[ { "path": ".devcontainer/devcontainer.json", "content": "{\n\t\"name\": \"PgSTAC\",\n\t\"dockerComposeFile\": \"../docker-compose.yml\",\n\t\"service\": \"pgstac\",\n\t\"workspaceFolder\": \"/opt/src\"\n}\n" }, { "path": ".dockerignore", "content": ".envrc*\n*/dist/*\n*.pyc\n*.egg-info\n*.eggs\nvenv/*\n*/.direnv/*\n*/.ruff_cache/*\n*/.pytest_cache/*\n*/.vscode/*\n*/.mypy_cache/*\n*/.pgadmin/*\n*/.ipynb_checkpoints/*\n*/.git/*\n*/.github/*\n*/env/*\nDockerfile\ndocker compose.yml\n*/.devcontainer/*\n" }, { "path": ".editorconfig", "content": "root = true\n\n[*]\nend_of_line = lf\ninsert_final_newline = true\ncharset = utf-8\ntrim_trailing_whitespace = true\n\n[*.sql]\nindent_style = space\nindent_size = 4\n\n[*.py]\nindent_style = space\nindent_size = 4\n\n[*.{yml,yaml}]\nindent_style = space\nindent_size = 2\n\n[*.md]\ntrim_trailing_whitespace = false\n\n[Makefile]\nindent_style = tab\n\n[*.sh]\nindent_style = space\nindent_size = 4\n" }, { "path": ".github/copilot-instructions.md", "content": "# Copilot Instructions for PgSTAC\n\nSee `CLAUDE.md` for comprehensive project instructions, architecture, and workflows.\nSee `AGENTS.md` for specialized agent definitions (sql-developer, migration-engineer, loader-developer).\n" }, { "path": ".github/dependabot.yml", "content": "version: 2\nupdates:\n - package-ecosystem: \"github-actions\"\n directory: \"/\"\n schedule:\n interval: \"weekly\"\n day: \"monday\"\n open-pull-requests-limit: 5\n groups:\n actions-all:\n applies-to: version-updates\n patterns:\n - \"*\"\n\n - package-ecosystem: \"docker\"\n directory: \"/\"\n schedule:\n interval: \"weekly\"\n day: \"monday\"\n open-pull-requests-limit: 5\n groups:\n docker-base-images:\n applies-to: version-updates\n patterns:\n - \"*\"\n\n - package-ecosystem: \"pip\"\n directory: \"/src/pypgstac\"\n schedule:\n interval: \"weekly\"\n day: \"monday\"\n open-pull-requests-limit: 5\n groups:\n python-dev-tooling:\n applies-to: version-updates\n patterns:\n - \"ruff\"\n - \"ty\"\n - \"pre-commit\"\n - \"types-*\"\n python-runtime:\n applies-to: version-updates\n patterns:\n - \"cachetools\"\n - \"fire\"\n - \"hydraters\"\n - \"orjson\"\n - \"plpygis\"\n - \"pydantic\"\n - \"python-dateutil\"\n - \"smart-open\"\n - \"tenacity\"\n - \"version-parser\"\n - \"psycopg*\"\n" }, { "path": ".github/instructions/migrations.instructions.md", "content": "---\napplyTo: \"src/pgstac/migrations/**\"\n---\n\n# Migration Files\n\nThese files are **generated** — see CLAUDE.md \"Migration Process\" for the full workflow.\n\n- **DO NOT** create, edit, or hand-modify migration files\n- Base (`pgstac.X.Y.Z.sql`) = full schema at that version\n- Incremental (`pgstac.X.Y.Z-A.B.C.sql`) = upgrade diff\n- Staged (`*.sql.staged`) = needs review before removing `.staged` suffix\n- Test: `scripts/test --migrations`\n" }, { "path": ".github/instructions/pypgstac.instructions.md", "content": "---\napplyTo: \"src/pypgstac/**\"\n---\n\n# pypgstac Python\n\nSee CLAUDE.md \"pypgstac Loader Internals\" for patterns. See AGENTS.md \"loader-developer\" for critical rules.\n\n- Uses psycopg v3 (not psycopg2), orjson (not json), tenacity, plpygis, fire\n- Materialize generators before retry boundaries\n- Query `partition_sys_meta` (live VIEW), never `partitions` (stale MATERIALIZED VIEW)\n- Test: `scripts/runinpypgstac --build test --pypgstac`\n" }, { "path": ".github/instructions/scripts.instructions.md", "content": "---\napplyTo: \"scripts/**\"\n---\n\n# Build Scripts\n\nSee CLAUDE.md \"Development Workflow\" for usage. All scripts require the Docker compose environment.\n\n- `runinpypgstac` is the foundation — most scripts delegate to it\n- `scripts/container-scripts/` contains the in-container script payload copied into the pypgstac image; keep host wrappers in `scripts/`\n- `stageversion` modifies version files AND generates migrations — see CLAUDE.md \"Migration Process\"\n- DO NOT run `stageversion` without understanding its side effects\n" }, { "path": ".github/instructions/sql-source.instructions.md", "content": "---\napplyTo: \"src/pgstac/sql/**\"\n---\n\n# SQL Source Files\n\nSee CLAUDE.md \"Critical Rules\" for full SQL conventions.\n\n- NEVER edit `pgstac.sql` — it is auto-generated\n- `CREATE OR REPLACE FUNCTION`, `IF NOT EXISTS`, `SECURITY DEFINER`\n- Grant permissions in `998_idempotent_post.sql`, not inline\n- `get_tstz_constraint()` regex must handle fractional seconds (`.` in timestamps)\n- Do NOT schema-qualify PostGIS calls — PostGIS may be in `public` or `postgis` schema\n- SQL functions used by GENERATED columns must be self-contained (no cross-function deps) — pg_dump orders functions alphabetically and breaks dependency chains\n- Test: `scripts/runinpypgstac --build test --pgtap --basicsql`\n" }, { "path": ".github/prompts/add-sql-function.prompt.md", "content": "---\ndescription: \"Add a new SQL function to PgSTAC\"\n---\n\nAdd a new SQL function following PgSTAC conventions:\n\n1. Determine the correct file in `src/pgstac/sql/` based on the prefix ranges in CLAUDE.md\n2. Use `CREATE OR REPLACE FUNCTION` in the `pgstac` schema\n3. Add `SECURITY DEFINER` if the function modifies tables\n4. Add permission grants in `src/pgstac/sql/998_idempotent_post.sql` if the function should be callable by `pgstac_ingest` or `pgstac_read`\n5. Add a test in `src/pgstac/tests/pgtap.sql` or a new `.sql`/`.sql.out` pair in `src/pgstac/tests/basic/`\n\nTest with: `scripts/runinpypgstac --build test --pgtap --basicsql`\n" }, { "path": ".github/prompts/debug-loader.prompt.md", "content": "---\ndescription: \"Debug a pypgstac data loading issue\"\n---\n\nDiagnose a pypgstac Loader problem:\n\n1. Check which load mode is being used (`insert`, `ignore`, `upsert`, `delsert`)\n2. Verify the loader queries `partition_sys_meta` (live VIEW), not `partitions` (stale MATERIALIZED VIEW)\n3. Check that generators are materialized to `list()` before `load_partition()` — generators can't survive tenacity retries\n4. Verify `item.pop(\"partition\", None)` uses `None` default for retry safety\n5. Check the retry decorator covers: `CheckViolation`, `DeadlockDetected`, `SerializationFailure`, `LockNotAvailable`, `ObjectInUse`\n6. Check `before_sleep` handler sets `partition.requires_update = True` on `CheckViolation`\n7. Verify `get_tstz_constraint()` regex handles fractional seconds (`.` in timestamps)\n\nKey files:\n- `src/pypgstac/src/pypgstac/load.py` — Loader class\n- `src/pgstac/sql/003b_partitions.sql` — partition constraint functions\n" }, { "path": ".github/prompts/stage-version.prompt.md", "content": "---\ndescription: \"Stage a new PgSTAC version and review the migration\"\n---\n\nGuide me through the PgSTAC release migration process:\n\n1. Confirm all SQL changes are in `src/pgstac/sql/` (never `pgstac.sql` directly)\n2. Run `scripts/stageversion {VERSION}` to generate base + incremental migrations\n3. Review the `.staged` migration file checking for:\n - Unintended `DROP TABLE` or `DROP COLUMN`\n - Unsafe `ALTER TABLE` for large tables\n - Bare `CREATE` instead of `CREATE OR REPLACE` for functions\n - Missing `IF NOT EXISTS` on indexes\n - Presence of `000_idempotent_pre.sql` and `998_idempotent_post.sql` content\n - `set_version()` called at the end\n4. Remove the `.staged` suffix\n5. Run `scripts/test --migrations` to validate the full migration chain\n6. Update CHANGELOG.md\n" }, { "path": ".github/workflows/continuous-integration.yml", "content": "name: CI\n\non:\n push:\n branches:\n - main\n pull_request:\n workflow_dispatch:\n schedule:\n - cron: '23 4 * * 0'\n\nenv:\n REGISTRY: ghcr.io\n IMAGE_NAME: ${{ github.repository }}\n DOCKER_BUILDKIT: 1\n PIP_BREAK_SYSTEM_PACKAGES: 1\n\njobs:\n changes:\n runs-on: ubuntu-latest\n permissions:\n pull-requests: read\n outputs:\n pgtagprefix: ${{ steps.check.outputs.pgtagprefix }}\n buildpgdocker: ${{ steps.check.outputs.buildpg }}\n pyrustdocker: ${{ steps.check.outputs.pytag }}\n buildpyrustdocker: ${{ steps.check.outputs.buildpy }}\n\n steps:\n - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2\n - uses: dorny/paths-filter@fbd0ab8f3e69293af611ebaee6363fc25e6d187d # v4.0.1\n id: filter\n with:\n filters: |\n pgstac:\n - 'docker/pgstac/**'\n pypgstac:\n - 'docker/pypgstac/**'\n - id: check\n run: |\n buildpg=false;\n ref=$(echo ${GITHUB_HEAD_REF:-${GITHUB_REF#refs/heads/}} | tr / _);\n pgref=$ref;\n [[ \"${{ steps.filter.outputs.pgstac }}\" == \"true\" ]] && buildpg=true || pgref=main;\n if [[ \"${{ github.event_name }}\" == \"schedule\" || \"${{ github.event_name }}\" == \"workflow_dispatch\" ]]; then\n buildpg=true;\n pgref=main;\n fi\n echo \"pgtagprefix=${{ env.REGISTRY }}/${GITHUB_REPOSITORY_OWNER}/pgstac-postgres:$pgref\" >>$GITHUB_OUTPUT;\n echo \"buildpg=$buildpg\" >>$GITHUB_OUTPUT;\n buildpy=false;\n pyref=$ref;\n [[ \"${{ steps.filter.outputs.pypgstac }}\" == \"true\" ]] && buildpy=true || pyref=main;\n if [[ \"${{ github.event_name }}\" == \"schedule\" || \"${{ github.event_name }}\" == \"workflow_dispatch\" ]]; then\n buildpy=true;\n pyref=main;\n fi\n echo \"pytag=${{ env.REGISTRY }}/${GITHUB_REPOSITORY_OWNER}/pgstac-pyrust:$pyref\" >>$GITHUB_OUTPUT;\n echo \"buildpy=$buildpy\" >>$GITHUB_OUTPUT;\n\n # This builds a base postgres image that has everything installed to be able to run pgstac. This image does not have pgstac itself installed.\n buildpg:\n name: Build and push base postgres image\n runs-on: ubuntu-latest\n needs: [changes]\n strategy:\n matrix:\n pg_major: [16, 17, 18]\n steps:\n - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2\n - uses: docker/setup-buildx-action@4d04d5d9486b7bd6fa91e7baf45bbb4f8b9deedd # v4.0.0\n - name: Log in to the Container registry\n uses: docker/login-action@4907a6ddec9925e35a0a9e82d7399ccc52663121 # v4.1.0\n with:\n registry: ${{ env.REGISTRY }}\n username: ${{ github.actor }}\n password: ${{ secrets.GITHUB_TOKEN }}\n - name: Build and Push Base Postgres\n if: ${{ needs.changes.outputs.buildpgdocker == 'true' }}\n uses: docker/build-push-action@bcafcacb16a39f128d818304e6c9c0c18556b85f # v7.1.0\n with:\n platforms: linux/amd64,linux/arm64\n context: .\n target: pgstacbase\n file: docker/pgstac/Dockerfile\n build-args: |\n PG_MAJOR=${{ matrix.pg_major }}\n tags: ${{ needs.changes.outputs.pgtagprefix }}-pg${{ matrix.pg_major }}\n push: true\n cache-from: type=gha\n cache-to: type=gha, mode=max\n\n buildpyrust:\n name: Build and push base pyrust\n runs-on: ubuntu-latest\n needs: [changes]\n steps:\n - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2\n - uses: docker/setup-buildx-action@4d04d5d9486b7bd6fa91e7baf45bbb4f8b9deedd # v4.0.0\n - name: Log in to the Container registry\n uses: docker/login-action@4907a6ddec9925e35a0a9e82d7399ccc52663121 # v4.1.0\n with:\n registry: ${{ env.REGISTRY }}\n username: ${{ github.actor }}\n password: ${{ secrets.GITHUB_TOKEN }}\n - name: Build and Push Base pyrust\n if: ${{ needs.changes.outputs.buildpyrustdocker == 'true' }}\n uses: docker/build-push-action@bcafcacb16a39f128d818304e6c9c0c18556b85f # v7.1.0\n with:\n platforms: linux/amd64,linux/arm64\n context: .\n target: pyrustbase\n file: docker/pypgstac/Dockerfile\n tags: ${{ needs.changes.outputs.pyrustdocker }}\n push: true\n cache-from: type=gha\n cache-to: type=gha, mode=max\n\n test:\n name: test\n needs: [changes, buildpg, buildpyrust]\n runs-on: ubuntu-latest\n strategy:\n matrix:\n pg_major: [16, 17, 18]\n flags:\n - \"\"\n - \"--resolution lowest-direct\"\n container:\n image: ${{ needs.changes.outputs.pyrustdocker }}\n options: --user root\n env:\n PGPASSWORD: postgres\n PGHOST: postgres\n PGDATABASE: postgres\n PGUSER: postgres\n UV_CACHE_DIR: /tmp/.uv-cache\n services:\n postgres:\n env:\n POSTGRES_PASSWORD: postgres\n image: ${{ needs.changes.outputs.pgtagprefix }}-pg${{ matrix.pg_major }}\n options: >-\n --health-cmd pg_isready\n --health-interval 10s\n --health-timeout 5s\n --health-retries 5\n steps:\n - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2\n - name: Ensure PostgreSQL client tools\n run: |\n set -euo pipefail\n apt-get update\n apt-get install -y --no-install-recommends gnupg ca-certificates curl\n install -d -m 0755 /etc/apt/keyrings\n curl -fsSL https://www.postgresql.org/media/keys/ACCC4CF8.asc | gpg --dearmor -o /etc/apt/keyrings/postgresql.gpg\n . /etc/os-release\n echo \"deb [signed-by=/etc/apt/keyrings/postgresql.gpg] http://apt.postgresql.org/pub/repos/apt ${VERSION_CODENAME}-pgdg main\" > /etc/apt/sources.list.d/pgdg.list\n apt-get update\n apt-get install -y --no-install-recommends \"postgresql-client-${{ matrix.pg_major }}\"\n\n client_major=\"$(pg_dump --version | sed -E 's/.* ([0-9]+)\\..*/\\1/')\"\n server_major=\"$(psql -X -tA -h postgres -d postgres -c 'show server_version' | sed -E 's/^([0-9]+).*/\\1/')\"\n if [[ \"$client_major\" != \"${{ matrix.pg_major }}\" ]]; then\n echo \"Expected pg_dump major ${{ matrix.pg_major }}, got ${client_major}\" >&2\n exit 1\n fi\n if [[ \"$client_major\" != \"$server_major\" ]]; then\n echo \"pg_dump major (${client_major}) does not match postgres major (${server_major})\" >&2\n exit 1\n fi\n - uses: astral-sh/setup-uv@08807647e7069bb48b6ef5acd8ec9567f424441b # v8.1.0\n - name: Install pypgstac\n working-directory: /__w/pgstac/pgstac/src/pypgstac\n run: |\n export UV_CACHE_DIR=/tmp/.uv-cache\n export XDG_CACHE_HOME=/tmp/.cache\n mkdir -p \"$UV_CACHE_DIR\" \"$XDG_CACHE_HOME\"\n uv --cache-dir \"$UV_CACHE_DIR\" venv /tmp/ci-venv\n uv --cache-dir \"$UV_CACHE_DIR\" pip install --python /tmp/ci-venv/bin/python ${{ matrix.flags }} .[dev,test,psycopg]\n echo \"/tmp/ci-venv/bin\" >> \"$GITHUB_PATH\"\n - name: Run tests\n working-directory: /__w/pgstac/pgstac\n run: scripts/container-scripts/test\n" }, { "path": ".github/workflows/deploy_mkdocs.yml", "content": "name: Publish docs via GitHub Pages\n\non:\n push:\n branches:\n - main\n paths:\n - 'README.md'\n - 'CHANGELOG.md'\n - 'CONTRIBUTING.md'\n - 'docs/**'\n pull_request:\n paths:\n - 'README.md'\n - 'CHANGELOG.md'\n - 'CONTRIBUTING.md'\n - 'docs/**'\n\njobs:\n docs:\n name: ${{ github.event_name == 'push' && 'Deploy docs' || 'Build docs' }}\n runs-on: ubuntu-latest\n steps:\n - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2\n\n - name: Set up Python 3.12\n uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0\n with:\n python-version: 3.12\n\n - name: Install dependencies\n run: |\n python -m pip install --upgrade pip\n python -m pip install mkdocs mkdocs-material mkdocs-jupyter pandas seaborn folium\n\n - name: Build docs\n if: github.event_name == 'pull_request'\n run: mkdocs build -f docs/mkdocs.yml\n\n - name: Deploy docs\n if: github.event_name == 'push'\n run: mkdocs gh-deploy --force -f docs/mkdocs.yml\n" }, { "path": ".github/workflows/release.yml", "content": "name: Release\n\non:\n push:\n tags:\n - \"*\"\n\nenv:\n REGISTRY: ghcr.io\n IMAGE_NAME: ${{ github.repository }}\n\njobs:\n tag:\n name: tag\n runs-on: ubuntu-latest\n steps:\n - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2\n - name: Tag Release\n uses: \"marvinpinto/action-automatic-releases@919008cf3f741b179569b7a6fb4d8860689ab7f0\" # v1.2.1\n with:\n repo_token: \"${{ secrets.GITHUB_TOKEN }}\"\n prerelease: false\n\n # This builds a base postgres image that has everything installed to be able to run pgstac.\n buildpg:\n name: Build and push base postgres image\n runs-on: ubuntu-latest\n steps:\n - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2\n - uses: docker/setup-qemu-action@ce360397dd3f832beb865e1373c09c0e9f86d70a # v4.0.0\n # Temporary workaround for ARM builds\n # https://github.com/docker/setup-qemu-action/issues/198\n with:\n image: tonistiigi/binfmt:qemu-v7.0.0-28\n - uses: docker/setup-buildx-action@4d04d5d9486b7bd6fa91e7baf45bbb4f8b9deedd # v4.0.0\n - name: Log in to the Container registry\n uses: docker/login-action@4907a6ddec9925e35a0a9e82d7399ccc52663121 # v4.1.0\n with:\n registry: ${{ env.REGISTRY }}\n username: ${{ github.actor }}\n password: ${{ secrets.GITHUB_TOKEN }}\n\n - name: Extract metadata (tags, labels) for Docker\n id: meta\n uses: docker/metadata-action@030e881283bb7a6894de51c315a6bfe6a94e05cf # v6.0.0\n with:\n images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}-postgres\n - name: Build and Push Base Postgres\n uses: docker/build-push-action@bcafcacb16a39f128d818304e6c9c0c18556b85f # v7.1.0\n with:\n platforms: linux/amd64,linux/arm64\n context: .\n target: pgstacbase\n file: docker/pgstac/Dockerfile\n tags: ${{ steps.meta.outputs.tags }}\n labels: ${{ steps.meta.outputs.labels }}\n push: true\n cache-from: type=gha\n cache-to: type=gha, mode=max\n\n # This builds a postgres image that already has pgstac installed to the tagged version\n buildpgstac:\n name: Build and push base postgres with pgstac installed\n runs-on: ubuntu-latest\n steps:\n - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2\n - uses: docker/setup-qemu-action@ce360397dd3f832beb865e1373c09c0e9f86d70a # v4.0.0\n # Temporary workaround for ARM builds\n # https://github.com/docker/setup-qemu-action/issues/198\n with:\n image: tonistiigi/binfmt:qemu-v7.0.0-28\n - uses: docker/setup-buildx-action@4d04d5d9486b7bd6fa91e7baf45bbb4f8b9deedd # v4.0.0\n - name: Log in to the Container registry\n uses: docker/login-action@4907a6ddec9925e35a0a9e82d7399ccc52663121 # v4.1.0\n with:\n registry: ${{ env.REGISTRY }}\n username: ${{ github.actor }}\n password: ${{ secrets.GITHUB_TOKEN }}\n\n - name: Extract metadata (tags, labels) for Docker\n id: meta\n uses: docker/metadata-action@030e881283bb7a6894de51c315a6bfe6a94e05cf # v6.0.0\n with:\n images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}\n - name: Build and Push Base Postgres\n uses: docker/build-push-action@bcafcacb16a39f128d818304e6c9c0c18556b85f # v7.1.0\n with:\n platforms: linux/amd64,linux/arm64\n context: .\n target: pgstac\n file: docker/pgstac/Dockerfile\n tags: ${{ steps.meta.outputs.tags }}\n labels: ${{ steps.meta.outputs.labels }}\n push: true\n cache-from: type=gha\n cache-to: type=gha, mode=max\n\n buildpyrust:\n name: Build and push base pyrust\n runs-on: ubuntu-latest\n steps:\n - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2\n - uses: docker/setup-qemu-action@ce360397dd3f832beb865e1373c09c0e9f86d70a # v4.0.0\n # Temporary workaround for ARM builds\n # https://github.com/docker/setup-qemu-action/issues/198\n with:\n image: tonistiigi/binfmt:qemu-v7.0.0-28\n - uses: docker/setup-buildx-action@4d04d5d9486b7bd6fa91e7baf45bbb4f8b9deedd # v4.0.0\n - name: Log in to the Container registry\n uses: docker/login-action@4907a6ddec9925e35a0a9e82d7399ccc52663121 # v4.1.0\n with:\n registry: ${{ env.REGISTRY }}\n username: ${{ github.actor }}\n password: ${{ secrets.GITHUB_TOKEN }}\n\n - name: Extract metadata (tags, labels) for Docker\n id: meta\n uses: docker/metadata-action@030e881283bb7a6894de51c315a6bfe6a94e05cf # v6.0.0\n with:\n images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}-pyrust\n - name: Build and Push Base Postgres\n uses: docker/build-push-action@bcafcacb16a39f128d818304e6c9c0c18556b85f # v7.1.0\n with:\n platforms: linux/amd64,linux/arm64\n context: .\n target: pyrustbase\n file: docker/pypgstac/Dockerfile\n tags: ${{ steps.meta.outputs.tags }}\n labels: ${{ steps.meta.outputs.labels }}\n push: true\n cache-from: type=gha\n cache-to: type=gha, mode=max\n\n buildpypgstac:\n name: Build and push base pyrust\n runs-on: ubuntu-latest\n steps:\n - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2\n - uses: docker/setup-qemu-action@ce360397dd3f832beb865e1373c09c0e9f86d70a # v4.0.0\n # Temporary workaround for ARM builds\n # https://github.com/docker/setup-qemu-action/issues/198\n with:\n image: tonistiigi/binfmt:qemu-v7.0.0-28\n - uses: docker/setup-buildx-action@4d04d5d9486b7bd6fa91e7baf45bbb4f8b9deedd # v4.0.0\n - name: Log in to the Container registry\n uses: docker/login-action@4907a6ddec9925e35a0a9e82d7399ccc52663121 # v4.1.0\n with:\n registry: ${{ env.REGISTRY }}\n username: ${{ github.actor }}\n password: ${{ secrets.GITHUB_TOKEN }}\n\n - name: Extract metadata (tags, labels) for Docker\n id: meta\n uses: docker/metadata-action@030e881283bb7a6894de51c315a6bfe6a94e05cf # v6.0.0\n with:\n images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}-pypgstac\n - name: Build and Push Base Postgres\n uses: docker/build-push-action@bcafcacb16a39f128d818304e6c9c0c18556b85f # v7.1.0\n with:\n platforms: linux/amd64,linux/arm64\n context: .\n target: pypgstac\n file: docker/pypgstac/Dockerfile\n tags: ${{ steps.meta.outputs.tags }}\n labels: ${{ steps.meta.outputs.labels }}\n push: true\n cache-from: type=gha\n cache-to: type=gha, mode=max\n\n buildpypgstacruntime:\n name: Build and push pypgstac runtime image\n runs-on: ubuntu-latest\n steps:\n - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2\n - uses: docker/setup-qemu-action@ce360397dd3f832beb865e1373c09c0e9f86d70a # v4.0.0\n with:\n image: tonistiigi/binfmt:qemu-v7.0.0-28\n - uses: docker/setup-buildx-action@4d04d5d9486b7bd6fa91e7baf45bbb4f8b9deedd # v4.0.0\n - name: Log in to the Container registry\n uses: docker/login-action@4907a6ddec9925e35a0a9e82d7399ccc52663121 # v4.1.0\n with:\n registry: ${{ env.REGISTRY }}\n username: ${{ github.actor }}\n password: ${{ secrets.GITHUB_TOKEN }}\n\n - name: Extract metadata (tags, labels) for Docker\n id: meta\n uses: docker/metadata-action@030e881283bb7a6894de51c315a6bfe6a94e05cf # v6.0.0\n with:\n images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}-pypgstac-runtime\n - name: Build and Push pypgstac runtime\n uses: docker/build-push-action@bcafcacb16a39f128d818304e6c9c0c18556b85f # v7.1.0\n with:\n platforms: linux/amd64,linux/arm64\n context: .\n target: pypgstac-runtime\n file: docker/pypgstac/Dockerfile\n tags: ${{ steps.meta.outputs.tags }}\n labels: ${{ steps.meta.outputs.labels }}\n push: true\n cache-from: type=gha\n cache-to: type=gha, mode=max\n\n releasetopypi:\n name: Release\n runs-on: ubuntu-latest\n permissions:\n id-token: write\n environment:\n name: pypi\n url: https://pypi.org/p/pypgstac\n steps:\n - name: Checkout\n uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2\n - name: Setup Python\n uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0\n with:\n python-version: \"3.x\"\n - name: Install build\n working-directory: /home/runner/work/pgstac/pgstac/src/pypgstac\n run: pip install build\n - name: Build\n working-directory: /home/runner/work/pgstac/pgstac/src/pypgstac\n run: python -m build\n - name: Publish to PyPI\n uses: pypa/gh-action-pypi-publish@cef221092ed1bacb1cc03d23a2d87d1d172e277b # release/v1\n with:\n packages-dir: /home/runner/work/pgstac/pgstac/src/pypgstac/dist\n" }, { "path": ".gitignore", "content": ".so\n.envrc\nsrc/pypgstac/dist\n*.pyc\n*.egg-info\n*.eggs\nvenv\nenv\n.direnv\nsrc/pypgstac/target\nsrc/pypgstac/python/pypgstac/*.so\n.vscode\n.ipynb_checkpoints\n.venv\n.pytest_cache\n.plans/\n.env\nsrc/pgstacrust/target/\n" }, { "path": ".pre-commit-config.yaml", "content": "# See https://pre-commit.com for more information\n# See https://pre-commit.com/hooks.html for more hooks\nrepos:\n- repo: https://github.com/pre-commit/pre-commit-hooks\n rev: v5.0.0\n hooks:\n - id: trailing-whitespace\n - id: check-yaml\n - id: check-added-large-files\n - id: check-toml\n - id: detect-aws-credentials\n args: [--allow-missing-credential]\n - id: detect-private-key\n - id: check-json\n - id: mixed-line-ending\n - id: check-merge-conflict\n - id: check-executables-have-shebangs\n - id: check-symlinks\n\n- repo: local\n hooks:\n - id: dockerbuild\n name: dockerbuild\n entry: scripts/update\n language: script\n pass_filenames: false\n verbose: true\n fail_fast: true\n files: Dockerfile$|\\.rs$\n - id: sql\n name: sql\n entry: scripts/test\n args: [--basicsql, --pgtap]\n language: script\n pass_filenames: false\n verbose: true\n fail_fast: true\n files: sql\\/.*\\.sql$\n - id: formatting\n name: formatting\n entry: scripts/test\n args: [--formatting]\n language: script\n pass_filenames: false\n verbose: true\n fail_fast: true\n files: ^(src/pypgstac/(src/pypgstac|tests)/.*\\.py|src/pypgstac/pyproject\\.toml)$\n - id: pypgstac\n name: pypgstac\n entry: scripts/test\n args: [--pypgstac]\n language: script\n pass_filenames: false\n verbose: true\n fail_fast: true\n files: pypgstac\\/.*\\.py$\n - id: migrations\n name: migrations\n entry: scripts/test\n args: [--migrations]\n language: script\n pass_filenames: false\n verbose: true\n fail_fast: true\n files: migrations\\/.*\\.sql$\n" }, { "path": "AGENTS.md", "content": "# PgSTAC Agents\n\n## sql-developer\n\nPostgreSQL SQL developer for PgSTAC. Works exclusively in `src/pgstac/sql/` files. See CLAUDE.md for full SQL rules, file map, and partition architecture.\n\n### Key Constraints\n\n- NEVER edit `pgstac.sql` — it is auto-generated\n- `CREATE OR REPLACE FUNCTION`, `IF NOT EXISTS`, `SECURITY DEFINER` for data-modifying functions\n- Grant permissions in `998_idempotent_post.sql`, not inline\n- Use `run_or_queue()` for deferrable operations\n- Do NOT schema-qualify PostGIS calls (PostGIS may be in `public` or `postgis` schema)\n- Avoid cross-function deps in SQL functions used by GENERATED columns — pg_dump orders alphabetically, so inline the logic (see `search_hash` pattern)\n- Test: `scripts/runinpypgstac --build test --pgtap --basicsql`\n\n---\n\n## migration-engineer\n\nMigration specialist for PgSTAC. See CLAUDE.md \"Migration Process\" for full workflow.\n\n### Quick Reference\n\n1. Edit SQL in `src/pgstac/sql/*.sql`\n2. `scripts/stageversion VERSION` → generates base + incremental `.staged` migration\n3. Review `.staged` file (watch for DROPs, unsafe ALTERs, missing `CREATE OR REPLACE`)\n4. Remove `.staged` suffix → `scripts/test --migrations`\n\n### Review Checklist\n\n- No unintended `DROP TABLE/COLUMN`, safe `ALTER TABLE` for large tables\n- `CREATE OR REPLACE` (not bare `CREATE`), `IF NOT EXISTS` for indexes\n- `000_idempotent_pre.sql` and `998_idempotent_post.sql` included\n- `set_version()` called at end\n\n---\n\n## loader-developer\n\nSpecialist in pypgstac bulk loading (`src/pypgstac/src/pypgstac/load.py`). See CLAUDE.md \"pypgstac Loader Internals\" for full details.\n\n### Critical Patterns\n\n- **Materialize generators**: `list(g)` before `load_partition()` — generators can't survive tenacity retries\n- **Live view only**: Query `partition_sys_meta` (VIEW), never `partitions` (stale MATERIALIZED VIEW)\n- **Retry safety**: `item.pop(\"partition\", None)` with `None` default; `before_sleep` sets `partition.requires_update = True` on `CheckViolation`\n- **Retry scope**: `CheckViolation`, `DeadlockDetected`, `SerializationFailure`, `LockNotAvailable`, `ObjectInUse`\n- **Load modes**: `insert`, `ignore`/`insert_ignore`, `upsert`, `delsert`\n- Test: `scripts/runinpypgstac --build test --pypgstac`\n" }, { "path": "CHANGELOG.md", "content": "# Changelog\n\nAll notable changes to this project will be documented in this file.\n\nThe format is based on [Keep a Changelog](http://keepachangelog.com/)\nand this project adheres to [Semantic Versioning](http://semver.org/).\n\n\n## [Unreleased]\n\n### Added\n- `scripts/makemigration` host wrapper for the in-container `makemigration` helper.\n- `.env.example` documenting all supported environment variables for local development.\n- All host-facing scripts (`test`, `format`, `migrate`, `server`, `stageversion`,\n `runinpypgstac`, `console`) now accept `--help` / `-h` and honor environment-variable\n counterparts for common flags (`PGSTAC_BUILD_POLICY`, `PGSTAC_FAST`, `PGSTAC_WATCH`,\n `PGSTAC_STRICT`).\n- `scripts/pgstacenv` gains `ensure_env_file` (auto-creates `.env` from `.env.example`\n on first run) and `first_available_pgport` (avoids port collisions on shared machines).\n- `scripts/test` expanded from a 3-line wrapper to a full-featured test runner supporting\n `--fast`, `--watch`, `--build-policy`, `--no-strict`, and stale-image detection.\n- PostgreSQL 16, 17, and 18-beta added to the CI and Docker build matrix. (Closes #334)\n- Weekly scheduled CI run (`cron: '23 4 * * 0'`) to catch upstream base-image CVEs without\n requiring a code change. (Closes #202)\n- `workflow_dispatch` trigger for manual CI runs.\n- `pg_tle` v1.5.2 built and pre-loaded in the `pgstacbase` image; database init runs\n `CREATE EXTENSION IF NOT EXISTS pg_tle`.\n- `pypgstac-runtime` Docker target: slim Python 3.13-trixie image without the Rust/build\n toolchain, for production deployments where the Rust build environment is not needed.\n- Dependabot coverage expanded to Docker base images and pip packages (two new\n ecosystems with grouped update policies).\n\n### Changed\n- In-container helper scripts moved from `docker/pypgstac/bin/` to\n `scripts/container-scripts/`; container `PATH` updated accordingly.\n- `docker/pgstac/Dockerfile` and `docker/pypgstac/Dockerfile` base images updated from\n `bullseye` to `trixie`. (Closes #231)\n- All Docker `RUN` layers now use BuildKit cache mounts for apt, uv, and git caches,\n significantly reducing incremental rebuild times.\n- `docker-compose.yml`: adds `env_file: .env`, explicit `PGHOST`/`PGPORT` defaults,\n a pgstac healthcheck, and a `service_healthy` dependency on pypgstac.\n- `runinpypgstac` gains `--build-policy {always,missing,never}` replacing the bare\n `--build` flag; `PGSTAC_BUILD_POLICY` env var provides a persistent default.\n- Dev tooling: `flake8`, `black`, and `mypy` removed in favour of `ruff==0.15.11` and\n `ty==0.0.31`. `pre-commit` pinned to `3.5.0`. `pre-commit-hooks` updated to v5.0.0.\n- `pypgstac` package floor raised to Python 3.11; metadata now advertises 3.11-3.14.\n- `pypgstac` settings now use `pydantic-settings` (`BaseSettings` from\n `pydantic_settings`) and require `pydantic>=2,<3`.\n- `cachetools` upper bound removed (`cachetools>=5.3.0`) since `pypgstac` only uses\n `cachetools.func.lru_cache`; no known incompatible API changes affect this usage.\n- `pypgstac` developer tooling config now consistently targets Ruff + ty:\n removes stale mypy config, pins Ruff to `0.15.11` to match pre-commit,\n and adds minimal `[tool.ty]` project settings.\n- Formatting/type-check pipeline now uses `scripts/test --formatting` as the\n single pre-commit entry point (removing duplicate direct Ruff pre-commit hooks)\n and aligns Ruff line-length handling with the formatter (`E501` ignored;\n explicit `line-length = 88`).\n- GitHub Actions updated: `dorny/paths-filter` v2→v3, `docker/build-push-action`\n v4→v6, `astral-sh/setup-uv` v8.0.0→v8.1.0; all SHA pins refreshed.\n- Dependabot groups reworked: `actions-all` (replaces `minor-and-patch`), new\n `docker-base-images`, `python-dev-tooling`, and `python-runtime` groups.\n- `docker-compose.yml` removes explicit `container_name` entries to avoid conflicts\n between concurrent local instances.\n\n### Removed\n- PL/Rust support: `pgstacbase-plrust` and `pgstac-plrust` Docker targets removed; the\n pgstac image no longer builds or ships PL/Rust or the Rust toolchain. (Closes #339)\n- `flake8`, `black`, and `mypy` removed from dev dependencies.\n\n### Fixed\n- `load.py`: Use timezone-aware `MIN_DATETIME_UTC` / `MAX_DATETIME_UTC` sentinel\n constants (instead of naive `datetime.min` / `datetime.max`) to avoid\n `TypeError: can't compare offset-naive and offset-aware datetimes`.\n- CI `lowest-direct` dependency install: avoid uv cache permission failures by using\n a writable temp cache path in the test job install step.\n- `pypgstac` dependency floor for `orjson` raised to `>=3.11.0` to avoid selecting\n the broken `3.9.0` sdist under `--resolution lowest-direct`.\n- `pydantic` minimum raised to `>=2.10` so `--resolution lowest-direct` on Python 3.13\n does not resolve to `pydantic-core==2.0.1`, which fails to build.\n\n\n## [v0.9.11]\n\n\n### Fixed\n- Fix timestamp regex in partition constraint parsing to handle fractional seconds (microseconds), preventing incorrect `(-infinity, infinity)` constraint bounds.\n- Add explicit ANALYZE before `st_estimatedextent()` in `update_partition_stats` for deterministic spatial extent calculation.\n- Consolidate materialized view refreshes in `update_partition_stats` to a single unconditional refresh, reducing redundant operations.\n- Use `partition_sys_meta` (live VIEW) instead of `partitions` (stale MATERIALIZED VIEW) in loader `_partition_update()` for real-time partition bounds.\n- Expand loader retry to 10 attempts and add `SerializationFailure`, `LockNotAvailable`, `ObjectInUse` to retryable exceptions.\n- Add `before_sleep` retry handler to force partition constraint refresh on `CheckViolation`.\n- Materialize `itertools.groupby` generators with `list()` before `load_partition()` to prevent silent data loss on retry.\n- Use safe `item.pop('partition', None)` to avoid `KeyError` on retry.\n- Inline `search_tohash` into `search_hash` to eliminate cross-function dependency that broke pg_dump/pg_restore (pg_dump orders functions alphabetically).\n\n### Added\n- `pgstac_restore` script for restoring pg_dump backups — installs a temporary event trigger to fix search_path during restore.\n- Race condition tests for sequential and concurrent loader operations with new-Loader-per-item pattern.\n- Search path independence tests verifying `partition_sys_meta`, `partition_stats`, `partitions`, `partitions_view`, and `partition_steps` work identically with and without `pgstac` in `search_path`.\n- pg_dump/pg_restore test (`--pgdump`) validating backup and restore of a pgstac database with sample data.\n- Documentation for pg_dump/pg_restore best practices with PgSTAC.\n\n## [v0.9.10]\n\n### Fixed\n- Improved performance and correctness of partition constraint parsing.\n\n##[v0.9.9]\n\n### Changed\n* changed container images to use non-root `user`\n* fix bug where closed and broken db connections are reused.\n\n### Fixed\n* replace space-separated terms with adjacency operator in free-text search (#387)\n\n## [v0.9.8]\n\n### Fixed\n- Allow array as q parameter for full text search\n\n\n## [v0.9.7]\n\n### Fixed\n- Fix bad handling of leading +/- terms in free-text search\n- Use consistent tsquery config in free-text search\n\n## [v0.9.6]\n\n### Added\n\n- Add `load_queryables` function to pypgstac for loading queryables from a JSON file\n- Add support for specifying collection IDs when loading queryables\n\n### Fixed\n- Added missing 0.8.6-0.9.0 migration script\n\n## [v0.9.5]\n\n### Changed\n\n - Pin to `plpygic>=0.5.0` and use `geom.ewkb` instead of `geom.wkt` when formatting items in `Loader.format_item`. Fixes (#357)\n\n## [v0.9.4]\n\n### Changed\n - Relax pypgstac dependencies\n\n## [v0.9.3]\n\n### Fixed\n\n- Fix CI issue with tests not running\n- Fix for issue with nulls in title or keywords for free text search\n\n### Changed\n\n- Replace hardcoded org name in CI\n\n## [v0.9.2]\n\n### Added\n\n- Add limited support for free-text search in the search functions. (Fixes #293)\n - the `q` parameter is converted from the\n [OGC API - Features syntax](https://docs.ogc.org/DRAFTS/24-031.html) into a `tsquery`\n statement which is used to compare to the description, title, and keywords fields in items or collection_search\n - the text search is un-indexed and will be very slow for item-level searches!\n - Add support for Postgres 17\n - Support for adding data to the private field using the pypgstac loader\n\n### Fixed\n\n- Add `open=True` in `psycopg.ConnectionPool` to avoid future behavior change\n- Switch from postgres `server_version` to `server_version_num` to get PG version (Fixes #300)\n- Allow read-only replicas work even when the context extension is enabled (Fixes #300)\n- Consistently ensure use of instantiated postgres fields when addressing with 'properties.' prefix\n\n### Changed\n- Move rust hydration to a separate repo\n\n## [v0.9.1]\n\n### Fixed\n\n- Fixed double nested extent when using trigger based update collection extent. (Fixes #274)\n- Fix time formatting (Fixes #275)\n- Relaxes smart-open dependency check (Fixes #273)\n- Switch to uv for docker image\n\n## [v0.9.0]\n\n### Breaking Changes\n\n- Context Extension has been deprecated. Context is now reported using OGC Features compliant numberMatched and numberReturned\n- Paging return from search using prev/next properties has been deprecated. Paging is now available in the spec compliant Links\n\n### Added\n\n- Add support for Casei and Accenti (Fixes #237). (Also, requires the addition of the unaccent extension)\n- Add numberReturned and numberMatched fields for ItemCollection. BREAKING CHANGE: As the context extension is deprecated, this also removes the \"context\" item from results.\n- Updated docs on automated updates of collection extents. (CLOSES #247)\n- stac search now returns paging information using standards compliant links rather than prev/next properties (Fixes #265)\n\n### Fixed\n\n- Fixes issue when there is a None rather than an empty dictionary in hydration.\n- Use \"debug\" log level rather than \"log\" to prevent growth in log messages due to differences in how client_min_messages and log_min_messages treat log levels. (Fixes #242)\n- Refactor search_query and search_where functions to eliminate race condition when running identical queries. (Fixes #233)\n- Fixes CQL2 Parser for Between operator (Fixes #251)\n- Update PyO3 for rust hydration performance improvements.\n\n## [v0.8.6]\n\n### Fixed\n\n - Relax version requirement for smart-open (Fixes #273)\n - Use uv pip in docker build\n\n## [v0.8.5]\n\n### Fixed\n\n- Fix issue when installing or migrating pgstac using a non superuser (particularly when using the default role found on RDS). (FIXES #239). Backports fix into migrations for 0.8.2, 0.8.3, and 0.8.4.\n- Adds fixes/updates to documentation\n- Fixes issue when using geometry with the strict queryables setting set.\n\n## [v0.8.4]\n\n### Fixed\n\n- Make release deployment use postgres images without plrust\n- Update versions of plrust in dockerfile (used for development, there is no plrust code yet)\n- Update incremental migration tests to start at v0.3.0 rather than v0.1.9 due to a breaking change in pg_partman at version 5 that has no ability to pin a version. Migrating from prior to v0.3.0 should still work fine as long as pg_partman has not been updated on the database.\n\n## [v0.8.3]\n\n### Added\n\n- Add support for arm64 to Docker images\n\n### Fixed\n\n- Fixes a critical bug when using the ingest_staging_upsert table or the upsert_item/upsert_items functions to update records with existing data where the existing row would get deleted, but the new row would not get added.\n\n## [v0.8.2]\n\n### Added\n\n- Add support functions and tests for Collection Search\n- Add configuration parameter for base_url to be able to generate absolute links\n- With this release, this is only used to create links for paging in collection_search\n- Adds read only mode to allow use of pgstac on read replicas\n- Note: Turning on romode disables any caching (particularly when context is turned on) and does not allow to store q query hash that can be used with geometry_search.\n- Add option to pypgstac loader \"--usequeue\" that forces use of the query queue for the loading process\n- Add \"pypgstac runqueue\" command to run any commands that are set in the query queue\n\n### Fixed\n\n- Fix bug with end_datetime constraint management leading to inability to add data outside of constraints\n- Fix bugs dealing with table ownership to ensure that all pgstac tables are owned by the pgstac_admin role\n- Fixes issues with errors/warnings caused when doing index maintenance\n- Fixes issues with errors/warnings caused with partition management\n- Make sure that pgstac_ingest role always has read/write permissions on all tables\n- Remove call to create_table_constraints from check_partition function. create_table_constraints was being called twice as it also gets called from update_partition_stats\n- Add NOT NULL constraint to collections table (FIXES #224)\n- Fix issue with indexes not getting created as the pg_admin role using SECURITY DEFINER\n\n### Changed\n\n- Revert pydantic requirement back to '>=1.7' and use basesettings conditionally from pydantic or pydantic.v1 to allow compatibility with pydantic 2 as well as with stac-fastapi that requires pydantic <2\n\n## [v0.8.1]\n\n### Fixed\n\n- Fix issue with CI building/pushing docker images\n\n## [v0.8.0]\n\n### Fixed\n\n- Revert an optimisation which limited the number of results from a search query to the number of item IDs specified in the query.\nThis fixes an issue where items with the same ID that are in multiple collections could be left out of search results.\n\n### Changed\n\n- update `pydantic` requirement to `~=2.0`\n- update docker and ci workflows to build binary wheels for rust additions to pypgstac\n- split docker into database service and python/rust container\n- Modify scripts to auto-generate unreleased migration\n- Add pre commit tasks to generate migration and to rebuild and compile pypgstac with maturin for rust\n- Add private jsonb column to items and collections table to hold private metadata that should not be returned as part of a stac item\n- Add generated columns to collections with the bounding box as a geometry and the datetime and end_datetime from the extents (this is to help with forthcoming work on collections search)\n- Add PLRust to the Docker postgres image for forthcoming work to add optional PLRust functions for expensive json manipulation (including hydration)\n- Remove default queryable for eo:cloud_cover\n\n## [v0.7.10]\n\n### Fixed\n\n- Return an empty jsonb array from all_collections() when the collections table is empty, instead of NULL. Fixes #186.\n- Add delete trigger to collections to clean up partition_stats records and remove any partitions. Fixes #185\n- Fixes boolean casting in get_setting_bool function\n\n## [v0.7.9]\n\n### Fixed\n\n- Update docker image to use postgis 3.3.3\n\n## [v0.7.8]\n\n### Fixed\n\n- Fix issue with search_query not returning all fields on first use of a query. Fixes #182\n\n## [v0.7.7]\n\n### Fixed\n\n- Fix migrations for 0.7.4->0.7.5 and 0.7.5->0.7.6 to use the partition_view rather than the materialized view to avoid issue with refreshing the materialized view when run in the same statement that is accessing the view. Fixes #180.\n\n### Added\n\n- Add a short cirucit for id searches that sets the limit to be no more than the number of ids in the filter.\n- Add 'timing' configuration variable that adds a \"timing\" element to the return object with the amount of time that it took to return a search.\n- Reduce locking when updating statistics in the search table. Use skip locked to skip updating last_used and count when there is a lock being held.\n\n## [v0.7.6]\n\n### Fixed\n\n- Fix issue with checking for existing collections in queryable trigger function that prevented adding scoped queryable entries.\n\n## [v0.7.5]\n\n### Fixed\n\n- Default sort not getting set when sortby not included in query with token (Fixes [#177](https://github.com/stac-utils/pgstac/issues/177))\n- Fixes regression in performance between with changes for partition structure at v0.7.0. Changes the normal view for partitions and partition_steps into indexed materialized views. Adds refreshing of the views to existing triggers to make sure they stay up to date.\n\n## [v0.7.4]\n\n### Added\n\n- Add --v and --vv options to scripts/test to change logging to notice / log when running tests.\n- Add framework for option to cache expensive item formatting/hydrating calls. Note: this only provides functionality to add and read from the cached calls, but does not have any wiring to remove any entries from the cache.\n- Update the costs for json formatting functions to 5000 to help the query planner choose to prefer using indexes on json fields.\n\n### Fixed\n\n- Fix bug in foreign key and unique collection detection in queryables trigger function, update tests to catch.\n- Add collection id to tokens to ensure uniqueness and improve speed when looking up token values. Update tests to use the new keys. Old item id only tokens are still valid, but new results will all contain the new keys.\n- Improve performance when looking for whether next/prev links should be added.\n- Update Search function to remove the use of cursors and temp tables.\n- Update get_token_filter to remove the use of temp tables.\n\n## [v0.7.3]\n\n### Fixed\n\n- Use IF EXISTS when dropping constraints to avoid race conditions\n- Rework function that finds indexes that need to be added to be added and to find functionally identical indexes better.\n\n## [v0.7.2]\n\n### Fixed\n\n- Use version_parser for parsing versions in pypgstac\n- Fix issue with dropping functions/procedures in 0.6.13->0.7.0 migrations\n- Fix issue with CREATE OR REPLACE TRIGGER on PG 13\n- Fix issue identifying duplicate indexes in maintain_partition_queries function\n- Ensure that pgstac_read role has read permissions to all partitions\n- Fix issue (and add tests) caused by bug in psycopg datetime types not being able to translate 'infinity', '-infinity'\n\n## [v0.7.1]\n\n### Fixed\n\n- Fix permission issue when running incremental migrations.\n- Make sure that pypgstac migrate runs in a single transaction\n- Don't try to use concurrently when building indexes by default (this was tripping things up when using with pg_cron)\n- Don't short circuit search for requests with ids (Fixes #159)\n- Fix for issue with pagination when sorting by columns with nulls (Fixes #161 Fixes #152)\n- Fixes issue where duplicate datetime,end_datetime index was being built.\n- Fix bug in pypgstac loader when using delsert option\n\n### Added\n\n- Add trigger to detect duplicate configurations for name/collection combination in queryables\n- Add trigger to ensure collections added to queryables exist\n- Add tests for queryables triggers\n- Add more tests for different pagination scenarios\n\n## [v0.7.0]\n\n### Added\n\n- Reorganize code base to create clearer separation between pgstac sql code and pypgstac.\n- Move Python tooling to use hatch with all python project configuration in pyproject.toml\n- Rework testing framework to not rely on pypgstac or migrations. This allows to run tests on any code updates without creating a version first. If a new version has been staged, the tests will still run through all incremental migrations to make sure they pass as well.\n- Add pre-commit to run formatting as well as the tests appropriate for which files have changed.\n- Add a query queue to allow for deferred processing of steps that do not change the ability to get results, but enhance performance. The query queue allows to use pg_cron or similar to run tasks that are placed in the queue.\n- Modify triggers to allow the use of the query queue for building indexes, adding constraints that are used solely for constraint exclusion, and updating partition and collection spatial and temporal extents. The use of the queue is controlled by the new configuration parameter \"use_queue\" which can be set as the pgstac.use_queue GUC or by setting in the pgstac_settings table.\n- Reorganize how partitions are created and updated to maintain more metadata about partition extents and better tie the constraints to the actual temporal extent of a partition.\n- Add \"partitions\" view that shows stats about number of records, the partition range, constraint ranges, actual date range and spatial extent of each partition.\n- Add ability to automatically update the extent object on a collection using the partition metadata via triggers. This is controlled by the new configuration parameter \"update_collection_extent\" which can be set as the pgstac.update_collection_extent GUC or by setting in the pgstac_settings table. This can be combined with \"use_queue\" to defer the processing.\n- Add many new tests.\n- Migrations now make sure that all objects in the pgstac schema are owned by the pgstac_admin role. Functions marked as \"SECURITY DEFINER\" have been moved to the lower level functions responsible for creating/altering partitions and adding records to the search/search_wheres tables. This should open the door for approaches to using Row Level Security.\n- Allow pypgstac loader to load data on pgstac databases that have the same major version even if minor version differs. [162] () Cherry picked from .\n\n### Fixed\n\n- Allow empty strings in datetime intervals\n- Set search_path and application_name upon connection rather than as kwargs for compatibility with RDS [156] ()\n\n## [v0.6.13]\n\n### Fixed\n\n- Fix issue with sorting and paging where in some circumstances the aggregation of data changed the expected order\n\n## [v0.6.12]\n\n### Added\n\n- Add ability to merge enum, min, and max from queryables where collections have different values.\n- Add tooling in pypgstac and pgstac to add stac_extension definitions to the database.\n- Modify missing_queryables function to try to use stac_extension definitions to populate queryable definitions from the stac_extension schemas.\n- Add validate_constraints procedure\n- Add analyze_items procedure\n- Add check_pgstac_settings function to check system and pgstac settings.\n\n### Fixed\n\n- Fix issue with upserts in the trigger for using the items_staging tables\n- Fix for generating token query for sorting. [152] ()\n\n## [v0.6.11]\n\n### Fixed\n\n- update pypgstac requirements to support python 3.11 [142](https://github.com/stac-utils/pgstac/pull/142)\n- rename pgstac setting `default-filter-lang` to `default_filter_lang` to allow pgstac on postgresql>=14\n\n## [v0.6.10]\n\n### Fixed\n\n- Makes sure that passing in a non-existing collection does not return a queryable object.\n\n## [v0.6.9]\n\n### Fixed\n\n- Set cursor_tuple_fraction to 1 in search function to let query planner know to expect the entire table result within the search function to be returned. The default cursor_tuple_fraction of .1 within that function was at times creating bad query plans leading to slow queries.\n\n## [v0.6.8]\n\n### Added\n\n- Add get_queryables function to return a composite queryables json for either a single collection (text), a list of collections(text[]), or for the full repository (null::text).\n- Add missing_queryables(collection text, tablesample int) function to help identify if there are any properties in a collection without entries in the queryables table. The tablesample parameter is an int <=100 that is the approximate percentage of the collection to scan to look for missing queryables rather than reading every item.\n- Add missing_queryables(tablesample int) function that scans all collections using a sample of records to identify missing queryables.\n\n## [v0.6.7]\n\n### Added\n\n- Add get_queryables function to return a composite queryables json for either a single collection (text), a list of collections(text[]), or for the full repository (null::text).\n- Add missing_queryables(collection text, tablesample int) function to help identify if there are any properties in a collection without entries in the queryables table. The tablesample parameter is an int <=100 that is the approximate percentage of the collection to scan to look for missing queryables rather than reading every item.\n- Add missing_queryables(tablesample int) function that scans all collections using a sample of records to identify missing queryables.\n\n## [v0.6.6]\n\n### Added\n\n- Add support for array operators in CQL2 (a_equals, a_contains, a_contained_by, a_overlaps).\n- Add check in loader to make sure that pypgstac and pgstac versions match before loading data [#123](https://github.com/stac-utils/pgstac/issues/123)\n\n## [v0.6.5]\n\n### Fixed\n\n- Fix for type casting when using the \"in\" operator [#122](https://github.com/stac-utils/pgstac/issues/122)\n- Fix failure of pypgstac load for large items [#121](https://github.com/stac-utils/pgstac/pull/121)\n\n## [v0.6.4]\n\n### Fixed\n\n- Fixed casts for numeric data when a property is not in the queryables table to use the type from the incoming json filter\n- Fixed issue loader grouping an unordered iterable by partition, speeding up loads of items with mixed partitions [#116](https://github.com/stac-utils/pgstac/pull/116)\n\n## [v0.6.3]\n\n### Fixed\n\n- Fixed content_hydrate argument ordering which caused incorrect behavior in database hydration [#115](https://github.com/stac-utils/pgstac/pull/115)\n\n### Added\n\n- Skip partition updates when unnecessary, which can drastically improve large ingest performance into existing partitions. [#114](https://github.com/stac-utils/pgstac/pull/114)\n\n## [v0.6.2]\n\n### Fixed\n\n- Ensure special keys are not in content when loaded [#112](https://github.com/stac-utils/pgstac/pull/112/files)\n\n## [v0.6.1]\n\n### Fixed\n\n- Fix issue where using equality operator against an array was only comparing the first element of the array\n\n## [v0.6.0]\n\n### Fixed\n\n- Fix function signatures for transactional functions (delete_item etc) to make sure that they are marked as volatile\n- Fix function for getting start/end dates from a stac item\n\n### Changed\n\n- Update hydration/dehydration logic to make sure that it matches hydration/dehydration in pypgstac\n- Update fields logic in pgstac to only use full paths and to match logic in stac-fastapi\n- Always include id and collection on features regardless of fields setting\n\n### Added\n\n- Add tests to ensure that pgstac and pypgstac hydration logic is equivalent\n- Add conf item to search to allow returning results without hydrating. This allows an application using pgstac to shift the CPU load of rehydrating items from the database onto the application server.\n- Add \"--dehydrated\" option to loader to be able to load a dehydrated file (or iterable) of items such as would be output using pg_dump or postgresql copy.\n- Add \"--chunksize\" option to loader that can split the processing of an iterable or file into chunks of n records at a time\n\n## [v0.5.1]\n\n### Fixed\n\n### Changed\n\n### Added\n\n- Add conf item to search to allow returning results without hydrating. This allows an application using pgstac to shift the CPU load of rehydrating items from the database onto the application server.\n\n## [v0.5.0]\n\nVersion 0.5.0 is a major refactor of how data is stored. It is recommended to start a new database from scratch and to move data over rather than to use the inbuilt migration which will be very slow for larger amounts of data.\n\n### Fixed\n\n### Changed\n\n- The partition layout has been changed from being hardcoded to a partition to week to using nested partitions. The first level is by collection, for each collection, there is an attribute partition_trunc which can be set to NULL (no temporal partitions), month, or year.\n\n- CQL1 and Query Code have been refactored to translate to CQL2 to reduce duplicated code in query parsing.\n\n- Unused functions have been stripped from the project.\n\n- Pypgstac has been changed to use Fire rather than Typo.\n\n- Pypgstac has been changed to use Psycopg3 rather than Asyncpg to enable easier use as both sync and async.\n\n- Indexing has been reworked to eliminate indexes that from logs were not being used. The global json index on properties has been removed. Indexes on individual properties can be added either globally or per collection using the new queryables table.\n\n- Triggers for maintaining partitions have been updated to reduce lock contention and to reflect the new data layout.\n\n- The data pager which optimizes \"order by datetime\" searches has been updated to get time periods from the new partition layout and partition metadata.\n\n- Tests have been updated to reflect the many changes.\n\n### Added\n\n- On ingest, the content in an item is compared to the metadata available at the collection level and duplicate information is stripped out (this is primarily data in the item_assets property). Logic is added in to merge this data back in on data usage.\n\n## [v0.4.5]\n\n### Fixed\n\n- Fixes support for using the intersects parameter at the base of a search (regression from changes in 0.4.4)\n- Fixes issue where results for a search on id returned [None] rather than [] (regression from changes in 0.4.4)\n\n### Changed\n\n- Changes requirement for PostgreSQL to 13+, the triggers used to main partitions are not available to be used on partitions prior to 13 ([#90](https://github.com/stac-utils/pgstac/pull/90))\n- Bump requirement for asyncpg to 0.25.0 ([#82](https://github.com/stac-utils/pgstac/pull/82))\n\n### Added\n\n- Added more tests.\n\n## [v0.4.4]\n\n### Added\n\n- Adds support for using ids, collections, datetime, bbox, and intersects parameters separated from the filter-lang (Fixes #85)\n - Previously use of these parameters was translated into cql-json and then to SQL, so was not available when using cql2-json\n - The deprecated query parameter is still only available when filter-lang is set to cql-json\n\n### Changed\n\n- Add PLPGSQL for item lookups by id so that the query plan for the simple query can be cached\n - Use item_by_id function when looking up records used for paging filters\n - Add a short circuit to search to use item_by_id lookup when using the ids parameter\n - This short circuit avoids using the query cache for this simple case\n - Ordering when using the ids parameter is hard coded to return results in the same order as the array passed in (this avoids the overhead of full parsing and additional overhead to sort)\n\n### Fixed\n\n- Fix to make sure that filtering on the search_wheres table leverages the functional index on the hash of the query rather than on the query itself.\n\n## [v0.4.3]\n\n### Fixed\n\n- Fix for optimization when using equals with json properties. Allow optimization for both \"eq\" and \"=\" (was only previously enabled for \"eq\")\n\n## [v0.4.2]\n\n### Changed\n\n- Add support for updated CQL2 spec to use timestamp or interval key\n\n### Fixed\n\n- Fix for 0.3.4 -> 0.3.5 migration making sure that partitions get renamed correctly\n\n## [v0.4.1]\n\n### Changed\n\n- Update `typer` to 0.4.0 to avoid clashes with `click` ([#76](https://github.com/stac-utils/pgstac/pull/76))\n\n### Fixed\n\n- Fix logic in getting settings to make sure that filter-lang set on query is respected. ([#77](https://github.com/stac-utils/pgstac/pull/77))\n- Fix for large queries in the query cache. ([#71](https://github.com/stac-utils/pgstac/pull/71))\n\n## [v0.4.0]\n\n### Fixed\n\n- Fixes syntax for IN, BETWEEN, ISNULL, and NOT in CQL 1 ([#69](https://github.com/stac-utils/pgstac/pull/69))\n\n### Added\n\n- Adds support for modifying settings through pgstac_settings table and by passing in 'conf' object in search json to support AWS RDS where custom user configuration settings are not allowed and changing settings on the fly for a given query.\n- Adds support for CQL2-JSON ([#67](https://github.com/stac-utils/pgstac/pull/67))\n - Adds tests for all examples in \n - filter-lang parameter controls which dialect of CQL to use\n - Adds 'default-filter-lang' setting to control what dialect to use when 'filter-lang' is not present\n - old style stac 'query' object and top level ids, collections, datetime, bbox, and intersects parameters are only available with cql-json\n\n## [v0.3.4]\n\n### Added\n\n- add `geometrysearch`, `geojsonsearch` and `xyzsearch` for optimized searches for tiled requets ([#39](https://github.com/stac-utils/pgstac/pull/39))\n- add `create_items` and `upsert_items` methods for bulk insert ([#39](https://github.com/stac-utils/pgstac/pull/39))\n\n## [v0.3.3]\n\n### Fixed\n\n- Fixed CQL term to be \"id\", not \"ids\" ([#46](https://github.com/stac-utils/pgstac/pull/46))\n- Make sure featureCollection response has empty features `[]` not `null` ([#46](https://github.com/stac-utils/pgstac/pull/46))\n- Fixed bugs for `sortby` and `pagination` ([#46](https://github.com/stac-utils/pgstac/pull/46))\n- Make sure pgtap errors get caught in CI ([#46](https://github.com/stac-utils/pgstac/pull/46))\n\n## [v0.3.2]\n\n## Fixed\n\n- Fixed CQL term to be \"collections\", not \"collection\" ([#43](https://github.com/stac-utils/pgstac/pull/43))\n\n## [v0.3.1]\n\n_TODO_\n\n## [v0.2.8]\n\n### Added\n\n- Type hints to pypgstac that pass mypy checks ([#18](https://github.com/stac-utils/pgstac/pull/18))\n\n### Fixed\n\n- Fixed issue with pypgstac loads which caused some writes to fail ([#18](https://github.com/stac-utils/pgstac/pull/18))\n\n[Unreleased]: https://github.com/stac-utils/pgstac/compare/v0.9.11...HEAD\n[v0.9.11]: https://github.com/stac-utils/pgstac/compare/v0.9.10...v0.9.11\n[v0.9.10]: https://github.com/stac-utils/pgstac/compare/v0.9.9...v0.9.10\n[v0.9.9]: https://github.com/stac-utils/pgstac/compare/v0.9.8...v0.9.9\n[v0.9.8]: https://github.com/stac-utils/pgstac/compare/v0.9.7...v0.9.8\n[v0.9.7]: https://github.com/stac-utils/pgstac/compare/v0.9.6...v0.9.7\n[v0.9.6]: https://github.com/stac-utils/pgstac/compare/v0.9.5...v0.9.6\n[v0.9.5]: https://github.com/stac-utils/pgstac/compare/v0.9.4...v0.9.5\n[v0.9.4]: https://github.com/stac-utils/pgstac/compare/v0.9.3...v0.9.4\n[v0.9.3]: https://github.com/stac-utils/pgstac/compare/v0.9.2...v0.9.3\n[v0.9.2]: https://github.com/stac-utils/pgstac/compare/v0.9.1...v0.9.2\n[v0.9.1]: https://github.com/stac-utils/pgstac/compare/v0.9.0...v0.9.1\n[v0.9.0]: https://github.com/stac-utils/pgstac/compare/v0.8.5...v0.9.0\n[v0.8.5]: https://github.com/stac-utils/pgstac/compare/v0.8.4...v0.8.5\n[v0.8.4]: https://github.com/stac-utils/pgstac/compare/v0.8.3...v0.8.4\n[v0.8.3]: https://github.com/stac-utils/pgstac/compare/v0.8.2...v0.8.3\n[v0.8.2]: https://github.com/stac-utils/pgstac/compare/v0.8.1...v0.8.2\n[v0.8.1]: https://github.com/stac-utils/pgstac/compare/v0.8.0...v0.8.1\n[v0.8.0]: https://github.com/stac-utils/pgstac/compare/v0.7.10...v0.8.0\n[v0.7.10]: https://github.com/stac-utils/pgstac/compare/v0.7.9...v0.7.10\n[v0.7.9]: https://github.com/stac-utils/pgstac/compare/v0.7.8...v0.7.9\n[v0.7.8]: https://github.com/stac-utils/pgstac/compare/v0.7.7...v0.7.8\n[v0.7.7]: https://github.com/stac-utils/pgstac/compare/v0.7.6...v0.7.7\n[v0.7.6]: https://github.com/stac-utils/pgstac/compare/v0.7.5...v0.7.6\n[v0.7.5]: https://github.com/stac-utils/pgstac/compare/v0.7.4...v0.7.5\n[v0.7.4]: https://github.com/stac-utils/pgstac/compare/v0.7.3...v0.7.4\n[v0.7.3]: https://github.com/stac-utils/pgstac/compare/v0.7.2...v0.7.3\n[v0.7.2]: https://github.com/stac-utils/pgstac/compare/v0.7.1...v0.7.2\n[v0.7.1]: https://github.com/stac-utils/pgstac/compare/v0.7.0...v0.7.1\n[v0.7.0]: https://github.com/stac-utils/pgstac/compare/v0.6.13...v0.7.0\n[v0.6.13]: https://github.com/stac-utils/pgstac/compare/v0.6.12...v0.6.13\n[v0.6.12]: https://github.com/stac-utils/pgstac/compare/v0.6.11...v0.6.12\n[v0.6.11]: https://github.com/stac-utils/pgstac/compare/v0.6.10...v0.6.11\n[v0.6.10]: https://github.com/stac-utils/pgstac/compare/v0.6.9...v0.6.10\n[v0.6.9]: https://github.com/stac-utils/pgstac/compare/v0.6.8...v0.6.9\n[v0.6.8]: https://github.com/stac-utils/pgstac/compare/v0.6.7...v0.6.8\n[v0.6.7]: https://github.com/stac-utils/pgstac/compare/v0.6.6...v0.6.7\n[v0.6.6]: https://github.com/stac-utils/pgstac/compare/v0.6.5...v0.6.6\n[v0.6.5]: https://github.com/stac-utils/pgstac/compare/v0.6.4...v0.6.5\n[v0.6.4]: https://github.com/stac-utils/pgstac/compare/v0.6.3...v0.6.4\n[v0.6.3]: https://github.com/stac-utils/pgstac/compare/v0.6.2...v0.6.3\n[v0.6.2]: https://github.com/stac-utils/pgstac/compare/v0.6.1...v0.6.2\n[v0.6.1]: https://github.com/stac-utils/pgstac/compare/v0.6.0...v0.6.1\n[v0.6.0]: https://github.com/stac-utils/pgstac/compare/v0.5.1...v0.6.0\n[v0.5.1]: https://github.com/stac-utils/pgstac/compare/v0.5.0...v0.5.1\n[v0.5.0]: https://github.com/stac-utils/pgstac/compare/v0.4.5...v0.5.0\n[v0.4.5]: https://github.com/stac-utils/pgstac/compare/v0.4.4...v0.4.5\n[v0.4.4]: https://github.com/stac-utils/pgstac/compare/v0.4.3...v0.4.4\n[v0.4.3]: https://github.com/stac-utils/pgstac/compare/v0.4.2...v0.4.3\n[v0.4.2]: https://github.com/stac-utils/pgstac/compare/v0.4.1...v0.4.2\n[v0.4.1]: https://github.com/stac-utils/pgstac/compare/v0.4.0...v0.4.1\n[v0.4.0]: https://github.com/stac-utils/pgstac/compare/v0.3.4...v0.4.0\n[v0.3.4]: https://github.com/stac-utils/pgstac/compare/v0.3.3...v0.3.4\n[v0.3.3]: https://github.com/stac-utils/pgstac/compare/v0.3.2...v0.3.3\n[v0.3.2]: https://github.com/stac-utils/pgstac/compare/v0.3.1...v0.3.2\n[v0.3.1]: https://github.com/stac-utils/pgstac/compare/v0.3.0...v0.3.1\n[v0.2.8]: https://github.com/stac-utils/pgstac/compare/ff02c9cee7bbb0a2de21530b0aeb34e823f2e95c...v0.2.8\n" }, { "path": "CLAUDE.md", "content": "# PgSTAC Development Instructions\n\n## Project Overview\n\nPgSTAC is a PostgreSQL extension (SQL functions + schema) for Spatio-Temporal Asset Catalogs (STAC), paired with pypgstac, a Python package for database migrations and bulk data ingestion.\n\n- **Repository**: stac-utils/pgstac\n- **License**: MIT\n- **Docs**: https://stac-utils.github.io/pgstac/\n\n## Architecture\n\n```\nsrc/pgstac/sql/ ← ALL SQL source files (edit ONLY here)\nsrc/pgstac/pgstac.sql ← Assembled output (DO NOT edit directly)\nsrc/pgstac/migrations/ ← Base + incremental migration files\nsrc/pgstac/tests/ ← PGTap and basic SQL tests\nsrc/pypgstac/src/pypgstac/ ← Python package source\nsrc/pypgstac/tests/ ← pytest tests\nscripts/ ← Host-facing entrypoint scripts\nscripts/container-scripts/ ← Scripts copied into the pypgstac container image\n```\n\n### Documentation Files\n\n- **`CHANGELOG.md`** — the single source of truth for release notes\n- **`docs/src/release-notes.md`** — a **manual copy** of `CHANGELOG.md`, served by mkdocs. Keep them identical; update both when changing either.\n\n### Database Roles\n\n- **pgstac_admin** – schema owner, migrations\n- **pgstac_ingest** – read/write, execute functions\n- **pgstac_read** – SELECT only\n\n## Critical Rules\n\n### SQL Changes\n\n**ONLY edit files in `src/pgstac/sql/`**. Never edit `pgstac.sql` directly — it is assembled by concatenating all `sql/*.sql` files in alphabetical order during `stageversion`.\n\nFile execution order (alphabetical by prefix):\n`000_idempotent_pre` → `001_core` → `001a_jsonutils` → `001s_stacutils` → `002_collections` → `002a_queryables` → `002b_cql` → `003a_items` → `003b_partitions` → `004_search` → `004a_collectionsearch` → `005_tileutils` → `006_tilesearch` → `997_maintenance` → `998_idempotent_post` → `999_version`\n\nPrefix ranges: `000-001` setup/core, `002` collections, `003` items/partitions, `004-006` search/tiles, `997-998` maintenance/post, `999` version (auto-generated).\n\n### Idempotency\n\n`000_idempotent_pre.sql` and `998_idempotent_post.sql` are included in both base installs and incremental migrations. Use `IF NOT EXISTS`, `CREATE OR REPLACE`, `ON CONFLICT DO NOTHING`.\n\n### Partitioning\n\nItems partitioned by `LIST(collection)`, optionally sub-partitioned by `RANGE(datetime)` (year/month via `collections.partition_trunc`). Naming: `_items_{key}[_{YYYY|YYYYMM}]`.\n\nKey functions: `check_partition()` (create/update), `update_partition_stats()` (recalculate constraints), `partition_sys_meta` (live VIEW — always current), `partitions` (MATERIALIZED VIEW — stale between refreshes).\n\n### Search Path\n\nPgSTAC installs into the `pgstac` schema. All connections must have `search_path` set to `pgstac, public`.\n\n### pg_dump / pg_restore Compatibility\n\nPgSTAC functions reference PostGIS functions (e.g. `st_makeenvelope`, `st_geomfromgeojson`) **without schema qualification** because PostGIS may be installed in either `public` or `postgis` schema. `pg_dump` clears `search_path` during restore, breaking these references.\n\n**Rules to maintain dump/restore compatibility:**\n\n- **Do NOT schema-qualify PostGIS function calls** in PgSTAC SQL\n- **Avoid cross-function dependencies in SQL functions used by GENERATED columns** — pg_dump orders functions alphabetically, so `func_a` calling `func_b` may be created before `func_b` exists. Inline the logic instead.\n- Use `pgstac_restore` (via `scripts/container-scripts/pgstac_restore` in the image) to restore dumps — it installs a temporary event trigger that sets the correct `search_path` before each DDL command\n- Test with `scripts/test --pgdump`\n\n## Development Workflow\n\n### Setup\n\n```bash\nscripts/setup # Build Docker images, start database\nscripts/server # Start database (use --detach for background)\n```\n\n### Running Tests\n\n```bash\nscripts/test # All test suites\nscripts/test --pypgstac # pytest only\nscripts/test --pgtap # PGTap SQL tests\nscripts/test --basicsql # SQL output comparison tests\nscripts/test --migrations # Full migration chain test\nscripts/test --formatting # ruff + ty\nscripts/test --pgdump # pg_dump/pg_restore round-trip test\n```\n\nAll tests run inside Docker via `scripts/runinpypgstac`. Use `--build` to rebuild images first.\n\n### Docker Architecture\n\n- **pgstac** container: PostgreSQL 17 + PostGIS 3 + extensions, port 5439→5432\n- **pypgstac** container: Python + Rust build tools, runs scripts\n- Credentials: `username` / `password`, database: `postgis`\n\n## Migration Process\n\n### Creating Migrations (Release)\n\n```bash\nscripts/stageversion 0.9.11\n```\n\nThis runs inside Docker and:\n1. Removes old `*unreleased*` migration files\n2. Writes `SELECT set_version('0.9.11');` to `999_version.sql`\n3. Concatenates all `sql/*.sql` → `migrations/pgstac.0.9.11.sql` (base migration)\n4. Copies the base migration to `pgstac.sql`\n5. Updates `version.py` and `pyproject.toml` version strings\n6. Runs `makemigration -f 0.9.10 -t 0.9.11` to generate incremental migration\n\n### How makemigration Works\n\n`makemigration` (copied from `scripts/container-scripts/makemigration` into the image) generates incremental migrations by diffing schemas:\n\n1. Creates two temp databases: `migra_from`, `migra_to`\n2. Loads old base migration into `migra_from`\n3. Loads new base migration into `migra_to`\n4. Runs `migra --schema pgstac --unsafe` to calculate the SQL diff\n5. Wraps the diff with `000_idempotent_pre.sql`, `998_idempotent_post.sql`, and `set_version()`\n6. Output: `migrations/pgstac.0.9.10-0.9.11.sql`\n\n**Important**: The generated migration is created with a `.staged` suffix. You MUST:\n1. Review the `.staged` file for correctness\n2. Remove the `.staged` suffix to enable it\n3. Run `scripts/test --migrations` to validate\n\n### Running Migrations\n\n```bash\npypgstac migrate # Migrate to current pypgstac version\npypgstac migrate --toversion 0.9.10 # Migrate to specific version\n```\n\nThe `Migrate` class (in `migrate.py`) builds a directed graph of all available migration files and uses BFS to find the shortest path from the current DB version to the target.\n\n## Testing Details\n\n### Test Database Setup\n\nTests create `pgstac_test_db_template` from `pgstac.sql`, then clone it per test suite:\n- `pgstac_test_pgtap` – PGTap tests\n- `pgstac_test_basicsql` – basic SQL tests\n- `pgstac_test_pypgstac` – pytest (function-scoped fixture creates fresh DB per test)\n\n### Test Types\n\n1. **PGTap**: SQL assertions in `src/pgstac/tests/pgtap.sql`\n2. **Basic SQL**: `.sql` files in `src/pgstac/tests/basic/`, output compared to `.sql.out`\n3. **Pytest**: `src/pypgstac/tests/test_load.py`, `test_benchmark.py`, `test_queryables.py`, `hydration/`\n4. **Migration**: Installs v0.3.0, migrates to latest, runs all test suites against migrated DB\n5. **pg_dump**: Dumps a database with sample data, restores via `pgstac_restore`, verifies counts match\n\n### Pytest Fixtures (conftest.py)\n\n- `db` – function-scoped `PgstacDB` connected to fresh test DB\n- `loader` – `Loader(db)` instance\n\n## PR Checklist\n\n1. Changes only in `src/pgstac/sql/` for SQL, `src/pypgstac/` for Python\n2. Tests added if appropriate\n3. `CHANGELOG.md` updated under `## [UNRELEASED]`\n4. `docs/src/release-notes.md` updated to match `CHANGELOG.md` (they must stay identical)\n5. Docs updated if needed\n6. All tests pass: `scripts/test` (or `scripts/runinpypgstac --build test --pypgstac`)\n\n## Release Checklist\n\n1. `scripts/stageversion VERSION`\n2. Review `.staged` migration, remove suffix\n3. `scripts/test --migrations`\n4. Move CHANGELOG \"Unreleased\" → new version\n5. Copy updated `CHANGELOG.md` to `docs/src/release-notes.md` (keep identical)\n6. Create PR, merge\n7. `git tag vVERSION && git push origin vVERSION`\n8. CI publishes to PyPI + ghcr.io\n\n## Common Patterns\n\n### Adding a new SQL function\n\n1. Edit the appropriate file in `src/pgstac/sql/` (use `CREATE OR REPLACE FUNCTION`)\n2. Add `SECURITY DEFINER` if the function modifies tables\n3. Grant execute in `998_idempotent_post.sql` if needed\n4. Add PGTap or basic SQL tests\n\n### Adding a new queryable\n\n```sql\nINSERT INTO queryables (name, definition, property_wrapper, property_index_type)\nVALUES ('prop_name', '{\"$ref\": \"...\"}', 'to_int', 'BTREE')\nON CONFLICT DO NOTHING;\n```\n\n### Loading test data\n\n```bash\nscripts/runinpypgstac --build loadsampledata\n```\n" }, { "path": "CONTRIBUTING.md", "content": "# Development - Contributing\n\nPgSTAC uses a dockerized development environment.\n\nLocal-only planning notes now live under `.plans/` and are intentionally gitignored.\nIf you keep local execution settings, start from `.env.example` and write overrides to\n`.env`.\n\nTo build the docker images and set up the test database, use:\n\n```bash\nscripts/setup\n```\n\nTo bring up the development database:\n```\nscripts/server\n```\n\nTo run tests, use:\n```bash\nscripts/test\n```\n\nTo set up pre-commit using the project uv workflow:\n\n```bash\nuv tool install pre-commit==3.5.0\npre-commit install\n```\n\nUseful options:\n\n```bash\nscripts/test --fast\nscripts/test --pypgstac\nscripts/test --build-policy always\n```\n\n`scripts/test` defaults to `PGSTAC_BUILD_POLICY=always` so the container image reflects\nyour current checkout. If you intentionally want to reuse an existing image, set\n`PGSTAC_BUILD_POLICY=missing` or pass `--build-policy missing`.\n\nTo rebuild docker images:\n```bash\nscripts/update\n```\n\nContainer-only helper scripts now live in `scripts/container-scripts/` and are copied\ninto the `pypgstac` image during build. Top-level `scripts/` remain the host-facing\nentrypoint surface.\n\nTo drop into a console, use\n```bash\nscripts/console\n```\n\nTo drop into a psql console on the database container, use:\n```bash\nscripts/console --db\n```\n\nTo run migrations on the development database, use\n```bash\nscripts/migrate\n```\n\nTo stage code and configurations and create template migrations for a version release, use\n```bash\nscripts/stageversion [version]\n```\n\nTo generate only the incremental migration, use:\n\n```bash\nscripts/makemigration --from 0.9.10 --to 0.9.11\n```\n\nExamples:\n```\nscripts/stageversion 0.2.8\n```\n\nThis will create a base migration for the new version and will create incremental migrations between any existing base migrations. The incremental migrations that are automatically generated by this script will have the extension \".staged\" on the file. You must manually review (and make any modifications necessary) this file and remove the \".staged\" extension to enable the migration.\n\n### Making Changes to SQL\nAll changes to SQL should only be made in the `/src/pgstac/sql` directory. SQL Files will be run in alphabetical order.\n\n### Adding Tests\n\nThere are three different types of tests within the project: (1) pgTap tests, (2) basic SQL tests, and (3) PyPgSTAC tests.\n\nPgSTAC tests can be written using PGTap or basic SQL output comparisons. Additional testing is available using PyTest in the PyPgSTAC module. Tests can be run using the `scripts/test` command.\n\nPGTap tests can be written using [PGTap](https://pgtap.org/) syntax. Tests should be added to the `/src/pgstac/tests/pgtap` directory. Any new SQL files added to this directory must be added to `/src/pgstac/tests/pgtap.sql`.\n\nThe Basic SQL tests will run any file ending in '.sql' in the `/src/pgstac/tests/basic` directory and will compare the exact results to the corresponding '.sql.out' file.\n\nPyPgSTAC tests are pytest tests, and they are located in `/src/pypgstac/tests`\n\nAll tests can be found in tests/pgtap.sql and are run using `scripts/test`.\n\nIndividual tests can be run with any combination of the following flags `--formatting --basicsql --pgtap --migrations --pypgstac`. The `--formatting` suite runs Ruff lint/format checks and Ty type checks. If pre-commit is installed, tests will be run on commit based on which files have changed.\n\n\n### To make a PR\n1) Make any changes.\n2) Make sure there are tests if appropriate.\n3) Update Changelog using \"### Unreleased\" as the version.\n4) Make any changes necessary to the docs.\n5) Ensure all tests pass (pre-commit will take care of this if installed and the tests will also run on CI)\n6) Create PR against the \"main\" branch.\n\n\n\n### Release Process\n1) Run \"scripts/stageversion VERSION\" (where version is the next version using semantic versioning ie 0.7.0\n2) Check the incremental migration created in the /src/pgstac/migrations file with the .staged extension to make sure that the generated SQL looks appropriate.\n3) Run the tests against the incremental migrations \"scripts/test --migrations\"\n4) Move any \"Unreleased\" changes in the CHANGELOG.md to the new version.\n5) Open a PR for the version change.\n6) Once the PR has been merged, start the release process.\n7) Create a git tag `git tag v0.2.8` using new version number\n8) Push the git tag `git push origin v0.2.8`\n9) The CI process will push pypgstac to PyPi, create a docker image on ghcr.io, and create a release on github.\n\n\n### Get Involved\n\nIssues and pull requests are more than welcome: https://github.com/stac-utils/pgstac/issues\n\n### A Note on Hydration and Dehydration\n\nDehydration refers to stripping redundant attributes of STAC items when storing them within the database. For many collections, dehydration saves a significant amount of memory.\n\nRehydration is the process of adding the stripped attributes back to the STAC items, such as during the export of an STAC collection or the response to a search query.\n\nPgSTAC, a versatile tool, is designed to seamlessly integrate with PyPgSTAC or alternative backends. This flexibility allows for direct calls for both rehydration and dehydration, giving developers and technical users a sense of control over the process.\n\nHydration and dehydration are de-facto settings that users can not opt out of. In the future, we may provide a configuration for use cases where the size benefits do not justify the added complexity.\n" }, { "path": "LICENSE", "content": " MIT License\n\n Copyright (c) Microsoft Corporation and other contributors.\n\n Permission is hereby granted, free of charge, to any person obtaining a copy\n of this software and associated documentation files (the \"Software\"), to deal\n in the Software without restriction, including without limitation the rights\n to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n copies of the Software, and to permit persons to whom the Software is\n furnished to do so, subject to the following conditions:\n\n The above copyright notice and this permission notice shall be included in all\n copies or substantial portions of the Software.\n\n THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\n IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\n FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\n AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\n LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\n OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\n SOFTWARE\n" }, { "path": "README.md", "content": "\n

\n \n

PostgreSQL schema and functions for Spatio-Temporal Asset Catalog (STAC)

\n

\n\n

\n \n \"Test\"\n \n \n \"Package\n \n \n \"License\"\n \n

\n\n---\n\n**Documentation**: https://stac-utils.github.io/pgstac/\n\n**Source Code**: https://github.com/stac-utils/pgstac\n\n---\n\n**PgSTAC** is a set of SQL functions and schema to build highly performant databases for Spatio-Temporal Asset Catalogs ([STAC](https://stacspec.org/)). The project also provides **pypgstac** (a Python module) to help with database migrations and document ingestion (collections and items).\n\nPgSTAC provides functionality for STAC Filters, CQL2 search, and utilities to help manage the indexing and partitioning of STAC Collections and Items.\n\nPgSTAC is used in production to scale to hundreds of millions of STAC items. PgSTAC implements core data models and functions to provide a STAC API from a PostgreSQL database. PgSTAC is entirely within the database and does not provide an HTTP-facing API. The [STAC FastAPI](https://github.com/stac-utils/stac-fastapi) PgSTAC backend and [Franklin](https://github.com/azavea/franklin) can be used to expose a PgSTAC catalog. Integrating PgSTAC with any other language with PostgreSQL drivers is also possible.\n\nPgSTAC Documentation: https://stac-utils.github.io/pgstac/pgstac\n\npyPgSTAC Documentation: https://stac-utils.github.io/pgstac/pypgstac\n\n## Project structure\n\n```\n/\n ├── src/pypgstac - pyPgSTAC python module\n ├── src/pypgstac/tests/ - pyPgSTAC tests\n ├── scripts/ - scripts to set up the environment, create migrations, and run tests\n ├── src/pgstac/sql/ - PgSTAC SQL code\n ├── src/pgstac/migrations/ - Migrations for incremental upgrades\n └── src/pgstac/tests/ - test suite\n```\n\n## Contribution & Development\n\nSee [CONTRIBUTING.md](https://github.com//stac-utils/pgstac/blob/master/CONTRIBUTING.md)\n\n## License\n\nSee [LICENSE](https://github.com//stac-utils/pgstac/blob/master/LICENSE)\n\n## Authors\n\nSee [contributors](https://github.com/stac-utils/pgstac/graphs/contributors) for a listing of individual contributors.\n\n## Changes\n\nSee [CHANGELOG.md](https://github.com/stac-utils/pgstac/blob/master/CHANGELOG.md).\n" }, { "path": "docker/pgstac/Dockerfile", "content": "# syntax=docker/dockerfile:1.7\nARG PG_MAJOR=17\nARG POSTGIS_MAJOR=3\nARG PG_TLE_VERSION=1.5.2\nARG DEBIAN_SUITE=trixie\n\n# Base postgres image that pgstac can be installed onto\nFROM postgres:${PG_MAJOR}-${DEBIAN_SUITE} AS pgstacbase\nARG POSTGIS_MAJOR\nARG PG_MAJOR\nARG PG_TLE_VERSION\nRUN --mount=type=cache,target=/var/cache/apt,sharing=locked \\\n --mount=type=cache,target=/var/lib/apt/lists,sharing=locked \\\n --mount=type=cache,target=/root/.cache/git,sharing=locked \\\n apt-get update \\\n && apt-get install -y --no-install-recommends \\\n postgresql-$PG_MAJOR-postgis-$POSTGIS_MAJOR \\\n postgresql-$PG_MAJOR-postgis-$POSTGIS_MAJOR-scripts \\\n postgresql-$PG_MAJOR-pgtap \\\n postgresql-$PG_MAJOR-plpgsql-check \\\n postgresql-$PG_MAJOR-partman \\\n postgresql-server-dev-$PG_MAJOR \\\n build-essential \\\n ca-certificates \\\n curl \\\n git \\\n flex \\\n bison \\\n libkrb5-dev \\\n && GIT_TERMINAL_PROMPT=0 git clone --branch v${PG_TLE_VERSION} --depth 1 https://github.com/aws/pg_tle.git /tmp/pg_tle \\\n && make -C /tmp/pg_tle \\\n && make -C /tmp/pg_tle install \\\n && rm -rf /tmp/pg_tle \\\n && sed -i \"s/^#shared_preload_libraries = .*/shared_preload_libraries = 'pg_tle'/\" /usr/share/postgresql/$PG_MAJOR/postgresql.conf.sample \\\n && sed -i \"s/^#shared_preload_libraries = .*/shared_preload_libraries = 'pg_tle'/\" /usr/share/postgresql/postgresql.conf.sample \\\n && apt-get purge -y --auto-remove \\\n postgresql-server-dev-$PG_MAJOR \\\n build-essential \\\n curl \\\n git \\\n flex \\\n bison \\\n libkrb5-dev \\\n && apt-get clean && apt-get -y autoremove \\\n && rm -rf /var/lib/apt/lists/*\n\n# The pgstacbase image with latest version of pgstac installed\nFROM pgstacbase AS pgstac\nWORKDIR /docker-entrypoint-initdb.d\nCOPY docker/pgstac/dbinit/pgstac.sh 990_pgstac.sh\nCOPY src/pgstac/pgstac.sql 999_pgstac.sql\n" }, { "path": "docker/pgstac/dbinit/pgstac.sh", "content": "SYSMEM=$(cat /proc/meminfo | grep -i 'memtotal' | grep -o '[[:digit:]]*')\nSHARED_BUFFERS=$(( $SYSMEM/4 ))\nEFFECTIVE_CACHE_SIZE=$(( $SYSMEM*3/4 ))\nMAINTENANCE_WORK_MEM=$(( $SYSMEM/8 ))\nWORK_MEM=$(( $SHARED_BUFFERS/50 ))\n\npsql -X -q -v ON_ERROR_STOP=1 </tmp/requirements.txt \\\n && uv pip install --system -r /tmp/requirements.txt\nCOPY scripts/container-scripts /opt/pgstac/container-scripts\nCOPY src/pypgstac /opt/src/pypgstac\nCOPY src/pgstac /opt/src/pgstac\nWORKDIR /opt/src/pypgstac\nRUN --mount=type=cache,target=/root/.cache/uv,sharing=locked \\\n uv pip install --system -e . \\\n && rm -rf /usr/local/cargo/registry\n\nRUN addgroup --gid 1000 user && \\\n adduser --uid 1000 --gid 1000 --disabled-password --gecos \"\" --home /home/user user && \\\n chown -R user:user /opt/src/pypgstac /opt/src/pgstac\nUSER user\n\n# Optional runtime-optimized image: no build toolchain, only pypgstac package + runtime deps.\nFROM python:3.13-slim-trixie AS pypgstac-runtime\nENV PYTHONWRITEBYTECODE=1\nENV PYTHONBUFFERED=1\nENV UV_BREAK_SYSTEM_PACKAGES=1\nENV PATH=\"/opt/pgstac/container-scripts:$PATH\"\nENV UV_CACHE_DIR=/root/.cache/uv\nARG PG_MAJOR=17\nRUN --mount=type=cache,target=/var/cache/apt,sharing=locked \\\n --mount=type=cache,target=/var/lib/apt/lists,sharing=locked \\\n --mount=type=cache,target=/root/.cache/uv,sharing=locked \\\n apt-get update \\\n && apt-get install -y --no-install-recommends \\\n ca-certificates \\\n curl \\\n postgresql-client-${PG_MAJOR} \\\n && curl -LsSf https://astral.sh/uv/install.sh | UV_INSTALL_DIR=/usr/local/bin sh \\\n && apt-get clean && rm -rf /var/lib/apt/lists/*\nCOPY ./src/pypgstac/pyproject.toml /tmp/pyproject.toml\nRUN --mount=type=cache,target=/root/.cache/uv,sharing=locked \\\n uv pip compile /tmp/pyproject.toml \\\n --extra psycopg \\\n --extra migrations \\\n >/tmp/requirements.txt \\\n && uv pip install --system -r /tmp/requirements.txt\nCOPY scripts/container-scripts /opt/pgstac/container-scripts\nCOPY src/pypgstac /opt/src/pypgstac\nCOPY src/pgstac /opt/src/pgstac\nWORKDIR /opt/src/pypgstac\nRUN --mount=type=cache,target=/root/.cache/uv,sharing=locked \\\n uv pip install --system .\n" }, { "path": "docker-compose.yml", "content": "services:\n pgstac:\n image: pgstac\n env_file:\n - .env\n build:\n context: .\n network: host\n dockerfile: docker/pgstac/Dockerfile\n target: pgstac\n platform: linux/amd64\n environment:\n - POSTGRES_USER=${POSTGRES_USER:-username}\n - POSTGRES_PASSWORD=${POSTGRES_PASSWORD:-password}\n - POSTGRES_DB=${POSTGRES_DB:-postgis}\n - PGHOST=/var/run/postgresql\n - PGPORT=5432\n - PGUSER=${PGUSER:-username}\n - PGPASSWORD=${PGPASSWORD:-password}\n - PGDATABASE=${PGDATABASE:-postgis}\n ports:\n - \"${PGPORT:-5439}:5432\"\n volumes:\n - pgstac-pgdata:/var/lib/postgresql/data\n command: postgres\n healthcheck:\n test: [\"CMD-SHELL\", \"pg_isready -h /var/run/postgresql -p 5432 -U ${PGUSER:-username} -d ${PGDATABASE:-postgis}\"]\n interval: 5s\n timeout: 5s\n retries: 12\n start_period: 5s\n pypgstac:\n image: pypgstac\n env_file:\n - .env\n build:\n context: .\n network: host\n dockerfile: docker/pypgstac/Dockerfile\n target: pypgstac\n platform: linux/amd64\n environment:\n - PGHOST=pgstac\n - PGPORT=5432\n - PGUSER=${PGUSER:-username}\n - PGPASSWORD=${PGPASSWORD:-password}\n - PGDATABASE=${PGDATABASE:-postgis}\n depends_on:\n pgstac:\n condition: service_healthy\nvolumes:\n pgstac-pgdata:\n" }, { "path": "docs/mkdocs.yml", "content": "site_name: pgstac\nsite_description: PostgreSQL schema and functions for Spatio-Temporal Asset Catalog (STAC).\n\ndocs_dir: 'src'\nsite_dir: 'build'\n\nrepo_name: \"stac-utils/pgstac\"\nrepo_url: \"https://github.com/stac-utils/pgstac\"\nedit_uri: \"blob/master/docs/src\"\nsite_url: \"https://stac-utils.github.io/pgstac/\"\n\nextra:\n social:\n - icon: \"fontawesome/brands/github\"\n link: \"https://github.com/stac-utils\"\n - icon: \"fontawesome/brands/twitter\"\n link: \"https://twitter.com/STACspec\"\n\nnav:\n - Home: \"index.md\"\n - PgSTAC: \"pgstac.md\"\n - pyPgSTAC: \"pypgstac.md\"\n - Performance:\n - item_size_analysis.ipynb\n - Development - Contributing: \"contributing.md\"\n - Release Notes: \"release-notes.md\"\n\nplugins:\n - search\n - mkdocs-jupyter:\n include_source: True\n include_requirejs: True\n execute: True\n show_input: False\n\ntheme:\n name: material\n palette:\n primary: indigo\n scheme: default\n\n# https://github.com/kylebarron/cogeo-mosaic/blob/mkdocs/mkdocs.yml#L50-L75\nmarkdown_extensions:\n - admonition\n - attr_list\n - codehilite:\n guess_lang: false\n - def_list\n - footnotes\n - pymdownx.arithmatex\n - pymdownx.betterem\n - pymdownx.caret:\n insert: false\n - pymdownx.details\n - pymdownx.emoji\n - pymdownx.escapeall:\n hardbreak: true\n nbsp: true\n - pymdownx.magiclink:\n hide_protocol: true\n repo_url_shortener: true\n - pymdownx.smartsymbols\n - pymdownx.superfences\n - pymdownx.tasklist:\n custom_checkbox: true\n - pymdownx.tilde\n - toc:\n permalink: true\n" }, { "path": "docs/src/benchmark.json", "content": "{\n \"machine_info\": {\n \"node\": \"quercus\",\n \"processor\": \"x86_64\",\n \"machine\": \"x86_64\",\n \"python_compiler\": \"GCC 13.2.0\",\n \"python_implementation\": \"CPython\",\n \"python_implementation_version\": \"3.12.3\",\n \"python_version\": \"3.12.3\",\n \"python_build\": [\n \"main\",\n \"Nov 6 2024 18:32:19\"\n ],\n \"release\": \"6.8.0-51-generic\",\n \"system\": \"Linux\",\n \"cpu\": {\n \"python_version\": \"3.12.3.final.0 (64 bit)\",\n \"cpuinfo_version\": [\n 9,\n 0,\n 0\n ],\n \"cpuinfo_version_string\": \"9.0.0\",\n \"arch\": \"X86_64\",\n \"bits\": 64,\n \"count\": 16,\n \"arch_string_raw\": \"x86_64\",\n \"vendor_id_raw\": \"AuthenticAMD\",\n \"brand_raw\": \"AMD Ryzen 7 7840U w/ Radeon 780M Graphics\",\n \"hz_advertised_friendly\": \"1.7866 GHz\",\n \"hz_actual_friendly\": \"1.7866 GHz\",\n \"hz_advertised\": [\n 1786629000,\n 0\n ],\n \"hz_actual\": [\n 1786629000,\n 0\n ],\n \"stepping\": 1,\n \"model\": 116,\n \"family\": 25,\n \"flags\": [\n \"3dnowext\",\n \"3dnowprefetch\",\n \"abm\",\n \"adx\",\n \"aes\",\n \"amd_lbr_v2\",\n \"aperfmperf\",\n \"apic\",\n \"arat\",\n \"avx\",\n \"avx2\",\n \"avx512_bf16\",\n \"avx512_bitalg\",\n \"avx512_vbmi2\",\n \"avx512_vnni\",\n \"avx512_vpopcntdq\",\n \"avx512bitalg\",\n \"avx512bw\",\n \"avx512cd\",\n \"avx512dq\",\n \"avx512f\",\n \"avx512ifma\",\n \"avx512vbmi\",\n \"avx512vbmi2\",\n \"avx512vl\",\n \"avx512vnni\",\n \"avx512vpopcntdq\",\n \"bmi1\",\n \"bmi2\",\n \"bpext\",\n \"cat_l3\",\n \"cdp_l3\",\n \"clflush\",\n \"clflushopt\",\n \"clwb\",\n \"clzero\",\n \"cmov\",\n \"cmp_legacy\",\n \"constant_tsc\",\n \"cpb\",\n \"cppc\",\n \"cpuid\",\n \"cqm\",\n \"cqm_llc\",\n \"cqm_mbm_local\",\n \"cqm_mbm_total\",\n \"cqm_occup_llc\",\n \"cr8_legacy\",\n \"cx16\",\n \"cx8\",\n \"dbx\",\n \"de\",\n \"decodeassists\",\n \"erms\",\n \"extapic\",\n \"extd_apicid\",\n \"f16c\",\n \"flush_l1d\",\n \"flushbyasid\",\n \"fma\",\n \"fpu\",\n \"fsgsbase\",\n \"fsrm\",\n \"fxsr\",\n \"fxsr_opt\",\n \"gfni\",\n \"ht\",\n \"hw_pstate\",\n \"ibpb\",\n \"ibrs\",\n \"ibrs_enhanced\",\n \"ibs\",\n \"invpcid\",\n \"irperf\",\n \"lahf_lm\",\n \"lbrv\",\n \"lm\",\n \"mba\",\n \"mca\",\n \"mce\",\n \"misalignsse\",\n \"mmx\",\n \"mmxext\",\n \"monitor\",\n \"movbe\",\n \"msr\",\n \"mtrr\",\n \"mwaitx\",\n \"nonstop_tsc\",\n \"nopl\",\n \"npt\",\n \"nrip_save\",\n \"nx\",\n \"ospke\",\n \"osvw\",\n \"osxsave\",\n \"overflow_recov\",\n \"pae\",\n \"pat\",\n \"pausefilter\",\n \"pci_l2i\",\n \"pclmulqdq\",\n \"pdpe1gb\",\n \"perfctr_core\",\n \"perfctr_llc\",\n \"perfctr_nb\",\n \"perfmon_v2\",\n \"pfthreshold\",\n \"pge\",\n \"pku\",\n \"pni\",\n \"popcnt\",\n \"pqe\",\n \"pqm\",\n \"pse\",\n \"pse36\",\n \"rapl\",\n \"rdpid\",\n \"rdpru\",\n \"rdrand\",\n \"rdrnd\",\n \"rdseed\",\n \"rdt_a\",\n \"rdtscp\",\n \"rep_good\",\n \"sep\",\n \"sha\",\n \"sha_ni\",\n \"skinit\",\n \"smap\",\n \"smca\",\n \"smep\",\n \"ssbd\",\n \"sse\",\n \"sse2\",\n \"sse4_1\",\n \"sse4_2\",\n \"sse4a\",\n \"ssse3\",\n \"stibp\",\n \"succor\",\n \"svm\",\n \"svm_lock\",\n \"syscall\",\n \"tce\",\n \"topoext\",\n \"tsc\",\n \"tsc_scale\",\n \"umip\",\n \"user_shstk\",\n \"v_spec_ctrl\",\n \"v_vmsave_vmload\",\n \"vaes\",\n \"vgif\",\n \"vmcb_clean\",\n \"vme\",\n \"vmmcall\",\n \"vnmi\",\n \"vpclmulqdq\",\n \"wbnoinvd\",\n \"wdt\",\n \"x2apic\",\n \"x2avic\",\n \"xgetbv1\",\n \"xsave\",\n \"xsavec\",\n \"xsaveerptr\",\n \"xsaveopt\",\n \"xsaves\"\n ],\n \"l3_cache_size\": 1048576,\n \"l2_cache_size\": 8388608,\n \"l1_data_cache_size\": 262144,\n \"l1_instruction_cache_size\": 262144,\n \"l2_cache_line_size\": 1024,\n \"l2_cache_associativity\": 6\n }\n },\n \"commit_info\": {\n \"id\": \"6da165b7c4f6da321d0b58bbb78da2887763a0d8\",\n \"time\": \"2024-12-09T10:50:39-06:00\",\n \"author_time\": \"2024-12-09T10:50:39-06:00\",\n \"dirty\": true,\n \"project\": \"pgstac\",\n \"branch\": \"feat/search-benchmarks\"\n },\n \"benchmarks\": [\n {\n \"group\": \"xyzsearch\",\n \"name\": \"test1[3-0.5]\",\n \"fullname\": \"src/pypgstac/tests/test_benchmark.py::test1[3-0.5]\",\n \"params\": {\n \"zoom\": 3,\n \"item_width\": 0.5\n },\n \"param\": \"3-0.5\",\n \"extra_info\": {},\n \"options\": {\n \"disable_gc\": false,\n \"timer\": \"perf_counter\",\n \"min_rounds\": 3,\n \"max_time\": 1.0,\n \"min_time\": 5e-06,\n \"warmup\": 2\n },\n \"stats\": {\n \"min\": 4.781100236999919,\n \"max\": 4.923354106998886,\n \"mean\": 4.84395678366612,\n \"stddev\": 0.07255507461212232,\n \"rounds\": 3,\n \"median\": 4.827416006999556,\n \"iqr\": 0.10669040249922546,\n \"q1\": 4.792679179499828,\n \"q3\": 4.8993695819990535,\n \"iqr_outliers\": 0,\n \"stddev_outliers\": 1,\n \"outliers\": \"1;0\",\n \"ld15iqr\": 4.781100236999919,\n \"hd15iqr\": 4.923354106998886,\n \"ops\": 0.20644279969053644,\n \"total\": 14.53187035099836,\n \"data\": [\n 4.781100236999919,\n 4.827416006999556,\n 4.923354106998886\n ],\n \"iterations\": 1\n }\n },\n {\n \"group\": \"xyzsearch\",\n \"name\": \"test1[3-0.75]\",\n \"fullname\": \"src/pypgstac/tests/test_benchmark.py::test1[3-0.75]\",\n \"params\": {\n \"zoom\": 3,\n \"item_width\": 0.75\n },\n \"param\": \"3-0.75\",\n \"extra_info\": {},\n \"options\": {\n \"disable_gc\": false,\n \"timer\": \"perf_counter\",\n \"min_rounds\": 3,\n \"max_time\": 1.0,\n \"min_time\": 5e-06,\n \"warmup\": 2\n },\n \"stats\": {\n \"min\": 2.147751219003112,\n \"max\": 2.1683643510004913,\n \"mean\": 2.158085888334123,\n \"stddev\": 0.010306680943796004,\n \"rounds\": 3,\n \"median\": 2.158142094998766,\n \"iqr\": 0.015459848998034431,\n \"q1\": 2.1503489380020255,\n \"q3\": 2.16580878700006,\n \"iqr_outliers\": 0,\n \"stddev_outliers\": 1,\n \"outliers\": \"1;0\",\n \"ld15iqr\": 2.147751219003112,\n \"hd15iqr\": 2.1683643510004913,\n \"ops\": 0.4633735874024566,\n \"total\": 6.474257665002369,\n \"data\": [\n 2.1683643510004913,\n 2.158142094998766,\n 2.147751219003112\n ],\n \"iterations\": 1\n }\n },\n {\n \"group\": \"xyzsearch\",\n \"name\": \"test1[3-1]\",\n \"fullname\": \"src/pypgstac/tests/test_benchmark.py::test1[3-1]\",\n \"params\": {\n \"zoom\": 3,\n \"item_width\": 1\n },\n \"param\": \"3-1\",\n \"extra_info\": {},\n \"options\": {\n \"disable_gc\": false,\n \"timer\": \"perf_counter\",\n \"min_rounds\": 3,\n \"max_time\": 1.0,\n \"min_time\": 5e-06,\n \"warmup\": 2\n },\n \"stats\": {\n \"min\": 1.1977020270023786,\n \"max\": 1.2047663939993072,\n \"mean\": 1.2008456296668253,\n \"stddev\": 0.003595734342317445,\n \"rounds\": 3,\n \"median\": 1.2000684679987899,\n \"iqr\": 0.005298275247696438,\n \"q1\": 1.1982936372514814,\n \"q3\": 1.2035919124991779,\n \"iqr_outliers\": 0,\n \"stddev_outliers\": 1,\n \"outliers\": \"1;0\",\n \"ld15iqr\": 1.1977020270023786,\n \"hd15iqr\": 1.2047663939993072,\n \"ops\": 0.8327465040426971,\n \"total\": 3.6025368890004756,\n \"data\": [\n 1.2047663939993072,\n 1.1977020270023786,\n 1.2000684679987899\n ],\n \"iterations\": 1\n }\n },\n {\n \"group\": \"xyzsearch\",\n \"name\": \"test1[3-1.5]\",\n \"fullname\": \"src/pypgstac/tests/test_benchmark.py::test1[3-1.5]\",\n \"params\": {\n \"zoom\": 3,\n \"item_width\": 1.5\n },\n \"param\": \"3-1.5\",\n \"extra_info\": {},\n \"options\": {\n \"disable_gc\": false,\n \"timer\": \"perf_counter\",\n \"min_rounds\": 3,\n \"max_time\": 1.0,\n \"min_time\": 5e-06,\n \"warmup\": 2\n },\n \"stats\": {\n \"min\": 0.5610035069985315,\n \"max\": 0.5700876880000578,\n \"mean\": 0.5648575563330572,\n \"stddev\": 0.004695826663797881,\n \"rounds\": 3,\n \"median\": 0.5634814740005822,\n \"iqr\": 0.006813135751144728,\n \"q1\": 0.5616229987490442,\n \"q3\": 0.5684361345001889,\n \"iqr_outliers\": 0,\n \"stddev_outliers\": 1,\n \"outliers\": \"1;0\",\n \"ld15iqr\": 0.5610035069985315,\n \"hd15iqr\": 0.5700876880000578,\n \"ops\": 1.7703578340914847,\n \"total\": 1.6945726689991716,\n \"data\": [\n 0.5610035069985315,\n 0.5634814740005822,\n 0.5700876880000578\n ],\n \"iterations\": 1\n }\n },\n {\n \"group\": \"xyzsearch\",\n \"name\": \"test1[3-2]\",\n \"fullname\": \"src/pypgstac/tests/test_benchmark.py::test1[3-2]\",\n \"params\": {\n \"zoom\": 3,\n \"item_width\": 2\n },\n \"param\": \"3-2\",\n \"extra_info\": {},\n \"options\": {\n \"disable_gc\": false,\n \"timer\": \"perf_counter\",\n \"min_rounds\": 3,\n \"max_time\": 1.0,\n \"min_time\": 5e-06,\n \"warmup\": 2\n },\n \"stats\": {\n \"min\": 0.3394572959987272,\n \"max\": 0.3474930239972309,\n \"mean\": 0.3423380219983301,\n \"stddev\": 0.0035336363516854353,\n \"rounds\": 4,\n \"median\": 0.3412008839986811,\n \"iqr\": 0.004039818997625844,\n \"q1\": 0.34031811249951716,\n \"q3\": 0.344357931497143,\n \"iqr_outliers\": 0,\n \"stddev_outliers\": 1,\n \"outliers\": \"1;0\",\n \"ld15iqr\": 0.3394572959987272,\n \"hd15iqr\": 0.3474930239972309,\n \"ops\": 2.921089495588889,\n \"total\": 1.3693520879933203,\n \"data\": [\n 0.3394572959987272,\n 0.3474930239972309,\n 0.3412228389970551,\n 0.3411789290003071\n ],\n \"iterations\": 1\n }\n },\n {\n \"group\": \"xyzsearch\",\n \"name\": \"test1[3-3]\",\n \"fullname\": \"src/pypgstac/tests/test_benchmark.py::test1[3-3]\",\n \"params\": {\n \"zoom\": 3,\n \"item_width\": 3\n },\n \"param\": \"3-3\",\n \"extra_info\": {},\n \"options\": {\n \"disable_gc\": false,\n \"timer\": \"perf_counter\",\n \"min_rounds\": 3,\n \"max_time\": 1.0,\n \"min_time\": 5e-06,\n \"warmup\": 2\n },\n \"stats\": {\n \"min\": 0.1616739819983195,\n \"max\": 0.1732538390024274,\n \"mean\": 0.1683614392864651,\n \"stddev\": 0.004645194475107718,\n \"rounds\": 7,\n \"median\": 0.16951855400111526,\n \"iqr\": 0.007849024001188809,\n \"q1\": 0.16395422725054232,\n \"q3\": 0.17180325125173113,\n \"iqr_outliers\": 0,\n \"stddev_outliers\": 3,\n \"outliers\": \"3;0\",\n \"ld15iqr\": 0.1616739819983195,\n \"hd15iqr\": 0.1732538390024274,\n \"ops\": 5.939602347414667,\n \"total\": 1.1785300750052556,\n \"data\": [\n 0.16951855400111526,\n 0.17012289500053157,\n 0.1732538390024274,\n 0.1616739819983195,\n 0.16210973700071918,\n 0.16948769800001173,\n 0.17236337000213098\n ],\n \"iterations\": 1\n }\n },\n {\n \"group\": \"xyzsearch\",\n \"name\": \"test1[3-4]\",\n \"fullname\": \"src/pypgstac/tests/test_benchmark.py::test1[3-4]\",\n \"params\": {\n \"zoom\": 3,\n \"item_width\": 4\n },\n \"param\": \"3-4\",\n \"extra_info\": {},\n \"options\": {\n \"disable_gc\": false,\n \"timer\": \"perf_counter\",\n \"min_rounds\": 3,\n \"max_time\": 1.0,\n \"min_time\": 5e-06,\n \"warmup\": 2\n },\n \"stats\": {\n \"min\": 0.11248960100056138,\n \"max\": 0.12265536899940344,\n \"mean\": 0.11735441244430452,\n \"stddev\": 0.0035863367674561345,\n \"rounds\": 9,\n \"median\": 0.11760097699880134,\n \"iqr\": 0.00569761775295774,\n \"q1\": 0.1145561129978887,\n \"q3\": 0.12025373075084644,\n \"iqr_outliers\": 0,\n \"stddev_outliers\": 3,\n \"outliers\": \"3;0\",\n \"ld15iqr\": 0.11248960100056138,\n \"hd15iqr\": 0.12265536899940344,\n \"ops\": 8.52119642688844,\n \"total\": 1.0561897119987407,\n \"data\": [\n 0.11948643000141601,\n 0.11310462300025392,\n 0.12265536899940344,\n 0.11510762000034447,\n 0.12055026199959684,\n 0.1150399429971003,\n 0.11760097699880134,\n 0.12015488700126298,\n 0.11248960100056138\n ],\n \"iterations\": 1\n }\n },\n {\n \"group\": \"xyzsearch\",\n \"name\": \"test1[3-5]\",\n \"fullname\": \"src/pypgstac/tests/test_benchmark.py::test1[3-5]\",\n \"params\": {\n \"zoom\": 3,\n \"item_width\": 5\n },\n \"param\": \"3-5\",\n \"extra_info\": {},\n \"options\": {\n \"disable_gc\": false,\n \"timer\": \"perf_counter\",\n \"min_rounds\": 3,\n \"max_time\": 1.0,\n \"min_time\": 5e-06,\n \"warmup\": 2\n },\n \"stats\": {\n \"min\": 0.08600778799882391,\n \"max\": 0.11819229899992933,\n \"mean\": 0.09380789238415534,\n \"stddev\": 0.008470383289679074,\n \"rounds\": 13,\n \"median\": 0.09278260499922908,\n \"iqr\": 0.009015565500703815,\n \"q1\": 0.08737023674893862,\n \"q3\": 0.09638580224964244,\n \"iqr_outliers\": 1,\n \"stddev_outliers\": 1,\n \"outliers\": \"1;1\",\n \"ld15iqr\": 0.08600778799882391,\n \"hd15iqr\": 0.11819229899992933,\n \"ops\": 10.66008386485086,\n \"total\": 1.2195026009940193,\n \"data\": [\n 0.0995918389999133,\n 0.09643121099725249,\n 0.09386423999967519,\n 0.09064780799963046,\n 0.08600778799882391,\n 0.09290014300131588,\n 0.09637066600043909,\n 0.08746462799899746,\n 0.11819229899992933,\n 0.09278260499922908,\n 0.08708706299876212,\n 0.09196810800131061,\n 0.08619420299874037\n ],\n \"iterations\": 1\n }\n },\n {\n \"group\": \"xyzsearch\",\n \"name\": \"test1[3-6]\",\n \"fullname\": \"src/pypgstac/tests/test_benchmark.py::test1[3-6]\",\n \"params\": {\n \"zoom\": 3,\n \"item_width\": 6\n },\n \"param\": \"3-6\",\n \"extra_info\": {},\n \"options\": {\n \"disable_gc\": false,\n \"timer\": \"perf_counter\",\n \"min_rounds\": 3,\n \"max_time\": 1.0,\n \"min_time\": 5e-06,\n \"warmup\": 2\n },\n \"stats\": {\n \"min\": 0.06886182499874849,\n \"max\": 0.08020066499739187,\n \"mean\": 0.0725205675997131,\n \"stddev\": 0.003679136903194388,\n \"rounds\": 15,\n \"median\": 0.07074371199996676,\n \"iqr\": 0.005597590749857773,\n \"q1\": 0.06958281574952707,\n \"q3\": 0.07518040649938484,\n \"iqr_outliers\": 0,\n \"stddev_outliers\": 2,\n \"outliers\": \"2;0\",\n \"ld15iqr\": 0.06886182499874849,\n \"hd15iqr\": 0.08020066499739187,\n \"ops\": 13.789191578306898,\n \"total\": 1.0878085139956966,\n \"data\": [\n 0.07045519399980549,\n 0.07074371199996676,\n 0.06936405300075421,\n 0.06957183299891767,\n 0.07550127799913753,\n 0.078494495999621,\n 0.08020066499739187,\n 0.07361344899982214,\n 0.07421779200012679,\n 0.07616023700029473,\n 0.06961576400135527,\n 0.06898055299825501,\n 0.07236440099950414,\n 0.06966326200199546,\n 0.06886182499874849\n ],\n \"iterations\": 1\n }\n },\n {\n \"group\": \"xyzsearch\",\n \"name\": \"test1[3-8]\",\n \"fullname\": \"src/pypgstac/tests/test_benchmark.py::test1[3-8]\",\n \"params\": {\n \"zoom\": 3,\n \"item_width\": 8\n },\n \"param\": \"3-8\",\n \"extra_info\": {},\n \"options\": {\n \"disable_gc\": false,\n \"timer\": \"perf_counter\",\n \"min_rounds\": 3,\n \"max_time\": 1.0,\n \"min_time\": 5e-06,\n \"warmup\": 2\n },\n \"stats\": {\n \"min\": 0.05374020599992946,\n \"max\": 0.05982637700071791,\n \"mean\": 0.0563469560555758,\n \"stddev\": 0.0016293739578158342,\n \"rounds\": 18,\n \"median\": 0.05618607450014679,\n \"iqr\": 0.0021514159998332616,\n \"q1\": 0.05518398200001684,\n \"q3\": 0.0573353979998501,\n \"iqr_outliers\": 0,\n \"stddev_outliers\": 4,\n \"outliers\": \"4;0\",\n \"ld15iqr\": 0.05374020599992946,\n \"hd15iqr\": 0.05982637700071791,\n \"ops\": 17.747187603420596,\n \"total\": 1.0142452090003644,\n \"data\": [\n 0.055682651000097394,\n 0.05374020599992946,\n 0.05650308599797427,\n 0.059472557997651165,\n 0.05982637700071791,\n 0.057392383001570124,\n 0.056449657000484876,\n 0.055928424997546244,\n 0.05518398200001684,\n 0.05479692000153591,\n 0.0573353979998501,\n 0.057961112001066795,\n 0.055098496999562485,\n 0.05566840800020145,\n 0.05707300100038992,\n 0.05644372400274733,\n 0.055237412998394575,\n 0.05445141100062756\n ],\n \"iterations\": 1\n }\n },\n {\n \"group\": \"xyzsearch\",\n \"name\": \"test1[3-10]\",\n \"fullname\": \"src/pypgstac/tests/test_benchmark.py::test1[3-10]\",\n \"params\": {\n \"zoom\": 3,\n \"item_width\": 10\n },\n \"param\": \"3-10\",\n \"extra_info\": {},\n \"options\": {\n \"disable_gc\": false,\n \"timer\": \"perf_counter\",\n \"min_rounds\": 3,\n \"max_time\": 1.0,\n \"min_time\": 5e-06,\n \"warmup\": 2\n },\n \"stats\": {\n \"min\": 0.047568563000822905,\n \"max\": 0.05683792700074264,\n \"mean\": 0.05166188815055648,\n \"stddev\": 0.0025601894529204182,\n \"rounds\": 20,\n \"median\": 0.05139943300127925,\n \"iqr\": 0.002679176501260372,\n \"q1\": 0.05012959849955223,\n \"q3\": 0.0528087750008126,\n \"iqr_outliers\": 1,\n \"stddev_outliers\": 6,\n \"outliers\": \"6;1\",\n \"ld15iqr\": 0.047568563000822905,\n \"hd15iqr\": 0.05683792700074264,\n \"ops\": 19.356628954128315,\n \"total\": 1.0332377630111296,\n \"data\": [\n 0.05180252200079849,\n 0.048698882001190213,\n 0.049534752000909066,\n 0.050387244002195075,\n 0.05197231200145325,\n 0.05162442900109454,\n 0.056070921000355156,\n 0.047568563000822905,\n 0.05047629700129619,\n 0.05343033399913111,\n 0.04820377700161771,\n 0.05592725600217818,\n 0.05117443700146396,\n 0.05683792700074264,\n 0.049871952996909386,\n 0.05052141399937682,\n 0.051092513000185136,\n 0.0516481759987073,\n 0.05420683799820836,\n 0.0521872160024941\n ],\n \"iterations\": 1\n }\n },\n {\n \"group\": \"xyzsearch\",\n \"name\": \"test1[4-0.5]\",\n \"fullname\": \"src/pypgstac/tests/test_benchmark.py::test1[4-0.5]\",\n \"params\": {\n \"zoom\": 4,\n \"item_width\": 0.5\n },\n \"param\": \"4-0.5\",\n \"extra_info\": {},\n \"options\": {\n \"disable_gc\": false,\n \"timer\": \"perf_counter\",\n \"min_rounds\": 3,\n \"max_time\": 1.0,\n \"min_time\": 5e-06,\n \"warmup\": 2\n },\n \"stats\": {\n \"min\": 1.1995517129980726,\n \"max\": 1.2396531550002692,\n \"mean\": 1.2173349196661245,\n \"stddev\": 0.020431746778374022,\n \"rounds\": 3,\n \"median\": 1.2127998910000315,\n \"iqr\": 0.030076081501647423,\n \"q1\": 1.2028637574985623,\n \"q3\": 1.2329398390002098,\n \"iqr_outliers\": 0,\n \"stddev_outliers\": 1,\n \"outliers\": \"1;0\",\n \"ld15iqr\": 1.1995517129980726,\n \"hd15iqr\": 1.2396531550002692,\n \"ops\": 0.8214666184670588,\n \"total\": 3.6520047589983733,\n \"data\": [\n 1.2396531550002692,\n 1.1995517129980726,\n 1.2127998910000315\n ],\n \"iterations\": 1\n }\n },\n {\n \"group\": \"xyzsearch\",\n \"name\": \"test1[4-0.75]\",\n \"fullname\": \"src/pypgstac/tests/test_benchmark.py::test1[4-0.75]\",\n \"params\": {\n \"zoom\": 4,\n \"item_width\": 0.75\n },\n \"param\": \"4-0.75\",\n \"extra_info\": {},\n \"options\": {\n \"disable_gc\": false,\n \"timer\": \"perf_counter\",\n \"min_rounds\": 3,\n \"max_time\": 1.0,\n \"min_time\": 5e-06,\n \"warmup\": 2\n },\n \"stats\": {\n \"min\": 0.5314996209999663,\n \"max\": 0.5415407899999991,\n \"mean\": 0.5360264716667492,\n \"stddev\": 0.0050928958379338005,\n \"rounds\": 3,\n \"median\": 0.5350390040002821,\n \"iqr\": 0.007530876750024618,\n \"q1\": 0.5323844667500452,\n \"q3\": 0.5399153435000699,\n \"iqr_outliers\": 0,\n \"stddev_outliers\": 1,\n \"outliers\": \"1;0\",\n \"ld15iqr\": 0.5314996209999663,\n \"hd15iqr\": 0.5415407899999991,\n \"ops\": 1.8655795055989435,\n \"total\": 1.6080794150002475,\n \"data\": [\n 0.5350390040002821,\n 0.5314996209999663,\n 0.5415407899999991\n ],\n \"iterations\": 1\n }\n },\n {\n \"group\": \"xyzsearch\",\n \"name\": \"test1[4-1]\",\n \"fullname\": \"src/pypgstac/tests/test_benchmark.py::test1[4-1]\",\n \"params\": {\n \"zoom\": 4,\n \"item_width\": 1\n },\n \"param\": \"4-1\",\n \"extra_info\": {},\n \"options\": {\n \"disable_gc\": false,\n \"timer\": \"perf_counter\",\n \"min_rounds\": 3,\n \"max_time\": 1.0,\n \"min_time\": 5e-06,\n \"warmup\": 2\n },\n \"stats\": {\n \"min\": 0.326569004002522,\n \"max\": 0.3346178480023809,\n \"mean\": 0.32932656225148094,\n \"stddev\": 0.003615016240426719,\n \"rounds\": 4,\n \"median\": 0.3280596985005104,\n \"iqr\": 0.004467878499781364,\n \"q1\": 0.32709262300159025,\n \"q3\": 0.3315605015013716,\n \"iqr_outliers\": 0,\n \"stddev_outliers\": 1,\n \"outliers\": \"1;0\",\n \"ld15iqr\": 0.326569004002522,\n \"hd15iqr\": 0.3346178480023809,\n \"ops\": 3.036499677291072,\n \"total\": 1.3173062490059237,\n \"data\": [\n 0.326569004002522,\n 0.3285031550003623,\n 0.32761624200065853,\n 0.3346178480023809\n ],\n \"iterations\": 1\n }\n },\n {\n \"group\": \"xyzsearch\",\n \"name\": \"test1[4-1.5]\",\n \"fullname\": \"src/pypgstac/tests/test_benchmark.py::test1[4-1.5]\",\n \"params\": {\n \"zoom\": 4,\n \"item_width\": 1.5\n },\n \"param\": \"4-1.5\",\n \"extra_info\": {},\n \"options\": {\n \"disable_gc\": false,\n \"timer\": \"perf_counter\",\n \"min_rounds\": 3,\n \"max_time\": 1.0,\n \"min_time\": 5e-06,\n \"warmup\": 2\n },\n \"stats\": {\n \"min\": 0.17086795599971083,\n \"max\": 0.18105750900213025,\n \"mean\": 0.17587526450127675,\n \"stddev\": 0.004035728323823642,\n \"rounds\": 6,\n \"median\": 0.17464539750108088,\n \"iqr\": 0.006878126998344669,\n \"q1\": 0.17357860000265646,\n \"q3\": 0.18045672700100113,\n \"iqr_outliers\": 0,\n \"stddev_outliers\": 3,\n \"outliers\": \"3;0\",\n \"ld15iqr\": 0.17086795599971083,\n \"hd15iqr\": 0.18105750900213025,\n \"ops\": 5.685847881085863,\n \"total\": 1.0552515870076604,\n \"data\": [\n 0.17497487199943862,\n 0.17431592300272314,\n 0.17086795599971083,\n 0.17357860000265646,\n 0.18105750900213025,\n 0.18045672700100113\n ],\n \"iterations\": 1\n }\n },\n {\n \"group\": \"xyzsearch\",\n \"name\": \"test1[4-2]\",\n \"fullname\": \"src/pypgstac/tests/test_benchmark.py::test1[4-2]\",\n \"params\": {\n \"zoom\": 4,\n \"item_width\": 2\n },\n \"param\": \"4-2\",\n \"extra_info\": {},\n \"options\": {\n \"disable_gc\": false,\n \"timer\": \"perf_counter\",\n \"min_rounds\": 3,\n \"max_time\": 1.0,\n \"min_time\": 5e-06,\n \"warmup\": 2\n },\n \"stats\": {\n \"min\": 0.11330923400237225,\n \"max\": 0.14413199599948712,\n \"mean\": 0.1252486348890266,\n \"stddev\": 0.009915628744136859,\n \"rounds\": 9,\n \"median\": 0.1253533779999998,\n \"iqr\": 0.013842049500453868,\n \"q1\": 0.11569722474996524,\n \"q3\": 0.1295392742504191,\n \"iqr_outliers\": 0,\n \"stddev_outliers\": 3,\n \"outliers\": \"3;0\",\n \"ld15iqr\": 0.11330923400237225,\n \"hd15iqr\": 0.14413199599948712,\n \"ops\": 7.984118955755683,\n \"total\": 1.1272377140012395,\n \"data\": [\n 0.1253533779999998,\n 0.11330923400237225,\n 0.12463980100073968,\n 0.1159379529999569,\n 0.11497503999999026,\n 0.12647421499787015,\n 0.12787050000042655,\n 0.14413199599948712,\n 0.1345455970003968\n ],\n \"iterations\": 1\n }\n },\n {\n \"group\": \"xyzsearch\",\n \"name\": \"test1[4-3]\",\n \"fullname\": \"src/pypgstac/tests/test_benchmark.py::test1[4-3]\",\n \"params\": {\n \"zoom\": 4,\n \"item_width\": 3\n },\n \"param\": \"4-3\",\n \"extra_info\": {},\n \"options\": {\n \"disable_gc\": false,\n \"timer\": \"perf_counter\",\n \"min_rounds\": 3,\n \"max_time\": 1.0,\n \"min_time\": 5e-06,\n \"warmup\": 2\n },\n \"stats\": {\n \"min\": 0.06794780299969716,\n \"max\": 0.07971649299724959,\n \"mean\": 0.07236937333267027,\n \"stddev\": 0.0032234028131627192,\n \"rounds\": 15,\n \"median\": 0.07156555899928208,\n \"iqr\": 0.004010758501863165,\n \"q1\": 0.0702488237475336,\n \"q3\": 0.07425958224939677,\n \"iqr_outliers\": 0,\n \"stddev_outliers\": 4,\n \"outliers\": \"4;0\",\n \"ld15iqr\": 0.06794780299969716,\n \"hd15iqr\": 0.07971649299724959,\n \"ops\": 13.817999990177643,\n \"total\": 1.085540599990054,\n \"data\": [\n 0.07195737400252256,\n 0.07060383200223441,\n 0.07430707499952405,\n 0.06810927799961064,\n 0.07010990799972205,\n 0.07129841300047701,\n 0.07156555899928208,\n 0.07971649299724959,\n 0.07411710399901494,\n 0.07689898899843683,\n 0.07484374299747287,\n 0.06794780299969716,\n 0.07022745199719793,\n 0.07031293899854063,\n 0.07352463799907127\n ],\n \"iterations\": 1\n }\n },\n {\n \"group\": \"xyzsearch\",\n \"name\": \"test1[4-4]\",\n \"fullname\": \"src/pypgstac/tests/test_benchmark.py::test1[4-4]\",\n \"params\": {\n \"zoom\": 4,\n \"item_width\": 4\n },\n \"param\": \"4-4\",\n \"extra_info\": {},\n \"options\": {\n \"disable_gc\": false,\n \"timer\": \"perf_counter\",\n \"min_rounds\": 3,\n \"max_time\": 1.0,\n \"min_time\": 5e-06,\n \"warmup\": 2\n },\n \"stats\": {\n \"min\": 0.055022680997353746,\n \"max\": 0.07121412800188409,\n \"mean\": 0.0596297509411456,\n \"stddev\": 0.004634585881486992,\n \"rounds\": 17,\n \"median\": 0.05803015600031358,\n \"iqr\": 0.0026907579995167907,\n \"q1\": 0.05688350125001307,\n \"q3\": 0.05957425924952986,\n \"iqr_outliers\": 3,\n \"stddev_outliers\": 3,\n \"outliers\": \"3;3\",\n \"ld15iqr\": 0.055022680997353746,\n \"hd15iqr\": 0.06487147699954221,\n \"ops\": 16.77015221792553,\n \"total\": 1.0137057659994753,\n \"data\": [\n 0.05694494400086114,\n 0.05621355599942035,\n 0.06983209000100032,\n 0.07121412800188409,\n 0.06487147699954221,\n 0.059164040998439305,\n 0.060804914002801524,\n 0.056618432001414476,\n 0.05837447899830295,\n 0.055022680997353746,\n 0.057981475998531096,\n 0.05858107200037921,\n 0.0578116910000972,\n 0.05820825400223839,\n 0.05803015600031358,\n 0.05669917299746885,\n 0.05733320199942682\n ],\n \"iterations\": 1\n }\n },\n {\n \"group\": \"xyzsearch\",\n \"name\": \"test1[4-5]\",\n \"fullname\": \"src/pypgstac/tests/test_benchmark.py::test1[4-5]\",\n \"params\": {\n \"zoom\": 4,\n \"item_width\": 5\n },\n \"param\": \"4-5\",\n \"extra_info\": {},\n \"options\": {\n \"disable_gc\": false,\n \"timer\": \"perf_counter\",\n \"min_rounds\": 3,\n \"max_time\": 1.0,\n \"min_time\": 5e-06,\n \"warmup\": 2\n },\n \"stats\": {\n \"min\": 0.048144559001229936,\n \"max\": 0.05387930500000948,\n \"mean\": 0.05032130766700166,\n \"stddev\": 0.0013144622688698053,\n \"rounds\": 21,\n \"median\": 0.050033582003379706,\n \"iqr\": 0.0009133455005212454,\n \"q1\": 0.049734376998458174,\n \"q3\": 0.05064772249897942,\n \"iqr_outliers\": 3,\n \"stddev_outliers\": 3,\n \"outliers\": \"3;3\",\n \"ld15iqr\": 0.04908135299774585,\n \"hd15iqr\": 0.0535943450013292,\n \"ops\": 19.872297568605372,\n \"total\": 1.0567474610070349,\n \"data\": [\n 0.0535943450013292,\n 0.05037315099980333,\n 0.049767618998885155,\n 0.05091931800052407,\n 0.05014399900028366,\n 0.050104821002605604,\n 0.05055837699910626,\n 0.049749811998481164,\n 0.04908135299774585,\n 0.049688071998389205,\n 0.050033582003379706,\n 0.0509157589985989,\n 0.050990560001082486,\n 0.04938530599974911,\n 0.049980152001808165,\n 0.04927488500106847,\n 0.04977712100298959,\n 0.048144559001229936,\n 0.04999440000028699,\n 0.050390964999678545,\n 0.05387930500000948\n ],\n \"iterations\": 1\n }\n },\n {\n \"group\": \"xyzsearch\",\n \"name\": \"test1[4-6]\",\n \"fullname\": \"src/pypgstac/tests/test_benchmark.py::test1[4-6]\",\n \"params\": {\n \"zoom\": 4,\n \"item_width\": 6\n },\n \"param\": \"4-6\",\n \"extra_info\": {},\n \"options\": {\n \"disable_gc\": false,\n \"timer\": \"perf_counter\",\n \"min_rounds\": 3,\n \"max_time\": 1.0,\n \"min_time\": 5e-06,\n \"warmup\": 2\n },\n \"stats\": {\n \"min\": 0.043895156999496976,\n \"max\": 0.04860524800096755,\n \"mean\": 0.046354956408487684,\n \"stddev\": 0.0013497296070651744,\n \"rounds\": 22,\n \"median\": 0.04642711300039082,\n \"iqr\": 0.0015482629969483241,\n \"q1\": 0.04566782300025807,\n \"q3\": 0.04721608599720639,\n \"iqr_outliers\": 0,\n \"stddev_outliers\": 8,\n \"outliers\": \"8;0\",\n \"ld15iqr\": 0.043895156999496976,\n \"hd15iqr\": 0.04860524800096755,\n \"ops\": 21.572666171613484,\n \"total\": 1.019809040986729,\n \"data\": [\n 0.04666873200039845,\n 0.04600739599845838,\n 0.045710566999332514,\n 0.04721608599720639,\n 0.043895156999496976,\n 0.04667348099974333,\n 0.04566782300025807,\n 0.04652031700243242,\n 0.04593021999971825,\n 0.04669129100147984,\n 0.04403644800186157,\n 0.04860524800096755,\n 0.04782636699746945,\n 0.04569038199770148,\n 0.04524751200005994,\n 0.04787148499963223,\n 0.04827398699853802,\n 0.046333908998349216,\n 0.04409462699914002,\n 0.04688363699824549,\n 0.04830604399830918,\n 0.04565832499793032\n ],\n \"iterations\": 1\n }\n },\n {\n \"group\": \"xyzsearch\",\n \"name\": \"test1[4-8]\",\n \"fullname\": \"src/pypgstac/tests/test_benchmark.py::test1[4-8]\",\n \"params\": {\n \"zoom\": 4,\n \"item_width\": 8\n },\n \"param\": \"4-8\",\n \"extra_info\": {},\n \"options\": {\n \"disable_gc\": false,\n \"timer\": \"perf_counter\",\n \"min_rounds\": 3,\n \"max_time\": 1.0,\n \"min_time\": 5e-06,\n \"warmup\": 2\n },\n \"stats\": {\n \"min\": 0.03777335199993104,\n \"max\": 0.0432480769995891,\n \"mean\": 0.04046258422170434,\n \"stddev\": 0.0014892429420970946,\n \"rounds\": 27,\n \"median\": 0.04061341799751972,\n \"iqr\": 0.0017895870014399407,\n \"q1\": 0.03936346749924269,\n \"q3\": 0.04115305450068263,\n \"iqr_outliers\": 0,\n \"stddev_outliers\": 9,\n \"outliers\": \"9;0\",\n \"ld15iqr\": 0.03777335199993104,\n \"hd15iqr\": 0.0432480769995891,\n \"ops\": 24.714190139729013,\n \"total\": 1.092489773986017,\n \"data\": [\n 0.039341798998066224,\n 0.0414837219977926,\n 0.038832438996905694,\n 0.0399212099982833,\n 0.03942847300277208,\n 0.04115958400143427,\n 0.04072858699873905,\n 0.0432480769995891,\n 0.041842292001092574,\n 0.03887399599989294,\n 0.03982503799852566,\n 0.03982266300226911,\n 0.039465280999138486,\n 0.04093755499707186,\n 0.039434409998648334,\n 0.04061341799751972,\n 0.038854998998431256,\n 0.04075708299933467,\n 0.04294649800067418,\n 0.04074164800113067,\n 0.03777335199993104,\n 0.039088900997739984,\n 0.04110259200024302,\n 0.038892993001354625,\n 0.043079480001324555,\n 0.04113346599842771,\n 0.04316021799968439\n ],\n \"iterations\": 1\n }\n },\n {\n \"group\": \"xyzsearch\",\n \"name\": \"test1[4-10]\",\n \"fullname\": \"src/pypgstac/tests/test_benchmark.py::test1[4-10]\",\n \"params\": {\n \"zoom\": 4,\n \"item_width\": 10\n },\n \"param\": \"4-10\",\n \"extra_info\": {},\n \"options\": {\n \"disable_gc\": false,\n \"timer\": \"perf_counter\",\n \"min_rounds\": 3,\n \"max_time\": 1.0,\n \"min_time\": 5e-06,\n \"warmup\": 2\n },\n \"stats\": {\n \"min\": 0.03276643600111129,\n \"max\": 0.03718326400121441,\n \"mean\": 0.03496493973090345,\n \"stddev\": 0.001112401201614798,\n \"rounds\": 26,\n \"median\": 0.03509476799990807,\n \"iqr\": 0.0016206899999815505,\n \"q1\": 0.03410217000055127,\n \"q3\": 0.03572286000053282,\n \"iqr_outliers\": 0,\n \"stddev_outliers\": 8,\n \"outliers\": \"8;0\",\n \"ld15iqr\": 0.03276643600111129,\n \"hd15iqr\": 0.03718326400121441,\n \"ops\": 28.600077897922386,\n \"total\": 0.9090884330034896,\n \"data\": [\n 0.03396681500089471,\n 0.03627852499994333,\n 0.035353603001567535,\n 0.03601612800048315,\n 0.03626427700146451,\n 0.03524199599996791,\n 0.03479556399906869,\n 0.034837120001611765,\n 0.03572286000053282,\n 0.033217616997717414,\n 0.03414016399983666,\n 0.03410217000055127,\n 0.03650292800011812,\n 0.03567655399820069,\n 0.03583802900175215,\n 0.03504727499966975,\n 0.03346695399886812,\n 0.03514226100014639,\n 0.03276643600111129,\n 0.03558394300125656,\n 0.03486442800203804,\n 0.03718326400121441,\n 0.03358331300114514,\n 0.03385520899973926,\n 0.03416866199768265,\n 0.0354723379969073\n ],\n \"iterations\": 1\n }\n },\n {\n \"group\": \"xyzsearch\",\n \"name\": \"test1[5-0.5]\",\n \"fullname\": \"src/pypgstac/tests/test_benchmark.py::test1[5-0.5]\",\n \"params\": {\n \"zoom\": 5,\n \"item_width\": 0.5\n },\n \"param\": \"5-0.5\",\n \"extra_info\": {},\n \"options\": {\n \"disable_gc\": false,\n \"timer\": \"perf_counter\",\n \"min_rounds\": 3,\n \"max_time\": 1.0,\n \"min_time\": 5e-06,\n \"warmup\": 2\n },\n \"stats\": {\n \"min\": 0.3528902499965625,\n \"max\": 0.3613380259994301,\n \"mean\": 0.3563750256653293,\n \"stddev\": 0.00441362560369112,\n \"rounds\": 3,\n \"median\": 0.3548968009999953,\n \"iqr\": 0.00633583200215071,\n \"q1\": 0.3533918877474207,\n \"q3\": 0.3597277197495714,\n \"iqr_outliers\": 0,\n \"stddev_outliers\": 1,\n \"outliers\": \"1;0\",\n \"ld15iqr\": 0.3528902499965625,\n \"hd15iqr\": 0.3613380259994301,\n \"ops\": 2.8060327688032127,\n \"total\": 1.069125076995988,\n \"data\": [\n 0.3548968009999953,\n 0.3613380259994301,\n 0.3528902499965625\n ],\n \"iterations\": 1\n }\n },\n {\n \"group\": \"xyzsearch\",\n \"name\": \"test1[5-0.75]\",\n \"fullname\": \"src/pypgstac/tests/test_benchmark.py::test1[5-0.75]\",\n \"params\": {\n \"zoom\": 5,\n \"item_width\": 0.75\n },\n \"param\": \"5-0.75\",\n \"extra_info\": {},\n \"options\": {\n \"disable_gc\": false,\n \"timer\": \"perf_counter\",\n \"min_rounds\": 3,\n \"max_time\": 1.0,\n \"min_time\": 5e-06,\n \"warmup\": 2\n },\n \"stats\": {\n \"min\": 0.1738603650010191,\n \"max\": 0.18968734400186804,\n \"mean\": 0.1816396906663916,\n \"stddev\": 0.006070635494990262,\n \"rounds\": 6,\n \"median\": 0.1798818664992723,\n \"iqr\": 0.00968734199705068,\n \"q1\": 0.17841967999993358,\n \"q3\": 0.18810702199698426,\n \"iqr_outliers\": 0,\n \"stddev_outliers\": 3,\n \"outliers\": \"3;0\",\n \"ld15iqr\": 0.1738603650010191,\n \"hd15iqr\": 0.18968734400186804,\n \"ops\": 5.50540466310664,\n \"total\": 1.0898381439983496,\n \"data\": [\n 0.1738603650010191,\n 0.17841967999993358,\n 0.18043218900129432,\n 0.17933154399725026,\n 0.18810702199698426,\n 0.18968734400186804\n ],\n \"iterations\": 1\n }\n },\n {\n \"group\": \"xyzsearch\",\n \"name\": \"test1[5-1]\",\n \"fullname\": \"src/pypgstac/tests/test_benchmark.py::test1[5-1]\",\n \"params\": {\n \"zoom\": 5,\n \"item_width\": 1\n },\n \"param\": \"5-1\",\n \"extra_info\": {},\n \"options\": {\n \"disable_gc\": false,\n \"timer\": \"perf_counter\",\n \"min_rounds\": 3,\n \"max_time\": 1.0,\n \"min_time\": 5e-06,\n \"warmup\": 2\n },\n \"stats\": {\n \"min\": 0.11908579000009922,\n \"max\": 0.14131123699917225,\n \"mean\": 0.13000069355580812,\n \"stddev\": 0.009213577351100307,\n \"rounds\": 9,\n \"median\": 0.1281640450033592,\n \"iqr\": 0.019572380000681733,\n \"q1\": 0.12101755949879589,\n \"q3\": 0.14058993949947762,\n \"iqr_outliers\": 0,\n \"stddev_outliers\": 5,\n \"outliers\": \"5;0\",\n \"ld15iqr\": 0.11908579000009922,\n \"hd15iqr\": 0.14131123699917225,\n \"ops\": 7.69226665372142,\n \"total\": 1.1700062420022732,\n \"data\": [\n 0.12073260299803223,\n 0.12111254499905044,\n 0.11908579000009922,\n 0.1252064270011033,\n 0.14116875800027628,\n 0.1281640450033592,\n 0.13282783700196887,\n 0.14131123699917225,\n 0.1403969999992114\n ],\n \"iterations\": 1\n }\n },\n {\n \"group\": \"xyzsearch\",\n \"name\": \"test1[5-1.5]\",\n \"fullname\": \"src/pypgstac/tests/test_benchmark.py::test1[5-1.5]\",\n \"params\": {\n \"zoom\": 5,\n \"item_width\": 1.5\n },\n \"param\": \"5-1.5\",\n \"extra_info\": {},\n \"options\": {\n \"disable_gc\": false,\n \"timer\": \"perf_counter\",\n \"min_rounds\": 3,\n \"max_time\": 1.0,\n \"min_time\": 5e-06,\n \"warmup\": 2\n },\n \"stats\": {\n \"min\": 0.07683741799701238,\n \"max\": 0.08334037099848501,\n \"mean\": 0.07977976357064367,\n \"stddev\": 0.001969583375656267,\n \"rounds\": 14,\n \"median\": 0.0792417389984621,\n \"iqr\": 0.003077533001487609,\n \"q1\": 0.0782598229998257,\n \"q3\": 0.08133735600131331,\n \"iqr_outliers\": 0,\n \"stddev_outliers\": 3,\n \"outliers\": \"3;0\",\n \"ld15iqr\": 0.07683741799701238,\n \"hd15iqr\": 0.08334037099848501,\n \"ops\": 12.534506938147496,\n \"total\": 1.1169166899890115,\n \"data\": [\n 0.07964304999768501,\n 0.07898171299893875,\n 0.0782598229998257,\n 0.08133735600131331,\n 0.0809538520006754,\n 0.07811615699756658,\n 0.08290818000023137,\n 0.08173985699977493,\n 0.07891759799895226,\n 0.0780294860014692,\n 0.07683741799701238,\n 0.07950176499798545,\n 0.07835006399909616,\n 0.08334037099848501\n ],\n \"iterations\": 1\n }\n },\n {\n \"group\": \"xyzsearch\",\n \"name\": \"test1[5-2]\",\n \"fullname\": \"src/pypgstac/tests/test_benchmark.py::test1[5-2]\",\n \"params\": {\n \"zoom\": 5,\n \"item_width\": 2\n },\n \"param\": \"5-2\",\n \"extra_info\": {},\n \"options\": {\n \"disable_gc\": false,\n \"timer\": \"perf_counter\",\n \"min_rounds\": 3,\n \"max_time\": 1.0,\n \"min_time\": 5e-06,\n \"warmup\": 2\n },\n \"stats\": {\n \"min\": 0.05488625500220223,\n \"max\": 0.0652931129989156,\n \"mean\": 0.05818601472209492,\n \"stddev\": 0.0029684394659227147,\n \"rounds\": 18,\n \"median\": 0.057458582497929456,\n \"iqr\": 0.0033470590024080593,\n \"q1\": 0.05617568099842174,\n \"q3\": 0.0595227400008298,\n \"iqr_outliers\": 2,\n \"stddev_outliers\": 4,\n \"outliers\": \"4;2\",\n \"ld15iqr\": 0.05488625500220223,\n \"hd15iqr\": 0.06502834500133758,\n \"ops\": 17.18626038879186,\n \"total\": 1.0473482649977086,\n \"data\": [\n 0.05642501900001662,\n 0.0652931129989156,\n 0.05617568099842174,\n 0.05500854599813465,\n 0.05781774499700987,\n 0.05565800999829662,\n 0.05653662999975495,\n 0.05488625500220223,\n 0.05593465900165029,\n 0.05695694200039725,\n 0.05792698200093582,\n 0.0595227400008298,\n 0.05911192699932144,\n 0.05862750099913683,\n 0.059550049001700245,\n 0.06502834500133758,\n 0.05978870100079803,\n 0.057099419998849044\n ],\n \"iterations\": 1\n }\n },\n {\n \"group\": \"xyzsearch\",\n \"name\": \"test1[5-3]\",\n \"fullname\": \"src/pypgstac/tests/test_benchmark.py::test1[5-3]\",\n \"params\": {\n \"zoom\": 5,\n \"item_width\": 3\n },\n \"param\": \"5-3\",\n \"extra_info\": {},\n \"options\": {\n \"disable_gc\": false,\n \"timer\": \"perf_counter\",\n \"min_rounds\": 3,\n \"max_time\": 1.0,\n \"min_time\": 5e-06,\n \"warmup\": 2\n },\n \"stats\": {\n \"min\": 0.04489852500046254,\n \"max\": 0.05440302599890856,\n \"mean\": 0.048053970475891786,\n \"stddev\": 0.002020601273945267,\n \"rounds\": 21,\n \"median\": 0.047797962000913685,\n \"iqr\": 0.0021989167489664396,\n \"q1\": 0.04697752375068376,\n \"q3\": 0.0491764404996502,\n \"iqr_outliers\": 1,\n \"stddev_outliers\": 5,\n \"outliers\": \"5;1\",\n \"ld15iqr\": 0.04489852500046254,\n \"hd15iqr\": 0.05440302599890856,\n \"ops\": 20.80993495639846,\n \"total\": 1.0091333799937274,\n \"data\": [\n 0.04763054699651548,\n 0.04489852500046254,\n 0.047797962000913685,\n 0.04580801299744053,\n 0.04957300499881967,\n 0.04843555300249136,\n 0.04801880300146877,\n 0.049190688998351106,\n 0.04728741300277761,\n 0.049315358002786525,\n 0.048169592999329325,\n 0.047464323997701285,\n 0.05080188200008706,\n 0.046753120001085335,\n 0.0470523250005499,\n 0.04726841699812212,\n 0.04648834699764848,\n 0.05440302599890856,\n 0.04917169100008323,\n 0.04577358000096865,\n 0.047831206997216213\n ],\n \"iterations\": 1\n }\n },\n {\n \"group\": \"xyzsearch\",\n \"name\": \"test1[5-4]\",\n \"fullname\": \"src/pypgstac/tests/test_benchmark.py::test1[5-4]\",\n \"params\": {\n \"zoom\": 5,\n \"item_width\": 4\n },\n \"param\": \"5-4\",\n \"extra_info\": {},\n \"options\": {\n \"disable_gc\": false,\n \"timer\": \"perf_counter\",\n \"min_rounds\": 3,\n \"max_time\": 1.0,\n \"min_time\": 5e-06,\n \"warmup\": 2\n },\n \"stats\": {\n \"min\": 0.03521118299977388,\n \"max\": 0.042654497003240976,\n \"mean\": 0.038113435110938516,\n \"stddev\": 0.0017210758691116373,\n \"rounds\": 27,\n \"median\": 0.037680810997699155,\n \"iqr\": 0.0016856995016496512,\n \"q1\": 0.03702540874837723,\n \"q3\": 0.03871110825002688,\n \"iqr_outliers\": 2,\n \"stddev_outliers\": 8,\n \"outliers\": \"8;2\",\n \"ld15iqr\": 0.03521118299977388,\n \"hd15iqr\": 0.04135794299872941,\n \"ops\": 26.237467105477485,\n \"total\": 1.02906274799534,\n \"data\": [\n 0.03717857399897184,\n 0.04093288200238021,\n 0.037019471998064546,\n 0.039954528998350725,\n 0.037556142000539694,\n 0.04005070199855254,\n 0.03872921500078519,\n 0.042654497003240976,\n 0.038476314999570604,\n 0.03755851599999005,\n 0.03729255600046599,\n 0.03865678799775196,\n 0.037043218999315286,\n 0.03655166799944709,\n 0.03736498299986124,\n 0.03565524099758477,\n 0.03813792900109547,\n 0.03521118299977388,\n 0.03823172700140276,\n 0.03802275899943197,\n 0.03685324800244416,\n 0.03694704699955764,\n 0.037680810997699155,\n 0.04135794299872941,\n 0.03627146300277673,\n 0.03928250799799571,\n 0.03839083099956042\n ],\n \"iterations\": 1\n }\n },\n {\n \"group\": \"xyzsearch\",\n \"name\": \"test1[5-5]\",\n \"fullname\": \"src/pypgstac/tests/test_benchmark.py::test1[5-5]\",\n \"params\": {\n \"zoom\": 5,\n \"item_width\": 5\n },\n \"param\": \"5-5\",\n \"extra_info\": {},\n \"options\": {\n \"disable_gc\": false,\n \"timer\": \"perf_counter\",\n \"min_rounds\": 3,\n \"max_time\": 1.0,\n \"min_time\": 5e-06,\n \"warmup\": 2\n },\n \"stats\": {\n \"min\": 0.03793965599834337,\n \"max\": 0.04555513000013889,\n \"mean\": 0.04055096839983889,\n \"stddev\": 0.001934729870115932,\n \"rounds\": 30,\n \"median\": 0.04005605349993857,\n \"iqr\": 0.0014117259997874498,\n \"q1\": 0.03940837000118336,\n \"q3\": 0.04082009600097081,\n \"iqr_outliers\": 4,\n \"stddev_outliers\": 7,\n \"outliers\": \"7;4\",\n \"ld15iqr\": 0.03793965599834337,\n \"hd15iqr\": 0.04337045899956138,\n \"ops\": 24.66032352519534,\n \"total\": 1.2165290519951668,\n \"data\": [\n 0.03940837000118336,\n 0.03993316499690991,\n 0.040693049999390496,\n 0.04137338500004262,\n 0.040091078997647855,\n 0.04555513000013889,\n 0.04147193300013896,\n 0.03979306199835264,\n 0.039633960001083324,\n 0.04014094599915552,\n 0.03979187299773912,\n 0.03993672700016759,\n 0.040509015998395626,\n 0.04054226000152994,\n 0.04337045899956138,\n 0.03941786699942895,\n 0.040662180002982495,\n 0.038028702001611236,\n 0.03920414999811328,\n 0.04003408700009459,\n 0.045365158999629784,\n 0.03905455000131042,\n 0.040078019999782555,\n 0.038662733997625764,\n 0.04082009600097081,\n 0.042519153001194354,\n 0.04443430100218393,\n 0.03793965599834337,\n 0.03873634800038417,\n 0.039327634000073886\n ],\n \"iterations\": 1\n }\n },\n {\n \"group\": \"xyzsearch\",\n \"name\": \"test1[5-6]\",\n \"fullname\": \"src/pypgstac/tests/test_benchmark.py::test1[5-6]\",\n \"params\": {\n \"zoom\": 5,\n \"item_width\": 6\n },\n \"param\": \"5-6\",\n \"extra_info\": {},\n \"options\": {\n \"disable_gc\": false,\n \"timer\": \"perf_counter\",\n \"min_rounds\": 3,\n \"max_time\": 1.0,\n \"min_time\": 5e-06,\n \"warmup\": 2\n },\n \"stats\": {\n \"min\": 0.032226272000116296,\n \"max\": 0.039702834001218434,\n \"mean\": 0.03612378748385386,\n \"stddev\": 0.0015639443296231721,\n \"rounds\": 31,\n \"median\": 0.03627503800089471,\n \"iqr\": 0.0017682202478681575,\n \"q1\": 0.03527353050230886,\n \"q3\": 0.037041750750177016,\n \"iqr_outliers\": 2,\n \"stddev_outliers\": 8,\n \"outliers\": \"8;2\",\n \"ld15iqr\": 0.03333166900119977,\n \"hd15iqr\": 0.039702834001218434,\n \"ops\": 27.682590050862384,\n \"total\": 1.1198374119994696,\n \"data\": [\n 0.03657423999902676,\n 0.03635696099809138,\n 0.03394788699733908,\n 0.03865205299734953,\n 0.035572141998272855,\n 0.03539641799943638,\n 0.03511977200105321,\n 0.037492039999051485,\n 0.03589865500180167,\n 0.03558995099956519,\n 0.035221881000325084,\n 0.03523375500299153,\n 0.03443706300095073,\n 0.03573005499856663,\n 0.03539285700026085,\n 0.037199960999714676,\n 0.03654574699976365,\n 0.032226272000116296,\n 0.03487518600013573,\n 0.03635221399963484,\n 0.0382186829992861,\n 0.03577517599842395,\n 0.038712609002686804,\n 0.03627503800089471,\n 0.03629640900180675,\n 0.03712872200048878,\n 0.03653862300052424,\n 0.03333166900119977,\n 0.03698030699888477,\n 0.037062232000607764,\n 0.039702834001218434\n ],\n \"iterations\": 1\n }\n },\n {\n \"group\": \"xyzsearch\",\n \"name\": \"test1[5-8]\",\n \"fullname\": \"src/pypgstac/tests/test_benchmark.py::test1[5-8]\",\n \"params\": {\n \"zoom\": 5,\n \"item_width\": 8\n },\n \"param\": \"5-8\",\n \"extra_info\": {},\n \"options\": {\n \"disable_gc\": false,\n \"timer\": \"perf_counter\",\n \"min_rounds\": 3,\n \"max_time\": 1.0,\n \"min_time\": 5e-06,\n \"warmup\": 2\n },\n \"stats\": {\n \"min\": 0.030467855001916178,\n \"max\": 0.03646382800070569,\n \"mean\": 0.03345623115592389,\n \"stddev\": 0.0012971261642753505,\n \"rounds\": 32,\n \"median\": 0.033450998498665285,\n \"iqr\": 0.0016723429998819483,\n \"q1\": 0.03265430600004038,\n \"q3\": 0.03432664899992233,\n \"iqr_outliers\": 0,\n \"stddev_outliers\": 8,\n \"outliers\": \"8;0\",\n \"ld15iqr\": 0.030467855001916178,\n \"hd15iqr\": 0.03646382800070569,\n \"ops\": 29.889798266262165,\n \"total\": 1.0705993969895644,\n \"data\": [\n 0.03300872199906735,\n 0.033955015998799354,\n 0.03407256100035738,\n 0.03350977099762531,\n 0.03443825599970296,\n 0.033723488999385154,\n 0.033799476997955935,\n 0.03339222599970526,\n 0.03351808200022788,\n 0.032727325000450946,\n 0.03438245200231904,\n 0.03174541200132808,\n 0.03243880700028967,\n 0.03119449600126245,\n 0.03338629100107937,\n 0.03163974199924269,\n 0.030467855001916178,\n 0.03351452199785854,\n 0.03527769399806857,\n 0.03313814099965384,\n 0.03427084599752561,\n 0.03279144299813197,\n 0.03513284099972225,\n 0.03254091800044989,\n 0.03258128699962981,\n 0.03308114999890677,\n 0.03646382800070569,\n 0.03286386999752722,\n 0.034547491999546764,\n 0.03583811000135029,\n 0.032556353999098064,\n 0.034600921000674134\n ],\n \"iterations\": 1\n }\n },\n {\n \"group\": \"xyzsearch\",\n \"name\": \"test1[5-10]\",\n \"fullname\": \"src/pypgstac/tests/test_benchmark.py::test1[5-10]\",\n \"params\": {\n \"zoom\": 5,\n \"item_width\": 10\n },\n \"param\": \"5-10\",\n \"extra_info\": {},\n \"options\": {\n \"disable_gc\": false,\n \"timer\": \"perf_counter\",\n \"min_rounds\": 3,\n \"max_time\": 1.0,\n \"min_time\": 5e-06,\n \"warmup\": 2\n },\n \"stats\": {\n \"min\": 0.03186058700157446,\n \"max\": 0.039468946000852156,\n \"mean\": 0.036132189964389126,\n \"stddev\": 0.0015354948988810695,\n \"rounds\": 28,\n \"median\": 0.0361070445014775,\n \"iqr\": 0.0010786809998535318,\n \"q1\": 0.035639239500596887,\n \"q3\": 0.03671792050045042,\n \"iqr_outliers\": 4,\n \"stddev_outliers\": 7,\n \"outliers\": \"7;4\",\n \"ld15iqr\": 0.03407731799961766,\n \"hd15iqr\": 0.03914836900003138,\n \"ops\": 27.676152510699513,\n \"total\": 1.0117013190028956,\n \"data\": [\n 0.03186058700157446,\n 0.03628336000110721,\n 0.03606133100038278,\n 0.03678203500021482,\n 0.034916751999844564,\n 0.03575975299827405,\n 0.03663480899922433,\n 0.036408030002348823,\n 0.03615038200223353,\n 0.03407731799961766,\n 0.03781619399887859,\n 0.03565883000192116,\n 0.039468946000852156,\n 0.03572532099860837,\n 0.03914836900003138,\n 0.03829468500043731,\n 0.03606370700072148,\n 0.03556978099732078,\n 0.03700169099829509,\n 0.0358013100012613,\n 0.03665380600068602,\n 0.036865149002551334,\n 0.03629761000047438,\n 0.03508416700060479,\n 0.03599721799764666,\n 0.036259615000744816,\n 0.03561964899927261,\n 0.033440913997765165\n ],\n \"iterations\": 1\n }\n },\n {\n \"group\": \"xyzsearch\",\n \"name\": \"test1[6-0.5]\",\n \"fullname\": \"src/pypgstac/tests/test_benchmark.py::test1[6-0.5]\",\n \"params\": {\n \"zoom\": 6,\n \"item_width\": 0.5\n },\n \"param\": \"6-0.5\",\n \"extra_info\": {},\n \"options\": {\n \"disable_gc\": false,\n \"timer\": \"perf_counter\",\n \"min_rounds\": 3,\n \"max_time\": 1.0,\n \"min_time\": 5e-06,\n \"warmup\": 2\n },\n \"stats\": {\n \"min\": 0.1268487230008759,\n \"max\": 0.14295829899856471,\n \"mean\": 0.1350773039998785,\n \"stddev\": 0.005595496365896534,\n \"rounds\": 8,\n \"median\": 0.1347099775011884,\n \"iqr\": 0.009163152501059812,\n \"q1\": 0.13076628749877273,\n \"q3\": 0.13992943999983254,\n \"iqr_outliers\": 0,\n \"stddev_outliers\": 3,\n \"outliers\": \"3;0\",\n \"ld15iqr\": 0.1268487230008759,\n \"hd15iqr\": 0.14295829899856471,\n \"ops\": 7.403168188794318,\n \"total\": 1.080618431999028,\n \"data\": [\n 0.14295829899856471,\n 0.13436564999938128,\n 0.13881217000016477,\n 0.1410467099995003,\n 0.13178797499858774,\n 0.1297445999989577,\n 0.13505430500299553,\n 0.1268487230008759\n ],\n \"iterations\": 1\n }\n },\n {\n \"group\": \"xyzsearch\",\n \"name\": \"test1[6-0.75]\",\n \"fullname\": \"src/pypgstac/tests/test_benchmark.py::test1[6-0.75]\",\n \"params\": {\n \"zoom\": 6,\n \"item_width\": 0.75\n },\n \"param\": \"6-0.75\",\n \"extra_info\": {},\n \"options\": {\n \"disable_gc\": false,\n \"timer\": \"perf_counter\",\n \"min_rounds\": 3,\n \"max_time\": 1.0,\n \"min_time\": 5e-06,\n \"warmup\": 2\n },\n \"stats\": {\n \"min\": 0.07463744200140354,\n \"max\": 0.08529722000093898,\n \"mean\": 0.07939257461503775,\n \"stddev\": 0.002691959172431208,\n \"rounds\": 13,\n \"median\": 0.0788595570011239,\n \"iqr\": 0.0035290212490508566,\n \"q1\": 0.07768084224881022,\n \"q3\": 0.08120986349786108,\n \"iqr_outliers\": 0,\n \"stddev_outliers\": 2,\n \"outliers\": \"2;0\",\n \"ld15iqr\": 0.07463744200140354,\n \"hd15iqr\": 0.08529722000093898,\n \"ops\": 12.595636365854672,\n \"total\": 1.0321034699954907,\n \"data\": [\n 0.07994239300023764,\n 0.07783370899778674,\n 0.0776900429991656,\n 0.0788595570011239,\n 0.07463744200140354,\n 0.08125438799834228,\n 0.0776532399977441,\n 0.08181836599760572,\n 0.08119502199770068,\n 0.07718187399950693,\n 0.08529722000093898,\n 0.0806061100010993,\n 0.07813410600283532\n ],\n \"iterations\": 1\n }\n },\n {\n \"group\": \"xyzsearch\",\n \"name\": \"test1[6-1]\",\n \"fullname\": \"src/pypgstac/tests/test_benchmark.py::test1[6-1]\",\n \"params\": {\n \"zoom\": 6,\n \"item_width\": 1\n },\n \"param\": \"6-1\",\n \"extra_info\": {},\n \"options\": {\n \"disable_gc\": false,\n \"timer\": \"perf_counter\",\n \"min_rounds\": 3,\n \"max_time\": 1.0,\n \"min_time\": 5e-06,\n \"warmup\": 2\n },\n \"stats\": {\n \"min\": 0.05851480500132311,\n \"max\": 0.06430774700129405,\n \"mean\": 0.06168284427783672,\n \"stddev\": 0.0016228951986026318,\n \"rounds\": 18,\n \"median\": 0.06135784850084747,\n \"iqr\": 0.0027047200019296724,\n \"q1\": 0.060831270999187836,\n \"q3\": 0.06353599100111751,\n \"iqr_outliers\": 0,\n \"stddev_outliers\": 7,\n \"outliers\": \"7;0\",\n \"ld15iqr\": 0.05851480500132311,\n \"hd15iqr\": 0.06430774700129405,\n \"ops\": 16.211963175623378,\n \"total\": 1.110291197001061,\n \"data\": [\n 0.06118627600153559,\n 0.06430774700129405,\n 0.06387675199948717,\n 0.06353599100111751,\n 0.06052612900020904,\n 0.060831270999187836,\n 0.062220438001531875,\n 0.05851480500132311,\n 0.06128720299966517,\n 0.06124683300004108,\n 0.05954658799964818,\n 0.06385656800193829,\n 0.061428494002029765,\n 0.06006544800038682,\n 0.06371408899940434,\n 0.06163390099754906,\n 0.06156741099766805,\n 0.06094525299704401\n ],\n \"iterations\": 1\n }\n },\n {\n \"group\": \"xyzsearch\",\n \"name\": \"test1[6-1.5]\",\n \"fullname\": \"src/pypgstac/tests/test_benchmark.py::test1[6-1.5]\",\n \"params\": {\n \"zoom\": 6,\n \"item_width\": 1.5\n },\n \"param\": \"6-1.5\",\n \"extra_info\": {},\n \"options\": {\n \"disable_gc\": false,\n \"timer\": \"perf_counter\",\n \"min_rounds\": 3,\n \"max_time\": 1.0,\n \"min_time\": 5e-06,\n \"warmup\": 2\n },\n \"stats\": {\n \"min\": 0.042510893999860855,\n \"max\": 0.049666887000057613,\n \"mean\": 0.04612569804472018,\n \"stddev\": 0.0016540591589389979,\n \"rounds\": 22,\n \"median\": 0.04573744499793975,\n \"iqr\": 0.0020493189986154903,\n \"q1\": 0.04529991500021424,\n \"q3\": 0.04734923399882973,\n \"iqr_outliers\": 0,\n \"stddev_outliers\": 8,\n \"outliers\": \"8;0\",\n \"ld15iqr\": 0.042510893999860855,\n \"hd15iqr\": 0.049666887000057613,\n \"ops\": 21.679888703916664,\n \"total\": 1.014765356983844,\n \"data\": [\n 0.04734923399882973,\n 0.04429187900313991,\n 0.042510893999860855,\n 0.04473474900078145,\n 0.04529991500021424,\n 0.04554212900256971,\n 0.04442604599898914,\n 0.045679858998482814,\n 0.047726801996759605,\n 0.04783722299907822,\n 0.04555756400077371,\n 0.04424794700025814,\n 0.048447506997035816,\n 0.04612629199982621,\n 0.04820410599859315,\n 0.046539479997591116,\n 0.04555162799806567,\n 0.04567985799803864,\n 0.046430245998635655,\n 0.049666887000057613,\n 0.04712008099886589,\n 0.04579503099739668\n ],\n \"iterations\": 1\n }\n },\n {\n \"group\": \"xyzsearch\",\n \"name\": \"test1[6-2]\",\n \"fullname\": \"src/pypgstac/tests/test_benchmark.py::test1[6-2]\",\n \"params\": {\n \"zoom\": 6,\n \"item_width\": 2\n },\n \"param\": \"6-2\",\n \"extra_info\": {},\n \"options\": {\n \"disable_gc\": false,\n \"timer\": \"perf_counter\",\n \"min_rounds\": 3,\n \"max_time\": 1.0,\n \"min_time\": 5e-06,\n \"warmup\": 2\n },\n \"stats\": {\n \"min\": 0.042301935001887614,\n \"max\": 0.0495885320015077,\n \"mean\": 0.044751182250062506,\n \"stddev\": 0.0019226710688970588,\n \"rounds\": 24,\n \"median\": 0.04421233649918577,\n \"iqr\": 0.0012057274998369394,\n \"q1\": 0.04363173349884164,\n \"q3\": 0.04483746099867858,\n \"iqr_outliers\": 4,\n \"stddev_outliers\": 6,\n \"outliers\": \"6;4\",\n \"ld15iqr\": 0.042301935001887614,\n \"hd15iqr\": 0.04736823900020681,\n \"ops\": 22.345778362058876,\n \"total\": 1.0740283740015002,\n \"data\": [\n 0.04329690899976413,\n 0.04402592499900493,\n 0.04481430699888733,\n 0.0441862139996374,\n 0.043782525001006434,\n 0.044828555997810327,\n 0.04367328999796882,\n 0.044974595999519806,\n 0.043363400000089314,\n 0.04945198999848799,\n 0.04400217899819836,\n 0.04736823900020681,\n 0.0480746960020042,\n 0.0495885320015077,\n 0.04359017699971446,\n 0.04392381500292686,\n 0.04484636599954683,\n 0.0427697380000609,\n 0.042301935001887614,\n 0.044630275002418784,\n 0.044303760998445796,\n 0.04423845899873413,\n 0.04352725300122984,\n 0.04446523700244143\n ],\n \"iterations\": 1\n }\n },\n {\n \"group\": \"xyzsearch\",\n \"name\": \"test1[6-3]\",\n \"fullname\": \"src/pypgstac/tests/test_benchmark.py::test1[6-3]\",\n \"params\": {\n \"zoom\": 6,\n \"item_width\": 3\n },\n \"param\": \"6-3\",\n \"extra_info\": {},\n \"options\": {\n \"disable_gc\": false,\n \"timer\": \"perf_counter\",\n \"min_rounds\": 3,\n \"max_time\": 1.0,\n \"min_time\": 5e-06,\n \"warmup\": 2\n },\n \"stats\": {\n \"min\": 0.03260270100145135,\n \"max\": 0.04594939700109535,\n \"mean\": 0.03932017700019599,\n \"stddev\": 0.0033886416698431653,\n \"rounds\": 27,\n \"median\": 0.039124662998801796,\n \"iqr\": 0.0042791097512235865,\n \"q1\": 0.037114231248779106,\n \"q3\": 0.04139334100000269,\n \"iqr_outliers\": 0,\n \"stddev_outliers\": 8,\n \"outliers\": \"8;0\",\n \"ld15iqr\": 0.03260270100145135,\n \"hd15iqr\": 0.04594939700109535,\n \"ops\": 25.432235465140852,\n \"total\": 1.0616447790052916,\n \"data\": [\n 0.03918521699961275,\n 0.039124662998801796,\n 0.04085459300040384,\n 0.03933006999795907,\n 0.038238919998548226,\n 0.04558013800124172,\n 0.045501774999138433,\n 0.0415183059994888,\n 0.03773312100020121,\n 0.04012320200126851,\n 0.03787322599964682,\n 0.04200154600039241,\n 0.04346076600268134,\n 0.04594939700109535,\n 0.0377901150022808,\n 0.042507349000516115,\n 0.04101844600154436,\n 0.040246685999591136,\n 0.035464149001199985,\n 0.037032602998806396,\n 0.037015980000433046,\n 0.03260270100145135,\n 0.037359115998697234,\n 0.034945289000461344,\n 0.03660397899875534,\n 0.03765594699871144,\n 0.034927479002362816\n ],\n \"iterations\": 1\n }\n },\n {\n \"group\": \"xyzsearch\",\n \"name\": \"test1[6-4]\",\n \"fullname\": \"src/pypgstac/tests/test_benchmark.py::test1[6-4]\",\n \"params\": {\n \"zoom\": 6,\n \"item_width\": 4\n },\n \"param\": \"6-4\",\n \"extra_info\": {},\n \"options\": {\n \"disable_gc\": false,\n \"timer\": \"perf_counter\",\n \"min_rounds\": 3,\n \"max_time\": 1.0,\n \"min_time\": 5e-06,\n \"warmup\": 2\n },\n \"stats\": {\n \"min\": 0.036126680999586824,\n \"max\": 0.04487012900062837,\n \"mean\": 0.03896573932161118,\n \"stddev\": 0.002392502885028669,\n \"rounds\": 28,\n \"median\": 0.03843424250044336,\n \"iqr\": 0.0034972590001416393,\n \"q1\": 0.037080099999002414,\n \"q3\": 0.04057735899914405,\n \"iqr_outliers\": 0,\n \"stddev_outliers\": 8,\n \"outliers\": \"8;0\",\n \"ld15iqr\": 0.036126680999586824,\n \"hd15iqr\": 0.04487012900062837,\n \"ops\": 25.663570547098022,\n \"total\": 1.091040701005113,\n \"data\": [\n 0.04196949600009248,\n 0.038218742000026396,\n 0.03735437100112904,\n 0.04305352200026391,\n 0.04069430999879842,\n 0.041581241002859315,\n 0.03641519799930393,\n 0.036954242998035625,\n 0.04046040799948969,\n 0.03879103300278075,\n 0.03729975600072066,\n 0.03675952399862581,\n 0.04079879699929734,\n 0.04363412499878905,\n 0.039502240000729216,\n 0.036126680999586824,\n 0.03678683200269006,\n 0.03678920699894661,\n 0.038649743000860326,\n 0.0372059569999692,\n 0.04487012900062837,\n 0.039962922000995604,\n 0.037445797002874315,\n 0.03743511100037722,\n 0.036178923000989016,\n 0.03801689899773919,\n 0.03899881599863875,\n 0.039086677999875974\n ],\n \"iterations\": 1\n }\n },\n {\n \"group\": \"xyzsearch\",\n \"name\": \"test1[6-5]\",\n \"fullname\": \"src/pypgstac/tests/test_benchmark.py::test1[6-5]\",\n \"params\": {\n \"zoom\": 6,\n \"item_width\": 5\n },\n \"param\": \"6-5\",\n \"extra_info\": {},\n \"options\": {\n \"disable_gc\": false,\n \"timer\": \"perf_counter\",\n \"min_rounds\": 3,\n \"max_time\": 1.0,\n \"min_time\": 5e-06,\n \"warmup\": 2\n },\n \"stats\": {\n \"min\": 0.03487881000182824,\n \"max\": 0.04174747499928344,\n \"mean\": 0.037700271428808003,\n \"stddev\": 0.002003219081389196,\n \"rounds\": 28,\n \"median\": 0.03710860299906926,\n \"iqr\": 0.0032580150000285357,\n \"q1\": 0.03605426000103762,\n \"q3\": 0.039312275001066155,\n \"iqr_outliers\": 0,\n \"stddev_outliers\": 10,\n \"outliers\": \"10;0\",\n \"ld15iqr\": 0.03487881000182824,\n \"hd15iqr\": 0.04174747499928344,\n \"ops\": 26.52500796680916,\n \"total\": 1.0556076000066241,\n \"data\": [\n 0.036404519003554014,\n 0.03566244100147742,\n 0.04006503599885036,\n 0.041011334000359057,\n 0.037138285999390064,\n 0.035888034999516094,\n 0.03707891999874846,\n 0.03828286599673447,\n 0.035546086000977084,\n 0.035413106001215056,\n 0.036980372002290096,\n 0.03811782800039509,\n 0.03661348899913719,\n 0.03766901900235098,\n 0.03919829200094682,\n 0.03589159600232961,\n 0.03487881000182824,\n 0.03893351800070377,\n 0.03942625800118549,\n 0.04164892799963127,\n 0.04174747499928344,\n 0.034936987998662516,\n 0.036812959999224404,\n 0.039606730999366846,\n 0.03621692399974563,\n 0.03673459600031492,\n 0.03782693399989512,\n 0.03987625299851061\n ],\n \"iterations\": 1\n }\n },\n {\n \"group\": \"xyzsearch\",\n \"name\": \"test1[6-6]\",\n \"fullname\": \"src/pypgstac/tests/test_benchmark.py::test1[6-6]\",\n \"params\": {\n \"zoom\": 6,\n \"item_width\": 6\n },\n \"param\": \"6-6\",\n \"extra_info\": {},\n \"options\": {\n \"disable_gc\": false,\n \"timer\": \"perf_counter\",\n \"min_rounds\": 3,\n \"max_time\": 1.0,\n \"min_time\": 5e-06,\n \"warmup\": 2\n },\n \"stats\": {\n \"min\": 0.030040474000998074,\n \"max\": 0.04595891800272511,\n \"mean\": 0.03633510730008614,\n \"stddev\": 0.0043045462862126484,\n \"rounds\": 30,\n \"median\": 0.036877674001516425,\n \"iqr\": 0.007704540999839082,\n \"q1\": 0.032137285998032894,\n \"q3\": 0.039841826997871976,\n \"iqr_outliers\": 0,\n \"stddev_outliers\": 10,\n \"outliers\": \"10;0\",\n \"ld15iqr\": 0.030040474000998074,\n \"hd15iqr\": 0.04595891800272511,\n \"ops\": 27.521592044331992,\n \"total\": 1.0900532190025842,\n \"data\": [\n 0.03704330600157846,\n 0.04039037099937559,\n 0.04017309099799604,\n 0.03581323899925337,\n 0.039841826997871976,\n 0.04328981499929796,\n 0.037152539000089746,\n 0.041258303997892654,\n 0.039796708999347175,\n 0.037076551001518965,\n 0.035965216000477085,\n 0.04216067000015755,\n 0.037044493001303636,\n 0.03418779300045571,\n 0.04595891800272511,\n 0.040525724998587975,\n 0.03705755299961311,\n 0.03956755600302131,\n 0.03671204200145439,\n 0.03355851100059226,\n 0.03365587200096343,\n 0.03159586599940667,\n 0.031034261999593582,\n 0.03230351200181758,\n 0.032137285998032894,\n 0.030491656998492545,\n 0.03206011000293074,\n 0.030754054998396896,\n 0.03140589599934174,\n 0.030040474000998074\n ],\n \"iterations\": 1\n }\n },\n {\n \"group\": \"xyzsearch\",\n \"name\": \"test1[6-8]\",\n \"fullname\": \"src/pypgstac/tests/test_benchmark.py::test1[6-8]\",\n \"params\": {\n \"zoom\": 6,\n \"item_width\": 8\n },\n \"param\": \"6-8\",\n \"extra_info\": {},\n \"options\": {\n \"disable_gc\": false,\n \"timer\": \"perf_counter\",\n \"min_rounds\": 3,\n \"max_time\": 1.0,\n \"min_time\": 5e-06,\n \"warmup\": 2\n },\n \"stats\": {\n \"min\": 0.029466998999851057,\n \"max\": 0.035118658997816965,\n \"mean\": 0.0317732569141039,\n \"stddev\": 0.0011116560823296769,\n \"rounds\": 35,\n \"median\": 0.03184639699975378,\n \"iqr\": 0.0011015400014002807,\n \"q1\": 0.03125985899987427,\n \"q3\": 0.03236139900127455,\n \"iqr_outliers\": 2,\n \"stddev_outliers\": 9,\n \"outliers\": \"9;2\",\n \"ld15iqr\": 0.02967003300000215,\n \"hd15iqr\": 0.035118658997816965,\n \"ops\": 31.473008974289563,\n \"total\": 1.1120639919936366,\n \"data\": [\n 0.03141539699936402,\n 0.03188201500233845,\n 0.032318950001354096,\n 0.03188201599914464,\n 0.03234269700260484,\n 0.03293398499954492,\n 0.030158020999806467,\n 0.032220401997619774,\n 0.03297316599855549,\n 0.03193544500027201,\n 0.03320944299775874,\n 0.029466998999851057,\n 0.03263121700001648,\n 0.03162436600177898,\n 0.029878999997890787,\n 0.03160299400042277,\n 0.03261221899811062,\n 0.035118658997816965,\n 0.03143676900072023,\n 0.031933069996739505,\n 0.030959464998886688,\n 0.030906036001397297,\n 0.033446908000769326,\n 0.030955902999266982,\n 0.03253385599964531,\n 0.03125273499972536,\n 0.031281231000320986,\n 0.03155075399990892,\n 0.03188320400295197,\n 0.031553127999359276,\n 0.031422522999491775,\n 0.03236763300083112,\n 0.02967003300000215,\n 0.03184639699975378,\n 0.03085735599961481\n ],\n \"iterations\": 1\n }\n },\n {\n \"group\": \"xyzsearch\",\n \"name\": \"test1[6-10]\",\n \"fullname\": \"src/pypgstac/tests/test_benchmark.py::test1[6-10]\",\n \"params\": {\n \"zoom\": 6,\n \"item_width\": 10\n },\n \"param\": \"6-10\",\n \"extra_info\": {},\n \"options\": {\n \"disable_gc\": false,\n \"timer\": \"perf_counter\",\n \"min_rounds\": 3,\n \"max_time\": 1.0,\n \"min_time\": 5e-06,\n \"warmup\": 2\n },\n \"stats\": {\n \"min\": 0.026975998996931594,\n \"max\": 0.04209300799993798,\n \"mean\": 0.03214526745681984,\n \"stddev\": 0.0029009541935060648,\n \"rounds\": 35,\n \"median\": 0.032535049998841714,\n \"iqr\": 0.003922621500350942,\n \"q1\": 0.029960930250126694,\n \"q3\": 0.033883551750477636,\n \"iqr_outliers\": 1,\n \"stddev_outliers\": 8,\n \"outliers\": \"8;1\",\n \"ld15iqr\": 0.026975998996931594,\n \"hd15iqr\": 0.04209300799993798,\n \"ops\": 31.10877834017969,\n \"total\": 1.1250843609886942,\n \"data\": [\n 0.029685470999538666,\n 0.03109126299750642,\n 0.028255932997126365,\n 0.026975998996931594,\n 0.029716340999584645,\n 0.02902888099924894,\n 0.030137841000396293,\n 0.030155651998938993,\n 0.029178483000578126,\n 0.03001792199938791,\n 0.029941933000372956,\n 0.029191544999775942,\n 0.028652499000600073,\n 0.030491664001601748,\n 0.030851424002321437,\n 0.03384704099880764,\n 0.03787088199896971,\n 0.032096925999212544,\n 0.034644923001906136,\n 0.032717895999667235,\n 0.03391234399896348,\n 0.03392421700118575,\n 0.033438600999943446,\n 0.0338957220010343,\n 0.03235932600000524,\n 0.03323913299755077,\n 0.03395390199875692,\n 0.034150997998949606,\n 0.0335490249999566,\n 0.033031350998498965,\n 0.034173557000030996,\n 0.04209300799993798,\n 0.032535049998841714,\n 0.03348016000018106,\n 0.032797448002384044\n ],\n \"iterations\": 1\n }\n },\n {\n \"group\": \"xyzsearch\",\n \"name\": \"test1[7-0.5]\",\n \"fullname\": \"src/pypgstac/tests/test_benchmark.py::test1[7-0.5]\",\n \"params\": {\n \"zoom\": 7,\n \"item_width\": 0.5\n },\n \"param\": \"7-0.5\",\n \"extra_info\": {},\n \"options\": {\n \"disable_gc\": false,\n \"timer\": \"perf_counter\",\n \"min_rounds\": 3,\n \"max_time\": 1.0,\n \"min_time\": 5e-06,\n \"warmup\": 2\n },\n \"stats\": {\n \"min\": 0.05863600000157021,\n \"max\": 0.06823670299854712,\n \"mean\": 0.06251938735273571,\n \"stddev\": 0.0022581559398568876,\n \"rounds\": 17,\n \"median\": 0.06236657000044943,\n \"iqr\": 0.002586880998023844,\n \"q1\": 0.06106378149979719,\n \"q3\": 0.06365066249782103,\n \"iqr_outliers\": 1,\n \"stddev_outliers\": 4,\n \"outliers\": \"4;1\",\n \"ld15iqr\": 0.05863600000157021,\n \"hd15iqr\": 0.06823670299854712,\n \"ops\": 15.995038376783489,\n \"total\": 1.062829584996507,\n \"data\": [\n 0.06236657000044943,\n 0.06587986099839327,\n 0.0637236809998285,\n 0.06111394300023676,\n 0.06390177900175331,\n 0.061628056999325054,\n 0.06823670299854712,\n 0.06362632299715187,\n 0.062382007999985944,\n 0.06094534600197221,\n 0.06093109799985541,\n 0.05984113399972557,\n 0.0634387259997311,\n 0.06276551399787422,\n 0.06230958200103487,\n 0.05863600000157021,\n 0.06110325999907218\n ],\n \"iterations\": 1\n }\n },\n {\n \"group\": \"xyzsearch\",\n \"name\": \"test1[7-0.75]\",\n \"fullname\": \"src/pypgstac/tests/test_benchmark.py::test1[7-0.75]\",\n \"params\": {\n \"zoom\": 7,\n \"item_width\": 0.75\n },\n \"param\": \"7-0.75\",\n \"extra_info\": {},\n \"options\": {\n \"disable_gc\": false,\n \"timer\": \"perf_counter\",\n \"min_rounds\": 3,\n \"max_time\": 1.0,\n \"min_time\": 5e-06,\n \"warmup\": 2\n },\n \"stats\": {\n \"min\": 0.04354155399778392,\n \"max\": 0.048738473997218534,\n \"mean\": 0.045375971043066114,\n \"stddev\": 0.0013899984103265528,\n \"rounds\": 23,\n \"median\": 0.04513969400068163,\n \"iqr\": 0.0020098417462577345,\n \"q1\": 0.04439405400171381,\n \"q3\": 0.046403895747971546,\n \"iqr_outliers\": 0,\n \"stddev_outliers\": 8,\n \"outliers\": \"8;0\",\n \"ld15iqr\": 0.04354155399778392,\n \"hd15iqr\": 0.048738473997218534,\n \"ops\": 22.038095869086856,\n \"total\": 1.0436473339905206,\n \"data\": [\n 0.044486663999123266,\n 0.04644456099777017,\n 0.04354155399778392,\n 0.046033746999455616,\n 0.04451278499982436,\n 0.04513969400068163,\n 0.04362229300022591,\n 0.048738473997218534,\n 0.04672833400036325,\n 0.04391912500068429,\n 0.04572267000185093,\n 0.04478943299909588,\n 0.044901042001583846,\n 0.044947346999833826,\n 0.04373271399890655,\n 0.04566924100072356,\n 0.046489682001265464,\n 0.04436318400257733,\n 0.04370896799809998,\n 0.04515987799823051,\n 0.04768887899990659,\n 0.04702516499673948,\n 0.04628189999857568\n ],\n \"iterations\": 1\n }\n },\n {\n \"group\": \"xyzsearch\",\n \"name\": \"test1[7-1]\",\n \"fullname\": \"src/pypgstac/tests/test_benchmark.py::test1[7-1]\",\n \"params\": {\n \"zoom\": 7,\n \"item_width\": 1\n },\n \"param\": \"7-1\",\n \"extra_info\": {},\n \"options\": {\n \"disable_gc\": false,\n \"timer\": \"perf_counter\",\n \"min_rounds\": 3,\n \"max_time\": 1.0,\n \"min_time\": 5e-06,\n \"warmup\": 2\n },\n \"stats\": {\n \"min\": 0.038548863001778955,\n \"max\": 0.044610154000110924,\n \"mean\": 0.04028780892568412,\n \"stddev\": 0.0012854161512075718,\n \"rounds\": 27,\n \"median\": 0.04029422899839119,\n \"iqr\": 0.0013024955014770967,\n \"q1\": 0.039389784499689995,\n \"q3\": 0.04069228000116709,\n \"iqr_outliers\": 1,\n \"stddev_outliers\": 6,\n \"outliers\": \"6;1\",\n \"ld15iqr\": 0.038548863001778955,\n \"hd15iqr\": 0.044610154000110924,\n \"ops\": 24.821404456236987,\n \"total\": 1.0877708409934712,\n \"data\": [\n 0.03955571199912811,\n 0.04021467699931236,\n 0.03939067399915075,\n 0.0399344689976715,\n 0.03902379099963582,\n 0.0392315739991318,\n 0.03859041900068405,\n 0.04195054699812317,\n 0.040376155000558356,\n 0.042304368998884456,\n 0.04080359200088424,\n 0.039497534999100026,\n 0.04029422899839119,\n 0.03935624400037341,\n 0.04052694500205689,\n 0.039895288999105105,\n 0.038548863001778955,\n 0.04146493200096302,\n 0.04038209099962842,\n 0.03889556200010702,\n 0.041216779998649145,\n 0.0405756250002014,\n 0.044610154000110924,\n 0.04053881900108536,\n 0.04073116500148899,\n 0.03938948799986974,\n 0.040471140997397015\n ],\n \"iterations\": 1\n }\n },\n {\n \"group\": \"xyzsearch\",\n \"name\": \"test1[7-1.5]\",\n \"fullname\": \"src/pypgstac/tests/test_benchmark.py::test1[7-1.5]\",\n \"params\": {\n \"zoom\": 7,\n \"item_width\": 1.5\n },\n \"param\": \"7-1.5\",\n \"extra_info\": {},\n \"options\": {\n \"disable_gc\": false,\n \"timer\": \"perf_counter\",\n \"min_rounds\": 3,\n \"max_time\": 1.0,\n \"min_time\": 5e-06,\n \"warmup\": 2\n },\n \"stats\": {\n \"min\": 0.03365115400083596,\n \"max\": 0.038333962998876814,\n \"mean\": 0.03571895903320789,\n \"stddev\": 0.0011727307440549206,\n \"rounds\": 30,\n \"median\": 0.03574915599892847,\n \"iqr\": 0.0013927309955761302,\n \"q1\": 0.03493702700143331,\n \"q3\": 0.03632975799700944,\n \"iqr_outliers\": 0,\n \"stddev_outliers\": 9,\n \"outliers\": \"9;0\",\n \"ld15iqr\": 0.03365115400083596,\n \"hd15iqr\": 0.038333962998876814,\n \"ops\": 27.996336597331986,\n \"total\": 1.0715687709962367,\n \"data\": [\n 0.03577052699984051,\n 0.03538939600184676,\n 0.03459745000145631,\n 0.03440985300039756,\n 0.03530628500084276,\n 0.03578596399893286,\n 0.03455233400018187,\n 0.03680350099966745,\n 0.03682012299759663,\n 0.03631788499842514,\n 0.03365115400083596,\n 0.03578833799838321,\n 0.034114211000996875,\n 0.038068002002546564,\n 0.03616472100111423,\n 0.035592429998359876,\n 0.03632975799700944,\n 0.03572778499801643,\n 0.03622289999839268,\n 0.03500470399740152,\n 0.03406196899959468,\n 0.03412727199975052,\n 0.03548913299891865,\n 0.037474339002073975,\n 0.03661590400224668,\n 0.038333962998876814,\n 0.03522910899846465,\n 0.03493702700143331,\n 0.035793087001366075,\n 0.03708964700126671\n ],\n \"iterations\": 1\n }\n },\n {\n \"group\": \"xyzsearch\",\n \"name\": \"test1[7-2]\",\n \"fullname\": \"src/pypgstac/tests/test_benchmark.py::test1[7-2]\",\n \"params\": {\n \"zoom\": 7,\n \"item_width\": 2\n },\n \"param\": \"7-2\",\n \"extra_info\": {},\n \"options\": {\n \"disable_gc\": false,\n \"timer\": \"perf_counter\",\n \"min_rounds\": 3,\n \"max_time\": 1.0,\n \"min_time\": 5e-06,\n \"warmup\": 2\n },\n \"stats\": {\n \"min\": 0.028906609000841854,\n \"max\": 0.035358532000827836,\n \"mean\": 0.0319112826878154,\n \"stddev\": 0.0015387649107082991,\n \"rounds\": 32,\n \"median\": 0.03188560699891241,\n \"iqr\": 0.0017311204992438434,\n \"q1\": 0.0308466970018344,\n \"q3\": 0.03257781750107824,\n \"iqr_outliers\": 1,\n \"stddev_outliers\": 9,\n \"outliers\": \"9;1\",\n \"ld15iqr\": 0.028906609000841854,\n \"hd15iqr\": 0.035358532000827836,\n \"ops\": 31.33687886453487,\n \"total\": 1.0211610460100928,\n \"data\": [\n 0.03288058399994043,\n 0.0333008970010269,\n 0.03068165800141287,\n 0.0319900909998978,\n 0.03119695799978217,\n 0.028906609000841854,\n 0.030351583001902327,\n 0.03415102299913997,\n 0.03243890100202407,\n 0.0317550020008639,\n 0.03226198900301824,\n 0.029107267000654247,\n 0.03183811399867409,\n 0.03240684199772659,\n 0.031771623998793075,\n 0.030382453001948306,\n 0.03186779699899489,\n 0.031715819000964984,\n 0.030463191997114336,\n 0.031903416998829925,\n 0.030586674001824576,\n 0.03101173600225593,\n 0.02950977000000421,\n 0.03208745200026897,\n 0.03271673400013242,\n 0.03142492500046501,\n 0.035358532000827836,\n 0.0322239939996507,\n 0.03410471700044582,\n 0.03201383800114854,\n 0.03500708399951691,\n 0.03374377000000095\n ],\n \"iterations\": 1\n }\n },\n {\n \"group\": \"xyzsearch\",\n \"name\": \"test1[7-3]\",\n \"fullname\": \"src/pypgstac/tests/test_benchmark.py::test1[7-3]\",\n \"params\": {\n \"zoom\": 7,\n \"item_width\": 3\n },\n \"param\": \"7-3\",\n \"extra_info\": {},\n \"options\": {\n \"disable_gc\": false,\n \"timer\": \"perf_counter\",\n \"min_rounds\": 3,\n \"max_time\": 1.0,\n \"min_time\": 5e-06,\n \"warmup\": 2\n },\n \"stats\": {\n \"min\": 0.029087087001244072,\n \"max\": 0.03676670400091098,\n \"mean\": 0.03163497772766277,\n \"stddev\": 0.0014185766524036627,\n \"rounds\": 33,\n \"median\": 0.03158284399978584,\n \"iqr\": 0.0012312565004322096,\n \"q1\": 0.030951482500313432,\n \"q3\": 0.03218273900074564,\n \"iqr_outliers\": 3,\n \"stddev_outliers\": 7,\n \"outliers\": \"7;3\",\n \"ld15iqr\": 0.0291345799996634,\n \"hd15iqr\": 0.03412371900049038,\n \"ops\": 31.610580181492075,\n \"total\": 1.0439542650128715,\n \"data\": [\n 0.032865152999875136,\n 0.031371498000225984,\n 0.03136912400077563,\n 0.032038774999819,\n 0.030763588001718745,\n 0.031986534002498956,\n 0.029087087001244072,\n 0.03213851100008469,\n 0.03412371900049038,\n 0.03289365000091493,\n 0.030415704000915866,\n 0.03158284399978584,\n 0.031763317001605174,\n 0.03144511400023475,\n 0.03193191800164641,\n 0.030003701001987793,\n 0.029894466999394353,\n 0.031568596001307014,\n 0.0324092209993978,\n 0.03182387099877815,\n 0.03147717200045008,\n 0.030071378998400178,\n 0.030258975999458926,\n 0.0291345799996634,\n 0.03154959799940116,\n 0.032324921998224454,\n 0.03129788599835592,\n 0.03676670400091098,\n 0.03182624500186648,\n 0.03243890500016278,\n 0.03200196900070296,\n 0.032315423002728494,\n 0.031014113999844994\n ],\n \"iterations\": 1\n }\n },\n {\n \"group\": \"xyzsearch\",\n \"name\": \"test1[7-4]\",\n \"fullname\": \"src/pypgstac/tests/test_benchmark.py::test1[7-4]\",\n \"params\": {\n \"zoom\": 7,\n \"item_width\": 4\n },\n \"param\": \"7-4\",\n \"extra_info\": {},\n \"options\": {\n \"disable_gc\": false,\n \"timer\": \"perf_counter\",\n \"min_rounds\": 3,\n \"max_time\": 1.0,\n \"min_time\": 5e-06,\n \"warmup\": 2\n },\n \"stats\": {\n \"min\": 0.027581562000705162,\n \"max\": 0.034331506001763046,\n \"mean\": 0.03041731352973062,\n \"stddev\": 0.001521611379613399,\n \"rounds\": 34,\n \"median\": 0.0303409059997648,\n \"iqr\": 0.0015363989987235982,\n \"q1\": 0.02964750799947069,\n \"q3\": 0.03118390699819429,\n \"iqr_outliers\": 1,\n \"stddev_outliers\": 12,\n \"outliers\": \"12;1\",\n \"ld15iqr\": 0.027581562000705162,\n \"hd15iqr\": 0.034331506001763046,\n \"ops\": 32.87601316344311,\n \"total\": 1.034188660010841,\n \"data\": [\n 0.03198772500036284,\n 0.0319449810012884,\n 0.031002244002593216,\n 0.030888261000654893,\n 0.02975436599808745,\n 0.030262541000411147,\n 0.0298434169999382,\n 0.03118390699819429,\n 0.030980874002125347,\n 0.03254339400155004,\n 0.028778386000340106,\n 0.028194221999001456,\n 0.027581562000705162,\n 0.02964750799947069,\n 0.028952922999451403,\n 0.029886160002206452,\n 0.030547499998647254,\n 0.03085501799796475,\n 0.03128482899774099,\n 0.029976397003338207,\n 0.030267292000644375,\n 0.03216226299991831,\n 0.0294052940007532,\n 0.031698018003226025,\n 0.03062823800064507,\n 0.027803591998235788,\n 0.028349762000289047,\n 0.0302103009998973,\n 0.028399630002240883,\n 0.034331506001763046,\n 0.030406208999920636,\n 0.033328215999063104,\n 0.030275602999608964,\n 0.030826521000562934\n ],\n \"iterations\": 1\n }\n },\n {\n \"group\": \"xyzsearch\",\n \"name\": \"test1[7-5]\",\n \"fullname\": \"src/pypgstac/tests/test_benchmark.py::test1[7-5]\",\n \"params\": {\n \"zoom\": 7,\n \"item_width\": 5\n },\n \"param\": \"7-5\",\n \"extra_info\": {},\n \"options\": {\n \"disable_gc\": false,\n \"timer\": \"perf_counter\",\n \"min_rounds\": 3,\n \"max_time\": 1.0,\n \"min_time\": 5e-06,\n \"warmup\": 2\n },\n \"stats\": {\n \"min\": 0.029624953000165988,\n \"max\": 0.03544047200193745,\n \"mean\": 0.03163227453126183,\n \"stddev\": 0.0013722909373689459,\n \"rounds\": 32,\n \"median\": 0.0315466385018226,\n \"iqr\": 0.0019151540018356172,\n \"q1\": 0.030518415498590912,\n \"q3\": 0.03243356950042653,\n \"iqr_outliers\": 1,\n \"stddev_outliers\": 10,\n \"outliers\": \"10;1\",\n \"ld15iqr\": 0.029624953000165988,\n \"hd15iqr\": 0.03544047200193745,\n \"ops\": 31.613281523960946,\n \"total\": 1.0122327850003785,\n \"data\": [\n 0.03236292300061905,\n 0.031078236999746878,\n 0.032562393997068284,\n 0.03205659299783292,\n 0.03353956299906713,\n 0.03250421600023401,\n 0.030679297000460792,\n 0.030446581997239264,\n 0.030127191999781644,\n 0.031597100001818035,\n 0.03289603400116903,\n 0.03085620999991079,\n 0.030072575002122903,\n 0.03132282899969141,\n 0.02995740400001523,\n 0.031661215998610714,\n 0.031255150999641046,\n 0.029624953000165988,\n 0.034652089001610875,\n 0.030336161002196604,\n 0.03097850300036953,\n 0.03012363000016194,\n 0.0320767800003523,\n 0.032229945001745364,\n 0.03059024899994256,\n 0.030080885997449514,\n 0.03328547599812737,\n 0.031496177001827164,\n 0.03174314100033371,\n 0.03544047200193745,\n 0.032525588998396415,\n 0.032073218000732595\n ],\n \"iterations\": 1\n }\n },\n {\n \"group\": \"xyzsearch\",\n \"name\": \"test1[7-6]\",\n \"fullname\": \"src/pypgstac/tests/test_benchmark.py::test1[7-6]\",\n \"params\": {\n \"zoom\": 7,\n \"item_width\": 6\n },\n \"param\": \"7-6\",\n \"extra_info\": {},\n \"options\": {\n \"disable_gc\": false,\n \"timer\": \"perf_counter\",\n \"min_rounds\": 3,\n \"max_time\": 1.0,\n \"min_time\": 5e-06,\n \"warmup\": 2\n },\n \"stats\": {\n \"min\": 0.028555175998917548,\n \"max\": 0.03251253300186363,\n \"mean\": 0.030253447555676556,\n \"stddev\": 0.0010558645538548761,\n \"rounds\": 36,\n \"median\": 0.03013194450068113,\n \"iqr\": 0.001688376998572494,\n \"q1\": 0.029296067001268966,\n \"q3\": 0.03098444399984146,\n \"iqr_outliers\": 0,\n \"stddev_outliers\": 14,\n \"outliers\": \"14;0\",\n \"ld15iqr\": 0.028555175998917548,\n \"hd15iqr\": 0.03251253300186363,\n \"ops\": 33.05408410594073,\n \"total\": 1.089124112004356,\n \"data\": [\n 0.030247113998484565,\n 0.030636556999525055,\n 0.029297254000994144,\n 0.029294880001543788,\n 0.02926163400115911,\n 0.03138338500139071,\n 0.029210578999482095,\n 0.029475352002918953,\n 0.030191310001100646,\n 0.02983748699989519,\n 0.030895395000698045,\n 0.02914765300010913,\n 0.030963072000304237,\n 0.03199960799975088,\n 0.02985529900252004,\n 0.029063352001685416,\n 0.03001083799972548,\n 0.031768078999448335,\n 0.030256614001700655,\n 0.029065726997941965,\n 0.032276254998578224,\n 0.030529698997270316,\n 0.030372971999895526,\n 0.028945805999683216,\n 0.02973894000024302,\n 0.02881163900019601,\n 0.03144750199862756,\n 0.03156029700039653,\n 0.028555175998917548,\n 0.02942786100174999,\n 0.030723233001481276,\n 0.030072579000261612,\n 0.03251253300186363,\n 0.031005815999378683,\n 0.029858859998057596,\n 0.03142375499737682\n ],\n \"iterations\": 1\n }\n },\n {\n \"group\": \"xyzsearch\",\n \"name\": \"test1[7-8]\",\n \"fullname\": \"src/pypgstac/tests/test_benchmark.py::test1[7-8]\",\n \"params\": {\n \"zoom\": 7,\n \"item_width\": 8\n },\n \"param\": \"7-8\",\n \"extra_info\": {},\n \"options\": {\n \"disable_gc\": false,\n \"timer\": \"perf_counter\",\n \"min_rounds\": 3,\n \"max_time\": 1.0,\n \"min_time\": 5e-06,\n \"warmup\": 2\n },\n \"stats\": {\n \"min\": 0.028266658999200445,\n \"max\": 0.03293759799998952,\n \"mean\": 0.030398328735078823,\n \"stddev\": 0.0010968732816062303,\n \"rounds\": 34,\n \"median\": 0.030405034001887543,\n \"iqr\": 0.0014794090020586737,\n \"q1\": 0.02978168599656783,\n \"q3\": 0.031261094998626504,\n \"iqr_outliers\": 0,\n \"stddev_outliers\": 11,\n \"outliers\": \"11;0\",\n \"ld15iqr\": 0.028266658999200445,\n \"hd15iqr\": 0.03293759799998952,\n \"ops\": 32.89654535665403,\n \"total\": 1.03354317699268,\n \"data\": [\n 0.03293759799998952,\n 0.031353705999208614,\n 0.028828263999457704,\n 0.028266658999200445,\n 0.02978168599656783,\n 0.030090391999692656,\n 0.029738943001575535,\n 0.02993010300269816,\n 0.030371787997864885,\n 0.03128009200008819,\n 0.03140357500160462,\n 0.030444216001342284,\n 0.030405034001887543,\n 0.031261094998626504,\n 0.030045274997974047,\n 0.03119460499874549,\n 0.03161017000093125,\n 0.03018062899718643,\n 0.029716384997300338,\n 0.02863117000015336,\n 0.02990754400161677,\n 0.030071395998675143,\n 0.031597108998539625,\n 0.030409783001232427,\n 0.029296071999851847,\n 0.030473897997580934,\n 0.028559929996845312,\n 0.030412158001126954,\n 0.031017694000183837,\n 0.030405034001887543,\n 0.03155436499946518,\n 0.028773649002687307,\n 0.032555280999076786,\n 0.031037879001814872\n ],\n \"iterations\": 1\n }\n },\n {\n \"group\": \"xyzsearch\",\n \"name\": \"test1[7-10]\",\n \"fullname\": \"src/pypgstac/tests/test_benchmark.py::test1[7-10]\",\n \"params\": {\n \"zoom\": 7,\n \"item_width\": 10\n },\n \"param\": \"7-10\",\n \"extra_info\": {},\n \"options\": {\n \"disable_gc\": false,\n \"timer\": \"perf_counter\",\n \"min_rounds\": 3,\n \"max_time\": 1.0,\n \"min_time\": 5e-06,\n \"warmup\": 2\n },\n \"stats\": {\n \"min\": 0.027302555998176103,\n \"max\": 0.03296134999982314,\n \"mean\": 0.029292249388870713,\n \"stddev\": 0.0012300904623758812,\n \"rounds\": 36,\n \"median\": 0.02918980949834804,\n \"iqr\": 0.0015280890020221705,\n \"q1\": 0.02840973649836087,\n \"q3\": 0.029937825500383042,\n \"iqr_outliers\": 1,\n \"stddev_outliers\": 12,\n \"outliers\": \"12;1\",\n \"ld15iqr\": 0.027302555998176103,\n \"hd15iqr\": 0.03296134999982314,\n \"ops\": 34.1387234119323,\n \"total\": 1.0545209779993456,\n \"data\": [\n 0.028638296000281116,\n 0.02874396799961687,\n 0.028501754000899382,\n 0.031631544003175804,\n 0.028944625999429263,\n 0.031008198002382414,\n 0.02980424899942591,\n 0.027741865000280086,\n 0.02796508199753589,\n 0.02837114799694973,\n 0.03085622000071453,\n 0.027866535001521697,\n 0.0292758909999975,\n 0.029145284999685828,\n 0.027721681999537395,\n 0.030627068001194857,\n 0.0274438470005407,\n 0.028330780001851963,\n 0.029078794999804813,\n 0.029919422002421925,\n 0.030410973999096313,\n 0.027302555998176103,\n 0.02914409799996065,\n 0.030698307000420755,\n 0.02995622899834416,\n 0.029785253998852568,\n 0.03296134999982314,\n 0.028354526999464724,\n 0.02957391000018106,\n 0.02928301499923691,\n 0.02968908000184456,\n 0.02860624000095413,\n 0.030064275000768248,\n 0.028448324999772012,\n 0.029392248998192372,\n 0.029234333997010253\n ],\n \"iterations\": 1\n }\n },\n {\n \"group\": \"xyzsearch\",\n \"name\": \"test1[8-0.5]\",\n \"fullname\": \"src/pypgstac/tests/test_benchmark.py::test1[8-0.5]\",\n \"params\": {\n \"zoom\": 8,\n \"item_width\": 0.5\n },\n \"param\": \"8-0.5\",\n \"extra_info\": {},\n \"options\": {\n \"disable_gc\": false,\n \"timer\": \"perf_counter\",\n \"min_rounds\": 3,\n \"max_time\": 1.0,\n \"min_time\": 5e-06,\n \"warmup\": 2\n },\n \"stats\": {\n \"min\": 0.03788044600150897,\n \"max\": 0.044045042002835544,\n \"mean\": 0.04064544235996436,\n \"stddev\": 0.0015116283612815955,\n \"rounds\": 25,\n \"median\": 0.04059704699830036,\n \"iqr\": 0.0015732069959994988,\n \"q1\": 0.039664106003328925,\n \"q3\": 0.041237312999328424,\n \"iqr_outliers\": 1,\n \"stddev_outliers\": 9,\n \"outliers\": \"9;1\",\n \"ld15iqr\": 0.03788044600150897,\n \"hd15iqr\": 0.044045042002835544,\n \"ops\": 24.603004468343464,\n \"total\": 1.016136058999109,\n \"data\": [\n 0.04063860200039926,\n 0.04158252599881962,\n 0.043160482000530465,\n 0.043389636997744674,\n 0.04110878299979959,\n 0.04084163500010618,\n 0.04137711899966234,\n 0.04247183400002541,\n 0.04102685999896494,\n 0.04059704699830036,\n 0.044045042002835544,\n 0.03788044600150897,\n 0.03829600999961258,\n 0.04054717999679269,\n 0.0391271369990136,\n 0.03996182800256065,\n 0.03905946099985158,\n 0.03968874300335301,\n 0.039131886998802656,\n 0.03985259399996721,\n 0.04119071099921712,\n 0.03959019500325667,\n 0.041112347000307636,\n 0.04018504600026063,\n 0.0402729069974157\n ],\n \"iterations\": 1\n }\n },\n {\n \"group\": \"xyzsearch\",\n \"name\": \"test1[8-0.75]\",\n \"fullname\": \"src/pypgstac/tests/test_benchmark.py::test1[8-0.75]\",\n \"params\": {\n \"zoom\": 8,\n \"item_width\": 0.75\n },\n \"param\": \"8-0.75\",\n \"extra_info\": {},\n \"options\": {\n \"disable_gc\": false,\n \"timer\": \"perf_counter\",\n \"min_rounds\": 3,\n \"max_time\": 1.0,\n \"min_time\": 5e-06,\n \"warmup\": 2\n },\n \"stats\": {\n \"min\": 0.03196281700002146,\n \"max\": 0.03790182000011555,\n \"mean\": 0.03527197655182982,\n \"stddev\": 0.0015415483599651623,\n \"rounds\": 29,\n \"median\": 0.03531463800027268,\n \"iqr\": 0.002420364248791884,\n \"q1\": 0.034211019249596575,\n \"q3\": 0.03663138349838846,\n \"iqr_outliers\": 0,\n \"stddev_outliers\": 9,\n \"outliers\": \"9;0\",\n \"ld15iqr\": 0.03196281700002146,\n \"hd15iqr\": 0.03790182000011555,\n \"ops\": 28.35111886997789,\n \"total\": 1.0228873200030648,\n \"data\": [\n 0.037126497001736425,\n 0.03454999899986433,\n 0.03531463800027268,\n 0.037422141002025455,\n 0.03790182000011555,\n 0.03617901100005838,\n 0.03618732199902297,\n 0.03487651400064351,\n 0.036152890002995264,\n 0.03387797300092643,\n 0.03531463900071685,\n 0.034897886998805916,\n 0.0366040749977401,\n 0.03727253900069627,\n 0.03284500099834986,\n 0.03511991800041869,\n 0.032935237999481615,\n 0.03481952400034061,\n 0.03531463900071685,\n 0.03671330900033354,\n 0.03501543299717014,\n 0.03671924599984777,\n 0.03426266799942823,\n 0.03196281700002146,\n 0.03557703899787157,\n 0.034056073000101605,\n 0.032747640001616674,\n 0.03402995199940051,\n 0.03709087800234556\n ],\n \"iterations\": 1\n }\n },\n {\n \"group\": \"xyzsearch\",\n \"name\": \"test1[8-1]\",\n \"fullname\": \"src/pypgstac/tests/test_benchmark.py::test1[8-1]\",\n \"params\": {\n \"zoom\": 8,\n \"item_width\": 1\n },\n \"param\": \"8-1\",\n \"extra_info\": {},\n \"options\": {\n \"disable_gc\": false,\n \"timer\": \"perf_counter\",\n \"min_rounds\": 3,\n \"max_time\": 1.0,\n \"min_time\": 5e-06,\n \"warmup\": 2\n },\n \"stats\": {\n \"min\": 0.0323878830022295,\n \"max\": 0.03815472799760755,\n \"mean\": 0.03524561979987387,\n \"stddev\": 0.0011380670339047108,\n \"rounds\": 30,\n \"median\": 0.035330077498656465,\n \"iqr\": 0.0012538160008261912,\n \"q1\": 0.03457849999904283,\n \"q3\": 0.035832315999869024,\n \"iqr_outliers\": 2,\n \"stddev_outliers\": 8,\n \"outliers\": \"8;2\",\n \"ld15iqr\": 0.03317626900025061,\n \"hd15iqr\": 0.03815472799760755,\n \"ops\": 28.3723198989844,\n \"total\": 1.057368593996216,\n \"data\": [\n 0.0357028979997267,\n 0.035832315999869024,\n 0.03457849999904283,\n 0.03444195699921693,\n 0.03695196500120801,\n 0.0339563410016126,\n 0.03526714899999206,\n 0.0323878830022295,\n 0.0353063309994468,\n 0.03535382399786613,\n 0.03384117100358708,\n 0.03504155900009209,\n 0.03548086899900227,\n 0.03815472799760755,\n 0.03578601199842524,\n 0.035990231997857336,\n 0.03428760700262501,\n 0.03587743599928217,\n 0.03542506499798037,\n 0.03317626900025061,\n 0.0349418229998264,\n 0.03529920800065156,\n 0.03653284000029089,\n 0.03491807700265781,\n 0.035834692000207724,\n 0.036848668998572975,\n 0.035555670998292044,\n 0.03547374499976286,\n 0.03494300999955158,\n 0.034180746999481926\n ],\n \"iterations\": 1\n }\n },\n {\n \"group\": \"xyzsearch\",\n \"name\": \"test1[8-1.5]\",\n \"fullname\": \"src/pypgstac/tests/test_benchmark.py::test1[8-1.5]\",\n \"params\": {\n \"zoom\": 8,\n \"item_width\": 1.5\n },\n \"param\": \"8-1.5\",\n \"extra_info\": {},\n \"options\": {\n \"disable_gc\": false,\n \"timer\": \"perf_counter\",\n \"min_rounds\": 3,\n \"max_time\": 1.0,\n \"min_time\": 5e-06,\n \"warmup\": 2\n },\n \"stats\": {\n \"min\": 0.030248326002038084,\n \"max\": 0.03439328300009947,\n \"mean\": 0.0319867942187102,\n \"stddev\": 0.0009411728636156087,\n \"rounds\": 32,\n \"median\": 0.031872588999249274,\n \"iqr\": 0.0011837645033665467,\n \"q1\": 0.03134007299922814,\n \"q3\": 0.03252383750259469,\n \"iqr_outliers\": 1,\n \"stddev_outliers\": 8,\n \"outliers\": \"8;1\",\n \"ld15iqr\": 0.030248326002038084,\n \"hd15iqr\": 0.03439328300009947,\n \"ops\": 31.26290159502964,\n \"total\": 1.0235774149987265,\n \"data\": [\n 0.03385304800031008,\n 0.03153063799982192,\n 0.03127417500218144,\n 0.032710840001527686,\n 0.033669012002064846,\n 0.031953325000358745,\n 0.03182746899983613,\n 0.03175504200044088,\n 0.032221661997027695,\n 0.031555571000353666,\n 0.03299698599948897,\n 0.032906748998357216,\n 0.03243419400314451,\n 0.031753855997521896,\n 0.03130860899909749,\n 0.03439328300009947,\n 0.032091056997160194,\n 0.032338019998860545,\n 0.032613481002044864,\n 0.03200913099863101,\n 0.03187615099886898,\n 0.030248326002038084,\n 0.030944098998588743,\n 0.032190791996981716,\n 0.031371536999358796,\n 0.031182752001768677,\n 0.03144158999930369,\n 0.0310485840018373,\n 0.03284263400200871,\n 0.030977344998973422,\n 0.03186902699962957,\n 0.030388430001039524\n ],\n \"iterations\": 1\n }\n },\n {\n \"group\": \"xyzsearch\",\n \"name\": \"test1[8-2]\",\n \"fullname\": \"src/pypgstac/tests/test_benchmark.py::test1[8-2]\",\n \"params\": {\n \"zoom\": 8,\n \"item_width\": 2\n },\n \"param\": \"8-2\",\n \"extra_info\": {},\n \"options\": {\n \"disable_gc\": false,\n \"timer\": \"perf_counter\",\n \"min_rounds\": 3,\n \"max_time\": 1.0,\n \"min_time\": 5e-06,\n \"warmup\": 2\n },\n \"stats\": {\n \"min\": 0.028950582000106806,\n \"max\": 0.03725355500137084,\n \"mean\": 0.031806940118151356,\n \"stddev\": 0.0014946978394731498,\n \"rounds\": 34,\n \"median\": 0.03164403200025845,\n \"iqr\": 0.002005395999731263,\n \"q1\": 0.030737507000594633,\n \"q3\": 0.032742903000325896,\n \"iqr_outliers\": 1,\n \"stddev_outliers\": 9,\n \"outliers\": \"9;1\",\n \"ld15iqr\": 0.028950582000106806,\n \"hd15iqr\": 0.03725355500137084,\n \"ops\": 31.43967939969577,\n \"total\": 1.081435964017146,\n \"data\": [\n 0.032860447998245945,\n 0.032930499997746665,\n 0.030135533001157455,\n 0.03163987600055407,\n 0.0333591240014357,\n 0.03725355500137084,\n 0.030737507000594633,\n 0.03192839600160369,\n 0.03176691900080186,\n 0.03258973600168247,\n 0.033474293999461224,\n 0.0323878910021449,\n 0.032742903000325896,\n 0.029632108002260793,\n 0.03152945599867962,\n 0.03024951699990197,\n 0.03317627700016601,\n 0.030543974000465823,\n 0.030230520002078265,\n 0.03336268700149958,\n 0.03216823700131499,\n 0.030519040999934077,\n 0.03144515500025591,\n 0.03164818799996283,\n 0.0310153430000355,\n 0.031651749999582535,\n 0.03140834799705772,\n 0.03270609600076568,\n 0.03134542000043439,\n 0.030598592002206715,\n 0.028950582000106806,\n 0.031205315000988776,\n 0.03287232200091239,\n 0.031370354001410306\n ],\n \"iterations\": 1\n }\n },\n {\n \"group\": \"xyzsearch\",\n \"name\": \"test1[8-3]\",\n \"fullname\": \"src/pypgstac/tests/test_benchmark.py::test1[8-3]\",\n \"params\": {\n \"zoom\": 8,\n \"item_width\": 3\n },\n \"param\": \"8-3\",\n \"extra_info\": {},\n \"options\": {\n \"disable_gc\": false,\n \"timer\": \"perf_counter\",\n \"min_rounds\": 3,\n \"max_time\": 1.0,\n \"min_time\": 5e-06,\n \"warmup\": 2\n },\n \"stats\": {\n \"min\": 0.029660606996912975,\n \"max\": 0.033832871999038616,\n \"mean\": 0.03176537699993898,\n \"stddev\": 0.0010999915228235328,\n \"rounds\": 33,\n \"median\": 0.03196758299964131,\n \"iqr\": 0.001908034250845958,\n \"q1\": 0.03072385799987387,\n \"q3\": 0.03263189225071983,\n \"iqr_outliers\": 0,\n \"stddev_outliers\": 11,\n \"outliers\": \"11;0\",\n \"ld15iqr\": 0.029660606996912975,\n \"hd15iqr\": 0.033832871999038616,\n \"ops\": 31.480816361849605,\n \"total\": 1.0482574409979861,\n \"data\": [\n 0.033832871999038616,\n 0.030280389997642487,\n 0.031687371996667935,\n 0.032386707000114257,\n 0.032153991000086535,\n 0.03178948200002196,\n 0.03289132200006861,\n 0.030627090000052704,\n 0.03196995599864749,\n 0.0311685109991231,\n 0.030679332001454895,\n 0.03199370299989823,\n 0.03062234100070782,\n 0.031891594000626355,\n 0.030738699999346863,\n 0.03196758299964131,\n 0.03349804699973902,\n 0.030088045001321007,\n 0.0327535930009617,\n 0.032258478000585455,\n 0.031011785002192482,\n 0.03239620800013654,\n 0.029660606996912975,\n 0.030378939998627175,\n 0.031088960997294635,\n 0.030111790998489596,\n 0.032640797002386535,\n 0.03272509700036608,\n 0.03262892400016426,\n 0.032580244002019754,\n 0.03271559900167631,\n 0.033615592001297045,\n 0.031423787000676384\n ],\n \"iterations\": 1\n }\n },\n {\n \"group\": \"xyzsearch\",\n \"name\": \"test1[8-4]\",\n \"fullname\": \"src/pypgstac/tests/test_benchmark.py::test1[8-4]\",\n \"params\": {\n \"zoom\": 8,\n \"item_width\": 4\n },\n \"param\": \"8-4\",\n \"extra_info\": {},\n \"options\": {\n \"disable_gc\": false,\n \"timer\": \"perf_counter\",\n \"min_rounds\": 3,\n \"max_time\": 1.0,\n \"min_time\": 5e-06,\n \"warmup\": 2\n },\n \"stats\": {\n \"min\": 0.027282394999929238,\n \"max\": 0.03426506200048607,\n \"mean\": 0.030401816441072323,\n \"stddev\": 0.001716866245843405,\n \"rounds\": 34,\n \"median\": 0.030336792500747833,\n \"iqr\": 0.0023212239975691773,\n \"q1\": 0.028907845000503585,\n \"q3\": 0.031229068998072762,\n \"iqr_outliers\": 0,\n \"stddev_outliers\": 12,\n \"outliers\": \"12;0\",\n \"ld15iqr\": 0.027282394999929238,\n \"hd15iqr\": 0.03426506200048607,\n \"ops\": 32.892771454570635,\n \"total\": 1.033661758996459,\n \"data\": [\n 0.030497674997604918,\n 0.029383961998973973,\n 0.029731849001109367,\n 0.03278327899897704,\n 0.03426506200048607,\n 0.030776695999520598,\n 0.03066746300100931,\n 0.028907845000503585,\n 0.028470907996961614,\n 0.030291079998278292,\n 0.031039095996675314,\n 0.03215993199773948,\n 0.029139373000361957,\n 0.03053567000097246,\n 0.03009160999863525,\n 0.031229068998072762,\n 0.0341736399968795,\n 0.02858489299978828,\n 0.030273272001068108,\n 0.027876059000846,\n 0.03098210600001039,\n 0.028721434999170015,\n 0.031764554001711076,\n 0.028535025001474423,\n 0.028791486998670734,\n 0.03335200999936205,\n 0.03073989100084873,\n 0.030266147001384525,\n 0.03132168100273702,\n 0.03256718700140482,\n 0.027282394999929238,\n 0.028671567000856157,\n 0.02940533600121853,\n 0.030382505003217375\n ],\n \"iterations\": 1\n }\n },\n {\n \"group\": \"xyzsearch\",\n \"name\": \"test1[8-5]\",\n \"fullname\": \"src/pypgstac/tests/test_benchmark.py::test1[8-5]\",\n \"params\": {\n \"zoom\": 8,\n \"item_width\": 5\n },\n \"param\": \"8-5\",\n \"extra_info\": {},\n \"options\": {\n \"disable_gc\": false,\n \"timer\": \"perf_counter\",\n \"min_rounds\": 3,\n \"max_time\": 1.0,\n \"min_time\": 5e-06,\n \"warmup\": 2\n },\n \"stats\": {\n \"min\": 0.027861814000061713,\n \"max\": 0.03240927499791724,\n \"mean\": 0.030260070090937945,\n \"stddev\": 0.0009621413396331114,\n \"rounds\": 33,\n \"median\": 0.030255463996581966,\n \"iqr\": 0.001138348749918805,\n \"q1\": 0.02963182200164738,\n \"q3\": 0.030770170751566184,\n \"iqr_outliers\": 1,\n \"stddev_outliers\": 8,\n \"outliers\": \"8;1\",\n \"ld15iqr\": 0.02877605499816127,\n \"hd15iqr\": 0.03240927499791724,\n \"ops\": 33.0468500897317,\n \"total\": 0.9985823130009521,\n \"data\": [\n 0.030935801998566603,\n 0.03057129299850203,\n 0.029934886002592975,\n 0.030205596001906088,\n 0.029239111998322187,\n 0.02877605499816127,\n 0.03240927499791724,\n 0.03159833000245271,\n 0.029677236001589336,\n 0.030676965001475764,\n 0.030527361999702407,\n 0.03076601500288234,\n 0.02989332899960573,\n 0.030255463996581966,\n 0.031093718000192894,\n 0.029296104999957606,\n 0.031021291000797646,\n 0.03225848599686287,\n 0.03181442499771947,\n 0.027861814000061713,\n 0.02947420399868861,\n 0.030294647000118857,\n 0.029697422000026563,\n 0.02994201100227656,\n 0.029326976000447758,\n 0.030294647000118857,\n 0.030476308002107544,\n 0.029499137999664526,\n 0.030782637997617712,\n 0.030115361001662677,\n 0.029486077000910882,\n 0.030704274999152403,\n 0.02967605000230833\n ],\n \"iterations\": 1\n }\n },\n {\n \"group\": \"xyzsearch\",\n \"name\": \"test1[8-6]\",\n \"fullname\": \"src/pypgstac/tests/test_benchmark.py::test1[8-6]\",\n \"params\": {\n \"zoom\": 8,\n \"item_width\": 6\n },\n \"param\": \"8-6\",\n \"extra_info\": {},\n \"options\": {\n \"disable_gc\": false,\n \"timer\": \"perf_counter\",\n \"min_rounds\": 3,\n \"max_time\": 1.0,\n \"min_time\": 5e-06,\n \"warmup\": 2\n },\n \"stats\": {\n \"min\": 0.028949408999324078,\n \"max\": 0.03259212699776981,\n \"mean\": 0.030504839750089257,\n \"stddev\": 0.0009478542205254766,\n \"rounds\": 36,\n \"median\": 0.03037479449994862,\n \"iqr\": 0.001157642998805386,\n \"q1\": 0.029859494999982417,\n \"q3\": 0.031017137998787803,\n \"iqr_outliers\": 0,\n \"stddev_outliers\": 12,\n \"outliers\": \"12;0\",\n \"ld15iqr\": 0.028949408999324078,\n \"hd15iqr\": 0.03259212699776981,\n \"ops\": 32.78168343752974,\n \"total\": 1.0981742310032132,\n \"data\": [\n 0.030885938002029434,\n 0.029247426999063464,\n 0.03185717199812643,\n 0.030361139000888215,\n 0.029714046002482064,\n 0.030447814999206457,\n 0.028966030000447063,\n 0.03104622699902393,\n 0.0324805180025578,\n 0.02979597200101125,\n 0.03259212699776981,\n 0.029552569998486433,\n 0.031183956998575013,\n 0.02986721200068132,\n 0.030602167000324698,\n 0.030076180999458302,\n 0.02997763299936196,\n 0.030777891999605345,\n 0.029904020000685705,\n 0.032488831002410734,\n 0.03183461400112719,\n 0.028949408999324078,\n 0.029995444001542637,\n 0.029851777999283513,\n 0.030988048998551676,\n 0.030612854003265966,\n 0.031489102002524305,\n 0.029530011997849215,\n 0.03022103599869297,\n 0.030209163000108674,\n 0.03128962999835494,\n 0.029380408999713836,\n 0.030675781999889296,\n 0.03028277700286708,\n 0.030388449999009026,\n 0.030650847998913378\n ],\n \"iterations\": 1\n }\n },\n {\n \"group\": \"xyzsearch\",\n \"name\": \"test1[8-8]\",\n \"fullname\": \"src/pypgstac/tests/test_benchmark.py::test1[8-8]\",\n \"params\": {\n \"zoom\": 8,\n \"item_width\": 8\n },\n \"param\": \"8-8\",\n \"extra_info\": {},\n \"options\": {\n \"disable_gc\": false,\n \"timer\": \"perf_counter\",\n \"min_rounds\": 3,\n \"max_time\": 1.0,\n \"min_time\": 5e-06,\n \"warmup\": 2\n },\n \"stats\": {\n \"min\": 0.027448631000879686,\n \"max\": 0.03276548099893262,\n \"mean\": 0.03027509761747805,\n \"stddev\": 0.001078552563011638,\n \"rounds\": 34,\n \"median\": 0.030215102500733337,\n \"iqr\": 0.0010638439998729154,\n \"q1\": 0.029684368000744144,\n \"q3\": 0.03074821200061706,\n \"iqr_outliers\": 3,\n \"stddev_outliers\": 9,\n \"outliers\": \"9;3\",\n \"ld15iqr\": 0.028456670999730704,\n \"hd15iqr\": 0.0326989909990516,\n \"ops\": 33.03044675973868,\n \"total\": 1.0293533189942536,\n \"data\": [\n 0.03276548099893262,\n 0.03144517299733707,\n 0.030481063997285673,\n 0.031204144997900585,\n 0.030190167999535333,\n 0.029652309000084642,\n 0.030148612000630237,\n 0.030088057999819284,\n 0.0326989909990516,\n 0.03090375200190465,\n 0.03074821200061706,\n 0.031194647002848797,\n 0.03150453899797867,\n 0.03056773799835355,\n 0.030211540000891546,\n 0.02932579400294344,\n 0.03017354699841235,\n 0.03028159399764263,\n 0.030238848998124013,\n 0.03188804700039327,\n 0.029565634999016766,\n 0.03021866500057513,\n 0.028456670999730704,\n 0.03047037899887073,\n 0.02943621700251242,\n 0.0299527040006069,\n 0.029684368000744144,\n 0.030378953997569624,\n 0.02917262999835657,\n 0.03024241200182587,\n 0.02867039000193472,\n 0.027448631000879686,\n 0.029856531000405084,\n 0.03008687200053828\n ],\n \"iterations\": 1\n }\n },\n {\n \"group\": \"xyzsearch\",\n \"name\": \"test1[8-10]\",\n \"fullname\": \"src/pypgstac/tests/test_benchmark.py::test1[8-10]\",\n \"params\": {\n \"zoom\": 8,\n \"item_width\": 10\n },\n \"param\": \"8-10\",\n \"extra_info\": {},\n \"options\": {\n \"disable_gc\": false,\n \"timer\": \"perf_counter\",\n \"min_rounds\": 3,\n \"max_time\": 1.0,\n \"min_time\": 5e-06,\n \"warmup\": 2\n },\n \"stats\": {\n \"min\": 0.027221853997616563,\n \"max\": 0.032586199002253124,\n \"mean\": 0.029672734942973227,\n \"stddev\": 0.0012590932030563811,\n \"rounds\": 35,\n \"median\": 0.02938160299891024,\n \"iqr\": 0.0017355754989694105,\n \"q1\": 0.028647240749705816,\n \"q3\": 0.030382816248675226,\n \"iqr_outliers\": 0,\n \"stddev_outliers\": 10,\n \"outliers\": \"10;0\",\n \"ld15iqr\": 0.027221853997616563,\n \"hd15iqr\": 0.032586199002253124,\n \"ops\": 33.700971680630644,\n \"total\": 1.038545723004063,\n \"data\": [\n 0.027221853997616563,\n 0.030416951998631703,\n 0.031562721997033805,\n 0.030825392001133878,\n 0.02917500699913944,\n 0.029343607002374483,\n 0.028673954999248963,\n 0.0297544229979394,\n 0.027784646998043172,\n 0.028380685002048267,\n 0.03194741600236739,\n 0.03218013200239511,\n 0.02952764300061972,\n 0.031147157002124004,\n 0.030203231999621494,\n 0.029350731001613894,\n 0.02928305300156353,\n 0.028575405998708447,\n 0.028595591000339482,\n 0.029159572997741634,\n 0.02995151999857626,\n 0.0286383359998581,\n 0.029634503000124823,\n 0.030280408998805797,\n 0.02938160299891024,\n 0.03072565800175653,\n 0.032586199002253124,\n 0.0293721039997763,\n 0.028415119002602296,\n 0.03014505399914924,\n 0.02973305200066534,\n 0.028434116000426002,\n 0.03099636800106964,\n 0.028245330999197904,\n 0.028897173000586918\n ],\n \"iterations\": 1\n }\n }\n ],\n \"datetime\": \"2025-01-07T03:09:36.802381+00:00\",\n \"version\": \"5.1.0\"\n}" }, { "path": "docs/src/item_size_analysis.ipynb", "content": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"id\": \"f7b04830-7b6a-4e37-a2a3-9961970e06df\",\n \"metadata\": {},\n \"source\": [\n \"# impacts of STAC item footprint size on dynamic tiling query performance\\n\",\n \"\\n\",\n \"**TL;DR:** If you have any control over the geographic footprint of the assets that you are cataloging with `pgstac` and you want to serve visualizations with a dynamic tiling application, try to maximize the size of the assets!\\n\",\n \"\\n\",\n \"Dynamic tiling applications like [`titiler-pgstac`](https://github.com/stac-utils/titiler-pgstac) send many queries to a `pgstac` database and clients are very sensitive to performance so it is worth considering a few basic ideas when building collections and items that may be used in this way.\\n\",\n \"\\n\",\n \"`pgstac`'s query functions perform relatively expensive spatial intersection operations so the fewer items there are in a collection x datetime partition, the faster the query will be. This is not a `pgstac`-specific problem (any application that needs to perform spatial intersections will take longer as the number of calculations increases), but it is worth demonstrating the influence of these factors in the dynamic tiling context.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"d34feaea-5288-4124-bca1-6bd4090fd27d\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"import json\\n\",\n \"import math\\n\",\n \"import uuid\\n\",\n \"from datetime import datetime, timezone\\n\",\n \"from typing import Any, Dict, Generator, Tuple\\n\",\n \"\\n\",\n \"import numpy as np\\n\",\n \"import pandas as pd\\n\",\n \"import seaborn as sns\\n\",\n \"from folium import Map, GeoJson, LayerControl\\n\",\n \"from matplotlib.colors import LogNorm\\n\",\n \"\\n\",\n \"\\n\",\n \"XMIN, YMIN = 0, 0\\n\",\n \"AOI_WIDTH = 50\\n\",\n \"AOI_HEIGHT = 50\\n\",\n \"ITEM_WIDTHS = [0.5, 1, 2, 4, 6, 8, 10]\\n\",\n \"\\n\",\n \"def generate_items(\\n\",\n \" item_size: Tuple[float, float],\\n\",\n \" collection_id: str,\\n\",\n \") -> Generator[Dict[str, Any], None, None]:\\n\",\n \" item_width, item_height = item_size\\n\",\n \"\\n\",\n \" cols = math.ceil(AOI_WIDTH / item_width)\\n\",\n \" rows = math.ceil(AOI_HEIGHT / item_height)\\n\",\n \"\\n\",\n \" # Generate items for each grid cell\\n\",\n \" for row in range(rows):\\n\",\n \" for col in range(cols):\\n\",\n \" left = XMIN + (col * item_width)\\n\",\n \" bottom = YMIN + (row * item_height)\\n\",\n \" right = left + item_width\\n\",\n \" top = bottom + item_height\\n\",\n \"\\n\",\n \" yield {\\n\",\n \" \\\"type\\\": \\\"Feature\\\",\\n\",\n \" \\\"stac_version\\\": \\\"1.0.0\\\",\\n\",\n \" \\\"id\\\": str(uuid.uuid4()),\\n\",\n \" \\\"collection\\\": collection_id,\\n\",\n \" \\\"geometry\\\": {\\n\",\n \" \\\"type\\\": \\\"Polygon\\\",\\n\",\n \" \\\"coordinates\\\": [\\n\",\n \" [\\n\",\n \" [left, bottom],\\n\",\n \" [right, bottom],\\n\",\n \" [right, top],\\n\",\n \" [left, top],\\n\",\n \" [left, bottom],\\n\",\n \" ],\\n\",\n \" ],\\n\",\n \" },\\n\",\n \" \\\"bbox\\\": [left, bottom, right, top],\\n\",\n \" \\\"properties\\\": {\\n\",\n \" \\\"datetime\\\": datetime.now(timezone.utc).isoformat(),\\n\",\n \" },\\n\",\n \" }\\n\",\n \"\\n\",\n \"\\n\",\n \"\\n\",\n \"def load_benchmark_results() -> pd.DataFrame:\\n\",\n \" \\\"\\\"\\\"Load benchmark results from JSON file into a pandas DataFrame.\\\"\\\"\\\"\\n\",\n \" with open(\\\"./benchmark.json\\\") as f:\\n\",\n \" data = json.load(f)\\n\",\n \"\\n\",\n \" # Extract the benchmarks into a list of records\\n\",\n \" records = []\\n\",\n \" for benchmark in data[\\\"benchmarks\\\"]:\\n\",\n \" record = {\\n\",\n \" \\\"item_width\\\": benchmark[\\\"params\\\"][\\\"item_width\\\"],\\n\",\n \" \\\"zoom\\\": benchmark[\\\"params\\\"][\\\"zoom\\\"],\\n\",\n \" \\\"mean\\\": benchmark[\\\"stats\\\"][\\\"mean\\\"],\\n\",\n \" \\\"stddev\\\": benchmark[\\\"stats\\\"][\\\"stddev\\\"],\\n\",\n \" \\\"median\\\": benchmark[\\\"stats\\\"][\\\"median\\\"],\\n\",\n \" }\\n\",\n \"\\n\",\n \" records.append(record)\\n\",\n \"\\n\",\n \" return pd.DataFrame(records).sort_values(by=[\\\"item_width\\\", \\\"zoom\\\"])\\n\",\n \"\\n\",\n \"\\n\",\n \"stac_items = {\\n\",\n \" item_width: list(\\n\",\n \" generate_items(\\n\",\n \" (item_width, item_width),\\n\",\n \" f\\\"{item_width} degrees\\\"\\n\",\n \" )\\n\",\n \" )\\n\",\n \" for item_width in ITEM_WIDTHS\\n\",\n \"}\\n\",\n \"\\n\",\n \"df = load_benchmark_results()\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"ceb365dc-67c1-4cbd-8b5e-7022a5773140\",\n \"metadata\": {},\n \"source\": [\n \"## Scenario\\n\",\n \"Imagine you have a continental-scale dataset of gridded data that will be stored as cloud-optimized geotiffs (COGs) and you get to decide how the individual files will be spatially arranged and cataloged in a `pgstac` database. You could make items as small as 0.5 degree squares or as large as 10 degree squares. In this case the assets will be non-overlapping rectangular grids.\\n\",\n \"\\n\",\n \"The assets will be publicly accessible, so smaller file sizes might be useful for some applications/users, but since the data will be stored as COGs and we also want to be able to serve raster tile visualizations in a web map with `titiler-pgstac`, smaller file sizes are not very important. However, the processing pipleline that generates the assets might have some resource constraints that push you to choose a smaller tile size.\\n\",\n \"\\n\",\n \"Consider the following options for tile sizes:\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"7ae1604f-df80-485b-a02a-4237f9ab0081\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"pd.DataFrame(\\n\",\n \" {\\\"tile width (degrees)\\\": item_width, \\\"# items\\\": len(items)}\\n\",\n \" for item_width, items in stac_items.items()\\n\",\n \")\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"fce001cb-ceb8-458e-9bda-e207d20362e7\",\n \"metadata\": {},\n \"source\": [\n \"The number of items is inversely proportional to the square of the tile width which means that small changes in tile size can have a large impact on the eventual number of items in your catalog!\\n\",\n \"\\n\",\n \"This map shows the spatial arrangement of the items for a range of tile sizes:\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"4c19eafb-125e-46ad-8b6f-e0053084287f\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"m = Map([25, 25], zoom_start=3)\\n\",\n \"for item_width in ITEM_WIDTHS:\\n\",\n \" layer_name = f\\\"{item_width} degrees\\\"\\n\",\n \" geojson = GeoJson(\\n\",\n \" {\\n\",\n \" \\\"type\\\": \\\"FeatureCollection\\\",\\n\",\n \" \\\"features\\\": stac_items[item_width],\\n\",\n \" },\\n\",\n \" name=layer_name,\\n\",\n \" overlay=True,\\n\",\n \" show=False,\\n\",\n \" )\\n\",\n \" geojson.add_to(m)\\n\",\n \" \\n\",\n \"LayerControl(collapsed=False, position=\\\"topright\\\").add_to(m)\\n\",\n \"\\n\",\n \"m\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"1c9714e7-c865-4b73-8305-83851864e486\",\n \"metadata\": {},\n \"source\": [\n \"## Performance comparison\\n\",\n \"To simulate the performance of queries made by a dynamic tiling application we have prepared a benchmarking procedure that uses the `pgstac` function `xyzsearch` to run an item query for an XYZ tile. By iterating over many combinations of tile sizes and zoom levels we can examine the response time with respect to item footprint size and tile zoom level. \\n\",\n \"\\n\",\n \"This figure shows average response time for `xyzsearch` to return a complete set of results for each zoom level for the range of item tile widths:\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"c61e8c44-df19-404d-8878-1efb5fddeb36\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"ax = sns.heatmap(\\n\",\n \" df.pivot(index=\\\"item_width\\\", columns=\\\"zoom\\\", values=\\\"median\\\"),\\n\",\n \" norm=LogNorm(vmin=1e-2, vmax=1e1),\\n\",\n \" cbar_kws={\\n\",\n \" \\\"ticks\\\": np.logspace(-2, 0, num=3),\\n\",\n \" \\\"format\\\": \\\"%.1e\\\",\\n\",\n \" }\\n\",\n \")\\n\",\n \"ax.set(xlabel=\\\"zoom level\\\", ylabel=\\\"item tile width\\\")\\n\",\n \"ax.xaxis.tick_top()\\n\",\n \"ax.xaxis.set_label_position(\\\"top\\\")\\n\",\n \"display(ax)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"cdb98e2e-e9da-4516-85f7-5576035b5915\",\n \"metadata\": {},\n \"source\": [\n \"Without details about the resource configuration for a specific `pgstac` deployment it is hard to say which zoom level becomes inoperable for a given tile size, but queries that take >0.5 seconds in this test would probably yield poor results in a deployed context.\"\n ]\n }\n ],\n \"metadata\": {\n \"kernelspec\": {\n \"display_name\": \"Python 3 (ipykernel)\",\n \"language\": \"python\",\n \"name\": \"python3\"\n },\n \"language_info\": {\n \"codemirror_mode\": {\n \"name\": \"ipython\",\n \"version\": 3\n },\n \"file_extension\": \".py\",\n \"mimetype\": \"text/x-python\",\n \"name\": \"python\",\n \"nbconvert_exporter\": \"python\",\n \"pygments_lexer\": \"ipython3\",\n \"version\": \"3.12.3\"\n }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 5\n}\n" }, { "path": "docs/src/pgstac.md", "content": "\nPGDatabase Schema and Functions for Storing and Accessing STAC collections and items in PostgreSQL\n\nSTAC Client that uses PgSTAC available in [STAC-FastAPI](https://github.com/stac-utils/stac-fastapi)\n\nPgSTAC requires **Postgresql>=13**, **PostGIS>=3** and **btree_gist**. Best performance will be had using PostGIS>=3.1.\n\n### PgSTAC Settings\nPgSTAC installs everything into the pgstac schema in the database. This schema must be in the search_path in the postgresql session while using pgstac.\n\n\n#### PgSTAC Users\nThe `pgstac_admin` role is the owner of all the objects within pgstac and should be used when running things such as migrations.\n\nThe `pgstac_ingest` role has read/write privileges on all tables and should be used for data ingest or if using the transactions extension with stac-fastapi-pgstac.\n\nThe `pgstac_read` role has read only access to the items and collections, but will still be able to write to the logging tables.\n\nYou can use the roles either directly and adding a password to them or by granting them to a role you are already using.\n\nTo use directly:\n```sql\nALTER ROLE pgstac_read LOGIN PASSWORD '';\n```\n\nTo grant pgstac permissions to a current postgresql user:\n```sql\nGRANT pgstac_read TO ;\n```\n\n#### PgSTAC Search Path\nThe search_path can be set at the database level or role level or by setting within the current session. The search_path is already set if you are directly using one of the pgstac users. If you are not logging in directly as one of the pgstac users, you will need to set the search_path by adding it to the search_path of the user you are using:\n```sql\nALTER ROLE SET SEARCH_PATH TO pgstac, public;\n```\nsetting the search_path on the database:\n```sql\nALTER DATABASE set search_path to pgstac, public;\n```\n\nIn psycopg the search_path can be set by passing it as a configuration when creating your connection:\n```python\nkwargs={\n \"options\": \"-c search_path=pgstac,public\"\n}\n```\n\n#### PgSTAC Settings Variables\nThere are additional variables that control the settings used for calculating and displaying context (total row count) for a search, as well as a variable to set the filter language (cql-json or cql2-json).\nThe context is \"off\" by default, and the default filter language is set to \"cql2-json\".\n\nVariables can be set either by passing them in via the connection options using your connection library, setting them in the pgstac_settings table or by setting them on the Role that is used to log in to the database.\n\nTurning \"context\" on can be **very** expensive on larger databases. Much of what PgSTAC does is to optimize the search of items sorted by time where only fewer than 10,000 records are returned at a time. It does this by searching for the data in chunks and is able to \"short circuit\" and return as soon as it has the number of records requested. Calculating the context (the total count for a query) requires a scan of all records that match the query parameters and can take a very long time. Setting \"context\" to auto will use database statistics to estimate the number of rows much more quickly, but for some queries, the estimate may be quite a bit off.\n\nExample for updating the pgstac_settings table with a new value:\n```sql\nINSERT INTO pgstac_settings (name, value)\nVALUES\n ('default-filter-lang', 'cql-json'),\n ('context', 'on')\n\nON CONFLICT ON CONSTRAINT pgstac_settings_pkey DO UPDATE SET value = excluded.value;\n```\n\nAlternatively, update the role:\n```sql\nALTER ROLE SET SEARCH_PATH to pgstac, public;\nALTER ROLE SET pgstac.context TO <'on','off','auto'>;\nALTER ROLE SET pgstac.context_estimated_count TO '';\nALTER ROLE SET pgstac.context_estimated_cost TO '';\nALTER ROLE SET pgstac.context_stats_ttl TO '>';\n```\n\nThe check_pgstac_settings function can be used to check what pgstac settings are being used and to check recommendations for system settings. It takes a single parameter which should be the amount of memory available on the database system.\n```sql\nSELECT check_pgstac_settings('16GB');\n```\n\n##### Read Only Mode\nThe pgstac.readonly setting can be used when using pgstac with a read replica.\nNote that when pgstac.readonly is set to TRUE that pgstac is unable to use a cache for calculating the total count for context which can make use of the context extension very expensive (see notes above). In readonly mode, pgstac is also unable to register the hash that is used to store queries that can be used with geometry_search (used by titiler-pgstac). A registered hash will still be readable, but new hashes cannot be created on the read only replica, they must be registered on the main database.\n\n#### Runtime Configurations\n\nRuntime configuration of variables can be made with search by passing in configuration in the search json \"conf\" item.\n\nRuntime configuration is available for **context**, **context_estimated_count**, **context_estimated_cost**, **context_stats_ttl**, and **nohydrate**.\n\nThe nohydrate conf item returns an unhydrated item bypassing the CPU intensive step of rehydrating data with data from the collection metadata. When using the nohydrate conf, the only fields that are respected in the fields extension are geometry and bbox.\n```sql\nSELECT search('{\"conf\":{\"nohydrate\"=true}}');\n```\n\n#### PgSTAC Partitioning\nBy default PgSTAC partitions data by collection (note: this is a change starting with version 0.5.0). Each collection can further be partitioned by either year or month. **Partitioning must be set up prior to loading any data!** Partitioning can be configured by setting the partition_trunc flag on a collection in the database.\n```sql\nUPDATE collections set partition_trunc='month' WHERE id='';\n```\n\nIn general, you should aim to keep each partition less than a few hundred thousand rows. Further partitioning (ie setting everything to 'month' when not needed to keep the partitions below a few hundred thousand rows) can be detrimental.\n\n#### PgSTAC Indexes / Queryables\n\nBy default, PgSTAC includes indexes on the id, datetime, collection, and geometry. Further indexing can be added for additional properties globally or only on particular collections by modifications to the queryables table.\n\nThe `queryables` table controls the indexes that PgSTAC will build as well as the metadata that is returned from a [STAC Queryables endpoint](https://github.com/stac-api-extensions/filter#queryables).\n\n| Column | Description | Type | Example |\n|-----------------------|--------------------------------------------------------------------------|------------|--------------------------------------------------------------------------------------------------------------------|\n| `id` | The id of the queryable | bigint[pk] | - |\n| `name` | The name of the property | text | `eo:cloud_cover` |\n| `collection_ids` | The collection ids that this queryable applies to | text[] | `{sentinel-2-l2a,landsat-c2-l2,aster-l1t}` or `NULL` |\n| `definition` | The queryable definition of the property | jsonb | `{\"title\": \"Cloud Cover\", \"type\": \"number\", \"minimum\": 0, \"maximum\": 100}` |\n| `property_wrapper` | The wrapper function to use to convert the property to a searchable type | text | One of `to_int`, `to_float`, `to_tstz`, `to_text` or `NULL` |\n| `property_index_type` | The index type to use for the property | text | `BTREE`, `NULL` or other valid [PostgreSQL index type](https://www.postgresql.org/docs/current/indexes-types.html) |\n\nEach record in the queryables table references a single property but can apply to any number of collections. If the `collection_ids` field is left as NULL, then that queryable will apply to all collections. There are constraints that allow only a single queryable record to be active per collection. If there is a queryable already set for a property field with collection_ids set to NULL, you will not be able to create a separate queryable entry that applies to that property with a specific collection as pgstac would not then be able to determine which queryable entry to use.\n\nBy default, any property may be used in filter expressions. If you wish to restrict it and only allow the queryables, you should either set the additional_properties setting variable to False or make the corresponding adjustment in the pgstac_settings table.\n\n##### Queryable Metadata\n\nWhen used with [stac-fastapi](https://stac-utils.github.io/stac-fastapi/), the metadata returned in the queryables endpoint is determined using the definition field on the `queryables` table. This is a jsonb field that will be returned as-is in the queryables response. The full queryable response for a collection will be determined by all the `queryables` records that have a match in `collection_ids` or have a NULL `collection_ids`.\n\nIf two or more collections in your catalog share a property name, but have different definitions (e.g., `platform` with different enum values), be sure to repeat the property for each collection id, each with a unique `definition`.\n\nThere is a utility SQL function that can be used to help populate the `queryables` table by looking at a sample of data for each collection. This utility can also look to the json schema for STAC extensions defined in the `stac_extensions` table.\n\nThe `stac_extensions` table contains a `url` field and a `content` field for each extension that should be introspected to compare for fields. This can either be filled in manually or by using the `pypgstac loadextensions` command included with pypgstac. This command will look at the `stac_extensions` attribute in all collections to populate the `stac_extensions` table, fetching the json content of each extension. If any urls were added manually to the stac_extensions table, it will also populate any records where the content is NULL.\n\nOnce the `stac_extensions` table has been filled in, you can run the `missing_queryables` function either for a single collection:\n\n```sql\nSELECT * FROM missing_queryables('mycollection', 5);\n```\n\nor for all collections:\n\n```sql\nSELECT * FROM missing_queryables(5);\n```\n\nThe numeric argument is the approximate percent of items that should be sampled to look for fields to include. This function will look for fields in the properties of items that do not already exist in the queryables table for each collection. It will then look to see if there is a field in any definition in the stac_extensions table to populate the definition for the queryable. If no definition was found, it will use the data type of the values for that field in the sample of items to fill in a generic definition with just the field type.\n\nIn order to populate the queryables table, you can then run the following query. Note we're casting the collection id to a text array:\n\n```sql\nINSERT INTO queryables (collection_ids, name, definition, property_wrapper)\n SELECT array[collection]::text[] as collection_ids, name, definition, property_wrapper\n FROM missing_queryables('mycollection', 5)\n```\n\nIf you run into conflicts due to the unique constraints on collection/name, you may need to create a temp table, make any changes to remove the conflicts, and then INSERT.\n\n```sql\nCREATE TEMP TABLE draft_queryables AS SELECT * FROM missing_queryables(5);\n```\n\nMake any edits to that table or the existing queryables, then:\n\n```sql\nINSERT INTO queryables (collection_ids, name, definition, property_wrapper) SELECT * FROM draft_queryables;\n```\n\n##### Indexing\n\nThe `queryables` table is also used to specify which item `properties` attributes to add indexes on.\n\nTo add a new global index across all collection partitions:\n\n```sql\nINSERT INTO pgstac.queryables (name, property_wrapper, property_index_type)\nVALUES (, , );\n```\n\nProperty wrapper should be one of `to_int`, `to_float`, `to_tstz`, or `to_text`. The index type should almost always be `BTREE`, but can be any PostgreSQL index type valid for the data type.\n\n**More indexes is not necessarily better.** You should only index the primary fields that are actively being used to search. Adding too many indexes can be very detrimental to performance and ingest speed. If your primary use case is delivering items sorted by datetime and you do not use the context extension, you likely will not need any further indexes.\n\nLeave `property_index_type` set to NULL if you do not want an index set for a property.\n\n### Maintenance Procedures\n\nThese are procedures that should be run periodically to make sure that statistics and constraints are kept up-to-date and validated. These can be made to run regularly using the pg_cron extension if available.\n```sql\nSELECT cron.schedule('0 * * * *', 'CALL validate_constraints();');\nSELECT cron.schedule('10, * * * *', 'CALL analyze_items();');\n```\n\n### System Checks\n\n#### System and pgSTAC Settings\nPgSTAC includes a function for checking common Postgres settings. You must pass in the amount of total memory available to get appropriate memory settings. Do note that these suggestions are merely a good first pass and figuring out the ideal parameters requires further analysis of a system and logs.\n```sql\nSELECT * FROM check_pgstac_settings('32GB');\n```\n\n#### Unused Indexes\nHaving too many indexes that do not get used can drastically hurt the performance of the database. The following query can be used to show indexes that are not getting used and should be considered for removal.\n```sql\nSELECT\n relname, indexrelname, idx_scan, last_idx_scan,idx_tup_read, idx_tup_fetch,\n pg_relation_size(relid) as index_bytes,\n pg_size_pretty(pg_relation_size(relid)) as index_size\nFROM pg_stat_user_indexes\nWHERE schemaname = 'pgstac'\nORDER BY idx_scan ASC\n;\n```\n\n### Backup and Restore (pg_dump / pg_restore)\n\nPgSTAC databases can be backed up and restored using standard PostgreSQL `pg_dump` and `pg_restore` tools. The custom format (`-Fc`) is recommended for flexibility and compression.\n\n#### Dumping\n\nSince PgSTAC installs everything into the `pgstac` schema, dump only that schema:\n\n```bash\npg_dump -Fc --schema=pgstac -f pgstac_backup.dump mydatabase\n```\n\nThis captures all tables, functions, views, indexes, and data within the `pgstac` schema.\n\n#### Restoring with `pgstac_restore`\n\n**Important:** `pg_restore` cannot be used directly with PgSTAC dumps. `pg_dump` sets `search_path` to empty during restore for safety, but PgSTAC SQL functions reference PostGIS functions (e.g. `st_makeenvelope`, `st_geomfromgeojson`) without schema qualification. Since PostGIS may be installed in either the `public` or `postgis` schema depending on the deployment, PgSTAC does not schema-qualify these calls. This means a raw `pg_restore` will fail when creating functions and tables that depend on PostGIS.\n\nPgSTAC provides the `pgstac_restore` script to handle this. It installs a temporary event trigger on the target database that sets the correct `search_path` (including the detected PostGIS schema) before each DDL command, then runs `pg_restore` directly:\n\n```bash\npgstac_restore -d target_database pgstac_backup.dump\n```\n\nThe script accepts several options:\n\n- `--create-extensions` — Create PostGIS, btree_gist, and unaccent on the target if they don't exist\n- `--no-roles` — Skip ownership and privilege restoration (use when the target doesn't have `pgstac_admin`/`pgstac_read`/`pgstac_ingest` roles)\n- `-d, --dbname` — Target database name (also reads `PGDATABASE`)\n\nCommon restore patterns:\n\n```bash\n# Restore into a fresh database, creating extensions, without requiring pgstac roles\npgstac_restore -d target_db --create-extensions --no-roles pgstac_backup.dump\n\n# Restore into a database that already has extensions and pgstac roles\npgstac_restore -d target_db pgstac_backup.dump\n```\n\nAfter restoring, ensure the `search_path` is set on the target database or role:\n\n```sql\nALTER DATABASE target_database SET search_path TO pgstac, public;\n```\n\n#### Notes\n\n- Always use `--schema=pgstac` on `pg_dump` to capture only the pgstac schema.\n- Do **not** use `pg_restore` directly — use `pgstac_restore` instead to handle the search_path issue.\n- After restoring, you may want to run `ANALYZE` on the restored database to update planner statistics.\n- Materialized views (`partitions`, `partition_steps`) are included in the dump. If you need to refresh them after restore, run `REFRESH MATERIALIZED VIEW pgstac.partitions; REFRESH MATERIALIZED VIEW pgstac.partition_steps;`.\n\n### Notification Triggers\n\nYou can add notification triggers alongside pgstac to get notified when items are inserted, updated, or deleted.\n\n**Important:** Do NOT create these in the `pgstac` schema as they could be removed in future migrations.\n\n**Note on pgSTAC's UPDATE behavior:** For performance and consistency, `update_item()` in pgSTAC uses a **DELETE+INSERT** pattern instead of a direct SQL `UPDATE`. As a result, standard `UPDATE` triggers will not fire during these operations. Applications that rely on change notifications should implement logic that accounts for this behavior.\n\n#### Example Notification Setup\n\nHere's an example of how to set up notification triggers for item changes:\n\n```sql\n-- Create the notification function\nCREATE OR REPLACE FUNCTION notify_items_change_func()\nRETURNS TRIGGER AS $$\nDECLARE\n\nBEGIN\n PERFORM pg_notify('pgstac_items_change'::text, json_build_object(\n 'operation', TG_OP,\n 'items', jsonb_agg(\n jsonb_build_object(\n 'collection', data.collection,\n 'id', data.id\n )\n )\n )::text\n )\n FROM data\n ;\n RETURN NULL;\nEND;\n$$ LANGUAGE plpgsql;\n\n-- Create triggers for INSERT operations\nCREATE OR REPLACE TRIGGER notify_items_change_insert\n AFTER INSERT ON pgstac.items\n REFERENCING NEW TABLE AS data\n FOR EACH STATEMENT EXECUTE FUNCTION notify_items_change_func()\n;\n\n-- Create triggers for UPDATE operations\nCREATE OR REPLACE TRIGGER notify_items_change_update\n AFTER UPDATE ON pgstac.items\n REFERENCING NEW TABLE AS data\n FOR EACH STATEMENT EXECUTE FUNCTION notify_items_change_func()\n;\n\n-- Create triggers for DELETE operations\nCREATE OR REPLACE TRIGGER notify_items_change_delete\n AFTER DELETE ON pgstac.items\n REFERENCING OLD TABLE AS data\n FOR EACH STATEMENT EXECUTE FUNCTION notify_items_change_func()\n;\n```\n\n#### Usage\n\nListen for notifications:\n```sql\nLISTEN pgstac_items_change;\n```\n\nPayload structure:\n\n```json\n{\n \"operation\": \"INSERT\",\n \"items\": [\n {\n \"collection\": \"sentinel-2-l2a\",\n \"id\": \"item-1\"\n },\n {\n \"collection\": \"sentinel-2-l2a\",\n \"id\": \"item-2\"\n }\n ]\n}\n```\n\n#### Customization\n\nYou may modify the function to include additional metadata, filter by collection, or use different channels. Only trigger on the `items` table, not `items_staging`.\n" }, { "path": "docs/src/pypgstac.md", "content": "\n\nPgSTAC includes a Python utility for bulk data loading and managing migrations.\n\npyPgSTAC is available on PyPI\n```\npython -m pip install pypgstac\n```\n\nBy default, pyPgSTAC does not install the `psycopg` dependency. If you want the database driver installed, use:\n\n```\npython -m pip install pypgstac[psycopg]\n```\n\nOr can be built locally\n```\ngit clone https://github.com/stac-utils/pgstac\ncd pgstac/pypgstac\npython -m pip install .\n```\n\n```\npypgstac --help\nUsage: pypgstac [OPTIONS] COMMAND [ARGS]...\n\nOptions:\n --install-completion Install completion for the current shell.\n --show-completion Show completion for the current shell, to copy it or\n customize the installation.\n\n --help Show this message and exit.\n\nCommands:\n initversion Get initial version.\n load Load STAC data into a pgstac database.\n migrate Migrate a pgstac database.\n pgready Wait for a pgstac database to accept connections.\n version Get version from a pgstac database.\n```\n\npyPgSTAC will get the database connection settings from the **standard PG environment variables**:\n\n- PGHOST=0.0.0.0\n- PGPORT=5432\n- PGUSER=username\n- PGDATABASE=postgis\n- PGPASSWORD=asupersecretpassword\n\nIt can also take a DSN database url \"postgresql://...\" via the **--dsn** flag.\n\n### Migrations\npyPgSTAC has a utility to help apply migrations to an existing PgSTAC instance to bring it up to date.\n\nThere are two types of migrations:\n\n - **Base migrations** install PgSTAC into a database with no current PgSTAC installation. These migrations follow the file pattern `\"pgstac.[version].sql\"`\n - **Incremental migrations** are used to move PgSTAC from one version to the next. These migrations follow the file pattern `\"pgstac.[version].[fromversion].sql\"`\n\nMigrations are stored in ```pypgstac/pypgstac/migrations``` and are distributed with the pyPgSTAC package.\n\n### Running Migrations\npyPgSTAC has a utility for checking the version of an existing PgSTAC database and applying the appropriate migrations in the correct order. It can also be used to setup a database from scratch.\n\nTo create an initial PgSTAC database or bring an existing one up to date, check you have the pypgstac version installed you want to migrate to and run:\n```\npypgstac migrate\n```\n\n### Bootstrapping an Empty Database\n\nWhen starting with an empty database, you have two options for initializing PgSTAC:\n\n#### Option 1: Execute as Power User\n\nThis approach uses a database user with administrative privileges (such as 'postgres') to run the migration, which will automatically create all necessary extensions and roles:\n\n```bash\n# Set environment variables for database connection\nexport PGHOST=localhost\nexport PGPORT=5432\nexport PGDATABASE=yourdatabase\nexport PGUSER=postgres # A user with admin privileges\nexport PGPASSWORD=yourpassword\n\n# Run the migration\npypgstac migrate\n```\n\nThe migration process will automatically:\n- Create required extensions (postgis, btree_gist, unaccent)\n- Create necessary roles (pgstac_admin, pgstac_read, pgstac_ingest)\n- Set up the pgstac schema and tables\n\nIn production environments, you should assign these roles to your application database user rather than continuing to use the postgres user:\n\n```sql\n-- Grant appropriate roles to your application user\nGRANT pgstac_read TO your_app_user;\nGRANT pgstac_ingest TO your_app_user;\nGRANT pgstac_admin TO your_app_user;\n\n-- Set the search path for your application user\nALTER USER your_app_user SET search_path TO pgstac, public;\n```\n\n#### Option 2: Create User with Initial Grants\n\nIf you don't have administrative privileges or prefer more control over the setup process, you can manually prepare the database before running migrations.\n\nConnect to your database as an administrator and execute:\n\n```sql\n\\c [database]\n\n-- Create required extensions\nCREATE EXTENSION IF NOT EXISTS postgis;\nCREATE EXTENSION IF NOT EXISTS btree_gist;\nCREATE EXTENSION IF NOT EXISTS unaccent;\n\n-- Create required roles\nCREATE ROLE pgstac_admin;\nCREATE ROLE pgstac_read;\nCREATE ROLE pgstac_ingest;\n\n-- Grant appropriate permissions\nALTER DATABASE [database] OWNER TO [user];\nALTER USER [user] SET search_path TO pgstac, public;\nALTER DATABASE [database] set search_path to pgstac, public;\nGRANT CONNECT ON DATABASE [database] TO [user];\nGRANT ALL PRIVILEGES ON TABLES TO [user];\nGRANT ALL PRIVILEGES ON SEQUENCES TO [user];\nGRANT pgstac_read TO [user] WITH ADMIN OPTION;\nGRANT pgstac_ingest TO [user] WITH ADMIN OPTION;\nGRANT pgstac_admin TO [user] WITH ADMIN OPTION;\n```\n\nThen run the migration as your non-admin user:\n\n```bash\n# Set environment variables for database connection\nexport PGHOST=localhost\nexport PGPORT=5432\nexport PGDATABASE=yourdatabase\nexport PGUSER=[user] # Your non-admin user\nexport PGPASSWORD=yourpassword\n\n# Run the migration\npypgstac migrate\n```\n\n### Verifying Migration\n\nTo verify that PgSTAC was installed correctly:\n\n```bash\n# Check the PgSTAC version\npypgstac version\n```\n\n### Bulk Data Loading\nA python utility is included which allows to load data from any source openable by smart-open using python in a memory efficient streaming manner using PostgreSQL copy. There are options for collections and items and can be used either as a command line or a library.\n\nTo load an ndjson of items directly using copy (will fail on any duplicate ids but is the fastest option to load new data you know will not conflict)\n```\npypgstac load items\n```\n\nTo load skipping any records that conflict with existing data\n```\npypgstac load items --method insert_ignore\n```\n\nTo upsert any records, adding anything new and replacing anything with the same id\n```\npypgstac load items --method upsert\n```\n\n### Loading Queryables\n\nQueryables are a mechanism that allows clients to discover what terms are available for use when writing filter expressions in a STAC API. The Filter Extension enables clients to filter collections and items based on their properties using the Common Query Language (CQL2).\n\nTo load queryables from a JSON file:\n\n```\npypgstac load_queryables queryables.json\n```\n\nTo load queryables for specific collections:\n\n```\npypgstac load_queryables queryables.json --collection_ids [collection1,collection2]\n```\n\nTo load queryables and delete properties not present in the file:\n\n```\npypgstac load_queryables queryables.json --delete_missing\n```\n\nTo load queryables and create indexes only for specific fields:\n\n```\npypgstac load_queryables queryables.json --index_fields [field1,field2]\n```\n\nBy default, no indexes are created when loading queryables. Using the `--index_fields` parameter allows you to selectively create indexes only for fields that require them. Creating too many indexes can degrade database performance, especially for write operations, so it's recommended to only index fields that are frequently used in queries.\n\nWhen using `--delete_missing` with specific collections, only properties for those collections will be deleted:\n\n```\npypgstac load_queryables queryables.json --collection_ids [collection1,collection2] --delete_missing\n```\n\nYou can combine all parameters as needed:\n\n```\npypgstac load_queryables queryables.json --collection_ids [collection1,collection2] --delete_missing --index_fields [field1,field2]\n```\n\nThe JSON file should follow the queryables schema as described in the [STAC API - Filter Extension](https://github.com/stac-api-extensions/filter#queryables). Here's an example:\n\n```json\n{\n \"$schema\": \"https://json-schema.org/draft/2019-09/schema\",\n \"$id\": \"https://example.com/stac/queryables\",\n \"type\": \"object\",\n \"title\": \"Queryables for Example STAC API\",\n \"description\": \"Queryable names for the Example STAC API\",\n \"properties\": {\n \"id\": {\n \"description\": \"Item identifier\",\n \"type\": \"string\"\n },\n \"datetime\": {\n \"description\": \"Datetime\",\n \"type\": \"string\",\n \"format\": \"date-time\"\n },\n \"eo:cloud_cover\": {\n \"description\": \"Cloud cover percentage\",\n \"type\": \"number\",\n \"minimum\": 0,\n \"maximum\": 100\n }\n },\n \"additionalProperties\": true\n}\n```\n\nThe command will extract the properties from the JSON file and create queryables in the database. It will also determine the appropriate property wrapper based on the type of each property and create the necessary indexes.\n\n### Automated Collection Extent Updates\n\nBy setting `pgstac.update_collection_extent` to `true`, a trigger is enabled to automatically adjust the spatial and temporal extents in collections when new items are ingested. This feature, while helpful, may increase overhead within data load transactions. To alleviate performance impact, combining this setting with `pgstac.use_queue` is beneficial. This approach necessitates a separate process, such as a scheduled task via the `pg_cron` extension, to periodically invoke `CALL run_queued_queries();`. Such asynchronous processing ensures efficient transactional performance and updated collection extents.\n\n*Note: The `pg_cron` extension must be properly installed and configured to manage the scheduling of the `run_queued_queries()` function.*\n" }, { "path": "scripts/cibuild", "content": "#!/bin/bash\nSCRIPT_DIR=$( cd -- \"$( dirname -- \"${BASH_SOURCE[0]}\" )\" &> /dev/null && pwd )\ncd $SCRIPT_DIR/..\nset -e\n\nif [[ \"${CI}\" ]]; then\n set -x\nfi\n\nfunction usage() {\n echo -n \\\n \"Usage: $(basename \"$0\")\nCI build for this project.\n\"\n}\n\nif [ \"${BASH_SOURCE[0]}\" = \"${0}\" ]; then\n scripts/setup\n scripts/test\nfi\n" }, { "path": "scripts/cipublish", "content": "#!/bin/bash\nSCRIPT_DIR=$( cd -- \"$( dirname -- \"${BASH_SOURCE[0]}\" )\" &> /dev/null && pwd )\ncd $SCRIPT_DIR/..\nset -e\n\nif [[ -n \"${CI}\" ]]; then\n set -x\nfi\n\nfunction usage() {\n echo -n \\\n \"Usage: $(basename \"$0\")\nPublish pypgstac\n\nOptions:\n--test Publish to test pypi\n\"\n}\n\nPOSITIONAL=()\nwhile [[ $# -gt 0 ]]\ndo\n key=\"$1\"\n case $key in\n\n --help)\n usage\n exit 0\n shift\n ;;\n\n --test)\n TEST_PYPI=\"--repository testpypi\"\n shift\n ;;\n\n *) # unknown option\n POSITIONAL+=(\"$1\") # save it in an array for later\n shift # past argument\n ;;\n esac\ndone\nset -- \"${POSITIONAL[@]}\" # restore positional parameters\n\n# Fail if this isn't CI and we aren't publishing to test pypi\nif [ -z \"${TEST_PYPI}\" ] && [ -z \"${CI}\" ]; then\n echo \"Only CI can publish to pypi\"\n exit 1\nfi\n\nif [ \"${BASH_SOURCE[0]}\" = \"${0}\" ]; then\n pushd pypgstac\n rm -rf dist\n pip install build twine\n python -m build --sdist --wheel\n twine upload ${TEST_PYPI} dist/*\n popd\nfi\n" }, { "path": "scripts/console", "content": "#!/bin/bash\nSCRIPT_DIR=$( cd -- \"$( dirname -- \"${BASH_SOURCE[0]}\" )\" &> /dev/null && pwd )\ncd $SCRIPT_DIR/..\nsource \"$SCRIPT_DIR/pgstacenv\"\nset -e\n\nif [[ \"${CI}\" ]]; then\n set -x\nfi\n\nfunction usage() {\n echo -n \\\n \"Usage: $(basename \"$0\") [--db] [--help]\nStart a console in the dev container\n\n--db: Instead, start a psql console in the database container.\n\"\n}\n\nwhile [[ \"$#\" > 0 ]]; do case $1 in\n --db)\n DB_CONSOLE=1\n shift\n ;;\n -h|--help)\n usage\n exit 0\n ;;\n *)\n echo \"Unknown parameter passed: $1\" >&2\n usage\n exit 1\n ;;\n esac; done\n\nif [ \"${BASH_SOURCE[0]}\" = \"${0}\" ]; then\n ensure_env_file\n\n docker compose up -d\n\n if [[ \"${DB_CONSOLE}\" ]]; then\n\n docker compose exec pgstac psql\n\n exit 0\n fi\n\n docker compose exec pypgstac /bin/bash\n\nfi\n" }, { "path": "scripts/container-scripts/format", "content": "#!/bin/bash\nset -e\n\nif [[ \"${CI}\" ]]; then\n set -x\nfi\nSCRIPT_DIR=$( cd -- \"$( dirname -- \"${BASH_SOURCE[0]}\" )\" &> /dev/null && pwd )\ncd ${PGSTAC_REPO_DIR:-/opt/src}\n\nfunction usage() {\n echo -n \\\n \"Usage: $(basename \"$0\")\nFormat code.\n\nThis script is meant to be run inside the dev container.\n\n\"\n}\n\nif [ \"${BASH_SOURCE[0]}\" = \"${0}\" ]; then\n echo \"Formatting pypgstac...\"\n ruff check --fix pypgstac/src/pypgstac pypgstac/tests\n ruff format pypgstac/src/pypgstac pypgstac/tests\nfi\n" }, { "path": "scripts/container-scripts/initpgstac", "content": "#!/bin/bash\nSCRIPT_DIR=$( cd -- \"$( dirname -- \"${BASH_SOURCE[0]}\" )\" &> /dev/null && pwd )\ncd ${PGSTAC_PGSTAC_DIR:-/opt/src/pgstac}\n\npsql -X -q -v ON_ERROR_STOP=1 < /dev/null && pwd )\ncd ${PGSTAC_PGSTAC_DIR:-/opt/src/pgstac}\npsql -f pgstac.sql\npsql -v ON_ERROR_STOP=1 <<-EOSQL\n \\copy collections (content) FROM 'tests/testdata/collections.ndjson'\n \\copy items_staging (content) FROM 'tests/testdata/items.ndjson'\nEOSQL\n" }, { "path": "scripts/container-scripts/makemigration", "content": "#!/bin/bash\nSCRIPT_DIR=$( cd -- \"$( dirname -- \"${BASH_SOURCE[0]}\" )\" &> /dev/null && pwd )\nSRCDIR=${PGSTAC_REPO_DIR:-/opt/src}\ncd $SRCDIR\n\nSHORT=f:,t:,o,d,h\nLONG=from:,to:,overwrite,debug,help\nOPTS=$(getopt --alternative --name $0 --options $SHORT --longoptions $LONG -- \"$@\")\n\neval set -- \"$OPTS\"\n\nwhile :\ndo\n case \"$1\" in\n -f | --from )\n FROM=\"$2\"\n shift 2\n ;;\n -t | --to )\n TO=\"$2\"\n shift 2\n ;;\n -o | --overwrite )\n OVERWRITE=1\n shift 1\n ;;\n -d | --debug )\n DEBUG=1\n shift 1\n ;;\n -h | --help)\n cat <&2\n exit 1\nfi\n\n\n\nBASEDIR=$SRCDIR\nPYPGSTACDIR=$BASEDIR/pypgstac\nMIGRATIONSDIR=$BASEDIR/pgstac/migrations\nSQLDIR=$BASEDIR/pgstac/sql\n\n# Check if from SQL file exists\nFROMSQL=$MIGRATIONSDIR/pgstac.$FROM.sql\nif [ -f $FROMSQL ]; then\n echo \"Migrating From: $FROMSQL\"\nelse\n echo \"From SQL $FROMSQL does not exist\"\n exit 1\nfi\n\n# Check if to SQL file exists\nTOSQL=$MIGRATIONSDIR/pgstac.$TO.sql\nif [ -f $TOSQL ]; then\n echo \"Migrating To: $TOSQL\"\nelse\n echo \"To SQL $TOSQL does not exist\"\n exit 1\nfi\n\nMIGRATIONSQL=$MIGRATIONSDIR/pgstac.$FROM-$TO.sql\nif [[ -f \"$MIGRATIONSQL\" ]]; then\n if [[ \"$OVERWRITE\" != 1 ]]; then\n echo \"ERROR: $MIGRATIONSQL already exists. Use --overwrite to replace.\" >&2\n exit 1\n else\n echo \"Removing existing $MIGRATIONSQL\"\n rm $MIGRATIONSQL\n fi\nelse\n echo \"Creating $MIGRATIONSQL\"\nfi\n\npg_isready -t 10\n# Create Databases to inspect to create migration\npsql -q >/dev/null 2>&1 <<-'EOSQL'\n DROP DATABASE IF EXISTS migra_from;\n CREATE DATABASE migra_from;\n DROP DATABASE IF EXISTS migra_to;\n CREATE DATABASE migra_to;\nEOSQL\n\nTODBURL=\"postgresql://${PGUSER}:${PGPASSWORD}@${PGHOST:-localhost}:${PGPORT:-5432}/migra_to\"\nFROMDBURL=\"postgresql://${PGUSER}:${PGPASSWORD}@${PGHOST:-localhost}:${PGPORT:-5432}/migra_from\"\n\n# Make sure to clean up migra databases\nfunction drop_migra_dbs(){\npsql -q >/dev/null 2>&1 <<-'EOSQL'\n DROP DATABASE IF EXISTS migra_from;\n DROP DATABASE IF EXISTS migra_to;\nEOSQL\n}\n\ntrap drop_migra_dbs 0 2 3 15\n\necho \"Creating Migration from $FROM to $TO\"\n\n# Install From into Database\npsql -q -X -1 -v ON_ERROR_STOP=1 -v CLIENT_MIN_MESSAGES=WARNING -f $FROMSQL $FROMDBURL >/dev/null || exit 1;\n\n# Install To into Database\npsql -q -X -1 -v ON_ERROR_STOP=1 -v CLIENT_MIN_MESSAGES=WARNING -f $TOSQL $TODBURL >/dev/null || exit 1;\n\n\n# Calculate the migration\nMIGRATION=$(mktemp)\ntrap \"rm $MIGRATION\" 0 2 3 15\n\nmigra --schema pgstac --unsafe $FROMDBURL $TODBURL >$MIGRATION\nif [[ $DEBUG == 1 ]]; then\n echo \"*************\"\n cat $MIGRATION\n echo \"*************\"\nfi\n\n# Append wrapper around created migration with idempotent and transaction statements\n\necho \"SET client_min_messages TO WARNING;\" >$MIGRATIONSQL\necho \"SET SEARCH_PATH to pgstac, public;\" >>$MIGRATIONSQL\ncat $SQLDIR/000_idempotent_pre.sql >>$MIGRATIONSQL\necho \"-- BEGIN migra calculated SQL\" >>$MIGRATIONSQL\ncat $MIGRATION >>$MIGRATIONSQL\necho \"-- END migra calculated SQL\" >>$MIGRATIONSQL\ncat $SQLDIR/998_idempotent_post.sql >>$MIGRATIONSQL\necho \"SELECT set_version('${TO}');\" >>$MIGRATIONSQL\n\necho \"Migration created at $MIGRATIONSQL.\"\nexit 0\n" }, { "path": "scripts/container-scripts/pgstac_restore", "content": "#!/bin/bash\nset -e\n\nfunction usage() {\n cat <\n\nRestore a PgSTAC pg_dump (-Fc) backup into a target database.\n\npg_dump clears the search_path during restore, which breaks PgSTAC\nfunctions that reference PostGIS functions without schema qualification.\nThis script works around that by installing a temporary event trigger\non the target database that resets search_path before every DDL command,\nthen running pg_restore directly (preserving its dependency ordering).\n\nPrerequisites on the target database:\n - Extensions: postgis, btree_gist, unaccent\n - Roles (optional): pgstac_admin, pgstac_read, pgstac_ingest\n (use --no-roles if roles don't exist; implies --no-owner --no-privileges)\n\nOptions:\n -d, --dbname NAME Target database (default: \\$PGDATABASE)\n -f, --file FILE Dump file path (also accepted as positional arg)\n --no-roles Skip role-dependent options (adds --no-owner --no-privileges)\n --create-extensions Create required extensions on target if missing\n -h, --help Show this help message\n\nEnvironment:\n Standard PG variables (PGHOST, PGPORT, PGUSER, PGPASSWORD, PGDATABASE)\n are respected for both psql and pg_restore.\n\nExamples:\n # Dump\n pg_dump -Fc --schema=pgstac -f pgstac_backup.dump mydb\n\n # Restore into an existing database with extensions already installed\n $(basename \"$0\") -d target_db pgstac_backup.dump\n\n # Restore, creating extensions and skipping ownership\n $(basename \"$0\") -d target_db --create-extensions --no-roles pgstac_backup.dump\nEOF\n exit \"${1:-0}\"\n}\n\nDBNAME=\"${PGDATABASE:-}\"\nDUMPFILE=\"\"\nNO_ROLES=0\nCREATE_EXTENSIONS=0\n\nwhile [[ $# -gt 0 ]]; do\n case \"$1\" in\n -d|--dbname)\n DBNAME=\"$2\"\n shift 2\n ;;\n -f|--file)\n DUMPFILE=\"$2\"\n shift 2\n ;;\n --no-roles)\n NO_ROLES=1\n shift\n ;;\n --create-extensions)\n CREATE_EXTENSIONS=1\n shift\n ;;\n -h|--help)\n usage 0\n ;;\n -*)\n echo \"Unknown option: $1\" >&2\n usage 1\n ;;\n *)\n if [ -z \"$DUMPFILE\" ]; then\n DUMPFILE=\"$1\"\n else\n echo \"Unexpected argument: $1\" >&2\n usage 1\n fi\n shift\n ;;\n esac\ndone\n\nif [ -z \"$DUMPFILE\" ]; then\n echo \"Error: dump file is required\" >&2\n usage 1\nfi\n\nif [ ! -f \"$DUMPFILE\" ]; then\n echo \"Error: dump file not found: $DUMPFILE\" >&2\n exit 1\nfi\n\nif [ -z \"$DBNAME\" ]; then\n echo \"Error: target database is required (-d or PGDATABASE)\" >&2\n usage 1\nfi\n\n# Create extensions if requested\nif [ \"$CREATE_EXTENSIONS\" -eq 1 ]; then\n echo \"Creating extensions on ${DBNAME} ...\"\n psql -X -q -v ON_ERROR_STOP=1 -d \"$DBNAME\" </dev/null || true)\n\nif [ -z \"$POSTGIS_SCHEMA\" ]; then\n echo \"Error: PostGIS extension not found on target database '${DBNAME}'.\" >&2\n echo \"Install PostGIS first or use --create-extensions.\" >&2\n exit 1\nfi\n\n# Build the search_path: pgstac + PostGIS schema + public (deduplicated)\nSEARCH_PATH=\"pgstac, ${POSTGIS_SCHEMA}\"\nif [ \"$POSTGIS_SCHEMA\" != \"public\" ]; then\n SEARCH_PATH=\"${SEARCH_PATH}, public\"\nfi\necho \"Using search_path: ${SEARCH_PATH}\"\n\n# Build pg_restore flags\nRESTORE_FLAGS=()\nif [ \"$NO_ROLES\" -eq 1 ]; then\n RESTORE_FLAGS+=(--no-owner --no-privileges)\nfi\n\necho \"Restoring ${DUMPFILE} into ${DBNAME} ...\"\n\n# pg_dump clears search_path during restore, which breaks PgSTAC functions\n# that reference PostGIS without schema qualification. We install a temporary\n# event trigger that resets search_path before every DDL command, then run\n# pg_restore directly.\n\n# Install temporary event trigger to fix search_path\npsql -X -v ON_ERROR_STOP=1 -d \"$DBNAME\" <&2\nfi\n\necho \"Restore complete.\"\n" }, { "path": "scripts/container-scripts/resetpgstac", "content": "#!/bin/bash\nSCRIPT_DIR=$( cd -- \"$( dirname -- \"${BASH_SOURCE[0]}\" )\" &> /dev/null && pwd )\ncd ${PGSTAC_PGSTAC_DIR:-/opt/src/pgstac}\nset -e\npsql -v ON_ERROR_STOP=1 <<-EOSQL\n DROP SCHEMA IF EXISTS pgstac CASCADE;\n \\i pgstac.sql\n SET SEARCH_PATH TO pgstac, public;\n \\copy collections (content) FROM 'tests/testdata/collections.ndjson'\n \\copy items_staging (content) FROM 'tests/testdata/items.ndjson'\nEOSQL\n" }, { "path": "scripts/container-scripts/stageversion", "content": "#!/bin/bash\nSCRIPT_DIR=$( cd -- \"$( dirname -- \"${BASH_SOURCE[0]}\" )\" &> /dev/null && pwd )\nSRCDIR=${PGSTAC_REPO_DIR:-/opt/src}\ncd $SRCDIR\nBASEDIR=$SRCDIR\nSQLDIR=$BASEDIR/pgstac/sql\nPYPGSTACDIR=$BASEDIR/pypgstac\nMIGRATIONSDIR=$BASEDIR/pgstac/migrations\n\nfunction usage() {\n cat <999_version.sql\ncat *.sql >$MIGRATIONSDIR/pgstac.${VERSION}.sql\ncd $BASEDIR/pgstac\n\n# make the base pgstac.sql a symbolic link to the most recent version\nrm pgstac.sql\ncp migrations/pgstac.${VERSION}.sql pgstac.sql\n\n# Update the version number in the appropriate places\n[[ $VERSION == 'unreleased' ]] && PYVERSION=\"${OLDVERSION}-dev\" || PYVERSION=\"$VERSION\"\necho \"Setting pypgstac version to $PYVERSION\"\ncat < $PYPGSTACDIR/src/pypgstac/version.py\n\"\"\"Version.\"\"\"\n__version__ = \"${PYVERSION}\"\nEOD\nsed -i \"s/^version[ ]*=[ ]*.*$/version = \\\"${PYVERSION}\\\"/\" $PYPGSTACDIR/pyproject.toml\n\nmakemigration -f $OLDVERSION -t $VERSION\n" }, { "path": "scripts/container-scripts/test", "content": "#!/bin/bash\nset -e\nSCRIPT_DIR=$( cd -- \"$( dirname -- \"${BASH_SOURCE[0]}\" )\" &> /dev/null && pwd )\nREPO_DIR=${PGSTAC_REPO_DIR:-/opt/src}\nexport PATH=\"$SCRIPT_DIR:$PATH\"\nif [[ -d \"$REPO_DIR/pgstac\" && -d \"$REPO_DIR/pypgstac\" ]]; then\n export SRCDIR=\"$REPO_DIR\"\nelif [[ -d \"$REPO_DIR/src/pgstac\" && -d \"$REPO_DIR/src/pypgstac\" ]]; then\n export SRCDIR=\"$REPO_DIR/src\"\nelif [[ -d \"$SCRIPT_DIR/../../src/pgstac\" && -d \"$SCRIPT_DIR/../../src/pypgstac\" ]]; then\n export SRCDIR=\"$(cd \"$SCRIPT_DIR/../../src\" && pwd)\"\nelse\n echo \"Unable to find pgstac/pypgstac sources under '$REPO_DIR'.\" >&2\n echo \"Set PGSTAC_REPO_DIR to either a directory containing pgstac/ and pypgstac/ or a repository root containing src/pgstac and src/pypgstac.\" >&2\n exit 1\nfi\nexport PGSTACDIR=$SRCDIR/pgstac\n\nif [[ \"${CI}\" ]]; then\n set -x\nfi\n\nfunction usage() {\n cat </dev/null 2>&1 || true\n psql -X -q -d postgres -c \"ALTER DATABASE postgis REFRESH COLLATION VERSION;\" >/dev/null 2>&1 || true\n}\n\nfunction test_formatting(){\n cd $SRCDIR/pypgstac\n\n echo \"Running ruff\"\n ruff check src/pypgstac tests\n ruff format --check src/pypgstac tests\n\n echo \"Running ty\"\n ty check\n\n echo \"Checking if there are any staged migrations.\"\n find $SRCDIR/pgstac/migrations | grep 'staged' && { echo \"There are staged migrations in pypgstac/migrations. Please check migrations and remove staged suffix.\"; exit 1; }\n\n\n VERSION=$(python -c \"from pypgstac.version import __version__; print(__version__)\")\n echo $VERSION\n if echo $VERSION | grep \"dev\"; then\n VERSION=\"unreleased\"\n fi\n\n echo \"Checking whether base sql migration exists for pypgstac version.\"\n [ -f $SRCDIR/pgstac/migrations/pgstac.\"${VERSION}\".sql ] || { echo \"****FAIL No Migration exists pypgstac/migrations/pgstac.${VERSION}.sql\"; exit 1; }\n\n echo \"Congratulations! All formatting tests pass.\"\n}\n\nfunction test_pgtap(){\ncd $PGSTACDIR\nTEMPLATEDB=${1:-pgstac_test_db_template}\npsql -X -q -v ON_ERROR_STOP=1 <\"$TMPFILE\"\n\n diff -Z -b -w -B --strip-trailing-cr --suppress-blank-empty -C 1 \"$TMPFILE\" $SQLOUTFILE && echo \"TEST $SQLFILE PASSED\" || { echo \"***TEST FOR $SQLFILE FAILED***\"; exit 1; }\n\ndone\npsql -X -q -c \"DROP DATABASE IF EXISTS pgstac_test_basicsql WITH (force);\";\n}\n\nfunction test_pypgstac(){\n[[ $MESSAGELOG == 1 ]] && VERBOSE=\"-vvv\"\nTEMPLATEDB=${1:-pgstac_test_db_template}\n cd $SRCDIR/pypgstac\n python -m venv venv\n source venv/bin/activate\n pip install --cache /tmp/.pipcache --upgrade pip\n pip install --cache /tmp/.pipcache -e . --no-deps\n psql -X -q -v ON_ERROR_STOP=1 </dev/null || stat -f%z \"$DUMPFILE\" 2>/dev/null)\n echo \"Dump file size: ${DUMPSIZE} bytes\"\n\n if [ \"$DUMPSIZE\" -lt 1000 ]; then\n echo \"***FAIL: pg_dump produced suspiciously small file (${DUMPSIZE} bytes)***\"\n exit 1\n fi\n\n # Create target database and restore using pgstac_restore.\n # pgstac_restore detects PostGIS schema on the target, fixes the\n # search_path that pg_dump clears, and pipes through psql.\n psql -X -q -v ON_ERROR_STOP=1 < /dev/null && pwd )\ncd $SCRIPT_DIR/..\n\nfunction usage() {\n\tcat <&2\n\t\t\tusage\n\t\t\texit 1\n\t\t\t;;\n\tesac\ndone\n\n$SCRIPT_DIR/runinpypgstac --build-policy \"$BUILD_POLICY\" --cpfiles format\n" }, { "path": "scripts/makemigration", "content": "#!/bin/bash\nSCRIPT_DIR=$( cd -- \"$( dirname -- \"${BASH_SOURCE[0]}\" )\" &> /dev/null && pwd )\ncd $SCRIPT_DIR/..\n\nfunction usage() {\n cat < /dev/null && pwd )\ncd $SCRIPT_DIR/..\n\nfunction usage() {\n\tcat < /dev/null && pwd )\ncd $SCRIPT_DIR/..\n\nexport PATH=$SCRIPT_DIR:$PATH\n\nset -e\n\nif [[ \"${CI}\" ]]; then\n set -x\nfi\n\nfunction first_available_pgport() {\n local port\n\n for port in 5439 5440 5441 5442 5443 5444 5445; do\n if ! ss -ltn \"( sport = :$port )\" 2>/dev/null | tail -n +2 | grep -q .; then\n echo \"$port\"\n return 0\n fi\n done\n\n echo 5439\n}\n\nfunction ensure_env_file() {\n if [[ ! -f .env && -f .env.example ]]; then\n cp .env.example .env\n local selected_port\n selected_port=$(first_available_pgport)\n if [[ \"$selected_port\" != \"5439\" ]]; then\n sed -i \"s/^PGPORT=.*/PGPORT=${selected_port}/\" .env\n fi\n echo \"Created .env from .env.example\"\n fi\n}\n" }, { "path": "scripts/runinpypgstac", "content": "#!/bin/bash\nSCRIPT_DIR=$( cd -- \"$( dirname -- \"${BASH_SOURCE[0]}\" )\" &> /dev/null && pwd )\ncd $SCRIPT_DIR/..\nsource \"$SCRIPT_DIR/pgstacenv\"\nset -e\n\nif [[ \"${CI}\" ]]; then\n set -x\nfi\n\nfunction usage() {\n cat < [command args...]\n\nRun a command inside the pypgstac development container.\n\nOptions:\n --build Equivalent to --build-policy always.\n --build-policy POLICY One of: always, missing, never.\n --no-cache Rebuild images without Docker layer cache.\n --cpfiles Copy /opt/src back to the host after the command runs.\n -h, --help Show this help text.\n\nExamples:\n $(basename \"$0\") --build test --pypgstac\n $(basename \"$0\") --build-policy missing format\n $(basename \"$0\") --build-policy always --cpfiles stageversion 0.9.11\nEOF\n}\n\nfunction ensure_images_exist() {\n docker image inspect pgstac >/dev/null 2>&1 && docker image inspect pypgstac >/dev/null 2>&1\n}\n\nfunction wait_for_pgstac() {\n local attempts=60\n local status=\"\"\n local container_id=\"\"\n\n while [[ $attempts -gt 0 ]]; do\n container_id=$(docker compose ps -q pgstac 2>/dev/null || true)\n status=$(docker inspect --format '{{if .State.Health}}{{.State.Health.Status}}{{else}}{{.State.Status}}{{end}}' \"$container_id\" 2>/dev/null || true)\n\n if [[ \"$status\" == \"healthy\" || \"$status\" == \"running\" ]]; then\n return 0\n fi\n\n attempts=$((attempts - 1))\n sleep 1\n done\n\n echo \"Timed out waiting for pgstac to become ready.\" >&2\n return 1\n}\n\n[ \"$#\" -eq 0 ] && usage && exit 1\n\nCONTAINER_ARGS=()\nBUILD_POLICY=\"${PGSTAC_BUILD_POLICY:-always}\"\n\nwhile [[ $# -gt 0 ]]; do\n case \"$1\" in\n --build)\n BUILD_POLICY=\"always\"\n shift\n ;;\n --build-policy)\n BUILD_POLICY=\"$2\"\n shift 2\n ;;\n --no-cache)\n NOCACHE=\"--no-cache\"\n shift\n ;;\n --cpfiles)\n CPFILES=1\n shift\n ;;\n -h|--help)\n usage\n exit 0\n ;;\n --)\n shift\n break\n ;;\n *)\n break\n ;;\n esac\ndone\n\nCONTAINER_ARGS=(\"$@\")\n\nif [[ ${#CONTAINER_ARGS[@]} -eq 0 ]]; then\n usage\n exit 1\nfi\n\ncase \"$BUILD_POLICY\" in\n always|missing|never)\n ;;\n *)\n echo \"Invalid build policy: $BUILD_POLICY\" >&2\n usage\n exit 1\n ;;\nesac\n\nif [[ -n \"$NOCACHE\" && \"$BUILD_POLICY\" == \"never\" ]]; then\n echo \"--no-cache cannot be used with --build-policy never.\" >&2\n exit 1\nfi\n\nif [[ \"$BUILD_POLICY\" == \"always\" || ( \"$BUILD_POLICY\" == \"missing\" && ! ensure_images_exist ) ]]; then\n echo \"Building docker images...\"\n ensure_env_file\n docker compose build ${NOCACHE}\nfi\n\nensure_env_file\nPGSTAC_RUNNING=$(docker compose ps pgstac --status running -q)\necho \"PGSTAC_RUNNING=$PGSTAC_RUNNING\"\n\nif [[ -z \"$PGSTAC_RUNNING\" ]]; then\n docker compose up -d pgstac\n wait_for_pgstac\nfi\n\nif [[ $CPFILES == 1 ]]; then\n echo \"Running pypgstac worker\"\n WORKER_ID=$(docker compose run -d --rm pypgstac tail -f /dev/null)\n echo \"Executing ${CONTAINER_ARGS[@]} in pypgstac worker\"\n docker exec \"$WORKER_ID\" \"${CONTAINER_ARGS[@]}\"\n echo \"copying datafiles to host\"\n docker cp \"$WORKER_ID\":/opt/src $SCRIPT_DIR/..\n echo \"killing pypgstac worker\"\n docker kill \"$WORKER_ID\" >/dev/null\nelse\n echo \"Running ${CONTAINER_ARGS[@]} in pypgstacworker\"\n docker compose run -T --rm pypgstac \"${CONTAINER_ARGS[@]}\"\nfi\nJOBEXITCODE=$?\n[[ $PGSTAC_RUNNING == \"\" ]] && docker compose stop pgstac\nexit $JOBEXITCODE\n" }, { "path": "scripts/server", "content": "#!/bin/bash\nSCRIPT_DIR=$( cd -- \"$( dirname -- \"${BASH_SOURCE[0]}\" )\" &> /dev/null && pwd )\ncd $SCRIPT_DIR/..\nsource \"$SCRIPT_DIR/pgstacenv\"\nset -e\n\nif [[ \"${CI}\" ]]; then\n set -x\nfi\n\nfunction usage() {\n if [[ -n \"$1\" ]]; then\n echo \"$1\" >&2\n echo >&2\n fi\n\n echo -n \\\n \"Usage: $(basename \"$0\") [--detach] [--no-cache] [--help]\nRuns the development database.\n\n--detach: Run in detached mode.\n--no-cache: Rebuild container images from scratch before startup.\n\"\n}\n\nDETACH_ARG=\"\"\n\nwhile [[ \"$#\" > 0 ]]; do case $1 in\n --detach)\n DETACH_ARG=\"--detach\"\n shift\n ;;\n --no-cache)\n NO_CACHE=\"--no-cache\"\n shift\n ;;\n -h|--help)\n usage\n exit 0\n ;;\n *)\n usage \"Unknown option: $1\"\n exit 1\n ;;\n esac; done\n\nif [ \"${BASH_SOURCE[0]}\" = \"${0}\" ]; then\n ensure_env_file\n docker compose build ${NO_CACHE}\n docker compose up ${DETACH_ARG} $@\nfi\n" }, { "path": "scripts/setup", "content": "#!/bin/bash\nSCRIPT_DIR=$( cd -- \"$( dirname -- \"${BASH_SOURCE[0]}\" )\" &> /dev/null && pwd )\ncd $SCRIPT_DIR/..\nset -e\n\nif [[ \"${CI}\" ]]; then\n set -x\nfi\n\nfunction usage() {\n echo -n \\\n \"Usage: $(basename \"$0\") [--help]\nSets up this project for development.\n\"\n}\n\nwhile [[ \"$#\" -gt 0 ]]; do\n case \"$1\" in\n -h|--help)\n usage\n exit 0\n ;;\n *)\n echo \"Unknown option: $1\" >&2\n usage\n exit 1\n ;;\n esac\ndone\n\nif [ \"${BASH_SOURCE[0]}\" = \"${0}\" ]; then\n\n # Build docker containers\n scripts/update\n\n echo \"Bringing up database...\"\n scripts/server --detach\n\n echo \"Done.\"\n\nfi\n" }, { "path": "scripts/stageversion", "content": "#!/bin/bash\nSCRIPT_DIR=$( cd -- \"$( dirname -- \"${BASH_SOURCE[0]}\" )\" &> /dev/null && pwd )\ncd $SCRIPT_DIR/..\n\nfunction usage() {\n\tcat < /dev/null && pwd )\n\nfunction usage() {\n\tcat </dev/null || true)\n\t[[ -n \"$changed\" ]]\n}\n\nif [[ \"$BUILD_POLICY\" != \"always\" && $STRICT -eq 1 ]]; then\n\tif [[ $WATCH -eq 1 ]]; then\n\t\techo \"Watch mode requires --build-policy always unless --no-strict is set.\" >&2\n\t\texit 1\n\tfi\n\n\tif has_rebuild_sensitive_changes; then\n\t\techo \"Refusing to run tests with build-policy '$BUILD_POLICY' while scripts/docker/src changes are present.\" >&2\n\t\techo \"Use --build-policy always or --no-strict if you intentionally want to reuse an existing image.\" >&2\n\t\texit 1\n\tfi\nfi\n\nRUN_ARGS=(--build-policy \"$BUILD_POLICY\" test)\nRUN_ARGS+=(\"${POSITIONAL[@]}\")\n\nfunction run_once() {\n\t\"$SCRIPT_DIR/runinpypgstac\" \"${RUN_ARGS[@]}\"\n}\n\nif [[ $WATCH -eq 1 ]]; then\n\tif ! command -v inotifywait >/dev/null 2>&1; then\n\t\tif [[ $STRICT -eq 1 ]]; then\n\t\t\techo \"Watch mode requires inotifywait (from inotify-tools).\" >&2\n\t\t\texit 1\n\t\tfi\n\n\t\techo \"Watch mode requested but inotifywait is unavailable; running tests once because --no-strict was set.\" >&2\n\t\trun_once\n\t\texit $?\n\tfi\n\n\twhile true; do\n\t\trun_once || true\n\t\techo \"Waiting for file changes...\"\n\t\tinotifywait -qq -r -e close_write,create,delete,move \"$SCRIPT_DIR/..\"/scripts \"$SCRIPT_DIR/..\"/docker \"$SCRIPT_DIR/..\"/src \"$SCRIPT_DIR/..\"/.github \"$SCRIPT_DIR/..\"/CONTRIBUTING.md \"$SCRIPT_DIR/..\"/CLAUDE.md\n\tdone\nfi\n\nrun_once\n" }, { "path": "scripts/update", "content": "#!/bin/bash\nSCRIPT_DIR=$( cd -- \"$( dirname -- \"${BASH_SOURCE[0]}\" )\" &> /dev/null && pwd )\ncd $SCRIPT_DIR/..\nsource \"$SCRIPT_DIR/pgstacenv\"\nset -e\n\nif [[ \"${CI}\" ]]; then\n set -x\nfi\n\nfunction usage() {\n echo -n \\\n \"Usage: $(basename \"$0\") [--no-cache]\nBuilds the docker containers for this project.\n\n--no-cache: Rebuild all containers from scratch.\n\"\n}\n\n# Parse args\nNO_CACHE=\"\";\nwhile [[ \"$#\" > 0 ]]; do case $1 in\n --no-cache) NO_CACHE=\"--no-cache\"; shift;;\n -h|--help) usage; exit 0;;\n *) echo \"Unknown parameter passed: $1\" >&2; usage; exit 1;;\nesac; done\n\nif [ \"${BASH_SOURCE[0]}\" = \"${0}\" ]; then\n\n echo \"==Building images...\"\n\n ensure_env_file\n docker compose build ${NO_CACHE}\n\nfi\n" }, { "path": "src/pgstac/migrations/pgstac.0.1.9-0.2.3.sql", "content": "SET SEARCH_PATH to pgstac, public;\nINSERT INTO migrations (version) VALUES ('0.2.3');\n" }, { "path": "src/pgstac/migrations/pgstac.0.1.9.sql", "content": "CREATE EXTENSION IF NOT EXISTS postgis;\nCREATE SCHEMA IF NOT EXISTS partman;\nCREATE EXTENSION IF NOT EXISTS pg_partman SCHEMA partman;\nCREATE SCHEMA IF NOT EXISTS pgstac;\n\n\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE TABLE migrations (\n version text,\n datetime timestamptz DEFAULT now() NOT NULL\n);\n\n/* converts a jsonb text array to a pg text[] array */\nCREATE OR REPLACE FUNCTION textarr(_js jsonb)\n RETURNS text[] AS $$\n SELECT ARRAY(SELECT jsonb_array_elements_text(_js));\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\n/*\nconverts a jsonb text array to comma delimited list of identifer quoted\nuseful for constructing column lists for selects\n*/\nCREATE OR REPLACE FUNCTION array_idents(_js jsonb)\n RETURNS text AS $$\n SELECT string_agg(quote_ident(v),',') FROM jsonb_array_elements_text(_js) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\n\n/* looks for a geometry in a stac item first from geometry and falling back to bbox */\nCREATE OR REPLACE FUNCTION stac_geom(value jsonb) RETURNS geometry AS $$\nSELECT\n CASE\n WHEN value->>'geometry' IS NOT NULL THEN\n ST_GeomFromGeoJSON(value->>'geometry')\n WHEN value->>'bbox' IS NOT NULL THEN\n ST_MakeEnvelope(\n (value->'bbox'->>0)::float,\n (value->'bbox'->>1)::float,\n (value->'bbox'->>2)::float,\n (value->'bbox'->>3)::float,\n 4326\n )\n ELSE NULL\n END as geometry\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION stac_datetime(value jsonb) RETURNS timestamptz AS $$\nSELECT (value->'properties'->>'datetime')::timestamptz;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION jsonb_paths (IN jdata jsonb, OUT path text[], OUT value jsonb) RETURNS\nSETOF RECORD AS $$\nwith recursive extract_all as\n(\n select\n ARRAY[key]::text[] as path,\n value\n FROM jsonb_each(jdata)\nunion all\n select\n path || coalesce(obj_key, (arr_key- 1)::text),\n coalesce(obj_value, arr_value)\n from extract_all\n left join lateral\n jsonb_each(case jsonb_typeof(value) when 'object' then value end)\n as o(obj_key, obj_value)\n on jsonb_typeof(value) = 'object'\n left join lateral\n jsonb_array_elements(case jsonb_typeof(value) when 'array' then value end)\n with ordinality as a(arr_value, arr_key)\n on jsonb_typeof(value) = 'array'\n where obj_key is not null or arr_key is not null\n)\nselect *\nfrom extract_all;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION jsonb_obj_paths (IN jdata jsonb, OUT path text[], OUT value jsonb) RETURNS\nSETOF RECORD AS $$\nwith recursive extract_all as\n(\n select\n ARRAY[key]::text[] as path,\n value\n FROM jsonb_each(jdata)\nunion all\n select\n path || obj_key,\n obj_value\n from extract_all\n left join lateral\n jsonb_each(case jsonb_typeof(value) when 'object' then value end)\n as o(obj_key, obj_value)\n on jsonb_typeof(value) = 'object'\n where obj_key is not null\n)\nselect *\nfrom extract_all;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION jsonb_val_paths (IN jdata jsonb, OUT path text[], OUT value jsonb) RETURNS\nSETOF RECORD AS $$\nSELECT * FROM jsonb_obj_paths(jdata) WHERE jsonb_typeof(value) not in ('object','array');\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION path_includes(IN path text[], IN includes text[]) RETURNS BOOLEAN AS $$\nWITH t AS (SELECT unnest(includes) i)\nSELECT EXISTS (\n SELECT 1 FROM t WHERE path @> string_to_array(i, '.')\n);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION path_excludes(IN path text[], IN excludes text[]) RETURNS BOOLEAN AS $$\nWITH t AS (SELECT unnest(excludes) e)\nSELECT NOT EXISTS (\n SELECT 1 FROM t WHERE path @> string_to_array(e, '.')\n);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION jsonb_obj_paths_filtered (\n IN jdata jsonb,\n IN includes text[] DEFAULT ARRAY[]::text[],\n IN excludes text[] DEFAULT ARRAY[]::text[],\n OUT path text[],\n OUT value jsonb\n) RETURNS\nSETOF RECORD AS $$\nSELECT path, value\nFROM jsonb_obj_paths(jdata)\nWHERE\n CASE WHEN cardinality(includes) > 0 THEN path_includes(path, includes) ELSE TRUE END\n AND\n path_excludes(path, excludes)\n\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION empty_arr(ANYARRAY) RETURNS BOOLEAN AS $$\nSELECT CASE\n WHEN $1 IS NULL THEN TRUE\n WHEN cardinality($1)<1 THEN TRUE\nELSE FALSE\nEND;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION filter_jsonb(\n IN jdata jsonb,\n IN includes text[] DEFAULT ARRAY[]::text[],\n IN excludes text[] DEFAULT ARRAY[]::text[]\n) RETURNS jsonb AS $$\nDECLARE\nrec RECORD;\noutj jsonb := '{}'::jsonb;\ncreated_paths text[] := '{}'::text[];\nBEGIN\n\nIF empty_arr(includes) AND empty_arr(excludes) THEN\nRAISE NOTICE 'no filter';\n RETURN jdata;\nEND IF;\nFOR rec in\nSELECT * FROM jsonb_obj_paths_filtered(jdata, includes, excludes)\nWHERE jsonb_typeof(value) != 'object'\nLOOP\n RAISE NOTICE 'path % val %', rec.path, rec.value;\n IF array_length(rec.path,1)>1 THEN\n FOR i IN 1..(array_length(rec.path,1)-1) LOOP\n IF NOT array_to_string(rec.path[1:i],'.') = ANY (created_paths) THEN\n outj := jsonb_set(outj, rec.path[1:i],'{}', true);\n created_paths := created_paths || array_to_string(rec.path[1:i],'.');\n END IF;\n END LOOP;\n END IF;\n outj := jsonb_set(outj, rec.path, rec.value, true);\n created_paths := created_paths || array_to_string(rec.path,'.');\nEND LOOP;\nRETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION properties_idx(_in jsonb) RETURNS jsonb AS $$\nWITH t AS (\n select array_to_string(path,'.') as path, lower(value::text)::jsonb as lowerval\n FROM jsonb_val_paths(_in)\n WHERE array_to_string(path,'.') not in ('datetime')\n)\nSELECT jsonb_object_agg(path, lowerval) FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE TABLE IF NOT EXISTS collections (\n id VARCHAR GENERATED ALWAYS AS (content->>'id') STORED PRIMARY KEY,\n content JSONB\n);\n\nCREATE OR REPLACE FUNCTION create_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection(data jsonb) RETURNS VOID AS $$\nDECLARE\nout collections%ROWTYPE;\nBEGIN\n UPDATE collections SET content=data WHERE id = data->>'id' RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ON CONFLICT (id) DO\n UPDATE\n SET content=EXCLUDED.content\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION delete_collection(_id text) RETURNS VOID AS $$\nDECLARE\nout collections%ROWTYPE;\nBEGIN\n DELETE FROM collections WHERE id = _id RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION get_collection(id text) RETURNS jsonb AS $$\nSELECT content FROM collections\nWHERE id=$1\n;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION all_collections() RETURNS jsonb AS $$\nSELECT jsonb_agg(content) FROM collections;\n;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\n\n\n\n\n\n/* CREATE OR REPLACE FUNCTION collections_trigger_func()\nRETURNS TRIGGER AS $$\nBEGIN\n IF pg_trigger_depth() = 1 THEN\n PERFORM create_collection(NEW.content);\n RETURN NULL;\n END IF;\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE TRIGGER collections_trigger\nBEFORE INSERT ON collections\nFOR EACH ROW EXECUTE PROCEDURE collections_trigger_func();\n */\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE TABLE IF NOT EXISTS items (\n id VARCHAR GENERATED ALWAYS AS (content->>'id') STORED NOT NULL,\n geometry geometry GENERATED ALWAYS AS (stac_geom(content)) STORED NOT NULL,\n properties jsonb GENERATED ALWAYS as (properties_idx(content->'properties')) STORED,\n collection_id text GENERATED ALWAYS AS (content->>'collection') STORED NOT NULL,\n datetime timestamptz GENERATED ALWAYS AS (stac_datetime(content)) STORED NOT NULL,\n content JSONB NOT NULL\n)\nPARTITION BY RANGE (stac_datetime(content))\n;\n\nALTER TABLE items ADD constraint items_collections_fk FOREIGN KEY (collection_id) REFERENCES collections(id) DEFERRABLE;\n\nCREATE TABLE items_template (\n LIKE items\n);\n\nALTER TABLE items_template ADD PRIMARY KEY (id);\n\n/*\nCREATE TABLE IF NOT EXISTS items_search (\n id text NOT NULL,\n geometry geometry NOT NULL,\n properties jsonb,\n collection_id text NOT NULL,\n datetime timestamptz NOT NULL\n)\nPARTITION BY RANGE (datetime)\n;\n\nCREATE TABLE IF NOT EXISTS items_search_template (\n LIKE items_search\n)\n;\nALTER TABLE items_search_template ADD PRIMARY KEY (id);\n*/\n\nDELETE from partman.part_config WHERE parent_table = 'pgstac.items';\nSELECT partman.create_parent(\n 'pgstac.items',\n 'datetime',\n 'native',\n 'weekly',\n p_template_table := 'pgstac.items_template',\n p_premake := 4\n);\n\nCREATE OR REPLACE FUNCTION make_partitions(st timestamptz, et timestamptz DEFAULT NULL) RETURNS BOOL AS $$\nWITH t AS (\n SELECT\n generate_series(\n date_trunc('week',st),\n date_trunc('week', coalesce(et, st)),\n '1 week'::interval\n ) w\n),\nw AS (SELECT array_agg(w) as w FROM t)\nSELECT CASE WHEN w IS NULL THEN NULL ELSE partman.create_partition_time('pgstac.items', w, true) END FROM w;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_partition(timestamptz) RETURNS text AS $$\nSELECT to_char($1, '\"items_p\"IYYY\"w\"IW');\n$$ LANGUAGE SQL;\n\nCREATE INDEX \"datetime_id_idx\" ON items (datetime, id);\nCREATE INDEX \"properties_idx\" ON items USING GIN (properties);\nCREATE INDEX \"collection_idx\" ON items (collection_id);\nCREATE INDEX \"geometry_idx\" ON items USING GIST (geometry);\n\n\nCREATE TYPE item AS (\n id text,\n geometry geometry,\n properties JSONB,\n collection_id text,\n datetime timestamptz\n);\n\n\n/*\nConverts single feature into an items row\n*/\n\n/*\nCREATE OR REPLACE FUNCTION feature_to_item(value jsonb) RETURNS item AS $$\n SELECT\n value->>'id' as id,\n CASE\n WHEN value->>'geometry' IS NOT NULL THEN\n ST_GeomFromGeoJSON(value->>'geometry')\n WHEN value->>'bbox' IS NOT NULL THEN\n ST_MakeEnvelope(\n (value->'bbox'->>0)::float,\n (value->'bbox'->>1)::float,\n (value->'bbox'->>2)::float,\n (value->'bbox'->>3)::float,\n 4326\n )\n ELSE NULL\n END as geometry,\n properties_idx(value ->'properties') as properties,\n value->>'collection' as collection_id,\n (value->'properties'->>'datetime')::timestamptz as datetime\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac,public;\n*/\n/*\nTakes a single feature, an array of features, or a feature collection\nand returns a set up individual items rows\n*/\n/*\nCREATE OR REPLACE FUNCTION features_to_items(value jsonb) RETURNS SETOF item AS $$\n WITH features AS (\n SELECT\n jsonb_array_elements(\n CASE\n WHEN jsonb_typeof(value) = 'array' THEN value\n WHEN value->>'type' = 'Feature' THEN '[]'::jsonb || value\n WHEN value->>'type' = 'FeatureCollection' THEN value->'features'\n ELSE NULL\n END\n ) as value\n )\n SELECT feature_to_item(value) FROM features\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac,public;\n*/\n\nCREATE OR REPLACE FUNCTION get_item(_id text) RETURNS jsonb AS $$\n SELECT content FROM items WHERE id=_id;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION delete_item(_id text) RETURNS VOID AS $$\nDECLARE\nout items%ROWTYPE;\nBEGIN\n DELETE FROM items WHERE id = _id RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION create_item(data jsonb) RETURNS VOID AS $$\n SELECT make_partitions(stac_datetime(data));\n INSERT INTO items (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION update_item(data jsonb) RETURNS VOID AS $$\nDECLARE\nout items%ROWTYPE;\nBEGIN\n UPDATE items SET content=data WHERE id = data->>'id' RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_item(data jsonb) RETURNS VOID AS $$\nDECLARE\npartition text;\nq text;\nnewcontent jsonb;\nBEGIN\n PERFORM make_partitions(stac_datetime(data));\n partition := get_partition(stac_datetime(data));\n q := format($q$\n INSERT INTO %I (content) VALUES ($1)\n ON CONFLICT (id) DO\n UPDATE SET content = EXCLUDED.content\n WHERE %I.content IS DISTINCT FROM EXCLUDED.content RETURNING content;\n $q$, partition, partition);\n EXECUTE q INTO newcontent USING (data);\n RAISE NOTICE 'newcontent: %', newcontent;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION analyze_empty_partitions() RETURNS VOID AS $$\nDECLARE\np text;\nBEGIN\nFOR p IN SELECT partition FROM all_items_partitions WHERE est_cnt = 0 LOOP\n RAISE NOTICE 'Analyzing %', p;\n EXECUTE format('ANALYZE %I;', p);\nEND LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n/* Trigger Function to cascade inserts/updates/deletes\nfrom items table to items_search table */\n/*\nALTER TABLE items_search ADD CONSTRAINT items_search_fk\nFOREIGN KEY (id) REFERENCES items(id)\nON DELETE CASCADE DEFERRABLE;\n\nCREATE OR REPLACE FUNCTION items_trigger_func()\nRETURNS TRIGGER AS $$\nDECLARE\nBEGIN\n IF TG_OP = 'UPDATE' THEN\n RAISE NOTICE 'DELETING % BEFORE UPDATE', OLD;\n DELETE FROM items_search WHERE id = OLD.id AND datetime = (OLD.content->'properties'->>'datetime')::timestamptz;\n END IF;\n\n INSERT INTO items_search SELECT * FROM feature_to_item(NEW.content);\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nDROP TRIGGER IF EXISTS items_insert_trigger ON items;\nCREATE TRIGGER items_insert_trigger\nAFTER INSERT ON items\nFOR EACH ROW EXECUTE PROCEDURE items_trigger_func();\n\nDROP TRIGGER IF EXISTS items_update_trigger ON items;\nCREATE TRIGGER items_update_trigger\nAFTER UPDATE ON items\nFOR EACH ROW\nWHEN (NEW.content IS DISTINCT FROM OLD.content)\nEXECUTE PROCEDURE items_trigger_func();\n*/\n\n/* Trigger Function to cascade inserts/updates/deletes\nfrom items table to items_search table */\n/*\nCREATE OR REPLACE FUNCTION items_search_trigger_delete_func()\nRETURNS TRIGGER AS $$\nDECLARE\nBEGIN\n RAISE NOTICE 'Deleting from items_search: % Depth: %', OLD, pg_trigger_depth();\n IF pg_trigger_depth()<3 THEN\n RAISE NOTICE 'DELETING WITH datetime';\n DELETE FROM items_search WHERE id=OLD.id AND datetime=OLD.datetime;\n RETURN NULL;\n END IF;\n RETURN OLD;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nDROP TRIGGER IF EXISTS items_search_delete_trigger ON items_search;\nCREATE TRIGGER items_search_delete_trigger\nBEFORE DELETE ON items_search\nFOR EACH ROW EXECUTE PROCEDURE items_search_trigger_delete_func();\n*/\nCREATE OR REPLACE FUNCTION backfill_partitions()\nRETURNS VOID AS $$\nDECLARE\nBEGIN\n IF EXISTS (SELECT 1 FROM items_default LIMIT 1) THEN\n RAISE NOTICE 'Creating new partitions and moving data from default';\n CREATE TEMP TABLE items_default_tmp ON COMMIT DROP AS SELECT datetime, content FROM items_default;\n TRUNCATE items_default;\n PERFORM make_partitions(min(datetime), max(datetime)) FROM items_default_tmp;\n INSERT INTO items (content) SELECT content FROM items_default_tmp;\n END IF;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION items_trigger_stmt_func()\nRETURNS TRIGGER AS $$\nDECLARE\nBEGIN\n PERFORM analyze_empty_partitions();\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nDROP TRIGGER IF EXISTS items_stmt_trigger ON items;\nCREATE TRIGGER items_stmt_trigger\nAFTER INSERT OR UPDATE OR DELETE ON items\nFOR EACH STATEMENT EXECUTE PROCEDURE items_trigger_stmt_func();\n\n\n/*\nView to get a table of available items partitions\nwith date ranges\n*/\n--DROP VIEW IF EXISTS all_items_partitions CASCADE;\nCREATE OR REPLACE VIEW all_items_partitions AS\nWITH base AS\n(SELECT\n c.oid::pg_catalog.regclass::text as partition,\n pg_catalog.pg_get_expr(c.relpartbound, c.oid) as _constraint,\n regexp_matches(\n pg_catalog.pg_get_expr(c.relpartbound, c.oid),\n E'\\\\(''\\([0-9 :+-]*\\)''\\\\).*\\\\(''\\([0-9 :+-]*\\)''\\\\)'\n ) as t,\n reltuples::bigint as est_cnt\nFROM pg_catalog.pg_class c, pg_catalog.pg_inherits i\nWHERE c.oid = i.inhrelid AND i.inhparent = 'items'::regclass)\nSELECT partition, tstzrange(\n t[1]::timestamptz,\n t[2]::timestamptz\n), est_cnt\nFROM base\nORDER BY 2 desc;\n\n--DROP VIEW IF EXISTS items_partitions;\nCREATE OR REPLACE VIEW items_partitions AS\nSELECT * FROM all_items_partitions WHERE est_cnt>0;\n\nCREATE OR REPLACE FUNCTION collection_bbox(id text) RETURNS jsonb AS $$\nSELECT (replace(replace(replace(st_extent(geometry)::text,'BOX(','[['),')',']]'),' ',','))::jsonb\nFROM items WHERE collection_id=$1;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION collection_temporal_extent(id text) RETURNS jsonb AS $$\nSELECT to_jsonb(array[array[min(datetime)::text, max(datetime)::text]])\nFROM items WHERE collection_id=$1;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection_extents() RETURNS VOID AS $$\nUPDATE collections SET\n content = content ||\n jsonb_build_object(\n 'extent', jsonb_build_object(\n 'spatial', jsonb_build_object(\n 'bbox', collection_bbox(collections.id)\n ),\n 'temporal', jsonb_build_object(\n 'interval', collection_temporal_extent(collections.id)\n )\n )\n )\n;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\nSET SEARCH_PATH TO pgstac, public;\n\n/*\nView to get a table of available items partitions\nwith date ranges\n*/\nDROP VIEW IF EXISTS items_partitions;\nCREATE VIEW items_partitions AS\nWITH base AS\n(SELECT\n c.oid::pg_catalog.regclass::text as partition,\n pg_catalog.pg_get_expr(c.relpartbound, c.oid) as _constraint,\n regexp_matches(\n pg_catalog.pg_get_expr(c.relpartbound, c.oid),\n E'\\\\(''\\([0-9 :+-]*\\)''\\\\).*\\\\(''\\([0-9 :+-]*\\)''\\\\)'\n ) as t,\n reltuples::bigint as est_cnt\nFROM pg_catalog.pg_class c, pg_catalog.pg_inherits i\nWHERE c.oid = i.inhrelid AND i.inhparent = 'items'::regclass)\nSELECT partition, tstzrange(\n t[1]::timestamptz,\n t[2]::timestamptz\n), est_cnt\nFROM base\nWHERE est_cnt >0\nORDER BY 2 desc;\n\n\nCREATE OR REPLACE FUNCTION items_by_partition(\n IN _where text DEFAULT 'TRUE',\n IN _dtrange tstzrange DEFAULT tstzrange('-infinity','infinity'),\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN _limit int DEFAULT 10\n) RETURNS SETOF items AS $$\nDECLARE\npartition_query text;\nmain_query text;\nbatchcount int;\ncounter int := 0;\np record;\nBEGIN\nIF _orderby ILIKE 'datetime d%' THEN\n partition_query := format($q$\n SELECT partition\n FROM items_partitions\n WHERE tstzrange && $1\n ORDER BY tstzrange DESC;\n $q$);\nELSIF _orderby ILIKE 'datetime a%' THEN\n partition_query := format($q$\n SELECT partition\n FROM items_partitions\n WHERE tstzrange && $1\n ORDER BY tstzrange ASC\n ;\n $q$);\nELSE\n partition_query := format($q$\n SELECT 'items' as partition WHERE $1 IS NOT NULL;\n $q$);\nEND IF;\nRAISE NOTICE 'Partition Query: %', partition_query;\nFOR p IN\n EXECUTE partition_query USING (_dtrange)\nLOOP\n IF lower(_dtrange)::timestamptz > '-infinity' THEN\n _where := concat(_where,format(' AND datetime >= %L',lower(_dtrange)::timestamptz::text));\n END IF;\n IF upper(_dtrange)::timestamptz < 'infinity' THEN\n _where := concat(_where,format(' AND datetime <= %L',upper(_dtrange)::timestamptz::text));\n END IF;\n\n main_query := format($q$\n SELECT * FROM %I\n WHERE %s\n ORDER BY %s\n LIMIT %s - $1\n $q$, p.partition::text, _where, _orderby, _limit\n );\n RAISE NOTICE 'Partition Query %', main_query;\n RAISE NOTICE '%', counter;\n RETURN QUERY EXECUTE main_query USING counter;\n\n GET DIAGNOSTICS batchcount = ROW_COUNT;\n counter := counter + batchcount;\n RAISE NOTICE 'FOUND %', batchcount;\n IF counter >= _limit THEN\n EXIT;\n END IF;\n RAISE NOTICE 'ADDED % FOR A TOTAL OF %', batchcount, counter;\nEND LOOP;\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION split_stac_path(IN path text, OUT col text, OUT dotpath text, OUT jspath text, OUT jspathtext text) AS $$\nWITH col AS (\n SELECT\n CASE WHEN\n split_part(path, '.', 1) IN ('id', 'stac_version', 'stac_extensions','geometry','properties','assets','collection_id','datetime','links', 'extra_fields') THEN split_part(path, '.', 1)\n ELSE 'properties'\n END AS col\n),\ndp AS (\n SELECT\n col, ltrim(replace(path, col , ''),'.') as dotpath\n FROM col\n),\npaths AS (\nSELECT\n col, dotpath,\n regexp_split_to_table(dotpath,E'\\\\.') as path FROM dp\n) SELECT\n col,\n btrim(concat(col,'.',dotpath),'.'),\n CASE WHEN btrim(concat(col,'.',dotpath),'.') != col THEN concat(col,'->',string_agg(concat('''',path,''''),'->')) ELSE col END,\n regexp_replace(\n CASE WHEN btrim(concat(col,'.',dotpath),'.') != col THEN concat(col,'->',string_agg(concat('''',path,''''),'->')) ELSE col END,\n E'>([^>]*)$','>>\\1'\n )\nFROM paths group by col, dotpath;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n/* Functions for searching items */\nCREATE OR REPLACE FUNCTION sort_base(\n IN _sort jsonb DEFAULT '[{\"field\":\"datetime\",\"direction\":\"desc\"}]',\n OUT key text,\n OUT col text,\n OUT dir text,\n OUT rdir text,\n OUT sort text,\n OUT rsort text\n) RETURNS SETOF RECORD AS $$\nWITH sorts AS (\n SELECT\n value->>'field' as key,\n (split_stac_path(value->>'field')).jspathtext as col,\n coalesce(upper(value->>'direction'),'ASC') as dir\n FROM jsonb_array_elements('[]'::jsonb || coalesce(_sort,'[{\"field\":\"datetime\",\"direction\":\"desc\"}]') )\n)\nSELECT\n key,\n col,\n dir,\n CASE dir WHEN 'DESC' THEN 'ASC' ELSE 'ASC' END as rdir,\n concat(col, ' ', dir, ' NULLS LAST ') AS sort,\n concat(col,' ', CASE dir WHEN 'DESC' THEN 'ASC' ELSE 'ASC' END, ' NULLS LAST ') AS rsort\nFROM sorts\nUNION ALL\nSELECT 'id', 'id', 'DESC', 'ASC', 'id DESC', 'id ASC'\n;\n$$ LANGUAGE SQL;\n\n\nCREATE OR REPLACE FUNCTION sort(_sort jsonb) RETURNS text AS $$\nSELECT string_agg(sort,', ') FROM sort_base(_sort);\n$$ LANGUAGE SQL PARALLEL SAFE SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION rsort(_sort jsonb) RETURNS text AS $$\nSELECT string_agg(rsort,', ') FROM sort_base(_sort);\n$$ LANGUAGE SQL PARALLEL SAFE SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION bbox_geom(_bbox jsonb) RETURNS box3d AS $$\nSELECT CASE jsonb_array_length(_bbox)\n WHEN 4 THEN\n ST_SetSRID(ST_MakeEnvelope(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float,\n (_bbox->>3)::float\n ),4326)\n WHEN 6 THEN\n ST_SetSRID(ST_3DMakeBox(\n ST_MakePoint(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float\n ),\n ST_MakePoint(\n (_bbox->>3)::float,\n (_bbox->>4)::float,\n (_bbox->>5)::float\n )\n ),4326)\n ELSE null END;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION in_array_q(col text, arr jsonb) RETURNS text AS $$\nSELECT CASE jsonb_typeof(arr) WHEN 'array' THEN format('%I = ANY(textarr(%L))', col, arr) ELSE format('%I = %L', col, arr) END;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION count_by_delim(text, text) RETURNS int AS $$\nSELECT count(*) FROM regexp_split_to_table($1,$2);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION stac_query_op(att text, _op text, val jsonb) RETURNS text AS $$\nDECLARE\nret text := '';\nop text;\njp text;\natt_parts RECORD;\nval_str text;\nBEGIN\nval_str := lower(jsonb_build_object('a',val)->>'a');\nRAISE NOTICE 'val_str %', val_str;\n\natt_parts := split_stac_path(att);\n\nop := CASE _op\n WHEN 'eq' THEN '='\n WHEN 'ge' THEN '>='\n WHEN 'gt' THEN '>'\n WHEN 'le' THEN '<='\n WHEN 'lt' THEN '<'\n WHEN 'ne' THEN '!='\n WHEN 'neq' THEN '!='\n WHEN 'startsWith' THEN 'LIKE'\n WHEN 'endsWith' THEN 'LIKE'\n WHEN 'contains' THEN 'LIKE'\n ELSE _op\nEND;\n\nval_str := CASE _op\n WHEN 'startsWith' THEN concat(val_str, '%')\n WHEN 'endsWith' THEN concat('%', val_str)\n WHEN 'contains' THEN concat('%',val_str,'%')\n ELSE val_str\nEND;\n\n\nRAISE NOTICE 'att_parts: % %', att_parts, count_by_delim(att_parts.dotpath,'\\.');\nIF\n op = '='\n AND att_parts.col = 'properties'\n --AND count_by_delim(att_parts.dotpath,'\\.') = 2\nTHEN\n -- use jsonpath query to leverage index for eqaulity tests on single level deep properties\n jp := btrim(format($jp$ $.%I[*] ? ( @ == %s ) $jp$, replace(att_parts.dotpath, 'properties.',''), lower(val::text)::jsonb));\n raise notice 'jp: %', jp;\n ret := format($q$ properties @? %L $q$, jp);\nELSIF jsonb_typeof(val) = 'number' THEN\n ret := format('(%s)::numeric %s %s', att_parts.jspathtext, op, val);\nELSE\n ret := format('%s %s %L', att_parts.jspathtext, op, val_str);\nEND IF;\nRAISE NOTICE 'Op Query: %', ret;\n\nreturn ret;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION stac_query(_query jsonb) RETURNS TEXT[] AS $$\nDECLARE\nqa text[];\natt text;\nops jsonb;\nop text;\nval jsonb;\nBEGIN\nFOR att, ops IN SELECT key, value FROM jsonb_each(_query)\nLOOP\n FOR op, val IN SELECT key, value FROM jsonb_each(ops)\n LOOP\n qa := array_append(qa, stac_query_op(att,op, val));\n RAISE NOTICE '% % %', att, op, val;\n END LOOP;\nEND LOOP;\nRETURN qa;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION filter_by_order(item_id text, _sort jsonb, _type text) RETURNS text AS $$\nDECLARE\nitem item;\nBEGIN\nSELECT * INTO item FROM items WHERE id=item_id;\nRETURN filter_by_order(item, _sort, _type);\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n-- Used to create filters used for paging using the items id from the token\nCREATE OR REPLACE FUNCTION filter_by_order(_item item, _sort jsonb, _type text) RETURNS text AS $$\nDECLARE\nsorts RECORD;\nfilts text[];\nitemval text;\nop text;\nidop text;\nret text;\neq_flag text;\n_item_j jsonb := to_jsonb(_item);\nBEGIN\nFOR sorts IN SELECT * FROM sort_base(_sort) LOOP\n IF sorts.col = 'datetime' THEN\n CONTINUE;\n END IF;\n IF sorts.col='id' AND _type IN ('prev','next') THEN\n eq_flag := '';\n ELSE\n eq_flag := '=';\n END IF;\n\n op := concat(\n CASE\n WHEN _type in ('prev','first') AND sorts.dir = 'ASC' THEN '<'\n WHEN _type in ('last','next') AND sorts.dir = 'ASC' THEN '>'\n WHEN _type in ('prev','first') AND sorts.dir = 'DESC' THEN '>'\n WHEN _type in ('last','next') AND sorts.dir = 'DESC' THEN '<'\n END,\n eq_flag\n );\n\n IF _item_j ? sorts.col THEN\n filts = array_append(filts, format('%s %s %L', sorts.col, op, _item_j->>sorts.col));\n END IF;\nEND LOOP;\nret := coalesce(array_to_string(filts,' AND '), 'TRUE');\nRAISE NOTICE 'Order Filter %', ret;\nRETURN ret;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION search_dtrange(IN _indate jsonb, OUT _tstzrange tstzrange) AS\n$$\nWITH t AS (\n SELECT CASE\n WHEN jsonb_typeof(_indate) = 'array' THEN\n textarr(_indate)\n ELSE\n regexp_split_to_array(\n btrim(_indate::text,'\"'),\n '/'\n )\n END AS arr\n)\n, t1 AS (\n SELECT\n CASE\n WHEN array_upper(arr,1) = 1 OR arr[1] = '..' OR arr[1] IS NULL THEN '-infinity'::timestamptz\n ELSE arr[1]::timestamptz\n END AS st,\n CASE\n WHEN array_upper(arr,1) = 1 THEN arr[1]::timestamptz\n WHEN arr[2] = '..' OR arr[2] IS NULL THEN 'infinity'::timestamptz\n ELSE arr[2]::timestamptz\n END AS et\n FROM t\n)\nSELECT\n tstzrange(st,et)\nFROM t1;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION flip_jsonb_array(j jsonb) RETURNS jsonb AS $$\nWITH t AS (\n SELECT i, row_number() over () as r FROM jsonb_array_elements(j) i\n), o AS (\n SELECT i FROM t ORDER BY r DESC\n)\nSELECT jsonb_agg(i) from o\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION search(_search jsonb = '{}'::jsonb) RETURNS SETOF jsonb AS $$\nDECLARE\nqstart timestamptz := clock_timestamp();\n_sort text := '';\n_rsort text := '';\n_limit int := 10;\n_geom geometry;\nqa text[];\npq text[];\nquery text;\npq_prop record;\npq_op record;\nprev_id text := NULL;\nnext_id text := NULL;\nwhereq text := 'TRUE';\nlinks jsonb := '[]'::jsonb;\ntoken text;\ntok_val text;\ntok_q text := 'TRUE';\ntok_sort text;\nfirst_id text;\nfirst_dt timestamptz;\nlast_id text;\nsort text;\nrsort text;\ndt text[];\ndqa text[];\ndq text;\nmq_where text;\nstartdt timestamptz;\nenddt timestamptz;\nitem items%ROWTYPE;\ncounter int := 0;\nbatchcount int;\nmonth timestamptz;\nm record;\n_dtrange tstzrange := tstzrange('-infinity','infinity');\n_dtsort text;\n_token_dtrange tstzrange := tstzrange('-infinity','infinity');\n_token_record items%ROWTYPE;\nis_prev boolean := false;\nincludes text[];\nexcludes text[];\nBEGIN\n-- Create table from sort query of items to sort\nCREATE TEMP TABLE pgstac_tmp_sorts ON COMMIT DROP AS SELECT * FROM sort_base(_search->'sortby');\n\n-- Get the datetime sort direction, necessary for efficient cycling through partitions\nSELECT INTO _dtsort dir FROM pgstac_tmp_sorts WHERE key='datetime';\nRAISE NOTICE '_dtsort: %',_dtsort;\n\nSELECT INTO _sort string_agg(s.sort,', ') FROM pgstac_tmp_sorts s;\nSELECT INTO _rsort string_agg(s.rsort,', ') FROM pgstac_tmp_sorts s;\ntok_sort := _sort;\n\n\n-- Get datetime from query as a tstzrange\nIF _search ? 'datetime' THEN\n _dtrange := search_dtrange(_search->'datetime');\n _token_dtrange := _dtrange;\nEND IF;\n\n-- Get the paging token\nIF _search ? 'token' THEN\n token := _search->>'token';\n tok_val := substr(token,6);\n IF starts_with(token, 'prev:') THEN\n is_prev := true;\n END IF;\n SELECT INTO _token_record * FROM items WHERE id=tok_val;\n IF\n (is_prev AND _dtsort = 'DESC')\n OR\n (not is_prev AND _dtsort = 'ASC')\n THEN\n _token_dtrange := tstzrange(_token_record.datetime, 'infinity');\n ELSIF\n _dtsort IS NOT NULL\n THEN\n _token_dtrange := tstzrange('-infinity',_token_record.datetime);\n END IF;\n IF is_prev THEN\n tok_q := filter_by_order(tok_val, _search->'sortby', 'first');\n _sort := _rsort;\n ELSIF starts_with(token, 'next:') THEN\n tok_q := filter_by_order(tok_val, _search->'sortby', 'last');\n END IF;\nEND IF;\nRAISE NOTICE 'timing: %', age(clock_timestamp(), qstart);\nRAISE NOTICE 'tok_q: % _token_dtrange: %', tok_q, _token_dtrange;\n\nIF _search ? 'ids' THEN\n RAISE NOTICE 'searching solely based on ids... %',_search;\n qa := array_append(qa, in_array_q('id', _search->'ids'));\nELSE\n IF _search ? 'intersects' THEN\n _geom := ST_SetSRID(ST_GeomFromGeoJSON(_search->>'intersects'), 4326);\n ELSIF _search ? 'bbox' THEN\n _geom := bbox_geom(_search->'bbox');\n END IF;\n\n IF _geom IS NOT NULL THEN\n qa := array_append(qa, format('st_intersects(geometry, %L::geometry)',_geom));\n END IF;\n\n IF _search ? 'collections' THEN\n qa := array_append(qa, in_array_q('collection_id', _search->'collections'));\n END IF;\n\n IF _search ? 'query' THEN\n qa := array_cat(qa,\n stac_query(_search->'query')\n );\n END IF;\nEND IF;\n\nIF _search ? 'limit' THEN\n _limit := (_search->>'limit')::int;\nEND IF;\n\nIF _search ? 'fields' THEN\n IF _search->'fields' ? 'exclude' THEN\n excludes=textarr(_search->'fields'->'exclude');\n END IF;\n IF _search->'fields' ? 'include' THEN\n includes=textarr(_search->'fields'->'include');\n IF array_length(includes, 1)>0 AND NOT 'id' = ANY (includes) THEN\n includes = includes || '{id}';\n END IF;\n END IF;\n RAISE NOTICE 'Includes: %, Excludes: %', includes, excludes;\nEND IF;\n\nwhereq := COALESCE(array_to_string(qa,' AND '),' TRUE ');\ndq := COALESCE(array_to_string(dqa,' AND '),' TRUE ');\nRAISE NOTICE 'timing before temp table: %', age(clock_timestamp(), qstart);\n\nCREATE TEMP TABLE results_page ON COMMIT DROP AS\nSELECT * FROM items_by_partition(\n concat(whereq, ' AND ', tok_q),\n _token_dtrange,\n _sort,\n _limit + 1\n);\nRAISE NOTICE 'timing after temp table: %', age(clock_timestamp(), qstart);\n\nRAISE NOTICE 'timing before min/max: %', age(clock_timestamp(), qstart);\n\nIF is_prev THEN\n SELECT INTO last_id, first_id, counter\n first_value(id) OVER (),\n last_value(id) OVER (),\n count(*) OVER ()\n FROM results_page;\nELSE\n SELECT INTO first_id, last_id, counter\n first_value(id) OVER (),\n last_value(id) OVER (),\n count(*) OVER ()\n FROM results_page;\nEND IF;\nRAISE NOTICE 'firstid: %, lastid %', first_id, last_id;\nRAISE NOTICE 'timing after min/max: %', age(clock_timestamp(), qstart);\n\n\n\n\nIF counter > _limit THEN\n next_id := last_id;\n RAISE NOTICE 'next_id: %', next_id;\nELSE\n RAISE NOTICE 'No more next';\nEND IF;\n\nIF tok_q = 'TRUE' THEN\n RAISE NOTICE 'Not a paging query, no previous item';\nELSE\n RAISE NOTICE 'Getting previous item id';\n RAISE NOTICE 'timing: %', age(clock_timestamp(), qstart);\n SELECT INTO _token_record * FROM items WHERE id=first_id;\n IF\n _dtsort = 'DESC'\n THEN\n _token_dtrange := _dtrange * tstzrange(_token_record.datetime, 'infinity');\n ELSE\n _token_dtrange := _dtrange * tstzrange('-infinity',_token_record.datetime);\n END IF;\n RAISE NOTICE '% %', _token_dtrange, _dtrange;\n SELECT id INTO prev_id FROM items_by_partition(\n concat(whereq, ' AND ', filter_by_order(first_id, _search->'sortby', 'prev')),\n _token_dtrange,\n _rsort,\n 1\n );\n RAISE NOTICE 'timing: %', age(clock_timestamp(), qstart);\n\n RAISE NOTICE 'prev_id: %', prev_id;\nEND IF;\n\n\nRETURN QUERY\nWITH features AS (\n SELECT filter_jsonb(content, includes, excludes) as content\n FROM results_page LIMIT _limit\n),\nj AS (SELECT jsonb_agg(content) as feature_arr FROM features)\nSELECT jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce (\n CASE WHEN is_prev THEN flip_jsonb_array(feature_arr) ELSE feature_arr END\n ,'[]'::jsonb),\n 'links', links,\n 'timeStamp', now(),\n 'next', next_id,\n 'prev', prev_id\n)\nFROM j\n;\n\n\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\nINSERT INTO migrations (version) VALUES ('0.1.9');\n" }, { "path": "src/pgstac/migrations/pgstac.0.2.3-0.2.4.sql", "content": "SET SEARCH_PATH to pgstac, public;\nINSERT INTO migrations (version) VALUES ('0.2.4');\n" }, { "path": "src/pgstac/migrations/pgstac.0.2.3.sql", "content": "CREATE EXTENSION IF NOT EXISTS postgis;\nCREATE SCHEMA IF NOT EXISTS partman;\nCREATE EXTENSION IF NOT EXISTS pg_partman SCHEMA partman;\nCREATE SCHEMA IF NOT EXISTS pgstac;\n\n\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE TABLE migrations (\n version text,\n datetime timestamptz DEFAULT now() NOT NULL\n);\n\n/* converts a jsonb text array to a pg text[] array */\nCREATE OR REPLACE FUNCTION textarr(_js jsonb)\n RETURNS text[] AS $$\n SELECT ARRAY(SELECT jsonb_array_elements_text(_js));\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\n/*\nconverts a jsonb text array to comma delimited list of identifer quoted\nuseful for constructing column lists for selects\n*/\nCREATE OR REPLACE FUNCTION array_idents(_js jsonb)\n RETURNS text AS $$\n SELECT string_agg(quote_ident(v),',') FROM jsonb_array_elements_text(_js) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\n\n/* looks for a geometry in a stac item first from geometry and falling back to bbox */\nCREATE OR REPLACE FUNCTION stac_geom(value jsonb) RETURNS geometry AS $$\nSELECT\n CASE\n WHEN value->>'geometry' IS NOT NULL THEN\n ST_GeomFromGeoJSON(value->>'geometry')\n WHEN value->>'bbox' IS NOT NULL THEN\n ST_MakeEnvelope(\n (value->'bbox'->>0)::float,\n (value->'bbox'->>1)::float,\n (value->'bbox'->>2)::float,\n (value->'bbox'->>3)::float,\n 4326\n )\n ELSE NULL\n END as geometry\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION stac_datetime(value jsonb) RETURNS timestamptz AS $$\nSELECT (value->'properties'->>'datetime')::timestamptz;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION jsonb_paths (IN jdata jsonb, OUT path text[], OUT value jsonb) RETURNS\nSETOF RECORD AS $$\nwith recursive extract_all as\n(\n select\n ARRAY[key]::text[] as path,\n value\n FROM jsonb_each(jdata)\nunion all\n select\n path || coalesce(obj_key, (arr_key- 1)::text),\n coalesce(obj_value, arr_value)\n from extract_all\n left join lateral\n jsonb_each(case jsonb_typeof(value) when 'object' then value end)\n as o(obj_key, obj_value)\n on jsonb_typeof(value) = 'object'\n left join lateral\n jsonb_array_elements(case jsonb_typeof(value) when 'array' then value end)\n with ordinality as a(arr_value, arr_key)\n on jsonb_typeof(value) = 'array'\n where obj_key is not null or arr_key is not null\n)\nselect *\nfrom extract_all;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION jsonb_obj_paths (IN jdata jsonb, OUT path text[], OUT value jsonb) RETURNS\nSETOF RECORD AS $$\nwith recursive extract_all as\n(\n select\n ARRAY[key]::text[] as path,\n value\n FROM jsonb_each(jdata)\nunion all\n select\n path || obj_key,\n obj_value\n from extract_all\n left join lateral\n jsonb_each(case jsonb_typeof(value) when 'object' then value end)\n as o(obj_key, obj_value)\n on jsonb_typeof(value) = 'object'\n where obj_key is not null\n)\nselect *\nfrom extract_all;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION jsonb_val_paths (IN jdata jsonb, OUT path text[], OUT value jsonb) RETURNS\nSETOF RECORD AS $$\nSELECT * FROM jsonb_obj_paths(jdata) WHERE jsonb_typeof(value) not in ('object','array');\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION path_includes(IN path text[], IN includes text[]) RETURNS BOOLEAN AS $$\nWITH t AS (SELECT unnest(includes) i)\nSELECT EXISTS (\n SELECT 1 FROM t WHERE path @> string_to_array(i, '.')\n);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION path_excludes(IN path text[], IN excludes text[]) RETURNS BOOLEAN AS $$\nWITH t AS (SELECT unnest(excludes) e)\nSELECT NOT EXISTS (\n SELECT 1 FROM t WHERE path @> string_to_array(e, '.')\n);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION jsonb_obj_paths_filtered (\n IN jdata jsonb,\n IN includes text[] DEFAULT ARRAY[]::text[],\n IN excludes text[] DEFAULT ARRAY[]::text[],\n OUT path text[],\n OUT value jsonb\n) RETURNS\nSETOF RECORD AS $$\nSELECT path, value\nFROM jsonb_obj_paths(jdata)\nWHERE\n CASE WHEN cardinality(includes) > 0 THEN path_includes(path, includes) ELSE TRUE END\n AND\n path_excludes(path, excludes)\n\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION empty_arr(ANYARRAY) RETURNS BOOLEAN AS $$\nSELECT CASE\n WHEN $1 IS NULL THEN TRUE\n WHEN cardinality($1)<1 THEN TRUE\nELSE FALSE\nEND;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION filter_jsonb(\n IN jdata jsonb,\n IN includes text[] DEFAULT ARRAY[]::text[],\n IN excludes text[] DEFAULT ARRAY[]::text[]\n) RETURNS jsonb AS $$\nDECLARE\nrec RECORD;\noutj jsonb := '{}'::jsonb;\ncreated_paths text[] := '{}'::text[];\nBEGIN\n\nIF empty_arr(includes) AND empty_arr(excludes) THEN\nRAISE NOTICE 'no filter';\n RETURN jdata;\nEND IF;\nFOR rec in\nSELECT * FROM jsonb_obj_paths_filtered(jdata, includes, excludes)\nWHERE jsonb_typeof(value) != 'object'\nLOOP\n RAISE NOTICE 'path % val %', rec.path, rec.value;\n IF array_length(rec.path,1)>1 THEN\n FOR i IN 1..(array_length(rec.path,1)-1) LOOP\n IF NOT array_to_string(rec.path[1:i],'.') = ANY (created_paths) THEN\n outj := jsonb_set(outj, rec.path[1:i],'{}', true);\n created_paths := created_paths || array_to_string(rec.path[1:i],'.');\n END IF;\n END LOOP;\n END IF;\n outj := jsonb_set(outj, rec.path, rec.value, true);\n created_paths := created_paths || array_to_string(rec.path,'.');\nEND LOOP;\nRETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION properties_idx(_in jsonb) RETURNS jsonb AS $$\nWITH t AS (\n select array_to_string(path,'.') as path, lower(value::text)::jsonb as lowerval\n FROM jsonb_val_paths(_in)\n WHERE array_to_string(path,'.') not in ('datetime')\n)\nSELECT jsonb_object_agg(path, lowerval) FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE TABLE IF NOT EXISTS collections (\n id VARCHAR GENERATED ALWAYS AS (content->>'id') STORED PRIMARY KEY,\n content JSONB\n);\n\nCREATE OR REPLACE FUNCTION create_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection(data jsonb) RETURNS VOID AS $$\nDECLARE\nout collections%ROWTYPE;\nBEGIN\n UPDATE collections SET content=data WHERE id = data->>'id' RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ON CONFLICT (id) DO\n UPDATE\n SET content=EXCLUDED.content\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION delete_collection(_id text) RETURNS VOID AS $$\nDECLARE\nout collections%ROWTYPE;\nBEGIN\n DELETE FROM collections WHERE id = _id RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION get_collection(id text) RETURNS jsonb AS $$\nSELECT content FROM collections\nWHERE id=$1\n;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION all_collections() RETURNS jsonb AS $$\nSELECT jsonb_agg(content) FROM collections;\n;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\n\n\n\n\n\n/* CREATE OR REPLACE FUNCTION collections_trigger_func()\nRETURNS TRIGGER AS $$\nBEGIN\n IF pg_trigger_depth() = 1 THEN\n PERFORM create_collection(NEW.content);\n RETURN NULL;\n END IF;\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE TRIGGER collections_trigger\nBEFORE INSERT ON collections\nFOR EACH ROW EXECUTE PROCEDURE collections_trigger_func();\n */\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE TABLE IF NOT EXISTS items (\n id VARCHAR GENERATED ALWAYS AS (content->>'id') STORED NOT NULL,\n geometry geometry GENERATED ALWAYS AS (stac_geom(content)) STORED NOT NULL,\n properties jsonb GENERATED ALWAYS as (properties_idx(content->'properties')) STORED,\n collection_id text GENERATED ALWAYS AS (content->>'collection') STORED NOT NULL,\n datetime timestamptz GENERATED ALWAYS AS (stac_datetime(content)) STORED NOT NULL,\n content JSONB NOT NULL\n)\nPARTITION BY RANGE (stac_datetime(content))\n;\n\nALTER TABLE items ADD constraint items_collections_fk FOREIGN KEY (collection_id) REFERENCES collections(id) DEFERRABLE;\n\nCREATE TABLE items_template (\n LIKE items\n);\n\nALTER TABLE items_template ADD PRIMARY KEY (id);\n\n/*\nCREATE TABLE IF NOT EXISTS items_search (\n id text NOT NULL,\n geometry geometry NOT NULL,\n properties jsonb,\n collection_id text NOT NULL,\n datetime timestamptz NOT NULL\n)\nPARTITION BY RANGE (datetime)\n;\n\nCREATE TABLE IF NOT EXISTS items_search_template (\n LIKE items_search\n)\n;\nALTER TABLE items_search_template ADD PRIMARY KEY (id);\n*/\n\nDELETE from partman.part_config WHERE parent_table = 'pgstac.items';\nSELECT partman.create_parent(\n 'pgstac.items',\n 'datetime',\n 'native',\n 'weekly',\n p_template_table := 'pgstac.items_template',\n p_premake := 4\n);\n\nCREATE OR REPLACE FUNCTION make_partitions(st timestamptz, et timestamptz DEFAULT NULL) RETURNS BOOL AS $$\nWITH t AS (\n SELECT\n generate_series(\n date_trunc('week',st),\n date_trunc('week', coalesce(et, st)),\n '1 week'::interval\n ) w\n),\nw AS (SELECT array_agg(w) as w FROM t)\nSELECT CASE WHEN w IS NULL THEN NULL ELSE partman.create_partition_time('pgstac.items', w, true) END FROM w;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_partition(timestamptz) RETURNS text AS $$\nSELECT to_char($1, '\"items_p\"IYYY\"w\"IW');\n$$ LANGUAGE SQL;\n\nCREATE INDEX \"datetime_id_idx\" ON items (datetime, id);\nCREATE INDEX \"properties_idx\" ON items USING GIN (properties);\nCREATE INDEX \"collection_idx\" ON items (collection_id);\nCREATE INDEX \"geometry_idx\" ON items USING GIST (geometry);\n\n\nCREATE TYPE item AS (\n id text,\n geometry geometry,\n properties JSONB,\n collection_id text,\n datetime timestamptz\n);\n\n\n/*\nConverts single feature into an items row\n*/\n\n/*\nCREATE OR REPLACE FUNCTION feature_to_item(value jsonb) RETURNS item AS $$\n SELECT\n value->>'id' as id,\n CASE\n WHEN value->>'geometry' IS NOT NULL THEN\n ST_GeomFromGeoJSON(value->>'geometry')\n WHEN value->>'bbox' IS NOT NULL THEN\n ST_MakeEnvelope(\n (value->'bbox'->>0)::float,\n (value->'bbox'->>1)::float,\n (value->'bbox'->>2)::float,\n (value->'bbox'->>3)::float,\n 4326\n )\n ELSE NULL\n END as geometry,\n properties_idx(value ->'properties') as properties,\n value->>'collection' as collection_id,\n (value->'properties'->>'datetime')::timestamptz as datetime\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac,public;\n*/\n/*\nTakes a single feature, an array of features, or a feature collection\nand returns a set up individual items rows\n*/\n/*\nCREATE OR REPLACE FUNCTION features_to_items(value jsonb) RETURNS SETOF item AS $$\n WITH features AS (\n SELECT\n jsonb_array_elements(\n CASE\n WHEN jsonb_typeof(value) = 'array' THEN value\n WHEN value->>'type' = 'Feature' THEN '[]'::jsonb || value\n WHEN value->>'type' = 'FeatureCollection' THEN value->'features'\n ELSE NULL\n END\n ) as value\n )\n SELECT feature_to_item(value) FROM features\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac,public;\n*/\n\nCREATE OR REPLACE FUNCTION get_item(_id text) RETURNS jsonb AS $$\n SELECT content FROM items WHERE id=_id;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION delete_item(_id text) RETURNS VOID AS $$\nDECLARE\nout items%ROWTYPE;\nBEGIN\n DELETE FROM items WHERE id = _id RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION create_item(data jsonb) RETURNS VOID AS $$\n SELECT make_partitions(stac_datetime(data));\n INSERT INTO items (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION update_item(data jsonb) RETURNS VOID AS $$\nDECLARE\nout items%ROWTYPE;\nBEGIN\n UPDATE items SET content=data WHERE id = data->>'id' RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_item(data jsonb) RETURNS VOID AS $$\nDECLARE\npartition text;\nq text;\nnewcontent jsonb;\nBEGIN\n PERFORM make_partitions(stac_datetime(data));\n partition := get_partition(stac_datetime(data));\n q := format($q$\n INSERT INTO %I (content) VALUES ($1)\n ON CONFLICT (id) DO\n UPDATE SET content = EXCLUDED.content\n WHERE %I.content IS DISTINCT FROM EXCLUDED.content RETURNING content;\n $q$, partition, partition);\n EXECUTE q INTO newcontent USING (data);\n RAISE NOTICE 'newcontent: %', newcontent;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION analyze_empty_partitions() RETURNS VOID AS $$\nDECLARE\np text;\nBEGIN\nFOR p IN SELECT partition FROM all_items_partitions WHERE est_cnt = 0 LOOP\n RAISE NOTICE 'Analyzing %', p;\n EXECUTE format('ANALYZE %I;', p);\nEND LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n/* Trigger Function to cascade inserts/updates/deletes\nfrom items table to items_search table */\n/*\nALTER TABLE items_search ADD CONSTRAINT items_search_fk\nFOREIGN KEY (id) REFERENCES items(id)\nON DELETE CASCADE DEFERRABLE;\n\nCREATE OR REPLACE FUNCTION items_trigger_func()\nRETURNS TRIGGER AS $$\nDECLARE\nBEGIN\n IF TG_OP = 'UPDATE' THEN\n RAISE NOTICE 'DELETING % BEFORE UPDATE', OLD;\n DELETE FROM items_search WHERE id = OLD.id AND datetime = (OLD.content->'properties'->>'datetime')::timestamptz;\n END IF;\n\n INSERT INTO items_search SELECT * FROM feature_to_item(NEW.content);\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nDROP TRIGGER IF EXISTS items_insert_trigger ON items;\nCREATE TRIGGER items_insert_trigger\nAFTER INSERT ON items\nFOR EACH ROW EXECUTE PROCEDURE items_trigger_func();\n\nDROP TRIGGER IF EXISTS items_update_trigger ON items;\nCREATE TRIGGER items_update_trigger\nAFTER UPDATE ON items\nFOR EACH ROW\nWHEN (NEW.content IS DISTINCT FROM OLD.content)\nEXECUTE PROCEDURE items_trigger_func();\n*/\n\n/* Trigger Function to cascade inserts/updates/deletes\nfrom items table to items_search table */\n/*\nCREATE OR REPLACE FUNCTION items_search_trigger_delete_func()\nRETURNS TRIGGER AS $$\nDECLARE\nBEGIN\n RAISE NOTICE 'Deleting from items_search: % Depth: %', OLD, pg_trigger_depth();\n IF pg_trigger_depth()<3 THEN\n RAISE NOTICE 'DELETING WITH datetime';\n DELETE FROM items_search WHERE id=OLD.id AND datetime=OLD.datetime;\n RETURN NULL;\n END IF;\n RETURN OLD;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nDROP TRIGGER IF EXISTS items_search_delete_trigger ON items_search;\nCREATE TRIGGER items_search_delete_trigger\nBEFORE DELETE ON items_search\nFOR EACH ROW EXECUTE PROCEDURE items_search_trigger_delete_func();\n*/\nCREATE OR REPLACE FUNCTION backfill_partitions()\nRETURNS VOID AS $$\nDECLARE\nBEGIN\n IF EXISTS (SELECT 1 FROM items_default LIMIT 1) THEN\n RAISE NOTICE 'Creating new partitions and moving data from default';\n CREATE TEMP TABLE items_default_tmp ON COMMIT DROP AS SELECT datetime, content FROM items_default;\n TRUNCATE items_default;\n PERFORM make_partitions(min(datetime), max(datetime)) FROM items_default_tmp;\n INSERT INTO items (content) SELECT content FROM items_default_tmp;\n END IF;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION items_trigger_stmt_func()\nRETURNS TRIGGER AS $$\nDECLARE\nBEGIN\n PERFORM analyze_empty_partitions();\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nDROP TRIGGER IF EXISTS items_stmt_trigger ON items;\nCREATE TRIGGER items_stmt_trigger\nAFTER INSERT OR UPDATE OR DELETE ON items\nFOR EACH STATEMENT EXECUTE PROCEDURE items_trigger_stmt_func();\n\n\n/*\nView to get a table of available items partitions\nwith date ranges\n*/\n--DROP VIEW IF EXISTS all_items_partitions CASCADE;\nCREATE OR REPLACE VIEW all_items_partitions AS\nWITH base AS\n(SELECT\n c.oid::pg_catalog.regclass::text as partition,\n pg_catalog.pg_get_expr(c.relpartbound, c.oid) as _constraint,\n regexp_matches(\n pg_catalog.pg_get_expr(c.relpartbound, c.oid),\n E'\\\\(''\\([0-9 :+-]*\\)''\\\\).*\\\\(''\\([0-9 :+-]*\\)''\\\\)'\n ) as t,\n reltuples::bigint as est_cnt\nFROM pg_catalog.pg_class c, pg_catalog.pg_inherits i\nWHERE c.oid = i.inhrelid AND i.inhparent = 'items'::regclass)\nSELECT partition, tstzrange(\n t[1]::timestamptz,\n t[2]::timestamptz\n), est_cnt\nFROM base\nORDER BY 2 desc;\n\n--DROP VIEW IF EXISTS items_partitions;\nCREATE OR REPLACE VIEW items_partitions AS\nSELECT * FROM all_items_partitions WHERE est_cnt>0;\n\nCREATE OR REPLACE FUNCTION collection_bbox(id text) RETURNS jsonb AS $$\nSELECT (replace(replace(replace(st_extent(geometry)::text,'BOX(','[['),')',']]'),' ',','))::jsonb\nFROM items WHERE collection_id=$1;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION collection_temporal_extent(id text) RETURNS jsonb AS $$\nSELECT to_jsonb(array[array[min(datetime)::text, max(datetime)::text]])\nFROM items WHERE collection_id=$1;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection_extents() RETURNS VOID AS $$\nUPDATE collections SET\n content = content ||\n jsonb_build_object(\n 'extent', jsonb_build_object(\n 'spatial', jsonb_build_object(\n 'bbox', collection_bbox(collections.id)\n ),\n 'temporal', jsonb_build_object(\n 'interval', collection_temporal_extent(collections.id)\n )\n )\n )\n;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\nSET SEARCH_PATH TO pgstac, public;\n\n/*\nView to get a table of available items partitions\nwith date ranges\n*/\nDROP VIEW IF EXISTS items_partitions;\nCREATE VIEW items_partitions AS\nWITH base AS\n(SELECT\n c.oid::pg_catalog.regclass::text as partition,\n pg_catalog.pg_get_expr(c.relpartbound, c.oid) as _constraint,\n regexp_matches(\n pg_catalog.pg_get_expr(c.relpartbound, c.oid),\n E'\\\\(''\\([0-9 :+-]*\\)''\\\\).*\\\\(''\\([0-9 :+-]*\\)''\\\\)'\n ) as t,\n reltuples::bigint as est_cnt\nFROM pg_catalog.pg_class c, pg_catalog.pg_inherits i\nWHERE c.oid = i.inhrelid AND i.inhparent = 'items'::regclass)\nSELECT partition, tstzrange(\n t[1]::timestamptz,\n t[2]::timestamptz\n), est_cnt\nFROM base\nWHERE est_cnt >0\nORDER BY 2 desc;\n\n\nCREATE OR REPLACE FUNCTION items_by_partition(\n IN _where text DEFAULT 'TRUE',\n IN _dtrange tstzrange DEFAULT tstzrange('-infinity','infinity'),\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN _limit int DEFAULT 10\n) RETURNS SETOF items AS $$\nDECLARE\npartition_query text;\nmain_query text;\nbatchcount int;\ncounter int := 0;\np record;\nBEGIN\nIF _orderby ILIKE 'datetime d%' THEN\n partition_query := format($q$\n SELECT partition\n FROM items_partitions\n WHERE tstzrange && $1\n ORDER BY tstzrange DESC;\n $q$);\nELSIF _orderby ILIKE 'datetime a%' THEN\n partition_query := format($q$\n SELECT partition\n FROM items_partitions\n WHERE tstzrange && $1\n ORDER BY tstzrange ASC\n ;\n $q$);\nELSE\n partition_query := format($q$\n SELECT 'items' as partition WHERE $1 IS NOT NULL;\n $q$);\nEND IF;\nRAISE NOTICE 'Partition Query: %', partition_query;\nFOR p IN\n EXECUTE partition_query USING (_dtrange)\nLOOP\n IF lower(_dtrange)::timestamptz > '-infinity' THEN\n _where := concat(_where,format(' AND datetime >= %L',lower(_dtrange)::timestamptz::text));\n END IF;\n IF upper(_dtrange)::timestamptz < 'infinity' THEN\n _where := concat(_where,format(' AND datetime <= %L',upper(_dtrange)::timestamptz::text));\n END IF;\n\n main_query := format($q$\n SELECT * FROM %I\n WHERE %s\n ORDER BY %s\n LIMIT %s - $1\n $q$, p.partition::text, _where, _orderby, _limit\n );\n RAISE NOTICE 'Partition Query %', main_query;\n RAISE NOTICE '%', counter;\n RETURN QUERY EXECUTE main_query USING counter;\n\n GET DIAGNOSTICS batchcount = ROW_COUNT;\n counter := counter + batchcount;\n RAISE NOTICE 'FOUND %', batchcount;\n IF counter >= _limit THEN\n EXIT;\n END IF;\n RAISE NOTICE 'ADDED % FOR A TOTAL OF %', batchcount, counter;\nEND LOOP;\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION split_stac_path(IN path text, OUT col text, OUT dotpath text, OUT jspath text, OUT jspathtext text) AS $$\nWITH col AS (\n SELECT\n CASE WHEN\n split_part(path, '.', 1) IN ('id', 'stac_version', 'stac_extensions','geometry','properties','assets','collection_id','datetime','links', 'extra_fields') THEN split_part(path, '.', 1)\n ELSE 'properties'\n END AS col\n),\ndp AS (\n SELECT\n col, ltrim(replace(path, col , ''),'.') as dotpath\n FROM col\n),\npaths AS (\nSELECT\n col, dotpath,\n regexp_split_to_table(dotpath,E'\\\\.') as path FROM dp\n) SELECT\n col,\n btrim(concat(col,'.',dotpath),'.'),\n CASE WHEN btrim(concat(col,'.',dotpath),'.') != col THEN concat(col,'->',string_agg(concat('''',path,''''),'->')) ELSE col END,\n regexp_replace(\n CASE WHEN btrim(concat(col,'.',dotpath),'.') != col THEN concat(col,'->',string_agg(concat('''',path,''''),'->')) ELSE col END,\n E'>([^>]*)$','>>\\1'\n )\nFROM paths group by col, dotpath;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n/* Functions for searching items */\nCREATE OR REPLACE FUNCTION sort_base(\n IN _sort jsonb DEFAULT '[{\"field\":\"datetime\",\"direction\":\"desc\"}]',\n OUT key text,\n OUT col text,\n OUT dir text,\n OUT rdir text,\n OUT sort text,\n OUT rsort text\n) RETURNS SETOF RECORD AS $$\nWITH sorts AS (\n SELECT\n value->>'field' as key,\n (split_stac_path(value->>'field')).jspathtext as col,\n coalesce(upper(value->>'direction'),'ASC') as dir\n FROM jsonb_array_elements('[]'::jsonb || coalesce(_sort,'[{\"field\":\"datetime\",\"direction\":\"desc\"}]') )\n)\nSELECT\n key,\n col,\n dir,\n CASE dir WHEN 'DESC' THEN 'ASC' ELSE 'ASC' END as rdir,\n concat(col, ' ', dir, ' NULLS LAST ') AS sort,\n concat(col,' ', CASE dir WHEN 'DESC' THEN 'ASC' ELSE 'ASC' END, ' NULLS LAST ') AS rsort\nFROM sorts\nUNION ALL\nSELECT 'id', 'id', 'DESC', 'ASC', 'id DESC', 'id ASC'\n;\n$$ LANGUAGE SQL;\n\n\nCREATE OR REPLACE FUNCTION sort(_sort jsonb) RETURNS text AS $$\nSELECT string_agg(sort,', ') FROM sort_base(_sort);\n$$ LANGUAGE SQL PARALLEL SAFE SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION rsort(_sort jsonb) RETURNS text AS $$\nSELECT string_agg(rsort,', ') FROM sort_base(_sort);\n$$ LANGUAGE SQL PARALLEL SAFE SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION bbox_geom(_bbox jsonb) RETURNS box3d AS $$\nSELECT CASE jsonb_array_length(_bbox)\n WHEN 4 THEN\n ST_SetSRID(ST_MakeEnvelope(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float,\n (_bbox->>3)::float\n ),4326)\n WHEN 6 THEN\n ST_SetSRID(ST_3DMakeBox(\n ST_MakePoint(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float\n ),\n ST_MakePoint(\n (_bbox->>3)::float,\n (_bbox->>4)::float,\n (_bbox->>5)::float\n )\n ),4326)\n ELSE null END;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION in_array_q(col text, arr jsonb) RETURNS text AS $$\nSELECT CASE jsonb_typeof(arr) WHEN 'array' THEN format('%I = ANY(textarr(%L))', col, arr) ELSE format('%I = %L', col, arr) END;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION count_by_delim(text, text) RETURNS int AS $$\nSELECT count(*) FROM regexp_split_to_table($1,$2);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION stac_query_op(att text, _op text, val jsonb) RETURNS text AS $$\nDECLARE\nret text := '';\nop text;\njp text;\natt_parts RECORD;\nval_str text;\nBEGIN\nval_str := lower(jsonb_build_object('a',val)->>'a');\nRAISE NOTICE 'val_str %', val_str;\n\natt_parts := split_stac_path(att);\n\nop := CASE _op\n WHEN 'eq' THEN '='\n WHEN 'ge' THEN '>='\n WHEN 'gt' THEN '>'\n WHEN 'le' THEN '<='\n WHEN 'lt' THEN '<'\n WHEN 'ne' THEN '!='\n WHEN 'neq' THEN '!='\n WHEN 'startsWith' THEN 'LIKE'\n WHEN 'endsWith' THEN 'LIKE'\n WHEN 'contains' THEN 'LIKE'\n ELSE _op\nEND;\n\nval_str := CASE _op\n WHEN 'startsWith' THEN concat(val_str, '%')\n WHEN 'endsWith' THEN concat('%', val_str)\n WHEN 'contains' THEN concat('%',val_str,'%')\n ELSE val_str\nEND;\n\n\nRAISE NOTICE 'att_parts: % %', att_parts, count_by_delim(att_parts.dotpath,'\\.');\nIF\n op = '='\n AND att_parts.col = 'properties'\n --AND count_by_delim(att_parts.dotpath,'\\.') = 2\nTHEN\n -- use jsonpath query to leverage index for eqaulity tests on single level deep properties\n jp := btrim(format($jp$ $.%I[*] ? ( @ == %s ) $jp$, replace(att_parts.dotpath, 'properties.',''), lower(val::text)::jsonb));\n raise notice 'jp: %', jp;\n ret := format($q$ properties @? %L $q$, jp);\nELSIF jsonb_typeof(val) = 'number' THEN\n ret := format('(%s)::numeric %s %s', att_parts.jspathtext, op, val);\nELSE\n ret := format('%s %s %L', att_parts.jspathtext, op, val_str);\nEND IF;\nRAISE NOTICE 'Op Query: %', ret;\n\nreturn ret;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION stac_query(_query jsonb) RETURNS TEXT[] AS $$\nDECLARE\nqa text[];\natt text;\nops jsonb;\nop text;\nval jsonb;\nBEGIN\nFOR att, ops IN SELECT key, value FROM jsonb_each(_query)\nLOOP\n FOR op, val IN SELECT key, value FROM jsonb_each(ops)\n LOOP\n qa := array_append(qa, stac_query_op(att,op, val));\n RAISE NOTICE '% % %', att, op, val;\n END LOOP;\nEND LOOP;\nRETURN qa;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION filter_by_order(item_id text, _sort jsonb, _type text) RETURNS text AS $$\nDECLARE\nitem item;\nBEGIN\nSELECT * INTO item FROM items WHERE id=item_id;\nRETURN filter_by_order(item, _sort, _type);\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n-- Used to create filters used for paging using the items id from the token\nCREATE OR REPLACE FUNCTION filter_by_order(_item item, _sort jsonb, _type text) RETURNS text AS $$\nDECLARE\nsorts RECORD;\nfilts text[];\nitemval text;\nop text;\nidop text;\nret text;\neq_flag text;\n_item_j jsonb := to_jsonb(_item);\nBEGIN\nFOR sorts IN SELECT * FROM sort_base(_sort) LOOP\n IF sorts.col = 'datetime' THEN\n CONTINUE;\n END IF;\n IF sorts.col='id' AND _type IN ('prev','next') THEN\n eq_flag := '';\n ELSE\n eq_flag := '=';\n END IF;\n\n op := concat(\n CASE\n WHEN _type in ('prev','first') AND sorts.dir = 'ASC' THEN '<'\n WHEN _type in ('last','next') AND sorts.dir = 'ASC' THEN '>'\n WHEN _type in ('prev','first') AND sorts.dir = 'DESC' THEN '>'\n WHEN _type in ('last','next') AND sorts.dir = 'DESC' THEN '<'\n END,\n eq_flag\n );\n\n IF _item_j ? sorts.col THEN\n filts = array_append(filts, format('%s %s %L', sorts.col, op, _item_j->>sorts.col));\n END IF;\nEND LOOP;\nret := coalesce(array_to_string(filts,' AND '), 'TRUE');\nRAISE NOTICE 'Order Filter %', ret;\nRETURN ret;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION search_dtrange(IN _indate jsonb, OUT _tstzrange tstzrange) AS\n$$\nWITH t AS (\n SELECT CASE\n WHEN jsonb_typeof(_indate) = 'array' THEN\n textarr(_indate)\n ELSE\n regexp_split_to_array(\n btrim(_indate::text,'\"'),\n '/'\n )\n END AS arr\n)\n, t1 AS (\n SELECT\n CASE\n WHEN array_upper(arr,1) = 1 OR arr[1] = '..' OR arr[1] IS NULL THEN '-infinity'::timestamptz\n ELSE arr[1]::timestamptz\n END AS st,\n CASE\n WHEN array_upper(arr,1) = 1 THEN arr[1]::timestamptz\n WHEN arr[2] = '..' OR arr[2] IS NULL THEN 'infinity'::timestamptz\n ELSE arr[2]::timestamptz\n END AS et\n FROM t\n)\nSELECT\n tstzrange(st,et)\nFROM t1;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION flip_jsonb_array(j jsonb) RETURNS jsonb AS $$\nWITH t AS (\n SELECT i, row_number() over () as r FROM jsonb_array_elements(j) i\n), o AS (\n SELECT i FROM t ORDER BY r DESC\n)\nSELECT jsonb_agg(i) from o\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION search(_search jsonb = '{}'::jsonb) RETURNS SETOF jsonb AS $$\nDECLARE\nqstart timestamptz := clock_timestamp();\n_sort text := '';\n_rsort text := '';\n_limit int := 10;\n_geom geometry;\nqa text[];\npq text[];\nquery text;\npq_prop record;\npq_op record;\nprev_id text := NULL;\nnext_id text := NULL;\nwhereq text := 'TRUE';\nlinks jsonb := '[]'::jsonb;\ntoken text;\ntok_val text;\ntok_q text := 'TRUE';\ntok_sort text;\nfirst_id text;\nfirst_dt timestamptz;\nlast_id text;\nsort text;\nrsort text;\ndt text[];\ndqa text[];\ndq text;\nmq_where text;\nstartdt timestamptz;\nenddt timestamptz;\nitem items%ROWTYPE;\ncounter int := 0;\nbatchcount int;\nmonth timestamptz;\nm record;\n_dtrange tstzrange := tstzrange('-infinity','infinity');\n_dtsort text;\n_token_dtrange tstzrange := tstzrange('-infinity','infinity');\n_token_record items%ROWTYPE;\nis_prev boolean := false;\nincludes text[];\nexcludes text[];\nBEGIN\n-- Create table from sort query of items to sort\nCREATE TEMP TABLE pgstac_tmp_sorts ON COMMIT DROP AS SELECT * FROM sort_base(_search->'sortby');\n\n-- Get the datetime sort direction, necessary for efficient cycling through partitions\nSELECT INTO _dtsort dir FROM pgstac_tmp_sorts WHERE key='datetime';\nRAISE NOTICE '_dtsort: %',_dtsort;\n\nSELECT INTO _sort string_agg(s.sort,', ') FROM pgstac_tmp_sorts s;\nSELECT INTO _rsort string_agg(s.rsort,', ') FROM pgstac_tmp_sorts s;\ntok_sort := _sort;\n\n\n-- Get datetime from query as a tstzrange\nIF _search ? 'datetime' THEN\n _dtrange := search_dtrange(_search->'datetime');\n _token_dtrange := _dtrange;\nEND IF;\n\n-- Get the paging token\nIF _search ? 'token' THEN\n token := _search->>'token';\n tok_val := substr(token,6);\n IF starts_with(token, 'prev:') THEN\n is_prev := true;\n END IF;\n SELECT INTO _token_record * FROM items WHERE id=tok_val;\n IF\n (is_prev AND _dtsort = 'DESC')\n OR\n (not is_prev AND _dtsort = 'ASC')\n THEN\n _token_dtrange := tstzrange(_token_record.datetime, 'infinity');\n ELSIF\n _dtsort IS NOT NULL\n THEN\n _token_dtrange := tstzrange('-infinity',_token_record.datetime);\n END IF;\n IF is_prev THEN\n tok_q := filter_by_order(tok_val, _search->'sortby', 'first');\n _sort := _rsort;\n ELSIF starts_with(token, 'next:') THEN\n tok_q := filter_by_order(tok_val, _search->'sortby', 'last');\n END IF;\nEND IF;\nRAISE NOTICE 'timing: %', age(clock_timestamp(), qstart);\nRAISE NOTICE 'tok_q: % _token_dtrange: %', tok_q, _token_dtrange;\n\nIF _search ? 'ids' THEN\n RAISE NOTICE 'searching solely based on ids... %',_search;\n qa := array_append(qa, in_array_q('id', _search->'ids'));\nELSE\n IF _search ? 'intersects' THEN\n _geom := ST_SetSRID(ST_GeomFromGeoJSON(_search->>'intersects'), 4326);\n ELSIF _search ? 'bbox' THEN\n _geom := bbox_geom(_search->'bbox');\n END IF;\n\n IF _geom IS NOT NULL THEN\n qa := array_append(qa, format('st_intersects(geometry, %L::geometry)',_geom));\n END IF;\n\n IF _search ? 'collections' THEN\n qa := array_append(qa, in_array_q('collection_id', _search->'collections'));\n END IF;\n\n IF _search ? 'query' THEN\n qa := array_cat(qa,\n stac_query(_search->'query')\n );\n END IF;\nEND IF;\n\nIF _search ? 'limit' THEN\n _limit := (_search->>'limit')::int;\nEND IF;\n\nIF _search ? 'fields' THEN\n IF _search->'fields' ? 'exclude' THEN\n excludes=textarr(_search->'fields'->'exclude');\n END IF;\n IF _search->'fields' ? 'include' THEN\n includes=textarr(_search->'fields'->'include');\n IF array_length(includes, 1)>0 AND NOT 'id' = ANY (includes) THEN\n includes = includes || '{id}';\n END IF;\n END IF;\n RAISE NOTICE 'Includes: %, Excludes: %', includes, excludes;\nEND IF;\n\nwhereq := COALESCE(array_to_string(qa,' AND '),' TRUE ');\ndq := COALESCE(array_to_string(dqa,' AND '),' TRUE ');\nRAISE NOTICE 'timing before temp table: %', age(clock_timestamp(), qstart);\n\nCREATE TEMP TABLE results_page ON COMMIT DROP AS\nSELECT * FROM items_by_partition(\n concat(whereq, ' AND ', tok_q),\n _token_dtrange,\n _sort,\n _limit + 1\n);\nRAISE NOTICE 'timing after temp table: %', age(clock_timestamp(), qstart);\n\nRAISE NOTICE 'timing before min/max: %', age(clock_timestamp(), qstart);\n\nIF is_prev THEN\n SELECT INTO last_id, first_id, counter\n first_value(id) OVER (),\n last_value(id) OVER (),\n count(*) OVER ()\n FROM results_page;\nELSE\n SELECT INTO first_id, last_id, counter\n first_value(id) OVER (),\n last_value(id) OVER (),\n count(*) OVER ()\n FROM results_page;\nEND IF;\nRAISE NOTICE 'firstid: %, lastid %', first_id, last_id;\nRAISE NOTICE 'timing after min/max: %', age(clock_timestamp(), qstart);\n\n\n\n\nIF counter > _limit THEN\n next_id := last_id;\n RAISE NOTICE 'next_id: %', next_id;\nELSE\n RAISE NOTICE 'No more next';\nEND IF;\n\nIF tok_q = 'TRUE' THEN\n RAISE NOTICE 'Not a paging query, no previous item';\nELSE\n RAISE NOTICE 'Getting previous item id';\n RAISE NOTICE 'timing: %', age(clock_timestamp(), qstart);\n SELECT INTO _token_record * FROM items WHERE id=first_id;\n IF\n _dtsort = 'DESC'\n THEN\n _token_dtrange := _dtrange * tstzrange(_token_record.datetime, 'infinity');\n ELSE\n _token_dtrange := _dtrange * tstzrange('-infinity',_token_record.datetime);\n END IF;\n RAISE NOTICE '% %', _token_dtrange, _dtrange;\n SELECT id INTO prev_id FROM items_by_partition(\n concat(whereq, ' AND ', filter_by_order(first_id, _search->'sortby', 'prev')),\n _token_dtrange,\n _rsort,\n 1\n );\n RAISE NOTICE 'timing: %', age(clock_timestamp(), qstart);\n\n RAISE NOTICE 'prev_id: %', prev_id;\nEND IF;\n\n\nRETURN QUERY\nWITH features AS (\n SELECT filter_jsonb(content, includes, excludes) as content\n FROM results_page LIMIT _limit\n),\nj AS (SELECT jsonb_agg(content) as feature_arr FROM features)\nSELECT jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce (\n CASE WHEN is_prev THEN flip_jsonb_array(feature_arr) ELSE feature_arr END\n ,'[]'::jsonb),\n 'links', links,\n 'timeStamp', now(),\n 'next', next_id,\n 'prev', prev_id\n)\nFROM j\n;\n\n\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\nINSERT INTO migrations (version) VALUES ('0.2.3');\n" }, { "path": "src/pgstac/migrations/pgstac.0.2.4-0.2.5.sql", "content": "SET SEARCH_PATH TO pgstac, public;\nBEGIN;\nDROP FUNCTION IF EXISTS bbox_geom;\nCREATE OR REPLACE FUNCTION bbox_geom(_bbox jsonb) RETURNS geometry AS $$\nSELECT CASE jsonb_array_length(_bbox)\n WHEN 4 THEN\n ST_SetSRID(ST_MakeEnvelope(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float,\n (_bbox->>3)::float\n ),4326)\n WHEN 6 THEN\n ST_SetSRID(ST_3DMakeBox(\n ST_MakePoint(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float\n ),\n ST_MakePoint(\n (_bbox->>3)::float,\n (_bbox->>4)::float,\n (_bbox->>5)::float\n )\n ),4326)\n ELSE null END;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION search(_search jsonb = '{}'::jsonb) RETURNS SETOF jsonb AS $$\nDECLARE\nqstart timestamptz := clock_timestamp();\n_sort text := '';\n_rsort text := '';\n_limit int := 10;\n_geom geometry;\nqa text[];\npq text[];\nquery text;\npq_prop record;\npq_op record;\nprev_id text := NULL;\nnext_id text := NULL;\nwhereq text := 'TRUE';\nlinks jsonb := '[]'::jsonb;\ntoken text;\ntok_val text;\ntok_q text := 'TRUE';\ntok_sort text;\nfirst_id text;\nfirst_dt timestamptz;\nlast_id text;\nsort text;\nrsort text;\ndt text[];\ndqa text[];\ndq text;\nmq_where text;\nstartdt timestamptz;\nenddt timestamptz;\nitem items%ROWTYPE;\ncounter int := 0;\nbatchcount int;\nmonth timestamptz;\nm record;\n_dtrange tstzrange := tstzrange('-infinity','infinity');\n_dtsort text;\n_token_dtrange tstzrange := tstzrange('-infinity','infinity');\n_token_record items%ROWTYPE;\nis_prev boolean := false;\nincludes text[];\nexcludes text[];\nBEGIN\n-- Create table from sort query of items to sort\nCREATE TEMP TABLE pgstac_tmp_sorts ON COMMIT DROP AS SELECT * FROM sort_base(_search->'sortby');\n\n-- Get the datetime sort direction, necessary for efficient cycling through partitions\nSELECT INTO _dtsort dir FROM pgstac_tmp_sorts WHERE key='datetime';\nRAISE NOTICE '_dtsort: %',_dtsort;\n\nSELECT INTO _sort string_agg(s.sort,', ') FROM pgstac_tmp_sorts s;\nSELECT INTO _rsort string_agg(s.rsort,', ') FROM pgstac_tmp_sorts s;\ntok_sort := _sort;\n\n\n-- Get datetime from query as a tstzrange\nIF _search ? 'datetime' THEN\n _dtrange := search_dtrange(_search->'datetime');\n _token_dtrange := _dtrange;\nEND IF;\n\n-- Get the paging token\nIF _search ? 'token' THEN\n token := _search->>'token';\n tok_val := substr(token,6);\n IF starts_with(token, 'prev:') THEN\n is_prev := true;\n END IF;\n SELECT INTO _token_record * FROM items WHERE id=tok_val;\n IF\n (is_prev AND _dtsort = 'DESC')\n OR\n (not is_prev AND _dtsort = 'ASC')\n THEN\n _token_dtrange := _dtrange * tstzrange(_token_record.datetime, 'infinity');\n ELSIF\n _dtsort IS NOT NULL\n THEN\n _token_dtrange := _dtrange * tstzrange('-infinity',_token_record.datetime);\n END IF;\n IF is_prev THEN\n tok_q := filter_by_order(tok_val, _search->'sortby', 'first');\n _sort := _rsort;\n ELSIF starts_with(token, 'next:') THEN\n tok_q := filter_by_order(tok_val, _search->'sortby', 'last');\n END IF;\nEND IF;\nRAISE NOTICE 'timing: %', age(clock_timestamp(), qstart);\nRAISE NOTICE 'tok_q: % _token_dtrange: %', tok_q, _token_dtrange;\n\nIF _search ? 'ids' THEN\n RAISE NOTICE 'searching solely based on ids... %',_search;\n qa := array_append(qa, in_array_q('id', _search->'ids'));\nELSE\n IF _search ? 'intersects' THEN\n _geom := ST_SetSRID(ST_GeomFromGeoJSON(_search->>'intersects'), 4326);\n ELSIF _search ? 'bbox' THEN\n _geom := bbox_geom(_search->'bbox');\n END IF;\n\n IF _geom IS NOT NULL THEN\n qa := array_append(qa, format('st_intersects(geometry, %L::geometry)',_geom));\n END IF;\n\n IF _search ? 'collections' THEN\n qa := array_append(qa, in_array_q('collection_id', _search->'collections'));\n END IF;\n\n IF _search ? 'query' THEN\n qa := array_cat(qa,\n stac_query(_search->'query')\n );\n END IF;\nEND IF;\n\nIF _search ? 'limit' THEN\n _limit := (_search->>'limit')::int;\nEND IF;\n\nIF _search ? 'fields' THEN\n IF _search->'fields' ? 'exclude' THEN\n excludes=textarr(_search->'fields'->'exclude');\n END IF;\n IF _search->'fields' ? 'include' THEN\n includes=textarr(_search->'fields'->'include');\n IF array_length(includes, 1)>0 AND NOT 'id' = ANY (includes) THEN\n includes = includes || '{id}';\n END IF;\n END IF;\n RAISE NOTICE 'Includes: %, Excludes: %', includes, excludes;\nEND IF;\n\nwhereq := COALESCE(array_to_string(qa,' AND '),' TRUE ');\ndq := COALESCE(array_to_string(dqa,' AND '),' TRUE ');\nRAISE NOTICE 'timing before temp table: %', age(clock_timestamp(), qstart);\n\nCREATE TEMP TABLE results_page ON COMMIT DROP AS\nSELECT * FROM items_by_partition(\n concat(whereq, ' AND ', tok_q),\n _token_dtrange,\n _sort,\n _limit + 1\n);\nRAISE NOTICE 'timing after temp table: %', age(clock_timestamp(), qstart);\n\nRAISE NOTICE 'timing before min/max: %', age(clock_timestamp(), qstart);\n\nIF is_prev THEN\n SELECT INTO last_id, first_id, counter\n first_value(id) OVER (),\n last_value(id) OVER (),\n count(*) OVER ()\n FROM results_page;\nELSE\n SELECT INTO first_id, last_id, counter\n first_value(id) OVER (),\n last_value(id) OVER (),\n count(*) OVER ()\n FROM results_page;\nEND IF;\nRAISE NOTICE 'firstid: %, lastid %', first_id, last_id;\nRAISE NOTICE 'timing after min/max: %', age(clock_timestamp(), qstart);\n\n\n\n\nIF counter > _limit THEN\n next_id := last_id;\n RAISE NOTICE 'next_id: %', next_id;\nELSE\n RAISE NOTICE 'No more next';\nEND IF;\n\nIF tok_q = 'TRUE' THEN\n RAISE NOTICE 'Not a paging query, no previous item';\nELSE\n RAISE NOTICE 'Getting previous item id';\n RAISE NOTICE 'timing: %', age(clock_timestamp(), qstart);\n SELECT INTO _token_record * FROM items WHERE id=first_id;\n IF\n _dtsort = 'DESC'\n THEN\n _token_dtrange := _dtrange * tstzrange(_token_record.datetime, 'infinity');\n ELSE\n _token_dtrange := _dtrange * tstzrange('-infinity',_token_record.datetime);\n END IF;\n RAISE NOTICE '% %', _token_dtrange, _dtrange;\n SELECT id INTO prev_id FROM items_by_partition(\n concat(whereq, ' AND ', filter_by_order(first_id, _search->'sortby', 'prev')),\n _token_dtrange,\n _rsort,\n 1\n );\n RAISE NOTICE 'timing: %', age(clock_timestamp(), qstart);\n\n RAISE NOTICE 'prev_id: %', prev_id;\nEND IF;\n\n\nRETURN QUERY\nWITH features AS (\n SELECT filter_jsonb(content, includes, excludes) as content\n FROM results_page LIMIT _limit\n),\nj AS (SELECT jsonb_agg(content) as feature_arr FROM features)\nSELECT jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce (\n CASE WHEN is_prev THEN flip_jsonb_array(feature_arr) ELSE feature_arr END\n ,'[]'::jsonb),\n 'links', links,\n 'timeStamp', now(),\n 'next', next_id,\n 'prev', prev_id\n)\nFROM j\n;\n\n\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\nINSERT INTO migrations (version) VALUES ('0.2.5');\n\nCOMMIT;\n" }, { "path": "src/pgstac/migrations/pgstac.0.2.4-0.2.7.sql", "content": "SET SEARCH_PATH TO pgstac, public;\nBEGIN;\n\nCREATE OR REPLACE FUNCTION stac_query_op(att text, _op text, val jsonb) RETURNS text AS $$\nDECLARE\nret text := '';\nop text;\njp text;\natt_parts RECORD;\nval_str text;\nprop_path text;\nBEGIN\nval_str := lower(jsonb_build_object('a',val)->>'a');\nRAISE NOTICE 'val_str %', val_str;\n\natt_parts := split_stac_path(att);\nprop_path := replace(att_parts.dotpath, 'properties.', '');\n\nop := CASE _op\n WHEN 'eq' THEN '='\n WHEN 'gte' THEN '>='\n WHEN 'gt' THEN '>'\n WHEN 'lte' THEN '<='\n WHEN 'lt' THEN '<'\n WHEN 'ne' THEN '!='\n WHEN 'neq' THEN '!='\n WHEN 'startsWith' THEN 'LIKE'\n WHEN 'endsWith' THEN 'LIKE'\n WHEN 'contains' THEN 'LIKE'\n ELSE _op\nEND;\n\nval_str := CASE _op\n WHEN 'startsWith' THEN concat(val_str, '%')\n WHEN 'endsWith' THEN concat('%', val_str)\n WHEN 'contains' THEN concat('%',val_str,'%')\n ELSE val_str\nEND;\n\n\nRAISE NOTICE 'att_parts: % %', att_parts, count_by_delim(att_parts.dotpath,'\\.');\nIF\n op = '='\n AND att_parts.col = 'properties'\n --AND count_by_delim(att_parts.dotpath,'\\.') = 2\nTHEN\n -- use jsonpath query to leverage index for eqaulity tests on single level deep properties\n jp := btrim(format($jp$ $.%I[*] ? ( @ == %s ) $jp$, replace(att_parts.dotpath, 'properties.',''), lower(val::text)::jsonb));\n raise notice 'jp: %', jp;\n ret := format($q$ properties @? %L $q$, jp);\nELSIF jsonb_typeof(val) = 'number' THEN\n ret := format('properties ? %L AND (%s)::numeric %s %s', prop_path, att_parts.jspathtext, op, val);\nELSE\n ret := format('properties ? %L AND %s %s %L', prop_path ,att_parts.jspathtext, op, val_str);\nEND IF;\nRAISE NOTICE 'Op Query: %', ret;\n\nreturn ret;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nDROP FUNCTION IF EXISTS bbox_geom;\nCREATE OR REPLACE FUNCTION bbox_geom(_bbox jsonb) RETURNS geometry AS $$\nSELECT CASE jsonb_array_length(_bbox)\n WHEN 4 THEN\n ST_SetSRID(ST_MakeEnvelope(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float,\n (_bbox->>3)::float\n ),4326)\n WHEN 6 THEN\n ST_SetSRID(ST_3DMakeBox(\n ST_MakePoint(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float\n ),\n ST_MakePoint(\n (_bbox->>3)::float,\n (_bbox->>4)::float,\n (_bbox->>5)::float\n )\n ),4326)\n ELSE null END;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION search(_search jsonb = '{}'::jsonb) RETURNS SETOF jsonb AS $$\nDECLARE\nqstart timestamptz := clock_timestamp();\n_sort text := '';\n_rsort text := '';\n_limit int := 10;\n_geom geometry;\nqa text[];\npq text[];\nquery text;\npq_prop record;\npq_op record;\nprev_id text := NULL;\nnext_id text := NULL;\nwhereq text := 'TRUE';\nlinks jsonb := '[]'::jsonb;\ntoken text;\ntok_val text;\ntok_q text := 'TRUE';\ntok_sort text;\nfirst_id text;\nfirst_dt timestamptz;\nlast_id text;\nsort text;\nrsort text;\ndt text[];\ndqa text[];\ndq text;\nmq_where text;\nstartdt timestamptz;\nenddt timestamptz;\nitem items%ROWTYPE;\ncounter int := 0;\nbatchcount int;\nmonth timestamptz;\nm record;\n_dtrange tstzrange := tstzrange('-infinity','infinity');\n_dtsort text;\n_token_dtrange tstzrange := tstzrange('-infinity','infinity');\n_token_record items%ROWTYPE;\nis_prev boolean := false;\nincludes text[];\nexcludes text[];\nBEGIN\n-- Create table from sort query of items to sort\nCREATE TEMP TABLE pgstac_tmp_sorts ON COMMIT DROP AS SELECT * FROM sort_base(_search->'sortby');\n\n-- Get the datetime sort direction, necessary for efficient cycling through partitions\nSELECT INTO _dtsort dir FROM pgstac_tmp_sorts WHERE key='datetime';\nRAISE NOTICE '_dtsort: %',_dtsort;\n\nSELECT INTO _sort string_agg(s.sort,', ') FROM pgstac_tmp_sorts s;\nSELECT INTO _rsort string_agg(s.rsort,', ') FROM pgstac_tmp_sorts s;\ntok_sort := _sort;\n\n\n-- Get datetime from query as a tstzrange\nIF _search ? 'datetime' THEN\n _dtrange := search_dtrange(_search->'datetime');\n _token_dtrange := _dtrange;\nEND IF;\n\n-- Get the paging token\nIF _search ? 'token' THEN\n token := _search->>'token';\n tok_val := substr(token,6);\n IF starts_with(token, 'prev:') THEN\n is_prev := true;\n END IF;\n SELECT INTO _token_record * FROM items WHERE id=tok_val;\n IF\n (is_prev AND _dtsort = 'DESC')\n OR\n (not is_prev AND _dtsort = 'ASC')\n THEN\n _token_dtrange := _dtrange * tstzrange(_token_record.datetime, 'infinity');\n ELSIF\n _dtsort IS NOT NULL\n THEN\n _token_dtrange := _dtrange * tstzrange('-infinity',_token_record.datetime);\n END IF;\n IF is_prev THEN\n tok_q := filter_by_order(tok_val, _search->'sortby', 'first');\n _sort := _rsort;\n ELSIF starts_with(token, 'next:') THEN\n tok_q := filter_by_order(tok_val, _search->'sortby', 'last');\n END IF;\nEND IF;\nRAISE NOTICE 'timing: %', age(clock_timestamp(), qstart);\nRAISE NOTICE 'tok_q: % _token_dtrange: %', tok_q, _token_dtrange;\n\nIF _search ? 'ids' THEN\n RAISE NOTICE 'searching solely based on ids... %',_search;\n qa := array_append(qa, in_array_q('id', _search->'ids'));\nELSE\n IF _search ? 'intersects' THEN\n _geom := ST_SetSRID(ST_GeomFromGeoJSON(_search->>'intersects'), 4326);\n ELSIF _search ? 'bbox' THEN\n _geom := bbox_geom(_search->'bbox');\n END IF;\n\n IF _geom IS NOT NULL THEN\n qa := array_append(qa, format('st_intersects(geometry, %L::geometry)',_geom));\n END IF;\n\n IF _search ? 'collections' THEN\n qa := array_append(qa, in_array_q('collection_id', _search->'collections'));\n END IF;\n\n IF _search ? 'query' THEN\n qa := array_cat(qa,\n stac_query(_search->'query')\n );\n END IF;\nEND IF;\n\nIF _search ? 'limit' THEN\n _limit := (_search->>'limit')::int;\nEND IF;\n\nIF _search ? 'fields' THEN\n IF _search->'fields' ? 'exclude' THEN\n excludes=textarr(_search->'fields'->'exclude');\n END IF;\n IF _search->'fields' ? 'include' THEN\n includes=textarr(_search->'fields'->'include');\n IF array_length(includes, 1)>0 AND NOT 'id' = ANY (includes) THEN\n includes = includes || '{id}';\n END IF;\n END IF;\n RAISE NOTICE 'Includes: %, Excludes: %', includes, excludes;\nEND IF;\n\nwhereq := COALESCE(array_to_string(qa,' AND '),' TRUE ');\ndq := COALESCE(array_to_string(dqa,' AND '),' TRUE ');\nRAISE NOTICE 'timing before temp table: %', age(clock_timestamp(), qstart);\n\nCREATE TEMP TABLE results_page ON COMMIT DROP AS\nSELECT * FROM items_by_partition(\n concat(whereq, ' AND ', tok_q),\n _token_dtrange,\n _sort,\n _limit + 1\n);\nRAISE NOTICE 'timing after temp table: %', age(clock_timestamp(), qstart);\n\nRAISE NOTICE 'timing before min/max: %', age(clock_timestamp(), qstart);\n\nIF is_prev THEN\n SELECT INTO last_id, first_id, counter\n first_value(id) OVER (),\n last_value(id) OVER (),\n count(*) OVER ()\n FROM results_page;\nELSE\n SELECT INTO first_id, last_id, counter\n first_value(id) OVER (),\n last_value(id) OVER (),\n count(*) OVER ()\n FROM results_page;\nEND IF;\nRAISE NOTICE 'firstid: %, lastid %', first_id, last_id;\nRAISE NOTICE 'timing after min/max: %', age(clock_timestamp(), qstart);\n\n\n\n\nIF counter > _limit THEN\n next_id := last_id;\n RAISE NOTICE 'next_id: %', next_id;\nELSE\n RAISE NOTICE 'No more next';\nEND IF;\n\nIF tok_q = 'TRUE' THEN\n RAISE NOTICE 'Not a paging query, no previous item';\nELSE\n RAISE NOTICE 'Getting previous item id';\n RAISE NOTICE 'timing: %', age(clock_timestamp(), qstart);\n SELECT INTO _token_record * FROM items WHERE id=first_id;\n IF\n _dtsort = 'DESC'\n THEN\n _token_dtrange := _dtrange * tstzrange(_token_record.datetime, 'infinity');\n ELSE\n _token_dtrange := _dtrange * tstzrange('-infinity',_token_record.datetime);\n END IF;\n RAISE NOTICE '% %', _token_dtrange, _dtrange;\n SELECT id INTO prev_id FROM items_by_partition(\n concat(whereq, ' AND ', filter_by_order(first_id, _search->'sortby', 'prev')),\n _token_dtrange,\n _rsort,\n 1\n );\n RAISE NOTICE 'timing: %', age(clock_timestamp(), qstart);\n\n RAISE NOTICE 'prev_id: %', prev_id;\nEND IF;\n\n\nRETURN QUERY\nWITH features AS (\n SELECT filter_jsonb(content, includes, excludes) as content\n FROM results_page LIMIT _limit\n),\nj AS (SELECT jsonb_agg(content) as feature_arr FROM features)\nSELECT jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce (\n CASE WHEN is_prev THEN flip_jsonb_array(feature_arr) ELSE feature_arr END\n ,'[]'::jsonb),\n 'links', links,\n 'timeStamp', now(),\n 'next', next_id,\n 'prev', prev_id\n)\nFROM j\n;\n\n\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\nINSERT INTO migrations (version) VALUES ('0.2.7');\n\nCOMMIT;\n" }, { "path": "src/pgstac/migrations/pgstac.0.2.4.sql", "content": "CREATE EXTENSION IF NOT EXISTS postgis;\nCREATE SCHEMA IF NOT EXISTS partman;\nCREATE EXTENSION IF NOT EXISTS pg_partman SCHEMA partman;\nCREATE SCHEMA IF NOT EXISTS pgstac;\n\n\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE TABLE migrations (\n version text,\n datetime timestamptz DEFAULT now() NOT NULL\n);\n\n/* converts a jsonb text array to a pg text[] array */\nCREATE OR REPLACE FUNCTION textarr(_js jsonb)\n RETURNS text[] AS $$\n SELECT ARRAY(SELECT jsonb_array_elements_text(_js));\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\n/*\nconverts a jsonb text array to comma delimited list of identifer quoted\nuseful for constructing column lists for selects\n*/\nCREATE OR REPLACE FUNCTION array_idents(_js jsonb)\n RETURNS text AS $$\n SELECT string_agg(quote_ident(v),',') FROM jsonb_array_elements_text(_js) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\n\n/* looks for a geometry in a stac item first from geometry and falling back to bbox */\nCREATE OR REPLACE FUNCTION stac_geom(value jsonb) RETURNS geometry AS $$\nSELECT\n CASE\n WHEN value->>'geometry' IS NOT NULL THEN\n ST_GeomFromGeoJSON(value->>'geometry')\n WHEN value->>'bbox' IS NOT NULL THEN\n ST_MakeEnvelope(\n (value->'bbox'->>0)::float,\n (value->'bbox'->>1)::float,\n (value->'bbox'->>2)::float,\n (value->'bbox'->>3)::float,\n 4326\n )\n ELSE NULL\n END as geometry\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION stac_datetime(value jsonb) RETURNS timestamptz AS $$\nSELECT (value->'properties'->>'datetime')::timestamptz;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION jsonb_paths (IN jdata jsonb, OUT path text[], OUT value jsonb) RETURNS\nSETOF RECORD AS $$\nwith recursive extract_all as\n(\n select\n ARRAY[key]::text[] as path,\n value\n FROM jsonb_each(jdata)\nunion all\n select\n path || coalesce(obj_key, (arr_key- 1)::text),\n coalesce(obj_value, arr_value)\n from extract_all\n left join lateral\n jsonb_each(case jsonb_typeof(value) when 'object' then value end)\n as o(obj_key, obj_value)\n on jsonb_typeof(value) = 'object'\n left join lateral\n jsonb_array_elements(case jsonb_typeof(value) when 'array' then value end)\n with ordinality as a(arr_value, arr_key)\n on jsonb_typeof(value) = 'array'\n where obj_key is not null or arr_key is not null\n)\nselect *\nfrom extract_all;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION jsonb_obj_paths (IN jdata jsonb, OUT path text[], OUT value jsonb) RETURNS\nSETOF RECORD AS $$\nwith recursive extract_all as\n(\n select\n ARRAY[key]::text[] as path,\n value\n FROM jsonb_each(jdata)\nunion all\n select\n path || obj_key,\n obj_value\n from extract_all\n left join lateral\n jsonb_each(case jsonb_typeof(value) when 'object' then value end)\n as o(obj_key, obj_value)\n on jsonb_typeof(value) = 'object'\n where obj_key is not null\n)\nselect *\nfrom extract_all;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION jsonb_val_paths (IN jdata jsonb, OUT path text[], OUT value jsonb) RETURNS\nSETOF RECORD AS $$\nSELECT * FROM jsonb_obj_paths(jdata) WHERE jsonb_typeof(value) not in ('object','array');\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION path_includes(IN path text[], IN includes text[]) RETURNS BOOLEAN AS $$\nWITH t AS (SELECT unnest(includes) i)\nSELECT EXISTS (\n SELECT 1 FROM t WHERE path @> string_to_array(i, '.')\n);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION path_excludes(IN path text[], IN excludes text[]) RETURNS BOOLEAN AS $$\nWITH t AS (SELECT unnest(excludes) e)\nSELECT NOT EXISTS (\n SELECT 1 FROM t WHERE path @> string_to_array(e, '.')\n);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION jsonb_obj_paths_filtered (\n IN jdata jsonb,\n IN includes text[] DEFAULT ARRAY[]::text[],\n IN excludes text[] DEFAULT ARRAY[]::text[],\n OUT path text[],\n OUT value jsonb\n) RETURNS\nSETOF RECORD AS $$\nSELECT path, value\nFROM jsonb_obj_paths(jdata)\nWHERE\n CASE WHEN cardinality(includes) > 0 THEN path_includes(path, includes) ELSE TRUE END\n AND\n path_excludes(path, excludes)\n\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION empty_arr(ANYARRAY) RETURNS BOOLEAN AS $$\nSELECT CASE\n WHEN $1 IS NULL THEN TRUE\n WHEN cardinality($1)<1 THEN TRUE\nELSE FALSE\nEND;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION filter_jsonb(\n IN jdata jsonb,\n IN includes text[] DEFAULT ARRAY[]::text[],\n IN excludes text[] DEFAULT ARRAY[]::text[]\n) RETURNS jsonb AS $$\nDECLARE\nrec RECORD;\noutj jsonb := '{}'::jsonb;\ncreated_paths text[] := '{}'::text[];\nBEGIN\n\nIF empty_arr(includes) AND empty_arr(excludes) THEN\nRAISE NOTICE 'no filter';\n RETURN jdata;\nEND IF;\nFOR rec in\nSELECT * FROM jsonb_obj_paths_filtered(jdata, includes, excludes)\nWHERE jsonb_typeof(value) != 'object'\nLOOP\n RAISE NOTICE 'path % val %', rec.path, rec.value;\n IF array_length(rec.path,1)>1 THEN\n FOR i IN 1..(array_length(rec.path,1)-1) LOOP\n IF NOT array_to_string(rec.path[1:i],'.') = ANY (created_paths) THEN\n outj := jsonb_set(outj, rec.path[1:i],'{}', true);\n created_paths := created_paths || array_to_string(rec.path[1:i],'.');\n END IF;\n END LOOP;\n END IF;\n outj := jsonb_set(outj, rec.path, rec.value, true);\n created_paths := created_paths || array_to_string(rec.path,'.');\nEND LOOP;\nRETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION properties_idx(_in jsonb) RETURNS jsonb AS $$\nWITH t AS (\n select array_to_string(path,'.') as path, lower(value::text)::jsonb as lowerval\n FROM jsonb_val_paths(_in)\n WHERE array_to_string(path,'.') not in ('datetime')\n)\nSELECT jsonb_object_agg(path, lowerval) FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE TABLE IF NOT EXISTS collections (\n id VARCHAR GENERATED ALWAYS AS (content->>'id') STORED PRIMARY KEY,\n content JSONB\n);\n\nCREATE OR REPLACE FUNCTION create_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection(data jsonb) RETURNS VOID AS $$\nDECLARE\nout collections%ROWTYPE;\nBEGIN\n UPDATE collections SET content=data WHERE id = data->>'id' RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ON CONFLICT (id) DO\n UPDATE\n SET content=EXCLUDED.content\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION delete_collection(_id text) RETURNS VOID AS $$\nDECLARE\nout collections%ROWTYPE;\nBEGIN\n DELETE FROM collections WHERE id = _id RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION get_collection(id text) RETURNS jsonb AS $$\nSELECT content FROM collections\nWHERE id=$1\n;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION all_collections() RETURNS jsonb AS $$\nSELECT jsonb_agg(content) FROM collections;\n;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\n\n\n\n\n\n/* CREATE OR REPLACE FUNCTION collections_trigger_func()\nRETURNS TRIGGER AS $$\nBEGIN\n IF pg_trigger_depth() = 1 THEN\n PERFORM create_collection(NEW.content);\n RETURN NULL;\n END IF;\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE TRIGGER collections_trigger\nBEFORE INSERT ON collections\nFOR EACH ROW EXECUTE PROCEDURE collections_trigger_func();\n */\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE TABLE IF NOT EXISTS items (\n id VARCHAR GENERATED ALWAYS AS (content->>'id') STORED NOT NULL,\n geometry geometry GENERATED ALWAYS AS (stac_geom(content)) STORED NOT NULL,\n properties jsonb GENERATED ALWAYS as (properties_idx(content->'properties')) STORED,\n collection_id text GENERATED ALWAYS AS (content->>'collection') STORED NOT NULL,\n datetime timestamptz GENERATED ALWAYS AS (stac_datetime(content)) STORED NOT NULL,\n content JSONB NOT NULL\n)\nPARTITION BY RANGE (stac_datetime(content))\n;\n\nALTER TABLE items ADD constraint items_collections_fk FOREIGN KEY (collection_id) REFERENCES collections(id) DEFERRABLE;\n\nCREATE TABLE items_template (\n LIKE items\n);\n\nALTER TABLE items_template ADD PRIMARY KEY (id);\n\n/*\nCREATE TABLE IF NOT EXISTS items_search (\n id text NOT NULL,\n geometry geometry NOT NULL,\n properties jsonb,\n collection_id text NOT NULL,\n datetime timestamptz NOT NULL\n)\nPARTITION BY RANGE (datetime)\n;\n\nCREATE TABLE IF NOT EXISTS items_search_template (\n LIKE items_search\n)\n;\nALTER TABLE items_search_template ADD PRIMARY KEY (id);\n*/\n\nDELETE from partman.part_config WHERE parent_table = 'pgstac.items';\nSELECT partman.create_parent(\n 'pgstac.items',\n 'datetime',\n 'native',\n 'weekly',\n p_template_table := 'pgstac.items_template',\n p_premake := 4\n);\n\nCREATE OR REPLACE FUNCTION make_partitions(st timestamptz, et timestamptz DEFAULT NULL) RETURNS BOOL AS $$\nWITH t AS (\n SELECT\n generate_series(\n date_trunc('week',st),\n date_trunc('week', coalesce(et, st)),\n '1 week'::interval\n ) w\n),\nw AS (SELECT array_agg(w) as w FROM t)\nSELECT CASE WHEN w IS NULL THEN NULL ELSE partman.create_partition_time('pgstac.items', w, true) END FROM w;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_partition(timestamptz) RETURNS text AS $$\nSELECT to_char($1, '\"items_p\"IYYY\"w\"IW');\n$$ LANGUAGE SQL;\n\nCREATE INDEX \"datetime_id_idx\" ON items (datetime, id);\nCREATE INDEX \"properties_idx\" ON items USING GIN (properties);\nCREATE INDEX \"collection_idx\" ON items (collection_id);\nCREATE INDEX \"geometry_idx\" ON items USING GIST (geometry);\n\n\nCREATE TYPE item AS (\n id text,\n geometry geometry,\n properties JSONB,\n collection_id text,\n datetime timestamptz\n);\n\n\n/*\nConverts single feature into an items row\n*/\n\n/*\nCREATE OR REPLACE FUNCTION feature_to_item(value jsonb) RETURNS item AS $$\n SELECT\n value->>'id' as id,\n CASE\n WHEN value->>'geometry' IS NOT NULL THEN\n ST_GeomFromGeoJSON(value->>'geometry')\n WHEN value->>'bbox' IS NOT NULL THEN\n ST_MakeEnvelope(\n (value->'bbox'->>0)::float,\n (value->'bbox'->>1)::float,\n (value->'bbox'->>2)::float,\n (value->'bbox'->>3)::float,\n 4326\n )\n ELSE NULL\n END as geometry,\n properties_idx(value ->'properties') as properties,\n value->>'collection' as collection_id,\n (value->'properties'->>'datetime')::timestamptz as datetime\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac,public;\n*/\n/*\nTakes a single feature, an array of features, or a feature collection\nand returns a set up individual items rows\n*/\n/*\nCREATE OR REPLACE FUNCTION features_to_items(value jsonb) RETURNS SETOF item AS $$\n WITH features AS (\n SELECT\n jsonb_array_elements(\n CASE\n WHEN jsonb_typeof(value) = 'array' THEN value\n WHEN value->>'type' = 'Feature' THEN '[]'::jsonb || value\n WHEN value->>'type' = 'FeatureCollection' THEN value->'features'\n ELSE NULL\n END\n ) as value\n )\n SELECT feature_to_item(value) FROM features\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac,public;\n*/\n\nCREATE OR REPLACE FUNCTION get_item(_id text) RETURNS jsonb AS $$\n SELECT content FROM items WHERE id=_id;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION delete_item(_id text) RETURNS VOID AS $$\nDECLARE\nout items%ROWTYPE;\nBEGIN\n DELETE FROM items WHERE id = _id RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION create_item(data jsonb) RETURNS VOID AS $$\n SELECT make_partitions(stac_datetime(data));\n INSERT INTO items (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION update_item(data jsonb) RETURNS VOID AS $$\nDECLARE\nout items%ROWTYPE;\nBEGIN\n UPDATE items SET content=data WHERE id = data->>'id' RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_item(data jsonb) RETURNS VOID AS $$\nDECLARE\npartition text;\nq text;\nnewcontent jsonb;\nBEGIN\n PERFORM make_partitions(stac_datetime(data));\n partition := get_partition(stac_datetime(data));\n q := format($q$\n INSERT INTO %I (content) VALUES ($1)\n ON CONFLICT (id) DO\n UPDATE SET content = EXCLUDED.content\n WHERE %I.content IS DISTINCT FROM EXCLUDED.content RETURNING content;\n $q$, partition, partition);\n EXECUTE q INTO newcontent USING (data);\n RAISE NOTICE 'newcontent: %', newcontent;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION analyze_empty_partitions() RETURNS VOID AS $$\nDECLARE\np text;\nBEGIN\nFOR p IN SELECT partition FROM all_items_partitions WHERE est_cnt = 0 LOOP\n RAISE NOTICE 'Analyzing %', p;\n EXECUTE format('ANALYZE %I;', p);\nEND LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n/* Trigger Function to cascade inserts/updates/deletes\nfrom items table to items_search table */\n/*\nALTER TABLE items_search ADD CONSTRAINT items_search_fk\nFOREIGN KEY (id) REFERENCES items(id)\nON DELETE CASCADE DEFERRABLE;\n\nCREATE OR REPLACE FUNCTION items_trigger_func()\nRETURNS TRIGGER AS $$\nDECLARE\nBEGIN\n IF TG_OP = 'UPDATE' THEN\n RAISE NOTICE 'DELETING % BEFORE UPDATE', OLD;\n DELETE FROM items_search WHERE id = OLD.id AND datetime = (OLD.content->'properties'->>'datetime')::timestamptz;\n END IF;\n\n INSERT INTO items_search SELECT * FROM feature_to_item(NEW.content);\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nDROP TRIGGER IF EXISTS items_insert_trigger ON items;\nCREATE TRIGGER items_insert_trigger\nAFTER INSERT ON items\nFOR EACH ROW EXECUTE PROCEDURE items_trigger_func();\n\nDROP TRIGGER IF EXISTS items_update_trigger ON items;\nCREATE TRIGGER items_update_trigger\nAFTER UPDATE ON items\nFOR EACH ROW\nWHEN (NEW.content IS DISTINCT FROM OLD.content)\nEXECUTE PROCEDURE items_trigger_func();\n*/\n\n/* Trigger Function to cascade inserts/updates/deletes\nfrom items table to items_search table */\n/*\nCREATE OR REPLACE FUNCTION items_search_trigger_delete_func()\nRETURNS TRIGGER AS $$\nDECLARE\nBEGIN\n RAISE NOTICE 'Deleting from items_search: % Depth: %', OLD, pg_trigger_depth();\n IF pg_trigger_depth()<3 THEN\n RAISE NOTICE 'DELETING WITH datetime';\n DELETE FROM items_search WHERE id=OLD.id AND datetime=OLD.datetime;\n RETURN NULL;\n END IF;\n RETURN OLD;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nDROP TRIGGER IF EXISTS items_search_delete_trigger ON items_search;\nCREATE TRIGGER items_search_delete_trigger\nBEFORE DELETE ON items_search\nFOR EACH ROW EXECUTE PROCEDURE items_search_trigger_delete_func();\n*/\nCREATE OR REPLACE FUNCTION backfill_partitions()\nRETURNS VOID AS $$\nDECLARE\nBEGIN\n IF EXISTS (SELECT 1 FROM items_default LIMIT 1) THEN\n RAISE NOTICE 'Creating new partitions and moving data from default';\n CREATE TEMP TABLE items_default_tmp ON COMMIT DROP AS SELECT datetime, content FROM items_default;\n TRUNCATE items_default;\n PERFORM make_partitions(min(datetime), max(datetime)) FROM items_default_tmp;\n INSERT INTO items (content) SELECT content FROM items_default_tmp;\n END IF;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION items_trigger_stmt_func()\nRETURNS TRIGGER AS $$\nDECLARE\nBEGIN\n PERFORM analyze_empty_partitions();\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nDROP TRIGGER IF EXISTS items_stmt_trigger ON items;\nCREATE TRIGGER items_stmt_trigger\nAFTER INSERT OR UPDATE OR DELETE ON items\nFOR EACH STATEMENT EXECUTE PROCEDURE items_trigger_stmt_func();\n\n\n/*\nView to get a table of available items partitions\nwith date ranges\n*/\n--DROP VIEW IF EXISTS all_items_partitions CASCADE;\nCREATE OR REPLACE VIEW all_items_partitions AS\nWITH base AS\n(SELECT\n c.oid::pg_catalog.regclass::text as partition,\n pg_catalog.pg_get_expr(c.relpartbound, c.oid) as _constraint,\n regexp_matches(\n pg_catalog.pg_get_expr(c.relpartbound, c.oid),\n E'\\\\(''\\([0-9 :+-]*\\)''\\\\).*\\\\(''\\([0-9 :+-]*\\)''\\\\)'\n ) as t,\n reltuples::bigint as est_cnt\nFROM pg_catalog.pg_class c, pg_catalog.pg_inherits i\nWHERE c.oid = i.inhrelid AND i.inhparent = 'items'::regclass)\nSELECT partition, tstzrange(\n t[1]::timestamptz,\n t[2]::timestamptz\n), est_cnt\nFROM base\nORDER BY 2 desc;\n\n--DROP VIEW IF EXISTS items_partitions;\nCREATE OR REPLACE VIEW items_partitions AS\nSELECT * FROM all_items_partitions WHERE est_cnt>0;\n\nCREATE OR REPLACE FUNCTION collection_bbox(id text) RETURNS jsonb AS $$\nSELECT (replace(replace(replace(st_extent(geometry)::text,'BOX(','[['),')',']]'),' ',','))::jsonb\nFROM items WHERE collection_id=$1;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION collection_temporal_extent(id text) RETURNS jsonb AS $$\nSELECT to_jsonb(array[array[min(datetime)::text, max(datetime)::text]])\nFROM items WHERE collection_id=$1;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection_extents() RETURNS VOID AS $$\nUPDATE collections SET\n content = content ||\n jsonb_build_object(\n 'extent', jsonb_build_object(\n 'spatial', jsonb_build_object(\n 'bbox', collection_bbox(collections.id)\n ),\n 'temporal', jsonb_build_object(\n 'interval', collection_temporal_extent(collections.id)\n )\n )\n )\n;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\nSET SEARCH_PATH TO pgstac, public;\n\n/*\nView to get a table of available items partitions\nwith date ranges\n*/\nDROP VIEW IF EXISTS items_partitions;\nCREATE VIEW items_partitions AS\nWITH base AS\n(SELECT\n c.oid::pg_catalog.regclass::text as partition,\n pg_catalog.pg_get_expr(c.relpartbound, c.oid) as _constraint,\n regexp_matches(\n pg_catalog.pg_get_expr(c.relpartbound, c.oid),\n E'\\\\(''\\([0-9 :+-]*\\)''\\\\).*\\\\(''\\([0-9 :+-]*\\)''\\\\)'\n ) as t,\n reltuples::bigint as est_cnt\nFROM pg_catalog.pg_class c, pg_catalog.pg_inherits i\nWHERE c.oid = i.inhrelid AND i.inhparent = 'items'::regclass)\nSELECT partition, tstzrange(\n t[1]::timestamptz,\n t[2]::timestamptz\n), est_cnt\nFROM base\nWHERE est_cnt >0\nORDER BY 2 desc;\n\n\nCREATE OR REPLACE FUNCTION items_by_partition(\n IN _where text DEFAULT 'TRUE',\n IN _dtrange tstzrange DEFAULT tstzrange('-infinity','infinity'),\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN _limit int DEFAULT 10\n) RETURNS SETOF items AS $$\nDECLARE\npartition_query text;\nmain_query text;\nbatchcount int;\ncounter int := 0;\np record;\nBEGIN\nIF _orderby ILIKE 'datetime d%' THEN\n partition_query := format($q$\n SELECT partition\n FROM items_partitions\n WHERE tstzrange && $1\n ORDER BY tstzrange DESC;\n $q$);\nELSIF _orderby ILIKE 'datetime a%' THEN\n partition_query := format($q$\n SELECT partition\n FROM items_partitions\n WHERE tstzrange && $1\n ORDER BY tstzrange ASC\n ;\n $q$);\nELSE\n partition_query := format($q$\n SELECT 'items' as partition WHERE $1 IS NOT NULL;\n $q$);\nEND IF;\nRAISE NOTICE 'Partition Query: %', partition_query;\nFOR p IN\n EXECUTE partition_query USING (_dtrange)\nLOOP\n IF lower(_dtrange)::timestamptz > '-infinity' THEN\n _where := concat(_where,format(' AND datetime >= %L',lower(_dtrange)::timestamptz::text));\n END IF;\n IF upper(_dtrange)::timestamptz < 'infinity' THEN\n _where := concat(_where,format(' AND datetime <= %L',upper(_dtrange)::timestamptz::text));\n END IF;\n\n main_query := format($q$\n SELECT * FROM %I\n WHERE %s\n ORDER BY %s\n LIMIT %s - $1\n $q$, p.partition::text, _where, _orderby, _limit\n );\n RAISE NOTICE 'Partition Query %', main_query;\n RAISE NOTICE '%', counter;\n RETURN QUERY EXECUTE main_query USING counter;\n\n GET DIAGNOSTICS batchcount = ROW_COUNT;\n counter := counter + batchcount;\n RAISE NOTICE 'FOUND %', batchcount;\n IF counter >= _limit THEN\n EXIT;\n END IF;\n RAISE NOTICE 'ADDED % FOR A TOTAL OF %', batchcount, counter;\nEND LOOP;\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION split_stac_path(IN path text, OUT col text, OUT dotpath text, OUT jspath text, OUT jspathtext text) AS $$\nWITH col AS (\n SELECT\n CASE WHEN\n split_part(path, '.', 1) IN ('id', 'stac_version', 'stac_extensions','geometry','properties','assets','collection_id','datetime','links', 'extra_fields') THEN split_part(path, '.', 1)\n ELSE 'properties'\n END AS col\n),\ndp AS (\n SELECT\n col, ltrim(replace(path, col , ''),'.') as dotpath\n FROM col\n),\npaths AS (\nSELECT\n col, dotpath,\n regexp_split_to_table(dotpath,E'\\\\.') as path FROM dp\n) SELECT\n col,\n btrim(concat(col,'.',dotpath),'.'),\n CASE WHEN btrim(concat(col,'.',dotpath),'.') != col THEN concat(col,'->',string_agg(concat('''',path,''''),'->')) ELSE col END,\n regexp_replace(\n CASE WHEN btrim(concat(col,'.',dotpath),'.') != col THEN concat(col,'->',string_agg(concat('''',path,''''),'->')) ELSE col END,\n E'>([^>]*)$','>>\\1'\n )\nFROM paths group by col, dotpath;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n/* Functions for searching items */\nCREATE OR REPLACE FUNCTION sort_base(\n IN _sort jsonb DEFAULT '[{\"field\":\"datetime\",\"direction\":\"desc\"}]',\n OUT key text,\n OUT col text,\n OUT dir text,\n OUT rdir text,\n OUT sort text,\n OUT rsort text\n) RETURNS SETOF RECORD AS $$\nWITH sorts AS (\n SELECT\n value->>'field' as key,\n (split_stac_path(value->>'field')).jspathtext as col,\n coalesce(upper(value->>'direction'),'ASC') as dir\n FROM jsonb_array_elements('[]'::jsonb || coalesce(_sort,'[{\"field\":\"datetime\",\"direction\":\"desc\"}]') )\n)\nSELECT\n key,\n col,\n dir,\n CASE dir WHEN 'DESC' THEN 'ASC' ELSE 'ASC' END as rdir,\n concat(col, ' ', dir, ' NULLS LAST ') AS sort,\n concat(col,' ', CASE dir WHEN 'DESC' THEN 'ASC' ELSE 'ASC' END, ' NULLS LAST ') AS rsort\nFROM sorts\nUNION ALL\nSELECT 'id', 'id', 'DESC', 'ASC', 'id DESC', 'id ASC'\n;\n$$ LANGUAGE SQL;\n\n\nCREATE OR REPLACE FUNCTION sort(_sort jsonb) RETURNS text AS $$\nSELECT string_agg(sort,', ') FROM sort_base(_sort);\n$$ LANGUAGE SQL PARALLEL SAFE SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION rsort(_sort jsonb) RETURNS text AS $$\nSELECT string_agg(rsort,', ') FROM sort_base(_sort);\n$$ LANGUAGE SQL PARALLEL SAFE SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION bbox_geom(_bbox jsonb) RETURNS box3d AS $$\nSELECT CASE jsonb_array_length(_bbox)\n WHEN 4 THEN\n ST_SetSRID(ST_MakeEnvelope(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float,\n (_bbox->>3)::float\n ),4326)\n WHEN 6 THEN\n ST_SetSRID(ST_3DMakeBox(\n ST_MakePoint(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float\n ),\n ST_MakePoint(\n (_bbox->>3)::float,\n (_bbox->>4)::float,\n (_bbox->>5)::float\n )\n ),4326)\n ELSE null END;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION in_array_q(col text, arr jsonb) RETURNS text AS $$\nSELECT CASE jsonb_typeof(arr) WHEN 'array' THEN format('%I = ANY(textarr(%L))', col, arr) ELSE format('%I = %L', col, arr) END;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION count_by_delim(text, text) RETURNS int AS $$\nSELECT count(*) FROM regexp_split_to_table($1,$2);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION stac_query_op(att text, _op text, val jsonb) RETURNS text AS $$\nDECLARE\nret text := '';\nop text;\njp text;\natt_parts RECORD;\nval_str text;\nBEGIN\nval_str := lower(jsonb_build_object('a',val)->>'a');\nRAISE NOTICE 'val_str %', val_str;\n\natt_parts := split_stac_path(att);\n\nop := CASE _op\n WHEN 'eq' THEN '='\n WHEN 'ge' THEN '>='\n WHEN 'gt' THEN '>'\n WHEN 'le' THEN '<='\n WHEN 'lt' THEN '<'\n WHEN 'ne' THEN '!='\n WHEN 'neq' THEN '!='\n WHEN 'startsWith' THEN 'LIKE'\n WHEN 'endsWith' THEN 'LIKE'\n WHEN 'contains' THEN 'LIKE'\n ELSE _op\nEND;\n\nval_str := CASE _op\n WHEN 'startsWith' THEN concat(val_str, '%')\n WHEN 'endsWith' THEN concat('%', val_str)\n WHEN 'contains' THEN concat('%',val_str,'%')\n ELSE val_str\nEND;\n\n\nRAISE NOTICE 'att_parts: % %', att_parts, count_by_delim(att_parts.dotpath,'\\.');\nIF\n op = '='\n AND att_parts.col = 'properties'\n --AND count_by_delim(att_parts.dotpath,'\\.') = 2\nTHEN\n -- use jsonpath query to leverage index for eqaulity tests on single level deep properties\n jp := btrim(format($jp$ $.%I[*] ? ( @ == %s ) $jp$, replace(att_parts.dotpath, 'properties.',''), lower(val::text)::jsonb));\n raise notice 'jp: %', jp;\n ret := format($q$ properties @? %L $q$, jp);\nELSIF jsonb_typeof(val) = 'number' THEN\n ret := format('(%s)::numeric %s %s', att_parts.jspathtext, op, val);\nELSE\n ret := format('%s %s %L', att_parts.jspathtext, op, val_str);\nEND IF;\nRAISE NOTICE 'Op Query: %', ret;\n\nreturn ret;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION stac_query(_query jsonb) RETURNS TEXT[] AS $$\nDECLARE\nqa text[];\natt text;\nops jsonb;\nop text;\nval jsonb;\nBEGIN\nFOR att, ops IN SELECT key, value FROM jsonb_each(_query)\nLOOP\n FOR op, val IN SELECT key, value FROM jsonb_each(ops)\n LOOP\n qa := array_append(qa, stac_query_op(att,op, val));\n RAISE NOTICE '% % %', att, op, val;\n END LOOP;\nEND LOOP;\nRETURN qa;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION filter_by_order(item_id text, _sort jsonb, _type text) RETURNS text AS $$\nDECLARE\nitem item;\nBEGIN\nSELECT * INTO item FROM items WHERE id=item_id;\nRETURN filter_by_order(item, _sort, _type);\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n-- Used to create filters used for paging using the items id from the token\nCREATE OR REPLACE FUNCTION filter_by_order(_item item, _sort jsonb, _type text) RETURNS text AS $$\nDECLARE\nsorts RECORD;\nfilts text[];\nitemval text;\nop text;\nidop text;\nret text;\neq_flag text;\n_item_j jsonb := to_jsonb(_item);\nBEGIN\nFOR sorts IN SELECT * FROM sort_base(_sort) LOOP\n IF sorts.col = 'datetime' THEN\n CONTINUE;\n END IF;\n IF sorts.col='id' AND _type IN ('prev','next') THEN\n eq_flag := '';\n ELSE\n eq_flag := '=';\n END IF;\n\n op := concat(\n CASE\n WHEN _type in ('prev','first') AND sorts.dir = 'ASC' THEN '<'\n WHEN _type in ('last','next') AND sorts.dir = 'ASC' THEN '>'\n WHEN _type in ('prev','first') AND sorts.dir = 'DESC' THEN '>'\n WHEN _type in ('last','next') AND sorts.dir = 'DESC' THEN '<'\n END,\n eq_flag\n );\n\n IF _item_j ? sorts.col THEN\n filts = array_append(filts, format('%s %s %L', sorts.col, op, _item_j->>sorts.col));\n END IF;\nEND LOOP;\nret := coalesce(array_to_string(filts,' AND '), 'TRUE');\nRAISE NOTICE 'Order Filter %', ret;\nRETURN ret;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION search_dtrange(IN _indate jsonb, OUT _tstzrange tstzrange) AS\n$$\nWITH t AS (\n SELECT CASE\n WHEN jsonb_typeof(_indate) = 'array' THEN\n textarr(_indate)\n ELSE\n regexp_split_to_array(\n btrim(_indate::text,'\"'),\n '/'\n )\n END AS arr\n)\n, t1 AS (\n SELECT\n CASE\n WHEN array_upper(arr,1) = 1 OR arr[1] = '..' OR arr[1] IS NULL THEN '-infinity'::timestamptz\n ELSE arr[1]::timestamptz\n END AS st,\n CASE\n WHEN array_upper(arr,1) = 1 THEN arr[1]::timestamptz\n WHEN arr[2] = '..' OR arr[2] IS NULL THEN 'infinity'::timestamptz\n ELSE arr[2]::timestamptz\n END AS et\n FROM t\n)\nSELECT\n tstzrange(st,et)\nFROM t1;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION flip_jsonb_array(j jsonb) RETURNS jsonb AS $$\nWITH t AS (\n SELECT i, row_number() over () as r FROM jsonb_array_elements(j) i\n), o AS (\n SELECT i FROM t ORDER BY r DESC\n)\nSELECT jsonb_agg(i) from o\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION search(_search jsonb = '{}'::jsonb) RETURNS SETOF jsonb AS $$\nDECLARE\nqstart timestamptz := clock_timestamp();\n_sort text := '';\n_rsort text := '';\n_limit int := 10;\n_geom geometry;\nqa text[];\npq text[];\nquery text;\npq_prop record;\npq_op record;\nprev_id text := NULL;\nnext_id text := NULL;\nwhereq text := 'TRUE';\nlinks jsonb := '[]'::jsonb;\ntoken text;\ntok_val text;\ntok_q text := 'TRUE';\ntok_sort text;\nfirst_id text;\nfirst_dt timestamptz;\nlast_id text;\nsort text;\nrsort text;\ndt text[];\ndqa text[];\ndq text;\nmq_where text;\nstartdt timestamptz;\nenddt timestamptz;\nitem items%ROWTYPE;\ncounter int := 0;\nbatchcount int;\nmonth timestamptz;\nm record;\n_dtrange tstzrange := tstzrange('-infinity','infinity');\n_dtsort text;\n_token_dtrange tstzrange := tstzrange('-infinity','infinity');\n_token_record items%ROWTYPE;\nis_prev boolean := false;\nincludes text[];\nexcludes text[];\nBEGIN\n-- Create table from sort query of items to sort\nCREATE TEMP TABLE pgstac_tmp_sorts ON COMMIT DROP AS SELECT * FROM sort_base(_search->'sortby');\n\n-- Get the datetime sort direction, necessary for efficient cycling through partitions\nSELECT INTO _dtsort dir FROM pgstac_tmp_sorts WHERE key='datetime';\nRAISE NOTICE '_dtsort: %',_dtsort;\n\nSELECT INTO _sort string_agg(s.sort,', ') FROM pgstac_tmp_sorts s;\nSELECT INTO _rsort string_agg(s.rsort,', ') FROM pgstac_tmp_sorts s;\ntok_sort := _sort;\n\n\n-- Get datetime from query as a tstzrange\nIF _search ? 'datetime' THEN\n _dtrange := search_dtrange(_search->'datetime');\n _token_dtrange := _dtrange;\nEND IF;\n\n-- Get the paging token\nIF _search ? 'token' THEN\n token := _search->>'token';\n tok_val := substr(token,6);\n IF starts_with(token, 'prev:') THEN\n is_prev := true;\n END IF;\n SELECT INTO _token_record * FROM items WHERE id=tok_val;\n IF\n (is_prev AND _dtsort = 'DESC')\n OR\n (not is_prev AND _dtsort = 'ASC')\n THEN\n _token_dtrange := tstzrange(_token_record.datetime, 'infinity');\n ELSIF\n _dtsort IS NOT NULL\n THEN\n _token_dtrange := tstzrange('-infinity',_token_record.datetime);\n END IF;\n IF is_prev THEN\n tok_q := filter_by_order(tok_val, _search->'sortby', 'first');\n _sort := _rsort;\n ELSIF starts_with(token, 'next:') THEN\n tok_q := filter_by_order(tok_val, _search->'sortby', 'last');\n END IF;\nEND IF;\nRAISE NOTICE 'timing: %', age(clock_timestamp(), qstart);\nRAISE NOTICE 'tok_q: % _token_dtrange: %', tok_q, _token_dtrange;\n\nIF _search ? 'ids' THEN\n RAISE NOTICE 'searching solely based on ids... %',_search;\n qa := array_append(qa, in_array_q('id', _search->'ids'));\nELSE\n IF _search ? 'intersects' THEN\n _geom := ST_SetSRID(ST_GeomFromGeoJSON(_search->>'intersects'), 4326);\n ELSIF _search ? 'bbox' THEN\n _geom := bbox_geom(_search->'bbox');\n END IF;\n\n IF _geom IS NOT NULL THEN\n qa := array_append(qa, format('st_intersects(geometry, %L::geometry)',_geom));\n END IF;\n\n IF _search ? 'collections' THEN\n qa := array_append(qa, in_array_q('collection_id', _search->'collections'));\n END IF;\n\n IF _search ? 'query' THEN\n qa := array_cat(qa,\n stac_query(_search->'query')\n );\n END IF;\nEND IF;\n\nIF _search ? 'limit' THEN\n _limit := (_search->>'limit')::int;\nEND IF;\n\nIF _search ? 'fields' THEN\n IF _search->'fields' ? 'exclude' THEN\n excludes=textarr(_search->'fields'->'exclude');\n END IF;\n IF _search->'fields' ? 'include' THEN\n includes=textarr(_search->'fields'->'include');\n IF array_length(includes, 1)>0 AND NOT 'id' = ANY (includes) THEN\n includes = includes || '{id}';\n END IF;\n END IF;\n RAISE NOTICE 'Includes: %, Excludes: %', includes, excludes;\nEND IF;\n\nwhereq := COALESCE(array_to_string(qa,' AND '),' TRUE ');\ndq := COALESCE(array_to_string(dqa,' AND '),' TRUE ');\nRAISE NOTICE 'timing before temp table: %', age(clock_timestamp(), qstart);\n\nCREATE TEMP TABLE results_page ON COMMIT DROP AS\nSELECT * FROM items_by_partition(\n concat(whereq, ' AND ', tok_q),\n _token_dtrange,\n _sort,\n _limit + 1\n);\nRAISE NOTICE 'timing after temp table: %', age(clock_timestamp(), qstart);\n\nRAISE NOTICE 'timing before min/max: %', age(clock_timestamp(), qstart);\n\nIF is_prev THEN\n SELECT INTO last_id, first_id, counter\n first_value(id) OVER (),\n last_value(id) OVER (),\n count(*) OVER ()\n FROM results_page;\nELSE\n SELECT INTO first_id, last_id, counter\n first_value(id) OVER (),\n last_value(id) OVER (),\n count(*) OVER ()\n FROM results_page;\nEND IF;\nRAISE NOTICE 'firstid: %, lastid %', first_id, last_id;\nRAISE NOTICE 'timing after min/max: %', age(clock_timestamp(), qstart);\n\n\n\n\nIF counter > _limit THEN\n next_id := last_id;\n RAISE NOTICE 'next_id: %', next_id;\nELSE\n RAISE NOTICE 'No more next';\nEND IF;\n\nIF tok_q = 'TRUE' THEN\n RAISE NOTICE 'Not a paging query, no previous item';\nELSE\n RAISE NOTICE 'Getting previous item id';\n RAISE NOTICE 'timing: %', age(clock_timestamp(), qstart);\n SELECT INTO _token_record * FROM items WHERE id=first_id;\n IF\n _dtsort = 'DESC'\n THEN\n _token_dtrange := _dtrange * tstzrange(_token_record.datetime, 'infinity');\n ELSE\n _token_dtrange := _dtrange * tstzrange('-infinity',_token_record.datetime);\n END IF;\n RAISE NOTICE '% %', _token_dtrange, _dtrange;\n SELECT id INTO prev_id FROM items_by_partition(\n concat(whereq, ' AND ', filter_by_order(first_id, _search->'sortby', 'prev')),\n _token_dtrange,\n _rsort,\n 1\n );\n RAISE NOTICE 'timing: %', age(clock_timestamp(), qstart);\n\n RAISE NOTICE 'prev_id: %', prev_id;\nEND IF;\n\n\nRETURN QUERY\nWITH features AS (\n SELECT filter_jsonb(content, includes, excludes) as content\n FROM results_page LIMIT _limit\n),\nj AS (SELECT jsonb_agg(content) as feature_arr FROM features)\nSELECT jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce (\n CASE WHEN is_prev THEN flip_jsonb_array(feature_arr) ELSE feature_arr END\n ,'[]'::jsonb),\n 'links', links,\n 'timeStamp', now(),\n 'next', next_id,\n 'prev', prev_id\n)\nFROM j\n;\n\n\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\nINSERT INTO migrations (version) VALUES ('0.2.4');\n" }, { "path": "src/pgstac/migrations/pgstac.0.2.5-0.2.7.sql", "content": "SET SEARCH_PATH TO pgstac, public;\nBEGIN;\nCREATE OR REPLACE FUNCTION stac_query_op(att text, _op text, val jsonb) RETURNS text AS $$\nDECLARE\nret text := '';\nop text;\njp text;\natt_parts RECORD;\nval_str text;\nprop_path text;\nBEGIN\nval_str := lower(jsonb_build_object('a',val)->>'a');\nRAISE NOTICE 'val_str %', val_str;\n\natt_parts := split_stac_path(att);\nprop_path := replace(att_parts.dotpath, 'properties.', '');\n\nop := CASE _op\n WHEN 'eq' THEN '='\n WHEN 'gte' THEN '>='\n WHEN 'gt' THEN '>'\n WHEN 'lte' THEN '<='\n WHEN 'lt' THEN '<'\n WHEN 'ne' THEN '!='\n WHEN 'neq' THEN '!='\n WHEN 'startsWith' THEN 'LIKE'\n WHEN 'endsWith' THEN 'LIKE'\n WHEN 'contains' THEN 'LIKE'\n ELSE _op\nEND;\n\nval_str := CASE _op\n WHEN 'startsWith' THEN concat(val_str, '%')\n WHEN 'endsWith' THEN concat('%', val_str)\n WHEN 'contains' THEN concat('%',val_str,'%')\n ELSE val_str\nEND;\n\n\nRAISE NOTICE 'att_parts: % %', att_parts, count_by_delim(att_parts.dotpath,'\\.');\nIF\n op = '='\n AND att_parts.col = 'properties'\n --AND count_by_delim(att_parts.dotpath,'\\.') = 2\nTHEN\n -- use jsonpath query to leverage index for eqaulity tests on single level deep properties\n jp := btrim(format($jp$ $.%I[*] ? ( @ == %s ) $jp$, replace(att_parts.dotpath, 'properties.',''), lower(val::text)::jsonb));\n raise notice 'jp: %', jp;\n ret := format($q$ properties @? %L $q$, jp);\nELSIF jsonb_typeof(val) = 'number' THEN\n ret := format('properties ? %L AND (%s)::numeric %s %s', prop_path, att_parts.jspathtext, op, val);\nELSE\n ret := format('properties ? %L AND %s %s %L', prop_path ,att_parts.jspathtext, op, val_str);\nEND IF;\nRAISE NOTICE 'Op Query: %', ret;\n\nreturn ret;\nEND;\n$$ LANGUAGE PLPGSQL;\nINSERT INTO migrations (version) VALUES ('0.2.7');\n\nCOMMIT;\n" }, { "path": "src/pgstac/migrations/pgstac.0.2.5.sql", "content": "CREATE EXTENSION IF NOT EXISTS postgis;\nCREATE SCHEMA IF NOT EXISTS partman;\nCREATE EXTENSION IF NOT EXISTS pg_partman SCHEMA partman;\nCREATE SCHEMA IF NOT EXISTS pgstac;\n\n\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE TABLE migrations (\n version text,\n datetime timestamptz DEFAULT now() NOT NULL\n);\n\n/* converts a jsonb text array to a pg text[] array */\nCREATE OR REPLACE FUNCTION textarr(_js jsonb)\n RETURNS text[] AS $$\n SELECT ARRAY(SELECT jsonb_array_elements_text(_js));\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\n/*\nconverts a jsonb text array to comma delimited list of identifer quoted\nuseful for constructing column lists for selects\n*/\nCREATE OR REPLACE FUNCTION array_idents(_js jsonb)\n RETURNS text AS $$\n SELECT string_agg(quote_ident(v),',') FROM jsonb_array_elements_text(_js) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\n\n/* looks for a geometry in a stac item first from geometry and falling back to bbox */\nCREATE OR REPLACE FUNCTION stac_geom(value jsonb) RETURNS geometry AS $$\nSELECT\n CASE\n WHEN value->>'geometry' IS NOT NULL THEN\n ST_GeomFromGeoJSON(value->>'geometry')\n WHEN value->>'bbox' IS NOT NULL THEN\n ST_MakeEnvelope(\n (value->'bbox'->>0)::float,\n (value->'bbox'->>1)::float,\n (value->'bbox'->>2)::float,\n (value->'bbox'->>3)::float,\n 4326\n )\n ELSE NULL\n END as geometry\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION stac_datetime(value jsonb) RETURNS timestamptz AS $$\nSELECT (value->'properties'->>'datetime')::timestamptz;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION jsonb_paths (IN jdata jsonb, OUT path text[], OUT value jsonb) RETURNS\nSETOF RECORD AS $$\nwith recursive extract_all as\n(\n select\n ARRAY[key]::text[] as path,\n value\n FROM jsonb_each(jdata)\nunion all\n select\n path || coalesce(obj_key, (arr_key- 1)::text),\n coalesce(obj_value, arr_value)\n from extract_all\n left join lateral\n jsonb_each(case jsonb_typeof(value) when 'object' then value end)\n as o(obj_key, obj_value)\n on jsonb_typeof(value) = 'object'\n left join lateral\n jsonb_array_elements(case jsonb_typeof(value) when 'array' then value end)\n with ordinality as a(arr_value, arr_key)\n on jsonb_typeof(value) = 'array'\n where obj_key is not null or arr_key is not null\n)\nselect *\nfrom extract_all;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION jsonb_obj_paths (IN jdata jsonb, OUT path text[], OUT value jsonb) RETURNS\nSETOF RECORD AS $$\nwith recursive extract_all as\n(\n select\n ARRAY[key]::text[] as path,\n value\n FROM jsonb_each(jdata)\nunion all\n select\n path || obj_key,\n obj_value\n from extract_all\n left join lateral\n jsonb_each(case jsonb_typeof(value) when 'object' then value end)\n as o(obj_key, obj_value)\n on jsonb_typeof(value) = 'object'\n where obj_key is not null\n)\nselect *\nfrom extract_all;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION jsonb_val_paths (IN jdata jsonb, OUT path text[], OUT value jsonb) RETURNS\nSETOF RECORD AS $$\nSELECT * FROM jsonb_obj_paths(jdata) WHERE jsonb_typeof(value) not in ('object','array');\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION path_includes(IN path text[], IN includes text[]) RETURNS BOOLEAN AS $$\nWITH t AS (SELECT unnest(includes) i)\nSELECT EXISTS (\n SELECT 1 FROM t WHERE path @> string_to_array(trim(i), '.')\n);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION path_excludes(IN path text[], IN excludes text[]) RETURNS BOOLEAN AS $$\nWITH t AS (SELECT unnest(excludes) e)\nSELECT NOT EXISTS (\n SELECT 1 FROM t WHERE path @> string_to_array(trim(e), '.')\n);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION jsonb_obj_paths_filtered (\n IN jdata jsonb,\n IN includes text[] DEFAULT ARRAY[]::text[],\n IN excludes text[] DEFAULT ARRAY[]::text[],\n OUT path text[],\n OUT value jsonb\n) RETURNS\nSETOF RECORD AS $$\nSELECT path, value\nFROM jsonb_obj_paths(jdata)\nWHERE\n CASE WHEN cardinality(includes) > 0 THEN path_includes(path, includes) ELSE TRUE END\n AND\n path_excludes(path, excludes)\n\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION empty_arr(ANYARRAY) RETURNS BOOLEAN AS $$\nSELECT CASE\n WHEN $1 IS NULL THEN TRUE\n WHEN cardinality($1)<1 THEN TRUE\nELSE FALSE\nEND;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION filter_jsonb(\n IN jdata jsonb,\n IN includes text[] DEFAULT ARRAY[]::text[],\n IN excludes text[] DEFAULT ARRAY[]::text[]\n) RETURNS jsonb AS $$\nDECLARE\nrec RECORD;\noutj jsonb := '{}'::jsonb;\ncreated_paths text[] := '{}'::text[];\nBEGIN\n\nIF empty_arr(includes) AND empty_arr(excludes) THEN\nRAISE NOTICE 'no filter';\n RETURN jdata;\nEND IF;\nFOR rec in\nSELECT * FROM jsonb_obj_paths_filtered(jdata, includes, excludes)\nWHERE jsonb_typeof(value) != 'object'\nLOOP\n IF array_length(rec.path,1)>1 THEN\n FOR i IN 1..(array_length(rec.path,1)-1) LOOP\n IF NOT array_to_string(rec.path[1:i],'.') = ANY (created_paths) THEN\n outj := jsonb_set(outj, rec.path[1:i],'{}', true);\n created_paths := created_paths || array_to_string(rec.path[1:i],'.');\n END IF;\n END LOOP;\n END IF;\n outj := jsonb_set(outj, rec.path, rec.value, true);\n created_paths := created_paths || array_to_string(rec.path,'.');\nEND LOOP;\nRETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION properties_idx(_in jsonb) RETURNS jsonb AS $$\nWITH t AS (\n select array_to_string(path,'.') as path, lower(value::text)::jsonb as lowerval\n FROM jsonb_val_paths(_in)\n WHERE array_to_string(path,'.') not in ('datetime')\n)\nSELECT jsonb_object_agg(path, lowerval) FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE TABLE IF NOT EXISTS collections (\n id VARCHAR GENERATED ALWAYS AS (content->>'id') STORED PRIMARY KEY,\n content JSONB\n);\n\nCREATE OR REPLACE FUNCTION create_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection(data jsonb) RETURNS VOID AS $$\nDECLARE\nout collections%ROWTYPE;\nBEGIN\n UPDATE collections SET content=data WHERE id = data->>'id' RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ON CONFLICT (id) DO\n UPDATE\n SET content=EXCLUDED.content\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION delete_collection(_id text) RETURNS VOID AS $$\nDECLARE\nout collections%ROWTYPE;\nBEGIN\n DELETE FROM collections WHERE id = _id RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION get_collection(id text) RETURNS jsonb AS $$\nSELECT content FROM collections\nWHERE id=$1\n;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION all_collections() RETURNS jsonb AS $$\nSELECT jsonb_agg(content) FROM collections;\n;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\n\n\n\n\n\n/* CREATE OR REPLACE FUNCTION collections_trigger_func()\nRETURNS TRIGGER AS $$\nBEGIN\n IF pg_trigger_depth() = 1 THEN\n PERFORM create_collection(NEW.content);\n RETURN NULL;\n END IF;\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE TRIGGER collections_trigger\nBEFORE INSERT ON collections\nFOR EACH ROW EXECUTE PROCEDURE collections_trigger_func();\n */\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE TABLE IF NOT EXISTS items (\n id VARCHAR GENERATED ALWAYS AS (content->>'id') STORED NOT NULL,\n geometry geometry GENERATED ALWAYS AS (stac_geom(content)) STORED NOT NULL,\n properties jsonb GENERATED ALWAYS as (properties_idx(content->'properties')) STORED,\n collection_id text GENERATED ALWAYS AS (content->>'collection') STORED NOT NULL,\n datetime timestamptz GENERATED ALWAYS AS (stac_datetime(content)) STORED NOT NULL,\n content JSONB NOT NULL\n)\nPARTITION BY RANGE (stac_datetime(content))\n;\n\nALTER TABLE items ADD constraint items_collections_fk FOREIGN KEY (collection_id) REFERENCES collections(id) DEFERRABLE;\n\nCREATE TABLE items_template (\n LIKE items\n);\n\nALTER TABLE items_template ADD PRIMARY KEY (id);\n\n/*\nCREATE TABLE IF NOT EXISTS items_search (\n id text NOT NULL,\n geometry geometry NOT NULL,\n properties jsonb,\n collection_id text NOT NULL,\n datetime timestamptz NOT NULL\n)\nPARTITION BY RANGE (datetime)\n;\n\nCREATE TABLE IF NOT EXISTS items_search_template (\n LIKE items_search\n)\n;\nALTER TABLE items_search_template ADD PRIMARY KEY (id);\n*/\n\nDELETE from partman.part_config WHERE parent_table = 'pgstac.items';\nSELECT partman.create_parent(\n 'pgstac.items',\n 'datetime',\n 'native',\n 'weekly',\n p_template_table := 'pgstac.items_template',\n p_premake := 4\n);\n\nCREATE OR REPLACE FUNCTION make_partitions(st timestamptz, et timestamptz DEFAULT NULL) RETURNS BOOL AS $$\nWITH t AS (\n SELECT\n generate_series(\n date_trunc('week',st),\n date_trunc('week', coalesce(et, st)),\n '1 week'::interval\n ) w\n),\nw AS (SELECT array_agg(w) as w FROM t)\nSELECT CASE WHEN w IS NULL THEN NULL ELSE partman.create_partition_time('pgstac.items', w, true) END FROM w;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_partition(timestamptz) RETURNS text AS $$\nSELECT to_char($1, '\"items_p\"IYYY\"w\"IW');\n$$ LANGUAGE SQL;\n\nCREATE INDEX \"datetime_id_idx\" ON items (datetime, id);\nCREATE INDEX \"properties_idx\" ON items USING GIN (properties);\nCREATE INDEX \"collection_idx\" ON items (collection_id);\nCREATE INDEX \"geometry_idx\" ON items USING GIST (geometry);\n\n\nCREATE TYPE item AS (\n id text,\n geometry geometry,\n properties JSONB,\n collection_id text,\n datetime timestamptz\n);\n\n\n/*\nConverts single feature into an items row\n*/\n\n/*\nCREATE OR REPLACE FUNCTION feature_to_item(value jsonb) RETURNS item AS $$\n SELECT\n value->>'id' as id,\n CASE\n WHEN value->>'geometry' IS NOT NULL THEN\n ST_GeomFromGeoJSON(value->>'geometry')\n WHEN value->>'bbox' IS NOT NULL THEN\n ST_MakeEnvelope(\n (value->'bbox'->>0)::float,\n (value->'bbox'->>1)::float,\n (value->'bbox'->>2)::float,\n (value->'bbox'->>3)::float,\n 4326\n )\n ELSE NULL\n END as geometry,\n properties_idx(value ->'properties') as properties,\n value->>'collection' as collection_id,\n (value->'properties'->>'datetime')::timestamptz as datetime\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac,public;\n*/\n/*\nTakes a single feature, an array of features, or a feature collection\nand returns a set up individual items rows\n*/\n/*\nCREATE OR REPLACE FUNCTION features_to_items(value jsonb) RETURNS SETOF item AS $$\n WITH features AS (\n SELECT\n jsonb_array_elements(\n CASE\n WHEN jsonb_typeof(value) = 'array' THEN value\n WHEN value->>'type' = 'Feature' THEN '[]'::jsonb || value\n WHEN value->>'type' = 'FeatureCollection' THEN value->'features'\n ELSE NULL\n END\n ) as value\n )\n SELECT feature_to_item(value) FROM features\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac,public;\n*/\n\nCREATE OR REPLACE FUNCTION get_item(_id text) RETURNS jsonb AS $$\n SELECT content FROM items WHERE id=_id;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION delete_item(_id text) RETURNS VOID AS $$\nDECLARE\nout items%ROWTYPE;\nBEGIN\n DELETE FROM items WHERE id = _id RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION create_item(data jsonb) RETURNS VOID AS $$\n SELECT make_partitions(stac_datetime(data));\n INSERT INTO items (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION update_item(data jsonb) RETURNS VOID AS $$\nDECLARE\nout items%ROWTYPE;\nBEGIN\n UPDATE items SET content=data WHERE id = data->>'id' RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_item(data jsonb) RETURNS VOID AS $$\nDECLARE\npartition text;\nq text;\nnewcontent jsonb;\nBEGIN\n PERFORM make_partitions(stac_datetime(data));\n partition := get_partition(stac_datetime(data));\n q := format($q$\n INSERT INTO %I (content) VALUES ($1)\n ON CONFLICT (id) DO\n UPDATE SET content = EXCLUDED.content\n WHERE %I.content IS DISTINCT FROM EXCLUDED.content RETURNING content;\n $q$, partition, partition);\n EXECUTE q INTO newcontent USING (data);\n RAISE NOTICE 'newcontent: %', newcontent;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION analyze_empty_partitions() RETURNS VOID AS $$\nDECLARE\np text;\nBEGIN\nFOR p IN SELECT partition FROM all_items_partitions WHERE est_cnt = 0 LOOP\n RAISE NOTICE 'Analyzing %', p;\n EXECUTE format('ANALYZE %I;', p);\nEND LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n/* Trigger Function to cascade inserts/updates/deletes\nfrom items table to items_search table */\n/*\nALTER TABLE items_search ADD CONSTRAINT items_search_fk\nFOREIGN KEY (id) REFERENCES items(id)\nON DELETE CASCADE DEFERRABLE;\n\nCREATE OR REPLACE FUNCTION items_trigger_func()\nRETURNS TRIGGER AS $$\nDECLARE\nBEGIN\n IF TG_OP = 'UPDATE' THEN\n RAISE NOTICE 'DELETING % BEFORE UPDATE', OLD;\n DELETE FROM items_search WHERE id = OLD.id AND datetime = (OLD.content->'properties'->>'datetime')::timestamptz;\n END IF;\n\n INSERT INTO items_search SELECT * FROM feature_to_item(NEW.content);\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nDROP TRIGGER IF EXISTS items_insert_trigger ON items;\nCREATE TRIGGER items_insert_trigger\nAFTER INSERT ON items\nFOR EACH ROW EXECUTE PROCEDURE items_trigger_func();\n\nDROP TRIGGER IF EXISTS items_update_trigger ON items;\nCREATE TRIGGER items_update_trigger\nAFTER UPDATE ON items\nFOR EACH ROW\nWHEN (NEW.content IS DISTINCT FROM OLD.content)\nEXECUTE PROCEDURE items_trigger_func();\n*/\n\n/* Trigger Function to cascade inserts/updates/deletes\nfrom items table to items_search table */\n/*\nCREATE OR REPLACE FUNCTION items_search_trigger_delete_func()\nRETURNS TRIGGER AS $$\nDECLARE\nBEGIN\n RAISE NOTICE 'Deleting from items_search: % Depth: %', OLD, pg_trigger_depth();\n IF pg_trigger_depth()<3 THEN\n RAISE NOTICE 'DELETING WITH datetime';\n DELETE FROM items_search WHERE id=OLD.id AND datetime=OLD.datetime;\n RETURN NULL;\n END IF;\n RETURN OLD;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nDROP TRIGGER IF EXISTS items_search_delete_trigger ON items_search;\nCREATE TRIGGER items_search_delete_trigger\nBEFORE DELETE ON items_search\nFOR EACH ROW EXECUTE PROCEDURE items_search_trigger_delete_func();\n*/\nCREATE OR REPLACE FUNCTION backfill_partitions()\nRETURNS VOID AS $$\nDECLARE\nBEGIN\n IF EXISTS (SELECT 1 FROM items_default LIMIT 1) THEN\n RAISE NOTICE 'Creating new partitions and moving data from default';\n CREATE TEMP TABLE items_default_tmp ON COMMIT DROP AS SELECT datetime, content FROM items_default;\n TRUNCATE items_default;\n PERFORM make_partitions(min(datetime), max(datetime)) FROM items_default_tmp;\n INSERT INTO items (content) SELECT content FROM items_default_tmp;\n END IF;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION items_trigger_stmt_func()\nRETURNS TRIGGER AS $$\nDECLARE\nBEGIN\n PERFORM analyze_empty_partitions();\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nDROP TRIGGER IF EXISTS items_stmt_trigger ON items;\nCREATE TRIGGER items_stmt_trigger\nAFTER INSERT OR UPDATE OR DELETE ON items\nFOR EACH STATEMENT EXECUTE PROCEDURE items_trigger_stmt_func();\n\n\n/*\nView to get a table of available items partitions\nwith date ranges\n*/\n--DROP VIEW IF EXISTS all_items_partitions CASCADE;\nCREATE OR REPLACE VIEW all_items_partitions AS\nWITH base AS\n(SELECT\n c.oid::pg_catalog.regclass::text as partition,\n pg_catalog.pg_get_expr(c.relpartbound, c.oid) as _constraint,\n regexp_matches(\n pg_catalog.pg_get_expr(c.relpartbound, c.oid),\n E'\\\\(''\\([0-9 :+-]*\\)''\\\\).*\\\\(''\\([0-9 :+-]*\\)''\\\\)'\n ) as t,\n reltuples::bigint as est_cnt\nFROM pg_catalog.pg_class c, pg_catalog.pg_inherits i\nWHERE c.oid = i.inhrelid AND i.inhparent = 'items'::regclass)\nSELECT partition, tstzrange(\n t[1]::timestamptz,\n t[2]::timestamptz\n), est_cnt\nFROM base\nORDER BY 2 desc;\n\n--DROP VIEW IF EXISTS items_partitions;\nCREATE OR REPLACE VIEW items_partitions AS\nSELECT * FROM all_items_partitions WHERE est_cnt>0;\n\nCREATE OR REPLACE FUNCTION collection_bbox(id text) RETURNS jsonb AS $$\nSELECT (replace(replace(replace(st_extent(geometry)::text,'BOX(','[['),')',']]'),' ',','))::jsonb\nFROM items WHERE collection_id=$1;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION collection_temporal_extent(id text) RETURNS jsonb AS $$\nSELECT to_jsonb(array[array[min(datetime)::text, max(datetime)::text]])\nFROM items WHERE collection_id=$1;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection_extents() RETURNS VOID AS $$\nUPDATE collections SET\n content = content ||\n jsonb_build_object(\n 'extent', jsonb_build_object(\n 'spatial', jsonb_build_object(\n 'bbox', collection_bbox(collections.id)\n ),\n 'temporal', jsonb_build_object(\n 'interval', collection_temporal_extent(collections.id)\n )\n )\n )\n;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\nSET SEARCH_PATH TO pgstac, public;\n\n/*\nView to get a table of available items partitions\nwith date ranges\n*/\nDROP VIEW IF EXISTS items_partitions;\nCREATE VIEW items_partitions AS\nWITH base AS\n(SELECT\n c.oid::pg_catalog.regclass::text as partition,\n pg_catalog.pg_get_expr(c.relpartbound, c.oid) as _constraint,\n regexp_matches(\n pg_catalog.pg_get_expr(c.relpartbound, c.oid),\n E'\\\\(''\\([0-9 :+-]*\\)''\\\\).*\\\\(''\\([0-9 :+-]*\\)''\\\\)'\n ) as t,\n reltuples::bigint as est_cnt\nFROM pg_catalog.pg_class c, pg_catalog.pg_inherits i\nWHERE c.oid = i.inhrelid AND i.inhparent = 'items'::regclass)\nSELECT partition, tstzrange(\n t[1]::timestamptz,\n t[2]::timestamptz\n), est_cnt\nFROM base\nWHERE est_cnt >0\nORDER BY 2 desc;\n\n\nCREATE OR REPLACE FUNCTION items_by_partition(\n IN _where text DEFAULT 'TRUE',\n IN _dtrange tstzrange DEFAULT tstzrange('-infinity','infinity'),\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN _limit int DEFAULT 10\n) RETURNS SETOF items AS $$\nDECLARE\npartition_query text;\nmain_query text;\nbatchcount int;\ncounter int := 0;\np record;\nBEGIN\nIF _orderby ILIKE 'datetime d%' THEN\n partition_query := format($q$\n SELECT partition\n FROM items_partitions\n WHERE tstzrange && $1\n ORDER BY tstzrange DESC;\n $q$);\nELSIF _orderby ILIKE 'datetime a%' THEN\n partition_query := format($q$\n SELECT partition\n FROM items_partitions\n WHERE tstzrange && $1\n ORDER BY tstzrange ASC\n ;\n $q$);\nELSE\n partition_query := format($q$\n SELECT 'items' as partition WHERE $1 IS NOT NULL;\n $q$);\nEND IF;\nRAISE NOTICE 'Partition Query: %', partition_query;\nFOR p IN\n EXECUTE partition_query USING (_dtrange)\nLOOP\n IF lower(_dtrange)::timestamptz > '-infinity' THEN\n _where := concat(_where,format(' AND datetime >= %L',lower(_dtrange)::timestamptz::text));\n END IF;\n IF upper(_dtrange)::timestamptz < 'infinity' THEN\n _where := concat(_where,format(' AND datetime <= %L',upper(_dtrange)::timestamptz::text));\n END IF;\n\n main_query := format($q$\n SELECT * FROM %I\n WHERE %s\n ORDER BY %s\n LIMIT %s - $1\n $q$, p.partition::text, _where, _orderby, _limit\n );\n RAISE NOTICE 'Partition Query %', main_query;\n RAISE NOTICE '%', counter;\n RETURN QUERY EXECUTE main_query USING counter;\n\n GET DIAGNOSTICS batchcount = ROW_COUNT;\n counter := counter + batchcount;\n RAISE NOTICE 'FOUND %', batchcount;\n IF counter >= _limit THEN\n EXIT;\n END IF;\n RAISE NOTICE 'ADDED % FOR A TOTAL OF %', batchcount, counter;\nEND LOOP;\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION split_stac_path(IN path text, OUT col text, OUT dotpath text, OUT jspath text, OUT jspathtext text) AS $$\nWITH col AS (\n SELECT\n CASE WHEN\n split_part(path, '.', 1) IN ('id', 'stac_version', 'stac_extensions','geometry','properties','assets','collection_id','datetime','links', 'extra_fields') THEN split_part(path, '.', 1)\n ELSE 'properties'\n END AS col\n),\ndp AS (\n SELECT\n col, ltrim(replace(path, col , ''),'.') as dotpath\n FROM col\n),\npaths AS (\nSELECT\n col, dotpath,\n regexp_split_to_table(dotpath,E'\\\\.') as path FROM dp\n) SELECT\n col,\n btrim(concat(col,'.',dotpath),'.'),\n CASE WHEN btrim(concat(col,'.',dotpath),'.') != col THEN concat(col,'->',string_agg(concat('''',path,''''),'->')) ELSE col END,\n regexp_replace(\n CASE WHEN btrim(concat(col,'.',dotpath),'.') != col THEN concat(col,'->',string_agg(concat('''',path,''''),'->')) ELSE col END,\n E'>([^>]*)$','>>\\1'\n )\nFROM paths group by col, dotpath;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n/* Functions for searching items */\nCREATE OR REPLACE FUNCTION sort_base(\n IN _sort jsonb DEFAULT '[{\"field\":\"datetime\",\"direction\":\"desc\"}]',\n OUT key text,\n OUT col text,\n OUT dir text,\n OUT rdir text,\n OUT sort text,\n OUT rsort text\n) RETURNS SETOF RECORD AS $$\nWITH sorts AS (\n SELECT\n value->>'field' as key,\n (split_stac_path(value->>'field')).jspathtext as col,\n coalesce(upper(value->>'direction'),'ASC') as dir\n FROM jsonb_array_elements('[]'::jsonb || coalesce(_sort,'[{\"field\":\"datetime\",\"direction\":\"desc\"}]') )\n)\nSELECT\n key,\n col,\n dir,\n CASE dir WHEN 'DESC' THEN 'ASC' ELSE 'ASC' END as rdir,\n concat(col, ' ', dir, ' NULLS LAST ') AS sort,\n concat(col,' ', CASE dir WHEN 'DESC' THEN 'ASC' ELSE 'ASC' END, ' NULLS LAST ') AS rsort\nFROM sorts\nUNION ALL\nSELECT 'id', 'id', 'DESC', 'ASC', 'id DESC', 'id ASC'\n;\n$$ LANGUAGE SQL;\n\n\nCREATE OR REPLACE FUNCTION sort(_sort jsonb) RETURNS text AS $$\nSELECT string_agg(sort,', ') FROM sort_base(_sort);\n$$ LANGUAGE SQL PARALLEL SAFE SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION rsort(_sort jsonb) RETURNS text AS $$\nSELECT string_agg(rsort,', ') FROM sort_base(_sort);\n$$ LANGUAGE SQL PARALLEL SAFE SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION bbox_geom(_bbox jsonb) RETURNS geometry AS $$\nSELECT CASE jsonb_array_length(_bbox)\n WHEN 4 THEN\n ST_SetSRID(ST_MakeEnvelope(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float,\n (_bbox->>3)::float\n ),4326)\n WHEN 6 THEN\n ST_SetSRID(ST_3DMakeBox(\n ST_MakePoint(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float\n ),\n ST_MakePoint(\n (_bbox->>3)::float,\n (_bbox->>4)::float,\n (_bbox->>5)::float\n )\n ),4326)\n ELSE null END;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION in_array_q(col text, arr jsonb) RETURNS text AS $$\nSELECT CASE jsonb_typeof(arr) WHEN 'array' THEN format('%I = ANY(textarr(%L))', col, arr) ELSE format('%I = %L', col, arr) END;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION count_by_delim(text, text) RETURNS int AS $$\nSELECT count(*) FROM regexp_split_to_table($1,$2);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION stac_query_op(att text, _op text, val jsonb) RETURNS text AS $$\nDECLARE\nret text := '';\nop text;\njp text;\natt_parts RECORD;\nval_str text;\nprop_path text;\nBEGIN\nval_str := lower(jsonb_build_object('a',val)->>'a');\nRAISE NOTICE 'val_str %', val_str;\n\natt_parts := split_stac_path(att);\nprop_path := replace(att_parts.dotpath, 'properties.', '');\n\nop := CASE _op\n WHEN 'eq' THEN '='\n WHEN 'gte' THEN '>='\n WHEN 'gt' THEN '>'\n WHEN 'lte' THEN '<='\n WHEN 'lt' THEN '<'\n WHEN 'ne' THEN '!='\n WHEN 'neq' THEN '!='\n WHEN 'startsWith' THEN 'LIKE'\n WHEN 'endsWith' THEN 'LIKE'\n WHEN 'contains' THEN 'LIKE'\n ELSE _op\nEND;\n\nval_str := CASE _op\n WHEN 'startsWith' THEN concat(val_str, '%')\n WHEN 'endsWith' THEN concat('%', val_str)\n WHEN 'contains' THEN concat('%',val_str,'%')\n ELSE val_str\nEND;\n\n\nRAISE NOTICE 'att_parts: % %', att_parts, count_by_delim(att_parts.dotpath,'\\.');\nIF\n op = '='\n AND att_parts.col = 'properties'\n --AND count_by_delim(att_parts.dotpath,'\\.') = 2\nTHEN\n -- use jsonpath query to leverage index for eqaulity tests on single level deep properties\n jp := btrim(format($jp$ $.%I[*] ? ( @ == %s ) $jp$, replace(att_parts.dotpath, 'properties.',''), lower(val::text)::jsonb));\n raise notice 'jp: %', jp;\n ret := format($q$ properties @? %L $q$, jp);\nELSIF jsonb_typeof(val) = 'number' THEN\n ret := format('properties ? %L AND (%s)::numeric %s %s', prop_path, att_parts.jspathtext, op, val);\nELSE\n ret := format('properties ? %L AND %s %s %L', prop_path ,att_parts.jspathtext, op, val_str);\nEND IF;\nRAISE NOTICE 'Op Query: %', ret;\n\nreturn ret;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION stac_query(_query jsonb) RETURNS TEXT[] AS $$\nDECLARE\nqa text[];\natt text;\nops jsonb;\nop text;\nval jsonb;\nBEGIN\nFOR att, ops IN SELECT key, value FROM jsonb_each(_query)\nLOOP\n FOR op, val IN SELECT key, value FROM jsonb_each(ops)\n LOOP\n qa := array_append(qa, stac_query_op(att,op, val));\n RAISE NOTICE '% % %', att, op, val;\n END LOOP;\nEND LOOP;\nRETURN qa;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION filter_by_order(item_id text, _sort jsonb, _type text) RETURNS text AS $$\nDECLARE\nitem item;\nBEGIN\nSELECT * INTO item FROM items WHERE id=item_id;\nRETURN filter_by_order(item, _sort, _type);\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n-- Used to create filters used for paging using the items id from the token\nCREATE OR REPLACE FUNCTION filter_by_order(_item item, _sort jsonb, _type text) RETURNS text AS $$\nDECLARE\nsorts RECORD;\nfilts text[];\nitemval text;\nop text;\nidop text;\nret text;\neq_flag text;\n_item_j jsonb := to_jsonb(_item);\nBEGIN\nFOR sorts IN SELECT * FROM sort_base(_sort) LOOP\n IF sorts.col = 'datetime' THEN\n CONTINUE;\n END IF;\n IF sorts.col='id' AND _type IN ('prev','next') THEN\n eq_flag := '';\n ELSE\n eq_flag := '=';\n END IF;\n\n op := concat(\n CASE\n WHEN _type in ('prev','first') AND sorts.dir = 'ASC' THEN '<'\n WHEN _type in ('last','next') AND sorts.dir = 'ASC' THEN '>'\n WHEN _type in ('prev','first') AND sorts.dir = 'DESC' THEN '>'\n WHEN _type in ('last','next') AND sorts.dir = 'DESC' THEN '<'\n END,\n eq_flag\n );\n\n IF _item_j ? sorts.col THEN\n filts = array_append(filts, format('%s %s %L', sorts.col, op, _item_j->>sorts.col));\n END IF;\nEND LOOP;\nret := coalesce(array_to_string(filts,' AND '), 'TRUE');\nRAISE NOTICE 'Order Filter %', ret;\nRETURN ret;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION search_dtrange(IN _indate jsonb, OUT _tstzrange tstzrange) AS\n$$\nWITH t AS (\n SELECT CASE\n WHEN jsonb_typeof(_indate) = 'array' THEN\n textarr(_indate)\n ELSE\n regexp_split_to_array(\n btrim(_indate::text,'\"'),\n '/'\n )\n END AS arr\n)\n, t1 AS (\n SELECT\n CASE\n WHEN array_upper(arr,1) = 1 OR arr[1] = '..' OR arr[1] IS NULL THEN '-infinity'::timestamptz\n ELSE arr[1]::timestamptz\n END AS st,\n CASE\n WHEN array_upper(arr,1) = 1 THEN arr[1]::timestamptz\n WHEN arr[2] = '..' OR arr[2] IS NULL THEN 'infinity'::timestamptz\n ELSE arr[2]::timestamptz\n END AS et\n FROM t\n)\nSELECT\n tstzrange(st,et)\nFROM t1;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION flip_jsonb_array(j jsonb) RETURNS jsonb AS $$\nWITH t AS (\n SELECT i, row_number() over () as r FROM jsonb_array_elements(j) i\n), o AS (\n SELECT i FROM t ORDER BY r DESC\n)\nSELECT jsonb_agg(i) from o\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION search(_search jsonb = '{}'::jsonb) RETURNS SETOF jsonb AS $$\nDECLARE\nqstart timestamptz := clock_timestamp();\n_sort text := '';\n_rsort text := '';\n_limit int := 10;\n_geom geometry;\nqa text[];\npq text[];\nquery text;\npq_prop record;\npq_op record;\nprev_id text := NULL;\nnext_id text := NULL;\nwhereq text := 'TRUE';\nlinks jsonb := '[]'::jsonb;\ntoken text;\ntok_val text;\ntok_q text := 'TRUE';\ntok_sort text;\nfirst_id text;\nfirst_dt timestamptz;\nlast_id text;\nsort text;\nrsort text;\ndt text[];\ndqa text[];\ndq text;\nmq_where text;\nstartdt timestamptz;\nenddt timestamptz;\nitem items%ROWTYPE;\ncounter int := 0;\nbatchcount int;\nmonth timestamptz;\nm record;\n_dtrange tstzrange := tstzrange('-infinity','infinity');\n_dtsort text;\n_token_dtrange tstzrange := tstzrange('-infinity','infinity');\n_token_record items%ROWTYPE;\nis_prev boolean := false;\nincludes text[];\nexcludes text[];\nBEGIN\n-- Create table from sort query of items to sort\nCREATE TEMP TABLE pgstac_tmp_sorts ON COMMIT DROP AS SELECT * FROM sort_base(_search->'sortby');\n\n-- Get the datetime sort direction, necessary for efficient cycling through partitions\nSELECT INTO _dtsort dir FROM pgstac_tmp_sorts WHERE key='datetime';\nRAISE NOTICE '_dtsort: %',_dtsort;\n\nSELECT INTO _sort string_agg(s.sort,', ') FROM pgstac_tmp_sorts s;\nSELECT INTO _rsort string_agg(s.rsort,', ') FROM pgstac_tmp_sorts s;\ntok_sort := _sort;\n\n\n-- Get datetime from query as a tstzrange\nIF _search ? 'datetime' THEN\n _dtrange := search_dtrange(_search->'datetime');\n _token_dtrange := _dtrange;\nEND IF;\n\n-- Get the paging token\nIF _search ? 'token' THEN\n token := _search->>'token';\n tok_val := substr(token,6);\n IF starts_with(token, 'prev:') THEN\n is_prev := true;\n END IF;\n SELECT INTO _token_record * FROM items WHERE id=tok_val;\n IF\n (is_prev AND _dtsort = 'DESC')\n OR\n (not is_prev AND _dtsort = 'ASC')\n THEN\n _token_dtrange := _dtrange * tstzrange(_token_record.datetime, 'infinity');\n ELSIF\n _dtsort IS NOT NULL\n THEN\n _token_dtrange := _dtrange * tstzrange('-infinity',_token_record.datetime);\n END IF;\n IF is_prev THEN\n tok_q := filter_by_order(tok_val, _search->'sortby', 'first');\n _sort := _rsort;\n ELSIF starts_with(token, 'next:') THEN\n tok_q := filter_by_order(tok_val, _search->'sortby', 'last');\n END IF;\nEND IF;\nRAISE NOTICE 'timing: %', age(clock_timestamp(), qstart);\nRAISE NOTICE 'tok_q: % _token_dtrange: %', tok_q, _token_dtrange;\n\nIF _search ? 'ids' THEN\n RAISE NOTICE 'searching solely based on ids... %',_search;\n qa := array_append(qa, in_array_q('id', _search->'ids'));\nELSE\n IF _search ? 'intersects' THEN\n _geom := ST_SetSRID(ST_GeomFromGeoJSON(_search->>'intersects'), 4326);\n ELSIF _search ? 'bbox' THEN\n _geom := bbox_geom(_search->'bbox');\n END IF;\n\n IF _geom IS NOT NULL THEN\n qa := array_append(qa, format('st_intersects(geometry, %L::geometry)',_geom));\n END IF;\n\n IF _search ? 'collections' THEN\n qa := array_append(qa, in_array_q('collection_id', _search->'collections'));\n END IF;\n\n IF _search ? 'query' THEN\n qa := array_cat(qa,\n stac_query(_search->'query')\n );\n END IF;\nEND IF;\n\nIF _search ? 'limit' THEN\n _limit := (_search->>'limit')::int;\nEND IF;\n\nIF _search ? 'fields' THEN\n IF _search->'fields' ? 'exclude' THEN\n excludes=textarr(_search->'fields'->'exclude');\n END IF;\n IF _search->'fields' ? 'include' THEN\n includes=textarr(_search->'fields'->'include');\n IF array_length(includes, 1)>0 AND NOT 'id' = ANY (includes) THEN\n includes = includes || '{id}';\n END IF;\n END IF;\n RAISE NOTICE 'Includes: %, Excludes: %', includes, excludes;\nEND IF;\n\nwhereq := COALESCE(array_to_string(qa,' AND '),' TRUE ');\ndq := COALESCE(array_to_string(dqa,' AND '),' TRUE ');\nRAISE NOTICE 'timing before temp table: %', age(clock_timestamp(), qstart);\n\nCREATE TEMP TABLE results_page ON COMMIT DROP AS\nSELECT * FROM items_by_partition(\n concat(whereq, ' AND ', tok_q),\n _token_dtrange,\n _sort,\n _limit + 1\n);\nRAISE NOTICE 'timing after temp table: %', age(clock_timestamp(), qstart);\n\nRAISE NOTICE 'timing before min/max: %', age(clock_timestamp(), qstart);\n\nIF is_prev THEN\n SELECT INTO last_id, first_id, counter\n first_value(id) OVER (),\n last_value(id) OVER (),\n count(*) OVER ()\n FROM results_page;\nELSE\n SELECT INTO first_id, last_id, counter\n first_value(id) OVER (),\n last_value(id) OVER (),\n count(*) OVER ()\n FROM results_page;\nEND IF;\nRAISE NOTICE 'firstid: %, lastid %', first_id, last_id;\nRAISE NOTICE 'timing after min/max: %', age(clock_timestamp(), qstart);\n\n\n\n\nIF counter > _limit THEN\n next_id := last_id;\n RAISE NOTICE 'next_id: %', next_id;\nELSE\n RAISE NOTICE 'No more next';\nEND IF;\n\nIF tok_q = 'TRUE' THEN\n RAISE NOTICE 'Not a paging query, no previous item';\nELSE\n RAISE NOTICE 'Getting previous item id';\n RAISE NOTICE 'timing: %', age(clock_timestamp(), qstart);\n SELECT INTO _token_record * FROM items WHERE id=first_id;\n IF\n _dtsort = 'DESC'\n THEN\n _token_dtrange := _dtrange * tstzrange(_token_record.datetime, 'infinity');\n ELSE\n _token_dtrange := _dtrange * tstzrange('-infinity',_token_record.datetime);\n END IF;\n RAISE NOTICE '% %', _token_dtrange, _dtrange;\n SELECT id INTO prev_id FROM items_by_partition(\n concat(whereq, ' AND ', filter_by_order(first_id, _search->'sortby', 'prev')),\n _token_dtrange,\n _rsort,\n 1\n );\n RAISE NOTICE 'timing: %', age(clock_timestamp(), qstart);\n\n RAISE NOTICE 'prev_id: %', prev_id;\nEND IF;\n\n\nRETURN QUERY\nWITH features AS (\n SELECT filter_jsonb(content, includes, excludes) as content\n FROM results_page LIMIT _limit\n),\nj AS (SELECT jsonb_agg(content) as feature_arr FROM features)\nSELECT jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce (\n CASE WHEN is_prev THEN flip_jsonb_array(feature_arr) ELSE feature_arr END\n ,'[]'::jsonb),\n 'links', links,\n 'timeStamp', now(),\n 'next', next_id,\n 'prev', prev_id\n)\nFROM j\n;\n\n\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\nINSERT INTO migrations (version) VALUES ('0.2.5');\n" }, { "path": "src/pgstac/migrations/pgstac.0.2.7-0.2.8.sql", "content": "SET SEARCH_PATH TO pgstac, public;\nBEGIN;\n\nINSERT INTO migrations (version) VALUES ('0.2.8');\n\nCOMMIT;\n" }, { "path": "src/pgstac/migrations/pgstac.0.2.7.sql", "content": "CREATE EXTENSION IF NOT EXISTS postgis;\nCREATE SCHEMA IF NOT EXISTS partman;\nCREATE EXTENSION IF NOT EXISTS pg_partman SCHEMA partman;\nCREATE SCHEMA IF NOT EXISTS pgstac;\n\n\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE TABLE migrations (\n version text,\n datetime timestamptz DEFAULT now() NOT NULL\n);\n\n/* converts a jsonb text array to a pg text[] array */\nCREATE OR REPLACE FUNCTION textarr(_js jsonb)\n RETURNS text[] AS $$\n SELECT ARRAY(SELECT jsonb_array_elements_text(_js));\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\n/*\nconverts a jsonb text array to comma delimited list of identifer quoted\nuseful for constructing column lists for selects\n*/\nCREATE OR REPLACE FUNCTION array_idents(_js jsonb)\n RETURNS text AS $$\n SELECT string_agg(quote_ident(v),',') FROM jsonb_array_elements_text(_js) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\n\n/* looks for a geometry in a stac item first from geometry and falling back to bbox */\nCREATE OR REPLACE FUNCTION stac_geom(value jsonb) RETURNS geometry AS $$\nSELECT\n CASE\n WHEN value->>'geometry' IS NOT NULL THEN\n ST_GeomFromGeoJSON(value->>'geometry')\n WHEN value->>'bbox' IS NOT NULL THEN\n ST_MakeEnvelope(\n (value->'bbox'->>0)::float,\n (value->'bbox'->>1)::float,\n (value->'bbox'->>2)::float,\n (value->'bbox'->>3)::float,\n 4326\n )\n ELSE NULL\n END as geometry\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION stac_datetime(value jsonb) RETURNS timestamptz AS $$\nSELECT (value->'properties'->>'datetime')::timestamptz;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION jsonb_paths (IN jdata jsonb, OUT path text[], OUT value jsonb) RETURNS\nSETOF RECORD AS $$\nwith recursive extract_all as\n(\n select\n ARRAY[key]::text[] as path,\n value\n FROM jsonb_each(jdata)\nunion all\n select\n path || coalesce(obj_key, (arr_key- 1)::text),\n coalesce(obj_value, arr_value)\n from extract_all\n left join lateral\n jsonb_each(case jsonb_typeof(value) when 'object' then value end)\n as o(obj_key, obj_value)\n on jsonb_typeof(value) = 'object'\n left join lateral\n jsonb_array_elements(case jsonb_typeof(value) when 'array' then value end)\n with ordinality as a(arr_value, arr_key)\n on jsonb_typeof(value) = 'array'\n where obj_key is not null or arr_key is not null\n)\nselect *\nfrom extract_all;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION jsonb_obj_paths (IN jdata jsonb, OUT path text[], OUT value jsonb) RETURNS\nSETOF RECORD AS $$\nwith recursive extract_all as\n(\n select\n ARRAY[key]::text[] as path,\n value\n FROM jsonb_each(jdata)\nunion all\n select\n path || obj_key,\n obj_value\n from extract_all\n left join lateral\n jsonb_each(case jsonb_typeof(value) when 'object' then value end)\n as o(obj_key, obj_value)\n on jsonb_typeof(value) = 'object'\n where obj_key is not null\n)\nselect *\nfrom extract_all;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION jsonb_val_paths (IN jdata jsonb, OUT path text[], OUT value jsonb) RETURNS\nSETOF RECORD AS $$\nSELECT * FROM jsonb_obj_paths(jdata) WHERE jsonb_typeof(value) not in ('object','array');\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION path_includes(IN path text[], IN includes text[]) RETURNS BOOLEAN AS $$\nWITH t AS (SELECT unnest(includes) i)\nSELECT EXISTS (\n SELECT 1 FROM t WHERE path @> string_to_array(trim(i), '.')\n);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION path_excludes(IN path text[], IN excludes text[]) RETURNS BOOLEAN AS $$\nWITH t AS (SELECT unnest(excludes) e)\nSELECT NOT EXISTS (\n SELECT 1 FROM t WHERE path @> string_to_array(trim(e), '.')\n);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION jsonb_obj_paths_filtered (\n IN jdata jsonb,\n IN includes text[] DEFAULT ARRAY[]::text[],\n IN excludes text[] DEFAULT ARRAY[]::text[],\n OUT path text[],\n OUT value jsonb\n) RETURNS\nSETOF RECORD AS $$\nSELECT path, value\nFROM jsonb_obj_paths(jdata)\nWHERE\n CASE WHEN cardinality(includes) > 0 THEN path_includes(path, includes) ELSE TRUE END\n AND\n path_excludes(path, excludes)\n\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION empty_arr(ANYARRAY) RETURNS BOOLEAN AS $$\nSELECT CASE\n WHEN $1 IS NULL THEN TRUE\n WHEN cardinality($1)<1 THEN TRUE\nELSE FALSE\nEND;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION filter_jsonb(\n IN jdata jsonb,\n IN includes text[] DEFAULT ARRAY[]::text[],\n IN excludes text[] DEFAULT ARRAY[]::text[]\n) RETURNS jsonb AS $$\nDECLARE\nrec RECORD;\noutj jsonb := '{}'::jsonb;\ncreated_paths text[] := '{}'::text[];\nBEGIN\n\nIF empty_arr(includes) AND empty_arr(excludes) THEN\nRAISE NOTICE 'no filter';\n RETURN jdata;\nEND IF;\nFOR rec in\nSELECT * FROM jsonb_obj_paths_filtered(jdata, includes, excludes)\nWHERE jsonb_typeof(value) != 'object'\nLOOP\n IF array_length(rec.path,1)>1 THEN\n FOR i IN 1..(array_length(rec.path,1)-1) LOOP\n IF NOT array_to_string(rec.path[1:i],'.') = ANY (created_paths) THEN\n outj := jsonb_set(outj, rec.path[1:i],'{}', true);\n created_paths := created_paths || array_to_string(rec.path[1:i],'.');\n END IF;\n END LOOP;\n END IF;\n outj := jsonb_set(outj, rec.path, rec.value, true);\n created_paths := created_paths || array_to_string(rec.path,'.');\nEND LOOP;\nRETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION properties_idx(_in jsonb) RETURNS jsonb AS $$\nWITH t AS (\n select array_to_string(path,'.') as path, lower(value::text)::jsonb as lowerval\n FROM jsonb_val_paths(_in)\n WHERE array_to_string(path,'.') not in ('datetime')\n)\nSELECT jsonb_object_agg(path, lowerval) FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE TABLE IF NOT EXISTS collections (\n id VARCHAR GENERATED ALWAYS AS (content->>'id') STORED PRIMARY KEY,\n content JSONB\n);\n\nCREATE OR REPLACE FUNCTION create_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection(data jsonb) RETURNS VOID AS $$\nDECLARE\nout collections%ROWTYPE;\nBEGIN\n UPDATE collections SET content=data WHERE id = data->>'id' RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ON CONFLICT (id) DO\n UPDATE\n SET content=EXCLUDED.content\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION delete_collection(_id text) RETURNS VOID AS $$\nDECLARE\nout collections%ROWTYPE;\nBEGIN\n DELETE FROM collections WHERE id = _id RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION get_collection(id text) RETURNS jsonb AS $$\nSELECT content FROM collections\nWHERE id=$1\n;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION all_collections() RETURNS jsonb AS $$\nSELECT jsonb_agg(content) FROM collections;\n;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\n\n\n\n\n\n/* CREATE OR REPLACE FUNCTION collections_trigger_func()\nRETURNS TRIGGER AS $$\nBEGIN\n IF pg_trigger_depth() = 1 THEN\n PERFORM create_collection(NEW.content);\n RETURN NULL;\n END IF;\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE TRIGGER collections_trigger\nBEFORE INSERT ON collections\nFOR EACH ROW EXECUTE PROCEDURE collections_trigger_func();\n */\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE TABLE IF NOT EXISTS items (\n id VARCHAR GENERATED ALWAYS AS (content->>'id') STORED NOT NULL,\n geometry geometry GENERATED ALWAYS AS (stac_geom(content)) STORED NOT NULL,\n properties jsonb GENERATED ALWAYS as (properties_idx(content->'properties')) STORED,\n collection_id text GENERATED ALWAYS AS (content->>'collection') STORED NOT NULL,\n datetime timestamptz GENERATED ALWAYS AS (stac_datetime(content)) STORED NOT NULL,\n content JSONB NOT NULL\n)\nPARTITION BY RANGE (stac_datetime(content))\n;\n\nALTER TABLE items ADD constraint items_collections_fk FOREIGN KEY (collection_id) REFERENCES collections(id) DEFERRABLE;\n\nCREATE TABLE items_template (\n LIKE items\n);\n\nALTER TABLE items_template ADD PRIMARY KEY (id);\n\n/*\nCREATE TABLE IF NOT EXISTS items_search (\n id text NOT NULL,\n geometry geometry NOT NULL,\n properties jsonb,\n collection_id text NOT NULL,\n datetime timestamptz NOT NULL\n)\nPARTITION BY RANGE (datetime)\n;\n\nCREATE TABLE IF NOT EXISTS items_search_template (\n LIKE items_search\n)\n;\nALTER TABLE items_search_template ADD PRIMARY KEY (id);\n*/\n\nDELETE from partman.part_config WHERE parent_table = 'pgstac.items';\nSELECT partman.create_parent(\n 'pgstac.items',\n 'datetime',\n 'native',\n 'weekly',\n p_template_table := 'pgstac.items_template',\n p_premake := 4\n);\n\nCREATE OR REPLACE FUNCTION make_partitions(st timestamptz, et timestamptz DEFAULT NULL) RETURNS BOOL AS $$\nWITH t AS (\n SELECT\n generate_series(\n date_trunc('week',st),\n date_trunc('week', coalesce(et, st)),\n '1 week'::interval\n ) w\n),\nw AS (SELECT array_agg(w) as w FROM t)\nSELECT CASE WHEN w IS NULL THEN NULL ELSE partman.create_partition_time('pgstac.items', w, true) END FROM w;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_partition(timestamptz) RETURNS text AS $$\nSELECT to_char($1, '\"items_p\"IYYY\"w\"IW');\n$$ LANGUAGE SQL;\n\nCREATE INDEX \"datetime_id_idx\" ON items (datetime, id);\nCREATE INDEX \"properties_idx\" ON items USING GIN (properties);\nCREATE INDEX \"collection_idx\" ON items (collection_id);\nCREATE INDEX \"geometry_idx\" ON items USING GIST (geometry);\n\n\nCREATE TYPE item AS (\n id text,\n geometry geometry,\n properties JSONB,\n collection_id text,\n datetime timestamptz\n);\n\n\n/*\nConverts single feature into an items row\n*/\n\n/*\nCREATE OR REPLACE FUNCTION feature_to_item(value jsonb) RETURNS item AS $$\n SELECT\n value->>'id' as id,\n CASE\n WHEN value->>'geometry' IS NOT NULL THEN\n ST_GeomFromGeoJSON(value->>'geometry')\n WHEN value->>'bbox' IS NOT NULL THEN\n ST_MakeEnvelope(\n (value->'bbox'->>0)::float,\n (value->'bbox'->>1)::float,\n (value->'bbox'->>2)::float,\n (value->'bbox'->>3)::float,\n 4326\n )\n ELSE NULL\n END as geometry,\n properties_idx(value ->'properties') as properties,\n value->>'collection' as collection_id,\n (value->'properties'->>'datetime')::timestamptz as datetime\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac,public;\n*/\n/*\nTakes a single feature, an array of features, or a feature collection\nand returns a set up individual items rows\n*/\n/*\nCREATE OR REPLACE FUNCTION features_to_items(value jsonb) RETURNS SETOF item AS $$\n WITH features AS (\n SELECT\n jsonb_array_elements(\n CASE\n WHEN jsonb_typeof(value) = 'array' THEN value\n WHEN value->>'type' = 'Feature' THEN '[]'::jsonb || value\n WHEN value->>'type' = 'FeatureCollection' THEN value->'features'\n ELSE NULL\n END\n ) as value\n )\n SELECT feature_to_item(value) FROM features\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac,public;\n*/\n\nCREATE OR REPLACE FUNCTION get_item(_id text) RETURNS jsonb AS $$\n SELECT content FROM items WHERE id=_id;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION delete_item(_id text) RETURNS VOID AS $$\nDECLARE\nout items%ROWTYPE;\nBEGIN\n DELETE FROM items WHERE id = _id RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION create_item(data jsonb) RETURNS VOID AS $$\n SELECT make_partitions(stac_datetime(data));\n INSERT INTO items (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION update_item(data jsonb) RETURNS VOID AS $$\nDECLARE\nout items%ROWTYPE;\nBEGIN\n UPDATE items SET content=data WHERE id = data->>'id' RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_item(data jsonb) RETURNS VOID AS $$\nDECLARE\npartition text;\nq text;\nnewcontent jsonb;\nBEGIN\n PERFORM make_partitions(stac_datetime(data));\n partition := get_partition(stac_datetime(data));\n q := format($q$\n INSERT INTO %I (content) VALUES ($1)\n ON CONFLICT (id) DO\n UPDATE SET content = EXCLUDED.content\n WHERE %I.content IS DISTINCT FROM EXCLUDED.content RETURNING content;\n $q$, partition, partition);\n EXECUTE q INTO newcontent USING (data);\n RAISE NOTICE 'newcontent: %', newcontent;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION analyze_empty_partitions() RETURNS VOID AS $$\nDECLARE\np text;\nBEGIN\nFOR p IN SELECT partition FROM all_items_partitions WHERE est_cnt = 0 LOOP\n RAISE NOTICE 'Analyzing %', p;\n EXECUTE format('ANALYZE %I;', p);\nEND LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n/* Trigger Function to cascade inserts/updates/deletes\nfrom items table to items_search table */\n/*\nALTER TABLE items_search ADD CONSTRAINT items_search_fk\nFOREIGN KEY (id) REFERENCES items(id)\nON DELETE CASCADE DEFERRABLE;\n\nCREATE OR REPLACE FUNCTION items_trigger_func()\nRETURNS TRIGGER AS $$\nDECLARE\nBEGIN\n IF TG_OP = 'UPDATE' THEN\n RAISE NOTICE 'DELETING % BEFORE UPDATE', OLD;\n DELETE FROM items_search WHERE id = OLD.id AND datetime = (OLD.content->'properties'->>'datetime')::timestamptz;\n END IF;\n\n INSERT INTO items_search SELECT * FROM feature_to_item(NEW.content);\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nDROP TRIGGER IF EXISTS items_insert_trigger ON items;\nCREATE TRIGGER items_insert_trigger\nAFTER INSERT ON items\nFOR EACH ROW EXECUTE PROCEDURE items_trigger_func();\n\nDROP TRIGGER IF EXISTS items_update_trigger ON items;\nCREATE TRIGGER items_update_trigger\nAFTER UPDATE ON items\nFOR EACH ROW\nWHEN (NEW.content IS DISTINCT FROM OLD.content)\nEXECUTE PROCEDURE items_trigger_func();\n*/\n\n/* Trigger Function to cascade inserts/updates/deletes\nfrom items table to items_search table */\n/*\nCREATE OR REPLACE FUNCTION items_search_trigger_delete_func()\nRETURNS TRIGGER AS $$\nDECLARE\nBEGIN\n RAISE NOTICE 'Deleting from items_search: % Depth: %', OLD, pg_trigger_depth();\n IF pg_trigger_depth()<3 THEN\n RAISE NOTICE 'DELETING WITH datetime';\n DELETE FROM items_search WHERE id=OLD.id AND datetime=OLD.datetime;\n RETURN NULL;\n END IF;\n RETURN OLD;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nDROP TRIGGER IF EXISTS items_search_delete_trigger ON items_search;\nCREATE TRIGGER items_search_delete_trigger\nBEFORE DELETE ON items_search\nFOR EACH ROW EXECUTE PROCEDURE items_search_trigger_delete_func();\n*/\nCREATE OR REPLACE FUNCTION backfill_partitions()\nRETURNS VOID AS $$\nDECLARE\nBEGIN\n IF EXISTS (SELECT 1 FROM items_default LIMIT 1) THEN\n RAISE NOTICE 'Creating new partitions and moving data from default';\n CREATE TEMP TABLE items_default_tmp ON COMMIT DROP AS SELECT datetime, content FROM items_default;\n TRUNCATE items_default;\n PERFORM make_partitions(min(datetime), max(datetime)) FROM items_default_tmp;\n INSERT INTO items (content) SELECT content FROM items_default_tmp;\n END IF;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION items_trigger_stmt_func()\nRETURNS TRIGGER AS $$\nDECLARE\nBEGIN\n PERFORM analyze_empty_partitions();\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nDROP TRIGGER IF EXISTS items_stmt_trigger ON items;\nCREATE TRIGGER items_stmt_trigger\nAFTER INSERT OR UPDATE OR DELETE ON items\nFOR EACH STATEMENT EXECUTE PROCEDURE items_trigger_stmt_func();\n\n\n/*\nView to get a table of available items partitions\nwith date ranges\n*/\n--DROP VIEW IF EXISTS all_items_partitions CASCADE;\nCREATE OR REPLACE VIEW all_items_partitions AS\nWITH base AS\n(SELECT\n c.oid::pg_catalog.regclass::text as partition,\n pg_catalog.pg_get_expr(c.relpartbound, c.oid) as _constraint,\n regexp_matches(\n pg_catalog.pg_get_expr(c.relpartbound, c.oid),\n E'\\\\(''\\([0-9 :+-]*\\)''\\\\).*\\\\(''\\([0-9 :+-]*\\)''\\\\)'\n ) as t,\n reltuples::bigint as est_cnt\nFROM pg_catalog.pg_class c, pg_catalog.pg_inherits i\nWHERE c.oid = i.inhrelid AND i.inhparent = 'items'::regclass)\nSELECT partition, tstzrange(\n t[1]::timestamptz,\n t[2]::timestamptz\n), est_cnt\nFROM base\nORDER BY 2 desc;\n\n--DROP VIEW IF EXISTS items_partitions;\nCREATE OR REPLACE VIEW items_partitions AS\nSELECT * FROM all_items_partitions WHERE est_cnt>0;\n\nCREATE OR REPLACE FUNCTION collection_bbox(id text) RETURNS jsonb AS $$\nSELECT (replace(replace(replace(st_extent(geometry)::text,'BOX(','[['),')',']]'),' ',','))::jsonb\nFROM items WHERE collection_id=$1;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION collection_temporal_extent(id text) RETURNS jsonb AS $$\nSELECT to_jsonb(array[array[min(datetime)::text, max(datetime)::text]])\nFROM items WHERE collection_id=$1;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection_extents() RETURNS VOID AS $$\nUPDATE collections SET\n content = content ||\n jsonb_build_object(\n 'extent', jsonb_build_object(\n 'spatial', jsonb_build_object(\n 'bbox', collection_bbox(collections.id)\n ),\n 'temporal', jsonb_build_object(\n 'interval', collection_temporal_extent(collections.id)\n )\n )\n )\n;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\nSET SEARCH_PATH TO pgstac, public;\n\n/*\nView to get a table of available items partitions\nwith date ranges\n*/\nDROP VIEW IF EXISTS items_partitions;\nCREATE VIEW items_partitions AS\nWITH base AS\n(SELECT\n c.oid::pg_catalog.regclass::text as partition,\n pg_catalog.pg_get_expr(c.relpartbound, c.oid) as _constraint,\n regexp_matches(\n pg_catalog.pg_get_expr(c.relpartbound, c.oid),\n E'\\\\(''\\([0-9 :+-]*\\)''\\\\).*\\\\(''\\([0-9 :+-]*\\)''\\\\)'\n ) as t,\n reltuples::bigint as est_cnt\nFROM pg_catalog.pg_class c, pg_catalog.pg_inherits i\nWHERE c.oid = i.inhrelid AND i.inhparent = 'items'::regclass)\nSELECT partition, tstzrange(\n t[1]::timestamptz,\n t[2]::timestamptz\n), est_cnt\nFROM base\nWHERE est_cnt >0\nORDER BY 2 desc;\n\n\nCREATE OR REPLACE FUNCTION items_by_partition(\n IN _where text DEFAULT 'TRUE',\n IN _dtrange tstzrange DEFAULT tstzrange('-infinity','infinity'),\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN _limit int DEFAULT 10\n) RETURNS SETOF items AS $$\nDECLARE\npartition_query text;\nmain_query text;\nbatchcount int;\ncounter int := 0;\np record;\nBEGIN\nIF _orderby ILIKE 'datetime d%' THEN\n partition_query := format($q$\n SELECT partition\n FROM items_partitions\n WHERE tstzrange && $1\n ORDER BY tstzrange DESC;\n $q$);\nELSIF _orderby ILIKE 'datetime a%' THEN\n partition_query := format($q$\n SELECT partition\n FROM items_partitions\n WHERE tstzrange && $1\n ORDER BY tstzrange ASC\n ;\n $q$);\nELSE\n partition_query := format($q$\n SELECT 'items' as partition WHERE $1 IS NOT NULL;\n $q$);\nEND IF;\nRAISE NOTICE 'Partition Query: %', partition_query;\nFOR p IN\n EXECUTE partition_query USING (_dtrange)\nLOOP\n IF lower(_dtrange)::timestamptz > '-infinity' THEN\n _where := concat(_where,format(' AND datetime >= %L',lower(_dtrange)::timestamptz::text));\n END IF;\n IF upper(_dtrange)::timestamptz < 'infinity' THEN\n _where := concat(_where,format(' AND datetime <= %L',upper(_dtrange)::timestamptz::text));\n END IF;\n\n main_query := format($q$\n SELECT * FROM %I\n WHERE %s\n ORDER BY %s\n LIMIT %s - $1\n $q$, p.partition::text, _where, _orderby, _limit\n );\n RAISE NOTICE 'Partition Query %', main_query;\n RAISE NOTICE '%', counter;\n RETURN QUERY EXECUTE main_query USING counter;\n\n GET DIAGNOSTICS batchcount = ROW_COUNT;\n counter := counter + batchcount;\n RAISE NOTICE 'FOUND %', batchcount;\n IF counter >= _limit THEN\n EXIT;\n END IF;\n RAISE NOTICE 'ADDED % FOR A TOTAL OF %', batchcount, counter;\nEND LOOP;\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION split_stac_path(IN path text, OUT col text, OUT dotpath text, OUT jspath text, OUT jspathtext text) AS $$\nWITH col AS (\n SELECT\n CASE WHEN\n split_part(path, '.', 1) IN ('id', 'stac_version', 'stac_extensions','geometry','properties','assets','collection_id','datetime','links', 'extra_fields') THEN split_part(path, '.', 1)\n ELSE 'properties'\n END AS col\n),\ndp AS (\n SELECT\n col, ltrim(replace(path, col , ''),'.') as dotpath\n FROM col\n),\npaths AS (\nSELECT\n col, dotpath,\n regexp_split_to_table(dotpath,E'\\\\.') as path FROM dp\n) SELECT\n col,\n btrim(concat(col,'.',dotpath),'.'),\n CASE WHEN btrim(concat(col,'.',dotpath),'.') != col THEN concat(col,'->',string_agg(concat('''',path,''''),'->')) ELSE col END,\n regexp_replace(\n CASE WHEN btrim(concat(col,'.',dotpath),'.') != col THEN concat(col,'->',string_agg(concat('''',path,''''),'->')) ELSE col END,\n E'>([^>]*)$','>>\\1'\n )\nFROM paths group by col, dotpath;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n/* Functions for searching items */\nCREATE OR REPLACE FUNCTION sort_base(\n IN _sort jsonb DEFAULT '[{\"field\":\"datetime\",\"direction\":\"desc\"}]',\n OUT key text,\n OUT col text,\n OUT dir text,\n OUT rdir text,\n OUT sort text,\n OUT rsort text\n) RETURNS SETOF RECORD AS $$\nWITH sorts AS (\n SELECT\n value->>'field' as key,\n (split_stac_path(value->>'field')).jspathtext as col,\n coalesce(upper(value->>'direction'),'ASC') as dir\n FROM jsonb_array_elements('[]'::jsonb || coalesce(_sort,'[{\"field\":\"datetime\",\"direction\":\"desc\"}]') )\n)\nSELECT\n key,\n col,\n dir,\n CASE dir WHEN 'DESC' THEN 'ASC' ELSE 'ASC' END as rdir,\n concat(col, ' ', dir, ' NULLS LAST ') AS sort,\n concat(col,' ', CASE dir WHEN 'DESC' THEN 'ASC' ELSE 'ASC' END, ' NULLS LAST ') AS rsort\nFROM sorts\nUNION ALL\nSELECT 'id', 'id', 'DESC', 'ASC', 'id DESC', 'id ASC'\n;\n$$ LANGUAGE SQL;\n\n\nCREATE OR REPLACE FUNCTION sort(_sort jsonb) RETURNS text AS $$\nSELECT string_agg(sort,', ') FROM sort_base(_sort);\n$$ LANGUAGE SQL PARALLEL SAFE SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION rsort(_sort jsonb) RETURNS text AS $$\nSELECT string_agg(rsort,', ') FROM sort_base(_sort);\n$$ LANGUAGE SQL PARALLEL SAFE SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION bbox_geom(_bbox jsonb) RETURNS geometry AS $$\nSELECT CASE jsonb_array_length(_bbox)\n WHEN 4 THEN\n ST_SetSRID(ST_MakeEnvelope(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float,\n (_bbox->>3)::float\n ),4326)\n WHEN 6 THEN\n ST_SetSRID(ST_3DMakeBox(\n ST_MakePoint(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float\n ),\n ST_MakePoint(\n (_bbox->>3)::float,\n (_bbox->>4)::float,\n (_bbox->>5)::float\n )\n ),4326)\n ELSE null END;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION in_array_q(col text, arr jsonb) RETURNS text AS $$\nSELECT CASE jsonb_typeof(arr) WHEN 'array' THEN format('%I = ANY(textarr(%L))', col, arr) ELSE format('%I = %L', col, arr) END;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION count_by_delim(text, text) RETURNS int AS $$\nSELECT count(*) FROM regexp_split_to_table($1,$2);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION stac_query_op(att text, _op text, val jsonb) RETURNS text AS $$\nDECLARE\nret text := '';\nop text;\njp text;\natt_parts RECORD;\nval_str text;\nprop_path text;\nBEGIN\nval_str := lower(jsonb_build_object('a',val)->>'a');\nRAISE NOTICE 'val_str %', val_str;\n\natt_parts := split_stac_path(att);\nprop_path := replace(att_parts.dotpath, 'properties.', '');\n\nop := CASE _op\n WHEN 'eq' THEN '='\n WHEN 'gte' THEN '>='\n WHEN 'gt' THEN '>'\n WHEN 'lte' THEN '<='\n WHEN 'lt' THEN '<'\n WHEN 'ne' THEN '!='\n WHEN 'neq' THEN '!='\n WHEN 'startsWith' THEN 'LIKE'\n WHEN 'endsWith' THEN 'LIKE'\n WHEN 'contains' THEN 'LIKE'\n ELSE _op\nEND;\n\nval_str := CASE _op\n WHEN 'startsWith' THEN concat(val_str, '%')\n WHEN 'endsWith' THEN concat('%', val_str)\n WHEN 'contains' THEN concat('%',val_str,'%')\n ELSE val_str\nEND;\n\n\nRAISE NOTICE 'att_parts: % %', att_parts, count_by_delim(att_parts.dotpath,'\\.');\nIF\n op = '='\n AND att_parts.col = 'properties'\n --AND count_by_delim(att_parts.dotpath,'\\.') = 2\nTHEN\n -- use jsonpath query to leverage index for eqaulity tests on single level deep properties\n jp := btrim(format($jp$ $.%I[*] ? ( @ == %s ) $jp$, replace(att_parts.dotpath, 'properties.',''), lower(val::text)::jsonb));\n raise notice 'jp: %', jp;\n ret := format($q$ properties @? %L $q$, jp);\nELSIF jsonb_typeof(val) = 'number' THEN\n ret := format('properties ? %L AND (%s)::numeric %s %s', prop_path, att_parts.jspathtext, op, val);\nELSE\n ret := format('properties ? %L AND %s %s %L', prop_path ,att_parts.jspathtext, op, val_str);\nEND IF;\nRAISE NOTICE 'Op Query: %', ret;\n\nreturn ret;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION stac_query(_query jsonb) RETURNS TEXT[] AS $$\nDECLARE\nqa text[];\natt text;\nops jsonb;\nop text;\nval jsonb;\nBEGIN\nFOR att, ops IN SELECT key, value FROM jsonb_each(_query)\nLOOP\n FOR op, val IN SELECT key, value FROM jsonb_each(ops)\n LOOP\n qa := array_append(qa, stac_query_op(att,op, val));\n RAISE NOTICE '% % %', att, op, val;\n END LOOP;\nEND LOOP;\nRETURN qa;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION filter_by_order(item_id text, _sort jsonb, _type text) RETURNS text AS $$\nDECLARE\nitem item;\nBEGIN\nSELECT * INTO item FROM items WHERE id=item_id;\nRETURN filter_by_order(item, _sort, _type);\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n-- Used to create filters used for paging using the items id from the token\nCREATE OR REPLACE FUNCTION filter_by_order(_item item, _sort jsonb, _type text) RETURNS text AS $$\nDECLARE\nsorts RECORD;\nfilts text[];\nitemval text;\nop text;\nidop text;\nret text;\neq_flag text;\n_item_j jsonb := to_jsonb(_item);\nBEGIN\nFOR sorts IN SELECT * FROM sort_base(_sort) LOOP\n IF sorts.col = 'datetime' THEN\n CONTINUE;\n END IF;\n IF sorts.col='id' AND _type IN ('prev','next') THEN\n eq_flag := '';\n ELSE\n eq_flag := '=';\n END IF;\n\n op := concat(\n CASE\n WHEN _type in ('prev','first') AND sorts.dir = 'ASC' THEN '<'\n WHEN _type in ('last','next') AND sorts.dir = 'ASC' THEN '>'\n WHEN _type in ('prev','first') AND sorts.dir = 'DESC' THEN '>'\n WHEN _type in ('last','next') AND sorts.dir = 'DESC' THEN '<'\n END,\n eq_flag\n );\n\n IF _item_j ? sorts.col THEN\n filts = array_append(filts, format('%s %s %L', sorts.col, op, _item_j->>sorts.col));\n END IF;\nEND LOOP;\nret := coalesce(array_to_string(filts,' AND '), 'TRUE');\nRAISE NOTICE 'Order Filter %', ret;\nRETURN ret;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION search_dtrange(IN _indate jsonb, OUT _tstzrange tstzrange) AS\n$$\nWITH t AS (\n SELECT CASE\n WHEN jsonb_typeof(_indate) = 'array' THEN\n textarr(_indate)\n ELSE\n regexp_split_to_array(\n btrim(_indate::text,'\"'),\n '/'\n )\n END AS arr\n)\n, t1 AS (\n SELECT\n CASE\n WHEN array_upper(arr,1) = 1 OR arr[1] = '..' OR arr[1] IS NULL THEN '-infinity'::timestamptz\n ELSE arr[1]::timestamptz\n END AS st,\n CASE\n WHEN array_upper(arr,1) = 1 THEN arr[1]::timestamptz\n WHEN arr[2] = '..' OR arr[2] IS NULL THEN 'infinity'::timestamptz\n ELSE arr[2]::timestamptz\n END AS et\n FROM t\n)\nSELECT\n tstzrange(st,et)\nFROM t1;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION flip_jsonb_array(j jsonb) RETURNS jsonb AS $$\nWITH t AS (\n SELECT i, row_number() over () as r FROM jsonb_array_elements(j) i\n), o AS (\n SELECT i FROM t ORDER BY r DESC\n)\nSELECT jsonb_agg(i) from o\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION search(_search jsonb = '{}'::jsonb) RETURNS SETOF jsonb AS $$\nDECLARE\nqstart timestamptz := clock_timestamp();\n_sort text := '';\n_rsort text := '';\n_limit int := 10;\n_geom geometry;\nqa text[];\npq text[];\nquery text;\npq_prop record;\npq_op record;\nprev_id text := NULL;\nnext_id text := NULL;\nwhereq text := 'TRUE';\nlinks jsonb := '[]'::jsonb;\ntoken text;\ntok_val text;\ntok_q text := 'TRUE';\ntok_sort text;\nfirst_id text;\nfirst_dt timestamptz;\nlast_id text;\nsort text;\nrsort text;\ndt text[];\ndqa text[];\ndq text;\nmq_where text;\nstartdt timestamptz;\nenddt timestamptz;\nitem items%ROWTYPE;\ncounter int := 0;\nbatchcount int;\nmonth timestamptz;\nm record;\n_dtrange tstzrange := tstzrange('-infinity','infinity');\n_dtsort text;\n_token_dtrange tstzrange := tstzrange('-infinity','infinity');\n_token_record items%ROWTYPE;\nis_prev boolean := false;\nincludes text[];\nexcludes text[];\nBEGIN\n-- Create table from sort query of items to sort\nCREATE TEMP TABLE pgstac_tmp_sorts ON COMMIT DROP AS SELECT * FROM sort_base(_search->'sortby');\n\n-- Get the datetime sort direction, necessary for efficient cycling through partitions\nSELECT INTO _dtsort dir FROM pgstac_tmp_sorts WHERE key='datetime';\nRAISE NOTICE '_dtsort: %',_dtsort;\n\nSELECT INTO _sort string_agg(s.sort,', ') FROM pgstac_tmp_sorts s;\nSELECT INTO _rsort string_agg(s.rsort,', ') FROM pgstac_tmp_sorts s;\ntok_sort := _sort;\n\n\n-- Get datetime from query as a tstzrange\nIF _search ? 'datetime' THEN\n _dtrange := search_dtrange(_search->'datetime');\n _token_dtrange := _dtrange;\nEND IF;\n\n-- Get the paging token\nIF _search ? 'token' THEN\n token := _search->>'token';\n tok_val := substr(token,6);\n IF starts_with(token, 'prev:') THEN\n is_prev := true;\n END IF;\n SELECT INTO _token_record * FROM items WHERE id=tok_val;\n IF\n (is_prev AND _dtsort = 'DESC')\n OR\n (not is_prev AND _dtsort = 'ASC')\n THEN\n _token_dtrange := _dtrange * tstzrange(_token_record.datetime, 'infinity');\n ELSIF\n _dtsort IS NOT NULL\n THEN\n _token_dtrange := _dtrange * tstzrange('-infinity',_token_record.datetime);\n END IF;\n IF is_prev THEN\n tok_q := filter_by_order(tok_val, _search->'sortby', 'first');\n _sort := _rsort;\n ELSIF starts_with(token, 'next:') THEN\n tok_q := filter_by_order(tok_val, _search->'sortby', 'last');\n END IF;\nEND IF;\nRAISE NOTICE 'timing: %', age(clock_timestamp(), qstart);\nRAISE NOTICE 'tok_q: % _token_dtrange: %', tok_q, _token_dtrange;\n\nIF _search ? 'ids' THEN\n RAISE NOTICE 'searching solely based on ids... %',_search;\n qa := array_append(qa, in_array_q('id', _search->'ids'));\nELSE\n IF _search ? 'intersects' THEN\n _geom := ST_SetSRID(ST_GeomFromGeoJSON(_search->>'intersects'), 4326);\n ELSIF _search ? 'bbox' THEN\n _geom := bbox_geom(_search->'bbox');\n END IF;\n\n IF _geom IS NOT NULL THEN\n qa := array_append(qa, format('st_intersects(geometry, %L::geometry)',_geom));\n END IF;\n\n IF _search ? 'collections' THEN\n qa := array_append(qa, in_array_q('collection_id', _search->'collections'));\n END IF;\n\n IF _search ? 'query' THEN\n qa := array_cat(qa,\n stac_query(_search->'query')\n );\n END IF;\nEND IF;\n\nIF _search ? 'limit' THEN\n _limit := (_search->>'limit')::int;\nEND IF;\n\nIF _search ? 'fields' THEN\n IF _search->'fields' ? 'exclude' THEN\n excludes=textarr(_search->'fields'->'exclude');\n END IF;\n IF _search->'fields' ? 'include' THEN\n includes=textarr(_search->'fields'->'include');\n IF array_length(includes, 1)>0 AND NOT 'id' = ANY (includes) THEN\n includes = includes || '{id}';\n END IF;\n END IF;\n RAISE NOTICE 'Includes: %, Excludes: %', includes, excludes;\nEND IF;\n\nwhereq := COALESCE(array_to_string(qa,' AND '),' TRUE ');\ndq := COALESCE(array_to_string(dqa,' AND '),' TRUE ');\nRAISE NOTICE 'timing before temp table: %', age(clock_timestamp(), qstart);\n\nCREATE TEMP TABLE results_page ON COMMIT DROP AS\nSELECT * FROM items_by_partition(\n concat(whereq, ' AND ', tok_q),\n _token_dtrange,\n _sort,\n _limit + 1\n);\nRAISE NOTICE 'timing after temp table: %', age(clock_timestamp(), qstart);\n\nRAISE NOTICE 'timing before min/max: %', age(clock_timestamp(), qstart);\n\nIF is_prev THEN\n SELECT INTO last_id, first_id, counter\n first_value(id) OVER (),\n last_value(id) OVER (),\n count(*) OVER ()\n FROM results_page;\nELSE\n SELECT INTO first_id, last_id, counter\n first_value(id) OVER (),\n last_value(id) OVER (),\n count(*) OVER ()\n FROM results_page;\nEND IF;\nRAISE NOTICE 'firstid: %, lastid %', first_id, last_id;\nRAISE NOTICE 'timing after min/max: %', age(clock_timestamp(), qstart);\n\n\n\n\nIF counter > _limit THEN\n next_id := last_id;\n RAISE NOTICE 'next_id: %', next_id;\nELSE\n RAISE NOTICE 'No more next';\nEND IF;\n\nIF tok_q = 'TRUE' THEN\n RAISE NOTICE 'Not a paging query, no previous item';\nELSE\n RAISE NOTICE 'Getting previous item id';\n RAISE NOTICE 'timing: %', age(clock_timestamp(), qstart);\n SELECT INTO _token_record * FROM items WHERE id=first_id;\n IF\n _dtsort = 'DESC'\n THEN\n _token_dtrange := _dtrange * tstzrange(_token_record.datetime, 'infinity');\n ELSE\n _token_dtrange := _dtrange * tstzrange('-infinity',_token_record.datetime);\n END IF;\n RAISE NOTICE '% %', _token_dtrange, _dtrange;\n SELECT id INTO prev_id FROM items_by_partition(\n concat(whereq, ' AND ', filter_by_order(first_id, _search->'sortby', 'prev')),\n _token_dtrange,\n _rsort,\n 1\n );\n RAISE NOTICE 'timing: %', age(clock_timestamp(), qstart);\n\n RAISE NOTICE 'prev_id: %', prev_id;\nEND IF;\n\n\nRETURN QUERY\nWITH features AS (\n SELECT filter_jsonb(content, includes, excludes) as content\n FROM results_page LIMIT _limit\n),\nj AS (SELECT jsonb_agg(content) as feature_arr FROM features)\nSELECT jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce (\n CASE WHEN is_prev THEN flip_jsonb_array(feature_arr) ELSE feature_arr END\n ,'[]'::jsonb),\n 'links', links,\n 'timeStamp', now(),\n 'next', next_id,\n 'prev', prev_id\n)\nFROM j\n;\n\n\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\nINSERT INTO migrations (version) VALUES ('0.2.7');\n" }, { "path": "src/pgstac/migrations/pgstac.0.2.8-0.2.9.sql", "content": "SET SEARCH_PATH to pgstac, public;\nINSERT INTO migrations (version) VALUES ('0.2.9');\n" }, { "path": "src/pgstac/migrations/pgstac.0.2.8.sql", "content": "CREATE EXTENSION IF NOT EXISTS postgis;\nCREATE SCHEMA IF NOT EXISTS partman;\nCREATE EXTENSION IF NOT EXISTS pg_partman SCHEMA partman;\nCREATE SCHEMA IF NOT EXISTS pgstac;\n\n\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE TABLE migrations (\n version text,\n datetime timestamptz DEFAULT now() NOT NULL\n);\n\n/* converts a jsonb text array to a pg text[] array */\nCREATE OR REPLACE FUNCTION textarr(_js jsonb)\n RETURNS text[] AS $$\n SELECT ARRAY(SELECT jsonb_array_elements_text(_js));\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\n/*\nconverts a jsonb text array to comma delimited list of identifer quoted\nuseful for constructing column lists for selects\n*/\nCREATE OR REPLACE FUNCTION array_idents(_js jsonb)\n RETURNS text AS $$\n SELECT string_agg(quote_ident(v),',') FROM jsonb_array_elements_text(_js) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\n\n/* looks for a geometry in a stac item first from geometry and falling back to bbox */\nCREATE OR REPLACE FUNCTION stac_geom(value jsonb) RETURNS geometry AS $$\nSELECT\n CASE\n WHEN value->>'geometry' IS NOT NULL THEN\n ST_GeomFromGeoJSON(value->>'geometry')\n WHEN value->>'bbox' IS NOT NULL THEN\n ST_MakeEnvelope(\n (value->'bbox'->>0)::float,\n (value->'bbox'->>1)::float,\n (value->'bbox'->>2)::float,\n (value->'bbox'->>3)::float,\n 4326\n )\n ELSE NULL\n END as geometry\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION stac_datetime(value jsonb) RETURNS timestamptz AS $$\nSELECT (value->'properties'->>'datetime')::timestamptz;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION jsonb_paths (IN jdata jsonb, OUT path text[], OUT value jsonb) RETURNS\nSETOF RECORD AS $$\nwith recursive extract_all as\n(\n select\n ARRAY[key]::text[] as path,\n value\n FROM jsonb_each(jdata)\nunion all\n select\n path || coalesce(obj_key, (arr_key- 1)::text),\n coalesce(obj_value, arr_value)\n from extract_all\n left join lateral\n jsonb_each(case jsonb_typeof(value) when 'object' then value end)\n as o(obj_key, obj_value)\n on jsonb_typeof(value) = 'object'\n left join lateral\n jsonb_array_elements(case jsonb_typeof(value) when 'array' then value end)\n with ordinality as a(arr_value, arr_key)\n on jsonb_typeof(value) = 'array'\n where obj_key is not null or arr_key is not null\n)\nselect *\nfrom extract_all;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION jsonb_obj_paths (IN jdata jsonb, OUT path text[], OUT value jsonb) RETURNS\nSETOF RECORD AS $$\nwith recursive extract_all as\n(\n select\n ARRAY[key]::text[] as path,\n value\n FROM jsonb_each(jdata)\nunion all\n select\n path || obj_key,\n obj_value\n from extract_all\n left join lateral\n jsonb_each(case jsonb_typeof(value) when 'object' then value end)\n as o(obj_key, obj_value)\n on jsonb_typeof(value) = 'object'\n where obj_key is not null\n)\nselect *\nfrom extract_all;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION jsonb_val_paths (IN jdata jsonb, OUT path text[], OUT value jsonb) RETURNS\nSETOF RECORD AS $$\nSELECT * FROM jsonb_obj_paths(jdata) WHERE jsonb_typeof(value) not in ('object','array');\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION path_includes(IN path text[], IN includes text[]) RETURNS BOOLEAN AS $$\nWITH t AS (SELECT unnest(includes) i)\nSELECT EXISTS (\n SELECT 1 FROM t WHERE path @> string_to_array(trim(i), '.')\n);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION path_excludes(IN path text[], IN excludes text[]) RETURNS BOOLEAN AS $$\nWITH t AS (SELECT unnest(excludes) e)\nSELECT NOT EXISTS (\n SELECT 1 FROM t WHERE path @> string_to_array(trim(e), '.')\n);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION jsonb_obj_paths_filtered (\n IN jdata jsonb,\n IN includes text[] DEFAULT ARRAY[]::text[],\n IN excludes text[] DEFAULT ARRAY[]::text[],\n OUT path text[],\n OUT value jsonb\n) RETURNS\nSETOF RECORD AS $$\nSELECT path, value\nFROM jsonb_obj_paths(jdata)\nWHERE\n CASE WHEN cardinality(includes) > 0 THEN path_includes(path, includes) ELSE TRUE END\n AND\n path_excludes(path, excludes)\n\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION empty_arr(ANYARRAY) RETURNS BOOLEAN AS $$\nSELECT CASE\n WHEN $1 IS NULL THEN TRUE\n WHEN cardinality($1)<1 THEN TRUE\nELSE FALSE\nEND;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION filter_jsonb(\n IN jdata jsonb,\n IN includes text[] DEFAULT ARRAY[]::text[],\n IN excludes text[] DEFAULT ARRAY[]::text[]\n) RETURNS jsonb AS $$\nDECLARE\nrec RECORD;\noutj jsonb := '{}'::jsonb;\ncreated_paths text[] := '{}'::text[];\nBEGIN\n\nIF empty_arr(includes) AND empty_arr(excludes) THEN\nRAISE NOTICE 'no filter';\n RETURN jdata;\nEND IF;\nFOR rec in\nSELECT * FROM jsonb_obj_paths_filtered(jdata, includes, excludes)\nWHERE jsonb_typeof(value) != 'object'\nLOOP\n IF array_length(rec.path,1)>1 THEN\n FOR i IN 1..(array_length(rec.path,1)-1) LOOP\n IF NOT array_to_string(rec.path[1:i],'.') = ANY (created_paths) THEN\n outj := jsonb_set(outj, rec.path[1:i],'{}', true);\n created_paths := created_paths || array_to_string(rec.path[1:i],'.');\n END IF;\n END LOOP;\n END IF;\n outj := jsonb_set(outj, rec.path, rec.value, true);\n created_paths := created_paths || array_to_string(rec.path,'.');\nEND LOOP;\nRETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION properties_idx(_in jsonb) RETURNS jsonb AS $$\nWITH t AS (\n select array_to_string(path,'.') as path, lower(value::text)::jsonb as lowerval\n FROM jsonb_val_paths(_in)\n WHERE array_to_string(path,'.') not in ('datetime')\n)\nSELECT jsonb_object_agg(path, lowerval) FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE TABLE IF NOT EXISTS collections (\n id VARCHAR GENERATED ALWAYS AS (content->>'id') STORED PRIMARY KEY,\n content JSONB\n);\n\nCREATE OR REPLACE FUNCTION create_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection(data jsonb) RETURNS VOID AS $$\nDECLARE\nout collections%ROWTYPE;\nBEGIN\n UPDATE collections SET content=data WHERE id = data->>'id' RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ON CONFLICT (id) DO\n UPDATE\n SET content=EXCLUDED.content\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION delete_collection(_id text) RETURNS VOID AS $$\nDECLARE\nout collections%ROWTYPE;\nBEGIN\n DELETE FROM collections WHERE id = _id RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION get_collection(id text) RETURNS jsonb AS $$\nSELECT content FROM collections\nWHERE id=$1\n;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION all_collections() RETURNS jsonb AS $$\nSELECT jsonb_agg(content) FROM collections;\n;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE TABLE IF NOT EXISTS items (\n id VARCHAR GENERATED ALWAYS AS (content->>'id') STORED NOT NULL,\n geometry geometry GENERATED ALWAYS AS (stac_geom(content)) STORED NOT NULL,\n properties jsonb GENERATED ALWAYS as (properties_idx(content->'properties')) STORED,\n collection_id text GENERATED ALWAYS AS (content->>'collection') STORED NOT NULL,\n datetime timestamptz GENERATED ALWAYS AS (stac_datetime(content)) STORED NOT NULL,\n content JSONB NOT NULL\n)\nPARTITION BY RANGE (stac_datetime(content))\n;\n\nALTER TABLE items ADD constraint items_collections_fk FOREIGN KEY (collection_id) REFERENCES collections(id) DEFERRABLE;\n\nCREATE TABLE items_template (\n LIKE items\n);\n\nALTER TABLE items_template ADD PRIMARY KEY (id);\n\n\nDELETE from partman.part_config WHERE parent_table = 'pgstac.items';\nSELECT partman.create_parent(\n 'pgstac.items',\n 'datetime',\n 'native',\n 'weekly',\n p_template_table := 'pgstac.items_template',\n p_premake := 4\n);\n\nCREATE OR REPLACE FUNCTION make_partitions(st timestamptz, et timestamptz DEFAULT NULL) RETURNS BOOL AS $$\nWITH t AS (\n SELECT\n generate_series(\n date_trunc('week',st),\n date_trunc('week', coalesce(et, st)),\n '1 week'::interval\n ) w\n),\nw AS (SELECT array_agg(w) as w FROM t)\nSELECT CASE WHEN w IS NULL THEN NULL ELSE partman.create_partition_time('pgstac.items', w, true) END FROM w;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_partition(timestamptz) RETURNS text AS $$\nSELECT to_char($1, '\"items_p\"IYYY\"w\"IW');\n$$ LANGUAGE SQL;\n\nCREATE INDEX \"datetime_id_idx\" ON items (datetime, id);\nCREATE INDEX \"properties_idx\" ON items USING GIN (properties);\nCREATE INDEX \"collection_idx\" ON items (collection_id);\nCREATE INDEX \"geometry_idx\" ON items USING GIST (geometry);\n\n\nCREATE TYPE item AS (\n id text,\n geometry geometry,\n properties JSONB,\n collection_id text,\n datetime timestamptz\n);\n\n\nCREATE OR REPLACE FUNCTION get_item(_id text) RETURNS jsonb AS $$\n SELECT content FROM items WHERE id=_id;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION delete_item(_id text) RETURNS VOID AS $$\nDECLARE\nout items%ROWTYPE;\nBEGIN\n DELETE FROM items WHERE id = _id RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION create_item(data jsonb) RETURNS VOID AS $$\n SELECT make_partitions(stac_datetime(data));\n INSERT INTO items (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION update_item(data jsonb) RETURNS VOID AS $$\nDECLARE\nout items%ROWTYPE;\nBEGIN\n UPDATE items SET content=data WHERE id = data->>'id' RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_item(data jsonb) RETURNS VOID AS $$\nDECLARE\npartition text;\nq text;\nnewcontent jsonb;\nBEGIN\n PERFORM make_partitions(stac_datetime(data));\n partition := get_partition(stac_datetime(data));\n q := format($q$\n INSERT INTO %I (content) VALUES ($1)\n ON CONFLICT (id) DO\n UPDATE SET content = EXCLUDED.content\n WHERE %I.content IS DISTINCT FROM EXCLUDED.content RETURNING content;\n $q$, partition, partition);\n EXECUTE q INTO newcontent USING (data);\n RAISE NOTICE 'newcontent: %', newcontent;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION analyze_empty_partitions() RETURNS VOID AS $$\nDECLARE\np text;\nBEGIN\nFOR p IN SELECT partition FROM all_items_partitions WHERE est_cnt = 0 LOOP\n RAISE NOTICE 'Analyzing %', p;\n EXECUTE format('ANALYZE %I;', p);\nEND LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION backfill_partitions()\nRETURNS VOID AS $$\nDECLARE\nBEGIN\n IF EXISTS (SELECT 1 FROM items_default LIMIT 1) THEN\n RAISE NOTICE 'Creating new partitions and moving data from default';\n CREATE TEMP TABLE items_default_tmp ON COMMIT DROP AS SELECT datetime, content FROM items_default;\n TRUNCATE items_default;\n PERFORM make_partitions(min(datetime), max(datetime)) FROM items_default_tmp;\n INSERT INTO items (content) SELECT content FROM items_default_tmp;\n END IF;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION items_trigger_stmt_func()\nRETURNS TRIGGER AS $$\nDECLARE\nBEGIN\n PERFORM analyze_empty_partitions();\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nDROP TRIGGER IF EXISTS items_stmt_trigger ON items;\nCREATE TRIGGER items_stmt_trigger\nAFTER INSERT OR UPDATE OR DELETE ON items\nFOR EACH STATEMENT EXECUTE PROCEDURE items_trigger_stmt_func();\n\n\n/*\nView to get a table of available items partitions\nwith date ranges\n*/\n--DROP VIEW IF EXISTS all_items_partitions CASCADE;\nCREATE OR REPLACE VIEW all_items_partitions AS\nWITH base AS\n(SELECT\n c.oid::pg_catalog.regclass::text as partition,\n pg_catalog.pg_get_expr(c.relpartbound, c.oid) as _constraint,\n regexp_matches(\n pg_catalog.pg_get_expr(c.relpartbound, c.oid),\n E'\\\\(''\\([0-9 :+-]*\\)''\\\\).*\\\\(''\\([0-9 :+-]*\\)''\\\\)'\n ) as t,\n reltuples::bigint as est_cnt\nFROM pg_catalog.pg_class c, pg_catalog.pg_inherits i\nWHERE c.oid = i.inhrelid AND i.inhparent = 'items'::regclass)\nSELECT partition, tstzrange(\n t[1]::timestamptz,\n t[2]::timestamptz\n), est_cnt\nFROM base\nORDER BY 2 desc;\n\n--DROP VIEW IF EXISTS items_partitions;\nCREATE OR REPLACE VIEW items_partitions AS\nSELECT * FROM all_items_partitions WHERE est_cnt>0;\n\nCREATE OR REPLACE FUNCTION collection_bbox(id text) RETURNS jsonb AS $$\nSELECT (replace(replace(replace(st_extent(geometry)::text,'BOX(','[['),')',']]'),' ',','))::jsonb\nFROM items WHERE collection_id=$1;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION collection_temporal_extent(id text) RETURNS jsonb AS $$\nSELECT to_jsonb(array[array[min(datetime)::text, max(datetime)::text]])\nFROM items WHERE collection_id=$1;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection_extents() RETURNS VOID AS $$\nUPDATE collections SET\n content = content ||\n jsonb_build_object(\n 'extent', jsonb_build_object(\n 'spatial', jsonb_build_object(\n 'bbox', collection_bbox(collections.id)\n ),\n 'temporal', jsonb_build_object(\n 'interval', collection_temporal_extent(collections.id)\n )\n )\n )\n;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\nSET SEARCH_PATH TO pgstac, public;\n\n\nCREATE OR REPLACE FUNCTION items_by_partition(\n IN _where text DEFAULT 'TRUE',\n IN _dtrange tstzrange DEFAULT tstzrange('-infinity','infinity'),\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN _limit int DEFAULT 10\n) RETURNS SETOF items AS $$\nDECLARE\npartition_query text;\nmain_query text;\nbatchcount int;\ncounter int := 0;\np record;\nBEGIN\nIF _orderby ILIKE 'datetime d%' THEN\n partition_query := format($q$\n SELECT partition\n FROM items_partitions\n WHERE tstzrange && $1\n ORDER BY tstzrange DESC;\n $q$);\nELSIF _orderby ILIKE 'datetime a%' THEN\n partition_query := format($q$\n SELECT partition\n FROM items_partitions\n WHERE tstzrange && $1\n ORDER BY tstzrange ASC\n ;\n $q$);\nELSE\n partition_query := format($q$\n SELECT 'items' as partition WHERE $1 IS NOT NULL;\n $q$);\nEND IF;\nRAISE NOTICE 'Partition Query: %', partition_query;\nFOR p IN\n EXECUTE partition_query USING (_dtrange)\nLOOP\n IF lower(_dtrange)::timestamptz > '-infinity' THEN\n _where := concat(_where,format(' AND datetime >= %L',lower(_dtrange)::timestamptz::text));\n END IF;\n IF upper(_dtrange)::timestamptz < 'infinity' THEN\n _where := concat(_where,format(' AND datetime <= %L',upper(_dtrange)::timestamptz::text));\n END IF;\n\n main_query := format($q$\n SELECT * FROM %I\n WHERE %s\n ORDER BY %s\n LIMIT %s - $1\n $q$, p.partition::text, _where, _orderby, _limit\n );\n RAISE NOTICE 'Partition Query %', main_query;\n RAISE NOTICE '%', counter;\n RETURN QUERY EXECUTE main_query USING counter;\n\n GET DIAGNOSTICS batchcount = ROW_COUNT;\n counter := counter + batchcount;\n RAISE NOTICE 'FOUND %', batchcount;\n IF counter >= _limit THEN\n EXIT;\n END IF;\n RAISE NOTICE 'ADDED % FOR A TOTAL OF %', batchcount, counter;\nEND LOOP;\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION split_stac_path(IN path text, OUT col text, OUT dotpath text, OUT jspath text, OUT jspathtext text) AS $$\nWITH col AS (\n SELECT\n CASE WHEN\n split_part(path, '.', 1) IN ('id', 'stac_version', 'stac_extensions','geometry','properties','assets','collection_id','datetime','links', 'extra_fields') THEN split_part(path, '.', 1)\n ELSE 'properties'\n END AS col\n),\ndp AS (\n SELECT\n col, ltrim(replace(path, col , ''),'.') as dotpath\n FROM col\n),\npaths AS (\nSELECT\n col, dotpath,\n regexp_split_to_table(dotpath,E'\\\\.') as path FROM dp\n) SELECT\n col,\n btrim(concat(col,'.',dotpath),'.'),\n CASE WHEN btrim(concat(col,'.',dotpath),'.') != col THEN concat(col,'->',string_agg(concat('''',path,''''),'->')) ELSE col END,\n regexp_replace(\n CASE WHEN btrim(concat(col,'.',dotpath),'.') != col THEN concat(col,'->',string_agg(concat('''',path,''''),'->')) ELSE col END,\n E'>([^>]*)$','>>\\1'\n )\nFROM paths group by col, dotpath;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n/* Functions for searching items */\nCREATE OR REPLACE FUNCTION sort_base(\n IN _sort jsonb DEFAULT '[{\"field\":\"datetime\",\"direction\":\"desc\"}]',\n OUT key text,\n OUT col text,\n OUT dir text,\n OUT rdir text,\n OUT sort text,\n OUT rsort text\n) RETURNS SETOF RECORD AS $$\nWITH sorts AS (\n SELECT\n value->>'field' as key,\n (split_stac_path(value->>'field')).jspathtext as col,\n coalesce(upper(value->>'direction'),'ASC') as dir\n FROM jsonb_array_elements('[]'::jsonb || coalesce(_sort,'[{\"field\":\"datetime\",\"direction\":\"desc\"}]') )\n)\nSELECT\n key,\n col,\n dir,\n CASE dir WHEN 'DESC' THEN 'ASC' ELSE 'ASC' END as rdir,\n concat(col, ' ', dir, ' NULLS LAST ') AS sort,\n concat(col,' ', CASE dir WHEN 'DESC' THEN 'ASC' ELSE 'ASC' END, ' NULLS LAST ') AS rsort\nFROM sorts\nUNION ALL\nSELECT 'id', 'id', 'DESC', 'ASC', 'id DESC', 'id ASC'\n;\n$$ LANGUAGE SQL;\n\n\nCREATE OR REPLACE FUNCTION sort(_sort jsonb) RETURNS text AS $$\nSELECT string_agg(sort,', ') FROM sort_base(_sort);\n$$ LANGUAGE SQL PARALLEL SAFE SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION rsort(_sort jsonb) RETURNS text AS $$\nSELECT string_agg(rsort,', ') FROM sort_base(_sort);\n$$ LANGUAGE SQL PARALLEL SAFE SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION bbox_geom(_bbox jsonb) RETURNS geometry AS $$\nSELECT CASE jsonb_array_length(_bbox)\n WHEN 4 THEN\n ST_SetSRID(ST_MakeEnvelope(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float,\n (_bbox->>3)::float\n ),4326)\n WHEN 6 THEN\n ST_SetSRID(ST_3DMakeBox(\n ST_MakePoint(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float\n ),\n ST_MakePoint(\n (_bbox->>3)::float,\n (_bbox->>4)::float,\n (_bbox->>5)::float\n )\n ),4326)\n ELSE null END;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION in_array_q(col text, arr jsonb) RETURNS text AS $$\nSELECT CASE jsonb_typeof(arr) WHEN 'array' THEN format('%I = ANY(textarr(%L))', col, arr) ELSE format('%I = %L', col, arr) END;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION count_by_delim(text, text) RETURNS int AS $$\nSELECT count(*) FROM regexp_split_to_table($1,$2);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION stac_query_op(att text, _op text, val jsonb) RETURNS text AS $$\nDECLARE\nret text := '';\nop text;\njp text;\natt_parts RECORD;\nval_str text;\nprop_path text;\nBEGIN\nval_str := lower(jsonb_build_object('a',val)->>'a');\nRAISE NOTICE 'val_str %', val_str;\n\natt_parts := split_stac_path(att);\nprop_path := replace(att_parts.dotpath, 'properties.', '');\n\nop := CASE _op\n WHEN 'eq' THEN '='\n WHEN 'gte' THEN '>='\n WHEN 'gt' THEN '>'\n WHEN 'lte' THEN '<='\n WHEN 'lt' THEN '<'\n WHEN 'ne' THEN '!='\n WHEN 'neq' THEN '!='\n WHEN 'startsWith' THEN 'LIKE'\n WHEN 'endsWith' THEN 'LIKE'\n WHEN 'contains' THEN 'LIKE'\n ELSE _op\nEND;\n\nval_str := CASE _op\n WHEN 'startsWith' THEN concat(val_str, '%')\n WHEN 'endsWith' THEN concat('%', val_str)\n WHEN 'contains' THEN concat('%',val_str,'%')\n ELSE val_str\nEND;\n\n\nRAISE NOTICE 'att_parts: % %', att_parts, count_by_delim(att_parts.dotpath,'\\.');\nIF\n op = '='\n AND att_parts.col = 'properties'\n --AND count_by_delim(att_parts.dotpath,'\\.') = 2\nTHEN\n -- use jsonpath query to leverage index for eqaulity tests on single level deep properties\n jp := btrim(format($jp$ $.%I[*] ? ( @ == %s ) $jp$, replace(att_parts.dotpath, 'properties.',''), lower(val::text)::jsonb));\n raise notice 'jp: %', jp;\n ret := format($q$ properties @? %L $q$, jp);\nELSIF jsonb_typeof(val) = 'number' THEN\n ret := format('properties ? %L AND (%s)::numeric %s %s', prop_path, att_parts.jspathtext, op, val);\nELSE\n ret := format('properties ? %L AND %s %s %L', prop_path ,att_parts.jspathtext, op, val_str);\nEND IF;\nRAISE NOTICE 'Op Query: %', ret;\n\nreturn ret;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION stac_query(_query jsonb) RETURNS TEXT[] AS $$\nDECLARE\nqa text[];\natt text;\nops jsonb;\nop text;\nval jsonb;\nBEGIN\nFOR att, ops IN SELECT key, value FROM jsonb_each(_query)\nLOOP\n FOR op, val IN SELECT key, value FROM jsonb_each(ops)\n LOOP\n qa := array_append(qa, stac_query_op(att,op, val));\n RAISE NOTICE '% % %', att, op, val;\n END LOOP;\nEND LOOP;\nRETURN qa;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION filter_by_order(item_id text, _sort jsonb, _type text) RETURNS text AS $$\nDECLARE\nitem item;\nBEGIN\nSELECT * INTO item FROM items WHERE id=item_id;\nRETURN filter_by_order(item, _sort, _type);\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n-- Used to create filters used for paging using the items id from the token\nCREATE OR REPLACE FUNCTION filter_by_order(_item item, _sort jsonb, _type text) RETURNS text AS $$\nDECLARE\nsorts RECORD;\nfilts text[];\nitemval text;\nop text;\nidop text;\nret text;\neq_flag text;\n_item_j jsonb := to_jsonb(_item);\nBEGIN\nFOR sorts IN SELECT * FROM sort_base(_sort) LOOP\n IF sorts.col = 'datetime' THEN\n CONTINUE;\n END IF;\n IF sorts.col='id' AND _type IN ('prev','next') THEN\n eq_flag := '';\n ELSE\n eq_flag := '=';\n END IF;\n\n op := concat(\n CASE\n WHEN _type in ('prev','first') AND sorts.dir = 'ASC' THEN '<'\n WHEN _type in ('last','next') AND sorts.dir = 'ASC' THEN '>'\n WHEN _type in ('prev','first') AND sorts.dir = 'DESC' THEN '>'\n WHEN _type in ('last','next') AND sorts.dir = 'DESC' THEN '<'\n END,\n eq_flag\n );\n\n IF _item_j ? sorts.col THEN\n filts = array_append(filts, format('%s %s %L', sorts.col, op, _item_j->>sorts.col));\n END IF;\nEND LOOP;\nret := coalesce(array_to_string(filts,' AND '), 'TRUE');\nRAISE NOTICE 'Order Filter %', ret;\nRETURN ret;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION search_dtrange(IN _indate jsonb, OUT _tstzrange tstzrange) AS\n$$\nWITH t AS (\n SELECT CASE\n WHEN jsonb_typeof(_indate) = 'array' THEN\n textarr(_indate)\n ELSE\n regexp_split_to_array(\n btrim(_indate::text,'\"'),\n '/'\n )\n END AS arr\n)\n, t1 AS (\n SELECT\n CASE\n WHEN array_upper(arr,1) = 1 OR arr[1] = '..' OR arr[1] IS NULL THEN '-infinity'::timestamptz\n ELSE arr[1]::timestamptz\n END AS st,\n CASE\n WHEN array_upper(arr,1) = 1 THEN arr[1]::timestamptz\n WHEN arr[2] = '..' OR arr[2] IS NULL THEN 'infinity'::timestamptz\n ELSE arr[2]::timestamptz\n END AS et\n FROM t\n)\nSELECT\n tstzrange(st,et)\nFROM t1;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION flip_jsonb_array(j jsonb) RETURNS jsonb AS $$\nWITH t AS (\n SELECT i, row_number() over () as r FROM jsonb_array_elements(j) i\n), o AS (\n SELECT i FROM t ORDER BY r DESC\n)\nSELECT jsonb_agg(i) from o\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION search(_search jsonb = '{}'::jsonb) RETURNS SETOF jsonb AS $$\nDECLARE\nqstart timestamptz := clock_timestamp();\n_sort text := '';\n_rsort text := '';\n_limit int := 10;\n_geom geometry;\nqa text[];\npq text[];\nquery text;\npq_prop record;\npq_op record;\nprev_id text := NULL;\nnext_id text := NULL;\nwhereq text := 'TRUE';\nlinks jsonb := '[]'::jsonb;\ntoken text;\ntok_val text;\ntok_q text := 'TRUE';\ntok_sort text;\nfirst_id text;\nfirst_dt timestamptz;\nlast_id text;\nsort text;\nrsort text;\ndt text[];\ndqa text[];\ndq text;\nmq_where text;\nstartdt timestamptz;\nenddt timestamptz;\nitem items%ROWTYPE;\ncounter int := 0;\nbatchcount int;\nmonth timestamptz;\nm record;\n_dtrange tstzrange := tstzrange('-infinity','infinity');\n_dtsort text;\n_token_dtrange tstzrange := tstzrange('-infinity','infinity');\n_token_record items%ROWTYPE;\nis_prev boolean := false;\nincludes text[];\nexcludes text[];\nBEGIN\n-- Create table from sort query of items to sort\nCREATE TEMP TABLE pgstac_tmp_sorts ON COMMIT DROP AS SELECT * FROM sort_base(_search->'sortby');\n\n-- Get the datetime sort direction, necessary for efficient cycling through partitions\nSELECT INTO _dtsort dir FROM pgstac_tmp_sorts WHERE key='datetime';\nRAISE NOTICE '_dtsort: %',_dtsort;\n\nSELECT INTO _sort string_agg(s.sort,', ') FROM pgstac_tmp_sorts s;\nSELECT INTO _rsort string_agg(s.rsort,', ') FROM pgstac_tmp_sorts s;\ntok_sort := _sort;\n\n\n-- Get datetime from query as a tstzrange\nIF _search ? 'datetime' THEN\n _dtrange := search_dtrange(_search->'datetime');\n _token_dtrange := _dtrange;\nEND IF;\n\n-- Get the paging token\nIF _search ? 'token' THEN\n token := _search->>'token';\n tok_val := substr(token,6);\n IF starts_with(token, 'prev:') THEN\n is_prev := true;\n END IF;\n SELECT INTO _token_record * FROM items WHERE id=tok_val;\n IF\n (is_prev AND _dtsort = 'DESC')\n OR\n (not is_prev AND _dtsort = 'ASC')\n THEN\n _token_dtrange := _dtrange * tstzrange(_token_record.datetime, 'infinity');\n ELSIF\n _dtsort IS NOT NULL\n THEN\n _token_dtrange := _dtrange * tstzrange('-infinity',_token_record.datetime);\n END IF;\n IF is_prev THEN\n tok_q := filter_by_order(tok_val, _search->'sortby', 'first');\n _sort := _rsort;\n ELSIF starts_with(token, 'next:') THEN\n tok_q := filter_by_order(tok_val, _search->'sortby', 'last');\n END IF;\nEND IF;\nRAISE NOTICE 'timing: %', age(clock_timestamp(), qstart);\nRAISE NOTICE 'tok_q: % _token_dtrange: %', tok_q, _token_dtrange;\n\nIF _search ? 'ids' THEN\n RAISE NOTICE 'searching solely based on ids... %',_search;\n qa := array_append(qa, in_array_q('id', _search->'ids'));\nELSE\n IF _search ? 'intersects' THEN\n _geom := ST_SetSRID(ST_GeomFromGeoJSON(_search->>'intersects'), 4326);\n ELSIF _search ? 'bbox' THEN\n _geom := bbox_geom(_search->'bbox');\n END IF;\n\n IF _geom IS NOT NULL THEN\n qa := array_append(qa, format('st_intersects(geometry, %L::geometry)',_geom));\n END IF;\n\n IF _search ? 'collections' THEN\n qa := array_append(qa, in_array_q('collection_id', _search->'collections'));\n END IF;\n\n IF _search ? 'query' THEN\n qa := array_cat(qa,\n stac_query(_search->'query')\n );\n END IF;\nEND IF;\n\nIF _search ? 'limit' THEN\n _limit := (_search->>'limit')::int;\nEND IF;\n\nIF _search ? 'fields' THEN\n IF _search->'fields' ? 'exclude' THEN\n excludes=textarr(_search->'fields'->'exclude');\n END IF;\n IF _search->'fields' ? 'include' THEN\n includes=textarr(_search->'fields'->'include');\n IF array_length(includes, 1)>0 AND NOT 'id' = ANY (includes) THEN\n includes = includes || '{id}';\n END IF;\n END IF;\n RAISE NOTICE 'Includes: %, Excludes: %', includes, excludes;\nEND IF;\n\nwhereq := COALESCE(array_to_string(qa,' AND '),' TRUE ');\ndq := COALESCE(array_to_string(dqa,' AND '),' TRUE ');\nRAISE NOTICE 'timing before temp table: %', age(clock_timestamp(), qstart);\n\nCREATE TEMP TABLE results_page ON COMMIT DROP AS\nSELECT * FROM items_by_partition(\n concat(whereq, ' AND ', tok_q),\n _token_dtrange,\n _sort,\n _limit + 1\n);\nRAISE NOTICE 'timing after temp table: %', age(clock_timestamp(), qstart);\n\nRAISE NOTICE 'timing before min/max: %', age(clock_timestamp(), qstart);\n\nIF is_prev THEN\n SELECT INTO last_id, first_id, counter\n first_value(id) OVER (),\n last_value(id) OVER (),\n count(*) OVER ()\n FROM results_page;\nELSE\n SELECT INTO first_id, last_id, counter\n first_value(id) OVER (),\n last_value(id) OVER (),\n count(*) OVER ()\n FROM results_page;\nEND IF;\nRAISE NOTICE 'firstid: %, lastid %', first_id, last_id;\nRAISE NOTICE 'timing after min/max: %', age(clock_timestamp(), qstart);\n\n\n\n\nIF counter > _limit THEN\n next_id := last_id;\n RAISE NOTICE 'next_id: %', next_id;\nELSE\n RAISE NOTICE 'No more next';\nEND IF;\n\nIF tok_q = 'TRUE' THEN\n RAISE NOTICE 'Not a paging query, no previous item';\nELSE\n RAISE NOTICE 'Getting previous item id';\n RAISE NOTICE 'timing: %', age(clock_timestamp(), qstart);\n SELECT INTO _token_record * FROM items WHERE id=first_id;\n IF\n _dtsort = 'DESC'\n THEN\n _token_dtrange := _dtrange * tstzrange(_token_record.datetime, 'infinity');\n ELSE\n _token_dtrange := _dtrange * tstzrange('-infinity',_token_record.datetime);\n END IF;\n RAISE NOTICE '% %', _token_dtrange, _dtrange;\n SELECT id INTO prev_id FROM items_by_partition(\n concat(whereq, ' AND ', filter_by_order(first_id, _search->'sortby', 'prev')),\n _token_dtrange,\n _rsort,\n 1\n );\n RAISE NOTICE 'timing: %', age(clock_timestamp(), qstart);\n\n RAISE NOTICE 'prev_id: %', prev_id;\nEND IF;\n\n\nRETURN QUERY\nWITH features AS (\n SELECT filter_jsonb(content, includes, excludes) as content\n FROM results_page LIMIT _limit\n),\nj AS (SELECT jsonb_agg(content) as feature_arr FROM features)\nSELECT jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce (\n CASE WHEN is_prev THEN flip_jsonb_array(feature_arr) ELSE feature_arr END\n ,'[]'::jsonb),\n 'links', links,\n 'timeStamp', now(),\n 'next', next_id,\n 'prev', prev_id\n)\nFROM j\n;\n\n\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\nINSERT INTO pgstac.migrations (version) VALUES ('0.2.8');\n" }, { "path": "src/pgstac/migrations/pgstac.0.2.9-0.3.0.sql", "content": "SET SEARCH_PATH to pgstac, public;\nALTER SCHEMA pgstac RENAME TO pgstac_bk;\n\nCREATE SCHEMA pgstac;\n\nALTER TABLE pgstac_bk.migrations SET SCHEMA pgstac;\nALTER TABLE pgstac_bk.collections SET SCHEMA pgstac;\n\nCREATE OR REPLACE FUNCTION notice(VARIADIC text[]) RETURNS boolean AS $$\nDECLARE\ndebug boolean := current_setting('pgstac.debug', true);\nBEGIN\n IF debug THEN\n RAISE NOTICE 'NOTICE FROM FUNC: % >>>>> %', concat_ws(' | ', $1), clock_timestamp();\n RETURN TRUE;\n END IF;\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION empty_arr(ANYARRAY) RETURNS BOOLEAN AS $$\nSELECT CASE\n WHEN $1 IS NULL THEN TRUE\n WHEN cardinality($1)<1 THEN TRUE\nELSE FALSE\nEND;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_map_ident(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_ident(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_map_literal(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_literal(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION estimated_count(_where text) RETURNS bigint AS $$\nDECLARE\nrec record;\nrows bigint;\nBEGIN\n FOR rec in EXECUTE format(\n $q$\n EXPLAIN SELECT 1 FROM items WHERE %s\n $q$,\n _where)\n LOOP\n rows := substring(rec.\"QUERY PLAN\" FROM ' rows=([[:digit:]]+)');\n EXIT WHEN rows IS NOT NULL;\n END LOOP;\n\n RETURN rows;\nEND;\n$$ LANGUAGE PLPGSQL;\n/* converts a jsonb text array to a pg text[] array */\nCREATE OR REPLACE FUNCTION textarr(_js jsonb)\n RETURNS text[] AS $$\n SELECT\n CASE jsonb_typeof(_js)\n WHEN 'array' THEN ARRAY(SELECT jsonb_array_elements_text(_js))\n ELSE ARRAY[_js->>0]\n END\n;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION jsonb_paths (IN jdata jsonb, OUT path text[], OUT value jsonb) RETURNS\nSETOF RECORD AS $$\nwith recursive extract_all as\n(\n select\n ARRAY[key]::text[] as path,\n value\n FROM jsonb_each(jdata)\nunion all\n select\n path || coalesce(obj_key, (arr_key- 1)::text),\n coalesce(obj_value, arr_value)\n from extract_all\n left join lateral\n jsonb_each(case jsonb_typeof(value) when 'object' then value end)\n as o(obj_key, obj_value)\n on jsonb_typeof(value) = 'object'\n left join lateral\n jsonb_array_elements(case jsonb_typeof(value) when 'array' then value end)\n with ordinality as a(arr_value, arr_key)\n on jsonb_typeof(value) = 'array'\n where obj_key is not null or arr_key is not null\n)\nselect *\nfrom extract_all;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION jsonb_obj_paths (IN jdata jsonb, OUT path text[], OUT value jsonb) RETURNS\nSETOF RECORD AS $$\nwith recursive extract_all as\n(\n select\n ARRAY[key]::text[] as path,\n value\n FROM jsonb_each(jdata)\nunion all\n select\n path || obj_key,\n obj_value\n from extract_all\n left join lateral\n jsonb_each(case jsonb_typeof(value) when 'object' then value end)\n as o(obj_key, obj_value)\n on jsonb_typeof(value) = 'object'\n where obj_key is not null\n)\nselect *\nfrom extract_all;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION jsonb_val_paths (IN jdata jsonb, OUT path text[], OUT value jsonb) RETURNS\nSETOF RECORD AS $$\nSELECT * FROM jsonb_obj_paths(jdata) WHERE jsonb_typeof(value) not in ('object','array');\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION path_includes(IN path text[], IN includes text[]) RETURNS BOOLEAN AS $$\nWITH t AS (SELECT unnest(includes) i)\nSELECT EXISTS (\n SELECT 1 FROM t WHERE path @> string_to_array(trim(i), '.')\n);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION path_excludes(IN path text[], IN excludes text[]) RETURNS BOOLEAN AS $$\nWITH t AS (SELECT unnest(excludes) e)\nSELECT NOT EXISTS (\n SELECT 1 FROM t WHERE path @> string_to_array(trim(e), '.')\n);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION jsonb_obj_paths_filtered (\n IN jdata jsonb,\n IN includes text[] DEFAULT ARRAY[]::text[],\n IN excludes text[] DEFAULT ARRAY[]::text[],\n OUT path text[],\n OUT value jsonb\n) RETURNS\nSETOF RECORD AS $$\nSELECT path, value\nFROM jsonb_obj_paths(jdata)\nWHERE\n CASE WHEN cardinality(includes) > 0 THEN path_includes(path, includes) ELSE TRUE END\n AND\n path_excludes(path, excludes)\n\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION filter_jsonb(\n IN jdata jsonb,\n IN includes text[] DEFAULT ARRAY[]::text[],\n IN excludes text[] DEFAULT ARRAY[]::text[]\n) RETURNS jsonb AS $$\nDECLARE\nrec RECORD;\noutj jsonb := '{}'::jsonb;\ncreated_paths text[] := '{}'::text[];\nBEGIN\n\nIF empty_arr(includes) AND empty_arr(excludes) THEN\nRAISE NOTICE 'no filter';\n RETURN jdata;\nEND IF;\nFOR rec in\nSELECT * FROM jsonb_obj_paths_filtered(jdata, includes, excludes)\nWHERE jsonb_typeof(value) != 'object'\nLOOP\n IF array_length(rec.path,1)>1 THEN\n FOR i IN 1..(array_length(rec.path,1)-1) LOOP\n IF NOT array_to_string(rec.path[1:i],'.') = ANY (created_paths) THEN\n outj := jsonb_set(outj, rec.path[1:i],'{}', true);\n created_paths := created_paths || array_to_string(rec.path[1:i],'.');\n END IF;\n END LOOP;\n END IF;\n outj := jsonb_set(outj, rec.path, rec.value, true);\n created_paths := created_paths || array_to_string(rec.path,'.');\nEND LOOP;\nRETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION flip_jsonb_array(j jsonb) RETURNS jsonb AS $$\nSELECT jsonb_agg(value) FROM (SELECT value FROM jsonb_array_elements(j) WITH ORDINALITY ORDER BY ordinality DESC) as t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n/* Functions to create an iterable of cursors over partitions. */\nCREATE OR REPLACE FUNCTION create_cursor(q text) RETURNS refcursor AS $$\nDECLARE\n curs refcursor;\nBEGIN\n OPEN curs FOR EXECUTE q;\n RETURN curs;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION partition_queries(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC'\n) RETURNS SETOF text AS $$\nDECLARE\n partition_query text;\n query text;\n p record;\n cursors refcursor;\nBEGIN\nIF _orderby ILIKE 'datetime d%' THEN\n partition_query := format($q$\n SELECT partition, tstzrange\n FROM items_partitions\n ORDER BY tstzrange DESC;\n $q$);\nELSIF _orderby ILIKE 'datetime a%' THEN\n partition_query := format($q$\n SELECT partition, tstzrange\n FROM items_partitions\n ORDER BY tstzrange ASC\n ;\n $q$);\nELSE\n query := format($q$\n SELECT * FROM items\n WHERE %s\n ORDER BY %s\n $q$, _where, _orderby\n );\n\n RETURN NEXT query;\n RETURN;\nEND IF;\n--RAISE NOTICE 'Partition Query: %', partition_query;\nFOR p IN\n EXECUTE partition_query\nLOOP\n query := format($q$\n SELECT * FROM items\n WHERE datetime >= %L AND datetime < %L AND %s\n ORDER BY %s\n $q$, lower(p.tstzrange), upper(p.tstzrange), _where, _orderby\n );\n --RAISE NOTICE 'query: %', query;\n RETURN NEXT query;\nEND LOOP;\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION partition_cursor(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC'\n) RETURNS SETOF refcursor AS $$\nDECLARE\n partition_query text;\n query text;\n p record;\n cursors refcursor;\nBEGIN\nFOR query IN SELECT * FROM partion_queries(_where, _orderby) LOOP\n RETURN NEXT create_cursor(query);\nEND LOOP;\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION partition_count(\n IN _where text DEFAULT 'TRUE'\n) RETURNS bigint AS $$\nDECLARE\n partition_query text;\n query text;\n p record;\n subtotal bigint;\n total bigint := 0;\nBEGIN\npartition_query := format($q$\n SELECT partition, tstzrange\n FROM items_partitions\n ORDER BY tstzrange DESC;\n$q$);\nRAISE NOTICE 'Partition Query: %', partition_query;\nFOR p IN\n EXECUTE partition_query\nLOOP\n query := format($q$\n SELECT count(*) FROM items\n WHERE datetime BETWEEN %L AND %L AND %s\n $q$, lower(p.tstzrange), upper(p.tstzrange), _where\n );\n RAISE NOTICE 'Query %', query;\n RAISE NOTICE 'Partition %, Count %, Total %',p.partition, subtotal, total;\n EXECUTE query INTO subtotal;\n total := subtotal + total;\nEND LOOP;\nRETURN total;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n/* looks for a geometry in a stac item first from geometry and falling back to bbox */\nCREATE OR REPLACE FUNCTION stac_geom(value jsonb) RETURNS geometry AS $$\nSELECT\n CASE\n WHEN value->>'geometry' IS NOT NULL THEN\n ST_GeomFromGeoJSON(value->>'geometry')\n WHEN value->>'bbox' IS NOT NULL THEN\n ST_MakeEnvelope(\n (value->'bbox'->>0)::float,\n (value->'bbox'->>1)::float,\n (value->'bbox'->>2)::float,\n (value->'bbox'->>3)::float,\n 4326\n )\n ELSE NULL\n END as geometry\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION stac_datetime(value jsonb) RETURNS timestamptz AS $$\nSELECT COALESCE(\n (value->'properties'->>'datetime')::timestamptz,\n (value->'properties'->>'start_datetime')::timestamptz\n);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_end_datetime(value jsonb) RETURNS timestamptz AS $$\nSELECT COALESCE(\n (value->'properties'->>'datetime')::timestamptz,\n (value->'properties'->>'end_datetime')::timestamptz\n);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\n\nCREATE OR REPLACE FUNCTION stac_daterange(value jsonb) RETURNS tstzrange AS $$\nSELECT tstzrange(stac_datetime(value),stac_end_datetime(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE TABLE IF NOT EXISTS collections (\n id VARCHAR GENERATED ALWAYS AS (content->>'id') STORED PRIMARY KEY,\n content JSONB\n);\n\nCREATE OR REPLACE FUNCTION create_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection(data jsonb) RETURNS VOID AS $$\nDECLARE\nout collections%ROWTYPE;\nBEGIN\n UPDATE collections SET content=data WHERE id = data->>'id' RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ON CONFLICT (id) DO\n UPDATE\n SET content=EXCLUDED.content\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION delete_collection(_id text) RETURNS VOID AS $$\nDECLARE\nout collections%ROWTYPE;\nBEGIN\n DELETE FROM collections WHERE id = _id RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION get_collection(id text) RETURNS jsonb AS $$\nSELECT content FROM collections\nWHERE id=$1\n;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION all_collections() RETURNS jsonb AS $$\nSELECT jsonb_agg(content) FROM collections;\n;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE TABLE IF NOT EXISTS items (\n id text NOT NULL,\n geometry geometry NOT NULL,\n collection_id text NOT NULL,\n datetime timestamptz NOT NULL,\n end_datetime timestamptz NOT NULL,\n properties jsonb NOT NULL,\n content JSONB NOT NULL\n)\nPARTITION BY RANGE (datetime)\n;\n\nCREATE OR REPLACE FUNCTION properties_idx (IN content jsonb) RETURNS jsonb AS $$\n with recursive extract_all as\n (\n select\n ARRAY[key]::text[] as path,\n ARRAY[key]::text[] as fullpath,\n value\n FROM jsonb_each(content->'properties')\n union all\n select\n CASE WHEN obj_key IS NOT NULL THEN path || obj_key ELSE path END,\n path || coalesce(obj_key, (arr_key- 1)::text),\n coalesce(obj_value, arr_value)\n from extract_all\n left join lateral\n jsonb_each(case jsonb_typeof(value) when 'object' then value end)\n as o(obj_key, obj_value)\n on jsonb_typeof(value) = 'object'\n left join lateral\n jsonb_array_elements(case jsonb_typeof(value) when 'array' then value end)\n with ordinality as a(arr_value, arr_key)\n on jsonb_typeof(value) = 'array'\n where obj_key is not null or arr_key is not null\n )\n , paths AS (\n select\n array_to_string(path, '.') as path,\n value\n FROM extract_all\n WHERE\n jsonb_typeof(value) NOT IN ('array','object')\n ), grouped AS (\n SELECT path, jsonb_agg(distinct value) vals FROM paths group by path\n ) SELECT coalesce(jsonb_object_agg(path, CASE WHEN jsonb_array_length(vals)=1 THEN vals->0 ELSE vals END) - '{datetime}'::text[], '{}'::jsonb) FROM grouped\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET JIT TO OFF;\n\n\nCREATE OR REPLACE FUNCTION analyze_empty_partitions() RETURNS VOID AS $$\nDECLARE\n p text;\nBEGIN\n FOR p IN SELECT partition FROM all_items_partitions WHERE est_cnt = 0 LOOP\n EXECUTE format('ANALYZE %I;', p);\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION items_partition_name(timestamptz) RETURNS text AS $$\n SELECT to_char($1, '\"items_p\"IYYY\"w\"IW');\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION items_partition_exists(text) RETURNS boolean AS $$\n SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_class WHERE relname=$1);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION items_partition_exists(timestamptz) RETURNS boolean AS $$\n SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_class WHERE relname=items_partition_name($1));\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION items_partition_create_worker(partition text, partition_start timestamptz, partition_end timestamptz) RETURNS VOID AS $$\nDECLARE\n err_context text;\nBEGIN\n EXECUTE format(\n $f$\n CREATE TABLE IF NOT EXISTS %1$I PARTITION OF items\n FOR VALUES FROM (%2$L) TO (%3$L);\n CREATE UNIQUE INDEX IF NOT EXISTS %4$I ON %1$I (id);\n $f$,\n partition,\n partition_start,\n partition_end,\n concat(partition, '_id_pk')\n );\nEXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', partition;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH to pgstac, public;\n\nCREATE OR REPLACE FUNCTION items_partition_create(ts timestamptz) RETURNS text AS $$\nDECLARE\n partition text := items_partition_name(ts);\n partition_start timestamptz;\n partition_end timestamptz;\nBEGIN\n IF items_partition_exists(partition) THEN\n RETURN partition;\n END IF;\n partition_start := date_trunc('week', ts);\n partition_end := partition_start + '1 week'::interval;\n PERFORM items_partition_create_worker(partition, partition_start, partition_end);\n RAISE NOTICE 'partition: %', partition;\n RETURN partition;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION items_partition_create(st timestamptz, et timestamptz) RETURNS SETOF text AS $$\nWITH t AS (\n SELECT\n generate_series(\n date_trunc('week',st),\n date_trunc('week', et),\n '1 week'::interval\n ) w\n)\nSELECT items_partition_create(w) FROM t;\n$$ LANGUAGE SQL;\n\n\nCREATE UNLOGGED TABLE items_staging (\n content JSONB NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION items_staging_insert_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n mindate timestamptz;\n maxdate timestamptz;\n partition text;\nBEGIN\n SELECT min(stac_datetime(content)), max(stac_datetime(content)) INTO mindate, maxdate FROM newdata;\n PERFORM items_partition_create(mindate, maxdate);\n INSERT INTO items (id, geometry, collection_id, datetime, end_datetime, properties, content)\n SELECT\n content->>'id',\n stac_geom(content),\n content->>'collection',\n stac_datetime(content),\n stac_end_datetime(content),\n properties_idx(content),\n content\n FROM newdata\n ;\n DELETE FROM items_staging;\n PERFORM analyze_empty_partitions();\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER items_staging_insert_trigger AFTER INSERT ON items_staging REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_insert_triggerfunc();\n\n\nCREATE UNLOGGED TABLE items_staging_ignore (\n content JSONB NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION items_staging_ignore_insert_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n mindate timestamptz;\n maxdate timestamptz;\n partition text;\nBEGIN\n SELECT min(stac_datetime(content)), max(stac_datetime(content)) INTO mindate, maxdate FROM newdata;\n PERFORM items_partition_create(mindate, maxdate);\n INSERT INTO items (id, geometry, collection_id, datetime, end_datetime, properties, content)\n SELECT\n content->>'id',\n stac_geom(content),\n content->>'collection',\n stac_datetime(content),\n stac_end_datetime(content),\n properties_idx(content),\n content\n FROM newdata\n ON CONFLICT DO NOTHING\n ;\n DELETE FROM items_staging_ignore;\n PERFORM analyze_empty_partitions();\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER items_staging_ignore_insert_trigger AFTER INSERT ON items_staging_ignore REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_ignore_insert_triggerfunc();\n\nCREATE UNLOGGED TABLE items_staging_upsert (\n content JSONB NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION items_staging_upsert_insert_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n mindate timestamptz;\n maxdate timestamptz;\n partition text;\nBEGIN\n SELECT min(stac_datetime(content)), max(stac_datetime(content)) INTO mindate, maxdate FROM newdata;\n PERFORM items_partition_create(mindate, maxdate);\n INSERT INTO items (id, geometry, collection_id, datetime, end_datetime, properties, content)\n SELECT\n content->>'id',\n stac_geom(content),\n content->>'collection',\n stac_datetime(content),\n stac_end_datetime(content),\n properties_idx(content),\n content\n FROM newdata\n ON CONFLICT (datetime, id) DO UPDATE SET\n content = EXCLUDED.content\n WHERE items.content IS DISTINCT FROM EXCLUDED.content\n ;\n DELETE FROM items_staging_upsert;\n PERFORM analyze_empty_partitions();\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER items_staging_upsert_insert_trigger AFTER INSERT ON items_staging_upsert REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_upsert_insert_triggerfunc();\n\nCREATE OR REPLACE FUNCTION items_update_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\nBEGIN\n NEW.id := NEW.content->>'id';\n NEW.datetime := stac_datetime(NEW.content);\n NEW.end_datetime := stac_end_datetime(NEW.content);\n NEW.collection_id := NEW.content->>'collection';\n NEW.geometry := stac_geom(NEW.content);\n NEW.properties := properties_idx(NEW.content);\n IF TG_OP = 'UPDATE' AND NEW IS NOT DISTINCT FROM OLD THEN\n RETURN NULL;\n END IF;\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER items_update_trigger BEFORE UPDATE ON items\n FOR EACH ROW EXECUTE PROCEDURE items_update_triggerfunc();\n\n/*\nView to get a table of available items partitions\nwith date ranges\n*/\nCREATE VIEW all_items_partitions AS\nWITH base AS\n(SELECT\n c.oid::pg_catalog.regclass::text as partition,\n pg_catalog.pg_get_expr(c.relpartbound, c.oid) as _constraint,\n regexp_matches(\n pg_catalog.pg_get_expr(c.relpartbound, c.oid),\n E'\\\\(''\\([0-9 :+-]*\\)''\\\\).*\\\\(''\\([0-9 :+-]*\\)''\\\\)'\n ) as t,\n reltuples::bigint as est_cnt\nFROM pg_catalog.pg_class c, pg_catalog.pg_inherits i\nWHERE c.oid = i.inhrelid AND i.inhparent = 'items'::regclass)\nSELECT partition, tstzrange(\n t[1]::timestamptz,\n t[2]::timestamptz\n), est_cnt\nFROM base\nORDER BY 2 desc;\n\nCREATE OR REPLACE VIEW items_partitions AS\nSELECT * FROM all_items_partitions WHERE est_cnt>0;\n\nCREATE OR REPLACE FUNCTION get_item(_id text) RETURNS jsonb AS $$\n SELECT content FROM items WHERE id=_id;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION delete_item(_id text) RETURNS VOID AS $$\nDECLARE\nout items%ROWTYPE;\nBEGIN\n DELETE FROM items WHERE id = _id RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION create_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION update_item(data jsonb) RETURNS VOID AS $$\nDECLARE\n out items%ROWTYPE;\nBEGIN\n UPDATE items SET content=data WHERE id = data->>'id' RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION collection_bbox(id text) RETURNS jsonb AS $$\nSELECT (replace(replace(replace(st_extent(geometry)::text,'BOX(','[['),')',']]'),' ',','))::jsonb\nFROM items WHERE collection_id=$1;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION collection_temporal_extent(id text) RETURNS jsonb AS $$\nSELECT to_jsonb(array[array[min(datetime)::text, max(datetime)::text]])\nFROM items WHERE collection_id=$1;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection_extents() RETURNS VOID AS $$\nUPDATE collections SET\n content = content ||\n jsonb_build_object(\n 'extent', jsonb_build_object(\n 'spatial', jsonb_build_object(\n 'bbox', collection_bbox(collections.id)\n ),\n 'temporal', jsonb_build_object(\n 'interval', collection_temporal_extent(collections.id)\n )\n )\n )\n;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\n----- BEGIN SEARCH\n\nCREATE OR REPLACE FUNCTION items_path(\n IN dotpath text,\n OUT field text,\n OUT path text,\n OUT path_txt text,\n OUT jsonpath text,\n OUT eq text\n) RETURNS RECORD AS $$\nDECLARE\npath_elements text[];\nlast_element text;\nBEGIN\ndotpath := replace(trim(dotpath), 'properties.', '');\n\nIF dotpath = '' THEN\n RETURN;\nEND IF;\n\npath_elements := string_to_array(dotpath, '.');\njsonpath := NULL;\n\nIF path_elements[1] IN ('id','geometry','datetime') THEN\n field := path_elements[1];\n path_elements := path_elements[2:];\nELSIF path_elements[1] = 'collection' THEN\n field := 'collection_id';\n path_elements := path_elements[2:];\nELSIF path_elements[1] IN ('links', 'assets', 'stac_version', 'stac_extensions') THEN\n field := 'content';\nELSE\n field := 'content';\n path_elements := '{properties}'::text[] || path_elements;\nEND IF;\nIF cardinality(path_elements)<1 THEN\n path := field;\n path_txt := field;\n jsonpath := '$';\n eq := NULL; -- format($F$ %s = %%s $F$, field);\n RETURN;\nEND IF;\n\n\nlast_element := path_elements[cardinality(path_elements)];\npath_elements := path_elements[1:cardinality(path_elements)-1];\njsonpath := concat(array_to_string('{$}'::text[] || array_map_ident(path_elements), '.'), '.', quote_ident(last_element));\npath_elements := array_map_literal(path_elements);\npath := format($F$ properties->%s $F$, quote_literal(dotpath));\npath_txt := format($F$ properties->>%s $F$, quote_literal(dotpath));\neq := format($F$ properties @? '$.%s[*] ? (@ == %%s) '$F$, quote_ident(dotpath));\n\nRAISE NOTICE 'ITEMS PATH -- % % % % %', field, path, path_txt, jsonpath, eq;\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION parse_dtrange(IN _indate jsonb, OUT _tstzrange tstzrange) AS $$\nWITH t AS (\n SELECT CASE\n WHEN jsonb_typeof(_indate) = 'array' THEN\n textarr(_indate)\n ELSE\n regexp_split_to_array(\n btrim(_indate::text,'\"'),\n '/'\n )\n END AS arr\n)\n, t1 AS (\n SELECT\n CASE\n WHEN array_upper(arr,1) = 1 OR arr[1] = '..' OR arr[1] IS NULL THEN '-infinity'::timestamptz\n ELSE arr[1]::timestamptz\n END AS st,\n CASE\n WHEN array_upper(arr,1) = 1 THEN arr[1]::timestamptz\n WHEN arr[2] = '..' OR arr[2] IS NULL THEN 'infinity'::timestamptz\n ELSE arr[2]::timestamptz\n END AS et\n FROM t\n)\nSELECT\n tstzrange(st,et)\nFROM t1;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION bbox_geom(_bbox jsonb) RETURNS geometry AS $$\nSELECT CASE jsonb_array_length(_bbox)\n WHEN 4 THEN\n ST_SetSRID(ST_MakeEnvelope(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float,\n (_bbox->>3)::float\n ),4326)\n WHEN 6 THEN\n ST_SetSRID(ST_3DMakeBox(\n ST_MakePoint(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float\n ),\n ST_MakePoint(\n (_bbox->>3)::float,\n (_bbox->>4)::float,\n (_bbox->>5)::float\n )\n ),4326)\n ELSE null END;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION cql_and_append(existing jsonb, newfilters jsonb) RETURNS jsonb AS $$\nSELECT CASE WHEN existing ? 'filter' AND newfilters IS NOT NULL THEN\n jsonb_build_object(\n 'and',\n jsonb_build_array(\n existing->'filter',\n newfilters\n )\n )\nELSE\n newfilters\nEND;\n$$ LANGUAGE SQL;\n\n\n-- ADDs base filters (ids, collections, datetime, bbox, intersects) that are\n-- added outside of the filter/query in the stac request\nCREATE OR REPLACE FUNCTION add_filters_to_cql(j jsonb) RETURNS jsonb AS $$\nDECLARE\nnewprop jsonb;\nnewprops jsonb := '[]'::jsonb;\nBEGIN\nIF j ? 'id' THEN\n newprop := jsonb_build_object(\n 'in',\n jsonb_build_array(\n '{\"property\":\"id\"}'::jsonb,\n j->'id'\n )\n );\n newprops := jsonb_insert(newprops, '{1}', newprop);\nEND IF;\nIF j ? 'collection' THEN\n newprop := jsonb_build_object(\n 'in',\n jsonb_build_array(\n '{\"property\":\"collection\"}'::jsonb,\n j->'collection'\n )\n );\n newprops := jsonb_insert(newprops, '{1}', newprop);\nEND IF;\n\nIF j ? 'datetime' THEN\n newprop := format(\n '{\"anyinteracts\":[{\"property\":\"datetime\"}, %s]}',\n j->'datetime'\n );\n newprops := jsonb_insert(newprops, '{1}', newprop);\nEND IF;\n\nIF j ? 'bbox' THEN\n newprop := format(\n '{\"intersects\":[{\"property\":\"geometry\"}, %s]}',\n j->'bbox'\n );\n newprops := jsonb_insert(newprops, '{1}', newprop);\nEND IF;\n\nIF j ? 'intersects' THEN\n newprop := format(\n '{\"intersects\":[{\"property\":\"geometry\"}, %s]}',\n j->'intersects'\n );\n newprops := jsonb_insert(newprops, '{1}', newprop);\nEND IF;\n\nRAISE NOTICE 'newprops: %', newprops;\n\nIF newprops IS NOT NULL AND jsonb_array_length(newprops) > 0 THEN\n return jsonb_set(\n j,\n '{filter}',\n cql_and_append(j, jsonb_build_object('and', newprops))\n ) - '{id,collection,datetime,bbox,intersects}'::text[];\nEND IF;\n\nreturn j;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION query_to_cqlfilter(j jsonb) RETURNS jsonb AS $$\n-- Translates anything passed in through the deprecated \"query\" into equivalent CQL\nWITH t AS (\n SELECT key as property, value as ops\n FROM jsonb_each(j->'query')\n), t2 AS (\n SELECT property, (jsonb_each(ops)).*\n FROM t WHERE jsonb_typeof(ops) = 'object'\n UNION ALL\n SELECT property, 'eq', ops\n FROM t WHERE jsonb_typeof(ops) != 'object'\n), t3 AS (\nSELECT\n jsonb_strip_nulls(jsonb_build_object(\n 'and',\n jsonb_agg(\n jsonb_build_object(\n key,\n jsonb_build_array(\n jsonb_build_object('property',property),\n value\n )\n )\n )\n )) as qcql FROM t2\n)\nSELECT\n CASE WHEN qcql IS NOT NULL THEN\n jsonb_set(j, '{filter}', cql_and_append(j, qcql)) - 'query'\n ELSE j\n END\nFROM t3\n;\n$$ LANGUAGE SQL;\n\n\n\nCREATE OR REPLACE FUNCTION temporal_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\nll text := 'datetime';\nlh text := 'end_datetime';\nrrange tstzrange;\nrl text;\nrh text;\noutq text;\nBEGIN\nrrange := parse_dtrange(args->1);\nRAISE NOTICE 'Constructing temporal query OP: %, ARGS: %, RRANGE: %', op, args, rrange;\nop := lower(op);\nrl := format('%L::timestamptz', lower(rrange));\nrh := format('%L::timestamptz', upper(rrange));\noutq := CASE op\n WHEN 't_before' THEN 'lh < rl'\n WHEN 't_after' THEN 'll > rh'\n WHEN 't_meets' THEN 'lh = rl'\n WHEN 't_metby' THEN 'll = rh'\n WHEN 't_overlaps' THEN 'll < rl AND rl < lh < rh'\n WHEN 't_overlappedby' THEN 'rl < ll < rh AND lh > rh'\n WHEN 't_starts' THEN 'll = rl AND lh < rh'\n WHEN 't_startedby' THEN 'll = rl AND lh > rh'\n WHEN 't_during' THEN 'll > rl AND lh < rh'\n WHEN 't_contains' THEN 'll < rl AND lh > rh'\n WHEN 't_finishes' THEN 'll > rl AND lh = rh'\n WHEN 't_finishedby' THEN 'll < rl AND lh = rh'\n WHEN 't_equals' THEN 'll = rl AND lh = rh'\n WHEN 't_disjoint' THEN 'NOT (ll <= rh AND lh >= rl)'\n WHEN 't_intersects' THEN 'll <= rh AND lh >= rl'\n WHEN 'anyinteracts' THEN 'll <= rh AND lh >= rl'\nEND;\noutq := regexp_replace(outq, '\\mll\\M', ll);\noutq := regexp_replace(outq, '\\mlh\\M', lh);\noutq := regexp_replace(outq, '\\mrl\\M', rl);\noutq := regexp_replace(outq, '\\mrh\\M', rh);\noutq := format('(%s)', outq);\nRETURN outq;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION spatial_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\ngeom text;\nj jsonb := args->1;\nBEGIN\nop := lower(op);\nRAISE NOTICE 'Constructing spatial query OP: %, ARGS: %', op, args;\nIF op NOT IN ('s_equals','s_disjoint','s_touches','s_within','s_overlaps','s_crosses','s_intersects','intersects','s_contains') THEN\n RAISE EXCEPTION 'Spatial Operator % Not Supported', op;\nEND IF;\nop := regexp_replace(op, '^s_', 'st_');\nIF op = 'intersects' THEN\n op := 'st_intersects';\nEND IF;\n-- Convert geometry to WKB string\nIF j ? 'type' AND j ? 'coordinates' THEN\n geom := st_geomfromgeojson(j)::text;\nELSIF jsonb_typeof(j) = 'array' THEN\n geom := bbox_geom(j)::text;\nEND IF;\n\nRETURN format('%s(geometry, %L::geometry)', op, geom);\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n/* cql_query_op -- Parses a CQL query operation, recursing when necessary\n IN jsonb -- a subelement from a valid stac query\n IN text -- the operator being used on elements passed in\n RETURNS a SQL fragment to be used in a WHERE clause\n*/\nCREATE OR REPLACE FUNCTION cql_query_op(j jsonb, _op text DEFAULT NULL) RETURNS text AS $$\nDECLARE\njtype text := jsonb_typeof(j);\nop text := lower(_op);\nops jsonb :=\n '{\n \"eq\": \"%s = %s\",\n \"lt\": \"%s < %s\",\n \"lte\": \"%s <= %s\",\n \"gt\": \"%s > %s\",\n \"gte\": \"%s >= %s\",\n \"like\": \"%s LIKE %s\",\n \"+\": \"%s + %s\",\n \"-\": \"%s - %s\",\n \"*\": \"%s * %s\",\n \"/\": \"%s / %s\",\n \"in\": \"%s = ANY (%s)\",\n \"not\": \"NOT (%s)\",\n \"between\": \"%s BETWEEN %s AND %s\",\n \"lower\":\"lower(%s)\"\n }'::jsonb;\nret text;\nargs text[] := NULL;\n\nBEGIN\nRAISE NOTICE 'j: %, op: %, jtype: %', j, op, jtype;\n\n-- Set Lower Case on Both Arguments When Case Insensitive Flag Set\nIF op in ('eq','lt','lte','gt','gte','like') AND jsonb_typeof(j->2) = 'boolean' THEN\n IF (j->>2)::boolean THEN\n RETURN format(concat('(',ops->>op,')'), cql_query_op(jsonb_build_array(j->0), 'lower'), cql_query_op(jsonb_build_array(j->1), 'lower'));\n END IF;\nEND IF;\n\n-- Special Case when comparing a property in a jsonb field to a string or number using eq\n-- Allows to leverage GIN index on jsonb fields\nIF op = 'eq' THEN\n IF j->0 ? 'property'\n AND jsonb_typeof(j->1) IN ('number','string')\n AND (items_path(j->0->>'property')).eq IS NOT NULL\n THEN\n RETURN format((items_path(j->0->>'property')).eq, j->1);\n END IF;\nEND IF;\n\nIF op ilike 't_%' or op = 'anyinteracts' THEN\n RETURN temporal_op_query(op, j);\nEND IF;\n\nIF op ilike 's_%' or op = 'intersects' THEN\n RETURN spatial_op_query(op, j);\nEND IF;\n\n\nIF jtype = 'object' THEN\n RAISE NOTICE 'parsing object';\n IF j ? 'property' THEN\n -- Convert the property to be used as an identifier\n return (items_path(j->>'property')).path_txt;\n ELSIF _op IS NULL THEN\n -- Iterate to convert elements in an object where the operator has not been set\n -- Combining with AND\n SELECT\n array_to_string(array_agg(cql_query_op(e.value, e.key)), ' AND ')\n INTO ret\n FROM jsonb_each(j) e;\n RETURN ret;\n END IF;\nEND IF;\n\nIF jtype = 'string' THEN\n RETURN quote_literal(j->>0);\nEND IF;\n\nIF jtype ='number' THEN\n RETURN (j->>0)::numeric;\nEND IF;\n\nIF jtype = 'array' AND op IS NULL THEN\n RAISE NOTICE 'Parsing array into array arg. j: %', j;\n SELECT format($f$ '{%s}'::text[] $f$, string_agg(e,',')) INTO ret FROM jsonb_array_elements_text(j) e;\n RETURN ret;\nEND IF;\n\n\n-- If the type of the passed json is an array\n-- Calculate the arguments that will be passed to functions/operators\nIF jtype = 'array' THEN\n RAISE NOTICE 'Parsing array into args. j: %', j;\n -- If any argument is numeric, cast any text arguments to numeric\n IF j @? '$[*] ? (@.type() == \"number\")' THEN\n SELECT INTO args\n array_agg(concat('(',cql_query_op(e),')::numeric'))\n FROM jsonb_array_elements(j) e;\n ELSE\n SELECT INTO args\n array_agg(cql_query_op(e))\n FROM jsonb_array_elements(j) e;\n END IF;\n --RETURN args;\nEND IF;\nRAISE NOTICE 'ARGS after array cleaning: %', args;\n\nIF op IS NULL THEN\n RETURN args::text[];\nEND IF;\n\nIF args IS NULL OR cardinality(args) < 1 THEN\n RAISE NOTICE 'No Args';\n RETURN '';\nEND IF;\n\nIF op IN ('and','or') THEN\n SELECT\n CONCAT(\n '(',\n array_to_string(args, UPPER(CONCAT(' ',op,' '))),\n ')'\n ) INTO ret\n FROM jsonb_array_elements(j) e;\n RETURN ret;\nEND IF;\n\n-- If the op is in the ops json then run using the template in the json\nIF ops ? op THEN\n RAISE NOTICE 'ARGS: % MAPPED: %',args, array_map_literal(args);\n\n RETURN format(concat('(',ops->>op,')'), VARIADIC args);\nEND IF;\n\nRETURN j->>0;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n\n\nCREATE OR REPLACE FUNCTION cql_to_where(_search jsonb = '{}'::jsonb) RETURNS text AS $$\nDECLARE\nsearch jsonb := _search;\n_where text;\nBEGIN\nRAISE NOTICE 'SEARCH CQL 1: %', search;\n\n-- Convert any old style stac query to cql\nsearch := query_to_cqlfilter(search);\n\nRAISE NOTICE 'SEARCH CQL 2: %', search;\n\n-- Convert item,collection,datetime,bbox,intersects to cql\nsearch := add_filters_to_cql(search);\n\nRAISE NOTICE 'SEARCH CQL Final: %', search;\n_where := cql_query_op(search->'filter');\n\nIF trim(_where) = '' THEN\n _where := NULL;\nEND IF;\n_where := coalesce(_where, ' TRUE ');\nRETURN _where;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION parse_sort_dir(_dir text, reverse boolean default false) RETURNS text AS $$\nWITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n) SELECT\n CASE\n WHEN NOT reverse THEN d\n WHEN d = 'ASC' THEN 'DESC'\n WHEN d = 'DESC' THEN 'ASC'\n END\nFROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION sort_dir_to_op(_dir text, prev boolean default false) RETURNS text AS $$\nWITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n) SELECT\n CASE\n WHEN d = 'ASC' AND prev THEN '<='\n WHEN d = 'DESC' AND prev THEN '>='\n WHEN d = 'ASC' THEN '>='\n WHEN d = 'DESC' THEN '<='\n END\nFROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION sort_sqlorderby(\n _search jsonb DEFAULT NULL,\n reverse boolean DEFAULT FALSE\n) RETURNS text AS $$\nWITH sorts AS (\n SELECT\n (items_path(value->>'field')).path as key,\n parse_sort_dir(value->>'direction', reverse) as dir\n FROM jsonb_array_elements(\n '[]'::jsonb\n ||\n coalesce(_search->'sort','[{\"field\":\"datetime\", \"direction\":\"desc\"}]')\n ||\n '[{\"field\":\"id\",\"direction\":\"desc\"}]'::jsonb\n )\n)\nSELECT array_to_string(\n array_agg(concat(key, ' ', dir)),\n ', '\n) FROM sorts;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_sort_dir(sort_item jsonb) RETURNS text AS $$\nSELECT CASE WHEN sort_item->>'direction' ILIKE 'desc%' THEN 'DESC' ELSE 'ASC' END;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION get_token_filter(_search jsonb = '{}'::jsonb, token_rec jsonb DEFAULT NULL) RETURNS text AS $$\nDECLARE\ntoken_id text;\nfilters text[] := '{}'::text[];\nprev boolean := TRUE;\nfield text;\ndir text;\nsort record;\norfilters text[] := '{}'::text[];\nandfilters text[] := '{}'::text[];\noutput text;\ntoken_where text;\nBEGIN\n-- If no token provided return NULL\nIF token_rec IS NULL THEN\n IF NOT (_search ? 'token' AND\n (\n (_search->>'token' ILIKE 'prev:%')\n OR\n (_search->>'token' ILIKE 'next:%')\n )\n ) THEN\n RETURN NULL;\n END IF;\n prev := (_search->>'token' ILIKE 'prev:%');\n token_id := substr(_search->>'token', 6);\n SELECT to_jsonb(items) INTO token_rec FROM items WHERE id=token_id;\nEND IF;\nRAISE NOTICE 'TOKEN ID: %', token_rec->'id';\n\nCREATE TEMP TABLE sorts (\n _row int GENERATED ALWAYS AS IDENTITY NOT NULL,\n _field text PRIMARY KEY,\n _dir text NOT NULL,\n _val text\n) ON COMMIT DROP;\n\n-- Make sure we only have distinct columns to sort with taking the first one we get\nINSERT INTO sorts (_field, _dir)\n SELECT\n (items_path(value->>'field')).path,\n get_sort_dir(value)\n FROM\n jsonb_array_elements(coalesce(_search->'sort','[{\"field\":\"datetime\",\"direction\":\"desc\"}]'))\nON CONFLICT DO NOTHING\n;\n\n-- Get the first sort direction provided. As the id is a primary key, if there are any\n-- sorts after id they won't do anything, so make sure that id is the last sort item.\nSELECT _dir INTO dir FROM sorts ORDER BY _row ASC LIMIT 1;\nIF EXISTS (SELECT 1 FROM sorts WHERE _field = 'id') THEN\n DELETE FROM sorts WHERE _row > (SELECT _row FROM sorts WHERE _field = 'id');\nELSE\n INSERT INTO sorts (_field, _dir) VALUES ('id', dir);\nEND IF;\n\n-- Add value from looked up item to the sorts table\nUPDATE sorts SET _val=quote_literal(token_rec->>_field);\n\n-- Check if all sorts are the same direction and use row comparison\n-- to filter\nIF (SELECT count(DISTINCT _dir) FROM sorts) = 1 THEN\n SELECT format(\n '(%s) %s (%s)',\n concat_ws(', ', VARIADIC array_agg(quote_ident(_field))),\n CASE WHEN (prev AND dir = 'ASC') OR (NOT prev AND dir = 'DESC') THEN '<' ELSE '>' END,\n concat_ws(', ', VARIADIC array_agg(_val))\n ) INTO output FROM sorts\n WHERE token_rec ? _field\n ;\nELSE\n FOR sort IN SELECT * FROM sorts ORDER BY _row asc LOOP\n RAISE NOTICE 'SORT: %', sort;\n IF sort._row = 1 THEN\n orfilters := orfilters || format('(%s %s %s)',\n quote_ident(sort._field),\n CASE WHEN (prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC') THEN '<' ELSE '>' END,\n sort._val\n );\n ELSE\n orfilters := orfilters || format('(%s AND %s %s %s)',\n array_to_string(andfilters, ' AND '),\n quote_ident(sort._field),\n CASE WHEN (prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC') THEN '<' ELSE '>' END,\n sort._val\n );\n\n END IF;\n andfilters := andfilters || format('%s = %s',\n quote_ident(sort._field),\n sort._val\n );\n END LOOP;\n output := array_to_string(orfilters, ' OR ');\nEND IF;\nDROP TABLE IF EXISTS sorts;\ntoken_where := concat('(',coalesce(output,'true'),')');\nIF trim(token_where) = '' THEN\n token_where := NULL;\nEND IF;\nRAISE NOTICE 'TOKEN_WHERE: |%|',token_where;\nRETURN token_where;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION search_tohash(jsonb) RETURNS jsonb AS $$\n SELECT $1 - '{token,limit,context,includes,excludes}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION search_hash(jsonb) RETURNS text AS $$\n SELECT md5(search_tohash($1)::text);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE TABLE IF NOT EXISTS searches(\n hash text GENERATED ALWAYS AS (search_hash(search)) STORED PRIMARY KEY,\n search jsonb NOT NULL,\n _where text,\n orderby text,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n statslastupdated timestamptz,\n estimated_count bigint,\n total_count bigint\n);\n\nCREATE OR REPLACE FUNCTION search_query(_search jsonb = '{}'::jsonb, updatestats boolean DEFAULT false) RETURNS searches AS $$\nDECLARE\n search searches%ROWTYPE;\nBEGIN\nINSERT INTO searches (search)\n VALUES (search_tohash(_search))\n ON CONFLICT DO NOTHING\n RETURNING * INTO search;\nIF search.hash IS NULL THEN\n SELECT * INTO search FROM searches WHERE hash=search_hash(_search);\nEND IF;\nIF search._where IS NULL THEN\n search._where := cql_to_where(_search);\nEND IF;\nIF search.orderby IS NULL THEN\n search.orderby := sort_sqlorderby(_search);\nEND IF;\n\nIF search.statslastupdated IS NULL OR age(search.statslastupdated) > '1 day'::interval OR (_search ? 'context' AND search.total_count IS NULL) THEN\n updatestats := TRUE;\nEND IF;\n\nIF updatestats THEN\n -- Get Estimated Stats\n RAISE NOTICE 'Getting stats for %', search._where;\n search.estimated_count := estimated_count(search._where);\n RAISE NOTICE 'Estimated Count: %', search.estimated_count;\n\n IF _search ? 'context' OR search.estimated_count < 10000 THEN\n --search.total_count := partition_count(search._where);\n EXECUTE format(\n 'SELECT count(*) FROM items WHERE %s',\n search._where\n ) INTO search.total_count;\n RAISE NOTICE 'Actual Count: %', search.total_count;\n ELSE\n search.total_count := NULL;\n END IF;\n search.statslastupdated := now();\nEND IF;\n\nsearch.lastused := now();\nsearch.usecount := coalesce(search.usecount,0) + 1;\nRAISE NOTICE 'SEARCH: %', search;\nUPDATE searches SET\n _where = search._where,\n orderby = search.orderby,\n lastused = search.lastused,\n usecount = search.usecount,\n statslastupdated = search.statslastupdated,\n estimated_count = search.estimated_count,\n total_count = search.total_count\nWHERE hash = search.hash\n;\nRETURN search;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n\nCREATE OR REPLACE FUNCTION search(_search jsonb = '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n searches searches%ROWTYPE;\n _where text;\n token_where text;\n full_where text;\n orderby text;\n query text;\n token_type text := substr(_search->>'token',1,4);\n _limit int := coalesce((_search->>'limit')::int, 10);\n curs refcursor;\n cntr int := 0;\n iter_record items%ROWTYPE;\n first_record items%ROWTYPE;\n last_record items%ROWTYPE;\n out_records jsonb := '[]'::jsonb;\n prev_query text;\n next text;\n prev_id text;\n has_next boolean := false;\n has_prev boolean := false;\n prev text;\n total_count bigint;\n context jsonb;\n collection jsonb;\n includes text[];\n excludes text[];\n exit_flag boolean := FALSE;\n batches int := 0;\n timer timestamptz := clock_timestamp();\nBEGIN\nsearches := search_query(_search);\n_where := searches._where;\norderby := searches.orderby;\ntotal_count := coalesce(searches.total_count, searches.estimated_count);\n\n\nIF token_type='prev' THEN\n token_where := get_token_filter(_search, null::jsonb);\n orderby := sort_sqlorderby(_search, TRUE);\nEND IF;\nIF token_type='next' THEN\n token_where := get_token_filter(_search, null::jsonb);\nEND IF;\n\nfull_where := concat_ws(' AND ', _where, token_where);\nRAISE NOTICE 'FULL QUERY % %', full_where, clock_timestamp()-timer;\ntimer := clock_timestamp();\n\nFOR query IN SELECT partition_queries(full_where, orderby) LOOP\n timer := clock_timestamp();\n query := format('%s LIMIT %L', query, _limit + 1);\n RAISE NOTICE 'Partition Query: %', query;\n batches := batches + 1;\n curs = create_cursor(query);\n LOOP\n FETCH curs into iter_record;\n EXIT WHEN NOT FOUND;\n cntr := cntr + 1;\n last_record := iter_record;\n IF cntr = 1 THEN\n first_record := last_record;\n END IF;\n IF cntr <= _limit THEN\n out_records := out_records || last_record.content;\n ELSIF cntr > _limit THEN\n has_next := true;\n exit_flag := true;\n EXIT;\n END IF;\n END LOOP;\n RAISE NOTICE 'Query took %', clock_timestamp()-timer;\n timer := clock_timestamp();\n EXIT WHEN exit_flag;\nEND LOOP;\nRAISE NOTICE 'Scanned through % partitions.', batches;\n\n\n-- Flip things around if this was the result of a prev token query\nIF token_type='prev' THEN\n out_records := flip_jsonb_array(out_records);\n first_record := last_record;\nEND IF;\n\n-- If this query has a token, see if there is data before the first record\nIF _search ? 'token' THEN\n prev_query := format(\n 'SELECT 1 FROM items WHERE %s LIMIT 1',\n concat_ws(\n ' AND ',\n _where,\n trim(get_token_filter(_search, to_jsonb(first_record)))\n )\n );\n RAISE NOTICE 'Query to get previous record: % --- %', prev_query, first_record;\n EXECUTE prev_query INTO has_prev;\n IF FOUND and has_prev IS NOT NULL THEN\n RAISE NOTICE 'Query results from prev query: %', has_prev;\n has_prev := TRUE;\n END IF;\nEND IF;\nhas_prev := COALESCE(has_prev, FALSE);\n\nRAISE NOTICE 'token_type: %, has_next: %, has_prev: %', token_type, has_next, has_prev;\nIF has_prev THEN\n prev := out_records->0->>'id';\nEND IF;\nIF has_next OR token_type='prev' THEN\n next := out_records->-1->>'id';\nEND IF;\n\n\n\n-- include/exclude any fields following fields extension\nIF _search ? 'fields' THEN\n IF _search->'fields' ? 'exclude' THEN\n excludes=textarr(_search->'fields'->'exclude');\n END IF;\n IF _search->'fields' ? 'include' THEN\n includes=textarr(_search->'fields'->'include');\n IF array_length(includes, 1)>0 AND NOT 'id' = ANY (includes) THEN\n includes = includes || '{id}';\n END IF;\n END IF;\n SELECT jsonb_agg(filter_jsonb(row, includes, excludes)) INTO out_records FROM jsonb_array_elements(out_records) row;\nEND IF;\n\n\ncontext := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'matched', total_count,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n));\n\ncollection := jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', out_records,\n 'next', next,\n 'prev', prev,\n 'context', context\n);\n\nRETURN collection;\nEND;\n$$ LANGUAGE PLPGSQL SET jit TO off;\n\n\n\n----- END SEARCH\n\nWITH t AS (SELECT items_partition_create_worker(partition, lower(tstzrange), upper(tstzrange)) FROM pgstac_bk.items_partitions) SELECT count(*) FROM t;\n\nINSERT INTO items (id, geometry, collection_id, datetime, end_datetime, properties, content)\nSELECT id, geometry, collection_id, stac_datetime(content), stac_end_datetime(content), properties_idx(content), content\nFROM pgstac_bk.items;\n\nDROP SCHEMA pgstac_bk CASCADE;\n\nCREATE INDEX \"datetime_idx\" ON items (datetime);\nCREATE INDEX \"end_datetime_idx\" ON items (end_datetime);\nCREATE INDEX \"properties_idx\" ON items USING GIN (properties jsonb_path_ops);\nCREATE INDEX \"collection_idx\" ON items (collection_id);\nCREATE INDEX \"geometry_idx\" ON items USING GIST (geometry);\nCREATE UNIQUE INDEX \"items_id_datetime_idx\" ON items (datetime, id);\n\nALTER TABLE items ADD CONSTRAINT items_collections_fk FOREIGN KEY (collection_id) REFERENCES collections(id) ON DELETE CASCADE DEFERRABLE;\n\nINSERT INTO pgstac.migrations (version) VALUES ('0.3.0');\n" }, { "path": "src/pgstac/migrations/pgstac.0.2.9.sql", "content": "CREATE EXTENSION IF NOT EXISTS postgis;\nCREATE SCHEMA IF NOT EXISTS partman;\nCREATE EXTENSION IF NOT EXISTS pg_partman SCHEMA partman;\nCREATE SCHEMA IF NOT EXISTS pgstac;\n\n\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE TABLE migrations (\n version text,\n datetime timestamptz DEFAULT now() NOT NULL\n);\n\n/* converts a jsonb text array to a pg text[] array */\nCREATE OR REPLACE FUNCTION textarr(_js jsonb)\n RETURNS text[] AS $$\n SELECT ARRAY(SELECT jsonb_array_elements_text(_js));\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\n/*\nconverts a jsonb text array to comma delimited list of identifer quoted\nuseful for constructing column lists for selects\n*/\nCREATE OR REPLACE FUNCTION array_idents(_js jsonb)\n RETURNS text AS $$\n SELECT string_agg(quote_ident(v),',') FROM jsonb_array_elements_text(_js) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\n\n/* looks for a geometry in a stac item first from geometry and falling back to bbox */\nCREATE OR REPLACE FUNCTION stac_geom(value jsonb) RETURNS geometry AS $$\nSELECT\n CASE\n WHEN value->>'geometry' IS NOT NULL THEN\n ST_GeomFromGeoJSON(value->>'geometry')\n WHEN value->>'bbox' IS NOT NULL THEN\n ST_MakeEnvelope(\n (value->'bbox'->>0)::float,\n (value->'bbox'->>1)::float,\n (value->'bbox'->>2)::float,\n (value->'bbox'->>3)::float,\n 4326\n )\n ELSE NULL\n END as geometry\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION stac_datetime(value jsonb) RETURNS timestamptz AS $$\nSELECT (value->'properties'->>'datetime')::timestamptz;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION jsonb_paths (IN jdata jsonb, OUT path text[], OUT value jsonb) RETURNS\nSETOF RECORD AS $$\nwith recursive extract_all as\n(\n select\n ARRAY[key]::text[] as path,\n value\n FROM jsonb_each(jdata)\nunion all\n select\n path || coalesce(obj_key, (arr_key- 1)::text),\n coalesce(obj_value, arr_value)\n from extract_all\n left join lateral\n jsonb_each(case jsonb_typeof(value) when 'object' then value end)\n as o(obj_key, obj_value)\n on jsonb_typeof(value) = 'object'\n left join lateral\n jsonb_array_elements(case jsonb_typeof(value) when 'array' then value end)\n with ordinality as a(arr_value, arr_key)\n on jsonb_typeof(value) = 'array'\n where obj_key is not null or arr_key is not null\n)\nselect *\nfrom extract_all;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION jsonb_obj_paths (IN jdata jsonb, OUT path text[], OUT value jsonb) RETURNS\nSETOF RECORD AS $$\nwith recursive extract_all as\n(\n select\n ARRAY[key]::text[] as path,\n value\n FROM jsonb_each(jdata)\nunion all\n select\n path || obj_key,\n obj_value\n from extract_all\n left join lateral\n jsonb_each(case jsonb_typeof(value) when 'object' then value end)\n as o(obj_key, obj_value)\n on jsonb_typeof(value) = 'object'\n where obj_key is not null\n)\nselect *\nfrom extract_all;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION jsonb_val_paths (IN jdata jsonb, OUT path text[], OUT value jsonb) RETURNS\nSETOF RECORD AS $$\nSELECT * FROM jsonb_obj_paths(jdata) WHERE jsonb_typeof(value) not in ('object','array');\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION path_includes(IN path text[], IN includes text[]) RETURNS BOOLEAN AS $$\nWITH t AS (SELECT unnest(includes) i)\nSELECT EXISTS (\n SELECT 1 FROM t WHERE path @> string_to_array(trim(i), '.')\n);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION path_excludes(IN path text[], IN excludes text[]) RETURNS BOOLEAN AS $$\nWITH t AS (SELECT unnest(excludes) e)\nSELECT NOT EXISTS (\n SELECT 1 FROM t WHERE path @> string_to_array(trim(e), '.')\n);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION jsonb_obj_paths_filtered (\n IN jdata jsonb,\n IN includes text[] DEFAULT ARRAY[]::text[],\n IN excludes text[] DEFAULT ARRAY[]::text[],\n OUT path text[],\n OUT value jsonb\n) RETURNS\nSETOF RECORD AS $$\nSELECT path, value\nFROM jsonb_obj_paths(jdata)\nWHERE\n CASE WHEN cardinality(includes) > 0 THEN path_includes(path, includes) ELSE TRUE END\n AND\n path_excludes(path, excludes)\n\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION empty_arr(ANYARRAY) RETURNS BOOLEAN AS $$\nSELECT CASE\n WHEN $1 IS NULL THEN TRUE\n WHEN cardinality($1)<1 THEN TRUE\nELSE FALSE\nEND;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION filter_jsonb(\n IN jdata jsonb,\n IN includes text[] DEFAULT ARRAY[]::text[],\n IN excludes text[] DEFAULT ARRAY[]::text[]\n) RETURNS jsonb AS $$\nDECLARE\nrec RECORD;\noutj jsonb := '{}'::jsonb;\ncreated_paths text[] := '{}'::text[];\nBEGIN\n\nIF empty_arr(includes) AND empty_arr(excludes) THEN\nRAISE NOTICE 'no filter';\n RETURN jdata;\nEND IF;\nFOR rec in\nSELECT * FROM jsonb_obj_paths_filtered(jdata, includes, excludes)\nWHERE jsonb_typeof(value) != 'object'\nLOOP\n IF array_length(rec.path,1)>1 THEN\n FOR i IN 1..(array_length(rec.path,1)-1) LOOP\n IF NOT array_to_string(rec.path[1:i],'.') = ANY (created_paths) THEN\n outj := jsonb_set(outj, rec.path[1:i],'{}', true);\n created_paths := created_paths || array_to_string(rec.path[1:i],'.');\n END IF;\n END LOOP;\n END IF;\n outj := jsonb_set(outj, rec.path, rec.value, true);\n created_paths := created_paths || array_to_string(rec.path,'.');\nEND LOOP;\nRETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION properties_idx(_in jsonb) RETURNS jsonb AS $$\nWITH t AS (\n select array_to_string(path,'.') as path, lower(value::text)::jsonb as lowerval\n FROM jsonb_val_paths(_in)\n WHERE array_to_string(path,'.') not in ('datetime')\n)\nSELECT jsonb_object_agg(path, lowerval) FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE TABLE IF NOT EXISTS collections (\n id VARCHAR GENERATED ALWAYS AS (content->>'id') STORED PRIMARY KEY,\n content JSONB\n);\n\nCREATE OR REPLACE FUNCTION create_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection(data jsonb) RETURNS VOID AS $$\nDECLARE\nout collections%ROWTYPE;\nBEGIN\n UPDATE collections SET content=data WHERE id = data->>'id' RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ON CONFLICT (id) DO\n UPDATE\n SET content=EXCLUDED.content\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION delete_collection(_id text) RETURNS VOID AS $$\nDECLARE\nout collections%ROWTYPE;\nBEGIN\n DELETE FROM collections WHERE id = _id RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION get_collection(id text) RETURNS jsonb AS $$\nSELECT content FROM collections\nWHERE id=$1\n;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION all_collections() RETURNS jsonb AS $$\nSELECT jsonb_agg(content) FROM collections;\n;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE TABLE IF NOT EXISTS items (\n id VARCHAR GENERATED ALWAYS AS (content->>'id') STORED NOT NULL,\n geometry geometry GENERATED ALWAYS AS (stac_geom(content)) STORED NOT NULL,\n properties jsonb GENERATED ALWAYS as (properties_idx(content->'properties')) STORED,\n collection_id text GENERATED ALWAYS AS (content->>'collection') STORED NOT NULL,\n datetime timestamptz GENERATED ALWAYS AS (stac_datetime(content)) STORED NOT NULL,\n content JSONB NOT NULL\n)\nPARTITION BY RANGE (stac_datetime(content))\n;\n\nALTER TABLE items ADD constraint items_collections_fk FOREIGN KEY (collection_id) REFERENCES collections(id) DEFERRABLE;\n\nCREATE TABLE items_template (\n LIKE items\n);\n\nALTER TABLE items_template ADD PRIMARY KEY (id);\n\n\nDELETE from partman.part_config WHERE parent_table = 'pgstac.items';\nSELECT partman.create_parent(\n 'pgstac.items',\n 'datetime',\n 'native',\n 'weekly',\n p_template_table := 'pgstac.items_template',\n p_premake := 4\n);\n\nCREATE OR REPLACE FUNCTION make_partitions(st timestamptz, et timestamptz DEFAULT NULL) RETURNS BOOL AS $$\nWITH t AS (\n SELECT\n generate_series(\n date_trunc('week',st),\n date_trunc('week', coalesce(et, st)),\n '1 week'::interval\n ) w\n),\nw AS (SELECT array_agg(w) as w FROM t)\nSELECT CASE WHEN w IS NULL THEN NULL ELSE partman.create_partition_time('pgstac.items', w, true) END FROM w;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_partition(timestamptz) RETURNS text AS $$\nSELECT to_char($1, '\"items_p\"IYYY\"w\"IW');\n$$ LANGUAGE SQL;\n\nCREATE INDEX \"datetime_id_idx\" ON items (datetime, id);\nCREATE INDEX \"properties_idx\" ON items USING GIN (properties);\nCREATE INDEX \"collection_idx\" ON items (collection_id);\nCREATE INDEX \"geometry_idx\" ON items USING GIST (geometry);\n\n\nCREATE TYPE item AS (\n id text,\n geometry geometry,\n properties JSONB,\n collection_id text,\n datetime timestamptz\n);\n\n\nCREATE OR REPLACE FUNCTION get_item(_id text) RETURNS jsonb AS $$\n SELECT content FROM items WHERE id=_id;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION delete_item(_id text) RETURNS VOID AS $$\nDECLARE\nout items%ROWTYPE;\nBEGIN\n DELETE FROM items WHERE id = _id RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION create_item(data jsonb) RETURNS VOID AS $$\n SELECT make_partitions(stac_datetime(data));\n INSERT INTO items (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION update_item(data jsonb) RETURNS VOID AS $$\nDECLARE\nout items%ROWTYPE;\nBEGIN\n UPDATE items SET content=data WHERE id = data->>'id' RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_item(data jsonb) RETURNS VOID AS $$\nDECLARE\npartition text;\nq text;\nnewcontent jsonb;\nBEGIN\n PERFORM make_partitions(stac_datetime(data));\n partition := get_partition(stac_datetime(data));\n q := format($q$\n INSERT INTO %I (content) VALUES ($1)\n ON CONFLICT (id) DO\n UPDATE SET content = EXCLUDED.content\n WHERE %I.content IS DISTINCT FROM EXCLUDED.content RETURNING content;\n $q$, partition, partition);\n EXECUTE q INTO newcontent USING (data);\n RAISE NOTICE 'newcontent: %', newcontent;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION analyze_empty_partitions() RETURNS VOID AS $$\nDECLARE\np text;\nBEGIN\nFOR p IN SELECT partition FROM all_items_partitions WHERE est_cnt = 0 LOOP\n RAISE NOTICE 'Analyzing %', p;\n EXECUTE format('ANALYZE %I;', p);\nEND LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION backfill_partitions()\nRETURNS VOID AS $$\nDECLARE\nBEGIN\n IF EXISTS (SELECT 1 FROM items_default LIMIT 1) THEN\n RAISE NOTICE 'Creating new partitions and moving data from default';\n CREATE TEMP TABLE items_default_tmp ON COMMIT DROP AS SELECT datetime, content FROM items_default;\n TRUNCATE items_default;\n PERFORM make_partitions(min(datetime), max(datetime)) FROM items_default_tmp;\n INSERT INTO items (content) SELECT content FROM items_default_tmp;\n END IF;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION items_trigger_stmt_func()\nRETURNS TRIGGER AS $$\nDECLARE\nBEGIN\n PERFORM analyze_empty_partitions();\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nDROP TRIGGER IF EXISTS items_stmt_trigger ON items;\nCREATE TRIGGER items_stmt_trigger\nAFTER INSERT OR UPDATE OR DELETE ON items\nFOR EACH STATEMENT EXECUTE PROCEDURE items_trigger_stmt_func();\n\n\n/*\nView to get a table of available items partitions\nwith date ranges\n*/\n--DROP VIEW IF EXISTS all_items_partitions CASCADE;\nCREATE OR REPLACE VIEW all_items_partitions AS\nWITH base AS\n(SELECT\n c.oid::pg_catalog.regclass::text as partition,\n pg_catalog.pg_get_expr(c.relpartbound, c.oid) as _constraint,\n regexp_matches(\n pg_catalog.pg_get_expr(c.relpartbound, c.oid),\n E'\\\\(''\\([0-9 :+-]*\\)''\\\\).*\\\\(''\\([0-9 :+-]*\\)''\\\\)'\n ) as t,\n reltuples::bigint as est_cnt\nFROM pg_catalog.pg_class c, pg_catalog.pg_inherits i\nWHERE c.oid = i.inhrelid AND i.inhparent = 'items'::regclass)\nSELECT partition, tstzrange(\n t[1]::timestamptz,\n t[2]::timestamptz\n), est_cnt\nFROM base\nORDER BY 2 desc;\n\n--DROP VIEW IF EXISTS items_partitions;\nCREATE OR REPLACE VIEW items_partitions AS\nSELECT * FROM all_items_partitions WHERE est_cnt>0;\n\nCREATE OR REPLACE FUNCTION collection_bbox(id text) RETURNS jsonb AS $$\nSELECT (replace(replace(replace(st_extent(geometry)::text,'BOX(','[['),')',']]'),' ',','))::jsonb\nFROM items WHERE collection_id=$1;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION collection_temporal_extent(id text) RETURNS jsonb AS $$\nSELECT to_jsonb(array[array[min(datetime)::text, max(datetime)::text]])\nFROM items WHERE collection_id=$1;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection_extents() RETURNS VOID AS $$\nUPDATE collections SET\n content = content ||\n jsonb_build_object(\n 'extent', jsonb_build_object(\n 'spatial', jsonb_build_object(\n 'bbox', collection_bbox(collections.id)\n ),\n 'temporal', jsonb_build_object(\n 'interval', collection_temporal_extent(collections.id)\n )\n )\n )\n;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\nSET SEARCH_PATH TO pgstac, public;\n\n\nCREATE OR REPLACE FUNCTION items_by_partition(\n IN _where text DEFAULT 'TRUE',\n IN _dtrange tstzrange DEFAULT tstzrange('-infinity','infinity'),\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN _limit int DEFAULT 10\n) RETURNS SETOF items AS $$\nDECLARE\npartition_query text;\nmain_query text;\nbatchcount int;\ncounter int := 0;\np record;\nBEGIN\nIF _orderby ILIKE 'datetime d%' THEN\n partition_query := format($q$\n SELECT partition\n FROM items_partitions\n WHERE tstzrange && $1\n ORDER BY tstzrange DESC;\n $q$);\nELSIF _orderby ILIKE 'datetime a%' THEN\n partition_query := format($q$\n SELECT partition\n FROM items_partitions\n WHERE tstzrange && $1\n ORDER BY tstzrange ASC\n ;\n $q$);\nELSE\n partition_query := format($q$\n SELECT 'items' as partition WHERE $1 IS NOT NULL;\n $q$);\nEND IF;\nRAISE NOTICE 'Partition Query: %', partition_query;\nFOR p IN\n EXECUTE partition_query USING (_dtrange)\nLOOP\n IF lower(_dtrange)::timestamptz > '-infinity' THEN\n _where := concat(_where,format(' AND datetime >= %L',lower(_dtrange)::timestamptz::text));\n END IF;\n IF upper(_dtrange)::timestamptz < 'infinity' THEN\n _where := concat(_where,format(' AND datetime <= %L',upper(_dtrange)::timestamptz::text));\n END IF;\n\n main_query := format($q$\n SELECT * FROM %I\n WHERE %s\n ORDER BY %s\n LIMIT %s - $1\n $q$, p.partition::text, _where, _orderby, _limit\n );\n RAISE NOTICE 'Partition Query %', main_query;\n RAISE NOTICE '%', counter;\n RETURN QUERY EXECUTE main_query USING counter;\n\n GET DIAGNOSTICS batchcount = ROW_COUNT;\n counter := counter + batchcount;\n RAISE NOTICE 'FOUND %', batchcount;\n IF counter >= _limit THEN\n EXIT;\n END IF;\n RAISE NOTICE 'ADDED % FOR A TOTAL OF %', batchcount, counter;\nEND LOOP;\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION split_stac_path(IN path text, OUT col text, OUT dotpath text, OUT jspath text, OUT jspathtext text) AS $$\nWITH col AS (\n SELECT\n CASE WHEN\n split_part(path, '.', 1) IN ('id', 'stac_version', 'stac_extensions','geometry','properties','assets','collection_id','datetime','links', 'extra_fields') THEN split_part(path, '.', 1)\n ELSE 'properties'\n END AS col\n),\ndp AS (\n SELECT\n col, ltrim(replace(path, col , ''),'.') as dotpath\n FROM col\n),\npaths AS (\nSELECT\n col, dotpath,\n regexp_split_to_table(dotpath,E'\\\\.') as path FROM dp\n) SELECT\n col,\n btrim(concat(col,'.',dotpath),'.'),\n CASE WHEN btrim(concat(col,'.',dotpath),'.') != col THEN concat(col,'->',string_agg(concat('''',path,''''),'->')) ELSE col END,\n regexp_replace(\n CASE WHEN btrim(concat(col,'.',dotpath),'.') != col THEN concat(col,'->',string_agg(concat('''',path,''''),'->')) ELSE col END,\n E'>([^>]*)$','>>\\1'\n )\nFROM paths group by col, dotpath;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n/* Functions for searching items */\nCREATE OR REPLACE FUNCTION sort_base(\n IN _sort jsonb DEFAULT '[{\"field\":\"datetime\",\"direction\":\"desc\"}]',\n OUT key text,\n OUT col text,\n OUT dir text,\n OUT rdir text,\n OUT sort text,\n OUT rsort text\n) RETURNS SETOF RECORD AS $$\nWITH sorts AS (\n SELECT\n value->>'field' as key,\n (split_stac_path(value->>'field')).jspathtext as col,\n coalesce(upper(value->>'direction'),'ASC') as dir\n FROM jsonb_array_elements('[]'::jsonb || coalesce(_sort,'[{\"field\":\"datetime\",\"direction\":\"desc\"}]') )\n)\nSELECT\n key,\n col,\n dir,\n CASE dir WHEN 'DESC' THEN 'ASC' ELSE 'ASC' END as rdir,\n concat(col, ' ', dir, ' NULLS LAST ') AS sort,\n concat(col,' ', CASE dir WHEN 'DESC' THEN 'ASC' ELSE 'ASC' END, ' NULLS LAST ') AS rsort\nFROM sorts\nUNION ALL\nSELECT 'id', 'id', 'DESC', 'ASC', 'id DESC', 'id ASC'\n;\n$$ LANGUAGE SQL;\n\n\nCREATE OR REPLACE FUNCTION sort(_sort jsonb) RETURNS text AS $$\nSELECT string_agg(sort,', ') FROM sort_base(_sort);\n$$ LANGUAGE SQL PARALLEL SAFE SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION rsort(_sort jsonb) RETURNS text AS $$\nSELECT string_agg(rsort,', ') FROM sort_base(_sort);\n$$ LANGUAGE SQL PARALLEL SAFE SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION bbox_geom(_bbox jsonb) RETURNS geometry AS $$\nSELECT CASE jsonb_array_length(_bbox)\n WHEN 4 THEN\n ST_SetSRID(ST_MakeEnvelope(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float,\n (_bbox->>3)::float\n ),4326)\n WHEN 6 THEN\n ST_SetSRID(ST_3DMakeBox(\n ST_MakePoint(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float\n ),\n ST_MakePoint(\n (_bbox->>3)::float,\n (_bbox->>4)::float,\n (_bbox->>5)::float\n )\n ),4326)\n ELSE null END;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION in_array_q(col text, arr jsonb) RETURNS text AS $$\nSELECT CASE jsonb_typeof(arr) WHEN 'array' THEN format('%I = ANY(textarr(%L))', col, arr) ELSE format('%I = %L', col, arr) END;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION count_by_delim(text, text) RETURNS int AS $$\nSELECT count(*) FROM regexp_split_to_table($1,$2);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION stac_query_op(att text, _op text, val jsonb) RETURNS text AS $$\nDECLARE\nret text := '';\nop text;\njp text;\natt_parts RECORD;\nval_str text;\nprop_path text;\nBEGIN\nval_str := lower(jsonb_build_object('a',val)->>'a');\nRAISE NOTICE 'val_str %', val_str;\n\natt_parts := split_stac_path(att);\nprop_path := replace(att_parts.dotpath, 'properties.', '');\n\nop := CASE _op\n WHEN 'eq' THEN '='\n WHEN 'gte' THEN '>='\n WHEN 'gt' THEN '>'\n WHEN 'lte' THEN '<='\n WHEN 'lt' THEN '<'\n WHEN 'ne' THEN '!='\n WHEN 'neq' THEN '!='\n WHEN 'startsWith' THEN 'LIKE'\n WHEN 'endsWith' THEN 'LIKE'\n WHEN 'contains' THEN 'LIKE'\n ELSE _op\nEND;\n\nval_str := CASE _op\n WHEN 'startsWith' THEN concat(val_str, '%')\n WHEN 'endsWith' THEN concat('%', val_str)\n WHEN 'contains' THEN concat('%',val_str,'%')\n ELSE val_str\nEND;\n\n\nRAISE NOTICE 'att_parts: % %', att_parts, count_by_delim(att_parts.dotpath,'\\.');\nIF\n op = '='\n AND att_parts.col = 'properties'\n --AND count_by_delim(att_parts.dotpath,'\\.') = 2\nTHEN\n -- use jsonpath query to leverage index for eqaulity tests on single level deep properties\n jp := btrim(format($jp$ $.%I[*] ? ( @ == %s ) $jp$, replace(att_parts.dotpath, 'properties.',''), lower(val::text)::jsonb));\n raise notice 'jp: %', jp;\n ret := format($q$ properties @? %L $q$, jp);\nELSIF jsonb_typeof(val) = 'number' THEN\n ret := format('properties ? %L AND (%s)::numeric %s %s', prop_path, att_parts.jspathtext, op, val);\nELSE\n ret := format('properties ? %L AND %s %s %L', prop_path ,att_parts.jspathtext, op, val_str);\nEND IF;\nRAISE NOTICE 'Op Query: %', ret;\n\nreturn ret;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION stac_query(_query jsonb) RETURNS TEXT[] AS $$\nDECLARE\nqa text[];\natt text;\nops jsonb;\nop text;\nval jsonb;\nBEGIN\nFOR att, ops IN SELECT key, value FROM jsonb_each(_query)\nLOOP\n FOR op, val IN SELECT key, value FROM jsonb_each(ops)\n LOOP\n qa := array_append(qa, stac_query_op(att,op, val));\n RAISE NOTICE '% % %', att, op, val;\n END LOOP;\nEND LOOP;\nRETURN qa;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION filter_by_order(item_id text, _sort jsonb, _type text) RETURNS text AS $$\nDECLARE\nitem item;\nBEGIN\nSELECT * INTO item FROM items WHERE id=item_id;\nRETURN filter_by_order(item, _sort, _type);\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n-- Used to create filters used for paging using the items id from the token\nCREATE OR REPLACE FUNCTION filter_by_order(_item item, _sort jsonb, _type text) RETURNS text AS $$\nDECLARE\nsorts RECORD;\nfilts text[];\nitemval text;\nop text;\nidop text;\nret text;\neq_flag text;\n_item_j jsonb := to_jsonb(_item);\nBEGIN\nFOR sorts IN SELECT * FROM sort_base(_sort) LOOP\n IF sorts.col = 'datetime' THEN\n CONTINUE;\n END IF;\n IF sorts.col='id' AND _type IN ('prev','next') THEN\n eq_flag := '';\n ELSE\n eq_flag := '=';\n END IF;\n\n op := concat(\n CASE\n WHEN _type in ('prev','first') AND sorts.dir = 'ASC' THEN '<'\n WHEN _type in ('last','next') AND sorts.dir = 'ASC' THEN '>'\n WHEN _type in ('prev','first') AND sorts.dir = 'DESC' THEN '>'\n WHEN _type in ('last','next') AND sorts.dir = 'DESC' THEN '<'\n END,\n eq_flag\n );\n\n IF _item_j ? sorts.col THEN\n filts = array_append(filts, format('%s %s %L', sorts.col, op, _item_j->>sorts.col));\n END IF;\nEND LOOP;\nret := coalesce(array_to_string(filts,' AND '), 'TRUE');\nRAISE NOTICE 'Order Filter %', ret;\nRETURN ret;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION search_dtrange(IN _indate jsonb, OUT _tstzrange tstzrange) AS\n$$\nWITH t AS (\n SELECT CASE\n WHEN jsonb_typeof(_indate) = 'array' THEN\n textarr(_indate)\n ELSE\n regexp_split_to_array(\n btrim(_indate::text,'\"'),\n '/'\n )\n END AS arr\n)\n, t1 AS (\n SELECT\n CASE\n WHEN array_upper(arr,1) = 1 OR arr[1] = '..' OR arr[1] IS NULL THEN '-infinity'::timestamptz\n ELSE arr[1]::timestamptz\n END AS st,\n CASE\n WHEN array_upper(arr,1) = 1 THEN arr[1]::timestamptz\n WHEN arr[2] = '..' OR arr[2] IS NULL THEN 'infinity'::timestamptz\n ELSE arr[2]::timestamptz\n END AS et\n FROM t\n)\nSELECT\n tstzrange(st,et)\nFROM t1;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION flip_jsonb_array(j jsonb) RETURNS jsonb AS $$\nWITH t AS (\n SELECT i, row_number() over () as r FROM jsonb_array_elements(j) i\n), o AS (\n SELECT i FROM t ORDER BY r DESC\n)\nSELECT jsonb_agg(i) from o\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION search(_search jsonb = '{}'::jsonb) RETURNS SETOF jsonb AS $$\nDECLARE\nqstart timestamptz := clock_timestamp();\n_sort text := '';\n_rsort text := '';\n_limit int := 10;\n_geom geometry;\nqa text[];\npq text[];\nquery text;\npq_prop record;\npq_op record;\nprev_id text := NULL;\nnext_id text := NULL;\nwhereq text := 'TRUE';\nlinks jsonb := '[]'::jsonb;\ntoken text;\ntok_val text;\ntok_q text := 'TRUE';\ntok_sort text;\nfirst_id text;\nfirst_dt timestamptz;\nlast_id text;\nsort text;\nrsort text;\ndt text[];\ndqa text[];\ndq text;\nmq_where text;\nstartdt timestamptz;\nenddt timestamptz;\nitem items%ROWTYPE;\ncounter int := 0;\nbatchcount int;\nmonth timestamptz;\nm record;\n_dtrange tstzrange := tstzrange('-infinity','infinity');\n_dtsort text;\n_token_dtrange tstzrange := tstzrange('-infinity','infinity');\n_token_record items%ROWTYPE;\nis_prev boolean := false;\nincludes text[];\nexcludes text[];\nBEGIN\n-- Create table from sort query of items to sort\nCREATE TEMP TABLE pgstac_tmp_sorts ON COMMIT DROP AS SELECT * FROM sort_base(_search->'sortby');\n\n-- Get the datetime sort direction, necessary for efficient cycling through partitions\nSELECT INTO _dtsort dir FROM pgstac_tmp_sorts WHERE key='datetime';\nRAISE NOTICE '_dtsort: %',_dtsort;\n\nSELECT INTO _sort string_agg(s.sort,', ') FROM pgstac_tmp_sorts s;\nSELECT INTO _rsort string_agg(s.rsort,', ') FROM pgstac_tmp_sorts s;\ntok_sort := _sort;\n\n\n-- Get datetime from query as a tstzrange\nIF _search ? 'datetime' THEN\n _dtrange := search_dtrange(_search->'datetime');\n _token_dtrange := _dtrange;\nEND IF;\n\n-- Get the paging token\nIF _search ? 'token' THEN\n token := _search->>'token';\n tok_val := substr(token,6);\n IF starts_with(token, 'prev:') THEN\n is_prev := true;\n END IF;\n SELECT INTO _token_record * FROM items WHERE id=tok_val;\n IF\n (is_prev AND _dtsort = 'DESC')\n OR\n (not is_prev AND _dtsort = 'ASC')\n THEN\n _token_dtrange := _dtrange * tstzrange(_token_record.datetime, 'infinity');\n ELSIF\n _dtsort IS NOT NULL\n THEN\n _token_dtrange := _dtrange * tstzrange('-infinity',_token_record.datetime);\n END IF;\n IF is_prev THEN\n tok_q := filter_by_order(tok_val, _search->'sortby', 'first');\n _sort := _rsort;\n ELSIF starts_with(token, 'next:') THEN\n tok_q := filter_by_order(tok_val, _search->'sortby', 'last');\n END IF;\nEND IF;\nRAISE NOTICE 'timing: %', age(clock_timestamp(), qstart);\nRAISE NOTICE 'tok_q: % _token_dtrange: %', tok_q, _token_dtrange;\n\nIF _search ? 'ids' THEN\n RAISE NOTICE 'searching solely based on ids... %',_search;\n qa := array_append(qa, in_array_q('id', _search->'ids'));\nELSE\n IF _search ? 'intersects' THEN\n _geom := ST_SetSRID(ST_GeomFromGeoJSON(_search->>'intersects'), 4326);\n ELSIF _search ? 'bbox' THEN\n _geom := bbox_geom(_search->'bbox');\n END IF;\n\n IF _geom IS NOT NULL THEN\n qa := array_append(qa, format('st_intersects(geometry, %L::geometry)',_geom));\n END IF;\n\n IF _search ? 'collections' THEN\n qa := array_append(qa, in_array_q('collection_id', _search->'collections'));\n END IF;\n\n IF _search ? 'query' THEN\n qa := array_cat(qa,\n stac_query(_search->'query')\n );\n END IF;\nEND IF;\n\nIF _search ? 'limit' THEN\n _limit := (_search->>'limit')::int;\nEND IF;\n\nIF _search ? 'fields' THEN\n IF _search->'fields' ? 'exclude' THEN\n excludes=textarr(_search->'fields'->'exclude');\n END IF;\n IF _search->'fields' ? 'include' THEN\n includes=textarr(_search->'fields'->'include');\n IF array_length(includes, 1)>0 AND NOT 'id' = ANY (includes) THEN\n includes = includes || '{id}';\n END IF;\n END IF;\n RAISE NOTICE 'Includes: %, Excludes: %', includes, excludes;\nEND IF;\n\nwhereq := COALESCE(array_to_string(qa,' AND '),' TRUE ');\ndq := COALESCE(array_to_string(dqa,' AND '),' TRUE ');\nRAISE NOTICE 'timing before temp table: %', age(clock_timestamp(), qstart);\n\nCREATE TEMP TABLE results_page ON COMMIT DROP AS\nSELECT * FROM items_by_partition(\n concat(whereq, ' AND ', tok_q),\n _token_dtrange,\n _sort,\n _limit + 1\n);\nRAISE NOTICE 'timing after temp table: %', age(clock_timestamp(), qstart);\n\nRAISE NOTICE 'timing before min/max: %', age(clock_timestamp(), qstart);\n\nIF is_prev THEN\n SELECT INTO last_id, first_id, counter\n first_value(id) OVER (),\n last_value(id) OVER (),\n count(*) OVER ()\n FROM results_page;\nELSE\n SELECT INTO first_id, last_id, counter\n first_value(id) OVER (),\n last_value(id) OVER (),\n count(*) OVER ()\n FROM results_page;\nEND IF;\nRAISE NOTICE 'firstid: %, lastid %', first_id, last_id;\nRAISE NOTICE 'timing after min/max: %', age(clock_timestamp(), qstart);\n\n\n\n\nIF counter > _limit THEN\n next_id := last_id;\n RAISE NOTICE 'next_id: %', next_id;\nELSE\n RAISE NOTICE 'No more next';\nEND IF;\n\nIF tok_q = 'TRUE' THEN\n RAISE NOTICE 'Not a paging query, no previous item';\nELSE\n RAISE NOTICE 'Getting previous item id';\n RAISE NOTICE 'timing: %', age(clock_timestamp(), qstart);\n SELECT INTO _token_record * FROM items WHERE id=first_id;\n IF\n _dtsort = 'DESC'\n THEN\n _token_dtrange := _dtrange * tstzrange(_token_record.datetime, 'infinity');\n ELSE\n _token_dtrange := _dtrange * tstzrange('-infinity',_token_record.datetime);\n END IF;\n RAISE NOTICE '% %', _token_dtrange, _dtrange;\n SELECT id INTO prev_id FROM items_by_partition(\n concat(whereq, ' AND ', filter_by_order(first_id, _search->'sortby', 'prev')),\n _token_dtrange,\n _rsort,\n 1\n );\n RAISE NOTICE 'timing: %', age(clock_timestamp(), qstart);\n\n RAISE NOTICE 'prev_id: %', prev_id;\nEND IF;\n\n\nRETURN QUERY\nWITH features AS (\n SELECT filter_jsonb(content, includes, excludes) as content\n FROM results_page LIMIT _limit\n),\nj AS (SELECT jsonb_agg(content) as feature_arr FROM features)\nSELECT jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce (\n CASE WHEN is_prev THEN flip_jsonb_array(feature_arr) ELSE feature_arr END\n ,'[]'::jsonb),\n 'links', links,\n 'timeStamp', now(),\n 'next', next_id,\n 'prev', prev_id\n)\nFROM j\n;\n\n\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\nINSERT INTO pgstac.migrations (version) VALUES ('0.2.9');\n" }, { "path": "src/pgstac/migrations/pgstac.0.3.0-0.3.1.sql", "content": "SET SEARCH_PATH to pgstac, public;\nset check_function_bodies = off;\n\nCREATE OR REPLACE FUNCTION pgstac.sort_sqlorderby(_search jsonb DEFAULT NULL::jsonb, reverse boolean DEFAULT false)\n RETURNS text\n LANGUAGE sql\nAS $function$\nWITH sorts AS (\n SELECT\n (items_path(value->>'field')).path as key,\n parse_sort_dir(value->>'direction', reverse) as dir\n FROM jsonb_array_elements(\n '[]'::jsonb\n ||\n coalesce(_search->'sortby','[{\"field\":\"datetime\", \"direction\":\"desc\"}]')\n ||\n '[{\"field\":\"id\",\"direction\":\"desc\"}]'::jsonb\n )\n)\nSELECT array_to_string(\n array_agg(concat(key, ' ', dir)),\n ', '\n) FROM sorts;\n$function$\n;\n\n\n\nINSERT INTO migrations (version) VALUES ('0.3.1');\n" }, { "path": "src/pgstac/migrations/pgstac.0.3.0.sql", "content": "CREATE EXTENSION IF NOT EXISTS postgis;\nCREATE SCHEMA IF NOT EXISTS pgstac;\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE TABLE migrations (\n version text,\n datetime timestamptz DEFAULT now() NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION notice(VARIADIC text[]) RETURNS boolean AS $$\nDECLARE\ndebug boolean := current_setting('pgstac.debug', true);\nBEGIN\n IF debug THEN\n RAISE NOTICE 'NOTICE FROM FUNC: % >>>>> %', concat_ws(' | ', $1), clock_timestamp();\n RETURN TRUE;\n END IF;\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION empty_arr(ANYARRAY) RETURNS BOOLEAN AS $$\nSELECT CASE\n WHEN $1 IS NULL THEN TRUE\n WHEN cardinality($1)<1 THEN TRUE\nELSE FALSE\nEND;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_map_ident(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_ident(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_map_literal(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_literal(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION estimated_count(_where text) RETURNS bigint AS $$\nDECLARE\nrec record;\nrows bigint;\nBEGIN\n FOR rec in EXECUTE format(\n $q$\n EXPLAIN SELECT 1 FROM items WHERE %s\n $q$,\n _where)\n LOOP\n rows := substring(rec.\"QUERY PLAN\" FROM ' rows=([[:digit:]]+)');\n EXIT WHEN rows IS NOT NULL;\n END LOOP;\n\n RETURN rows;\nEND;\n$$ LANGUAGE PLPGSQL;\n/* converts a jsonb text array to a pg text[] array */\nCREATE OR REPLACE FUNCTION textarr(_js jsonb)\n RETURNS text[] AS $$\n SELECT\n CASE jsonb_typeof(_js)\n WHEN 'array' THEN ARRAY(SELECT jsonb_array_elements_text(_js))\n ELSE ARRAY[_js->>0]\n END\n;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION jsonb_paths (IN jdata jsonb, OUT path text[], OUT value jsonb) RETURNS\nSETOF RECORD AS $$\nwith recursive extract_all as\n(\n select\n ARRAY[key]::text[] as path,\n value\n FROM jsonb_each(jdata)\nunion all\n select\n path || coalesce(obj_key, (arr_key- 1)::text),\n coalesce(obj_value, arr_value)\n from extract_all\n left join lateral\n jsonb_each(case jsonb_typeof(value) when 'object' then value end)\n as o(obj_key, obj_value)\n on jsonb_typeof(value) = 'object'\n left join lateral\n jsonb_array_elements(case jsonb_typeof(value) when 'array' then value end)\n with ordinality as a(arr_value, arr_key)\n on jsonb_typeof(value) = 'array'\n where obj_key is not null or arr_key is not null\n)\nselect *\nfrom extract_all;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION jsonb_obj_paths (IN jdata jsonb, OUT path text[], OUT value jsonb) RETURNS\nSETOF RECORD AS $$\nwith recursive extract_all as\n(\n select\n ARRAY[key]::text[] as path,\n value\n FROM jsonb_each(jdata)\nunion all\n select\n path || obj_key,\n obj_value\n from extract_all\n left join lateral\n jsonb_each(case jsonb_typeof(value) when 'object' then value end)\n as o(obj_key, obj_value)\n on jsonb_typeof(value) = 'object'\n where obj_key is not null\n)\nselect *\nfrom extract_all;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION jsonb_val_paths (IN jdata jsonb, OUT path text[], OUT value jsonb) RETURNS\nSETOF RECORD AS $$\nSELECT * FROM jsonb_obj_paths(jdata) WHERE jsonb_typeof(value) not in ('object','array');\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION path_includes(IN path text[], IN includes text[]) RETURNS BOOLEAN AS $$\nWITH t AS (SELECT unnest(includes) i)\nSELECT EXISTS (\n SELECT 1 FROM t WHERE path @> string_to_array(trim(i), '.')\n);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION path_excludes(IN path text[], IN excludes text[]) RETURNS BOOLEAN AS $$\nWITH t AS (SELECT unnest(excludes) e)\nSELECT NOT EXISTS (\n SELECT 1 FROM t WHERE path @> string_to_array(trim(e), '.')\n);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION jsonb_obj_paths_filtered (\n IN jdata jsonb,\n IN includes text[] DEFAULT ARRAY[]::text[],\n IN excludes text[] DEFAULT ARRAY[]::text[],\n OUT path text[],\n OUT value jsonb\n) RETURNS\nSETOF RECORD AS $$\nSELECT path, value\nFROM jsonb_obj_paths(jdata)\nWHERE\n CASE WHEN cardinality(includes) > 0 THEN path_includes(path, includes) ELSE TRUE END\n AND\n path_excludes(path, excludes)\n\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION filter_jsonb(\n IN jdata jsonb,\n IN includes text[] DEFAULT ARRAY[]::text[],\n IN excludes text[] DEFAULT ARRAY[]::text[]\n) RETURNS jsonb AS $$\nDECLARE\nrec RECORD;\noutj jsonb := '{}'::jsonb;\ncreated_paths text[] := '{}'::text[];\nBEGIN\n\nIF empty_arr(includes) AND empty_arr(excludes) THEN\nRAISE NOTICE 'no filter';\n RETURN jdata;\nEND IF;\nFOR rec in\nSELECT * FROM jsonb_obj_paths_filtered(jdata, includes, excludes)\nWHERE jsonb_typeof(value) != 'object'\nLOOP\n IF array_length(rec.path,1)>1 THEN\n FOR i IN 1..(array_length(rec.path,1)-1) LOOP\n IF NOT array_to_string(rec.path[1:i],'.') = ANY (created_paths) THEN\n outj := jsonb_set(outj, rec.path[1:i],'{}', true);\n created_paths := created_paths || array_to_string(rec.path[1:i],'.');\n END IF;\n END LOOP;\n END IF;\n outj := jsonb_set(outj, rec.path, rec.value, true);\n created_paths := created_paths || array_to_string(rec.path,'.');\nEND LOOP;\nRETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION flip_jsonb_array(j jsonb) RETURNS jsonb AS $$\nSELECT jsonb_agg(value) FROM (SELECT value FROM jsonb_array_elements(j) WITH ORDINALITY ORDER BY ordinality DESC) as t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n/* Functions to create an iterable of cursors over partitions. */\nCREATE OR REPLACE FUNCTION create_cursor(q text) RETURNS refcursor AS $$\nDECLARE\n curs refcursor;\nBEGIN\n OPEN curs FOR EXECUTE q;\n RETURN curs;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION partition_queries(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC'\n) RETURNS SETOF text AS $$\nDECLARE\n partition_query text;\n query text;\n p record;\n cursors refcursor;\nBEGIN\nIF _orderby ILIKE 'datetime d%' THEN\n partition_query := format($q$\n SELECT partition, tstzrange\n FROM items_partitions\n ORDER BY tstzrange DESC;\n $q$);\nELSIF _orderby ILIKE 'datetime a%' THEN\n partition_query := format($q$\n SELECT partition, tstzrange\n FROM items_partitions\n ORDER BY tstzrange ASC\n ;\n $q$);\nELSE\n query := format($q$\n SELECT * FROM items\n WHERE %s\n ORDER BY %s\n $q$, _where, _orderby\n );\n\n RETURN NEXT query;\n RETURN;\nEND IF;\nFOR p IN\n EXECUTE partition_query\nLOOP\n query := format($q$\n SELECT * FROM items\n WHERE datetime >= %L AND datetime < %L AND %s\n ORDER BY %s\n $q$, lower(p.tstzrange), upper(p.tstzrange), _where, _orderby\n );\n RETURN NEXT query;\nEND LOOP;\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION partition_cursor(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC'\n) RETURNS SETOF refcursor AS $$\nDECLARE\n partition_query text;\n query text;\n p record;\n cursors refcursor;\nBEGIN\nFOR query IN SELECT * FROM partion_queries(_where, _orderby) LOOP\n RETURN NEXT create_cursor(query);\nEND LOOP;\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION partition_count(\n IN _where text DEFAULT 'TRUE'\n) RETURNS bigint AS $$\nDECLARE\n partition_query text;\n query text;\n p record;\n subtotal bigint;\n total bigint := 0;\nBEGIN\npartition_query := format($q$\n SELECT partition, tstzrange\n FROM items_partitions\n ORDER BY tstzrange DESC;\n$q$);\nRAISE NOTICE 'Partition Query: %', partition_query;\nFOR p IN\n EXECUTE partition_query\nLOOP\n query := format($q$\n SELECT count(*) FROM items\n WHERE datetime BETWEEN %L AND %L AND %s\n $q$, lower(p.tstzrange), upper(p.tstzrange), _where\n );\n RAISE NOTICE 'Query %', query;\n RAISE NOTICE 'Partition %, Count %, Total %',p.partition, subtotal, total;\n EXECUTE query INTO subtotal;\n total := subtotal + total;\nEND LOOP;\nRETURN total;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n/* looks for a geometry in a stac item first from geometry and falling back to bbox */\nCREATE OR REPLACE FUNCTION stac_geom(value jsonb) RETURNS geometry AS $$\nSELECT\n CASE\n WHEN value->>'geometry' IS NOT NULL THEN\n ST_GeomFromGeoJSON(value->>'geometry')\n WHEN value->>'bbox' IS NOT NULL THEN\n ST_MakeEnvelope(\n (value->'bbox'->>0)::float,\n (value->'bbox'->>1)::float,\n (value->'bbox'->>2)::float,\n (value->'bbox'->>3)::float,\n 4326\n )\n ELSE NULL\n END as geometry\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION stac_datetime(value jsonb) RETURNS timestamptz AS $$\nSELECT COALESCE(\n (value->'properties'->>'datetime')::timestamptz,\n (value->'properties'->>'start_datetime')::timestamptz\n);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_end_datetime(value jsonb) RETURNS timestamptz AS $$\nSELECT COALESCE(\n (value->'properties'->>'datetime')::timestamptz,\n (value->'properties'->>'end_datetime')::timestamptz\n);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\n\nCREATE OR REPLACE FUNCTION stac_daterange(value jsonb) RETURNS tstzrange AS $$\nSELECT tstzrange(stac_datetime(value),stac_end_datetime(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE TABLE IF NOT EXISTS collections (\n id VARCHAR GENERATED ALWAYS AS (content->>'id') STORED PRIMARY KEY,\n content JSONB\n);\n\nCREATE OR REPLACE FUNCTION create_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection(data jsonb) RETURNS VOID AS $$\nDECLARE\nout collections%ROWTYPE;\nBEGIN\n UPDATE collections SET content=data WHERE id = data->>'id' RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ON CONFLICT (id) DO\n UPDATE\n SET content=EXCLUDED.content\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION delete_collection(_id text) RETURNS VOID AS $$\nDECLARE\nout collections%ROWTYPE;\nBEGIN\n DELETE FROM collections WHERE id = _id RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION get_collection(id text) RETURNS jsonb AS $$\nSELECT content FROM collections\nWHERE id=$1\n;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION all_collections() RETURNS jsonb AS $$\nSELECT jsonb_agg(content) FROM collections;\n;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE TABLE items (\n id text NOT NULL,\n geometry geometry NOT NULL,\n collection_id text NOT NULL,\n datetime timestamptz NOT NULL,\n end_datetime timestamptz NOT NULL,\n properties jsonb NOT NULL,\n content JSONB NOT NULL\n)\nPARTITION BY RANGE (datetime)\n;\n\nCREATE OR REPLACE FUNCTION properties_idx (IN content jsonb) RETURNS jsonb AS $$\n with recursive extract_all as\n (\n select\n ARRAY[key]::text[] as path,\n ARRAY[key]::text[] as fullpath,\n value\n FROM jsonb_each(content->'properties')\n union all\n select\n CASE WHEN obj_key IS NOT NULL THEN path || obj_key ELSE path END,\n path || coalesce(obj_key, (arr_key- 1)::text),\n coalesce(obj_value, arr_value)\n from extract_all\n left join lateral\n jsonb_each(case jsonb_typeof(value) when 'object' then value end)\n as o(obj_key, obj_value)\n on jsonb_typeof(value) = 'object'\n left join lateral\n jsonb_array_elements(case jsonb_typeof(value) when 'array' then value end)\n with ordinality as a(arr_value, arr_key)\n on jsonb_typeof(value) = 'array'\n where obj_key is not null or arr_key is not null\n )\n , paths AS (\n select\n array_to_string(path, '.') as path,\n value\n FROM extract_all\n WHERE\n jsonb_typeof(value) NOT IN ('array','object')\n ), grouped AS (\n SELECT path, jsonb_agg(distinct value) vals FROM paths group by path\n ) SELECT coalesce(jsonb_object_agg(path, CASE WHEN jsonb_array_length(vals)=1 THEN vals->0 ELSE vals END) - '{datetime}'::text[], '{}'::jsonb) FROM grouped\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET JIT TO OFF;\n\nCREATE INDEX \"datetime_idx\" ON items (datetime);\nCREATE INDEX \"end_datetime_idx\" ON items (end_datetime);\nCREATE INDEX \"properties_idx\" ON items USING GIN (properties jsonb_path_ops);\nCREATE INDEX \"collection_idx\" ON items (collection_id);\nCREATE INDEX \"geometry_idx\" ON items USING GIST (geometry);\nCREATE UNIQUE INDEX \"items_id_datetime_idx\" ON items (datetime, id);\n\nALTER TABLE items ADD CONSTRAINT items_collections_fk FOREIGN KEY (collection_id) REFERENCES collections(id) ON DELETE CASCADE DEFERRABLE;\n\nCREATE OR REPLACE FUNCTION analyze_empty_partitions() RETURNS VOID AS $$\nDECLARE\n p text;\nBEGIN\n FOR p IN SELECT partition FROM all_items_partitions WHERE est_cnt = 0 LOOP\n EXECUTE format('ANALYZE %I;', p);\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION items_partition_name(timestamptz) RETURNS text AS $$\n SELECT to_char($1, '\"items_p\"IYYY\"w\"IW');\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION items_partition_exists(text) RETURNS boolean AS $$\n SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_class WHERE relname=$1);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION items_partition_exists(timestamptz) RETURNS boolean AS $$\n SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_class WHERE relname=items_partition_name($1));\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION items_partition_create_worker(partition text, partition_start timestamptz, partition_end timestamptz) RETURNS VOID AS $$\nDECLARE\n err_context text;\nBEGIN\n EXECUTE format(\n $f$\n CREATE TABLE IF NOT EXISTS %1$I PARTITION OF items\n FOR VALUES FROM (%2$L) TO (%3$L);\n CREATE UNIQUE INDEX IF NOT EXISTS %4$I ON %1$I (id);\n $f$,\n partition,\n partition_start,\n partition_end,\n concat(partition, '_id_pk')\n );\nEXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', partition;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH to pgstac, public;\n\nCREATE OR REPLACE FUNCTION items_partition_create(ts timestamptz) RETURNS text AS $$\nDECLARE\n partition text := items_partition_name(ts);\n partition_start timestamptz;\n partition_end timestamptz;\nBEGIN\n IF items_partition_exists(partition) THEN\n RETURN partition;\n END IF;\n partition_start := date_trunc('week', ts);\n partition_end := partition_start + '1 week'::interval;\n PERFORM items_partition_create_worker(partition, partition_start, partition_end);\n RAISE NOTICE 'partition: %', partition;\n RETURN partition;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION items_partition_create(st timestamptz, et timestamptz) RETURNS SETOF text AS $$\nWITH t AS (\n SELECT\n generate_series(\n date_trunc('week',st),\n date_trunc('week', et),\n '1 week'::interval\n ) w\n)\nSELECT items_partition_create(w) FROM t;\n$$ LANGUAGE SQL;\n\n\nCREATE UNLOGGED TABLE items_staging (\n content JSONB NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION items_staging_insert_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n mindate timestamptz;\n maxdate timestamptz;\n partition text;\nBEGIN\n SELECT min(stac_datetime(content)), max(stac_datetime(content)) INTO mindate, maxdate FROM newdata;\n PERFORM items_partition_create(mindate, maxdate);\n INSERT INTO items (id, geometry, collection_id, datetime, end_datetime, properties, content)\n SELECT\n content->>'id',\n stac_geom(content),\n content->>'collection',\n stac_datetime(content),\n stac_end_datetime(content),\n properties_idx(content),\n content\n FROM newdata\n ;\n DELETE FROM items_staging;\n PERFORM analyze_empty_partitions();\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER items_staging_insert_trigger AFTER INSERT ON items_staging REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_insert_triggerfunc();\n\n\nCREATE UNLOGGED TABLE items_staging_ignore (\n content JSONB NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION items_staging_ignore_insert_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n mindate timestamptz;\n maxdate timestamptz;\n partition text;\nBEGIN\n SELECT min(stac_datetime(content)), max(stac_datetime(content)) INTO mindate, maxdate FROM newdata;\n PERFORM items_partition_create(mindate, maxdate);\n INSERT INTO items (id, geometry, collection_id, datetime, end_datetime, properties, content)\n SELECT\n content->>'id',\n stac_geom(content),\n content->>'collection',\n stac_datetime(content),\n stac_end_datetime(content),\n properties_idx(content),\n content\n FROM newdata\n ON CONFLICT DO NOTHING\n ;\n DELETE FROM items_staging_ignore;\n PERFORM analyze_empty_partitions();\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER items_staging_ignore_insert_trigger AFTER INSERT ON items_staging_ignore REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_ignore_insert_triggerfunc();\n\nCREATE UNLOGGED TABLE items_staging_upsert (\n content JSONB NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION items_staging_upsert_insert_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n mindate timestamptz;\n maxdate timestamptz;\n partition text;\nBEGIN\n SELECT min(stac_datetime(content)), max(stac_datetime(content)) INTO mindate, maxdate FROM newdata;\n PERFORM items_partition_create(mindate, maxdate);\n INSERT INTO items (id, geometry, collection_id, datetime, end_datetime, properties, content)\n SELECT\n content->>'id',\n stac_geom(content),\n content->>'collection',\n stac_datetime(content),\n stac_end_datetime(content),\n properties_idx(content),\n content\n FROM newdata\n ON CONFLICT (datetime, id) DO UPDATE SET\n content = EXCLUDED.content\n WHERE items.content IS DISTINCT FROM EXCLUDED.content\n ;\n DELETE FROM items_staging_upsert;\n PERFORM analyze_empty_partitions();\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER items_staging_upsert_insert_trigger AFTER INSERT ON items_staging_upsert REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_upsert_insert_triggerfunc();\n\nCREATE OR REPLACE FUNCTION items_update_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\nBEGIN\n NEW.id := NEW.content->>'id';\n NEW.datetime := stac_datetime(NEW.content);\n NEW.end_datetime := stac_end_datetime(NEW.content);\n NEW.collection_id := NEW.content->>'collection';\n NEW.geometry := stac_geom(NEW.content);\n NEW.properties := properties_idx(NEW.content);\n IF TG_OP = 'UPDATE' AND NEW IS NOT DISTINCT FROM OLD THEN\n RETURN NULL;\n END IF;\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER items_update_trigger BEFORE UPDATE ON items\n FOR EACH ROW EXECUTE PROCEDURE items_update_triggerfunc();\n\n/*\nView to get a table of available items partitions\nwith date ranges\n*/\nCREATE VIEW all_items_partitions AS\nWITH base AS\n(SELECT\n c.oid::pg_catalog.regclass::text as partition,\n pg_catalog.pg_get_expr(c.relpartbound, c.oid) as _constraint,\n regexp_matches(\n pg_catalog.pg_get_expr(c.relpartbound, c.oid),\n E'\\\\(''\\([0-9 :+-]*\\)''\\\\).*\\\\(''\\([0-9 :+-]*\\)''\\\\)'\n ) as t,\n reltuples::bigint as est_cnt\nFROM pg_catalog.pg_class c, pg_catalog.pg_inherits i\nWHERE c.oid = i.inhrelid AND i.inhparent = 'items'::regclass)\nSELECT partition, tstzrange(\n t[1]::timestamptz,\n t[2]::timestamptz\n), est_cnt\nFROM base\nORDER BY 2 desc;\n\nCREATE OR REPLACE VIEW items_partitions AS\nSELECT * FROM all_items_partitions WHERE est_cnt>0;\n\nCREATE OR REPLACE FUNCTION get_item(_id text) RETURNS jsonb AS $$\n SELECT content FROM items WHERE id=_id;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION delete_item(_id text) RETURNS VOID AS $$\nDECLARE\nout items%ROWTYPE;\nBEGIN\n DELETE FROM items WHERE id = _id RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION create_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION update_item(data jsonb) RETURNS VOID AS $$\nDECLARE\n out items%ROWTYPE;\nBEGIN\n UPDATE items SET content=data WHERE id = data->>'id' RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION collection_bbox(id text) RETURNS jsonb AS $$\nSELECT (replace(replace(replace(st_extent(geometry)::text,'BOX(','[['),')',']]'),' ',','))::jsonb\nFROM items WHERE collection_id=$1;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION collection_temporal_extent(id text) RETURNS jsonb AS $$\nSELECT to_jsonb(array[array[min(datetime)::text, max(datetime)::text]])\nFROM items WHERE collection_id=$1;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection_extents() RETURNS VOID AS $$\nUPDATE collections SET\n content = content ||\n jsonb_build_object(\n 'extent', jsonb_build_object(\n 'spatial', jsonb_build_object(\n 'bbox', collection_bbox(collections.id)\n ),\n 'temporal', jsonb_build_object(\n 'interval', collection_temporal_extent(collections.id)\n )\n )\n )\n;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION items_path(\n IN dotpath text,\n OUT field text,\n OUT path text,\n OUT path_txt text,\n OUT jsonpath text,\n OUT eq text\n) RETURNS RECORD AS $$\nDECLARE\npath_elements text[];\nlast_element text;\nBEGIN\ndotpath := replace(trim(dotpath), 'properties.', '');\n\nIF dotpath = '' THEN\n RETURN;\nEND IF;\n\npath_elements := string_to_array(dotpath, '.');\njsonpath := NULL;\n\nIF path_elements[1] IN ('id','geometry','datetime') THEN\n field := path_elements[1];\n path_elements := path_elements[2:];\nELSIF path_elements[1] = 'collection' THEN\n field := 'collection_id';\n path_elements := path_elements[2:];\nELSIF path_elements[1] IN ('links', 'assets', 'stac_version', 'stac_extensions') THEN\n field := 'content';\nELSE\n field := 'content';\n path_elements := '{properties}'::text[] || path_elements;\nEND IF;\nIF cardinality(path_elements)<1 THEN\n path := field;\n path_txt := field;\n jsonpath := '$';\n eq := NULL; -- format($F$ %s = %%s $F$, field);\n RETURN;\nEND IF;\n\n\nlast_element := path_elements[cardinality(path_elements)];\npath_elements := path_elements[1:cardinality(path_elements)-1];\njsonpath := concat(array_to_string('{$}'::text[] || array_map_ident(path_elements), '.'), '.', quote_ident(last_element));\npath_elements := array_map_literal(path_elements);\npath := format($F$ properties->%s $F$, quote_literal(dotpath));\npath_txt := format($F$ properties->>%s $F$, quote_literal(dotpath));\neq := format($F$ properties @? '$.%s[*] ? (@ == %%s) '$F$, quote_ident(dotpath));\n\nRAISE NOTICE 'ITEMS PATH -- % % % % %', field, path, path_txt, jsonpath, eq;\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION parse_dtrange(IN _indate jsonb, OUT _tstzrange tstzrange) AS $$\nWITH t AS (\n SELECT CASE\n WHEN jsonb_typeof(_indate) = 'array' THEN\n textarr(_indate)\n ELSE\n regexp_split_to_array(\n btrim(_indate::text,'\"'),\n '/'\n )\n END AS arr\n)\n, t1 AS (\n SELECT\n CASE\n WHEN array_upper(arr,1) = 1 OR arr[1] = '..' OR arr[1] IS NULL THEN '-infinity'::timestamptz\n ELSE arr[1]::timestamptz\n END AS st,\n CASE\n WHEN array_upper(arr,1) = 1 THEN arr[1]::timestamptz\n WHEN arr[2] = '..' OR arr[2] IS NULL THEN 'infinity'::timestamptz\n ELSE arr[2]::timestamptz\n END AS et\n FROM t\n)\nSELECT\n tstzrange(st,et)\nFROM t1;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION bbox_geom(_bbox jsonb) RETURNS geometry AS $$\nSELECT CASE jsonb_array_length(_bbox)\n WHEN 4 THEN\n ST_SetSRID(ST_MakeEnvelope(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float,\n (_bbox->>3)::float\n ),4326)\n WHEN 6 THEN\n ST_SetSRID(ST_3DMakeBox(\n ST_MakePoint(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float\n ),\n ST_MakePoint(\n (_bbox->>3)::float,\n (_bbox->>4)::float,\n (_bbox->>5)::float\n )\n ),4326)\n ELSE null END;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION cql_and_append(existing jsonb, newfilters jsonb) RETURNS jsonb AS $$\nSELECT CASE WHEN existing ? 'filter' AND newfilters IS NOT NULL THEN\n jsonb_build_object(\n 'and',\n jsonb_build_array(\n existing->'filter',\n newfilters\n )\n )\nELSE\n newfilters\nEND;\n$$ LANGUAGE SQL;\n\n\n-- ADDs base filters (ids, collections, datetime, bbox, intersects) that are\n-- added outside of the filter/query in the stac request\nCREATE OR REPLACE FUNCTION add_filters_to_cql(j jsonb) RETURNS jsonb AS $$\nDECLARE\nnewprop jsonb;\nnewprops jsonb := '[]'::jsonb;\nBEGIN\nIF j ? 'id' THEN\n newprop := jsonb_build_object(\n 'in',\n jsonb_build_array(\n '{\"property\":\"id\"}'::jsonb,\n j->'id'\n )\n );\n newprops := jsonb_insert(newprops, '{1}', newprop);\nEND IF;\nIF j ? 'collection' THEN\n newprop := jsonb_build_object(\n 'in',\n jsonb_build_array(\n '{\"property\":\"collection\"}'::jsonb,\n j->'collection'\n )\n );\n newprops := jsonb_insert(newprops, '{1}', newprop);\nEND IF;\n\nIF j ? 'datetime' THEN\n newprop := format(\n '{\"anyinteracts\":[{\"property\":\"datetime\"}, %s]}',\n j->'datetime'\n );\n newprops := jsonb_insert(newprops, '{1}', newprop);\nEND IF;\n\nIF j ? 'bbox' THEN\n newprop := format(\n '{\"intersects\":[{\"property\":\"geometry\"}, %s]}',\n j->'bbox'\n );\n newprops := jsonb_insert(newprops, '{1}', newprop);\nEND IF;\n\nIF j ? 'intersects' THEN\n newprop := format(\n '{\"intersects\":[{\"property\":\"geometry\"}, %s]}',\n j->'intersects'\n );\n newprops := jsonb_insert(newprops, '{1}', newprop);\nEND IF;\n\nRAISE NOTICE 'newprops: %', newprops;\n\nIF newprops IS NOT NULL AND jsonb_array_length(newprops) > 0 THEN\n return jsonb_set(\n j,\n '{filter}',\n cql_and_append(j, jsonb_build_object('and', newprops))\n ) - '{id,collection,datetime,bbox,intersects}'::text[];\nEND IF;\n\nreturn j;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION query_to_cqlfilter(j jsonb) RETURNS jsonb AS $$\n-- Translates anything passed in through the deprecated \"query\" into equivalent CQL\nWITH t AS (\n SELECT key as property, value as ops\n FROM jsonb_each(j->'query')\n), t2 AS (\n SELECT property, (jsonb_each(ops)).*\n FROM t WHERE jsonb_typeof(ops) = 'object'\n UNION ALL\n SELECT property, 'eq', ops\n FROM t WHERE jsonb_typeof(ops) != 'object'\n), t3 AS (\nSELECT\n jsonb_strip_nulls(jsonb_build_object(\n 'and',\n jsonb_agg(\n jsonb_build_object(\n key,\n jsonb_build_array(\n jsonb_build_object('property',property),\n value\n )\n )\n )\n )) as qcql FROM t2\n)\nSELECT\n CASE WHEN qcql IS NOT NULL THEN\n jsonb_set(j, '{filter}', cql_and_append(j, qcql)) - 'query'\n ELSE j\n END\nFROM t3\n;\n$$ LANGUAGE SQL;\n\n\n\nCREATE OR REPLACE FUNCTION temporal_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\nll text := 'datetime';\nlh text := 'end_datetime';\nrrange tstzrange;\nrl text;\nrh text;\noutq text;\nBEGIN\nrrange := parse_dtrange(args->1);\nRAISE NOTICE 'Constructing temporal query OP: %, ARGS: %, RRANGE: %', op, args, rrange;\nop := lower(op);\nrl := format('%L::timestamptz', lower(rrange));\nrh := format('%L::timestamptz', upper(rrange));\noutq := CASE op\n WHEN 't_before' THEN 'lh < rl'\n WHEN 't_after' THEN 'll > rh'\n WHEN 't_meets' THEN 'lh = rl'\n WHEN 't_metby' THEN 'll = rh'\n WHEN 't_overlaps' THEN 'll < rl AND rl < lh < rh'\n WHEN 't_overlappedby' THEN 'rl < ll < rh AND lh > rh'\n WHEN 't_starts' THEN 'll = rl AND lh < rh'\n WHEN 't_startedby' THEN 'll = rl AND lh > rh'\n WHEN 't_during' THEN 'll > rl AND lh < rh'\n WHEN 't_contains' THEN 'll < rl AND lh > rh'\n WHEN 't_finishes' THEN 'll > rl AND lh = rh'\n WHEN 't_finishedby' THEN 'll < rl AND lh = rh'\n WHEN 't_equals' THEN 'll = rl AND lh = rh'\n WHEN 't_disjoint' THEN 'NOT (ll <= rh AND lh >= rl)'\n WHEN 't_intersects' THEN 'll <= rh AND lh >= rl'\n WHEN 'anyinteracts' THEN 'll <= rh AND lh >= rl'\nEND;\noutq := regexp_replace(outq, '\\mll\\M', ll);\noutq := regexp_replace(outq, '\\mlh\\M', lh);\noutq := regexp_replace(outq, '\\mrl\\M', rl);\noutq := regexp_replace(outq, '\\mrh\\M', rh);\noutq := format('(%s)', outq);\nRETURN outq;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION spatial_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\ngeom text;\nj jsonb := args->1;\nBEGIN\nop := lower(op);\nRAISE NOTICE 'Constructing spatial query OP: %, ARGS: %', op, args;\nIF op NOT IN ('s_equals','s_disjoint','s_touches','s_within','s_overlaps','s_crosses','s_intersects','intersects','s_contains') THEN\n RAISE EXCEPTION 'Spatial Operator % Not Supported', op;\nEND IF;\nop := regexp_replace(op, '^s_', 'st_');\nIF op = 'intersects' THEN\n op := 'st_intersects';\nEND IF;\n-- Convert geometry to WKB string\nIF j ? 'type' AND j ? 'coordinates' THEN\n geom := st_geomfromgeojson(j)::text;\nELSIF jsonb_typeof(j) = 'array' THEN\n geom := bbox_geom(j)::text;\nEND IF;\n\nRETURN format('%s(geometry, %L::geometry)', op, geom);\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n/* cql_query_op -- Parses a CQL query operation, recursing when necessary\n IN jsonb -- a subelement from a valid stac query\n IN text -- the operator being used on elements passed in\n RETURNS a SQL fragment to be used in a WHERE clause\n*/\nCREATE OR REPLACE FUNCTION cql_query_op(j jsonb, _op text DEFAULT NULL) RETURNS text AS $$\nDECLARE\njtype text := jsonb_typeof(j);\nop text := lower(_op);\nops jsonb :=\n '{\n \"eq\": \"%s = %s\",\n \"lt\": \"%s < %s\",\n \"lte\": \"%s <= %s\",\n \"gt\": \"%s > %s\",\n \"gte\": \"%s >= %s\",\n \"like\": \"%s LIKE %s\",\n \"+\": \"%s + %s\",\n \"-\": \"%s - %s\",\n \"*\": \"%s * %s\",\n \"/\": \"%s / %s\",\n \"in\": \"%s = ANY (%s)\",\n \"not\": \"NOT (%s)\",\n \"between\": \"%s BETWEEN %s AND %s\",\n \"lower\":\"lower(%s)\"\n }'::jsonb;\nret text;\nargs text[] := NULL;\n\nBEGIN\nRAISE NOTICE 'j: %, op: %, jtype: %', j, op, jtype;\n\n-- Set Lower Case on Both Arguments When Case Insensitive Flag Set\nIF op in ('eq','lt','lte','gt','gte','like') AND jsonb_typeof(j->2) = 'boolean' THEN\n IF (j->>2)::boolean THEN\n RETURN format(concat('(',ops->>op,')'), cql_query_op(jsonb_build_array(j->0), 'lower'), cql_query_op(jsonb_build_array(j->1), 'lower'));\n END IF;\nEND IF;\n\n-- Special Case when comparing a property in a jsonb field to a string or number using eq\n-- Allows to leverage GIN index on jsonb fields\nIF op = 'eq' THEN\n IF j->0 ? 'property'\n AND jsonb_typeof(j->1) IN ('number','string')\n AND (items_path(j->0->>'property')).eq IS NOT NULL\n THEN\n RETURN format((items_path(j->0->>'property')).eq, j->1);\n END IF;\nEND IF;\n\nIF op ilike 't_%' or op = 'anyinteracts' THEN\n RETURN temporal_op_query(op, j);\nEND IF;\n\nIF op ilike 's_%' or op = 'intersects' THEN\n RETURN spatial_op_query(op, j);\nEND IF;\n\n\nIF jtype = 'object' THEN\n RAISE NOTICE 'parsing object';\n IF j ? 'property' THEN\n -- Convert the property to be used as an identifier\n return (items_path(j->>'property')).path_txt;\n ELSIF _op IS NULL THEN\n -- Iterate to convert elements in an object where the operator has not been set\n -- Combining with AND\n SELECT\n array_to_string(array_agg(cql_query_op(e.value, e.key)), ' AND ')\n INTO ret\n FROM jsonb_each(j) e;\n RETURN ret;\n END IF;\nEND IF;\n\nIF jtype = 'string' THEN\n RETURN quote_literal(j->>0);\nEND IF;\n\nIF jtype ='number' THEN\n RETURN (j->>0)::numeric;\nEND IF;\n\nIF jtype = 'array' AND op IS NULL THEN\n RAISE NOTICE 'Parsing array into array arg. j: %', j;\n SELECT format($f$ '{%s}'::text[] $f$, string_agg(e,',')) INTO ret FROM jsonb_array_elements_text(j) e;\n RETURN ret;\nEND IF;\n\n\n-- If the type of the passed json is an array\n-- Calculate the arguments that will be passed to functions/operators\nIF jtype = 'array' THEN\n RAISE NOTICE 'Parsing array into args. j: %', j;\n -- If any argument is numeric, cast any text arguments to numeric\n IF j @? '$[*] ? (@.type() == \"number\")' THEN\n SELECT INTO args\n array_agg(concat('(',cql_query_op(e),')::numeric'))\n FROM jsonb_array_elements(j) e;\n ELSE\n SELECT INTO args\n array_agg(cql_query_op(e))\n FROM jsonb_array_elements(j) e;\n END IF;\n --RETURN args;\nEND IF;\nRAISE NOTICE 'ARGS after array cleaning: %', args;\n\nIF op IS NULL THEN\n RETURN args::text[];\nEND IF;\n\nIF args IS NULL OR cardinality(args) < 1 THEN\n RAISE NOTICE 'No Args';\n RETURN '';\nEND IF;\n\nIF op IN ('and','or') THEN\n SELECT\n CONCAT(\n '(',\n array_to_string(args, UPPER(CONCAT(' ',op,' '))),\n ')'\n ) INTO ret\n FROM jsonb_array_elements(j) e;\n RETURN ret;\nEND IF;\n\n-- If the op is in the ops json then run using the template in the json\nIF ops ? op THEN\n RAISE NOTICE 'ARGS: % MAPPED: %',args, array_map_literal(args);\n\n RETURN format(concat('(',ops->>op,')'), VARIADIC args);\nEND IF;\n\nRETURN j->>0;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n\n\nCREATE OR REPLACE FUNCTION cql_to_where(_search jsonb = '{}'::jsonb) RETURNS text AS $$\nDECLARE\nsearch jsonb := _search;\n_where text;\nBEGIN\nRAISE NOTICE 'SEARCH CQL 1: %', search;\n\n-- Convert any old style stac query to cql\nsearch := query_to_cqlfilter(search);\n\nRAISE NOTICE 'SEARCH CQL 2: %', search;\n\n-- Convert item,collection,datetime,bbox,intersects to cql\nsearch := add_filters_to_cql(search);\n\nRAISE NOTICE 'SEARCH CQL Final: %', search;\n_where := cql_query_op(search->'filter');\n\nIF trim(_where) = '' THEN\n _where := NULL;\nEND IF;\n_where := coalesce(_where, ' TRUE ');\nRETURN _where;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION parse_sort_dir(_dir text, reverse boolean default false) RETURNS text AS $$\nWITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n) SELECT\n CASE\n WHEN NOT reverse THEN d\n WHEN d = 'ASC' THEN 'DESC'\n WHEN d = 'DESC' THEN 'ASC'\n END\nFROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION sort_dir_to_op(_dir text, prev boolean default false) RETURNS text AS $$\nWITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n) SELECT\n CASE\n WHEN d = 'ASC' AND prev THEN '<='\n WHEN d = 'DESC' AND prev THEN '>='\n WHEN d = 'ASC' THEN '>='\n WHEN d = 'DESC' THEN '<='\n END\nFROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION sort_sqlorderby(\n _search jsonb DEFAULT NULL,\n reverse boolean DEFAULT FALSE\n) RETURNS text AS $$\nWITH sorts AS (\n SELECT\n (items_path(value->>'field')).path as key,\n parse_sort_dir(value->>'direction', reverse) as dir\n FROM jsonb_array_elements(\n '[]'::jsonb\n ||\n coalesce(_search->'sort','[{\"field\":\"datetime\", \"direction\":\"desc\"}]')\n ||\n '[{\"field\":\"id\",\"direction\":\"desc\"}]'::jsonb\n )\n)\nSELECT array_to_string(\n array_agg(concat(key, ' ', dir)),\n ', '\n) FROM sorts;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_sort_dir(sort_item jsonb) RETURNS text AS $$\nSELECT CASE WHEN sort_item->>'direction' ILIKE 'desc%' THEN 'DESC' ELSE 'ASC' END;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION get_token_filter(_search jsonb = '{}'::jsonb, token_rec jsonb DEFAULT NULL) RETURNS text AS $$\nDECLARE\ntoken_id text;\nfilters text[] := '{}'::text[];\nprev boolean := TRUE;\nfield text;\ndir text;\nsort record;\norfilters text[] := '{}'::text[];\nandfilters text[] := '{}'::text[];\noutput text;\ntoken_where text;\nBEGIN\n-- If no token provided return NULL\nIF token_rec IS NULL THEN\n IF NOT (_search ? 'token' AND\n (\n (_search->>'token' ILIKE 'prev:%')\n OR\n (_search->>'token' ILIKE 'next:%')\n )\n ) THEN\n RETURN NULL;\n END IF;\n prev := (_search->>'token' ILIKE 'prev:%');\n token_id := substr(_search->>'token', 6);\n SELECT to_jsonb(items) INTO token_rec FROM items WHERE id=token_id;\nEND IF;\nRAISE NOTICE 'TOKEN ID: %', token_rec->'id';\n\nCREATE TEMP TABLE sorts (\n _row int GENERATED ALWAYS AS IDENTITY NOT NULL,\n _field text PRIMARY KEY,\n _dir text NOT NULL,\n _val text\n) ON COMMIT DROP;\n\n-- Make sure we only have distinct columns to sort with taking the first one we get\nINSERT INTO sorts (_field, _dir)\n SELECT\n (items_path(value->>'field')).path,\n get_sort_dir(value)\n FROM\n jsonb_array_elements(coalesce(_search->'sort','[{\"field\":\"datetime\",\"direction\":\"desc\"}]'))\nON CONFLICT DO NOTHING\n;\n\n-- Get the first sort direction provided. As the id is a primary key, if there are any\n-- sorts after id they won't do anything, so make sure that id is the last sort item.\nSELECT _dir INTO dir FROM sorts ORDER BY _row ASC LIMIT 1;\nIF EXISTS (SELECT 1 FROM sorts WHERE _field = 'id') THEN\n DELETE FROM sorts WHERE _row > (SELECT _row FROM sorts WHERE _field = 'id');\nELSE\n INSERT INTO sorts (_field, _dir) VALUES ('id', dir);\nEND IF;\n\n-- Add value from looked up item to the sorts table\nUPDATE sorts SET _val=quote_literal(token_rec->>_field);\n\n-- Check if all sorts are the same direction and use row comparison\n-- to filter\nIF (SELECT count(DISTINCT _dir) FROM sorts) = 1 THEN\n SELECT format(\n '(%s) %s (%s)',\n concat_ws(', ', VARIADIC array_agg(quote_ident(_field))),\n CASE WHEN (prev AND dir = 'ASC') OR (NOT prev AND dir = 'DESC') THEN '<' ELSE '>' END,\n concat_ws(', ', VARIADIC array_agg(_val))\n ) INTO output FROM sorts\n WHERE token_rec ? _field\n ;\nELSE\n FOR sort IN SELECT * FROM sorts ORDER BY _row asc LOOP\n RAISE NOTICE 'SORT: %', sort;\n IF sort._row = 1 THEN\n orfilters := orfilters || format('(%s %s %s)',\n quote_ident(sort._field),\n CASE WHEN (prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC') THEN '<' ELSE '>' END,\n sort._val\n );\n ELSE\n orfilters := orfilters || format('(%s AND %s %s %s)',\n array_to_string(andfilters, ' AND '),\n quote_ident(sort._field),\n CASE WHEN (prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC') THEN '<' ELSE '>' END,\n sort._val\n );\n\n END IF;\n andfilters := andfilters || format('%s = %s',\n quote_ident(sort._field),\n sort._val\n );\n END LOOP;\n output := array_to_string(orfilters, ' OR ');\nEND IF;\nDROP TABLE IF EXISTS sorts;\ntoken_where := concat('(',coalesce(output,'true'),')');\nIF trim(token_where) = '' THEN\n token_where := NULL;\nEND IF;\nRAISE NOTICE 'TOKEN_WHERE: |%|',token_where;\nRETURN token_where;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION search_tohash(jsonb) RETURNS jsonb AS $$\n SELECT $1 - '{token,limit,context,includes,excludes}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION search_hash(jsonb) RETURNS text AS $$\n SELECT md5(search_tohash($1)::text);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE TABLE IF NOT EXISTS searches(\n hash text GENERATED ALWAYS AS (search_hash(search)) STORED PRIMARY KEY,\n search jsonb NOT NULL,\n _where text,\n orderby text,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n statslastupdated timestamptz,\n estimated_count bigint,\n total_count bigint\n);\n\nCREATE OR REPLACE FUNCTION search_query(_search jsonb = '{}'::jsonb, updatestats boolean DEFAULT false) RETURNS searches AS $$\nDECLARE\n search searches%ROWTYPE;\nBEGIN\nINSERT INTO searches (search)\n VALUES (search_tohash(_search))\n ON CONFLICT DO NOTHING\n RETURNING * INTO search;\nIF search.hash IS NULL THEN\n SELECT * INTO search FROM searches WHERE hash=search_hash(_search);\nEND IF;\nIF search._where IS NULL THEN\n search._where := cql_to_where(_search);\nEND IF;\nIF search.orderby IS NULL THEN\n search.orderby := sort_sqlorderby(_search);\nEND IF;\n\nIF search.statslastupdated IS NULL OR age(search.statslastupdated) > '1 day'::interval OR (_search ? 'context' AND search.total_count IS NULL) THEN\n updatestats := TRUE;\nEND IF;\n\nIF updatestats THEN\n -- Get Estimated Stats\n RAISE NOTICE 'Getting stats for %', search._where;\n search.estimated_count := estimated_count(search._where);\n RAISE NOTICE 'Estimated Count: %', search.estimated_count;\n\n IF _search ? 'context' OR search.estimated_count < 10000 THEN\n --search.total_count := partition_count(search._where);\n EXECUTE format(\n 'SELECT count(*) FROM items WHERE %s',\n search._where\n ) INTO search.total_count;\n RAISE NOTICE 'Actual Count: %', search.total_count;\n ELSE\n search.total_count := NULL;\n END IF;\n search.statslastupdated := now();\nEND IF;\n\nsearch.lastused := now();\nsearch.usecount := coalesce(search.usecount,0) + 1;\nRAISE NOTICE 'SEARCH: %', search;\nUPDATE searches SET\n _where = search._where,\n orderby = search.orderby,\n lastused = search.lastused,\n usecount = search.usecount,\n statslastupdated = search.statslastupdated,\n estimated_count = search.estimated_count,\n total_count = search.total_count\nWHERE hash = search.hash\n;\nRETURN search;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n\nCREATE OR REPLACE FUNCTION search(_search jsonb = '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n searches searches%ROWTYPE;\n _where text;\n token_where text;\n full_where text;\n orderby text;\n query text;\n token_type text := substr(_search->>'token',1,4);\n _limit int := coalesce((_search->>'limit')::int, 10);\n curs refcursor;\n cntr int := 0;\n iter_record items%ROWTYPE;\n first_record items%ROWTYPE;\n last_record items%ROWTYPE;\n out_records jsonb := '[]'::jsonb;\n prev_query text;\n next text;\n prev_id text;\n has_next boolean := false;\n has_prev boolean := false;\n prev text;\n total_count bigint;\n context jsonb;\n collection jsonb;\n includes text[];\n excludes text[];\n exit_flag boolean := FALSE;\n batches int := 0;\n timer timestamptz := clock_timestamp();\nBEGIN\nsearches := search_query(_search);\n_where := searches._where;\norderby := searches.orderby;\ntotal_count := coalesce(searches.total_count, searches.estimated_count);\n\n\nIF token_type='prev' THEN\n token_where := get_token_filter(_search, null::jsonb);\n orderby := sort_sqlorderby(_search, TRUE);\nEND IF;\nIF token_type='next' THEN\n token_where := get_token_filter(_search, null::jsonb);\nEND IF;\n\nfull_where := concat_ws(' AND ', _where, token_where);\nRAISE NOTICE 'FULL QUERY % %', full_where, clock_timestamp()-timer;\ntimer := clock_timestamp();\n\nFOR query IN SELECT partition_queries(full_where, orderby) LOOP\n timer := clock_timestamp();\n query := format('%s LIMIT %L', query, _limit + 1);\n RAISE NOTICE 'Partition Query: %', query;\n batches := batches + 1;\n curs = create_cursor(query);\n LOOP\n FETCH curs into iter_record;\n EXIT WHEN NOT FOUND;\n cntr := cntr + 1;\n last_record := iter_record;\n IF cntr = 1 THEN\n first_record := last_record;\n END IF;\n IF cntr <= _limit THEN\n out_records := out_records || last_record.content;\n ELSIF cntr > _limit THEN\n has_next := true;\n exit_flag := true;\n EXIT;\n END IF;\n END LOOP;\n RAISE NOTICE 'Query took %', clock_timestamp()-timer;\n timer := clock_timestamp();\n EXIT WHEN exit_flag;\nEND LOOP;\nRAISE NOTICE 'Scanned through % partitions.', batches;\n\n\n-- Flip things around if this was the result of a prev token query\nIF token_type='prev' THEN\n out_records := flip_jsonb_array(out_records);\n first_record := last_record;\nEND IF;\n\n-- If this query has a token, see if there is data before the first record\nIF _search ? 'token' THEN\n prev_query := format(\n 'SELECT 1 FROM items WHERE %s LIMIT 1',\n concat_ws(\n ' AND ',\n _where,\n trim(get_token_filter(_search, to_jsonb(first_record)))\n )\n );\n RAISE NOTICE 'Query to get previous record: % --- %', prev_query, first_record;\n EXECUTE prev_query INTO has_prev;\n IF FOUND and has_prev IS NOT NULL THEN\n RAISE NOTICE 'Query results from prev query: %', has_prev;\n has_prev := TRUE;\n END IF;\nEND IF;\nhas_prev := COALESCE(has_prev, FALSE);\n\nRAISE NOTICE 'token_type: %, has_next: %, has_prev: %', token_type, has_next, has_prev;\nIF has_prev THEN\n prev := out_records->0->>'id';\nEND IF;\nIF has_next OR token_type='prev' THEN\n next := out_records->-1->>'id';\nEND IF;\n\n\n\n-- include/exclude any fields following fields extension\nIF _search ? 'fields' THEN\n IF _search->'fields' ? 'exclude' THEN\n excludes=textarr(_search->'fields'->'exclude');\n END IF;\n IF _search->'fields' ? 'include' THEN\n includes=textarr(_search->'fields'->'include');\n IF array_length(includes, 1)>0 AND NOT 'id' = ANY (includes) THEN\n includes = includes || '{id}';\n END IF;\n END IF;\n SELECT jsonb_agg(filter_jsonb(row, includes, excludes)) INTO out_records FROM jsonb_array_elements(out_records) row;\nEND IF;\n\n\ncontext := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'matched', total_count,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n));\n\ncollection := jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', out_records,\n 'next', next,\n 'prev', prev,\n 'context', context\n);\n\nRETURN collection;\nEND;\n$$ LANGUAGE PLPGSQL SET jit TO off;\nINSERT INTO pgstac.migrations (version) VALUES ('0.3.0');\n" }, { "path": "src/pgstac/migrations/pgstac.0.3.1-0.3.2.sql", "content": "SET SEARCH_PATH to pgstac, public;\nset check_function_bodies = off;\n\nCREATE OR REPLACE FUNCTION pgstac.add_filters_to_cql(j jsonb)\n RETURNS jsonb\n LANGUAGE plpgsql\nAS $function$\nDECLARE\nnewprop jsonb;\nnewprops jsonb := '[]'::jsonb;\nBEGIN\nIF j ? 'id' THEN\n newprop := jsonb_build_object(\n 'in',\n jsonb_build_array(\n '{\"property\":\"id\"}'::jsonb,\n j->'id'\n )\n );\n newprops := jsonb_insert(newprops, '{1}', newprop);\nEND IF;\nIF j ? 'collections' THEN\n newprop := jsonb_build_object(\n 'in',\n jsonb_build_array(\n '{\"property\":\"collection\"}'::jsonb,\n j->'collections'\n )\n );\n newprops := jsonb_insert(newprops, '{1}', newprop);\nEND IF;\n\nIF j ? 'datetime' THEN\n newprop := format(\n '{\"anyinteracts\":[{\"property\":\"datetime\"}, %s]}',\n j->'datetime'\n );\n newprops := jsonb_insert(newprops, '{1}', newprop);\nEND IF;\n\nIF j ? 'bbox' THEN\n newprop := format(\n '{\"intersects\":[{\"property\":\"geometry\"}, %s]}',\n j->'bbox'\n );\n newprops := jsonb_insert(newprops, '{1}', newprop);\nEND IF;\n\nIF j ? 'intersects' THEN\n newprop := format(\n '{\"intersects\":[{\"property\":\"geometry\"}, %s]}',\n j->'intersects'\n );\n newprops := jsonb_insert(newprops, '{1}', newprop);\nEND IF;\n\nRAISE NOTICE 'newprops: %', newprops;\n\nIF newprops IS NOT NULL AND jsonb_array_length(newprops) > 0 THEN\n return jsonb_set(\n j,\n '{filter}',\n cql_and_append(j, jsonb_build_object('and', newprops))\n ) - '{id,collections,datetime,bbox,intersects}'::text[];\nEND IF;\n\nreturn j;\nEND;\n$function$\n;\n\n\n\nINSERT INTO migrations (version) VALUES ('0.3.2');\n" }, { "path": "src/pgstac/migrations/pgstac.0.3.1.sql", "content": "CREATE EXTENSION IF NOT EXISTS postgis;\nCREATE SCHEMA IF NOT EXISTS pgstac;\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE TABLE migrations (\n version text,\n datetime timestamptz DEFAULT now() NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION notice(VARIADIC text[]) RETURNS boolean AS $$\nDECLARE\ndebug boolean := current_setting('pgstac.debug', true);\nBEGIN\n IF debug THEN\n RAISE NOTICE 'NOTICE FROM FUNC: % >>>>> %', concat_ws(' | ', $1), clock_timestamp();\n RETURN TRUE;\n END IF;\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION empty_arr(ANYARRAY) RETURNS BOOLEAN AS $$\nSELECT CASE\n WHEN $1 IS NULL THEN TRUE\n WHEN cardinality($1)<1 THEN TRUE\nELSE FALSE\nEND;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_map_ident(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_ident(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_map_literal(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_literal(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION estimated_count(_where text) RETURNS bigint AS $$\nDECLARE\nrec record;\nrows bigint;\nBEGIN\n FOR rec in EXECUTE format(\n $q$\n EXPLAIN SELECT 1 FROM items WHERE %s\n $q$,\n _where)\n LOOP\n rows := substring(rec.\"QUERY PLAN\" FROM ' rows=([[:digit:]]+)');\n EXIT WHEN rows IS NOT NULL;\n END LOOP;\n\n RETURN rows;\nEND;\n$$ LANGUAGE PLPGSQL;\n/* converts a jsonb text array to a pg text[] array */\nCREATE OR REPLACE FUNCTION textarr(_js jsonb)\n RETURNS text[] AS $$\n SELECT\n CASE jsonb_typeof(_js)\n WHEN 'array' THEN ARRAY(SELECT jsonb_array_elements_text(_js))\n ELSE ARRAY[_js->>0]\n END\n;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION jsonb_paths (IN jdata jsonb, OUT path text[], OUT value jsonb) RETURNS\nSETOF RECORD AS $$\nwith recursive extract_all as\n(\n select\n ARRAY[key]::text[] as path,\n value\n FROM jsonb_each(jdata)\nunion all\n select\n path || coalesce(obj_key, (arr_key- 1)::text),\n coalesce(obj_value, arr_value)\n from extract_all\n left join lateral\n jsonb_each(case jsonb_typeof(value) when 'object' then value end)\n as o(obj_key, obj_value)\n on jsonb_typeof(value) = 'object'\n left join lateral\n jsonb_array_elements(case jsonb_typeof(value) when 'array' then value end)\n with ordinality as a(arr_value, arr_key)\n on jsonb_typeof(value) = 'array'\n where obj_key is not null or arr_key is not null\n)\nselect *\nfrom extract_all;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION jsonb_obj_paths (IN jdata jsonb, OUT path text[], OUT value jsonb) RETURNS\nSETOF RECORD AS $$\nwith recursive extract_all as\n(\n select\n ARRAY[key]::text[] as path,\n value\n FROM jsonb_each(jdata)\nunion all\n select\n path || obj_key,\n obj_value\n from extract_all\n left join lateral\n jsonb_each(case jsonb_typeof(value) when 'object' then value end)\n as o(obj_key, obj_value)\n on jsonb_typeof(value) = 'object'\n where obj_key is not null\n)\nselect *\nfrom extract_all;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION jsonb_val_paths (IN jdata jsonb, OUT path text[], OUT value jsonb) RETURNS\nSETOF RECORD AS $$\nSELECT * FROM jsonb_obj_paths(jdata) WHERE jsonb_typeof(value) not in ('object','array');\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION path_includes(IN path text[], IN includes text[]) RETURNS BOOLEAN AS $$\nWITH t AS (SELECT unnest(includes) i)\nSELECT EXISTS (\n SELECT 1 FROM t WHERE path @> string_to_array(trim(i), '.')\n);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION path_excludes(IN path text[], IN excludes text[]) RETURNS BOOLEAN AS $$\nWITH t AS (SELECT unnest(excludes) e)\nSELECT NOT EXISTS (\n SELECT 1 FROM t WHERE path @> string_to_array(trim(e), '.')\n);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION jsonb_obj_paths_filtered (\n IN jdata jsonb,\n IN includes text[] DEFAULT ARRAY[]::text[],\n IN excludes text[] DEFAULT ARRAY[]::text[],\n OUT path text[],\n OUT value jsonb\n) RETURNS\nSETOF RECORD AS $$\nSELECT path, value\nFROM jsonb_obj_paths(jdata)\nWHERE\n CASE WHEN cardinality(includes) > 0 THEN path_includes(path, includes) ELSE TRUE END\n AND\n path_excludes(path, excludes)\n\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION filter_jsonb(\n IN jdata jsonb,\n IN includes text[] DEFAULT ARRAY[]::text[],\n IN excludes text[] DEFAULT ARRAY[]::text[]\n) RETURNS jsonb AS $$\nDECLARE\nrec RECORD;\noutj jsonb := '{}'::jsonb;\ncreated_paths text[] := '{}'::text[];\nBEGIN\n\nIF empty_arr(includes) AND empty_arr(excludes) THEN\nRAISE NOTICE 'no filter';\n RETURN jdata;\nEND IF;\nFOR rec in\nSELECT * FROM jsonb_obj_paths_filtered(jdata, includes, excludes)\nWHERE jsonb_typeof(value) != 'object'\nLOOP\n IF array_length(rec.path,1)>1 THEN\n FOR i IN 1..(array_length(rec.path,1)-1) LOOP\n IF NOT array_to_string(rec.path[1:i],'.') = ANY (created_paths) THEN\n outj := jsonb_set(outj, rec.path[1:i],'{}', true);\n created_paths := created_paths || array_to_string(rec.path[1:i],'.');\n END IF;\n END LOOP;\n END IF;\n outj := jsonb_set(outj, rec.path, rec.value, true);\n created_paths := created_paths || array_to_string(rec.path,'.');\nEND LOOP;\nRETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION flip_jsonb_array(j jsonb) RETURNS jsonb AS $$\nSELECT jsonb_agg(value) FROM (SELECT value FROM jsonb_array_elements(j) WITH ORDINALITY ORDER BY ordinality DESC) as t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n/* Functions to create an iterable of cursors over partitions. */\nCREATE OR REPLACE FUNCTION create_cursor(q text) RETURNS refcursor AS $$\nDECLARE\n curs refcursor;\nBEGIN\n OPEN curs FOR EXECUTE q;\n RETURN curs;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION partition_queries(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC'\n) RETURNS SETOF text AS $$\nDECLARE\n partition_query text;\n query text;\n p record;\n cursors refcursor;\nBEGIN\nIF _orderby ILIKE 'datetime d%' THEN\n partition_query := format($q$\n SELECT partition, tstzrange\n FROM items_partitions\n ORDER BY tstzrange DESC;\n $q$);\nELSIF _orderby ILIKE 'datetime a%' THEN\n partition_query := format($q$\n SELECT partition, tstzrange\n FROM items_partitions\n ORDER BY tstzrange ASC\n ;\n $q$);\nELSE\n query := format($q$\n SELECT * FROM items\n WHERE %s\n ORDER BY %s\n $q$, _where, _orderby\n );\n\n RETURN NEXT query;\n RETURN;\nEND IF;\nFOR p IN\n EXECUTE partition_query\nLOOP\n query := format($q$\n SELECT * FROM items\n WHERE datetime >= %L AND datetime < %L AND %s\n ORDER BY %s\n $q$, lower(p.tstzrange), upper(p.tstzrange), _where, _orderby\n );\n RETURN NEXT query;\nEND LOOP;\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION partition_cursor(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC'\n) RETURNS SETOF refcursor AS $$\nDECLARE\n partition_query text;\n query text;\n p record;\n cursors refcursor;\nBEGIN\nFOR query IN SELECT * FROM partion_queries(_where, _orderby) LOOP\n RETURN NEXT create_cursor(query);\nEND LOOP;\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION partition_count(\n IN _where text DEFAULT 'TRUE'\n) RETURNS bigint AS $$\nDECLARE\n partition_query text;\n query text;\n p record;\n subtotal bigint;\n total bigint := 0;\nBEGIN\npartition_query := format($q$\n SELECT partition, tstzrange\n FROM items_partitions\n ORDER BY tstzrange DESC;\n$q$);\nRAISE NOTICE 'Partition Query: %', partition_query;\nFOR p IN\n EXECUTE partition_query\nLOOP\n query := format($q$\n SELECT count(*) FROM items\n WHERE datetime BETWEEN %L AND %L AND %s\n $q$, lower(p.tstzrange), upper(p.tstzrange), _where\n );\n RAISE NOTICE 'Query %', query;\n RAISE NOTICE 'Partition %, Count %, Total %',p.partition, subtotal, total;\n EXECUTE query INTO subtotal;\n total := subtotal + total;\nEND LOOP;\nRETURN total;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n/* looks for a geometry in a stac item first from geometry and falling back to bbox */\nCREATE OR REPLACE FUNCTION stac_geom(value jsonb) RETURNS geometry AS $$\nSELECT\n CASE\n WHEN value->>'geometry' IS NOT NULL THEN\n ST_GeomFromGeoJSON(value->>'geometry')\n WHEN value->>'bbox' IS NOT NULL THEN\n ST_MakeEnvelope(\n (value->'bbox'->>0)::float,\n (value->'bbox'->>1)::float,\n (value->'bbox'->>2)::float,\n (value->'bbox'->>3)::float,\n 4326\n )\n ELSE NULL\n END as geometry\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION stac_datetime(value jsonb) RETURNS timestamptz AS $$\nSELECT COALESCE(\n (value->'properties'->>'datetime')::timestamptz,\n (value->'properties'->>'start_datetime')::timestamptz\n);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_end_datetime(value jsonb) RETURNS timestamptz AS $$\nSELECT COALESCE(\n (value->'properties'->>'datetime')::timestamptz,\n (value->'properties'->>'end_datetime')::timestamptz\n);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\n\nCREATE OR REPLACE FUNCTION stac_daterange(value jsonb) RETURNS tstzrange AS $$\nSELECT tstzrange(stac_datetime(value),stac_end_datetime(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE TABLE IF NOT EXISTS collections (\n id VARCHAR GENERATED ALWAYS AS (content->>'id') STORED PRIMARY KEY,\n content JSONB\n);\n\nCREATE OR REPLACE FUNCTION create_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection(data jsonb) RETURNS VOID AS $$\nDECLARE\nout collections%ROWTYPE;\nBEGIN\n UPDATE collections SET content=data WHERE id = data->>'id' RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ON CONFLICT (id) DO\n UPDATE\n SET content=EXCLUDED.content\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION delete_collection(_id text) RETURNS VOID AS $$\nDECLARE\nout collections%ROWTYPE;\nBEGIN\n DELETE FROM collections WHERE id = _id RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION get_collection(id text) RETURNS jsonb AS $$\nSELECT content FROM collections\nWHERE id=$1\n;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION all_collections() RETURNS jsonb AS $$\nSELECT jsonb_agg(content) FROM collections;\n;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE TABLE items (\n id text NOT NULL,\n geometry geometry NOT NULL,\n collection_id text NOT NULL,\n datetime timestamptz NOT NULL,\n end_datetime timestamptz NOT NULL,\n properties jsonb NOT NULL,\n content JSONB NOT NULL\n)\nPARTITION BY RANGE (datetime)\n;\n\nCREATE OR REPLACE FUNCTION properties_idx (IN content jsonb) RETURNS jsonb AS $$\n with recursive extract_all as\n (\n select\n ARRAY[key]::text[] as path,\n ARRAY[key]::text[] as fullpath,\n value\n FROM jsonb_each(content->'properties')\n union all\n select\n CASE WHEN obj_key IS NOT NULL THEN path || obj_key ELSE path END,\n path || coalesce(obj_key, (arr_key- 1)::text),\n coalesce(obj_value, arr_value)\n from extract_all\n left join lateral\n jsonb_each(case jsonb_typeof(value) when 'object' then value end)\n as o(obj_key, obj_value)\n on jsonb_typeof(value) = 'object'\n left join lateral\n jsonb_array_elements(case jsonb_typeof(value) when 'array' then value end)\n with ordinality as a(arr_value, arr_key)\n on jsonb_typeof(value) = 'array'\n where obj_key is not null or arr_key is not null\n )\n , paths AS (\n select\n array_to_string(path, '.') as path,\n value\n FROM extract_all\n WHERE\n jsonb_typeof(value) NOT IN ('array','object')\n ), grouped AS (\n SELECT path, jsonb_agg(distinct value) vals FROM paths group by path\n ) SELECT coalesce(jsonb_object_agg(path, CASE WHEN jsonb_array_length(vals)=1 THEN vals->0 ELSE vals END) - '{datetime}'::text[], '{}'::jsonb) FROM grouped\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET JIT TO OFF;\n\nCREATE INDEX \"datetime_idx\" ON items (datetime);\nCREATE INDEX \"end_datetime_idx\" ON items (end_datetime);\nCREATE INDEX \"properties_idx\" ON items USING GIN (properties jsonb_path_ops);\nCREATE INDEX \"collection_idx\" ON items (collection_id);\nCREATE INDEX \"geometry_idx\" ON items USING GIST (geometry);\nCREATE UNIQUE INDEX \"items_id_datetime_idx\" ON items (datetime, id);\n\nALTER TABLE items ADD CONSTRAINT items_collections_fk FOREIGN KEY (collection_id) REFERENCES collections(id) ON DELETE CASCADE DEFERRABLE;\n\nCREATE OR REPLACE FUNCTION analyze_empty_partitions() RETURNS VOID AS $$\nDECLARE\n p text;\nBEGIN\n FOR p IN SELECT partition FROM all_items_partitions WHERE est_cnt = 0 LOOP\n EXECUTE format('ANALYZE %I;', p);\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION items_partition_name(timestamptz) RETURNS text AS $$\n SELECT to_char($1, '\"items_p\"IYYY\"w\"IW');\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION items_partition_exists(text) RETURNS boolean AS $$\n SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_class WHERE relname=$1);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION items_partition_exists(timestamptz) RETURNS boolean AS $$\n SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_class WHERE relname=items_partition_name($1));\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION items_partition_create_worker(partition text, partition_start timestamptz, partition_end timestamptz) RETURNS VOID AS $$\nDECLARE\n err_context text;\nBEGIN\n EXECUTE format(\n $f$\n CREATE TABLE IF NOT EXISTS %1$I PARTITION OF items\n FOR VALUES FROM (%2$L) TO (%3$L);\n CREATE UNIQUE INDEX IF NOT EXISTS %4$I ON %1$I (id);\n $f$,\n partition,\n partition_start,\n partition_end,\n concat(partition, '_id_pk')\n );\nEXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', partition;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH to pgstac, public;\n\nCREATE OR REPLACE FUNCTION items_partition_create(ts timestamptz) RETURNS text AS $$\nDECLARE\n partition text := items_partition_name(ts);\n partition_start timestamptz;\n partition_end timestamptz;\nBEGIN\n IF items_partition_exists(partition) THEN\n RETURN partition;\n END IF;\n partition_start := date_trunc('week', ts);\n partition_end := partition_start + '1 week'::interval;\n PERFORM items_partition_create_worker(partition, partition_start, partition_end);\n RAISE NOTICE 'partition: %', partition;\n RETURN partition;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION items_partition_create(st timestamptz, et timestamptz) RETURNS SETOF text AS $$\nWITH t AS (\n SELECT\n generate_series(\n date_trunc('week',st),\n date_trunc('week', et),\n '1 week'::interval\n ) w\n)\nSELECT items_partition_create(w) FROM t;\n$$ LANGUAGE SQL;\n\n\nCREATE UNLOGGED TABLE items_staging (\n content JSONB NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION items_staging_insert_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n mindate timestamptz;\n maxdate timestamptz;\n partition text;\nBEGIN\n SELECT min(stac_datetime(content)), max(stac_datetime(content)) INTO mindate, maxdate FROM newdata;\n PERFORM items_partition_create(mindate, maxdate);\n INSERT INTO items (id, geometry, collection_id, datetime, end_datetime, properties, content)\n SELECT\n content->>'id',\n stac_geom(content),\n content->>'collection',\n stac_datetime(content),\n stac_end_datetime(content),\n properties_idx(content),\n content\n FROM newdata\n ;\n DELETE FROM items_staging;\n PERFORM analyze_empty_partitions();\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER items_staging_insert_trigger AFTER INSERT ON items_staging REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_insert_triggerfunc();\n\n\nCREATE UNLOGGED TABLE items_staging_ignore (\n content JSONB NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION items_staging_ignore_insert_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n mindate timestamptz;\n maxdate timestamptz;\n partition text;\nBEGIN\n SELECT min(stac_datetime(content)), max(stac_datetime(content)) INTO mindate, maxdate FROM newdata;\n PERFORM items_partition_create(mindate, maxdate);\n INSERT INTO items (id, geometry, collection_id, datetime, end_datetime, properties, content)\n SELECT\n content->>'id',\n stac_geom(content),\n content->>'collection',\n stac_datetime(content),\n stac_end_datetime(content),\n properties_idx(content),\n content\n FROM newdata\n ON CONFLICT DO NOTHING\n ;\n DELETE FROM items_staging_ignore;\n PERFORM analyze_empty_partitions();\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER items_staging_ignore_insert_trigger AFTER INSERT ON items_staging_ignore REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_ignore_insert_triggerfunc();\n\nCREATE UNLOGGED TABLE items_staging_upsert (\n content JSONB NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION items_staging_upsert_insert_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n mindate timestamptz;\n maxdate timestamptz;\n partition text;\nBEGIN\n SELECT min(stac_datetime(content)), max(stac_datetime(content)) INTO mindate, maxdate FROM newdata;\n PERFORM items_partition_create(mindate, maxdate);\n INSERT INTO items (id, geometry, collection_id, datetime, end_datetime, properties, content)\n SELECT\n content->>'id',\n stac_geom(content),\n content->>'collection',\n stac_datetime(content),\n stac_end_datetime(content),\n properties_idx(content),\n content\n FROM newdata\n ON CONFLICT (datetime, id) DO UPDATE SET\n content = EXCLUDED.content\n WHERE items.content IS DISTINCT FROM EXCLUDED.content\n ;\n DELETE FROM items_staging_upsert;\n PERFORM analyze_empty_partitions();\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER items_staging_upsert_insert_trigger AFTER INSERT ON items_staging_upsert REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_upsert_insert_triggerfunc();\n\nCREATE OR REPLACE FUNCTION items_update_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\nBEGIN\n NEW.id := NEW.content->>'id';\n NEW.datetime := stac_datetime(NEW.content);\n NEW.end_datetime := stac_end_datetime(NEW.content);\n NEW.collection_id := NEW.content->>'collection';\n NEW.geometry := stac_geom(NEW.content);\n NEW.properties := properties_idx(NEW.content);\n IF TG_OP = 'UPDATE' AND NEW IS NOT DISTINCT FROM OLD THEN\n RETURN NULL;\n END IF;\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER items_update_trigger BEFORE UPDATE ON items\n FOR EACH ROW EXECUTE PROCEDURE items_update_triggerfunc();\n\n/*\nView to get a table of available items partitions\nwith date ranges\n*/\nCREATE VIEW all_items_partitions AS\nWITH base AS\n(SELECT\n c.oid::pg_catalog.regclass::text as partition,\n pg_catalog.pg_get_expr(c.relpartbound, c.oid) as _constraint,\n regexp_matches(\n pg_catalog.pg_get_expr(c.relpartbound, c.oid),\n E'\\\\(''\\([0-9 :+-]*\\)''\\\\).*\\\\(''\\([0-9 :+-]*\\)''\\\\)'\n ) as t,\n reltuples::bigint as est_cnt\nFROM pg_catalog.pg_class c, pg_catalog.pg_inherits i\nWHERE c.oid = i.inhrelid AND i.inhparent = 'items'::regclass)\nSELECT partition, tstzrange(\n t[1]::timestamptz,\n t[2]::timestamptz\n), est_cnt\nFROM base\nORDER BY 2 desc;\n\nCREATE OR REPLACE VIEW items_partitions AS\nSELECT * FROM all_items_partitions WHERE est_cnt>0;\n\nCREATE OR REPLACE FUNCTION get_item(_id text) RETURNS jsonb AS $$\n SELECT content FROM items WHERE id=_id;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION delete_item(_id text) RETURNS VOID AS $$\nDECLARE\nout items%ROWTYPE;\nBEGIN\n DELETE FROM items WHERE id = _id RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION create_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION update_item(data jsonb) RETURNS VOID AS $$\nDECLARE\n out items%ROWTYPE;\nBEGIN\n UPDATE items SET content=data WHERE id = data->>'id' RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION collection_bbox(id text) RETURNS jsonb AS $$\nSELECT (replace(replace(replace(st_extent(geometry)::text,'BOX(','[['),')',']]'),' ',','))::jsonb\nFROM items WHERE collection_id=$1;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION collection_temporal_extent(id text) RETURNS jsonb AS $$\nSELECT to_jsonb(array[array[min(datetime)::text, max(datetime)::text]])\nFROM items WHERE collection_id=$1;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection_extents() RETURNS VOID AS $$\nUPDATE collections SET\n content = content ||\n jsonb_build_object(\n 'extent', jsonb_build_object(\n 'spatial', jsonb_build_object(\n 'bbox', collection_bbox(collections.id)\n ),\n 'temporal', jsonb_build_object(\n 'interval', collection_temporal_extent(collections.id)\n )\n )\n )\n;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION items_path(\n IN dotpath text,\n OUT field text,\n OUT path text,\n OUT path_txt text,\n OUT jsonpath text,\n OUT eq text\n) RETURNS RECORD AS $$\nDECLARE\npath_elements text[];\nlast_element text;\nBEGIN\ndotpath := replace(trim(dotpath), 'properties.', '');\n\nIF dotpath = '' THEN\n RETURN;\nEND IF;\n\npath_elements := string_to_array(dotpath, '.');\njsonpath := NULL;\n\nIF path_elements[1] IN ('id','geometry','datetime') THEN\n field := path_elements[1];\n path_elements := path_elements[2:];\nELSIF path_elements[1] = 'collection' THEN\n field := 'collection_id';\n path_elements := path_elements[2:];\nELSIF path_elements[1] IN ('links', 'assets', 'stac_version', 'stac_extensions') THEN\n field := 'content';\nELSE\n field := 'content';\n path_elements := '{properties}'::text[] || path_elements;\nEND IF;\nIF cardinality(path_elements)<1 THEN\n path := field;\n path_txt := field;\n jsonpath := '$';\n eq := NULL; -- format($F$ %s = %%s $F$, field);\n RETURN;\nEND IF;\n\n\nlast_element := path_elements[cardinality(path_elements)];\npath_elements := path_elements[1:cardinality(path_elements)-1];\njsonpath := concat(array_to_string('{$}'::text[] || array_map_ident(path_elements), '.'), '.', quote_ident(last_element));\npath_elements := array_map_literal(path_elements);\npath := format($F$ properties->%s $F$, quote_literal(dotpath));\npath_txt := format($F$ properties->>%s $F$, quote_literal(dotpath));\neq := format($F$ properties @? '$.%s[*] ? (@ == %%s) '$F$, quote_ident(dotpath));\n\nRAISE NOTICE 'ITEMS PATH -- % % % % %', field, path, path_txt, jsonpath, eq;\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION parse_dtrange(IN _indate jsonb, OUT _tstzrange tstzrange) AS $$\nWITH t AS (\n SELECT CASE\n WHEN jsonb_typeof(_indate) = 'array' THEN\n textarr(_indate)\n ELSE\n regexp_split_to_array(\n btrim(_indate::text,'\"'),\n '/'\n )\n END AS arr\n)\n, t1 AS (\n SELECT\n CASE\n WHEN array_upper(arr,1) = 1 OR arr[1] = '..' OR arr[1] IS NULL THEN '-infinity'::timestamptz\n ELSE arr[1]::timestamptz\n END AS st,\n CASE\n WHEN array_upper(arr,1) = 1 THEN arr[1]::timestamptz\n WHEN arr[2] = '..' OR arr[2] IS NULL THEN 'infinity'::timestamptz\n ELSE arr[2]::timestamptz\n END AS et\n FROM t\n)\nSELECT\n tstzrange(st,et)\nFROM t1;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION bbox_geom(_bbox jsonb) RETURNS geometry AS $$\nSELECT CASE jsonb_array_length(_bbox)\n WHEN 4 THEN\n ST_SetSRID(ST_MakeEnvelope(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float,\n (_bbox->>3)::float\n ),4326)\n WHEN 6 THEN\n ST_SetSRID(ST_3DMakeBox(\n ST_MakePoint(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float\n ),\n ST_MakePoint(\n (_bbox->>3)::float,\n (_bbox->>4)::float,\n (_bbox->>5)::float\n )\n ),4326)\n ELSE null END;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION cql_and_append(existing jsonb, newfilters jsonb) RETURNS jsonb AS $$\nSELECT CASE WHEN existing ? 'filter' AND newfilters IS NOT NULL THEN\n jsonb_build_object(\n 'and',\n jsonb_build_array(\n existing->'filter',\n newfilters\n )\n )\nELSE\n newfilters\nEND;\n$$ LANGUAGE SQL;\n\n\n-- ADDs base filters (ids, collections, datetime, bbox, intersects) that are\n-- added outside of the filter/query in the stac request\nCREATE OR REPLACE FUNCTION add_filters_to_cql(j jsonb) RETURNS jsonb AS $$\nDECLARE\nnewprop jsonb;\nnewprops jsonb := '[]'::jsonb;\nBEGIN\nIF j ? 'id' THEN\n newprop := jsonb_build_object(\n 'in',\n jsonb_build_array(\n '{\"property\":\"id\"}'::jsonb,\n j->'id'\n )\n );\n newprops := jsonb_insert(newprops, '{1}', newprop);\nEND IF;\nIF j ? 'collection' THEN\n newprop := jsonb_build_object(\n 'in',\n jsonb_build_array(\n '{\"property\":\"collection\"}'::jsonb,\n j->'collection'\n )\n );\n newprops := jsonb_insert(newprops, '{1}', newprop);\nEND IF;\n\nIF j ? 'datetime' THEN\n newprop := format(\n '{\"anyinteracts\":[{\"property\":\"datetime\"}, %s]}',\n j->'datetime'\n );\n newprops := jsonb_insert(newprops, '{1}', newprop);\nEND IF;\n\nIF j ? 'bbox' THEN\n newprop := format(\n '{\"intersects\":[{\"property\":\"geometry\"}, %s]}',\n j->'bbox'\n );\n newprops := jsonb_insert(newprops, '{1}', newprop);\nEND IF;\n\nIF j ? 'intersects' THEN\n newprop := format(\n '{\"intersects\":[{\"property\":\"geometry\"}, %s]}',\n j->'intersects'\n );\n newprops := jsonb_insert(newprops, '{1}', newprop);\nEND IF;\n\nRAISE NOTICE 'newprops: %', newprops;\n\nIF newprops IS NOT NULL AND jsonb_array_length(newprops) > 0 THEN\n return jsonb_set(\n j,\n '{filter}',\n cql_and_append(j, jsonb_build_object('and', newprops))\n ) - '{id,collection,datetime,bbox,intersects}'::text[];\nEND IF;\n\nreturn j;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION query_to_cqlfilter(j jsonb) RETURNS jsonb AS $$\n-- Translates anything passed in through the deprecated \"query\" into equivalent CQL\nWITH t AS (\n SELECT key as property, value as ops\n FROM jsonb_each(j->'query')\n), t2 AS (\n SELECT property, (jsonb_each(ops)).*\n FROM t WHERE jsonb_typeof(ops) = 'object'\n UNION ALL\n SELECT property, 'eq', ops\n FROM t WHERE jsonb_typeof(ops) != 'object'\n), t3 AS (\nSELECT\n jsonb_strip_nulls(jsonb_build_object(\n 'and',\n jsonb_agg(\n jsonb_build_object(\n key,\n jsonb_build_array(\n jsonb_build_object('property',property),\n value\n )\n )\n )\n )) as qcql FROM t2\n)\nSELECT\n CASE WHEN qcql IS NOT NULL THEN\n jsonb_set(j, '{filter}', cql_and_append(j, qcql)) - 'query'\n ELSE j\n END\nFROM t3\n;\n$$ LANGUAGE SQL;\n\n\n\nCREATE OR REPLACE FUNCTION temporal_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\nll text := 'datetime';\nlh text := 'end_datetime';\nrrange tstzrange;\nrl text;\nrh text;\noutq text;\nBEGIN\nrrange := parse_dtrange(args->1);\nRAISE NOTICE 'Constructing temporal query OP: %, ARGS: %, RRANGE: %', op, args, rrange;\nop := lower(op);\nrl := format('%L::timestamptz', lower(rrange));\nrh := format('%L::timestamptz', upper(rrange));\noutq := CASE op\n WHEN 't_before' THEN 'lh < rl'\n WHEN 't_after' THEN 'll > rh'\n WHEN 't_meets' THEN 'lh = rl'\n WHEN 't_metby' THEN 'll = rh'\n WHEN 't_overlaps' THEN 'll < rl AND rl < lh < rh'\n WHEN 't_overlappedby' THEN 'rl < ll < rh AND lh > rh'\n WHEN 't_starts' THEN 'll = rl AND lh < rh'\n WHEN 't_startedby' THEN 'll = rl AND lh > rh'\n WHEN 't_during' THEN 'll > rl AND lh < rh'\n WHEN 't_contains' THEN 'll < rl AND lh > rh'\n WHEN 't_finishes' THEN 'll > rl AND lh = rh'\n WHEN 't_finishedby' THEN 'll < rl AND lh = rh'\n WHEN 't_equals' THEN 'll = rl AND lh = rh'\n WHEN 't_disjoint' THEN 'NOT (ll <= rh AND lh >= rl)'\n WHEN 't_intersects' THEN 'll <= rh AND lh >= rl'\n WHEN 'anyinteracts' THEN 'll <= rh AND lh >= rl'\nEND;\noutq := regexp_replace(outq, '\\mll\\M', ll);\noutq := regexp_replace(outq, '\\mlh\\M', lh);\noutq := regexp_replace(outq, '\\mrl\\M', rl);\noutq := regexp_replace(outq, '\\mrh\\M', rh);\noutq := format('(%s)', outq);\nRETURN outq;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION spatial_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\ngeom text;\nj jsonb := args->1;\nBEGIN\nop := lower(op);\nRAISE NOTICE 'Constructing spatial query OP: %, ARGS: %', op, args;\nIF op NOT IN ('s_equals','s_disjoint','s_touches','s_within','s_overlaps','s_crosses','s_intersects','intersects','s_contains') THEN\n RAISE EXCEPTION 'Spatial Operator % Not Supported', op;\nEND IF;\nop := regexp_replace(op, '^s_', 'st_');\nIF op = 'intersects' THEN\n op := 'st_intersects';\nEND IF;\n-- Convert geometry to WKB string\nIF j ? 'type' AND j ? 'coordinates' THEN\n geom := st_geomfromgeojson(j)::text;\nELSIF jsonb_typeof(j) = 'array' THEN\n geom := bbox_geom(j)::text;\nEND IF;\n\nRETURN format('%s(geometry, %L::geometry)', op, geom);\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n/* cql_query_op -- Parses a CQL query operation, recursing when necessary\n IN jsonb -- a subelement from a valid stac query\n IN text -- the operator being used on elements passed in\n RETURNS a SQL fragment to be used in a WHERE clause\n*/\nCREATE OR REPLACE FUNCTION cql_query_op(j jsonb, _op text DEFAULT NULL) RETURNS text AS $$\nDECLARE\njtype text := jsonb_typeof(j);\nop text := lower(_op);\nops jsonb :=\n '{\n \"eq\": \"%s = %s\",\n \"lt\": \"%s < %s\",\n \"lte\": \"%s <= %s\",\n \"gt\": \"%s > %s\",\n \"gte\": \"%s >= %s\",\n \"like\": \"%s LIKE %s\",\n \"+\": \"%s + %s\",\n \"-\": \"%s - %s\",\n \"*\": \"%s * %s\",\n \"/\": \"%s / %s\",\n \"in\": \"%s = ANY (%s)\",\n \"not\": \"NOT (%s)\",\n \"between\": \"%s BETWEEN %s AND %s\",\n \"lower\":\"lower(%s)\"\n }'::jsonb;\nret text;\nargs text[] := NULL;\n\nBEGIN\nRAISE NOTICE 'j: %, op: %, jtype: %', j, op, jtype;\n\n-- Set Lower Case on Both Arguments When Case Insensitive Flag Set\nIF op in ('eq','lt','lte','gt','gte','like') AND jsonb_typeof(j->2) = 'boolean' THEN\n IF (j->>2)::boolean THEN\n RETURN format(concat('(',ops->>op,')'), cql_query_op(jsonb_build_array(j->0), 'lower'), cql_query_op(jsonb_build_array(j->1), 'lower'));\n END IF;\nEND IF;\n\n-- Special Case when comparing a property in a jsonb field to a string or number using eq\n-- Allows to leverage GIN index on jsonb fields\nIF op = 'eq' THEN\n IF j->0 ? 'property'\n AND jsonb_typeof(j->1) IN ('number','string')\n AND (items_path(j->0->>'property')).eq IS NOT NULL\n THEN\n RETURN format((items_path(j->0->>'property')).eq, j->1);\n END IF;\nEND IF;\n\nIF op ilike 't_%' or op = 'anyinteracts' THEN\n RETURN temporal_op_query(op, j);\nEND IF;\n\nIF op ilike 's_%' or op = 'intersects' THEN\n RETURN spatial_op_query(op, j);\nEND IF;\n\n\nIF jtype = 'object' THEN\n RAISE NOTICE 'parsing object';\n IF j ? 'property' THEN\n -- Convert the property to be used as an identifier\n return (items_path(j->>'property')).path_txt;\n ELSIF _op IS NULL THEN\n -- Iterate to convert elements in an object where the operator has not been set\n -- Combining with AND\n SELECT\n array_to_string(array_agg(cql_query_op(e.value, e.key)), ' AND ')\n INTO ret\n FROM jsonb_each(j) e;\n RETURN ret;\n END IF;\nEND IF;\n\nIF jtype = 'string' THEN\n RETURN quote_literal(j->>0);\nEND IF;\n\nIF jtype ='number' THEN\n RETURN (j->>0)::numeric;\nEND IF;\n\nIF jtype = 'array' AND op IS NULL THEN\n RAISE NOTICE 'Parsing array into array arg. j: %', j;\n SELECT format($f$ '{%s}'::text[] $f$, string_agg(e,',')) INTO ret FROM jsonb_array_elements_text(j) e;\n RETURN ret;\nEND IF;\n\n\n-- If the type of the passed json is an array\n-- Calculate the arguments that will be passed to functions/operators\nIF jtype = 'array' THEN\n RAISE NOTICE 'Parsing array into args. j: %', j;\n -- If any argument is numeric, cast any text arguments to numeric\n IF j @? '$[*] ? (@.type() == \"number\")' THEN\n SELECT INTO args\n array_agg(concat('(',cql_query_op(e),')::numeric'))\n FROM jsonb_array_elements(j) e;\n ELSE\n SELECT INTO args\n array_agg(cql_query_op(e))\n FROM jsonb_array_elements(j) e;\n END IF;\n --RETURN args;\nEND IF;\nRAISE NOTICE 'ARGS after array cleaning: %', args;\n\nIF op IS NULL THEN\n RETURN args::text[];\nEND IF;\n\nIF args IS NULL OR cardinality(args) < 1 THEN\n RAISE NOTICE 'No Args';\n RETURN '';\nEND IF;\n\nIF op IN ('and','or') THEN\n SELECT\n CONCAT(\n '(',\n array_to_string(args, UPPER(CONCAT(' ',op,' '))),\n ')'\n ) INTO ret\n FROM jsonb_array_elements(j) e;\n RETURN ret;\nEND IF;\n\n-- If the op is in the ops json then run using the template in the json\nIF ops ? op THEN\n RAISE NOTICE 'ARGS: % MAPPED: %',args, array_map_literal(args);\n\n RETURN format(concat('(',ops->>op,')'), VARIADIC args);\nEND IF;\n\nRETURN j->>0;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n\n\nCREATE OR REPLACE FUNCTION cql_to_where(_search jsonb = '{}'::jsonb) RETURNS text AS $$\nDECLARE\nsearch jsonb := _search;\n_where text;\nBEGIN\nRAISE NOTICE 'SEARCH CQL 1: %', search;\n\n-- Convert any old style stac query to cql\nsearch := query_to_cqlfilter(search);\n\nRAISE NOTICE 'SEARCH CQL 2: %', search;\n\n-- Convert item,collection,datetime,bbox,intersects to cql\nsearch := add_filters_to_cql(search);\n\nRAISE NOTICE 'SEARCH CQL Final: %', search;\n_where := cql_query_op(search->'filter');\n\nIF trim(_where) = '' THEN\n _where := NULL;\nEND IF;\n_where := coalesce(_where, ' TRUE ');\nRETURN _where;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION parse_sort_dir(_dir text, reverse boolean default false) RETURNS text AS $$\nWITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n) SELECT\n CASE\n WHEN NOT reverse THEN d\n WHEN d = 'ASC' THEN 'DESC'\n WHEN d = 'DESC' THEN 'ASC'\n END\nFROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION sort_dir_to_op(_dir text, prev boolean default false) RETURNS text AS $$\nWITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n) SELECT\n CASE\n WHEN d = 'ASC' AND prev THEN '<='\n WHEN d = 'DESC' AND prev THEN '>='\n WHEN d = 'ASC' THEN '>='\n WHEN d = 'DESC' THEN '<='\n END\nFROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION sort_sqlorderby(\n _search jsonb DEFAULT NULL,\n reverse boolean DEFAULT FALSE\n) RETURNS text AS $$\nWITH sorts AS (\n SELECT\n (items_path(value->>'field')).path as key,\n parse_sort_dir(value->>'direction', reverse) as dir\n FROM jsonb_array_elements(\n '[]'::jsonb\n ||\n coalesce(_search->'sortby','[{\"field\":\"datetime\", \"direction\":\"desc\"}]')\n ||\n '[{\"field\":\"id\",\"direction\":\"desc\"}]'::jsonb\n )\n)\nSELECT array_to_string(\n array_agg(concat(key, ' ', dir)),\n ', '\n) FROM sorts;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_sort_dir(sort_item jsonb) RETURNS text AS $$\nSELECT CASE WHEN sort_item->>'direction' ILIKE 'desc%' THEN 'DESC' ELSE 'ASC' END;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION get_token_filter(_search jsonb = '{}'::jsonb, token_rec jsonb DEFAULT NULL) RETURNS text AS $$\nDECLARE\ntoken_id text;\nfilters text[] := '{}'::text[];\nprev boolean := TRUE;\nfield text;\ndir text;\nsort record;\norfilters text[] := '{}'::text[];\nandfilters text[] := '{}'::text[];\noutput text;\ntoken_where text;\nBEGIN\n-- If no token provided return NULL\nIF token_rec IS NULL THEN\n IF NOT (_search ? 'token' AND\n (\n (_search->>'token' ILIKE 'prev:%')\n OR\n (_search->>'token' ILIKE 'next:%')\n )\n ) THEN\n RETURN NULL;\n END IF;\n prev := (_search->>'token' ILIKE 'prev:%');\n token_id := substr(_search->>'token', 6);\n SELECT to_jsonb(items) INTO token_rec FROM items WHERE id=token_id;\nEND IF;\nRAISE NOTICE 'TOKEN ID: %', token_rec->'id';\n\nCREATE TEMP TABLE sorts (\n _row int GENERATED ALWAYS AS IDENTITY NOT NULL,\n _field text PRIMARY KEY,\n _dir text NOT NULL,\n _val text\n) ON COMMIT DROP;\n\n-- Make sure we only have distinct columns to sort with taking the first one we get\nINSERT INTO sorts (_field, _dir)\n SELECT\n (items_path(value->>'field')).path,\n get_sort_dir(value)\n FROM\n jsonb_array_elements(coalesce(_search->'sort','[{\"field\":\"datetime\",\"direction\":\"desc\"}]'))\nON CONFLICT DO NOTHING\n;\n\n-- Get the first sort direction provided. As the id is a primary key, if there are any\n-- sorts after id they won't do anything, so make sure that id is the last sort item.\nSELECT _dir INTO dir FROM sorts ORDER BY _row ASC LIMIT 1;\nIF EXISTS (SELECT 1 FROM sorts WHERE _field = 'id') THEN\n DELETE FROM sorts WHERE _row > (SELECT _row FROM sorts WHERE _field = 'id');\nELSE\n INSERT INTO sorts (_field, _dir) VALUES ('id', dir);\nEND IF;\n\n-- Add value from looked up item to the sorts table\nUPDATE sorts SET _val=quote_literal(token_rec->>_field);\n\n-- Check if all sorts are the same direction and use row comparison\n-- to filter\nIF (SELECT count(DISTINCT _dir) FROM sorts) = 1 THEN\n SELECT format(\n '(%s) %s (%s)',\n concat_ws(', ', VARIADIC array_agg(quote_ident(_field))),\n CASE WHEN (prev AND dir = 'ASC') OR (NOT prev AND dir = 'DESC') THEN '<' ELSE '>' END,\n concat_ws(', ', VARIADIC array_agg(_val))\n ) INTO output FROM sorts\n WHERE token_rec ? _field\n ;\nELSE\n FOR sort IN SELECT * FROM sorts ORDER BY _row asc LOOP\n RAISE NOTICE 'SORT: %', sort;\n IF sort._row = 1 THEN\n orfilters := orfilters || format('(%s %s %s)',\n quote_ident(sort._field),\n CASE WHEN (prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC') THEN '<' ELSE '>' END,\n sort._val\n );\n ELSE\n orfilters := orfilters || format('(%s AND %s %s %s)',\n array_to_string(andfilters, ' AND '),\n quote_ident(sort._field),\n CASE WHEN (prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC') THEN '<' ELSE '>' END,\n sort._val\n );\n\n END IF;\n andfilters := andfilters || format('%s = %s',\n quote_ident(sort._field),\n sort._val\n );\n END LOOP;\n output := array_to_string(orfilters, ' OR ');\nEND IF;\nDROP TABLE IF EXISTS sorts;\ntoken_where := concat('(',coalesce(output,'true'),')');\nIF trim(token_where) = '' THEN\n token_where := NULL;\nEND IF;\nRAISE NOTICE 'TOKEN_WHERE: |%|',token_where;\nRETURN token_where;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION search_tohash(jsonb) RETURNS jsonb AS $$\n SELECT $1 - '{token,limit,context,includes,excludes}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION search_hash(jsonb) RETURNS text AS $$\n SELECT md5(search_tohash($1)::text);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE TABLE IF NOT EXISTS searches(\n hash text GENERATED ALWAYS AS (search_hash(search)) STORED PRIMARY KEY,\n search jsonb NOT NULL,\n _where text,\n orderby text,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n statslastupdated timestamptz,\n estimated_count bigint,\n total_count bigint\n);\n\nCREATE OR REPLACE FUNCTION search_query(_search jsonb = '{}'::jsonb, updatestats boolean DEFAULT false) RETURNS searches AS $$\nDECLARE\n search searches%ROWTYPE;\nBEGIN\nINSERT INTO searches (search)\n VALUES (search_tohash(_search))\n ON CONFLICT DO NOTHING\n RETURNING * INTO search;\nIF search.hash IS NULL THEN\n SELECT * INTO search FROM searches WHERE hash=search_hash(_search);\nEND IF;\nIF search._where IS NULL THEN\n search._where := cql_to_where(_search);\nEND IF;\nIF search.orderby IS NULL THEN\n search.orderby := sort_sqlorderby(_search);\nEND IF;\n\nIF search.statslastupdated IS NULL OR age(search.statslastupdated) > '1 day'::interval OR (_search ? 'context' AND search.total_count IS NULL) THEN\n updatestats := TRUE;\nEND IF;\n\nIF updatestats THEN\n -- Get Estimated Stats\n RAISE NOTICE 'Getting stats for %', search._where;\n search.estimated_count := estimated_count(search._where);\n RAISE NOTICE 'Estimated Count: %', search.estimated_count;\n\n IF _search ? 'context' OR search.estimated_count < 10000 THEN\n --search.total_count := partition_count(search._where);\n EXECUTE format(\n 'SELECT count(*) FROM items WHERE %s',\n search._where\n ) INTO search.total_count;\n RAISE NOTICE 'Actual Count: %', search.total_count;\n ELSE\n search.total_count := NULL;\n END IF;\n search.statslastupdated := now();\nEND IF;\n\nsearch.lastused := now();\nsearch.usecount := coalesce(search.usecount,0) + 1;\nRAISE NOTICE 'SEARCH: %', search;\nUPDATE searches SET\n _where = search._where,\n orderby = search.orderby,\n lastused = search.lastused,\n usecount = search.usecount,\n statslastupdated = search.statslastupdated,\n estimated_count = search.estimated_count,\n total_count = search.total_count\nWHERE hash = search.hash\n;\nRETURN search;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n\nCREATE OR REPLACE FUNCTION search(_search jsonb = '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n searches searches%ROWTYPE;\n _where text;\n token_where text;\n full_where text;\n orderby text;\n query text;\n token_type text := substr(_search->>'token',1,4);\n _limit int := coalesce((_search->>'limit')::int, 10);\n curs refcursor;\n cntr int := 0;\n iter_record items%ROWTYPE;\n first_record items%ROWTYPE;\n last_record items%ROWTYPE;\n out_records jsonb := '[]'::jsonb;\n prev_query text;\n next text;\n prev_id text;\n has_next boolean := false;\n has_prev boolean := false;\n prev text;\n total_count bigint;\n context jsonb;\n collection jsonb;\n includes text[];\n excludes text[];\n exit_flag boolean := FALSE;\n batches int := 0;\n timer timestamptz := clock_timestamp();\nBEGIN\nsearches := search_query(_search);\n_where := searches._where;\norderby := searches.orderby;\ntotal_count := coalesce(searches.total_count, searches.estimated_count);\n\n\nIF token_type='prev' THEN\n token_where := get_token_filter(_search, null::jsonb);\n orderby := sort_sqlorderby(_search, TRUE);\nEND IF;\nIF token_type='next' THEN\n token_where := get_token_filter(_search, null::jsonb);\nEND IF;\n\nfull_where := concat_ws(' AND ', _where, token_where);\nRAISE NOTICE 'FULL QUERY % %', full_where, clock_timestamp()-timer;\ntimer := clock_timestamp();\n\nFOR query IN SELECT partition_queries(full_where, orderby) LOOP\n timer := clock_timestamp();\n query := format('%s LIMIT %L', query, _limit + 1);\n RAISE NOTICE 'Partition Query: %', query;\n batches := batches + 1;\n curs = create_cursor(query);\n LOOP\n FETCH curs into iter_record;\n EXIT WHEN NOT FOUND;\n cntr := cntr + 1;\n last_record := iter_record;\n IF cntr = 1 THEN\n first_record := last_record;\n END IF;\n IF cntr <= _limit THEN\n out_records := out_records || last_record.content;\n ELSIF cntr > _limit THEN\n has_next := true;\n exit_flag := true;\n EXIT;\n END IF;\n END LOOP;\n RAISE NOTICE 'Query took %', clock_timestamp()-timer;\n timer := clock_timestamp();\n EXIT WHEN exit_flag;\nEND LOOP;\nRAISE NOTICE 'Scanned through % partitions.', batches;\n\n\n-- Flip things around if this was the result of a prev token query\nIF token_type='prev' THEN\n out_records := flip_jsonb_array(out_records);\n first_record := last_record;\nEND IF;\n\n-- If this query has a token, see if there is data before the first record\nIF _search ? 'token' THEN\n prev_query := format(\n 'SELECT 1 FROM items WHERE %s LIMIT 1',\n concat_ws(\n ' AND ',\n _where,\n trim(get_token_filter(_search, to_jsonb(first_record)))\n )\n );\n RAISE NOTICE 'Query to get previous record: % --- %', prev_query, first_record;\n EXECUTE prev_query INTO has_prev;\n IF FOUND and has_prev IS NOT NULL THEN\n RAISE NOTICE 'Query results from prev query: %', has_prev;\n has_prev := TRUE;\n END IF;\nEND IF;\nhas_prev := COALESCE(has_prev, FALSE);\n\nRAISE NOTICE 'token_type: %, has_next: %, has_prev: %', token_type, has_next, has_prev;\nIF has_prev THEN\n prev := out_records->0->>'id';\nEND IF;\nIF has_next OR token_type='prev' THEN\n next := out_records->-1->>'id';\nEND IF;\n\n\n\n-- include/exclude any fields following fields extension\nIF _search ? 'fields' THEN\n IF _search->'fields' ? 'exclude' THEN\n excludes=textarr(_search->'fields'->'exclude');\n END IF;\n IF _search->'fields' ? 'include' THEN\n includes=textarr(_search->'fields'->'include');\n IF array_length(includes, 1)>0 AND NOT 'id' = ANY (includes) THEN\n includes = includes || '{id}';\n END IF;\n END IF;\n SELECT jsonb_agg(filter_jsonb(row, includes, excludes)) INTO out_records FROM jsonb_array_elements(out_records) row;\nEND IF;\n\n\ncontext := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'matched', total_count,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n));\n\ncollection := jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', out_records,\n 'next', next,\n 'prev', prev,\n 'context', context\n);\n\nRETURN collection;\nEND;\n$$ LANGUAGE PLPGSQL SET jit TO off;\nINSERT INTO pgstac.migrations (version) VALUES ('0.3.1');\n" }, { "path": "src/pgstac/migrations/pgstac.0.3.2-0.3.3.sql", "content": "SET SEARCH_PATH to pgstac, public;\nset check_function_bodies = off;\n\nCREATE OR REPLACE FUNCTION pgstac.add_filters_to_cql(j jsonb)\n RETURNS jsonb\n LANGUAGE plpgsql\nAS $function$\nDECLARE\nnewprop jsonb;\nnewprops jsonb := '[]'::jsonb;\nBEGIN\nIF j ? 'ids' THEN\n newprop := jsonb_build_object(\n 'in',\n jsonb_build_array(\n '{\"property\":\"id\"}'::jsonb,\n j->'ids'\n )\n );\n newprops := jsonb_insert(newprops, '{1}', newprop);\nEND IF;\nIF j ? 'collections' THEN\n newprop := jsonb_build_object(\n 'in',\n jsonb_build_array(\n '{\"property\":\"collection\"}'::jsonb,\n j->'collections'\n )\n );\n newprops := jsonb_insert(newprops, '{1}', newprop);\nEND IF;\n\nIF j ? 'datetime' THEN\n newprop := format(\n '{\"anyinteracts\":[{\"property\":\"datetime\"}, %s]}',\n j->'datetime'\n );\n newprops := jsonb_insert(newprops, '{1}', newprop);\nEND IF;\n\nIF j ? 'bbox' THEN\n newprop := format(\n '{\"intersects\":[{\"property\":\"geometry\"}, %s]}',\n j->'bbox'\n );\n newprops := jsonb_insert(newprops, '{1}', newprop);\nEND IF;\n\nIF j ? 'intersects' THEN\n newprop := format(\n '{\"intersects\":[{\"property\":\"geometry\"}, %s]}',\n j->'intersects'\n );\n newprops := jsonb_insert(newprops, '{1}', newprop);\nEND IF;\n\nRAISE NOTICE 'newprops: %', newprops;\n\nIF newprops IS NOT NULL AND jsonb_array_length(newprops) > 0 THEN\n return jsonb_set(\n j,\n '{filter}',\n cql_and_append(j, jsonb_build_object('and', newprops))\n ) - '{ids,collections,datetime,bbox,intersects}'::text[];\nEND IF;\n\nreturn j;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.get_token_filter(_search jsonb DEFAULT '{}'::jsonb, token_rec jsonb DEFAULT NULL::jsonb)\n RETURNS text\n LANGUAGE plpgsql\nAS $function$\nDECLARE\n token_id text;\n filters text[] := '{}'::text[];\n prev boolean := TRUE;\n field text;\n dir text;\n sort record;\n orfilters text[] := '{}'::text[];\n andfilters text[] := '{}'::text[];\n output text;\n token_where text;\nBEGIN\n -- If no token provided return NULL\n IF token_rec IS NULL THEN\n IF NOT (_search ? 'token' AND\n (\n (_search->>'token' ILIKE 'prev:%')\n OR\n (_search->>'token' ILIKE 'next:%')\n )\n ) THEN\n RETURN NULL;\n END IF;\n prev := (_search->>'token' ILIKE 'prev:%');\n token_id := substr(_search->>'token', 6);\n SELECT to_jsonb(items) INTO token_rec FROM items WHERE id=token_id;\n END IF;\n RAISE NOTICE 'TOKEN ID: %', token_rec->'id';\n\n CREATE TEMP TABLE sorts (\n _row int GENERATED ALWAYS AS IDENTITY NOT NULL,\n _field text PRIMARY KEY,\n _dir text NOT NULL,\n _val text\n ) ON COMMIT DROP;\n\n -- Make sure we only have distinct columns to sort with taking the first one we get\n INSERT INTO sorts (_field, _dir)\n SELECT\n (items_path(value->>'field')).path,\n get_sort_dir(value)\n FROM\n jsonb_array_elements(coalesce(_search->'sortby','[{\"field\":\"datetime\",\"direction\":\"desc\"}]'))\n ON CONFLICT DO NOTHING\n ;\n RAISE NOTICE 'sorts 1: %', (SELECT jsonb_agg(to_json(sorts)) FROM sorts);\n -- Get the first sort direction provided. As the id is a primary key, if there are any\n -- sorts after id they won't do anything, so make sure that id is the last sort item.\n SELECT _dir INTO dir FROM sorts ORDER BY _row ASC LIMIT 1;\n IF EXISTS (SELECT 1 FROM sorts WHERE _field = 'id') THEN\n DELETE FROM sorts WHERE _row > (SELECT _row FROM sorts WHERE _field = 'id' ORDER BY _row ASC);\n ELSE\n INSERT INTO sorts (_field, _dir) VALUES ('id', dir);\n END IF;\n\n -- Add value from looked up item to the sorts table\n UPDATE sorts SET _val=quote_literal(token_rec->>_field);\n\n -- Check if all sorts are the same direction and use row comparison\n -- to filter\n RAISE NOTICE 'sorts 2: %', (SELECT jsonb_agg(to_json(sorts)) FROM sorts);\n\n IF (SELECT count(DISTINCT _dir) FROM sorts) = 1 THEN\n SELECT format(\n '(%s) %s (%s)',\n concat_ws(', ', VARIADIC array_agg(quote_ident(_field))),\n CASE WHEN (prev AND dir = 'ASC') OR (NOT prev AND dir = 'DESC') THEN '<' ELSE '>' END,\n concat_ws(', ', VARIADIC array_agg(_val))\n ) INTO output FROM sorts\n WHERE token_rec ? _field\n ;\n ELSE\n FOR sort IN SELECT * FROM sorts ORDER BY _row asc LOOP\n RAISE NOTICE 'SORT: %', sort;\n IF sort._row = 1 THEN\n orfilters := orfilters || format('(%s %s %s)',\n quote_ident(sort._field),\n CASE WHEN (prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC') THEN '<' ELSE '>' END,\n sort._val\n );\n ELSE\n orfilters := orfilters || format('(%s AND %s %s %s)',\n array_to_string(andfilters, ' AND '),\n quote_ident(sort._field),\n CASE WHEN (prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC') THEN '<' ELSE '>' END,\n sort._val\n );\n\n END IF;\n andfilters := andfilters || format('%s = %s',\n quote_ident(sort._field),\n sort._val\n );\n END LOOP;\n output := array_to_string(orfilters, ' OR ');\n END IF;\n DROP TABLE IF EXISTS sorts;\n token_where := concat('(',coalesce(output,'true'),')');\n IF trim(token_where) = '' THEN\n token_where := NULL;\n END IF;\n RAISE NOTICE 'TOKEN_WHERE: |%|',token_where;\n RETURN token_where;\n END;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.search(_search jsonb DEFAULT '{}'::jsonb)\n RETURNS jsonb\n LANGUAGE plpgsql\n SET jit TO 'off'\nAS $function$\nDECLARE\n searches searches%ROWTYPE;\n _where text;\n token_where text;\n full_where text;\n orderby text;\n query text;\n token_type text := substr(_search->>'token',1,4);\n _limit int := coalesce((_search->>'limit')::int, 10);\n curs refcursor;\n cntr int := 0;\n iter_record items%ROWTYPE;\n first_record items%ROWTYPE;\n last_record items%ROWTYPE;\n out_records jsonb := '[]'::jsonb;\n prev_query text;\n next text;\n prev_id text;\n has_next boolean := false;\n has_prev boolean := false;\n prev text;\n total_count bigint;\n context jsonb;\n collection jsonb;\n includes text[];\n excludes text[];\n exit_flag boolean := FALSE;\n batches int := 0;\n timer timestamptz := clock_timestamp();\nBEGIN\nsearches := search_query(_search);\n_where := searches._where;\norderby := searches.orderby;\ntotal_count := coalesce(searches.total_count, searches.estimated_count);\n\n\nIF token_type='prev' THEN\n token_where := get_token_filter(_search, null::jsonb);\n orderby := sort_sqlorderby(_search, TRUE);\nEND IF;\nIF token_type='next' THEN\n token_where := get_token_filter(_search, null::jsonb);\nEND IF;\n\nfull_where := concat_ws(' AND ', _where, token_where);\nRAISE NOTICE 'FULL QUERY % %', full_where, clock_timestamp()-timer;\ntimer := clock_timestamp();\n\nFOR query IN SELECT partition_queries(full_where, orderby) LOOP\n timer := clock_timestamp();\n query := format('%s LIMIT %L', query, _limit + 1);\n RAISE NOTICE 'Partition Query: %', query;\n batches := batches + 1;\n curs = create_cursor(query);\n LOOP\n FETCH curs into iter_record;\n EXIT WHEN NOT FOUND;\n cntr := cntr + 1;\n last_record := iter_record;\n IF cntr = 1 THEN\n first_record := last_record;\n END IF;\n IF cntr <= _limit THEN\n out_records := out_records || last_record.content;\n ELSIF cntr > _limit THEN\n has_next := true;\n exit_flag := true;\n EXIT;\n END IF;\n END LOOP;\n RAISE NOTICE 'Query took %', clock_timestamp()-timer;\n timer := clock_timestamp();\n EXIT WHEN exit_flag;\nEND LOOP;\nRAISE NOTICE 'Scanned through % partitions.', batches;\n\n\n-- Flip things around if this was the result of a prev token query\nIF token_type='prev' THEN\n out_records := flip_jsonb_array(out_records);\n first_record := last_record;\nEND IF;\n\n-- If this query has a token, see if there is data before the first record\nIF _search ? 'token' THEN\n prev_query := format(\n 'SELECT 1 FROM items WHERE %s LIMIT 1',\n concat_ws(\n ' AND ',\n _where,\n trim(get_token_filter(_search, to_jsonb(first_record)))\n )\n );\n RAISE NOTICE 'Query to get previous record: % --- %', prev_query, first_record;\n EXECUTE prev_query INTO has_prev;\n IF FOUND and has_prev IS NOT NULL THEN\n RAISE NOTICE 'Query results from prev query: %', has_prev;\n has_prev := TRUE;\n END IF;\nEND IF;\nhas_prev := COALESCE(has_prev, FALSE);\n\nRAISE NOTICE 'token_type: %, has_next: %, has_prev: %', token_type, has_next, has_prev;\nIF has_prev THEN\n prev := out_records->0->>'id';\nEND IF;\nIF has_next OR token_type='prev' THEN\n next := out_records->-1->>'id';\nEND IF;\n\n\n-- include/exclude any fields following fields extension\nIF _search ? 'fields' THEN\n IF _search->'fields' ? 'exclude' THEN\n excludes=textarr(_search->'fields'->'exclude');\n END IF;\n IF _search->'fields' ? 'include' THEN\n includes=textarr(_search->'fields'->'include');\n IF array_length(includes, 1)>0 AND NOT 'id' = ANY (includes) THEN\n includes = includes || '{id}';\n END IF;\n END IF;\n SELECT jsonb_agg(filter_jsonb(row, includes, excludes)) INTO out_records FROM jsonb_array_elements(out_records) row;\nEND IF;\n\ncontext := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'matched', total_count,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n));\n\ncollection := jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb),\n 'next', next,\n 'prev', prev,\n 'context', context\n);\n\nRETURN collection;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.sort_sqlorderby(_search jsonb DEFAULT NULL::jsonb, reverse boolean DEFAULT false)\n RETURNS text\n LANGUAGE sql\nAS $function$\nWITH sortby AS (\n SELECT coalesce(_search->'sortby','[{\"field\":\"datetime\", \"direction\":\"desc\"}]') as sort\n), withid AS (\n SELECT CASE\n WHEN sort @? '$[*] ? (@.field == \"id\")' THEN sort\n ELSE sort || '[{\"field\":\"id\", \"direction\":\"desc\"}]'::jsonb\n END as sort\n FROM sortby\n), withid_rows AS (\n SELECT jsonb_array_elements(sort) as value FROM withid\n),sorts AS (\n SELECT\n (items_path(value->>'field')).path as key,\n parse_sort_dir(value->>'direction', reverse) as dir\n FROM withid_rows\n)\nSELECT array_to_string(\n array_agg(concat(key, ' ', dir)),\n ', '\n) FROM sorts;\n$function$\n;\n\n\n\nINSERT INTO migrations (version) VALUES ('0.3.3');\n" }, { "path": "src/pgstac/migrations/pgstac.0.3.2.sql", "content": "CREATE EXTENSION IF NOT EXISTS postgis;\nCREATE SCHEMA IF NOT EXISTS pgstac;\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE TABLE migrations (\n version text,\n datetime timestamptz DEFAULT now() NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION notice(VARIADIC text[]) RETURNS boolean AS $$\nDECLARE\ndebug boolean := current_setting('pgstac.debug', true);\nBEGIN\n IF debug THEN\n RAISE NOTICE 'NOTICE FROM FUNC: % >>>>> %', concat_ws(' | ', $1), clock_timestamp();\n RETURN TRUE;\n END IF;\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION empty_arr(ANYARRAY) RETURNS BOOLEAN AS $$\nSELECT CASE\n WHEN $1 IS NULL THEN TRUE\n WHEN cardinality($1)<1 THEN TRUE\nELSE FALSE\nEND;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_map_ident(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_ident(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_map_literal(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_literal(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION estimated_count(_where text) RETURNS bigint AS $$\nDECLARE\nrec record;\nrows bigint;\nBEGIN\n FOR rec in EXECUTE format(\n $q$\n EXPLAIN SELECT 1 FROM items WHERE %s\n $q$,\n _where)\n LOOP\n rows := substring(rec.\"QUERY PLAN\" FROM ' rows=([[:digit:]]+)');\n EXIT WHEN rows IS NOT NULL;\n END LOOP;\n\n RETURN rows;\nEND;\n$$ LANGUAGE PLPGSQL;\n/* converts a jsonb text array to a pg text[] array */\nCREATE OR REPLACE FUNCTION textarr(_js jsonb)\n RETURNS text[] AS $$\n SELECT\n CASE jsonb_typeof(_js)\n WHEN 'array' THEN ARRAY(SELECT jsonb_array_elements_text(_js))\n ELSE ARRAY[_js->>0]\n END\n;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION jsonb_paths (IN jdata jsonb, OUT path text[], OUT value jsonb) RETURNS\nSETOF RECORD AS $$\nwith recursive extract_all as\n(\n select\n ARRAY[key]::text[] as path,\n value\n FROM jsonb_each(jdata)\nunion all\n select\n path || coalesce(obj_key, (arr_key- 1)::text),\n coalesce(obj_value, arr_value)\n from extract_all\n left join lateral\n jsonb_each(case jsonb_typeof(value) when 'object' then value end)\n as o(obj_key, obj_value)\n on jsonb_typeof(value) = 'object'\n left join lateral\n jsonb_array_elements(case jsonb_typeof(value) when 'array' then value end)\n with ordinality as a(arr_value, arr_key)\n on jsonb_typeof(value) = 'array'\n where obj_key is not null or arr_key is not null\n)\nselect *\nfrom extract_all;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION jsonb_obj_paths (IN jdata jsonb, OUT path text[], OUT value jsonb) RETURNS\nSETOF RECORD AS $$\nwith recursive extract_all as\n(\n select\n ARRAY[key]::text[] as path,\n value\n FROM jsonb_each(jdata)\nunion all\n select\n path || obj_key,\n obj_value\n from extract_all\n left join lateral\n jsonb_each(case jsonb_typeof(value) when 'object' then value end)\n as o(obj_key, obj_value)\n on jsonb_typeof(value) = 'object'\n where obj_key is not null\n)\nselect *\nfrom extract_all;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION jsonb_val_paths (IN jdata jsonb, OUT path text[], OUT value jsonb) RETURNS\nSETOF RECORD AS $$\nSELECT * FROM jsonb_obj_paths(jdata) WHERE jsonb_typeof(value) not in ('object','array');\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION path_includes(IN path text[], IN includes text[]) RETURNS BOOLEAN AS $$\nWITH t AS (SELECT unnest(includes) i)\nSELECT EXISTS (\n SELECT 1 FROM t WHERE path @> string_to_array(trim(i), '.')\n);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION path_excludes(IN path text[], IN excludes text[]) RETURNS BOOLEAN AS $$\nWITH t AS (SELECT unnest(excludes) e)\nSELECT NOT EXISTS (\n SELECT 1 FROM t WHERE path @> string_to_array(trim(e), '.')\n);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION jsonb_obj_paths_filtered (\n IN jdata jsonb,\n IN includes text[] DEFAULT ARRAY[]::text[],\n IN excludes text[] DEFAULT ARRAY[]::text[],\n OUT path text[],\n OUT value jsonb\n) RETURNS\nSETOF RECORD AS $$\nSELECT path, value\nFROM jsonb_obj_paths(jdata)\nWHERE\n CASE WHEN cardinality(includes) > 0 THEN path_includes(path, includes) ELSE TRUE END\n AND\n path_excludes(path, excludes)\n\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION filter_jsonb(\n IN jdata jsonb,\n IN includes text[] DEFAULT ARRAY[]::text[],\n IN excludes text[] DEFAULT ARRAY[]::text[]\n) RETURNS jsonb AS $$\nDECLARE\nrec RECORD;\noutj jsonb := '{}'::jsonb;\ncreated_paths text[] := '{}'::text[];\nBEGIN\n\nIF empty_arr(includes) AND empty_arr(excludes) THEN\nRAISE NOTICE 'no filter';\n RETURN jdata;\nEND IF;\nFOR rec in\nSELECT * FROM jsonb_obj_paths_filtered(jdata, includes, excludes)\nWHERE jsonb_typeof(value) != 'object'\nLOOP\n IF array_length(rec.path,1)>1 THEN\n FOR i IN 1..(array_length(rec.path,1)-1) LOOP\n IF NOT array_to_string(rec.path[1:i],'.') = ANY (created_paths) THEN\n outj := jsonb_set(outj, rec.path[1:i],'{}', true);\n created_paths := created_paths || array_to_string(rec.path[1:i],'.');\n END IF;\n END LOOP;\n END IF;\n outj := jsonb_set(outj, rec.path, rec.value, true);\n created_paths := created_paths || array_to_string(rec.path,'.');\nEND LOOP;\nRETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION flip_jsonb_array(j jsonb) RETURNS jsonb AS $$\nSELECT jsonb_agg(value) FROM (SELECT value FROM jsonb_array_elements(j) WITH ORDINALITY ORDER BY ordinality DESC) as t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n/* Functions to create an iterable of cursors over partitions. */\nCREATE OR REPLACE FUNCTION create_cursor(q text) RETURNS refcursor AS $$\nDECLARE\n curs refcursor;\nBEGIN\n OPEN curs FOR EXECUTE q;\n RETURN curs;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION partition_queries(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC'\n) RETURNS SETOF text AS $$\nDECLARE\n partition_query text;\n query text;\n p record;\n cursors refcursor;\nBEGIN\nIF _orderby ILIKE 'datetime d%' THEN\n partition_query := format($q$\n SELECT partition, tstzrange\n FROM items_partitions\n ORDER BY tstzrange DESC;\n $q$);\nELSIF _orderby ILIKE 'datetime a%' THEN\n partition_query := format($q$\n SELECT partition, tstzrange\n FROM items_partitions\n ORDER BY tstzrange ASC\n ;\n $q$);\nELSE\n query := format($q$\n SELECT * FROM items\n WHERE %s\n ORDER BY %s\n $q$, _where, _orderby\n );\n\n RETURN NEXT query;\n RETURN;\nEND IF;\nFOR p IN\n EXECUTE partition_query\nLOOP\n query := format($q$\n SELECT * FROM items\n WHERE datetime >= %L AND datetime < %L AND %s\n ORDER BY %s\n $q$, lower(p.tstzrange), upper(p.tstzrange), _where, _orderby\n );\n RETURN NEXT query;\nEND LOOP;\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION partition_cursor(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC'\n) RETURNS SETOF refcursor AS $$\nDECLARE\n partition_query text;\n query text;\n p record;\n cursors refcursor;\nBEGIN\nFOR query IN SELECT * FROM partion_queries(_where, _orderby) LOOP\n RETURN NEXT create_cursor(query);\nEND LOOP;\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION partition_count(\n IN _where text DEFAULT 'TRUE'\n) RETURNS bigint AS $$\nDECLARE\n partition_query text;\n query text;\n p record;\n subtotal bigint;\n total bigint := 0;\nBEGIN\npartition_query := format($q$\n SELECT partition, tstzrange\n FROM items_partitions\n ORDER BY tstzrange DESC;\n$q$);\nRAISE NOTICE 'Partition Query: %', partition_query;\nFOR p IN\n EXECUTE partition_query\nLOOP\n query := format($q$\n SELECT count(*) FROM items\n WHERE datetime BETWEEN %L AND %L AND %s\n $q$, lower(p.tstzrange), upper(p.tstzrange), _where\n );\n RAISE NOTICE 'Query %', query;\n RAISE NOTICE 'Partition %, Count %, Total %',p.partition, subtotal, total;\n EXECUTE query INTO subtotal;\n total := subtotal + total;\nEND LOOP;\nRETURN total;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n/* looks for a geometry in a stac item first from geometry and falling back to bbox */\nCREATE OR REPLACE FUNCTION stac_geom(value jsonb) RETURNS geometry AS $$\nSELECT\n CASE\n WHEN value->>'geometry' IS NOT NULL THEN\n ST_GeomFromGeoJSON(value->>'geometry')\n WHEN value->>'bbox' IS NOT NULL THEN\n ST_MakeEnvelope(\n (value->'bbox'->>0)::float,\n (value->'bbox'->>1)::float,\n (value->'bbox'->>2)::float,\n (value->'bbox'->>3)::float,\n 4326\n )\n ELSE NULL\n END as geometry\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION stac_datetime(value jsonb) RETURNS timestamptz AS $$\nSELECT COALESCE(\n (value->'properties'->>'datetime')::timestamptz,\n (value->'properties'->>'start_datetime')::timestamptz\n);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_end_datetime(value jsonb) RETURNS timestamptz AS $$\nSELECT COALESCE(\n (value->'properties'->>'datetime')::timestamptz,\n (value->'properties'->>'end_datetime')::timestamptz\n);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\n\nCREATE OR REPLACE FUNCTION stac_daterange(value jsonb) RETURNS tstzrange AS $$\nSELECT tstzrange(stac_datetime(value),stac_end_datetime(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE TABLE IF NOT EXISTS collections (\n id VARCHAR GENERATED ALWAYS AS (content->>'id') STORED PRIMARY KEY,\n content JSONB\n);\n\nCREATE OR REPLACE FUNCTION create_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection(data jsonb) RETURNS VOID AS $$\nDECLARE\nout collections%ROWTYPE;\nBEGIN\n UPDATE collections SET content=data WHERE id = data->>'id' RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ON CONFLICT (id) DO\n UPDATE\n SET content=EXCLUDED.content\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION delete_collection(_id text) RETURNS VOID AS $$\nDECLARE\nout collections%ROWTYPE;\nBEGIN\n DELETE FROM collections WHERE id = _id RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION get_collection(id text) RETURNS jsonb AS $$\nSELECT content FROM collections\nWHERE id=$1\n;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION all_collections() RETURNS jsonb AS $$\nSELECT jsonb_agg(content) FROM collections;\n;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE TABLE items (\n id text NOT NULL,\n geometry geometry NOT NULL,\n collection_id text NOT NULL,\n datetime timestamptz NOT NULL,\n end_datetime timestamptz NOT NULL,\n properties jsonb NOT NULL,\n content JSONB NOT NULL\n)\nPARTITION BY RANGE (datetime)\n;\n\nCREATE OR REPLACE FUNCTION properties_idx (IN content jsonb) RETURNS jsonb AS $$\n with recursive extract_all as\n (\n select\n ARRAY[key]::text[] as path,\n ARRAY[key]::text[] as fullpath,\n value\n FROM jsonb_each(content->'properties')\n union all\n select\n CASE WHEN obj_key IS NOT NULL THEN path || obj_key ELSE path END,\n path || coalesce(obj_key, (arr_key- 1)::text),\n coalesce(obj_value, arr_value)\n from extract_all\n left join lateral\n jsonb_each(case jsonb_typeof(value) when 'object' then value end)\n as o(obj_key, obj_value)\n on jsonb_typeof(value) = 'object'\n left join lateral\n jsonb_array_elements(case jsonb_typeof(value) when 'array' then value end)\n with ordinality as a(arr_value, arr_key)\n on jsonb_typeof(value) = 'array'\n where obj_key is not null or arr_key is not null\n )\n , paths AS (\n select\n array_to_string(path, '.') as path,\n value\n FROM extract_all\n WHERE\n jsonb_typeof(value) NOT IN ('array','object')\n ), grouped AS (\n SELECT path, jsonb_agg(distinct value) vals FROM paths group by path\n ) SELECT coalesce(jsonb_object_agg(path, CASE WHEN jsonb_array_length(vals)=1 THEN vals->0 ELSE vals END) - '{datetime}'::text[], '{}'::jsonb) FROM grouped\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET JIT TO OFF;\n\nCREATE INDEX \"datetime_idx\" ON items (datetime);\nCREATE INDEX \"end_datetime_idx\" ON items (end_datetime);\nCREATE INDEX \"properties_idx\" ON items USING GIN (properties jsonb_path_ops);\nCREATE INDEX \"collection_idx\" ON items (collection_id);\nCREATE INDEX \"geometry_idx\" ON items USING GIST (geometry);\nCREATE UNIQUE INDEX \"items_id_datetime_idx\" ON items (datetime, id);\n\nALTER TABLE items ADD CONSTRAINT items_collections_fk FOREIGN KEY (collection_id) REFERENCES collections(id) ON DELETE CASCADE DEFERRABLE;\n\nCREATE OR REPLACE FUNCTION analyze_empty_partitions() RETURNS VOID AS $$\nDECLARE\n p text;\nBEGIN\n FOR p IN SELECT partition FROM all_items_partitions WHERE est_cnt = 0 LOOP\n EXECUTE format('ANALYZE %I;', p);\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION items_partition_name(timestamptz) RETURNS text AS $$\n SELECT to_char($1, '\"items_p\"IYYY\"w\"IW');\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION items_partition_exists(text) RETURNS boolean AS $$\n SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_class WHERE relname=$1);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION items_partition_exists(timestamptz) RETURNS boolean AS $$\n SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_class WHERE relname=items_partition_name($1));\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION items_partition_create_worker(partition text, partition_start timestamptz, partition_end timestamptz) RETURNS VOID AS $$\nDECLARE\n err_context text;\nBEGIN\n EXECUTE format(\n $f$\n CREATE TABLE IF NOT EXISTS %1$I PARTITION OF items\n FOR VALUES FROM (%2$L) TO (%3$L);\n CREATE UNIQUE INDEX IF NOT EXISTS %4$I ON %1$I (id);\n $f$,\n partition,\n partition_start,\n partition_end,\n concat(partition, '_id_pk')\n );\nEXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', partition;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH to pgstac, public;\n\nCREATE OR REPLACE FUNCTION items_partition_create(ts timestamptz) RETURNS text AS $$\nDECLARE\n partition text := items_partition_name(ts);\n partition_start timestamptz;\n partition_end timestamptz;\nBEGIN\n IF items_partition_exists(partition) THEN\n RETURN partition;\n END IF;\n partition_start := date_trunc('week', ts);\n partition_end := partition_start + '1 week'::interval;\n PERFORM items_partition_create_worker(partition, partition_start, partition_end);\n RAISE NOTICE 'partition: %', partition;\n RETURN partition;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION items_partition_create(st timestamptz, et timestamptz) RETURNS SETOF text AS $$\nWITH t AS (\n SELECT\n generate_series(\n date_trunc('week',st),\n date_trunc('week', et),\n '1 week'::interval\n ) w\n)\nSELECT items_partition_create(w) FROM t;\n$$ LANGUAGE SQL;\n\n\nCREATE UNLOGGED TABLE items_staging (\n content JSONB NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION items_staging_insert_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n mindate timestamptz;\n maxdate timestamptz;\n partition text;\nBEGIN\n SELECT min(stac_datetime(content)), max(stac_datetime(content)) INTO mindate, maxdate FROM newdata;\n PERFORM items_partition_create(mindate, maxdate);\n INSERT INTO items (id, geometry, collection_id, datetime, end_datetime, properties, content)\n SELECT\n content->>'id',\n stac_geom(content),\n content->>'collection',\n stac_datetime(content),\n stac_end_datetime(content),\n properties_idx(content),\n content\n FROM newdata\n ;\n DELETE FROM items_staging;\n PERFORM analyze_empty_partitions();\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER items_staging_insert_trigger AFTER INSERT ON items_staging REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_insert_triggerfunc();\n\n\nCREATE UNLOGGED TABLE items_staging_ignore (\n content JSONB NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION items_staging_ignore_insert_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n mindate timestamptz;\n maxdate timestamptz;\n partition text;\nBEGIN\n SELECT min(stac_datetime(content)), max(stac_datetime(content)) INTO mindate, maxdate FROM newdata;\n PERFORM items_partition_create(mindate, maxdate);\n INSERT INTO items (id, geometry, collection_id, datetime, end_datetime, properties, content)\n SELECT\n content->>'id',\n stac_geom(content),\n content->>'collection',\n stac_datetime(content),\n stac_end_datetime(content),\n properties_idx(content),\n content\n FROM newdata\n ON CONFLICT DO NOTHING\n ;\n DELETE FROM items_staging_ignore;\n PERFORM analyze_empty_partitions();\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER items_staging_ignore_insert_trigger AFTER INSERT ON items_staging_ignore REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_ignore_insert_triggerfunc();\n\nCREATE UNLOGGED TABLE items_staging_upsert (\n content JSONB NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION items_staging_upsert_insert_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n mindate timestamptz;\n maxdate timestamptz;\n partition text;\nBEGIN\n SELECT min(stac_datetime(content)), max(stac_datetime(content)) INTO mindate, maxdate FROM newdata;\n PERFORM items_partition_create(mindate, maxdate);\n INSERT INTO items (id, geometry, collection_id, datetime, end_datetime, properties, content)\n SELECT\n content->>'id',\n stac_geom(content),\n content->>'collection',\n stac_datetime(content),\n stac_end_datetime(content),\n properties_idx(content),\n content\n FROM newdata\n ON CONFLICT (datetime, id) DO UPDATE SET\n content = EXCLUDED.content\n WHERE items.content IS DISTINCT FROM EXCLUDED.content\n ;\n DELETE FROM items_staging_upsert;\n PERFORM analyze_empty_partitions();\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER items_staging_upsert_insert_trigger AFTER INSERT ON items_staging_upsert REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_upsert_insert_triggerfunc();\n\nCREATE OR REPLACE FUNCTION items_update_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\nBEGIN\n NEW.id := NEW.content->>'id';\n NEW.datetime := stac_datetime(NEW.content);\n NEW.end_datetime := stac_end_datetime(NEW.content);\n NEW.collection_id := NEW.content->>'collection';\n NEW.geometry := stac_geom(NEW.content);\n NEW.properties := properties_idx(NEW.content);\n IF TG_OP = 'UPDATE' AND NEW IS NOT DISTINCT FROM OLD THEN\n RETURN NULL;\n END IF;\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER items_update_trigger BEFORE UPDATE ON items\n FOR EACH ROW EXECUTE PROCEDURE items_update_triggerfunc();\n\n/*\nView to get a table of available items partitions\nwith date ranges\n*/\nCREATE VIEW all_items_partitions AS\nWITH base AS\n(SELECT\n c.oid::pg_catalog.regclass::text as partition,\n pg_catalog.pg_get_expr(c.relpartbound, c.oid) as _constraint,\n regexp_matches(\n pg_catalog.pg_get_expr(c.relpartbound, c.oid),\n E'\\\\(''\\([0-9 :+-]*\\)''\\\\).*\\\\(''\\([0-9 :+-]*\\)''\\\\)'\n ) as t,\n reltuples::bigint as est_cnt\nFROM pg_catalog.pg_class c, pg_catalog.pg_inherits i\nWHERE c.oid = i.inhrelid AND i.inhparent = 'items'::regclass)\nSELECT partition, tstzrange(\n t[1]::timestamptz,\n t[2]::timestamptz\n), est_cnt\nFROM base\nORDER BY 2 desc;\n\nCREATE OR REPLACE VIEW items_partitions AS\nSELECT * FROM all_items_partitions WHERE est_cnt>0;\n\nCREATE OR REPLACE FUNCTION get_item(_id text) RETURNS jsonb AS $$\n SELECT content FROM items WHERE id=_id;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION delete_item(_id text) RETURNS VOID AS $$\nDECLARE\nout items%ROWTYPE;\nBEGIN\n DELETE FROM items WHERE id = _id RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION create_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION update_item(data jsonb) RETURNS VOID AS $$\nDECLARE\n out items%ROWTYPE;\nBEGIN\n UPDATE items SET content=data WHERE id = data->>'id' RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION collection_bbox(id text) RETURNS jsonb AS $$\nSELECT (replace(replace(replace(st_extent(geometry)::text,'BOX(','[['),')',']]'),' ',','))::jsonb\nFROM items WHERE collection_id=$1;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION collection_temporal_extent(id text) RETURNS jsonb AS $$\nSELECT to_jsonb(array[array[min(datetime)::text, max(datetime)::text]])\nFROM items WHERE collection_id=$1;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection_extents() RETURNS VOID AS $$\nUPDATE collections SET\n content = content ||\n jsonb_build_object(\n 'extent', jsonb_build_object(\n 'spatial', jsonb_build_object(\n 'bbox', collection_bbox(collections.id)\n ),\n 'temporal', jsonb_build_object(\n 'interval', collection_temporal_extent(collections.id)\n )\n )\n )\n;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION items_path(\n IN dotpath text,\n OUT field text,\n OUT path text,\n OUT path_txt text,\n OUT jsonpath text,\n OUT eq text\n) RETURNS RECORD AS $$\nDECLARE\npath_elements text[];\nlast_element text;\nBEGIN\ndotpath := replace(trim(dotpath), 'properties.', '');\n\nIF dotpath = '' THEN\n RETURN;\nEND IF;\n\npath_elements := string_to_array(dotpath, '.');\njsonpath := NULL;\n\nIF path_elements[1] IN ('id','geometry','datetime') THEN\n field := path_elements[1];\n path_elements := path_elements[2:];\nELSIF path_elements[1] = 'collection' THEN\n field := 'collection_id';\n path_elements := path_elements[2:];\nELSIF path_elements[1] IN ('links', 'assets', 'stac_version', 'stac_extensions') THEN\n field := 'content';\nELSE\n field := 'content';\n path_elements := '{properties}'::text[] || path_elements;\nEND IF;\nIF cardinality(path_elements)<1 THEN\n path := field;\n path_txt := field;\n jsonpath := '$';\n eq := NULL; -- format($F$ %s = %%s $F$, field);\n RETURN;\nEND IF;\n\n\nlast_element := path_elements[cardinality(path_elements)];\npath_elements := path_elements[1:cardinality(path_elements)-1];\njsonpath := concat(array_to_string('{$}'::text[] || array_map_ident(path_elements), '.'), '.', quote_ident(last_element));\npath_elements := array_map_literal(path_elements);\npath := format($F$ properties->%s $F$, quote_literal(dotpath));\npath_txt := format($F$ properties->>%s $F$, quote_literal(dotpath));\neq := format($F$ properties @? '$.%s[*] ? (@ == %%s) '$F$, quote_ident(dotpath));\n\nRAISE NOTICE 'ITEMS PATH -- % % % % %', field, path, path_txt, jsonpath, eq;\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION parse_dtrange(IN _indate jsonb, OUT _tstzrange tstzrange) AS $$\nWITH t AS (\n SELECT CASE\n WHEN jsonb_typeof(_indate) = 'array' THEN\n textarr(_indate)\n ELSE\n regexp_split_to_array(\n btrim(_indate::text,'\"'),\n '/'\n )\n END AS arr\n)\n, t1 AS (\n SELECT\n CASE\n WHEN array_upper(arr,1) = 1 OR arr[1] = '..' OR arr[1] IS NULL THEN '-infinity'::timestamptz\n ELSE arr[1]::timestamptz\n END AS st,\n CASE\n WHEN array_upper(arr,1) = 1 THEN arr[1]::timestamptz\n WHEN arr[2] = '..' OR arr[2] IS NULL THEN 'infinity'::timestamptz\n ELSE arr[2]::timestamptz\n END AS et\n FROM t\n)\nSELECT\n tstzrange(st,et)\nFROM t1;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION bbox_geom(_bbox jsonb) RETURNS geometry AS $$\nSELECT CASE jsonb_array_length(_bbox)\n WHEN 4 THEN\n ST_SetSRID(ST_MakeEnvelope(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float,\n (_bbox->>3)::float\n ),4326)\n WHEN 6 THEN\n ST_SetSRID(ST_3DMakeBox(\n ST_MakePoint(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float\n ),\n ST_MakePoint(\n (_bbox->>3)::float,\n (_bbox->>4)::float,\n (_bbox->>5)::float\n )\n ),4326)\n ELSE null END;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION cql_and_append(existing jsonb, newfilters jsonb) RETURNS jsonb AS $$\nSELECT CASE WHEN existing ? 'filter' AND newfilters IS NOT NULL THEN\n jsonb_build_object(\n 'and',\n jsonb_build_array(\n existing->'filter',\n newfilters\n )\n )\nELSE\n newfilters\nEND;\n$$ LANGUAGE SQL;\n\n\n-- ADDs base filters (ids, collections, datetime, bbox, intersects) that are\n-- added outside of the filter/query in the stac request\nCREATE OR REPLACE FUNCTION add_filters_to_cql(j jsonb) RETURNS jsonb AS $$\nDECLARE\nnewprop jsonb;\nnewprops jsonb := '[]'::jsonb;\nBEGIN\nIF j ? 'id' THEN\n newprop := jsonb_build_object(\n 'in',\n jsonb_build_array(\n '{\"property\":\"id\"}'::jsonb,\n j->'id'\n )\n );\n newprops := jsonb_insert(newprops, '{1}', newprop);\nEND IF;\nIF j ? 'collections' THEN\n newprop := jsonb_build_object(\n 'in',\n jsonb_build_array(\n '{\"property\":\"collection\"}'::jsonb,\n j->'collections'\n )\n );\n newprops := jsonb_insert(newprops, '{1}', newprop);\nEND IF;\n\nIF j ? 'datetime' THEN\n newprop := format(\n '{\"anyinteracts\":[{\"property\":\"datetime\"}, %s]}',\n j->'datetime'\n );\n newprops := jsonb_insert(newprops, '{1}', newprop);\nEND IF;\n\nIF j ? 'bbox' THEN\n newprop := format(\n '{\"intersects\":[{\"property\":\"geometry\"}, %s]}',\n j->'bbox'\n );\n newprops := jsonb_insert(newprops, '{1}', newprop);\nEND IF;\n\nIF j ? 'intersects' THEN\n newprop := format(\n '{\"intersects\":[{\"property\":\"geometry\"}, %s]}',\n j->'intersects'\n );\n newprops := jsonb_insert(newprops, '{1}', newprop);\nEND IF;\n\nRAISE NOTICE 'newprops: %', newprops;\n\nIF newprops IS NOT NULL AND jsonb_array_length(newprops) > 0 THEN\n return jsonb_set(\n j,\n '{filter}',\n cql_and_append(j, jsonb_build_object('and', newprops))\n ) - '{id,collections,datetime,bbox,intersects}'::text[];\nEND IF;\n\nreturn j;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION query_to_cqlfilter(j jsonb) RETURNS jsonb AS $$\n-- Translates anything passed in through the deprecated \"query\" into equivalent CQL\nWITH t AS (\n SELECT key as property, value as ops\n FROM jsonb_each(j->'query')\n), t2 AS (\n SELECT property, (jsonb_each(ops)).*\n FROM t WHERE jsonb_typeof(ops) = 'object'\n UNION ALL\n SELECT property, 'eq', ops\n FROM t WHERE jsonb_typeof(ops) != 'object'\n), t3 AS (\nSELECT\n jsonb_strip_nulls(jsonb_build_object(\n 'and',\n jsonb_agg(\n jsonb_build_object(\n key,\n jsonb_build_array(\n jsonb_build_object('property',property),\n value\n )\n )\n )\n )) as qcql FROM t2\n)\nSELECT\n CASE WHEN qcql IS NOT NULL THEN\n jsonb_set(j, '{filter}', cql_and_append(j, qcql)) - 'query'\n ELSE j\n END\nFROM t3\n;\n$$ LANGUAGE SQL;\n\n\n\nCREATE OR REPLACE FUNCTION temporal_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\nll text := 'datetime';\nlh text := 'end_datetime';\nrrange tstzrange;\nrl text;\nrh text;\noutq text;\nBEGIN\nrrange := parse_dtrange(args->1);\nRAISE NOTICE 'Constructing temporal query OP: %, ARGS: %, RRANGE: %', op, args, rrange;\nop := lower(op);\nrl := format('%L::timestamptz', lower(rrange));\nrh := format('%L::timestamptz', upper(rrange));\noutq := CASE op\n WHEN 't_before' THEN 'lh < rl'\n WHEN 't_after' THEN 'll > rh'\n WHEN 't_meets' THEN 'lh = rl'\n WHEN 't_metby' THEN 'll = rh'\n WHEN 't_overlaps' THEN 'll < rl AND rl < lh < rh'\n WHEN 't_overlappedby' THEN 'rl < ll < rh AND lh > rh'\n WHEN 't_starts' THEN 'll = rl AND lh < rh'\n WHEN 't_startedby' THEN 'll = rl AND lh > rh'\n WHEN 't_during' THEN 'll > rl AND lh < rh'\n WHEN 't_contains' THEN 'll < rl AND lh > rh'\n WHEN 't_finishes' THEN 'll > rl AND lh = rh'\n WHEN 't_finishedby' THEN 'll < rl AND lh = rh'\n WHEN 't_equals' THEN 'll = rl AND lh = rh'\n WHEN 't_disjoint' THEN 'NOT (ll <= rh AND lh >= rl)'\n WHEN 't_intersects' THEN 'll <= rh AND lh >= rl'\n WHEN 'anyinteracts' THEN 'll <= rh AND lh >= rl'\nEND;\noutq := regexp_replace(outq, '\\mll\\M', ll);\noutq := regexp_replace(outq, '\\mlh\\M', lh);\noutq := regexp_replace(outq, '\\mrl\\M', rl);\noutq := regexp_replace(outq, '\\mrh\\M', rh);\noutq := format('(%s)', outq);\nRETURN outq;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION spatial_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\ngeom text;\nj jsonb := args->1;\nBEGIN\nop := lower(op);\nRAISE NOTICE 'Constructing spatial query OP: %, ARGS: %', op, args;\nIF op NOT IN ('s_equals','s_disjoint','s_touches','s_within','s_overlaps','s_crosses','s_intersects','intersects','s_contains') THEN\n RAISE EXCEPTION 'Spatial Operator % Not Supported', op;\nEND IF;\nop := regexp_replace(op, '^s_', 'st_');\nIF op = 'intersects' THEN\n op := 'st_intersects';\nEND IF;\n-- Convert geometry to WKB string\nIF j ? 'type' AND j ? 'coordinates' THEN\n geom := st_geomfromgeojson(j)::text;\nELSIF jsonb_typeof(j) = 'array' THEN\n geom := bbox_geom(j)::text;\nEND IF;\n\nRETURN format('%s(geometry, %L::geometry)', op, geom);\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n/* cql_query_op -- Parses a CQL query operation, recursing when necessary\n IN jsonb -- a subelement from a valid stac query\n IN text -- the operator being used on elements passed in\n RETURNS a SQL fragment to be used in a WHERE clause\n*/\nCREATE OR REPLACE FUNCTION cql_query_op(j jsonb, _op text DEFAULT NULL) RETURNS text AS $$\nDECLARE\njtype text := jsonb_typeof(j);\nop text := lower(_op);\nops jsonb :=\n '{\n \"eq\": \"%s = %s\",\n \"lt\": \"%s < %s\",\n \"lte\": \"%s <= %s\",\n \"gt\": \"%s > %s\",\n \"gte\": \"%s >= %s\",\n \"like\": \"%s LIKE %s\",\n \"+\": \"%s + %s\",\n \"-\": \"%s - %s\",\n \"*\": \"%s * %s\",\n \"/\": \"%s / %s\",\n \"in\": \"%s = ANY (%s)\",\n \"not\": \"NOT (%s)\",\n \"between\": \"%s BETWEEN %s AND %s\",\n \"lower\":\"lower(%s)\"\n }'::jsonb;\nret text;\nargs text[] := NULL;\n\nBEGIN\nRAISE NOTICE 'j: %, op: %, jtype: %', j, op, jtype;\n\n-- Set Lower Case on Both Arguments When Case Insensitive Flag Set\nIF op in ('eq','lt','lte','gt','gte','like') AND jsonb_typeof(j->2) = 'boolean' THEN\n IF (j->>2)::boolean THEN\n RETURN format(concat('(',ops->>op,')'), cql_query_op(jsonb_build_array(j->0), 'lower'), cql_query_op(jsonb_build_array(j->1), 'lower'));\n END IF;\nEND IF;\n\n-- Special Case when comparing a property in a jsonb field to a string or number using eq\n-- Allows to leverage GIN index on jsonb fields\nIF op = 'eq' THEN\n IF j->0 ? 'property'\n AND jsonb_typeof(j->1) IN ('number','string')\n AND (items_path(j->0->>'property')).eq IS NOT NULL\n THEN\n RETURN format((items_path(j->0->>'property')).eq, j->1);\n END IF;\nEND IF;\n\nIF op ilike 't_%' or op = 'anyinteracts' THEN\n RETURN temporal_op_query(op, j);\nEND IF;\n\nIF op ilike 's_%' or op = 'intersects' THEN\n RETURN spatial_op_query(op, j);\nEND IF;\n\n\nIF jtype = 'object' THEN\n RAISE NOTICE 'parsing object';\n IF j ? 'property' THEN\n -- Convert the property to be used as an identifier\n return (items_path(j->>'property')).path_txt;\n ELSIF _op IS NULL THEN\n -- Iterate to convert elements in an object where the operator has not been set\n -- Combining with AND\n SELECT\n array_to_string(array_agg(cql_query_op(e.value, e.key)), ' AND ')\n INTO ret\n FROM jsonb_each(j) e;\n RETURN ret;\n END IF;\nEND IF;\n\nIF jtype = 'string' THEN\n RETURN quote_literal(j->>0);\nEND IF;\n\nIF jtype ='number' THEN\n RETURN (j->>0)::numeric;\nEND IF;\n\nIF jtype = 'array' AND op IS NULL THEN\n RAISE NOTICE 'Parsing array into array arg. j: %', j;\n SELECT format($f$ '{%s}'::text[] $f$, string_agg(e,',')) INTO ret FROM jsonb_array_elements_text(j) e;\n RETURN ret;\nEND IF;\n\n\n-- If the type of the passed json is an array\n-- Calculate the arguments that will be passed to functions/operators\nIF jtype = 'array' THEN\n RAISE NOTICE 'Parsing array into args. j: %', j;\n -- If any argument is numeric, cast any text arguments to numeric\n IF j @? '$[*] ? (@.type() == \"number\")' THEN\n SELECT INTO args\n array_agg(concat('(',cql_query_op(e),')::numeric'))\n FROM jsonb_array_elements(j) e;\n ELSE\n SELECT INTO args\n array_agg(cql_query_op(e))\n FROM jsonb_array_elements(j) e;\n END IF;\n --RETURN args;\nEND IF;\nRAISE NOTICE 'ARGS after array cleaning: %', args;\n\nIF op IS NULL THEN\n RETURN args::text[];\nEND IF;\n\nIF args IS NULL OR cardinality(args) < 1 THEN\n RAISE NOTICE 'No Args';\n RETURN '';\nEND IF;\n\nIF op IN ('and','or') THEN\n SELECT\n CONCAT(\n '(',\n array_to_string(args, UPPER(CONCAT(' ',op,' '))),\n ')'\n ) INTO ret\n FROM jsonb_array_elements(j) e;\n RETURN ret;\nEND IF;\n\n-- If the op is in the ops json then run using the template in the json\nIF ops ? op THEN\n RAISE NOTICE 'ARGS: % MAPPED: %',args, array_map_literal(args);\n\n RETURN format(concat('(',ops->>op,')'), VARIADIC args);\nEND IF;\n\nRETURN j->>0;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n\n\nCREATE OR REPLACE FUNCTION cql_to_where(_search jsonb = '{}'::jsonb) RETURNS text AS $$\nDECLARE\nsearch jsonb := _search;\n_where text;\nBEGIN\nRAISE NOTICE 'SEARCH CQL 1: %', search;\n\n-- Convert any old style stac query to cql\nsearch := query_to_cqlfilter(search);\n\nRAISE NOTICE 'SEARCH CQL 2: %', search;\n\n-- Convert item,collection,datetime,bbox,intersects to cql\nsearch := add_filters_to_cql(search);\n\nRAISE NOTICE 'SEARCH CQL Final: %', search;\n_where := cql_query_op(search->'filter');\n\nIF trim(_where) = '' THEN\n _where := NULL;\nEND IF;\n_where := coalesce(_where, ' TRUE ');\nRETURN _where;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION parse_sort_dir(_dir text, reverse boolean default false) RETURNS text AS $$\nWITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n) SELECT\n CASE\n WHEN NOT reverse THEN d\n WHEN d = 'ASC' THEN 'DESC'\n WHEN d = 'DESC' THEN 'ASC'\n END\nFROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION sort_dir_to_op(_dir text, prev boolean default false) RETURNS text AS $$\nWITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n) SELECT\n CASE\n WHEN d = 'ASC' AND prev THEN '<='\n WHEN d = 'DESC' AND prev THEN '>='\n WHEN d = 'ASC' THEN '>='\n WHEN d = 'DESC' THEN '<='\n END\nFROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION sort_sqlorderby(\n _search jsonb DEFAULT NULL,\n reverse boolean DEFAULT FALSE\n) RETURNS text AS $$\nWITH sorts AS (\n SELECT\n (items_path(value->>'field')).path as key,\n parse_sort_dir(value->>'direction', reverse) as dir\n FROM jsonb_array_elements(\n '[]'::jsonb\n ||\n coalesce(_search->'sortby','[{\"field\":\"datetime\", \"direction\":\"desc\"}]')\n ||\n '[{\"field\":\"id\",\"direction\":\"desc\"}]'::jsonb\n )\n)\nSELECT array_to_string(\n array_agg(concat(key, ' ', dir)),\n ', '\n) FROM sorts;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_sort_dir(sort_item jsonb) RETURNS text AS $$\nSELECT CASE WHEN sort_item->>'direction' ILIKE 'desc%' THEN 'DESC' ELSE 'ASC' END;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION get_token_filter(_search jsonb = '{}'::jsonb, token_rec jsonb DEFAULT NULL) RETURNS text AS $$\nDECLARE\ntoken_id text;\nfilters text[] := '{}'::text[];\nprev boolean := TRUE;\nfield text;\ndir text;\nsort record;\norfilters text[] := '{}'::text[];\nandfilters text[] := '{}'::text[];\noutput text;\ntoken_where text;\nBEGIN\n-- If no token provided return NULL\nIF token_rec IS NULL THEN\n IF NOT (_search ? 'token' AND\n (\n (_search->>'token' ILIKE 'prev:%')\n OR\n (_search->>'token' ILIKE 'next:%')\n )\n ) THEN\n RETURN NULL;\n END IF;\n prev := (_search->>'token' ILIKE 'prev:%');\n token_id := substr(_search->>'token', 6);\n SELECT to_jsonb(items) INTO token_rec FROM items WHERE id=token_id;\nEND IF;\nRAISE NOTICE 'TOKEN ID: %', token_rec->'id';\n\nCREATE TEMP TABLE sorts (\n _row int GENERATED ALWAYS AS IDENTITY NOT NULL,\n _field text PRIMARY KEY,\n _dir text NOT NULL,\n _val text\n) ON COMMIT DROP;\n\n-- Make sure we only have distinct columns to sort with taking the first one we get\nINSERT INTO sorts (_field, _dir)\n SELECT\n (items_path(value->>'field')).path,\n get_sort_dir(value)\n FROM\n jsonb_array_elements(coalesce(_search->'sort','[{\"field\":\"datetime\",\"direction\":\"desc\"}]'))\nON CONFLICT DO NOTHING\n;\n\n-- Get the first sort direction provided. As the id is a primary key, if there are any\n-- sorts after id they won't do anything, so make sure that id is the last sort item.\nSELECT _dir INTO dir FROM sorts ORDER BY _row ASC LIMIT 1;\nIF EXISTS (SELECT 1 FROM sorts WHERE _field = 'id') THEN\n DELETE FROM sorts WHERE _row > (SELECT _row FROM sorts WHERE _field = 'id');\nELSE\n INSERT INTO sorts (_field, _dir) VALUES ('id', dir);\nEND IF;\n\n-- Add value from looked up item to the sorts table\nUPDATE sorts SET _val=quote_literal(token_rec->>_field);\n\n-- Check if all sorts are the same direction and use row comparison\n-- to filter\nIF (SELECT count(DISTINCT _dir) FROM sorts) = 1 THEN\n SELECT format(\n '(%s) %s (%s)',\n concat_ws(', ', VARIADIC array_agg(quote_ident(_field))),\n CASE WHEN (prev AND dir = 'ASC') OR (NOT prev AND dir = 'DESC') THEN '<' ELSE '>' END,\n concat_ws(', ', VARIADIC array_agg(_val))\n ) INTO output FROM sorts\n WHERE token_rec ? _field\n ;\nELSE\n FOR sort IN SELECT * FROM sorts ORDER BY _row asc LOOP\n RAISE NOTICE 'SORT: %', sort;\n IF sort._row = 1 THEN\n orfilters := orfilters || format('(%s %s %s)',\n quote_ident(sort._field),\n CASE WHEN (prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC') THEN '<' ELSE '>' END,\n sort._val\n );\n ELSE\n orfilters := orfilters || format('(%s AND %s %s %s)',\n array_to_string(andfilters, ' AND '),\n quote_ident(sort._field),\n CASE WHEN (prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC') THEN '<' ELSE '>' END,\n sort._val\n );\n\n END IF;\n andfilters := andfilters || format('%s = %s',\n quote_ident(sort._field),\n sort._val\n );\n END LOOP;\n output := array_to_string(orfilters, ' OR ');\nEND IF;\nDROP TABLE IF EXISTS sorts;\ntoken_where := concat('(',coalesce(output,'true'),')');\nIF trim(token_where) = '' THEN\n token_where := NULL;\nEND IF;\nRAISE NOTICE 'TOKEN_WHERE: |%|',token_where;\nRETURN token_where;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION search_tohash(jsonb) RETURNS jsonb AS $$\n SELECT $1 - '{token,limit,context,includes,excludes}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION search_hash(jsonb) RETURNS text AS $$\n SELECT md5(search_tohash($1)::text);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE TABLE IF NOT EXISTS searches(\n hash text GENERATED ALWAYS AS (search_hash(search)) STORED PRIMARY KEY,\n search jsonb NOT NULL,\n _where text,\n orderby text,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n statslastupdated timestamptz,\n estimated_count bigint,\n total_count bigint\n);\n\nCREATE OR REPLACE FUNCTION search_query(_search jsonb = '{}'::jsonb, updatestats boolean DEFAULT false) RETURNS searches AS $$\nDECLARE\n search searches%ROWTYPE;\nBEGIN\nINSERT INTO searches (search)\n VALUES (search_tohash(_search))\n ON CONFLICT DO NOTHING\n RETURNING * INTO search;\nIF search.hash IS NULL THEN\n SELECT * INTO search FROM searches WHERE hash=search_hash(_search);\nEND IF;\nIF search._where IS NULL THEN\n search._where := cql_to_where(_search);\nEND IF;\nIF search.orderby IS NULL THEN\n search.orderby := sort_sqlorderby(_search);\nEND IF;\n\nIF search.statslastupdated IS NULL OR age(search.statslastupdated) > '1 day'::interval OR (_search ? 'context' AND search.total_count IS NULL) THEN\n updatestats := TRUE;\nEND IF;\n\nIF updatestats THEN\n -- Get Estimated Stats\n RAISE NOTICE 'Getting stats for %', search._where;\n search.estimated_count := estimated_count(search._where);\n RAISE NOTICE 'Estimated Count: %', search.estimated_count;\n\n IF _search ? 'context' OR search.estimated_count < 10000 THEN\n --search.total_count := partition_count(search._where);\n EXECUTE format(\n 'SELECT count(*) FROM items WHERE %s',\n search._where\n ) INTO search.total_count;\n RAISE NOTICE 'Actual Count: %', search.total_count;\n ELSE\n search.total_count := NULL;\n END IF;\n search.statslastupdated := now();\nEND IF;\n\nsearch.lastused := now();\nsearch.usecount := coalesce(search.usecount,0) + 1;\nRAISE NOTICE 'SEARCH: %', search;\nUPDATE searches SET\n _where = search._where,\n orderby = search.orderby,\n lastused = search.lastused,\n usecount = search.usecount,\n statslastupdated = search.statslastupdated,\n estimated_count = search.estimated_count,\n total_count = search.total_count\nWHERE hash = search.hash\n;\nRETURN search;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n\nCREATE OR REPLACE FUNCTION search(_search jsonb = '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n searches searches%ROWTYPE;\n _where text;\n token_where text;\n full_where text;\n orderby text;\n query text;\n token_type text := substr(_search->>'token',1,4);\n _limit int := coalesce((_search->>'limit')::int, 10);\n curs refcursor;\n cntr int := 0;\n iter_record items%ROWTYPE;\n first_record items%ROWTYPE;\n last_record items%ROWTYPE;\n out_records jsonb := '[]'::jsonb;\n prev_query text;\n next text;\n prev_id text;\n has_next boolean := false;\n has_prev boolean := false;\n prev text;\n total_count bigint;\n context jsonb;\n collection jsonb;\n includes text[];\n excludes text[];\n exit_flag boolean := FALSE;\n batches int := 0;\n timer timestamptz := clock_timestamp();\nBEGIN\nsearches := search_query(_search);\n_where := searches._where;\norderby := searches.orderby;\ntotal_count := coalesce(searches.total_count, searches.estimated_count);\n\n\nIF token_type='prev' THEN\n token_where := get_token_filter(_search, null::jsonb);\n orderby := sort_sqlorderby(_search, TRUE);\nEND IF;\nIF token_type='next' THEN\n token_where := get_token_filter(_search, null::jsonb);\nEND IF;\n\nfull_where := concat_ws(' AND ', _where, token_where);\nRAISE NOTICE 'FULL QUERY % %', full_where, clock_timestamp()-timer;\ntimer := clock_timestamp();\n\nFOR query IN SELECT partition_queries(full_where, orderby) LOOP\n timer := clock_timestamp();\n query := format('%s LIMIT %L', query, _limit + 1);\n RAISE NOTICE 'Partition Query: %', query;\n batches := batches + 1;\n curs = create_cursor(query);\n LOOP\n FETCH curs into iter_record;\n EXIT WHEN NOT FOUND;\n cntr := cntr + 1;\n last_record := iter_record;\n IF cntr = 1 THEN\n first_record := last_record;\n END IF;\n IF cntr <= _limit THEN\n out_records := out_records || last_record.content;\n ELSIF cntr > _limit THEN\n has_next := true;\n exit_flag := true;\n EXIT;\n END IF;\n END LOOP;\n RAISE NOTICE 'Query took %', clock_timestamp()-timer;\n timer := clock_timestamp();\n EXIT WHEN exit_flag;\nEND LOOP;\nRAISE NOTICE 'Scanned through % partitions.', batches;\n\n\n-- Flip things around if this was the result of a prev token query\nIF token_type='prev' THEN\n out_records := flip_jsonb_array(out_records);\n first_record := last_record;\nEND IF;\n\n-- If this query has a token, see if there is data before the first record\nIF _search ? 'token' THEN\n prev_query := format(\n 'SELECT 1 FROM items WHERE %s LIMIT 1',\n concat_ws(\n ' AND ',\n _where,\n trim(get_token_filter(_search, to_jsonb(first_record)))\n )\n );\n RAISE NOTICE 'Query to get previous record: % --- %', prev_query, first_record;\n EXECUTE prev_query INTO has_prev;\n IF FOUND and has_prev IS NOT NULL THEN\n RAISE NOTICE 'Query results from prev query: %', has_prev;\n has_prev := TRUE;\n END IF;\nEND IF;\nhas_prev := COALESCE(has_prev, FALSE);\n\nRAISE NOTICE 'token_type: %, has_next: %, has_prev: %', token_type, has_next, has_prev;\nIF has_prev THEN\n prev := out_records->0->>'id';\nEND IF;\nIF has_next OR token_type='prev' THEN\n next := out_records->-1->>'id';\nEND IF;\n\n\n\n-- include/exclude any fields following fields extension\nIF _search ? 'fields' THEN\n IF _search->'fields' ? 'exclude' THEN\n excludes=textarr(_search->'fields'->'exclude');\n END IF;\n IF _search->'fields' ? 'include' THEN\n includes=textarr(_search->'fields'->'include');\n IF array_length(includes, 1)>0 AND NOT 'id' = ANY (includes) THEN\n includes = includes || '{id}';\n END IF;\n END IF;\n SELECT jsonb_agg(filter_jsonb(row, includes, excludes)) INTO out_records FROM jsonb_array_elements(out_records) row;\nEND IF;\n\n\ncontext := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'matched', total_count,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n));\n\ncollection := jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', out_records,\n 'next', next,\n 'prev', prev,\n 'context', context\n);\n\nRETURN collection;\nEND;\n$$ LANGUAGE PLPGSQL SET jit TO off;\nINSERT INTO pgstac.migrations (version) VALUES ('0.3.2');\n" }, { "path": "src/pgstac/migrations/pgstac.0.3.3-0.3.4.sql", "content": "SET SEARCH_PATH to pgstac, public;\nset check_function_bodies = off;\n\nCREATE OR REPLACE FUNCTION pgstac.create_items(data jsonb)\n RETURNS void\n LANGUAGE sql\n SET search_path TO 'pgstac', 'public'\nAS $function$\n INSERT INTO items_staging (content)\n SELECT * FROM jsonb_array_elements(data);\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.ftime()\n RETURNS interval\n LANGUAGE sql\nAS $function$\nSELECT age(clock_timestamp(), transaction_timestamp());\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.geojsonsearch(geojson jsonb, queryhash text, fields jsonb DEFAULT NULL::jsonb, _scanlimit integer DEFAULT 10000, _limit integer DEFAULT 100, _timelimit interval DEFAULT '00:00:05'::interval, exitwhenfull boolean DEFAULT true, skipcovered boolean DEFAULT true)\n RETURNS jsonb\n LANGUAGE sql\nAS $function$\n SELECT * FROM geometrysearch(\n st_geomfromgeojson(geojson),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.geometrysearch(geom geometry, queryhash text, fields jsonb DEFAULT NULL::jsonb, _scanlimit integer DEFAULT 10000, _limit integer DEFAULT 100, _timelimit interval DEFAULT '00:00:05'::interval, exitwhenfull boolean DEFAULT true, skipcovered boolean DEFAULT true)\n RETURNS jsonb\n LANGUAGE plpgsql\nAS $function$\nDECLARE\n search searches%ROWTYPE;\n curs refcursor;\n _where text;\n query text;\n iter_record items%ROWTYPE;\n out_records jsonb[] := '{}'::jsonb[];\n exit_flag boolean := FALSE;\n counter int := 1;\n scancounter int := 1;\n remaining_limit int := _scanlimit;\n tilearea float;\n unionedgeom geometry;\n clippedgeom geometry;\n unionedgeom_area float := 0;\n prev_area float := 0;\n excludes text[];\n includes text[];\n\nBEGIN\n -- If skipcovered is true then you will always want to exit when the passed in geometry is full\n IF skipcovered THEN\n exitwhenfull := TRUE;\n END IF;\n\n SELECT * INTO search FROM searches WHERE hash=queryhash;\n\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Search with Query Hash % Not Found', queryhash;\n END IF;\n\n tilearea := st_area(geom);\n _where := format('%s AND st_intersects(geometry, %L::geometry)', search._where, geom);\n\n IF fields IS NOT NULL THEN\n IF fields ? 'fields' THEN\n fields := fields->'fields';\n END IF;\n IF fields ? 'exclude' THEN\n excludes=textarr(fields->'exclude');\n END IF;\n IF fields ? 'include' THEN\n includes=textarr(fields->'include');\n IF array_length(includes, 1)>0 AND NOT 'id' = ANY (includes) THEN\n includes = includes || '{id}';\n END IF;\n END IF;\n END IF;\n RAISE NOTICE 'fields: %, includes: %, excludes: %', fields, includes, excludes;\n\n FOR query IN SELECT * FROM partition_queries(_where, search.orderby) LOOP\n query := format('%s LIMIT %L', query, remaining_limit);\n RAISE NOTICE '%', query;\n curs = create_cursor(query);\n LOOP\n FETCH curs INTO iter_record;\n EXIT WHEN NOT FOUND;\n IF exitwhenfull OR skipcovered THEN -- If we are not using exitwhenfull or skipcovered, we do not need to do expensive geometry operations\n clippedgeom := st_intersection(geom, iter_record.geometry);\n\n IF unionedgeom IS NULL THEN\n unionedgeom := clippedgeom;\n ELSE\n unionedgeom := st_union(unionedgeom, clippedgeom);\n END IF;\n\n unionedgeom_area := st_area(unionedgeom);\n\n IF skipcovered AND prev_area = unionedgeom_area THEN\n scancounter := scancounter + 1;\n CONTINUE;\n END IF;\n\n prev_area := unionedgeom_area;\n\n RAISE NOTICE '% % % %', unionedgeom_area/tilearea, counter, scancounter, ftime();\n END IF;\n\n IF fields IS NOT NULL THEN\n out_records := out_records || filter_jsonb(iter_record.content, includes, excludes);\n ELSE\n out_records := out_records || iter_record.content;\n END IF;\n IF counter >= _limit\n OR scancounter > _scanlimit\n OR ftime() > _timelimit\n OR (exitwhenfull AND unionedgeom_area >= tilearea)\n THEN\n exit_flag := TRUE;\n EXIT;\n END IF;\n counter := counter + 1;\n scancounter := scancounter + 1;\n\n END LOOP;\n EXIT WHEN exit_flag;\n remaining_limit := _scanlimit - scancounter;\n END LOOP;\n\n RETURN jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', array_to_json(out_records)::jsonb\n );\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.tileenvelope(zoom integer, x integer, y integer)\n RETURNS geometry\n LANGUAGE sql\n IMMUTABLE PARALLEL SAFE\nAS $function$\nWITH t AS (\n SELECT\n 20037508.3427892 as merc_max,\n -20037508.3427892 as merc_min,\n (2 * 20037508.3427892) / (2 ^ zoom) as tile_size\n)\nSELECT st_makeenvelope(\n merc_min + (tile_size * x),\n merc_max - (tile_size * (y + 1)),\n merc_min + (tile_size * (x + 1)),\n merc_max - (tile_size * y),\n 3857\n) FROM t;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.upsert_items(data jsonb)\n RETURNS void\n LANGUAGE sql\n SET search_path TO 'pgstac', 'public'\nAS $function$\n INSERT INTO items_staging_upsert (content)\n SELECT * FROM jsonb_array_elements(data);\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.xyzsearch(_x integer, _y integer, _z integer, queryhash text, fields jsonb DEFAULT NULL::jsonb, _scanlimit integer DEFAULT 10000, _limit integer DEFAULT 100, _timelimit interval DEFAULT '00:00:05'::interval, exitwhenfull boolean DEFAULT true, skipcovered boolean DEFAULT true)\n RETURNS jsonb\n LANGUAGE sql\nAS $function$\n SELECT * FROM geometrysearch(\n st_transform(tileenvelope(_z, _x, _y), 4326),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$function$\n;\n\n\n\nINSERT INTO migrations (version) VALUES ('0.3.4');\n" }, { "path": "src/pgstac/migrations/pgstac.0.3.3.sql", "content": "CREATE EXTENSION IF NOT EXISTS postgis;\nCREATE SCHEMA IF NOT EXISTS pgstac;\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE TABLE migrations (\n version text,\n datetime timestamptz DEFAULT now() NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION notice(VARIADIC text[]) RETURNS boolean AS $$\nDECLARE\ndebug boolean := current_setting('pgstac.debug', true);\nBEGIN\n IF debug THEN\n RAISE NOTICE 'NOTICE FROM FUNC: % >>>>> %', concat_ws(' | ', $1), clock_timestamp();\n RETURN TRUE;\n END IF;\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION empty_arr(ANYARRAY) RETURNS BOOLEAN AS $$\nSELECT CASE\n WHEN $1 IS NULL THEN TRUE\n WHEN cardinality($1)<1 THEN TRUE\nELSE FALSE\nEND;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_map_ident(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_ident(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_map_literal(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_literal(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION estimated_count(_where text) RETURNS bigint AS $$\nDECLARE\nrec record;\nrows bigint;\nBEGIN\n FOR rec in EXECUTE format(\n $q$\n EXPLAIN SELECT 1 FROM items WHERE %s\n $q$,\n _where)\n LOOP\n rows := substring(rec.\"QUERY PLAN\" FROM ' rows=([[:digit:]]+)');\n EXIT WHEN rows IS NOT NULL;\n END LOOP;\n\n RETURN rows;\nEND;\n$$ LANGUAGE PLPGSQL;\n/* converts a jsonb text array to a pg text[] array */\nCREATE OR REPLACE FUNCTION textarr(_js jsonb)\n RETURNS text[] AS $$\n SELECT\n CASE jsonb_typeof(_js)\n WHEN 'array' THEN ARRAY(SELECT jsonb_array_elements_text(_js))\n ELSE ARRAY[_js->>0]\n END\n;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION jsonb_paths (IN jdata jsonb, OUT path text[], OUT value jsonb) RETURNS\nSETOF RECORD AS $$\nwith recursive extract_all as\n(\n select\n ARRAY[key]::text[] as path,\n value\n FROM jsonb_each(jdata)\nunion all\n select\n path || coalesce(obj_key, (arr_key- 1)::text),\n coalesce(obj_value, arr_value)\n from extract_all\n left join lateral\n jsonb_each(case jsonb_typeof(value) when 'object' then value end)\n as o(obj_key, obj_value)\n on jsonb_typeof(value) = 'object'\n left join lateral\n jsonb_array_elements(case jsonb_typeof(value) when 'array' then value end)\n with ordinality as a(arr_value, arr_key)\n on jsonb_typeof(value) = 'array'\n where obj_key is not null or arr_key is not null\n)\nselect *\nfrom extract_all;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION jsonb_obj_paths (IN jdata jsonb, OUT path text[], OUT value jsonb) RETURNS\nSETOF RECORD AS $$\nwith recursive extract_all as\n(\n select\n ARRAY[key]::text[] as path,\n value\n FROM jsonb_each(jdata)\nunion all\n select\n path || obj_key,\n obj_value\n from extract_all\n left join lateral\n jsonb_each(case jsonb_typeof(value) when 'object' then value end)\n as o(obj_key, obj_value)\n on jsonb_typeof(value) = 'object'\n where obj_key is not null\n)\nselect *\nfrom extract_all;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION jsonb_val_paths (IN jdata jsonb, OUT path text[], OUT value jsonb) RETURNS\nSETOF RECORD AS $$\nSELECT * FROM jsonb_obj_paths(jdata) WHERE jsonb_typeof(value) not in ('object','array');\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION path_includes(IN path text[], IN includes text[]) RETURNS BOOLEAN AS $$\nWITH t AS (SELECT unnest(includes) i)\nSELECT EXISTS (\n SELECT 1 FROM t WHERE path @> string_to_array(trim(i), '.')\n);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION path_excludes(IN path text[], IN excludes text[]) RETURNS BOOLEAN AS $$\nWITH t AS (SELECT unnest(excludes) e)\nSELECT NOT EXISTS (\n SELECT 1 FROM t WHERE path @> string_to_array(trim(e), '.')\n);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION jsonb_obj_paths_filtered (\n IN jdata jsonb,\n IN includes text[] DEFAULT ARRAY[]::text[],\n IN excludes text[] DEFAULT ARRAY[]::text[],\n OUT path text[],\n OUT value jsonb\n) RETURNS\nSETOF RECORD AS $$\nSELECT path, value\nFROM jsonb_obj_paths(jdata)\nWHERE\n CASE WHEN cardinality(includes) > 0 THEN path_includes(path, includes) ELSE TRUE END\n AND\n path_excludes(path, excludes)\n\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION filter_jsonb(\n IN jdata jsonb,\n IN includes text[] DEFAULT ARRAY[]::text[],\n IN excludes text[] DEFAULT ARRAY[]::text[]\n) RETURNS jsonb AS $$\nDECLARE\nrec RECORD;\noutj jsonb := '{}'::jsonb;\ncreated_paths text[] := '{}'::text[];\nBEGIN\n\nIF empty_arr(includes) AND empty_arr(excludes) THEN\nRAISE NOTICE 'no filter';\n RETURN jdata;\nEND IF;\nFOR rec in\nSELECT * FROM jsonb_obj_paths_filtered(jdata, includes, excludes)\nWHERE jsonb_typeof(value) != 'object'\nLOOP\n IF array_length(rec.path,1)>1 THEN\n FOR i IN 1..(array_length(rec.path,1)-1) LOOP\n IF NOT array_to_string(rec.path[1:i],'.') = ANY (created_paths) THEN\n outj := jsonb_set(outj, rec.path[1:i],'{}', true);\n created_paths := created_paths || array_to_string(rec.path[1:i],'.');\n END IF;\n END LOOP;\n END IF;\n outj := jsonb_set(outj, rec.path, rec.value, true);\n created_paths := created_paths || array_to_string(rec.path,'.');\nEND LOOP;\nRETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION flip_jsonb_array(j jsonb) RETURNS jsonb AS $$\nSELECT jsonb_agg(value) FROM (SELECT value FROM jsonb_array_elements(j) WITH ORDINALITY ORDER BY ordinality DESC) as t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n/* Functions to create an iterable of cursors over partitions. */\nCREATE OR REPLACE FUNCTION create_cursor(q text) RETURNS refcursor AS $$\nDECLARE\n curs refcursor;\nBEGIN\n OPEN curs FOR EXECUTE q;\n RETURN curs;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION partition_queries(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC'\n) RETURNS SETOF text AS $$\nDECLARE\n partition_query text;\n query text;\n p record;\n cursors refcursor;\nBEGIN\nIF _orderby ILIKE 'datetime d%' THEN\n partition_query := format($q$\n SELECT partition, tstzrange\n FROM items_partitions\n ORDER BY tstzrange DESC;\n $q$);\nELSIF _orderby ILIKE 'datetime a%' THEN\n partition_query := format($q$\n SELECT partition, tstzrange\n FROM items_partitions\n ORDER BY tstzrange ASC\n ;\n $q$);\nELSE\n query := format($q$\n SELECT * FROM items\n WHERE %s\n ORDER BY %s\n $q$, _where, _orderby\n );\n\n RETURN NEXT query;\n RETURN;\nEND IF;\nFOR p IN\n EXECUTE partition_query\nLOOP\n query := format($q$\n SELECT * FROM items\n WHERE datetime >= %L AND datetime < %L AND %s\n ORDER BY %s\n $q$, lower(p.tstzrange), upper(p.tstzrange), _where, _orderby\n );\n RETURN NEXT query;\nEND LOOP;\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION partition_cursor(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC'\n) RETURNS SETOF refcursor AS $$\nDECLARE\n partition_query text;\n query text;\n p record;\n cursors refcursor;\nBEGIN\nFOR query IN SELECT * FROM partion_queries(_where, _orderby) LOOP\n RETURN NEXT create_cursor(query);\nEND LOOP;\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION partition_count(\n IN _where text DEFAULT 'TRUE'\n) RETURNS bigint AS $$\nDECLARE\n partition_query text;\n query text;\n p record;\n subtotal bigint;\n total bigint := 0;\nBEGIN\npartition_query := format($q$\n SELECT partition, tstzrange\n FROM items_partitions\n ORDER BY tstzrange DESC;\n$q$);\nRAISE NOTICE 'Partition Query: %', partition_query;\nFOR p IN\n EXECUTE partition_query\nLOOP\n query := format($q$\n SELECT count(*) FROM items\n WHERE datetime BETWEEN %L AND %L AND %s\n $q$, lower(p.tstzrange), upper(p.tstzrange), _where\n );\n RAISE NOTICE 'Query %', query;\n RAISE NOTICE 'Partition %, Count %, Total %',p.partition, subtotal, total;\n EXECUTE query INTO subtotal;\n total := subtotal + total;\nEND LOOP;\nRETURN total;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n/* looks for a geometry in a stac item first from geometry and falling back to bbox */\nCREATE OR REPLACE FUNCTION stac_geom(value jsonb) RETURNS geometry AS $$\nSELECT\n CASE\n WHEN value->>'geometry' IS NOT NULL THEN\n ST_GeomFromGeoJSON(value->>'geometry')\n WHEN value->>'bbox' IS NOT NULL THEN\n ST_MakeEnvelope(\n (value->'bbox'->>0)::float,\n (value->'bbox'->>1)::float,\n (value->'bbox'->>2)::float,\n (value->'bbox'->>3)::float,\n 4326\n )\n ELSE NULL\n END as geometry\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION stac_datetime(value jsonb) RETURNS timestamptz AS $$\nSELECT COALESCE(\n (value->'properties'->>'datetime')::timestamptz,\n (value->'properties'->>'start_datetime')::timestamptz\n);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_end_datetime(value jsonb) RETURNS timestamptz AS $$\nSELECT COALESCE(\n (value->'properties'->>'datetime')::timestamptz,\n (value->'properties'->>'end_datetime')::timestamptz\n);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\n\nCREATE OR REPLACE FUNCTION stac_daterange(value jsonb) RETURNS tstzrange AS $$\nSELECT tstzrange(stac_datetime(value),stac_end_datetime(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE TABLE IF NOT EXISTS collections (\n id VARCHAR GENERATED ALWAYS AS (content->>'id') STORED PRIMARY KEY,\n content JSONB\n);\n\nCREATE OR REPLACE FUNCTION create_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection(data jsonb) RETURNS VOID AS $$\nDECLARE\nout collections%ROWTYPE;\nBEGIN\n UPDATE collections SET content=data WHERE id = data->>'id' RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ON CONFLICT (id) DO\n UPDATE\n SET content=EXCLUDED.content\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION delete_collection(_id text) RETURNS VOID AS $$\nDECLARE\nout collections%ROWTYPE;\nBEGIN\n DELETE FROM collections WHERE id = _id RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION get_collection(id text) RETURNS jsonb AS $$\nSELECT content FROM collections\nWHERE id=$1\n;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION all_collections() RETURNS jsonb AS $$\nSELECT jsonb_agg(content) FROM collections;\n;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE TABLE items (\n id text NOT NULL,\n geometry geometry NOT NULL,\n collection_id text NOT NULL,\n datetime timestamptz NOT NULL,\n end_datetime timestamptz NOT NULL,\n properties jsonb NOT NULL,\n content JSONB NOT NULL\n)\nPARTITION BY RANGE (datetime)\n;\n\nCREATE OR REPLACE FUNCTION properties_idx (IN content jsonb) RETURNS jsonb AS $$\n with recursive extract_all as\n (\n select\n ARRAY[key]::text[] as path,\n ARRAY[key]::text[] as fullpath,\n value\n FROM jsonb_each(content->'properties')\n union all\n select\n CASE WHEN obj_key IS NOT NULL THEN path || obj_key ELSE path END,\n path || coalesce(obj_key, (arr_key- 1)::text),\n coalesce(obj_value, arr_value)\n from extract_all\n left join lateral\n jsonb_each(case jsonb_typeof(value) when 'object' then value end)\n as o(obj_key, obj_value)\n on jsonb_typeof(value) = 'object'\n left join lateral\n jsonb_array_elements(case jsonb_typeof(value) when 'array' then value end)\n with ordinality as a(arr_value, arr_key)\n on jsonb_typeof(value) = 'array'\n where obj_key is not null or arr_key is not null\n )\n , paths AS (\n select\n array_to_string(path, '.') as path,\n value\n FROM extract_all\n WHERE\n jsonb_typeof(value) NOT IN ('array','object')\n ), grouped AS (\n SELECT path, jsonb_agg(distinct value) vals FROM paths group by path\n ) SELECT coalesce(jsonb_object_agg(path, CASE WHEN jsonb_array_length(vals)=1 THEN vals->0 ELSE vals END) - '{datetime}'::text[], '{}'::jsonb) FROM grouped\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET JIT TO OFF;\n\nCREATE INDEX \"datetime_idx\" ON items (datetime);\nCREATE INDEX \"end_datetime_idx\" ON items (end_datetime);\nCREATE INDEX \"properties_idx\" ON items USING GIN (properties jsonb_path_ops);\nCREATE INDEX \"collection_idx\" ON items (collection_id);\nCREATE INDEX \"geometry_idx\" ON items USING GIST (geometry);\nCREATE UNIQUE INDEX \"items_id_datetime_idx\" ON items (datetime, id);\n\nALTER TABLE items ADD CONSTRAINT items_collections_fk FOREIGN KEY (collection_id) REFERENCES collections(id) ON DELETE CASCADE DEFERRABLE;\n\nCREATE OR REPLACE FUNCTION analyze_empty_partitions() RETURNS VOID AS $$\nDECLARE\n p text;\nBEGIN\n FOR p IN SELECT partition FROM all_items_partitions WHERE est_cnt = 0 LOOP\n EXECUTE format('ANALYZE %I;', p);\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION items_partition_name(timestamptz) RETURNS text AS $$\n SELECT to_char($1, '\"items_p\"IYYY\"w\"IW');\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION items_partition_exists(text) RETURNS boolean AS $$\n SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_class WHERE relname=$1);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION items_partition_exists(timestamptz) RETURNS boolean AS $$\n SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_class WHERE relname=items_partition_name($1));\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION items_partition_create_worker(partition text, partition_start timestamptz, partition_end timestamptz) RETURNS VOID AS $$\nDECLARE\n err_context text;\nBEGIN\n EXECUTE format(\n $f$\n CREATE TABLE IF NOT EXISTS %1$I PARTITION OF items\n FOR VALUES FROM (%2$L) TO (%3$L);\n CREATE UNIQUE INDEX IF NOT EXISTS %4$I ON %1$I (id);\n $f$,\n partition,\n partition_start,\n partition_end,\n concat(partition, '_id_pk')\n );\nEXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', partition;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH to pgstac, public;\n\nCREATE OR REPLACE FUNCTION items_partition_create(ts timestamptz) RETURNS text AS $$\nDECLARE\n partition text := items_partition_name(ts);\n partition_start timestamptz;\n partition_end timestamptz;\nBEGIN\n IF items_partition_exists(partition) THEN\n RETURN partition;\n END IF;\n partition_start := date_trunc('week', ts);\n partition_end := partition_start + '1 week'::interval;\n PERFORM items_partition_create_worker(partition, partition_start, partition_end);\n RAISE NOTICE 'partition: %', partition;\n RETURN partition;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION items_partition_create(st timestamptz, et timestamptz) RETURNS SETOF text AS $$\nWITH t AS (\n SELECT\n generate_series(\n date_trunc('week',st),\n date_trunc('week', et),\n '1 week'::interval\n ) w\n)\nSELECT items_partition_create(w) FROM t;\n$$ LANGUAGE SQL;\n\n\nCREATE UNLOGGED TABLE items_staging (\n content JSONB NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION items_staging_insert_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n mindate timestamptz;\n maxdate timestamptz;\n partition text;\nBEGIN\n SELECT min(stac_datetime(content)), max(stac_datetime(content)) INTO mindate, maxdate FROM newdata;\n PERFORM items_partition_create(mindate, maxdate);\n INSERT INTO items (id, geometry, collection_id, datetime, end_datetime, properties, content)\n SELECT\n content->>'id',\n stac_geom(content),\n content->>'collection',\n stac_datetime(content),\n stac_end_datetime(content),\n properties_idx(content),\n content\n FROM newdata\n ;\n DELETE FROM items_staging;\n PERFORM analyze_empty_partitions();\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER items_staging_insert_trigger AFTER INSERT ON items_staging REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_insert_triggerfunc();\n\n\nCREATE UNLOGGED TABLE items_staging_ignore (\n content JSONB NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION items_staging_ignore_insert_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n mindate timestamptz;\n maxdate timestamptz;\n partition text;\nBEGIN\n SELECT min(stac_datetime(content)), max(stac_datetime(content)) INTO mindate, maxdate FROM newdata;\n PERFORM items_partition_create(mindate, maxdate);\n INSERT INTO items (id, geometry, collection_id, datetime, end_datetime, properties, content)\n SELECT\n content->>'id',\n stac_geom(content),\n content->>'collection',\n stac_datetime(content),\n stac_end_datetime(content),\n properties_idx(content),\n content\n FROM newdata\n ON CONFLICT DO NOTHING\n ;\n DELETE FROM items_staging_ignore;\n PERFORM analyze_empty_partitions();\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER items_staging_ignore_insert_trigger AFTER INSERT ON items_staging_ignore REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_ignore_insert_triggerfunc();\n\nCREATE UNLOGGED TABLE items_staging_upsert (\n content JSONB NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION items_staging_upsert_insert_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n mindate timestamptz;\n maxdate timestamptz;\n partition text;\nBEGIN\n SELECT min(stac_datetime(content)), max(stac_datetime(content)) INTO mindate, maxdate FROM newdata;\n PERFORM items_partition_create(mindate, maxdate);\n INSERT INTO items (id, geometry, collection_id, datetime, end_datetime, properties, content)\n SELECT\n content->>'id',\n stac_geom(content),\n content->>'collection',\n stac_datetime(content),\n stac_end_datetime(content),\n properties_idx(content),\n content\n FROM newdata\n ON CONFLICT (datetime, id) DO UPDATE SET\n content = EXCLUDED.content\n WHERE items.content IS DISTINCT FROM EXCLUDED.content\n ;\n DELETE FROM items_staging_upsert;\n PERFORM analyze_empty_partitions();\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER items_staging_upsert_insert_trigger AFTER INSERT ON items_staging_upsert REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_upsert_insert_triggerfunc();\n\nCREATE OR REPLACE FUNCTION items_update_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\nBEGIN\n NEW.id := NEW.content->>'id';\n NEW.datetime := stac_datetime(NEW.content);\n NEW.end_datetime := stac_end_datetime(NEW.content);\n NEW.collection_id := NEW.content->>'collection';\n NEW.geometry := stac_geom(NEW.content);\n NEW.properties := properties_idx(NEW.content);\n IF TG_OP = 'UPDATE' AND NEW IS NOT DISTINCT FROM OLD THEN\n RETURN NULL;\n END IF;\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER items_update_trigger BEFORE UPDATE ON items\n FOR EACH ROW EXECUTE PROCEDURE items_update_triggerfunc();\n\n/*\nView to get a table of available items partitions\nwith date ranges\n*/\nCREATE VIEW all_items_partitions AS\nWITH base AS\n(SELECT\n c.oid::pg_catalog.regclass::text as partition,\n pg_catalog.pg_get_expr(c.relpartbound, c.oid) as _constraint,\n regexp_matches(\n pg_catalog.pg_get_expr(c.relpartbound, c.oid),\n E'\\\\(''\\([0-9 :+-]*\\)''\\\\).*\\\\(''\\([0-9 :+-]*\\)''\\\\)'\n ) as t,\n reltuples::bigint as est_cnt\nFROM pg_catalog.pg_class c, pg_catalog.pg_inherits i\nWHERE c.oid = i.inhrelid AND i.inhparent = 'items'::regclass)\nSELECT partition, tstzrange(\n t[1]::timestamptz,\n t[2]::timestamptz\n), est_cnt\nFROM base\nORDER BY 2 desc;\n\nCREATE OR REPLACE VIEW items_partitions AS\nSELECT * FROM all_items_partitions WHERE est_cnt>0;\n\nCREATE OR REPLACE FUNCTION get_item(_id text) RETURNS jsonb AS $$\n SELECT content FROM items WHERE id=_id;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION delete_item(_id text) RETURNS VOID AS $$\nDECLARE\nout items%ROWTYPE;\nBEGIN\n DELETE FROM items WHERE id = _id RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION create_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION update_item(data jsonb) RETURNS VOID AS $$\nDECLARE\n out items%ROWTYPE;\nBEGIN\n UPDATE items SET content=data WHERE id = data->>'id' RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION collection_bbox(id text) RETURNS jsonb AS $$\nSELECT (replace(replace(replace(st_extent(geometry)::text,'BOX(','[['),')',']]'),' ',','))::jsonb\nFROM items WHERE collection_id=$1;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION collection_temporal_extent(id text) RETURNS jsonb AS $$\nSELECT to_jsonb(array[array[min(datetime)::text, max(datetime)::text]])\nFROM items WHERE collection_id=$1;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection_extents() RETURNS VOID AS $$\nUPDATE collections SET\n content = content ||\n jsonb_build_object(\n 'extent', jsonb_build_object(\n 'spatial', jsonb_build_object(\n 'bbox', collection_bbox(collections.id)\n ),\n 'temporal', jsonb_build_object(\n 'interval', collection_temporal_extent(collections.id)\n )\n )\n )\n;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION items_path(\n IN dotpath text,\n OUT field text,\n OUT path text,\n OUT path_txt text,\n OUT jsonpath text,\n OUT eq text\n) RETURNS RECORD AS $$\nDECLARE\npath_elements text[];\nlast_element text;\nBEGIN\ndotpath := replace(trim(dotpath), 'properties.', '');\n\nIF dotpath = '' THEN\n RETURN;\nEND IF;\n\npath_elements := string_to_array(dotpath, '.');\njsonpath := NULL;\n\nIF path_elements[1] IN ('id','geometry','datetime') THEN\n field := path_elements[1];\n path_elements := path_elements[2:];\nELSIF path_elements[1] = 'collection' THEN\n field := 'collection_id';\n path_elements := path_elements[2:];\nELSIF path_elements[1] IN ('links', 'assets', 'stac_version', 'stac_extensions') THEN\n field := 'content';\nELSE\n field := 'content';\n path_elements := '{properties}'::text[] || path_elements;\nEND IF;\nIF cardinality(path_elements)<1 THEN\n path := field;\n path_txt := field;\n jsonpath := '$';\n eq := NULL; -- format($F$ %s = %%s $F$, field);\n RETURN;\nEND IF;\n\n\nlast_element := path_elements[cardinality(path_elements)];\npath_elements := path_elements[1:cardinality(path_elements)-1];\njsonpath := concat(array_to_string('{$}'::text[] || array_map_ident(path_elements), '.'), '.', quote_ident(last_element));\npath_elements := array_map_literal(path_elements);\npath := format($F$ properties->%s $F$, quote_literal(dotpath));\npath_txt := format($F$ properties->>%s $F$, quote_literal(dotpath));\neq := format($F$ properties @? '$.%s[*] ? (@ == %%s) '$F$, quote_ident(dotpath));\n\nRAISE NOTICE 'ITEMS PATH -- % % % % %', field, path, path_txt, jsonpath, eq;\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION parse_dtrange(IN _indate jsonb, OUT _tstzrange tstzrange) AS $$\nWITH t AS (\n SELECT CASE\n WHEN jsonb_typeof(_indate) = 'array' THEN\n textarr(_indate)\n ELSE\n regexp_split_to_array(\n btrim(_indate::text,'\"'),\n '/'\n )\n END AS arr\n)\n, t1 AS (\n SELECT\n CASE\n WHEN array_upper(arr,1) = 1 OR arr[1] = '..' OR arr[1] IS NULL THEN '-infinity'::timestamptz\n ELSE arr[1]::timestamptz\n END AS st,\n CASE\n WHEN array_upper(arr,1) = 1 THEN arr[1]::timestamptz\n WHEN arr[2] = '..' OR arr[2] IS NULL THEN 'infinity'::timestamptz\n ELSE arr[2]::timestamptz\n END AS et\n FROM t\n)\nSELECT\n tstzrange(st,et)\nFROM t1;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION bbox_geom(_bbox jsonb) RETURNS geometry AS $$\nSELECT CASE jsonb_array_length(_bbox)\n WHEN 4 THEN\n ST_SetSRID(ST_MakeEnvelope(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float,\n (_bbox->>3)::float\n ),4326)\n WHEN 6 THEN\n ST_SetSRID(ST_3DMakeBox(\n ST_MakePoint(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float\n ),\n ST_MakePoint(\n (_bbox->>3)::float,\n (_bbox->>4)::float,\n (_bbox->>5)::float\n )\n ),4326)\n ELSE null END;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION cql_and_append(existing jsonb, newfilters jsonb) RETURNS jsonb AS $$\nSELECT CASE WHEN existing ? 'filter' AND newfilters IS NOT NULL THEN\n jsonb_build_object(\n 'and',\n jsonb_build_array(\n existing->'filter',\n newfilters\n )\n )\nELSE\n newfilters\nEND;\n$$ LANGUAGE SQL;\n\n\n-- ADDs base filters (ids, collections, datetime, bbox, intersects) that are\n-- added outside of the filter/query in the stac request\nCREATE OR REPLACE FUNCTION add_filters_to_cql(j jsonb) RETURNS jsonb AS $$\nDECLARE\nnewprop jsonb;\nnewprops jsonb := '[]'::jsonb;\nBEGIN\nIF j ? 'ids' THEN\n newprop := jsonb_build_object(\n 'in',\n jsonb_build_array(\n '{\"property\":\"id\"}'::jsonb,\n j->'ids'\n )\n );\n newprops := jsonb_insert(newprops, '{1}', newprop);\nEND IF;\nIF j ? 'collections' THEN\n newprop := jsonb_build_object(\n 'in',\n jsonb_build_array(\n '{\"property\":\"collection\"}'::jsonb,\n j->'collections'\n )\n );\n newprops := jsonb_insert(newprops, '{1}', newprop);\nEND IF;\n\nIF j ? 'datetime' THEN\n newprop := format(\n '{\"anyinteracts\":[{\"property\":\"datetime\"}, %s]}',\n j->'datetime'\n );\n newprops := jsonb_insert(newprops, '{1}', newprop);\nEND IF;\n\nIF j ? 'bbox' THEN\n newprop := format(\n '{\"intersects\":[{\"property\":\"geometry\"}, %s]}',\n j->'bbox'\n );\n newprops := jsonb_insert(newprops, '{1}', newprop);\nEND IF;\n\nIF j ? 'intersects' THEN\n newprop := format(\n '{\"intersects\":[{\"property\":\"geometry\"}, %s]}',\n j->'intersects'\n );\n newprops := jsonb_insert(newprops, '{1}', newprop);\nEND IF;\n\nRAISE NOTICE 'newprops: %', newprops;\n\nIF newprops IS NOT NULL AND jsonb_array_length(newprops) > 0 THEN\n return jsonb_set(\n j,\n '{filter}',\n cql_and_append(j, jsonb_build_object('and', newprops))\n ) - '{ids,collections,datetime,bbox,intersects}'::text[];\nEND IF;\n\nreturn j;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION query_to_cqlfilter(j jsonb) RETURNS jsonb AS $$\n-- Translates anything passed in through the deprecated \"query\" into equivalent CQL\nWITH t AS (\n SELECT key as property, value as ops\n FROM jsonb_each(j->'query')\n), t2 AS (\n SELECT property, (jsonb_each(ops)).*\n FROM t WHERE jsonb_typeof(ops) = 'object'\n UNION ALL\n SELECT property, 'eq', ops\n FROM t WHERE jsonb_typeof(ops) != 'object'\n), t3 AS (\nSELECT\n jsonb_strip_nulls(jsonb_build_object(\n 'and',\n jsonb_agg(\n jsonb_build_object(\n key,\n jsonb_build_array(\n jsonb_build_object('property',property),\n value\n )\n )\n )\n )) as qcql FROM t2\n)\nSELECT\n CASE WHEN qcql IS NOT NULL THEN\n jsonb_set(j, '{filter}', cql_and_append(j, qcql)) - 'query'\n ELSE j\n END\nFROM t3\n;\n$$ LANGUAGE SQL;\n\n\n\nCREATE OR REPLACE FUNCTION temporal_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\nll text := 'datetime';\nlh text := 'end_datetime';\nrrange tstzrange;\nrl text;\nrh text;\noutq text;\nBEGIN\nrrange := parse_dtrange(args->1);\nRAISE NOTICE 'Constructing temporal query OP: %, ARGS: %, RRANGE: %', op, args, rrange;\nop := lower(op);\nrl := format('%L::timestamptz', lower(rrange));\nrh := format('%L::timestamptz', upper(rrange));\noutq := CASE op\n WHEN 't_before' THEN 'lh < rl'\n WHEN 't_after' THEN 'll > rh'\n WHEN 't_meets' THEN 'lh = rl'\n WHEN 't_metby' THEN 'll = rh'\n WHEN 't_overlaps' THEN 'll < rl AND rl < lh < rh'\n WHEN 't_overlappedby' THEN 'rl < ll < rh AND lh > rh'\n WHEN 't_starts' THEN 'll = rl AND lh < rh'\n WHEN 't_startedby' THEN 'll = rl AND lh > rh'\n WHEN 't_during' THEN 'll > rl AND lh < rh'\n WHEN 't_contains' THEN 'll < rl AND lh > rh'\n WHEN 't_finishes' THEN 'll > rl AND lh = rh'\n WHEN 't_finishedby' THEN 'll < rl AND lh = rh'\n WHEN 't_equals' THEN 'll = rl AND lh = rh'\n WHEN 't_disjoint' THEN 'NOT (ll <= rh AND lh >= rl)'\n WHEN 't_intersects' THEN 'll <= rh AND lh >= rl'\n WHEN 'anyinteracts' THEN 'll <= rh AND lh >= rl'\nEND;\noutq := regexp_replace(outq, '\\mll\\M', ll);\noutq := regexp_replace(outq, '\\mlh\\M', lh);\noutq := regexp_replace(outq, '\\mrl\\M', rl);\noutq := regexp_replace(outq, '\\mrh\\M', rh);\noutq := format('(%s)', outq);\nRETURN outq;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION spatial_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\ngeom text;\nj jsonb := args->1;\nBEGIN\nop := lower(op);\nRAISE NOTICE 'Constructing spatial query OP: %, ARGS: %', op, args;\nIF op NOT IN ('s_equals','s_disjoint','s_touches','s_within','s_overlaps','s_crosses','s_intersects','intersects','s_contains') THEN\n RAISE EXCEPTION 'Spatial Operator % Not Supported', op;\nEND IF;\nop := regexp_replace(op, '^s_', 'st_');\nIF op = 'intersects' THEN\n op := 'st_intersects';\nEND IF;\n-- Convert geometry to WKB string\nIF j ? 'type' AND j ? 'coordinates' THEN\n geom := st_geomfromgeojson(j)::text;\nELSIF jsonb_typeof(j) = 'array' THEN\n geom := bbox_geom(j)::text;\nEND IF;\n\nRETURN format('%s(geometry, %L::geometry)', op, geom);\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n/* cql_query_op -- Parses a CQL query operation, recursing when necessary\n IN jsonb -- a subelement from a valid stac query\n IN text -- the operator being used on elements passed in\n RETURNS a SQL fragment to be used in a WHERE clause\n*/\nCREATE OR REPLACE FUNCTION cql_query_op(j jsonb, _op text DEFAULT NULL) RETURNS text AS $$\nDECLARE\njtype text := jsonb_typeof(j);\nop text := lower(_op);\nops jsonb :=\n '{\n \"eq\": \"%s = %s\",\n \"lt\": \"%s < %s\",\n \"lte\": \"%s <= %s\",\n \"gt\": \"%s > %s\",\n \"gte\": \"%s >= %s\",\n \"like\": \"%s LIKE %s\",\n \"+\": \"%s + %s\",\n \"-\": \"%s - %s\",\n \"*\": \"%s * %s\",\n \"/\": \"%s / %s\",\n \"in\": \"%s = ANY (%s)\",\n \"not\": \"NOT (%s)\",\n \"between\": \"%s BETWEEN %s AND %s\",\n \"lower\":\"lower(%s)\"\n }'::jsonb;\nret text;\nargs text[] := NULL;\n\nBEGIN\nRAISE NOTICE 'j: %, op: %, jtype: %', j, op, jtype;\n\n-- Set Lower Case on Both Arguments When Case Insensitive Flag Set\nIF op in ('eq','lt','lte','gt','gte','like') AND jsonb_typeof(j->2) = 'boolean' THEN\n IF (j->>2)::boolean THEN\n RETURN format(concat('(',ops->>op,')'), cql_query_op(jsonb_build_array(j->0), 'lower'), cql_query_op(jsonb_build_array(j->1), 'lower'));\n END IF;\nEND IF;\n\n-- Special Case when comparing a property in a jsonb field to a string or number using eq\n-- Allows to leverage GIN index on jsonb fields\nIF op = 'eq' THEN\n IF j->0 ? 'property'\n AND jsonb_typeof(j->1) IN ('number','string')\n AND (items_path(j->0->>'property')).eq IS NOT NULL\n THEN\n RETURN format((items_path(j->0->>'property')).eq, j->1);\n END IF;\nEND IF;\n\nIF op ilike 't_%' or op = 'anyinteracts' THEN\n RETURN temporal_op_query(op, j);\nEND IF;\n\nIF op ilike 's_%' or op = 'intersects' THEN\n RETURN spatial_op_query(op, j);\nEND IF;\n\n\nIF jtype = 'object' THEN\n RAISE NOTICE 'parsing object';\n IF j ? 'property' THEN\n -- Convert the property to be used as an identifier\n return (items_path(j->>'property')).path_txt;\n ELSIF _op IS NULL THEN\n -- Iterate to convert elements in an object where the operator has not been set\n -- Combining with AND\n SELECT\n array_to_string(array_agg(cql_query_op(e.value, e.key)), ' AND ')\n INTO ret\n FROM jsonb_each(j) e;\n RETURN ret;\n END IF;\nEND IF;\n\nIF jtype = 'string' THEN\n RETURN quote_literal(j->>0);\nEND IF;\n\nIF jtype ='number' THEN\n RETURN (j->>0)::numeric;\nEND IF;\n\nIF jtype = 'array' AND op IS NULL THEN\n RAISE NOTICE 'Parsing array into array arg. j: %', j;\n SELECT format($f$ '{%s}'::text[] $f$, string_agg(e,',')) INTO ret FROM jsonb_array_elements_text(j) e;\n RETURN ret;\nEND IF;\n\n\n-- If the type of the passed json is an array\n-- Calculate the arguments that will be passed to functions/operators\nIF jtype = 'array' THEN\n RAISE NOTICE 'Parsing array into args. j: %', j;\n -- If any argument is numeric, cast any text arguments to numeric\n IF j @? '$[*] ? (@.type() == \"number\")' THEN\n SELECT INTO args\n array_agg(concat('(',cql_query_op(e),')::numeric'))\n FROM jsonb_array_elements(j) e;\n ELSE\n SELECT INTO args\n array_agg(cql_query_op(e))\n FROM jsonb_array_elements(j) e;\n END IF;\n --RETURN args;\nEND IF;\nRAISE NOTICE 'ARGS after array cleaning: %', args;\n\nIF op IS NULL THEN\n RETURN args::text[];\nEND IF;\n\nIF args IS NULL OR cardinality(args) < 1 THEN\n RAISE NOTICE 'No Args';\n RETURN '';\nEND IF;\n\nIF op IN ('and','or') THEN\n SELECT\n CONCAT(\n '(',\n array_to_string(args, UPPER(CONCAT(' ',op,' '))),\n ')'\n ) INTO ret\n FROM jsonb_array_elements(j) e;\n RETURN ret;\nEND IF;\n\n-- If the op is in the ops json then run using the template in the json\nIF ops ? op THEN\n RAISE NOTICE 'ARGS: % MAPPED: %',args, array_map_literal(args);\n\n RETURN format(concat('(',ops->>op,')'), VARIADIC args);\nEND IF;\n\nRETURN j->>0;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n\n\nCREATE OR REPLACE FUNCTION cql_to_where(_search jsonb = '{}'::jsonb) RETURNS text AS $$\nDECLARE\nsearch jsonb := _search;\n_where text;\nBEGIN\nRAISE NOTICE 'SEARCH CQL 1: %', search;\n\n-- Convert any old style stac query to cql\nsearch := query_to_cqlfilter(search);\n\nRAISE NOTICE 'SEARCH CQL 2: %', search;\n\n-- Convert item,collection,datetime,bbox,intersects to cql\nsearch := add_filters_to_cql(search);\n\nRAISE NOTICE 'SEARCH CQL Final: %', search;\n_where := cql_query_op(search->'filter');\n\nIF trim(_where) = '' THEN\n _where := NULL;\nEND IF;\n_where := coalesce(_where, ' TRUE ');\nRETURN _where;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION parse_sort_dir(_dir text, reverse boolean default false) RETURNS text AS $$\nWITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n) SELECT\n CASE\n WHEN NOT reverse THEN d\n WHEN d = 'ASC' THEN 'DESC'\n WHEN d = 'DESC' THEN 'ASC'\n END\nFROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION sort_dir_to_op(_dir text, prev boolean default false) RETURNS text AS $$\nWITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n) SELECT\n CASE\n WHEN d = 'ASC' AND prev THEN '<='\n WHEN d = 'DESC' AND prev THEN '>='\n WHEN d = 'ASC' THEN '>='\n WHEN d = 'DESC' THEN '<='\n END\nFROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION sort_sqlorderby(\n _search jsonb DEFAULT NULL,\n reverse boolean DEFAULT FALSE\n) RETURNS text AS $$\nWITH sortby AS (\n SELECT coalesce(_search->'sortby','[{\"field\":\"datetime\", \"direction\":\"desc\"}]') as sort\n), withid AS (\n SELECT CASE\n WHEN sort @? '$[*] ? (@.field == \"id\")' THEN sort\n ELSE sort || '[{\"field\":\"id\", \"direction\":\"desc\"}]'::jsonb\n END as sort\n FROM sortby\n), withid_rows AS (\n SELECT jsonb_array_elements(sort) as value FROM withid\n),sorts AS (\n SELECT\n (items_path(value->>'field')).path as key,\n parse_sort_dir(value->>'direction', reverse) as dir\n FROM withid_rows\n)\nSELECT array_to_string(\n array_agg(concat(key, ' ', dir)),\n ', '\n) FROM sorts;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_sort_dir(sort_item jsonb) RETURNS text AS $$\nSELECT CASE WHEN sort_item->>'direction' ILIKE 'desc%' THEN 'DESC' ELSE 'ASC' END;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION get_token_filter(_search jsonb = '{}'::jsonb, token_rec jsonb DEFAULT NULL) RETURNS text AS $$\nDECLARE\n token_id text;\n filters text[] := '{}'::text[];\n prev boolean := TRUE;\n field text;\n dir text;\n sort record;\n orfilters text[] := '{}'::text[];\n andfilters text[] := '{}'::text[];\n output text;\n token_where text;\nBEGIN\n -- If no token provided return NULL\n IF token_rec IS NULL THEN\n IF NOT (_search ? 'token' AND\n (\n (_search->>'token' ILIKE 'prev:%')\n OR\n (_search->>'token' ILIKE 'next:%')\n )\n ) THEN\n RETURN NULL;\n END IF;\n prev := (_search->>'token' ILIKE 'prev:%');\n token_id := substr(_search->>'token', 6);\n SELECT to_jsonb(items) INTO token_rec FROM items WHERE id=token_id;\n END IF;\n RAISE NOTICE 'TOKEN ID: %', token_rec->'id';\n\n CREATE TEMP TABLE sorts (\n _row int GENERATED ALWAYS AS IDENTITY NOT NULL,\n _field text PRIMARY KEY,\n _dir text NOT NULL,\n _val text\n ) ON COMMIT DROP;\n\n -- Make sure we only have distinct columns to sort with taking the first one we get\n INSERT INTO sorts (_field, _dir)\n SELECT\n (items_path(value->>'field')).path,\n get_sort_dir(value)\n FROM\n jsonb_array_elements(coalesce(_search->'sortby','[{\"field\":\"datetime\",\"direction\":\"desc\"}]'))\n ON CONFLICT DO NOTHING\n ;\n RAISE NOTICE 'sorts 1: %', (SELECT jsonb_agg(to_json(sorts)) FROM sorts);\n -- Get the first sort direction provided. As the id is a primary key, if there are any\n -- sorts after id they won't do anything, so make sure that id is the last sort item.\n SELECT _dir INTO dir FROM sorts ORDER BY _row ASC LIMIT 1;\n IF EXISTS (SELECT 1 FROM sorts WHERE _field = 'id') THEN\n DELETE FROM sorts WHERE _row > (SELECT _row FROM sorts WHERE _field = 'id' ORDER BY _row ASC);\n ELSE\n INSERT INTO sorts (_field, _dir) VALUES ('id', dir);\n END IF;\n\n -- Add value from looked up item to the sorts table\n UPDATE sorts SET _val=quote_literal(token_rec->>_field);\n\n -- Check if all sorts are the same direction and use row comparison\n -- to filter\n RAISE NOTICE 'sorts 2: %', (SELECT jsonb_agg(to_json(sorts)) FROM sorts);\n\n IF (SELECT count(DISTINCT _dir) FROM sorts) = 1 THEN\n SELECT format(\n '(%s) %s (%s)',\n concat_ws(', ', VARIADIC array_agg(quote_ident(_field))),\n CASE WHEN (prev AND dir = 'ASC') OR (NOT prev AND dir = 'DESC') THEN '<' ELSE '>' END,\n concat_ws(', ', VARIADIC array_agg(_val))\n ) INTO output FROM sorts\n WHERE token_rec ? _field\n ;\n ELSE\n FOR sort IN SELECT * FROM sorts ORDER BY _row asc LOOP\n RAISE NOTICE 'SORT: %', sort;\n IF sort._row = 1 THEN\n orfilters := orfilters || format('(%s %s %s)',\n quote_ident(sort._field),\n CASE WHEN (prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC') THEN '<' ELSE '>' END,\n sort._val\n );\n ELSE\n orfilters := orfilters || format('(%s AND %s %s %s)',\n array_to_string(andfilters, ' AND '),\n quote_ident(sort._field),\n CASE WHEN (prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC') THEN '<' ELSE '>' END,\n sort._val\n );\n\n END IF;\n andfilters := andfilters || format('%s = %s',\n quote_ident(sort._field),\n sort._val\n );\n END LOOP;\n output := array_to_string(orfilters, ' OR ');\n END IF;\n DROP TABLE IF EXISTS sorts;\n token_where := concat('(',coalesce(output,'true'),')');\n IF trim(token_where) = '' THEN\n token_where := NULL;\n END IF;\n RAISE NOTICE 'TOKEN_WHERE: |%|',token_where;\n RETURN token_where;\n END;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION search_tohash(jsonb) RETURNS jsonb AS $$\n SELECT $1 - '{token,limit,context,includes,excludes}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION search_hash(jsonb) RETURNS text AS $$\n SELECT md5(search_tohash($1)::text);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE TABLE IF NOT EXISTS searches(\n hash text GENERATED ALWAYS AS (search_hash(search)) STORED PRIMARY KEY,\n search jsonb NOT NULL,\n _where text,\n orderby text,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n statslastupdated timestamptz,\n estimated_count bigint,\n total_count bigint\n);\n\nCREATE OR REPLACE FUNCTION search_query(_search jsonb = '{}'::jsonb, updatestats boolean DEFAULT false) RETURNS searches AS $$\nDECLARE\n search searches%ROWTYPE;\nBEGIN\nINSERT INTO searches (search)\n VALUES (search_tohash(_search))\n ON CONFLICT DO NOTHING\n RETURNING * INTO search;\nIF search.hash IS NULL THEN\n SELECT * INTO search FROM searches WHERE hash=search_hash(_search);\nEND IF;\nIF search._where IS NULL THEN\n search._where := cql_to_where(_search);\nEND IF;\nIF search.orderby IS NULL THEN\n search.orderby := sort_sqlorderby(_search);\nEND IF;\n\nIF search.statslastupdated IS NULL OR age(search.statslastupdated) > '1 day'::interval OR (_search ? 'context' AND search.total_count IS NULL) THEN\n updatestats := TRUE;\nEND IF;\n\nIF updatestats THEN\n -- Get Estimated Stats\n RAISE NOTICE 'Getting stats for %', search._where;\n search.estimated_count := estimated_count(search._where);\n RAISE NOTICE 'Estimated Count: %', search.estimated_count;\n\n IF _search ? 'context' OR search.estimated_count < 10000 THEN\n --search.total_count := partition_count(search._where);\n EXECUTE format(\n 'SELECT count(*) FROM items WHERE %s',\n search._where\n ) INTO search.total_count;\n RAISE NOTICE 'Actual Count: %', search.total_count;\n ELSE\n search.total_count := NULL;\n END IF;\n search.statslastupdated := now();\nEND IF;\n\nsearch.lastused := now();\nsearch.usecount := coalesce(search.usecount,0) + 1;\nRAISE NOTICE 'SEARCH: %', search;\nUPDATE searches SET\n _where = search._where,\n orderby = search.orderby,\n lastused = search.lastused,\n usecount = search.usecount,\n statslastupdated = search.statslastupdated,\n estimated_count = search.estimated_count,\n total_count = search.total_count\nWHERE hash = search.hash\n;\nRETURN search;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n\nCREATE OR REPLACE FUNCTION search(_search jsonb = '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n searches searches%ROWTYPE;\n _where text;\n token_where text;\n full_where text;\n orderby text;\n query text;\n token_type text := substr(_search->>'token',1,4);\n _limit int := coalesce((_search->>'limit')::int, 10);\n curs refcursor;\n cntr int := 0;\n iter_record items%ROWTYPE;\n first_record items%ROWTYPE;\n last_record items%ROWTYPE;\n out_records jsonb := '[]'::jsonb;\n prev_query text;\n next text;\n prev_id text;\n has_next boolean := false;\n has_prev boolean := false;\n prev text;\n total_count bigint;\n context jsonb;\n collection jsonb;\n includes text[];\n excludes text[];\n exit_flag boolean := FALSE;\n batches int := 0;\n timer timestamptz := clock_timestamp();\nBEGIN\nsearches := search_query(_search);\n_where := searches._where;\norderby := searches.orderby;\ntotal_count := coalesce(searches.total_count, searches.estimated_count);\n\n\nIF token_type='prev' THEN\n token_where := get_token_filter(_search, null::jsonb);\n orderby := sort_sqlorderby(_search, TRUE);\nEND IF;\nIF token_type='next' THEN\n token_where := get_token_filter(_search, null::jsonb);\nEND IF;\n\nfull_where := concat_ws(' AND ', _where, token_where);\nRAISE NOTICE 'FULL QUERY % %', full_where, clock_timestamp()-timer;\ntimer := clock_timestamp();\n\nFOR query IN SELECT partition_queries(full_where, orderby) LOOP\n timer := clock_timestamp();\n query := format('%s LIMIT %L', query, _limit + 1);\n RAISE NOTICE 'Partition Query: %', query;\n batches := batches + 1;\n curs = create_cursor(query);\n LOOP\n FETCH curs into iter_record;\n EXIT WHEN NOT FOUND;\n cntr := cntr + 1;\n last_record := iter_record;\n IF cntr = 1 THEN\n first_record := last_record;\n END IF;\n IF cntr <= _limit THEN\n out_records := out_records || last_record.content;\n ELSIF cntr > _limit THEN\n has_next := true;\n exit_flag := true;\n EXIT;\n END IF;\n END LOOP;\n RAISE NOTICE 'Query took %', clock_timestamp()-timer;\n timer := clock_timestamp();\n EXIT WHEN exit_flag;\nEND LOOP;\nRAISE NOTICE 'Scanned through % partitions.', batches;\n\n\n-- Flip things around if this was the result of a prev token query\nIF token_type='prev' THEN\n out_records := flip_jsonb_array(out_records);\n first_record := last_record;\nEND IF;\n\n-- If this query has a token, see if there is data before the first record\nIF _search ? 'token' THEN\n prev_query := format(\n 'SELECT 1 FROM items WHERE %s LIMIT 1',\n concat_ws(\n ' AND ',\n _where,\n trim(get_token_filter(_search, to_jsonb(first_record)))\n )\n );\n RAISE NOTICE 'Query to get previous record: % --- %', prev_query, first_record;\n EXECUTE prev_query INTO has_prev;\n IF FOUND and has_prev IS NOT NULL THEN\n RAISE NOTICE 'Query results from prev query: %', has_prev;\n has_prev := TRUE;\n END IF;\nEND IF;\nhas_prev := COALESCE(has_prev, FALSE);\n\nRAISE NOTICE 'token_type: %, has_next: %, has_prev: %', token_type, has_next, has_prev;\nIF has_prev THEN\n prev := out_records->0->>'id';\nEND IF;\nIF has_next OR token_type='prev' THEN\n next := out_records->-1->>'id';\nEND IF;\n\n\n-- include/exclude any fields following fields extension\nIF _search ? 'fields' THEN\n IF _search->'fields' ? 'exclude' THEN\n excludes=textarr(_search->'fields'->'exclude');\n END IF;\n IF _search->'fields' ? 'include' THEN\n includes=textarr(_search->'fields'->'include');\n IF array_length(includes, 1)>0 AND NOT 'id' = ANY (includes) THEN\n includes = includes || '{id}';\n END IF;\n END IF;\n SELECT jsonb_agg(filter_jsonb(row, includes, excludes)) INTO out_records FROM jsonb_array_elements(out_records) row;\nEND IF;\n\ncontext := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'matched', total_count,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n));\n\ncollection := jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb),\n 'next', next,\n 'prev', prev,\n 'context', context\n);\n\nRETURN collection;\nEND;\n$$ LANGUAGE PLPGSQL SET jit TO off;\nINSERT INTO pgstac.migrations (version) VALUES ('0.3.3');\n" }, { "path": "src/pgstac/migrations/pgstac.0.3.4-0.3.5.sql", "content": "SET SEARCH_PATH to pgstac, public;\n\nCREATE TEMP TABLE temp_migrations AS SELECT version, max(datetime) as datetime from migrations group by 1;\nTRUNCATE pgstac.migrations;\nINSERT INTO pgstac.migrations SELECT * FROM temp_migrations;\n\ndrop function if exists \"pgstac\".\"partition_queries\"(_where text, _orderby text);\n\n--drop function if exists \"pgstac\".\"search_hash\"(jsonb);\n\ndrop function if exists \"pgstac\".\"search_query\"(_search jsonb, updatestats boolean);\n\ndrop view if exists \"pgstac\".\"items_partitions\";\n\ndrop view if exists \"pgstac\".\"all_items_partitions\";\n\ncreate table \"pgstac\".\"search_wheres\" (\n \"_where\" text not null,\n \"lastused\" timestamp with time zone default now(),\n \"usecount\" bigint default 0,\n \"statslastupdated\" timestamp with time zone,\n \"estimated_count\" bigint,\n \"estimated_cost\" double precision,\n \"time_to_estimate\" double precision,\n \"total_count\" bigint,\n \"time_to_count\" double precision,\n \"partitions\" text[]\n);\n\n\nalter table \"pgstac\".\"migrations\" alter column \"datetime\" set default clock_timestamp();\n\nalter table \"pgstac\".\"migrations\" alter column \"version\" set not null;\n\nalter table \"pgstac\".\"searches\" drop column \"estimated_count\";\n\nalter table \"pgstac\".\"searches\" drop column \"statslastupdated\";\n\nalter table \"pgstac\".\"searches\" drop column \"total_count\";\n\nalter table \"pgstac\".\"searches\" add column \"metadata\" jsonb not null default '{}'::jsonb;\n\n--alter table \"pgstac\".\"searches\" alter column \"hash\" set default pgstac.search_hash(search, metadata);\n\n\nCREATE OR REPLACE FUNCTION pgstac.search_hash(jsonb, jsonb)\n RETURNS text\n LANGUAGE sql\n IMMUTABLE PARALLEL SAFE\nAS $function$\n SELECT md5(concat(search_tohash($1)::text,$2::text));\n$function$\n;\n\nalter table \"pgstac\".\"searches\" DROP COLUMN hash;\nalter table \"pgstac\".\"searches\" add column \"hash\" text generated always as (\"pgstac\".search_hash(search, metadata)) stored primary key;\n\nCREATE UNIQUE INDEX migrations_pkey ON pgstac.migrations USING btree (version);\n\nCREATE UNIQUE INDEX search_wheres_pkey ON pgstac.search_wheres USING btree (_where);\n\nalter table \"pgstac\".\"migrations\" add constraint \"migrations_pkey\" PRIMARY KEY using index \"migrations_pkey\";\n\nalter table \"pgstac\".\"search_wheres\" add constraint \"search_wheres_pkey\" PRIMARY KEY using index \"search_wheres_pkey\";\n\nset check_function_bodies = off;\n\nCREATE OR REPLACE FUNCTION pgstac.array_reverse(anyarray)\n RETURNS anyarray\n LANGUAGE sql\n IMMUTABLE STRICT\nAS $function$\nSELECT ARRAY(\n SELECT $1[i]\n FROM generate_subscripts($1,1) AS s(i)\n ORDER BY i DESC\n);\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.context()\n RETURNS text\n LANGUAGE sql\nAS $function$\n SELECT get_setting('pgstac.context','off'::text);\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.context_estimated_cost()\n RETURNS double precision\n LANGUAGE sql\nAS $function$\n SELECT get_setting('pgstac.context_estimated_cost', 1000000::float);\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.context_estimated_count()\n RETURNS integer\n LANGUAGE sql\nAS $function$\n SELECT get_setting('pgstac.context_estimated_count', 100000::int);\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.context_stats_ttl()\n RETURNS interval\n LANGUAGE sql\nAS $function$\n SELECT get_setting('pgstac.context_stats_ttl', '1 day'::interval);\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.drop_partition_constraints(partition text)\n RETURNS void\n LANGUAGE plpgsql\nAS $function$\nDECLARE\n q text;\n end_datetime_constraint text := concat(partition, '_end_datetime_constraint');\n collections_constraint text := concat(partition, '_collections_constraint');\nBEGIN\n q := format($q$\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n $q$,\n partition,\n end_datetime_constraint,\n partition,\n collections_constraint\n );\n\n EXECUTE q;\n RETURN;\n\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.field_orderby(p text)\n RETURNS text\n LANGUAGE sql\nAS $function$\nWITH t AS (\n SELECT\n replace(trim(substring(indexdef from 'btree \\((.*)\\)')),' ','')as s\n FROM pg_indexes WHERE schemaname='pgstac' AND tablename='items' AND indexdef ~* 'btree' AND indexdef ~* 'properties'\n) SELECT s FROM t WHERE strpos(s, lower(trim(p)))>0;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.get_setting(setting text, INOUT _default anynonarray DEFAULT NULL::text)\n RETURNS anynonarray\n LANGUAGE plpgsql\nAS $function$\nDECLARE\n_type text;\nBEGIN\n SELECT pg_typeof(_default) INTO _type;\n IF _type = 'unknown' THEN _type='text'; END IF;\n EXECUTE format($q$\n SELECT COALESCE(\n CAST(current_setting($1,TRUE) AS %s),\n $2\n )\n $q$, _type)\n INTO _default\n USING setting, _default\n ;\n RETURN;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.get_version()\n RETURNS text\n LANGUAGE sql\n SET search_path TO 'pgstac', 'public'\nAS $function$\n SELECT version FROM migrations ORDER BY datetime DESC, version DESC LIMIT 1;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.items_count(_where text)\n RETURNS bigint\n LANGUAGE plpgsql\nAS $function$\nDECLARE\ncnt bigint;\nBEGIN\nEXECUTE format('SELECT count(*) FROM items WHERE %s', _where) INTO cnt;\nRETURN cnt;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.partition_checks(partition text, OUT min_datetime timestamp with time zone, OUT max_datetime timestamp with time zone, OUT min_end_datetime timestamp with time zone, OUT max_end_datetime timestamp with time zone, OUT collections text[], OUT cnt bigint)\n RETURNS record\n LANGUAGE plpgsql\nAS $function$\nDECLARE\nq text;\nend_datetime_constraint text := concat(partition, '_end_datetime_constraint');\ncollections_constraint text := concat(partition, '_collections_constraint');\nBEGIN\nRAISE NOTICE 'CREATING CONSTRAINTS FOR %', partition;\nq := format($q$\n SELECT\n min(datetime),\n max(datetime),\n min(end_datetime),\n max(end_datetime),\n array_agg(DISTINCT collection_id),\n count(*)\n FROM %I;\n $q$,\n partition\n);\nEXECUTE q INTO min_datetime, max_datetime, min_end_datetime, max_end_datetime, collections, cnt;\nRAISE NOTICE '% % % % % %', min_datetime, max_datetime, min_end_datetime, max_end_datetime, collections, cnt;\nIF cnt IS NULL or cnt = 0 THEN\n RAISE NOTICE 'Partition % is empty, removing...', partition;\n q := format($q$\n DROP TABLE IF EXISTS %I;\n $q$, partition\n );\n EXECUTE q;\n RETURN;\nEND IF;\nq := format($q$\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n ALTER TABLE %I ADD CONSTRAINT %I\n check((end_datetime >= %L) AND (end_datetime <= %L));\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n ALTER TABLE %I ADD CONSTRAINT %I\n check((collection_id = ANY(%L)));\n ANALYZE %I;\n $q$,\n partition,\n end_datetime_constraint,\n partition,\n end_datetime_constraint,\n min_end_datetime,\n max_end_datetime,\n partition,\n collections_constraint,\n partition,\n collections_constraint,\n collections,\n partition\n);\n\nEXECUTE q;\nRETURN;\n\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.partition_queries(_where text DEFAULT 'TRUE'::text, _orderby text DEFAULT 'datetime DESC, id DESC'::text, partitions text[] DEFAULT '{items}'::text[])\n RETURNS SETOF text\n LANGUAGE plpgsql\n SET search_path TO 'pgstac', 'public'\nAS $function$\nDECLARE\n partition_query text;\n query text;\n p text;\n cursors refcursor;\n dstart timestamptz;\n dend timestamptz;\n step interval := '10 weeks'::interval;\nBEGIN\n\nIF _orderby ILIKE 'datetime d%' THEN\n partitions := partitions;\nELSIF _orderby ILIKE 'datetime a%' THEN\n partitions := array_reverse(partitions);\nELSE\n query := format($q$\n SELECT * FROM items\n WHERE %s\n ORDER BY %s\n $q$, _where, _orderby\n );\n\n RETURN NEXT query;\n RETURN;\nEND IF;\nRAISE NOTICE 'PARTITIONS ---> %',partitions;\nIF cardinality(partitions) > 0 THEN\n FOREACH p IN ARRAY partitions\n --EXECUTE partition_query\n LOOP\n query := format($q$\n SELECT * FROM %I\n WHERE %s\n ORDER BY %s\n $q$,\n p,\n _where,\n _orderby\n );\n RETURN NEXT query;\n END LOOP;\nEND IF;\nRETURN;\nEND;\n$function$\n;\n\n\nCREATE OR REPLACE FUNCTION pgstac.search_query(_search jsonb DEFAULT '{}'::jsonb, updatestats boolean DEFAULT false, _metadata jsonb DEFAULT '{}'::jsonb)\n RETURNS pgstac.searches\n LANGUAGE plpgsql\nAS $function$\nDECLARE\n search searches%ROWTYPE;\n pexplain jsonb;\n t timestamptz;\n i interval;\nBEGIN\nSELECT * INTO search FROM searches\nWHERE hash=search_hash(_search, _metadata) FOR UPDATE;\n\n-- Calculate the where clause if not already calculated\nIF search._where IS NULL THEN\n search._where := cql_to_where(_search);\nEND IF;\n\n-- Calculate the order by clause if not already calculated\nIF search.orderby IS NULL THEN\n search.orderby := sort_sqlorderby(_search);\nEND IF;\n\nPERFORM where_stats(search._where, updatestats);\n\nsearch.lastused := now();\nsearch.usecount := coalesce(search.usecount, 0) + 1;\nINSERT INTO searches (search, _where, orderby, lastused, usecount, metadata)\nVALUES (_search, search._where, search.orderby, search.lastused, search.usecount, _metadata)\nON CONFLICT (hash) DO\nUPDATE SET\n _where = EXCLUDED._where,\n orderby = EXCLUDED.orderby,\n lastused = EXCLUDED.lastused,\n usecount = EXCLUDED.usecount,\n metadata = EXCLUDED.metadata\nRETURNING * INTO search\n;\nRETURN search;\n\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.set_version(text)\n RETURNS text\n LANGUAGE sql\n SET search_path TO 'pgstac', 'public'\nAS $function$\n INSERT INTO migrations (version) VALUES ($1)\n ON CONFLICT DO NOTHING\n RETURNING version;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.where_stats(inwhere text, updatestats boolean DEFAULT false)\n RETURNS pgstac.search_wheres\n LANGUAGE plpgsql\nAS $function$\nDECLARE\n t timestamptz;\n i interval;\n explain_json jsonb;\n partitions text[];\n sw search_wheres%ROWTYPE;\nBEGIN\n SELECT * INTO sw FROM search_wheres WHERE _where=inwhere FOR UPDATE;\n\n -- Update statistics if explicitly set, if statistics do not exist, or statistics ttl has expired\n IF NOT updatestats THEN\n RAISE NOTICE 'Checking if update is needed.';\n RAISE NOTICE 'Stats Last Updated: %', sw.statslastupdated;\n RAISE NOTICE 'TTL: %, Age: %', context_stats_ttl(), now() - sw.statslastupdated;\n RAISE NOTICE 'Context: %, Existing Total: %', context(), sw.total_count;\n IF\n sw.statslastupdated IS NULL\n OR (now() - sw.statslastupdated) > context_stats_ttl()\n OR (context() != 'off' AND sw.total_count IS NULL)\n THEN\n updatestats := TRUE;\n END IF;\n END IF;\n\n sw._where := inwhere;\n sw.lastused := now();\n sw.usecount := coalesce(sw.usecount,0) + 1;\n\n IF NOT updatestats THEN\n UPDATE search_wheres SET\n lastused = sw.lastused,\n usecount = sw.usecount\n WHERE _where = inwhere\n RETURNING * INTO sw\n ;\n RETURN sw;\n END IF;\n -- Use explain to get estimated count/cost and a list of the partitions that would be hit by the query\n t := clock_timestamp();\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s', inwhere)\n INTO explain_json;\n RAISE NOTICE 'Time for just the explain: %', clock_timestamp() - t;\n WITH t AS (\n SELECT j->>0 as p FROM\n jsonb_path_query(\n explain_json,\n 'strict $.**.\"Relation Name\" ? (@ != null)'\n ) j\n ), ordered AS (\n SELECT p FROM t ORDER BY p DESC\n -- SELECT p FROM t JOIN items_partitions\n -- ON (t.p = items_partitions.partition)\n -- ORDER BY pstart DESC\n )\n SELECT array_agg(p) INTO partitions FROM ordered;\n i := clock_timestamp() - t;\n RAISE NOTICE 'Time for explain + join: %', clock_timestamp() - t;\n\n\n\n sw.statslastupdated := now();\n sw.estimated_count := explain_json->0->'Plan'->'Plan Rows';\n sw.estimated_cost := explain_json->0->'Plan'->'Total Cost';\n sw.time_to_estimate := extract(epoch from i);\n sw.partitions := partitions;\n\n -- Do a full count of rows if context is set to on or if auto is set and estimates are low enough\n IF\n context() = 'on'\n OR\n ( context() = 'auto' AND\n (\n sw.estimated_count < context_estimated_count()\n OR\n sw.estimated_cost < context_estimated_cost()\n )\n )\n THEN\n t := clock_timestamp();\n RAISE NOTICE 'Calculating actual count...';\n EXECUTE format(\n 'SELECT count(*) FROM items WHERE %s',\n inwhere\n ) INTO sw.total_count;\n i := clock_timestamp() - t;\n RAISE NOTICE 'Actual Count: % -- %', sw.total_count, i;\n sw.time_to_count := extract(epoch FROM i);\n ELSE\n sw.total_count := NULL;\n sw.time_to_count := NULL;\n END IF;\n\n\n INSERT INTO search_wheres SELECT sw.*\n ON CONFLICT (_where)\n DO UPDATE\n SET\n lastused = sw.lastused,\n usecount = sw.usecount,\n statslastupdated = sw.statslastupdated,\n estimated_count = sw.estimated_count,\n estimated_cost = sw.estimated_cost,\n time_to_estimate = sw.time_to_estimate,\n partitions = sw.partitions,\n total_count = sw.total_count,\n time_to_count = sw.time_to_count\n ;\n RETURN sw;\nEND;\n$function$\n;\n\ncreate or replace view \"pgstac\".\"all_items_partitions\" as WITH base AS (\n SELECT ((c.oid)::regclass)::text AS partition,\n pg_get_expr(c.relpartbound, c.oid) AS _constraint,\n regexp_matches(pg_get_expr(c.relpartbound, c.oid), '\\(''([0-9 :+-]*)''\\).*\\(''([0-9 :+-]*)''\\)'::text) AS t,\n (c.reltuples)::bigint AS est_cnt\n FROM pg_class c,\n pg_inherits i\n WHERE ((c.oid = i.inhrelid) AND (i.inhparent = ('pgstac.items'::regclass)::oid))\n )\n SELECT base.partition,\n tstzrange((base.t[1])::timestamp with time zone, (base.t[2])::timestamp with time zone) AS tstzrange,\n (base.t[1])::timestamp with time zone AS pstart,\n (base.t[2])::timestamp with time zone AS pend,\n base.est_cnt\n FROM base\n ORDER BY (tstzrange((base.t[1])::timestamp with time zone, (base.t[2])::timestamp with time zone)) DESC;\n\n\nCREATE OR REPLACE FUNCTION pgstac.items_partition_name(timestamp with time zone)\n RETURNS text\n LANGUAGE sql\n IMMUTABLE PARALLEL SAFE\nAS $function$\n SELECT to_char($1, '\"items_p\"IYYY\"w\"IW');\n$function$\n;\n\ncreate or replace view \"pgstac\".\"items_partitions\" as SELECT all_items_partitions.partition,\n all_items_partitions.tstzrange,\n all_items_partitions.pstart,\n all_items_partitions.pend,\n all_items_partitions.est_cnt\n FROM pgstac.all_items_partitions\n WHERE (all_items_partitions.est_cnt > 0);\n\n\nCREATE OR REPLACE FUNCTION pgstac.items_path(dotpath text, OUT field text, OUT path text, OUT path_txt text, OUT jsonpath text, OUT eq text)\n RETURNS record\n LANGUAGE plpgsql\n IMMUTABLE PARALLEL SAFE\nAS $function$\nDECLARE\npath_elements text[];\nlast_element text;\nBEGIN\ndotpath := replace(trim(dotpath), 'properties.', '');\n\nIF dotpath = '' THEN\n RETURN;\nEND IF;\n\npath_elements := string_to_array(dotpath, '.');\njsonpath := NULL;\n\nIF path_elements[1] IN ('id','geometry','datetime') THEN\n field := path_elements[1];\n path_elements := path_elements[2:];\nELSIF path_elements[1] = 'collection' THEN\n field := 'collection_id';\n path_elements := path_elements[2:];\nELSIF path_elements[1] IN ('links', 'assets', 'stac_version', 'stac_extensions') THEN\n field := 'content';\nELSE\n field := 'content';\n path_elements := '{properties}'::text[] || path_elements;\nEND IF;\nIF cardinality(path_elements)<1 THEN\n path := field;\n path_txt := field;\n jsonpath := '$';\n eq := NULL;\n RETURN;\nEND IF;\n\n\nlast_element := path_elements[cardinality(path_elements)];\npath_elements := path_elements[1:cardinality(path_elements)-1];\njsonpath := concat(array_to_string('{$}'::text[] || array_map_ident(path_elements), '.'), '.', quote_ident(last_element));\npath_elements := array_map_literal(path_elements);\npath := format($F$ properties->%s $F$, quote_literal(dotpath));\npath_txt := format($F$ properties->>%s $F$, quote_literal(dotpath));\neq := format($F$ properties @? '$.%s[*] ? (@ == %%s) '$F$, quote_ident(dotpath));\n\nRAISE NOTICE 'ITEMS PATH -- % % % % %', field, path, path_txt, jsonpath, eq;\nRETURN;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.items_staging_ignore_insert_triggerfunc()\n RETURNS trigger\n LANGUAGE plpgsql\nAS $function$\nDECLARE\n p record;\nBEGIN\n CREATE TEMP TABLE new_partitions ON COMMIT DROP AS\n SELECT\n items_partition_name(stac_datetime(content)) as partition,\n date_trunc('week', min(stac_datetime(content))) as partition_start\n FROM newdata\n GROUP BY 1;\n\n FOR p IN SELECT new_partitions.partition, new_partitions.partition_start, new_partitions.partition_start + '1 week'::interval as partition_end FROM new_partitions\n LOOP\n RAISE NOTICE 'Getting partition % ready.', p.partition;\n IF NOT items_partition_exists(p.partition) THEN\n RAISE NOTICE 'Creating partition %.', p.partition;\n PERFORM items_partition_create_worker(p.partition, p.partition_start, p.partition_end);\n END IF;\n PERFORM drop_partition_constraints(p.partition);\n END LOOP;\n\n INSERT INTO items (id, geometry, collection_id, datetime, end_datetime, properties, content)\n SELECT\n content->>'id',\n stac_geom(content),\n content->>'collection',\n stac_datetime(content),\n stac_end_datetime(content),\n properties_idx(content),\n content\n FROM newdata\n ON CONFLICT (datetime, id) DO NOTHING\n ;\n DELETE FROM items_staging;\n\n\n FOR p IN SELECT new_partitions.partition FROM new_partitions\n LOOP\n RAISE NOTICE 'Setting constraints for partition %.', p.partition;\n PERFORM partition_checks(p.partition);\n END LOOP;\n DROP TABLE IF EXISTS new_partitions;\n RETURN NULL;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.items_staging_insert_triggerfunc()\n RETURNS trigger\n LANGUAGE plpgsql\nAS $function$\nDECLARE\n mindate timestamptz;\n maxdate timestamptz;\n partition text;\n p record;\nBEGIN\n CREATE TEMP TABLE new_partitions ON COMMIT DROP AS\n SELECT\n items_partition_name(stac_datetime(content)) as partition,\n date_trunc('week', min(stac_datetime(content))) as partition_start\n FROM newdata\n GROUP BY 1;\n\n FOR p IN SELECT new_partitions.partition, new_partitions.partition_start, new_partitions.partition_start + '1 week'::interval as partition_end FROM new_partitions\n LOOP\n RAISE NOTICE 'Getting partition % ready.', p.partition;\n IF NOT items_partition_exists(p.partition) THEN\n RAISE NOTICE 'Creating partition %.', p.partition;\n PERFORM items_partition_create_worker(p.partition, p.partition_start, p.partition_end);\n END IF;\n PERFORM drop_partition_constraints(p.partition);\n END LOOP;\n\n INSERT INTO items (id, geometry, collection_id, datetime, end_datetime, properties, content)\n SELECT\n content->>'id',\n stac_geom(content),\n content->>'collection',\n stac_datetime(content),\n stac_end_datetime(content),\n properties_idx(content),\n content\n FROM newdata\n ;\n DELETE FROM items_staging;\n\n\n FOR p IN SELECT new_partitions.partition FROM new_partitions\n LOOP\n RAISE NOTICE 'Setting constraints for partition %.', p.partition;\n PERFORM partition_checks(p.partition);\n END LOOP;\n DROP TABLE IF EXISTS new_partitions;\n RETURN NULL;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.items_staging_upsert_insert_triggerfunc()\n RETURNS trigger\n LANGUAGE plpgsql\nAS $function$\nDECLARE\n p record;\nBEGIN\n CREATE TEMP TABLE new_partitions ON COMMIT DROP AS\n SELECT\n items_partition_name(stac_datetime(content)) as partition,\n date_trunc('week', min(stac_datetime(content))) as partition_start\n FROM newdata\n GROUP BY 1;\n\n FOR p IN SELECT new_partitions.partition, new_partitions.partition_start, new_partitions.partition_start + '1 week'::interval as partition_end FROM new_partitions\n LOOP\n RAISE NOTICE 'Getting partition % ready.', p.partition;\n IF NOT items_partition_exists(p.partition) THEN\n RAISE NOTICE 'Creating partition %.', p.partition;\n PERFORM items_partition_create_worker(p.partition, p.partition_start, p.partition_end);\n END IF;\n PERFORM drop_partition_constraints(p.partition);\n END LOOP;\n\n INSERT INTO items (id, geometry, collection_id, datetime, end_datetime, properties, content)\n SELECT\n content->>'id',\n stac_geom(content),\n content->>'collection',\n stac_datetime(content),\n stac_end_datetime(content),\n properties_idx(content),\n content\n FROM newdata\n ON CONFLICT (datetime, id) DO UPDATE SET\n content = EXCLUDED.content\n WHERE items.content IS DISTINCT FROM EXCLUDED.content\n ;\n DELETE FROM items_staging;\n\n\n FOR p IN SELECT new_partitions.partition FROM new_partitions\n LOOP\n RAISE NOTICE 'Setting constraints for partition %.', p.partition;\n PERFORM partition_checks(p.partition);\n END LOOP;\n DROP TABLE IF EXISTS new_partitions;\n RETURN NULL;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.partition_cursor(_where text DEFAULT 'TRUE'::text, _orderby text DEFAULT 'datetime DESC, id DESC'::text)\n RETURNS SETOF refcursor\n LANGUAGE plpgsql\n SET search_path TO 'pgstac', 'public'\nAS $function$\nDECLARE\n partition_query text;\n query text;\n p record;\n cursors refcursor;\nBEGIN\nFOR query IN SELECT * FROM partition_queries(_where, _orderby) LOOP\n RETURN NEXT create_cursor(query);\nEND LOOP;\nRETURN;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.search(_search jsonb DEFAULT '{}'::jsonb)\n RETURNS jsonb\n LANGUAGE plpgsql\n SET jit TO 'off'\nAS $function$\nDECLARE\n searches searches%ROWTYPE;\n _where text;\n token_where text;\n full_where text;\n orderby text;\n query text;\n token_type text := substr(_search->>'token',1,4);\n _limit int := coalesce((_search->>'limit')::int, 10);\n curs refcursor;\n cntr int := 0;\n iter_record items%ROWTYPE;\n first_record items%ROWTYPE;\n last_record items%ROWTYPE;\n out_records jsonb := '[]'::jsonb;\n prev_query text;\n next text;\n prev_id text;\n has_next boolean := false;\n has_prev boolean := false;\n prev text;\n total_count bigint;\n context jsonb;\n collection jsonb;\n includes text[];\n excludes text[];\n exit_flag boolean := FALSE;\n batches int := 0;\n timer timestamptz := clock_timestamp();\n pstart timestamptz;\n pend timestamptz;\n pcurs refcursor;\n search_where search_wheres%ROWTYPE;\nBEGIN\nsearches := search_query(_search);\n_where := searches._where;\norderby := searches.orderby;\nsearch_where := where_stats(_where);\ntotal_count := coalesce(search_where.total_count, search_where.estimated_count);\n\n\nIF token_type='prev' THEN\n token_where := get_token_filter(_search, null::jsonb);\n orderby := sort_sqlorderby(_search, TRUE);\nEND IF;\nIF token_type='next' THEN\n token_where := get_token_filter(_search, null::jsonb);\nEND IF;\n\nfull_where := concat_ws(' AND ', _where, token_where);\nRAISE NOTICE 'FULL QUERY % %', full_where, clock_timestamp()-timer;\ntimer := clock_timestamp();\n\nFOR query IN SELECT partition_queries(full_where, orderby, search_where.partitions) LOOP\n timer := clock_timestamp();\n query := format('%s LIMIT %s', query, _limit + 1);\n RAISE NOTICE 'Partition Query: %', query;\n batches := batches + 1;\n curs = create_cursor(query);\n --OPEN curs\n LOOP\n FETCH curs into iter_record;\n EXIT WHEN NOT FOUND;\n cntr := cntr + 1;\n last_record := iter_record;\n IF cntr = 1 THEN\n first_record := last_record;\n END IF;\n IF cntr <= _limit THEN\n out_records := out_records || last_record.content;\n ELSIF cntr > _limit THEN\n has_next := true;\n exit_flag := true;\n EXIT;\n END IF;\n END LOOP;\n RAISE NOTICE 'Query took %. Total Time %', clock_timestamp()-timer, ftime();\n timer := clock_timestamp();\n EXIT WHEN exit_flag;\nEND LOOP;\nRAISE NOTICE 'Scanned through % partitions.', batches;\n\n\n-- Flip things around if this was the result of a prev token query\nIF token_type='prev' THEN\n out_records := flip_jsonb_array(out_records);\n first_record := last_record;\nEND IF;\n\n-- If this query has a token, see if there is data before the first record\nIF _search ? 'token' THEN\n prev_query := format(\n 'SELECT 1 FROM items WHERE %s LIMIT 1',\n concat_ws(\n ' AND ',\n _where,\n trim(get_token_filter(_search, to_jsonb(first_record)))\n )\n );\n RAISE NOTICE 'Query to get previous record: % --- %', prev_query, first_record;\n EXECUTE prev_query INTO has_prev;\n IF FOUND and has_prev IS NOT NULL THEN\n RAISE NOTICE 'Query results from prev query: %', has_prev;\n has_prev := TRUE;\n END IF;\nEND IF;\nhas_prev := COALESCE(has_prev, FALSE);\n\nRAISE NOTICE 'token_type: %, has_next: %, has_prev: %', token_type, has_next, has_prev;\nIF has_prev THEN\n prev := out_records->0->>'id';\nEND IF;\nIF has_next OR token_type='prev' THEN\n next := out_records->-1->>'id';\nEND IF;\n\n\n-- include/exclude any fields following fields extension\nIF _search ? 'fields' THEN\n IF _search->'fields' ? 'exclude' THEN\n excludes=textarr(_search->'fields'->'exclude');\n END IF;\n IF _search->'fields' ? 'include' THEN\n includes=textarr(_search->'fields'->'include');\n IF array_length(includes, 1)>0 AND NOT 'id' = ANY (includes) THEN\n includes = includes || '{id}';\n END IF;\n END IF;\n SELECT jsonb_agg(filter_jsonb(row, includes, excludes)) INTO out_records FROM jsonb_array_elements(out_records) row;\nEND IF;\n\nIF context() != 'off' THEN\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'matched', total_count,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\nELSE\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\nEND IF;\n\ncollection := jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb),\n 'next', next,\n 'prev', prev,\n 'context', context\n);\n\nRETURN collection;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.sort_sqlorderby(_search jsonb DEFAULT NULL::jsonb, reverse boolean DEFAULT false)\n RETURNS text\n LANGUAGE sql\nAS $function$\nWITH sortby AS (\n SELECT coalesce(_search->'sortby','[{\"field\":\"datetime\", \"direction\":\"desc\"}]') as sort\n), withid AS (\n SELECT CASE\n WHEN sort @? '$[*] ? (@.field == \"id\")' THEN sort\n ELSE sort || '[{\"field\":\"id\", \"direction\":\"desc\"}]'::jsonb\n END as sort\n FROM sortby\n), withid_rows AS (\n SELECT jsonb_array_elements(sort) as value FROM withid\n),sorts AS (\n SELECT\n coalesce(field_orderby((items_path(value->>'field')).path_txt), (items_path(value->>'field')).path) as key,\n parse_sort_dir(value->>'direction', reverse) as dir\n FROM withid_rows\n)\nSELECT array_to_string(\n array_agg(concat(key, ' ', dir)),\n ', '\n) FROM sorts;\n$function$\n;\n\nDO $$\nDECLARE\n partition text;\n topartition text;\n q text;\n matches text[];\nBEGIN\nFOR partition IN\n SELECT\n (to_json(parse_ident(c.oid::pg_catalog.regclass::text)))->>-1\n FROM pg_catalog.pg_class c, pg_catalog.pg_inherits i\n WHERE c.oid=i.inhrelid and i.inhparent='items'::regclass\n LOOP\n RAISE NOTICE 'Partition: %', partition;\n topartition := (to_json(regexp_split_to_array(partition, '\\.')))->>-1;\n IF partition != topartition THEN\n RAISE NOTICE 'Renaming partition % to %', partition, topartition;\n q := format($q$\n ALTER TABLE %I RENAME TO %I;\n $q$, partition, topartition);\n RAISE NOTICE '%', q;\n EXECUTE q;\n END IF;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nSELECT partition_checks(partition) FROM all_items_partitions;\n\nSELECT set_version('0.3.5');\n" }, { "path": "src/pgstac/migrations/pgstac.0.3.4.sql", "content": "CREATE EXTENSION IF NOT EXISTS postgis;\nCREATE SCHEMA IF NOT EXISTS pgstac;\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE TABLE migrations (\n version text,\n datetime timestamptz DEFAULT now() NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION notice(VARIADIC text[]) RETURNS boolean AS $$\nDECLARE\ndebug boolean := current_setting('pgstac.debug', true);\nBEGIN\n IF debug THEN\n RAISE NOTICE 'NOTICE FROM FUNC: % >>>>> %', concat_ws(' | ', $1), clock_timestamp();\n RETURN TRUE;\n END IF;\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION empty_arr(ANYARRAY) RETURNS BOOLEAN AS $$\nSELECT CASE\n WHEN $1 IS NULL THEN TRUE\n WHEN cardinality($1)<1 THEN TRUE\nELSE FALSE\nEND;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_map_ident(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_ident(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_map_literal(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_literal(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION estimated_count(_where text) RETURNS bigint AS $$\nDECLARE\nrec record;\nrows bigint;\nBEGIN\n FOR rec in EXECUTE format(\n $q$\n EXPLAIN SELECT 1 FROM items WHERE %s\n $q$,\n _where)\n LOOP\n rows := substring(rec.\"QUERY PLAN\" FROM ' rows=([[:digit:]]+)');\n EXIT WHEN rows IS NOT NULL;\n END LOOP;\n\n RETURN rows;\nEND;\n$$ LANGUAGE PLPGSQL;\n/* converts a jsonb text array to a pg text[] array */\nCREATE OR REPLACE FUNCTION textarr(_js jsonb)\n RETURNS text[] AS $$\n SELECT\n CASE jsonb_typeof(_js)\n WHEN 'array' THEN ARRAY(SELECT jsonb_array_elements_text(_js))\n ELSE ARRAY[_js->>0]\n END\n;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION jsonb_paths (IN jdata jsonb, OUT path text[], OUT value jsonb) RETURNS\nSETOF RECORD AS $$\nwith recursive extract_all as\n(\n select\n ARRAY[key]::text[] as path,\n value\n FROM jsonb_each(jdata)\nunion all\n select\n path || coalesce(obj_key, (arr_key- 1)::text),\n coalesce(obj_value, arr_value)\n from extract_all\n left join lateral\n jsonb_each(case jsonb_typeof(value) when 'object' then value end)\n as o(obj_key, obj_value)\n on jsonb_typeof(value) = 'object'\n left join lateral\n jsonb_array_elements(case jsonb_typeof(value) when 'array' then value end)\n with ordinality as a(arr_value, arr_key)\n on jsonb_typeof(value) = 'array'\n where obj_key is not null or arr_key is not null\n)\nselect *\nfrom extract_all;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION jsonb_obj_paths (IN jdata jsonb, OUT path text[], OUT value jsonb) RETURNS\nSETOF RECORD AS $$\nwith recursive extract_all as\n(\n select\n ARRAY[key]::text[] as path,\n value\n FROM jsonb_each(jdata)\nunion all\n select\n path || obj_key,\n obj_value\n from extract_all\n left join lateral\n jsonb_each(case jsonb_typeof(value) when 'object' then value end)\n as o(obj_key, obj_value)\n on jsonb_typeof(value) = 'object'\n where obj_key is not null\n)\nselect *\nfrom extract_all;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION jsonb_val_paths (IN jdata jsonb, OUT path text[], OUT value jsonb) RETURNS\nSETOF RECORD AS $$\nSELECT * FROM jsonb_obj_paths(jdata) WHERE jsonb_typeof(value) not in ('object','array');\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION path_includes(IN path text[], IN includes text[]) RETURNS BOOLEAN AS $$\nWITH t AS (SELECT unnest(includes) i)\nSELECT EXISTS (\n SELECT 1 FROM t WHERE path @> string_to_array(trim(i), '.')\n);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION path_excludes(IN path text[], IN excludes text[]) RETURNS BOOLEAN AS $$\nWITH t AS (SELECT unnest(excludes) e)\nSELECT NOT EXISTS (\n SELECT 1 FROM t WHERE path @> string_to_array(trim(e), '.')\n);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION jsonb_obj_paths_filtered (\n IN jdata jsonb,\n IN includes text[] DEFAULT ARRAY[]::text[],\n IN excludes text[] DEFAULT ARRAY[]::text[],\n OUT path text[],\n OUT value jsonb\n) RETURNS\nSETOF RECORD AS $$\nSELECT path, value\nFROM jsonb_obj_paths(jdata)\nWHERE\n CASE WHEN cardinality(includes) > 0 THEN path_includes(path, includes) ELSE TRUE END\n AND\n path_excludes(path, excludes)\n\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION filter_jsonb(\n IN jdata jsonb,\n IN includes text[] DEFAULT ARRAY[]::text[],\n IN excludes text[] DEFAULT ARRAY[]::text[]\n) RETURNS jsonb AS $$\nDECLARE\nrec RECORD;\noutj jsonb := '{}'::jsonb;\ncreated_paths text[] := '{}'::text[];\nBEGIN\n\nIF empty_arr(includes) AND empty_arr(excludes) THEN\nRAISE NOTICE 'no filter';\n RETURN jdata;\nEND IF;\nFOR rec in\nSELECT * FROM jsonb_obj_paths_filtered(jdata, includes, excludes)\nWHERE jsonb_typeof(value) != 'object'\nLOOP\n IF array_length(rec.path,1)>1 THEN\n FOR i IN 1..(array_length(rec.path,1)-1) LOOP\n IF NOT array_to_string(rec.path[1:i],'.') = ANY (created_paths) THEN\n outj := jsonb_set(outj, rec.path[1:i],'{}', true);\n created_paths := created_paths || array_to_string(rec.path[1:i],'.');\n END IF;\n END LOOP;\n END IF;\n outj := jsonb_set(outj, rec.path, rec.value, true);\n created_paths := created_paths || array_to_string(rec.path,'.');\nEND LOOP;\nRETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION flip_jsonb_array(j jsonb) RETURNS jsonb AS $$\nSELECT jsonb_agg(value) FROM (SELECT value FROM jsonb_array_elements(j) WITH ORDINALITY ORDER BY ordinality DESC) as t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n/* Functions to create an iterable of cursors over partitions. */\nCREATE OR REPLACE FUNCTION create_cursor(q text) RETURNS refcursor AS $$\nDECLARE\n curs refcursor;\nBEGIN\n OPEN curs FOR EXECUTE q;\n RETURN curs;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION partition_queries(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC'\n) RETURNS SETOF text AS $$\nDECLARE\n partition_query text;\n query text;\n p record;\n cursors refcursor;\nBEGIN\nIF _orderby ILIKE 'datetime d%' THEN\n partition_query := format($q$\n SELECT partition, tstzrange\n FROM items_partitions\n ORDER BY tstzrange DESC;\n $q$);\nELSIF _orderby ILIKE 'datetime a%' THEN\n partition_query := format($q$\n SELECT partition, tstzrange\n FROM items_partitions\n ORDER BY tstzrange ASC\n ;\n $q$);\nELSE\n query := format($q$\n SELECT * FROM items\n WHERE %s\n ORDER BY %s\n $q$, _where, _orderby\n );\n\n RETURN NEXT query;\n RETURN;\nEND IF;\nFOR p IN\n EXECUTE partition_query\nLOOP\n query := format($q$\n SELECT * FROM items\n WHERE datetime >= %L AND datetime < %L AND %s\n ORDER BY %s\n $q$, lower(p.tstzrange), upper(p.tstzrange), _where, _orderby\n );\n RETURN NEXT query;\nEND LOOP;\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION partition_cursor(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC'\n) RETURNS SETOF refcursor AS $$\nDECLARE\n partition_query text;\n query text;\n p record;\n cursors refcursor;\nBEGIN\nFOR query IN SELECT * FROM partion_queries(_where, _orderby) LOOP\n RETURN NEXT create_cursor(query);\nEND LOOP;\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION partition_count(\n IN _where text DEFAULT 'TRUE'\n) RETURNS bigint AS $$\nDECLARE\n partition_query text;\n query text;\n p record;\n subtotal bigint;\n total bigint := 0;\nBEGIN\npartition_query := format($q$\n SELECT partition, tstzrange\n FROM items_partitions\n ORDER BY tstzrange DESC;\n$q$);\nRAISE NOTICE 'Partition Query: %', partition_query;\nFOR p IN\n EXECUTE partition_query\nLOOP\n query := format($q$\n SELECT count(*) FROM items\n WHERE datetime BETWEEN %L AND %L AND %s\n $q$, lower(p.tstzrange), upper(p.tstzrange), _where\n );\n RAISE NOTICE 'Query %', query;\n RAISE NOTICE 'Partition %, Count %, Total %',p.partition, subtotal, total;\n EXECUTE query INTO subtotal;\n total := subtotal + total;\nEND LOOP;\nRETURN total;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n/* looks for a geometry in a stac item first from geometry and falling back to bbox */\nCREATE OR REPLACE FUNCTION stac_geom(value jsonb) RETURNS geometry AS $$\nSELECT\n CASE\n WHEN value->>'geometry' IS NOT NULL THEN\n ST_GeomFromGeoJSON(value->>'geometry')\n WHEN value->>'bbox' IS NOT NULL THEN\n ST_MakeEnvelope(\n (value->'bbox'->>0)::float,\n (value->'bbox'->>1)::float,\n (value->'bbox'->>2)::float,\n (value->'bbox'->>3)::float,\n 4326\n )\n ELSE NULL\n END as geometry\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION stac_datetime(value jsonb) RETURNS timestamptz AS $$\nSELECT COALESCE(\n (value->'properties'->>'datetime')::timestamptz,\n (value->'properties'->>'start_datetime')::timestamptz\n);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_end_datetime(value jsonb) RETURNS timestamptz AS $$\nSELECT COALESCE(\n (value->'properties'->>'datetime')::timestamptz,\n (value->'properties'->>'end_datetime')::timestamptz\n);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\n\nCREATE OR REPLACE FUNCTION stac_daterange(value jsonb) RETURNS tstzrange AS $$\nSELECT tstzrange(stac_datetime(value),stac_end_datetime(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE TABLE IF NOT EXISTS collections (\n id VARCHAR GENERATED ALWAYS AS (content->>'id') STORED PRIMARY KEY,\n content JSONB\n);\n\nCREATE OR REPLACE FUNCTION create_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection(data jsonb) RETURNS VOID AS $$\nDECLARE\nout collections%ROWTYPE;\nBEGIN\n UPDATE collections SET content=data WHERE id = data->>'id' RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ON CONFLICT (id) DO\n UPDATE\n SET content=EXCLUDED.content\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION delete_collection(_id text) RETURNS VOID AS $$\nDECLARE\nout collections%ROWTYPE;\nBEGIN\n DELETE FROM collections WHERE id = _id RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION get_collection(id text) RETURNS jsonb AS $$\nSELECT content FROM collections\nWHERE id=$1\n;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION all_collections() RETURNS jsonb AS $$\nSELECT jsonb_agg(content) FROM collections;\n;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE TABLE items (\n id text NOT NULL,\n geometry geometry NOT NULL,\n collection_id text NOT NULL,\n datetime timestamptz NOT NULL,\n end_datetime timestamptz NOT NULL,\n properties jsonb NOT NULL,\n content JSONB NOT NULL\n)\nPARTITION BY RANGE (datetime)\n;\n\nCREATE OR REPLACE FUNCTION properties_idx (IN content jsonb) RETURNS jsonb AS $$\n with recursive extract_all as\n (\n select\n ARRAY[key]::text[] as path,\n ARRAY[key]::text[] as fullpath,\n value\n FROM jsonb_each(content->'properties')\n union all\n select\n CASE WHEN obj_key IS NOT NULL THEN path || obj_key ELSE path END,\n path || coalesce(obj_key, (arr_key- 1)::text),\n coalesce(obj_value, arr_value)\n from extract_all\n left join lateral\n jsonb_each(case jsonb_typeof(value) when 'object' then value end)\n as o(obj_key, obj_value)\n on jsonb_typeof(value) = 'object'\n left join lateral\n jsonb_array_elements(case jsonb_typeof(value) when 'array' then value end)\n with ordinality as a(arr_value, arr_key)\n on jsonb_typeof(value) = 'array'\n where obj_key is not null or arr_key is not null\n )\n , paths AS (\n select\n array_to_string(path, '.') as path,\n value\n FROM extract_all\n WHERE\n jsonb_typeof(value) NOT IN ('array','object')\n ), grouped AS (\n SELECT path, jsonb_agg(distinct value) vals FROM paths group by path\n ) SELECT coalesce(jsonb_object_agg(path, CASE WHEN jsonb_array_length(vals)=1 THEN vals->0 ELSE vals END) - '{datetime}'::text[], '{}'::jsonb) FROM grouped\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET JIT TO OFF;\n\nCREATE INDEX \"datetime_idx\" ON items (datetime);\nCREATE INDEX \"end_datetime_idx\" ON items (end_datetime);\nCREATE INDEX \"properties_idx\" ON items USING GIN (properties jsonb_path_ops);\nCREATE INDEX \"collection_idx\" ON items (collection_id);\nCREATE INDEX \"geometry_idx\" ON items USING GIST (geometry);\nCREATE UNIQUE INDEX \"items_id_datetime_idx\" ON items (datetime, id);\n\nALTER TABLE items ADD CONSTRAINT items_collections_fk FOREIGN KEY (collection_id) REFERENCES collections(id) ON DELETE CASCADE DEFERRABLE;\n\nCREATE OR REPLACE FUNCTION analyze_empty_partitions() RETURNS VOID AS $$\nDECLARE\n p text;\nBEGIN\n FOR p IN SELECT partition FROM all_items_partitions WHERE est_cnt = 0 LOOP\n EXECUTE format('ANALYZE %I;', p);\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION items_partition_name(timestamptz) RETURNS text AS $$\n SELECT to_char($1, '\"items_p\"IYYY\"w\"IW');\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION items_partition_exists(text) RETURNS boolean AS $$\n SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_class WHERE relname=$1);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION items_partition_exists(timestamptz) RETURNS boolean AS $$\n SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_class WHERE relname=items_partition_name($1));\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION items_partition_create_worker(partition text, partition_start timestamptz, partition_end timestamptz) RETURNS VOID AS $$\nDECLARE\n err_context text;\nBEGIN\n EXECUTE format(\n $f$\n CREATE TABLE IF NOT EXISTS %1$I PARTITION OF items\n FOR VALUES FROM (%2$L) TO (%3$L);\n CREATE UNIQUE INDEX IF NOT EXISTS %4$I ON %1$I (id);\n $f$,\n partition,\n partition_start,\n partition_end,\n concat(partition, '_id_pk')\n );\nEXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', partition;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH to pgstac, public;\n\nCREATE OR REPLACE FUNCTION items_partition_create(ts timestamptz) RETURNS text AS $$\nDECLARE\n partition text := items_partition_name(ts);\n partition_start timestamptz;\n partition_end timestamptz;\nBEGIN\n IF items_partition_exists(partition) THEN\n RETURN partition;\n END IF;\n partition_start := date_trunc('week', ts);\n partition_end := partition_start + '1 week'::interval;\n PERFORM items_partition_create_worker(partition, partition_start, partition_end);\n RAISE NOTICE 'partition: %', partition;\n RETURN partition;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION items_partition_create(st timestamptz, et timestamptz) RETURNS SETOF text AS $$\nWITH t AS (\n SELECT\n generate_series(\n date_trunc('week',st),\n date_trunc('week', et),\n '1 week'::interval\n ) w\n)\nSELECT items_partition_create(w) FROM t;\n$$ LANGUAGE SQL;\n\n\nCREATE UNLOGGED TABLE items_staging (\n content JSONB NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION items_staging_insert_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n mindate timestamptz;\n maxdate timestamptz;\n partition text;\nBEGIN\n SELECT min(stac_datetime(content)), max(stac_datetime(content)) INTO mindate, maxdate FROM newdata;\n PERFORM items_partition_create(mindate, maxdate);\n INSERT INTO items (id, geometry, collection_id, datetime, end_datetime, properties, content)\n SELECT\n content->>'id',\n stac_geom(content),\n content->>'collection',\n stac_datetime(content),\n stac_end_datetime(content),\n properties_idx(content),\n content\n FROM newdata\n ;\n DELETE FROM items_staging;\n PERFORM analyze_empty_partitions();\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER items_staging_insert_trigger AFTER INSERT ON items_staging REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_insert_triggerfunc();\n\n\nCREATE UNLOGGED TABLE items_staging_ignore (\n content JSONB NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION items_staging_ignore_insert_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n mindate timestamptz;\n maxdate timestamptz;\n partition text;\nBEGIN\n SELECT min(stac_datetime(content)), max(stac_datetime(content)) INTO mindate, maxdate FROM newdata;\n PERFORM items_partition_create(mindate, maxdate);\n INSERT INTO items (id, geometry, collection_id, datetime, end_datetime, properties, content)\n SELECT\n content->>'id',\n stac_geom(content),\n content->>'collection',\n stac_datetime(content),\n stac_end_datetime(content),\n properties_idx(content),\n content\n FROM newdata\n ON CONFLICT DO NOTHING\n ;\n DELETE FROM items_staging_ignore;\n PERFORM analyze_empty_partitions();\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER items_staging_ignore_insert_trigger AFTER INSERT ON items_staging_ignore REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_ignore_insert_triggerfunc();\n\nCREATE UNLOGGED TABLE items_staging_upsert (\n content JSONB NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION items_staging_upsert_insert_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n mindate timestamptz;\n maxdate timestamptz;\n partition text;\nBEGIN\n SELECT min(stac_datetime(content)), max(stac_datetime(content)) INTO mindate, maxdate FROM newdata;\n PERFORM items_partition_create(mindate, maxdate);\n INSERT INTO items (id, geometry, collection_id, datetime, end_datetime, properties, content)\n SELECT\n content->>'id',\n stac_geom(content),\n content->>'collection',\n stac_datetime(content),\n stac_end_datetime(content),\n properties_idx(content),\n content\n FROM newdata\n ON CONFLICT (datetime, id) DO UPDATE SET\n content = EXCLUDED.content\n WHERE items.content IS DISTINCT FROM EXCLUDED.content\n ;\n DELETE FROM items_staging_upsert;\n PERFORM analyze_empty_partitions();\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER items_staging_upsert_insert_trigger AFTER INSERT ON items_staging_upsert REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_upsert_insert_triggerfunc();\n\nCREATE OR REPLACE FUNCTION items_update_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\nBEGIN\n NEW.id := NEW.content->>'id';\n NEW.datetime := stac_datetime(NEW.content);\n NEW.end_datetime := stac_end_datetime(NEW.content);\n NEW.collection_id := NEW.content->>'collection';\n NEW.geometry := stac_geom(NEW.content);\n NEW.properties := properties_idx(NEW.content);\n IF TG_OP = 'UPDATE' AND NEW IS NOT DISTINCT FROM OLD THEN\n RETURN NULL;\n END IF;\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER items_update_trigger BEFORE UPDATE ON items\n FOR EACH ROW EXECUTE PROCEDURE items_update_triggerfunc();\n\n/*\nView to get a table of available items partitions\nwith date ranges\n*/\nCREATE VIEW all_items_partitions AS\nWITH base AS\n(SELECT\n c.oid::pg_catalog.regclass::text as partition,\n pg_catalog.pg_get_expr(c.relpartbound, c.oid) as _constraint,\n regexp_matches(\n pg_catalog.pg_get_expr(c.relpartbound, c.oid),\n E'\\\\(''\\([0-9 :+-]*\\)''\\\\).*\\\\(''\\([0-9 :+-]*\\)''\\\\)'\n ) as t,\n reltuples::bigint as est_cnt\nFROM pg_catalog.pg_class c, pg_catalog.pg_inherits i\nWHERE c.oid = i.inhrelid AND i.inhparent = 'items'::regclass)\nSELECT partition, tstzrange(\n t[1]::timestamptz,\n t[2]::timestamptz\n), est_cnt\nFROM base\nORDER BY 2 desc;\n\nCREATE OR REPLACE VIEW items_partitions AS\nSELECT * FROM all_items_partitions WHERE est_cnt>0;\n\nCREATE OR REPLACE FUNCTION get_item(_id text) RETURNS jsonb AS $$\n SELECT content FROM items WHERE id=_id;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION delete_item(_id text) RETURNS VOID AS $$\nDECLARE\nout items%ROWTYPE;\nBEGIN\n DELETE FROM items WHERE id = _id RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION create_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION update_item(data jsonb) RETURNS VOID AS $$\nDECLARE\n out items%ROWTYPE;\nBEGIN\n UPDATE items SET content=data WHERE id = data->>'id' RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION create_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION collection_bbox(id text) RETURNS jsonb AS $$\nSELECT (replace(replace(replace(st_extent(geometry)::text,'BOX(','[['),')',']]'),' ',','))::jsonb\nFROM items WHERE collection_id=$1;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION collection_temporal_extent(id text) RETURNS jsonb AS $$\nSELECT to_jsonb(array[array[min(datetime)::text, max(datetime)::text]])\nFROM items WHERE collection_id=$1;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection_extents() RETURNS VOID AS $$\nUPDATE collections SET\n content = content ||\n jsonb_build_object(\n 'extent', jsonb_build_object(\n 'spatial', jsonb_build_object(\n 'bbox', collection_bbox(collections.id)\n ),\n 'temporal', jsonb_build_object(\n 'interval', collection_temporal_extent(collections.id)\n )\n )\n )\n;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION items_path(\n IN dotpath text,\n OUT field text,\n OUT path text,\n OUT path_txt text,\n OUT jsonpath text,\n OUT eq text\n) RETURNS RECORD AS $$\nDECLARE\npath_elements text[];\nlast_element text;\nBEGIN\ndotpath := replace(trim(dotpath), 'properties.', '');\n\nIF dotpath = '' THEN\n RETURN;\nEND IF;\n\npath_elements := string_to_array(dotpath, '.');\njsonpath := NULL;\n\nIF path_elements[1] IN ('id','geometry','datetime') THEN\n field := path_elements[1];\n path_elements := path_elements[2:];\nELSIF path_elements[1] = 'collection' THEN\n field := 'collection_id';\n path_elements := path_elements[2:];\nELSIF path_elements[1] IN ('links', 'assets', 'stac_version', 'stac_extensions') THEN\n field := 'content';\nELSE\n field := 'content';\n path_elements := '{properties}'::text[] || path_elements;\nEND IF;\nIF cardinality(path_elements)<1 THEN\n path := field;\n path_txt := field;\n jsonpath := '$';\n eq := NULL; -- format($F$ %s = %%s $F$, field);\n RETURN;\nEND IF;\n\n\nlast_element := path_elements[cardinality(path_elements)];\npath_elements := path_elements[1:cardinality(path_elements)-1];\njsonpath := concat(array_to_string('{$}'::text[] || array_map_ident(path_elements), '.'), '.', quote_ident(last_element));\npath_elements := array_map_literal(path_elements);\npath := format($F$ properties->%s $F$, quote_literal(dotpath));\npath_txt := format($F$ properties->>%s $F$, quote_literal(dotpath));\neq := format($F$ properties @? '$.%s[*] ? (@ == %%s) '$F$, quote_ident(dotpath));\n\nRAISE NOTICE 'ITEMS PATH -- % % % % %', field, path, path_txt, jsonpath, eq;\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION parse_dtrange(IN _indate jsonb, OUT _tstzrange tstzrange) AS $$\nWITH t AS (\n SELECT CASE\n WHEN jsonb_typeof(_indate) = 'array' THEN\n textarr(_indate)\n ELSE\n regexp_split_to_array(\n btrim(_indate::text,'\"'),\n '/'\n )\n END AS arr\n)\n, t1 AS (\n SELECT\n CASE\n WHEN array_upper(arr,1) = 1 OR arr[1] = '..' OR arr[1] IS NULL THEN '-infinity'::timestamptz\n ELSE arr[1]::timestamptz\n END AS st,\n CASE\n WHEN array_upper(arr,1) = 1 THEN arr[1]::timestamptz\n WHEN arr[2] = '..' OR arr[2] IS NULL THEN 'infinity'::timestamptz\n ELSE arr[2]::timestamptz\n END AS et\n FROM t\n)\nSELECT\n tstzrange(st,et)\nFROM t1;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION bbox_geom(_bbox jsonb) RETURNS geometry AS $$\nSELECT CASE jsonb_array_length(_bbox)\n WHEN 4 THEN\n ST_SetSRID(ST_MakeEnvelope(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float,\n (_bbox->>3)::float\n ),4326)\n WHEN 6 THEN\n ST_SetSRID(ST_3DMakeBox(\n ST_MakePoint(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float\n ),\n ST_MakePoint(\n (_bbox->>3)::float,\n (_bbox->>4)::float,\n (_bbox->>5)::float\n )\n ),4326)\n ELSE null END;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION cql_and_append(existing jsonb, newfilters jsonb) RETURNS jsonb AS $$\nSELECT CASE WHEN existing ? 'filter' AND newfilters IS NOT NULL THEN\n jsonb_build_object(\n 'and',\n jsonb_build_array(\n existing->'filter',\n newfilters\n )\n )\nELSE\n newfilters\nEND;\n$$ LANGUAGE SQL;\n\n\n-- ADDs base filters (ids, collections, datetime, bbox, intersects) that are\n-- added outside of the filter/query in the stac request\nCREATE OR REPLACE FUNCTION add_filters_to_cql(j jsonb) RETURNS jsonb AS $$\nDECLARE\nnewprop jsonb;\nnewprops jsonb := '[]'::jsonb;\nBEGIN\nIF j ? 'ids' THEN\n newprop := jsonb_build_object(\n 'in',\n jsonb_build_array(\n '{\"property\":\"id\"}'::jsonb,\n j->'ids'\n )\n );\n newprops := jsonb_insert(newprops, '{1}', newprop);\nEND IF;\nIF j ? 'collections' THEN\n newprop := jsonb_build_object(\n 'in',\n jsonb_build_array(\n '{\"property\":\"collection\"}'::jsonb,\n j->'collections'\n )\n );\n newprops := jsonb_insert(newprops, '{1}', newprop);\nEND IF;\n\nIF j ? 'datetime' THEN\n newprop := format(\n '{\"anyinteracts\":[{\"property\":\"datetime\"}, %s]}',\n j->'datetime'\n );\n newprops := jsonb_insert(newprops, '{1}', newprop);\nEND IF;\n\nIF j ? 'bbox' THEN\n newprop := format(\n '{\"intersects\":[{\"property\":\"geometry\"}, %s]}',\n j->'bbox'\n );\n newprops := jsonb_insert(newprops, '{1}', newprop);\nEND IF;\n\nIF j ? 'intersects' THEN\n newprop := format(\n '{\"intersects\":[{\"property\":\"geometry\"}, %s]}',\n j->'intersects'\n );\n newprops := jsonb_insert(newprops, '{1}', newprop);\nEND IF;\n\nRAISE NOTICE 'newprops: %', newprops;\n\nIF newprops IS NOT NULL AND jsonb_array_length(newprops) > 0 THEN\n return jsonb_set(\n j,\n '{filter}',\n cql_and_append(j, jsonb_build_object('and', newprops))\n ) - '{ids,collections,datetime,bbox,intersects}'::text[];\nEND IF;\n\nreturn j;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION query_to_cqlfilter(j jsonb) RETURNS jsonb AS $$\n-- Translates anything passed in through the deprecated \"query\" into equivalent CQL\nWITH t AS (\n SELECT key as property, value as ops\n FROM jsonb_each(j->'query')\n), t2 AS (\n SELECT property, (jsonb_each(ops)).*\n FROM t WHERE jsonb_typeof(ops) = 'object'\n UNION ALL\n SELECT property, 'eq', ops\n FROM t WHERE jsonb_typeof(ops) != 'object'\n), t3 AS (\nSELECT\n jsonb_strip_nulls(jsonb_build_object(\n 'and',\n jsonb_agg(\n jsonb_build_object(\n key,\n jsonb_build_array(\n jsonb_build_object('property',property),\n value\n )\n )\n )\n )) as qcql FROM t2\n)\nSELECT\n CASE WHEN qcql IS NOT NULL THEN\n jsonb_set(j, '{filter}', cql_and_append(j, qcql)) - 'query'\n ELSE j\n END\nFROM t3\n;\n$$ LANGUAGE SQL;\n\n\n\nCREATE OR REPLACE FUNCTION temporal_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\nll text := 'datetime';\nlh text := 'end_datetime';\nrrange tstzrange;\nrl text;\nrh text;\noutq text;\nBEGIN\nrrange := parse_dtrange(args->1);\nRAISE NOTICE 'Constructing temporal query OP: %, ARGS: %, RRANGE: %', op, args, rrange;\nop := lower(op);\nrl := format('%L::timestamptz', lower(rrange));\nrh := format('%L::timestamptz', upper(rrange));\noutq := CASE op\n WHEN 't_before' THEN 'lh < rl'\n WHEN 't_after' THEN 'll > rh'\n WHEN 't_meets' THEN 'lh = rl'\n WHEN 't_metby' THEN 'll = rh'\n WHEN 't_overlaps' THEN 'll < rl AND rl < lh < rh'\n WHEN 't_overlappedby' THEN 'rl < ll < rh AND lh > rh'\n WHEN 't_starts' THEN 'll = rl AND lh < rh'\n WHEN 't_startedby' THEN 'll = rl AND lh > rh'\n WHEN 't_during' THEN 'll > rl AND lh < rh'\n WHEN 't_contains' THEN 'll < rl AND lh > rh'\n WHEN 't_finishes' THEN 'll > rl AND lh = rh'\n WHEN 't_finishedby' THEN 'll < rl AND lh = rh'\n WHEN 't_equals' THEN 'll = rl AND lh = rh'\n WHEN 't_disjoint' THEN 'NOT (ll <= rh AND lh >= rl)'\n WHEN 't_intersects' THEN 'll <= rh AND lh >= rl'\n WHEN 'anyinteracts' THEN 'll <= rh AND lh >= rl'\nEND;\noutq := regexp_replace(outq, '\\mll\\M', ll);\noutq := regexp_replace(outq, '\\mlh\\M', lh);\noutq := regexp_replace(outq, '\\mrl\\M', rl);\noutq := regexp_replace(outq, '\\mrh\\M', rh);\noutq := format('(%s)', outq);\nRETURN outq;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION spatial_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\ngeom text;\nj jsonb := args->1;\nBEGIN\nop := lower(op);\nRAISE NOTICE 'Constructing spatial query OP: %, ARGS: %', op, args;\nIF op NOT IN ('s_equals','s_disjoint','s_touches','s_within','s_overlaps','s_crosses','s_intersects','intersects','s_contains') THEN\n RAISE EXCEPTION 'Spatial Operator % Not Supported', op;\nEND IF;\nop := regexp_replace(op, '^s_', 'st_');\nIF op = 'intersects' THEN\n op := 'st_intersects';\nEND IF;\n-- Convert geometry to WKB string\nIF j ? 'type' AND j ? 'coordinates' THEN\n geom := st_geomfromgeojson(j)::text;\nELSIF jsonb_typeof(j) = 'array' THEN\n geom := bbox_geom(j)::text;\nEND IF;\n\nRETURN format('%s(geometry, %L::geometry)', op, geom);\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n/* cql_query_op -- Parses a CQL query operation, recursing when necessary\n IN jsonb -- a subelement from a valid stac query\n IN text -- the operator being used on elements passed in\n RETURNS a SQL fragment to be used in a WHERE clause\n*/\nCREATE OR REPLACE FUNCTION cql_query_op(j jsonb, _op text DEFAULT NULL) RETURNS text AS $$\nDECLARE\njtype text := jsonb_typeof(j);\nop text := lower(_op);\nops jsonb :=\n '{\n \"eq\": \"%s = %s\",\n \"lt\": \"%s < %s\",\n \"lte\": \"%s <= %s\",\n \"gt\": \"%s > %s\",\n \"gte\": \"%s >= %s\",\n \"like\": \"%s LIKE %s\",\n \"+\": \"%s + %s\",\n \"-\": \"%s - %s\",\n \"*\": \"%s * %s\",\n \"/\": \"%s / %s\",\n \"in\": \"%s = ANY (%s)\",\n \"not\": \"NOT (%s)\",\n \"between\": \"%s BETWEEN %s AND %s\",\n \"lower\":\"lower(%s)\"\n }'::jsonb;\nret text;\nargs text[] := NULL;\n\nBEGIN\nRAISE NOTICE 'j: %, op: %, jtype: %', j, op, jtype;\n\n-- Set Lower Case on Both Arguments When Case Insensitive Flag Set\nIF op in ('eq','lt','lte','gt','gte','like') AND jsonb_typeof(j->2) = 'boolean' THEN\n IF (j->>2)::boolean THEN\n RETURN format(concat('(',ops->>op,')'), cql_query_op(jsonb_build_array(j->0), 'lower'), cql_query_op(jsonb_build_array(j->1), 'lower'));\n END IF;\nEND IF;\n\n-- Special Case when comparing a property in a jsonb field to a string or number using eq\n-- Allows to leverage GIN index on jsonb fields\nIF op = 'eq' THEN\n IF j->0 ? 'property'\n AND jsonb_typeof(j->1) IN ('number','string')\n AND (items_path(j->0->>'property')).eq IS NOT NULL\n THEN\n RETURN format((items_path(j->0->>'property')).eq, j->1);\n END IF;\nEND IF;\n\nIF op ilike 't_%' or op = 'anyinteracts' THEN\n RETURN temporal_op_query(op, j);\nEND IF;\n\nIF op ilike 's_%' or op = 'intersects' THEN\n RETURN spatial_op_query(op, j);\nEND IF;\n\n\nIF jtype = 'object' THEN\n RAISE NOTICE 'parsing object';\n IF j ? 'property' THEN\n -- Convert the property to be used as an identifier\n return (items_path(j->>'property')).path_txt;\n ELSIF _op IS NULL THEN\n -- Iterate to convert elements in an object where the operator has not been set\n -- Combining with AND\n SELECT\n array_to_string(array_agg(cql_query_op(e.value, e.key)), ' AND ')\n INTO ret\n FROM jsonb_each(j) e;\n RETURN ret;\n END IF;\nEND IF;\n\nIF jtype = 'string' THEN\n RETURN quote_literal(j->>0);\nEND IF;\n\nIF jtype ='number' THEN\n RETURN (j->>0)::numeric;\nEND IF;\n\nIF jtype = 'array' AND op IS NULL THEN\n RAISE NOTICE 'Parsing array into array arg. j: %', j;\n SELECT format($f$ '{%s}'::text[] $f$, string_agg(e,',')) INTO ret FROM jsonb_array_elements_text(j) e;\n RETURN ret;\nEND IF;\n\n\n-- If the type of the passed json is an array\n-- Calculate the arguments that will be passed to functions/operators\nIF jtype = 'array' THEN\n RAISE NOTICE 'Parsing array into args. j: %', j;\n -- If any argument is numeric, cast any text arguments to numeric\n IF j @? '$[*] ? (@.type() == \"number\")' THEN\n SELECT INTO args\n array_agg(concat('(',cql_query_op(e),')::numeric'))\n FROM jsonb_array_elements(j) e;\n ELSE\n SELECT INTO args\n array_agg(cql_query_op(e))\n FROM jsonb_array_elements(j) e;\n END IF;\n --RETURN args;\nEND IF;\nRAISE NOTICE 'ARGS after array cleaning: %', args;\n\nIF op IS NULL THEN\n RETURN args::text[];\nEND IF;\n\nIF args IS NULL OR cardinality(args) < 1 THEN\n RAISE NOTICE 'No Args';\n RETURN '';\nEND IF;\n\nIF op IN ('and','or') THEN\n SELECT\n CONCAT(\n '(',\n array_to_string(args, UPPER(CONCAT(' ',op,' '))),\n ')'\n ) INTO ret\n FROM jsonb_array_elements(j) e;\n RETURN ret;\nEND IF;\n\n-- If the op is in the ops json then run using the template in the json\nIF ops ? op THEN\n RAISE NOTICE 'ARGS: % MAPPED: %',args, array_map_literal(args);\n\n RETURN format(concat('(',ops->>op,')'), VARIADIC args);\nEND IF;\n\nRETURN j->>0;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n\n\nCREATE OR REPLACE FUNCTION cql_to_where(_search jsonb = '{}'::jsonb) RETURNS text AS $$\nDECLARE\nsearch jsonb := _search;\n_where text;\nBEGIN\nRAISE NOTICE 'SEARCH CQL 1: %', search;\n\n-- Convert any old style stac query to cql\nsearch := query_to_cqlfilter(search);\n\nRAISE NOTICE 'SEARCH CQL 2: %', search;\n\n-- Convert item,collection,datetime,bbox,intersects to cql\nsearch := add_filters_to_cql(search);\n\nRAISE NOTICE 'SEARCH CQL Final: %', search;\n_where := cql_query_op(search->'filter');\n\nIF trim(_where) = '' THEN\n _where := NULL;\nEND IF;\n_where := coalesce(_where, ' TRUE ');\nRETURN _where;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION parse_sort_dir(_dir text, reverse boolean default false) RETURNS text AS $$\nWITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n) SELECT\n CASE\n WHEN NOT reverse THEN d\n WHEN d = 'ASC' THEN 'DESC'\n WHEN d = 'DESC' THEN 'ASC'\n END\nFROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION sort_dir_to_op(_dir text, prev boolean default false) RETURNS text AS $$\nWITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n) SELECT\n CASE\n WHEN d = 'ASC' AND prev THEN '<='\n WHEN d = 'DESC' AND prev THEN '>='\n WHEN d = 'ASC' THEN '>='\n WHEN d = 'DESC' THEN '<='\n END\nFROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION sort_sqlorderby(\n _search jsonb DEFAULT NULL,\n reverse boolean DEFAULT FALSE\n) RETURNS text AS $$\nWITH sortby AS (\n SELECT coalesce(_search->'sortby','[{\"field\":\"datetime\", \"direction\":\"desc\"}]') as sort\n), withid AS (\n SELECT CASE\n WHEN sort @? '$[*] ? (@.field == \"id\")' THEN sort\n ELSE sort || '[{\"field\":\"id\", \"direction\":\"desc\"}]'::jsonb\n END as sort\n FROM sortby\n), withid_rows AS (\n SELECT jsonb_array_elements(sort) as value FROM withid\n),sorts AS (\n SELECT\n (items_path(value->>'field')).path as key,\n parse_sort_dir(value->>'direction', reverse) as dir\n FROM withid_rows\n)\nSELECT array_to_string(\n array_agg(concat(key, ' ', dir)),\n ', '\n) FROM sorts;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_sort_dir(sort_item jsonb) RETURNS text AS $$\nSELECT CASE WHEN sort_item->>'direction' ILIKE 'desc%' THEN 'DESC' ELSE 'ASC' END;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION get_token_filter(_search jsonb = '{}'::jsonb, token_rec jsonb DEFAULT NULL) RETURNS text AS $$\nDECLARE\n token_id text;\n filters text[] := '{}'::text[];\n prev boolean := TRUE;\n field text;\n dir text;\n sort record;\n orfilters text[] := '{}'::text[];\n andfilters text[] := '{}'::text[];\n output text;\n token_where text;\nBEGIN\n -- If no token provided return NULL\n IF token_rec IS NULL THEN\n IF NOT (_search ? 'token' AND\n (\n (_search->>'token' ILIKE 'prev:%')\n OR\n (_search->>'token' ILIKE 'next:%')\n )\n ) THEN\n RETURN NULL;\n END IF;\n prev := (_search->>'token' ILIKE 'prev:%');\n token_id := substr(_search->>'token', 6);\n SELECT to_jsonb(items) INTO token_rec FROM items WHERE id=token_id;\n END IF;\n RAISE NOTICE 'TOKEN ID: %', token_rec->'id';\n\n CREATE TEMP TABLE sorts (\n _row int GENERATED ALWAYS AS IDENTITY NOT NULL,\n _field text PRIMARY KEY,\n _dir text NOT NULL,\n _val text\n ) ON COMMIT DROP;\n\n -- Make sure we only have distinct columns to sort with taking the first one we get\n INSERT INTO sorts (_field, _dir)\n SELECT\n (items_path(value->>'field')).path,\n get_sort_dir(value)\n FROM\n jsonb_array_elements(coalesce(_search->'sortby','[{\"field\":\"datetime\",\"direction\":\"desc\"}]'))\n ON CONFLICT DO NOTHING\n ;\n RAISE NOTICE 'sorts 1: %', (SELECT jsonb_agg(to_json(sorts)) FROM sorts);\n -- Get the first sort direction provided. As the id is a primary key, if there are any\n -- sorts after id they won't do anything, so make sure that id is the last sort item.\n SELECT _dir INTO dir FROM sorts ORDER BY _row ASC LIMIT 1;\n IF EXISTS (SELECT 1 FROM sorts WHERE _field = 'id') THEN\n DELETE FROM sorts WHERE _row > (SELECT _row FROM sorts WHERE _field = 'id' ORDER BY _row ASC);\n ELSE\n INSERT INTO sorts (_field, _dir) VALUES ('id', dir);\n END IF;\n\n -- Add value from looked up item to the sorts table\n UPDATE sorts SET _val=quote_literal(token_rec->>_field);\n\n -- Check if all sorts are the same direction and use row comparison\n -- to filter\n RAISE NOTICE 'sorts 2: %', (SELECT jsonb_agg(to_json(sorts)) FROM sorts);\n\n IF (SELECT count(DISTINCT _dir) FROM sorts) = 1 THEN\n SELECT format(\n '(%s) %s (%s)',\n concat_ws(', ', VARIADIC array_agg(quote_ident(_field))),\n CASE WHEN (prev AND dir = 'ASC') OR (NOT prev AND dir = 'DESC') THEN '<' ELSE '>' END,\n concat_ws(', ', VARIADIC array_agg(_val))\n ) INTO output FROM sorts\n WHERE token_rec ? _field\n ;\n ELSE\n FOR sort IN SELECT * FROM sorts ORDER BY _row asc LOOP\n RAISE NOTICE 'SORT: %', sort;\n IF sort._row = 1 THEN\n orfilters := orfilters || format('(%s %s %s)',\n quote_ident(sort._field),\n CASE WHEN (prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC') THEN '<' ELSE '>' END,\n sort._val\n );\n ELSE\n orfilters := orfilters || format('(%s AND %s %s %s)',\n array_to_string(andfilters, ' AND '),\n quote_ident(sort._field),\n CASE WHEN (prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC') THEN '<' ELSE '>' END,\n sort._val\n );\n\n END IF;\n andfilters := andfilters || format('%s = %s',\n quote_ident(sort._field),\n sort._val\n );\n END LOOP;\n output := array_to_string(orfilters, ' OR ');\n END IF;\n DROP TABLE IF EXISTS sorts;\n token_where := concat('(',coalesce(output,'true'),')');\n IF trim(token_where) = '' THEN\n token_where := NULL;\n END IF;\n RAISE NOTICE 'TOKEN_WHERE: |%|',token_where;\n RETURN token_where;\n END;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION search_tohash(jsonb) RETURNS jsonb AS $$\n SELECT $1 - '{token,limit,context,includes,excludes}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION search_hash(jsonb) RETURNS text AS $$\n SELECT md5(search_tohash($1)::text);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE TABLE IF NOT EXISTS searches(\n hash text GENERATED ALWAYS AS (search_hash(search)) STORED PRIMARY KEY,\n search jsonb NOT NULL,\n _where text,\n orderby text,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n statslastupdated timestamptz,\n estimated_count bigint,\n total_count bigint\n);\n\nCREATE OR REPLACE FUNCTION search_query(_search jsonb = '{}'::jsonb, updatestats boolean DEFAULT false) RETURNS searches AS $$\nDECLARE\n search searches%ROWTYPE;\nBEGIN\nINSERT INTO searches (search)\n VALUES (search_tohash(_search))\n ON CONFLICT DO NOTHING\n RETURNING * INTO search;\nIF search.hash IS NULL THEN\n SELECT * INTO search FROM searches WHERE hash=search_hash(_search);\nEND IF;\nIF search._where IS NULL THEN\n search._where := cql_to_where(_search);\nEND IF;\nIF search.orderby IS NULL THEN\n search.orderby := sort_sqlorderby(_search);\nEND IF;\n\nIF search.statslastupdated IS NULL OR age(search.statslastupdated) > '1 day'::interval OR (_search ? 'context' AND search.total_count IS NULL) THEN\n updatestats := TRUE;\nEND IF;\n\nIF updatestats THEN\n -- Get Estimated Stats\n RAISE NOTICE 'Getting stats for %', search._where;\n search.estimated_count := estimated_count(search._where);\n RAISE NOTICE 'Estimated Count: %', search.estimated_count;\n\n IF _search ? 'context' OR search.estimated_count < 10000 THEN\n --search.total_count := partition_count(search._where);\n EXECUTE format(\n 'SELECT count(*) FROM items WHERE %s',\n search._where\n ) INTO search.total_count;\n RAISE NOTICE 'Actual Count: %', search.total_count;\n ELSE\n search.total_count := NULL;\n END IF;\n search.statslastupdated := now();\nEND IF;\n\nsearch.lastused := now();\nsearch.usecount := coalesce(search.usecount,0) + 1;\nRAISE NOTICE 'SEARCH: %', search;\nUPDATE searches SET\n _where = search._where,\n orderby = search.orderby,\n lastused = search.lastused,\n usecount = search.usecount,\n statslastupdated = search.statslastupdated,\n estimated_count = search.estimated_count,\n total_count = search.total_count\nWHERE hash = search.hash\n;\nRETURN search;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n\nCREATE OR REPLACE FUNCTION search(_search jsonb = '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n searches searches%ROWTYPE;\n _where text;\n token_where text;\n full_where text;\n orderby text;\n query text;\n token_type text := substr(_search->>'token',1,4);\n _limit int := coalesce((_search->>'limit')::int, 10);\n curs refcursor;\n cntr int := 0;\n iter_record items%ROWTYPE;\n first_record items%ROWTYPE;\n last_record items%ROWTYPE;\n out_records jsonb := '[]'::jsonb;\n prev_query text;\n next text;\n prev_id text;\n has_next boolean := false;\n has_prev boolean := false;\n prev text;\n total_count bigint;\n context jsonb;\n collection jsonb;\n includes text[];\n excludes text[];\n exit_flag boolean := FALSE;\n batches int := 0;\n timer timestamptz := clock_timestamp();\nBEGIN\nsearches := search_query(_search);\n_where := searches._where;\norderby := searches.orderby;\ntotal_count := coalesce(searches.total_count, searches.estimated_count);\n\n\nIF token_type='prev' THEN\n token_where := get_token_filter(_search, null::jsonb);\n orderby := sort_sqlorderby(_search, TRUE);\nEND IF;\nIF token_type='next' THEN\n token_where := get_token_filter(_search, null::jsonb);\nEND IF;\n\nfull_where := concat_ws(' AND ', _where, token_where);\nRAISE NOTICE 'FULL QUERY % %', full_where, clock_timestamp()-timer;\ntimer := clock_timestamp();\n\nFOR query IN SELECT partition_queries(full_where, orderby) LOOP\n timer := clock_timestamp();\n query := format('%s LIMIT %L', query, _limit + 1);\n RAISE NOTICE 'Partition Query: %', query;\n batches := batches + 1;\n curs = create_cursor(query);\n LOOP\n FETCH curs into iter_record;\n EXIT WHEN NOT FOUND;\n cntr := cntr + 1;\n last_record := iter_record;\n IF cntr = 1 THEN\n first_record := last_record;\n END IF;\n IF cntr <= _limit THEN\n out_records := out_records || last_record.content;\n ELSIF cntr > _limit THEN\n has_next := true;\n exit_flag := true;\n EXIT;\n END IF;\n END LOOP;\n RAISE NOTICE 'Query took %', clock_timestamp()-timer;\n timer := clock_timestamp();\n EXIT WHEN exit_flag;\nEND LOOP;\nRAISE NOTICE 'Scanned through % partitions.', batches;\n\n\n-- Flip things around if this was the result of a prev token query\nIF token_type='prev' THEN\n out_records := flip_jsonb_array(out_records);\n first_record := last_record;\nEND IF;\n\n-- If this query has a token, see if there is data before the first record\nIF _search ? 'token' THEN\n prev_query := format(\n 'SELECT 1 FROM items WHERE %s LIMIT 1',\n concat_ws(\n ' AND ',\n _where,\n trim(get_token_filter(_search, to_jsonb(first_record)))\n )\n );\n RAISE NOTICE 'Query to get previous record: % --- %', prev_query, first_record;\n EXECUTE prev_query INTO has_prev;\n IF FOUND and has_prev IS NOT NULL THEN\n RAISE NOTICE 'Query results from prev query: %', has_prev;\n has_prev := TRUE;\n END IF;\nEND IF;\nhas_prev := COALESCE(has_prev, FALSE);\n\nRAISE NOTICE 'token_type: %, has_next: %, has_prev: %', token_type, has_next, has_prev;\nIF has_prev THEN\n prev := out_records->0->>'id';\nEND IF;\nIF has_next OR token_type='prev' THEN\n next := out_records->-1->>'id';\nEND IF;\n\n\n-- include/exclude any fields following fields extension\nIF _search ? 'fields' THEN\n IF _search->'fields' ? 'exclude' THEN\n excludes=textarr(_search->'fields'->'exclude');\n END IF;\n IF _search->'fields' ? 'include' THEN\n includes=textarr(_search->'fields'->'include');\n IF array_length(includes, 1)>0 AND NOT 'id' = ANY (includes) THEN\n includes = includes || '{id}';\n END IF;\n END IF;\n SELECT jsonb_agg(filter_jsonb(row, includes, excludes)) INTO out_records FROM jsonb_array_elements(out_records) row;\nEND IF;\n\ncontext := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'matched', total_count,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n));\n\ncollection := jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb),\n 'next', next,\n 'prev', prev,\n 'context', context\n);\n\nRETURN collection;\nEND;\n$$ LANGUAGE PLPGSQL SET jit TO off;\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION tileenvelope(zoom int, x int, y int) RETURNS geometry AS $$\nWITH t AS (\n SELECT\n 20037508.3427892 as merc_max,\n -20037508.3427892 as merc_min,\n (2 * 20037508.3427892) / (2 ^ zoom) as tile_size\n)\nSELECT st_makeenvelope(\n merc_min + (tile_size * x),\n merc_max - (tile_size * (y + 1)),\n merc_min + (tile_size * (x + 1)),\n merc_max - (tile_size * y),\n 3857\n) FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;DROP FUNCTION IF EXISTS mercgrid;\n\n\nCREATE OR REPLACE FUNCTION ftime() RETURNS interval as $$\nSELECT age(clock_timestamp(), transaction_timestamp());\n$$ LANGUAGE SQL;\nSET SEARCH_PATH to pgstac, public;\n\nDROP FUNCTION IF EXISTS geometrysearch;\nCREATE OR REPLACE FUNCTION geometrysearch(\n IN geom geometry,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE, -- Return as soon as the passed in geometry is full covered\n IN skipcovered boolean DEFAULT TRUE -- Skip any items that would show up completely under the previous items\n) RETURNS jsonb AS $$\nDECLARE\n search searches%ROWTYPE;\n curs refcursor;\n _where text;\n query text;\n iter_record items%ROWTYPE;\n out_records jsonb[] := '{}'::jsonb[];\n exit_flag boolean := FALSE;\n counter int := 1;\n scancounter int := 1;\n remaining_limit int := _scanlimit;\n tilearea float;\n unionedgeom geometry;\n clippedgeom geometry;\n unionedgeom_area float := 0;\n prev_area float := 0;\n excludes text[];\n includes text[];\n\nBEGIN\n -- If skipcovered is true then you will always want to exit when the passed in geometry is full\n IF skipcovered THEN\n exitwhenfull := TRUE;\n END IF;\n\n SELECT * INTO search FROM searches WHERE hash=queryhash;\n\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Search with Query Hash % Not Found', queryhash;\n END IF;\n\n tilearea := st_area(geom);\n _where := format('%s AND st_intersects(geometry, %L::geometry)', search._where, geom);\n\n IF fields IS NOT NULL THEN\n IF fields ? 'fields' THEN\n fields := fields->'fields';\n END IF;\n IF fields ? 'exclude' THEN\n excludes=textarr(fields->'exclude');\n END IF;\n IF fields ? 'include' THEN\n includes=textarr(fields->'include');\n IF array_length(includes, 1)>0 AND NOT 'id' = ANY (includes) THEN\n includes = includes || '{id}';\n END IF;\n END IF;\n END IF;\n RAISE NOTICE 'fields: %, includes: %, excludes: %', fields, includes, excludes;\n\n FOR query IN SELECT * FROM partition_queries(_where, search.orderby) LOOP\n query := format('%s LIMIT %L', query, remaining_limit);\n RAISE NOTICE '%', query;\n curs = create_cursor(query);\n LOOP\n FETCH curs INTO iter_record;\n EXIT WHEN NOT FOUND;\n IF exitwhenfull OR skipcovered THEN -- If we are not using exitwhenfull or skipcovered, we do not need to do expensive geometry operations\n clippedgeom := st_intersection(geom, iter_record.geometry);\n\n IF unionedgeom IS NULL THEN\n unionedgeom := clippedgeom;\n ELSE\n unionedgeom := st_union(unionedgeom, clippedgeom);\n END IF;\n\n unionedgeom_area := st_area(unionedgeom);\n\n IF skipcovered AND prev_area = unionedgeom_area THEN\n scancounter := scancounter + 1;\n CONTINUE;\n END IF;\n\n prev_area := unionedgeom_area;\n\n RAISE NOTICE '% % % %', unionedgeom_area/tilearea, counter, scancounter, ftime();\n END IF;\n\n IF fields IS NOT NULL THEN\n out_records := out_records || filter_jsonb(iter_record.content, includes, excludes);\n ELSE\n out_records := out_records || iter_record.content;\n END IF;\n IF counter >= _limit\n OR scancounter > _scanlimit\n OR ftime() > _timelimit\n OR (exitwhenfull AND unionedgeom_area >= tilearea)\n THEN\n exit_flag := TRUE;\n EXIT;\n END IF;\n counter := counter + 1;\n scancounter := scancounter + 1;\n\n END LOOP;\n EXIT WHEN exit_flag;\n remaining_limit := _scanlimit - scancounter;\n END LOOP;\n\n RETURN jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', array_to_json(out_records)::jsonb\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDROP FUNCTION IF EXISTS geojsonsearch;\nCREATE OR REPLACE FUNCTION geojsonsearch(\n IN geojson jsonb,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_geomfromgeojson(geojson),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS xyzsearch;\nCREATE OR REPLACE FUNCTION xyzsearch(\n IN _x int,\n IN _y int,\n IN _z int,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_transform(tileenvelope(_z, _x, _y), 4326),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\nINSERT INTO pgstac.migrations (version) VALUES ('0.3.4');\n" }, { "path": "src/pgstac/migrations/pgstac.0.3.5-0.3.6.sql", "content": "SET SEARCH_PATH to pgstac, public;\nCREATE INDEX search_wheres_partitions ON pgstac.search_wheres USING gin (partitions);\n\nset check_function_bodies = off;\n\nCREATE OR REPLACE FUNCTION pgstac.validate_constraints()\n RETURNS void\n LANGUAGE plpgsql\nAS $function$\nDECLARE\nq text;\nBEGIN\nFOR q IN\n SELECT FORMAT(\n 'ALTER TABLE %I.%I.%I VALIDATE CONSTRAINT %I;',\n current_database(),\n nsp.nspname,\n cls.relname,\n con.conname\n )\n FROM pg_constraint AS con\n JOIN pg_class AS cls\n ON con.conrelid = cls.oid\n JOIN pg_namespace AS nsp\n ON cls.relnamespace = nsp.oid\n WHERE convalidated IS FALSE\n AND nsp.nspname = 'pgstac'\nLOOP\n EXECUTE q;\nEND LOOP;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.drop_partition_constraints(partition text)\n RETURNS void\n LANGUAGE plpgsql\nAS $function$\nDECLARE\n q text;\n end_datetime_constraint text := concat(partition, '_end_datetime_constraint');\n collections_constraint text := concat(partition, '_collections_constraint');\nBEGIN\n q := format($q$\n ALTER TABLE %I\n DROP CONSTRAINT IF EXISTS %I,\n DROP CONSTRAINT IF EXISTS %I;\n $q$,\n partition,\n end_datetime_constraint,\n collections_constraint\n );\n\n EXECUTE q;\n RETURN;\n\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.items_staging_ignore_insert_triggerfunc()\n RETURNS trigger\n LANGUAGE plpgsql\nAS $function$\nDECLARE\n p record;\n _partitions text[];\nBEGIN\n CREATE TEMP TABLE new_partitions ON COMMIT DROP AS\n SELECT\n items_partition_name(stac_datetime(content)) as partition,\n date_trunc('week', min(stac_datetime(content))) as partition_start\n FROM newdata\n GROUP BY 1;\n\n -- set statslastupdated in cache to be old enough cache always regenerated\n\n SELECT array_agg(partition) INTO _partitions FROM new_partitions;\n UPDATE search_wheres\n SET\n statslastupdated = NULL\n WHERE partitions && _partitions\n ;\n\n FOR p IN SELECT new_partitions.partition, new_partitions.partition_start, new_partitions.partition_start + '1 week'::interval as partition_end FROM new_partitions\n LOOP\n RAISE NOTICE 'Getting partition % ready.', p.partition;\n IF NOT items_partition_exists(p.partition) THEN\n RAISE NOTICE 'Creating partition %.', p.partition;\n PERFORM items_partition_create_worker(p.partition, p.partition_start, p.partition_end);\n END IF;\n PERFORM drop_partition_constraints(p.partition);\n END LOOP;\n\n INSERT INTO items (id, geometry, collection_id, datetime, end_datetime, properties, content)\n SELECT\n content->>'id',\n stac_geom(content),\n content->>'collection',\n stac_datetime(content),\n stac_end_datetime(content),\n properties_idx(content),\n content\n FROM newdata\n ON CONFLICT (datetime, id) DO NOTHING\n ;\n DELETE FROM items_staging;\n\n\n FOR p IN SELECT new_partitions.partition FROM new_partitions\n LOOP\n RAISE NOTICE 'Setting constraints for partition %.', p.partition;\n PERFORM partition_checks(p.partition);\n END LOOP;\n DROP TABLE IF EXISTS new_partitions;\n RETURN NULL;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.items_staging_insert_triggerfunc()\n RETURNS trigger\n LANGUAGE plpgsql\nAS $function$\nDECLARE\n p record;\n _partitions text[];\nBEGIN\n CREATE TEMP TABLE new_partitions ON COMMIT DROP AS\n SELECT\n items_partition_name(stac_datetime(content)) as partition,\n date_trunc('week', min(stac_datetime(content))) as partition_start\n FROM newdata\n GROUP BY 1;\n\n -- set statslastupdated in cache to be old enough cache always regenerated\n\n SELECT array_agg(partition) INTO _partitions FROM new_partitions;\n UPDATE search_wheres\n SET\n statslastupdated = NULL\n WHERE partitions && _partitions\n ;\n\n FOR p IN SELECT new_partitions.partition, new_partitions.partition_start, new_partitions.partition_start + '1 week'::interval as partition_end FROM new_partitions\n LOOP\n RAISE NOTICE 'Getting partition % ready.', p.partition;\n IF NOT items_partition_exists(p.partition) THEN\n RAISE NOTICE 'Creating partition %.', p.partition;\n PERFORM items_partition_create_worker(p.partition, p.partition_start, p.partition_end);\n END IF;\n PERFORM drop_partition_constraints(p.partition);\n END LOOP;\n INSERT INTO items (id, geometry, collection_id, datetime, end_datetime, properties, content)\n SELECT\n content->>'id',\n stac_geom(content),\n content->>'collection',\n stac_datetime(content),\n stac_end_datetime(content),\n properties_idx(content),\n content\n FROM newdata\n ;\n DELETE FROM items_staging;\n\n\n FOR p IN SELECT new_partitions.partition FROM new_partitions\n LOOP\n RAISE NOTICE 'Setting constraints for partition %.', p.partition;\n PERFORM partition_checks(p.partition);\n END LOOP;\n DROP TABLE IF EXISTS new_partitions;\n RETURN NULL;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.items_staging_upsert_insert_triggerfunc()\n RETURNS trigger\n LANGUAGE plpgsql\nAS $function$\nDECLARE\n p record;\n _partitions text[];\nBEGIN\n CREATE TEMP TABLE new_partitions ON COMMIT DROP AS\n SELECT\n items_partition_name(stac_datetime(content)) as partition,\n date_trunc('week', min(stac_datetime(content))) as partition_start\n FROM newdata\n GROUP BY 1;\n\n -- set statslastupdated in cache to be old enough cache always regenerated\n\n SELECT array_agg(partition) INTO _partitions FROM new_partitions;\n UPDATE search_wheres\n SET\n statslastupdated = NULL\n WHERE partitions && _partitions\n ;\n\n FOR p IN SELECT new_partitions.partition, new_partitions.partition_start, new_partitions.partition_start + '1 week'::interval as partition_end FROM new_partitions\n LOOP\n RAISE NOTICE 'Getting partition % ready.', p.partition;\n IF NOT items_partition_exists(p.partition) THEN\n RAISE NOTICE 'Creating partition %.', p.partition;\n PERFORM items_partition_create_worker(p.partition, p.partition_start, p.partition_end);\n END IF;\n PERFORM drop_partition_constraints(p.partition);\n END LOOP;\n\n INSERT INTO items (id, geometry, collection_id, datetime, end_datetime, properties, content)\n SELECT\n content->>'id',\n stac_geom(content),\n content->>'collection',\n stac_datetime(content),\n stac_end_datetime(content),\n properties_idx(content),\n content\n FROM newdata\n ON CONFLICT (datetime, id) DO UPDATE SET\n content = EXCLUDED.content\n WHERE items.content IS DISTINCT FROM EXCLUDED.content\n ;\n DELETE FROM items_staging;\n\n\n FOR p IN SELECT new_partitions.partition FROM new_partitions\n LOOP\n RAISE NOTICE 'Setting constraints for partition %.', p.partition;\n PERFORM partition_checks(p.partition);\n END LOOP;\n DROP TABLE IF EXISTS new_partitions;\n RETURN NULL;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.partition_checks(partition text, OUT min_datetime timestamp with time zone, OUT max_datetime timestamp with time zone, OUT min_end_datetime timestamp with time zone, OUT max_end_datetime timestamp with time zone, OUT collections text[], OUT cnt bigint)\n RETURNS record\n LANGUAGE plpgsql\nAS $function$\nDECLARE\nq text;\nend_datetime_constraint text := concat(partition, '_end_datetime_constraint');\ncollections_constraint text := concat(partition, '_collections_constraint');\nBEGIN\nRAISE NOTICE 'CREATING CONSTRAINTS FOR %', partition;\nq := format($q$\n SELECT\n min(datetime),\n max(datetime),\n min(end_datetime),\n max(end_datetime),\n array_agg(DISTINCT collection_id),\n count(*)\n FROM %I;\n $q$,\n partition\n);\nEXECUTE q INTO min_datetime, max_datetime, min_end_datetime, max_end_datetime, collections, cnt;\nRAISE NOTICE '% % % % % % %', min_datetime, max_datetime, min_end_datetime, max_end_datetime, collections, cnt, ftime();\nIF cnt IS NULL or cnt = 0 THEN\n RAISE NOTICE 'Partition % is empty, removing...', partition;\n q := format($q$\n DROP TABLE IF EXISTS %I;\n $q$, partition\n );\n EXECUTE q;\n RETURN;\nEND IF;\nRAISE NOTICE 'Running Constraint DDL %', ftime();\nq := format($q$\n ALTER TABLE %I\n DROP CONSTRAINT IF EXISTS %I,\n ADD CONSTRAINT %I\n check((end_datetime >= %L) AND (end_datetime <= %L)) NOT VALID,\n DROP CONSTRAINT IF EXISTS %I,\n ADD CONSTRAINT %I\n check((collection_id = ANY(%L))) NOT VALID;\n $q$,\n partition,\n end_datetime_constraint,\n end_datetime_constraint,\n min_end_datetime,\n max_end_datetime,\n collections_constraint,\n collections_constraint,\n collections,\n partition\n);\nRAISE NOTICE 'q: %', q;\n\nEXECUTE q;\nRAISE NOTICE 'Returning %', ftime();\nRETURN;\n\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.search(_search jsonb DEFAULT '{}'::jsonb)\n RETURNS jsonb\n LANGUAGE plpgsql\n SET jit TO 'off'\nAS $function$\nDECLARE\n searches searches%ROWTYPE;\n _where text;\n token_where text;\n full_where text;\n orderby text;\n query text;\n token_type text := substr(_search->>'token',1,4);\n _limit int := coalesce((_search->>'limit')::int, 10);\n curs refcursor;\n cntr int := 0;\n iter_record items%ROWTYPE;\n first_record items%ROWTYPE;\n last_record items%ROWTYPE;\n out_records jsonb := '[]'::jsonb;\n prev_query text;\n next text;\n prev_id text;\n has_next boolean := false;\n has_prev boolean := false;\n prev text;\n total_count bigint;\n context jsonb;\n collection jsonb;\n includes text[];\n excludes text[];\n exit_flag boolean := FALSE;\n batches int := 0;\n timer timestamptz := clock_timestamp();\n pstart timestamptz;\n pend timestamptz;\n pcurs refcursor;\n search_where search_wheres%ROWTYPE;\nBEGIN\nsearches := search_query(_search);\n_where := searches._where;\norderby := searches.orderby;\nsearch_where := where_stats(_where);\ntotal_count := coalesce(search_where.total_count, search_where.estimated_count);\n\n\nIF token_type='prev' THEN\n token_where := get_token_filter(_search, null::jsonb);\n orderby := sort_sqlorderby(_search, TRUE);\nEND IF;\nIF token_type='next' THEN\n token_where := get_token_filter(_search, null::jsonb);\nEND IF;\n\nfull_where := concat_ws(' AND ', _where, token_where);\nRAISE NOTICE 'FULL QUERY % %', full_where, clock_timestamp()-timer;\ntimer := clock_timestamp();\n\nCREATE TEMP TABLE results (content jsonb) ON COMMIT DROP;\n\n\nFOR query IN SELECT partition_queries(full_where, orderby, search_where.partitions) LOOP\n timer := clock_timestamp();\n query := format('%s LIMIT %s', query, _limit + 1);\n RAISE NOTICE 'Partition Query: %', query;\n batches := batches + 1;\n -- curs = create_cursor(query);\n OPEN curs FOR EXECUTE query;\n LOOP\n FETCH curs into iter_record;\n EXIT WHEN NOT FOUND;\n cntr := cntr + 1;\n last_record := iter_record;\n IF cntr = 1 THEN\n first_record := last_record;\n END IF;\n IF cntr <= _limit THEN\n INSERT INTO results (content) VALUES (last_record.content);\n -- out_records := out_records || last_record.content;\n\n ELSIF cntr > _limit THEN\n has_next := true;\n exit_flag := true;\n EXIT;\n END IF;\n END LOOP;\n CLOSE curs;\n RAISE NOTICE 'Query took %. Total Time %', clock_timestamp()-timer, ftime();\n timer := clock_timestamp();\n EXIT WHEN exit_flag;\nEND LOOP;\nRAISE NOTICE 'Scanned through % partitions.', batches;\n\nSELECT jsonb_agg(content) INTO out_records FROM results;\n\nDROP TABLE results;\n\n\n-- Flip things around if this was the result of a prev token query\nIF token_type='prev' THEN\n out_records := flip_jsonb_array(out_records);\n first_record := last_record;\nEND IF;\n\n-- If this query has a token, see if there is data before the first record\nIF _search ? 'token' THEN\n prev_query := format(\n 'SELECT 1 FROM items WHERE %s LIMIT 1',\n concat_ws(\n ' AND ',\n _where,\n trim(get_token_filter(_search, to_jsonb(first_record)))\n )\n );\n RAISE NOTICE 'Query to get previous record: % --- %', prev_query, first_record;\n EXECUTE prev_query INTO has_prev;\n IF FOUND and has_prev IS NOT NULL THEN\n RAISE NOTICE 'Query results from prev query: %', has_prev;\n has_prev := TRUE;\n END IF;\nEND IF;\nhas_prev := COALESCE(has_prev, FALSE);\n\nRAISE NOTICE 'token_type: %, has_next: %, has_prev: %', token_type, has_next, has_prev;\nIF has_prev THEN\n prev := out_records->0->>'id';\nEND IF;\nIF has_next OR token_type='prev' THEN\n next := out_records->-1->>'id';\nEND IF;\n\n\n-- include/exclude any fields following fields extension\nIF _search ? 'fields' THEN\n IF _search->'fields' ? 'exclude' THEN\n excludes=textarr(_search->'fields'->'exclude');\n END IF;\n IF _search->'fields' ? 'include' THEN\n includes=textarr(_search->'fields'->'include');\n IF array_length(includes, 1)>0 AND NOT 'id' = ANY (includes) THEN\n includes = includes || '{id}';\n END IF;\n END IF;\n SELECT jsonb_agg(filter_jsonb(row, includes, excludes)) INTO out_records FROM jsonb_array_elements(out_records) row;\nEND IF;\n\nIF context() != 'off' THEN\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'matched', total_count,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\nELSE\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\nEND IF;\n\ncollection := jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb),\n 'next', next,\n 'prev', prev,\n 'context', context\n);\n\nRETURN collection;\nEND;\n$function$\n;\n\n\n\nSELECT set_version('0.3.6');\n" }, { "path": "src/pgstac/migrations/pgstac.0.3.5.sql", "content": "CREATE EXTENSION IF NOT EXISTS postgis;\nCREATE SCHEMA IF NOT EXISTS pgstac;\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE TABLE migrations (\n version text PRIMARY KEY,\n datetime timestamptz DEFAULT clock_timestamp() NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION get_version() RETURNS text AS $$\n SELECT version FROM migrations ORDER BY datetime DESC, version DESC LIMIT 1;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION set_version(text) RETURNS text AS $$\n INSERT INTO migrations (version) VALUES ($1)\n ON CONFLICT DO NOTHING\n RETURNING version;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\n\nCREATE OR REPLACE FUNCTION get_setting(IN setting text, INOUT _default anynonarray = null::text ) AS $$\nDECLARE\n_type text;\nBEGIN\n SELECT pg_typeof(_default) INTO _type;\n IF _type = 'unknown' THEN _type='text'; END IF;\n EXECUTE format($q$\n SELECT COALESCE(\n CAST(current_setting($1,TRUE) AS %s),\n $2\n )\n $q$, _type)\n INTO _default\n USING setting, _default\n ;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION context() RETURNS text AS $$\n SELECT get_setting('pgstac.context','off'::text);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context_estimated_count() RETURNS int AS $$\n SELECT get_setting('pgstac.context_estimated_count', 100000::int);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context_estimated_cost() RETURNS float AS $$\n SELECT get_setting('pgstac.context_estimated_cost', 1000000::float);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context_stats_ttl() RETURNS interval AS $$\n SELECT get_setting('pgstac.context_stats_ttl', '1 day'::interval);\n$$ LANGUAGE SQL;\n\n\nCREATE OR REPLACE FUNCTION notice(VARIADIC text[]) RETURNS boolean AS $$\nDECLARE\ndebug boolean := current_setting('pgstac.debug', true);\nBEGIN\n IF debug THEN\n RAISE NOTICE 'NOTICE FROM FUNC: % >>>>> %', concat_ws(' | ', $1), clock_timestamp();\n RETURN TRUE;\n END IF;\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION empty_arr(ANYARRAY) RETURNS BOOLEAN AS $$\nSELECT CASE\n WHEN $1 IS NULL THEN TRUE\n WHEN cardinality($1)<1 THEN TRUE\nELSE FALSE\nEND;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_map_ident(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_ident(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_map_literal(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_literal(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION estimated_count(_where text) RETURNS bigint AS $$\nDECLARE\nrec record;\nrows bigint;\nBEGIN\n FOR rec in EXECUTE format(\n $q$\n EXPLAIN SELECT 1 FROM items WHERE %s\n $q$,\n _where)\n LOOP\n rows := substring(rec.\"QUERY PLAN\" FROM ' rows=([[:digit:]]+)');\n EXIT WHEN rows IS NOT NULL;\n END LOOP;\n\n RETURN rows;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION array_reverse(anyarray) RETURNS anyarray AS $$\nSELECT ARRAY(\n SELECT $1[i]\n FROM generate_subscripts($1,1) AS s(i)\n ORDER BY i DESC\n);\n$$ LANGUAGE 'sql' STRICT IMMUTABLE;\n/* converts a jsonb text array to a pg text[] array */\nCREATE OR REPLACE FUNCTION textarr(_js jsonb)\n RETURNS text[] AS $$\n SELECT\n CASE jsonb_typeof(_js)\n WHEN 'array' THEN ARRAY(SELECT jsonb_array_elements_text(_js))\n ELSE ARRAY[_js->>0]\n END\n;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION jsonb_paths (IN jdata jsonb, OUT path text[], OUT value jsonb) RETURNS\nSETOF RECORD AS $$\nwith recursive extract_all as\n(\n select\n ARRAY[key]::text[] as path,\n value\n FROM jsonb_each(jdata)\nunion all\n select\n path || coalesce(obj_key, (arr_key- 1)::text),\n coalesce(obj_value, arr_value)\n from extract_all\n left join lateral\n jsonb_each(case jsonb_typeof(value) when 'object' then value end)\n as o(obj_key, obj_value)\n on jsonb_typeof(value) = 'object'\n left join lateral\n jsonb_array_elements(case jsonb_typeof(value) when 'array' then value end)\n with ordinality as a(arr_value, arr_key)\n on jsonb_typeof(value) = 'array'\n where obj_key is not null or arr_key is not null\n)\nselect *\nfrom extract_all;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION jsonb_obj_paths (IN jdata jsonb, OUT path text[], OUT value jsonb) RETURNS\nSETOF RECORD AS $$\nwith recursive extract_all as\n(\n select\n ARRAY[key]::text[] as path,\n value\n FROM jsonb_each(jdata)\nunion all\n select\n path || obj_key,\n obj_value\n from extract_all\n left join lateral\n jsonb_each(case jsonb_typeof(value) when 'object' then value end)\n as o(obj_key, obj_value)\n on jsonb_typeof(value) = 'object'\n where obj_key is not null\n)\nselect *\nfrom extract_all;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION jsonb_val_paths (IN jdata jsonb, OUT path text[], OUT value jsonb) RETURNS\nSETOF RECORD AS $$\nSELECT * FROM jsonb_obj_paths(jdata) WHERE jsonb_typeof(value) not in ('object','array');\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION path_includes(IN path text[], IN includes text[]) RETURNS BOOLEAN AS $$\nWITH t AS (SELECT unnest(includes) i)\nSELECT EXISTS (\n SELECT 1 FROM t WHERE path @> string_to_array(trim(i), '.')\n);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION path_excludes(IN path text[], IN excludes text[]) RETURNS BOOLEAN AS $$\nWITH t AS (SELECT unnest(excludes) e)\nSELECT NOT EXISTS (\n SELECT 1 FROM t WHERE path @> string_to_array(trim(e), '.')\n);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION jsonb_obj_paths_filtered (\n IN jdata jsonb,\n IN includes text[] DEFAULT ARRAY[]::text[],\n IN excludes text[] DEFAULT ARRAY[]::text[],\n OUT path text[],\n OUT value jsonb\n) RETURNS\nSETOF RECORD AS $$\nSELECT path, value\nFROM jsonb_obj_paths(jdata)\nWHERE\n CASE WHEN cardinality(includes) > 0 THEN path_includes(path, includes) ELSE TRUE END\n AND\n path_excludes(path, excludes)\n\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION filter_jsonb(\n IN jdata jsonb,\n IN includes text[] DEFAULT ARRAY[]::text[],\n IN excludes text[] DEFAULT ARRAY[]::text[]\n) RETURNS jsonb AS $$\nDECLARE\nrec RECORD;\noutj jsonb := '{}'::jsonb;\ncreated_paths text[] := '{}'::text[];\nBEGIN\n\nIF empty_arr(includes) AND empty_arr(excludes) THEN\nRAISE NOTICE 'no filter';\n RETURN jdata;\nEND IF;\nFOR rec in\nSELECT * FROM jsonb_obj_paths_filtered(jdata, includes, excludes)\nWHERE jsonb_typeof(value) != 'object'\nLOOP\n IF array_length(rec.path,1)>1 THEN\n FOR i IN 1..(array_length(rec.path,1)-1) LOOP\n IF NOT array_to_string(rec.path[1:i],'.') = ANY (created_paths) THEN\n outj := jsonb_set(outj, rec.path[1:i],'{}', true);\n created_paths := created_paths || array_to_string(rec.path[1:i],'.');\n END IF;\n END LOOP;\n END IF;\n outj := jsonb_set(outj, rec.path, rec.value, true);\n created_paths := created_paths || array_to_string(rec.path,'.');\nEND LOOP;\nRETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION flip_jsonb_array(j jsonb) RETURNS jsonb AS $$\nSELECT jsonb_agg(value) FROM (SELECT value FROM jsonb_array_elements(j) WITH ORDINALITY ORDER BY ordinality DESC) as t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n/* Functions to create an iterable of cursors over partitions. */\nCREATE OR REPLACE FUNCTION create_cursor(q text) RETURNS refcursor AS $$\nDECLARE\n curs refcursor;\nBEGIN\n OPEN curs FOR EXECUTE q;\n RETURN curs;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDROP FUNCTION IF EXISTS partition_queries;\nCREATE OR REPLACE FUNCTION partition_queries(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN partitions text[] DEFAULT '{items}'\n) RETURNS SETOF text AS $$\nDECLARE\n partition_query text;\n query text;\n p text;\n cursors refcursor;\n dstart timestamptz;\n dend timestamptz;\n step interval := '10 weeks'::interval;\nBEGIN\n\nIF _orderby ILIKE 'datetime d%' THEN\n partitions := partitions;\nELSIF _orderby ILIKE 'datetime a%' THEN\n partitions := array_reverse(partitions);\nELSE\n query := format($q$\n SELECT * FROM items\n WHERE %s\n ORDER BY %s\n $q$, _where, _orderby\n );\n\n RETURN NEXT query;\n RETURN;\nEND IF;\nRAISE NOTICE 'PARTITIONS ---> %',partitions;\nIF cardinality(partitions) > 0 THEN\n FOREACH p IN ARRAY partitions\n --EXECUTE partition_query\n LOOP\n query := format($q$\n SELECT * FROM %I\n WHERE %s\n ORDER BY %s\n $q$,\n p,\n _where,\n _orderby\n );\n RETURN NEXT query;\n END LOOP;\nEND IF;\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION partition_cursor(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC'\n) RETURNS SETOF refcursor AS $$\nDECLARE\n partition_query text;\n query text;\n p record;\n cursors refcursor;\nBEGIN\nFOR query IN SELECT * FROM partition_queries(_where, _orderby) LOOP\n RETURN NEXT create_cursor(query);\nEND LOOP;\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION partition_count(\n IN _where text DEFAULT 'TRUE'\n) RETURNS bigint AS $$\nDECLARE\n partition_query text;\n query text;\n p record;\n subtotal bigint;\n total bigint := 0;\nBEGIN\npartition_query := format($q$\n SELECT partition, tstzrange\n FROM items_partitions\n ORDER BY tstzrange DESC;\n$q$);\nRAISE NOTICE 'Partition Query: %', partition_query;\nFOR p IN\n EXECUTE partition_query\nLOOP\n query := format($q$\n SELECT count(*) FROM items\n WHERE datetime BETWEEN %L AND %L AND %s\n $q$, lower(p.tstzrange), upper(p.tstzrange), _where\n );\n RAISE NOTICE 'Query %', query;\n RAISE NOTICE 'Partition %, Count %, Total %',p.partition, subtotal, total;\n EXECUTE query INTO subtotal;\n total := subtotal + total;\nEND LOOP;\nRETURN total;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION drop_partition_constraints(IN partition text) RETURNS VOID AS $$\nDECLARE\n q text;\n end_datetime_constraint text := concat(partition, '_end_datetime_constraint');\n collections_constraint text := concat(partition, '_collections_constraint');\nBEGIN\n q := format($q$\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n $q$,\n partition,\n end_datetime_constraint,\n partition,\n collections_constraint\n );\n\n EXECUTE q;\n RETURN;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDROP FUNCTION IF EXISTS partition_checks;\nCREATE OR REPLACE FUNCTION partition_checks(\n IN partition text,\n OUT min_datetime timestamptz,\n OUT max_datetime timestamptz,\n OUT min_end_datetime timestamptz,\n OUT max_end_datetime timestamptz,\n OUT collections text[],\n OUT cnt bigint\n) RETURNS RECORD AS $$\nDECLARE\nq text;\nend_datetime_constraint text := concat(partition, '_end_datetime_constraint');\ncollections_constraint text := concat(partition, '_collections_constraint');\nBEGIN\nRAISE NOTICE 'CREATING CONSTRAINTS FOR %', partition;\nq := format($q$\n SELECT\n min(datetime),\n max(datetime),\n min(end_datetime),\n max(end_datetime),\n array_agg(DISTINCT collection_id),\n count(*)\n FROM %I;\n $q$,\n partition\n);\nEXECUTE q INTO min_datetime, max_datetime, min_end_datetime, max_end_datetime, collections, cnt;\nRAISE NOTICE '% % % % % %', min_datetime, max_datetime, min_end_datetime, max_end_datetime, collections, cnt;\nIF cnt IS NULL or cnt = 0 THEN\n RAISE NOTICE 'Partition % is empty, removing...', partition;\n q := format($q$\n DROP TABLE IF EXISTS %I;\n $q$, partition\n );\n EXECUTE q;\n RETURN;\nEND IF;\nq := format($q$\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n ALTER TABLE %I ADD CONSTRAINT %I\n check((end_datetime >= %L) AND (end_datetime <= %L));\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n ALTER TABLE %I ADD CONSTRAINT %I\n check((collection_id = ANY(%L)));\n ANALYZE %I;\n $q$,\n partition,\n end_datetime_constraint,\n partition,\n end_datetime_constraint,\n min_end_datetime,\n max_end_datetime,\n partition,\n collections_constraint,\n partition,\n collections_constraint,\n collections,\n partition\n);\n\nEXECUTE q;\nRETURN;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n/* looks for a geometry in a stac item first from geometry and falling back to bbox */\nCREATE OR REPLACE FUNCTION stac_geom(value jsonb) RETURNS geometry AS $$\nSELECT\n CASE\n WHEN value->>'geometry' IS NOT NULL THEN\n ST_GeomFromGeoJSON(value->>'geometry')\n WHEN value->>'bbox' IS NOT NULL THEN\n ST_MakeEnvelope(\n (value->'bbox'->>0)::float,\n (value->'bbox'->>1)::float,\n (value->'bbox'->>2)::float,\n (value->'bbox'->>3)::float,\n 4326\n )\n ELSE NULL\n END as geometry\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION stac_datetime(value jsonb) RETURNS timestamptz AS $$\nSELECT COALESCE(\n (value->'properties'->>'datetime')::timestamptz,\n (value->'properties'->>'start_datetime')::timestamptz\n);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_end_datetime(value jsonb) RETURNS timestamptz AS $$\nSELECT COALESCE(\n (value->'properties'->>'datetime')::timestamptz,\n (value->'properties'->>'end_datetime')::timestamptz\n);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\n\nCREATE OR REPLACE FUNCTION stac_daterange(value jsonb) RETURNS tstzrange AS $$\nSELECT tstzrange(stac_datetime(value),stac_end_datetime(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE TABLE IF NOT EXISTS collections (\n id VARCHAR GENERATED ALWAYS AS (content->>'id') STORED PRIMARY KEY,\n content JSONB\n);\n\nCREATE OR REPLACE FUNCTION create_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection(data jsonb) RETURNS VOID AS $$\nDECLARE\nout collections%ROWTYPE;\nBEGIN\n UPDATE collections SET content=data WHERE id = data->>'id' RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ON CONFLICT (id) DO\n UPDATE\n SET content=EXCLUDED.content\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION delete_collection(_id text) RETURNS VOID AS $$\nDECLARE\nout collections%ROWTYPE;\nBEGIN\n DELETE FROM collections WHERE id = _id RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION get_collection(id text) RETURNS jsonb AS $$\nSELECT content FROM collections\nWHERE id=$1\n;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION all_collections() RETURNS jsonb AS $$\nSELECT jsonb_agg(content) FROM collections;\n;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE TABLE items (\n id text NOT NULL,\n geometry geometry NOT NULL,\n collection_id text NOT NULL,\n datetime timestamptz NOT NULL,\n end_datetime timestamptz NOT NULL,\n properties jsonb NOT NULL,\n content JSONB NOT NULL\n)\nPARTITION BY RANGE (datetime)\n;\n\nCREATE OR REPLACE FUNCTION properties_idx (IN content jsonb) RETURNS jsonb AS $$\n with recursive extract_all as\n (\n select\n ARRAY[key]::text[] as path,\n ARRAY[key]::text[] as fullpath,\n value\n FROM jsonb_each(content->'properties')\n union all\n select\n CASE WHEN obj_key IS NOT NULL THEN path || obj_key ELSE path END,\n path || coalesce(obj_key, (arr_key- 1)::text),\n coalesce(obj_value, arr_value)\n from extract_all\n left join lateral\n jsonb_each(case jsonb_typeof(value) when 'object' then value end)\n as o(obj_key, obj_value)\n on jsonb_typeof(value) = 'object'\n left join lateral\n jsonb_array_elements(case jsonb_typeof(value) when 'array' then value end)\n with ordinality as a(arr_value, arr_key)\n on jsonb_typeof(value) = 'array'\n where obj_key is not null or arr_key is not null\n )\n , paths AS (\n select\n array_to_string(path, '.') as path,\n value\n FROM extract_all\n WHERE\n jsonb_typeof(value) NOT IN ('array','object')\n ), grouped AS (\n SELECT path, jsonb_agg(distinct value) vals FROM paths group by path\n ) SELECT coalesce(jsonb_object_agg(path, CASE WHEN jsonb_array_length(vals)=1 THEN vals->0 ELSE vals END) - '{datetime}'::text[], '{}'::jsonb) FROM grouped\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET JIT TO OFF;\n\nCREATE INDEX \"datetime_idx\" ON items (datetime);\nCREATE INDEX \"end_datetime_idx\" ON items (end_datetime);\nCREATE INDEX \"properties_idx\" ON items USING GIN (properties jsonb_path_ops);\nCREATE INDEX \"collection_idx\" ON items (collection_id);\nCREATE INDEX \"geometry_idx\" ON items USING GIST (geometry);\nCREATE UNIQUE INDEX \"items_id_datetime_idx\" ON items (datetime, id);\n\nALTER TABLE items ADD CONSTRAINT items_collections_fk FOREIGN KEY (collection_id) REFERENCES collections(id) ON DELETE CASCADE DEFERRABLE;\n\nCREATE OR REPLACE FUNCTION analyze_empty_partitions() RETURNS VOID AS $$\nDECLARE\n p text;\nBEGIN\n FOR p IN SELECT partition FROM all_items_partitions WHERE est_cnt = 0 LOOP\n EXECUTE format('ANALYZE %I;', p);\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION items_partition_name(timestamptz) RETURNS text AS $$\n SELECT to_char($1, '\"items_p\"IYYY\"w\"IW');\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION items_partition_exists(text) RETURNS boolean AS $$\n SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_class WHERE relname=$1);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION items_partition_exists(timestamptz) RETURNS boolean AS $$\n SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_class WHERE relname=items_partition_name($1));\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION items_partition_create_worker(partition text, partition_start timestamptz, partition_end timestamptz) RETURNS VOID AS $$\nDECLARE\n err_context text;\nBEGIN\n EXECUTE format(\n $f$\n CREATE TABLE IF NOT EXISTS %1$I PARTITION OF items\n FOR VALUES FROM (%2$L) TO (%3$L);\n CREATE UNIQUE INDEX IF NOT EXISTS %4$I ON %1$I (id);\n $f$,\n partition,\n partition_start,\n partition_end,\n concat(partition, '_id_pk')\n );\nEXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', partition;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH to pgstac, public;\n\nCREATE OR REPLACE FUNCTION items_partition_create(ts timestamptz) RETURNS text AS $$\nDECLARE\n partition text := items_partition_name(ts);\n partition_start timestamptz;\n partition_end timestamptz;\nBEGIN\n IF items_partition_exists(partition) THEN\n RETURN partition;\n END IF;\n partition_start := date_trunc('week', ts);\n partition_end := partition_start + '1 week'::interval;\n PERFORM items_partition_create_worker(partition, partition_start, partition_end);\n RAISE NOTICE 'partition: %', partition;\n RETURN partition;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION items_partition_create(st timestamptz, et timestamptz) RETURNS SETOF text AS $$\nWITH t AS (\n SELECT\n generate_series(\n date_trunc('week',st),\n date_trunc('week', et),\n '1 week'::interval\n ) w\n)\nSELECT items_partition_create(w) FROM t;\n$$ LANGUAGE SQL;\n\n\nCREATE UNLOGGED TABLE items_staging (\n content JSONB NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION items_staging_insert_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n mindate timestamptz;\n maxdate timestamptz;\n partition text;\n p record;\nBEGIN\n CREATE TEMP TABLE new_partitions ON COMMIT DROP AS\n SELECT\n items_partition_name(stac_datetime(content)) as partition,\n date_trunc('week', min(stac_datetime(content))) as partition_start\n FROM newdata\n GROUP BY 1;\n\n FOR p IN SELECT new_partitions.partition, new_partitions.partition_start, new_partitions.partition_start + '1 week'::interval as partition_end FROM new_partitions\n LOOP\n RAISE NOTICE 'Getting partition % ready.', p.partition;\n IF NOT items_partition_exists(p.partition) THEN\n RAISE NOTICE 'Creating partition %.', p.partition;\n PERFORM items_partition_create_worker(p.partition, p.partition_start, p.partition_end);\n END IF;\n PERFORM drop_partition_constraints(p.partition);\n END LOOP;\n\n INSERT INTO items (id, geometry, collection_id, datetime, end_datetime, properties, content)\n SELECT\n content->>'id',\n stac_geom(content),\n content->>'collection',\n stac_datetime(content),\n stac_end_datetime(content),\n properties_idx(content),\n content\n FROM newdata\n ;\n DELETE FROM items_staging;\n\n\n FOR p IN SELECT new_partitions.partition FROM new_partitions\n LOOP\n RAISE NOTICE 'Setting constraints for partition %.', p.partition;\n PERFORM partition_checks(p.partition);\n END LOOP;\n DROP TABLE IF EXISTS new_partitions;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER items_staging_insert_trigger AFTER INSERT ON items_staging REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_insert_triggerfunc();\n\n\nCREATE UNLOGGED TABLE items_staging_ignore (\n content JSONB NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION items_staging_ignore_insert_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p record;\nBEGIN\n CREATE TEMP TABLE new_partitions ON COMMIT DROP AS\n SELECT\n items_partition_name(stac_datetime(content)) as partition,\n date_trunc('week', min(stac_datetime(content))) as partition_start\n FROM newdata\n GROUP BY 1;\n\n FOR p IN SELECT new_partitions.partition, new_partitions.partition_start, new_partitions.partition_start + '1 week'::interval as partition_end FROM new_partitions\n LOOP\n RAISE NOTICE 'Getting partition % ready.', p.partition;\n IF NOT items_partition_exists(p.partition) THEN\n RAISE NOTICE 'Creating partition %.', p.partition;\n PERFORM items_partition_create_worker(p.partition, p.partition_start, p.partition_end);\n END IF;\n PERFORM drop_partition_constraints(p.partition);\n END LOOP;\n\n INSERT INTO items (id, geometry, collection_id, datetime, end_datetime, properties, content)\n SELECT\n content->>'id',\n stac_geom(content),\n content->>'collection',\n stac_datetime(content),\n stac_end_datetime(content),\n properties_idx(content),\n content\n FROM newdata\n ON CONFLICT (datetime, id) DO NOTHING\n ;\n DELETE FROM items_staging;\n\n\n FOR p IN SELECT new_partitions.partition FROM new_partitions\n LOOP\n RAISE NOTICE 'Setting constraints for partition %.', p.partition;\n PERFORM partition_checks(p.partition);\n END LOOP;\n DROP TABLE IF EXISTS new_partitions;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER items_staging_ignore_insert_trigger AFTER INSERT ON items_staging_ignore REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_ignore_insert_triggerfunc();\n\nCREATE UNLOGGED TABLE items_staging_upsert (\n content JSONB NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION items_staging_upsert_insert_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p record;\nBEGIN\n CREATE TEMP TABLE new_partitions ON COMMIT DROP AS\n SELECT\n items_partition_name(stac_datetime(content)) as partition,\n date_trunc('week', min(stac_datetime(content))) as partition_start\n FROM newdata\n GROUP BY 1;\n\n FOR p IN SELECT new_partitions.partition, new_partitions.partition_start, new_partitions.partition_start + '1 week'::interval as partition_end FROM new_partitions\n LOOP\n RAISE NOTICE 'Getting partition % ready.', p.partition;\n IF NOT items_partition_exists(p.partition) THEN\n RAISE NOTICE 'Creating partition %.', p.partition;\n PERFORM items_partition_create_worker(p.partition, p.partition_start, p.partition_end);\n END IF;\n PERFORM drop_partition_constraints(p.partition);\n END LOOP;\n\n INSERT INTO items (id, geometry, collection_id, datetime, end_datetime, properties, content)\n SELECT\n content->>'id',\n stac_geom(content),\n content->>'collection',\n stac_datetime(content),\n stac_end_datetime(content),\n properties_idx(content),\n content\n FROM newdata\n ON CONFLICT (datetime, id) DO UPDATE SET\n content = EXCLUDED.content\n WHERE items.content IS DISTINCT FROM EXCLUDED.content\n ;\n DELETE FROM items_staging;\n\n\n FOR p IN SELECT new_partitions.partition FROM new_partitions\n LOOP\n RAISE NOTICE 'Setting constraints for partition %.', p.partition;\n PERFORM partition_checks(p.partition);\n END LOOP;\n DROP TABLE IF EXISTS new_partitions;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER items_staging_upsert_insert_trigger AFTER INSERT ON items_staging_upsert REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_upsert_insert_triggerfunc();\n\nCREATE OR REPLACE FUNCTION items_update_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\nBEGIN\n NEW.id := NEW.content->>'id';\n NEW.datetime := stac_datetime(NEW.content);\n NEW.end_datetime := stac_end_datetime(NEW.content);\n NEW.collection_id := NEW.content->>'collection';\n NEW.geometry := stac_geom(NEW.content);\n NEW.properties := properties_idx(NEW.content);\n IF TG_OP = 'UPDATE' AND NEW IS NOT DISTINCT FROM OLD THEN\n RETURN NULL;\n END IF;\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER items_update_trigger BEFORE UPDATE ON items\n FOR EACH ROW EXECUTE PROCEDURE items_update_triggerfunc();\n\n/*\nView to get a table of available items partitions\nwith date ranges\n*/\nDROP VIEW IF EXISTS all_items_partitions CASCADE;\nCREATE VIEW all_items_partitions AS\nWITH base AS\n(SELECT\n c.oid::pg_catalog.regclass::text as partition,\n pg_catalog.pg_get_expr(c.relpartbound, c.oid) as _constraint,\n regexp_matches(\n pg_catalog.pg_get_expr(c.relpartbound, c.oid),\n E'\\\\(''\\([0-9 :+-]*\\)''\\\\).*\\\\(''\\([0-9 :+-]*\\)''\\\\)'\n ) as t,\n reltuples::bigint as est_cnt\nFROM pg_catalog.pg_class c, pg_catalog.pg_inherits i\nWHERE c.oid = i.inhrelid AND i.inhparent = 'items'::regclass)\nSELECT partition, tstzrange(\n t[1]::timestamptz,\n t[2]::timestamptz\n), t[1]::timestamptz as pstart,\n t[2]::timestamptz as pend, est_cnt\nFROM base\nORDER BY 2 desc;\n\nCREATE OR REPLACE VIEW items_partitions AS\nSELECT * FROM all_items_partitions WHERE est_cnt>0;\n\nCREATE OR REPLACE FUNCTION get_item(_id text) RETURNS jsonb AS $$\n SELECT content FROM items WHERE id=_id;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION delete_item(_id text) RETURNS VOID AS $$\nDECLARE\nout items%ROWTYPE;\nBEGIN\n DELETE FROM items WHERE id = _id RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION create_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION update_item(data jsonb) RETURNS VOID AS $$\nDECLARE\n out items%ROWTYPE;\nBEGIN\n UPDATE items SET content=data WHERE id = data->>'id' RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION create_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION collection_bbox(id text) RETURNS jsonb AS $$\nSELECT (replace(replace(replace(st_extent(geometry)::text,'BOX(','[['),')',']]'),' ',','))::jsonb\nFROM items WHERE collection_id=$1;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION collection_temporal_extent(id text) RETURNS jsonb AS $$\nSELECT to_jsonb(array[array[min(datetime)::text, max(datetime)::text]])\nFROM items WHERE collection_id=$1;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection_extents() RETURNS VOID AS $$\nUPDATE collections SET\n content = content ||\n jsonb_build_object(\n 'extent', jsonb_build_object(\n 'spatial', jsonb_build_object(\n 'bbox', collection_bbox(collections.id)\n ),\n 'temporal', jsonb_build_object(\n 'interval', collection_temporal_extent(collections.id)\n )\n )\n )\n;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION items_path(\n IN dotpath text,\n OUT field text,\n OUT path text,\n OUT path_txt text,\n OUT jsonpath text,\n OUT eq text\n) RETURNS RECORD AS $$\nDECLARE\npath_elements text[];\nlast_element text;\nBEGIN\ndotpath := replace(trim(dotpath), 'properties.', '');\n\nIF dotpath = '' THEN\n RETURN;\nEND IF;\n\npath_elements := string_to_array(dotpath, '.');\njsonpath := NULL;\n\nIF path_elements[1] IN ('id','geometry','datetime') THEN\n field := path_elements[1];\n path_elements := path_elements[2:];\nELSIF path_elements[1] = 'collection' THEN\n field := 'collection_id';\n path_elements := path_elements[2:];\nELSIF path_elements[1] IN ('links', 'assets', 'stac_version', 'stac_extensions') THEN\n field := 'content';\nELSE\n field := 'content';\n path_elements := '{properties}'::text[] || path_elements;\nEND IF;\nIF cardinality(path_elements)<1 THEN\n path := field;\n path_txt := field;\n jsonpath := '$';\n eq := NULL;\n RETURN;\nEND IF;\n\n\nlast_element := path_elements[cardinality(path_elements)];\npath_elements := path_elements[1:cardinality(path_elements)-1];\njsonpath := concat(array_to_string('{$}'::text[] || array_map_ident(path_elements), '.'), '.', quote_ident(last_element));\npath_elements := array_map_literal(path_elements);\npath := format($F$ properties->%s $F$, quote_literal(dotpath));\npath_txt := format($F$ properties->>%s $F$, quote_literal(dotpath));\neq := format($F$ properties @? '$.%s[*] ? (@ == %%s) '$F$, quote_ident(dotpath));\n\nRAISE NOTICE 'ITEMS PATH -- % % % % %', field, path, path_txt, jsonpath, eq;\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION parse_dtrange(IN _indate jsonb, OUT _tstzrange tstzrange) AS $$\nWITH t AS (\n SELECT CASE\n WHEN jsonb_typeof(_indate) = 'array' THEN\n textarr(_indate)\n ELSE\n regexp_split_to_array(\n btrim(_indate::text,'\"'),\n '/'\n )\n END AS arr\n)\n, t1 AS (\n SELECT\n CASE\n WHEN array_upper(arr,1) = 1 OR arr[1] = '..' OR arr[1] IS NULL THEN '-infinity'::timestamptz\n ELSE arr[1]::timestamptz\n END AS st,\n CASE\n WHEN array_upper(arr,1) = 1 THEN arr[1]::timestamptz\n WHEN arr[2] = '..' OR arr[2] IS NULL THEN 'infinity'::timestamptz\n ELSE arr[2]::timestamptz\n END AS et\n FROM t\n)\nSELECT\n tstzrange(st,et)\nFROM t1;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION bbox_geom(_bbox jsonb) RETURNS geometry AS $$\nSELECT CASE jsonb_array_length(_bbox)\n WHEN 4 THEN\n ST_SetSRID(ST_MakeEnvelope(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float,\n (_bbox->>3)::float\n ),4326)\n WHEN 6 THEN\n ST_SetSRID(ST_3DMakeBox(\n ST_MakePoint(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float\n ),\n ST_MakePoint(\n (_bbox->>3)::float,\n (_bbox->>4)::float,\n (_bbox->>5)::float\n )\n ),4326)\n ELSE null END;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION cql_and_append(existing jsonb, newfilters jsonb) RETURNS jsonb AS $$\nSELECT CASE WHEN existing ? 'filter' AND newfilters IS NOT NULL THEN\n jsonb_build_object(\n 'and',\n jsonb_build_array(\n existing->'filter',\n newfilters\n )\n )\nELSE\n newfilters\nEND;\n$$ LANGUAGE SQL;\n\n\n-- ADDs base filters (ids, collections, datetime, bbox, intersects) that are\n-- added outside of the filter/query in the stac request\nCREATE OR REPLACE FUNCTION add_filters_to_cql(j jsonb) RETURNS jsonb AS $$\nDECLARE\nnewprop jsonb;\nnewprops jsonb := '[]'::jsonb;\nBEGIN\nIF j ? 'ids' THEN\n newprop := jsonb_build_object(\n 'in',\n jsonb_build_array(\n '{\"property\":\"id\"}'::jsonb,\n j->'ids'\n )\n );\n newprops := jsonb_insert(newprops, '{1}', newprop);\nEND IF;\nIF j ? 'collections' THEN\n newprop := jsonb_build_object(\n 'in',\n jsonb_build_array(\n '{\"property\":\"collection\"}'::jsonb,\n j->'collections'\n )\n );\n newprops := jsonb_insert(newprops, '{1}', newprop);\nEND IF;\n\nIF j ? 'datetime' THEN\n newprop := format(\n '{\"anyinteracts\":[{\"property\":\"datetime\"}, %s]}',\n j->'datetime'\n );\n newprops := jsonb_insert(newprops, '{1}', newprop);\nEND IF;\n\nIF j ? 'bbox' THEN\n newprop := format(\n '{\"intersects\":[{\"property\":\"geometry\"}, %s]}',\n j->'bbox'\n );\n newprops := jsonb_insert(newprops, '{1}', newprop);\nEND IF;\n\nIF j ? 'intersects' THEN\n newprop := format(\n '{\"intersects\":[{\"property\":\"geometry\"}, %s]}',\n j->'intersects'\n );\n newprops := jsonb_insert(newprops, '{1}', newprop);\nEND IF;\n\nRAISE NOTICE 'newprops: %', newprops;\n\nIF newprops IS NOT NULL AND jsonb_array_length(newprops) > 0 THEN\n return jsonb_set(\n j,\n '{filter}',\n cql_and_append(j, jsonb_build_object('and', newprops))\n ) - '{ids,collections,datetime,bbox,intersects}'::text[];\nEND IF;\n\nreturn j;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION query_to_cqlfilter(j jsonb) RETURNS jsonb AS $$\n-- Translates anything passed in through the deprecated \"query\" into equivalent CQL\nWITH t AS (\n SELECT key as property, value as ops\n FROM jsonb_each(j->'query')\n), t2 AS (\n SELECT property, (jsonb_each(ops)).*\n FROM t WHERE jsonb_typeof(ops) = 'object'\n UNION ALL\n SELECT property, 'eq', ops\n FROM t WHERE jsonb_typeof(ops) != 'object'\n), t3 AS (\nSELECT\n jsonb_strip_nulls(jsonb_build_object(\n 'and',\n jsonb_agg(\n jsonb_build_object(\n key,\n jsonb_build_array(\n jsonb_build_object('property',property),\n value\n )\n )\n )\n )) as qcql FROM t2\n)\nSELECT\n CASE WHEN qcql IS NOT NULL THEN\n jsonb_set(j, '{filter}', cql_and_append(j, qcql)) - 'query'\n ELSE j\n END\nFROM t3\n;\n$$ LANGUAGE SQL;\n\n\n\nCREATE OR REPLACE FUNCTION temporal_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\nll text := 'datetime';\nlh text := 'end_datetime';\nrrange tstzrange;\nrl text;\nrh text;\noutq text;\nBEGIN\nrrange := parse_dtrange(args->1);\nRAISE NOTICE 'Constructing temporal query OP: %, ARGS: %, RRANGE: %', op, args, rrange;\nop := lower(op);\nrl := format('%L::timestamptz', lower(rrange));\nrh := format('%L::timestamptz', upper(rrange));\noutq := CASE op\n WHEN 't_before' THEN 'lh < rl'\n WHEN 't_after' THEN 'll > rh'\n WHEN 't_meets' THEN 'lh = rl'\n WHEN 't_metby' THEN 'll = rh'\n WHEN 't_overlaps' THEN 'll < rl AND rl < lh < rh'\n WHEN 't_overlappedby' THEN 'rl < ll < rh AND lh > rh'\n WHEN 't_starts' THEN 'll = rl AND lh < rh'\n WHEN 't_startedby' THEN 'll = rl AND lh > rh'\n WHEN 't_during' THEN 'll > rl AND lh < rh'\n WHEN 't_contains' THEN 'll < rl AND lh > rh'\n WHEN 't_finishes' THEN 'll > rl AND lh = rh'\n WHEN 't_finishedby' THEN 'll < rl AND lh = rh'\n WHEN 't_equals' THEN 'll = rl AND lh = rh'\n WHEN 't_disjoint' THEN 'NOT (ll <= rh AND lh >= rl)'\n WHEN 't_intersects' THEN 'll <= rh AND lh >= rl'\n WHEN 'anyinteracts' THEN 'll <= rh AND lh >= rl'\nEND;\noutq := regexp_replace(outq, '\\mll\\M', ll);\noutq := regexp_replace(outq, '\\mlh\\M', lh);\noutq := regexp_replace(outq, '\\mrl\\M', rl);\noutq := regexp_replace(outq, '\\mrh\\M', rh);\noutq := format('(%s)', outq);\nRETURN outq;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION spatial_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\ngeom text;\nj jsonb := args->1;\nBEGIN\nop := lower(op);\nRAISE NOTICE 'Constructing spatial query OP: %, ARGS: %', op, args;\nIF op NOT IN ('s_equals','s_disjoint','s_touches','s_within','s_overlaps','s_crosses','s_intersects','intersects','s_contains') THEN\n RAISE EXCEPTION 'Spatial Operator % Not Supported', op;\nEND IF;\nop := regexp_replace(op, '^s_', 'st_');\nIF op = 'intersects' THEN\n op := 'st_intersects';\nEND IF;\n-- Convert geometry to WKB string\nIF j ? 'type' AND j ? 'coordinates' THEN\n geom := st_geomfromgeojson(j)::text;\nELSIF jsonb_typeof(j) = 'array' THEN\n geom := bbox_geom(j)::text;\nEND IF;\n\nRETURN format('%s(geometry, %L::geometry)', op, geom);\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n/* cql_query_op -- Parses a CQL query operation, recursing when necessary\n IN jsonb -- a subelement from a valid stac query\n IN text -- the operator being used on elements passed in\n RETURNS a SQL fragment to be used in a WHERE clause\n*/\nCREATE OR REPLACE FUNCTION cql_query_op(j jsonb, _op text DEFAULT NULL) RETURNS text AS $$\nDECLARE\njtype text := jsonb_typeof(j);\nop text := lower(_op);\nops jsonb :=\n '{\n \"eq\": \"%s = %s\",\n \"lt\": \"%s < %s\",\n \"lte\": \"%s <= %s\",\n \"gt\": \"%s > %s\",\n \"gte\": \"%s >= %s\",\n \"like\": \"%s LIKE %s\",\n \"+\": \"%s + %s\",\n \"-\": \"%s - %s\",\n \"*\": \"%s * %s\",\n \"/\": \"%s / %s\",\n \"in\": \"%s = ANY (%s)\",\n \"not\": \"NOT (%s)\",\n \"between\": \"%s BETWEEN %s AND %s\",\n \"lower\":\"lower(%s)\"\n }'::jsonb;\nret text;\nargs text[] := NULL;\n\nBEGIN\nRAISE NOTICE 'j: %, op: %, jtype: %', j, op, jtype;\n\n-- Set Lower Case on Both Arguments When Case Insensitive Flag Set\nIF op in ('eq','lt','lte','gt','gte','like') AND jsonb_typeof(j->2) = 'boolean' THEN\n IF (j->>2)::boolean THEN\n RETURN format(concat('(',ops->>op,')'), cql_query_op(jsonb_build_array(j->0), 'lower'), cql_query_op(jsonb_build_array(j->1), 'lower'));\n END IF;\nEND IF;\n\n-- Special Case when comparing a property in a jsonb field to a string or number using eq\n-- Allows to leverage GIN index on jsonb fields\nIF op = 'eq' THEN\n IF j->0 ? 'property'\n AND jsonb_typeof(j->1) IN ('number','string')\n AND (items_path(j->0->>'property')).eq IS NOT NULL\n THEN\n RETURN format((items_path(j->0->>'property')).eq, j->1);\n END IF;\nEND IF;\n\nIF op ilike 't_%' or op = 'anyinteracts' THEN\n RETURN temporal_op_query(op, j);\nEND IF;\n\nIF op ilike 's_%' or op = 'intersects' THEN\n RETURN spatial_op_query(op, j);\nEND IF;\n\n\nIF jtype = 'object' THEN\n RAISE NOTICE 'parsing object';\n IF j ? 'property' THEN\n -- Convert the property to be used as an identifier\n return (items_path(j->>'property')).path_txt;\n ELSIF _op IS NULL THEN\n -- Iterate to convert elements in an object where the operator has not been set\n -- Combining with AND\n SELECT\n array_to_string(array_agg(cql_query_op(e.value, e.key)), ' AND ')\n INTO ret\n FROM jsonb_each(j) e;\n RETURN ret;\n END IF;\nEND IF;\n\nIF jtype = 'string' THEN\n RETURN quote_literal(j->>0);\nEND IF;\n\nIF jtype ='number' THEN\n RETURN (j->>0)::numeric;\nEND IF;\n\nIF jtype = 'array' AND op IS NULL THEN\n RAISE NOTICE 'Parsing array into array arg. j: %', j;\n SELECT format($f$ '{%s}'::text[] $f$, string_agg(e,',')) INTO ret FROM jsonb_array_elements_text(j) e;\n RETURN ret;\nEND IF;\n\n\n-- If the type of the passed json is an array\n-- Calculate the arguments that will be passed to functions/operators\nIF jtype = 'array' THEN\n RAISE NOTICE 'Parsing array into args. j: %', j;\n -- If any argument is numeric, cast any text arguments to numeric\n IF j @? '$[*] ? (@.type() == \"number\")' THEN\n SELECT INTO args\n array_agg(concat('(',cql_query_op(e),')::numeric'))\n FROM jsonb_array_elements(j) e;\n ELSE\n SELECT INTO args\n array_agg(cql_query_op(e))\n FROM jsonb_array_elements(j) e;\n END IF;\n --RETURN args;\nEND IF;\nRAISE NOTICE 'ARGS after array cleaning: %', args;\n\nIF op IS NULL THEN\n RETURN args::text[];\nEND IF;\n\nIF args IS NULL OR cardinality(args) < 1 THEN\n RAISE NOTICE 'No Args';\n RETURN '';\nEND IF;\n\nIF op IN ('and','or') THEN\n SELECT\n CONCAT(\n '(',\n array_to_string(args, UPPER(CONCAT(' ',op,' '))),\n ')'\n ) INTO ret\n FROM jsonb_array_elements(j) e;\n RETURN ret;\nEND IF;\n\n-- If the op is in the ops json then run using the template in the json\nIF ops ? op THEN\n RAISE NOTICE 'ARGS: % MAPPED: %',args, array_map_literal(args);\n\n RETURN format(concat('(',ops->>op,')'), VARIADIC args);\nEND IF;\n\nRETURN j->>0;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n\n\nCREATE OR REPLACE FUNCTION cql_to_where(_search jsonb = '{}'::jsonb) RETURNS text AS $$\nDECLARE\nsearch jsonb := _search;\n_where text;\nBEGIN\nRAISE NOTICE 'SEARCH CQL 1: %', search;\n\n-- Convert any old style stac query to cql\nsearch := query_to_cqlfilter(search);\n\nRAISE NOTICE 'SEARCH CQL 2: %', search;\n\n-- Convert item,collection,datetime,bbox,intersects to cql\nsearch := add_filters_to_cql(search);\n\nRAISE NOTICE 'SEARCH CQL Final: %', search;\n_where := cql_query_op(search->'filter');\n\nIF trim(_where) = '' THEN\n _where := NULL;\nEND IF;\n_where := coalesce(_where, ' TRUE ');\nRETURN _where;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION parse_sort_dir(_dir text, reverse boolean default false) RETURNS text AS $$\nWITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n) SELECT\n CASE\n WHEN NOT reverse THEN d\n WHEN d = 'ASC' THEN 'DESC'\n WHEN d = 'DESC' THEN 'ASC'\n END\nFROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION sort_dir_to_op(_dir text, prev boolean default false) RETURNS text AS $$\nWITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n) SELECT\n CASE\n WHEN d = 'ASC' AND prev THEN '<='\n WHEN d = 'DESC' AND prev THEN '>='\n WHEN d = 'ASC' THEN '>='\n WHEN d = 'DESC' THEN '<='\n END\nFROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION field_orderby(p text) RETURNS text AS $$\nWITH t AS (\n SELECT\n replace(trim(substring(indexdef from 'btree \\((.*)\\)')),' ','')as s\n FROM pg_indexes WHERE schemaname='pgstac' AND tablename='items' AND indexdef ~* 'btree' AND indexdef ~* 'properties'\n) SELECT s FROM t WHERE strpos(s, lower(trim(p)))>0;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION sort_sqlorderby(\n _search jsonb DEFAULT NULL,\n reverse boolean DEFAULT FALSE\n) RETURNS text AS $$\nWITH sortby AS (\n SELECT coalesce(_search->'sortby','[{\"field\":\"datetime\", \"direction\":\"desc\"}]') as sort\n), withid AS (\n SELECT CASE\n WHEN sort @? '$[*] ? (@.field == \"id\")' THEN sort\n ELSE sort || '[{\"field\":\"id\", \"direction\":\"desc\"}]'::jsonb\n END as sort\n FROM sortby\n), withid_rows AS (\n SELECT jsonb_array_elements(sort) as value FROM withid\n),sorts AS (\n SELECT\n coalesce(field_orderby((items_path(value->>'field')).path_txt), (items_path(value->>'field')).path) as key,\n parse_sort_dir(value->>'direction', reverse) as dir\n FROM withid_rows\n)\nSELECT array_to_string(\n array_agg(concat(key, ' ', dir)),\n ', '\n) FROM sorts;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_sort_dir(sort_item jsonb) RETURNS text AS $$\nSELECT CASE WHEN sort_item->>'direction' ILIKE 'desc%' THEN 'DESC' ELSE 'ASC' END;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION get_token_filter(_search jsonb = '{}'::jsonb, token_rec jsonb DEFAULT NULL) RETURNS text AS $$\nDECLARE\n token_id text;\n filters text[] := '{}'::text[];\n prev boolean := TRUE;\n field text;\n dir text;\n sort record;\n orfilters text[] := '{}'::text[];\n andfilters text[] := '{}'::text[];\n output text;\n token_where text;\nBEGIN\n -- If no token provided return NULL\n IF token_rec IS NULL THEN\n IF NOT (_search ? 'token' AND\n (\n (_search->>'token' ILIKE 'prev:%')\n OR\n (_search->>'token' ILIKE 'next:%')\n )\n ) THEN\n RETURN NULL;\n END IF;\n prev := (_search->>'token' ILIKE 'prev:%');\n token_id := substr(_search->>'token', 6);\n SELECT to_jsonb(items) INTO token_rec FROM items WHERE id=token_id;\n END IF;\n RAISE NOTICE 'TOKEN ID: %', token_rec->'id';\n\n CREATE TEMP TABLE sorts (\n _row int GENERATED ALWAYS AS IDENTITY NOT NULL,\n _field text PRIMARY KEY,\n _dir text NOT NULL,\n _val text\n ) ON COMMIT DROP;\n\n -- Make sure we only have distinct columns to sort with taking the first one we get\n INSERT INTO sorts (_field, _dir)\n SELECT\n (items_path(value->>'field')).path,\n get_sort_dir(value)\n FROM\n jsonb_array_elements(coalesce(_search->'sortby','[{\"field\":\"datetime\",\"direction\":\"desc\"}]'))\n ON CONFLICT DO NOTHING\n ;\n RAISE NOTICE 'sorts 1: %', (SELECT jsonb_agg(to_json(sorts)) FROM sorts);\n -- Get the first sort direction provided. As the id is a primary key, if there are any\n -- sorts after id they won't do anything, so make sure that id is the last sort item.\n SELECT _dir INTO dir FROM sorts ORDER BY _row ASC LIMIT 1;\n IF EXISTS (SELECT 1 FROM sorts WHERE _field = 'id') THEN\n DELETE FROM sorts WHERE _row > (SELECT _row FROM sorts WHERE _field = 'id' ORDER BY _row ASC);\n ELSE\n INSERT INTO sorts (_field, _dir) VALUES ('id', dir);\n END IF;\n\n -- Add value from looked up item to the sorts table\n UPDATE sorts SET _val=quote_literal(token_rec->>_field);\n\n -- Check if all sorts are the same direction and use row comparison\n -- to filter\n RAISE NOTICE 'sorts 2: %', (SELECT jsonb_agg(to_json(sorts)) FROM sorts);\n\n IF (SELECT count(DISTINCT _dir) FROM sorts) = 1 THEN\n SELECT format(\n '(%s) %s (%s)',\n concat_ws(', ', VARIADIC array_agg(quote_ident(_field))),\n CASE WHEN (prev AND dir = 'ASC') OR (NOT prev AND dir = 'DESC') THEN '<' ELSE '>' END,\n concat_ws(', ', VARIADIC array_agg(_val))\n ) INTO output FROM sorts\n WHERE token_rec ? _field\n ;\n ELSE\n FOR sort IN SELECT * FROM sorts ORDER BY _row asc LOOP\n RAISE NOTICE 'SORT: %', sort;\n IF sort._row = 1 THEN\n orfilters := orfilters || format('(%s %s %s)',\n quote_ident(sort._field),\n CASE WHEN (prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC') THEN '<' ELSE '>' END,\n sort._val\n );\n ELSE\n orfilters := orfilters || format('(%s AND %s %s %s)',\n array_to_string(andfilters, ' AND '),\n quote_ident(sort._field),\n CASE WHEN (prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC') THEN '<' ELSE '>' END,\n sort._val\n );\n\n END IF;\n andfilters := andfilters || format('%s = %s',\n quote_ident(sort._field),\n sort._val\n );\n END LOOP;\n output := array_to_string(orfilters, ' OR ');\n END IF;\n DROP TABLE IF EXISTS sorts;\n token_where := concat('(',coalesce(output,'true'),')');\n IF trim(token_where) = '' THEN\n token_where := NULL;\n END IF;\n RAISE NOTICE 'TOKEN_WHERE: |%|',token_where;\n RETURN token_where;\n END;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION search_tohash(jsonb) RETURNS jsonb AS $$\n SELECT $1 - '{token,limit,context,includes,excludes}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION search_hash(jsonb, jsonb) RETURNS text AS $$\n SELECT md5(concat(search_tohash($1)::text,$2::text));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TABLE IF NOT EXISTS searches(\n hash text GENERATED ALWAYS AS (search_hash(search, metadata)) STORED PRIMARY KEY,\n search jsonb NOT NULL,\n _where text,\n orderby text,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n metadata jsonb DEFAULT '{}'::jsonb NOT NULL\n);\n\nCREATE TABLE IF NOT EXISTS search_wheres(\n _where text PRIMARY KEY,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n statslastupdated timestamptz,\n estimated_count bigint,\n estimated_cost float,\n time_to_estimate float,\n total_count bigint,\n time_to_count float,\n partitions text[]\n);\n\n\nCREATE OR REPLACE FUNCTION where_stats(inwhere text, updatestats boolean default false) RETURNS search_wheres AS $$\nDECLARE\n t timestamptz;\n i interval;\n explain_json jsonb;\n partitions text[];\n sw search_wheres%ROWTYPE;\nBEGIN\n SELECT * INTO sw FROM search_wheres WHERE _where=inwhere FOR UPDATE;\n\n -- Update statistics if explicitly set, if statistics do not exist, or statistics ttl has expired\n IF NOT updatestats THEN\n RAISE NOTICE 'Checking if update is needed.';\n RAISE NOTICE 'Stats Last Updated: %', sw.statslastupdated;\n RAISE NOTICE 'TTL: %, Age: %', context_stats_ttl(), now() - sw.statslastupdated;\n RAISE NOTICE 'Context: %, Existing Total: %', context(), sw.total_count;\n IF\n sw.statslastupdated IS NULL\n OR (now() - sw.statslastupdated) > context_stats_ttl()\n OR (context() != 'off' AND sw.total_count IS NULL)\n THEN\n updatestats := TRUE;\n END IF;\n END IF;\n\n sw._where := inwhere;\n sw.lastused := now();\n sw.usecount := coalesce(sw.usecount,0) + 1;\n\n IF NOT updatestats THEN\n UPDATE search_wheres SET\n lastused = sw.lastused,\n usecount = sw.usecount\n WHERE _where = inwhere\n RETURNING * INTO sw\n ;\n RETURN sw;\n END IF;\n -- Use explain to get estimated count/cost and a list of the partitions that would be hit by the query\n t := clock_timestamp();\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s', inwhere)\n INTO explain_json;\n RAISE NOTICE 'Time for just the explain: %', clock_timestamp() - t;\n WITH t AS (\n SELECT j->>0 as p FROM\n jsonb_path_query(\n explain_json,\n 'strict $.**.\"Relation Name\" ? (@ != null)'\n ) j\n ), ordered AS (\n SELECT p FROM t ORDER BY p DESC\n -- SELECT p FROM t JOIN items_partitions\n -- ON (t.p = items_partitions.partition)\n -- ORDER BY pstart DESC\n )\n SELECT array_agg(p) INTO partitions FROM ordered;\n i := clock_timestamp() - t;\n RAISE NOTICE 'Time for explain + join: %', clock_timestamp() - t;\n\n\n\n sw.statslastupdated := now();\n sw.estimated_count := explain_json->0->'Plan'->'Plan Rows';\n sw.estimated_cost := explain_json->0->'Plan'->'Total Cost';\n sw.time_to_estimate := extract(epoch from i);\n sw.partitions := partitions;\n\n -- Do a full count of rows if context is set to on or if auto is set and estimates are low enough\n IF\n context() = 'on'\n OR\n ( context() = 'auto' AND\n (\n sw.estimated_count < context_estimated_count()\n OR\n sw.estimated_cost < context_estimated_cost()\n )\n )\n THEN\n t := clock_timestamp();\n RAISE NOTICE 'Calculating actual count...';\n EXECUTE format(\n 'SELECT count(*) FROM items WHERE %s',\n inwhere\n ) INTO sw.total_count;\n i := clock_timestamp() - t;\n RAISE NOTICE 'Actual Count: % -- %', sw.total_count, i;\n sw.time_to_count := extract(epoch FROM i);\n ELSE\n sw.total_count := NULL;\n sw.time_to_count := NULL;\n END IF;\n\n\n INSERT INTO search_wheres SELECT sw.*\n ON CONFLICT (_where)\n DO UPDATE\n SET\n lastused = sw.lastused,\n usecount = sw.usecount,\n statslastupdated = sw.statslastupdated,\n estimated_count = sw.estimated_count,\n estimated_cost = sw.estimated_cost,\n time_to_estimate = sw.time_to_estimate,\n partitions = sw.partitions,\n total_count = sw.total_count,\n time_to_count = sw.time_to_count\n ;\n RETURN sw;\nEND;\n$$ LANGUAGE PLPGSQL ;\n\n\n\nCREATE OR REPLACE FUNCTION items_count(_where text) RETURNS bigint AS $$\nDECLARE\ncnt bigint;\nBEGIN\nEXECUTE format('SELECT count(*) FROM items WHERE %s', _where) INTO cnt;\nRETURN cnt;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n\n\n\nDROP FUNCTION IF EXISTS search_query;\nCREATE OR REPLACE FUNCTION search_query(\n _search jsonb = '{}'::jsonb,\n updatestats boolean = false,\n _metadata jsonb = '{}'::jsonb\n) RETURNS searches AS $$\nDECLARE\n search searches%ROWTYPE;\n pexplain jsonb;\n t timestamptz;\n i interval;\nBEGIN\nSELECT * INTO search FROM searches\nWHERE hash=search_hash(_search, _metadata) FOR UPDATE;\n\n-- Calculate the where clause if not already calculated\nIF search._where IS NULL THEN\n search._where := cql_to_where(_search);\nEND IF;\n\n-- Calculate the order by clause if not already calculated\nIF search.orderby IS NULL THEN\n search.orderby := sort_sqlorderby(_search);\nEND IF;\n\nPERFORM where_stats(search._where, updatestats);\n\nsearch.lastused := now();\nsearch.usecount := coalesce(search.usecount, 0) + 1;\nINSERT INTO searches (search, _where, orderby, lastused, usecount, metadata)\nVALUES (_search, search._where, search.orderby, search.lastused, search.usecount, _metadata)\nON CONFLICT (hash) DO\nUPDATE SET\n _where = EXCLUDED._where,\n orderby = EXCLUDED.orderby,\n lastused = EXCLUDED.lastused,\n usecount = EXCLUDED.usecount,\n metadata = EXCLUDED.metadata\nRETURNING * INTO search\n;\nRETURN search;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n\nCREATE OR REPLACE FUNCTION search(_search jsonb = '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n searches searches%ROWTYPE;\n _where text;\n token_where text;\n full_where text;\n orderby text;\n query text;\n token_type text := substr(_search->>'token',1,4);\n _limit int := coalesce((_search->>'limit')::int, 10);\n curs refcursor;\n cntr int := 0;\n iter_record items%ROWTYPE;\n first_record items%ROWTYPE;\n last_record items%ROWTYPE;\n out_records jsonb := '[]'::jsonb;\n prev_query text;\n next text;\n prev_id text;\n has_next boolean := false;\n has_prev boolean := false;\n prev text;\n total_count bigint;\n context jsonb;\n collection jsonb;\n includes text[];\n excludes text[];\n exit_flag boolean := FALSE;\n batches int := 0;\n timer timestamptz := clock_timestamp();\n pstart timestamptz;\n pend timestamptz;\n pcurs refcursor;\n search_where search_wheres%ROWTYPE;\nBEGIN\nsearches := search_query(_search);\n_where := searches._where;\norderby := searches.orderby;\nsearch_where := where_stats(_where);\ntotal_count := coalesce(search_where.total_count, search_where.estimated_count);\n\n\nIF token_type='prev' THEN\n token_where := get_token_filter(_search, null::jsonb);\n orderby := sort_sqlorderby(_search, TRUE);\nEND IF;\nIF token_type='next' THEN\n token_where := get_token_filter(_search, null::jsonb);\nEND IF;\n\nfull_where := concat_ws(' AND ', _where, token_where);\nRAISE NOTICE 'FULL QUERY % %', full_where, clock_timestamp()-timer;\ntimer := clock_timestamp();\n\nFOR query IN SELECT partition_queries(full_where, orderby, search_where.partitions) LOOP\n timer := clock_timestamp();\n query := format('%s LIMIT %s', query, _limit + 1);\n RAISE NOTICE 'Partition Query: %', query;\n batches := batches + 1;\n curs = create_cursor(query);\n --OPEN curs\n LOOP\n FETCH curs into iter_record;\n EXIT WHEN NOT FOUND;\n cntr := cntr + 1;\n last_record := iter_record;\n IF cntr = 1 THEN\n first_record := last_record;\n END IF;\n IF cntr <= _limit THEN\n out_records := out_records || last_record.content;\n ELSIF cntr > _limit THEN\n has_next := true;\n exit_flag := true;\n EXIT;\n END IF;\n END LOOP;\n RAISE NOTICE 'Query took %. Total Time %', clock_timestamp()-timer, ftime();\n timer := clock_timestamp();\n EXIT WHEN exit_flag;\nEND LOOP;\nRAISE NOTICE 'Scanned through % partitions.', batches;\n\n\n-- Flip things around if this was the result of a prev token query\nIF token_type='prev' THEN\n out_records := flip_jsonb_array(out_records);\n first_record := last_record;\nEND IF;\n\n-- If this query has a token, see if there is data before the first record\nIF _search ? 'token' THEN\n prev_query := format(\n 'SELECT 1 FROM items WHERE %s LIMIT 1',\n concat_ws(\n ' AND ',\n _where,\n trim(get_token_filter(_search, to_jsonb(first_record)))\n )\n );\n RAISE NOTICE 'Query to get previous record: % --- %', prev_query, first_record;\n EXECUTE prev_query INTO has_prev;\n IF FOUND and has_prev IS NOT NULL THEN\n RAISE NOTICE 'Query results from prev query: %', has_prev;\n has_prev := TRUE;\n END IF;\nEND IF;\nhas_prev := COALESCE(has_prev, FALSE);\n\nRAISE NOTICE 'token_type: %, has_next: %, has_prev: %', token_type, has_next, has_prev;\nIF has_prev THEN\n prev := out_records->0->>'id';\nEND IF;\nIF has_next OR token_type='prev' THEN\n next := out_records->-1->>'id';\nEND IF;\n\n\n-- include/exclude any fields following fields extension\nIF _search ? 'fields' THEN\n IF _search->'fields' ? 'exclude' THEN\n excludes=textarr(_search->'fields'->'exclude');\n END IF;\n IF _search->'fields' ? 'include' THEN\n includes=textarr(_search->'fields'->'include');\n IF array_length(includes, 1)>0 AND NOT 'id' = ANY (includes) THEN\n includes = includes || '{id}';\n END IF;\n END IF;\n SELECT jsonb_agg(filter_jsonb(row, includes, excludes)) INTO out_records FROM jsonb_array_elements(out_records) row;\nEND IF;\n\nIF context() != 'off' THEN\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'matched', total_count,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\nELSE\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\nEND IF;\n\ncollection := jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb),\n 'next', next,\n 'prev', prev,\n 'context', context\n);\n\nRETURN collection;\nEND;\n$$ LANGUAGE PLPGSQL\nSET jit TO off\n;\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION tileenvelope(zoom int, x int, y int) RETURNS geometry AS $$\nWITH t AS (\n SELECT\n 20037508.3427892 as merc_max,\n -20037508.3427892 as merc_min,\n (2 * 20037508.3427892) / (2 ^ zoom) as tile_size\n)\nSELECT st_makeenvelope(\n merc_min + (tile_size * x),\n merc_max - (tile_size * (y + 1)),\n merc_min + (tile_size * (x + 1)),\n merc_max - (tile_size * y),\n 3857\n) FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;DROP FUNCTION IF EXISTS mercgrid;\n\n\nCREATE OR REPLACE FUNCTION ftime() RETURNS interval as $$\nSELECT age(clock_timestamp(), transaction_timestamp());\n$$ LANGUAGE SQL;\nSET SEARCH_PATH to pgstac, public;\n\nDROP FUNCTION IF EXISTS geometrysearch;\nCREATE OR REPLACE FUNCTION geometrysearch(\n IN geom geometry,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE, -- Return as soon as the passed in geometry is full covered\n IN skipcovered boolean DEFAULT TRUE -- Skip any items that would show up completely under the previous items\n) RETURNS jsonb AS $$\nDECLARE\n search searches%ROWTYPE;\n curs refcursor;\n _where text;\n query text;\n iter_record items%ROWTYPE;\n out_records jsonb[] := '{}'::jsonb[];\n exit_flag boolean := FALSE;\n counter int := 1;\n scancounter int := 1;\n remaining_limit int := _scanlimit;\n tilearea float;\n unionedgeom geometry;\n clippedgeom geometry;\n unionedgeom_area float := 0;\n prev_area float := 0;\n excludes text[];\n includes text[];\n\nBEGIN\n -- If skipcovered is true then you will always want to exit when the passed in geometry is full\n IF skipcovered THEN\n exitwhenfull := TRUE;\n END IF;\n\n SELECT * INTO search FROM searches WHERE hash=queryhash;\n\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Search with Query Hash % Not Found', queryhash;\n END IF;\n\n tilearea := st_area(geom);\n _where := format('%s AND st_intersects(geometry, %L::geometry)', search._where, geom);\n\n IF fields IS NOT NULL THEN\n IF fields ? 'fields' THEN\n fields := fields->'fields';\n END IF;\n IF fields ? 'exclude' THEN\n excludes=textarr(fields->'exclude');\n END IF;\n IF fields ? 'include' THEN\n includes=textarr(fields->'include');\n IF array_length(includes, 1)>0 AND NOT 'id' = ANY (includes) THEN\n includes = includes || '{id}';\n END IF;\n END IF;\n END IF;\n RAISE NOTICE 'fields: %, includes: %, excludes: %', fields, includes, excludes;\n\n FOR query IN SELECT * FROM partition_queries(_where, search.orderby) LOOP\n query := format('%s LIMIT %L', query, remaining_limit);\n RAISE NOTICE '%', query;\n curs = create_cursor(query);\n LOOP\n FETCH curs INTO iter_record;\n EXIT WHEN NOT FOUND;\n IF exitwhenfull OR skipcovered THEN -- If we are not using exitwhenfull or skipcovered, we do not need to do expensive geometry operations\n clippedgeom := st_intersection(geom, iter_record.geometry);\n\n IF unionedgeom IS NULL THEN\n unionedgeom := clippedgeom;\n ELSE\n unionedgeom := st_union(unionedgeom, clippedgeom);\n END IF;\n\n unionedgeom_area := st_area(unionedgeom);\n\n IF skipcovered AND prev_area = unionedgeom_area THEN\n scancounter := scancounter + 1;\n CONTINUE;\n END IF;\n\n prev_area := unionedgeom_area;\n\n RAISE NOTICE '% % % %', unionedgeom_area/tilearea, counter, scancounter, ftime();\n END IF;\n\n IF fields IS NOT NULL THEN\n out_records := out_records || filter_jsonb(iter_record.content, includes, excludes);\n ELSE\n out_records := out_records || iter_record.content;\n END IF;\n IF counter >= _limit\n OR scancounter > _scanlimit\n OR ftime() > _timelimit\n OR (exitwhenfull AND unionedgeom_area >= tilearea)\n THEN\n exit_flag := TRUE;\n EXIT;\n END IF;\n counter := counter + 1;\n scancounter := scancounter + 1;\n\n END LOOP;\n EXIT WHEN exit_flag;\n remaining_limit := _scanlimit - scancounter;\n END LOOP;\n\n RETURN jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', array_to_json(out_records)::jsonb\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDROP FUNCTION IF EXISTS geojsonsearch;\nCREATE OR REPLACE FUNCTION geojsonsearch(\n IN geojson jsonb,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_geomfromgeojson(geojson),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS xyzsearch;\nCREATE OR REPLACE FUNCTION xyzsearch(\n IN _x int,\n IN _y int,\n IN _z int,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_transform(tileenvelope(_z, _x, _y), 4326),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\nSELECT set_version('0.3.5');\n" }, { "path": "src/pgstac/migrations/pgstac.0.3.6-0.4.0.sql", "content": "SET SEARCH_PATH to pgstac, public;\ndrop function if exists \"pgstac\".\"context\"();\n\ndrop function if exists \"pgstac\".\"context_estimated_cost\"();\n\ndrop function if exists \"pgstac\".\"context_estimated_count\"();\n\ndrop function if exists \"pgstac\".\"context_stats_ttl\"();\n\ndrop function if exists \"pgstac\".\"get_setting\"(setting text, INOUT _default anynonarray);\n\ndrop function if exists \"pgstac\".\"where_stats\"(inwhere text, updatestats boolean);\n\ncreate table \"pgstac\".\"pgstac_settings\" (\n \"name\" text not null,\n \"value\" text not null\n);\n\n\nCREATE UNIQUE INDEX pgstac_settings_pkey ON pgstac.pgstac_settings USING btree (name);\n\nalter table \"pgstac\".\"pgstac_settings\" add constraint \"pgstac_settings_pkey\" PRIMARY KEY using index \"pgstac_settings_pkey\";\n\nset check_function_bodies = off;\n\nCREATE OR REPLACE FUNCTION pgstac.context(conf jsonb DEFAULT NULL::jsonb)\n RETURNS text\n LANGUAGE sql\nAS $function$\n SELECT get_setting('context', conf);\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.context_estimated_cost(conf jsonb DEFAULT NULL::jsonb)\n RETURNS double precision\n LANGUAGE sql\nAS $function$\n SELECT get_setting('context_estimated_cost', conf)::float;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.context_estimated_count(conf jsonb DEFAULT NULL::jsonb)\n RETURNS integer\n LANGUAGE sql\nAS $function$\n SELECT get_setting('context_estimated_count', conf)::int;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.context_stats_ttl(conf jsonb DEFAULT NULL::jsonb)\n RETURNS interval\n LANGUAGE sql\nAS $function$\n SELECT get_setting('context_stats_ttl', conf)::interval;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.cql2_query(j jsonb, recursion integer DEFAULT 0)\n RETURNS text\n LANGUAGE plpgsql\nAS $function$\nDECLARE\nargs jsonb := j->'args';\njtype text := jsonb_typeof(j->'args');\nop text := lower(j->>'op');\narg jsonb;\nargtext text;\nargstext text[] := '{}'::text[];\ninobj jsonb;\n_numeric text := '';\nops jsonb :=\n '{\n \"eq\": \"%s = %s\",\n \"lt\": \"%s < %s\",\n \"lte\": \"%s <= %s\",\n \"gt\": \"%s > %s\",\n \"gte\": \"%s >= %s\",\n \"le\": \"%s <= %s\",\n \"ge\": \"%s >= %s\",\n \"=\": \"%s = %s\",\n \"<\": \"%s < %s\",\n \"<=\": \"%s <= %s\",\n \">\": \"%s > %s\",\n \">=\": \"%s >= %s\",\n \"like\": \"%s LIKE %s\",\n \"ilike\": \"%s ILIKE %s\",\n \"+\": \"%s + %s\",\n \"-\": \"%s - %s\",\n \"*\": \"%s * %s\",\n \"/\": \"%s / %s\",\n \"in\": \"%s = ANY (%s)\",\n \"not\": \"NOT (%s)\",\n \"between\": \"%s BETWEEN (%2$s)[1] AND (%2$s)[2]\",\n \"lower\":\" lower(%s)\",\n \"upper\":\" upper(%s)\",\n \"isnull\": \"%s IS NULL\"\n }'::jsonb;\nret text;\n\nBEGIN\nRAISE NOTICE 'j: %s', j;\nIF j ? 'filter' THEN\n RETURN cql2_query(j->'filter');\nEND IF;\n\nIF j ? 'upper' THEN\nRAISE NOTICE 'upper %s',jsonb_build_object(\n 'op', 'upper',\n 'args', jsonb_build_array( j-> 'upper')\n ) ;\n RETURN cql2_query(\n jsonb_build_object(\n 'op', 'upper',\n 'args', jsonb_build_array( j-> 'upper')\n )\n );\nEND IF;\n\nIF j ? 'lower' THEN\n RETURN cql2_query(\n jsonb_build_object(\n 'op', 'lower',\n 'args', jsonb_build_array( j-> 'lower')\n )\n );\nEND IF;\n\nIF j ? 'args' AND jsonb_typeof(args) != 'array' THEN\n args := jsonb_build_array(args);\nEND IF;\n-- END Cases where no further nesting is expected\nIF j ? 'op' THEN\n -- Special case to use JSONB index for equality\n IF op = 'eq'\n AND args->0 ? 'property'\n AND jsonb_typeof(args->1) IN ('number', 'string')\n AND (items_path(args->0->>'property')).eq IS NOT NULL\n THEN\n RETURN format((items_path(args->0->>'property')).eq, args->1);\n END IF;\n\n -- Temporal Query\n IF op ilike 't_%' or op = 'anyinteracts' THEN\n RETURN temporal_op_query(op, args);\n END IF;\n\n -- Spatial Query\n IF op ilike 's_%' or op = 'intersects' THEN\n RETURN spatial_op_query(op, args);\n END IF;\n\n -- In Query - separate into separate eq statements so that we can use eq jsonb optimization\n IF op = 'in' THEN\n RAISE NOTICE '% IN args: %', repeat(' ', recursion), args;\n SELECT INTO inobj\n jsonb_agg(\n jsonb_build_object(\n 'op', 'eq',\n 'args', jsonb_build_array( args->0 , v)\n )\n )\n FROM jsonb_array_elements( args->1) v;\n RETURN cql2_query(jsonb_build_object('op','or','args',inobj));\n END IF;\nEND IF;\n\nIF j ? 'property' THEN\n RETURN (items_path(j->>'property')).path_txt;\nEND IF;\n\nRAISE NOTICE '%jtype: %',repeat(' ', recursion), jtype;\nIF jsonb_typeof(j) = 'number' THEN\n RETURN format('%L::numeric', j->>0);\nEND IF;\n\nIF jsonb_typeof(j) = 'string' THEN\n RETURN quote_literal(j->>0);\nEND IF;\n\nIF jsonb_typeof(j) = 'array' THEN\n IF j @? '$[*] ? (@.type() == \"number\")' THEN\n RETURN CONCAT(quote_literal(textarr(j)::text), '::numeric[]');\n ELSE\n RETURN CONCAT(quote_literal(textarr(j)::text), '::text[]');\n END IF;\nEND IF;\nRAISE NOTICE 'ARGS after array cleaning: %', args;\n\nRAISE NOTICE '%beforeargs op: %, args: %',repeat(' ', recursion), op, args;\nIF j ? 'args' THEN\n FOR arg in SELECT * FROM jsonb_array_elements(args) LOOP\n argtext := cql2_query(arg, recursion + 1);\n RAISE NOTICE '% -- arg: %, argtext: %', repeat(' ', recursion), arg, argtext;\n argstext := argstext || argtext;\n END LOOP;\nEND IF;\nRAISE NOTICE '%afterargs op: %, argstext: %',repeat(' ', recursion), op, argstext;\n\n\nIF op IN ('and', 'or') THEN\n RAISE NOTICE 'inand op: %, argstext: %', op, argstext;\n SELECT\n concat(' ( ',array_to_string(array_agg(e), concat(' ',op,' ')),' ) ')\n INTO ret\n FROM unnest(argstext) e;\n RETURN ret;\nEND IF;\n\nIF ops ? op THEN\n IF argstext[2] ~* 'numeric' THEN\n argstext := ARRAY[concat('(',argstext[1],')::numeric')] || argstext[2:3];\n END IF;\n RETURN format(concat('(',ops->>op,')'), VARIADIC argstext);\nEND IF;\n\nRAISE NOTICE '%op: %, argstext: %',repeat(' ', recursion), op, argstext;\n\nRETURN NULL;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.get_setting(_setting text, conf jsonb DEFAULT NULL::jsonb)\n RETURNS text\n LANGUAGE sql\nAS $function$\nSELECT COALESCE(\n conf->>_setting,\n current_setting(concat('pgstac.',_setting), TRUE),\n (SELECT value FROM pgstac_settings WHERE name=_setting)\n);\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.where_stats(inwhere text, updatestats boolean DEFAULT false, conf jsonb DEFAULT NULL::jsonb)\n RETURNS search_wheres\n LANGUAGE plpgsql\nAS $function$\nDECLARE\n t timestamptz;\n i interval;\n explain_json jsonb;\n partitions text[];\n sw search_wheres%ROWTYPE;\nBEGIN\n SELECT * INTO sw FROM search_wheres WHERE _where=inwhere FOR UPDATE;\n\n -- Update statistics if explicitly set, if statistics do not exist, or statistics ttl has expired\n IF NOT updatestats THEN\n RAISE NOTICE 'Checking if update is needed.';\n RAISE NOTICE 'Stats Last Updated: %', sw.statslastupdated;\n RAISE NOTICE 'TTL: %, Age: %', context_stats_ttl(conf), now() - sw.statslastupdated;\n RAISE NOTICE 'Context: %, Existing Total: %', context(conf), sw.total_count;\n IF\n sw.statslastupdated IS NULL\n OR (now() - sw.statslastupdated) > context_stats_ttl(conf)\n OR (context(conf) != 'off' AND sw.total_count IS NULL)\n THEN\n updatestats := TRUE;\n END IF;\n END IF;\n\n sw._where := inwhere;\n sw.lastused := now();\n sw.usecount := coalesce(sw.usecount,0) + 1;\n\n IF NOT updatestats THEN\n UPDATE search_wheres SET\n lastused = sw.lastused,\n usecount = sw.usecount\n WHERE _where = inwhere\n RETURNING * INTO sw\n ;\n RETURN sw;\n END IF;\n -- Use explain to get estimated count/cost and a list of the partitions that would be hit by the query\n t := clock_timestamp();\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s', inwhere)\n INTO explain_json;\n RAISE NOTICE 'Time for just the explain: %', clock_timestamp() - t;\n WITH t AS (\n SELECT j->>0 as p FROM\n jsonb_path_query(\n explain_json,\n 'strict $.**.\"Relation Name\" ? (@ != null)'\n ) j\n ), ordered AS (\n SELECT p FROM t ORDER BY p DESC\n -- SELECT p FROM t JOIN items_partitions\n -- ON (t.p = items_partitions.partition)\n -- ORDER BY pstart DESC\n )\n SELECT array_agg(p) INTO partitions FROM ordered;\n i := clock_timestamp() - t;\n RAISE NOTICE 'Time for explain + join: %', clock_timestamp() - t;\n\n\n\n sw.statslastupdated := now();\n sw.estimated_count := explain_json->0->'Plan'->'Plan Rows';\n sw.estimated_cost := explain_json->0->'Plan'->'Total Cost';\n sw.time_to_estimate := extract(epoch from i);\n sw.partitions := partitions;\n\n -- Do a full count of rows if context is set to on or if auto is set and estimates are low enough\n IF\n context(conf) = 'on'\n OR\n ( context(conf) = 'auto' AND\n (\n sw.estimated_count < context_estimated_count(conf)\n OR\n sw.estimated_cost < context_estimated_cost(conf)\n )\n )\n THEN\n t := clock_timestamp();\n RAISE NOTICE 'Calculating actual count...';\n EXECUTE format(\n 'SELECT count(*) FROM items WHERE %s',\n inwhere\n ) INTO sw.total_count;\n i := clock_timestamp() - t;\n RAISE NOTICE 'Actual Count: % -- %', sw.total_count, i;\n sw.time_to_count := extract(epoch FROM i);\n ELSE\n sw.total_count := NULL;\n sw.time_to_count := NULL;\n END IF;\n\n\n INSERT INTO search_wheres SELECT sw.*\n ON CONFLICT (_where)\n DO UPDATE\n SET\n lastused = sw.lastused,\n usecount = sw.usecount,\n statslastupdated = sw.statslastupdated,\n estimated_count = sw.estimated_count,\n estimated_cost = sw.estimated_cost,\n time_to_estimate = sw.time_to_estimate,\n partitions = sw.partitions,\n total_count = sw.total_count,\n time_to_count = sw.time_to_count\n ;\n RETURN sw;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.cql_query_op(j jsonb, _op text DEFAULT NULL::text)\n RETURNS text\n LANGUAGE plpgsql\nAS $function$\nDECLARE\njtype text := jsonb_typeof(j);\nop text := lower(_op);\nops jsonb :=\n '{\n \"eq\": \"%s = %s\",\n \"lt\": \"%s < %s\",\n \"lte\": \"%s <= %s\",\n \"gt\": \"%s > %s\",\n \"gte\": \"%s >= %s\",\n \"le\": \"%s <= %s\",\n \"ge\": \"%s >= %s\",\n \"=\": \"%s = %s\",\n \"<\": \"%s < %s\",\n \"<=\": \"%s <= %s\",\n \">\": \"%s > %s\",\n \">=\": \"%s >= %s\",\n \"like\": \"%s LIKE %s\",\n \"ilike\": \"%s ILIKE %s\",\n \"+\": \"%s + %s\",\n \"-\": \"%s - %s\",\n \"*\": \"%s * %s\",\n \"/\": \"%s / %s\",\n \"in\": \"%s = ANY (%s)\",\n \"not\": \"NOT (%s)\",\n \"between\": \"%s BETWEEN %s AND %s\",\n \"lower\":\" lower(%s)\",\n \"upper\":\" upper(%s)\",\n \"isnull\": \"%s IS NULL\"\n }'::jsonb;\nret text;\nargs text[] := NULL;\n\nBEGIN\nRAISE NOTICE 'j: %, op: %, jtype: %', j, op, jtype;\n\n-- for in, convert value, list to array syntax to match other ops\nIF op = 'in' and j ? 'value' and j ? 'list' THEN\n j := jsonb_build_array( j->'value', j->'list');\n jtype := 'array';\n RAISE NOTICE 'IN: j: %, jtype: %', j, jtype;\nEND IF;\n\nIF op = 'between' and j ? 'value' and j ? 'lower' and j ? 'upper' THEN\n j := jsonb_build_array( j->'value', j->'lower', j->'upper');\n jtype := 'array';\n RAISE NOTICE 'BETWEEN: j: %, jtype: %', j, jtype;\nEND IF;\n\nIF op = 'not' AND jtype = 'object' THEN\n j := jsonb_build_array( j );\n jtype := 'array';\n RAISE NOTICE 'NOT: j: %, jtype: %', j, jtype;\nEND IF;\n\n-- Set Lower Case on Both Arguments When Case Insensitive Flag Set\nIF op in ('eq','lt','lte','gt','gte','like') AND jsonb_typeof(j->2) = 'boolean' THEN\n IF (j->>2)::boolean THEN\n RETURN format(concat('(',ops->>op,')'), cql_query_op(jsonb_build_array(j->0), 'lower'), cql_query_op(jsonb_build_array(j->1), 'lower'));\n END IF;\nEND IF;\n\n-- Special Case when comparing a property in a jsonb field to a string or number using eq\n-- Allows to leverage GIN index on jsonb fields\nIF op = 'eq' THEN\n IF j->0 ? 'property'\n AND jsonb_typeof(j->1) IN ('number','string')\n AND (items_path(j->0->>'property')).eq IS NOT NULL\n THEN\n RETURN format((items_path(j->0->>'property')).eq, j->1);\n END IF;\nEND IF;\n\n\n\nIF op ilike 't_%' or op = 'anyinteracts' THEN\n RETURN temporal_op_query(op, j);\nEND IF;\n\nIF op ilike 's_%' or op = 'intersects' THEN\n RETURN spatial_op_query(op, j);\nEND IF;\n\n\nIF jtype = 'object' THEN\n RAISE NOTICE 'parsing object';\n IF j ? 'property' THEN\n -- Convert the property to be used as an identifier\n return (items_path(j->>'property')).path_txt;\n ELSIF _op IS NULL THEN\n -- Iterate to convert elements in an object where the operator has not been set\n -- Combining with AND\n SELECT\n array_to_string(array_agg(cql_query_op(e.value, e.key)), ' AND ')\n INTO ret\n FROM jsonb_each(j) e;\n RETURN ret;\n END IF;\nEND IF;\n\nIF jtype = 'string' THEN\n RETURN quote_literal(j->>0);\nEND IF;\n\nIF jtype ='number' THEN\n RETURN (j->>0)::numeric;\nEND IF;\n\nIF jtype = 'array' AND op IS NULL THEN\n RAISE NOTICE 'Parsing array into array arg. j: %', j;\n SELECT format($f$ '{%s}'::text[] $f$, string_agg(e,',')) INTO ret FROM jsonb_array_elements_text(j) e;\n RETURN ret;\nEND IF;\n\n\n-- If the type of the passed json is an array\n-- Calculate the arguments that will be passed to functions/operators\nIF jtype = 'array' THEN\n RAISE NOTICE 'Parsing array into args. j: %', j;\n -- If any argument is numeric, cast any text arguments to numeric\n IF j @? '$[*] ? (@.type() == \"number\")' THEN\n SELECT INTO args\n array_agg(concat('(',cql_query_op(e),')::numeric'))\n FROM jsonb_array_elements(j) e;\n ELSE\n SELECT INTO args\n array_agg(cql_query_op(e))\n FROM jsonb_array_elements(j) e;\n END IF;\n --RETURN args;\nEND IF;\nRAISE NOTICE 'ARGS after array cleaning: %', args;\n\nIF op IS NULL THEN\n RETURN args::text[];\nEND IF;\n\nIF args IS NULL OR cardinality(args) < 1 THEN\n RAISE NOTICE 'No Args';\n RETURN '';\nEND IF;\n\nIF op IN ('and','or') THEN\n SELECT\n CONCAT(\n '(',\n array_to_string(args, UPPER(CONCAT(' ',op,' '))),\n ')'\n ) INTO ret\n FROM jsonb_array_elements(j) e;\n RETURN ret;\nEND IF;\n\n-- If the op is in the ops json then run using the template in the json\nIF ops ? op THEN\n RAISE NOTICE 'ARGS: % MAPPED: %',args, array_map_literal(args);\n\n RETURN format(concat('(',ops->>op,')'), VARIADIC args);\nEND IF;\n\nRETURN j->>0;\n\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.cql_to_where(_search jsonb DEFAULT '{}'::jsonb)\n RETURNS text\n LANGUAGE plpgsql\nAS $function$\nDECLARE\nsearch jsonb := _search;\n_where text;\nBEGIN\n\nRAISE NOTICE 'SEARCH CQL Final: %', search;\nIF (search ? 'filter-lang' AND search->>'filter-lang' = 'cql-json') OR get_setting('default-filter-lang', _search->'conf')='cql-json' THEN\n search := query_to_cqlfilter(search);\n search := add_filters_to_cql(search);\n _where := cql_query_op(search->'filter');\nELSE\n _where := cql2_query(search->'filter');\nEND IF;\n\nIF trim(_where) = '' THEN\n _where := NULL;\nEND IF;\n_where := coalesce(_where, ' TRUE ');\nRETURN _where;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.items_staging_ignore_insert_triggerfunc()\n RETURNS trigger\n LANGUAGE plpgsql\nAS $function$\nDECLARE\n p record;\n _partitions text[];\nBEGIN\n CREATE TEMP TABLE new_partitions ON COMMIT DROP AS\n SELECT\n items_partition_name(stac_datetime(content)) as partition,\n date_trunc('week', min(stac_datetime(content))) as partition_start\n FROM newdata\n GROUP BY 1;\n\n -- set statslastupdated in cache to be old enough cache always regenerated\n\n SELECT array_agg(partition) INTO _partitions FROM new_partitions;\n UPDATE search_wheres\n SET\n statslastupdated = NULL\n WHERE _where IN (\n SELECT _where FROM search_wheres sw WHERE sw.partitions && _partitions\n FOR UPDATE SKIP LOCKED\n )\n ;\n\n FOR p IN SELECT new_partitions.partition, new_partitions.partition_start, new_partitions.partition_start + '1 week'::interval as partition_end FROM new_partitions\n LOOP\n RAISE NOTICE 'Getting partition % ready.', p.partition;\n IF NOT items_partition_exists(p.partition) THEN\n RAISE NOTICE 'Creating partition %.', p.partition;\n PERFORM items_partition_create_worker(p.partition, p.partition_start, p.partition_end);\n END IF;\n PERFORM drop_partition_constraints(p.partition);\n END LOOP;\n\n INSERT INTO items (id, geometry, collection_id, datetime, end_datetime, properties, content)\n SELECT\n content->>'id',\n stac_geom(content),\n content->>'collection',\n stac_datetime(content),\n stac_end_datetime(content),\n properties_idx(content),\n content\n FROM newdata\n ON CONFLICT (datetime, id) DO NOTHING\n ;\n DELETE FROM items_staging;\n\n\n FOR p IN SELECT new_partitions.partition FROM new_partitions\n LOOP\n RAISE NOTICE 'Setting constraints for partition %.', p.partition;\n PERFORM partition_checks(p.partition);\n END LOOP;\n DROP TABLE IF EXISTS new_partitions;\n RETURN NULL;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.items_staging_insert_triggerfunc()\n RETURNS trigger\n LANGUAGE plpgsql\nAS $function$\nDECLARE\n p record;\n _partitions text[];\nBEGIN\n CREATE TEMP TABLE new_partitions ON COMMIT DROP AS\n SELECT\n items_partition_name(stac_datetime(content)) as partition,\n date_trunc('week', min(stac_datetime(content))) as partition_start\n FROM newdata\n GROUP BY 1;\n\n -- set statslastupdated in cache to be old enough cache always regenerated\n\n SELECT array_agg(partition) INTO _partitions FROM new_partitions;\n UPDATE search_wheres\n SET\n statslastupdated = NULL\n WHERE _where IN (\n SELECT _where FROM search_wheres sw WHERE sw.partitions && _partitions\n FOR UPDATE SKIP LOCKED\n )\n ;\n FOR p IN SELECT new_partitions.partition, new_partitions.partition_start, new_partitions.partition_start + '1 week'::interval as partition_end FROM new_partitions\n LOOP\n RAISE NOTICE 'Getting partition % ready.', p.partition;\n IF NOT items_partition_exists(p.partition) THEN\n RAISE NOTICE 'Creating partition %.', p.partition;\n PERFORM items_partition_create_worker(p.partition, p.partition_start, p.partition_end);\n END IF;\n PERFORM drop_partition_constraints(p.partition);\n END LOOP;\n INSERT INTO items (id, geometry, collection_id, datetime, end_datetime, properties, content)\n SELECT\n content->>'id',\n stac_geom(content),\n content->>'collection',\n stac_datetime(content),\n stac_end_datetime(content),\n properties_idx(content),\n content\n FROM newdata\n ;\n DELETE FROM items_staging;\n\n\n FOR p IN SELECT new_partitions.partition FROM new_partitions\n LOOP\n RAISE NOTICE 'Setting constraints for partition %.', p.partition;\n PERFORM partition_checks(p.partition);\n END LOOP;\n DROP TABLE IF EXISTS new_partitions;\n RETURN NULL;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.items_staging_upsert_insert_triggerfunc()\n RETURNS trigger\n LANGUAGE plpgsql\nAS $function$\nDECLARE\n p record;\n _partitions text[];\nBEGIN\n CREATE TEMP TABLE new_partitions ON COMMIT DROP AS\n SELECT\n items_partition_name(stac_datetime(content)) as partition,\n date_trunc('week', min(stac_datetime(content))) as partition_start\n FROM newdata\n GROUP BY 1;\n\n -- set statslastupdated in cache to be old enough cache always regenerated\n\n SELECT array_agg(partition) INTO _partitions FROM new_partitions;\n UPDATE search_wheres\n SET\n statslastupdated = NULL\n WHERE _where IN (\n SELECT _where FROM search_wheres sw WHERE sw.partitions && _partitions\n FOR UPDATE SKIP LOCKED\n )\n ;\n\n FOR p IN SELECT new_partitions.partition, new_partitions.partition_start, new_partitions.partition_start + '1 week'::interval as partition_end FROM new_partitions\n LOOP\n RAISE NOTICE 'Getting partition % ready.', p.partition;\n IF NOT items_partition_exists(p.partition) THEN\n RAISE NOTICE 'Creating partition %.', p.partition;\n PERFORM items_partition_create_worker(p.partition, p.partition_start, p.partition_end);\n END IF;\n PERFORM drop_partition_constraints(p.partition);\n END LOOP;\n\n INSERT INTO items (id, geometry, collection_id, datetime, end_datetime, properties, content)\n SELECT\n content->>'id',\n stac_geom(content),\n content->>'collection',\n stac_datetime(content),\n stac_end_datetime(content),\n properties_idx(content),\n content\n FROM newdata\n ON CONFLICT (datetime, id) DO UPDATE SET\n content = EXCLUDED.content\n WHERE items.content IS DISTINCT FROM EXCLUDED.content\n ;\n DELETE FROM items_staging;\n\n\n FOR p IN SELECT new_partitions.partition FROM new_partitions\n LOOP\n RAISE NOTICE 'Setting constraints for partition %.', p.partition;\n PERFORM partition_checks(p.partition);\n END LOOP;\n DROP TABLE IF EXISTS new_partitions;\n RETURN NULL;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.search(_search jsonb DEFAULT '{}'::jsonb)\n RETURNS jsonb\n LANGUAGE plpgsql\n SET jit TO 'off'\nAS $function$\nDECLARE\n searches searches%ROWTYPE;\n _where text;\n token_where text;\n full_where text;\n orderby text;\n query text;\n token_type text := substr(_search->>'token',1,4);\n _limit int := coalesce((_search->>'limit')::int, 10);\n curs refcursor;\n cntr int := 0;\n iter_record items%ROWTYPE;\n first_record items%ROWTYPE;\n last_record items%ROWTYPE;\n out_records jsonb := '[]'::jsonb;\n prev_query text;\n next text;\n prev_id text;\n has_next boolean := false;\n has_prev boolean := false;\n prev text;\n total_count bigint;\n context jsonb;\n collection jsonb;\n includes text[];\n excludes text[];\n exit_flag boolean := FALSE;\n batches int := 0;\n timer timestamptz := clock_timestamp();\n pstart timestamptz;\n pend timestamptz;\n pcurs refcursor;\n search_where search_wheres%ROWTYPE;\nBEGIN\nsearches := search_query(_search);\n_where := searches._where;\norderby := searches.orderby;\nsearch_where := where_stats(_where);\ntotal_count := coalesce(search_where.total_count, search_where.estimated_count);\n\n\nIF token_type='prev' THEN\n token_where := get_token_filter(_search, null::jsonb);\n orderby := sort_sqlorderby(_search, TRUE);\nEND IF;\nIF token_type='next' THEN\n token_where := get_token_filter(_search, null::jsonb);\nEND IF;\n\nfull_where := concat_ws(' AND ', _where, token_where);\nRAISE NOTICE 'FULL QUERY % %', full_where, clock_timestamp()-timer;\ntimer := clock_timestamp();\n\nCREATE TEMP TABLE results (content jsonb) ON COMMIT DROP;\n\n\nFOR query IN SELECT partition_queries(full_where, orderby, search_where.partitions) LOOP\n timer := clock_timestamp();\n query := format('%s LIMIT %s', query, _limit + 1);\n RAISE NOTICE 'Partition Query: %', query;\n batches := batches + 1;\n -- curs = create_cursor(query);\n OPEN curs FOR EXECUTE query;\n LOOP\n FETCH curs into iter_record;\n EXIT WHEN NOT FOUND;\n cntr := cntr + 1;\n last_record := iter_record;\n IF cntr = 1 THEN\n first_record := last_record;\n END IF;\n IF cntr <= _limit THEN\n INSERT INTO results (content) VALUES (last_record.content);\n -- out_records := out_records || last_record.content;\n\n ELSIF cntr > _limit THEN\n has_next := true;\n exit_flag := true;\n EXIT;\n END IF;\n END LOOP;\n CLOSE curs;\n RAISE NOTICE 'Query took %. Total Time %', clock_timestamp()-timer, ftime();\n timer := clock_timestamp();\n EXIT WHEN exit_flag;\nEND LOOP;\nRAISE NOTICE 'Scanned through % partitions.', batches;\n\nSELECT jsonb_agg(content) INTO out_records FROM results;\n\nDROP TABLE results;\n\n\n-- Flip things around if this was the result of a prev token query\nIF token_type='prev' THEN\n out_records := flip_jsonb_array(out_records);\n first_record := last_record;\nEND IF;\n\n-- If this query has a token, see if there is data before the first record\nIF _search ? 'token' THEN\n prev_query := format(\n 'SELECT 1 FROM items WHERE %s LIMIT 1',\n concat_ws(\n ' AND ',\n _where,\n trim(get_token_filter(_search, to_jsonb(first_record)))\n )\n );\n RAISE NOTICE 'Query to get previous record: % --- %', prev_query, first_record;\n EXECUTE prev_query INTO has_prev;\n IF FOUND and has_prev IS NOT NULL THEN\n RAISE NOTICE 'Query results from prev query: %', has_prev;\n has_prev := TRUE;\n END IF;\nEND IF;\nhas_prev := COALESCE(has_prev, FALSE);\n\nRAISE NOTICE 'token_type: %, has_next: %, has_prev: %', token_type, has_next, has_prev;\nIF has_prev THEN\n prev := out_records->0->>'id';\nEND IF;\nIF has_next OR token_type='prev' THEN\n next := out_records->-1->>'id';\nEND IF;\n\n\n-- include/exclude any fields following fields extension\nIF _search ? 'fields' THEN\n IF _search->'fields' ? 'exclude' THEN\n excludes=textarr(_search->'fields'->'exclude');\n END IF;\n IF _search->'fields' ? 'include' THEN\n includes=textarr(_search->'fields'->'include');\n IF array_length(includes, 1)>0 AND NOT 'id' = ANY (includes) THEN\n includes = includes || '{id}';\n END IF;\n END IF;\n SELECT jsonb_agg(filter_jsonb(row, includes, excludes)) INTO out_records FROM jsonb_array_elements(out_records) row;\nEND IF;\n\nIF context(_search->'conf') != 'off' THEN\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'matched', total_count,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\nELSE\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\nEND IF;\n\ncollection := jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb),\n 'next', next,\n 'prev', prev,\n 'context', context\n);\n\nRETURN collection;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.search_query(_search jsonb DEFAULT '{}'::jsonb, updatestats boolean DEFAULT false, _metadata jsonb DEFAULT '{}'::jsonb)\n RETURNS searches\n LANGUAGE plpgsql\nAS $function$\nDECLARE\n search searches%ROWTYPE;\n pexplain jsonb;\n t timestamptz;\n i interval;\nBEGIN\nSELECT * INTO search FROM searches\nWHERE hash=search_hash(_search, _metadata) FOR UPDATE;\n\n-- Calculate the where clause if not already calculated\nIF search._where IS NULL THEN\n search._where := cql_to_where(_search);\nEND IF;\n\n-- Calculate the order by clause if not already calculated\nIF search.orderby IS NULL THEN\n search.orderby := sort_sqlorderby(_search);\nEND IF;\n\nPERFORM where_stats(search._where, updatestats, _search->'conf');\n\nsearch.lastused := now();\nsearch.usecount := coalesce(search.usecount, 0) + 1;\nINSERT INTO searches (search, _where, orderby, lastused, usecount, metadata)\nVALUES (_search, search._where, search.orderby, search.lastused, search.usecount, _metadata)\nON CONFLICT (hash) DO\nUPDATE SET\n _where = EXCLUDED._where,\n orderby = EXCLUDED.orderby,\n lastused = EXCLUDED.lastused,\n usecount = EXCLUDED.usecount,\n metadata = EXCLUDED.metadata\nRETURNING * INTO search\n;\nRETURN search;\n\nEND;\n$function$\n;\n\nINSERT INTO pgstac_settings (name, value) VALUES\n ('context', 'off'),\n ('context_estimated_count', '100000'),\n ('context_estimated_cost', '1000000'),\n ('context_stats_ttl', '1 day'),\n ('default-filter-lang', 'cql2-json')\nON CONFLICT DO NOTHING\n;\n\n\nSELECT set_version('0.4.0');\n" }, { "path": "src/pgstac/migrations/pgstac.0.3.6.sql", "content": "CREATE EXTENSION IF NOT EXISTS postgis;\nCREATE SCHEMA IF NOT EXISTS pgstac;\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE TABLE migrations (\n version text PRIMARY KEY,\n datetime timestamptz DEFAULT clock_timestamp() NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION get_version() RETURNS text AS $$\n SELECT version FROM migrations ORDER BY datetime DESC, version DESC LIMIT 1;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION set_version(text) RETURNS text AS $$\n INSERT INTO migrations (version) VALUES ($1)\n ON CONFLICT DO NOTHING\n RETURNING version;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\n\nCREATE OR REPLACE FUNCTION get_setting(IN setting text, INOUT _default anynonarray = null::text ) AS $$\nDECLARE\n_type text;\nBEGIN\n SELECT pg_typeof(_default) INTO _type;\n IF _type = 'unknown' THEN _type='text'; END IF;\n EXECUTE format($q$\n SELECT COALESCE(\n CAST(current_setting($1,TRUE) AS %s),\n $2\n )\n $q$, _type)\n INTO _default\n USING setting, _default\n ;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION context() RETURNS text AS $$\n SELECT get_setting('pgstac.context','off'::text);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context_estimated_count() RETURNS int AS $$\n SELECT get_setting('pgstac.context_estimated_count', 100000::int);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context_estimated_cost() RETURNS float AS $$\n SELECT get_setting('pgstac.context_estimated_cost', 1000000::float);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context_stats_ttl() RETURNS interval AS $$\n SELECT get_setting('pgstac.context_stats_ttl', '1 day'::interval);\n$$ LANGUAGE SQL;\n\n\nCREATE OR REPLACE FUNCTION notice(VARIADIC text[]) RETURNS boolean AS $$\nDECLARE\ndebug boolean := current_setting('pgstac.debug', true);\nBEGIN\n IF debug THEN\n RAISE NOTICE 'NOTICE FROM FUNC: % >>>>> %', concat_ws(' | ', $1), clock_timestamp();\n RETURN TRUE;\n END IF;\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION empty_arr(ANYARRAY) RETURNS BOOLEAN AS $$\nSELECT CASE\n WHEN $1 IS NULL THEN TRUE\n WHEN cardinality($1)<1 THEN TRUE\nELSE FALSE\nEND;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_map_ident(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_ident(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_map_literal(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_literal(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION estimated_count(_where text) RETURNS bigint AS $$\nDECLARE\nrec record;\nrows bigint;\nBEGIN\n FOR rec in EXECUTE format(\n $q$\n EXPLAIN SELECT 1 FROM items WHERE %s\n $q$,\n _where)\n LOOP\n rows := substring(rec.\"QUERY PLAN\" FROM ' rows=([[:digit:]]+)');\n EXIT WHEN rows IS NOT NULL;\n END LOOP;\n\n RETURN rows;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION array_reverse(anyarray) RETURNS anyarray AS $$\nSELECT ARRAY(\n SELECT $1[i]\n FROM generate_subscripts($1,1) AS s(i)\n ORDER BY i DESC\n);\n$$ LANGUAGE 'sql' STRICT IMMUTABLE;\n/* converts a jsonb text array to a pg text[] array */\nCREATE OR REPLACE FUNCTION textarr(_js jsonb)\n RETURNS text[] AS $$\n SELECT\n CASE jsonb_typeof(_js)\n WHEN 'array' THEN ARRAY(SELECT jsonb_array_elements_text(_js))\n ELSE ARRAY[_js->>0]\n END\n;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION jsonb_paths (IN jdata jsonb, OUT path text[], OUT value jsonb) RETURNS\nSETOF RECORD AS $$\nwith recursive extract_all as\n(\n select\n ARRAY[key]::text[] as path,\n value\n FROM jsonb_each(jdata)\nunion all\n select\n path || coalesce(obj_key, (arr_key- 1)::text),\n coalesce(obj_value, arr_value)\n from extract_all\n left join lateral\n jsonb_each(case jsonb_typeof(value) when 'object' then value end)\n as o(obj_key, obj_value)\n on jsonb_typeof(value) = 'object'\n left join lateral\n jsonb_array_elements(case jsonb_typeof(value) when 'array' then value end)\n with ordinality as a(arr_value, arr_key)\n on jsonb_typeof(value) = 'array'\n where obj_key is not null or arr_key is not null\n)\nselect *\nfrom extract_all;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION jsonb_obj_paths (IN jdata jsonb, OUT path text[], OUT value jsonb) RETURNS\nSETOF RECORD AS $$\nwith recursive extract_all as\n(\n select\n ARRAY[key]::text[] as path,\n value\n FROM jsonb_each(jdata)\nunion all\n select\n path || obj_key,\n obj_value\n from extract_all\n left join lateral\n jsonb_each(case jsonb_typeof(value) when 'object' then value end)\n as o(obj_key, obj_value)\n on jsonb_typeof(value) = 'object'\n where obj_key is not null\n)\nselect *\nfrom extract_all;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION jsonb_val_paths (IN jdata jsonb, OUT path text[], OUT value jsonb) RETURNS\nSETOF RECORD AS $$\nSELECT * FROM jsonb_obj_paths(jdata) WHERE jsonb_typeof(value) not in ('object','array');\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION path_includes(IN path text[], IN includes text[]) RETURNS BOOLEAN AS $$\nWITH t AS (SELECT unnest(includes) i)\nSELECT EXISTS (\n SELECT 1 FROM t WHERE path @> string_to_array(trim(i), '.')\n);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION path_excludes(IN path text[], IN excludes text[]) RETURNS BOOLEAN AS $$\nWITH t AS (SELECT unnest(excludes) e)\nSELECT NOT EXISTS (\n SELECT 1 FROM t WHERE path @> string_to_array(trim(e), '.')\n);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION jsonb_obj_paths_filtered (\n IN jdata jsonb,\n IN includes text[] DEFAULT ARRAY[]::text[],\n IN excludes text[] DEFAULT ARRAY[]::text[],\n OUT path text[],\n OUT value jsonb\n) RETURNS\nSETOF RECORD AS $$\nSELECT path, value\nFROM jsonb_obj_paths(jdata)\nWHERE\n CASE WHEN cardinality(includes) > 0 THEN path_includes(path, includes) ELSE TRUE END\n AND\n path_excludes(path, excludes)\n\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION filter_jsonb(\n IN jdata jsonb,\n IN includes text[] DEFAULT ARRAY[]::text[],\n IN excludes text[] DEFAULT ARRAY[]::text[]\n) RETURNS jsonb AS $$\nDECLARE\nrec RECORD;\noutj jsonb := '{}'::jsonb;\ncreated_paths text[] := '{}'::text[];\nBEGIN\n\nIF empty_arr(includes) AND empty_arr(excludes) THEN\nRAISE NOTICE 'no filter';\n RETURN jdata;\nEND IF;\nFOR rec in\nSELECT * FROM jsonb_obj_paths_filtered(jdata, includes, excludes)\nWHERE jsonb_typeof(value) != 'object'\nLOOP\n IF array_length(rec.path,1)>1 THEN\n FOR i IN 1..(array_length(rec.path,1)-1) LOOP\n IF NOT array_to_string(rec.path[1:i],'.') = ANY (created_paths) THEN\n outj := jsonb_set(outj, rec.path[1:i],'{}', true);\n created_paths := created_paths || array_to_string(rec.path[1:i],'.');\n END IF;\n END LOOP;\n END IF;\n outj := jsonb_set(outj, rec.path, rec.value, true);\n created_paths := created_paths || array_to_string(rec.path,'.');\nEND LOOP;\nRETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION flip_jsonb_array(j jsonb) RETURNS jsonb AS $$\nSELECT jsonb_agg(value) FROM (SELECT value FROM jsonb_array_elements(j) WITH ORDINALITY ORDER BY ordinality DESC) as t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n/* Functions to create an iterable of cursors over partitions. */\nCREATE OR REPLACE FUNCTION create_cursor(q text) RETURNS refcursor AS $$\nDECLARE\n curs refcursor;\nBEGIN\n OPEN curs FOR EXECUTE q;\n RETURN curs;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDROP FUNCTION IF EXISTS partition_queries;\nCREATE OR REPLACE FUNCTION partition_queries(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN partitions text[] DEFAULT '{items}'\n) RETURNS SETOF text AS $$\nDECLARE\n partition_query text;\n query text;\n p text;\n cursors refcursor;\n dstart timestamptz;\n dend timestamptz;\n step interval := '10 weeks'::interval;\nBEGIN\n\nIF _orderby ILIKE 'datetime d%' THEN\n partitions := partitions;\nELSIF _orderby ILIKE 'datetime a%' THEN\n partitions := array_reverse(partitions);\nELSE\n query := format($q$\n SELECT * FROM items\n WHERE %s\n ORDER BY %s\n $q$, _where, _orderby\n );\n\n RETURN NEXT query;\n RETURN;\nEND IF;\nRAISE NOTICE 'PARTITIONS ---> %',partitions;\nIF cardinality(partitions) > 0 THEN\n FOREACH p IN ARRAY partitions\n --EXECUTE partition_query\n LOOP\n query := format($q$\n SELECT * FROM %I\n WHERE %s\n ORDER BY %s\n $q$,\n p,\n _where,\n _orderby\n );\n RETURN NEXT query;\n END LOOP;\nEND IF;\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION partition_cursor(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC'\n) RETURNS SETOF refcursor AS $$\nDECLARE\n partition_query text;\n query text;\n p record;\n cursors refcursor;\nBEGIN\nFOR query IN SELECT * FROM partition_queries(_where, _orderby) LOOP\n RETURN NEXT create_cursor(query);\nEND LOOP;\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION partition_count(\n IN _where text DEFAULT 'TRUE'\n) RETURNS bigint AS $$\nDECLARE\n partition_query text;\n query text;\n p record;\n subtotal bigint;\n total bigint := 0;\nBEGIN\npartition_query := format($q$\n SELECT partition, tstzrange\n FROM items_partitions\n ORDER BY tstzrange DESC;\n$q$);\nRAISE NOTICE 'Partition Query: %', partition_query;\nFOR p IN\n EXECUTE partition_query\nLOOP\n query := format($q$\n SELECT count(*) FROM items\n WHERE datetime BETWEEN %L AND %L AND %s\n $q$, lower(p.tstzrange), upper(p.tstzrange), _where\n );\n RAISE NOTICE 'Query %', query;\n RAISE NOTICE 'Partition %, Count %, Total %',p.partition, subtotal, total;\n EXECUTE query INTO subtotal;\n total := subtotal + total;\nEND LOOP;\nRETURN total;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION drop_partition_constraints(IN partition text) RETURNS VOID AS $$\nDECLARE\n q text;\n end_datetime_constraint text := concat(partition, '_end_datetime_constraint');\n collections_constraint text := concat(partition, '_collections_constraint');\nBEGIN\n q := format($q$\n ALTER TABLE %I\n DROP CONSTRAINT IF EXISTS %I,\n DROP CONSTRAINT IF EXISTS %I;\n $q$,\n partition,\n end_datetime_constraint,\n collections_constraint\n );\n\n EXECUTE q;\n RETURN;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDROP FUNCTION IF EXISTS partition_checks;\nCREATE OR REPLACE FUNCTION partition_checks(\n IN partition text,\n OUT min_datetime timestamptz,\n OUT max_datetime timestamptz,\n OUT min_end_datetime timestamptz,\n OUT max_end_datetime timestamptz,\n OUT collections text[],\n OUT cnt bigint\n) RETURNS RECORD AS $$\nDECLARE\nq text;\nend_datetime_constraint text := concat(partition, '_end_datetime_constraint');\ncollections_constraint text := concat(partition, '_collections_constraint');\nBEGIN\nRAISE NOTICE 'CREATING CONSTRAINTS FOR %', partition;\nq := format($q$\n SELECT\n min(datetime),\n max(datetime),\n min(end_datetime),\n max(end_datetime),\n array_agg(DISTINCT collection_id),\n count(*)\n FROM %I;\n $q$,\n partition\n);\nEXECUTE q INTO min_datetime, max_datetime, min_end_datetime, max_end_datetime, collections, cnt;\nRAISE NOTICE '% % % % % % %', min_datetime, max_datetime, min_end_datetime, max_end_datetime, collections, cnt, ftime();\nIF cnt IS NULL or cnt = 0 THEN\n RAISE NOTICE 'Partition % is empty, removing...', partition;\n q := format($q$\n DROP TABLE IF EXISTS %I;\n $q$, partition\n );\n EXECUTE q;\n RETURN;\nEND IF;\nRAISE NOTICE 'Running Constraint DDL %', ftime();\nq := format($q$\n ALTER TABLE %I\n DROP CONSTRAINT IF EXISTS %I,\n ADD CONSTRAINT %I\n check((end_datetime >= %L) AND (end_datetime <= %L)) NOT VALID,\n DROP CONSTRAINT IF EXISTS %I,\n ADD CONSTRAINT %I\n check((collection_id = ANY(%L))) NOT VALID;\n $q$,\n partition,\n end_datetime_constraint,\n end_datetime_constraint,\n min_end_datetime,\n max_end_datetime,\n collections_constraint,\n collections_constraint,\n collections,\n partition\n);\nRAISE NOTICE 'q: %', q;\n\nEXECUTE q;\nRAISE NOTICE 'Returning %', ftime();\nRETURN;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION validate_constraints() RETURNS VOID AS $$\nDECLARE\nq text;\nBEGIN\nFOR q IN\n SELECT FORMAT(\n 'ALTER TABLE %I.%I.%I VALIDATE CONSTRAINT %I;',\n current_database(),\n nsp.nspname,\n cls.relname,\n con.conname\n )\n FROM pg_constraint AS con\n JOIN pg_class AS cls\n ON con.conrelid = cls.oid\n JOIN pg_namespace AS nsp\n ON cls.relnamespace = nsp.oid\n WHERE convalidated IS FALSE\n AND nsp.nspname = 'pgstac'\nLOOP\n EXECUTE q;\nEND LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n/* looks for a geometry in a stac item first from geometry and falling back to bbox */\nCREATE OR REPLACE FUNCTION stac_geom(value jsonb) RETURNS geometry AS $$\nSELECT\n CASE\n WHEN value->>'geometry' IS NOT NULL THEN\n ST_GeomFromGeoJSON(value->>'geometry')\n WHEN value->>'bbox' IS NOT NULL THEN\n ST_MakeEnvelope(\n (value->'bbox'->>0)::float,\n (value->'bbox'->>1)::float,\n (value->'bbox'->>2)::float,\n (value->'bbox'->>3)::float,\n 4326\n )\n ELSE NULL\n END as geometry\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION stac_datetime(value jsonb) RETURNS timestamptz AS $$\nSELECT COALESCE(\n (value->'properties'->>'datetime')::timestamptz,\n (value->'properties'->>'start_datetime')::timestamptz\n);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_end_datetime(value jsonb) RETURNS timestamptz AS $$\nSELECT COALESCE(\n (value->'properties'->>'datetime')::timestamptz,\n (value->'properties'->>'end_datetime')::timestamptz\n);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\n\nCREATE OR REPLACE FUNCTION stac_daterange(value jsonb) RETURNS tstzrange AS $$\nSELECT tstzrange(stac_datetime(value),stac_end_datetime(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE TABLE IF NOT EXISTS collections (\n id VARCHAR GENERATED ALWAYS AS (content->>'id') STORED PRIMARY KEY,\n content JSONB\n);\n\nCREATE OR REPLACE FUNCTION create_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection(data jsonb) RETURNS VOID AS $$\nDECLARE\nout collections%ROWTYPE;\nBEGIN\n UPDATE collections SET content=data WHERE id = data->>'id' RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ON CONFLICT (id) DO\n UPDATE\n SET content=EXCLUDED.content\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION delete_collection(_id text) RETURNS VOID AS $$\nDECLARE\nout collections%ROWTYPE;\nBEGIN\n DELETE FROM collections WHERE id = _id RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION get_collection(id text) RETURNS jsonb AS $$\nSELECT content FROM collections\nWHERE id=$1\n;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION all_collections() RETURNS jsonb AS $$\nSELECT jsonb_agg(content) FROM collections;\n;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE TABLE items (\n id text NOT NULL,\n geometry geometry NOT NULL,\n collection_id text NOT NULL,\n datetime timestamptz NOT NULL,\n end_datetime timestamptz NOT NULL,\n properties jsonb NOT NULL,\n content JSONB NOT NULL\n)\nPARTITION BY RANGE (datetime)\n;\n\nCREATE OR REPLACE FUNCTION properties_idx (IN content jsonb) RETURNS jsonb AS $$\n with recursive extract_all as\n (\n select\n ARRAY[key]::text[] as path,\n ARRAY[key]::text[] as fullpath,\n value\n FROM jsonb_each(content->'properties')\n union all\n select\n CASE WHEN obj_key IS NOT NULL THEN path || obj_key ELSE path END,\n path || coalesce(obj_key, (arr_key- 1)::text),\n coalesce(obj_value, arr_value)\n from extract_all\n left join lateral\n jsonb_each(case jsonb_typeof(value) when 'object' then value end)\n as o(obj_key, obj_value)\n on jsonb_typeof(value) = 'object'\n left join lateral\n jsonb_array_elements(case jsonb_typeof(value) when 'array' then value end)\n with ordinality as a(arr_value, arr_key)\n on jsonb_typeof(value) = 'array'\n where obj_key is not null or arr_key is not null\n )\n , paths AS (\n select\n array_to_string(path, '.') as path,\n value\n FROM extract_all\n WHERE\n jsonb_typeof(value) NOT IN ('array','object')\n ), grouped AS (\n SELECT path, jsonb_agg(distinct value) vals FROM paths group by path\n ) SELECT coalesce(jsonb_object_agg(path, CASE WHEN jsonb_array_length(vals)=1 THEN vals->0 ELSE vals END) - '{datetime}'::text[], '{}'::jsonb) FROM grouped\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET JIT TO OFF;\n\nCREATE INDEX \"datetime_idx\" ON items (datetime);\nCREATE INDEX \"end_datetime_idx\" ON items (end_datetime);\nCREATE INDEX \"properties_idx\" ON items USING GIN (properties jsonb_path_ops);\nCREATE INDEX \"collection_idx\" ON items (collection_id);\nCREATE INDEX \"geometry_idx\" ON items USING GIST (geometry);\nCREATE UNIQUE INDEX \"items_id_datetime_idx\" ON items (datetime, id);\n\nALTER TABLE items ADD CONSTRAINT items_collections_fk FOREIGN KEY (collection_id) REFERENCES collections(id) ON DELETE CASCADE DEFERRABLE;\n\nCREATE OR REPLACE FUNCTION analyze_empty_partitions() RETURNS VOID AS $$\nDECLARE\n p text;\nBEGIN\n FOR p IN SELECT partition FROM all_items_partitions WHERE est_cnt = 0 LOOP\n EXECUTE format('ANALYZE %I;', p);\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION items_partition_name(timestamptz) RETURNS text AS $$\n SELECT to_char($1, '\"items_p\"IYYY\"w\"IW');\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION items_partition_exists(text) RETURNS boolean AS $$\n SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_class WHERE relname=$1);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION items_partition_exists(timestamptz) RETURNS boolean AS $$\n SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_class WHERE relname=items_partition_name($1));\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION items_partition_create_worker(partition text, partition_start timestamptz, partition_end timestamptz) RETURNS VOID AS $$\nDECLARE\n err_context text;\nBEGIN\n EXECUTE format(\n $f$\n CREATE TABLE IF NOT EXISTS %1$I PARTITION OF items\n FOR VALUES FROM (%2$L) TO (%3$L);\n CREATE UNIQUE INDEX IF NOT EXISTS %4$I ON %1$I (id);\n $f$,\n partition,\n partition_start,\n partition_end,\n concat(partition, '_id_pk')\n );\nEXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', partition;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH to pgstac, public;\n\nCREATE OR REPLACE FUNCTION items_partition_create(ts timestamptz) RETURNS text AS $$\nDECLARE\n partition text := items_partition_name(ts);\n partition_start timestamptz;\n partition_end timestamptz;\nBEGIN\n IF items_partition_exists(partition) THEN\n RETURN partition;\n END IF;\n partition_start := date_trunc('week', ts);\n partition_end := partition_start + '1 week'::interval;\n PERFORM items_partition_create_worker(partition, partition_start, partition_end);\n RAISE NOTICE 'partition: %', partition;\n RETURN partition;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION items_partition_create(st timestamptz, et timestamptz) RETURNS SETOF text AS $$\nWITH t AS (\n SELECT\n generate_series(\n date_trunc('week',st),\n date_trunc('week', et),\n '1 week'::interval\n ) w\n)\nSELECT items_partition_create(w) FROM t;\n$$ LANGUAGE SQL;\n\n\nCREATE UNLOGGED TABLE items_staging (\n content JSONB NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION items_staging_insert_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p record;\n _partitions text[];\nBEGIN\n CREATE TEMP TABLE new_partitions ON COMMIT DROP AS\n SELECT\n items_partition_name(stac_datetime(content)) as partition,\n date_trunc('week', min(stac_datetime(content))) as partition_start\n FROM newdata\n GROUP BY 1;\n\n -- set statslastupdated in cache to be old enough cache always regenerated\n\n SELECT array_agg(partition) INTO _partitions FROM new_partitions;\n UPDATE search_wheres\n SET\n statslastupdated = NULL\n WHERE partitions && _partitions\n ;\n\n FOR p IN SELECT new_partitions.partition, new_partitions.partition_start, new_partitions.partition_start + '1 week'::interval as partition_end FROM new_partitions\n LOOP\n RAISE NOTICE 'Getting partition % ready.', p.partition;\n IF NOT items_partition_exists(p.partition) THEN\n RAISE NOTICE 'Creating partition %.', p.partition;\n PERFORM items_partition_create_worker(p.partition, p.partition_start, p.partition_end);\n END IF;\n PERFORM drop_partition_constraints(p.partition);\n END LOOP;\n INSERT INTO items (id, geometry, collection_id, datetime, end_datetime, properties, content)\n SELECT\n content->>'id',\n stac_geom(content),\n content->>'collection',\n stac_datetime(content),\n stac_end_datetime(content),\n properties_idx(content),\n content\n FROM newdata\n ;\n DELETE FROM items_staging;\n\n\n FOR p IN SELECT new_partitions.partition FROM new_partitions\n LOOP\n RAISE NOTICE 'Setting constraints for partition %.', p.partition;\n PERFORM partition_checks(p.partition);\n END LOOP;\n DROP TABLE IF EXISTS new_partitions;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER items_staging_insert_trigger AFTER INSERT ON items_staging REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_insert_triggerfunc();\n\n\nCREATE UNLOGGED TABLE items_staging_ignore (\n content JSONB NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION items_staging_ignore_insert_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p record;\n _partitions text[];\nBEGIN\n CREATE TEMP TABLE new_partitions ON COMMIT DROP AS\n SELECT\n items_partition_name(stac_datetime(content)) as partition,\n date_trunc('week', min(stac_datetime(content))) as partition_start\n FROM newdata\n GROUP BY 1;\n\n -- set statslastupdated in cache to be old enough cache always regenerated\n\n SELECT array_agg(partition) INTO _partitions FROM new_partitions;\n UPDATE search_wheres\n SET\n statslastupdated = NULL\n WHERE partitions && _partitions\n ;\n\n FOR p IN SELECT new_partitions.partition, new_partitions.partition_start, new_partitions.partition_start + '1 week'::interval as partition_end FROM new_partitions\n LOOP\n RAISE NOTICE 'Getting partition % ready.', p.partition;\n IF NOT items_partition_exists(p.partition) THEN\n RAISE NOTICE 'Creating partition %.', p.partition;\n PERFORM items_partition_create_worker(p.partition, p.partition_start, p.partition_end);\n END IF;\n PERFORM drop_partition_constraints(p.partition);\n END LOOP;\n\n INSERT INTO items (id, geometry, collection_id, datetime, end_datetime, properties, content)\n SELECT\n content->>'id',\n stac_geom(content),\n content->>'collection',\n stac_datetime(content),\n stac_end_datetime(content),\n properties_idx(content),\n content\n FROM newdata\n ON CONFLICT (datetime, id) DO NOTHING\n ;\n DELETE FROM items_staging;\n\n\n FOR p IN SELECT new_partitions.partition FROM new_partitions\n LOOP\n RAISE NOTICE 'Setting constraints for partition %.', p.partition;\n PERFORM partition_checks(p.partition);\n END LOOP;\n DROP TABLE IF EXISTS new_partitions;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER items_staging_ignore_insert_trigger AFTER INSERT ON items_staging_ignore REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_ignore_insert_triggerfunc();\n\nCREATE UNLOGGED TABLE items_staging_upsert (\n content JSONB NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION items_staging_upsert_insert_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p record;\n _partitions text[];\nBEGIN\n CREATE TEMP TABLE new_partitions ON COMMIT DROP AS\n SELECT\n items_partition_name(stac_datetime(content)) as partition,\n date_trunc('week', min(stac_datetime(content))) as partition_start\n FROM newdata\n GROUP BY 1;\n\n -- set statslastupdated in cache to be old enough cache always regenerated\n\n SELECT array_agg(partition) INTO _partitions FROM new_partitions;\n UPDATE search_wheres\n SET\n statslastupdated = NULL\n WHERE partitions && _partitions\n ;\n\n FOR p IN SELECT new_partitions.partition, new_partitions.partition_start, new_partitions.partition_start + '1 week'::interval as partition_end FROM new_partitions\n LOOP\n RAISE NOTICE 'Getting partition % ready.', p.partition;\n IF NOT items_partition_exists(p.partition) THEN\n RAISE NOTICE 'Creating partition %.', p.partition;\n PERFORM items_partition_create_worker(p.partition, p.partition_start, p.partition_end);\n END IF;\n PERFORM drop_partition_constraints(p.partition);\n END LOOP;\n\n INSERT INTO items (id, geometry, collection_id, datetime, end_datetime, properties, content)\n SELECT\n content->>'id',\n stac_geom(content),\n content->>'collection',\n stac_datetime(content),\n stac_end_datetime(content),\n properties_idx(content),\n content\n FROM newdata\n ON CONFLICT (datetime, id) DO UPDATE SET\n content = EXCLUDED.content\n WHERE items.content IS DISTINCT FROM EXCLUDED.content\n ;\n DELETE FROM items_staging;\n\n\n FOR p IN SELECT new_partitions.partition FROM new_partitions\n LOOP\n RAISE NOTICE 'Setting constraints for partition %.', p.partition;\n PERFORM partition_checks(p.partition);\n END LOOP;\n DROP TABLE IF EXISTS new_partitions;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER items_staging_upsert_insert_trigger AFTER INSERT ON items_staging_upsert REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_upsert_insert_triggerfunc();\n\nCREATE OR REPLACE FUNCTION items_update_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\nBEGIN\n NEW.id := NEW.content->>'id';\n NEW.datetime := stac_datetime(NEW.content);\n NEW.end_datetime := stac_end_datetime(NEW.content);\n NEW.collection_id := NEW.content->>'collection';\n NEW.geometry := stac_geom(NEW.content);\n NEW.properties := properties_idx(NEW.content);\n IF TG_OP = 'UPDATE' AND NEW IS NOT DISTINCT FROM OLD THEN\n RETURN NULL;\n END IF;\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER items_update_trigger BEFORE UPDATE ON items\n FOR EACH ROW EXECUTE PROCEDURE items_update_triggerfunc();\n\n/*\nView to get a table of available items partitions\nwith date ranges\n*/\nDROP VIEW IF EXISTS all_items_partitions CASCADE;\nCREATE VIEW all_items_partitions AS\nWITH base AS\n(SELECT\n c.oid::pg_catalog.regclass::text as partition,\n pg_catalog.pg_get_expr(c.relpartbound, c.oid) as _constraint,\n regexp_matches(\n pg_catalog.pg_get_expr(c.relpartbound, c.oid),\n E'\\\\(''\\([0-9 :+-]*\\)''\\\\).*\\\\(''\\([0-9 :+-]*\\)''\\\\)'\n ) as t,\n reltuples::bigint as est_cnt\nFROM pg_catalog.pg_class c, pg_catalog.pg_inherits i\nWHERE c.oid = i.inhrelid AND i.inhparent = 'items'::regclass)\nSELECT partition, tstzrange(\n t[1]::timestamptz,\n t[2]::timestamptz\n), t[1]::timestamptz as pstart,\n t[2]::timestamptz as pend, est_cnt\nFROM base\nORDER BY 2 desc;\n\nCREATE OR REPLACE VIEW items_partitions AS\nSELECT * FROM all_items_partitions WHERE est_cnt>0;\n\nCREATE OR REPLACE FUNCTION get_item(_id text) RETURNS jsonb AS $$\n SELECT content FROM items WHERE id=_id;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION delete_item(_id text) RETURNS VOID AS $$\nDECLARE\nout items%ROWTYPE;\nBEGIN\n DELETE FROM items WHERE id = _id RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION create_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION update_item(data jsonb) RETURNS VOID AS $$\nDECLARE\n out items%ROWTYPE;\nBEGIN\n UPDATE items SET content=data WHERE id = data->>'id' RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION create_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION collection_bbox(id text) RETURNS jsonb AS $$\nSELECT (replace(replace(replace(st_extent(geometry)::text,'BOX(','[['),')',']]'),' ',','))::jsonb\nFROM items WHERE collection_id=$1;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION collection_temporal_extent(id text) RETURNS jsonb AS $$\nSELECT to_jsonb(array[array[min(datetime)::text, max(datetime)::text]])\nFROM items WHERE collection_id=$1;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection_extents() RETURNS VOID AS $$\nUPDATE collections SET\n content = content ||\n jsonb_build_object(\n 'extent', jsonb_build_object(\n 'spatial', jsonb_build_object(\n 'bbox', collection_bbox(collections.id)\n ),\n 'temporal', jsonb_build_object(\n 'interval', collection_temporal_extent(collections.id)\n )\n )\n )\n;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION items_path(\n IN dotpath text,\n OUT field text,\n OUT path text,\n OUT path_txt text,\n OUT jsonpath text,\n OUT eq text\n) RETURNS RECORD AS $$\nDECLARE\npath_elements text[];\nlast_element text;\nBEGIN\ndotpath := replace(trim(dotpath), 'properties.', '');\n\nIF dotpath = '' THEN\n RETURN;\nEND IF;\n\npath_elements := string_to_array(dotpath, '.');\njsonpath := NULL;\n\nIF path_elements[1] IN ('id','geometry','datetime') THEN\n field := path_elements[1];\n path_elements := path_elements[2:];\nELSIF path_elements[1] = 'collection' THEN\n field := 'collection_id';\n path_elements := path_elements[2:];\nELSIF path_elements[1] IN ('links', 'assets', 'stac_version', 'stac_extensions') THEN\n field := 'content';\nELSE\n field := 'content';\n path_elements := '{properties}'::text[] || path_elements;\nEND IF;\nIF cardinality(path_elements)<1 THEN\n path := field;\n path_txt := field;\n jsonpath := '$';\n eq := NULL;\n RETURN;\nEND IF;\n\n\nlast_element := path_elements[cardinality(path_elements)];\npath_elements := path_elements[1:cardinality(path_elements)-1];\njsonpath := concat(array_to_string('{$}'::text[] || array_map_ident(path_elements), '.'), '.', quote_ident(last_element));\npath_elements := array_map_literal(path_elements);\npath := format($F$ properties->%s $F$, quote_literal(dotpath));\npath_txt := format($F$ properties->>%s $F$, quote_literal(dotpath));\neq := format($F$ properties @? '$.%s[*] ? (@ == %%s) '$F$, quote_ident(dotpath));\n\nRAISE NOTICE 'ITEMS PATH -- % % % % %', field, path, path_txt, jsonpath, eq;\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION parse_dtrange(IN _indate jsonb, OUT _tstzrange tstzrange) AS $$\nWITH t AS (\n SELECT CASE\n WHEN jsonb_typeof(_indate) = 'array' THEN\n textarr(_indate)\n ELSE\n regexp_split_to_array(\n btrim(_indate::text,'\"'),\n '/'\n )\n END AS arr\n)\n, t1 AS (\n SELECT\n CASE\n WHEN array_upper(arr,1) = 1 OR arr[1] = '..' OR arr[1] IS NULL THEN '-infinity'::timestamptz\n ELSE arr[1]::timestamptz\n END AS st,\n CASE\n WHEN array_upper(arr,1) = 1 THEN arr[1]::timestamptz\n WHEN arr[2] = '..' OR arr[2] IS NULL THEN 'infinity'::timestamptz\n ELSE arr[2]::timestamptz\n END AS et\n FROM t\n)\nSELECT\n tstzrange(st,et)\nFROM t1;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION bbox_geom(_bbox jsonb) RETURNS geometry AS $$\nSELECT CASE jsonb_array_length(_bbox)\n WHEN 4 THEN\n ST_SetSRID(ST_MakeEnvelope(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float,\n (_bbox->>3)::float\n ),4326)\n WHEN 6 THEN\n ST_SetSRID(ST_3DMakeBox(\n ST_MakePoint(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float\n ),\n ST_MakePoint(\n (_bbox->>3)::float,\n (_bbox->>4)::float,\n (_bbox->>5)::float\n )\n ),4326)\n ELSE null END;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION cql_and_append(existing jsonb, newfilters jsonb) RETURNS jsonb AS $$\nSELECT CASE WHEN existing ? 'filter' AND newfilters IS NOT NULL THEN\n jsonb_build_object(\n 'and',\n jsonb_build_array(\n existing->'filter',\n newfilters\n )\n )\nELSE\n newfilters\nEND;\n$$ LANGUAGE SQL;\n\n\n-- ADDs base filters (ids, collections, datetime, bbox, intersects) that are\n-- added outside of the filter/query in the stac request\nCREATE OR REPLACE FUNCTION add_filters_to_cql(j jsonb) RETURNS jsonb AS $$\nDECLARE\nnewprop jsonb;\nnewprops jsonb := '[]'::jsonb;\nBEGIN\nIF j ? 'ids' THEN\n newprop := jsonb_build_object(\n 'in',\n jsonb_build_array(\n '{\"property\":\"id\"}'::jsonb,\n j->'ids'\n )\n );\n newprops := jsonb_insert(newprops, '{1}', newprop);\nEND IF;\nIF j ? 'collections' THEN\n newprop := jsonb_build_object(\n 'in',\n jsonb_build_array(\n '{\"property\":\"collection\"}'::jsonb,\n j->'collections'\n )\n );\n newprops := jsonb_insert(newprops, '{1}', newprop);\nEND IF;\n\nIF j ? 'datetime' THEN\n newprop := format(\n '{\"anyinteracts\":[{\"property\":\"datetime\"}, %s]}',\n j->'datetime'\n );\n newprops := jsonb_insert(newprops, '{1}', newprop);\nEND IF;\n\nIF j ? 'bbox' THEN\n newprop := format(\n '{\"intersects\":[{\"property\":\"geometry\"}, %s]}',\n j->'bbox'\n );\n newprops := jsonb_insert(newprops, '{1}', newprop);\nEND IF;\n\nIF j ? 'intersects' THEN\n newprop := format(\n '{\"intersects\":[{\"property\":\"geometry\"}, %s]}',\n j->'intersects'\n );\n newprops := jsonb_insert(newprops, '{1}', newprop);\nEND IF;\n\nRAISE NOTICE 'newprops: %', newprops;\n\nIF newprops IS NOT NULL AND jsonb_array_length(newprops) > 0 THEN\n return jsonb_set(\n j,\n '{filter}',\n cql_and_append(j, jsonb_build_object('and', newprops))\n ) - '{ids,collections,datetime,bbox,intersects}'::text[];\nEND IF;\n\nreturn j;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION query_to_cqlfilter(j jsonb) RETURNS jsonb AS $$\n-- Translates anything passed in through the deprecated \"query\" into equivalent CQL\nWITH t AS (\n SELECT key as property, value as ops\n FROM jsonb_each(j->'query')\n), t2 AS (\n SELECT property, (jsonb_each(ops)).*\n FROM t WHERE jsonb_typeof(ops) = 'object'\n UNION ALL\n SELECT property, 'eq', ops\n FROM t WHERE jsonb_typeof(ops) != 'object'\n), t3 AS (\nSELECT\n jsonb_strip_nulls(jsonb_build_object(\n 'and',\n jsonb_agg(\n jsonb_build_object(\n key,\n jsonb_build_array(\n jsonb_build_object('property',property),\n value\n )\n )\n )\n )) as qcql FROM t2\n)\nSELECT\n CASE WHEN qcql IS NOT NULL THEN\n jsonb_set(j, '{filter}', cql_and_append(j, qcql)) - 'query'\n ELSE j\n END\nFROM t3\n;\n$$ LANGUAGE SQL;\n\n\n\nCREATE OR REPLACE FUNCTION temporal_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\nll text := 'datetime';\nlh text := 'end_datetime';\nrrange tstzrange;\nrl text;\nrh text;\noutq text;\nBEGIN\nrrange := parse_dtrange(args->1);\nRAISE NOTICE 'Constructing temporal query OP: %, ARGS: %, RRANGE: %', op, args, rrange;\nop := lower(op);\nrl := format('%L::timestamptz', lower(rrange));\nrh := format('%L::timestamptz', upper(rrange));\noutq := CASE op\n WHEN 't_before' THEN 'lh < rl'\n WHEN 't_after' THEN 'll > rh'\n WHEN 't_meets' THEN 'lh = rl'\n WHEN 't_metby' THEN 'll = rh'\n WHEN 't_overlaps' THEN 'll < rl AND rl < lh < rh'\n WHEN 't_overlappedby' THEN 'rl < ll < rh AND lh > rh'\n WHEN 't_starts' THEN 'll = rl AND lh < rh'\n WHEN 't_startedby' THEN 'll = rl AND lh > rh'\n WHEN 't_during' THEN 'll > rl AND lh < rh'\n WHEN 't_contains' THEN 'll < rl AND lh > rh'\n WHEN 't_finishes' THEN 'll > rl AND lh = rh'\n WHEN 't_finishedby' THEN 'll < rl AND lh = rh'\n WHEN 't_equals' THEN 'll = rl AND lh = rh'\n WHEN 't_disjoint' THEN 'NOT (ll <= rh AND lh >= rl)'\n WHEN 't_intersects' THEN 'll <= rh AND lh >= rl'\n WHEN 'anyinteracts' THEN 'll <= rh AND lh >= rl'\nEND;\noutq := regexp_replace(outq, '\\mll\\M', ll);\noutq := regexp_replace(outq, '\\mlh\\M', lh);\noutq := regexp_replace(outq, '\\mrl\\M', rl);\noutq := regexp_replace(outq, '\\mrh\\M', rh);\noutq := format('(%s)', outq);\nRETURN outq;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION spatial_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\ngeom text;\nj jsonb := args->1;\nBEGIN\nop := lower(op);\nRAISE NOTICE 'Constructing spatial query OP: %, ARGS: %', op, args;\nIF op NOT IN ('s_equals','s_disjoint','s_touches','s_within','s_overlaps','s_crosses','s_intersects','intersects','s_contains') THEN\n RAISE EXCEPTION 'Spatial Operator % Not Supported', op;\nEND IF;\nop := regexp_replace(op, '^s_', 'st_');\nIF op = 'intersects' THEN\n op := 'st_intersects';\nEND IF;\n-- Convert geometry to WKB string\nIF j ? 'type' AND j ? 'coordinates' THEN\n geom := st_geomfromgeojson(j)::text;\nELSIF jsonb_typeof(j) = 'array' THEN\n geom := bbox_geom(j)::text;\nEND IF;\n\nRETURN format('%s(geometry, %L::geometry)', op, geom);\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n/* cql_query_op -- Parses a CQL query operation, recursing when necessary\n IN jsonb -- a subelement from a valid stac query\n IN text -- the operator being used on elements passed in\n RETURNS a SQL fragment to be used in a WHERE clause\n*/\nCREATE OR REPLACE FUNCTION cql_query_op(j jsonb, _op text DEFAULT NULL) RETURNS text AS $$\nDECLARE\njtype text := jsonb_typeof(j);\nop text := lower(_op);\nops jsonb :=\n '{\n \"eq\": \"%s = %s\",\n \"lt\": \"%s < %s\",\n \"lte\": \"%s <= %s\",\n \"gt\": \"%s > %s\",\n \"gte\": \"%s >= %s\",\n \"like\": \"%s LIKE %s\",\n \"+\": \"%s + %s\",\n \"-\": \"%s - %s\",\n \"*\": \"%s * %s\",\n \"/\": \"%s / %s\",\n \"in\": \"%s = ANY (%s)\",\n \"not\": \"NOT (%s)\",\n \"between\": \"%s BETWEEN %s AND %s\",\n \"lower\":\"lower(%s)\"\n }'::jsonb;\nret text;\nargs text[] := NULL;\n\nBEGIN\nRAISE NOTICE 'j: %, op: %, jtype: %', j, op, jtype;\n\n-- Set Lower Case on Both Arguments When Case Insensitive Flag Set\nIF op in ('eq','lt','lte','gt','gte','like') AND jsonb_typeof(j->2) = 'boolean' THEN\n IF (j->>2)::boolean THEN\n RETURN format(concat('(',ops->>op,')'), cql_query_op(jsonb_build_array(j->0), 'lower'), cql_query_op(jsonb_build_array(j->1), 'lower'));\n END IF;\nEND IF;\n\n-- Special Case when comparing a property in a jsonb field to a string or number using eq\n-- Allows to leverage GIN index on jsonb fields\nIF op = 'eq' THEN\n IF j->0 ? 'property'\n AND jsonb_typeof(j->1) IN ('number','string')\n AND (items_path(j->0->>'property')).eq IS NOT NULL\n THEN\n RETURN format((items_path(j->0->>'property')).eq, j->1);\n END IF;\nEND IF;\n\nIF op ilike 't_%' or op = 'anyinteracts' THEN\n RETURN temporal_op_query(op, j);\nEND IF;\n\nIF op ilike 's_%' or op = 'intersects' THEN\n RETURN spatial_op_query(op, j);\nEND IF;\n\n\nIF jtype = 'object' THEN\n RAISE NOTICE 'parsing object';\n IF j ? 'property' THEN\n -- Convert the property to be used as an identifier\n return (items_path(j->>'property')).path_txt;\n ELSIF _op IS NULL THEN\n -- Iterate to convert elements in an object where the operator has not been set\n -- Combining with AND\n SELECT\n array_to_string(array_agg(cql_query_op(e.value, e.key)), ' AND ')\n INTO ret\n FROM jsonb_each(j) e;\n RETURN ret;\n END IF;\nEND IF;\n\nIF jtype = 'string' THEN\n RETURN quote_literal(j->>0);\nEND IF;\n\nIF jtype ='number' THEN\n RETURN (j->>0)::numeric;\nEND IF;\n\nIF jtype = 'array' AND op IS NULL THEN\n RAISE NOTICE 'Parsing array into array arg. j: %', j;\n SELECT format($f$ '{%s}'::text[] $f$, string_agg(e,',')) INTO ret FROM jsonb_array_elements_text(j) e;\n RETURN ret;\nEND IF;\n\n\n-- If the type of the passed json is an array\n-- Calculate the arguments that will be passed to functions/operators\nIF jtype = 'array' THEN\n RAISE NOTICE 'Parsing array into args. j: %', j;\n -- If any argument is numeric, cast any text arguments to numeric\n IF j @? '$[*] ? (@.type() == \"number\")' THEN\n SELECT INTO args\n array_agg(concat('(',cql_query_op(e),')::numeric'))\n FROM jsonb_array_elements(j) e;\n ELSE\n SELECT INTO args\n array_agg(cql_query_op(e))\n FROM jsonb_array_elements(j) e;\n END IF;\n --RETURN args;\nEND IF;\nRAISE NOTICE 'ARGS after array cleaning: %', args;\n\nIF op IS NULL THEN\n RETURN args::text[];\nEND IF;\n\nIF args IS NULL OR cardinality(args) < 1 THEN\n RAISE NOTICE 'No Args';\n RETURN '';\nEND IF;\n\nIF op IN ('and','or') THEN\n SELECT\n CONCAT(\n '(',\n array_to_string(args, UPPER(CONCAT(' ',op,' '))),\n ')'\n ) INTO ret\n FROM jsonb_array_elements(j) e;\n RETURN ret;\nEND IF;\n\n-- If the op is in the ops json then run using the template in the json\nIF ops ? op THEN\n RAISE NOTICE 'ARGS: % MAPPED: %',args, array_map_literal(args);\n\n RETURN format(concat('(',ops->>op,')'), VARIADIC args);\nEND IF;\n\nRETURN j->>0;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n\n\nCREATE OR REPLACE FUNCTION cql_to_where(_search jsonb = '{}'::jsonb) RETURNS text AS $$\nDECLARE\nsearch jsonb := _search;\n_where text;\nBEGIN\nRAISE NOTICE 'SEARCH CQL 1: %', search;\n\n-- Convert any old style stac query to cql\nsearch := query_to_cqlfilter(search);\n\nRAISE NOTICE 'SEARCH CQL 2: %', search;\n\n-- Convert item,collection,datetime,bbox,intersects to cql\nsearch := add_filters_to_cql(search);\n\nRAISE NOTICE 'SEARCH CQL Final: %', search;\n_where := cql_query_op(search->'filter');\n\nIF trim(_where) = '' THEN\n _where := NULL;\nEND IF;\n_where := coalesce(_where, ' TRUE ');\nRETURN _where;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION parse_sort_dir(_dir text, reverse boolean default false) RETURNS text AS $$\nWITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n) SELECT\n CASE\n WHEN NOT reverse THEN d\n WHEN d = 'ASC' THEN 'DESC'\n WHEN d = 'DESC' THEN 'ASC'\n END\nFROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION sort_dir_to_op(_dir text, prev boolean default false) RETURNS text AS $$\nWITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n) SELECT\n CASE\n WHEN d = 'ASC' AND prev THEN '<='\n WHEN d = 'DESC' AND prev THEN '>='\n WHEN d = 'ASC' THEN '>='\n WHEN d = 'DESC' THEN '<='\n END\nFROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION field_orderby(p text) RETURNS text AS $$\nWITH t AS (\n SELECT\n replace(trim(substring(indexdef from 'btree \\((.*)\\)')),' ','')as s\n FROM pg_indexes WHERE schemaname='pgstac' AND tablename='items' AND indexdef ~* 'btree' AND indexdef ~* 'properties'\n) SELECT s FROM t WHERE strpos(s, lower(trim(p)))>0;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION sort_sqlorderby(\n _search jsonb DEFAULT NULL,\n reverse boolean DEFAULT FALSE\n) RETURNS text AS $$\nWITH sortby AS (\n SELECT coalesce(_search->'sortby','[{\"field\":\"datetime\", \"direction\":\"desc\"}]') as sort\n), withid AS (\n SELECT CASE\n WHEN sort @? '$[*] ? (@.field == \"id\")' THEN sort\n ELSE sort || '[{\"field\":\"id\", \"direction\":\"desc\"}]'::jsonb\n END as sort\n FROM sortby\n), withid_rows AS (\n SELECT jsonb_array_elements(sort) as value FROM withid\n),sorts AS (\n SELECT\n coalesce(field_orderby((items_path(value->>'field')).path_txt), (items_path(value->>'field')).path) as key,\n parse_sort_dir(value->>'direction', reverse) as dir\n FROM withid_rows\n)\nSELECT array_to_string(\n array_agg(concat(key, ' ', dir)),\n ', '\n) FROM sorts;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_sort_dir(sort_item jsonb) RETURNS text AS $$\nSELECT CASE WHEN sort_item->>'direction' ILIKE 'desc%' THEN 'DESC' ELSE 'ASC' END;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION get_token_filter(_search jsonb = '{}'::jsonb, token_rec jsonb DEFAULT NULL) RETURNS text AS $$\nDECLARE\n token_id text;\n filters text[] := '{}'::text[];\n prev boolean := TRUE;\n field text;\n dir text;\n sort record;\n orfilters text[] := '{}'::text[];\n andfilters text[] := '{}'::text[];\n output text;\n token_where text;\nBEGIN\n -- If no token provided return NULL\n IF token_rec IS NULL THEN\n IF NOT (_search ? 'token' AND\n (\n (_search->>'token' ILIKE 'prev:%')\n OR\n (_search->>'token' ILIKE 'next:%')\n )\n ) THEN\n RETURN NULL;\n END IF;\n prev := (_search->>'token' ILIKE 'prev:%');\n token_id := substr(_search->>'token', 6);\n SELECT to_jsonb(items) INTO token_rec FROM items WHERE id=token_id;\n END IF;\n RAISE NOTICE 'TOKEN ID: %', token_rec->'id';\n\n CREATE TEMP TABLE sorts (\n _row int GENERATED ALWAYS AS IDENTITY NOT NULL,\n _field text PRIMARY KEY,\n _dir text NOT NULL,\n _val text\n ) ON COMMIT DROP;\n\n -- Make sure we only have distinct columns to sort with taking the first one we get\n INSERT INTO sorts (_field, _dir)\n SELECT\n (items_path(value->>'field')).path,\n get_sort_dir(value)\n FROM\n jsonb_array_elements(coalesce(_search->'sortby','[{\"field\":\"datetime\",\"direction\":\"desc\"}]'))\n ON CONFLICT DO NOTHING\n ;\n RAISE NOTICE 'sorts 1: %', (SELECT jsonb_agg(to_json(sorts)) FROM sorts);\n -- Get the first sort direction provided. As the id is a primary key, if there are any\n -- sorts after id they won't do anything, so make sure that id is the last sort item.\n SELECT _dir INTO dir FROM sorts ORDER BY _row ASC LIMIT 1;\n IF EXISTS (SELECT 1 FROM sorts WHERE _field = 'id') THEN\n DELETE FROM sorts WHERE _row > (SELECT _row FROM sorts WHERE _field = 'id' ORDER BY _row ASC);\n ELSE\n INSERT INTO sorts (_field, _dir) VALUES ('id', dir);\n END IF;\n\n -- Add value from looked up item to the sorts table\n UPDATE sorts SET _val=quote_literal(token_rec->>_field);\n\n -- Check if all sorts are the same direction and use row comparison\n -- to filter\n RAISE NOTICE 'sorts 2: %', (SELECT jsonb_agg(to_json(sorts)) FROM sorts);\n\n IF (SELECT count(DISTINCT _dir) FROM sorts) = 1 THEN\n SELECT format(\n '(%s) %s (%s)',\n concat_ws(', ', VARIADIC array_agg(quote_ident(_field))),\n CASE WHEN (prev AND dir = 'ASC') OR (NOT prev AND dir = 'DESC') THEN '<' ELSE '>' END,\n concat_ws(', ', VARIADIC array_agg(_val))\n ) INTO output FROM sorts\n WHERE token_rec ? _field\n ;\n ELSE\n FOR sort IN SELECT * FROM sorts ORDER BY _row asc LOOP\n RAISE NOTICE 'SORT: %', sort;\n IF sort._row = 1 THEN\n orfilters := orfilters || format('(%s %s %s)',\n quote_ident(sort._field),\n CASE WHEN (prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC') THEN '<' ELSE '>' END,\n sort._val\n );\n ELSE\n orfilters := orfilters || format('(%s AND %s %s %s)',\n array_to_string(andfilters, ' AND '),\n quote_ident(sort._field),\n CASE WHEN (prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC') THEN '<' ELSE '>' END,\n sort._val\n );\n\n END IF;\n andfilters := andfilters || format('%s = %s',\n quote_ident(sort._field),\n sort._val\n );\n END LOOP;\n output := array_to_string(orfilters, ' OR ');\n END IF;\n DROP TABLE IF EXISTS sorts;\n token_where := concat('(',coalesce(output,'true'),')');\n IF trim(token_where) = '' THEN\n token_where := NULL;\n END IF;\n RAISE NOTICE 'TOKEN_WHERE: |%|',token_where;\n RETURN token_where;\n END;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION search_tohash(jsonb) RETURNS jsonb AS $$\n SELECT $1 - '{token,limit,context,includes,excludes}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION search_hash(jsonb, jsonb) RETURNS text AS $$\n SELECT md5(concat(search_tohash($1)::text,$2::text));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TABLE IF NOT EXISTS searches(\n hash text GENERATED ALWAYS AS (search_hash(search, metadata)) STORED PRIMARY KEY,\n search jsonb NOT NULL,\n _where text,\n orderby text,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n metadata jsonb DEFAULT '{}'::jsonb NOT NULL\n);\n\nCREATE TABLE IF NOT EXISTS search_wheres(\n _where text PRIMARY KEY,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n statslastupdated timestamptz,\n estimated_count bigint,\n estimated_cost float,\n time_to_estimate float,\n total_count bigint,\n time_to_count float,\n partitions text[]\n);\n\nCREATE INDEX IF NOT EXISTS search_wheres_partitions ON search_wheres USING GIN (partitions);\n\nCREATE OR REPLACE FUNCTION where_stats(inwhere text, updatestats boolean default false) RETURNS search_wheres AS $$\nDECLARE\n t timestamptz;\n i interval;\n explain_json jsonb;\n partitions text[];\n sw search_wheres%ROWTYPE;\nBEGIN\n SELECT * INTO sw FROM search_wheres WHERE _where=inwhere FOR UPDATE;\n\n -- Update statistics if explicitly set, if statistics do not exist, or statistics ttl has expired\n IF NOT updatestats THEN\n RAISE NOTICE 'Checking if update is needed.';\n RAISE NOTICE 'Stats Last Updated: %', sw.statslastupdated;\n RAISE NOTICE 'TTL: %, Age: %', context_stats_ttl(), now() - sw.statslastupdated;\n RAISE NOTICE 'Context: %, Existing Total: %', context(), sw.total_count;\n IF\n sw.statslastupdated IS NULL\n OR (now() - sw.statslastupdated) > context_stats_ttl()\n OR (context() != 'off' AND sw.total_count IS NULL)\n THEN\n updatestats := TRUE;\n END IF;\n END IF;\n\n sw._where := inwhere;\n sw.lastused := now();\n sw.usecount := coalesce(sw.usecount,0) + 1;\n\n IF NOT updatestats THEN\n UPDATE search_wheres SET\n lastused = sw.lastused,\n usecount = sw.usecount\n WHERE _where = inwhere\n RETURNING * INTO sw\n ;\n RETURN sw;\n END IF;\n -- Use explain to get estimated count/cost and a list of the partitions that would be hit by the query\n t := clock_timestamp();\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s', inwhere)\n INTO explain_json;\n RAISE NOTICE 'Time for just the explain: %', clock_timestamp() - t;\n WITH t AS (\n SELECT j->>0 as p FROM\n jsonb_path_query(\n explain_json,\n 'strict $.**.\"Relation Name\" ? (@ != null)'\n ) j\n ), ordered AS (\n SELECT p FROM t ORDER BY p DESC\n -- SELECT p FROM t JOIN items_partitions\n -- ON (t.p = items_partitions.partition)\n -- ORDER BY pstart DESC\n )\n SELECT array_agg(p) INTO partitions FROM ordered;\n i := clock_timestamp() - t;\n RAISE NOTICE 'Time for explain + join: %', clock_timestamp() - t;\n\n\n\n sw.statslastupdated := now();\n sw.estimated_count := explain_json->0->'Plan'->'Plan Rows';\n sw.estimated_cost := explain_json->0->'Plan'->'Total Cost';\n sw.time_to_estimate := extract(epoch from i);\n sw.partitions := partitions;\n\n -- Do a full count of rows if context is set to on or if auto is set and estimates are low enough\n IF\n context() = 'on'\n OR\n ( context() = 'auto' AND\n (\n sw.estimated_count < context_estimated_count()\n OR\n sw.estimated_cost < context_estimated_cost()\n )\n )\n THEN\n t := clock_timestamp();\n RAISE NOTICE 'Calculating actual count...';\n EXECUTE format(\n 'SELECT count(*) FROM items WHERE %s',\n inwhere\n ) INTO sw.total_count;\n i := clock_timestamp() - t;\n RAISE NOTICE 'Actual Count: % -- %', sw.total_count, i;\n sw.time_to_count := extract(epoch FROM i);\n ELSE\n sw.total_count := NULL;\n sw.time_to_count := NULL;\n END IF;\n\n\n INSERT INTO search_wheres SELECT sw.*\n ON CONFLICT (_where)\n DO UPDATE\n SET\n lastused = sw.lastused,\n usecount = sw.usecount,\n statslastupdated = sw.statslastupdated,\n estimated_count = sw.estimated_count,\n estimated_cost = sw.estimated_cost,\n time_to_estimate = sw.time_to_estimate,\n partitions = sw.partitions,\n total_count = sw.total_count,\n time_to_count = sw.time_to_count\n ;\n RETURN sw;\nEND;\n$$ LANGUAGE PLPGSQL ;\n\n\n\nCREATE OR REPLACE FUNCTION items_count(_where text) RETURNS bigint AS $$\nDECLARE\ncnt bigint;\nBEGIN\nEXECUTE format('SELECT count(*) FROM items WHERE %s', _where) INTO cnt;\nRETURN cnt;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n\n\n\nDROP FUNCTION IF EXISTS search_query;\nCREATE OR REPLACE FUNCTION search_query(\n _search jsonb = '{}'::jsonb,\n updatestats boolean = false,\n _metadata jsonb = '{}'::jsonb\n) RETURNS searches AS $$\nDECLARE\n search searches%ROWTYPE;\n pexplain jsonb;\n t timestamptz;\n i interval;\nBEGIN\nSELECT * INTO search FROM searches\nWHERE hash=search_hash(_search, _metadata) FOR UPDATE;\n\n-- Calculate the where clause if not already calculated\nIF search._where IS NULL THEN\n search._where := cql_to_where(_search);\nEND IF;\n\n-- Calculate the order by clause if not already calculated\nIF search.orderby IS NULL THEN\n search.orderby := sort_sqlorderby(_search);\nEND IF;\n\nPERFORM where_stats(search._where, updatestats);\n\nsearch.lastused := now();\nsearch.usecount := coalesce(search.usecount, 0) + 1;\nINSERT INTO searches (search, _where, orderby, lastused, usecount, metadata)\nVALUES (_search, search._where, search.orderby, search.lastused, search.usecount, _metadata)\nON CONFLICT (hash) DO\nUPDATE SET\n _where = EXCLUDED._where,\n orderby = EXCLUDED.orderby,\n lastused = EXCLUDED.lastused,\n usecount = EXCLUDED.usecount,\n metadata = EXCLUDED.metadata\nRETURNING * INTO search\n;\nRETURN search;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n\nCREATE OR REPLACE FUNCTION search(_search jsonb = '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n searches searches%ROWTYPE;\n _where text;\n token_where text;\n full_where text;\n orderby text;\n query text;\n token_type text := substr(_search->>'token',1,4);\n _limit int := coalesce((_search->>'limit')::int, 10);\n curs refcursor;\n cntr int := 0;\n iter_record items%ROWTYPE;\n first_record items%ROWTYPE;\n last_record items%ROWTYPE;\n out_records jsonb := '[]'::jsonb;\n prev_query text;\n next text;\n prev_id text;\n has_next boolean := false;\n has_prev boolean := false;\n prev text;\n total_count bigint;\n context jsonb;\n collection jsonb;\n includes text[];\n excludes text[];\n exit_flag boolean := FALSE;\n batches int := 0;\n timer timestamptz := clock_timestamp();\n pstart timestamptz;\n pend timestamptz;\n pcurs refcursor;\n search_where search_wheres%ROWTYPE;\nBEGIN\nsearches := search_query(_search);\n_where := searches._where;\norderby := searches.orderby;\nsearch_where := where_stats(_where);\ntotal_count := coalesce(search_where.total_count, search_where.estimated_count);\n\n\nIF token_type='prev' THEN\n token_where := get_token_filter(_search, null::jsonb);\n orderby := sort_sqlorderby(_search, TRUE);\nEND IF;\nIF token_type='next' THEN\n token_where := get_token_filter(_search, null::jsonb);\nEND IF;\n\nfull_where := concat_ws(' AND ', _where, token_where);\nRAISE NOTICE 'FULL QUERY % %', full_where, clock_timestamp()-timer;\ntimer := clock_timestamp();\n\nCREATE TEMP TABLE results (content jsonb) ON COMMIT DROP;\n\n\nFOR query IN SELECT partition_queries(full_where, orderby, search_where.partitions) LOOP\n timer := clock_timestamp();\n query := format('%s LIMIT %s', query, _limit + 1);\n RAISE NOTICE 'Partition Query: %', query;\n batches := batches + 1;\n -- curs = create_cursor(query);\n OPEN curs FOR EXECUTE query;\n LOOP\n FETCH curs into iter_record;\n EXIT WHEN NOT FOUND;\n cntr := cntr + 1;\n last_record := iter_record;\n IF cntr = 1 THEN\n first_record := last_record;\n END IF;\n IF cntr <= _limit THEN\n INSERT INTO results (content) VALUES (last_record.content);\n -- out_records := out_records || last_record.content;\n\n ELSIF cntr > _limit THEN\n has_next := true;\n exit_flag := true;\n EXIT;\n END IF;\n END LOOP;\n CLOSE curs;\n RAISE NOTICE 'Query took %. Total Time %', clock_timestamp()-timer, ftime();\n timer := clock_timestamp();\n EXIT WHEN exit_flag;\nEND LOOP;\nRAISE NOTICE 'Scanned through % partitions.', batches;\n\nSELECT jsonb_agg(content) INTO out_records FROM results;\n\nDROP TABLE results;\n\n\n-- Flip things around if this was the result of a prev token query\nIF token_type='prev' THEN\n out_records := flip_jsonb_array(out_records);\n first_record := last_record;\nEND IF;\n\n-- If this query has a token, see if there is data before the first record\nIF _search ? 'token' THEN\n prev_query := format(\n 'SELECT 1 FROM items WHERE %s LIMIT 1',\n concat_ws(\n ' AND ',\n _where,\n trim(get_token_filter(_search, to_jsonb(first_record)))\n )\n );\n RAISE NOTICE 'Query to get previous record: % --- %', prev_query, first_record;\n EXECUTE prev_query INTO has_prev;\n IF FOUND and has_prev IS NOT NULL THEN\n RAISE NOTICE 'Query results from prev query: %', has_prev;\n has_prev := TRUE;\n END IF;\nEND IF;\nhas_prev := COALESCE(has_prev, FALSE);\n\nRAISE NOTICE 'token_type: %, has_next: %, has_prev: %', token_type, has_next, has_prev;\nIF has_prev THEN\n prev := out_records->0->>'id';\nEND IF;\nIF has_next OR token_type='prev' THEN\n next := out_records->-1->>'id';\nEND IF;\n\n\n-- include/exclude any fields following fields extension\nIF _search ? 'fields' THEN\n IF _search->'fields' ? 'exclude' THEN\n excludes=textarr(_search->'fields'->'exclude');\n END IF;\n IF _search->'fields' ? 'include' THEN\n includes=textarr(_search->'fields'->'include');\n IF array_length(includes, 1)>0 AND NOT 'id' = ANY (includes) THEN\n includes = includes || '{id}';\n END IF;\n END IF;\n SELECT jsonb_agg(filter_jsonb(row, includes, excludes)) INTO out_records FROM jsonb_array_elements(out_records) row;\nEND IF;\n\nIF context() != 'off' THEN\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'matched', total_count,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\nELSE\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\nEND IF;\n\ncollection := jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb),\n 'next', next,\n 'prev', prev,\n 'context', context\n);\n\nRETURN collection;\nEND;\n$$ LANGUAGE PLPGSQL\nSET jit TO off\n;\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION tileenvelope(zoom int, x int, y int) RETURNS geometry AS $$\nWITH t AS (\n SELECT\n 20037508.3427892 as merc_max,\n -20037508.3427892 as merc_min,\n (2 * 20037508.3427892) / (2 ^ zoom) as tile_size\n)\nSELECT st_makeenvelope(\n merc_min + (tile_size * x),\n merc_max - (tile_size * (y + 1)),\n merc_min + (tile_size * (x + 1)),\n merc_max - (tile_size * y),\n 3857\n) FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;DROP FUNCTION IF EXISTS mercgrid;\n\n\nCREATE OR REPLACE FUNCTION ftime() RETURNS interval as $$\nSELECT age(clock_timestamp(), transaction_timestamp());\n$$ LANGUAGE SQL;\nSET SEARCH_PATH to pgstac, public;\n\nDROP FUNCTION IF EXISTS geometrysearch;\nCREATE OR REPLACE FUNCTION geometrysearch(\n IN geom geometry,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE, -- Return as soon as the passed in geometry is full covered\n IN skipcovered boolean DEFAULT TRUE -- Skip any items that would show up completely under the previous items\n) RETURNS jsonb AS $$\nDECLARE\n search searches%ROWTYPE;\n curs refcursor;\n _where text;\n query text;\n iter_record items%ROWTYPE;\n out_records jsonb[] := '{}'::jsonb[];\n exit_flag boolean := FALSE;\n counter int := 1;\n scancounter int := 1;\n remaining_limit int := _scanlimit;\n tilearea float;\n unionedgeom geometry;\n clippedgeom geometry;\n unionedgeom_area float := 0;\n prev_area float := 0;\n excludes text[];\n includes text[];\n\nBEGIN\n -- If skipcovered is true then you will always want to exit when the passed in geometry is full\n IF skipcovered THEN\n exitwhenfull := TRUE;\n END IF;\n\n SELECT * INTO search FROM searches WHERE hash=queryhash;\n\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Search with Query Hash % Not Found', queryhash;\n END IF;\n\n tilearea := st_area(geom);\n _where := format('%s AND st_intersects(geometry, %L::geometry)', search._where, geom);\n\n IF fields IS NOT NULL THEN\n IF fields ? 'fields' THEN\n fields := fields->'fields';\n END IF;\n IF fields ? 'exclude' THEN\n excludes=textarr(fields->'exclude');\n END IF;\n IF fields ? 'include' THEN\n includes=textarr(fields->'include');\n IF array_length(includes, 1)>0 AND NOT 'id' = ANY (includes) THEN\n includes = includes || '{id}';\n END IF;\n END IF;\n END IF;\n RAISE NOTICE 'fields: %, includes: %, excludes: %', fields, includes, excludes;\n\n FOR query IN SELECT * FROM partition_queries(_where, search.orderby) LOOP\n query := format('%s LIMIT %L', query, remaining_limit);\n RAISE NOTICE '%', query;\n curs = create_cursor(query);\n LOOP\n FETCH curs INTO iter_record;\n EXIT WHEN NOT FOUND;\n IF exitwhenfull OR skipcovered THEN -- If we are not using exitwhenfull or skipcovered, we do not need to do expensive geometry operations\n clippedgeom := st_intersection(geom, iter_record.geometry);\n\n IF unionedgeom IS NULL THEN\n unionedgeom := clippedgeom;\n ELSE\n unionedgeom := st_union(unionedgeom, clippedgeom);\n END IF;\n\n unionedgeom_area := st_area(unionedgeom);\n\n IF skipcovered AND prev_area = unionedgeom_area THEN\n scancounter := scancounter + 1;\n CONTINUE;\n END IF;\n\n prev_area := unionedgeom_area;\n\n RAISE NOTICE '% % % %', unionedgeom_area/tilearea, counter, scancounter, ftime();\n END IF;\n\n IF fields IS NOT NULL THEN\n out_records := out_records || filter_jsonb(iter_record.content, includes, excludes);\n ELSE\n out_records := out_records || iter_record.content;\n END IF;\n IF counter >= _limit\n OR scancounter > _scanlimit\n OR ftime() > _timelimit\n OR (exitwhenfull AND unionedgeom_area >= tilearea)\n THEN\n exit_flag := TRUE;\n EXIT;\n END IF;\n counter := counter + 1;\n scancounter := scancounter + 1;\n\n END LOOP;\n EXIT WHEN exit_flag;\n remaining_limit := _scanlimit - scancounter;\n END LOOP;\n\n RETURN jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', array_to_json(out_records)::jsonb\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDROP FUNCTION IF EXISTS geojsonsearch;\nCREATE OR REPLACE FUNCTION geojsonsearch(\n IN geojson jsonb,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_geomfromgeojson(geojson),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS xyzsearch;\nCREATE OR REPLACE FUNCTION xyzsearch(\n IN _x int,\n IN _y int,\n IN _z int,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_transform(tileenvelope(_z, _x, _y), 4326),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\nSELECT set_version('0.3.6');\n" }, { "path": "src/pgstac/migrations/pgstac.0.4.0-0.4.1.sql", "content": "SET SEARCH_PATH to pgstac, public;\nalter table \"pgstac\".\"search_wheres\" drop constraint \"search_wheres_pkey\";\n\ndrop index if exists \"pgstac\".\"search_wheres_pkey\";\n\nalter table \"pgstac\".\"search_wheres\" add column \"id\" bigint generated always as identity not null;\n\nCREATE UNIQUE INDEX search_wheres_where ON pgstac.search_wheres USING btree (md5(_where));\n\nCREATE UNIQUE INDEX search_wheres_pkey ON pgstac.search_wheres USING btree (id);\n\nalter table \"pgstac\".\"search_wheres\" add constraint \"search_wheres_pkey\" PRIMARY KEY using index \"search_wheres_pkey\";\n\nset check_function_bodies = off;\n\nCREATE OR REPLACE FUNCTION pgstac.cql_to_where(_search jsonb DEFAULT '{}'::jsonb)\n RETURNS text\n LANGUAGE plpgsql\nAS $function$\nDECLARE\nfilterlang text;\nsearch jsonb := _search;\n_where text;\nBEGIN\n\nRAISE NOTICE 'SEARCH CQL Final: %', search;\nfilterlang := COALESCE(\n search->>'filter-lang',\n get_setting('default-filter-lang', _search->'conf')\n);\n\nIF filterlang = 'cql-json' THEN\n search := query_to_cqlfilter(search);\n search := add_filters_to_cql(search);\n _where := cql_query_op(search->'filter');\nELSE\n _where := cql2_query(search->'filter');\nEND IF;\n\nIF trim(_where) = '' THEN\n _where := NULL;\nEND IF;\n_where := coalesce(_where, ' TRUE ');\nRETURN _where;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.where_stats(inwhere text, updatestats boolean DEFAULT false, conf jsonb DEFAULT NULL::jsonb)\n RETURNS search_wheres\n LANGUAGE plpgsql\nAS $function$\nDECLARE\n t timestamptz;\n i interval;\n explain_json jsonb;\n partitions text[];\n sw search_wheres%ROWTYPE;\nBEGIN\n SELECT * INTO sw FROM search_wheres WHERE _where=inwhere FOR UPDATE;\n\n -- Update statistics if explicitly set, if statistics do not exist, or statistics ttl has expired\n IF NOT updatestats THEN\n RAISE NOTICE 'Checking if update is needed.';\n RAISE NOTICE 'Stats Last Updated: %', sw.statslastupdated;\n RAISE NOTICE 'TTL: %, Age: %', context_stats_ttl(conf), now() - sw.statslastupdated;\n RAISE NOTICE 'Context: %, Existing Total: %', context(conf), sw.total_count;\n IF\n sw.statslastupdated IS NULL\n OR (now() - sw.statslastupdated) > context_stats_ttl(conf)\n OR (context(conf) != 'off' AND sw.total_count IS NULL)\n THEN\n updatestats := TRUE;\n END IF;\n END IF;\n\n sw._where := inwhere;\n sw.lastused := now();\n sw.usecount := coalesce(sw.usecount,0) + 1;\n\n IF NOT updatestats THEN\n UPDATE search_wheres SET\n lastused = sw.lastused,\n usecount = sw.usecount\n WHERE _where = inwhere\n RETURNING * INTO sw\n ;\n RETURN sw;\n END IF;\n -- Use explain to get estimated count/cost and a list of the partitions that would be hit by the query\n t := clock_timestamp();\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s', inwhere)\n INTO explain_json;\n RAISE NOTICE 'Time for just the explain: %', clock_timestamp() - t;\n WITH t AS (\n SELECT j->>0 as p FROM\n jsonb_path_query(\n explain_json,\n 'strict $.**.\"Relation Name\" ? (@ != null)'\n ) j\n ), ordered AS (\n SELECT p FROM t ORDER BY p DESC\n -- SELECT p FROM t JOIN items_partitions\n -- ON (t.p = items_partitions.partition)\n -- ORDER BY pstart DESC\n )\n SELECT array_agg(p) INTO partitions FROM ordered;\n i := clock_timestamp() - t;\n RAISE NOTICE 'Time for explain + join: %', clock_timestamp() - t;\n\n\n\n sw.statslastupdated := now();\n sw.estimated_count := explain_json->0->'Plan'->'Plan Rows';\n sw.estimated_cost := explain_json->0->'Plan'->'Total Cost';\n sw.time_to_estimate := extract(epoch from i);\n sw.partitions := partitions;\n\n -- Do a full count of rows if context is set to on or if auto is set and estimates are low enough\n IF\n context(conf) = 'on'\n OR\n ( context(conf) = 'auto' AND\n (\n sw.estimated_count < context_estimated_count(conf)\n OR\n sw.estimated_cost < context_estimated_cost(conf)\n )\n )\n THEN\n t := clock_timestamp();\n RAISE NOTICE 'Calculating actual count...';\n EXECUTE format(\n 'SELECT count(*) FROM items WHERE %s',\n inwhere\n ) INTO sw.total_count;\n i := clock_timestamp() - t;\n RAISE NOTICE 'Actual Count: % -- %', sw.total_count, i;\n sw.time_to_count := extract(epoch FROM i);\n ELSE\n sw.total_count := NULL;\n sw.time_to_count := NULL;\n END IF;\n\n\n INSERT INTO search_wheres\n (_where, lastused, usecount, statslastupdated, estimated_count, estimated_cost, time_to_estimate, partitions, total_count, time_to_count)\n SELECT sw._where, sw.lastused, sw.usecount, sw.statslastupdated, sw.estimated_count, sw.estimated_cost, sw.time_to_estimate, sw.partitions, sw.total_count, sw.time_to_count\n ON CONFLICT ((md5(_where)))\n DO UPDATE\n SET\n lastused = sw.lastused,\n usecount = sw.usecount,\n statslastupdated = sw.statslastupdated,\n estimated_count = sw.estimated_count,\n estimated_cost = sw.estimated_cost,\n time_to_estimate = sw.time_to_estimate,\n partitions = sw.partitions,\n total_count = sw.total_count,\n time_to_count = sw.time_to_count\n ;\n RETURN sw;\nEND;\n$function$\n;\n\n\n\nSELECT set_version('0.4.1');\n" }, { "path": "src/pgstac/migrations/pgstac.0.4.0.sql", "content": "CREATE EXTENSION IF NOT EXISTS postgis;\nCREATE SCHEMA IF NOT EXISTS pgstac;\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE TABLE migrations (\n version text PRIMARY KEY,\n datetime timestamptz DEFAULT clock_timestamp() NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION get_version() RETURNS text AS $$\n SELECT version FROM migrations ORDER BY datetime DESC, version DESC LIMIT 1;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION set_version(text) RETURNS text AS $$\n INSERT INTO migrations (version) VALUES ($1)\n ON CONFLICT DO NOTHING\n RETURNING version;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\n\nCREATE TABLE IF NOT EXISTS pgstac_settings (\n name text PRIMARY KEY,\n value text NOT NULL\n);\n\nINSERT INTO pgstac_settings (name, value) VALUES\n ('context', 'off'),\n ('context_estimated_count', '100000'),\n ('context_estimated_cost', '1000000'),\n ('context_stats_ttl', '1 day'),\n ('default-filter-lang', 'cql2-json')\nON CONFLICT DO NOTHING\n;\n\n\nCREATE OR REPLACE FUNCTION get_setting(IN _setting text, IN conf jsonb DEFAULT NULL) RETURNS text AS $$\nSELECT COALESCE(\n conf->>_setting,\n current_setting(concat('pgstac.',_setting), TRUE),\n (SELECT value FROM pgstac_settings WHERE name=_setting)\n);\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context();\nCREATE OR REPLACE FUNCTION context(conf jsonb DEFAULT NULL) RETURNS text AS $$\n SELECT get_setting('context', conf);\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_estimated_count();\nCREATE OR REPLACE FUNCTION context_estimated_count(conf jsonb DEFAULT NULL) RETURNS int AS $$\n SELECT get_setting('context_estimated_count', conf)::int;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_estimated_cost();\nCREATE OR REPLACE FUNCTION context_estimated_cost(conf jsonb DEFAULT NULL) RETURNS float AS $$\n SELECT get_setting('context_estimated_cost', conf)::float;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_stats_ttl();\nCREATE OR REPLACE FUNCTION context_stats_ttl(conf jsonb DEFAULT NULL) RETURNS interval AS $$\n SELECT get_setting('context_stats_ttl', conf)::interval;\n$$ LANGUAGE SQL;\n\n\nCREATE OR REPLACE FUNCTION notice(VARIADIC text[]) RETURNS boolean AS $$\nDECLARE\ndebug boolean := current_setting('pgstac.debug', true);\nBEGIN\n IF debug THEN\n RAISE NOTICE 'NOTICE FROM FUNC: % >>>>> %', concat_ws(' | ', $1), clock_timestamp();\n RETURN TRUE;\n END IF;\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION empty_arr(ANYARRAY) RETURNS BOOLEAN AS $$\nSELECT CASE\n WHEN $1 IS NULL THEN TRUE\n WHEN cardinality($1)<1 THEN TRUE\nELSE FALSE\nEND;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_map_ident(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_ident(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_map_literal(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_literal(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION estimated_count(_where text) RETURNS bigint AS $$\nDECLARE\nrec record;\nrows bigint;\nBEGIN\n FOR rec in EXECUTE format(\n $q$\n EXPLAIN SELECT 1 FROM items WHERE %s\n $q$,\n _where)\n LOOP\n rows := substring(rec.\"QUERY PLAN\" FROM ' rows=([[:digit:]]+)');\n EXIT WHEN rows IS NOT NULL;\n END LOOP;\n\n RETURN rows;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION array_reverse(anyarray) RETURNS anyarray AS $$\nSELECT ARRAY(\n SELECT $1[i]\n FROM generate_subscripts($1,1) AS s(i)\n ORDER BY i DESC\n);\n$$ LANGUAGE 'sql' STRICT IMMUTABLE;\n/* converts a jsonb text array to a pg text[] array */\nCREATE OR REPLACE FUNCTION textarr(_js jsonb)\n RETURNS text[] AS $$\n SELECT\n CASE jsonb_typeof(_js)\n WHEN 'array' THEN ARRAY(SELECT jsonb_array_elements_text(_js))\n ELSE ARRAY[_js->>0]\n END\n;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION jsonb_paths (IN jdata jsonb, OUT path text[], OUT value jsonb) RETURNS\nSETOF RECORD AS $$\nwith recursive extract_all as\n(\n select\n ARRAY[key]::text[] as path,\n value\n FROM jsonb_each(jdata)\nunion all\n select\n path || coalesce(obj_key, (arr_key- 1)::text),\n coalesce(obj_value, arr_value)\n from extract_all\n left join lateral\n jsonb_each(case jsonb_typeof(value) when 'object' then value end)\n as o(obj_key, obj_value)\n on jsonb_typeof(value) = 'object'\n left join lateral\n jsonb_array_elements(case jsonb_typeof(value) when 'array' then value end)\n with ordinality as a(arr_value, arr_key)\n on jsonb_typeof(value) = 'array'\n where obj_key is not null or arr_key is not null\n)\nselect *\nfrom extract_all;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION jsonb_obj_paths (IN jdata jsonb, OUT path text[], OUT value jsonb) RETURNS\nSETOF RECORD AS $$\nwith recursive extract_all as\n(\n select\n ARRAY[key]::text[] as path,\n value\n FROM jsonb_each(jdata)\nunion all\n select\n path || obj_key,\n obj_value\n from extract_all\n left join lateral\n jsonb_each(case jsonb_typeof(value) when 'object' then value end)\n as o(obj_key, obj_value)\n on jsonb_typeof(value) = 'object'\n where obj_key is not null\n)\nselect *\nfrom extract_all;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION jsonb_val_paths (IN jdata jsonb, OUT path text[], OUT value jsonb) RETURNS\nSETOF RECORD AS $$\nSELECT * FROM jsonb_obj_paths(jdata) WHERE jsonb_typeof(value) not in ('object','array');\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION path_includes(IN path text[], IN includes text[]) RETURNS BOOLEAN AS $$\nWITH t AS (SELECT unnest(includes) i)\nSELECT EXISTS (\n SELECT 1 FROM t WHERE path @> string_to_array(trim(i), '.')\n);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION path_excludes(IN path text[], IN excludes text[]) RETURNS BOOLEAN AS $$\nWITH t AS (SELECT unnest(excludes) e)\nSELECT NOT EXISTS (\n SELECT 1 FROM t WHERE path @> string_to_array(trim(e), '.')\n);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION jsonb_obj_paths_filtered (\n IN jdata jsonb,\n IN includes text[] DEFAULT ARRAY[]::text[],\n IN excludes text[] DEFAULT ARRAY[]::text[],\n OUT path text[],\n OUT value jsonb\n) RETURNS\nSETOF RECORD AS $$\nSELECT path, value\nFROM jsonb_obj_paths(jdata)\nWHERE\n CASE WHEN cardinality(includes) > 0 THEN path_includes(path, includes) ELSE TRUE END\n AND\n path_excludes(path, excludes)\n\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION filter_jsonb(\n IN jdata jsonb,\n IN includes text[] DEFAULT ARRAY[]::text[],\n IN excludes text[] DEFAULT ARRAY[]::text[]\n) RETURNS jsonb AS $$\nDECLARE\nrec RECORD;\noutj jsonb := '{}'::jsonb;\ncreated_paths text[] := '{}'::text[];\nBEGIN\n\nIF empty_arr(includes) AND empty_arr(excludes) THEN\nRAISE NOTICE 'no filter';\n RETURN jdata;\nEND IF;\nFOR rec in\nSELECT * FROM jsonb_obj_paths_filtered(jdata, includes, excludes)\nWHERE jsonb_typeof(value) != 'object'\nLOOP\n IF array_length(rec.path,1)>1 THEN\n FOR i IN 1..(array_length(rec.path,1)-1) LOOP\n IF NOT array_to_string(rec.path[1:i],'.') = ANY (created_paths) THEN\n outj := jsonb_set(outj, rec.path[1:i],'{}', true);\n created_paths := created_paths || array_to_string(rec.path[1:i],'.');\n END IF;\n END LOOP;\n END IF;\n outj := jsonb_set(outj, rec.path, rec.value, true);\n created_paths := created_paths || array_to_string(rec.path,'.');\nEND LOOP;\nRETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION flip_jsonb_array(j jsonb) RETURNS jsonb AS $$\nSELECT jsonb_agg(value) FROM (SELECT value FROM jsonb_array_elements(j) WITH ORDINALITY ORDER BY ordinality DESC) as t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n/* Functions to create an iterable of cursors over partitions. */\nCREATE OR REPLACE FUNCTION create_cursor(q text) RETURNS refcursor AS $$\nDECLARE\n curs refcursor;\nBEGIN\n OPEN curs FOR EXECUTE q;\n RETURN curs;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDROP FUNCTION IF EXISTS partition_queries;\nCREATE OR REPLACE FUNCTION partition_queries(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN partitions text[] DEFAULT '{items}'\n) RETURNS SETOF text AS $$\nDECLARE\n partition_query text;\n query text;\n p text;\n cursors refcursor;\n dstart timestamptz;\n dend timestamptz;\n step interval := '10 weeks'::interval;\nBEGIN\n\nIF _orderby ILIKE 'datetime d%' THEN\n partitions := partitions;\nELSIF _orderby ILIKE 'datetime a%' THEN\n partitions := array_reverse(partitions);\nELSE\n query := format($q$\n SELECT * FROM items\n WHERE %s\n ORDER BY %s\n $q$, _where, _orderby\n );\n\n RETURN NEXT query;\n RETURN;\nEND IF;\nRAISE NOTICE 'PARTITIONS ---> %',partitions;\nIF cardinality(partitions) > 0 THEN\n FOREACH p IN ARRAY partitions\n --EXECUTE partition_query\n LOOP\n query := format($q$\n SELECT * FROM %I\n WHERE %s\n ORDER BY %s\n $q$,\n p,\n _where,\n _orderby\n );\n RETURN NEXT query;\n END LOOP;\nEND IF;\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION partition_cursor(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC'\n) RETURNS SETOF refcursor AS $$\nDECLARE\n partition_query text;\n query text;\n p record;\n cursors refcursor;\nBEGIN\nFOR query IN SELECT * FROM partition_queries(_where, _orderby) LOOP\n RETURN NEXT create_cursor(query);\nEND LOOP;\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION partition_count(\n IN _where text DEFAULT 'TRUE'\n) RETURNS bigint AS $$\nDECLARE\n partition_query text;\n query text;\n p record;\n subtotal bigint;\n total bigint := 0;\nBEGIN\npartition_query := format($q$\n SELECT partition, tstzrange\n FROM items_partitions\n ORDER BY tstzrange DESC;\n$q$);\nRAISE NOTICE 'Partition Query: %', partition_query;\nFOR p IN\n EXECUTE partition_query\nLOOP\n query := format($q$\n SELECT count(*) FROM items\n WHERE datetime BETWEEN %L AND %L AND %s\n $q$, lower(p.tstzrange), upper(p.tstzrange), _where\n );\n RAISE NOTICE 'Query %', query;\n RAISE NOTICE 'Partition %, Count %, Total %',p.partition, subtotal, total;\n EXECUTE query INTO subtotal;\n total := subtotal + total;\nEND LOOP;\nRETURN total;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION drop_partition_constraints(IN partition text) RETURNS VOID AS $$\nDECLARE\n q text;\n end_datetime_constraint text := concat(partition, '_end_datetime_constraint');\n collections_constraint text := concat(partition, '_collections_constraint');\nBEGIN\n q := format($q$\n ALTER TABLE %I\n DROP CONSTRAINT IF EXISTS %I,\n DROP CONSTRAINT IF EXISTS %I;\n $q$,\n partition,\n end_datetime_constraint,\n collections_constraint\n );\n\n EXECUTE q;\n RETURN;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDROP FUNCTION IF EXISTS partition_checks;\nCREATE OR REPLACE FUNCTION partition_checks(\n IN partition text,\n OUT min_datetime timestamptz,\n OUT max_datetime timestamptz,\n OUT min_end_datetime timestamptz,\n OUT max_end_datetime timestamptz,\n OUT collections text[],\n OUT cnt bigint\n) RETURNS RECORD AS $$\nDECLARE\nq text;\nend_datetime_constraint text := concat(partition, '_end_datetime_constraint');\ncollections_constraint text := concat(partition, '_collections_constraint');\nBEGIN\nRAISE NOTICE 'CREATING CONSTRAINTS FOR %', partition;\nq := format($q$\n SELECT\n min(datetime),\n max(datetime),\n min(end_datetime),\n max(end_datetime),\n array_agg(DISTINCT collection_id),\n count(*)\n FROM %I;\n $q$,\n partition\n);\nEXECUTE q INTO min_datetime, max_datetime, min_end_datetime, max_end_datetime, collections, cnt;\nRAISE NOTICE '% % % % % % %', min_datetime, max_datetime, min_end_datetime, max_end_datetime, collections, cnt, ftime();\nIF cnt IS NULL or cnt = 0 THEN\n RAISE NOTICE 'Partition % is empty, removing...', partition;\n q := format($q$\n DROP TABLE IF EXISTS %I;\n $q$, partition\n );\n EXECUTE q;\n RETURN;\nEND IF;\nRAISE NOTICE 'Running Constraint DDL %', ftime();\nq := format($q$\n ALTER TABLE %I\n DROP CONSTRAINT IF EXISTS %I,\n ADD CONSTRAINT %I\n check((end_datetime >= %L) AND (end_datetime <= %L)) NOT VALID,\n DROP CONSTRAINT IF EXISTS %I,\n ADD CONSTRAINT %I\n check((collection_id = ANY(%L))) NOT VALID;\n $q$,\n partition,\n end_datetime_constraint,\n end_datetime_constraint,\n min_end_datetime,\n max_end_datetime,\n collections_constraint,\n collections_constraint,\n collections,\n partition\n);\nRAISE NOTICE 'q: %', q;\n\nEXECUTE q;\nRAISE NOTICE 'Returning %', ftime();\nRETURN;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION validate_constraints() RETURNS VOID AS $$\nDECLARE\nq text;\nBEGIN\nFOR q IN\n SELECT FORMAT(\n 'ALTER TABLE %I.%I.%I VALIDATE CONSTRAINT %I;',\n current_database(),\n nsp.nspname,\n cls.relname,\n con.conname\n )\n FROM pg_constraint AS con\n JOIN pg_class AS cls\n ON con.conrelid = cls.oid\n JOIN pg_namespace AS nsp\n ON cls.relnamespace = nsp.oid\n WHERE convalidated IS FALSE\n AND nsp.nspname = 'pgstac'\nLOOP\n EXECUTE q;\nEND LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n/* looks for a geometry in a stac item first from geometry and falling back to bbox */\nCREATE OR REPLACE FUNCTION stac_geom(value jsonb) RETURNS geometry AS $$\nSELECT\n CASE\n WHEN value->>'geometry' IS NOT NULL THEN\n ST_GeomFromGeoJSON(value->>'geometry')\n WHEN value->>'bbox' IS NOT NULL THEN\n ST_MakeEnvelope(\n (value->'bbox'->>0)::float,\n (value->'bbox'->>1)::float,\n (value->'bbox'->>2)::float,\n (value->'bbox'->>3)::float,\n 4326\n )\n ELSE NULL\n END as geometry\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION stac_datetime(value jsonb) RETURNS timestamptz AS $$\nSELECT COALESCE(\n (value->'properties'->>'datetime')::timestamptz,\n (value->'properties'->>'start_datetime')::timestamptz\n);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_end_datetime(value jsonb) RETURNS timestamptz AS $$\nSELECT COALESCE(\n (value->'properties'->>'datetime')::timestamptz,\n (value->'properties'->>'end_datetime')::timestamptz\n);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\n\nCREATE OR REPLACE FUNCTION stac_daterange(value jsonb) RETURNS tstzrange AS $$\nSELECT tstzrange(stac_datetime(value),stac_end_datetime(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE TABLE IF NOT EXISTS collections (\n id VARCHAR GENERATED ALWAYS AS (content->>'id') STORED PRIMARY KEY,\n content JSONB\n);\n\nCREATE OR REPLACE FUNCTION create_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection(data jsonb) RETURNS VOID AS $$\nDECLARE\nout collections%ROWTYPE;\nBEGIN\n UPDATE collections SET content=data WHERE id = data->>'id' RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ON CONFLICT (id) DO\n UPDATE\n SET content=EXCLUDED.content\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION delete_collection(_id text) RETURNS VOID AS $$\nDECLARE\nout collections%ROWTYPE;\nBEGIN\n DELETE FROM collections WHERE id = _id RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION get_collection(id text) RETURNS jsonb AS $$\nSELECT content FROM collections\nWHERE id=$1\n;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION all_collections() RETURNS jsonb AS $$\nSELECT jsonb_agg(content) FROM collections;\n;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE TABLE items (\n id text NOT NULL,\n geometry geometry NOT NULL,\n collection_id text NOT NULL,\n datetime timestamptz NOT NULL,\n end_datetime timestamptz NOT NULL,\n properties jsonb NOT NULL,\n content JSONB NOT NULL\n)\nPARTITION BY RANGE (datetime)\n;\n\nCREATE OR REPLACE FUNCTION properties_idx (IN content jsonb) RETURNS jsonb AS $$\n with recursive extract_all as\n (\n select\n ARRAY[key]::text[] as path,\n ARRAY[key]::text[] as fullpath,\n value\n FROM jsonb_each(content->'properties')\n union all\n select\n CASE WHEN obj_key IS NOT NULL THEN path || obj_key ELSE path END,\n path || coalesce(obj_key, (arr_key- 1)::text),\n coalesce(obj_value, arr_value)\n from extract_all\n left join lateral\n jsonb_each(case jsonb_typeof(value) when 'object' then value end)\n as o(obj_key, obj_value)\n on jsonb_typeof(value) = 'object'\n left join lateral\n jsonb_array_elements(case jsonb_typeof(value) when 'array' then value end)\n with ordinality as a(arr_value, arr_key)\n on jsonb_typeof(value) = 'array'\n where obj_key is not null or arr_key is not null\n )\n , paths AS (\n select\n array_to_string(path, '.') as path,\n value\n FROM extract_all\n WHERE\n jsonb_typeof(value) NOT IN ('array','object')\n ), grouped AS (\n SELECT path, jsonb_agg(distinct value) vals FROM paths group by path\n ) SELECT coalesce(jsonb_object_agg(path, CASE WHEN jsonb_array_length(vals)=1 THEN vals->0 ELSE vals END) - '{datetime}'::text[], '{}'::jsonb) FROM grouped\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET JIT TO OFF;\n\nCREATE INDEX \"datetime_idx\" ON items (datetime);\nCREATE INDEX \"end_datetime_idx\" ON items (end_datetime);\nCREATE INDEX \"properties_idx\" ON items USING GIN (properties jsonb_path_ops);\nCREATE INDEX \"collection_idx\" ON items (collection_id);\nCREATE INDEX \"geometry_idx\" ON items USING GIST (geometry);\nCREATE UNIQUE INDEX \"items_id_datetime_idx\" ON items (datetime, id);\n\nALTER TABLE items ADD CONSTRAINT items_collections_fk FOREIGN KEY (collection_id) REFERENCES collections(id) ON DELETE CASCADE DEFERRABLE;\n\nCREATE OR REPLACE FUNCTION analyze_empty_partitions() RETURNS VOID AS $$\nDECLARE\n p text;\nBEGIN\n FOR p IN SELECT partition FROM all_items_partitions WHERE est_cnt = 0 LOOP\n EXECUTE format('ANALYZE %I;', p);\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION items_partition_name(timestamptz) RETURNS text AS $$\n SELECT to_char($1, '\"items_p\"IYYY\"w\"IW');\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION items_partition_exists(text) RETURNS boolean AS $$\n SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_class WHERE relname=$1);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION items_partition_exists(timestamptz) RETURNS boolean AS $$\n SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_class WHERE relname=items_partition_name($1));\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION items_partition_create_worker(partition text, partition_start timestamptz, partition_end timestamptz) RETURNS VOID AS $$\nDECLARE\n err_context text;\nBEGIN\n EXECUTE format(\n $f$\n CREATE TABLE IF NOT EXISTS %1$I PARTITION OF items\n FOR VALUES FROM (%2$L) TO (%3$L);\n CREATE UNIQUE INDEX IF NOT EXISTS %4$I ON %1$I (id);\n $f$,\n partition,\n partition_start,\n partition_end,\n concat(partition, '_id_pk')\n );\nEXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', partition;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH to pgstac, public;\n\nCREATE OR REPLACE FUNCTION items_partition_create(ts timestamptz) RETURNS text AS $$\nDECLARE\n partition text := items_partition_name(ts);\n partition_start timestamptz;\n partition_end timestamptz;\nBEGIN\n IF items_partition_exists(partition) THEN\n RETURN partition;\n END IF;\n partition_start := date_trunc('week', ts);\n partition_end := partition_start + '1 week'::interval;\n PERFORM items_partition_create_worker(partition, partition_start, partition_end);\n RAISE NOTICE 'partition: %', partition;\n RETURN partition;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION items_partition_create(st timestamptz, et timestamptz) RETURNS SETOF text AS $$\nWITH t AS (\n SELECT\n generate_series(\n date_trunc('week',st),\n date_trunc('week', et),\n '1 week'::interval\n ) w\n)\nSELECT items_partition_create(w) FROM t;\n$$ LANGUAGE SQL;\n\n\nCREATE UNLOGGED TABLE items_staging (\n content JSONB NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION items_staging_insert_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p record;\n _partitions text[];\nBEGIN\n CREATE TEMP TABLE new_partitions ON COMMIT DROP AS\n SELECT\n items_partition_name(stac_datetime(content)) as partition,\n date_trunc('week', min(stac_datetime(content))) as partition_start\n FROM newdata\n GROUP BY 1;\n\n -- set statslastupdated in cache to be old enough cache always regenerated\n\n SELECT array_agg(partition) INTO _partitions FROM new_partitions;\n UPDATE search_wheres\n SET\n statslastupdated = NULL\n WHERE _where IN (\n SELECT _where FROM search_wheres sw WHERE sw.partitions && _partitions\n FOR UPDATE SKIP LOCKED\n )\n ;\n FOR p IN SELECT new_partitions.partition, new_partitions.partition_start, new_partitions.partition_start + '1 week'::interval as partition_end FROM new_partitions\n LOOP\n RAISE NOTICE 'Getting partition % ready.', p.partition;\n IF NOT items_partition_exists(p.partition) THEN\n RAISE NOTICE 'Creating partition %.', p.partition;\n PERFORM items_partition_create_worker(p.partition, p.partition_start, p.partition_end);\n END IF;\n PERFORM drop_partition_constraints(p.partition);\n END LOOP;\n INSERT INTO items (id, geometry, collection_id, datetime, end_datetime, properties, content)\n SELECT\n content->>'id',\n stac_geom(content),\n content->>'collection',\n stac_datetime(content),\n stac_end_datetime(content),\n properties_idx(content),\n content\n FROM newdata\n ;\n DELETE FROM items_staging;\n\n\n FOR p IN SELECT new_partitions.partition FROM new_partitions\n LOOP\n RAISE NOTICE 'Setting constraints for partition %.', p.partition;\n PERFORM partition_checks(p.partition);\n END LOOP;\n DROP TABLE IF EXISTS new_partitions;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER items_staging_insert_trigger AFTER INSERT ON items_staging REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_insert_triggerfunc();\n\n\nCREATE UNLOGGED TABLE items_staging_ignore (\n content JSONB NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION items_staging_ignore_insert_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p record;\n _partitions text[];\nBEGIN\n CREATE TEMP TABLE new_partitions ON COMMIT DROP AS\n SELECT\n items_partition_name(stac_datetime(content)) as partition,\n date_trunc('week', min(stac_datetime(content))) as partition_start\n FROM newdata\n GROUP BY 1;\n\n -- set statslastupdated in cache to be old enough cache always regenerated\n\n SELECT array_agg(partition) INTO _partitions FROM new_partitions;\n UPDATE search_wheres\n SET\n statslastupdated = NULL\n WHERE _where IN (\n SELECT _where FROM search_wheres sw WHERE sw.partitions && _partitions\n FOR UPDATE SKIP LOCKED\n )\n ;\n\n FOR p IN SELECT new_partitions.partition, new_partitions.partition_start, new_partitions.partition_start + '1 week'::interval as partition_end FROM new_partitions\n LOOP\n RAISE NOTICE 'Getting partition % ready.', p.partition;\n IF NOT items_partition_exists(p.partition) THEN\n RAISE NOTICE 'Creating partition %.', p.partition;\n PERFORM items_partition_create_worker(p.partition, p.partition_start, p.partition_end);\n END IF;\n PERFORM drop_partition_constraints(p.partition);\n END LOOP;\n\n INSERT INTO items (id, geometry, collection_id, datetime, end_datetime, properties, content)\n SELECT\n content->>'id',\n stac_geom(content),\n content->>'collection',\n stac_datetime(content),\n stac_end_datetime(content),\n properties_idx(content),\n content\n FROM newdata\n ON CONFLICT (datetime, id) DO NOTHING\n ;\n DELETE FROM items_staging;\n\n\n FOR p IN SELECT new_partitions.partition FROM new_partitions\n LOOP\n RAISE NOTICE 'Setting constraints for partition %.', p.partition;\n PERFORM partition_checks(p.partition);\n END LOOP;\n DROP TABLE IF EXISTS new_partitions;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER items_staging_ignore_insert_trigger AFTER INSERT ON items_staging_ignore REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_ignore_insert_triggerfunc();\n\nCREATE UNLOGGED TABLE items_staging_upsert (\n content JSONB NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION items_staging_upsert_insert_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p record;\n _partitions text[];\nBEGIN\n CREATE TEMP TABLE new_partitions ON COMMIT DROP AS\n SELECT\n items_partition_name(stac_datetime(content)) as partition,\n date_trunc('week', min(stac_datetime(content))) as partition_start\n FROM newdata\n GROUP BY 1;\n\n -- set statslastupdated in cache to be old enough cache always regenerated\n\n SELECT array_agg(partition) INTO _partitions FROM new_partitions;\n UPDATE search_wheres\n SET\n statslastupdated = NULL\n WHERE _where IN (\n SELECT _where FROM search_wheres sw WHERE sw.partitions && _partitions\n FOR UPDATE SKIP LOCKED\n )\n ;\n\n FOR p IN SELECT new_partitions.partition, new_partitions.partition_start, new_partitions.partition_start + '1 week'::interval as partition_end FROM new_partitions\n LOOP\n RAISE NOTICE 'Getting partition % ready.', p.partition;\n IF NOT items_partition_exists(p.partition) THEN\n RAISE NOTICE 'Creating partition %.', p.partition;\n PERFORM items_partition_create_worker(p.partition, p.partition_start, p.partition_end);\n END IF;\n PERFORM drop_partition_constraints(p.partition);\n END LOOP;\n\n INSERT INTO items (id, geometry, collection_id, datetime, end_datetime, properties, content)\n SELECT\n content->>'id',\n stac_geom(content),\n content->>'collection',\n stac_datetime(content),\n stac_end_datetime(content),\n properties_idx(content),\n content\n FROM newdata\n ON CONFLICT (datetime, id) DO UPDATE SET\n content = EXCLUDED.content\n WHERE items.content IS DISTINCT FROM EXCLUDED.content\n ;\n DELETE FROM items_staging;\n\n\n FOR p IN SELECT new_partitions.partition FROM new_partitions\n LOOP\n RAISE NOTICE 'Setting constraints for partition %.', p.partition;\n PERFORM partition_checks(p.partition);\n END LOOP;\n DROP TABLE IF EXISTS new_partitions;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER items_staging_upsert_insert_trigger AFTER INSERT ON items_staging_upsert REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_upsert_insert_triggerfunc();\n\nCREATE OR REPLACE FUNCTION items_update_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\nBEGIN\n NEW.id := NEW.content->>'id';\n NEW.datetime := stac_datetime(NEW.content);\n NEW.end_datetime := stac_end_datetime(NEW.content);\n NEW.collection_id := NEW.content->>'collection';\n NEW.geometry := stac_geom(NEW.content);\n NEW.properties := properties_idx(NEW.content);\n IF TG_OP = 'UPDATE' AND NEW IS NOT DISTINCT FROM OLD THEN\n RETURN NULL;\n END IF;\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER items_update_trigger BEFORE UPDATE ON items\n FOR EACH ROW EXECUTE PROCEDURE items_update_triggerfunc();\n\n/*\nView to get a table of available items partitions\nwith date ranges\n*/\nDROP VIEW IF EXISTS all_items_partitions CASCADE;\nCREATE VIEW all_items_partitions AS\nWITH base AS\n(SELECT\n c.oid::pg_catalog.regclass::text as partition,\n pg_catalog.pg_get_expr(c.relpartbound, c.oid) as _constraint,\n regexp_matches(\n pg_catalog.pg_get_expr(c.relpartbound, c.oid),\n E'\\\\(''\\([0-9 :+-]*\\)''\\\\).*\\\\(''\\([0-9 :+-]*\\)''\\\\)'\n ) as t,\n reltuples::bigint as est_cnt\nFROM pg_catalog.pg_class c, pg_catalog.pg_inherits i\nWHERE c.oid = i.inhrelid AND i.inhparent = 'items'::regclass)\nSELECT partition, tstzrange(\n t[1]::timestamptz,\n t[2]::timestamptz\n), t[1]::timestamptz as pstart,\n t[2]::timestamptz as pend, est_cnt\nFROM base\nORDER BY 2 desc;\n\nCREATE OR REPLACE VIEW items_partitions AS\nSELECT * FROM all_items_partitions WHERE est_cnt>0;\n\nCREATE OR REPLACE FUNCTION get_item(_id text) RETURNS jsonb AS $$\n SELECT content FROM items WHERE id=_id;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION delete_item(_id text) RETURNS VOID AS $$\nDECLARE\nout items%ROWTYPE;\nBEGIN\n DELETE FROM items WHERE id = _id RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION create_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION update_item(data jsonb) RETURNS VOID AS $$\nDECLARE\n out items%ROWTYPE;\nBEGIN\n UPDATE items SET content=data WHERE id = data->>'id' RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION create_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION collection_bbox(id text) RETURNS jsonb AS $$\nSELECT (replace(replace(replace(st_extent(geometry)::text,'BOX(','[['),')',']]'),' ',','))::jsonb\nFROM items WHERE collection_id=$1;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION collection_temporal_extent(id text) RETURNS jsonb AS $$\nSELECT to_jsonb(array[array[min(datetime)::text, max(datetime)::text]])\nFROM items WHERE collection_id=$1;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection_extents() RETURNS VOID AS $$\nUPDATE collections SET\n content = content ||\n jsonb_build_object(\n 'extent', jsonb_build_object(\n 'spatial', jsonb_build_object(\n 'bbox', collection_bbox(collections.id)\n ),\n 'temporal', jsonb_build_object(\n 'interval', collection_temporal_extent(collections.id)\n )\n )\n )\n;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION items_path(\n IN dotpath text,\n OUT field text,\n OUT path text,\n OUT path_txt text,\n OUT jsonpath text,\n OUT eq text\n) RETURNS RECORD AS $$\nDECLARE\npath_elements text[];\nlast_element text;\nBEGIN\ndotpath := replace(trim(dotpath), 'properties.', '');\n\nIF dotpath = '' THEN\n RETURN;\nEND IF;\n\npath_elements := string_to_array(dotpath, '.');\njsonpath := NULL;\n\nIF path_elements[1] IN ('id','geometry','datetime') THEN\n field := path_elements[1];\n path_elements := path_elements[2:];\nELSIF path_elements[1] = 'collection' THEN\n field := 'collection_id';\n path_elements := path_elements[2:];\nELSIF path_elements[1] IN ('links', 'assets', 'stac_version', 'stac_extensions') THEN\n field := 'content';\nELSE\n field := 'content';\n path_elements := '{properties}'::text[] || path_elements;\nEND IF;\nIF cardinality(path_elements)<1 THEN\n path := field;\n path_txt := field;\n jsonpath := '$';\n eq := NULL;\n RETURN;\nEND IF;\n\n\nlast_element := path_elements[cardinality(path_elements)];\npath_elements := path_elements[1:cardinality(path_elements)-1];\njsonpath := concat(array_to_string('{$}'::text[] || array_map_ident(path_elements), '.'), '.', quote_ident(last_element));\npath_elements := array_map_literal(path_elements);\npath := format($F$ properties->%s $F$, quote_literal(dotpath));\npath_txt := format($F$ properties->>%s $F$, quote_literal(dotpath));\neq := format($F$ properties @? '$.%s[*] ? (@ == %%s) '$F$, quote_ident(dotpath));\n\nRAISE NOTICE 'ITEMS PATH -- % % % % %', field, path, path_txt, jsonpath, eq;\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION parse_dtrange(IN _indate jsonb, OUT _tstzrange tstzrange) AS $$\nWITH t AS (\n SELECT CASE\n WHEN jsonb_typeof(_indate) = 'array' THEN\n textarr(_indate)\n ELSE\n regexp_split_to_array(\n btrim(_indate::text,'\"'),\n '/'\n )\n END AS arr\n)\n, t1 AS (\n SELECT\n CASE\n WHEN array_upper(arr,1) = 1 OR arr[1] = '..' OR arr[1] IS NULL THEN '-infinity'::timestamptz\n ELSE arr[1]::timestamptz\n END AS st,\n CASE\n WHEN array_upper(arr,1) = 1 THEN arr[1]::timestamptz\n WHEN arr[2] = '..' OR arr[2] IS NULL THEN 'infinity'::timestamptz\n ELSE arr[2]::timestamptz\n END AS et\n FROM t\n)\nSELECT\n tstzrange(st,et)\nFROM t1;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION bbox_geom(_bbox jsonb) RETURNS geometry AS $$\nSELECT CASE jsonb_array_length(_bbox)\n WHEN 4 THEN\n ST_SetSRID(ST_MakeEnvelope(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float,\n (_bbox->>3)::float\n ),4326)\n WHEN 6 THEN\n ST_SetSRID(ST_3DMakeBox(\n ST_MakePoint(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float\n ),\n ST_MakePoint(\n (_bbox->>3)::float,\n (_bbox->>4)::float,\n (_bbox->>5)::float\n )\n ),4326)\n ELSE null END;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION cql_and_append(existing jsonb, newfilters jsonb) RETURNS jsonb AS $$\nSELECT CASE WHEN existing ? 'filter' AND newfilters IS NOT NULL THEN\n jsonb_build_object(\n 'and',\n jsonb_build_array(\n existing->'filter',\n newfilters\n )\n )\nELSE\n newfilters\nEND;\n$$ LANGUAGE SQL;\n\n\n-- ADDs base filters (ids, collections, datetime, bbox, intersects) that are\n-- added outside of the filter/query in the stac request\nCREATE OR REPLACE FUNCTION add_filters_to_cql(j jsonb) RETURNS jsonb AS $$\nDECLARE\nnewprop jsonb;\nnewprops jsonb := '[]'::jsonb;\nBEGIN\nIF j ? 'ids' THEN\n newprop := jsonb_build_object(\n 'in',\n jsonb_build_array(\n '{\"property\":\"id\"}'::jsonb,\n j->'ids'\n )\n );\n newprops := jsonb_insert(newprops, '{1}', newprop);\nEND IF;\nIF j ? 'collections' THEN\n newprop := jsonb_build_object(\n 'in',\n jsonb_build_array(\n '{\"property\":\"collection\"}'::jsonb,\n j->'collections'\n )\n );\n newprops := jsonb_insert(newprops, '{1}', newprop);\nEND IF;\n\nIF j ? 'datetime' THEN\n newprop := format(\n '{\"anyinteracts\":[{\"property\":\"datetime\"}, %s]}',\n j->'datetime'\n );\n newprops := jsonb_insert(newprops, '{1}', newprop);\nEND IF;\n\nIF j ? 'bbox' THEN\n newprop := format(\n '{\"intersects\":[{\"property\":\"geometry\"}, %s]}',\n j->'bbox'\n );\n newprops := jsonb_insert(newprops, '{1}', newprop);\nEND IF;\n\nIF j ? 'intersects' THEN\n newprop := format(\n '{\"intersects\":[{\"property\":\"geometry\"}, %s]}',\n j->'intersects'\n );\n newprops := jsonb_insert(newprops, '{1}', newprop);\nEND IF;\n\nRAISE NOTICE 'newprops: %', newprops;\n\nIF newprops IS NOT NULL AND jsonb_array_length(newprops) > 0 THEN\n return jsonb_set(\n j,\n '{filter}',\n cql_and_append(j, jsonb_build_object('and', newprops))\n ) - '{ids,collections,datetime,bbox,intersects}'::text[];\nEND IF;\n\nreturn j;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION query_to_cqlfilter(j jsonb) RETURNS jsonb AS $$\n-- Translates anything passed in through the deprecated \"query\" into equivalent CQL\nWITH t AS (\n SELECT key as property, value as ops\n FROM jsonb_each(j->'query')\n), t2 AS (\n SELECT property, (jsonb_each(ops)).*\n FROM t WHERE jsonb_typeof(ops) = 'object'\n UNION ALL\n SELECT property, 'eq', ops\n FROM t WHERE jsonb_typeof(ops) != 'object'\n), t3 AS (\nSELECT\n jsonb_strip_nulls(jsonb_build_object(\n 'and',\n jsonb_agg(\n jsonb_build_object(\n key,\n jsonb_build_array(\n jsonb_build_object('property',property),\n value\n )\n )\n )\n )) as qcql FROM t2\n)\nSELECT\n CASE WHEN qcql IS NOT NULL THEN\n jsonb_set(j, '{filter}', cql_and_append(j, qcql)) - 'query'\n ELSE j\n END\nFROM t3\n;\n$$ LANGUAGE SQL;\n\n\n\nCREATE OR REPLACE FUNCTION temporal_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\nll text := 'datetime';\nlh text := 'end_datetime';\nrrange tstzrange;\nrl text;\nrh text;\noutq text;\nBEGIN\nrrange := parse_dtrange(args->1);\nRAISE NOTICE 'Constructing temporal query OP: %, ARGS: %, RRANGE: %', op, args, rrange;\nop := lower(op);\nrl := format('%L::timestamptz', lower(rrange));\nrh := format('%L::timestamptz', upper(rrange));\noutq := CASE op\n WHEN 't_before' THEN 'lh < rl'\n WHEN 't_after' THEN 'll > rh'\n WHEN 't_meets' THEN 'lh = rl'\n WHEN 't_metby' THEN 'll = rh'\n WHEN 't_overlaps' THEN 'll < rl AND rl < lh < rh'\n WHEN 't_overlappedby' THEN 'rl < ll < rh AND lh > rh'\n WHEN 't_starts' THEN 'll = rl AND lh < rh'\n WHEN 't_startedby' THEN 'll = rl AND lh > rh'\n WHEN 't_during' THEN 'll > rl AND lh < rh'\n WHEN 't_contains' THEN 'll < rl AND lh > rh'\n WHEN 't_finishes' THEN 'll > rl AND lh = rh'\n WHEN 't_finishedby' THEN 'll < rl AND lh = rh'\n WHEN 't_equals' THEN 'll = rl AND lh = rh'\n WHEN 't_disjoint' THEN 'NOT (ll <= rh AND lh >= rl)'\n WHEN 't_intersects' THEN 'll <= rh AND lh >= rl'\n WHEN 'anyinteracts' THEN 'll <= rh AND lh >= rl'\nEND;\noutq := regexp_replace(outq, '\\mll\\M', ll);\noutq := regexp_replace(outq, '\\mlh\\M', lh);\noutq := regexp_replace(outq, '\\mrl\\M', rl);\noutq := regexp_replace(outq, '\\mrh\\M', rh);\noutq := format('(%s)', outq);\nRETURN outq;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION spatial_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\ngeom text;\nj jsonb := args->1;\nBEGIN\nop := lower(op);\nRAISE NOTICE 'Constructing spatial query OP: %, ARGS: %', op, args;\nIF op NOT IN ('s_equals','s_disjoint','s_touches','s_within','s_overlaps','s_crosses','s_intersects','intersects','s_contains') THEN\n RAISE EXCEPTION 'Spatial Operator % Not Supported', op;\nEND IF;\nop := regexp_replace(op, '^s_', 'st_');\nIF op = 'intersects' THEN\n op := 'st_intersects';\nEND IF;\n-- Convert geometry to WKB string\nIF j ? 'type' AND j ? 'coordinates' THEN\n geom := st_geomfromgeojson(j)::text;\nELSIF jsonb_typeof(j) = 'array' THEN\n geom := bbox_geom(j)::text;\nEND IF;\n\nRETURN format('%s(geometry, %L::geometry)', op, geom);\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n/* cql_query_op -- Parses a CQL query operation, recursing when necessary\n IN jsonb -- a subelement from a valid stac query\n IN text -- the operator being used on elements passed in\n RETURNS a SQL fragment to be used in a WHERE clause\n*/\nCREATE OR REPLACE FUNCTION cql_query_op(j jsonb, _op text DEFAULT NULL) RETURNS text AS $$\nDECLARE\njtype text := jsonb_typeof(j);\nop text := lower(_op);\nops jsonb :=\n '{\n \"eq\": \"%s = %s\",\n \"lt\": \"%s < %s\",\n \"lte\": \"%s <= %s\",\n \"gt\": \"%s > %s\",\n \"gte\": \"%s >= %s\",\n \"le\": \"%s <= %s\",\n \"ge\": \"%s >= %s\",\n \"=\": \"%s = %s\",\n \"<\": \"%s < %s\",\n \"<=\": \"%s <= %s\",\n \">\": \"%s > %s\",\n \">=\": \"%s >= %s\",\n \"like\": \"%s LIKE %s\",\n \"ilike\": \"%s ILIKE %s\",\n \"+\": \"%s + %s\",\n \"-\": \"%s - %s\",\n \"*\": \"%s * %s\",\n \"/\": \"%s / %s\",\n \"in\": \"%s = ANY (%s)\",\n \"not\": \"NOT (%s)\",\n \"between\": \"%s BETWEEN %s AND %s\",\n \"lower\":\" lower(%s)\",\n \"upper\":\" upper(%s)\",\n \"isnull\": \"%s IS NULL\"\n }'::jsonb;\nret text;\nargs text[] := NULL;\n\nBEGIN\nRAISE NOTICE 'j: %, op: %, jtype: %', j, op, jtype;\n\n-- for in, convert value, list to array syntax to match other ops\nIF op = 'in' and j ? 'value' and j ? 'list' THEN\n j := jsonb_build_array( j->'value', j->'list');\n jtype := 'array';\n RAISE NOTICE 'IN: j: %, jtype: %', j, jtype;\nEND IF;\n\nIF op = 'between' and j ? 'value' and j ? 'lower' and j ? 'upper' THEN\n j := jsonb_build_array( j->'value', j->'lower', j->'upper');\n jtype := 'array';\n RAISE NOTICE 'BETWEEN: j: %, jtype: %', j, jtype;\nEND IF;\n\nIF op = 'not' AND jtype = 'object' THEN\n j := jsonb_build_array( j );\n jtype := 'array';\n RAISE NOTICE 'NOT: j: %, jtype: %', j, jtype;\nEND IF;\n\n-- Set Lower Case on Both Arguments When Case Insensitive Flag Set\nIF op in ('eq','lt','lte','gt','gte','like') AND jsonb_typeof(j->2) = 'boolean' THEN\n IF (j->>2)::boolean THEN\n RETURN format(concat('(',ops->>op,')'), cql_query_op(jsonb_build_array(j->0), 'lower'), cql_query_op(jsonb_build_array(j->1), 'lower'));\n END IF;\nEND IF;\n\n-- Special Case when comparing a property in a jsonb field to a string or number using eq\n-- Allows to leverage GIN index on jsonb fields\nIF op = 'eq' THEN\n IF j->0 ? 'property'\n AND jsonb_typeof(j->1) IN ('number','string')\n AND (items_path(j->0->>'property')).eq IS NOT NULL\n THEN\n RETURN format((items_path(j->0->>'property')).eq, j->1);\n END IF;\nEND IF;\n\n\n\nIF op ilike 't_%' or op = 'anyinteracts' THEN\n RETURN temporal_op_query(op, j);\nEND IF;\n\nIF op ilike 's_%' or op = 'intersects' THEN\n RETURN spatial_op_query(op, j);\nEND IF;\n\n\nIF jtype = 'object' THEN\n RAISE NOTICE 'parsing object';\n IF j ? 'property' THEN\n -- Convert the property to be used as an identifier\n return (items_path(j->>'property')).path_txt;\n ELSIF _op IS NULL THEN\n -- Iterate to convert elements in an object where the operator has not been set\n -- Combining with AND\n SELECT\n array_to_string(array_agg(cql_query_op(e.value, e.key)), ' AND ')\n INTO ret\n FROM jsonb_each(j) e;\n RETURN ret;\n END IF;\nEND IF;\n\nIF jtype = 'string' THEN\n RETURN quote_literal(j->>0);\nEND IF;\n\nIF jtype ='number' THEN\n RETURN (j->>0)::numeric;\nEND IF;\n\nIF jtype = 'array' AND op IS NULL THEN\n RAISE NOTICE 'Parsing array into array arg. j: %', j;\n SELECT format($f$ '{%s}'::text[] $f$, string_agg(e,',')) INTO ret FROM jsonb_array_elements_text(j) e;\n RETURN ret;\nEND IF;\n\n\n-- If the type of the passed json is an array\n-- Calculate the arguments that will be passed to functions/operators\nIF jtype = 'array' THEN\n RAISE NOTICE 'Parsing array into args. j: %', j;\n -- If any argument is numeric, cast any text arguments to numeric\n IF j @? '$[*] ? (@.type() == \"number\")' THEN\n SELECT INTO args\n array_agg(concat('(',cql_query_op(e),')::numeric'))\n FROM jsonb_array_elements(j) e;\n ELSE\n SELECT INTO args\n array_agg(cql_query_op(e))\n FROM jsonb_array_elements(j) e;\n END IF;\n --RETURN args;\nEND IF;\nRAISE NOTICE 'ARGS after array cleaning: %', args;\n\nIF op IS NULL THEN\n RETURN args::text[];\nEND IF;\n\nIF args IS NULL OR cardinality(args) < 1 THEN\n RAISE NOTICE 'No Args';\n RETURN '';\nEND IF;\n\nIF op IN ('and','or') THEN\n SELECT\n CONCAT(\n '(',\n array_to_string(args, UPPER(CONCAT(' ',op,' '))),\n ')'\n ) INTO ret\n FROM jsonb_array_elements(j) e;\n RETURN ret;\nEND IF;\n\n-- If the op is in the ops json then run using the template in the json\nIF ops ? op THEN\n RAISE NOTICE 'ARGS: % MAPPED: %',args, array_map_literal(args);\n\n RETURN format(concat('(',ops->>op,')'), VARIADIC args);\nEND IF;\n\nRETURN j->>0;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n/* cql_query_op -- Parses a CQL query operation, recursing when necessary\n IN jsonb -- a subelement from a valid stac query\n IN text -- the operator being used on elements passed in\n RETURNS a SQL fragment to be used in a WHERE clause\n*/\nDROP FUNCTION IF EXISTS cql2_query;\nCREATE OR REPLACE FUNCTION cql2_query(j jsonb, recursion int DEFAULT 0) RETURNS text AS $$\nDECLARE\nargs jsonb := j->'args';\njtype text := jsonb_typeof(j->'args');\nop text := lower(j->>'op');\narg jsonb;\nargtext text;\nargstext text[] := '{}'::text[];\ninobj jsonb;\n_numeric text := '';\nops jsonb :=\n '{\n \"eq\": \"%s = %s\",\n \"lt\": \"%s < %s\",\n \"lte\": \"%s <= %s\",\n \"gt\": \"%s > %s\",\n \"gte\": \"%s >= %s\",\n \"le\": \"%s <= %s\",\n \"ge\": \"%s >= %s\",\n \"=\": \"%s = %s\",\n \"<\": \"%s < %s\",\n \"<=\": \"%s <= %s\",\n \">\": \"%s > %s\",\n \">=\": \"%s >= %s\",\n \"like\": \"%s LIKE %s\",\n \"ilike\": \"%s ILIKE %s\",\n \"+\": \"%s + %s\",\n \"-\": \"%s - %s\",\n \"*\": \"%s * %s\",\n \"/\": \"%s / %s\",\n \"in\": \"%s = ANY (%s)\",\n \"not\": \"NOT (%s)\",\n \"between\": \"%s BETWEEN (%2$s)[1] AND (%2$s)[2]\",\n \"lower\":\" lower(%s)\",\n \"upper\":\" upper(%s)\",\n \"isnull\": \"%s IS NULL\"\n }'::jsonb;\nret text;\n\nBEGIN\nRAISE NOTICE 'j: %s', j;\nIF j ? 'filter' THEN\n RETURN cql2_query(j->'filter');\nEND IF;\n\nIF j ? 'upper' THEN\nRAISE NOTICE 'upper %s',jsonb_build_object(\n 'op', 'upper',\n 'args', jsonb_build_array( j-> 'upper')\n ) ;\n RETURN cql2_query(\n jsonb_build_object(\n 'op', 'upper',\n 'args', jsonb_build_array( j-> 'upper')\n )\n );\nEND IF;\n\nIF j ? 'lower' THEN\n RETURN cql2_query(\n jsonb_build_object(\n 'op', 'lower',\n 'args', jsonb_build_array( j-> 'lower')\n )\n );\nEND IF;\n\nIF j ? 'args' AND jsonb_typeof(args) != 'array' THEN\n args := jsonb_build_array(args);\nEND IF;\n-- END Cases where no further nesting is expected\nIF j ? 'op' THEN\n -- Special case to use JSONB index for equality\n IF op = 'eq'\n AND args->0 ? 'property'\n AND jsonb_typeof(args->1) IN ('number', 'string')\n AND (items_path(args->0->>'property')).eq IS NOT NULL\n THEN\n RETURN format((items_path(args->0->>'property')).eq, args->1);\n END IF;\n\n -- Temporal Query\n IF op ilike 't_%' or op = 'anyinteracts' THEN\n RETURN temporal_op_query(op, args);\n END IF;\n\n -- Spatial Query\n IF op ilike 's_%' or op = 'intersects' THEN\n RETURN spatial_op_query(op, args);\n END IF;\n\n -- In Query - separate into separate eq statements so that we can use eq jsonb optimization\n IF op = 'in' THEN\n RAISE NOTICE '% IN args: %', repeat(' ', recursion), args;\n SELECT INTO inobj\n jsonb_agg(\n jsonb_build_object(\n 'op', 'eq',\n 'args', jsonb_build_array( args->0 , v)\n )\n )\n FROM jsonb_array_elements( args->1) v;\n RETURN cql2_query(jsonb_build_object('op','or','args',inobj));\n END IF;\nEND IF;\n\nIF j ? 'property' THEN\n RETURN (items_path(j->>'property')).path_txt;\nEND IF;\n\nRAISE NOTICE '%jtype: %',repeat(' ', recursion), jtype;\nIF jsonb_typeof(j) = 'number' THEN\n RETURN format('%L::numeric', j->>0);\nEND IF;\n\nIF jsonb_typeof(j) = 'string' THEN\n RETURN quote_literal(j->>0);\nEND IF;\n\nIF jsonb_typeof(j) = 'array' THEN\n IF j @? '$[*] ? (@.type() == \"number\")' THEN\n RETURN CONCAT(quote_literal(textarr(j)::text), '::numeric[]');\n ELSE\n RETURN CONCAT(quote_literal(textarr(j)::text), '::text[]');\n END IF;\nEND IF;\nRAISE NOTICE 'ARGS after array cleaning: %', args;\n\nRAISE NOTICE '%beforeargs op: %, args: %',repeat(' ', recursion), op, args;\nIF j ? 'args' THEN\n FOR arg in SELECT * FROM jsonb_array_elements(args) LOOP\n argtext := cql2_query(arg, recursion + 1);\n RAISE NOTICE '% -- arg: %, argtext: %', repeat(' ', recursion), arg, argtext;\n argstext := argstext || argtext;\n END LOOP;\nEND IF;\nRAISE NOTICE '%afterargs op: %, argstext: %',repeat(' ', recursion), op, argstext;\n\n\nIF op IN ('and', 'or') THEN\n RAISE NOTICE 'inand op: %, argstext: %', op, argstext;\n SELECT\n concat(' ( ',array_to_string(array_agg(e), concat(' ',op,' ')),' ) ')\n INTO ret\n FROM unnest(argstext) e;\n RETURN ret;\nEND IF;\n\nIF ops ? op THEN\n IF argstext[2] ~* 'numeric' THEN\n argstext := ARRAY[concat('(',argstext[1],')::numeric')] || argstext[2:3];\n END IF;\n RETURN format(concat('(',ops->>op,')'), VARIADIC argstext);\nEND IF;\n\nRAISE NOTICE '%op: %, argstext: %',repeat(' ', recursion), op, argstext;\n\nRETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n\nCREATE OR REPLACE FUNCTION cql_to_where(_search jsonb = '{}'::jsonb) RETURNS text AS $$\nDECLARE\nsearch jsonb := _search;\n_where text;\nBEGIN\n\nRAISE NOTICE 'SEARCH CQL Final: %', search;\nIF (search ? 'filter-lang' AND search->>'filter-lang' = 'cql-json') OR get_setting('default-filter-lang', _search->'conf')='cql-json' THEN\n search := query_to_cqlfilter(search);\n search := add_filters_to_cql(search);\n _where := cql_query_op(search->'filter');\nELSE\n _where := cql2_query(search->'filter');\nEND IF;\n\nIF trim(_where) = '' THEN\n _where := NULL;\nEND IF;\n_where := coalesce(_where, ' TRUE ');\nRETURN _where;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION parse_sort_dir(_dir text, reverse boolean default false) RETURNS text AS $$\nWITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n) SELECT\n CASE\n WHEN NOT reverse THEN d\n WHEN d = 'ASC' THEN 'DESC'\n WHEN d = 'DESC' THEN 'ASC'\n END\nFROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION sort_dir_to_op(_dir text, prev boolean default false) RETURNS text AS $$\nWITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n) SELECT\n CASE\n WHEN d = 'ASC' AND prev THEN '<='\n WHEN d = 'DESC' AND prev THEN '>='\n WHEN d = 'ASC' THEN '>='\n WHEN d = 'DESC' THEN '<='\n END\nFROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION field_orderby(p text) RETURNS text AS $$\nWITH t AS (\n SELECT\n replace(trim(substring(indexdef from 'btree \\((.*)\\)')),' ','')as s\n FROM pg_indexes WHERE schemaname='pgstac' AND tablename='items' AND indexdef ~* 'btree' AND indexdef ~* 'properties'\n) SELECT s FROM t WHERE strpos(s, lower(trim(p)))>0;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION sort_sqlorderby(\n _search jsonb DEFAULT NULL,\n reverse boolean DEFAULT FALSE\n) RETURNS text AS $$\nWITH sortby AS (\n SELECT coalesce(_search->'sortby','[{\"field\":\"datetime\", \"direction\":\"desc\"}]') as sort\n), withid AS (\n SELECT CASE\n WHEN sort @? '$[*] ? (@.field == \"id\")' THEN sort\n ELSE sort || '[{\"field\":\"id\", \"direction\":\"desc\"}]'::jsonb\n END as sort\n FROM sortby\n), withid_rows AS (\n SELECT jsonb_array_elements(sort) as value FROM withid\n),sorts AS (\n SELECT\n coalesce(field_orderby((items_path(value->>'field')).path_txt), (items_path(value->>'field')).path) as key,\n parse_sort_dir(value->>'direction', reverse) as dir\n FROM withid_rows\n)\nSELECT array_to_string(\n array_agg(concat(key, ' ', dir)),\n ', '\n) FROM sorts;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_sort_dir(sort_item jsonb) RETURNS text AS $$\nSELECT CASE WHEN sort_item->>'direction' ILIKE 'desc%' THEN 'DESC' ELSE 'ASC' END;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION get_token_filter(_search jsonb = '{}'::jsonb, token_rec jsonb DEFAULT NULL) RETURNS text AS $$\nDECLARE\n token_id text;\n filters text[] := '{}'::text[];\n prev boolean := TRUE;\n field text;\n dir text;\n sort record;\n orfilters text[] := '{}'::text[];\n andfilters text[] := '{}'::text[];\n output text;\n token_where text;\nBEGIN\n -- If no token provided return NULL\n IF token_rec IS NULL THEN\n IF NOT (_search ? 'token' AND\n (\n (_search->>'token' ILIKE 'prev:%')\n OR\n (_search->>'token' ILIKE 'next:%')\n )\n ) THEN\n RETURN NULL;\n END IF;\n prev := (_search->>'token' ILIKE 'prev:%');\n token_id := substr(_search->>'token', 6);\n SELECT to_jsonb(items) INTO token_rec FROM items WHERE id=token_id;\n END IF;\n RAISE NOTICE 'TOKEN ID: %', token_rec->'id';\n\n CREATE TEMP TABLE sorts (\n _row int GENERATED ALWAYS AS IDENTITY NOT NULL,\n _field text PRIMARY KEY,\n _dir text NOT NULL,\n _val text\n ) ON COMMIT DROP;\n\n -- Make sure we only have distinct columns to sort with taking the first one we get\n INSERT INTO sorts (_field, _dir)\n SELECT\n (items_path(value->>'field')).path,\n get_sort_dir(value)\n FROM\n jsonb_array_elements(coalesce(_search->'sortby','[{\"field\":\"datetime\",\"direction\":\"desc\"}]'))\n ON CONFLICT DO NOTHING\n ;\n RAISE NOTICE 'sorts 1: %', (SELECT jsonb_agg(to_json(sorts)) FROM sorts);\n -- Get the first sort direction provided. As the id is a primary key, if there are any\n -- sorts after id they won't do anything, so make sure that id is the last sort item.\n SELECT _dir INTO dir FROM sorts ORDER BY _row ASC LIMIT 1;\n IF EXISTS (SELECT 1 FROM sorts WHERE _field = 'id') THEN\n DELETE FROM sorts WHERE _row > (SELECT _row FROM sorts WHERE _field = 'id' ORDER BY _row ASC);\n ELSE\n INSERT INTO sorts (_field, _dir) VALUES ('id', dir);\n END IF;\n\n -- Add value from looked up item to the sorts table\n UPDATE sorts SET _val=quote_literal(token_rec->>_field);\n\n -- Check if all sorts are the same direction and use row comparison\n -- to filter\n RAISE NOTICE 'sorts 2: %', (SELECT jsonb_agg(to_json(sorts)) FROM sorts);\n\n IF (SELECT count(DISTINCT _dir) FROM sorts) = 1 THEN\n SELECT format(\n '(%s) %s (%s)',\n concat_ws(', ', VARIADIC array_agg(quote_ident(_field))),\n CASE WHEN (prev AND dir = 'ASC') OR (NOT prev AND dir = 'DESC') THEN '<' ELSE '>' END,\n concat_ws(', ', VARIADIC array_agg(_val))\n ) INTO output FROM sorts\n WHERE token_rec ? _field\n ;\n ELSE\n FOR sort IN SELECT * FROM sorts ORDER BY _row asc LOOP\n RAISE NOTICE 'SORT: %', sort;\n IF sort._row = 1 THEN\n orfilters := orfilters || format('(%s %s %s)',\n quote_ident(sort._field),\n CASE WHEN (prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC') THEN '<' ELSE '>' END,\n sort._val\n );\n ELSE\n orfilters := orfilters || format('(%s AND %s %s %s)',\n array_to_string(andfilters, ' AND '),\n quote_ident(sort._field),\n CASE WHEN (prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC') THEN '<' ELSE '>' END,\n sort._val\n );\n\n END IF;\n andfilters := andfilters || format('%s = %s',\n quote_ident(sort._field),\n sort._val\n );\n END LOOP;\n output := array_to_string(orfilters, ' OR ');\n END IF;\n DROP TABLE IF EXISTS sorts;\n token_where := concat('(',coalesce(output,'true'),')');\n IF trim(token_where) = '' THEN\n token_where := NULL;\n END IF;\n RAISE NOTICE 'TOKEN_WHERE: |%|',token_where;\n RETURN token_where;\n END;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION search_tohash(jsonb) RETURNS jsonb AS $$\n SELECT $1 - '{token,limit,context,includes,excludes}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION search_hash(jsonb, jsonb) RETURNS text AS $$\n SELECT md5(concat(search_tohash($1)::text,$2::text));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TABLE IF NOT EXISTS searches(\n hash text GENERATED ALWAYS AS (search_hash(search, metadata)) STORED PRIMARY KEY,\n search jsonb NOT NULL,\n _where text,\n orderby text,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n metadata jsonb DEFAULT '{}'::jsonb NOT NULL\n);\n\nCREATE TABLE IF NOT EXISTS search_wheres(\n _where text PRIMARY KEY,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n statslastupdated timestamptz,\n estimated_count bigint,\n estimated_cost float,\n time_to_estimate float,\n total_count bigint,\n time_to_count float,\n partitions text[]\n);\n\nCREATE INDEX IF NOT EXISTS search_wheres_partitions ON search_wheres USING GIN (partitions);\n\nCREATE OR REPLACE FUNCTION where_stats(inwhere text, updatestats boolean default false, conf jsonb default null) RETURNS search_wheres AS $$\nDECLARE\n t timestamptz;\n i interval;\n explain_json jsonb;\n partitions text[];\n sw search_wheres%ROWTYPE;\nBEGIN\n SELECT * INTO sw FROM search_wheres WHERE _where=inwhere FOR UPDATE;\n\n -- Update statistics if explicitly set, if statistics do not exist, or statistics ttl has expired\n IF NOT updatestats THEN\n RAISE NOTICE 'Checking if update is needed.';\n RAISE NOTICE 'Stats Last Updated: %', sw.statslastupdated;\n RAISE NOTICE 'TTL: %, Age: %', context_stats_ttl(conf), now() - sw.statslastupdated;\n RAISE NOTICE 'Context: %, Existing Total: %', context(conf), sw.total_count;\n IF\n sw.statslastupdated IS NULL\n OR (now() - sw.statslastupdated) > context_stats_ttl(conf)\n OR (context(conf) != 'off' AND sw.total_count IS NULL)\n THEN\n updatestats := TRUE;\n END IF;\n END IF;\n\n sw._where := inwhere;\n sw.lastused := now();\n sw.usecount := coalesce(sw.usecount,0) + 1;\n\n IF NOT updatestats THEN\n UPDATE search_wheres SET\n lastused = sw.lastused,\n usecount = sw.usecount\n WHERE _where = inwhere\n RETURNING * INTO sw\n ;\n RETURN sw;\n END IF;\n -- Use explain to get estimated count/cost and a list of the partitions that would be hit by the query\n t := clock_timestamp();\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s', inwhere)\n INTO explain_json;\n RAISE NOTICE 'Time for just the explain: %', clock_timestamp() - t;\n WITH t AS (\n SELECT j->>0 as p FROM\n jsonb_path_query(\n explain_json,\n 'strict $.**.\"Relation Name\" ? (@ != null)'\n ) j\n ), ordered AS (\n SELECT p FROM t ORDER BY p DESC\n -- SELECT p FROM t JOIN items_partitions\n -- ON (t.p = items_partitions.partition)\n -- ORDER BY pstart DESC\n )\n SELECT array_agg(p) INTO partitions FROM ordered;\n i := clock_timestamp() - t;\n RAISE NOTICE 'Time for explain + join: %', clock_timestamp() - t;\n\n\n\n sw.statslastupdated := now();\n sw.estimated_count := explain_json->0->'Plan'->'Plan Rows';\n sw.estimated_cost := explain_json->0->'Plan'->'Total Cost';\n sw.time_to_estimate := extract(epoch from i);\n sw.partitions := partitions;\n\n -- Do a full count of rows if context is set to on or if auto is set and estimates are low enough\n IF\n context(conf) = 'on'\n OR\n ( context(conf) = 'auto' AND\n (\n sw.estimated_count < context_estimated_count(conf)\n OR\n sw.estimated_cost < context_estimated_cost(conf)\n )\n )\n THEN\n t := clock_timestamp();\n RAISE NOTICE 'Calculating actual count...';\n EXECUTE format(\n 'SELECT count(*) FROM items WHERE %s',\n inwhere\n ) INTO sw.total_count;\n i := clock_timestamp() - t;\n RAISE NOTICE 'Actual Count: % -- %', sw.total_count, i;\n sw.time_to_count := extract(epoch FROM i);\n ELSE\n sw.total_count := NULL;\n sw.time_to_count := NULL;\n END IF;\n\n\n INSERT INTO search_wheres SELECT sw.*\n ON CONFLICT (_where)\n DO UPDATE\n SET\n lastused = sw.lastused,\n usecount = sw.usecount,\n statslastupdated = sw.statslastupdated,\n estimated_count = sw.estimated_count,\n estimated_cost = sw.estimated_cost,\n time_to_estimate = sw.time_to_estimate,\n partitions = sw.partitions,\n total_count = sw.total_count,\n time_to_count = sw.time_to_count\n ;\n RETURN sw;\nEND;\n$$ LANGUAGE PLPGSQL ;\n\n\n\nCREATE OR REPLACE FUNCTION items_count(_where text) RETURNS bigint AS $$\nDECLARE\ncnt bigint;\nBEGIN\nEXECUTE format('SELECT count(*) FROM items WHERE %s', _where) INTO cnt;\nRETURN cnt;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n\n\n\nDROP FUNCTION IF EXISTS search_query;\nCREATE OR REPLACE FUNCTION search_query(\n _search jsonb = '{}'::jsonb,\n updatestats boolean = false,\n _metadata jsonb = '{}'::jsonb\n) RETURNS searches AS $$\nDECLARE\n search searches%ROWTYPE;\n pexplain jsonb;\n t timestamptz;\n i interval;\nBEGIN\nSELECT * INTO search FROM searches\nWHERE hash=search_hash(_search, _metadata) FOR UPDATE;\n\n-- Calculate the where clause if not already calculated\nIF search._where IS NULL THEN\n search._where := cql_to_where(_search);\nEND IF;\n\n-- Calculate the order by clause if not already calculated\nIF search.orderby IS NULL THEN\n search.orderby := sort_sqlorderby(_search);\nEND IF;\n\nPERFORM where_stats(search._where, updatestats, _search->'conf');\n\nsearch.lastused := now();\nsearch.usecount := coalesce(search.usecount, 0) + 1;\nINSERT INTO searches (search, _where, orderby, lastused, usecount, metadata)\nVALUES (_search, search._where, search.orderby, search.lastused, search.usecount, _metadata)\nON CONFLICT (hash) DO\nUPDATE SET\n _where = EXCLUDED._where,\n orderby = EXCLUDED.orderby,\n lastused = EXCLUDED.lastused,\n usecount = EXCLUDED.usecount,\n metadata = EXCLUDED.metadata\nRETURNING * INTO search\n;\nRETURN search;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n\nCREATE OR REPLACE FUNCTION search(_search jsonb = '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n searches searches%ROWTYPE;\n _where text;\n token_where text;\n full_where text;\n orderby text;\n query text;\n token_type text := substr(_search->>'token',1,4);\n _limit int := coalesce((_search->>'limit')::int, 10);\n curs refcursor;\n cntr int := 0;\n iter_record items%ROWTYPE;\n first_record items%ROWTYPE;\n last_record items%ROWTYPE;\n out_records jsonb := '[]'::jsonb;\n prev_query text;\n next text;\n prev_id text;\n has_next boolean := false;\n has_prev boolean := false;\n prev text;\n total_count bigint;\n context jsonb;\n collection jsonb;\n includes text[];\n excludes text[];\n exit_flag boolean := FALSE;\n batches int := 0;\n timer timestamptz := clock_timestamp();\n pstart timestamptz;\n pend timestamptz;\n pcurs refcursor;\n search_where search_wheres%ROWTYPE;\nBEGIN\nsearches := search_query(_search);\n_where := searches._where;\norderby := searches.orderby;\nsearch_where := where_stats(_where);\ntotal_count := coalesce(search_where.total_count, search_where.estimated_count);\n\n\nIF token_type='prev' THEN\n token_where := get_token_filter(_search, null::jsonb);\n orderby := sort_sqlorderby(_search, TRUE);\nEND IF;\nIF token_type='next' THEN\n token_where := get_token_filter(_search, null::jsonb);\nEND IF;\n\nfull_where := concat_ws(' AND ', _where, token_where);\nRAISE NOTICE 'FULL QUERY % %', full_where, clock_timestamp()-timer;\ntimer := clock_timestamp();\n\nCREATE TEMP TABLE results (content jsonb) ON COMMIT DROP;\n\n\nFOR query IN SELECT partition_queries(full_where, orderby, search_where.partitions) LOOP\n timer := clock_timestamp();\n query := format('%s LIMIT %s', query, _limit + 1);\n RAISE NOTICE 'Partition Query: %', query;\n batches := batches + 1;\n -- curs = create_cursor(query);\n OPEN curs FOR EXECUTE query;\n LOOP\n FETCH curs into iter_record;\n EXIT WHEN NOT FOUND;\n cntr := cntr + 1;\n last_record := iter_record;\n IF cntr = 1 THEN\n first_record := last_record;\n END IF;\n IF cntr <= _limit THEN\n INSERT INTO results (content) VALUES (last_record.content);\n -- out_records := out_records || last_record.content;\n\n ELSIF cntr > _limit THEN\n has_next := true;\n exit_flag := true;\n EXIT;\n END IF;\n END LOOP;\n CLOSE curs;\n RAISE NOTICE 'Query took %. Total Time %', clock_timestamp()-timer, ftime();\n timer := clock_timestamp();\n EXIT WHEN exit_flag;\nEND LOOP;\nRAISE NOTICE 'Scanned through % partitions.', batches;\n\nSELECT jsonb_agg(content) INTO out_records FROM results;\n\nDROP TABLE results;\n\n\n-- Flip things around if this was the result of a prev token query\nIF token_type='prev' THEN\n out_records := flip_jsonb_array(out_records);\n first_record := last_record;\nEND IF;\n\n-- If this query has a token, see if there is data before the first record\nIF _search ? 'token' THEN\n prev_query := format(\n 'SELECT 1 FROM items WHERE %s LIMIT 1',\n concat_ws(\n ' AND ',\n _where,\n trim(get_token_filter(_search, to_jsonb(first_record)))\n )\n );\n RAISE NOTICE 'Query to get previous record: % --- %', prev_query, first_record;\n EXECUTE prev_query INTO has_prev;\n IF FOUND and has_prev IS NOT NULL THEN\n RAISE NOTICE 'Query results from prev query: %', has_prev;\n has_prev := TRUE;\n END IF;\nEND IF;\nhas_prev := COALESCE(has_prev, FALSE);\n\nRAISE NOTICE 'token_type: %, has_next: %, has_prev: %', token_type, has_next, has_prev;\nIF has_prev THEN\n prev := out_records->0->>'id';\nEND IF;\nIF has_next OR token_type='prev' THEN\n next := out_records->-1->>'id';\nEND IF;\n\n\n-- include/exclude any fields following fields extension\nIF _search ? 'fields' THEN\n IF _search->'fields' ? 'exclude' THEN\n excludes=textarr(_search->'fields'->'exclude');\n END IF;\n IF _search->'fields' ? 'include' THEN\n includes=textarr(_search->'fields'->'include');\n IF array_length(includes, 1)>0 AND NOT 'id' = ANY (includes) THEN\n includes = includes || '{id}';\n END IF;\n END IF;\n SELECT jsonb_agg(filter_jsonb(row, includes, excludes)) INTO out_records FROM jsonb_array_elements(out_records) row;\nEND IF;\n\nIF context(_search->'conf') != 'off' THEN\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'matched', total_count,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\nELSE\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\nEND IF;\n\ncollection := jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb),\n 'next', next,\n 'prev', prev,\n 'context', context\n);\n\nRETURN collection;\nEND;\n$$ LANGUAGE PLPGSQL\nSET jit TO off\n;\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION tileenvelope(zoom int, x int, y int) RETURNS geometry AS $$\nWITH t AS (\n SELECT\n 20037508.3427892 as merc_max,\n -20037508.3427892 as merc_min,\n (2 * 20037508.3427892) / (2 ^ zoom) as tile_size\n)\nSELECT st_makeenvelope(\n merc_min + (tile_size * x),\n merc_max - (tile_size * (y + 1)),\n merc_min + (tile_size * (x + 1)),\n merc_max - (tile_size * y),\n 3857\n) FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;DROP FUNCTION IF EXISTS mercgrid;\n\n\nCREATE OR REPLACE FUNCTION ftime() RETURNS interval as $$\nSELECT age(clock_timestamp(), transaction_timestamp());\n$$ LANGUAGE SQL;\nSET SEARCH_PATH to pgstac, public;\n\nDROP FUNCTION IF EXISTS geometrysearch;\nCREATE OR REPLACE FUNCTION geometrysearch(\n IN geom geometry,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE, -- Return as soon as the passed in geometry is full covered\n IN skipcovered boolean DEFAULT TRUE -- Skip any items that would show up completely under the previous items\n) RETURNS jsonb AS $$\nDECLARE\n search searches%ROWTYPE;\n curs refcursor;\n _where text;\n query text;\n iter_record items%ROWTYPE;\n out_records jsonb[] := '{}'::jsonb[];\n exit_flag boolean := FALSE;\n counter int := 1;\n scancounter int := 1;\n remaining_limit int := _scanlimit;\n tilearea float;\n unionedgeom geometry;\n clippedgeom geometry;\n unionedgeom_area float := 0;\n prev_area float := 0;\n excludes text[];\n includes text[];\n\nBEGIN\n -- If skipcovered is true then you will always want to exit when the passed in geometry is full\n IF skipcovered THEN\n exitwhenfull := TRUE;\n END IF;\n\n SELECT * INTO search FROM searches WHERE hash=queryhash;\n\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Search with Query Hash % Not Found', queryhash;\n END IF;\n\n tilearea := st_area(geom);\n _where := format('%s AND st_intersects(geometry, %L::geometry)', search._where, geom);\n\n IF fields IS NOT NULL THEN\n IF fields ? 'fields' THEN\n fields := fields->'fields';\n END IF;\n IF fields ? 'exclude' THEN\n excludes=textarr(fields->'exclude');\n END IF;\n IF fields ? 'include' THEN\n includes=textarr(fields->'include');\n IF array_length(includes, 1)>0 AND NOT 'id' = ANY (includes) THEN\n includes = includes || '{id}';\n END IF;\n END IF;\n END IF;\n RAISE NOTICE 'fields: %, includes: %, excludes: %', fields, includes, excludes;\n\n FOR query IN SELECT * FROM partition_queries(_where, search.orderby) LOOP\n query := format('%s LIMIT %L', query, remaining_limit);\n RAISE NOTICE '%', query;\n curs = create_cursor(query);\n LOOP\n FETCH curs INTO iter_record;\n EXIT WHEN NOT FOUND;\n IF exitwhenfull OR skipcovered THEN -- If we are not using exitwhenfull or skipcovered, we do not need to do expensive geometry operations\n clippedgeom := st_intersection(geom, iter_record.geometry);\n\n IF unionedgeom IS NULL THEN\n unionedgeom := clippedgeom;\n ELSE\n unionedgeom := st_union(unionedgeom, clippedgeom);\n END IF;\n\n unionedgeom_area := st_area(unionedgeom);\n\n IF skipcovered AND prev_area = unionedgeom_area THEN\n scancounter := scancounter + 1;\n CONTINUE;\n END IF;\n\n prev_area := unionedgeom_area;\n\n RAISE NOTICE '% % % %', unionedgeom_area/tilearea, counter, scancounter, ftime();\n END IF;\n\n IF fields IS NOT NULL THEN\n out_records := out_records || filter_jsonb(iter_record.content, includes, excludes);\n ELSE\n out_records := out_records || iter_record.content;\n END IF;\n IF counter >= _limit\n OR scancounter > _scanlimit\n OR ftime() > _timelimit\n OR (exitwhenfull AND unionedgeom_area >= tilearea)\n THEN\n exit_flag := TRUE;\n EXIT;\n END IF;\n counter := counter + 1;\n scancounter := scancounter + 1;\n\n END LOOP;\n EXIT WHEN exit_flag;\n remaining_limit := _scanlimit - scancounter;\n END LOOP;\n\n RETURN jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', array_to_json(out_records)::jsonb\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDROP FUNCTION IF EXISTS geojsonsearch;\nCREATE OR REPLACE FUNCTION geojsonsearch(\n IN geojson jsonb,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_geomfromgeojson(geojson),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS xyzsearch;\nCREATE OR REPLACE FUNCTION xyzsearch(\n IN _x int,\n IN _y int,\n IN _z int,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_transform(tileenvelope(_z, _x, _y), 4326),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\nSELECT set_version('0.4.0');\n" }, { "path": "src/pgstac/migrations/pgstac.0.4.1-0.4.2.sql", "content": "SET SEARCH_PATH to pgstac, public;\nset check_function_bodies = off;\n\nCREATE OR REPLACE FUNCTION pgstac.cql2_query(j jsonb, recursion integer DEFAULT 0)\n RETURNS text\n LANGUAGE plpgsql\nAS $function$\nDECLARE\nargs jsonb := j->'args';\njtype text := jsonb_typeof(j->'args');\nop text := lower(j->>'op');\narg jsonb;\nargtext text;\nargstext text[] := '{}'::text[];\ninobj jsonb;\n_numeric text := '';\nops jsonb :=\n '{\n \"eq\": \"%s = %s\",\n \"lt\": \"%s < %s\",\n \"lte\": \"%s <= %s\",\n \"gt\": \"%s > %s\",\n \"gte\": \"%s >= %s\",\n \"le\": \"%s <= %s\",\n \"ge\": \"%s >= %s\",\n \"=\": \"%s = %s\",\n \"<\": \"%s < %s\",\n \"<=\": \"%s <= %s\",\n \">\": \"%s > %s\",\n \">=\": \"%s >= %s\",\n \"like\": \"%s LIKE %s\",\n \"ilike\": \"%s ILIKE %s\",\n \"+\": \"%s + %s\",\n \"-\": \"%s - %s\",\n \"*\": \"%s * %s\",\n \"/\": \"%s / %s\",\n \"in\": \"%s = ANY (%s)\",\n \"not\": \"NOT (%s)\",\n \"between\": \"%s BETWEEN (%2$s)[1] AND (%2$s)[2]\",\n \"lower\":\" lower(%s)\",\n \"upper\":\" upper(%s)\",\n \"isnull\": \"%s IS NULL\"\n }'::jsonb;\nret text;\n\nBEGIN\nRAISE NOTICE 'j: %s', j;\nIF j ? 'filter' THEN\n RETURN cql2_query(j->'filter');\nEND IF;\n\nIF j ? 'upper' THEN\nRAISE NOTICE 'upper %s',jsonb_build_object(\n 'op', 'upper',\n 'args', jsonb_build_array( j-> 'upper')\n ) ;\n RETURN cql2_query(\n jsonb_build_object(\n 'op', 'upper',\n 'args', jsonb_build_array( j-> 'upper')\n )\n );\nEND IF;\n\nIF j ? 'lower' THEN\n RETURN cql2_query(\n jsonb_build_object(\n 'op', 'lower',\n 'args', jsonb_build_array( j-> 'lower')\n )\n );\nEND IF;\n\nIF j ? 'args' AND jsonb_typeof(args) != 'array' THEN\n args := jsonb_build_array(args);\nEND IF;\n-- END Cases where no further nesting is expected\nIF j ? 'op' THEN\n -- Special case to use JSONB index for equality\n IF op = 'eq'\n AND args->0 ? 'property'\n AND jsonb_typeof(args->1) IN ('number', 'string')\n AND (items_path(args->0->>'property')).eq IS NOT NULL\n THEN\n RETURN format((items_path(args->0->>'property')).eq, args->1);\n END IF;\n\n -- Temporal Query\n IF op ilike 't_%' or op = 'anyinteracts' THEN\n RETURN temporal_op_query(op, args);\n END IF;\n\n -- Spatial Query\n IF op ilike 's_%' or op = 'intersects' THEN\n RETURN spatial_op_query(op, args);\n END IF;\n\n -- In Query - separate into separate eq statements so that we can use eq jsonb optimization\n IF op = 'in' THEN\n RAISE NOTICE '% IN args: %', repeat(' ', recursion), args;\n SELECT INTO inobj\n jsonb_agg(\n jsonb_build_object(\n 'op', 'eq',\n 'args', jsonb_build_array( args->0 , v)\n )\n )\n FROM jsonb_array_elements( args->1) v;\n RETURN cql2_query(jsonb_build_object('op','or','args',inobj));\n END IF;\nEND IF;\n\nIF j ? 'property' THEN\n RETURN (items_path(j->>'property')).path_txt;\nEND IF;\n\nIF j ? 'timestamp' THEN\n RETURN quote_literal(j->>'timestamp');\nEND IF;\n\nRAISE NOTICE '%jtype: %',repeat(' ', recursion), jtype;\nIF jsonb_typeof(j) = 'number' THEN\n RETURN format('%L::numeric', j->>0);\nEND IF;\n\nIF jsonb_typeof(j) = 'string' THEN\n RETURN quote_literal(j->>0);\nEND IF;\n\nIF jsonb_typeof(j) = 'array' THEN\n IF j @? '$[*] ? (@.type() == \"number\")' THEN\n RETURN CONCAT(quote_literal(textarr(j)::text), '::numeric[]');\n ELSE\n RETURN CONCAT(quote_literal(textarr(j)::text), '::text[]');\n END IF;\nEND IF;\nRAISE NOTICE 'ARGS after array cleaning: %', args;\n\nRAISE NOTICE '%beforeargs op: %, args: %',repeat(' ', recursion), op, args;\nIF j ? 'args' THEN\n FOR arg in SELECT * FROM jsonb_array_elements(args) LOOP\n argtext := cql2_query(arg, recursion + 1);\n RAISE NOTICE '% -- arg: %, argtext: %', repeat(' ', recursion), arg, argtext;\n argstext := argstext || argtext;\n END LOOP;\nEND IF;\nRAISE NOTICE '%afterargs op: %, argstext: %',repeat(' ', recursion), op, argstext;\n\n\nIF op IN ('and', 'or') THEN\n RAISE NOTICE 'inand op: %, argstext: %', op, argstext;\n SELECT\n concat(' ( ',array_to_string(array_agg(e), concat(' ',op,' ')),' ) ')\n INTO ret\n FROM unnest(argstext) e;\n RETURN ret;\nEND IF;\n\nIF ops ? op THEN\n IF argstext[2] ~* 'numeric' THEN\n argstext := ARRAY[concat('(',argstext[1],')::numeric')] || argstext[2:3];\n END IF;\n RETURN format(concat('(',ops->>op,')'), VARIADIC argstext);\nEND IF;\n\nRAISE NOTICE '%op: %, argstext: %',repeat(' ', recursion), op, argstext;\n\nRETURN NULL;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.parse_dtrange(_indate jsonb, OUT _tstzrange tstzrange)\n RETURNS tstzrange\n LANGUAGE sql\n IMMUTABLE PARALLEL SAFE\nAS $function$\nWITH t AS (\n SELECT CASE\n WHEN _indate ? 'timestamp' THEN\n ARRAY[_indate->>'timestamp', 'infinity']\n WHEN _indate ? 'interval' THEN\n textarr(_indate->'interval')\n WHEN jsonb_typeof(_indate) = 'array' THEN\n textarr(_indate)\n ELSE\n regexp_split_to_array(\n btrim(_indate::text,'\"'),\n '/'\n )\n END AS arr\n)\n, t1 AS (\n SELECT\n CASE\n WHEN array_upper(arr,1) = 1 OR arr[1] = '..' OR arr[1] IS NULL THEN '-infinity'::timestamptz\n ELSE arr[1]::timestamptz\n END AS st,\n CASE\n WHEN array_upper(arr,1) = 1 THEN arr[1]::timestamptz\n WHEN arr[2] = '..' OR arr[2] IS NULL THEN 'infinity'::timestamptz\n ELSE arr[2]::timestamptz\n END AS et\n FROM t\n)\nSELECT\n tstzrange(st,et)\nFROM t1;\n$function$\n;\n\n\n\nSELECT set_version('0.4.2');\n" }, { "path": "src/pgstac/migrations/pgstac.0.4.1.sql", "content": "CREATE EXTENSION IF NOT EXISTS postgis;\nCREATE SCHEMA IF NOT EXISTS pgstac;\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE TABLE migrations (\n version text PRIMARY KEY,\n datetime timestamptz DEFAULT clock_timestamp() NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION get_version() RETURNS text AS $$\n SELECT version FROM migrations ORDER BY datetime DESC, version DESC LIMIT 1;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION set_version(text) RETURNS text AS $$\n INSERT INTO migrations (version) VALUES ($1)\n ON CONFLICT DO NOTHING\n RETURNING version;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\n\nCREATE TABLE IF NOT EXISTS pgstac_settings (\n name text PRIMARY KEY,\n value text NOT NULL\n);\n\nINSERT INTO pgstac_settings (name, value) VALUES\n ('context', 'off'),\n ('context_estimated_count', '100000'),\n ('context_estimated_cost', '1000000'),\n ('context_stats_ttl', '1 day'),\n ('default-filter-lang', 'cql2-json')\nON CONFLICT DO NOTHING\n;\n\n\nCREATE OR REPLACE FUNCTION get_setting(IN _setting text, IN conf jsonb DEFAULT NULL) RETURNS text AS $$\nSELECT COALESCE(\n conf->>_setting,\n current_setting(concat('pgstac.',_setting), TRUE),\n (SELECT value FROM pgstac_settings WHERE name=_setting)\n);\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context();\nCREATE OR REPLACE FUNCTION context(conf jsonb DEFAULT NULL) RETURNS text AS $$\n SELECT get_setting('context', conf);\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_estimated_count();\nCREATE OR REPLACE FUNCTION context_estimated_count(conf jsonb DEFAULT NULL) RETURNS int AS $$\n SELECT get_setting('context_estimated_count', conf)::int;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_estimated_cost();\nCREATE OR REPLACE FUNCTION context_estimated_cost(conf jsonb DEFAULT NULL) RETURNS float AS $$\n SELECT get_setting('context_estimated_cost', conf)::float;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_stats_ttl();\nCREATE OR REPLACE FUNCTION context_stats_ttl(conf jsonb DEFAULT NULL) RETURNS interval AS $$\n SELECT get_setting('context_stats_ttl', conf)::interval;\n$$ LANGUAGE SQL;\n\n\nCREATE OR REPLACE FUNCTION notice(VARIADIC text[]) RETURNS boolean AS $$\nDECLARE\ndebug boolean := current_setting('pgstac.debug', true);\nBEGIN\n IF debug THEN\n RAISE NOTICE 'NOTICE FROM FUNC: % >>>>> %', concat_ws(' | ', $1), clock_timestamp();\n RETURN TRUE;\n END IF;\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION empty_arr(ANYARRAY) RETURNS BOOLEAN AS $$\nSELECT CASE\n WHEN $1 IS NULL THEN TRUE\n WHEN cardinality($1)<1 THEN TRUE\nELSE FALSE\nEND;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_map_ident(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_ident(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_map_literal(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_literal(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION estimated_count(_where text) RETURNS bigint AS $$\nDECLARE\nrec record;\nrows bigint;\nBEGIN\n FOR rec in EXECUTE format(\n $q$\n EXPLAIN SELECT 1 FROM items WHERE %s\n $q$,\n _where)\n LOOP\n rows := substring(rec.\"QUERY PLAN\" FROM ' rows=([[:digit:]]+)');\n EXIT WHEN rows IS NOT NULL;\n END LOOP;\n\n RETURN rows;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION array_reverse(anyarray) RETURNS anyarray AS $$\nSELECT ARRAY(\n SELECT $1[i]\n FROM generate_subscripts($1,1) AS s(i)\n ORDER BY i DESC\n);\n$$ LANGUAGE 'sql' STRICT IMMUTABLE;\n/* converts a jsonb text array to a pg text[] array */\nCREATE OR REPLACE FUNCTION textarr(_js jsonb)\n RETURNS text[] AS $$\n SELECT\n CASE jsonb_typeof(_js)\n WHEN 'array' THEN ARRAY(SELECT jsonb_array_elements_text(_js))\n ELSE ARRAY[_js->>0]\n END\n;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION jsonb_paths (IN jdata jsonb, OUT path text[], OUT value jsonb) RETURNS\nSETOF RECORD AS $$\nwith recursive extract_all as\n(\n select\n ARRAY[key]::text[] as path,\n value\n FROM jsonb_each(jdata)\nunion all\n select\n path || coalesce(obj_key, (arr_key- 1)::text),\n coalesce(obj_value, arr_value)\n from extract_all\n left join lateral\n jsonb_each(case jsonb_typeof(value) when 'object' then value end)\n as o(obj_key, obj_value)\n on jsonb_typeof(value) = 'object'\n left join lateral\n jsonb_array_elements(case jsonb_typeof(value) when 'array' then value end)\n with ordinality as a(arr_value, arr_key)\n on jsonb_typeof(value) = 'array'\n where obj_key is not null or arr_key is not null\n)\nselect *\nfrom extract_all;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION jsonb_obj_paths (IN jdata jsonb, OUT path text[], OUT value jsonb) RETURNS\nSETOF RECORD AS $$\nwith recursive extract_all as\n(\n select\n ARRAY[key]::text[] as path,\n value\n FROM jsonb_each(jdata)\nunion all\n select\n path || obj_key,\n obj_value\n from extract_all\n left join lateral\n jsonb_each(case jsonb_typeof(value) when 'object' then value end)\n as o(obj_key, obj_value)\n on jsonb_typeof(value) = 'object'\n where obj_key is not null\n)\nselect *\nfrom extract_all;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION jsonb_val_paths (IN jdata jsonb, OUT path text[], OUT value jsonb) RETURNS\nSETOF RECORD AS $$\nSELECT * FROM jsonb_obj_paths(jdata) WHERE jsonb_typeof(value) not in ('object','array');\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION path_includes(IN path text[], IN includes text[]) RETURNS BOOLEAN AS $$\nWITH t AS (SELECT unnest(includes) i)\nSELECT EXISTS (\n SELECT 1 FROM t WHERE path @> string_to_array(trim(i), '.')\n);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION path_excludes(IN path text[], IN excludes text[]) RETURNS BOOLEAN AS $$\nWITH t AS (SELECT unnest(excludes) e)\nSELECT NOT EXISTS (\n SELECT 1 FROM t WHERE path @> string_to_array(trim(e), '.')\n);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION jsonb_obj_paths_filtered (\n IN jdata jsonb,\n IN includes text[] DEFAULT ARRAY[]::text[],\n IN excludes text[] DEFAULT ARRAY[]::text[],\n OUT path text[],\n OUT value jsonb\n) RETURNS\nSETOF RECORD AS $$\nSELECT path, value\nFROM jsonb_obj_paths(jdata)\nWHERE\n CASE WHEN cardinality(includes) > 0 THEN path_includes(path, includes) ELSE TRUE END\n AND\n path_excludes(path, excludes)\n\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION filter_jsonb(\n IN jdata jsonb,\n IN includes text[] DEFAULT ARRAY[]::text[],\n IN excludes text[] DEFAULT ARRAY[]::text[]\n) RETURNS jsonb AS $$\nDECLARE\nrec RECORD;\noutj jsonb := '{}'::jsonb;\ncreated_paths text[] := '{}'::text[];\nBEGIN\n\nIF empty_arr(includes) AND empty_arr(excludes) THEN\nRAISE NOTICE 'no filter';\n RETURN jdata;\nEND IF;\nFOR rec in\nSELECT * FROM jsonb_obj_paths_filtered(jdata, includes, excludes)\nWHERE jsonb_typeof(value) != 'object'\nLOOP\n IF array_length(rec.path,1)>1 THEN\n FOR i IN 1..(array_length(rec.path,1)-1) LOOP\n IF NOT array_to_string(rec.path[1:i],'.') = ANY (created_paths) THEN\n outj := jsonb_set(outj, rec.path[1:i],'{}', true);\n created_paths := created_paths || array_to_string(rec.path[1:i],'.');\n END IF;\n END LOOP;\n END IF;\n outj := jsonb_set(outj, rec.path, rec.value, true);\n created_paths := created_paths || array_to_string(rec.path,'.');\nEND LOOP;\nRETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION flip_jsonb_array(j jsonb) RETURNS jsonb AS $$\nSELECT jsonb_agg(value) FROM (SELECT value FROM jsonb_array_elements(j) WITH ORDINALITY ORDER BY ordinality DESC) as t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n/* Functions to create an iterable of cursors over partitions. */\nCREATE OR REPLACE FUNCTION create_cursor(q text) RETURNS refcursor AS $$\nDECLARE\n curs refcursor;\nBEGIN\n OPEN curs FOR EXECUTE q;\n RETURN curs;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDROP FUNCTION IF EXISTS partition_queries;\nCREATE OR REPLACE FUNCTION partition_queries(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN partitions text[] DEFAULT '{items}'\n) RETURNS SETOF text AS $$\nDECLARE\n partition_query text;\n query text;\n p text;\n cursors refcursor;\n dstart timestamptz;\n dend timestamptz;\n step interval := '10 weeks'::interval;\nBEGIN\n\nIF _orderby ILIKE 'datetime d%' THEN\n partitions := partitions;\nELSIF _orderby ILIKE 'datetime a%' THEN\n partitions := array_reverse(partitions);\nELSE\n query := format($q$\n SELECT * FROM items\n WHERE %s\n ORDER BY %s\n $q$, _where, _orderby\n );\n\n RETURN NEXT query;\n RETURN;\nEND IF;\nRAISE NOTICE 'PARTITIONS ---> %',partitions;\nIF cardinality(partitions) > 0 THEN\n FOREACH p IN ARRAY partitions\n --EXECUTE partition_query\n LOOP\n query := format($q$\n SELECT * FROM %I\n WHERE %s\n ORDER BY %s\n $q$,\n p,\n _where,\n _orderby\n );\n RETURN NEXT query;\n END LOOP;\nEND IF;\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION partition_cursor(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC'\n) RETURNS SETOF refcursor AS $$\nDECLARE\n partition_query text;\n query text;\n p record;\n cursors refcursor;\nBEGIN\nFOR query IN SELECT * FROM partition_queries(_where, _orderby) LOOP\n RETURN NEXT create_cursor(query);\nEND LOOP;\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION partition_count(\n IN _where text DEFAULT 'TRUE'\n) RETURNS bigint AS $$\nDECLARE\n partition_query text;\n query text;\n p record;\n subtotal bigint;\n total bigint := 0;\nBEGIN\npartition_query := format($q$\n SELECT partition, tstzrange\n FROM items_partitions\n ORDER BY tstzrange DESC;\n$q$);\nRAISE NOTICE 'Partition Query: %', partition_query;\nFOR p IN\n EXECUTE partition_query\nLOOP\n query := format($q$\n SELECT count(*) FROM items\n WHERE datetime BETWEEN %L AND %L AND %s\n $q$, lower(p.tstzrange), upper(p.tstzrange), _where\n );\n RAISE NOTICE 'Query %', query;\n RAISE NOTICE 'Partition %, Count %, Total %',p.partition, subtotal, total;\n EXECUTE query INTO subtotal;\n total := subtotal + total;\nEND LOOP;\nRETURN total;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION drop_partition_constraints(IN partition text) RETURNS VOID AS $$\nDECLARE\n q text;\n end_datetime_constraint text := concat(partition, '_end_datetime_constraint');\n collections_constraint text := concat(partition, '_collections_constraint');\nBEGIN\n q := format($q$\n ALTER TABLE %I\n DROP CONSTRAINT IF EXISTS %I,\n DROP CONSTRAINT IF EXISTS %I;\n $q$,\n partition,\n end_datetime_constraint,\n collections_constraint\n );\n\n EXECUTE q;\n RETURN;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDROP FUNCTION IF EXISTS partition_checks;\nCREATE OR REPLACE FUNCTION partition_checks(\n IN partition text,\n OUT min_datetime timestamptz,\n OUT max_datetime timestamptz,\n OUT min_end_datetime timestamptz,\n OUT max_end_datetime timestamptz,\n OUT collections text[],\n OUT cnt bigint\n) RETURNS RECORD AS $$\nDECLARE\nq text;\nend_datetime_constraint text := concat(partition, '_end_datetime_constraint');\ncollections_constraint text := concat(partition, '_collections_constraint');\nBEGIN\nRAISE NOTICE 'CREATING CONSTRAINTS FOR %', partition;\nq := format($q$\n SELECT\n min(datetime),\n max(datetime),\n min(end_datetime),\n max(end_datetime),\n array_agg(DISTINCT collection_id),\n count(*)\n FROM %I;\n $q$,\n partition\n);\nEXECUTE q INTO min_datetime, max_datetime, min_end_datetime, max_end_datetime, collections, cnt;\nRAISE NOTICE '% % % % % % %', min_datetime, max_datetime, min_end_datetime, max_end_datetime, collections, cnt, ftime();\nIF cnt IS NULL or cnt = 0 THEN\n RAISE NOTICE 'Partition % is empty, removing...', partition;\n q := format($q$\n DROP TABLE IF EXISTS %I;\n $q$, partition\n );\n EXECUTE q;\n RETURN;\nEND IF;\nRAISE NOTICE 'Running Constraint DDL %', ftime();\nq := format($q$\n ALTER TABLE %I\n DROP CONSTRAINT IF EXISTS %I,\n ADD CONSTRAINT %I\n check((end_datetime >= %L) AND (end_datetime <= %L)) NOT VALID,\n DROP CONSTRAINT IF EXISTS %I,\n ADD CONSTRAINT %I\n check((collection_id = ANY(%L))) NOT VALID;\n $q$,\n partition,\n end_datetime_constraint,\n end_datetime_constraint,\n min_end_datetime,\n max_end_datetime,\n collections_constraint,\n collections_constraint,\n collections,\n partition\n);\nRAISE NOTICE 'q: %', q;\n\nEXECUTE q;\nRAISE NOTICE 'Returning %', ftime();\nRETURN;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION validate_constraints() RETURNS VOID AS $$\nDECLARE\nq text;\nBEGIN\nFOR q IN\n SELECT FORMAT(\n 'ALTER TABLE %I.%I.%I VALIDATE CONSTRAINT %I;',\n current_database(),\n nsp.nspname,\n cls.relname,\n con.conname\n )\n FROM pg_constraint AS con\n JOIN pg_class AS cls\n ON con.conrelid = cls.oid\n JOIN pg_namespace AS nsp\n ON cls.relnamespace = nsp.oid\n WHERE convalidated IS FALSE\n AND nsp.nspname = 'pgstac'\nLOOP\n EXECUTE q;\nEND LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n/* looks for a geometry in a stac item first from geometry and falling back to bbox */\nCREATE OR REPLACE FUNCTION stac_geom(value jsonb) RETURNS geometry AS $$\nSELECT\n CASE\n WHEN value->>'geometry' IS NOT NULL THEN\n ST_GeomFromGeoJSON(value->>'geometry')\n WHEN value->>'bbox' IS NOT NULL THEN\n ST_MakeEnvelope(\n (value->'bbox'->>0)::float,\n (value->'bbox'->>1)::float,\n (value->'bbox'->>2)::float,\n (value->'bbox'->>3)::float,\n 4326\n )\n ELSE NULL\n END as geometry\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION stac_datetime(value jsonb) RETURNS timestamptz AS $$\nSELECT COALESCE(\n (value->'properties'->>'datetime')::timestamptz,\n (value->'properties'->>'start_datetime')::timestamptz\n);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_end_datetime(value jsonb) RETURNS timestamptz AS $$\nSELECT COALESCE(\n (value->'properties'->>'datetime')::timestamptz,\n (value->'properties'->>'end_datetime')::timestamptz\n);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\n\nCREATE OR REPLACE FUNCTION stac_daterange(value jsonb) RETURNS tstzrange AS $$\nSELECT tstzrange(stac_datetime(value),stac_end_datetime(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE TABLE IF NOT EXISTS collections (\n id VARCHAR GENERATED ALWAYS AS (content->>'id') STORED PRIMARY KEY,\n content JSONB\n);\n\nCREATE OR REPLACE FUNCTION create_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection(data jsonb) RETURNS VOID AS $$\nDECLARE\nout collections%ROWTYPE;\nBEGIN\n UPDATE collections SET content=data WHERE id = data->>'id' RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ON CONFLICT (id) DO\n UPDATE\n SET content=EXCLUDED.content\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION delete_collection(_id text) RETURNS VOID AS $$\nDECLARE\nout collections%ROWTYPE;\nBEGIN\n DELETE FROM collections WHERE id = _id RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION get_collection(id text) RETURNS jsonb AS $$\nSELECT content FROM collections\nWHERE id=$1\n;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION all_collections() RETURNS jsonb AS $$\nSELECT jsonb_agg(content) FROM collections;\n;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE TABLE items (\n id text NOT NULL,\n geometry geometry NOT NULL,\n collection_id text NOT NULL,\n datetime timestamptz NOT NULL,\n end_datetime timestamptz NOT NULL,\n properties jsonb NOT NULL,\n content JSONB NOT NULL\n)\nPARTITION BY RANGE (datetime)\n;\n\nCREATE OR REPLACE FUNCTION properties_idx (IN content jsonb) RETURNS jsonb AS $$\n with recursive extract_all as\n (\n select\n ARRAY[key]::text[] as path,\n ARRAY[key]::text[] as fullpath,\n value\n FROM jsonb_each(content->'properties')\n union all\n select\n CASE WHEN obj_key IS NOT NULL THEN path || obj_key ELSE path END,\n path || coalesce(obj_key, (arr_key- 1)::text),\n coalesce(obj_value, arr_value)\n from extract_all\n left join lateral\n jsonb_each(case jsonb_typeof(value) when 'object' then value end)\n as o(obj_key, obj_value)\n on jsonb_typeof(value) = 'object'\n left join lateral\n jsonb_array_elements(case jsonb_typeof(value) when 'array' then value end)\n with ordinality as a(arr_value, arr_key)\n on jsonb_typeof(value) = 'array'\n where obj_key is not null or arr_key is not null\n )\n , paths AS (\n select\n array_to_string(path, '.') as path,\n value\n FROM extract_all\n WHERE\n jsonb_typeof(value) NOT IN ('array','object')\n ), grouped AS (\n SELECT path, jsonb_agg(distinct value) vals FROM paths group by path\n ) SELECT coalesce(jsonb_object_agg(path, CASE WHEN jsonb_array_length(vals)=1 THEN vals->0 ELSE vals END) - '{datetime}'::text[], '{}'::jsonb) FROM grouped\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET JIT TO OFF;\n\nCREATE INDEX \"datetime_idx\" ON items (datetime);\nCREATE INDEX \"end_datetime_idx\" ON items (end_datetime);\nCREATE INDEX \"properties_idx\" ON items USING GIN (properties jsonb_path_ops);\nCREATE INDEX \"collection_idx\" ON items (collection_id);\nCREATE INDEX \"geometry_idx\" ON items USING GIST (geometry);\nCREATE UNIQUE INDEX \"items_id_datetime_idx\" ON items (datetime, id);\n\nALTER TABLE items ADD CONSTRAINT items_collections_fk FOREIGN KEY (collection_id) REFERENCES collections(id) ON DELETE CASCADE DEFERRABLE;\n\nCREATE OR REPLACE FUNCTION analyze_empty_partitions() RETURNS VOID AS $$\nDECLARE\n p text;\nBEGIN\n FOR p IN SELECT partition FROM all_items_partitions WHERE est_cnt = 0 LOOP\n EXECUTE format('ANALYZE %I;', p);\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION items_partition_name(timestamptz) RETURNS text AS $$\n SELECT to_char($1, '\"items_p\"IYYY\"w\"IW');\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION items_partition_exists(text) RETURNS boolean AS $$\n SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_class WHERE relname=$1);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION items_partition_exists(timestamptz) RETURNS boolean AS $$\n SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_class WHERE relname=items_partition_name($1));\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION items_partition_create_worker(partition text, partition_start timestamptz, partition_end timestamptz) RETURNS VOID AS $$\nDECLARE\n err_context text;\nBEGIN\n EXECUTE format(\n $f$\n CREATE TABLE IF NOT EXISTS %1$I PARTITION OF items\n FOR VALUES FROM (%2$L) TO (%3$L);\n CREATE UNIQUE INDEX IF NOT EXISTS %4$I ON %1$I (id);\n $f$,\n partition,\n partition_start,\n partition_end,\n concat(partition, '_id_pk')\n );\nEXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', partition;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH to pgstac, public;\n\nCREATE OR REPLACE FUNCTION items_partition_create(ts timestamptz) RETURNS text AS $$\nDECLARE\n partition text := items_partition_name(ts);\n partition_start timestamptz;\n partition_end timestamptz;\nBEGIN\n IF items_partition_exists(partition) THEN\n RETURN partition;\n END IF;\n partition_start := date_trunc('week', ts);\n partition_end := partition_start + '1 week'::interval;\n PERFORM items_partition_create_worker(partition, partition_start, partition_end);\n RAISE NOTICE 'partition: %', partition;\n RETURN partition;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION items_partition_create(st timestamptz, et timestamptz) RETURNS SETOF text AS $$\nWITH t AS (\n SELECT\n generate_series(\n date_trunc('week',st),\n date_trunc('week', et),\n '1 week'::interval\n ) w\n)\nSELECT items_partition_create(w) FROM t;\n$$ LANGUAGE SQL;\n\n\nCREATE UNLOGGED TABLE items_staging (\n content JSONB NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION items_staging_insert_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p record;\n _partitions text[];\nBEGIN\n CREATE TEMP TABLE new_partitions ON COMMIT DROP AS\n SELECT\n items_partition_name(stac_datetime(content)) as partition,\n date_trunc('week', min(stac_datetime(content))) as partition_start\n FROM newdata\n GROUP BY 1;\n\n -- set statslastupdated in cache to be old enough cache always regenerated\n\n SELECT array_agg(partition) INTO _partitions FROM new_partitions;\n UPDATE search_wheres\n SET\n statslastupdated = NULL\n WHERE _where IN (\n SELECT _where FROM search_wheres sw WHERE sw.partitions && _partitions\n FOR UPDATE SKIP LOCKED\n )\n ;\n FOR p IN SELECT new_partitions.partition, new_partitions.partition_start, new_partitions.partition_start + '1 week'::interval as partition_end FROM new_partitions\n LOOP\n RAISE NOTICE 'Getting partition % ready.', p.partition;\n IF NOT items_partition_exists(p.partition) THEN\n RAISE NOTICE 'Creating partition %.', p.partition;\n PERFORM items_partition_create_worker(p.partition, p.partition_start, p.partition_end);\n END IF;\n PERFORM drop_partition_constraints(p.partition);\n END LOOP;\n INSERT INTO items (id, geometry, collection_id, datetime, end_datetime, properties, content)\n SELECT\n content->>'id',\n stac_geom(content),\n content->>'collection',\n stac_datetime(content),\n stac_end_datetime(content),\n properties_idx(content),\n content\n FROM newdata\n ;\n DELETE FROM items_staging;\n\n\n FOR p IN SELECT new_partitions.partition FROM new_partitions\n LOOP\n RAISE NOTICE 'Setting constraints for partition %.', p.partition;\n PERFORM partition_checks(p.partition);\n END LOOP;\n DROP TABLE IF EXISTS new_partitions;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER items_staging_insert_trigger AFTER INSERT ON items_staging REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_insert_triggerfunc();\n\n\nCREATE UNLOGGED TABLE items_staging_ignore (\n content JSONB NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION items_staging_ignore_insert_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p record;\n _partitions text[];\nBEGIN\n CREATE TEMP TABLE new_partitions ON COMMIT DROP AS\n SELECT\n items_partition_name(stac_datetime(content)) as partition,\n date_trunc('week', min(stac_datetime(content))) as partition_start\n FROM newdata\n GROUP BY 1;\n\n -- set statslastupdated in cache to be old enough cache always regenerated\n\n SELECT array_agg(partition) INTO _partitions FROM new_partitions;\n UPDATE search_wheres\n SET\n statslastupdated = NULL\n WHERE _where IN (\n SELECT _where FROM search_wheres sw WHERE sw.partitions && _partitions\n FOR UPDATE SKIP LOCKED\n )\n ;\n\n FOR p IN SELECT new_partitions.partition, new_partitions.partition_start, new_partitions.partition_start + '1 week'::interval as partition_end FROM new_partitions\n LOOP\n RAISE NOTICE 'Getting partition % ready.', p.partition;\n IF NOT items_partition_exists(p.partition) THEN\n RAISE NOTICE 'Creating partition %.', p.partition;\n PERFORM items_partition_create_worker(p.partition, p.partition_start, p.partition_end);\n END IF;\n PERFORM drop_partition_constraints(p.partition);\n END LOOP;\n\n INSERT INTO items (id, geometry, collection_id, datetime, end_datetime, properties, content)\n SELECT\n content->>'id',\n stac_geom(content),\n content->>'collection',\n stac_datetime(content),\n stac_end_datetime(content),\n properties_idx(content),\n content\n FROM newdata\n ON CONFLICT (datetime, id) DO NOTHING\n ;\n DELETE FROM items_staging;\n\n\n FOR p IN SELECT new_partitions.partition FROM new_partitions\n LOOP\n RAISE NOTICE 'Setting constraints for partition %.', p.partition;\n PERFORM partition_checks(p.partition);\n END LOOP;\n DROP TABLE IF EXISTS new_partitions;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER items_staging_ignore_insert_trigger AFTER INSERT ON items_staging_ignore REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_ignore_insert_triggerfunc();\n\nCREATE UNLOGGED TABLE items_staging_upsert (\n content JSONB NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION items_staging_upsert_insert_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p record;\n _partitions text[];\nBEGIN\n CREATE TEMP TABLE new_partitions ON COMMIT DROP AS\n SELECT\n items_partition_name(stac_datetime(content)) as partition,\n date_trunc('week', min(stac_datetime(content))) as partition_start\n FROM newdata\n GROUP BY 1;\n\n -- set statslastupdated in cache to be old enough cache always regenerated\n\n SELECT array_agg(partition) INTO _partitions FROM new_partitions;\n UPDATE search_wheres\n SET\n statslastupdated = NULL\n WHERE _where IN (\n SELECT _where FROM search_wheres sw WHERE sw.partitions && _partitions\n FOR UPDATE SKIP LOCKED\n )\n ;\n\n FOR p IN SELECT new_partitions.partition, new_partitions.partition_start, new_partitions.partition_start + '1 week'::interval as partition_end FROM new_partitions\n LOOP\n RAISE NOTICE 'Getting partition % ready.', p.partition;\n IF NOT items_partition_exists(p.partition) THEN\n RAISE NOTICE 'Creating partition %.', p.partition;\n PERFORM items_partition_create_worker(p.partition, p.partition_start, p.partition_end);\n END IF;\n PERFORM drop_partition_constraints(p.partition);\n END LOOP;\n\n INSERT INTO items (id, geometry, collection_id, datetime, end_datetime, properties, content)\n SELECT\n content->>'id',\n stac_geom(content),\n content->>'collection',\n stac_datetime(content),\n stac_end_datetime(content),\n properties_idx(content),\n content\n FROM newdata\n ON CONFLICT (datetime, id) DO UPDATE SET\n content = EXCLUDED.content\n WHERE items.content IS DISTINCT FROM EXCLUDED.content\n ;\n DELETE FROM items_staging;\n\n\n FOR p IN SELECT new_partitions.partition FROM new_partitions\n LOOP\n RAISE NOTICE 'Setting constraints for partition %.', p.partition;\n PERFORM partition_checks(p.partition);\n END LOOP;\n DROP TABLE IF EXISTS new_partitions;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER items_staging_upsert_insert_trigger AFTER INSERT ON items_staging_upsert REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_upsert_insert_triggerfunc();\n\nCREATE OR REPLACE FUNCTION items_update_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\nBEGIN\n NEW.id := NEW.content->>'id';\n NEW.datetime := stac_datetime(NEW.content);\n NEW.end_datetime := stac_end_datetime(NEW.content);\n NEW.collection_id := NEW.content->>'collection';\n NEW.geometry := stac_geom(NEW.content);\n NEW.properties := properties_idx(NEW.content);\n IF TG_OP = 'UPDATE' AND NEW IS NOT DISTINCT FROM OLD THEN\n RETURN NULL;\n END IF;\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER items_update_trigger BEFORE UPDATE ON items\n FOR EACH ROW EXECUTE PROCEDURE items_update_triggerfunc();\n\n/*\nView to get a table of available items partitions\nwith date ranges\n*/\nDROP VIEW IF EXISTS all_items_partitions CASCADE;\nCREATE VIEW all_items_partitions AS\nWITH base AS\n(SELECT\n c.oid::pg_catalog.regclass::text as partition,\n pg_catalog.pg_get_expr(c.relpartbound, c.oid) as _constraint,\n regexp_matches(\n pg_catalog.pg_get_expr(c.relpartbound, c.oid),\n E'\\\\(''\\([0-9 :+-]*\\)''\\\\).*\\\\(''\\([0-9 :+-]*\\)''\\\\)'\n ) as t,\n reltuples::bigint as est_cnt\nFROM pg_catalog.pg_class c, pg_catalog.pg_inherits i\nWHERE c.oid = i.inhrelid AND i.inhparent = 'items'::regclass)\nSELECT partition, tstzrange(\n t[1]::timestamptz,\n t[2]::timestamptz\n), t[1]::timestamptz as pstart,\n t[2]::timestamptz as pend, est_cnt\nFROM base\nORDER BY 2 desc;\n\nCREATE OR REPLACE VIEW items_partitions AS\nSELECT * FROM all_items_partitions WHERE est_cnt>0;\n\nCREATE OR REPLACE FUNCTION get_item(_id text) RETURNS jsonb AS $$\n SELECT content FROM items WHERE id=_id;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION delete_item(_id text) RETURNS VOID AS $$\nDECLARE\nout items%ROWTYPE;\nBEGIN\n DELETE FROM items WHERE id = _id RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION create_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION update_item(data jsonb) RETURNS VOID AS $$\nDECLARE\n out items%ROWTYPE;\nBEGIN\n UPDATE items SET content=data WHERE id = data->>'id' RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION create_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION collection_bbox(id text) RETURNS jsonb AS $$\nSELECT (replace(replace(replace(st_extent(geometry)::text,'BOX(','[['),')',']]'),' ',','))::jsonb\nFROM items WHERE collection_id=$1;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION collection_temporal_extent(id text) RETURNS jsonb AS $$\nSELECT to_jsonb(array[array[min(datetime)::text, max(datetime)::text]])\nFROM items WHERE collection_id=$1;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection_extents() RETURNS VOID AS $$\nUPDATE collections SET\n content = content ||\n jsonb_build_object(\n 'extent', jsonb_build_object(\n 'spatial', jsonb_build_object(\n 'bbox', collection_bbox(collections.id)\n ),\n 'temporal', jsonb_build_object(\n 'interval', collection_temporal_extent(collections.id)\n )\n )\n )\n;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION items_path(\n IN dotpath text,\n OUT field text,\n OUT path text,\n OUT path_txt text,\n OUT jsonpath text,\n OUT eq text\n) RETURNS RECORD AS $$\nDECLARE\npath_elements text[];\nlast_element text;\nBEGIN\ndotpath := replace(trim(dotpath), 'properties.', '');\n\nIF dotpath = '' THEN\n RETURN;\nEND IF;\n\npath_elements := string_to_array(dotpath, '.');\njsonpath := NULL;\n\nIF path_elements[1] IN ('id','geometry','datetime') THEN\n field := path_elements[1];\n path_elements := path_elements[2:];\nELSIF path_elements[1] = 'collection' THEN\n field := 'collection_id';\n path_elements := path_elements[2:];\nELSIF path_elements[1] IN ('links', 'assets', 'stac_version', 'stac_extensions') THEN\n field := 'content';\nELSE\n field := 'content';\n path_elements := '{properties}'::text[] || path_elements;\nEND IF;\nIF cardinality(path_elements)<1 THEN\n path := field;\n path_txt := field;\n jsonpath := '$';\n eq := NULL;\n RETURN;\nEND IF;\n\n\nlast_element := path_elements[cardinality(path_elements)];\npath_elements := path_elements[1:cardinality(path_elements)-1];\njsonpath := concat(array_to_string('{$}'::text[] || array_map_ident(path_elements), '.'), '.', quote_ident(last_element));\npath_elements := array_map_literal(path_elements);\npath := format($F$ properties->%s $F$, quote_literal(dotpath));\npath_txt := format($F$ properties->>%s $F$, quote_literal(dotpath));\neq := format($F$ properties @? '$.%s[*] ? (@ == %%s) '$F$, quote_ident(dotpath));\n\nRAISE NOTICE 'ITEMS PATH -- % % % % %', field, path, path_txt, jsonpath, eq;\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION parse_dtrange(IN _indate jsonb, OUT _tstzrange tstzrange) AS $$\nWITH t AS (\n SELECT CASE\n WHEN jsonb_typeof(_indate) = 'array' THEN\n textarr(_indate)\n ELSE\n regexp_split_to_array(\n btrim(_indate::text,'\"'),\n '/'\n )\n END AS arr\n)\n, t1 AS (\n SELECT\n CASE\n WHEN array_upper(arr,1) = 1 OR arr[1] = '..' OR arr[1] IS NULL THEN '-infinity'::timestamptz\n ELSE arr[1]::timestamptz\n END AS st,\n CASE\n WHEN array_upper(arr,1) = 1 THEN arr[1]::timestamptz\n WHEN arr[2] = '..' OR arr[2] IS NULL THEN 'infinity'::timestamptz\n ELSE arr[2]::timestamptz\n END AS et\n FROM t\n)\nSELECT\n tstzrange(st,et)\nFROM t1;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION bbox_geom(_bbox jsonb) RETURNS geometry AS $$\nSELECT CASE jsonb_array_length(_bbox)\n WHEN 4 THEN\n ST_SetSRID(ST_MakeEnvelope(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float,\n (_bbox->>3)::float\n ),4326)\n WHEN 6 THEN\n ST_SetSRID(ST_3DMakeBox(\n ST_MakePoint(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float\n ),\n ST_MakePoint(\n (_bbox->>3)::float,\n (_bbox->>4)::float,\n (_bbox->>5)::float\n )\n ),4326)\n ELSE null END;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION cql_and_append(existing jsonb, newfilters jsonb) RETURNS jsonb AS $$\nSELECT CASE WHEN existing ? 'filter' AND newfilters IS NOT NULL THEN\n jsonb_build_object(\n 'and',\n jsonb_build_array(\n existing->'filter',\n newfilters\n )\n )\nELSE\n newfilters\nEND;\n$$ LANGUAGE SQL;\n\n\n-- ADDs base filters (ids, collections, datetime, bbox, intersects) that are\n-- added outside of the filter/query in the stac request\nCREATE OR REPLACE FUNCTION add_filters_to_cql(j jsonb) RETURNS jsonb AS $$\nDECLARE\nnewprop jsonb;\nnewprops jsonb := '[]'::jsonb;\nBEGIN\nIF j ? 'ids' THEN\n newprop := jsonb_build_object(\n 'in',\n jsonb_build_array(\n '{\"property\":\"id\"}'::jsonb,\n j->'ids'\n )\n );\n newprops := jsonb_insert(newprops, '{1}', newprop);\nEND IF;\nIF j ? 'collections' THEN\n newprop := jsonb_build_object(\n 'in',\n jsonb_build_array(\n '{\"property\":\"collection\"}'::jsonb,\n j->'collections'\n )\n );\n newprops := jsonb_insert(newprops, '{1}', newprop);\nEND IF;\n\nIF j ? 'datetime' THEN\n newprop := format(\n '{\"anyinteracts\":[{\"property\":\"datetime\"}, %s]}',\n j->'datetime'\n );\n newprops := jsonb_insert(newprops, '{1}', newprop);\nEND IF;\n\nIF j ? 'bbox' THEN\n newprop := format(\n '{\"intersects\":[{\"property\":\"geometry\"}, %s]}',\n j->'bbox'\n );\n newprops := jsonb_insert(newprops, '{1}', newprop);\nEND IF;\n\nIF j ? 'intersects' THEN\n newprop := format(\n '{\"intersects\":[{\"property\":\"geometry\"}, %s]}',\n j->'intersects'\n );\n newprops := jsonb_insert(newprops, '{1}', newprop);\nEND IF;\n\nRAISE NOTICE 'newprops: %', newprops;\n\nIF newprops IS NOT NULL AND jsonb_array_length(newprops) > 0 THEN\n return jsonb_set(\n j,\n '{filter}',\n cql_and_append(j, jsonb_build_object('and', newprops))\n ) - '{ids,collections,datetime,bbox,intersects}'::text[];\nEND IF;\n\nreturn j;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION query_to_cqlfilter(j jsonb) RETURNS jsonb AS $$\n-- Translates anything passed in through the deprecated \"query\" into equivalent CQL\nWITH t AS (\n SELECT key as property, value as ops\n FROM jsonb_each(j->'query')\n), t2 AS (\n SELECT property, (jsonb_each(ops)).*\n FROM t WHERE jsonb_typeof(ops) = 'object'\n UNION ALL\n SELECT property, 'eq', ops\n FROM t WHERE jsonb_typeof(ops) != 'object'\n), t3 AS (\nSELECT\n jsonb_strip_nulls(jsonb_build_object(\n 'and',\n jsonb_agg(\n jsonb_build_object(\n key,\n jsonb_build_array(\n jsonb_build_object('property',property),\n value\n )\n )\n )\n )) as qcql FROM t2\n)\nSELECT\n CASE WHEN qcql IS NOT NULL THEN\n jsonb_set(j, '{filter}', cql_and_append(j, qcql)) - 'query'\n ELSE j\n END\nFROM t3\n;\n$$ LANGUAGE SQL;\n\n\n\nCREATE OR REPLACE FUNCTION temporal_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\nll text := 'datetime';\nlh text := 'end_datetime';\nrrange tstzrange;\nrl text;\nrh text;\noutq text;\nBEGIN\nrrange := parse_dtrange(args->1);\nRAISE NOTICE 'Constructing temporal query OP: %, ARGS: %, RRANGE: %', op, args, rrange;\nop := lower(op);\nrl := format('%L::timestamptz', lower(rrange));\nrh := format('%L::timestamptz', upper(rrange));\noutq := CASE op\n WHEN 't_before' THEN 'lh < rl'\n WHEN 't_after' THEN 'll > rh'\n WHEN 't_meets' THEN 'lh = rl'\n WHEN 't_metby' THEN 'll = rh'\n WHEN 't_overlaps' THEN 'll < rl AND rl < lh < rh'\n WHEN 't_overlappedby' THEN 'rl < ll < rh AND lh > rh'\n WHEN 't_starts' THEN 'll = rl AND lh < rh'\n WHEN 't_startedby' THEN 'll = rl AND lh > rh'\n WHEN 't_during' THEN 'll > rl AND lh < rh'\n WHEN 't_contains' THEN 'll < rl AND lh > rh'\n WHEN 't_finishes' THEN 'll > rl AND lh = rh'\n WHEN 't_finishedby' THEN 'll < rl AND lh = rh'\n WHEN 't_equals' THEN 'll = rl AND lh = rh'\n WHEN 't_disjoint' THEN 'NOT (ll <= rh AND lh >= rl)'\n WHEN 't_intersects' THEN 'll <= rh AND lh >= rl'\n WHEN 'anyinteracts' THEN 'll <= rh AND lh >= rl'\nEND;\noutq := regexp_replace(outq, '\\mll\\M', ll);\noutq := regexp_replace(outq, '\\mlh\\M', lh);\noutq := regexp_replace(outq, '\\mrl\\M', rl);\noutq := regexp_replace(outq, '\\mrh\\M', rh);\noutq := format('(%s)', outq);\nRETURN outq;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION spatial_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\ngeom text;\nj jsonb := args->1;\nBEGIN\nop := lower(op);\nRAISE NOTICE 'Constructing spatial query OP: %, ARGS: %', op, args;\nIF op NOT IN ('s_equals','s_disjoint','s_touches','s_within','s_overlaps','s_crosses','s_intersects','intersects','s_contains') THEN\n RAISE EXCEPTION 'Spatial Operator % Not Supported', op;\nEND IF;\nop := regexp_replace(op, '^s_', 'st_');\nIF op = 'intersects' THEN\n op := 'st_intersects';\nEND IF;\n-- Convert geometry to WKB string\nIF j ? 'type' AND j ? 'coordinates' THEN\n geom := st_geomfromgeojson(j)::text;\nELSIF jsonb_typeof(j) = 'array' THEN\n geom := bbox_geom(j)::text;\nEND IF;\n\nRETURN format('%s(geometry, %L::geometry)', op, geom);\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n/* cql_query_op -- Parses a CQL query operation, recursing when necessary\n IN jsonb -- a subelement from a valid stac query\n IN text -- the operator being used on elements passed in\n RETURNS a SQL fragment to be used in a WHERE clause\n*/\nCREATE OR REPLACE FUNCTION cql_query_op(j jsonb, _op text DEFAULT NULL) RETURNS text AS $$\nDECLARE\njtype text := jsonb_typeof(j);\nop text := lower(_op);\nops jsonb :=\n '{\n \"eq\": \"%s = %s\",\n \"lt\": \"%s < %s\",\n \"lte\": \"%s <= %s\",\n \"gt\": \"%s > %s\",\n \"gte\": \"%s >= %s\",\n \"le\": \"%s <= %s\",\n \"ge\": \"%s >= %s\",\n \"=\": \"%s = %s\",\n \"<\": \"%s < %s\",\n \"<=\": \"%s <= %s\",\n \">\": \"%s > %s\",\n \">=\": \"%s >= %s\",\n \"like\": \"%s LIKE %s\",\n \"ilike\": \"%s ILIKE %s\",\n \"+\": \"%s + %s\",\n \"-\": \"%s - %s\",\n \"*\": \"%s * %s\",\n \"/\": \"%s / %s\",\n \"in\": \"%s = ANY (%s)\",\n \"not\": \"NOT (%s)\",\n \"between\": \"%s BETWEEN %s AND %s\",\n \"lower\":\" lower(%s)\",\n \"upper\":\" upper(%s)\",\n \"isnull\": \"%s IS NULL\"\n }'::jsonb;\nret text;\nargs text[] := NULL;\n\nBEGIN\nRAISE NOTICE 'j: %, op: %, jtype: %', j, op, jtype;\n\n-- for in, convert value, list to array syntax to match other ops\nIF op = 'in' and j ? 'value' and j ? 'list' THEN\n j := jsonb_build_array( j->'value', j->'list');\n jtype := 'array';\n RAISE NOTICE 'IN: j: %, jtype: %', j, jtype;\nEND IF;\n\nIF op = 'between' and j ? 'value' and j ? 'lower' and j ? 'upper' THEN\n j := jsonb_build_array( j->'value', j->'lower', j->'upper');\n jtype := 'array';\n RAISE NOTICE 'BETWEEN: j: %, jtype: %', j, jtype;\nEND IF;\n\nIF op = 'not' AND jtype = 'object' THEN\n j := jsonb_build_array( j );\n jtype := 'array';\n RAISE NOTICE 'NOT: j: %, jtype: %', j, jtype;\nEND IF;\n\n-- Set Lower Case on Both Arguments When Case Insensitive Flag Set\nIF op in ('eq','lt','lte','gt','gte','like') AND jsonb_typeof(j->2) = 'boolean' THEN\n IF (j->>2)::boolean THEN\n RETURN format(concat('(',ops->>op,')'), cql_query_op(jsonb_build_array(j->0), 'lower'), cql_query_op(jsonb_build_array(j->1), 'lower'));\n END IF;\nEND IF;\n\n-- Special Case when comparing a property in a jsonb field to a string or number using eq\n-- Allows to leverage GIN index on jsonb fields\nIF op = 'eq' THEN\n IF j->0 ? 'property'\n AND jsonb_typeof(j->1) IN ('number','string')\n AND (items_path(j->0->>'property')).eq IS NOT NULL\n THEN\n RETURN format((items_path(j->0->>'property')).eq, j->1);\n END IF;\nEND IF;\n\n\n\nIF op ilike 't_%' or op = 'anyinteracts' THEN\n RETURN temporal_op_query(op, j);\nEND IF;\n\nIF op ilike 's_%' or op = 'intersects' THEN\n RETURN spatial_op_query(op, j);\nEND IF;\n\n\nIF jtype = 'object' THEN\n RAISE NOTICE 'parsing object';\n IF j ? 'property' THEN\n -- Convert the property to be used as an identifier\n return (items_path(j->>'property')).path_txt;\n ELSIF _op IS NULL THEN\n -- Iterate to convert elements in an object where the operator has not been set\n -- Combining with AND\n SELECT\n array_to_string(array_agg(cql_query_op(e.value, e.key)), ' AND ')\n INTO ret\n FROM jsonb_each(j) e;\n RETURN ret;\n END IF;\nEND IF;\n\nIF jtype = 'string' THEN\n RETURN quote_literal(j->>0);\nEND IF;\n\nIF jtype ='number' THEN\n RETURN (j->>0)::numeric;\nEND IF;\n\nIF jtype = 'array' AND op IS NULL THEN\n RAISE NOTICE 'Parsing array into array arg. j: %', j;\n SELECT format($f$ '{%s}'::text[] $f$, string_agg(e,',')) INTO ret FROM jsonb_array_elements_text(j) e;\n RETURN ret;\nEND IF;\n\n\n-- If the type of the passed json is an array\n-- Calculate the arguments that will be passed to functions/operators\nIF jtype = 'array' THEN\n RAISE NOTICE 'Parsing array into args. j: %', j;\n -- If any argument is numeric, cast any text arguments to numeric\n IF j @? '$[*] ? (@.type() == \"number\")' THEN\n SELECT INTO args\n array_agg(concat('(',cql_query_op(e),')::numeric'))\n FROM jsonb_array_elements(j) e;\n ELSE\n SELECT INTO args\n array_agg(cql_query_op(e))\n FROM jsonb_array_elements(j) e;\n END IF;\n --RETURN args;\nEND IF;\nRAISE NOTICE 'ARGS after array cleaning: %', args;\n\nIF op IS NULL THEN\n RETURN args::text[];\nEND IF;\n\nIF args IS NULL OR cardinality(args) < 1 THEN\n RAISE NOTICE 'No Args';\n RETURN '';\nEND IF;\n\nIF op IN ('and','or') THEN\n SELECT\n CONCAT(\n '(',\n array_to_string(args, UPPER(CONCAT(' ',op,' '))),\n ')'\n ) INTO ret\n FROM jsonb_array_elements(j) e;\n RETURN ret;\nEND IF;\n\n-- If the op is in the ops json then run using the template in the json\nIF ops ? op THEN\n RAISE NOTICE 'ARGS: % MAPPED: %',args, array_map_literal(args);\n\n RETURN format(concat('(',ops->>op,')'), VARIADIC args);\nEND IF;\n\nRETURN j->>0;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n/* cql_query_op -- Parses a CQL query operation, recursing when necessary\n IN jsonb -- a subelement from a valid stac query\n IN text -- the operator being used on elements passed in\n RETURNS a SQL fragment to be used in a WHERE clause\n*/\nDROP FUNCTION IF EXISTS cql2_query;\nCREATE OR REPLACE FUNCTION cql2_query(j jsonb, recursion int DEFAULT 0) RETURNS text AS $$\nDECLARE\nargs jsonb := j->'args';\njtype text := jsonb_typeof(j->'args');\nop text := lower(j->>'op');\narg jsonb;\nargtext text;\nargstext text[] := '{}'::text[];\ninobj jsonb;\n_numeric text := '';\nops jsonb :=\n '{\n \"eq\": \"%s = %s\",\n \"lt\": \"%s < %s\",\n \"lte\": \"%s <= %s\",\n \"gt\": \"%s > %s\",\n \"gte\": \"%s >= %s\",\n \"le\": \"%s <= %s\",\n \"ge\": \"%s >= %s\",\n \"=\": \"%s = %s\",\n \"<\": \"%s < %s\",\n \"<=\": \"%s <= %s\",\n \">\": \"%s > %s\",\n \">=\": \"%s >= %s\",\n \"like\": \"%s LIKE %s\",\n \"ilike\": \"%s ILIKE %s\",\n \"+\": \"%s + %s\",\n \"-\": \"%s - %s\",\n \"*\": \"%s * %s\",\n \"/\": \"%s / %s\",\n \"in\": \"%s = ANY (%s)\",\n \"not\": \"NOT (%s)\",\n \"between\": \"%s BETWEEN (%2$s)[1] AND (%2$s)[2]\",\n \"lower\":\" lower(%s)\",\n \"upper\":\" upper(%s)\",\n \"isnull\": \"%s IS NULL\"\n }'::jsonb;\nret text;\n\nBEGIN\nRAISE NOTICE 'j: %s', j;\nIF j ? 'filter' THEN\n RETURN cql2_query(j->'filter');\nEND IF;\n\nIF j ? 'upper' THEN\nRAISE NOTICE 'upper %s',jsonb_build_object(\n 'op', 'upper',\n 'args', jsonb_build_array( j-> 'upper')\n ) ;\n RETURN cql2_query(\n jsonb_build_object(\n 'op', 'upper',\n 'args', jsonb_build_array( j-> 'upper')\n )\n );\nEND IF;\n\nIF j ? 'lower' THEN\n RETURN cql2_query(\n jsonb_build_object(\n 'op', 'lower',\n 'args', jsonb_build_array( j-> 'lower')\n )\n );\nEND IF;\n\nIF j ? 'args' AND jsonb_typeof(args) != 'array' THEN\n args := jsonb_build_array(args);\nEND IF;\n-- END Cases where no further nesting is expected\nIF j ? 'op' THEN\n -- Special case to use JSONB index for equality\n IF op = 'eq'\n AND args->0 ? 'property'\n AND jsonb_typeof(args->1) IN ('number', 'string')\n AND (items_path(args->0->>'property')).eq IS NOT NULL\n THEN\n RETURN format((items_path(args->0->>'property')).eq, args->1);\n END IF;\n\n -- Temporal Query\n IF op ilike 't_%' or op = 'anyinteracts' THEN\n RETURN temporal_op_query(op, args);\n END IF;\n\n -- Spatial Query\n IF op ilike 's_%' or op = 'intersects' THEN\n RETURN spatial_op_query(op, args);\n END IF;\n\n -- In Query - separate into separate eq statements so that we can use eq jsonb optimization\n IF op = 'in' THEN\n RAISE NOTICE '% IN args: %', repeat(' ', recursion), args;\n SELECT INTO inobj\n jsonb_agg(\n jsonb_build_object(\n 'op', 'eq',\n 'args', jsonb_build_array( args->0 , v)\n )\n )\n FROM jsonb_array_elements( args->1) v;\n RETURN cql2_query(jsonb_build_object('op','or','args',inobj));\n END IF;\nEND IF;\n\nIF j ? 'property' THEN\n RETURN (items_path(j->>'property')).path_txt;\nEND IF;\n\nRAISE NOTICE '%jtype: %',repeat(' ', recursion), jtype;\nIF jsonb_typeof(j) = 'number' THEN\n RETURN format('%L::numeric', j->>0);\nEND IF;\n\nIF jsonb_typeof(j) = 'string' THEN\n RETURN quote_literal(j->>0);\nEND IF;\n\nIF jsonb_typeof(j) = 'array' THEN\n IF j @? '$[*] ? (@.type() == \"number\")' THEN\n RETURN CONCAT(quote_literal(textarr(j)::text), '::numeric[]');\n ELSE\n RETURN CONCAT(quote_literal(textarr(j)::text), '::text[]');\n END IF;\nEND IF;\nRAISE NOTICE 'ARGS after array cleaning: %', args;\n\nRAISE NOTICE '%beforeargs op: %, args: %',repeat(' ', recursion), op, args;\nIF j ? 'args' THEN\n FOR arg in SELECT * FROM jsonb_array_elements(args) LOOP\n argtext := cql2_query(arg, recursion + 1);\n RAISE NOTICE '% -- arg: %, argtext: %', repeat(' ', recursion), arg, argtext;\n argstext := argstext || argtext;\n END LOOP;\nEND IF;\nRAISE NOTICE '%afterargs op: %, argstext: %',repeat(' ', recursion), op, argstext;\n\n\nIF op IN ('and', 'or') THEN\n RAISE NOTICE 'inand op: %, argstext: %', op, argstext;\n SELECT\n concat(' ( ',array_to_string(array_agg(e), concat(' ',op,' ')),' ) ')\n INTO ret\n FROM unnest(argstext) e;\n RETURN ret;\nEND IF;\n\nIF ops ? op THEN\n IF argstext[2] ~* 'numeric' THEN\n argstext := ARRAY[concat('(',argstext[1],')::numeric')] || argstext[2:3];\n END IF;\n RETURN format(concat('(',ops->>op,')'), VARIADIC argstext);\nEND IF;\n\nRAISE NOTICE '%op: %, argstext: %',repeat(' ', recursion), op, argstext;\n\nRETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n\nCREATE OR REPLACE FUNCTION cql_to_where(_search jsonb = '{}'::jsonb) RETURNS text AS $$\nDECLARE\nfilterlang text;\nsearch jsonb := _search;\n_where text;\nBEGIN\n\nRAISE NOTICE 'SEARCH CQL Final: %', search;\nfilterlang := COALESCE(\n search->>'filter-lang',\n get_setting('default-filter-lang', _search->'conf')\n);\n\nIF filterlang = 'cql-json' THEN\n search := query_to_cqlfilter(search);\n search := add_filters_to_cql(search);\n _where := cql_query_op(search->'filter');\nELSE\n _where := cql2_query(search->'filter');\nEND IF;\n\nIF trim(_where) = '' THEN\n _where := NULL;\nEND IF;\n_where := coalesce(_where, ' TRUE ');\nRETURN _where;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION parse_sort_dir(_dir text, reverse boolean default false) RETURNS text AS $$\nWITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n) SELECT\n CASE\n WHEN NOT reverse THEN d\n WHEN d = 'ASC' THEN 'DESC'\n WHEN d = 'DESC' THEN 'ASC'\n END\nFROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION sort_dir_to_op(_dir text, prev boolean default false) RETURNS text AS $$\nWITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n) SELECT\n CASE\n WHEN d = 'ASC' AND prev THEN '<='\n WHEN d = 'DESC' AND prev THEN '>='\n WHEN d = 'ASC' THEN '>='\n WHEN d = 'DESC' THEN '<='\n END\nFROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION field_orderby(p text) RETURNS text AS $$\nWITH t AS (\n SELECT\n replace(trim(substring(indexdef from 'btree \\((.*)\\)')),' ','')as s\n FROM pg_indexes WHERE schemaname='pgstac' AND tablename='items' AND indexdef ~* 'btree' AND indexdef ~* 'properties'\n) SELECT s FROM t WHERE strpos(s, lower(trim(p)))>0;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION sort_sqlorderby(\n _search jsonb DEFAULT NULL,\n reverse boolean DEFAULT FALSE\n) RETURNS text AS $$\nWITH sortby AS (\n SELECT coalesce(_search->'sortby','[{\"field\":\"datetime\", \"direction\":\"desc\"}]') as sort\n), withid AS (\n SELECT CASE\n WHEN sort @? '$[*] ? (@.field == \"id\")' THEN sort\n ELSE sort || '[{\"field\":\"id\", \"direction\":\"desc\"}]'::jsonb\n END as sort\n FROM sortby\n), withid_rows AS (\n SELECT jsonb_array_elements(sort) as value FROM withid\n),sorts AS (\n SELECT\n coalesce(field_orderby((items_path(value->>'field')).path_txt), (items_path(value->>'field')).path) as key,\n parse_sort_dir(value->>'direction', reverse) as dir\n FROM withid_rows\n)\nSELECT array_to_string(\n array_agg(concat(key, ' ', dir)),\n ', '\n) FROM sorts;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_sort_dir(sort_item jsonb) RETURNS text AS $$\nSELECT CASE WHEN sort_item->>'direction' ILIKE 'desc%' THEN 'DESC' ELSE 'ASC' END;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION get_token_filter(_search jsonb = '{}'::jsonb, token_rec jsonb DEFAULT NULL) RETURNS text AS $$\nDECLARE\n token_id text;\n filters text[] := '{}'::text[];\n prev boolean := TRUE;\n field text;\n dir text;\n sort record;\n orfilters text[] := '{}'::text[];\n andfilters text[] := '{}'::text[];\n output text;\n token_where text;\nBEGIN\n -- If no token provided return NULL\n IF token_rec IS NULL THEN\n IF NOT (_search ? 'token' AND\n (\n (_search->>'token' ILIKE 'prev:%')\n OR\n (_search->>'token' ILIKE 'next:%')\n )\n ) THEN\n RETURN NULL;\n END IF;\n prev := (_search->>'token' ILIKE 'prev:%');\n token_id := substr(_search->>'token', 6);\n SELECT to_jsonb(items) INTO token_rec FROM items WHERE id=token_id;\n END IF;\n RAISE NOTICE 'TOKEN ID: %', token_rec->'id';\n\n CREATE TEMP TABLE sorts (\n _row int GENERATED ALWAYS AS IDENTITY NOT NULL,\n _field text PRIMARY KEY,\n _dir text NOT NULL,\n _val text\n ) ON COMMIT DROP;\n\n -- Make sure we only have distinct columns to sort with taking the first one we get\n INSERT INTO sorts (_field, _dir)\n SELECT\n (items_path(value->>'field')).path,\n get_sort_dir(value)\n FROM\n jsonb_array_elements(coalesce(_search->'sortby','[{\"field\":\"datetime\",\"direction\":\"desc\"}]'))\n ON CONFLICT DO NOTHING\n ;\n RAISE NOTICE 'sorts 1: %', (SELECT jsonb_agg(to_json(sorts)) FROM sorts);\n -- Get the first sort direction provided. As the id is a primary key, if there are any\n -- sorts after id they won't do anything, so make sure that id is the last sort item.\n SELECT _dir INTO dir FROM sorts ORDER BY _row ASC LIMIT 1;\n IF EXISTS (SELECT 1 FROM sorts WHERE _field = 'id') THEN\n DELETE FROM sorts WHERE _row > (SELECT _row FROM sorts WHERE _field = 'id' ORDER BY _row ASC);\n ELSE\n INSERT INTO sorts (_field, _dir) VALUES ('id', dir);\n END IF;\n\n -- Add value from looked up item to the sorts table\n UPDATE sorts SET _val=quote_literal(token_rec->>_field);\n\n -- Check if all sorts are the same direction and use row comparison\n -- to filter\n RAISE NOTICE 'sorts 2: %', (SELECT jsonb_agg(to_json(sorts)) FROM sorts);\n\n IF (SELECT count(DISTINCT _dir) FROM sorts) = 1 THEN\n SELECT format(\n '(%s) %s (%s)',\n concat_ws(', ', VARIADIC array_agg(quote_ident(_field))),\n CASE WHEN (prev AND dir = 'ASC') OR (NOT prev AND dir = 'DESC') THEN '<' ELSE '>' END,\n concat_ws(', ', VARIADIC array_agg(_val))\n ) INTO output FROM sorts\n WHERE token_rec ? _field\n ;\n ELSE\n FOR sort IN SELECT * FROM sorts ORDER BY _row asc LOOP\n RAISE NOTICE 'SORT: %', sort;\n IF sort._row = 1 THEN\n orfilters := orfilters || format('(%s %s %s)',\n quote_ident(sort._field),\n CASE WHEN (prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC') THEN '<' ELSE '>' END,\n sort._val\n );\n ELSE\n orfilters := orfilters || format('(%s AND %s %s %s)',\n array_to_string(andfilters, ' AND '),\n quote_ident(sort._field),\n CASE WHEN (prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC') THEN '<' ELSE '>' END,\n sort._val\n );\n\n END IF;\n andfilters := andfilters || format('%s = %s',\n quote_ident(sort._field),\n sort._val\n );\n END LOOP;\n output := array_to_string(orfilters, ' OR ');\n END IF;\n DROP TABLE IF EXISTS sorts;\n token_where := concat('(',coalesce(output,'true'),')');\n IF trim(token_where) = '' THEN\n token_where := NULL;\n END IF;\n RAISE NOTICE 'TOKEN_WHERE: |%|',token_where;\n RETURN token_where;\n END;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION search_tohash(jsonb) RETURNS jsonb AS $$\n SELECT $1 - '{token,limit,context,includes,excludes}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION search_hash(jsonb, jsonb) RETURNS text AS $$\n SELECT md5(concat(search_tohash($1)::text,$2::text));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TABLE IF NOT EXISTS searches(\n hash text GENERATED ALWAYS AS (search_hash(search, metadata)) STORED PRIMARY KEY,\n search jsonb NOT NULL,\n _where text,\n orderby text,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n metadata jsonb DEFAULT '{}'::jsonb NOT NULL\n);\nCREATE TABLE IF NOT EXISTS search_wheres(\n id bigint generated always as identity primary key,\n _where text NOT NULL,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n statslastupdated timestamptz,\n estimated_count bigint,\n estimated_cost float,\n time_to_estimate float,\n total_count bigint,\n time_to_count float,\n partitions text[]\n);\n\nCREATE INDEX IF NOT EXISTS search_wheres_partitions ON search_wheres USING GIN (partitions);\nCREATE UNIQUE INDEX IF NOT EXISTS search_wheres_where ON search_wheres ((md5(_where)));\n\nCREATE OR REPLACE FUNCTION where_stats(inwhere text, updatestats boolean default false, conf jsonb default null) RETURNS search_wheres AS $$\nDECLARE\n t timestamptz;\n i interval;\n explain_json jsonb;\n partitions text[];\n sw search_wheres%ROWTYPE;\nBEGIN\n SELECT * INTO sw FROM search_wheres WHERE _where=inwhere FOR UPDATE;\n\n -- Update statistics if explicitly set, if statistics do not exist, or statistics ttl has expired\n IF NOT updatestats THEN\n RAISE NOTICE 'Checking if update is needed.';\n RAISE NOTICE 'Stats Last Updated: %', sw.statslastupdated;\n RAISE NOTICE 'TTL: %, Age: %', context_stats_ttl(conf), now() - sw.statslastupdated;\n RAISE NOTICE 'Context: %, Existing Total: %', context(conf), sw.total_count;\n IF\n sw.statslastupdated IS NULL\n OR (now() - sw.statslastupdated) > context_stats_ttl(conf)\n OR (context(conf) != 'off' AND sw.total_count IS NULL)\n THEN\n updatestats := TRUE;\n END IF;\n END IF;\n\n sw._where := inwhere;\n sw.lastused := now();\n sw.usecount := coalesce(sw.usecount,0) + 1;\n\n IF NOT updatestats THEN\n UPDATE search_wheres SET\n lastused = sw.lastused,\n usecount = sw.usecount\n WHERE _where = inwhere\n RETURNING * INTO sw\n ;\n RETURN sw;\n END IF;\n -- Use explain to get estimated count/cost and a list of the partitions that would be hit by the query\n t := clock_timestamp();\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s', inwhere)\n INTO explain_json;\n RAISE NOTICE 'Time for just the explain: %', clock_timestamp() - t;\n WITH t AS (\n SELECT j->>0 as p FROM\n jsonb_path_query(\n explain_json,\n 'strict $.**.\"Relation Name\" ? (@ != null)'\n ) j\n ), ordered AS (\n SELECT p FROM t ORDER BY p DESC\n -- SELECT p FROM t JOIN items_partitions\n -- ON (t.p = items_partitions.partition)\n -- ORDER BY pstart DESC\n )\n SELECT array_agg(p) INTO partitions FROM ordered;\n i := clock_timestamp() - t;\n RAISE NOTICE 'Time for explain + join: %', clock_timestamp() - t;\n\n\n\n sw.statslastupdated := now();\n sw.estimated_count := explain_json->0->'Plan'->'Plan Rows';\n sw.estimated_cost := explain_json->0->'Plan'->'Total Cost';\n sw.time_to_estimate := extract(epoch from i);\n sw.partitions := partitions;\n\n -- Do a full count of rows if context is set to on or if auto is set and estimates are low enough\n IF\n context(conf) = 'on'\n OR\n ( context(conf) = 'auto' AND\n (\n sw.estimated_count < context_estimated_count(conf)\n OR\n sw.estimated_cost < context_estimated_cost(conf)\n )\n )\n THEN\n t := clock_timestamp();\n RAISE NOTICE 'Calculating actual count...';\n EXECUTE format(\n 'SELECT count(*) FROM items WHERE %s',\n inwhere\n ) INTO sw.total_count;\n i := clock_timestamp() - t;\n RAISE NOTICE 'Actual Count: % -- %', sw.total_count, i;\n sw.time_to_count := extract(epoch FROM i);\n ELSE\n sw.total_count := NULL;\n sw.time_to_count := NULL;\n END IF;\n\n\n INSERT INTO search_wheres\n (_where, lastused, usecount, statslastupdated, estimated_count, estimated_cost, time_to_estimate, partitions, total_count, time_to_count)\n SELECT sw._where, sw.lastused, sw.usecount, sw.statslastupdated, sw.estimated_count, sw.estimated_cost, sw.time_to_estimate, sw.partitions, sw.total_count, sw.time_to_count\n ON CONFLICT ((md5(_where)))\n DO UPDATE\n SET\n lastused = sw.lastused,\n usecount = sw.usecount,\n statslastupdated = sw.statslastupdated,\n estimated_count = sw.estimated_count,\n estimated_cost = sw.estimated_cost,\n time_to_estimate = sw.time_to_estimate,\n partitions = sw.partitions,\n total_count = sw.total_count,\n time_to_count = sw.time_to_count\n ;\n RETURN sw;\nEND;\n$$ LANGUAGE PLPGSQL ;\n\n\nCREATE OR REPLACE FUNCTION items_count(_where text) RETURNS bigint AS $$\nDECLARE\ncnt bigint;\nBEGIN\nEXECUTE format('SELECT count(*) FROM items WHERE %s', _where) INTO cnt;\nRETURN cnt;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n\n\n\nDROP FUNCTION IF EXISTS search_query;\nCREATE OR REPLACE FUNCTION search_query(\n _search jsonb = '{}'::jsonb,\n updatestats boolean = false,\n _metadata jsonb = '{}'::jsonb\n) RETURNS searches AS $$\nDECLARE\n search searches%ROWTYPE;\n pexplain jsonb;\n t timestamptz;\n i interval;\nBEGIN\nSELECT * INTO search FROM searches\nWHERE hash=search_hash(_search, _metadata) FOR UPDATE;\n\n-- Calculate the where clause if not already calculated\nIF search._where IS NULL THEN\n search._where := cql_to_where(_search);\nEND IF;\n\n-- Calculate the order by clause if not already calculated\nIF search.orderby IS NULL THEN\n search.orderby := sort_sqlorderby(_search);\nEND IF;\n\nPERFORM where_stats(search._where, updatestats, _search->'conf');\n\nsearch.lastused := now();\nsearch.usecount := coalesce(search.usecount, 0) + 1;\nINSERT INTO searches (search, _where, orderby, lastused, usecount, metadata)\nVALUES (_search, search._where, search.orderby, search.lastused, search.usecount, _metadata)\nON CONFLICT (hash) DO\nUPDATE SET\n _where = EXCLUDED._where,\n orderby = EXCLUDED.orderby,\n lastused = EXCLUDED.lastused,\n usecount = EXCLUDED.usecount,\n metadata = EXCLUDED.metadata\nRETURNING * INTO search\n;\nRETURN search;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n\nCREATE OR REPLACE FUNCTION search(_search jsonb = '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n searches searches%ROWTYPE;\n _where text;\n token_where text;\n full_where text;\n orderby text;\n query text;\n token_type text := substr(_search->>'token',1,4);\n _limit int := coalesce((_search->>'limit')::int, 10);\n curs refcursor;\n cntr int := 0;\n iter_record items%ROWTYPE;\n first_record items%ROWTYPE;\n last_record items%ROWTYPE;\n out_records jsonb := '[]'::jsonb;\n prev_query text;\n next text;\n prev_id text;\n has_next boolean := false;\n has_prev boolean := false;\n prev text;\n total_count bigint;\n context jsonb;\n collection jsonb;\n includes text[];\n excludes text[];\n exit_flag boolean := FALSE;\n batches int := 0;\n timer timestamptz := clock_timestamp();\n pstart timestamptz;\n pend timestamptz;\n pcurs refcursor;\n search_where search_wheres%ROWTYPE;\nBEGIN\nsearches := search_query(_search);\n_where := searches._where;\norderby := searches.orderby;\nsearch_where := where_stats(_where);\ntotal_count := coalesce(search_where.total_count, search_where.estimated_count);\n\n\nIF token_type='prev' THEN\n token_where := get_token_filter(_search, null::jsonb);\n orderby := sort_sqlorderby(_search, TRUE);\nEND IF;\nIF token_type='next' THEN\n token_where := get_token_filter(_search, null::jsonb);\nEND IF;\n\nfull_where := concat_ws(' AND ', _where, token_where);\nRAISE NOTICE 'FULL QUERY % %', full_where, clock_timestamp()-timer;\ntimer := clock_timestamp();\n\nCREATE TEMP TABLE results (content jsonb) ON COMMIT DROP;\n\n\nFOR query IN SELECT partition_queries(full_where, orderby, search_where.partitions) LOOP\n timer := clock_timestamp();\n query := format('%s LIMIT %s', query, _limit + 1);\n RAISE NOTICE 'Partition Query: %', query;\n batches := batches + 1;\n -- curs = create_cursor(query);\n OPEN curs FOR EXECUTE query;\n LOOP\n FETCH curs into iter_record;\n EXIT WHEN NOT FOUND;\n cntr := cntr + 1;\n last_record := iter_record;\n IF cntr = 1 THEN\n first_record := last_record;\n END IF;\n IF cntr <= _limit THEN\n INSERT INTO results (content) VALUES (last_record.content);\n -- out_records := out_records || last_record.content;\n\n ELSIF cntr > _limit THEN\n has_next := true;\n exit_flag := true;\n EXIT;\n END IF;\n END LOOP;\n CLOSE curs;\n RAISE NOTICE 'Query took %. Total Time %', clock_timestamp()-timer, ftime();\n timer := clock_timestamp();\n EXIT WHEN exit_flag;\nEND LOOP;\nRAISE NOTICE 'Scanned through % partitions.', batches;\n\nSELECT jsonb_agg(content) INTO out_records FROM results;\n\nDROP TABLE results;\n\n\n-- Flip things around if this was the result of a prev token query\nIF token_type='prev' THEN\n out_records := flip_jsonb_array(out_records);\n first_record := last_record;\nEND IF;\n\n-- If this query has a token, see if there is data before the first record\nIF _search ? 'token' THEN\n prev_query := format(\n 'SELECT 1 FROM items WHERE %s LIMIT 1',\n concat_ws(\n ' AND ',\n _where,\n trim(get_token_filter(_search, to_jsonb(first_record)))\n )\n );\n RAISE NOTICE 'Query to get previous record: % --- %', prev_query, first_record;\n EXECUTE prev_query INTO has_prev;\n IF FOUND and has_prev IS NOT NULL THEN\n RAISE NOTICE 'Query results from prev query: %', has_prev;\n has_prev := TRUE;\n END IF;\nEND IF;\nhas_prev := COALESCE(has_prev, FALSE);\n\nRAISE NOTICE 'token_type: %, has_next: %, has_prev: %', token_type, has_next, has_prev;\nIF has_prev THEN\n prev := out_records->0->>'id';\nEND IF;\nIF has_next OR token_type='prev' THEN\n next := out_records->-1->>'id';\nEND IF;\n\n\n-- include/exclude any fields following fields extension\nIF _search ? 'fields' THEN\n IF _search->'fields' ? 'exclude' THEN\n excludes=textarr(_search->'fields'->'exclude');\n END IF;\n IF _search->'fields' ? 'include' THEN\n includes=textarr(_search->'fields'->'include');\n IF array_length(includes, 1)>0 AND NOT 'id' = ANY (includes) THEN\n includes = includes || '{id}';\n END IF;\n END IF;\n SELECT jsonb_agg(filter_jsonb(row, includes, excludes)) INTO out_records FROM jsonb_array_elements(out_records) row;\nEND IF;\n\nIF context(_search->'conf') != 'off' THEN\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'matched', total_count,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\nELSE\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\nEND IF;\n\ncollection := jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb),\n 'next', next,\n 'prev', prev,\n 'context', context\n);\n\nRETURN collection;\nEND;\n$$ LANGUAGE PLPGSQL\nSET jit TO off\n;\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION tileenvelope(zoom int, x int, y int) RETURNS geometry AS $$\nWITH t AS (\n SELECT\n 20037508.3427892 as merc_max,\n -20037508.3427892 as merc_min,\n (2 * 20037508.3427892) / (2 ^ zoom) as tile_size\n)\nSELECT st_makeenvelope(\n merc_min + (tile_size * x),\n merc_max - (tile_size * (y + 1)),\n merc_min + (tile_size * (x + 1)),\n merc_max - (tile_size * y),\n 3857\n) FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;DROP FUNCTION IF EXISTS mercgrid;\n\n\nCREATE OR REPLACE FUNCTION ftime() RETURNS interval as $$\nSELECT age(clock_timestamp(), transaction_timestamp());\n$$ LANGUAGE SQL;\nSET SEARCH_PATH to pgstac, public;\n\nDROP FUNCTION IF EXISTS geometrysearch;\nCREATE OR REPLACE FUNCTION geometrysearch(\n IN geom geometry,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE, -- Return as soon as the passed in geometry is full covered\n IN skipcovered boolean DEFAULT TRUE -- Skip any items that would show up completely under the previous items\n) RETURNS jsonb AS $$\nDECLARE\n search searches%ROWTYPE;\n curs refcursor;\n _where text;\n query text;\n iter_record items%ROWTYPE;\n out_records jsonb[] := '{}'::jsonb[];\n exit_flag boolean := FALSE;\n counter int := 1;\n scancounter int := 1;\n remaining_limit int := _scanlimit;\n tilearea float;\n unionedgeom geometry;\n clippedgeom geometry;\n unionedgeom_area float := 0;\n prev_area float := 0;\n excludes text[];\n includes text[];\n\nBEGIN\n -- If skipcovered is true then you will always want to exit when the passed in geometry is full\n IF skipcovered THEN\n exitwhenfull := TRUE;\n END IF;\n\n SELECT * INTO search FROM searches WHERE hash=queryhash;\n\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Search with Query Hash % Not Found', queryhash;\n END IF;\n\n tilearea := st_area(geom);\n _where := format('%s AND st_intersects(geometry, %L::geometry)', search._where, geom);\n\n IF fields IS NOT NULL THEN\n IF fields ? 'fields' THEN\n fields := fields->'fields';\n END IF;\n IF fields ? 'exclude' THEN\n excludes=textarr(fields->'exclude');\n END IF;\n IF fields ? 'include' THEN\n includes=textarr(fields->'include');\n IF array_length(includes, 1)>0 AND NOT 'id' = ANY (includes) THEN\n includes = includes || '{id}';\n END IF;\n END IF;\n END IF;\n RAISE NOTICE 'fields: %, includes: %, excludes: %', fields, includes, excludes;\n\n FOR query IN SELECT * FROM partition_queries(_where, search.orderby) LOOP\n query := format('%s LIMIT %L', query, remaining_limit);\n RAISE NOTICE '%', query;\n curs = create_cursor(query);\n LOOP\n FETCH curs INTO iter_record;\n EXIT WHEN NOT FOUND;\n IF exitwhenfull OR skipcovered THEN -- If we are not using exitwhenfull or skipcovered, we do not need to do expensive geometry operations\n clippedgeom := st_intersection(geom, iter_record.geometry);\n\n IF unionedgeom IS NULL THEN\n unionedgeom := clippedgeom;\n ELSE\n unionedgeom := st_union(unionedgeom, clippedgeom);\n END IF;\n\n unionedgeom_area := st_area(unionedgeom);\n\n IF skipcovered AND prev_area = unionedgeom_area THEN\n scancounter := scancounter + 1;\n CONTINUE;\n END IF;\n\n prev_area := unionedgeom_area;\n\n RAISE NOTICE '% % % %', unionedgeom_area/tilearea, counter, scancounter, ftime();\n END IF;\n\n IF fields IS NOT NULL THEN\n out_records := out_records || filter_jsonb(iter_record.content, includes, excludes);\n ELSE\n out_records := out_records || iter_record.content;\n END IF;\n IF counter >= _limit\n OR scancounter > _scanlimit\n OR ftime() > _timelimit\n OR (exitwhenfull AND unionedgeom_area >= tilearea)\n THEN\n exit_flag := TRUE;\n EXIT;\n END IF;\n counter := counter + 1;\n scancounter := scancounter + 1;\n\n END LOOP;\n EXIT WHEN exit_flag;\n remaining_limit := _scanlimit - scancounter;\n END LOOP;\n\n RETURN jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', array_to_json(out_records)::jsonb\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDROP FUNCTION IF EXISTS geojsonsearch;\nCREATE OR REPLACE FUNCTION geojsonsearch(\n IN geojson jsonb,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_geomfromgeojson(geojson),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS xyzsearch;\nCREATE OR REPLACE FUNCTION xyzsearch(\n IN _x int,\n IN _y int,\n IN _z int,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_transform(tileenvelope(_z, _x, _y), 4326),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\nSELECT set_version('0.4.1');\n" }, { "path": "src/pgstac/migrations/pgstac.0.4.2-0.4.3.sql", "content": "SET SEARCH_PATH to pgstac, public;\nset check_function_bodies = off;\n\nCREATE OR REPLACE FUNCTION pgstac.cql2_query(j jsonb, recursion integer DEFAULT 0)\n RETURNS text\n LANGUAGE plpgsql\nAS $function$\nDECLARE\nargs jsonb := j->'args';\njtype text := jsonb_typeof(j->'args');\nop text := lower(j->>'op');\narg jsonb;\nargtext text;\nargstext text[] := '{}'::text[];\ninobj jsonb;\n_numeric text := '';\nops jsonb :=\n '{\n \"eq\": \"%s = %s\",\n \"lt\": \"%s < %s\",\n \"lte\": \"%s <= %s\",\n \"gt\": \"%s > %s\",\n \"gte\": \"%s >= %s\",\n \"le\": \"%s <= %s\",\n \"ge\": \"%s >= %s\",\n \"=\": \"%s = %s\",\n \"<\": \"%s < %s\",\n \"<=\": \"%s <= %s\",\n \">\": \"%s > %s\",\n \">=\": \"%s >= %s\",\n \"like\": \"%s LIKE %s\",\n \"ilike\": \"%s ILIKE %s\",\n \"+\": \"%s + %s\",\n \"-\": \"%s - %s\",\n \"*\": \"%s * %s\",\n \"/\": \"%s / %s\",\n \"in\": \"%s = ANY (%s)\",\n \"not\": \"NOT (%s)\",\n \"between\": \"%s BETWEEN (%2$s)[1] AND (%2$s)[2]\",\n \"lower\":\" lower(%s)\",\n \"upper\":\" upper(%s)\",\n \"isnull\": \"%s IS NULL\"\n }'::jsonb;\nret text;\n\nBEGIN\nRAISE NOTICE 'j: %s', j;\nIF j ? 'filter' THEN\n RETURN cql2_query(j->'filter');\nEND IF;\n\nIF j ? 'upper' THEN\nRAISE NOTICE 'upper %s',jsonb_build_object(\n 'op', 'upper',\n 'args', jsonb_build_array( j-> 'upper')\n ) ;\n RETURN cql2_query(\n jsonb_build_object(\n 'op', 'upper',\n 'args', jsonb_build_array( j-> 'upper')\n )\n );\nEND IF;\n\nIF j ? 'lower' THEN\n RETURN cql2_query(\n jsonb_build_object(\n 'op', 'lower',\n 'args', jsonb_build_array( j-> 'lower')\n )\n );\nEND IF;\n\nIF j ? 'args' AND jsonb_typeof(args) != 'array' THEN\n args := jsonb_build_array(args);\nEND IF;\n-- END Cases where no further nesting is expected\nIF j ? 'op' THEN\n -- Special case to use JSONB index for equality\n IF op IN ('eq', '=')\n AND args->0 ? 'property'\n AND jsonb_typeof(args->1) IN ('number', 'string')\n AND (items_path(args->0->>'property')).eq IS NOT NULL\n THEN\n RETURN format((items_path(args->0->>'property')).eq, args->1);\n END IF;\n\n -- Temporal Query\n IF op ilike 't_%' or op = 'anyinteracts' THEN\n RETURN temporal_op_query(op, args);\n END IF;\n\n -- Spatial Query\n IF op ilike 's_%' or op = 'intersects' THEN\n RETURN spatial_op_query(op, args);\n END IF;\n\n -- In Query - separate into separate eq statements so that we can use eq jsonb optimization\n IF op = 'in' THEN\n RAISE NOTICE '% IN args: %', repeat(' ', recursion), args;\n SELECT INTO inobj\n jsonb_agg(\n jsonb_build_object(\n 'op', 'eq',\n 'args', jsonb_build_array( args->0 , v)\n )\n )\n FROM jsonb_array_elements( args->1) v;\n RETURN cql2_query(jsonb_build_object('op','or','args',inobj));\n END IF;\nEND IF;\n\nIF j ? 'property' THEN\n RETURN (items_path(j->>'property')).path_txt;\nEND IF;\n\nIF j ? 'timestamp' THEN\n RETURN quote_literal(j->>'timestamp');\nEND IF;\n\nRAISE NOTICE '%jtype: %',repeat(' ', recursion), jtype;\nIF jsonb_typeof(j) = 'number' THEN\n RETURN format('%L::numeric', j->>0);\nEND IF;\n\nIF jsonb_typeof(j) = 'string' THEN\n RETURN quote_literal(j->>0);\nEND IF;\n\nIF jsonb_typeof(j) = 'array' THEN\n IF j @? '$[*] ? (@.type() == \"number\")' THEN\n RETURN CONCAT(quote_literal(textarr(j)::text), '::numeric[]');\n ELSE\n RETURN CONCAT(quote_literal(textarr(j)::text), '::text[]');\n END IF;\nEND IF;\nRAISE NOTICE 'ARGS after array cleaning: %', args;\n\nRAISE NOTICE '%beforeargs op: %, args: %',repeat(' ', recursion), op, args;\nIF j ? 'args' THEN\n FOR arg in SELECT * FROM jsonb_array_elements(args) LOOP\n argtext := cql2_query(arg, recursion + 1);\n RAISE NOTICE '% -- arg: %, argtext: %', repeat(' ', recursion), arg, argtext;\n argstext := argstext || argtext;\n END LOOP;\nEND IF;\nRAISE NOTICE '%afterargs op: %, argstext: %',repeat(' ', recursion), op, argstext;\n\n\nIF op IN ('and', 'or') THEN\n RAISE NOTICE 'inand op: %, argstext: %', op, argstext;\n SELECT\n concat(' ( ',array_to_string(array_agg(e), concat(' ',op,' ')),' ) ')\n INTO ret\n FROM unnest(argstext) e;\n RETURN ret;\nEND IF;\n\nIF ops ? op THEN\n IF argstext[2] ~* 'numeric' THEN\n argstext := ARRAY[concat('(',argstext[1],')::numeric')] || argstext[2:3];\n END IF;\n RETURN format(concat('(',ops->>op,')'), VARIADIC argstext);\nEND IF;\n\nRAISE NOTICE '%op: %, argstext: %',repeat(' ', recursion), op, argstext;\n\nRETURN NULL;\nEND;\n$function$\n;\n\n\n\nSELECT set_version('0.4.3');\n" }, { "path": "src/pgstac/migrations/pgstac.0.4.2.sql", "content": "CREATE EXTENSION IF NOT EXISTS postgis;\nCREATE SCHEMA IF NOT EXISTS pgstac;\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE TABLE migrations (\n version text PRIMARY KEY,\n datetime timestamptz DEFAULT clock_timestamp() NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION get_version() RETURNS text AS $$\n SELECT version FROM migrations ORDER BY datetime DESC, version DESC LIMIT 1;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION set_version(text) RETURNS text AS $$\n INSERT INTO migrations (version) VALUES ($1)\n ON CONFLICT DO NOTHING\n RETURNING version;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\n\nCREATE TABLE IF NOT EXISTS pgstac_settings (\n name text PRIMARY KEY,\n value text NOT NULL\n);\n\nINSERT INTO pgstac_settings (name, value) VALUES\n ('context', 'off'),\n ('context_estimated_count', '100000'),\n ('context_estimated_cost', '1000000'),\n ('context_stats_ttl', '1 day'),\n ('default-filter-lang', 'cql2-json')\nON CONFLICT DO NOTHING\n;\n\n\nCREATE OR REPLACE FUNCTION get_setting(IN _setting text, IN conf jsonb DEFAULT NULL) RETURNS text AS $$\nSELECT COALESCE(\n conf->>_setting,\n current_setting(concat('pgstac.',_setting), TRUE),\n (SELECT value FROM pgstac_settings WHERE name=_setting)\n);\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context();\nCREATE OR REPLACE FUNCTION context(conf jsonb DEFAULT NULL) RETURNS text AS $$\n SELECT get_setting('context', conf);\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_estimated_count();\nCREATE OR REPLACE FUNCTION context_estimated_count(conf jsonb DEFAULT NULL) RETURNS int AS $$\n SELECT get_setting('context_estimated_count', conf)::int;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_estimated_cost();\nCREATE OR REPLACE FUNCTION context_estimated_cost(conf jsonb DEFAULT NULL) RETURNS float AS $$\n SELECT get_setting('context_estimated_cost', conf)::float;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_stats_ttl();\nCREATE OR REPLACE FUNCTION context_stats_ttl(conf jsonb DEFAULT NULL) RETURNS interval AS $$\n SELECT get_setting('context_stats_ttl', conf)::interval;\n$$ LANGUAGE SQL;\n\n\nCREATE OR REPLACE FUNCTION notice(VARIADIC text[]) RETURNS boolean AS $$\nDECLARE\ndebug boolean := current_setting('pgstac.debug', true);\nBEGIN\n IF debug THEN\n RAISE NOTICE 'NOTICE FROM FUNC: % >>>>> %', concat_ws(' | ', $1), clock_timestamp();\n RETURN TRUE;\n END IF;\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION empty_arr(ANYARRAY) RETURNS BOOLEAN AS $$\nSELECT CASE\n WHEN $1 IS NULL THEN TRUE\n WHEN cardinality($1)<1 THEN TRUE\nELSE FALSE\nEND;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_map_ident(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_ident(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_map_literal(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_literal(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION estimated_count(_where text) RETURNS bigint AS $$\nDECLARE\nrec record;\nrows bigint;\nBEGIN\n FOR rec in EXECUTE format(\n $q$\n EXPLAIN SELECT 1 FROM items WHERE %s\n $q$,\n _where)\n LOOP\n rows := substring(rec.\"QUERY PLAN\" FROM ' rows=([[:digit:]]+)');\n EXIT WHEN rows IS NOT NULL;\n END LOOP;\n\n RETURN rows;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION array_reverse(anyarray) RETURNS anyarray AS $$\nSELECT ARRAY(\n SELECT $1[i]\n FROM generate_subscripts($1,1) AS s(i)\n ORDER BY i DESC\n);\n$$ LANGUAGE 'sql' STRICT IMMUTABLE;\n/* converts a jsonb text array to a pg text[] array */\nCREATE OR REPLACE FUNCTION textarr(_js jsonb)\n RETURNS text[] AS $$\n SELECT\n CASE jsonb_typeof(_js)\n WHEN 'array' THEN ARRAY(SELECT jsonb_array_elements_text(_js))\n ELSE ARRAY[_js->>0]\n END\n;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION jsonb_paths (IN jdata jsonb, OUT path text[], OUT value jsonb) RETURNS\nSETOF RECORD AS $$\nwith recursive extract_all as\n(\n select\n ARRAY[key]::text[] as path,\n value\n FROM jsonb_each(jdata)\nunion all\n select\n path || coalesce(obj_key, (arr_key- 1)::text),\n coalesce(obj_value, arr_value)\n from extract_all\n left join lateral\n jsonb_each(case jsonb_typeof(value) when 'object' then value end)\n as o(obj_key, obj_value)\n on jsonb_typeof(value) = 'object'\n left join lateral\n jsonb_array_elements(case jsonb_typeof(value) when 'array' then value end)\n with ordinality as a(arr_value, arr_key)\n on jsonb_typeof(value) = 'array'\n where obj_key is not null or arr_key is not null\n)\nselect *\nfrom extract_all;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION jsonb_obj_paths (IN jdata jsonb, OUT path text[], OUT value jsonb) RETURNS\nSETOF RECORD AS $$\nwith recursive extract_all as\n(\n select\n ARRAY[key]::text[] as path,\n value\n FROM jsonb_each(jdata)\nunion all\n select\n path || obj_key,\n obj_value\n from extract_all\n left join lateral\n jsonb_each(case jsonb_typeof(value) when 'object' then value end)\n as o(obj_key, obj_value)\n on jsonb_typeof(value) = 'object'\n where obj_key is not null\n)\nselect *\nfrom extract_all;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION jsonb_val_paths (IN jdata jsonb, OUT path text[], OUT value jsonb) RETURNS\nSETOF RECORD AS $$\nSELECT * FROM jsonb_obj_paths(jdata) WHERE jsonb_typeof(value) not in ('object','array');\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION path_includes(IN path text[], IN includes text[]) RETURNS BOOLEAN AS $$\nWITH t AS (SELECT unnest(includes) i)\nSELECT EXISTS (\n SELECT 1 FROM t WHERE path @> string_to_array(trim(i), '.')\n);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION path_excludes(IN path text[], IN excludes text[]) RETURNS BOOLEAN AS $$\nWITH t AS (SELECT unnest(excludes) e)\nSELECT NOT EXISTS (\n SELECT 1 FROM t WHERE path @> string_to_array(trim(e), '.')\n);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION jsonb_obj_paths_filtered (\n IN jdata jsonb,\n IN includes text[] DEFAULT ARRAY[]::text[],\n IN excludes text[] DEFAULT ARRAY[]::text[],\n OUT path text[],\n OUT value jsonb\n) RETURNS\nSETOF RECORD AS $$\nSELECT path, value\nFROM jsonb_obj_paths(jdata)\nWHERE\n CASE WHEN cardinality(includes) > 0 THEN path_includes(path, includes) ELSE TRUE END\n AND\n path_excludes(path, excludes)\n\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION filter_jsonb(\n IN jdata jsonb,\n IN includes text[] DEFAULT ARRAY[]::text[],\n IN excludes text[] DEFAULT ARRAY[]::text[]\n) RETURNS jsonb AS $$\nDECLARE\nrec RECORD;\noutj jsonb := '{}'::jsonb;\ncreated_paths text[] := '{}'::text[];\nBEGIN\n\nIF empty_arr(includes) AND empty_arr(excludes) THEN\nRAISE NOTICE 'no filter';\n RETURN jdata;\nEND IF;\nFOR rec in\nSELECT * FROM jsonb_obj_paths_filtered(jdata, includes, excludes)\nWHERE jsonb_typeof(value) != 'object'\nLOOP\n IF array_length(rec.path,1)>1 THEN\n FOR i IN 1..(array_length(rec.path,1)-1) LOOP\n IF NOT array_to_string(rec.path[1:i],'.') = ANY (created_paths) THEN\n outj := jsonb_set(outj, rec.path[1:i],'{}', true);\n created_paths := created_paths || array_to_string(rec.path[1:i],'.');\n END IF;\n END LOOP;\n END IF;\n outj := jsonb_set(outj, rec.path, rec.value, true);\n created_paths := created_paths || array_to_string(rec.path,'.');\nEND LOOP;\nRETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION flip_jsonb_array(j jsonb) RETURNS jsonb AS $$\nSELECT jsonb_agg(value) FROM (SELECT value FROM jsonb_array_elements(j) WITH ORDINALITY ORDER BY ordinality DESC) as t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n/* Functions to create an iterable of cursors over partitions. */\nCREATE OR REPLACE FUNCTION create_cursor(q text) RETURNS refcursor AS $$\nDECLARE\n curs refcursor;\nBEGIN\n OPEN curs FOR EXECUTE q;\n RETURN curs;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDROP FUNCTION IF EXISTS partition_queries;\nCREATE OR REPLACE FUNCTION partition_queries(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN partitions text[] DEFAULT '{items}'\n) RETURNS SETOF text AS $$\nDECLARE\n partition_query text;\n query text;\n p text;\n cursors refcursor;\n dstart timestamptz;\n dend timestamptz;\n step interval := '10 weeks'::interval;\nBEGIN\n\nIF _orderby ILIKE 'datetime d%' THEN\n partitions := partitions;\nELSIF _orderby ILIKE 'datetime a%' THEN\n partitions := array_reverse(partitions);\nELSE\n query := format($q$\n SELECT * FROM items\n WHERE %s\n ORDER BY %s\n $q$, _where, _orderby\n );\n\n RETURN NEXT query;\n RETURN;\nEND IF;\nRAISE NOTICE 'PARTITIONS ---> %',partitions;\nIF cardinality(partitions) > 0 THEN\n FOREACH p IN ARRAY partitions\n --EXECUTE partition_query\n LOOP\n query := format($q$\n SELECT * FROM %I\n WHERE %s\n ORDER BY %s\n $q$,\n p,\n _where,\n _orderby\n );\n RETURN NEXT query;\n END LOOP;\nEND IF;\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION partition_cursor(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC'\n) RETURNS SETOF refcursor AS $$\nDECLARE\n partition_query text;\n query text;\n p record;\n cursors refcursor;\nBEGIN\nFOR query IN SELECT * FROM partition_queries(_where, _orderby) LOOP\n RETURN NEXT create_cursor(query);\nEND LOOP;\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION partition_count(\n IN _where text DEFAULT 'TRUE'\n) RETURNS bigint AS $$\nDECLARE\n partition_query text;\n query text;\n p record;\n subtotal bigint;\n total bigint := 0;\nBEGIN\npartition_query := format($q$\n SELECT partition, tstzrange\n FROM items_partitions\n ORDER BY tstzrange DESC;\n$q$);\nRAISE NOTICE 'Partition Query: %', partition_query;\nFOR p IN\n EXECUTE partition_query\nLOOP\n query := format($q$\n SELECT count(*) FROM items\n WHERE datetime BETWEEN %L AND %L AND %s\n $q$, lower(p.tstzrange), upper(p.tstzrange), _where\n );\n RAISE NOTICE 'Query %', query;\n RAISE NOTICE 'Partition %, Count %, Total %',p.partition, subtotal, total;\n EXECUTE query INTO subtotal;\n total := subtotal + total;\nEND LOOP;\nRETURN total;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION drop_partition_constraints(IN partition text) RETURNS VOID AS $$\nDECLARE\n q text;\n end_datetime_constraint text := concat(partition, '_end_datetime_constraint');\n collections_constraint text := concat(partition, '_collections_constraint');\nBEGIN\n q := format($q$\n ALTER TABLE %I\n DROP CONSTRAINT IF EXISTS %I,\n DROP CONSTRAINT IF EXISTS %I;\n $q$,\n partition,\n end_datetime_constraint,\n collections_constraint\n );\n\n EXECUTE q;\n RETURN;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDROP FUNCTION IF EXISTS partition_checks;\nCREATE OR REPLACE FUNCTION partition_checks(\n IN partition text,\n OUT min_datetime timestamptz,\n OUT max_datetime timestamptz,\n OUT min_end_datetime timestamptz,\n OUT max_end_datetime timestamptz,\n OUT collections text[],\n OUT cnt bigint\n) RETURNS RECORD AS $$\nDECLARE\nq text;\nend_datetime_constraint text := concat(partition, '_end_datetime_constraint');\ncollections_constraint text := concat(partition, '_collections_constraint');\nBEGIN\nRAISE NOTICE 'CREATING CONSTRAINTS FOR %', partition;\nq := format($q$\n SELECT\n min(datetime),\n max(datetime),\n min(end_datetime),\n max(end_datetime),\n array_agg(DISTINCT collection_id),\n count(*)\n FROM %I;\n $q$,\n partition\n);\nEXECUTE q INTO min_datetime, max_datetime, min_end_datetime, max_end_datetime, collections, cnt;\nRAISE NOTICE '% % % % % % %', min_datetime, max_datetime, min_end_datetime, max_end_datetime, collections, cnt, ftime();\nIF cnt IS NULL or cnt = 0 THEN\n RAISE NOTICE 'Partition % is empty, removing...', partition;\n q := format($q$\n DROP TABLE IF EXISTS %I;\n $q$, partition\n );\n EXECUTE q;\n RETURN;\nEND IF;\nRAISE NOTICE 'Running Constraint DDL %', ftime();\nq := format($q$\n ALTER TABLE %I\n DROP CONSTRAINT IF EXISTS %I,\n ADD CONSTRAINT %I\n check((end_datetime >= %L) AND (end_datetime <= %L)) NOT VALID,\n DROP CONSTRAINT IF EXISTS %I,\n ADD CONSTRAINT %I\n check((collection_id = ANY(%L))) NOT VALID;\n $q$,\n partition,\n end_datetime_constraint,\n end_datetime_constraint,\n min_end_datetime,\n max_end_datetime,\n collections_constraint,\n collections_constraint,\n collections,\n partition\n);\nRAISE NOTICE 'q: %', q;\n\nEXECUTE q;\nRAISE NOTICE 'Returning %', ftime();\nRETURN;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION validate_constraints() RETURNS VOID AS $$\nDECLARE\nq text;\nBEGIN\nFOR q IN\n SELECT FORMAT(\n 'ALTER TABLE %I.%I.%I VALIDATE CONSTRAINT %I;',\n current_database(),\n nsp.nspname,\n cls.relname,\n con.conname\n )\n FROM pg_constraint AS con\n JOIN pg_class AS cls\n ON con.conrelid = cls.oid\n JOIN pg_namespace AS nsp\n ON cls.relnamespace = nsp.oid\n WHERE convalidated IS FALSE\n AND nsp.nspname = 'pgstac'\nLOOP\n EXECUTE q;\nEND LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n/* looks for a geometry in a stac item first from geometry and falling back to bbox */\nCREATE OR REPLACE FUNCTION stac_geom(value jsonb) RETURNS geometry AS $$\nSELECT\n CASE\n WHEN value->>'geometry' IS NOT NULL THEN\n ST_GeomFromGeoJSON(value->>'geometry')\n WHEN value->>'bbox' IS NOT NULL THEN\n ST_MakeEnvelope(\n (value->'bbox'->>0)::float,\n (value->'bbox'->>1)::float,\n (value->'bbox'->>2)::float,\n (value->'bbox'->>3)::float,\n 4326\n )\n ELSE NULL\n END as geometry\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION stac_datetime(value jsonb) RETURNS timestamptz AS $$\nSELECT COALESCE(\n (value->'properties'->>'datetime')::timestamptz,\n (value->'properties'->>'start_datetime')::timestamptz\n);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_end_datetime(value jsonb) RETURNS timestamptz AS $$\nSELECT COALESCE(\n (value->'properties'->>'datetime')::timestamptz,\n (value->'properties'->>'end_datetime')::timestamptz\n);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\n\nCREATE OR REPLACE FUNCTION stac_daterange(value jsonb) RETURNS tstzrange AS $$\nSELECT tstzrange(stac_datetime(value),stac_end_datetime(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE TABLE IF NOT EXISTS collections (\n id VARCHAR GENERATED ALWAYS AS (content->>'id') STORED PRIMARY KEY,\n content JSONB\n);\n\nCREATE OR REPLACE FUNCTION create_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection(data jsonb) RETURNS VOID AS $$\nDECLARE\nout collections%ROWTYPE;\nBEGIN\n UPDATE collections SET content=data WHERE id = data->>'id' RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ON CONFLICT (id) DO\n UPDATE\n SET content=EXCLUDED.content\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION delete_collection(_id text) RETURNS VOID AS $$\nDECLARE\nout collections%ROWTYPE;\nBEGIN\n DELETE FROM collections WHERE id = _id RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION get_collection(id text) RETURNS jsonb AS $$\nSELECT content FROM collections\nWHERE id=$1\n;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION all_collections() RETURNS jsonb AS $$\nSELECT jsonb_agg(content) FROM collections;\n;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE TABLE items (\n id text NOT NULL,\n geometry geometry NOT NULL,\n collection_id text NOT NULL,\n datetime timestamptz NOT NULL,\n end_datetime timestamptz NOT NULL,\n properties jsonb NOT NULL,\n content JSONB NOT NULL\n)\nPARTITION BY RANGE (datetime)\n;\n\nCREATE OR REPLACE FUNCTION properties_idx (IN content jsonb) RETURNS jsonb AS $$\n with recursive extract_all as\n (\n select\n ARRAY[key]::text[] as path,\n ARRAY[key]::text[] as fullpath,\n value\n FROM jsonb_each(content->'properties')\n union all\n select\n CASE WHEN obj_key IS NOT NULL THEN path || obj_key ELSE path END,\n path || coalesce(obj_key, (arr_key- 1)::text),\n coalesce(obj_value, arr_value)\n from extract_all\n left join lateral\n jsonb_each(case jsonb_typeof(value) when 'object' then value end)\n as o(obj_key, obj_value)\n on jsonb_typeof(value) = 'object'\n left join lateral\n jsonb_array_elements(case jsonb_typeof(value) when 'array' then value end)\n with ordinality as a(arr_value, arr_key)\n on jsonb_typeof(value) = 'array'\n where obj_key is not null or arr_key is not null\n )\n , paths AS (\n select\n array_to_string(path, '.') as path,\n value\n FROM extract_all\n WHERE\n jsonb_typeof(value) NOT IN ('array','object')\n ), grouped AS (\n SELECT path, jsonb_agg(distinct value) vals FROM paths group by path\n ) SELECT coalesce(jsonb_object_agg(path, CASE WHEN jsonb_array_length(vals)=1 THEN vals->0 ELSE vals END) - '{datetime}'::text[], '{}'::jsonb) FROM grouped\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET JIT TO OFF;\n\nCREATE INDEX \"datetime_idx\" ON items (datetime);\nCREATE INDEX \"end_datetime_idx\" ON items (end_datetime);\nCREATE INDEX \"properties_idx\" ON items USING GIN (properties jsonb_path_ops);\nCREATE INDEX \"collection_idx\" ON items (collection_id);\nCREATE INDEX \"geometry_idx\" ON items USING GIST (geometry);\nCREATE UNIQUE INDEX \"items_id_datetime_idx\" ON items (datetime, id);\n\nALTER TABLE items ADD CONSTRAINT items_collections_fk FOREIGN KEY (collection_id) REFERENCES collections(id) ON DELETE CASCADE DEFERRABLE;\n\nCREATE OR REPLACE FUNCTION analyze_empty_partitions() RETURNS VOID AS $$\nDECLARE\n p text;\nBEGIN\n FOR p IN SELECT partition FROM all_items_partitions WHERE est_cnt = 0 LOOP\n EXECUTE format('ANALYZE %I;', p);\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION items_partition_name(timestamptz) RETURNS text AS $$\n SELECT to_char($1, '\"items_p\"IYYY\"w\"IW');\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION items_partition_exists(text) RETURNS boolean AS $$\n SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_class WHERE relname=$1);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION items_partition_exists(timestamptz) RETURNS boolean AS $$\n SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_class WHERE relname=items_partition_name($1));\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION items_partition_create_worker(partition text, partition_start timestamptz, partition_end timestamptz) RETURNS VOID AS $$\nDECLARE\n err_context text;\nBEGIN\n EXECUTE format(\n $f$\n CREATE TABLE IF NOT EXISTS %1$I PARTITION OF items\n FOR VALUES FROM (%2$L) TO (%3$L);\n CREATE UNIQUE INDEX IF NOT EXISTS %4$I ON %1$I (id);\n $f$,\n partition,\n partition_start,\n partition_end,\n concat(partition, '_id_pk')\n );\nEXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', partition;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH to pgstac, public;\n\nCREATE OR REPLACE FUNCTION items_partition_create(ts timestamptz) RETURNS text AS $$\nDECLARE\n partition text := items_partition_name(ts);\n partition_start timestamptz;\n partition_end timestamptz;\nBEGIN\n IF items_partition_exists(partition) THEN\n RETURN partition;\n END IF;\n partition_start := date_trunc('week', ts);\n partition_end := partition_start + '1 week'::interval;\n PERFORM items_partition_create_worker(partition, partition_start, partition_end);\n RAISE NOTICE 'partition: %', partition;\n RETURN partition;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION items_partition_create(st timestamptz, et timestamptz) RETURNS SETOF text AS $$\nWITH t AS (\n SELECT\n generate_series(\n date_trunc('week',st),\n date_trunc('week', et),\n '1 week'::interval\n ) w\n)\nSELECT items_partition_create(w) FROM t;\n$$ LANGUAGE SQL;\n\n\nCREATE UNLOGGED TABLE items_staging (\n content JSONB NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION items_staging_insert_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p record;\n _partitions text[];\nBEGIN\n CREATE TEMP TABLE new_partitions ON COMMIT DROP AS\n SELECT\n items_partition_name(stac_datetime(content)) as partition,\n date_trunc('week', min(stac_datetime(content))) as partition_start\n FROM newdata\n GROUP BY 1;\n\n -- set statslastupdated in cache to be old enough cache always regenerated\n\n SELECT array_agg(partition) INTO _partitions FROM new_partitions;\n UPDATE search_wheres\n SET\n statslastupdated = NULL\n WHERE _where IN (\n SELECT _where FROM search_wheres sw WHERE sw.partitions && _partitions\n FOR UPDATE SKIP LOCKED\n )\n ;\n FOR p IN SELECT new_partitions.partition, new_partitions.partition_start, new_partitions.partition_start + '1 week'::interval as partition_end FROM new_partitions\n LOOP\n RAISE NOTICE 'Getting partition % ready.', p.partition;\n IF NOT items_partition_exists(p.partition) THEN\n RAISE NOTICE 'Creating partition %.', p.partition;\n PERFORM items_partition_create_worker(p.partition, p.partition_start, p.partition_end);\n END IF;\n PERFORM drop_partition_constraints(p.partition);\n END LOOP;\n INSERT INTO items (id, geometry, collection_id, datetime, end_datetime, properties, content)\n SELECT\n content->>'id',\n stac_geom(content),\n content->>'collection',\n stac_datetime(content),\n stac_end_datetime(content),\n properties_idx(content),\n content\n FROM newdata\n ;\n DELETE FROM items_staging;\n\n\n FOR p IN SELECT new_partitions.partition FROM new_partitions\n LOOP\n RAISE NOTICE 'Setting constraints for partition %.', p.partition;\n PERFORM partition_checks(p.partition);\n END LOOP;\n DROP TABLE IF EXISTS new_partitions;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER items_staging_insert_trigger AFTER INSERT ON items_staging REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_insert_triggerfunc();\n\n\nCREATE UNLOGGED TABLE items_staging_ignore (\n content JSONB NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION items_staging_ignore_insert_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p record;\n _partitions text[];\nBEGIN\n CREATE TEMP TABLE new_partitions ON COMMIT DROP AS\n SELECT\n items_partition_name(stac_datetime(content)) as partition,\n date_trunc('week', min(stac_datetime(content))) as partition_start\n FROM newdata\n GROUP BY 1;\n\n -- set statslastupdated in cache to be old enough cache always regenerated\n\n SELECT array_agg(partition) INTO _partitions FROM new_partitions;\n UPDATE search_wheres\n SET\n statslastupdated = NULL\n WHERE _where IN (\n SELECT _where FROM search_wheres sw WHERE sw.partitions && _partitions\n FOR UPDATE SKIP LOCKED\n )\n ;\n\n FOR p IN SELECT new_partitions.partition, new_partitions.partition_start, new_partitions.partition_start + '1 week'::interval as partition_end FROM new_partitions\n LOOP\n RAISE NOTICE 'Getting partition % ready.', p.partition;\n IF NOT items_partition_exists(p.partition) THEN\n RAISE NOTICE 'Creating partition %.', p.partition;\n PERFORM items_partition_create_worker(p.partition, p.partition_start, p.partition_end);\n END IF;\n PERFORM drop_partition_constraints(p.partition);\n END LOOP;\n\n INSERT INTO items (id, geometry, collection_id, datetime, end_datetime, properties, content)\n SELECT\n content->>'id',\n stac_geom(content),\n content->>'collection',\n stac_datetime(content),\n stac_end_datetime(content),\n properties_idx(content),\n content\n FROM newdata\n ON CONFLICT (datetime, id) DO NOTHING\n ;\n DELETE FROM items_staging;\n\n\n FOR p IN SELECT new_partitions.partition FROM new_partitions\n LOOP\n RAISE NOTICE 'Setting constraints for partition %.', p.partition;\n PERFORM partition_checks(p.partition);\n END LOOP;\n DROP TABLE IF EXISTS new_partitions;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER items_staging_ignore_insert_trigger AFTER INSERT ON items_staging_ignore REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_ignore_insert_triggerfunc();\n\nCREATE UNLOGGED TABLE items_staging_upsert (\n content JSONB NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION items_staging_upsert_insert_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p record;\n _partitions text[];\nBEGIN\n CREATE TEMP TABLE new_partitions ON COMMIT DROP AS\n SELECT\n items_partition_name(stac_datetime(content)) as partition,\n date_trunc('week', min(stac_datetime(content))) as partition_start\n FROM newdata\n GROUP BY 1;\n\n -- set statslastupdated in cache to be old enough cache always regenerated\n\n SELECT array_agg(partition) INTO _partitions FROM new_partitions;\n UPDATE search_wheres\n SET\n statslastupdated = NULL\n WHERE _where IN (\n SELECT _where FROM search_wheres sw WHERE sw.partitions && _partitions\n FOR UPDATE SKIP LOCKED\n )\n ;\n\n FOR p IN SELECT new_partitions.partition, new_partitions.partition_start, new_partitions.partition_start + '1 week'::interval as partition_end FROM new_partitions\n LOOP\n RAISE NOTICE 'Getting partition % ready.', p.partition;\n IF NOT items_partition_exists(p.partition) THEN\n RAISE NOTICE 'Creating partition %.', p.partition;\n PERFORM items_partition_create_worker(p.partition, p.partition_start, p.partition_end);\n END IF;\n PERFORM drop_partition_constraints(p.partition);\n END LOOP;\n\n INSERT INTO items (id, geometry, collection_id, datetime, end_datetime, properties, content)\n SELECT\n content->>'id',\n stac_geom(content),\n content->>'collection',\n stac_datetime(content),\n stac_end_datetime(content),\n properties_idx(content),\n content\n FROM newdata\n ON CONFLICT (datetime, id) DO UPDATE SET\n content = EXCLUDED.content\n WHERE items.content IS DISTINCT FROM EXCLUDED.content\n ;\n DELETE FROM items_staging;\n\n\n FOR p IN SELECT new_partitions.partition FROM new_partitions\n LOOP\n RAISE NOTICE 'Setting constraints for partition %.', p.partition;\n PERFORM partition_checks(p.partition);\n END LOOP;\n DROP TABLE IF EXISTS new_partitions;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER items_staging_upsert_insert_trigger AFTER INSERT ON items_staging_upsert REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_upsert_insert_triggerfunc();\n\nCREATE OR REPLACE FUNCTION items_update_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\nBEGIN\n NEW.id := NEW.content->>'id';\n NEW.datetime := stac_datetime(NEW.content);\n NEW.end_datetime := stac_end_datetime(NEW.content);\n NEW.collection_id := NEW.content->>'collection';\n NEW.geometry := stac_geom(NEW.content);\n NEW.properties := properties_idx(NEW.content);\n IF TG_OP = 'UPDATE' AND NEW IS NOT DISTINCT FROM OLD THEN\n RETURN NULL;\n END IF;\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER items_update_trigger BEFORE UPDATE ON items\n FOR EACH ROW EXECUTE PROCEDURE items_update_triggerfunc();\n\n/*\nView to get a table of available items partitions\nwith date ranges\n*/\nDROP VIEW IF EXISTS all_items_partitions CASCADE;\nCREATE VIEW all_items_partitions AS\nWITH base AS\n(SELECT\n c.oid::pg_catalog.regclass::text as partition,\n pg_catalog.pg_get_expr(c.relpartbound, c.oid) as _constraint,\n regexp_matches(\n pg_catalog.pg_get_expr(c.relpartbound, c.oid),\n E'\\\\(''\\([0-9 :+-]*\\)''\\\\).*\\\\(''\\([0-9 :+-]*\\)''\\\\)'\n ) as t,\n reltuples::bigint as est_cnt\nFROM pg_catalog.pg_class c, pg_catalog.pg_inherits i\nWHERE c.oid = i.inhrelid AND i.inhparent = 'items'::regclass)\nSELECT partition, tstzrange(\n t[1]::timestamptz,\n t[2]::timestamptz\n), t[1]::timestamptz as pstart,\n t[2]::timestamptz as pend, est_cnt\nFROM base\nORDER BY 2 desc;\n\nCREATE OR REPLACE VIEW items_partitions AS\nSELECT * FROM all_items_partitions WHERE est_cnt>0;\n\nCREATE OR REPLACE FUNCTION get_item(_id text) RETURNS jsonb AS $$\n SELECT content FROM items WHERE id=_id;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION delete_item(_id text) RETURNS VOID AS $$\nDECLARE\nout items%ROWTYPE;\nBEGIN\n DELETE FROM items WHERE id = _id RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION create_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION update_item(data jsonb) RETURNS VOID AS $$\nDECLARE\n out items%ROWTYPE;\nBEGIN\n UPDATE items SET content=data WHERE id = data->>'id' RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION create_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION collection_bbox(id text) RETURNS jsonb AS $$\nSELECT (replace(replace(replace(st_extent(geometry)::text,'BOX(','[['),')',']]'),' ',','))::jsonb\nFROM items WHERE collection_id=$1;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION collection_temporal_extent(id text) RETURNS jsonb AS $$\nSELECT to_jsonb(array[array[min(datetime)::text, max(datetime)::text]])\nFROM items WHERE collection_id=$1;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection_extents() RETURNS VOID AS $$\nUPDATE collections SET\n content = content ||\n jsonb_build_object(\n 'extent', jsonb_build_object(\n 'spatial', jsonb_build_object(\n 'bbox', collection_bbox(collections.id)\n ),\n 'temporal', jsonb_build_object(\n 'interval', collection_temporal_extent(collections.id)\n )\n )\n )\n;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION items_path(\n IN dotpath text,\n OUT field text,\n OUT path text,\n OUT path_txt text,\n OUT jsonpath text,\n OUT eq text\n) RETURNS RECORD AS $$\nDECLARE\npath_elements text[];\nlast_element text;\nBEGIN\ndotpath := replace(trim(dotpath), 'properties.', '');\n\nIF dotpath = '' THEN\n RETURN;\nEND IF;\n\npath_elements := string_to_array(dotpath, '.');\njsonpath := NULL;\n\nIF path_elements[1] IN ('id','geometry','datetime') THEN\n field := path_elements[1];\n path_elements := path_elements[2:];\nELSIF path_elements[1] = 'collection' THEN\n field := 'collection_id';\n path_elements := path_elements[2:];\nELSIF path_elements[1] IN ('links', 'assets', 'stac_version', 'stac_extensions') THEN\n field := 'content';\nELSE\n field := 'content';\n path_elements := '{properties}'::text[] || path_elements;\nEND IF;\nIF cardinality(path_elements)<1 THEN\n path := field;\n path_txt := field;\n jsonpath := '$';\n eq := NULL;\n RETURN;\nEND IF;\n\n\nlast_element := path_elements[cardinality(path_elements)];\npath_elements := path_elements[1:cardinality(path_elements)-1];\njsonpath := concat(array_to_string('{$}'::text[] || array_map_ident(path_elements), '.'), '.', quote_ident(last_element));\npath_elements := array_map_literal(path_elements);\npath := format($F$ properties->%s $F$, quote_literal(dotpath));\npath_txt := format($F$ properties->>%s $F$, quote_literal(dotpath));\neq := format($F$ properties @? '$.%s[*] ? (@ == %%s) '$F$, quote_ident(dotpath));\n\nRAISE NOTICE 'ITEMS PATH -- % % % % %', field, path, path_txt, jsonpath, eq;\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION parse_dtrange(IN _indate jsonb, OUT _tstzrange tstzrange) AS $$\nWITH t AS (\n SELECT CASE\n WHEN _indate ? 'timestamp' THEN\n ARRAY[_indate->>'timestamp', 'infinity']\n WHEN _indate ? 'interval' THEN\n textarr(_indate->'interval')\n WHEN jsonb_typeof(_indate) = 'array' THEN\n textarr(_indate)\n ELSE\n regexp_split_to_array(\n btrim(_indate::text,'\"'),\n '/'\n )\n END AS arr\n)\n, t1 AS (\n SELECT\n CASE\n WHEN array_upper(arr,1) = 1 OR arr[1] = '..' OR arr[1] IS NULL THEN '-infinity'::timestamptz\n ELSE arr[1]::timestamptz\n END AS st,\n CASE\n WHEN array_upper(arr,1) = 1 THEN arr[1]::timestamptz\n WHEN arr[2] = '..' OR arr[2] IS NULL THEN 'infinity'::timestamptz\n ELSE arr[2]::timestamptz\n END AS et\n FROM t\n)\nSELECT\n tstzrange(st,et)\nFROM t1;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION bbox_geom(_bbox jsonb) RETURNS geometry AS $$\nSELECT CASE jsonb_array_length(_bbox)\n WHEN 4 THEN\n ST_SetSRID(ST_MakeEnvelope(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float,\n (_bbox->>3)::float\n ),4326)\n WHEN 6 THEN\n ST_SetSRID(ST_3DMakeBox(\n ST_MakePoint(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float\n ),\n ST_MakePoint(\n (_bbox->>3)::float,\n (_bbox->>4)::float,\n (_bbox->>5)::float\n )\n ),4326)\n ELSE null END;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION cql_and_append(existing jsonb, newfilters jsonb) RETURNS jsonb AS $$\nSELECT CASE WHEN existing ? 'filter' AND newfilters IS NOT NULL THEN\n jsonb_build_object(\n 'and',\n jsonb_build_array(\n existing->'filter',\n newfilters\n )\n )\nELSE\n newfilters\nEND;\n$$ LANGUAGE SQL;\n\n\n-- ADDs base filters (ids, collections, datetime, bbox, intersects) that are\n-- added outside of the filter/query in the stac request\nCREATE OR REPLACE FUNCTION add_filters_to_cql(j jsonb) RETURNS jsonb AS $$\nDECLARE\nnewprop jsonb;\nnewprops jsonb := '[]'::jsonb;\nBEGIN\nIF j ? 'ids' THEN\n newprop := jsonb_build_object(\n 'in',\n jsonb_build_array(\n '{\"property\":\"id\"}'::jsonb,\n j->'ids'\n )\n );\n newprops := jsonb_insert(newprops, '{1}', newprop);\nEND IF;\nIF j ? 'collections' THEN\n newprop := jsonb_build_object(\n 'in',\n jsonb_build_array(\n '{\"property\":\"collection\"}'::jsonb,\n j->'collections'\n )\n );\n newprops := jsonb_insert(newprops, '{1}', newprop);\nEND IF;\n\nIF j ? 'datetime' THEN\n newprop := format(\n '{\"anyinteracts\":[{\"property\":\"datetime\"}, %s]}',\n j->'datetime'\n );\n newprops := jsonb_insert(newprops, '{1}', newprop);\nEND IF;\n\nIF j ? 'bbox' THEN\n newprop := format(\n '{\"intersects\":[{\"property\":\"geometry\"}, %s]}',\n j->'bbox'\n );\n newprops := jsonb_insert(newprops, '{1}', newprop);\nEND IF;\n\nIF j ? 'intersects' THEN\n newprop := format(\n '{\"intersects\":[{\"property\":\"geometry\"}, %s]}',\n j->'intersects'\n );\n newprops := jsonb_insert(newprops, '{1}', newprop);\nEND IF;\n\nRAISE NOTICE 'newprops: %', newprops;\n\nIF newprops IS NOT NULL AND jsonb_array_length(newprops) > 0 THEN\n return jsonb_set(\n j,\n '{filter}',\n cql_and_append(j, jsonb_build_object('and', newprops))\n ) - '{ids,collections,datetime,bbox,intersects}'::text[];\nEND IF;\n\nreturn j;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION query_to_cqlfilter(j jsonb) RETURNS jsonb AS $$\n-- Translates anything passed in through the deprecated \"query\" into equivalent CQL\nWITH t AS (\n SELECT key as property, value as ops\n FROM jsonb_each(j->'query')\n), t2 AS (\n SELECT property, (jsonb_each(ops)).*\n FROM t WHERE jsonb_typeof(ops) = 'object'\n UNION ALL\n SELECT property, 'eq', ops\n FROM t WHERE jsonb_typeof(ops) != 'object'\n), t3 AS (\nSELECT\n jsonb_strip_nulls(jsonb_build_object(\n 'and',\n jsonb_agg(\n jsonb_build_object(\n key,\n jsonb_build_array(\n jsonb_build_object('property',property),\n value\n )\n )\n )\n )) as qcql FROM t2\n)\nSELECT\n CASE WHEN qcql IS NOT NULL THEN\n jsonb_set(j, '{filter}', cql_and_append(j, qcql)) - 'query'\n ELSE j\n END\nFROM t3\n;\n$$ LANGUAGE SQL;\n\n\n\nCREATE OR REPLACE FUNCTION temporal_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\nll text := 'datetime';\nlh text := 'end_datetime';\nrrange tstzrange;\nrl text;\nrh text;\noutq text;\nBEGIN\nrrange := parse_dtrange(args->1);\nRAISE NOTICE 'Constructing temporal query OP: %, ARGS: %, RRANGE: %', op, args, rrange;\nop := lower(op);\nrl := format('%L::timestamptz', lower(rrange));\nrh := format('%L::timestamptz', upper(rrange));\noutq := CASE op\n WHEN 't_before' THEN 'lh < rl'\n WHEN 't_after' THEN 'll > rh'\n WHEN 't_meets' THEN 'lh = rl'\n WHEN 't_metby' THEN 'll = rh'\n WHEN 't_overlaps' THEN 'll < rl AND rl < lh < rh'\n WHEN 't_overlappedby' THEN 'rl < ll < rh AND lh > rh'\n WHEN 't_starts' THEN 'll = rl AND lh < rh'\n WHEN 't_startedby' THEN 'll = rl AND lh > rh'\n WHEN 't_during' THEN 'll > rl AND lh < rh'\n WHEN 't_contains' THEN 'll < rl AND lh > rh'\n WHEN 't_finishes' THEN 'll > rl AND lh = rh'\n WHEN 't_finishedby' THEN 'll < rl AND lh = rh'\n WHEN 't_equals' THEN 'll = rl AND lh = rh'\n WHEN 't_disjoint' THEN 'NOT (ll <= rh AND lh >= rl)'\n WHEN 't_intersects' THEN 'll <= rh AND lh >= rl'\n WHEN 'anyinteracts' THEN 'll <= rh AND lh >= rl'\nEND;\noutq := regexp_replace(outq, '\\mll\\M', ll);\noutq := regexp_replace(outq, '\\mlh\\M', lh);\noutq := regexp_replace(outq, '\\mrl\\M', rl);\noutq := regexp_replace(outq, '\\mrh\\M', rh);\noutq := format('(%s)', outq);\nRETURN outq;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION spatial_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\ngeom text;\nj jsonb := args->1;\nBEGIN\nop := lower(op);\nRAISE NOTICE 'Constructing spatial query OP: %, ARGS: %', op, args;\nIF op NOT IN ('s_equals','s_disjoint','s_touches','s_within','s_overlaps','s_crosses','s_intersects','intersects','s_contains') THEN\n RAISE EXCEPTION 'Spatial Operator % Not Supported', op;\nEND IF;\nop := regexp_replace(op, '^s_', 'st_');\nIF op = 'intersects' THEN\n op := 'st_intersects';\nEND IF;\n-- Convert geometry to WKB string\nIF j ? 'type' AND j ? 'coordinates' THEN\n geom := st_geomfromgeojson(j)::text;\nELSIF jsonb_typeof(j) = 'array' THEN\n geom := bbox_geom(j)::text;\nEND IF;\n\nRETURN format('%s(geometry, %L::geometry)', op, geom);\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n/* cql_query_op -- Parses a CQL query operation, recursing when necessary\n IN jsonb -- a subelement from a valid stac query\n IN text -- the operator being used on elements passed in\n RETURNS a SQL fragment to be used in a WHERE clause\n*/\nCREATE OR REPLACE FUNCTION cql_query_op(j jsonb, _op text DEFAULT NULL) RETURNS text AS $$\nDECLARE\njtype text := jsonb_typeof(j);\nop text := lower(_op);\nops jsonb :=\n '{\n \"eq\": \"%s = %s\",\n \"lt\": \"%s < %s\",\n \"lte\": \"%s <= %s\",\n \"gt\": \"%s > %s\",\n \"gte\": \"%s >= %s\",\n \"le\": \"%s <= %s\",\n \"ge\": \"%s >= %s\",\n \"=\": \"%s = %s\",\n \"<\": \"%s < %s\",\n \"<=\": \"%s <= %s\",\n \">\": \"%s > %s\",\n \">=\": \"%s >= %s\",\n \"like\": \"%s LIKE %s\",\n \"ilike\": \"%s ILIKE %s\",\n \"+\": \"%s + %s\",\n \"-\": \"%s - %s\",\n \"*\": \"%s * %s\",\n \"/\": \"%s / %s\",\n \"in\": \"%s = ANY (%s)\",\n \"not\": \"NOT (%s)\",\n \"between\": \"%s BETWEEN %s AND %s\",\n \"lower\":\" lower(%s)\",\n \"upper\":\" upper(%s)\",\n \"isnull\": \"%s IS NULL\"\n }'::jsonb;\nret text;\nargs text[] := NULL;\n\nBEGIN\nRAISE NOTICE 'j: %, op: %, jtype: %', j, op, jtype;\n\n-- for in, convert value, list to array syntax to match other ops\nIF op = 'in' and j ? 'value' and j ? 'list' THEN\n j := jsonb_build_array( j->'value', j->'list');\n jtype := 'array';\n RAISE NOTICE 'IN: j: %, jtype: %', j, jtype;\nEND IF;\n\nIF op = 'between' and j ? 'value' and j ? 'lower' and j ? 'upper' THEN\n j := jsonb_build_array( j->'value', j->'lower', j->'upper');\n jtype := 'array';\n RAISE NOTICE 'BETWEEN: j: %, jtype: %', j, jtype;\nEND IF;\n\nIF op = 'not' AND jtype = 'object' THEN\n j := jsonb_build_array( j );\n jtype := 'array';\n RAISE NOTICE 'NOT: j: %, jtype: %', j, jtype;\nEND IF;\n\n-- Set Lower Case on Both Arguments When Case Insensitive Flag Set\nIF op in ('eq','lt','lte','gt','gte','like') AND jsonb_typeof(j->2) = 'boolean' THEN\n IF (j->>2)::boolean THEN\n RETURN format(concat('(',ops->>op,')'), cql_query_op(jsonb_build_array(j->0), 'lower'), cql_query_op(jsonb_build_array(j->1), 'lower'));\n END IF;\nEND IF;\n\n-- Special Case when comparing a property in a jsonb field to a string or number using eq\n-- Allows to leverage GIN index on jsonb fields\nIF op = 'eq' THEN\n IF j->0 ? 'property'\n AND jsonb_typeof(j->1) IN ('number','string')\n AND (items_path(j->0->>'property')).eq IS NOT NULL\n THEN\n RETURN format((items_path(j->0->>'property')).eq, j->1);\n END IF;\nEND IF;\n\n\n\nIF op ilike 't_%' or op = 'anyinteracts' THEN\n RETURN temporal_op_query(op, j);\nEND IF;\n\nIF op ilike 's_%' or op = 'intersects' THEN\n RETURN spatial_op_query(op, j);\nEND IF;\n\n\nIF jtype = 'object' THEN\n RAISE NOTICE 'parsing object';\n IF j ? 'property' THEN\n -- Convert the property to be used as an identifier\n return (items_path(j->>'property')).path_txt;\n ELSIF _op IS NULL THEN\n -- Iterate to convert elements in an object where the operator has not been set\n -- Combining with AND\n SELECT\n array_to_string(array_agg(cql_query_op(e.value, e.key)), ' AND ')\n INTO ret\n FROM jsonb_each(j) e;\n RETURN ret;\n END IF;\nEND IF;\n\nIF jtype = 'string' THEN\n RETURN quote_literal(j->>0);\nEND IF;\n\nIF jtype ='number' THEN\n RETURN (j->>0)::numeric;\nEND IF;\n\nIF jtype = 'array' AND op IS NULL THEN\n RAISE NOTICE 'Parsing array into array arg. j: %', j;\n SELECT format($f$ '{%s}'::text[] $f$, string_agg(e,',')) INTO ret FROM jsonb_array_elements_text(j) e;\n RETURN ret;\nEND IF;\n\n\n-- If the type of the passed json is an array\n-- Calculate the arguments that will be passed to functions/operators\nIF jtype = 'array' THEN\n RAISE NOTICE 'Parsing array into args. j: %', j;\n -- If any argument is numeric, cast any text arguments to numeric\n IF j @? '$[*] ? (@.type() == \"number\")' THEN\n SELECT INTO args\n array_agg(concat('(',cql_query_op(e),')::numeric'))\n FROM jsonb_array_elements(j) e;\n ELSE\n SELECT INTO args\n array_agg(cql_query_op(e))\n FROM jsonb_array_elements(j) e;\n END IF;\n --RETURN args;\nEND IF;\nRAISE NOTICE 'ARGS after array cleaning: %', args;\n\nIF op IS NULL THEN\n RETURN args::text[];\nEND IF;\n\nIF args IS NULL OR cardinality(args) < 1 THEN\n RAISE NOTICE 'No Args';\n RETURN '';\nEND IF;\n\nIF op IN ('and','or') THEN\n SELECT\n CONCAT(\n '(',\n array_to_string(args, UPPER(CONCAT(' ',op,' '))),\n ')'\n ) INTO ret\n FROM jsonb_array_elements(j) e;\n RETURN ret;\nEND IF;\n\n-- If the op is in the ops json then run using the template in the json\nIF ops ? op THEN\n RAISE NOTICE 'ARGS: % MAPPED: %',args, array_map_literal(args);\n\n RETURN format(concat('(',ops->>op,')'), VARIADIC args);\nEND IF;\n\nRETURN j->>0;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n/* cql_query_op -- Parses a CQL query operation, recursing when necessary\n IN jsonb -- a subelement from a valid stac query\n IN text -- the operator being used on elements passed in\n RETURNS a SQL fragment to be used in a WHERE clause\n*/\nDROP FUNCTION IF EXISTS cql2_query;\nCREATE OR REPLACE FUNCTION cql2_query(j jsonb, recursion int DEFAULT 0) RETURNS text AS $$\nDECLARE\nargs jsonb := j->'args';\njtype text := jsonb_typeof(j->'args');\nop text := lower(j->>'op');\narg jsonb;\nargtext text;\nargstext text[] := '{}'::text[];\ninobj jsonb;\n_numeric text := '';\nops jsonb :=\n '{\n \"eq\": \"%s = %s\",\n \"lt\": \"%s < %s\",\n \"lte\": \"%s <= %s\",\n \"gt\": \"%s > %s\",\n \"gte\": \"%s >= %s\",\n \"le\": \"%s <= %s\",\n \"ge\": \"%s >= %s\",\n \"=\": \"%s = %s\",\n \"<\": \"%s < %s\",\n \"<=\": \"%s <= %s\",\n \">\": \"%s > %s\",\n \">=\": \"%s >= %s\",\n \"like\": \"%s LIKE %s\",\n \"ilike\": \"%s ILIKE %s\",\n \"+\": \"%s + %s\",\n \"-\": \"%s - %s\",\n \"*\": \"%s * %s\",\n \"/\": \"%s / %s\",\n \"in\": \"%s = ANY (%s)\",\n \"not\": \"NOT (%s)\",\n \"between\": \"%s BETWEEN (%2$s)[1] AND (%2$s)[2]\",\n \"lower\":\" lower(%s)\",\n \"upper\":\" upper(%s)\",\n \"isnull\": \"%s IS NULL\"\n }'::jsonb;\nret text;\n\nBEGIN\nRAISE NOTICE 'j: %s', j;\nIF j ? 'filter' THEN\n RETURN cql2_query(j->'filter');\nEND IF;\n\nIF j ? 'upper' THEN\nRAISE NOTICE 'upper %s',jsonb_build_object(\n 'op', 'upper',\n 'args', jsonb_build_array( j-> 'upper')\n ) ;\n RETURN cql2_query(\n jsonb_build_object(\n 'op', 'upper',\n 'args', jsonb_build_array( j-> 'upper')\n )\n );\nEND IF;\n\nIF j ? 'lower' THEN\n RETURN cql2_query(\n jsonb_build_object(\n 'op', 'lower',\n 'args', jsonb_build_array( j-> 'lower')\n )\n );\nEND IF;\n\nIF j ? 'args' AND jsonb_typeof(args) != 'array' THEN\n args := jsonb_build_array(args);\nEND IF;\n-- END Cases where no further nesting is expected\nIF j ? 'op' THEN\n -- Special case to use JSONB index for equality\n IF op = 'eq'\n AND args->0 ? 'property'\n AND jsonb_typeof(args->1) IN ('number', 'string')\n AND (items_path(args->0->>'property')).eq IS NOT NULL\n THEN\n RETURN format((items_path(args->0->>'property')).eq, args->1);\n END IF;\n\n -- Temporal Query\n IF op ilike 't_%' or op = 'anyinteracts' THEN\n RETURN temporal_op_query(op, args);\n END IF;\n\n -- Spatial Query\n IF op ilike 's_%' or op = 'intersects' THEN\n RETURN spatial_op_query(op, args);\n END IF;\n\n -- In Query - separate into separate eq statements so that we can use eq jsonb optimization\n IF op = 'in' THEN\n RAISE NOTICE '% IN args: %', repeat(' ', recursion), args;\n SELECT INTO inobj\n jsonb_agg(\n jsonb_build_object(\n 'op', 'eq',\n 'args', jsonb_build_array( args->0 , v)\n )\n )\n FROM jsonb_array_elements( args->1) v;\n RETURN cql2_query(jsonb_build_object('op','or','args',inobj));\n END IF;\nEND IF;\n\nIF j ? 'property' THEN\n RETURN (items_path(j->>'property')).path_txt;\nEND IF;\n\nIF j ? 'timestamp' THEN\n RETURN quote_literal(j->>'timestamp');\nEND IF;\n\nRAISE NOTICE '%jtype: %',repeat(' ', recursion), jtype;\nIF jsonb_typeof(j) = 'number' THEN\n RETURN format('%L::numeric', j->>0);\nEND IF;\n\nIF jsonb_typeof(j) = 'string' THEN\n RETURN quote_literal(j->>0);\nEND IF;\n\nIF jsonb_typeof(j) = 'array' THEN\n IF j @? '$[*] ? (@.type() == \"number\")' THEN\n RETURN CONCAT(quote_literal(textarr(j)::text), '::numeric[]');\n ELSE\n RETURN CONCAT(quote_literal(textarr(j)::text), '::text[]');\n END IF;\nEND IF;\nRAISE NOTICE 'ARGS after array cleaning: %', args;\n\nRAISE NOTICE '%beforeargs op: %, args: %',repeat(' ', recursion), op, args;\nIF j ? 'args' THEN\n FOR arg in SELECT * FROM jsonb_array_elements(args) LOOP\n argtext := cql2_query(arg, recursion + 1);\n RAISE NOTICE '% -- arg: %, argtext: %', repeat(' ', recursion), arg, argtext;\n argstext := argstext || argtext;\n END LOOP;\nEND IF;\nRAISE NOTICE '%afterargs op: %, argstext: %',repeat(' ', recursion), op, argstext;\n\n\nIF op IN ('and', 'or') THEN\n RAISE NOTICE 'inand op: %, argstext: %', op, argstext;\n SELECT\n concat(' ( ',array_to_string(array_agg(e), concat(' ',op,' ')),' ) ')\n INTO ret\n FROM unnest(argstext) e;\n RETURN ret;\nEND IF;\n\nIF ops ? op THEN\n IF argstext[2] ~* 'numeric' THEN\n argstext := ARRAY[concat('(',argstext[1],')::numeric')] || argstext[2:3];\n END IF;\n RETURN format(concat('(',ops->>op,')'), VARIADIC argstext);\nEND IF;\n\nRAISE NOTICE '%op: %, argstext: %',repeat(' ', recursion), op, argstext;\n\nRETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n\nCREATE OR REPLACE FUNCTION cql_to_where(_search jsonb = '{}'::jsonb) RETURNS text AS $$\nDECLARE\nfilterlang text;\nsearch jsonb := _search;\n_where text;\nBEGIN\n\nRAISE NOTICE 'SEARCH CQL Final: %', search;\nfilterlang := COALESCE(\n search->>'filter-lang',\n get_setting('default-filter-lang', _search->'conf')\n);\n\nIF filterlang = 'cql-json' THEN\n search := query_to_cqlfilter(search);\n search := add_filters_to_cql(search);\n _where := cql_query_op(search->'filter');\nELSE\n _where := cql2_query(search->'filter');\nEND IF;\n\nIF trim(_where) = '' THEN\n _where := NULL;\nEND IF;\n_where := coalesce(_where, ' TRUE ');\nRETURN _where;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION parse_sort_dir(_dir text, reverse boolean default false) RETURNS text AS $$\nWITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n) SELECT\n CASE\n WHEN NOT reverse THEN d\n WHEN d = 'ASC' THEN 'DESC'\n WHEN d = 'DESC' THEN 'ASC'\n END\nFROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION sort_dir_to_op(_dir text, prev boolean default false) RETURNS text AS $$\nWITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n) SELECT\n CASE\n WHEN d = 'ASC' AND prev THEN '<='\n WHEN d = 'DESC' AND prev THEN '>='\n WHEN d = 'ASC' THEN '>='\n WHEN d = 'DESC' THEN '<='\n END\nFROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION field_orderby(p text) RETURNS text AS $$\nWITH t AS (\n SELECT\n replace(trim(substring(indexdef from 'btree \\((.*)\\)')),' ','')as s\n FROM pg_indexes WHERE schemaname='pgstac' AND tablename='items' AND indexdef ~* 'btree' AND indexdef ~* 'properties'\n) SELECT s FROM t WHERE strpos(s, lower(trim(p)))>0;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION sort_sqlorderby(\n _search jsonb DEFAULT NULL,\n reverse boolean DEFAULT FALSE\n) RETURNS text AS $$\nWITH sortby AS (\n SELECT coalesce(_search->'sortby','[{\"field\":\"datetime\", \"direction\":\"desc\"}]') as sort\n), withid AS (\n SELECT CASE\n WHEN sort @? '$[*] ? (@.field == \"id\")' THEN sort\n ELSE sort || '[{\"field\":\"id\", \"direction\":\"desc\"}]'::jsonb\n END as sort\n FROM sortby\n), withid_rows AS (\n SELECT jsonb_array_elements(sort) as value FROM withid\n),sorts AS (\n SELECT\n coalesce(field_orderby((items_path(value->>'field')).path_txt), (items_path(value->>'field')).path) as key,\n parse_sort_dir(value->>'direction', reverse) as dir\n FROM withid_rows\n)\nSELECT array_to_string(\n array_agg(concat(key, ' ', dir)),\n ', '\n) FROM sorts;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_sort_dir(sort_item jsonb) RETURNS text AS $$\nSELECT CASE WHEN sort_item->>'direction' ILIKE 'desc%' THEN 'DESC' ELSE 'ASC' END;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION get_token_filter(_search jsonb = '{}'::jsonb, token_rec jsonb DEFAULT NULL) RETURNS text AS $$\nDECLARE\n token_id text;\n filters text[] := '{}'::text[];\n prev boolean := TRUE;\n field text;\n dir text;\n sort record;\n orfilters text[] := '{}'::text[];\n andfilters text[] := '{}'::text[];\n output text;\n token_where text;\nBEGIN\n -- If no token provided return NULL\n IF token_rec IS NULL THEN\n IF NOT (_search ? 'token' AND\n (\n (_search->>'token' ILIKE 'prev:%')\n OR\n (_search->>'token' ILIKE 'next:%')\n )\n ) THEN\n RETURN NULL;\n END IF;\n prev := (_search->>'token' ILIKE 'prev:%');\n token_id := substr(_search->>'token', 6);\n SELECT to_jsonb(items) INTO token_rec FROM items WHERE id=token_id;\n END IF;\n RAISE NOTICE 'TOKEN ID: %', token_rec->'id';\n\n CREATE TEMP TABLE sorts (\n _row int GENERATED ALWAYS AS IDENTITY NOT NULL,\n _field text PRIMARY KEY,\n _dir text NOT NULL,\n _val text\n ) ON COMMIT DROP;\n\n -- Make sure we only have distinct columns to sort with taking the first one we get\n INSERT INTO sorts (_field, _dir)\n SELECT\n (items_path(value->>'field')).path,\n get_sort_dir(value)\n FROM\n jsonb_array_elements(coalesce(_search->'sortby','[{\"field\":\"datetime\",\"direction\":\"desc\"}]'))\n ON CONFLICT DO NOTHING\n ;\n RAISE NOTICE 'sorts 1: %', (SELECT jsonb_agg(to_json(sorts)) FROM sorts);\n -- Get the first sort direction provided. As the id is a primary key, if there are any\n -- sorts after id they won't do anything, so make sure that id is the last sort item.\n SELECT _dir INTO dir FROM sorts ORDER BY _row ASC LIMIT 1;\n IF EXISTS (SELECT 1 FROM sorts WHERE _field = 'id') THEN\n DELETE FROM sorts WHERE _row > (SELECT _row FROM sorts WHERE _field = 'id' ORDER BY _row ASC);\n ELSE\n INSERT INTO sorts (_field, _dir) VALUES ('id', dir);\n END IF;\n\n -- Add value from looked up item to the sorts table\n UPDATE sorts SET _val=quote_literal(token_rec->>_field);\n\n -- Check if all sorts are the same direction and use row comparison\n -- to filter\n RAISE NOTICE 'sorts 2: %', (SELECT jsonb_agg(to_json(sorts)) FROM sorts);\n\n IF (SELECT count(DISTINCT _dir) FROM sorts) = 1 THEN\n SELECT format(\n '(%s) %s (%s)',\n concat_ws(', ', VARIADIC array_agg(quote_ident(_field))),\n CASE WHEN (prev AND dir = 'ASC') OR (NOT prev AND dir = 'DESC') THEN '<' ELSE '>' END,\n concat_ws(', ', VARIADIC array_agg(_val))\n ) INTO output FROM sorts\n WHERE token_rec ? _field\n ;\n ELSE\n FOR sort IN SELECT * FROM sorts ORDER BY _row asc LOOP\n RAISE NOTICE 'SORT: %', sort;\n IF sort._row = 1 THEN\n orfilters := orfilters || format('(%s %s %s)',\n quote_ident(sort._field),\n CASE WHEN (prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC') THEN '<' ELSE '>' END,\n sort._val\n );\n ELSE\n orfilters := orfilters || format('(%s AND %s %s %s)',\n array_to_string(andfilters, ' AND '),\n quote_ident(sort._field),\n CASE WHEN (prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC') THEN '<' ELSE '>' END,\n sort._val\n );\n\n END IF;\n andfilters := andfilters || format('%s = %s',\n quote_ident(sort._field),\n sort._val\n );\n END LOOP;\n output := array_to_string(orfilters, ' OR ');\n END IF;\n DROP TABLE IF EXISTS sorts;\n token_where := concat('(',coalesce(output,'true'),')');\n IF trim(token_where) = '' THEN\n token_where := NULL;\n END IF;\n RAISE NOTICE 'TOKEN_WHERE: |%|',token_where;\n RETURN token_where;\n END;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION search_tohash(jsonb) RETURNS jsonb AS $$\n SELECT $1 - '{token,limit,context,includes,excludes}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION search_hash(jsonb, jsonb) RETURNS text AS $$\n SELECT md5(concat(search_tohash($1)::text,$2::text));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TABLE IF NOT EXISTS searches(\n hash text GENERATED ALWAYS AS (search_hash(search, metadata)) STORED PRIMARY KEY,\n search jsonb NOT NULL,\n _where text,\n orderby text,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n metadata jsonb DEFAULT '{}'::jsonb NOT NULL\n);\nCREATE TABLE IF NOT EXISTS search_wheres(\n id bigint generated always as identity primary key,\n _where text NOT NULL,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n statslastupdated timestamptz,\n estimated_count bigint,\n estimated_cost float,\n time_to_estimate float,\n total_count bigint,\n time_to_count float,\n partitions text[]\n);\n\nCREATE INDEX IF NOT EXISTS search_wheres_partitions ON search_wheres USING GIN (partitions);\nCREATE UNIQUE INDEX IF NOT EXISTS search_wheres_where ON search_wheres ((md5(_where)));\n\nCREATE OR REPLACE FUNCTION where_stats(inwhere text, updatestats boolean default false, conf jsonb default null) RETURNS search_wheres AS $$\nDECLARE\n t timestamptz;\n i interval;\n explain_json jsonb;\n partitions text[];\n sw search_wheres%ROWTYPE;\nBEGIN\n SELECT * INTO sw FROM search_wheres WHERE _where=inwhere FOR UPDATE;\n\n -- Update statistics if explicitly set, if statistics do not exist, or statistics ttl has expired\n IF NOT updatestats THEN\n RAISE NOTICE 'Checking if update is needed.';\n RAISE NOTICE 'Stats Last Updated: %', sw.statslastupdated;\n RAISE NOTICE 'TTL: %, Age: %', context_stats_ttl(conf), now() - sw.statslastupdated;\n RAISE NOTICE 'Context: %, Existing Total: %', context(conf), sw.total_count;\n IF\n sw.statslastupdated IS NULL\n OR (now() - sw.statslastupdated) > context_stats_ttl(conf)\n OR (context(conf) != 'off' AND sw.total_count IS NULL)\n THEN\n updatestats := TRUE;\n END IF;\n END IF;\n\n sw._where := inwhere;\n sw.lastused := now();\n sw.usecount := coalesce(sw.usecount,0) + 1;\n\n IF NOT updatestats THEN\n UPDATE search_wheres SET\n lastused = sw.lastused,\n usecount = sw.usecount\n WHERE _where = inwhere\n RETURNING * INTO sw\n ;\n RETURN sw;\n END IF;\n -- Use explain to get estimated count/cost and a list of the partitions that would be hit by the query\n t := clock_timestamp();\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s', inwhere)\n INTO explain_json;\n RAISE NOTICE 'Time for just the explain: %', clock_timestamp() - t;\n WITH t AS (\n SELECT j->>0 as p FROM\n jsonb_path_query(\n explain_json,\n 'strict $.**.\"Relation Name\" ? (@ != null)'\n ) j\n ), ordered AS (\n SELECT p FROM t ORDER BY p DESC\n -- SELECT p FROM t JOIN items_partitions\n -- ON (t.p = items_partitions.partition)\n -- ORDER BY pstart DESC\n )\n SELECT array_agg(p) INTO partitions FROM ordered;\n i := clock_timestamp() - t;\n RAISE NOTICE 'Time for explain + join: %', clock_timestamp() - t;\n\n\n\n sw.statslastupdated := now();\n sw.estimated_count := explain_json->0->'Plan'->'Plan Rows';\n sw.estimated_cost := explain_json->0->'Plan'->'Total Cost';\n sw.time_to_estimate := extract(epoch from i);\n sw.partitions := partitions;\n\n -- Do a full count of rows if context is set to on or if auto is set and estimates are low enough\n IF\n context(conf) = 'on'\n OR\n ( context(conf) = 'auto' AND\n (\n sw.estimated_count < context_estimated_count(conf)\n OR\n sw.estimated_cost < context_estimated_cost(conf)\n )\n )\n THEN\n t := clock_timestamp();\n RAISE NOTICE 'Calculating actual count...';\n EXECUTE format(\n 'SELECT count(*) FROM items WHERE %s',\n inwhere\n ) INTO sw.total_count;\n i := clock_timestamp() - t;\n RAISE NOTICE 'Actual Count: % -- %', sw.total_count, i;\n sw.time_to_count := extract(epoch FROM i);\n ELSE\n sw.total_count := NULL;\n sw.time_to_count := NULL;\n END IF;\n\n\n INSERT INTO search_wheres\n (_where, lastused, usecount, statslastupdated, estimated_count, estimated_cost, time_to_estimate, partitions, total_count, time_to_count)\n SELECT sw._where, sw.lastused, sw.usecount, sw.statslastupdated, sw.estimated_count, sw.estimated_cost, sw.time_to_estimate, sw.partitions, sw.total_count, sw.time_to_count\n ON CONFLICT ((md5(_where)))\n DO UPDATE\n SET\n lastused = sw.lastused,\n usecount = sw.usecount,\n statslastupdated = sw.statslastupdated,\n estimated_count = sw.estimated_count,\n estimated_cost = sw.estimated_cost,\n time_to_estimate = sw.time_to_estimate,\n partitions = sw.partitions,\n total_count = sw.total_count,\n time_to_count = sw.time_to_count\n ;\n RETURN sw;\nEND;\n$$ LANGUAGE PLPGSQL ;\n\n\nCREATE OR REPLACE FUNCTION items_count(_where text) RETURNS bigint AS $$\nDECLARE\ncnt bigint;\nBEGIN\nEXECUTE format('SELECT count(*) FROM items WHERE %s', _where) INTO cnt;\nRETURN cnt;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n\n\n\nDROP FUNCTION IF EXISTS search_query;\nCREATE OR REPLACE FUNCTION search_query(\n _search jsonb = '{}'::jsonb,\n updatestats boolean = false,\n _metadata jsonb = '{}'::jsonb\n) RETURNS searches AS $$\nDECLARE\n search searches%ROWTYPE;\n pexplain jsonb;\n t timestamptz;\n i interval;\nBEGIN\nSELECT * INTO search FROM searches\nWHERE hash=search_hash(_search, _metadata) FOR UPDATE;\n\n-- Calculate the where clause if not already calculated\nIF search._where IS NULL THEN\n search._where := cql_to_where(_search);\nEND IF;\n\n-- Calculate the order by clause if not already calculated\nIF search.orderby IS NULL THEN\n search.orderby := sort_sqlorderby(_search);\nEND IF;\n\nPERFORM where_stats(search._where, updatestats, _search->'conf');\n\nsearch.lastused := now();\nsearch.usecount := coalesce(search.usecount, 0) + 1;\nINSERT INTO searches (search, _where, orderby, lastused, usecount, metadata)\nVALUES (_search, search._where, search.orderby, search.lastused, search.usecount, _metadata)\nON CONFLICT (hash) DO\nUPDATE SET\n _where = EXCLUDED._where,\n orderby = EXCLUDED.orderby,\n lastused = EXCLUDED.lastused,\n usecount = EXCLUDED.usecount,\n metadata = EXCLUDED.metadata\nRETURNING * INTO search\n;\nRETURN search;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n\nCREATE OR REPLACE FUNCTION search(_search jsonb = '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n searches searches%ROWTYPE;\n _where text;\n token_where text;\n full_where text;\n orderby text;\n query text;\n token_type text := substr(_search->>'token',1,4);\n _limit int := coalesce((_search->>'limit')::int, 10);\n curs refcursor;\n cntr int := 0;\n iter_record items%ROWTYPE;\n first_record items%ROWTYPE;\n last_record items%ROWTYPE;\n out_records jsonb := '[]'::jsonb;\n prev_query text;\n next text;\n prev_id text;\n has_next boolean := false;\n has_prev boolean := false;\n prev text;\n total_count bigint;\n context jsonb;\n collection jsonb;\n includes text[];\n excludes text[];\n exit_flag boolean := FALSE;\n batches int := 0;\n timer timestamptz := clock_timestamp();\n pstart timestamptz;\n pend timestamptz;\n pcurs refcursor;\n search_where search_wheres%ROWTYPE;\nBEGIN\nsearches := search_query(_search);\n_where := searches._where;\norderby := searches.orderby;\nsearch_where := where_stats(_where);\ntotal_count := coalesce(search_where.total_count, search_where.estimated_count);\n\n\nIF token_type='prev' THEN\n token_where := get_token_filter(_search, null::jsonb);\n orderby := sort_sqlorderby(_search, TRUE);\nEND IF;\nIF token_type='next' THEN\n token_where := get_token_filter(_search, null::jsonb);\nEND IF;\n\nfull_where := concat_ws(' AND ', _where, token_where);\nRAISE NOTICE 'FULL QUERY % %', full_where, clock_timestamp()-timer;\ntimer := clock_timestamp();\n\nCREATE TEMP TABLE results (content jsonb) ON COMMIT DROP;\n\n\nFOR query IN SELECT partition_queries(full_where, orderby, search_where.partitions) LOOP\n timer := clock_timestamp();\n query := format('%s LIMIT %s', query, _limit + 1);\n RAISE NOTICE 'Partition Query: %', query;\n batches := batches + 1;\n -- curs = create_cursor(query);\n OPEN curs FOR EXECUTE query;\n LOOP\n FETCH curs into iter_record;\n EXIT WHEN NOT FOUND;\n cntr := cntr + 1;\n last_record := iter_record;\n IF cntr = 1 THEN\n first_record := last_record;\n END IF;\n IF cntr <= _limit THEN\n INSERT INTO results (content) VALUES (last_record.content);\n -- out_records := out_records || last_record.content;\n\n ELSIF cntr > _limit THEN\n has_next := true;\n exit_flag := true;\n EXIT;\n END IF;\n END LOOP;\n CLOSE curs;\n RAISE NOTICE 'Query took %. Total Time %', clock_timestamp()-timer, ftime();\n timer := clock_timestamp();\n EXIT WHEN exit_flag;\nEND LOOP;\nRAISE NOTICE 'Scanned through % partitions.', batches;\n\nSELECT jsonb_agg(content) INTO out_records FROM results;\n\nDROP TABLE results;\n\n\n-- Flip things around if this was the result of a prev token query\nIF token_type='prev' THEN\n out_records := flip_jsonb_array(out_records);\n first_record := last_record;\nEND IF;\n\n-- If this query has a token, see if there is data before the first record\nIF _search ? 'token' THEN\n prev_query := format(\n 'SELECT 1 FROM items WHERE %s LIMIT 1',\n concat_ws(\n ' AND ',\n _where,\n trim(get_token_filter(_search, to_jsonb(first_record)))\n )\n );\n RAISE NOTICE 'Query to get previous record: % --- %', prev_query, first_record;\n EXECUTE prev_query INTO has_prev;\n IF FOUND and has_prev IS NOT NULL THEN\n RAISE NOTICE 'Query results from prev query: %', has_prev;\n has_prev := TRUE;\n END IF;\nEND IF;\nhas_prev := COALESCE(has_prev, FALSE);\n\nRAISE NOTICE 'token_type: %, has_next: %, has_prev: %', token_type, has_next, has_prev;\nIF has_prev THEN\n prev := out_records->0->>'id';\nEND IF;\nIF has_next OR token_type='prev' THEN\n next := out_records->-1->>'id';\nEND IF;\n\n\n-- include/exclude any fields following fields extension\nIF _search ? 'fields' THEN\n IF _search->'fields' ? 'exclude' THEN\n excludes=textarr(_search->'fields'->'exclude');\n END IF;\n IF _search->'fields' ? 'include' THEN\n includes=textarr(_search->'fields'->'include');\n IF array_length(includes, 1)>0 AND NOT 'id' = ANY (includes) THEN\n includes = includes || '{id}';\n END IF;\n END IF;\n SELECT jsonb_agg(filter_jsonb(row, includes, excludes)) INTO out_records FROM jsonb_array_elements(out_records) row;\nEND IF;\n\nIF context(_search->'conf') != 'off' THEN\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'matched', total_count,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\nELSE\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\nEND IF;\n\ncollection := jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb),\n 'next', next,\n 'prev', prev,\n 'context', context\n);\n\nRETURN collection;\nEND;\n$$ LANGUAGE PLPGSQL\nSET jit TO off\n;\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION tileenvelope(zoom int, x int, y int) RETURNS geometry AS $$\nWITH t AS (\n SELECT\n 20037508.3427892 as merc_max,\n -20037508.3427892 as merc_min,\n (2 * 20037508.3427892) / (2 ^ zoom) as tile_size\n)\nSELECT st_makeenvelope(\n merc_min + (tile_size * x),\n merc_max - (tile_size * (y + 1)),\n merc_min + (tile_size * (x + 1)),\n merc_max - (tile_size * y),\n 3857\n) FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;DROP FUNCTION IF EXISTS mercgrid;\n\n\nCREATE OR REPLACE FUNCTION ftime() RETURNS interval as $$\nSELECT age(clock_timestamp(), transaction_timestamp());\n$$ LANGUAGE SQL;\nSET SEARCH_PATH to pgstac, public;\n\nDROP FUNCTION IF EXISTS geometrysearch;\nCREATE OR REPLACE FUNCTION geometrysearch(\n IN geom geometry,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE, -- Return as soon as the passed in geometry is full covered\n IN skipcovered boolean DEFAULT TRUE -- Skip any items that would show up completely under the previous items\n) RETURNS jsonb AS $$\nDECLARE\n search searches%ROWTYPE;\n curs refcursor;\n _where text;\n query text;\n iter_record items%ROWTYPE;\n out_records jsonb[] := '{}'::jsonb[];\n exit_flag boolean := FALSE;\n counter int := 1;\n scancounter int := 1;\n remaining_limit int := _scanlimit;\n tilearea float;\n unionedgeom geometry;\n clippedgeom geometry;\n unionedgeom_area float := 0;\n prev_area float := 0;\n excludes text[];\n includes text[];\n\nBEGIN\n -- If skipcovered is true then you will always want to exit when the passed in geometry is full\n IF skipcovered THEN\n exitwhenfull := TRUE;\n END IF;\n\n SELECT * INTO search FROM searches WHERE hash=queryhash;\n\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Search with Query Hash % Not Found', queryhash;\n END IF;\n\n tilearea := st_area(geom);\n _where := format('%s AND st_intersects(geometry, %L::geometry)', search._where, geom);\n\n IF fields IS NOT NULL THEN\n IF fields ? 'fields' THEN\n fields := fields->'fields';\n END IF;\n IF fields ? 'exclude' THEN\n excludes=textarr(fields->'exclude');\n END IF;\n IF fields ? 'include' THEN\n includes=textarr(fields->'include');\n IF array_length(includes, 1)>0 AND NOT 'id' = ANY (includes) THEN\n includes = includes || '{id}';\n END IF;\n END IF;\n END IF;\n RAISE NOTICE 'fields: %, includes: %, excludes: %', fields, includes, excludes;\n\n FOR query IN SELECT * FROM partition_queries(_where, search.orderby) LOOP\n query := format('%s LIMIT %L', query, remaining_limit);\n RAISE NOTICE '%', query;\n curs = create_cursor(query);\n LOOP\n FETCH curs INTO iter_record;\n EXIT WHEN NOT FOUND;\n IF exitwhenfull OR skipcovered THEN -- If we are not using exitwhenfull or skipcovered, we do not need to do expensive geometry operations\n clippedgeom := st_intersection(geom, iter_record.geometry);\n\n IF unionedgeom IS NULL THEN\n unionedgeom := clippedgeom;\n ELSE\n unionedgeom := st_union(unionedgeom, clippedgeom);\n END IF;\n\n unionedgeom_area := st_area(unionedgeom);\n\n IF skipcovered AND prev_area = unionedgeom_area THEN\n scancounter := scancounter + 1;\n CONTINUE;\n END IF;\n\n prev_area := unionedgeom_area;\n\n RAISE NOTICE '% % % %', unionedgeom_area/tilearea, counter, scancounter, ftime();\n END IF;\n\n IF fields IS NOT NULL THEN\n out_records := out_records || filter_jsonb(iter_record.content, includes, excludes);\n ELSE\n out_records := out_records || iter_record.content;\n END IF;\n IF counter >= _limit\n OR scancounter > _scanlimit\n OR ftime() > _timelimit\n OR (exitwhenfull AND unionedgeom_area >= tilearea)\n THEN\n exit_flag := TRUE;\n EXIT;\n END IF;\n counter := counter + 1;\n scancounter := scancounter + 1;\n\n END LOOP;\n EXIT WHEN exit_flag;\n remaining_limit := _scanlimit - scancounter;\n END LOOP;\n\n RETURN jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', array_to_json(out_records)::jsonb\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDROP FUNCTION IF EXISTS geojsonsearch;\nCREATE OR REPLACE FUNCTION geojsonsearch(\n IN geojson jsonb,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_geomfromgeojson(geojson),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS xyzsearch;\nCREATE OR REPLACE FUNCTION xyzsearch(\n IN _x int,\n IN _y int,\n IN _z int,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_transform(tileenvelope(_z, _x, _y), 4326),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\nSELECT set_version('0.4.2');\n" }, { "path": "src/pgstac/migrations/pgstac.0.4.3-0.4.4.sql", "content": "SET SEARCH_PATH to pgstac, public;\nset check_function_bodies = off;\n\nCREATE OR REPLACE FUNCTION pgstac.base_stac_query(j jsonb)\n RETURNS text\n LANGUAGE plpgsql\nAS $function$\nDECLARE\n_where text := '';\ndtrange tstzrange;\ngeom geometry;\nBEGIN\n\n IF j ? 'ids' THEN\n _where := format('%s AND id = ANY (%L) ', _where, textarr(j->'ids'));\n END IF;\n IF j ? 'collections' THEN\n _where := format('%s AND collection_id = ANY (%L) ', _where, textarr(j->'collections'));\n END IF;\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n _where := format('%s AND datetime <= %L::timestamptz AND end_datetime >= %L::timestamptz ',\n _where,\n upper(dtrange),\n lower(dtrange)\n );\n END IF;\n\n IF j ? 'bbox' THEN\n geom := bbox_geom(j->'bbox');\n _where := format('%s AND geometry && %L',\n _where,\n geom\n );\n END IF;\n\n IF j ? 'intersects' THEN\n geom := st_fromgeojson(j->>'intersects');\n _where := format('%s AND st_intersects(geometry, %L)',\n _where,\n geom\n );\n END IF;\n\n RETURN _where;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.item_by_id(_id text)\n RETURNS pgstac.items\n LANGUAGE plpgsql\nAS $function$\nDECLARE\n i items%ROWTYPE;\nBEGIN\n SELECT * INTO i FROM items WHERE id=_id LIMIT 1;\n RETURN i;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.cql_to_where(_search jsonb DEFAULT '{}'::jsonb)\n RETURNS text\n LANGUAGE plpgsql\nAS $function$\nDECLARE\nfilterlang text;\nsearch jsonb := _search;\nbase_where text;\n_where text;\nBEGIN\n\nRAISE NOTICE 'SEARCH CQL Final: %', search;\nfilterlang := COALESCE(\n search->>'filter-lang',\n get_setting('default-filter-lang', _search->'conf')\n);\n\nbase_where := base_stac_query(search);\n\nIF filterlang = 'cql-json' THEN\n search := query_to_cqlfilter(search);\n -- search := add_filters_to_cql(search);\n _where := cql_query_op(search->'filter');\nELSE\n _where := cql2_query(search->'filter');\nEND IF;\n\nIF trim(_where) = '' THEN\n _where := NULL;\nEND IF;\n_where := coalesce(_where, ' TRUE ');\nRETURN format('( %s ) %s ', _where, base_where);\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.get_token_filter(_search jsonb DEFAULT '{}'::jsonb, token_rec jsonb DEFAULT NULL::jsonb)\n RETURNS text\n LANGUAGE plpgsql\nAS $function$\nDECLARE\n token_id text;\n filters text[] := '{}'::text[];\n prev boolean := TRUE;\n field text;\n dir text;\n sort record;\n orfilters text[] := '{}'::text[];\n andfilters text[] := '{}'::text[];\n output text;\n token_where text;\nBEGIN\n -- If no token provided return NULL\n IF token_rec IS NULL THEN\n IF NOT (_search ? 'token' AND\n (\n (_search->>'token' ILIKE 'prev:%')\n OR\n (_search->>'token' ILIKE 'next:%')\n )\n ) THEN\n RETURN NULL;\n END IF;\n prev := (_search->>'token' ILIKE 'prev:%');\n token_id := substr(_search->>'token', 6);\n SELECT to_jsonb(items) INTO token_rec FROM item_by_id(token_id) as items;\n END IF;\n RAISE NOTICE 'TOKEN ID: %', token_rec->'id';\n\n CREATE TEMP TABLE sorts (\n _row int GENERATED ALWAYS AS IDENTITY NOT NULL,\n _field text PRIMARY KEY,\n _dir text NOT NULL,\n _val text\n ) ON COMMIT DROP;\n\n -- Make sure we only have distinct columns to sort with taking the first one we get\n INSERT INTO sorts (_field, _dir)\n SELECT\n (items_path(value->>'field')).path,\n get_sort_dir(value)\n FROM\n jsonb_array_elements(coalesce(_search->'sortby','[{\"field\":\"datetime\",\"direction\":\"desc\"}]'))\n ON CONFLICT DO NOTHING\n ;\n RAISE NOTICE 'sorts 1: %', (SELECT jsonb_agg(to_json(sorts)) FROM sorts);\n -- Get the first sort direction provided. As the id is a primary key, if there are any\n -- sorts after id they won't do anything, so make sure that id is the last sort item.\n SELECT _dir INTO dir FROM sorts ORDER BY _row ASC LIMIT 1;\n IF EXISTS (SELECT 1 FROM sorts WHERE _field = 'id') THEN\n DELETE FROM sorts WHERE _row > (SELECT _row FROM sorts WHERE _field = 'id' ORDER BY _row ASC);\n ELSE\n INSERT INTO sorts (_field, _dir) VALUES ('id', dir);\n END IF;\n\n -- Add value from looked up item to the sorts table\n UPDATE sorts SET _val=quote_literal(token_rec->>_field);\n\n -- Check if all sorts are the same direction and use row comparison\n -- to filter\n RAISE NOTICE 'sorts 2: %', (SELECT jsonb_agg(to_json(sorts)) FROM sorts);\n\n IF (SELECT count(DISTINCT _dir) FROM sorts) = 1 THEN\n SELECT format(\n '(%s) %s (%s)',\n concat_ws(', ', VARIADIC array_agg(quote_ident(_field))),\n CASE WHEN (prev AND dir = 'ASC') OR (NOT prev AND dir = 'DESC') THEN '<' ELSE '>' END,\n concat_ws(', ', VARIADIC array_agg(_val))\n ) INTO output FROM sorts\n WHERE token_rec ? _field\n ;\n ELSE\n FOR sort IN SELECT * FROM sorts ORDER BY _row asc LOOP\n RAISE NOTICE 'SORT: %', sort;\n IF sort._row = 1 THEN\n orfilters := orfilters || format('(%s %s %s)',\n quote_ident(sort._field),\n CASE WHEN (prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC') THEN '<' ELSE '>' END,\n sort._val\n );\n ELSE\n orfilters := orfilters || format('(%s AND %s %s %s)',\n array_to_string(andfilters, ' AND '),\n quote_ident(sort._field),\n CASE WHEN (prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC') THEN '<' ELSE '>' END,\n sort._val\n );\n\n END IF;\n andfilters := andfilters || format('%s = %s',\n quote_ident(sort._field),\n sort._val\n );\n END LOOP;\n output := array_to_string(orfilters, ' OR ');\n END IF;\n DROP TABLE IF EXISTS sorts;\n token_where := concat('(',coalesce(output,'true'),')');\n IF trim(token_where) = '' THEN\n token_where := NULL;\n END IF;\n RAISE NOTICE 'TOKEN_WHERE: |%|',token_where;\n RETURN token_where;\n END;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.search(_search jsonb DEFAULT '{}'::jsonb)\n RETURNS jsonb\n LANGUAGE plpgsql\n SET jit TO 'off'\nAS $function$\nDECLARE\n searches searches%ROWTYPE;\n _where text;\n token_where text;\n full_where text;\n orderby text;\n query text;\n token_type text := substr(_search->>'token',1,4);\n _limit int := coalesce((_search->>'limit')::int, 10);\n curs refcursor;\n cntr int := 0;\n iter_record items%ROWTYPE;\n first_record items%ROWTYPE;\n last_record items%ROWTYPE;\n out_records jsonb := '[]'::jsonb;\n prev_query text;\n next text;\n prev_id text;\n has_next boolean := false;\n has_prev boolean := false;\n prev text;\n total_count bigint;\n context jsonb;\n collection jsonb;\n includes text[];\n excludes text[];\n exit_flag boolean := FALSE;\n batches int := 0;\n timer timestamptz := clock_timestamp();\n pstart timestamptz;\n pend timestamptz;\n pcurs refcursor;\n search_where search_wheres%ROWTYPE;\n id text;\nBEGIN\nCREATE TEMP TABLE results (content jsonb) ON COMMIT DROP;\n-- if ids is set, short circuit and just use direct ids query for each id\n-- skip any paging or caching\n-- hard codes ordering in the same order as the array of ids\nIF _search ? 'ids' THEN\n FOR id IN SELECT jsonb_array_elements_text(_search->'ids') LOOP\n INSERT INTO results (content) SELECT content FROM item_by_id(id);\n END LOOP;\n SELECT INTO total_count count(*) FROM results;\nELSE\n searches := search_query(_search);\n _where := searches._where;\n orderby := searches.orderby;\n search_where := where_stats(_where);\n total_count := coalesce(search_where.total_count, search_where.estimated_count);\n\n\n\n IF token_type='prev' THEN\n token_where := get_token_filter(_search, null::jsonb);\n orderby := sort_sqlorderby(_search, TRUE);\n END IF;\n IF token_type='next' THEN\n token_where := get_token_filter(_search, null::jsonb);\n END IF;\n\n full_where := concat_ws(' AND ', _where, token_where);\n RAISE NOTICE 'FULL QUERY % %', full_where, clock_timestamp()-timer;\n timer := clock_timestamp();\n\n\n\n\n FOR query IN SELECT partition_queries(full_where, orderby, search_where.partitions) LOOP\n timer := clock_timestamp();\n query := format('%s LIMIT %s', query, _limit + 1);\n RAISE NOTICE 'Partition Query: %', query;\n batches := batches + 1;\n -- curs = create_cursor(query);\n OPEN curs FOR EXECUTE query;\n LOOP\n FETCH curs into iter_record;\n EXIT WHEN NOT FOUND;\n cntr := cntr + 1;\n last_record := iter_record;\n IF cntr = 1 THEN\n first_record := last_record;\n END IF;\n IF cntr <= _limit THEN\n INSERT INTO results (content) VALUES (last_record.content);\n -- out_records := out_records || last_record.content;\n\n ELSIF cntr > _limit THEN\n has_next := true;\n exit_flag := true;\n EXIT;\n END IF;\n END LOOP;\n CLOSE curs;\n RAISE NOTICE 'Query took %. Total Time %', clock_timestamp()-timer, ftime();\n timer := clock_timestamp();\n EXIT WHEN exit_flag;\n END LOOP;\n RAISE NOTICE 'Scanned through % partitions.', batches;\nEND IF;\n\nSELECT jsonb_agg(content) INTO out_records FROM results;\n\nDROP TABLE results;\n\n\n-- Flip things around if this was the result of a prev token query\nIF token_type='prev' THEN\n out_records := flip_jsonb_array(out_records);\n first_record := last_record;\nEND IF;\n\n-- If this query has a token, see if there is data before the first record\nIF _search ? 'token' THEN\n prev_query := format(\n 'SELECT 1 FROM items WHERE %s LIMIT 1',\n concat_ws(\n ' AND ',\n _where,\n trim(get_token_filter(_search, to_jsonb(first_record)))\n )\n );\n RAISE NOTICE 'Query to get previous record: % --- %', prev_query, first_record;\n EXECUTE prev_query INTO has_prev;\n IF FOUND and has_prev IS NOT NULL THEN\n RAISE NOTICE 'Query results from prev query: %', has_prev;\n has_prev := TRUE;\n END IF;\nEND IF;\nhas_prev := COALESCE(has_prev, FALSE);\n\nRAISE NOTICE 'token_type: %, has_next: %, has_prev: %', token_type, has_next, has_prev;\nIF has_prev THEN\n prev := out_records->0->>'id';\nEND IF;\nIF has_next OR token_type='prev' THEN\n next := out_records->-1->>'id';\nEND IF;\n\n\n-- include/exclude any fields following fields extension\nIF _search ? 'fields' THEN\n IF _search->'fields' ? 'exclude' THEN\n excludes=textarr(_search->'fields'->'exclude');\n END IF;\n IF _search->'fields' ? 'include' THEN\n includes=textarr(_search->'fields'->'include');\n IF array_length(includes, 1)>0 AND NOT 'id' = ANY (includes) THEN\n includes = includes || '{id}';\n END IF;\n END IF;\n SELECT jsonb_agg(filter_jsonb(row, includes, excludes)) INTO out_records FROM jsonb_array_elements(out_records) row;\nEND IF;\n\nIF context(_search->'conf') != 'off' THEN\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'matched', total_count,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\nELSE\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\nEND IF;\n\ncollection := jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb),\n 'next', next,\n 'prev', prev,\n 'context', context\n);\n\nRETURN collection;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.where_stats(inwhere text, updatestats boolean DEFAULT false, conf jsonb DEFAULT NULL::jsonb)\n RETURNS pgstac.search_wheres\n LANGUAGE plpgsql\nAS $function$\nDECLARE\n t timestamptz;\n i interval;\n explain_json jsonb;\n partitions text[];\n sw search_wheres%ROWTYPE;\n inwhere_hash text := md5(inwhere);\nBEGIN\n SELECT * INTO sw FROM search_wheres WHERE _where=inwhere_hash FOR UPDATE;\n\n -- Update statistics if explicitly set, if statistics do not exist, or statistics ttl has expired\n IF NOT updatestats THEN\n RAISE NOTICE 'Checking if update is needed.';\n RAISE NOTICE 'Stats Last Updated: %', sw.statslastupdated;\n RAISE NOTICE 'TTL: %, Age: %', context_stats_ttl(conf), now() - sw.statslastupdated;\n RAISE NOTICE 'Context: %, Existing Total: %', context(conf), sw.total_count;\n IF\n sw.statslastupdated IS NULL\n OR (now() - sw.statslastupdated) > context_stats_ttl(conf)\n OR (context(conf) != 'off' AND sw.total_count IS NULL)\n THEN\n updatestats := TRUE;\n END IF;\n END IF;\n\n sw._where := inwhere;\n sw.lastused := now();\n sw.usecount := coalesce(sw.usecount,0) + 1;\n\n IF NOT updatestats THEN\n UPDATE search_wheres SET\n lastused = sw.lastused,\n usecount = sw.usecount\n WHERE _where = inwhere_hash\n RETURNING * INTO sw\n ;\n RETURN sw;\n END IF;\n -- Use explain to get estimated count/cost and a list of the partitions that would be hit by the query\n t := clock_timestamp();\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s', inwhere)\n INTO explain_json;\n RAISE NOTICE 'Time for just the explain: %', clock_timestamp() - t;\n WITH t AS (\n SELECT j->>0 as p FROM\n jsonb_path_query(\n explain_json,\n 'strict $.**.\"Relation Name\" ? (@ != null)'\n ) j\n ), ordered AS (\n SELECT p FROM t ORDER BY p DESC\n -- SELECT p FROM t JOIN items_partitions\n -- ON (t.p = items_partitions.partition)\n -- ORDER BY pstart DESC\n )\n SELECT array_agg(p) INTO partitions FROM ordered;\n i := clock_timestamp() - t;\n RAISE NOTICE 'Time for explain + join: %', clock_timestamp() - t;\n\n\n\n sw.statslastupdated := now();\n sw.estimated_count := explain_json->0->'Plan'->'Plan Rows';\n sw.estimated_cost := explain_json->0->'Plan'->'Total Cost';\n sw.time_to_estimate := extract(epoch from i);\n sw.partitions := partitions;\n\n -- Do a full count of rows if context is set to on or if auto is set and estimates are low enough\n IF\n context(conf) = 'on'\n OR\n ( context(conf) = 'auto' AND\n (\n sw.estimated_count < context_estimated_count(conf)\n OR\n sw.estimated_cost < context_estimated_cost(conf)\n )\n )\n THEN\n t := clock_timestamp();\n RAISE NOTICE 'Calculating actual count...';\n EXECUTE format(\n 'SELECT count(*) FROM items WHERE %s',\n inwhere\n ) INTO sw.total_count;\n i := clock_timestamp() - t;\n RAISE NOTICE 'Actual Count: % -- %', sw.total_count, i;\n sw.time_to_count := extract(epoch FROM i);\n ELSE\n sw.total_count := NULL;\n sw.time_to_count := NULL;\n END IF;\n\n\n INSERT INTO search_wheres\n (_where, lastused, usecount, statslastupdated, estimated_count, estimated_cost, time_to_estimate, partitions, total_count, time_to_count)\n SELECT sw._where, sw.lastused, sw.usecount, sw.statslastupdated, sw.estimated_count, sw.estimated_cost, sw.time_to_estimate, sw.partitions, sw.total_count, sw.time_to_count\n ON CONFLICT ((md5(_where)))\n DO UPDATE\n SET\n lastused = sw.lastused,\n usecount = sw.usecount,\n statslastupdated = sw.statslastupdated,\n estimated_count = sw.estimated_count,\n estimated_cost = sw.estimated_cost,\n time_to_estimate = sw.time_to_estimate,\n partitions = sw.partitions,\n total_count = sw.total_count,\n time_to_count = sw.time_to_count\n ;\n RETURN sw;\nEND;\n$function$\n;\n\n\n\nSELECT set_version('0.4.4');\n" }, { "path": "src/pgstac/migrations/pgstac.0.4.3.sql", "content": "CREATE EXTENSION IF NOT EXISTS postgis;\nCREATE SCHEMA IF NOT EXISTS pgstac;\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE TABLE migrations (\n version text PRIMARY KEY,\n datetime timestamptz DEFAULT clock_timestamp() NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION get_version() RETURNS text AS $$\n SELECT version FROM migrations ORDER BY datetime DESC, version DESC LIMIT 1;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION set_version(text) RETURNS text AS $$\n INSERT INTO migrations (version) VALUES ($1)\n ON CONFLICT DO NOTHING\n RETURNING version;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\n\nCREATE TABLE IF NOT EXISTS pgstac_settings (\n name text PRIMARY KEY,\n value text NOT NULL\n);\n\nINSERT INTO pgstac_settings (name, value) VALUES\n ('context', 'off'),\n ('context_estimated_count', '100000'),\n ('context_estimated_cost', '1000000'),\n ('context_stats_ttl', '1 day'),\n ('default-filter-lang', 'cql2-json')\nON CONFLICT DO NOTHING\n;\n\n\nCREATE OR REPLACE FUNCTION get_setting(IN _setting text, IN conf jsonb DEFAULT NULL) RETURNS text AS $$\nSELECT COALESCE(\n conf->>_setting,\n current_setting(concat('pgstac.',_setting), TRUE),\n (SELECT value FROM pgstac_settings WHERE name=_setting)\n);\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context();\nCREATE OR REPLACE FUNCTION context(conf jsonb DEFAULT NULL) RETURNS text AS $$\n SELECT get_setting('context', conf);\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_estimated_count();\nCREATE OR REPLACE FUNCTION context_estimated_count(conf jsonb DEFAULT NULL) RETURNS int AS $$\n SELECT get_setting('context_estimated_count', conf)::int;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_estimated_cost();\nCREATE OR REPLACE FUNCTION context_estimated_cost(conf jsonb DEFAULT NULL) RETURNS float AS $$\n SELECT get_setting('context_estimated_cost', conf)::float;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_stats_ttl();\nCREATE OR REPLACE FUNCTION context_stats_ttl(conf jsonb DEFAULT NULL) RETURNS interval AS $$\n SELECT get_setting('context_stats_ttl', conf)::interval;\n$$ LANGUAGE SQL;\n\n\nCREATE OR REPLACE FUNCTION notice(VARIADIC text[]) RETURNS boolean AS $$\nDECLARE\ndebug boolean := current_setting('pgstac.debug', true);\nBEGIN\n IF debug THEN\n RAISE NOTICE 'NOTICE FROM FUNC: % >>>>> %', concat_ws(' | ', $1), clock_timestamp();\n RETURN TRUE;\n END IF;\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION empty_arr(ANYARRAY) RETURNS BOOLEAN AS $$\nSELECT CASE\n WHEN $1 IS NULL THEN TRUE\n WHEN cardinality($1)<1 THEN TRUE\nELSE FALSE\nEND;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_map_ident(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_ident(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_map_literal(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_literal(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION estimated_count(_where text) RETURNS bigint AS $$\nDECLARE\nrec record;\nrows bigint;\nBEGIN\n FOR rec in EXECUTE format(\n $q$\n EXPLAIN SELECT 1 FROM items WHERE %s\n $q$,\n _where)\n LOOP\n rows := substring(rec.\"QUERY PLAN\" FROM ' rows=([[:digit:]]+)');\n EXIT WHEN rows IS NOT NULL;\n END LOOP;\n\n RETURN rows;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION array_reverse(anyarray) RETURNS anyarray AS $$\nSELECT ARRAY(\n SELECT $1[i]\n FROM generate_subscripts($1,1) AS s(i)\n ORDER BY i DESC\n);\n$$ LANGUAGE 'sql' STRICT IMMUTABLE;\n/* converts a jsonb text array to a pg text[] array */\nCREATE OR REPLACE FUNCTION textarr(_js jsonb)\n RETURNS text[] AS $$\n SELECT\n CASE jsonb_typeof(_js)\n WHEN 'array' THEN ARRAY(SELECT jsonb_array_elements_text(_js))\n ELSE ARRAY[_js->>0]\n END\n;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION jsonb_paths (IN jdata jsonb, OUT path text[], OUT value jsonb) RETURNS\nSETOF RECORD AS $$\nwith recursive extract_all as\n(\n select\n ARRAY[key]::text[] as path,\n value\n FROM jsonb_each(jdata)\nunion all\n select\n path || coalesce(obj_key, (arr_key- 1)::text),\n coalesce(obj_value, arr_value)\n from extract_all\n left join lateral\n jsonb_each(case jsonb_typeof(value) when 'object' then value end)\n as o(obj_key, obj_value)\n on jsonb_typeof(value) = 'object'\n left join lateral\n jsonb_array_elements(case jsonb_typeof(value) when 'array' then value end)\n with ordinality as a(arr_value, arr_key)\n on jsonb_typeof(value) = 'array'\n where obj_key is not null or arr_key is not null\n)\nselect *\nfrom extract_all;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION jsonb_obj_paths (IN jdata jsonb, OUT path text[], OUT value jsonb) RETURNS\nSETOF RECORD AS $$\nwith recursive extract_all as\n(\n select\n ARRAY[key]::text[] as path,\n value\n FROM jsonb_each(jdata)\nunion all\n select\n path || obj_key,\n obj_value\n from extract_all\n left join lateral\n jsonb_each(case jsonb_typeof(value) when 'object' then value end)\n as o(obj_key, obj_value)\n on jsonb_typeof(value) = 'object'\n where obj_key is not null\n)\nselect *\nfrom extract_all;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION jsonb_val_paths (IN jdata jsonb, OUT path text[], OUT value jsonb) RETURNS\nSETOF RECORD AS $$\nSELECT * FROM jsonb_obj_paths(jdata) WHERE jsonb_typeof(value) not in ('object','array');\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION path_includes(IN path text[], IN includes text[]) RETURNS BOOLEAN AS $$\nWITH t AS (SELECT unnest(includes) i)\nSELECT EXISTS (\n SELECT 1 FROM t WHERE path @> string_to_array(trim(i), '.')\n);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION path_excludes(IN path text[], IN excludes text[]) RETURNS BOOLEAN AS $$\nWITH t AS (SELECT unnest(excludes) e)\nSELECT NOT EXISTS (\n SELECT 1 FROM t WHERE path @> string_to_array(trim(e), '.')\n);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION jsonb_obj_paths_filtered (\n IN jdata jsonb,\n IN includes text[] DEFAULT ARRAY[]::text[],\n IN excludes text[] DEFAULT ARRAY[]::text[],\n OUT path text[],\n OUT value jsonb\n) RETURNS\nSETOF RECORD AS $$\nSELECT path, value\nFROM jsonb_obj_paths(jdata)\nWHERE\n CASE WHEN cardinality(includes) > 0 THEN path_includes(path, includes) ELSE TRUE END\n AND\n path_excludes(path, excludes)\n\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION filter_jsonb(\n IN jdata jsonb,\n IN includes text[] DEFAULT ARRAY[]::text[],\n IN excludes text[] DEFAULT ARRAY[]::text[]\n) RETURNS jsonb AS $$\nDECLARE\nrec RECORD;\noutj jsonb := '{}'::jsonb;\ncreated_paths text[] := '{}'::text[];\nBEGIN\n\nIF empty_arr(includes) AND empty_arr(excludes) THEN\nRAISE NOTICE 'no filter';\n RETURN jdata;\nEND IF;\nFOR rec in\nSELECT * FROM jsonb_obj_paths_filtered(jdata, includes, excludes)\nWHERE jsonb_typeof(value) != 'object'\nLOOP\n IF array_length(rec.path,1)>1 THEN\n FOR i IN 1..(array_length(rec.path,1)-1) LOOP\n IF NOT array_to_string(rec.path[1:i],'.') = ANY (created_paths) THEN\n outj := jsonb_set(outj, rec.path[1:i],'{}', true);\n created_paths := created_paths || array_to_string(rec.path[1:i],'.');\n END IF;\n END LOOP;\n END IF;\n outj := jsonb_set(outj, rec.path, rec.value, true);\n created_paths := created_paths || array_to_string(rec.path,'.');\nEND LOOP;\nRETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION flip_jsonb_array(j jsonb) RETURNS jsonb AS $$\nSELECT jsonb_agg(value) FROM (SELECT value FROM jsonb_array_elements(j) WITH ORDINALITY ORDER BY ordinality DESC) as t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n/* Functions to create an iterable of cursors over partitions. */\nCREATE OR REPLACE FUNCTION create_cursor(q text) RETURNS refcursor AS $$\nDECLARE\n curs refcursor;\nBEGIN\n OPEN curs FOR EXECUTE q;\n RETURN curs;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDROP FUNCTION IF EXISTS partition_queries;\nCREATE OR REPLACE FUNCTION partition_queries(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN partitions text[] DEFAULT '{items}'\n) RETURNS SETOF text AS $$\nDECLARE\n partition_query text;\n query text;\n p text;\n cursors refcursor;\n dstart timestamptz;\n dend timestamptz;\n step interval := '10 weeks'::interval;\nBEGIN\n\nIF _orderby ILIKE 'datetime d%' THEN\n partitions := partitions;\nELSIF _orderby ILIKE 'datetime a%' THEN\n partitions := array_reverse(partitions);\nELSE\n query := format($q$\n SELECT * FROM items\n WHERE %s\n ORDER BY %s\n $q$, _where, _orderby\n );\n\n RETURN NEXT query;\n RETURN;\nEND IF;\nRAISE NOTICE 'PARTITIONS ---> %',partitions;\nIF cardinality(partitions) > 0 THEN\n FOREACH p IN ARRAY partitions\n --EXECUTE partition_query\n LOOP\n query := format($q$\n SELECT * FROM %I\n WHERE %s\n ORDER BY %s\n $q$,\n p,\n _where,\n _orderby\n );\n RETURN NEXT query;\n END LOOP;\nEND IF;\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION partition_cursor(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC'\n) RETURNS SETOF refcursor AS $$\nDECLARE\n partition_query text;\n query text;\n p record;\n cursors refcursor;\nBEGIN\nFOR query IN SELECT * FROM partition_queries(_where, _orderby) LOOP\n RETURN NEXT create_cursor(query);\nEND LOOP;\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION partition_count(\n IN _where text DEFAULT 'TRUE'\n) RETURNS bigint AS $$\nDECLARE\n partition_query text;\n query text;\n p record;\n subtotal bigint;\n total bigint := 0;\nBEGIN\npartition_query := format($q$\n SELECT partition, tstzrange\n FROM items_partitions\n ORDER BY tstzrange DESC;\n$q$);\nRAISE NOTICE 'Partition Query: %', partition_query;\nFOR p IN\n EXECUTE partition_query\nLOOP\n query := format($q$\n SELECT count(*) FROM items\n WHERE datetime BETWEEN %L AND %L AND %s\n $q$, lower(p.tstzrange), upper(p.tstzrange), _where\n );\n RAISE NOTICE 'Query %', query;\n RAISE NOTICE 'Partition %, Count %, Total %',p.partition, subtotal, total;\n EXECUTE query INTO subtotal;\n total := subtotal + total;\nEND LOOP;\nRETURN total;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION drop_partition_constraints(IN partition text) RETURNS VOID AS $$\nDECLARE\n q text;\n end_datetime_constraint text := concat(partition, '_end_datetime_constraint');\n collections_constraint text := concat(partition, '_collections_constraint');\nBEGIN\n q := format($q$\n ALTER TABLE %I\n DROP CONSTRAINT IF EXISTS %I,\n DROP CONSTRAINT IF EXISTS %I;\n $q$,\n partition,\n end_datetime_constraint,\n collections_constraint\n );\n\n EXECUTE q;\n RETURN;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDROP FUNCTION IF EXISTS partition_checks;\nCREATE OR REPLACE FUNCTION partition_checks(\n IN partition text,\n OUT min_datetime timestamptz,\n OUT max_datetime timestamptz,\n OUT min_end_datetime timestamptz,\n OUT max_end_datetime timestamptz,\n OUT collections text[],\n OUT cnt bigint\n) RETURNS RECORD AS $$\nDECLARE\nq text;\nend_datetime_constraint text := concat(partition, '_end_datetime_constraint');\ncollections_constraint text := concat(partition, '_collections_constraint');\nBEGIN\nRAISE NOTICE 'CREATING CONSTRAINTS FOR %', partition;\nq := format($q$\n SELECT\n min(datetime),\n max(datetime),\n min(end_datetime),\n max(end_datetime),\n array_agg(DISTINCT collection_id),\n count(*)\n FROM %I;\n $q$,\n partition\n);\nEXECUTE q INTO min_datetime, max_datetime, min_end_datetime, max_end_datetime, collections, cnt;\nRAISE NOTICE '% % % % % % %', min_datetime, max_datetime, min_end_datetime, max_end_datetime, collections, cnt, ftime();\nIF cnt IS NULL or cnt = 0 THEN\n RAISE NOTICE 'Partition % is empty, removing...', partition;\n q := format($q$\n DROP TABLE IF EXISTS %I;\n $q$, partition\n );\n EXECUTE q;\n RETURN;\nEND IF;\nRAISE NOTICE 'Running Constraint DDL %', ftime();\nq := format($q$\n ALTER TABLE %I\n DROP CONSTRAINT IF EXISTS %I,\n ADD CONSTRAINT %I\n check((end_datetime >= %L) AND (end_datetime <= %L)) NOT VALID,\n DROP CONSTRAINT IF EXISTS %I,\n ADD CONSTRAINT %I\n check((collection_id = ANY(%L))) NOT VALID;\n $q$,\n partition,\n end_datetime_constraint,\n end_datetime_constraint,\n min_end_datetime,\n max_end_datetime,\n collections_constraint,\n collections_constraint,\n collections,\n partition\n);\nRAISE NOTICE 'q: %', q;\n\nEXECUTE q;\nRAISE NOTICE 'Returning %', ftime();\nRETURN;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION validate_constraints() RETURNS VOID AS $$\nDECLARE\nq text;\nBEGIN\nFOR q IN\n SELECT FORMAT(\n 'ALTER TABLE %I.%I.%I VALIDATE CONSTRAINT %I;',\n current_database(),\n nsp.nspname,\n cls.relname,\n con.conname\n )\n FROM pg_constraint AS con\n JOIN pg_class AS cls\n ON con.conrelid = cls.oid\n JOIN pg_namespace AS nsp\n ON cls.relnamespace = nsp.oid\n WHERE convalidated IS FALSE\n AND nsp.nspname = 'pgstac'\nLOOP\n EXECUTE q;\nEND LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n/* looks for a geometry in a stac item first from geometry and falling back to bbox */\nCREATE OR REPLACE FUNCTION stac_geom(value jsonb) RETURNS geometry AS $$\nSELECT\n CASE\n WHEN value->>'geometry' IS NOT NULL THEN\n ST_GeomFromGeoJSON(value->>'geometry')\n WHEN value->>'bbox' IS NOT NULL THEN\n ST_MakeEnvelope(\n (value->'bbox'->>0)::float,\n (value->'bbox'->>1)::float,\n (value->'bbox'->>2)::float,\n (value->'bbox'->>3)::float,\n 4326\n )\n ELSE NULL\n END as geometry\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION stac_datetime(value jsonb) RETURNS timestamptz AS $$\nSELECT COALESCE(\n (value->'properties'->>'datetime')::timestamptz,\n (value->'properties'->>'start_datetime')::timestamptz\n);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_end_datetime(value jsonb) RETURNS timestamptz AS $$\nSELECT COALESCE(\n (value->'properties'->>'datetime')::timestamptz,\n (value->'properties'->>'end_datetime')::timestamptz\n);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\n\nCREATE OR REPLACE FUNCTION stac_daterange(value jsonb) RETURNS tstzrange AS $$\nSELECT tstzrange(stac_datetime(value),stac_end_datetime(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE TABLE IF NOT EXISTS collections (\n id VARCHAR GENERATED ALWAYS AS (content->>'id') STORED PRIMARY KEY,\n content JSONB\n);\n\nCREATE OR REPLACE FUNCTION create_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection(data jsonb) RETURNS VOID AS $$\nDECLARE\nout collections%ROWTYPE;\nBEGIN\n UPDATE collections SET content=data WHERE id = data->>'id' RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ON CONFLICT (id) DO\n UPDATE\n SET content=EXCLUDED.content\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION delete_collection(_id text) RETURNS VOID AS $$\nDECLARE\nout collections%ROWTYPE;\nBEGIN\n DELETE FROM collections WHERE id = _id RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION get_collection(id text) RETURNS jsonb AS $$\nSELECT content FROM collections\nWHERE id=$1\n;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION all_collections() RETURNS jsonb AS $$\nSELECT jsonb_agg(content) FROM collections;\n;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE TABLE items (\n id text NOT NULL,\n geometry geometry NOT NULL,\n collection_id text NOT NULL,\n datetime timestamptz NOT NULL,\n end_datetime timestamptz NOT NULL,\n properties jsonb NOT NULL,\n content JSONB NOT NULL\n)\nPARTITION BY RANGE (datetime)\n;\n\nCREATE OR REPLACE FUNCTION properties_idx (IN content jsonb) RETURNS jsonb AS $$\n with recursive extract_all as\n (\n select\n ARRAY[key]::text[] as path,\n ARRAY[key]::text[] as fullpath,\n value\n FROM jsonb_each(content->'properties')\n union all\n select\n CASE WHEN obj_key IS NOT NULL THEN path || obj_key ELSE path END,\n path || coalesce(obj_key, (arr_key- 1)::text),\n coalesce(obj_value, arr_value)\n from extract_all\n left join lateral\n jsonb_each(case jsonb_typeof(value) when 'object' then value end)\n as o(obj_key, obj_value)\n on jsonb_typeof(value) = 'object'\n left join lateral\n jsonb_array_elements(case jsonb_typeof(value) when 'array' then value end)\n with ordinality as a(arr_value, arr_key)\n on jsonb_typeof(value) = 'array'\n where obj_key is not null or arr_key is not null\n )\n , paths AS (\n select\n array_to_string(path, '.') as path,\n value\n FROM extract_all\n WHERE\n jsonb_typeof(value) NOT IN ('array','object')\n ), grouped AS (\n SELECT path, jsonb_agg(distinct value) vals FROM paths group by path\n ) SELECT coalesce(jsonb_object_agg(path, CASE WHEN jsonb_array_length(vals)=1 THEN vals->0 ELSE vals END) - '{datetime}'::text[], '{}'::jsonb) FROM grouped\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET JIT TO OFF;\n\nCREATE INDEX \"datetime_idx\" ON items (datetime);\nCREATE INDEX \"end_datetime_idx\" ON items (end_datetime);\nCREATE INDEX \"properties_idx\" ON items USING GIN (properties jsonb_path_ops);\nCREATE INDEX \"collection_idx\" ON items (collection_id);\nCREATE INDEX \"geometry_idx\" ON items USING GIST (geometry);\nCREATE UNIQUE INDEX \"items_id_datetime_idx\" ON items (datetime, id);\n\nALTER TABLE items ADD CONSTRAINT items_collections_fk FOREIGN KEY (collection_id) REFERENCES collections(id) ON DELETE CASCADE DEFERRABLE;\n\nCREATE OR REPLACE FUNCTION analyze_empty_partitions() RETURNS VOID AS $$\nDECLARE\n p text;\nBEGIN\n FOR p IN SELECT partition FROM all_items_partitions WHERE est_cnt = 0 LOOP\n EXECUTE format('ANALYZE %I;', p);\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION items_partition_name(timestamptz) RETURNS text AS $$\n SELECT to_char($1, '\"items_p\"IYYY\"w\"IW');\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION items_partition_exists(text) RETURNS boolean AS $$\n SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_class WHERE relname=$1);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION items_partition_exists(timestamptz) RETURNS boolean AS $$\n SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_class WHERE relname=items_partition_name($1));\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION items_partition_create_worker(partition text, partition_start timestamptz, partition_end timestamptz) RETURNS VOID AS $$\nDECLARE\n err_context text;\nBEGIN\n EXECUTE format(\n $f$\n CREATE TABLE IF NOT EXISTS %1$I PARTITION OF items\n FOR VALUES FROM (%2$L) TO (%3$L);\n CREATE UNIQUE INDEX IF NOT EXISTS %4$I ON %1$I (id);\n $f$,\n partition,\n partition_start,\n partition_end,\n concat(partition, '_id_pk')\n );\nEXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', partition;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH to pgstac, public;\n\nCREATE OR REPLACE FUNCTION items_partition_create(ts timestamptz) RETURNS text AS $$\nDECLARE\n partition text := items_partition_name(ts);\n partition_start timestamptz;\n partition_end timestamptz;\nBEGIN\n IF items_partition_exists(partition) THEN\n RETURN partition;\n END IF;\n partition_start := date_trunc('week', ts);\n partition_end := partition_start + '1 week'::interval;\n PERFORM items_partition_create_worker(partition, partition_start, partition_end);\n RAISE NOTICE 'partition: %', partition;\n RETURN partition;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION items_partition_create(st timestamptz, et timestamptz) RETURNS SETOF text AS $$\nWITH t AS (\n SELECT\n generate_series(\n date_trunc('week',st),\n date_trunc('week', et),\n '1 week'::interval\n ) w\n)\nSELECT items_partition_create(w) FROM t;\n$$ LANGUAGE SQL;\n\n\nCREATE UNLOGGED TABLE items_staging (\n content JSONB NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION items_staging_insert_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p record;\n _partitions text[];\nBEGIN\n CREATE TEMP TABLE new_partitions ON COMMIT DROP AS\n SELECT\n items_partition_name(stac_datetime(content)) as partition,\n date_trunc('week', min(stac_datetime(content))) as partition_start\n FROM newdata\n GROUP BY 1;\n\n -- set statslastupdated in cache to be old enough cache always regenerated\n\n SELECT array_agg(partition) INTO _partitions FROM new_partitions;\n UPDATE search_wheres\n SET\n statslastupdated = NULL\n WHERE _where IN (\n SELECT _where FROM search_wheres sw WHERE sw.partitions && _partitions\n FOR UPDATE SKIP LOCKED\n )\n ;\n FOR p IN SELECT new_partitions.partition, new_partitions.partition_start, new_partitions.partition_start + '1 week'::interval as partition_end FROM new_partitions\n LOOP\n RAISE NOTICE 'Getting partition % ready.', p.partition;\n IF NOT items_partition_exists(p.partition) THEN\n RAISE NOTICE 'Creating partition %.', p.partition;\n PERFORM items_partition_create_worker(p.partition, p.partition_start, p.partition_end);\n END IF;\n PERFORM drop_partition_constraints(p.partition);\n END LOOP;\n INSERT INTO items (id, geometry, collection_id, datetime, end_datetime, properties, content)\n SELECT\n content->>'id',\n stac_geom(content),\n content->>'collection',\n stac_datetime(content),\n stac_end_datetime(content),\n properties_idx(content),\n content\n FROM newdata\n ;\n DELETE FROM items_staging;\n\n\n FOR p IN SELECT new_partitions.partition FROM new_partitions\n LOOP\n RAISE NOTICE 'Setting constraints for partition %.', p.partition;\n PERFORM partition_checks(p.partition);\n END LOOP;\n DROP TABLE IF EXISTS new_partitions;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER items_staging_insert_trigger AFTER INSERT ON items_staging REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_insert_triggerfunc();\n\n\nCREATE UNLOGGED TABLE items_staging_ignore (\n content JSONB NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION items_staging_ignore_insert_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p record;\n _partitions text[];\nBEGIN\n CREATE TEMP TABLE new_partitions ON COMMIT DROP AS\n SELECT\n items_partition_name(stac_datetime(content)) as partition,\n date_trunc('week', min(stac_datetime(content))) as partition_start\n FROM newdata\n GROUP BY 1;\n\n -- set statslastupdated in cache to be old enough cache always regenerated\n\n SELECT array_agg(partition) INTO _partitions FROM new_partitions;\n UPDATE search_wheres\n SET\n statslastupdated = NULL\n WHERE _where IN (\n SELECT _where FROM search_wheres sw WHERE sw.partitions && _partitions\n FOR UPDATE SKIP LOCKED\n )\n ;\n\n FOR p IN SELECT new_partitions.partition, new_partitions.partition_start, new_partitions.partition_start + '1 week'::interval as partition_end FROM new_partitions\n LOOP\n RAISE NOTICE 'Getting partition % ready.', p.partition;\n IF NOT items_partition_exists(p.partition) THEN\n RAISE NOTICE 'Creating partition %.', p.partition;\n PERFORM items_partition_create_worker(p.partition, p.partition_start, p.partition_end);\n END IF;\n PERFORM drop_partition_constraints(p.partition);\n END LOOP;\n\n INSERT INTO items (id, geometry, collection_id, datetime, end_datetime, properties, content)\n SELECT\n content->>'id',\n stac_geom(content),\n content->>'collection',\n stac_datetime(content),\n stac_end_datetime(content),\n properties_idx(content),\n content\n FROM newdata\n ON CONFLICT (datetime, id) DO NOTHING\n ;\n DELETE FROM items_staging;\n\n\n FOR p IN SELECT new_partitions.partition FROM new_partitions\n LOOP\n RAISE NOTICE 'Setting constraints for partition %.', p.partition;\n PERFORM partition_checks(p.partition);\n END LOOP;\n DROP TABLE IF EXISTS new_partitions;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER items_staging_ignore_insert_trigger AFTER INSERT ON items_staging_ignore REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_ignore_insert_triggerfunc();\n\nCREATE UNLOGGED TABLE items_staging_upsert (\n content JSONB NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION items_staging_upsert_insert_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p record;\n _partitions text[];\nBEGIN\n CREATE TEMP TABLE new_partitions ON COMMIT DROP AS\n SELECT\n items_partition_name(stac_datetime(content)) as partition,\n date_trunc('week', min(stac_datetime(content))) as partition_start\n FROM newdata\n GROUP BY 1;\n\n -- set statslastupdated in cache to be old enough cache always regenerated\n\n SELECT array_agg(partition) INTO _partitions FROM new_partitions;\n UPDATE search_wheres\n SET\n statslastupdated = NULL\n WHERE _where IN (\n SELECT _where FROM search_wheres sw WHERE sw.partitions && _partitions\n FOR UPDATE SKIP LOCKED\n )\n ;\n\n FOR p IN SELECT new_partitions.partition, new_partitions.partition_start, new_partitions.partition_start + '1 week'::interval as partition_end FROM new_partitions\n LOOP\n RAISE NOTICE 'Getting partition % ready.', p.partition;\n IF NOT items_partition_exists(p.partition) THEN\n RAISE NOTICE 'Creating partition %.', p.partition;\n PERFORM items_partition_create_worker(p.partition, p.partition_start, p.partition_end);\n END IF;\n PERFORM drop_partition_constraints(p.partition);\n END LOOP;\n\n INSERT INTO items (id, geometry, collection_id, datetime, end_datetime, properties, content)\n SELECT\n content->>'id',\n stac_geom(content),\n content->>'collection',\n stac_datetime(content),\n stac_end_datetime(content),\n properties_idx(content),\n content\n FROM newdata\n ON CONFLICT (datetime, id) DO UPDATE SET\n content = EXCLUDED.content\n WHERE items.content IS DISTINCT FROM EXCLUDED.content\n ;\n DELETE FROM items_staging;\n\n\n FOR p IN SELECT new_partitions.partition FROM new_partitions\n LOOP\n RAISE NOTICE 'Setting constraints for partition %.', p.partition;\n PERFORM partition_checks(p.partition);\n END LOOP;\n DROP TABLE IF EXISTS new_partitions;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER items_staging_upsert_insert_trigger AFTER INSERT ON items_staging_upsert REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_upsert_insert_triggerfunc();\n\nCREATE OR REPLACE FUNCTION items_update_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\nBEGIN\n NEW.id := NEW.content->>'id';\n NEW.datetime := stac_datetime(NEW.content);\n NEW.end_datetime := stac_end_datetime(NEW.content);\n NEW.collection_id := NEW.content->>'collection';\n NEW.geometry := stac_geom(NEW.content);\n NEW.properties := properties_idx(NEW.content);\n IF TG_OP = 'UPDATE' AND NEW IS NOT DISTINCT FROM OLD THEN\n RETURN NULL;\n END IF;\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER items_update_trigger BEFORE UPDATE ON items\n FOR EACH ROW EXECUTE PROCEDURE items_update_triggerfunc();\n\n/*\nView to get a table of available items partitions\nwith date ranges\n*/\nDROP VIEW IF EXISTS all_items_partitions CASCADE;\nCREATE VIEW all_items_partitions AS\nWITH base AS\n(SELECT\n c.oid::pg_catalog.regclass::text as partition,\n pg_catalog.pg_get_expr(c.relpartbound, c.oid) as _constraint,\n regexp_matches(\n pg_catalog.pg_get_expr(c.relpartbound, c.oid),\n E'\\\\(''\\([0-9 :+-]*\\)''\\\\).*\\\\(''\\([0-9 :+-]*\\)''\\\\)'\n ) as t,\n reltuples::bigint as est_cnt\nFROM pg_catalog.pg_class c, pg_catalog.pg_inherits i\nWHERE c.oid = i.inhrelid AND i.inhparent = 'items'::regclass)\nSELECT partition, tstzrange(\n t[1]::timestamptz,\n t[2]::timestamptz\n), t[1]::timestamptz as pstart,\n t[2]::timestamptz as pend, est_cnt\nFROM base\nORDER BY 2 desc;\n\nCREATE OR REPLACE VIEW items_partitions AS\nSELECT * FROM all_items_partitions WHERE est_cnt>0;\n\nCREATE OR REPLACE FUNCTION get_item(_id text) RETURNS jsonb AS $$\n SELECT content FROM items WHERE id=_id;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION delete_item(_id text) RETURNS VOID AS $$\nDECLARE\nout items%ROWTYPE;\nBEGIN\n DELETE FROM items WHERE id = _id RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION create_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION update_item(data jsonb) RETURNS VOID AS $$\nDECLARE\n out items%ROWTYPE;\nBEGIN\n UPDATE items SET content=data WHERE id = data->>'id' RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION create_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION collection_bbox(id text) RETURNS jsonb AS $$\nSELECT (replace(replace(replace(st_extent(geometry)::text,'BOX(','[['),')',']]'),' ',','))::jsonb\nFROM items WHERE collection_id=$1;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION collection_temporal_extent(id text) RETURNS jsonb AS $$\nSELECT to_jsonb(array[array[min(datetime)::text, max(datetime)::text]])\nFROM items WHERE collection_id=$1;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection_extents() RETURNS VOID AS $$\nUPDATE collections SET\n content = content ||\n jsonb_build_object(\n 'extent', jsonb_build_object(\n 'spatial', jsonb_build_object(\n 'bbox', collection_bbox(collections.id)\n ),\n 'temporal', jsonb_build_object(\n 'interval', collection_temporal_extent(collections.id)\n )\n )\n )\n;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION items_path(\n IN dotpath text,\n OUT field text,\n OUT path text,\n OUT path_txt text,\n OUT jsonpath text,\n OUT eq text\n) RETURNS RECORD AS $$\nDECLARE\npath_elements text[];\nlast_element text;\nBEGIN\ndotpath := replace(trim(dotpath), 'properties.', '');\n\nIF dotpath = '' THEN\n RETURN;\nEND IF;\n\npath_elements := string_to_array(dotpath, '.');\njsonpath := NULL;\n\nIF path_elements[1] IN ('id','geometry','datetime') THEN\n field := path_elements[1];\n path_elements := path_elements[2:];\nELSIF path_elements[1] = 'collection' THEN\n field := 'collection_id';\n path_elements := path_elements[2:];\nELSIF path_elements[1] IN ('links', 'assets', 'stac_version', 'stac_extensions') THEN\n field := 'content';\nELSE\n field := 'content';\n path_elements := '{properties}'::text[] || path_elements;\nEND IF;\nIF cardinality(path_elements)<1 THEN\n path := field;\n path_txt := field;\n jsonpath := '$';\n eq := NULL;\n RETURN;\nEND IF;\n\n\nlast_element := path_elements[cardinality(path_elements)];\npath_elements := path_elements[1:cardinality(path_elements)-1];\njsonpath := concat(array_to_string('{$}'::text[] || array_map_ident(path_elements), '.'), '.', quote_ident(last_element));\npath_elements := array_map_literal(path_elements);\npath := format($F$ properties->%s $F$, quote_literal(dotpath));\npath_txt := format($F$ properties->>%s $F$, quote_literal(dotpath));\neq := format($F$ properties @? '$.%s[*] ? (@ == %%s) '$F$, quote_ident(dotpath));\n\nRAISE NOTICE 'ITEMS PATH -- % % % % %', field, path, path_txt, jsonpath, eq;\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION parse_dtrange(IN _indate jsonb, OUT _tstzrange tstzrange) AS $$\nWITH t AS (\n SELECT CASE\n WHEN _indate ? 'timestamp' THEN\n ARRAY[_indate->>'timestamp', 'infinity']\n WHEN _indate ? 'interval' THEN\n textarr(_indate->'interval')\n WHEN jsonb_typeof(_indate) = 'array' THEN\n textarr(_indate)\n ELSE\n regexp_split_to_array(\n btrim(_indate::text,'\"'),\n '/'\n )\n END AS arr\n)\n, t1 AS (\n SELECT\n CASE\n WHEN array_upper(arr,1) = 1 OR arr[1] = '..' OR arr[1] IS NULL THEN '-infinity'::timestamptz\n ELSE arr[1]::timestamptz\n END AS st,\n CASE\n WHEN array_upper(arr,1) = 1 THEN arr[1]::timestamptz\n WHEN arr[2] = '..' OR arr[2] IS NULL THEN 'infinity'::timestamptz\n ELSE arr[2]::timestamptz\n END AS et\n FROM t\n)\nSELECT\n tstzrange(st,et)\nFROM t1;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION bbox_geom(_bbox jsonb) RETURNS geometry AS $$\nSELECT CASE jsonb_array_length(_bbox)\n WHEN 4 THEN\n ST_SetSRID(ST_MakeEnvelope(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float,\n (_bbox->>3)::float\n ),4326)\n WHEN 6 THEN\n ST_SetSRID(ST_3DMakeBox(\n ST_MakePoint(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float\n ),\n ST_MakePoint(\n (_bbox->>3)::float,\n (_bbox->>4)::float,\n (_bbox->>5)::float\n )\n ),4326)\n ELSE null END;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION cql_and_append(existing jsonb, newfilters jsonb) RETURNS jsonb AS $$\nSELECT CASE WHEN existing ? 'filter' AND newfilters IS NOT NULL THEN\n jsonb_build_object(\n 'and',\n jsonb_build_array(\n existing->'filter',\n newfilters\n )\n )\nELSE\n newfilters\nEND;\n$$ LANGUAGE SQL;\n\n\n-- ADDs base filters (ids, collections, datetime, bbox, intersects) that are\n-- added outside of the filter/query in the stac request\nCREATE OR REPLACE FUNCTION add_filters_to_cql(j jsonb) RETURNS jsonb AS $$\nDECLARE\nnewprop jsonb;\nnewprops jsonb := '[]'::jsonb;\nBEGIN\nIF j ? 'ids' THEN\n newprop := jsonb_build_object(\n 'in',\n jsonb_build_array(\n '{\"property\":\"id\"}'::jsonb,\n j->'ids'\n )\n );\n newprops := jsonb_insert(newprops, '{1}', newprop);\nEND IF;\nIF j ? 'collections' THEN\n newprop := jsonb_build_object(\n 'in',\n jsonb_build_array(\n '{\"property\":\"collection\"}'::jsonb,\n j->'collections'\n )\n );\n newprops := jsonb_insert(newprops, '{1}', newprop);\nEND IF;\n\nIF j ? 'datetime' THEN\n newprop := format(\n '{\"anyinteracts\":[{\"property\":\"datetime\"}, %s]}',\n j->'datetime'\n );\n newprops := jsonb_insert(newprops, '{1}', newprop);\nEND IF;\n\nIF j ? 'bbox' THEN\n newprop := format(\n '{\"intersects\":[{\"property\":\"geometry\"}, %s]}',\n j->'bbox'\n );\n newprops := jsonb_insert(newprops, '{1}', newprop);\nEND IF;\n\nIF j ? 'intersects' THEN\n newprop := format(\n '{\"intersects\":[{\"property\":\"geometry\"}, %s]}',\n j->'intersects'\n );\n newprops := jsonb_insert(newprops, '{1}', newprop);\nEND IF;\n\nRAISE NOTICE 'newprops: %', newprops;\n\nIF newprops IS NOT NULL AND jsonb_array_length(newprops) > 0 THEN\n return jsonb_set(\n j,\n '{filter}',\n cql_and_append(j, jsonb_build_object('and', newprops))\n ) - '{ids,collections,datetime,bbox,intersects}'::text[];\nEND IF;\n\nreturn j;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION query_to_cqlfilter(j jsonb) RETURNS jsonb AS $$\n-- Translates anything passed in through the deprecated \"query\" into equivalent CQL\nWITH t AS (\n SELECT key as property, value as ops\n FROM jsonb_each(j->'query')\n), t2 AS (\n SELECT property, (jsonb_each(ops)).*\n FROM t WHERE jsonb_typeof(ops) = 'object'\n UNION ALL\n SELECT property, 'eq', ops\n FROM t WHERE jsonb_typeof(ops) != 'object'\n), t3 AS (\nSELECT\n jsonb_strip_nulls(jsonb_build_object(\n 'and',\n jsonb_agg(\n jsonb_build_object(\n key,\n jsonb_build_array(\n jsonb_build_object('property',property),\n value\n )\n )\n )\n )) as qcql FROM t2\n)\nSELECT\n CASE WHEN qcql IS NOT NULL THEN\n jsonb_set(j, '{filter}', cql_and_append(j, qcql)) - 'query'\n ELSE j\n END\nFROM t3\n;\n$$ LANGUAGE SQL;\n\n\n\nCREATE OR REPLACE FUNCTION temporal_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\nll text := 'datetime';\nlh text := 'end_datetime';\nrrange tstzrange;\nrl text;\nrh text;\noutq text;\nBEGIN\nrrange := parse_dtrange(args->1);\nRAISE NOTICE 'Constructing temporal query OP: %, ARGS: %, RRANGE: %', op, args, rrange;\nop := lower(op);\nrl := format('%L::timestamptz', lower(rrange));\nrh := format('%L::timestamptz', upper(rrange));\noutq := CASE op\n WHEN 't_before' THEN 'lh < rl'\n WHEN 't_after' THEN 'll > rh'\n WHEN 't_meets' THEN 'lh = rl'\n WHEN 't_metby' THEN 'll = rh'\n WHEN 't_overlaps' THEN 'll < rl AND rl < lh < rh'\n WHEN 't_overlappedby' THEN 'rl < ll < rh AND lh > rh'\n WHEN 't_starts' THEN 'll = rl AND lh < rh'\n WHEN 't_startedby' THEN 'll = rl AND lh > rh'\n WHEN 't_during' THEN 'll > rl AND lh < rh'\n WHEN 't_contains' THEN 'll < rl AND lh > rh'\n WHEN 't_finishes' THEN 'll > rl AND lh = rh'\n WHEN 't_finishedby' THEN 'll < rl AND lh = rh'\n WHEN 't_equals' THEN 'll = rl AND lh = rh'\n WHEN 't_disjoint' THEN 'NOT (ll <= rh AND lh >= rl)'\n WHEN 't_intersects' THEN 'll <= rh AND lh >= rl'\n WHEN 'anyinteracts' THEN 'll <= rh AND lh >= rl'\nEND;\noutq := regexp_replace(outq, '\\mll\\M', ll);\noutq := regexp_replace(outq, '\\mlh\\M', lh);\noutq := regexp_replace(outq, '\\mrl\\M', rl);\noutq := regexp_replace(outq, '\\mrh\\M', rh);\noutq := format('(%s)', outq);\nRETURN outq;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION spatial_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\ngeom text;\nj jsonb := args->1;\nBEGIN\nop := lower(op);\nRAISE NOTICE 'Constructing spatial query OP: %, ARGS: %', op, args;\nIF op NOT IN ('s_equals','s_disjoint','s_touches','s_within','s_overlaps','s_crosses','s_intersects','intersects','s_contains') THEN\n RAISE EXCEPTION 'Spatial Operator % Not Supported', op;\nEND IF;\nop := regexp_replace(op, '^s_', 'st_');\nIF op = 'intersects' THEN\n op := 'st_intersects';\nEND IF;\n-- Convert geometry to WKB string\nIF j ? 'type' AND j ? 'coordinates' THEN\n geom := st_geomfromgeojson(j)::text;\nELSIF jsonb_typeof(j) = 'array' THEN\n geom := bbox_geom(j)::text;\nEND IF;\n\nRETURN format('%s(geometry, %L::geometry)', op, geom);\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n/* cql_query_op -- Parses a CQL query operation, recursing when necessary\n IN jsonb -- a subelement from a valid stac query\n IN text -- the operator being used on elements passed in\n RETURNS a SQL fragment to be used in a WHERE clause\n*/\nCREATE OR REPLACE FUNCTION cql_query_op(j jsonb, _op text DEFAULT NULL) RETURNS text AS $$\nDECLARE\njtype text := jsonb_typeof(j);\nop text := lower(_op);\nops jsonb :=\n '{\n \"eq\": \"%s = %s\",\n \"lt\": \"%s < %s\",\n \"lte\": \"%s <= %s\",\n \"gt\": \"%s > %s\",\n \"gte\": \"%s >= %s\",\n \"le\": \"%s <= %s\",\n \"ge\": \"%s >= %s\",\n \"=\": \"%s = %s\",\n \"<\": \"%s < %s\",\n \"<=\": \"%s <= %s\",\n \">\": \"%s > %s\",\n \">=\": \"%s >= %s\",\n \"like\": \"%s LIKE %s\",\n \"ilike\": \"%s ILIKE %s\",\n \"+\": \"%s + %s\",\n \"-\": \"%s - %s\",\n \"*\": \"%s * %s\",\n \"/\": \"%s / %s\",\n \"in\": \"%s = ANY (%s)\",\n \"not\": \"NOT (%s)\",\n \"between\": \"%s BETWEEN %s AND %s\",\n \"lower\":\" lower(%s)\",\n \"upper\":\" upper(%s)\",\n \"isnull\": \"%s IS NULL\"\n }'::jsonb;\nret text;\nargs text[] := NULL;\n\nBEGIN\nRAISE NOTICE 'j: %, op: %, jtype: %', j, op, jtype;\n\n-- for in, convert value, list to array syntax to match other ops\nIF op = 'in' and j ? 'value' and j ? 'list' THEN\n j := jsonb_build_array( j->'value', j->'list');\n jtype := 'array';\n RAISE NOTICE 'IN: j: %, jtype: %', j, jtype;\nEND IF;\n\nIF op = 'between' and j ? 'value' and j ? 'lower' and j ? 'upper' THEN\n j := jsonb_build_array( j->'value', j->'lower', j->'upper');\n jtype := 'array';\n RAISE NOTICE 'BETWEEN: j: %, jtype: %', j, jtype;\nEND IF;\n\nIF op = 'not' AND jtype = 'object' THEN\n j := jsonb_build_array( j );\n jtype := 'array';\n RAISE NOTICE 'NOT: j: %, jtype: %', j, jtype;\nEND IF;\n\n-- Set Lower Case on Both Arguments When Case Insensitive Flag Set\nIF op in ('eq','lt','lte','gt','gte','like') AND jsonb_typeof(j->2) = 'boolean' THEN\n IF (j->>2)::boolean THEN\n RETURN format(concat('(',ops->>op,')'), cql_query_op(jsonb_build_array(j->0), 'lower'), cql_query_op(jsonb_build_array(j->1), 'lower'));\n END IF;\nEND IF;\n\n-- Special Case when comparing a property in a jsonb field to a string or number using eq\n-- Allows to leverage GIN index on jsonb fields\nIF op = 'eq' THEN\n IF j->0 ? 'property'\n AND jsonb_typeof(j->1) IN ('number','string')\n AND (items_path(j->0->>'property')).eq IS NOT NULL\n THEN\n RETURN format((items_path(j->0->>'property')).eq, j->1);\n END IF;\nEND IF;\n\n\n\nIF op ilike 't_%' or op = 'anyinteracts' THEN\n RETURN temporal_op_query(op, j);\nEND IF;\n\nIF op ilike 's_%' or op = 'intersects' THEN\n RETURN spatial_op_query(op, j);\nEND IF;\n\n\nIF jtype = 'object' THEN\n RAISE NOTICE 'parsing object';\n IF j ? 'property' THEN\n -- Convert the property to be used as an identifier\n return (items_path(j->>'property')).path_txt;\n ELSIF _op IS NULL THEN\n -- Iterate to convert elements in an object where the operator has not been set\n -- Combining with AND\n SELECT\n array_to_string(array_agg(cql_query_op(e.value, e.key)), ' AND ')\n INTO ret\n FROM jsonb_each(j) e;\n RETURN ret;\n END IF;\nEND IF;\n\nIF jtype = 'string' THEN\n RETURN quote_literal(j->>0);\nEND IF;\n\nIF jtype ='number' THEN\n RETURN (j->>0)::numeric;\nEND IF;\n\nIF jtype = 'array' AND op IS NULL THEN\n RAISE NOTICE 'Parsing array into array arg. j: %', j;\n SELECT format($f$ '{%s}'::text[] $f$, string_agg(e,',')) INTO ret FROM jsonb_array_elements_text(j) e;\n RETURN ret;\nEND IF;\n\n\n-- If the type of the passed json is an array\n-- Calculate the arguments that will be passed to functions/operators\nIF jtype = 'array' THEN\n RAISE NOTICE 'Parsing array into args. j: %', j;\n -- If any argument is numeric, cast any text arguments to numeric\n IF j @? '$[*] ? (@.type() == \"number\")' THEN\n SELECT INTO args\n array_agg(concat('(',cql_query_op(e),')::numeric'))\n FROM jsonb_array_elements(j) e;\n ELSE\n SELECT INTO args\n array_agg(cql_query_op(e))\n FROM jsonb_array_elements(j) e;\n END IF;\n --RETURN args;\nEND IF;\nRAISE NOTICE 'ARGS after array cleaning: %', args;\n\nIF op IS NULL THEN\n RETURN args::text[];\nEND IF;\n\nIF args IS NULL OR cardinality(args) < 1 THEN\n RAISE NOTICE 'No Args';\n RETURN '';\nEND IF;\n\nIF op IN ('and','or') THEN\n SELECT\n CONCAT(\n '(',\n array_to_string(args, UPPER(CONCAT(' ',op,' '))),\n ')'\n ) INTO ret\n FROM jsonb_array_elements(j) e;\n RETURN ret;\nEND IF;\n\n-- If the op is in the ops json then run using the template in the json\nIF ops ? op THEN\n RAISE NOTICE 'ARGS: % MAPPED: %',args, array_map_literal(args);\n\n RETURN format(concat('(',ops->>op,')'), VARIADIC args);\nEND IF;\n\nRETURN j->>0;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n/* cql_query_op -- Parses a CQL query operation, recursing when necessary\n IN jsonb -- a subelement from a valid stac query\n IN text -- the operator being used on elements passed in\n RETURNS a SQL fragment to be used in a WHERE clause\n*/\nDROP FUNCTION IF EXISTS cql2_query;\nCREATE OR REPLACE FUNCTION cql2_query(j jsonb, recursion int DEFAULT 0) RETURNS text AS $$\nDECLARE\nargs jsonb := j->'args';\njtype text := jsonb_typeof(j->'args');\nop text := lower(j->>'op');\narg jsonb;\nargtext text;\nargstext text[] := '{}'::text[];\ninobj jsonb;\n_numeric text := '';\nops jsonb :=\n '{\n \"eq\": \"%s = %s\",\n \"lt\": \"%s < %s\",\n \"lte\": \"%s <= %s\",\n \"gt\": \"%s > %s\",\n \"gte\": \"%s >= %s\",\n \"le\": \"%s <= %s\",\n \"ge\": \"%s >= %s\",\n \"=\": \"%s = %s\",\n \"<\": \"%s < %s\",\n \"<=\": \"%s <= %s\",\n \">\": \"%s > %s\",\n \">=\": \"%s >= %s\",\n \"like\": \"%s LIKE %s\",\n \"ilike\": \"%s ILIKE %s\",\n \"+\": \"%s + %s\",\n \"-\": \"%s - %s\",\n \"*\": \"%s * %s\",\n \"/\": \"%s / %s\",\n \"in\": \"%s = ANY (%s)\",\n \"not\": \"NOT (%s)\",\n \"between\": \"%s BETWEEN (%2$s)[1] AND (%2$s)[2]\",\n \"lower\":\" lower(%s)\",\n \"upper\":\" upper(%s)\",\n \"isnull\": \"%s IS NULL\"\n }'::jsonb;\nret text;\n\nBEGIN\nRAISE NOTICE 'j: %s', j;\nIF j ? 'filter' THEN\n RETURN cql2_query(j->'filter');\nEND IF;\n\nIF j ? 'upper' THEN\nRAISE NOTICE 'upper %s',jsonb_build_object(\n 'op', 'upper',\n 'args', jsonb_build_array( j-> 'upper')\n ) ;\n RETURN cql2_query(\n jsonb_build_object(\n 'op', 'upper',\n 'args', jsonb_build_array( j-> 'upper')\n )\n );\nEND IF;\n\nIF j ? 'lower' THEN\n RETURN cql2_query(\n jsonb_build_object(\n 'op', 'lower',\n 'args', jsonb_build_array( j-> 'lower')\n )\n );\nEND IF;\n\nIF j ? 'args' AND jsonb_typeof(args) != 'array' THEN\n args := jsonb_build_array(args);\nEND IF;\n-- END Cases where no further nesting is expected\nIF j ? 'op' THEN\n -- Special case to use JSONB index for equality\n IF op IN ('eq', '=')\n AND args->0 ? 'property'\n AND jsonb_typeof(args->1) IN ('number', 'string')\n AND (items_path(args->0->>'property')).eq IS NOT NULL\n THEN\n RETURN format((items_path(args->0->>'property')).eq, args->1);\n END IF;\n\n -- Temporal Query\n IF op ilike 't_%' or op = 'anyinteracts' THEN\n RETURN temporal_op_query(op, args);\n END IF;\n\n -- Spatial Query\n IF op ilike 's_%' or op = 'intersects' THEN\n RETURN spatial_op_query(op, args);\n END IF;\n\n -- In Query - separate into separate eq statements so that we can use eq jsonb optimization\n IF op = 'in' THEN\n RAISE NOTICE '% IN args: %', repeat(' ', recursion), args;\n SELECT INTO inobj\n jsonb_agg(\n jsonb_build_object(\n 'op', 'eq',\n 'args', jsonb_build_array( args->0 , v)\n )\n )\n FROM jsonb_array_elements( args->1) v;\n RETURN cql2_query(jsonb_build_object('op','or','args',inobj));\n END IF;\nEND IF;\n\nIF j ? 'property' THEN\n RETURN (items_path(j->>'property')).path_txt;\nEND IF;\n\nIF j ? 'timestamp' THEN\n RETURN quote_literal(j->>'timestamp');\nEND IF;\n\nRAISE NOTICE '%jtype: %',repeat(' ', recursion), jtype;\nIF jsonb_typeof(j) = 'number' THEN\n RETURN format('%L::numeric', j->>0);\nEND IF;\n\nIF jsonb_typeof(j) = 'string' THEN\n RETURN quote_literal(j->>0);\nEND IF;\n\nIF jsonb_typeof(j) = 'array' THEN\n IF j @? '$[*] ? (@.type() == \"number\")' THEN\n RETURN CONCAT(quote_literal(textarr(j)::text), '::numeric[]');\n ELSE\n RETURN CONCAT(quote_literal(textarr(j)::text), '::text[]');\n END IF;\nEND IF;\nRAISE NOTICE 'ARGS after array cleaning: %', args;\n\nRAISE NOTICE '%beforeargs op: %, args: %',repeat(' ', recursion), op, args;\nIF j ? 'args' THEN\n FOR arg in SELECT * FROM jsonb_array_elements(args) LOOP\n argtext := cql2_query(arg, recursion + 1);\n RAISE NOTICE '% -- arg: %, argtext: %', repeat(' ', recursion), arg, argtext;\n argstext := argstext || argtext;\n END LOOP;\nEND IF;\nRAISE NOTICE '%afterargs op: %, argstext: %',repeat(' ', recursion), op, argstext;\n\n\nIF op IN ('and', 'or') THEN\n RAISE NOTICE 'inand op: %, argstext: %', op, argstext;\n SELECT\n concat(' ( ',array_to_string(array_agg(e), concat(' ',op,' ')),' ) ')\n INTO ret\n FROM unnest(argstext) e;\n RETURN ret;\nEND IF;\n\nIF ops ? op THEN\n IF argstext[2] ~* 'numeric' THEN\n argstext := ARRAY[concat('(',argstext[1],')::numeric')] || argstext[2:3];\n END IF;\n RETURN format(concat('(',ops->>op,')'), VARIADIC argstext);\nEND IF;\n\nRAISE NOTICE '%op: %, argstext: %',repeat(' ', recursion), op, argstext;\n\nRETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n\nCREATE OR REPLACE FUNCTION cql_to_where(_search jsonb = '{}'::jsonb) RETURNS text AS $$\nDECLARE\nfilterlang text;\nsearch jsonb := _search;\n_where text;\nBEGIN\n\nRAISE NOTICE 'SEARCH CQL Final: %', search;\nfilterlang := COALESCE(\n search->>'filter-lang',\n get_setting('default-filter-lang', _search->'conf')\n);\n\nIF filterlang = 'cql-json' THEN\n search := query_to_cqlfilter(search);\n search := add_filters_to_cql(search);\n _where := cql_query_op(search->'filter');\nELSE\n _where := cql2_query(search->'filter');\nEND IF;\n\nIF trim(_where) = '' THEN\n _where := NULL;\nEND IF;\n_where := coalesce(_where, ' TRUE ');\nRETURN _where;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION parse_sort_dir(_dir text, reverse boolean default false) RETURNS text AS $$\nWITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n) SELECT\n CASE\n WHEN NOT reverse THEN d\n WHEN d = 'ASC' THEN 'DESC'\n WHEN d = 'DESC' THEN 'ASC'\n END\nFROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION sort_dir_to_op(_dir text, prev boolean default false) RETURNS text AS $$\nWITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n) SELECT\n CASE\n WHEN d = 'ASC' AND prev THEN '<='\n WHEN d = 'DESC' AND prev THEN '>='\n WHEN d = 'ASC' THEN '>='\n WHEN d = 'DESC' THEN '<='\n END\nFROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION field_orderby(p text) RETURNS text AS $$\nWITH t AS (\n SELECT\n replace(trim(substring(indexdef from 'btree \\((.*)\\)')),' ','')as s\n FROM pg_indexes WHERE schemaname='pgstac' AND tablename='items' AND indexdef ~* 'btree' AND indexdef ~* 'properties'\n) SELECT s FROM t WHERE strpos(s, lower(trim(p)))>0;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION sort_sqlorderby(\n _search jsonb DEFAULT NULL,\n reverse boolean DEFAULT FALSE\n) RETURNS text AS $$\nWITH sortby AS (\n SELECT coalesce(_search->'sortby','[{\"field\":\"datetime\", \"direction\":\"desc\"}]') as sort\n), withid AS (\n SELECT CASE\n WHEN sort @? '$[*] ? (@.field == \"id\")' THEN sort\n ELSE sort || '[{\"field\":\"id\", \"direction\":\"desc\"}]'::jsonb\n END as sort\n FROM sortby\n), withid_rows AS (\n SELECT jsonb_array_elements(sort) as value FROM withid\n),sorts AS (\n SELECT\n coalesce(field_orderby((items_path(value->>'field')).path_txt), (items_path(value->>'field')).path) as key,\n parse_sort_dir(value->>'direction', reverse) as dir\n FROM withid_rows\n)\nSELECT array_to_string(\n array_agg(concat(key, ' ', dir)),\n ', '\n) FROM sorts;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_sort_dir(sort_item jsonb) RETURNS text AS $$\nSELECT CASE WHEN sort_item->>'direction' ILIKE 'desc%' THEN 'DESC' ELSE 'ASC' END;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION get_token_filter(_search jsonb = '{}'::jsonb, token_rec jsonb DEFAULT NULL) RETURNS text AS $$\nDECLARE\n token_id text;\n filters text[] := '{}'::text[];\n prev boolean := TRUE;\n field text;\n dir text;\n sort record;\n orfilters text[] := '{}'::text[];\n andfilters text[] := '{}'::text[];\n output text;\n token_where text;\nBEGIN\n -- If no token provided return NULL\n IF token_rec IS NULL THEN\n IF NOT (_search ? 'token' AND\n (\n (_search->>'token' ILIKE 'prev:%')\n OR\n (_search->>'token' ILIKE 'next:%')\n )\n ) THEN\n RETURN NULL;\n END IF;\n prev := (_search->>'token' ILIKE 'prev:%');\n token_id := substr(_search->>'token', 6);\n SELECT to_jsonb(items) INTO token_rec FROM items WHERE id=token_id;\n END IF;\n RAISE NOTICE 'TOKEN ID: %', token_rec->'id';\n\n CREATE TEMP TABLE sorts (\n _row int GENERATED ALWAYS AS IDENTITY NOT NULL,\n _field text PRIMARY KEY,\n _dir text NOT NULL,\n _val text\n ) ON COMMIT DROP;\n\n -- Make sure we only have distinct columns to sort with taking the first one we get\n INSERT INTO sorts (_field, _dir)\n SELECT\n (items_path(value->>'field')).path,\n get_sort_dir(value)\n FROM\n jsonb_array_elements(coalesce(_search->'sortby','[{\"field\":\"datetime\",\"direction\":\"desc\"}]'))\n ON CONFLICT DO NOTHING\n ;\n RAISE NOTICE 'sorts 1: %', (SELECT jsonb_agg(to_json(sorts)) FROM sorts);\n -- Get the first sort direction provided. As the id is a primary key, if there are any\n -- sorts after id they won't do anything, so make sure that id is the last sort item.\n SELECT _dir INTO dir FROM sorts ORDER BY _row ASC LIMIT 1;\n IF EXISTS (SELECT 1 FROM sorts WHERE _field = 'id') THEN\n DELETE FROM sorts WHERE _row > (SELECT _row FROM sorts WHERE _field = 'id' ORDER BY _row ASC);\n ELSE\n INSERT INTO sorts (_field, _dir) VALUES ('id', dir);\n END IF;\n\n -- Add value from looked up item to the sorts table\n UPDATE sorts SET _val=quote_literal(token_rec->>_field);\n\n -- Check if all sorts are the same direction and use row comparison\n -- to filter\n RAISE NOTICE 'sorts 2: %', (SELECT jsonb_agg(to_json(sorts)) FROM sorts);\n\n IF (SELECT count(DISTINCT _dir) FROM sorts) = 1 THEN\n SELECT format(\n '(%s) %s (%s)',\n concat_ws(', ', VARIADIC array_agg(quote_ident(_field))),\n CASE WHEN (prev AND dir = 'ASC') OR (NOT prev AND dir = 'DESC') THEN '<' ELSE '>' END,\n concat_ws(', ', VARIADIC array_agg(_val))\n ) INTO output FROM sorts\n WHERE token_rec ? _field\n ;\n ELSE\n FOR sort IN SELECT * FROM sorts ORDER BY _row asc LOOP\n RAISE NOTICE 'SORT: %', sort;\n IF sort._row = 1 THEN\n orfilters := orfilters || format('(%s %s %s)',\n quote_ident(sort._field),\n CASE WHEN (prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC') THEN '<' ELSE '>' END,\n sort._val\n );\n ELSE\n orfilters := orfilters || format('(%s AND %s %s %s)',\n array_to_string(andfilters, ' AND '),\n quote_ident(sort._field),\n CASE WHEN (prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC') THEN '<' ELSE '>' END,\n sort._val\n );\n\n END IF;\n andfilters := andfilters || format('%s = %s',\n quote_ident(sort._field),\n sort._val\n );\n END LOOP;\n output := array_to_string(orfilters, ' OR ');\n END IF;\n DROP TABLE IF EXISTS sorts;\n token_where := concat('(',coalesce(output,'true'),')');\n IF trim(token_where) = '' THEN\n token_where := NULL;\n END IF;\n RAISE NOTICE 'TOKEN_WHERE: |%|',token_where;\n RETURN token_where;\n END;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION search_tohash(jsonb) RETURNS jsonb AS $$\n SELECT $1 - '{token,limit,context,includes,excludes}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION search_hash(jsonb, jsonb) RETURNS text AS $$\n SELECT md5(concat(search_tohash($1)::text,$2::text));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TABLE IF NOT EXISTS searches(\n hash text GENERATED ALWAYS AS (search_hash(search, metadata)) STORED PRIMARY KEY,\n search jsonb NOT NULL,\n _where text,\n orderby text,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n metadata jsonb DEFAULT '{}'::jsonb NOT NULL\n);\nCREATE TABLE IF NOT EXISTS search_wheres(\n id bigint generated always as identity primary key,\n _where text NOT NULL,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n statslastupdated timestamptz,\n estimated_count bigint,\n estimated_cost float,\n time_to_estimate float,\n total_count bigint,\n time_to_count float,\n partitions text[]\n);\n\nCREATE INDEX IF NOT EXISTS search_wheres_partitions ON search_wheres USING GIN (partitions);\nCREATE UNIQUE INDEX IF NOT EXISTS search_wheres_where ON search_wheres ((md5(_where)));\n\nCREATE OR REPLACE FUNCTION where_stats(inwhere text, updatestats boolean default false, conf jsonb default null) RETURNS search_wheres AS $$\nDECLARE\n t timestamptz;\n i interval;\n explain_json jsonb;\n partitions text[];\n sw search_wheres%ROWTYPE;\nBEGIN\n SELECT * INTO sw FROM search_wheres WHERE _where=inwhere FOR UPDATE;\n\n -- Update statistics if explicitly set, if statistics do not exist, or statistics ttl has expired\n IF NOT updatestats THEN\n RAISE NOTICE 'Checking if update is needed.';\n RAISE NOTICE 'Stats Last Updated: %', sw.statslastupdated;\n RAISE NOTICE 'TTL: %, Age: %', context_stats_ttl(conf), now() - sw.statslastupdated;\n RAISE NOTICE 'Context: %, Existing Total: %', context(conf), sw.total_count;\n IF\n sw.statslastupdated IS NULL\n OR (now() - sw.statslastupdated) > context_stats_ttl(conf)\n OR (context(conf) != 'off' AND sw.total_count IS NULL)\n THEN\n updatestats := TRUE;\n END IF;\n END IF;\n\n sw._where := inwhere;\n sw.lastused := now();\n sw.usecount := coalesce(sw.usecount,0) + 1;\n\n IF NOT updatestats THEN\n UPDATE search_wheres SET\n lastused = sw.lastused,\n usecount = sw.usecount\n WHERE _where = inwhere\n RETURNING * INTO sw\n ;\n RETURN sw;\n END IF;\n -- Use explain to get estimated count/cost and a list of the partitions that would be hit by the query\n t := clock_timestamp();\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s', inwhere)\n INTO explain_json;\n RAISE NOTICE 'Time for just the explain: %', clock_timestamp() - t;\n WITH t AS (\n SELECT j->>0 as p FROM\n jsonb_path_query(\n explain_json,\n 'strict $.**.\"Relation Name\" ? (@ != null)'\n ) j\n ), ordered AS (\n SELECT p FROM t ORDER BY p DESC\n -- SELECT p FROM t JOIN items_partitions\n -- ON (t.p = items_partitions.partition)\n -- ORDER BY pstart DESC\n )\n SELECT array_agg(p) INTO partitions FROM ordered;\n i := clock_timestamp() - t;\n RAISE NOTICE 'Time for explain + join: %', clock_timestamp() - t;\n\n\n\n sw.statslastupdated := now();\n sw.estimated_count := explain_json->0->'Plan'->'Plan Rows';\n sw.estimated_cost := explain_json->0->'Plan'->'Total Cost';\n sw.time_to_estimate := extract(epoch from i);\n sw.partitions := partitions;\n\n -- Do a full count of rows if context is set to on or if auto is set and estimates are low enough\n IF\n context(conf) = 'on'\n OR\n ( context(conf) = 'auto' AND\n (\n sw.estimated_count < context_estimated_count(conf)\n OR\n sw.estimated_cost < context_estimated_cost(conf)\n )\n )\n THEN\n t := clock_timestamp();\n RAISE NOTICE 'Calculating actual count...';\n EXECUTE format(\n 'SELECT count(*) FROM items WHERE %s',\n inwhere\n ) INTO sw.total_count;\n i := clock_timestamp() - t;\n RAISE NOTICE 'Actual Count: % -- %', sw.total_count, i;\n sw.time_to_count := extract(epoch FROM i);\n ELSE\n sw.total_count := NULL;\n sw.time_to_count := NULL;\n END IF;\n\n\n INSERT INTO search_wheres\n (_where, lastused, usecount, statslastupdated, estimated_count, estimated_cost, time_to_estimate, partitions, total_count, time_to_count)\n SELECT sw._where, sw.lastused, sw.usecount, sw.statslastupdated, sw.estimated_count, sw.estimated_cost, sw.time_to_estimate, sw.partitions, sw.total_count, sw.time_to_count\n ON CONFLICT ((md5(_where)))\n DO UPDATE\n SET\n lastused = sw.lastused,\n usecount = sw.usecount,\n statslastupdated = sw.statslastupdated,\n estimated_count = sw.estimated_count,\n estimated_cost = sw.estimated_cost,\n time_to_estimate = sw.time_to_estimate,\n partitions = sw.partitions,\n total_count = sw.total_count,\n time_to_count = sw.time_to_count\n ;\n RETURN sw;\nEND;\n$$ LANGUAGE PLPGSQL ;\n\n\nCREATE OR REPLACE FUNCTION items_count(_where text) RETURNS bigint AS $$\nDECLARE\ncnt bigint;\nBEGIN\nEXECUTE format('SELECT count(*) FROM items WHERE %s', _where) INTO cnt;\nRETURN cnt;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n\n\n\nDROP FUNCTION IF EXISTS search_query;\nCREATE OR REPLACE FUNCTION search_query(\n _search jsonb = '{}'::jsonb,\n updatestats boolean = false,\n _metadata jsonb = '{}'::jsonb\n) RETURNS searches AS $$\nDECLARE\n search searches%ROWTYPE;\n pexplain jsonb;\n t timestamptz;\n i interval;\nBEGIN\nSELECT * INTO search FROM searches\nWHERE hash=search_hash(_search, _metadata) FOR UPDATE;\n\n-- Calculate the where clause if not already calculated\nIF search._where IS NULL THEN\n search._where := cql_to_where(_search);\nEND IF;\n\n-- Calculate the order by clause if not already calculated\nIF search.orderby IS NULL THEN\n search.orderby := sort_sqlorderby(_search);\nEND IF;\n\nPERFORM where_stats(search._where, updatestats, _search->'conf');\n\nsearch.lastused := now();\nsearch.usecount := coalesce(search.usecount, 0) + 1;\nINSERT INTO searches (search, _where, orderby, lastused, usecount, metadata)\nVALUES (_search, search._where, search.orderby, search.lastused, search.usecount, _metadata)\nON CONFLICT (hash) DO\nUPDATE SET\n _where = EXCLUDED._where,\n orderby = EXCLUDED.orderby,\n lastused = EXCLUDED.lastused,\n usecount = EXCLUDED.usecount,\n metadata = EXCLUDED.metadata\nRETURNING * INTO search\n;\nRETURN search;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n\nCREATE OR REPLACE FUNCTION search(_search jsonb = '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n searches searches%ROWTYPE;\n _where text;\n token_where text;\n full_where text;\n orderby text;\n query text;\n token_type text := substr(_search->>'token',1,4);\n _limit int := coalesce((_search->>'limit')::int, 10);\n curs refcursor;\n cntr int := 0;\n iter_record items%ROWTYPE;\n first_record items%ROWTYPE;\n last_record items%ROWTYPE;\n out_records jsonb := '[]'::jsonb;\n prev_query text;\n next text;\n prev_id text;\n has_next boolean := false;\n has_prev boolean := false;\n prev text;\n total_count bigint;\n context jsonb;\n collection jsonb;\n includes text[];\n excludes text[];\n exit_flag boolean := FALSE;\n batches int := 0;\n timer timestamptz := clock_timestamp();\n pstart timestamptz;\n pend timestamptz;\n pcurs refcursor;\n search_where search_wheres%ROWTYPE;\nBEGIN\nsearches := search_query(_search);\n_where := searches._where;\norderby := searches.orderby;\nsearch_where := where_stats(_where);\ntotal_count := coalesce(search_where.total_count, search_where.estimated_count);\n\n\nIF token_type='prev' THEN\n token_where := get_token_filter(_search, null::jsonb);\n orderby := sort_sqlorderby(_search, TRUE);\nEND IF;\nIF token_type='next' THEN\n token_where := get_token_filter(_search, null::jsonb);\nEND IF;\n\nfull_where := concat_ws(' AND ', _where, token_where);\nRAISE NOTICE 'FULL QUERY % %', full_where, clock_timestamp()-timer;\ntimer := clock_timestamp();\n\nCREATE TEMP TABLE results (content jsonb) ON COMMIT DROP;\n\n\nFOR query IN SELECT partition_queries(full_where, orderby, search_where.partitions) LOOP\n timer := clock_timestamp();\n query := format('%s LIMIT %s', query, _limit + 1);\n RAISE NOTICE 'Partition Query: %', query;\n batches := batches + 1;\n -- curs = create_cursor(query);\n OPEN curs FOR EXECUTE query;\n LOOP\n FETCH curs into iter_record;\n EXIT WHEN NOT FOUND;\n cntr := cntr + 1;\n last_record := iter_record;\n IF cntr = 1 THEN\n first_record := last_record;\n END IF;\n IF cntr <= _limit THEN\n INSERT INTO results (content) VALUES (last_record.content);\n -- out_records := out_records || last_record.content;\n\n ELSIF cntr > _limit THEN\n has_next := true;\n exit_flag := true;\n EXIT;\n END IF;\n END LOOP;\n CLOSE curs;\n RAISE NOTICE 'Query took %. Total Time %', clock_timestamp()-timer, ftime();\n timer := clock_timestamp();\n EXIT WHEN exit_flag;\nEND LOOP;\nRAISE NOTICE 'Scanned through % partitions.', batches;\n\nSELECT jsonb_agg(content) INTO out_records FROM results;\n\nDROP TABLE results;\n\n\n-- Flip things around if this was the result of a prev token query\nIF token_type='prev' THEN\n out_records := flip_jsonb_array(out_records);\n first_record := last_record;\nEND IF;\n\n-- If this query has a token, see if there is data before the first record\nIF _search ? 'token' THEN\n prev_query := format(\n 'SELECT 1 FROM items WHERE %s LIMIT 1',\n concat_ws(\n ' AND ',\n _where,\n trim(get_token_filter(_search, to_jsonb(first_record)))\n )\n );\n RAISE NOTICE 'Query to get previous record: % --- %', prev_query, first_record;\n EXECUTE prev_query INTO has_prev;\n IF FOUND and has_prev IS NOT NULL THEN\n RAISE NOTICE 'Query results from prev query: %', has_prev;\n has_prev := TRUE;\n END IF;\nEND IF;\nhas_prev := COALESCE(has_prev, FALSE);\n\nRAISE NOTICE 'token_type: %, has_next: %, has_prev: %', token_type, has_next, has_prev;\nIF has_prev THEN\n prev := out_records->0->>'id';\nEND IF;\nIF has_next OR token_type='prev' THEN\n next := out_records->-1->>'id';\nEND IF;\n\n\n-- include/exclude any fields following fields extension\nIF _search ? 'fields' THEN\n IF _search->'fields' ? 'exclude' THEN\n excludes=textarr(_search->'fields'->'exclude');\n END IF;\n IF _search->'fields' ? 'include' THEN\n includes=textarr(_search->'fields'->'include');\n IF array_length(includes, 1)>0 AND NOT 'id' = ANY (includes) THEN\n includes = includes || '{id}';\n END IF;\n END IF;\n SELECT jsonb_agg(filter_jsonb(row, includes, excludes)) INTO out_records FROM jsonb_array_elements(out_records) row;\nEND IF;\n\nIF context(_search->'conf') != 'off' THEN\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'matched', total_count,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\nELSE\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\nEND IF;\n\ncollection := jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb),\n 'next', next,\n 'prev', prev,\n 'context', context\n);\n\nRETURN collection;\nEND;\n$$ LANGUAGE PLPGSQL\nSET jit TO off\n;\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION tileenvelope(zoom int, x int, y int) RETURNS geometry AS $$\nWITH t AS (\n SELECT\n 20037508.3427892 as merc_max,\n -20037508.3427892 as merc_min,\n (2 * 20037508.3427892) / (2 ^ zoom) as tile_size\n)\nSELECT st_makeenvelope(\n merc_min + (tile_size * x),\n merc_max - (tile_size * (y + 1)),\n merc_min + (tile_size * (x + 1)),\n merc_max - (tile_size * y),\n 3857\n) FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;DROP FUNCTION IF EXISTS mercgrid;\n\n\nCREATE OR REPLACE FUNCTION ftime() RETURNS interval as $$\nSELECT age(clock_timestamp(), transaction_timestamp());\n$$ LANGUAGE SQL;\nSET SEARCH_PATH to pgstac, public;\n\nDROP FUNCTION IF EXISTS geometrysearch;\nCREATE OR REPLACE FUNCTION geometrysearch(\n IN geom geometry,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE, -- Return as soon as the passed in geometry is full covered\n IN skipcovered boolean DEFAULT TRUE -- Skip any items that would show up completely under the previous items\n) RETURNS jsonb AS $$\nDECLARE\n search searches%ROWTYPE;\n curs refcursor;\n _where text;\n query text;\n iter_record items%ROWTYPE;\n out_records jsonb[] := '{}'::jsonb[];\n exit_flag boolean := FALSE;\n counter int := 1;\n scancounter int := 1;\n remaining_limit int := _scanlimit;\n tilearea float;\n unionedgeom geometry;\n clippedgeom geometry;\n unionedgeom_area float := 0;\n prev_area float := 0;\n excludes text[];\n includes text[];\n\nBEGIN\n -- If skipcovered is true then you will always want to exit when the passed in geometry is full\n IF skipcovered THEN\n exitwhenfull := TRUE;\n END IF;\n\n SELECT * INTO search FROM searches WHERE hash=queryhash;\n\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Search with Query Hash % Not Found', queryhash;\n END IF;\n\n tilearea := st_area(geom);\n _where := format('%s AND st_intersects(geometry, %L::geometry)', search._where, geom);\n\n IF fields IS NOT NULL THEN\n IF fields ? 'fields' THEN\n fields := fields->'fields';\n END IF;\n IF fields ? 'exclude' THEN\n excludes=textarr(fields->'exclude');\n END IF;\n IF fields ? 'include' THEN\n includes=textarr(fields->'include');\n IF array_length(includes, 1)>0 AND NOT 'id' = ANY (includes) THEN\n includes = includes || '{id}';\n END IF;\n END IF;\n END IF;\n RAISE NOTICE 'fields: %, includes: %, excludes: %', fields, includes, excludes;\n\n FOR query IN SELECT * FROM partition_queries(_where, search.orderby) LOOP\n query := format('%s LIMIT %L', query, remaining_limit);\n RAISE NOTICE '%', query;\n curs = create_cursor(query);\n LOOP\n FETCH curs INTO iter_record;\n EXIT WHEN NOT FOUND;\n IF exitwhenfull OR skipcovered THEN -- If we are not using exitwhenfull or skipcovered, we do not need to do expensive geometry operations\n clippedgeom := st_intersection(geom, iter_record.geometry);\n\n IF unionedgeom IS NULL THEN\n unionedgeom := clippedgeom;\n ELSE\n unionedgeom := st_union(unionedgeom, clippedgeom);\n END IF;\n\n unionedgeom_area := st_area(unionedgeom);\n\n IF skipcovered AND prev_area = unionedgeom_area THEN\n scancounter := scancounter + 1;\n CONTINUE;\n END IF;\n\n prev_area := unionedgeom_area;\n\n RAISE NOTICE '% % % %', unionedgeom_area/tilearea, counter, scancounter, ftime();\n END IF;\n\n IF fields IS NOT NULL THEN\n out_records := out_records || filter_jsonb(iter_record.content, includes, excludes);\n ELSE\n out_records := out_records || iter_record.content;\n END IF;\n IF counter >= _limit\n OR scancounter > _scanlimit\n OR ftime() > _timelimit\n OR (exitwhenfull AND unionedgeom_area >= tilearea)\n THEN\n exit_flag := TRUE;\n EXIT;\n END IF;\n counter := counter + 1;\n scancounter := scancounter + 1;\n\n END LOOP;\n EXIT WHEN exit_flag;\n remaining_limit := _scanlimit - scancounter;\n END LOOP;\n\n RETURN jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', array_to_json(out_records)::jsonb\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDROP FUNCTION IF EXISTS geojsonsearch;\nCREATE OR REPLACE FUNCTION geojsonsearch(\n IN geojson jsonb,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_geomfromgeojson(geojson),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS xyzsearch;\nCREATE OR REPLACE FUNCTION xyzsearch(\n IN _x int,\n IN _y int,\n IN _z int,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_transform(tileenvelope(_z, _x, _y), 4326),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\nSELECT set_version('0.4.3');\n" }, { "path": "src/pgstac/migrations/pgstac.0.4.4-0.4.5.sql", "content": "SET SEARCH_PATH to pgstac, public;\nset check_function_bodies = off;\n\nCREATE OR REPLACE FUNCTION pgstac.base_stac_query(j jsonb)\n RETURNS text\n LANGUAGE plpgsql\nAS $function$\nDECLARE\n_where text := '';\ndtrange tstzrange;\ngeom geometry;\nBEGIN\n\n IF j ? 'ids' THEN\n _where := format('%s AND id = ANY (%L) ', _where, textarr(j->'ids'));\n END IF;\n IF j ? 'collections' THEN\n _where := format('%s AND collection_id = ANY (%L) ', _where, textarr(j->'collections'));\n END IF;\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n _where := format('%s AND datetime <= %L::timestamptz AND end_datetime >= %L::timestamptz ',\n _where,\n upper(dtrange),\n lower(dtrange)\n );\n END IF;\n\n IF j ? 'bbox' THEN\n geom := bbox_geom(j->'bbox');\n _where := format('%s AND geometry && %L',\n _where,\n geom\n );\n END IF;\n\n IF j ? 'intersects' THEN\n geom := st_geomfromgeojson(j->>'intersects');\n _where := format('%s AND st_intersects(geometry, %L)',\n _where,\n geom\n );\n END IF;\n\n RETURN _where;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.items_staging_ignore_insert_triggerfunc()\n RETURNS trigger\n LANGUAGE plpgsql\nAS $function$\nDECLARE\n p record;\n _partitions text[];\nBEGIN\n CREATE TEMP TABLE new_partitions ON COMMIT DROP AS\n SELECT\n items_partition_name(stac_datetime(content)) as partition,\n date_trunc('week', min(stac_datetime(content))) as partition_start\n FROM newdata\n GROUP BY 1;\n\n -- set statslastupdated in cache to be old enough cache always regenerated\n\n SELECT array_agg(partition) INTO _partitions FROM new_partitions;\n UPDATE search_wheres\n SET\n statslastupdated = NULL\n WHERE _where IN (\n SELECT _where FROM search_wheres sw WHERE sw.partitions && _partitions\n FOR UPDATE SKIP LOCKED\n )\n ;\n\n FOR p IN SELECT new_partitions.partition, new_partitions.partition_start, new_partitions.partition_start + '1 week'::interval as partition_end FROM new_partitions\n LOOP\n RAISE NOTICE 'Getting partition % ready.', p.partition;\n IF NOT items_partition_exists(p.partition) THEN\n RAISE NOTICE 'Creating partition %.', p.partition;\n PERFORM items_partition_create_worker(p.partition, p.partition_start, p.partition_end);\n END IF;\n PERFORM drop_partition_constraints(p.partition);\n END LOOP;\n\n INSERT INTO items (id, geometry, collection_id, datetime, end_datetime, properties, content)\n SELECT\n content->>'id',\n stac_geom(content),\n content->>'collection',\n stac_datetime(content),\n stac_end_datetime(content),\n properties_idx(content),\n content\n FROM newdata\n ON CONFLICT DO NOTHING\n ;\n DELETE FROM items_staging;\n\n\n FOR p IN SELECT new_partitions.partition FROM new_partitions\n LOOP\n RAISE NOTICE 'Setting constraints for partition %.', p.partition;\n PERFORM partition_checks(p.partition);\n END LOOP;\n DROP TABLE IF EXISTS new_partitions;\n RETURN NULL;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.search(_search jsonb DEFAULT '{}'::jsonb)\n RETURNS jsonb\n LANGUAGE plpgsql\n SET jit TO 'off'\nAS $function$\nDECLARE\n searches searches%ROWTYPE;\n _where text;\n token_where text;\n full_where text;\n orderby text;\n query text;\n token_type text := substr(_search->>'token',1,4);\n _limit int := coalesce((_search->>'limit')::int, 10);\n curs refcursor;\n cntr int := 0;\n iter_record items%ROWTYPE;\n first_record items%ROWTYPE;\n last_record items%ROWTYPE;\n out_records jsonb := '[]'::jsonb;\n prev_query text;\n next text;\n prev_id text;\n has_next boolean := false;\n has_prev boolean := false;\n prev text;\n total_count bigint;\n context jsonb;\n collection jsonb;\n includes text[];\n excludes text[];\n exit_flag boolean := FALSE;\n batches int := 0;\n timer timestamptz := clock_timestamp();\n pstart timestamptz;\n pend timestamptz;\n pcurs refcursor;\n search_where search_wheres%ROWTYPE;\n id text;\nBEGIN\nCREATE TEMP TABLE results (content jsonb) ON COMMIT DROP;\n-- if ids is set, short circuit and just use direct ids query for each id\n-- skip any paging or caching\n-- hard codes ordering in the same order as the array of ids\nIF _search ? 'ids' THEN\n FOR id IN SELECT jsonb_array_elements_text(_search->'ids') LOOP\n INSERT INTO results (content) SELECT content FROM item_by_id(id) WHERE content IS NOT NULL;\n END LOOP;\n SELECT INTO total_count count(*) FROM results;\nELSE\n searches := search_query(_search);\n _where := searches._where;\n orderby := searches.orderby;\n search_where := where_stats(_where);\n total_count := coalesce(search_where.total_count, search_where.estimated_count);\n\n\n\n IF token_type='prev' THEN\n token_where := get_token_filter(_search, null::jsonb);\n orderby := sort_sqlorderby(_search, TRUE);\n END IF;\n IF token_type='next' THEN\n token_where := get_token_filter(_search, null::jsonb);\n END IF;\n\n full_where := concat_ws(' AND ', _where, token_where);\n RAISE NOTICE 'FULL QUERY % %', full_where, clock_timestamp()-timer;\n timer := clock_timestamp();\n\n\n\n\n FOR query IN SELECT partition_queries(full_where, orderby, search_where.partitions) LOOP\n timer := clock_timestamp();\n query := format('%s LIMIT %s', query, _limit + 1);\n RAISE NOTICE 'Partition Query: %', query;\n batches := batches + 1;\n -- curs = create_cursor(query);\n OPEN curs FOR EXECUTE query;\n LOOP\n FETCH curs into iter_record;\n EXIT WHEN NOT FOUND;\n cntr := cntr + 1;\n last_record := iter_record;\n IF cntr = 1 THEN\n first_record := last_record;\n END IF;\n IF cntr <= _limit THEN\n INSERT INTO results (content) VALUES (last_record.content);\n -- out_records := out_records || last_record.content;\n\n ELSIF cntr > _limit THEN\n has_next := true;\n exit_flag := true;\n EXIT;\n END IF;\n END LOOP;\n CLOSE curs;\n RAISE NOTICE 'Query took %. Total Time %', clock_timestamp()-timer, ftime();\n timer := clock_timestamp();\n EXIT WHEN exit_flag;\n END LOOP;\n RAISE NOTICE 'Scanned through % partitions.', batches;\nEND IF;\n\nSELECT jsonb_agg(content) INTO out_records FROM results WHERE content is not NULL;\n\nDROP TABLE results;\n\n\n-- Flip things around if this was the result of a prev token query\nIF token_type='prev' THEN\n out_records := flip_jsonb_array(out_records);\n first_record := last_record;\nEND IF;\n\n-- If this query has a token, see if there is data before the first record\nIF _search ? 'token' THEN\n prev_query := format(\n 'SELECT 1 FROM items WHERE %s LIMIT 1',\n concat_ws(\n ' AND ',\n _where,\n trim(get_token_filter(_search, to_jsonb(first_record)))\n )\n );\n RAISE NOTICE 'Query to get previous record: % --- %', prev_query, first_record;\n EXECUTE prev_query INTO has_prev;\n IF FOUND and has_prev IS NOT NULL THEN\n RAISE NOTICE 'Query results from prev query: %', has_prev;\n has_prev := TRUE;\n END IF;\nEND IF;\nhas_prev := COALESCE(has_prev, FALSE);\n\nRAISE NOTICE 'token_type: %, has_next: %, has_prev: %', token_type, has_next, has_prev;\nIF has_prev THEN\n prev := out_records->0->>'id';\nEND IF;\nIF has_next OR token_type='prev' THEN\n next := out_records->-1->>'id';\nEND IF;\n\n\n-- include/exclude any fields following fields extension\nIF _search ? 'fields' THEN\n IF _search->'fields' ? 'exclude' THEN\n excludes=textarr(_search->'fields'->'exclude');\n END IF;\n IF _search->'fields' ? 'include' THEN\n includes=textarr(_search->'fields'->'include');\n IF array_length(includes, 1)>0 AND NOT 'id' = ANY (includes) THEN\n includes = includes || '{id}';\n END IF;\n END IF;\n SELECT jsonb_agg(filter_jsonb(row, includes, excludes)) INTO out_records FROM jsonb_array_elements(out_records) row;\nEND IF;\n\nIF context(_search->'conf') != 'off' THEN\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'matched', total_count,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\nELSE\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\nEND IF;\n\ncollection := jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb),\n 'next', next,\n 'prev', prev,\n 'context', context\n);\n\nRETURN collection;\nEND;\n$function$\n;\n\n\n\nSELECT set_version('0.4.5');\n" }, { "path": "src/pgstac/migrations/pgstac.0.4.4.sql", "content": "CREATE EXTENSION IF NOT EXISTS postgis;\nCREATE SCHEMA IF NOT EXISTS pgstac;\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE TABLE migrations (\n version text PRIMARY KEY,\n datetime timestamptz DEFAULT clock_timestamp() NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION get_version() RETURNS text AS $$\n SELECT version FROM migrations ORDER BY datetime DESC, version DESC LIMIT 1;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION set_version(text) RETURNS text AS $$\n INSERT INTO migrations (version) VALUES ($1)\n ON CONFLICT DO NOTHING\n RETURNING version;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\n\nCREATE TABLE IF NOT EXISTS pgstac_settings (\n name text PRIMARY KEY,\n value text NOT NULL\n);\n\nINSERT INTO pgstac_settings (name, value) VALUES\n ('context', 'off'),\n ('context_estimated_count', '100000'),\n ('context_estimated_cost', '1000000'),\n ('context_stats_ttl', '1 day'),\n ('default-filter-lang', 'cql2-json')\nON CONFLICT DO NOTHING\n;\n\n\nCREATE OR REPLACE FUNCTION get_setting(IN _setting text, IN conf jsonb DEFAULT NULL) RETURNS text AS $$\nSELECT COALESCE(\n conf->>_setting,\n current_setting(concat('pgstac.',_setting), TRUE),\n (SELECT value FROM pgstac_settings WHERE name=_setting)\n);\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context();\nCREATE OR REPLACE FUNCTION context(conf jsonb DEFAULT NULL) RETURNS text AS $$\n SELECT get_setting('context', conf);\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_estimated_count();\nCREATE OR REPLACE FUNCTION context_estimated_count(conf jsonb DEFAULT NULL) RETURNS int AS $$\n SELECT get_setting('context_estimated_count', conf)::int;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_estimated_cost();\nCREATE OR REPLACE FUNCTION context_estimated_cost(conf jsonb DEFAULT NULL) RETURNS float AS $$\n SELECT get_setting('context_estimated_cost', conf)::float;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_stats_ttl();\nCREATE OR REPLACE FUNCTION context_stats_ttl(conf jsonb DEFAULT NULL) RETURNS interval AS $$\n SELECT get_setting('context_stats_ttl', conf)::interval;\n$$ LANGUAGE SQL;\n\n\nCREATE OR REPLACE FUNCTION notice(VARIADIC text[]) RETURNS boolean AS $$\nDECLARE\ndebug boolean := current_setting('pgstac.debug', true);\nBEGIN\n IF debug THEN\n RAISE NOTICE 'NOTICE FROM FUNC: % >>>>> %', concat_ws(' | ', $1), clock_timestamp();\n RETURN TRUE;\n END IF;\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION empty_arr(ANYARRAY) RETURNS BOOLEAN AS $$\nSELECT CASE\n WHEN $1 IS NULL THEN TRUE\n WHEN cardinality($1)<1 THEN TRUE\nELSE FALSE\nEND;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_map_ident(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_ident(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_map_literal(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_literal(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION estimated_count(_where text) RETURNS bigint AS $$\nDECLARE\nrec record;\nrows bigint;\nBEGIN\n FOR rec in EXECUTE format(\n $q$\n EXPLAIN SELECT 1 FROM items WHERE %s\n $q$,\n _where)\n LOOP\n rows := substring(rec.\"QUERY PLAN\" FROM ' rows=([[:digit:]]+)');\n EXIT WHEN rows IS NOT NULL;\n END LOOP;\n\n RETURN rows;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION array_reverse(anyarray) RETURNS anyarray AS $$\nSELECT ARRAY(\n SELECT $1[i]\n FROM generate_subscripts($1,1) AS s(i)\n ORDER BY i DESC\n);\n$$ LANGUAGE 'sql' STRICT IMMUTABLE;\n/* converts a jsonb text array to a pg text[] array */\nCREATE OR REPLACE FUNCTION textarr(_js jsonb)\n RETURNS text[] AS $$\n SELECT\n CASE jsonb_typeof(_js)\n WHEN 'array' THEN ARRAY(SELECT jsonb_array_elements_text(_js))\n ELSE ARRAY[_js->>0]\n END\n;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION jsonb_paths (IN jdata jsonb, OUT path text[], OUT value jsonb) RETURNS\nSETOF RECORD AS $$\nwith recursive extract_all as\n(\n select\n ARRAY[key]::text[] as path,\n value\n FROM jsonb_each(jdata)\nunion all\n select\n path || coalesce(obj_key, (arr_key- 1)::text),\n coalesce(obj_value, arr_value)\n from extract_all\n left join lateral\n jsonb_each(case jsonb_typeof(value) when 'object' then value end)\n as o(obj_key, obj_value)\n on jsonb_typeof(value) = 'object'\n left join lateral\n jsonb_array_elements(case jsonb_typeof(value) when 'array' then value end)\n with ordinality as a(arr_value, arr_key)\n on jsonb_typeof(value) = 'array'\n where obj_key is not null or arr_key is not null\n)\nselect *\nfrom extract_all;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION jsonb_obj_paths (IN jdata jsonb, OUT path text[], OUT value jsonb) RETURNS\nSETOF RECORD AS $$\nwith recursive extract_all as\n(\n select\n ARRAY[key]::text[] as path,\n value\n FROM jsonb_each(jdata)\nunion all\n select\n path || obj_key,\n obj_value\n from extract_all\n left join lateral\n jsonb_each(case jsonb_typeof(value) when 'object' then value end)\n as o(obj_key, obj_value)\n on jsonb_typeof(value) = 'object'\n where obj_key is not null\n)\nselect *\nfrom extract_all;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION jsonb_val_paths (IN jdata jsonb, OUT path text[], OUT value jsonb) RETURNS\nSETOF RECORD AS $$\nSELECT * FROM jsonb_obj_paths(jdata) WHERE jsonb_typeof(value) not in ('object','array');\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION path_includes(IN path text[], IN includes text[]) RETURNS BOOLEAN AS $$\nWITH t AS (SELECT unnest(includes) i)\nSELECT EXISTS (\n SELECT 1 FROM t WHERE path @> string_to_array(trim(i), '.')\n);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION path_excludes(IN path text[], IN excludes text[]) RETURNS BOOLEAN AS $$\nWITH t AS (SELECT unnest(excludes) e)\nSELECT NOT EXISTS (\n SELECT 1 FROM t WHERE path @> string_to_array(trim(e), '.')\n);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION jsonb_obj_paths_filtered (\n IN jdata jsonb,\n IN includes text[] DEFAULT ARRAY[]::text[],\n IN excludes text[] DEFAULT ARRAY[]::text[],\n OUT path text[],\n OUT value jsonb\n) RETURNS\nSETOF RECORD AS $$\nSELECT path, value\nFROM jsonb_obj_paths(jdata)\nWHERE\n CASE WHEN cardinality(includes) > 0 THEN path_includes(path, includes) ELSE TRUE END\n AND\n path_excludes(path, excludes)\n\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION filter_jsonb(\n IN jdata jsonb,\n IN includes text[] DEFAULT ARRAY[]::text[],\n IN excludes text[] DEFAULT ARRAY[]::text[]\n) RETURNS jsonb AS $$\nDECLARE\nrec RECORD;\noutj jsonb := '{}'::jsonb;\ncreated_paths text[] := '{}'::text[];\nBEGIN\n\nIF empty_arr(includes) AND empty_arr(excludes) THEN\nRAISE NOTICE 'no filter';\n RETURN jdata;\nEND IF;\nFOR rec in\nSELECT * FROM jsonb_obj_paths_filtered(jdata, includes, excludes)\nWHERE jsonb_typeof(value) != 'object'\nLOOP\n IF array_length(rec.path,1)>1 THEN\n FOR i IN 1..(array_length(rec.path,1)-1) LOOP\n IF NOT array_to_string(rec.path[1:i],'.') = ANY (created_paths) THEN\n outj := jsonb_set(outj, rec.path[1:i],'{}', true);\n created_paths := created_paths || array_to_string(rec.path[1:i],'.');\n END IF;\n END LOOP;\n END IF;\n outj := jsonb_set(outj, rec.path, rec.value, true);\n created_paths := created_paths || array_to_string(rec.path,'.');\nEND LOOP;\nRETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION flip_jsonb_array(j jsonb) RETURNS jsonb AS $$\nSELECT jsonb_agg(value) FROM (SELECT value FROM jsonb_array_elements(j) WITH ORDINALITY ORDER BY ordinality DESC) as t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n/* Functions to create an iterable of cursors over partitions. */\nCREATE OR REPLACE FUNCTION create_cursor(q text) RETURNS refcursor AS $$\nDECLARE\n curs refcursor;\nBEGIN\n OPEN curs FOR EXECUTE q;\n RETURN curs;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDROP FUNCTION IF EXISTS partition_queries;\nCREATE OR REPLACE FUNCTION partition_queries(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN partitions text[] DEFAULT '{items}'\n) RETURNS SETOF text AS $$\nDECLARE\n partition_query text;\n query text;\n p text;\n cursors refcursor;\n dstart timestamptz;\n dend timestamptz;\n step interval := '10 weeks'::interval;\nBEGIN\n\nIF _orderby ILIKE 'datetime d%' THEN\n partitions := partitions;\nELSIF _orderby ILIKE 'datetime a%' THEN\n partitions := array_reverse(partitions);\nELSE\n query := format($q$\n SELECT * FROM items\n WHERE %s\n ORDER BY %s\n $q$, _where, _orderby\n );\n\n RETURN NEXT query;\n RETURN;\nEND IF;\nRAISE NOTICE 'PARTITIONS ---> %',partitions;\nIF cardinality(partitions) > 0 THEN\n FOREACH p IN ARRAY partitions\n --EXECUTE partition_query\n LOOP\n query := format($q$\n SELECT * FROM %I\n WHERE %s\n ORDER BY %s\n $q$,\n p,\n _where,\n _orderby\n );\n RETURN NEXT query;\n END LOOP;\nEND IF;\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION partition_cursor(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC'\n) RETURNS SETOF refcursor AS $$\nDECLARE\n partition_query text;\n query text;\n p record;\n cursors refcursor;\nBEGIN\nFOR query IN SELECT * FROM partition_queries(_where, _orderby) LOOP\n RETURN NEXT create_cursor(query);\nEND LOOP;\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION partition_count(\n IN _where text DEFAULT 'TRUE'\n) RETURNS bigint AS $$\nDECLARE\n partition_query text;\n query text;\n p record;\n subtotal bigint;\n total bigint := 0;\nBEGIN\npartition_query := format($q$\n SELECT partition, tstzrange\n FROM items_partitions\n ORDER BY tstzrange DESC;\n$q$);\nRAISE NOTICE 'Partition Query: %', partition_query;\nFOR p IN\n EXECUTE partition_query\nLOOP\n query := format($q$\n SELECT count(*) FROM items\n WHERE datetime BETWEEN %L AND %L AND %s\n $q$, lower(p.tstzrange), upper(p.tstzrange), _where\n );\n RAISE NOTICE 'Query %', query;\n RAISE NOTICE 'Partition %, Count %, Total %',p.partition, subtotal, total;\n EXECUTE query INTO subtotal;\n total := subtotal + total;\nEND LOOP;\nRETURN total;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION drop_partition_constraints(IN partition text) RETURNS VOID AS $$\nDECLARE\n q text;\n end_datetime_constraint text := concat(partition, '_end_datetime_constraint');\n collections_constraint text := concat(partition, '_collections_constraint');\nBEGIN\n q := format($q$\n ALTER TABLE %I\n DROP CONSTRAINT IF EXISTS %I,\n DROP CONSTRAINT IF EXISTS %I;\n $q$,\n partition,\n end_datetime_constraint,\n collections_constraint\n );\n\n EXECUTE q;\n RETURN;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDROP FUNCTION IF EXISTS partition_checks;\nCREATE OR REPLACE FUNCTION partition_checks(\n IN partition text,\n OUT min_datetime timestamptz,\n OUT max_datetime timestamptz,\n OUT min_end_datetime timestamptz,\n OUT max_end_datetime timestamptz,\n OUT collections text[],\n OUT cnt bigint\n) RETURNS RECORD AS $$\nDECLARE\nq text;\nend_datetime_constraint text := concat(partition, '_end_datetime_constraint');\ncollections_constraint text := concat(partition, '_collections_constraint');\nBEGIN\nRAISE NOTICE 'CREATING CONSTRAINTS FOR %', partition;\nq := format($q$\n SELECT\n min(datetime),\n max(datetime),\n min(end_datetime),\n max(end_datetime),\n array_agg(DISTINCT collection_id),\n count(*)\n FROM %I;\n $q$,\n partition\n);\nEXECUTE q INTO min_datetime, max_datetime, min_end_datetime, max_end_datetime, collections, cnt;\nRAISE NOTICE '% % % % % % %', min_datetime, max_datetime, min_end_datetime, max_end_datetime, collections, cnt, ftime();\nIF cnt IS NULL or cnt = 0 THEN\n RAISE NOTICE 'Partition % is empty, removing...', partition;\n q := format($q$\n DROP TABLE IF EXISTS %I;\n $q$, partition\n );\n EXECUTE q;\n RETURN;\nEND IF;\nRAISE NOTICE 'Running Constraint DDL %', ftime();\nq := format($q$\n ALTER TABLE %I\n DROP CONSTRAINT IF EXISTS %I,\n ADD CONSTRAINT %I\n check((end_datetime >= %L) AND (end_datetime <= %L)) NOT VALID,\n DROP CONSTRAINT IF EXISTS %I,\n ADD CONSTRAINT %I\n check((collection_id = ANY(%L))) NOT VALID;\n $q$,\n partition,\n end_datetime_constraint,\n end_datetime_constraint,\n min_end_datetime,\n max_end_datetime,\n collections_constraint,\n collections_constraint,\n collections,\n partition\n);\nRAISE NOTICE 'q: %', q;\n\nEXECUTE q;\nRAISE NOTICE 'Returning %', ftime();\nRETURN;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION validate_constraints() RETURNS VOID AS $$\nDECLARE\nq text;\nBEGIN\nFOR q IN\n SELECT FORMAT(\n 'ALTER TABLE %I.%I.%I VALIDATE CONSTRAINT %I;',\n current_database(),\n nsp.nspname,\n cls.relname,\n con.conname\n )\n FROM pg_constraint AS con\n JOIN pg_class AS cls\n ON con.conrelid = cls.oid\n JOIN pg_namespace AS nsp\n ON cls.relnamespace = nsp.oid\n WHERE convalidated IS FALSE\n AND nsp.nspname = 'pgstac'\nLOOP\n EXECUTE q;\nEND LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n/* looks for a geometry in a stac item first from geometry and falling back to bbox */\nCREATE OR REPLACE FUNCTION stac_geom(value jsonb) RETURNS geometry AS $$\nSELECT\n CASE\n WHEN value->>'geometry' IS NOT NULL THEN\n ST_GeomFromGeoJSON(value->>'geometry')\n WHEN value->>'bbox' IS NOT NULL THEN\n ST_MakeEnvelope(\n (value->'bbox'->>0)::float,\n (value->'bbox'->>1)::float,\n (value->'bbox'->>2)::float,\n (value->'bbox'->>3)::float,\n 4326\n )\n ELSE NULL\n END as geometry\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION stac_datetime(value jsonb) RETURNS timestamptz AS $$\nSELECT COALESCE(\n (value->'properties'->>'datetime')::timestamptz,\n (value->'properties'->>'start_datetime')::timestamptz\n);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_end_datetime(value jsonb) RETURNS timestamptz AS $$\nSELECT COALESCE(\n (value->'properties'->>'datetime')::timestamptz,\n (value->'properties'->>'end_datetime')::timestamptz\n);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\n\nCREATE OR REPLACE FUNCTION stac_daterange(value jsonb) RETURNS tstzrange AS $$\nSELECT tstzrange(stac_datetime(value),stac_end_datetime(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE TABLE IF NOT EXISTS collections (\n id VARCHAR GENERATED ALWAYS AS (content->>'id') STORED PRIMARY KEY,\n content JSONB\n);\n\nCREATE OR REPLACE FUNCTION create_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection(data jsonb) RETURNS VOID AS $$\nDECLARE\nout collections%ROWTYPE;\nBEGIN\n UPDATE collections SET content=data WHERE id = data->>'id' RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ON CONFLICT (id) DO\n UPDATE\n SET content=EXCLUDED.content\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION delete_collection(_id text) RETURNS VOID AS $$\nDECLARE\nout collections%ROWTYPE;\nBEGIN\n DELETE FROM collections WHERE id = _id RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION get_collection(id text) RETURNS jsonb AS $$\nSELECT content FROM collections\nWHERE id=$1\n;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION all_collections() RETURNS jsonb AS $$\nSELECT jsonb_agg(content) FROM collections;\n;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE TABLE items (\n id text NOT NULL,\n geometry geometry NOT NULL,\n collection_id text NOT NULL,\n datetime timestamptz NOT NULL,\n end_datetime timestamptz NOT NULL,\n properties jsonb NOT NULL,\n content JSONB NOT NULL\n)\nPARTITION BY RANGE (datetime)\n;\n\nCREATE OR REPLACE FUNCTION properties_idx (IN content jsonb) RETURNS jsonb AS $$\n with recursive extract_all as\n (\n select\n ARRAY[key]::text[] as path,\n ARRAY[key]::text[] as fullpath,\n value\n FROM jsonb_each(content->'properties')\n union all\n select\n CASE WHEN obj_key IS NOT NULL THEN path || obj_key ELSE path END,\n path || coalesce(obj_key, (arr_key- 1)::text),\n coalesce(obj_value, arr_value)\n from extract_all\n left join lateral\n jsonb_each(case jsonb_typeof(value) when 'object' then value end)\n as o(obj_key, obj_value)\n on jsonb_typeof(value) = 'object'\n left join lateral\n jsonb_array_elements(case jsonb_typeof(value) when 'array' then value end)\n with ordinality as a(arr_value, arr_key)\n on jsonb_typeof(value) = 'array'\n where obj_key is not null or arr_key is not null\n )\n , paths AS (\n select\n array_to_string(path, '.') as path,\n value\n FROM extract_all\n WHERE\n jsonb_typeof(value) NOT IN ('array','object')\n ), grouped AS (\n SELECT path, jsonb_agg(distinct value) vals FROM paths group by path\n ) SELECT coalesce(jsonb_object_agg(path, CASE WHEN jsonb_array_length(vals)=1 THEN vals->0 ELSE vals END) - '{datetime}'::text[], '{}'::jsonb) FROM grouped\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET JIT TO OFF;\n\nCREATE INDEX \"datetime_idx\" ON items (datetime);\nCREATE INDEX \"end_datetime_idx\" ON items (end_datetime);\nCREATE INDEX \"properties_idx\" ON items USING GIN (properties jsonb_path_ops);\nCREATE INDEX \"collection_idx\" ON items (collection_id);\nCREATE INDEX \"geometry_idx\" ON items USING GIST (geometry);\nCREATE UNIQUE INDEX \"items_id_datetime_idx\" ON items (datetime, id);\n\nALTER TABLE items ADD CONSTRAINT items_collections_fk FOREIGN KEY (collection_id) REFERENCES collections(id) ON DELETE CASCADE DEFERRABLE;\n\nCREATE OR REPLACE FUNCTION analyze_empty_partitions() RETURNS VOID AS $$\nDECLARE\n p text;\nBEGIN\n FOR p IN SELECT partition FROM all_items_partitions WHERE est_cnt = 0 LOOP\n EXECUTE format('ANALYZE %I;', p);\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION items_partition_name(timestamptz) RETURNS text AS $$\n SELECT to_char($1, '\"items_p\"IYYY\"w\"IW');\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION items_partition_exists(text) RETURNS boolean AS $$\n SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_class WHERE relname=$1);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION items_partition_exists(timestamptz) RETURNS boolean AS $$\n SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_class WHERE relname=items_partition_name($1));\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION items_partition_create_worker(partition text, partition_start timestamptz, partition_end timestamptz) RETURNS VOID AS $$\nDECLARE\n err_context text;\nBEGIN\n EXECUTE format(\n $f$\n CREATE TABLE IF NOT EXISTS %1$I PARTITION OF items\n FOR VALUES FROM (%2$L) TO (%3$L);\n CREATE UNIQUE INDEX IF NOT EXISTS %4$I ON %1$I (id);\n $f$,\n partition,\n partition_start,\n partition_end,\n concat(partition, '_id_pk')\n );\nEXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', partition;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH to pgstac, public;\n\nCREATE OR REPLACE FUNCTION items_partition_create(ts timestamptz) RETURNS text AS $$\nDECLARE\n partition text := items_partition_name(ts);\n partition_start timestamptz;\n partition_end timestamptz;\nBEGIN\n IF items_partition_exists(partition) THEN\n RETURN partition;\n END IF;\n partition_start := date_trunc('week', ts);\n partition_end := partition_start + '1 week'::interval;\n PERFORM items_partition_create_worker(partition, partition_start, partition_end);\n RAISE NOTICE 'partition: %', partition;\n RETURN partition;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION items_partition_create(st timestamptz, et timestamptz) RETURNS SETOF text AS $$\nWITH t AS (\n SELECT\n generate_series(\n date_trunc('week',st),\n date_trunc('week', et),\n '1 week'::interval\n ) w\n)\nSELECT items_partition_create(w) FROM t;\n$$ LANGUAGE SQL;\n\n\nCREATE UNLOGGED TABLE items_staging (\n content JSONB NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION items_staging_insert_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p record;\n _partitions text[];\nBEGIN\n CREATE TEMP TABLE new_partitions ON COMMIT DROP AS\n SELECT\n items_partition_name(stac_datetime(content)) as partition,\n date_trunc('week', min(stac_datetime(content))) as partition_start\n FROM newdata\n GROUP BY 1;\n\n -- set statslastupdated in cache to be old enough cache always regenerated\n\n SELECT array_agg(partition) INTO _partitions FROM new_partitions;\n UPDATE search_wheres\n SET\n statslastupdated = NULL\n WHERE _where IN (\n SELECT _where FROM search_wheres sw WHERE sw.partitions && _partitions\n FOR UPDATE SKIP LOCKED\n )\n ;\n FOR p IN SELECT new_partitions.partition, new_partitions.partition_start, new_partitions.partition_start + '1 week'::interval as partition_end FROM new_partitions\n LOOP\n RAISE NOTICE 'Getting partition % ready.', p.partition;\n IF NOT items_partition_exists(p.partition) THEN\n RAISE NOTICE 'Creating partition %.', p.partition;\n PERFORM items_partition_create_worker(p.partition, p.partition_start, p.partition_end);\n END IF;\n PERFORM drop_partition_constraints(p.partition);\n END LOOP;\n INSERT INTO items (id, geometry, collection_id, datetime, end_datetime, properties, content)\n SELECT\n content->>'id',\n stac_geom(content),\n content->>'collection',\n stac_datetime(content),\n stac_end_datetime(content),\n properties_idx(content),\n content\n FROM newdata\n ;\n DELETE FROM items_staging;\n\n\n FOR p IN SELECT new_partitions.partition FROM new_partitions\n LOOP\n RAISE NOTICE 'Setting constraints for partition %.', p.partition;\n PERFORM partition_checks(p.partition);\n END LOOP;\n DROP TABLE IF EXISTS new_partitions;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER items_staging_insert_trigger AFTER INSERT ON items_staging REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_insert_triggerfunc();\n\n\nCREATE UNLOGGED TABLE items_staging_ignore (\n content JSONB NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION items_staging_ignore_insert_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p record;\n _partitions text[];\nBEGIN\n CREATE TEMP TABLE new_partitions ON COMMIT DROP AS\n SELECT\n items_partition_name(stac_datetime(content)) as partition,\n date_trunc('week', min(stac_datetime(content))) as partition_start\n FROM newdata\n GROUP BY 1;\n\n -- set statslastupdated in cache to be old enough cache always regenerated\n\n SELECT array_agg(partition) INTO _partitions FROM new_partitions;\n UPDATE search_wheres\n SET\n statslastupdated = NULL\n WHERE _where IN (\n SELECT _where FROM search_wheres sw WHERE sw.partitions && _partitions\n FOR UPDATE SKIP LOCKED\n )\n ;\n\n FOR p IN SELECT new_partitions.partition, new_partitions.partition_start, new_partitions.partition_start + '1 week'::interval as partition_end FROM new_partitions\n LOOP\n RAISE NOTICE 'Getting partition % ready.', p.partition;\n IF NOT items_partition_exists(p.partition) THEN\n RAISE NOTICE 'Creating partition %.', p.partition;\n PERFORM items_partition_create_worker(p.partition, p.partition_start, p.partition_end);\n END IF;\n PERFORM drop_partition_constraints(p.partition);\n END LOOP;\n\n INSERT INTO items (id, geometry, collection_id, datetime, end_datetime, properties, content)\n SELECT\n content->>'id',\n stac_geom(content),\n content->>'collection',\n stac_datetime(content),\n stac_end_datetime(content),\n properties_idx(content),\n content\n FROM newdata\n ON CONFLICT (datetime, id) DO NOTHING\n ;\n DELETE FROM items_staging;\n\n\n FOR p IN SELECT new_partitions.partition FROM new_partitions\n LOOP\n RAISE NOTICE 'Setting constraints for partition %.', p.partition;\n PERFORM partition_checks(p.partition);\n END LOOP;\n DROP TABLE IF EXISTS new_partitions;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER items_staging_ignore_insert_trigger AFTER INSERT ON items_staging_ignore REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_ignore_insert_triggerfunc();\n\nCREATE UNLOGGED TABLE items_staging_upsert (\n content JSONB NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION items_staging_upsert_insert_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p record;\n _partitions text[];\nBEGIN\n CREATE TEMP TABLE new_partitions ON COMMIT DROP AS\n SELECT\n items_partition_name(stac_datetime(content)) as partition,\n date_trunc('week', min(stac_datetime(content))) as partition_start\n FROM newdata\n GROUP BY 1;\n\n -- set statslastupdated in cache to be old enough cache always regenerated\n\n SELECT array_agg(partition) INTO _partitions FROM new_partitions;\n UPDATE search_wheres\n SET\n statslastupdated = NULL\n WHERE _where IN (\n SELECT _where FROM search_wheres sw WHERE sw.partitions && _partitions\n FOR UPDATE SKIP LOCKED\n )\n ;\n\n FOR p IN SELECT new_partitions.partition, new_partitions.partition_start, new_partitions.partition_start + '1 week'::interval as partition_end FROM new_partitions\n LOOP\n RAISE NOTICE 'Getting partition % ready.', p.partition;\n IF NOT items_partition_exists(p.partition) THEN\n RAISE NOTICE 'Creating partition %.', p.partition;\n PERFORM items_partition_create_worker(p.partition, p.partition_start, p.partition_end);\n END IF;\n PERFORM drop_partition_constraints(p.partition);\n END LOOP;\n\n INSERT INTO items (id, geometry, collection_id, datetime, end_datetime, properties, content)\n SELECT\n content->>'id',\n stac_geom(content),\n content->>'collection',\n stac_datetime(content),\n stac_end_datetime(content),\n properties_idx(content),\n content\n FROM newdata\n ON CONFLICT (datetime, id) DO UPDATE SET\n content = EXCLUDED.content\n WHERE items.content IS DISTINCT FROM EXCLUDED.content\n ;\n DELETE FROM items_staging;\n\n\n FOR p IN SELECT new_partitions.partition FROM new_partitions\n LOOP\n RAISE NOTICE 'Setting constraints for partition %.', p.partition;\n PERFORM partition_checks(p.partition);\n END LOOP;\n DROP TABLE IF EXISTS new_partitions;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER items_staging_upsert_insert_trigger AFTER INSERT ON items_staging_upsert REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_upsert_insert_triggerfunc();\n\nCREATE OR REPLACE FUNCTION items_update_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\nBEGIN\n NEW.id := NEW.content->>'id';\n NEW.datetime := stac_datetime(NEW.content);\n NEW.end_datetime := stac_end_datetime(NEW.content);\n NEW.collection_id := NEW.content->>'collection';\n NEW.geometry := stac_geom(NEW.content);\n NEW.properties := properties_idx(NEW.content);\n IF TG_OP = 'UPDATE' AND NEW IS NOT DISTINCT FROM OLD THEN\n RETURN NULL;\n END IF;\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER items_update_trigger BEFORE UPDATE ON items\n FOR EACH ROW EXECUTE PROCEDURE items_update_triggerfunc();\n\n/*\nView to get a table of available items partitions\nwith date ranges\n*/\nDROP VIEW IF EXISTS all_items_partitions CASCADE;\nCREATE VIEW all_items_partitions AS\nWITH base AS\n(SELECT\n c.oid::pg_catalog.regclass::text as partition,\n pg_catalog.pg_get_expr(c.relpartbound, c.oid) as _constraint,\n regexp_matches(\n pg_catalog.pg_get_expr(c.relpartbound, c.oid),\n E'\\\\(''\\([0-9 :+-]*\\)''\\\\).*\\\\(''\\([0-9 :+-]*\\)''\\\\)'\n ) as t,\n reltuples::bigint as est_cnt\nFROM pg_catalog.pg_class c, pg_catalog.pg_inherits i\nWHERE c.oid = i.inhrelid AND i.inhparent = 'items'::regclass)\nSELECT partition, tstzrange(\n t[1]::timestamptz,\n t[2]::timestamptz\n), t[1]::timestamptz as pstart,\n t[2]::timestamptz as pend, est_cnt\nFROM base\nORDER BY 2 desc;\n\nCREATE OR REPLACE VIEW items_partitions AS\nSELECT * FROM all_items_partitions WHERE est_cnt>0;\n\nCREATE OR REPLACE FUNCTION item_by_id(_id text) RETURNS items AS\n$$\nDECLARE\n i items%ROWTYPE;\nBEGIN\n SELECT * INTO i FROM items WHERE id=_id LIMIT 1;\n RETURN i;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION get_item(_id text) RETURNS jsonb AS $$\n SELECT content FROM items WHERE id=_id;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION delete_item(_id text) RETURNS VOID AS $$\nDECLARE\nout items%ROWTYPE;\nBEGIN\n DELETE FROM items WHERE id = _id RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION create_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION update_item(data jsonb) RETURNS VOID AS $$\nDECLARE\n out items%ROWTYPE;\nBEGIN\n UPDATE items SET content=data WHERE id = data->>'id' RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION create_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION collection_bbox(id text) RETURNS jsonb AS $$\nSELECT (replace(replace(replace(st_extent(geometry)::text,'BOX(','[['),')',']]'),' ',','))::jsonb\nFROM items WHERE collection_id=$1;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION collection_temporal_extent(id text) RETURNS jsonb AS $$\nSELECT to_jsonb(array[array[min(datetime)::text, max(datetime)::text]])\nFROM items WHERE collection_id=$1;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection_extents() RETURNS VOID AS $$\nUPDATE collections SET\n content = content ||\n jsonb_build_object(\n 'extent', jsonb_build_object(\n 'spatial', jsonb_build_object(\n 'bbox', collection_bbox(collections.id)\n ),\n 'temporal', jsonb_build_object(\n 'interval', collection_temporal_extent(collections.id)\n )\n )\n )\n;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION items_path(\n IN dotpath text,\n OUT field text,\n OUT path text,\n OUT path_txt text,\n OUT jsonpath text,\n OUT eq text\n) RETURNS RECORD AS $$\nDECLARE\npath_elements text[];\nlast_element text;\nBEGIN\ndotpath := replace(trim(dotpath), 'properties.', '');\n\nIF dotpath = '' THEN\n RETURN;\nEND IF;\n\npath_elements := string_to_array(dotpath, '.');\njsonpath := NULL;\n\nIF path_elements[1] IN ('id','geometry','datetime') THEN\n field := path_elements[1];\n path_elements := path_elements[2:];\nELSIF path_elements[1] = 'collection' THEN\n field := 'collection_id';\n path_elements := path_elements[2:];\nELSIF path_elements[1] IN ('links', 'assets', 'stac_version', 'stac_extensions') THEN\n field := 'content';\nELSE\n field := 'content';\n path_elements := '{properties}'::text[] || path_elements;\nEND IF;\nIF cardinality(path_elements)<1 THEN\n path := field;\n path_txt := field;\n jsonpath := '$';\n eq := NULL;\n RETURN;\nEND IF;\n\n\nlast_element := path_elements[cardinality(path_elements)];\npath_elements := path_elements[1:cardinality(path_elements)-1];\njsonpath := concat(array_to_string('{$}'::text[] || array_map_ident(path_elements), '.'), '.', quote_ident(last_element));\npath_elements := array_map_literal(path_elements);\npath := format($F$ properties->%s $F$, quote_literal(dotpath));\npath_txt := format($F$ properties->>%s $F$, quote_literal(dotpath));\neq := format($F$ properties @? '$.%s[*] ? (@ == %%s) '$F$, quote_ident(dotpath));\n\nRAISE NOTICE 'ITEMS PATH -- % % % % %', field, path, path_txt, jsonpath, eq;\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION parse_dtrange(IN _indate jsonb, OUT _tstzrange tstzrange) AS $$\nWITH t AS (\n SELECT CASE\n WHEN _indate ? 'timestamp' THEN\n ARRAY[_indate->>'timestamp', 'infinity']\n WHEN _indate ? 'interval' THEN\n textarr(_indate->'interval')\n WHEN jsonb_typeof(_indate) = 'array' THEN\n textarr(_indate)\n ELSE\n regexp_split_to_array(\n btrim(_indate::text,'\"'),\n '/'\n )\n END AS arr\n)\n, t1 AS (\n SELECT\n CASE\n WHEN array_upper(arr,1) = 1 OR arr[1] = '..' OR arr[1] IS NULL THEN '-infinity'::timestamptz\n ELSE arr[1]::timestamptz\n END AS st,\n CASE\n WHEN array_upper(arr,1) = 1 THEN arr[1]::timestamptz\n WHEN arr[2] = '..' OR arr[2] IS NULL THEN 'infinity'::timestamptz\n ELSE arr[2]::timestamptz\n END AS et\n FROM t\n)\nSELECT\n tstzrange(st,et)\nFROM t1;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION bbox_geom(_bbox jsonb) RETURNS geometry AS $$\nSELECT CASE jsonb_array_length(_bbox)\n WHEN 4 THEN\n ST_SetSRID(ST_MakeEnvelope(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float,\n (_bbox->>3)::float\n ),4326)\n WHEN 6 THEN\n ST_SetSRID(ST_3DMakeBox(\n ST_MakePoint(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float\n ),\n ST_MakePoint(\n (_bbox->>3)::float,\n (_bbox->>4)::float,\n (_bbox->>5)::float\n )\n ),4326)\n ELSE null END;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION cql_and_append(existing jsonb, newfilters jsonb) RETURNS jsonb AS $$\nSELECT CASE WHEN existing ? 'filter' AND newfilters IS NOT NULL THEN\n jsonb_build_object(\n 'and',\n jsonb_build_array(\n existing->'filter',\n newfilters\n )\n )\nELSE\n newfilters\nEND;\n$$ LANGUAGE SQL;\n\n\n-- ADDs base filters (ids, collections, datetime, bbox, intersects) that are\n-- added outside of the filter/query in the stac request\nCREATE OR REPLACE FUNCTION add_filters_to_cql(j jsonb) RETURNS jsonb AS $$\nDECLARE\nnewprop jsonb;\nnewprops jsonb := '[]'::jsonb;\nBEGIN\nIF j ? 'ids' THEN\n newprop := jsonb_build_object(\n 'in',\n jsonb_build_array(\n '{\"property\":\"id\"}'::jsonb,\n j->'ids'\n )\n );\n newprops := jsonb_insert(newprops, '{1}', newprop);\nEND IF;\nIF j ? 'collections' THEN\n newprop := jsonb_build_object(\n 'in',\n jsonb_build_array(\n '{\"property\":\"collection\"}'::jsonb,\n j->'collections'\n )\n );\n newprops := jsonb_insert(newprops, '{1}', newprop);\nEND IF;\n\nIF j ? 'datetime' THEN\n newprop := format(\n '{\"anyinteracts\":[{\"property\":\"datetime\"}, %s]}',\n j->'datetime'\n );\n newprops := jsonb_insert(newprops, '{1}', newprop);\nEND IF;\n\nIF j ? 'bbox' THEN\n newprop := format(\n '{\"intersects\":[{\"property\":\"geometry\"}, %s]}',\n j->'bbox'\n );\n newprops := jsonb_insert(newprops, '{1}', newprop);\nEND IF;\n\nIF j ? 'intersects' THEN\n newprop := format(\n '{\"intersects\":[{\"property\":\"geometry\"}, %s]}',\n j->'intersects'\n );\n newprops := jsonb_insert(newprops, '{1}', newprop);\nEND IF;\n\nRAISE NOTICE 'newprops: %', newprops;\n\nIF newprops IS NOT NULL AND jsonb_array_length(newprops) > 0 THEN\n return jsonb_set(\n j,\n '{filter}',\n cql_and_append(j, jsonb_build_object('and', newprops))\n ) - '{ids,collections,datetime,bbox,intersects}'::text[];\nEND IF;\n\nreturn j;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION query_to_cqlfilter(j jsonb) RETURNS jsonb AS $$\n-- Translates anything passed in through the deprecated \"query\" into equivalent CQL\nWITH t AS (\n SELECT key as property, value as ops\n FROM jsonb_each(j->'query')\n), t2 AS (\n SELECT property, (jsonb_each(ops)).*\n FROM t WHERE jsonb_typeof(ops) = 'object'\n UNION ALL\n SELECT property, 'eq', ops\n FROM t WHERE jsonb_typeof(ops) != 'object'\n), t3 AS (\nSELECT\n jsonb_strip_nulls(jsonb_build_object(\n 'and',\n jsonb_agg(\n jsonb_build_object(\n key,\n jsonb_build_array(\n jsonb_build_object('property',property),\n value\n )\n )\n )\n )) as qcql FROM t2\n)\nSELECT\n CASE WHEN qcql IS NOT NULL THEN\n jsonb_set(j, '{filter}', cql_and_append(j, qcql)) - 'query'\n ELSE j\n END\nFROM t3\n;\n$$ LANGUAGE SQL;\n\n\nCREATE OR REPLACE FUNCTION base_stac_query(j jsonb) RETURNS text AS $$\nDECLARE\n_where text := '';\ndtrange tstzrange;\ngeom geometry;\nBEGIN\n\n IF j ? 'ids' THEN\n _where := format('%s AND id = ANY (%L) ', _where, textarr(j->'ids'));\n END IF;\n IF j ? 'collections' THEN\n _where := format('%s AND collection_id = ANY (%L) ', _where, textarr(j->'collections'));\n END IF;\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n _where := format('%s AND datetime <= %L::timestamptz AND end_datetime >= %L::timestamptz ',\n _where,\n upper(dtrange),\n lower(dtrange)\n );\n END IF;\n\n IF j ? 'bbox' THEN\n geom := bbox_geom(j->'bbox');\n _where := format('%s AND geometry && %L',\n _where,\n geom\n );\n END IF;\n\n IF j ? 'intersects' THEN\n geom := st_fromgeojson(j->>'intersects');\n _where := format('%s AND st_intersects(geometry, %L)',\n _where,\n geom\n );\n END IF;\n\n RETURN _where;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION temporal_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\nll text := 'datetime';\nlh text := 'end_datetime';\nrrange tstzrange;\nrl text;\nrh text;\noutq text;\nBEGIN\nrrange := parse_dtrange(args->1);\nRAISE NOTICE 'Constructing temporal query OP: %, ARGS: %, RRANGE: %', op, args, rrange;\nop := lower(op);\nrl := format('%L::timestamptz', lower(rrange));\nrh := format('%L::timestamptz', upper(rrange));\noutq := CASE op\n WHEN 't_before' THEN 'lh < rl'\n WHEN 't_after' THEN 'll > rh'\n WHEN 't_meets' THEN 'lh = rl'\n WHEN 't_metby' THEN 'll = rh'\n WHEN 't_overlaps' THEN 'll < rl AND rl < lh < rh'\n WHEN 't_overlappedby' THEN 'rl < ll < rh AND lh > rh'\n WHEN 't_starts' THEN 'll = rl AND lh < rh'\n WHEN 't_startedby' THEN 'll = rl AND lh > rh'\n WHEN 't_during' THEN 'll > rl AND lh < rh'\n WHEN 't_contains' THEN 'll < rl AND lh > rh'\n WHEN 't_finishes' THEN 'll > rl AND lh = rh'\n WHEN 't_finishedby' THEN 'll < rl AND lh = rh'\n WHEN 't_equals' THEN 'll = rl AND lh = rh'\n WHEN 't_disjoint' THEN 'NOT (ll <= rh AND lh >= rl)'\n WHEN 't_intersects' THEN 'll <= rh AND lh >= rl'\n WHEN 'anyinteracts' THEN 'll <= rh AND lh >= rl'\nEND;\noutq := regexp_replace(outq, '\\mll\\M', ll);\noutq := regexp_replace(outq, '\\mlh\\M', lh);\noutq := regexp_replace(outq, '\\mrl\\M', rl);\noutq := regexp_replace(outq, '\\mrh\\M', rh);\noutq := format('(%s)', outq);\nRETURN outq;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION spatial_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\ngeom text;\nj jsonb := args->1;\nBEGIN\nop := lower(op);\nRAISE NOTICE 'Constructing spatial query OP: %, ARGS: %', op, args;\nIF op NOT IN ('s_equals','s_disjoint','s_touches','s_within','s_overlaps','s_crosses','s_intersects','intersects','s_contains') THEN\n RAISE EXCEPTION 'Spatial Operator % Not Supported', op;\nEND IF;\nop := regexp_replace(op, '^s_', 'st_');\nIF op = 'intersects' THEN\n op := 'st_intersects';\nEND IF;\n-- Convert geometry to WKB string\nIF j ? 'type' AND j ? 'coordinates' THEN\n geom := st_geomfromgeojson(j)::text;\nELSIF jsonb_typeof(j) = 'array' THEN\n geom := bbox_geom(j)::text;\nEND IF;\n\nRETURN format('%s(geometry, %L::geometry)', op, geom);\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n/* cql_query_op -- Parses a CQL query operation, recursing when necessary\n IN jsonb -- a subelement from a valid stac query\n IN text -- the operator being used on elements passed in\n RETURNS a SQL fragment to be used in a WHERE clause\n*/\nCREATE OR REPLACE FUNCTION cql_query_op(j jsonb, _op text DEFAULT NULL) RETURNS text AS $$\nDECLARE\njtype text := jsonb_typeof(j);\nop text := lower(_op);\nops jsonb :=\n '{\n \"eq\": \"%s = %s\",\n \"lt\": \"%s < %s\",\n \"lte\": \"%s <= %s\",\n \"gt\": \"%s > %s\",\n \"gte\": \"%s >= %s\",\n \"le\": \"%s <= %s\",\n \"ge\": \"%s >= %s\",\n \"=\": \"%s = %s\",\n \"<\": \"%s < %s\",\n \"<=\": \"%s <= %s\",\n \">\": \"%s > %s\",\n \">=\": \"%s >= %s\",\n \"like\": \"%s LIKE %s\",\n \"ilike\": \"%s ILIKE %s\",\n \"+\": \"%s + %s\",\n \"-\": \"%s - %s\",\n \"*\": \"%s * %s\",\n \"/\": \"%s / %s\",\n \"in\": \"%s = ANY (%s)\",\n \"not\": \"NOT (%s)\",\n \"between\": \"%s BETWEEN %s AND %s\",\n \"lower\":\" lower(%s)\",\n \"upper\":\" upper(%s)\",\n \"isnull\": \"%s IS NULL\"\n }'::jsonb;\nret text;\nargs text[] := NULL;\n\nBEGIN\nRAISE NOTICE 'j: %, op: %, jtype: %', j, op, jtype;\n\n-- for in, convert value, list to array syntax to match other ops\nIF op = 'in' and j ? 'value' and j ? 'list' THEN\n j := jsonb_build_array( j->'value', j->'list');\n jtype := 'array';\n RAISE NOTICE 'IN: j: %, jtype: %', j, jtype;\nEND IF;\n\nIF op = 'between' and j ? 'value' and j ? 'lower' and j ? 'upper' THEN\n j := jsonb_build_array( j->'value', j->'lower', j->'upper');\n jtype := 'array';\n RAISE NOTICE 'BETWEEN: j: %, jtype: %', j, jtype;\nEND IF;\n\nIF op = 'not' AND jtype = 'object' THEN\n j := jsonb_build_array( j );\n jtype := 'array';\n RAISE NOTICE 'NOT: j: %, jtype: %', j, jtype;\nEND IF;\n\n-- Set Lower Case on Both Arguments When Case Insensitive Flag Set\nIF op in ('eq','lt','lte','gt','gte','like') AND jsonb_typeof(j->2) = 'boolean' THEN\n IF (j->>2)::boolean THEN\n RETURN format(concat('(',ops->>op,')'), cql_query_op(jsonb_build_array(j->0), 'lower'), cql_query_op(jsonb_build_array(j->1), 'lower'));\n END IF;\nEND IF;\n\n-- Special Case when comparing a property in a jsonb field to a string or number using eq\n-- Allows to leverage GIN index on jsonb fields\nIF op = 'eq' THEN\n IF j->0 ? 'property'\n AND jsonb_typeof(j->1) IN ('number','string')\n AND (items_path(j->0->>'property')).eq IS NOT NULL\n THEN\n RETURN format((items_path(j->0->>'property')).eq, j->1);\n END IF;\nEND IF;\n\n\n\nIF op ilike 't_%' or op = 'anyinteracts' THEN\n RETURN temporal_op_query(op, j);\nEND IF;\n\nIF op ilike 's_%' or op = 'intersects' THEN\n RETURN spatial_op_query(op, j);\nEND IF;\n\n\nIF jtype = 'object' THEN\n RAISE NOTICE 'parsing object';\n IF j ? 'property' THEN\n -- Convert the property to be used as an identifier\n return (items_path(j->>'property')).path_txt;\n ELSIF _op IS NULL THEN\n -- Iterate to convert elements in an object where the operator has not been set\n -- Combining with AND\n SELECT\n array_to_string(array_agg(cql_query_op(e.value, e.key)), ' AND ')\n INTO ret\n FROM jsonb_each(j) e;\n RETURN ret;\n END IF;\nEND IF;\n\nIF jtype = 'string' THEN\n RETURN quote_literal(j->>0);\nEND IF;\n\nIF jtype ='number' THEN\n RETURN (j->>0)::numeric;\nEND IF;\n\nIF jtype = 'array' AND op IS NULL THEN\n RAISE NOTICE 'Parsing array into array arg. j: %', j;\n SELECT format($f$ '{%s}'::text[] $f$, string_agg(e,',')) INTO ret FROM jsonb_array_elements_text(j) e;\n RETURN ret;\nEND IF;\n\n\n-- If the type of the passed json is an array\n-- Calculate the arguments that will be passed to functions/operators\nIF jtype = 'array' THEN\n RAISE NOTICE 'Parsing array into args. j: %', j;\n -- If any argument is numeric, cast any text arguments to numeric\n IF j @? '$[*] ? (@.type() == \"number\")' THEN\n SELECT INTO args\n array_agg(concat('(',cql_query_op(e),')::numeric'))\n FROM jsonb_array_elements(j) e;\n ELSE\n SELECT INTO args\n array_agg(cql_query_op(e))\n FROM jsonb_array_elements(j) e;\n END IF;\n --RETURN args;\nEND IF;\nRAISE NOTICE 'ARGS after array cleaning: %', args;\n\nIF op IS NULL THEN\n RETURN args::text[];\nEND IF;\n\nIF args IS NULL OR cardinality(args) < 1 THEN\n RAISE NOTICE 'No Args';\n RETURN '';\nEND IF;\n\nIF op IN ('and','or') THEN\n SELECT\n CONCAT(\n '(',\n array_to_string(args, UPPER(CONCAT(' ',op,' '))),\n ')'\n ) INTO ret\n FROM jsonb_array_elements(j) e;\n RETURN ret;\nEND IF;\n\n-- If the op is in the ops json then run using the template in the json\nIF ops ? op THEN\n RAISE NOTICE 'ARGS: % MAPPED: %',args, array_map_literal(args);\n\n RETURN format(concat('(',ops->>op,')'), VARIADIC args);\nEND IF;\n\nRETURN j->>0;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n/* cql_query_op -- Parses a CQL query operation, recursing when necessary\n IN jsonb -- a subelement from a valid stac query\n IN text -- the operator being used on elements passed in\n RETURNS a SQL fragment to be used in a WHERE clause\n*/\nDROP FUNCTION IF EXISTS cql2_query;\nCREATE OR REPLACE FUNCTION cql2_query(j jsonb, recursion int DEFAULT 0) RETURNS text AS $$\nDECLARE\nargs jsonb := j->'args';\njtype text := jsonb_typeof(j->'args');\nop text := lower(j->>'op');\narg jsonb;\nargtext text;\nargstext text[] := '{}'::text[];\ninobj jsonb;\n_numeric text := '';\nops jsonb :=\n '{\n \"eq\": \"%s = %s\",\n \"lt\": \"%s < %s\",\n \"lte\": \"%s <= %s\",\n \"gt\": \"%s > %s\",\n \"gte\": \"%s >= %s\",\n \"le\": \"%s <= %s\",\n \"ge\": \"%s >= %s\",\n \"=\": \"%s = %s\",\n \"<\": \"%s < %s\",\n \"<=\": \"%s <= %s\",\n \">\": \"%s > %s\",\n \">=\": \"%s >= %s\",\n \"like\": \"%s LIKE %s\",\n \"ilike\": \"%s ILIKE %s\",\n \"+\": \"%s + %s\",\n \"-\": \"%s - %s\",\n \"*\": \"%s * %s\",\n \"/\": \"%s / %s\",\n \"in\": \"%s = ANY (%s)\",\n \"not\": \"NOT (%s)\",\n \"between\": \"%s BETWEEN (%2$s)[1] AND (%2$s)[2]\",\n \"lower\":\" lower(%s)\",\n \"upper\":\" upper(%s)\",\n \"isnull\": \"%s IS NULL\"\n }'::jsonb;\nret text;\n\nBEGIN\nRAISE NOTICE 'j: %s', j;\nIF j ? 'filter' THEN\n RETURN cql2_query(j->'filter');\nEND IF;\n\nIF j ? 'upper' THEN\nRAISE NOTICE 'upper %s',jsonb_build_object(\n 'op', 'upper',\n 'args', jsonb_build_array( j-> 'upper')\n ) ;\n RETURN cql2_query(\n jsonb_build_object(\n 'op', 'upper',\n 'args', jsonb_build_array( j-> 'upper')\n )\n );\nEND IF;\n\nIF j ? 'lower' THEN\n RETURN cql2_query(\n jsonb_build_object(\n 'op', 'lower',\n 'args', jsonb_build_array( j-> 'lower')\n )\n );\nEND IF;\n\nIF j ? 'args' AND jsonb_typeof(args) != 'array' THEN\n args := jsonb_build_array(args);\nEND IF;\n-- END Cases where no further nesting is expected\nIF j ? 'op' THEN\n -- Special case to use JSONB index for equality\n IF op IN ('eq', '=')\n AND args->0 ? 'property'\n AND jsonb_typeof(args->1) IN ('number', 'string')\n AND (items_path(args->0->>'property')).eq IS NOT NULL\n THEN\n RETURN format((items_path(args->0->>'property')).eq, args->1);\n END IF;\n\n -- Temporal Query\n IF op ilike 't_%' or op = 'anyinteracts' THEN\n RETURN temporal_op_query(op, args);\n END IF;\n\n -- Spatial Query\n IF op ilike 's_%' or op = 'intersects' THEN\n RETURN spatial_op_query(op, args);\n END IF;\n\n -- In Query - separate into separate eq statements so that we can use eq jsonb optimization\n IF op = 'in' THEN\n RAISE NOTICE '% IN args: %', repeat(' ', recursion), args;\n SELECT INTO inobj\n jsonb_agg(\n jsonb_build_object(\n 'op', 'eq',\n 'args', jsonb_build_array( args->0 , v)\n )\n )\n FROM jsonb_array_elements( args->1) v;\n RETURN cql2_query(jsonb_build_object('op','or','args',inobj));\n END IF;\nEND IF;\n\nIF j ? 'property' THEN\n RETURN (items_path(j->>'property')).path_txt;\nEND IF;\n\nIF j ? 'timestamp' THEN\n RETURN quote_literal(j->>'timestamp');\nEND IF;\n\nRAISE NOTICE '%jtype: %',repeat(' ', recursion), jtype;\nIF jsonb_typeof(j) = 'number' THEN\n RETURN format('%L::numeric', j->>0);\nEND IF;\n\nIF jsonb_typeof(j) = 'string' THEN\n RETURN quote_literal(j->>0);\nEND IF;\n\nIF jsonb_typeof(j) = 'array' THEN\n IF j @? '$[*] ? (@.type() == \"number\")' THEN\n RETURN CONCAT(quote_literal(textarr(j)::text), '::numeric[]');\n ELSE\n RETURN CONCAT(quote_literal(textarr(j)::text), '::text[]');\n END IF;\nEND IF;\nRAISE NOTICE 'ARGS after array cleaning: %', args;\n\nRAISE NOTICE '%beforeargs op: %, args: %',repeat(' ', recursion), op, args;\nIF j ? 'args' THEN\n FOR arg in SELECT * FROM jsonb_array_elements(args) LOOP\n argtext := cql2_query(arg, recursion + 1);\n RAISE NOTICE '% -- arg: %, argtext: %', repeat(' ', recursion), arg, argtext;\n argstext := argstext || argtext;\n END LOOP;\nEND IF;\nRAISE NOTICE '%afterargs op: %, argstext: %',repeat(' ', recursion), op, argstext;\n\n\nIF op IN ('and', 'or') THEN\n RAISE NOTICE 'inand op: %, argstext: %', op, argstext;\n SELECT\n concat(' ( ',array_to_string(array_agg(e), concat(' ',op,' ')),' ) ')\n INTO ret\n FROM unnest(argstext) e;\n RETURN ret;\nEND IF;\n\nIF ops ? op THEN\n IF argstext[2] ~* 'numeric' THEN\n argstext := ARRAY[concat('(',argstext[1],')::numeric')] || argstext[2:3];\n END IF;\n RETURN format(concat('(',ops->>op,')'), VARIADIC argstext);\nEND IF;\n\nRAISE NOTICE '%op: %, argstext: %',repeat(' ', recursion), op, argstext;\n\nRETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n\nCREATE OR REPLACE FUNCTION cql_to_where(_search jsonb = '{}'::jsonb) RETURNS text AS $$\nDECLARE\nfilterlang text;\nsearch jsonb := _search;\nbase_where text;\n_where text;\nBEGIN\n\nRAISE NOTICE 'SEARCH CQL Final: %', search;\nfilterlang := COALESCE(\n search->>'filter-lang',\n get_setting('default-filter-lang', _search->'conf')\n);\n\nbase_where := base_stac_query(search);\n\nIF filterlang = 'cql-json' THEN\n search := query_to_cqlfilter(search);\n -- search := add_filters_to_cql(search);\n _where := cql_query_op(search->'filter');\nELSE\n _where := cql2_query(search->'filter');\nEND IF;\n\nIF trim(_where) = '' THEN\n _where := NULL;\nEND IF;\n_where := coalesce(_where, ' TRUE ');\nRETURN format('( %s ) %s ', _where, base_where);\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION parse_sort_dir(_dir text, reverse boolean default false) RETURNS text AS $$\nWITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n) SELECT\n CASE\n WHEN NOT reverse THEN d\n WHEN d = 'ASC' THEN 'DESC'\n WHEN d = 'DESC' THEN 'ASC'\n END\nFROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION sort_dir_to_op(_dir text, prev boolean default false) RETURNS text AS $$\nWITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n) SELECT\n CASE\n WHEN d = 'ASC' AND prev THEN '<='\n WHEN d = 'DESC' AND prev THEN '>='\n WHEN d = 'ASC' THEN '>='\n WHEN d = 'DESC' THEN '<='\n END\nFROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION field_orderby(p text) RETURNS text AS $$\nWITH t AS (\n SELECT\n replace(trim(substring(indexdef from 'btree \\((.*)\\)')),' ','')as s\n FROM pg_indexes WHERE schemaname='pgstac' AND tablename='items' AND indexdef ~* 'btree' AND indexdef ~* 'properties'\n) SELECT s FROM t WHERE strpos(s, lower(trim(p)))>0;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION sort_sqlorderby(\n _search jsonb DEFAULT NULL,\n reverse boolean DEFAULT FALSE\n) RETURNS text AS $$\nWITH sortby AS (\n SELECT coalesce(_search->'sortby','[{\"field\":\"datetime\", \"direction\":\"desc\"}]') as sort\n), withid AS (\n SELECT CASE\n WHEN sort @? '$[*] ? (@.field == \"id\")' THEN sort\n ELSE sort || '[{\"field\":\"id\", \"direction\":\"desc\"}]'::jsonb\n END as sort\n FROM sortby\n), withid_rows AS (\n SELECT jsonb_array_elements(sort) as value FROM withid\n),sorts AS (\n SELECT\n coalesce(field_orderby((items_path(value->>'field')).path_txt), (items_path(value->>'field')).path) as key,\n parse_sort_dir(value->>'direction', reverse) as dir\n FROM withid_rows\n)\nSELECT array_to_string(\n array_agg(concat(key, ' ', dir)),\n ', '\n) FROM sorts;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_sort_dir(sort_item jsonb) RETURNS text AS $$\nSELECT CASE WHEN sort_item->>'direction' ILIKE 'desc%' THEN 'DESC' ELSE 'ASC' END;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION get_token_filter(_search jsonb = '{}'::jsonb, token_rec jsonb DEFAULT NULL) RETURNS text AS $$\nDECLARE\n token_id text;\n filters text[] := '{}'::text[];\n prev boolean := TRUE;\n field text;\n dir text;\n sort record;\n orfilters text[] := '{}'::text[];\n andfilters text[] := '{}'::text[];\n output text;\n token_where text;\nBEGIN\n -- If no token provided return NULL\n IF token_rec IS NULL THEN\n IF NOT (_search ? 'token' AND\n (\n (_search->>'token' ILIKE 'prev:%')\n OR\n (_search->>'token' ILIKE 'next:%')\n )\n ) THEN\n RETURN NULL;\n END IF;\n prev := (_search->>'token' ILIKE 'prev:%');\n token_id := substr(_search->>'token', 6);\n SELECT to_jsonb(items) INTO token_rec FROM item_by_id(token_id) as items;\n END IF;\n RAISE NOTICE 'TOKEN ID: %', token_rec->'id';\n\n CREATE TEMP TABLE sorts (\n _row int GENERATED ALWAYS AS IDENTITY NOT NULL,\n _field text PRIMARY KEY,\n _dir text NOT NULL,\n _val text\n ) ON COMMIT DROP;\n\n -- Make sure we only have distinct columns to sort with taking the first one we get\n INSERT INTO sorts (_field, _dir)\n SELECT\n (items_path(value->>'field')).path,\n get_sort_dir(value)\n FROM\n jsonb_array_elements(coalesce(_search->'sortby','[{\"field\":\"datetime\",\"direction\":\"desc\"}]'))\n ON CONFLICT DO NOTHING\n ;\n RAISE NOTICE 'sorts 1: %', (SELECT jsonb_agg(to_json(sorts)) FROM sorts);\n -- Get the first sort direction provided. As the id is a primary key, if there are any\n -- sorts after id they won't do anything, so make sure that id is the last sort item.\n SELECT _dir INTO dir FROM sorts ORDER BY _row ASC LIMIT 1;\n IF EXISTS (SELECT 1 FROM sorts WHERE _field = 'id') THEN\n DELETE FROM sorts WHERE _row > (SELECT _row FROM sorts WHERE _field = 'id' ORDER BY _row ASC);\n ELSE\n INSERT INTO sorts (_field, _dir) VALUES ('id', dir);\n END IF;\n\n -- Add value from looked up item to the sorts table\n UPDATE sorts SET _val=quote_literal(token_rec->>_field);\n\n -- Check if all sorts are the same direction and use row comparison\n -- to filter\n RAISE NOTICE 'sorts 2: %', (SELECT jsonb_agg(to_json(sorts)) FROM sorts);\n\n IF (SELECT count(DISTINCT _dir) FROM sorts) = 1 THEN\n SELECT format(\n '(%s) %s (%s)',\n concat_ws(', ', VARIADIC array_agg(quote_ident(_field))),\n CASE WHEN (prev AND dir = 'ASC') OR (NOT prev AND dir = 'DESC') THEN '<' ELSE '>' END,\n concat_ws(', ', VARIADIC array_agg(_val))\n ) INTO output FROM sorts\n WHERE token_rec ? _field\n ;\n ELSE\n FOR sort IN SELECT * FROM sorts ORDER BY _row asc LOOP\n RAISE NOTICE 'SORT: %', sort;\n IF sort._row = 1 THEN\n orfilters := orfilters || format('(%s %s %s)',\n quote_ident(sort._field),\n CASE WHEN (prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC') THEN '<' ELSE '>' END,\n sort._val\n );\n ELSE\n orfilters := orfilters || format('(%s AND %s %s %s)',\n array_to_string(andfilters, ' AND '),\n quote_ident(sort._field),\n CASE WHEN (prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC') THEN '<' ELSE '>' END,\n sort._val\n );\n\n END IF;\n andfilters := andfilters || format('%s = %s',\n quote_ident(sort._field),\n sort._val\n );\n END LOOP;\n output := array_to_string(orfilters, ' OR ');\n END IF;\n DROP TABLE IF EXISTS sorts;\n token_where := concat('(',coalesce(output,'true'),')');\n IF trim(token_where) = '' THEN\n token_where := NULL;\n END IF;\n RAISE NOTICE 'TOKEN_WHERE: |%|',token_where;\n RETURN token_where;\n END;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION search_tohash(jsonb) RETURNS jsonb AS $$\n SELECT $1 - '{token,limit,context,includes,excludes}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION search_hash(jsonb, jsonb) RETURNS text AS $$\n SELECT md5(concat(search_tohash($1)::text,$2::text));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TABLE IF NOT EXISTS searches(\n hash text GENERATED ALWAYS AS (search_hash(search, metadata)) STORED PRIMARY KEY,\n search jsonb NOT NULL,\n _where text,\n orderby text,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n metadata jsonb DEFAULT '{}'::jsonb NOT NULL\n);\nCREATE TABLE IF NOT EXISTS search_wheres(\n id bigint generated always as identity primary key,\n _where text NOT NULL,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n statslastupdated timestamptz,\n estimated_count bigint,\n estimated_cost float,\n time_to_estimate float,\n total_count bigint,\n time_to_count float,\n partitions text[]\n);\n\nCREATE INDEX IF NOT EXISTS search_wheres_partitions ON search_wheres USING GIN (partitions);\nCREATE UNIQUE INDEX IF NOT EXISTS search_wheres_where ON search_wheres ((md5(_where)));\n\nCREATE OR REPLACE FUNCTION where_stats(inwhere text, updatestats boolean default false, conf jsonb default null) RETURNS search_wheres AS $$\nDECLARE\n t timestamptz;\n i interval;\n explain_json jsonb;\n partitions text[];\n sw search_wheres%ROWTYPE;\n inwhere_hash text := md5(inwhere);\nBEGIN\n SELECT * INTO sw FROM search_wheres WHERE _where=inwhere_hash FOR UPDATE;\n\n -- Update statistics if explicitly set, if statistics do not exist, or statistics ttl has expired\n IF NOT updatestats THEN\n RAISE NOTICE 'Checking if update is needed.';\n RAISE NOTICE 'Stats Last Updated: %', sw.statslastupdated;\n RAISE NOTICE 'TTL: %, Age: %', context_stats_ttl(conf), now() - sw.statslastupdated;\n RAISE NOTICE 'Context: %, Existing Total: %', context(conf), sw.total_count;\n IF\n sw.statslastupdated IS NULL\n OR (now() - sw.statslastupdated) > context_stats_ttl(conf)\n OR (context(conf) != 'off' AND sw.total_count IS NULL)\n THEN\n updatestats := TRUE;\n END IF;\n END IF;\n\n sw._where := inwhere;\n sw.lastused := now();\n sw.usecount := coalesce(sw.usecount,0) + 1;\n\n IF NOT updatestats THEN\n UPDATE search_wheres SET\n lastused = sw.lastused,\n usecount = sw.usecount\n WHERE _where = inwhere_hash\n RETURNING * INTO sw\n ;\n RETURN sw;\n END IF;\n -- Use explain to get estimated count/cost and a list of the partitions that would be hit by the query\n t := clock_timestamp();\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s', inwhere)\n INTO explain_json;\n RAISE NOTICE 'Time for just the explain: %', clock_timestamp() - t;\n WITH t AS (\n SELECT j->>0 as p FROM\n jsonb_path_query(\n explain_json,\n 'strict $.**.\"Relation Name\" ? (@ != null)'\n ) j\n ), ordered AS (\n SELECT p FROM t ORDER BY p DESC\n -- SELECT p FROM t JOIN items_partitions\n -- ON (t.p = items_partitions.partition)\n -- ORDER BY pstart DESC\n )\n SELECT array_agg(p) INTO partitions FROM ordered;\n i := clock_timestamp() - t;\n RAISE NOTICE 'Time for explain + join: %', clock_timestamp() - t;\n\n\n\n sw.statslastupdated := now();\n sw.estimated_count := explain_json->0->'Plan'->'Plan Rows';\n sw.estimated_cost := explain_json->0->'Plan'->'Total Cost';\n sw.time_to_estimate := extract(epoch from i);\n sw.partitions := partitions;\n\n -- Do a full count of rows if context is set to on or if auto is set and estimates are low enough\n IF\n context(conf) = 'on'\n OR\n ( context(conf) = 'auto' AND\n (\n sw.estimated_count < context_estimated_count(conf)\n OR\n sw.estimated_cost < context_estimated_cost(conf)\n )\n )\n THEN\n t := clock_timestamp();\n RAISE NOTICE 'Calculating actual count...';\n EXECUTE format(\n 'SELECT count(*) FROM items WHERE %s',\n inwhere\n ) INTO sw.total_count;\n i := clock_timestamp() - t;\n RAISE NOTICE 'Actual Count: % -- %', sw.total_count, i;\n sw.time_to_count := extract(epoch FROM i);\n ELSE\n sw.total_count := NULL;\n sw.time_to_count := NULL;\n END IF;\n\n\n INSERT INTO search_wheres\n (_where, lastused, usecount, statslastupdated, estimated_count, estimated_cost, time_to_estimate, partitions, total_count, time_to_count)\n SELECT sw._where, sw.lastused, sw.usecount, sw.statslastupdated, sw.estimated_count, sw.estimated_cost, sw.time_to_estimate, sw.partitions, sw.total_count, sw.time_to_count\n ON CONFLICT ((md5(_where)))\n DO UPDATE\n SET\n lastused = sw.lastused,\n usecount = sw.usecount,\n statslastupdated = sw.statslastupdated,\n estimated_count = sw.estimated_count,\n estimated_cost = sw.estimated_cost,\n time_to_estimate = sw.time_to_estimate,\n partitions = sw.partitions,\n total_count = sw.total_count,\n time_to_count = sw.time_to_count\n ;\n RETURN sw;\nEND;\n$$ LANGUAGE PLPGSQL ;\n\n\nCREATE OR REPLACE FUNCTION items_count(_where text) RETURNS bigint AS $$\nDECLARE\ncnt bigint;\nBEGIN\nEXECUTE format('SELECT count(*) FROM items WHERE %s', _where) INTO cnt;\nRETURN cnt;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n\n\n\nDROP FUNCTION IF EXISTS search_query;\nCREATE OR REPLACE FUNCTION search_query(\n _search jsonb = '{}'::jsonb,\n updatestats boolean = false,\n _metadata jsonb = '{}'::jsonb\n) RETURNS searches AS $$\nDECLARE\n search searches%ROWTYPE;\n pexplain jsonb;\n t timestamptz;\n i interval;\nBEGIN\nSELECT * INTO search FROM searches\nWHERE hash=search_hash(_search, _metadata) FOR UPDATE;\n\n-- Calculate the where clause if not already calculated\nIF search._where IS NULL THEN\n search._where := cql_to_where(_search);\nEND IF;\n\n-- Calculate the order by clause if not already calculated\nIF search.orderby IS NULL THEN\n search.orderby := sort_sqlorderby(_search);\nEND IF;\n\nPERFORM where_stats(search._where, updatestats, _search->'conf');\n\nsearch.lastused := now();\nsearch.usecount := coalesce(search.usecount, 0) + 1;\nINSERT INTO searches (search, _where, orderby, lastused, usecount, metadata)\nVALUES (_search, search._where, search.orderby, search.lastused, search.usecount, _metadata)\nON CONFLICT (hash) DO\nUPDATE SET\n _where = EXCLUDED._where,\n orderby = EXCLUDED.orderby,\n lastused = EXCLUDED.lastused,\n usecount = EXCLUDED.usecount,\n metadata = EXCLUDED.metadata\nRETURNING * INTO search\n;\nRETURN search;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n\nCREATE OR REPLACE FUNCTION search(_search jsonb = '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n searches searches%ROWTYPE;\n _where text;\n token_where text;\n full_where text;\n orderby text;\n query text;\n token_type text := substr(_search->>'token',1,4);\n _limit int := coalesce((_search->>'limit')::int, 10);\n curs refcursor;\n cntr int := 0;\n iter_record items%ROWTYPE;\n first_record items%ROWTYPE;\n last_record items%ROWTYPE;\n out_records jsonb := '[]'::jsonb;\n prev_query text;\n next text;\n prev_id text;\n has_next boolean := false;\n has_prev boolean := false;\n prev text;\n total_count bigint;\n context jsonb;\n collection jsonb;\n includes text[];\n excludes text[];\n exit_flag boolean := FALSE;\n batches int := 0;\n timer timestamptz := clock_timestamp();\n pstart timestamptz;\n pend timestamptz;\n pcurs refcursor;\n search_where search_wheres%ROWTYPE;\n id text;\nBEGIN\nCREATE TEMP TABLE results (content jsonb) ON COMMIT DROP;\n-- if ids is set, short circuit and just use direct ids query for each id\n-- skip any paging or caching\n-- hard codes ordering in the same order as the array of ids\nIF _search ? 'ids' THEN\n FOR id IN SELECT jsonb_array_elements_text(_search->'ids') LOOP\n INSERT INTO results (content) SELECT content FROM item_by_id(id);\n END LOOP;\n SELECT INTO total_count count(*) FROM results;\nELSE\n searches := search_query(_search);\n _where := searches._where;\n orderby := searches.orderby;\n search_where := where_stats(_where);\n total_count := coalesce(search_where.total_count, search_where.estimated_count);\n\n\n\n IF token_type='prev' THEN\n token_where := get_token_filter(_search, null::jsonb);\n orderby := sort_sqlorderby(_search, TRUE);\n END IF;\n IF token_type='next' THEN\n token_where := get_token_filter(_search, null::jsonb);\n END IF;\n\n full_where := concat_ws(' AND ', _where, token_where);\n RAISE NOTICE 'FULL QUERY % %', full_where, clock_timestamp()-timer;\n timer := clock_timestamp();\n\n\n\n\n FOR query IN SELECT partition_queries(full_where, orderby, search_where.partitions) LOOP\n timer := clock_timestamp();\n query := format('%s LIMIT %s', query, _limit + 1);\n RAISE NOTICE 'Partition Query: %', query;\n batches := batches + 1;\n -- curs = create_cursor(query);\n OPEN curs FOR EXECUTE query;\n LOOP\n FETCH curs into iter_record;\n EXIT WHEN NOT FOUND;\n cntr := cntr + 1;\n last_record := iter_record;\n IF cntr = 1 THEN\n first_record := last_record;\n END IF;\n IF cntr <= _limit THEN\n INSERT INTO results (content) VALUES (last_record.content);\n -- out_records := out_records || last_record.content;\n\n ELSIF cntr > _limit THEN\n has_next := true;\n exit_flag := true;\n EXIT;\n END IF;\n END LOOP;\n CLOSE curs;\n RAISE NOTICE 'Query took %. Total Time %', clock_timestamp()-timer, ftime();\n timer := clock_timestamp();\n EXIT WHEN exit_flag;\n END LOOP;\n RAISE NOTICE 'Scanned through % partitions.', batches;\nEND IF;\n\nSELECT jsonb_agg(content) INTO out_records FROM results;\n\nDROP TABLE results;\n\n\n-- Flip things around if this was the result of a prev token query\nIF token_type='prev' THEN\n out_records := flip_jsonb_array(out_records);\n first_record := last_record;\nEND IF;\n\n-- If this query has a token, see if there is data before the first record\nIF _search ? 'token' THEN\n prev_query := format(\n 'SELECT 1 FROM items WHERE %s LIMIT 1',\n concat_ws(\n ' AND ',\n _where,\n trim(get_token_filter(_search, to_jsonb(first_record)))\n )\n );\n RAISE NOTICE 'Query to get previous record: % --- %', prev_query, first_record;\n EXECUTE prev_query INTO has_prev;\n IF FOUND and has_prev IS NOT NULL THEN\n RAISE NOTICE 'Query results from prev query: %', has_prev;\n has_prev := TRUE;\n END IF;\nEND IF;\nhas_prev := COALESCE(has_prev, FALSE);\n\nRAISE NOTICE 'token_type: %, has_next: %, has_prev: %', token_type, has_next, has_prev;\nIF has_prev THEN\n prev := out_records->0->>'id';\nEND IF;\nIF has_next OR token_type='prev' THEN\n next := out_records->-1->>'id';\nEND IF;\n\n\n-- include/exclude any fields following fields extension\nIF _search ? 'fields' THEN\n IF _search->'fields' ? 'exclude' THEN\n excludes=textarr(_search->'fields'->'exclude');\n END IF;\n IF _search->'fields' ? 'include' THEN\n includes=textarr(_search->'fields'->'include');\n IF array_length(includes, 1)>0 AND NOT 'id' = ANY (includes) THEN\n includes = includes || '{id}';\n END IF;\n END IF;\n SELECT jsonb_agg(filter_jsonb(row, includes, excludes)) INTO out_records FROM jsonb_array_elements(out_records) row;\nEND IF;\n\nIF context(_search->'conf') != 'off' THEN\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'matched', total_count,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\nELSE\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\nEND IF;\n\ncollection := jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb),\n 'next', next,\n 'prev', prev,\n 'context', context\n);\n\nRETURN collection;\nEND;\n$$ LANGUAGE PLPGSQL\nSET jit TO off\n;\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION tileenvelope(zoom int, x int, y int) RETURNS geometry AS $$\nWITH t AS (\n SELECT\n 20037508.3427892 as merc_max,\n -20037508.3427892 as merc_min,\n (2 * 20037508.3427892) / (2 ^ zoom) as tile_size\n)\nSELECT st_makeenvelope(\n merc_min + (tile_size * x),\n merc_max - (tile_size * (y + 1)),\n merc_min + (tile_size * (x + 1)),\n merc_max - (tile_size * y),\n 3857\n) FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;DROP FUNCTION IF EXISTS mercgrid;\n\n\nCREATE OR REPLACE FUNCTION ftime() RETURNS interval as $$\nSELECT age(clock_timestamp(), transaction_timestamp());\n$$ LANGUAGE SQL;\nSET SEARCH_PATH to pgstac, public;\n\nDROP FUNCTION IF EXISTS geometrysearch;\nCREATE OR REPLACE FUNCTION geometrysearch(\n IN geom geometry,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE, -- Return as soon as the passed in geometry is full covered\n IN skipcovered boolean DEFAULT TRUE -- Skip any items that would show up completely under the previous items\n) RETURNS jsonb AS $$\nDECLARE\n search searches%ROWTYPE;\n curs refcursor;\n _where text;\n query text;\n iter_record items%ROWTYPE;\n out_records jsonb[] := '{}'::jsonb[];\n exit_flag boolean := FALSE;\n counter int := 1;\n scancounter int := 1;\n remaining_limit int := _scanlimit;\n tilearea float;\n unionedgeom geometry;\n clippedgeom geometry;\n unionedgeom_area float := 0;\n prev_area float := 0;\n excludes text[];\n includes text[];\n\nBEGIN\n -- If skipcovered is true then you will always want to exit when the passed in geometry is full\n IF skipcovered THEN\n exitwhenfull := TRUE;\n END IF;\n\n SELECT * INTO search FROM searches WHERE hash=queryhash;\n\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Search with Query Hash % Not Found', queryhash;\n END IF;\n\n tilearea := st_area(geom);\n _where := format('%s AND st_intersects(geometry, %L::geometry)', search._where, geom);\n\n IF fields IS NOT NULL THEN\n IF fields ? 'fields' THEN\n fields := fields->'fields';\n END IF;\n IF fields ? 'exclude' THEN\n excludes=textarr(fields->'exclude');\n END IF;\n IF fields ? 'include' THEN\n includes=textarr(fields->'include');\n IF array_length(includes, 1)>0 AND NOT 'id' = ANY (includes) THEN\n includes = includes || '{id}';\n END IF;\n END IF;\n END IF;\n RAISE NOTICE 'fields: %, includes: %, excludes: %', fields, includes, excludes;\n\n FOR query IN SELECT * FROM partition_queries(_where, search.orderby) LOOP\n query := format('%s LIMIT %L', query, remaining_limit);\n RAISE NOTICE '%', query;\n curs = create_cursor(query);\n LOOP\n FETCH curs INTO iter_record;\n EXIT WHEN NOT FOUND;\n IF exitwhenfull OR skipcovered THEN -- If we are not using exitwhenfull or skipcovered, we do not need to do expensive geometry operations\n clippedgeom := st_intersection(geom, iter_record.geometry);\n\n IF unionedgeom IS NULL THEN\n unionedgeom := clippedgeom;\n ELSE\n unionedgeom := st_union(unionedgeom, clippedgeom);\n END IF;\n\n unionedgeom_area := st_area(unionedgeom);\n\n IF skipcovered AND prev_area = unionedgeom_area THEN\n scancounter := scancounter + 1;\n CONTINUE;\n END IF;\n\n prev_area := unionedgeom_area;\n\n RAISE NOTICE '% % % %', unionedgeom_area/tilearea, counter, scancounter, ftime();\n END IF;\n\n IF fields IS NOT NULL THEN\n out_records := out_records || filter_jsonb(iter_record.content, includes, excludes);\n ELSE\n out_records := out_records || iter_record.content;\n END IF;\n IF counter >= _limit\n OR scancounter > _scanlimit\n OR ftime() > _timelimit\n OR (exitwhenfull AND unionedgeom_area >= tilearea)\n THEN\n exit_flag := TRUE;\n EXIT;\n END IF;\n counter := counter + 1;\n scancounter := scancounter + 1;\n\n END LOOP;\n EXIT WHEN exit_flag;\n remaining_limit := _scanlimit - scancounter;\n END LOOP;\n\n RETURN jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', array_to_json(out_records)::jsonb\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDROP FUNCTION IF EXISTS geojsonsearch;\nCREATE OR REPLACE FUNCTION geojsonsearch(\n IN geojson jsonb,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_geomfromgeojson(geojson),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS xyzsearch;\nCREATE OR REPLACE FUNCTION xyzsearch(\n IN _x int,\n IN _y int,\n IN _z int,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_transform(tileenvelope(_z, _x, _y), 4326),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\nSELECT set_version('0.4.4');\n" }, { "path": "src/pgstac/migrations/pgstac.0.4.5-0.5.0.sql", "content": "CREATE EXTENSION IF NOT EXISTS postgis;\nCREATE EXTENSION IF NOT EXISTS btree_gist;\n\nALTER SCHEMA pgstac rename to pgstac_045;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_admin;\n CREATE ROLE pgstac_read;\n CREATE ROLE pgstac_ingest;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nGRANT pgstac_admin TO current_user;\n\nCREATE SCHEMA IF NOT EXISTS pgstac AUTHORIZATION pgstac_admin;\n\nALTER ROLE pgstac_admin SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_read SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_ingest SET SEARCH_PATH TO pgstac, public;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\n\nGRANT pgstac_read TO pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\n\nSET ROLE pgstac_admin;\n\nSET SEARCH_PATH TO pgstac, public;\n\n\nCREATE TABLE IF NOT EXISTS migrations (\n version text PRIMARY KEY,\n datetime timestamptz DEFAULT clock_timestamp() NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION get_version() RETURNS text AS $$\n SELECT version FROM pgstac.migrations ORDER BY datetime DESC, version DESC LIMIT 1;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION set_version(text) RETURNS text AS $$\n INSERT INTO pgstac.migrations (version) VALUES ($1)\n ON CONFLICT DO NOTHING\n RETURNING version;\n$$ LANGUAGE SQL;\n\n\nCREATE TABLE IF NOT EXISTS pgstac_settings (\n name text PRIMARY KEY,\n value text NOT NULL\n);\n\nINSERT INTO pgstac_settings (name, value) VALUES\n ('context', 'off'),\n ('context_estimated_count', '100000'),\n ('context_estimated_cost', '100000'),\n ('context_stats_ttl', '1 day'),\n ('default-filter-lang', 'cql2-json'),\n ('additional_properties', 'true')\nON CONFLICT DO NOTHING\n;\n\n\nCREATE OR REPLACE FUNCTION get_setting(IN _setting text, IN conf jsonb DEFAULT NULL) RETURNS text AS $$\nSELECT COALESCE(\n conf->>_setting,\n current_setting(concat('pgstac.',_setting), TRUE),\n (SELECT value FROM pgstac.pgstac_settings WHERE name=_setting)\n);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context(conf jsonb DEFAULT NULL) RETURNS text AS $$\n SELECT pgstac.get_setting('context', conf);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context_estimated_count(conf jsonb DEFAULT NULL) RETURNS int AS $$\n SELECT pgstac.get_setting('context_estimated_count', conf)::int;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_estimated_cost();\nCREATE OR REPLACE FUNCTION context_estimated_cost(conf jsonb DEFAULT NULL) RETURNS float AS $$\n SELECT pgstac.get_setting('context_estimated_cost', conf)::float;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_stats_ttl();\nCREATE OR REPLACE FUNCTION context_stats_ttl(conf jsonb DEFAULT NULL) RETURNS interval AS $$\n SELECT pgstac.get_setting('context_stats_ttl', conf)::interval;\n$$ LANGUAGE SQL;\n\n\nCREATE OR REPLACE FUNCTION notice(VARIADIC text[]) RETURNS boolean AS $$\nDECLARE\ndebug boolean := current_setting('pgstac.debug', true);\nBEGIN\n IF debug THEN\n RAISE NOTICE 'NOTICE FROM FUNC: % >>>>> %', concat_ws(' | ', $1), clock_timestamp();\n RETURN TRUE;\n END IF;\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION empty_arr(ANYARRAY) RETURNS BOOLEAN AS $$\nSELECT CASE\n WHEN $1 IS NULL THEN TRUE\n WHEN cardinality($1)<1 THEN TRUE\nELSE FALSE\nEND;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_intersection(_a ANYARRAY, _b ANYARRAY) RETURNS ANYARRAY AS $$\n SELECT ARRAY ( SELECT unnest(_a) INTERSECT SELECT UNNEST(_b) );\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION array_map_ident(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_ident(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_map_literal(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_literal(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION array_reverse(anyarray) RETURNS anyarray AS $$\nSELECT ARRAY(\n SELECT $1[i]\n FROM generate_subscripts($1,1) AS s(i)\n ORDER BY i DESC\n);\n$$ LANGUAGE SQL STRICT IMMUTABLE;\nCREATE OR REPLACE FUNCTION to_int(jsonb) RETURNS int AS $$\n SELECT floor(($1->>0)::float)::int;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION to_float(jsonb) RETURNS float AS $$\n SELECT ($1->>0)::float;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION to_tstz(jsonb) RETURNS timestamptz AS $$\n SELECT ($1->>0)::timestamptz;\n$$ LANGUAGE SQL IMMUTABLE STRICT SET TIME ZONE 'UTC';\n\n\nCREATE OR REPLACE FUNCTION to_text(jsonb) RETURNS text AS $$\n SELECT $1->>0;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION to_text_array(jsonb) RETURNS text[] AS $$\n SELECT\n CASE jsonb_typeof($1)\n WHEN 'array' THEN ARRAY(SELECT jsonb_array_elements_text($1))\n ELSE ARRAY[$1->>0]\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\nCREATE OR REPLACE FUNCTION bbox_geom(_bbox jsonb) RETURNS geometry AS $$\nSELECT CASE jsonb_array_length(_bbox)\n WHEN 4 THEN\n ST_SetSRID(ST_MakeEnvelope(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float,\n (_bbox->>3)::float\n ),4326)\n WHEN 6 THEN\n ST_SetSRID(ST_3DMakeBox(\n ST_MakePoint(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float\n ),\n ST_MakePoint(\n (_bbox->>3)::float,\n (_bbox->>4)::float,\n (_bbox->>5)::float\n )\n ),4326)\n ELSE null END;\n;\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION geom_bbox(_geom geometry) RETURNS jsonb AS $$\n SELECT jsonb_build_array(\n st_xmin(_geom),\n st_ymin(_geom),\n st_xmax(_geom),\n st_ymax(_geom)\n );\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION flip_jsonb_array(j jsonb) RETURNS jsonb AS $$\n SELECT jsonb_agg(value) FROM (SELECT value FROM jsonb_array_elements(j) WITH ORDINALITY ORDER BY ordinality DESC) as t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n/* looks for a geometry in a stac item first from geometry and falling back to bbox */\nCREATE OR REPLACE FUNCTION stac_geom(value jsonb) RETURNS geometry AS $$\nSELECT\n CASE\n WHEN value ? 'intersects' THEN\n ST_GeomFromGeoJSON(value->>'intersects')\n WHEN value ? 'geometry' THEN\n ST_GeomFromGeoJSON(value->>'geometry')\n WHEN value ? 'bbox' THEN\n pgstac.bbox_geom(value->'bbox')\n ELSE NULL\n END as geometry\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION stac_daterange(\n value jsonb\n) RETURNS tstzrange AS $$\nDECLARE\n props jsonb := value;\n dt timestamptz;\n edt timestamptz;\nBEGIN\n IF props ? 'properties' THEN\n props := props->'properties';\n END IF;\n IF props ? 'start_datetime' AND props ? 'end_datetime' THEN\n dt := props->'start_datetime';\n edt := props->'end_datetime';\n IF dt > edt THEN\n RAISE EXCEPTION 'start_datetime must be < end_datetime';\n END IF;\n ELSE\n dt := props->'datetime';\n edt := props->'datetime';\n END IF;\n IF dt is NULL OR edt IS NULL THEN\n RAISE EXCEPTION 'Either datetime or both start_datetime and end_datetime must be set.';\n END IF;\n RETURN tstzrange(dt, edt, '[]');\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_datetime(value jsonb) RETURNS timestamptz AS $$\n SELECT lower(stac_daterange(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_end_datetime(value jsonb) RETURNS timestamptz AS $$\n SELECT upper(stac_daterange(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\n\nCREATE TABLE IF NOT EXISTS stac_extensions(\n name text PRIMARY KEY,\n url text,\n enbabled_by_default boolean NOT NULL DEFAULT TRUE,\n enableable boolean NOT NULL DEFAULT TRUE\n);\n\nINSERT INTO stac_extensions (name, url) VALUES\n ('fields', 'https://api.stacspec.org/v1.0.0-beta.5/item-search#fields'),\n ('sort','https://api.stacspec.org/v1.0.0-beta.5/item-search#sort'),\n ('context','https://api.stacspec.org/v1.0.0-beta.5/item-search#context'),\n ('filter', 'https://api.stacspec.org/v1.0.0-beta.5/item-search#filter'),\n ('query', 'https://api.stacspec.org/v1.0.0-beta.5/item-search#query')\nON CONFLICT (name) DO UPDATE SET url=EXCLUDED.url;\n\n\n\nCREATE OR REPLACE FUNCTION collection_base_item(content jsonb) RETURNS jsonb AS $$\n SELECT jsonb_build_object(\n 'type', 'Feature',\n 'stac_version', content->'stac_version',\n 'assets', content->'item_assets',\n 'collection', content->'id',\n 'links', '[]'::jsonb\n );\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TYPE partition_trunc_strategy AS ENUM ('year', 'month');\n\nCREATE TABLE IF NOT EXISTS collections (\n key bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,\n id text GENERATED ALWAYS AS (content->>'id') STORED UNIQUE,\n content JSONB NOT NULL,\n base_item jsonb GENERATED ALWAYS AS (pgstac.collection_base_item(content)) STORED,\n partition_trunc partition_trunc_strategy\n);\n\n\nCREATE OR REPLACE FUNCTION collection_base_item(cid text) RETURNS jsonb AS $$\n SELECT pgstac.collection_base_item(content) FROM pgstac.collections WHERE id = cid LIMIT 1;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION table_empty(text) RETURNS boolean AS $$\nDECLARE\n retval boolean;\nBEGIN\n EXECUTE format($q$\n SELECT NOT EXISTS (SELECT 1 FROM %I LIMIT 1)\n $q$,\n $1\n ) INTO retval;\n RETURN retval;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION collections_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n q text;\n partition_name text := format('_items_%s', NEW.key);\n partition_exists boolean := false;\n partition_empty boolean := true;\n err_context text;\n loadtemp boolean := FALSE;\nBEGIN\n RAISE NOTICE 'Collection Trigger. % %', NEW.id, NEW.key;\n SELECT relid::text INTO partition_name\n FROM pg_partition_tree('items')\n WHERE relid::text = partition_name;\n IF FOUND THEN\n partition_exists := true;\n partition_empty := table_empty(partition_name);\n ELSE\n partition_exists := false;\n partition_empty := true;\n partition_name := format('_items_%s', NEW.key);\n END IF;\n IF TG_OP = 'UPDATE' AND NEW.partition_trunc IS DISTINCT FROM OLD.partition_trunc AND partition_empty THEN\n q := format($q$\n DROP TABLE IF EXISTS %I CASCADE;\n $q$,\n partition_name\n );\n EXECUTE q;\n END IF;\n IF TG_OP = 'UPDATE' AND NEW.partition_trunc IS DISTINCT FROM OLD.partition_trunc AND partition_exists AND NOT partition_empty THEN\n q := format($q$\n CREATE TEMP TABLE changepartitionstaging ON COMMIT DROP AS SELECT * FROM %I;\n DROP TABLE IF EXISTS %I CASCADE;\n $q$,\n partition_name,\n partition_name\n );\n EXECUTE q;\n loadtemp := TRUE;\n partition_empty := TRUE;\n partition_exists := FALSE;\n END IF;\n IF TG_OP = 'UPDATE' AND NEW.partition_trunc IS NOT DISTINCT FROM OLD.partition_trunc THEN\n RETURN NEW;\n END IF;\n IF NEW.partition_trunc IS NULL AND partition_empty THEN\n RAISE NOTICE '% % % %',\n partition_name,\n NEW.id,\n concat(partition_name,'_id_idx'),\n partition_name\n ;\n q := format($q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n $q$,\n partition_name,\n NEW.id,\n concat(partition_name,'_id_idx'),\n partition_name\n );\n RAISE NOTICE 'q: %', q;\n BEGIN\n EXECUTE q;\n EXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', partition_name;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\n END;\n DELETE FROM partitions WHERE collection=NEW.id AND name=partition_name;\n INSERT INTO partitions (collection, name) VALUES (NEW.id, partition_name);\n ELSIF partition_empty THEN\n q := format($q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L)\n PARTITION BY RANGE (datetime);\n $q$,\n partition_name,\n NEW.id\n );\n RAISE NOTICE 'q: %', q;\n BEGIN\n EXECUTE q;\n EXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', partition_name;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\n END;\n DELETE FROM partitions WHERE collection=NEW.id AND name=partition_name;\n ELSE\n RAISE EXCEPTION 'Cannot modify partition % unless empty', partition_name;\n END IF;\n IF loadtemp THEN\n RAISE NOTICE 'Moving data into new partitions.';\n q := format($q$\n WITH p AS (\n SELECT\n collection,\n datetime as datetime,\n end_datetime as end_datetime,\n (partition_name(\n collection,\n datetime\n )).partition_name as name\n FROM changepartitionstaging\n )\n INSERT INTO partitions (collection, datetime_range, end_datetime_range)\n SELECT\n collection,\n tstzrange(min(datetime), max(datetime), '[]') as datetime_range,\n tstzrange(min(end_datetime), max(end_datetime), '[]') as end_datetime_range\n FROM p\n GROUP BY collection, name\n ON CONFLICT (name) DO UPDATE SET\n datetime_range = EXCLUDED.datetime_range,\n end_datetime_range = EXCLUDED.end_datetime_range\n ;\n INSERT INTO %I SELECT * FROM changepartitionstaging;\n DROP TABLE IF EXISTS changepartitionstaging;\n $q$,\n partition_name\n );\n EXECUTE q;\n END IF;\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE TRIGGER collections_trigger AFTER INSERT OR UPDATE ON collections FOR EACH ROW\nEXECUTE FUNCTION collections_trigger_func();\n\nCREATE OR REPLACE FUNCTION partition_collection(collection text, strategy partition_trunc_strategy) RETURNS text AS $$\n UPDATE collections SET partition_trunc=strategy WHERE id=collection RETURNING partition_trunc;\n$$ LANGUAGE SQL;\n\nCREATE TABLE IF NOT EXISTS partitions (\n collection text REFERENCES collections(id),\n name text PRIMARY KEY,\n partition_range tstzrange NOT NULL DEFAULT tstzrange('-infinity'::timestamptz,'infinity'::timestamptz, '[]'),\n datetime_range tstzrange,\n end_datetime_range tstzrange,\n CONSTRAINT prange EXCLUDE USING GIST (\n collection WITH =,\n partition_range WITH &&\n )\n) WITH (FILLFACTOR=90);\nCREATE INDEX partitions_range_idx ON partitions USING GIST(partition_range);\n\n\n\nCREATE OR REPLACE FUNCTION partition_name(\n IN collection text,\n IN dt timestamptz,\n OUT partition_name text,\n OUT partition_range tstzrange\n) AS $$\nDECLARE\n c RECORD;\n parent_name text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', collection;\n END IF;\n parent_name := format('_items_%s', c.key);\n\n\n IF c.partition_trunc = 'year' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYY'));\n ELSIF c.partition_trunc = 'month' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYYMM'));\n ELSE\n partition_name := parent_name;\n partition_range := tstzrange('-infinity'::timestamptz, 'infinity'::timestamptz, '[]');\n END IF;\n IF partition_range IS NULL THEN\n partition_range := tstzrange(\n date_trunc(c.partition_trunc::text, dt),\n date_trunc(c.partition_trunc::text, dt) + concat('1 ', c.partition_trunc)::interval\n );\n END IF;\n RETURN;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\n\nCREATE OR REPLACE FUNCTION partitions_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n q text;\n cq text;\n parent_name text;\n partition_trunc text;\n partition_name text := NEW.name;\n partition_exists boolean := false;\n partition_empty boolean := true;\n partition_range tstzrange;\n datetime_range tstzrange;\n end_datetime_range tstzrange;\n err_context text;\n mindt timestamptz := lower(NEW.datetime_range);\n maxdt timestamptz := upper(NEW.datetime_range);\n minedt timestamptz := lower(NEW.end_datetime_range);\n maxedt timestamptz := upper(NEW.end_datetime_range);\n t_mindt timestamptz;\n t_maxdt timestamptz;\n t_minedt timestamptz;\n t_maxedt timestamptz;\nBEGIN\n RAISE NOTICE 'Partitions Trigger. %', NEW;\n datetime_range := NEW.datetime_range;\n end_datetime_range := NEW.end_datetime_range;\n\n SELECT\n format('_items_%s', key),\n c.partition_trunc::text\n INTO\n parent_name,\n partition_trunc\n FROM pgstac.collections c\n WHERE c.id = NEW.collection;\n SELECT (pgstac.partition_name(NEW.collection, mindt)).* INTO partition_name, partition_range;\n NEW.name := partition_name;\n\n IF partition_range IS NULL OR partition_range = 'empty'::tstzrange THEN\n partition_range := tstzrange('-infinity'::timestamptz, 'infinity'::timestamptz, '[]');\n END IF;\n\n NEW.partition_range := partition_range;\n IF TG_OP = 'UPDATE' THEN\n mindt := least(mindt, lower(OLD.datetime_range));\n maxdt := greatest(maxdt, upper(OLD.datetime_range));\n minedt := least(minedt, lower(OLD.end_datetime_range));\n maxedt := greatest(maxedt, upper(OLD.end_datetime_range));\n NEW.datetime_range := tstzrange(mindt, maxdt, '[]');\n NEW.end_datetime_range := tstzrange(minedt, maxedt, '[]');\n END IF;\n IF TG_OP = 'INSERT' THEN\n\n IF partition_range != tstzrange('-infinity'::timestamptz, 'infinity'::timestamptz, '[]') THEN\n\n RAISE NOTICE '% % %', partition_name, parent_name, partition_range;\n q := format($q$\n CREATE TABLE IF NOT EXISTS %I partition OF %I FOR VALUES FROM (%L) TO (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n $q$,\n partition_name,\n parent_name,\n lower(partition_range),\n upper(partition_range),\n format('%s_pkey', partition_name),\n partition_name\n );\n BEGIN\n EXECUTE q;\n EXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', partition_name;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\n END;\n END IF;\n\n END IF;\n\n -- Update constraints\n EXECUTE format($q$\n SELECT\n min(datetime),\n max(datetime),\n min(end_datetime),\n max(end_datetime)\n FROM %I;\n $q$, partition_name)\n INTO t_mindt, t_maxdt, t_minedt, t_maxedt;\n mindt := least(mindt, t_mindt);\n maxdt := greatest(maxdt, t_maxdt);\n minedt := least(mindt, minedt, t_minedt);\n maxedt := greatest(maxdt, maxedt, t_maxedt);\n\n mindt := date_trunc(coalesce(partition_trunc, 'year'), mindt);\n maxdt := date_trunc(coalesce(partition_trunc, 'year'), maxdt - '1 second'::interval) + concat('1 ',coalesce(partition_trunc, 'year'))::interval;\n minedt := date_trunc(coalesce(partition_trunc, 'year'), minedt);\n maxedt := date_trunc(coalesce(partition_trunc, 'year'), maxedt - '1 second'::interval) + concat('1 ',coalesce(partition_trunc, 'year'))::interval;\n\n\n IF mindt IS NOT NULL AND maxdt IS NOT NULL AND minedt IS NOT NULL AND maxedt IS NOT NULL THEN\n NEW.datetime_range := tstzrange(mindt, maxdt, '[]');\n NEW.end_datetime_range := tstzrange(minedt, maxedt, '[]');\n IF\n TG_OP='UPDATE'\n AND OLD.datetime_range @> NEW.datetime_range\n AND OLD.end_datetime_range @> NEW.end_datetime_range\n THEN\n RAISE NOTICE 'Range unchanged, not updating constraints.';\n ELSE\n\n RAISE NOTICE '\n SETTING CONSTRAINTS\n mindt: %, maxdt: %\n minedt: %, maxedt: %\n ', mindt, maxdt, minedt, maxedt;\n IF partition_trunc IS NULL THEN\n cq := format($q$\n ALTER TABLE %7$I\n DROP CONSTRAINT IF EXISTS %1$I,\n DROP CONSTRAINT IF EXISTS %2$I,\n ADD CONSTRAINT %1$I\n CHECK (\n (datetime >= %3$L)\n AND (datetime <= %4$L)\n AND (end_datetime >= %5$L)\n AND (end_datetime <= %6$L)\n ) NOT VALID\n ;\n ALTER TABLE %7$I\n VALIDATE CONSTRAINT %1$I;\n $q$,\n format('%s_dt', partition_name),\n format('%s_edt', partition_name),\n mindt,\n maxdt,\n minedt,\n maxedt,\n partition_name\n );\n ELSE\n cq := format($q$\n ALTER TABLE %5$I\n DROP CONSTRAINT IF EXISTS %1$I,\n DROP CONSTRAINT IF EXISTS %2$I,\n ADD CONSTRAINT %2$I\n CHECK ((end_datetime >= %3$L) AND (end_datetime <= %4$L)) NOT VALID\n ;\n ALTER TABLE %5$I\n VALIDATE CONSTRAINT %2$I;\n $q$,\n format('%s_dt', partition_name),\n format('%s_edt', partition_name),\n minedt,\n maxedt,\n partition_name\n );\n\n END IF;\n RAISE NOTICE 'Altering Constraints. %', cq;\n EXECUTE cq;\n END IF;\n ELSE\n NEW.datetime_range = NULL;\n NEW.end_datetime_range = NULL;\n\n cq := format($q$\n ALTER TABLE %3$I\n DROP CONSTRAINT IF EXISTS %1$I,\n DROP CONSTRAINT IF EXISTS %2$I,\n ADD CONSTRAINT %1$I\n CHECK ((datetime IS NULL AND end_datetime IS NULL)) NOT VALID\n ;\n ALTER TABLE %3$I\n VALIDATE CONSTRAINT %1$I;\n $q$,\n format('%s_dt', partition_name),\n format('%s_edt', partition_name),\n partition_name\n );\n EXECUTE cq;\n END IF;\n\n RETURN NEW;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER partitions_trigger BEFORE INSERT OR UPDATE ON partitions FOR EACH ROW\nEXECUTE FUNCTION partitions_trigger_func();\n\n\nCREATE OR REPLACE FUNCTION create_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection(data jsonb) RETURNS VOID AS $$\nDECLARE\n out collections%ROWTYPE;\nBEGIN\n UPDATE collections SET content=data WHERE id = data->>'id' RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ON CONFLICT (id) DO\n UPDATE\n SET content=EXCLUDED.content\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION delete_collection(_id text) RETURNS VOID AS $$\nDECLARE\n out collections%ROWTYPE;\nBEGIN\n DELETE FROM collections WHERE id = _id RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION get_collection(id text) RETURNS jsonb AS $$\n SELECT content FROM collections\n WHERE id=$1\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION all_collections() RETURNS jsonb AS $$\n SELECT jsonb_agg(content) FROM collections;\n;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\nCREATE TABLE queryables (\n id bigint GENERATED ALWAYS AS identity PRIMARY KEY,\n name text UNIQUE NOT NULL,\n collection_ids text[], -- used to determine what partitions to create indexes on\n definition jsonb,\n property_path text,\n property_wrapper text,\n property_index_type text\n);\nCREATE INDEX queryables_name_idx ON queryables (name);\nCREATE INDEX queryables_property_wrapper_idx ON queryables (property_wrapper);\n\n\nINSERT INTO queryables (name, definition) VALUES\n('id', '{\"title\": \"Item ID\",\"description\": \"Item identifier\",\"$ref\": \"https://schemas.stacspec.org/v1.0.0/item-spec/json-schema/item.json#/definitions/core/allOf/2/properties/id\"}'),\n('datetime','{\"description\": \"Datetime\",\"type\": \"string\",\"title\": \"Acquired\",\"format\": \"date-time\",\"pattern\": \"(\\\\+00:00|Z)$\"}')\nON CONFLICT DO NOTHING;\n\n\n\nINSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n('eo:cloud_cover','{\"$ref\": \"https://stac-extensions.github.io/eo/v1.0.0/schema.json#/definitions/fieldsproperties/eo:cloud_cover\"}','to_int','BTREE')\nON CONFLICT DO NOTHING;\n\nCREATE OR REPLACE FUNCTION array_to_path(arr text[]) RETURNS text AS $$\n SELECT string_agg(\n quote_literal(v),\n '->'\n ) FROM unnest(arr) v;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\n\n\nCREATE OR REPLACE FUNCTION queryable(\n IN dotpath text,\n OUT path text,\n OUT expression text,\n OUT wrapper text\n) AS $$\nDECLARE\n q RECORD;\n path_elements text[];\nBEGIN\n IF dotpath IN ('id', 'geometry', 'datetime', 'end_datetime', 'collection') THEN\n path := dotpath;\n expression := dotpath;\n wrapper := NULL;\n RETURN;\n END IF;\n SELECT * INTO q FROM queryables WHERE name=dotpath;\n IF q.property_wrapper IS NULL THEN\n IF q.definition->>'type' = 'number' THEN\n wrapper := 'to_float';\n ELSIF q.definition->>'format' = 'date-time' THEN\n wrapper := 'to_tstz';\n ELSE\n wrapper := 'to_text';\n END IF;\n ELSE\n wrapper := q.property_wrapper;\n END IF;\n IF q.property_path IS NOT NULL THEN\n path := q.property_path;\n ELSE\n path_elements := string_to_array(dotpath, '.');\n IF path_elements[1] IN ('links', 'assets', 'stac_version', 'stac_extensions') THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSIF path_elements[1] = 'properties' THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSE\n path := format($F$content->'properties'->%s$F$, array_to_path(path_elements));\n END IF;\n END IF;\n expression := format('%I(%s)', wrapper, path);\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\nCREATE OR REPLACE FUNCTION create_queryable_indexes() RETURNS VOID AS $$\nDECLARE\n queryable RECORD;\n q text;\nBEGIN\n FOR queryable IN\n SELECT\n queryables.id as qid,\n CASE WHEN collections.key IS NULL THEN 'items' ELSE format('_items_%s',collections.key) END AS part,\n property_index_type,\n expression\n FROM\n queryables\n LEFT JOIN collections ON (collections.id = ANY (queryables.collection_ids))\n JOIN LATERAL queryable(queryables.name) ON (queryables.property_index_type IS NOT NULL)\n LOOP\n q := format(\n $q$\n CREATE INDEX IF NOT EXISTS %I ON %I USING %s ((%s));\n $q$,\n format('%s_%s_idx', queryable.part, queryable.qid),\n queryable.part,\n COALESCE(queryable.property_index_type, 'to_text'),\n queryable.expression\n );\n RAISE NOTICE '%',q;\n EXECUTE q;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION queryables_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\nBEGIN\nPERFORM create_queryable_indexes();\nRETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER queryables_trigger AFTER INSERT OR UPDATE ON queryables\nFOR EACH STATEMENT EXECUTE PROCEDURE queryables_trigger_func();\n\nCREATE TRIGGER queryables_collection_trigger AFTER INSERT OR UPDATE ON collections\nFOR EACH STATEMENT EXECUTE PROCEDURE queryables_trigger_func();\nCREATE OR REPLACE FUNCTION parse_dtrange(\n _indate jsonb,\n relative_base timestamptz DEFAULT date_trunc('hour', CURRENT_TIMESTAMP)\n) RETURNS tstzrange AS $$\nDECLARE\n timestrs text[];\n s timestamptz;\n e timestamptz;\nBEGIN\n timestrs :=\n CASE\n WHEN _indate ? 'timestamp' THEN\n ARRAY[_indate->>'timestamp']\n WHEN _indate ? 'interval' THEN\n to_text_array(_indate->'interval')\n WHEN jsonb_typeof(_indate) = 'array' THEN\n to_text_array(_indate)\n ELSE\n regexp_split_to_array(\n _indate->>0,\n '/'\n )\n END;\n RAISE NOTICE 'TIMESTRS %', timestrs;\n IF cardinality(timestrs) = 1 THEN\n IF timestrs[1] ILIKE 'P%' THEN\n RETURN tstzrange(relative_base - upper(timestrs[1])::interval, relative_base, '[)');\n END IF;\n s := timestrs[1]::timestamptz;\n RETURN tstzrange(s, s, '[]');\n END IF;\n\n IF cardinality(timestrs) != 2 THEN\n RAISE EXCEPTION 'Timestamp cannot have more than 2 values';\n END IF;\n\n IF timestrs[1] = '..' THEN\n s := '-infinity'::timestamptz;\n e := timestrs[2]::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] = '..' THEN\n s := timestrs[1]::timestamptz;\n e := 'infinity'::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[1] ILIKE 'P%' AND timestrs[2] NOT ILIKE 'P%' THEN\n e := timestrs[2]::timestamptz;\n s := e - upper(timestrs[1])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] ILIKE 'P%' AND timestrs[1] NOT ILIKE 'P%' THEN\n s := timestrs[1]::timestamptz;\n e := s + upper(timestrs[2])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n s := timestrs[1]::timestamptz;\n e := timestrs[2]::timestamptz;\n\n RETURN tstzrange(s,e,'[)');\n\n RETURN NULL;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT PARALLEL SAFE SET TIME ZONE 'UTC';\n\nCREATE OR REPLACE FUNCTION parse_dtrange(\n _indate text,\n relative_base timestamptz DEFAULT CURRENT_TIMESTAMP\n) RETURNS tstzrange AS $$\n SELECT parse_dtrange(to_jsonb(_indate), relative_base);\n$$ LANGUAGE SQL STABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION temporal_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\n ll text := 'datetime';\n lh text := 'end_datetime';\n rrange tstzrange;\n rl text;\n rh text;\n outq text;\nBEGIN\n rrange := parse_dtrange(args->1);\n RAISE NOTICE 'Constructing temporal query OP: %, ARGS: %, RRANGE: %', op, args, rrange;\n op := lower(op);\n rl := format('%L::timestamptz', lower(rrange));\n rh := format('%L::timestamptz', upper(rrange));\n outq := CASE op\n WHEN 't_before' THEN 'lh < rl'\n WHEN 't_after' THEN 'll > rh'\n WHEN 't_meets' THEN 'lh = rl'\n WHEN 't_metby' THEN 'll = rh'\n WHEN 't_overlaps' THEN 'll < rl AND rl < lh < rh'\n WHEN 't_overlappedby' THEN 'rl < ll < rh AND lh > rh'\n WHEN 't_starts' THEN 'll = rl AND lh < rh'\n WHEN 't_startedby' THEN 'll = rl AND lh > rh'\n WHEN 't_during' THEN 'll > rl AND lh < rh'\n WHEN 't_contains' THEN 'll < rl AND lh > rh'\n WHEN 't_finishes' THEN 'll > rl AND lh = rh'\n WHEN 't_finishedby' THEN 'll < rl AND lh = rh'\n WHEN 't_equals' THEN 'll = rl AND lh = rh'\n WHEN 't_disjoint' THEN 'NOT (ll <= rh AND lh >= rl)'\n WHEN 't_intersects' THEN 'll <= rh AND lh >= rl'\n WHEN 'anyinteracts' THEN 'll <= rh AND lh >= rl'\n END;\n outq := regexp_replace(outq, '\\mll\\M', ll);\n outq := regexp_replace(outq, '\\mlh\\M', lh);\n outq := regexp_replace(outq, '\\mrl\\M', rl);\n outq := regexp_replace(outq, '\\mrh\\M', rh);\n outq := format('(%s)', outq);\n RETURN outq;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\n\n\nCREATE OR REPLACE FUNCTION spatial_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\n geom text;\n j jsonb := args->1;\nBEGIN\n op := lower(op);\n RAISE NOTICE 'Constructing spatial query OP: %, ARGS: %', op, args;\n IF op NOT IN ('s_equals','s_disjoint','s_touches','s_within','s_overlaps','s_crosses','s_intersects','intersects','s_contains') THEN\n RAISE EXCEPTION 'Spatial Operator % Not Supported', op;\n END IF;\n op := regexp_replace(op, '^s_', 'st_');\n IF op = 'intersects' THEN\n op := 'st_intersects';\n END IF;\n -- Convert geometry to WKB string\n IF j ? 'type' AND j ? 'coordinates' THEN\n geom := st_geomfromgeojson(j)::text;\n ELSIF jsonb_typeof(j) = 'array' THEN\n geom := bbox_geom(j)::text;\n END IF;\n\n RETURN format('%s(geometry, %L::geometry)', op, geom);\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION query_to_cql2(q jsonb) RETURNS jsonb AS $$\n-- Translates anything passed in through the deprecated \"query\" into equivalent CQL2\nWITH t AS (\n SELECT key as property, value as ops\n FROM jsonb_each(q)\n), t2 AS (\n SELECT property, (jsonb_each(ops)).*\n FROM t WHERE jsonb_typeof(ops) = 'object'\n UNION ALL\n SELECT property, 'eq', ops\n FROM t WHERE jsonb_typeof(ops) != 'object'\n)\nSELECT\n jsonb_strip_nulls(jsonb_build_object(\n 'op', 'and',\n 'args', jsonb_agg(\n jsonb_build_object(\n 'op', key,\n 'args', jsonb_build_array(\n jsonb_build_object('property',property),\n value\n )\n )\n )\n )\n) as qcql FROM t2\n;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\nCREATE OR REPLACE FUNCTION cql1_to_cql2(j jsonb) RETURNS jsonb AS $$\nDECLARE\n args jsonb;\n ret jsonb;\nBEGIN\n RAISE NOTICE 'CQL1_TO_CQL2: %', j;\n IF j ? 'filter' THEN\n RETURN cql1_to_cql2(j->'filter');\n END IF;\n IF j ? 'property' THEN\n RETURN j;\n END IF;\n IF jsonb_typeof(j) = 'array' THEN\n SELECT jsonb_agg(cql1_to_cql2(el)) INTO args FROM jsonb_array_elements(j) el;\n RETURN args;\n END IF;\n IF jsonb_typeof(j) = 'number' THEN\n RETURN j;\n END IF;\n IF jsonb_typeof(j) = 'string' THEN\n RETURN j;\n END IF;\n\n IF jsonb_typeof(j) = 'object' THEN\n SELECT jsonb_build_object(\n 'op', key,\n 'args', cql1_to_cql2(value)\n ) INTO ret\n FROM jsonb_each(j)\n WHERE j IS NOT NULL;\n RETURN ret;\n END IF;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE STRICT;\n\nCREATE TABLE cql2_ops (\n op text PRIMARY KEY,\n template text,\n types text[]\n);\nINSERT INTO cql2_ops (op, template, types) VALUES\n ('eq', '%s = %s', NULL),\n ('lt', '%s < %s', NULL),\n ('lte', '%s <= %s', NULL),\n ('gt', '%s > %s', NULL),\n ('gte', '%s >= %s', NULL),\n ('le', '%s <= %s', NULL),\n ('ge', '%s >= %s', NULL),\n ('=', '%s = %s', NULL),\n ('<', '%s < %s', NULL),\n ('<=', '%s <= %s', NULL),\n ('>', '%s > %s', NULL),\n ('>=', '%s >= %s', NULL),\n ('like', '%s LIKE %s', NULL),\n ('ilike', '%s ILIKE %s', NULL),\n ('+', '%s + %s', NULL),\n ('-', '%s - %s', NULL),\n ('*', '%s * %s', NULL),\n ('/', '%s / %s', NULL),\n ('in', '%s = ANY (%s)', NULL),\n ('not', 'NOT (%s)', NULL),\n ('between', '%s BETWEEN (%2$s)[1] AND (%2$s)[2]', NULL),\n ('isnull', '%s IS NULL', NULL)\nON CONFLICT (op) DO UPDATE\n SET\n template = EXCLUDED.template;\n;\n\n\nCREATE OR REPLACE FUNCTION cql2_query(j jsonb, wrapper text DEFAULT NULL) RETURNS text AS $$\n#variable_conflict use_variable\nDECLARE\n args jsonb := j->'args';\n arg jsonb;\n op text := lower(j->>'op');\n cql2op RECORD;\n literal text;\nBEGIN\n IF j IS NULL OR (op IS NOT NULL AND args IS NULL) THEN\n RETURN NULL;\n END IF;\n RAISE NOTICE 'CQL2_QUERY: %', j;\n IF j ? 'filter' THEN\n RETURN cql2_query(j->'filter');\n END IF;\n\n IF j ? 'upper' THEN\n RETURN format('upper(%s)', cql2_query(j->'upper'));\n END IF;\n\n IF j ? 'lower' THEN\n RETURN format('lower(%s)', cql2_query(j->'lower'));\n END IF;\n\n -- Temporal Query\n IF op ilike 't_%' or op = 'anyinteracts' THEN\n RETURN temporal_op_query(op, args);\n END IF;\n\n -- If property is a timestamp convert it to text to use with\n -- general operators\n IF j ? 'timestamp' THEN\n RETURN format('%L::timestamptz', to_tstz(j->'timestamp'));\n END IF;\n IF j ? 'interval' THEN\n RAISE EXCEPTION 'Please use temporal operators when using intervals.';\n RETURN NONE;\n END IF;\n\n -- Spatial Query\n IF op ilike 's_%' or op = 'intersects' THEN\n RETURN spatial_op_query(op, args);\n END IF;\n\n\n IF op = 'in' THEN\n RETURN format(\n '%s = ANY (%L)',\n cql2_query(args->0),\n to_text_array(args->1)\n );\n END IF;\n\n\n\n IF op = 'between' THEN\n SELECT (queryable(a->>'property')).wrapper INTO wrapper\n FROM jsonb_array_elements(args) a\n WHERE a ? 'property' LIMIT 1;\n\n RETURN format(\n '%s BETWEEN %s and %s',\n cql2_query(args->0, wrapper),\n cql2_query(args->1->0, wrapper),\n cql2_query(args->1->1, wrapper)\n );\n END IF;\n\n -- Make sure that args is an array and run cql2_query on\n -- each element of the array\n RAISE NOTICE 'ARGS PRE: %', args;\n IF j ? 'args' THEN\n IF jsonb_typeof(args) != 'array' THEN\n args := jsonb_build_array(args);\n END IF;\n\n SELECT (queryable(a->>'property')).wrapper INTO wrapper\n FROM jsonb_array_elements(args) a\n WHERE a ? 'property' LIMIT 1;\n\n SELECT jsonb_agg(cql2_query(a, wrapper))\n INTO args\n FROM jsonb_array_elements(args) a;\n END IF;\n RAISE NOTICE 'ARGS: %', args;\n\n IF op IN ('and', 'or') THEN\n RETURN\n format(\n '(%s)',\n array_to_string(to_text_array(args), format(' %s ', upper(op)))\n );\n END IF;\n\n -- Look up template from cql2_ops\n IF j ? 'op' THEN\n SELECT * INTO cql2op FROM cql2_ops WHERE cql2_ops.op ilike op;\n IF FOUND THEN\n -- If specific index set in queryables for a property cast other arguments to that type\n RETURN format(\n cql2op.template,\n VARIADIC (to_text_array(args))\n );\n ELSE\n RAISE EXCEPTION 'Operator % Not Supported.', op;\n END IF;\n END IF;\n\n\n IF j ? 'property' THEN\n RETURN (queryable(j->>'property')).expression;\n END IF;\n\n IF wrapper IS NOT NULL THEN\n EXECUTE format('SELECT %I(%L)', wrapper, j) INTO literal;\n RAISE NOTICE '% % %',wrapper, j, literal;\n RETURN format('%I(%L)', wrapper, j);\n END IF;\n\n RETURN quote_literal(to_text(j));\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION paging_dtrange(\n j jsonb\n) RETURNS tstzrange AS $$\nDECLARE\n op text;\n filter jsonb := j->'filter';\n dtrange tstzrange := tstzrange('-infinity'::timestamptz,'infinity'::timestamptz);\n sdate timestamptz := '-infinity'::timestamptz;\n edate timestamptz := 'infinity'::timestamptz;\n jpitem jsonb;\nBEGIN\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n END IF;\n IF NOT (filter @? '$.**.op ? (@ == \"or\" || @ == \"not\")') THEN\n FOR jpitem IN SELECT j FROM jsonb_path_query(filter,'strict $.** ? (@.args[*].property == \"datetime\")'::jsonpath) j LOOP\n op := lower(jpitem->>'op');\n dtrange := parse_dtrange(jpitem->'args'->1);\n IF op IN ('<=', 'lt', 'lte', '<', 'le', 't_before') THEN\n sdate := greatest(sdate,'-infinity');\n edate := least(edate, upper(dtrange));\n ELSIF op IN ('>=', '>', 'gt', 'gte', 'ge', 't_after') THEN\n edate := least(edate, 'infinity');\n sdate := greatest(sdate, lower(dtrange));\n ELSIF op IN ('=', 'eq') THEN\n edate := least(edate, upper(dtrange));\n sdate := greatest(sdate, lower(dtrange));\n END IF;\n RAISE NOTICE '2 OP: %, ARGS: %, DTRANGE: %, SDATE: %, EDATE: %', op, jpitem->'args'->1, dtrange, sdate, edate;\n END LOOP;\n END IF;\n IF sdate > edate THEN\n RETURN 'empty'::tstzrange;\n END IF;\n RETURN tstzrange(sdate,edate, '[]');\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT SET TIME ZONE 'UTC';\n\nCREATE OR REPLACE FUNCTION paging_collections(\n IN j jsonb\n) RETURNS text[] AS $$\nDECLARE\n filter jsonb := j->'filter';\n jpitem jsonb;\n op text;\n args jsonb;\n arg jsonb;\n collections text[];\nBEGIN\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n END IF;\n IF NOT (filter @? '$.**.op ? (@ == \"or\" || @ == \"not\")') THEN\n FOR jpitem IN SELECT j FROM jsonb_path_query(filter,'strict $.** ? (@.args[*].property == \"collection\")'::jsonpath) j LOOP\n RAISE NOTICE 'JPITEM: %', jpitem;\n op := jpitem->>'op';\n args := jpitem->'args';\n IF op IN ('=', 'eq', 'in') THEN\n FOR arg IN SELECT a FROM jsonb_array_elements(args) a LOOP\n IF jsonb_typeof(arg) IN ('string', 'array') THEN\n RAISE NOTICE 'arg: %, collections: %', arg, collections;\n IF collections IS NULL OR collections = '{}'::text[] THEN\n collections := to_text_array(arg);\n ELSE\n collections := array_intersection(collections, to_text_array(arg));\n END IF;\n END IF;\n END LOOP;\n END IF;\n END LOOP;\n END IF;\n IF collections = '{}'::text[] THEN\n RETURN NULL;\n END IF;\n RETURN collections;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\nCREATE TABLE items (\n id text NOT NULL,\n geometry geometry NOT NULL,\n collection text NOT NULL,\n datetime timestamptz NOT NULL,\n end_datetime timestamptz NOT NULL,\n content JSONB NOT NULL\n)\nPARTITION BY LIST (collection)\n;\n\nCREATE INDEX \"datetime_idx\" ON items USING BTREE (datetime DESC, end_datetime ASC);\nCREATE INDEX \"geometry_idx\" ON items USING GIST (geometry);\n\nCREATE STATISTICS datetime_stats (dependencies) on datetime, end_datetime from items;\n\n\nALTER TABLE items ADD CONSTRAINT items_collections_fk FOREIGN KEY (collection) REFERENCES collections(id) ON DELETE CASCADE DEFERRABLE;\n\n\nCREATE OR REPLACE FUNCTION content_slim(_item jsonb, _collection jsonb) RETURNS jsonb AS $$\n SELECT\n jsonb_object_agg(\n key,\n CASE\n WHEN\n jsonb_typeof(c.value) = 'object'\n AND\n jsonb_typeof(i.value) = 'object'\n THEN content_slim(i.value, c.value)\n ELSE i.value\n END\n )\n FROM\n jsonb_each(_item) as i\n LEFT JOIN\n jsonb_each(_collection) as c\n USING (key)\n WHERE\n i.value IS DISTINCT FROM c.value\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_slim(_item jsonb) RETURNS jsonb AS $$\n SELECT content_slim(_item - '{id,type,collection,geometry,bbox}'::text[], collection_base_item(_item->>'collection'));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_dehydrate(content jsonb) RETURNS items AS $$\n SELECT\n content->>'id' as id,\n stac_geom(content) as geometry,\n content->>'collection' as collection,\n stac_datetime(content) as datetime,\n stac_end_datetime(content) as end_datetime,\n content_slim(content) as content\n ;\n$$ LANGUAGE SQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION include_field(f text, fields jsonb DEFAULT '{}'::jsonb) RETURNS boolean AS $$\nDECLARE\n includes jsonb := coalesce(fields->'includes', fields->'include', '[]'::jsonb);\n excludes jsonb := coalesce(fields->'excludes', fields->'exclude', '[]'::jsonb);\nBEGIN\n IF f IS NULL THEN\n RETURN NULL;\n ELSIF jsonb_array_length(includes)>0 AND includes ? f THEN\n RETURN TRUE;\n ELSIF jsonb_array_length(excludes)>0 AND excludes ? f THEN\n RETURN FALSE;\n ELSIF jsonb_array_length(includes)>0 AND NOT includes ? f THEN\n RETURN FALSE;\n END IF;\n RETURN TRUE;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION key_filter(IN k text, IN val jsonb, INOUT kf jsonb, OUT include boolean) AS $$\nDECLARE\n includes jsonb := coalesce(kf->'includes', kf->'include', '[]'::jsonb);\n excludes jsonb := coalesce(kf->'excludes', kf->'exclude', '[]'::jsonb);\nBEGIN\n RAISE NOTICE '% % %', k, val, kf;\n\n include := TRUE;\n IF k = 'properties' AND NOT excludes ? 'properties' THEN\n excludes := excludes || '[\"properties\"]';\n include := TRUE;\n RAISE NOTICE 'Prop include %', include;\n ELSIF\n jsonb_array_length(excludes)>0 AND excludes ? k THEN\n include := FALSE;\n ELSIF\n jsonb_array_length(includes)>0 AND NOT includes ? k THEN\n include := FALSE;\n ELSIF\n jsonb_array_length(includes)>0 AND includes ? k THEN\n includes := '[]'::jsonb;\n RAISE NOTICE 'KF: %', kf;\n END IF;\n kf := jsonb_build_object('includes', includes, 'excludes', excludes);\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION strip_assets(a jsonb) RETURNS jsonb AS $$\n WITH t AS (SELECT * FROM jsonb_each(a))\n SELECT jsonb_object_agg(key, value) FROM t\n WHERE value ? 'href';\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION content_hydrate(\n _item jsonb,\n _collection jsonb,\n fields jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\n SELECT\n jsonb_strip_nulls(jsonb_object_agg(\n key,\n CASE\n WHEN key = 'properties' AND include_field('properties', fields) THEN\n i.value\n WHEN key = 'properties' THEN\n content_hydrate(i.value, c.value, kf)\n WHEN\n c.value IS NULL AND key != 'properties'\n THEN i.value\n WHEN\n key = 'assets'\n AND\n jsonb_typeof(c.value) = 'object'\n AND\n jsonb_typeof(i.value) = 'object'\n THEN strip_assets(content_hydrate(i.value, c.value, kf))\n WHEN\n jsonb_typeof(c.value) = 'object'\n AND\n jsonb_typeof(i.value) = 'object'\n THEN content_hydrate(i.value, c.value, kf)\n ELSE coalesce(i.value, c.value)\n END\n ))\n FROM\n jsonb_each(coalesce(_item,'{}'::jsonb)) as i\n FULL JOIN\n jsonb_each(coalesce(_collection,'{}'::jsonb)) as c\n USING (key)\n JOIN LATERAL (\n SELECT kf, include FROM key_filter(key, i.value, fields)\n ) as k ON (include)\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION content_hydrate(_item items, _collection collections, fields jsonb DEFAULT '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\n content jsonb;\n base_item jsonb := _collection.base_item;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry)::jsonb;\n END IF;\n IF include_field('bbox', fields) THEN\n bbox := geom_bbox(_item.geometry)::jsonb;\n END IF;\n output := content_hydrate(\n jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'bbox',bbox,\n 'collection', _item.collection\n ) || _item.content,\n _collection.base_item,\n fields\n );\n\n RETURN output;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_hydrate(_item items, fields jsonb DEFAULT '{}'::jsonb) RETURNS jsonb AS $$\n SELECT content_hydrate(\n _item,\n (SELECT c FROM collections c WHERE id=_item.collection LIMIT 1),\n fields\n );\n$$ LANGUAGE SQL STABLE;\n\n\nCREATE UNLOGGED TABLE items_staging (\n content JSONB NOT NULL\n);\nCREATE UNLOGGED TABLE items_staging_ignore (\n content JSONB NOT NULL\n);\nCREATE UNLOGGED TABLE items_staging_upsert (\n content JSONB NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION items_staging_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p record;\n _partitions text[];\n ts timestamptz := clock_timestamp();\nBEGIN\n RAISE NOTICE 'Creating Partitions. %', clock_timestamp() - ts;\n WITH ranges AS (\n SELECT\n n.content->>'collection' as collection,\n stac_daterange(n.content->'properties') as dtr\n FROM newdata n\n ), p AS (\n SELECT\n collection,\n lower(dtr) as datetime,\n upper(dtr) as end_datetime,\n (partition_name(\n collection,\n lower(dtr)\n )).partition_name as name\n FROM ranges\n )\n INSERT INTO partitions (collection, datetime_range, end_datetime_range)\n SELECT\n collection,\n tstzrange(min(datetime), max(datetime), '[]') as datetime_range,\n tstzrange(min(end_datetime), max(end_datetime), '[]') as end_datetime_range\n FROM p\n GROUP BY collection, name\n ON CONFLICT (name) DO UPDATE SET\n datetime_range = EXCLUDED.datetime_range,\n end_datetime_range = EXCLUDED.end_datetime_range\n ;\n\n RAISE NOTICE 'Doing the insert. %', clock_timestamp() - ts;\n IF TG_TABLE_NAME = 'items_staging' THEN\n INSERT INTO items\n SELECT\n (content_dehydrate(content)).*\n FROM newdata;\n DELETE FROM items_staging;\n ELSIF TG_TABLE_NAME = 'items_staging_ignore' THEN\n INSERT INTO items\n SELECT\n (content_dehydrate(content)).*\n FROM newdata\n ON CONFLICT DO NOTHING;\n DELETE FROM items_staging_ignore;\n ELSIF TG_TABLE_NAME = 'items_staging_upsert' THEN\n WITH staging_formatted AS (\n SELECT (content_dehydrate(content)).* FROM newdata\n ), deletes AS (\n DELETE FROM items i USING staging_formatted s\n WHERE\n i.id = s.id\n AND i.collection = s.collection\n AND i IS DISTINCT FROM s\n RETURNING i.id, i.collection\n )\n INSERT INTO items\n SELECT s.* FROM\n staging_formatted s\n JOIN deletes d\n USING (id, collection);\n DELETE FROM items_staging_upsert;\n END IF;\n RAISE NOTICE 'Done. %', clock_timestamp() - ts;\n\n RETURN NULL;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER items_staging_insert_trigger AFTER INSERT ON items_staging REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\nCREATE TRIGGER items_staging_insert_ignore_trigger AFTER INSERT ON items_staging_ignore REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\nCREATE TRIGGER items_staging_insert_upsert_trigger AFTER INSERT ON items_staging_upsert REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\n\n\n\nCREATE OR REPLACE FUNCTION item_by_id(_id text, _collection text DEFAULT NULL) RETURNS items AS\n$$\nDECLARE\n i items%ROWTYPE;\nBEGIN\n SELECT * INTO i FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection) LIMIT 1;\n RETURN i;\nEND;\n$$ LANGUAGE PLPGSQL STABLE SECURITY DEFINER SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION get_item(_id text, _collection text DEFAULT NULL) RETURNS jsonb AS $$\n SELECT content_hydrate(items) FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection);\n$$ LANGUAGE SQL STABLE SECURITY DEFINER SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION delete_item(_id text, _collection text DEFAULT NULL) RETURNS VOID AS $$\nDECLARE\nout items%ROWTYPE;\nBEGIN\n DELETE FROM items WHERE id = _id AND (_collection IS NULL OR collection=_collection) RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n--/*\nCREATE OR REPLACE FUNCTION create_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION update_item(content jsonb) RETURNS VOID AS $$\nDECLARE\n old items %ROWTYPE;\n out items%ROWTYPE;\nBEGIN\n SELECT delete_item(content->>'id', content->>'collection');\n SELECT create_item(content);\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION create_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION collection_bbox(id text) RETURNS jsonb AS $$\n SELECT (replace(replace(replace(st_extent(geometry)::text,'BOX(','[['),')',']]'),' ',','))::jsonb\n FROM items WHERE collection=$1;\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION collection_temporal_extent(id text) RETURNS jsonb AS $$\n SELECT to_jsonb(array[array[min(datetime)::text, max(datetime)::text]])\n FROM items WHERE collection=$1;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection_extents() RETURNS VOID AS $$\nUPDATE collections SET\n content = content ||\n jsonb_build_object(\n 'extent', jsonb_build_object(\n 'spatial', jsonb_build_object(\n 'bbox', collection_bbox(collections.id)\n ),\n 'temporal', jsonb_build_object(\n 'interval', collection_temporal_extent(collections.id)\n )\n )\n )\n;\n$$ LANGUAGE SQL;\nCREATE VIEW partition_steps AS\nSELECT\n name,\n date_trunc('month',lower(datetime_range)) as sdate,\n date_trunc('month', upper(datetime_range)) + '1 month'::interval as edate\n FROM partitions WHERE datetime_range IS NOT NULL AND datetime_range != 'empty'::tstzrange\n ORDER BY datetime_range ASC\n;\n\nCREATE OR REPLACE FUNCTION chunker(\n IN _where text,\n OUT s timestamptz,\n OUT e timestamptz\n) RETURNS SETOF RECORD AS $$\nDECLARE\n explain jsonb;\nBEGIN\n IF _where IS NULL THEN\n _where := ' TRUE ';\n END IF;\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s;', _where)\n INTO explain;\n\n RETURN QUERY\n WITH t AS (\n SELECT j->>0 as p FROM\n jsonb_path_query(\n explain,\n 'strict $.**.\"Relation Name\" ? (@ != null)'\n ) j\n ),\n parts AS (\n SELECT sdate, edate FROM t JOIN partition_steps ON (t.p = name)\n ),\n times AS (\n SELECT sdate FROM parts\n UNION\n SELECT edate FROM parts\n ),\n uniq AS (\n SELECT DISTINCT sdate FROM times ORDER BY sdate\n ),\n last AS (\n SELECT sdate, lead(sdate, 1) over () as edate FROM uniq\n )\n SELECT sdate, edate FROM last WHERE edate IS NOT NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION partition_queries(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN partitions text[] DEFAULT NULL\n) RETURNS SETOF text AS $$\nDECLARE\n query text;\n sdate timestamptz;\n edate timestamptz;\nBEGIN\nIF _where IS NULL OR trim(_where) = '' THEN\n _where = ' TRUE ';\nEND IF;\nRAISE NOTICE 'Getting chunks for % %', _where, _orderby;\nIF _orderby ILIKE 'datetime d%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 DESC LOOP\n RETURN NEXT format($q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n $q$,\n sdate,\n edate,\n _where,\n _orderby\n );\n END LOOP;\nELSIF _orderby ILIKE 'datetime a%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 ASC LOOP\n RETURN NEXT format($q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n $q$,\n sdate,\n edate,\n _where,\n _orderby\n );\n END LOOP;\nELSE\n query := format($q$\n SELECT * FROM items\n WHERE %s\n ORDER BY %s\n $q$, _where, _orderby\n );\n\n RETURN NEXT query;\n RETURN;\nEND IF;\n\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION partition_query_view(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN _limit int DEFAULT 10\n) RETURNS text AS $$\n WITH p AS (\n SELECT * FROM partition_queries(_where, _orderby) p\n )\n SELECT\n CASE WHEN EXISTS (SELECT 1 FROM p) THEN\n (SELECT format($q$\n SELECT * FROM (\n %s\n ) total LIMIT %s\n $q$,\n string_agg(\n format($q$ SELECT * FROM ( %s ) AS sub $q$, p),\n '\n UNION ALL\n '\n ),\n _limit\n ))\n ELSE NULL\n END FROM p;\n$$ LANGUAGE SQL IMMUTABLE;\n\n\n\n\nCREATE OR REPLACE FUNCTION stac_search_to_where(j jsonb) RETURNS text AS $$\nDECLARE\n where_segments text[];\n _where text;\n dtrange tstzrange;\n collections text[];\n geom geometry;\n sdate timestamptz;\n edate timestamptz;\n filterlang text;\n filter jsonb := j->'filter';\nBEGIN\n IF j ? 'ids' THEN\n where_segments := where_segments || format('id = ANY (%L) ', to_text_array(j->'ids'));\n END IF;\n\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n where_segments := where_segments || format('collection = ANY (%L) ', collections);\n END IF;\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n\n where_segments := where_segments || format(' datetime <= %L::timestamptz AND end_datetime >= %L::timestamptz ',\n edate,\n sdate\n );\n END IF;\n\n geom := stac_geom(j);\n IF geom IS NOT NULL THEN\n where_segments := where_segments || format('st_intersects(geometry, %L)',geom);\n END IF;\n\n filterlang := COALESCE(\n j->>'filter-lang',\n get_setting('default-filter-lang', j->'conf')\n );\n IF NOT filter @? '$.**.op' THEN\n filterlang := 'cql-json';\n END IF;\n\n IF filterlang NOT IN ('cql-json','cql2-json') AND j ? 'filter' THEN\n RAISE EXCEPTION '% is not a supported filter-lang. Please use cql-json or cql2-json.', filterlang;\n END IF;\n\n IF j ? 'query' AND j ? 'filter' THEN\n RAISE EXCEPTION 'Can only use either query or filter at one time.';\n END IF;\n\n IF j ? 'query' THEN\n filter := query_to_cql2(j->'query');\n ELSIF filterlang = 'cql-json' THEN\n filter := cql1_to_cql2(filter);\n END IF;\n RAISE NOTICE 'FILTER: %', filter;\n where_segments := where_segments || cql2_query(filter);\n IF cardinality(where_segments) < 1 THEN\n RETURN ' TRUE ';\n END IF;\n\n _where := array_to_string(array_remove(where_segments, NULL), ' AND ');\n\n IF _where IS NULL OR BTRIM(_where) = '' THEN\n RETURN ' TRUE ';\n END IF;\n RETURN _where;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION parse_sort_dir(_dir text, reverse boolean default false) RETURNS text AS $$\n WITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n ) SELECT\n CASE\n WHEN NOT reverse THEN d\n WHEN d = 'ASC' THEN 'DESC'\n WHEN d = 'DESC' THEN 'ASC'\n END\n FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION sort_dir_to_op(_dir text, prev boolean default false) RETURNS text AS $$\n WITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n ) SELECT\n CASE\n WHEN d = 'ASC' AND prev THEN '<='\n WHEN d = 'DESC' AND prev THEN '>='\n WHEN d = 'ASC' THEN '>='\n WHEN d = 'DESC' THEN '<='\n END\n FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION sort_sqlorderby(\n _search jsonb DEFAULT NULL,\n reverse boolean DEFAULT FALSE\n) RETURNS text AS $$\n WITH sortby AS (\n SELECT coalesce(_search->'sortby','[{\"field\":\"datetime\", \"direction\":\"desc\"}]') as sort\n ), withid AS (\n SELECT CASE\n WHEN sort @? '$[*] ? (@.field == \"id\")' THEN sort\n ELSE sort || '[{\"field\":\"id\", \"direction\":\"desc\"}]'::jsonb\n END as sort\n FROM sortby\n ), withid_rows AS (\n SELECT jsonb_array_elements(sort) as value FROM withid\n ),sorts AS (\n SELECT\n coalesce(\n -- field_orderby((items_path(value->>'field')).path_txt),\n (queryable(value->>'field')).expression\n ) as key,\n parse_sort_dir(value->>'direction', reverse) as dir\n FROM withid_rows\n )\n SELECT array_to_string(\n array_agg(concat(key, ' ', dir)),\n ', '\n ) FROM sorts;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_sort_dir(sort_item jsonb) RETURNS text AS $$\n SELECT CASE WHEN sort_item->>'direction' ILIKE 'desc%' THEN 'DESC' ELSE 'ASC' END;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION get_token_filter(_search jsonb = '{}'::jsonb, token_rec jsonb DEFAULT NULL) RETURNS text AS $$\nDECLARE\n token_id text;\n filters text[] := '{}'::text[];\n prev boolean := TRUE;\n field text;\n dir text;\n sort record;\n orfilters text[] := '{}'::text[];\n andfilters text[] := '{}'::text[];\n output text;\n token_where text;\nBEGIN\n RAISE NOTICE 'Getting Token Filter. % %', _search, token_rec;\n -- If no token provided return NULL\n IF token_rec IS NULL THEN\n IF NOT (_search ? 'token' AND\n (\n (_search->>'token' ILIKE 'prev:%')\n OR\n (_search->>'token' ILIKE 'next:%')\n )\n ) THEN\n RETURN NULL;\n END IF;\n prev := (_search->>'token' ILIKE 'prev:%');\n token_id := substr(_search->>'token', 6);\n SELECT to_jsonb(items) INTO token_rec\n FROM items WHERE id=token_id;\n END IF;\n RAISE NOTICE 'TOKEN ID: % %', token_rec, token_rec->'id';\n\n CREATE TEMP TABLE sorts (\n _row int GENERATED ALWAYS AS IDENTITY NOT NULL,\n _field text PRIMARY KEY,\n _dir text NOT NULL,\n _val text\n ) ON COMMIT DROP;\n\n -- Make sure we only have distinct columns to sort with taking the first one we get\n INSERT INTO sorts (_field, _dir)\n SELECT\n (queryable(value->>'field')).expression,\n get_sort_dir(value)\n FROM\n jsonb_array_elements(coalesce(_search->'sortby','[{\"field\":\"datetime\",\"direction\":\"desc\"}]'))\n ON CONFLICT DO NOTHING\n ;\n RAISE NOTICE 'sorts 1: %', (SELECT jsonb_agg(to_json(sorts)) FROM sorts);\n -- Get the first sort direction provided. As the id is a primary key, if there are any\n -- sorts after id they won't do anything, so make sure that id is the last sort item.\n SELECT _dir INTO dir FROM sorts ORDER BY _row ASC LIMIT 1;\n IF EXISTS (SELECT 1 FROM sorts WHERE _field = 'id') THEN\n DELETE FROM sorts WHERE _row > (SELECT _row FROM sorts WHERE _field = 'id' ORDER BY _row ASC);\n ELSE\n INSERT INTO sorts (_field, _dir) VALUES ('id', dir);\n END IF;\n\n -- Add value from looked up item to the sorts table\n UPDATE sorts SET _val=quote_literal(token_rec->>_field);\n\n -- Check if all sorts are the same direction and use row comparison\n -- to filter\n RAISE NOTICE 'sorts 2: %', (SELECT jsonb_agg(to_json(sorts)) FROM sorts);\n\n IF (SELECT count(DISTINCT _dir) FROM sorts) = 1 THEN\n SELECT format(\n '(%s) %s (%s)',\n concat_ws(', ', VARIADIC array_agg(quote_ident(_field))),\n CASE WHEN (prev AND dir = 'ASC') OR (NOT prev AND dir = 'DESC') THEN '<' ELSE '>' END,\n concat_ws(', ', VARIADIC array_agg(_val))\n ) INTO output FROM sorts\n WHERE token_rec ? _field\n ;\n ELSE\n FOR sort IN SELECT * FROM sorts ORDER BY _row asc LOOP\n RAISE NOTICE 'SORT: %', sort;\n IF sort._row = 1 THEN\n orfilters := orfilters || format('(%s %s %s)',\n quote_ident(sort._field),\n CASE WHEN (prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC') THEN '<' ELSE '>' END,\n sort._val\n );\n ELSE\n orfilters := orfilters || format('(%s AND %s %s %s)',\n array_to_string(andfilters, ' AND '),\n quote_ident(sort._field),\n CASE WHEN (prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC') THEN '<' ELSE '>' END,\n sort._val\n );\n\n END IF;\n andfilters := andfilters || format('%s = %s',\n quote_ident(sort._field),\n sort._val\n );\n END LOOP;\n output := array_to_string(orfilters, ' OR ');\n END IF;\n DROP TABLE IF EXISTS sorts;\n token_where := concat('(',coalesce(output,'true'),')');\n IF trim(token_where) = '' THEN\n token_where := NULL;\n END IF;\n RAISE NOTICE 'TOKEN_WHERE: |%|',token_where;\n RETURN token_where;\n END;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION search_tohash(jsonb) RETURNS jsonb AS $$\n SELECT $1 - '{token,limit,context,includes,excludes}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION search_hash(jsonb, jsonb) RETURNS text AS $$\n SELECT md5(concat(search_tohash($1)::text,$2::text));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TABLE IF NOT EXISTS searches(\n hash text GENERATED ALWAYS AS (search_hash(search, metadata)) STORED PRIMARY KEY,\n search jsonb NOT NULL,\n _where text,\n orderby text,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n metadata jsonb DEFAULT '{}'::jsonb NOT NULL\n);\nCREATE TABLE IF NOT EXISTS search_wheres(\n id bigint generated always as identity primary key,\n _where text NOT NULL,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n statslastupdated timestamptz,\n estimated_count bigint,\n estimated_cost float,\n time_to_estimate float,\n total_count bigint,\n time_to_count float,\n partitions text[]\n);\n\nCREATE INDEX IF NOT EXISTS search_wheres_partitions ON search_wheres USING GIN (partitions);\nCREATE UNIQUE INDEX IF NOT EXISTS search_wheres_where ON search_wheres ((md5(_where)));\n\nCREATE OR REPLACE FUNCTION where_stats(inwhere text, updatestats boolean default false, conf jsonb default null) RETURNS search_wheres AS $$\nDECLARE\n t timestamptz;\n i interval;\n explain_json jsonb;\n partitions text[];\n sw search_wheres%ROWTYPE;\n inwhere_hash text := md5(inwhere);\n _context text := lower(context(conf));\n _stats_ttl interval := context_stats_ttl(conf);\n _estimated_cost float := context_estimated_cost(conf);\n _estimated_count int := context_estimated_count(conf);\nBEGIN\n IF _context = 'off' THEN\n sw._where := inwhere;\n return sw;\n END IF;\n\n SELECT * INTO sw FROM search_wheres WHERE md5(_where)=inwhere_hash FOR UPDATE;\n\n -- Update statistics if explicitly set, if statistics do not exist, or statistics ttl has expired\n IF NOT updatestats THEN\n RAISE NOTICE 'Checking if update is needed for: % .', inwhere;\n RAISE NOTICE 'Stats Last Updated: %', sw.statslastupdated;\n RAISE NOTICE 'TTL: %, Age: %', _stats_ttl, now() - sw.statslastupdated;\n RAISE NOTICE 'Context: %, Existing Total: %', _context, sw.total_count;\n IF\n sw.statslastupdated IS NULL\n OR (now() - sw.statslastupdated) > _stats_ttl\n OR (context(conf) != 'off' AND sw.total_count IS NULL)\n THEN\n updatestats := TRUE;\n END IF;\n END IF;\n\n sw._where := inwhere;\n sw.lastused := now();\n sw.usecount := coalesce(sw.usecount,0) + 1;\n\n IF NOT updatestats THEN\n UPDATE search_wheres SET\n lastused = sw.lastused,\n usecount = sw.usecount\n WHERE md5(_where) = inwhere_hash\n RETURNING * INTO sw\n ;\n RETURN sw;\n END IF;\n\n -- Use explain to get estimated count/cost and a list of the partitions that would be hit by the query\n t := clock_timestamp();\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s', inwhere)\n INTO explain_json;\n RAISE NOTICE 'Time for just the explain: %', clock_timestamp() - t;\n i := clock_timestamp() - t;\n\n sw.statslastupdated := now();\n sw.estimated_count := explain_json->0->'Plan'->'Plan Rows';\n sw.estimated_cost := explain_json->0->'Plan'->'Total Cost';\n sw.time_to_estimate := extract(epoch from i);\n\n RAISE NOTICE 'ESTIMATED_COUNT: % < %', sw.estimated_count, _estimated_count;\n RAISE NOTICE 'ESTIMATED_COST: % < %', sw.estimated_cost, _estimated_cost;\n\n -- Do a full count of rows if context is set to on or if auto is set and estimates are low enough\n IF\n _context = 'on'\n OR\n ( _context = 'auto' AND\n (\n sw.estimated_count < _estimated_count\n AND\n sw.estimated_cost < _estimated_cost\n )\n )\n THEN\n t := clock_timestamp();\n RAISE NOTICE 'Calculating actual count...';\n EXECUTE format(\n 'SELECT count(*) FROM items WHERE %s',\n inwhere\n ) INTO sw.total_count;\n i := clock_timestamp() - t;\n RAISE NOTICE 'Actual Count: % -- %', sw.total_count, i;\n sw.time_to_count := extract(epoch FROM i);\n ELSE\n sw.total_count := NULL;\n sw.time_to_count := NULL;\n END IF;\n\n\n INSERT INTO search_wheres\n (_where, lastused, usecount, statslastupdated, estimated_count, estimated_cost, time_to_estimate, partitions, total_count, time_to_count)\n SELECT sw._where, sw.lastused, sw.usecount, sw.statslastupdated, sw.estimated_count, sw.estimated_cost, sw.time_to_estimate, sw.partitions, sw.total_count, sw.time_to_count\n ON CONFLICT ((md5(_where)))\n DO UPDATE\n SET\n lastused = sw.lastused,\n usecount = sw.usecount,\n statslastupdated = sw.statslastupdated,\n estimated_count = sw.estimated_count,\n estimated_cost = sw.estimated_cost,\n time_to_estimate = sw.time_to_estimate,\n total_count = sw.total_count,\n time_to_count = sw.time_to_count\n ;\n RETURN sw;\nEND;\n$$ LANGUAGE PLPGSQL ;\n\n\n\nDROP FUNCTION IF EXISTS search_query;\nCREATE OR REPLACE FUNCTION search_query(\n _search jsonb = '{}'::jsonb,\n updatestats boolean = false,\n _metadata jsonb = '{}'::jsonb\n) RETURNS searches AS $$\nDECLARE\n search searches%ROWTYPE;\n pexplain jsonb;\n t timestamptz;\n i interval;\nBEGIN\n SELECT * INTO search FROM searches\n WHERE hash=search_hash(_search, _metadata) FOR UPDATE;\n\n -- Calculate the where clause if not already calculated\n IF search._where IS NULL THEN\n search._where := stac_search_to_where(_search);\n END IF;\n\n -- Calculate the order by clause if not already calculated\n IF search.orderby IS NULL THEN\n search.orderby := sort_sqlorderby(_search);\n END IF;\n\n PERFORM where_stats(search._where, updatestats, _search->'conf');\n\n search.lastused := now();\n search.usecount := coalesce(search.usecount, 0) + 1;\n INSERT INTO searches (search, _where, orderby, lastused, usecount, metadata)\n VALUES (_search, search._where, search.orderby, search.lastused, search.usecount, _metadata)\n ON CONFLICT (hash) DO\n UPDATE SET\n _where = EXCLUDED._where,\n orderby = EXCLUDED.orderby,\n lastused = EXCLUDED.lastused,\n usecount = EXCLUDED.usecount,\n metadata = EXCLUDED.metadata\n RETURNING * INTO search\n ;\n RETURN search;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION search(_search jsonb = '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n searches searches%ROWTYPE;\n _where text;\n token_where text;\n full_where text;\n orderby text;\n query text;\n token_type text := substr(_search->>'token',1,4);\n _limit int := coalesce((_search->>'limit')::int, 10);\n curs refcursor;\n cntr int := 0;\n iter_record items%ROWTYPE;\n first_record jsonb;\n last_record jsonb;\n out_records jsonb := '[]'::jsonb;\n prev_query text;\n next text;\n prev_id text;\n has_next boolean := false;\n has_prev boolean := false;\n prev text;\n total_count bigint;\n context jsonb;\n collection jsonb;\n includes text[];\n excludes text[];\n exit_flag boolean := FALSE;\n batches int := 0;\n timer timestamptz := clock_timestamp();\n pstart timestamptz;\n pend timestamptz;\n pcurs refcursor;\n search_where search_wheres%ROWTYPE;\n id text;\nBEGIN\nCREATE TEMP TABLE results (content jsonb) ON COMMIT DROP;\n-- if ids is set, short circuit and just use direct ids query for each id\n-- skip any paging or caching\n-- hard codes ordering in the same order as the array of ids\nIF _search ? 'ids' THEN\n INSERT INTO results\n SELECT content_hydrate(items, _search->'fields')\n FROM items WHERE\n items.id = ANY(to_text_array(_search->'ids'))\n AND\n CASE WHEN _search ? 'collections' THEN\n items.collection = ANY(to_text_array(_search->'collections'))\n ELSE TRUE\n END\n ORDER BY items.datetime desc, items.id desc\n ;\n SELECT INTO total_count count(*) FROM results;\nELSE\n searches := search_query(_search);\n _where := searches._where;\n orderby := searches.orderby;\n search_where := where_stats(_where);\n total_count := coalesce(search_where.total_count, search_where.estimated_count);\n\n IF token_type='prev' THEN\n token_where := get_token_filter(_search, null::jsonb);\n orderby := sort_sqlorderby(_search, TRUE);\n END IF;\n IF token_type='next' THEN\n token_where := get_token_filter(_search, null::jsonb);\n END IF;\n\n full_where := concat_ws(' AND ', _where, token_where);\n RAISE NOTICE 'FULL QUERY % %', full_where, clock_timestamp()-timer;\n timer := clock_timestamp();\n\n FOR query IN SELECT partition_queries(full_where, orderby, search_where.partitions) LOOP\n timer := clock_timestamp();\n query := format('%s LIMIT %s', query, _limit + 1);\n RAISE NOTICE 'Partition Query: %', query;\n batches := batches + 1;\n -- curs = create_cursor(query);\n OPEN curs FOR EXECUTE query;\n LOOP\n FETCH curs into iter_record;\n EXIT WHEN NOT FOUND;\n cntr := cntr + 1;\n last_record := content_hydrate(iter_record, _search->'fields');\n IF cntr = 1 THEN\n first_record := last_record;\n END IF;\n IF cntr <= _limit THEN\n INSERT INTO results (content) VALUES (last_record);\n ELSIF cntr > _limit THEN\n has_next := true;\n exit_flag := true;\n EXIT;\n END IF;\n END LOOP;\n CLOSE curs;\n RAISE NOTICE 'Query took %.', clock_timestamp()-timer;\n timer := clock_timestamp();\n EXIT WHEN exit_flag;\n END LOOP;\n RAISE NOTICE 'Scanned through % partitions.', batches;\nEND IF;\n\nSELECT jsonb_agg(content) INTO out_records FROM results WHERE content is not NULL;\n\nDROP TABLE results;\n\n\n-- Flip things around if this was the result of a prev token query\nIF token_type='prev' THEN\n out_records := flip_jsonb_array(out_records);\n first_record := last_record;\nEND IF;\n\n-- If this query has a token, see if there is data before the first record\nIF _search ? 'token' THEN\n prev_query := format(\n 'SELECT 1 FROM items WHERE %s LIMIT 1',\n concat_ws(\n ' AND ',\n _where,\n trim(get_token_filter(_search, to_jsonb(content_dehydrate(first_record))))\n )\n );\n RAISE NOTICE 'Query to get previous record: % --- %', prev_query, first_record;\n EXECUTE prev_query INTO has_prev;\n IF FOUND and has_prev IS NOT NULL THEN\n RAISE NOTICE 'Query results from prev query: %', has_prev;\n has_prev := TRUE;\n END IF;\nEND IF;\nhas_prev := COALESCE(has_prev, FALSE);\n\nIF has_prev THEN\n prev := out_records->0->>'id';\nEND IF;\nIF has_next OR token_type='prev' THEN\n next := out_records->-1->>'id';\nEND IF;\n\nIF context(_search->'conf') != 'off' THEN\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'matched', total_count,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\nELSE\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\nEND IF;\n\ncollection := jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb),\n 'next', next,\n 'prev', prev,\n 'context', context\n);\n\nRETURN collection;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER SET SEARCH_PATH TO pgstac, public;\n\n\nCREATE OR REPLACE FUNCTION search_cursor(_search jsonb = '{}'::jsonb) RETURNS refcursor AS $$\nDECLARE\n curs refcursor;\n searches searches%ROWTYPE;\n _where text;\n _orderby text;\n q text;\n\nBEGIN\n searches := search_query(_search);\n _where := searches._where;\n _orderby := searches.orderby;\n\n OPEN curs FOR\n WITH p AS (\n SELECT * FROM partition_queries(_where, _orderby) p\n )\n SELECT\n CASE WHEN EXISTS (SELECT 1 FROM p) THEN\n (SELECT format($q$\n SELECT * FROM (\n %s\n ) total\n $q$,\n string_agg(\n format($q$ SELECT * FROM ( %s ) AS sub $q$, p),\n '\n UNION ALL\n '\n )\n ))\n ELSE NULL\n END FROM p;\n RETURN curs;\nEND;\n$$ LANGUAGE PLPGSQL;\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION tileenvelope(zoom int, x int, y int) RETURNS geometry AS $$\nWITH t AS (\n SELECT\n 20037508.3427892 as merc_max,\n -20037508.3427892 as merc_min,\n (2 * 20037508.3427892) / (2 ^ zoom) as tile_size\n)\nSELECT st_makeenvelope(\n merc_min + (tile_size * x),\n merc_max - (tile_size * (y + 1)),\n merc_min + (tile_size * (x + 1)),\n merc_max - (tile_size * y),\n 3857\n) FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;DROP FUNCTION IF EXISTS mercgrid;\n\n\nCREATE OR REPLACE FUNCTION ftime() RETURNS interval as $$\nSELECT age(clock_timestamp(), transaction_timestamp());\n$$ LANGUAGE SQL;\nSET SEARCH_PATH to pgstac, public;\n\nDROP FUNCTION IF EXISTS geometrysearch;\nCREATE OR REPLACE FUNCTION geometrysearch(\n IN geom geometry,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE, -- Return as soon as the passed in geometry is full covered\n IN skipcovered boolean DEFAULT TRUE -- Skip any items that would show up completely under the previous items\n) RETURNS jsonb AS $$\nDECLARE\n search searches%ROWTYPE;\n curs refcursor;\n _where text;\n query text;\n iter_record items%ROWTYPE;\n out_records jsonb := '{}'::jsonb[];\n exit_flag boolean := FALSE;\n counter int := 1;\n scancounter int := 1;\n remaining_limit int := _scanlimit;\n tilearea float;\n unionedgeom geometry;\n clippedgeom geometry;\n unionedgeom_area float := 0;\n prev_area float := 0;\n excludes text[];\n includes text[];\n\nBEGIN\n DROP TABLE IF EXISTS pgstac_results;\n CREATE TEMP TABLE pgstac_results (content jsonb) ON COMMIT DROP;\n\n -- If skipcovered is true then you will always want to exit when the passed in geometry is full\n IF skipcovered THEN\n exitwhenfull := TRUE;\n END IF;\n\n SELECT * INTO search FROM searches WHERE hash=queryhash;\n\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Search with Query Hash % Not Found', queryhash;\n END IF;\n\n tilearea := st_area(geom);\n _where := format('%s AND st_intersects(geometry, %L::geometry)', search._where, geom);\n\n\n FOR query IN SELECT * FROM partition_queries(_where, search.orderby) LOOP\n query := format('%s LIMIT %L', query, remaining_limit);\n RAISE NOTICE '%', query;\n OPEN curs FOR EXECUTE query;\n LOOP\n FETCH curs INTO iter_record;\n EXIT WHEN NOT FOUND;\n IF exitwhenfull OR skipcovered THEN -- If we are not using exitwhenfull or skipcovered, we do not need to do expensive geometry operations\n clippedgeom := st_intersection(geom, iter_record.geometry);\n\n IF unionedgeom IS NULL THEN\n unionedgeom := clippedgeom;\n ELSE\n unionedgeom := st_union(unionedgeom, clippedgeom);\n END IF;\n\n unionedgeom_area := st_area(unionedgeom);\n\n IF skipcovered AND prev_area = unionedgeom_area THEN\n scancounter := scancounter + 1;\n CONTINUE;\n END IF;\n\n prev_area := unionedgeom_area;\n\n RAISE NOTICE '% % % %', unionedgeom_area/tilearea, counter, scancounter, ftime();\n END IF;\n RAISE NOTICE '% %', iter_record, content_hydrate(iter_record, fields);\n INSERT INTO pgstac_results (content) VALUES (content_hydrate(iter_record, fields));\n\n IF counter >= _limit\n OR scancounter > _scanlimit\n OR ftime() > _timelimit\n OR (exitwhenfull AND unionedgeom_area >= tilearea)\n THEN\n exit_flag := TRUE;\n EXIT;\n END IF;\n counter := counter + 1;\n scancounter := scancounter + 1;\n\n END LOOP;\n CLOSE curs;\n EXIT WHEN exit_flag;\n remaining_limit := _scanlimit - scancounter;\n END LOOP;\n\n SELECT jsonb_agg(content) INTO out_records FROM pgstac_results WHERE content IS NOT NULL;\n\n RETURN jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb)\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDROP FUNCTION IF EXISTS geojsonsearch;\nCREATE OR REPLACE FUNCTION geojsonsearch(\n IN geojson jsonb,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_geomfromgeojson(geojson),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS xyzsearch;\nCREATE OR REPLACE FUNCTION xyzsearch(\n IN _x int,\n IN _y int,\n IN _z int,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_transform(tileenvelope(_z, _x, _y), 4326),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nGRANT ALL ON SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac to pgstac_admin;\n\n-- pgstac_read role limited to using function apis\nGRANT EXECUTE ON FUNCTION search TO pgstac_read;\nGRANT EXECUTE ON FUNCTION search_query TO pgstac_read;\nGRANT EXECUTE ON FUNCTION item_by_id TO pgstac_read;\nGRANT EXECUTE ON FUNCTION get_item TO pgstac_read;\n\nGRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_ingest;\nGRANT USAGE ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_ingest;\n\nRESET ROLE;\n\nINSERT INTO pgstac.collections (content) SELECT content FROM pgstac_045.collections;\nINSERT INTO pgstac.items_staging SELECT content FROM pgstac_045.items;\n\n\n\nSELECT set_version('0.5.0');\n" }, { "path": "src/pgstac/migrations/pgstac.0.4.5.sql", "content": "CREATE EXTENSION IF NOT EXISTS postgis;\nCREATE SCHEMA IF NOT EXISTS pgstac;\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE TABLE migrations (\n version text PRIMARY KEY,\n datetime timestamptz DEFAULT clock_timestamp() NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION get_version() RETURNS text AS $$\n SELECT version FROM migrations ORDER BY datetime DESC, version DESC LIMIT 1;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION set_version(text) RETURNS text AS $$\n INSERT INTO migrations (version) VALUES ($1)\n ON CONFLICT DO NOTHING\n RETURNING version;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\n\nCREATE TABLE IF NOT EXISTS pgstac_settings (\n name text PRIMARY KEY,\n value text NOT NULL\n);\n\nINSERT INTO pgstac_settings (name, value) VALUES\n ('context', 'off'),\n ('context_estimated_count', '100000'),\n ('context_estimated_cost', '1000000'),\n ('context_stats_ttl', '1 day'),\n ('default-filter-lang', 'cql2-json')\nON CONFLICT DO NOTHING\n;\n\n\nCREATE OR REPLACE FUNCTION get_setting(IN _setting text, IN conf jsonb DEFAULT NULL) RETURNS text AS $$\nSELECT COALESCE(\n conf->>_setting,\n current_setting(concat('pgstac.',_setting), TRUE),\n (SELECT value FROM pgstac_settings WHERE name=_setting)\n);\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context();\nCREATE OR REPLACE FUNCTION context(conf jsonb DEFAULT NULL) RETURNS text AS $$\n SELECT get_setting('context', conf);\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_estimated_count();\nCREATE OR REPLACE FUNCTION context_estimated_count(conf jsonb DEFAULT NULL) RETURNS int AS $$\n SELECT get_setting('context_estimated_count', conf)::int;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_estimated_cost();\nCREATE OR REPLACE FUNCTION context_estimated_cost(conf jsonb DEFAULT NULL) RETURNS float AS $$\n SELECT get_setting('context_estimated_cost', conf)::float;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_stats_ttl();\nCREATE OR REPLACE FUNCTION context_stats_ttl(conf jsonb DEFAULT NULL) RETURNS interval AS $$\n SELECT get_setting('context_stats_ttl', conf)::interval;\n$$ LANGUAGE SQL;\n\n\nCREATE OR REPLACE FUNCTION notice(VARIADIC text[]) RETURNS boolean AS $$\nDECLARE\ndebug boolean := current_setting('pgstac.debug', true);\nBEGIN\n IF debug THEN\n RAISE NOTICE 'NOTICE FROM FUNC: % >>>>> %', concat_ws(' | ', $1), clock_timestamp();\n RETURN TRUE;\n END IF;\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION empty_arr(ANYARRAY) RETURNS BOOLEAN AS $$\nSELECT CASE\n WHEN $1 IS NULL THEN TRUE\n WHEN cardinality($1)<1 THEN TRUE\nELSE FALSE\nEND;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_map_ident(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_ident(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_map_literal(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_literal(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION estimated_count(_where text) RETURNS bigint AS $$\nDECLARE\nrec record;\nrows bigint;\nBEGIN\n FOR rec in EXECUTE format(\n $q$\n EXPLAIN SELECT 1 FROM items WHERE %s\n $q$,\n _where)\n LOOP\n rows := substring(rec.\"QUERY PLAN\" FROM ' rows=([[:digit:]]+)');\n EXIT WHEN rows IS NOT NULL;\n END LOOP;\n\n RETURN rows;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION array_reverse(anyarray) RETURNS anyarray AS $$\nSELECT ARRAY(\n SELECT $1[i]\n FROM generate_subscripts($1,1) AS s(i)\n ORDER BY i DESC\n);\n$$ LANGUAGE 'sql' STRICT IMMUTABLE;\n/* converts a jsonb text array to a pg text[] array */\nCREATE OR REPLACE FUNCTION textarr(_js jsonb)\n RETURNS text[] AS $$\n SELECT\n CASE jsonb_typeof(_js)\n WHEN 'array' THEN ARRAY(SELECT jsonb_array_elements_text(_js))\n ELSE ARRAY[_js->>0]\n END\n;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION jsonb_paths (IN jdata jsonb, OUT path text[], OUT value jsonb) RETURNS\nSETOF RECORD AS $$\nwith recursive extract_all as\n(\n select\n ARRAY[key]::text[] as path,\n value\n FROM jsonb_each(jdata)\nunion all\n select\n path || coalesce(obj_key, (arr_key- 1)::text),\n coalesce(obj_value, arr_value)\n from extract_all\n left join lateral\n jsonb_each(case jsonb_typeof(value) when 'object' then value end)\n as o(obj_key, obj_value)\n on jsonb_typeof(value) = 'object'\n left join lateral\n jsonb_array_elements(case jsonb_typeof(value) when 'array' then value end)\n with ordinality as a(arr_value, arr_key)\n on jsonb_typeof(value) = 'array'\n where obj_key is not null or arr_key is not null\n)\nselect *\nfrom extract_all;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION jsonb_obj_paths (IN jdata jsonb, OUT path text[], OUT value jsonb) RETURNS\nSETOF RECORD AS $$\nwith recursive extract_all as\n(\n select\n ARRAY[key]::text[] as path,\n value\n FROM jsonb_each(jdata)\nunion all\n select\n path || obj_key,\n obj_value\n from extract_all\n left join lateral\n jsonb_each(case jsonb_typeof(value) when 'object' then value end)\n as o(obj_key, obj_value)\n on jsonb_typeof(value) = 'object'\n where obj_key is not null\n)\nselect *\nfrom extract_all;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION jsonb_val_paths (IN jdata jsonb, OUT path text[], OUT value jsonb) RETURNS\nSETOF RECORD AS $$\nSELECT * FROM jsonb_obj_paths(jdata) WHERE jsonb_typeof(value) not in ('object','array');\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION path_includes(IN path text[], IN includes text[]) RETURNS BOOLEAN AS $$\nWITH t AS (SELECT unnest(includes) i)\nSELECT EXISTS (\n SELECT 1 FROM t WHERE path @> string_to_array(trim(i), '.')\n);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION path_excludes(IN path text[], IN excludes text[]) RETURNS BOOLEAN AS $$\nWITH t AS (SELECT unnest(excludes) e)\nSELECT NOT EXISTS (\n SELECT 1 FROM t WHERE path @> string_to_array(trim(e), '.')\n);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION jsonb_obj_paths_filtered (\n IN jdata jsonb,\n IN includes text[] DEFAULT ARRAY[]::text[],\n IN excludes text[] DEFAULT ARRAY[]::text[],\n OUT path text[],\n OUT value jsonb\n) RETURNS\nSETOF RECORD AS $$\nSELECT path, value\nFROM jsonb_obj_paths(jdata)\nWHERE\n CASE WHEN cardinality(includes) > 0 THEN path_includes(path, includes) ELSE TRUE END\n AND\n path_excludes(path, excludes)\n\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION filter_jsonb(\n IN jdata jsonb,\n IN includes text[] DEFAULT ARRAY[]::text[],\n IN excludes text[] DEFAULT ARRAY[]::text[]\n) RETURNS jsonb AS $$\nDECLARE\nrec RECORD;\noutj jsonb := '{}'::jsonb;\ncreated_paths text[] := '{}'::text[];\nBEGIN\n\nIF empty_arr(includes) AND empty_arr(excludes) THEN\nRAISE NOTICE 'no filter';\n RETURN jdata;\nEND IF;\nFOR rec in\nSELECT * FROM jsonb_obj_paths_filtered(jdata, includes, excludes)\nWHERE jsonb_typeof(value) != 'object'\nLOOP\n IF array_length(rec.path,1)>1 THEN\n FOR i IN 1..(array_length(rec.path,1)-1) LOOP\n IF NOT array_to_string(rec.path[1:i],'.') = ANY (created_paths) THEN\n outj := jsonb_set(outj, rec.path[1:i],'{}', true);\n created_paths := created_paths || array_to_string(rec.path[1:i],'.');\n END IF;\n END LOOP;\n END IF;\n outj := jsonb_set(outj, rec.path, rec.value, true);\n created_paths := created_paths || array_to_string(rec.path,'.');\nEND LOOP;\nRETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION flip_jsonb_array(j jsonb) RETURNS jsonb AS $$\nSELECT jsonb_agg(value) FROM (SELECT value FROM jsonb_array_elements(j) WITH ORDINALITY ORDER BY ordinality DESC) as t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n/* Functions to create an iterable of cursors over partitions. */\nCREATE OR REPLACE FUNCTION create_cursor(q text) RETURNS refcursor AS $$\nDECLARE\n curs refcursor;\nBEGIN\n OPEN curs FOR EXECUTE q;\n RETURN curs;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDROP FUNCTION IF EXISTS partition_queries;\nCREATE OR REPLACE FUNCTION partition_queries(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN partitions text[] DEFAULT '{items}'\n) RETURNS SETOF text AS $$\nDECLARE\n partition_query text;\n query text;\n p text;\n cursors refcursor;\n dstart timestamptz;\n dend timestamptz;\n step interval := '10 weeks'::interval;\nBEGIN\n\nIF _orderby ILIKE 'datetime d%' THEN\n partitions := partitions;\nELSIF _orderby ILIKE 'datetime a%' THEN\n partitions := array_reverse(partitions);\nELSE\n query := format($q$\n SELECT * FROM items\n WHERE %s\n ORDER BY %s\n $q$, _where, _orderby\n );\n\n RETURN NEXT query;\n RETURN;\nEND IF;\nRAISE NOTICE 'PARTITIONS ---> %',partitions;\nIF cardinality(partitions) > 0 THEN\n FOREACH p IN ARRAY partitions\n --EXECUTE partition_query\n LOOP\n query := format($q$\n SELECT * FROM %I\n WHERE %s\n ORDER BY %s\n $q$,\n p,\n _where,\n _orderby\n );\n RETURN NEXT query;\n END LOOP;\nEND IF;\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION partition_cursor(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC'\n) RETURNS SETOF refcursor AS $$\nDECLARE\n partition_query text;\n query text;\n p record;\n cursors refcursor;\nBEGIN\nFOR query IN SELECT * FROM partition_queries(_where, _orderby) LOOP\n RETURN NEXT create_cursor(query);\nEND LOOP;\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION partition_count(\n IN _where text DEFAULT 'TRUE'\n) RETURNS bigint AS $$\nDECLARE\n partition_query text;\n query text;\n p record;\n subtotal bigint;\n total bigint := 0;\nBEGIN\npartition_query := format($q$\n SELECT partition, tstzrange\n FROM items_partitions\n ORDER BY tstzrange DESC;\n$q$);\nRAISE NOTICE 'Partition Query: %', partition_query;\nFOR p IN\n EXECUTE partition_query\nLOOP\n query := format($q$\n SELECT count(*) FROM items\n WHERE datetime BETWEEN %L AND %L AND %s\n $q$, lower(p.tstzrange), upper(p.tstzrange), _where\n );\n RAISE NOTICE 'Query %', query;\n RAISE NOTICE 'Partition %, Count %, Total %',p.partition, subtotal, total;\n EXECUTE query INTO subtotal;\n total := subtotal + total;\nEND LOOP;\nRETURN total;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION drop_partition_constraints(IN partition text) RETURNS VOID AS $$\nDECLARE\n q text;\n end_datetime_constraint text := concat(partition, '_end_datetime_constraint');\n collections_constraint text := concat(partition, '_collections_constraint');\nBEGIN\n q := format($q$\n ALTER TABLE %I\n DROP CONSTRAINT IF EXISTS %I,\n DROP CONSTRAINT IF EXISTS %I;\n $q$,\n partition,\n end_datetime_constraint,\n collections_constraint\n );\n\n EXECUTE q;\n RETURN;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDROP FUNCTION IF EXISTS partition_checks;\nCREATE OR REPLACE FUNCTION partition_checks(\n IN partition text,\n OUT min_datetime timestamptz,\n OUT max_datetime timestamptz,\n OUT min_end_datetime timestamptz,\n OUT max_end_datetime timestamptz,\n OUT collections text[],\n OUT cnt bigint\n) RETURNS RECORD AS $$\nDECLARE\nq text;\nend_datetime_constraint text := concat(partition, '_end_datetime_constraint');\ncollections_constraint text := concat(partition, '_collections_constraint');\nBEGIN\nRAISE NOTICE 'CREATING CONSTRAINTS FOR %', partition;\nq := format($q$\n SELECT\n min(datetime),\n max(datetime),\n min(end_datetime),\n max(end_datetime),\n array_agg(DISTINCT collection_id),\n count(*)\n FROM %I;\n $q$,\n partition\n);\nEXECUTE q INTO min_datetime, max_datetime, min_end_datetime, max_end_datetime, collections, cnt;\nRAISE NOTICE '% % % % % % %', min_datetime, max_datetime, min_end_datetime, max_end_datetime, collections, cnt, ftime();\nIF cnt IS NULL or cnt = 0 THEN\n RAISE NOTICE 'Partition % is empty, removing...', partition;\n q := format($q$\n DROP TABLE IF EXISTS %I;\n $q$, partition\n );\n EXECUTE q;\n RETURN;\nEND IF;\nRAISE NOTICE 'Running Constraint DDL %', ftime();\nq := format($q$\n ALTER TABLE %I\n DROP CONSTRAINT IF EXISTS %I,\n ADD CONSTRAINT %I\n check((end_datetime >= %L) AND (end_datetime <= %L)) NOT VALID,\n DROP CONSTRAINT IF EXISTS %I,\n ADD CONSTRAINT %I\n check((collection_id = ANY(%L))) NOT VALID;\n $q$,\n partition,\n end_datetime_constraint,\n end_datetime_constraint,\n min_end_datetime,\n max_end_datetime,\n collections_constraint,\n collections_constraint,\n collections,\n partition\n);\nRAISE NOTICE 'q: %', q;\n\nEXECUTE q;\nRAISE NOTICE 'Returning %', ftime();\nRETURN;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION validate_constraints() RETURNS VOID AS $$\nDECLARE\nq text;\nBEGIN\nFOR q IN\n SELECT FORMAT(\n 'ALTER TABLE %I.%I.%I VALIDATE CONSTRAINT %I;',\n current_database(),\n nsp.nspname,\n cls.relname,\n con.conname\n )\n FROM pg_constraint AS con\n JOIN pg_class AS cls\n ON con.conrelid = cls.oid\n JOIN pg_namespace AS nsp\n ON cls.relnamespace = nsp.oid\n WHERE convalidated IS FALSE\n AND nsp.nspname = 'pgstac'\nLOOP\n EXECUTE q;\nEND LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n/* looks for a geometry in a stac item first from geometry and falling back to bbox */\nCREATE OR REPLACE FUNCTION stac_geom(value jsonb) RETURNS geometry AS $$\nSELECT\n CASE\n WHEN value->>'geometry' IS NOT NULL THEN\n ST_GeomFromGeoJSON(value->>'geometry')\n WHEN value->>'bbox' IS NOT NULL THEN\n ST_MakeEnvelope(\n (value->'bbox'->>0)::float,\n (value->'bbox'->>1)::float,\n (value->'bbox'->>2)::float,\n (value->'bbox'->>3)::float,\n 4326\n )\n ELSE NULL\n END as geometry\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION stac_datetime(value jsonb) RETURNS timestamptz AS $$\nSELECT COALESCE(\n (value->'properties'->>'datetime')::timestamptz,\n (value->'properties'->>'start_datetime')::timestamptz\n);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_end_datetime(value jsonb) RETURNS timestamptz AS $$\nSELECT COALESCE(\n (value->'properties'->>'datetime')::timestamptz,\n (value->'properties'->>'end_datetime')::timestamptz\n);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\n\nCREATE OR REPLACE FUNCTION stac_daterange(value jsonb) RETURNS tstzrange AS $$\nSELECT tstzrange(stac_datetime(value),stac_end_datetime(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE TABLE IF NOT EXISTS collections (\n id VARCHAR GENERATED ALWAYS AS (content->>'id') STORED PRIMARY KEY,\n content JSONB\n);\n\nCREATE OR REPLACE FUNCTION create_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection(data jsonb) RETURNS VOID AS $$\nDECLARE\nout collections%ROWTYPE;\nBEGIN\n UPDATE collections SET content=data WHERE id = data->>'id' RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ON CONFLICT (id) DO\n UPDATE\n SET content=EXCLUDED.content\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION delete_collection(_id text) RETURNS VOID AS $$\nDECLARE\nout collections%ROWTYPE;\nBEGIN\n DELETE FROM collections WHERE id = _id RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION get_collection(id text) RETURNS jsonb AS $$\nSELECT content FROM collections\nWHERE id=$1\n;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION all_collections() RETURNS jsonb AS $$\nSELECT jsonb_agg(content) FROM collections;\n;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE TABLE items (\n id text NOT NULL,\n geometry geometry NOT NULL,\n collection_id text NOT NULL,\n datetime timestamptz NOT NULL,\n end_datetime timestamptz NOT NULL,\n properties jsonb NOT NULL,\n content JSONB NOT NULL\n)\nPARTITION BY RANGE (datetime)\n;\n\nCREATE OR REPLACE FUNCTION properties_idx (IN content jsonb) RETURNS jsonb AS $$\n with recursive extract_all as\n (\n select\n ARRAY[key]::text[] as path,\n ARRAY[key]::text[] as fullpath,\n value\n FROM jsonb_each(content->'properties')\n union all\n select\n CASE WHEN obj_key IS NOT NULL THEN path || obj_key ELSE path END,\n path || coalesce(obj_key, (arr_key- 1)::text),\n coalesce(obj_value, arr_value)\n from extract_all\n left join lateral\n jsonb_each(case jsonb_typeof(value) when 'object' then value end)\n as o(obj_key, obj_value)\n on jsonb_typeof(value) = 'object'\n left join lateral\n jsonb_array_elements(case jsonb_typeof(value) when 'array' then value end)\n with ordinality as a(arr_value, arr_key)\n on jsonb_typeof(value) = 'array'\n where obj_key is not null or arr_key is not null\n )\n , paths AS (\n select\n array_to_string(path, '.') as path,\n value\n FROM extract_all\n WHERE\n jsonb_typeof(value) NOT IN ('array','object')\n ), grouped AS (\n SELECT path, jsonb_agg(distinct value) vals FROM paths group by path\n ) SELECT coalesce(jsonb_object_agg(path, CASE WHEN jsonb_array_length(vals)=1 THEN vals->0 ELSE vals END) - '{datetime}'::text[], '{}'::jsonb) FROM grouped\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET JIT TO OFF;\n\nCREATE INDEX \"datetime_idx\" ON items (datetime);\nCREATE INDEX \"end_datetime_idx\" ON items (end_datetime);\nCREATE INDEX \"properties_idx\" ON items USING GIN (properties jsonb_path_ops);\nCREATE INDEX \"collection_idx\" ON items (collection_id);\nCREATE INDEX \"geometry_idx\" ON items USING GIST (geometry);\nCREATE UNIQUE INDEX \"items_id_datetime_idx\" ON items (datetime, id);\n\nALTER TABLE items ADD CONSTRAINT items_collections_fk FOREIGN KEY (collection_id) REFERENCES collections(id) ON DELETE CASCADE DEFERRABLE;\n\nCREATE OR REPLACE FUNCTION analyze_empty_partitions() RETURNS VOID AS $$\nDECLARE\n p text;\nBEGIN\n FOR p IN SELECT partition FROM all_items_partitions WHERE est_cnt = 0 LOOP\n EXECUTE format('ANALYZE %I;', p);\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION items_partition_name(timestamptz) RETURNS text AS $$\n SELECT to_char($1, '\"items_p\"IYYY\"w\"IW');\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION items_partition_exists(text) RETURNS boolean AS $$\n SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_class WHERE relname=$1);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION items_partition_exists(timestamptz) RETURNS boolean AS $$\n SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_class WHERE relname=items_partition_name($1));\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION items_partition_create_worker(partition text, partition_start timestamptz, partition_end timestamptz) RETURNS VOID AS $$\nDECLARE\n err_context text;\nBEGIN\n EXECUTE format(\n $f$\n CREATE TABLE IF NOT EXISTS %1$I PARTITION OF items\n FOR VALUES FROM (%2$L) TO (%3$L);\n CREATE UNIQUE INDEX IF NOT EXISTS %4$I ON %1$I (id);\n $f$,\n partition,\n partition_start,\n partition_end,\n concat(partition, '_id_pk')\n );\nEXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', partition;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH to pgstac, public;\n\nCREATE OR REPLACE FUNCTION items_partition_create(ts timestamptz) RETURNS text AS $$\nDECLARE\n partition text := items_partition_name(ts);\n partition_start timestamptz;\n partition_end timestamptz;\nBEGIN\n IF items_partition_exists(partition) THEN\n RETURN partition;\n END IF;\n partition_start := date_trunc('week', ts);\n partition_end := partition_start + '1 week'::interval;\n PERFORM items_partition_create_worker(partition, partition_start, partition_end);\n RAISE NOTICE 'partition: %', partition;\n RETURN partition;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION items_partition_create(st timestamptz, et timestamptz) RETURNS SETOF text AS $$\nWITH t AS (\n SELECT\n generate_series(\n date_trunc('week',st),\n date_trunc('week', et),\n '1 week'::interval\n ) w\n)\nSELECT items_partition_create(w) FROM t;\n$$ LANGUAGE SQL;\n\n\nCREATE UNLOGGED TABLE items_staging (\n content JSONB NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION items_staging_insert_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p record;\n _partitions text[];\nBEGIN\n CREATE TEMP TABLE new_partitions ON COMMIT DROP AS\n SELECT\n items_partition_name(stac_datetime(content)) as partition,\n date_trunc('week', min(stac_datetime(content))) as partition_start\n FROM newdata\n GROUP BY 1;\n\n -- set statslastupdated in cache to be old enough cache always regenerated\n\n SELECT array_agg(partition) INTO _partitions FROM new_partitions;\n UPDATE search_wheres\n SET\n statslastupdated = NULL\n WHERE _where IN (\n SELECT _where FROM search_wheres sw WHERE sw.partitions && _partitions\n FOR UPDATE SKIP LOCKED\n )\n ;\n FOR p IN SELECT new_partitions.partition, new_partitions.partition_start, new_partitions.partition_start + '1 week'::interval as partition_end FROM new_partitions\n LOOP\n RAISE NOTICE 'Getting partition % ready.', p.partition;\n IF NOT items_partition_exists(p.partition) THEN\n RAISE NOTICE 'Creating partition %.', p.partition;\n PERFORM items_partition_create_worker(p.partition, p.partition_start, p.partition_end);\n END IF;\n PERFORM drop_partition_constraints(p.partition);\n END LOOP;\n INSERT INTO items (id, geometry, collection_id, datetime, end_datetime, properties, content)\n SELECT\n content->>'id',\n stac_geom(content),\n content->>'collection',\n stac_datetime(content),\n stac_end_datetime(content),\n properties_idx(content),\n content\n FROM newdata\n ;\n DELETE FROM items_staging;\n\n\n FOR p IN SELECT new_partitions.partition FROM new_partitions\n LOOP\n RAISE NOTICE 'Setting constraints for partition %.', p.partition;\n PERFORM partition_checks(p.partition);\n END LOOP;\n DROP TABLE IF EXISTS new_partitions;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER items_staging_insert_trigger AFTER INSERT ON items_staging REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_insert_triggerfunc();\n\n\nCREATE UNLOGGED TABLE items_staging_ignore (\n content JSONB NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION items_staging_ignore_insert_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p record;\n _partitions text[];\nBEGIN\n CREATE TEMP TABLE new_partitions ON COMMIT DROP AS\n SELECT\n items_partition_name(stac_datetime(content)) as partition,\n date_trunc('week', min(stac_datetime(content))) as partition_start\n FROM newdata\n GROUP BY 1;\n\n -- set statslastupdated in cache to be old enough cache always regenerated\n\n SELECT array_agg(partition) INTO _partitions FROM new_partitions;\n UPDATE search_wheres\n SET\n statslastupdated = NULL\n WHERE _where IN (\n SELECT _where FROM search_wheres sw WHERE sw.partitions && _partitions\n FOR UPDATE SKIP LOCKED\n )\n ;\n\n FOR p IN SELECT new_partitions.partition, new_partitions.partition_start, new_partitions.partition_start + '1 week'::interval as partition_end FROM new_partitions\n LOOP\n RAISE NOTICE 'Getting partition % ready.', p.partition;\n IF NOT items_partition_exists(p.partition) THEN\n RAISE NOTICE 'Creating partition %.', p.partition;\n PERFORM items_partition_create_worker(p.partition, p.partition_start, p.partition_end);\n END IF;\n PERFORM drop_partition_constraints(p.partition);\n END LOOP;\n\n INSERT INTO items (id, geometry, collection_id, datetime, end_datetime, properties, content)\n SELECT\n content->>'id',\n stac_geom(content),\n content->>'collection',\n stac_datetime(content),\n stac_end_datetime(content),\n properties_idx(content),\n content\n FROM newdata\n ON CONFLICT DO NOTHING\n ;\n DELETE FROM items_staging;\n\n\n FOR p IN SELECT new_partitions.partition FROM new_partitions\n LOOP\n RAISE NOTICE 'Setting constraints for partition %.', p.partition;\n PERFORM partition_checks(p.partition);\n END LOOP;\n DROP TABLE IF EXISTS new_partitions;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER items_staging_ignore_insert_trigger AFTER INSERT ON items_staging_ignore REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_ignore_insert_triggerfunc();\n\nCREATE UNLOGGED TABLE items_staging_upsert (\n content JSONB NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION items_staging_upsert_insert_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p record;\n _partitions text[];\nBEGIN\n CREATE TEMP TABLE new_partitions ON COMMIT DROP AS\n SELECT\n items_partition_name(stac_datetime(content)) as partition,\n date_trunc('week', min(stac_datetime(content))) as partition_start\n FROM newdata\n GROUP BY 1;\n\n -- set statslastupdated in cache to be old enough cache always regenerated\n\n SELECT array_agg(partition) INTO _partitions FROM new_partitions;\n UPDATE search_wheres\n SET\n statslastupdated = NULL\n WHERE _where IN (\n SELECT _where FROM search_wheres sw WHERE sw.partitions && _partitions\n FOR UPDATE SKIP LOCKED\n )\n ;\n\n FOR p IN SELECT new_partitions.partition, new_partitions.partition_start, new_partitions.partition_start + '1 week'::interval as partition_end FROM new_partitions\n LOOP\n RAISE NOTICE 'Getting partition % ready.', p.partition;\n IF NOT items_partition_exists(p.partition) THEN\n RAISE NOTICE 'Creating partition %.', p.partition;\n PERFORM items_partition_create_worker(p.partition, p.partition_start, p.partition_end);\n END IF;\n PERFORM drop_partition_constraints(p.partition);\n END LOOP;\n\n INSERT INTO items (id, geometry, collection_id, datetime, end_datetime, properties, content)\n SELECT\n content->>'id',\n stac_geom(content),\n content->>'collection',\n stac_datetime(content),\n stac_end_datetime(content),\n properties_idx(content),\n content\n FROM newdata\n ON CONFLICT (datetime, id) DO UPDATE SET\n content = EXCLUDED.content\n WHERE items.content IS DISTINCT FROM EXCLUDED.content\n ;\n DELETE FROM items_staging;\n\n\n FOR p IN SELECT new_partitions.partition FROM new_partitions\n LOOP\n RAISE NOTICE 'Setting constraints for partition %.', p.partition;\n PERFORM partition_checks(p.partition);\n END LOOP;\n DROP TABLE IF EXISTS new_partitions;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER items_staging_upsert_insert_trigger AFTER INSERT ON items_staging_upsert REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_upsert_insert_triggerfunc();\n\nCREATE OR REPLACE FUNCTION items_update_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\nBEGIN\n NEW.id := NEW.content->>'id';\n NEW.datetime := stac_datetime(NEW.content);\n NEW.end_datetime := stac_end_datetime(NEW.content);\n NEW.collection_id := NEW.content->>'collection';\n NEW.geometry := stac_geom(NEW.content);\n NEW.properties := properties_idx(NEW.content);\n IF TG_OP = 'UPDATE' AND NEW IS NOT DISTINCT FROM OLD THEN\n RETURN NULL;\n END IF;\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER items_update_trigger BEFORE UPDATE ON items\n FOR EACH ROW EXECUTE PROCEDURE items_update_triggerfunc();\n\n/*\nView to get a table of available items partitions\nwith date ranges\n*/\nDROP VIEW IF EXISTS all_items_partitions CASCADE;\nCREATE VIEW all_items_partitions AS\nWITH base AS\n(SELECT\n c.oid::pg_catalog.regclass::text as partition,\n pg_catalog.pg_get_expr(c.relpartbound, c.oid) as _constraint,\n regexp_matches(\n pg_catalog.pg_get_expr(c.relpartbound, c.oid),\n E'\\\\(''\\([0-9 :+-]*\\)''\\\\).*\\\\(''\\([0-9 :+-]*\\)''\\\\)'\n ) as t,\n reltuples::bigint as est_cnt\nFROM pg_catalog.pg_class c, pg_catalog.pg_inherits i\nWHERE c.oid = i.inhrelid AND i.inhparent = 'items'::regclass)\nSELECT partition, tstzrange(\n t[1]::timestamptz,\n t[2]::timestamptz\n), t[1]::timestamptz as pstart,\n t[2]::timestamptz as pend, est_cnt\nFROM base\nORDER BY 2 desc;\n\nCREATE OR REPLACE VIEW items_partitions AS\nSELECT * FROM all_items_partitions WHERE est_cnt>0;\n\nCREATE OR REPLACE FUNCTION item_by_id(_id text) RETURNS items AS\n$$\nDECLARE\n i items%ROWTYPE;\nBEGIN\n SELECT * INTO i FROM items WHERE id=_id LIMIT 1;\n RETURN i;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION get_item(_id text) RETURNS jsonb AS $$\n SELECT content FROM items WHERE id=_id;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION delete_item(_id text) RETURNS VOID AS $$\nDECLARE\nout items%ROWTYPE;\nBEGIN\n DELETE FROM items WHERE id = _id RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION create_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION update_item(data jsonb) RETURNS VOID AS $$\nDECLARE\n out items%ROWTYPE;\nBEGIN\n UPDATE items SET content=data WHERE id = data->>'id' RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION create_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION collection_bbox(id text) RETURNS jsonb AS $$\nSELECT (replace(replace(replace(st_extent(geometry)::text,'BOX(','[['),')',']]'),' ',','))::jsonb\nFROM items WHERE collection_id=$1;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION collection_temporal_extent(id text) RETURNS jsonb AS $$\nSELECT to_jsonb(array[array[min(datetime)::text, max(datetime)::text]])\nFROM items WHERE collection_id=$1;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection_extents() RETURNS VOID AS $$\nUPDATE collections SET\n content = content ||\n jsonb_build_object(\n 'extent', jsonb_build_object(\n 'spatial', jsonb_build_object(\n 'bbox', collection_bbox(collections.id)\n ),\n 'temporal', jsonb_build_object(\n 'interval', collection_temporal_extent(collections.id)\n )\n )\n )\n;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION items_path(\n IN dotpath text,\n OUT field text,\n OUT path text,\n OUT path_txt text,\n OUT jsonpath text,\n OUT eq text\n) RETURNS RECORD AS $$\nDECLARE\npath_elements text[];\nlast_element text;\nBEGIN\ndotpath := replace(trim(dotpath), 'properties.', '');\n\nIF dotpath = '' THEN\n RETURN;\nEND IF;\n\npath_elements := string_to_array(dotpath, '.');\njsonpath := NULL;\n\nIF path_elements[1] IN ('id','geometry','datetime') THEN\n field := path_elements[1];\n path_elements := path_elements[2:];\nELSIF path_elements[1] = 'collection' THEN\n field := 'collection_id';\n path_elements := path_elements[2:];\nELSIF path_elements[1] IN ('links', 'assets', 'stac_version', 'stac_extensions') THEN\n field := 'content';\nELSE\n field := 'content';\n path_elements := '{properties}'::text[] || path_elements;\nEND IF;\nIF cardinality(path_elements)<1 THEN\n path := field;\n path_txt := field;\n jsonpath := '$';\n eq := NULL;\n RETURN;\nEND IF;\n\n\nlast_element := path_elements[cardinality(path_elements)];\npath_elements := path_elements[1:cardinality(path_elements)-1];\njsonpath := concat(array_to_string('{$}'::text[] || array_map_ident(path_elements), '.'), '.', quote_ident(last_element));\npath_elements := array_map_literal(path_elements);\npath := format($F$ properties->%s $F$, quote_literal(dotpath));\npath_txt := format($F$ properties->>%s $F$, quote_literal(dotpath));\neq := format($F$ properties @? '$.%s[*] ? (@ == %%s) '$F$, quote_ident(dotpath));\n\nRAISE NOTICE 'ITEMS PATH -- % % % % %', field, path, path_txt, jsonpath, eq;\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION parse_dtrange(IN _indate jsonb, OUT _tstzrange tstzrange) AS $$\nWITH t AS (\n SELECT CASE\n WHEN _indate ? 'timestamp' THEN\n ARRAY[_indate->>'timestamp', 'infinity']\n WHEN _indate ? 'interval' THEN\n textarr(_indate->'interval')\n WHEN jsonb_typeof(_indate) = 'array' THEN\n textarr(_indate)\n ELSE\n regexp_split_to_array(\n btrim(_indate::text,'\"'),\n '/'\n )\n END AS arr\n)\n, t1 AS (\n SELECT\n CASE\n WHEN array_upper(arr,1) = 1 OR arr[1] = '..' OR arr[1] IS NULL THEN '-infinity'::timestamptz\n ELSE arr[1]::timestamptz\n END AS st,\n CASE\n WHEN array_upper(arr,1) = 1 THEN arr[1]::timestamptz\n WHEN arr[2] = '..' OR arr[2] IS NULL THEN 'infinity'::timestamptz\n ELSE arr[2]::timestamptz\n END AS et\n FROM t\n)\nSELECT\n tstzrange(st,et)\nFROM t1;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION bbox_geom(_bbox jsonb) RETURNS geometry AS $$\nSELECT CASE jsonb_array_length(_bbox)\n WHEN 4 THEN\n ST_SetSRID(ST_MakeEnvelope(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float,\n (_bbox->>3)::float\n ),4326)\n WHEN 6 THEN\n ST_SetSRID(ST_3DMakeBox(\n ST_MakePoint(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float\n ),\n ST_MakePoint(\n (_bbox->>3)::float,\n (_bbox->>4)::float,\n (_bbox->>5)::float\n )\n ),4326)\n ELSE null END;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION cql_and_append(existing jsonb, newfilters jsonb) RETURNS jsonb AS $$\nSELECT CASE WHEN existing ? 'filter' AND newfilters IS NOT NULL THEN\n jsonb_build_object(\n 'and',\n jsonb_build_array(\n existing->'filter',\n newfilters\n )\n )\nELSE\n newfilters\nEND;\n$$ LANGUAGE SQL;\n\n\n-- ADDs base filters (ids, collections, datetime, bbox, intersects) that are\n-- added outside of the filter/query in the stac request\nCREATE OR REPLACE FUNCTION add_filters_to_cql(j jsonb) RETURNS jsonb AS $$\nDECLARE\nnewprop jsonb;\nnewprops jsonb := '[]'::jsonb;\nBEGIN\nIF j ? 'ids' THEN\n newprop := jsonb_build_object(\n 'in',\n jsonb_build_array(\n '{\"property\":\"id\"}'::jsonb,\n j->'ids'\n )\n );\n newprops := jsonb_insert(newprops, '{1}', newprop);\nEND IF;\nIF j ? 'collections' THEN\n newprop := jsonb_build_object(\n 'in',\n jsonb_build_array(\n '{\"property\":\"collection\"}'::jsonb,\n j->'collections'\n )\n );\n newprops := jsonb_insert(newprops, '{1}', newprop);\nEND IF;\n\nIF j ? 'datetime' THEN\n newprop := format(\n '{\"anyinteracts\":[{\"property\":\"datetime\"}, %s]}',\n j->'datetime'\n );\n newprops := jsonb_insert(newprops, '{1}', newprop);\nEND IF;\n\nIF j ? 'bbox' THEN\n newprop := format(\n '{\"intersects\":[{\"property\":\"geometry\"}, %s]}',\n j->'bbox'\n );\n newprops := jsonb_insert(newprops, '{1}', newprop);\nEND IF;\n\nIF j ? 'intersects' THEN\n newprop := format(\n '{\"intersects\":[{\"property\":\"geometry\"}, %s]}',\n j->'intersects'\n );\n newprops := jsonb_insert(newprops, '{1}', newprop);\nEND IF;\n\nRAISE NOTICE 'newprops: %', newprops;\n\nIF newprops IS NOT NULL AND jsonb_array_length(newprops) > 0 THEN\n return jsonb_set(\n j,\n '{filter}',\n cql_and_append(j, jsonb_build_object('and', newprops))\n ) - '{ids,collections,datetime,bbox,intersects}'::text[];\nEND IF;\n\nreturn j;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION query_to_cqlfilter(j jsonb) RETURNS jsonb AS $$\n-- Translates anything passed in through the deprecated \"query\" into equivalent CQL\nWITH t AS (\n SELECT key as property, value as ops\n FROM jsonb_each(j->'query')\n), t2 AS (\n SELECT property, (jsonb_each(ops)).*\n FROM t WHERE jsonb_typeof(ops) = 'object'\n UNION ALL\n SELECT property, 'eq', ops\n FROM t WHERE jsonb_typeof(ops) != 'object'\n), t3 AS (\nSELECT\n jsonb_strip_nulls(jsonb_build_object(\n 'and',\n jsonb_agg(\n jsonb_build_object(\n key,\n jsonb_build_array(\n jsonb_build_object('property',property),\n value\n )\n )\n )\n )) as qcql FROM t2\n)\nSELECT\n CASE WHEN qcql IS NOT NULL THEN\n jsonb_set(j, '{filter}', cql_and_append(j, qcql)) - 'query'\n ELSE j\n END\nFROM t3\n;\n$$ LANGUAGE SQL;\n\n\nCREATE OR REPLACE FUNCTION base_stac_query(j jsonb) RETURNS text AS $$\nDECLARE\n_where text := '';\ndtrange tstzrange;\ngeom geometry;\nBEGIN\n\n IF j ? 'ids' THEN\n _where := format('%s AND id = ANY (%L) ', _where, textarr(j->'ids'));\n END IF;\n IF j ? 'collections' THEN\n _where := format('%s AND collection_id = ANY (%L) ', _where, textarr(j->'collections'));\n END IF;\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n _where := format('%s AND datetime <= %L::timestamptz AND end_datetime >= %L::timestamptz ',\n _where,\n upper(dtrange),\n lower(dtrange)\n );\n END IF;\n\n IF j ? 'bbox' THEN\n geom := bbox_geom(j->'bbox');\n _where := format('%s AND geometry && %L',\n _where,\n geom\n );\n END IF;\n\n IF j ? 'intersects' THEN\n geom := st_geomfromgeojson(j->>'intersects');\n _where := format('%s AND st_intersects(geometry, %L)',\n _where,\n geom\n );\n END IF;\n\n RETURN _where;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION temporal_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\nll text := 'datetime';\nlh text := 'end_datetime';\nrrange tstzrange;\nrl text;\nrh text;\noutq text;\nBEGIN\nrrange := parse_dtrange(args->1);\nRAISE NOTICE 'Constructing temporal query OP: %, ARGS: %, RRANGE: %', op, args, rrange;\nop := lower(op);\nrl := format('%L::timestamptz', lower(rrange));\nrh := format('%L::timestamptz', upper(rrange));\noutq := CASE op\n WHEN 't_before' THEN 'lh < rl'\n WHEN 't_after' THEN 'll > rh'\n WHEN 't_meets' THEN 'lh = rl'\n WHEN 't_metby' THEN 'll = rh'\n WHEN 't_overlaps' THEN 'll < rl AND rl < lh < rh'\n WHEN 't_overlappedby' THEN 'rl < ll < rh AND lh > rh'\n WHEN 't_starts' THEN 'll = rl AND lh < rh'\n WHEN 't_startedby' THEN 'll = rl AND lh > rh'\n WHEN 't_during' THEN 'll > rl AND lh < rh'\n WHEN 't_contains' THEN 'll < rl AND lh > rh'\n WHEN 't_finishes' THEN 'll > rl AND lh = rh'\n WHEN 't_finishedby' THEN 'll < rl AND lh = rh'\n WHEN 't_equals' THEN 'll = rl AND lh = rh'\n WHEN 't_disjoint' THEN 'NOT (ll <= rh AND lh >= rl)'\n WHEN 't_intersects' THEN 'll <= rh AND lh >= rl'\n WHEN 'anyinteracts' THEN 'll <= rh AND lh >= rl'\nEND;\noutq := regexp_replace(outq, '\\mll\\M', ll);\noutq := regexp_replace(outq, '\\mlh\\M', lh);\noutq := regexp_replace(outq, '\\mrl\\M', rl);\noutq := regexp_replace(outq, '\\mrh\\M', rh);\noutq := format('(%s)', outq);\nRETURN outq;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION spatial_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\ngeom text;\nj jsonb := args->1;\nBEGIN\nop := lower(op);\nRAISE NOTICE 'Constructing spatial query OP: %, ARGS: %', op, args;\nIF op NOT IN ('s_equals','s_disjoint','s_touches','s_within','s_overlaps','s_crosses','s_intersects','intersects','s_contains') THEN\n RAISE EXCEPTION 'Spatial Operator % Not Supported', op;\nEND IF;\nop := regexp_replace(op, '^s_', 'st_');\nIF op = 'intersects' THEN\n op := 'st_intersects';\nEND IF;\n-- Convert geometry to WKB string\nIF j ? 'type' AND j ? 'coordinates' THEN\n geom := st_geomfromgeojson(j)::text;\nELSIF jsonb_typeof(j) = 'array' THEN\n geom := bbox_geom(j)::text;\nEND IF;\n\nRETURN format('%s(geometry, %L::geometry)', op, geom);\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n/* cql_query_op -- Parses a CQL query operation, recursing when necessary\n IN jsonb -- a subelement from a valid stac query\n IN text -- the operator being used on elements passed in\n RETURNS a SQL fragment to be used in a WHERE clause\n*/\nCREATE OR REPLACE FUNCTION cql_query_op(j jsonb, _op text DEFAULT NULL) RETURNS text AS $$\nDECLARE\njtype text := jsonb_typeof(j);\nop text := lower(_op);\nops jsonb :=\n '{\n \"eq\": \"%s = %s\",\n \"lt\": \"%s < %s\",\n \"lte\": \"%s <= %s\",\n \"gt\": \"%s > %s\",\n \"gte\": \"%s >= %s\",\n \"le\": \"%s <= %s\",\n \"ge\": \"%s >= %s\",\n \"=\": \"%s = %s\",\n \"<\": \"%s < %s\",\n \"<=\": \"%s <= %s\",\n \">\": \"%s > %s\",\n \">=\": \"%s >= %s\",\n \"like\": \"%s LIKE %s\",\n \"ilike\": \"%s ILIKE %s\",\n \"+\": \"%s + %s\",\n \"-\": \"%s - %s\",\n \"*\": \"%s * %s\",\n \"/\": \"%s / %s\",\n \"in\": \"%s = ANY (%s)\",\n \"not\": \"NOT (%s)\",\n \"between\": \"%s BETWEEN %s AND %s\",\n \"lower\":\" lower(%s)\",\n \"upper\":\" upper(%s)\",\n \"isnull\": \"%s IS NULL\"\n }'::jsonb;\nret text;\nargs text[] := NULL;\n\nBEGIN\nRAISE NOTICE 'j: %, op: %, jtype: %', j, op, jtype;\n\n-- for in, convert value, list to array syntax to match other ops\nIF op = 'in' and j ? 'value' and j ? 'list' THEN\n j := jsonb_build_array( j->'value', j->'list');\n jtype := 'array';\n RAISE NOTICE 'IN: j: %, jtype: %', j, jtype;\nEND IF;\n\nIF op = 'between' and j ? 'value' and j ? 'lower' and j ? 'upper' THEN\n j := jsonb_build_array( j->'value', j->'lower', j->'upper');\n jtype := 'array';\n RAISE NOTICE 'BETWEEN: j: %, jtype: %', j, jtype;\nEND IF;\n\nIF op = 'not' AND jtype = 'object' THEN\n j := jsonb_build_array( j );\n jtype := 'array';\n RAISE NOTICE 'NOT: j: %, jtype: %', j, jtype;\nEND IF;\n\n-- Set Lower Case on Both Arguments When Case Insensitive Flag Set\nIF op in ('eq','lt','lte','gt','gte','like') AND jsonb_typeof(j->2) = 'boolean' THEN\n IF (j->>2)::boolean THEN\n RETURN format(concat('(',ops->>op,')'), cql_query_op(jsonb_build_array(j->0), 'lower'), cql_query_op(jsonb_build_array(j->1), 'lower'));\n END IF;\nEND IF;\n\n-- Special Case when comparing a property in a jsonb field to a string or number using eq\n-- Allows to leverage GIN index on jsonb fields\nIF op = 'eq' THEN\n IF j->0 ? 'property'\n AND jsonb_typeof(j->1) IN ('number','string')\n AND (items_path(j->0->>'property')).eq IS NOT NULL\n THEN\n RETURN format((items_path(j->0->>'property')).eq, j->1);\n END IF;\nEND IF;\n\n\n\nIF op ilike 't_%' or op = 'anyinteracts' THEN\n RETURN temporal_op_query(op, j);\nEND IF;\n\nIF op ilike 's_%' or op = 'intersects' THEN\n RETURN spatial_op_query(op, j);\nEND IF;\n\n\nIF jtype = 'object' THEN\n RAISE NOTICE 'parsing object';\n IF j ? 'property' THEN\n -- Convert the property to be used as an identifier\n return (items_path(j->>'property')).path_txt;\n ELSIF _op IS NULL THEN\n -- Iterate to convert elements in an object where the operator has not been set\n -- Combining with AND\n SELECT\n array_to_string(array_agg(cql_query_op(e.value, e.key)), ' AND ')\n INTO ret\n FROM jsonb_each(j) e;\n RETURN ret;\n END IF;\nEND IF;\n\nIF jtype = 'string' THEN\n RETURN quote_literal(j->>0);\nEND IF;\n\nIF jtype ='number' THEN\n RETURN (j->>0)::numeric;\nEND IF;\n\nIF jtype = 'array' AND op IS NULL THEN\n RAISE NOTICE 'Parsing array into array arg. j: %', j;\n SELECT format($f$ '{%s}'::text[] $f$, string_agg(e,',')) INTO ret FROM jsonb_array_elements_text(j) e;\n RETURN ret;\nEND IF;\n\n\n-- If the type of the passed json is an array\n-- Calculate the arguments that will be passed to functions/operators\nIF jtype = 'array' THEN\n RAISE NOTICE 'Parsing array into args. j: %', j;\n -- If any argument is numeric, cast any text arguments to numeric\n IF j @? '$[*] ? (@.type() == \"number\")' THEN\n SELECT INTO args\n array_agg(concat('(',cql_query_op(e),')::numeric'))\n FROM jsonb_array_elements(j) e;\n ELSE\n SELECT INTO args\n array_agg(cql_query_op(e))\n FROM jsonb_array_elements(j) e;\n END IF;\n --RETURN args;\nEND IF;\nRAISE NOTICE 'ARGS after array cleaning: %', args;\n\nIF op IS NULL THEN\n RETURN args::text[];\nEND IF;\n\nIF args IS NULL OR cardinality(args) < 1 THEN\n RAISE NOTICE 'No Args';\n RETURN '';\nEND IF;\n\nIF op IN ('and','or') THEN\n SELECT\n CONCAT(\n '(',\n array_to_string(args, UPPER(CONCAT(' ',op,' '))),\n ')'\n ) INTO ret\n FROM jsonb_array_elements(j) e;\n RETURN ret;\nEND IF;\n\n-- If the op is in the ops json then run using the template in the json\nIF ops ? op THEN\n RAISE NOTICE 'ARGS: % MAPPED: %',args, array_map_literal(args);\n\n RETURN format(concat('(',ops->>op,')'), VARIADIC args);\nEND IF;\n\nRETURN j->>0;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n/* cql_query_op -- Parses a CQL query operation, recursing when necessary\n IN jsonb -- a subelement from a valid stac query\n IN text -- the operator being used on elements passed in\n RETURNS a SQL fragment to be used in a WHERE clause\n*/\nDROP FUNCTION IF EXISTS cql2_query;\nCREATE OR REPLACE FUNCTION cql2_query(j jsonb, recursion int DEFAULT 0) RETURNS text AS $$\nDECLARE\nargs jsonb := j->'args';\njtype text := jsonb_typeof(j->'args');\nop text := lower(j->>'op');\narg jsonb;\nargtext text;\nargstext text[] := '{}'::text[];\ninobj jsonb;\n_numeric text := '';\nops jsonb :=\n '{\n \"eq\": \"%s = %s\",\n \"lt\": \"%s < %s\",\n \"lte\": \"%s <= %s\",\n \"gt\": \"%s > %s\",\n \"gte\": \"%s >= %s\",\n \"le\": \"%s <= %s\",\n \"ge\": \"%s >= %s\",\n \"=\": \"%s = %s\",\n \"<\": \"%s < %s\",\n \"<=\": \"%s <= %s\",\n \">\": \"%s > %s\",\n \">=\": \"%s >= %s\",\n \"like\": \"%s LIKE %s\",\n \"ilike\": \"%s ILIKE %s\",\n \"+\": \"%s + %s\",\n \"-\": \"%s - %s\",\n \"*\": \"%s * %s\",\n \"/\": \"%s / %s\",\n \"in\": \"%s = ANY (%s)\",\n \"not\": \"NOT (%s)\",\n \"between\": \"%s BETWEEN (%2$s)[1] AND (%2$s)[2]\",\n \"lower\":\" lower(%s)\",\n \"upper\":\" upper(%s)\",\n \"isnull\": \"%s IS NULL\"\n }'::jsonb;\nret text;\n\nBEGIN\nRAISE NOTICE 'j: %s', j;\nIF j ? 'filter' THEN\n RETURN cql2_query(j->'filter');\nEND IF;\n\nIF j ? 'upper' THEN\nRAISE NOTICE 'upper %s',jsonb_build_object(\n 'op', 'upper',\n 'args', jsonb_build_array( j-> 'upper')\n ) ;\n RETURN cql2_query(\n jsonb_build_object(\n 'op', 'upper',\n 'args', jsonb_build_array( j-> 'upper')\n )\n );\nEND IF;\n\nIF j ? 'lower' THEN\n RETURN cql2_query(\n jsonb_build_object(\n 'op', 'lower',\n 'args', jsonb_build_array( j-> 'lower')\n )\n );\nEND IF;\n\nIF j ? 'args' AND jsonb_typeof(args) != 'array' THEN\n args := jsonb_build_array(args);\nEND IF;\n-- END Cases where no further nesting is expected\nIF j ? 'op' THEN\n -- Special case to use JSONB index for equality\n IF op IN ('eq', '=')\n AND args->0 ? 'property'\n AND jsonb_typeof(args->1) IN ('number', 'string')\n AND (items_path(args->0->>'property')).eq IS NOT NULL\n THEN\n RETURN format((items_path(args->0->>'property')).eq, args->1);\n END IF;\n\n -- Temporal Query\n IF op ilike 't_%' or op = 'anyinteracts' THEN\n RETURN temporal_op_query(op, args);\n END IF;\n\n -- Spatial Query\n IF op ilike 's_%' or op = 'intersects' THEN\n RETURN spatial_op_query(op, args);\n END IF;\n\n -- In Query - separate into separate eq statements so that we can use eq jsonb optimization\n IF op = 'in' THEN\n RAISE NOTICE '% IN args: %', repeat(' ', recursion), args;\n SELECT INTO inobj\n jsonb_agg(\n jsonb_build_object(\n 'op', 'eq',\n 'args', jsonb_build_array( args->0 , v)\n )\n )\n FROM jsonb_array_elements( args->1) v;\n RETURN cql2_query(jsonb_build_object('op','or','args',inobj));\n END IF;\nEND IF;\n\nIF j ? 'property' THEN\n RETURN (items_path(j->>'property')).path_txt;\nEND IF;\n\nIF j ? 'timestamp' THEN\n RETURN quote_literal(j->>'timestamp');\nEND IF;\n\nRAISE NOTICE '%jtype: %',repeat(' ', recursion), jtype;\nIF jsonb_typeof(j) = 'number' THEN\n RETURN format('%L::numeric', j->>0);\nEND IF;\n\nIF jsonb_typeof(j) = 'string' THEN\n RETURN quote_literal(j->>0);\nEND IF;\n\nIF jsonb_typeof(j) = 'array' THEN\n IF j @? '$[*] ? (@.type() == \"number\")' THEN\n RETURN CONCAT(quote_literal(textarr(j)::text), '::numeric[]');\n ELSE\n RETURN CONCAT(quote_literal(textarr(j)::text), '::text[]');\n END IF;\nEND IF;\nRAISE NOTICE 'ARGS after array cleaning: %', args;\n\nRAISE NOTICE '%beforeargs op: %, args: %',repeat(' ', recursion), op, args;\nIF j ? 'args' THEN\n FOR arg in SELECT * FROM jsonb_array_elements(args) LOOP\n argtext := cql2_query(arg, recursion + 1);\n RAISE NOTICE '% -- arg: %, argtext: %', repeat(' ', recursion), arg, argtext;\n argstext := argstext || argtext;\n END LOOP;\nEND IF;\nRAISE NOTICE '%afterargs op: %, argstext: %',repeat(' ', recursion), op, argstext;\n\n\nIF op IN ('and', 'or') THEN\n RAISE NOTICE 'inand op: %, argstext: %', op, argstext;\n SELECT\n concat(' ( ',array_to_string(array_agg(e), concat(' ',op,' ')),' ) ')\n INTO ret\n FROM unnest(argstext) e;\n RETURN ret;\nEND IF;\n\nIF ops ? op THEN\n IF argstext[2] ~* 'numeric' THEN\n argstext := ARRAY[concat('(',argstext[1],')::numeric')] || argstext[2:3];\n END IF;\n RETURN format(concat('(',ops->>op,')'), VARIADIC argstext);\nEND IF;\n\nRAISE NOTICE '%op: %, argstext: %',repeat(' ', recursion), op, argstext;\n\nRETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n\nCREATE OR REPLACE FUNCTION cql_to_where(_search jsonb = '{}'::jsonb) RETURNS text AS $$\nDECLARE\nfilterlang text;\nsearch jsonb := _search;\nbase_where text;\n_where text;\nBEGIN\n\nRAISE NOTICE 'SEARCH CQL Final: %', search;\nfilterlang := COALESCE(\n search->>'filter-lang',\n get_setting('default-filter-lang', _search->'conf')\n);\n\nbase_where := base_stac_query(search);\n\nIF filterlang = 'cql-json' THEN\n search := query_to_cqlfilter(search);\n -- search := add_filters_to_cql(search);\n _where := cql_query_op(search->'filter');\nELSE\n _where := cql2_query(search->'filter');\nEND IF;\n\nIF trim(_where) = '' THEN\n _where := NULL;\nEND IF;\n_where := coalesce(_where, ' TRUE ');\nRETURN format('( %s ) %s ', _where, base_where);\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION parse_sort_dir(_dir text, reverse boolean default false) RETURNS text AS $$\nWITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n) SELECT\n CASE\n WHEN NOT reverse THEN d\n WHEN d = 'ASC' THEN 'DESC'\n WHEN d = 'DESC' THEN 'ASC'\n END\nFROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION sort_dir_to_op(_dir text, prev boolean default false) RETURNS text AS $$\nWITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n) SELECT\n CASE\n WHEN d = 'ASC' AND prev THEN '<='\n WHEN d = 'DESC' AND prev THEN '>='\n WHEN d = 'ASC' THEN '>='\n WHEN d = 'DESC' THEN '<='\n END\nFROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION field_orderby(p text) RETURNS text AS $$\nWITH t AS (\n SELECT\n replace(trim(substring(indexdef from 'btree \\((.*)\\)')),' ','')as s\n FROM pg_indexes WHERE schemaname='pgstac' AND tablename='items' AND indexdef ~* 'btree' AND indexdef ~* 'properties'\n) SELECT s FROM t WHERE strpos(s, lower(trim(p)))>0;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION sort_sqlorderby(\n _search jsonb DEFAULT NULL,\n reverse boolean DEFAULT FALSE\n) RETURNS text AS $$\nWITH sortby AS (\n SELECT coalesce(_search->'sortby','[{\"field\":\"datetime\", \"direction\":\"desc\"}]') as sort\n), withid AS (\n SELECT CASE\n WHEN sort @? '$[*] ? (@.field == \"id\")' THEN sort\n ELSE sort || '[{\"field\":\"id\", \"direction\":\"desc\"}]'::jsonb\n END as sort\n FROM sortby\n), withid_rows AS (\n SELECT jsonb_array_elements(sort) as value FROM withid\n),sorts AS (\n SELECT\n coalesce(field_orderby((items_path(value->>'field')).path_txt), (items_path(value->>'field')).path) as key,\n parse_sort_dir(value->>'direction', reverse) as dir\n FROM withid_rows\n)\nSELECT array_to_string(\n array_agg(concat(key, ' ', dir)),\n ', '\n) FROM sorts;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_sort_dir(sort_item jsonb) RETURNS text AS $$\nSELECT CASE WHEN sort_item->>'direction' ILIKE 'desc%' THEN 'DESC' ELSE 'ASC' END;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION get_token_filter(_search jsonb = '{}'::jsonb, token_rec jsonb DEFAULT NULL) RETURNS text AS $$\nDECLARE\n token_id text;\n filters text[] := '{}'::text[];\n prev boolean := TRUE;\n field text;\n dir text;\n sort record;\n orfilters text[] := '{}'::text[];\n andfilters text[] := '{}'::text[];\n output text;\n token_where text;\nBEGIN\n -- If no token provided return NULL\n IF token_rec IS NULL THEN\n IF NOT (_search ? 'token' AND\n (\n (_search->>'token' ILIKE 'prev:%')\n OR\n (_search->>'token' ILIKE 'next:%')\n )\n ) THEN\n RETURN NULL;\n END IF;\n prev := (_search->>'token' ILIKE 'prev:%');\n token_id := substr(_search->>'token', 6);\n SELECT to_jsonb(items) INTO token_rec FROM item_by_id(token_id) as items;\n END IF;\n RAISE NOTICE 'TOKEN ID: %', token_rec->'id';\n\n CREATE TEMP TABLE sorts (\n _row int GENERATED ALWAYS AS IDENTITY NOT NULL,\n _field text PRIMARY KEY,\n _dir text NOT NULL,\n _val text\n ) ON COMMIT DROP;\n\n -- Make sure we only have distinct columns to sort with taking the first one we get\n INSERT INTO sorts (_field, _dir)\n SELECT\n (items_path(value->>'field')).path,\n get_sort_dir(value)\n FROM\n jsonb_array_elements(coalesce(_search->'sortby','[{\"field\":\"datetime\",\"direction\":\"desc\"}]'))\n ON CONFLICT DO NOTHING\n ;\n RAISE NOTICE 'sorts 1: %', (SELECT jsonb_agg(to_json(sorts)) FROM sorts);\n -- Get the first sort direction provided. As the id is a primary key, if there are any\n -- sorts after id they won't do anything, so make sure that id is the last sort item.\n SELECT _dir INTO dir FROM sorts ORDER BY _row ASC LIMIT 1;\n IF EXISTS (SELECT 1 FROM sorts WHERE _field = 'id') THEN\n DELETE FROM sorts WHERE _row > (SELECT _row FROM sorts WHERE _field = 'id' ORDER BY _row ASC);\n ELSE\n INSERT INTO sorts (_field, _dir) VALUES ('id', dir);\n END IF;\n\n -- Add value from looked up item to the sorts table\n UPDATE sorts SET _val=quote_literal(token_rec->>_field);\n\n -- Check if all sorts are the same direction and use row comparison\n -- to filter\n RAISE NOTICE 'sorts 2: %', (SELECT jsonb_agg(to_json(sorts)) FROM sorts);\n\n IF (SELECT count(DISTINCT _dir) FROM sorts) = 1 THEN\n SELECT format(\n '(%s) %s (%s)',\n concat_ws(', ', VARIADIC array_agg(quote_ident(_field))),\n CASE WHEN (prev AND dir = 'ASC') OR (NOT prev AND dir = 'DESC') THEN '<' ELSE '>' END,\n concat_ws(', ', VARIADIC array_agg(_val))\n ) INTO output FROM sorts\n WHERE token_rec ? _field\n ;\n ELSE\n FOR sort IN SELECT * FROM sorts ORDER BY _row asc LOOP\n RAISE NOTICE 'SORT: %', sort;\n IF sort._row = 1 THEN\n orfilters := orfilters || format('(%s %s %s)',\n quote_ident(sort._field),\n CASE WHEN (prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC') THEN '<' ELSE '>' END,\n sort._val\n );\n ELSE\n orfilters := orfilters || format('(%s AND %s %s %s)',\n array_to_string(andfilters, ' AND '),\n quote_ident(sort._field),\n CASE WHEN (prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC') THEN '<' ELSE '>' END,\n sort._val\n );\n\n END IF;\n andfilters := andfilters || format('%s = %s',\n quote_ident(sort._field),\n sort._val\n );\n END LOOP;\n output := array_to_string(orfilters, ' OR ');\n END IF;\n DROP TABLE IF EXISTS sorts;\n token_where := concat('(',coalesce(output,'true'),')');\n IF trim(token_where) = '' THEN\n token_where := NULL;\n END IF;\n RAISE NOTICE 'TOKEN_WHERE: |%|',token_where;\n RETURN token_where;\n END;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION search_tohash(jsonb) RETURNS jsonb AS $$\n SELECT $1 - '{token,limit,context,includes,excludes}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION search_hash(jsonb, jsonb) RETURNS text AS $$\n SELECT md5(concat(search_tohash($1)::text,$2::text));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TABLE IF NOT EXISTS searches(\n hash text GENERATED ALWAYS AS (search_hash(search, metadata)) STORED PRIMARY KEY,\n search jsonb NOT NULL,\n _where text,\n orderby text,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n metadata jsonb DEFAULT '{}'::jsonb NOT NULL\n);\nCREATE TABLE IF NOT EXISTS search_wheres(\n id bigint generated always as identity primary key,\n _where text NOT NULL,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n statslastupdated timestamptz,\n estimated_count bigint,\n estimated_cost float,\n time_to_estimate float,\n total_count bigint,\n time_to_count float,\n partitions text[]\n);\n\nCREATE INDEX IF NOT EXISTS search_wheres_partitions ON search_wheres USING GIN (partitions);\nCREATE UNIQUE INDEX IF NOT EXISTS search_wheres_where ON search_wheres ((md5(_where)));\n\nCREATE OR REPLACE FUNCTION where_stats(inwhere text, updatestats boolean default false, conf jsonb default null) RETURNS search_wheres AS $$\nDECLARE\n t timestamptz;\n i interval;\n explain_json jsonb;\n partitions text[];\n sw search_wheres%ROWTYPE;\n inwhere_hash text := md5(inwhere);\nBEGIN\n SELECT * INTO sw FROM search_wheres WHERE _where=inwhere_hash FOR UPDATE;\n\n -- Update statistics if explicitly set, if statistics do not exist, or statistics ttl has expired\n IF NOT updatestats THEN\n RAISE NOTICE 'Checking if update is needed.';\n RAISE NOTICE 'Stats Last Updated: %', sw.statslastupdated;\n RAISE NOTICE 'TTL: %, Age: %', context_stats_ttl(conf), now() - sw.statslastupdated;\n RAISE NOTICE 'Context: %, Existing Total: %', context(conf), sw.total_count;\n IF\n sw.statslastupdated IS NULL\n OR (now() - sw.statslastupdated) > context_stats_ttl(conf)\n OR (context(conf) != 'off' AND sw.total_count IS NULL)\n THEN\n updatestats := TRUE;\n END IF;\n END IF;\n\n sw._where := inwhere;\n sw.lastused := now();\n sw.usecount := coalesce(sw.usecount,0) + 1;\n\n IF NOT updatestats THEN\n UPDATE search_wheres SET\n lastused = sw.lastused,\n usecount = sw.usecount\n WHERE _where = inwhere_hash\n RETURNING * INTO sw\n ;\n RETURN sw;\n END IF;\n -- Use explain to get estimated count/cost and a list of the partitions that would be hit by the query\n t := clock_timestamp();\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s', inwhere)\n INTO explain_json;\n RAISE NOTICE 'Time for just the explain: %', clock_timestamp() - t;\n WITH t AS (\n SELECT j->>0 as p FROM\n jsonb_path_query(\n explain_json,\n 'strict $.**.\"Relation Name\" ? (@ != null)'\n ) j\n ), ordered AS (\n SELECT p FROM t ORDER BY p DESC\n -- SELECT p FROM t JOIN items_partitions\n -- ON (t.p = items_partitions.partition)\n -- ORDER BY pstart DESC\n )\n SELECT array_agg(p) INTO partitions FROM ordered;\n i := clock_timestamp() - t;\n RAISE NOTICE 'Time for explain + join: %', clock_timestamp() - t;\n\n\n\n sw.statslastupdated := now();\n sw.estimated_count := explain_json->0->'Plan'->'Plan Rows';\n sw.estimated_cost := explain_json->0->'Plan'->'Total Cost';\n sw.time_to_estimate := extract(epoch from i);\n sw.partitions := partitions;\n\n -- Do a full count of rows if context is set to on or if auto is set and estimates are low enough\n IF\n context(conf) = 'on'\n OR\n ( context(conf) = 'auto' AND\n (\n sw.estimated_count < context_estimated_count(conf)\n OR\n sw.estimated_cost < context_estimated_cost(conf)\n )\n )\n THEN\n t := clock_timestamp();\n RAISE NOTICE 'Calculating actual count...';\n EXECUTE format(\n 'SELECT count(*) FROM items WHERE %s',\n inwhere\n ) INTO sw.total_count;\n i := clock_timestamp() - t;\n RAISE NOTICE 'Actual Count: % -- %', sw.total_count, i;\n sw.time_to_count := extract(epoch FROM i);\n ELSE\n sw.total_count := NULL;\n sw.time_to_count := NULL;\n END IF;\n\n\n INSERT INTO search_wheres\n (_where, lastused, usecount, statslastupdated, estimated_count, estimated_cost, time_to_estimate, partitions, total_count, time_to_count)\n SELECT sw._where, sw.lastused, sw.usecount, sw.statslastupdated, sw.estimated_count, sw.estimated_cost, sw.time_to_estimate, sw.partitions, sw.total_count, sw.time_to_count\n ON CONFLICT ((md5(_where)))\n DO UPDATE\n SET\n lastused = sw.lastused,\n usecount = sw.usecount,\n statslastupdated = sw.statslastupdated,\n estimated_count = sw.estimated_count,\n estimated_cost = sw.estimated_cost,\n time_to_estimate = sw.time_to_estimate,\n partitions = sw.partitions,\n total_count = sw.total_count,\n time_to_count = sw.time_to_count\n ;\n RETURN sw;\nEND;\n$$ LANGUAGE PLPGSQL ;\n\n\nCREATE OR REPLACE FUNCTION items_count(_where text) RETURNS bigint AS $$\nDECLARE\ncnt bigint;\nBEGIN\nEXECUTE format('SELECT count(*) FROM items WHERE %s', _where) INTO cnt;\nRETURN cnt;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n\n\n\nDROP FUNCTION IF EXISTS search_query;\nCREATE OR REPLACE FUNCTION search_query(\n _search jsonb = '{}'::jsonb,\n updatestats boolean = false,\n _metadata jsonb = '{}'::jsonb\n) RETURNS searches AS $$\nDECLARE\n search searches%ROWTYPE;\n pexplain jsonb;\n t timestamptz;\n i interval;\nBEGIN\nSELECT * INTO search FROM searches\nWHERE hash=search_hash(_search, _metadata) FOR UPDATE;\n\n-- Calculate the where clause if not already calculated\nIF search._where IS NULL THEN\n search._where := cql_to_where(_search);\nEND IF;\n\n-- Calculate the order by clause if not already calculated\nIF search.orderby IS NULL THEN\n search.orderby := sort_sqlorderby(_search);\nEND IF;\n\nPERFORM where_stats(search._where, updatestats, _search->'conf');\n\nsearch.lastused := now();\nsearch.usecount := coalesce(search.usecount, 0) + 1;\nINSERT INTO searches (search, _where, orderby, lastused, usecount, metadata)\nVALUES (_search, search._where, search.orderby, search.lastused, search.usecount, _metadata)\nON CONFLICT (hash) DO\nUPDATE SET\n _where = EXCLUDED._where,\n orderby = EXCLUDED.orderby,\n lastused = EXCLUDED.lastused,\n usecount = EXCLUDED.usecount,\n metadata = EXCLUDED.metadata\nRETURNING * INTO search\n;\nRETURN search;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n\nCREATE OR REPLACE FUNCTION search(_search jsonb = '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n searches searches%ROWTYPE;\n _where text;\n token_where text;\n full_where text;\n orderby text;\n query text;\n token_type text := substr(_search->>'token',1,4);\n _limit int := coalesce((_search->>'limit')::int, 10);\n curs refcursor;\n cntr int := 0;\n iter_record items%ROWTYPE;\n first_record items%ROWTYPE;\n last_record items%ROWTYPE;\n out_records jsonb := '[]'::jsonb;\n prev_query text;\n next text;\n prev_id text;\n has_next boolean := false;\n has_prev boolean := false;\n prev text;\n total_count bigint;\n context jsonb;\n collection jsonb;\n includes text[];\n excludes text[];\n exit_flag boolean := FALSE;\n batches int := 0;\n timer timestamptz := clock_timestamp();\n pstart timestamptz;\n pend timestamptz;\n pcurs refcursor;\n search_where search_wheres%ROWTYPE;\n id text;\nBEGIN\nCREATE TEMP TABLE results (content jsonb) ON COMMIT DROP;\n-- if ids is set, short circuit and just use direct ids query for each id\n-- skip any paging or caching\n-- hard codes ordering in the same order as the array of ids\nIF _search ? 'ids' THEN\n FOR id IN SELECT jsonb_array_elements_text(_search->'ids') LOOP\n INSERT INTO results (content) SELECT content FROM item_by_id(id) WHERE content IS NOT NULL;\n END LOOP;\n SELECT INTO total_count count(*) FROM results;\nELSE\n searches := search_query(_search);\n _where := searches._where;\n orderby := searches.orderby;\n search_where := where_stats(_where);\n total_count := coalesce(search_where.total_count, search_where.estimated_count);\n\n\n\n IF token_type='prev' THEN\n token_where := get_token_filter(_search, null::jsonb);\n orderby := sort_sqlorderby(_search, TRUE);\n END IF;\n IF token_type='next' THEN\n token_where := get_token_filter(_search, null::jsonb);\n END IF;\n\n full_where := concat_ws(' AND ', _where, token_where);\n RAISE NOTICE 'FULL QUERY % %', full_where, clock_timestamp()-timer;\n timer := clock_timestamp();\n\n\n\n\n FOR query IN SELECT partition_queries(full_where, orderby, search_where.partitions) LOOP\n timer := clock_timestamp();\n query := format('%s LIMIT %s', query, _limit + 1);\n RAISE NOTICE 'Partition Query: %', query;\n batches := batches + 1;\n -- curs = create_cursor(query);\n OPEN curs FOR EXECUTE query;\n LOOP\n FETCH curs into iter_record;\n EXIT WHEN NOT FOUND;\n cntr := cntr + 1;\n last_record := iter_record;\n IF cntr = 1 THEN\n first_record := last_record;\n END IF;\n IF cntr <= _limit THEN\n INSERT INTO results (content) VALUES (last_record.content);\n -- out_records := out_records || last_record.content;\n\n ELSIF cntr > _limit THEN\n has_next := true;\n exit_flag := true;\n EXIT;\n END IF;\n END LOOP;\n CLOSE curs;\n RAISE NOTICE 'Query took %. Total Time %', clock_timestamp()-timer, ftime();\n timer := clock_timestamp();\n EXIT WHEN exit_flag;\n END LOOP;\n RAISE NOTICE 'Scanned through % partitions.', batches;\nEND IF;\n\nSELECT jsonb_agg(content) INTO out_records FROM results WHERE content is not NULL;\n\nDROP TABLE results;\n\n\n-- Flip things around if this was the result of a prev token query\nIF token_type='prev' THEN\n out_records := flip_jsonb_array(out_records);\n first_record := last_record;\nEND IF;\n\n-- If this query has a token, see if there is data before the first record\nIF _search ? 'token' THEN\n prev_query := format(\n 'SELECT 1 FROM items WHERE %s LIMIT 1',\n concat_ws(\n ' AND ',\n _where,\n trim(get_token_filter(_search, to_jsonb(first_record)))\n )\n );\n RAISE NOTICE 'Query to get previous record: % --- %', prev_query, first_record;\n EXECUTE prev_query INTO has_prev;\n IF FOUND and has_prev IS NOT NULL THEN\n RAISE NOTICE 'Query results from prev query: %', has_prev;\n has_prev := TRUE;\n END IF;\nEND IF;\nhas_prev := COALESCE(has_prev, FALSE);\n\nRAISE NOTICE 'token_type: %, has_next: %, has_prev: %', token_type, has_next, has_prev;\nIF has_prev THEN\n prev := out_records->0->>'id';\nEND IF;\nIF has_next OR token_type='prev' THEN\n next := out_records->-1->>'id';\nEND IF;\n\n\n-- include/exclude any fields following fields extension\nIF _search ? 'fields' THEN\n IF _search->'fields' ? 'exclude' THEN\n excludes=textarr(_search->'fields'->'exclude');\n END IF;\n IF _search->'fields' ? 'include' THEN\n includes=textarr(_search->'fields'->'include');\n IF array_length(includes, 1)>0 AND NOT 'id' = ANY (includes) THEN\n includes = includes || '{id}';\n END IF;\n END IF;\n SELECT jsonb_agg(filter_jsonb(row, includes, excludes)) INTO out_records FROM jsonb_array_elements(out_records) row;\nEND IF;\n\nIF context(_search->'conf') != 'off' THEN\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'matched', total_count,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\nELSE\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\nEND IF;\n\ncollection := jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb),\n 'next', next,\n 'prev', prev,\n 'context', context\n);\n\nRETURN collection;\nEND;\n$$ LANGUAGE PLPGSQL\nSET jit TO off\n;\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION tileenvelope(zoom int, x int, y int) RETURNS geometry AS $$\nWITH t AS (\n SELECT\n 20037508.3427892 as merc_max,\n -20037508.3427892 as merc_min,\n (2 * 20037508.3427892) / (2 ^ zoom) as tile_size\n)\nSELECT st_makeenvelope(\n merc_min + (tile_size * x),\n merc_max - (tile_size * (y + 1)),\n merc_min + (tile_size * (x + 1)),\n merc_max - (tile_size * y),\n 3857\n) FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;DROP FUNCTION IF EXISTS mercgrid;\n\n\nCREATE OR REPLACE FUNCTION ftime() RETURNS interval as $$\nSELECT age(clock_timestamp(), transaction_timestamp());\n$$ LANGUAGE SQL;\nSET SEARCH_PATH to pgstac, public;\n\nDROP FUNCTION IF EXISTS geometrysearch;\nCREATE OR REPLACE FUNCTION geometrysearch(\n IN geom geometry,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE, -- Return as soon as the passed in geometry is full covered\n IN skipcovered boolean DEFAULT TRUE -- Skip any items that would show up completely under the previous items\n) RETURNS jsonb AS $$\nDECLARE\n search searches%ROWTYPE;\n curs refcursor;\n _where text;\n query text;\n iter_record items%ROWTYPE;\n out_records jsonb[] := '{}'::jsonb[];\n exit_flag boolean := FALSE;\n counter int := 1;\n scancounter int := 1;\n remaining_limit int := _scanlimit;\n tilearea float;\n unionedgeom geometry;\n clippedgeom geometry;\n unionedgeom_area float := 0;\n prev_area float := 0;\n excludes text[];\n includes text[];\n\nBEGIN\n -- If skipcovered is true then you will always want to exit when the passed in geometry is full\n IF skipcovered THEN\n exitwhenfull := TRUE;\n END IF;\n\n SELECT * INTO search FROM searches WHERE hash=queryhash;\n\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Search with Query Hash % Not Found', queryhash;\n END IF;\n\n tilearea := st_area(geom);\n _where := format('%s AND st_intersects(geometry, %L::geometry)', search._where, geom);\n\n IF fields IS NOT NULL THEN\n IF fields ? 'fields' THEN\n fields := fields->'fields';\n END IF;\n IF fields ? 'exclude' THEN\n excludes=textarr(fields->'exclude');\n END IF;\n IF fields ? 'include' THEN\n includes=textarr(fields->'include');\n IF array_length(includes, 1)>0 AND NOT 'id' = ANY (includes) THEN\n includes = includes || '{id}';\n END IF;\n END IF;\n END IF;\n RAISE NOTICE 'fields: %, includes: %, excludes: %', fields, includes, excludes;\n\n FOR query IN SELECT * FROM partition_queries(_where, search.orderby) LOOP\n query := format('%s LIMIT %L', query, remaining_limit);\n RAISE NOTICE '%', query;\n curs = create_cursor(query);\n LOOP\n FETCH curs INTO iter_record;\n EXIT WHEN NOT FOUND;\n IF exitwhenfull OR skipcovered THEN -- If we are not using exitwhenfull or skipcovered, we do not need to do expensive geometry operations\n clippedgeom := st_intersection(geom, iter_record.geometry);\n\n IF unionedgeom IS NULL THEN\n unionedgeom := clippedgeom;\n ELSE\n unionedgeom := st_union(unionedgeom, clippedgeom);\n END IF;\n\n unionedgeom_area := st_area(unionedgeom);\n\n IF skipcovered AND prev_area = unionedgeom_area THEN\n scancounter := scancounter + 1;\n CONTINUE;\n END IF;\n\n prev_area := unionedgeom_area;\n\n RAISE NOTICE '% % % %', unionedgeom_area/tilearea, counter, scancounter, ftime();\n END IF;\n\n IF fields IS NOT NULL THEN\n out_records := out_records || filter_jsonb(iter_record.content, includes, excludes);\n ELSE\n out_records := out_records || iter_record.content;\n END IF;\n IF counter >= _limit\n OR scancounter > _scanlimit\n OR ftime() > _timelimit\n OR (exitwhenfull AND unionedgeom_area >= tilearea)\n THEN\n exit_flag := TRUE;\n EXIT;\n END IF;\n counter := counter + 1;\n scancounter := scancounter + 1;\n\n END LOOP;\n EXIT WHEN exit_flag;\n remaining_limit := _scanlimit - scancounter;\n END LOOP;\n\n RETURN jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', array_to_json(out_records)::jsonb\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDROP FUNCTION IF EXISTS geojsonsearch;\nCREATE OR REPLACE FUNCTION geojsonsearch(\n IN geojson jsonb,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_geomfromgeojson(geojson),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS xyzsearch;\nCREATE OR REPLACE FUNCTION xyzsearch(\n IN _x int,\n IN _y int,\n IN _z int,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_transform(tileenvelope(_z, _x, _y), 4326),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\nSELECT set_version('0.4.5');\n" }, { "path": "src/pgstac/migrations/pgstac.0.5.0-0.5.1.sql", "content": "SET SEARCH_PATH to pgstac, public;\nset check_function_bodies = off;\n\nCREATE OR REPLACE FUNCTION pgstac.content_nonhydrated(_item pgstac.items, fields jsonb DEFAULT '{}'::jsonb)\n RETURNS jsonb\n LANGUAGE plpgsql\n STABLE PARALLEL SAFE\nAS $function$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry)::jsonb;\n END IF;\n IF include_field('bbox', fields) THEN\n bbox := geom_bbox(_item.geometry)::jsonb;\n END IF;\n output := jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'bbox',bbox,\n 'collection', _item.collection\n ) || _item.content;\n RETURN output;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.search(_search jsonb DEFAULT '{}'::jsonb)\n RETURNS jsonb\n LANGUAGE plpgsql\n SECURITY DEFINER\n SET search_path TO 'pgstac', 'public'\nAS $function$\nDECLARE\n searches searches%ROWTYPE;\n _where text;\n token_where text;\n full_where text;\n orderby text;\n query text;\n token_type text := substr(_search->>'token',1,4);\n _limit int := coalesce((_search->>'limit')::int, 10);\n curs refcursor;\n cntr int := 0;\n iter_record items%ROWTYPE;\n first_record jsonb;\n last_record jsonb;\n out_records jsonb := '[]'::jsonb;\n prev_query text;\n next text;\n prev_id text;\n has_next boolean := false;\n has_prev boolean := false;\n prev text;\n total_count bigint;\n context jsonb;\n collection jsonb;\n includes text[];\n excludes text[];\n exit_flag boolean := FALSE;\n batches int := 0;\n timer timestamptz := clock_timestamp();\n pstart timestamptz;\n pend timestamptz;\n pcurs refcursor;\n search_where search_wheres%ROWTYPE;\n id text;\nBEGIN\nCREATE TEMP TABLE results (content jsonb) ON COMMIT DROP;\n-- if ids is set, short circuit and just use direct ids query for each id\n-- skip any paging or caching\n-- hard codes ordering in the same order as the array of ids\nIF _search ? 'ids' THEN\n INSERT INTO results\n SELECT\n CASE WHEN _search->'conf'->>'nohydrate' IS NOT NULL AND (_search->'conf'->>'nohydrate')::boolean = true THEN\n content_nonhydrated(items, _search->'fields')\n ELSE\n content_hydrate(items, _search->'fields')\n END\n FROM items WHERE\n items.id = ANY(to_text_array(_search->'ids'))\n AND\n CASE WHEN _search ? 'collections' THEN\n items.collection = ANY(to_text_array(_search->'collections'))\n ELSE TRUE\n END\n ORDER BY items.datetime desc, items.id desc\n ;\n SELECT INTO total_count count(*) FROM results;\nELSE\n searches := search_query(_search);\n _where := searches._where;\n orderby := searches.orderby;\n search_where := where_stats(_where);\n total_count := coalesce(search_where.total_count, search_where.estimated_count);\n\n IF token_type='prev' THEN\n token_where := get_token_filter(_search, null::jsonb);\n orderby := sort_sqlorderby(_search, TRUE);\n END IF;\n IF token_type='next' THEN\n token_where := get_token_filter(_search, null::jsonb);\n END IF;\n\n full_where := concat_ws(' AND ', _where, token_where);\n RAISE NOTICE 'FULL QUERY % %', full_where, clock_timestamp()-timer;\n timer := clock_timestamp();\n\n FOR query IN SELECT partition_queries(full_where, orderby, search_where.partitions) LOOP\n timer := clock_timestamp();\n query := format('%s LIMIT %s', query, _limit + 1);\n RAISE NOTICE 'Partition Query: %', query;\n batches := batches + 1;\n -- curs = create_cursor(query);\n OPEN curs FOR EXECUTE query;\n LOOP\n FETCH curs into iter_record;\n EXIT WHEN NOT FOUND;\n cntr := cntr + 1;\n\n IF _search->'conf'->>'nohydrate' IS NOT NULL AND (_search->'conf'->>'nohydrate')::boolean = true THEN\n last_record := content_nonhydrated(iter_record, _search->'fields');\n ELSE\n last_record := content_hydrate(iter_record, _search->'fields');\n END IF;\n IF cntr = 1 THEN\n first_record := last_record;\n END IF;\n IF cntr <= _limit THEN\n INSERT INTO results (content) VALUES (last_record);\n ELSIF cntr > _limit THEN\n has_next := true;\n exit_flag := true;\n EXIT;\n END IF;\n END LOOP;\n CLOSE curs;\n RAISE NOTICE 'Query took %.', clock_timestamp()-timer;\n timer := clock_timestamp();\n EXIT WHEN exit_flag;\n END LOOP;\n RAISE NOTICE 'Scanned through % partitions.', batches;\nEND IF;\n\nSELECT jsonb_agg(content) INTO out_records FROM results WHERE content is not NULL;\n\nDROP TABLE results;\n\n\n-- Flip things around if this was the result of a prev token query\nIF token_type='prev' THEN\n out_records := flip_jsonb_array(out_records);\n first_record := last_record;\nEND IF;\n\n-- If this query has a token, see if there is data before the first record\nIF _search ? 'token' THEN\n prev_query := format(\n 'SELECT 1 FROM items WHERE %s LIMIT 1',\n concat_ws(\n ' AND ',\n _where,\n trim(get_token_filter(_search, to_jsonb(content_dehydrate(first_record))))\n )\n );\n RAISE NOTICE 'Query to get previous record: % --- %', prev_query, first_record;\n EXECUTE prev_query INTO has_prev;\n IF FOUND and has_prev IS NOT NULL THEN\n RAISE NOTICE 'Query results from prev query: %', has_prev;\n has_prev := TRUE;\n END IF;\nEND IF;\nhas_prev := COALESCE(has_prev, FALSE);\n\nIF has_prev THEN\n prev := out_records->0->>'id';\nEND IF;\nIF has_next OR token_type='prev' THEN\n next := out_records->-1->>'id';\nEND IF;\n\nIF context(_search->'conf') != 'off' THEN\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'matched', total_count,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\nELSE\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\nEND IF;\n\ncollection := jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb),\n 'next', next,\n 'prev', prev,\n 'context', context\n);\n\nRETURN collection;\nEND;\n$function$\n;\n\n\n\nSELECT set_version('0.5.1');\n" }, { "path": "src/pgstac/migrations/pgstac.0.5.0.sql", "content": "CREATE EXTENSION IF NOT EXISTS postgis;\nCREATE EXTENSION IF NOT EXISTS btree_gist;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_admin;\n CREATE ROLE pgstac_read;\n CREATE ROLE pgstac_ingest;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nGRANT pgstac_admin TO current_user;\n\nCREATE SCHEMA IF NOT EXISTS pgstac AUTHORIZATION pgstac_admin;\n\nALTER ROLE pgstac_admin SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_read SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_ingest SET SEARCH_PATH TO pgstac, public;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\n\nGRANT pgstac_read TO pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\n\nSET ROLE pgstac_admin;\n\nSET SEARCH_PATH TO pgstac, public;\n\n\nCREATE TABLE IF NOT EXISTS migrations (\n version text PRIMARY KEY,\n datetime timestamptz DEFAULT clock_timestamp() NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION get_version() RETURNS text AS $$\n SELECT version FROM pgstac.migrations ORDER BY datetime DESC, version DESC LIMIT 1;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION set_version(text) RETURNS text AS $$\n INSERT INTO pgstac.migrations (version) VALUES ($1)\n ON CONFLICT DO NOTHING\n RETURNING version;\n$$ LANGUAGE SQL;\n\n\nCREATE TABLE IF NOT EXISTS pgstac_settings (\n name text PRIMARY KEY,\n value text NOT NULL\n);\n\nINSERT INTO pgstac_settings (name, value) VALUES\n ('context', 'off'),\n ('context_estimated_count', '100000'),\n ('context_estimated_cost', '100000'),\n ('context_stats_ttl', '1 day'),\n ('default-filter-lang', 'cql2-json'),\n ('additional_properties', 'true')\nON CONFLICT DO NOTHING\n;\n\n\nCREATE OR REPLACE FUNCTION get_setting(IN _setting text, IN conf jsonb DEFAULT NULL) RETURNS text AS $$\nSELECT COALESCE(\n conf->>_setting,\n current_setting(concat('pgstac.',_setting), TRUE),\n (SELECT value FROM pgstac.pgstac_settings WHERE name=_setting)\n);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context(conf jsonb DEFAULT NULL) RETURNS text AS $$\n SELECT pgstac.get_setting('context', conf);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context_estimated_count(conf jsonb DEFAULT NULL) RETURNS int AS $$\n SELECT pgstac.get_setting('context_estimated_count', conf)::int;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_estimated_cost();\nCREATE OR REPLACE FUNCTION context_estimated_cost(conf jsonb DEFAULT NULL) RETURNS float AS $$\n SELECT pgstac.get_setting('context_estimated_cost', conf)::float;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_stats_ttl();\nCREATE OR REPLACE FUNCTION context_stats_ttl(conf jsonb DEFAULT NULL) RETURNS interval AS $$\n SELECT pgstac.get_setting('context_stats_ttl', conf)::interval;\n$$ LANGUAGE SQL;\n\n\nCREATE OR REPLACE FUNCTION notice(VARIADIC text[]) RETURNS boolean AS $$\nDECLARE\ndebug boolean := current_setting('pgstac.debug', true);\nBEGIN\n IF debug THEN\n RAISE NOTICE 'NOTICE FROM FUNC: % >>>>> %', concat_ws(' | ', $1), clock_timestamp();\n RETURN TRUE;\n END IF;\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION empty_arr(ANYARRAY) RETURNS BOOLEAN AS $$\nSELECT CASE\n WHEN $1 IS NULL THEN TRUE\n WHEN cardinality($1)<1 THEN TRUE\nELSE FALSE\nEND;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_intersection(_a ANYARRAY, _b ANYARRAY) RETURNS ANYARRAY AS $$\n SELECT ARRAY ( SELECT unnest(_a) INTERSECT SELECT UNNEST(_b) );\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION array_map_ident(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_ident(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_map_literal(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_literal(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION array_reverse(anyarray) RETURNS anyarray AS $$\nSELECT ARRAY(\n SELECT $1[i]\n FROM generate_subscripts($1,1) AS s(i)\n ORDER BY i DESC\n);\n$$ LANGUAGE SQL STRICT IMMUTABLE;\nCREATE OR REPLACE FUNCTION to_int(jsonb) RETURNS int AS $$\n SELECT floor(($1->>0)::float)::int;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION to_float(jsonb) RETURNS float AS $$\n SELECT ($1->>0)::float;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION to_tstz(jsonb) RETURNS timestamptz AS $$\n SELECT ($1->>0)::timestamptz;\n$$ LANGUAGE SQL IMMUTABLE STRICT SET TIME ZONE 'UTC';\n\n\nCREATE OR REPLACE FUNCTION to_text(jsonb) RETURNS text AS $$\n SELECT $1->>0;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION to_text_array(jsonb) RETURNS text[] AS $$\n SELECT\n CASE jsonb_typeof($1)\n WHEN 'array' THEN ARRAY(SELECT jsonb_array_elements_text($1))\n ELSE ARRAY[$1->>0]\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\nCREATE OR REPLACE FUNCTION bbox_geom(_bbox jsonb) RETURNS geometry AS $$\nSELECT CASE jsonb_array_length(_bbox)\n WHEN 4 THEN\n ST_SetSRID(ST_MakeEnvelope(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float,\n (_bbox->>3)::float\n ),4326)\n WHEN 6 THEN\n ST_SetSRID(ST_3DMakeBox(\n ST_MakePoint(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float\n ),\n ST_MakePoint(\n (_bbox->>3)::float,\n (_bbox->>4)::float,\n (_bbox->>5)::float\n )\n ),4326)\n ELSE null END;\n;\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION geom_bbox(_geom geometry) RETURNS jsonb AS $$\n SELECT jsonb_build_array(\n st_xmin(_geom),\n st_ymin(_geom),\n st_xmax(_geom),\n st_ymax(_geom)\n );\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION flip_jsonb_array(j jsonb) RETURNS jsonb AS $$\n SELECT jsonb_agg(value) FROM (SELECT value FROM jsonb_array_elements(j) WITH ORDINALITY ORDER BY ordinality DESC) as t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n/* looks for a geometry in a stac item first from geometry and falling back to bbox */\nCREATE OR REPLACE FUNCTION stac_geom(value jsonb) RETURNS geometry AS $$\nSELECT\n CASE\n WHEN value ? 'intersects' THEN\n ST_GeomFromGeoJSON(value->>'intersects')\n WHEN value ? 'geometry' THEN\n ST_GeomFromGeoJSON(value->>'geometry')\n WHEN value ? 'bbox' THEN\n pgstac.bbox_geom(value->'bbox')\n ELSE NULL\n END as geometry\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION stac_daterange(\n value jsonb\n) RETURNS tstzrange AS $$\nDECLARE\n props jsonb := value;\n dt timestamptz;\n edt timestamptz;\nBEGIN\n IF props ? 'properties' THEN\n props := props->'properties';\n END IF;\n IF props ? 'start_datetime' AND props ? 'end_datetime' THEN\n dt := props->'start_datetime';\n edt := props->'end_datetime';\n IF dt > edt THEN\n RAISE EXCEPTION 'start_datetime must be < end_datetime';\n END IF;\n ELSE\n dt := props->'datetime';\n edt := props->'datetime';\n END IF;\n IF dt is NULL OR edt IS NULL THEN\n RAISE EXCEPTION 'Either datetime or both start_datetime and end_datetime must be set.';\n END IF;\n RETURN tstzrange(dt, edt, '[]');\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_datetime(value jsonb) RETURNS timestamptz AS $$\n SELECT lower(stac_daterange(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_end_datetime(value jsonb) RETURNS timestamptz AS $$\n SELECT upper(stac_daterange(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\n\nCREATE TABLE IF NOT EXISTS stac_extensions(\n name text PRIMARY KEY,\n url text,\n enbabled_by_default boolean NOT NULL DEFAULT TRUE,\n enableable boolean NOT NULL DEFAULT TRUE\n);\n\nINSERT INTO stac_extensions (name, url) VALUES\n ('fields', 'https://api.stacspec.org/v1.0.0-beta.5/item-search#fields'),\n ('sort','https://api.stacspec.org/v1.0.0-beta.5/item-search#sort'),\n ('context','https://api.stacspec.org/v1.0.0-beta.5/item-search#context'),\n ('filter', 'https://api.stacspec.org/v1.0.0-beta.5/item-search#filter'),\n ('query', 'https://api.stacspec.org/v1.0.0-beta.5/item-search#query')\nON CONFLICT (name) DO UPDATE SET url=EXCLUDED.url;\n\n\n\nCREATE OR REPLACE FUNCTION collection_base_item(content jsonb) RETURNS jsonb AS $$\n SELECT jsonb_build_object(\n 'type', 'Feature',\n 'stac_version', content->'stac_version',\n 'assets', content->'item_assets',\n 'collection', content->'id',\n 'links', '[]'::jsonb\n );\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TYPE partition_trunc_strategy AS ENUM ('year', 'month');\n\nCREATE TABLE IF NOT EXISTS collections (\n key bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,\n id text GENERATED ALWAYS AS (content->>'id') STORED UNIQUE,\n content JSONB NOT NULL,\n base_item jsonb GENERATED ALWAYS AS (pgstac.collection_base_item(content)) STORED,\n partition_trunc partition_trunc_strategy\n);\n\n\nCREATE OR REPLACE FUNCTION collection_base_item(cid text) RETURNS jsonb AS $$\n SELECT pgstac.collection_base_item(content) FROM pgstac.collections WHERE id = cid LIMIT 1;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION table_empty(text) RETURNS boolean AS $$\nDECLARE\n retval boolean;\nBEGIN\n EXECUTE format($q$\n SELECT NOT EXISTS (SELECT 1 FROM %I LIMIT 1)\n $q$,\n $1\n ) INTO retval;\n RETURN retval;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION collections_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n q text;\n partition_name text := format('_items_%s', NEW.key);\n partition_exists boolean := false;\n partition_empty boolean := true;\n err_context text;\n loadtemp boolean := FALSE;\nBEGIN\n RAISE NOTICE 'Collection Trigger. % %', NEW.id, NEW.key;\n SELECT relid::text INTO partition_name\n FROM pg_partition_tree('items')\n WHERE relid::text = partition_name;\n IF FOUND THEN\n partition_exists := true;\n partition_empty := table_empty(partition_name);\n ELSE\n partition_exists := false;\n partition_empty := true;\n partition_name := format('_items_%s', NEW.key);\n END IF;\n IF TG_OP = 'UPDATE' AND NEW.partition_trunc IS DISTINCT FROM OLD.partition_trunc AND partition_empty THEN\n q := format($q$\n DROP TABLE IF EXISTS %I CASCADE;\n $q$,\n partition_name\n );\n EXECUTE q;\n END IF;\n IF TG_OP = 'UPDATE' AND NEW.partition_trunc IS DISTINCT FROM OLD.partition_trunc AND partition_exists AND NOT partition_empty THEN\n q := format($q$\n CREATE TEMP TABLE changepartitionstaging ON COMMIT DROP AS SELECT * FROM %I;\n DROP TABLE IF EXISTS %I CASCADE;\n $q$,\n partition_name,\n partition_name\n );\n EXECUTE q;\n loadtemp := TRUE;\n partition_empty := TRUE;\n partition_exists := FALSE;\n END IF;\n IF TG_OP = 'UPDATE' AND NEW.partition_trunc IS NOT DISTINCT FROM OLD.partition_trunc THEN\n RETURN NEW;\n END IF;\n IF NEW.partition_trunc IS NULL AND partition_empty THEN\n RAISE NOTICE '% % % %',\n partition_name,\n NEW.id,\n concat(partition_name,'_id_idx'),\n partition_name\n ;\n q := format($q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n $q$,\n partition_name,\n NEW.id,\n concat(partition_name,'_id_idx'),\n partition_name\n );\n RAISE NOTICE 'q: %', q;\n BEGIN\n EXECUTE q;\n EXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', partition_name;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\n END;\n DELETE FROM partitions WHERE collection=NEW.id AND name=partition_name;\n INSERT INTO partitions (collection, name) VALUES (NEW.id, partition_name);\n ELSIF partition_empty THEN\n q := format($q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L)\n PARTITION BY RANGE (datetime);\n $q$,\n partition_name,\n NEW.id\n );\n RAISE NOTICE 'q: %', q;\n BEGIN\n EXECUTE q;\n EXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', partition_name;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\n END;\n DELETE FROM partitions WHERE collection=NEW.id AND name=partition_name;\n ELSE\n RAISE EXCEPTION 'Cannot modify partition % unless empty', partition_name;\n END IF;\n IF loadtemp THEN\n RAISE NOTICE 'Moving data into new partitions.';\n q := format($q$\n WITH p AS (\n SELECT\n collection,\n datetime as datetime,\n end_datetime as end_datetime,\n (partition_name(\n collection,\n datetime\n )).partition_name as name\n FROM changepartitionstaging\n )\n INSERT INTO partitions (collection, datetime_range, end_datetime_range)\n SELECT\n collection,\n tstzrange(min(datetime), max(datetime), '[]') as datetime_range,\n tstzrange(min(end_datetime), max(end_datetime), '[]') as end_datetime_range\n FROM p\n GROUP BY collection, name\n ON CONFLICT (name) DO UPDATE SET\n datetime_range = EXCLUDED.datetime_range,\n end_datetime_range = EXCLUDED.end_datetime_range\n ;\n INSERT INTO %I SELECT * FROM changepartitionstaging;\n DROP TABLE IF EXISTS changepartitionstaging;\n $q$,\n partition_name\n );\n EXECUTE q;\n END IF;\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE TRIGGER collections_trigger AFTER INSERT OR UPDATE ON collections FOR EACH ROW\nEXECUTE FUNCTION collections_trigger_func();\n\nCREATE OR REPLACE FUNCTION partition_collection(collection text, strategy partition_trunc_strategy) RETURNS text AS $$\n UPDATE collections SET partition_trunc=strategy WHERE id=collection RETURNING partition_trunc;\n$$ LANGUAGE SQL;\n\nCREATE TABLE IF NOT EXISTS partitions (\n collection text REFERENCES collections(id),\n name text PRIMARY KEY,\n partition_range tstzrange NOT NULL DEFAULT tstzrange('-infinity'::timestamptz,'infinity'::timestamptz, '[]'),\n datetime_range tstzrange,\n end_datetime_range tstzrange,\n CONSTRAINT prange EXCLUDE USING GIST (\n collection WITH =,\n partition_range WITH &&\n )\n) WITH (FILLFACTOR=90);\nCREATE INDEX partitions_range_idx ON partitions USING GIST(partition_range);\n\n\n\nCREATE OR REPLACE FUNCTION partition_name(\n IN collection text,\n IN dt timestamptz,\n OUT partition_name text,\n OUT partition_range tstzrange\n) AS $$\nDECLARE\n c RECORD;\n parent_name text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', collection;\n END IF;\n parent_name := format('_items_%s', c.key);\n\n\n IF c.partition_trunc = 'year' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYY'));\n ELSIF c.partition_trunc = 'month' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYYMM'));\n ELSE\n partition_name := parent_name;\n partition_range := tstzrange('-infinity'::timestamptz, 'infinity'::timestamptz, '[]');\n END IF;\n IF partition_range IS NULL THEN\n partition_range := tstzrange(\n date_trunc(c.partition_trunc::text, dt),\n date_trunc(c.partition_trunc::text, dt) + concat('1 ', c.partition_trunc)::interval\n );\n END IF;\n RETURN;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\n\nCREATE OR REPLACE FUNCTION partitions_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n q text;\n cq text;\n parent_name text;\n partition_trunc text;\n partition_name text := NEW.name;\n partition_exists boolean := false;\n partition_empty boolean := true;\n partition_range tstzrange;\n datetime_range tstzrange;\n end_datetime_range tstzrange;\n err_context text;\n mindt timestamptz := lower(NEW.datetime_range);\n maxdt timestamptz := upper(NEW.datetime_range);\n minedt timestamptz := lower(NEW.end_datetime_range);\n maxedt timestamptz := upper(NEW.end_datetime_range);\n t_mindt timestamptz;\n t_maxdt timestamptz;\n t_minedt timestamptz;\n t_maxedt timestamptz;\nBEGIN\n RAISE NOTICE 'Partitions Trigger. %', NEW;\n datetime_range := NEW.datetime_range;\n end_datetime_range := NEW.end_datetime_range;\n\n SELECT\n format('_items_%s', key),\n c.partition_trunc::text\n INTO\n parent_name,\n partition_trunc\n FROM pgstac.collections c\n WHERE c.id = NEW.collection;\n SELECT (pgstac.partition_name(NEW.collection, mindt)).* INTO partition_name, partition_range;\n NEW.name := partition_name;\n\n IF partition_range IS NULL OR partition_range = 'empty'::tstzrange THEN\n partition_range := tstzrange('-infinity'::timestamptz, 'infinity'::timestamptz, '[]');\n END IF;\n\n NEW.partition_range := partition_range;\n IF TG_OP = 'UPDATE' THEN\n mindt := least(mindt, lower(OLD.datetime_range));\n maxdt := greatest(maxdt, upper(OLD.datetime_range));\n minedt := least(minedt, lower(OLD.end_datetime_range));\n maxedt := greatest(maxedt, upper(OLD.end_datetime_range));\n NEW.datetime_range := tstzrange(mindt, maxdt, '[]');\n NEW.end_datetime_range := tstzrange(minedt, maxedt, '[]');\n END IF;\n IF TG_OP = 'INSERT' THEN\n\n IF partition_range != tstzrange('-infinity'::timestamptz, 'infinity'::timestamptz, '[]') THEN\n\n RAISE NOTICE '% % %', partition_name, parent_name, partition_range;\n q := format($q$\n CREATE TABLE IF NOT EXISTS %I partition OF %I FOR VALUES FROM (%L) TO (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n $q$,\n partition_name,\n parent_name,\n lower(partition_range),\n upper(partition_range),\n format('%s_pkey', partition_name),\n partition_name\n );\n BEGIN\n EXECUTE q;\n EXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', partition_name;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\n END;\n END IF;\n\n END IF;\n\n -- Update constraints\n EXECUTE format($q$\n SELECT\n min(datetime),\n max(datetime),\n min(end_datetime),\n max(end_datetime)\n FROM %I;\n $q$, partition_name)\n INTO t_mindt, t_maxdt, t_minedt, t_maxedt;\n mindt := least(mindt, t_mindt);\n maxdt := greatest(maxdt, t_maxdt);\n minedt := least(mindt, minedt, t_minedt);\n maxedt := greatest(maxdt, maxedt, t_maxedt);\n\n mindt := date_trunc(coalesce(partition_trunc, 'year'), mindt);\n maxdt := date_trunc(coalesce(partition_trunc, 'year'), maxdt - '1 second'::interval) + concat('1 ',coalesce(partition_trunc, 'year'))::interval;\n minedt := date_trunc(coalesce(partition_trunc, 'year'), minedt);\n maxedt := date_trunc(coalesce(partition_trunc, 'year'), maxedt - '1 second'::interval) + concat('1 ',coalesce(partition_trunc, 'year'))::interval;\n\n\n IF mindt IS NOT NULL AND maxdt IS NOT NULL AND minedt IS NOT NULL AND maxedt IS NOT NULL THEN\n NEW.datetime_range := tstzrange(mindt, maxdt, '[]');\n NEW.end_datetime_range := tstzrange(minedt, maxedt, '[]');\n IF\n TG_OP='UPDATE'\n AND OLD.datetime_range @> NEW.datetime_range\n AND OLD.end_datetime_range @> NEW.end_datetime_range\n THEN\n RAISE NOTICE 'Range unchanged, not updating constraints.';\n ELSE\n\n RAISE NOTICE '\n SETTING CONSTRAINTS\n mindt: %, maxdt: %\n minedt: %, maxedt: %\n ', mindt, maxdt, minedt, maxedt;\n IF partition_trunc IS NULL THEN\n cq := format($q$\n ALTER TABLE %7$I\n DROP CONSTRAINT IF EXISTS %1$I,\n DROP CONSTRAINT IF EXISTS %2$I,\n ADD CONSTRAINT %1$I\n CHECK (\n (datetime >= %3$L)\n AND (datetime <= %4$L)\n AND (end_datetime >= %5$L)\n AND (end_datetime <= %6$L)\n ) NOT VALID\n ;\n ALTER TABLE %7$I\n VALIDATE CONSTRAINT %1$I;\n $q$,\n format('%s_dt', partition_name),\n format('%s_edt', partition_name),\n mindt,\n maxdt,\n minedt,\n maxedt,\n partition_name\n );\n ELSE\n cq := format($q$\n ALTER TABLE %5$I\n DROP CONSTRAINT IF EXISTS %1$I,\n DROP CONSTRAINT IF EXISTS %2$I,\n ADD CONSTRAINT %2$I\n CHECK ((end_datetime >= %3$L) AND (end_datetime <= %4$L)) NOT VALID\n ;\n ALTER TABLE %5$I\n VALIDATE CONSTRAINT %2$I;\n $q$,\n format('%s_dt', partition_name),\n format('%s_edt', partition_name),\n minedt,\n maxedt,\n partition_name\n );\n\n END IF;\n RAISE NOTICE 'Altering Constraints. %', cq;\n EXECUTE cq;\n END IF;\n ELSE\n NEW.datetime_range = NULL;\n NEW.end_datetime_range = NULL;\n\n cq := format($q$\n ALTER TABLE %3$I\n DROP CONSTRAINT IF EXISTS %1$I,\n DROP CONSTRAINT IF EXISTS %2$I,\n ADD CONSTRAINT %1$I\n CHECK ((datetime IS NULL AND end_datetime IS NULL)) NOT VALID\n ;\n ALTER TABLE %3$I\n VALIDATE CONSTRAINT %1$I;\n $q$,\n format('%s_dt', partition_name),\n format('%s_edt', partition_name),\n partition_name\n );\n EXECUTE cq;\n END IF;\n\n RETURN NEW;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER partitions_trigger BEFORE INSERT OR UPDATE ON partitions FOR EACH ROW\nEXECUTE FUNCTION partitions_trigger_func();\n\n\nCREATE OR REPLACE FUNCTION create_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection(data jsonb) RETURNS VOID AS $$\nDECLARE\n out collections%ROWTYPE;\nBEGIN\n UPDATE collections SET content=data WHERE id = data->>'id' RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ON CONFLICT (id) DO\n UPDATE\n SET content=EXCLUDED.content\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION delete_collection(_id text) RETURNS VOID AS $$\nDECLARE\n out collections%ROWTYPE;\nBEGIN\n DELETE FROM collections WHERE id = _id RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION get_collection(id text) RETURNS jsonb AS $$\n SELECT content FROM collections\n WHERE id=$1\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION all_collections() RETURNS jsonb AS $$\n SELECT jsonb_agg(content) FROM collections;\n;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\nCREATE TABLE queryables (\n id bigint GENERATED ALWAYS AS identity PRIMARY KEY,\n name text UNIQUE NOT NULL,\n collection_ids text[], -- used to determine what partitions to create indexes on\n definition jsonb,\n property_path text,\n property_wrapper text,\n property_index_type text\n);\nCREATE INDEX queryables_name_idx ON queryables (name);\nCREATE INDEX queryables_property_wrapper_idx ON queryables (property_wrapper);\n\n\nINSERT INTO queryables (name, definition) VALUES\n('id', '{\"title\": \"Item ID\",\"description\": \"Item identifier\",\"$ref\": \"https://schemas.stacspec.org/v1.0.0/item-spec/json-schema/item.json#/definitions/core/allOf/2/properties/id\"}'),\n('datetime','{\"description\": \"Datetime\",\"type\": \"string\",\"title\": \"Acquired\",\"format\": \"date-time\",\"pattern\": \"(\\\\+00:00|Z)$\"}')\nON CONFLICT DO NOTHING;\n\n\n\nINSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n('eo:cloud_cover','{\"$ref\": \"https://stac-extensions.github.io/eo/v1.0.0/schema.json#/definitions/fieldsproperties/eo:cloud_cover\"}','to_int','BTREE')\nON CONFLICT DO NOTHING;\n\nCREATE OR REPLACE FUNCTION array_to_path(arr text[]) RETURNS text AS $$\n SELECT string_agg(\n quote_literal(v),\n '->'\n ) FROM unnest(arr) v;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\n\n\nCREATE OR REPLACE FUNCTION queryable(\n IN dotpath text,\n OUT path text,\n OUT expression text,\n OUT wrapper text\n) AS $$\nDECLARE\n q RECORD;\n path_elements text[];\nBEGIN\n IF dotpath IN ('id', 'geometry', 'datetime', 'end_datetime', 'collection') THEN\n path := dotpath;\n expression := dotpath;\n wrapper := NULL;\n RETURN;\n END IF;\n SELECT * INTO q FROM queryables WHERE name=dotpath;\n IF q.property_wrapper IS NULL THEN\n IF q.definition->>'type' = 'number' THEN\n wrapper := 'to_float';\n ELSIF q.definition->>'format' = 'date-time' THEN\n wrapper := 'to_tstz';\n ELSE\n wrapper := 'to_text';\n END IF;\n ELSE\n wrapper := q.property_wrapper;\n END IF;\n IF q.property_path IS NOT NULL THEN\n path := q.property_path;\n ELSE\n path_elements := string_to_array(dotpath, '.');\n IF path_elements[1] IN ('links', 'assets', 'stac_version', 'stac_extensions') THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSIF path_elements[1] = 'properties' THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSE\n path := format($F$content->'properties'->%s$F$, array_to_path(path_elements));\n END IF;\n END IF;\n expression := format('%I(%s)', wrapper, path);\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\nCREATE OR REPLACE FUNCTION create_queryable_indexes() RETURNS VOID AS $$\nDECLARE\n queryable RECORD;\n q text;\nBEGIN\n FOR queryable IN\n SELECT\n queryables.id as qid,\n CASE WHEN collections.key IS NULL THEN 'items' ELSE format('_items_%s',collections.key) END AS part,\n property_index_type,\n expression\n FROM\n queryables\n LEFT JOIN collections ON (collections.id = ANY (queryables.collection_ids))\n JOIN LATERAL queryable(queryables.name) ON (queryables.property_index_type IS NOT NULL)\n LOOP\n q := format(\n $q$\n CREATE INDEX IF NOT EXISTS %I ON %I USING %s ((%s));\n $q$,\n format('%s_%s_idx', queryable.part, queryable.qid),\n queryable.part,\n COALESCE(queryable.property_index_type, 'to_text'),\n queryable.expression\n );\n RAISE NOTICE '%',q;\n EXECUTE q;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION queryables_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\nBEGIN\nPERFORM create_queryable_indexes();\nRETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER queryables_trigger AFTER INSERT OR UPDATE ON queryables\nFOR EACH STATEMENT EXECUTE PROCEDURE queryables_trigger_func();\n\nCREATE TRIGGER queryables_collection_trigger AFTER INSERT OR UPDATE ON collections\nFOR EACH STATEMENT EXECUTE PROCEDURE queryables_trigger_func();\nCREATE OR REPLACE FUNCTION parse_dtrange(\n _indate jsonb,\n relative_base timestamptz DEFAULT date_trunc('hour', CURRENT_TIMESTAMP)\n) RETURNS tstzrange AS $$\nDECLARE\n timestrs text[];\n s timestamptz;\n e timestamptz;\nBEGIN\n timestrs :=\n CASE\n WHEN _indate ? 'timestamp' THEN\n ARRAY[_indate->>'timestamp']\n WHEN _indate ? 'interval' THEN\n to_text_array(_indate->'interval')\n WHEN jsonb_typeof(_indate) = 'array' THEN\n to_text_array(_indate)\n ELSE\n regexp_split_to_array(\n _indate->>0,\n '/'\n )\n END;\n RAISE NOTICE 'TIMESTRS %', timestrs;\n IF cardinality(timestrs) = 1 THEN\n IF timestrs[1] ILIKE 'P%' THEN\n RETURN tstzrange(relative_base - upper(timestrs[1])::interval, relative_base, '[)');\n END IF;\n s := timestrs[1]::timestamptz;\n RETURN tstzrange(s, s, '[]');\n END IF;\n\n IF cardinality(timestrs) != 2 THEN\n RAISE EXCEPTION 'Timestamp cannot have more than 2 values';\n END IF;\n\n IF timestrs[1] = '..' THEN\n s := '-infinity'::timestamptz;\n e := timestrs[2]::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] = '..' THEN\n s := timestrs[1]::timestamptz;\n e := 'infinity'::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[1] ILIKE 'P%' AND timestrs[2] NOT ILIKE 'P%' THEN\n e := timestrs[2]::timestamptz;\n s := e - upper(timestrs[1])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] ILIKE 'P%' AND timestrs[1] NOT ILIKE 'P%' THEN\n s := timestrs[1]::timestamptz;\n e := s + upper(timestrs[2])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n s := timestrs[1]::timestamptz;\n e := timestrs[2]::timestamptz;\n\n RETURN tstzrange(s,e,'[)');\n\n RETURN NULL;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT PARALLEL SAFE SET TIME ZONE 'UTC';\n\nCREATE OR REPLACE FUNCTION parse_dtrange(\n _indate text,\n relative_base timestamptz DEFAULT CURRENT_TIMESTAMP\n) RETURNS tstzrange AS $$\n SELECT parse_dtrange(to_jsonb(_indate), relative_base);\n$$ LANGUAGE SQL STABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION temporal_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\n ll text := 'datetime';\n lh text := 'end_datetime';\n rrange tstzrange;\n rl text;\n rh text;\n outq text;\nBEGIN\n rrange := parse_dtrange(args->1);\n RAISE NOTICE 'Constructing temporal query OP: %, ARGS: %, RRANGE: %', op, args, rrange;\n op := lower(op);\n rl := format('%L::timestamptz', lower(rrange));\n rh := format('%L::timestamptz', upper(rrange));\n outq := CASE op\n WHEN 't_before' THEN 'lh < rl'\n WHEN 't_after' THEN 'll > rh'\n WHEN 't_meets' THEN 'lh = rl'\n WHEN 't_metby' THEN 'll = rh'\n WHEN 't_overlaps' THEN 'll < rl AND rl < lh < rh'\n WHEN 't_overlappedby' THEN 'rl < ll < rh AND lh > rh'\n WHEN 't_starts' THEN 'll = rl AND lh < rh'\n WHEN 't_startedby' THEN 'll = rl AND lh > rh'\n WHEN 't_during' THEN 'll > rl AND lh < rh'\n WHEN 't_contains' THEN 'll < rl AND lh > rh'\n WHEN 't_finishes' THEN 'll > rl AND lh = rh'\n WHEN 't_finishedby' THEN 'll < rl AND lh = rh'\n WHEN 't_equals' THEN 'll = rl AND lh = rh'\n WHEN 't_disjoint' THEN 'NOT (ll <= rh AND lh >= rl)'\n WHEN 't_intersects' THEN 'll <= rh AND lh >= rl'\n WHEN 'anyinteracts' THEN 'll <= rh AND lh >= rl'\n END;\n outq := regexp_replace(outq, '\\mll\\M', ll);\n outq := regexp_replace(outq, '\\mlh\\M', lh);\n outq := regexp_replace(outq, '\\mrl\\M', rl);\n outq := regexp_replace(outq, '\\mrh\\M', rh);\n outq := format('(%s)', outq);\n RETURN outq;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\n\n\nCREATE OR REPLACE FUNCTION spatial_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\n geom text;\n j jsonb := args->1;\nBEGIN\n op := lower(op);\n RAISE NOTICE 'Constructing spatial query OP: %, ARGS: %', op, args;\n IF op NOT IN ('s_equals','s_disjoint','s_touches','s_within','s_overlaps','s_crosses','s_intersects','intersects','s_contains') THEN\n RAISE EXCEPTION 'Spatial Operator % Not Supported', op;\n END IF;\n op := regexp_replace(op, '^s_', 'st_');\n IF op = 'intersects' THEN\n op := 'st_intersects';\n END IF;\n -- Convert geometry to WKB string\n IF j ? 'type' AND j ? 'coordinates' THEN\n geom := st_geomfromgeojson(j)::text;\n ELSIF jsonb_typeof(j) = 'array' THEN\n geom := bbox_geom(j)::text;\n END IF;\n\n RETURN format('%s(geometry, %L::geometry)', op, geom);\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION query_to_cql2(q jsonb) RETURNS jsonb AS $$\n-- Translates anything passed in through the deprecated \"query\" into equivalent CQL2\nWITH t AS (\n SELECT key as property, value as ops\n FROM jsonb_each(q)\n), t2 AS (\n SELECT property, (jsonb_each(ops)).*\n FROM t WHERE jsonb_typeof(ops) = 'object'\n UNION ALL\n SELECT property, 'eq', ops\n FROM t WHERE jsonb_typeof(ops) != 'object'\n)\nSELECT\n jsonb_strip_nulls(jsonb_build_object(\n 'op', 'and',\n 'args', jsonb_agg(\n jsonb_build_object(\n 'op', key,\n 'args', jsonb_build_array(\n jsonb_build_object('property',property),\n value\n )\n )\n )\n )\n) as qcql FROM t2\n;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\nCREATE OR REPLACE FUNCTION cql1_to_cql2(j jsonb) RETURNS jsonb AS $$\nDECLARE\n args jsonb;\n ret jsonb;\nBEGIN\n RAISE NOTICE 'CQL1_TO_CQL2: %', j;\n IF j ? 'filter' THEN\n RETURN cql1_to_cql2(j->'filter');\n END IF;\n IF j ? 'property' THEN\n RETURN j;\n END IF;\n IF jsonb_typeof(j) = 'array' THEN\n SELECT jsonb_agg(cql1_to_cql2(el)) INTO args FROM jsonb_array_elements(j) el;\n RETURN args;\n END IF;\n IF jsonb_typeof(j) = 'number' THEN\n RETURN j;\n END IF;\n IF jsonb_typeof(j) = 'string' THEN\n RETURN j;\n END IF;\n\n IF jsonb_typeof(j) = 'object' THEN\n SELECT jsonb_build_object(\n 'op', key,\n 'args', cql1_to_cql2(value)\n ) INTO ret\n FROM jsonb_each(j)\n WHERE j IS NOT NULL;\n RETURN ret;\n END IF;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE STRICT;\n\nCREATE TABLE cql2_ops (\n op text PRIMARY KEY,\n template text,\n types text[]\n);\nINSERT INTO cql2_ops (op, template, types) VALUES\n ('eq', '%s = %s', NULL),\n ('lt', '%s < %s', NULL),\n ('lte', '%s <= %s', NULL),\n ('gt', '%s > %s', NULL),\n ('gte', '%s >= %s', NULL),\n ('le', '%s <= %s', NULL),\n ('ge', '%s >= %s', NULL),\n ('=', '%s = %s', NULL),\n ('<', '%s < %s', NULL),\n ('<=', '%s <= %s', NULL),\n ('>', '%s > %s', NULL),\n ('>=', '%s >= %s', NULL),\n ('like', '%s LIKE %s', NULL),\n ('ilike', '%s ILIKE %s', NULL),\n ('+', '%s + %s', NULL),\n ('-', '%s - %s', NULL),\n ('*', '%s * %s', NULL),\n ('/', '%s / %s', NULL),\n ('in', '%s = ANY (%s)', NULL),\n ('not', 'NOT (%s)', NULL),\n ('between', '%s BETWEEN (%2$s)[1] AND (%2$s)[2]', NULL),\n ('isnull', '%s IS NULL', NULL)\nON CONFLICT (op) DO UPDATE\n SET\n template = EXCLUDED.template;\n;\n\n\nCREATE OR REPLACE FUNCTION cql2_query(j jsonb, wrapper text DEFAULT NULL) RETURNS text AS $$\n#variable_conflict use_variable\nDECLARE\n args jsonb := j->'args';\n arg jsonb;\n op text := lower(j->>'op');\n cql2op RECORD;\n literal text;\nBEGIN\n IF j IS NULL OR (op IS NOT NULL AND args IS NULL) THEN\n RETURN NULL;\n END IF;\n RAISE NOTICE 'CQL2_QUERY: %', j;\n IF j ? 'filter' THEN\n RETURN cql2_query(j->'filter');\n END IF;\n\n IF j ? 'upper' THEN\n RETURN format('upper(%s)', cql2_query(j->'upper'));\n END IF;\n\n IF j ? 'lower' THEN\n RETURN format('lower(%s)', cql2_query(j->'lower'));\n END IF;\n\n -- Temporal Query\n IF op ilike 't_%' or op = 'anyinteracts' THEN\n RETURN temporal_op_query(op, args);\n END IF;\n\n -- If property is a timestamp convert it to text to use with\n -- general operators\n IF j ? 'timestamp' THEN\n RETURN format('%L::timestamptz', to_tstz(j->'timestamp'));\n END IF;\n IF j ? 'interval' THEN\n RAISE EXCEPTION 'Please use temporal operators when using intervals.';\n RETURN NONE;\n END IF;\n\n -- Spatial Query\n IF op ilike 's_%' or op = 'intersects' THEN\n RETURN spatial_op_query(op, args);\n END IF;\n\n\n IF op = 'in' THEN\n RETURN format(\n '%s = ANY (%L)',\n cql2_query(args->0),\n to_text_array(args->1)\n );\n END IF;\n\n\n\n IF op = 'between' THEN\n SELECT (queryable(a->>'property')).wrapper INTO wrapper\n FROM jsonb_array_elements(args) a\n WHERE a ? 'property' LIMIT 1;\n\n RETURN format(\n '%s BETWEEN %s and %s',\n cql2_query(args->0, wrapper),\n cql2_query(args->1->0, wrapper),\n cql2_query(args->1->1, wrapper)\n );\n END IF;\n\n -- Make sure that args is an array and run cql2_query on\n -- each element of the array\n RAISE NOTICE 'ARGS PRE: %', args;\n IF j ? 'args' THEN\n IF jsonb_typeof(args) != 'array' THEN\n args := jsonb_build_array(args);\n END IF;\n\n SELECT (queryable(a->>'property')).wrapper INTO wrapper\n FROM jsonb_array_elements(args) a\n WHERE a ? 'property' LIMIT 1;\n\n SELECT jsonb_agg(cql2_query(a, wrapper))\n INTO args\n FROM jsonb_array_elements(args) a;\n END IF;\n RAISE NOTICE 'ARGS: %', args;\n\n IF op IN ('and', 'or') THEN\n RETURN\n format(\n '(%s)',\n array_to_string(to_text_array(args), format(' %s ', upper(op)))\n );\n END IF;\n\n -- Look up template from cql2_ops\n IF j ? 'op' THEN\n SELECT * INTO cql2op FROM cql2_ops WHERE cql2_ops.op ilike op;\n IF FOUND THEN\n -- If specific index set in queryables for a property cast other arguments to that type\n RETURN format(\n cql2op.template,\n VARIADIC (to_text_array(args))\n );\n ELSE\n RAISE EXCEPTION 'Operator % Not Supported.', op;\n END IF;\n END IF;\n\n\n IF j ? 'property' THEN\n RETURN (queryable(j->>'property')).expression;\n END IF;\n\n IF wrapper IS NOT NULL THEN\n EXECUTE format('SELECT %I(%L)', wrapper, j) INTO literal;\n RAISE NOTICE '% % %',wrapper, j, literal;\n RETURN format('%I(%L)', wrapper, j);\n END IF;\n\n RETURN quote_literal(to_text(j));\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION paging_dtrange(\n j jsonb\n) RETURNS tstzrange AS $$\nDECLARE\n op text;\n filter jsonb := j->'filter';\n dtrange tstzrange := tstzrange('-infinity'::timestamptz,'infinity'::timestamptz);\n sdate timestamptz := '-infinity'::timestamptz;\n edate timestamptz := 'infinity'::timestamptz;\n jpitem jsonb;\nBEGIN\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n END IF;\n IF NOT (filter @? '$.**.op ? (@ == \"or\" || @ == \"not\")') THEN\n FOR jpitem IN SELECT j FROM jsonb_path_query(filter,'strict $.** ? (@.args[*].property == \"datetime\")'::jsonpath) j LOOP\n op := lower(jpitem->>'op');\n dtrange := parse_dtrange(jpitem->'args'->1);\n IF op IN ('<=', 'lt', 'lte', '<', 'le', 't_before') THEN\n sdate := greatest(sdate,'-infinity');\n edate := least(edate, upper(dtrange));\n ELSIF op IN ('>=', '>', 'gt', 'gte', 'ge', 't_after') THEN\n edate := least(edate, 'infinity');\n sdate := greatest(sdate, lower(dtrange));\n ELSIF op IN ('=', 'eq') THEN\n edate := least(edate, upper(dtrange));\n sdate := greatest(sdate, lower(dtrange));\n END IF;\n RAISE NOTICE '2 OP: %, ARGS: %, DTRANGE: %, SDATE: %, EDATE: %', op, jpitem->'args'->1, dtrange, sdate, edate;\n END LOOP;\n END IF;\n IF sdate > edate THEN\n RETURN 'empty'::tstzrange;\n END IF;\n RETURN tstzrange(sdate,edate, '[]');\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT SET TIME ZONE 'UTC';\n\nCREATE OR REPLACE FUNCTION paging_collections(\n IN j jsonb\n) RETURNS text[] AS $$\nDECLARE\n filter jsonb := j->'filter';\n jpitem jsonb;\n op text;\n args jsonb;\n arg jsonb;\n collections text[];\nBEGIN\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n END IF;\n IF NOT (filter @? '$.**.op ? (@ == \"or\" || @ == \"not\")') THEN\n FOR jpitem IN SELECT j FROM jsonb_path_query(filter,'strict $.** ? (@.args[*].property == \"collection\")'::jsonpath) j LOOP\n RAISE NOTICE 'JPITEM: %', jpitem;\n op := jpitem->>'op';\n args := jpitem->'args';\n IF op IN ('=', 'eq', 'in') THEN\n FOR arg IN SELECT a FROM jsonb_array_elements(args) a LOOP\n IF jsonb_typeof(arg) IN ('string', 'array') THEN\n RAISE NOTICE 'arg: %, collections: %', arg, collections;\n IF collections IS NULL OR collections = '{}'::text[] THEN\n collections := to_text_array(arg);\n ELSE\n collections := array_intersection(collections, to_text_array(arg));\n END IF;\n END IF;\n END LOOP;\n END IF;\n END LOOP;\n END IF;\n IF collections = '{}'::text[] THEN\n RETURN NULL;\n END IF;\n RETURN collections;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\nCREATE TABLE items (\n id text NOT NULL,\n geometry geometry NOT NULL,\n collection text NOT NULL,\n datetime timestamptz NOT NULL,\n end_datetime timestamptz NOT NULL,\n content JSONB NOT NULL\n)\nPARTITION BY LIST (collection)\n;\n\nCREATE INDEX \"datetime_idx\" ON items USING BTREE (datetime DESC, end_datetime ASC);\nCREATE INDEX \"geometry_idx\" ON items USING GIST (geometry);\n\nCREATE STATISTICS datetime_stats (dependencies) on datetime, end_datetime from items;\n\n\nALTER TABLE items ADD CONSTRAINT items_collections_fk FOREIGN KEY (collection) REFERENCES collections(id) ON DELETE CASCADE DEFERRABLE;\n\n\nCREATE OR REPLACE FUNCTION content_slim(_item jsonb, _collection jsonb) RETURNS jsonb AS $$\n SELECT\n jsonb_object_agg(\n key,\n CASE\n WHEN\n jsonb_typeof(c.value) = 'object'\n AND\n jsonb_typeof(i.value) = 'object'\n THEN content_slim(i.value, c.value)\n ELSE i.value\n END\n )\n FROM\n jsonb_each(_item) as i\n LEFT JOIN\n jsonb_each(_collection) as c\n USING (key)\n WHERE\n i.value IS DISTINCT FROM c.value\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_slim(_item jsonb) RETURNS jsonb AS $$\n SELECT content_slim(_item - '{id,type,collection,geometry,bbox}'::text[], collection_base_item(_item->>'collection'));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_dehydrate(content jsonb) RETURNS items AS $$\n SELECT\n content->>'id' as id,\n stac_geom(content) as geometry,\n content->>'collection' as collection,\n stac_datetime(content) as datetime,\n stac_end_datetime(content) as end_datetime,\n content_slim(content) as content\n ;\n$$ LANGUAGE SQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION include_field(f text, fields jsonb DEFAULT '{}'::jsonb) RETURNS boolean AS $$\nDECLARE\n includes jsonb := coalesce(fields->'includes', fields->'include', '[]'::jsonb);\n excludes jsonb := coalesce(fields->'excludes', fields->'exclude', '[]'::jsonb);\nBEGIN\n IF f IS NULL THEN\n RETURN NULL;\n ELSIF jsonb_array_length(includes)>0 AND includes ? f THEN\n RETURN TRUE;\n ELSIF jsonb_array_length(excludes)>0 AND excludes ? f THEN\n RETURN FALSE;\n ELSIF jsonb_array_length(includes)>0 AND NOT includes ? f THEN\n RETURN FALSE;\n END IF;\n RETURN TRUE;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION key_filter(IN k text, IN val jsonb, INOUT kf jsonb, OUT include boolean) AS $$\nDECLARE\n includes jsonb := coalesce(kf->'includes', kf->'include', '[]'::jsonb);\n excludes jsonb := coalesce(kf->'excludes', kf->'exclude', '[]'::jsonb);\nBEGIN\n RAISE NOTICE '% % %', k, val, kf;\n\n include := TRUE;\n IF k = 'properties' AND NOT excludes ? 'properties' THEN\n excludes := excludes || '[\"properties\"]';\n include := TRUE;\n RAISE NOTICE 'Prop include %', include;\n ELSIF\n jsonb_array_length(excludes)>0 AND excludes ? k THEN\n include := FALSE;\n ELSIF\n jsonb_array_length(includes)>0 AND NOT includes ? k THEN\n include := FALSE;\n ELSIF\n jsonb_array_length(includes)>0 AND includes ? k THEN\n includes := '[]'::jsonb;\n RAISE NOTICE 'KF: %', kf;\n END IF;\n kf := jsonb_build_object('includes', includes, 'excludes', excludes);\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION strip_assets(a jsonb) RETURNS jsonb AS $$\n WITH t AS (SELECT * FROM jsonb_each(a))\n SELECT jsonb_object_agg(key, value) FROM t\n WHERE value ? 'href';\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION content_hydrate(\n _item jsonb,\n _collection jsonb,\n fields jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\n SELECT\n jsonb_strip_nulls(jsonb_object_agg(\n key,\n CASE\n WHEN key = 'properties' AND include_field('properties', fields) THEN\n i.value\n WHEN key = 'properties' THEN\n content_hydrate(i.value, c.value, kf)\n WHEN\n c.value IS NULL AND key != 'properties'\n THEN i.value\n WHEN\n key = 'assets'\n AND\n jsonb_typeof(c.value) = 'object'\n AND\n jsonb_typeof(i.value) = 'object'\n THEN strip_assets(content_hydrate(i.value, c.value, kf))\n WHEN\n jsonb_typeof(c.value) = 'object'\n AND\n jsonb_typeof(i.value) = 'object'\n THEN content_hydrate(i.value, c.value, kf)\n ELSE coalesce(i.value, c.value)\n END\n ))\n FROM\n jsonb_each(coalesce(_item,'{}'::jsonb)) as i\n FULL JOIN\n jsonb_each(coalesce(_collection,'{}'::jsonb)) as c\n USING (key)\n JOIN LATERAL (\n SELECT kf, include FROM key_filter(key, i.value, fields)\n ) as k ON (include)\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION content_hydrate(_item items, _collection collections, fields jsonb DEFAULT '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\n content jsonb;\n base_item jsonb := _collection.base_item;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry)::jsonb;\n END IF;\n IF include_field('bbox', fields) THEN\n bbox := geom_bbox(_item.geometry)::jsonb;\n END IF;\n output := content_hydrate(\n jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'bbox',bbox,\n 'collection', _item.collection\n ) || _item.content,\n _collection.base_item,\n fields\n );\n\n RETURN output;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_hydrate(_item items, fields jsonb DEFAULT '{}'::jsonb) RETURNS jsonb AS $$\n SELECT content_hydrate(\n _item,\n (SELECT c FROM collections c WHERE id=_item.collection LIMIT 1),\n fields\n );\n$$ LANGUAGE SQL STABLE;\n\n\nCREATE UNLOGGED TABLE items_staging (\n content JSONB NOT NULL\n);\nCREATE UNLOGGED TABLE items_staging_ignore (\n content JSONB NOT NULL\n);\nCREATE UNLOGGED TABLE items_staging_upsert (\n content JSONB NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION items_staging_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p record;\n _partitions text[];\n ts timestamptz := clock_timestamp();\nBEGIN\n RAISE NOTICE 'Creating Partitions. %', clock_timestamp() - ts;\n WITH ranges AS (\n SELECT\n n.content->>'collection' as collection,\n stac_daterange(n.content->'properties') as dtr\n FROM newdata n\n ), p AS (\n SELECT\n collection,\n lower(dtr) as datetime,\n upper(dtr) as end_datetime,\n (partition_name(\n collection,\n lower(dtr)\n )).partition_name as name\n FROM ranges\n )\n INSERT INTO partitions (collection, datetime_range, end_datetime_range)\n SELECT\n collection,\n tstzrange(min(datetime), max(datetime), '[]') as datetime_range,\n tstzrange(min(end_datetime), max(end_datetime), '[]') as end_datetime_range\n FROM p\n GROUP BY collection, name\n ON CONFLICT (name) DO UPDATE SET\n datetime_range = EXCLUDED.datetime_range,\n end_datetime_range = EXCLUDED.end_datetime_range\n ;\n\n RAISE NOTICE 'Doing the insert. %', clock_timestamp() - ts;\n IF TG_TABLE_NAME = 'items_staging' THEN\n INSERT INTO items\n SELECT\n (content_dehydrate(content)).*\n FROM newdata;\n DELETE FROM items_staging;\n ELSIF TG_TABLE_NAME = 'items_staging_ignore' THEN\n INSERT INTO items\n SELECT\n (content_dehydrate(content)).*\n FROM newdata\n ON CONFLICT DO NOTHING;\n DELETE FROM items_staging_ignore;\n ELSIF TG_TABLE_NAME = 'items_staging_upsert' THEN\n WITH staging_formatted AS (\n SELECT (content_dehydrate(content)).* FROM newdata\n ), deletes AS (\n DELETE FROM items i USING staging_formatted s\n WHERE\n i.id = s.id\n AND i.collection = s.collection\n AND i IS DISTINCT FROM s\n RETURNING i.id, i.collection\n )\n INSERT INTO items\n SELECT s.* FROM\n staging_formatted s\n JOIN deletes d\n USING (id, collection);\n DELETE FROM items_staging_upsert;\n END IF;\n RAISE NOTICE 'Done. %', clock_timestamp() - ts;\n\n RETURN NULL;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER items_staging_insert_trigger AFTER INSERT ON items_staging REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\nCREATE TRIGGER items_staging_insert_ignore_trigger AFTER INSERT ON items_staging_ignore REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\nCREATE TRIGGER items_staging_insert_upsert_trigger AFTER INSERT ON items_staging_upsert REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\n\n\n\nCREATE OR REPLACE FUNCTION item_by_id(_id text, _collection text DEFAULT NULL) RETURNS items AS\n$$\nDECLARE\n i items%ROWTYPE;\nBEGIN\n SELECT * INTO i FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection) LIMIT 1;\n RETURN i;\nEND;\n$$ LANGUAGE PLPGSQL STABLE SECURITY DEFINER SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION get_item(_id text, _collection text DEFAULT NULL) RETURNS jsonb AS $$\n SELECT content_hydrate(items) FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection);\n$$ LANGUAGE SQL STABLE SECURITY DEFINER SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION delete_item(_id text, _collection text DEFAULT NULL) RETURNS VOID AS $$\nDECLARE\nout items%ROWTYPE;\nBEGIN\n DELETE FROM items WHERE id = _id AND (_collection IS NULL OR collection=_collection) RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n--/*\nCREATE OR REPLACE FUNCTION create_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION update_item(content jsonb) RETURNS VOID AS $$\nDECLARE\n old items %ROWTYPE;\n out items%ROWTYPE;\nBEGIN\n SELECT delete_item(content->>'id', content->>'collection');\n SELECT create_item(content);\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION create_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION collection_bbox(id text) RETURNS jsonb AS $$\n SELECT (replace(replace(replace(st_extent(geometry)::text,'BOX(','[['),')',']]'),' ',','))::jsonb\n FROM items WHERE collection=$1;\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION collection_temporal_extent(id text) RETURNS jsonb AS $$\n SELECT to_jsonb(array[array[min(datetime)::text, max(datetime)::text]])\n FROM items WHERE collection=$1;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection_extents() RETURNS VOID AS $$\nUPDATE collections SET\n content = content ||\n jsonb_build_object(\n 'extent', jsonb_build_object(\n 'spatial', jsonb_build_object(\n 'bbox', collection_bbox(collections.id)\n ),\n 'temporal', jsonb_build_object(\n 'interval', collection_temporal_extent(collections.id)\n )\n )\n )\n;\n$$ LANGUAGE SQL;\nCREATE VIEW partition_steps AS\nSELECT\n name,\n date_trunc('month',lower(datetime_range)) as sdate,\n date_trunc('month', upper(datetime_range)) + '1 month'::interval as edate\n FROM partitions WHERE datetime_range IS NOT NULL AND datetime_range != 'empty'::tstzrange\n ORDER BY datetime_range ASC\n;\n\nCREATE OR REPLACE FUNCTION chunker(\n IN _where text,\n OUT s timestamptz,\n OUT e timestamptz\n) RETURNS SETOF RECORD AS $$\nDECLARE\n explain jsonb;\nBEGIN\n IF _where IS NULL THEN\n _where := ' TRUE ';\n END IF;\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s;', _where)\n INTO explain;\n\n RETURN QUERY\n WITH t AS (\n SELECT j->>0 as p FROM\n jsonb_path_query(\n explain,\n 'strict $.**.\"Relation Name\" ? (@ != null)'\n ) j\n ),\n parts AS (\n SELECT sdate, edate FROM t JOIN partition_steps ON (t.p = name)\n ),\n times AS (\n SELECT sdate FROM parts\n UNION\n SELECT edate FROM parts\n ),\n uniq AS (\n SELECT DISTINCT sdate FROM times ORDER BY sdate\n ),\n last AS (\n SELECT sdate, lead(sdate, 1) over () as edate FROM uniq\n )\n SELECT sdate, edate FROM last WHERE edate IS NOT NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION partition_queries(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN partitions text[] DEFAULT NULL\n) RETURNS SETOF text AS $$\nDECLARE\n query text;\n sdate timestamptz;\n edate timestamptz;\nBEGIN\nIF _where IS NULL OR trim(_where) = '' THEN\n _where = ' TRUE ';\nEND IF;\nRAISE NOTICE 'Getting chunks for % %', _where, _orderby;\nIF _orderby ILIKE 'datetime d%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 DESC LOOP\n RETURN NEXT format($q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n $q$,\n sdate,\n edate,\n _where,\n _orderby\n );\n END LOOP;\nELSIF _orderby ILIKE 'datetime a%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 ASC LOOP\n RETURN NEXT format($q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n $q$,\n sdate,\n edate,\n _where,\n _orderby\n );\n END LOOP;\nELSE\n query := format($q$\n SELECT * FROM items\n WHERE %s\n ORDER BY %s\n $q$, _where, _orderby\n );\n\n RETURN NEXT query;\n RETURN;\nEND IF;\n\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION partition_query_view(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN _limit int DEFAULT 10\n) RETURNS text AS $$\n WITH p AS (\n SELECT * FROM partition_queries(_where, _orderby) p\n )\n SELECT\n CASE WHEN EXISTS (SELECT 1 FROM p) THEN\n (SELECT format($q$\n SELECT * FROM (\n %s\n ) total LIMIT %s\n $q$,\n string_agg(\n format($q$ SELECT * FROM ( %s ) AS sub $q$, p),\n '\n UNION ALL\n '\n ),\n _limit\n ))\n ELSE NULL\n END FROM p;\n$$ LANGUAGE SQL IMMUTABLE;\n\n\n\n\nCREATE OR REPLACE FUNCTION stac_search_to_where(j jsonb) RETURNS text AS $$\nDECLARE\n where_segments text[];\n _where text;\n dtrange tstzrange;\n collections text[];\n geom geometry;\n sdate timestamptz;\n edate timestamptz;\n filterlang text;\n filter jsonb := j->'filter';\nBEGIN\n IF j ? 'ids' THEN\n where_segments := where_segments || format('id = ANY (%L) ', to_text_array(j->'ids'));\n END IF;\n\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n where_segments := where_segments || format('collection = ANY (%L) ', collections);\n END IF;\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n\n where_segments := where_segments || format(' datetime <= %L::timestamptz AND end_datetime >= %L::timestamptz ',\n edate,\n sdate\n );\n END IF;\n\n geom := stac_geom(j);\n IF geom IS NOT NULL THEN\n where_segments := where_segments || format('st_intersects(geometry, %L)',geom);\n END IF;\n\n filterlang := COALESCE(\n j->>'filter-lang',\n get_setting('default-filter-lang', j->'conf')\n );\n IF NOT filter @? '$.**.op' THEN\n filterlang := 'cql-json';\n END IF;\n\n IF filterlang NOT IN ('cql-json','cql2-json') AND j ? 'filter' THEN\n RAISE EXCEPTION '% is not a supported filter-lang. Please use cql-json or cql2-json.', filterlang;\n END IF;\n\n IF j ? 'query' AND j ? 'filter' THEN\n RAISE EXCEPTION 'Can only use either query or filter at one time.';\n END IF;\n\n IF j ? 'query' THEN\n filter := query_to_cql2(j->'query');\n ELSIF filterlang = 'cql-json' THEN\n filter := cql1_to_cql2(filter);\n END IF;\n RAISE NOTICE 'FILTER: %', filter;\n where_segments := where_segments || cql2_query(filter);\n IF cardinality(where_segments) < 1 THEN\n RETURN ' TRUE ';\n END IF;\n\n _where := array_to_string(array_remove(where_segments, NULL), ' AND ');\n\n IF _where IS NULL OR BTRIM(_where) = '' THEN\n RETURN ' TRUE ';\n END IF;\n RETURN _where;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION parse_sort_dir(_dir text, reverse boolean default false) RETURNS text AS $$\n WITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n ) SELECT\n CASE\n WHEN NOT reverse THEN d\n WHEN d = 'ASC' THEN 'DESC'\n WHEN d = 'DESC' THEN 'ASC'\n END\n FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION sort_dir_to_op(_dir text, prev boolean default false) RETURNS text AS $$\n WITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n ) SELECT\n CASE\n WHEN d = 'ASC' AND prev THEN '<='\n WHEN d = 'DESC' AND prev THEN '>='\n WHEN d = 'ASC' THEN '>='\n WHEN d = 'DESC' THEN '<='\n END\n FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION sort_sqlorderby(\n _search jsonb DEFAULT NULL,\n reverse boolean DEFAULT FALSE\n) RETURNS text AS $$\n WITH sortby AS (\n SELECT coalesce(_search->'sortby','[{\"field\":\"datetime\", \"direction\":\"desc\"}]') as sort\n ), withid AS (\n SELECT CASE\n WHEN sort @? '$[*] ? (@.field == \"id\")' THEN sort\n ELSE sort || '[{\"field\":\"id\", \"direction\":\"desc\"}]'::jsonb\n END as sort\n FROM sortby\n ), withid_rows AS (\n SELECT jsonb_array_elements(sort) as value FROM withid\n ),sorts AS (\n SELECT\n coalesce(\n -- field_orderby((items_path(value->>'field')).path_txt),\n (queryable(value->>'field')).expression\n ) as key,\n parse_sort_dir(value->>'direction', reverse) as dir\n FROM withid_rows\n )\n SELECT array_to_string(\n array_agg(concat(key, ' ', dir)),\n ', '\n ) FROM sorts;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_sort_dir(sort_item jsonb) RETURNS text AS $$\n SELECT CASE WHEN sort_item->>'direction' ILIKE 'desc%' THEN 'DESC' ELSE 'ASC' END;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION get_token_filter(_search jsonb = '{}'::jsonb, token_rec jsonb DEFAULT NULL) RETURNS text AS $$\nDECLARE\n token_id text;\n filters text[] := '{}'::text[];\n prev boolean := TRUE;\n field text;\n dir text;\n sort record;\n orfilters text[] := '{}'::text[];\n andfilters text[] := '{}'::text[];\n output text;\n token_where text;\nBEGIN\n RAISE NOTICE 'Getting Token Filter. % %', _search, token_rec;\n -- If no token provided return NULL\n IF token_rec IS NULL THEN\n IF NOT (_search ? 'token' AND\n (\n (_search->>'token' ILIKE 'prev:%')\n OR\n (_search->>'token' ILIKE 'next:%')\n )\n ) THEN\n RETURN NULL;\n END IF;\n prev := (_search->>'token' ILIKE 'prev:%');\n token_id := substr(_search->>'token', 6);\n SELECT to_jsonb(items) INTO token_rec\n FROM items WHERE id=token_id;\n END IF;\n RAISE NOTICE 'TOKEN ID: % %', token_rec, token_rec->'id';\n\n CREATE TEMP TABLE sorts (\n _row int GENERATED ALWAYS AS IDENTITY NOT NULL,\n _field text PRIMARY KEY,\n _dir text NOT NULL,\n _val text\n ) ON COMMIT DROP;\n\n -- Make sure we only have distinct columns to sort with taking the first one we get\n INSERT INTO sorts (_field, _dir)\n SELECT\n (queryable(value->>'field')).expression,\n get_sort_dir(value)\n FROM\n jsonb_array_elements(coalesce(_search->'sortby','[{\"field\":\"datetime\",\"direction\":\"desc\"}]'))\n ON CONFLICT DO NOTHING\n ;\n RAISE NOTICE 'sorts 1: %', (SELECT jsonb_agg(to_json(sorts)) FROM sorts);\n -- Get the first sort direction provided. As the id is a primary key, if there are any\n -- sorts after id they won't do anything, so make sure that id is the last sort item.\n SELECT _dir INTO dir FROM sorts ORDER BY _row ASC LIMIT 1;\n IF EXISTS (SELECT 1 FROM sorts WHERE _field = 'id') THEN\n DELETE FROM sorts WHERE _row > (SELECT _row FROM sorts WHERE _field = 'id' ORDER BY _row ASC);\n ELSE\n INSERT INTO sorts (_field, _dir) VALUES ('id', dir);\n END IF;\n\n -- Add value from looked up item to the sorts table\n UPDATE sorts SET _val=quote_literal(token_rec->>_field);\n\n -- Check if all sorts are the same direction and use row comparison\n -- to filter\n RAISE NOTICE 'sorts 2: %', (SELECT jsonb_agg(to_json(sorts)) FROM sorts);\n\n IF (SELECT count(DISTINCT _dir) FROM sorts) = 1 THEN\n SELECT format(\n '(%s) %s (%s)',\n concat_ws(', ', VARIADIC array_agg(quote_ident(_field))),\n CASE WHEN (prev AND dir = 'ASC') OR (NOT prev AND dir = 'DESC') THEN '<' ELSE '>' END,\n concat_ws(', ', VARIADIC array_agg(_val))\n ) INTO output FROM sorts\n WHERE token_rec ? _field\n ;\n ELSE\n FOR sort IN SELECT * FROM sorts ORDER BY _row asc LOOP\n RAISE NOTICE 'SORT: %', sort;\n IF sort._row = 1 THEN\n orfilters := orfilters || format('(%s %s %s)',\n quote_ident(sort._field),\n CASE WHEN (prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC') THEN '<' ELSE '>' END,\n sort._val\n );\n ELSE\n orfilters := orfilters || format('(%s AND %s %s %s)',\n array_to_string(andfilters, ' AND '),\n quote_ident(sort._field),\n CASE WHEN (prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC') THEN '<' ELSE '>' END,\n sort._val\n );\n\n END IF;\n andfilters := andfilters || format('%s = %s',\n quote_ident(sort._field),\n sort._val\n );\n END LOOP;\n output := array_to_string(orfilters, ' OR ');\n END IF;\n DROP TABLE IF EXISTS sorts;\n token_where := concat('(',coalesce(output,'true'),')');\n IF trim(token_where) = '' THEN\n token_where := NULL;\n END IF;\n RAISE NOTICE 'TOKEN_WHERE: |%|',token_where;\n RETURN token_where;\n END;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION search_tohash(jsonb) RETURNS jsonb AS $$\n SELECT $1 - '{token,limit,context,includes,excludes}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION search_hash(jsonb, jsonb) RETURNS text AS $$\n SELECT md5(concat(search_tohash($1)::text,$2::text));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TABLE IF NOT EXISTS searches(\n hash text GENERATED ALWAYS AS (search_hash(search, metadata)) STORED PRIMARY KEY,\n search jsonb NOT NULL,\n _where text,\n orderby text,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n metadata jsonb DEFAULT '{}'::jsonb NOT NULL\n);\nCREATE TABLE IF NOT EXISTS search_wheres(\n id bigint generated always as identity primary key,\n _where text NOT NULL,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n statslastupdated timestamptz,\n estimated_count bigint,\n estimated_cost float,\n time_to_estimate float,\n total_count bigint,\n time_to_count float,\n partitions text[]\n);\n\nCREATE INDEX IF NOT EXISTS search_wheres_partitions ON search_wheres USING GIN (partitions);\nCREATE UNIQUE INDEX IF NOT EXISTS search_wheres_where ON search_wheres ((md5(_where)));\n\nCREATE OR REPLACE FUNCTION where_stats(inwhere text, updatestats boolean default false, conf jsonb default null) RETURNS search_wheres AS $$\nDECLARE\n t timestamptz;\n i interval;\n explain_json jsonb;\n partitions text[];\n sw search_wheres%ROWTYPE;\n inwhere_hash text := md5(inwhere);\n _context text := lower(context(conf));\n _stats_ttl interval := context_stats_ttl(conf);\n _estimated_cost float := context_estimated_cost(conf);\n _estimated_count int := context_estimated_count(conf);\nBEGIN\n IF _context = 'off' THEN\n sw._where := inwhere;\n return sw;\n END IF;\n\n SELECT * INTO sw FROM search_wheres WHERE md5(_where)=inwhere_hash FOR UPDATE;\n\n -- Update statistics if explicitly set, if statistics do not exist, or statistics ttl has expired\n IF NOT updatestats THEN\n RAISE NOTICE 'Checking if update is needed for: % .', inwhere;\n RAISE NOTICE 'Stats Last Updated: %', sw.statslastupdated;\n RAISE NOTICE 'TTL: %, Age: %', _stats_ttl, now() - sw.statslastupdated;\n RAISE NOTICE 'Context: %, Existing Total: %', _context, sw.total_count;\n IF\n sw.statslastupdated IS NULL\n OR (now() - sw.statslastupdated) > _stats_ttl\n OR (context(conf) != 'off' AND sw.total_count IS NULL)\n THEN\n updatestats := TRUE;\n END IF;\n END IF;\n\n sw._where := inwhere;\n sw.lastused := now();\n sw.usecount := coalesce(sw.usecount,0) + 1;\n\n IF NOT updatestats THEN\n UPDATE search_wheres SET\n lastused = sw.lastused,\n usecount = sw.usecount\n WHERE md5(_where) = inwhere_hash\n RETURNING * INTO sw\n ;\n RETURN sw;\n END IF;\n\n -- Use explain to get estimated count/cost and a list of the partitions that would be hit by the query\n t := clock_timestamp();\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s', inwhere)\n INTO explain_json;\n RAISE NOTICE 'Time for just the explain: %', clock_timestamp() - t;\n i := clock_timestamp() - t;\n\n sw.statslastupdated := now();\n sw.estimated_count := explain_json->0->'Plan'->'Plan Rows';\n sw.estimated_cost := explain_json->0->'Plan'->'Total Cost';\n sw.time_to_estimate := extract(epoch from i);\n\n RAISE NOTICE 'ESTIMATED_COUNT: % < %', sw.estimated_count, _estimated_count;\n RAISE NOTICE 'ESTIMATED_COST: % < %', sw.estimated_cost, _estimated_cost;\n\n -- Do a full count of rows if context is set to on or if auto is set and estimates are low enough\n IF\n _context = 'on'\n OR\n ( _context = 'auto' AND\n (\n sw.estimated_count < _estimated_count\n AND\n sw.estimated_cost < _estimated_cost\n )\n )\n THEN\n t := clock_timestamp();\n RAISE NOTICE 'Calculating actual count...';\n EXECUTE format(\n 'SELECT count(*) FROM items WHERE %s',\n inwhere\n ) INTO sw.total_count;\n i := clock_timestamp() - t;\n RAISE NOTICE 'Actual Count: % -- %', sw.total_count, i;\n sw.time_to_count := extract(epoch FROM i);\n ELSE\n sw.total_count := NULL;\n sw.time_to_count := NULL;\n END IF;\n\n\n INSERT INTO search_wheres\n (_where, lastused, usecount, statslastupdated, estimated_count, estimated_cost, time_to_estimate, partitions, total_count, time_to_count)\n SELECT sw._where, sw.lastused, sw.usecount, sw.statslastupdated, sw.estimated_count, sw.estimated_cost, sw.time_to_estimate, sw.partitions, sw.total_count, sw.time_to_count\n ON CONFLICT ((md5(_where)))\n DO UPDATE\n SET\n lastused = sw.lastused,\n usecount = sw.usecount,\n statslastupdated = sw.statslastupdated,\n estimated_count = sw.estimated_count,\n estimated_cost = sw.estimated_cost,\n time_to_estimate = sw.time_to_estimate,\n total_count = sw.total_count,\n time_to_count = sw.time_to_count\n ;\n RETURN sw;\nEND;\n$$ LANGUAGE PLPGSQL ;\n\n\n\nDROP FUNCTION IF EXISTS search_query;\nCREATE OR REPLACE FUNCTION search_query(\n _search jsonb = '{}'::jsonb,\n updatestats boolean = false,\n _metadata jsonb = '{}'::jsonb\n) RETURNS searches AS $$\nDECLARE\n search searches%ROWTYPE;\n pexplain jsonb;\n t timestamptz;\n i interval;\nBEGIN\n SELECT * INTO search FROM searches\n WHERE hash=search_hash(_search, _metadata) FOR UPDATE;\n\n -- Calculate the where clause if not already calculated\n IF search._where IS NULL THEN\n search._where := stac_search_to_where(_search);\n END IF;\n\n -- Calculate the order by clause if not already calculated\n IF search.orderby IS NULL THEN\n search.orderby := sort_sqlorderby(_search);\n END IF;\n\n PERFORM where_stats(search._where, updatestats, _search->'conf');\n\n search.lastused := now();\n search.usecount := coalesce(search.usecount, 0) + 1;\n INSERT INTO searches (search, _where, orderby, lastused, usecount, metadata)\n VALUES (_search, search._where, search.orderby, search.lastused, search.usecount, _metadata)\n ON CONFLICT (hash) DO\n UPDATE SET\n _where = EXCLUDED._where,\n orderby = EXCLUDED.orderby,\n lastused = EXCLUDED.lastused,\n usecount = EXCLUDED.usecount,\n metadata = EXCLUDED.metadata\n RETURNING * INTO search\n ;\n RETURN search;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION search(_search jsonb = '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n searches searches%ROWTYPE;\n _where text;\n token_where text;\n full_where text;\n orderby text;\n query text;\n token_type text := substr(_search->>'token',1,4);\n _limit int := coalesce((_search->>'limit')::int, 10);\n curs refcursor;\n cntr int := 0;\n iter_record items%ROWTYPE;\n first_record jsonb;\n last_record jsonb;\n out_records jsonb := '[]'::jsonb;\n prev_query text;\n next text;\n prev_id text;\n has_next boolean := false;\n has_prev boolean := false;\n prev text;\n total_count bigint;\n context jsonb;\n collection jsonb;\n includes text[];\n excludes text[];\n exit_flag boolean := FALSE;\n batches int := 0;\n timer timestamptz := clock_timestamp();\n pstart timestamptz;\n pend timestamptz;\n pcurs refcursor;\n search_where search_wheres%ROWTYPE;\n id text;\nBEGIN\nCREATE TEMP TABLE results (content jsonb) ON COMMIT DROP;\n-- if ids is set, short circuit and just use direct ids query for each id\n-- skip any paging or caching\n-- hard codes ordering in the same order as the array of ids\nIF _search ? 'ids' THEN\n INSERT INTO results\n SELECT content_hydrate(items, _search->'fields')\n FROM items WHERE\n items.id = ANY(to_text_array(_search->'ids'))\n AND\n CASE WHEN _search ? 'collections' THEN\n items.collection = ANY(to_text_array(_search->'collections'))\n ELSE TRUE\n END\n ORDER BY items.datetime desc, items.id desc\n ;\n SELECT INTO total_count count(*) FROM results;\nELSE\n searches := search_query(_search);\n _where := searches._where;\n orderby := searches.orderby;\n search_where := where_stats(_where);\n total_count := coalesce(search_where.total_count, search_where.estimated_count);\n\n IF token_type='prev' THEN\n token_where := get_token_filter(_search, null::jsonb);\n orderby := sort_sqlorderby(_search, TRUE);\n END IF;\n IF token_type='next' THEN\n token_where := get_token_filter(_search, null::jsonb);\n END IF;\n\n full_where := concat_ws(' AND ', _where, token_where);\n RAISE NOTICE 'FULL QUERY % %', full_where, clock_timestamp()-timer;\n timer := clock_timestamp();\n\n FOR query IN SELECT partition_queries(full_where, orderby, search_where.partitions) LOOP\n timer := clock_timestamp();\n query := format('%s LIMIT %s', query, _limit + 1);\n RAISE NOTICE 'Partition Query: %', query;\n batches := batches + 1;\n -- curs = create_cursor(query);\n OPEN curs FOR EXECUTE query;\n LOOP\n FETCH curs into iter_record;\n EXIT WHEN NOT FOUND;\n cntr := cntr + 1;\n last_record := content_hydrate(iter_record, _search->'fields');\n IF cntr = 1 THEN\n first_record := last_record;\n END IF;\n IF cntr <= _limit THEN\n INSERT INTO results (content) VALUES (last_record);\n ELSIF cntr > _limit THEN\n has_next := true;\n exit_flag := true;\n EXIT;\n END IF;\n END LOOP;\n CLOSE curs;\n RAISE NOTICE 'Query took %.', clock_timestamp()-timer;\n timer := clock_timestamp();\n EXIT WHEN exit_flag;\n END LOOP;\n RAISE NOTICE 'Scanned through % partitions.', batches;\nEND IF;\n\nSELECT jsonb_agg(content) INTO out_records FROM results WHERE content is not NULL;\n\nDROP TABLE results;\n\n\n-- Flip things around if this was the result of a prev token query\nIF token_type='prev' THEN\n out_records := flip_jsonb_array(out_records);\n first_record := last_record;\nEND IF;\n\n-- If this query has a token, see if there is data before the first record\nIF _search ? 'token' THEN\n prev_query := format(\n 'SELECT 1 FROM items WHERE %s LIMIT 1',\n concat_ws(\n ' AND ',\n _where,\n trim(get_token_filter(_search, to_jsonb(content_dehydrate(first_record))))\n )\n );\n RAISE NOTICE 'Query to get previous record: % --- %', prev_query, first_record;\n EXECUTE prev_query INTO has_prev;\n IF FOUND and has_prev IS NOT NULL THEN\n RAISE NOTICE 'Query results from prev query: %', has_prev;\n has_prev := TRUE;\n END IF;\nEND IF;\nhas_prev := COALESCE(has_prev, FALSE);\n\nIF has_prev THEN\n prev := out_records->0->>'id';\nEND IF;\nIF has_next OR token_type='prev' THEN\n next := out_records->-1->>'id';\nEND IF;\n\nIF context(_search->'conf') != 'off' THEN\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'matched', total_count,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\nELSE\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\nEND IF;\n\ncollection := jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb),\n 'next', next,\n 'prev', prev,\n 'context', context\n);\n\nRETURN collection;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER SET SEARCH_PATH TO pgstac, public;\n\n\nCREATE OR REPLACE FUNCTION search_cursor(_search jsonb = '{}'::jsonb) RETURNS refcursor AS $$\nDECLARE\n curs refcursor;\n searches searches%ROWTYPE;\n _where text;\n _orderby text;\n q text;\n\nBEGIN\n searches := search_query(_search);\n _where := searches._where;\n _orderby := searches.orderby;\n\n OPEN curs FOR\n WITH p AS (\n SELECT * FROM partition_queries(_where, _orderby) p\n )\n SELECT\n CASE WHEN EXISTS (SELECT 1 FROM p) THEN\n (SELECT format($q$\n SELECT * FROM (\n %s\n ) total\n $q$,\n string_agg(\n format($q$ SELECT * FROM ( %s ) AS sub $q$, p),\n '\n UNION ALL\n '\n )\n ))\n ELSE NULL\n END FROM p;\n RETURN curs;\nEND;\n$$ LANGUAGE PLPGSQL;\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION tileenvelope(zoom int, x int, y int) RETURNS geometry AS $$\nWITH t AS (\n SELECT\n 20037508.3427892 as merc_max,\n -20037508.3427892 as merc_min,\n (2 * 20037508.3427892) / (2 ^ zoom) as tile_size\n)\nSELECT st_makeenvelope(\n merc_min + (tile_size * x),\n merc_max - (tile_size * (y + 1)),\n merc_min + (tile_size * (x + 1)),\n merc_max - (tile_size * y),\n 3857\n) FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;DROP FUNCTION IF EXISTS mercgrid;\n\n\nCREATE OR REPLACE FUNCTION ftime() RETURNS interval as $$\nSELECT age(clock_timestamp(), transaction_timestamp());\n$$ LANGUAGE SQL;\nSET SEARCH_PATH to pgstac, public;\n\nDROP FUNCTION IF EXISTS geometrysearch;\nCREATE OR REPLACE FUNCTION geometrysearch(\n IN geom geometry,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE, -- Return as soon as the passed in geometry is full covered\n IN skipcovered boolean DEFAULT TRUE -- Skip any items that would show up completely under the previous items\n) RETURNS jsonb AS $$\nDECLARE\n search searches%ROWTYPE;\n curs refcursor;\n _where text;\n query text;\n iter_record items%ROWTYPE;\n out_records jsonb := '{}'::jsonb[];\n exit_flag boolean := FALSE;\n counter int := 1;\n scancounter int := 1;\n remaining_limit int := _scanlimit;\n tilearea float;\n unionedgeom geometry;\n clippedgeom geometry;\n unionedgeom_area float := 0;\n prev_area float := 0;\n excludes text[];\n includes text[];\n\nBEGIN\n DROP TABLE IF EXISTS pgstac_results;\n CREATE TEMP TABLE pgstac_results (content jsonb) ON COMMIT DROP;\n\n -- If skipcovered is true then you will always want to exit when the passed in geometry is full\n IF skipcovered THEN\n exitwhenfull := TRUE;\n END IF;\n\n SELECT * INTO search FROM searches WHERE hash=queryhash;\n\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Search with Query Hash % Not Found', queryhash;\n END IF;\n\n tilearea := st_area(geom);\n _where := format('%s AND st_intersects(geometry, %L::geometry)', search._where, geom);\n\n\n FOR query IN SELECT * FROM partition_queries(_where, search.orderby) LOOP\n query := format('%s LIMIT %L', query, remaining_limit);\n RAISE NOTICE '%', query;\n OPEN curs FOR EXECUTE query;\n LOOP\n FETCH curs INTO iter_record;\n EXIT WHEN NOT FOUND;\n IF exitwhenfull OR skipcovered THEN -- If we are not using exitwhenfull or skipcovered, we do not need to do expensive geometry operations\n clippedgeom := st_intersection(geom, iter_record.geometry);\n\n IF unionedgeom IS NULL THEN\n unionedgeom := clippedgeom;\n ELSE\n unionedgeom := st_union(unionedgeom, clippedgeom);\n END IF;\n\n unionedgeom_area := st_area(unionedgeom);\n\n IF skipcovered AND prev_area = unionedgeom_area THEN\n scancounter := scancounter + 1;\n CONTINUE;\n END IF;\n\n prev_area := unionedgeom_area;\n\n RAISE NOTICE '% % % %', unionedgeom_area/tilearea, counter, scancounter, ftime();\n END IF;\n RAISE NOTICE '% %', iter_record, content_hydrate(iter_record, fields);\n INSERT INTO pgstac_results (content) VALUES (content_hydrate(iter_record, fields));\n\n IF counter >= _limit\n OR scancounter > _scanlimit\n OR ftime() > _timelimit\n OR (exitwhenfull AND unionedgeom_area >= tilearea)\n THEN\n exit_flag := TRUE;\n EXIT;\n END IF;\n counter := counter + 1;\n scancounter := scancounter + 1;\n\n END LOOP;\n CLOSE curs;\n EXIT WHEN exit_flag;\n remaining_limit := _scanlimit - scancounter;\n END LOOP;\n\n SELECT jsonb_agg(content) INTO out_records FROM pgstac_results WHERE content IS NOT NULL;\n\n RETURN jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb)\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDROP FUNCTION IF EXISTS geojsonsearch;\nCREATE OR REPLACE FUNCTION geojsonsearch(\n IN geojson jsonb,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_geomfromgeojson(geojson),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS xyzsearch;\nCREATE OR REPLACE FUNCTION xyzsearch(\n IN _x int,\n IN _y int,\n IN _z int,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_transform(tileenvelope(_z, _x, _y), 4326),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nGRANT ALL ON SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac to pgstac_admin;\n\n-- pgstac_read role limited to using function apis\nGRANT EXECUTE ON FUNCTION search TO pgstac_read;\nGRANT EXECUTE ON FUNCTION search_query TO pgstac_read;\nGRANT EXECUTE ON FUNCTION item_by_id TO pgstac_read;\nGRANT EXECUTE ON FUNCTION get_item TO pgstac_read;\n\nGRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_ingest;\nGRANT USAGE ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_ingest;\nSELECT set_version('0.5.0');\n" }, { "path": "src/pgstac/migrations/pgstac.0.5.1-0.6.0.sql", "content": "SET SEARCH_PATH to pgstac, public;\nalter table \"pgstac\".\"partitions\" drop constraint \"partitions_collection_fkey\";\n\ndrop function if exists \"pgstac\".\"content_hydrate\"(_item jsonb, _collection jsonb, fields jsonb);\n\ndrop function if exists \"pgstac\".\"content_slim\"(_item jsonb, _collection jsonb);\n\ndrop function if exists \"pgstac\".\"key_filter\"(k text, val jsonb, INOUT kf jsonb, OUT include boolean);\n\ndrop function if exists \"pgstac\".\"strip_assets\"(a jsonb);\n\nalter table \"pgstac\".\"partitions\" add constraint \"partitions_collection_fkey\" FOREIGN KEY (collection) REFERENCES pgstac.collections(id) ON DELETE CASCADE;\n\nset check_function_bodies = off;\n\nCREATE OR REPLACE FUNCTION pgstac.content_hydrate(_base_item jsonb, _item jsonb, fields jsonb DEFAULT '{}'::jsonb)\n RETURNS jsonb\n LANGUAGE sql\n IMMUTABLE PARALLEL SAFE\nAS $function$\n SELECT merge_jsonb(\n jsonb_fields(_item, fields),\n jsonb_fields(_base_item, fields)\n );\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.explode_dotpaths(j jsonb)\n RETURNS SETOF text[]\n LANGUAGE sql\n IMMUTABLE PARALLEL SAFE\nAS $function$\n SELECT string_to_array(p, '.') as e FROM jsonb_array_elements_text(j) p;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.explode_dotpaths_recurse(j jsonb)\n RETURNS SETOF text[]\n LANGUAGE sql\n IMMUTABLE PARALLEL SAFE\nAS $function$\n WITH RECURSIVE t AS (\n SELECT e FROM explode_dotpaths(j) e\n UNION ALL\n SELECT e[1:cardinality(e)-1]\n FROM t\n WHERE cardinality(e)>1\n ) SELECT e FROM t;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.jsonb_exclude(j jsonb, f jsonb)\n RETURNS jsonb\n LANGUAGE plpgsql\n IMMUTABLE\nAS $function$\nDECLARE\n excludes jsonb := f-> 'exclude';\n outj jsonb := j;\n path text[];\nBEGIN\n IF\n excludes IS NULL\n OR jsonb_array_length(excludes) = 0\n THEN\n RETURN j;\n ELSE\n FOR path IN SELECT explode_dotpaths(excludes) LOOP\n outj := outj #- path;\n END LOOP;\n END IF;\n RETURN outj;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.jsonb_fields(j jsonb, f jsonb DEFAULT '{\"fields\": []}'::jsonb)\n RETURNS jsonb\n LANGUAGE sql\n IMMUTABLE\nAS $function$\n SELECT jsonb_exclude(jsonb_include(j, f), f);\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.jsonb_include(j jsonb, f jsonb)\n RETURNS jsonb\n LANGUAGE plpgsql\n IMMUTABLE\nAS $function$\nDECLARE\n includes jsonb := f-> 'include';\n outj jsonb := '{}'::jsonb;\n path text[];\nBEGIN\n IF\n includes IS NULL\n OR jsonb_array_length(includes) = 0\n THEN\n RETURN j;\n ELSE\n includes := includes || '[\"id\",\"collection\"]'::jsonb;\n FOR path IN SELECT explode_dotpaths(includes) LOOP\n outj := jsonb_set_nested(outj, path, j #> path);\n END LOOP;\n END IF;\n RETURN outj;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.jsonb_set_nested(j jsonb, path text[], val jsonb)\n RETURNS jsonb\n LANGUAGE plpgsql\n IMMUTABLE\nAS $function$\nDECLARE\nBEGIN\n IF cardinality(path) > 1 THEN\n FOR i IN 1..(cardinality(path)-1) LOOP\n IF j #> path[:i] IS NULL THEN\n j := jsonb_set_lax(j, path[:i], '{}', TRUE);\n END IF;\n END LOOP;\n END IF;\n RETURN jsonb_set_lax(j, path, val, true);\n\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.merge_jsonb(_a jsonb, _b jsonb)\n RETURNS jsonb\n LANGUAGE sql\n IMMUTABLE\nAS $function$\n SELECT\n CASE\n WHEN _a = '\"𒍟※\"'::jsonb THEN NULL\n WHEN _a IS NULL OR jsonb_typeof(_a) = 'null' THEN _b\n WHEN jsonb_typeof(_a) = 'object' AND jsonb_typeof(_b) = 'object' THEN\n (\n SELECT\n jsonb_strip_nulls(\n jsonb_object_agg(\n key,\n merge_jsonb(a.value, b.value)\n )\n )\n FROM\n jsonb_each(coalesce(_a,'{}'::jsonb)) as a\n FULL JOIN\n jsonb_each(coalesce(_b,'{}'::jsonb)) as b\n USING (key)\n )\n WHEN\n jsonb_typeof(_a) = 'array'\n AND jsonb_typeof(_b) = 'array'\n AND jsonb_array_length(_a) = jsonb_array_length(_b)\n THEN\n (\n SELECT jsonb_agg(m) FROM\n ( SELECT\n merge_jsonb(\n jsonb_array_elements(_a),\n jsonb_array_elements(_b)\n ) as m\n ) as l\n )\n ELSE _a\n END\n ;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.partitions_delete_trigger_func()\n RETURNS trigger\n LANGUAGE plpgsql\nAS $function$\nDECLARE\n q text;\nBEGIN\n RAISE NOTICE 'Partition Delete Trigger. %', OLD.name;\n EXECUTE format($q$\n DROP TABLE IF EXISTS %I CASCADE;\n $q$,\n OLD.name\n );\n RAISE NOTICE 'Dropped partition.';\n RETURN OLD;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.strip_jsonb(_a jsonb, _b jsonb)\n RETURNS jsonb\n LANGUAGE sql\n IMMUTABLE\nAS $function$\n SELECT\n CASE\n\n WHEN (_a IS NULL OR jsonb_typeof(_a) = 'null') AND _b IS NOT NULL AND jsonb_typeof(_b) != 'null' THEN '\"𒍟※\"'::jsonb\n WHEN _b IS NULL OR jsonb_typeof(_a) = 'null' THEN _a\n WHEN _a = _b AND jsonb_typeof(_a) = 'object' THEN '{}'::jsonb\n WHEN _a = _b THEN NULL\n WHEN jsonb_typeof(_a) = 'object' AND jsonb_typeof(_b) = 'object' THEN\n (\n SELECT\n jsonb_strip_nulls(\n jsonb_object_agg(\n key,\n strip_jsonb(a.value, b.value)\n )\n )\n FROM\n jsonb_each(_a) as a\n FULL JOIN\n jsonb_each(_b) as b\n USING (key)\n )\n WHEN\n jsonb_typeof(_a) = 'array'\n AND jsonb_typeof(_b) = 'array'\n AND jsonb_array_length(_a) = jsonb_array_length(_b)\n THEN\n (\n SELECT jsonb_agg(m) FROM\n ( SELECT\n strip_jsonb(\n jsonb_array_elements(_a),\n jsonb_array_elements(_b)\n ) as m\n ) as l\n )\n ELSE _a\n END\n ;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.collection_base_item(content jsonb)\n RETURNS jsonb\n LANGUAGE sql\n IMMUTABLE PARALLEL SAFE\nAS $function$\n SELECT jsonb_build_object(\n 'type', 'Feature',\n 'stac_version', content->'stac_version',\n 'assets', content->'item_assets',\n 'collection', content->'id'\n );\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.collections_trigger_func()\n RETURNS trigger\n LANGUAGE plpgsql\n SET search_path TO 'pgstac', 'public'\nAS $function$\nDECLARE\n q text;\n partition_name text := format('_items_%s', NEW.key);\n partition_exists boolean := false;\n partition_empty boolean := true;\n err_context text;\n loadtemp boolean := FALSE;\nBEGIN\n RAISE NOTICE 'Collection Trigger. % %', NEW.id, NEW.key;\n SELECT relid::text INTO partition_name\n FROM pg_partition_tree('items')\n WHERE relid::text = partition_name;\n IF FOUND THEN\n partition_exists := true;\n partition_empty := table_empty(partition_name);\n ELSE\n partition_exists := false;\n partition_empty := true;\n partition_name := format('_items_%s', NEW.key);\n END IF;\n IF TG_OP = 'UPDATE' AND NEW.partition_trunc IS DISTINCT FROM OLD.partition_trunc AND partition_empty THEN\n q := format($q$\n DROP TABLE IF EXISTS %I CASCADE;\n $q$,\n partition_name\n );\n EXECUTE q;\n END IF;\n IF TG_OP = 'UPDATE' AND NEW.partition_trunc IS DISTINCT FROM OLD.partition_trunc AND partition_exists AND NOT partition_empty THEN\n q := format($q$\n CREATE TEMP TABLE changepartitionstaging ON COMMIT DROP AS SELECT * FROM %I;\n DROP TABLE IF EXISTS %I CASCADE;\n $q$,\n partition_name,\n partition_name\n );\n EXECUTE q;\n loadtemp := TRUE;\n partition_empty := TRUE;\n partition_exists := FALSE;\n END IF;\n IF TG_OP = 'UPDATE' AND NEW.partition_trunc IS NOT DISTINCT FROM OLD.partition_trunc THEN\n RETURN NEW;\n END IF;\n IF NEW.partition_trunc IS NULL AND partition_empty THEN\n RAISE NOTICE '% % % %',\n partition_name,\n NEW.id,\n concat(partition_name,'_id_idx'),\n partition_name\n ;\n q := format($q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n $q$,\n partition_name,\n NEW.id,\n concat(partition_name,'_id_idx'),\n partition_name\n );\n RAISE NOTICE 'q: %', q;\n BEGIN\n EXECUTE q;\n EXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', partition_name;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\n END;\n\n ALTER TABLE partitions DISABLE TRIGGER partitions_delete_trigger;\n DELETE FROM partitions WHERE collection=NEW.id AND name=partition_name;\n ALTER TABLE partitions ENABLE TRIGGER partitions_delete_trigger;\n\n INSERT INTO partitions (collection, name) VALUES (NEW.id, partition_name);\n ELSIF partition_empty THEN\n q := format($q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L)\n PARTITION BY RANGE (datetime);\n $q$,\n partition_name,\n NEW.id\n );\n RAISE NOTICE 'q: %', q;\n BEGIN\n EXECUTE q;\n EXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', partition_name;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\n END;\n ALTER TABLE partitions DISABLE TRIGGER partitions_delete_trigger;\n DELETE FROM partitions WHERE collection=NEW.id AND name=partition_name;\n ALTER TABLE partitions ENABLE TRIGGER partitions_delete_trigger;\n ELSE\n RAISE EXCEPTION 'Cannot modify partition % unless empty', partition_name;\n END IF;\n IF loadtemp THEN\n RAISE NOTICE 'Moving data into new partitions.';\n q := format($q$\n WITH p AS (\n SELECT\n collection,\n datetime as datetime,\n end_datetime as end_datetime,\n (partition_name(\n collection,\n datetime\n )).partition_name as name\n FROM changepartitionstaging\n )\n INSERT INTO partitions (collection, datetime_range, end_datetime_range)\n SELECT\n collection,\n tstzrange(min(datetime), max(datetime), '[]') as datetime_range,\n tstzrange(min(end_datetime), max(end_datetime), '[]') as end_datetime_range\n FROM p\n GROUP BY collection, name\n ON CONFLICT (name) DO UPDATE SET\n datetime_range = EXCLUDED.datetime_range,\n end_datetime_range = EXCLUDED.end_datetime_range\n ;\n INSERT INTO %I SELECT * FROM changepartitionstaging;\n DROP TABLE IF EXISTS changepartitionstaging;\n $q$,\n partition_name\n );\n EXECUTE q;\n END IF;\n RETURN NEW;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.content_hydrate(_item pgstac.items, _collection pgstac.collections, fields jsonb DEFAULT '{}'::jsonb)\n RETURNS jsonb\n LANGUAGE plpgsql\n STABLE PARALLEL SAFE\nAS $function$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\n content jsonb;\n base_item jsonb := _collection.base_item;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry, 20)::jsonb;\n END IF;\n output := content_hydrate(\n jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'collection', _item.collection,\n 'type', 'Feature'\n ) || _item.content,\n _collection.base_item,\n fields\n );\n\n RETURN output;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.content_nonhydrated(_item pgstac.items, fields jsonb DEFAULT '{}'::jsonb)\n RETURNS jsonb\n LANGUAGE plpgsql\n STABLE PARALLEL SAFE\nAS $function$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry, 20)::jsonb;\n END IF;\n output := jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'collection', _item.collection,\n 'type', 'Feature'\n ) || _item.content;\n RETURN output;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.content_slim(_item jsonb)\n RETURNS jsonb\n LANGUAGE sql\n IMMUTABLE PARALLEL SAFE\nAS $function$\n SELECT strip_jsonb(_item - '{id,geometry,collection,type}'::text[], collection_base_item(_item->>'collection')) - '{id,geometry,collection,type}'::text[];\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.delete_item(_id text, _collection text DEFAULT NULL::text)\n RETURNS void\n LANGUAGE plpgsql\nAS $function$\nDECLARE\nout items%ROWTYPE;\nBEGIN\n DELETE FROM items WHERE id = _id AND (_collection IS NULL OR collection=_collection) RETURNING * INTO STRICT out;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.include_field(f text, fields jsonb DEFAULT '{}'::jsonb)\n RETURNS boolean\n LANGUAGE plpgsql\n IMMUTABLE\nAS $function$\nDECLARE\n includes jsonb := fields->'include';\n excludes jsonb := fields->'exclude';\nBEGIN\n IF f IS NULL THEN\n RETURN NULL;\n END IF;\n\n\n IF\n jsonb_typeof(excludes) = 'array'\n AND jsonb_array_length(excludes)>0\n AND excludes ? f\n THEN\n RETURN FALSE;\n END IF;\n\n IF\n (\n jsonb_typeof(includes) = 'array'\n AND jsonb_array_length(includes) > 0\n AND includes ? f\n ) OR\n (\n includes IS NULL\n OR jsonb_typeof(includes) = 'null'\n OR jsonb_array_length(includes) = 0\n )\n THEN\n RETURN TRUE;\n END IF;\n\n RETURN FALSE;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.partition_name(collection text, dt timestamp with time zone, OUT partition_name text, OUT partition_range tstzrange)\n RETURNS record\n LANGUAGE plpgsql\n STABLE\nAS $function$\nDECLARE\n c RECORD;\n parent_name text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n parent_name := format('_items_%s', c.key);\n\n\n IF c.partition_trunc = 'year' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYY'));\n ELSIF c.partition_trunc = 'month' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYYMM'));\n ELSE\n partition_name := parent_name;\n partition_range := tstzrange('-infinity'::timestamptz, 'infinity'::timestamptz, '[]');\n END IF;\n IF partition_range IS NULL THEN\n partition_range := tstzrange(\n date_trunc(c.partition_trunc::text, dt),\n date_trunc(c.partition_trunc::text, dt) + concat('1 ', c.partition_trunc)::interval\n );\n END IF;\n RETURN;\n\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.search(_search jsonb DEFAULT '{}'::jsonb)\n RETURNS jsonb\n LANGUAGE plpgsql\n SECURITY DEFINER\n SET search_path TO 'pgstac', 'public'\nAS $function$\nDECLARE\n searches searches%ROWTYPE;\n _where text;\n token_where text;\n full_where text;\n orderby text;\n query text;\n token_type text := substr(_search->>'token',1,4);\n _limit int := coalesce((_search->>'limit')::int, 10);\n curs refcursor;\n cntr int := 0;\n iter_record items%ROWTYPE;\n first_record jsonb;\n first_item items%ROWTYPE;\n last_item items%ROWTYPE;\n last_record jsonb;\n out_records jsonb := '[]'::jsonb;\n prev_query text;\n next text;\n prev_id text;\n has_next boolean := false;\n has_prev boolean := false;\n prev text;\n total_count bigint;\n context jsonb;\n collection jsonb;\n includes text[];\n excludes text[];\n exit_flag boolean := FALSE;\n batches int := 0;\n timer timestamptz := clock_timestamp();\n pstart timestamptz;\n pend timestamptz;\n pcurs refcursor;\n search_where search_wheres%ROWTYPE;\n id text;\nBEGIN\nCREATE TEMP TABLE results (content jsonb) ON COMMIT DROP;\n-- if ids is set, short circuit and just use direct ids query for each id\n-- skip any paging or caching\n-- hard codes ordering in the same order as the array of ids\nIF _search ? 'ids' THEN\n INSERT INTO results\n SELECT\n CASE WHEN _search->'conf'->>'nohydrate' IS NOT NULL AND (_search->'conf'->>'nohydrate')::boolean = true THEN\n content_nonhydrated(items, _search->'fields')\n ELSE\n content_hydrate(items, _search->'fields')\n END\n FROM items WHERE\n items.id = ANY(to_text_array(_search->'ids'))\n AND\n CASE WHEN _search ? 'collections' THEN\n items.collection = ANY(to_text_array(_search->'collections'))\n ELSE TRUE\n END\n ORDER BY items.datetime desc, items.id desc\n ;\n SELECT INTO total_count count(*) FROM results;\nELSE\n searches := search_query(_search);\n _where := searches._where;\n orderby := searches.orderby;\n search_where := where_stats(_where);\n total_count := coalesce(search_where.total_count, search_where.estimated_count);\n\n IF token_type='prev' THEN\n token_where := get_token_filter(_search, null::jsonb);\n orderby := sort_sqlorderby(_search, TRUE);\n END IF;\n IF token_type='next' THEN\n token_where := get_token_filter(_search, null::jsonb);\n END IF;\n\n full_where := concat_ws(' AND ', _where, token_where);\n RAISE NOTICE 'FULL QUERY % %', full_where, clock_timestamp()-timer;\n timer := clock_timestamp();\n\n FOR query IN SELECT partition_queries(full_where, orderby, search_where.partitions) LOOP\n timer := clock_timestamp();\n query := format('%s LIMIT %s', query, _limit + 1);\n RAISE NOTICE 'Partition Query: %', query;\n batches := batches + 1;\n -- curs = create_cursor(query);\n OPEN curs FOR EXECUTE query;\n LOOP\n FETCH curs into iter_record;\n EXIT WHEN NOT FOUND;\n cntr := cntr + 1;\n\n IF _search->'conf'->>'nohydrate' IS NOT NULL AND (_search->'conf'->>'nohydrate')::boolean = true THEN\n last_record := content_nonhydrated(iter_record, _search->'fields');\n ELSE\n last_record := content_hydrate(iter_record, _search->'fields');\n END IF;\n last_item := iter_record;\n IF cntr = 1 THEN\n first_item := last_item;\n first_record := last_record;\n END IF;\n IF cntr <= _limit THEN\n INSERT INTO results (content) VALUES (last_record);\n ELSIF cntr > _limit THEN\n has_next := true;\n exit_flag := true;\n EXIT;\n END IF;\n END LOOP;\n CLOSE curs;\n RAISE NOTICE 'Query took %.', clock_timestamp()-timer;\n timer := clock_timestamp();\n EXIT WHEN exit_flag;\n END LOOP;\n RAISE NOTICE 'Scanned through % partitions.', batches;\nEND IF;\n\nSELECT jsonb_agg(content) INTO out_records FROM results WHERE content is not NULL;\n\nDROP TABLE results;\n\n\n-- Flip things around if this was the result of a prev token query\nIF token_type='prev' THEN\n out_records := flip_jsonb_array(out_records);\n first_record := last_record;\nEND IF;\n\n-- If this query has a token, see if there is data before the first record\nIF _search ? 'token' THEN\n prev_query := format(\n 'SELECT 1 FROM items WHERE %s LIMIT 1',\n concat_ws(\n ' AND ',\n _where,\n trim(get_token_filter(_search, to_jsonb(first_item)))\n )\n );\n RAISE NOTICE 'Query to get previous record: % --- %', prev_query, first_record;\n EXECUTE prev_query INTO has_prev;\n IF FOUND and has_prev IS NOT NULL THEN\n RAISE NOTICE 'Query results from prev query: %', has_prev;\n has_prev := TRUE;\n END IF;\nEND IF;\nhas_prev := COALESCE(has_prev, FALSE);\n\nIF has_prev THEN\n prev := out_records->0->>'id';\nEND IF;\nIF has_next OR token_type='prev' THEN\n next := out_records->-1->>'id';\nEND IF;\n\nIF context(_search->'conf') != 'off' THEN\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'matched', total_count,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\nELSE\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\nEND IF;\n\ncollection := jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb),\n 'next', next,\n 'prev', prev,\n 'context', context\n);\n\nRETURN collection;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.stac_daterange(value jsonb)\n RETURNS tstzrange\n LANGUAGE plpgsql\n IMMUTABLE PARALLEL SAFE\n SET \"TimeZone\" TO 'UTC'\nAS $function$\nDECLARE\n props jsonb := value;\n dt timestamptz;\n edt timestamptz;\nBEGIN\n IF props ? 'properties' THEN\n props := props->'properties';\n END IF;\n IF\n props ? 'start_datetime'\n AND props->>'start_datetime' IS NOT NULL\n AND props ? 'end_datetime'\n AND props->>'end_datetime' IS NOT NULL\n THEN\n dt := props->>'start_datetime';\n edt := props->>'end_datetime';\n IF dt > edt THEN\n RAISE EXCEPTION 'start_datetime must be < end_datetime';\n END IF;\n ELSE\n dt := props->>'datetime';\n edt := props->>'datetime';\n END IF;\n IF dt is NULL OR edt IS NULL THEN\n RAISE NOTICE 'DT: %, EDT: %', dt, edt;\n RAISE EXCEPTION 'Either datetime (%) or both start_datetime (%) and end_datetime (%) must be set.', props->>'datetime',props->>'start_datetime',props->>'end_datetime';\n END IF;\n RETURN tstzrange(dt, edt, '[]');\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.update_item(content jsonb)\n RETURNS void\n LANGUAGE plpgsql\n SET search_path TO 'pgstac', 'public'\nAS $function$\nDECLARE\n old items %ROWTYPE;\n out items%ROWTYPE;\nBEGIN\n PERFORM delete_item(content->>'id', content->>'collection');\n PERFORM create_item(content);\nEND;\n$function$\n;\n\nCREATE TRIGGER partitions_delete_trigger BEFORE DELETE ON pgstac.partitions FOR EACH ROW EXECUTE FUNCTION pgstac.partitions_delete_trigger_func();\n\n\n\nSELECT set_version('0.6.0');\n" }, { "path": "src/pgstac/migrations/pgstac.0.5.1.sql", "content": "CREATE EXTENSION IF NOT EXISTS postgis;\nCREATE EXTENSION IF NOT EXISTS btree_gist;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_admin;\n CREATE ROLE pgstac_read;\n CREATE ROLE pgstac_ingest;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nGRANT pgstac_admin TO current_user;\n\nCREATE SCHEMA IF NOT EXISTS pgstac AUTHORIZATION pgstac_admin;\n\nALTER ROLE pgstac_admin SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_read SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_ingest SET SEARCH_PATH TO pgstac, public;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\n\nGRANT pgstac_read TO pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\n\nSET ROLE pgstac_admin;\n\nSET SEARCH_PATH TO pgstac, public;\n\n\nCREATE TABLE IF NOT EXISTS migrations (\n version text PRIMARY KEY,\n datetime timestamptz DEFAULT clock_timestamp() NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION get_version() RETURNS text AS $$\n SELECT version FROM pgstac.migrations ORDER BY datetime DESC, version DESC LIMIT 1;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION set_version(text) RETURNS text AS $$\n INSERT INTO pgstac.migrations (version) VALUES ($1)\n ON CONFLICT DO NOTHING\n RETURNING version;\n$$ LANGUAGE SQL;\n\n\nCREATE TABLE IF NOT EXISTS pgstac_settings (\n name text PRIMARY KEY,\n value text NOT NULL\n);\n\nINSERT INTO pgstac_settings (name, value) VALUES\n ('context', 'off'),\n ('context_estimated_count', '100000'),\n ('context_estimated_cost', '100000'),\n ('context_stats_ttl', '1 day'),\n ('default-filter-lang', 'cql2-json'),\n ('additional_properties', 'true')\nON CONFLICT DO NOTHING\n;\n\n\nCREATE OR REPLACE FUNCTION get_setting(IN _setting text, IN conf jsonb DEFAULT NULL) RETURNS text AS $$\nSELECT COALESCE(\n conf->>_setting,\n current_setting(concat('pgstac.',_setting), TRUE),\n (SELECT value FROM pgstac.pgstac_settings WHERE name=_setting)\n);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context(conf jsonb DEFAULT NULL) RETURNS text AS $$\n SELECT pgstac.get_setting('context', conf);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context_estimated_count(conf jsonb DEFAULT NULL) RETURNS int AS $$\n SELECT pgstac.get_setting('context_estimated_count', conf)::int;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_estimated_cost();\nCREATE OR REPLACE FUNCTION context_estimated_cost(conf jsonb DEFAULT NULL) RETURNS float AS $$\n SELECT pgstac.get_setting('context_estimated_cost', conf)::float;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_stats_ttl();\nCREATE OR REPLACE FUNCTION context_stats_ttl(conf jsonb DEFAULT NULL) RETURNS interval AS $$\n SELECT pgstac.get_setting('context_stats_ttl', conf)::interval;\n$$ LANGUAGE SQL;\n\n\nCREATE OR REPLACE FUNCTION notice(VARIADIC text[]) RETURNS boolean AS $$\nDECLARE\ndebug boolean := current_setting('pgstac.debug', true);\nBEGIN\n IF debug THEN\n RAISE NOTICE 'NOTICE FROM FUNC: % >>>>> %', concat_ws(' | ', $1), clock_timestamp();\n RETURN TRUE;\n END IF;\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION empty_arr(ANYARRAY) RETURNS BOOLEAN AS $$\nSELECT CASE\n WHEN $1 IS NULL THEN TRUE\n WHEN cardinality($1)<1 THEN TRUE\nELSE FALSE\nEND;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_intersection(_a ANYARRAY, _b ANYARRAY) RETURNS ANYARRAY AS $$\n SELECT ARRAY ( SELECT unnest(_a) INTERSECT SELECT UNNEST(_b) );\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION array_map_ident(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_ident(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_map_literal(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_literal(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION array_reverse(anyarray) RETURNS anyarray AS $$\nSELECT ARRAY(\n SELECT $1[i]\n FROM generate_subscripts($1,1) AS s(i)\n ORDER BY i DESC\n);\n$$ LANGUAGE SQL STRICT IMMUTABLE;\nCREATE OR REPLACE FUNCTION to_int(jsonb) RETURNS int AS $$\n SELECT floor(($1->>0)::float)::int;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION to_float(jsonb) RETURNS float AS $$\n SELECT ($1->>0)::float;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION to_tstz(jsonb) RETURNS timestamptz AS $$\n SELECT ($1->>0)::timestamptz;\n$$ LANGUAGE SQL IMMUTABLE STRICT SET TIME ZONE 'UTC';\n\n\nCREATE OR REPLACE FUNCTION to_text(jsonb) RETURNS text AS $$\n SELECT $1->>0;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION to_text_array(jsonb) RETURNS text[] AS $$\n SELECT\n CASE jsonb_typeof($1)\n WHEN 'array' THEN ARRAY(SELECT jsonb_array_elements_text($1))\n ELSE ARRAY[$1->>0]\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\nCREATE OR REPLACE FUNCTION bbox_geom(_bbox jsonb) RETURNS geometry AS $$\nSELECT CASE jsonb_array_length(_bbox)\n WHEN 4 THEN\n ST_SetSRID(ST_MakeEnvelope(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float,\n (_bbox->>3)::float\n ),4326)\n WHEN 6 THEN\n ST_SetSRID(ST_3DMakeBox(\n ST_MakePoint(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float\n ),\n ST_MakePoint(\n (_bbox->>3)::float,\n (_bbox->>4)::float,\n (_bbox->>5)::float\n )\n ),4326)\n ELSE null END;\n;\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION geom_bbox(_geom geometry) RETURNS jsonb AS $$\n SELECT jsonb_build_array(\n st_xmin(_geom),\n st_ymin(_geom),\n st_xmax(_geom),\n st_ymax(_geom)\n );\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION flip_jsonb_array(j jsonb) RETURNS jsonb AS $$\n SELECT jsonb_agg(value) FROM (SELECT value FROM jsonb_array_elements(j) WITH ORDINALITY ORDER BY ordinality DESC) as t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n/* looks for a geometry in a stac item first from geometry and falling back to bbox */\nCREATE OR REPLACE FUNCTION stac_geom(value jsonb) RETURNS geometry AS $$\nSELECT\n CASE\n WHEN value ? 'intersects' THEN\n ST_GeomFromGeoJSON(value->>'intersects')\n WHEN value ? 'geometry' THEN\n ST_GeomFromGeoJSON(value->>'geometry')\n WHEN value ? 'bbox' THEN\n pgstac.bbox_geom(value->'bbox')\n ELSE NULL\n END as geometry\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION stac_daterange(\n value jsonb\n) RETURNS tstzrange AS $$\nDECLARE\n props jsonb := value;\n dt timestamptz;\n edt timestamptz;\nBEGIN\n IF props ? 'properties' THEN\n props := props->'properties';\n END IF;\n IF props ? 'start_datetime' AND props ? 'end_datetime' THEN\n dt := props->'start_datetime';\n edt := props->'end_datetime';\n IF dt > edt THEN\n RAISE EXCEPTION 'start_datetime must be < end_datetime';\n END IF;\n ELSE\n dt := props->'datetime';\n edt := props->'datetime';\n END IF;\n IF dt is NULL OR edt IS NULL THEN\n RAISE EXCEPTION 'Either datetime or both start_datetime and end_datetime must be set.';\n END IF;\n RETURN tstzrange(dt, edt, '[]');\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_datetime(value jsonb) RETURNS timestamptz AS $$\n SELECT lower(stac_daterange(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_end_datetime(value jsonb) RETURNS timestamptz AS $$\n SELECT upper(stac_daterange(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\n\nCREATE TABLE IF NOT EXISTS stac_extensions(\n name text PRIMARY KEY,\n url text,\n enbabled_by_default boolean NOT NULL DEFAULT TRUE,\n enableable boolean NOT NULL DEFAULT TRUE\n);\n\nINSERT INTO stac_extensions (name, url) VALUES\n ('fields', 'https://api.stacspec.org/v1.0.0-beta.5/item-search#fields'),\n ('sort','https://api.stacspec.org/v1.0.0-beta.5/item-search#sort'),\n ('context','https://api.stacspec.org/v1.0.0-beta.5/item-search#context'),\n ('filter', 'https://api.stacspec.org/v1.0.0-beta.5/item-search#filter'),\n ('query', 'https://api.stacspec.org/v1.0.0-beta.5/item-search#query')\nON CONFLICT (name) DO UPDATE SET url=EXCLUDED.url;\n\n\n\nCREATE OR REPLACE FUNCTION collection_base_item(content jsonb) RETURNS jsonb AS $$\n SELECT jsonb_build_object(\n 'type', 'Feature',\n 'stac_version', content->'stac_version',\n 'assets', content->'item_assets',\n 'collection', content->'id',\n 'links', '[]'::jsonb\n );\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TYPE partition_trunc_strategy AS ENUM ('year', 'month');\n\nCREATE TABLE IF NOT EXISTS collections (\n key bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,\n id text GENERATED ALWAYS AS (content->>'id') STORED UNIQUE,\n content JSONB NOT NULL,\n base_item jsonb GENERATED ALWAYS AS (pgstac.collection_base_item(content)) STORED,\n partition_trunc partition_trunc_strategy\n);\n\n\nCREATE OR REPLACE FUNCTION collection_base_item(cid text) RETURNS jsonb AS $$\n SELECT pgstac.collection_base_item(content) FROM pgstac.collections WHERE id = cid LIMIT 1;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION table_empty(text) RETURNS boolean AS $$\nDECLARE\n retval boolean;\nBEGIN\n EXECUTE format($q$\n SELECT NOT EXISTS (SELECT 1 FROM %I LIMIT 1)\n $q$,\n $1\n ) INTO retval;\n RETURN retval;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION collections_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n q text;\n partition_name text := format('_items_%s', NEW.key);\n partition_exists boolean := false;\n partition_empty boolean := true;\n err_context text;\n loadtemp boolean := FALSE;\nBEGIN\n RAISE NOTICE 'Collection Trigger. % %', NEW.id, NEW.key;\n SELECT relid::text INTO partition_name\n FROM pg_partition_tree('items')\n WHERE relid::text = partition_name;\n IF FOUND THEN\n partition_exists := true;\n partition_empty := table_empty(partition_name);\n ELSE\n partition_exists := false;\n partition_empty := true;\n partition_name := format('_items_%s', NEW.key);\n END IF;\n IF TG_OP = 'UPDATE' AND NEW.partition_trunc IS DISTINCT FROM OLD.partition_trunc AND partition_empty THEN\n q := format($q$\n DROP TABLE IF EXISTS %I CASCADE;\n $q$,\n partition_name\n );\n EXECUTE q;\n END IF;\n IF TG_OP = 'UPDATE' AND NEW.partition_trunc IS DISTINCT FROM OLD.partition_trunc AND partition_exists AND NOT partition_empty THEN\n q := format($q$\n CREATE TEMP TABLE changepartitionstaging ON COMMIT DROP AS SELECT * FROM %I;\n DROP TABLE IF EXISTS %I CASCADE;\n $q$,\n partition_name,\n partition_name\n );\n EXECUTE q;\n loadtemp := TRUE;\n partition_empty := TRUE;\n partition_exists := FALSE;\n END IF;\n IF TG_OP = 'UPDATE' AND NEW.partition_trunc IS NOT DISTINCT FROM OLD.partition_trunc THEN\n RETURN NEW;\n END IF;\n IF NEW.partition_trunc IS NULL AND partition_empty THEN\n RAISE NOTICE '% % % %',\n partition_name,\n NEW.id,\n concat(partition_name,'_id_idx'),\n partition_name\n ;\n q := format($q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n $q$,\n partition_name,\n NEW.id,\n concat(partition_name,'_id_idx'),\n partition_name\n );\n RAISE NOTICE 'q: %', q;\n BEGIN\n EXECUTE q;\n EXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', partition_name;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\n END;\n DELETE FROM partitions WHERE collection=NEW.id AND name=partition_name;\n INSERT INTO partitions (collection, name) VALUES (NEW.id, partition_name);\n ELSIF partition_empty THEN\n q := format($q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L)\n PARTITION BY RANGE (datetime);\n $q$,\n partition_name,\n NEW.id\n );\n RAISE NOTICE 'q: %', q;\n BEGIN\n EXECUTE q;\n EXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', partition_name;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\n END;\n DELETE FROM partitions WHERE collection=NEW.id AND name=partition_name;\n ELSE\n RAISE EXCEPTION 'Cannot modify partition % unless empty', partition_name;\n END IF;\n IF loadtemp THEN\n RAISE NOTICE 'Moving data into new partitions.';\n q := format($q$\n WITH p AS (\n SELECT\n collection,\n datetime as datetime,\n end_datetime as end_datetime,\n (partition_name(\n collection,\n datetime\n )).partition_name as name\n FROM changepartitionstaging\n )\n INSERT INTO partitions (collection, datetime_range, end_datetime_range)\n SELECT\n collection,\n tstzrange(min(datetime), max(datetime), '[]') as datetime_range,\n tstzrange(min(end_datetime), max(end_datetime), '[]') as end_datetime_range\n FROM p\n GROUP BY collection, name\n ON CONFLICT (name) DO UPDATE SET\n datetime_range = EXCLUDED.datetime_range,\n end_datetime_range = EXCLUDED.end_datetime_range\n ;\n INSERT INTO %I SELECT * FROM changepartitionstaging;\n DROP TABLE IF EXISTS changepartitionstaging;\n $q$,\n partition_name\n );\n EXECUTE q;\n END IF;\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE TRIGGER collections_trigger AFTER INSERT OR UPDATE ON collections FOR EACH ROW\nEXECUTE FUNCTION collections_trigger_func();\n\nCREATE OR REPLACE FUNCTION partition_collection(collection text, strategy partition_trunc_strategy) RETURNS text AS $$\n UPDATE collections SET partition_trunc=strategy WHERE id=collection RETURNING partition_trunc;\n$$ LANGUAGE SQL;\n\nCREATE TABLE IF NOT EXISTS partitions (\n collection text REFERENCES collections(id),\n name text PRIMARY KEY,\n partition_range tstzrange NOT NULL DEFAULT tstzrange('-infinity'::timestamptz,'infinity'::timestamptz, '[]'),\n datetime_range tstzrange,\n end_datetime_range tstzrange,\n CONSTRAINT prange EXCLUDE USING GIST (\n collection WITH =,\n partition_range WITH &&\n )\n) WITH (FILLFACTOR=90);\nCREATE INDEX partitions_range_idx ON partitions USING GIST(partition_range);\n\n\n\nCREATE OR REPLACE FUNCTION partition_name(\n IN collection text,\n IN dt timestamptz,\n OUT partition_name text,\n OUT partition_range tstzrange\n) AS $$\nDECLARE\n c RECORD;\n parent_name text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', collection;\n END IF;\n parent_name := format('_items_%s', c.key);\n\n\n IF c.partition_trunc = 'year' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYY'));\n ELSIF c.partition_trunc = 'month' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYYMM'));\n ELSE\n partition_name := parent_name;\n partition_range := tstzrange('-infinity'::timestamptz, 'infinity'::timestamptz, '[]');\n END IF;\n IF partition_range IS NULL THEN\n partition_range := tstzrange(\n date_trunc(c.partition_trunc::text, dt),\n date_trunc(c.partition_trunc::text, dt) + concat('1 ', c.partition_trunc)::interval\n );\n END IF;\n RETURN;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\n\nCREATE OR REPLACE FUNCTION partitions_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n q text;\n cq text;\n parent_name text;\n partition_trunc text;\n partition_name text := NEW.name;\n partition_exists boolean := false;\n partition_empty boolean := true;\n partition_range tstzrange;\n datetime_range tstzrange;\n end_datetime_range tstzrange;\n err_context text;\n mindt timestamptz := lower(NEW.datetime_range);\n maxdt timestamptz := upper(NEW.datetime_range);\n minedt timestamptz := lower(NEW.end_datetime_range);\n maxedt timestamptz := upper(NEW.end_datetime_range);\n t_mindt timestamptz;\n t_maxdt timestamptz;\n t_minedt timestamptz;\n t_maxedt timestamptz;\nBEGIN\n RAISE NOTICE 'Partitions Trigger. %', NEW;\n datetime_range := NEW.datetime_range;\n end_datetime_range := NEW.end_datetime_range;\n\n SELECT\n format('_items_%s', key),\n c.partition_trunc::text\n INTO\n parent_name,\n partition_trunc\n FROM pgstac.collections c\n WHERE c.id = NEW.collection;\n SELECT (pgstac.partition_name(NEW.collection, mindt)).* INTO partition_name, partition_range;\n NEW.name := partition_name;\n\n IF partition_range IS NULL OR partition_range = 'empty'::tstzrange THEN\n partition_range := tstzrange('-infinity'::timestamptz, 'infinity'::timestamptz, '[]');\n END IF;\n\n NEW.partition_range := partition_range;\n IF TG_OP = 'UPDATE' THEN\n mindt := least(mindt, lower(OLD.datetime_range));\n maxdt := greatest(maxdt, upper(OLD.datetime_range));\n minedt := least(minedt, lower(OLD.end_datetime_range));\n maxedt := greatest(maxedt, upper(OLD.end_datetime_range));\n NEW.datetime_range := tstzrange(mindt, maxdt, '[]');\n NEW.end_datetime_range := tstzrange(minedt, maxedt, '[]');\n END IF;\n IF TG_OP = 'INSERT' THEN\n\n IF partition_range != tstzrange('-infinity'::timestamptz, 'infinity'::timestamptz, '[]') THEN\n\n RAISE NOTICE '% % %', partition_name, parent_name, partition_range;\n q := format($q$\n CREATE TABLE IF NOT EXISTS %I partition OF %I FOR VALUES FROM (%L) TO (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n $q$,\n partition_name,\n parent_name,\n lower(partition_range),\n upper(partition_range),\n format('%s_pkey', partition_name),\n partition_name\n );\n BEGIN\n EXECUTE q;\n EXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', partition_name;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\n END;\n END IF;\n\n END IF;\n\n -- Update constraints\n EXECUTE format($q$\n SELECT\n min(datetime),\n max(datetime),\n min(end_datetime),\n max(end_datetime)\n FROM %I;\n $q$, partition_name)\n INTO t_mindt, t_maxdt, t_minedt, t_maxedt;\n mindt := least(mindt, t_mindt);\n maxdt := greatest(maxdt, t_maxdt);\n minedt := least(mindt, minedt, t_minedt);\n maxedt := greatest(maxdt, maxedt, t_maxedt);\n\n mindt := date_trunc(coalesce(partition_trunc, 'year'), mindt);\n maxdt := date_trunc(coalesce(partition_trunc, 'year'), maxdt - '1 second'::interval) + concat('1 ',coalesce(partition_trunc, 'year'))::interval;\n minedt := date_trunc(coalesce(partition_trunc, 'year'), minedt);\n maxedt := date_trunc(coalesce(partition_trunc, 'year'), maxedt - '1 second'::interval) + concat('1 ',coalesce(partition_trunc, 'year'))::interval;\n\n\n IF mindt IS NOT NULL AND maxdt IS NOT NULL AND minedt IS NOT NULL AND maxedt IS NOT NULL THEN\n NEW.datetime_range := tstzrange(mindt, maxdt, '[]');\n NEW.end_datetime_range := tstzrange(minedt, maxedt, '[]');\n IF\n TG_OP='UPDATE'\n AND OLD.datetime_range @> NEW.datetime_range\n AND OLD.end_datetime_range @> NEW.end_datetime_range\n THEN\n RAISE NOTICE 'Range unchanged, not updating constraints.';\n ELSE\n\n RAISE NOTICE '\n SETTING CONSTRAINTS\n mindt: %, maxdt: %\n minedt: %, maxedt: %\n ', mindt, maxdt, minedt, maxedt;\n IF partition_trunc IS NULL THEN\n cq := format($q$\n ALTER TABLE %7$I\n DROP CONSTRAINT IF EXISTS %1$I,\n DROP CONSTRAINT IF EXISTS %2$I,\n ADD CONSTRAINT %1$I\n CHECK (\n (datetime >= %3$L)\n AND (datetime <= %4$L)\n AND (end_datetime >= %5$L)\n AND (end_datetime <= %6$L)\n ) NOT VALID\n ;\n ALTER TABLE %7$I\n VALIDATE CONSTRAINT %1$I;\n $q$,\n format('%s_dt', partition_name),\n format('%s_edt', partition_name),\n mindt,\n maxdt,\n minedt,\n maxedt,\n partition_name\n );\n ELSE\n cq := format($q$\n ALTER TABLE %5$I\n DROP CONSTRAINT IF EXISTS %1$I,\n DROP CONSTRAINT IF EXISTS %2$I,\n ADD CONSTRAINT %2$I\n CHECK ((end_datetime >= %3$L) AND (end_datetime <= %4$L)) NOT VALID\n ;\n ALTER TABLE %5$I\n VALIDATE CONSTRAINT %2$I;\n $q$,\n format('%s_dt', partition_name),\n format('%s_edt', partition_name),\n minedt,\n maxedt,\n partition_name\n );\n\n END IF;\n RAISE NOTICE 'Altering Constraints. %', cq;\n EXECUTE cq;\n END IF;\n ELSE\n NEW.datetime_range = NULL;\n NEW.end_datetime_range = NULL;\n\n cq := format($q$\n ALTER TABLE %3$I\n DROP CONSTRAINT IF EXISTS %1$I,\n DROP CONSTRAINT IF EXISTS %2$I,\n ADD CONSTRAINT %1$I\n CHECK ((datetime IS NULL AND end_datetime IS NULL)) NOT VALID\n ;\n ALTER TABLE %3$I\n VALIDATE CONSTRAINT %1$I;\n $q$,\n format('%s_dt', partition_name),\n format('%s_edt', partition_name),\n partition_name\n );\n EXECUTE cq;\n END IF;\n\n RETURN NEW;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER partitions_trigger BEFORE INSERT OR UPDATE ON partitions FOR EACH ROW\nEXECUTE FUNCTION partitions_trigger_func();\n\n\nCREATE OR REPLACE FUNCTION create_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection(data jsonb) RETURNS VOID AS $$\nDECLARE\n out collections%ROWTYPE;\nBEGIN\n UPDATE collections SET content=data WHERE id = data->>'id' RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ON CONFLICT (id) DO\n UPDATE\n SET content=EXCLUDED.content\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION delete_collection(_id text) RETURNS VOID AS $$\nDECLARE\n out collections%ROWTYPE;\nBEGIN\n DELETE FROM collections WHERE id = _id RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION get_collection(id text) RETURNS jsonb AS $$\n SELECT content FROM collections\n WHERE id=$1\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION all_collections() RETURNS jsonb AS $$\n SELECT jsonb_agg(content) FROM collections;\n;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\nCREATE TABLE queryables (\n id bigint GENERATED ALWAYS AS identity PRIMARY KEY,\n name text UNIQUE NOT NULL,\n collection_ids text[], -- used to determine what partitions to create indexes on\n definition jsonb,\n property_path text,\n property_wrapper text,\n property_index_type text\n);\nCREATE INDEX queryables_name_idx ON queryables (name);\nCREATE INDEX queryables_property_wrapper_idx ON queryables (property_wrapper);\n\n\nINSERT INTO queryables (name, definition) VALUES\n('id', '{\"title\": \"Item ID\",\"description\": \"Item identifier\",\"$ref\": \"https://schemas.stacspec.org/v1.0.0/item-spec/json-schema/item.json#/definitions/core/allOf/2/properties/id\"}'),\n('datetime','{\"description\": \"Datetime\",\"type\": \"string\",\"title\": \"Acquired\",\"format\": \"date-time\",\"pattern\": \"(\\\\+00:00|Z)$\"}')\nON CONFLICT DO NOTHING;\n\n\n\nINSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n('eo:cloud_cover','{\"$ref\": \"https://stac-extensions.github.io/eo/v1.0.0/schema.json#/definitions/fieldsproperties/eo:cloud_cover\"}','to_int','BTREE')\nON CONFLICT DO NOTHING;\n\nCREATE OR REPLACE FUNCTION array_to_path(arr text[]) RETURNS text AS $$\n SELECT string_agg(\n quote_literal(v),\n '->'\n ) FROM unnest(arr) v;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\n\n\nCREATE OR REPLACE FUNCTION queryable(\n IN dotpath text,\n OUT path text,\n OUT expression text,\n OUT wrapper text\n) AS $$\nDECLARE\n q RECORD;\n path_elements text[];\nBEGIN\n IF dotpath IN ('id', 'geometry', 'datetime', 'end_datetime', 'collection') THEN\n path := dotpath;\n expression := dotpath;\n wrapper := NULL;\n RETURN;\n END IF;\n SELECT * INTO q FROM queryables WHERE name=dotpath;\n IF q.property_wrapper IS NULL THEN\n IF q.definition->>'type' = 'number' THEN\n wrapper := 'to_float';\n ELSIF q.definition->>'format' = 'date-time' THEN\n wrapper := 'to_tstz';\n ELSE\n wrapper := 'to_text';\n END IF;\n ELSE\n wrapper := q.property_wrapper;\n END IF;\n IF q.property_path IS NOT NULL THEN\n path := q.property_path;\n ELSE\n path_elements := string_to_array(dotpath, '.');\n IF path_elements[1] IN ('links', 'assets', 'stac_version', 'stac_extensions') THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSIF path_elements[1] = 'properties' THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSE\n path := format($F$content->'properties'->%s$F$, array_to_path(path_elements));\n END IF;\n END IF;\n expression := format('%I(%s)', wrapper, path);\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\nCREATE OR REPLACE FUNCTION create_queryable_indexes() RETURNS VOID AS $$\nDECLARE\n queryable RECORD;\n q text;\nBEGIN\n FOR queryable IN\n SELECT\n queryables.id as qid,\n CASE WHEN collections.key IS NULL THEN 'items' ELSE format('_items_%s',collections.key) END AS part,\n property_index_type,\n expression\n FROM\n queryables\n LEFT JOIN collections ON (collections.id = ANY (queryables.collection_ids))\n JOIN LATERAL queryable(queryables.name) ON (queryables.property_index_type IS NOT NULL)\n LOOP\n q := format(\n $q$\n CREATE INDEX IF NOT EXISTS %I ON %I USING %s ((%s));\n $q$,\n format('%s_%s_idx', queryable.part, queryable.qid),\n queryable.part,\n COALESCE(queryable.property_index_type, 'to_text'),\n queryable.expression\n );\n RAISE NOTICE '%',q;\n EXECUTE q;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION queryables_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\nBEGIN\nPERFORM create_queryable_indexes();\nRETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER queryables_trigger AFTER INSERT OR UPDATE ON queryables\nFOR EACH STATEMENT EXECUTE PROCEDURE queryables_trigger_func();\n\nCREATE TRIGGER queryables_collection_trigger AFTER INSERT OR UPDATE ON collections\nFOR EACH STATEMENT EXECUTE PROCEDURE queryables_trigger_func();\nCREATE OR REPLACE FUNCTION parse_dtrange(\n _indate jsonb,\n relative_base timestamptz DEFAULT date_trunc('hour', CURRENT_TIMESTAMP)\n) RETURNS tstzrange AS $$\nDECLARE\n timestrs text[];\n s timestamptz;\n e timestamptz;\nBEGIN\n timestrs :=\n CASE\n WHEN _indate ? 'timestamp' THEN\n ARRAY[_indate->>'timestamp']\n WHEN _indate ? 'interval' THEN\n to_text_array(_indate->'interval')\n WHEN jsonb_typeof(_indate) = 'array' THEN\n to_text_array(_indate)\n ELSE\n regexp_split_to_array(\n _indate->>0,\n '/'\n )\n END;\n RAISE NOTICE 'TIMESTRS %', timestrs;\n IF cardinality(timestrs) = 1 THEN\n IF timestrs[1] ILIKE 'P%' THEN\n RETURN tstzrange(relative_base - upper(timestrs[1])::interval, relative_base, '[)');\n END IF;\n s := timestrs[1]::timestamptz;\n RETURN tstzrange(s, s, '[]');\n END IF;\n\n IF cardinality(timestrs) != 2 THEN\n RAISE EXCEPTION 'Timestamp cannot have more than 2 values';\n END IF;\n\n IF timestrs[1] = '..' THEN\n s := '-infinity'::timestamptz;\n e := timestrs[2]::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] = '..' THEN\n s := timestrs[1]::timestamptz;\n e := 'infinity'::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[1] ILIKE 'P%' AND timestrs[2] NOT ILIKE 'P%' THEN\n e := timestrs[2]::timestamptz;\n s := e - upper(timestrs[1])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] ILIKE 'P%' AND timestrs[1] NOT ILIKE 'P%' THEN\n s := timestrs[1]::timestamptz;\n e := s + upper(timestrs[2])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n s := timestrs[1]::timestamptz;\n e := timestrs[2]::timestamptz;\n\n RETURN tstzrange(s,e,'[)');\n\n RETURN NULL;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT PARALLEL SAFE SET TIME ZONE 'UTC';\n\nCREATE OR REPLACE FUNCTION parse_dtrange(\n _indate text,\n relative_base timestamptz DEFAULT CURRENT_TIMESTAMP\n) RETURNS tstzrange AS $$\n SELECT parse_dtrange(to_jsonb(_indate), relative_base);\n$$ LANGUAGE SQL STABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION temporal_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\n ll text := 'datetime';\n lh text := 'end_datetime';\n rrange tstzrange;\n rl text;\n rh text;\n outq text;\nBEGIN\n rrange := parse_dtrange(args->1);\n RAISE NOTICE 'Constructing temporal query OP: %, ARGS: %, RRANGE: %', op, args, rrange;\n op := lower(op);\n rl := format('%L::timestamptz', lower(rrange));\n rh := format('%L::timestamptz', upper(rrange));\n outq := CASE op\n WHEN 't_before' THEN 'lh < rl'\n WHEN 't_after' THEN 'll > rh'\n WHEN 't_meets' THEN 'lh = rl'\n WHEN 't_metby' THEN 'll = rh'\n WHEN 't_overlaps' THEN 'll < rl AND rl < lh < rh'\n WHEN 't_overlappedby' THEN 'rl < ll < rh AND lh > rh'\n WHEN 't_starts' THEN 'll = rl AND lh < rh'\n WHEN 't_startedby' THEN 'll = rl AND lh > rh'\n WHEN 't_during' THEN 'll > rl AND lh < rh'\n WHEN 't_contains' THEN 'll < rl AND lh > rh'\n WHEN 't_finishes' THEN 'll > rl AND lh = rh'\n WHEN 't_finishedby' THEN 'll < rl AND lh = rh'\n WHEN 't_equals' THEN 'll = rl AND lh = rh'\n WHEN 't_disjoint' THEN 'NOT (ll <= rh AND lh >= rl)'\n WHEN 't_intersects' THEN 'll <= rh AND lh >= rl'\n WHEN 'anyinteracts' THEN 'll <= rh AND lh >= rl'\n END;\n outq := regexp_replace(outq, '\\mll\\M', ll);\n outq := regexp_replace(outq, '\\mlh\\M', lh);\n outq := regexp_replace(outq, '\\mrl\\M', rl);\n outq := regexp_replace(outq, '\\mrh\\M', rh);\n outq := format('(%s)', outq);\n RETURN outq;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\n\n\nCREATE OR REPLACE FUNCTION spatial_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\n geom text;\n j jsonb := args->1;\nBEGIN\n op := lower(op);\n RAISE NOTICE 'Constructing spatial query OP: %, ARGS: %', op, args;\n IF op NOT IN ('s_equals','s_disjoint','s_touches','s_within','s_overlaps','s_crosses','s_intersects','intersects','s_contains') THEN\n RAISE EXCEPTION 'Spatial Operator % Not Supported', op;\n END IF;\n op := regexp_replace(op, '^s_', 'st_');\n IF op = 'intersects' THEN\n op := 'st_intersects';\n END IF;\n -- Convert geometry to WKB string\n IF j ? 'type' AND j ? 'coordinates' THEN\n geom := st_geomfromgeojson(j)::text;\n ELSIF jsonb_typeof(j) = 'array' THEN\n geom := bbox_geom(j)::text;\n END IF;\n\n RETURN format('%s(geometry, %L::geometry)', op, geom);\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION query_to_cql2(q jsonb) RETURNS jsonb AS $$\n-- Translates anything passed in through the deprecated \"query\" into equivalent CQL2\nWITH t AS (\n SELECT key as property, value as ops\n FROM jsonb_each(q)\n), t2 AS (\n SELECT property, (jsonb_each(ops)).*\n FROM t WHERE jsonb_typeof(ops) = 'object'\n UNION ALL\n SELECT property, 'eq', ops\n FROM t WHERE jsonb_typeof(ops) != 'object'\n)\nSELECT\n jsonb_strip_nulls(jsonb_build_object(\n 'op', 'and',\n 'args', jsonb_agg(\n jsonb_build_object(\n 'op', key,\n 'args', jsonb_build_array(\n jsonb_build_object('property',property),\n value\n )\n )\n )\n )\n) as qcql FROM t2\n;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\nCREATE OR REPLACE FUNCTION cql1_to_cql2(j jsonb) RETURNS jsonb AS $$\nDECLARE\n args jsonb;\n ret jsonb;\nBEGIN\n RAISE NOTICE 'CQL1_TO_CQL2: %', j;\n IF j ? 'filter' THEN\n RETURN cql1_to_cql2(j->'filter');\n END IF;\n IF j ? 'property' THEN\n RETURN j;\n END IF;\n IF jsonb_typeof(j) = 'array' THEN\n SELECT jsonb_agg(cql1_to_cql2(el)) INTO args FROM jsonb_array_elements(j) el;\n RETURN args;\n END IF;\n IF jsonb_typeof(j) = 'number' THEN\n RETURN j;\n END IF;\n IF jsonb_typeof(j) = 'string' THEN\n RETURN j;\n END IF;\n\n IF jsonb_typeof(j) = 'object' THEN\n SELECT jsonb_build_object(\n 'op', key,\n 'args', cql1_to_cql2(value)\n ) INTO ret\n FROM jsonb_each(j)\n WHERE j IS NOT NULL;\n RETURN ret;\n END IF;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE STRICT;\n\nCREATE TABLE cql2_ops (\n op text PRIMARY KEY,\n template text,\n types text[]\n);\nINSERT INTO cql2_ops (op, template, types) VALUES\n ('eq', '%s = %s', NULL),\n ('lt', '%s < %s', NULL),\n ('lte', '%s <= %s', NULL),\n ('gt', '%s > %s', NULL),\n ('gte', '%s >= %s', NULL),\n ('le', '%s <= %s', NULL),\n ('ge', '%s >= %s', NULL),\n ('=', '%s = %s', NULL),\n ('<', '%s < %s', NULL),\n ('<=', '%s <= %s', NULL),\n ('>', '%s > %s', NULL),\n ('>=', '%s >= %s', NULL),\n ('like', '%s LIKE %s', NULL),\n ('ilike', '%s ILIKE %s', NULL),\n ('+', '%s + %s', NULL),\n ('-', '%s - %s', NULL),\n ('*', '%s * %s', NULL),\n ('/', '%s / %s', NULL),\n ('in', '%s = ANY (%s)', NULL),\n ('not', 'NOT (%s)', NULL),\n ('between', '%s BETWEEN (%2$s)[1] AND (%2$s)[2]', NULL),\n ('isnull', '%s IS NULL', NULL)\nON CONFLICT (op) DO UPDATE\n SET\n template = EXCLUDED.template;\n;\n\n\nCREATE OR REPLACE FUNCTION cql2_query(j jsonb, wrapper text DEFAULT NULL) RETURNS text AS $$\n#variable_conflict use_variable\nDECLARE\n args jsonb := j->'args';\n arg jsonb;\n op text := lower(j->>'op');\n cql2op RECORD;\n literal text;\nBEGIN\n IF j IS NULL OR (op IS NOT NULL AND args IS NULL) THEN\n RETURN NULL;\n END IF;\n RAISE NOTICE 'CQL2_QUERY: %', j;\n IF j ? 'filter' THEN\n RETURN cql2_query(j->'filter');\n END IF;\n\n IF j ? 'upper' THEN\n RETURN format('upper(%s)', cql2_query(j->'upper'));\n END IF;\n\n IF j ? 'lower' THEN\n RETURN format('lower(%s)', cql2_query(j->'lower'));\n END IF;\n\n -- Temporal Query\n IF op ilike 't_%' or op = 'anyinteracts' THEN\n RETURN temporal_op_query(op, args);\n END IF;\n\n -- If property is a timestamp convert it to text to use with\n -- general operators\n IF j ? 'timestamp' THEN\n RETURN format('%L::timestamptz', to_tstz(j->'timestamp'));\n END IF;\n IF j ? 'interval' THEN\n RAISE EXCEPTION 'Please use temporal operators when using intervals.';\n RETURN NONE;\n END IF;\n\n -- Spatial Query\n IF op ilike 's_%' or op = 'intersects' THEN\n RETURN spatial_op_query(op, args);\n END IF;\n\n\n IF op = 'in' THEN\n RETURN format(\n '%s = ANY (%L)',\n cql2_query(args->0),\n to_text_array(args->1)\n );\n END IF;\n\n\n\n IF op = 'between' THEN\n SELECT (queryable(a->>'property')).wrapper INTO wrapper\n FROM jsonb_array_elements(args) a\n WHERE a ? 'property' LIMIT 1;\n\n RETURN format(\n '%s BETWEEN %s and %s',\n cql2_query(args->0, wrapper),\n cql2_query(args->1->0, wrapper),\n cql2_query(args->1->1, wrapper)\n );\n END IF;\n\n -- Make sure that args is an array and run cql2_query on\n -- each element of the array\n RAISE NOTICE 'ARGS PRE: %', args;\n IF j ? 'args' THEN\n IF jsonb_typeof(args) != 'array' THEN\n args := jsonb_build_array(args);\n END IF;\n\n SELECT (queryable(a->>'property')).wrapper INTO wrapper\n FROM jsonb_array_elements(args) a\n WHERE a ? 'property' LIMIT 1;\n\n SELECT jsonb_agg(cql2_query(a, wrapper))\n INTO args\n FROM jsonb_array_elements(args) a;\n END IF;\n RAISE NOTICE 'ARGS: %', args;\n\n IF op IN ('and', 'or') THEN\n RETURN\n format(\n '(%s)',\n array_to_string(to_text_array(args), format(' %s ', upper(op)))\n );\n END IF;\n\n -- Look up template from cql2_ops\n IF j ? 'op' THEN\n SELECT * INTO cql2op FROM cql2_ops WHERE cql2_ops.op ilike op;\n IF FOUND THEN\n -- If specific index set in queryables for a property cast other arguments to that type\n RETURN format(\n cql2op.template,\n VARIADIC (to_text_array(args))\n );\n ELSE\n RAISE EXCEPTION 'Operator % Not Supported.', op;\n END IF;\n END IF;\n\n\n IF j ? 'property' THEN\n RETURN (queryable(j->>'property')).expression;\n END IF;\n\n IF wrapper IS NOT NULL THEN\n EXECUTE format('SELECT %I(%L)', wrapper, j) INTO literal;\n RAISE NOTICE '% % %',wrapper, j, literal;\n RETURN format('%I(%L)', wrapper, j);\n END IF;\n\n RETURN quote_literal(to_text(j));\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION paging_dtrange(\n j jsonb\n) RETURNS tstzrange AS $$\nDECLARE\n op text;\n filter jsonb := j->'filter';\n dtrange tstzrange := tstzrange('-infinity'::timestamptz,'infinity'::timestamptz);\n sdate timestamptz := '-infinity'::timestamptz;\n edate timestamptz := 'infinity'::timestamptz;\n jpitem jsonb;\nBEGIN\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n END IF;\n IF NOT (filter @? '$.**.op ? (@ == \"or\" || @ == \"not\")') THEN\n FOR jpitem IN SELECT j FROM jsonb_path_query(filter,'strict $.** ? (@.args[*].property == \"datetime\")'::jsonpath) j LOOP\n op := lower(jpitem->>'op');\n dtrange := parse_dtrange(jpitem->'args'->1);\n IF op IN ('<=', 'lt', 'lte', '<', 'le', 't_before') THEN\n sdate := greatest(sdate,'-infinity');\n edate := least(edate, upper(dtrange));\n ELSIF op IN ('>=', '>', 'gt', 'gte', 'ge', 't_after') THEN\n edate := least(edate, 'infinity');\n sdate := greatest(sdate, lower(dtrange));\n ELSIF op IN ('=', 'eq') THEN\n edate := least(edate, upper(dtrange));\n sdate := greatest(sdate, lower(dtrange));\n END IF;\n RAISE NOTICE '2 OP: %, ARGS: %, DTRANGE: %, SDATE: %, EDATE: %', op, jpitem->'args'->1, dtrange, sdate, edate;\n END LOOP;\n END IF;\n IF sdate > edate THEN\n RETURN 'empty'::tstzrange;\n END IF;\n RETURN tstzrange(sdate,edate, '[]');\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT SET TIME ZONE 'UTC';\n\nCREATE OR REPLACE FUNCTION paging_collections(\n IN j jsonb\n) RETURNS text[] AS $$\nDECLARE\n filter jsonb := j->'filter';\n jpitem jsonb;\n op text;\n args jsonb;\n arg jsonb;\n collections text[];\nBEGIN\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n END IF;\n IF NOT (filter @? '$.**.op ? (@ == \"or\" || @ == \"not\")') THEN\n FOR jpitem IN SELECT j FROM jsonb_path_query(filter,'strict $.** ? (@.args[*].property == \"collection\")'::jsonpath) j LOOP\n RAISE NOTICE 'JPITEM: %', jpitem;\n op := jpitem->>'op';\n args := jpitem->'args';\n IF op IN ('=', 'eq', 'in') THEN\n FOR arg IN SELECT a FROM jsonb_array_elements(args) a LOOP\n IF jsonb_typeof(arg) IN ('string', 'array') THEN\n RAISE NOTICE 'arg: %, collections: %', arg, collections;\n IF collections IS NULL OR collections = '{}'::text[] THEN\n collections := to_text_array(arg);\n ELSE\n collections := array_intersection(collections, to_text_array(arg));\n END IF;\n END IF;\n END LOOP;\n END IF;\n END LOOP;\n END IF;\n IF collections = '{}'::text[] THEN\n RETURN NULL;\n END IF;\n RETURN collections;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\nCREATE TABLE items (\n id text NOT NULL,\n geometry geometry NOT NULL,\n collection text NOT NULL,\n datetime timestamptz NOT NULL,\n end_datetime timestamptz NOT NULL,\n content JSONB NOT NULL\n)\nPARTITION BY LIST (collection)\n;\n\nCREATE INDEX \"datetime_idx\" ON items USING BTREE (datetime DESC, end_datetime ASC);\nCREATE INDEX \"geometry_idx\" ON items USING GIST (geometry);\n\nCREATE STATISTICS datetime_stats (dependencies) on datetime, end_datetime from items;\n\n\nALTER TABLE items ADD CONSTRAINT items_collections_fk FOREIGN KEY (collection) REFERENCES collections(id) ON DELETE CASCADE DEFERRABLE;\n\n\nCREATE OR REPLACE FUNCTION content_slim(_item jsonb, _collection jsonb) RETURNS jsonb AS $$\n SELECT\n jsonb_object_agg(\n key,\n CASE\n WHEN\n jsonb_typeof(c.value) = 'object'\n AND\n jsonb_typeof(i.value) = 'object'\n THEN content_slim(i.value, c.value)\n ELSE i.value\n END\n )\n FROM\n jsonb_each(_item) as i\n LEFT JOIN\n jsonb_each(_collection) as c\n USING (key)\n WHERE\n i.value IS DISTINCT FROM c.value\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_slim(_item jsonb) RETURNS jsonb AS $$\n SELECT content_slim(_item - '{id,type,collection,geometry,bbox}'::text[], collection_base_item(_item->>'collection'));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_dehydrate(content jsonb) RETURNS items AS $$\n SELECT\n content->>'id' as id,\n stac_geom(content) as geometry,\n content->>'collection' as collection,\n stac_datetime(content) as datetime,\n stac_end_datetime(content) as end_datetime,\n content_slim(content) as content\n ;\n$$ LANGUAGE SQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION include_field(f text, fields jsonb DEFAULT '{}'::jsonb) RETURNS boolean AS $$\nDECLARE\n includes jsonb := coalesce(fields->'includes', fields->'include', '[]'::jsonb);\n excludes jsonb := coalesce(fields->'excludes', fields->'exclude', '[]'::jsonb);\nBEGIN\n IF f IS NULL THEN\n RETURN NULL;\n ELSIF jsonb_array_length(includes)>0 AND includes ? f THEN\n RETURN TRUE;\n ELSIF jsonb_array_length(excludes)>0 AND excludes ? f THEN\n RETURN FALSE;\n ELSIF jsonb_array_length(includes)>0 AND NOT includes ? f THEN\n RETURN FALSE;\n END IF;\n RETURN TRUE;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION key_filter(IN k text, IN val jsonb, INOUT kf jsonb, OUT include boolean) AS $$\nDECLARE\n includes jsonb := coalesce(kf->'includes', kf->'include', '[]'::jsonb);\n excludes jsonb := coalesce(kf->'excludes', kf->'exclude', '[]'::jsonb);\nBEGIN\n RAISE NOTICE '% % %', k, val, kf;\n\n include := TRUE;\n IF k = 'properties' AND NOT excludes ? 'properties' THEN\n excludes := excludes || '[\"properties\"]';\n include := TRUE;\n RAISE NOTICE 'Prop include %', include;\n ELSIF\n jsonb_array_length(excludes)>0 AND excludes ? k THEN\n include := FALSE;\n ELSIF\n jsonb_array_length(includes)>0 AND NOT includes ? k THEN\n include := FALSE;\n ELSIF\n jsonb_array_length(includes)>0 AND includes ? k THEN\n includes := '[]'::jsonb;\n RAISE NOTICE 'KF: %', kf;\n END IF;\n kf := jsonb_build_object('includes', includes, 'excludes', excludes);\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION strip_assets(a jsonb) RETURNS jsonb AS $$\n WITH t AS (SELECT * FROM jsonb_each(a))\n SELECT jsonb_object_agg(key, value) FROM t\n WHERE value ? 'href';\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION content_hydrate(\n _item jsonb,\n _collection jsonb,\n fields jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\n SELECT\n jsonb_strip_nulls(jsonb_object_agg(\n key,\n CASE\n WHEN key = 'properties' AND include_field('properties', fields) THEN\n i.value\n WHEN key = 'properties' THEN\n content_hydrate(i.value, c.value, kf)\n WHEN\n c.value IS NULL AND key != 'properties'\n THEN i.value\n WHEN\n key = 'assets'\n AND\n jsonb_typeof(c.value) = 'object'\n AND\n jsonb_typeof(i.value) = 'object'\n THEN strip_assets(content_hydrate(i.value, c.value, kf))\n WHEN\n jsonb_typeof(c.value) = 'object'\n AND\n jsonb_typeof(i.value) = 'object'\n THEN content_hydrate(i.value, c.value, kf)\n ELSE coalesce(i.value, c.value)\n END\n ))\n FROM\n jsonb_each(coalesce(_item,'{}'::jsonb)) as i\n FULL JOIN\n jsonb_each(coalesce(_collection,'{}'::jsonb)) as c\n USING (key)\n JOIN LATERAL (\n SELECT kf, include FROM key_filter(key, i.value, fields)\n ) as k ON (include)\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION content_hydrate(_item items, _collection collections, fields jsonb DEFAULT '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\n content jsonb;\n base_item jsonb := _collection.base_item;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry)::jsonb;\n END IF;\n IF include_field('bbox', fields) THEN\n bbox := geom_bbox(_item.geometry)::jsonb;\n END IF;\n output := content_hydrate(\n jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'bbox',bbox,\n 'collection', _item.collection\n ) || _item.content,\n _collection.base_item,\n fields\n );\n\n RETURN output;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_nonhydrated(\n _item items,\n fields jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry)::jsonb;\n END IF;\n IF include_field('bbox', fields) THEN\n bbox := geom_bbox(_item.geometry)::jsonb;\n END IF;\n output := jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'bbox',bbox,\n 'collection', _item.collection\n ) || _item.content;\n RETURN output;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_hydrate(_item items, fields jsonb DEFAULT '{}'::jsonb) RETURNS jsonb AS $$\n SELECT content_hydrate(\n _item,\n (SELECT c FROM collections c WHERE id=_item.collection LIMIT 1),\n fields\n );\n$$ LANGUAGE SQL STABLE;\n\n\nCREATE UNLOGGED TABLE items_staging (\n content JSONB NOT NULL\n);\nCREATE UNLOGGED TABLE items_staging_ignore (\n content JSONB NOT NULL\n);\nCREATE UNLOGGED TABLE items_staging_upsert (\n content JSONB NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION items_staging_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p record;\n _partitions text[];\n ts timestamptz := clock_timestamp();\nBEGIN\n RAISE NOTICE 'Creating Partitions. %', clock_timestamp() - ts;\n WITH ranges AS (\n SELECT\n n.content->>'collection' as collection,\n stac_daterange(n.content->'properties') as dtr\n FROM newdata n\n ), p AS (\n SELECT\n collection,\n lower(dtr) as datetime,\n upper(dtr) as end_datetime,\n (partition_name(\n collection,\n lower(dtr)\n )).partition_name as name\n FROM ranges\n )\n INSERT INTO partitions (collection, datetime_range, end_datetime_range)\n SELECT\n collection,\n tstzrange(min(datetime), max(datetime), '[]') as datetime_range,\n tstzrange(min(end_datetime), max(end_datetime), '[]') as end_datetime_range\n FROM p\n GROUP BY collection, name\n ON CONFLICT (name) DO UPDATE SET\n datetime_range = EXCLUDED.datetime_range,\n end_datetime_range = EXCLUDED.end_datetime_range\n ;\n\n RAISE NOTICE 'Doing the insert. %', clock_timestamp() - ts;\n IF TG_TABLE_NAME = 'items_staging' THEN\n INSERT INTO items\n SELECT\n (content_dehydrate(content)).*\n FROM newdata;\n DELETE FROM items_staging;\n ELSIF TG_TABLE_NAME = 'items_staging_ignore' THEN\n INSERT INTO items\n SELECT\n (content_dehydrate(content)).*\n FROM newdata\n ON CONFLICT DO NOTHING;\n DELETE FROM items_staging_ignore;\n ELSIF TG_TABLE_NAME = 'items_staging_upsert' THEN\n WITH staging_formatted AS (\n SELECT (content_dehydrate(content)).* FROM newdata\n ), deletes AS (\n DELETE FROM items i USING staging_formatted s\n WHERE\n i.id = s.id\n AND i.collection = s.collection\n AND i IS DISTINCT FROM s\n RETURNING i.id, i.collection\n )\n INSERT INTO items\n SELECT s.* FROM\n staging_formatted s\n JOIN deletes d\n USING (id, collection);\n DELETE FROM items_staging_upsert;\n END IF;\n RAISE NOTICE 'Done. %', clock_timestamp() - ts;\n\n RETURN NULL;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER items_staging_insert_trigger AFTER INSERT ON items_staging REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\nCREATE TRIGGER items_staging_insert_ignore_trigger AFTER INSERT ON items_staging_ignore REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\nCREATE TRIGGER items_staging_insert_upsert_trigger AFTER INSERT ON items_staging_upsert REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\n\n\n\nCREATE OR REPLACE FUNCTION item_by_id(_id text, _collection text DEFAULT NULL) RETURNS items AS\n$$\nDECLARE\n i items%ROWTYPE;\nBEGIN\n SELECT * INTO i FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection) LIMIT 1;\n RETURN i;\nEND;\n$$ LANGUAGE PLPGSQL STABLE SECURITY DEFINER SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION get_item(_id text, _collection text DEFAULT NULL) RETURNS jsonb AS $$\n SELECT content_hydrate(items) FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection);\n$$ LANGUAGE SQL STABLE SECURITY DEFINER SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION delete_item(_id text, _collection text DEFAULT NULL) RETURNS VOID AS $$\nDECLARE\nout items%ROWTYPE;\nBEGIN\n DELETE FROM items WHERE id = _id AND (_collection IS NULL OR collection=_collection) RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n--/*\nCREATE OR REPLACE FUNCTION create_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION update_item(content jsonb) RETURNS VOID AS $$\nDECLARE\n old items %ROWTYPE;\n out items%ROWTYPE;\nBEGIN\n SELECT delete_item(content->>'id', content->>'collection');\n SELECT create_item(content);\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION create_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION collection_bbox(id text) RETURNS jsonb AS $$\n SELECT (replace(replace(replace(st_extent(geometry)::text,'BOX(','[['),')',']]'),' ',','))::jsonb\n FROM items WHERE collection=$1;\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION collection_temporal_extent(id text) RETURNS jsonb AS $$\n SELECT to_jsonb(array[array[min(datetime)::text, max(datetime)::text]])\n FROM items WHERE collection=$1;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection_extents() RETURNS VOID AS $$\nUPDATE collections SET\n content = content ||\n jsonb_build_object(\n 'extent', jsonb_build_object(\n 'spatial', jsonb_build_object(\n 'bbox', collection_bbox(collections.id)\n ),\n 'temporal', jsonb_build_object(\n 'interval', collection_temporal_extent(collections.id)\n )\n )\n )\n;\n$$ LANGUAGE SQL;\nCREATE VIEW partition_steps AS\nSELECT\n name,\n date_trunc('month',lower(datetime_range)) as sdate,\n date_trunc('month', upper(datetime_range)) + '1 month'::interval as edate\n FROM partitions WHERE datetime_range IS NOT NULL AND datetime_range != 'empty'::tstzrange\n ORDER BY datetime_range ASC\n;\n\nCREATE OR REPLACE FUNCTION chunker(\n IN _where text,\n OUT s timestamptz,\n OUT e timestamptz\n) RETURNS SETOF RECORD AS $$\nDECLARE\n explain jsonb;\nBEGIN\n IF _where IS NULL THEN\n _where := ' TRUE ';\n END IF;\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s;', _where)\n INTO explain;\n\n RETURN QUERY\n WITH t AS (\n SELECT j->>0 as p FROM\n jsonb_path_query(\n explain,\n 'strict $.**.\"Relation Name\" ? (@ != null)'\n ) j\n ),\n parts AS (\n SELECT sdate, edate FROM t JOIN partition_steps ON (t.p = name)\n ),\n times AS (\n SELECT sdate FROM parts\n UNION\n SELECT edate FROM parts\n ),\n uniq AS (\n SELECT DISTINCT sdate FROM times ORDER BY sdate\n ),\n last AS (\n SELECT sdate, lead(sdate, 1) over () as edate FROM uniq\n )\n SELECT sdate, edate FROM last WHERE edate IS NOT NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION partition_queries(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN partitions text[] DEFAULT NULL\n) RETURNS SETOF text AS $$\nDECLARE\n query text;\n sdate timestamptz;\n edate timestamptz;\nBEGIN\nIF _where IS NULL OR trim(_where) = '' THEN\n _where = ' TRUE ';\nEND IF;\nRAISE NOTICE 'Getting chunks for % %', _where, _orderby;\nIF _orderby ILIKE 'datetime d%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 DESC LOOP\n RETURN NEXT format($q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n $q$,\n sdate,\n edate,\n _where,\n _orderby\n );\n END LOOP;\nELSIF _orderby ILIKE 'datetime a%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 ASC LOOP\n RETURN NEXT format($q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n $q$,\n sdate,\n edate,\n _where,\n _orderby\n );\n END LOOP;\nELSE\n query := format($q$\n SELECT * FROM items\n WHERE %s\n ORDER BY %s\n $q$, _where, _orderby\n );\n\n RETURN NEXT query;\n RETURN;\nEND IF;\n\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION partition_query_view(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN _limit int DEFAULT 10\n) RETURNS text AS $$\n WITH p AS (\n SELECT * FROM partition_queries(_where, _orderby) p\n )\n SELECT\n CASE WHEN EXISTS (SELECT 1 FROM p) THEN\n (SELECT format($q$\n SELECT * FROM (\n %s\n ) total LIMIT %s\n $q$,\n string_agg(\n format($q$ SELECT * FROM ( %s ) AS sub $q$, p),\n '\n UNION ALL\n '\n ),\n _limit\n ))\n ELSE NULL\n END FROM p;\n$$ LANGUAGE SQL IMMUTABLE;\n\n\n\n\nCREATE OR REPLACE FUNCTION stac_search_to_where(j jsonb) RETURNS text AS $$\nDECLARE\n where_segments text[];\n _where text;\n dtrange tstzrange;\n collections text[];\n geom geometry;\n sdate timestamptz;\n edate timestamptz;\n filterlang text;\n filter jsonb := j->'filter';\nBEGIN\n IF j ? 'ids' THEN\n where_segments := where_segments || format('id = ANY (%L) ', to_text_array(j->'ids'));\n END IF;\n\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n where_segments := where_segments || format('collection = ANY (%L) ', collections);\n END IF;\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n\n where_segments := where_segments || format(' datetime <= %L::timestamptz AND end_datetime >= %L::timestamptz ',\n edate,\n sdate\n );\n END IF;\n\n geom := stac_geom(j);\n IF geom IS NOT NULL THEN\n where_segments := where_segments || format('st_intersects(geometry, %L)',geom);\n END IF;\n\n filterlang := COALESCE(\n j->>'filter-lang',\n get_setting('default-filter-lang', j->'conf')\n );\n IF NOT filter @? '$.**.op' THEN\n filterlang := 'cql-json';\n END IF;\n\n IF filterlang NOT IN ('cql-json','cql2-json') AND j ? 'filter' THEN\n RAISE EXCEPTION '% is not a supported filter-lang. Please use cql-json or cql2-json.', filterlang;\n END IF;\n\n IF j ? 'query' AND j ? 'filter' THEN\n RAISE EXCEPTION 'Can only use either query or filter at one time.';\n END IF;\n\n IF j ? 'query' THEN\n filter := query_to_cql2(j->'query');\n ELSIF filterlang = 'cql-json' THEN\n filter := cql1_to_cql2(filter);\n END IF;\n RAISE NOTICE 'FILTER: %', filter;\n where_segments := where_segments || cql2_query(filter);\n IF cardinality(where_segments) < 1 THEN\n RETURN ' TRUE ';\n END IF;\n\n _where := array_to_string(array_remove(where_segments, NULL), ' AND ');\n\n IF _where IS NULL OR BTRIM(_where) = '' THEN\n RETURN ' TRUE ';\n END IF;\n RETURN _where;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION parse_sort_dir(_dir text, reverse boolean default false) RETURNS text AS $$\n WITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n ) SELECT\n CASE\n WHEN NOT reverse THEN d\n WHEN d = 'ASC' THEN 'DESC'\n WHEN d = 'DESC' THEN 'ASC'\n END\n FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION sort_dir_to_op(_dir text, prev boolean default false) RETURNS text AS $$\n WITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n ) SELECT\n CASE\n WHEN d = 'ASC' AND prev THEN '<='\n WHEN d = 'DESC' AND prev THEN '>='\n WHEN d = 'ASC' THEN '>='\n WHEN d = 'DESC' THEN '<='\n END\n FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION sort_sqlorderby(\n _search jsonb DEFAULT NULL,\n reverse boolean DEFAULT FALSE\n) RETURNS text AS $$\n WITH sortby AS (\n SELECT coalesce(_search->'sortby','[{\"field\":\"datetime\", \"direction\":\"desc\"}]') as sort\n ), withid AS (\n SELECT CASE\n WHEN sort @? '$[*] ? (@.field == \"id\")' THEN sort\n ELSE sort || '[{\"field\":\"id\", \"direction\":\"desc\"}]'::jsonb\n END as sort\n FROM sortby\n ), withid_rows AS (\n SELECT jsonb_array_elements(sort) as value FROM withid\n ),sorts AS (\n SELECT\n coalesce(\n -- field_orderby((items_path(value->>'field')).path_txt),\n (queryable(value->>'field')).expression\n ) as key,\n parse_sort_dir(value->>'direction', reverse) as dir\n FROM withid_rows\n )\n SELECT array_to_string(\n array_agg(concat(key, ' ', dir)),\n ', '\n ) FROM sorts;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_sort_dir(sort_item jsonb) RETURNS text AS $$\n SELECT CASE WHEN sort_item->>'direction' ILIKE 'desc%' THEN 'DESC' ELSE 'ASC' END;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION get_token_filter(_search jsonb = '{}'::jsonb, token_rec jsonb DEFAULT NULL) RETURNS text AS $$\nDECLARE\n token_id text;\n filters text[] := '{}'::text[];\n prev boolean := TRUE;\n field text;\n dir text;\n sort record;\n orfilters text[] := '{}'::text[];\n andfilters text[] := '{}'::text[];\n output text;\n token_where text;\nBEGIN\n RAISE NOTICE 'Getting Token Filter. % %', _search, token_rec;\n -- If no token provided return NULL\n IF token_rec IS NULL THEN\n IF NOT (_search ? 'token' AND\n (\n (_search->>'token' ILIKE 'prev:%')\n OR\n (_search->>'token' ILIKE 'next:%')\n )\n ) THEN\n RETURN NULL;\n END IF;\n prev := (_search->>'token' ILIKE 'prev:%');\n token_id := substr(_search->>'token', 6);\n SELECT to_jsonb(items) INTO token_rec\n FROM items WHERE id=token_id;\n END IF;\n RAISE NOTICE 'TOKEN ID: % %', token_rec, token_rec->'id';\n\n CREATE TEMP TABLE sorts (\n _row int GENERATED ALWAYS AS IDENTITY NOT NULL,\n _field text PRIMARY KEY,\n _dir text NOT NULL,\n _val text\n ) ON COMMIT DROP;\n\n -- Make sure we only have distinct columns to sort with taking the first one we get\n INSERT INTO sorts (_field, _dir)\n SELECT\n (queryable(value->>'field')).expression,\n get_sort_dir(value)\n FROM\n jsonb_array_elements(coalesce(_search->'sortby','[{\"field\":\"datetime\",\"direction\":\"desc\"}]'))\n ON CONFLICT DO NOTHING\n ;\n RAISE NOTICE 'sorts 1: %', (SELECT jsonb_agg(to_json(sorts)) FROM sorts);\n -- Get the first sort direction provided. As the id is a primary key, if there are any\n -- sorts after id they won't do anything, so make sure that id is the last sort item.\n SELECT _dir INTO dir FROM sorts ORDER BY _row ASC LIMIT 1;\n IF EXISTS (SELECT 1 FROM sorts WHERE _field = 'id') THEN\n DELETE FROM sorts WHERE _row > (SELECT _row FROM sorts WHERE _field = 'id' ORDER BY _row ASC);\n ELSE\n INSERT INTO sorts (_field, _dir) VALUES ('id', dir);\n END IF;\n\n -- Add value from looked up item to the sorts table\n UPDATE sorts SET _val=quote_literal(token_rec->>_field);\n\n -- Check if all sorts are the same direction and use row comparison\n -- to filter\n RAISE NOTICE 'sorts 2: %', (SELECT jsonb_agg(to_json(sorts)) FROM sorts);\n\n IF (SELECT count(DISTINCT _dir) FROM sorts) = 1 THEN\n SELECT format(\n '(%s) %s (%s)',\n concat_ws(', ', VARIADIC array_agg(quote_ident(_field))),\n CASE WHEN (prev AND dir = 'ASC') OR (NOT prev AND dir = 'DESC') THEN '<' ELSE '>' END,\n concat_ws(', ', VARIADIC array_agg(_val))\n ) INTO output FROM sorts\n WHERE token_rec ? _field\n ;\n ELSE\n FOR sort IN SELECT * FROM sorts ORDER BY _row asc LOOP\n RAISE NOTICE 'SORT: %', sort;\n IF sort._row = 1 THEN\n orfilters := orfilters || format('(%s %s %s)',\n quote_ident(sort._field),\n CASE WHEN (prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC') THEN '<' ELSE '>' END,\n sort._val\n );\n ELSE\n orfilters := orfilters || format('(%s AND %s %s %s)',\n array_to_string(andfilters, ' AND '),\n quote_ident(sort._field),\n CASE WHEN (prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC') THEN '<' ELSE '>' END,\n sort._val\n );\n\n END IF;\n andfilters := andfilters || format('%s = %s',\n quote_ident(sort._field),\n sort._val\n );\n END LOOP;\n output := array_to_string(orfilters, ' OR ');\n END IF;\n DROP TABLE IF EXISTS sorts;\n token_where := concat('(',coalesce(output,'true'),')');\n IF trim(token_where) = '' THEN\n token_where := NULL;\n END IF;\n RAISE NOTICE 'TOKEN_WHERE: |%|',token_where;\n RETURN token_where;\n END;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION search_tohash(jsonb) RETURNS jsonb AS $$\n SELECT $1 - '{token,limit,context,includes,excludes}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION search_hash(jsonb, jsonb) RETURNS text AS $$\n SELECT md5(concat(search_tohash($1)::text,$2::text));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TABLE IF NOT EXISTS searches(\n hash text GENERATED ALWAYS AS (search_hash(search, metadata)) STORED PRIMARY KEY,\n search jsonb NOT NULL,\n _where text,\n orderby text,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n metadata jsonb DEFAULT '{}'::jsonb NOT NULL\n);\nCREATE TABLE IF NOT EXISTS search_wheres(\n id bigint generated always as identity primary key,\n _where text NOT NULL,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n statslastupdated timestamptz,\n estimated_count bigint,\n estimated_cost float,\n time_to_estimate float,\n total_count bigint,\n time_to_count float,\n partitions text[]\n);\n\nCREATE INDEX IF NOT EXISTS search_wheres_partitions ON search_wheres USING GIN (partitions);\nCREATE UNIQUE INDEX IF NOT EXISTS search_wheres_where ON search_wheres ((md5(_where)));\n\nCREATE OR REPLACE FUNCTION where_stats(inwhere text, updatestats boolean default false, conf jsonb default null) RETURNS search_wheres AS $$\nDECLARE\n t timestamptz;\n i interval;\n explain_json jsonb;\n partitions text[];\n sw search_wheres%ROWTYPE;\n inwhere_hash text := md5(inwhere);\n _context text := lower(context(conf));\n _stats_ttl interval := context_stats_ttl(conf);\n _estimated_cost float := context_estimated_cost(conf);\n _estimated_count int := context_estimated_count(conf);\nBEGIN\n IF _context = 'off' THEN\n sw._where := inwhere;\n return sw;\n END IF;\n\n SELECT * INTO sw FROM search_wheres WHERE md5(_where)=inwhere_hash FOR UPDATE;\n\n -- Update statistics if explicitly set, if statistics do not exist, or statistics ttl has expired\n IF NOT updatestats THEN\n RAISE NOTICE 'Checking if update is needed for: % .', inwhere;\n RAISE NOTICE 'Stats Last Updated: %', sw.statslastupdated;\n RAISE NOTICE 'TTL: %, Age: %', _stats_ttl, now() - sw.statslastupdated;\n RAISE NOTICE 'Context: %, Existing Total: %', _context, sw.total_count;\n IF\n sw.statslastupdated IS NULL\n OR (now() - sw.statslastupdated) > _stats_ttl\n OR (context(conf) != 'off' AND sw.total_count IS NULL)\n THEN\n updatestats := TRUE;\n END IF;\n END IF;\n\n sw._where := inwhere;\n sw.lastused := now();\n sw.usecount := coalesce(sw.usecount,0) + 1;\n\n IF NOT updatestats THEN\n UPDATE search_wheres SET\n lastused = sw.lastused,\n usecount = sw.usecount\n WHERE md5(_where) = inwhere_hash\n RETURNING * INTO sw\n ;\n RETURN sw;\n END IF;\n\n -- Use explain to get estimated count/cost and a list of the partitions that would be hit by the query\n t := clock_timestamp();\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s', inwhere)\n INTO explain_json;\n RAISE NOTICE 'Time for just the explain: %', clock_timestamp() - t;\n i := clock_timestamp() - t;\n\n sw.statslastupdated := now();\n sw.estimated_count := explain_json->0->'Plan'->'Plan Rows';\n sw.estimated_cost := explain_json->0->'Plan'->'Total Cost';\n sw.time_to_estimate := extract(epoch from i);\n\n RAISE NOTICE 'ESTIMATED_COUNT: % < %', sw.estimated_count, _estimated_count;\n RAISE NOTICE 'ESTIMATED_COST: % < %', sw.estimated_cost, _estimated_cost;\n\n -- Do a full count of rows if context is set to on or if auto is set and estimates are low enough\n IF\n _context = 'on'\n OR\n ( _context = 'auto' AND\n (\n sw.estimated_count < _estimated_count\n AND\n sw.estimated_cost < _estimated_cost\n )\n )\n THEN\n t := clock_timestamp();\n RAISE NOTICE 'Calculating actual count...';\n EXECUTE format(\n 'SELECT count(*) FROM items WHERE %s',\n inwhere\n ) INTO sw.total_count;\n i := clock_timestamp() - t;\n RAISE NOTICE 'Actual Count: % -- %', sw.total_count, i;\n sw.time_to_count := extract(epoch FROM i);\n ELSE\n sw.total_count := NULL;\n sw.time_to_count := NULL;\n END IF;\n\n\n INSERT INTO search_wheres\n (_where, lastused, usecount, statslastupdated, estimated_count, estimated_cost, time_to_estimate, partitions, total_count, time_to_count)\n SELECT sw._where, sw.lastused, sw.usecount, sw.statslastupdated, sw.estimated_count, sw.estimated_cost, sw.time_to_estimate, sw.partitions, sw.total_count, sw.time_to_count\n ON CONFLICT ((md5(_where)))\n DO UPDATE\n SET\n lastused = sw.lastused,\n usecount = sw.usecount,\n statslastupdated = sw.statslastupdated,\n estimated_count = sw.estimated_count,\n estimated_cost = sw.estimated_cost,\n time_to_estimate = sw.time_to_estimate,\n total_count = sw.total_count,\n time_to_count = sw.time_to_count\n ;\n RETURN sw;\nEND;\n$$ LANGUAGE PLPGSQL ;\n\n\n\nDROP FUNCTION IF EXISTS search_query;\nCREATE OR REPLACE FUNCTION search_query(\n _search jsonb = '{}'::jsonb,\n updatestats boolean = false,\n _metadata jsonb = '{}'::jsonb\n) RETURNS searches AS $$\nDECLARE\n search searches%ROWTYPE;\n pexplain jsonb;\n t timestamptz;\n i interval;\nBEGIN\n SELECT * INTO search FROM searches\n WHERE hash=search_hash(_search, _metadata) FOR UPDATE;\n\n -- Calculate the where clause if not already calculated\n IF search._where IS NULL THEN\n search._where := stac_search_to_where(_search);\n END IF;\n\n -- Calculate the order by clause if not already calculated\n IF search.orderby IS NULL THEN\n search.orderby := sort_sqlorderby(_search);\n END IF;\n\n PERFORM where_stats(search._where, updatestats, _search->'conf');\n\n search.lastused := now();\n search.usecount := coalesce(search.usecount, 0) + 1;\n INSERT INTO searches (search, _where, orderby, lastused, usecount, metadata)\n VALUES (_search, search._where, search.orderby, search.lastused, search.usecount, _metadata)\n ON CONFLICT (hash) DO\n UPDATE SET\n _where = EXCLUDED._where,\n orderby = EXCLUDED.orderby,\n lastused = EXCLUDED.lastused,\n usecount = EXCLUDED.usecount,\n metadata = EXCLUDED.metadata\n RETURNING * INTO search\n ;\n RETURN search;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION search(_search jsonb = '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n searches searches%ROWTYPE;\n _where text;\n token_where text;\n full_where text;\n orderby text;\n query text;\n token_type text := substr(_search->>'token',1,4);\n _limit int := coalesce((_search->>'limit')::int, 10);\n curs refcursor;\n cntr int := 0;\n iter_record items%ROWTYPE;\n first_record jsonb;\n last_record jsonb;\n out_records jsonb := '[]'::jsonb;\n prev_query text;\n next text;\n prev_id text;\n has_next boolean := false;\n has_prev boolean := false;\n prev text;\n total_count bigint;\n context jsonb;\n collection jsonb;\n includes text[];\n excludes text[];\n exit_flag boolean := FALSE;\n batches int := 0;\n timer timestamptz := clock_timestamp();\n pstart timestamptz;\n pend timestamptz;\n pcurs refcursor;\n search_where search_wheres%ROWTYPE;\n id text;\nBEGIN\nCREATE TEMP TABLE results (content jsonb) ON COMMIT DROP;\n-- if ids is set, short circuit and just use direct ids query for each id\n-- skip any paging or caching\n-- hard codes ordering in the same order as the array of ids\nIF _search ? 'ids' THEN\n INSERT INTO results\n SELECT\n CASE WHEN _search->'conf'->>'nohydrate' IS NOT NULL AND (_search->'conf'->>'nohydrate')::boolean = true THEN\n content_nonhydrated(items, _search->'fields')\n ELSE\n content_hydrate(items, _search->'fields')\n END\n FROM items WHERE\n items.id = ANY(to_text_array(_search->'ids'))\n AND\n CASE WHEN _search ? 'collections' THEN\n items.collection = ANY(to_text_array(_search->'collections'))\n ELSE TRUE\n END\n ORDER BY items.datetime desc, items.id desc\n ;\n SELECT INTO total_count count(*) FROM results;\nELSE\n searches := search_query(_search);\n _where := searches._where;\n orderby := searches.orderby;\n search_where := where_stats(_where);\n total_count := coalesce(search_where.total_count, search_where.estimated_count);\n\n IF token_type='prev' THEN\n token_where := get_token_filter(_search, null::jsonb);\n orderby := sort_sqlorderby(_search, TRUE);\n END IF;\n IF token_type='next' THEN\n token_where := get_token_filter(_search, null::jsonb);\n END IF;\n\n full_where := concat_ws(' AND ', _where, token_where);\n RAISE NOTICE 'FULL QUERY % %', full_where, clock_timestamp()-timer;\n timer := clock_timestamp();\n\n FOR query IN SELECT partition_queries(full_where, orderby, search_where.partitions) LOOP\n timer := clock_timestamp();\n query := format('%s LIMIT %s', query, _limit + 1);\n RAISE NOTICE 'Partition Query: %', query;\n batches := batches + 1;\n -- curs = create_cursor(query);\n OPEN curs FOR EXECUTE query;\n LOOP\n FETCH curs into iter_record;\n EXIT WHEN NOT FOUND;\n cntr := cntr + 1;\n\n IF _search->'conf'->>'nohydrate' IS NOT NULL AND (_search->'conf'->>'nohydrate')::boolean = true THEN\n last_record := content_nonhydrated(iter_record, _search->'fields');\n ELSE\n last_record := content_hydrate(iter_record, _search->'fields');\n END IF;\n IF cntr = 1 THEN\n first_record := last_record;\n END IF;\n IF cntr <= _limit THEN\n INSERT INTO results (content) VALUES (last_record);\n ELSIF cntr > _limit THEN\n has_next := true;\n exit_flag := true;\n EXIT;\n END IF;\n END LOOP;\n CLOSE curs;\n RAISE NOTICE 'Query took %.', clock_timestamp()-timer;\n timer := clock_timestamp();\n EXIT WHEN exit_flag;\n END LOOP;\n RAISE NOTICE 'Scanned through % partitions.', batches;\nEND IF;\n\nSELECT jsonb_agg(content) INTO out_records FROM results WHERE content is not NULL;\n\nDROP TABLE results;\n\n\n-- Flip things around if this was the result of a prev token query\nIF token_type='prev' THEN\n out_records := flip_jsonb_array(out_records);\n first_record := last_record;\nEND IF;\n\n-- If this query has a token, see if there is data before the first record\nIF _search ? 'token' THEN\n prev_query := format(\n 'SELECT 1 FROM items WHERE %s LIMIT 1',\n concat_ws(\n ' AND ',\n _where,\n trim(get_token_filter(_search, to_jsonb(content_dehydrate(first_record))))\n )\n );\n RAISE NOTICE 'Query to get previous record: % --- %', prev_query, first_record;\n EXECUTE prev_query INTO has_prev;\n IF FOUND and has_prev IS NOT NULL THEN\n RAISE NOTICE 'Query results from prev query: %', has_prev;\n has_prev := TRUE;\n END IF;\nEND IF;\nhas_prev := COALESCE(has_prev, FALSE);\n\nIF has_prev THEN\n prev := out_records->0->>'id';\nEND IF;\nIF has_next OR token_type='prev' THEN\n next := out_records->-1->>'id';\nEND IF;\n\nIF context(_search->'conf') != 'off' THEN\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'matched', total_count,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\nELSE\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\nEND IF;\n\ncollection := jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb),\n 'next', next,\n 'prev', prev,\n 'context', context\n);\n\nRETURN collection;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER SET SEARCH_PATH TO pgstac, public;\n\n\nCREATE OR REPLACE FUNCTION search_cursor(_search jsonb = '{}'::jsonb) RETURNS refcursor AS $$\nDECLARE\n curs refcursor;\n searches searches%ROWTYPE;\n _where text;\n _orderby text;\n q text;\n\nBEGIN\n searches := search_query(_search);\n _where := searches._where;\n _orderby := searches.orderby;\n\n OPEN curs FOR\n WITH p AS (\n SELECT * FROM partition_queries(_where, _orderby) p\n )\n SELECT\n CASE WHEN EXISTS (SELECT 1 FROM p) THEN\n (SELECT format($q$\n SELECT * FROM (\n %s\n ) total\n $q$,\n string_agg(\n format($q$ SELECT * FROM ( %s ) AS sub $q$, p),\n '\n UNION ALL\n '\n )\n ))\n ELSE NULL\n END FROM p;\n RETURN curs;\nEND;\n$$ LANGUAGE PLPGSQL;\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION tileenvelope(zoom int, x int, y int) RETURNS geometry AS $$\nWITH t AS (\n SELECT\n 20037508.3427892 as merc_max,\n -20037508.3427892 as merc_min,\n (2 * 20037508.3427892) / (2 ^ zoom) as tile_size\n)\nSELECT st_makeenvelope(\n merc_min + (tile_size * x),\n merc_max - (tile_size * (y + 1)),\n merc_min + (tile_size * (x + 1)),\n merc_max - (tile_size * y),\n 3857\n) FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;DROP FUNCTION IF EXISTS mercgrid;\n\n\nCREATE OR REPLACE FUNCTION ftime() RETURNS interval as $$\nSELECT age(clock_timestamp(), transaction_timestamp());\n$$ LANGUAGE SQL;\nSET SEARCH_PATH to pgstac, public;\n\nDROP FUNCTION IF EXISTS geometrysearch;\nCREATE OR REPLACE FUNCTION geometrysearch(\n IN geom geometry,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE, -- Return as soon as the passed in geometry is full covered\n IN skipcovered boolean DEFAULT TRUE -- Skip any items that would show up completely under the previous items\n) RETURNS jsonb AS $$\nDECLARE\n search searches%ROWTYPE;\n curs refcursor;\n _where text;\n query text;\n iter_record items%ROWTYPE;\n out_records jsonb := '{}'::jsonb[];\n exit_flag boolean := FALSE;\n counter int := 1;\n scancounter int := 1;\n remaining_limit int := _scanlimit;\n tilearea float;\n unionedgeom geometry;\n clippedgeom geometry;\n unionedgeom_area float := 0;\n prev_area float := 0;\n excludes text[];\n includes text[];\n\nBEGIN\n DROP TABLE IF EXISTS pgstac_results;\n CREATE TEMP TABLE pgstac_results (content jsonb) ON COMMIT DROP;\n\n -- If skipcovered is true then you will always want to exit when the passed in geometry is full\n IF skipcovered THEN\n exitwhenfull := TRUE;\n END IF;\n\n SELECT * INTO search FROM searches WHERE hash=queryhash;\n\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Search with Query Hash % Not Found', queryhash;\n END IF;\n\n tilearea := st_area(geom);\n _where := format('%s AND st_intersects(geometry, %L::geometry)', search._where, geom);\n\n\n FOR query IN SELECT * FROM partition_queries(_where, search.orderby) LOOP\n query := format('%s LIMIT %L', query, remaining_limit);\n RAISE NOTICE '%', query;\n OPEN curs FOR EXECUTE query;\n LOOP\n FETCH curs INTO iter_record;\n EXIT WHEN NOT FOUND;\n IF exitwhenfull OR skipcovered THEN -- If we are not using exitwhenfull or skipcovered, we do not need to do expensive geometry operations\n clippedgeom := st_intersection(geom, iter_record.geometry);\n\n IF unionedgeom IS NULL THEN\n unionedgeom := clippedgeom;\n ELSE\n unionedgeom := st_union(unionedgeom, clippedgeom);\n END IF;\n\n unionedgeom_area := st_area(unionedgeom);\n\n IF skipcovered AND prev_area = unionedgeom_area THEN\n scancounter := scancounter + 1;\n CONTINUE;\n END IF;\n\n prev_area := unionedgeom_area;\n\n RAISE NOTICE '% % % %', unionedgeom_area/tilearea, counter, scancounter, ftime();\n END IF;\n RAISE NOTICE '% %', iter_record, content_hydrate(iter_record, fields);\n INSERT INTO pgstac_results (content) VALUES (content_hydrate(iter_record, fields));\n\n IF counter >= _limit\n OR scancounter > _scanlimit\n OR ftime() > _timelimit\n OR (exitwhenfull AND unionedgeom_area >= tilearea)\n THEN\n exit_flag := TRUE;\n EXIT;\n END IF;\n counter := counter + 1;\n scancounter := scancounter + 1;\n\n END LOOP;\n CLOSE curs;\n EXIT WHEN exit_flag;\n remaining_limit := _scanlimit - scancounter;\n END LOOP;\n\n SELECT jsonb_agg(content) INTO out_records FROM pgstac_results WHERE content IS NOT NULL;\n\n RETURN jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb)\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDROP FUNCTION IF EXISTS geojsonsearch;\nCREATE OR REPLACE FUNCTION geojsonsearch(\n IN geojson jsonb,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_geomfromgeojson(geojson),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS xyzsearch;\nCREATE OR REPLACE FUNCTION xyzsearch(\n IN _x int,\n IN _y int,\n IN _z int,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_transform(tileenvelope(_z, _x, _y), 4326),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nGRANT ALL ON SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac to pgstac_admin;\n\n-- pgstac_read role limited to using function apis\nGRANT EXECUTE ON FUNCTION search TO pgstac_read;\nGRANT EXECUTE ON FUNCTION search_query TO pgstac_read;\nGRANT EXECUTE ON FUNCTION item_by_id TO pgstac_read;\nGRANT EXECUTE ON FUNCTION get_item TO pgstac_read;\n\nGRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_ingest;\nGRANT USAGE ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_ingest;\nSELECT set_version('0.5.1');\n" }, { "path": "src/pgstac/migrations/pgstac.0.6.0-0.6.1.sql", "content": "SET SEARCH_PATH to pgstac, public;\nset check_function_bodies = off;\n\nCREATE OR REPLACE FUNCTION pgstac.to_text(jsonb)\n RETURNS text\n LANGUAGE sql\n IMMUTABLE STRICT\nAS $function$\n SELECT CASE WHEN jsonb_typeof($1) IN ('array','object') THEN $1::text ELSE $1->>0 END;\n$function$\n;\n\n\n\nSELECT set_version('0.6.1');\n" }, { "path": "src/pgstac/migrations/pgstac.0.6.0.sql", "content": "CREATE EXTENSION IF NOT EXISTS postgis;\nCREATE EXTENSION IF NOT EXISTS btree_gist;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_admin;\n CREATE ROLE pgstac_read;\n CREATE ROLE pgstac_ingest;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nGRANT pgstac_admin TO current_user;\n\nCREATE SCHEMA IF NOT EXISTS pgstac AUTHORIZATION pgstac_admin;\n\nALTER ROLE pgstac_admin SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_read SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_ingest SET SEARCH_PATH TO pgstac, public;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\n\nGRANT pgstac_read TO pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\n\nSET ROLE pgstac_admin;\n\nSET SEARCH_PATH TO pgstac, public;\n\n\nCREATE TABLE IF NOT EXISTS migrations (\n version text PRIMARY KEY,\n datetime timestamptz DEFAULT clock_timestamp() NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION get_version() RETURNS text AS $$\n SELECT version FROM pgstac.migrations ORDER BY datetime DESC, version DESC LIMIT 1;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION set_version(text) RETURNS text AS $$\n INSERT INTO pgstac.migrations (version) VALUES ($1)\n ON CONFLICT DO NOTHING\n RETURNING version;\n$$ LANGUAGE SQL;\n\n\nCREATE TABLE IF NOT EXISTS pgstac_settings (\n name text PRIMARY KEY,\n value text NOT NULL\n);\n\nINSERT INTO pgstac_settings (name, value) VALUES\n ('context', 'off'),\n ('context_estimated_count', '100000'),\n ('context_estimated_cost', '100000'),\n ('context_stats_ttl', '1 day'),\n ('default-filter-lang', 'cql2-json'),\n ('additional_properties', 'true')\nON CONFLICT DO NOTHING\n;\n\n\nCREATE OR REPLACE FUNCTION get_setting(IN _setting text, IN conf jsonb DEFAULT NULL) RETURNS text AS $$\nSELECT COALESCE(\n conf->>_setting,\n current_setting(concat('pgstac.',_setting), TRUE),\n (SELECT value FROM pgstac.pgstac_settings WHERE name=_setting)\n);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context(conf jsonb DEFAULT NULL) RETURNS text AS $$\n SELECT pgstac.get_setting('context', conf);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context_estimated_count(conf jsonb DEFAULT NULL) RETURNS int AS $$\n SELECT pgstac.get_setting('context_estimated_count', conf)::int;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_estimated_cost();\nCREATE OR REPLACE FUNCTION context_estimated_cost(conf jsonb DEFAULT NULL) RETURNS float AS $$\n SELECT pgstac.get_setting('context_estimated_cost', conf)::float;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_stats_ttl();\nCREATE OR REPLACE FUNCTION context_stats_ttl(conf jsonb DEFAULT NULL) RETURNS interval AS $$\n SELECT pgstac.get_setting('context_stats_ttl', conf)::interval;\n$$ LANGUAGE SQL;\n\n\nCREATE OR REPLACE FUNCTION notice(VARIADIC text[]) RETURNS boolean AS $$\nDECLARE\ndebug boolean := current_setting('pgstac.debug', true);\nBEGIN\n IF debug THEN\n RAISE NOTICE 'NOTICE FROM FUNC: % >>>>> %', concat_ws(' | ', $1), clock_timestamp();\n RETURN TRUE;\n END IF;\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION empty_arr(ANYARRAY) RETURNS BOOLEAN AS $$\nSELECT CASE\n WHEN $1 IS NULL THEN TRUE\n WHEN cardinality($1)<1 THEN TRUE\nELSE FALSE\nEND;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_intersection(_a ANYARRAY, _b ANYARRAY) RETURNS ANYARRAY AS $$\n SELECT ARRAY ( SELECT unnest(_a) INTERSECT SELECT UNNEST(_b) );\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION array_map_ident(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_ident(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_map_literal(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_literal(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION array_reverse(anyarray) RETURNS anyarray AS $$\nSELECT ARRAY(\n SELECT $1[i]\n FROM generate_subscripts($1,1) AS s(i)\n ORDER BY i DESC\n);\n$$ LANGUAGE SQL STRICT IMMUTABLE;\nCREATE OR REPLACE FUNCTION to_int(jsonb) RETURNS int AS $$\n SELECT floor(($1->>0)::float)::int;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION to_float(jsonb) RETURNS float AS $$\n SELECT ($1->>0)::float;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION to_tstz(jsonb) RETURNS timestamptz AS $$\n SELECT ($1->>0)::timestamptz;\n$$ LANGUAGE SQL IMMUTABLE STRICT SET TIME ZONE 'UTC';\n\n\nCREATE OR REPLACE FUNCTION to_text(jsonb) RETURNS text AS $$\n SELECT $1->>0;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION to_text_array(jsonb) RETURNS text[] AS $$\n SELECT\n CASE jsonb_typeof($1)\n WHEN 'array' THEN ARRAY(SELECT jsonb_array_elements_text($1))\n ELSE ARRAY[$1->>0]\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\nCREATE OR REPLACE FUNCTION bbox_geom(_bbox jsonb) RETURNS geometry AS $$\nSELECT CASE jsonb_array_length(_bbox)\n WHEN 4 THEN\n ST_SetSRID(ST_MakeEnvelope(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float,\n (_bbox->>3)::float\n ),4326)\n WHEN 6 THEN\n ST_SetSRID(ST_3DMakeBox(\n ST_MakePoint(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float\n ),\n ST_MakePoint(\n (_bbox->>3)::float,\n (_bbox->>4)::float,\n (_bbox->>5)::float\n )\n ),4326)\n ELSE null END;\n;\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION geom_bbox(_geom geometry) RETURNS jsonb AS $$\n SELECT jsonb_build_array(\n st_xmin(_geom),\n st_ymin(_geom),\n st_xmax(_geom),\n st_ymax(_geom)\n );\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION flip_jsonb_array(j jsonb) RETURNS jsonb AS $$\n SELECT jsonb_agg(value) FROM (SELECT value FROM jsonb_array_elements(j) WITH ORDINALITY ORDER BY ordinality DESC) as t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION explode_dotpaths(j jsonb) RETURNS SETOF text[] AS $$\n SELECT string_to_array(p, '.') as e FROM jsonb_array_elements_text(j) p;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION explode_dotpaths_recurse(IN j jsonb) RETURNS SETOF text[] AS $$\n WITH RECURSIVE t AS (\n SELECT e FROM explode_dotpaths(j) e\n UNION ALL\n SELECT e[1:cardinality(e)-1]\n FROM t\n WHERE cardinality(e)>1\n ) SELECT e FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION jsonb_set_nested(j jsonb, path text[], val jsonb) RETURNS jsonb AS $$\nDECLARE\nBEGIN\n IF cardinality(path) > 1 THEN\n FOR i IN 1..(cardinality(path)-1) LOOP\n IF j #> path[:i] IS NULL THEN\n j := jsonb_set_lax(j, path[:i], '{}', TRUE);\n END IF;\n END LOOP;\n END IF;\n RETURN jsonb_set_lax(j, path, val, true);\n\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\n\n\nCREATE OR REPLACE FUNCTION jsonb_include(j jsonb, f jsonb) RETURNS jsonb AS $$\nDECLARE\n includes jsonb := f-> 'include';\n outj jsonb := '{}'::jsonb;\n path text[];\nBEGIN\n IF\n includes IS NULL\n OR jsonb_array_length(includes) = 0\n THEN\n RETURN j;\n ELSE\n includes := includes || '[\"id\",\"collection\"]'::jsonb;\n FOR path IN SELECT explode_dotpaths(includes) LOOP\n outj := jsonb_set_nested(outj, path, j #> path);\n END LOOP;\n END IF;\n RETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_exclude(j jsonb, f jsonb) RETURNS jsonb AS $$\nDECLARE\n excludes jsonb := f-> 'exclude';\n outj jsonb := j;\n path text[];\nBEGIN\n IF\n excludes IS NULL\n OR jsonb_array_length(excludes) = 0\n THEN\n RETURN j;\n ELSE\n FOR path IN SELECT explode_dotpaths(excludes) LOOP\n outj := outj #- path;\n END LOOP;\n END IF;\n RETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_fields(j jsonb, f jsonb DEFAULT '{\"fields\":[]}') RETURNS jsonb AS $$\n SELECT jsonb_exclude(jsonb_include(j, f), f);\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION merge_jsonb(_a jsonb, _b jsonb) RETURNS jsonb AS $$\n SELECT\n CASE\n WHEN _a = '\"𒍟※\"'::jsonb THEN NULL\n WHEN _a IS NULL OR jsonb_typeof(_a) = 'null' THEN _b\n WHEN jsonb_typeof(_a) = 'object' AND jsonb_typeof(_b) = 'object' THEN\n (\n SELECT\n jsonb_strip_nulls(\n jsonb_object_agg(\n key,\n merge_jsonb(a.value, b.value)\n )\n )\n FROM\n jsonb_each(coalesce(_a,'{}'::jsonb)) as a\n FULL JOIN\n jsonb_each(coalesce(_b,'{}'::jsonb)) as b\n USING (key)\n )\n WHEN\n jsonb_typeof(_a) = 'array'\n AND jsonb_typeof(_b) = 'array'\n AND jsonb_array_length(_a) = jsonb_array_length(_b)\n THEN\n (\n SELECT jsonb_agg(m) FROM\n ( SELECT\n merge_jsonb(\n jsonb_array_elements(_a),\n jsonb_array_elements(_b)\n ) as m\n ) as l\n )\n ELSE _a\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION strip_jsonb(_a jsonb, _b jsonb) RETURNS jsonb AS $$\n SELECT\n CASE\n\n WHEN (_a IS NULL OR jsonb_typeof(_a) = 'null') AND _b IS NOT NULL AND jsonb_typeof(_b) != 'null' THEN '\"𒍟※\"'::jsonb\n WHEN _b IS NULL OR jsonb_typeof(_a) = 'null' THEN _a\n WHEN _a = _b AND jsonb_typeof(_a) = 'object' THEN '{}'::jsonb\n WHEN _a = _b THEN NULL\n WHEN jsonb_typeof(_a) = 'object' AND jsonb_typeof(_b) = 'object' THEN\n (\n SELECT\n jsonb_strip_nulls(\n jsonb_object_agg(\n key,\n strip_jsonb(a.value, b.value)\n )\n )\n FROM\n jsonb_each(_a) as a\n FULL JOIN\n jsonb_each(_b) as b\n USING (key)\n )\n WHEN\n jsonb_typeof(_a) = 'array'\n AND jsonb_typeof(_b) = 'array'\n AND jsonb_array_length(_a) = jsonb_array_length(_b)\n THEN\n (\n SELECT jsonb_agg(m) FROM\n ( SELECT\n strip_jsonb(\n jsonb_array_elements(_a),\n jsonb_array_elements(_b)\n ) as m\n ) as l\n )\n ELSE _a\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE;\n/* looks for a geometry in a stac item first from geometry and falling back to bbox */\nCREATE OR REPLACE FUNCTION stac_geom(value jsonb) RETURNS geometry AS $$\nSELECT\n CASE\n WHEN value ? 'intersects' THEN\n ST_GeomFromGeoJSON(value->>'intersects')\n WHEN value ? 'geometry' THEN\n ST_GeomFromGeoJSON(value->>'geometry')\n WHEN value ? 'bbox' THEN\n pgstac.bbox_geom(value->'bbox')\n ELSE NULL\n END as geometry\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION stac_daterange(\n value jsonb\n) RETURNS tstzrange AS $$\nDECLARE\n props jsonb := value;\n dt timestamptz;\n edt timestamptz;\nBEGIN\n IF props ? 'properties' THEN\n props := props->'properties';\n END IF;\n IF\n props ? 'start_datetime'\n AND props->>'start_datetime' IS NOT NULL\n AND props ? 'end_datetime'\n AND props->>'end_datetime' IS NOT NULL\n THEN\n dt := props->>'start_datetime';\n edt := props->>'end_datetime';\n IF dt > edt THEN\n RAISE EXCEPTION 'start_datetime must be < end_datetime';\n END IF;\n ELSE\n dt := props->>'datetime';\n edt := props->>'datetime';\n END IF;\n IF dt is NULL OR edt IS NULL THEN\n RAISE NOTICE 'DT: %, EDT: %', dt, edt;\n RAISE EXCEPTION 'Either datetime (%) or both start_datetime (%) and end_datetime (%) must be set.', props->>'datetime',props->>'start_datetime',props->>'end_datetime';\n END IF;\n RETURN tstzrange(dt, edt, '[]');\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_datetime(value jsonb) RETURNS timestamptz AS $$\n SELECT lower(stac_daterange(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_end_datetime(value jsonb) RETURNS timestamptz AS $$\n SELECT upper(stac_daterange(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\n\nCREATE TABLE IF NOT EXISTS stac_extensions(\n name text PRIMARY KEY,\n url text,\n enbabled_by_default boolean NOT NULL DEFAULT TRUE,\n enableable boolean NOT NULL DEFAULT TRUE\n);\n\nINSERT INTO stac_extensions (name, url) VALUES\n ('fields', 'https://api.stacspec.org/v1.0.0-beta.5/item-search#fields'),\n ('sort','https://api.stacspec.org/v1.0.0-beta.5/item-search#sort'),\n ('context','https://api.stacspec.org/v1.0.0-beta.5/item-search#context'),\n ('filter', 'https://api.stacspec.org/v1.0.0-beta.5/item-search#filter'),\n ('query', 'https://api.stacspec.org/v1.0.0-beta.5/item-search#query')\nON CONFLICT (name) DO UPDATE SET url=EXCLUDED.url;\n\n\n\nCREATE OR REPLACE FUNCTION collection_base_item(content jsonb) RETURNS jsonb AS $$\n SELECT jsonb_build_object(\n 'type', 'Feature',\n 'stac_version', content->'stac_version',\n 'assets', content->'item_assets',\n 'collection', content->'id'\n );\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TYPE partition_trunc_strategy AS ENUM ('year', 'month');\n\nCREATE TABLE IF NOT EXISTS collections (\n key bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,\n id text GENERATED ALWAYS AS (content->>'id') STORED UNIQUE,\n content JSONB NOT NULL,\n base_item jsonb GENERATED ALWAYS AS (pgstac.collection_base_item(content)) STORED,\n partition_trunc partition_trunc_strategy\n);\n\n\nCREATE OR REPLACE FUNCTION collection_base_item(cid text) RETURNS jsonb AS $$\n SELECT pgstac.collection_base_item(content) FROM pgstac.collections WHERE id = cid LIMIT 1;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION table_empty(text) RETURNS boolean AS $$\nDECLARE\n retval boolean;\nBEGIN\n EXECUTE format($q$\n SELECT NOT EXISTS (SELECT 1 FROM %I LIMIT 1)\n $q$,\n $1\n ) INTO retval;\n RETURN retval;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION collections_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n q text;\n partition_name text := format('_items_%s', NEW.key);\n partition_exists boolean := false;\n partition_empty boolean := true;\n err_context text;\n loadtemp boolean := FALSE;\nBEGIN\n RAISE NOTICE 'Collection Trigger. % %', NEW.id, NEW.key;\n SELECT relid::text INTO partition_name\n FROM pg_partition_tree('items')\n WHERE relid::text = partition_name;\n IF FOUND THEN\n partition_exists := true;\n partition_empty := table_empty(partition_name);\n ELSE\n partition_exists := false;\n partition_empty := true;\n partition_name := format('_items_%s', NEW.key);\n END IF;\n IF TG_OP = 'UPDATE' AND NEW.partition_trunc IS DISTINCT FROM OLD.partition_trunc AND partition_empty THEN\n q := format($q$\n DROP TABLE IF EXISTS %I CASCADE;\n $q$,\n partition_name\n );\n EXECUTE q;\n END IF;\n IF TG_OP = 'UPDATE' AND NEW.partition_trunc IS DISTINCT FROM OLD.partition_trunc AND partition_exists AND NOT partition_empty THEN\n q := format($q$\n CREATE TEMP TABLE changepartitionstaging ON COMMIT DROP AS SELECT * FROM %I;\n DROP TABLE IF EXISTS %I CASCADE;\n $q$,\n partition_name,\n partition_name\n );\n EXECUTE q;\n loadtemp := TRUE;\n partition_empty := TRUE;\n partition_exists := FALSE;\n END IF;\n IF TG_OP = 'UPDATE' AND NEW.partition_trunc IS NOT DISTINCT FROM OLD.partition_trunc THEN\n RETURN NEW;\n END IF;\n IF NEW.partition_trunc IS NULL AND partition_empty THEN\n RAISE NOTICE '% % % %',\n partition_name,\n NEW.id,\n concat(partition_name,'_id_idx'),\n partition_name\n ;\n q := format($q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n $q$,\n partition_name,\n NEW.id,\n concat(partition_name,'_id_idx'),\n partition_name\n );\n RAISE NOTICE 'q: %', q;\n BEGIN\n EXECUTE q;\n EXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', partition_name;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\n END;\n\n ALTER TABLE partitions DISABLE TRIGGER partitions_delete_trigger;\n DELETE FROM partitions WHERE collection=NEW.id AND name=partition_name;\n ALTER TABLE partitions ENABLE TRIGGER partitions_delete_trigger;\n\n INSERT INTO partitions (collection, name) VALUES (NEW.id, partition_name);\n ELSIF partition_empty THEN\n q := format($q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L)\n PARTITION BY RANGE (datetime);\n $q$,\n partition_name,\n NEW.id\n );\n RAISE NOTICE 'q: %', q;\n BEGIN\n EXECUTE q;\n EXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', partition_name;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\n END;\n ALTER TABLE partitions DISABLE TRIGGER partitions_delete_trigger;\n DELETE FROM partitions WHERE collection=NEW.id AND name=partition_name;\n ALTER TABLE partitions ENABLE TRIGGER partitions_delete_trigger;\n ELSE\n RAISE EXCEPTION 'Cannot modify partition % unless empty', partition_name;\n END IF;\n IF loadtemp THEN\n RAISE NOTICE 'Moving data into new partitions.';\n q := format($q$\n WITH p AS (\n SELECT\n collection,\n datetime as datetime,\n end_datetime as end_datetime,\n (partition_name(\n collection,\n datetime\n )).partition_name as name\n FROM changepartitionstaging\n )\n INSERT INTO partitions (collection, datetime_range, end_datetime_range)\n SELECT\n collection,\n tstzrange(min(datetime), max(datetime), '[]') as datetime_range,\n tstzrange(min(end_datetime), max(end_datetime), '[]') as end_datetime_range\n FROM p\n GROUP BY collection, name\n ON CONFLICT (name) DO UPDATE SET\n datetime_range = EXCLUDED.datetime_range,\n end_datetime_range = EXCLUDED.end_datetime_range\n ;\n INSERT INTO %I SELECT * FROM changepartitionstaging;\n DROP TABLE IF EXISTS changepartitionstaging;\n $q$,\n partition_name\n );\n EXECUTE q;\n END IF;\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE TRIGGER collections_trigger AFTER INSERT OR UPDATE ON collections FOR EACH ROW\nEXECUTE FUNCTION collections_trigger_func();\n\nCREATE OR REPLACE FUNCTION partition_collection(collection text, strategy partition_trunc_strategy) RETURNS text AS $$\n UPDATE collections SET partition_trunc=strategy WHERE id=collection RETURNING partition_trunc;\n$$ LANGUAGE SQL;\n\nCREATE TABLE IF NOT EXISTS partitions (\n collection text REFERENCES collections(id) ON DELETE CASCADE,\n name text PRIMARY KEY,\n partition_range tstzrange NOT NULL DEFAULT tstzrange('-infinity'::timestamptz,'infinity'::timestamptz, '[]'),\n datetime_range tstzrange,\n end_datetime_range tstzrange,\n CONSTRAINT prange EXCLUDE USING GIST (\n collection WITH =,\n partition_range WITH &&\n )\n) WITH (FILLFACTOR=90);\nCREATE INDEX partitions_range_idx ON partitions USING GIST(partition_range);\n\nCREATE OR REPLACE FUNCTION partitions_delete_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n q text;\nBEGIN\n RAISE NOTICE 'Partition Delete Trigger. %', OLD.name;\n EXECUTE format($q$\n DROP TABLE IF EXISTS %I CASCADE;\n $q$,\n OLD.name\n );\n RAISE NOTICE 'Dropped partition.';\n RETURN OLD;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER partitions_delete_trigger BEFORE DELETE ON partitions FOR EACH ROW\nEXECUTE FUNCTION partitions_delete_trigger_func();\n\nCREATE OR REPLACE FUNCTION partition_name(\n IN collection text,\n IN dt timestamptz,\n OUT partition_name text,\n OUT partition_range tstzrange\n) AS $$\nDECLARE\n c RECORD;\n parent_name text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n parent_name := format('_items_%s', c.key);\n\n\n IF c.partition_trunc = 'year' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYY'));\n ELSIF c.partition_trunc = 'month' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYYMM'));\n ELSE\n partition_name := parent_name;\n partition_range := tstzrange('-infinity'::timestamptz, 'infinity'::timestamptz, '[]');\n END IF;\n IF partition_range IS NULL THEN\n partition_range := tstzrange(\n date_trunc(c.partition_trunc::text, dt),\n date_trunc(c.partition_trunc::text, dt) + concat('1 ', c.partition_trunc)::interval\n );\n END IF;\n RETURN;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\n\nCREATE OR REPLACE FUNCTION partitions_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n q text;\n cq text;\n parent_name text;\n partition_trunc text;\n partition_name text := NEW.name;\n partition_exists boolean := false;\n partition_empty boolean := true;\n partition_range tstzrange;\n datetime_range tstzrange;\n end_datetime_range tstzrange;\n err_context text;\n mindt timestamptz := lower(NEW.datetime_range);\n maxdt timestamptz := upper(NEW.datetime_range);\n minedt timestamptz := lower(NEW.end_datetime_range);\n maxedt timestamptz := upper(NEW.end_datetime_range);\n t_mindt timestamptz;\n t_maxdt timestamptz;\n t_minedt timestamptz;\n t_maxedt timestamptz;\nBEGIN\n RAISE NOTICE 'Partitions Trigger. %', NEW;\n datetime_range := NEW.datetime_range;\n end_datetime_range := NEW.end_datetime_range;\n\n SELECT\n format('_items_%s', key),\n c.partition_trunc::text\n INTO\n parent_name,\n partition_trunc\n FROM pgstac.collections c\n WHERE c.id = NEW.collection;\n SELECT (pgstac.partition_name(NEW.collection, mindt)).* INTO partition_name, partition_range;\n NEW.name := partition_name;\n\n IF partition_range IS NULL OR partition_range = 'empty'::tstzrange THEN\n partition_range := tstzrange('-infinity'::timestamptz, 'infinity'::timestamptz, '[]');\n END IF;\n\n NEW.partition_range := partition_range;\n IF TG_OP = 'UPDATE' THEN\n mindt := least(mindt, lower(OLD.datetime_range));\n maxdt := greatest(maxdt, upper(OLD.datetime_range));\n minedt := least(minedt, lower(OLD.end_datetime_range));\n maxedt := greatest(maxedt, upper(OLD.end_datetime_range));\n NEW.datetime_range := tstzrange(mindt, maxdt, '[]');\n NEW.end_datetime_range := tstzrange(minedt, maxedt, '[]');\n END IF;\n IF TG_OP = 'INSERT' THEN\n\n IF partition_range != tstzrange('-infinity'::timestamptz, 'infinity'::timestamptz, '[]') THEN\n\n RAISE NOTICE '% % %', partition_name, parent_name, partition_range;\n q := format($q$\n CREATE TABLE IF NOT EXISTS %I partition OF %I FOR VALUES FROM (%L) TO (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n $q$,\n partition_name,\n parent_name,\n lower(partition_range),\n upper(partition_range),\n format('%s_pkey', partition_name),\n partition_name\n );\n BEGIN\n EXECUTE q;\n EXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', partition_name;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\n END;\n END IF;\n\n END IF;\n\n -- Update constraints\n EXECUTE format($q$\n SELECT\n min(datetime),\n max(datetime),\n min(end_datetime),\n max(end_datetime)\n FROM %I;\n $q$, partition_name)\n INTO t_mindt, t_maxdt, t_minedt, t_maxedt;\n mindt := least(mindt, t_mindt);\n maxdt := greatest(maxdt, t_maxdt);\n minedt := least(mindt, minedt, t_minedt);\n maxedt := greatest(maxdt, maxedt, t_maxedt);\n\n mindt := date_trunc(coalesce(partition_trunc, 'year'), mindt);\n maxdt := date_trunc(coalesce(partition_trunc, 'year'), maxdt - '1 second'::interval) + concat('1 ',coalesce(partition_trunc, 'year'))::interval;\n minedt := date_trunc(coalesce(partition_trunc, 'year'), minedt);\n maxedt := date_trunc(coalesce(partition_trunc, 'year'), maxedt - '1 second'::interval) + concat('1 ',coalesce(partition_trunc, 'year'))::interval;\n\n\n IF mindt IS NOT NULL AND maxdt IS NOT NULL AND minedt IS NOT NULL AND maxedt IS NOT NULL THEN\n NEW.datetime_range := tstzrange(mindt, maxdt, '[]');\n NEW.end_datetime_range := tstzrange(minedt, maxedt, '[]');\n IF\n TG_OP='UPDATE'\n AND OLD.datetime_range @> NEW.datetime_range\n AND OLD.end_datetime_range @> NEW.end_datetime_range\n THEN\n RAISE NOTICE 'Range unchanged, not updating constraints.';\n ELSE\n\n RAISE NOTICE '\n SETTING CONSTRAINTS\n mindt: %, maxdt: %\n minedt: %, maxedt: %\n ', mindt, maxdt, minedt, maxedt;\n IF partition_trunc IS NULL THEN\n cq := format($q$\n ALTER TABLE %7$I\n DROP CONSTRAINT IF EXISTS %1$I,\n DROP CONSTRAINT IF EXISTS %2$I,\n ADD CONSTRAINT %1$I\n CHECK (\n (datetime >= %3$L)\n AND (datetime <= %4$L)\n AND (end_datetime >= %5$L)\n AND (end_datetime <= %6$L)\n ) NOT VALID\n ;\n ALTER TABLE %7$I\n VALIDATE CONSTRAINT %1$I;\n $q$,\n format('%s_dt', partition_name),\n format('%s_edt', partition_name),\n mindt,\n maxdt,\n minedt,\n maxedt,\n partition_name\n );\n ELSE\n cq := format($q$\n ALTER TABLE %5$I\n DROP CONSTRAINT IF EXISTS %1$I,\n DROP CONSTRAINT IF EXISTS %2$I,\n ADD CONSTRAINT %2$I\n CHECK ((end_datetime >= %3$L) AND (end_datetime <= %4$L)) NOT VALID\n ;\n ALTER TABLE %5$I\n VALIDATE CONSTRAINT %2$I;\n $q$,\n format('%s_dt', partition_name),\n format('%s_edt', partition_name),\n minedt,\n maxedt,\n partition_name\n );\n\n END IF;\n RAISE NOTICE 'Altering Constraints. %', cq;\n EXECUTE cq;\n END IF;\n ELSE\n NEW.datetime_range = NULL;\n NEW.end_datetime_range = NULL;\n\n cq := format($q$\n ALTER TABLE %3$I\n DROP CONSTRAINT IF EXISTS %1$I,\n DROP CONSTRAINT IF EXISTS %2$I,\n ADD CONSTRAINT %1$I\n CHECK ((datetime IS NULL AND end_datetime IS NULL)) NOT VALID\n ;\n ALTER TABLE %3$I\n VALIDATE CONSTRAINT %1$I;\n $q$,\n format('%s_dt', partition_name),\n format('%s_edt', partition_name),\n partition_name\n );\n EXECUTE cq;\n END IF;\n\n RETURN NEW;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER partitions_trigger BEFORE INSERT OR UPDATE ON partitions FOR EACH ROW\nEXECUTE FUNCTION partitions_trigger_func();\n\n\nCREATE OR REPLACE FUNCTION create_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection(data jsonb) RETURNS VOID AS $$\nDECLARE\n out collections%ROWTYPE;\nBEGIN\n UPDATE collections SET content=data WHERE id = data->>'id' RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ON CONFLICT (id) DO\n UPDATE\n SET content=EXCLUDED.content\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION delete_collection(_id text) RETURNS VOID AS $$\nDECLARE\n out collections%ROWTYPE;\nBEGIN\n DELETE FROM collections WHERE id = _id RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION get_collection(id text) RETURNS jsonb AS $$\n SELECT content FROM collections\n WHERE id=$1\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION all_collections() RETURNS jsonb AS $$\n SELECT jsonb_agg(content) FROM collections;\n;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\nCREATE TABLE queryables (\n id bigint GENERATED ALWAYS AS identity PRIMARY KEY,\n name text UNIQUE NOT NULL,\n collection_ids text[], -- used to determine what partitions to create indexes on\n definition jsonb,\n property_path text,\n property_wrapper text,\n property_index_type text\n);\nCREATE INDEX queryables_name_idx ON queryables (name);\nCREATE INDEX queryables_property_wrapper_idx ON queryables (property_wrapper);\n\n\nINSERT INTO queryables (name, definition) VALUES\n('id', '{\"title\": \"Item ID\",\"description\": \"Item identifier\",\"$ref\": \"https://schemas.stacspec.org/v1.0.0/item-spec/json-schema/item.json#/definitions/core/allOf/2/properties/id\"}'),\n('datetime','{\"description\": \"Datetime\",\"type\": \"string\",\"title\": \"Acquired\",\"format\": \"date-time\",\"pattern\": \"(\\\\+00:00|Z)$\"}')\nON CONFLICT DO NOTHING;\n\n\n\nINSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n('eo:cloud_cover','{\"$ref\": \"https://stac-extensions.github.io/eo/v1.0.0/schema.json#/definitions/fieldsproperties/eo:cloud_cover\"}','to_int','BTREE')\nON CONFLICT DO NOTHING;\n\nCREATE OR REPLACE FUNCTION array_to_path(arr text[]) RETURNS text AS $$\n SELECT string_agg(\n quote_literal(v),\n '->'\n ) FROM unnest(arr) v;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\n\n\nCREATE OR REPLACE FUNCTION queryable(\n IN dotpath text,\n OUT path text,\n OUT expression text,\n OUT wrapper text\n) AS $$\nDECLARE\n q RECORD;\n path_elements text[];\nBEGIN\n IF dotpath IN ('id', 'geometry', 'datetime', 'end_datetime', 'collection') THEN\n path := dotpath;\n expression := dotpath;\n wrapper := NULL;\n RETURN;\n END IF;\n SELECT * INTO q FROM queryables WHERE name=dotpath;\n IF q.property_wrapper IS NULL THEN\n IF q.definition->>'type' = 'number' THEN\n wrapper := 'to_float';\n ELSIF q.definition->>'format' = 'date-time' THEN\n wrapper := 'to_tstz';\n ELSE\n wrapper := 'to_text';\n END IF;\n ELSE\n wrapper := q.property_wrapper;\n END IF;\n IF q.property_path IS NOT NULL THEN\n path := q.property_path;\n ELSE\n path_elements := string_to_array(dotpath, '.');\n IF path_elements[1] IN ('links', 'assets', 'stac_version', 'stac_extensions') THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSIF path_elements[1] = 'properties' THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSE\n path := format($F$content->'properties'->%s$F$, array_to_path(path_elements));\n END IF;\n END IF;\n expression := format('%I(%s)', wrapper, path);\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\nCREATE OR REPLACE FUNCTION create_queryable_indexes() RETURNS VOID AS $$\nDECLARE\n queryable RECORD;\n q text;\nBEGIN\n FOR queryable IN\n SELECT\n queryables.id as qid,\n CASE WHEN collections.key IS NULL THEN 'items' ELSE format('_items_%s',collections.key) END AS part,\n property_index_type,\n expression\n FROM\n queryables\n LEFT JOIN collections ON (collections.id = ANY (queryables.collection_ids))\n JOIN LATERAL queryable(queryables.name) ON (queryables.property_index_type IS NOT NULL)\n LOOP\n q := format(\n $q$\n CREATE INDEX IF NOT EXISTS %I ON %I USING %s ((%s));\n $q$,\n format('%s_%s_idx', queryable.part, queryable.qid),\n queryable.part,\n COALESCE(queryable.property_index_type, 'to_text'),\n queryable.expression\n );\n RAISE NOTICE '%',q;\n EXECUTE q;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION queryables_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\nBEGIN\nPERFORM create_queryable_indexes();\nRETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER queryables_trigger AFTER INSERT OR UPDATE ON queryables\nFOR EACH STATEMENT EXECUTE PROCEDURE queryables_trigger_func();\n\nCREATE TRIGGER queryables_collection_trigger AFTER INSERT OR UPDATE ON collections\nFOR EACH STATEMENT EXECUTE PROCEDURE queryables_trigger_func();\nCREATE OR REPLACE FUNCTION parse_dtrange(\n _indate jsonb,\n relative_base timestamptz DEFAULT date_trunc('hour', CURRENT_TIMESTAMP)\n) RETURNS tstzrange AS $$\nDECLARE\n timestrs text[];\n s timestamptz;\n e timestamptz;\nBEGIN\n timestrs :=\n CASE\n WHEN _indate ? 'timestamp' THEN\n ARRAY[_indate->>'timestamp']\n WHEN _indate ? 'interval' THEN\n to_text_array(_indate->'interval')\n WHEN jsonb_typeof(_indate) = 'array' THEN\n to_text_array(_indate)\n ELSE\n regexp_split_to_array(\n _indate->>0,\n '/'\n )\n END;\n RAISE NOTICE 'TIMESTRS %', timestrs;\n IF cardinality(timestrs) = 1 THEN\n IF timestrs[1] ILIKE 'P%' THEN\n RETURN tstzrange(relative_base - upper(timestrs[1])::interval, relative_base, '[)');\n END IF;\n s := timestrs[1]::timestamptz;\n RETURN tstzrange(s, s, '[]');\n END IF;\n\n IF cardinality(timestrs) != 2 THEN\n RAISE EXCEPTION 'Timestamp cannot have more than 2 values';\n END IF;\n\n IF timestrs[1] = '..' THEN\n s := '-infinity'::timestamptz;\n e := timestrs[2]::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] = '..' THEN\n s := timestrs[1]::timestamptz;\n e := 'infinity'::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[1] ILIKE 'P%' AND timestrs[2] NOT ILIKE 'P%' THEN\n e := timestrs[2]::timestamptz;\n s := e - upper(timestrs[1])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] ILIKE 'P%' AND timestrs[1] NOT ILIKE 'P%' THEN\n s := timestrs[1]::timestamptz;\n e := s + upper(timestrs[2])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n s := timestrs[1]::timestamptz;\n e := timestrs[2]::timestamptz;\n\n RETURN tstzrange(s,e,'[)');\n\n RETURN NULL;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT PARALLEL SAFE SET TIME ZONE 'UTC';\n\nCREATE OR REPLACE FUNCTION parse_dtrange(\n _indate text,\n relative_base timestamptz DEFAULT CURRENT_TIMESTAMP\n) RETURNS tstzrange AS $$\n SELECT parse_dtrange(to_jsonb(_indate), relative_base);\n$$ LANGUAGE SQL STABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION temporal_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\n ll text := 'datetime';\n lh text := 'end_datetime';\n rrange tstzrange;\n rl text;\n rh text;\n outq text;\nBEGIN\n rrange := parse_dtrange(args->1);\n RAISE NOTICE 'Constructing temporal query OP: %, ARGS: %, RRANGE: %', op, args, rrange;\n op := lower(op);\n rl := format('%L::timestamptz', lower(rrange));\n rh := format('%L::timestamptz', upper(rrange));\n outq := CASE op\n WHEN 't_before' THEN 'lh < rl'\n WHEN 't_after' THEN 'll > rh'\n WHEN 't_meets' THEN 'lh = rl'\n WHEN 't_metby' THEN 'll = rh'\n WHEN 't_overlaps' THEN 'll < rl AND rl < lh < rh'\n WHEN 't_overlappedby' THEN 'rl < ll < rh AND lh > rh'\n WHEN 't_starts' THEN 'll = rl AND lh < rh'\n WHEN 't_startedby' THEN 'll = rl AND lh > rh'\n WHEN 't_during' THEN 'll > rl AND lh < rh'\n WHEN 't_contains' THEN 'll < rl AND lh > rh'\n WHEN 't_finishes' THEN 'll > rl AND lh = rh'\n WHEN 't_finishedby' THEN 'll < rl AND lh = rh'\n WHEN 't_equals' THEN 'll = rl AND lh = rh'\n WHEN 't_disjoint' THEN 'NOT (ll <= rh AND lh >= rl)'\n WHEN 't_intersects' THEN 'll <= rh AND lh >= rl'\n WHEN 'anyinteracts' THEN 'll <= rh AND lh >= rl'\n END;\n outq := regexp_replace(outq, '\\mll\\M', ll);\n outq := regexp_replace(outq, '\\mlh\\M', lh);\n outq := regexp_replace(outq, '\\mrl\\M', rl);\n outq := regexp_replace(outq, '\\mrh\\M', rh);\n outq := format('(%s)', outq);\n RETURN outq;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\n\n\nCREATE OR REPLACE FUNCTION spatial_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\n geom text;\n j jsonb := args->1;\nBEGIN\n op := lower(op);\n RAISE NOTICE 'Constructing spatial query OP: %, ARGS: %', op, args;\n IF op NOT IN ('s_equals','s_disjoint','s_touches','s_within','s_overlaps','s_crosses','s_intersects','intersects','s_contains') THEN\n RAISE EXCEPTION 'Spatial Operator % Not Supported', op;\n END IF;\n op := regexp_replace(op, '^s_', 'st_');\n IF op = 'intersects' THEN\n op := 'st_intersects';\n END IF;\n -- Convert geometry to WKB string\n IF j ? 'type' AND j ? 'coordinates' THEN\n geom := st_geomfromgeojson(j)::text;\n ELSIF jsonb_typeof(j) = 'array' THEN\n geom := bbox_geom(j)::text;\n END IF;\n\n RETURN format('%s(geometry, %L::geometry)', op, geom);\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION query_to_cql2(q jsonb) RETURNS jsonb AS $$\n-- Translates anything passed in through the deprecated \"query\" into equivalent CQL2\nWITH t AS (\n SELECT key as property, value as ops\n FROM jsonb_each(q)\n), t2 AS (\n SELECT property, (jsonb_each(ops)).*\n FROM t WHERE jsonb_typeof(ops) = 'object'\n UNION ALL\n SELECT property, 'eq', ops\n FROM t WHERE jsonb_typeof(ops) != 'object'\n)\nSELECT\n jsonb_strip_nulls(jsonb_build_object(\n 'op', 'and',\n 'args', jsonb_agg(\n jsonb_build_object(\n 'op', key,\n 'args', jsonb_build_array(\n jsonb_build_object('property',property),\n value\n )\n )\n )\n )\n) as qcql FROM t2\n;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\nCREATE OR REPLACE FUNCTION cql1_to_cql2(j jsonb) RETURNS jsonb AS $$\nDECLARE\n args jsonb;\n ret jsonb;\nBEGIN\n RAISE NOTICE 'CQL1_TO_CQL2: %', j;\n IF j ? 'filter' THEN\n RETURN cql1_to_cql2(j->'filter');\n END IF;\n IF j ? 'property' THEN\n RETURN j;\n END IF;\n IF jsonb_typeof(j) = 'array' THEN\n SELECT jsonb_agg(cql1_to_cql2(el)) INTO args FROM jsonb_array_elements(j) el;\n RETURN args;\n END IF;\n IF jsonb_typeof(j) = 'number' THEN\n RETURN j;\n END IF;\n IF jsonb_typeof(j) = 'string' THEN\n RETURN j;\n END IF;\n\n IF jsonb_typeof(j) = 'object' THEN\n SELECT jsonb_build_object(\n 'op', key,\n 'args', cql1_to_cql2(value)\n ) INTO ret\n FROM jsonb_each(j)\n WHERE j IS NOT NULL;\n RETURN ret;\n END IF;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE STRICT;\n\nCREATE TABLE cql2_ops (\n op text PRIMARY KEY,\n template text,\n types text[]\n);\nINSERT INTO cql2_ops (op, template, types) VALUES\n ('eq', '%s = %s', NULL),\n ('lt', '%s < %s', NULL),\n ('lte', '%s <= %s', NULL),\n ('gt', '%s > %s', NULL),\n ('gte', '%s >= %s', NULL),\n ('le', '%s <= %s', NULL),\n ('ge', '%s >= %s', NULL),\n ('=', '%s = %s', NULL),\n ('<', '%s < %s', NULL),\n ('<=', '%s <= %s', NULL),\n ('>', '%s > %s', NULL),\n ('>=', '%s >= %s', NULL),\n ('like', '%s LIKE %s', NULL),\n ('ilike', '%s ILIKE %s', NULL),\n ('+', '%s + %s', NULL),\n ('-', '%s - %s', NULL),\n ('*', '%s * %s', NULL),\n ('/', '%s / %s', NULL),\n ('in', '%s = ANY (%s)', NULL),\n ('not', 'NOT (%s)', NULL),\n ('between', '%s BETWEEN (%2$s)[1] AND (%2$s)[2]', NULL),\n ('isnull', '%s IS NULL', NULL)\nON CONFLICT (op) DO UPDATE\n SET\n template = EXCLUDED.template;\n;\n\n\nCREATE OR REPLACE FUNCTION cql2_query(j jsonb, wrapper text DEFAULT NULL) RETURNS text AS $$\n#variable_conflict use_variable\nDECLARE\n args jsonb := j->'args';\n arg jsonb;\n op text := lower(j->>'op');\n cql2op RECORD;\n literal text;\nBEGIN\n IF j IS NULL OR (op IS NOT NULL AND args IS NULL) THEN\n RETURN NULL;\n END IF;\n RAISE NOTICE 'CQL2_QUERY: %', j;\n IF j ? 'filter' THEN\n RETURN cql2_query(j->'filter');\n END IF;\n\n IF j ? 'upper' THEN\n RETURN format('upper(%s)', cql2_query(j->'upper'));\n END IF;\n\n IF j ? 'lower' THEN\n RETURN format('lower(%s)', cql2_query(j->'lower'));\n END IF;\n\n -- Temporal Query\n IF op ilike 't_%' or op = 'anyinteracts' THEN\n RETURN temporal_op_query(op, args);\n END IF;\n\n -- If property is a timestamp convert it to text to use with\n -- general operators\n IF j ? 'timestamp' THEN\n RETURN format('%L::timestamptz', to_tstz(j->'timestamp'));\n END IF;\n IF j ? 'interval' THEN\n RAISE EXCEPTION 'Please use temporal operators when using intervals.';\n RETURN NONE;\n END IF;\n\n -- Spatial Query\n IF op ilike 's_%' or op = 'intersects' THEN\n RETURN spatial_op_query(op, args);\n END IF;\n\n\n IF op = 'in' THEN\n RETURN format(\n '%s = ANY (%L)',\n cql2_query(args->0),\n to_text_array(args->1)\n );\n END IF;\n\n\n\n IF op = 'between' THEN\n SELECT (queryable(a->>'property')).wrapper INTO wrapper\n FROM jsonb_array_elements(args) a\n WHERE a ? 'property' LIMIT 1;\n\n RETURN format(\n '%s BETWEEN %s and %s',\n cql2_query(args->0, wrapper),\n cql2_query(args->1->0, wrapper),\n cql2_query(args->1->1, wrapper)\n );\n END IF;\n\n -- Make sure that args is an array and run cql2_query on\n -- each element of the array\n RAISE NOTICE 'ARGS PRE: %', args;\n IF j ? 'args' THEN\n IF jsonb_typeof(args) != 'array' THEN\n args := jsonb_build_array(args);\n END IF;\n\n SELECT (queryable(a->>'property')).wrapper INTO wrapper\n FROM jsonb_array_elements(args) a\n WHERE a ? 'property' LIMIT 1;\n\n SELECT jsonb_agg(cql2_query(a, wrapper))\n INTO args\n FROM jsonb_array_elements(args) a;\n END IF;\n RAISE NOTICE 'ARGS: %', args;\n\n IF op IN ('and', 'or') THEN\n RETURN\n format(\n '(%s)',\n array_to_string(to_text_array(args), format(' %s ', upper(op)))\n );\n END IF;\n\n -- Look up template from cql2_ops\n IF j ? 'op' THEN\n SELECT * INTO cql2op FROM cql2_ops WHERE cql2_ops.op ilike op;\n IF FOUND THEN\n -- If specific index set in queryables for a property cast other arguments to that type\n RETURN format(\n cql2op.template,\n VARIADIC (to_text_array(args))\n );\n ELSE\n RAISE EXCEPTION 'Operator % Not Supported.', op;\n END IF;\n END IF;\n\n\n IF j ? 'property' THEN\n RETURN (queryable(j->>'property')).expression;\n END IF;\n\n IF wrapper IS NOT NULL THEN\n EXECUTE format('SELECT %I(%L)', wrapper, j) INTO literal;\n RAISE NOTICE '% % %',wrapper, j, literal;\n RETURN format('%I(%L)', wrapper, j);\n END IF;\n\n RETURN quote_literal(to_text(j));\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION paging_dtrange(\n j jsonb\n) RETURNS tstzrange AS $$\nDECLARE\n op text;\n filter jsonb := j->'filter';\n dtrange tstzrange := tstzrange('-infinity'::timestamptz,'infinity'::timestamptz);\n sdate timestamptz := '-infinity'::timestamptz;\n edate timestamptz := 'infinity'::timestamptz;\n jpitem jsonb;\nBEGIN\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n END IF;\n IF NOT (filter @? '$.**.op ? (@ == \"or\" || @ == \"not\")') THEN\n FOR jpitem IN SELECT j FROM jsonb_path_query(filter,'strict $.** ? (@.args[*].property == \"datetime\")'::jsonpath) j LOOP\n op := lower(jpitem->>'op');\n dtrange := parse_dtrange(jpitem->'args'->1);\n IF op IN ('<=', 'lt', 'lte', '<', 'le', 't_before') THEN\n sdate := greatest(sdate,'-infinity');\n edate := least(edate, upper(dtrange));\n ELSIF op IN ('>=', '>', 'gt', 'gte', 'ge', 't_after') THEN\n edate := least(edate, 'infinity');\n sdate := greatest(sdate, lower(dtrange));\n ELSIF op IN ('=', 'eq') THEN\n edate := least(edate, upper(dtrange));\n sdate := greatest(sdate, lower(dtrange));\n END IF;\n RAISE NOTICE '2 OP: %, ARGS: %, DTRANGE: %, SDATE: %, EDATE: %', op, jpitem->'args'->1, dtrange, sdate, edate;\n END LOOP;\n END IF;\n IF sdate > edate THEN\n RETURN 'empty'::tstzrange;\n END IF;\n RETURN tstzrange(sdate,edate, '[]');\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT SET TIME ZONE 'UTC';\n\nCREATE OR REPLACE FUNCTION paging_collections(\n IN j jsonb\n) RETURNS text[] AS $$\nDECLARE\n filter jsonb := j->'filter';\n jpitem jsonb;\n op text;\n args jsonb;\n arg jsonb;\n collections text[];\nBEGIN\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n END IF;\n IF NOT (filter @? '$.**.op ? (@ == \"or\" || @ == \"not\")') THEN\n FOR jpitem IN SELECT j FROM jsonb_path_query(filter,'strict $.** ? (@.args[*].property == \"collection\")'::jsonpath) j LOOP\n RAISE NOTICE 'JPITEM: %', jpitem;\n op := jpitem->>'op';\n args := jpitem->'args';\n IF op IN ('=', 'eq', 'in') THEN\n FOR arg IN SELECT a FROM jsonb_array_elements(args) a LOOP\n IF jsonb_typeof(arg) IN ('string', 'array') THEN\n RAISE NOTICE 'arg: %, collections: %', arg, collections;\n IF collections IS NULL OR collections = '{}'::text[] THEN\n collections := to_text_array(arg);\n ELSE\n collections := array_intersection(collections, to_text_array(arg));\n END IF;\n END IF;\n END LOOP;\n END IF;\n END LOOP;\n END IF;\n IF collections = '{}'::text[] THEN\n RETURN NULL;\n END IF;\n RETURN collections;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\nCREATE TABLE items (\n id text NOT NULL,\n geometry geometry NOT NULL,\n collection text NOT NULL,\n datetime timestamptz NOT NULL,\n end_datetime timestamptz NOT NULL,\n content JSONB NOT NULL\n)\nPARTITION BY LIST (collection)\n;\n\nCREATE INDEX \"datetime_idx\" ON items USING BTREE (datetime DESC, end_datetime ASC);\nCREATE INDEX \"geometry_idx\" ON items USING GIST (geometry);\n\nCREATE STATISTICS datetime_stats (dependencies) on datetime, end_datetime from items;\n\n\nALTER TABLE items ADD CONSTRAINT items_collections_fk FOREIGN KEY (collection) REFERENCES collections(id) ON DELETE CASCADE DEFERRABLE;\n\n\nCREATE OR REPLACE FUNCTION content_slim(_item jsonb) RETURNS jsonb AS $$\n SELECT strip_jsonb(_item - '{id,geometry,collection,type}'::text[], collection_base_item(_item->>'collection')) - '{id,geometry,collection,type}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_dehydrate(content jsonb) RETURNS items AS $$\n SELECT\n content->>'id' as id,\n stac_geom(content) as geometry,\n content->>'collection' as collection,\n stac_datetime(content) as datetime,\n stac_end_datetime(content) as end_datetime,\n content_slim(content) as content\n ;\n$$ LANGUAGE SQL STABLE;\n\nCREATE OR REPLACE FUNCTION include_field(f text, fields jsonb DEFAULT '{}'::jsonb) RETURNS boolean AS $$\nDECLARE\n includes jsonb := fields->'include';\n excludes jsonb := fields->'exclude';\nBEGIN\n IF f IS NULL THEN\n RETURN NULL;\n END IF;\n\n\n IF\n jsonb_typeof(excludes) = 'array'\n AND jsonb_array_length(excludes)>0\n AND excludes ? f\n THEN\n RETURN FALSE;\n END IF;\n\n IF\n (\n jsonb_typeof(includes) = 'array'\n AND jsonb_array_length(includes) > 0\n AND includes ? f\n ) OR\n (\n includes IS NULL\n OR jsonb_typeof(includes) = 'null'\n OR jsonb_array_length(includes) = 0\n )\n THEN\n RETURN TRUE;\n END IF;\n\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nDROP FUNCTION IF EXISTS content_hydrate(jsonb, jsonb, jsonb);\nCREATE OR REPLACE FUNCTION content_hydrate(\n _base_item jsonb,\n _item jsonb,\n fields jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\n SELECT merge_jsonb(\n jsonb_fields(_item, fields),\n jsonb_fields(_base_item, fields)\n );\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION content_hydrate(_item items, _collection collections, fields jsonb DEFAULT '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\n content jsonb;\n base_item jsonb := _collection.base_item;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry, 20)::jsonb;\n END IF;\n output := content_hydrate(\n jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'collection', _item.collection,\n 'type', 'Feature'\n ) || _item.content,\n _collection.base_item,\n fields\n );\n\n RETURN output;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_nonhydrated(\n _item items,\n fields jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry, 20)::jsonb;\n END IF;\n output := jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'collection', _item.collection,\n 'type', 'Feature'\n ) || _item.content;\n RETURN output;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_hydrate(_item items, fields jsonb DEFAULT '{}'::jsonb) RETURNS jsonb AS $$\n SELECT content_hydrate(\n _item,\n (SELECT c FROM collections c WHERE id=_item.collection LIMIT 1),\n fields\n );\n$$ LANGUAGE SQL STABLE;\n\n\nCREATE UNLOGGED TABLE items_staging (\n content JSONB NOT NULL\n);\nCREATE UNLOGGED TABLE items_staging_ignore (\n content JSONB NOT NULL\n);\nCREATE UNLOGGED TABLE items_staging_upsert (\n content JSONB NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION items_staging_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p record;\n _partitions text[];\n ts timestamptz := clock_timestamp();\nBEGIN\n RAISE NOTICE 'Creating Partitions. %', clock_timestamp() - ts;\n WITH ranges AS (\n SELECT\n n.content->>'collection' as collection,\n stac_daterange(n.content->'properties') as dtr\n FROM newdata n\n ), p AS (\n SELECT\n collection,\n lower(dtr) as datetime,\n upper(dtr) as end_datetime,\n (partition_name(\n collection,\n lower(dtr)\n )).partition_name as name\n FROM ranges\n )\n INSERT INTO partitions (collection, datetime_range, end_datetime_range)\n SELECT\n collection,\n tstzrange(min(datetime), max(datetime), '[]') as datetime_range,\n tstzrange(min(end_datetime), max(end_datetime), '[]') as end_datetime_range\n FROM p\n GROUP BY collection, name\n ON CONFLICT (name) DO UPDATE SET\n datetime_range = EXCLUDED.datetime_range,\n end_datetime_range = EXCLUDED.end_datetime_range\n ;\n\n RAISE NOTICE 'Doing the insert. %', clock_timestamp() - ts;\n IF TG_TABLE_NAME = 'items_staging' THEN\n INSERT INTO items\n SELECT\n (content_dehydrate(content)).*\n FROM newdata;\n DELETE FROM items_staging;\n ELSIF TG_TABLE_NAME = 'items_staging_ignore' THEN\n INSERT INTO items\n SELECT\n (content_dehydrate(content)).*\n FROM newdata\n ON CONFLICT DO NOTHING;\n DELETE FROM items_staging_ignore;\n ELSIF TG_TABLE_NAME = 'items_staging_upsert' THEN\n WITH staging_formatted AS (\n SELECT (content_dehydrate(content)).* FROM newdata\n ), deletes AS (\n DELETE FROM items i USING staging_formatted s\n WHERE\n i.id = s.id\n AND i.collection = s.collection\n AND i IS DISTINCT FROM s\n RETURNING i.id, i.collection\n )\n INSERT INTO items\n SELECT s.* FROM\n staging_formatted s\n JOIN deletes d\n USING (id, collection);\n DELETE FROM items_staging_upsert;\n END IF;\n RAISE NOTICE 'Done. %', clock_timestamp() - ts;\n\n RETURN NULL;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER items_staging_insert_trigger AFTER INSERT ON items_staging REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\nCREATE TRIGGER items_staging_insert_ignore_trigger AFTER INSERT ON items_staging_ignore REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\nCREATE TRIGGER items_staging_insert_upsert_trigger AFTER INSERT ON items_staging_upsert REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\n\n\n\nCREATE OR REPLACE FUNCTION item_by_id(_id text, _collection text DEFAULT NULL) RETURNS items AS\n$$\nDECLARE\n i items%ROWTYPE;\nBEGIN\n SELECT * INTO i FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection) LIMIT 1;\n RETURN i;\nEND;\n$$ LANGUAGE PLPGSQL STABLE SECURITY DEFINER SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION get_item(_id text, _collection text DEFAULT NULL) RETURNS jsonb AS $$\n SELECT content_hydrate(items) FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection);\n$$ LANGUAGE SQL STABLE SECURITY DEFINER SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION delete_item(_id text, _collection text DEFAULT NULL) RETURNS VOID AS $$\nDECLARE\nout items%ROWTYPE;\nBEGIN\n DELETE FROM items WHERE id = _id AND (_collection IS NULL OR collection=_collection) RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n--/*\nCREATE OR REPLACE FUNCTION create_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION update_item(content jsonb) RETURNS VOID AS $$\nDECLARE\n old items %ROWTYPE;\n out items%ROWTYPE;\nBEGIN\n PERFORM delete_item(content->>'id', content->>'collection');\n PERFORM create_item(content);\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION create_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION collection_bbox(id text) RETURNS jsonb AS $$\n SELECT (replace(replace(replace(st_extent(geometry)::text,'BOX(','[['),')',']]'),' ',','))::jsonb\n FROM items WHERE collection=$1;\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION collection_temporal_extent(id text) RETURNS jsonb AS $$\n SELECT to_jsonb(array[array[min(datetime)::text, max(datetime)::text]])\n FROM items WHERE collection=$1;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection_extents() RETURNS VOID AS $$\nUPDATE collections SET\n content = content ||\n jsonb_build_object(\n 'extent', jsonb_build_object(\n 'spatial', jsonb_build_object(\n 'bbox', collection_bbox(collections.id)\n ),\n 'temporal', jsonb_build_object(\n 'interval', collection_temporal_extent(collections.id)\n )\n )\n )\n;\n$$ LANGUAGE SQL;\nCREATE VIEW partition_steps AS\nSELECT\n name,\n date_trunc('month',lower(datetime_range)) as sdate,\n date_trunc('month', upper(datetime_range)) + '1 month'::interval as edate\n FROM partitions WHERE datetime_range IS NOT NULL AND datetime_range != 'empty'::tstzrange\n ORDER BY datetime_range ASC\n;\n\nCREATE OR REPLACE FUNCTION chunker(\n IN _where text,\n OUT s timestamptz,\n OUT e timestamptz\n) RETURNS SETOF RECORD AS $$\nDECLARE\n explain jsonb;\nBEGIN\n IF _where IS NULL THEN\n _where := ' TRUE ';\n END IF;\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s;', _where)\n INTO explain;\n\n RETURN QUERY\n WITH t AS (\n SELECT j->>0 as p FROM\n jsonb_path_query(\n explain,\n 'strict $.**.\"Relation Name\" ? (@ != null)'\n ) j\n ),\n parts AS (\n SELECT sdate, edate FROM t JOIN partition_steps ON (t.p = name)\n ),\n times AS (\n SELECT sdate FROM parts\n UNION\n SELECT edate FROM parts\n ),\n uniq AS (\n SELECT DISTINCT sdate FROM times ORDER BY sdate\n ),\n last AS (\n SELECT sdate, lead(sdate, 1) over () as edate FROM uniq\n )\n SELECT sdate, edate FROM last WHERE edate IS NOT NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION partition_queries(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN partitions text[] DEFAULT NULL\n) RETURNS SETOF text AS $$\nDECLARE\n query text;\n sdate timestamptz;\n edate timestamptz;\nBEGIN\nIF _where IS NULL OR trim(_where) = '' THEN\n _where = ' TRUE ';\nEND IF;\nRAISE NOTICE 'Getting chunks for % %', _where, _orderby;\nIF _orderby ILIKE 'datetime d%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 DESC LOOP\n RETURN NEXT format($q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n $q$,\n sdate,\n edate,\n _where,\n _orderby\n );\n END LOOP;\nELSIF _orderby ILIKE 'datetime a%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 ASC LOOP\n RETURN NEXT format($q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n $q$,\n sdate,\n edate,\n _where,\n _orderby\n );\n END LOOP;\nELSE\n query := format($q$\n SELECT * FROM items\n WHERE %s\n ORDER BY %s\n $q$, _where, _orderby\n );\n\n RETURN NEXT query;\n RETURN;\nEND IF;\n\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION partition_query_view(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN _limit int DEFAULT 10\n) RETURNS text AS $$\n WITH p AS (\n SELECT * FROM partition_queries(_where, _orderby) p\n )\n SELECT\n CASE WHEN EXISTS (SELECT 1 FROM p) THEN\n (SELECT format($q$\n SELECT * FROM (\n %s\n ) total LIMIT %s\n $q$,\n string_agg(\n format($q$ SELECT * FROM ( %s ) AS sub $q$, p),\n '\n UNION ALL\n '\n ),\n _limit\n ))\n ELSE NULL\n END FROM p;\n$$ LANGUAGE SQL IMMUTABLE;\n\n\n\n\nCREATE OR REPLACE FUNCTION stac_search_to_where(j jsonb) RETURNS text AS $$\nDECLARE\n where_segments text[];\n _where text;\n dtrange tstzrange;\n collections text[];\n geom geometry;\n sdate timestamptz;\n edate timestamptz;\n filterlang text;\n filter jsonb := j->'filter';\nBEGIN\n IF j ? 'ids' THEN\n where_segments := where_segments || format('id = ANY (%L) ', to_text_array(j->'ids'));\n END IF;\n\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n where_segments := where_segments || format('collection = ANY (%L) ', collections);\n END IF;\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n\n where_segments := where_segments || format(' datetime <= %L::timestamptz AND end_datetime >= %L::timestamptz ',\n edate,\n sdate\n );\n END IF;\n\n geom := stac_geom(j);\n IF geom IS NOT NULL THEN\n where_segments := where_segments || format('st_intersects(geometry, %L)',geom);\n END IF;\n\n filterlang := COALESCE(\n j->>'filter-lang',\n get_setting('default-filter-lang', j->'conf')\n );\n IF NOT filter @? '$.**.op' THEN\n filterlang := 'cql-json';\n END IF;\n\n IF filterlang NOT IN ('cql-json','cql2-json') AND j ? 'filter' THEN\n RAISE EXCEPTION '% is not a supported filter-lang. Please use cql-json or cql2-json.', filterlang;\n END IF;\n\n IF j ? 'query' AND j ? 'filter' THEN\n RAISE EXCEPTION 'Can only use either query or filter at one time.';\n END IF;\n\n IF j ? 'query' THEN\n filter := query_to_cql2(j->'query');\n ELSIF filterlang = 'cql-json' THEN\n filter := cql1_to_cql2(filter);\n END IF;\n RAISE NOTICE 'FILTER: %', filter;\n where_segments := where_segments || cql2_query(filter);\n IF cardinality(where_segments) < 1 THEN\n RETURN ' TRUE ';\n END IF;\n\n _where := array_to_string(array_remove(where_segments, NULL), ' AND ');\n\n IF _where IS NULL OR BTRIM(_where) = '' THEN\n RETURN ' TRUE ';\n END IF;\n RETURN _where;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION parse_sort_dir(_dir text, reverse boolean default false) RETURNS text AS $$\n WITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n ) SELECT\n CASE\n WHEN NOT reverse THEN d\n WHEN d = 'ASC' THEN 'DESC'\n WHEN d = 'DESC' THEN 'ASC'\n END\n FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION sort_dir_to_op(_dir text, prev boolean default false) RETURNS text AS $$\n WITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n ) SELECT\n CASE\n WHEN d = 'ASC' AND prev THEN '<='\n WHEN d = 'DESC' AND prev THEN '>='\n WHEN d = 'ASC' THEN '>='\n WHEN d = 'DESC' THEN '<='\n END\n FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION sort_sqlorderby(\n _search jsonb DEFAULT NULL,\n reverse boolean DEFAULT FALSE\n) RETURNS text AS $$\n WITH sortby AS (\n SELECT coalesce(_search->'sortby','[{\"field\":\"datetime\", \"direction\":\"desc\"}]') as sort\n ), withid AS (\n SELECT CASE\n WHEN sort @? '$[*] ? (@.field == \"id\")' THEN sort\n ELSE sort || '[{\"field\":\"id\", \"direction\":\"desc\"}]'::jsonb\n END as sort\n FROM sortby\n ), withid_rows AS (\n SELECT jsonb_array_elements(sort) as value FROM withid\n ),sorts AS (\n SELECT\n coalesce(\n -- field_orderby((items_path(value->>'field')).path_txt),\n (queryable(value->>'field')).expression\n ) as key,\n parse_sort_dir(value->>'direction', reverse) as dir\n FROM withid_rows\n )\n SELECT array_to_string(\n array_agg(concat(key, ' ', dir)),\n ', '\n ) FROM sorts;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_sort_dir(sort_item jsonb) RETURNS text AS $$\n SELECT CASE WHEN sort_item->>'direction' ILIKE 'desc%' THEN 'DESC' ELSE 'ASC' END;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION get_token_filter(_search jsonb = '{}'::jsonb, token_rec jsonb DEFAULT NULL) RETURNS text AS $$\nDECLARE\n token_id text;\n filters text[] := '{}'::text[];\n prev boolean := TRUE;\n field text;\n dir text;\n sort record;\n orfilters text[] := '{}'::text[];\n andfilters text[] := '{}'::text[];\n output text;\n token_where text;\nBEGIN\n RAISE NOTICE 'Getting Token Filter. % %', _search, token_rec;\n -- If no token provided return NULL\n IF token_rec IS NULL THEN\n IF NOT (_search ? 'token' AND\n (\n (_search->>'token' ILIKE 'prev:%')\n OR\n (_search->>'token' ILIKE 'next:%')\n )\n ) THEN\n RETURN NULL;\n END IF;\n prev := (_search->>'token' ILIKE 'prev:%');\n token_id := substr(_search->>'token', 6);\n SELECT to_jsonb(items) INTO token_rec\n FROM items WHERE id=token_id;\n END IF;\n RAISE NOTICE 'TOKEN ID: % %', token_rec, token_rec->'id';\n\n CREATE TEMP TABLE sorts (\n _row int GENERATED ALWAYS AS IDENTITY NOT NULL,\n _field text PRIMARY KEY,\n _dir text NOT NULL,\n _val text\n ) ON COMMIT DROP;\n\n -- Make sure we only have distinct columns to sort with taking the first one we get\n INSERT INTO sorts (_field, _dir)\n SELECT\n (queryable(value->>'field')).expression,\n get_sort_dir(value)\n FROM\n jsonb_array_elements(coalesce(_search->'sortby','[{\"field\":\"datetime\",\"direction\":\"desc\"}]'))\n ON CONFLICT DO NOTHING\n ;\n RAISE NOTICE 'sorts 1: %', (SELECT jsonb_agg(to_json(sorts)) FROM sorts);\n -- Get the first sort direction provided. As the id is a primary key, if there are any\n -- sorts after id they won't do anything, so make sure that id is the last sort item.\n SELECT _dir INTO dir FROM sorts ORDER BY _row ASC LIMIT 1;\n IF EXISTS (SELECT 1 FROM sorts WHERE _field = 'id') THEN\n DELETE FROM sorts WHERE _row > (SELECT _row FROM sorts WHERE _field = 'id' ORDER BY _row ASC);\n ELSE\n INSERT INTO sorts (_field, _dir) VALUES ('id', dir);\n END IF;\n\n -- Add value from looked up item to the sorts table\n UPDATE sorts SET _val=quote_literal(token_rec->>_field);\n\n -- Check if all sorts are the same direction and use row comparison\n -- to filter\n RAISE NOTICE 'sorts 2: %', (SELECT jsonb_agg(to_json(sorts)) FROM sorts);\n\n IF (SELECT count(DISTINCT _dir) FROM sorts) = 1 THEN\n SELECT format(\n '(%s) %s (%s)',\n concat_ws(', ', VARIADIC array_agg(quote_ident(_field))),\n CASE WHEN (prev AND dir = 'ASC') OR (NOT prev AND dir = 'DESC') THEN '<' ELSE '>' END,\n concat_ws(', ', VARIADIC array_agg(_val))\n ) INTO output FROM sorts\n WHERE token_rec ? _field\n ;\n ELSE\n FOR sort IN SELECT * FROM sorts ORDER BY _row asc LOOP\n RAISE NOTICE 'SORT: %', sort;\n IF sort._row = 1 THEN\n orfilters := orfilters || format('(%s %s %s)',\n quote_ident(sort._field),\n CASE WHEN (prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC') THEN '<' ELSE '>' END,\n sort._val\n );\n ELSE\n orfilters := orfilters || format('(%s AND %s %s %s)',\n array_to_string(andfilters, ' AND '),\n quote_ident(sort._field),\n CASE WHEN (prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC') THEN '<' ELSE '>' END,\n sort._val\n );\n\n END IF;\n andfilters := andfilters || format('%s = %s',\n quote_ident(sort._field),\n sort._val\n );\n END LOOP;\n output := array_to_string(orfilters, ' OR ');\n END IF;\n DROP TABLE IF EXISTS sorts;\n token_where := concat('(',coalesce(output,'true'),')');\n IF trim(token_where) = '' THEN\n token_where := NULL;\n END IF;\n RAISE NOTICE 'TOKEN_WHERE: |%|',token_where;\n RETURN token_where;\n END;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION search_tohash(jsonb) RETURNS jsonb AS $$\n SELECT $1 - '{token,limit,context,includes,excludes}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION search_hash(jsonb, jsonb) RETURNS text AS $$\n SELECT md5(concat(search_tohash($1)::text,$2::text));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TABLE IF NOT EXISTS searches(\n hash text GENERATED ALWAYS AS (search_hash(search, metadata)) STORED PRIMARY KEY,\n search jsonb NOT NULL,\n _where text,\n orderby text,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n metadata jsonb DEFAULT '{}'::jsonb NOT NULL\n);\nCREATE TABLE IF NOT EXISTS search_wheres(\n id bigint generated always as identity primary key,\n _where text NOT NULL,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n statslastupdated timestamptz,\n estimated_count bigint,\n estimated_cost float,\n time_to_estimate float,\n total_count bigint,\n time_to_count float,\n partitions text[]\n);\n\nCREATE INDEX IF NOT EXISTS search_wheres_partitions ON search_wheres USING GIN (partitions);\nCREATE UNIQUE INDEX IF NOT EXISTS search_wheres_where ON search_wheres ((md5(_where)));\n\nCREATE OR REPLACE FUNCTION where_stats(inwhere text, updatestats boolean default false, conf jsonb default null) RETURNS search_wheres AS $$\nDECLARE\n t timestamptz;\n i interval;\n explain_json jsonb;\n partitions text[];\n sw search_wheres%ROWTYPE;\n inwhere_hash text := md5(inwhere);\n _context text := lower(context(conf));\n _stats_ttl interval := context_stats_ttl(conf);\n _estimated_cost float := context_estimated_cost(conf);\n _estimated_count int := context_estimated_count(conf);\nBEGIN\n IF _context = 'off' THEN\n sw._where := inwhere;\n return sw;\n END IF;\n\n SELECT * INTO sw FROM search_wheres WHERE md5(_where)=inwhere_hash FOR UPDATE;\n\n -- Update statistics if explicitly set, if statistics do not exist, or statistics ttl has expired\n IF NOT updatestats THEN\n RAISE NOTICE 'Checking if update is needed for: % .', inwhere;\n RAISE NOTICE 'Stats Last Updated: %', sw.statslastupdated;\n RAISE NOTICE 'TTL: %, Age: %', _stats_ttl, now() - sw.statslastupdated;\n RAISE NOTICE 'Context: %, Existing Total: %', _context, sw.total_count;\n IF\n sw.statslastupdated IS NULL\n OR (now() - sw.statslastupdated) > _stats_ttl\n OR (context(conf) != 'off' AND sw.total_count IS NULL)\n THEN\n updatestats := TRUE;\n END IF;\n END IF;\n\n sw._where := inwhere;\n sw.lastused := now();\n sw.usecount := coalesce(sw.usecount,0) + 1;\n\n IF NOT updatestats THEN\n UPDATE search_wheres SET\n lastused = sw.lastused,\n usecount = sw.usecount\n WHERE md5(_where) = inwhere_hash\n RETURNING * INTO sw\n ;\n RETURN sw;\n END IF;\n\n -- Use explain to get estimated count/cost and a list of the partitions that would be hit by the query\n t := clock_timestamp();\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s', inwhere)\n INTO explain_json;\n RAISE NOTICE 'Time for just the explain: %', clock_timestamp() - t;\n i := clock_timestamp() - t;\n\n sw.statslastupdated := now();\n sw.estimated_count := explain_json->0->'Plan'->'Plan Rows';\n sw.estimated_cost := explain_json->0->'Plan'->'Total Cost';\n sw.time_to_estimate := extract(epoch from i);\n\n RAISE NOTICE 'ESTIMATED_COUNT: % < %', sw.estimated_count, _estimated_count;\n RAISE NOTICE 'ESTIMATED_COST: % < %', sw.estimated_cost, _estimated_cost;\n\n -- Do a full count of rows if context is set to on or if auto is set and estimates are low enough\n IF\n _context = 'on'\n OR\n ( _context = 'auto' AND\n (\n sw.estimated_count < _estimated_count\n AND\n sw.estimated_cost < _estimated_cost\n )\n )\n THEN\n t := clock_timestamp();\n RAISE NOTICE 'Calculating actual count...';\n EXECUTE format(\n 'SELECT count(*) FROM items WHERE %s',\n inwhere\n ) INTO sw.total_count;\n i := clock_timestamp() - t;\n RAISE NOTICE 'Actual Count: % -- %', sw.total_count, i;\n sw.time_to_count := extract(epoch FROM i);\n ELSE\n sw.total_count := NULL;\n sw.time_to_count := NULL;\n END IF;\n\n\n INSERT INTO search_wheres\n (_where, lastused, usecount, statslastupdated, estimated_count, estimated_cost, time_to_estimate, partitions, total_count, time_to_count)\n SELECT sw._where, sw.lastused, sw.usecount, sw.statslastupdated, sw.estimated_count, sw.estimated_cost, sw.time_to_estimate, sw.partitions, sw.total_count, sw.time_to_count\n ON CONFLICT ((md5(_where)))\n DO UPDATE\n SET\n lastused = sw.lastused,\n usecount = sw.usecount,\n statslastupdated = sw.statslastupdated,\n estimated_count = sw.estimated_count,\n estimated_cost = sw.estimated_cost,\n time_to_estimate = sw.time_to_estimate,\n total_count = sw.total_count,\n time_to_count = sw.time_to_count\n ;\n RETURN sw;\nEND;\n$$ LANGUAGE PLPGSQL ;\n\n\n\nDROP FUNCTION IF EXISTS search_query;\nCREATE OR REPLACE FUNCTION search_query(\n _search jsonb = '{}'::jsonb,\n updatestats boolean = false,\n _metadata jsonb = '{}'::jsonb\n) RETURNS searches AS $$\nDECLARE\n search searches%ROWTYPE;\n pexplain jsonb;\n t timestamptz;\n i interval;\nBEGIN\n SELECT * INTO search FROM searches\n WHERE hash=search_hash(_search, _metadata) FOR UPDATE;\n\n -- Calculate the where clause if not already calculated\n IF search._where IS NULL THEN\n search._where := stac_search_to_where(_search);\n END IF;\n\n -- Calculate the order by clause if not already calculated\n IF search.orderby IS NULL THEN\n search.orderby := sort_sqlorderby(_search);\n END IF;\n\n PERFORM where_stats(search._where, updatestats, _search->'conf');\n\n search.lastused := now();\n search.usecount := coalesce(search.usecount, 0) + 1;\n INSERT INTO searches (search, _where, orderby, lastused, usecount, metadata)\n VALUES (_search, search._where, search.orderby, search.lastused, search.usecount, _metadata)\n ON CONFLICT (hash) DO\n UPDATE SET\n _where = EXCLUDED._where,\n orderby = EXCLUDED.orderby,\n lastused = EXCLUDED.lastused,\n usecount = EXCLUDED.usecount,\n metadata = EXCLUDED.metadata\n RETURNING * INTO search\n ;\n RETURN search;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION search(_search jsonb = '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n searches searches%ROWTYPE;\n _where text;\n token_where text;\n full_where text;\n orderby text;\n query text;\n token_type text := substr(_search->>'token',1,4);\n _limit int := coalesce((_search->>'limit')::int, 10);\n curs refcursor;\n cntr int := 0;\n iter_record items%ROWTYPE;\n first_record jsonb;\n first_item items%ROWTYPE;\n last_item items%ROWTYPE;\n last_record jsonb;\n out_records jsonb := '[]'::jsonb;\n prev_query text;\n next text;\n prev_id text;\n has_next boolean := false;\n has_prev boolean := false;\n prev text;\n total_count bigint;\n context jsonb;\n collection jsonb;\n includes text[];\n excludes text[];\n exit_flag boolean := FALSE;\n batches int := 0;\n timer timestamptz := clock_timestamp();\n pstart timestamptz;\n pend timestamptz;\n pcurs refcursor;\n search_where search_wheres%ROWTYPE;\n id text;\nBEGIN\nCREATE TEMP TABLE results (content jsonb) ON COMMIT DROP;\n-- if ids is set, short circuit and just use direct ids query for each id\n-- skip any paging or caching\n-- hard codes ordering in the same order as the array of ids\nIF _search ? 'ids' THEN\n INSERT INTO results\n SELECT\n CASE WHEN _search->'conf'->>'nohydrate' IS NOT NULL AND (_search->'conf'->>'nohydrate')::boolean = true THEN\n content_nonhydrated(items, _search->'fields')\n ELSE\n content_hydrate(items, _search->'fields')\n END\n FROM items WHERE\n items.id = ANY(to_text_array(_search->'ids'))\n AND\n CASE WHEN _search ? 'collections' THEN\n items.collection = ANY(to_text_array(_search->'collections'))\n ELSE TRUE\n END\n ORDER BY items.datetime desc, items.id desc\n ;\n SELECT INTO total_count count(*) FROM results;\nELSE\n searches := search_query(_search);\n _where := searches._where;\n orderby := searches.orderby;\n search_where := where_stats(_where);\n total_count := coalesce(search_where.total_count, search_where.estimated_count);\n\n IF token_type='prev' THEN\n token_where := get_token_filter(_search, null::jsonb);\n orderby := sort_sqlorderby(_search, TRUE);\n END IF;\n IF token_type='next' THEN\n token_where := get_token_filter(_search, null::jsonb);\n END IF;\n\n full_where := concat_ws(' AND ', _where, token_where);\n RAISE NOTICE 'FULL QUERY % %', full_where, clock_timestamp()-timer;\n timer := clock_timestamp();\n\n FOR query IN SELECT partition_queries(full_where, orderby, search_where.partitions) LOOP\n timer := clock_timestamp();\n query := format('%s LIMIT %s', query, _limit + 1);\n RAISE NOTICE 'Partition Query: %', query;\n batches := batches + 1;\n -- curs = create_cursor(query);\n OPEN curs FOR EXECUTE query;\n LOOP\n FETCH curs into iter_record;\n EXIT WHEN NOT FOUND;\n cntr := cntr + 1;\n\n IF _search->'conf'->>'nohydrate' IS NOT NULL AND (_search->'conf'->>'nohydrate')::boolean = true THEN\n last_record := content_nonhydrated(iter_record, _search->'fields');\n ELSE\n last_record := content_hydrate(iter_record, _search->'fields');\n END IF;\n last_item := iter_record;\n IF cntr = 1 THEN\n first_item := last_item;\n first_record := last_record;\n END IF;\n IF cntr <= _limit THEN\n INSERT INTO results (content) VALUES (last_record);\n ELSIF cntr > _limit THEN\n has_next := true;\n exit_flag := true;\n EXIT;\n END IF;\n END LOOP;\n CLOSE curs;\n RAISE NOTICE 'Query took %.', clock_timestamp()-timer;\n timer := clock_timestamp();\n EXIT WHEN exit_flag;\n END LOOP;\n RAISE NOTICE 'Scanned through % partitions.', batches;\nEND IF;\n\nSELECT jsonb_agg(content) INTO out_records FROM results WHERE content is not NULL;\n\nDROP TABLE results;\n\n\n-- Flip things around if this was the result of a prev token query\nIF token_type='prev' THEN\n out_records := flip_jsonb_array(out_records);\n first_record := last_record;\nEND IF;\n\n-- If this query has a token, see if there is data before the first record\nIF _search ? 'token' THEN\n prev_query := format(\n 'SELECT 1 FROM items WHERE %s LIMIT 1',\n concat_ws(\n ' AND ',\n _where,\n trim(get_token_filter(_search, to_jsonb(first_item)))\n )\n );\n RAISE NOTICE 'Query to get previous record: % --- %', prev_query, first_record;\n EXECUTE prev_query INTO has_prev;\n IF FOUND and has_prev IS NOT NULL THEN\n RAISE NOTICE 'Query results from prev query: %', has_prev;\n has_prev := TRUE;\n END IF;\nEND IF;\nhas_prev := COALESCE(has_prev, FALSE);\n\nIF has_prev THEN\n prev := out_records->0->>'id';\nEND IF;\nIF has_next OR token_type='prev' THEN\n next := out_records->-1->>'id';\nEND IF;\n\nIF context(_search->'conf') != 'off' THEN\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'matched', total_count,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\nELSE\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\nEND IF;\n\ncollection := jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb),\n 'next', next,\n 'prev', prev,\n 'context', context\n);\n\nRETURN collection;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER SET SEARCH_PATH TO pgstac, public;\n\n\nCREATE OR REPLACE FUNCTION search_cursor(_search jsonb = '{}'::jsonb) RETURNS refcursor AS $$\nDECLARE\n curs refcursor;\n searches searches%ROWTYPE;\n _where text;\n _orderby text;\n q text;\n\nBEGIN\n searches := search_query(_search);\n _where := searches._where;\n _orderby := searches.orderby;\n\n OPEN curs FOR\n WITH p AS (\n SELECT * FROM partition_queries(_where, _orderby) p\n )\n SELECT\n CASE WHEN EXISTS (SELECT 1 FROM p) THEN\n (SELECT format($q$\n SELECT * FROM (\n %s\n ) total\n $q$,\n string_agg(\n format($q$ SELECT * FROM ( %s ) AS sub $q$, p),\n '\n UNION ALL\n '\n )\n ))\n ELSE NULL\n END FROM p;\n RETURN curs;\nEND;\n$$ LANGUAGE PLPGSQL;\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION tileenvelope(zoom int, x int, y int) RETURNS geometry AS $$\nWITH t AS (\n SELECT\n 20037508.3427892 as merc_max,\n -20037508.3427892 as merc_min,\n (2 * 20037508.3427892) / (2 ^ zoom) as tile_size\n)\nSELECT st_makeenvelope(\n merc_min + (tile_size * x),\n merc_max - (tile_size * (y + 1)),\n merc_min + (tile_size * (x + 1)),\n merc_max - (tile_size * y),\n 3857\n) FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;DROP FUNCTION IF EXISTS mercgrid;\n\n\nCREATE OR REPLACE FUNCTION ftime() RETURNS interval as $$\nSELECT age(clock_timestamp(), transaction_timestamp());\n$$ LANGUAGE SQL;\nSET SEARCH_PATH to pgstac, public;\n\nDROP FUNCTION IF EXISTS geometrysearch;\nCREATE OR REPLACE FUNCTION geometrysearch(\n IN geom geometry,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE, -- Return as soon as the passed in geometry is full covered\n IN skipcovered boolean DEFAULT TRUE -- Skip any items that would show up completely under the previous items\n) RETURNS jsonb AS $$\nDECLARE\n search searches%ROWTYPE;\n curs refcursor;\n _where text;\n query text;\n iter_record items%ROWTYPE;\n out_records jsonb := '{}'::jsonb[];\n exit_flag boolean := FALSE;\n counter int := 1;\n scancounter int := 1;\n remaining_limit int := _scanlimit;\n tilearea float;\n unionedgeom geometry;\n clippedgeom geometry;\n unionedgeom_area float := 0;\n prev_area float := 0;\n excludes text[];\n includes text[];\n\nBEGIN\n DROP TABLE IF EXISTS pgstac_results;\n CREATE TEMP TABLE pgstac_results (content jsonb) ON COMMIT DROP;\n\n -- If skipcovered is true then you will always want to exit when the passed in geometry is full\n IF skipcovered THEN\n exitwhenfull := TRUE;\n END IF;\n\n SELECT * INTO search FROM searches WHERE hash=queryhash;\n\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Search with Query Hash % Not Found', queryhash;\n END IF;\n\n tilearea := st_area(geom);\n _where := format('%s AND st_intersects(geometry, %L::geometry)', search._where, geom);\n\n\n FOR query IN SELECT * FROM partition_queries(_where, search.orderby) LOOP\n query := format('%s LIMIT %L', query, remaining_limit);\n RAISE NOTICE '%', query;\n OPEN curs FOR EXECUTE query;\n LOOP\n FETCH curs INTO iter_record;\n EXIT WHEN NOT FOUND;\n IF exitwhenfull OR skipcovered THEN -- If we are not using exitwhenfull or skipcovered, we do not need to do expensive geometry operations\n clippedgeom := st_intersection(geom, iter_record.geometry);\n\n IF unionedgeom IS NULL THEN\n unionedgeom := clippedgeom;\n ELSE\n unionedgeom := st_union(unionedgeom, clippedgeom);\n END IF;\n\n unionedgeom_area := st_area(unionedgeom);\n\n IF skipcovered AND prev_area = unionedgeom_area THEN\n scancounter := scancounter + 1;\n CONTINUE;\n END IF;\n\n prev_area := unionedgeom_area;\n\n RAISE NOTICE '% % % %', unionedgeom_area/tilearea, counter, scancounter, ftime();\n END IF;\n RAISE NOTICE '% %', iter_record, content_hydrate(iter_record, fields);\n INSERT INTO pgstac_results (content) VALUES (content_hydrate(iter_record, fields));\n\n IF counter >= _limit\n OR scancounter > _scanlimit\n OR ftime() > _timelimit\n OR (exitwhenfull AND unionedgeom_area >= tilearea)\n THEN\n exit_flag := TRUE;\n EXIT;\n END IF;\n counter := counter + 1;\n scancounter := scancounter + 1;\n\n END LOOP;\n CLOSE curs;\n EXIT WHEN exit_flag;\n remaining_limit := _scanlimit - scancounter;\n END LOOP;\n\n SELECT jsonb_agg(content) INTO out_records FROM pgstac_results WHERE content IS NOT NULL;\n\n RETURN jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb)\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDROP FUNCTION IF EXISTS geojsonsearch;\nCREATE OR REPLACE FUNCTION geojsonsearch(\n IN geojson jsonb,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_geomfromgeojson(geojson),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS xyzsearch;\nCREATE OR REPLACE FUNCTION xyzsearch(\n IN _x int,\n IN _y int,\n IN _z int,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_transform(tileenvelope(_z, _x, _y), 4326),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nGRANT ALL ON SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac to pgstac_admin;\n\n-- pgstac_read role limited to using function apis\nGRANT EXECUTE ON FUNCTION search TO pgstac_read;\nGRANT EXECUTE ON FUNCTION search_query TO pgstac_read;\nGRANT EXECUTE ON FUNCTION item_by_id TO pgstac_read;\nGRANT EXECUTE ON FUNCTION get_item TO pgstac_read;\n\nGRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_ingest;\nGRANT USAGE ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_ingest;\nSELECT set_version('0.6.0');\n" }, { "path": "src/pgstac/migrations/pgstac.0.6.1-0.6.2.sql", "content": "SET SEARCH_PATH to pgstac, public;\nSELECT set_version('0.6.2');\n" }, { "path": "src/pgstac/migrations/pgstac.0.6.1.sql", "content": "CREATE EXTENSION IF NOT EXISTS postgis;\nCREATE EXTENSION IF NOT EXISTS btree_gist;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_admin;\n CREATE ROLE pgstac_read;\n CREATE ROLE pgstac_ingest;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nGRANT pgstac_admin TO current_user;\n\nCREATE SCHEMA IF NOT EXISTS pgstac AUTHORIZATION pgstac_admin;\n\nALTER ROLE pgstac_admin SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_read SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_ingest SET SEARCH_PATH TO pgstac, public;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\n\nGRANT pgstac_read TO pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\n\nSET ROLE pgstac_admin;\n\nSET SEARCH_PATH TO pgstac, public;\n\n\nCREATE TABLE IF NOT EXISTS migrations (\n version text PRIMARY KEY,\n datetime timestamptz DEFAULT clock_timestamp() NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION get_version() RETURNS text AS $$\n SELECT version FROM pgstac.migrations ORDER BY datetime DESC, version DESC LIMIT 1;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION set_version(text) RETURNS text AS $$\n INSERT INTO pgstac.migrations (version) VALUES ($1)\n ON CONFLICT DO NOTHING\n RETURNING version;\n$$ LANGUAGE SQL;\n\n\nCREATE TABLE IF NOT EXISTS pgstac_settings (\n name text PRIMARY KEY,\n value text NOT NULL\n);\n\nINSERT INTO pgstac_settings (name, value) VALUES\n ('context', 'off'),\n ('context_estimated_count', '100000'),\n ('context_estimated_cost', '100000'),\n ('context_stats_ttl', '1 day'),\n ('default-filter-lang', 'cql2-json'),\n ('additional_properties', 'true')\nON CONFLICT DO NOTHING\n;\n\n\nCREATE OR REPLACE FUNCTION get_setting(IN _setting text, IN conf jsonb DEFAULT NULL) RETURNS text AS $$\nSELECT COALESCE(\n conf->>_setting,\n current_setting(concat('pgstac.',_setting), TRUE),\n (SELECT value FROM pgstac.pgstac_settings WHERE name=_setting)\n);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context(conf jsonb DEFAULT NULL) RETURNS text AS $$\n SELECT pgstac.get_setting('context', conf);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context_estimated_count(conf jsonb DEFAULT NULL) RETURNS int AS $$\n SELECT pgstac.get_setting('context_estimated_count', conf)::int;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_estimated_cost();\nCREATE OR REPLACE FUNCTION context_estimated_cost(conf jsonb DEFAULT NULL) RETURNS float AS $$\n SELECT pgstac.get_setting('context_estimated_cost', conf)::float;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_stats_ttl();\nCREATE OR REPLACE FUNCTION context_stats_ttl(conf jsonb DEFAULT NULL) RETURNS interval AS $$\n SELECT pgstac.get_setting('context_stats_ttl', conf)::interval;\n$$ LANGUAGE SQL;\n\n\nCREATE OR REPLACE FUNCTION notice(VARIADIC text[]) RETURNS boolean AS $$\nDECLARE\ndebug boolean := current_setting('pgstac.debug', true);\nBEGIN\n IF debug THEN\n RAISE NOTICE 'NOTICE FROM FUNC: % >>>>> %', concat_ws(' | ', $1), clock_timestamp();\n RETURN TRUE;\n END IF;\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION empty_arr(ANYARRAY) RETURNS BOOLEAN AS $$\nSELECT CASE\n WHEN $1 IS NULL THEN TRUE\n WHEN cardinality($1)<1 THEN TRUE\nELSE FALSE\nEND;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_intersection(_a ANYARRAY, _b ANYARRAY) RETURNS ANYARRAY AS $$\n SELECT ARRAY ( SELECT unnest(_a) INTERSECT SELECT UNNEST(_b) );\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION array_map_ident(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_ident(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_map_literal(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_literal(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION array_reverse(anyarray) RETURNS anyarray AS $$\nSELECT ARRAY(\n SELECT $1[i]\n FROM generate_subscripts($1,1) AS s(i)\n ORDER BY i DESC\n);\n$$ LANGUAGE SQL STRICT IMMUTABLE;\nCREATE OR REPLACE FUNCTION to_int(jsonb) RETURNS int AS $$\n SELECT floor(($1->>0)::float)::int;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION to_float(jsonb) RETURNS float AS $$\n SELECT ($1->>0)::float;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION to_tstz(jsonb) RETURNS timestamptz AS $$\n SELECT ($1->>0)::timestamptz;\n$$ LANGUAGE SQL IMMUTABLE STRICT SET TIME ZONE 'UTC';\n\n\nCREATE OR REPLACE FUNCTION to_text(jsonb) RETURNS text AS $$\n SELECT CASE WHEN jsonb_typeof($1) IN ('array','object') THEN $1::text ELSE $1->>0 END;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION to_text_array(jsonb) RETURNS text[] AS $$\n SELECT\n CASE jsonb_typeof($1)\n WHEN 'array' THEN ARRAY(SELECT jsonb_array_elements_text($1))\n ELSE ARRAY[$1->>0]\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\nCREATE OR REPLACE FUNCTION bbox_geom(_bbox jsonb) RETURNS geometry AS $$\nSELECT CASE jsonb_array_length(_bbox)\n WHEN 4 THEN\n ST_SetSRID(ST_MakeEnvelope(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float,\n (_bbox->>3)::float\n ),4326)\n WHEN 6 THEN\n ST_SetSRID(ST_3DMakeBox(\n ST_MakePoint(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float\n ),\n ST_MakePoint(\n (_bbox->>3)::float,\n (_bbox->>4)::float,\n (_bbox->>5)::float\n )\n ),4326)\n ELSE null END;\n;\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION geom_bbox(_geom geometry) RETURNS jsonb AS $$\n SELECT jsonb_build_array(\n st_xmin(_geom),\n st_ymin(_geom),\n st_xmax(_geom),\n st_ymax(_geom)\n );\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION flip_jsonb_array(j jsonb) RETURNS jsonb AS $$\n SELECT jsonb_agg(value) FROM (SELECT value FROM jsonb_array_elements(j) WITH ORDINALITY ORDER BY ordinality DESC) as t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION explode_dotpaths(j jsonb) RETURNS SETOF text[] AS $$\n SELECT string_to_array(p, '.') as e FROM jsonb_array_elements_text(j) p;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION explode_dotpaths_recurse(IN j jsonb) RETURNS SETOF text[] AS $$\n WITH RECURSIVE t AS (\n SELECT e FROM explode_dotpaths(j) e\n UNION ALL\n SELECT e[1:cardinality(e)-1]\n FROM t\n WHERE cardinality(e)>1\n ) SELECT e FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION jsonb_set_nested(j jsonb, path text[], val jsonb) RETURNS jsonb AS $$\nDECLARE\nBEGIN\n IF cardinality(path) > 1 THEN\n FOR i IN 1..(cardinality(path)-1) LOOP\n IF j #> path[:i] IS NULL THEN\n j := jsonb_set_lax(j, path[:i], '{}', TRUE);\n END IF;\n END LOOP;\n END IF;\n RETURN jsonb_set_lax(j, path, val, true);\n\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\n\n\nCREATE OR REPLACE FUNCTION jsonb_include(j jsonb, f jsonb) RETURNS jsonb AS $$\nDECLARE\n includes jsonb := f-> 'include';\n outj jsonb := '{}'::jsonb;\n path text[];\nBEGIN\n IF\n includes IS NULL\n OR jsonb_array_length(includes) = 0\n THEN\n RETURN j;\n ELSE\n includes := includes || '[\"id\",\"collection\"]'::jsonb;\n FOR path IN SELECT explode_dotpaths(includes) LOOP\n outj := jsonb_set_nested(outj, path, j #> path);\n END LOOP;\n END IF;\n RETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_exclude(j jsonb, f jsonb) RETURNS jsonb AS $$\nDECLARE\n excludes jsonb := f-> 'exclude';\n outj jsonb := j;\n path text[];\nBEGIN\n IF\n excludes IS NULL\n OR jsonb_array_length(excludes) = 0\n THEN\n RETURN j;\n ELSE\n FOR path IN SELECT explode_dotpaths(excludes) LOOP\n outj := outj #- path;\n END LOOP;\n END IF;\n RETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_fields(j jsonb, f jsonb DEFAULT '{\"fields\":[]}') RETURNS jsonb AS $$\n SELECT jsonb_exclude(jsonb_include(j, f), f);\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION merge_jsonb(_a jsonb, _b jsonb) RETURNS jsonb AS $$\n SELECT\n CASE\n WHEN _a = '\"𒍟※\"'::jsonb THEN NULL\n WHEN _a IS NULL OR jsonb_typeof(_a) = 'null' THEN _b\n WHEN jsonb_typeof(_a) = 'object' AND jsonb_typeof(_b) = 'object' THEN\n (\n SELECT\n jsonb_strip_nulls(\n jsonb_object_agg(\n key,\n merge_jsonb(a.value, b.value)\n )\n )\n FROM\n jsonb_each(coalesce(_a,'{}'::jsonb)) as a\n FULL JOIN\n jsonb_each(coalesce(_b,'{}'::jsonb)) as b\n USING (key)\n )\n WHEN\n jsonb_typeof(_a) = 'array'\n AND jsonb_typeof(_b) = 'array'\n AND jsonb_array_length(_a) = jsonb_array_length(_b)\n THEN\n (\n SELECT jsonb_agg(m) FROM\n ( SELECT\n merge_jsonb(\n jsonb_array_elements(_a),\n jsonb_array_elements(_b)\n ) as m\n ) as l\n )\n ELSE _a\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION strip_jsonb(_a jsonb, _b jsonb) RETURNS jsonb AS $$\n SELECT\n CASE\n\n WHEN (_a IS NULL OR jsonb_typeof(_a) = 'null') AND _b IS NOT NULL AND jsonb_typeof(_b) != 'null' THEN '\"𒍟※\"'::jsonb\n WHEN _b IS NULL OR jsonb_typeof(_a) = 'null' THEN _a\n WHEN _a = _b AND jsonb_typeof(_a) = 'object' THEN '{}'::jsonb\n WHEN _a = _b THEN NULL\n WHEN jsonb_typeof(_a) = 'object' AND jsonb_typeof(_b) = 'object' THEN\n (\n SELECT\n jsonb_strip_nulls(\n jsonb_object_agg(\n key,\n strip_jsonb(a.value, b.value)\n )\n )\n FROM\n jsonb_each(_a) as a\n FULL JOIN\n jsonb_each(_b) as b\n USING (key)\n )\n WHEN\n jsonb_typeof(_a) = 'array'\n AND jsonb_typeof(_b) = 'array'\n AND jsonb_array_length(_a) = jsonb_array_length(_b)\n THEN\n (\n SELECT jsonb_agg(m) FROM\n ( SELECT\n strip_jsonb(\n jsonb_array_elements(_a),\n jsonb_array_elements(_b)\n ) as m\n ) as l\n )\n ELSE _a\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE;\n/* looks for a geometry in a stac item first from geometry and falling back to bbox */\nCREATE OR REPLACE FUNCTION stac_geom(value jsonb) RETURNS geometry AS $$\nSELECT\n CASE\n WHEN value ? 'intersects' THEN\n ST_GeomFromGeoJSON(value->>'intersects')\n WHEN value ? 'geometry' THEN\n ST_GeomFromGeoJSON(value->>'geometry')\n WHEN value ? 'bbox' THEN\n pgstac.bbox_geom(value->'bbox')\n ELSE NULL\n END as geometry\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION stac_daterange(\n value jsonb\n) RETURNS tstzrange AS $$\nDECLARE\n props jsonb := value;\n dt timestamptz;\n edt timestamptz;\nBEGIN\n IF props ? 'properties' THEN\n props := props->'properties';\n END IF;\n IF\n props ? 'start_datetime'\n AND props->>'start_datetime' IS NOT NULL\n AND props ? 'end_datetime'\n AND props->>'end_datetime' IS NOT NULL\n THEN\n dt := props->>'start_datetime';\n edt := props->>'end_datetime';\n IF dt > edt THEN\n RAISE EXCEPTION 'start_datetime must be < end_datetime';\n END IF;\n ELSE\n dt := props->>'datetime';\n edt := props->>'datetime';\n END IF;\n IF dt is NULL OR edt IS NULL THEN\n RAISE NOTICE 'DT: %, EDT: %', dt, edt;\n RAISE EXCEPTION 'Either datetime (%) or both start_datetime (%) and end_datetime (%) must be set.', props->>'datetime',props->>'start_datetime',props->>'end_datetime';\n END IF;\n RETURN tstzrange(dt, edt, '[]');\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_datetime(value jsonb) RETURNS timestamptz AS $$\n SELECT lower(stac_daterange(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_end_datetime(value jsonb) RETURNS timestamptz AS $$\n SELECT upper(stac_daterange(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\n\nCREATE TABLE IF NOT EXISTS stac_extensions(\n name text PRIMARY KEY,\n url text,\n enbabled_by_default boolean NOT NULL DEFAULT TRUE,\n enableable boolean NOT NULL DEFAULT TRUE\n);\n\nINSERT INTO stac_extensions (name, url) VALUES\n ('fields', 'https://api.stacspec.org/v1.0.0-beta.5/item-search#fields'),\n ('sort','https://api.stacspec.org/v1.0.0-beta.5/item-search#sort'),\n ('context','https://api.stacspec.org/v1.0.0-beta.5/item-search#context'),\n ('filter', 'https://api.stacspec.org/v1.0.0-beta.5/item-search#filter'),\n ('query', 'https://api.stacspec.org/v1.0.0-beta.5/item-search#query')\nON CONFLICT (name) DO UPDATE SET url=EXCLUDED.url;\n\n\n\nCREATE OR REPLACE FUNCTION collection_base_item(content jsonb) RETURNS jsonb AS $$\n SELECT jsonb_build_object(\n 'type', 'Feature',\n 'stac_version', content->'stac_version',\n 'assets', content->'item_assets',\n 'collection', content->'id'\n );\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TYPE partition_trunc_strategy AS ENUM ('year', 'month');\n\nCREATE TABLE IF NOT EXISTS collections (\n key bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,\n id text GENERATED ALWAYS AS (content->>'id') STORED UNIQUE,\n content JSONB NOT NULL,\n base_item jsonb GENERATED ALWAYS AS (pgstac.collection_base_item(content)) STORED,\n partition_trunc partition_trunc_strategy\n);\n\n\nCREATE OR REPLACE FUNCTION collection_base_item(cid text) RETURNS jsonb AS $$\n SELECT pgstac.collection_base_item(content) FROM pgstac.collections WHERE id = cid LIMIT 1;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION table_empty(text) RETURNS boolean AS $$\nDECLARE\n retval boolean;\nBEGIN\n EXECUTE format($q$\n SELECT NOT EXISTS (SELECT 1 FROM %I LIMIT 1)\n $q$,\n $1\n ) INTO retval;\n RETURN retval;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION collections_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n q text;\n partition_name text := format('_items_%s', NEW.key);\n partition_exists boolean := false;\n partition_empty boolean := true;\n err_context text;\n loadtemp boolean := FALSE;\nBEGIN\n RAISE NOTICE 'Collection Trigger. % %', NEW.id, NEW.key;\n SELECT relid::text INTO partition_name\n FROM pg_partition_tree('items')\n WHERE relid::text = partition_name;\n IF FOUND THEN\n partition_exists := true;\n partition_empty := table_empty(partition_name);\n ELSE\n partition_exists := false;\n partition_empty := true;\n partition_name := format('_items_%s', NEW.key);\n END IF;\n IF TG_OP = 'UPDATE' AND NEW.partition_trunc IS DISTINCT FROM OLD.partition_trunc AND partition_empty THEN\n q := format($q$\n DROP TABLE IF EXISTS %I CASCADE;\n $q$,\n partition_name\n );\n EXECUTE q;\n END IF;\n IF TG_OP = 'UPDATE' AND NEW.partition_trunc IS DISTINCT FROM OLD.partition_trunc AND partition_exists AND NOT partition_empty THEN\n q := format($q$\n CREATE TEMP TABLE changepartitionstaging ON COMMIT DROP AS SELECT * FROM %I;\n DROP TABLE IF EXISTS %I CASCADE;\n $q$,\n partition_name,\n partition_name\n );\n EXECUTE q;\n loadtemp := TRUE;\n partition_empty := TRUE;\n partition_exists := FALSE;\n END IF;\n IF TG_OP = 'UPDATE' AND NEW.partition_trunc IS NOT DISTINCT FROM OLD.partition_trunc THEN\n RETURN NEW;\n END IF;\n IF NEW.partition_trunc IS NULL AND partition_empty THEN\n RAISE NOTICE '% % % %',\n partition_name,\n NEW.id,\n concat(partition_name,'_id_idx'),\n partition_name\n ;\n q := format($q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n $q$,\n partition_name,\n NEW.id,\n concat(partition_name,'_id_idx'),\n partition_name\n );\n RAISE NOTICE 'q: %', q;\n BEGIN\n EXECUTE q;\n EXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', partition_name;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\n END;\n\n ALTER TABLE partitions DISABLE TRIGGER partitions_delete_trigger;\n DELETE FROM partitions WHERE collection=NEW.id AND name=partition_name;\n ALTER TABLE partitions ENABLE TRIGGER partitions_delete_trigger;\n\n INSERT INTO partitions (collection, name) VALUES (NEW.id, partition_name);\n ELSIF partition_empty THEN\n q := format($q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L)\n PARTITION BY RANGE (datetime);\n $q$,\n partition_name,\n NEW.id\n );\n RAISE NOTICE 'q: %', q;\n BEGIN\n EXECUTE q;\n EXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', partition_name;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\n END;\n ALTER TABLE partitions DISABLE TRIGGER partitions_delete_trigger;\n DELETE FROM partitions WHERE collection=NEW.id AND name=partition_name;\n ALTER TABLE partitions ENABLE TRIGGER partitions_delete_trigger;\n ELSE\n RAISE EXCEPTION 'Cannot modify partition % unless empty', partition_name;\n END IF;\n IF loadtemp THEN\n RAISE NOTICE 'Moving data into new partitions.';\n q := format($q$\n WITH p AS (\n SELECT\n collection,\n datetime as datetime,\n end_datetime as end_datetime,\n (partition_name(\n collection,\n datetime\n )).partition_name as name\n FROM changepartitionstaging\n )\n INSERT INTO partitions (collection, datetime_range, end_datetime_range)\n SELECT\n collection,\n tstzrange(min(datetime), max(datetime), '[]') as datetime_range,\n tstzrange(min(end_datetime), max(end_datetime), '[]') as end_datetime_range\n FROM p\n GROUP BY collection, name\n ON CONFLICT (name) DO UPDATE SET\n datetime_range = EXCLUDED.datetime_range,\n end_datetime_range = EXCLUDED.end_datetime_range\n ;\n INSERT INTO %I SELECT * FROM changepartitionstaging;\n DROP TABLE IF EXISTS changepartitionstaging;\n $q$,\n partition_name\n );\n EXECUTE q;\n END IF;\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE TRIGGER collections_trigger AFTER INSERT OR UPDATE ON collections FOR EACH ROW\nEXECUTE FUNCTION collections_trigger_func();\n\nCREATE OR REPLACE FUNCTION partition_collection(collection text, strategy partition_trunc_strategy) RETURNS text AS $$\n UPDATE collections SET partition_trunc=strategy WHERE id=collection RETURNING partition_trunc;\n$$ LANGUAGE SQL;\n\nCREATE TABLE IF NOT EXISTS partitions (\n collection text REFERENCES collections(id) ON DELETE CASCADE,\n name text PRIMARY KEY,\n partition_range tstzrange NOT NULL DEFAULT tstzrange('-infinity'::timestamptz,'infinity'::timestamptz, '[]'),\n datetime_range tstzrange,\n end_datetime_range tstzrange,\n CONSTRAINT prange EXCLUDE USING GIST (\n collection WITH =,\n partition_range WITH &&\n )\n) WITH (FILLFACTOR=90);\nCREATE INDEX partitions_range_idx ON partitions USING GIST(partition_range);\n\nCREATE OR REPLACE FUNCTION partitions_delete_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n q text;\nBEGIN\n RAISE NOTICE 'Partition Delete Trigger. %', OLD.name;\n EXECUTE format($q$\n DROP TABLE IF EXISTS %I CASCADE;\n $q$,\n OLD.name\n );\n RAISE NOTICE 'Dropped partition.';\n RETURN OLD;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER partitions_delete_trigger BEFORE DELETE ON partitions FOR EACH ROW\nEXECUTE FUNCTION partitions_delete_trigger_func();\n\nCREATE OR REPLACE FUNCTION partition_name(\n IN collection text,\n IN dt timestamptz,\n OUT partition_name text,\n OUT partition_range tstzrange\n) AS $$\nDECLARE\n c RECORD;\n parent_name text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n parent_name := format('_items_%s', c.key);\n\n\n IF c.partition_trunc = 'year' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYY'));\n ELSIF c.partition_trunc = 'month' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYYMM'));\n ELSE\n partition_name := parent_name;\n partition_range := tstzrange('-infinity'::timestamptz, 'infinity'::timestamptz, '[]');\n END IF;\n IF partition_range IS NULL THEN\n partition_range := tstzrange(\n date_trunc(c.partition_trunc::text, dt),\n date_trunc(c.partition_trunc::text, dt) + concat('1 ', c.partition_trunc)::interval\n );\n END IF;\n RETURN;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\n\nCREATE OR REPLACE FUNCTION partitions_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n q text;\n cq text;\n parent_name text;\n partition_trunc text;\n partition_name text := NEW.name;\n partition_exists boolean := false;\n partition_empty boolean := true;\n partition_range tstzrange;\n datetime_range tstzrange;\n end_datetime_range tstzrange;\n err_context text;\n mindt timestamptz := lower(NEW.datetime_range);\n maxdt timestamptz := upper(NEW.datetime_range);\n minedt timestamptz := lower(NEW.end_datetime_range);\n maxedt timestamptz := upper(NEW.end_datetime_range);\n t_mindt timestamptz;\n t_maxdt timestamptz;\n t_minedt timestamptz;\n t_maxedt timestamptz;\nBEGIN\n RAISE NOTICE 'Partitions Trigger. %', NEW;\n datetime_range := NEW.datetime_range;\n end_datetime_range := NEW.end_datetime_range;\n\n SELECT\n format('_items_%s', key),\n c.partition_trunc::text\n INTO\n parent_name,\n partition_trunc\n FROM pgstac.collections c\n WHERE c.id = NEW.collection;\n SELECT (pgstac.partition_name(NEW.collection, mindt)).* INTO partition_name, partition_range;\n NEW.name := partition_name;\n\n IF partition_range IS NULL OR partition_range = 'empty'::tstzrange THEN\n partition_range := tstzrange('-infinity'::timestamptz, 'infinity'::timestamptz, '[]');\n END IF;\n\n NEW.partition_range := partition_range;\n IF TG_OP = 'UPDATE' THEN\n mindt := least(mindt, lower(OLD.datetime_range));\n maxdt := greatest(maxdt, upper(OLD.datetime_range));\n minedt := least(minedt, lower(OLD.end_datetime_range));\n maxedt := greatest(maxedt, upper(OLD.end_datetime_range));\n NEW.datetime_range := tstzrange(mindt, maxdt, '[]');\n NEW.end_datetime_range := tstzrange(minedt, maxedt, '[]');\n END IF;\n IF TG_OP = 'INSERT' THEN\n\n IF partition_range != tstzrange('-infinity'::timestamptz, 'infinity'::timestamptz, '[]') THEN\n\n RAISE NOTICE '% % %', partition_name, parent_name, partition_range;\n q := format($q$\n CREATE TABLE IF NOT EXISTS %I partition OF %I FOR VALUES FROM (%L) TO (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n $q$,\n partition_name,\n parent_name,\n lower(partition_range),\n upper(partition_range),\n format('%s_pkey', partition_name),\n partition_name\n );\n BEGIN\n EXECUTE q;\n EXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', partition_name;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\n END;\n END IF;\n\n END IF;\n\n -- Update constraints\n EXECUTE format($q$\n SELECT\n min(datetime),\n max(datetime),\n min(end_datetime),\n max(end_datetime)\n FROM %I;\n $q$, partition_name)\n INTO t_mindt, t_maxdt, t_minedt, t_maxedt;\n mindt := least(mindt, t_mindt);\n maxdt := greatest(maxdt, t_maxdt);\n minedt := least(mindt, minedt, t_minedt);\n maxedt := greatest(maxdt, maxedt, t_maxedt);\n\n mindt := date_trunc(coalesce(partition_trunc, 'year'), mindt);\n maxdt := date_trunc(coalesce(partition_trunc, 'year'), maxdt - '1 second'::interval) + concat('1 ',coalesce(partition_trunc, 'year'))::interval;\n minedt := date_trunc(coalesce(partition_trunc, 'year'), minedt);\n maxedt := date_trunc(coalesce(partition_trunc, 'year'), maxedt - '1 second'::interval) + concat('1 ',coalesce(partition_trunc, 'year'))::interval;\n\n\n IF mindt IS NOT NULL AND maxdt IS NOT NULL AND minedt IS NOT NULL AND maxedt IS NOT NULL THEN\n NEW.datetime_range := tstzrange(mindt, maxdt, '[]');\n NEW.end_datetime_range := tstzrange(minedt, maxedt, '[]');\n IF\n TG_OP='UPDATE'\n AND OLD.datetime_range @> NEW.datetime_range\n AND OLD.end_datetime_range @> NEW.end_datetime_range\n THEN\n RAISE NOTICE 'Range unchanged, not updating constraints.';\n ELSE\n\n RAISE NOTICE '\n SETTING CONSTRAINTS\n mindt: %, maxdt: %\n minedt: %, maxedt: %\n ', mindt, maxdt, minedt, maxedt;\n IF partition_trunc IS NULL THEN\n cq := format($q$\n ALTER TABLE %7$I\n DROP CONSTRAINT IF EXISTS %1$I,\n DROP CONSTRAINT IF EXISTS %2$I,\n ADD CONSTRAINT %1$I\n CHECK (\n (datetime >= %3$L)\n AND (datetime <= %4$L)\n AND (end_datetime >= %5$L)\n AND (end_datetime <= %6$L)\n ) NOT VALID\n ;\n ALTER TABLE %7$I\n VALIDATE CONSTRAINT %1$I;\n $q$,\n format('%s_dt', partition_name),\n format('%s_edt', partition_name),\n mindt,\n maxdt,\n minedt,\n maxedt,\n partition_name\n );\n ELSE\n cq := format($q$\n ALTER TABLE %5$I\n DROP CONSTRAINT IF EXISTS %1$I,\n DROP CONSTRAINT IF EXISTS %2$I,\n ADD CONSTRAINT %2$I\n CHECK ((end_datetime >= %3$L) AND (end_datetime <= %4$L)) NOT VALID\n ;\n ALTER TABLE %5$I\n VALIDATE CONSTRAINT %2$I;\n $q$,\n format('%s_dt', partition_name),\n format('%s_edt', partition_name),\n minedt,\n maxedt,\n partition_name\n );\n\n END IF;\n RAISE NOTICE 'Altering Constraints. %', cq;\n EXECUTE cq;\n END IF;\n ELSE\n NEW.datetime_range = NULL;\n NEW.end_datetime_range = NULL;\n\n cq := format($q$\n ALTER TABLE %3$I\n DROP CONSTRAINT IF EXISTS %1$I,\n DROP CONSTRAINT IF EXISTS %2$I,\n ADD CONSTRAINT %1$I\n CHECK ((datetime IS NULL AND end_datetime IS NULL)) NOT VALID\n ;\n ALTER TABLE %3$I\n VALIDATE CONSTRAINT %1$I;\n $q$,\n format('%s_dt', partition_name),\n format('%s_edt', partition_name),\n partition_name\n );\n EXECUTE cq;\n END IF;\n\n RETURN NEW;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER partitions_trigger BEFORE INSERT OR UPDATE ON partitions FOR EACH ROW\nEXECUTE FUNCTION partitions_trigger_func();\n\n\nCREATE OR REPLACE FUNCTION create_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection(data jsonb) RETURNS VOID AS $$\nDECLARE\n out collections%ROWTYPE;\nBEGIN\n UPDATE collections SET content=data WHERE id = data->>'id' RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ON CONFLICT (id) DO\n UPDATE\n SET content=EXCLUDED.content\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION delete_collection(_id text) RETURNS VOID AS $$\nDECLARE\n out collections%ROWTYPE;\nBEGIN\n DELETE FROM collections WHERE id = _id RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION get_collection(id text) RETURNS jsonb AS $$\n SELECT content FROM collections\n WHERE id=$1\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION all_collections() RETURNS jsonb AS $$\n SELECT jsonb_agg(content) FROM collections;\n;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\nCREATE TABLE queryables (\n id bigint GENERATED ALWAYS AS identity PRIMARY KEY,\n name text UNIQUE NOT NULL,\n collection_ids text[], -- used to determine what partitions to create indexes on\n definition jsonb,\n property_path text,\n property_wrapper text,\n property_index_type text\n);\nCREATE INDEX queryables_name_idx ON queryables (name);\nCREATE INDEX queryables_property_wrapper_idx ON queryables (property_wrapper);\n\n\nINSERT INTO queryables (name, definition) VALUES\n('id', '{\"title\": \"Item ID\",\"description\": \"Item identifier\",\"$ref\": \"https://schemas.stacspec.org/v1.0.0/item-spec/json-schema/item.json#/definitions/core/allOf/2/properties/id\"}'),\n('datetime','{\"description\": \"Datetime\",\"type\": \"string\",\"title\": \"Acquired\",\"format\": \"date-time\",\"pattern\": \"(\\\\+00:00|Z)$\"}')\nON CONFLICT DO NOTHING;\n\n\n\nINSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n('eo:cloud_cover','{\"$ref\": \"https://stac-extensions.github.io/eo/v1.0.0/schema.json#/definitions/fieldsproperties/eo:cloud_cover\"}','to_int','BTREE')\nON CONFLICT DO NOTHING;\n\nCREATE OR REPLACE FUNCTION array_to_path(arr text[]) RETURNS text AS $$\n SELECT string_agg(\n quote_literal(v),\n '->'\n ) FROM unnest(arr) v;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\n\n\nCREATE OR REPLACE FUNCTION queryable(\n IN dotpath text,\n OUT path text,\n OUT expression text,\n OUT wrapper text\n) AS $$\nDECLARE\n q RECORD;\n path_elements text[];\nBEGIN\n IF dotpath IN ('id', 'geometry', 'datetime', 'end_datetime', 'collection') THEN\n path := dotpath;\n expression := dotpath;\n wrapper := NULL;\n RETURN;\n END IF;\n SELECT * INTO q FROM queryables WHERE name=dotpath;\n IF q.property_wrapper IS NULL THEN\n IF q.definition->>'type' = 'number' THEN\n wrapper := 'to_float';\n ELSIF q.definition->>'format' = 'date-time' THEN\n wrapper := 'to_tstz';\n ELSE\n wrapper := 'to_text';\n END IF;\n ELSE\n wrapper := q.property_wrapper;\n END IF;\n IF q.property_path IS NOT NULL THEN\n path := q.property_path;\n ELSE\n path_elements := string_to_array(dotpath, '.');\n IF path_elements[1] IN ('links', 'assets', 'stac_version', 'stac_extensions') THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSIF path_elements[1] = 'properties' THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSE\n path := format($F$content->'properties'->%s$F$, array_to_path(path_elements));\n END IF;\n END IF;\n expression := format('%I(%s)', wrapper, path);\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\nCREATE OR REPLACE FUNCTION create_queryable_indexes() RETURNS VOID AS $$\nDECLARE\n queryable RECORD;\n q text;\nBEGIN\n FOR queryable IN\n SELECT\n queryables.id as qid,\n CASE WHEN collections.key IS NULL THEN 'items' ELSE format('_items_%s',collections.key) END AS part,\n property_index_type,\n expression\n FROM\n queryables\n LEFT JOIN collections ON (collections.id = ANY (queryables.collection_ids))\n JOIN LATERAL queryable(queryables.name) ON (queryables.property_index_type IS NOT NULL)\n LOOP\n q := format(\n $q$\n CREATE INDEX IF NOT EXISTS %I ON %I USING %s ((%s));\n $q$,\n format('%s_%s_idx', queryable.part, queryable.qid),\n queryable.part,\n COALESCE(queryable.property_index_type, 'to_text'),\n queryable.expression\n );\n RAISE NOTICE '%',q;\n EXECUTE q;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION queryables_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\nBEGIN\nPERFORM create_queryable_indexes();\nRETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER queryables_trigger AFTER INSERT OR UPDATE ON queryables\nFOR EACH STATEMENT EXECUTE PROCEDURE queryables_trigger_func();\n\nCREATE TRIGGER queryables_collection_trigger AFTER INSERT OR UPDATE ON collections\nFOR EACH STATEMENT EXECUTE PROCEDURE queryables_trigger_func();\nCREATE OR REPLACE FUNCTION parse_dtrange(\n _indate jsonb,\n relative_base timestamptz DEFAULT date_trunc('hour', CURRENT_TIMESTAMP)\n) RETURNS tstzrange AS $$\nDECLARE\n timestrs text[];\n s timestamptz;\n e timestamptz;\nBEGIN\n timestrs :=\n CASE\n WHEN _indate ? 'timestamp' THEN\n ARRAY[_indate->>'timestamp']\n WHEN _indate ? 'interval' THEN\n to_text_array(_indate->'interval')\n WHEN jsonb_typeof(_indate) = 'array' THEN\n to_text_array(_indate)\n ELSE\n regexp_split_to_array(\n _indate->>0,\n '/'\n )\n END;\n RAISE NOTICE 'TIMESTRS %', timestrs;\n IF cardinality(timestrs) = 1 THEN\n IF timestrs[1] ILIKE 'P%' THEN\n RETURN tstzrange(relative_base - upper(timestrs[1])::interval, relative_base, '[)');\n END IF;\n s := timestrs[1]::timestamptz;\n RETURN tstzrange(s, s, '[]');\n END IF;\n\n IF cardinality(timestrs) != 2 THEN\n RAISE EXCEPTION 'Timestamp cannot have more than 2 values';\n END IF;\n\n IF timestrs[1] = '..' THEN\n s := '-infinity'::timestamptz;\n e := timestrs[2]::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] = '..' THEN\n s := timestrs[1]::timestamptz;\n e := 'infinity'::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[1] ILIKE 'P%' AND timestrs[2] NOT ILIKE 'P%' THEN\n e := timestrs[2]::timestamptz;\n s := e - upper(timestrs[1])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] ILIKE 'P%' AND timestrs[1] NOT ILIKE 'P%' THEN\n s := timestrs[1]::timestamptz;\n e := s + upper(timestrs[2])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n s := timestrs[1]::timestamptz;\n e := timestrs[2]::timestamptz;\n\n RETURN tstzrange(s,e,'[)');\n\n RETURN NULL;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT PARALLEL SAFE SET TIME ZONE 'UTC';\n\nCREATE OR REPLACE FUNCTION parse_dtrange(\n _indate text,\n relative_base timestamptz DEFAULT CURRENT_TIMESTAMP\n) RETURNS tstzrange AS $$\n SELECT parse_dtrange(to_jsonb(_indate), relative_base);\n$$ LANGUAGE SQL STABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION temporal_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\n ll text := 'datetime';\n lh text := 'end_datetime';\n rrange tstzrange;\n rl text;\n rh text;\n outq text;\nBEGIN\n rrange := parse_dtrange(args->1);\n RAISE NOTICE 'Constructing temporal query OP: %, ARGS: %, RRANGE: %', op, args, rrange;\n op := lower(op);\n rl := format('%L::timestamptz', lower(rrange));\n rh := format('%L::timestamptz', upper(rrange));\n outq := CASE op\n WHEN 't_before' THEN 'lh < rl'\n WHEN 't_after' THEN 'll > rh'\n WHEN 't_meets' THEN 'lh = rl'\n WHEN 't_metby' THEN 'll = rh'\n WHEN 't_overlaps' THEN 'll < rl AND rl < lh < rh'\n WHEN 't_overlappedby' THEN 'rl < ll < rh AND lh > rh'\n WHEN 't_starts' THEN 'll = rl AND lh < rh'\n WHEN 't_startedby' THEN 'll = rl AND lh > rh'\n WHEN 't_during' THEN 'll > rl AND lh < rh'\n WHEN 't_contains' THEN 'll < rl AND lh > rh'\n WHEN 't_finishes' THEN 'll > rl AND lh = rh'\n WHEN 't_finishedby' THEN 'll < rl AND lh = rh'\n WHEN 't_equals' THEN 'll = rl AND lh = rh'\n WHEN 't_disjoint' THEN 'NOT (ll <= rh AND lh >= rl)'\n WHEN 't_intersects' THEN 'll <= rh AND lh >= rl'\n WHEN 'anyinteracts' THEN 'll <= rh AND lh >= rl'\n END;\n outq := regexp_replace(outq, '\\mll\\M', ll);\n outq := regexp_replace(outq, '\\mlh\\M', lh);\n outq := regexp_replace(outq, '\\mrl\\M', rl);\n outq := regexp_replace(outq, '\\mrh\\M', rh);\n outq := format('(%s)', outq);\n RETURN outq;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\n\n\nCREATE OR REPLACE FUNCTION spatial_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\n geom text;\n j jsonb := args->1;\nBEGIN\n op := lower(op);\n RAISE NOTICE 'Constructing spatial query OP: %, ARGS: %', op, args;\n IF op NOT IN ('s_equals','s_disjoint','s_touches','s_within','s_overlaps','s_crosses','s_intersects','intersects','s_contains') THEN\n RAISE EXCEPTION 'Spatial Operator % Not Supported', op;\n END IF;\n op := regexp_replace(op, '^s_', 'st_');\n IF op = 'intersects' THEN\n op := 'st_intersects';\n END IF;\n -- Convert geometry to WKB string\n IF j ? 'type' AND j ? 'coordinates' THEN\n geom := st_geomfromgeojson(j)::text;\n ELSIF jsonb_typeof(j) = 'array' THEN\n geom := bbox_geom(j)::text;\n END IF;\n\n RETURN format('%s(geometry, %L::geometry)', op, geom);\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION query_to_cql2(q jsonb) RETURNS jsonb AS $$\n-- Translates anything passed in through the deprecated \"query\" into equivalent CQL2\nWITH t AS (\n SELECT key as property, value as ops\n FROM jsonb_each(q)\n), t2 AS (\n SELECT property, (jsonb_each(ops)).*\n FROM t WHERE jsonb_typeof(ops) = 'object'\n UNION ALL\n SELECT property, 'eq', ops\n FROM t WHERE jsonb_typeof(ops) != 'object'\n)\nSELECT\n jsonb_strip_nulls(jsonb_build_object(\n 'op', 'and',\n 'args', jsonb_agg(\n jsonb_build_object(\n 'op', key,\n 'args', jsonb_build_array(\n jsonb_build_object('property',property),\n value\n )\n )\n )\n )\n) as qcql FROM t2\n;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\nCREATE OR REPLACE FUNCTION cql1_to_cql2(j jsonb) RETURNS jsonb AS $$\nDECLARE\n args jsonb;\n ret jsonb;\nBEGIN\n RAISE NOTICE 'CQL1_TO_CQL2: %', j;\n IF j ? 'filter' THEN\n RETURN cql1_to_cql2(j->'filter');\n END IF;\n IF j ? 'property' THEN\n RETURN j;\n END IF;\n IF jsonb_typeof(j) = 'array' THEN\n SELECT jsonb_agg(cql1_to_cql2(el)) INTO args FROM jsonb_array_elements(j) el;\n RETURN args;\n END IF;\n IF jsonb_typeof(j) = 'number' THEN\n RETURN j;\n END IF;\n IF jsonb_typeof(j) = 'string' THEN\n RETURN j;\n END IF;\n\n IF jsonb_typeof(j) = 'object' THEN\n SELECT jsonb_build_object(\n 'op', key,\n 'args', cql1_to_cql2(value)\n ) INTO ret\n FROM jsonb_each(j)\n WHERE j IS NOT NULL;\n RETURN ret;\n END IF;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE STRICT;\n\nCREATE TABLE cql2_ops (\n op text PRIMARY KEY,\n template text,\n types text[]\n);\nINSERT INTO cql2_ops (op, template, types) VALUES\n ('eq', '%s = %s', NULL),\n ('lt', '%s < %s', NULL),\n ('lte', '%s <= %s', NULL),\n ('gt', '%s > %s', NULL),\n ('gte', '%s >= %s', NULL),\n ('le', '%s <= %s', NULL),\n ('ge', '%s >= %s', NULL),\n ('=', '%s = %s', NULL),\n ('<', '%s < %s', NULL),\n ('<=', '%s <= %s', NULL),\n ('>', '%s > %s', NULL),\n ('>=', '%s >= %s', NULL),\n ('like', '%s LIKE %s', NULL),\n ('ilike', '%s ILIKE %s', NULL),\n ('+', '%s + %s', NULL),\n ('-', '%s - %s', NULL),\n ('*', '%s * %s', NULL),\n ('/', '%s / %s', NULL),\n ('in', '%s = ANY (%s)', NULL),\n ('not', 'NOT (%s)', NULL),\n ('between', '%s BETWEEN (%2$s)[1] AND (%2$s)[2]', NULL),\n ('isnull', '%s IS NULL', NULL)\nON CONFLICT (op) DO UPDATE\n SET\n template = EXCLUDED.template;\n;\n\n\nCREATE OR REPLACE FUNCTION cql2_query(j jsonb, wrapper text DEFAULT NULL) RETURNS text AS $$\n#variable_conflict use_variable\nDECLARE\n args jsonb := j->'args';\n arg jsonb;\n op text := lower(j->>'op');\n cql2op RECORD;\n literal text;\nBEGIN\n IF j IS NULL OR (op IS NOT NULL AND args IS NULL) THEN\n RETURN NULL;\n END IF;\n RAISE NOTICE 'CQL2_QUERY: %', j;\n IF j ? 'filter' THEN\n RETURN cql2_query(j->'filter');\n END IF;\n\n IF j ? 'upper' THEN\n RETURN format('upper(%s)', cql2_query(j->'upper'));\n END IF;\n\n IF j ? 'lower' THEN\n RETURN format('lower(%s)', cql2_query(j->'lower'));\n END IF;\n\n -- Temporal Query\n IF op ilike 't_%' or op = 'anyinteracts' THEN\n RETURN temporal_op_query(op, args);\n END IF;\n\n -- If property is a timestamp convert it to text to use with\n -- general operators\n IF j ? 'timestamp' THEN\n RETURN format('%L::timestamptz', to_tstz(j->'timestamp'));\n END IF;\n IF j ? 'interval' THEN\n RAISE EXCEPTION 'Please use temporal operators when using intervals.';\n RETURN NONE;\n END IF;\n\n -- Spatial Query\n IF op ilike 's_%' or op = 'intersects' THEN\n RETURN spatial_op_query(op, args);\n END IF;\n\n\n IF op = 'in' THEN\n RETURN format(\n '%s = ANY (%L)',\n cql2_query(args->0),\n to_text_array(args->1)\n );\n END IF;\n\n\n\n IF op = 'between' THEN\n SELECT (queryable(a->>'property')).wrapper INTO wrapper\n FROM jsonb_array_elements(args) a\n WHERE a ? 'property' LIMIT 1;\n\n RETURN format(\n '%s BETWEEN %s and %s',\n cql2_query(args->0, wrapper),\n cql2_query(args->1->0, wrapper),\n cql2_query(args->1->1, wrapper)\n );\n END IF;\n\n -- Make sure that args is an array and run cql2_query on\n -- each element of the array\n RAISE NOTICE 'ARGS PRE: %', args;\n IF j ? 'args' THEN\n IF jsonb_typeof(args) != 'array' THEN\n args := jsonb_build_array(args);\n END IF;\n\n SELECT (queryable(a->>'property')).wrapper INTO wrapper\n FROM jsonb_array_elements(args) a\n WHERE a ? 'property' LIMIT 1;\n\n SELECT jsonb_agg(cql2_query(a, wrapper))\n INTO args\n FROM jsonb_array_elements(args) a;\n END IF;\n RAISE NOTICE 'ARGS: %', args;\n\n IF op IN ('and', 'or') THEN\n RETURN\n format(\n '(%s)',\n array_to_string(to_text_array(args), format(' %s ', upper(op)))\n );\n END IF;\n\n -- Look up template from cql2_ops\n IF j ? 'op' THEN\n SELECT * INTO cql2op FROM cql2_ops WHERE cql2_ops.op ilike op;\n IF FOUND THEN\n -- If specific index set in queryables for a property cast other arguments to that type\n RETURN format(\n cql2op.template,\n VARIADIC (to_text_array(args))\n );\n ELSE\n RAISE EXCEPTION 'Operator % Not Supported.', op;\n END IF;\n END IF;\n\n\n IF j ? 'property' THEN\n RETURN (queryable(j->>'property')).expression;\n END IF;\n\n IF wrapper IS NOT NULL THEN\n EXECUTE format('SELECT %I(%L)', wrapper, j) INTO literal;\n RAISE NOTICE '% % %',wrapper, j, literal;\n RETURN format('%I(%L)', wrapper, j);\n END IF;\n\n RETURN quote_literal(to_text(j));\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION paging_dtrange(\n j jsonb\n) RETURNS tstzrange AS $$\nDECLARE\n op text;\n filter jsonb := j->'filter';\n dtrange tstzrange := tstzrange('-infinity'::timestamptz,'infinity'::timestamptz);\n sdate timestamptz := '-infinity'::timestamptz;\n edate timestamptz := 'infinity'::timestamptz;\n jpitem jsonb;\nBEGIN\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n END IF;\n IF NOT (filter @? '$.**.op ? (@ == \"or\" || @ == \"not\")') THEN\n FOR jpitem IN SELECT j FROM jsonb_path_query(filter,'strict $.** ? (@.args[*].property == \"datetime\")'::jsonpath) j LOOP\n op := lower(jpitem->>'op');\n dtrange := parse_dtrange(jpitem->'args'->1);\n IF op IN ('<=', 'lt', 'lte', '<', 'le', 't_before') THEN\n sdate := greatest(sdate,'-infinity');\n edate := least(edate, upper(dtrange));\n ELSIF op IN ('>=', '>', 'gt', 'gte', 'ge', 't_after') THEN\n edate := least(edate, 'infinity');\n sdate := greatest(sdate, lower(dtrange));\n ELSIF op IN ('=', 'eq') THEN\n edate := least(edate, upper(dtrange));\n sdate := greatest(sdate, lower(dtrange));\n END IF;\n RAISE NOTICE '2 OP: %, ARGS: %, DTRANGE: %, SDATE: %, EDATE: %', op, jpitem->'args'->1, dtrange, sdate, edate;\n END LOOP;\n END IF;\n IF sdate > edate THEN\n RETURN 'empty'::tstzrange;\n END IF;\n RETURN tstzrange(sdate,edate, '[]');\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT SET TIME ZONE 'UTC';\n\nCREATE OR REPLACE FUNCTION paging_collections(\n IN j jsonb\n) RETURNS text[] AS $$\nDECLARE\n filter jsonb := j->'filter';\n jpitem jsonb;\n op text;\n args jsonb;\n arg jsonb;\n collections text[];\nBEGIN\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n END IF;\n IF NOT (filter @? '$.**.op ? (@ == \"or\" || @ == \"not\")') THEN\n FOR jpitem IN SELECT j FROM jsonb_path_query(filter,'strict $.** ? (@.args[*].property == \"collection\")'::jsonpath) j LOOP\n RAISE NOTICE 'JPITEM: %', jpitem;\n op := jpitem->>'op';\n args := jpitem->'args';\n IF op IN ('=', 'eq', 'in') THEN\n FOR arg IN SELECT a FROM jsonb_array_elements(args) a LOOP\n IF jsonb_typeof(arg) IN ('string', 'array') THEN\n RAISE NOTICE 'arg: %, collections: %', arg, collections;\n IF collections IS NULL OR collections = '{}'::text[] THEN\n collections := to_text_array(arg);\n ELSE\n collections := array_intersection(collections, to_text_array(arg));\n END IF;\n END IF;\n END LOOP;\n END IF;\n END LOOP;\n END IF;\n IF collections = '{}'::text[] THEN\n RETURN NULL;\n END IF;\n RETURN collections;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\nCREATE TABLE items (\n id text NOT NULL,\n geometry geometry NOT NULL,\n collection text NOT NULL,\n datetime timestamptz NOT NULL,\n end_datetime timestamptz NOT NULL,\n content JSONB NOT NULL\n)\nPARTITION BY LIST (collection)\n;\n\nCREATE INDEX \"datetime_idx\" ON items USING BTREE (datetime DESC, end_datetime ASC);\nCREATE INDEX \"geometry_idx\" ON items USING GIST (geometry);\n\nCREATE STATISTICS datetime_stats (dependencies) on datetime, end_datetime from items;\n\n\nALTER TABLE items ADD CONSTRAINT items_collections_fk FOREIGN KEY (collection) REFERENCES collections(id) ON DELETE CASCADE DEFERRABLE;\n\n\nCREATE OR REPLACE FUNCTION content_slim(_item jsonb) RETURNS jsonb AS $$\n SELECT strip_jsonb(_item - '{id,geometry,collection,type}'::text[], collection_base_item(_item->>'collection')) - '{id,geometry,collection,type}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_dehydrate(content jsonb) RETURNS items AS $$\n SELECT\n content->>'id' as id,\n stac_geom(content) as geometry,\n content->>'collection' as collection,\n stac_datetime(content) as datetime,\n stac_end_datetime(content) as end_datetime,\n content_slim(content) as content\n ;\n$$ LANGUAGE SQL STABLE;\n\nCREATE OR REPLACE FUNCTION include_field(f text, fields jsonb DEFAULT '{}'::jsonb) RETURNS boolean AS $$\nDECLARE\n includes jsonb := fields->'include';\n excludes jsonb := fields->'exclude';\nBEGIN\n IF f IS NULL THEN\n RETURN NULL;\n END IF;\n\n\n IF\n jsonb_typeof(excludes) = 'array'\n AND jsonb_array_length(excludes)>0\n AND excludes ? f\n THEN\n RETURN FALSE;\n END IF;\n\n IF\n (\n jsonb_typeof(includes) = 'array'\n AND jsonb_array_length(includes) > 0\n AND includes ? f\n ) OR\n (\n includes IS NULL\n OR jsonb_typeof(includes) = 'null'\n OR jsonb_array_length(includes) = 0\n )\n THEN\n RETURN TRUE;\n END IF;\n\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nDROP FUNCTION IF EXISTS content_hydrate(jsonb, jsonb, jsonb);\nCREATE OR REPLACE FUNCTION content_hydrate(\n _base_item jsonb,\n _item jsonb,\n fields jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\n SELECT merge_jsonb(\n jsonb_fields(_item, fields),\n jsonb_fields(_base_item, fields)\n );\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION content_hydrate(_item items, _collection collections, fields jsonb DEFAULT '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\n content jsonb;\n base_item jsonb := _collection.base_item;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry, 20)::jsonb;\n END IF;\n output := content_hydrate(\n jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'collection', _item.collection,\n 'type', 'Feature'\n ) || _item.content,\n _collection.base_item,\n fields\n );\n\n RETURN output;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_nonhydrated(\n _item items,\n fields jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry, 20)::jsonb;\n END IF;\n output := jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'collection', _item.collection,\n 'type', 'Feature'\n ) || _item.content;\n RETURN output;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_hydrate(_item items, fields jsonb DEFAULT '{}'::jsonb) RETURNS jsonb AS $$\n SELECT content_hydrate(\n _item,\n (SELECT c FROM collections c WHERE id=_item.collection LIMIT 1),\n fields\n );\n$$ LANGUAGE SQL STABLE;\n\n\nCREATE UNLOGGED TABLE items_staging (\n content JSONB NOT NULL\n);\nCREATE UNLOGGED TABLE items_staging_ignore (\n content JSONB NOT NULL\n);\nCREATE UNLOGGED TABLE items_staging_upsert (\n content JSONB NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION items_staging_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p record;\n _partitions text[];\n ts timestamptz := clock_timestamp();\nBEGIN\n RAISE NOTICE 'Creating Partitions. %', clock_timestamp() - ts;\n WITH ranges AS (\n SELECT\n n.content->>'collection' as collection,\n stac_daterange(n.content->'properties') as dtr\n FROM newdata n\n ), p AS (\n SELECT\n collection,\n lower(dtr) as datetime,\n upper(dtr) as end_datetime,\n (partition_name(\n collection,\n lower(dtr)\n )).partition_name as name\n FROM ranges\n )\n INSERT INTO partitions (collection, datetime_range, end_datetime_range)\n SELECT\n collection,\n tstzrange(min(datetime), max(datetime), '[]') as datetime_range,\n tstzrange(min(end_datetime), max(end_datetime), '[]') as end_datetime_range\n FROM p\n GROUP BY collection, name\n ON CONFLICT (name) DO UPDATE SET\n datetime_range = EXCLUDED.datetime_range,\n end_datetime_range = EXCLUDED.end_datetime_range\n ;\n\n RAISE NOTICE 'Doing the insert. %', clock_timestamp() - ts;\n IF TG_TABLE_NAME = 'items_staging' THEN\n INSERT INTO items\n SELECT\n (content_dehydrate(content)).*\n FROM newdata;\n DELETE FROM items_staging;\n ELSIF TG_TABLE_NAME = 'items_staging_ignore' THEN\n INSERT INTO items\n SELECT\n (content_dehydrate(content)).*\n FROM newdata\n ON CONFLICT DO NOTHING;\n DELETE FROM items_staging_ignore;\n ELSIF TG_TABLE_NAME = 'items_staging_upsert' THEN\n WITH staging_formatted AS (\n SELECT (content_dehydrate(content)).* FROM newdata\n ), deletes AS (\n DELETE FROM items i USING staging_formatted s\n WHERE\n i.id = s.id\n AND i.collection = s.collection\n AND i IS DISTINCT FROM s\n RETURNING i.id, i.collection\n )\n INSERT INTO items\n SELECT s.* FROM\n staging_formatted s\n JOIN deletes d\n USING (id, collection);\n DELETE FROM items_staging_upsert;\n END IF;\n RAISE NOTICE 'Done. %', clock_timestamp() - ts;\n\n RETURN NULL;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER items_staging_insert_trigger AFTER INSERT ON items_staging REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\nCREATE TRIGGER items_staging_insert_ignore_trigger AFTER INSERT ON items_staging_ignore REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\nCREATE TRIGGER items_staging_insert_upsert_trigger AFTER INSERT ON items_staging_upsert REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\n\n\n\nCREATE OR REPLACE FUNCTION item_by_id(_id text, _collection text DEFAULT NULL) RETURNS items AS\n$$\nDECLARE\n i items%ROWTYPE;\nBEGIN\n SELECT * INTO i FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection) LIMIT 1;\n RETURN i;\nEND;\n$$ LANGUAGE PLPGSQL STABLE SECURITY DEFINER SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION get_item(_id text, _collection text DEFAULT NULL) RETURNS jsonb AS $$\n SELECT content_hydrate(items) FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection);\n$$ LANGUAGE SQL STABLE SECURITY DEFINER SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION delete_item(_id text, _collection text DEFAULT NULL) RETURNS VOID AS $$\nDECLARE\nout items%ROWTYPE;\nBEGIN\n DELETE FROM items WHERE id = _id AND (_collection IS NULL OR collection=_collection) RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n--/*\nCREATE OR REPLACE FUNCTION create_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION update_item(content jsonb) RETURNS VOID AS $$\nDECLARE\n old items %ROWTYPE;\n out items%ROWTYPE;\nBEGIN\n PERFORM delete_item(content->>'id', content->>'collection');\n PERFORM create_item(content);\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION create_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION collection_bbox(id text) RETURNS jsonb AS $$\n SELECT (replace(replace(replace(st_extent(geometry)::text,'BOX(','[['),')',']]'),' ',','))::jsonb\n FROM items WHERE collection=$1;\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION collection_temporal_extent(id text) RETURNS jsonb AS $$\n SELECT to_jsonb(array[array[min(datetime)::text, max(datetime)::text]])\n FROM items WHERE collection=$1;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection_extents() RETURNS VOID AS $$\nUPDATE collections SET\n content = content ||\n jsonb_build_object(\n 'extent', jsonb_build_object(\n 'spatial', jsonb_build_object(\n 'bbox', collection_bbox(collections.id)\n ),\n 'temporal', jsonb_build_object(\n 'interval', collection_temporal_extent(collections.id)\n )\n )\n )\n;\n$$ LANGUAGE SQL;\nCREATE VIEW partition_steps AS\nSELECT\n name,\n date_trunc('month',lower(datetime_range)) as sdate,\n date_trunc('month', upper(datetime_range)) + '1 month'::interval as edate\n FROM partitions WHERE datetime_range IS NOT NULL AND datetime_range != 'empty'::tstzrange\n ORDER BY datetime_range ASC\n;\n\nCREATE OR REPLACE FUNCTION chunker(\n IN _where text,\n OUT s timestamptz,\n OUT e timestamptz\n) RETURNS SETOF RECORD AS $$\nDECLARE\n explain jsonb;\nBEGIN\n IF _where IS NULL THEN\n _where := ' TRUE ';\n END IF;\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s;', _where)\n INTO explain;\n\n RETURN QUERY\n WITH t AS (\n SELECT j->>0 as p FROM\n jsonb_path_query(\n explain,\n 'strict $.**.\"Relation Name\" ? (@ != null)'\n ) j\n ),\n parts AS (\n SELECT sdate, edate FROM t JOIN partition_steps ON (t.p = name)\n ),\n times AS (\n SELECT sdate FROM parts\n UNION\n SELECT edate FROM parts\n ),\n uniq AS (\n SELECT DISTINCT sdate FROM times ORDER BY sdate\n ),\n last AS (\n SELECT sdate, lead(sdate, 1) over () as edate FROM uniq\n )\n SELECT sdate, edate FROM last WHERE edate IS NOT NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION partition_queries(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN partitions text[] DEFAULT NULL\n) RETURNS SETOF text AS $$\nDECLARE\n query text;\n sdate timestamptz;\n edate timestamptz;\nBEGIN\nIF _where IS NULL OR trim(_where) = '' THEN\n _where = ' TRUE ';\nEND IF;\nRAISE NOTICE 'Getting chunks for % %', _where, _orderby;\nIF _orderby ILIKE 'datetime d%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 DESC LOOP\n RETURN NEXT format($q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n $q$,\n sdate,\n edate,\n _where,\n _orderby\n );\n END LOOP;\nELSIF _orderby ILIKE 'datetime a%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 ASC LOOP\n RETURN NEXT format($q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n $q$,\n sdate,\n edate,\n _where,\n _orderby\n );\n END LOOP;\nELSE\n query := format($q$\n SELECT * FROM items\n WHERE %s\n ORDER BY %s\n $q$, _where, _orderby\n );\n\n RETURN NEXT query;\n RETURN;\nEND IF;\n\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION partition_query_view(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN _limit int DEFAULT 10\n) RETURNS text AS $$\n WITH p AS (\n SELECT * FROM partition_queries(_where, _orderby) p\n )\n SELECT\n CASE WHEN EXISTS (SELECT 1 FROM p) THEN\n (SELECT format($q$\n SELECT * FROM (\n %s\n ) total LIMIT %s\n $q$,\n string_agg(\n format($q$ SELECT * FROM ( %s ) AS sub $q$, p),\n '\n UNION ALL\n '\n ),\n _limit\n ))\n ELSE NULL\n END FROM p;\n$$ LANGUAGE SQL IMMUTABLE;\n\n\n\n\nCREATE OR REPLACE FUNCTION stac_search_to_where(j jsonb) RETURNS text AS $$\nDECLARE\n where_segments text[];\n _where text;\n dtrange tstzrange;\n collections text[];\n geom geometry;\n sdate timestamptz;\n edate timestamptz;\n filterlang text;\n filter jsonb := j->'filter';\nBEGIN\n IF j ? 'ids' THEN\n where_segments := where_segments || format('id = ANY (%L) ', to_text_array(j->'ids'));\n END IF;\n\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n where_segments := where_segments || format('collection = ANY (%L) ', collections);\n END IF;\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n\n where_segments := where_segments || format(' datetime <= %L::timestamptz AND end_datetime >= %L::timestamptz ',\n edate,\n sdate\n );\n END IF;\n\n geom := stac_geom(j);\n IF geom IS NOT NULL THEN\n where_segments := where_segments || format('st_intersects(geometry, %L)',geom);\n END IF;\n\n filterlang := COALESCE(\n j->>'filter-lang',\n get_setting('default-filter-lang', j->'conf')\n );\n IF NOT filter @? '$.**.op' THEN\n filterlang := 'cql-json';\n END IF;\n\n IF filterlang NOT IN ('cql-json','cql2-json') AND j ? 'filter' THEN\n RAISE EXCEPTION '% is not a supported filter-lang. Please use cql-json or cql2-json.', filterlang;\n END IF;\n\n IF j ? 'query' AND j ? 'filter' THEN\n RAISE EXCEPTION 'Can only use either query or filter at one time.';\n END IF;\n\n IF j ? 'query' THEN\n filter := query_to_cql2(j->'query');\n ELSIF filterlang = 'cql-json' THEN\n filter := cql1_to_cql2(filter);\n END IF;\n RAISE NOTICE 'FILTER: %', filter;\n where_segments := where_segments || cql2_query(filter);\n IF cardinality(where_segments) < 1 THEN\n RETURN ' TRUE ';\n END IF;\n\n _where := array_to_string(array_remove(where_segments, NULL), ' AND ');\n\n IF _where IS NULL OR BTRIM(_where) = '' THEN\n RETURN ' TRUE ';\n END IF;\n RETURN _where;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION parse_sort_dir(_dir text, reverse boolean default false) RETURNS text AS $$\n WITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n ) SELECT\n CASE\n WHEN NOT reverse THEN d\n WHEN d = 'ASC' THEN 'DESC'\n WHEN d = 'DESC' THEN 'ASC'\n END\n FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION sort_dir_to_op(_dir text, prev boolean default false) RETURNS text AS $$\n WITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n ) SELECT\n CASE\n WHEN d = 'ASC' AND prev THEN '<='\n WHEN d = 'DESC' AND prev THEN '>='\n WHEN d = 'ASC' THEN '>='\n WHEN d = 'DESC' THEN '<='\n END\n FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION sort_sqlorderby(\n _search jsonb DEFAULT NULL,\n reverse boolean DEFAULT FALSE\n) RETURNS text AS $$\n WITH sortby AS (\n SELECT coalesce(_search->'sortby','[{\"field\":\"datetime\", \"direction\":\"desc\"}]') as sort\n ), withid AS (\n SELECT CASE\n WHEN sort @? '$[*] ? (@.field == \"id\")' THEN sort\n ELSE sort || '[{\"field\":\"id\", \"direction\":\"desc\"}]'::jsonb\n END as sort\n FROM sortby\n ), withid_rows AS (\n SELECT jsonb_array_elements(sort) as value FROM withid\n ),sorts AS (\n SELECT\n coalesce(\n -- field_orderby((items_path(value->>'field')).path_txt),\n (queryable(value->>'field')).expression\n ) as key,\n parse_sort_dir(value->>'direction', reverse) as dir\n FROM withid_rows\n )\n SELECT array_to_string(\n array_agg(concat(key, ' ', dir)),\n ', '\n ) FROM sorts;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_sort_dir(sort_item jsonb) RETURNS text AS $$\n SELECT CASE WHEN sort_item->>'direction' ILIKE 'desc%' THEN 'DESC' ELSE 'ASC' END;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION get_token_filter(_search jsonb = '{}'::jsonb, token_rec jsonb DEFAULT NULL) RETURNS text AS $$\nDECLARE\n token_id text;\n filters text[] := '{}'::text[];\n prev boolean := TRUE;\n field text;\n dir text;\n sort record;\n orfilters text[] := '{}'::text[];\n andfilters text[] := '{}'::text[];\n output text;\n token_where text;\nBEGIN\n RAISE NOTICE 'Getting Token Filter. % %', _search, token_rec;\n -- If no token provided return NULL\n IF token_rec IS NULL THEN\n IF NOT (_search ? 'token' AND\n (\n (_search->>'token' ILIKE 'prev:%')\n OR\n (_search->>'token' ILIKE 'next:%')\n )\n ) THEN\n RETURN NULL;\n END IF;\n prev := (_search->>'token' ILIKE 'prev:%');\n token_id := substr(_search->>'token', 6);\n SELECT to_jsonb(items) INTO token_rec\n FROM items WHERE id=token_id;\n END IF;\n RAISE NOTICE 'TOKEN ID: % %', token_rec, token_rec->'id';\n\n CREATE TEMP TABLE sorts (\n _row int GENERATED ALWAYS AS IDENTITY NOT NULL,\n _field text PRIMARY KEY,\n _dir text NOT NULL,\n _val text\n ) ON COMMIT DROP;\n\n -- Make sure we only have distinct columns to sort with taking the first one we get\n INSERT INTO sorts (_field, _dir)\n SELECT\n (queryable(value->>'field')).expression,\n get_sort_dir(value)\n FROM\n jsonb_array_elements(coalesce(_search->'sortby','[{\"field\":\"datetime\",\"direction\":\"desc\"}]'))\n ON CONFLICT DO NOTHING\n ;\n RAISE NOTICE 'sorts 1: %', (SELECT jsonb_agg(to_json(sorts)) FROM sorts);\n -- Get the first sort direction provided. As the id is a primary key, if there are any\n -- sorts after id they won't do anything, so make sure that id is the last sort item.\n SELECT _dir INTO dir FROM sorts ORDER BY _row ASC LIMIT 1;\n IF EXISTS (SELECT 1 FROM sorts WHERE _field = 'id') THEN\n DELETE FROM sorts WHERE _row > (SELECT _row FROM sorts WHERE _field = 'id' ORDER BY _row ASC);\n ELSE\n INSERT INTO sorts (_field, _dir) VALUES ('id', dir);\n END IF;\n\n -- Add value from looked up item to the sorts table\n UPDATE sorts SET _val=quote_literal(token_rec->>_field);\n\n -- Check if all sorts are the same direction and use row comparison\n -- to filter\n RAISE NOTICE 'sorts 2: %', (SELECT jsonb_agg(to_json(sorts)) FROM sorts);\n\n IF (SELECT count(DISTINCT _dir) FROM sorts) = 1 THEN\n SELECT format(\n '(%s) %s (%s)',\n concat_ws(', ', VARIADIC array_agg(quote_ident(_field))),\n CASE WHEN (prev AND dir = 'ASC') OR (NOT prev AND dir = 'DESC') THEN '<' ELSE '>' END,\n concat_ws(', ', VARIADIC array_agg(_val))\n ) INTO output FROM sorts\n WHERE token_rec ? _field\n ;\n ELSE\n FOR sort IN SELECT * FROM sorts ORDER BY _row asc LOOP\n RAISE NOTICE 'SORT: %', sort;\n IF sort._row = 1 THEN\n orfilters := orfilters || format('(%s %s %s)',\n quote_ident(sort._field),\n CASE WHEN (prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC') THEN '<' ELSE '>' END,\n sort._val\n );\n ELSE\n orfilters := orfilters || format('(%s AND %s %s %s)',\n array_to_string(andfilters, ' AND '),\n quote_ident(sort._field),\n CASE WHEN (prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC') THEN '<' ELSE '>' END,\n sort._val\n );\n\n END IF;\n andfilters := andfilters || format('%s = %s',\n quote_ident(sort._field),\n sort._val\n );\n END LOOP;\n output := array_to_string(orfilters, ' OR ');\n END IF;\n DROP TABLE IF EXISTS sorts;\n token_where := concat('(',coalesce(output,'true'),')');\n IF trim(token_where) = '' THEN\n token_where := NULL;\n END IF;\n RAISE NOTICE 'TOKEN_WHERE: |%|',token_where;\n RETURN token_where;\n END;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION search_tohash(jsonb) RETURNS jsonb AS $$\n SELECT $1 - '{token,limit,context,includes,excludes}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION search_hash(jsonb, jsonb) RETURNS text AS $$\n SELECT md5(concat(search_tohash($1)::text,$2::text));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TABLE IF NOT EXISTS searches(\n hash text GENERATED ALWAYS AS (search_hash(search, metadata)) STORED PRIMARY KEY,\n search jsonb NOT NULL,\n _where text,\n orderby text,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n metadata jsonb DEFAULT '{}'::jsonb NOT NULL\n);\nCREATE TABLE IF NOT EXISTS search_wheres(\n id bigint generated always as identity primary key,\n _where text NOT NULL,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n statslastupdated timestamptz,\n estimated_count bigint,\n estimated_cost float,\n time_to_estimate float,\n total_count bigint,\n time_to_count float,\n partitions text[]\n);\n\nCREATE INDEX IF NOT EXISTS search_wheres_partitions ON search_wheres USING GIN (partitions);\nCREATE UNIQUE INDEX IF NOT EXISTS search_wheres_where ON search_wheres ((md5(_where)));\n\nCREATE OR REPLACE FUNCTION where_stats(inwhere text, updatestats boolean default false, conf jsonb default null) RETURNS search_wheres AS $$\nDECLARE\n t timestamptz;\n i interval;\n explain_json jsonb;\n partitions text[];\n sw search_wheres%ROWTYPE;\n inwhere_hash text := md5(inwhere);\n _context text := lower(context(conf));\n _stats_ttl interval := context_stats_ttl(conf);\n _estimated_cost float := context_estimated_cost(conf);\n _estimated_count int := context_estimated_count(conf);\nBEGIN\n IF _context = 'off' THEN\n sw._where := inwhere;\n return sw;\n END IF;\n\n SELECT * INTO sw FROM search_wheres WHERE md5(_where)=inwhere_hash FOR UPDATE;\n\n -- Update statistics if explicitly set, if statistics do not exist, or statistics ttl has expired\n IF NOT updatestats THEN\n RAISE NOTICE 'Checking if update is needed for: % .', inwhere;\n RAISE NOTICE 'Stats Last Updated: %', sw.statslastupdated;\n RAISE NOTICE 'TTL: %, Age: %', _stats_ttl, now() - sw.statslastupdated;\n RAISE NOTICE 'Context: %, Existing Total: %', _context, sw.total_count;\n IF\n sw.statslastupdated IS NULL\n OR (now() - sw.statslastupdated) > _stats_ttl\n OR (context(conf) != 'off' AND sw.total_count IS NULL)\n THEN\n updatestats := TRUE;\n END IF;\n END IF;\n\n sw._where := inwhere;\n sw.lastused := now();\n sw.usecount := coalesce(sw.usecount,0) + 1;\n\n IF NOT updatestats THEN\n UPDATE search_wheres SET\n lastused = sw.lastused,\n usecount = sw.usecount\n WHERE md5(_where) = inwhere_hash\n RETURNING * INTO sw\n ;\n RETURN sw;\n END IF;\n\n -- Use explain to get estimated count/cost and a list of the partitions that would be hit by the query\n t := clock_timestamp();\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s', inwhere)\n INTO explain_json;\n RAISE NOTICE 'Time for just the explain: %', clock_timestamp() - t;\n i := clock_timestamp() - t;\n\n sw.statslastupdated := now();\n sw.estimated_count := explain_json->0->'Plan'->'Plan Rows';\n sw.estimated_cost := explain_json->0->'Plan'->'Total Cost';\n sw.time_to_estimate := extract(epoch from i);\n\n RAISE NOTICE 'ESTIMATED_COUNT: % < %', sw.estimated_count, _estimated_count;\n RAISE NOTICE 'ESTIMATED_COST: % < %', sw.estimated_cost, _estimated_cost;\n\n -- Do a full count of rows if context is set to on or if auto is set and estimates are low enough\n IF\n _context = 'on'\n OR\n ( _context = 'auto' AND\n (\n sw.estimated_count < _estimated_count\n AND\n sw.estimated_cost < _estimated_cost\n )\n )\n THEN\n t := clock_timestamp();\n RAISE NOTICE 'Calculating actual count...';\n EXECUTE format(\n 'SELECT count(*) FROM items WHERE %s',\n inwhere\n ) INTO sw.total_count;\n i := clock_timestamp() - t;\n RAISE NOTICE 'Actual Count: % -- %', sw.total_count, i;\n sw.time_to_count := extract(epoch FROM i);\n ELSE\n sw.total_count := NULL;\n sw.time_to_count := NULL;\n END IF;\n\n\n INSERT INTO search_wheres\n (_where, lastused, usecount, statslastupdated, estimated_count, estimated_cost, time_to_estimate, partitions, total_count, time_to_count)\n SELECT sw._where, sw.lastused, sw.usecount, sw.statslastupdated, sw.estimated_count, sw.estimated_cost, sw.time_to_estimate, sw.partitions, sw.total_count, sw.time_to_count\n ON CONFLICT ((md5(_where)))\n DO UPDATE\n SET\n lastused = sw.lastused,\n usecount = sw.usecount,\n statslastupdated = sw.statslastupdated,\n estimated_count = sw.estimated_count,\n estimated_cost = sw.estimated_cost,\n time_to_estimate = sw.time_to_estimate,\n total_count = sw.total_count,\n time_to_count = sw.time_to_count\n ;\n RETURN sw;\nEND;\n$$ LANGUAGE PLPGSQL ;\n\n\n\nDROP FUNCTION IF EXISTS search_query;\nCREATE OR REPLACE FUNCTION search_query(\n _search jsonb = '{}'::jsonb,\n updatestats boolean = false,\n _metadata jsonb = '{}'::jsonb\n) RETURNS searches AS $$\nDECLARE\n search searches%ROWTYPE;\n pexplain jsonb;\n t timestamptz;\n i interval;\nBEGIN\n SELECT * INTO search FROM searches\n WHERE hash=search_hash(_search, _metadata) FOR UPDATE;\n\n -- Calculate the where clause if not already calculated\n IF search._where IS NULL THEN\n search._where := stac_search_to_where(_search);\n END IF;\n\n -- Calculate the order by clause if not already calculated\n IF search.orderby IS NULL THEN\n search.orderby := sort_sqlorderby(_search);\n END IF;\n\n PERFORM where_stats(search._where, updatestats, _search->'conf');\n\n search.lastused := now();\n search.usecount := coalesce(search.usecount, 0) + 1;\n INSERT INTO searches (search, _where, orderby, lastused, usecount, metadata)\n VALUES (_search, search._where, search.orderby, search.lastused, search.usecount, _metadata)\n ON CONFLICT (hash) DO\n UPDATE SET\n _where = EXCLUDED._where,\n orderby = EXCLUDED.orderby,\n lastused = EXCLUDED.lastused,\n usecount = EXCLUDED.usecount,\n metadata = EXCLUDED.metadata\n RETURNING * INTO search\n ;\n RETURN search;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION search(_search jsonb = '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n searches searches%ROWTYPE;\n _where text;\n token_where text;\n full_where text;\n orderby text;\n query text;\n token_type text := substr(_search->>'token',1,4);\n _limit int := coalesce((_search->>'limit')::int, 10);\n curs refcursor;\n cntr int := 0;\n iter_record items%ROWTYPE;\n first_record jsonb;\n first_item items%ROWTYPE;\n last_item items%ROWTYPE;\n last_record jsonb;\n out_records jsonb := '[]'::jsonb;\n prev_query text;\n next text;\n prev_id text;\n has_next boolean := false;\n has_prev boolean := false;\n prev text;\n total_count bigint;\n context jsonb;\n collection jsonb;\n includes text[];\n excludes text[];\n exit_flag boolean := FALSE;\n batches int := 0;\n timer timestamptz := clock_timestamp();\n pstart timestamptz;\n pend timestamptz;\n pcurs refcursor;\n search_where search_wheres%ROWTYPE;\n id text;\nBEGIN\nCREATE TEMP TABLE results (content jsonb) ON COMMIT DROP;\n-- if ids is set, short circuit and just use direct ids query for each id\n-- skip any paging or caching\n-- hard codes ordering in the same order as the array of ids\nIF _search ? 'ids' THEN\n INSERT INTO results\n SELECT\n CASE WHEN _search->'conf'->>'nohydrate' IS NOT NULL AND (_search->'conf'->>'nohydrate')::boolean = true THEN\n content_nonhydrated(items, _search->'fields')\n ELSE\n content_hydrate(items, _search->'fields')\n END\n FROM items WHERE\n items.id = ANY(to_text_array(_search->'ids'))\n AND\n CASE WHEN _search ? 'collections' THEN\n items.collection = ANY(to_text_array(_search->'collections'))\n ELSE TRUE\n END\n ORDER BY items.datetime desc, items.id desc\n ;\n SELECT INTO total_count count(*) FROM results;\nELSE\n searches := search_query(_search);\n _where := searches._where;\n orderby := searches.orderby;\n search_where := where_stats(_where);\n total_count := coalesce(search_where.total_count, search_where.estimated_count);\n\n IF token_type='prev' THEN\n token_where := get_token_filter(_search, null::jsonb);\n orderby := sort_sqlorderby(_search, TRUE);\n END IF;\n IF token_type='next' THEN\n token_where := get_token_filter(_search, null::jsonb);\n END IF;\n\n full_where := concat_ws(' AND ', _where, token_where);\n RAISE NOTICE 'FULL QUERY % %', full_where, clock_timestamp()-timer;\n timer := clock_timestamp();\n\n FOR query IN SELECT partition_queries(full_where, orderby, search_where.partitions) LOOP\n timer := clock_timestamp();\n query := format('%s LIMIT %s', query, _limit + 1);\n RAISE NOTICE 'Partition Query: %', query;\n batches := batches + 1;\n -- curs = create_cursor(query);\n OPEN curs FOR EXECUTE query;\n LOOP\n FETCH curs into iter_record;\n EXIT WHEN NOT FOUND;\n cntr := cntr + 1;\n\n IF _search->'conf'->>'nohydrate' IS NOT NULL AND (_search->'conf'->>'nohydrate')::boolean = true THEN\n last_record := content_nonhydrated(iter_record, _search->'fields');\n ELSE\n last_record := content_hydrate(iter_record, _search->'fields');\n END IF;\n last_item := iter_record;\n IF cntr = 1 THEN\n first_item := last_item;\n first_record := last_record;\n END IF;\n IF cntr <= _limit THEN\n INSERT INTO results (content) VALUES (last_record);\n ELSIF cntr > _limit THEN\n has_next := true;\n exit_flag := true;\n EXIT;\n END IF;\n END LOOP;\n CLOSE curs;\n RAISE NOTICE 'Query took %.', clock_timestamp()-timer;\n timer := clock_timestamp();\n EXIT WHEN exit_flag;\n END LOOP;\n RAISE NOTICE 'Scanned through % partitions.', batches;\nEND IF;\n\nSELECT jsonb_agg(content) INTO out_records FROM results WHERE content is not NULL;\n\nDROP TABLE results;\n\n\n-- Flip things around if this was the result of a prev token query\nIF token_type='prev' THEN\n out_records := flip_jsonb_array(out_records);\n first_record := last_record;\nEND IF;\n\n-- If this query has a token, see if there is data before the first record\nIF _search ? 'token' THEN\n prev_query := format(\n 'SELECT 1 FROM items WHERE %s LIMIT 1',\n concat_ws(\n ' AND ',\n _where,\n trim(get_token_filter(_search, to_jsonb(first_item)))\n )\n );\n RAISE NOTICE 'Query to get previous record: % --- %', prev_query, first_record;\n EXECUTE prev_query INTO has_prev;\n IF FOUND and has_prev IS NOT NULL THEN\n RAISE NOTICE 'Query results from prev query: %', has_prev;\n has_prev := TRUE;\n END IF;\nEND IF;\nhas_prev := COALESCE(has_prev, FALSE);\n\nIF has_prev THEN\n prev := out_records->0->>'id';\nEND IF;\nIF has_next OR token_type='prev' THEN\n next := out_records->-1->>'id';\nEND IF;\n\nIF context(_search->'conf') != 'off' THEN\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'matched', total_count,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\nELSE\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\nEND IF;\n\ncollection := jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb),\n 'next', next,\n 'prev', prev,\n 'context', context\n);\n\nRETURN collection;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER SET SEARCH_PATH TO pgstac, public;\n\n\nCREATE OR REPLACE FUNCTION search_cursor(_search jsonb = '{}'::jsonb) RETURNS refcursor AS $$\nDECLARE\n curs refcursor;\n searches searches%ROWTYPE;\n _where text;\n _orderby text;\n q text;\n\nBEGIN\n searches := search_query(_search);\n _where := searches._where;\n _orderby := searches.orderby;\n\n OPEN curs FOR\n WITH p AS (\n SELECT * FROM partition_queries(_where, _orderby) p\n )\n SELECT\n CASE WHEN EXISTS (SELECT 1 FROM p) THEN\n (SELECT format($q$\n SELECT * FROM (\n %s\n ) total\n $q$,\n string_agg(\n format($q$ SELECT * FROM ( %s ) AS sub $q$, p),\n '\n UNION ALL\n '\n )\n ))\n ELSE NULL\n END FROM p;\n RETURN curs;\nEND;\n$$ LANGUAGE PLPGSQL;\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION tileenvelope(zoom int, x int, y int) RETURNS geometry AS $$\nWITH t AS (\n SELECT\n 20037508.3427892 as merc_max,\n -20037508.3427892 as merc_min,\n (2 * 20037508.3427892) / (2 ^ zoom) as tile_size\n)\nSELECT st_makeenvelope(\n merc_min + (tile_size * x),\n merc_max - (tile_size * (y + 1)),\n merc_min + (tile_size * (x + 1)),\n merc_max - (tile_size * y),\n 3857\n) FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;DROP FUNCTION IF EXISTS mercgrid;\n\n\nCREATE OR REPLACE FUNCTION ftime() RETURNS interval as $$\nSELECT age(clock_timestamp(), transaction_timestamp());\n$$ LANGUAGE SQL;\nSET SEARCH_PATH to pgstac, public;\n\nDROP FUNCTION IF EXISTS geometrysearch;\nCREATE OR REPLACE FUNCTION geometrysearch(\n IN geom geometry,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE, -- Return as soon as the passed in geometry is full covered\n IN skipcovered boolean DEFAULT TRUE -- Skip any items that would show up completely under the previous items\n) RETURNS jsonb AS $$\nDECLARE\n search searches%ROWTYPE;\n curs refcursor;\n _where text;\n query text;\n iter_record items%ROWTYPE;\n out_records jsonb := '{}'::jsonb[];\n exit_flag boolean := FALSE;\n counter int := 1;\n scancounter int := 1;\n remaining_limit int := _scanlimit;\n tilearea float;\n unionedgeom geometry;\n clippedgeom geometry;\n unionedgeom_area float := 0;\n prev_area float := 0;\n excludes text[];\n includes text[];\n\nBEGIN\n DROP TABLE IF EXISTS pgstac_results;\n CREATE TEMP TABLE pgstac_results (content jsonb) ON COMMIT DROP;\n\n -- If skipcovered is true then you will always want to exit when the passed in geometry is full\n IF skipcovered THEN\n exitwhenfull := TRUE;\n END IF;\n\n SELECT * INTO search FROM searches WHERE hash=queryhash;\n\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Search with Query Hash % Not Found', queryhash;\n END IF;\n\n tilearea := st_area(geom);\n _where := format('%s AND st_intersects(geometry, %L::geometry)', search._where, geom);\n\n\n FOR query IN SELECT * FROM partition_queries(_where, search.orderby) LOOP\n query := format('%s LIMIT %L', query, remaining_limit);\n RAISE NOTICE '%', query;\n OPEN curs FOR EXECUTE query;\n LOOP\n FETCH curs INTO iter_record;\n EXIT WHEN NOT FOUND;\n IF exitwhenfull OR skipcovered THEN -- If we are not using exitwhenfull or skipcovered, we do not need to do expensive geometry operations\n clippedgeom := st_intersection(geom, iter_record.geometry);\n\n IF unionedgeom IS NULL THEN\n unionedgeom := clippedgeom;\n ELSE\n unionedgeom := st_union(unionedgeom, clippedgeom);\n END IF;\n\n unionedgeom_area := st_area(unionedgeom);\n\n IF skipcovered AND prev_area = unionedgeom_area THEN\n scancounter := scancounter + 1;\n CONTINUE;\n END IF;\n\n prev_area := unionedgeom_area;\n\n RAISE NOTICE '% % % %', unionedgeom_area/tilearea, counter, scancounter, ftime();\n END IF;\n RAISE NOTICE '% %', iter_record, content_hydrate(iter_record, fields);\n INSERT INTO pgstac_results (content) VALUES (content_hydrate(iter_record, fields));\n\n IF counter >= _limit\n OR scancounter > _scanlimit\n OR ftime() > _timelimit\n OR (exitwhenfull AND unionedgeom_area >= tilearea)\n THEN\n exit_flag := TRUE;\n EXIT;\n END IF;\n counter := counter + 1;\n scancounter := scancounter + 1;\n\n END LOOP;\n CLOSE curs;\n EXIT WHEN exit_flag;\n remaining_limit := _scanlimit - scancounter;\n END LOOP;\n\n SELECT jsonb_agg(content) INTO out_records FROM pgstac_results WHERE content IS NOT NULL;\n\n RETURN jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb)\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDROP FUNCTION IF EXISTS geojsonsearch;\nCREATE OR REPLACE FUNCTION geojsonsearch(\n IN geojson jsonb,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_geomfromgeojson(geojson),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS xyzsearch;\nCREATE OR REPLACE FUNCTION xyzsearch(\n IN _x int,\n IN _y int,\n IN _z int,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_transform(tileenvelope(_z, _x, _y), 4326),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nGRANT ALL ON SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac to pgstac_admin;\n\n-- pgstac_read role limited to using function apis\nGRANT EXECUTE ON FUNCTION search TO pgstac_read;\nGRANT EXECUTE ON FUNCTION search_query TO pgstac_read;\nGRANT EXECUTE ON FUNCTION item_by_id TO pgstac_read;\nGRANT EXECUTE ON FUNCTION get_item TO pgstac_read;\n\nGRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_ingest;\nGRANT USAGE ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_ingest;\nSELECT set_version('0.6.1');\n" }, { "path": "src/pgstac/migrations/pgstac.0.6.10-0.6.11.sql", "content": "SET SEARCH_PATH to pgstac, public;\nset check_function_bodies = off;\n\nUPDATE pgstac_settings SET name='default_filter_lang' WHERE name='default-filter-lang';\n\nCREATE OR REPLACE FUNCTION pgstac.stac_search_to_where(j jsonb)\n RETURNS text\n LANGUAGE plpgsql\n STABLE\nAS $function$\nDECLARE\n where_segments text[];\n _where text;\n dtrange tstzrange;\n collections text[];\n geom geometry;\n sdate timestamptz;\n edate timestamptz;\n filterlang text;\n filter jsonb := j->'filter';\nBEGIN\n IF j ? 'ids' THEN\n where_segments := where_segments || format('id = ANY (%L) ', to_text_array(j->'ids'));\n END IF;\n\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n where_segments := where_segments || format('collection = ANY (%L) ', collections);\n END IF;\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n\n where_segments := where_segments || format(' datetime <= %L::timestamptz AND end_datetime >= %L::timestamptz ',\n edate,\n sdate\n );\n END IF;\n\n geom := stac_geom(j);\n IF geom IS NOT NULL THEN\n where_segments := where_segments || format('st_intersects(geometry, %L)',geom);\n END IF;\n\n filterlang := COALESCE(\n j->>'filter-lang',\n get_setting('default_filter_lang', j->'conf')\n );\n IF NOT filter @? '$.**.op' THEN\n filterlang := 'cql-json';\n END IF;\n\n IF filterlang NOT IN ('cql-json','cql2-json') AND j ? 'filter' THEN\n RAISE EXCEPTION '% is not a supported filter-lang. Please use cql-json or cql2-json.', filterlang;\n END IF;\n\n IF j ? 'query' AND j ? 'filter' THEN\n RAISE EXCEPTION 'Can only use either query or filter at one time.';\n END IF;\n\n IF j ? 'query' THEN\n filter := query_to_cql2(j->'query');\n ELSIF filterlang = 'cql-json' THEN\n filter := cql1_to_cql2(filter);\n END IF;\n RAISE NOTICE 'FILTER: %', filter;\n where_segments := where_segments || cql2_query(filter);\n IF cardinality(where_segments) < 1 THEN\n RETURN ' TRUE ';\n END IF;\n\n _where := array_to_string(array_remove(where_segments, NULL), ' AND ');\n\n IF _where IS NULL OR BTRIM(_where) = '' THEN\n RETURN ' TRUE ';\n END IF;\n RETURN _where;\n\nEND;\n$function$\n;\n\n\n\nSELECT set_version('0.6.11');\n" }, { "path": "src/pgstac/migrations/pgstac.0.6.10.sql", "content": "CREATE EXTENSION IF NOT EXISTS postgis;\nCREATE EXTENSION IF NOT EXISTS btree_gist;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_admin;\n CREATE ROLE pgstac_read;\n CREATE ROLE pgstac_ingest;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nGRANT pgstac_admin TO current_user;\n\nCREATE SCHEMA IF NOT EXISTS pgstac AUTHORIZATION pgstac_admin;\n\nALTER ROLE pgstac_admin SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_read SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_ingest SET SEARCH_PATH TO pgstac, public;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\n\nGRANT pgstac_read TO pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\n\nSET ROLE pgstac_admin;\n\nSET SEARCH_PATH TO pgstac, public;\n\n\nCREATE TABLE IF NOT EXISTS migrations (\n version text PRIMARY KEY,\n datetime timestamptz DEFAULT clock_timestamp() NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION get_version() RETURNS text AS $$\n SELECT version FROM pgstac.migrations ORDER BY datetime DESC, version DESC LIMIT 1;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION set_version(text) RETURNS text AS $$\n INSERT INTO pgstac.migrations (version) VALUES ($1)\n ON CONFLICT DO NOTHING\n RETURNING version;\n$$ LANGUAGE SQL;\n\n\nCREATE TABLE IF NOT EXISTS pgstac_settings (\n name text PRIMARY KEY,\n value text NOT NULL\n);\n\nINSERT INTO pgstac_settings (name, value) VALUES\n ('context', 'off'),\n ('context_estimated_count', '100000'),\n ('context_estimated_cost', '100000'),\n ('context_stats_ttl', '1 day'),\n ('default-filter-lang', 'cql2-json'),\n ('additional_properties', 'true')\nON CONFLICT DO NOTHING\n;\n\n\nCREATE OR REPLACE FUNCTION get_setting(IN _setting text, IN conf jsonb DEFAULT NULL) RETURNS text AS $$\nSELECT COALESCE(\n conf->>_setting,\n current_setting(concat('pgstac.',_setting), TRUE),\n (SELECT value FROM pgstac.pgstac_settings WHERE name=_setting)\n);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context(conf jsonb DEFAULT NULL) RETURNS text AS $$\n SELECT pgstac.get_setting('context', conf);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context_estimated_count(conf jsonb DEFAULT NULL) RETURNS int AS $$\n SELECT pgstac.get_setting('context_estimated_count', conf)::int;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_estimated_cost();\nCREATE OR REPLACE FUNCTION context_estimated_cost(conf jsonb DEFAULT NULL) RETURNS float AS $$\n SELECT pgstac.get_setting('context_estimated_cost', conf)::float;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_stats_ttl();\nCREATE OR REPLACE FUNCTION context_stats_ttl(conf jsonb DEFAULT NULL) RETURNS interval AS $$\n SELECT pgstac.get_setting('context_stats_ttl', conf)::interval;\n$$ LANGUAGE SQL;\n\n\nCREATE OR REPLACE FUNCTION notice(VARIADIC text[]) RETURNS boolean AS $$\nDECLARE\ndebug boolean := current_setting('pgstac.debug', true);\nBEGIN\n IF debug THEN\n RAISE NOTICE 'NOTICE FROM FUNC: % >>>>> %', concat_ws(' | ', $1), clock_timestamp();\n RETURN TRUE;\n END IF;\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION empty_arr(ANYARRAY) RETURNS BOOLEAN AS $$\nSELECT CASE\n WHEN $1 IS NULL THEN TRUE\n WHEN cardinality($1)<1 THEN TRUE\nELSE FALSE\nEND;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_intersection(_a ANYARRAY, _b ANYARRAY) RETURNS ANYARRAY AS $$\n SELECT ARRAY ( SELECT unnest(_a) INTERSECT SELECT UNNEST(_b) );\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION array_map_ident(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_ident(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_map_literal(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_literal(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION array_reverse(anyarray) RETURNS anyarray AS $$\nSELECT ARRAY(\n SELECT $1[i]\n FROM generate_subscripts($1,1) AS s(i)\n ORDER BY i DESC\n);\n$$ LANGUAGE SQL STRICT IMMUTABLE;\nCREATE OR REPLACE FUNCTION to_int(jsonb) RETURNS int AS $$\n SELECT floor(($1->>0)::float)::int;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION to_float(jsonb) RETURNS float AS $$\n SELECT ($1->>0)::float;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION to_tstz(jsonb) RETURNS timestamptz AS $$\n SELECT ($1->>0)::timestamptz;\n$$ LANGUAGE SQL IMMUTABLE STRICT SET TIME ZONE 'UTC';\n\n\nCREATE OR REPLACE FUNCTION to_text(jsonb) RETURNS text AS $$\n SELECT CASE WHEN jsonb_typeof($1) IN ('array','object') THEN $1::text ELSE $1->>0 END;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION to_text_array(jsonb) RETURNS text[] AS $$\n SELECT\n CASE jsonb_typeof($1)\n WHEN 'array' THEN ARRAY(SELECT jsonb_array_elements_text($1))\n ELSE ARRAY[$1->>0]\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION bbox_geom(_bbox jsonb) RETURNS geometry AS $$\nSELECT CASE jsonb_array_length(_bbox)\n WHEN 4 THEN\n ST_SetSRID(ST_MakeEnvelope(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float,\n (_bbox->>3)::float\n ),4326)\n WHEN 6 THEN\n ST_SetSRID(ST_3DMakeBox(\n ST_MakePoint(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float\n ),\n ST_MakePoint(\n (_bbox->>3)::float,\n (_bbox->>4)::float,\n (_bbox->>5)::float\n )\n ),4326)\n ELSE null END;\n;\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION geom_bbox(_geom geometry) RETURNS jsonb AS $$\n SELECT jsonb_build_array(\n st_xmin(_geom),\n st_ymin(_geom),\n st_xmax(_geom),\n st_ymax(_geom)\n );\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION flip_jsonb_array(j jsonb) RETURNS jsonb AS $$\n SELECT jsonb_agg(value) FROM (SELECT value FROM jsonb_array_elements(j) WITH ORDINALITY ORDER BY ordinality DESC) as t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION explode_dotpaths(j jsonb) RETURNS SETOF text[] AS $$\n SELECT string_to_array(p, '.') as e FROM jsonb_array_elements_text(j) p;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION explode_dotpaths_recurse(IN j jsonb) RETURNS SETOF text[] AS $$\n WITH RECURSIVE t AS (\n SELECT e FROM explode_dotpaths(j) e\n UNION ALL\n SELECT e[1:cardinality(e)-1]\n FROM t\n WHERE cardinality(e)>1\n ) SELECT e FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION jsonb_set_nested(j jsonb, path text[], val jsonb) RETURNS jsonb AS $$\nDECLARE\nBEGIN\n IF cardinality(path) > 1 THEN\n FOR i IN 1..(cardinality(path)-1) LOOP\n IF j #> path[:i] IS NULL THEN\n j := jsonb_set_lax(j, path[:i], '{}', TRUE);\n END IF;\n END LOOP;\n END IF;\n RETURN jsonb_set_lax(j, path, val, true);\n\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\n\n\nCREATE OR REPLACE FUNCTION jsonb_include(j jsonb, f jsonb) RETURNS jsonb AS $$\nDECLARE\n includes jsonb := f-> 'include';\n outj jsonb := '{}'::jsonb;\n path text[];\nBEGIN\n IF\n includes IS NULL\n OR jsonb_array_length(includes) = 0\n THEN\n RETURN j;\n ELSE\n includes := includes || '[\"id\",\"collection\"]'::jsonb;\n FOR path IN SELECT explode_dotpaths(includes) LOOP\n outj := jsonb_set_nested(outj, path, j #> path);\n END LOOP;\n END IF;\n RETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_exclude(j jsonb, f jsonb) RETURNS jsonb AS $$\nDECLARE\n excludes jsonb := f-> 'exclude';\n outj jsonb := j;\n path text[];\nBEGIN\n IF\n excludes IS NULL\n OR jsonb_array_length(excludes) = 0\n THEN\n RETURN j;\n ELSE\n FOR path IN SELECT explode_dotpaths(excludes) LOOP\n outj := outj #- path;\n END LOOP;\n END IF;\n RETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_fields(j jsonb, f jsonb DEFAULT '{\"fields\":[]}') RETURNS jsonb AS $$\n SELECT jsonb_exclude(jsonb_include(j, f), f);\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION merge_jsonb(_a jsonb, _b jsonb) RETURNS jsonb AS $$\n SELECT\n CASE\n WHEN _a = '\"𒍟※\"'::jsonb THEN NULL\n WHEN _a IS NULL OR jsonb_typeof(_a) = 'null' THEN _b\n WHEN jsonb_typeof(_a) = 'object' AND jsonb_typeof(_b) = 'object' THEN\n (\n SELECT\n jsonb_strip_nulls(\n jsonb_object_agg(\n key,\n merge_jsonb(a.value, b.value)\n )\n )\n FROM\n jsonb_each(coalesce(_a,'{}'::jsonb)) as a\n FULL JOIN\n jsonb_each(coalesce(_b,'{}'::jsonb)) as b\n USING (key)\n )\n WHEN\n jsonb_typeof(_a) = 'array'\n AND jsonb_typeof(_b) = 'array'\n AND jsonb_array_length(_a) = jsonb_array_length(_b)\n THEN\n (\n SELECT jsonb_agg(m) FROM\n ( SELECT\n merge_jsonb(\n jsonb_array_elements(_a),\n jsonb_array_elements(_b)\n ) as m\n ) as l\n )\n ELSE _a\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION strip_jsonb(_a jsonb, _b jsonb) RETURNS jsonb AS $$\n SELECT\n CASE\n\n WHEN (_a IS NULL OR jsonb_typeof(_a) = 'null') AND _b IS NOT NULL AND jsonb_typeof(_b) != 'null' THEN '\"𒍟※\"'::jsonb\n WHEN _b IS NULL OR jsonb_typeof(_a) = 'null' THEN _a\n WHEN _a = _b AND jsonb_typeof(_a) = 'object' THEN '{}'::jsonb\n WHEN _a = _b THEN NULL\n WHEN jsonb_typeof(_a) = 'object' AND jsonb_typeof(_b) = 'object' THEN\n (\n SELECT\n jsonb_strip_nulls(\n jsonb_object_agg(\n key,\n strip_jsonb(a.value, b.value)\n )\n )\n FROM\n jsonb_each(_a) as a\n FULL JOIN\n jsonb_each(_b) as b\n USING (key)\n )\n WHEN\n jsonb_typeof(_a) = 'array'\n AND jsonb_typeof(_b) = 'array'\n AND jsonb_array_length(_a) = jsonb_array_length(_b)\n THEN\n (\n SELECT jsonb_agg(m) FROM\n ( SELECT\n strip_jsonb(\n jsonb_array_elements(_a),\n jsonb_array_elements(_b)\n ) as m\n ) as l\n )\n ELSE _a\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE;\n/* looks for a geometry in a stac item first from geometry and falling back to bbox */\nCREATE OR REPLACE FUNCTION stac_geom(value jsonb) RETURNS geometry AS $$\nSELECT\n CASE\n WHEN value ? 'intersects' THEN\n ST_GeomFromGeoJSON(value->>'intersects')\n WHEN value ? 'geometry' THEN\n ST_GeomFromGeoJSON(value->>'geometry')\n WHEN value ? 'bbox' THEN\n pgstac.bbox_geom(value->'bbox')\n ELSE NULL\n END as geometry\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION stac_daterange(\n value jsonb\n) RETURNS tstzrange AS $$\nDECLARE\n props jsonb := value;\n dt timestamptz;\n edt timestamptz;\nBEGIN\n IF props ? 'properties' THEN\n props := props->'properties';\n END IF;\n IF\n props ? 'start_datetime'\n AND props->>'start_datetime' IS NOT NULL\n AND props ? 'end_datetime'\n AND props->>'end_datetime' IS NOT NULL\n THEN\n dt := props->>'start_datetime';\n edt := props->>'end_datetime';\n IF dt > edt THEN\n RAISE EXCEPTION 'start_datetime must be < end_datetime';\n END IF;\n ELSE\n dt := props->>'datetime';\n edt := props->>'datetime';\n END IF;\n IF dt is NULL OR edt IS NULL THEN\n RAISE NOTICE 'DT: %, EDT: %', dt, edt;\n RAISE EXCEPTION 'Either datetime (%) or both start_datetime (%) and end_datetime (%) must be set.', props->>'datetime',props->>'start_datetime',props->>'end_datetime';\n END IF;\n RETURN tstzrange(dt, edt, '[]');\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_datetime(value jsonb) RETURNS timestamptz AS $$\n SELECT lower(stac_daterange(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_end_datetime(value jsonb) RETURNS timestamptz AS $$\n SELECT upper(stac_daterange(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\n\nCREATE TABLE IF NOT EXISTS stac_extensions(\n name text PRIMARY KEY,\n url text,\n enbabled_by_default boolean NOT NULL DEFAULT TRUE,\n enableable boolean NOT NULL DEFAULT TRUE\n);\n\nINSERT INTO stac_extensions (name, url) VALUES\n ('fields', 'https://api.stacspec.org/v1.0.0-beta.5/item-search#fields'),\n ('sort','https://api.stacspec.org/v1.0.0-beta.5/item-search#sort'),\n ('context','https://api.stacspec.org/v1.0.0-beta.5/item-search#context'),\n ('filter', 'https://api.stacspec.org/v1.0.0-beta.5/item-search#filter'),\n ('query', 'https://api.stacspec.org/v1.0.0-beta.5/item-search#query')\nON CONFLICT (name) DO UPDATE SET url=EXCLUDED.url;\n\n\n\nCREATE OR REPLACE FUNCTION collection_base_item(content jsonb) RETURNS jsonb AS $$\n SELECT jsonb_build_object(\n 'type', 'Feature',\n 'stac_version', content->'stac_version',\n 'assets', content->'item_assets',\n 'collection', content->'id'\n );\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TYPE partition_trunc_strategy AS ENUM ('year', 'month');\n\nCREATE TABLE IF NOT EXISTS collections (\n key bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,\n id text GENERATED ALWAYS AS (content->>'id') STORED UNIQUE,\n content JSONB NOT NULL,\n base_item jsonb GENERATED ALWAYS AS (pgstac.collection_base_item(content)) STORED,\n partition_trunc partition_trunc_strategy\n);\n\n\nCREATE OR REPLACE FUNCTION collection_base_item(cid text) RETURNS jsonb AS $$\n SELECT pgstac.collection_base_item(content) FROM pgstac.collections WHERE id = cid LIMIT 1;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION table_empty(text) RETURNS boolean AS $$\nDECLARE\n retval boolean;\nBEGIN\n EXECUTE format($q$\n SELECT NOT EXISTS (SELECT 1 FROM %I LIMIT 1)\n $q$,\n $1\n ) INTO retval;\n RETURN retval;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION collections_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n q text;\n partition_name text := format('_items_%s', NEW.key);\n partition_exists boolean := false;\n partition_empty boolean := true;\n err_context text;\n loadtemp boolean := FALSE;\nBEGIN\n RAISE NOTICE 'Collection Trigger. % %', NEW.id, NEW.key;\n SELECT relid::text INTO partition_name\n FROM pg_partition_tree('items')\n WHERE relid::text = partition_name;\n IF FOUND THEN\n partition_exists := true;\n partition_empty := table_empty(partition_name);\n ELSE\n partition_exists := false;\n partition_empty := true;\n partition_name := format('_items_%s', NEW.key);\n END IF;\n IF TG_OP = 'UPDATE' AND NEW.partition_trunc IS DISTINCT FROM OLD.partition_trunc AND partition_empty THEN\n q := format($q$\n DROP TABLE IF EXISTS %I CASCADE;\n $q$,\n partition_name\n );\n EXECUTE q;\n END IF;\n IF TG_OP = 'UPDATE' AND NEW.partition_trunc IS DISTINCT FROM OLD.partition_trunc AND partition_exists AND NOT partition_empty THEN\n q := format($q$\n CREATE TEMP TABLE changepartitionstaging ON COMMIT DROP AS SELECT * FROM %I;\n DROP TABLE IF EXISTS %I CASCADE;\n $q$,\n partition_name,\n partition_name\n );\n EXECUTE q;\n loadtemp := TRUE;\n partition_empty := TRUE;\n partition_exists := FALSE;\n END IF;\n IF TG_OP = 'UPDATE' AND NEW.partition_trunc IS NOT DISTINCT FROM OLD.partition_trunc THEN\n RETURN NEW;\n END IF;\n IF NEW.partition_trunc IS NULL AND partition_empty THEN\n RAISE NOTICE '% % % %',\n partition_name,\n NEW.id,\n concat(partition_name,'_id_idx'),\n partition_name\n ;\n q := format($q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n $q$,\n partition_name,\n NEW.id,\n concat(partition_name,'_id_idx'),\n partition_name\n );\n RAISE NOTICE 'q: %', q;\n BEGIN\n EXECUTE q;\n EXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', partition_name;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\n END;\n\n ALTER TABLE partitions DISABLE TRIGGER partitions_delete_trigger;\n DELETE FROM partitions WHERE collection=NEW.id AND name=partition_name;\n ALTER TABLE partitions ENABLE TRIGGER partitions_delete_trigger;\n\n INSERT INTO partitions (collection, name) VALUES (NEW.id, partition_name);\n ELSIF partition_empty THEN\n q := format($q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L)\n PARTITION BY RANGE (datetime);\n $q$,\n partition_name,\n NEW.id\n );\n RAISE NOTICE 'q: %', q;\n BEGIN\n EXECUTE q;\n EXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', partition_name;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\n END;\n ALTER TABLE partitions DISABLE TRIGGER partitions_delete_trigger;\n DELETE FROM partitions WHERE collection=NEW.id AND name=partition_name;\n ALTER TABLE partitions ENABLE TRIGGER partitions_delete_trigger;\n ELSE\n RAISE EXCEPTION 'Cannot modify partition % unless empty', partition_name;\n END IF;\n IF loadtemp THEN\n RAISE NOTICE 'Moving data into new partitions.';\n q := format($q$\n WITH p AS (\n SELECT\n collection,\n datetime as datetime,\n end_datetime as end_datetime,\n (partition_name(\n collection,\n datetime\n )).partition_name as name\n FROM changepartitionstaging\n )\n INSERT INTO partitions (collection, datetime_range, end_datetime_range)\n SELECT\n collection,\n tstzrange(min(datetime), max(datetime), '[]') as datetime_range,\n tstzrange(min(end_datetime), max(end_datetime), '[]') as end_datetime_range\n FROM p\n GROUP BY collection, name\n ON CONFLICT (name) DO UPDATE SET\n datetime_range = EXCLUDED.datetime_range,\n end_datetime_range = EXCLUDED.end_datetime_range\n ;\n INSERT INTO %I SELECT * FROM changepartitionstaging;\n DROP TABLE IF EXISTS changepartitionstaging;\n $q$,\n partition_name\n );\n EXECUTE q;\n END IF;\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE TRIGGER collections_trigger AFTER INSERT OR UPDATE ON collections FOR EACH ROW\nEXECUTE FUNCTION collections_trigger_func();\n\nCREATE OR REPLACE FUNCTION partition_collection(collection text, strategy partition_trunc_strategy) RETURNS text AS $$\n UPDATE collections SET partition_trunc=strategy WHERE id=collection RETURNING partition_trunc;\n$$ LANGUAGE SQL;\n\nCREATE TABLE IF NOT EXISTS partitions (\n collection text REFERENCES collections(id) ON DELETE CASCADE,\n name text PRIMARY KEY,\n partition_range tstzrange NOT NULL DEFAULT tstzrange('-infinity'::timestamptz,'infinity'::timestamptz, '[]'),\n datetime_range tstzrange,\n end_datetime_range tstzrange,\n CONSTRAINT prange EXCLUDE USING GIST (\n collection WITH =,\n partition_range WITH &&\n )\n) WITH (FILLFACTOR=90);\nCREATE INDEX partitions_range_idx ON partitions USING GIST(partition_range);\n\nCREATE OR REPLACE FUNCTION partitions_delete_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n q text;\nBEGIN\n RAISE NOTICE 'Partition Delete Trigger. %', OLD.name;\n EXECUTE format($q$\n DROP TABLE IF EXISTS %I CASCADE;\n $q$,\n OLD.name\n );\n RAISE NOTICE 'Dropped partition.';\n RETURN OLD;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER partitions_delete_trigger BEFORE DELETE ON partitions FOR EACH ROW\nEXECUTE FUNCTION partitions_delete_trigger_func();\n\nCREATE OR REPLACE FUNCTION partition_name(\n IN collection text,\n IN dt timestamptz,\n OUT partition_name text,\n OUT partition_range tstzrange\n) AS $$\nDECLARE\n c RECORD;\n parent_name text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n parent_name := format('_items_%s', c.key);\n\n\n IF c.partition_trunc = 'year' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYY'));\n ELSIF c.partition_trunc = 'month' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYYMM'));\n ELSE\n partition_name := parent_name;\n partition_range := tstzrange('-infinity'::timestamptz, 'infinity'::timestamptz, '[]');\n END IF;\n IF partition_range IS NULL THEN\n partition_range := tstzrange(\n date_trunc(c.partition_trunc::text, dt),\n date_trunc(c.partition_trunc::text, dt) + concat('1 ', c.partition_trunc)::interval\n );\n END IF;\n RETURN;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\n\nCREATE OR REPLACE FUNCTION partitions_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n q text;\n cq text;\n parent_name text;\n partition_trunc text;\n partition_name text := NEW.name;\n partition_exists boolean := false;\n partition_empty boolean := true;\n partition_range tstzrange;\n datetime_range tstzrange;\n end_datetime_range tstzrange;\n err_context text;\n mindt timestamptz := lower(NEW.datetime_range);\n maxdt timestamptz := upper(NEW.datetime_range);\n minedt timestamptz := lower(NEW.end_datetime_range);\n maxedt timestamptz := upper(NEW.end_datetime_range);\n t_mindt timestamptz;\n t_maxdt timestamptz;\n t_minedt timestamptz;\n t_maxedt timestamptz;\nBEGIN\n RAISE NOTICE 'Partitions Trigger. %', NEW;\n datetime_range := NEW.datetime_range;\n end_datetime_range := NEW.end_datetime_range;\n\n SELECT\n format('_items_%s', key),\n c.partition_trunc::text\n INTO\n parent_name,\n partition_trunc\n FROM pgstac.collections c\n WHERE c.id = NEW.collection;\n SELECT (pgstac.partition_name(NEW.collection, mindt)).* INTO partition_name, partition_range;\n NEW.name := partition_name;\n\n IF partition_range IS NULL OR partition_range = 'empty'::tstzrange THEN\n partition_range := tstzrange('-infinity'::timestamptz, 'infinity'::timestamptz, '[]');\n END IF;\n\n NEW.partition_range := partition_range;\n IF TG_OP = 'UPDATE' THEN\n mindt := least(mindt, lower(OLD.datetime_range));\n maxdt := greatest(maxdt, upper(OLD.datetime_range));\n minedt := least(minedt, lower(OLD.end_datetime_range));\n maxedt := greatest(maxedt, upper(OLD.end_datetime_range));\n NEW.datetime_range := tstzrange(mindt, maxdt, '[]');\n NEW.end_datetime_range := tstzrange(minedt, maxedt, '[]');\n END IF;\n IF TG_OP = 'INSERT' THEN\n\n IF partition_range != tstzrange('-infinity'::timestamptz, 'infinity'::timestamptz, '[]') THEN\n\n RAISE NOTICE '% % %', partition_name, parent_name, partition_range;\n q := format($q$\n CREATE TABLE IF NOT EXISTS %I partition OF %I FOR VALUES FROM (%L) TO (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n $q$,\n partition_name,\n parent_name,\n lower(partition_range),\n upper(partition_range),\n format('%s_pkey', partition_name),\n partition_name\n );\n BEGIN\n EXECUTE q;\n EXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', partition_name;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\n END;\n END IF;\n\n END IF;\n\n -- Update constraints\n EXECUTE format($q$\n SELECT\n min(datetime),\n max(datetime),\n min(end_datetime),\n max(end_datetime)\n FROM %I;\n $q$, partition_name)\n INTO t_mindt, t_maxdt, t_minedt, t_maxedt;\n mindt := least(mindt, t_mindt);\n maxdt := greatest(maxdt, t_maxdt);\n minedt := least(mindt, minedt, t_minedt);\n maxedt := greatest(maxdt, maxedt, t_maxedt);\n\n mindt := date_trunc(coalesce(partition_trunc, 'year'), mindt);\n maxdt := date_trunc(coalesce(partition_trunc, 'year'), maxdt - '1 second'::interval) + concat('1 ',coalesce(partition_trunc, 'year'))::interval;\n minedt := date_trunc(coalesce(partition_trunc, 'year'), minedt);\n maxedt := date_trunc(coalesce(partition_trunc, 'year'), maxedt - '1 second'::interval) + concat('1 ',coalesce(partition_trunc, 'year'))::interval;\n\n\n IF mindt IS NOT NULL AND maxdt IS NOT NULL AND minedt IS NOT NULL AND maxedt IS NOT NULL THEN\n NEW.datetime_range := tstzrange(mindt, maxdt, '[]');\n NEW.end_datetime_range := tstzrange(minedt, maxedt, '[]');\n IF\n TG_OP='UPDATE'\n AND OLD.datetime_range @> NEW.datetime_range\n AND OLD.end_datetime_range @> NEW.end_datetime_range\n THEN\n RAISE NOTICE 'Range unchanged, not updating constraints.';\n ELSE\n\n RAISE NOTICE '\n SETTING CONSTRAINTS\n mindt: %, maxdt: %\n minedt: %, maxedt: %\n ', mindt, maxdt, minedt, maxedt;\n IF partition_trunc IS NULL THEN\n cq := format($q$\n ALTER TABLE %7$I\n DROP CONSTRAINT IF EXISTS %1$I,\n DROP CONSTRAINT IF EXISTS %2$I,\n ADD CONSTRAINT %1$I\n CHECK (\n (datetime >= %3$L)\n AND (datetime <= %4$L)\n AND (end_datetime >= %5$L)\n AND (end_datetime <= %6$L)\n ) NOT VALID\n ;\n ALTER TABLE %7$I\n VALIDATE CONSTRAINT %1$I;\n $q$,\n format('%s_dt', partition_name),\n format('%s_edt', partition_name),\n mindt,\n maxdt,\n minedt,\n maxedt,\n partition_name\n );\n ELSE\n cq := format($q$\n ALTER TABLE %5$I\n DROP CONSTRAINT IF EXISTS %1$I,\n DROP CONSTRAINT IF EXISTS %2$I,\n ADD CONSTRAINT %2$I\n CHECK ((end_datetime >= %3$L) AND (end_datetime <= %4$L)) NOT VALID\n ;\n ALTER TABLE %5$I\n VALIDATE CONSTRAINT %2$I;\n $q$,\n format('%s_dt', partition_name),\n format('%s_edt', partition_name),\n minedt,\n maxedt,\n partition_name\n );\n\n END IF;\n RAISE NOTICE 'Altering Constraints. %', cq;\n EXECUTE cq;\n END IF;\n ELSE\n NEW.datetime_range = NULL;\n NEW.end_datetime_range = NULL;\n\n cq := format($q$\n ALTER TABLE %3$I\n DROP CONSTRAINT IF EXISTS %1$I,\n DROP CONSTRAINT IF EXISTS %2$I,\n ADD CONSTRAINT %1$I\n CHECK ((datetime IS NULL AND end_datetime IS NULL)) NOT VALID\n ;\n ALTER TABLE %3$I\n VALIDATE CONSTRAINT %1$I;\n $q$,\n format('%s_dt', partition_name),\n format('%s_edt', partition_name),\n partition_name\n );\n EXECUTE cq;\n END IF;\n\n RETURN NEW;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER partitions_trigger BEFORE INSERT OR UPDATE ON partitions FOR EACH ROW\nEXECUTE FUNCTION partitions_trigger_func();\n\n\nCREATE OR REPLACE FUNCTION create_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection(data jsonb) RETURNS VOID AS $$\nDECLARE\n out collections%ROWTYPE;\nBEGIN\n UPDATE collections SET content=data WHERE id = data->>'id' RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ON CONFLICT (id) DO\n UPDATE\n SET content=EXCLUDED.content\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION delete_collection(_id text) RETURNS VOID AS $$\nDECLARE\n out collections%ROWTYPE;\nBEGIN\n DELETE FROM collections WHERE id = _id RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION get_collection(id text) RETURNS jsonb AS $$\n SELECT content FROM collections\n WHERE id=$1\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION all_collections() RETURNS jsonb AS $$\n SELECT jsonb_agg(content) FROM collections;\n;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\nCREATE TABLE queryables (\n id bigint GENERATED ALWAYS AS identity PRIMARY KEY,\n name text UNIQUE NOT NULL,\n collection_ids text[], -- used to determine what partitions to create indexes on\n definition jsonb,\n property_path text,\n property_wrapper text,\n property_index_type text\n);\nCREATE INDEX queryables_name_idx ON queryables (name);\nCREATE INDEX queryables_property_wrapper_idx ON queryables (property_wrapper);\n\n\nINSERT INTO queryables (name, definition) VALUES\n('id', '{\"title\": \"Item ID\",\"description\": \"Item identifier\",\"$ref\": \"https://schemas.stacspec.org/v1.0.0/item-spec/json-schema/item.json#/definitions/core/allOf/2/properties/id\"}'),\n('datetime','{\"description\": \"Datetime\",\"type\": \"string\",\"title\": \"Acquired\",\"format\": \"date-time\",\"pattern\": \"(\\\\+00:00|Z)$\"}'),\n('geometry', '{\"title\": \"Item Geometry\",\"description\": \"Item Geometry\",\"$ref\": \"https://geojson.org/schema/Feature.json\"}')\nON CONFLICT DO NOTHING;\n\nINSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n('eo:cloud_cover','{\"$ref\": \"https://stac-extensions.github.io/eo/v1.0.0/schema.json#/definitions/fieldsproperties/eo:cloud_cover\"}','to_int','BTREE')\nON CONFLICT DO NOTHING;\n\nCREATE OR REPLACE FUNCTION array_to_path(arr text[]) RETURNS text AS $$\n SELECT string_agg(\n quote_literal(v),\n '->'\n ) FROM unnest(arr) v;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\n\n\nCREATE OR REPLACE FUNCTION queryable(\n IN dotpath text,\n OUT path text,\n OUT expression text,\n OUT wrapper text,\n OUT nulled_wrapper text\n) AS $$\nDECLARE\n q RECORD;\n path_elements text[];\nBEGIN\n IF dotpath IN ('id', 'geometry', 'datetime', 'end_datetime', 'collection') THEN\n path := dotpath;\n expression := dotpath;\n wrapper := NULL;\n RETURN;\n END IF;\n SELECT * INTO q FROM queryables\n WHERE\n name=dotpath\n OR name = 'properties.' || dotpath\n OR name = replace(dotpath, 'properties.', '')\n ;\n IF q.property_wrapper IS NULL THEN\n IF q.definition->>'type' = 'number' THEN\n wrapper := 'to_float';\n nulled_wrapper := wrapper;\n ELSIF q.definition->>'format' = 'date-time' THEN\n wrapper := 'to_tstz';\n nulled_wrapper := wrapper;\n ELSE\n nulled_wrapper := NULL;\n wrapper := 'to_text';\n END IF;\n ELSE\n wrapper := q.property_wrapper;\n nulled_wrapper := wrapper;\n END IF;\n IF q.property_path IS NOT NULL THEN\n path := q.property_path;\n ELSE\n path_elements := string_to_array(dotpath, '.');\n IF path_elements[1] IN ('links', 'assets', 'stac_version', 'stac_extensions') THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSIF path_elements[1] = 'properties' THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSE\n path := format($F$content->'properties'->%s$F$, array_to_path(path_elements));\n END IF;\n END IF;\n expression := format('%I(%s)', wrapper, path);\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\nCREATE OR REPLACE FUNCTION create_queryable_indexes() RETURNS VOID AS $$\nDECLARE\n queryable RECORD;\n q text;\nBEGIN\n FOR queryable IN\n SELECT\n queryables.id as qid,\n CASE WHEN collections.key IS NULL THEN 'items' ELSE format('_items_%s',collections.key) END AS part,\n property_index_type,\n expression\n FROM\n queryables\n LEFT JOIN collections ON (collections.id = ANY (queryables.collection_ids))\n JOIN LATERAL queryable(queryables.name) ON (queryables.property_index_type IS NOT NULL)\n LOOP\n q := format(\n $q$\n CREATE INDEX IF NOT EXISTS %I ON %I USING %s ((%s));\n $q$,\n format('%s_%s_idx', queryable.part, queryable.qid),\n queryable.part,\n COALESCE(queryable.property_index_type, 'to_text'),\n queryable.expression\n );\n RAISE NOTICE '%',q;\n EXECUTE q;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION queryables_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\nBEGIN\nPERFORM create_queryable_indexes();\nRETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER queryables_trigger AFTER INSERT OR UPDATE ON queryables\nFOR EACH STATEMENT EXECUTE PROCEDURE queryables_trigger_func();\n\nCREATE TRIGGER queryables_collection_trigger AFTER INSERT OR UPDATE ON collections\nFOR EACH STATEMENT EXECUTE PROCEDURE queryables_trigger_func();\n\nCREATE OR REPLACE FUNCTION get_queryables(_collection_ids text[] DEFAULT NULL) RETURNS jsonb AS $$\nBEGIN\n -- Build up queryables if the input contains valid collection ids or is empty\n IF EXISTS (\n SELECT 1 FROM collections\n WHERE\n _collection_ids IS NULL\n OR cardinality(_collection_ids) = 0\n OR id = ANY(_collection_ids)\n )\n THEN\n RETURN (\n SELECT\n jsonb_build_object(\n '$schema', 'http://json-schema.org/draft-07/schema#',\n '$id', 'https://example.org/queryables',\n 'type', 'object',\n 'title', 'STAC Queryables.',\n 'properties', jsonb_object_agg(\n name,\n definition\n )\n )\n FROM queryables\n WHERE\n _collection_ids IS NULL OR\n cardinality(_collection_ids) = 0 OR\n collection_ids IS NULL OR\n _collection_ids && collection_ids\n );\n ELSE\n RETURN NULL;\n END IF;\nEND;\n\n$$ LANGUAGE PLPGSQL STABLE;\n\nCREATE OR REPLACE FUNCTION get_queryables(_collection text DEFAULT NULL) RETURNS jsonb AS $$\n SELECT\n CASE\n WHEN _collection IS NULL THEN get_queryables(NULL::text[])\n ELSE get_queryables(ARRAY[_collection])\n END\n ;\n$$ LANGUAGE SQL;\n\n\nCREATE OR REPLACE FUNCTION missing_queryables(_collection text, _tablesample int DEFAULT 5) RETURNS TABLE(collection text, name text, definition jsonb, property_wrapper text) AS $$\nDECLARE\n q text;\n _partition text;\n explain_json json;\n psize bigint;\nBEGIN\n SELECT format('_items_%s', key) INTO _partition FROM collections WHERE id=_collection;\n\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM %I;', _partition)\n INTO explain_json;\n psize := explain_json->0->'Plan'->'Plan Rows';\n IF _tablesample * .01 * psize < 10 THEN\n _tablesample := 100;\n END IF;\n RAISE NOTICE 'Using tablesample % to find missing queryables from % % that has ~% rows', _tablesample, _collection, _partition, psize;\n\n q := format(\n $q$\n WITH q AS (\n SELECT * FROM queryables\n WHERE\n collection_ids IS NULL\n OR %L = ANY(collection_ids)\n ), t AS (\n SELECT\n content->'properties' AS properties\n FROM\n %I\n TABLESAMPLE SYSTEM(%L)\n ), p AS (\n SELECT DISTINCT ON (key)\n key,\n value\n FROM t\n JOIN LATERAL jsonb_each(properties) ON TRUE\n LEFT JOIN q ON (q.name=key)\n WHERE q.definition IS NULL\n )\n SELECT\n %L,\n key,\n jsonb_build_object('type',jsonb_typeof(value)) as definition,\n CASE jsonb_typeof(value)\n WHEN 'number' THEN 'to_float'\n WHEN 'array' THEN 'to_text_array'\n ELSE 'to_text'\n END\n FROM p;\n $q$,\n _collection,\n _partition,\n _tablesample,\n _collection\n );\n RETURN QUERY EXECUTE q;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION missing_queryables(_tablesample int DEFAULT 5) RETURNS TABLE(collection_ids text[], name text, definition jsonb, property_wrapper text) AS $$\n SELECT\n array_agg(collection),\n name,\n definition,\n property_wrapper\n FROM\n collections\n JOIN LATERAL\n missing_queryables(id, _tablesample) c\n ON TRUE\n GROUP BY\n 2,3,4\n ORDER BY 2,1\n ;\n$$ LANGUAGE SQL;\nCREATE OR REPLACE FUNCTION parse_dtrange(\n _indate jsonb,\n relative_base timestamptz DEFAULT date_trunc('hour', CURRENT_TIMESTAMP)\n) RETURNS tstzrange AS $$\nDECLARE\n timestrs text[];\n s timestamptz;\n e timestamptz;\nBEGIN\n timestrs :=\n CASE\n WHEN _indate ? 'timestamp' THEN\n ARRAY[_indate->>'timestamp']\n WHEN _indate ? 'interval' THEN\n to_text_array(_indate->'interval')\n WHEN jsonb_typeof(_indate) = 'array' THEN\n to_text_array(_indate)\n ELSE\n regexp_split_to_array(\n _indate->>0,\n '/'\n )\n END;\n RAISE NOTICE 'TIMESTRS %', timestrs;\n IF cardinality(timestrs) = 1 THEN\n IF timestrs[1] ILIKE 'P%' THEN\n RETURN tstzrange(relative_base - upper(timestrs[1])::interval, relative_base, '[)');\n END IF;\n s := timestrs[1]::timestamptz;\n RETURN tstzrange(s, s, '[]');\n END IF;\n\n IF cardinality(timestrs) != 2 THEN\n RAISE EXCEPTION 'Timestamp cannot have more than 2 values';\n END IF;\n\n IF timestrs[1] = '..' THEN\n s := '-infinity'::timestamptz;\n e := timestrs[2]::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] = '..' THEN\n s := timestrs[1]::timestamptz;\n e := 'infinity'::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[1] ILIKE 'P%' AND timestrs[2] NOT ILIKE 'P%' THEN\n e := timestrs[2]::timestamptz;\n s := e - upper(timestrs[1])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] ILIKE 'P%' AND timestrs[1] NOT ILIKE 'P%' THEN\n s := timestrs[1]::timestamptz;\n e := s + upper(timestrs[2])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n s := timestrs[1]::timestamptz;\n e := timestrs[2]::timestamptz;\n\n RETURN tstzrange(s,e,'[)');\n\n RETURN NULL;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT PARALLEL SAFE SET TIME ZONE 'UTC';\n\nCREATE OR REPLACE FUNCTION parse_dtrange(\n _indate text,\n relative_base timestamptz DEFAULT CURRENT_TIMESTAMP\n) RETURNS tstzrange AS $$\n SELECT parse_dtrange(to_jsonb(_indate), relative_base);\n$$ LANGUAGE SQL STABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION temporal_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\n ll text := 'datetime';\n lh text := 'end_datetime';\n rrange tstzrange;\n rl text;\n rh text;\n outq text;\nBEGIN\n rrange := parse_dtrange(args->1);\n RAISE NOTICE 'Constructing temporal query OP: %, ARGS: %, RRANGE: %', op, args, rrange;\n op := lower(op);\n rl := format('%L::timestamptz', lower(rrange));\n rh := format('%L::timestamptz', upper(rrange));\n outq := CASE op\n WHEN 't_before' THEN 'lh < rl'\n WHEN 't_after' THEN 'll > rh'\n WHEN 't_meets' THEN 'lh = rl'\n WHEN 't_metby' THEN 'll = rh'\n WHEN 't_overlaps' THEN 'll < rl AND rl < lh < rh'\n WHEN 't_overlappedby' THEN 'rl < ll < rh AND lh > rh'\n WHEN 't_starts' THEN 'll = rl AND lh < rh'\n WHEN 't_startedby' THEN 'll = rl AND lh > rh'\n WHEN 't_during' THEN 'll > rl AND lh < rh'\n WHEN 't_contains' THEN 'll < rl AND lh > rh'\n WHEN 't_finishes' THEN 'll > rl AND lh = rh'\n WHEN 't_finishedby' THEN 'll < rl AND lh = rh'\n WHEN 't_equals' THEN 'll = rl AND lh = rh'\n WHEN 't_disjoint' THEN 'NOT (ll <= rh AND lh >= rl)'\n WHEN 't_intersects' THEN 'll <= rh AND lh >= rl'\n WHEN 'anyinteracts' THEN 'll <= rh AND lh >= rl'\n END;\n outq := regexp_replace(outq, '\\mll\\M', ll);\n outq := regexp_replace(outq, '\\mlh\\M', lh);\n outq := regexp_replace(outq, '\\mrl\\M', rl);\n outq := regexp_replace(outq, '\\mrh\\M', rh);\n outq := format('(%s)', outq);\n RETURN outq;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\n\n\nCREATE OR REPLACE FUNCTION spatial_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\n geom text;\n j jsonb := args->1;\nBEGIN\n op := lower(op);\n RAISE NOTICE 'Constructing spatial query OP: %, ARGS: %', op, args;\n IF op NOT IN ('s_equals','s_disjoint','s_touches','s_within','s_overlaps','s_crosses','s_intersects','intersects','s_contains') THEN\n RAISE EXCEPTION 'Spatial Operator % Not Supported', op;\n END IF;\n op := regexp_replace(op, '^s_', 'st_');\n IF op = 'intersects' THEN\n op := 'st_intersects';\n END IF;\n -- Convert geometry to WKB string\n IF j ? 'type' AND j ? 'coordinates' THEN\n geom := st_geomfromgeojson(j)::text;\n ELSIF jsonb_typeof(j) = 'array' THEN\n geom := bbox_geom(j)::text;\n END IF;\n\n RETURN format('%s(geometry, %L::geometry)', op, geom);\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION query_to_cql2(q jsonb) RETURNS jsonb AS $$\n-- Translates anything passed in through the deprecated \"query\" into equivalent CQL2\nWITH t AS (\n SELECT key as property, value as ops\n FROM jsonb_each(q)\n), t2 AS (\n SELECT property, (jsonb_each(ops)).*\n FROM t WHERE jsonb_typeof(ops) = 'object'\n UNION ALL\n SELECT property, 'eq', ops\n FROM t WHERE jsonb_typeof(ops) != 'object'\n)\nSELECT\n jsonb_strip_nulls(jsonb_build_object(\n 'op', 'and',\n 'args', jsonb_agg(\n jsonb_build_object(\n 'op', key,\n 'args', jsonb_build_array(\n jsonb_build_object('property',property),\n value\n )\n )\n )\n )\n) as qcql FROM t2\n;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\nCREATE OR REPLACE FUNCTION cql1_to_cql2(j jsonb) RETURNS jsonb AS $$\nDECLARE\n args jsonb;\n ret jsonb;\nBEGIN\n RAISE NOTICE 'CQL1_TO_CQL2: %', j;\n IF j ? 'filter' THEN\n RETURN cql1_to_cql2(j->'filter');\n END IF;\n IF j ? 'property' THEN\n RETURN j;\n END IF;\n IF jsonb_typeof(j) = 'array' THEN\n SELECT jsonb_agg(cql1_to_cql2(el)) INTO args FROM jsonb_array_elements(j) el;\n RETURN args;\n END IF;\n IF jsonb_typeof(j) = 'number' THEN\n RETURN j;\n END IF;\n IF jsonb_typeof(j) = 'string' THEN\n RETURN j;\n END IF;\n\n IF jsonb_typeof(j) = 'object' THEN\n SELECT jsonb_build_object(\n 'op', key,\n 'args', cql1_to_cql2(value)\n ) INTO ret\n FROM jsonb_each(j)\n WHERE j IS NOT NULL;\n RETURN ret;\n END IF;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE STRICT;\n\nCREATE TABLE cql2_ops (\n op text PRIMARY KEY,\n template text,\n types text[]\n);\nINSERT INTO cql2_ops (op, template, types) VALUES\n ('eq', '%s = %s', NULL),\n ('neq', '%s != %s', NULL),\n ('ne', '%s != %s', NULL),\n ('!=', '%s != %s', NULL),\n ('<>', '%s != %s', NULL),\n ('lt', '%s < %s', NULL),\n ('lte', '%s <= %s', NULL),\n ('gt', '%s > %s', NULL),\n ('gte', '%s >= %s', NULL),\n ('le', '%s <= %s', NULL),\n ('ge', '%s >= %s', NULL),\n ('=', '%s = %s', NULL),\n ('<', '%s < %s', NULL),\n ('<=', '%s <= %s', NULL),\n ('>', '%s > %s', NULL),\n ('>=', '%s >= %s', NULL),\n ('like', '%s LIKE %s', NULL),\n ('ilike', '%s ILIKE %s', NULL),\n ('+', '%s + %s', NULL),\n ('-', '%s - %s', NULL),\n ('*', '%s * %s', NULL),\n ('/', '%s / %s', NULL),\n ('not', 'NOT (%s)', NULL),\n ('between', '%s BETWEEN %s AND %s', NULL),\n ('isnull', '%s IS NULL', NULL),\n ('upper', 'upper(%s)', NULL),\n ('lower', 'lower(%s)', NULL)\nON CONFLICT (op) DO UPDATE\n SET\n template = EXCLUDED.template\n;\n\n\nCREATE OR REPLACE FUNCTION cql2_query(j jsonb, wrapper text DEFAULT NULL) RETURNS text AS $$\n#variable_conflict use_variable\nDECLARE\n args jsonb := j->'args';\n arg jsonb;\n op text := lower(j->>'op');\n cql2op RECORD;\n literal text;\n _wrapper text;\n leftarg text;\n rightarg text;\nBEGIN\n IF j IS NULL OR (op IS NOT NULL AND args IS NULL) THEN\n RETURN NULL;\n END IF;\n RAISE NOTICE 'CQL2_QUERY: %', j;\n IF j ? 'filter' THEN\n RETURN cql2_query(j->'filter');\n END IF;\n\n IF j ? 'upper' THEN\n RETURN cql2_query(jsonb_build_object('op', 'upper', 'args', j->'upper'));\n END IF;\n\n IF j ? 'lower' THEN\n RETURN cql2_query(jsonb_build_object('op', 'lower', 'args', j->'lower'));\n END IF;\n\n -- Temporal Query\n IF op ilike 't_%' or op = 'anyinteracts' THEN\n RETURN temporal_op_query(op, args);\n END IF;\n\n -- If property is a timestamp convert it to text to use with\n -- general operators\n IF j ? 'timestamp' THEN\n RETURN format('%L::timestamptz', to_tstz(j->'timestamp'));\n END IF;\n IF j ? 'interval' THEN\n RAISE EXCEPTION 'Please use temporal operators when using intervals.';\n RETURN NONE;\n END IF;\n\n -- Spatial Query\n IF op ilike 's_%' or op = 'intersects' THEN\n RETURN spatial_op_query(op, args);\n END IF;\n\n IF op IN ('a_equals','a_contains','a_contained_by','a_overlaps') THEN\n IF args->0 ? 'property' THEN\n leftarg := format('to_text_array(%s)', (queryable(args->0->>'property')).path);\n END IF;\n IF args->1 ? 'property' THEN\n rightarg := format('to_text_array(%s)', (queryable(args->1->>'property')).path);\n END IF;\n RETURN FORMAT(\n '%s %s %s',\n COALESCE(leftarg, quote_literal(to_text_array(args->0))),\n CASE op\n WHEN 'a_equals' THEN '='\n WHEN 'a_contains' THEN '@>'\n WHEN 'a_contained_by' THEN '<@'\n WHEN 'a_overlaps' THEN '&&'\n END,\n COALESCE(rightarg, quote_literal(to_text_array(args->1)))\n );\n END IF;\n\n IF op = 'in' THEN\n RAISE NOTICE 'IN : % % %', args, jsonb_build_array(args->0), args->1;\n args := jsonb_build_array(args->0) || (args->1);\n RAISE NOTICE 'IN2 : %', args;\n END IF;\n\n\n\n IF op = 'between' THEN\n args = jsonb_build_array(\n args->0,\n args->1->0,\n args->1->1\n );\n END IF;\n\n -- Make sure that args is an array and run cql2_query on\n -- each element of the array\n RAISE NOTICE 'ARGS PRE: %', args;\n IF j ? 'args' THEN\n IF jsonb_typeof(args) != 'array' THEN\n args := jsonb_build_array(args);\n END IF;\n\n IF jsonb_path_exists(args, '$[*] ? (@.property == \"id\" || @.property == \"datetime\" || @.property == \"end_datetime\" || @.property == \"collection\")') THEN\n wrapper := NULL;\n ELSE\n -- if any of the arguments are a property, try to get the property_wrapper\n FOR arg IN SELECT jsonb_path_query(args, '$[*] ? (@.property != null)') LOOP\n RAISE NOTICE 'Arg: %', arg;\n wrapper := (queryable(arg->>'property')).nulled_wrapper;\n RAISE NOTICE 'Property: %, Wrapper: %', arg, wrapper;\n IF wrapper IS NOT NULL THEN\n EXIT;\n END IF;\n END LOOP;\n\n -- if the property was not in queryables, see if any args were numbers\n IF\n wrapper IS NULL\n AND jsonb_path_exists(args, '$[*] ? (@.type()==\"number\")')\n THEN\n wrapper := 'to_float';\n END IF;\n wrapper := coalesce(wrapper, 'to_text');\n END IF;\n\n SELECT jsonb_agg(cql2_query(a, wrapper))\n INTO args\n FROM jsonb_array_elements(args) a;\n END IF;\n RAISE NOTICE 'ARGS: %', args;\n\n IF op IN ('and', 'or') THEN\n RETURN\n format(\n '(%s)',\n array_to_string(to_text_array(args), format(' %s ', upper(op)))\n );\n END IF;\n\n IF op = 'in' THEN\n RAISE NOTICE 'IN -- % %', args->0, to_text(args->0);\n RETURN format(\n '%s IN (%s)',\n to_text(args->0),\n array_to_string((to_text_array(args))[2:], ',')\n );\n END IF;\n\n -- Look up template from cql2_ops\n IF j ? 'op' THEN\n SELECT * INTO cql2op FROM cql2_ops WHERE cql2_ops.op ilike op;\n IF FOUND THEN\n -- If specific index set in queryables for a property cast other arguments to that type\n\n RETURN format(\n cql2op.template,\n VARIADIC (to_text_array(args))\n );\n ELSE\n RAISE EXCEPTION 'Operator % Not Supported.', op;\n END IF;\n END IF;\n\n\n IF wrapper IS NOT NULL THEN\n RAISE NOTICE 'Wrapping % with %', j, wrapper;\n IF j ? 'property' THEN\n RETURN format('%I(%s)', wrapper, (queryable(j->>'property')).path);\n ELSE\n RETURN format('%I(%L)', wrapper, j);\n END IF;\n ELSIF j ? 'property' THEN\n RETURN quote_ident(j->>'property');\n END IF;\n\n RETURN quote_literal(to_text(j));\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION paging_dtrange(\n j jsonb\n) RETURNS tstzrange AS $$\nDECLARE\n op text;\n filter jsonb := j->'filter';\n dtrange tstzrange := tstzrange('-infinity'::timestamptz,'infinity'::timestamptz);\n sdate timestamptz := '-infinity'::timestamptz;\n edate timestamptz := 'infinity'::timestamptz;\n jpitem jsonb;\nBEGIN\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n END IF;\n IF NOT (filter @? '$.**.op ? (@ == \"or\" || @ == \"not\")') THEN\n FOR jpitem IN SELECT j FROM jsonb_path_query(filter,'strict $.** ? (@.args[*].property == \"datetime\")'::jsonpath) j LOOP\n op := lower(jpitem->>'op');\n dtrange := parse_dtrange(jpitem->'args'->1);\n IF op IN ('<=', 'lt', 'lte', '<', 'le', 't_before') THEN\n sdate := greatest(sdate,'-infinity');\n edate := least(edate, upper(dtrange));\n ELSIF op IN ('>=', '>', 'gt', 'gte', 'ge', 't_after') THEN\n edate := least(edate, 'infinity');\n sdate := greatest(sdate, lower(dtrange));\n ELSIF op IN ('=', 'eq') THEN\n edate := least(edate, upper(dtrange));\n sdate := greatest(sdate, lower(dtrange));\n END IF;\n RAISE NOTICE '2 OP: %, ARGS: %, DTRANGE: %, SDATE: %, EDATE: %', op, jpitem->'args'->1, dtrange, sdate, edate;\n END LOOP;\n END IF;\n IF sdate > edate THEN\n RETURN 'empty'::tstzrange;\n END IF;\n RETURN tstzrange(sdate,edate, '[]');\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT SET TIME ZONE 'UTC';\n\nCREATE OR REPLACE FUNCTION paging_collections(\n IN j jsonb\n) RETURNS text[] AS $$\nDECLARE\n filter jsonb := j->'filter';\n jpitem jsonb;\n op text;\n args jsonb;\n arg jsonb;\n collections text[];\nBEGIN\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n END IF;\n IF NOT (filter @? '$.**.op ? (@ == \"or\" || @ == \"not\")') THEN\n FOR jpitem IN SELECT j FROM jsonb_path_query(filter,'strict $.** ? (@.args[*].property == \"collection\")'::jsonpath) j LOOP\n RAISE NOTICE 'JPITEM: %', jpitem;\n op := jpitem->>'op';\n args := jpitem->'args';\n IF op IN ('=', 'eq', 'in') THEN\n FOR arg IN SELECT a FROM jsonb_array_elements(args) a LOOP\n IF jsonb_typeof(arg) IN ('string', 'array') THEN\n RAISE NOTICE 'arg: %, collections: %', arg, collections;\n IF collections IS NULL OR collections = '{}'::text[] THEN\n collections := to_text_array(arg);\n ELSE\n collections := array_intersection(collections, to_text_array(arg));\n END IF;\n END IF;\n END LOOP;\n END IF;\n END LOOP;\n END IF;\n IF collections = '{}'::text[] THEN\n RETURN NULL;\n END IF;\n RETURN collections;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\nCREATE TABLE items (\n id text NOT NULL,\n geometry geometry NOT NULL,\n collection text NOT NULL,\n datetime timestamptz NOT NULL,\n end_datetime timestamptz NOT NULL,\n content JSONB NOT NULL\n)\nPARTITION BY LIST (collection)\n;\n\nCREATE INDEX \"datetime_idx\" ON items USING BTREE (datetime DESC, end_datetime ASC);\nCREATE INDEX \"geometry_idx\" ON items USING GIST (geometry);\n\nCREATE STATISTICS datetime_stats (dependencies) on datetime, end_datetime from items;\n\n\nALTER TABLE items ADD CONSTRAINT items_collections_fk FOREIGN KEY (collection) REFERENCES collections(id) ON DELETE CASCADE DEFERRABLE;\n\n\nCREATE OR REPLACE FUNCTION content_slim(_item jsonb) RETURNS jsonb AS $$\n SELECT strip_jsonb(_item - '{id,geometry,collection,type}'::text[], collection_base_item(_item->>'collection')) - '{id,geometry,collection,type}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_dehydrate(content jsonb) RETURNS items AS $$\n SELECT\n content->>'id' as id,\n stac_geom(content) as geometry,\n content->>'collection' as collection,\n stac_datetime(content) as datetime,\n stac_end_datetime(content) as end_datetime,\n content_slim(content) as content\n ;\n$$ LANGUAGE SQL STABLE;\n\nCREATE OR REPLACE FUNCTION include_field(f text, fields jsonb DEFAULT '{}'::jsonb) RETURNS boolean AS $$\nDECLARE\n includes jsonb := fields->'include';\n excludes jsonb := fields->'exclude';\nBEGIN\n IF f IS NULL THEN\n RETURN NULL;\n END IF;\n\n\n IF\n jsonb_typeof(excludes) = 'array'\n AND jsonb_array_length(excludes)>0\n AND excludes ? f\n THEN\n RETURN FALSE;\n END IF;\n\n IF\n (\n jsonb_typeof(includes) = 'array'\n AND jsonb_array_length(includes) > 0\n AND includes ? f\n ) OR\n (\n includes IS NULL\n OR jsonb_typeof(includes) = 'null'\n OR jsonb_array_length(includes) = 0\n )\n THEN\n RETURN TRUE;\n END IF;\n\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nDROP FUNCTION IF EXISTS content_hydrate(jsonb, jsonb, jsonb);\nCREATE OR REPLACE FUNCTION content_hydrate(\n _item jsonb,\n _base_item jsonb,\n fields jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\n SELECT merge_jsonb(\n jsonb_fields(_item, fields),\n jsonb_fields(_base_item, fields)\n );\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION content_hydrate(_item items, _collection collections, fields jsonb DEFAULT '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\n content jsonb;\n base_item jsonb := _collection.base_item;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry, 20)::jsonb;\n END IF;\n output := content_hydrate(\n jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'collection', _item.collection,\n 'type', 'Feature'\n ) || _item.content,\n _collection.base_item,\n fields\n );\n\n RETURN output;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_nonhydrated(\n _item items,\n fields jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry, 20)::jsonb;\n END IF;\n output := jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'collection', _item.collection,\n 'type', 'Feature'\n ) || _item.content;\n RETURN output;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_hydrate(_item items, fields jsonb DEFAULT '{}'::jsonb) RETURNS jsonb AS $$\n SELECT content_hydrate(\n _item,\n (SELECT c FROM collections c WHERE id=_item.collection LIMIT 1),\n fields\n );\n$$ LANGUAGE SQL STABLE;\n\n\nCREATE UNLOGGED TABLE items_staging (\n content JSONB NOT NULL\n);\nCREATE UNLOGGED TABLE items_staging_ignore (\n content JSONB NOT NULL\n);\nCREATE UNLOGGED TABLE items_staging_upsert (\n content JSONB NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION items_staging_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p record;\n _partitions text[];\n ts timestamptz := clock_timestamp();\nBEGIN\n RAISE NOTICE 'Creating Partitions. %', clock_timestamp() - ts;\n WITH ranges AS (\n SELECT\n n.content->>'collection' as collection,\n stac_daterange(n.content->'properties') as dtr\n FROM newdata n\n ), p AS (\n SELECT\n collection,\n lower(dtr) as datetime,\n upper(dtr) as end_datetime,\n (partition_name(\n collection,\n lower(dtr)\n )).partition_name as name\n FROM ranges\n )\n INSERT INTO partitions (collection, datetime_range, end_datetime_range)\n SELECT\n collection,\n tstzrange(min(datetime), max(datetime), '[]') as datetime_range,\n tstzrange(min(end_datetime), max(end_datetime), '[]') as end_datetime_range\n FROM p\n GROUP BY collection, name\n ON CONFLICT (name) DO UPDATE SET\n datetime_range = EXCLUDED.datetime_range,\n end_datetime_range = EXCLUDED.end_datetime_range\n ;\n\n RAISE NOTICE 'Doing the insert. %', clock_timestamp() - ts;\n IF TG_TABLE_NAME = 'items_staging' THEN\n INSERT INTO items\n SELECT\n (content_dehydrate(content)).*\n FROM newdata;\n DELETE FROM items_staging;\n ELSIF TG_TABLE_NAME = 'items_staging_ignore' THEN\n INSERT INTO items\n SELECT\n (content_dehydrate(content)).*\n FROM newdata\n ON CONFLICT DO NOTHING;\n DELETE FROM items_staging_ignore;\n ELSIF TG_TABLE_NAME = 'items_staging_upsert' THEN\n WITH staging_formatted AS (\n SELECT (content_dehydrate(content)).* FROM newdata\n ), deletes AS (\n DELETE FROM items i USING staging_formatted s\n WHERE\n i.id = s.id\n AND i.collection = s.collection\n AND i IS DISTINCT FROM s\n RETURNING i.id, i.collection\n )\n INSERT INTO items\n SELECT s.* FROM\n staging_formatted s\n JOIN deletes d\n USING (id, collection);\n DELETE FROM items_staging_upsert;\n END IF;\n RAISE NOTICE 'Done. %', clock_timestamp() - ts;\n\n RETURN NULL;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER items_staging_insert_trigger AFTER INSERT ON items_staging REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\nCREATE TRIGGER items_staging_insert_ignore_trigger AFTER INSERT ON items_staging_ignore REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\nCREATE TRIGGER items_staging_insert_upsert_trigger AFTER INSERT ON items_staging_upsert REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\n\n\n\nCREATE OR REPLACE FUNCTION item_by_id(_id text, _collection text DEFAULT NULL) RETURNS items AS\n$$\nDECLARE\n i items%ROWTYPE;\nBEGIN\n SELECT * INTO i FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection) LIMIT 1;\n RETURN i;\nEND;\n$$ LANGUAGE PLPGSQL STABLE SECURITY DEFINER SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION get_item(_id text, _collection text DEFAULT NULL) RETURNS jsonb AS $$\n SELECT content_hydrate(items) FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection);\n$$ LANGUAGE SQL STABLE SECURITY DEFINER SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION delete_item(_id text, _collection text DEFAULT NULL) RETURNS VOID AS $$\nDECLARE\nout items%ROWTYPE;\nBEGIN\n DELETE FROM items WHERE id = _id AND (_collection IS NULL OR collection=_collection) RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n--/*\nCREATE OR REPLACE FUNCTION create_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION update_item(content jsonb) RETURNS VOID AS $$\nDECLARE\n old items %ROWTYPE;\n out items%ROWTYPE;\nBEGIN\n PERFORM delete_item(content->>'id', content->>'collection');\n PERFORM create_item(content);\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION create_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION collection_bbox(id text) RETURNS jsonb AS $$\n SELECT (replace(replace(replace(st_extent(geometry)::text,'BOX(','[['),')',']]'),' ',','))::jsonb\n FROM items WHERE collection=$1;\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION collection_temporal_extent(id text) RETURNS jsonb AS $$\n SELECT to_jsonb(array[array[min(datetime)::text, max(datetime)::text]])\n FROM items WHERE collection=$1;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection_extents() RETURNS VOID AS $$\nUPDATE collections SET\n content = content ||\n jsonb_build_object(\n 'extent', jsonb_build_object(\n 'spatial', jsonb_build_object(\n 'bbox', collection_bbox(collections.id)\n ),\n 'temporal', jsonb_build_object(\n 'interval', collection_temporal_extent(collections.id)\n )\n )\n )\n;\n$$ LANGUAGE SQL;\nCREATE VIEW partition_steps AS\nSELECT\n name,\n date_trunc('month',lower(datetime_range)) as sdate,\n date_trunc('month', upper(datetime_range)) + '1 month'::interval as edate\n FROM partitions WHERE datetime_range IS NOT NULL AND datetime_range != 'empty'::tstzrange\n ORDER BY datetime_range ASC\n;\n\nCREATE OR REPLACE FUNCTION chunker(\n IN _where text,\n OUT s timestamptz,\n OUT e timestamptz\n) RETURNS SETOF RECORD AS $$\nDECLARE\n explain jsonb;\nBEGIN\n IF _where IS NULL THEN\n _where := ' TRUE ';\n END IF;\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s;', _where)\n INTO explain;\n\n RETURN QUERY\n WITH t AS (\n SELECT j->>0 as p FROM\n jsonb_path_query(\n explain,\n 'strict $.**.\"Relation Name\" ? (@ != null)'\n ) j\n ),\n parts AS (\n SELECT sdate, edate FROM t JOIN partition_steps ON (t.p = name)\n ),\n times AS (\n SELECT sdate FROM parts\n UNION\n SELECT edate FROM parts\n ),\n uniq AS (\n SELECT DISTINCT sdate FROM times ORDER BY sdate\n ),\n last AS (\n SELECT sdate, lead(sdate, 1) over () as edate FROM uniq\n )\n SELECT sdate, edate FROM last WHERE edate IS NOT NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION partition_queries(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN partitions text[] DEFAULT NULL\n) RETURNS SETOF text AS $$\nDECLARE\n query text;\n sdate timestamptz;\n edate timestamptz;\nBEGIN\nIF _where IS NULL OR trim(_where) = '' THEN\n _where = ' TRUE ';\nEND IF;\nRAISE NOTICE 'Getting chunks for % %', _where, _orderby;\nIF _orderby ILIKE 'datetime d%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 DESC LOOP\n RETURN NEXT format($q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n $q$,\n sdate,\n edate,\n _where,\n _orderby\n );\n END LOOP;\nELSIF _orderby ILIKE 'datetime a%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 ASC LOOP\n RETURN NEXT format($q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n $q$,\n sdate,\n edate,\n _where,\n _orderby\n );\n END LOOP;\nELSE\n query := format($q$\n SELECT * FROM items\n WHERE %s\n ORDER BY %s\n $q$, _where, _orderby\n );\n\n RETURN NEXT query;\n RETURN;\nEND IF;\n\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION partition_query_view(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN _limit int DEFAULT 10\n) RETURNS text AS $$\n WITH p AS (\n SELECT * FROM partition_queries(_where, _orderby) p\n )\n SELECT\n CASE WHEN EXISTS (SELECT 1 FROM p) THEN\n (SELECT format($q$\n SELECT * FROM (\n %s\n ) total LIMIT %s\n $q$,\n string_agg(\n format($q$ SELECT * FROM ( %s ) AS sub $q$, p),\n '\n UNION ALL\n '\n ),\n _limit\n ))\n ELSE NULL\n END FROM p;\n$$ LANGUAGE SQL IMMUTABLE;\n\n\n\n\nCREATE OR REPLACE FUNCTION stac_search_to_where(j jsonb) RETURNS text AS $$\nDECLARE\n where_segments text[];\n _where text;\n dtrange tstzrange;\n collections text[];\n geom geometry;\n sdate timestamptz;\n edate timestamptz;\n filterlang text;\n filter jsonb := j->'filter';\nBEGIN\n IF j ? 'ids' THEN\n where_segments := where_segments || format('id = ANY (%L) ', to_text_array(j->'ids'));\n END IF;\n\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n where_segments := where_segments || format('collection = ANY (%L) ', collections);\n END IF;\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n\n where_segments := where_segments || format(' datetime <= %L::timestamptz AND end_datetime >= %L::timestamptz ',\n edate,\n sdate\n );\n END IF;\n\n geom := stac_geom(j);\n IF geom IS NOT NULL THEN\n where_segments := where_segments || format('st_intersects(geometry, %L)',geom);\n END IF;\n\n filterlang := COALESCE(\n j->>'filter-lang',\n get_setting('default-filter-lang', j->'conf')\n );\n IF NOT filter @? '$.**.op' THEN\n filterlang := 'cql-json';\n END IF;\n\n IF filterlang NOT IN ('cql-json','cql2-json') AND j ? 'filter' THEN\n RAISE EXCEPTION '% is not a supported filter-lang. Please use cql-json or cql2-json.', filterlang;\n END IF;\n\n IF j ? 'query' AND j ? 'filter' THEN\n RAISE EXCEPTION 'Can only use either query or filter at one time.';\n END IF;\n\n IF j ? 'query' THEN\n filter := query_to_cql2(j->'query');\n ELSIF filterlang = 'cql-json' THEN\n filter := cql1_to_cql2(filter);\n END IF;\n RAISE NOTICE 'FILTER: %', filter;\n where_segments := where_segments || cql2_query(filter);\n IF cardinality(where_segments) < 1 THEN\n RETURN ' TRUE ';\n END IF;\n\n _where := array_to_string(array_remove(where_segments, NULL), ' AND ');\n\n IF _where IS NULL OR BTRIM(_where) = '' THEN\n RETURN ' TRUE ';\n END IF;\n RETURN _where;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION parse_sort_dir(_dir text, reverse boolean default false) RETURNS text AS $$\n WITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n ) SELECT\n CASE\n WHEN NOT reverse THEN d\n WHEN d = 'ASC' THEN 'DESC'\n WHEN d = 'DESC' THEN 'ASC'\n END\n FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION sort_dir_to_op(_dir text, prev boolean default false) RETURNS text AS $$\n WITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n ) SELECT\n CASE\n WHEN d = 'ASC' AND prev THEN '<='\n WHEN d = 'DESC' AND prev THEN '>='\n WHEN d = 'ASC' THEN '>='\n WHEN d = 'DESC' THEN '<='\n END\n FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION sort_sqlorderby(\n _search jsonb DEFAULT NULL,\n reverse boolean DEFAULT FALSE\n) RETURNS text AS $$\n WITH sortby AS (\n SELECT coalesce(_search->'sortby','[{\"field\":\"datetime\", \"direction\":\"desc\"}]') as sort\n ), withid AS (\n SELECT CASE\n WHEN sort @? '$[*] ? (@.field == \"id\")' THEN sort\n ELSE sort || '[{\"field\":\"id\", \"direction\":\"desc\"}]'::jsonb\n END as sort\n FROM sortby\n ), withid_rows AS (\n SELECT jsonb_array_elements(sort) as value FROM withid\n ),sorts AS (\n SELECT\n coalesce(\n -- field_orderby((items_path(value->>'field')).path_txt),\n (queryable(value->>'field')).expression\n ) as key,\n parse_sort_dir(value->>'direction', reverse) as dir\n FROM withid_rows\n )\n SELECT array_to_string(\n array_agg(concat(key, ' ', dir)),\n ', '\n ) FROM sorts;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_sort_dir(sort_item jsonb) RETURNS text AS $$\n SELECT CASE WHEN sort_item->>'direction' ILIKE 'desc%' THEN 'DESC' ELSE 'ASC' END;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION get_token_filter(_search jsonb = '{}'::jsonb, token_rec jsonb DEFAULT NULL) RETURNS text AS $$\nDECLARE\n token_id text;\n filters text[] := '{}'::text[];\n prev boolean := TRUE;\n field text;\n dir text;\n sort record;\n orfilters text[] := '{}'::text[];\n andfilters text[] := '{}'::text[];\n output text;\n token_where text;\nBEGIN\n RAISE NOTICE 'Getting Token Filter. % %', _search, token_rec;\n -- If no token provided return NULL\n IF token_rec IS NULL THEN\n IF NOT (_search ? 'token' AND\n (\n (_search->>'token' ILIKE 'prev:%')\n OR\n (_search->>'token' ILIKE 'next:%')\n )\n ) THEN\n RETURN NULL;\n END IF;\n prev := (_search->>'token' ILIKE 'prev:%');\n token_id := substr(_search->>'token', 6);\n SELECT to_jsonb(items) INTO token_rec\n FROM items WHERE id=token_id;\n END IF;\n RAISE NOTICE 'TOKEN ID: % %', token_rec, token_rec->'id';\n\n CREATE TEMP TABLE sorts (\n _row int GENERATED ALWAYS AS IDENTITY NOT NULL,\n _field text PRIMARY KEY,\n _dir text NOT NULL,\n _val text\n ) ON COMMIT DROP;\n\n -- Make sure we only have distinct columns to sort with taking the first one we get\n INSERT INTO sorts (_field, _dir)\n SELECT\n (queryable(value->>'field')).expression,\n get_sort_dir(value)\n FROM\n jsonb_array_elements(coalesce(_search->'sortby','[{\"field\":\"datetime\",\"direction\":\"desc\"}]'))\n ON CONFLICT DO NOTHING\n ;\n RAISE NOTICE 'sorts 1: %', (SELECT jsonb_agg(to_json(sorts)) FROM sorts);\n -- Get the first sort direction provided. As the id is a primary key, if there are any\n -- sorts after id they won't do anything, so make sure that id is the last sort item.\n SELECT _dir INTO dir FROM sorts ORDER BY _row ASC LIMIT 1;\n IF EXISTS (SELECT 1 FROM sorts WHERE _field = 'id') THEN\n DELETE FROM sorts WHERE _row > (SELECT _row FROM sorts WHERE _field = 'id' ORDER BY _row ASC);\n ELSE\n INSERT INTO sorts (_field, _dir) VALUES ('id', dir);\n END IF;\n\n -- Add value from looked up item to the sorts table\n UPDATE sorts SET _val=quote_literal(token_rec->>_field);\n\n -- Check if all sorts are the same direction and use row comparison\n -- to filter\n RAISE NOTICE 'sorts 2: %', (SELECT jsonb_agg(to_json(sorts)) FROM sorts);\n\n IF (SELECT count(DISTINCT _dir) FROM sorts) = 1 THEN\n SELECT format(\n '(%s) %s (%s)',\n concat_ws(', ', VARIADIC array_agg(quote_ident(_field))),\n CASE WHEN (prev AND dir = 'ASC') OR (NOT prev AND dir = 'DESC') THEN '<' ELSE '>' END,\n concat_ws(', ', VARIADIC array_agg(_val))\n ) INTO output FROM sorts\n WHERE token_rec ? _field\n ;\n ELSE\n FOR sort IN SELECT * FROM sorts ORDER BY _row asc LOOP\n RAISE NOTICE 'SORT: %', sort;\n IF sort._row = 1 THEN\n orfilters := orfilters || format('(%s %s %s)',\n quote_ident(sort._field),\n CASE WHEN (prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC') THEN '<' ELSE '>' END,\n sort._val\n );\n ELSE\n orfilters := orfilters || format('(%s AND %s %s %s)',\n array_to_string(andfilters, ' AND '),\n quote_ident(sort._field),\n CASE WHEN (prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC') THEN '<' ELSE '>' END,\n sort._val\n );\n\n END IF;\n andfilters := andfilters || format('%s = %s',\n quote_ident(sort._field),\n sort._val\n );\n END LOOP;\n output := array_to_string(orfilters, ' OR ');\n END IF;\n DROP TABLE IF EXISTS sorts;\n token_where := concat('(',coalesce(output,'true'),')');\n IF trim(token_where) = '' THEN\n token_where := NULL;\n END IF;\n RAISE NOTICE 'TOKEN_WHERE: |%|',token_where;\n RETURN token_where;\n END;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION search_tohash(jsonb) RETURNS jsonb AS $$\n SELECT $1 - '{token,limit,context,includes,excludes}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION search_hash(jsonb, jsonb) RETURNS text AS $$\n SELECT md5(concat(search_tohash($1)::text,$2::text));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TABLE IF NOT EXISTS searches(\n hash text GENERATED ALWAYS AS (search_hash(search, metadata)) STORED PRIMARY KEY,\n search jsonb NOT NULL,\n _where text,\n orderby text,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n metadata jsonb DEFAULT '{}'::jsonb NOT NULL\n);\nCREATE TABLE IF NOT EXISTS search_wheres(\n id bigint generated always as identity primary key,\n _where text NOT NULL,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n statslastupdated timestamptz,\n estimated_count bigint,\n estimated_cost float,\n time_to_estimate float,\n total_count bigint,\n time_to_count float,\n partitions text[]\n);\n\nCREATE INDEX IF NOT EXISTS search_wheres_partitions ON search_wheres USING GIN (partitions);\nCREATE UNIQUE INDEX IF NOT EXISTS search_wheres_where ON search_wheres ((md5(_where)));\n\nCREATE OR REPLACE FUNCTION where_stats(inwhere text, updatestats boolean default false, conf jsonb default null) RETURNS search_wheres AS $$\nDECLARE\n t timestamptz;\n i interval;\n explain_json jsonb;\n partitions text[];\n sw search_wheres%ROWTYPE;\n inwhere_hash text := md5(inwhere);\n _context text := lower(context(conf));\n _stats_ttl interval := context_stats_ttl(conf);\n _estimated_cost float := context_estimated_cost(conf);\n _estimated_count int := context_estimated_count(conf);\nBEGIN\n IF _context = 'off' THEN\n sw._where := inwhere;\n return sw;\n END IF;\n\n SELECT * INTO sw FROM search_wheres WHERE md5(_where)=inwhere_hash FOR UPDATE;\n\n -- Update statistics if explicitly set, if statistics do not exist, or statistics ttl has expired\n IF NOT updatestats THEN\n RAISE NOTICE 'Checking if update is needed for: % .', inwhere;\n RAISE NOTICE 'Stats Last Updated: %', sw.statslastupdated;\n RAISE NOTICE 'TTL: %, Age: %', _stats_ttl, now() - sw.statslastupdated;\n RAISE NOTICE 'Context: %, Existing Total: %', _context, sw.total_count;\n IF\n sw.statslastupdated IS NULL\n OR (now() - sw.statslastupdated) > _stats_ttl\n OR (context(conf) != 'off' AND sw.total_count IS NULL)\n THEN\n updatestats := TRUE;\n END IF;\n END IF;\n\n sw._where := inwhere;\n sw.lastused := now();\n sw.usecount := coalesce(sw.usecount,0) + 1;\n\n IF NOT updatestats THEN\n UPDATE search_wheres SET\n lastused = sw.lastused,\n usecount = sw.usecount\n WHERE md5(_where) = inwhere_hash\n RETURNING * INTO sw\n ;\n RETURN sw;\n END IF;\n\n -- Use explain to get estimated count/cost and a list of the partitions that would be hit by the query\n t := clock_timestamp();\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s', inwhere)\n INTO explain_json;\n RAISE NOTICE 'Time for just the explain: %', clock_timestamp() - t;\n i := clock_timestamp() - t;\n\n sw.statslastupdated := now();\n sw.estimated_count := explain_json->0->'Plan'->'Plan Rows';\n sw.estimated_cost := explain_json->0->'Plan'->'Total Cost';\n sw.time_to_estimate := extract(epoch from i);\n\n RAISE NOTICE 'ESTIMATED_COUNT: % < %', sw.estimated_count, _estimated_count;\n RAISE NOTICE 'ESTIMATED_COST: % < %', sw.estimated_cost, _estimated_cost;\n\n -- Do a full count of rows if context is set to on or if auto is set and estimates are low enough\n IF\n _context = 'on'\n OR\n ( _context = 'auto' AND\n (\n sw.estimated_count < _estimated_count\n AND\n sw.estimated_cost < _estimated_cost\n )\n )\n THEN\n t := clock_timestamp();\n RAISE NOTICE 'Calculating actual count...';\n EXECUTE format(\n 'SELECT count(*) FROM items WHERE %s',\n inwhere\n ) INTO sw.total_count;\n i := clock_timestamp() - t;\n RAISE NOTICE 'Actual Count: % -- %', sw.total_count, i;\n sw.time_to_count := extract(epoch FROM i);\n ELSE\n sw.total_count := NULL;\n sw.time_to_count := NULL;\n END IF;\n\n\n INSERT INTO search_wheres\n (_where, lastused, usecount, statslastupdated, estimated_count, estimated_cost, time_to_estimate, partitions, total_count, time_to_count)\n SELECT sw._where, sw.lastused, sw.usecount, sw.statslastupdated, sw.estimated_count, sw.estimated_cost, sw.time_to_estimate, sw.partitions, sw.total_count, sw.time_to_count\n ON CONFLICT ((md5(_where)))\n DO UPDATE\n SET\n lastused = sw.lastused,\n usecount = sw.usecount,\n statslastupdated = sw.statslastupdated,\n estimated_count = sw.estimated_count,\n estimated_cost = sw.estimated_cost,\n time_to_estimate = sw.time_to_estimate,\n total_count = sw.total_count,\n time_to_count = sw.time_to_count\n ;\n RETURN sw;\nEND;\n$$ LANGUAGE PLPGSQL ;\n\n\n\nDROP FUNCTION IF EXISTS search_query;\nCREATE OR REPLACE FUNCTION search_query(\n _search jsonb = '{}'::jsonb,\n updatestats boolean = false,\n _metadata jsonb = '{}'::jsonb\n) RETURNS searches AS $$\nDECLARE\n search searches%ROWTYPE;\n pexplain jsonb;\n t timestamptz;\n i interval;\nBEGIN\n SELECT * INTO search FROM searches\n WHERE hash=search_hash(_search, _metadata) FOR UPDATE;\n\n -- Calculate the where clause if not already calculated\n IF search._where IS NULL THEN\n search._where := stac_search_to_where(_search);\n END IF;\n\n -- Calculate the order by clause if not already calculated\n IF search.orderby IS NULL THEN\n search.orderby := sort_sqlorderby(_search);\n END IF;\n\n PERFORM where_stats(search._where, updatestats, _search->'conf');\n\n search.lastused := now();\n search.usecount := coalesce(search.usecount, 0) + 1;\n INSERT INTO searches (search, _where, orderby, lastused, usecount, metadata)\n VALUES (_search, search._where, search.orderby, search.lastused, search.usecount, _metadata)\n ON CONFLICT (hash) DO\n UPDATE SET\n _where = EXCLUDED._where,\n orderby = EXCLUDED.orderby,\n lastused = EXCLUDED.lastused,\n usecount = EXCLUDED.usecount,\n metadata = EXCLUDED.metadata\n RETURNING * INTO search\n ;\n RETURN search;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION search(_search jsonb = '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n searches searches%ROWTYPE;\n _where text;\n token_where text;\n full_where text;\n orderby text;\n query text;\n token_type text := substr(_search->>'token',1,4);\n _limit int := coalesce((_search->>'limit')::int, 10);\n curs refcursor;\n cntr int := 0;\n iter_record items%ROWTYPE;\n first_record jsonb;\n first_item items%ROWTYPE;\n last_item items%ROWTYPE;\n last_record jsonb;\n out_records jsonb := '[]'::jsonb;\n prev_query text;\n next text;\n prev_id text;\n has_next boolean := false;\n has_prev boolean := false;\n prev text;\n total_count bigint;\n context jsonb;\n collection jsonb;\n includes text[];\n excludes text[];\n exit_flag boolean := FALSE;\n batches int := 0;\n timer timestamptz := clock_timestamp();\n pstart timestamptz;\n pend timestamptz;\n pcurs refcursor;\n search_where search_wheres%ROWTYPE;\n id text;\nBEGIN\nCREATE TEMP TABLE results (content jsonb) ON COMMIT DROP;\n-- if ids is set, short circuit and just use direct ids query for each id\n-- skip any paging or caching\n-- hard codes ordering in the same order as the array of ids\nIF _search ? 'ids' THEN\n INSERT INTO results\n SELECT\n CASE WHEN _search->'conf'->>'nohydrate' IS NOT NULL AND (_search->'conf'->>'nohydrate')::boolean = true THEN\n content_nonhydrated(items, _search->'fields')\n ELSE\n content_hydrate(items, _search->'fields')\n END\n FROM items WHERE\n items.id = ANY(to_text_array(_search->'ids'))\n AND\n CASE WHEN _search ? 'collections' THEN\n items.collection = ANY(to_text_array(_search->'collections'))\n ELSE TRUE\n END\n ORDER BY items.datetime desc, items.id desc\n ;\n SELECT INTO total_count count(*) FROM results;\nELSE\n searches := search_query(_search);\n _where := searches._where;\n orderby := searches.orderby;\n search_where := where_stats(_where);\n total_count := coalesce(search_where.total_count, search_where.estimated_count);\n\n IF token_type='prev' THEN\n token_where := get_token_filter(_search, null::jsonb);\n orderby := sort_sqlorderby(_search, TRUE);\n END IF;\n IF token_type='next' THEN\n token_where := get_token_filter(_search, null::jsonb);\n END IF;\n\n full_where := concat_ws(' AND ', _where, token_where);\n RAISE NOTICE 'FULL QUERY % %', full_where, clock_timestamp()-timer;\n timer := clock_timestamp();\n\n FOR query IN SELECT partition_queries(full_where, orderby, search_where.partitions) LOOP\n timer := clock_timestamp();\n query := format('%s LIMIT %s', query, _limit + 1);\n RAISE NOTICE 'Partition Query: %', query;\n batches := batches + 1;\n -- curs = create_cursor(query);\n RAISE NOTICE 'cursor_tuple_fraction: %', current_setting('cursor_tuple_fraction');\n OPEN curs FOR EXECUTE query;\n LOOP\n FETCH curs into iter_record;\n EXIT WHEN NOT FOUND;\n cntr := cntr + 1;\n\n IF _search->'conf'->>'nohydrate' IS NOT NULL AND (_search->'conf'->>'nohydrate')::boolean = true THEN\n last_record := content_nonhydrated(iter_record, _search->'fields');\n ELSE\n last_record := content_hydrate(iter_record, _search->'fields');\n END IF;\n last_item := iter_record;\n IF cntr = 1 THEN\n first_item := last_item;\n first_record := last_record;\n END IF;\n IF cntr <= _limit THEN\n INSERT INTO results (content) VALUES (last_record);\n ELSIF cntr > _limit THEN\n has_next := true;\n exit_flag := true;\n EXIT;\n END IF;\n END LOOP;\n CLOSE curs;\n RAISE NOTICE 'Query took %.', clock_timestamp()-timer;\n timer := clock_timestamp();\n EXIT WHEN exit_flag;\n END LOOP;\n RAISE NOTICE 'Scanned through % partitions.', batches;\nEND IF;\n\nSELECT jsonb_agg(content) INTO out_records FROM results WHERE content is not NULL;\n\nDROP TABLE results;\n\n\n-- Flip things around if this was the result of a prev token query\nIF token_type='prev' THEN\n out_records := flip_jsonb_array(out_records);\n first_item := last_item;\n first_record := last_record;\nEND IF;\n\n-- If this query has a token, see if there is data before the first record\nIF _search ? 'token' THEN\n prev_query := format(\n 'SELECT 1 FROM items WHERE %s LIMIT 1',\n concat_ws(\n ' AND ',\n _where,\n trim(get_token_filter(_search, to_jsonb(first_item)))\n )\n );\n RAISE NOTICE 'Query to get previous record: % --- %', prev_query, first_record;\n EXECUTE prev_query INTO has_prev;\n IF FOUND and has_prev IS NOT NULL THEN\n RAISE NOTICE 'Query results from prev query: %', has_prev;\n has_prev := TRUE;\n END IF;\nEND IF;\nhas_prev := COALESCE(has_prev, FALSE);\n\nIF has_prev THEN\n prev := out_records->0->>'id';\nEND IF;\nIF has_next OR token_type='prev' THEN\n next := out_records->-1->>'id';\nEND IF;\n\nIF context(_search->'conf') != 'off' THEN\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'matched', total_count,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\nELSE\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\nEND IF;\n\ncollection := jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb),\n 'next', next,\n 'prev', prev,\n 'context', context\n);\n\nRETURN collection;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER SET SEARCH_PATH TO pgstac, public SET cursor_tuple_fraction TO 1;\n\n\nCREATE OR REPLACE FUNCTION search_cursor(_search jsonb = '{}'::jsonb) RETURNS refcursor AS $$\nDECLARE\n curs refcursor;\n searches searches%ROWTYPE;\n _where text;\n _orderby text;\n q text;\n\nBEGIN\n searches := search_query(_search);\n _where := searches._where;\n _orderby := searches.orderby;\n\n OPEN curs FOR\n WITH p AS (\n SELECT * FROM partition_queries(_where, _orderby) p\n )\n SELECT\n CASE WHEN EXISTS (SELECT 1 FROM p) THEN\n (SELECT format($q$\n SELECT * FROM (\n %s\n ) total\n $q$,\n string_agg(\n format($q$ SELECT * FROM ( %s ) AS sub $q$, p),\n '\n UNION ALL\n '\n )\n ))\n ELSE NULL\n END FROM p;\n RETURN curs;\nEND;\n$$ LANGUAGE PLPGSQL;\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION tileenvelope(zoom int, x int, y int) RETURNS geometry AS $$\nWITH t AS (\n SELECT\n 20037508.3427892 as merc_max,\n -20037508.3427892 as merc_min,\n (2 * 20037508.3427892) / (2 ^ zoom) as tile_size\n)\nSELECT st_makeenvelope(\n merc_min + (tile_size * x),\n merc_max - (tile_size * (y + 1)),\n merc_min + (tile_size * (x + 1)),\n merc_max - (tile_size * y),\n 3857\n) FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;DROP FUNCTION IF EXISTS mercgrid;\n\n\nCREATE OR REPLACE FUNCTION ftime() RETURNS interval as $$\nSELECT age(clock_timestamp(), transaction_timestamp());\n$$ LANGUAGE SQL;\nSET SEARCH_PATH to pgstac, public;\n\nDROP FUNCTION IF EXISTS geometrysearch;\nCREATE OR REPLACE FUNCTION geometrysearch(\n IN geom geometry,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE, -- Return as soon as the passed in geometry is full covered\n IN skipcovered boolean DEFAULT TRUE -- Skip any items that would show up completely under the previous items\n) RETURNS jsonb AS $$\nDECLARE\n search searches%ROWTYPE;\n curs refcursor;\n _where text;\n query text;\n iter_record items%ROWTYPE;\n out_records jsonb := '{}'::jsonb[];\n exit_flag boolean := FALSE;\n counter int := 1;\n scancounter int := 1;\n remaining_limit int := _scanlimit;\n tilearea float;\n unionedgeom geometry;\n clippedgeom geometry;\n unionedgeom_area float := 0;\n prev_area float := 0;\n excludes text[];\n includes text[];\n\nBEGIN\n DROP TABLE IF EXISTS pgstac_results;\n CREATE TEMP TABLE pgstac_results (content jsonb) ON COMMIT DROP;\n\n -- If skipcovered is true then you will always want to exit when the passed in geometry is full\n IF skipcovered THEN\n exitwhenfull := TRUE;\n END IF;\n\n SELECT * INTO search FROM searches WHERE hash=queryhash;\n\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Search with Query Hash % Not Found', queryhash;\n END IF;\n\n tilearea := st_area(geom);\n _where := format('%s AND st_intersects(geometry, %L::geometry)', search._where, geom);\n\n\n FOR query IN SELECT * FROM partition_queries(_where, search.orderby) LOOP\n query := format('%s LIMIT %L', query, remaining_limit);\n RAISE NOTICE '%', query;\n OPEN curs FOR EXECUTE query;\n LOOP\n FETCH curs INTO iter_record;\n EXIT WHEN NOT FOUND;\n IF exitwhenfull OR skipcovered THEN -- If we are not using exitwhenfull or skipcovered, we do not need to do expensive geometry operations\n clippedgeom := st_intersection(geom, iter_record.geometry);\n\n IF unionedgeom IS NULL THEN\n unionedgeom := clippedgeom;\n ELSE\n unionedgeom := st_union(unionedgeom, clippedgeom);\n END IF;\n\n unionedgeom_area := st_area(unionedgeom);\n\n IF skipcovered AND prev_area = unionedgeom_area THEN\n scancounter := scancounter + 1;\n CONTINUE;\n END IF;\n\n prev_area := unionedgeom_area;\n\n RAISE NOTICE '% % % %', unionedgeom_area/tilearea, counter, scancounter, ftime();\n END IF;\n RAISE NOTICE '% %', iter_record, content_hydrate(iter_record, fields);\n INSERT INTO pgstac_results (content) VALUES (content_hydrate(iter_record, fields));\n\n IF counter >= _limit\n OR scancounter > _scanlimit\n OR ftime() > _timelimit\n OR (exitwhenfull AND unionedgeom_area >= tilearea)\n THEN\n exit_flag := TRUE;\n EXIT;\n END IF;\n counter := counter + 1;\n scancounter := scancounter + 1;\n\n END LOOP;\n CLOSE curs;\n EXIT WHEN exit_flag;\n remaining_limit := _scanlimit - scancounter;\n END LOOP;\n\n SELECT jsonb_agg(content) INTO out_records FROM pgstac_results WHERE content IS NOT NULL;\n\n RETURN jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb)\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDROP FUNCTION IF EXISTS geojsonsearch;\nCREATE OR REPLACE FUNCTION geojsonsearch(\n IN geojson jsonb,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_geomfromgeojson(geojson),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS xyzsearch;\nCREATE OR REPLACE FUNCTION xyzsearch(\n IN _x int,\n IN _y int,\n IN _z int,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_transform(tileenvelope(_z, _x, _y), 4326),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nGRANT ALL ON SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac to pgstac_admin;\n\n-- pgstac_read role limited to using function apis\nGRANT EXECUTE ON FUNCTION search TO pgstac_read;\nGRANT EXECUTE ON FUNCTION search_query TO pgstac_read;\nGRANT EXECUTE ON FUNCTION item_by_id TO pgstac_read;\nGRANT EXECUTE ON FUNCTION get_item TO pgstac_read;\n\nGRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_ingest;\nGRANT USAGE ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_ingest;\nSELECT set_version('0.6.10');\n" }, { "path": "src/pgstac/migrations/pgstac.0.6.11-0.6.12.sql", "content": "SET client_min_messages TO WARNING;\nSET SEARCH_PATH to pgstac, public;\nalter table \"pgstac\".\"queryables\" drop constraint \"queryables_name_key\";\n\ndrop function if exists \"pgstac\".\"missing_queryables\"(_collection text, _tablesample integer);\n\ndrop function if exists \"pgstac\".\"missing_queryables\"(_tablesample integer);\n\nalter table \"pgstac\".\"stac_extensions\" drop constraint \"stac_extensions_pkey\";\n\ndrop index if exists \"pgstac\".\"queryables_name_key\";\n\ndrop index if exists \"pgstac\".\"stac_extensions_pkey\";\n\nalter table \"pgstac\".\"stac_extensions\" drop column \"enableable\";\n\nalter table \"pgstac\".\"stac_extensions\" drop column \"enbabled_by_default\";\n\nalter table \"pgstac\".\"stac_extensions\" drop column \"name\";\n\nalter table \"pgstac\".\"stac_extensions\" add column \"content\" jsonb;\n\nalter table \"pgstac\".\"stac_extensions\" alter column \"url\" set not null;\n\nCREATE UNIQUE INDEX stac_extensions_pkey ON pgstac.stac_extensions USING btree (url);\n\nalter table \"pgstac\".\"stac_extensions\" add constraint \"stac_extensions_pkey\" PRIMARY KEY using index \"stac_extensions_pkey\";\n\nset check_function_bodies = off;\n\nCREATE OR REPLACE PROCEDURE pgstac.analyze_items()\n LANGUAGE plpgsql\nAS $procedure$\nDECLARE\nq text;\nBEGIN\nFOR q IN\n SELECT format('ANALYZE (VERBOSE, SKIP_LOCKED) %I;', relname)\n FROM pg_stat_user_tables\n WHERE relname like '_item%' AND (n_mod_since_analyze>0 OR last_analyze IS NULL)\nLOOP\n RAISE NOTICE '%', q;\n EXECUTE q;\n COMMIT;\nEND LOOP;\nEND;\n$procedure$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.check_pgstac_settings(_sysmem text DEFAULT NULL::text)\n RETURNS void\n LANGUAGE plpgsql\n SET search_path TO 'pgstac', 'public'\n SET client_min_messages TO 'notice'\nAS $function$\nDECLARE\n settingval text;\n sysmem bigint := pg_size_bytes(_sysmem);\n effective_cache_size bigint := pg_size_bytes(current_setting('effective_cache_size', TRUE));\n shared_buffers bigint := pg_size_bytes(current_setting('shared_buffers', TRUE));\n work_mem bigint := pg_size_bytes(current_setting('work_mem', TRUE));\n max_connections int := current_setting('max_connections', TRUE);\n maintenance_work_mem bigint := pg_size_bytes(current_setting('maintenance_work_mem', TRUE));\n seq_page_cost float := current_setting('seq_page_cost', TRUE);\n random_page_cost float := current_setting('random_page_cost', TRUE);\n temp_buffers bigint := pg_size_bytes(current_setting('temp_buffers', TRUE));\n r record;\nBEGIN\n IF _sysmem IS NULL THEN\n RAISE NOTICE 'Call function with the size of your system memory `SELECT check_pgstac_settings(''4GB'')` to get pg system setting recommendations.';\n ELSE\n IF effective_cache_size < (sysmem * 0.5) THEN\n RAISE WARNING 'effective_cache_size of % is set low for a system with %. Recomended value between % and %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.5), pg_size_pretty(sysmem * 0.75);\n ELSIF effective_cache_size > (sysmem * 0.75) THEN\n RAISE WARNING 'effective_cache_size of % is set high for a system with %. Recomended value between % and %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.5), pg_size_pretty(sysmem * 0.75);\n ELSE\n RAISE NOTICE 'effective_cache_size of % is set appropriately for a system with %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem);\n END IF;\n\n IF shared_buffers < (sysmem * 0.2) THEN\n RAISE WARNING 'shared_buffers of % is set low for a system with %. Recomended value between % and %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.2), pg_size_pretty(sysmem * 0.3);\n ELSIF shared_buffers > (sysmem * 0.3) THEN\n RAISE WARNING 'shared_buffers of % is set high for a system with %. Recomended value between % and %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.2), pg_size_pretty(sysmem * 0.3);\n ELSE\n RAISE NOTICE 'shared_buffers of % is set appropriately for a system with %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem);\n END IF;\n shared_buffers = sysmem * 0.3;\n IF maintenance_work_mem < (sysmem * 0.2) THEN\n RAISE WARNING 'maintenance_work_mem of % is set low for shared_buffers of %. Recomended value between % and %', pg_size_pretty(maintenance_work_mem), pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers * 0.2), pg_size_pretty(shared_buffers * 0.3);\n ELSIF maintenance_work_mem > (shared_buffers * 0.3) THEN\n RAISE WARNING 'maintenance_work_mem of % is set high for shared_buffers of %. Recomended value between % and %', pg_size_pretty(maintenance_work_mem), pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers * 0.2), pg_size_pretty(shared_buffers * 0.3);\n ELSE\n RAISE NOTICE 'maintenance_work_mem of % is set appropriately for shared_buffers of %', pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers);\n END IF;\n\n IF work_mem * max_connections > shared_buffers THEN\n RAISE WARNING 'work_mem setting of % is set high for % max_connections please reduce work_mem to % or decrease max_connections to %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers/max_connections), floor(shared_buffers/work_mem);\n ELSIF work_mem * max_connections < (shared_buffers * 0.75) THEN\n RAISE WARNING 'work_mem setting of % is set low for % max_connections you may consider raising work_mem to % or increasing max_connections to %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers/max_connections), floor(shared_buffers/work_mem);\n ELSE\n RAISE NOTICE 'work_mem setting of % and max_connections of % are adequate for shared_buffers of %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers);\n END IF;\n\n IF random_page_cost / seq_page_cost != 1.1 THEN\n RAISE WARNING 'random_page_cost (%) /seq_page_cost (%) should be set to 1.1 for SSD. Change random_page_cost to %', random_page_cost, seq_page_cost, 1.1 * seq_page_cost;\n ELSE\n RAISE NOTICE 'random_page_cost and seq_page_cost set appropriately for SSD';\n END IF;\n\n IF temp_buffers < greatest(pg_size_bytes('128MB'),(maintenance_work_mem / 2)) THEN\n RAISE WARNING 'pgstac makes heavy use of temp tables, consider raising temp_buffers from % to %', pg_size_pretty(temp_buffers), greatest('128MB', pg_size_pretty((shared_buffers / 16)));\n END IF;\n END IF;\n\n RAISE NOTICE 'VALUES FOR PGSTAC VARIABLES';\n RAISE NOTICE 'These can be set either as GUC system variables or by setting in the pgstac_settings table.';\n\n FOR r IN SELECT name, get_setting(name) as setting, CASE WHEN current_setting(concat('pgstac.',name), TRUE) IS NOT NULL THEN concat('pgstac.',name, ' GUC') WHEN value IS NOT NULL THEN 'pgstac_settings table' ELSE 'Not Set' END as loc FROM pgstac_settings LOOP\n RAISE NOTICE '% is set to % from the %', r.name, r.setting, r.loc;\n END LOOP;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pg_cron';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider intalling pg_cron which can be used to automate tasks';\n ELSE\n RAISE NOTICE 'pg_cron % is installed', settingval;\n END IF;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pgstattuple';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider installing the pgstattuple extension which can be used to help maintain tables and indexes.';\n ELSE\n RAISE NOTICE 'pgstattuple % is installed', settingval;\n END IF;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pg_stat_statements';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider installing the pg_stat_statements extension which is very helpful for tracking the types of queries on the system';\n ELSE\n RAISE NOTICE 'pgstattuple % is installed', settingval;\n END IF;\n\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.first_notnull_sfunc(anyelement, anyelement)\n RETURNS anyelement\n LANGUAGE sql\n IMMUTABLE PARALLEL SAFE\nAS $function$\n SELECT COALESCE($1,$2);\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.get_queryables()\n RETURNS jsonb\n LANGUAGE sql\nAS $function$\n SELECT get_queryables(NULL::text[]);\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.get_token_val_str(_field text, _item pgstac.items)\n RETURNS text\n LANGUAGE plpgsql\nAS $function$\nDECLARE\nliteral text;\nBEGIN\nRAISE NOTICE '% %', _field, _item;\nCREATE TEMP TABLE _token_item ON COMMIT DROP AS SELECT (_item).*;\nEXECUTE format($q$ SELECT quote_literal(%s) FROM _token_item $q$, _field) INTO literal;\nDROP TABLE IF EXISTS _token_item;\nRETURN literal;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.jsonb_array_unique(j jsonb)\n RETURNS jsonb\n LANGUAGE sql\n IMMUTABLE\nAS $function$\n SELECT nullif_jsonbnullempty(jsonb_agg(DISTINCT a)) v FROM jsonb_array_elements(j) a;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.jsonb_concat_ignorenull(a jsonb, b jsonb)\n RETURNS jsonb\n LANGUAGE sql\n IMMUTABLE\nAS $function$\n SELECT coalesce(a,'[]'::jsonb) || coalesce(b,'[]'::jsonb);\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.jsonb_greatest(a jsonb, b jsonb)\n RETURNS jsonb\n LANGUAGE sql\n IMMUTABLE\nAS $function$\n SELECT nullif_jsonbnullempty(greatest(a, b));\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.jsonb_least(a jsonb, b jsonb)\n RETURNS jsonb\n LANGUAGE sql\n IMMUTABLE\nAS $function$\n SELECT nullif_jsonbnullempty(least(nullif_jsonbnullempty(a), nullif_jsonbnullempty(b)));\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.missing_queryables(_collection text, _tablesample double precision DEFAULT 5, minrows double precision DEFAULT 10)\n RETURNS TABLE(collection text, name text, definition jsonb, property_wrapper text)\n LANGUAGE plpgsql\nAS $function$\nDECLARE\n q text;\n _partition text;\n explain_json json;\n psize float;\n estrows float;\nBEGIN\n SELECT format('_items_%s', key) INTO _partition FROM collections WHERE id=_collection;\n\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM %I;', _partition)\n INTO explain_json;\n psize := explain_json->0->'Plan'->'Plan Rows';\n estrows := _tablesample * .01 * psize;\n IF estrows < minrows THEN\n _tablesample := least(100,greatest(_tablesample, (estrows / psize) / 100));\n RAISE NOTICE '%', (psize / estrows) / 100;\n END IF;\n RAISE NOTICE 'Using tablesample % to find missing queryables from % % that has ~% rows estrows: %', _tablesample, _collection, _partition, psize, estrows;\n\n q := format(\n $q$\n WITH q AS (\n SELECT * FROM queryables\n WHERE\n collection_ids IS NULL\n OR %L = ANY(collection_ids)\n ), t AS (\n SELECT\n content->'properties' AS properties\n FROM\n %I\n TABLESAMPLE SYSTEM(%L)\n ), p AS (\n SELECT DISTINCT ON (key)\n key,\n value,\n s.definition\n FROM t\n JOIN LATERAL jsonb_each(properties) ON TRUE\n LEFT JOIN q ON (q.name=key)\n LEFT JOIN stac_extension_queryables s ON (s.name=key)\n WHERE q.definition IS NULL\n )\n SELECT\n %L,\n key,\n COALESCE(definition, jsonb_build_object('type',jsonb_typeof(value))) as definition,\n CASE\n WHEN definition->>'type' = 'integer' THEN 'to_int'\n WHEN COALESCE(definition->>'type', jsonb_typeof(value)) = 'number' THEN 'to_float'\n WHEN COALESCE(definition->>'type', jsonb_typeof(value)) = 'array' THEN 'to_text_array'\n ELSE 'to_text'\n END\n FROM p;\n $q$,\n _collection,\n _partition,\n _tablesample,\n _collection\n );\n RETURN QUERY EXECUTE q;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.missing_queryables(_tablesample double precision DEFAULT 5)\n RETURNS TABLE(collection_ids text[], name text, definition jsonb, property_wrapper text)\n LANGUAGE sql\nAS $function$\n SELECT\n array_agg(collection),\n name,\n definition,\n property_wrapper\n FROM\n collections\n JOIN LATERAL\n missing_queryables(id, _tablesample) c\n ON TRUE\n GROUP BY\n 2,3,4\n ORDER BY 2,1\n ;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.nullif_jsonbnullempty(j jsonb)\n RETURNS jsonb\n LANGUAGE sql\n IMMUTABLE PARALLEL SAFE STRICT\nAS $function$\n SELECT nullif(nullif(nullif(j,'null'::jsonb),'{}'::jsonb),'[]'::jsonb);\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.schema_qualify_refs(url text, j jsonb)\n RETURNS jsonb\n LANGUAGE sql\n IMMUTABLE PARALLEL SAFE STRICT\nAS $function$\n SELECT regexp_replace(j::text, '\"\\$ref\": \"#', concat('\"$ref\": \"', url, '#'), 'g')::jsonb;\n$function$\n;\n\ncreate or replace view \"pgstac\".\"stac_extension_queryables\" as SELECT DISTINCT j.key AS name,\n pgstac.schema_qualify_refs(e.url, j.value) AS definition\n FROM pgstac.stac_extensions e,\n LATERAL jsonb_each((((e.content -> 'definitions'::text) -> 'fields'::text) -> 'properties'::text)) j(key, value);\n\n\nCREATE OR REPLACE PROCEDURE pgstac.validate_constraints()\n LANGUAGE plpgsql\nAS $procedure$\nDECLARE\n q text;\nBEGIN\n FOR q IN\n SELECT\n FORMAT(\n 'ALTER TABLE %I.%I VALIDATE CONSTRAINT %I;',\n nsp.nspname,\n cls.relname,\n con.conname\n )\n\n FROM pg_constraint AS con\n JOIN pg_class AS cls\n ON con.conrelid = cls.oid\n JOIN pg_namespace AS nsp\n ON cls.relnamespace = nsp.oid\n WHERE convalidated = FALSE AND contype in ('c','f')\n AND nsp.nspname = 'pgstac'\n LOOP\n RAISE NOTICE '%', q;\n EXECUTE q;\n COMMIT;\n END LOOP;\nEND;\n$procedure$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.get_queryables(_collection_ids text[] DEFAULT NULL::text[])\n RETURNS jsonb\n LANGUAGE plpgsql\n STABLE\nAS $function$\nDECLARE\nBEGIN\n -- Build up queryables if the input contains valid collection ids or is empty\n IF EXISTS (\n SELECT 1 FROM collections\n WHERE\n _collection_ids IS NULL\n OR cardinality(_collection_ids) = 0\n OR id = ANY(_collection_ids)\n )\n THEN\n RETURN (\n WITH base AS (\n SELECT\n unnest(collection_ids) as collection_id,\n name,\n coalesce(definition, '{\"type\":\"string\"}'::jsonb) as definition\n FROM queryables\n WHERE\n _collection_ids IS NULL OR\n _collection_ids = '{}'::text[] OR\n _collection_ids && collection_ids\n UNION ALL\n SELECT null, name, coalesce(definition, '{\"type\":\"string\"}'::jsonb) as definition\n FROM queryables WHERE collection_ids IS NULL OR collection_ids = '{}'::text[]\n ), g AS (\n SELECT\n name,\n first_notnull(definition) as definition,\n jsonb_array_unique_merge(definition->'enum') as enum,\n jsonb_min(definition->'minimum') as minimum,\n jsonb_min(definition->'maxiumn') as maximum\n FROM base\n GROUP BY 1\n )\n SELECT\n jsonb_build_object(\n '$schema', 'http://json-schema.org/draft-07/schema#',\n '$id', '',\n 'type', 'object',\n 'title', 'STAC Queryables.',\n 'properties', jsonb_object_agg(\n name,\n definition\n ||\n jsonb_strip_nulls(jsonb_build_object(\n 'enum', enum,\n 'minimum', minimum,\n 'maximum', maximum\n ))\n )\n )\n FROM g\n );\n ELSE\n RETURN NULL;\n END IF;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.get_token_filter(_search jsonb DEFAULT '{}'::jsonb, token_rec jsonb DEFAULT NULL::jsonb)\n RETURNS text\n LANGUAGE plpgsql\nAS $function$\nDECLARE\n token_id text;\n filters text[] := '{}'::text[];\n prev boolean := TRUE;\n field text;\n dir text;\n sort record;\n orfilters text[] := '{}'::text[];\n andfilters text[] := '{}'::text[];\n output text;\n token_where text;\n token_item items%ROWTYPE;\nBEGIN\n RAISE NOTICE 'Getting Token Filter. % %', _search, token_rec;\n -- If no token provided return NULL\n IF token_rec IS NULL THEN\n IF NOT (_search ? 'token' AND\n (\n (_search->>'token' ILIKE 'prev:%')\n OR\n (_search->>'token' ILIKE 'next:%')\n )\n ) THEN\n RETURN NULL;\n END IF;\n prev := (_search->>'token' ILIKE 'prev:%');\n token_id := substr(_search->>'token', 6);\n SELECT to_jsonb(items) INTO token_rec\n FROM items WHERE id=token_id;\n END IF;\n RAISE NOTICE 'TOKEN ID: % %', token_rec, token_rec->'id';\n\n\n RAISE NOTICE 'TOKEN ID: % %', token_rec, token_rec->'id';\n token_item := jsonb_populate_record(null::items, token_rec);\n RAISE NOTICE 'TOKEN ITEM ----- %', token_item;\n\n\n CREATE TEMP TABLE sorts (\n _row int GENERATED ALWAYS AS IDENTITY NOT NULL,\n _field text PRIMARY KEY,\n _dir text NOT NULL,\n _val text\n ) ON COMMIT DROP;\n\n -- Make sure we only have distinct columns to sort with taking the first one we get\n INSERT INTO sorts (_field, _dir)\n SELECT\n (queryable(value->>'field')).expression,\n get_sort_dir(value)\n FROM\n jsonb_array_elements(coalesce(_search->'sortby','[{\"field\":\"datetime\",\"direction\":\"desc\"}]'))\n ON CONFLICT DO NOTHING\n ;\n RAISE NOTICE 'sorts 1: %', (SELECT jsonb_agg(to_json(sorts)) FROM sorts);\n -- Get the first sort direction provided. As the id is a primary key, if there are any\n -- sorts after id they won't do anything, so make sure that id is the last sort item.\n SELECT _dir INTO dir FROM sorts ORDER BY _row ASC LIMIT 1;\n IF EXISTS (SELECT 1 FROM sorts WHERE _field = 'id') THEN\n DELETE FROM sorts WHERE _row > (SELECT _row FROM sorts WHERE _field = 'id' ORDER BY _row ASC);\n ELSE\n INSERT INTO sorts (_field, _dir) VALUES ('id', dir);\n END IF;\n\n -- Add value from looked up item to the sorts table\n UPDATE sorts SET _val=get_token_val_str(_field, token_item);\n\n -- Check if all sorts are the same direction and use row comparison\n -- to filter\n RAISE NOTICE 'sorts 2: %', (SELECT jsonb_agg(to_json(sorts)) FROM sorts);\n\n FOR sort IN SELECT * FROM sorts ORDER BY _row asc LOOP\n RAISE NOTICE 'SORT: %', sort;\n IF sort._row = 1 THEN\n IF sort._val IS NULL THEN\n orfilters := orfilters || format('(%s IS NOT NULL)', sort._field);\n ELSE\n orfilters := orfilters || format('(%s %s %s)',\n sort._field,\n CASE WHEN (prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC') THEN '<' ELSE '>' END,\n sort._val\n );\n END IF;\n ELSE\n IF sort._val IS NULL THEN\n orfilters := orfilters || format('(%s AND %s IS NOT NULL)',\n array_to_string(andfilters, ' AND '), sort._field);\n ELSE\n orfilters := orfilters || format('(%s AND %s %s %s)',\n array_to_string(andfilters, ' AND '),\n sort._field,\n CASE WHEN (prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC') THEN '<' ELSE '>' END,\n sort._val\n );\n END IF;\n END IF;\n IF sort._val IS NULL THEN\n andfilters := andfilters || format('%s IS NULL',\n sort._field\n );\n ELSE\n andfilters := andfilters || format('%s = %s',\n sort._field,\n sort._val\n );\n END IF;\n END LOOP;\n output := array_to_string(orfilters, ' OR ');\n\n DROP TABLE IF EXISTS sorts;\n token_where := concat('(',coalesce(output,'true'),')');\n IF trim(token_where) = '' THEN\n token_where := NULL;\n END IF;\n RAISE NOTICE 'TOKEN_WHERE: |%|',token_where;\n RETURN token_where;\n END;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.items_staging_triggerfunc()\n RETURNS trigger\n LANGUAGE plpgsql\nAS $function$\nDECLARE\n p record;\n _partitions text[];\n ts timestamptz := clock_timestamp();\nBEGIN\n RAISE NOTICE 'Creating Partitions. %', clock_timestamp() - ts;\n WITH ranges AS (\n SELECT\n n.content->>'collection' as collection,\n stac_daterange(n.content->'properties') as dtr\n FROM newdata n\n ), p AS (\n SELECT\n collection,\n lower(dtr) as datetime,\n upper(dtr) as end_datetime,\n (partition_name(\n collection,\n lower(dtr)\n )).partition_name as name\n FROM ranges\n )\n INSERT INTO partitions (collection, datetime_range, end_datetime_range)\n SELECT\n collection,\n tstzrange(min(datetime), max(datetime), '[]') as datetime_range,\n tstzrange(min(end_datetime), max(end_datetime), '[]') as end_datetime_range\n FROM p\n GROUP BY collection, name\n ON CONFLICT (name) DO UPDATE SET\n datetime_range = EXCLUDED.datetime_range,\n end_datetime_range = EXCLUDED.end_datetime_range\n ;\n\n RAISE NOTICE 'Doing the insert. %', clock_timestamp() - ts;\n IF TG_TABLE_NAME = 'items_staging' THEN\n INSERT INTO items\n SELECT\n (content_dehydrate(content)).*\n FROM newdata;\n DELETE FROM items_staging;\n ELSIF TG_TABLE_NAME = 'items_staging_ignore' THEN\n INSERT INTO items\n SELECT\n (content_dehydrate(content)).*\n FROM newdata\n ON CONFLICT DO NOTHING;\n DELETE FROM items_staging_ignore;\n ELSIF TG_TABLE_NAME = 'items_staging_upsert' THEN\n WITH staging_formatted AS (\n SELECT (content_dehydrate(content)).* FROM newdata\n ), deletes AS (\n DELETE FROM items i USING staging_formatted s\n WHERE\n i.id = s.id\n AND i.collection = s.collection\n AND i IS DISTINCT FROM s\n RETURNING i.id, i.collection\n )\n INSERT INTO items\n SELECT s.* FROM\n staging_formatted s\n ON CONFLICT DO NOTHING;\n DELETE FROM items_staging_upsert;\n END IF;\n RAISE NOTICE 'Done. %', clock_timestamp() - ts;\n\n RETURN NULL;\n\nEND;\n$function$\n;\n\n\nCREATE OR REPLACE AGGREGATE first_notnull(anyelement)(\n SFUNC = first_notnull_sfunc,\n STYPE = anyelement\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_array_unique_merge(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_concat_ignorenull,\n FINALFUNC = jsonb_array_unique\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_min(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_least\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_max(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_greatest\n);\n\n\nSELECT set_version('0.6.12');\n" }, { "path": "src/pgstac/migrations/pgstac.0.6.11.sql", "content": "CREATE EXTENSION IF NOT EXISTS postgis;\nCREATE EXTENSION IF NOT EXISTS btree_gist;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_admin;\n CREATE ROLE pgstac_read;\n CREATE ROLE pgstac_ingest;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nGRANT pgstac_admin TO current_user;\n\nCREATE SCHEMA IF NOT EXISTS pgstac AUTHORIZATION pgstac_admin;\n\nALTER ROLE pgstac_admin SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_read SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_ingest SET SEARCH_PATH TO pgstac, public;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\n\nGRANT pgstac_read TO pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\n\nSET ROLE pgstac_admin;\n\nSET SEARCH_PATH TO pgstac, public;\n\n\nCREATE TABLE IF NOT EXISTS migrations (\n version text PRIMARY KEY,\n datetime timestamptz DEFAULT clock_timestamp() NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION get_version() RETURNS text AS $$\n SELECT version FROM pgstac.migrations ORDER BY datetime DESC, version DESC LIMIT 1;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION set_version(text) RETURNS text AS $$\n INSERT INTO pgstac.migrations (version) VALUES ($1)\n ON CONFLICT DO NOTHING\n RETURNING version;\n$$ LANGUAGE SQL;\n\n\nCREATE TABLE IF NOT EXISTS pgstac_settings (\n name text PRIMARY KEY,\n value text NOT NULL\n);\n\nINSERT INTO pgstac_settings (name, value) VALUES\n ('context', 'off'),\n ('context_estimated_count', '100000'),\n ('context_estimated_cost', '100000'),\n ('context_stats_ttl', '1 day'),\n ('default_filter_lang', 'cql2-json'),\n ('additional_properties', 'true')\nON CONFLICT DO NOTHING\n;\n\n\nCREATE OR REPLACE FUNCTION get_setting(IN _setting text, IN conf jsonb DEFAULT NULL) RETURNS text AS $$\nSELECT COALESCE(\n conf->>_setting,\n current_setting(concat('pgstac.',_setting), TRUE),\n (SELECT value FROM pgstac.pgstac_settings WHERE name=_setting)\n);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context(conf jsonb DEFAULT NULL) RETURNS text AS $$\n SELECT pgstac.get_setting('context', conf);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context_estimated_count(conf jsonb DEFAULT NULL) RETURNS int AS $$\n SELECT pgstac.get_setting('context_estimated_count', conf)::int;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_estimated_cost();\nCREATE OR REPLACE FUNCTION context_estimated_cost(conf jsonb DEFAULT NULL) RETURNS float AS $$\n SELECT pgstac.get_setting('context_estimated_cost', conf)::float;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_stats_ttl();\nCREATE OR REPLACE FUNCTION context_stats_ttl(conf jsonb DEFAULT NULL) RETURNS interval AS $$\n SELECT pgstac.get_setting('context_stats_ttl', conf)::interval;\n$$ LANGUAGE SQL;\n\n\nCREATE OR REPLACE FUNCTION notice(VARIADIC text[]) RETURNS boolean AS $$\nDECLARE\ndebug boolean := current_setting('pgstac.debug', true);\nBEGIN\n IF debug THEN\n RAISE NOTICE 'NOTICE FROM FUNC: % >>>>> %', concat_ws(' | ', $1), clock_timestamp();\n RETURN TRUE;\n END IF;\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION empty_arr(ANYARRAY) RETURNS BOOLEAN AS $$\nSELECT CASE\n WHEN $1 IS NULL THEN TRUE\n WHEN cardinality($1)<1 THEN TRUE\nELSE FALSE\nEND;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_intersection(_a ANYARRAY, _b ANYARRAY) RETURNS ANYARRAY AS $$\n SELECT ARRAY ( SELECT unnest(_a) INTERSECT SELECT UNNEST(_b) );\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION array_map_ident(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_ident(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_map_literal(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_literal(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION array_reverse(anyarray) RETURNS anyarray AS $$\nSELECT ARRAY(\n SELECT $1[i]\n FROM generate_subscripts($1,1) AS s(i)\n ORDER BY i DESC\n);\n$$ LANGUAGE SQL STRICT IMMUTABLE;\nCREATE OR REPLACE FUNCTION to_int(jsonb) RETURNS int AS $$\n SELECT floor(($1->>0)::float)::int;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION to_float(jsonb) RETURNS float AS $$\n SELECT ($1->>0)::float;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION to_tstz(jsonb) RETURNS timestamptz AS $$\n SELECT ($1->>0)::timestamptz;\n$$ LANGUAGE SQL IMMUTABLE STRICT SET TIME ZONE 'UTC';\n\n\nCREATE OR REPLACE FUNCTION to_text(jsonb) RETURNS text AS $$\n SELECT CASE WHEN jsonb_typeof($1) IN ('array','object') THEN $1::text ELSE $1->>0 END;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION to_text_array(jsonb) RETURNS text[] AS $$\n SELECT\n CASE jsonb_typeof($1)\n WHEN 'array' THEN ARRAY(SELECT jsonb_array_elements_text($1))\n ELSE ARRAY[$1->>0]\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION bbox_geom(_bbox jsonb) RETURNS geometry AS $$\nSELECT CASE jsonb_array_length(_bbox)\n WHEN 4 THEN\n ST_SetSRID(ST_MakeEnvelope(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float,\n (_bbox->>3)::float\n ),4326)\n WHEN 6 THEN\n ST_SetSRID(ST_3DMakeBox(\n ST_MakePoint(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float\n ),\n ST_MakePoint(\n (_bbox->>3)::float,\n (_bbox->>4)::float,\n (_bbox->>5)::float\n )\n ),4326)\n ELSE null END;\n;\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION geom_bbox(_geom geometry) RETURNS jsonb AS $$\n SELECT jsonb_build_array(\n st_xmin(_geom),\n st_ymin(_geom),\n st_xmax(_geom),\n st_ymax(_geom)\n );\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION flip_jsonb_array(j jsonb) RETURNS jsonb AS $$\n SELECT jsonb_agg(value) FROM (SELECT value FROM jsonb_array_elements(j) WITH ORDINALITY ORDER BY ordinality DESC) as t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION explode_dotpaths(j jsonb) RETURNS SETOF text[] AS $$\n SELECT string_to_array(p, '.') as e FROM jsonb_array_elements_text(j) p;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION explode_dotpaths_recurse(IN j jsonb) RETURNS SETOF text[] AS $$\n WITH RECURSIVE t AS (\n SELECT e FROM explode_dotpaths(j) e\n UNION ALL\n SELECT e[1:cardinality(e)-1]\n FROM t\n WHERE cardinality(e)>1\n ) SELECT e FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION jsonb_set_nested(j jsonb, path text[], val jsonb) RETURNS jsonb AS $$\nDECLARE\nBEGIN\n IF cardinality(path) > 1 THEN\n FOR i IN 1..(cardinality(path)-1) LOOP\n IF j #> path[:i] IS NULL THEN\n j := jsonb_set_lax(j, path[:i], '{}', TRUE);\n END IF;\n END LOOP;\n END IF;\n RETURN jsonb_set_lax(j, path, val, true);\n\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\n\n\nCREATE OR REPLACE FUNCTION jsonb_include(j jsonb, f jsonb) RETURNS jsonb AS $$\nDECLARE\n includes jsonb := f-> 'include';\n outj jsonb := '{}'::jsonb;\n path text[];\nBEGIN\n IF\n includes IS NULL\n OR jsonb_array_length(includes) = 0\n THEN\n RETURN j;\n ELSE\n includes := includes || '[\"id\",\"collection\"]'::jsonb;\n FOR path IN SELECT explode_dotpaths(includes) LOOP\n outj := jsonb_set_nested(outj, path, j #> path);\n END LOOP;\n END IF;\n RETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_exclude(j jsonb, f jsonb) RETURNS jsonb AS $$\nDECLARE\n excludes jsonb := f-> 'exclude';\n outj jsonb := j;\n path text[];\nBEGIN\n IF\n excludes IS NULL\n OR jsonb_array_length(excludes) = 0\n THEN\n RETURN j;\n ELSE\n FOR path IN SELECT explode_dotpaths(excludes) LOOP\n outj := outj #- path;\n END LOOP;\n END IF;\n RETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_fields(j jsonb, f jsonb DEFAULT '{\"fields\":[]}') RETURNS jsonb AS $$\n SELECT jsonb_exclude(jsonb_include(j, f), f);\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION merge_jsonb(_a jsonb, _b jsonb) RETURNS jsonb AS $$\n SELECT\n CASE\n WHEN _a = '\"𒍟※\"'::jsonb THEN NULL\n WHEN _a IS NULL OR jsonb_typeof(_a) = 'null' THEN _b\n WHEN jsonb_typeof(_a) = 'object' AND jsonb_typeof(_b) = 'object' THEN\n (\n SELECT\n jsonb_strip_nulls(\n jsonb_object_agg(\n key,\n merge_jsonb(a.value, b.value)\n )\n )\n FROM\n jsonb_each(coalesce(_a,'{}'::jsonb)) as a\n FULL JOIN\n jsonb_each(coalesce(_b,'{}'::jsonb)) as b\n USING (key)\n )\n WHEN\n jsonb_typeof(_a) = 'array'\n AND jsonb_typeof(_b) = 'array'\n AND jsonb_array_length(_a) = jsonb_array_length(_b)\n THEN\n (\n SELECT jsonb_agg(m) FROM\n ( SELECT\n merge_jsonb(\n jsonb_array_elements(_a),\n jsonb_array_elements(_b)\n ) as m\n ) as l\n )\n ELSE _a\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION strip_jsonb(_a jsonb, _b jsonb) RETURNS jsonb AS $$\n SELECT\n CASE\n\n WHEN (_a IS NULL OR jsonb_typeof(_a) = 'null') AND _b IS NOT NULL AND jsonb_typeof(_b) != 'null' THEN '\"𒍟※\"'::jsonb\n WHEN _b IS NULL OR jsonb_typeof(_a) = 'null' THEN _a\n WHEN _a = _b AND jsonb_typeof(_a) = 'object' THEN '{}'::jsonb\n WHEN _a = _b THEN NULL\n WHEN jsonb_typeof(_a) = 'object' AND jsonb_typeof(_b) = 'object' THEN\n (\n SELECT\n jsonb_strip_nulls(\n jsonb_object_agg(\n key,\n strip_jsonb(a.value, b.value)\n )\n )\n FROM\n jsonb_each(_a) as a\n FULL JOIN\n jsonb_each(_b) as b\n USING (key)\n )\n WHEN\n jsonb_typeof(_a) = 'array'\n AND jsonb_typeof(_b) = 'array'\n AND jsonb_array_length(_a) = jsonb_array_length(_b)\n THEN\n (\n SELECT jsonb_agg(m) FROM\n ( SELECT\n strip_jsonb(\n jsonb_array_elements(_a),\n jsonb_array_elements(_b)\n ) as m\n ) as l\n )\n ELSE _a\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE;\n/* looks for a geometry in a stac item first from geometry and falling back to bbox */\nCREATE OR REPLACE FUNCTION stac_geom(value jsonb) RETURNS geometry AS $$\nSELECT\n CASE\n WHEN value ? 'intersects' THEN\n ST_GeomFromGeoJSON(value->>'intersects')\n WHEN value ? 'geometry' THEN\n ST_GeomFromGeoJSON(value->>'geometry')\n WHEN value ? 'bbox' THEN\n pgstac.bbox_geom(value->'bbox')\n ELSE NULL\n END as geometry\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION stac_daterange(\n value jsonb\n) RETURNS tstzrange AS $$\nDECLARE\n props jsonb := value;\n dt timestamptz;\n edt timestamptz;\nBEGIN\n IF props ? 'properties' THEN\n props := props->'properties';\n END IF;\n IF\n props ? 'start_datetime'\n AND props->>'start_datetime' IS NOT NULL\n AND props ? 'end_datetime'\n AND props->>'end_datetime' IS NOT NULL\n THEN\n dt := props->>'start_datetime';\n edt := props->>'end_datetime';\n IF dt > edt THEN\n RAISE EXCEPTION 'start_datetime must be < end_datetime';\n END IF;\n ELSE\n dt := props->>'datetime';\n edt := props->>'datetime';\n END IF;\n IF dt is NULL OR edt IS NULL THEN\n RAISE NOTICE 'DT: %, EDT: %', dt, edt;\n RAISE EXCEPTION 'Either datetime (%) or both start_datetime (%) and end_datetime (%) must be set.', props->>'datetime',props->>'start_datetime',props->>'end_datetime';\n END IF;\n RETURN tstzrange(dt, edt, '[]');\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_datetime(value jsonb) RETURNS timestamptz AS $$\n SELECT lower(stac_daterange(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_end_datetime(value jsonb) RETURNS timestamptz AS $$\n SELECT upper(stac_daterange(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\n\nCREATE TABLE IF NOT EXISTS stac_extensions(\n name text PRIMARY KEY,\n url text,\n enbabled_by_default boolean NOT NULL DEFAULT TRUE,\n enableable boolean NOT NULL DEFAULT TRUE\n);\n\nINSERT INTO stac_extensions (name, url) VALUES\n ('fields', 'https://api.stacspec.org/v1.0.0-beta.5/item-search#fields'),\n ('sort','https://api.stacspec.org/v1.0.0-beta.5/item-search#sort'),\n ('context','https://api.stacspec.org/v1.0.0-beta.5/item-search#context'),\n ('filter', 'https://api.stacspec.org/v1.0.0-beta.5/item-search#filter'),\n ('query', 'https://api.stacspec.org/v1.0.0-beta.5/item-search#query')\nON CONFLICT (name) DO UPDATE SET url=EXCLUDED.url;\n\n\n\nCREATE OR REPLACE FUNCTION collection_base_item(content jsonb) RETURNS jsonb AS $$\n SELECT jsonb_build_object(\n 'type', 'Feature',\n 'stac_version', content->'stac_version',\n 'assets', content->'item_assets',\n 'collection', content->'id'\n );\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TYPE partition_trunc_strategy AS ENUM ('year', 'month');\n\nCREATE TABLE IF NOT EXISTS collections (\n key bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,\n id text GENERATED ALWAYS AS (content->>'id') STORED UNIQUE,\n content JSONB NOT NULL,\n base_item jsonb GENERATED ALWAYS AS (pgstac.collection_base_item(content)) STORED,\n partition_trunc partition_trunc_strategy\n);\n\n\nCREATE OR REPLACE FUNCTION collection_base_item(cid text) RETURNS jsonb AS $$\n SELECT pgstac.collection_base_item(content) FROM pgstac.collections WHERE id = cid LIMIT 1;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION table_empty(text) RETURNS boolean AS $$\nDECLARE\n retval boolean;\nBEGIN\n EXECUTE format($q$\n SELECT NOT EXISTS (SELECT 1 FROM %I LIMIT 1)\n $q$,\n $1\n ) INTO retval;\n RETURN retval;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION collections_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n q text;\n partition_name text := format('_items_%s', NEW.key);\n partition_exists boolean := false;\n partition_empty boolean := true;\n err_context text;\n loadtemp boolean := FALSE;\nBEGIN\n RAISE NOTICE 'Collection Trigger. % %', NEW.id, NEW.key;\n SELECT relid::text INTO partition_name\n FROM pg_partition_tree('items')\n WHERE relid::text = partition_name;\n IF FOUND THEN\n partition_exists := true;\n partition_empty := table_empty(partition_name);\n ELSE\n partition_exists := false;\n partition_empty := true;\n partition_name := format('_items_%s', NEW.key);\n END IF;\n IF TG_OP = 'UPDATE' AND NEW.partition_trunc IS DISTINCT FROM OLD.partition_trunc AND partition_empty THEN\n q := format($q$\n DROP TABLE IF EXISTS %I CASCADE;\n $q$,\n partition_name\n );\n EXECUTE q;\n END IF;\n IF TG_OP = 'UPDATE' AND NEW.partition_trunc IS DISTINCT FROM OLD.partition_trunc AND partition_exists AND NOT partition_empty THEN\n q := format($q$\n CREATE TEMP TABLE changepartitionstaging ON COMMIT DROP AS SELECT * FROM %I;\n DROP TABLE IF EXISTS %I CASCADE;\n $q$,\n partition_name,\n partition_name\n );\n EXECUTE q;\n loadtemp := TRUE;\n partition_empty := TRUE;\n partition_exists := FALSE;\n END IF;\n IF TG_OP = 'UPDATE' AND NEW.partition_trunc IS NOT DISTINCT FROM OLD.partition_trunc THEN\n RETURN NEW;\n END IF;\n IF NEW.partition_trunc IS NULL AND partition_empty THEN\n RAISE NOTICE '% % % %',\n partition_name,\n NEW.id,\n concat(partition_name,'_id_idx'),\n partition_name\n ;\n q := format($q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n $q$,\n partition_name,\n NEW.id,\n concat(partition_name,'_id_idx'),\n partition_name\n );\n RAISE NOTICE 'q: %', q;\n BEGIN\n EXECUTE q;\n EXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', partition_name;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\n END;\n\n ALTER TABLE partitions DISABLE TRIGGER partitions_delete_trigger;\n DELETE FROM partitions WHERE collection=NEW.id AND name=partition_name;\n ALTER TABLE partitions ENABLE TRIGGER partitions_delete_trigger;\n\n INSERT INTO partitions (collection, name) VALUES (NEW.id, partition_name);\n ELSIF partition_empty THEN\n q := format($q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L)\n PARTITION BY RANGE (datetime);\n $q$,\n partition_name,\n NEW.id\n );\n RAISE NOTICE 'q: %', q;\n BEGIN\n EXECUTE q;\n EXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', partition_name;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\n END;\n ALTER TABLE partitions DISABLE TRIGGER partitions_delete_trigger;\n DELETE FROM partitions WHERE collection=NEW.id AND name=partition_name;\n ALTER TABLE partitions ENABLE TRIGGER partitions_delete_trigger;\n ELSE\n RAISE EXCEPTION 'Cannot modify partition % unless empty', partition_name;\n END IF;\n IF loadtemp THEN\n RAISE NOTICE 'Moving data into new partitions.';\n q := format($q$\n WITH p AS (\n SELECT\n collection,\n datetime as datetime,\n end_datetime as end_datetime,\n (partition_name(\n collection,\n datetime\n )).partition_name as name\n FROM changepartitionstaging\n )\n INSERT INTO partitions (collection, datetime_range, end_datetime_range)\n SELECT\n collection,\n tstzrange(min(datetime), max(datetime), '[]') as datetime_range,\n tstzrange(min(end_datetime), max(end_datetime), '[]') as end_datetime_range\n FROM p\n GROUP BY collection, name\n ON CONFLICT (name) DO UPDATE SET\n datetime_range = EXCLUDED.datetime_range,\n end_datetime_range = EXCLUDED.end_datetime_range\n ;\n INSERT INTO %I SELECT * FROM changepartitionstaging;\n DROP TABLE IF EXISTS changepartitionstaging;\n $q$,\n partition_name\n );\n EXECUTE q;\n END IF;\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE TRIGGER collections_trigger AFTER INSERT OR UPDATE ON collections FOR EACH ROW\nEXECUTE FUNCTION collections_trigger_func();\n\nCREATE OR REPLACE FUNCTION partition_collection(collection text, strategy partition_trunc_strategy) RETURNS text AS $$\n UPDATE collections SET partition_trunc=strategy WHERE id=collection RETURNING partition_trunc;\n$$ LANGUAGE SQL;\n\nCREATE TABLE IF NOT EXISTS partitions (\n collection text REFERENCES collections(id) ON DELETE CASCADE,\n name text PRIMARY KEY,\n partition_range tstzrange NOT NULL DEFAULT tstzrange('-infinity'::timestamptz,'infinity'::timestamptz, '[]'),\n datetime_range tstzrange,\n end_datetime_range tstzrange,\n CONSTRAINT prange EXCLUDE USING GIST (\n collection WITH =,\n partition_range WITH &&\n )\n) WITH (FILLFACTOR=90);\nCREATE INDEX partitions_range_idx ON partitions USING GIST(partition_range);\n\nCREATE OR REPLACE FUNCTION partitions_delete_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n q text;\nBEGIN\n RAISE NOTICE 'Partition Delete Trigger. %', OLD.name;\n EXECUTE format($q$\n DROP TABLE IF EXISTS %I CASCADE;\n $q$,\n OLD.name\n );\n RAISE NOTICE 'Dropped partition.';\n RETURN OLD;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER partitions_delete_trigger BEFORE DELETE ON partitions FOR EACH ROW\nEXECUTE FUNCTION partitions_delete_trigger_func();\n\nCREATE OR REPLACE FUNCTION partition_name(\n IN collection text,\n IN dt timestamptz,\n OUT partition_name text,\n OUT partition_range tstzrange\n) AS $$\nDECLARE\n c RECORD;\n parent_name text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n parent_name := format('_items_%s', c.key);\n\n\n IF c.partition_trunc = 'year' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYY'));\n ELSIF c.partition_trunc = 'month' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYYMM'));\n ELSE\n partition_name := parent_name;\n partition_range := tstzrange('-infinity'::timestamptz, 'infinity'::timestamptz, '[]');\n END IF;\n IF partition_range IS NULL THEN\n partition_range := tstzrange(\n date_trunc(c.partition_trunc::text, dt),\n date_trunc(c.partition_trunc::text, dt) + concat('1 ', c.partition_trunc)::interval\n );\n END IF;\n RETURN;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\n\nCREATE OR REPLACE FUNCTION partitions_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n q text;\n cq text;\n parent_name text;\n partition_trunc text;\n partition_name text := NEW.name;\n partition_exists boolean := false;\n partition_empty boolean := true;\n partition_range tstzrange;\n datetime_range tstzrange;\n end_datetime_range tstzrange;\n err_context text;\n mindt timestamptz := lower(NEW.datetime_range);\n maxdt timestamptz := upper(NEW.datetime_range);\n minedt timestamptz := lower(NEW.end_datetime_range);\n maxedt timestamptz := upper(NEW.end_datetime_range);\n t_mindt timestamptz;\n t_maxdt timestamptz;\n t_minedt timestamptz;\n t_maxedt timestamptz;\nBEGIN\n RAISE NOTICE 'Partitions Trigger. %', NEW;\n datetime_range := NEW.datetime_range;\n end_datetime_range := NEW.end_datetime_range;\n\n SELECT\n format('_items_%s', key),\n c.partition_trunc::text\n INTO\n parent_name,\n partition_trunc\n FROM pgstac.collections c\n WHERE c.id = NEW.collection;\n SELECT (pgstac.partition_name(NEW.collection, mindt)).* INTO partition_name, partition_range;\n NEW.name := partition_name;\n\n IF partition_range IS NULL OR partition_range = 'empty'::tstzrange THEN\n partition_range := tstzrange('-infinity'::timestamptz, 'infinity'::timestamptz, '[]');\n END IF;\n\n NEW.partition_range := partition_range;\n IF TG_OP = 'UPDATE' THEN\n mindt := least(mindt, lower(OLD.datetime_range));\n maxdt := greatest(maxdt, upper(OLD.datetime_range));\n minedt := least(minedt, lower(OLD.end_datetime_range));\n maxedt := greatest(maxedt, upper(OLD.end_datetime_range));\n NEW.datetime_range := tstzrange(mindt, maxdt, '[]');\n NEW.end_datetime_range := tstzrange(minedt, maxedt, '[]');\n END IF;\n IF TG_OP = 'INSERT' THEN\n\n IF partition_range != tstzrange('-infinity'::timestamptz, 'infinity'::timestamptz, '[]') THEN\n\n RAISE NOTICE '% % %', partition_name, parent_name, partition_range;\n q := format($q$\n CREATE TABLE IF NOT EXISTS %I partition OF %I FOR VALUES FROM (%L) TO (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n $q$,\n partition_name,\n parent_name,\n lower(partition_range),\n upper(partition_range),\n format('%s_pkey', partition_name),\n partition_name\n );\n BEGIN\n EXECUTE q;\n EXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', partition_name;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\n END;\n END IF;\n\n END IF;\n\n -- Update constraints\n EXECUTE format($q$\n SELECT\n min(datetime),\n max(datetime),\n min(end_datetime),\n max(end_datetime)\n FROM %I;\n $q$, partition_name)\n INTO t_mindt, t_maxdt, t_minedt, t_maxedt;\n mindt := least(mindt, t_mindt);\n maxdt := greatest(maxdt, t_maxdt);\n minedt := least(mindt, minedt, t_minedt);\n maxedt := greatest(maxdt, maxedt, t_maxedt);\n\n mindt := date_trunc(coalesce(partition_trunc, 'year'), mindt);\n maxdt := date_trunc(coalesce(partition_trunc, 'year'), maxdt - '1 second'::interval) + concat('1 ',coalesce(partition_trunc, 'year'))::interval;\n minedt := date_trunc(coalesce(partition_trunc, 'year'), minedt);\n maxedt := date_trunc(coalesce(partition_trunc, 'year'), maxedt - '1 second'::interval) + concat('1 ',coalesce(partition_trunc, 'year'))::interval;\n\n\n IF mindt IS NOT NULL AND maxdt IS NOT NULL AND minedt IS NOT NULL AND maxedt IS NOT NULL THEN\n NEW.datetime_range := tstzrange(mindt, maxdt, '[]');\n NEW.end_datetime_range := tstzrange(minedt, maxedt, '[]');\n IF\n TG_OP='UPDATE'\n AND OLD.datetime_range @> NEW.datetime_range\n AND OLD.end_datetime_range @> NEW.end_datetime_range\n THEN\n RAISE NOTICE 'Range unchanged, not updating constraints.';\n ELSE\n\n RAISE NOTICE '\n SETTING CONSTRAINTS\n mindt: %, maxdt: %\n minedt: %, maxedt: %\n ', mindt, maxdt, minedt, maxedt;\n IF partition_trunc IS NULL THEN\n cq := format($q$\n ALTER TABLE %7$I\n DROP CONSTRAINT IF EXISTS %1$I,\n DROP CONSTRAINT IF EXISTS %2$I,\n ADD CONSTRAINT %1$I\n CHECK (\n (datetime >= %3$L)\n AND (datetime <= %4$L)\n AND (end_datetime >= %5$L)\n AND (end_datetime <= %6$L)\n ) NOT VALID\n ;\n ALTER TABLE %7$I\n VALIDATE CONSTRAINT %1$I;\n $q$,\n format('%s_dt', partition_name),\n format('%s_edt', partition_name),\n mindt,\n maxdt,\n minedt,\n maxedt,\n partition_name\n );\n ELSE\n cq := format($q$\n ALTER TABLE %5$I\n DROP CONSTRAINT IF EXISTS %1$I,\n DROP CONSTRAINT IF EXISTS %2$I,\n ADD CONSTRAINT %2$I\n CHECK ((end_datetime >= %3$L) AND (end_datetime <= %4$L)) NOT VALID\n ;\n ALTER TABLE %5$I\n VALIDATE CONSTRAINT %2$I;\n $q$,\n format('%s_dt', partition_name),\n format('%s_edt', partition_name),\n minedt,\n maxedt,\n partition_name\n );\n\n END IF;\n RAISE NOTICE 'Altering Constraints. %', cq;\n EXECUTE cq;\n END IF;\n ELSE\n NEW.datetime_range = NULL;\n NEW.end_datetime_range = NULL;\n\n cq := format($q$\n ALTER TABLE %3$I\n DROP CONSTRAINT IF EXISTS %1$I,\n DROP CONSTRAINT IF EXISTS %2$I,\n ADD CONSTRAINT %1$I\n CHECK ((datetime IS NULL AND end_datetime IS NULL)) NOT VALID\n ;\n ALTER TABLE %3$I\n VALIDATE CONSTRAINT %1$I;\n $q$,\n format('%s_dt', partition_name),\n format('%s_edt', partition_name),\n partition_name\n );\n EXECUTE cq;\n END IF;\n\n RETURN NEW;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER partitions_trigger BEFORE INSERT OR UPDATE ON partitions FOR EACH ROW\nEXECUTE FUNCTION partitions_trigger_func();\n\n\nCREATE OR REPLACE FUNCTION create_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection(data jsonb) RETURNS VOID AS $$\nDECLARE\n out collections%ROWTYPE;\nBEGIN\n UPDATE collections SET content=data WHERE id = data->>'id' RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ON CONFLICT (id) DO\n UPDATE\n SET content=EXCLUDED.content\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION delete_collection(_id text) RETURNS VOID AS $$\nDECLARE\n out collections%ROWTYPE;\nBEGIN\n DELETE FROM collections WHERE id = _id RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION get_collection(id text) RETURNS jsonb AS $$\n SELECT content FROM collections\n WHERE id=$1\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION all_collections() RETURNS jsonb AS $$\n SELECT jsonb_agg(content) FROM collections;\n;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\nCREATE TABLE queryables (\n id bigint GENERATED ALWAYS AS identity PRIMARY KEY,\n name text UNIQUE NOT NULL,\n collection_ids text[], -- used to determine what partitions to create indexes on\n definition jsonb,\n property_path text,\n property_wrapper text,\n property_index_type text\n);\nCREATE INDEX queryables_name_idx ON queryables (name);\nCREATE INDEX queryables_property_wrapper_idx ON queryables (property_wrapper);\n\n\nINSERT INTO queryables (name, definition) VALUES\n('id', '{\"title\": \"Item ID\",\"description\": \"Item identifier\",\"$ref\": \"https://schemas.stacspec.org/v1.0.0/item-spec/json-schema/item.json#/definitions/core/allOf/2/properties/id\"}'),\n('datetime','{\"description\": \"Datetime\",\"type\": \"string\",\"title\": \"Acquired\",\"format\": \"date-time\",\"pattern\": \"(\\\\+00:00|Z)$\"}'),\n('geometry', '{\"title\": \"Item Geometry\",\"description\": \"Item Geometry\",\"$ref\": \"https://geojson.org/schema/Feature.json\"}')\nON CONFLICT DO NOTHING;\n\nINSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n('eo:cloud_cover','{\"$ref\": \"https://stac-extensions.github.io/eo/v1.0.0/schema.json#/definitions/fieldsproperties/eo:cloud_cover\"}','to_int','BTREE')\nON CONFLICT DO NOTHING;\n\nCREATE OR REPLACE FUNCTION array_to_path(arr text[]) RETURNS text AS $$\n SELECT string_agg(\n quote_literal(v),\n '->'\n ) FROM unnest(arr) v;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\n\n\nCREATE OR REPLACE FUNCTION queryable(\n IN dotpath text,\n OUT path text,\n OUT expression text,\n OUT wrapper text,\n OUT nulled_wrapper text\n) AS $$\nDECLARE\n q RECORD;\n path_elements text[];\nBEGIN\n IF dotpath IN ('id', 'geometry', 'datetime', 'end_datetime', 'collection') THEN\n path := dotpath;\n expression := dotpath;\n wrapper := NULL;\n RETURN;\n END IF;\n SELECT * INTO q FROM queryables\n WHERE\n name=dotpath\n OR name = 'properties.' || dotpath\n OR name = replace(dotpath, 'properties.', '')\n ;\n IF q.property_wrapper IS NULL THEN\n IF q.definition->>'type' = 'number' THEN\n wrapper := 'to_float';\n nulled_wrapper := wrapper;\n ELSIF q.definition->>'format' = 'date-time' THEN\n wrapper := 'to_tstz';\n nulled_wrapper := wrapper;\n ELSE\n nulled_wrapper := NULL;\n wrapper := 'to_text';\n END IF;\n ELSE\n wrapper := q.property_wrapper;\n nulled_wrapper := wrapper;\n END IF;\n IF q.property_path IS NOT NULL THEN\n path := q.property_path;\n ELSE\n path_elements := string_to_array(dotpath, '.');\n IF path_elements[1] IN ('links', 'assets', 'stac_version', 'stac_extensions') THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSIF path_elements[1] = 'properties' THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSE\n path := format($F$content->'properties'->%s$F$, array_to_path(path_elements));\n END IF;\n END IF;\n expression := format('%I(%s)', wrapper, path);\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\nCREATE OR REPLACE FUNCTION create_queryable_indexes() RETURNS VOID AS $$\nDECLARE\n queryable RECORD;\n q text;\nBEGIN\n FOR queryable IN\n SELECT\n queryables.id as qid,\n CASE WHEN collections.key IS NULL THEN 'items' ELSE format('_items_%s',collections.key) END AS part,\n property_index_type,\n expression\n FROM\n queryables\n LEFT JOIN collections ON (collections.id = ANY (queryables.collection_ids))\n JOIN LATERAL queryable(queryables.name) ON (queryables.property_index_type IS NOT NULL)\n LOOP\n q := format(\n $q$\n CREATE INDEX IF NOT EXISTS %I ON %I USING %s ((%s));\n $q$,\n format('%s_%s_idx', queryable.part, queryable.qid),\n queryable.part,\n COALESCE(queryable.property_index_type, 'to_text'),\n queryable.expression\n );\n RAISE NOTICE '%',q;\n EXECUTE q;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION queryables_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\nBEGIN\nPERFORM create_queryable_indexes();\nRETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER queryables_trigger AFTER INSERT OR UPDATE ON queryables\nFOR EACH STATEMENT EXECUTE PROCEDURE queryables_trigger_func();\n\nCREATE TRIGGER queryables_collection_trigger AFTER INSERT OR UPDATE ON collections\nFOR EACH STATEMENT EXECUTE PROCEDURE queryables_trigger_func();\n\nCREATE OR REPLACE FUNCTION get_queryables(_collection_ids text[] DEFAULT NULL) RETURNS jsonb AS $$\nBEGIN\n -- Build up queryables if the input contains valid collection ids or is empty\n IF EXISTS (\n SELECT 1 FROM collections\n WHERE\n _collection_ids IS NULL\n OR cardinality(_collection_ids) = 0\n OR id = ANY(_collection_ids)\n )\n THEN\n RETURN (\n SELECT\n jsonb_build_object(\n '$schema', 'http://json-schema.org/draft-07/schema#',\n '$id', 'https://example.org/queryables',\n 'type', 'object',\n 'title', 'STAC Queryables.',\n 'properties', jsonb_object_agg(\n name,\n definition\n )\n )\n FROM queryables\n WHERE\n _collection_ids IS NULL OR\n cardinality(_collection_ids) = 0 OR\n collection_ids IS NULL OR\n _collection_ids && collection_ids\n );\n ELSE\n RETURN NULL;\n END IF;\nEND;\n\n$$ LANGUAGE PLPGSQL STABLE;\n\nCREATE OR REPLACE FUNCTION get_queryables(_collection text DEFAULT NULL) RETURNS jsonb AS $$\n SELECT\n CASE\n WHEN _collection IS NULL THEN get_queryables(NULL::text[])\n ELSE get_queryables(ARRAY[_collection])\n END\n ;\n$$ LANGUAGE SQL;\n\n\nCREATE OR REPLACE FUNCTION missing_queryables(_collection text, _tablesample int DEFAULT 5) RETURNS TABLE(collection text, name text, definition jsonb, property_wrapper text) AS $$\nDECLARE\n q text;\n _partition text;\n explain_json json;\n psize bigint;\nBEGIN\n SELECT format('_items_%s', key) INTO _partition FROM collections WHERE id=_collection;\n\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM %I;', _partition)\n INTO explain_json;\n psize := explain_json->0->'Plan'->'Plan Rows';\n IF _tablesample * .01 * psize < 10 THEN\n _tablesample := 100;\n END IF;\n RAISE NOTICE 'Using tablesample % to find missing queryables from % % that has ~% rows', _tablesample, _collection, _partition, psize;\n\n q := format(\n $q$\n WITH q AS (\n SELECT * FROM queryables\n WHERE\n collection_ids IS NULL\n OR %L = ANY(collection_ids)\n ), t AS (\n SELECT\n content->'properties' AS properties\n FROM\n %I\n TABLESAMPLE SYSTEM(%L)\n ), p AS (\n SELECT DISTINCT ON (key)\n key,\n value\n FROM t\n JOIN LATERAL jsonb_each(properties) ON TRUE\n LEFT JOIN q ON (q.name=key)\n WHERE q.definition IS NULL\n )\n SELECT\n %L,\n key,\n jsonb_build_object('type',jsonb_typeof(value)) as definition,\n CASE jsonb_typeof(value)\n WHEN 'number' THEN 'to_float'\n WHEN 'array' THEN 'to_text_array'\n ELSE 'to_text'\n END\n FROM p;\n $q$,\n _collection,\n _partition,\n _tablesample,\n _collection\n );\n RETURN QUERY EXECUTE q;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION missing_queryables(_tablesample int DEFAULT 5) RETURNS TABLE(collection_ids text[], name text, definition jsonb, property_wrapper text) AS $$\n SELECT\n array_agg(collection),\n name,\n definition,\n property_wrapper\n FROM\n collections\n JOIN LATERAL\n missing_queryables(id, _tablesample) c\n ON TRUE\n GROUP BY\n 2,3,4\n ORDER BY 2,1\n ;\n$$ LANGUAGE SQL;\nCREATE OR REPLACE FUNCTION parse_dtrange(\n _indate jsonb,\n relative_base timestamptz DEFAULT date_trunc('hour', CURRENT_TIMESTAMP)\n) RETURNS tstzrange AS $$\nDECLARE\n timestrs text[];\n s timestamptz;\n e timestamptz;\nBEGIN\n timestrs :=\n CASE\n WHEN _indate ? 'timestamp' THEN\n ARRAY[_indate->>'timestamp']\n WHEN _indate ? 'interval' THEN\n to_text_array(_indate->'interval')\n WHEN jsonb_typeof(_indate) = 'array' THEN\n to_text_array(_indate)\n ELSE\n regexp_split_to_array(\n _indate->>0,\n '/'\n )\n END;\n RAISE NOTICE 'TIMESTRS %', timestrs;\n IF cardinality(timestrs) = 1 THEN\n IF timestrs[1] ILIKE 'P%' THEN\n RETURN tstzrange(relative_base - upper(timestrs[1])::interval, relative_base, '[)');\n END IF;\n s := timestrs[1]::timestamptz;\n RETURN tstzrange(s, s, '[]');\n END IF;\n\n IF cardinality(timestrs) != 2 THEN\n RAISE EXCEPTION 'Timestamp cannot have more than 2 values';\n END IF;\n\n IF timestrs[1] = '..' THEN\n s := '-infinity'::timestamptz;\n e := timestrs[2]::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] = '..' THEN\n s := timestrs[1]::timestamptz;\n e := 'infinity'::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[1] ILIKE 'P%' AND timestrs[2] NOT ILIKE 'P%' THEN\n e := timestrs[2]::timestamptz;\n s := e - upper(timestrs[1])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] ILIKE 'P%' AND timestrs[1] NOT ILIKE 'P%' THEN\n s := timestrs[1]::timestamptz;\n e := s + upper(timestrs[2])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n s := timestrs[1]::timestamptz;\n e := timestrs[2]::timestamptz;\n\n RETURN tstzrange(s,e,'[)');\n\n RETURN NULL;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT PARALLEL SAFE SET TIME ZONE 'UTC';\n\nCREATE OR REPLACE FUNCTION parse_dtrange(\n _indate text,\n relative_base timestamptz DEFAULT CURRENT_TIMESTAMP\n) RETURNS tstzrange AS $$\n SELECT parse_dtrange(to_jsonb(_indate), relative_base);\n$$ LANGUAGE SQL STABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION temporal_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\n ll text := 'datetime';\n lh text := 'end_datetime';\n rrange tstzrange;\n rl text;\n rh text;\n outq text;\nBEGIN\n rrange := parse_dtrange(args->1);\n RAISE NOTICE 'Constructing temporal query OP: %, ARGS: %, RRANGE: %', op, args, rrange;\n op := lower(op);\n rl := format('%L::timestamptz', lower(rrange));\n rh := format('%L::timestamptz', upper(rrange));\n outq := CASE op\n WHEN 't_before' THEN 'lh < rl'\n WHEN 't_after' THEN 'll > rh'\n WHEN 't_meets' THEN 'lh = rl'\n WHEN 't_metby' THEN 'll = rh'\n WHEN 't_overlaps' THEN 'll < rl AND rl < lh < rh'\n WHEN 't_overlappedby' THEN 'rl < ll < rh AND lh > rh'\n WHEN 't_starts' THEN 'll = rl AND lh < rh'\n WHEN 't_startedby' THEN 'll = rl AND lh > rh'\n WHEN 't_during' THEN 'll > rl AND lh < rh'\n WHEN 't_contains' THEN 'll < rl AND lh > rh'\n WHEN 't_finishes' THEN 'll > rl AND lh = rh'\n WHEN 't_finishedby' THEN 'll < rl AND lh = rh'\n WHEN 't_equals' THEN 'll = rl AND lh = rh'\n WHEN 't_disjoint' THEN 'NOT (ll <= rh AND lh >= rl)'\n WHEN 't_intersects' THEN 'll <= rh AND lh >= rl'\n WHEN 'anyinteracts' THEN 'll <= rh AND lh >= rl'\n END;\n outq := regexp_replace(outq, '\\mll\\M', ll);\n outq := regexp_replace(outq, '\\mlh\\M', lh);\n outq := regexp_replace(outq, '\\mrl\\M', rl);\n outq := regexp_replace(outq, '\\mrh\\M', rh);\n outq := format('(%s)', outq);\n RETURN outq;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\n\n\nCREATE OR REPLACE FUNCTION spatial_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\n geom text;\n j jsonb := args->1;\nBEGIN\n op := lower(op);\n RAISE NOTICE 'Constructing spatial query OP: %, ARGS: %', op, args;\n IF op NOT IN ('s_equals','s_disjoint','s_touches','s_within','s_overlaps','s_crosses','s_intersects','intersects','s_contains') THEN\n RAISE EXCEPTION 'Spatial Operator % Not Supported', op;\n END IF;\n op := regexp_replace(op, '^s_', 'st_');\n IF op = 'intersects' THEN\n op := 'st_intersects';\n END IF;\n -- Convert geometry to WKB string\n IF j ? 'type' AND j ? 'coordinates' THEN\n geom := st_geomfromgeojson(j)::text;\n ELSIF jsonb_typeof(j) = 'array' THEN\n geom := bbox_geom(j)::text;\n END IF;\n\n RETURN format('%s(geometry, %L::geometry)', op, geom);\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION query_to_cql2(q jsonb) RETURNS jsonb AS $$\n-- Translates anything passed in through the deprecated \"query\" into equivalent CQL2\nWITH t AS (\n SELECT key as property, value as ops\n FROM jsonb_each(q)\n), t2 AS (\n SELECT property, (jsonb_each(ops)).*\n FROM t WHERE jsonb_typeof(ops) = 'object'\n UNION ALL\n SELECT property, 'eq', ops\n FROM t WHERE jsonb_typeof(ops) != 'object'\n)\nSELECT\n jsonb_strip_nulls(jsonb_build_object(\n 'op', 'and',\n 'args', jsonb_agg(\n jsonb_build_object(\n 'op', key,\n 'args', jsonb_build_array(\n jsonb_build_object('property',property),\n value\n )\n )\n )\n )\n) as qcql FROM t2\n;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\nCREATE OR REPLACE FUNCTION cql1_to_cql2(j jsonb) RETURNS jsonb AS $$\nDECLARE\n args jsonb;\n ret jsonb;\nBEGIN\n RAISE NOTICE 'CQL1_TO_CQL2: %', j;\n IF j ? 'filter' THEN\n RETURN cql1_to_cql2(j->'filter');\n END IF;\n IF j ? 'property' THEN\n RETURN j;\n END IF;\n IF jsonb_typeof(j) = 'array' THEN\n SELECT jsonb_agg(cql1_to_cql2(el)) INTO args FROM jsonb_array_elements(j) el;\n RETURN args;\n END IF;\n IF jsonb_typeof(j) = 'number' THEN\n RETURN j;\n END IF;\n IF jsonb_typeof(j) = 'string' THEN\n RETURN j;\n END IF;\n\n IF jsonb_typeof(j) = 'object' THEN\n SELECT jsonb_build_object(\n 'op', key,\n 'args', cql1_to_cql2(value)\n ) INTO ret\n FROM jsonb_each(j)\n WHERE j IS NOT NULL;\n RETURN ret;\n END IF;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE STRICT;\n\nCREATE TABLE cql2_ops (\n op text PRIMARY KEY,\n template text,\n types text[]\n);\nINSERT INTO cql2_ops (op, template, types) VALUES\n ('eq', '%s = %s', NULL),\n ('neq', '%s != %s', NULL),\n ('ne', '%s != %s', NULL),\n ('!=', '%s != %s', NULL),\n ('<>', '%s != %s', NULL),\n ('lt', '%s < %s', NULL),\n ('lte', '%s <= %s', NULL),\n ('gt', '%s > %s', NULL),\n ('gte', '%s >= %s', NULL),\n ('le', '%s <= %s', NULL),\n ('ge', '%s >= %s', NULL),\n ('=', '%s = %s', NULL),\n ('<', '%s < %s', NULL),\n ('<=', '%s <= %s', NULL),\n ('>', '%s > %s', NULL),\n ('>=', '%s >= %s', NULL),\n ('like', '%s LIKE %s', NULL),\n ('ilike', '%s ILIKE %s', NULL),\n ('+', '%s + %s', NULL),\n ('-', '%s - %s', NULL),\n ('*', '%s * %s', NULL),\n ('/', '%s / %s', NULL),\n ('not', 'NOT (%s)', NULL),\n ('between', '%s BETWEEN %s AND %s', NULL),\n ('isnull', '%s IS NULL', NULL),\n ('upper', 'upper(%s)', NULL),\n ('lower', 'lower(%s)', NULL)\nON CONFLICT (op) DO UPDATE\n SET\n template = EXCLUDED.template\n;\n\n\nCREATE OR REPLACE FUNCTION cql2_query(j jsonb, wrapper text DEFAULT NULL) RETURNS text AS $$\n#variable_conflict use_variable\nDECLARE\n args jsonb := j->'args';\n arg jsonb;\n op text := lower(j->>'op');\n cql2op RECORD;\n literal text;\n _wrapper text;\n leftarg text;\n rightarg text;\nBEGIN\n IF j IS NULL OR (op IS NOT NULL AND args IS NULL) THEN\n RETURN NULL;\n END IF;\n RAISE NOTICE 'CQL2_QUERY: %', j;\n IF j ? 'filter' THEN\n RETURN cql2_query(j->'filter');\n END IF;\n\n IF j ? 'upper' THEN\n RETURN cql2_query(jsonb_build_object('op', 'upper', 'args', j->'upper'));\n END IF;\n\n IF j ? 'lower' THEN\n RETURN cql2_query(jsonb_build_object('op', 'lower', 'args', j->'lower'));\n END IF;\n\n -- Temporal Query\n IF op ilike 't_%' or op = 'anyinteracts' THEN\n RETURN temporal_op_query(op, args);\n END IF;\n\n -- If property is a timestamp convert it to text to use with\n -- general operators\n IF j ? 'timestamp' THEN\n RETURN format('%L::timestamptz', to_tstz(j->'timestamp'));\n END IF;\n IF j ? 'interval' THEN\n RAISE EXCEPTION 'Please use temporal operators when using intervals.';\n RETURN NONE;\n END IF;\n\n -- Spatial Query\n IF op ilike 's_%' or op = 'intersects' THEN\n RETURN spatial_op_query(op, args);\n END IF;\n\n IF op IN ('a_equals','a_contains','a_contained_by','a_overlaps') THEN\n IF args->0 ? 'property' THEN\n leftarg := format('to_text_array(%s)', (queryable(args->0->>'property')).path);\n END IF;\n IF args->1 ? 'property' THEN\n rightarg := format('to_text_array(%s)', (queryable(args->1->>'property')).path);\n END IF;\n RETURN FORMAT(\n '%s %s %s',\n COALESCE(leftarg, quote_literal(to_text_array(args->0))),\n CASE op\n WHEN 'a_equals' THEN '='\n WHEN 'a_contains' THEN '@>'\n WHEN 'a_contained_by' THEN '<@'\n WHEN 'a_overlaps' THEN '&&'\n END,\n COALESCE(rightarg, quote_literal(to_text_array(args->1)))\n );\n END IF;\n\n IF op = 'in' THEN\n RAISE NOTICE 'IN : % % %', args, jsonb_build_array(args->0), args->1;\n args := jsonb_build_array(args->0) || (args->1);\n RAISE NOTICE 'IN2 : %', args;\n END IF;\n\n\n\n IF op = 'between' THEN\n args = jsonb_build_array(\n args->0,\n args->1->0,\n args->1->1\n );\n END IF;\n\n -- Make sure that args is an array and run cql2_query on\n -- each element of the array\n RAISE NOTICE 'ARGS PRE: %', args;\n IF j ? 'args' THEN\n IF jsonb_typeof(args) != 'array' THEN\n args := jsonb_build_array(args);\n END IF;\n\n IF jsonb_path_exists(args, '$[*] ? (@.property == \"id\" || @.property == \"datetime\" || @.property == \"end_datetime\" || @.property == \"collection\")') THEN\n wrapper := NULL;\n ELSE\n -- if any of the arguments are a property, try to get the property_wrapper\n FOR arg IN SELECT jsonb_path_query(args, '$[*] ? (@.property != null)') LOOP\n RAISE NOTICE 'Arg: %', arg;\n wrapper := (queryable(arg->>'property')).nulled_wrapper;\n RAISE NOTICE 'Property: %, Wrapper: %', arg, wrapper;\n IF wrapper IS NOT NULL THEN\n EXIT;\n END IF;\n END LOOP;\n\n -- if the property was not in queryables, see if any args were numbers\n IF\n wrapper IS NULL\n AND jsonb_path_exists(args, '$[*] ? (@.type()==\"number\")')\n THEN\n wrapper := 'to_float';\n END IF;\n wrapper := coalesce(wrapper, 'to_text');\n END IF;\n\n SELECT jsonb_agg(cql2_query(a, wrapper))\n INTO args\n FROM jsonb_array_elements(args) a;\n END IF;\n RAISE NOTICE 'ARGS: %', args;\n\n IF op IN ('and', 'or') THEN\n RETURN\n format(\n '(%s)',\n array_to_string(to_text_array(args), format(' %s ', upper(op)))\n );\n END IF;\n\n IF op = 'in' THEN\n RAISE NOTICE 'IN -- % %', args->0, to_text(args->0);\n RETURN format(\n '%s IN (%s)',\n to_text(args->0),\n array_to_string((to_text_array(args))[2:], ',')\n );\n END IF;\n\n -- Look up template from cql2_ops\n IF j ? 'op' THEN\n SELECT * INTO cql2op FROM cql2_ops WHERE cql2_ops.op ilike op;\n IF FOUND THEN\n -- If specific index set in queryables for a property cast other arguments to that type\n\n RETURN format(\n cql2op.template,\n VARIADIC (to_text_array(args))\n );\n ELSE\n RAISE EXCEPTION 'Operator % Not Supported.', op;\n END IF;\n END IF;\n\n\n IF wrapper IS NOT NULL THEN\n RAISE NOTICE 'Wrapping % with %', j, wrapper;\n IF j ? 'property' THEN\n RETURN format('%I(%s)', wrapper, (queryable(j->>'property')).path);\n ELSE\n RETURN format('%I(%L)', wrapper, j);\n END IF;\n ELSIF j ? 'property' THEN\n RETURN quote_ident(j->>'property');\n END IF;\n\n RETURN quote_literal(to_text(j));\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION paging_dtrange(\n j jsonb\n) RETURNS tstzrange AS $$\nDECLARE\n op text;\n filter jsonb := j->'filter';\n dtrange tstzrange := tstzrange('-infinity'::timestamptz,'infinity'::timestamptz);\n sdate timestamptz := '-infinity'::timestamptz;\n edate timestamptz := 'infinity'::timestamptz;\n jpitem jsonb;\nBEGIN\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n END IF;\n IF NOT (filter @? '$.**.op ? (@ == \"or\" || @ == \"not\")') THEN\n FOR jpitem IN SELECT j FROM jsonb_path_query(filter,'strict $.** ? (@.args[*].property == \"datetime\")'::jsonpath) j LOOP\n op := lower(jpitem->>'op');\n dtrange := parse_dtrange(jpitem->'args'->1);\n IF op IN ('<=', 'lt', 'lte', '<', 'le', 't_before') THEN\n sdate := greatest(sdate,'-infinity');\n edate := least(edate, upper(dtrange));\n ELSIF op IN ('>=', '>', 'gt', 'gte', 'ge', 't_after') THEN\n edate := least(edate, 'infinity');\n sdate := greatest(sdate, lower(dtrange));\n ELSIF op IN ('=', 'eq') THEN\n edate := least(edate, upper(dtrange));\n sdate := greatest(sdate, lower(dtrange));\n END IF;\n RAISE NOTICE '2 OP: %, ARGS: %, DTRANGE: %, SDATE: %, EDATE: %', op, jpitem->'args'->1, dtrange, sdate, edate;\n END LOOP;\n END IF;\n IF sdate > edate THEN\n RETURN 'empty'::tstzrange;\n END IF;\n RETURN tstzrange(sdate,edate, '[]');\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT SET TIME ZONE 'UTC';\n\nCREATE OR REPLACE FUNCTION paging_collections(\n IN j jsonb\n) RETURNS text[] AS $$\nDECLARE\n filter jsonb := j->'filter';\n jpitem jsonb;\n op text;\n args jsonb;\n arg jsonb;\n collections text[];\nBEGIN\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n END IF;\n IF NOT (filter @? '$.**.op ? (@ == \"or\" || @ == \"not\")') THEN\n FOR jpitem IN SELECT j FROM jsonb_path_query(filter,'strict $.** ? (@.args[*].property == \"collection\")'::jsonpath) j LOOP\n RAISE NOTICE 'JPITEM: %', jpitem;\n op := jpitem->>'op';\n args := jpitem->'args';\n IF op IN ('=', 'eq', 'in') THEN\n FOR arg IN SELECT a FROM jsonb_array_elements(args) a LOOP\n IF jsonb_typeof(arg) IN ('string', 'array') THEN\n RAISE NOTICE 'arg: %, collections: %', arg, collections;\n IF collections IS NULL OR collections = '{}'::text[] THEN\n collections := to_text_array(arg);\n ELSE\n collections := array_intersection(collections, to_text_array(arg));\n END IF;\n END IF;\n END LOOP;\n END IF;\n END LOOP;\n END IF;\n IF collections = '{}'::text[] THEN\n RETURN NULL;\n END IF;\n RETURN collections;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\nCREATE TABLE items (\n id text NOT NULL,\n geometry geometry NOT NULL,\n collection text NOT NULL,\n datetime timestamptz NOT NULL,\n end_datetime timestamptz NOT NULL,\n content JSONB NOT NULL\n)\nPARTITION BY LIST (collection)\n;\n\nCREATE INDEX \"datetime_idx\" ON items USING BTREE (datetime DESC, end_datetime ASC);\nCREATE INDEX \"geometry_idx\" ON items USING GIST (geometry);\n\nCREATE STATISTICS datetime_stats (dependencies) on datetime, end_datetime from items;\n\n\nALTER TABLE items ADD CONSTRAINT items_collections_fk FOREIGN KEY (collection) REFERENCES collections(id) ON DELETE CASCADE DEFERRABLE;\n\n\nCREATE OR REPLACE FUNCTION content_slim(_item jsonb) RETURNS jsonb AS $$\n SELECT strip_jsonb(_item - '{id,geometry,collection,type}'::text[], collection_base_item(_item->>'collection')) - '{id,geometry,collection,type}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_dehydrate(content jsonb) RETURNS items AS $$\n SELECT\n content->>'id' as id,\n stac_geom(content) as geometry,\n content->>'collection' as collection,\n stac_datetime(content) as datetime,\n stac_end_datetime(content) as end_datetime,\n content_slim(content) as content\n ;\n$$ LANGUAGE SQL STABLE;\n\nCREATE OR REPLACE FUNCTION include_field(f text, fields jsonb DEFAULT '{}'::jsonb) RETURNS boolean AS $$\nDECLARE\n includes jsonb := fields->'include';\n excludes jsonb := fields->'exclude';\nBEGIN\n IF f IS NULL THEN\n RETURN NULL;\n END IF;\n\n\n IF\n jsonb_typeof(excludes) = 'array'\n AND jsonb_array_length(excludes)>0\n AND excludes ? f\n THEN\n RETURN FALSE;\n END IF;\n\n IF\n (\n jsonb_typeof(includes) = 'array'\n AND jsonb_array_length(includes) > 0\n AND includes ? f\n ) OR\n (\n includes IS NULL\n OR jsonb_typeof(includes) = 'null'\n OR jsonb_array_length(includes) = 0\n )\n THEN\n RETURN TRUE;\n END IF;\n\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nDROP FUNCTION IF EXISTS content_hydrate(jsonb, jsonb, jsonb);\nCREATE OR REPLACE FUNCTION content_hydrate(\n _item jsonb,\n _base_item jsonb,\n fields jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\n SELECT merge_jsonb(\n jsonb_fields(_item, fields),\n jsonb_fields(_base_item, fields)\n );\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION content_hydrate(_item items, _collection collections, fields jsonb DEFAULT '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\n content jsonb;\n base_item jsonb := _collection.base_item;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry, 20)::jsonb;\n END IF;\n output := content_hydrate(\n jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'collection', _item.collection,\n 'type', 'Feature'\n ) || _item.content,\n _collection.base_item,\n fields\n );\n\n RETURN output;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_nonhydrated(\n _item items,\n fields jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry, 20)::jsonb;\n END IF;\n output := jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'collection', _item.collection,\n 'type', 'Feature'\n ) || _item.content;\n RETURN output;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_hydrate(_item items, fields jsonb DEFAULT '{}'::jsonb) RETURNS jsonb AS $$\n SELECT content_hydrate(\n _item,\n (SELECT c FROM collections c WHERE id=_item.collection LIMIT 1),\n fields\n );\n$$ LANGUAGE SQL STABLE;\n\n\nCREATE UNLOGGED TABLE items_staging (\n content JSONB NOT NULL\n);\nCREATE UNLOGGED TABLE items_staging_ignore (\n content JSONB NOT NULL\n);\nCREATE UNLOGGED TABLE items_staging_upsert (\n content JSONB NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION items_staging_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p record;\n _partitions text[];\n ts timestamptz := clock_timestamp();\nBEGIN\n RAISE NOTICE 'Creating Partitions. %', clock_timestamp() - ts;\n WITH ranges AS (\n SELECT\n n.content->>'collection' as collection,\n stac_daterange(n.content->'properties') as dtr\n FROM newdata n\n ), p AS (\n SELECT\n collection,\n lower(dtr) as datetime,\n upper(dtr) as end_datetime,\n (partition_name(\n collection,\n lower(dtr)\n )).partition_name as name\n FROM ranges\n )\n INSERT INTO partitions (collection, datetime_range, end_datetime_range)\n SELECT\n collection,\n tstzrange(min(datetime), max(datetime), '[]') as datetime_range,\n tstzrange(min(end_datetime), max(end_datetime), '[]') as end_datetime_range\n FROM p\n GROUP BY collection, name\n ON CONFLICT (name) DO UPDATE SET\n datetime_range = EXCLUDED.datetime_range,\n end_datetime_range = EXCLUDED.end_datetime_range\n ;\n\n RAISE NOTICE 'Doing the insert. %', clock_timestamp() - ts;\n IF TG_TABLE_NAME = 'items_staging' THEN\n INSERT INTO items\n SELECT\n (content_dehydrate(content)).*\n FROM newdata;\n DELETE FROM items_staging;\n ELSIF TG_TABLE_NAME = 'items_staging_ignore' THEN\n INSERT INTO items\n SELECT\n (content_dehydrate(content)).*\n FROM newdata\n ON CONFLICT DO NOTHING;\n DELETE FROM items_staging_ignore;\n ELSIF TG_TABLE_NAME = 'items_staging_upsert' THEN\n WITH staging_formatted AS (\n SELECT (content_dehydrate(content)).* FROM newdata\n ), deletes AS (\n DELETE FROM items i USING staging_formatted s\n WHERE\n i.id = s.id\n AND i.collection = s.collection\n AND i IS DISTINCT FROM s\n RETURNING i.id, i.collection\n )\n INSERT INTO items\n SELECT s.* FROM\n staging_formatted s\n JOIN deletes d\n USING (id, collection);\n DELETE FROM items_staging_upsert;\n END IF;\n RAISE NOTICE 'Done. %', clock_timestamp() - ts;\n\n RETURN NULL;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER items_staging_insert_trigger AFTER INSERT ON items_staging REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\nCREATE TRIGGER items_staging_insert_ignore_trigger AFTER INSERT ON items_staging_ignore REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\nCREATE TRIGGER items_staging_insert_upsert_trigger AFTER INSERT ON items_staging_upsert REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\n\n\n\nCREATE OR REPLACE FUNCTION item_by_id(_id text, _collection text DEFAULT NULL) RETURNS items AS\n$$\nDECLARE\n i items%ROWTYPE;\nBEGIN\n SELECT * INTO i FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection) LIMIT 1;\n RETURN i;\nEND;\n$$ LANGUAGE PLPGSQL STABLE SECURITY DEFINER SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION get_item(_id text, _collection text DEFAULT NULL) RETURNS jsonb AS $$\n SELECT content_hydrate(items) FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection);\n$$ LANGUAGE SQL STABLE SECURITY DEFINER SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION delete_item(_id text, _collection text DEFAULT NULL) RETURNS VOID AS $$\nDECLARE\nout items%ROWTYPE;\nBEGIN\n DELETE FROM items WHERE id = _id AND (_collection IS NULL OR collection=_collection) RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n--/*\nCREATE OR REPLACE FUNCTION create_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION update_item(content jsonb) RETURNS VOID AS $$\nDECLARE\n old items %ROWTYPE;\n out items%ROWTYPE;\nBEGIN\n PERFORM delete_item(content->>'id', content->>'collection');\n PERFORM create_item(content);\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION create_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION collection_bbox(id text) RETURNS jsonb AS $$\n SELECT (replace(replace(replace(st_extent(geometry)::text,'BOX(','[['),')',']]'),' ',','))::jsonb\n FROM items WHERE collection=$1;\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION collection_temporal_extent(id text) RETURNS jsonb AS $$\n SELECT to_jsonb(array[array[min(datetime)::text, max(datetime)::text]])\n FROM items WHERE collection=$1;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection_extents() RETURNS VOID AS $$\nUPDATE collections SET\n content = content ||\n jsonb_build_object(\n 'extent', jsonb_build_object(\n 'spatial', jsonb_build_object(\n 'bbox', collection_bbox(collections.id)\n ),\n 'temporal', jsonb_build_object(\n 'interval', collection_temporal_extent(collections.id)\n )\n )\n )\n;\n$$ LANGUAGE SQL;\nCREATE VIEW partition_steps AS\nSELECT\n name,\n date_trunc('month',lower(datetime_range)) as sdate,\n date_trunc('month', upper(datetime_range)) + '1 month'::interval as edate\n FROM partitions WHERE datetime_range IS NOT NULL AND datetime_range != 'empty'::tstzrange\n ORDER BY datetime_range ASC\n;\n\nCREATE OR REPLACE FUNCTION chunker(\n IN _where text,\n OUT s timestamptz,\n OUT e timestamptz\n) RETURNS SETOF RECORD AS $$\nDECLARE\n explain jsonb;\nBEGIN\n IF _where IS NULL THEN\n _where := ' TRUE ';\n END IF;\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s;', _where)\n INTO explain;\n\n RETURN QUERY\n WITH t AS (\n SELECT j->>0 as p FROM\n jsonb_path_query(\n explain,\n 'strict $.**.\"Relation Name\" ? (@ != null)'\n ) j\n ),\n parts AS (\n SELECT sdate, edate FROM t JOIN partition_steps ON (t.p = name)\n ),\n times AS (\n SELECT sdate FROM parts\n UNION\n SELECT edate FROM parts\n ),\n uniq AS (\n SELECT DISTINCT sdate FROM times ORDER BY sdate\n ),\n last AS (\n SELECT sdate, lead(sdate, 1) over () as edate FROM uniq\n )\n SELECT sdate, edate FROM last WHERE edate IS NOT NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION partition_queries(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN partitions text[] DEFAULT NULL\n) RETURNS SETOF text AS $$\nDECLARE\n query text;\n sdate timestamptz;\n edate timestamptz;\nBEGIN\nIF _where IS NULL OR trim(_where) = '' THEN\n _where = ' TRUE ';\nEND IF;\nRAISE NOTICE 'Getting chunks for % %', _where, _orderby;\nIF _orderby ILIKE 'datetime d%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 DESC LOOP\n RETURN NEXT format($q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n $q$,\n sdate,\n edate,\n _where,\n _orderby\n );\n END LOOP;\nELSIF _orderby ILIKE 'datetime a%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 ASC LOOP\n RETURN NEXT format($q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n $q$,\n sdate,\n edate,\n _where,\n _orderby\n );\n END LOOP;\nELSE\n query := format($q$\n SELECT * FROM items\n WHERE %s\n ORDER BY %s\n $q$, _where, _orderby\n );\n\n RETURN NEXT query;\n RETURN;\nEND IF;\n\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION partition_query_view(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN _limit int DEFAULT 10\n) RETURNS text AS $$\n WITH p AS (\n SELECT * FROM partition_queries(_where, _orderby) p\n )\n SELECT\n CASE WHEN EXISTS (SELECT 1 FROM p) THEN\n (SELECT format($q$\n SELECT * FROM (\n %s\n ) total LIMIT %s\n $q$,\n string_agg(\n format($q$ SELECT * FROM ( %s ) AS sub $q$, p),\n '\n UNION ALL\n '\n ),\n _limit\n ))\n ELSE NULL\n END FROM p;\n$$ LANGUAGE SQL IMMUTABLE;\n\n\n\n\nCREATE OR REPLACE FUNCTION stac_search_to_where(j jsonb) RETURNS text AS $$\nDECLARE\n where_segments text[];\n _where text;\n dtrange tstzrange;\n collections text[];\n geom geometry;\n sdate timestamptz;\n edate timestamptz;\n filterlang text;\n filter jsonb := j->'filter';\nBEGIN\n IF j ? 'ids' THEN\n where_segments := where_segments || format('id = ANY (%L) ', to_text_array(j->'ids'));\n END IF;\n\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n where_segments := where_segments || format('collection = ANY (%L) ', collections);\n END IF;\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n\n where_segments := where_segments || format(' datetime <= %L::timestamptz AND end_datetime >= %L::timestamptz ',\n edate,\n sdate\n );\n END IF;\n\n geom := stac_geom(j);\n IF geom IS NOT NULL THEN\n where_segments := where_segments || format('st_intersects(geometry, %L)',geom);\n END IF;\n\n filterlang := COALESCE(\n j->>'filter-lang',\n get_setting('default_filter_lang', j->'conf')\n );\n IF NOT filter @? '$.**.op' THEN\n filterlang := 'cql-json';\n END IF;\n\n IF filterlang NOT IN ('cql-json','cql2-json') AND j ? 'filter' THEN\n RAISE EXCEPTION '% is not a supported filter-lang. Please use cql-json or cql2-json.', filterlang;\n END IF;\n\n IF j ? 'query' AND j ? 'filter' THEN\n RAISE EXCEPTION 'Can only use either query or filter at one time.';\n END IF;\n\n IF j ? 'query' THEN\n filter := query_to_cql2(j->'query');\n ELSIF filterlang = 'cql-json' THEN\n filter := cql1_to_cql2(filter);\n END IF;\n RAISE NOTICE 'FILTER: %', filter;\n where_segments := where_segments || cql2_query(filter);\n IF cardinality(where_segments) < 1 THEN\n RETURN ' TRUE ';\n END IF;\n\n _where := array_to_string(array_remove(where_segments, NULL), ' AND ');\n\n IF _where IS NULL OR BTRIM(_where) = '' THEN\n RETURN ' TRUE ';\n END IF;\n RETURN _where;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION parse_sort_dir(_dir text, reverse boolean default false) RETURNS text AS $$\n WITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n ) SELECT\n CASE\n WHEN NOT reverse THEN d\n WHEN d = 'ASC' THEN 'DESC'\n WHEN d = 'DESC' THEN 'ASC'\n END\n FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION sort_dir_to_op(_dir text, prev boolean default false) RETURNS text AS $$\n WITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n ) SELECT\n CASE\n WHEN d = 'ASC' AND prev THEN '<='\n WHEN d = 'DESC' AND prev THEN '>='\n WHEN d = 'ASC' THEN '>='\n WHEN d = 'DESC' THEN '<='\n END\n FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION sort_sqlorderby(\n _search jsonb DEFAULT NULL,\n reverse boolean DEFAULT FALSE\n) RETURNS text AS $$\n WITH sortby AS (\n SELECT coalesce(_search->'sortby','[{\"field\":\"datetime\", \"direction\":\"desc\"}]') as sort\n ), withid AS (\n SELECT CASE\n WHEN sort @? '$[*] ? (@.field == \"id\")' THEN sort\n ELSE sort || '[{\"field\":\"id\", \"direction\":\"desc\"}]'::jsonb\n END as sort\n FROM sortby\n ), withid_rows AS (\n SELECT jsonb_array_elements(sort) as value FROM withid\n ),sorts AS (\n SELECT\n coalesce(\n -- field_orderby((items_path(value->>'field')).path_txt),\n (queryable(value->>'field')).expression\n ) as key,\n parse_sort_dir(value->>'direction', reverse) as dir\n FROM withid_rows\n )\n SELECT array_to_string(\n array_agg(concat(key, ' ', dir)),\n ', '\n ) FROM sorts;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_sort_dir(sort_item jsonb) RETURNS text AS $$\n SELECT CASE WHEN sort_item->>'direction' ILIKE 'desc%' THEN 'DESC' ELSE 'ASC' END;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION get_token_filter(_search jsonb = '{}'::jsonb, token_rec jsonb DEFAULT NULL) RETURNS text AS $$\nDECLARE\n token_id text;\n filters text[] := '{}'::text[];\n prev boolean := TRUE;\n field text;\n dir text;\n sort record;\n orfilters text[] := '{}'::text[];\n andfilters text[] := '{}'::text[];\n output text;\n token_where text;\nBEGIN\n RAISE NOTICE 'Getting Token Filter. % %', _search, token_rec;\n -- If no token provided return NULL\n IF token_rec IS NULL THEN\n IF NOT (_search ? 'token' AND\n (\n (_search->>'token' ILIKE 'prev:%')\n OR\n (_search->>'token' ILIKE 'next:%')\n )\n ) THEN\n RETURN NULL;\n END IF;\n prev := (_search->>'token' ILIKE 'prev:%');\n token_id := substr(_search->>'token', 6);\n SELECT to_jsonb(items) INTO token_rec\n FROM items WHERE id=token_id;\n END IF;\n RAISE NOTICE 'TOKEN ID: % %', token_rec, token_rec->'id';\n\n CREATE TEMP TABLE sorts (\n _row int GENERATED ALWAYS AS IDENTITY NOT NULL,\n _field text PRIMARY KEY,\n _dir text NOT NULL,\n _val text\n ) ON COMMIT DROP;\n\n -- Make sure we only have distinct columns to sort with taking the first one we get\n INSERT INTO sorts (_field, _dir)\n SELECT\n (queryable(value->>'field')).expression,\n get_sort_dir(value)\n FROM\n jsonb_array_elements(coalesce(_search->'sortby','[{\"field\":\"datetime\",\"direction\":\"desc\"}]'))\n ON CONFLICT DO NOTHING\n ;\n RAISE NOTICE 'sorts 1: %', (SELECT jsonb_agg(to_json(sorts)) FROM sorts);\n -- Get the first sort direction provided. As the id is a primary key, if there are any\n -- sorts after id they won't do anything, so make sure that id is the last sort item.\n SELECT _dir INTO dir FROM sorts ORDER BY _row ASC LIMIT 1;\n IF EXISTS (SELECT 1 FROM sorts WHERE _field = 'id') THEN\n DELETE FROM sorts WHERE _row > (SELECT _row FROM sorts WHERE _field = 'id' ORDER BY _row ASC);\n ELSE\n INSERT INTO sorts (_field, _dir) VALUES ('id', dir);\n END IF;\n\n -- Add value from looked up item to the sorts table\n UPDATE sorts SET _val=quote_literal(token_rec->>_field);\n\n -- Check if all sorts are the same direction and use row comparison\n -- to filter\n RAISE NOTICE 'sorts 2: %', (SELECT jsonb_agg(to_json(sorts)) FROM sorts);\n\n IF (SELECT count(DISTINCT _dir) FROM sorts) = 1 THEN\n SELECT format(\n '(%s) %s (%s)',\n concat_ws(', ', VARIADIC array_agg(quote_ident(_field))),\n CASE WHEN (prev AND dir = 'ASC') OR (NOT prev AND dir = 'DESC') THEN '<' ELSE '>' END,\n concat_ws(', ', VARIADIC array_agg(_val))\n ) INTO output FROM sorts\n WHERE token_rec ? _field\n ;\n ELSE\n FOR sort IN SELECT * FROM sorts ORDER BY _row asc LOOP\n RAISE NOTICE 'SORT: %', sort;\n IF sort._row = 1 THEN\n orfilters := orfilters || format('(%s %s %s)',\n quote_ident(sort._field),\n CASE WHEN (prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC') THEN '<' ELSE '>' END,\n sort._val\n );\n ELSE\n orfilters := orfilters || format('(%s AND %s %s %s)',\n array_to_string(andfilters, ' AND '),\n quote_ident(sort._field),\n CASE WHEN (prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC') THEN '<' ELSE '>' END,\n sort._val\n );\n\n END IF;\n andfilters := andfilters || format('%s = %s',\n quote_ident(sort._field),\n sort._val\n );\n END LOOP;\n output := array_to_string(orfilters, ' OR ');\n END IF;\n DROP TABLE IF EXISTS sorts;\n token_where := concat('(',coalesce(output,'true'),')');\n IF trim(token_where) = '' THEN\n token_where := NULL;\n END IF;\n RAISE NOTICE 'TOKEN_WHERE: |%|',token_where;\n RETURN token_where;\n END;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION search_tohash(jsonb) RETURNS jsonb AS $$\n SELECT $1 - '{token,limit,context,includes,excludes}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION search_hash(jsonb, jsonb) RETURNS text AS $$\n SELECT md5(concat(search_tohash($1)::text,$2::text));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TABLE IF NOT EXISTS searches(\n hash text GENERATED ALWAYS AS (search_hash(search, metadata)) STORED PRIMARY KEY,\n search jsonb NOT NULL,\n _where text,\n orderby text,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n metadata jsonb DEFAULT '{}'::jsonb NOT NULL\n);\nCREATE TABLE IF NOT EXISTS search_wheres(\n id bigint generated always as identity primary key,\n _where text NOT NULL,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n statslastupdated timestamptz,\n estimated_count bigint,\n estimated_cost float,\n time_to_estimate float,\n total_count bigint,\n time_to_count float,\n partitions text[]\n);\n\nCREATE INDEX IF NOT EXISTS search_wheres_partitions ON search_wheres USING GIN (partitions);\nCREATE UNIQUE INDEX IF NOT EXISTS search_wheres_where ON search_wheres ((md5(_where)));\n\nCREATE OR REPLACE FUNCTION where_stats(inwhere text, updatestats boolean default false, conf jsonb default null) RETURNS search_wheres AS $$\nDECLARE\n t timestamptz;\n i interval;\n explain_json jsonb;\n partitions text[];\n sw search_wheres%ROWTYPE;\n inwhere_hash text := md5(inwhere);\n _context text := lower(context(conf));\n _stats_ttl interval := context_stats_ttl(conf);\n _estimated_cost float := context_estimated_cost(conf);\n _estimated_count int := context_estimated_count(conf);\nBEGIN\n IF _context = 'off' THEN\n sw._where := inwhere;\n return sw;\n END IF;\n\n SELECT * INTO sw FROM search_wheres WHERE md5(_where)=inwhere_hash FOR UPDATE;\n\n -- Update statistics if explicitly set, if statistics do not exist, or statistics ttl has expired\n IF NOT updatestats THEN\n RAISE NOTICE 'Checking if update is needed for: % .', inwhere;\n RAISE NOTICE 'Stats Last Updated: %', sw.statslastupdated;\n RAISE NOTICE 'TTL: %, Age: %', _stats_ttl, now() - sw.statslastupdated;\n RAISE NOTICE 'Context: %, Existing Total: %', _context, sw.total_count;\n IF\n sw.statslastupdated IS NULL\n OR (now() - sw.statslastupdated) > _stats_ttl\n OR (context(conf) != 'off' AND sw.total_count IS NULL)\n THEN\n updatestats := TRUE;\n END IF;\n END IF;\n\n sw._where := inwhere;\n sw.lastused := now();\n sw.usecount := coalesce(sw.usecount,0) + 1;\n\n IF NOT updatestats THEN\n UPDATE search_wheres SET\n lastused = sw.lastused,\n usecount = sw.usecount\n WHERE md5(_where) = inwhere_hash\n RETURNING * INTO sw\n ;\n RETURN sw;\n END IF;\n\n -- Use explain to get estimated count/cost and a list of the partitions that would be hit by the query\n t := clock_timestamp();\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s', inwhere)\n INTO explain_json;\n RAISE NOTICE 'Time for just the explain: %', clock_timestamp() - t;\n i := clock_timestamp() - t;\n\n sw.statslastupdated := now();\n sw.estimated_count := explain_json->0->'Plan'->'Plan Rows';\n sw.estimated_cost := explain_json->0->'Plan'->'Total Cost';\n sw.time_to_estimate := extract(epoch from i);\n\n RAISE NOTICE 'ESTIMATED_COUNT: % < %', sw.estimated_count, _estimated_count;\n RAISE NOTICE 'ESTIMATED_COST: % < %', sw.estimated_cost, _estimated_cost;\n\n -- Do a full count of rows if context is set to on or if auto is set and estimates are low enough\n IF\n _context = 'on'\n OR\n ( _context = 'auto' AND\n (\n sw.estimated_count < _estimated_count\n AND\n sw.estimated_cost < _estimated_cost\n )\n )\n THEN\n t := clock_timestamp();\n RAISE NOTICE 'Calculating actual count...';\n EXECUTE format(\n 'SELECT count(*) FROM items WHERE %s',\n inwhere\n ) INTO sw.total_count;\n i := clock_timestamp() - t;\n RAISE NOTICE 'Actual Count: % -- %', sw.total_count, i;\n sw.time_to_count := extract(epoch FROM i);\n ELSE\n sw.total_count := NULL;\n sw.time_to_count := NULL;\n END IF;\n\n\n INSERT INTO search_wheres\n (_where, lastused, usecount, statslastupdated, estimated_count, estimated_cost, time_to_estimate, partitions, total_count, time_to_count)\n SELECT sw._where, sw.lastused, sw.usecount, sw.statslastupdated, sw.estimated_count, sw.estimated_cost, sw.time_to_estimate, sw.partitions, sw.total_count, sw.time_to_count\n ON CONFLICT ((md5(_where)))\n DO UPDATE\n SET\n lastused = sw.lastused,\n usecount = sw.usecount,\n statslastupdated = sw.statslastupdated,\n estimated_count = sw.estimated_count,\n estimated_cost = sw.estimated_cost,\n time_to_estimate = sw.time_to_estimate,\n total_count = sw.total_count,\n time_to_count = sw.time_to_count\n ;\n RETURN sw;\nEND;\n$$ LANGUAGE PLPGSQL ;\n\n\n\nDROP FUNCTION IF EXISTS search_query;\nCREATE OR REPLACE FUNCTION search_query(\n _search jsonb = '{}'::jsonb,\n updatestats boolean = false,\n _metadata jsonb = '{}'::jsonb\n) RETURNS searches AS $$\nDECLARE\n search searches%ROWTYPE;\n pexplain jsonb;\n t timestamptz;\n i interval;\nBEGIN\n SELECT * INTO search FROM searches\n WHERE hash=search_hash(_search, _metadata) FOR UPDATE;\n\n -- Calculate the where clause if not already calculated\n IF search._where IS NULL THEN\n search._where := stac_search_to_where(_search);\n END IF;\n\n -- Calculate the order by clause if not already calculated\n IF search.orderby IS NULL THEN\n search.orderby := sort_sqlorderby(_search);\n END IF;\n\n PERFORM where_stats(search._where, updatestats, _search->'conf');\n\n search.lastused := now();\n search.usecount := coalesce(search.usecount, 0) + 1;\n INSERT INTO searches (search, _where, orderby, lastused, usecount, metadata)\n VALUES (_search, search._where, search.orderby, search.lastused, search.usecount, _metadata)\n ON CONFLICT (hash) DO\n UPDATE SET\n _where = EXCLUDED._where,\n orderby = EXCLUDED.orderby,\n lastused = EXCLUDED.lastused,\n usecount = EXCLUDED.usecount,\n metadata = EXCLUDED.metadata\n RETURNING * INTO search\n ;\n RETURN search;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION search(_search jsonb = '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n searches searches%ROWTYPE;\n _where text;\n token_where text;\n full_where text;\n orderby text;\n query text;\n token_type text := substr(_search->>'token',1,4);\n _limit int := coalesce((_search->>'limit')::int, 10);\n curs refcursor;\n cntr int := 0;\n iter_record items%ROWTYPE;\n first_record jsonb;\n first_item items%ROWTYPE;\n last_item items%ROWTYPE;\n last_record jsonb;\n out_records jsonb := '[]'::jsonb;\n prev_query text;\n next text;\n prev_id text;\n has_next boolean := false;\n has_prev boolean := false;\n prev text;\n total_count bigint;\n context jsonb;\n collection jsonb;\n includes text[];\n excludes text[];\n exit_flag boolean := FALSE;\n batches int := 0;\n timer timestamptz := clock_timestamp();\n pstart timestamptz;\n pend timestamptz;\n pcurs refcursor;\n search_where search_wheres%ROWTYPE;\n id text;\nBEGIN\nCREATE TEMP TABLE results (content jsonb) ON COMMIT DROP;\n-- if ids is set, short circuit and just use direct ids query for each id\n-- skip any paging or caching\n-- hard codes ordering in the same order as the array of ids\nIF _search ? 'ids' THEN\n INSERT INTO results\n SELECT\n CASE WHEN _search->'conf'->>'nohydrate' IS NOT NULL AND (_search->'conf'->>'nohydrate')::boolean = true THEN\n content_nonhydrated(items, _search->'fields')\n ELSE\n content_hydrate(items, _search->'fields')\n END\n FROM items WHERE\n items.id = ANY(to_text_array(_search->'ids'))\n AND\n CASE WHEN _search ? 'collections' THEN\n items.collection = ANY(to_text_array(_search->'collections'))\n ELSE TRUE\n END\n ORDER BY items.datetime desc, items.id desc\n ;\n SELECT INTO total_count count(*) FROM results;\nELSE\n searches := search_query(_search);\n _where := searches._where;\n orderby := searches.orderby;\n search_where := where_stats(_where);\n total_count := coalesce(search_where.total_count, search_where.estimated_count);\n\n IF token_type='prev' THEN\n token_where := get_token_filter(_search, null::jsonb);\n orderby := sort_sqlorderby(_search, TRUE);\n END IF;\n IF token_type='next' THEN\n token_where := get_token_filter(_search, null::jsonb);\n END IF;\n\n full_where := concat_ws(' AND ', _where, token_where);\n RAISE NOTICE 'FULL QUERY % %', full_where, clock_timestamp()-timer;\n timer := clock_timestamp();\n\n FOR query IN SELECT partition_queries(full_where, orderby, search_where.partitions) LOOP\n timer := clock_timestamp();\n query := format('%s LIMIT %s', query, _limit + 1);\n RAISE NOTICE 'Partition Query: %', query;\n batches := batches + 1;\n -- curs = create_cursor(query);\n RAISE NOTICE 'cursor_tuple_fraction: %', current_setting('cursor_tuple_fraction');\n OPEN curs FOR EXECUTE query;\n LOOP\n FETCH curs into iter_record;\n EXIT WHEN NOT FOUND;\n cntr := cntr + 1;\n\n IF _search->'conf'->>'nohydrate' IS NOT NULL AND (_search->'conf'->>'nohydrate')::boolean = true THEN\n last_record := content_nonhydrated(iter_record, _search->'fields');\n ELSE\n last_record := content_hydrate(iter_record, _search->'fields');\n END IF;\n last_item := iter_record;\n IF cntr = 1 THEN\n first_item := last_item;\n first_record := last_record;\n END IF;\n IF cntr <= _limit THEN\n INSERT INTO results (content) VALUES (last_record);\n ELSIF cntr > _limit THEN\n has_next := true;\n exit_flag := true;\n EXIT;\n END IF;\n END LOOP;\n CLOSE curs;\n RAISE NOTICE 'Query took %.', clock_timestamp()-timer;\n timer := clock_timestamp();\n EXIT WHEN exit_flag;\n END LOOP;\n RAISE NOTICE 'Scanned through % partitions.', batches;\nEND IF;\n\nSELECT jsonb_agg(content) INTO out_records FROM results WHERE content is not NULL;\n\nDROP TABLE results;\n\n\n-- Flip things around if this was the result of a prev token query\nIF token_type='prev' THEN\n out_records := flip_jsonb_array(out_records);\n first_item := last_item;\n first_record := last_record;\nEND IF;\n\n-- If this query has a token, see if there is data before the first record\nIF _search ? 'token' THEN\n prev_query := format(\n 'SELECT 1 FROM items WHERE %s LIMIT 1',\n concat_ws(\n ' AND ',\n _where,\n trim(get_token_filter(_search, to_jsonb(first_item)))\n )\n );\n RAISE NOTICE 'Query to get previous record: % --- %', prev_query, first_record;\n EXECUTE prev_query INTO has_prev;\n IF FOUND and has_prev IS NOT NULL THEN\n RAISE NOTICE 'Query results from prev query: %', has_prev;\n has_prev := TRUE;\n END IF;\nEND IF;\nhas_prev := COALESCE(has_prev, FALSE);\n\nIF has_prev THEN\n prev := out_records->0->>'id';\nEND IF;\nIF has_next OR token_type='prev' THEN\n next := out_records->-1->>'id';\nEND IF;\n\nIF context(_search->'conf') != 'off' THEN\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'matched', total_count,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\nELSE\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\nEND IF;\n\ncollection := jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb),\n 'next', next,\n 'prev', prev,\n 'context', context\n);\n\nRETURN collection;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER SET SEARCH_PATH TO pgstac, public SET cursor_tuple_fraction TO 1;\n\n\nCREATE OR REPLACE FUNCTION search_cursor(_search jsonb = '{}'::jsonb) RETURNS refcursor AS $$\nDECLARE\n curs refcursor;\n searches searches%ROWTYPE;\n _where text;\n _orderby text;\n q text;\n\nBEGIN\n searches := search_query(_search);\n _where := searches._where;\n _orderby := searches.orderby;\n\n OPEN curs FOR\n WITH p AS (\n SELECT * FROM partition_queries(_where, _orderby) p\n )\n SELECT\n CASE WHEN EXISTS (SELECT 1 FROM p) THEN\n (SELECT format($q$\n SELECT * FROM (\n %s\n ) total\n $q$,\n string_agg(\n format($q$ SELECT * FROM ( %s ) AS sub $q$, p),\n '\n UNION ALL\n '\n )\n ))\n ELSE NULL\n END FROM p;\n RETURN curs;\nEND;\n$$ LANGUAGE PLPGSQL;\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION tileenvelope(zoom int, x int, y int) RETURNS geometry AS $$\nWITH t AS (\n SELECT\n 20037508.3427892 as merc_max,\n -20037508.3427892 as merc_min,\n (2 * 20037508.3427892) / (2 ^ zoom) as tile_size\n)\nSELECT st_makeenvelope(\n merc_min + (tile_size * x),\n merc_max - (tile_size * (y + 1)),\n merc_min + (tile_size * (x + 1)),\n merc_max - (tile_size * y),\n 3857\n) FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;DROP FUNCTION IF EXISTS mercgrid;\n\n\nCREATE OR REPLACE FUNCTION ftime() RETURNS interval as $$\nSELECT age(clock_timestamp(), transaction_timestamp());\n$$ LANGUAGE SQL;\nSET SEARCH_PATH to pgstac, public;\n\nDROP FUNCTION IF EXISTS geometrysearch;\nCREATE OR REPLACE FUNCTION geometrysearch(\n IN geom geometry,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE, -- Return as soon as the passed in geometry is full covered\n IN skipcovered boolean DEFAULT TRUE -- Skip any items that would show up completely under the previous items\n) RETURNS jsonb AS $$\nDECLARE\n search searches%ROWTYPE;\n curs refcursor;\n _where text;\n query text;\n iter_record items%ROWTYPE;\n out_records jsonb := '{}'::jsonb[];\n exit_flag boolean := FALSE;\n counter int := 1;\n scancounter int := 1;\n remaining_limit int := _scanlimit;\n tilearea float;\n unionedgeom geometry;\n clippedgeom geometry;\n unionedgeom_area float := 0;\n prev_area float := 0;\n excludes text[];\n includes text[];\n\nBEGIN\n DROP TABLE IF EXISTS pgstac_results;\n CREATE TEMP TABLE pgstac_results (content jsonb) ON COMMIT DROP;\n\n -- If skipcovered is true then you will always want to exit when the passed in geometry is full\n IF skipcovered THEN\n exitwhenfull := TRUE;\n END IF;\n\n SELECT * INTO search FROM searches WHERE hash=queryhash;\n\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Search with Query Hash % Not Found', queryhash;\n END IF;\n\n tilearea := st_area(geom);\n _where := format('%s AND st_intersects(geometry, %L::geometry)', search._where, geom);\n\n\n FOR query IN SELECT * FROM partition_queries(_where, search.orderby) LOOP\n query := format('%s LIMIT %L', query, remaining_limit);\n RAISE NOTICE '%', query;\n OPEN curs FOR EXECUTE query;\n LOOP\n FETCH curs INTO iter_record;\n EXIT WHEN NOT FOUND;\n IF exitwhenfull OR skipcovered THEN -- If we are not using exitwhenfull or skipcovered, we do not need to do expensive geometry operations\n clippedgeom := st_intersection(geom, iter_record.geometry);\n\n IF unionedgeom IS NULL THEN\n unionedgeom := clippedgeom;\n ELSE\n unionedgeom := st_union(unionedgeom, clippedgeom);\n END IF;\n\n unionedgeom_area := st_area(unionedgeom);\n\n IF skipcovered AND prev_area = unionedgeom_area THEN\n scancounter := scancounter + 1;\n CONTINUE;\n END IF;\n\n prev_area := unionedgeom_area;\n\n RAISE NOTICE '% % % %', unionedgeom_area/tilearea, counter, scancounter, ftime();\n END IF;\n RAISE NOTICE '% %', iter_record, content_hydrate(iter_record, fields);\n INSERT INTO pgstac_results (content) VALUES (content_hydrate(iter_record, fields));\n\n IF counter >= _limit\n OR scancounter > _scanlimit\n OR ftime() > _timelimit\n OR (exitwhenfull AND unionedgeom_area >= tilearea)\n THEN\n exit_flag := TRUE;\n EXIT;\n END IF;\n counter := counter + 1;\n scancounter := scancounter + 1;\n\n END LOOP;\n CLOSE curs;\n EXIT WHEN exit_flag;\n remaining_limit := _scanlimit - scancounter;\n END LOOP;\n\n SELECT jsonb_agg(content) INTO out_records FROM pgstac_results WHERE content IS NOT NULL;\n\n RETURN jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb)\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDROP FUNCTION IF EXISTS geojsonsearch;\nCREATE OR REPLACE FUNCTION geojsonsearch(\n IN geojson jsonb,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_geomfromgeojson(geojson),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS xyzsearch;\nCREATE OR REPLACE FUNCTION xyzsearch(\n IN _x int,\n IN _y int,\n IN _z int,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_transform(tileenvelope(_z, _x, _y), 4326),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nGRANT ALL ON SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac to pgstac_admin;\n\n-- pgstac_read role limited to using function apis\nGRANT EXECUTE ON FUNCTION search TO pgstac_read;\nGRANT EXECUTE ON FUNCTION search_query TO pgstac_read;\nGRANT EXECUTE ON FUNCTION item_by_id TO pgstac_read;\nGRANT EXECUTE ON FUNCTION get_item TO pgstac_read;\n\nGRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_ingest;\nGRANT USAGE ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_ingest;\nSELECT set_version('0.6.11');\n" }, { "path": "src/pgstac/migrations/pgstac.0.6.12-0.6.13.sql", "content": "SET client_min_messages TO WARNING;\nSET SEARCH_PATH to pgstac, public;\nset check_function_bodies = off;\n\nCREATE OR REPLACE FUNCTION pgstac.search(_search jsonb DEFAULT '{}'::jsonb)\n RETURNS jsonb\n LANGUAGE plpgsql\n SECURITY DEFINER\n SET search_path TO 'pgstac', 'public'\n SET cursor_tuple_fraction TO '1'\nAS $function$\nDECLARE\n searches searches%ROWTYPE;\n _where text;\n token_where text;\n full_where text;\n orderby text;\n query text;\n token_type text := substr(_search->>'token',1,4);\n _limit int := coalesce((_search->>'limit')::int, 10);\n curs refcursor;\n cntr int := 0;\n iter_record items%ROWTYPE;\n first_record jsonb;\n first_item items%ROWTYPE;\n last_item items%ROWTYPE;\n last_record jsonb;\n out_records jsonb := '[]'::jsonb;\n prev_query text;\n next text;\n prev_id text;\n has_next boolean := false;\n has_prev boolean := false;\n prev text;\n total_count bigint;\n context jsonb;\n collection jsonb;\n includes text[];\n excludes text[];\n exit_flag boolean := FALSE;\n batches int := 0;\n timer timestamptz := clock_timestamp();\n pstart timestamptz;\n pend timestamptz;\n pcurs refcursor;\n search_where search_wheres%ROWTYPE;\n id text;\nBEGIN\nCREATE TEMP TABLE results (i int GENERATED ALWAYS AS IDENTITY, content jsonb) ON COMMIT DROP;\n-- if ids is set, short circuit and just use direct ids query for each id\n-- skip any paging or caching\n-- hard codes ordering in the same order as the array of ids\nIF _search ? 'ids' THEN\n INSERT INTO results (content)\n SELECT\n CASE WHEN _search->'conf'->>'nohydrate' IS NOT NULL AND (_search->'conf'->>'nohydrate')::boolean = true THEN\n content_nonhydrated(items, _search->'fields')\n ELSE\n content_hydrate(items, _search->'fields')\n END\n FROM items WHERE\n items.id = ANY(to_text_array(_search->'ids'))\n AND\n CASE WHEN _search ? 'collections' THEN\n items.collection = ANY(to_text_array(_search->'collections'))\n ELSE TRUE\n END\n ORDER BY items.datetime desc, items.id desc\n ;\n SELECT INTO total_count count(*) FROM results;\nELSE\n searches := search_query(_search);\n _where := searches._where;\n orderby := searches.orderby;\n search_where := where_stats(_where);\n total_count := coalesce(search_where.total_count, search_where.estimated_count);\n\n IF token_type='prev' THEN\n token_where := get_token_filter(_search, null::jsonb);\n orderby := sort_sqlorderby(_search, TRUE);\n END IF;\n IF token_type='next' THEN\n token_where := get_token_filter(_search, null::jsonb);\n END IF;\n\n full_where := concat_ws(' AND ', _where, token_where);\n RAISE NOTICE 'FULL QUERY % %', full_where, clock_timestamp()-timer;\n timer := clock_timestamp();\n\n FOR query IN SELECT partition_queries(full_where, orderby, search_where.partitions) LOOP\n timer := clock_timestamp();\n query := format('%s LIMIT %s', query, _limit + 1);\n RAISE NOTICE 'Partition Query: %', query;\n batches := batches + 1;\n -- curs = create_cursor(query);\n RAISE NOTICE 'cursor_tuple_fraction: %', current_setting('cursor_tuple_fraction');\n OPEN curs FOR EXECUTE query;\n LOOP\n FETCH curs into iter_record;\n EXIT WHEN NOT FOUND;\n cntr := cntr + 1;\n\n IF _search->'conf'->>'nohydrate' IS NOT NULL AND (_search->'conf'->>'nohydrate')::boolean = true THEN\n last_record := content_nonhydrated(iter_record, _search->'fields');\n ELSE\n last_record := content_hydrate(iter_record, _search->'fields');\n END IF;\n last_item := iter_record;\n IF cntr = 1 THEN\n first_item := last_item;\n first_record := last_record;\n END IF;\n IF cntr <= _limit THEN\n INSERT INTO results (content) VALUES (last_record);\n ELSIF cntr > _limit THEN\n has_next := true;\n exit_flag := true;\n EXIT;\n END IF;\n END LOOP;\n CLOSE curs;\n RAISE NOTICE 'Query took %.', clock_timestamp()-timer;\n timer := clock_timestamp();\n EXIT WHEN exit_flag;\n END LOOP;\n RAISE NOTICE 'Scanned through % partitions.', batches;\nEND IF;\n\nWITH ordered AS (SELECT * FROM results WHERE content IS NOT NULL ORDER BY i)\nSELECT jsonb_agg(content) INTO out_records FROM ordered;\n\nDROP TABLE results;\n\n\n-- Flip things around if this was the result of a prev token query\nIF token_type='prev' THEN\n out_records := flip_jsonb_array(out_records);\n first_item := last_item;\n first_record := last_record;\nEND IF;\n\n-- If this query has a token, see if there is data before the first record\nIF _search ? 'token' THEN\n prev_query := format(\n 'SELECT 1 FROM items WHERE %s LIMIT 1',\n concat_ws(\n ' AND ',\n _where,\n trim(get_token_filter(_search, to_jsonb(first_item)))\n )\n );\n RAISE NOTICE 'Query to get previous record: % --- %', prev_query, first_record;\n EXECUTE prev_query INTO has_prev;\n IF FOUND and has_prev IS NOT NULL THEN\n RAISE NOTICE 'Query results from prev query: %', has_prev;\n has_prev := TRUE;\n END IF;\nEND IF;\nhas_prev := COALESCE(has_prev, FALSE);\n\nIF has_prev THEN\n prev := out_records->0->>'id';\nEND IF;\nIF has_next OR token_type='prev' THEN\n next := out_records->-1->>'id';\nEND IF;\n\nIF context(_search->'conf') != 'off' THEN\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'matched', total_count,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\nELSE\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\nEND IF;\n\ncollection := jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb),\n 'next', next,\n 'prev', prev,\n 'context', context\n);\n\nRETURN collection;\nEND;\n$function$\n;\n\n\n\nSELECT set_version('0.6.13');\n" }, { "path": "src/pgstac/migrations/pgstac.0.6.12.sql", "content": "CREATE EXTENSION IF NOT EXISTS postgis;\nCREATE EXTENSION IF NOT EXISTS btree_gist;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_admin;\n CREATE ROLE pgstac_read;\n CREATE ROLE pgstac_ingest;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nGRANT pgstac_admin TO current_user;\n\nCREATE SCHEMA IF NOT EXISTS pgstac AUTHORIZATION pgstac_admin;\n\nALTER ROLE pgstac_admin SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_read SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_ingest SET SEARCH_PATH TO pgstac, public;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\n\nGRANT pgstac_read TO pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\n\nSET ROLE pgstac_admin;\n\nSET SEARCH_PATH TO pgstac, public;\n\n\nCREATE TABLE IF NOT EXISTS migrations (\n version text PRIMARY KEY,\n datetime timestamptz DEFAULT clock_timestamp() NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION get_version() RETURNS text AS $$\n SELECT version FROM pgstac.migrations ORDER BY datetime DESC, version DESC LIMIT 1;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION set_version(text) RETURNS text AS $$\n INSERT INTO pgstac.migrations (version) VALUES ($1)\n ON CONFLICT DO NOTHING\n RETURNING version;\n$$ LANGUAGE SQL;\n\n\nCREATE TABLE IF NOT EXISTS pgstac_settings (\n name text PRIMARY KEY,\n value text NOT NULL\n);\n\nINSERT INTO pgstac_settings (name, value) VALUES\n ('context', 'off'),\n ('context_estimated_count', '100000'),\n ('context_estimated_cost', '100000'),\n ('context_stats_ttl', '1 day'),\n ('default_filter_lang', 'cql2-json'),\n ('additional_properties', 'true')\nON CONFLICT DO NOTHING\n;\n\n\nCREATE OR REPLACE FUNCTION get_setting(IN _setting text, IN conf jsonb DEFAULT NULL) RETURNS text AS $$\nSELECT COALESCE(\n conf->>_setting,\n current_setting(concat('pgstac.',_setting), TRUE),\n (SELECT value FROM pgstac.pgstac_settings WHERE name=_setting)\n);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context(conf jsonb DEFAULT NULL) RETURNS text AS $$\n SELECT pgstac.get_setting('context', conf);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context_estimated_count(conf jsonb DEFAULT NULL) RETURNS int AS $$\n SELECT pgstac.get_setting('context_estimated_count', conf)::int;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_estimated_cost();\nCREATE OR REPLACE FUNCTION context_estimated_cost(conf jsonb DEFAULT NULL) RETURNS float AS $$\n SELECT pgstac.get_setting('context_estimated_cost', conf)::float;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_stats_ttl();\nCREATE OR REPLACE FUNCTION context_stats_ttl(conf jsonb DEFAULT NULL) RETURNS interval AS $$\n SELECT pgstac.get_setting('context_stats_ttl', conf)::interval;\n$$ LANGUAGE SQL;\n\n\nCREATE OR REPLACE FUNCTION notice(VARIADIC text[]) RETURNS boolean AS $$\nDECLARE\ndebug boolean := current_setting('pgstac.debug', true);\nBEGIN\n IF debug THEN\n RAISE NOTICE 'NOTICE FROM FUNC: % >>>>> %', concat_ws(' | ', $1), clock_timestamp();\n RETURN TRUE;\n END IF;\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION empty_arr(ANYARRAY) RETURNS BOOLEAN AS $$\nSELECT CASE\n WHEN $1 IS NULL THEN TRUE\n WHEN cardinality($1)<1 THEN TRUE\nELSE FALSE\nEND;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_intersection(_a ANYARRAY, _b ANYARRAY) RETURNS ANYARRAY AS $$\n SELECT ARRAY ( SELECT unnest(_a) INTERSECT SELECT UNNEST(_b) );\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION array_map_ident(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_ident(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_map_literal(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_literal(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION array_reverse(anyarray) RETURNS anyarray AS $$\nSELECT ARRAY(\n SELECT $1[i]\n FROM generate_subscripts($1,1) AS s(i)\n ORDER BY i DESC\n);\n$$ LANGUAGE SQL STRICT IMMUTABLE;\n\n\nDROP FUNCTION IF EXISTS check_pgstac_settings;\nCREATE OR REPLACE FUNCTION check_pgstac_settings(_sysmem text DEFAULT NULL) RETURNS VOID AS $$\nDECLARE\n settingval text;\n sysmem bigint := pg_size_bytes(_sysmem);\n effective_cache_size bigint := pg_size_bytes(current_setting('effective_cache_size', TRUE));\n shared_buffers bigint := pg_size_bytes(current_setting('shared_buffers', TRUE));\n work_mem bigint := pg_size_bytes(current_setting('work_mem', TRUE));\n max_connections int := current_setting('max_connections', TRUE);\n maintenance_work_mem bigint := pg_size_bytes(current_setting('maintenance_work_mem', TRUE));\n seq_page_cost float := current_setting('seq_page_cost', TRUE);\n random_page_cost float := current_setting('random_page_cost', TRUE);\n temp_buffers bigint := pg_size_bytes(current_setting('temp_buffers', TRUE));\n r record;\nBEGIN\n IF _sysmem IS NULL THEN\n RAISE NOTICE 'Call function with the size of your system memory `SELECT check_pgstac_settings(''4GB'')` to get pg system setting recommendations.';\n ELSE\n IF effective_cache_size < (sysmem * 0.5) THEN\n RAISE WARNING 'effective_cache_size of % is set low for a system with %. Recomended value between % and %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.5), pg_size_pretty(sysmem * 0.75);\n ELSIF effective_cache_size > (sysmem * 0.75) THEN\n RAISE WARNING 'effective_cache_size of % is set high for a system with %. Recomended value between % and %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.5), pg_size_pretty(sysmem * 0.75);\n ELSE\n RAISE NOTICE 'effective_cache_size of % is set appropriately for a system with %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem);\n END IF;\n\n IF shared_buffers < (sysmem * 0.2) THEN\n RAISE WARNING 'shared_buffers of % is set low for a system with %. Recomended value between % and %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.2), pg_size_pretty(sysmem * 0.3);\n ELSIF shared_buffers > (sysmem * 0.3) THEN\n RAISE WARNING 'shared_buffers of % is set high for a system with %. Recomended value between % and %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.2), pg_size_pretty(sysmem * 0.3);\n ELSE\n RAISE NOTICE 'shared_buffers of % is set appropriately for a system with %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem);\n END IF;\n shared_buffers = sysmem * 0.3;\n IF maintenance_work_mem < (sysmem * 0.2) THEN\n RAISE WARNING 'maintenance_work_mem of % is set low for shared_buffers of %. Recomended value between % and %', pg_size_pretty(maintenance_work_mem), pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers * 0.2), pg_size_pretty(shared_buffers * 0.3);\n ELSIF maintenance_work_mem > (shared_buffers * 0.3) THEN\n RAISE WARNING 'maintenance_work_mem of % is set high for shared_buffers of %. Recomended value between % and %', pg_size_pretty(maintenance_work_mem), pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers * 0.2), pg_size_pretty(shared_buffers * 0.3);\n ELSE\n RAISE NOTICE 'maintenance_work_mem of % is set appropriately for shared_buffers of %', pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers);\n END IF;\n\n IF work_mem * max_connections > shared_buffers THEN\n RAISE WARNING 'work_mem setting of % is set high for % max_connections please reduce work_mem to % or decrease max_connections to %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers/max_connections), floor(shared_buffers/work_mem);\n ELSIF work_mem * max_connections < (shared_buffers * 0.75) THEN\n RAISE WARNING 'work_mem setting of % is set low for % max_connections you may consider raising work_mem to % or increasing max_connections to %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers/max_connections), floor(shared_buffers/work_mem);\n ELSE\n RAISE NOTICE 'work_mem setting of % and max_connections of % are adequate for shared_buffers of %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers);\n END IF;\n\n IF random_page_cost / seq_page_cost != 1.1 THEN\n RAISE WARNING 'random_page_cost (%) /seq_page_cost (%) should be set to 1.1 for SSD. Change random_page_cost to %', random_page_cost, seq_page_cost, 1.1 * seq_page_cost;\n ELSE\n RAISE NOTICE 'random_page_cost and seq_page_cost set appropriately for SSD';\n END IF;\n\n IF temp_buffers < greatest(pg_size_bytes('128MB'),(maintenance_work_mem / 2)) THEN\n RAISE WARNING 'pgstac makes heavy use of temp tables, consider raising temp_buffers from % to %', pg_size_pretty(temp_buffers), greatest('128MB', pg_size_pretty((shared_buffers / 16)));\n END IF;\n END IF;\n\n RAISE NOTICE 'VALUES FOR PGSTAC VARIABLES';\n RAISE NOTICE 'These can be set either as GUC system variables or by setting in the pgstac_settings table.';\n\n FOR r IN SELECT name, get_setting(name) as setting, CASE WHEN current_setting(concat('pgstac.',name), TRUE) IS NOT NULL THEN concat('pgstac.',name, ' GUC') WHEN value IS NOT NULL THEN 'pgstac_settings table' ELSE 'Not Set' END as loc FROM pgstac_settings LOOP\n RAISE NOTICE '% is set to % from the %', r.name, r.setting, r.loc;\n END LOOP;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pg_cron';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider intalling pg_cron which can be used to automate tasks';\n ELSE\n RAISE NOTICE 'pg_cron % is installed', settingval;\n END IF;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pgstattuple';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider installing the pgstattuple extension which can be used to help maintain tables and indexes.';\n ELSE\n RAISE NOTICE 'pgstattuple % is installed', settingval;\n END IF;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pg_stat_statements';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider installing the pg_stat_statements extension which is very helpful for tracking the types of queries on the system';\n ELSE\n RAISE NOTICE 'pgstattuple % is installed', settingval;\n END IF;\n\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac, public SET CLIENT_MIN_MESSAGES TO NOTICE;\nCREATE OR REPLACE FUNCTION to_int(jsonb) RETURNS int AS $$\n SELECT floor(($1->>0)::float)::int;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION to_float(jsonb) RETURNS float AS $$\n SELECT ($1->>0)::float;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION to_tstz(jsonb) RETURNS timestamptz AS $$\n SELECT ($1->>0)::timestamptz;\n$$ LANGUAGE SQL IMMUTABLE STRICT SET TIME ZONE 'UTC';\n\n\nCREATE OR REPLACE FUNCTION to_text(jsonb) RETURNS text AS $$\n SELECT CASE WHEN jsonb_typeof($1) IN ('array','object') THEN $1::text ELSE $1->>0 END;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION to_text_array(jsonb) RETURNS text[] AS $$\n SELECT\n CASE jsonb_typeof($1)\n WHEN 'array' THEN ARRAY(SELECT jsonb_array_elements_text($1))\n ELSE ARRAY[$1->>0]\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION bbox_geom(_bbox jsonb) RETURNS geometry AS $$\nSELECT CASE jsonb_array_length(_bbox)\n WHEN 4 THEN\n ST_SetSRID(ST_MakeEnvelope(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float,\n (_bbox->>3)::float\n ),4326)\n WHEN 6 THEN\n ST_SetSRID(ST_3DMakeBox(\n ST_MakePoint(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float\n ),\n ST_MakePoint(\n (_bbox->>3)::float,\n (_bbox->>4)::float,\n (_bbox->>5)::float\n )\n ),4326)\n ELSE null END;\n;\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION geom_bbox(_geom geometry) RETURNS jsonb AS $$\n SELECT jsonb_build_array(\n st_xmin(_geom),\n st_ymin(_geom),\n st_xmax(_geom),\n st_ymax(_geom)\n );\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION flip_jsonb_array(j jsonb) RETURNS jsonb AS $$\n SELECT jsonb_agg(value) FROM (SELECT value FROM jsonb_array_elements(j) WITH ORDINALITY ORDER BY ordinality DESC) as t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION explode_dotpaths(j jsonb) RETURNS SETOF text[] AS $$\n SELECT string_to_array(p, '.') as e FROM jsonb_array_elements_text(j) p;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION explode_dotpaths_recurse(IN j jsonb) RETURNS SETOF text[] AS $$\n WITH RECURSIVE t AS (\n SELECT e FROM explode_dotpaths(j) e\n UNION ALL\n SELECT e[1:cardinality(e)-1]\n FROM t\n WHERE cardinality(e)>1\n ) SELECT e FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION jsonb_set_nested(j jsonb, path text[], val jsonb) RETURNS jsonb AS $$\nDECLARE\nBEGIN\n IF cardinality(path) > 1 THEN\n FOR i IN 1..(cardinality(path)-1) LOOP\n IF j #> path[:i] IS NULL THEN\n j := jsonb_set_lax(j, path[:i], '{}', TRUE);\n END IF;\n END LOOP;\n END IF;\n RETURN jsonb_set_lax(j, path, val, true);\n\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\n\n\nCREATE OR REPLACE FUNCTION jsonb_include(j jsonb, f jsonb) RETURNS jsonb AS $$\nDECLARE\n includes jsonb := f-> 'include';\n outj jsonb := '{}'::jsonb;\n path text[];\nBEGIN\n IF\n includes IS NULL\n OR jsonb_array_length(includes) = 0\n THEN\n RETURN j;\n ELSE\n includes := includes || '[\"id\",\"collection\"]'::jsonb;\n FOR path IN SELECT explode_dotpaths(includes) LOOP\n outj := jsonb_set_nested(outj, path, j #> path);\n END LOOP;\n END IF;\n RETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_exclude(j jsonb, f jsonb) RETURNS jsonb AS $$\nDECLARE\n excludes jsonb := f-> 'exclude';\n outj jsonb := j;\n path text[];\nBEGIN\n IF\n excludes IS NULL\n OR jsonb_array_length(excludes) = 0\n THEN\n RETURN j;\n ELSE\n FOR path IN SELECT explode_dotpaths(excludes) LOOP\n outj := outj #- path;\n END LOOP;\n END IF;\n RETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_fields(j jsonb, f jsonb DEFAULT '{\"fields\":[]}') RETURNS jsonb AS $$\n SELECT jsonb_exclude(jsonb_include(j, f), f);\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION merge_jsonb(_a jsonb, _b jsonb) RETURNS jsonb AS $$\n SELECT\n CASE\n WHEN _a = '\"𒍟※\"'::jsonb THEN NULL\n WHEN _a IS NULL OR jsonb_typeof(_a) = 'null' THEN _b\n WHEN jsonb_typeof(_a) = 'object' AND jsonb_typeof(_b) = 'object' THEN\n (\n SELECT\n jsonb_strip_nulls(\n jsonb_object_agg(\n key,\n merge_jsonb(a.value, b.value)\n )\n )\n FROM\n jsonb_each(coalesce(_a,'{}'::jsonb)) as a\n FULL JOIN\n jsonb_each(coalesce(_b,'{}'::jsonb)) as b\n USING (key)\n )\n WHEN\n jsonb_typeof(_a) = 'array'\n AND jsonb_typeof(_b) = 'array'\n AND jsonb_array_length(_a) = jsonb_array_length(_b)\n THEN\n (\n SELECT jsonb_agg(m) FROM\n ( SELECT\n merge_jsonb(\n jsonb_array_elements(_a),\n jsonb_array_elements(_b)\n ) as m\n ) as l\n )\n ELSE _a\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION strip_jsonb(_a jsonb, _b jsonb) RETURNS jsonb AS $$\n SELECT\n CASE\n\n WHEN (_a IS NULL OR jsonb_typeof(_a) = 'null') AND _b IS NOT NULL AND jsonb_typeof(_b) != 'null' THEN '\"𒍟※\"'::jsonb\n WHEN _b IS NULL OR jsonb_typeof(_a) = 'null' THEN _a\n WHEN _a = _b AND jsonb_typeof(_a) = 'object' THEN '{}'::jsonb\n WHEN _a = _b THEN NULL\n WHEN jsonb_typeof(_a) = 'object' AND jsonb_typeof(_b) = 'object' THEN\n (\n SELECT\n jsonb_strip_nulls(\n jsonb_object_agg(\n key,\n strip_jsonb(a.value, b.value)\n )\n )\n FROM\n jsonb_each(_a) as a\n FULL JOIN\n jsonb_each(_b) as b\n USING (key)\n )\n WHEN\n jsonb_typeof(_a) = 'array'\n AND jsonb_typeof(_b) = 'array'\n AND jsonb_array_length(_a) = jsonb_array_length(_b)\n THEN\n (\n SELECT jsonb_agg(m) FROM\n ( SELECT\n strip_jsonb(\n jsonb_array_elements(_a),\n jsonb_array_elements(_b)\n ) as m\n ) as l\n )\n ELSE _a\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION nullif_jsonbnullempty(j jsonb) RETURNS jsonb AS $$\n SELECT nullif(nullif(nullif(j,'null'::jsonb),'{}'::jsonb),'[]'::jsonb);\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION jsonb_array_unique(j jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(jsonb_agg(DISTINCT a)) v FROM jsonb_array_elements(j) a;\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_concat_ignorenull(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT coalesce(a,'[]'::jsonb) || coalesce(b,'[]'::jsonb);\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_least(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(least(nullif_jsonbnullempty(a), nullif_jsonbnullempty(b)));\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_greatest(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(greatest(a, b));\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION first_notnull_sfunc(anyelement, anyelement) RETURNS anyelement AS $$\n SELECT COALESCE($1,$2);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE AGGREGATE first_notnull(anyelement)(\n SFUNC = first_notnull_sfunc,\n STYPE = anyelement\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_array_unique_merge(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_concat_ignorenull,\n FINALFUNC = jsonb_array_unique\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_min(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_least\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_max(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_greatest\n);\n/* looks for a geometry in a stac item first from geometry and falling back to bbox */\nCREATE OR REPLACE FUNCTION stac_geom(value jsonb) RETURNS geometry AS $$\nSELECT\n CASE\n WHEN value ? 'intersects' THEN\n ST_GeomFromGeoJSON(value->>'intersects')\n WHEN value ? 'geometry' THEN\n ST_GeomFromGeoJSON(value->>'geometry')\n WHEN value ? 'bbox' THEN\n pgstac.bbox_geom(value->'bbox')\n ELSE NULL\n END as geometry\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION stac_daterange(\n value jsonb\n) RETURNS tstzrange AS $$\nDECLARE\n props jsonb := value;\n dt timestamptz;\n edt timestamptz;\nBEGIN\n IF props ? 'properties' THEN\n props := props->'properties';\n END IF;\n IF\n props ? 'start_datetime'\n AND props->>'start_datetime' IS NOT NULL\n AND props ? 'end_datetime'\n AND props->>'end_datetime' IS NOT NULL\n THEN\n dt := props->>'start_datetime';\n edt := props->>'end_datetime';\n IF dt > edt THEN\n RAISE EXCEPTION 'start_datetime must be < end_datetime';\n END IF;\n ELSE\n dt := props->>'datetime';\n edt := props->>'datetime';\n END IF;\n IF dt is NULL OR edt IS NULL THEN\n RAISE NOTICE 'DT: %, EDT: %', dt, edt;\n RAISE EXCEPTION 'Either datetime (%) or both start_datetime (%) and end_datetime (%) must be set.', props->>'datetime',props->>'start_datetime',props->>'end_datetime';\n END IF;\n RETURN tstzrange(dt, edt, '[]');\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_datetime(value jsonb) RETURNS timestamptz AS $$\n SELECT lower(stac_daterange(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_end_datetime(value jsonb) RETURNS timestamptz AS $$\n SELECT upper(stac_daterange(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE TABLE stac_extensions(\n url text PRIMARY KEY,\n content jsonb\n);\n\n\n\nCREATE OR REPLACE FUNCTION collection_base_item(content jsonb) RETURNS jsonb AS $$\n SELECT jsonb_build_object(\n 'type', 'Feature',\n 'stac_version', content->'stac_version',\n 'assets', content->'item_assets',\n 'collection', content->'id'\n );\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TYPE partition_trunc_strategy AS ENUM ('year', 'month');\n\nCREATE TABLE IF NOT EXISTS collections (\n key bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,\n id text GENERATED ALWAYS AS (content->>'id') STORED UNIQUE,\n content JSONB NOT NULL,\n base_item jsonb GENERATED ALWAYS AS (pgstac.collection_base_item(content)) STORED,\n partition_trunc partition_trunc_strategy\n);\n\n\nCREATE OR REPLACE FUNCTION collection_base_item(cid text) RETURNS jsonb AS $$\n SELECT pgstac.collection_base_item(content) FROM pgstac.collections WHERE id = cid LIMIT 1;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION table_empty(text) RETURNS boolean AS $$\nDECLARE\n retval boolean;\nBEGIN\n EXECUTE format($q$\n SELECT NOT EXISTS (SELECT 1 FROM %I LIMIT 1)\n $q$,\n $1\n ) INTO retval;\n RETURN retval;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION collections_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n q text;\n partition_name text := format('_items_%s', NEW.key);\n partition_exists boolean := false;\n partition_empty boolean := true;\n err_context text;\n loadtemp boolean := FALSE;\nBEGIN\n RAISE NOTICE 'Collection Trigger. % %', NEW.id, NEW.key;\n SELECT relid::text INTO partition_name\n FROM pg_partition_tree('items')\n WHERE relid::text = partition_name;\n IF FOUND THEN\n partition_exists := true;\n partition_empty := table_empty(partition_name);\n ELSE\n partition_exists := false;\n partition_empty := true;\n partition_name := format('_items_%s', NEW.key);\n END IF;\n IF TG_OP = 'UPDATE' AND NEW.partition_trunc IS DISTINCT FROM OLD.partition_trunc AND partition_empty THEN\n q := format($q$\n DROP TABLE IF EXISTS %I CASCADE;\n $q$,\n partition_name\n );\n EXECUTE q;\n END IF;\n IF TG_OP = 'UPDATE' AND NEW.partition_trunc IS DISTINCT FROM OLD.partition_trunc AND partition_exists AND NOT partition_empty THEN\n q := format($q$\n CREATE TEMP TABLE changepartitionstaging ON COMMIT DROP AS SELECT * FROM %I;\n DROP TABLE IF EXISTS %I CASCADE;\n $q$,\n partition_name,\n partition_name\n );\n EXECUTE q;\n loadtemp := TRUE;\n partition_empty := TRUE;\n partition_exists := FALSE;\n END IF;\n IF TG_OP = 'UPDATE' AND NEW.partition_trunc IS NOT DISTINCT FROM OLD.partition_trunc THEN\n RETURN NEW;\n END IF;\n IF NEW.partition_trunc IS NULL AND partition_empty THEN\n RAISE NOTICE '% % % %',\n partition_name,\n NEW.id,\n concat(partition_name,'_id_idx'),\n partition_name\n ;\n q := format($q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n $q$,\n partition_name,\n NEW.id,\n concat(partition_name,'_id_idx'),\n partition_name\n );\n RAISE NOTICE 'q: %', q;\n BEGIN\n EXECUTE q;\n EXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', partition_name;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\n END;\n\n ALTER TABLE partitions DISABLE TRIGGER partitions_delete_trigger;\n DELETE FROM partitions WHERE collection=NEW.id AND name=partition_name;\n ALTER TABLE partitions ENABLE TRIGGER partitions_delete_trigger;\n\n INSERT INTO partitions (collection, name) VALUES (NEW.id, partition_name);\n ELSIF partition_empty THEN\n q := format($q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L)\n PARTITION BY RANGE (datetime);\n $q$,\n partition_name,\n NEW.id\n );\n RAISE NOTICE 'q: %', q;\n BEGIN\n EXECUTE q;\n EXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', partition_name;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\n END;\n ALTER TABLE partitions DISABLE TRIGGER partitions_delete_trigger;\n DELETE FROM partitions WHERE collection=NEW.id AND name=partition_name;\n ALTER TABLE partitions ENABLE TRIGGER partitions_delete_trigger;\n ELSE\n RAISE EXCEPTION 'Cannot modify partition % unless empty', partition_name;\n END IF;\n IF loadtemp THEN\n RAISE NOTICE 'Moving data into new partitions.';\n q := format($q$\n WITH p AS (\n SELECT\n collection,\n datetime as datetime,\n end_datetime as end_datetime,\n (partition_name(\n collection,\n datetime\n )).partition_name as name\n FROM changepartitionstaging\n )\n INSERT INTO partitions (collection, datetime_range, end_datetime_range)\n SELECT\n collection,\n tstzrange(min(datetime), max(datetime), '[]') as datetime_range,\n tstzrange(min(end_datetime), max(end_datetime), '[]') as end_datetime_range\n FROM p\n GROUP BY collection, name\n ON CONFLICT (name) DO UPDATE SET\n datetime_range = EXCLUDED.datetime_range,\n end_datetime_range = EXCLUDED.end_datetime_range\n ;\n INSERT INTO %I SELECT * FROM changepartitionstaging;\n DROP TABLE IF EXISTS changepartitionstaging;\n $q$,\n partition_name\n );\n EXECUTE q;\n END IF;\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE TRIGGER collections_trigger AFTER INSERT OR UPDATE ON collections FOR EACH ROW\nEXECUTE FUNCTION collections_trigger_func();\n\nCREATE OR REPLACE FUNCTION partition_collection(collection text, strategy partition_trunc_strategy) RETURNS text AS $$\n UPDATE collections SET partition_trunc=strategy WHERE id=collection RETURNING partition_trunc;\n$$ LANGUAGE SQL;\n\nCREATE TABLE IF NOT EXISTS partitions (\n collection text REFERENCES collections(id) ON DELETE CASCADE,\n name text PRIMARY KEY,\n partition_range tstzrange NOT NULL DEFAULT tstzrange('-infinity'::timestamptz,'infinity'::timestamptz, '[]'),\n datetime_range tstzrange,\n end_datetime_range tstzrange,\n CONSTRAINT prange EXCLUDE USING GIST (\n collection WITH =,\n partition_range WITH &&\n )\n) WITH (FILLFACTOR=90);\nCREATE INDEX partitions_range_idx ON partitions USING GIST(partition_range);\n\nCREATE OR REPLACE FUNCTION partitions_delete_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n q text;\nBEGIN\n RAISE NOTICE 'Partition Delete Trigger. %', OLD.name;\n EXECUTE format($q$\n DROP TABLE IF EXISTS %I CASCADE;\n $q$,\n OLD.name\n );\n RAISE NOTICE 'Dropped partition.';\n RETURN OLD;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER partitions_delete_trigger BEFORE DELETE ON partitions FOR EACH ROW\nEXECUTE FUNCTION partitions_delete_trigger_func();\n\nCREATE OR REPLACE FUNCTION partition_name(\n IN collection text,\n IN dt timestamptz,\n OUT partition_name text,\n OUT partition_range tstzrange\n) AS $$\nDECLARE\n c RECORD;\n parent_name text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n parent_name := format('_items_%s', c.key);\n\n\n IF c.partition_trunc = 'year' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYY'));\n ELSIF c.partition_trunc = 'month' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYYMM'));\n ELSE\n partition_name := parent_name;\n partition_range := tstzrange('-infinity'::timestamptz, 'infinity'::timestamptz, '[]');\n END IF;\n IF partition_range IS NULL THEN\n partition_range := tstzrange(\n date_trunc(c.partition_trunc::text, dt),\n date_trunc(c.partition_trunc::text, dt) + concat('1 ', c.partition_trunc)::interval\n );\n END IF;\n RETURN;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\n\nCREATE OR REPLACE FUNCTION partitions_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n q text;\n cq text;\n parent_name text;\n partition_trunc text;\n partition_name text := NEW.name;\n partition_exists boolean := false;\n partition_empty boolean := true;\n partition_range tstzrange;\n datetime_range tstzrange;\n end_datetime_range tstzrange;\n err_context text;\n mindt timestamptz := lower(NEW.datetime_range);\n maxdt timestamptz := upper(NEW.datetime_range);\n minedt timestamptz := lower(NEW.end_datetime_range);\n maxedt timestamptz := upper(NEW.end_datetime_range);\n t_mindt timestamptz;\n t_maxdt timestamptz;\n t_minedt timestamptz;\n t_maxedt timestamptz;\nBEGIN\n RAISE NOTICE 'Partitions Trigger. %', NEW;\n datetime_range := NEW.datetime_range;\n end_datetime_range := NEW.end_datetime_range;\n\n SELECT\n format('_items_%s', key),\n c.partition_trunc::text\n INTO\n parent_name,\n partition_trunc\n FROM pgstac.collections c\n WHERE c.id = NEW.collection;\n SELECT (pgstac.partition_name(NEW.collection, mindt)).* INTO partition_name, partition_range;\n NEW.name := partition_name;\n\n IF partition_range IS NULL OR partition_range = 'empty'::tstzrange THEN\n partition_range := tstzrange('-infinity'::timestamptz, 'infinity'::timestamptz, '[]');\n END IF;\n\n NEW.partition_range := partition_range;\n IF TG_OP = 'UPDATE' THEN\n mindt := least(mindt, lower(OLD.datetime_range));\n maxdt := greatest(maxdt, upper(OLD.datetime_range));\n minedt := least(minedt, lower(OLD.end_datetime_range));\n maxedt := greatest(maxedt, upper(OLD.end_datetime_range));\n NEW.datetime_range := tstzrange(mindt, maxdt, '[]');\n NEW.end_datetime_range := tstzrange(minedt, maxedt, '[]');\n END IF;\n IF TG_OP = 'INSERT' THEN\n\n IF partition_range != tstzrange('-infinity'::timestamptz, 'infinity'::timestamptz, '[]') THEN\n\n RAISE NOTICE '% % %', partition_name, parent_name, partition_range;\n q := format($q$\n CREATE TABLE IF NOT EXISTS %I partition OF %I FOR VALUES FROM (%L) TO (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n $q$,\n partition_name,\n parent_name,\n lower(partition_range),\n upper(partition_range),\n format('%s_pkey', partition_name),\n partition_name\n );\n BEGIN\n EXECUTE q;\n EXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', partition_name;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\n END;\n END IF;\n\n END IF;\n\n -- Update constraints\n EXECUTE format($q$\n SELECT\n min(datetime),\n max(datetime),\n min(end_datetime),\n max(end_datetime)\n FROM %I;\n $q$, partition_name)\n INTO t_mindt, t_maxdt, t_minedt, t_maxedt;\n mindt := least(mindt, t_mindt);\n maxdt := greatest(maxdt, t_maxdt);\n minedt := least(mindt, minedt, t_minedt);\n maxedt := greatest(maxdt, maxedt, t_maxedt);\n\n mindt := date_trunc(coalesce(partition_trunc, 'year'), mindt);\n maxdt := date_trunc(coalesce(partition_trunc, 'year'), maxdt - '1 second'::interval) + concat('1 ',coalesce(partition_trunc, 'year'))::interval;\n minedt := date_trunc(coalesce(partition_trunc, 'year'), minedt);\n maxedt := date_trunc(coalesce(partition_trunc, 'year'), maxedt - '1 second'::interval) + concat('1 ',coalesce(partition_trunc, 'year'))::interval;\n\n\n IF mindt IS NOT NULL AND maxdt IS NOT NULL AND minedt IS NOT NULL AND maxedt IS NOT NULL THEN\n NEW.datetime_range := tstzrange(mindt, maxdt, '[]');\n NEW.end_datetime_range := tstzrange(minedt, maxedt, '[]');\n IF\n TG_OP='UPDATE'\n AND OLD.datetime_range @> NEW.datetime_range\n AND OLD.end_datetime_range @> NEW.end_datetime_range\n THEN\n RAISE NOTICE 'Range unchanged, not updating constraints.';\n ELSE\n\n RAISE NOTICE '\n SETTING CONSTRAINTS\n mindt: %, maxdt: %\n minedt: %, maxedt: %\n ', mindt, maxdt, minedt, maxedt;\n IF partition_trunc IS NULL THEN\n cq := format($q$\n ALTER TABLE %7$I\n DROP CONSTRAINT IF EXISTS %1$I,\n DROP CONSTRAINT IF EXISTS %2$I,\n ADD CONSTRAINT %1$I\n CHECK (\n (datetime >= %3$L)\n AND (datetime <= %4$L)\n AND (end_datetime >= %5$L)\n AND (end_datetime <= %6$L)\n ) NOT VALID\n ;\n ALTER TABLE %7$I\n VALIDATE CONSTRAINT %1$I;\n $q$,\n format('%s_dt', partition_name),\n format('%s_edt', partition_name),\n mindt,\n maxdt,\n minedt,\n maxedt,\n partition_name\n );\n ELSE\n cq := format($q$\n ALTER TABLE %5$I\n DROP CONSTRAINT IF EXISTS %1$I,\n DROP CONSTRAINT IF EXISTS %2$I,\n ADD CONSTRAINT %2$I\n CHECK ((end_datetime >= %3$L) AND (end_datetime <= %4$L)) NOT VALID\n ;\n ALTER TABLE %5$I\n VALIDATE CONSTRAINT %2$I;\n $q$,\n format('%s_dt', partition_name),\n format('%s_edt', partition_name),\n minedt,\n maxedt,\n partition_name\n );\n\n END IF;\n RAISE NOTICE 'Altering Constraints. %', cq;\n EXECUTE cq;\n END IF;\n ELSE\n NEW.datetime_range = NULL;\n NEW.end_datetime_range = NULL;\n\n cq := format($q$\n ALTER TABLE %3$I\n DROP CONSTRAINT IF EXISTS %1$I,\n DROP CONSTRAINT IF EXISTS %2$I,\n ADD CONSTRAINT %1$I\n CHECK ((datetime IS NULL AND end_datetime IS NULL)) NOT VALID\n ;\n ALTER TABLE %3$I\n VALIDATE CONSTRAINT %1$I;\n $q$,\n format('%s_dt', partition_name),\n format('%s_edt', partition_name),\n partition_name\n );\n EXECUTE cq;\n END IF;\n\n RETURN NEW;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER partitions_trigger BEFORE INSERT OR UPDATE ON partitions FOR EACH ROW\nEXECUTE FUNCTION partitions_trigger_func();\n\n\nCREATE OR REPLACE FUNCTION create_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection(data jsonb) RETURNS VOID AS $$\nDECLARE\n out collections%ROWTYPE;\nBEGIN\n UPDATE collections SET content=data WHERE id = data->>'id' RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ON CONFLICT (id) DO\n UPDATE\n SET content=EXCLUDED.content\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION delete_collection(_id text) RETURNS VOID AS $$\nDECLARE\n out collections%ROWTYPE;\nBEGIN\n DELETE FROM collections WHERE id = _id RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION get_collection(id text) RETURNS jsonb AS $$\n SELECT content FROM collections\n WHERE id=$1\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION all_collections() RETURNS jsonb AS $$\n SELECT jsonb_agg(content) FROM collections;\n;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\nCREATE TABLE queryables (\n id bigint GENERATED ALWAYS AS identity PRIMARY KEY,\n name text NOT NULL,\n collection_ids text[], -- used to determine what partitions to create indexes on\n definition jsonb,\n property_path text,\n property_wrapper text,\n property_index_type text\n);\nCREATE INDEX queryables_name_idx ON queryables (name);\nCREATE INDEX queryables_property_wrapper_idx ON queryables (property_wrapper);\n\n\nINSERT INTO queryables (name, definition) VALUES\n('id', '{\"title\": \"Item ID\",\"description\": \"Item identifier\",\"$ref\": \"https://schemas.stacspec.org/v1.0.0/item-spec/json-schema/item.json#/definitions/core/allOf/2/properties/id\"}'),\n('datetime','{\"description\": \"Datetime\",\"type\": \"string\",\"title\": \"Acquired\",\"format\": \"date-time\",\"pattern\": \"(\\\\+00:00|Z)$\"}'),\n('geometry', '{\"title\": \"Item Geometry\",\"description\": \"Item Geometry\",\"$ref\": \"https://geojson.org/schema/Feature.json\"}')\nON CONFLICT DO NOTHING;\n\nINSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n('eo:cloud_cover','{\"$ref\": \"https://stac-extensions.github.io/eo/v1.0.0/schema.json#/definitions/fieldsproperties/eo:cloud_cover\"}','to_int','BTREE')\nON CONFLICT DO NOTHING;\n\nCREATE OR REPLACE FUNCTION array_to_path(arr text[]) RETURNS text AS $$\n SELECT string_agg(\n quote_literal(v),\n '->'\n ) FROM unnest(arr) v;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\n\n\nCREATE OR REPLACE FUNCTION queryable(\n IN dotpath text,\n OUT path text,\n OUT expression text,\n OUT wrapper text,\n OUT nulled_wrapper text\n) AS $$\nDECLARE\n q RECORD;\n path_elements text[];\nBEGIN\n IF dotpath IN ('id', 'geometry', 'datetime', 'end_datetime', 'collection') THEN\n path := dotpath;\n expression := dotpath;\n wrapper := NULL;\n RETURN;\n END IF;\n SELECT * INTO q FROM queryables\n WHERE\n name=dotpath\n OR name = 'properties.' || dotpath\n OR name = replace(dotpath, 'properties.', '')\n ;\n IF q.property_wrapper IS NULL THEN\n IF q.definition->>'type' = 'number' THEN\n wrapper := 'to_float';\n nulled_wrapper := wrapper;\n ELSIF q.definition->>'format' = 'date-time' THEN\n wrapper := 'to_tstz';\n nulled_wrapper := wrapper;\n ELSE\n nulled_wrapper := NULL;\n wrapper := 'to_text';\n END IF;\n ELSE\n wrapper := q.property_wrapper;\n nulled_wrapper := wrapper;\n END IF;\n IF q.property_path IS NOT NULL THEN\n path := q.property_path;\n ELSE\n path_elements := string_to_array(dotpath, '.');\n IF path_elements[1] IN ('links', 'assets', 'stac_version', 'stac_extensions') THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSIF path_elements[1] = 'properties' THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSE\n path := format($F$content->'properties'->%s$F$, array_to_path(path_elements));\n END IF;\n END IF;\n expression := format('%I(%s)', wrapper, path);\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\nCREATE OR REPLACE FUNCTION create_queryable_indexes() RETURNS VOID AS $$\nDECLARE\n queryable RECORD;\n q text;\nBEGIN\n FOR queryable IN\n SELECT\n queryables.id as qid,\n CASE WHEN collections.key IS NULL THEN 'items' ELSE format('_items_%s',collections.key) END AS part,\n property_index_type,\n expression\n FROM\n queryables\n LEFT JOIN collections ON (collections.id = ANY (queryables.collection_ids))\n JOIN LATERAL queryable(queryables.name) ON (queryables.property_index_type IS NOT NULL)\n LOOP\n q := format(\n $q$\n CREATE INDEX IF NOT EXISTS %I ON %I USING %s ((%s));\n $q$,\n format('%s_%s_idx', queryable.part, queryable.qid),\n queryable.part,\n COALESCE(queryable.property_index_type, 'to_text'),\n queryable.expression\n );\n RAISE NOTICE '%',q;\n EXECUTE q;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION queryables_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\nBEGIN\nPERFORM create_queryable_indexes();\nRETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER queryables_trigger AFTER INSERT OR UPDATE ON queryables\nFOR EACH STATEMENT EXECUTE PROCEDURE queryables_trigger_func();\n\nCREATE TRIGGER queryables_collection_trigger AFTER INSERT OR UPDATE ON collections\nFOR EACH STATEMENT EXECUTE PROCEDURE queryables_trigger_func();\n\n\nCREATE OR REPLACE FUNCTION get_queryables(_collection_ids text[] DEFAULT NULL) RETURNS jsonb AS $$\nDECLARE\nBEGIN\n -- Build up queryables if the input contains valid collection ids or is empty\n IF EXISTS (\n SELECT 1 FROM collections\n WHERE\n _collection_ids IS NULL\n OR cardinality(_collection_ids) = 0\n OR id = ANY(_collection_ids)\n )\n THEN\n RETURN (\n WITH base AS (\n SELECT\n unnest(collection_ids) as collection_id,\n name,\n coalesce(definition, '{\"type\":\"string\"}'::jsonb) as definition\n FROM queryables\n WHERE\n _collection_ids IS NULL OR\n _collection_ids = '{}'::text[] OR\n _collection_ids && collection_ids\n UNION ALL\n SELECT null, name, coalesce(definition, '{\"type\":\"string\"}'::jsonb) as definition\n FROM queryables WHERE collection_ids IS NULL OR collection_ids = '{}'::text[]\n ), g AS (\n SELECT\n name,\n first_notnull(definition) as definition,\n jsonb_array_unique_merge(definition->'enum') as enum,\n jsonb_min(definition->'minimum') as minimum,\n jsonb_min(definition->'maxiumn') as maximum\n FROM base\n GROUP BY 1\n )\n SELECT\n jsonb_build_object(\n '$schema', 'http://json-schema.org/draft-07/schema#',\n '$id', '',\n 'type', 'object',\n 'title', 'STAC Queryables.',\n 'properties', jsonb_object_agg(\n name,\n definition\n ||\n jsonb_strip_nulls(jsonb_build_object(\n 'enum', enum,\n 'minimum', minimum,\n 'maximum', maximum\n ))\n )\n )\n FROM g\n );\n ELSE\n RETURN NULL;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\nCREATE OR REPLACE FUNCTION get_queryables(_collection text DEFAULT NULL) RETURNS jsonb AS $$\n SELECT\n CASE\n WHEN _collection IS NULL THEN get_queryables(NULL::text[])\n ELSE get_queryables(ARRAY[_collection])\n END\n ;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_queryables() RETURNS jsonb AS $$\n SELECT get_queryables(NULL::text[]);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION schema_qualify_refs(url text, j jsonb) returns jsonb as $$\n SELECT regexp_replace(j::text, '\"\\$ref\": \"#', concat('\"$ref\": \"', url, '#'), 'g')::jsonb;\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE VIEW stac_extension_queryables AS\nSELECT DISTINCT key as name, schema_qualify_refs(e.url, j.value) as definition FROM stac_extensions e, jsonb_each(e.content->'definitions'->'fields'->'properties') j;\n\n\nCREATE OR REPLACE FUNCTION missing_queryables(_collection text, _tablesample float DEFAULT 5, minrows float DEFAULT 10) RETURNS TABLE(collection text, name text, definition jsonb, property_wrapper text) AS $$\nDECLARE\n q text;\n _partition text;\n explain_json json;\n psize float;\n estrows float;\nBEGIN\n SELECT format('_items_%s', key) INTO _partition FROM collections WHERE id=_collection;\n\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM %I;', _partition)\n INTO explain_json;\n psize := explain_json->0->'Plan'->'Plan Rows';\n estrows := _tablesample * .01 * psize;\n IF estrows < minrows THEN\n _tablesample := least(100,greatest(_tablesample, (estrows / psize) / 100));\n RAISE NOTICE '%', (psize / estrows) / 100;\n END IF;\n RAISE NOTICE 'Using tablesample % to find missing queryables from % % that has ~% rows estrows: %', _tablesample, _collection, _partition, psize, estrows;\n\n q := format(\n $q$\n WITH q AS (\n SELECT * FROM queryables\n WHERE\n collection_ids IS NULL\n OR %L = ANY(collection_ids)\n ), t AS (\n SELECT\n content->'properties' AS properties\n FROM\n %I\n TABLESAMPLE SYSTEM(%L)\n ), p AS (\n SELECT DISTINCT ON (key)\n key,\n value,\n s.definition\n FROM t\n JOIN LATERAL jsonb_each(properties) ON TRUE\n LEFT JOIN q ON (q.name=key)\n LEFT JOIN stac_extension_queryables s ON (s.name=key)\n WHERE q.definition IS NULL\n )\n SELECT\n %L,\n key,\n COALESCE(definition, jsonb_build_object('type',jsonb_typeof(value))) as definition,\n CASE\n WHEN definition->>'type' = 'integer' THEN 'to_int'\n WHEN COALESCE(definition->>'type', jsonb_typeof(value)) = 'number' THEN 'to_float'\n WHEN COALESCE(definition->>'type', jsonb_typeof(value)) = 'array' THEN 'to_text_array'\n ELSE 'to_text'\n END\n FROM p;\n $q$,\n _collection,\n _partition,\n _tablesample,\n _collection\n );\n RETURN QUERY EXECUTE q;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION missing_queryables(_tablesample float DEFAULT 5) RETURNS TABLE(collection_ids text[], name text, definition jsonb, property_wrapper text) AS $$\n SELECT\n array_agg(collection),\n name,\n definition,\n property_wrapper\n FROM\n collections\n JOIN LATERAL\n missing_queryables(id, _tablesample) c\n ON TRUE\n GROUP BY\n 2,3,4\n ORDER BY 2,1\n ;\n$$ LANGUAGE SQL;\nCREATE OR REPLACE FUNCTION parse_dtrange(\n _indate jsonb,\n relative_base timestamptz DEFAULT date_trunc('hour', CURRENT_TIMESTAMP)\n) RETURNS tstzrange AS $$\nDECLARE\n timestrs text[];\n s timestamptz;\n e timestamptz;\nBEGIN\n timestrs :=\n CASE\n WHEN _indate ? 'timestamp' THEN\n ARRAY[_indate->>'timestamp']\n WHEN _indate ? 'interval' THEN\n to_text_array(_indate->'interval')\n WHEN jsonb_typeof(_indate) = 'array' THEN\n to_text_array(_indate)\n ELSE\n regexp_split_to_array(\n _indate->>0,\n '/'\n )\n END;\n RAISE NOTICE 'TIMESTRS %', timestrs;\n IF cardinality(timestrs) = 1 THEN\n IF timestrs[1] ILIKE 'P%' THEN\n RETURN tstzrange(relative_base - upper(timestrs[1])::interval, relative_base, '[)');\n END IF;\n s := timestrs[1]::timestamptz;\n RETURN tstzrange(s, s, '[]');\n END IF;\n\n IF cardinality(timestrs) != 2 THEN\n RAISE EXCEPTION 'Timestamp cannot have more than 2 values';\n END IF;\n\n IF timestrs[1] = '..' THEN\n s := '-infinity'::timestamptz;\n e := timestrs[2]::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] = '..' THEN\n s := timestrs[1]::timestamptz;\n e := 'infinity'::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[1] ILIKE 'P%' AND timestrs[2] NOT ILIKE 'P%' THEN\n e := timestrs[2]::timestamptz;\n s := e - upper(timestrs[1])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] ILIKE 'P%' AND timestrs[1] NOT ILIKE 'P%' THEN\n s := timestrs[1]::timestamptz;\n e := s + upper(timestrs[2])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n s := timestrs[1]::timestamptz;\n e := timestrs[2]::timestamptz;\n\n RETURN tstzrange(s,e,'[)');\n\n RETURN NULL;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT PARALLEL SAFE SET TIME ZONE 'UTC';\n\nCREATE OR REPLACE FUNCTION parse_dtrange(\n _indate text,\n relative_base timestamptz DEFAULT CURRENT_TIMESTAMP\n) RETURNS tstzrange AS $$\n SELECT parse_dtrange(to_jsonb(_indate), relative_base);\n$$ LANGUAGE SQL STABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION temporal_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\n ll text := 'datetime';\n lh text := 'end_datetime';\n rrange tstzrange;\n rl text;\n rh text;\n outq text;\nBEGIN\n rrange := parse_dtrange(args->1);\n RAISE NOTICE 'Constructing temporal query OP: %, ARGS: %, RRANGE: %', op, args, rrange;\n op := lower(op);\n rl := format('%L::timestamptz', lower(rrange));\n rh := format('%L::timestamptz', upper(rrange));\n outq := CASE op\n WHEN 't_before' THEN 'lh < rl'\n WHEN 't_after' THEN 'll > rh'\n WHEN 't_meets' THEN 'lh = rl'\n WHEN 't_metby' THEN 'll = rh'\n WHEN 't_overlaps' THEN 'll < rl AND rl < lh < rh'\n WHEN 't_overlappedby' THEN 'rl < ll < rh AND lh > rh'\n WHEN 't_starts' THEN 'll = rl AND lh < rh'\n WHEN 't_startedby' THEN 'll = rl AND lh > rh'\n WHEN 't_during' THEN 'll > rl AND lh < rh'\n WHEN 't_contains' THEN 'll < rl AND lh > rh'\n WHEN 't_finishes' THEN 'll > rl AND lh = rh'\n WHEN 't_finishedby' THEN 'll < rl AND lh = rh'\n WHEN 't_equals' THEN 'll = rl AND lh = rh'\n WHEN 't_disjoint' THEN 'NOT (ll <= rh AND lh >= rl)'\n WHEN 't_intersects' THEN 'll <= rh AND lh >= rl'\n WHEN 'anyinteracts' THEN 'll <= rh AND lh >= rl'\n END;\n outq := regexp_replace(outq, '\\mll\\M', ll);\n outq := regexp_replace(outq, '\\mlh\\M', lh);\n outq := regexp_replace(outq, '\\mrl\\M', rl);\n outq := regexp_replace(outq, '\\mrh\\M', rh);\n outq := format('(%s)', outq);\n RETURN outq;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\n\n\nCREATE OR REPLACE FUNCTION spatial_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\n geom text;\n j jsonb := args->1;\nBEGIN\n op := lower(op);\n RAISE NOTICE 'Constructing spatial query OP: %, ARGS: %', op, args;\n IF op NOT IN ('s_equals','s_disjoint','s_touches','s_within','s_overlaps','s_crosses','s_intersects','intersects','s_contains') THEN\n RAISE EXCEPTION 'Spatial Operator % Not Supported', op;\n END IF;\n op := regexp_replace(op, '^s_', 'st_');\n IF op = 'intersects' THEN\n op := 'st_intersects';\n END IF;\n -- Convert geometry to WKB string\n IF j ? 'type' AND j ? 'coordinates' THEN\n geom := st_geomfromgeojson(j)::text;\n ELSIF jsonb_typeof(j) = 'array' THEN\n geom := bbox_geom(j)::text;\n END IF;\n\n RETURN format('%s(geometry, %L::geometry)', op, geom);\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION query_to_cql2(q jsonb) RETURNS jsonb AS $$\n-- Translates anything passed in through the deprecated \"query\" into equivalent CQL2\nWITH t AS (\n SELECT key as property, value as ops\n FROM jsonb_each(q)\n), t2 AS (\n SELECT property, (jsonb_each(ops)).*\n FROM t WHERE jsonb_typeof(ops) = 'object'\n UNION ALL\n SELECT property, 'eq', ops\n FROM t WHERE jsonb_typeof(ops) != 'object'\n)\nSELECT\n jsonb_strip_nulls(jsonb_build_object(\n 'op', 'and',\n 'args', jsonb_agg(\n jsonb_build_object(\n 'op', key,\n 'args', jsonb_build_array(\n jsonb_build_object('property',property),\n value\n )\n )\n )\n )\n) as qcql FROM t2\n;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\nCREATE OR REPLACE FUNCTION cql1_to_cql2(j jsonb) RETURNS jsonb AS $$\nDECLARE\n args jsonb;\n ret jsonb;\nBEGIN\n RAISE NOTICE 'CQL1_TO_CQL2: %', j;\n IF j ? 'filter' THEN\n RETURN cql1_to_cql2(j->'filter');\n END IF;\n IF j ? 'property' THEN\n RETURN j;\n END IF;\n IF jsonb_typeof(j) = 'array' THEN\n SELECT jsonb_agg(cql1_to_cql2(el)) INTO args FROM jsonb_array_elements(j) el;\n RETURN args;\n END IF;\n IF jsonb_typeof(j) = 'number' THEN\n RETURN j;\n END IF;\n IF jsonb_typeof(j) = 'string' THEN\n RETURN j;\n END IF;\n\n IF jsonb_typeof(j) = 'object' THEN\n SELECT jsonb_build_object(\n 'op', key,\n 'args', cql1_to_cql2(value)\n ) INTO ret\n FROM jsonb_each(j)\n WHERE j IS NOT NULL;\n RETURN ret;\n END IF;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE STRICT;\n\nCREATE TABLE cql2_ops (\n op text PRIMARY KEY,\n template text,\n types text[]\n);\nINSERT INTO cql2_ops (op, template, types) VALUES\n ('eq', '%s = %s', NULL),\n ('neq', '%s != %s', NULL),\n ('ne', '%s != %s', NULL),\n ('!=', '%s != %s', NULL),\n ('<>', '%s != %s', NULL),\n ('lt', '%s < %s', NULL),\n ('lte', '%s <= %s', NULL),\n ('gt', '%s > %s', NULL),\n ('gte', '%s >= %s', NULL),\n ('le', '%s <= %s', NULL),\n ('ge', '%s >= %s', NULL),\n ('=', '%s = %s', NULL),\n ('<', '%s < %s', NULL),\n ('<=', '%s <= %s', NULL),\n ('>', '%s > %s', NULL),\n ('>=', '%s >= %s', NULL),\n ('like', '%s LIKE %s', NULL),\n ('ilike', '%s ILIKE %s', NULL),\n ('+', '%s + %s', NULL),\n ('-', '%s - %s', NULL),\n ('*', '%s * %s', NULL),\n ('/', '%s / %s', NULL),\n ('not', 'NOT (%s)', NULL),\n ('between', '%s BETWEEN %s AND %s', NULL),\n ('isnull', '%s IS NULL', NULL),\n ('upper', 'upper(%s)', NULL),\n ('lower', 'lower(%s)', NULL)\nON CONFLICT (op) DO UPDATE\n SET\n template = EXCLUDED.template\n;\n\n\nCREATE OR REPLACE FUNCTION cql2_query(j jsonb, wrapper text DEFAULT NULL) RETURNS text AS $$\n#variable_conflict use_variable\nDECLARE\n args jsonb := j->'args';\n arg jsonb;\n op text := lower(j->>'op');\n cql2op RECORD;\n literal text;\n _wrapper text;\n leftarg text;\n rightarg text;\nBEGIN\n IF j IS NULL OR (op IS NOT NULL AND args IS NULL) THEN\n RETURN NULL;\n END IF;\n RAISE NOTICE 'CQL2_QUERY: %', j;\n IF j ? 'filter' THEN\n RETURN cql2_query(j->'filter');\n END IF;\n\n IF j ? 'upper' THEN\n RETURN cql2_query(jsonb_build_object('op', 'upper', 'args', j->'upper'));\n END IF;\n\n IF j ? 'lower' THEN\n RETURN cql2_query(jsonb_build_object('op', 'lower', 'args', j->'lower'));\n END IF;\n\n -- Temporal Query\n IF op ilike 't_%' or op = 'anyinteracts' THEN\n RETURN temporal_op_query(op, args);\n END IF;\n\n -- If property is a timestamp convert it to text to use with\n -- general operators\n IF j ? 'timestamp' THEN\n RETURN format('%L::timestamptz', to_tstz(j->'timestamp'));\n END IF;\n IF j ? 'interval' THEN\n RAISE EXCEPTION 'Please use temporal operators when using intervals.';\n RETURN NONE;\n END IF;\n\n -- Spatial Query\n IF op ilike 's_%' or op = 'intersects' THEN\n RETURN spatial_op_query(op, args);\n END IF;\n\n IF op IN ('a_equals','a_contains','a_contained_by','a_overlaps') THEN\n IF args->0 ? 'property' THEN\n leftarg := format('to_text_array(%s)', (queryable(args->0->>'property')).path);\n END IF;\n IF args->1 ? 'property' THEN\n rightarg := format('to_text_array(%s)', (queryable(args->1->>'property')).path);\n END IF;\n RETURN FORMAT(\n '%s %s %s',\n COALESCE(leftarg, quote_literal(to_text_array(args->0))),\n CASE op\n WHEN 'a_equals' THEN '='\n WHEN 'a_contains' THEN '@>'\n WHEN 'a_contained_by' THEN '<@'\n WHEN 'a_overlaps' THEN '&&'\n END,\n COALESCE(rightarg, quote_literal(to_text_array(args->1)))\n );\n END IF;\n\n IF op = 'in' THEN\n RAISE NOTICE 'IN : % % %', args, jsonb_build_array(args->0), args->1;\n args := jsonb_build_array(args->0) || (args->1);\n RAISE NOTICE 'IN2 : %', args;\n END IF;\n\n\n\n IF op = 'between' THEN\n args = jsonb_build_array(\n args->0,\n args->1->0,\n args->1->1\n );\n END IF;\n\n -- Make sure that args is an array and run cql2_query on\n -- each element of the array\n RAISE NOTICE 'ARGS PRE: %', args;\n IF j ? 'args' THEN\n IF jsonb_typeof(args) != 'array' THEN\n args := jsonb_build_array(args);\n END IF;\n\n IF jsonb_path_exists(args, '$[*] ? (@.property == \"id\" || @.property == \"datetime\" || @.property == \"end_datetime\" || @.property == \"collection\")') THEN\n wrapper := NULL;\n ELSE\n -- if any of the arguments are a property, try to get the property_wrapper\n FOR arg IN SELECT jsonb_path_query(args, '$[*] ? (@.property != null)') LOOP\n RAISE NOTICE 'Arg: %', arg;\n wrapper := (queryable(arg->>'property')).nulled_wrapper;\n RAISE NOTICE 'Property: %, Wrapper: %', arg, wrapper;\n IF wrapper IS NOT NULL THEN\n EXIT;\n END IF;\n END LOOP;\n\n -- if the property was not in queryables, see if any args were numbers\n IF\n wrapper IS NULL\n AND jsonb_path_exists(args, '$[*] ? (@.type()==\"number\")')\n THEN\n wrapper := 'to_float';\n END IF;\n wrapper := coalesce(wrapper, 'to_text');\n END IF;\n\n SELECT jsonb_agg(cql2_query(a, wrapper))\n INTO args\n FROM jsonb_array_elements(args) a;\n END IF;\n RAISE NOTICE 'ARGS: %', args;\n\n IF op IN ('and', 'or') THEN\n RETURN\n format(\n '(%s)',\n array_to_string(to_text_array(args), format(' %s ', upper(op)))\n );\n END IF;\n\n IF op = 'in' THEN\n RAISE NOTICE 'IN -- % %', args->0, to_text(args->0);\n RETURN format(\n '%s IN (%s)',\n to_text(args->0),\n array_to_string((to_text_array(args))[2:], ',')\n );\n END IF;\n\n -- Look up template from cql2_ops\n IF j ? 'op' THEN\n SELECT * INTO cql2op FROM cql2_ops WHERE cql2_ops.op ilike op;\n IF FOUND THEN\n -- If specific index set in queryables for a property cast other arguments to that type\n\n RETURN format(\n cql2op.template,\n VARIADIC (to_text_array(args))\n );\n ELSE\n RAISE EXCEPTION 'Operator % Not Supported.', op;\n END IF;\n END IF;\n\n\n IF wrapper IS NOT NULL THEN\n RAISE NOTICE 'Wrapping % with %', j, wrapper;\n IF j ? 'property' THEN\n RETURN format('%I(%s)', wrapper, (queryable(j->>'property')).path);\n ELSE\n RETURN format('%I(%L)', wrapper, j);\n END IF;\n ELSIF j ? 'property' THEN\n RETURN quote_ident(j->>'property');\n END IF;\n\n RETURN quote_literal(to_text(j));\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION paging_dtrange(\n j jsonb\n) RETURNS tstzrange AS $$\nDECLARE\n op text;\n filter jsonb := j->'filter';\n dtrange tstzrange := tstzrange('-infinity'::timestamptz,'infinity'::timestamptz);\n sdate timestamptz := '-infinity'::timestamptz;\n edate timestamptz := 'infinity'::timestamptz;\n jpitem jsonb;\nBEGIN\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n END IF;\n IF NOT (filter @? '$.**.op ? (@ == \"or\" || @ == \"not\")') THEN\n FOR jpitem IN SELECT j FROM jsonb_path_query(filter,'strict $.** ? (@.args[*].property == \"datetime\")'::jsonpath) j LOOP\n op := lower(jpitem->>'op');\n dtrange := parse_dtrange(jpitem->'args'->1);\n IF op IN ('<=', 'lt', 'lte', '<', 'le', 't_before') THEN\n sdate := greatest(sdate,'-infinity');\n edate := least(edate, upper(dtrange));\n ELSIF op IN ('>=', '>', 'gt', 'gte', 'ge', 't_after') THEN\n edate := least(edate, 'infinity');\n sdate := greatest(sdate, lower(dtrange));\n ELSIF op IN ('=', 'eq') THEN\n edate := least(edate, upper(dtrange));\n sdate := greatest(sdate, lower(dtrange));\n END IF;\n RAISE NOTICE '2 OP: %, ARGS: %, DTRANGE: %, SDATE: %, EDATE: %', op, jpitem->'args'->1, dtrange, sdate, edate;\n END LOOP;\n END IF;\n IF sdate > edate THEN\n RETURN 'empty'::tstzrange;\n END IF;\n RETURN tstzrange(sdate,edate, '[]');\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT SET TIME ZONE 'UTC';\n\nCREATE OR REPLACE FUNCTION paging_collections(\n IN j jsonb\n) RETURNS text[] AS $$\nDECLARE\n filter jsonb := j->'filter';\n jpitem jsonb;\n op text;\n args jsonb;\n arg jsonb;\n collections text[];\nBEGIN\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n END IF;\n IF NOT (filter @? '$.**.op ? (@ == \"or\" || @ == \"not\")') THEN\n FOR jpitem IN SELECT j FROM jsonb_path_query(filter,'strict $.** ? (@.args[*].property == \"collection\")'::jsonpath) j LOOP\n RAISE NOTICE 'JPITEM: %', jpitem;\n op := jpitem->>'op';\n args := jpitem->'args';\n IF op IN ('=', 'eq', 'in') THEN\n FOR arg IN SELECT a FROM jsonb_array_elements(args) a LOOP\n IF jsonb_typeof(arg) IN ('string', 'array') THEN\n RAISE NOTICE 'arg: %, collections: %', arg, collections;\n IF collections IS NULL OR collections = '{}'::text[] THEN\n collections := to_text_array(arg);\n ELSE\n collections := array_intersection(collections, to_text_array(arg));\n END IF;\n END IF;\n END LOOP;\n END IF;\n END LOOP;\n END IF;\n IF collections = '{}'::text[] THEN\n RETURN NULL;\n END IF;\n RETURN collections;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\nCREATE TABLE items (\n id text NOT NULL,\n geometry geometry NOT NULL,\n collection text NOT NULL,\n datetime timestamptz NOT NULL,\n end_datetime timestamptz NOT NULL,\n content JSONB NOT NULL\n)\nPARTITION BY LIST (collection)\n;\n\nCREATE INDEX \"datetime_idx\" ON items USING BTREE (datetime DESC, end_datetime ASC);\nCREATE INDEX \"geometry_idx\" ON items USING GIST (geometry);\n\nCREATE STATISTICS datetime_stats (dependencies) on datetime, end_datetime from items;\n\n\nALTER TABLE items ADD CONSTRAINT items_collections_fk FOREIGN KEY (collection) REFERENCES collections(id) ON DELETE CASCADE DEFERRABLE;\n\n\nCREATE OR REPLACE FUNCTION content_slim(_item jsonb) RETURNS jsonb AS $$\n SELECT strip_jsonb(_item - '{id,geometry,collection,type}'::text[], collection_base_item(_item->>'collection')) - '{id,geometry,collection,type}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_dehydrate(content jsonb) RETURNS items AS $$\n SELECT\n content->>'id' as id,\n stac_geom(content) as geometry,\n content->>'collection' as collection,\n stac_datetime(content) as datetime,\n stac_end_datetime(content) as end_datetime,\n content_slim(content) as content\n ;\n$$ LANGUAGE SQL STABLE;\n\nCREATE OR REPLACE FUNCTION include_field(f text, fields jsonb DEFAULT '{}'::jsonb) RETURNS boolean AS $$\nDECLARE\n includes jsonb := fields->'include';\n excludes jsonb := fields->'exclude';\nBEGIN\n IF f IS NULL THEN\n RETURN NULL;\n END IF;\n\n\n IF\n jsonb_typeof(excludes) = 'array'\n AND jsonb_array_length(excludes)>0\n AND excludes ? f\n THEN\n RETURN FALSE;\n END IF;\n\n IF\n (\n jsonb_typeof(includes) = 'array'\n AND jsonb_array_length(includes) > 0\n AND includes ? f\n ) OR\n (\n includes IS NULL\n OR jsonb_typeof(includes) = 'null'\n OR jsonb_array_length(includes) = 0\n )\n THEN\n RETURN TRUE;\n END IF;\n\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nDROP FUNCTION IF EXISTS content_hydrate(jsonb, jsonb, jsonb);\nCREATE OR REPLACE FUNCTION content_hydrate(\n _item jsonb,\n _base_item jsonb,\n fields jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\n SELECT merge_jsonb(\n jsonb_fields(_item, fields),\n jsonb_fields(_base_item, fields)\n );\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION content_hydrate(_item items, _collection collections, fields jsonb DEFAULT '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\n content jsonb;\n base_item jsonb := _collection.base_item;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry, 20)::jsonb;\n END IF;\n output := content_hydrate(\n jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'collection', _item.collection,\n 'type', 'Feature'\n ) || _item.content,\n _collection.base_item,\n fields\n );\n\n RETURN output;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_nonhydrated(\n _item items,\n fields jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry, 20)::jsonb;\n END IF;\n output := jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'collection', _item.collection,\n 'type', 'Feature'\n ) || _item.content;\n RETURN output;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_hydrate(_item items, fields jsonb DEFAULT '{}'::jsonb) RETURNS jsonb AS $$\n SELECT content_hydrate(\n _item,\n (SELECT c FROM collections c WHERE id=_item.collection LIMIT 1),\n fields\n );\n$$ LANGUAGE SQL STABLE;\n\n\nCREATE UNLOGGED TABLE items_staging (\n content JSONB NOT NULL\n);\nCREATE UNLOGGED TABLE items_staging_ignore (\n content JSONB NOT NULL\n);\nCREATE UNLOGGED TABLE items_staging_upsert (\n content JSONB NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION items_staging_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p record;\n _partitions text[];\n ts timestamptz := clock_timestamp();\nBEGIN\n RAISE NOTICE 'Creating Partitions. %', clock_timestamp() - ts;\n WITH ranges AS (\n SELECT\n n.content->>'collection' as collection,\n stac_daterange(n.content->'properties') as dtr\n FROM newdata n\n ), p AS (\n SELECT\n collection,\n lower(dtr) as datetime,\n upper(dtr) as end_datetime,\n (partition_name(\n collection,\n lower(dtr)\n )).partition_name as name\n FROM ranges\n )\n INSERT INTO partitions (collection, datetime_range, end_datetime_range)\n SELECT\n collection,\n tstzrange(min(datetime), max(datetime), '[]') as datetime_range,\n tstzrange(min(end_datetime), max(end_datetime), '[]') as end_datetime_range\n FROM p\n GROUP BY collection, name\n ON CONFLICT (name) DO UPDATE SET\n datetime_range = EXCLUDED.datetime_range,\n end_datetime_range = EXCLUDED.end_datetime_range\n ;\n\n RAISE NOTICE 'Doing the insert. %', clock_timestamp() - ts;\n IF TG_TABLE_NAME = 'items_staging' THEN\n INSERT INTO items\n SELECT\n (content_dehydrate(content)).*\n FROM newdata;\n DELETE FROM items_staging;\n ELSIF TG_TABLE_NAME = 'items_staging_ignore' THEN\n INSERT INTO items\n SELECT\n (content_dehydrate(content)).*\n FROM newdata\n ON CONFLICT DO NOTHING;\n DELETE FROM items_staging_ignore;\n ELSIF TG_TABLE_NAME = 'items_staging_upsert' THEN\n WITH staging_formatted AS (\n SELECT (content_dehydrate(content)).* FROM newdata\n ), deletes AS (\n DELETE FROM items i USING staging_formatted s\n WHERE\n i.id = s.id\n AND i.collection = s.collection\n AND i IS DISTINCT FROM s\n RETURNING i.id, i.collection\n )\n INSERT INTO items\n SELECT s.* FROM\n staging_formatted s\n ON CONFLICT DO NOTHING;\n DELETE FROM items_staging_upsert;\n END IF;\n RAISE NOTICE 'Done. %', clock_timestamp() - ts;\n\n RETURN NULL;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER items_staging_insert_trigger AFTER INSERT ON items_staging REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\nCREATE TRIGGER items_staging_insert_ignore_trigger AFTER INSERT ON items_staging_ignore REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\nCREATE TRIGGER items_staging_insert_upsert_trigger AFTER INSERT ON items_staging_upsert REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\n\n\n\nCREATE OR REPLACE FUNCTION item_by_id(_id text, _collection text DEFAULT NULL) RETURNS items AS\n$$\nDECLARE\n i items%ROWTYPE;\nBEGIN\n SELECT * INTO i FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection) LIMIT 1;\n RETURN i;\nEND;\n$$ LANGUAGE PLPGSQL STABLE SECURITY DEFINER SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION get_item(_id text, _collection text DEFAULT NULL) RETURNS jsonb AS $$\n SELECT content_hydrate(items) FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection);\n$$ LANGUAGE SQL STABLE SECURITY DEFINER SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION delete_item(_id text, _collection text DEFAULT NULL) RETURNS VOID AS $$\nDECLARE\nout items%ROWTYPE;\nBEGIN\n DELETE FROM items WHERE id = _id AND (_collection IS NULL OR collection=_collection) RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n--/*\nCREATE OR REPLACE FUNCTION create_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION update_item(content jsonb) RETURNS VOID AS $$\nDECLARE\n old items %ROWTYPE;\n out items%ROWTYPE;\nBEGIN\n PERFORM delete_item(content->>'id', content->>'collection');\n PERFORM create_item(content);\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION create_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION collection_bbox(id text) RETURNS jsonb AS $$\n SELECT (replace(replace(replace(st_extent(geometry)::text,'BOX(','[['),')',']]'),' ',','))::jsonb\n FROM items WHERE collection=$1;\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION collection_temporal_extent(id text) RETURNS jsonb AS $$\n SELECT to_jsonb(array[array[min(datetime)::text, max(datetime)::text]])\n FROM items WHERE collection=$1;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection_extents() RETURNS VOID AS $$\nUPDATE collections SET\n content = content ||\n jsonb_build_object(\n 'extent', jsonb_build_object(\n 'spatial', jsonb_build_object(\n 'bbox', collection_bbox(collections.id)\n ),\n 'temporal', jsonb_build_object(\n 'interval', collection_temporal_extent(collections.id)\n )\n )\n )\n;\n$$ LANGUAGE SQL;\n\n\nCREATE OR REPLACE PROCEDURE analyze_items() AS $$\nDECLARE\nq text;\nBEGIN\nFOR q IN\n SELECT format('ANALYZE (VERBOSE, SKIP_LOCKED) %I;', relname)\n FROM pg_stat_user_tables\n WHERE relname like '_item%' AND (n_mod_since_analyze>0 OR last_analyze IS NULL)\nLOOP\n RAISE NOTICE '%', q;\n EXECUTE q;\n COMMIT;\nEND LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE PROCEDURE validate_constraints() AS $$\nDECLARE\n q text;\nBEGIN\n FOR q IN\n SELECT\n FORMAT(\n 'ALTER TABLE %I.%I VALIDATE CONSTRAINT %I;',\n nsp.nspname,\n cls.relname,\n con.conname\n )\n\n FROM pg_constraint AS con\n JOIN pg_class AS cls\n ON con.conrelid = cls.oid\n JOIN pg_namespace AS nsp\n ON cls.relnamespace = nsp.oid\n WHERE convalidated = FALSE AND contype in ('c','f')\n AND nsp.nspname = 'pgstac'\n LOOP\n RAISE NOTICE '%', q;\n EXECUTE q;\n COMMIT;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\nCREATE VIEW partition_steps AS\nSELECT\n name,\n date_trunc('month',lower(datetime_range)) as sdate,\n date_trunc('month', upper(datetime_range)) + '1 month'::interval as edate\n FROM partitions WHERE datetime_range IS NOT NULL AND datetime_range != 'empty'::tstzrange\n ORDER BY datetime_range ASC\n;\n\nCREATE OR REPLACE FUNCTION chunker(\n IN _where text,\n OUT s timestamptz,\n OUT e timestamptz\n) RETURNS SETOF RECORD AS $$\nDECLARE\n explain jsonb;\nBEGIN\n IF _where IS NULL THEN\n _where := ' TRUE ';\n END IF;\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s;', _where)\n INTO explain;\n\n RETURN QUERY\n WITH t AS (\n SELECT j->>0 as p FROM\n jsonb_path_query(\n explain,\n 'strict $.**.\"Relation Name\" ? (@ != null)'\n ) j\n ),\n parts AS (\n SELECT sdate, edate FROM t JOIN partition_steps ON (t.p = name)\n ),\n times AS (\n SELECT sdate FROM parts\n UNION\n SELECT edate FROM parts\n ),\n uniq AS (\n SELECT DISTINCT sdate FROM times ORDER BY sdate\n ),\n last AS (\n SELECT sdate, lead(sdate, 1) over () as edate FROM uniq\n )\n SELECT sdate, edate FROM last WHERE edate IS NOT NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION partition_queries(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN partitions text[] DEFAULT NULL\n) RETURNS SETOF text AS $$\nDECLARE\n query text;\n sdate timestamptz;\n edate timestamptz;\nBEGIN\nIF _where IS NULL OR trim(_where) = '' THEN\n _where = ' TRUE ';\nEND IF;\nRAISE NOTICE 'Getting chunks for % %', _where, _orderby;\nIF _orderby ILIKE 'datetime d%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 DESC LOOP\n RETURN NEXT format($q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n $q$,\n sdate,\n edate,\n _where,\n _orderby\n );\n END LOOP;\nELSIF _orderby ILIKE 'datetime a%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 ASC LOOP\n RETURN NEXT format($q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n $q$,\n sdate,\n edate,\n _where,\n _orderby\n );\n END LOOP;\nELSE\n query := format($q$\n SELECT * FROM items\n WHERE %s\n ORDER BY %s\n $q$, _where, _orderby\n );\n\n RETURN NEXT query;\n RETURN;\nEND IF;\n\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION partition_query_view(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN _limit int DEFAULT 10\n) RETURNS text AS $$\n WITH p AS (\n SELECT * FROM partition_queries(_where, _orderby) p\n )\n SELECT\n CASE WHEN EXISTS (SELECT 1 FROM p) THEN\n (SELECT format($q$\n SELECT * FROM (\n %s\n ) total LIMIT %s\n $q$,\n string_agg(\n format($q$ SELECT * FROM ( %s ) AS sub $q$, p),\n '\n UNION ALL\n '\n ),\n _limit\n ))\n ELSE NULL\n END FROM p;\n$$ LANGUAGE SQL IMMUTABLE;\n\n\n\n\nCREATE OR REPLACE FUNCTION stac_search_to_where(j jsonb) RETURNS text AS $$\nDECLARE\n where_segments text[];\n _where text;\n dtrange tstzrange;\n collections text[];\n geom geometry;\n sdate timestamptz;\n edate timestamptz;\n filterlang text;\n filter jsonb := j->'filter';\nBEGIN\n IF j ? 'ids' THEN\n where_segments := where_segments || format('id = ANY (%L) ', to_text_array(j->'ids'));\n END IF;\n\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n where_segments := where_segments || format('collection = ANY (%L) ', collections);\n END IF;\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n\n where_segments := where_segments || format(' datetime <= %L::timestamptz AND end_datetime >= %L::timestamptz ',\n edate,\n sdate\n );\n END IF;\n\n geom := stac_geom(j);\n IF geom IS NOT NULL THEN\n where_segments := where_segments || format('st_intersects(geometry, %L)',geom);\n END IF;\n\n filterlang := COALESCE(\n j->>'filter-lang',\n get_setting('default_filter_lang', j->'conf')\n );\n IF NOT filter @? '$.**.op' THEN\n filterlang := 'cql-json';\n END IF;\n\n IF filterlang NOT IN ('cql-json','cql2-json') AND j ? 'filter' THEN\n RAISE EXCEPTION '% is not a supported filter-lang. Please use cql-json or cql2-json.', filterlang;\n END IF;\n\n IF j ? 'query' AND j ? 'filter' THEN\n RAISE EXCEPTION 'Can only use either query or filter at one time.';\n END IF;\n\n IF j ? 'query' THEN\n filter := query_to_cql2(j->'query');\n ELSIF filterlang = 'cql-json' THEN\n filter := cql1_to_cql2(filter);\n END IF;\n RAISE NOTICE 'FILTER: %', filter;\n where_segments := where_segments || cql2_query(filter);\n IF cardinality(where_segments) < 1 THEN\n RETURN ' TRUE ';\n END IF;\n\n _where := array_to_string(array_remove(where_segments, NULL), ' AND ');\n\n IF _where IS NULL OR BTRIM(_where) = '' THEN\n RETURN ' TRUE ';\n END IF;\n RETURN _where;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION parse_sort_dir(_dir text, reverse boolean default false) RETURNS text AS $$\n WITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n ) SELECT\n CASE\n WHEN NOT reverse THEN d\n WHEN d = 'ASC' THEN 'DESC'\n WHEN d = 'DESC' THEN 'ASC'\n END\n FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION sort_dir_to_op(_dir text, prev boolean default false) RETURNS text AS $$\n WITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n ) SELECT\n CASE\n WHEN d = 'ASC' AND prev THEN '<='\n WHEN d = 'DESC' AND prev THEN '>='\n WHEN d = 'ASC' THEN '>='\n WHEN d = 'DESC' THEN '<='\n END\n FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION sort_sqlorderby(\n _search jsonb DEFAULT NULL,\n reverse boolean DEFAULT FALSE\n) RETURNS text AS $$\n WITH sortby AS (\n SELECT coalesce(_search->'sortby','[{\"field\":\"datetime\", \"direction\":\"desc\"}]') as sort\n ), withid AS (\n SELECT CASE\n WHEN sort @? '$[*] ? (@.field == \"id\")' THEN sort\n ELSE sort || '[{\"field\":\"id\", \"direction\":\"desc\"}]'::jsonb\n END as sort\n FROM sortby\n ), withid_rows AS (\n SELECT jsonb_array_elements(sort) as value FROM withid\n ),sorts AS (\n SELECT\n coalesce(\n -- field_orderby((items_path(value->>'field')).path_txt),\n (queryable(value->>'field')).expression\n ) as key,\n parse_sort_dir(value->>'direction', reverse) as dir\n FROM withid_rows\n )\n SELECT array_to_string(\n array_agg(concat(key, ' ', dir)),\n ', '\n ) FROM sorts;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_sort_dir(sort_item jsonb) RETURNS text AS $$\n SELECT CASE WHEN sort_item->>'direction' ILIKE 'desc%' THEN 'DESC' ELSE 'ASC' END;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION get_token_val_str(\n _field text,\n _item items\n) RETURNS text AS $$\nDECLARE\nliteral text;\nBEGIN\nRAISE NOTICE '% %', _field, _item;\nCREATE TEMP TABLE _token_item ON COMMIT DROP AS SELECT (_item).*;\nEXECUTE format($q$ SELECT quote_literal(%s) FROM _token_item $q$, _field) INTO literal;\nDROP TABLE IF EXISTS _token_item;\nRETURN literal;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION get_token_filter(_search jsonb = '{}'::jsonb, token_rec jsonb DEFAULT NULL) RETURNS text AS $$\nDECLARE\n token_id text;\n filters text[] := '{}'::text[];\n prev boolean := TRUE;\n field text;\n dir text;\n sort record;\n orfilters text[] := '{}'::text[];\n andfilters text[] := '{}'::text[];\n output text;\n token_where text;\n token_item items%ROWTYPE;\nBEGIN\n RAISE NOTICE 'Getting Token Filter. % %', _search, token_rec;\n -- If no token provided return NULL\n IF token_rec IS NULL THEN\n IF NOT (_search ? 'token' AND\n (\n (_search->>'token' ILIKE 'prev:%')\n OR\n (_search->>'token' ILIKE 'next:%')\n )\n ) THEN\n RETURN NULL;\n END IF;\n prev := (_search->>'token' ILIKE 'prev:%');\n token_id := substr(_search->>'token', 6);\n SELECT to_jsonb(items) INTO token_rec\n FROM items WHERE id=token_id;\n END IF;\n RAISE NOTICE 'TOKEN ID: % %', token_rec, token_rec->'id';\n\n\n RAISE NOTICE 'TOKEN ID: % %', token_rec, token_rec->'id';\n token_item := jsonb_populate_record(null::items, token_rec);\n RAISE NOTICE 'TOKEN ITEM ----- %', token_item;\n\n\n CREATE TEMP TABLE sorts (\n _row int GENERATED ALWAYS AS IDENTITY NOT NULL,\n _field text PRIMARY KEY,\n _dir text NOT NULL,\n _val text\n ) ON COMMIT DROP;\n\n -- Make sure we only have distinct columns to sort with taking the first one we get\n INSERT INTO sorts (_field, _dir)\n SELECT\n (queryable(value->>'field')).expression,\n get_sort_dir(value)\n FROM\n jsonb_array_elements(coalesce(_search->'sortby','[{\"field\":\"datetime\",\"direction\":\"desc\"}]'))\n ON CONFLICT DO NOTHING\n ;\n RAISE NOTICE 'sorts 1: %', (SELECT jsonb_agg(to_json(sorts)) FROM sorts);\n -- Get the first sort direction provided. As the id is a primary key, if there are any\n -- sorts after id they won't do anything, so make sure that id is the last sort item.\n SELECT _dir INTO dir FROM sorts ORDER BY _row ASC LIMIT 1;\n IF EXISTS (SELECT 1 FROM sorts WHERE _field = 'id') THEN\n DELETE FROM sorts WHERE _row > (SELECT _row FROM sorts WHERE _field = 'id' ORDER BY _row ASC);\n ELSE\n INSERT INTO sorts (_field, _dir) VALUES ('id', dir);\n END IF;\n\n -- Add value from looked up item to the sorts table\n UPDATE sorts SET _val=get_token_val_str(_field, token_item);\n\n -- Check if all sorts are the same direction and use row comparison\n -- to filter\n RAISE NOTICE 'sorts 2: %', (SELECT jsonb_agg(to_json(sorts)) FROM sorts);\n\n FOR sort IN SELECT * FROM sorts ORDER BY _row asc LOOP\n RAISE NOTICE 'SORT: %', sort;\n IF sort._row = 1 THEN\n IF sort._val IS NULL THEN\n orfilters := orfilters || format('(%s IS NOT NULL)', sort._field);\n ELSE\n orfilters := orfilters || format('(%s %s %s)',\n sort._field,\n CASE WHEN (prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC') THEN '<' ELSE '>' END,\n sort._val\n );\n END IF;\n ELSE\n IF sort._val IS NULL THEN\n orfilters := orfilters || format('(%s AND %s IS NOT NULL)',\n array_to_string(andfilters, ' AND '), sort._field);\n ELSE\n orfilters := orfilters || format('(%s AND %s %s %s)',\n array_to_string(andfilters, ' AND '),\n sort._field,\n CASE WHEN (prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC') THEN '<' ELSE '>' END,\n sort._val\n );\n END IF;\n END IF;\n IF sort._val IS NULL THEN\n andfilters := andfilters || format('%s IS NULL',\n sort._field\n );\n ELSE\n andfilters := andfilters || format('%s = %s',\n sort._field,\n sort._val\n );\n END IF;\n END LOOP;\n output := array_to_string(orfilters, ' OR ');\n\n DROP TABLE IF EXISTS sorts;\n token_where := concat('(',coalesce(output,'true'),')');\n IF trim(token_where) = '' THEN\n token_where := NULL;\n END IF;\n RAISE NOTICE 'TOKEN_WHERE: |%|',token_where;\n RETURN token_where;\n END;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION search_tohash(jsonb) RETURNS jsonb AS $$\n SELECT $1 - '{token,limit,context,includes,excludes}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION search_hash(jsonb, jsonb) RETURNS text AS $$\n SELECT md5(concat(search_tohash($1)::text,$2::text));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TABLE IF NOT EXISTS searches(\n hash text GENERATED ALWAYS AS (search_hash(search, metadata)) STORED PRIMARY KEY,\n search jsonb NOT NULL,\n _where text,\n orderby text,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n metadata jsonb DEFAULT '{}'::jsonb NOT NULL\n);\nCREATE TABLE IF NOT EXISTS search_wheres(\n id bigint generated always as identity primary key,\n _where text NOT NULL,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n statslastupdated timestamptz,\n estimated_count bigint,\n estimated_cost float,\n time_to_estimate float,\n total_count bigint,\n time_to_count float,\n partitions text[]\n);\n\nCREATE INDEX IF NOT EXISTS search_wheres_partitions ON search_wheres USING GIN (partitions);\nCREATE UNIQUE INDEX IF NOT EXISTS search_wheres_where ON search_wheres ((md5(_where)));\n\nCREATE OR REPLACE FUNCTION where_stats(inwhere text, updatestats boolean default false, conf jsonb default null) RETURNS search_wheres AS $$\nDECLARE\n t timestamptz;\n i interval;\n explain_json jsonb;\n partitions text[];\n sw search_wheres%ROWTYPE;\n inwhere_hash text := md5(inwhere);\n _context text := lower(context(conf));\n _stats_ttl interval := context_stats_ttl(conf);\n _estimated_cost float := context_estimated_cost(conf);\n _estimated_count int := context_estimated_count(conf);\nBEGIN\n IF _context = 'off' THEN\n sw._where := inwhere;\n return sw;\n END IF;\n\n SELECT * INTO sw FROM search_wheres WHERE md5(_where)=inwhere_hash FOR UPDATE;\n\n -- Update statistics if explicitly set, if statistics do not exist, or statistics ttl has expired\n IF NOT updatestats THEN\n RAISE NOTICE 'Checking if update is needed for: % .', inwhere;\n RAISE NOTICE 'Stats Last Updated: %', sw.statslastupdated;\n RAISE NOTICE 'TTL: %, Age: %', _stats_ttl, now() - sw.statslastupdated;\n RAISE NOTICE 'Context: %, Existing Total: %', _context, sw.total_count;\n IF\n sw.statslastupdated IS NULL\n OR (now() - sw.statslastupdated) > _stats_ttl\n OR (context(conf) != 'off' AND sw.total_count IS NULL)\n THEN\n updatestats := TRUE;\n END IF;\n END IF;\n\n sw._where := inwhere;\n sw.lastused := now();\n sw.usecount := coalesce(sw.usecount,0) + 1;\n\n IF NOT updatestats THEN\n UPDATE search_wheres SET\n lastused = sw.lastused,\n usecount = sw.usecount\n WHERE md5(_where) = inwhere_hash\n RETURNING * INTO sw\n ;\n RETURN sw;\n END IF;\n\n -- Use explain to get estimated count/cost and a list of the partitions that would be hit by the query\n t := clock_timestamp();\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s', inwhere)\n INTO explain_json;\n RAISE NOTICE 'Time for just the explain: %', clock_timestamp() - t;\n i := clock_timestamp() - t;\n\n sw.statslastupdated := now();\n sw.estimated_count := explain_json->0->'Plan'->'Plan Rows';\n sw.estimated_cost := explain_json->0->'Plan'->'Total Cost';\n sw.time_to_estimate := extract(epoch from i);\n\n RAISE NOTICE 'ESTIMATED_COUNT: % < %', sw.estimated_count, _estimated_count;\n RAISE NOTICE 'ESTIMATED_COST: % < %', sw.estimated_cost, _estimated_cost;\n\n -- Do a full count of rows if context is set to on or if auto is set and estimates are low enough\n IF\n _context = 'on'\n OR\n ( _context = 'auto' AND\n (\n sw.estimated_count < _estimated_count\n AND\n sw.estimated_cost < _estimated_cost\n )\n )\n THEN\n t := clock_timestamp();\n RAISE NOTICE 'Calculating actual count...';\n EXECUTE format(\n 'SELECT count(*) FROM items WHERE %s',\n inwhere\n ) INTO sw.total_count;\n i := clock_timestamp() - t;\n RAISE NOTICE 'Actual Count: % -- %', sw.total_count, i;\n sw.time_to_count := extract(epoch FROM i);\n ELSE\n sw.total_count := NULL;\n sw.time_to_count := NULL;\n END IF;\n\n\n INSERT INTO search_wheres\n (_where, lastused, usecount, statslastupdated, estimated_count, estimated_cost, time_to_estimate, partitions, total_count, time_to_count)\n SELECT sw._where, sw.lastused, sw.usecount, sw.statslastupdated, sw.estimated_count, sw.estimated_cost, sw.time_to_estimate, sw.partitions, sw.total_count, sw.time_to_count\n ON CONFLICT ((md5(_where)))\n DO UPDATE\n SET\n lastused = sw.lastused,\n usecount = sw.usecount,\n statslastupdated = sw.statslastupdated,\n estimated_count = sw.estimated_count,\n estimated_cost = sw.estimated_cost,\n time_to_estimate = sw.time_to_estimate,\n total_count = sw.total_count,\n time_to_count = sw.time_to_count\n ;\n RETURN sw;\nEND;\n$$ LANGUAGE PLPGSQL ;\n\n\n\nDROP FUNCTION IF EXISTS search_query;\nCREATE OR REPLACE FUNCTION search_query(\n _search jsonb = '{}'::jsonb,\n updatestats boolean = false,\n _metadata jsonb = '{}'::jsonb\n) RETURNS searches AS $$\nDECLARE\n search searches%ROWTYPE;\n pexplain jsonb;\n t timestamptz;\n i interval;\nBEGIN\n SELECT * INTO search FROM searches\n WHERE hash=search_hash(_search, _metadata) FOR UPDATE;\n\n -- Calculate the where clause if not already calculated\n IF search._where IS NULL THEN\n search._where := stac_search_to_where(_search);\n END IF;\n\n -- Calculate the order by clause if not already calculated\n IF search.orderby IS NULL THEN\n search.orderby := sort_sqlorderby(_search);\n END IF;\n\n PERFORM where_stats(search._where, updatestats, _search->'conf');\n\n search.lastused := now();\n search.usecount := coalesce(search.usecount, 0) + 1;\n INSERT INTO searches (search, _where, orderby, lastused, usecount, metadata)\n VALUES (_search, search._where, search.orderby, search.lastused, search.usecount, _metadata)\n ON CONFLICT (hash) DO\n UPDATE SET\n _where = EXCLUDED._where,\n orderby = EXCLUDED.orderby,\n lastused = EXCLUDED.lastused,\n usecount = EXCLUDED.usecount,\n metadata = EXCLUDED.metadata\n RETURNING * INTO search\n ;\n RETURN search;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION search(_search jsonb = '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n searches searches%ROWTYPE;\n _where text;\n token_where text;\n full_where text;\n orderby text;\n query text;\n token_type text := substr(_search->>'token',1,4);\n _limit int := coalesce((_search->>'limit')::int, 10);\n curs refcursor;\n cntr int := 0;\n iter_record items%ROWTYPE;\n first_record jsonb;\n first_item items%ROWTYPE;\n last_item items%ROWTYPE;\n last_record jsonb;\n out_records jsonb := '[]'::jsonb;\n prev_query text;\n next text;\n prev_id text;\n has_next boolean := false;\n has_prev boolean := false;\n prev text;\n total_count bigint;\n context jsonb;\n collection jsonb;\n includes text[];\n excludes text[];\n exit_flag boolean := FALSE;\n batches int := 0;\n timer timestamptz := clock_timestamp();\n pstart timestamptz;\n pend timestamptz;\n pcurs refcursor;\n search_where search_wheres%ROWTYPE;\n id text;\nBEGIN\nCREATE TEMP TABLE results (content jsonb) ON COMMIT DROP;\n-- if ids is set, short circuit and just use direct ids query for each id\n-- skip any paging or caching\n-- hard codes ordering in the same order as the array of ids\nIF _search ? 'ids' THEN\n INSERT INTO results\n SELECT\n CASE WHEN _search->'conf'->>'nohydrate' IS NOT NULL AND (_search->'conf'->>'nohydrate')::boolean = true THEN\n content_nonhydrated(items, _search->'fields')\n ELSE\n content_hydrate(items, _search->'fields')\n END\n FROM items WHERE\n items.id = ANY(to_text_array(_search->'ids'))\n AND\n CASE WHEN _search ? 'collections' THEN\n items.collection = ANY(to_text_array(_search->'collections'))\n ELSE TRUE\n END\n ORDER BY items.datetime desc, items.id desc\n ;\n SELECT INTO total_count count(*) FROM results;\nELSE\n searches := search_query(_search);\n _where := searches._where;\n orderby := searches.orderby;\n search_where := where_stats(_where);\n total_count := coalesce(search_where.total_count, search_where.estimated_count);\n\n IF token_type='prev' THEN\n token_where := get_token_filter(_search, null::jsonb);\n orderby := sort_sqlorderby(_search, TRUE);\n END IF;\n IF token_type='next' THEN\n token_where := get_token_filter(_search, null::jsonb);\n END IF;\n\n full_where := concat_ws(' AND ', _where, token_where);\n RAISE NOTICE 'FULL QUERY % %', full_where, clock_timestamp()-timer;\n timer := clock_timestamp();\n\n FOR query IN SELECT partition_queries(full_where, orderby, search_where.partitions) LOOP\n timer := clock_timestamp();\n query := format('%s LIMIT %s', query, _limit + 1);\n RAISE NOTICE 'Partition Query: %', query;\n batches := batches + 1;\n -- curs = create_cursor(query);\n RAISE NOTICE 'cursor_tuple_fraction: %', current_setting('cursor_tuple_fraction');\n OPEN curs FOR EXECUTE query;\n LOOP\n FETCH curs into iter_record;\n EXIT WHEN NOT FOUND;\n cntr := cntr + 1;\n\n IF _search->'conf'->>'nohydrate' IS NOT NULL AND (_search->'conf'->>'nohydrate')::boolean = true THEN\n last_record := content_nonhydrated(iter_record, _search->'fields');\n ELSE\n last_record := content_hydrate(iter_record, _search->'fields');\n END IF;\n last_item := iter_record;\n IF cntr = 1 THEN\n first_item := last_item;\n first_record := last_record;\n END IF;\n IF cntr <= _limit THEN\n INSERT INTO results (content) VALUES (last_record);\n ELSIF cntr > _limit THEN\n has_next := true;\n exit_flag := true;\n EXIT;\n END IF;\n END LOOP;\n CLOSE curs;\n RAISE NOTICE 'Query took %.', clock_timestamp()-timer;\n timer := clock_timestamp();\n EXIT WHEN exit_flag;\n END LOOP;\n RAISE NOTICE 'Scanned through % partitions.', batches;\nEND IF;\n\nSELECT jsonb_agg(content) INTO out_records FROM results WHERE content is not NULL;\n\nDROP TABLE results;\n\n\n-- Flip things around if this was the result of a prev token query\nIF token_type='prev' THEN\n out_records := flip_jsonb_array(out_records);\n first_item := last_item;\n first_record := last_record;\nEND IF;\n\n-- If this query has a token, see if there is data before the first record\nIF _search ? 'token' THEN\n prev_query := format(\n 'SELECT 1 FROM items WHERE %s LIMIT 1',\n concat_ws(\n ' AND ',\n _where,\n trim(get_token_filter(_search, to_jsonb(first_item)))\n )\n );\n RAISE NOTICE 'Query to get previous record: % --- %', prev_query, first_record;\n EXECUTE prev_query INTO has_prev;\n IF FOUND and has_prev IS NOT NULL THEN\n RAISE NOTICE 'Query results from prev query: %', has_prev;\n has_prev := TRUE;\n END IF;\nEND IF;\nhas_prev := COALESCE(has_prev, FALSE);\n\nIF has_prev THEN\n prev := out_records->0->>'id';\nEND IF;\nIF has_next OR token_type='prev' THEN\n next := out_records->-1->>'id';\nEND IF;\n\nIF context(_search->'conf') != 'off' THEN\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'matched', total_count,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\nELSE\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\nEND IF;\n\ncollection := jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb),\n 'next', next,\n 'prev', prev,\n 'context', context\n);\n\nRETURN collection;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER SET SEARCH_PATH TO pgstac, public SET cursor_tuple_fraction TO 1;\n\n\nCREATE OR REPLACE FUNCTION search_cursor(_search jsonb = '{}'::jsonb) RETURNS refcursor AS $$\nDECLARE\n curs refcursor;\n searches searches%ROWTYPE;\n _where text;\n _orderby text;\n q text;\n\nBEGIN\n searches := search_query(_search);\n _where := searches._where;\n _orderby := searches.orderby;\n\n OPEN curs FOR\n WITH p AS (\n SELECT * FROM partition_queries(_where, _orderby) p\n )\n SELECT\n CASE WHEN EXISTS (SELECT 1 FROM p) THEN\n (SELECT format($q$\n SELECT * FROM (\n %s\n ) total\n $q$,\n string_agg(\n format($q$ SELECT * FROM ( %s ) AS sub $q$, p),\n '\n UNION ALL\n '\n )\n ))\n ELSE NULL\n END FROM p;\n RETURN curs;\nEND;\n$$ LANGUAGE PLPGSQL;\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION tileenvelope(zoom int, x int, y int) RETURNS geometry AS $$\nWITH t AS (\n SELECT\n 20037508.3427892 as merc_max,\n -20037508.3427892 as merc_min,\n (2 * 20037508.3427892) / (2 ^ zoom) as tile_size\n)\nSELECT st_makeenvelope(\n merc_min + (tile_size * x),\n merc_max - (tile_size * (y + 1)),\n merc_min + (tile_size * (x + 1)),\n merc_max - (tile_size * y),\n 3857\n) FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;DROP FUNCTION IF EXISTS mercgrid;\n\n\nCREATE OR REPLACE FUNCTION ftime() RETURNS interval as $$\nSELECT age(clock_timestamp(), transaction_timestamp());\n$$ LANGUAGE SQL;\nSET SEARCH_PATH to pgstac, public;\n\nDROP FUNCTION IF EXISTS geometrysearch;\nCREATE OR REPLACE FUNCTION geometrysearch(\n IN geom geometry,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE, -- Return as soon as the passed in geometry is full covered\n IN skipcovered boolean DEFAULT TRUE -- Skip any items that would show up completely under the previous items\n) RETURNS jsonb AS $$\nDECLARE\n search searches%ROWTYPE;\n curs refcursor;\n _where text;\n query text;\n iter_record items%ROWTYPE;\n out_records jsonb := '{}'::jsonb[];\n exit_flag boolean := FALSE;\n counter int := 1;\n scancounter int := 1;\n remaining_limit int := _scanlimit;\n tilearea float;\n unionedgeom geometry;\n clippedgeom geometry;\n unionedgeom_area float := 0;\n prev_area float := 0;\n excludes text[];\n includes text[];\n\nBEGIN\n DROP TABLE IF EXISTS pgstac_results;\n CREATE TEMP TABLE pgstac_results (content jsonb) ON COMMIT DROP;\n\n -- If skipcovered is true then you will always want to exit when the passed in geometry is full\n IF skipcovered THEN\n exitwhenfull := TRUE;\n END IF;\n\n SELECT * INTO search FROM searches WHERE hash=queryhash;\n\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Search with Query Hash % Not Found', queryhash;\n END IF;\n\n tilearea := st_area(geom);\n _where := format('%s AND st_intersects(geometry, %L::geometry)', search._where, geom);\n\n\n FOR query IN SELECT * FROM partition_queries(_where, search.orderby) LOOP\n query := format('%s LIMIT %L', query, remaining_limit);\n RAISE NOTICE '%', query;\n OPEN curs FOR EXECUTE query;\n LOOP\n FETCH curs INTO iter_record;\n EXIT WHEN NOT FOUND;\n IF exitwhenfull OR skipcovered THEN -- If we are not using exitwhenfull or skipcovered, we do not need to do expensive geometry operations\n clippedgeom := st_intersection(geom, iter_record.geometry);\n\n IF unionedgeom IS NULL THEN\n unionedgeom := clippedgeom;\n ELSE\n unionedgeom := st_union(unionedgeom, clippedgeom);\n END IF;\n\n unionedgeom_area := st_area(unionedgeom);\n\n IF skipcovered AND prev_area = unionedgeom_area THEN\n scancounter := scancounter + 1;\n CONTINUE;\n END IF;\n\n prev_area := unionedgeom_area;\n\n RAISE NOTICE '% % % %', unionedgeom_area/tilearea, counter, scancounter, ftime();\n END IF;\n RAISE NOTICE '% %', iter_record, content_hydrate(iter_record, fields);\n INSERT INTO pgstac_results (content) VALUES (content_hydrate(iter_record, fields));\n\n IF counter >= _limit\n OR scancounter > _scanlimit\n OR ftime() > _timelimit\n OR (exitwhenfull AND unionedgeom_area >= tilearea)\n THEN\n exit_flag := TRUE;\n EXIT;\n END IF;\n counter := counter + 1;\n scancounter := scancounter + 1;\n\n END LOOP;\n CLOSE curs;\n EXIT WHEN exit_flag;\n remaining_limit := _scanlimit - scancounter;\n END LOOP;\n\n SELECT jsonb_agg(content) INTO out_records FROM pgstac_results WHERE content IS NOT NULL;\n\n RETURN jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb)\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDROP FUNCTION IF EXISTS geojsonsearch;\nCREATE OR REPLACE FUNCTION geojsonsearch(\n IN geojson jsonb,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_geomfromgeojson(geojson),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS xyzsearch;\nCREATE OR REPLACE FUNCTION xyzsearch(\n IN _x int,\n IN _y int,\n IN _z int,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_transform(tileenvelope(_z, _x, _y), 4326),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nGRANT ALL ON SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac to pgstac_admin;\n\n-- pgstac_read role limited to using function apis\nGRANT EXECUTE ON FUNCTION search TO pgstac_read;\nGRANT EXECUTE ON FUNCTION search_query TO pgstac_read;\nGRANT EXECUTE ON FUNCTION item_by_id TO pgstac_read;\nGRANT EXECUTE ON FUNCTION get_item TO pgstac_read;\n\nGRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_ingest;\nGRANT USAGE ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_ingest;\nSELECT set_version('0.6.12');\n" }, { "path": "src/pgstac/migrations/pgstac.0.6.13-0.7.0.sql", "content": "RESET ROLE;\nDO $$\nDECLARE\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='postgis') THEN\n CREATE EXTENSION IF NOT EXISTS postgis;\n END IF;\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='btree_gist') THEN\n CREATE EXTENSION IF NOT EXISTS btree_gist;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_admin;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_read;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_ingest;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n\nGRANT pgstac_admin TO current_user;\n\n-- Function to make sure pgstac_admin is the owner of items\nCREATE OR REPLACE FUNCTION pgstac_admin_owns() RETURNS VOID AS $$\nDECLARE\n f RECORD;\nBEGIN\n FOR f IN (\n SELECT\n concat(\n oid::regproc::text,\n '(',\n coalesce(pg_get_function_identity_arguments(oid),''),\n ')'\n ) AS name,\n CASE prokind WHEN 'f' THEN 'FUNCTION' WHEN 'p' THEN 'PROCEDURE' WHEN 'a' THEN 'AGGREGATE' END as typ\n FROM pg_proc\n WHERE\n pronamespace=to_regnamespace('pgstac')\n AND proowner != to_regrole('pgstac_admin')\n AND proname NOT LIKE 'pg_stat%'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n FOR f IN (\n SELECT\n oid::regclass::text as name,\n CASE relkind\n WHEN 'i' THEN 'INDEX'\n WHEN 'I' THEN 'INDEX'\n WHEN 'p' THEN 'TABLE'\n WHEN 'r' THEN 'TABLE'\n WHEN 'v' THEN 'VIEW'\n WHEN 'S' THEN 'SEQUENCE'\n ELSE NULL\n END as typ\n FROM pg_class\n WHERE relnamespace=to_regnamespace('pgstac') and relowner != to_regrole('pgstac_admin') AND relkind IN ('r','p','v','S') AND relname NOT LIKE 'pg_stat'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\nSELECT pgstac_admin_owns();\n\nCREATE SCHEMA IF NOT EXISTS pgstac AUTHORIZATION pgstac_admin;\n\nGRANT ALL ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_admin;\n\nALTER ROLE pgstac_admin SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_read SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_ingest SET SEARCH_PATH TO pgstac, public;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\n\nGRANT pgstac_read TO pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\n\nSET ROLE pgstac_admin;\n\nSET SEARCH_PATH TO pgstac, public;\n\n\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS analyze_items;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS validate_constraints;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nSET client_min_messages TO WARNING;\nSET SEARCH_PATH to pgstac, public;\n-- BEGIN migra calculated SQL\ndrop trigger if exists \"queryables_collection_trigger\" on \"pgstac\".\"collections\";\n\ndrop trigger if exists \"partitions_delete_trigger\" on \"pgstac\".\"partitions\";\n\ndrop trigger if exists \"partitions_trigger\" on \"pgstac\".\"partitions\";\n\nalter table \"pgstac\".\"partitions\" drop constraint \"partitions_collection_fkey\";\n\nalter table \"pgstac\".\"partitions\" drop constraint \"prange\";\n\ndrop function if exists \"pgstac\".\"create_queryable_indexes\"();\n\ndrop function if exists \"pgstac\".\"partition_collection\"(collection text, strategy pgstac.partition_trunc_strategy);\n\ndrop function if exists \"pgstac\".\"partitions_delete_trigger_func\"();\n\ndrop function if exists \"pgstac\".\"partitions_trigger_func\"();\n\ndrop function if exists \"pgstac\".\"parse_dtrange\"(_indate jsonb, relative_base timestamp with time zone);\n\ndrop view if exists \"pgstac\".\"partition_steps\";\n\nalter table \"pgstac\".\"partitions\" drop constraint \"partitions_pkey\";\n\ndrop index if exists \"pgstac\".\"partitions_pkey\";\n\nselect 1; -- drop index if exists \"pgstac\".\"prange\";\n\ndrop index if exists \"pgstac\".\"partitions_range_idx\";\n\ndrop table \"pgstac\".\"partitions\";\n\ncreate table \"pgstac\".\"partition_stats\" (\n \"partition\" text not null,\n \"dtrange\" tstzrange,\n \"edtrange\" tstzrange,\n \"spatial\" geometry,\n \"last_updated\" timestamp with time zone,\n \"keys\" text[]\n);\n\n\ncreate table \"pgstac\".\"query_queue\" (\n \"query\" text not null,\n \"added\" timestamp with time zone default now()\n);\n\n\ncreate table \"pgstac\".\"query_queue_history\" (\n \"query\" text,\n \"added\" timestamp with time zone not null,\n \"finished\" timestamp with time zone not null default now(),\n \"error\" text\n);\n\n\nalter table \"pgstac\".\"collections\" alter column \"partition_trunc\" set data type text using \"partition_trunc\"::text;\n\ndrop type \"pgstac\".\"partition_trunc_strategy\";\n\nCREATE UNIQUE INDEX partition_stats_pkey ON pgstac.partition_stats USING btree (partition);\n\nCREATE UNIQUE INDEX query_queue_pkey ON pgstac.query_queue USING btree (query);\n\nCREATE INDEX partitions_range_idx ON pgstac.partition_stats USING gist (dtrange);\n\nalter table \"pgstac\".\"partition_stats\" add constraint \"partition_stats_pkey\" PRIMARY KEY using index \"partition_stats_pkey\";\n\nalter table \"pgstac\".\"query_queue\" add constraint \"query_queue_pkey\" PRIMARY KEY using index \"query_queue_pkey\";\n\nalter table \"pgstac\".\"collections\" add constraint \"collections_partition_trunc_check\" CHECK ((partition_trunc = ANY (ARRAY['year'::text, 'month'::text]))) not valid;\n\nalter table \"pgstac\".\"collections\" validate constraint \"collections_partition_trunc_check\";\n\nset check_function_bodies = off;\n\nCREATE OR REPLACE FUNCTION pgstac.check_partition(_collection text, _dtrange tstzrange, _edtrange tstzrange)\n RETURNS text\n LANGUAGE plpgsql\n SECURITY DEFINER\nAS $function$\nDECLARE\n c RECORD;\n pm RECORD;\n _partition_name text;\n _partition_dtrange tstzrange;\n _constraint_dtrange tstzrange;\n _constraint_edtrange tstzrange;\n q text;\n deferrable_q text;\n err_context text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=_collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', _collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n\n IF c.partition_trunc IS NOT NULL THEN\n _partition_dtrange := tstzrange(\n date_trunc(c.partition_trunc, lower(_dtrange)),\n date_trunc(c.partition_trunc, lower(_dtrange)) + (concat('1 ', c.partition_trunc))::interval,\n '[)'\n );\n ELSE\n _partition_dtrange := '[-infinity, infinity]'::tstzrange;\n END IF;\n\n IF NOT _partition_dtrange @> _dtrange THEN\n RAISE EXCEPTION 'dtrange % is greater than the partition size % for collection %', _dtrange, c.partition_trunc, _collection;\n END IF;\n\n\n IF c.partition_trunc = 'year' THEN\n _partition_name := format('_items_%s_%s', c.key, to_char(lower(_partition_dtrange),'YYYY'));\n ELSIF c.partition_trunc = 'month' THEN\n _partition_name := format('_items_%s_%s', c.key, to_char(lower(_partition_dtrange),'YYYYMM'));\n ELSE\n _partition_name := format('_items_%s', c.key);\n END IF;\n\n SELECT * INTO pm FROM partition_sys_meta WHERE collection=_collection AND partition_dtrange @> _dtrange;\n IF FOUND THEN\n RAISE NOTICE '% % %', _edtrange, _dtrange, pm;\n _constraint_edtrange :=\n tstzrange(\n least(\n lower(_edtrange),\n nullif(lower(pm.constraint_edtrange), '-infinity')\n ),\n greatest(\n upper(_edtrange),\n nullif(upper(pm.constraint_edtrange), 'infinity')\n ),\n '[]'\n );\n _constraint_dtrange :=\n tstzrange(\n least(\n lower(_dtrange),\n nullif(lower(pm.constraint_dtrange), '-infinity')\n ),\n greatest(\n upper(_dtrange),\n nullif(upper(pm.constraint_dtrange), 'infinity')\n ),\n '[]'\n );\n\n IF pm.constraint_edtrange @> _edtrange AND pm.constraint_dtrange @> _dtrange THEN\n RETURN pm.partition;\n ELSE\n PERFORM drop_table_constraints(_partition_name);\n END IF;\n ELSE\n _constraint_edtrange := _edtrange;\n _constraint_dtrange := _dtrange;\n END IF;\n RAISE NOTICE 'Creating partition % %', _partition_name, _partition_dtrange;\n IF c.partition_trunc IS NULL THEN\n q := format(\n $q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n $q$,\n _partition_name,\n _collection,\n concat(_partition_name,'_pk'),\n _partition_name\n );\n ELSE\n q := format(\n $q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L) PARTITION BY RANGE (datetime);\n CREATE TABLE IF NOT EXISTS %I partition OF %I FOR VALUES FROM (%L) TO (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n $q$,\n format('_items_%s', c.key),\n _collection,\n _partition_name,\n format('_items_%s', c.key),\n lower(_partition_dtrange),\n upper(_partition_dtrange),\n format('%s_pk', _partition_name),\n _partition_name\n );\n END IF;\n\n BEGIN\n EXECUTE q;\n EXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', _partition_name;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\n END;\n PERFORM create_table_constraints(_partition_name, _constraint_dtrange, _constraint_edtrange);\n PERFORM maintain_partitions(_partition_name);\n PERFORM update_partition_stats_q(_partition_name, true);\n RETURN _partition_name;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.collection_extent(_collection text, runupdate boolean DEFAULT false)\n RETURNS jsonb\n LANGUAGE plpgsql\nAS $function$\nDECLARE\n geom_extent geometry;\n mind timestamptz;\n maxd timestamptz;\n extent jsonb;\nBEGIN\n IF runupdate THEN\n PERFORM update_partition_stats_q(partition)\n FROM partitions WHERE collection=_collection;\n END IF;\n SELECT\n min(lower(dtrange)),\n max(upper(edtrange)),\n st_extent(spatial)\n INTO\n mind,\n maxd,\n geom_extent\n FROM partitions\n WHERE collection=_collection;\n\n IF geom_extent IS NOT NULL AND mind IS NOT NULL AND maxd IS NOT NULL THEN\n extent := jsonb_build_object(\n 'extent', jsonb_build_object(\n 'spatial', jsonb_build_object(\n 'bbox', to_jsonb(array[array[st_xmin(geom_extent), st_ymin(geom_extent), st_xmax(geom_extent), st_ymax(geom_extent)]])\n ),\n 'temporal', jsonb_build_object(\n 'interval', to_jsonb(array[array[mind, maxd]])\n )\n )\n );\n RETURN extent;\n END IF;\n RETURN NULL;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.constraint_tstzrange(expr text)\n RETURNS tstzrange\n LANGUAGE sql\n IMMUTABLE PARALLEL SAFE STRICT\nAS $function$\n WITH t AS (\n SELECT regexp_matches(\n expr,\n E'\\\\(''\\([0-9 :+-]*\\)''\\\\).*\\\\(''\\([0-9 :+-]*\\)''\\\\)'\n ) AS m\n ) SELECT tstzrange(m[1]::timestamptz, m[2]::timestamptz) FROM t\n ;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.create_table_constraints(t text, _dtrange tstzrange, _edtrange tstzrange)\n RETURNS text\n LANGUAGE plpgsql\n SECURITY DEFINER\nAS $function$\nDECLARE\n q text;\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM partitions WHERE partition=t) THEN\n RETURN NULL;\n END IF;\n RAISE NOTICE 'Creating Table Constraints for % % %', t, _dtrange, _edtrange;\n IF _dtrange = 'empty' AND _edtrange = 'empty' THEN\n q :=format(\n $q$\n ALTER TABLE %I\n ADD CONSTRAINT %I\n CHECK (((datetime IS NULL) AND (end_datetime IS NULL))) NOT VALID\n ;\n ALTER TABLE %I\n VALIDATE CONSTRAINT %I\n ;\n $q$,\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t)\n );\n ELSE\n q :=format(\n $q$\n ALTER TABLE %I\n ADD CONSTRAINT %I\n CHECK (\n (datetime >= %L)\n AND (datetime <= %L)\n AND (end_datetime >= %L)\n AND (end_datetime <= %L)\n ) NOT VALID\n ;\n ALTER TABLE %I\n VALIDATE CONSTRAINT %I\n ;\n $q$,\n t,\n format('%s_dt', t),\n lower(_dtrange),\n upper(_dtrange),\n lower(_edtrange),\n upper(_edtrange),\n t,\n format('%s_dt', t)\n );\n END IF;\n PERFORM run_or_queue(q);\n RETURN t;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.drop_table_constraints(t text)\n RETURNS text\n LANGUAGE plpgsql\n SECURITY DEFINER\nAS $function$\nDECLARE\n q text;\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM partitions WHERE partition=t) THEN\n RETURN NULL;\n END IF;\n FOR q IN SELECT FORMAT(\n $q$\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n $q$,\n t,\n conname\n ) FROM pg_constraint\n WHERE conrelid=t::regclass::oid AND contype='c'\n LOOP\n EXECUTE q;\n END LOOP;\n RETURN t;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.dt_constraint(coid oid, OUT dt tstzrange, OUT edt tstzrange)\n RETURNS record\n LANGUAGE plpgsql\n STABLE STRICT\nAS $function$\nDECLARE\n expr text := pg_get_constraintdef(coid);\n matches timestamptz[];\nBEGIN\n IF expr LIKE '%NULL%' THEN\n dt := tstzrange(null::timestamptz, null::timestamptz);\n edt := tstzrange(null::timestamptz, null::timestamptz);\n RETURN;\n END IF;\n WITH f AS (SELECT (regexp_matches(expr, E'([0-9]{4}-[0-1][0-9]-[0-3][0-9] [0-2][0-9]:[0-5][0-9]:[0-5][0-9]\\.?[0-9]*)', 'g'))[1] f)\n SELECT array_agg(f::timestamptz) INTO matches FROM f;\n IF cardinality(matches) = 4 THEN\n dt := tstzrange(matches[1], matches[2],'[]');\n edt := tstzrange(matches[3], matches[4], '[]');\n RETURN;\n ELSIF cardinality(matches) = 2 THEN\n edt := tstzrange(matches[1], matches[2],'[]');\n RETURN;\n END IF;\n RETURN;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.get_setting_bool(_setting text, conf jsonb DEFAULT NULL::jsonb)\n RETURNS boolean\n LANGUAGE sql\nAS $function$\nSELECT COALESCE(\n conf->>_setting,\n current_setting(concat('pgstac.',_setting), TRUE),\n (SELECT value FROM pgstac.pgstac_settings WHERE name=_setting),\n 'FALSE'\n)::boolean;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.maintain_partition_queries(part text DEFAULT 'items'::text, dropindexes boolean DEFAULT false, rebuildindexes boolean DEFAULT false)\n RETURNS SETOF text\n LANGUAGE plpgsql\nAS $function$\nDECLARE\n parent text;\n level int;\n isleaf bool;\n collection collections%ROWTYPE;\n subpart text;\n baseidx text;\n queryable_name text;\n queryable_property_index_type text;\n queryable_property_wrapper text;\n queryable_parsed RECORD;\n deletedidx pg_indexes%ROWTYPE;\n q text;\n idx text;\n collection_partition bigint;\n _concurrently text := '';\nBEGIN\n RAISE NOTICE 'Maintaining partition: %', part;\n IF get_setting_bool('use_queue') THEN\n _concurrently='CONCURRENTLY';\n END IF;\n\n -- Get root partition\n SELECT parentrelid::text, pt.isleaf, pt.level\n INTO parent, isleaf, level\n FROM pg_partition_tree('items') pt\n WHERE relid::text = part;\n IF NOT FOUND THEN\n RAISE NOTICE 'Partition % Does Not Exist In Partition Tree', part;\n RETURN;\n END IF;\n\n -- If this is a parent partition, recurse to leaves\n IF NOT isleaf THEN\n FOR subpart IN\n SELECT relid::text\n FROM pg_partition_tree(part)\n WHERE relid::text != part\n LOOP\n RAISE NOTICE 'Recursing to %', subpart;\n RETURN QUERY SELECT * FROM maintain_partition_queries(subpart, dropindexes, rebuildindexes);\n END LOOP;\n RETURN; -- Don't continue since not an end leaf\n END IF;\n\n\n -- Get collection\n collection_partition := ((regexp_match(part, E'^_items_([0-9]+)'))[1])::bigint;\n RAISE NOTICE 'COLLECTION PARTITION: %', collection_partition;\n SELECT * INTO STRICT collection\n FROM collections\n WHERE key = collection_partition;\n RAISE NOTICE 'COLLECTION ID: %s', collection.id;\n\n\n -- Create temp table with existing indexes\n CREATE TEMP TABLE existing_indexes ON COMMIT DROP AS\n SELECT *\n FROM pg_indexes\n WHERE schemaname='pgstac' AND tablename=part;\n\n\n -- Check if index exists for each queryable.\n FOR\n queryable_name,\n queryable_property_index_type,\n queryable_property_wrapper\n IN\n SELECT\n name,\n COALESCE(property_index_type, 'BTREE'),\n COALESCE(property_wrapper, 'to_text')\n FROM queryables\n WHERE\n name NOT in ('id', 'datetime', 'geometry')\n AND (\n collection_ids IS NULL\n OR collection_ids = '{}'::text[]\n OR collection.id = ANY (collection_ids)\n )\n UNION ALL\n SELECT 'datetime desc, end_datetime', 'BTREE', ''\n UNION ALL\n SELECT 'geometry', 'GIST', ''\n UNION ALL\n SELECT 'id', 'BTREE', ''\n LOOP\n baseidx := format(\n $q$ ON %I USING %s (%s(((content -> 'properties'::text) -> %L::text)))$q$,\n part,\n queryable_property_index_type,\n queryable_property_wrapper,\n queryable_name\n );\n RAISE NOTICE 'BASEIDX: %', baseidx;\n RAISE NOTICE 'IDXSEARCH: %', format($q$[(']%s[')]$q$, queryable_name);\n -- If index already exists, delete it from existing indexes type table\n FOR deletedidx IN\n DELETE FROM existing_indexes\n WHERE indexdef ~* format($q$[(']%s[')]$q$, queryable_name)\n RETURNING *\n LOOP\n RAISE NOTICE 'EXISTING INDEX: %', deletedidx;\n IF NOT FOUND THEN -- index did not exist, create it\n RETURN NEXT format('CREATE INDEX %s %s;', _concurrently, baseidx);\n ELSIF rebuildindexes THEN\n RETURN NEXT format('REINDEX %I %s;', deletedidx.indexname, _concurrently);\n END IF;\n END LOOP;\n END LOOP;\n\n -- Remove indexes that were not expected\n FOR idx IN SELECT indexname::text FROM existing_indexes\n LOOP\n RAISE WARNING 'Index: % is not defined by queryables.', idx;\n IF dropindexes THEN\n RETURN NEXT format('DROP INDEX IF EXISTS %I;', idx);\n END IF;\n END LOOP;\n\n DROP TABLE existing_indexes;\n RAISE NOTICE 'Returning from maintain_partition_queries.';\n RETURN;\n\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.maintain_partitions(part text DEFAULT 'items'::text, dropindexes boolean DEFAULT false, rebuildindexes boolean DEFAULT false)\n RETURNS void\n LANGUAGE sql\nAS $function$\n WITH t AS (\n SELECT run_or_queue(q) FROM maintain_partition_queries(part, dropindexes, rebuildindexes) q\n ) SELECT count(*) FROM t;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.partition_after_triggerfunc()\n RETURNS trigger\n LANGUAGE plpgsql\n SECURITY DEFINER\nAS $function$\nDECLARE\n p text;\n t timestamptz := clock_timestamp();\nBEGIN\n RAISE NOTICE 'Updating partition stats %', t;\n FOR p IN SELECT DISTINCT partition\n FROM newdata n JOIN partition_sys_meta p\n ON (n.collection=p.collection AND n.datetime <@ p.partition_dtrange)\n LOOP\n PERFORM run_or_queue(format('SELECT update_partition_stats(%L, %L);', p, true));\n END LOOP;\n RAISE NOTICE 't: % %', t, clock_timestamp() - t;\n RETURN NULL;\nEND;\n$function$\n;\n\ncreate or replace view \"pgstac\".\"partition_sys_meta\" as SELECT (pg_partition_tree.relid)::text AS partition,\n replace(replace(\n CASE\n WHEN (pg_partition_tree.level = 1) THEN pg_get_expr(c.relpartbound, c.oid)\n ELSE pg_get_expr(parent.relpartbound, parent.oid)\n END, 'FOR VALUES IN ('''::text, ''::text), ''')'::text, ''::text) AS collection,\n pg_partition_tree.level,\n c.reltuples,\n c.relhastriggers,\n COALESCE(pgstac.constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity'::timestamp with time zone, 'infinity'::timestamp with time zone, '[]'::text)) AS partition_dtrange,\n COALESCE((pgstac.dt_constraint(edt.oid)).dt, pgstac.constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity'::timestamp with time zone, 'infinity'::timestamp with time zone, '[]'::text)) AS constraint_dtrange,\n COALESCE((pgstac.dt_constraint(edt.oid)).edt, tstzrange('-infinity'::timestamp with time zone, 'infinity'::timestamp with time zone, '[]'::text)) AS constraint_edtrange\n FROM (((pg_partition_tree('pgstac.items'::regclass) pg_partition_tree(relid, parentrelid, isleaf, level)\n JOIN pg_class c ON (((pg_partition_tree.relid)::oid = c.oid)))\n JOIN pg_class parent ON ((((pg_partition_tree.parentrelid)::oid = parent.oid) AND pg_partition_tree.isleaf)))\n LEFT JOIN pg_constraint edt ON (((edt.conrelid = c.oid) AND (edt.contype = 'c'::\"char\"))))\n WHERE pg_partition_tree.isleaf;\n\n\ncreate or replace view \"pgstac\".\"partitions\" as SELECT partition_sys_meta.partition,\n partition_sys_meta.collection,\n partition_sys_meta.level,\n partition_sys_meta.reltuples,\n partition_sys_meta.relhastriggers,\n partition_sys_meta.partition_dtrange,\n partition_sys_meta.constraint_dtrange,\n partition_sys_meta.constraint_edtrange,\n partition_stats.dtrange,\n partition_stats.edtrange,\n partition_stats.spatial,\n partition_stats.last_updated,\n partition_stats.keys\n FROM (pgstac.partition_sys_meta\n LEFT JOIN pgstac.partition_stats USING (partition));\n\n\ncreate or replace view \"pgstac\".\"pgstac_indexes\" as SELECT i.schemaname,\n i.tablename,\n i.indexname,\n i.indexdef,\n COALESCE((regexp_match(i.indexdef, '\\(([a-zA-Z]+)\\)'::text))[1], (regexp_match(i.indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_]+)''::text'::text))[1],\n CASE\n WHEN (i.indexdef ~* '\\(datetime desc, end_datetime\\)'::text) THEN 'datetime_end_datetime'::text\n ELSE NULL::text\n END) AS field,\n pg_table_size(((i.indexname)::text)::regclass) AS index_size,\n pg_size_pretty(pg_table_size(((i.indexname)::text)::regclass)) AS index_size_pretty\n FROM pg_indexes i\n WHERE ((i.schemaname = 'pgstac'::name) AND (i.tablename ~ '_items_'::text));\n\n\ncreate or replace view \"pgstac\".\"pgstac_indexes_stats\" as SELECT i.schemaname,\n i.tablename,\n i.indexname,\n i.indexdef,\n COALESCE((regexp_match(i.indexdef, '\\(([a-zA-Z]+)\\)'::text))[1], (regexp_match(i.indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_]+)''::text'::text))[1],\n CASE\n WHEN (i.indexdef ~* '\\(datetime desc, end_datetime\\)'::text) THEN 'datetime_end_datetime'::text\n ELSE NULL::text\n END) AS field,\n pg_table_size(((i.indexname)::text)::regclass) AS index_size,\n pg_size_pretty(pg_table_size(((i.indexname)::text)::regclass)) AS index_size_pretty,\n s.n_distinct,\n ((s.most_common_vals)::text)::text[] AS most_common_vals,\n ((s.most_common_freqs)::text)::text[] AS most_common_freqs,\n ((s.histogram_bounds)::text)::text[] AS histogram_bounds,\n s.correlation\n FROM (pg_indexes i\n LEFT JOIN pg_stats s ON ((s.tablename = i.indexname)))\n WHERE ((i.schemaname = 'pgstac'::name) AND (i.tablename ~ '_items_'::text));\n\n\nCREATE OR REPLACE FUNCTION pgstac.queue_timeout()\n RETURNS interval\n LANGUAGE sql\nAS $function$\n SELECT set_config(\n 'statement_timeout',\n t2s(coalesce(\n get_setting('queue_timeout'),\n '1h'\n )),\n false\n )::interval;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.repartition(_collection text, _partition_trunc text, triggered boolean DEFAULT false)\n RETURNS text\n LANGUAGE plpgsql\n SECURITY DEFINER\nAS $function$\nDECLARE\n c RECORD;\n q text;\n from_trunc text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=_collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', _collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n IF triggered THEN\n RAISE NOTICE 'Converting % to % partitioning via Trigger', _collection, _partition_trunc;\n ELSE\n RAISE NOTICE 'Converting % from using % to % partitioning', _collection, c.partition_trunc, _partition_trunc;\n IF c.partition_trunc IS NOT DISTINCT FROM _partition_trunc THEN\n RAISE NOTICE 'Collection % already set to use partition by %', _collection, _partition_trunc;\n RETURN _collection;\n END IF;\n END IF;\n\n IF EXISTS (SELECT 1 FROM partitions WHERE collection=_collection LIMIT 1) THEN\n EXECUTE format(\n $q$\n CREATE TEMP TABLE changepartitionstaging ON COMMIT DROP AS SELECT * FROM %I;\n DROP TABLE IF EXISTS %I CASCADE;\n WITH p AS (\n SELECT\n collection,\n CASE WHEN %L IS NULL THEN '-infinity'::timestamptz\n ELSE date_trunc(%L, datetime)\n END as d,\n tstzrange(min(datetime),max(datetime),'[]') as dtrange,\n tstzrange(min(datetime),max(datetime),'[]') as edtrange\n FROM changepartitionstaging\n GROUP BY 1,2\n ) SELECT check_partition(collection, dtrange, edtrange) FROM p;\n INSERT INTO items SELECT * FROM changepartitionstaging;\n DROP TABLE changepartitionstaging;\n $q$,\n concat('_items_', c.key),\n concat('_items_', c.key),\n c.partition_trunc,\n c.partition_trunc\n );\n END IF;\n RETURN _collection;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.run_or_queue(query text)\n RETURNS void\n LANGUAGE plpgsql\nAS $function$\nDECLARE\n use_queue text := COALESCE(get_setting('use_queue'), 'FALSE')::boolean;\nBEGIN\n IF get_setting_bool('debug') THEN\n RAISE NOTICE '%', query;\n END IF;\n IF use_queue THEN\n INSERT INTO query_queue (query) VALUES (query) ON CONFLICT DO NOTHING;\n ELSE\n EXECUTE query;\n END IF;\n RETURN;\nEND;\n$function$\n;\n\nCREATE OR REPLACE PROCEDURE pgstac.run_queued_queries()\n LANGUAGE plpgsql\nAS $procedure$\nDECLARE\n qitem query_queue%ROWTYPE;\n timeout_ts timestamptz;\n error text;\n cnt int := 0;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n DELETE FROM query_queue WHERE query = (SELECT query FROM query_queue ORDER BY added DESC LIMIT 1 FOR UPDATE SKIP LOCKED) RETURNING * INTO qitem;\n IF NOT FOUND THEN\n EXIT;\n END IF;\n cnt := cnt + 1;\n BEGIN\n RAISE NOTICE 'RUNNING QUERY: %', qitem.query;\n EXECUTE qitem.query;\n EXCEPTION WHEN others THEN\n error := format('%s | %s', SQLERRM, SQLSTATE);\n END;\n INSERT INTO query_queue_history (query, added, finished, error)\n VALUES (qitem.query, qitem.added, clock_timestamp(), error);\n COMMIT;\n END LOOP;\nEND;\n$procedure$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.run_queued_queries_intransaction()\n RETURNS integer\n LANGUAGE plpgsql\nAS $function$\nDECLARE\n qitem query_queue%ROWTYPE;\n timeout_ts timestamptz;\n error text;\n cnt int := 0;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n DELETE FROM query_queue WHERE query = (SELECT query FROM query_queue ORDER BY added DESC LIMIT 1 FOR UPDATE SKIP LOCKED) RETURNING * INTO qitem;\n IF NOT FOUND THEN\n RETURN cnt;\n END IF;\n cnt := cnt + 1;\n BEGIN\n qitem.query := regexp_replace(qitem.query, 'CONCURRENTLY', '');\n RAISE NOTICE 'RUNNING QUERY: %', qitem.query;\n\n EXECUTE qitem.query;\n EXCEPTION WHEN others THEN\n error := format('%s | %s', SQLERRM, SQLSTATE);\n RAISE WARNING '%', error;\n END;\n INSERT INTO query_queue_history (query, added, finished, error)\n VALUES (qitem.query, qitem.added, clock_timestamp(), error);\n END LOOP;\n RETURN cnt;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.t2s(text)\n RETURNS text\n LANGUAGE sql\n IMMUTABLE PARALLEL SAFE STRICT\nAS $function$\n SELECT extract(epoch FROM $1::interval)::text || ' s';\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.update_partition_stats(_partition text, istrigger boolean DEFAULT false)\n RETURNS void\n LANGUAGE plpgsql\n STRICT\nAS $function$\nDECLARE\n dtrange tstzrange;\n edtrange tstzrange;\n cdtrange tstzrange;\n cedtrange tstzrange;\n extent geometry;\n collection text;\nBEGIN\n RAISE NOTICE 'Updating stats for %.', _partition;\n EXECUTE format(\n $q$\n SELECT\n tstzrange(min(datetime), max(datetime),'[]'),\n tstzrange(min(end_datetime), max(end_datetime), '[]')\n FROM %I\n $q$,\n _partition\n ) INTO dtrange, edtrange;\n extent := st_estimatedextent('pgstac', _partition, 'geometry');\n INSERT INTO partition_stats (partition, dtrange, edtrange, spatial, last_updated)\n SELECT _partition, dtrange, edtrange, extent, now()\n ON CONFLICT (partition) DO\n UPDATE SET\n dtrange=EXCLUDED.dtrange,\n edtrange=EXCLUDED.edtrange,\n spatial=EXCLUDED.spatial,\n last_updated=EXCLUDED.last_updated\n ;\n SELECT\n constraint_dtrange, constraint_edtrange, partitions.collection\n INTO cdtrange, cedtrange, collection\n FROM partitions WHERE partition = _partition;\n\n RAISE NOTICE 'Checking if we need to modify constraints.';\n IF\n (cdtrange IS DISTINCT FROM dtrange OR edtrange IS DISTINCT FROM cedtrange)\n AND NOT istrigger\n THEN\n RAISE NOTICE 'Modifying Constraints';\n RAISE NOTICE 'Existing % %', cdtrange, cedtrange;\n RAISE NOTICE 'New % %', dtrange, edtrange;\n PERFORM drop_table_constraints(_partition);\n PERFORM create_table_constraints(_partition, dtrange, edtrange);\n END IF;\n RAISE NOTICE 'Checking if we need to update collection extents.';\n IF get_setting_bool('update_collection_extent') THEN\n RAISE NOTICE 'updating collection extent for %', collection;\n PERFORM run_or_queue(format($q$\n UPDATE collections\n SET content = jsonb_set_lax(\n content,\n '{extent}'::text[],\n collection_extent(%L),\n true,\n 'use_json_null'\n ) WHERE id=%L\n ;\n $q$, collection, collection));\n ELSE\n RAISE NOTICE 'Not updating collection extent for %', collection;\n END IF;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.update_partition_stats_q(_partition text, istrigger boolean DEFAULT false)\n RETURNS void\n LANGUAGE plpgsql\nAS $function$\nDECLARE\nBEGIN\n PERFORM run_or_queue(\n format('SELECT update_partition_stats(%L, %L);', _partition, istrigger)\n );\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.all_collections()\n RETURNS jsonb\n LANGUAGE sql\n SET search_path TO 'pgstac', 'public'\nAS $function$\n SELECT jsonb_agg(content) FROM collections;\n$function$\n;\n\nCREATE OR REPLACE PROCEDURE pgstac.analyze_items()\n LANGUAGE plpgsql\nAS $procedure$\nDECLARE\n q text;\n timeout_ts timestamptz;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n RAISE NOTICE '% % %', clock_timestamp(), timeout_ts, current_setting('statement_timeout', TRUE);\n SELECT format('ANALYZE (VERBOSE, SKIP_LOCKED) %I;', relname) INTO q\n FROM pg_stat_user_tables\n WHERE relname like '_item%' AND (n_mod_since_analyze>0 OR last_analyze IS NULL) LIMIT 1;\n IF NOT FOUND THEN\n EXIT;\n END IF;\n RAISE NOTICE '%', q;\n EXECUTE q;\n COMMIT;\n RAISE NOTICE '%', queue_timeout();\n END LOOP;\nEND;\n$procedure$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.check_pgstac_settings(_sysmem text DEFAULT NULL::text)\n RETURNS void\n LANGUAGE plpgsql\n SET search_path TO 'pgstac', 'public'\n SET client_min_messages TO 'notice'\nAS $function$\nDECLARE\n settingval text;\n sysmem bigint := pg_size_bytes(_sysmem);\n effective_cache_size bigint := pg_size_bytes(current_setting('effective_cache_size', TRUE));\n shared_buffers bigint := pg_size_bytes(current_setting('shared_buffers', TRUE));\n work_mem bigint := pg_size_bytes(current_setting('work_mem', TRUE));\n max_connections int := current_setting('max_connections', TRUE);\n maintenance_work_mem bigint := pg_size_bytes(current_setting('maintenance_work_mem', TRUE));\n seq_page_cost float := current_setting('seq_page_cost', TRUE);\n random_page_cost float := current_setting('random_page_cost', TRUE);\n temp_buffers bigint := pg_size_bytes(current_setting('temp_buffers', TRUE));\n r record;\nBEGIN\n IF _sysmem IS NULL THEN\n RAISE NOTICE 'Call function with the size of your system memory `SELECT check_pgstac_settings(''4GB'')` to get pg system setting recommendations.';\n ELSE\n IF effective_cache_size < (sysmem * 0.5) THEN\n RAISE WARNING 'effective_cache_size of % is set low for a system with %. Recomended value between % and %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.5), pg_size_pretty(sysmem * 0.75);\n ELSIF effective_cache_size > (sysmem * 0.75) THEN\n RAISE WARNING 'effective_cache_size of % is set high for a system with %. Recomended value between % and %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.5), pg_size_pretty(sysmem * 0.75);\n ELSE\n RAISE NOTICE 'effective_cache_size of % is set appropriately for a system with %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem);\n END IF;\n\n IF shared_buffers < (sysmem * 0.2) THEN\n RAISE WARNING 'shared_buffers of % is set low for a system with %. Recomended value between % and %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.2), pg_size_pretty(sysmem * 0.3);\n ELSIF shared_buffers > (sysmem * 0.3) THEN\n RAISE WARNING 'shared_buffers of % is set high for a system with %. Recomended value between % and %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.2), pg_size_pretty(sysmem * 0.3);\n ELSE\n RAISE NOTICE 'shared_buffers of % is set appropriately for a system with %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem);\n END IF;\n shared_buffers = sysmem * 0.3;\n IF maintenance_work_mem < (sysmem * 0.2) THEN\n RAISE WARNING 'maintenance_work_mem of % is set low for shared_buffers of %. Recomended value between % and %', pg_size_pretty(maintenance_work_mem), pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers * 0.2), pg_size_pretty(shared_buffers * 0.3);\n ELSIF maintenance_work_mem > (shared_buffers * 0.3) THEN\n RAISE WARNING 'maintenance_work_mem of % is set high for shared_buffers of %. Recomended value between % and %', pg_size_pretty(maintenance_work_mem), pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers * 0.2), pg_size_pretty(shared_buffers * 0.3);\n ELSE\n RAISE NOTICE 'maintenance_work_mem of % is set appropriately for shared_buffers of %', pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers);\n END IF;\n\n IF work_mem * max_connections > shared_buffers THEN\n RAISE WARNING 'work_mem setting of % is set high for % max_connections please reduce work_mem to % or decrease max_connections to %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers/max_connections), floor(shared_buffers/work_mem);\n ELSIF work_mem * max_connections < (shared_buffers * 0.75) THEN\n RAISE WARNING 'work_mem setting of % is set low for % max_connections you may consider raising work_mem to % or increasing max_connections to %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers/max_connections), floor(shared_buffers/work_mem);\n ELSE\n RAISE NOTICE 'work_mem setting of % and max_connections of % are adequate for shared_buffers of %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers);\n END IF;\n\n IF random_page_cost / seq_page_cost != 1.1 THEN\n RAISE WARNING 'random_page_cost (%) /seq_page_cost (%) should be set to 1.1 for SSD. Change random_page_cost to %', random_page_cost, seq_page_cost, 1.1 * seq_page_cost;\n ELSE\n RAISE NOTICE 'random_page_cost and seq_page_cost set appropriately for SSD';\n END IF;\n\n IF temp_buffers < greatest(pg_size_bytes('128MB'),(maintenance_work_mem / 2)) THEN\n RAISE WARNING 'pgstac makes heavy use of temp tables, consider raising temp_buffers from % to %', pg_size_pretty(temp_buffers), greatest('128MB', pg_size_pretty((shared_buffers / 16)));\n END IF;\n END IF;\n\n RAISE NOTICE 'VALUES FOR PGSTAC VARIABLES';\n RAISE NOTICE 'These can be set either as GUC system variables or by setting in the pgstac_settings table.';\n\n FOR r IN SELECT name, get_setting(name) as setting, CASE WHEN current_setting(concat('pgstac.',name), TRUE) IS NOT NULL THEN concat('pgstac.',name, ' GUC') WHEN value IS NOT NULL THEN 'pgstac_settings table' ELSE 'Not Set' END as loc FROM pgstac_settings LOOP\n RAISE NOTICE '% is set to % from the %', r.name, r.setting, r.loc;\n END LOOP;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pg_cron';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider intalling pg_cron which can be used to automate tasks';\n ELSE\n RAISE NOTICE 'pg_cron % is installed', settingval;\n END IF;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pgstattuple';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider installing the pgstattuple extension which can be used to help maintain tables and indexes.';\n ELSE\n RAISE NOTICE 'pgstattuple % is installed', settingval;\n END IF;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pg_stat_statements';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider installing the pg_stat_statements extension which is very helpful for tracking the types of queries on the system';\n ELSE\n RAISE NOTICE 'pg_stat_statements % is installed', settingval;\n IF current_setting('pg_stat_statements.track_statements', TRUE) IS DISTINCT FROM 'all' THEN\n RAISE WARNING 'SET pg_stat_statements.track_statements TO ''all''; --In order to track statements within functions.';\n END IF;\n END IF;\n\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.collections_trigger_func()\n RETURNS trigger\n LANGUAGE plpgsql\nAS $function$\nDECLARE\n q text;\n partition_name text := format('_items_%s', NEW.key);\n partition_exists boolean := false;\n partition_empty boolean := true;\n err_context text;\n loadtemp boolean := FALSE;\nBEGIN\n RAISE NOTICE 'Collection Trigger. % %', NEW.id, NEW.key;\n IF TG_OP = 'UPDATE' AND NEW.partition_trunc IS DISTINCT FROM OLD.partition_trunc THEN\n PERFORM repartition(NEW.id, NEW.partition_trunc, TRUE);\n END IF;\n RETURN NEW;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.get_item(_id text, _collection text DEFAULT NULL::text)\n RETURNS jsonb\n LANGUAGE sql\n STABLE\n SET search_path TO 'pgstac', 'public'\nAS $function$\n SELECT content_hydrate(items) FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection);\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.get_setting(_setting text, conf jsonb DEFAULT NULL::jsonb)\n RETURNS text\n LANGUAGE sql\nAS $function$\nSELECT COALESCE(\n nullif(conf->>_setting, ''),\n nullif(current_setting(concat('pgstac.',_setting), TRUE),''),\n nullif((SELECT value FROM pgstac.pgstac_settings WHERE name=_setting),'')\n);\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.item_by_id(_id text, _collection text DEFAULT NULL::text)\n RETURNS pgstac.items\n LANGUAGE plpgsql\n STABLE\n SET search_path TO 'pgstac', 'public'\nAS $function$\nDECLARE\n i items%ROWTYPE;\nBEGIN\n SELECT * INTO i FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection) LIMIT 1;\n RETURN i;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.items_staging_triggerfunc()\n RETURNS trigger\n LANGUAGE plpgsql\nAS $function$\nDECLARE\n p record;\n _partitions text[];\n part text;\n ts timestamptz := clock_timestamp();\nBEGIN\n RAISE NOTICE 'Creating Partitions. %', clock_timestamp() - ts;\n\n FOR part IN WITH t AS (\n SELECT\n n.content->>'collection' as collection,\n stac_daterange(n.content->'properties') as dtr,\n partition_trunc\n FROM newdata n JOIN collections ON (n.content->>'collection'=collections.id)\n ), p AS (\n SELECT\n collection,\n COALESCE(date_trunc(partition_trunc::text, lower(dtr)),'-infinity') as d,\n tstzrange(min(lower(dtr)),max(lower(dtr)),'[]') as dtrange,\n tstzrange(min(upper(dtr)),max(upper(dtr)),'[]') as edtrange\n FROM t\n GROUP BY 1,2\n ) SELECT check_partition(collection, dtrange, edtrange) FROM p LOOP\n RAISE NOTICE 'Partition %', part;\n END LOOP;\n\n RAISE NOTICE 'Doing the insert. %', clock_timestamp() - ts;\n IF TG_TABLE_NAME = 'items_staging' THEN\n INSERT INTO items\n SELECT\n (content_dehydrate(content)).*\n FROM newdata;\n RAISE NOTICE 'Doing the delete. %', clock_timestamp() - ts;\n DELETE FROM items_staging;\n ELSIF TG_TABLE_NAME = 'items_staging_ignore' THEN\n INSERT INTO items\n SELECT\n (content_dehydrate(content)).*\n FROM newdata\n ON CONFLICT DO NOTHING;\n RAISE NOTICE 'Doing the delete. %', clock_timestamp() - ts;\n DELETE FROM items_staging_ignore;\n ELSIF TG_TABLE_NAME = 'items_staging_upsert' THEN\n WITH staging_formatted AS (\n SELECT (content_dehydrate(content)).* FROM newdata\n ), deletes AS (\n DELETE FROM items i USING staging_formatted s\n WHERE\n i.id = s.id\n AND i.collection = s.collection\n AND i IS DISTINCT FROM s\n RETURNING i.id, i.collection\n )\n INSERT INTO items\n SELECT s.* FROM\n staging_formatted s\n ON CONFLICT DO NOTHING;\n RAISE NOTICE 'Doing the delete. %', clock_timestamp() - ts;\n DELETE FROM items_staging_upsert;\n END IF;\n RAISE NOTICE 'Done. %', clock_timestamp() - ts;\n\n RETURN NULL;\n\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.parse_dtrange(_indate jsonb, relative_base timestamp with time zone DEFAULT date_trunc('hour'::text, CURRENT_TIMESTAMP))\n RETURNS tstzrange\n LANGUAGE plpgsql\n STABLE PARALLEL SAFE STRICT\n SET \"TimeZone\" TO 'UTC'\nAS $function$\nDECLARE\n timestrs text[];\n s timestamptz;\n e timestamptz;\nBEGIN\n timestrs :=\n CASE\n WHEN _indate ? 'timestamp' THEN\n ARRAY[_indate->>'timestamp']\n WHEN _indate ? 'interval' THEN\n to_text_array(_indate->'interval')\n WHEN jsonb_typeof(_indate) = 'array' THEN\n to_text_array(_indate)\n ELSE\n regexp_split_to_array(\n _indate->>0,\n '/'\n )\n END;\n RAISE NOTICE 'TIMESTRS %', timestrs;\n IF cardinality(timestrs) = 1 THEN\n IF timestrs[1] ILIKE 'P%' THEN\n RETURN tstzrange(relative_base - upper(timestrs[1])::interval, relative_base, '[)');\n END IF;\n s := timestrs[1]::timestamptz;\n RETURN tstzrange(s, s, '[]');\n END IF;\n\n IF cardinality(timestrs) != 2 THEN\n RAISE EXCEPTION 'Timestamp cannot have more than 2 values';\n END IF;\n\n IF timestrs[1] = '..' OR timestrs[1] = '' THEN\n s := '-infinity'::timestamptz;\n e := timestrs[2]::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] = '..' OR timestrs[2] = '' THEN\n s := timestrs[1]::timestamptz;\n e := 'infinity'::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[1] ILIKE 'P%' AND timestrs[2] NOT ILIKE 'P%' THEN\n e := timestrs[2]::timestamptz;\n s := e - upper(timestrs[1])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] ILIKE 'P%' AND timestrs[1] NOT ILIKE 'P%' THEN\n s := timestrs[1]::timestamptz;\n e := s + upper(timestrs[2])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n s := timestrs[1]::timestamptz;\n e := timestrs[2]::timestamptz;\n\n RETURN tstzrange(s,e,'[)');\n\n RETURN NULL;\n\nEND;\n$function$\n;\n\ncreate or replace view \"pgstac\".\"partition_steps\" as SELECT partitions.partition AS name,\n date_trunc('month'::text, lower(partitions.partition_dtrange)) AS sdate,\n (date_trunc('month'::text, upper(partitions.partition_dtrange)) + '1 mon'::interval) AS edate\n FROM pgstac.partitions\n WHERE ((partitions.partition_dtrange IS NOT NULL) AND (partitions.partition_dtrange <> 'empty'::tstzrange))\n ORDER BY partitions.dtrange;\n\n\nCREATE OR REPLACE FUNCTION pgstac.queryables_trigger_func()\n RETURNS trigger\n LANGUAGE plpgsql\nAS $function$\nDECLARE\nBEGIN\n PERFORM maintain_partitions();\n RETURN NULL;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.search(_search jsonb DEFAULT '{}'::jsonb)\n RETURNS jsonb\n LANGUAGE plpgsql\n SET search_path TO 'pgstac', 'public'\n SET cursor_tuple_fraction TO '1'\nAS $function$\nDECLARE\n searches searches%ROWTYPE;\n _where text;\n token_where text;\n full_where text;\n orderby text;\n query text;\n token_type text := substr(_search->>'token',1,4);\n _limit int := coalesce((_search->>'limit')::int, 10);\n curs refcursor;\n cntr int := 0;\n iter_record items%ROWTYPE;\n first_record jsonb;\n first_item items%ROWTYPE;\n last_item items%ROWTYPE;\n last_record jsonb;\n out_records jsonb := '[]'::jsonb;\n prev_query text;\n next text;\n prev_id text;\n has_next boolean := false;\n has_prev boolean := false;\n prev text;\n total_count bigint;\n context jsonb;\n collection jsonb;\n includes text[];\n excludes text[];\n exit_flag boolean := FALSE;\n batches int := 0;\n timer timestamptz := clock_timestamp();\n pstart timestamptz;\n pend timestamptz;\n pcurs refcursor;\n search_where search_wheres%ROWTYPE;\n id text;\nBEGIN\nCREATE TEMP TABLE results (i int GENERATED ALWAYS AS IDENTITY, content jsonb) ON COMMIT DROP;\n-- if ids is set, short circuit and just use direct ids query for each id\n-- skip any paging or caching\n-- hard codes ordering in the same order as the array of ids\nIF _search ? 'ids' THEN\n INSERT INTO results (content)\n SELECT\n CASE WHEN _search->'conf'->>'nohydrate' IS NOT NULL AND (_search->'conf'->>'nohydrate')::boolean = true THEN\n content_nonhydrated(items, _search->'fields')\n ELSE\n content_hydrate(items, _search->'fields')\n END\n FROM items WHERE\n items.id = ANY(to_text_array(_search->'ids'))\n AND\n CASE WHEN _search ? 'collections' THEN\n items.collection = ANY(to_text_array(_search->'collections'))\n ELSE TRUE\n END\n ORDER BY items.datetime desc, items.id desc\n ;\n SELECT INTO total_count count(*) FROM results;\nELSE\n searches := search_query(_search);\n _where := searches._where;\n orderby := searches.orderby;\n search_where := where_stats(_where);\n total_count := coalesce(search_where.total_count, search_where.estimated_count);\n\n IF token_type='prev' THEN\n token_where := get_token_filter(_search, null::jsonb);\n orderby := sort_sqlorderby(_search, TRUE);\n END IF;\n IF token_type='next' THEN\n token_where := get_token_filter(_search, null::jsonb);\n END IF;\n\n full_where := concat_ws(' AND ', _where, token_where);\n RAISE NOTICE 'FULL QUERY % %', full_where, clock_timestamp()-timer;\n timer := clock_timestamp();\n\n FOR query IN SELECT partition_queries(full_where, orderby, search_where.partitions) LOOP\n timer := clock_timestamp();\n query := format('%s LIMIT %s', query, _limit + 1);\n RAISE NOTICE 'Partition Query: %', query;\n batches := batches + 1;\n -- curs = create_cursor(query);\n RAISE NOTICE 'cursor_tuple_fraction: %', current_setting('cursor_tuple_fraction');\n OPEN curs FOR EXECUTE query;\n LOOP\n FETCH curs into iter_record;\n EXIT WHEN NOT FOUND;\n cntr := cntr + 1;\n\n IF _search->'conf'->>'nohydrate' IS NOT NULL AND (_search->'conf'->>'nohydrate')::boolean = true THEN\n last_record := content_nonhydrated(iter_record, _search->'fields');\n ELSE\n last_record := content_hydrate(iter_record, _search->'fields');\n END IF;\n last_item := iter_record;\n IF cntr = 1 THEN\n first_item := last_item;\n first_record := last_record;\n END IF;\n IF cntr <= _limit THEN\n INSERT INTO results (content) VALUES (last_record);\n ELSIF cntr > _limit THEN\n has_next := true;\n exit_flag := true;\n EXIT;\n END IF;\n END LOOP;\n CLOSE curs;\n RAISE NOTICE 'Query took %.', clock_timestamp()-timer;\n timer := clock_timestamp();\n EXIT WHEN exit_flag;\n END LOOP;\n RAISE NOTICE 'Scanned through % partitions.', batches;\nEND IF;\n\nWITH ordered AS (SELECT * FROM results WHERE content IS NOT NULL ORDER BY i)\nSELECT jsonb_agg(content) INTO out_records FROM ordered;\n\nDROP TABLE results;\n\n\n-- Flip things around if this was the result of a prev token query\nIF token_type='prev' THEN\n out_records := flip_jsonb_array(out_records);\n first_item := last_item;\n first_record := last_record;\nEND IF;\n\n-- If this query has a token, see if there is data before the first record\nIF _search ? 'token' THEN\n prev_query := format(\n 'SELECT 1 FROM items WHERE %s LIMIT 1',\n concat_ws(\n ' AND ',\n _where,\n trim(get_token_filter(_search, to_jsonb(first_item)))\n )\n );\n RAISE NOTICE 'Query to get previous record: % --- %', prev_query, first_record;\n EXECUTE prev_query INTO has_prev;\n IF FOUND and has_prev IS NOT NULL THEN\n RAISE NOTICE 'Query results from prev query: %', has_prev;\n has_prev := TRUE;\n END IF;\nEND IF;\nhas_prev := COALESCE(has_prev, FALSE);\n\nIF has_prev THEN\n prev := out_records->0->>'id';\nEND IF;\nIF has_next OR token_type='prev' THEN\n next := out_records->-1->>'id';\nEND IF;\n\nIF context(_search->'conf') != 'off' THEN\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'matched', total_count,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\nELSE\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\nEND IF;\n\ncollection := jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb),\n 'next', next,\n 'prev', prev,\n 'context', context\n);\n\nRETURN collection;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.search_query(_search jsonb DEFAULT '{}'::jsonb, updatestats boolean DEFAULT false, _metadata jsonb DEFAULT '{}'::jsonb)\n RETURNS pgstac.searches\n LANGUAGE plpgsql\n SECURITY DEFINER\nAS $function$\nDECLARE\n search searches%ROWTYPE;\n pexplain jsonb;\n t timestamptz;\n i interval;\nBEGIN\n SELECT * INTO search FROM searches\n WHERE hash=search_hash(_search, _metadata) FOR UPDATE;\n\n -- Calculate the where clause if not already calculated\n IF search._where IS NULL THEN\n search._where := stac_search_to_where(_search);\n END IF;\n\n -- Calculate the order by clause if not already calculated\n IF search.orderby IS NULL THEN\n search.orderby := sort_sqlorderby(_search);\n END IF;\n\n PERFORM where_stats(search._where, updatestats, _search->'conf');\n\n search.lastused := now();\n search.usecount := coalesce(search.usecount, 0) + 1;\n INSERT INTO searches (search, _where, orderby, lastused, usecount, metadata)\n VALUES (_search, search._where, search.orderby, search.lastused, search.usecount, _metadata)\n ON CONFLICT (hash) DO\n UPDATE SET\n _where = EXCLUDED._where,\n orderby = EXCLUDED.orderby,\n lastused = EXCLUDED.lastused,\n usecount = EXCLUDED.usecount,\n metadata = EXCLUDED.metadata\n RETURNING * INTO search\n ;\n RETURN search;\n\nEND;\n$function$\n;\n\nCREATE OR REPLACE PROCEDURE pgstac.validate_constraints()\n LANGUAGE plpgsql\nAS $procedure$\nDECLARE\n q text;\nBEGIN\n FOR q IN\n SELECT\n FORMAT(\n 'ALTER TABLE %I.%I VALIDATE CONSTRAINT %I;',\n nsp.nspname,\n cls.relname,\n con.conname\n )\n\n FROM pg_constraint AS con\n JOIN pg_class AS cls\n ON con.conrelid = cls.oid\n JOIN pg_namespace AS nsp\n ON cls.relnamespace = nsp.oid\n WHERE convalidated = FALSE AND contype in ('c','f')\n AND nsp.nspname = 'pgstac'\n LOOP\n RAISE NOTICE '%', q;\n PERFORM run_or_queue(q);\n COMMIT;\n END LOOP;\nEND;\n$procedure$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.where_stats(inwhere text, updatestats boolean DEFAULT false, conf jsonb DEFAULT NULL::jsonb)\n RETURNS pgstac.search_wheres\n LANGUAGE plpgsql\n SECURITY DEFINER\nAS $function$\nDECLARE\n t timestamptz;\n i interval;\n explain_json jsonb;\n partitions text[];\n sw search_wheres%ROWTYPE;\n inwhere_hash text := md5(inwhere);\n _context text := lower(context(conf));\n _stats_ttl interval := context_stats_ttl(conf);\n _estimated_cost float := context_estimated_cost(conf);\n _estimated_count int := context_estimated_count(conf);\nBEGIN\n IF _context = 'off' THEN\n sw._where := inwhere;\n return sw;\n END IF;\n\n SELECT * INTO sw FROM search_wheres WHERE md5(_where)=inwhere_hash FOR UPDATE;\n\n -- Update statistics if explicitly set, if statistics do not exist, or statistics ttl has expired\n IF NOT updatestats THEN\n RAISE NOTICE 'Checking if update is needed for: % .', inwhere;\n RAISE NOTICE 'Stats Last Updated: %', sw.statslastupdated;\n RAISE NOTICE 'TTL: %, Age: %', _stats_ttl, now() - sw.statslastupdated;\n RAISE NOTICE 'Context: %, Existing Total: %', _context, sw.total_count;\n IF\n sw.statslastupdated IS NULL\n OR (now() - sw.statslastupdated) > _stats_ttl\n OR (context(conf) != 'off' AND sw.total_count IS NULL)\n THEN\n updatestats := TRUE;\n END IF;\n END IF;\n\n sw._where := inwhere;\n sw.lastused := now();\n sw.usecount := coalesce(sw.usecount,0) + 1;\n\n IF NOT updatestats THEN\n UPDATE search_wheres SET\n lastused = sw.lastused,\n usecount = sw.usecount\n WHERE md5(_where) = inwhere_hash\n RETURNING * INTO sw\n ;\n RETURN sw;\n END IF;\n\n -- Use explain to get estimated count/cost and a list of the partitions that would be hit by the query\n t := clock_timestamp();\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s', inwhere)\n INTO explain_json;\n RAISE NOTICE 'Time for just the explain: %', clock_timestamp() - t;\n i := clock_timestamp() - t;\n\n sw.statslastupdated := now();\n sw.estimated_count := explain_json->0->'Plan'->'Plan Rows';\n sw.estimated_cost := explain_json->0->'Plan'->'Total Cost';\n sw.time_to_estimate := extract(epoch from i);\n\n RAISE NOTICE 'ESTIMATED_COUNT: % < %', sw.estimated_count, _estimated_count;\n RAISE NOTICE 'ESTIMATED_COST: % < %', sw.estimated_cost, _estimated_cost;\n\n -- Do a full count of rows if context is set to on or if auto is set and estimates are low enough\n IF\n _context = 'on'\n OR\n ( _context = 'auto' AND\n (\n sw.estimated_count < _estimated_count\n AND\n sw.estimated_cost < _estimated_cost\n )\n )\n THEN\n t := clock_timestamp();\n RAISE NOTICE 'Calculating actual count...';\n EXECUTE format(\n 'SELECT count(*) FROM items WHERE %s',\n inwhere\n ) INTO sw.total_count;\n i := clock_timestamp() - t;\n RAISE NOTICE 'Actual Count: % -- %', sw.total_count, i;\n sw.time_to_count := extract(epoch FROM i);\n ELSE\n sw.total_count := NULL;\n sw.time_to_count := NULL;\n END IF;\n\n\n INSERT INTO search_wheres\n (_where, lastused, usecount, statslastupdated, estimated_count, estimated_cost, time_to_estimate, partitions, total_count, time_to_count)\n SELECT sw._where, sw.lastused, sw.usecount, sw.statslastupdated, sw.estimated_count, sw.estimated_cost, sw.time_to_estimate, sw.partitions, sw.total_count, sw.time_to_count\n ON CONFLICT ((md5(_where)))\n DO UPDATE\n SET\n lastused = sw.lastused,\n usecount = sw.usecount,\n statslastupdated = sw.statslastupdated,\n estimated_count = sw.estimated_count,\n estimated_cost = sw.estimated_cost,\n time_to_estimate = sw.time_to_estimate,\n total_count = sw.total_count,\n time_to_count = sw.time_to_count\n ;\n RETURN sw;\nEND;\n$function$\n;\n\nCREATE TRIGGER items_after_delete_trigger AFTER UPDATE ON pgstac.items REFERENCING NEW TABLE AS newdata FOR EACH STATEMENT EXECUTE FUNCTION pgstac.partition_after_triggerfunc();\n\nCREATE TRIGGER items_after_insert_trigger AFTER INSERT ON pgstac.items REFERENCING NEW TABLE AS newdata FOR EACH STATEMENT EXECUTE FUNCTION pgstac.partition_after_triggerfunc();\n\nCREATE TRIGGER items_after_update_trigger AFTER DELETE ON pgstac.items REFERENCING OLD TABLE AS newdata FOR EACH STATEMENT EXECUTE FUNCTION pgstac.partition_after_triggerfunc();\n\n\n\n-- END migra calculated SQL\nINSERT INTO queryables (name, definition) VALUES\n('id', '{\"title\": \"Item ID\",\"description\": \"Item identifier\",\"$ref\": \"https://schemas.stacspec.org/v1.0.0/item-spec/json-schema/item.json#/definitions/core/allOf/2/properties/id\"}'),\n('datetime','{\"description\": \"Datetime\",\"type\": \"string\",\"title\": \"Acquired\",\"format\": \"date-time\",\"pattern\": \"(\\\\+00:00|Z)$\"}'),\n('geometry', '{\"title\": \"Item Geometry\",\"description\": \"Item Geometry\",\"$ref\": \"https://geojson.org/schema/Feature.json\"}')\nON CONFLICT DO NOTHING;\n\nINSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n('eo:cloud_cover','{\"$ref\": \"https://stac-extensions.github.io/eo/v1.0.0/schema.json#/definitions/fieldsproperties/eo:cloud_cover\"}','to_int','BTREE')\nON CONFLICT DO NOTHING;\n\n\nINSERT INTO pgstac_settings (name, value) VALUES\n ('context', 'off'),\n ('context_estimated_count', '100000'),\n ('context_estimated_cost', '100000'),\n ('context_stats_ttl', '1 day'),\n ('default_filter_lang', 'cql2-json'),\n ('additional_properties', 'true'),\n ('use_queue', 'false'),\n ('queue_timeout', '10 minutes'),\n ('update_collection_extent', 'true')\nON CONFLICT DO NOTHING\n;\n\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nGRANT ALL ON SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac to pgstac_admin;\n\n-- pgstac_read role limited to using function apis\nGRANT EXECUTE ON FUNCTION search TO pgstac_read;\nGRANT EXECUTE ON FUNCTION search_query TO pgstac_read;\nGRANT EXECUTE ON FUNCTION item_by_id TO pgstac_read;\nGRANT EXECUTE ON FUNCTION get_item TO pgstac_read;\n\nGRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_ingest;\nGRANT USAGE ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_ingest;\n\nSELECT update_partition_stats_q(partition) FROM partitions;\nSELECT set_version('0.7.0');\n" }, { "path": "src/pgstac/migrations/pgstac.0.6.13-0.7.3.sql", "content": "RESET ROLE;\nDO $$\nDECLARE\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='postgis') THEN\n CREATE EXTENSION IF NOT EXISTS postgis;\n END IF;\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='btree_gist') THEN\n CREATE EXTENSION IF NOT EXISTS btree_gist;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_admin;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_read;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_ingest;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n\nGRANT pgstac_admin TO current_user;\n\n-- Function to make sure pgstac_admin is the owner of items\nCREATE OR REPLACE FUNCTION pgstac_admin_owns() RETURNS VOID AS $$\nDECLARE\n f RECORD;\nBEGIN\n FOR f IN (\n SELECT\n concat(\n oid::regproc::text,\n '(',\n coalesce(pg_get_function_identity_arguments(oid),''),\n ')'\n ) AS name,\n CASE prokind WHEN 'f' THEN 'FUNCTION' WHEN 'p' THEN 'PROCEDURE' WHEN 'a' THEN 'AGGREGATE' END as typ\n FROM pg_proc\n WHERE\n pronamespace=to_regnamespace('pgstac')\n AND proowner != to_regrole('pgstac_admin')\n AND proname NOT LIKE 'pg_stat%'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n FOR f IN (\n SELECT\n oid::regclass::text as name,\n CASE relkind\n WHEN 'i' THEN 'INDEX'\n WHEN 'I' THEN 'INDEX'\n WHEN 'p' THEN 'TABLE'\n WHEN 'r' THEN 'TABLE'\n WHEN 'v' THEN 'VIEW'\n WHEN 'S' THEN 'SEQUENCE'\n ELSE NULL\n END as typ\n FROM pg_class\n WHERE relnamespace=to_regnamespace('pgstac') and relowner != to_regrole('pgstac_admin') AND relkind IN ('r','p','v','S') AND relname NOT LIKE 'pg_stat'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\nSELECT pgstac_admin_owns();\n\nCREATE SCHEMA IF NOT EXISTS pgstac AUTHORIZATION pgstac_admin;\n\nGRANT ALL ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_admin;\n\nALTER ROLE pgstac_admin SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_read SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_ingest SET SEARCH_PATH TO pgstac, public;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\n\nGRANT pgstac_read TO pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\n\nSET ROLE pgstac_admin;\n\nSET SEARCH_PATH TO pgstac, public;\n\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS analyze_items;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS validate_constraints;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\nSET client_min_messages TO WARNING;\nSET SEARCH_PATH to pgstac, public;\n-- BEGIN migra calculated SQL\ndrop trigger if exists \"queryables_collection_trigger\" on \"pgstac\".\"collections\";\n\ndrop trigger if exists \"partitions_delete_trigger\" on \"pgstac\".\"partitions\";\n\ndrop trigger if exists \"partitions_trigger\" on \"pgstac\".\"partitions\";\n\nalter table \"pgstac\".\"partitions\" drop constraint \"partitions_collection_fkey\";\n\nalter table \"pgstac\".\"partitions\" drop constraint \"prange\";\n\ndrop function if exists \"pgstac\".\"create_queryable_indexes\"();\n\ndrop function if exists \"pgstac\".\"partition_collection\"(collection text, strategy pgstac.partition_trunc_strategy);\n\ndrop function if exists \"pgstac\".\"partitions_delete_trigger_func\"();\n\ndrop function if exists \"pgstac\".\"partitions_trigger_func\"();\n\ndrop view if exists \"pgstac\".\"partition_steps\";\n\nalter table \"pgstac\".\"partitions\" drop constraint \"partitions_pkey\";\n\ndrop index if exists \"pgstac\".\"partitions_pkey\";\n\nselect 1; -- drop index if exists \"pgstac\".\"prange\";\n\ndrop index if exists \"pgstac\".\"partitions_range_idx\";\n\ndrop table \"pgstac\".\"partitions\";\n\ncreate table \"pgstac\".\"partition_stats\" (\n \"partition\" text not null,\n \"dtrange\" tstzrange,\n \"edtrange\" tstzrange,\n \"spatial\" geometry,\n \"last_updated\" timestamp with time zone,\n \"keys\" text[]\n);\n\n\ncreate table \"pgstac\".\"query_queue\" (\n \"query\" text not null,\n \"added\" timestamp with time zone default now()\n);\n\n\ncreate table \"pgstac\".\"query_queue_history\" (\n \"query\" text,\n \"added\" timestamp with time zone not null,\n \"finished\" timestamp with time zone not null default now(),\n \"error\" text\n);\n\n\nalter table \"pgstac\".\"collections\" alter column \"partition_trunc\" set data type text using \"partition_trunc\"::text;\n\ndrop type \"pgstac\".\"partition_trunc_strategy\";\n\nCREATE UNIQUE INDEX partition_stats_pkey ON pgstac.partition_stats USING btree (partition);\n\nCREATE UNIQUE INDEX query_queue_pkey ON pgstac.query_queue USING btree (query);\n\nCREATE INDEX queryables_collection_idx ON pgstac.queryables USING gin (collection_ids);\n\nCREATE INDEX partitions_range_idx ON pgstac.partition_stats USING gist (dtrange);\n\nalter table \"pgstac\".\"partition_stats\" add constraint \"partition_stats_pkey\" PRIMARY KEY using index \"partition_stats_pkey\";\n\nalter table \"pgstac\".\"query_queue\" add constraint \"query_queue_pkey\" PRIMARY KEY using index \"query_queue_pkey\";\n\nalter table \"pgstac\".\"collections\" add constraint \"collections_partition_trunc_check\" CHECK ((partition_trunc = ANY (ARRAY['year'::text, 'month'::text]))) not valid;\n\nalter table \"pgstac\".\"collections\" validate constraint \"collections_partition_trunc_check\";\n\nset check_function_bodies = off;\n\nCREATE OR REPLACE FUNCTION pgstac.check_partition(_collection text, _dtrange tstzrange, _edtrange tstzrange)\n RETURNS text\n LANGUAGE plpgsql\n SECURITY DEFINER\nAS $function$\nDECLARE\n c RECORD;\n pm RECORD;\n _partition_name text;\n _partition_dtrange tstzrange;\n _constraint_dtrange tstzrange;\n _constraint_edtrange tstzrange;\n q text;\n deferrable_q text;\n err_context text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=_collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', _collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n\n IF c.partition_trunc IS NOT NULL THEN\n _partition_dtrange := tstzrange(\n date_trunc(c.partition_trunc, lower(_dtrange)),\n date_trunc(c.partition_trunc, lower(_dtrange)) + (concat('1 ', c.partition_trunc))::interval,\n '[)'\n );\n ELSE\n _partition_dtrange := '[-infinity, infinity]'::tstzrange;\n END IF;\n\n IF NOT _partition_dtrange @> _dtrange THEN\n RAISE EXCEPTION 'dtrange % is greater than the partition size % for collection %', _dtrange, c.partition_trunc, _collection;\n END IF;\n\n\n IF c.partition_trunc = 'year' THEN\n _partition_name := format('_items_%s_%s', c.key, to_char(lower(_partition_dtrange),'YYYY'));\n ELSIF c.partition_trunc = 'month' THEN\n _partition_name := format('_items_%s_%s', c.key, to_char(lower(_partition_dtrange),'YYYYMM'));\n ELSE\n _partition_name := format('_items_%s', c.key);\n END IF;\n\n SELECT * INTO pm FROM partition_sys_meta WHERE collection=_collection AND partition_dtrange @> _dtrange;\n IF FOUND THEN\n RAISE NOTICE '% % %', _edtrange, _dtrange, pm;\n _constraint_edtrange :=\n tstzrange(\n least(\n lower(_edtrange),\n nullif(lower(pm.constraint_edtrange), '-infinity')\n ),\n greatest(\n upper(_edtrange),\n nullif(upper(pm.constraint_edtrange), 'infinity')\n ),\n '[]'\n );\n _constraint_dtrange :=\n tstzrange(\n least(\n lower(_dtrange),\n nullif(lower(pm.constraint_dtrange), '-infinity')\n ),\n greatest(\n upper(_dtrange),\n nullif(upper(pm.constraint_dtrange), 'infinity')\n ),\n '[]'\n );\n\n IF pm.constraint_edtrange @> _edtrange AND pm.constraint_dtrange @> _dtrange THEN\n RETURN pm.partition;\n ELSE\n PERFORM drop_table_constraints(_partition_name);\n END IF;\n ELSE\n _constraint_edtrange := _edtrange;\n _constraint_dtrange := _dtrange;\n END IF;\n RAISE NOTICE 'Creating partition % %', _partition_name, _partition_dtrange;\n IF c.partition_trunc IS NULL THEN\n q := format(\n $q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n $q$,\n _partition_name,\n _collection,\n concat(_partition_name,'_pk'),\n _partition_name\n );\n ELSE\n q := format(\n $q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L) PARTITION BY RANGE (datetime);\n CREATE TABLE IF NOT EXISTS %I partition OF %I FOR VALUES FROM (%L) TO (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n $q$,\n format('_items_%s', c.key),\n _collection,\n _partition_name,\n format('_items_%s', c.key),\n lower(_partition_dtrange),\n upper(_partition_dtrange),\n format('%s_pk', _partition_name),\n _partition_name\n );\n END IF;\n\n BEGIN\n EXECUTE q;\n EXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', _partition_name;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\n END;\n PERFORM create_table_constraints(_partition_name, _constraint_dtrange, _constraint_edtrange);\n PERFORM maintain_partitions(_partition_name);\n PERFORM update_partition_stats_q(_partition_name, true);\n RETURN _partition_name;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.collection_extent(_collection text, runupdate boolean DEFAULT false)\n RETURNS jsonb\n LANGUAGE plpgsql\nAS $function$\nDECLARE\n geom_extent geometry;\n mind timestamptz;\n maxd timestamptz;\n extent jsonb;\nBEGIN\n IF runupdate THEN\n PERFORM update_partition_stats_q(partition)\n FROM partitions WHERE collection=_collection;\n END IF;\n SELECT\n min(lower(dtrange)),\n max(upper(edtrange)),\n st_extent(spatial)\n INTO\n mind,\n maxd,\n geom_extent\n FROM partitions\n WHERE collection=_collection;\n\n IF geom_extent IS NOT NULL AND mind IS NOT NULL AND maxd IS NOT NULL THEN\n extent := jsonb_build_object(\n 'extent', jsonb_build_object(\n 'spatial', jsonb_build_object(\n 'bbox', to_jsonb(array[array[st_xmin(geom_extent), st_ymin(geom_extent), st_xmax(geom_extent), st_ymax(geom_extent)]])\n ),\n 'temporal', jsonb_build_object(\n 'interval', to_jsonb(array[array[mind, maxd]])\n )\n )\n );\n RETURN extent;\n END IF;\n RETURN NULL;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.constraint_tstzrange(expr text)\n RETURNS tstzrange\n LANGUAGE sql\n IMMUTABLE PARALLEL SAFE STRICT\nAS $function$\n WITH t AS (\n SELECT regexp_matches(\n expr,\n E'\\\\(''\\([0-9 :+-]*\\)''\\\\).*\\\\(''\\([0-9 :+-]*\\)''\\\\)'\n ) AS m\n ) SELECT tstzrange(m[1]::timestamptz, m[2]::timestamptz) FROM t\n ;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.create_table_constraints(t text, _dtrange tstzrange, _edtrange tstzrange)\n RETURNS text\n LANGUAGE plpgsql\n SECURITY DEFINER\nAS $function$\nDECLARE\n q text;\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM partitions WHERE partition=t) THEN\n RETURN NULL;\n END IF;\n RAISE NOTICE 'Creating Table Constraints for % % %', t, _dtrange, _edtrange;\n IF _dtrange = 'empty' AND _edtrange = 'empty' THEN\n q :=format(\n $q$\n DO $block$\n BEGIN\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n ALTER TABLE %I\n ADD CONSTRAINT %I\n CHECK (((datetime IS NULL) AND (end_datetime IS NULL))) NOT VALID\n ;\n ALTER TABLE %I\n VALIDATE CONSTRAINT %I\n ;\n\n EXCEPTION WHEN others THEN\n RAISE WARNING '%%, Issue Altering Constraints. Please run update_partition_stats(%I)', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n $block$;\n $q$,\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t),\n t\n );\n ELSE\n q :=format(\n $q$\n DO $block$\n BEGIN\n\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n ALTER TABLE %I\n ADD CONSTRAINT %I\n CHECK (\n (datetime >= %L)\n AND (datetime <= %L)\n AND (end_datetime >= %L)\n AND (end_datetime <= %L)\n ) NOT VALID\n ;\n ALTER TABLE %I\n VALIDATE CONSTRAINT %I\n ;\n\n EXCEPTION WHEN others THEN\n RAISE WARNING '%%, Issue Altering Constraints. Please run update_partition_stats(%I)', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n $block$;\n $q$,\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t),\n lower(_dtrange),\n upper(_dtrange),\n lower(_edtrange),\n upper(_edtrange),\n t,\n format('%s_dt', t),\n t\n );\n END IF;\n PERFORM run_or_queue(q);\n RETURN t;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.drop_table_constraints(t text)\n RETURNS text\n LANGUAGE plpgsql\n SECURITY DEFINER\nAS $function$\nDECLARE\n q text;\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM partitions WHERE partition=t) THEN\n RETURN NULL;\n END IF;\n FOR q IN SELECT FORMAT(\n $q$\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n $q$,\n t,\n conname\n ) FROM pg_constraint\n WHERE conrelid=t::regclass::oid AND contype='c'\n LOOP\n EXECUTE q;\n END LOOP;\n RETURN t;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.dt_constraint(coid oid, OUT dt tstzrange, OUT edt tstzrange)\n RETURNS record\n LANGUAGE plpgsql\n STABLE STRICT\nAS $function$\nDECLARE\n expr text := pg_get_constraintdef(coid);\n matches timestamptz[];\nBEGIN\n IF expr LIKE '%NULL%' THEN\n dt := tstzrange(null::timestamptz, null::timestamptz);\n edt := tstzrange(null::timestamptz, null::timestamptz);\n RETURN;\n END IF;\n WITH f AS (SELECT (regexp_matches(expr, E'([0-9]{4}-[0-1][0-9]-[0-3][0-9] [0-2][0-9]:[0-5][0-9]:[0-5][0-9]\\.?[0-9]*)', 'g'))[1] f)\n SELECT array_agg(f::timestamptz) INTO matches FROM f;\n IF cardinality(matches) = 4 THEN\n dt := tstzrange(matches[1], matches[2],'[]');\n edt := tstzrange(matches[3], matches[4], '[]');\n RETURN;\n ELSIF cardinality(matches) = 2 THEN\n edt := tstzrange(matches[1], matches[2],'[]');\n RETURN;\n END IF;\n RETURN;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.get_setting_bool(_setting text, conf jsonb DEFAULT NULL::jsonb)\n RETURNS boolean\n LANGUAGE sql\nAS $function$\nSELECT COALESCE(\n conf->>_setting,\n current_setting(concat('pgstac.',_setting), TRUE),\n (SELECT value FROM pgstac.pgstac_settings WHERE name=_setting),\n 'FALSE'\n)::boolean;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.indexdef(q pgstac.queryables)\n RETURNS text\n LANGUAGE plpgsql\n IMMUTABLE\nAS $function$\n DECLARE\n out text;\n BEGIN\n IF q.name = 'id' THEN\n out := 'CREATE UNIQUE INDEX ON %I USING btree (id)';\n ELSIF q.name = 'datetime' THEN\n out := 'CREATE INDEX ON %I USING btree (datetime DESC, end_datetime)';\n ELSIF q.name = 'geometry' THEN\n out := 'CREATE INDEX ON %I USING gist (geometry)';\n ELSE\n out := format($q$CREATE INDEX ON %%I USING %s (%s(((content -> 'properties'::text) -> %L::text)))$q$,\n lower(COALESCE(q.property_index_type, 'BTREE')),\n lower(COALESCE(q.property_wrapper, 'to_text')),\n q.name\n );\n END IF;\n RETURN btrim(out, ' \\n\\t');\n END;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.maintain_partition_queries(part text DEFAULT 'items'::text, dropindexes boolean DEFAULT false, rebuildindexes boolean DEFAULT false, idxconcurrently boolean DEFAULT false)\n RETURNS SETOF text\n LANGUAGE plpgsql\nAS $function$\nDECLARE\n rec record;\nBEGIN\n FOR rec IN (\n WITH p AS (\n SELECT\n relid::text as partition,\n replace(replace(\n CASE\n WHEN level = 1 THEN pg_get_expr(c.relpartbound, c.oid)\n ELSE pg_get_expr(parent.relpartbound, parent.oid)\n END,\n 'FOR VALUES IN (''',''), ''')',\n ''\n ) AS collection\n FROM pg_partition_tree('items')\n JOIN pg_class c ON (relid::regclass = c.oid)\n JOIN pg_class parent ON (parentrelid::regclass = parent.oid AND isleaf)\n ), i AS (\n SELECT\n partition,\n indexname,\n regexp_replace(btrim(replace(replace(indexdef, indexname, ''),'pgstac.',''),' \\t\\n'), '[ ]+', ' ', 'g') as iidx,\n COALESCE(\n (regexp_match(indexdef, '\\(([a-zA-Z]+)\\)'))[1],\n (regexp_match(indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_-]+)''::text'))[1],\n CASE WHEN indexdef ~* '\\(datetime desc, end_datetime\\)' THEN 'datetime' ELSE NULL END\n ) AS field\n FROM\n pg_indexes\n JOIN p ON (tablename=partition)\n ), q AS (\n SELECT\n name AS field,\n collection,\n partition,\n format(indexdef(queryables), partition) as qidx\n FROM queryables, unnest_collection(queryables.collection_ids) collection\n JOIN p USING (collection)\n WHERE property_index_type IS NOT NULL OR name IN ('datetime','geometry','id')\n )\n SELECT * FROM i FULL JOIN q USING (field, partition)\n WHERE lower(iidx) IS DISTINCT FROM lower(qidx)\n ) LOOP\n IF rec.iidx IS NULL THEN\n IF idxconcurrently THEN\n RETURN NEXT replace(rec.qidx, 'INDEX', 'INDEX CONCURRENTLY');\n ELSE\n RETURN NEXT rec.qidx;\n END IF;\n ELSIF rec.qidx IS NULL AND dropindexes THEN\n RETURN NEXT format('DROP INDEX IF EXISTS %I;', rec.indexname);\n ELSIF lower(rec.qidx) != lower(rec.iidx) THEN\n IF dropindexes THEN\n RETURN NEXT format('DROP INDEX IF EXISTS %I; %s;', rec.indexname, rec.qidx);\n ELSE\n IF idxconcurrently THEN\n RETURN NEXT replace(rec.qidx, 'INDEX', 'INDEX CONCURRENTLY');\n ELSE\n RETURN NEXT rec.qidx;\n END IF;\n END IF;\n ELSIF rebuildindexes and rec.indexname IS NOT NULL THEN\n IF idxconcurrently THEN\n RETURN NEXT format('REINDEX INDEX CONCURRENTLY %I;', rec.indexname);\n ELSE\n RETURN NEXT format('REINDEX INDEX %I;', rec.indexname);\n END IF;\n END IF;\n END LOOP;\n RETURN;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.maintain_partitions(part text DEFAULT 'items'::text, dropindexes boolean DEFAULT false, rebuildindexes boolean DEFAULT false)\n RETURNS void\n LANGUAGE sql\nAS $function$\n WITH t AS (\n SELECT run_or_queue(q) FROM maintain_partition_queries(part, dropindexes, rebuildindexes) q\n ) SELECT count(*) FROM t;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.normalize_indexdef(def text)\n RETURNS text\n LANGUAGE plpgsql\n IMMUTABLE PARALLEL SAFE STRICT\nAS $function$\nDECLARE\nBEGIN\n def := btrim(def, ' \\n\\t');\n\tdef := regexp_replace(def, '^CREATE (UNIQUE )?INDEX ([^ ]* )?ON (ONLY )?([^ ]* )?', '', 'i');\n RETURN def;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.partition_after_triggerfunc()\n RETURNS trigger\n LANGUAGE plpgsql\n SECURITY DEFINER\nAS $function$\nDECLARE\n p text;\n t timestamptz := clock_timestamp();\nBEGIN\n RAISE NOTICE 'Updating partition stats %', t;\n FOR p IN SELECT DISTINCT partition\n FROM newdata n JOIN partition_sys_meta p\n ON (n.collection=p.collection AND n.datetime <@ p.partition_dtrange)\n LOOP\n PERFORM run_or_queue(format('SELECT update_partition_stats(%L, %L);', p, true));\n END LOOP;\n RAISE NOTICE 't: % %', t, clock_timestamp() - t;\n RETURN NULL;\nEND;\n$function$\n;\n\ncreate or replace view \"pgstac\".\"partition_sys_meta\" as SELECT (pg_partition_tree.relid)::text AS partition,\n replace(replace(\n CASE\n WHEN (pg_partition_tree.level = 1) THEN pg_get_expr(c.relpartbound, c.oid)\n ELSE pg_get_expr(parent.relpartbound, parent.oid)\n END, 'FOR VALUES IN ('''::text, ''::text), ''')'::text, ''::text) AS collection,\n pg_partition_tree.level,\n c.reltuples,\n c.relhastriggers,\n COALESCE(pgstac.constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity'::timestamp with time zone, 'infinity'::timestamp with time zone, '[]'::text)) AS partition_dtrange,\n COALESCE((pgstac.dt_constraint(edt.oid)).dt, pgstac.constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity'::timestamp with time zone, 'infinity'::timestamp with time zone, '[]'::text)) AS constraint_dtrange,\n COALESCE((pgstac.dt_constraint(edt.oid)).edt, tstzrange('-infinity'::timestamp with time zone, 'infinity'::timestamp with time zone, '[]'::text)) AS constraint_edtrange\n FROM (((pg_partition_tree('pgstac.items'::regclass) pg_partition_tree(relid, parentrelid, isleaf, level)\n JOIN pg_class c ON (((pg_partition_tree.relid)::oid = c.oid)))\n JOIN pg_class parent ON ((((pg_partition_tree.parentrelid)::oid = parent.oid) AND pg_partition_tree.isleaf)))\n LEFT JOIN pg_constraint edt ON (((edt.conrelid = c.oid) AND (edt.contype = 'c'::\"char\"))))\n WHERE pg_partition_tree.isleaf;\n\n\ncreate or replace view \"pgstac\".\"partitions\" as SELECT partition_sys_meta.partition,\n partition_sys_meta.collection,\n partition_sys_meta.level,\n partition_sys_meta.reltuples,\n partition_sys_meta.relhastriggers,\n partition_sys_meta.partition_dtrange,\n partition_sys_meta.constraint_dtrange,\n partition_sys_meta.constraint_edtrange,\n partition_stats.dtrange,\n partition_stats.edtrange,\n partition_stats.spatial,\n partition_stats.last_updated,\n partition_stats.keys\n FROM (pgstac.partition_sys_meta\n LEFT JOIN pgstac.partition_stats USING (partition));\n\n\ncreate or replace view \"pgstac\".\"pgstac_indexes\" as SELECT i.schemaname,\n i.tablename,\n i.indexname,\n regexp_replace(btrim(replace(replace(i.indexdef, (i.indexname)::text, ''::text), 'pgstac.'::text, ''::text), ' \\t\\n'::text), '[ ]+'::text, ' '::text, 'g'::text) AS idx,\n COALESCE((regexp_match(i.indexdef, '\\(([a-zA-Z]+)\\)'::text))[1], (regexp_match(i.indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_-]+)''::text'::text))[1],\n CASE\n WHEN (i.indexdef ~* '\\(datetime desc, end_datetime\\)'::text) THEN 'datetime'::text\n ELSE NULL::text\n END) AS field,\n pg_table_size(((i.indexname)::text)::regclass) AS index_size,\n pg_size_pretty(pg_table_size(((i.indexname)::text)::regclass)) AS index_size_pretty\n FROM pg_indexes i\n WHERE ((i.schemaname = 'pgstac'::name) AND (i.tablename ~ '_items_'::text) AND (i.indexdef !~* ' only '::text));\n\n\ncreate or replace view \"pgstac\".\"pgstac_indexes_stats\" as SELECT i.schemaname,\n i.tablename,\n i.indexname,\n i.indexdef,\n COALESCE((regexp_match(i.indexdef, '\\(([a-zA-Z]+)\\)'::text))[1], (regexp_match(i.indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_]+)''::text'::text))[1],\n CASE\n WHEN (i.indexdef ~* '\\(datetime desc, end_datetime\\)'::text) THEN 'datetime_end_datetime'::text\n ELSE NULL::text\n END) AS field,\n pg_table_size(((i.indexname)::text)::regclass) AS index_size,\n pg_size_pretty(pg_table_size(((i.indexname)::text)::regclass)) AS index_size_pretty,\n s.n_distinct,\n ((s.most_common_vals)::text)::text[] AS most_common_vals,\n ((s.most_common_freqs)::text)::text[] AS most_common_freqs,\n ((s.histogram_bounds)::text)::text[] AS histogram_bounds,\n s.correlation\n FROM (pg_indexes i\n LEFT JOIN pg_stats s ON ((s.tablename = i.indexname)))\n WHERE ((i.schemaname = 'pgstac'::name) AND (i.tablename ~ '_items_'::text));\n\n\nCREATE OR REPLACE FUNCTION pgstac.queryable_signature(n text, c text[])\n RETURNS text\n LANGUAGE sql\n IMMUTABLE PARALLEL SAFE\nAS $function$\n SELECT concat(n, c);\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.queryables_constraint_triggerfunc()\n RETURNS trigger\n LANGUAGE plpgsql\nAS $function$\nDECLARE\n allcollections text[];\nBEGIN\n RAISE NOTICE 'Making sure that name/collection is unique for queryables';\n IF NEW.collection_ids IS NOT NULL THEN\n IF EXISTS (\n SELECT 1 FROM\n collections\n LEFT JOIN\n unnest(NEW.collection_ids) c\n ON (collections.id = c)\n WHERE c IS NULL\n ) THEN\n RAISE foreign_key_violation;\n RETURN NULL;\n END IF;\n END IF;\n IF TG_OP = 'INSERT' THEN\n IF EXISTS (\n SELECT 1 FROM queryables q\n WHERE\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n )\n ) THEN\n RAISE unique_violation;\n RETURN NULL;\n END IF;\n END IF;\n IF TG_OP = 'UPDATE' THEN\n IF EXISTS (\n SELECT 1 FROM queryables q\n WHERE\n q.id != NEW.id\n AND\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n )\n ) THEN\n RAISE unique_violation\n USING MESSAGE = format(\n 'There is already a queryable for %s for a collection in %s',\n NEW.name,\n NEW.collection_ids\n );\n RETURN NULL;\n END IF;\n END IF;\n\n RETURN NEW;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.queue_timeout()\n RETURNS interval\n LANGUAGE sql\nAS $function$\n SELECT t2s(coalesce(\n get_setting('queue_timeout'),\n '1h'\n ))::interval;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.repartition(_collection text, _partition_trunc text, triggered boolean DEFAULT false)\n RETURNS text\n LANGUAGE plpgsql\n SECURITY DEFINER\nAS $function$\nDECLARE\n c RECORD;\n q text;\n from_trunc text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=_collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', _collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n IF triggered THEN\n RAISE NOTICE 'Converting % to % partitioning via Trigger', _collection, _partition_trunc;\n ELSE\n RAISE NOTICE 'Converting % from using % to % partitioning', _collection, c.partition_trunc, _partition_trunc;\n IF c.partition_trunc IS NOT DISTINCT FROM _partition_trunc THEN\n RAISE NOTICE 'Collection % already set to use partition by %', _collection, _partition_trunc;\n RETURN _collection;\n END IF;\n END IF;\n\n IF EXISTS (SELECT 1 FROM partitions WHERE collection=_collection LIMIT 1) THEN\n EXECUTE format(\n $q$\n CREATE TEMP TABLE changepartitionstaging ON COMMIT DROP AS SELECT * FROM %I;\n DROP TABLE IF EXISTS %I CASCADE;\n WITH p AS (\n SELECT\n collection,\n CASE WHEN %L IS NULL THEN '-infinity'::timestamptz\n ELSE date_trunc(%L, datetime)\n END as d,\n tstzrange(min(datetime),max(datetime),'[]') as dtrange,\n tstzrange(min(datetime),max(datetime),'[]') as edtrange\n FROM changepartitionstaging\n GROUP BY 1,2\n ) SELECT check_partition(collection, dtrange, edtrange) FROM p;\n INSERT INTO items SELECT * FROM changepartitionstaging;\n DROP TABLE changepartitionstaging;\n $q$,\n concat('_items_', c.key),\n concat('_items_', c.key),\n c.partition_trunc,\n c.partition_trunc\n );\n END IF;\n RETURN _collection;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.run_or_queue(query text)\n RETURNS void\n LANGUAGE plpgsql\nAS $function$\nDECLARE\n use_queue text := COALESCE(get_setting('use_queue'), 'FALSE')::boolean;\nBEGIN\n IF get_setting_bool('debug') THEN\n RAISE NOTICE '%', query;\n END IF;\n IF use_queue THEN\n INSERT INTO query_queue (query) VALUES (query) ON CONFLICT DO NOTHING;\n ELSE\n EXECUTE query;\n END IF;\n RETURN;\nEND;\n$function$\n;\n\nCREATE OR REPLACE PROCEDURE pgstac.run_queued_queries()\n LANGUAGE plpgsql\nAS $procedure$\nDECLARE\n qitem query_queue%ROWTYPE;\n timeout_ts timestamptz;\n error text;\n cnt int := 0;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n DELETE FROM query_queue WHERE query = (SELECT query FROM query_queue ORDER BY added DESC LIMIT 1 FOR UPDATE SKIP LOCKED) RETURNING * INTO qitem;\n IF NOT FOUND THEN\n EXIT;\n END IF;\n cnt := cnt + 1;\n BEGIN\n RAISE NOTICE 'RUNNING QUERY: %', qitem.query;\n EXECUTE qitem.query;\n EXCEPTION WHEN others THEN\n error := format('%s | %s', SQLERRM, SQLSTATE);\n END;\n INSERT INTO query_queue_history (query, added, finished, error)\n VALUES (qitem.query, qitem.added, clock_timestamp(), error);\n COMMIT;\n END LOOP;\nEND;\n$procedure$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.run_queued_queries_intransaction()\n RETURNS integer\n LANGUAGE plpgsql\nAS $function$\nDECLARE\n qitem query_queue%ROWTYPE;\n timeout_ts timestamptz;\n error text;\n cnt int := 0;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n DELETE FROM query_queue WHERE query = (SELECT query FROM query_queue ORDER BY added DESC LIMIT 1 FOR UPDATE SKIP LOCKED) RETURNING * INTO qitem;\n IF NOT FOUND THEN\n RETURN cnt;\n END IF;\n cnt := cnt + 1;\n BEGIN\n qitem.query := regexp_replace(qitem.query, 'CONCURRENTLY', '');\n RAISE NOTICE 'RUNNING QUERY: %', qitem.query;\n\n EXECUTE qitem.query;\n EXCEPTION WHEN others THEN\n error := format('%s | %s', SQLERRM, SQLSTATE);\n RAISE WARNING '%', error;\n END;\n INSERT INTO query_queue_history (query, added, finished, error)\n VALUES (qitem.query, qitem.added, clock_timestamp(), error);\n END LOOP;\n RETURN cnt;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.t2s(text)\n RETURNS text\n LANGUAGE sql\n IMMUTABLE PARALLEL SAFE STRICT\nAS $function$\n SELECT extract(epoch FROM $1::interval)::text || ' s';\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.unnest_collection(collection_ids text[] DEFAULT NULL::text[])\n RETURNS SETOF text\n LANGUAGE plpgsql\n STABLE\nAS $function$\n DECLARE\n BEGIN\n IF collection_ids IS NULL THEN\n RETURN QUERY SELECT id FROM collections;\n END IF;\n RETURN QUERY SELECT unnest(collection_ids);\n END;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.update_partition_stats(_partition text, istrigger boolean DEFAULT false)\n RETURNS void\n LANGUAGE plpgsql\n STRICT\nAS $function$\nDECLARE\n dtrange tstzrange;\n edtrange tstzrange;\n cdtrange tstzrange;\n cedtrange tstzrange;\n extent geometry;\n collection text;\nBEGIN\n RAISE NOTICE 'Updating stats for %.', _partition;\n EXECUTE format(\n $q$\n SELECT\n tstzrange(min(datetime), max(datetime),'[]'),\n tstzrange(min(end_datetime), max(end_datetime), '[]')\n FROM %I\n $q$,\n _partition\n ) INTO dtrange, edtrange;\n extent := st_estimatedextent('pgstac', _partition, 'geometry');\n INSERT INTO partition_stats (partition, dtrange, edtrange, spatial, last_updated)\n SELECT _partition, dtrange, edtrange, extent, now()\n ON CONFLICT (partition) DO\n UPDATE SET\n dtrange=EXCLUDED.dtrange,\n edtrange=EXCLUDED.edtrange,\n spatial=EXCLUDED.spatial,\n last_updated=EXCLUDED.last_updated\n ;\n SELECT\n constraint_dtrange, constraint_edtrange, partitions.collection\n INTO cdtrange, cedtrange, collection\n FROM partitions WHERE partition = _partition;\n\n RAISE NOTICE 'Checking if we need to modify constraints.';\n IF\n (cdtrange IS DISTINCT FROM dtrange OR edtrange IS DISTINCT FROM cedtrange)\n AND NOT istrigger\n THEN\n RAISE NOTICE 'Modifying Constraints';\n RAISE NOTICE 'Existing % %', cdtrange, cedtrange;\n RAISE NOTICE 'New % %', dtrange, edtrange;\n PERFORM drop_table_constraints(_partition);\n PERFORM create_table_constraints(_partition, dtrange, edtrange);\n END IF;\n RAISE NOTICE 'Checking if we need to update collection extents.';\n IF get_setting_bool('update_collection_extent') THEN\n RAISE NOTICE 'updating collection extent for %', collection;\n PERFORM run_or_queue(format($q$\n UPDATE collections\n SET content = jsonb_set_lax(\n content,\n '{extent}'::text[],\n collection_extent(%L),\n true,\n 'use_json_null'\n ) WHERE id=%L\n ;\n $q$, collection, collection));\n ELSE\n RAISE NOTICE 'Not updating collection extent for %', collection;\n END IF;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.update_partition_stats_q(_partition text, istrigger boolean DEFAULT false)\n RETURNS void\n LANGUAGE plpgsql\nAS $function$\nDECLARE\nBEGIN\n PERFORM run_or_queue(\n format('SELECT update_partition_stats(%L, %L);', _partition, istrigger)\n );\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.all_collections()\n RETURNS jsonb\n LANGUAGE sql\n SET search_path TO 'pgstac', 'public'\nAS $function$\n SELECT jsonb_agg(content) FROM collections;\n$function$\n;\n\nCREATE OR REPLACE PROCEDURE pgstac.analyze_items()\n LANGUAGE plpgsql\nAS $procedure$\nDECLARE\n q text;\n timeout_ts timestamptz;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n SELECT format('ANALYZE (VERBOSE, SKIP_LOCKED) %I;', relname) INTO q\n FROM pg_stat_user_tables\n WHERE relname like '_item%' AND (n_mod_since_analyze>0 OR last_analyze IS NULL) LIMIT 1;\n IF NOT FOUND THEN\n EXIT;\n END IF;\n RAISE NOTICE '%', q;\n EXECUTE q;\n COMMIT;\n END LOOP;\nEND;\n$procedure$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.check_pgstac_settings(_sysmem text DEFAULT NULL::text)\n RETURNS void\n LANGUAGE plpgsql\n SET search_path TO 'pgstac', 'public'\n SET client_min_messages TO 'notice'\nAS $function$\nDECLARE\n settingval text;\n sysmem bigint := pg_size_bytes(_sysmem);\n effective_cache_size bigint := pg_size_bytes(current_setting('effective_cache_size', TRUE));\n shared_buffers bigint := pg_size_bytes(current_setting('shared_buffers', TRUE));\n work_mem bigint := pg_size_bytes(current_setting('work_mem', TRUE));\n max_connections int := current_setting('max_connections', TRUE);\n maintenance_work_mem bigint := pg_size_bytes(current_setting('maintenance_work_mem', TRUE));\n seq_page_cost float := current_setting('seq_page_cost', TRUE);\n random_page_cost float := current_setting('random_page_cost', TRUE);\n temp_buffers bigint := pg_size_bytes(current_setting('temp_buffers', TRUE));\n r record;\nBEGIN\n IF _sysmem IS NULL THEN\n RAISE NOTICE 'Call function with the size of your system memory `SELECT check_pgstac_settings(''4GB'')` to get pg system setting recommendations.';\n ELSE\n IF effective_cache_size < (sysmem * 0.5) THEN\n RAISE WARNING 'effective_cache_size of % is set low for a system with %. Recomended value between % and %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.5), pg_size_pretty(sysmem * 0.75);\n ELSIF effective_cache_size > (sysmem * 0.75) THEN\n RAISE WARNING 'effective_cache_size of % is set high for a system with %. Recomended value between % and %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.5), pg_size_pretty(sysmem * 0.75);\n ELSE\n RAISE NOTICE 'effective_cache_size of % is set appropriately for a system with %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem);\n END IF;\n\n IF shared_buffers < (sysmem * 0.2) THEN\n RAISE WARNING 'shared_buffers of % is set low for a system with %. Recomended value between % and %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.2), pg_size_pretty(sysmem * 0.3);\n ELSIF shared_buffers > (sysmem * 0.3) THEN\n RAISE WARNING 'shared_buffers of % is set high for a system with %. Recomended value between % and %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.2), pg_size_pretty(sysmem * 0.3);\n ELSE\n RAISE NOTICE 'shared_buffers of % is set appropriately for a system with %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem);\n END IF;\n shared_buffers = sysmem * 0.3;\n IF maintenance_work_mem < (sysmem * 0.2) THEN\n RAISE WARNING 'maintenance_work_mem of % is set low for shared_buffers of %. Recomended value between % and %', pg_size_pretty(maintenance_work_mem), pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers * 0.2), pg_size_pretty(shared_buffers * 0.3);\n ELSIF maintenance_work_mem > (shared_buffers * 0.3) THEN\n RAISE WARNING 'maintenance_work_mem of % is set high for shared_buffers of %. Recomended value between % and %', pg_size_pretty(maintenance_work_mem), pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers * 0.2), pg_size_pretty(shared_buffers * 0.3);\n ELSE\n RAISE NOTICE 'maintenance_work_mem of % is set appropriately for shared_buffers of %', pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers);\n END IF;\n\n IF work_mem * max_connections > shared_buffers THEN\n RAISE WARNING 'work_mem setting of % is set high for % max_connections please reduce work_mem to % or decrease max_connections to %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers/max_connections), floor(shared_buffers/work_mem);\n ELSIF work_mem * max_connections < (shared_buffers * 0.75) THEN\n RAISE WARNING 'work_mem setting of % is set low for % max_connections you may consider raising work_mem to % or increasing max_connections to %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers/max_connections), floor(shared_buffers/work_mem);\n ELSE\n RAISE NOTICE 'work_mem setting of % and max_connections of % are adequate for shared_buffers of %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers);\n END IF;\n\n IF random_page_cost / seq_page_cost != 1.1 THEN\n RAISE WARNING 'random_page_cost (%) /seq_page_cost (%) should be set to 1.1 for SSD. Change random_page_cost to %', random_page_cost, seq_page_cost, 1.1 * seq_page_cost;\n ELSE\n RAISE NOTICE 'random_page_cost and seq_page_cost set appropriately for SSD';\n END IF;\n\n IF temp_buffers < greatest(pg_size_bytes('128MB'),(maintenance_work_mem / 2)) THEN\n RAISE WARNING 'pgstac makes heavy use of temp tables, consider raising temp_buffers from % to %', pg_size_pretty(temp_buffers), greatest('128MB', pg_size_pretty((shared_buffers / 16)));\n END IF;\n END IF;\n\n RAISE NOTICE 'VALUES FOR PGSTAC VARIABLES';\n RAISE NOTICE 'These can be set either as GUC system variables or by setting in the pgstac_settings table.';\n\n FOR r IN SELECT name, get_setting(name) as setting, CASE WHEN current_setting(concat('pgstac.',name), TRUE) IS NOT NULL THEN concat('pgstac.',name, ' GUC') WHEN value IS NOT NULL THEN 'pgstac_settings table' ELSE 'Not Set' END as loc FROM pgstac_settings LOOP\n RAISE NOTICE '% is set to % from the %', r.name, r.setting, r.loc;\n END LOOP;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pg_cron';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider intalling pg_cron which can be used to automate tasks';\n ELSE\n RAISE NOTICE 'pg_cron % is installed', settingval;\n END IF;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pgstattuple';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider installing the pgstattuple extension which can be used to help maintain tables and indexes.';\n ELSE\n RAISE NOTICE 'pgstattuple % is installed', settingval;\n END IF;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pg_stat_statements';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider installing the pg_stat_statements extension which is very helpful for tracking the types of queries on the system';\n ELSE\n RAISE NOTICE 'pg_stat_statements % is installed', settingval;\n IF current_setting('pg_stat_statements.track_statements', TRUE) IS DISTINCT FROM 'all' THEN\n RAISE WARNING 'SET pg_stat_statements.track_statements TO ''all''; --In order to track statements within functions.';\n END IF;\n END IF;\n\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.collections_trigger_func()\n RETURNS trigger\n LANGUAGE plpgsql\nAS $function$\nDECLARE\n q text;\n partition_name text := format('_items_%s', NEW.key);\n partition_exists boolean := false;\n partition_empty boolean := true;\n err_context text;\n loadtemp boolean := FALSE;\nBEGIN\n RAISE NOTICE 'Collection Trigger. % %', NEW.id, NEW.key;\n IF TG_OP = 'UPDATE' AND NEW.partition_trunc IS DISTINCT FROM OLD.partition_trunc THEN\n PERFORM repartition(NEW.id, NEW.partition_trunc, TRUE);\n END IF;\n RETURN NEW;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.get_item(_id text, _collection text DEFAULT NULL::text)\n RETURNS jsonb\n LANGUAGE sql\n STABLE\n SET search_path TO 'pgstac', 'public'\nAS $function$\n SELECT content_hydrate(items) FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection);\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.get_setting(_setting text, conf jsonb DEFAULT NULL::jsonb)\n RETURNS text\n LANGUAGE sql\nAS $function$\nSELECT COALESCE(\n nullif(conf->>_setting, ''),\n nullif(current_setting(concat('pgstac.',_setting), TRUE),''),\n nullif((SELECT value FROM pgstac.pgstac_settings WHERE name=_setting),'')\n);\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.get_token_filter(_search jsonb DEFAULT '{}'::jsonb, token_rec jsonb DEFAULT NULL::jsonb)\n RETURNS text\n LANGUAGE plpgsql\n SET transform_null_equals TO 'true'\nAS $function$\nDECLARE\n token_id text;\n filters text[] := '{}'::text[];\n prev boolean := TRUE;\n field text;\n dir text;\n sort record;\n orfilter text := '';\n orfilters text[] := '{}'::text[];\n andfilters text[] := '{}'::text[];\n output text;\n token_where text;\n token_item items%ROWTYPE;\nBEGIN\n RAISE NOTICE 'Getting Token Filter. % %', _search, token_rec;\n -- If no token provided return NULL\n IF token_rec IS NULL THEN\n IF NOT (_search ? 'token' AND\n (\n (_search->>'token' ILIKE 'prev:%')\n OR\n (_search->>'token' ILIKE 'next:%')\n )\n ) THEN\n RETURN NULL;\n END IF;\n prev := (_search->>'token' ILIKE 'prev:%');\n token_id := substr(_search->>'token', 6);\n IF token_id IS NULL OR token_id = '' THEN\n RAISE WARNING 'next or prev set, but no token id found';\n RETURN NULL;\n END IF;\n SELECT to_jsonb(items) INTO token_rec\n FROM items WHERE id=token_id;\n END IF;\n RAISE NOTICE 'TOKEN ID: % %', token_rec, token_rec->'id';\n\n\n RAISE NOTICE 'TOKEN ID: % %', token_rec, token_rec->'id';\n token_item := jsonb_populate_record(null::items, token_rec);\n RAISE NOTICE 'TOKEN ITEM ----- %', token_item;\n\n\n CREATE TEMP TABLE sorts (\n _row int GENERATED ALWAYS AS IDENTITY NOT NULL,\n _field text PRIMARY KEY,\n _dir text NOT NULL,\n _val text\n ) ON COMMIT DROP;\n\n -- Make sure we only have distinct columns to sort with taking the first one we get\n INSERT INTO sorts (_field, _dir)\n SELECT\n (queryable(value->>'field')).expression,\n get_sort_dir(value)\n FROM\n jsonb_array_elements(coalesce(_search->'sortby','[{\"field\":\"datetime\",\"direction\":\"desc\"}]'))\n ON CONFLICT DO NOTHING\n ;\n RAISE NOTICE 'sorts 1: %', (SELECT jsonb_agg(to_json(sorts)) FROM sorts);\n -- Get the first sort direction provided. As the id is a primary key, if there are any\n -- sorts after id they won't do anything, so make sure that id is the last sort item.\n SELECT _dir INTO dir FROM sorts ORDER BY _row ASC LIMIT 1;\n IF EXISTS (SELECT 1 FROM sorts WHERE _field = 'id') THEN\n DELETE FROM sorts WHERE _row > (SELECT _row FROM sorts WHERE _field = 'id' ORDER BY _row ASC);\n ELSE\n INSERT INTO sorts (_field, _dir) VALUES ('id', dir);\n END IF;\n\n -- Add value from looked up item to the sorts table\n UPDATE sorts SET _val=get_token_val_str(_field, token_item);\n\n -- Check if all sorts are the same direction and use row comparison\n -- to filter\n RAISE NOTICE 'sorts 2: %', (SELECT jsonb_agg(to_json(sorts)) FROM sorts);\n FOR sort IN SELECT * FROM sorts ORDER BY _row asc LOOP\n orfilter := NULL;\n RAISE NOTICE 'SORT: %', sort;\n IF sort._val IS NOT NULL AND ((prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC')) THEN\n orfilter := format($f$(\n (%s < %s) OR (%s IS NULL)\n )$f$,\n sort._field,\n sort._val,\n sort._val\n );\n ELSIF sort._val IS NULL AND ((prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC')) THEN\n RAISE NOTICE '< but null';\n orfilter := format('%s IS NOT NULL', sort._field);\n ELSIF sort._val IS NULL THEN\n RAISE NOTICE '> but null';\n --orfilter := format('%s IS NULL', sort._field);\n ELSE\n orfilter := format($f$(\n (%s > %s) OR (%s IS NULL)\n )$f$,\n sort._field,\n sort._val,\n sort._field\n );\n END IF;\n RAISE NOTICE 'ORFILTER: %', orfilter;\n\n IF orfilter IS NOT NULL THEN\n IF sort._row = 1 THEN\n orfilters := orfilters || orfilter;\n ELSE\n orfilters := orfilters || format('(%s AND %s)', array_to_string(andfilters, ' AND '), orfilter);\n END IF;\n END IF;\n IF sort._val IS NOT NULL THEN\n andfilters := andfilters || format('%s = %s', sort._field, sort._val);\n ELSE\n andfilters := andfilters || format('%s IS NULL', sort._field);\n END IF;\n END LOOP;\n output := array_to_string(orfilters, ' OR ');\n\n DROP TABLE IF EXISTS sorts;\n token_where := concat('(',coalesce(output,'true'),')');\n IF trim(token_where) = '' THEN\n token_where := NULL;\n END IF;\n RAISE NOTICE 'TOKEN_WHERE: %',token_where;\n RETURN token_where;\n END;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.item_by_id(_id text, _collection text DEFAULT NULL::text)\n RETURNS pgstac.items\n LANGUAGE plpgsql\n STABLE\n SET search_path TO 'pgstac', 'public'\nAS $function$\nDECLARE\n i items%ROWTYPE;\nBEGIN\n SELECT * INTO i FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection) LIMIT 1;\n RETURN i;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.items_staging_triggerfunc()\n RETURNS trigger\n LANGUAGE plpgsql\nAS $function$\nDECLARE\n p record;\n _partitions text[];\n part text;\n ts timestamptz := clock_timestamp();\nBEGIN\n RAISE NOTICE 'Creating Partitions. %', clock_timestamp() - ts;\n\n FOR part IN WITH t AS (\n SELECT\n n.content->>'collection' as collection,\n stac_daterange(n.content->'properties') as dtr,\n partition_trunc\n FROM newdata n JOIN collections ON (n.content->>'collection'=collections.id)\n ), p AS (\n SELECT\n collection,\n COALESCE(date_trunc(partition_trunc::text, lower(dtr)),'-infinity') as d,\n tstzrange(min(lower(dtr)),max(lower(dtr)),'[]') as dtrange,\n tstzrange(min(upper(dtr)),max(upper(dtr)),'[]') as edtrange\n FROM t\n GROUP BY 1,2\n ) SELECT check_partition(collection, dtrange, edtrange) FROM p LOOP\n RAISE NOTICE 'Partition %', part;\n END LOOP;\n\n RAISE NOTICE 'Doing the insert. %', clock_timestamp() - ts;\n IF TG_TABLE_NAME = 'items_staging' THEN\n INSERT INTO items\n SELECT\n (content_dehydrate(content)).*\n FROM newdata;\n RAISE NOTICE 'Doing the delete. %', clock_timestamp() - ts;\n DELETE FROM items_staging;\n ELSIF TG_TABLE_NAME = 'items_staging_ignore' THEN\n INSERT INTO items\n SELECT\n (content_dehydrate(content)).*\n FROM newdata\n ON CONFLICT DO NOTHING;\n RAISE NOTICE 'Doing the delete. %', clock_timestamp() - ts;\n DELETE FROM items_staging_ignore;\n ELSIF TG_TABLE_NAME = 'items_staging_upsert' THEN\n WITH staging_formatted AS (\n SELECT (content_dehydrate(content)).* FROM newdata\n ), deletes AS (\n DELETE FROM items i USING staging_formatted s\n WHERE\n i.id = s.id\n AND i.collection = s.collection\n AND i IS DISTINCT FROM s\n RETURNING i.id, i.collection\n )\n INSERT INTO items\n SELECT s.* FROM\n staging_formatted s\n ON CONFLICT DO NOTHING;\n RAISE NOTICE 'Doing the delete. %', clock_timestamp() - ts;\n DELETE FROM items_staging_upsert;\n END IF;\n RAISE NOTICE 'Done. %', clock_timestamp() - ts;\n\n RETURN NULL;\n\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.parse_dtrange(_indate jsonb, relative_base timestamp with time zone DEFAULT date_trunc('hour'::text, CURRENT_TIMESTAMP))\n RETURNS tstzrange\n LANGUAGE plpgsql\n STABLE PARALLEL SAFE STRICT\n SET \"TimeZone\" TO 'UTC'\nAS $function$\nDECLARE\n timestrs text[];\n s timestamptz;\n e timestamptz;\nBEGIN\n timestrs :=\n CASE\n WHEN _indate ? 'timestamp' THEN\n ARRAY[_indate->>'timestamp']\n WHEN _indate ? 'interval' THEN\n to_text_array(_indate->'interval')\n WHEN jsonb_typeof(_indate) = 'array' THEN\n to_text_array(_indate)\n ELSE\n regexp_split_to_array(\n _indate->>0,\n '/'\n )\n END;\n RAISE NOTICE 'TIMESTRS %', timestrs;\n IF cardinality(timestrs) = 1 THEN\n IF timestrs[1] ILIKE 'P%' THEN\n RETURN tstzrange(relative_base - upper(timestrs[1])::interval, relative_base, '[)');\n END IF;\n s := timestrs[1]::timestamptz;\n RETURN tstzrange(s, s, '[]');\n END IF;\n\n IF cardinality(timestrs) != 2 THEN\n RAISE EXCEPTION 'Timestamp cannot have more than 2 values';\n END IF;\n\n IF timestrs[1] = '..' OR timestrs[1] = '' THEN\n s := '-infinity'::timestamptz;\n e := timestrs[2]::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] = '..' OR timestrs[2] = '' THEN\n s := timestrs[1]::timestamptz;\n e := 'infinity'::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[1] ILIKE 'P%' AND timestrs[2] NOT ILIKE 'P%' THEN\n e := timestrs[2]::timestamptz;\n s := e - upper(timestrs[1])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] ILIKE 'P%' AND timestrs[1] NOT ILIKE 'P%' THEN\n s := timestrs[1]::timestamptz;\n e := s + upper(timestrs[2])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n s := timestrs[1]::timestamptz;\n e := timestrs[2]::timestamptz;\n\n RETURN tstzrange(s,e,'[)');\n\n RETURN NULL;\n\nEND;\n$function$\n;\n\ncreate or replace view \"pgstac\".\"partition_steps\" as SELECT partitions.partition AS name,\n date_trunc('month'::text, lower(partitions.partition_dtrange)) AS sdate,\n (date_trunc('month'::text, upper(partitions.partition_dtrange)) + '1 mon'::interval) AS edate\n FROM pgstac.partitions\n WHERE ((partitions.partition_dtrange IS NOT NULL) AND (partitions.partition_dtrange <> 'empty'::tstzrange))\n ORDER BY partitions.dtrange;\n\n\nCREATE OR REPLACE FUNCTION pgstac.queryables_trigger_func()\n RETURNS trigger\n LANGUAGE plpgsql\nAS $function$\nDECLARE\nBEGIN\n PERFORM maintain_partitions();\n RETURN NULL;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.search(_search jsonb DEFAULT '{}'::jsonb)\n RETURNS jsonb\n LANGUAGE plpgsql\n SET search_path TO 'pgstac', 'public'\n SET cursor_tuple_fraction TO '1'\nAS $function$\nDECLARE\n searches searches%ROWTYPE;\n _where text;\n token_where text;\n full_where text;\n orderby text;\n query text;\n token_type text := substr(_search->>'token',1,4);\n _limit int := coalesce((_search->>'limit')::int, 10);\n curs refcursor;\n cntr int := 0;\n iter_record items%ROWTYPE;\n first_record jsonb;\n first_item items%ROWTYPE;\n last_item items%ROWTYPE;\n last_record jsonb;\n out_records jsonb := '[]'::jsonb;\n prev_query text;\n next text;\n prev_id text;\n has_next boolean := false;\n has_prev boolean := false;\n prev text;\n total_count bigint;\n context jsonb;\n collection jsonb;\n includes text[];\n excludes text[];\n exit_flag boolean := FALSE;\n batches int := 0;\n timer timestamptz := clock_timestamp();\n pstart timestamptz;\n pend timestamptz;\n pcurs refcursor;\n search_where search_wheres%ROWTYPE;\n id text;\nBEGIN\nCREATE TEMP TABLE results (i int GENERATED ALWAYS AS IDENTITY, content jsonb) ON COMMIT DROP;\n searches := search_query(_search);\n _where := searches._where;\n orderby := searches.orderby;\n search_where := where_stats(_where);\n total_count := coalesce(search_where.total_count, search_where.estimated_count);\n\n IF token_type='prev' THEN\n token_where := get_token_filter(_search, null::jsonb);\n orderby := sort_sqlorderby(_search, TRUE);\n END IF;\n IF token_type='next' THEN\n token_where := get_token_filter(_search, null::jsonb);\n END IF;\n\n full_where := concat_ws(' AND ', _where, token_where);\n RAISE NOTICE 'FULL QUERY % %', full_where, clock_timestamp()-timer;\n timer := clock_timestamp();\n\n FOR query IN SELECT partition_queries(full_where, orderby, search_where.partitions) LOOP\n timer := clock_timestamp();\n query := format('%s LIMIT %s', query, _limit + 1);\n RAISE NOTICE 'Partition Query: %', query;\n batches := batches + 1;\n -- curs = create_cursor(query);\n RAISE NOTICE 'cursor_tuple_fraction: %', current_setting('cursor_tuple_fraction');\n OPEN curs FOR EXECUTE query;\n LOOP\n FETCH curs into iter_record;\n EXIT WHEN NOT FOUND;\n cntr := cntr + 1;\n\n IF _search->'conf'->>'nohydrate' IS NOT NULL AND (_search->'conf'->>'nohydrate')::boolean = true THEN\n last_record := content_nonhydrated(iter_record, _search->'fields');\n ELSE\n last_record := content_hydrate(iter_record, _search->'fields');\n END IF;\n last_item := iter_record;\n IF cntr = 1 THEN\n first_item := last_item;\n first_record := last_record;\n END IF;\n IF cntr <= _limit THEN\n INSERT INTO results (content) VALUES (last_record);\n ELSIF cntr > _limit THEN\n has_next := true;\n exit_flag := true;\n EXIT;\n END IF;\n END LOOP;\n CLOSE curs;\n RAISE NOTICE 'Query took %.', clock_timestamp()-timer;\n timer := clock_timestamp();\n EXIT WHEN exit_flag;\n END LOOP;\n RAISE NOTICE 'Scanned through % partitions.', batches;\n\nWITH ordered AS (SELECT * FROM results WHERE content IS NOT NULL ORDER BY i)\nSELECT jsonb_agg(content) INTO out_records FROM ordered;\n\nDROP TABLE results;\n\n\n-- Flip things around if this was the result of a prev token query\nIF token_type='prev' THEN\n out_records := flip_jsonb_array(out_records);\n first_item := last_item;\n first_record := last_record;\nEND IF;\n\n-- If this query has a token, see if there is data before the first record\nIF _search ? 'token' THEN\n prev_query := format(\n 'SELECT 1 FROM items WHERE %s LIMIT 1',\n concat_ws(\n ' AND ',\n _where,\n trim(get_token_filter(_search, to_jsonb(first_item)))\n )\n );\n RAISE NOTICE 'Query to get previous record: % --- %', prev_query, first_record;\n EXECUTE prev_query INTO has_prev;\n IF FOUND and has_prev IS NOT NULL THEN\n RAISE NOTICE 'Query results from prev query: %', has_prev;\n has_prev := TRUE;\n END IF;\nEND IF;\nhas_prev := COALESCE(has_prev, FALSE);\n\nIF has_prev THEN\n prev := out_records->0->>'id';\nEND IF;\nIF has_next OR token_type='prev' THEN\n next := out_records->-1->>'id';\nEND IF;\n\nIF context(_search->'conf') != 'off' THEN\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'matched', total_count,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\nELSE\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\nEND IF;\n\ncollection := jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb),\n 'next', next,\n 'prev', prev,\n 'context', context\n);\n\nRETURN collection;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.search_query(_search jsonb DEFAULT '{}'::jsonb, updatestats boolean DEFAULT false, _metadata jsonb DEFAULT '{}'::jsonb)\n RETURNS pgstac.searches\n LANGUAGE plpgsql\n SECURITY DEFINER\nAS $function$\nDECLARE\n search searches%ROWTYPE;\n pexplain jsonb;\n t timestamptz;\n i interval;\nBEGIN\n SELECT * INTO search FROM searches\n WHERE hash=search_hash(_search, _metadata) FOR UPDATE;\n\n -- Calculate the where clause if not already calculated\n IF search._where IS NULL THEN\n search._where := stac_search_to_where(_search);\n END IF;\n\n -- Calculate the order by clause if not already calculated\n IF search.orderby IS NULL THEN\n search.orderby := sort_sqlorderby(_search);\n END IF;\n\n PERFORM where_stats(search._where, updatestats, _search->'conf');\n\n search.lastused := now();\n search.usecount := coalesce(search.usecount, 0) + 1;\n INSERT INTO searches (search, _where, orderby, lastused, usecount, metadata)\n VALUES (_search, search._where, search.orderby, search.lastused, search.usecount, _metadata)\n ON CONFLICT (hash) DO\n UPDATE SET\n _where = EXCLUDED._where,\n orderby = EXCLUDED.orderby,\n lastused = EXCLUDED.lastused,\n usecount = EXCLUDED.usecount,\n metadata = EXCLUDED.metadata\n RETURNING * INTO search\n ;\n RETURN search;\n\nEND;\n$function$\n;\n\nCREATE OR REPLACE PROCEDURE pgstac.validate_constraints()\n LANGUAGE plpgsql\nAS $procedure$\nDECLARE\n q text;\nBEGIN\n FOR q IN\n SELECT\n FORMAT(\n 'ALTER TABLE %I.%I VALIDATE CONSTRAINT %I;',\n nsp.nspname,\n cls.relname,\n con.conname\n )\n\n FROM pg_constraint AS con\n JOIN pg_class AS cls\n ON con.conrelid = cls.oid\n JOIN pg_namespace AS nsp\n ON cls.relnamespace = nsp.oid\n WHERE convalidated = FALSE AND contype in ('c','f')\n AND nsp.nspname = 'pgstac'\n LOOP\n RAISE NOTICE '%', q;\n PERFORM run_or_queue(q);\n COMMIT;\n END LOOP;\nEND;\n$procedure$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.where_stats(inwhere text, updatestats boolean DEFAULT false, conf jsonb DEFAULT NULL::jsonb)\n RETURNS pgstac.search_wheres\n LANGUAGE plpgsql\n SECURITY DEFINER\nAS $function$\nDECLARE\n t timestamptz;\n i interval;\n explain_json jsonb;\n partitions text[];\n sw search_wheres%ROWTYPE;\n inwhere_hash text := md5(inwhere);\n _context text := lower(context(conf));\n _stats_ttl interval := context_stats_ttl(conf);\n _estimated_cost float := context_estimated_cost(conf);\n _estimated_count int := context_estimated_count(conf);\nBEGIN\n IF _context = 'off' THEN\n sw._where := inwhere;\n return sw;\n END IF;\n\n SELECT * INTO sw FROM search_wheres WHERE md5(_where)=inwhere_hash FOR UPDATE;\n\n -- Update statistics if explicitly set, if statistics do not exist, or statistics ttl has expired\n IF NOT updatestats THEN\n RAISE NOTICE 'Checking if update is needed for: % .', inwhere;\n RAISE NOTICE 'Stats Last Updated: %', sw.statslastupdated;\n RAISE NOTICE 'TTL: %, Age: %', _stats_ttl, now() - sw.statslastupdated;\n RAISE NOTICE 'Context: %, Existing Total: %', _context, sw.total_count;\n IF\n sw.statslastupdated IS NULL\n OR (now() - sw.statslastupdated) > _stats_ttl\n OR (context(conf) != 'off' AND sw.total_count IS NULL)\n THEN\n updatestats := TRUE;\n END IF;\n END IF;\n\n sw._where := inwhere;\n sw.lastused := now();\n sw.usecount := coalesce(sw.usecount,0) + 1;\n\n IF NOT updatestats THEN\n UPDATE search_wheres SET\n lastused = sw.lastused,\n usecount = sw.usecount\n WHERE md5(_where) = inwhere_hash\n RETURNING * INTO sw\n ;\n RETURN sw;\n END IF;\n\n -- Use explain to get estimated count/cost and a list of the partitions that would be hit by the query\n t := clock_timestamp();\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s', inwhere)\n INTO explain_json;\n RAISE NOTICE 'Time for just the explain: %', clock_timestamp() - t;\n i := clock_timestamp() - t;\n\n sw.statslastupdated := now();\n sw.estimated_count := explain_json->0->'Plan'->'Plan Rows';\n sw.estimated_cost := explain_json->0->'Plan'->'Total Cost';\n sw.time_to_estimate := extract(epoch from i);\n\n RAISE NOTICE 'ESTIMATED_COUNT: % < %', sw.estimated_count, _estimated_count;\n RAISE NOTICE 'ESTIMATED_COST: % < %', sw.estimated_cost, _estimated_cost;\n\n -- Do a full count of rows if context is set to on or if auto is set and estimates are low enough\n IF\n _context = 'on'\n OR\n ( _context = 'auto' AND\n (\n sw.estimated_count < _estimated_count\n AND\n sw.estimated_cost < _estimated_cost\n )\n )\n THEN\n t := clock_timestamp();\n RAISE NOTICE 'Calculating actual count...';\n EXECUTE format(\n 'SELECT count(*) FROM items WHERE %s',\n inwhere\n ) INTO sw.total_count;\n i := clock_timestamp() - t;\n RAISE NOTICE 'Actual Count: % -- %', sw.total_count, i;\n sw.time_to_count := extract(epoch FROM i);\n ELSE\n sw.total_count := NULL;\n sw.time_to_count := NULL;\n END IF;\n\n\n INSERT INTO search_wheres\n (_where, lastused, usecount, statslastupdated, estimated_count, estimated_cost, time_to_estimate, partitions, total_count, time_to_count)\n SELECT sw._where, sw.lastused, sw.usecount, sw.statslastupdated, sw.estimated_count, sw.estimated_cost, sw.time_to_estimate, sw.partitions, sw.total_count, sw.time_to_count\n ON CONFLICT ((md5(_where)))\n DO UPDATE\n SET\n lastused = sw.lastused,\n usecount = sw.usecount,\n statslastupdated = sw.statslastupdated,\n estimated_count = sw.estimated_count,\n estimated_cost = sw.estimated_cost,\n time_to_estimate = sw.time_to_estimate,\n total_count = sw.total_count,\n time_to_count = sw.time_to_count\n ;\n RETURN sw;\nEND;\n$function$\n;\n\nCREATE TRIGGER items_after_delete_trigger AFTER UPDATE ON pgstac.items REFERENCING NEW TABLE AS newdata FOR EACH STATEMENT EXECUTE FUNCTION pgstac.partition_after_triggerfunc();\n\nCREATE TRIGGER items_after_insert_trigger AFTER INSERT ON pgstac.items REFERENCING NEW TABLE AS newdata FOR EACH STATEMENT EXECUTE FUNCTION pgstac.partition_after_triggerfunc();\n\nCREATE TRIGGER items_after_update_trigger AFTER DELETE ON pgstac.items REFERENCING OLD TABLE AS newdata FOR EACH STATEMENT EXECUTE FUNCTION pgstac.partition_after_triggerfunc();\n\nCREATE TRIGGER queryables_constraint_insert_trigger BEFORE INSERT ON pgstac.queryables FOR EACH ROW EXECUTE FUNCTION pgstac.queryables_constraint_triggerfunc();\n\nCREATE TRIGGER queryables_constraint_update_trigger BEFORE UPDATE ON pgstac.queryables FOR EACH ROW WHEN (((new.name = old.name) AND (new.collection_ids IS DISTINCT FROM old.collection_ids))) EXECUTE FUNCTION pgstac.queryables_constraint_triggerfunc();\n\n\n\n-- END migra calculated SQL\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('id', '{\"title\": \"Item ID\",\"description\": \"Item identifier\",\"$ref\": \"https://schemas.stacspec.org/v1.0.0/item-spec/json-schema/item.json#/definitions/core/allOf/2/properties/id\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('geometry', '{\"title\": \"Item Geometry\",\"description\": \"Item Geometry\",\"$ref\": \"https://geojson.org/schema/Feature.json\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('datetime','{\"description\": \"Datetime\",\"type\": \"string\",\"title\": \"Acquired\",\"format\": \"date-time\",\"pattern\": \"(\\\\+00:00|Z)$\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('eo:cloud_cover','{\"$ref\": \"https://stac-extensions.github.io/eo/v1.0.0/schema.json#/definitions/fieldsproperties/eo:cloud_cover\"}','to_int','BTREE');\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDELETE FROM queryables a USING queryables b\n WHERE a.name = b.name AND a.collection_ids IS NOT DISTINCT FROM b.collection_ids AND a.id > b.id;\n\n\nINSERT INTO pgstac_settings (name, value) VALUES\n ('context', 'off'),\n ('context_estimated_count', '100000'),\n ('context_estimated_cost', '100000'),\n ('context_stats_ttl', '1 day'),\n ('default_filter_lang', 'cql2-json'),\n ('additional_properties', 'true'),\n ('use_queue', 'false'),\n ('queue_timeout', '10 minutes'),\n ('update_collection_extent', 'false')\nON CONFLICT DO NOTHING\n;\n\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nGRANT ALL ON SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac to pgstac_admin;\n\n-- pgstac_read role limited to using function apis\nGRANT EXECUTE ON FUNCTION search TO pgstac_read;\nGRANT EXECUTE ON FUNCTION search_query TO pgstac_read;\nGRANT EXECUTE ON FUNCTION item_by_id TO pgstac_read;\nGRANT EXECUTE ON FUNCTION get_item TO pgstac_read;\nGRANT SELECT ON ALL TABLES IN SCHEMA pgstac TO pgstac_read;\n\n\nGRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_ingest;\nGRANT USAGE ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_ingest;\n\nSELECT update_partition_stats_q(partition) FROM partitions;\nSELECT set_version('0.7.3');\n" }, { "path": "src/pgstac/migrations/pgstac.0.6.13.sql", "content": "CREATE EXTENSION IF NOT EXISTS postgis;\nCREATE EXTENSION IF NOT EXISTS btree_gist;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_admin;\n CREATE ROLE pgstac_read;\n CREATE ROLE pgstac_ingest;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nGRANT pgstac_admin TO current_user;\n\nCREATE SCHEMA IF NOT EXISTS pgstac AUTHORIZATION pgstac_admin;\n\nALTER ROLE pgstac_admin SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_read SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_ingest SET SEARCH_PATH TO pgstac, public;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\n\nGRANT pgstac_read TO pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\n\nSET ROLE pgstac_admin;\n\nSET SEARCH_PATH TO pgstac, public;\n\n\nCREATE TABLE IF NOT EXISTS migrations (\n version text PRIMARY KEY,\n datetime timestamptz DEFAULT clock_timestamp() NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION get_version() RETURNS text AS $$\n SELECT version FROM pgstac.migrations ORDER BY datetime DESC, version DESC LIMIT 1;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION set_version(text) RETURNS text AS $$\n INSERT INTO pgstac.migrations (version) VALUES ($1)\n ON CONFLICT DO NOTHING\n RETURNING version;\n$$ LANGUAGE SQL;\n\n\nCREATE TABLE IF NOT EXISTS pgstac_settings (\n name text PRIMARY KEY,\n value text NOT NULL\n);\n\nINSERT INTO pgstac_settings (name, value) VALUES\n ('context', 'off'),\n ('context_estimated_count', '100000'),\n ('context_estimated_cost', '100000'),\n ('context_stats_ttl', '1 day'),\n ('default_filter_lang', 'cql2-json'),\n ('additional_properties', 'true')\nON CONFLICT DO NOTHING\n;\n\n\nCREATE OR REPLACE FUNCTION get_setting(IN _setting text, IN conf jsonb DEFAULT NULL) RETURNS text AS $$\nSELECT COALESCE(\n conf->>_setting,\n current_setting(concat('pgstac.',_setting), TRUE),\n (SELECT value FROM pgstac.pgstac_settings WHERE name=_setting)\n);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context(conf jsonb DEFAULT NULL) RETURNS text AS $$\n SELECT pgstac.get_setting('context', conf);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context_estimated_count(conf jsonb DEFAULT NULL) RETURNS int AS $$\n SELECT pgstac.get_setting('context_estimated_count', conf)::int;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_estimated_cost();\nCREATE OR REPLACE FUNCTION context_estimated_cost(conf jsonb DEFAULT NULL) RETURNS float AS $$\n SELECT pgstac.get_setting('context_estimated_cost', conf)::float;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_stats_ttl();\nCREATE OR REPLACE FUNCTION context_stats_ttl(conf jsonb DEFAULT NULL) RETURNS interval AS $$\n SELECT pgstac.get_setting('context_stats_ttl', conf)::interval;\n$$ LANGUAGE SQL;\n\n\nCREATE OR REPLACE FUNCTION notice(VARIADIC text[]) RETURNS boolean AS $$\nDECLARE\ndebug boolean := current_setting('pgstac.debug', true);\nBEGIN\n IF debug THEN\n RAISE NOTICE 'NOTICE FROM FUNC: % >>>>> %', concat_ws(' | ', $1), clock_timestamp();\n RETURN TRUE;\n END IF;\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION empty_arr(ANYARRAY) RETURNS BOOLEAN AS $$\nSELECT CASE\n WHEN $1 IS NULL THEN TRUE\n WHEN cardinality($1)<1 THEN TRUE\nELSE FALSE\nEND;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_intersection(_a ANYARRAY, _b ANYARRAY) RETURNS ANYARRAY AS $$\n SELECT ARRAY ( SELECT unnest(_a) INTERSECT SELECT UNNEST(_b) );\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION array_map_ident(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_ident(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_map_literal(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_literal(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION array_reverse(anyarray) RETURNS anyarray AS $$\nSELECT ARRAY(\n SELECT $1[i]\n FROM generate_subscripts($1,1) AS s(i)\n ORDER BY i DESC\n);\n$$ LANGUAGE SQL STRICT IMMUTABLE;\n\n\nDROP FUNCTION IF EXISTS check_pgstac_settings;\nCREATE OR REPLACE FUNCTION check_pgstac_settings(_sysmem text DEFAULT NULL) RETURNS VOID AS $$\nDECLARE\n settingval text;\n sysmem bigint := pg_size_bytes(_sysmem);\n effective_cache_size bigint := pg_size_bytes(current_setting('effective_cache_size', TRUE));\n shared_buffers bigint := pg_size_bytes(current_setting('shared_buffers', TRUE));\n work_mem bigint := pg_size_bytes(current_setting('work_mem', TRUE));\n max_connections int := current_setting('max_connections', TRUE);\n maintenance_work_mem bigint := pg_size_bytes(current_setting('maintenance_work_mem', TRUE));\n seq_page_cost float := current_setting('seq_page_cost', TRUE);\n random_page_cost float := current_setting('random_page_cost', TRUE);\n temp_buffers bigint := pg_size_bytes(current_setting('temp_buffers', TRUE));\n r record;\nBEGIN\n IF _sysmem IS NULL THEN\n RAISE NOTICE 'Call function with the size of your system memory `SELECT check_pgstac_settings(''4GB'')` to get pg system setting recommendations.';\n ELSE\n IF effective_cache_size < (sysmem * 0.5) THEN\n RAISE WARNING 'effective_cache_size of % is set low for a system with %. Recomended value between % and %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.5), pg_size_pretty(sysmem * 0.75);\n ELSIF effective_cache_size > (sysmem * 0.75) THEN\n RAISE WARNING 'effective_cache_size of % is set high for a system with %. Recomended value between % and %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.5), pg_size_pretty(sysmem * 0.75);\n ELSE\n RAISE NOTICE 'effective_cache_size of % is set appropriately for a system with %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem);\n END IF;\n\n IF shared_buffers < (sysmem * 0.2) THEN\n RAISE WARNING 'shared_buffers of % is set low for a system with %. Recomended value between % and %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.2), pg_size_pretty(sysmem * 0.3);\n ELSIF shared_buffers > (sysmem * 0.3) THEN\n RAISE WARNING 'shared_buffers of % is set high for a system with %. Recomended value between % and %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.2), pg_size_pretty(sysmem * 0.3);\n ELSE\n RAISE NOTICE 'shared_buffers of % is set appropriately for a system with %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem);\n END IF;\n shared_buffers = sysmem * 0.3;\n IF maintenance_work_mem < (sysmem * 0.2) THEN\n RAISE WARNING 'maintenance_work_mem of % is set low for shared_buffers of %. Recomended value between % and %', pg_size_pretty(maintenance_work_mem), pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers * 0.2), pg_size_pretty(shared_buffers * 0.3);\n ELSIF maintenance_work_mem > (shared_buffers * 0.3) THEN\n RAISE WARNING 'maintenance_work_mem of % is set high for shared_buffers of %. Recomended value between % and %', pg_size_pretty(maintenance_work_mem), pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers * 0.2), pg_size_pretty(shared_buffers * 0.3);\n ELSE\n RAISE NOTICE 'maintenance_work_mem of % is set appropriately for shared_buffers of %', pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers);\n END IF;\n\n IF work_mem * max_connections > shared_buffers THEN\n RAISE WARNING 'work_mem setting of % is set high for % max_connections please reduce work_mem to % or decrease max_connections to %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers/max_connections), floor(shared_buffers/work_mem);\n ELSIF work_mem * max_connections < (shared_buffers * 0.75) THEN\n RAISE WARNING 'work_mem setting of % is set low for % max_connections you may consider raising work_mem to % or increasing max_connections to %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers/max_connections), floor(shared_buffers/work_mem);\n ELSE\n RAISE NOTICE 'work_mem setting of % and max_connections of % are adequate for shared_buffers of %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers);\n END IF;\n\n IF random_page_cost / seq_page_cost != 1.1 THEN\n RAISE WARNING 'random_page_cost (%) /seq_page_cost (%) should be set to 1.1 for SSD. Change random_page_cost to %', random_page_cost, seq_page_cost, 1.1 * seq_page_cost;\n ELSE\n RAISE NOTICE 'random_page_cost and seq_page_cost set appropriately for SSD';\n END IF;\n\n IF temp_buffers < greatest(pg_size_bytes('128MB'),(maintenance_work_mem / 2)) THEN\n RAISE WARNING 'pgstac makes heavy use of temp tables, consider raising temp_buffers from % to %', pg_size_pretty(temp_buffers), greatest('128MB', pg_size_pretty((shared_buffers / 16)));\n END IF;\n END IF;\n\n RAISE NOTICE 'VALUES FOR PGSTAC VARIABLES';\n RAISE NOTICE 'These can be set either as GUC system variables or by setting in the pgstac_settings table.';\n\n FOR r IN SELECT name, get_setting(name) as setting, CASE WHEN current_setting(concat('pgstac.',name), TRUE) IS NOT NULL THEN concat('pgstac.',name, ' GUC') WHEN value IS NOT NULL THEN 'pgstac_settings table' ELSE 'Not Set' END as loc FROM pgstac_settings LOOP\n RAISE NOTICE '% is set to % from the %', r.name, r.setting, r.loc;\n END LOOP;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pg_cron';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider intalling pg_cron which can be used to automate tasks';\n ELSE\n RAISE NOTICE 'pg_cron % is installed', settingval;\n END IF;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pgstattuple';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider installing the pgstattuple extension which can be used to help maintain tables and indexes.';\n ELSE\n RAISE NOTICE 'pgstattuple % is installed', settingval;\n END IF;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pg_stat_statements';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider installing the pg_stat_statements extension which is very helpful for tracking the types of queries on the system';\n ELSE\n RAISE NOTICE 'pgstattuple % is installed', settingval;\n END IF;\n\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac, public SET CLIENT_MIN_MESSAGES TO NOTICE;\nCREATE OR REPLACE FUNCTION to_int(jsonb) RETURNS int AS $$\n SELECT floor(($1->>0)::float)::int;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION to_float(jsonb) RETURNS float AS $$\n SELECT ($1->>0)::float;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION to_tstz(jsonb) RETURNS timestamptz AS $$\n SELECT ($1->>0)::timestamptz;\n$$ LANGUAGE SQL IMMUTABLE STRICT SET TIME ZONE 'UTC';\n\n\nCREATE OR REPLACE FUNCTION to_text(jsonb) RETURNS text AS $$\n SELECT CASE WHEN jsonb_typeof($1) IN ('array','object') THEN $1::text ELSE $1->>0 END;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION to_text_array(jsonb) RETURNS text[] AS $$\n SELECT\n CASE jsonb_typeof($1)\n WHEN 'array' THEN ARRAY(SELECT jsonb_array_elements_text($1))\n ELSE ARRAY[$1->>0]\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION bbox_geom(_bbox jsonb) RETURNS geometry AS $$\nSELECT CASE jsonb_array_length(_bbox)\n WHEN 4 THEN\n ST_SetSRID(ST_MakeEnvelope(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float,\n (_bbox->>3)::float\n ),4326)\n WHEN 6 THEN\n ST_SetSRID(ST_3DMakeBox(\n ST_MakePoint(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float\n ),\n ST_MakePoint(\n (_bbox->>3)::float,\n (_bbox->>4)::float,\n (_bbox->>5)::float\n )\n ),4326)\n ELSE null END;\n;\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION geom_bbox(_geom geometry) RETURNS jsonb AS $$\n SELECT jsonb_build_array(\n st_xmin(_geom),\n st_ymin(_geom),\n st_xmax(_geom),\n st_ymax(_geom)\n );\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION flip_jsonb_array(j jsonb) RETURNS jsonb AS $$\n SELECT jsonb_agg(value) FROM (SELECT value FROM jsonb_array_elements(j) WITH ORDINALITY ORDER BY ordinality DESC) as t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION explode_dotpaths(j jsonb) RETURNS SETOF text[] AS $$\n SELECT string_to_array(p, '.') as e FROM jsonb_array_elements_text(j) p;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION explode_dotpaths_recurse(IN j jsonb) RETURNS SETOF text[] AS $$\n WITH RECURSIVE t AS (\n SELECT e FROM explode_dotpaths(j) e\n UNION ALL\n SELECT e[1:cardinality(e)-1]\n FROM t\n WHERE cardinality(e)>1\n ) SELECT e FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION jsonb_set_nested(j jsonb, path text[], val jsonb) RETURNS jsonb AS $$\nDECLARE\nBEGIN\n IF cardinality(path) > 1 THEN\n FOR i IN 1..(cardinality(path)-1) LOOP\n IF j #> path[:i] IS NULL THEN\n j := jsonb_set_lax(j, path[:i], '{}', TRUE);\n END IF;\n END LOOP;\n END IF;\n RETURN jsonb_set_lax(j, path, val, true);\n\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\n\n\nCREATE OR REPLACE FUNCTION jsonb_include(j jsonb, f jsonb) RETURNS jsonb AS $$\nDECLARE\n includes jsonb := f-> 'include';\n outj jsonb := '{}'::jsonb;\n path text[];\nBEGIN\n IF\n includes IS NULL\n OR jsonb_array_length(includes) = 0\n THEN\n RETURN j;\n ELSE\n includes := includes || '[\"id\",\"collection\"]'::jsonb;\n FOR path IN SELECT explode_dotpaths(includes) LOOP\n outj := jsonb_set_nested(outj, path, j #> path);\n END LOOP;\n END IF;\n RETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_exclude(j jsonb, f jsonb) RETURNS jsonb AS $$\nDECLARE\n excludes jsonb := f-> 'exclude';\n outj jsonb := j;\n path text[];\nBEGIN\n IF\n excludes IS NULL\n OR jsonb_array_length(excludes) = 0\n THEN\n RETURN j;\n ELSE\n FOR path IN SELECT explode_dotpaths(excludes) LOOP\n outj := outj #- path;\n END LOOP;\n END IF;\n RETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_fields(j jsonb, f jsonb DEFAULT '{\"fields\":[]}') RETURNS jsonb AS $$\n SELECT jsonb_exclude(jsonb_include(j, f), f);\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION merge_jsonb(_a jsonb, _b jsonb) RETURNS jsonb AS $$\n SELECT\n CASE\n WHEN _a = '\"𒍟※\"'::jsonb THEN NULL\n WHEN _a IS NULL OR jsonb_typeof(_a) = 'null' THEN _b\n WHEN jsonb_typeof(_a) = 'object' AND jsonb_typeof(_b) = 'object' THEN\n (\n SELECT\n jsonb_strip_nulls(\n jsonb_object_agg(\n key,\n merge_jsonb(a.value, b.value)\n )\n )\n FROM\n jsonb_each(coalesce(_a,'{}'::jsonb)) as a\n FULL JOIN\n jsonb_each(coalesce(_b,'{}'::jsonb)) as b\n USING (key)\n )\n WHEN\n jsonb_typeof(_a) = 'array'\n AND jsonb_typeof(_b) = 'array'\n AND jsonb_array_length(_a) = jsonb_array_length(_b)\n THEN\n (\n SELECT jsonb_agg(m) FROM\n ( SELECT\n merge_jsonb(\n jsonb_array_elements(_a),\n jsonb_array_elements(_b)\n ) as m\n ) as l\n )\n ELSE _a\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION strip_jsonb(_a jsonb, _b jsonb) RETURNS jsonb AS $$\n SELECT\n CASE\n\n WHEN (_a IS NULL OR jsonb_typeof(_a) = 'null') AND _b IS NOT NULL AND jsonb_typeof(_b) != 'null' THEN '\"𒍟※\"'::jsonb\n WHEN _b IS NULL OR jsonb_typeof(_a) = 'null' THEN _a\n WHEN _a = _b AND jsonb_typeof(_a) = 'object' THEN '{}'::jsonb\n WHEN _a = _b THEN NULL\n WHEN jsonb_typeof(_a) = 'object' AND jsonb_typeof(_b) = 'object' THEN\n (\n SELECT\n jsonb_strip_nulls(\n jsonb_object_agg(\n key,\n strip_jsonb(a.value, b.value)\n )\n )\n FROM\n jsonb_each(_a) as a\n FULL JOIN\n jsonb_each(_b) as b\n USING (key)\n )\n WHEN\n jsonb_typeof(_a) = 'array'\n AND jsonb_typeof(_b) = 'array'\n AND jsonb_array_length(_a) = jsonb_array_length(_b)\n THEN\n (\n SELECT jsonb_agg(m) FROM\n ( SELECT\n strip_jsonb(\n jsonb_array_elements(_a),\n jsonb_array_elements(_b)\n ) as m\n ) as l\n )\n ELSE _a\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION nullif_jsonbnullempty(j jsonb) RETURNS jsonb AS $$\n SELECT nullif(nullif(nullif(j,'null'::jsonb),'{}'::jsonb),'[]'::jsonb);\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION jsonb_array_unique(j jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(jsonb_agg(DISTINCT a)) v FROM jsonb_array_elements(j) a;\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_concat_ignorenull(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT coalesce(a,'[]'::jsonb) || coalesce(b,'[]'::jsonb);\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_least(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(least(nullif_jsonbnullempty(a), nullif_jsonbnullempty(b)));\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_greatest(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(greatest(a, b));\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION first_notnull_sfunc(anyelement, anyelement) RETURNS anyelement AS $$\n SELECT COALESCE($1,$2);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE AGGREGATE first_notnull(anyelement)(\n SFUNC = first_notnull_sfunc,\n STYPE = anyelement\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_array_unique_merge(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_concat_ignorenull,\n FINALFUNC = jsonb_array_unique\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_min(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_least\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_max(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_greatest\n);\n/* looks for a geometry in a stac item first from geometry and falling back to bbox */\nCREATE OR REPLACE FUNCTION stac_geom(value jsonb) RETURNS geometry AS $$\nSELECT\n CASE\n WHEN value ? 'intersects' THEN\n ST_GeomFromGeoJSON(value->>'intersects')\n WHEN value ? 'geometry' THEN\n ST_GeomFromGeoJSON(value->>'geometry')\n WHEN value ? 'bbox' THEN\n pgstac.bbox_geom(value->'bbox')\n ELSE NULL\n END as geometry\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION stac_daterange(\n value jsonb\n) RETURNS tstzrange AS $$\nDECLARE\n props jsonb := value;\n dt timestamptz;\n edt timestamptz;\nBEGIN\n IF props ? 'properties' THEN\n props := props->'properties';\n END IF;\n IF\n props ? 'start_datetime'\n AND props->>'start_datetime' IS NOT NULL\n AND props ? 'end_datetime'\n AND props->>'end_datetime' IS NOT NULL\n THEN\n dt := props->>'start_datetime';\n edt := props->>'end_datetime';\n IF dt > edt THEN\n RAISE EXCEPTION 'start_datetime must be < end_datetime';\n END IF;\n ELSE\n dt := props->>'datetime';\n edt := props->>'datetime';\n END IF;\n IF dt is NULL OR edt IS NULL THEN\n RAISE NOTICE 'DT: %, EDT: %', dt, edt;\n RAISE EXCEPTION 'Either datetime (%) or both start_datetime (%) and end_datetime (%) must be set.', props->>'datetime',props->>'start_datetime',props->>'end_datetime';\n END IF;\n RETURN tstzrange(dt, edt, '[]');\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_datetime(value jsonb) RETURNS timestamptz AS $$\n SELECT lower(stac_daterange(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_end_datetime(value jsonb) RETURNS timestamptz AS $$\n SELECT upper(stac_daterange(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE TABLE stac_extensions(\n url text PRIMARY KEY,\n content jsonb\n);\n\n\n\nCREATE OR REPLACE FUNCTION collection_base_item(content jsonb) RETURNS jsonb AS $$\n SELECT jsonb_build_object(\n 'type', 'Feature',\n 'stac_version', content->'stac_version',\n 'assets', content->'item_assets',\n 'collection', content->'id'\n );\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TYPE partition_trunc_strategy AS ENUM ('year', 'month');\n\nCREATE TABLE IF NOT EXISTS collections (\n key bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,\n id text GENERATED ALWAYS AS (content->>'id') STORED UNIQUE,\n content JSONB NOT NULL,\n base_item jsonb GENERATED ALWAYS AS (pgstac.collection_base_item(content)) STORED,\n partition_trunc partition_trunc_strategy\n);\n\n\nCREATE OR REPLACE FUNCTION collection_base_item(cid text) RETURNS jsonb AS $$\n SELECT pgstac.collection_base_item(content) FROM pgstac.collections WHERE id = cid LIMIT 1;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION table_empty(text) RETURNS boolean AS $$\nDECLARE\n retval boolean;\nBEGIN\n EXECUTE format($q$\n SELECT NOT EXISTS (SELECT 1 FROM %I LIMIT 1)\n $q$,\n $1\n ) INTO retval;\n RETURN retval;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION collections_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n q text;\n partition_name text := format('_items_%s', NEW.key);\n partition_exists boolean := false;\n partition_empty boolean := true;\n err_context text;\n loadtemp boolean := FALSE;\nBEGIN\n RAISE NOTICE 'Collection Trigger. % %', NEW.id, NEW.key;\n SELECT relid::text INTO partition_name\n FROM pg_partition_tree('items')\n WHERE relid::text = partition_name;\n IF FOUND THEN\n partition_exists := true;\n partition_empty := table_empty(partition_name);\n ELSE\n partition_exists := false;\n partition_empty := true;\n partition_name := format('_items_%s', NEW.key);\n END IF;\n IF TG_OP = 'UPDATE' AND NEW.partition_trunc IS DISTINCT FROM OLD.partition_trunc AND partition_empty THEN\n q := format($q$\n DROP TABLE IF EXISTS %I CASCADE;\n $q$,\n partition_name\n );\n EXECUTE q;\n END IF;\n IF TG_OP = 'UPDATE' AND NEW.partition_trunc IS DISTINCT FROM OLD.partition_trunc AND partition_exists AND NOT partition_empty THEN\n q := format($q$\n CREATE TEMP TABLE changepartitionstaging ON COMMIT DROP AS SELECT * FROM %I;\n DROP TABLE IF EXISTS %I CASCADE;\n $q$,\n partition_name,\n partition_name\n );\n EXECUTE q;\n loadtemp := TRUE;\n partition_empty := TRUE;\n partition_exists := FALSE;\n END IF;\n IF TG_OP = 'UPDATE' AND NEW.partition_trunc IS NOT DISTINCT FROM OLD.partition_trunc THEN\n RETURN NEW;\n END IF;\n IF NEW.partition_trunc IS NULL AND partition_empty THEN\n RAISE NOTICE '% % % %',\n partition_name,\n NEW.id,\n concat(partition_name,'_id_idx'),\n partition_name\n ;\n q := format($q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n $q$,\n partition_name,\n NEW.id,\n concat(partition_name,'_id_idx'),\n partition_name\n );\n RAISE NOTICE 'q: %', q;\n BEGIN\n EXECUTE q;\n EXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', partition_name;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\n END;\n\n ALTER TABLE partitions DISABLE TRIGGER partitions_delete_trigger;\n DELETE FROM partitions WHERE collection=NEW.id AND name=partition_name;\n ALTER TABLE partitions ENABLE TRIGGER partitions_delete_trigger;\n\n INSERT INTO partitions (collection, name) VALUES (NEW.id, partition_name);\n ELSIF partition_empty THEN\n q := format($q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L)\n PARTITION BY RANGE (datetime);\n $q$,\n partition_name,\n NEW.id\n );\n RAISE NOTICE 'q: %', q;\n BEGIN\n EXECUTE q;\n EXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', partition_name;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\n END;\n ALTER TABLE partitions DISABLE TRIGGER partitions_delete_trigger;\n DELETE FROM partitions WHERE collection=NEW.id AND name=partition_name;\n ALTER TABLE partitions ENABLE TRIGGER partitions_delete_trigger;\n ELSE\n RAISE EXCEPTION 'Cannot modify partition % unless empty', partition_name;\n END IF;\n IF loadtemp THEN\n RAISE NOTICE 'Moving data into new partitions.';\n q := format($q$\n WITH p AS (\n SELECT\n collection,\n datetime as datetime,\n end_datetime as end_datetime,\n (partition_name(\n collection,\n datetime\n )).partition_name as name\n FROM changepartitionstaging\n )\n INSERT INTO partitions (collection, datetime_range, end_datetime_range)\n SELECT\n collection,\n tstzrange(min(datetime), max(datetime), '[]') as datetime_range,\n tstzrange(min(end_datetime), max(end_datetime), '[]') as end_datetime_range\n FROM p\n GROUP BY collection, name\n ON CONFLICT (name) DO UPDATE SET\n datetime_range = EXCLUDED.datetime_range,\n end_datetime_range = EXCLUDED.end_datetime_range\n ;\n INSERT INTO %I SELECT * FROM changepartitionstaging;\n DROP TABLE IF EXISTS changepartitionstaging;\n $q$,\n partition_name\n );\n EXECUTE q;\n END IF;\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE TRIGGER collections_trigger AFTER INSERT OR UPDATE ON collections FOR EACH ROW\nEXECUTE FUNCTION collections_trigger_func();\n\nCREATE OR REPLACE FUNCTION partition_collection(collection text, strategy partition_trunc_strategy) RETURNS text AS $$\n UPDATE collections SET partition_trunc=strategy WHERE id=collection RETURNING partition_trunc;\n$$ LANGUAGE SQL;\n\nCREATE TABLE IF NOT EXISTS partitions (\n collection text REFERENCES collections(id) ON DELETE CASCADE,\n name text PRIMARY KEY,\n partition_range tstzrange NOT NULL DEFAULT tstzrange('-infinity'::timestamptz,'infinity'::timestamptz, '[]'),\n datetime_range tstzrange,\n end_datetime_range tstzrange,\n CONSTRAINT prange EXCLUDE USING GIST (\n collection WITH =,\n partition_range WITH &&\n )\n) WITH (FILLFACTOR=90);\nCREATE INDEX partitions_range_idx ON partitions USING GIST(partition_range);\n\nCREATE OR REPLACE FUNCTION partitions_delete_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n q text;\nBEGIN\n RAISE NOTICE 'Partition Delete Trigger. %', OLD.name;\n EXECUTE format($q$\n DROP TABLE IF EXISTS %I CASCADE;\n $q$,\n OLD.name\n );\n RAISE NOTICE 'Dropped partition.';\n RETURN OLD;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER partitions_delete_trigger BEFORE DELETE ON partitions FOR EACH ROW\nEXECUTE FUNCTION partitions_delete_trigger_func();\n\nCREATE OR REPLACE FUNCTION partition_name(\n IN collection text,\n IN dt timestamptz,\n OUT partition_name text,\n OUT partition_range tstzrange\n) AS $$\nDECLARE\n c RECORD;\n parent_name text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n parent_name := format('_items_%s', c.key);\n\n\n IF c.partition_trunc = 'year' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYY'));\n ELSIF c.partition_trunc = 'month' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYYMM'));\n ELSE\n partition_name := parent_name;\n partition_range := tstzrange('-infinity'::timestamptz, 'infinity'::timestamptz, '[]');\n END IF;\n IF partition_range IS NULL THEN\n partition_range := tstzrange(\n date_trunc(c.partition_trunc::text, dt),\n date_trunc(c.partition_trunc::text, dt) + concat('1 ', c.partition_trunc)::interval\n );\n END IF;\n RETURN;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\n\nCREATE OR REPLACE FUNCTION partitions_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n q text;\n cq text;\n parent_name text;\n partition_trunc text;\n partition_name text := NEW.name;\n partition_exists boolean := false;\n partition_empty boolean := true;\n partition_range tstzrange;\n datetime_range tstzrange;\n end_datetime_range tstzrange;\n err_context text;\n mindt timestamptz := lower(NEW.datetime_range);\n maxdt timestamptz := upper(NEW.datetime_range);\n minedt timestamptz := lower(NEW.end_datetime_range);\n maxedt timestamptz := upper(NEW.end_datetime_range);\n t_mindt timestamptz;\n t_maxdt timestamptz;\n t_minedt timestamptz;\n t_maxedt timestamptz;\nBEGIN\n RAISE NOTICE 'Partitions Trigger. %', NEW;\n datetime_range := NEW.datetime_range;\n end_datetime_range := NEW.end_datetime_range;\n\n SELECT\n format('_items_%s', key),\n c.partition_trunc::text\n INTO\n parent_name,\n partition_trunc\n FROM pgstac.collections c\n WHERE c.id = NEW.collection;\n SELECT (pgstac.partition_name(NEW.collection, mindt)).* INTO partition_name, partition_range;\n NEW.name := partition_name;\n\n IF partition_range IS NULL OR partition_range = 'empty'::tstzrange THEN\n partition_range := tstzrange('-infinity'::timestamptz, 'infinity'::timestamptz, '[]');\n END IF;\n\n NEW.partition_range := partition_range;\n IF TG_OP = 'UPDATE' THEN\n mindt := least(mindt, lower(OLD.datetime_range));\n maxdt := greatest(maxdt, upper(OLD.datetime_range));\n minedt := least(minedt, lower(OLD.end_datetime_range));\n maxedt := greatest(maxedt, upper(OLD.end_datetime_range));\n NEW.datetime_range := tstzrange(mindt, maxdt, '[]');\n NEW.end_datetime_range := tstzrange(minedt, maxedt, '[]');\n END IF;\n IF TG_OP = 'INSERT' THEN\n\n IF partition_range != tstzrange('-infinity'::timestamptz, 'infinity'::timestamptz, '[]') THEN\n\n RAISE NOTICE '% % %', partition_name, parent_name, partition_range;\n q := format($q$\n CREATE TABLE IF NOT EXISTS %I partition OF %I FOR VALUES FROM (%L) TO (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n $q$,\n partition_name,\n parent_name,\n lower(partition_range),\n upper(partition_range),\n format('%s_pkey', partition_name),\n partition_name\n );\n BEGIN\n EXECUTE q;\n EXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', partition_name;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\n END;\n END IF;\n\n END IF;\n\n -- Update constraints\n EXECUTE format($q$\n SELECT\n min(datetime),\n max(datetime),\n min(end_datetime),\n max(end_datetime)\n FROM %I;\n $q$, partition_name)\n INTO t_mindt, t_maxdt, t_minedt, t_maxedt;\n mindt := least(mindt, t_mindt);\n maxdt := greatest(maxdt, t_maxdt);\n minedt := least(mindt, minedt, t_minedt);\n maxedt := greatest(maxdt, maxedt, t_maxedt);\n\n mindt := date_trunc(coalesce(partition_trunc, 'year'), mindt);\n maxdt := date_trunc(coalesce(partition_trunc, 'year'), maxdt - '1 second'::interval) + concat('1 ',coalesce(partition_trunc, 'year'))::interval;\n minedt := date_trunc(coalesce(partition_trunc, 'year'), minedt);\n maxedt := date_trunc(coalesce(partition_trunc, 'year'), maxedt - '1 second'::interval) + concat('1 ',coalesce(partition_trunc, 'year'))::interval;\n\n\n IF mindt IS NOT NULL AND maxdt IS NOT NULL AND minedt IS NOT NULL AND maxedt IS NOT NULL THEN\n NEW.datetime_range := tstzrange(mindt, maxdt, '[]');\n NEW.end_datetime_range := tstzrange(minedt, maxedt, '[]');\n IF\n TG_OP='UPDATE'\n AND OLD.datetime_range @> NEW.datetime_range\n AND OLD.end_datetime_range @> NEW.end_datetime_range\n THEN\n RAISE NOTICE 'Range unchanged, not updating constraints.';\n ELSE\n\n RAISE NOTICE '\n SETTING CONSTRAINTS\n mindt: %, maxdt: %\n minedt: %, maxedt: %\n ', mindt, maxdt, minedt, maxedt;\n IF partition_trunc IS NULL THEN\n cq := format($q$\n ALTER TABLE %7$I\n DROP CONSTRAINT IF EXISTS %1$I,\n DROP CONSTRAINT IF EXISTS %2$I,\n ADD CONSTRAINT %1$I\n CHECK (\n (datetime >= %3$L)\n AND (datetime <= %4$L)\n AND (end_datetime >= %5$L)\n AND (end_datetime <= %6$L)\n ) NOT VALID\n ;\n ALTER TABLE %7$I\n VALIDATE CONSTRAINT %1$I;\n $q$,\n format('%s_dt', partition_name),\n format('%s_edt', partition_name),\n mindt,\n maxdt,\n minedt,\n maxedt,\n partition_name\n );\n ELSE\n cq := format($q$\n ALTER TABLE %5$I\n DROP CONSTRAINT IF EXISTS %1$I,\n DROP CONSTRAINT IF EXISTS %2$I,\n ADD CONSTRAINT %2$I\n CHECK ((end_datetime >= %3$L) AND (end_datetime <= %4$L)) NOT VALID\n ;\n ALTER TABLE %5$I\n VALIDATE CONSTRAINT %2$I;\n $q$,\n format('%s_dt', partition_name),\n format('%s_edt', partition_name),\n minedt,\n maxedt,\n partition_name\n );\n\n END IF;\n RAISE NOTICE 'Altering Constraints. %', cq;\n EXECUTE cq;\n END IF;\n ELSE\n NEW.datetime_range = NULL;\n NEW.end_datetime_range = NULL;\n\n cq := format($q$\n ALTER TABLE %3$I\n DROP CONSTRAINT IF EXISTS %1$I,\n DROP CONSTRAINT IF EXISTS %2$I,\n ADD CONSTRAINT %1$I\n CHECK ((datetime IS NULL AND end_datetime IS NULL)) NOT VALID\n ;\n ALTER TABLE %3$I\n VALIDATE CONSTRAINT %1$I;\n $q$,\n format('%s_dt', partition_name),\n format('%s_edt', partition_name),\n partition_name\n );\n EXECUTE cq;\n END IF;\n\n RETURN NEW;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER partitions_trigger BEFORE INSERT OR UPDATE ON partitions FOR EACH ROW\nEXECUTE FUNCTION partitions_trigger_func();\n\n\nCREATE OR REPLACE FUNCTION create_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection(data jsonb) RETURNS VOID AS $$\nDECLARE\n out collections%ROWTYPE;\nBEGIN\n UPDATE collections SET content=data WHERE id = data->>'id' RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ON CONFLICT (id) DO\n UPDATE\n SET content=EXCLUDED.content\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION delete_collection(_id text) RETURNS VOID AS $$\nDECLARE\n out collections%ROWTYPE;\nBEGIN\n DELETE FROM collections WHERE id = _id RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION get_collection(id text) RETURNS jsonb AS $$\n SELECT content FROM collections\n WHERE id=$1\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION all_collections() RETURNS jsonb AS $$\n SELECT jsonb_agg(content) FROM collections;\n;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\nCREATE TABLE queryables (\n id bigint GENERATED ALWAYS AS identity PRIMARY KEY,\n name text NOT NULL,\n collection_ids text[], -- used to determine what partitions to create indexes on\n definition jsonb,\n property_path text,\n property_wrapper text,\n property_index_type text\n);\nCREATE INDEX queryables_name_idx ON queryables (name);\nCREATE INDEX queryables_property_wrapper_idx ON queryables (property_wrapper);\n\n\nINSERT INTO queryables (name, definition) VALUES\n('id', '{\"title\": \"Item ID\",\"description\": \"Item identifier\",\"$ref\": \"https://schemas.stacspec.org/v1.0.0/item-spec/json-schema/item.json#/definitions/core/allOf/2/properties/id\"}'),\n('datetime','{\"description\": \"Datetime\",\"type\": \"string\",\"title\": \"Acquired\",\"format\": \"date-time\",\"pattern\": \"(\\\\+00:00|Z)$\"}'),\n('geometry', '{\"title\": \"Item Geometry\",\"description\": \"Item Geometry\",\"$ref\": \"https://geojson.org/schema/Feature.json\"}')\nON CONFLICT DO NOTHING;\n\nINSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n('eo:cloud_cover','{\"$ref\": \"https://stac-extensions.github.io/eo/v1.0.0/schema.json#/definitions/fieldsproperties/eo:cloud_cover\"}','to_int','BTREE')\nON CONFLICT DO NOTHING;\n\nCREATE OR REPLACE FUNCTION array_to_path(arr text[]) RETURNS text AS $$\n SELECT string_agg(\n quote_literal(v),\n '->'\n ) FROM unnest(arr) v;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\n\n\nCREATE OR REPLACE FUNCTION queryable(\n IN dotpath text,\n OUT path text,\n OUT expression text,\n OUT wrapper text,\n OUT nulled_wrapper text\n) AS $$\nDECLARE\n q RECORD;\n path_elements text[];\nBEGIN\n IF dotpath IN ('id', 'geometry', 'datetime', 'end_datetime', 'collection') THEN\n path := dotpath;\n expression := dotpath;\n wrapper := NULL;\n RETURN;\n END IF;\n SELECT * INTO q FROM queryables\n WHERE\n name=dotpath\n OR name = 'properties.' || dotpath\n OR name = replace(dotpath, 'properties.', '')\n ;\n IF q.property_wrapper IS NULL THEN\n IF q.definition->>'type' = 'number' THEN\n wrapper := 'to_float';\n nulled_wrapper := wrapper;\n ELSIF q.definition->>'format' = 'date-time' THEN\n wrapper := 'to_tstz';\n nulled_wrapper := wrapper;\n ELSE\n nulled_wrapper := NULL;\n wrapper := 'to_text';\n END IF;\n ELSE\n wrapper := q.property_wrapper;\n nulled_wrapper := wrapper;\n END IF;\n IF q.property_path IS NOT NULL THEN\n path := q.property_path;\n ELSE\n path_elements := string_to_array(dotpath, '.');\n IF path_elements[1] IN ('links', 'assets', 'stac_version', 'stac_extensions') THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSIF path_elements[1] = 'properties' THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSE\n path := format($F$content->'properties'->%s$F$, array_to_path(path_elements));\n END IF;\n END IF;\n expression := format('%I(%s)', wrapper, path);\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\nCREATE OR REPLACE FUNCTION create_queryable_indexes() RETURNS VOID AS $$\nDECLARE\n queryable RECORD;\n q text;\nBEGIN\n FOR queryable IN\n SELECT\n queryables.id as qid,\n CASE WHEN collections.key IS NULL THEN 'items' ELSE format('_items_%s',collections.key) END AS part,\n property_index_type,\n expression\n FROM\n queryables\n LEFT JOIN collections ON (collections.id = ANY (queryables.collection_ids))\n JOIN LATERAL queryable(queryables.name) ON (queryables.property_index_type IS NOT NULL)\n LOOP\n q := format(\n $q$\n CREATE INDEX IF NOT EXISTS %I ON %I USING %s ((%s));\n $q$,\n format('%s_%s_idx', queryable.part, queryable.qid),\n queryable.part,\n COALESCE(queryable.property_index_type, 'to_text'),\n queryable.expression\n );\n RAISE NOTICE '%',q;\n EXECUTE q;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION queryables_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\nBEGIN\nPERFORM create_queryable_indexes();\nRETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER queryables_trigger AFTER INSERT OR UPDATE ON queryables\nFOR EACH STATEMENT EXECUTE PROCEDURE queryables_trigger_func();\n\nCREATE TRIGGER queryables_collection_trigger AFTER INSERT OR UPDATE ON collections\nFOR EACH STATEMENT EXECUTE PROCEDURE queryables_trigger_func();\n\n\nCREATE OR REPLACE FUNCTION get_queryables(_collection_ids text[] DEFAULT NULL) RETURNS jsonb AS $$\nDECLARE\nBEGIN\n -- Build up queryables if the input contains valid collection ids or is empty\n IF EXISTS (\n SELECT 1 FROM collections\n WHERE\n _collection_ids IS NULL\n OR cardinality(_collection_ids) = 0\n OR id = ANY(_collection_ids)\n )\n THEN\n RETURN (\n WITH base AS (\n SELECT\n unnest(collection_ids) as collection_id,\n name,\n coalesce(definition, '{\"type\":\"string\"}'::jsonb) as definition\n FROM queryables\n WHERE\n _collection_ids IS NULL OR\n _collection_ids = '{}'::text[] OR\n _collection_ids && collection_ids\n UNION ALL\n SELECT null, name, coalesce(definition, '{\"type\":\"string\"}'::jsonb) as definition\n FROM queryables WHERE collection_ids IS NULL OR collection_ids = '{}'::text[]\n ), g AS (\n SELECT\n name,\n first_notnull(definition) as definition,\n jsonb_array_unique_merge(definition->'enum') as enum,\n jsonb_min(definition->'minimum') as minimum,\n jsonb_min(definition->'maxiumn') as maximum\n FROM base\n GROUP BY 1\n )\n SELECT\n jsonb_build_object(\n '$schema', 'http://json-schema.org/draft-07/schema#',\n '$id', '',\n 'type', 'object',\n 'title', 'STAC Queryables.',\n 'properties', jsonb_object_agg(\n name,\n definition\n ||\n jsonb_strip_nulls(jsonb_build_object(\n 'enum', enum,\n 'minimum', minimum,\n 'maximum', maximum\n ))\n )\n )\n FROM g\n );\n ELSE\n RETURN NULL;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\nCREATE OR REPLACE FUNCTION get_queryables(_collection text DEFAULT NULL) RETURNS jsonb AS $$\n SELECT\n CASE\n WHEN _collection IS NULL THEN get_queryables(NULL::text[])\n ELSE get_queryables(ARRAY[_collection])\n END\n ;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_queryables() RETURNS jsonb AS $$\n SELECT get_queryables(NULL::text[]);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION schema_qualify_refs(url text, j jsonb) returns jsonb as $$\n SELECT regexp_replace(j::text, '\"\\$ref\": \"#', concat('\"$ref\": \"', url, '#'), 'g')::jsonb;\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE VIEW stac_extension_queryables AS\nSELECT DISTINCT key as name, schema_qualify_refs(e.url, j.value) as definition FROM stac_extensions e, jsonb_each(e.content->'definitions'->'fields'->'properties') j;\n\n\nCREATE OR REPLACE FUNCTION missing_queryables(_collection text, _tablesample float DEFAULT 5, minrows float DEFAULT 10) RETURNS TABLE(collection text, name text, definition jsonb, property_wrapper text) AS $$\nDECLARE\n q text;\n _partition text;\n explain_json json;\n psize float;\n estrows float;\nBEGIN\n SELECT format('_items_%s', key) INTO _partition FROM collections WHERE id=_collection;\n\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM %I;', _partition)\n INTO explain_json;\n psize := explain_json->0->'Plan'->'Plan Rows';\n estrows := _tablesample * .01 * psize;\n IF estrows < minrows THEN\n _tablesample := least(100,greatest(_tablesample, (estrows / psize) / 100));\n RAISE NOTICE '%', (psize / estrows) / 100;\n END IF;\n RAISE NOTICE 'Using tablesample % to find missing queryables from % % that has ~% rows estrows: %', _tablesample, _collection, _partition, psize, estrows;\n\n q := format(\n $q$\n WITH q AS (\n SELECT * FROM queryables\n WHERE\n collection_ids IS NULL\n OR %L = ANY(collection_ids)\n ), t AS (\n SELECT\n content->'properties' AS properties\n FROM\n %I\n TABLESAMPLE SYSTEM(%L)\n ), p AS (\n SELECT DISTINCT ON (key)\n key,\n value,\n s.definition\n FROM t\n JOIN LATERAL jsonb_each(properties) ON TRUE\n LEFT JOIN q ON (q.name=key)\n LEFT JOIN stac_extension_queryables s ON (s.name=key)\n WHERE q.definition IS NULL\n )\n SELECT\n %L,\n key,\n COALESCE(definition, jsonb_build_object('type',jsonb_typeof(value))) as definition,\n CASE\n WHEN definition->>'type' = 'integer' THEN 'to_int'\n WHEN COALESCE(definition->>'type', jsonb_typeof(value)) = 'number' THEN 'to_float'\n WHEN COALESCE(definition->>'type', jsonb_typeof(value)) = 'array' THEN 'to_text_array'\n ELSE 'to_text'\n END\n FROM p;\n $q$,\n _collection,\n _partition,\n _tablesample,\n _collection\n );\n RETURN QUERY EXECUTE q;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION missing_queryables(_tablesample float DEFAULT 5) RETURNS TABLE(collection_ids text[], name text, definition jsonb, property_wrapper text) AS $$\n SELECT\n array_agg(collection),\n name,\n definition,\n property_wrapper\n FROM\n collections\n JOIN LATERAL\n missing_queryables(id, _tablesample) c\n ON TRUE\n GROUP BY\n 2,3,4\n ORDER BY 2,1\n ;\n$$ LANGUAGE SQL;\nCREATE OR REPLACE FUNCTION parse_dtrange(\n _indate jsonb,\n relative_base timestamptz DEFAULT date_trunc('hour', CURRENT_TIMESTAMP)\n) RETURNS tstzrange AS $$\nDECLARE\n timestrs text[];\n s timestamptz;\n e timestamptz;\nBEGIN\n timestrs :=\n CASE\n WHEN _indate ? 'timestamp' THEN\n ARRAY[_indate->>'timestamp']\n WHEN _indate ? 'interval' THEN\n to_text_array(_indate->'interval')\n WHEN jsonb_typeof(_indate) = 'array' THEN\n to_text_array(_indate)\n ELSE\n regexp_split_to_array(\n _indate->>0,\n '/'\n )\n END;\n RAISE NOTICE 'TIMESTRS %', timestrs;\n IF cardinality(timestrs) = 1 THEN\n IF timestrs[1] ILIKE 'P%' THEN\n RETURN tstzrange(relative_base - upper(timestrs[1])::interval, relative_base, '[)');\n END IF;\n s := timestrs[1]::timestamptz;\n RETURN tstzrange(s, s, '[]');\n END IF;\n\n IF cardinality(timestrs) != 2 THEN\n RAISE EXCEPTION 'Timestamp cannot have more than 2 values';\n END IF;\n\n IF timestrs[1] = '..' THEN\n s := '-infinity'::timestamptz;\n e := timestrs[2]::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] = '..' THEN\n s := timestrs[1]::timestamptz;\n e := 'infinity'::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[1] ILIKE 'P%' AND timestrs[2] NOT ILIKE 'P%' THEN\n e := timestrs[2]::timestamptz;\n s := e - upper(timestrs[1])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] ILIKE 'P%' AND timestrs[1] NOT ILIKE 'P%' THEN\n s := timestrs[1]::timestamptz;\n e := s + upper(timestrs[2])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n s := timestrs[1]::timestamptz;\n e := timestrs[2]::timestamptz;\n\n RETURN tstzrange(s,e,'[)');\n\n RETURN NULL;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT PARALLEL SAFE SET TIME ZONE 'UTC';\n\nCREATE OR REPLACE FUNCTION parse_dtrange(\n _indate text,\n relative_base timestamptz DEFAULT CURRENT_TIMESTAMP\n) RETURNS tstzrange AS $$\n SELECT parse_dtrange(to_jsonb(_indate), relative_base);\n$$ LANGUAGE SQL STABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION temporal_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\n ll text := 'datetime';\n lh text := 'end_datetime';\n rrange tstzrange;\n rl text;\n rh text;\n outq text;\nBEGIN\n rrange := parse_dtrange(args->1);\n RAISE NOTICE 'Constructing temporal query OP: %, ARGS: %, RRANGE: %', op, args, rrange;\n op := lower(op);\n rl := format('%L::timestamptz', lower(rrange));\n rh := format('%L::timestamptz', upper(rrange));\n outq := CASE op\n WHEN 't_before' THEN 'lh < rl'\n WHEN 't_after' THEN 'll > rh'\n WHEN 't_meets' THEN 'lh = rl'\n WHEN 't_metby' THEN 'll = rh'\n WHEN 't_overlaps' THEN 'll < rl AND rl < lh < rh'\n WHEN 't_overlappedby' THEN 'rl < ll < rh AND lh > rh'\n WHEN 't_starts' THEN 'll = rl AND lh < rh'\n WHEN 't_startedby' THEN 'll = rl AND lh > rh'\n WHEN 't_during' THEN 'll > rl AND lh < rh'\n WHEN 't_contains' THEN 'll < rl AND lh > rh'\n WHEN 't_finishes' THEN 'll > rl AND lh = rh'\n WHEN 't_finishedby' THEN 'll < rl AND lh = rh'\n WHEN 't_equals' THEN 'll = rl AND lh = rh'\n WHEN 't_disjoint' THEN 'NOT (ll <= rh AND lh >= rl)'\n WHEN 't_intersects' THEN 'll <= rh AND lh >= rl'\n WHEN 'anyinteracts' THEN 'll <= rh AND lh >= rl'\n END;\n outq := regexp_replace(outq, '\\mll\\M', ll);\n outq := regexp_replace(outq, '\\mlh\\M', lh);\n outq := regexp_replace(outq, '\\mrl\\M', rl);\n outq := regexp_replace(outq, '\\mrh\\M', rh);\n outq := format('(%s)', outq);\n RETURN outq;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\n\n\nCREATE OR REPLACE FUNCTION spatial_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\n geom text;\n j jsonb := args->1;\nBEGIN\n op := lower(op);\n RAISE NOTICE 'Constructing spatial query OP: %, ARGS: %', op, args;\n IF op NOT IN ('s_equals','s_disjoint','s_touches','s_within','s_overlaps','s_crosses','s_intersects','intersects','s_contains') THEN\n RAISE EXCEPTION 'Spatial Operator % Not Supported', op;\n END IF;\n op := regexp_replace(op, '^s_', 'st_');\n IF op = 'intersects' THEN\n op := 'st_intersects';\n END IF;\n -- Convert geometry to WKB string\n IF j ? 'type' AND j ? 'coordinates' THEN\n geom := st_geomfromgeojson(j)::text;\n ELSIF jsonb_typeof(j) = 'array' THEN\n geom := bbox_geom(j)::text;\n END IF;\n\n RETURN format('%s(geometry, %L::geometry)', op, geom);\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION query_to_cql2(q jsonb) RETURNS jsonb AS $$\n-- Translates anything passed in through the deprecated \"query\" into equivalent CQL2\nWITH t AS (\n SELECT key as property, value as ops\n FROM jsonb_each(q)\n), t2 AS (\n SELECT property, (jsonb_each(ops)).*\n FROM t WHERE jsonb_typeof(ops) = 'object'\n UNION ALL\n SELECT property, 'eq', ops\n FROM t WHERE jsonb_typeof(ops) != 'object'\n)\nSELECT\n jsonb_strip_nulls(jsonb_build_object(\n 'op', 'and',\n 'args', jsonb_agg(\n jsonb_build_object(\n 'op', key,\n 'args', jsonb_build_array(\n jsonb_build_object('property',property),\n value\n )\n )\n )\n )\n) as qcql FROM t2\n;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\nCREATE OR REPLACE FUNCTION cql1_to_cql2(j jsonb) RETURNS jsonb AS $$\nDECLARE\n args jsonb;\n ret jsonb;\nBEGIN\n RAISE NOTICE 'CQL1_TO_CQL2: %', j;\n IF j ? 'filter' THEN\n RETURN cql1_to_cql2(j->'filter');\n END IF;\n IF j ? 'property' THEN\n RETURN j;\n END IF;\n IF jsonb_typeof(j) = 'array' THEN\n SELECT jsonb_agg(cql1_to_cql2(el)) INTO args FROM jsonb_array_elements(j) el;\n RETURN args;\n END IF;\n IF jsonb_typeof(j) = 'number' THEN\n RETURN j;\n END IF;\n IF jsonb_typeof(j) = 'string' THEN\n RETURN j;\n END IF;\n\n IF jsonb_typeof(j) = 'object' THEN\n SELECT jsonb_build_object(\n 'op', key,\n 'args', cql1_to_cql2(value)\n ) INTO ret\n FROM jsonb_each(j)\n WHERE j IS NOT NULL;\n RETURN ret;\n END IF;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE STRICT;\n\nCREATE TABLE cql2_ops (\n op text PRIMARY KEY,\n template text,\n types text[]\n);\nINSERT INTO cql2_ops (op, template, types) VALUES\n ('eq', '%s = %s', NULL),\n ('neq', '%s != %s', NULL),\n ('ne', '%s != %s', NULL),\n ('!=', '%s != %s', NULL),\n ('<>', '%s != %s', NULL),\n ('lt', '%s < %s', NULL),\n ('lte', '%s <= %s', NULL),\n ('gt', '%s > %s', NULL),\n ('gte', '%s >= %s', NULL),\n ('le', '%s <= %s', NULL),\n ('ge', '%s >= %s', NULL),\n ('=', '%s = %s', NULL),\n ('<', '%s < %s', NULL),\n ('<=', '%s <= %s', NULL),\n ('>', '%s > %s', NULL),\n ('>=', '%s >= %s', NULL),\n ('like', '%s LIKE %s', NULL),\n ('ilike', '%s ILIKE %s', NULL),\n ('+', '%s + %s', NULL),\n ('-', '%s - %s', NULL),\n ('*', '%s * %s', NULL),\n ('/', '%s / %s', NULL),\n ('not', 'NOT (%s)', NULL),\n ('between', '%s BETWEEN %s AND %s', NULL),\n ('isnull', '%s IS NULL', NULL),\n ('upper', 'upper(%s)', NULL),\n ('lower', 'lower(%s)', NULL)\nON CONFLICT (op) DO UPDATE\n SET\n template = EXCLUDED.template\n;\n\n\nCREATE OR REPLACE FUNCTION cql2_query(j jsonb, wrapper text DEFAULT NULL) RETURNS text AS $$\n#variable_conflict use_variable\nDECLARE\n args jsonb := j->'args';\n arg jsonb;\n op text := lower(j->>'op');\n cql2op RECORD;\n literal text;\n _wrapper text;\n leftarg text;\n rightarg text;\nBEGIN\n IF j IS NULL OR (op IS NOT NULL AND args IS NULL) THEN\n RETURN NULL;\n END IF;\n RAISE NOTICE 'CQL2_QUERY: %', j;\n IF j ? 'filter' THEN\n RETURN cql2_query(j->'filter');\n END IF;\n\n IF j ? 'upper' THEN\n RETURN cql2_query(jsonb_build_object('op', 'upper', 'args', j->'upper'));\n END IF;\n\n IF j ? 'lower' THEN\n RETURN cql2_query(jsonb_build_object('op', 'lower', 'args', j->'lower'));\n END IF;\n\n -- Temporal Query\n IF op ilike 't_%' or op = 'anyinteracts' THEN\n RETURN temporal_op_query(op, args);\n END IF;\n\n -- If property is a timestamp convert it to text to use with\n -- general operators\n IF j ? 'timestamp' THEN\n RETURN format('%L::timestamptz', to_tstz(j->'timestamp'));\n END IF;\n IF j ? 'interval' THEN\n RAISE EXCEPTION 'Please use temporal operators when using intervals.';\n RETURN NONE;\n END IF;\n\n -- Spatial Query\n IF op ilike 's_%' or op = 'intersects' THEN\n RETURN spatial_op_query(op, args);\n END IF;\n\n IF op IN ('a_equals','a_contains','a_contained_by','a_overlaps') THEN\n IF args->0 ? 'property' THEN\n leftarg := format('to_text_array(%s)', (queryable(args->0->>'property')).path);\n END IF;\n IF args->1 ? 'property' THEN\n rightarg := format('to_text_array(%s)', (queryable(args->1->>'property')).path);\n END IF;\n RETURN FORMAT(\n '%s %s %s',\n COALESCE(leftarg, quote_literal(to_text_array(args->0))),\n CASE op\n WHEN 'a_equals' THEN '='\n WHEN 'a_contains' THEN '@>'\n WHEN 'a_contained_by' THEN '<@'\n WHEN 'a_overlaps' THEN '&&'\n END,\n COALESCE(rightarg, quote_literal(to_text_array(args->1)))\n );\n END IF;\n\n IF op = 'in' THEN\n RAISE NOTICE 'IN : % % %', args, jsonb_build_array(args->0), args->1;\n args := jsonb_build_array(args->0) || (args->1);\n RAISE NOTICE 'IN2 : %', args;\n END IF;\n\n\n\n IF op = 'between' THEN\n args = jsonb_build_array(\n args->0,\n args->1->0,\n args->1->1\n );\n END IF;\n\n -- Make sure that args is an array and run cql2_query on\n -- each element of the array\n RAISE NOTICE 'ARGS PRE: %', args;\n IF j ? 'args' THEN\n IF jsonb_typeof(args) != 'array' THEN\n args := jsonb_build_array(args);\n END IF;\n\n IF jsonb_path_exists(args, '$[*] ? (@.property == \"id\" || @.property == \"datetime\" || @.property == \"end_datetime\" || @.property == \"collection\")') THEN\n wrapper := NULL;\n ELSE\n -- if any of the arguments are a property, try to get the property_wrapper\n FOR arg IN SELECT jsonb_path_query(args, '$[*] ? (@.property != null)') LOOP\n RAISE NOTICE 'Arg: %', arg;\n wrapper := (queryable(arg->>'property')).nulled_wrapper;\n RAISE NOTICE 'Property: %, Wrapper: %', arg, wrapper;\n IF wrapper IS NOT NULL THEN\n EXIT;\n END IF;\n END LOOP;\n\n -- if the property was not in queryables, see if any args were numbers\n IF\n wrapper IS NULL\n AND jsonb_path_exists(args, '$[*] ? (@.type()==\"number\")')\n THEN\n wrapper := 'to_float';\n END IF;\n wrapper := coalesce(wrapper, 'to_text');\n END IF;\n\n SELECT jsonb_agg(cql2_query(a, wrapper))\n INTO args\n FROM jsonb_array_elements(args) a;\n END IF;\n RAISE NOTICE 'ARGS: %', args;\n\n IF op IN ('and', 'or') THEN\n RETURN\n format(\n '(%s)',\n array_to_string(to_text_array(args), format(' %s ', upper(op)))\n );\n END IF;\n\n IF op = 'in' THEN\n RAISE NOTICE 'IN -- % %', args->0, to_text(args->0);\n RETURN format(\n '%s IN (%s)',\n to_text(args->0),\n array_to_string((to_text_array(args))[2:], ',')\n );\n END IF;\n\n -- Look up template from cql2_ops\n IF j ? 'op' THEN\n SELECT * INTO cql2op FROM cql2_ops WHERE cql2_ops.op ilike op;\n IF FOUND THEN\n -- If specific index set in queryables for a property cast other arguments to that type\n\n RETURN format(\n cql2op.template,\n VARIADIC (to_text_array(args))\n );\n ELSE\n RAISE EXCEPTION 'Operator % Not Supported.', op;\n END IF;\n END IF;\n\n\n IF wrapper IS NOT NULL THEN\n RAISE NOTICE 'Wrapping % with %', j, wrapper;\n IF j ? 'property' THEN\n RETURN format('%I(%s)', wrapper, (queryable(j->>'property')).path);\n ELSE\n RETURN format('%I(%L)', wrapper, j);\n END IF;\n ELSIF j ? 'property' THEN\n RETURN quote_ident(j->>'property');\n END IF;\n\n RETURN quote_literal(to_text(j));\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION paging_dtrange(\n j jsonb\n) RETURNS tstzrange AS $$\nDECLARE\n op text;\n filter jsonb := j->'filter';\n dtrange tstzrange := tstzrange('-infinity'::timestamptz,'infinity'::timestamptz);\n sdate timestamptz := '-infinity'::timestamptz;\n edate timestamptz := 'infinity'::timestamptz;\n jpitem jsonb;\nBEGIN\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n END IF;\n IF NOT (filter @? '$.**.op ? (@ == \"or\" || @ == \"not\")') THEN\n FOR jpitem IN SELECT j FROM jsonb_path_query(filter,'strict $.** ? (@.args[*].property == \"datetime\")'::jsonpath) j LOOP\n op := lower(jpitem->>'op');\n dtrange := parse_dtrange(jpitem->'args'->1);\n IF op IN ('<=', 'lt', 'lte', '<', 'le', 't_before') THEN\n sdate := greatest(sdate,'-infinity');\n edate := least(edate, upper(dtrange));\n ELSIF op IN ('>=', '>', 'gt', 'gte', 'ge', 't_after') THEN\n edate := least(edate, 'infinity');\n sdate := greatest(sdate, lower(dtrange));\n ELSIF op IN ('=', 'eq') THEN\n edate := least(edate, upper(dtrange));\n sdate := greatest(sdate, lower(dtrange));\n END IF;\n RAISE NOTICE '2 OP: %, ARGS: %, DTRANGE: %, SDATE: %, EDATE: %', op, jpitem->'args'->1, dtrange, sdate, edate;\n END LOOP;\n END IF;\n IF sdate > edate THEN\n RETURN 'empty'::tstzrange;\n END IF;\n RETURN tstzrange(sdate,edate, '[]');\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT SET TIME ZONE 'UTC';\n\nCREATE OR REPLACE FUNCTION paging_collections(\n IN j jsonb\n) RETURNS text[] AS $$\nDECLARE\n filter jsonb := j->'filter';\n jpitem jsonb;\n op text;\n args jsonb;\n arg jsonb;\n collections text[];\nBEGIN\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n END IF;\n IF NOT (filter @? '$.**.op ? (@ == \"or\" || @ == \"not\")') THEN\n FOR jpitem IN SELECT j FROM jsonb_path_query(filter,'strict $.** ? (@.args[*].property == \"collection\")'::jsonpath) j LOOP\n RAISE NOTICE 'JPITEM: %', jpitem;\n op := jpitem->>'op';\n args := jpitem->'args';\n IF op IN ('=', 'eq', 'in') THEN\n FOR arg IN SELECT a FROM jsonb_array_elements(args) a LOOP\n IF jsonb_typeof(arg) IN ('string', 'array') THEN\n RAISE NOTICE 'arg: %, collections: %', arg, collections;\n IF collections IS NULL OR collections = '{}'::text[] THEN\n collections := to_text_array(arg);\n ELSE\n collections := array_intersection(collections, to_text_array(arg));\n END IF;\n END IF;\n END LOOP;\n END IF;\n END LOOP;\n END IF;\n IF collections = '{}'::text[] THEN\n RETURN NULL;\n END IF;\n RETURN collections;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\nCREATE TABLE items (\n id text NOT NULL,\n geometry geometry NOT NULL,\n collection text NOT NULL,\n datetime timestamptz NOT NULL,\n end_datetime timestamptz NOT NULL,\n content JSONB NOT NULL\n)\nPARTITION BY LIST (collection)\n;\n\nCREATE INDEX \"datetime_idx\" ON items USING BTREE (datetime DESC, end_datetime ASC);\nCREATE INDEX \"geometry_idx\" ON items USING GIST (geometry);\n\nCREATE STATISTICS datetime_stats (dependencies) on datetime, end_datetime from items;\n\n\nALTER TABLE items ADD CONSTRAINT items_collections_fk FOREIGN KEY (collection) REFERENCES collections(id) ON DELETE CASCADE DEFERRABLE;\n\n\nCREATE OR REPLACE FUNCTION content_slim(_item jsonb) RETURNS jsonb AS $$\n SELECT strip_jsonb(_item - '{id,geometry,collection,type}'::text[], collection_base_item(_item->>'collection')) - '{id,geometry,collection,type}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_dehydrate(content jsonb) RETURNS items AS $$\n SELECT\n content->>'id' as id,\n stac_geom(content) as geometry,\n content->>'collection' as collection,\n stac_datetime(content) as datetime,\n stac_end_datetime(content) as end_datetime,\n content_slim(content) as content\n ;\n$$ LANGUAGE SQL STABLE;\n\nCREATE OR REPLACE FUNCTION include_field(f text, fields jsonb DEFAULT '{}'::jsonb) RETURNS boolean AS $$\nDECLARE\n includes jsonb := fields->'include';\n excludes jsonb := fields->'exclude';\nBEGIN\n IF f IS NULL THEN\n RETURN NULL;\n END IF;\n\n\n IF\n jsonb_typeof(excludes) = 'array'\n AND jsonb_array_length(excludes)>0\n AND excludes ? f\n THEN\n RETURN FALSE;\n END IF;\n\n IF\n (\n jsonb_typeof(includes) = 'array'\n AND jsonb_array_length(includes) > 0\n AND includes ? f\n ) OR\n (\n includes IS NULL\n OR jsonb_typeof(includes) = 'null'\n OR jsonb_array_length(includes) = 0\n )\n THEN\n RETURN TRUE;\n END IF;\n\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nDROP FUNCTION IF EXISTS content_hydrate(jsonb, jsonb, jsonb);\nCREATE OR REPLACE FUNCTION content_hydrate(\n _item jsonb,\n _base_item jsonb,\n fields jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\n SELECT merge_jsonb(\n jsonb_fields(_item, fields),\n jsonb_fields(_base_item, fields)\n );\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION content_hydrate(_item items, _collection collections, fields jsonb DEFAULT '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\n content jsonb;\n base_item jsonb := _collection.base_item;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry, 20)::jsonb;\n END IF;\n output := content_hydrate(\n jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'collection', _item.collection,\n 'type', 'Feature'\n ) || _item.content,\n _collection.base_item,\n fields\n );\n\n RETURN output;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_nonhydrated(\n _item items,\n fields jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry, 20)::jsonb;\n END IF;\n output := jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'collection', _item.collection,\n 'type', 'Feature'\n ) || _item.content;\n RETURN output;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_hydrate(_item items, fields jsonb DEFAULT '{}'::jsonb) RETURNS jsonb AS $$\n SELECT content_hydrate(\n _item,\n (SELECT c FROM collections c WHERE id=_item.collection LIMIT 1),\n fields\n );\n$$ LANGUAGE SQL STABLE;\n\n\nCREATE UNLOGGED TABLE items_staging (\n content JSONB NOT NULL\n);\nCREATE UNLOGGED TABLE items_staging_ignore (\n content JSONB NOT NULL\n);\nCREATE UNLOGGED TABLE items_staging_upsert (\n content JSONB NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION items_staging_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p record;\n _partitions text[];\n ts timestamptz := clock_timestamp();\nBEGIN\n RAISE NOTICE 'Creating Partitions. %', clock_timestamp() - ts;\n WITH ranges AS (\n SELECT\n n.content->>'collection' as collection,\n stac_daterange(n.content->'properties') as dtr\n FROM newdata n\n ), p AS (\n SELECT\n collection,\n lower(dtr) as datetime,\n upper(dtr) as end_datetime,\n (partition_name(\n collection,\n lower(dtr)\n )).partition_name as name\n FROM ranges\n )\n INSERT INTO partitions (collection, datetime_range, end_datetime_range)\n SELECT\n collection,\n tstzrange(min(datetime), max(datetime), '[]') as datetime_range,\n tstzrange(min(end_datetime), max(end_datetime), '[]') as end_datetime_range\n FROM p\n GROUP BY collection, name\n ON CONFLICT (name) DO UPDATE SET\n datetime_range = EXCLUDED.datetime_range,\n end_datetime_range = EXCLUDED.end_datetime_range\n ;\n\n RAISE NOTICE 'Doing the insert. %', clock_timestamp() - ts;\n IF TG_TABLE_NAME = 'items_staging' THEN\n INSERT INTO items\n SELECT\n (content_dehydrate(content)).*\n FROM newdata;\n DELETE FROM items_staging;\n ELSIF TG_TABLE_NAME = 'items_staging_ignore' THEN\n INSERT INTO items\n SELECT\n (content_dehydrate(content)).*\n FROM newdata\n ON CONFLICT DO NOTHING;\n DELETE FROM items_staging_ignore;\n ELSIF TG_TABLE_NAME = 'items_staging_upsert' THEN\n WITH staging_formatted AS (\n SELECT (content_dehydrate(content)).* FROM newdata\n ), deletes AS (\n DELETE FROM items i USING staging_formatted s\n WHERE\n i.id = s.id\n AND i.collection = s.collection\n AND i IS DISTINCT FROM s\n RETURNING i.id, i.collection\n )\n INSERT INTO items\n SELECT s.* FROM\n staging_formatted s\n ON CONFLICT DO NOTHING;\n DELETE FROM items_staging_upsert;\n END IF;\n RAISE NOTICE 'Done. %', clock_timestamp() - ts;\n\n RETURN NULL;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER items_staging_insert_trigger AFTER INSERT ON items_staging REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\nCREATE TRIGGER items_staging_insert_ignore_trigger AFTER INSERT ON items_staging_ignore REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\nCREATE TRIGGER items_staging_insert_upsert_trigger AFTER INSERT ON items_staging_upsert REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\n\n\n\nCREATE OR REPLACE FUNCTION item_by_id(_id text, _collection text DEFAULT NULL) RETURNS items AS\n$$\nDECLARE\n i items%ROWTYPE;\nBEGIN\n SELECT * INTO i FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection) LIMIT 1;\n RETURN i;\nEND;\n$$ LANGUAGE PLPGSQL STABLE SECURITY DEFINER SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION get_item(_id text, _collection text DEFAULT NULL) RETURNS jsonb AS $$\n SELECT content_hydrate(items) FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection);\n$$ LANGUAGE SQL STABLE SECURITY DEFINER SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION delete_item(_id text, _collection text DEFAULT NULL) RETURNS VOID AS $$\nDECLARE\nout items%ROWTYPE;\nBEGIN\n DELETE FROM items WHERE id = _id AND (_collection IS NULL OR collection=_collection) RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n--/*\nCREATE OR REPLACE FUNCTION create_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION update_item(content jsonb) RETURNS VOID AS $$\nDECLARE\n old items %ROWTYPE;\n out items%ROWTYPE;\nBEGIN\n PERFORM delete_item(content->>'id', content->>'collection');\n PERFORM create_item(content);\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION create_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION collection_bbox(id text) RETURNS jsonb AS $$\n SELECT (replace(replace(replace(st_extent(geometry)::text,'BOX(','[['),')',']]'),' ',','))::jsonb\n FROM items WHERE collection=$1;\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION collection_temporal_extent(id text) RETURNS jsonb AS $$\n SELECT to_jsonb(array[array[min(datetime)::text, max(datetime)::text]])\n FROM items WHERE collection=$1;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection_extents() RETURNS VOID AS $$\nUPDATE collections SET\n content = content ||\n jsonb_build_object(\n 'extent', jsonb_build_object(\n 'spatial', jsonb_build_object(\n 'bbox', collection_bbox(collections.id)\n ),\n 'temporal', jsonb_build_object(\n 'interval', collection_temporal_extent(collections.id)\n )\n )\n )\n;\n$$ LANGUAGE SQL;\n\n\nCREATE OR REPLACE PROCEDURE analyze_items() AS $$\nDECLARE\nq text;\nBEGIN\nFOR q IN\n SELECT format('ANALYZE (VERBOSE, SKIP_LOCKED) %I;', relname)\n FROM pg_stat_user_tables\n WHERE relname like '_item%' AND (n_mod_since_analyze>0 OR last_analyze IS NULL)\nLOOP\n RAISE NOTICE '%', q;\n EXECUTE q;\n COMMIT;\nEND LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE PROCEDURE validate_constraints() AS $$\nDECLARE\n q text;\nBEGIN\n FOR q IN\n SELECT\n FORMAT(\n 'ALTER TABLE %I.%I VALIDATE CONSTRAINT %I;',\n nsp.nspname,\n cls.relname,\n con.conname\n )\n\n FROM pg_constraint AS con\n JOIN pg_class AS cls\n ON con.conrelid = cls.oid\n JOIN pg_namespace AS nsp\n ON cls.relnamespace = nsp.oid\n WHERE convalidated = FALSE AND contype in ('c','f')\n AND nsp.nspname = 'pgstac'\n LOOP\n RAISE NOTICE '%', q;\n EXECUTE q;\n COMMIT;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\nCREATE VIEW partition_steps AS\nSELECT\n name,\n date_trunc('month',lower(datetime_range)) as sdate,\n date_trunc('month', upper(datetime_range)) + '1 month'::interval as edate\n FROM partitions WHERE datetime_range IS NOT NULL AND datetime_range != 'empty'::tstzrange\n ORDER BY datetime_range ASC\n;\n\nCREATE OR REPLACE FUNCTION chunker(\n IN _where text,\n OUT s timestamptz,\n OUT e timestamptz\n) RETURNS SETOF RECORD AS $$\nDECLARE\n explain jsonb;\nBEGIN\n IF _where IS NULL THEN\n _where := ' TRUE ';\n END IF;\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s;', _where)\n INTO explain;\n\n RETURN QUERY\n WITH t AS (\n SELECT j->>0 as p FROM\n jsonb_path_query(\n explain,\n 'strict $.**.\"Relation Name\" ? (@ != null)'\n ) j\n ),\n parts AS (\n SELECT sdate, edate FROM t JOIN partition_steps ON (t.p = name)\n ),\n times AS (\n SELECT sdate FROM parts\n UNION\n SELECT edate FROM parts\n ),\n uniq AS (\n SELECT DISTINCT sdate FROM times ORDER BY sdate\n ),\n last AS (\n SELECT sdate, lead(sdate, 1) over () as edate FROM uniq\n )\n SELECT sdate, edate FROM last WHERE edate IS NOT NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION partition_queries(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN partitions text[] DEFAULT NULL\n) RETURNS SETOF text AS $$\nDECLARE\n query text;\n sdate timestamptz;\n edate timestamptz;\nBEGIN\nIF _where IS NULL OR trim(_where) = '' THEN\n _where = ' TRUE ';\nEND IF;\nRAISE NOTICE 'Getting chunks for % %', _where, _orderby;\nIF _orderby ILIKE 'datetime d%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 DESC LOOP\n RETURN NEXT format($q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n $q$,\n sdate,\n edate,\n _where,\n _orderby\n );\n END LOOP;\nELSIF _orderby ILIKE 'datetime a%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 ASC LOOP\n RETURN NEXT format($q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n $q$,\n sdate,\n edate,\n _where,\n _orderby\n );\n END LOOP;\nELSE\n query := format($q$\n SELECT * FROM items\n WHERE %s\n ORDER BY %s\n $q$, _where, _orderby\n );\n\n RETURN NEXT query;\n RETURN;\nEND IF;\n\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION partition_query_view(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN _limit int DEFAULT 10\n) RETURNS text AS $$\n WITH p AS (\n SELECT * FROM partition_queries(_where, _orderby) p\n )\n SELECT\n CASE WHEN EXISTS (SELECT 1 FROM p) THEN\n (SELECT format($q$\n SELECT * FROM (\n %s\n ) total LIMIT %s\n $q$,\n string_agg(\n format($q$ SELECT * FROM ( %s ) AS sub $q$, p),\n '\n UNION ALL\n '\n ),\n _limit\n ))\n ELSE NULL\n END FROM p;\n$$ LANGUAGE SQL IMMUTABLE;\n\n\n\n\nCREATE OR REPLACE FUNCTION stac_search_to_where(j jsonb) RETURNS text AS $$\nDECLARE\n where_segments text[];\n _where text;\n dtrange tstzrange;\n collections text[];\n geom geometry;\n sdate timestamptz;\n edate timestamptz;\n filterlang text;\n filter jsonb := j->'filter';\nBEGIN\n IF j ? 'ids' THEN\n where_segments := where_segments || format('id = ANY (%L) ', to_text_array(j->'ids'));\n END IF;\n\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n where_segments := where_segments || format('collection = ANY (%L) ', collections);\n END IF;\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n\n where_segments := where_segments || format(' datetime <= %L::timestamptz AND end_datetime >= %L::timestamptz ',\n edate,\n sdate\n );\n END IF;\n\n geom := stac_geom(j);\n IF geom IS NOT NULL THEN\n where_segments := where_segments || format('st_intersects(geometry, %L)',geom);\n END IF;\n\n filterlang := COALESCE(\n j->>'filter-lang',\n get_setting('default_filter_lang', j->'conf')\n );\n IF NOT filter @? '$.**.op' THEN\n filterlang := 'cql-json';\n END IF;\n\n IF filterlang NOT IN ('cql-json','cql2-json') AND j ? 'filter' THEN\n RAISE EXCEPTION '% is not a supported filter-lang. Please use cql-json or cql2-json.', filterlang;\n END IF;\n\n IF j ? 'query' AND j ? 'filter' THEN\n RAISE EXCEPTION 'Can only use either query or filter at one time.';\n END IF;\n\n IF j ? 'query' THEN\n filter := query_to_cql2(j->'query');\n ELSIF filterlang = 'cql-json' THEN\n filter := cql1_to_cql2(filter);\n END IF;\n RAISE NOTICE 'FILTER: %', filter;\n where_segments := where_segments || cql2_query(filter);\n IF cardinality(where_segments) < 1 THEN\n RETURN ' TRUE ';\n END IF;\n\n _where := array_to_string(array_remove(where_segments, NULL), ' AND ');\n\n IF _where IS NULL OR BTRIM(_where) = '' THEN\n RETURN ' TRUE ';\n END IF;\n RETURN _where;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION parse_sort_dir(_dir text, reverse boolean default false) RETURNS text AS $$\n WITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n ) SELECT\n CASE\n WHEN NOT reverse THEN d\n WHEN d = 'ASC' THEN 'DESC'\n WHEN d = 'DESC' THEN 'ASC'\n END\n FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION sort_dir_to_op(_dir text, prev boolean default false) RETURNS text AS $$\n WITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n ) SELECT\n CASE\n WHEN d = 'ASC' AND prev THEN '<='\n WHEN d = 'DESC' AND prev THEN '>='\n WHEN d = 'ASC' THEN '>='\n WHEN d = 'DESC' THEN '<='\n END\n FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION sort_sqlorderby(\n _search jsonb DEFAULT NULL,\n reverse boolean DEFAULT FALSE\n) RETURNS text AS $$\n WITH sortby AS (\n SELECT coalesce(_search->'sortby','[{\"field\":\"datetime\", \"direction\":\"desc\"}]') as sort\n ), withid AS (\n SELECT CASE\n WHEN sort @? '$[*] ? (@.field == \"id\")' THEN sort\n ELSE sort || '[{\"field\":\"id\", \"direction\":\"desc\"}]'::jsonb\n END as sort\n FROM sortby\n ), withid_rows AS (\n SELECT jsonb_array_elements(sort) as value FROM withid\n ),sorts AS (\n SELECT\n coalesce(\n -- field_orderby((items_path(value->>'field')).path_txt),\n (queryable(value->>'field')).expression\n ) as key,\n parse_sort_dir(value->>'direction', reverse) as dir\n FROM withid_rows\n )\n SELECT array_to_string(\n array_agg(concat(key, ' ', dir)),\n ', '\n ) FROM sorts;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_sort_dir(sort_item jsonb) RETURNS text AS $$\n SELECT CASE WHEN sort_item->>'direction' ILIKE 'desc%' THEN 'DESC' ELSE 'ASC' END;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION get_token_val_str(\n _field text,\n _item items\n) RETURNS text AS $$\nDECLARE\nliteral text;\nBEGIN\nRAISE NOTICE '% %', _field, _item;\nCREATE TEMP TABLE _token_item ON COMMIT DROP AS SELECT (_item).*;\nEXECUTE format($q$ SELECT quote_literal(%s) FROM _token_item $q$, _field) INTO literal;\nDROP TABLE IF EXISTS _token_item;\nRETURN literal;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION get_token_filter(_search jsonb = '{}'::jsonb, token_rec jsonb DEFAULT NULL) RETURNS text AS $$\nDECLARE\n token_id text;\n filters text[] := '{}'::text[];\n prev boolean := TRUE;\n field text;\n dir text;\n sort record;\n orfilters text[] := '{}'::text[];\n andfilters text[] := '{}'::text[];\n output text;\n token_where text;\n token_item items%ROWTYPE;\nBEGIN\n RAISE NOTICE 'Getting Token Filter. % %', _search, token_rec;\n -- If no token provided return NULL\n IF token_rec IS NULL THEN\n IF NOT (_search ? 'token' AND\n (\n (_search->>'token' ILIKE 'prev:%')\n OR\n (_search->>'token' ILIKE 'next:%')\n )\n ) THEN\n RETURN NULL;\n END IF;\n prev := (_search->>'token' ILIKE 'prev:%');\n token_id := substr(_search->>'token', 6);\n SELECT to_jsonb(items) INTO token_rec\n FROM items WHERE id=token_id;\n END IF;\n RAISE NOTICE 'TOKEN ID: % %', token_rec, token_rec->'id';\n\n\n RAISE NOTICE 'TOKEN ID: % %', token_rec, token_rec->'id';\n token_item := jsonb_populate_record(null::items, token_rec);\n RAISE NOTICE 'TOKEN ITEM ----- %', token_item;\n\n\n CREATE TEMP TABLE sorts (\n _row int GENERATED ALWAYS AS IDENTITY NOT NULL,\n _field text PRIMARY KEY,\n _dir text NOT NULL,\n _val text\n ) ON COMMIT DROP;\n\n -- Make sure we only have distinct columns to sort with taking the first one we get\n INSERT INTO sorts (_field, _dir)\n SELECT\n (queryable(value->>'field')).expression,\n get_sort_dir(value)\n FROM\n jsonb_array_elements(coalesce(_search->'sortby','[{\"field\":\"datetime\",\"direction\":\"desc\"}]'))\n ON CONFLICT DO NOTHING\n ;\n RAISE NOTICE 'sorts 1: %', (SELECT jsonb_agg(to_json(sorts)) FROM sorts);\n -- Get the first sort direction provided. As the id is a primary key, if there are any\n -- sorts after id they won't do anything, so make sure that id is the last sort item.\n SELECT _dir INTO dir FROM sorts ORDER BY _row ASC LIMIT 1;\n IF EXISTS (SELECT 1 FROM sorts WHERE _field = 'id') THEN\n DELETE FROM sorts WHERE _row > (SELECT _row FROM sorts WHERE _field = 'id' ORDER BY _row ASC);\n ELSE\n INSERT INTO sorts (_field, _dir) VALUES ('id', dir);\n END IF;\n\n -- Add value from looked up item to the sorts table\n UPDATE sorts SET _val=get_token_val_str(_field, token_item);\n\n -- Check if all sorts are the same direction and use row comparison\n -- to filter\n RAISE NOTICE 'sorts 2: %', (SELECT jsonb_agg(to_json(sorts)) FROM sorts);\n\n FOR sort IN SELECT * FROM sorts ORDER BY _row asc LOOP\n RAISE NOTICE 'SORT: %', sort;\n IF sort._row = 1 THEN\n IF sort._val IS NULL THEN\n orfilters := orfilters || format('(%s IS NOT NULL)', sort._field);\n ELSE\n orfilters := orfilters || format('(%s %s %s)',\n sort._field,\n CASE WHEN (prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC') THEN '<' ELSE '>' END,\n sort._val\n );\n END IF;\n ELSE\n IF sort._val IS NULL THEN\n orfilters := orfilters || format('(%s AND %s IS NOT NULL)',\n array_to_string(andfilters, ' AND '), sort._field);\n ELSE\n orfilters := orfilters || format('(%s AND %s %s %s)',\n array_to_string(andfilters, ' AND '),\n sort._field,\n CASE WHEN (prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC') THEN '<' ELSE '>' END,\n sort._val\n );\n END IF;\n END IF;\n IF sort._val IS NULL THEN\n andfilters := andfilters || format('%s IS NULL',\n sort._field\n );\n ELSE\n andfilters := andfilters || format('%s = %s',\n sort._field,\n sort._val\n );\n END IF;\n END LOOP;\n output := array_to_string(orfilters, ' OR ');\n\n DROP TABLE IF EXISTS sorts;\n token_where := concat('(',coalesce(output,'true'),')');\n IF trim(token_where) = '' THEN\n token_where := NULL;\n END IF;\n RAISE NOTICE 'TOKEN_WHERE: |%|',token_where;\n RETURN token_where;\n END;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION search_tohash(jsonb) RETURNS jsonb AS $$\n SELECT $1 - '{token,limit,context,includes,excludes}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION search_hash(jsonb, jsonb) RETURNS text AS $$\n SELECT md5(concat(search_tohash($1)::text,$2::text));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TABLE IF NOT EXISTS searches(\n hash text GENERATED ALWAYS AS (search_hash(search, metadata)) STORED PRIMARY KEY,\n search jsonb NOT NULL,\n _where text,\n orderby text,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n metadata jsonb DEFAULT '{}'::jsonb NOT NULL\n);\nCREATE TABLE IF NOT EXISTS search_wheres(\n id bigint generated always as identity primary key,\n _where text NOT NULL,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n statslastupdated timestamptz,\n estimated_count bigint,\n estimated_cost float,\n time_to_estimate float,\n total_count bigint,\n time_to_count float,\n partitions text[]\n);\n\nCREATE INDEX IF NOT EXISTS search_wheres_partitions ON search_wheres USING GIN (partitions);\nCREATE UNIQUE INDEX IF NOT EXISTS search_wheres_where ON search_wheres ((md5(_where)));\n\nCREATE OR REPLACE FUNCTION where_stats(inwhere text, updatestats boolean default false, conf jsonb default null) RETURNS search_wheres AS $$\nDECLARE\n t timestamptz;\n i interval;\n explain_json jsonb;\n partitions text[];\n sw search_wheres%ROWTYPE;\n inwhere_hash text := md5(inwhere);\n _context text := lower(context(conf));\n _stats_ttl interval := context_stats_ttl(conf);\n _estimated_cost float := context_estimated_cost(conf);\n _estimated_count int := context_estimated_count(conf);\nBEGIN\n IF _context = 'off' THEN\n sw._where := inwhere;\n return sw;\n END IF;\n\n SELECT * INTO sw FROM search_wheres WHERE md5(_where)=inwhere_hash FOR UPDATE;\n\n -- Update statistics if explicitly set, if statistics do not exist, or statistics ttl has expired\n IF NOT updatestats THEN\n RAISE NOTICE 'Checking if update is needed for: % .', inwhere;\n RAISE NOTICE 'Stats Last Updated: %', sw.statslastupdated;\n RAISE NOTICE 'TTL: %, Age: %', _stats_ttl, now() - sw.statslastupdated;\n RAISE NOTICE 'Context: %, Existing Total: %', _context, sw.total_count;\n IF\n sw.statslastupdated IS NULL\n OR (now() - sw.statslastupdated) > _stats_ttl\n OR (context(conf) != 'off' AND sw.total_count IS NULL)\n THEN\n updatestats := TRUE;\n END IF;\n END IF;\n\n sw._where := inwhere;\n sw.lastused := now();\n sw.usecount := coalesce(sw.usecount,0) + 1;\n\n IF NOT updatestats THEN\n UPDATE search_wheres SET\n lastused = sw.lastused,\n usecount = sw.usecount\n WHERE md5(_where) = inwhere_hash\n RETURNING * INTO sw\n ;\n RETURN sw;\n END IF;\n\n -- Use explain to get estimated count/cost and a list of the partitions that would be hit by the query\n t := clock_timestamp();\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s', inwhere)\n INTO explain_json;\n RAISE NOTICE 'Time for just the explain: %', clock_timestamp() - t;\n i := clock_timestamp() - t;\n\n sw.statslastupdated := now();\n sw.estimated_count := explain_json->0->'Plan'->'Plan Rows';\n sw.estimated_cost := explain_json->0->'Plan'->'Total Cost';\n sw.time_to_estimate := extract(epoch from i);\n\n RAISE NOTICE 'ESTIMATED_COUNT: % < %', sw.estimated_count, _estimated_count;\n RAISE NOTICE 'ESTIMATED_COST: % < %', sw.estimated_cost, _estimated_cost;\n\n -- Do a full count of rows if context is set to on or if auto is set and estimates are low enough\n IF\n _context = 'on'\n OR\n ( _context = 'auto' AND\n (\n sw.estimated_count < _estimated_count\n AND\n sw.estimated_cost < _estimated_cost\n )\n )\n THEN\n t := clock_timestamp();\n RAISE NOTICE 'Calculating actual count...';\n EXECUTE format(\n 'SELECT count(*) FROM items WHERE %s',\n inwhere\n ) INTO sw.total_count;\n i := clock_timestamp() - t;\n RAISE NOTICE 'Actual Count: % -- %', sw.total_count, i;\n sw.time_to_count := extract(epoch FROM i);\n ELSE\n sw.total_count := NULL;\n sw.time_to_count := NULL;\n END IF;\n\n\n INSERT INTO search_wheres\n (_where, lastused, usecount, statslastupdated, estimated_count, estimated_cost, time_to_estimate, partitions, total_count, time_to_count)\n SELECT sw._where, sw.lastused, sw.usecount, sw.statslastupdated, sw.estimated_count, sw.estimated_cost, sw.time_to_estimate, sw.partitions, sw.total_count, sw.time_to_count\n ON CONFLICT ((md5(_where)))\n DO UPDATE\n SET\n lastused = sw.lastused,\n usecount = sw.usecount,\n statslastupdated = sw.statslastupdated,\n estimated_count = sw.estimated_count,\n estimated_cost = sw.estimated_cost,\n time_to_estimate = sw.time_to_estimate,\n total_count = sw.total_count,\n time_to_count = sw.time_to_count\n ;\n RETURN sw;\nEND;\n$$ LANGUAGE PLPGSQL ;\n\n\n\nDROP FUNCTION IF EXISTS search_query;\nCREATE OR REPLACE FUNCTION search_query(\n _search jsonb = '{}'::jsonb,\n updatestats boolean = false,\n _metadata jsonb = '{}'::jsonb\n) RETURNS searches AS $$\nDECLARE\n search searches%ROWTYPE;\n pexplain jsonb;\n t timestamptz;\n i interval;\nBEGIN\n SELECT * INTO search FROM searches\n WHERE hash=search_hash(_search, _metadata) FOR UPDATE;\n\n -- Calculate the where clause if not already calculated\n IF search._where IS NULL THEN\n search._where := stac_search_to_where(_search);\n END IF;\n\n -- Calculate the order by clause if not already calculated\n IF search.orderby IS NULL THEN\n search.orderby := sort_sqlorderby(_search);\n END IF;\n\n PERFORM where_stats(search._where, updatestats, _search->'conf');\n\n search.lastused := now();\n search.usecount := coalesce(search.usecount, 0) + 1;\n INSERT INTO searches (search, _where, orderby, lastused, usecount, metadata)\n VALUES (_search, search._where, search.orderby, search.lastused, search.usecount, _metadata)\n ON CONFLICT (hash) DO\n UPDATE SET\n _where = EXCLUDED._where,\n orderby = EXCLUDED.orderby,\n lastused = EXCLUDED.lastused,\n usecount = EXCLUDED.usecount,\n metadata = EXCLUDED.metadata\n RETURNING * INTO search\n ;\n RETURN search;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION search(_search jsonb = '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n searches searches%ROWTYPE;\n _where text;\n token_where text;\n full_where text;\n orderby text;\n query text;\n token_type text := substr(_search->>'token',1,4);\n _limit int := coalesce((_search->>'limit')::int, 10);\n curs refcursor;\n cntr int := 0;\n iter_record items%ROWTYPE;\n first_record jsonb;\n first_item items%ROWTYPE;\n last_item items%ROWTYPE;\n last_record jsonb;\n out_records jsonb := '[]'::jsonb;\n prev_query text;\n next text;\n prev_id text;\n has_next boolean := false;\n has_prev boolean := false;\n prev text;\n total_count bigint;\n context jsonb;\n collection jsonb;\n includes text[];\n excludes text[];\n exit_flag boolean := FALSE;\n batches int := 0;\n timer timestamptz := clock_timestamp();\n pstart timestamptz;\n pend timestamptz;\n pcurs refcursor;\n search_where search_wheres%ROWTYPE;\n id text;\nBEGIN\nCREATE TEMP TABLE results (i int GENERATED ALWAYS AS IDENTITY, content jsonb) ON COMMIT DROP;\n-- if ids is set, short circuit and just use direct ids query for each id\n-- skip any paging or caching\n-- hard codes ordering in the same order as the array of ids\nIF _search ? 'ids' THEN\n INSERT INTO results (content)\n SELECT\n CASE WHEN _search->'conf'->>'nohydrate' IS NOT NULL AND (_search->'conf'->>'nohydrate')::boolean = true THEN\n content_nonhydrated(items, _search->'fields')\n ELSE\n content_hydrate(items, _search->'fields')\n END\n FROM items WHERE\n items.id = ANY(to_text_array(_search->'ids'))\n AND\n CASE WHEN _search ? 'collections' THEN\n items.collection = ANY(to_text_array(_search->'collections'))\n ELSE TRUE\n END\n ORDER BY items.datetime desc, items.id desc\n ;\n SELECT INTO total_count count(*) FROM results;\nELSE\n searches := search_query(_search);\n _where := searches._where;\n orderby := searches.orderby;\n search_where := where_stats(_where);\n total_count := coalesce(search_where.total_count, search_where.estimated_count);\n\n IF token_type='prev' THEN\n token_where := get_token_filter(_search, null::jsonb);\n orderby := sort_sqlorderby(_search, TRUE);\n END IF;\n IF token_type='next' THEN\n token_where := get_token_filter(_search, null::jsonb);\n END IF;\n\n full_where := concat_ws(' AND ', _where, token_where);\n RAISE NOTICE 'FULL QUERY % %', full_where, clock_timestamp()-timer;\n timer := clock_timestamp();\n\n FOR query IN SELECT partition_queries(full_where, orderby, search_where.partitions) LOOP\n timer := clock_timestamp();\n query := format('%s LIMIT %s', query, _limit + 1);\n RAISE NOTICE 'Partition Query: %', query;\n batches := batches + 1;\n -- curs = create_cursor(query);\n RAISE NOTICE 'cursor_tuple_fraction: %', current_setting('cursor_tuple_fraction');\n OPEN curs FOR EXECUTE query;\n LOOP\n FETCH curs into iter_record;\n EXIT WHEN NOT FOUND;\n cntr := cntr + 1;\n\n IF _search->'conf'->>'nohydrate' IS NOT NULL AND (_search->'conf'->>'nohydrate')::boolean = true THEN\n last_record := content_nonhydrated(iter_record, _search->'fields');\n ELSE\n last_record := content_hydrate(iter_record, _search->'fields');\n END IF;\n last_item := iter_record;\n IF cntr = 1 THEN\n first_item := last_item;\n first_record := last_record;\n END IF;\n IF cntr <= _limit THEN\n INSERT INTO results (content) VALUES (last_record);\n ELSIF cntr > _limit THEN\n has_next := true;\n exit_flag := true;\n EXIT;\n END IF;\n END LOOP;\n CLOSE curs;\n RAISE NOTICE 'Query took %.', clock_timestamp()-timer;\n timer := clock_timestamp();\n EXIT WHEN exit_flag;\n END LOOP;\n RAISE NOTICE 'Scanned through % partitions.', batches;\nEND IF;\n\nWITH ordered AS (SELECT * FROM results WHERE content IS NOT NULL ORDER BY i)\nSELECT jsonb_agg(content) INTO out_records FROM ordered;\n\nDROP TABLE results;\n\n\n-- Flip things around if this was the result of a prev token query\nIF token_type='prev' THEN\n out_records := flip_jsonb_array(out_records);\n first_item := last_item;\n first_record := last_record;\nEND IF;\n\n-- If this query has a token, see if there is data before the first record\nIF _search ? 'token' THEN\n prev_query := format(\n 'SELECT 1 FROM items WHERE %s LIMIT 1',\n concat_ws(\n ' AND ',\n _where,\n trim(get_token_filter(_search, to_jsonb(first_item)))\n )\n );\n RAISE NOTICE 'Query to get previous record: % --- %', prev_query, first_record;\n EXECUTE prev_query INTO has_prev;\n IF FOUND and has_prev IS NOT NULL THEN\n RAISE NOTICE 'Query results from prev query: %', has_prev;\n has_prev := TRUE;\n END IF;\nEND IF;\nhas_prev := COALESCE(has_prev, FALSE);\n\nIF has_prev THEN\n prev := out_records->0->>'id';\nEND IF;\nIF has_next OR token_type='prev' THEN\n next := out_records->-1->>'id';\nEND IF;\n\nIF context(_search->'conf') != 'off' THEN\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'matched', total_count,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\nELSE\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\nEND IF;\n\ncollection := jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb),\n 'next', next,\n 'prev', prev,\n 'context', context\n);\n\nRETURN collection;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER SET SEARCH_PATH TO pgstac, public SET cursor_tuple_fraction TO 1;\n\n\nCREATE OR REPLACE FUNCTION search_cursor(_search jsonb = '{}'::jsonb) RETURNS refcursor AS $$\nDECLARE\n curs refcursor;\n searches searches%ROWTYPE;\n _where text;\n _orderby text;\n q text;\n\nBEGIN\n searches := search_query(_search);\n _where := searches._where;\n _orderby := searches.orderby;\n\n OPEN curs FOR\n WITH p AS (\n SELECT * FROM partition_queries(_where, _orderby) p\n )\n SELECT\n CASE WHEN EXISTS (SELECT 1 FROM p) THEN\n (SELECT format($q$\n SELECT * FROM (\n %s\n ) total\n $q$,\n string_agg(\n format($q$ SELECT * FROM ( %s ) AS sub $q$, p),\n '\n UNION ALL\n '\n )\n ))\n ELSE NULL\n END FROM p;\n RETURN curs;\nEND;\n$$ LANGUAGE PLPGSQL;\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION tileenvelope(zoom int, x int, y int) RETURNS geometry AS $$\nWITH t AS (\n SELECT\n 20037508.3427892 as merc_max,\n -20037508.3427892 as merc_min,\n (2 * 20037508.3427892) / (2 ^ zoom) as tile_size\n)\nSELECT st_makeenvelope(\n merc_min + (tile_size * x),\n merc_max - (tile_size * (y + 1)),\n merc_min + (tile_size * (x + 1)),\n merc_max - (tile_size * y),\n 3857\n) FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;DROP FUNCTION IF EXISTS mercgrid;\n\n\nCREATE OR REPLACE FUNCTION ftime() RETURNS interval as $$\nSELECT age(clock_timestamp(), transaction_timestamp());\n$$ LANGUAGE SQL;\nSET SEARCH_PATH to pgstac, public;\n\nDROP FUNCTION IF EXISTS geometrysearch;\nCREATE OR REPLACE FUNCTION geometrysearch(\n IN geom geometry,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE, -- Return as soon as the passed in geometry is full covered\n IN skipcovered boolean DEFAULT TRUE -- Skip any items that would show up completely under the previous items\n) RETURNS jsonb AS $$\nDECLARE\n search searches%ROWTYPE;\n curs refcursor;\n _where text;\n query text;\n iter_record items%ROWTYPE;\n out_records jsonb := '{}'::jsonb[];\n exit_flag boolean := FALSE;\n counter int := 1;\n scancounter int := 1;\n remaining_limit int := _scanlimit;\n tilearea float;\n unionedgeom geometry;\n clippedgeom geometry;\n unionedgeom_area float := 0;\n prev_area float := 0;\n excludes text[];\n includes text[];\n\nBEGIN\n DROP TABLE IF EXISTS pgstac_results;\n CREATE TEMP TABLE pgstac_results (content jsonb) ON COMMIT DROP;\n\n -- If skipcovered is true then you will always want to exit when the passed in geometry is full\n IF skipcovered THEN\n exitwhenfull := TRUE;\n END IF;\n\n SELECT * INTO search FROM searches WHERE hash=queryhash;\n\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Search with Query Hash % Not Found', queryhash;\n END IF;\n\n tilearea := st_area(geom);\n _where := format('%s AND st_intersects(geometry, %L::geometry)', search._where, geom);\n\n\n FOR query IN SELECT * FROM partition_queries(_where, search.orderby) LOOP\n query := format('%s LIMIT %L', query, remaining_limit);\n RAISE NOTICE '%', query;\n OPEN curs FOR EXECUTE query;\n LOOP\n FETCH curs INTO iter_record;\n EXIT WHEN NOT FOUND;\n IF exitwhenfull OR skipcovered THEN -- If we are not using exitwhenfull or skipcovered, we do not need to do expensive geometry operations\n clippedgeom := st_intersection(geom, iter_record.geometry);\n\n IF unionedgeom IS NULL THEN\n unionedgeom := clippedgeom;\n ELSE\n unionedgeom := st_union(unionedgeom, clippedgeom);\n END IF;\n\n unionedgeom_area := st_area(unionedgeom);\n\n IF skipcovered AND prev_area = unionedgeom_area THEN\n scancounter := scancounter + 1;\n CONTINUE;\n END IF;\n\n prev_area := unionedgeom_area;\n\n RAISE NOTICE '% % % %', unionedgeom_area/tilearea, counter, scancounter, ftime();\n END IF;\n RAISE NOTICE '% %', iter_record, content_hydrate(iter_record, fields);\n INSERT INTO pgstac_results (content) VALUES (content_hydrate(iter_record, fields));\n\n IF counter >= _limit\n OR scancounter > _scanlimit\n OR ftime() > _timelimit\n OR (exitwhenfull AND unionedgeom_area >= tilearea)\n THEN\n exit_flag := TRUE;\n EXIT;\n END IF;\n counter := counter + 1;\n scancounter := scancounter + 1;\n\n END LOOP;\n CLOSE curs;\n EXIT WHEN exit_flag;\n remaining_limit := _scanlimit - scancounter;\n END LOOP;\n\n SELECT jsonb_agg(content) INTO out_records FROM pgstac_results WHERE content IS NOT NULL;\n\n RETURN jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb)\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDROP FUNCTION IF EXISTS geojsonsearch;\nCREATE OR REPLACE FUNCTION geojsonsearch(\n IN geojson jsonb,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_geomfromgeojson(geojson),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS xyzsearch;\nCREATE OR REPLACE FUNCTION xyzsearch(\n IN _x int,\n IN _y int,\n IN _z int,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_transform(tileenvelope(_z, _x, _y), 4326),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nGRANT ALL ON SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac to pgstac_admin;\n\n-- pgstac_read role limited to using function apis\nGRANT EXECUTE ON FUNCTION search TO pgstac_read;\nGRANT EXECUTE ON FUNCTION search_query TO pgstac_read;\nGRANT EXECUTE ON FUNCTION item_by_id TO pgstac_read;\nGRANT EXECUTE ON FUNCTION get_item TO pgstac_read;\n\nGRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_ingest;\nGRANT USAGE ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_ingest;\nSELECT set_version('0.6.13');\n" }, { "path": "src/pgstac/migrations/pgstac.0.6.2-0.6.3.sql", "content": "SET SEARCH_PATH to pgstac, public;\ndrop function if exists \"pgstac\".\"content_hydrate\"(_base_item jsonb, _item jsonb, fields jsonb);\n\nset check_function_bodies = off;\n\nCREATE OR REPLACE FUNCTION pgstac.content_hydrate(_item jsonb, _base_item jsonb, fields jsonb DEFAULT '{}'::jsonb)\n RETURNS jsonb\n LANGUAGE sql\n IMMUTABLE PARALLEL SAFE\nAS $function$\n SELECT merge_jsonb(\n jsonb_fields(_item, fields),\n jsonb_fields(_base_item, fields)\n );\n$function$\n;\n\n\n\nSELECT set_version('0.6.3');\n" }, { "path": "src/pgstac/migrations/pgstac.0.6.2.sql", "content": "CREATE EXTENSION IF NOT EXISTS postgis;\nCREATE EXTENSION IF NOT EXISTS btree_gist;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_admin;\n CREATE ROLE pgstac_read;\n CREATE ROLE pgstac_ingest;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nGRANT pgstac_admin TO current_user;\n\nCREATE SCHEMA IF NOT EXISTS pgstac AUTHORIZATION pgstac_admin;\n\nALTER ROLE pgstac_admin SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_read SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_ingest SET SEARCH_PATH TO pgstac, public;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\n\nGRANT pgstac_read TO pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\n\nSET ROLE pgstac_admin;\n\nSET SEARCH_PATH TO pgstac, public;\n\n\nCREATE TABLE IF NOT EXISTS migrations (\n version text PRIMARY KEY,\n datetime timestamptz DEFAULT clock_timestamp() NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION get_version() RETURNS text AS $$\n SELECT version FROM pgstac.migrations ORDER BY datetime DESC, version DESC LIMIT 1;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION set_version(text) RETURNS text AS $$\n INSERT INTO pgstac.migrations (version) VALUES ($1)\n ON CONFLICT DO NOTHING\n RETURNING version;\n$$ LANGUAGE SQL;\n\n\nCREATE TABLE IF NOT EXISTS pgstac_settings (\n name text PRIMARY KEY,\n value text NOT NULL\n);\n\nINSERT INTO pgstac_settings (name, value) VALUES\n ('context', 'off'),\n ('context_estimated_count', '100000'),\n ('context_estimated_cost', '100000'),\n ('context_stats_ttl', '1 day'),\n ('default-filter-lang', 'cql2-json'),\n ('additional_properties', 'true')\nON CONFLICT DO NOTHING\n;\n\n\nCREATE OR REPLACE FUNCTION get_setting(IN _setting text, IN conf jsonb DEFAULT NULL) RETURNS text AS $$\nSELECT COALESCE(\n conf->>_setting,\n current_setting(concat('pgstac.',_setting), TRUE),\n (SELECT value FROM pgstac.pgstac_settings WHERE name=_setting)\n);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context(conf jsonb DEFAULT NULL) RETURNS text AS $$\n SELECT pgstac.get_setting('context', conf);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context_estimated_count(conf jsonb DEFAULT NULL) RETURNS int AS $$\n SELECT pgstac.get_setting('context_estimated_count', conf)::int;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_estimated_cost();\nCREATE OR REPLACE FUNCTION context_estimated_cost(conf jsonb DEFAULT NULL) RETURNS float AS $$\n SELECT pgstac.get_setting('context_estimated_cost', conf)::float;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_stats_ttl();\nCREATE OR REPLACE FUNCTION context_stats_ttl(conf jsonb DEFAULT NULL) RETURNS interval AS $$\n SELECT pgstac.get_setting('context_stats_ttl', conf)::interval;\n$$ LANGUAGE SQL;\n\n\nCREATE OR REPLACE FUNCTION notice(VARIADIC text[]) RETURNS boolean AS $$\nDECLARE\ndebug boolean := current_setting('pgstac.debug', true);\nBEGIN\n IF debug THEN\n RAISE NOTICE 'NOTICE FROM FUNC: % >>>>> %', concat_ws(' | ', $1), clock_timestamp();\n RETURN TRUE;\n END IF;\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION empty_arr(ANYARRAY) RETURNS BOOLEAN AS $$\nSELECT CASE\n WHEN $1 IS NULL THEN TRUE\n WHEN cardinality($1)<1 THEN TRUE\nELSE FALSE\nEND;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_intersection(_a ANYARRAY, _b ANYARRAY) RETURNS ANYARRAY AS $$\n SELECT ARRAY ( SELECT unnest(_a) INTERSECT SELECT UNNEST(_b) );\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION array_map_ident(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_ident(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_map_literal(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_literal(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION array_reverse(anyarray) RETURNS anyarray AS $$\nSELECT ARRAY(\n SELECT $1[i]\n FROM generate_subscripts($1,1) AS s(i)\n ORDER BY i DESC\n);\n$$ LANGUAGE SQL STRICT IMMUTABLE;\nCREATE OR REPLACE FUNCTION to_int(jsonb) RETURNS int AS $$\n SELECT floor(($1->>0)::float)::int;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION to_float(jsonb) RETURNS float AS $$\n SELECT ($1->>0)::float;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION to_tstz(jsonb) RETURNS timestamptz AS $$\n SELECT ($1->>0)::timestamptz;\n$$ LANGUAGE SQL IMMUTABLE STRICT SET TIME ZONE 'UTC';\n\n\nCREATE OR REPLACE FUNCTION to_text(jsonb) RETURNS text AS $$\n SELECT CASE WHEN jsonb_typeof($1) IN ('array','object') THEN $1::text ELSE $1->>0 END;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION to_text_array(jsonb) RETURNS text[] AS $$\n SELECT\n CASE jsonb_typeof($1)\n WHEN 'array' THEN ARRAY(SELECT jsonb_array_elements_text($1))\n ELSE ARRAY[$1->>0]\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\nCREATE OR REPLACE FUNCTION bbox_geom(_bbox jsonb) RETURNS geometry AS $$\nSELECT CASE jsonb_array_length(_bbox)\n WHEN 4 THEN\n ST_SetSRID(ST_MakeEnvelope(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float,\n (_bbox->>3)::float\n ),4326)\n WHEN 6 THEN\n ST_SetSRID(ST_3DMakeBox(\n ST_MakePoint(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float\n ),\n ST_MakePoint(\n (_bbox->>3)::float,\n (_bbox->>4)::float,\n (_bbox->>5)::float\n )\n ),4326)\n ELSE null END;\n;\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION geom_bbox(_geom geometry) RETURNS jsonb AS $$\n SELECT jsonb_build_array(\n st_xmin(_geom),\n st_ymin(_geom),\n st_xmax(_geom),\n st_ymax(_geom)\n );\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION flip_jsonb_array(j jsonb) RETURNS jsonb AS $$\n SELECT jsonb_agg(value) FROM (SELECT value FROM jsonb_array_elements(j) WITH ORDINALITY ORDER BY ordinality DESC) as t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION explode_dotpaths(j jsonb) RETURNS SETOF text[] AS $$\n SELECT string_to_array(p, '.') as e FROM jsonb_array_elements_text(j) p;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION explode_dotpaths_recurse(IN j jsonb) RETURNS SETOF text[] AS $$\n WITH RECURSIVE t AS (\n SELECT e FROM explode_dotpaths(j) e\n UNION ALL\n SELECT e[1:cardinality(e)-1]\n FROM t\n WHERE cardinality(e)>1\n ) SELECT e FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION jsonb_set_nested(j jsonb, path text[], val jsonb) RETURNS jsonb AS $$\nDECLARE\nBEGIN\n IF cardinality(path) > 1 THEN\n FOR i IN 1..(cardinality(path)-1) LOOP\n IF j #> path[:i] IS NULL THEN\n j := jsonb_set_lax(j, path[:i], '{}', TRUE);\n END IF;\n END LOOP;\n END IF;\n RETURN jsonb_set_lax(j, path, val, true);\n\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\n\n\nCREATE OR REPLACE FUNCTION jsonb_include(j jsonb, f jsonb) RETURNS jsonb AS $$\nDECLARE\n includes jsonb := f-> 'include';\n outj jsonb := '{}'::jsonb;\n path text[];\nBEGIN\n IF\n includes IS NULL\n OR jsonb_array_length(includes) = 0\n THEN\n RETURN j;\n ELSE\n includes := includes || '[\"id\",\"collection\"]'::jsonb;\n FOR path IN SELECT explode_dotpaths(includes) LOOP\n outj := jsonb_set_nested(outj, path, j #> path);\n END LOOP;\n END IF;\n RETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_exclude(j jsonb, f jsonb) RETURNS jsonb AS $$\nDECLARE\n excludes jsonb := f-> 'exclude';\n outj jsonb := j;\n path text[];\nBEGIN\n IF\n excludes IS NULL\n OR jsonb_array_length(excludes) = 0\n THEN\n RETURN j;\n ELSE\n FOR path IN SELECT explode_dotpaths(excludes) LOOP\n outj := outj #- path;\n END LOOP;\n END IF;\n RETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_fields(j jsonb, f jsonb DEFAULT '{\"fields\":[]}') RETURNS jsonb AS $$\n SELECT jsonb_exclude(jsonb_include(j, f), f);\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION merge_jsonb(_a jsonb, _b jsonb) RETURNS jsonb AS $$\n SELECT\n CASE\n WHEN _a = '\"𒍟※\"'::jsonb THEN NULL\n WHEN _a IS NULL OR jsonb_typeof(_a) = 'null' THEN _b\n WHEN jsonb_typeof(_a) = 'object' AND jsonb_typeof(_b) = 'object' THEN\n (\n SELECT\n jsonb_strip_nulls(\n jsonb_object_agg(\n key,\n merge_jsonb(a.value, b.value)\n )\n )\n FROM\n jsonb_each(coalesce(_a,'{}'::jsonb)) as a\n FULL JOIN\n jsonb_each(coalesce(_b,'{}'::jsonb)) as b\n USING (key)\n )\n WHEN\n jsonb_typeof(_a) = 'array'\n AND jsonb_typeof(_b) = 'array'\n AND jsonb_array_length(_a) = jsonb_array_length(_b)\n THEN\n (\n SELECT jsonb_agg(m) FROM\n ( SELECT\n merge_jsonb(\n jsonb_array_elements(_a),\n jsonb_array_elements(_b)\n ) as m\n ) as l\n )\n ELSE _a\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION strip_jsonb(_a jsonb, _b jsonb) RETURNS jsonb AS $$\n SELECT\n CASE\n\n WHEN (_a IS NULL OR jsonb_typeof(_a) = 'null') AND _b IS NOT NULL AND jsonb_typeof(_b) != 'null' THEN '\"𒍟※\"'::jsonb\n WHEN _b IS NULL OR jsonb_typeof(_a) = 'null' THEN _a\n WHEN _a = _b AND jsonb_typeof(_a) = 'object' THEN '{}'::jsonb\n WHEN _a = _b THEN NULL\n WHEN jsonb_typeof(_a) = 'object' AND jsonb_typeof(_b) = 'object' THEN\n (\n SELECT\n jsonb_strip_nulls(\n jsonb_object_agg(\n key,\n strip_jsonb(a.value, b.value)\n )\n )\n FROM\n jsonb_each(_a) as a\n FULL JOIN\n jsonb_each(_b) as b\n USING (key)\n )\n WHEN\n jsonb_typeof(_a) = 'array'\n AND jsonb_typeof(_b) = 'array'\n AND jsonb_array_length(_a) = jsonb_array_length(_b)\n THEN\n (\n SELECT jsonb_agg(m) FROM\n ( SELECT\n strip_jsonb(\n jsonb_array_elements(_a),\n jsonb_array_elements(_b)\n ) as m\n ) as l\n )\n ELSE _a\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE;\n/* looks for a geometry in a stac item first from geometry and falling back to bbox */\nCREATE OR REPLACE FUNCTION stac_geom(value jsonb) RETURNS geometry AS $$\nSELECT\n CASE\n WHEN value ? 'intersects' THEN\n ST_GeomFromGeoJSON(value->>'intersects')\n WHEN value ? 'geometry' THEN\n ST_GeomFromGeoJSON(value->>'geometry')\n WHEN value ? 'bbox' THEN\n pgstac.bbox_geom(value->'bbox')\n ELSE NULL\n END as geometry\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION stac_daterange(\n value jsonb\n) RETURNS tstzrange AS $$\nDECLARE\n props jsonb := value;\n dt timestamptz;\n edt timestamptz;\nBEGIN\n IF props ? 'properties' THEN\n props := props->'properties';\n END IF;\n IF\n props ? 'start_datetime'\n AND props->>'start_datetime' IS NOT NULL\n AND props ? 'end_datetime'\n AND props->>'end_datetime' IS NOT NULL\n THEN\n dt := props->>'start_datetime';\n edt := props->>'end_datetime';\n IF dt > edt THEN\n RAISE EXCEPTION 'start_datetime must be < end_datetime';\n END IF;\n ELSE\n dt := props->>'datetime';\n edt := props->>'datetime';\n END IF;\n IF dt is NULL OR edt IS NULL THEN\n RAISE NOTICE 'DT: %, EDT: %', dt, edt;\n RAISE EXCEPTION 'Either datetime (%) or both start_datetime (%) and end_datetime (%) must be set.', props->>'datetime',props->>'start_datetime',props->>'end_datetime';\n END IF;\n RETURN tstzrange(dt, edt, '[]');\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_datetime(value jsonb) RETURNS timestamptz AS $$\n SELECT lower(stac_daterange(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_end_datetime(value jsonb) RETURNS timestamptz AS $$\n SELECT upper(stac_daterange(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\n\nCREATE TABLE IF NOT EXISTS stac_extensions(\n name text PRIMARY KEY,\n url text,\n enbabled_by_default boolean NOT NULL DEFAULT TRUE,\n enableable boolean NOT NULL DEFAULT TRUE\n);\n\nINSERT INTO stac_extensions (name, url) VALUES\n ('fields', 'https://api.stacspec.org/v1.0.0-beta.5/item-search#fields'),\n ('sort','https://api.stacspec.org/v1.0.0-beta.5/item-search#sort'),\n ('context','https://api.stacspec.org/v1.0.0-beta.5/item-search#context'),\n ('filter', 'https://api.stacspec.org/v1.0.0-beta.5/item-search#filter'),\n ('query', 'https://api.stacspec.org/v1.0.0-beta.5/item-search#query')\nON CONFLICT (name) DO UPDATE SET url=EXCLUDED.url;\n\n\n\nCREATE OR REPLACE FUNCTION collection_base_item(content jsonb) RETURNS jsonb AS $$\n SELECT jsonb_build_object(\n 'type', 'Feature',\n 'stac_version', content->'stac_version',\n 'assets', content->'item_assets',\n 'collection', content->'id'\n );\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TYPE partition_trunc_strategy AS ENUM ('year', 'month');\n\nCREATE TABLE IF NOT EXISTS collections (\n key bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,\n id text GENERATED ALWAYS AS (content->>'id') STORED UNIQUE,\n content JSONB NOT NULL,\n base_item jsonb GENERATED ALWAYS AS (pgstac.collection_base_item(content)) STORED,\n partition_trunc partition_trunc_strategy\n);\n\n\nCREATE OR REPLACE FUNCTION collection_base_item(cid text) RETURNS jsonb AS $$\n SELECT pgstac.collection_base_item(content) FROM pgstac.collections WHERE id = cid LIMIT 1;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION table_empty(text) RETURNS boolean AS $$\nDECLARE\n retval boolean;\nBEGIN\n EXECUTE format($q$\n SELECT NOT EXISTS (SELECT 1 FROM %I LIMIT 1)\n $q$,\n $1\n ) INTO retval;\n RETURN retval;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION collections_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n q text;\n partition_name text := format('_items_%s', NEW.key);\n partition_exists boolean := false;\n partition_empty boolean := true;\n err_context text;\n loadtemp boolean := FALSE;\nBEGIN\n RAISE NOTICE 'Collection Trigger. % %', NEW.id, NEW.key;\n SELECT relid::text INTO partition_name\n FROM pg_partition_tree('items')\n WHERE relid::text = partition_name;\n IF FOUND THEN\n partition_exists := true;\n partition_empty := table_empty(partition_name);\n ELSE\n partition_exists := false;\n partition_empty := true;\n partition_name := format('_items_%s', NEW.key);\n END IF;\n IF TG_OP = 'UPDATE' AND NEW.partition_trunc IS DISTINCT FROM OLD.partition_trunc AND partition_empty THEN\n q := format($q$\n DROP TABLE IF EXISTS %I CASCADE;\n $q$,\n partition_name\n );\n EXECUTE q;\n END IF;\n IF TG_OP = 'UPDATE' AND NEW.partition_trunc IS DISTINCT FROM OLD.partition_trunc AND partition_exists AND NOT partition_empty THEN\n q := format($q$\n CREATE TEMP TABLE changepartitionstaging ON COMMIT DROP AS SELECT * FROM %I;\n DROP TABLE IF EXISTS %I CASCADE;\n $q$,\n partition_name,\n partition_name\n );\n EXECUTE q;\n loadtemp := TRUE;\n partition_empty := TRUE;\n partition_exists := FALSE;\n END IF;\n IF TG_OP = 'UPDATE' AND NEW.partition_trunc IS NOT DISTINCT FROM OLD.partition_trunc THEN\n RETURN NEW;\n END IF;\n IF NEW.partition_trunc IS NULL AND partition_empty THEN\n RAISE NOTICE '% % % %',\n partition_name,\n NEW.id,\n concat(partition_name,'_id_idx'),\n partition_name\n ;\n q := format($q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n $q$,\n partition_name,\n NEW.id,\n concat(partition_name,'_id_idx'),\n partition_name\n );\n RAISE NOTICE 'q: %', q;\n BEGIN\n EXECUTE q;\n EXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', partition_name;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\n END;\n\n ALTER TABLE partitions DISABLE TRIGGER partitions_delete_trigger;\n DELETE FROM partitions WHERE collection=NEW.id AND name=partition_name;\n ALTER TABLE partitions ENABLE TRIGGER partitions_delete_trigger;\n\n INSERT INTO partitions (collection, name) VALUES (NEW.id, partition_name);\n ELSIF partition_empty THEN\n q := format($q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L)\n PARTITION BY RANGE (datetime);\n $q$,\n partition_name,\n NEW.id\n );\n RAISE NOTICE 'q: %', q;\n BEGIN\n EXECUTE q;\n EXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', partition_name;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\n END;\n ALTER TABLE partitions DISABLE TRIGGER partitions_delete_trigger;\n DELETE FROM partitions WHERE collection=NEW.id AND name=partition_name;\n ALTER TABLE partitions ENABLE TRIGGER partitions_delete_trigger;\n ELSE\n RAISE EXCEPTION 'Cannot modify partition % unless empty', partition_name;\n END IF;\n IF loadtemp THEN\n RAISE NOTICE 'Moving data into new partitions.';\n q := format($q$\n WITH p AS (\n SELECT\n collection,\n datetime as datetime,\n end_datetime as end_datetime,\n (partition_name(\n collection,\n datetime\n )).partition_name as name\n FROM changepartitionstaging\n )\n INSERT INTO partitions (collection, datetime_range, end_datetime_range)\n SELECT\n collection,\n tstzrange(min(datetime), max(datetime), '[]') as datetime_range,\n tstzrange(min(end_datetime), max(end_datetime), '[]') as end_datetime_range\n FROM p\n GROUP BY collection, name\n ON CONFLICT (name) DO UPDATE SET\n datetime_range = EXCLUDED.datetime_range,\n end_datetime_range = EXCLUDED.end_datetime_range\n ;\n INSERT INTO %I SELECT * FROM changepartitionstaging;\n DROP TABLE IF EXISTS changepartitionstaging;\n $q$,\n partition_name\n );\n EXECUTE q;\n END IF;\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE TRIGGER collections_trigger AFTER INSERT OR UPDATE ON collections FOR EACH ROW\nEXECUTE FUNCTION collections_trigger_func();\n\nCREATE OR REPLACE FUNCTION partition_collection(collection text, strategy partition_trunc_strategy) RETURNS text AS $$\n UPDATE collections SET partition_trunc=strategy WHERE id=collection RETURNING partition_trunc;\n$$ LANGUAGE SQL;\n\nCREATE TABLE IF NOT EXISTS partitions (\n collection text REFERENCES collections(id) ON DELETE CASCADE,\n name text PRIMARY KEY,\n partition_range tstzrange NOT NULL DEFAULT tstzrange('-infinity'::timestamptz,'infinity'::timestamptz, '[]'),\n datetime_range tstzrange,\n end_datetime_range tstzrange,\n CONSTRAINT prange EXCLUDE USING GIST (\n collection WITH =,\n partition_range WITH &&\n )\n) WITH (FILLFACTOR=90);\nCREATE INDEX partitions_range_idx ON partitions USING GIST(partition_range);\n\nCREATE OR REPLACE FUNCTION partitions_delete_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n q text;\nBEGIN\n RAISE NOTICE 'Partition Delete Trigger. %', OLD.name;\n EXECUTE format($q$\n DROP TABLE IF EXISTS %I CASCADE;\n $q$,\n OLD.name\n );\n RAISE NOTICE 'Dropped partition.';\n RETURN OLD;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER partitions_delete_trigger BEFORE DELETE ON partitions FOR EACH ROW\nEXECUTE FUNCTION partitions_delete_trigger_func();\n\nCREATE OR REPLACE FUNCTION partition_name(\n IN collection text,\n IN dt timestamptz,\n OUT partition_name text,\n OUT partition_range tstzrange\n) AS $$\nDECLARE\n c RECORD;\n parent_name text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n parent_name := format('_items_%s', c.key);\n\n\n IF c.partition_trunc = 'year' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYY'));\n ELSIF c.partition_trunc = 'month' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYYMM'));\n ELSE\n partition_name := parent_name;\n partition_range := tstzrange('-infinity'::timestamptz, 'infinity'::timestamptz, '[]');\n END IF;\n IF partition_range IS NULL THEN\n partition_range := tstzrange(\n date_trunc(c.partition_trunc::text, dt),\n date_trunc(c.partition_trunc::text, dt) + concat('1 ', c.partition_trunc)::interval\n );\n END IF;\n RETURN;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\n\nCREATE OR REPLACE FUNCTION partitions_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n q text;\n cq text;\n parent_name text;\n partition_trunc text;\n partition_name text := NEW.name;\n partition_exists boolean := false;\n partition_empty boolean := true;\n partition_range tstzrange;\n datetime_range tstzrange;\n end_datetime_range tstzrange;\n err_context text;\n mindt timestamptz := lower(NEW.datetime_range);\n maxdt timestamptz := upper(NEW.datetime_range);\n minedt timestamptz := lower(NEW.end_datetime_range);\n maxedt timestamptz := upper(NEW.end_datetime_range);\n t_mindt timestamptz;\n t_maxdt timestamptz;\n t_minedt timestamptz;\n t_maxedt timestamptz;\nBEGIN\n RAISE NOTICE 'Partitions Trigger. %', NEW;\n datetime_range := NEW.datetime_range;\n end_datetime_range := NEW.end_datetime_range;\n\n SELECT\n format('_items_%s', key),\n c.partition_trunc::text\n INTO\n parent_name,\n partition_trunc\n FROM pgstac.collections c\n WHERE c.id = NEW.collection;\n SELECT (pgstac.partition_name(NEW.collection, mindt)).* INTO partition_name, partition_range;\n NEW.name := partition_name;\n\n IF partition_range IS NULL OR partition_range = 'empty'::tstzrange THEN\n partition_range := tstzrange('-infinity'::timestamptz, 'infinity'::timestamptz, '[]');\n END IF;\n\n NEW.partition_range := partition_range;\n IF TG_OP = 'UPDATE' THEN\n mindt := least(mindt, lower(OLD.datetime_range));\n maxdt := greatest(maxdt, upper(OLD.datetime_range));\n minedt := least(minedt, lower(OLD.end_datetime_range));\n maxedt := greatest(maxedt, upper(OLD.end_datetime_range));\n NEW.datetime_range := tstzrange(mindt, maxdt, '[]');\n NEW.end_datetime_range := tstzrange(minedt, maxedt, '[]');\n END IF;\n IF TG_OP = 'INSERT' THEN\n\n IF partition_range != tstzrange('-infinity'::timestamptz, 'infinity'::timestamptz, '[]') THEN\n\n RAISE NOTICE '% % %', partition_name, parent_name, partition_range;\n q := format($q$\n CREATE TABLE IF NOT EXISTS %I partition OF %I FOR VALUES FROM (%L) TO (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n $q$,\n partition_name,\n parent_name,\n lower(partition_range),\n upper(partition_range),\n format('%s_pkey', partition_name),\n partition_name\n );\n BEGIN\n EXECUTE q;\n EXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', partition_name;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\n END;\n END IF;\n\n END IF;\n\n -- Update constraints\n EXECUTE format($q$\n SELECT\n min(datetime),\n max(datetime),\n min(end_datetime),\n max(end_datetime)\n FROM %I;\n $q$, partition_name)\n INTO t_mindt, t_maxdt, t_minedt, t_maxedt;\n mindt := least(mindt, t_mindt);\n maxdt := greatest(maxdt, t_maxdt);\n minedt := least(mindt, minedt, t_minedt);\n maxedt := greatest(maxdt, maxedt, t_maxedt);\n\n mindt := date_trunc(coalesce(partition_trunc, 'year'), mindt);\n maxdt := date_trunc(coalesce(partition_trunc, 'year'), maxdt - '1 second'::interval) + concat('1 ',coalesce(partition_trunc, 'year'))::interval;\n minedt := date_trunc(coalesce(partition_trunc, 'year'), minedt);\n maxedt := date_trunc(coalesce(partition_trunc, 'year'), maxedt - '1 second'::interval) + concat('1 ',coalesce(partition_trunc, 'year'))::interval;\n\n\n IF mindt IS NOT NULL AND maxdt IS NOT NULL AND minedt IS NOT NULL AND maxedt IS NOT NULL THEN\n NEW.datetime_range := tstzrange(mindt, maxdt, '[]');\n NEW.end_datetime_range := tstzrange(minedt, maxedt, '[]');\n IF\n TG_OP='UPDATE'\n AND OLD.datetime_range @> NEW.datetime_range\n AND OLD.end_datetime_range @> NEW.end_datetime_range\n THEN\n RAISE NOTICE 'Range unchanged, not updating constraints.';\n ELSE\n\n RAISE NOTICE '\n SETTING CONSTRAINTS\n mindt: %, maxdt: %\n minedt: %, maxedt: %\n ', mindt, maxdt, minedt, maxedt;\n IF partition_trunc IS NULL THEN\n cq := format($q$\n ALTER TABLE %7$I\n DROP CONSTRAINT IF EXISTS %1$I,\n DROP CONSTRAINT IF EXISTS %2$I,\n ADD CONSTRAINT %1$I\n CHECK (\n (datetime >= %3$L)\n AND (datetime <= %4$L)\n AND (end_datetime >= %5$L)\n AND (end_datetime <= %6$L)\n ) NOT VALID\n ;\n ALTER TABLE %7$I\n VALIDATE CONSTRAINT %1$I;\n $q$,\n format('%s_dt', partition_name),\n format('%s_edt', partition_name),\n mindt,\n maxdt,\n minedt,\n maxedt,\n partition_name\n );\n ELSE\n cq := format($q$\n ALTER TABLE %5$I\n DROP CONSTRAINT IF EXISTS %1$I,\n DROP CONSTRAINT IF EXISTS %2$I,\n ADD CONSTRAINT %2$I\n CHECK ((end_datetime >= %3$L) AND (end_datetime <= %4$L)) NOT VALID\n ;\n ALTER TABLE %5$I\n VALIDATE CONSTRAINT %2$I;\n $q$,\n format('%s_dt', partition_name),\n format('%s_edt', partition_name),\n minedt,\n maxedt,\n partition_name\n );\n\n END IF;\n RAISE NOTICE 'Altering Constraints. %', cq;\n EXECUTE cq;\n END IF;\n ELSE\n NEW.datetime_range = NULL;\n NEW.end_datetime_range = NULL;\n\n cq := format($q$\n ALTER TABLE %3$I\n DROP CONSTRAINT IF EXISTS %1$I,\n DROP CONSTRAINT IF EXISTS %2$I,\n ADD CONSTRAINT %1$I\n CHECK ((datetime IS NULL AND end_datetime IS NULL)) NOT VALID\n ;\n ALTER TABLE %3$I\n VALIDATE CONSTRAINT %1$I;\n $q$,\n format('%s_dt', partition_name),\n format('%s_edt', partition_name),\n partition_name\n );\n EXECUTE cq;\n END IF;\n\n RETURN NEW;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER partitions_trigger BEFORE INSERT OR UPDATE ON partitions FOR EACH ROW\nEXECUTE FUNCTION partitions_trigger_func();\n\n\nCREATE OR REPLACE FUNCTION create_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection(data jsonb) RETURNS VOID AS $$\nDECLARE\n out collections%ROWTYPE;\nBEGIN\n UPDATE collections SET content=data WHERE id = data->>'id' RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ON CONFLICT (id) DO\n UPDATE\n SET content=EXCLUDED.content\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION delete_collection(_id text) RETURNS VOID AS $$\nDECLARE\n out collections%ROWTYPE;\nBEGIN\n DELETE FROM collections WHERE id = _id RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION get_collection(id text) RETURNS jsonb AS $$\n SELECT content FROM collections\n WHERE id=$1\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION all_collections() RETURNS jsonb AS $$\n SELECT jsonb_agg(content) FROM collections;\n;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\nCREATE TABLE queryables (\n id bigint GENERATED ALWAYS AS identity PRIMARY KEY,\n name text UNIQUE NOT NULL,\n collection_ids text[], -- used to determine what partitions to create indexes on\n definition jsonb,\n property_path text,\n property_wrapper text,\n property_index_type text\n);\nCREATE INDEX queryables_name_idx ON queryables (name);\nCREATE INDEX queryables_property_wrapper_idx ON queryables (property_wrapper);\n\n\nINSERT INTO queryables (name, definition) VALUES\n('id', '{\"title\": \"Item ID\",\"description\": \"Item identifier\",\"$ref\": \"https://schemas.stacspec.org/v1.0.0/item-spec/json-schema/item.json#/definitions/core/allOf/2/properties/id\"}'),\n('datetime','{\"description\": \"Datetime\",\"type\": \"string\",\"title\": \"Acquired\",\"format\": \"date-time\",\"pattern\": \"(\\\\+00:00|Z)$\"}')\nON CONFLICT DO NOTHING;\n\n\n\nINSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n('eo:cloud_cover','{\"$ref\": \"https://stac-extensions.github.io/eo/v1.0.0/schema.json#/definitions/fieldsproperties/eo:cloud_cover\"}','to_int','BTREE')\nON CONFLICT DO NOTHING;\n\nCREATE OR REPLACE FUNCTION array_to_path(arr text[]) RETURNS text AS $$\n SELECT string_agg(\n quote_literal(v),\n '->'\n ) FROM unnest(arr) v;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\n\n\nCREATE OR REPLACE FUNCTION queryable(\n IN dotpath text,\n OUT path text,\n OUT expression text,\n OUT wrapper text\n) AS $$\nDECLARE\n q RECORD;\n path_elements text[];\nBEGIN\n IF dotpath IN ('id', 'geometry', 'datetime', 'end_datetime', 'collection') THEN\n path := dotpath;\n expression := dotpath;\n wrapper := NULL;\n RETURN;\n END IF;\n SELECT * INTO q FROM queryables WHERE name=dotpath;\n IF q.property_wrapper IS NULL THEN\n IF q.definition->>'type' = 'number' THEN\n wrapper := 'to_float';\n ELSIF q.definition->>'format' = 'date-time' THEN\n wrapper := 'to_tstz';\n ELSE\n wrapper := 'to_text';\n END IF;\n ELSE\n wrapper := q.property_wrapper;\n END IF;\n IF q.property_path IS NOT NULL THEN\n path := q.property_path;\n ELSE\n path_elements := string_to_array(dotpath, '.');\n IF path_elements[1] IN ('links', 'assets', 'stac_version', 'stac_extensions') THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSIF path_elements[1] = 'properties' THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSE\n path := format($F$content->'properties'->%s$F$, array_to_path(path_elements));\n END IF;\n END IF;\n expression := format('%I(%s)', wrapper, path);\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\nCREATE OR REPLACE FUNCTION create_queryable_indexes() RETURNS VOID AS $$\nDECLARE\n queryable RECORD;\n q text;\nBEGIN\n FOR queryable IN\n SELECT\n queryables.id as qid,\n CASE WHEN collections.key IS NULL THEN 'items' ELSE format('_items_%s',collections.key) END AS part,\n property_index_type,\n expression\n FROM\n queryables\n LEFT JOIN collections ON (collections.id = ANY (queryables.collection_ids))\n JOIN LATERAL queryable(queryables.name) ON (queryables.property_index_type IS NOT NULL)\n LOOP\n q := format(\n $q$\n CREATE INDEX IF NOT EXISTS %I ON %I USING %s ((%s));\n $q$,\n format('%s_%s_idx', queryable.part, queryable.qid),\n queryable.part,\n COALESCE(queryable.property_index_type, 'to_text'),\n queryable.expression\n );\n RAISE NOTICE '%',q;\n EXECUTE q;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION queryables_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\nBEGIN\nPERFORM create_queryable_indexes();\nRETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER queryables_trigger AFTER INSERT OR UPDATE ON queryables\nFOR EACH STATEMENT EXECUTE PROCEDURE queryables_trigger_func();\n\nCREATE TRIGGER queryables_collection_trigger AFTER INSERT OR UPDATE ON collections\nFOR EACH STATEMENT EXECUTE PROCEDURE queryables_trigger_func();\nCREATE OR REPLACE FUNCTION parse_dtrange(\n _indate jsonb,\n relative_base timestamptz DEFAULT date_trunc('hour', CURRENT_TIMESTAMP)\n) RETURNS tstzrange AS $$\nDECLARE\n timestrs text[];\n s timestamptz;\n e timestamptz;\nBEGIN\n timestrs :=\n CASE\n WHEN _indate ? 'timestamp' THEN\n ARRAY[_indate->>'timestamp']\n WHEN _indate ? 'interval' THEN\n to_text_array(_indate->'interval')\n WHEN jsonb_typeof(_indate) = 'array' THEN\n to_text_array(_indate)\n ELSE\n regexp_split_to_array(\n _indate->>0,\n '/'\n )\n END;\n RAISE NOTICE 'TIMESTRS %', timestrs;\n IF cardinality(timestrs) = 1 THEN\n IF timestrs[1] ILIKE 'P%' THEN\n RETURN tstzrange(relative_base - upper(timestrs[1])::interval, relative_base, '[)');\n END IF;\n s := timestrs[1]::timestamptz;\n RETURN tstzrange(s, s, '[]');\n END IF;\n\n IF cardinality(timestrs) != 2 THEN\n RAISE EXCEPTION 'Timestamp cannot have more than 2 values';\n END IF;\n\n IF timestrs[1] = '..' THEN\n s := '-infinity'::timestamptz;\n e := timestrs[2]::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] = '..' THEN\n s := timestrs[1]::timestamptz;\n e := 'infinity'::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[1] ILIKE 'P%' AND timestrs[2] NOT ILIKE 'P%' THEN\n e := timestrs[2]::timestamptz;\n s := e - upper(timestrs[1])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] ILIKE 'P%' AND timestrs[1] NOT ILIKE 'P%' THEN\n s := timestrs[1]::timestamptz;\n e := s + upper(timestrs[2])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n s := timestrs[1]::timestamptz;\n e := timestrs[2]::timestamptz;\n\n RETURN tstzrange(s,e,'[)');\n\n RETURN NULL;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT PARALLEL SAFE SET TIME ZONE 'UTC';\n\nCREATE OR REPLACE FUNCTION parse_dtrange(\n _indate text,\n relative_base timestamptz DEFAULT CURRENT_TIMESTAMP\n) RETURNS tstzrange AS $$\n SELECT parse_dtrange(to_jsonb(_indate), relative_base);\n$$ LANGUAGE SQL STABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION temporal_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\n ll text := 'datetime';\n lh text := 'end_datetime';\n rrange tstzrange;\n rl text;\n rh text;\n outq text;\nBEGIN\n rrange := parse_dtrange(args->1);\n RAISE NOTICE 'Constructing temporal query OP: %, ARGS: %, RRANGE: %', op, args, rrange;\n op := lower(op);\n rl := format('%L::timestamptz', lower(rrange));\n rh := format('%L::timestamptz', upper(rrange));\n outq := CASE op\n WHEN 't_before' THEN 'lh < rl'\n WHEN 't_after' THEN 'll > rh'\n WHEN 't_meets' THEN 'lh = rl'\n WHEN 't_metby' THEN 'll = rh'\n WHEN 't_overlaps' THEN 'll < rl AND rl < lh < rh'\n WHEN 't_overlappedby' THEN 'rl < ll < rh AND lh > rh'\n WHEN 't_starts' THEN 'll = rl AND lh < rh'\n WHEN 't_startedby' THEN 'll = rl AND lh > rh'\n WHEN 't_during' THEN 'll > rl AND lh < rh'\n WHEN 't_contains' THEN 'll < rl AND lh > rh'\n WHEN 't_finishes' THEN 'll > rl AND lh = rh'\n WHEN 't_finishedby' THEN 'll < rl AND lh = rh'\n WHEN 't_equals' THEN 'll = rl AND lh = rh'\n WHEN 't_disjoint' THEN 'NOT (ll <= rh AND lh >= rl)'\n WHEN 't_intersects' THEN 'll <= rh AND lh >= rl'\n WHEN 'anyinteracts' THEN 'll <= rh AND lh >= rl'\n END;\n outq := regexp_replace(outq, '\\mll\\M', ll);\n outq := regexp_replace(outq, '\\mlh\\M', lh);\n outq := regexp_replace(outq, '\\mrl\\M', rl);\n outq := regexp_replace(outq, '\\mrh\\M', rh);\n outq := format('(%s)', outq);\n RETURN outq;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\n\n\nCREATE OR REPLACE FUNCTION spatial_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\n geom text;\n j jsonb := args->1;\nBEGIN\n op := lower(op);\n RAISE NOTICE 'Constructing spatial query OP: %, ARGS: %', op, args;\n IF op NOT IN ('s_equals','s_disjoint','s_touches','s_within','s_overlaps','s_crosses','s_intersects','intersects','s_contains') THEN\n RAISE EXCEPTION 'Spatial Operator % Not Supported', op;\n END IF;\n op := regexp_replace(op, '^s_', 'st_');\n IF op = 'intersects' THEN\n op := 'st_intersects';\n END IF;\n -- Convert geometry to WKB string\n IF j ? 'type' AND j ? 'coordinates' THEN\n geom := st_geomfromgeojson(j)::text;\n ELSIF jsonb_typeof(j) = 'array' THEN\n geom := bbox_geom(j)::text;\n END IF;\n\n RETURN format('%s(geometry, %L::geometry)', op, geom);\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION query_to_cql2(q jsonb) RETURNS jsonb AS $$\n-- Translates anything passed in through the deprecated \"query\" into equivalent CQL2\nWITH t AS (\n SELECT key as property, value as ops\n FROM jsonb_each(q)\n), t2 AS (\n SELECT property, (jsonb_each(ops)).*\n FROM t WHERE jsonb_typeof(ops) = 'object'\n UNION ALL\n SELECT property, 'eq', ops\n FROM t WHERE jsonb_typeof(ops) != 'object'\n)\nSELECT\n jsonb_strip_nulls(jsonb_build_object(\n 'op', 'and',\n 'args', jsonb_agg(\n jsonb_build_object(\n 'op', key,\n 'args', jsonb_build_array(\n jsonb_build_object('property',property),\n value\n )\n )\n )\n )\n) as qcql FROM t2\n;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\nCREATE OR REPLACE FUNCTION cql1_to_cql2(j jsonb) RETURNS jsonb AS $$\nDECLARE\n args jsonb;\n ret jsonb;\nBEGIN\n RAISE NOTICE 'CQL1_TO_CQL2: %', j;\n IF j ? 'filter' THEN\n RETURN cql1_to_cql2(j->'filter');\n END IF;\n IF j ? 'property' THEN\n RETURN j;\n END IF;\n IF jsonb_typeof(j) = 'array' THEN\n SELECT jsonb_agg(cql1_to_cql2(el)) INTO args FROM jsonb_array_elements(j) el;\n RETURN args;\n END IF;\n IF jsonb_typeof(j) = 'number' THEN\n RETURN j;\n END IF;\n IF jsonb_typeof(j) = 'string' THEN\n RETURN j;\n END IF;\n\n IF jsonb_typeof(j) = 'object' THEN\n SELECT jsonb_build_object(\n 'op', key,\n 'args', cql1_to_cql2(value)\n ) INTO ret\n FROM jsonb_each(j)\n WHERE j IS NOT NULL;\n RETURN ret;\n END IF;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE STRICT;\n\nCREATE TABLE cql2_ops (\n op text PRIMARY KEY,\n template text,\n types text[]\n);\nINSERT INTO cql2_ops (op, template, types) VALUES\n ('eq', '%s = %s', NULL),\n ('lt', '%s < %s', NULL),\n ('lte', '%s <= %s', NULL),\n ('gt', '%s > %s', NULL),\n ('gte', '%s >= %s', NULL),\n ('le', '%s <= %s', NULL),\n ('ge', '%s >= %s', NULL),\n ('=', '%s = %s', NULL),\n ('<', '%s < %s', NULL),\n ('<=', '%s <= %s', NULL),\n ('>', '%s > %s', NULL),\n ('>=', '%s >= %s', NULL),\n ('like', '%s LIKE %s', NULL),\n ('ilike', '%s ILIKE %s', NULL),\n ('+', '%s + %s', NULL),\n ('-', '%s - %s', NULL),\n ('*', '%s * %s', NULL),\n ('/', '%s / %s', NULL),\n ('in', '%s = ANY (%s)', NULL),\n ('not', 'NOT (%s)', NULL),\n ('between', '%s BETWEEN (%2$s)[1] AND (%2$s)[2]', NULL),\n ('isnull', '%s IS NULL', NULL)\nON CONFLICT (op) DO UPDATE\n SET\n template = EXCLUDED.template;\n;\n\n\nCREATE OR REPLACE FUNCTION cql2_query(j jsonb, wrapper text DEFAULT NULL) RETURNS text AS $$\n#variable_conflict use_variable\nDECLARE\n args jsonb := j->'args';\n arg jsonb;\n op text := lower(j->>'op');\n cql2op RECORD;\n literal text;\nBEGIN\n IF j IS NULL OR (op IS NOT NULL AND args IS NULL) THEN\n RETURN NULL;\n END IF;\n RAISE NOTICE 'CQL2_QUERY: %', j;\n IF j ? 'filter' THEN\n RETURN cql2_query(j->'filter');\n END IF;\n\n IF j ? 'upper' THEN\n RETURN format('upper(%s)', cql2_query(j->'upper'));\n END IF;\n\n IF j ? 'lower' THEN\n RETURN format('lower(%s)', cql2_query(j->'lower'));\n END IF;\n\n -- Temporal Query\n IF op ilike 't_%' or op = 'anyinteracts' THEN\n RETURN temporal_op_query(op, args);\n END IF;\n\n -- If property is a timestamp convert it to text to use with\n -- general operators\n IF j ? 'timestamp' THEN\n RETURN format('%L::timestamptz', to_tstz(j->'timestamp'));\n END IF;\n IF j ? 'interval' THEN\n RAISE EXCEPTION 'Please use temporal operators when using intervals.';\n RETURN NONE;\n END IF;\n\n -- Spatial Query\n IF op ilike 's_%' or op = 'intersects' THEN\n RETURN spatial_op_query(op, args);\n END IF;\n\n\n IF op = 'in' THEN\n RETURN format(\n '%s = ANY (%L)',\n cql2_query(args->0),\n to_text_array(args->1)\n );\n END IF;\n\n\n\n IF op = 'between' THEN\n SELECT (queryable(a->>'property')).wrapper INTO wrapper\n FROM jsonb_array_elements(args) a\n WHERE a ? 'property' LIMIT 1;\n\n RETURN format(\n '%s BETWEEN %s and %s',\n cql2_query(args->0, wrapper),\n cql2_query(args->1->0, wrapper),\n cql2_query(args->1->1, wrapper)\n );\n END IF;\n\n -- Make sure that args is an array and run cql2_query on\n -- each element of the array\n RAISE NOTICE 'ARGS PRE: %', args;\n IF j ? 'args' THEN\n IF jsonb_typeof(args) != 'array' THEN\n args := jsonb_build_array(args);\n END IF;\n\n SELECT (queryable(a->>'property')).wrapper INTO wrapper\n FROM jsonb_array_elements(args) a\n WHERE a ? 'property' LIMIT 1;\n\n SELECT jsonb_agg(cql2_query(a, wrapper))\n INTO args\n FROM jsonb_array_elements(args) a;\n END IF;\n RAISE NOTICE 'ARGS: %', args;\n\n IF op IN ('and', 'or') THEN\n RETURN\n format(\n '(%s)',\n array_to_string(to_text_array(args), format(' %s ', upper(op)))\n );\n END IF;\n\n -- Look up template from cql2_ops\n IF j ? 'op' THEN\n SELECT * INTO cql2op FROM cql2_ops WHERE cql2_ops.op ilike op;\n IF FOUND THEN\n -- If specific index set in queryables for a property cast other arguments to that type\n RETURN format(\n cql2op.template,\n VARIADIC (to_text_array(args))\n );\n ELSE\n RAISE EXCEPTION 'Operator % Not Supported.', op;\n END IF;\n END IF;\n\n\n IF j ? 'property' THEN\n RETURN (queryable(j->>'property')).expression;\n END IF;\n\n IF wrapper IS NOT NULL THEN\n EXECUTE format('SELECT %I(%L)', wrapper, j) INTO literal;\n RAISE NOTICE '% % %',wrapper, j, literal;\n RETURN format('%I(%L)', wrapper, j);\n END IF;\n\n RETURN quote_literal(to_text(j));\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION paging_dtrange(\n j jsonb\n) RETURNS tstzrange AS $$\nDECLARE\n op text;\n filter jsonb := j->'filter';\n dtrange tstzrange := tstzrange('-infinity'::timestamptz,'infinity'::timestamptz);\n sdate timestamptz := '-infinity'::timestamptz;\n edate timestamptz := 'infinity'::timestamptz;\n jpitem jsonb;\nBEGIN\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n END IF;\n IF NOT (filter @? '$.**.op ? (@ == \"or\" || @ == \"not\")') THEN\n FOR jpitem IN SELECT j FROM jsonb_path_query(filter,'strict $.** ? (@.args[*].property == \"datetime\")'::jsonpath) j LOOP\n op := lower(jpitem->>'op');\n dtrange := parse_dtrange(jpitem->'args'->1);\n IF op IN ('<=', 'lt', 'lte', '<', 'le', 't_before') THEN\n sdate := greatest(sdate,'-infinity');\n edate := least(edate, upper(dtrange));\n ELSIF op IN ('>=', '>', 'gt', 'gte', 'ge', 't_after') THEN\n edate := least(edate, 'infinity');\n sdate := greatest(sdate, lower(dtrange));\n ELSIF op IN ('=', 'eq') THEN\n edate := least(edate, upper(dtrange));\n sdate := greatest(sdate, lower(dtrange));\n END IF;\n RAISE NOTICE '2 OP: %, ARGS: %, DTRANGE: %, SDATE: %, EDATE: %', op, jpitem->'args'->1, dtrange, sdate, edate;\n END LOOP;\n END IF;\n IF sdate > edate THEN\n RETURN 'empty'::tstzrange;\n END IF;\n RETURN tstzrange(sdate,edate, '[]');\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT SET TIME ZONE 'UTC';\n\nCREATE OR REPLACE FUNCTION paging_collections(\n IN j jsonb\n) RETURNS text[] AS $$\nDECLARE\n filter jsonb := j->'filter';\n jpitem jsonb;\n op text;\n args jsonb;\n arg jsonb;\n collections text[];\nBEGIN\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n END IF;\n IF NOT (filter @? '$.**.op ? (@ == \"or\" || @ == \"not\")') THEN\n FOR jpitem IN SELECT j FROM jsonb_path_query(filter,'strict $.** ? (@.args[*].property == \"collection\")'::jsonpath) j LOOP\n RAISE NOTICE 'JPITEM: %', jpitem;\n op := jpitem->>'op';\n args := jpitem->'args';\n IF op IN ('=', 'eq', 'in') THEN\n FOR arg IN SELECT a FROM jsonb_array_elements(args) a LOOP\n IF jsonb_typeof(arg) IN ('string', 'array') THEN\n RAISE NOTICE 'arg: %, collections: %', arg, collections;\n IF collections IS NULL OR collections = '{}'::text[] THEN\n collections := to_text_array(arg);\n ELSE\n collections := array_intersection(collections, to_text_array(arg));\n END IF;\n END IF;\n END LOOP;\n END IF;\n END LOOP;\n END IF;\n IF collections = '{}'::text[] THEN\n RETURN NULL;\n END IF;\n RETURN collections;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\nCREATE TABLE items (\n id text NOT NULL,\n geometry geometry NOT NULL,\n collection text NOT NULL,\n datetime timestamptz NOT NULL,\n end_datetime timestamptz NOT NULL,\n content JSONB NOT NULL\n)\nPARTITION BY LIST (collection)\n;\n\nCREATE INDEX \"datetime_idx\" ON items USING BTREE (datetime DESC, end_datetime ASC);\nCREATE INDEX \"geometry_idx\" ON items USING GIST (geometry);\n\nCREATE STATISTICS datetime_stats (dependencies) on datetime, end_datetime from items;\n\n\nALTER TABLE items ADD CONSTRAINT items_collections_fk FOREIGN KEY (collection) REFERENCES collections(id) ON DELETE CASCADE DEFERRABLE;\n\n\nCREATE OR REPLACE FUNCTION content_slim(_item jsonb) RETURNS jsonb AS $$\n SELECT strip_jsonb(_item - '{id,geometry,collection,type}'::text[], collection_base_item(_item->>'collection')) - '{id,geometry,collection,type}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_dehydrate(content jsonb) RETURNS items AS $$\n SELECT\n content->>'id' as id,\n stac_geom(content) as geometry,\n content->>'collection' as collection,\n stac_datetime(content) as datetime,\n stac_end_datetime(content) as end_datetime,\n content_slim(content) as content\n ;\n$$ LANGUAGE SQL STABLE;\n\nCREATE OR REPLACE FUNCTION include_field(f text, fields jsonb DEFAULT '{}'::jsonb) RETURNS boolean AS $$\nDECLARE\n includes jsonb := fields->'include';\n excludes jsonb := fields->'exclude';\nBEGIN\n IF f IS NULL THEN\n RETURN NULL;\n END IF;\n\n\n IF\n jsonb_typeof(excludes) = 'array'\n AND jsonb_array_length(excludes)>0\n AND excludes ? f\n THEN\n RETURN FALSE;\n END IF;\n\n IF\n (\n jsonb_typeof(includes) = 'array'\n AND jsonb_array_length(includes) > 0\n AND includes ? f\n ) OR\n (\n includes IS NULL\n OR jsonb_typeof(includes) = 'null'\n OR jsonb_array_length(includes) = 0\n )\n THEN\n RETURN TRUE;\n END IF;\n\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nDROP FUNCTION IF EXISTS content_hydrate(jsonb, jsonb, jsonb);\nCREATE OR REPLACE FUNCTION content_hydrate(\n _base_item jsonb,\n _item jsonb,\n fields jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\n SELECT merge_jsonb(\n jsonb_fields(_item, fields),\n jsonb_fields(_base_item, fields)\n );\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION content_hydrate(_item items, _collection collections, fields jsonb DEFAULT '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\n content jsonb;\n base_item jsonb := _collection.base_item;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry, 20)::jsonb;\n END IF;\n output := content_hydrate(\n jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'collection', _item.collection,\n 'type', 'Feature'\n ) || _item.content,\n _collection.base_item,\n fields\n );\n\n RETURN output;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_nonhydrated(\n _item items,\n fields jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry, 20)::jsonb;\n END IF;\n output := jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'collection', _item.collection,\n 'type', 'Feature'\n ) || _item.content;\n RETURN output;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_hydrate(_item items, fields jsonb DEFAULT '{}'::jsonb) RETURNS jsonb AS $$\n SELECT content_hydrate(\n _item,\n (SELECT c FROM collections c WHERE id=_item.collection LIMIT 1),\n fields\n );\n$$ LANGUAGE SQL STABLE;\n\n\nCREATE UNLOGGED TABLE items_staging (\n content JSONB NOT NULL\n);\nCREATE UNLOGGED TABLE items_staging_ignore (\n content JSONB NOT NULL\n);\nCREATE UNLOGGED TABLE items_staging_upsert (\n content JSONB NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION items_staging_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p record;\n _partitions text[];\n ts timestamptz := clock_timestamp();\nBEGIN\n RAISE NOTICE 'Creating Partitions. %', clock_timestamp() - ts;\n WITH ranges AS (\n SELECT\n n.content->>'collection' as collection,\n stac_daterange(n.content->'properties') as dtr\n FROM newdata n\n ), p AS (\n SELECT\n collection,\n lower(dtr) as datetime,\n upper(dtr) as end_datetime,\n (partition_name(\n collection,\n lower(dtr)\n )).partition_name as name\n FROM ranges\n )\n INSERT INTO partitions (collection, datetime_range, end_datetime_range)\n SELECT\n collection,\n tstzrange(min(datetime), max(datetime), '[]') as datetime_range,\n tstzrange(min(end_datetime), max(end_datetime), '[]') as end_datetime_range\n FROM p\n GROUP BY collection, name\n ON CONFLICT (name) DO UPDATE SET\n datetime_range = EXCLUDED.datetime_range,\n end_datetime_range = EXCLUDED.end_datetime_range\n ;\n\n RAISE NOTICE 'Doing the insert. %', clock_timestamp() - ts;\n IF TG_TABLE_NAME = 'items_staging' THEN\n INSERT INTO items\n SELECT\n (content_dehydrate(content)).*\n FROM newdata;\n DELETE FROM items_staging;\n ELSIF TG_TABLE_NAME = 'items_staging_ignore' THEN\n INSERT INTO items\n SELECT\n (content_dehydrate(content)).*\n FROM newdata\n ON CONFLICT DO NOTHING;\n DELETE FROM items_staging_ignore;\n ELSIF TG_TABLE_NAME = 'items_staging_upsert' THEN\n WITH staging_formatted AS (\n SELECT (content_dehydrate(content)).* FROM newdata\n ), deletes AS (\n DELETE FROM items i USING staging_formatted s\n WHERE\n i.id = s.id\n AND i.collection = s.collection\n AND i IS DISTINCT FROM s\n RETURNING i.id, i.collection\n )\n INSERT INTO items\n SELECT s.* FROM\n staging_formatted s\n JOIN deletes d\n USING (id, collection);\n DELETE FROM items_staging_upsert;\n END IF;\n RAISE NOTICE 'Done. %', clock_timestamp() - ts;\n\n RETURN NULL;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER items_staging_insert_trigger AFTER INSERT ON items_staging REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\nCREATE TRIGGER items_staging_insert_ignore_trigger AFTER INSERT ON items_staging_ignore REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\nCREATE TRIGGER items_staging_insert_upsert_trigger AFTER INSERT ON items_staging_upsert REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\n\n\n\nCREATE OR REPLACE FUNCTION item_by_id(_id text, _collection text DEFAULT NULL) RETURNS items AS\n$$\nDECLARE\n i items%ROWTYPE;\nBEGIN\n SELECT * INTO i FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection) LIMIT 1;\n RETURN i;\nEND;\n$$ LANGUAGE PLPGSQL STABLE SECURITY DEFINER SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION get_item(_id text, _collection text DEFAULT NULL) RETURNS jsonb AS $$\n SELECT content_hydrate(items) FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection);\n$$ LANGUAGE SQL STABLE SECURITY DEFINER SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION delete_item(_id text, _collection text DEFAULT NULL) RETURNS VOID AS $$\nDECLARE\nout items%ROWTYPE;\nBEGIN\n DELETE FROM items WHERE id = _id AND (_collection IS NULL OR collection=_collection) RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n--/*\nCREATE OR REPLACE FUNCTION create_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION update_item(content jsonb) RETURNS VOID AS $$\nDECLARE\n old items %ROWTYPE;\n out items%ROWTYPE;\nBEGIN\n PERFORM delete_item(content->>'id', content->>'collection');\n PERFORM create_item(content);\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION create_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION collection_bbox(id text) RETURNS jsonb AS $$\n SELECT (replace(replace(replace(st_extent(geometry)::text,'BOX(','[['),')',']]'),' ',','))::jsonb\n FROM items WHERE collection=$1;\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION collection_temporal_extent(id text) RETURNS jsonb AS $$\n SELECT to_jsonb(array[array[min(datetime)::text, max(datetime)::text]])\n FROM items WHERE collection=$1;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection_extents() RETURNS VOID AS $$\nUPDATE collections SET\n content = content ||\n jsonb_build_object(\n 'extent', jsonb_build_object(\n 'spatial', jsonb_build_object(\n 'bbox', collection_bbox(collections.id)\n ),\n 'temporal', jsonb_build_object(\n 'interval', collection_temporal_extent(collections.id)\n )\n )\n )\n;\n$$ LANGUAGE SQL;\nCREATE VIEW partition_steps AS\nSELECT\n name,\n date_trunc('month',lower(datetime_range)) as sdate,\n date_trunc('month', upper(datetime_range)) + '1 month'::interval as edate\n FROM partitions WHERE datetime_range IS NOT NULL AND datetime_range != 'empty'::tstzrange\n ORDER BY datetime_range ASC\n;\n\nCREATE OR REPLACE FUNCTION chunker(\n IN _where text,\n OUT s timestamptz,\n OUT e timestamptz\n) RETURNS SETOF RECORD AS $$\nDECLARE\n explain jsonb;\nBEGIN\n IF _where IS NULL THEN\n _where := ' TRUE ';\n END IF;\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s;', _where)\n INTO explain;\n\n RETURN QUERY\n WITH t AS (\n SELECT j->>0 as p FROM\n jsonb_path_query(\n explain,\n 'strict $.**.\"Relation Name\" ? (@ != null)'\n ) j\n ),\n parts AS (\n SELECT sdate, edate FROM t JOIN partition_steps ON (t.p = name)\n ),\n times AS (\n SELECT sdate FROM parts\n UNION\n SELECT edate FROM parts\n ),\n uniq AS (\n SELECT DISTINCT sdate FROM times ORDER BY sdate\n ),\n last AS (\n SELECT sdate, lead(sdate, 1) over () as edate FROM uniq\n )\n SELECT sdate, edate FROM last WHERE edate IS NOT NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION partition_queries(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN partitions text[] DEFAULT NULL\n) RETURNS SETOF text AS $$\nDECLARE\n query text;\n sdate timestamptz;\n edate timestamptz;\nBEGIN\nIF _where IS NULL OR trim(_where) = '' THEN\n _where = ' TRUE ';\nEND IF;\nRAISE NOTICE 'Getting chunks for % %', _where, _orderby;\nIF _orderby ILIKE 'datetime d%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 DESC LOOP\n RETURN NEXT format($q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n $q$,\n sdate,\n edate,\n _where,\n _orderby\n );\n END LOOP;\nELSIF _orderby ILIKE 'datetime a%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 ASC LOOP\n RETURN NEXT format($q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n $q$,\n sdate,\n edate,\n _where,\n _orderby\n );\n END LOOP;\nELSE\n query := format($q$\n SELECT * FROM items\n WHERE %s\n ORDER BY %s\n $q$, _where, _orderby\n );\n\n RETURN NEXT query;\n RETURN;\nEND IF;\n\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION partition_query_view(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN _limit int DEFAULT 10\n) RETURNS text AS $$\n WITH p AS (\n SELECT * FROM partition_queries(_where, _orderby) p\n )\n SELECT\n CASE WHEN EXISTS (SELECT 1 FROM p) THEN\n (SELECT format($q$\n SELECT * FROM (\n %s\n ) total LIMIT %s\n $q$,\n string_agg(\n format($q$ SELECT * FROM ( %s ) AS sub $q$, p),\n '\n UNION ALL\n '\n ),\n _limit\n ))\n ELSE NULL\n END FROM p;\n$$ LANGUAGE SQL IMMUTABLE;\n\n\n\n\nCREATE OR REPLACE FUNCTION stac_search_to_where(j jsonb) RETURNS text AS $$\nDECLARE\n where_segments text[];\n _where text;\n dtrange tstzrange;\n collections text[];\n geom geometry;\n sdate timestamptz;\n edate timestamptz;\n filterlang text;\n filter jsonb := j->'filter';\nBEGIN\n IF j ? 'ids' THEN\n where_segments := where_segments || format('id = ANY (%L) ', to_text_array(j->'ids'));\n END IF;\n\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n where_segments := where_segments || format('collection = ANY (%L) ', collections);\n END IF;\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n\n where_segments := where_segments || format(' datetime <= %L::timestamptz AND end_datetime >= %L::timestamptz ',\n edate,\n sdate\n );\n END IF;\n\n geom := stac_geom(j);\n IF geom IS NOT NULL THEN\n where_segments := where_segments || format('st_intersects(geometry, %L)',geom);\n END IF;\n\n filterlang := COALESCE(\n j->>'filter-lang',\n get_setting('default-filter-lang', j->'conf')\n );\n IF NOT filter @? '$.**.op' THEN\n filterlang := 'cql-json';\n END IF;\n\n IF filterlang NOT IN ('cql-json','cql2-json') AND j ? 'filter' THEN\n RAISE EXCEPTION '% is not a supported filter-lang. Please use cql-json or cql2-json.', filterlang;\n END IF;\n\n IF j ? 'query' AND j ? 'filter' THEN\n RAISE EXCEPTION 'Can only use either query or filter at one time.';\n END IF;\n\n IF j ? 'query' THEN\n filter := query_to_cql2(j->'query');\n ELSIF filterlang = 'cql-json' THEN\n filter := cql1_to_cql2(filter);\n END IF;\n RAISE NOTICE 'FILTER: %', filter;\n where_segments := where_segments || cql2_query(filter);\n IF cardinality(where_segments) < 1 THEN\n RETURN ' TRUE ';\n END IF;\n\n _where := array_to_string(array_remove(where_segments, NULL), ' AND ');\n\n IF _where IS NULL OR BTRIM(_where) = '' THEN\n RETURN ' TRUE ';\n END IF;\n RETURN _where;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION parse_sort_dir(_dir text, reverse boolean default false) RETURNS text AS $$\n WITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n ) SELECT\n CASE\n WHEN NOT reverse THEN d\n WHEN d = 'ASC' THEN 'DESC'\n WHEN d = 'DESC' THEN 'ASC'\n END\n FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION sort_dir_to_op(_dir text, prev boolean default false) RETURNS text AS $$\n WITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n ) SELECT\n CASE\n WHEN d = 'ASC' AND prev THEN '<='\n WHEN d = 'DESC' AND prev THEN '>='\n WHEN d = 'ASC' THEN '>='\n WHEN d = 'DESC' THEN '<='\n END\n FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION sort_sqlorderby(\n _search jsonb DEFAULT NULL,\n reverse boolean DEFAULT FALSE\n) RETURNS text AS $$\n WITH sortby AS (\n SELECT coalesce(_search->'sortby','[{\"field\":\"datetime\", \"direction\":\"desc\"}]') as sort\n ), withid AS (\n SELECT CASE\n WHEN sort @? '$[*] ? (@.field == \"id\")' THEN sort\n ELSE sort || '[{\"field\":\"id\", \"direction\":\"desc\"}]'::jsonb\n END as sort\n FROM sortby\n ), withid_rows AS (\n SELECT jsonb_array_elements(sort) as value FROM withid\n ),sorts AS (\n SELECT\n coalesce(\n -- field_orderby((items_path(value->>'field')).path_txt),\n (queryable(value->>'field')).expression\n ) as key,\n parse_sort_dir(value->>'direction', reverse) as dir\n FROM withid_rows\n )\n SELECT array_to_string(\n array_agg(concat(key, ' ', dir)),\n ', '\n ) FROM sorts;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_sort_dir(sort_item jsonb) RETURNS text AS $$\n SELECT CASE WHEN sort_item->>'direction' ILIKE 'desc%' THEN 'DESC' ELSE 'ASC' END;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION get_token_filter(_search jsonb = '{}'::jsonb, token_rec jsonb DEFAULT NULL) RETURNS text AS $$\nDECLARE\n token_id text;\n filters text[] := '{}'::text[];\n prev boolean := TRUE;\n field text;\n dir text;\n sort record;\n orfilters text[] := '{}'::text[];\n andfilters text[] := '{}'::text[];\n output text;\n token_where text;\nBEGIN\n RAISE NOTICE 'Getting Token Filter. % %', _search, token_rec;\n -- If no token provided return NULL\n IF token_rec IS NULL THEN\n IF NOT (_search ? 'token' AND\n (\n (_search->>'token' ILIKE 'prev:%')\n OR\n (_search->>'token' ILIKE 'next:%')\n )\n ) THEN\n RETURN NULL;\n END IF;\n prev := (_search->>'token' ILIKE 'prev:%');\n token_id := substr(_search->>'token', 6);\n SELECT to_jsonb(items) INTO token_rec\n FROM items WHERE id=token_id;\n END IF;\n RAISE NOTICE 'TOKEN ID: % %', token_rec, token_rec->'id';\n\n CREATE TEMP TABLE sorts (\n _row int GENERATED ALWAYS AS IDENTITY NOT NULL,\n _field text PRIMARY KEY,\n _dir text NOT NULL,\n _val text\n ) ON COMMIT DROP;\n\n -- Make sure we only have distinct columns to sort with taking the first one we get\n INSERT INTO sorts (_field, _dir)\n SELECT\n (queryable(value->>'field')).expression,\n get_sort_dir(value)\n FROM\n jsonb_array_elements(coalesce(_search->'sortby','[{\"field\":\"datetime\",\"direction\":\"desc\"}]'))\n ON CONFLICT DO NOTHING\n ;\n RAISE NOTICE 'sorts 1: %', (SELECT jsonb_agg(to_json(sorts)) FROM sorts);\n -- Get the first sort direction provided. As the id is a primary key, if there are any\n -- sorts after id they won't do anything, so make sure that id is the last sort item.\n SELECT _dir INTO dir FROM sorts ORDER BY _row ASC LIMIT 1;\n IF EXISTS (SELECT 1 FROM sorts WHERE _field = 'id') THEN\n DELETE FROM sorts WHERE _row > (SELECT _row FROM sorts WHERE _field = 'id' ORDER BY _row ASC);\n ELSE\n INSERT INTO sorts (_field, _dir) VALUES ('id', dir);\n END IF;\n\n -- Add value from looked up item to the sorts table\n UPDATE sorts SET _val=quote_literal(token_rec->>_field);\n\n -- Check if all sorts are the same direction and use row comparison\n -- to filter\n RAISE NOTICE 'sorts 2: %', (SELECT jsonb_agg(to_json(sorts)) FROM sorts);\n\n IF (SELECT count(DISTINCT _dir) FROM sorts) = 1 THEN\n SELECT format(\n '(%s) %s (%s)',\n concat_ws(', ', VARIADIC array_agg(quote_ident(_field))),\n CASE WHEN (prev AND dir = 'ASC') OR (NOT prev AND dir = 'DESC') THEN '<' ELSE '>' END,\n concat_ws(', ', VARIADIC array_agg(_val))\n ) INTO output FROM sorts\n WHERE token_rec ? _field\n ;\n ELSE\n FOR sort IN SELECT * FROM sorts ORDER BY _row asc LOOP\n RAISE NOTICE 'SORT: %', sort;\n IF sort._row = 1 THEN\n orfilters := orfilters || format('(%s %s %s)',\n quote_ident(sort._field),\n CASE WHEN (prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC') THEN '<' ELSE '>' END,\n sort._val\n );\n ELSE\n orfilters := orfilters || format('(%s AND %s %s %s)',\n array_to_string(andfilters, ' AND '),\n quote_ident(sort._field),\n CASE WHEN (prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC') THEN '<' ELSE '>' END,\n sort._val\n );\n\n END IF;\n andfilters := andfilters || format('%s = %s',\n quote_ident(sort._field),\n sort._val\n );\n END LOOP;\n output := array_to_string(orfilters, ' OR ');\n END IF;\n DROP TABLE IF EXISTS sorts;\n token_where := concat('(',coalesce(output,'true'),')');\n IF trim(token_where) = '' THEN\n token_where := NULL;\n END IF;\n RAISE NOTICE 'TOKEN_WHERE: |%|',token_where;\n RETURN token_where;\n END;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION search_tohash(jsonb) RETURNS jsonb AS $$\n SELECT $1 - '{token,limit,context,includes,excludes}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION search_hash(jsonb, jsonb) RETURNS text AS $$\n SELECT md5(concat(search_tohash($1)::text,$2::text));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TABLE IF NOT EXISTS searches(\n hash text GENERATED ALWAYS AS (search_hash(search, metadata)) STORED PRIMARY KEY,\n search jsonb NOT NULL,\n _where text,\n orderby text,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n metadata jsonb DEFAULT '{}'::jsonb NOT NULL\n);\nCREATE TABLE IF NOT EXISTS search_wheres(\n id bigint generated always as identity primary key,\n _where text NOT NULL,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n statslastupdated timestamptz,\n estimated_count bigint,\n estimated_cost float,\n time_to_estimate float,\n total_count bigint,\n time_to_count float,\n partitions text[]\n);\n\nCREATE INDEX IF NOT EXISTS search_wheres_partitions ON search_wheres USING GIN (partitions);\nCREATE UNIQUE INDEX IF NOT EXISTS search_wheres_where ON search_wheres ((md5(_where)));\n\nCREATE OR REPLACE FUNCTION where_stats(inwhere text, updatestats boolean default false, conf jsonb default null) RETURNS search_wheres AS $$\nDECLARE\n t timestamptz;\n i interval;\n explain_json jsonb;\n partitions text[];\n sw search_wheres%ROWTYPE;\n inwhere_hash text := md5(inwhere);\n _context text := lower(context(conf));\n _stats_ttl interval := context_stats_ttl(conf);\n _estimated_cost float := context_estimated_cost(conf);\n _estimated_count int := context_estimated_count(conf);\nBEGIN\n IF _context = 'off' THEN\n sw._where := inwhere;\n return sw;\n END IF;\n\n SELECT * INTO sw FROM search_wheres WHERE md5(_where)=inwhere_hash FOR UPDATE;\n\n -- Update statistics if explicitly set, if statistics do not exist, or statistics ttl has expired\n IF NOT updatestats THEN\n RAISE NOTICE 'Checking if update is needed for: % .', inwhere;\n RAISE NOTICE 'Stats Last Updated: %', sw.statslastupdated;\n RAISE NOTICE 'TTL: %, Age: %', _stats_ttl, now() - sw.statslastupdated;\n RAISE NOTICE 'Context: %, Existing Total: %', _context, sw.total_count;\n IF\n sw.statslastupdated IS NULL\n OR (now() - sw.statslastupdated) > _stats_ttl\n OR (context(conf) != 'off' AND sw.total_count IS NULL)\n THEN\n updatestats := TRUE;\n END IF;\n END IF;\n\n sw._where := inwhere;\n sw.lastused := now();\n sw.usecount := coalesce(sw.usecount,0) + 1;\n\n IF NOT updatestats THEN\n UPDATE search_wheres SET\n lastused = sw.lastused,\n usecount = sw.usecount\n WHERE md5(_where) = inwhere_hash\n RETURNING * INTO sw\n ;\n RETURN sw;\n END IF;\n\n -- Use explain to get estimated count/cost and a list of the partitions that would be hit by the query\n t := clock_timestamp();\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s', inwhere)\n INTO explain_json;\n RAISE NOTICE 'Time for just the explain: %', clock_timestamp() - t;\n i := clock_timestamp() - t;\n\n sw.statslastupdated := now();\n sw.estimated_count := explain_json->0->'Plan'->'Plan Rows';\n sw.estimated_cost := explain_json->0->'Plan'->'Total Cost';\n sw.time_to_estimate := extract(epoch from i);\n\n RAISE NOTICE 'ESTIMATED_COUNT: % < %', sw.estimated_count, _estimated_count;\n RAISE NOTICE 'ESTIMATED_COST: % < %', sw.estimated_cost, _estimated_cost;\n\n -- Do a full count of rows if context is set to on or if auto is set and estimates are low enough\n IF\n _context = 'on'\n OR\n ( _context = 'auto' AND\n (\n sw.estimated_count < _estimated_count\n AND\n sw.estimated_cost < _estimated_cost\n )\n )\n THEN\n t := clock_timestamp();\n RAISE NOTICE 'Calculating actual count...';\n EXECUTE format(\n 'SELECT count(*) FROM items WHERE %s',\n inwhere\n ) INTO sw.total_count;\n i := clock_timestamp() - t;\n RAISE NOTICE 'Actual Count: % -- %', sw.total_count, i;\n sw.time_to_count := extract(epoch FROM i);\n ELSE\n sw.total_count := NULL;\n sw.time_to_count := NULL;\n END IF;\n\n\n INSERT INTO search_wheres\n (_where, lastused, usecount, statslastupdated, estimated_count, estimated_cost, time_to_estimate, partitions, total_count, time_to_count)\n SELECT sw._where, sw.lastused, sw.usecount, sw.statslastupdated, sw.estimated_count, sw.estimated_cost, sw.time_to_estimate, sw.partitions, sw.total_count, sw.time_to_count\n ON CONFLICT ((md5(_where)))\n DO UPDATE\n SET\n lastused = sw.lastused,\n usecount = sw.usecount,\n statslastupdated = sw.statslastupdated,\n estimated_count = sw.estimated_count,\n estimated_cost = sw.estimated_cost,\n time_to_estimate = sw.time_to_estimate,\n total_count = sw.total_count,\n time_to_count = sw.time_to_count\n ;\n RETURN sw;\nEND;\n$$ LANGUAGE PLPGSQL ;\n\n\n\nDROP FUNCTION IF EXISTS search_query;\nCREATE OR REPLACE FUNCTION search_query(\n _search jsonb = '{}'::jsonb,\n updatestats boolean = false,\n _metadata jsonb = '{}'::jsonb\n) RETURNS searches AS $$\nDECLARE\n search searches%ROWTYPE;\n pexplain jsonb;\n t timestamptz;\n i interval;\nBEGIN\n SELECT * INTO search FROM searches\n WHERE hash=search_hash(_search, _metadata) FOR UPDATE;\n\n -- Calculate the where clause if not already calculated\n IF search._where IS NULL THEN\n search._where := stac_search_to_where(_search);\n END IF;\n\n -- Calculate the order by clause if not already calculated\n IF search.orderby IS NULL THEN\n search.orderby := sort_sqlorderby(_search);\n END IF;\n\n PERFORM where_stats(search._where, updatestats, _search->'conf');\n\n search.lastused := now();\n search.usecount := coalesce(search.usecount, 0) + 1;\n INSERT INTO searches (search, _where, orderby, lastused, usecount, metadata)\n VALUES (_search, search._where, search.orderby, search.lastused, search.usecount, _metadata)\n ON CONFLICT (hash) DO\n UPDATE SET\n _where = EXCLUDED._where,\n orderby = EXCLUDED.orderby,\n lastused = EXCLUDED.lastused,\n usecount = EXCLUDED.usecount,\n metadata = EXCLUDED.metadata\n RETURNING * INTO search\n ;\n RETURN search;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION search(_search jsonb = '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n searches searches%ROWTYPE;\n _where text;\n token_where text;\n full_where text;\n orderby text;\n query text;\n token_type text := substr(_search->>'token',1,4);\n _limit int := coalesce((_search->>'limit')::int, 10);\n curs refcursor;\n cntr int := 0;\n iter_record items%ROWTYPE;\n first_record jsonb;\n first_item items%ROWTYPE;\n last_item items%ROWTYPE;\n last_record jsonb;\n out_records jsonb := '[]'::jsonb;\n prev_query text;\n next text;\n prev_id text;\n has_next boolean := false;\n has_prev boolean := false;\n prev text;\n total_count bigint;\n context jsonb;\n collection jsonb;\n includes text[];\n excludes text[];\n exit_flag boolean := FALSE;\n batches int := 0;\n timer timestamptz := clock_timestamp();\n pstart timestamptz;\n pend timestamptz;\n pcurs refcursor;\n search_where search_wheres%ROWTYPE;\n id text;\nBEGIN\nCREATE TEMP TABLE results (content jsonb) ON COMMIT DROP;\n-- if ids is set, short circuit and just use direct ids query for each id\n-- skip any paging or caching\n-- hard codes ordering in the same order as the array of ids\nIF _search ? 'ids' THEN\n INSERT INTO results\n SELECT\n CASE WHEN _search->'conf'->>'nohydrate' IS NOT NULL AND (_search->'conf'->>'nohydrate')::boolean = true THEN\n content_nonhydrated(items, _search->'fields')\n ELSE\n content_hydrate(items, _search->'fields')\n END\n FROM items WHERE\n items.id = ANY(to_text_array(_search->'ids'))\n AND\n CASE WHEN _search ? 'collections' THEN\n items.collection = ANY(to_text_array(_search->'collections'))\n ELSE TRUE\n END\n ORDER BY items.datetime desc, items.id desc\n ;\n SELECT INTO total_count count(*) FROM results;\nELSE\n searches := search_query(_search);\n _where := searches._where;\n orderby := searches.orderby;\n search_where := where_stats(_where);\n total_count := coalesce(search_where.total_count, search_where.estimated_count);\n\n IF token_type='prev' THEN\n token_where := get_token_filter(_search, null::jsonb);\n orderby := sort_sqlorderby(_search, TRUE);\n END IF;\n IF token_type='next' THEN\n token_where := get_token_filter(_search, null::jsonb);\n END IF;\n\n full_where := concat_ws(' AND ', _where, token_where);\n RAISE NOTICE 'FULL QUERY % %', full_where, clock_timestamp()-timer;\n timer := clock_timestamp();\n\n FOR query IN SELECT partition_queries(full_where, orderby, search_where.partitions) LOOP\n timer := clock_timestamp();\n query := format('%s LIMIT %s', query, _limit + 1);\n RAISE NOTICE 'Partition Query: %', query;\n batches := batches + 1;\n -- curs = create_cursor(query);\n OPEN curs FOR EXECUTE query;\n LOOP\n FETCH curs into iter_record;\n EXIT WHEN NOT FOUND;\n cntr := cntr + 1;\n\n IF _search->'conf'->>'nohydrate' IS NOT NULL AND (_search->'conf'->>'nohydrate')::boolean = true THEN\n last_record := content_nonhydrated(iter_record, _search->'fields');\n ELSE\n last_record := content_hydrate(iter_record, _search->'fields');\n END IF;\n last_item := iter_record;\n IF cntr = 1 THEN\n first_item := last_item;\n first_record := last_record;\n END IF;\n IF cntr <= _limit THEN\n INSERT INTO results (content) VALUES (last_record);\n ELSIF cntr > _limit THEN\n has_next := true;\n exit_flag := true;\n EXIT;\n END IF;\n END LOOP;\n CLOSE curs;\n RAISE NOTICE 'Query took %.', clock_timestamp()-timer;\n timer := clock_timestamp();\n EXIT WHEN exit_flag;\n END LOOP;\n RAISE NOTICE 'Scanned through % partitions.', batches;\nEND IF;\n\nSELECT jsonb_agg(content) INTO out_records FROM results WHERE content is not NULL;\n\nDROP TABLE results;\n\n\n-- Flip things around if this was the result of a prev token query\nIF token_type='prev' THEN\n out_records := flip_jsonb_array(out_records);\n first_record := last_record;\nEND IF;\n\n-- If this query has a token, see if there is data before the first record\nIF _search ? 'token' THEN\n prev_query := format(\n 'SELECT 1 FROM items WHERE %s LIMIT 1',\n concat_ws(\n ' AND ',\n _where,\n trim(get_token_filter(_search, to_jsonb(first_item)))\n )\n );\n RAISE NOTICE 'Query to get previous record: % --- %', prev_query, first_record;\n EXECUTE prev_query INTO has_prev;\n IF FOUND and has_prev IS NOT NULL THEN\n RAISE NOTICE 'Query results from prev query: %', has_prev;\n has_prev := TRUE;\n END IF;\nEND IF;\nhas_prev := COALESCE(has_prev, FALSE);\n\nIF has_prev THEN\n prev := out_records->0->>'id';\nEND IF;\nIF has_next OR token_type='prev' THEN\n next := out_records->-1->>'id';\nEND IF;\n\nIF context(_search->'conf') != 'off' THEN\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'matched', total_count,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\nELSE\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\nEND IF;\n\ncollection := jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb),\n 'next', next,\n 'prev', prev,\n 'context', context\n);\n\nRETURN collection;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER SET SEARCH_PATH TO pgstac, public;\n\n\nCREATE OR REPLACE FUNCTION search_cursor(_search jsonb = '{}'::jsonb) RETURNS refcursor AS $$\nDECLARE\n curs refcursor;\n searches searches%ROWTYPE;\n _where text;\n _orderby text;\n q text;\n\nBEGIN\n searches := search_query(_search);\n _where := searches._where;\n _orderby := searches.orderby;\n\n OPEN curs FOR\n WITH p AS (\n SELECT * FROM partition_queries(_where, _orderby) p\n )\n SELECT\n CASE WHEN EXISTS (SELECT 1 FROM p) THEN\n (SELECT format($q$\n SELECT * FROM (\n %s\n ) total\n $q$,\n string_agg(\n format($q$ SELECT * FROM ( %s ) AS sub $q$, p),\n '\n UNION ALL\n '\n )\n ))\n ELSE NULL\n END FROM p;\n RETURN curs;\nEND;\n$$ LANGUAGE PLPGSQL;\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION tileenvelope(zoom int, x int, y int) RETURNS geometry AS $$\nWITH t AS (\n SELECT\n 20037508.3427892 as merc_max,\n -20037508.3427892 as merc_min,\n (2 * 20037508.3427892) / (2 ^ zoom) as tile_size\n)\nSELECT st_makeenvelope(\n merc_min + (tile_size * x),\n merc_max - (tile_size * (y + 1)),\n merc_min + (tile_size * (x + 1)),\n merc_max - (tile_size * y),\n 3857\n) FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;DROP FUNCTION IF EXISTS mercgrid;\n\n\nCREATE OR REPLACE FUNCTION ftime() RETURNS interval as $$\nSELECT age(clock_timestamp(), transaction_timestamp());\n$$ LANGUAGE SQL;\nSET SEARCH_PATH to pgstac, public;\n\nDROP FUNCTION IF EXISTS geometrysearch;\nCREATE OR REPLACE FUNCTION geometrysearch(\n IN geom geometry,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE, -- Return as soon as the passed in geometry is full covered\n IN skipcovered boolean DEFAULT TRUE -- Skip any items that would show up completely under the previous items\n) RETURNS jsonb AS $$\nDECLARE\n search searches%ROWTYPE;\n curs refcursor;\n _where text;\n query text;\n iter_record items%ROWTYPE;\n out_records jsonb := '{}'::jsonb[];\n exit_flag boolean := FALSE;\n counter int := 1;\n scancounter int := 1;\n remaining_limit int := _scanlimit;\n tilearea float;\n unionedgeom geometry;\n clippedgeom geometry;\n unionedgeom_area float := 0;\n prev_area float := 0;\n excludes text[];\n includes text[];\n\nBEGIN\n DROP TABLE IF EXISTS pgstac_results;\n CREATE TEMP TABLE pgstac_results (content jsonb) ON COMMIT DROP;\n\n -- If skipcovered is true then you will always want to exit when the passed in geometry is full\n IF skipcovered THEN\n exitwhenfull := TRUE;\n END IF;\n\n SELECT * INTO search FROM searches WHERE hash=queryhash;\n\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Search with Query Hash % Not Found', queryhash;\n END IF;\n\n tilearea := st_area(geom);\n _where := format('%s AND st_intersects(geometry, %L::geometry)', search._where, geom);\n\n\n FOR query IN SELECT * FROM partition_queries(_where, search.orderby) LOOP\n query := format('%s LIMIT %L', query, remaining_limit);\n RAISE NOTICE '%', query;\n OPEN curs FOR EXECUTE query;\n LOOP\n FETCH curs INTO iter_record;\n EXIT WHEN NOT FOUND;\n IF exitwhenfull OR skipcovered THEN -- If we are not using exitwhenfull or skipcovered, we do not need to do expensive geometry operations\n clippedgeom := st_intersection(geom, iter_record.geometry);\n\n IF unionedgeom IS NULL THEN\n unionedgeom := clippedgeom;\n ELSE\n unionedgeom := st_union(unionedgeom, clippedgeom);\n END IF;\n\n unionedgeom_area := st_area(unionedgeom);\n\n IF skipcovered AND prev_area = unionedgeom_area THEN\n scancounter := scancounter + 1;\n CONTINUE;\n END IF;\n\n prev_area := unionedgeom_area;\n\n RAISE NOTICE '% % % %', unionedgeom_area/tilearea, counter, scancounter, ftime();\n END IF;\n RAISE NOTICE '% %', iter_record, content_hydrate(iter_record, fields);\n INSERT INTO pgstac_results (content) VALUES (content_hydrate(iter_record, fields));\n\n IF counter >= _limit\n OR scancounter > _scanlimit\n OR ftime() > _timelimit\n OR (exitwhenfull AND unionedgeom_area >= tilearea)\n THEN\n exit_flag := TRUE;\n EXIT;\n END IF;\n counter := counter + 1;\n scancounter := scancounter + 1;\n\n END LOOP;\n CLOSE curs;\n EXIT WHEN exit_flag;\n remaining_limit := _scanlimit - scancounter;\n END LOOP;\n\n SELECT jsonb_agg(content) INTO out_records FROM pgstac_results WHERE content IS NOT NULL;\n\n RETURN jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb)\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDROP FUNCTION IF EXISTS geojsonsearch;\nCREATE OR REPLACE FUNCTION geojsonsearch(\n IN geojson jsonb,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_geomfromgeojson(geojson),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS xyzsearch;\nCREATE OR REPLACE FUNCTION xyzsearch(\n IN _x int,\n IN _y int,\n IN _z int,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_transform(tileenvelope(_z, _x, _y), 4326),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nGRANT ALL ON SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac to pgstac_admin;\n\n-- pgstac_read role limited to using function apis\nGRANT EXECUTE ON FUNCTION search TO pgstac_read;\nGRANT EXECUTE ON FUNCTION search_query TO pgstac_read;\nGRANT EXECUTE ON FUNCTION item_by_id TO pgstac_read;\nGRANT EXECUTE ON FUNCTION get_item TO pgstac_read;\n\nGRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_ingest;\nGRANT USAGE ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_ingest;\nSELECT set_version('0.6.2');\n" }, { "path": "src/pgstac/migrations/pgstac.0.6.3-0.6.4.sql", "content": "SET SEARCH_PATH to pgstac, public;\nset check_function_bodies = off;\n\nINSERT INTO cql2_ops (op, template, types) VALUES\n ('eq', '%s = %s', NULL),\n ('lt', '%s < %s', NULL),\n ('lte', '%s <= %s', NULL),\n ('gt', '%s > %s', NULL),\n ('gte', '%s >= %s', NULL),\n ('le', '%s <= %s', NULL),\n ('ge', '%s >= %s', NULL),\n ('=', '%s = %s', NULL),\n ('<', '%s < %s', NULL),\n ('<=', '%s <= %s', NULL),\n ('>', '%s > %s', NULL),\n ('>=', '%s >= %s', NULL),\n ('like', '%s LIKE %s', NULL),\n ('ilike', '%s ILIKE %s', NULL),\n ('+', '%s + %s', NULL),\n ('-', '%s - %s', NULL),\n ('*', '%s * %s', NULL),\n ('/', '%s / %s', NULL),\n ('in', '%s = ANY (%s)', NULL),\n ('not', 'NOT (%s)', NULL),\n ('between', '%s BETWEEN %s AND %s', NULL),\n ('isnull', '%s IS NULL', NULL),\n ('upper', 'upper(%s)', NULL),\n ('lower', 'lower(%s)', NULL)\nON CONFLICT (op) DO UPDATE\n SET\n template = EXCLUDED.template\n;\n\n\nCREATE OR REPLACE FUNCTION pgstac.cql2_query(j jsonb, wrapper text DEFAULT NULL::text)\n RETURNS text\n LANGUAGE plpgsql\n STABLE\nAS $function$\n#variable_conflict use_variable\nDECLARE\n args jsonb := j->'args';\n arg jsonb;\n op text := lower(j->>'op');\n cql2op RECORD;\n literal text;\n _wrapper text;\nBEGIN\n IF j IS NULL OR (op IS NOT NULL AND args IS NULL) THEN\n RETURN NULL;\n END IF;\n RAISE NOTICE 'CQL2_QUERY: %', j;\n IF j ? 'filter' THEN\n RETURN cql2_query(j->'filter');\n END IF;\n\n IF j ? 'upper' THEN\n RETURN cql2_query(jsonb_build_object('op', 'upper', 'args', j->'upper'));\n END IF;\n\n IF j ? 'lower' THEN\n RETURN cql2_query(jsonb_build_object('op', 'lower', 'args', j->'lower'));\n END IF;\n\n -- Temporal Query\n IF op ilike 't_%' or op = 'anyinteracts' THEN\n RETURN temporal_op_query(op, args);\n END IF;\n\n -- If property is a timestamp convert it to text to use with\n -- general operators\n IF j ? 'timestamp' THEN\n RETURN format('%L::timestamptz', to_tstz(j->'timestamp'));\n END IF;\n IF j ? 'interval' THEN\n RAISE EXCEPTION 'Please use temporal operators when using intervals.';\n RETURN NONE;\n END IF;\n\n -- Spatial Query\n IF op ilike 's_%' or op = 'intersects' THEN\n RETURN spatial_op_query(op, args);\n END IF;\n\n\n IF op = 'in' THEN\n RETURN format(\n '%s = ANY (%L)',\n cql2_query(args->0),\n to_text_array(args->1)\n );\n END IF;\n\n\n\n IF op = 'between' THEN\n args = jsonb_build_array(\n args->0,\n args->1->0,\n args->1->1\n );\n END IF;\n\n -- Make sure that args is an array and run cql2_query on\n -- each element of the array\n RAISE NOTICE 'ARGS PRE: %', args;\n IF j ? 'args' THEN\n IF jsonb_typeof(args) != 'array' THEN\n args := jsonb_build_array(args);\n END IF;\n\n IF jsonb_path_exists(args, '$[*] ? (@.property == \"id\" || @.property == \"datetime\" || @.property == \"end_datetime\" || @.property == \"collection\")') THEN\n wrapper := NULL;\n ELSE\n -- if any of the arguments are a property, try to get the property_wrapper\n FOR arg IN SELECT jsonb_path_query(args, '$[*] ? (@.property != null)') LOOP\n RAISE NOTICE 'Arg: %', arg;\n SELECT property_wrapper INTO wrapper\n FROM queryables\n WHERE name=(arg->>'property')\n LIMIT 1;\n RAISE NOTICE 'Property: %, Wrapper: %', arg, wrapper;\n IF wrapper IS NOT NULL THEN\n EXIT;\n END IF;\n END LOOP;\n\n -- if the property was not in queryables, see if any args were numbers\n IF\n wrapper IS NULL\n AND jsonb_path_exists(args, '$[*] ? (@.type()==\"number\")')\n THEN\n wrapper := 'to_float';\n END IF;\n wrapper := coalesce(wrapper, 'to_text');\n END IF;\n\n SELECT jsonb_agg(cql2_query(a, wrapper))\n INTO args\n FROM jsonb_array_elements(args) a;\n END IF;\n RAISE NOTICE 'ARGS: %', args;\n\n IF op IN ('and', 'or') THEN\n RETURN\n format(\n '(%s)',\n array_to_string(to_text_array(args), format(' %s ', upper(op)))\n );\n END IF;\n\n -- Look up template from cql2_ops\n IF j ? 'op' THEN\n SELECT * INTO cql2op FROM cql2_ops WHERE cql2_ops.op ilike op;\n IF FOUND THEN\n -- If specific index set in queryables for a property cast other arguments to that type\n RETURN format(\n cql2op.template,\n VARIADIC (to_text_array(args))\n );\n ELSE\n RAISE EXCEPTION 'Operator % Not Supported.', op;\n END IF;\n END IF;\n\n\n IF wrapper IS NOT NULL THEN\n RAISE NOTICE 'Wrapping % with %', j, wrapper;\n IF j ? 'property' THEN\n RETURN format('%I(%s)', wrapper, (queryable(j->>'property')).path);\n ELSE\n RETURN format('%I(%L)', wrapper, j);\n END IF;\n ELSIF j ? 'property' THEN\n RETURN quote_ident(j->>'property');\n END IF;\n\n RETURN quote_literal(to_text(j));\nEND;\n$function$\n;\n\n\n\nSELECT set_version('0.6.4');\n" }, { "path": "src/pgstac/migrations/pgstac.0.6.3.sql", "content": "CREATE EXTENSION IF NOT EXISTS postgis;\nCREATE EXTENSION IF NOT EXISTS btree_gist;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_admin;\n CREATE ROLE pgstac_read;\n CREATE ROLE pgstac_ingest;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nGRANT pgstac_admin TO current_user;\n\nCREATE SCHEMA IF NOT EXISTS pgstac AUTHORIZATION pgstac_admin;\n\nALTER ROLE pgstac_admin SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_read SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_ingest SET SEARCH_PATH TO pgstac, public;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\n\nGRANT pgstac_read TO pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\n\nSET ROLE pgstac_admin;\n\nSET SEARCH_PATH TO pgstac, public;\n\n\nCREATE TABLE IF NOT EXISTS migrations (\n version text PRIMARY KEY,\n datetime timestamptz DEFAULT clock_timestamp() NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION get_version() RETURNS text AS $$\n SELECT version FROM pgstac.migrations ORDER BY datetime DESC, version DESC LIMIT 1;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION set_version(text) RETURNS text AS $$\n INSERT INTO pgstac.migrations (version) VALUES ($1)\n ON CONFLICT DO NOTHING\n RETURNING version;\n$$ LANGUAGE SQL;\n\n\nCREATE TABLE IF NOT EXISTS pgstac_settings (\n name text PRIMARY KEY,\n value text NOT NULL\n);\n\nINSERT INTO pgstac_settings (name, value) VALUES\n ('context', 'off'),\n ('context_estimated_count', '100000'),\n ('context_estimated_cost', '100000'),\n ('context_stats_ttl', '1 day'),\n ('default-filter-lang', 'cql2-json'),\n ('additional_properties', 'true')\nON CONFLICT DO NOTHING\n;\n\n\nCREATE OR REPLACE FUNCTION get_setting(IN _setting text, IN conf jsonb DEFAULT NULL) RETURNS text AS $$\nSELECT COALESCE(\n conf->>_setting,\n current_setting(concat('pgstac.',_setting), TRUE),\n (SELECT value FROM pgstac.pgstac_settings WHERE name=_setting)\n);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context(conf jsonb DEFAULT NULL) RETURNS text AS $$\n SELECT pgstac.get_setting('context', conf);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context_estimated_count(conf jsonb DEFAULT NULL) RETURNS int AS $$\n SELECT pgstac.get_setting('context_estimated_count', conf)::int;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_estimated_cost();\nCREATE OR REPLACE FUNCTION context_estimated_cost(conf jsonb DEFAULT NULL) RETURNS float AS $$\n SELECT pgstac.get_setting('context_estimated_cost', conf)::float;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_stats_ttl();\nCREATE OR REPLACE FUNCTION context_stats_ttl(conf jsonb DEFAULT NULL) RETURNS interval AS $$\n SELECT pgstac.get_setting('context_stats_ttl', conf)::interval;\n$$ LANGUAGE SQL;\n\n\nCREATE OR REPLACE FUNCTION notice(VARIADIC text[]) RETURNS boolean AS $$\nDECLARE\ndebug boolean := current_setting('pgstac.debug', true);\nBEGIN\n IF debug THEN\n RAISE NOTICE 'NOTICE FROM FUNC: % >>>>> %', concat_ws(' | ', $1), clock_timestamp();\n RETURN TRUE;\n END IF;\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION empty_arr(ANYARRAY) RETURNS BOOLEAN AS $$\nSELECT CASE\n WHEN $1 IS NULL THEN TRUE\n WHEN cardinality($1)<1 THEN TRUE\nELSE FALSE\nEND;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_intersection(_a ANYARRAY, _b ANYARRAY) RETURNS ANYARRAY AS $$\n SELECT ARRAY ( SELECT unnest(_a) INTERSECT SELECT UNNEST(_b) );\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION array_map_ident(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_ident(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_map_literal(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_literal(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION array_reverse(anyarray) RETURNS anyarray AS $$\nSELECT ARRAY(\n SELECT $1[i]\n FROM generate_subscripts($1,1) AS s(i)\n ORDER BY i DESC\n);\n$$ LANGUAGE SQL STRICT IMMUTABLE;\nCREATE OR REPLACE FUNCTION to_int(jsonb) RETURNS int AS $$\n SELECT floor(($1->>0)::float)::int;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION to_float(jsonb) RETURNS float AS $$\n SELECT ($1->>0)::float;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION to_tstz(jsonb) RETURNS timestamptz AS $$\n SELECT ($1->>0)::timestamptz;\n$$ LANGUAGE SQL IMMUTABLE STRICT SET TIME ZONE 'UTC';\n\n\nCREATE OR REPLACE FUNCTION to_text(jsonb) RETURNS text AS $$\n SELECT CASE WHEN jsonb_typeof($1) IN ('array','object') THEN $1::text ELSE $1->>0 END;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION to_text_array(jsonb) RETURNS text[] AS $$\n SELECT\n CASE jsonb_typeof($1)\n WHEN 'array' THEN ARRAY(SELECT jsonb_array_elements_text($1))\n ELSE ARRAY[$1->>0]\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\nCREATE OR REPLACE FUNCTION bbox_geom(_bbox jsonb) RETURNS geometry AS $$\nSELECT CASE jsonb_array_length(_bbox)\n WHEN 4 THEN\n ST_SetSRID(ST_MakeEnvelope(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float,\n (_bbox->>3)::float\n ),4326)\n WHEN 6 THEN\n ST_SetSRID(ST_3DMakeBox(\n ST_MakePoint(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float\n ),\n ST_MakePoint(\n (_bbox->>3)::float,\n (_bbox->>4)::float,\n (_bbox->>5)::float\n )\n ),4326)\n ELSE null END;\n;\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION geom_bbox(_geom geometry) RETURNS jsonb AS $$\n SELECT jsonb_build_array(\n st_xmin(_geom),\n st_ymin(_geom),\n st_xmax(_geom),\n st_ymax(_geom)\n );\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION flip_jsonb_array(j jsonb) RETURNS jsonb AS $$\n SELECT jsonb_agg(value) FROM (SELECT value FROM jsonb_array_elements(j) WITH ORDINALITY ORDER BY ordinality DESC) as t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION explode_dotpaths(j jsonb) RETURNS SETOF text[] AS $$\n SELECT string_to_array(p, '.') as e FROM jsonb_array_elements_text(j) p;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION explode_dotpaths_recurse(IN j jsonb) RETURNS SETOF text[] AS $$\n WITH RECURSIVE t AS (\n SELECT e FROM explode_dotpaths(j) e\n UNION ALL\n SELECT e[1:cardinality(e)-1]\n FROM t\n WHERE cardinality(e)>1\n ) SELECT e FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION jsonb_set_nested(j jsonb, path text[], val jsonb) RETURNS jsonb AS $$\nDECLARE\nBEGIN\n IF cardinality(path) > 1 THEN\n FOR i IN 1..(cardinality(path)-1) LOOP\n IF j #> path[:i] IS NULL THEN\n j := jsonb_set_lax(j, path[:i], '{}', TRUE);\n END IF;\n END LOOP;\n END IF;\n RETURN jsonb_set_lax(j, path, val, true);\n\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\n\n\nCREATE OR REPLACE FUNCTION jsonb_include(j jsonb, f jsonb) RETURNS jsonb AS $$\nDECLARE\n includes jsonb := f-> 'include';\n outj jsonb := '{}'::jsonb;\n path text[];\nBEGIN\n IF\n includes IS NULL\n OR jsonb_array_length(includes) = 0\n THEN\n RETURN j;\n ELSE\n includes := includes || '[\"id\",\"collection\"]'::jsonb;\n FOR path IN SELECT explode_dotpaths(includes) LOOP\n outj := jsonb_set_nested(outj, path, j #> path);\n END LOOP;\n END IF;\n RETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_exclude(j jsonb, f jsonb) RETURNS jsonb AS $$\nDECLARE\n excludes jsonb := f-> 'exclude';\n outj jsonb := j;\n path text[];\nBEGIN\n IF\n excludes IS NULL\n OR jsonb_array_length(excludes) = 0\n THEN\n RETURN j;\n ELSE\n FOR path IN SELECT explode_dotpaths(excludes) LOOP\n outj := outj #- path;\n END LOOP;\n END IF;\n RETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_fields(j jsonb, f jsonb DEFAULT '{\"fields\":[]}') RETURNS jsonb AS $$\n SELECT jsonb_exclude(jsonb_include(j, f), f);\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION merge_jsonb(_a jsonb, _b jsonb) RETURNS jsonb AS $$\n SELECT\n CASE\n WHEN _a = '\"𒍟※\"'::jsonb THEN NULL\n WHEN _a IS NULL OR jsonb_typeof(_a) = 'null' THEN _b\n WHEN jsonb_typeof(_a) = 'object' AND jsonb_typeof(_b) = 'object' THEN\n (\n SELECT\n jsonb_strip_nulls(\n jsonb_object_agg(\n key,\n merge_jsonb(a.value, b.value)\n )\n )\n FROM\n jsonb_each(coalesce(_a,'{}'::jsonb)) as a\n FULL JOIN\n jsonb_each(coalesce(_b,'{}'::jsonb)) as b\n USING (key)\n )\n WHEN\n jsonb_typeof(_a) = 'array'\n AND jsonb_typeof(_b) = 'array'\n AND jsonb_array_length(_a) = jsonb_array_length(_b)\n THEN\n (\n SELECT jsonb_agg(m) FROM\n ( SELECT\n merge_jsonb(\n jsonb_array_elements(_a),\n jsonb_array_elements(_b)\n ) as m\n ) as l\n )\n ELSE _a\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION strip_jsonb(_a jsonb, _b jsonb) RETURNS jsonb AS $$\n SELECT\n CASE\n\n WHEN (_a IS NULL OR jsonb_typeof(_a) = 'null') AND _b IS NOT NULL AND jsonb_typeof(_b) != 'null' THEN '\"𒍟※\"'::jsonb\n WHEN _b IS NULL OR jsonb_typeof(_a) = 'null' THEN _a\n WHEN _a = _b AND jsonb_typeof(_a) = 'object' THEN '{}'::jsonb\n WHEN _a = _b THEN NULL\n WHEN jsonb_typeof(_a) = 'object' AND jsonb_typeof(_b) = 'object' THEN\n (\n SELECT\n jsonb_strip_nulls(\n jsonb_object_agg(\n key,\n strip_jsonb(a.value, b.value)\n )\n )\n FROM\n jsonb_each(_a) as a\n FULL JOIN\n jsonb_each(_b) as b\n USING (key)\n )\n WHEN\n jsonb_typeof(_a) = 'array'\n AND jsonb_typeof(_b) = 'array'\n AND jsonb_array_length(_a) = jsonb_array_length(_b)\n THEN\n (\n SELECT jsonb_agg(m) FROM\n ( SELECT\n strip_jsonb(\n jsonb_array_elements(_a),\n jsonb_array_elements(_b)\n ) as m\n ) as l\n )\n ELSE _a\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE;\n/* looks for a geometry in a stac item first from geometry and falling back to bbox */\nCREATE OR REPLACE FUNCTION stac_geom(value jsonb) RETURNS geometry AS $$\nSELECT\n CASE\n WHEN value ? 'intersects' THEN\n ST_GeomFromGeoJSON(value->>'intersects')\n WHEN value ? 'geometry' THEN\n ST_GeomFromGeoJSON(value->>'geometry')\n WHEN value ? 'bbox' THEN\n pgstac.bbox_geom(value->'bbox')\n ELSE NULL\n END as geometry\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION stac_daterange(\n value jsonb\n) RETURNS tstzrange AS $$\nDECLARE\n props jsonb := value;\n dt timestamptz;\n edt timestamptz;\nBEGIN\n IF props ? 'properties' THEN\n props := props->'properties';\n END IF;\n IF\n props ? 'start_datetime'\n AND props->>'start_datetime' IS NOT NULL\n AND props ? 'end_datetime'\n AND props->>'end_datetime' IS NOT NULL\n THEN\n dt := props->>'start_datetime';\n edt := props->>'end_datetime';\n IF dt > edt THEN\n RAISE EXCEPTION 'start_datetime must be < end_datetime';\n END IF;\n ELSE\n dt := props->>'datetime';\n edt := props->>'datetime';\n END IF;\n IF dt is NULL OR edt IS NULL THEN\n RAISE NOTICE 'DT: %, EDT: %', dt, edt;\n RAISE EXCEPTION 'Either datetime (%) or both start_datetime (%) and end_datetime (%) must be set.', props->>'datetime',props->>'start_datetime',props->>'end_datetime';\n END IF;\n RETURN tstzrange(dt, edt, '[]');\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_datetime(value jsonb) RETURNS timestamptz AS $$\n SELECT lower(stac_daterange(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_end_datetime(value jsonb) RETURNS timestamptz AS $$\n SELECT upper(stac_daterange(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\n\nCREATE TABLE IF NOT EXISTS stac_extensions(\n name text PRIMARY KEY,\n url text,\n enbabled_by_default boolean NOT NULL DEFAULT TRUE,\n enableable boolean NOT NULL DEFAULT TRUE\n);\n\nINSERT INTO stac_extensions (name, url) VALUES\n ('fields', 'https://api.stacspec.org/v1.0.0-beta.5/item-search#fields'),\n ('sort','https://api.stacspec.org/v1.0.0-beta.5/item-search#sort'),\n ('context','https://api.stacspec.org/v1.0.0-beta.5/item-search#context'),\n ('filter', 'https://api.stacspec.org/v1.0.0-beta.5/item-search#filter'),\n ('query', 'https://api.stacspec.org/v1.0.0-beta.5/item-search#query')\nON CONFLICT (name) DO UPDATE SET url=EXCLUDED.url;\n\n\n\nCREATE OR REPLACE FUNCTION collection_base_item(content jsonb) RETURNS jsonb AS $$\n SELECT jsonb_build_object(\n 'type', 'Feature',\n 'stac_version', content->'stac_version',\n 'assets', content->'item_assets',\n 'collection', content->'id'\n );\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TYPE partition_trunc_strategy AS ENUM ('year', 'month');\n\nCREATE TABLE IF NOT EXISTS collections (\n key bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,\n id text GENERATED ALWAYS AS (content->>'id') STORED UNIQUE,\n content JSONB NOT NULL,\n base_item jsonb GENERATED ALWAYS AS (pgstac.collection_base_item(content)) STORED,\n partition_trunc partition_trunc_strategy\n);\n\n\nCREATE OR REPLACE FUNCTION collection_base_item(cid text) RETURNS jsonb AS $$\n SELECT pgstac.collection_base_item(content) FROM pgstac.collections WHERE id = cid LIMIT 1;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION table_empty(text) RETURNS boolean AS $$\nDECLARE\n retval boolean;\nBEGIN\n EXECUTE format($q$\n SELECT NOT EXISTS (SELECT 1 FROM %I LIMIT 1)\n $q$,\n $1\n ) INTO retval;\n RETURN retval;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION collections_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n q text;\n partition_name text := format('_items_%s', NEW.key);\n partition_exists boolean := false;\n partition_empty boolean := true;\n err_context text;\n loadtemp boolean := FALSE;\nBEGIN\n RAISE NOTICE 'Collection Trigger. % %', NEW.id, NEW.key;\n SELECT relid::text INTO partition_name\n FROM pg_partition_tree('items')\n WHERE relid::text = partition_name;\n IF FOUND THEN\n partition_exists := true;\n partition_empty := table_empty(partition_name);\n ELSE\n partition_exists := false;\n partition_empty := true;\n partition_name := format('_items_%s', NEW.key);\n END IF;\n IF TG_OP = 'UPDATE' AND NEW.partition_trunc IS DISTINCT FROM OLD.partition_trunc AND partition_empty THEN\n q := format($q$\n DROP TABLE IF EXISTS %I CASCADE;\n $q$,\n partition_name\n );\n EXECUTE q;\n END IF;\n IF TG_OP = 'UPDATE' AND NEW.partition_trunc IS DISTINCT FROM OLD.partition_trunc AND partition_exists AND NOT partition_empty THEN\n q := format($q$\n CREATE TEMP TABLE changepartitionstaging ON COMMIT DROP AS SELECT * FROM %I;\n DROP TABLE IF EXISTS %I CASCADE;\n $q$,\n partition_name,\n partition_name\n );\n EXECUTE q;\n loadtemp := TRUE;\n partition_empty := TRUE;\n partition_exists := FALSE;\n END IF;\n IF TG_OP = 'UPDATE' AND NEW.partition_trunc IS NOT DISTINCT FROM OLD.partition_trunc THEN\n RETURN NEW;\n END IF;\n IF NEW.partition_trunc IS NULL AND partition_empty THEN\n RAISE NOTICE '% % % %',\n partition_name,\n NEW.id,\n concat(partition_name,'_id_idx'),\n partition_name\n ;\n q := format($q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n $q$,\n partition_name,\n NEW.id,\n concat(partition_name,'_id_idx'),\n partition_name\n );\n RAISE NOTICE 'q: %', q;\n BEGIN\n EXECUTE q;\n EXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', partition_name;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\n END;\n\n ALTER TABLE partitions DISABLE TRIGGER partitions_delete_trigger;\n DELETE FROM partitions WHERE collection=NEW.id AND name=partition_name;\n ALTER TABLE partitions ENABLE TRIGGER partitions_delete_trigger;\n\n INSERT INTO partitions (collection, name) VALUES (NEW.id, partition_name);\n ELSIF partition_empty THEN\n q := format($q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L)\n PARTITION BY RANGE (datetime);\n $q$,\n partition_name,\n NEW.id\n );\n RAISE NOTICE 'q: %', q;\n BEGIN\n EXECUTE q;\n EXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', partition_name;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\n END;\n ALTER TABLE partitions DISABLE TRIGGER partitions_delete_trigger;\n DELETE FROM partitions WHERE collection=NEW.id AND name=partition_name;\n ALTER TABLE partitions ENABLE TRIGGER partitions_delete_trigger;\n ELSE\n RAISE EXCEPTION 'Cannot modify partition % unless empty', partition_name;\n END IF;\n IF loadtemp THEN\n RAISE NOTICE 'Moving data into new partitions.';\n q := format($q$\n WITH p AS (\n SELECT\n collection,\n datetime as datetime,\n end_datetime as end_datetime,\n (partition_name(\n collection,\n datetime\n )).partition_name as name\n FROM changepartitionstaging\n )\n INSERT INTO partitions (collection, datetime_range, end_datetime_range)\n SELECT\n collection,\n tstzrange(min(datetime), max(datetime), '[]') as datetime_range,\n tstzrange(min(end_datetime), max(end_datetime), '[]') as end_datetime_range\n FROM p\n GROUP BY collection, name\n ON CONFLICT (name) DO UPDATE SET\n datetime_range = EXCLUDED.datetime_range,\n end_datetime_range = EXCLUDED.end_datetime_range\n ;\n INSERT INTO %I SELECT * FROM changepartitionstaging;\n DROP TABLE IF EXISTS changepartitionstaging;\n $q$,\n partition_name\n );\n EXECUTE q;\n END IF;\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE TRIGGER collections_trigger AFTER INSERT OR UPDATE ON collections FOR EACH ROW\nEXECUTE FUNCTION collections_trigger_func();\n\nCREATE OR REPLACE FUNCTION partition_collection(collection text, strategy partition_trunc_strategy) RETURNS text AS $$\n UPDATE collections SET partition_trunc=strategy WHERE id=collection RETURNING partition_trunc;\n$$ LANGUAGE SQL;\n\nCREATE TABLE IF NOT EXISTS partitions (\n collection text REFERENCES collections(id) ON DELETE CASCADE,\n name text PRIMARY KEY,\n partition_range tstzrange NOT NULL DEFAULT tstzrange('-infinity'::timestamptz,'infinity'::timestamptz, '[]'),\n datetime_range tstzrange,\n end_datetime_range tstzrange,\n CONSTRAINT prange EXCLUDE USING GIST (\n collection WITH =,\n partition_range WITH &&\n )\n) WITH (FILLFACTOR=90);\nCREATE INDEX partitions_range_idx ON partitions USING GIST(partition_range);\n\nCREATE OR REPLACE FUNCTION partitions_delete_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n q text;\nBEGIN\n RAISE NOTICE 'Partition Delete Trigger. %', OLD.name;\n EXECUTE format($q$\n DROP TABLE IF EXISTS %I CASCADE;\n $q$,\n OLD.name\n );\n RAISE NOTICE 'Dropped partition.';\n RETURN OLD;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER partitions_delete_trigger BEFORE DELETE ON partitions FOR EACH ROW\nEXECUTE FUNCTION partitions_delete_trigger_func();\n\nCREATE OR REPLACE FUNCTION partition_name(\n IN collection text,\n IN dt timestamptz,\n OUT partition_name text,\n OUT partition_range tstzrange\n) AS $$\nDECLARE\n c RECORD;\n parent_name text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n parent_name := format('_items_%s', c.key);\n\n\n IF c.partition_trunc = 'year' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYY'));\n ELSIF c.partition_trunc = 'month' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYYMM'));\n ELSE\n partition_name := parent_name;\n partition_range := tstzrange('-infinity'::timestamptz, 'infinity'::timestamptz, '[]');\n END IF;\n IF partition_range IS NULL THEN\n partition_range := tstzrange(\n date_trunc(c.partition_trunc::text, dt),\n date_trunc(c.partition_trunc::text, dt) + concat('1 ', c.partition_trunc)::interval\n );\n END IF;\n RETURN;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\n\nCREATE OR REPLACE FUNCTION partitions_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n q text;\n cq text;\n parent_name text;\n partition_trunc text;\n partition_name text := NEW.name;\n partition_exists boolean := false;\n partition_empty boolean := true;\n partition_range tstzrange;\n datetime_range tstzrange;\n end_datetime_range tstzrange;\n err_context text;\n mindt timestamptz := lower(NEW.datetime_range);\n maxdt timestamptz := upper(NEW.datetime_range);\n minedt timestamptz := lower(NEW.end_datetime_range);\n maxedt timestamptz := upper(NEW.end_datetime_range);\n t_mindt timestamptz;\n t_maxdt timestamptz;\n t_minedt timestamptz;\n t_maxedt timestamptz;\nBEGIN\n RAISE NOTICE 'Partitions Trigger. %', NEW;\n datetime_range := NEW.datetime_range;\n end_datetime_range := NEW.end_datetime_range;\n\n SELECT\n format('_items_%s', key),\n c.partition_trunc::text\n INTO\n parent_name,\n partition_trunc\n FROM pgstac.collections c\n WHERE c.id = NEW.collection;\n SELECT (pgstac.partition_name(NEW.collection, mindt)).* INTO partition_name, partition_range;\n NEW.name := partition_name;\n\n IF partition_range IS NULL OR partition_range = 'empty'::tstzrange THEN\n partition_range := tstzrange('-infinity'::timestamptz, 'infinity'::timestamptz, '[]');\n END IF;\n\n NEW.partition_range := partition_range;\n IF TG_OP = 'UPDATE' THEN\n mindt := least(mindt, lower(OLD.datetime_range));\n maxdt := greatest(maxdt, upper(OLD.datetime_range));\n minedt := least(minedt, lower(OLD.end_datetime_range));\n maxedt := greatest(maxedt, upper(OLD.end_datetime_range));\n NEW.datetime_range := tstzrange(mindt, maxdt, '[]');\n NEW.end_datetime_range := tstzrange(minedt, maxedt, '[]');\n END IF;\n IF TG_OP = 'INSERT' THEN\n\n IF partition_range != tstzrange('-infinity'::timestamptz, 'infinity'::timestamptz, '[]') THEN\n\n RAISE NOTICE '% % %', partition_name, parent_name, partition_range;\n q := format($q$\n CREATE TABLE IF NOT EXISTS %I partition OF %I FOR VALUES FROM (%L) TO (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n $q$,\n partition_name,\n parent_name,\n lower(partition_range),\n upper(partition_range),\n format('%s_pkey', partition_name),\n partition_name\n );\n BEGIN\n EXECUTE q;\n EXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', partition_name;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\n END;\n END IF;\n\n END IF;\n\n -- Update constraints\n EXECUTE format($q$\n SELECT\n min(datetime),\n max(datetime),\n min(end_datetime),\n max(end_datetime)\n FROM %I;\n $q$, partition_name)\n INTO t_mindt, t_maxdt, t_minedt, t_maxedt;\n mindt := least(mindt, t_mindt);\n maxdt := greatest(maxdt, t_maxdt);\n minedt := least(mindt, minedt, t_minedt);\n maxedt := greatest(maxdt, maxedt, t_maxedt);\n\n mindt := date_trunc(coalesce(partition_trunc, 'year'), mindt);\n maxdt := date_trunc(coalesce(partition_trunc, 'year'), maxdt - '1 second'::interval) + concat('1 ',coalesce(partition_trunc, 'year'))::interval;\n minedt := date_trunc(coalesce(partition_trunc, 'year'), minedt);\n maxedt := date_trunc(coalesce(partition_trunc, 'year'), maxedt - '1 second'::interval) + concat('1 ',coalesce(partition_trunc, 'year'))::interval;\n\n\n IF mindt IS NOT NULL AND maxdt IS NOT NULL AND minedt IS NOT NULL AND maxedt IS NOT NULL THEN\n NEW.datetime_range := tstzrange(mindt, maxdt, '[]');\n NEW.end_datetime_range := tstzrange(minedt, maxedt, '[]');\n IF\n TG_OP='UPDATE'\n AND OLD.datetime_range @> NEW.datetime_range\n AND OLD.end_datetime_range @> NEW.end_datetime_range\n THEN\n RAISE NOTICE 'Range unchanged, not updating constraints.';\n ELSE\n\n RAISE NOTICE '\n SETTING CONSTRAINTS\n mindt: %, maxdt: %\n minedt: %, maxedt: %\n ', mindt, maxdt, minedt, maxedt;\n IF partition_trunc IS NULL THEN\n cq := format($q$\n ALTER TABLE %7$I\n DROP CONSTRAINT IF EXISTS %1$I,\n DROP CONSTRAINT IF EXISTS %2$I,\n ADD CONSTRAINT %1$I\n CHECK (\n (datetime >= %3$L)\n AND (datetime <= %4$L)\n AND (end_datetime >= %5$L)\n AND (end_datetime <= %6$L)\n ) NOT VALID\n ;\n ALTER TABLE %7$I\n VALIDATE CONSTRAINT %1$I;\n $q$,\n format('%s_dt', partition_name),\n format('%s_edt', partition_name),\n mindt,\n maxdt,\n minedt,\n maxedt,\n partition_name\n );\n ELSE\n cq := format($q$\n ALTER TABLE %5$I\n DROP CONSTRAINT IF EXISTS %1$I,\n DROP CONSTRAINT IF EXISTS %2$I,\n ADD CONSTRAINT %2$I\n CHECK ((end_datetime >= %3$L) AND (end_datetime <= %4$L)) NOT VALID\n ;\n ALTER TABLE %5$I\n VALIDATE CONSTRAINT %2$I;\n $q$,\n format('%s_dt', partition_name),\n format('%s_edt', partition_name),\n minedt,\n maxedt,\n partition_name\n );\n\n END IF;\n RAISE NOTICE 'Altering Constraints. %', cq;\n EXECUTE cq;\n END IF;\n ELSE\n NEW.datetime_range = NULL;\n NEW.end_datetime_range = NULL;\n\n cq := format($q$\n ALTER TABLE %3$I\n DROP CONSTRAINT IF EXISTS %1$I,\n DROP CONSTRAINT IF EXISTS %2$I,\n ADD CONSTRAINT %1$I\n CHECK ((datetime IS NULL AND end_datetime IS NULL)) NOT VALID\n ;\n ALTER TABLE %3$I\n VALIDATE CONSTRAINT %1$I;\n $q$,\n format('%s_dt', partition_name),\n format('%s_edt', partition_name),\n partition_name\n );\n EXECUTE cq;\n END IF;\n\n RETURN NEW;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER partitions_trigger BEFORE INSERT OR UPDATE ON partitions FOR EACH ROW\nEXECUTE FUNCTION partitions_trigger_func();\n\n\nCREATE OR REPLACE FUNCTION create_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection(data jsonb) RETURNS VOID AS $$\nDECLARE\n out collections%ROWTYPE;\nBEGIN\n UPDATE collections SET content=data WHERE id = data->>'id' RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ON CONFLICT (id) DO\n UPDATE\n SET content=EXCLUDED.content\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION delete_collection(_id text) RETURNS VOID AS $$\nDECLARE\n out collections%ROWTYPE;\nBEGIN\n DELETE FROM collections WHERE id = _id RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION get_collection(id text) RETURNS jsonb AS $$\n SELECT content FROM collections\n WHERE id=$1\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION all_collections() RETURNS jsonb AS $$\n SELECT jsonb_agg(content) FROM collections;\n;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\nCREATE TABLE queryables (\n id bigint GENERATED ALWAYS AS identity PRIMARY KEY,\n name text UNIQUE NOT NULL,\n collection_ids text[], -- used to determine what partitions to create indexes on\n definition jsonb,\n property_path text,\n property_wrapper text,\n property_index_type text\n);\nCREATE INDEX queryables_name_idx ON queryables (name);\nCREATE INDEX queryables_property_wrapper_idx ON queryables (property_wrapper);\n\n\nINSERT INTO queryables (name, definition) VALUES\n('id', '{\"title\": \"Item ID\",\"description\": \"Item identifier\",\"$ref\": \"https://schemas.stacspec.org/v1.0.0/item-spec/json-schema/item.json#/definitions/core/allOf/2/properties/id\"}'),\n('datetime','{\"description\": \"Datetime\",\"type\": \"string\",\"title\": \"Acquired\",\"format\": \"date-time\",\"pattern\": \"(\\\\+00:00|Z)$\"}')\nON CONFLICT DO NOTHING;\n\n\n\nINSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n('eo:cloud_cover','{\"$ref\": \"https://stac-extensions.github.io/eo/v1.0.0/schema.json#/definitions/fieldsproperties/eo:cloud_cover\"}','to_int','BTREE')\nON CONFLICT DO NOTHING;\n\nCREATE OR REPLACE FUNCTION array_to_path(arr text[]) RETURNS text AS $$\n SELECT string_agg(\n quote_literal(v),\n '->'\n ) FROM unnest(arr) v;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\n\n\nCREATE OR REPLACE FUNCTION queryable(\n IN dotpath text,\n OUT path text,\n OUT expression text,\n OUT wrapper text\n) AS $$\nDECLARE\n q RECORD;\n path_elements text[];\nBEGIN\n IF dotpath IN ('id', 'geometry', 'datetime', 'end_datetime', 'collection') THEN\n path := dotpath;\n expression := dotpath;\n wrapper := NULL;\n RETURN;\n END IF;\n SELECT * INTO q FROM queryables WHERE name=dotpath;\n IF q.property_wrapper IS NULL THEN\n IF q.definition->>'type' = 'number' THEN\n wrapper := 'to_float';\n ELSIF q.definition->>'format' = 'date-time' THEN\n wrapper := 'to_tstz';\n ELSE\n wrapper := 'to_text';\n END IF;\n ELSE\n wrapper := q.property_wrapper;\n END IF;\n IF q.property_path IS NOT NULL THEN\n path := q.property_path;\n ELSE\n path_elements := string_to_array(dotpath, '.');\n IF path_elements[1] IN ('links', 'assets', 'stac_version', 'stac_extensions') THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSIF path_elements[1] = 'properties' THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSE\n path := format($F$content->'properties'->%s$F$, array_to_path(path_elements));\n END IF;\n END IF;\n expression := format('%I(%s)', wrapper, path);\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\nCREATE OR REPLACE FUNCTION create_queryable_indexes() RETURNS VOID AS $$\nDECLARE\n queryable RECORD;\n q text;\nBEGIN\n FOR queryable IN\n SELECT\n queryables.id as qid,\n CASE WHEN collections.key IS NULL THEN 'items' ELSE format('_items_%s',collections.key) END AS part,\n property_index_type,\n expression\n FROM\n queryables\n LEFT JOIN collections ON (collections.id = ANY (queryables.collection_ids))\n JOIN LATERAL queryable(queryables.name) ON (queryables.property_index_type IS NOT NULL)\n LOOP\n q := format(\n $q$\n CREATE INDEX IF NOT EXISTS %I ON %I USING %s ((%s));\n $q$,\n format('%s_%s_idx', queryable.part, queryable.qid),\n queryable.part,\n COALESCE(queryable.property_index_type, 'to_text'),\n queryable.expression\n );\n RAISE NOTICE '%',q;\n EXECUTE q;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION queryables_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\nBEGIN\nPERFORM create_queryable_indexes();\nRETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER queryables_trigger AFTER INSERT OR UPDATE ON queryables\nFOR EACH STATEMENT EXECUTE PROCEDURE queryables_trigger_func();\n\nCREATE TRIGGER queryables_collection_trigger AFTER INSERT OR UPDATE ON collections\nFOR EACH STATEMENT EXECUTE PROCEDURE queryables_trigger_func();\nCREATE OR REPLACE FUNCTION parse_dtrange(\n _indate jsonb,\n relative_base timestamptz DEFAULT date_trunc('hour', CURRENT_TIMESTAMP)\n) RETURNS tstzrange AS $$\nDECLARE\n timestrs text[];\n s timestamptz;\n e timestamptz;\nBEGIN\n timestrs :=\n CASE\n WHEN _indate ? 'timestamp' THEN\n ARRAY[_indate->>'timestamp']\n WHEN _indate ? 'interval' THEN\n to_text_array(_indate->'interval')\n WHEN jsonb_typeof(_indate) = 'array' THEN\n to_text_array(_indate)\n ELSE\n regexp_split_to_array(\n _indate->>0,\n '/'\n )\n END;\n RAISE NOTICE 'TIMESTRS %', timestrs;\n IF cardinality(timestrs) = 1 THEN\n IF timestrs[1] ILIKE 'P%' THEN\n RETURN tstzrange(relative_base - upper(timestrs[1])::interval, relative_base, '[)');\n END IF;\n s := timestrs[1]::timestamptz;\n RETURN tstzrange(s, s, '[]');\n END IF;\n\n IF cardinality(timestrs) != 2 THEN\n RAISE EXCEPTION 'Timestamp cannot have more than 2 values';\n END IF;\n\n IF timestrs[1] = '..' THEN\n s := '-infinity'::timestamptz;\n e := timestrs[2]::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] = '..' THEN\n s := timestrs[1]::timestamptz;\n e := 'infinity'::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[1] ILIKE 'P%' AND timestrs[2] NOT ILIKE 'P%' THEN\n e := timestrs[2]::timestamptz;\n s := e - upper(timestrs[1])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] ILIKE 'P%' AND timestrs[1] NOT ILIKE 'P%' THEN\n s := timestrs[1]::timestamptz;\n e := s + upper(timestrs[2])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n s := timestrs[1]::timestamptz;\n e := timestrs[2]::timestamptz;\n\n RETURN tstzrange(s,e,'[)');\n\n RETURN NULL;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT PARALLEL SAFE SET TIME ZONE 'UTC';\n\nCREATE OR REPLACE FUNCTION parse_dtrange(\n _indate text,\n relative_base timestamptz DEFAULT CURRENT_TIMESTAMP\n) RETURNS tstzrange AS $$\n SELECT parse_dtrange(to_jsonb(_indate), relative_base);\n$$ LANGUAGE SQL STABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION temporal_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\n ll text := 'datetime';\n lh text := 'end_datetime';\n rrange tstzrange;\n rl text;\n rh text;\n outq text;\nBEGIN\n rrange := parse_dtrange(args->1);\n RAISE NOTICE 'Constructing temporal query OP: %, ARGS: %, RRANGE: %', op, args, rrange;\n op := lower(op);\n rl := format('%L::timestamptz', lower(rrange));\n rh := format('%L::timestamptz', upper(rrange));\n outq := CASE op\n WHEN 't_before' THEN 'lh < rl'\n WHEN 't_after' THEN 'll > rh'\n WHEN 't_meets' THEN 'lh = rl'\n WHEN 't_metby' THEN 'll = rh'\n WHEN 't_overlaps' THEN 'll < rl AND rl < lh < rh'\n WHEN 't_overlappedby' THEN 'rl < ll < rh AND lh > rh'\n WHEN 't_starts' THEN 'll = rl AND lh < rh'\n WHEN 't_startedby' THEN 'll = rl AND lh > rh'\n WHEN 't_during' THEN 'll > rl AND lh < rh'\n WHEN 't_contains' THEN 'll < rl AND lh > rh'\n WHEN 't_finishes' THEN 'll > rl AND lh = rh'\n WHEN 't_finishedby' THEN 'll < rl AND lh = rh'\n WHEN 't_equals' THEN 'll = rl AND lh = rh'\n WHEN 't_disjoint' THEN 'NOT (ll <= rh AND lh >= rl)'\n WHEN 't_intersects' THEN 'll <= rh AND lh >= rl'\n WHEN 'anyinteracts' THEN 'll <= rh AND lh >= rl'\n END;\n outq := regexp_replace(outq, '\\mll\\M', ll);\n outq := regexp_replace(outq, '\\mlh\\M', lh);\n outq := regexp_replace(outq, '\\mrl\\M', rl);\n outq := regexp_replace(outq, '\\mrh\\M', rh);\n outq := format('(%s)', outq);\n RETURN outq;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\n\n\nCREATE OR REPLACE FUNCTION spatial_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\n geom text;\n j jsonb := args->1;\nBEGIN\n op := lower(op);\n RAISE NOTICE 'Constructing spatial query OP: %, ARGS: %', op, args;\n IF op NOT IN ('s_equals','s_disjoint','s_touches','s_within','s_overlaps','s_crosses','s_intersects','intersects','s_contains') THEN\n RAISE EXCEPTION 'Spatial Operator % Not Supported', op;\n END IF;\n op := regexp_replace(op, '^s_', 'st_');\n IF op = 'intersects' THEN\n op := 'st_intersects';\n END IF;\n -- Convert geometry to WKB string\n IF j ? 'type' AND j ? 'coordinates' THEN\n geom := st_geomfromgeojson(j)::text;\n ELSIF jsonb_typeof(j) = 'array' THEN\n geom := bbox_geom(j)::text;\n END IF;\n\n RETURN format('%s(geometry, %L::geometry)', op, geom);\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION query_to_cql2(q jsonb) RETURNS jsonb AS $$\n-- Translates anything passed in through the deprecated \"query\" into equivalent CQL2\nWITH t AS (\n SELECT key as property, value as ops\n FROM jsonb_each(q)\n), t2 AS (\n SELECT property, (jsonb_each(ops)).*\n FROM t WHERE jsonb_typeof(ops) = 'object'\n UNION ALL\n SELECT property, 'eq', ops\n FROM t WHERE jsonb_typeof(ops) != 'object'\n)\nSELECT\n jsonb_strip_nulls(jsonb_build_object(\n 'op', 'and',\n 'args', jsonb_agg(\n jsonb_build_object(\n 'op', key,\n 'args', jsonb_build_array(\n jsonb_build_object('property',property),\n value\n )\n )\n )\n )\n) as qcql FROM t2\n;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\nCREATE OR REPLACE FUNCTION cql1_to_cql2(j jsonb) RETURNS jsonb AS $$\nDECLARE\n args jsonb;\n ret jsonb;\nBEGIN\n RAISE NOTICE 'CQL1_TO_CQL2: %', j;\n IF j ? 'filter' THEN\n RETURN cql1_to_cql2(j->'filter');\n END IF;\n IF j ? 'property' THEN\n RETURN j;\n END IF;\n IF jsonb_typeof(j) = 'array' THEN\n SELECT jsonb_agg(cql1_to_cql2(el)) INTO args FROM jsonb_array_elements(j) el;\n RETURN args;\n END IF;\n IF jsonb_typeof(j) = 'number' THEN\n RETURN j;\n END IF;\n IF jsonb_typeof(j) = 'string' THEN\n RETURN j;\n END IF;\n\n IF jsonb_typeof(j) = 'object' THEN\n SELECT jsonb_build_object(\n 'op', key,\n 'args', cql1_to_cql2(value)\n ) INTO ret\n FROM jsonb_each(j)\n WHERE j IS NOT NULL;\n RETURN ret;\n END IF;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE STRICT;\n\nCREATE TABLE cql2_ops (\n op text PRIMARY KEY,\n template text,\n types text[]\n);\nINSERT INTO cql2_ops (op, template, types) VALUES\n ('eq', '%s = %s', NULL),\n ('lt', '%s < %s', NULL),\n ('lte', '%s <= %s', NULL),\n ('gt', '%s > %s', NULL),\n ('gte', '%s >= %s', NULL),\n ('le', '%s <= %s', NULL),\n ('ge', '%s >= %s', NULL),\n ('=', '%s = %s', NULL),\n ('<', '%s < %s', NULL),\n ('<=', '%s <= %s', NULL),\n ('>', '%s > %s', NULL),\n ('>=', '%s >= %s', NULL),\n ('like', '%s LIKE %s', NULL),\n ('ilike', '%s ILIKE %s', NULL),\n ('+', '%s + %s', NULL),\n ('-', '%s - %s', NULL),\n ('*', '%s * %s', NULL),\n ('/', '%s / %s', NULL),\n ('in', '%s = ANY (%s)', NULL),\n ('not', 'NOT (%s)', NULL),\n ('between', '%s BETWEEN (%2$s)[1] AND (%2$s)[2]', NULL),\n ('isnull', '%s IS NULL', NULL)\nON CONFLICT (op) DO UPDATE\n SET\n template = EXCLUDED.template;\n;\n\n\nCREATE OR REPLACE FUNCTION cql2_query(j jsonb, wrapper text DEFAULT NULL) RETURNS text AS $$\n#variable_conflict use_variable\nDECLARE\n args jsonb := j->'args';\n arg jsonb;\n op text := lower(j->>'op');\n cql2op RECORD;\n literal text;\nBEGIN\n IF j IS NULL OR (op IS NOT NULL AND args IS NULL) THEN\n RETURN NULL;\n END IF;\n RAISE NOTICE 'CQL2_QUERY: %', j;\n IF j ? 'filter' THEN\n RETURN cql2_query(j->'filter');\n END IF;\n\n IF j ? 'upper' THEN\n RETURN format('upper(%s)', cql2_query(j->'upper'));\n END IF;\n\n IF j ? 'lower' THEN\n RETURN format('lower(%s)', cql2_query(j->'lower'));\n END IF;\n\n -- Temporal Query\n IF op ilike 't_%' or op = 'anyinteracts' THEN\n RETURN temporal_op_query(op, args);\n END IF;\n\n -- If property is a timestamp convert it to text to use with\n -- general operators\n IF j ? 'timestamp' THEN\n RETURN format('%L::timestamptz', to_tstz(j->'timestamp'));\n END IF;\n IF j ? 'interval' THEN\n RAISE EXCEPTION 'Please use temporal operators when using intervals.';\n RETURN NONE;\n END IF;\n\n -- Spatial Query\n IF op ilike 's_%' or op = 'intersects' THEN\n RETURN spatial_op_query(op, args);\n END IF;\n\n\n IF op = 'in' THEN\n RETURN format(\n '%s = ANY (%L)',\n cql2_query(args->0),\n to_text_array(args->1)\n );\n END IF;\n\n\n\n IF op = 'between' THEN\n SELECT (queryable(a->>'property')).wrapper INTO wrapper\n FROM jsonb_array_elements(args) a\n WHERE a ? 'property' LIMIT 1;\n\n RETURN format(\n '%s BETWEEN %s and %s',\n cql2_query(args->0, wrapper),\n cql2_query(args->1->0, wrapper),\n cql2_query(args->1->1, wrapper)\n );\n END IF;\n\n -- Make sure that args is an array and run cql2_query on\n -- each element of the array\n RAISE NOTICE 'ARGS PRE: %', args;\n IF j ? 'args' THEN\n IF jsonb_typeof(args) != 'array' THEN\n args := jsonb_build_array(args);\n END IF;\n\n SELECT (queryable(a->>'property')).wrapper INTO wrapper\n FROM jsonb_array_elements(args) a\n WHERE a ? 'property' LIMIT 1;\n\n SELECT jsonb_agg(cql2_query(a, wrapper))\n INTO args\n FROM jsonb_array_elements(args) a;\n END IF;\n RAISE NOTICE 'ARGS: %', args;\n\n IF op IN ('and', 'or') THEN\n RETURN\n format(\n '(%s)',\n array_to_string(to_text_array(args), format(' %s ', upper(op)))\n );\n END IF;\n\n -- Look up template from cql2_ops\n IF j ? 'op' THEN\n SELECT * INTO cql2op FROM cql2_ops WHERE cql2_ops.op ilike op;\n IF FOUND THEN\n -- If specific index set in queryables for a property cast other arguments to that type\n RETURN format(\n cql2op.template,\n VARIADIC (to_text_array(args))\n );\n ELSE\n RAISE EXCEPTION 'Operator % Not Supported.', op;\n END IF;\n END IF;\n\n\n IF j ? 'property' THEN\n RETURN (queryable(j->>'property')).expression;\n END IF;\n\n IF wrapper IS NOT NULL THEN\n EXECUTE format('SELECT %I(%L)', wrapper, j) INTO literal;\n RAISE NOTICE '% % %',wrapper, j, literal;\n RETURN format('%I(%L)', wrapper, j);\n END IF;\n\n RETURN quote_literal(to_text(j));\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION paging_dtrange(\n j jsonb\n) RETURNS tstzrange AS $$\nDECLARE\n op text;\n filter jsonb := j->'filter';\n dtrange tstzrange := tstzrange('-infinity'::timestamptz,'infinity'::timestamptz);\n sdate timestamptz := '-infinity'::timestamptz;\n edate timestamptz := 'infinity'::timestamptz;\n jpitem jsonb;\nBEGIN\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n END IF;\n IF NOT (filter @? '$.**.op ? (@ == \"or\" || @ == \"not\")') THEN\n FOR jpitem IN SELECT j FROM jsonb_path_query(filter,'strict $.** ? (@.args[*].property == \"datetime\")'::jsonpath) j LOOP\n op := lower(jpitem->>'op');\n dtrange := parse_dtrange(jpitem->'args'->1);\n IF op IN ('<=', 'lt', 'lte', '<', 'le', 't_before') THEN\n sdate := greatest(sdate,'-infinity');\n edate := least(edate, upper(dtrange));\n ELSIF op IN ('>=', '>', 'gt', 'gte', 'ge', 't_after') THEN\n edate := least(edate, 'infinity');\n sdate := greatest(sdate, lower(dtrange));\n ELSIF op IN ('=', 'eq') THEN\n edate := least(edate, upper(dtrange));\n sdate := greatest(sdate, lower(dtrange));\n END IF;\n RAISE NOTICE '2 OP: %, ARGS: %, DTRANGE: %, SDATE: %, EDATE: %', op, jpitem->'args'->1, dtrange, sdate, edate;\n END LOOP;\n END IF;\n IF sdate > edate THEN\n RETURN 'empty'::tstzrange;\n END IF;\n RETURN tstzrange(sdate,edate, '[]');\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT SET TIME ZONE 'UTC';\n\nCREATE OR REPLACE FUNCTION paging_collections(\n IN j jsonb\n) RETURNS text[] AS $$\nDECLARE\n filter jsonb := j->'filter';\n jpitem jsonb;\n op text;\n args jsonb;\n arg jsonb;\n collections text[];\nBEGIN\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n END IF;\n IF NOT (filter @? '$.**.op ? (@ == \"or\" || @ == \"not\")') THEN\n FOR jpitem IN SELECT j FROM jsonb_path_query(filter,'strict $.** ? (@.args[*].property == \"collection\")'::jsonpath) j LOOP\n RAISE NOTICE 'JPITEM: %', jpitem;\n op := jpitem->>'op';\n args := jpitem->'args';\n IF op IN ('=', 'eq', 'in') THEN\n FOR arg IN SELECT a FROM jsonb_array_elements(args) a LOOP\n IF jsonb_typeof(arg) IN ('string', 'array') THEN\n RAISE NOTICE 'arg: %, collections: %', arg, collections;\n IF collections IS NULL OR collections = '{}'::text[] THEN\n collections := to_text_array(arg);\n ELSE\n collections := array_intersection(collections, to_text_array(arg));\n END IF;\n END IF;\n END LOOP;\n END IF;\n END LOOP;\n END IF;\n IF collections = '{}'::text[] THEN\n RETURN NULL;\n END IF;\n RETURN collections;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\nCREATE TABLE items (\n id text NOT NULL,\n geometry geometry NOT NULL,\n collection text NOT NULL,\n datetime timestamptz NOT NULL,\n end_datetime timestamptz NOT NULL,\n content JSONB NOT NULL\n)\nPARTITION BY LIST (collection)\n;\n\nCREATE INDEX \"datetime_idx\" ON items USING BTREE (datetime DESC, end_datetime ASC);\nCREATE INDEX \"geometry_idx\" ON items USING GIST (geometry);\n\nCREATE STATISTICS datetime_stats (dependencies) on datetime, end_datetime from items;\n\n\nALTER TABLE items ADD CONSTRAINT items_collections_fk FOREIGN KEY (collection) REFERENCES collections(id) ON DELETE CASCADE DEFERRABLE;\n\n\nCREATE OR REPLACE FUNCTION content_slim(_item jsonb) RETURNS jsonb AS $$\n SELECT strip_jsonb(_item - '{id,geometry,collection,type}'::text[], collection_base_item(_item->>'collection')) - '{id,geometry,collection,type}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_dehydrate(content jsonb) RETURNS items AS $$\n SELECT\n content->>'id' as id,\n stac_geom(content) as geometry,\n content->>'collection' as collection,\n stac_datetime(content) as datetime,\n stac_end_datetime(content) as end_datetime,\n content_slim(content) as content\n ;\n$$ LANGUAGE SQL STABLE;\n\nCREATE OR REPLACE FUNCTION include_field(f text, fields jsonb DEFAULT '{}'::jsonb) RETURNS boolean AS $$\nDECLARE\n includes jsonb := fields->'include';\n excludes jsonb := fields->'exclude';\nBEGIN\n IF f IS NULL THEN\n RETURN NULL;\n END IF;\n\n\n IF\n jsonb_typeof(excludes) = 'array'\n AND jsonb_array_length(excludes)>0\n AND excludes ? f\n THEN\n RETURN FALSE;\n END IF;\n\n IF\n (\n jsonb_typeof(includes) = 'array'\n AND jsonb_array_length(includes) > 0\n AND includes ? f\n ) OR\n (\n includes IS NULL\n OR jsonb_typeof(includes) = 'null'\n OR jsonb_array_length(includes) = 0\n )\n THEN\n RETURN TRUE;\n END IF;\n\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nDROP FUNCTION IF EXISTS content_hydrate(jsonb, jsonb, jsonb);\nCREATE OR REPLACE FUNCTION content_hydrate(\n _item jsonb,\n _base_item jsonb,\n fields jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\n SELECT merge_jsonb(\n jsonb_fields(_item, fields),\n jsonb_fields(_base_item, fields)\n );\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION content_hydrate(_item items, _collection collections, fields jsonb DEFAULT '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\n content jsonb;\n base_item jsonb := _collection.base_item;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry, 20)::jsonb;\n END IF;\n output := content_hydrate(\n jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'collection', _item.collection,\n 'type', 'Feature'\n ) || _item.content,\n _collection.base_item,\n fields\n );\n\n RETURN output;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_nonhydrated(\n _item items,\n fields jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry, 20)::jsonb;\n END IF;\n output := jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'collection', _item.collection,\n 'type', 'Feature'\n ) || _item.content;\n RETURN output;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_hydrate(_item items, fields jsonb DEFAULT '{}'::jsonb) RETURNS jsonb AS $$\n SELECT content_hydrate(\n _item,\n (SELECT c FROM collections c WHERE id=_item.collection LIMIT 1),\n fields\n );\n$$ LANGUAGE SQL STABLE;\n\n\nCREATE UNLOGGED TABLE items_staging (\n content JSONB NOT NULL\n);\nCREATE UNLOGGED TABLE items_staging_ignore (\n content JSONB NOT NULL\n);\nCREATE UNLOGGED TABLE items_staging_upsert (\n content JSONB NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION items_staging_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p record;\n _partitions text[];\n ts timestamptz := clock_timestamp();\nBEGIN\n RAISE NOTICE 'Creating Partitions. %', clock_timestamp() - ts;\n WITH ranges AS (\n SELECT\n n.content->>'collection' as collection,\n stac_daterange(n.content->'properties') as dtr\n FROM newdata n\n ), p AS (\n SELECT\n collection,\n lower(dtr) as datetime,\n upper(dtr) as end_datetime,\n (partition_name(\n collection,\n lower(dtr)\n )).partition_name as name\n FROM ranges\n )\n INSERT INTO partitions (collection, datetime_range, end_datetime_range)\n SELECT\n collection,\n tstzrange(min(datetime), max(datetime), '[]') as datetime_range,\n tstzrange(min(end_datetime), max(end_datetime), '[]') as end_datetime_range\n FROM p\n GROUP BY collection, name\n ON CONFLICT (name) DO UPDATE SET\n datetime_range = EXCLUDED.datetime_range,\n end_datetime_range = EXCLUDED.end_datetime_range\n ;\n\n RAISE NOTICE 'Doing the insert. %', clock_timestamp() - ts;\n IF TG_TABLE_NAME = 'items_staging' THEN\n INSERT INTO items\n SELECT\n (content_dehydrate(content)).*\n FROM newdata;\n DELETE FROM items_staging;\n ELSIF TG_TABLE_NAME = 'items_staging_ignore' THEN\n INSERT INTO items\n SELECT\n (content_dehydrate(content)).*\n FROM newdata\n ON CONFLICT DO NOTHING;\n DELETE FROM items_staging_ignore;\n ELSIF TG_TABLE_NAME = 'items_staging_upsert' THEN\n WITH staging_formatted AS (\n SELECT (content_dehydrate(content)).* FROM newdata\n ), deletes AS (\n DELETE FROM items i USING staging_formatted s\n WHERE\n i.id = s.id\n AND i.collection = s.collection\n AND i IS DISTINCT FROM s\n RETURNING i.id, i.collection\n )\n INSERT INTO items\n SELECT s.* FROM\n staging_formatted s\n JOIN deletes d\n USING (id, collection);\n DELETE FROM items_staging_upsert;\n END IF;\n RAISE NOTICE 'Done. %', clock_timestamp() - ts;\n\n RETURN NULL;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER items_staging_insert_trigger AFTER INSERT ON items_staging REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\nCREATE TRIGGER items_staging_insert_ignore_trigger AFTER INSERT ON items_staging_ignore REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\nCREATE TRIGGER items_staging_insert_upsert_trigger AFTER INSERT ON items_staging_upsert REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\n\n\n\nCREATE OR REPLACE FUNCTION item_by_id(_id text, _collection text DEFAULT NULL) RETURNS items AS\n$$\nDECLARE\n i items%ROWTYPE;\nBEGIN\n SELECT * INTO i FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection) LIMIT 1;\n RETURN i;\nEND;\n$$ LANGUAGE PLPGSQL STABLE SECURITY DEFINER SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION get_item(_id text, _collection text DEFAULT NULL) RETURNS jsonb AS $$\n SELECT content_hydrate(items) FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection);\n$$ LANGUAGE SQL STABLE SECURITY DEFINER SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION delete_item(_id text, _collection text DEFAULT NULL) RETURNS VOID AS $$\nDECLARE\nout items%ROWTYPE;\nBEGIN\n DELETE FROM items WHERE id = _id AND (_collection IS NULL OR collection=_collection) RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n--/*\nCREATE OR REPLACE FUNCTION create_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION update_item(content jsonb) RETURNS VOID AS $$\nDECLARE\n old items %ROWTYPE;\n out items%ROWTYPE;\nBEGIN\n PERFORM delete_item(content->>'id', content->>'collection');\n PERFORM create_item(content);\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION create_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION collection_bbox(id text) RETURNS jsonb AS $$\n SELECT (replace(replace(replace(st_extent(geometry)::text,'BOX(','[['),')',']]'),' ',','))::jsonb\n FROM items WHERE collection=$1;\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION collection_temporal_extent(id text) RETURNS jsonb AS $$\n SELECT to_jsonb(array[array[min(datetime)::text, max(datetime)::text]])\n FROM items WHERE collection=$1;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection_extents() RETURNS VOID AS $$\nUPDATE collections SET\n content = content ||\n jsonb_build_object(\n 'extent', jsonb_build_object(\n 'spatial', jsonb_build_object(\n 'bbox', collection_bbox(collections.id)\n ),\n 'temporal', jsonb_build_object(\n 'interval', collection_temporal_extent(collections.id)\n )\n )\n )\n;\n$$ LANGUAGE SQL;\nCREATE VIEW partition_steps AS\nSELECT\n name,\n date_trunc('month',lower(datetime_range)) as sdate,\n date_trunc('month', upper(datetime_range)) + '1 month'::interval as edate\n FROM partitions WHERE datetime_range IS NOT NULL AND datetime_range != 'empty'::tstzrange\n ORDER BY datetime_range ASC\n;\n\nCREATE OR REPLACE FUNCTION chunker(\n IN _where text,\n OUT s timestamptz,\n OUT e timestamptz\n) RETURNS SETOF RECORD AS $$\nDECLARE\n explain jsonb;\nBEGIN\n IF _where IS NULL THEN\n _where := ' TRUE ';\n END IF;\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s;', _where)\n INTO explain;\n\n RETURN QUERY\n WITH t AS (\n SELECT j->>0 as p FROM\n jsonb_path_query(\n explain,\n 'strict $.**.\"Relation Name\" ? (@ != null)'\n ) j\n ),\n parts AS (\n SELECT sdate, edate FROM t JOIN partition_steps ON (t.p = name)\n ),\n times AS (\n SELECT sdate FROM parts\n UNION\n SELECT edate FROM parts\n ),\n uniq AS (\n SELECT DISTINCT sdate FROM times ORDER BY sdate\n ),\n last AS (\n SELECT sdate, lead(sdate, 1) over () as edate FROM uniq\n )\n SELECT sdate, edate FROM last WHERE edate IS NOT NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION partition_queries(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN partitions text[] DEFAULT NULL\n) RETURNS SETOF text AS $$\nDECLARE\n query text;\n sdate timestamptz;\n edate timestamptz;\nBEGIN\nIF _where IS NULL OR trim(_where) = '' THEN\n _where = ' TRUE ';\nEND IF;\nRAISE NOTICE 'Getting chunks for % %', _where, _orderby;\nIF _orderby ILIKE 'datetime d%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 DESC LOOP\n RETURN NEXT format($q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n $q$,\n sdate,\n edate,\n _where,\n _orderby\n );\n END LOOP;\nELSIF _orderby ILIKE 'datetime a%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 ASC LOOP\n RETURN NEXT format($q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n $q$,\n sdate,\n edate,\n _where,\n _orderby\n );\n END LOOP;\nELSE\n query := format($q$\n SELECT * FROM items\n WHERE %s\n ORDER BY %s\n $q$, _where, _orderby\n );\n\n RETURN NEXT query;\n RETURN;\nEND IF;\n\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION partition_query_view(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN _limit int DEFAULT 10\n) RETURNS text AS $$\n WITH p AS (\n SELECT * FROM partition_queries(_where, _orderby) p\n )\n SELECT\n CASE WHEN EXISTS (SELECT 1 FROM p) THEN\n (SELECT format($q$\n SELECT * FROM (\n %s\n ) total LIMIT %s\n $q$,\n string_agg(\n format($q$ SELECT * FROM ( %s ) AS sub $q$, p),\n '\n UNION ALL\n '\n ),\n _limit\n ))\n ELSE NULL\n END FROM p;\n$$ LANGUAGE SQL IMMUTABLE;\n\n\n\n\nCREATE OR REPLACE FUNCTION stac_search_to_where(j jsonb) RETURNS text AS $$\nDECLARE\n where_segments text[];\n _where text;\n dtrange tstzrange;\n collections text[];\n geom geometry;\n sdate timestamptz;\n edate timestamptz;\n filterlang text;\n filter jsonb := j->'filter';\nBEGIN\n IF j ? 'ids' THEN\n where_segments := where_segments || format('id = ANY (%L) ', to_text_array(j->'ids'));\n END IF;\n\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n where_segments := where_segments || format('collection = ANY (%L) ', collections);\n END IF;\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n\n where_segments := where_segments || format(' datetime <= %L::timestamptz AND end_datetime >= %L::timestamptz ',\n edate,\n sdate\n );\n END IF;\n\n geom := stac_geom(j);\n IF geom IS NOT NULL THEN\n where_segments := where_segments || format('st_intersects(geometry, %L)',geom);\n END IF;\n\n filterlang := COALESCE(\n j->>'filter-lang',\n get_setting('default-filter-lang', j->'conf')\n );\n IF NOT filter @? '$.**.op' THEN\n filterlang := 'cql-json';\n END IF;\n\n IF filterlang NOT IN ('cql-json','cql2-json') AND j ? 'filter' THEN\n RAISE EXCEPTION '% is not a supported filter-lang. Please use cql-json or cql2-json.', filterlang;\n END IF;\n\n IF j ? 'query' AND j ? 'filter' THEN\n RAISE EXCEPTION 'Can only use either query or filter at one time.';\n END IF;\n\n IF j ? 'query' THEN\n filter := query_to_cql2(j->'query');\n ELSIF filterlang = 'cql-json' THEN\n filter := cql1_to_cql2(filter);\n END IF;\n RAISE NOTICE 'FILTER: %', filter;\n where_segments := where_segments || cql2_query(filter);\n IF cardinality(where_segments) < 1 THEN\n RETURN ' TRUE ';\n END IF;\n\n _where := array_to_string(array_remove(where_segments, NULL), ' AND ');\n\n IF _where IS NULL OR BTRIM(_where) = '' THEN\n RETURN ' TRUE ';\n END IF;\n RETURN _where;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION parse_sort_dir(_dir text, reverse boolean default false) RETURNS text AS $$\n WITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n ) SELECT\n CASE\n WHEN NOT reverse THEN d\n WHEN d = 'ASC' THEN 'DESC'\n WHEN d = 'DESC' THEN 'ASC'\n END\n FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION sort_dir_to_op(_dir text, prev boolean default false) RETURNS text AS $$\n WITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n ) SELECT\n CASE\n WHEN d = 'ASC' AND prev THEN '<='\n WHEN d = 'DESC' AND prev THEN '>='\n WHEN d = 'ASC' THEN '>='\n WHEN d = 'DESC' THEN '<='\n END\n FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION sort_sqlorderby(\n _search jsonb DEFAULT NULL,\n reverse boolean DEFAULT FALSE\n) RETURNS text AS $$\n WITH sortby AS (\n SELECT coalesce(_search->'sortby','[{\"field\":\"datetime\", \"direction\":\"desc\"}]') as sort\n ), withid AS (\n SELECT CASE\n WHEN sort @? '$[*] ? (@.field == \"id\")' THEN sort\n ELSE sort || '[{\"field\":\"id\", \"direction\":\"desc\"}]'::jsonb\n END as sort\n FROM sortby\n ), withid_rows AS (\n SELECT jsonb_array_elements(sort) as value FROM withid\n ),sorts AS (\n SELECT\n coalesce(\n -- field_orderby((items_path(value->>'field')).path_txt),\n (queryable(value->>'field')).expression\n ) as key,\n parse_sort_dir(value->>'direction', reverse) as dir\n FROM withid_rows\n )\n SELECT array_to_string(\n array_agg(concat(key, ' ', dir)),\n ', '\n ) FROM sorts;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_sort_dir(sort_item jsonb) RETURNS text AS $$\n SELECT CASE WHEN sort_item->>'direction' ILIKE 'desc%' THEN 'DESC' ELSE 'ASC' END;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION get_token_filter(_search jsonb = '{}'::jsonb, token_rec jsonb DEFAULT NULL) RETURNS text AS $$\nDECLARE\n token_id text;\n filters text[] := '{}'::text[];\n prev boolean := TRUE;\n field text;\n dir text;\n sort record;\n orfilters text[] := '{}'::text[];\n andfilters text[] := '{}'::text[];\n output text;\n token_where text;\nBEGIN\n RAISE NOTICE 'Getting Token Filter. % %', _search, token_rec;\n -- If no token provided return NULL\n IF token_rec IS NULL THEN\n IF NOT (_search ? 'token' AND\n (\n (_search->>'token' ILIKE 'prev:%')\n OR\n (_search->>'token' ILIKE 'next:%')\n )\n ) THEN\n RETURN NULL;\n END IF;\n prev := (_search->>'token' ILIKE 'prev:%');\n token_id := substr(_search->>'token', 6);\n SELECT to_jsonb(items) INTO token_rec\n FROM items WHERE id=token_id;\n END IF;\n RAISE NOTICE 'TOKEN ID: % %', token_rec, token_rec->'id';\n\n CREATE TEMP TABLE sorts (\n _row int GENERATED ALWAYS AS IDENTITY NOT NULL,\n _field text PRIMARY KEY,\n _dir text NOT NULL,\n _val text\n ) ON COMMIT DROP;\n\n -- Make sure we only have distinct columns to sort with taking the first one we get\n INSERT INTO sorts (_field, _dir)\n SELECT\n (queryable(value->>'field')).expression,\n get_sort_dir(value)\n FROM\n jsonb_array_elements(coalesce(_search->'sortby','[{\"field\":\"datetime\",\"direction\":\"desc\"}]'))\n ON CONFLICT DO NOTHING\n ;\n RAISE NOTICE 'sorts 1: %', (SELECT jsonb_agg(to_json(sorts)) FROM sorts);\n -- Get the first sort direction provided. As the id is a primary key, if there are any\n -- sorts after id they won't do anything, so make sure that id is the last sort item.\n SELECT _dir INTO dir FROM sorts ORDER BY _row ASC LIMIT 1;\n IF EXISTS (SELECT 1 FROM sorts WHERE _field = 'id') THEN\n DELETE FROM sorts WHERE _row > (SELECT _row FROM sorts WHERE _field = 'id' ORDER BY _row ASC);\n ELSE\n INSERT INTO sorts (_field, _dir) VALUES ('id', dir);\n END IF;\n\n -- Add value from looked up item to the sorts table\n UPDATE sorts SET _val=quote_literal(token_rec->>_field);\n\n -- Check if all sorts are the same direction and use row comparison\n -- to filter\n RAISE NOTICE 'sorts 2: %', (SELECT jsonb_agg(to_json(sorts)) FROM sorts);\n\n IF (SELECT count(DISTINCT _dir) FROM sorts) = 1 THEN\n SELECT format(\n '(%s) %s (%s)',\n concat_ws(', ', VARIADIC array_agg(quote_ident(_field))),\n CASE WHEN (prev AND dir = 'ASC') OR (NOT prev AND dir = 'DESC') THEN '<' ELSE '>' END,\n concat_ws(', ', VARIADIC array_agg(_val))\n ) INTO output FROM sorts\n WHERE token_rec ? _field\n ;\n ELSE\n FOR sort IN SELECT * FROM sorts ORDER BY _row asc LOOP\n RAISE NOTICE 'SORT: %', sort;\n IF sort._row = 1 THEN\n orfilters := orfilters || format('(%s %s %s)',\n quote_ident(sort._field),\n CASE WHEN (prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC') THEN '<' ELSE '>' END,\n sort._val\n );\n ELSE\n orfilters := orfilters || format('(%s AND %s %s %s)',\n array_to_string(andfilters, ' AND '),\n quote_ident(sort._field),\n CASE WHEN (prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC') THEN '<' ELSE '>' END,\n sort._val\n );\n\n END IF;\n andfilters := andfilters || format('%s = %s',\n quote_ident(sort._field),\n sort._val\n );\n END LOOP;\n output := array_to_string(orfilters, ' OR ');\n END IF;\n DROP TABLE IF EXISTS sorts;\n token_where := concat('(',coalesce(output,'true'),')');\n IF trim(token_where) = '' THEN\n token_where := NULL;\n END IF;\n RAISE NOTICE 'TOKEN_WHERE: |%|',token_where;\n RETURN token_where;\n END;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION search_tohash(jsonb) RETURNS jsonb AS $$\n SELECT $1 - '{token,limit,context,includes,excludes}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION search_hash(jsonb, jsonb) RETURNS text AS $$\n SELECT md5(concat(search_tohash($1)::text,$2::text));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TABLE IF NOT EXISTS searches(\n hash text GENERATED ALWAYS AS (search_hash(search, metadata)) STORED PRIMARY KEY,\n search jsonb NOT NULL,\n _where text,\n orderby text,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n metadata jsonb DEFAULT '{}'::jsonb NOT NULL\n);\nCREATE TABLE IF NOT EXISTS search_wheres(\n id bigint generated always as identity primary key,\n _where text NOT NULL,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n statslastupdated timestamptz,\n estimated_count bigint,\n estimated_cost float,\n time_to_estimate float,\n total_count bigint,\n time_to_count float,\n partitions text[]\n);\n\nCREATE INDEX IF NOT EXISTS search_wheres_partitions ON search_wheres USING GIN (partitions);\nCREATE UNIQUE INDEX IF NOT EXISTS search_wheres_where ON search_wheres ((md5(_where)));\n\nCREATE OR REPLACE FUNCTION where_stats(inwhere text, updatestats boolean default false, conf jsonb default null) RETURNS search_wheres AS $$\nDECLARE\n t timestamptz;\n i interval;\n explain_json jsonb;\n partitions text[];\n sw search_wheres%ROWTYPE;\n inwhere_hash text := md5(inwhere);\n _context text := lower(context(conf));\n _stats_ttl interval := context_stats_ttl(conf);\n _estimated_cost float := context_estimated_cost(conf);\n _estimated_count int := context_estimated_count(conf);\nBEGIN\n IF _context = 'off' THEN\n sw._where := inwhere;\n return sw;\n END IF;\n\n SELECT * INTO sw FROM search_wheres WHERE md5(_where)=inwhere_hash FOR UPDATE;\n\n -- Update statistics if explicitly set, if statistics do not exist, or statistics ttl has expired\n IF NOT updatestats THEN\n RAISE NOTICE 'Checking if update is needed for: % .', inwhere;\n RAISE NOTICE 'Stats Last Updated: %', sw.statslastupdated;\n RAISE NOTICE 'TTL: %, Age: %', _stats_ttl, now() - sw.statslastupdated;\n RAISE NOTICE 'Context: %, Existing Total: %', _context, sw.total_count;\n IF\n sw.statslastupdated IS NULL\n OR (now() - sw.statslastupdated) > _stats_ttl\n OR (context(conf) != 'off' AND sw.total_count IS NULL)\n THEN\n updatestats := TRUE;\n END IF;\n END IF;\n\n sw._where := inwhere;\n sw.lastused := now();\n sw.usecount := coalesce(sw.usecount,0) + 1;\n\n IF NOT updatestats THEN\n UPDATE search_wheres SET\n lastused = sw.lastused,\n usecount = sw.usecount\n WHERE md5(_where) = inwhere_hash\n RETURNING * INTO sw\n ;\n RETURN sw;\n END IF;\n\n -- Use explain to get estimated count/cost and a list of the partitions that would be hit by the query\n t := clock_timestamp();\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s', inwhere)\n INTO explain_json;\n RAISE NOTICE 'Time for just the explain: %', clock_timestamp() - t;\n i := clock_timestamp() - t;\n\n sw.statslastupdated := now();\n sw.estimated_count := explain_json->0->'Plan'->'Plan Rows';\n sw.estimated_cost := explain_json->0->'Plan'->'Total Cost';\n sw.time_to_estimate := extract(epoch from i);\n\n RAISE NOTICE 'ESTIMATED_COUNT: % < %', sw.estimated_count, _estimated_count;\n RAISE NOTICE 'ESTIMATED_COST: % < %', sw.estimated_cost, _estimated_cost;\n\n -- Do a full count of rows if context is set to on or if auto is set and estimates are low enough\n IF\n _context = 'on'\n OR\n ( _context = 'auto' AND\n (\n sw.estimated_count < _estimated_count\n AND\n sw.estimated_cost < _estimated_cost\n )\n )\n THEN\n t := clock_timestamp();\n RAISE NOTICE 'Calculating actual count...';\n EXECUTE format(\n 'SELECT count(*) FROM items WHERE %s',\n inwhere\n ) INTO sw.total_count;\n i := clock_timestamp() - t;\n RAISE NOTICE 'Actual Count: % -- %', sw.total_count, i;\n sw.time_to_count := extract(epoch FROM i);\n ELSE\n sw.total_count := NULL;\n sw.time_to_count := NULL;\n END IF;\n\n\n INSERT INTO search_wheres\n (_where, lastused, usecount, statslastupdated, estimated_count, estimated_cost, time_to_estimate, partitions, total_count, time_to_count)\n SELECT sw._where, sw.lastused, sw.usecount, sw.statslastupdated, sw.estimated_count, sw.estimated_cost, sw.time_to_estimate, sw.partitions, sw.total_count, sw.time_to_count\n ON CONFLICT ((md5(_where)))\n DO UPDATE\n SET\n lastused = sw.lastused,\n usecount = sw.usecount,\n statslastupdated = sw.statslastupdated,\n estimated_count = sw.estimated_count,\n estimated_cost = sw.estimated_cost,\n time_to_estimate = sw.time_to_estimate,\n total_count = sw.total_count,\n time_to_count = sw.time_to_count\n ;\n RETURN sw;\nEND;\n$$ LANGUAGE PLPGSQL ;\n\n\n\nDROP FUNCTION IF EXISTS search_query;\nCREATE OR REPLACE FUNCTION search_query(\n _search jsonb = '{}'::jsonb,\n updatestats boolean = false,\n _metadata jsonb = '{}'::jsonb\n) RETURNS searches AS $$\nDECLARE\n search searches%ROWTYPE;\n pexplain jsonb;\n t timestamptz;\n i interval;\nBEGIN\n SELECT * INTO search FROM searches\n WHERE hash=search_hash(_search, _metadata) FOR UPDATE;\n\n -- Calculate the where clause if not already calculated\n IF search._where IS NULL THEN\n search._where := stac_search_to_where(_search);\n END IF;\n\n -- Calculate the order by clause if not already calculated\n IF search.orderby IS NULL THEN\n search.orderby := sort_sqlorderby(_search);\n END IF;\n\n PERFORM where_stats(search._where, updatestats, _search->'conf');\n\n search.lastused := now();\n search.usecount := coalesce(search.usecount, 0) + 1;\n INSERT INTO searches (search, _where, orderby, lastused, usecount, metadata)\n VALUES (_search, search._where, search.orderby, search.lastused, search.usecount, _metadata)\n ON CONFLICT (hash) DO\n UPDATE SET\n _where = EXCLUDED._where,\n orderby = EXCLUDED.orderby,\n lastused = EXCLUDED.lastused,\n usecount = EXCLUDED.usecount,\n metadata = EXCLUDED.metadata\n RETURNING * INTO search\n ;\n RETURN search;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION search(_search jsonb = '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n searches searches%ROWTYPE;\n _where text;\n token_where text;\n full_where text;\n orderby text;\n query text;\n token_type text := substr(_search->>'token',1,4);\n _limit int := coalesce((_search->>'limit')::int, 10);\n curs refcursor;\n cntr int := 0;\n iter_record items%ROWTYPE;\n first_record jsonb;\n first_item items%ROWTYPE;\n last_item items%ROWTYPE;\n last_record jsonb;\n out_records jsonb := '[]'::jsonb;\n prev_query text;\n next text;\n prev_id text;\n has_next boolean := false;\n has_prev boolean := false;\n prev text;\n total_count bigint;\n context jsonb;\n collection jsonb;\n includes text[];\n excludes text[];\n exit_flag boolean := FALSE;\n batches int := 0;\n timer timestamptz := clock_timestamp();\n pstart timestamptz;\n pend timestamptz;\n pcurs refcursor;\n search_where search_wheres%ROWTYPE;\n id text;\nBEGIN\nCREATE TEMP TABLE results (content jsonb) ON COMMIT DROP;\n-- if ids is set, short circuit and just use direct ids query for each id\n-- skip any paging or caching\n-- hard codes ordering in the same order as the array of ids\nIF _search ? 'ids' THEN\n INSERT INTO results\n SELECT\n CASE WHEN _search->'conf'->>'nohydrate' IS NOT NULL AND (_search->'conf'->>'nohydrate')::boolean = true THEN\n content_nonhydrated(items, _search->'fields')\n ELSE\n content_hydrate(items, _search->'fields')\n END\n FROM items WHERE\n items.id = ANY(to_text_array(_search->'ids'))\n AND\n CASE WHEN _search ? 'collections' THEN\n items.collection = ANY(to_text_array(_search->'collections'))\n ELSE TRUE\n END\n ORDER BY items.datetime desc, items.id desc\n ;\n SELECT INTO total_count count(*) FROM results;\nELSE\n searches := search_query(_search);\n _where := searches._where;\n orderby := searches.orderby;\n search_where := where_stats(_where);\n total_count := coalesce(search_where.total_count, search_where.estimated_count);\n\n IF token_type='prev' THEN\n token_where := get_token_filter(_search, null::jsonb);\n orderby := sort_sqlorderby(_search, TRUE);\n END IF;\n IF token_type='next' THEN\n token_where := get_token_filter(_search, null::jsonb);\n END IF;\n\n full_where := concat_ws(' AND ', _where, token_where);\n RAISE NOTICE 'FULL QUERY % %', full_where, clock_timestamp()-timer;\n timer := clock_timestamp();\n\n FOR query IN SELECT partition_queries(full_where, orderby, search_where.partitions) LOOP\n timer := clock_timestamp();\n query := format('%s LIMIT %s', query, _limit + 1);\n RAISE NOTICE 'Partition Query: %', query;\n batches := batches + 1;\n -- curs = create_cursor(query);\n OPEN curs FOR EXECUTE query;\n LOOP\n FETCH curs into iter_record;\n EXIT WHEN NOT FOUND;\n cntr := cntr + 1;\n\n IF _search->'conf'->>'nohydrate' IS NOT NULL AND (_search->'conf'->>'nohydrate')::boolean = true THEN\n last_record := content_nonhydrated(iter_record, _search->'fields');\n ELSE\n last_record := content_hydrate(iter_record, _search->'fields');\n END IF;\n last_item := iter_record;\n IF cntr = 1 THEN\n first_item := last_item;\n first_record := last_record;\n END IF;\n IF cntr <= _limit THEN\n INSERT INTO results (content) VALUES (last_record);\n ELSIF cntr > _limit THEN\n has_next := true;\n exit_flag := true;\n EXIT;\n END IF;\n END LOOP;\n CLOSE curs;\n RAISE NOTICE 'Query took %.', clock_timestamp()-timer;\n timer := clock_timestamp();\n EXIT WHEN exit_flag;\n END LOOP;\n RAISE NOTICE 'Scanned through % partitions.', batches;\nEND IF;\n\nSELECT jsonb_agg(content) INTO out_records FROM results WHERE content is not NULL;\n\nDROP TABLE results;\n\n\n-- Flip things around if this was the result of a prev token query\nIF token_type='prev' THEN\n out_records := flip_jsonb_array(out_records);\n first_record := last_record;\nEND IF;\n\n-- If this query has a token, see if there is data before the first record\nIF _search ? 'token' THEN\n prev_query := format(\n 'SELECT 1 FROM items WHERE %s LIMIT 1',\n concat_ws(\n ' AND ',\n _where,\n trim(get_token_filter(_search, to_jsonb(first_item)))\n )\n );\n RAISE NOTICE 'Query to get previous record: % --- %', prev_query, first_record;\n EXECUTE prev_query INTO has_prev;\n IF FOUND and has_prev IS NOT NULL THEN\n RAISE NOTICE 'Query results from prev query: %', has_prev;\n has_prev := TRUE;\n END IF;\nEND IF;\nhas_prev := COALESCE(has_prev, FALSE);\n\nIF has_prev THEN\n prev := out_records->0->>'id';\nEND IF;\nIF has_next OR token_type='prev' THEN\n next := out_records->-1->>'id';\nEND IF;\n\nIF context(_search->'conf') != 'off' THEN\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'matched', total_count,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\nELSE\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\nEND IF;\n\ncollection := jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb),\n 'next', next,\n 'prev', prev,\n 'context', context\n);\n\nRETURN collection;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER SET SEARCH_PATH TO pgstac, public;\n\n\nCREATE OR REPLACE FUNCTION search_cursor(_search jsonb = '{}'::jsonb) RETURNS refcursor AS $$\nDECLARE\n curs refcursor;\n searches searches%ROWTYPE;\n _where text;\n _orderby text;\n q text;\n\nBEGIN\n searches := search_query(_search);\n _where := searches._where;\n _orderby := searches.orderby;\n\n OPEN curs FOR\n WITH p AS (\n SELECT * FROM partition_queries(_where, _orderby) p\n )\n SELECT\n CASE WHEN EXISTS (SELECT 1 FROM p) THEN\n (SELECT format($q$\n SELECT * FROM (\n %s\n ) total\n $q$,\n string_agg(\n format($q$ SELECT * FROM ( %s ) AS sub $q$, p),\n '\n UNION ALL\n '\n )\n ))\n ELSE NULL\n END FROM p;\n RETURN curs;\nEND;\n$$ LANGUAGE PLPGSQL;\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION tileenvelope(zoom int, x int, y int) RETURNS geometry AS $$\nWITH t AS (\n SELECT\n 20037508.3427892 as merc_max,\n -20037508.3427892 as merc_min,\n (2 * 20037508.3427892) / (2 ^ zoom) as tile_size\n)\nSELECT st_makeenvelope(\n merc_min + (tile_size * x),\n merc_max - (tile_size * (y + 1)),\n merc_min + (tile_size * (x + 1)),\n merc_max - (tile_size * y),\n 3857\n) FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;DROP FUNCTION IF EXISTS mercgrid;\n\n\nCREATE OR REPLACE FUNCTION ftime() RETURNS interval as $$\nSELECT age(clock_timestamp(), transaction_timestamp());\n$$ LANGUAGE SQL;\nSET SEARCH_PATH to pgstac, public;\n\nDROP FUNCTION IF EXISTS geometrysearch;\nCREATE OR REPLACE FUNCTION geometrysearch(\n IN geom geometry,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE, -- Return as soon as the passed in geometry is full covered\n IN skipcovered boolean DEFAULT TRUE -- Skip any items that would show up completely under the previous items\n) RETURNS jsonb AS $$\nDECLARE\n search searches%ROWTYPE;\n curs refcursor;\n _where text;\n query text;\n iter_record items%ROWTYPE;\n out_records jsonb := '{}'::jsonb[];\n exit_flag boolean := FALSE;\n counter int := 1;\n scancounter int := 1;\n remaining_limit int := _scanlimit;\n tilearea float;\n unionedgeom geometry;\n clippedgeom geometry;\n unionedgeom_area float := 0;\n prev_area float := 0;\n excludes text[];\n includes text[];\n\nBEGIN\n DROP TABLE IF EXISTS pgstac_results;\n CREATE TEMP TABLE pgstac_results (content jsonb) ON COMMIT DROP;\n\n -- If skipcovered is true then you will always want to exit when the passed in geometry is full\n IF skipcovered THEN\n exitwhenfull := TRUE;\n END IF;\n\n SELECT * INTO search FROM searches WHERE hash=queryhash;\n\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Search with Query Hash % Not Found', queryhash;\n END IF;\n\n tilearea := st_area(geom);\n _where := format('%s AND st_intersects(geometry, %L::geometry)', search._where, geom);\n\n\n FOR query IN SELECT * FROM partition_queries(_where, search.orderby) LOOP\n query := format('%s LIMIT %L', query, remaining_limit);\n RAISE NOTICE '%', query;\n OPEN curs FOR EXECUTE query;\n LOOP\n FETCH curs INTO iter_record;\n EXIT WHEN NOT FOUND;\n IF exitwhenfull OR skipcovered THEN -- If we are not using exitwhenfull or skipcovered, we do not need to do expensive geometry operations\n clippedgeom := st_intersection(geom, iter_record.geometry);\n\n IF unionedgeom IS NULL THEN\n unionedgeom := clippedgeom;\n ELSE\n unionedgeom := st_union(unionedgeom, clippedgeom);\n END IF;\n\n unionedgeom_area := st_area(unionedgeom);\n\n IF skipcovered AND prev_area = unionedgeom_area THEN\n scancounter := scancounter + 1;\n CONTINUE;\n END IF;\n\n prev_area := unionedgeom_area;\n\n RAISE NOTICE '% % % %', unionedgeom_area/tilearea, counter, scancounter, ftime();\n END IF;\n RAISE NOTICE '% %', iter_record, content_hydrate(iter_record, fields);\n INSERT INTO pgstac_results (content) VALUES (content_hydrate(iter_record, fields));\n\n IF counter >= _limit\n OR scancounter > _scanlimit\n OR ftime() > _timelimit\n OR (exitwhenfull AND unionedgeom_area >= tilearea)\n THEN\n exit_flag := TRUE;\n EXIT;\n END IF;\n counter := counter + 1;\n scancounter := scancounter + 1;\n\n END LOOP;\n CLOSE curs;\n EXIT WHEN exit_flag;\n remaining_limit := _scanlimit - scancounter;\n END LOOP;\n\n SELECT jsonb_agg(content) INTO out_records FROM pgstac_results WHERE content IS NOT NULL;\n\n RETURN jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb)\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDROP FUNCTION IF EXISTS geojsonsearch;\nCREATE OR REPLACE FUNCTION geojsonsearch(\n IN geojson jsonb,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_geomfromgeojson(geojson),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS xyzsearch;\nCREATE OR REPLACE FUNCTION xyzsearch(\n IN _x int,\n IN _y int,\n IN _z int,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_transform(tileenvelope(_z, _x, _y), 4326),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nGRANT ALL ON SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac to pgstac_admin;\n\n-- pgstac_read role limited to using function apis\nGRANT EXECUTE ON FUNCTION search TO pgstac_read;\nGRANT EXECUTE ON FUNCTION search_query TO pgstac_read;\nGRANT EXECUTE ON FUNCTION item_by_id TO pgstac_read;\nGRANT EXECUTE ON FUNCTION get_item TO pgstac_read;\n\nGRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_ingest;\nGRANT USAGE ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_ingest;\nSELECT set_version('0.6.3');\n" }, { "path": "src/pgstac/migrations/pgstac.0.6.4-0.6.5.sql", "content": "SET SEARCH_PATH to pgstac, public;\nset check_function_bodies = off;\n\nCREATE OR REPLACE FUNCTION pgstac.cql2_query(j jsonb, wrapper text DEFAULT NULL::text)\n RETURNS text\n LANGUAGE plpgsql\n STABLE\nAS $function$\n#variable_conflict use_variable\nDECLARE\n args jsonb := j->'args';\n arg jsonb;\n op text := lower(j->>'op');\n cql2op RECORD;\n literal text;\n _wrapper text;\nBEGIN\n IF j IS NULL OR (op IS NOT NULL AND args IS NULL) THEN\n RETURN NULL;\n END IF;\n RAISE NOTICE 'CQL2_QUERY: %', j;\n IF j ? 'filter' THEN\n RETURN cql2_query(j->'filter');\n END IF;\n\n IF j ? 'upper' THEN\n RETURN cql2_query(jsonb_build_object('op', 'upper', 'args', j->'upper'));\n END IF;\n\n IF j ? 'lower' THEN\n RETURN cql2_query(jsonb_build_object('op', 'lower', 'args', j->'lower'));\n END IF;\n\n -- Temporal Query\n IF op ilike 't_%' or op = 'anyinteracts' THEN\n RETURN temporal_op_query(op, args);\n END IF;\n\n -- If property is a timestamp convert it to text to use with\n -- general operators\n IF j ? 'timestamp' THEN\n RETURN format('%L::timestamptz', to_tstz(j->'timestamp'));\n END IF;\n IF j ? 'interval' THEN\n RAISE EXCEPTION 'Please use temporal operators when using intervals.';\n RETURN NONE;\n END IF;\n\n -- Spatial Query\n IF op ilike 's_%' or op = 'intersects' THEN\n RETURN spatial_op_query(op, args);\n END IF;\n\n\n IF op = 'in' THEN\n RAISE NOTICE 'IN : % % %', args, jsonb_build_array(args->0), args->1;\n args := jsonb_build_array(args->0) || (args->1);\n RAISE NOTICE 'IN2 : %', args;\n -- RETURN format(\n -- '%s = ANY (%L)',\n -- cql2_query(args->0),\n -- to_text_array(args->1)\n -- );\n END IF;\n\n\n\n IF op = 'between' THEN\n args = jsonb_build_array(\n args->0,\n args->1->0,\n args->1->1\n );\n END IF;\n\n -- Make sure that args is an array and run cql2_query on\n -- each element of the array\n RAISE NOTICE 'ARGS PRE: %', args;\n IF j ? 'args' THEN\n IF jsonb_typeof(args) != 'array' THEN\n args := jsonb_build_array(args);\n END IF;\n\n IF jsonb_path_exists(args, '$[*] ? (@.property == \"id\" || @.property == \"datetime\" || @.property == \"end_datetime\" || @.property == \"collection\")') THEN\n wrapper := NULL;\n ELSE\n -- if any of the arguments are a property, try to get the property_wrapper\n FOR arg IN SELECT jsonb_path_query(args, '$[*] ? (@.property != null)') LOOP\n RAISE NOTICE 'Arg: %', arg;\n SELECT property_wrapper INTO wrapper\n FROM queryables\n WHERE name=(arg->>'property')\n LIMIT 1;\n RAISE NOTICE 'Property: %, Wrapper: %', arg, wrapper;\n IF wrapper IS NOT NULL THEN\n EXIT;\n END IF;\n END LOOP;\n\n -- if the property was not in queryables, see if any args were numbers\n IF\n wrapper IS NULL\n AND jsonb_path_exists(args, '$[*] ? (@.type()==\"number\")')\n THEN\n wrapper := 'to_float';\n END IF;\n wrapper := coalesce(wrapper, 'to_text');\n END IF;\n\n SELECT jsonb_agg(cql2_query(a, wrapper))\n INTO args\n FROM jsonb_array_elements(args) a;\n END IF;\n RAISE NOTICE 'ARGS: %', args;\n\n IF op IN ('and', 'or') THEN\n RETURN\n format(\n '(%s)',\n array_to_string(to_text_array(args), format(' %s ', upper(op)))\n );\n END IF;\n\n IF op = 'in' THEN\n RAISE NOTICE 'IN -- % %', args->0, to_text(args->0);\n RETURN format(\n '%s IN (%s)',\n to_text(args->0),\n array_to_string((to_text_array(args))[2:], ',')\n );\n END IF;\n\n -- Look up template from cql2_ops\n IF j ? 'op' THEN\n SELECT * INTO cql2op FROM cql2_ops WHERE cql2_ops.op ilike op;\n IF FOUND THEN\n -- If specific index set in queryables for a property cast other arguments to that type\n\n RETURN format(\n cql2op.template,\n VARIADIC (to_text_array(args))\n );\n ELSE\n RAISE EXCEPTION 'Operator % Not Supported.', op;\n END IF;\n END IF;\n\n\n IF wrapper IS NOT NULL THEN\n RAISE NOTICE 'Wrapping % with %', j, wrapper;\n IF j ? 'property' THEN\n RETURN format('%I(%s)', wrapper, (queryable(j->>'property')).path);\n ELSE\n RETURN format('%I(%L)', wrapper, j);\n END IF;\n ELSIF j ? 'property' THEN\n RETURN quote_ident(j->>'property');\n END IF;\n\n RETURN quote_literal(to_text(j));\nEND;\n$function$\n;\n\n\n\nSELECT set_version('0.6.5');\n" }, { "path": "src/pgstac/migrations/pgstac.0.6.4.sql", "content": "CREATE EXTENSION IF NOT EXISTS postgis;\nCREATE EXTENSION IF NOT EXISTS btree_gist;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_admin;\n CREATE ROLE pgstac_read;\n CREATE ROLE pgstac_ingest;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nGRANT pgstac_admin TO current_user;\n\nCREATE SCHEMA IF NOT EXISTS pgstac AUTHORIZATION pgstac_admin;\n\nALTER ROLE pgstac_admin SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_read SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_ingest SET SEARCH_PATH TO pgstac, public;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\n\nGRANT pgstac_read TO pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\n\nSET ROLE pgstac_admin;\n\nSET SEARCH_PATH TO pgstac, public;\n\n\nCREATE TABLE IF NOT EXISTS migrations (\n version text PRIMARY KEY,\n datetime timestamptz DEFAULT clock_timestamp() NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION get_version() RETURNS text AS $$\n SELECT version FROM pgstac.migrations ORDER BY datetime DESC, version DESC LIMIT 1;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION set_version(text) RETURNS text AS $$\n INSERT INTO pgstac.migrations (version) VALUES ($1)\n ON CONFLICT DO NOTHING\n RETURNING version;\n$$ LANGUAGE SQL;\n\n\nCREATE TABLE IF NOT EXISTS pgstac_settings (\n name text PRIMARY KEY,\n value text NOT NULL\n);\n\nINSERT INTO pgstac_settings (name, value) VALUES\n ('context', 'off'),\n ('context_estimated_count', '100000'),\n ('context_estimated_cost', '100000'),\n ('context_stats_ttl', '1 day'),\n ('default-filter-lang', 'cql2-json'),\n ('additional_properties', 'true')\nON CONFLICT DO NOTHING\n;\n\n\nCREATE OR REPLACE FUNCTION get_setting(IN _setting text, IN conf jsonb DEFAULT NULL) RETURNS text AS $$\nSELECT COALESCE(\n conf->>_setting,\n current_setting(concat('pgstac.',_setting), TRUE),\n (SELECT value FROM pgstac.pgstac_settings WHERE name=_setting)\n);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context(conf jsonb DEFAULT NULL) RETURNS text AS $$\n SELECT pgstac.get_setting('context', conf);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context_estimated_count(conf jsonb DEFAULT NULL) RETURNS int AS $$\n SELECT pgstac.get_setting('context_estimated_count', conf)::int;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_estimated_cost();\nCREATE OR REPLACE FUNCTION context_estimated_cost(conf jsonb DEFAULT NULL) RETURNS float AS $$\n SELECT pgstac.get_setting('context_estimated_cost', conf)::float;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_stats_ttl();\nCREATE OR REPLACE FUNCTION context_stats_ttl(conf jsonb DEFAULT NULL) RETURNS interval AS $$\n SELECT pgstac.get_setting('context_stats_ttl', conf)::interval;\n$$ LANGUAGE SQL;\n\n\nCREATE OR REPLACE FUNCTION notice(VARIADIC text[]) RETURNS boolean AS $$\nDECLARE\ndebug boolean := current_setting('pgstac.debug', true);\nBEGIN\n IF debug THEN\n RAISE NOTICE 'NOTICE FROM FUNC: % >>>>> %', concat_ws(' | ', $1), clock_timestamp();\n RETURN TRUE;\n END IF;\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION empty_arr(ANYARRAY) RETURNS BOOLEAN AS $$\nSELECT CASE\n WHEN $1 IS NULL THEN TRUE\n WHEN cardinality($1)<1 THEN TRUE\nELSE FALSE\nEND;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_intersection(_a ANYARRAY, _b ANYARRAY) RETURNS ANYARRAY AS $$\n SELECT ARRAY ( SELECT unnest(_a) INTERSECT SELECT UNNEST(_b) );\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION array_map_ident(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_ident(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_map_literal(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_literal(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION array_reverse(anyarray) RETURNS anyarray AS $$\nSELECT ARRAY(\n SELECT $1[i]\n FROM generate_subscripts($1,1) AS s(i)\n ORDER BY i DESC\n);\n$$ LANGUAGE SQL STRICT IMMUTABLE;\nCREATE OR REPLACE FUNCTION to_int(jsonb) RETURNS int AS $$\n SELECT floor(($1->>0)::float)::int;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION to_float(jsonb) RETURNS float AS $$\n SELECT ($1->>0)::float;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION to_tstz(jsonb) RETURNS timestamptz AS $$\n SELECT ($1->>0)::timestamptz;\n$$ LANGUAGE SQL IMMUTABLE STRICT SET TIME ZONE 'UTC';\n\n\nCREATE OR REPLACE FUNCTION to_text(jsonb) RETURNS text AS $$\n SELECT CASE WHEN jsonb_typeof($1) IN ('array','object') THEN $1::text ELSE $1->>0 END;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION to_text_array(jsonb) RETURNS text[] AS $$\n SELECT\n CASE jsonb_typeof($1)\n WHEN 'array' THEN ARRAY(SELECT jsonb_array_elements_text($1))\n ELSE ARRAY[$1->>0]\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\nCREATE OR REPLACE FUNCTION bbox_geom(_bbox jsonb) RETURNS geometry AS $$\nSELECT CASE jsonb_array_length(_bbox)\n WHEN 4 THEN\n ST_SetSRID(ST_MakeEnvelope(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float,\n (_bbox->>3)::float\n ),4326)\n WHEN 6 THEN\n ST_SetSRID(ST_3DMakeBox(\n ST_MakePoint(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float\n ),\n ST_MakePoint(\n (_bbox->>3)::float,\n (_bbox->>4)::float,\n (_bbox->>5)::float\n )\n ),4326)\n ELSE null END;\n;\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION geom_bbox(_geom geometry) RETURNS jsonb AS $$\n SELECT jsonb_build_array(\n st_xmin(_geom),\n st_ymin(_geom),\n st_xmax(_geom),\n st_ymax(_geom)\n );\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION flip_jsonb_array(j jsonb) RETURNS jsonb AS $$\n SELECT jsonb_agg(value) FROM (SELECT value FROM jsonb_array_elements(j) WITH ORDINALITY ORDER BY ordinality DESC) as t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION explode_dotpaths(j jsonb) RETURNS SETOF text[] AS $$\n SELECT string_to_array(p, '.') as e FROM jsonb_array_elements_text(j) p;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION explode_dotpaths_recurse(IN j jsonb) RETURNS SETOF text[] AS $$\n WITH RECURSIVE t AS (\n SELECT e FROM explode_dotpaths(j) e\n UNION ALL\n SELECT e[1:cardinality(e)-1]\n FROM t\n WHERE cardinality(e)>1\n ) SELECT e FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION jsonb_set_nested(j jsonb, path text[], val jsonb) RETURNS jsonb AS $$\nDECLARE\nBEGIN\n IF cardinality(path) > 1 THEN\n FOR i IN 1..(cardinality(path)-1) LOOP\n IF j #> path[:i] IS NULL THEN\n j := jsonb_set_lax(j, path[:i], '{}', TRUE);\n END IF;\n END LOOP;\n END IF;\n RETURN jsonb_set_lax(j, path, val, true);\n\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\n\n\nCREATE OR REPLACE FUNCTION jsonb_include(j jsonb, f jsonb) RETURNS jsonb AS $$\nDECLARE\n includes jsonb := f-> 'include';\n outj jsonb := '{}'::jsonb;\n path text[];\nBEGIN\n IF\n includes IS NULL\n OR jsonb_array_length(includes) = 0\n THEN\n RETURN j;\n ELSE\n includes := includes || '[\"id\",\"collection\"]'::jsonb;\n FOR path IN SELECT explode_dotpaths(includes) LOOP\n outj := jsonb_set_nested(outj, path, j #> path);\n END LOOP;\n END IF;\n RETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_exclude(j jsonb, f jsonb) RETURNS jsonb AS $$\nDECLARE\n excludes jsonb := f-> 'exclude';\n outj jsonb := j;\n path text[];\nBEGIN\n IF\n excludes IS NULL\n OR jsonb_array_length(excludes) = 0\n THEN\n RETURN j;\n ELSE\n FOR path IN SELECT explode_dotpaths(excludes) LOOP\n outj := outj #- path;\n END LOOP;\n END IF;\n RETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_fields(j jsonb, f jsonb DEFAULT '{\"fields\":[]}') RETURNS jsonb AS $$\n SELECT jsonb_exclude(jsonb_include(j, f), f);\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION merge_jsonb(_a jsonb, _b jsonb) RETURNS jsonb AS $$\n SELECT\n CASE\n WHEN _a = '\"𒍟※\"'::jsonb THEN NULL\n WHEN _a IS NULL OR jsonb_typeof(_a) = 'null' THEN _b\n WHEN jsonb_typeof(_a) = 'object' AND jsonb_typeof(_b) = 'object' THEN\n (\n SELECT\n jsonb_strip_nulls(\n jsonb_object_agg(\n key,\n merge_jsonb(a.value, b.value)\n )\n )\n FROM\n jsonb_each(coalesce(_a,'{}'::jsonb)) as a\n FULL JOIN\n jsonb_each(coalesce(_b,'{}'::jsonb)) as b\n USING (key)\n )\n WHEN\n jsonb_typeof(_a) = 'array'\n AND jsonb_typeof(_b) = 'array'\n AND jsonb_array_length(_a) = jsonb_array_length(_b)\n THEN\n (\n SELECT jsonb_agg(m) FROM\n ( SELECT\n merge_jsonb(\n jsonb_array_elements(_a),\n jsonb_array_elements(_b)\n ) as m\n ) as l\n )\n ELSE _a\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION strip_jsonb(_a jsonb, _b jsonb) RETURNS jsonb AS $$\n SELECT\n CASE\n\n WHEN (_a IS NULL OR jsonb_typeof(_a) = 'null') AND _b IS NOT NULL AND jsonb_typeof(_b) != 'null' THEN '\"𒍟※\"'::jsonb\n WHEN _b IS NULL OR jsonb_typeof(_a) = 'null' THEN _a\n WHEN _a = _b AND jsonb_typeof(_a) = 'object' THEN '{}'::jsonb\n WHEN _a = _b THEN NULL\n WHEN jsonb_typeof(_a) = 'object' AND jsonb_typeof(_b) = 'object' THEN\n (\n SELECT\n jsonb_strip_nulls(\n jsonb_object_agg(\n key,\n strip_jsonb(a.value, b.value)\n )\n )\n FROM\n jsonb_each(_a) as a\n FULL JOIN\n jsonb_each(_b) as b\n USING (key)\n )\n WHEN\n jsonb_typeof(_a) = 'array'\n AND jsonb_typeof(_b) = 'array'\n AND jsonb_array_length(_a) = jsonb_array_length(_b)\n THEN\n (\n SELECT jsonb_agg(m) FROM\n ( SELECT\n strip_jsonb(\n jsonb_array_elements(_a),\n jsonb_array_elements(_b)\n ) as m\n ) as l\n )\n ELSE _a\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE;\n/* looks for a geometry in a stac item first from geometry and falling back to bbox */\nCREATE OR REPLACE FUNCTION stac_geom(value jsonb) RETURNS geometry AS $$\nSELECT\n CASE\n WHEN value ? 'intersects' THEN\n ST_GeomFromGeoJSON(value->>'intersects')\n WHEN value ? 'geometry' THEN\n ST_GeomFromGeoJSON(value->>'geometry')\n WHEN value ? 'bbox' THEN\n pgstac.bbox_geom(value->'bbox')\n ELSE NULL\n END as geometry\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION stac_daterange(\n value jsonb\n) RETURNS tstzrange AS $$\nDECLARE\n props jsonb := value;\n dt timestamptz;\n edt timestamptz;\nBEGIN\n IF props ? 'properties' THEN\n props := props->'properties';\n END IF;\n IF\n props ? 'start_datetime'\n AND props->>'start_datetime' IS NOT NULL\n AND props ? 'end_datetime'\n AND props->>'end_datetime' IS NOT NULL\n THEN\n dt := props->>'start_datetime';\n edt := props->>'end_datetime';\n IF dt > edt THEN\n RAISE EXCEPTION 'start_datetime must be < end_datetime';\n END IF;\n ELSE\n dt := props->>'datetime';\n edt := props->>'datetime';\n END IF;\n IF dt is NULL OR edt IS NULL THEN\n RAISE NOTICE 'DT: %, EDT: %', dt, edt;\n RAISE EXCEPTION 'Either datetime (%) or both start_datetime (%) and end_datetime (%) must be set.', props->>'datetime',props->>'start_datetime',props->>'end_datetime';\n END IF;\n RETURN tstzrange(dt, edt, '[]');\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_datetime(value jsonb) RETURNS timestamptz AS $$\n SELECT lower(stac_daterange(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_end_datetime(value jsonb) RETURNS timestamptz AS $$\n SELECT upper(stac_daterange(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\n\nCREATE TABLE IF NOT EXISTS stac_extensions(\n name text PRIMARY KEY,\n url text,\n enbabled_by_default boolean NOT NULL DEFAULT TRUE,\n enableable boolean NOT NULL DEFAULT TRUE\n);\n\nINSERT INTO stac_extensions (name, url) VALUES\n ('fields', 'https://api.stacspec.org/v1.0.0-beta.5/item-search#fields'),\n ('sort','https://api.stacspec.org/v1.0.0-beta.5/item-search#sort'),\n ('context','https://api.stacspec.org/v1.0.0-beta.5/item-search#context'),\n ('filter', 'https://api.stacspec.org/v1.0.0-beta.5/item-search#filter'),\n ('query', 'https://api.stacspec.org/v1.0.0-beta.5/item-search#query')\nON CONFLICT (name) DO UPDATE SET url=EXCLUDED.url;\n\n\n\nCREATE OR REPLACE FUNCTION collection_base_item(content jsonb) RETURNS jsonb AS $$\n SELECT jsonb_build_object(\n 'type', 'Feature',\n 'stac_version', content->'stac_version',\n 'assets', content->'item_assets',\n 'collection', content->'id'\n );\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TYPE partition_trunc_strategy AS ENUM ('year', 'month');\n\nCREATE TABLE IF NOT EXISTS collections (\n key bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,\n id text GENERATED ALWAYS AS (content->>'id') STORED UNIQUE,\n content JSONB NOT NULL,\n base_item jsonb GENERATED ALWAYS AS (pgstac.collection_base_item(content)) STORED,\n partition_trunc partition_trunc_strategy\n);\n\n\nCREATE OR REPLACE FUNCTION collection_base_item(cid text) RETURNS jsonb AS $$\n SELECT pgstac.collection_base_item(content) FROM pgstac.collections WHERE id = cid LIMIT 1;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION table_empty(text) RETURNS boolean AS $$\nDECLARE\n retval boolean;\nBEGIN\n EXECUTE format($q$\n SELECT NOT EXISTS (SELECT 1 FROM %I LIMIT 1)\n $q$,\n $1\n ) INTO retval;\n RETURN retval;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION collections_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n q text;\n partition_name text := format('_items_%s', NEW.key);\n partition_exists boolean := false;\n partition_empty boolean := true;\n err_context text;\n loadtemp boolean := FALSE;\nBEGIN\n RAISE NOTICE 'Collection Trigger. % %', NEW.id, NEW.key;\n SELECT relid::text INTO partition_name\n FROM pg_partition_tree('items')\n WHERE relid::text = partition_name;\n IF FOUND THEN\n partition_exists := true;\n partition_empty := table_empty(partition_name);\n ELSE\n partition_exists := false;\n partition_empty := true;\n partition_name := format('_items_%s', NEW.key);\n END IF;\n IF TG_OP = 'UPDATE' AND NEW.partition_trunc IS DISTINCT FROM OLD.partition_trunc AND partition_empty THEN\n q := format($q$\n DROP TABLE IF EXISTS %I CASCADE;\n $q$,\n partition_name\n );\n EXECUTE q;\n END IF;\n IF TG_OP = 'UPDATE' AND NEW.partition_trunc IS DISTINCT FROM OLD.partition_trunc AND partition_exists AND NOT partition_empty THEN\n q := format($q$\n CREATE TEMP TABLE changepartitionstaging ON COMMIT DROP AS SELECT * FROM %I;\n DROP TABLE IF EXISTS %I CASCADE;\n $q$,\n partition_name,\n partition_name\n );\n EXECUTE q;\n loadtemp := TRUE;\n partition_empty := TRUE;\n partition_exists := FALSE;\n END IF;\n IF TG_OP = 'UPDATE' AND NEW.partition_trunc IS NOT DISTINCT FROM OLD.partition_trunc THEN\n RETURN NEW;\n END IF;\n IF NEW.partition_trunc IS NULL AND partition_empty THEN\n RAISE NOTICE '% % % %',\n partition_name,\n NEW.id,\n concat(partition_name,'_id_idx'),\n partition_name\n ;\n q := format($q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n $q$,\n partition_name,\n NEW.id,\n concat(partition_name,'_id_idx'),\n partition_name\n );\n RAISE NOTICE 'q: %', q;\n BEGIN\n EXECUTE q;\n EXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', partition_name;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\n END;\n\n ALTER TABLE partitions DISABLE TRIGGER partitions_delete_trigger;\n DELETE FROM partitions WHERE collection=NEW.id AND name=partition_name;\n ALTER TABLE partitions ENABLE TRIGGER partitions_delete_trigger;\n\n INSERT INTO partitions (collection, name) VALUES (NEW.id, partition_name);\n ELSIF partition_empty THEN\n q := format($q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L)\n PARTITION BY RANGE (datetime);\n $q$,\n partition_name,\n NEW.id\n );\n RAISE NOTICE 'q: %', q;\n BEGIN\n EXECUTE q;\n EXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', partition_name;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\n END;\n ALTER TABLE partitions DISABLE TRIGGER partitions_delete_trigger;\n DELETE FROM partitions WHERE collection=NEW.id AND name=partition_name;\n ALTER TABLE partitions ENABLE TRIGGER partitions_delete_trigger;\n ELSE\n RAISE EXCEPTION 'Cannot modify partition % unless empty', partition_name;\n END IF;\n IF loadtemp THEN\n RAISE NOTICE 'Moving data into new partitions.';\n q := format($q$\n WITH p AS (\n SELECT\n collection,\n datetime as datetime,\n end_datetime as end_datetime,\n (partition_name(\n collection,\n datetime\n )).partition_name as name\n FROM changepartitionstaging\n )\n INSERT INTO partitions (collection, datetime_range, end_datetime_range)\n SELECT\n collection,\n tstzrange(min(datetime), max(datetime), '[]') as datetime_range,\n tstzrange(min(end_datetime), max(end_datetime), '[]') as end_datetime_range\n FROM p\n GROUP BY collection, name\n ON CONFLICT (name) DO UPDATE SET\n datetime_range = EXCLUDED.datetime_range,\n end_datetime_range = EXCLUDED.end_datetime_range\n ;\n INSERT INTO %I SELECT * FROM changepartitionstaging;\n DROP TABLE IF EXISTS changepartitionstaging;\n $q$,\n partition_name\n );\n EXECUTE q;\n END IF;\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE TRIGGER collections_trigger AFTER INSERT OR UPDATE ON collections FOR EACH ROW\nEXECUTE FUNCTION collections_trigger_func();\n\nCREATE OR REPLACE FUNCTION partition_collection(collection text, strategy partition_trunc_strategy) RETURNS text AS $$\n UPDATE collections SET partition_trunc=strategy WHERE id=collection RETURNING partition_trunc;\n$$ LANGUAGE SQL;\n\nCREATE TABLE IF NOT EXISTS partitions (\n collection text REFERENCES collections(id) ON DELETE CASCADE,\n name text PRIMARY KEY,\n partition_range tstzrange NOT NULL DEFAULT tstzrange('-infinity'::timestamptz,'infinity'::timestamptz, '[]'),\n datetime_range tstzrange,\n end_datetime_range tstzrange,\n CONSTRAINT prange EXCLUDE USING GIST (\n collection WITH =,\n partition_range WITH &&\n )\n) WITH (FILLFACTOR=90);\nCREATE INDEX partitions_range_idx ON partitions USING GIST(partition_range);\n\nCREATE OR REPLACE FUNCTION partitions_delete_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n q text;\nBEGIN\n RAISE NOTICE 'Partition Delete Trigger. %', OLD.name;\n EXECUTE format($q$\n DROP TABLE IF EXISTS %I CASCADE;\n $q$,\n OLD.name\n );\n RAISE NOTICE 'Dropped partition.';\n RETURN OLD;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER partitions_delete_trigger BEFORE DELETE ON partitions FOR EACH ROW\nEXECUTE FUNCTION partitions_delete_trigger_func();\n\nCREATE OR REPLACE FUNCTION partition_name(\n IN collection text,\n IN dt timestamptz,\n OUT partition_name text,\n OUT partition_range tstzrange\n) AS $$\nDECLARE\n c RECORD;\n parent_name text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n parent_name := format('_items_%s', c.key);\n\n\n IF c.partition_trunc = 'year' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYY'));\n ELSIF c.partition_trunc = 'month' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYYMM'));\n ELSE\n partition_name := parent_name;\n partition_range := tstzrange('-infinity'::timestamptz, 'infinity'::timestamptz, '[]');\n END IF;\n IF partition_range IS NULL THEN\n partition_range := tstzrange(\n date_trunc(c.partition_trunc::text, dt),\n date_trunc(c.partition_trunc::text, dt) + concat('1 ', c.partition_trunc)::interval\n );\n END IF;\n RETURN;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\n\nCREATE OR REPLACE FUNCTION partitions_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n q text;\n cq text;\n parent_name text;\n partition_trunc text;\n partition_name text := NEW.name;\n partition_exists boolean := false;\n partition_empty boolean := true;\n partition_range tstzrange;\n datetime_range tstzrange;\n end_datetime_range tstzrange;\n err_context text;\n mindt timestamptz := lower(NEW.datetime_range);\n maxdt timestamptz := upper(NEW.datetime_range);\n minedt timestamptz := lower(NEW.end_datetime_range);\n maxedt timestamptz := upper(NEW.end_datetime_range);\n t_mindt timestamptz;\n t_maxdt timestamptz;\n t_minedt timestamptz;\n t_maxedt timestamptz;\nBEGIN\n RAISE NOTICE 'Partitions Trigger. %', NEW;\n datetime_range := NEW.datetime_range;\n end_datetime_range := NEW.end_datetime_range;\n\n SELECT\n format('_items_%s', key),\n c.partition_trunc::text\n INTO\n parent_name,\n partition_trunc\n FROM pgstac.collections c\n WHERE c.id = NEW.collection;\n SELECT (pgstac.partition_name(NEW.collection, mindt)).* INTO partition_name, partition_range;\n NEW.name := partition_name;\n\n IF partition_range IS NULL OR partition_range = 'empty'::tstzrange THEN\n partition_range := tstzrange('-infinity'::timestamptz, 'infinity'::timestamptz, '[]');\n END IF;\n\n NEW.partition_range := partition_range;\n IF TG_OP = 'UPDATE' THEN\n mindt := least(mindt, lower(OLD.datetime_range));\n maxdt := greatest(maxdt, upper(OLD.datetime_range));\n minedt := least(minedt, lower(OLD.end_datetime_range));\n maxedt := greatest(maxedt, upper(OLD.end_datetime_range));\n NEW.datetime_range := tstzrange(mindt, maxdt, '[]');\n NEW.end_datetime_range := tstzrange(minedt, maxedt, '[]');\n END IF;\n IF TG_OP = 'INSERT' THEN\n\n IF partition_range != tstzrange('-infinity'::timestamptz, 'infinity'::timestamptz, '[]') THEN\n\n RAISE NOTICE '% % %', partition_name, parent_name, partition_range;\n q := format($q$\n CREATE TABLE IF NOT EXISTS %I partition OF %I FOR VALUES FROM (%L) TO (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n $q$,\n partition_name,\n parent_name,\n lower(partition_range),\n upper(partition_range),\n format('%s_pkey', partition_name),\n partition_name\n );\n BEGIN\n EXECUTE q;\n EXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', partition_name;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\n END;\n END IF;\n\n END IF;\n\n -- Update constraints\n EXECUTE format($q$\n SELECT\n min(datetime),\n max(datetime),\n min(end_datetime),\n max(end_datetime)\n FROM %I;\n $q$, partition_name)\n INTO t_mindt, t_maxdt, t_minedt, t_maxedt;\n mindt := least(mindt, t_mindt);\n maxdt := greatest(maxdt, t_maxdt);\n minedt := least(mindt, minedt, t_minedt);\n maxedt := greatest(maxdt, maxedt, t_maxedt);\n\n mindt := date_trunc(coalesce(partition_trunc, 'year'), mindt);\n maxdt := date_trunc(coalesce(partition_trunc, 'year'), maxdt - '1 second'::interval) + concat('1 ',coalesce(partition_trunc, 'year'))::interval;\n minedt := date_trunc(coalesce(partition_trunc, 'year'), minedt);\n maxedt := date_trunc(coalesce(partition_trunc, 'year'), maxedt - '1 second'::interval) + concat('1 ',coalesce(partition_trunc, 'year'))::interval;\n\n\n IF mindt IS NOT NULL AND maxdt IS NOT NULL AND minedt IS NOT NULL AND maxedt IS NOT NULL THEN\n NEW.datetime_range := tstzrange(mindt, maxdt, '[]');\n NEW.end_datetime_range := tstzrange(minedt, maxedt, '[]');\n IF\n TG_OP='UPDATE'\n AND OLD.datetime_range @> NEW.datetime_range\n AND OLD.end_datetime_range @> NEW.end_datetime_range\n THEN\n RAISE NOTICE 'Range unchanged, not updating constraints.';\n ELSE\n\n RAISE NOTICE '\n SETTING CONSTRAINTS\n mindt: %, maxdt: %\n minedt: %, maxedt: %\n ', mindt, maxdt, minedt, maxedt;\n IF partition_trunc IS NULL THEN\n cq := format($q$\n ALTER TABLE %7$I\n DROP CONSTRAINT IF EXISTS %1$I,\n DROP CONSTRAINT IF EXISTS %2$I,\n ADD CONSTRAINT %1$I\n CHECK (\n (datetime >= %3$L)\n AND (datetime <= %4$L)\n AND (end_datetime >= %5$L)\n AND (end_datetime <= %6$L)\n ) NOT VALID\n ;\n ALTER TABLE %7$I\n VALIDATE CONSTRAINT %1$I;\n $q$,\n format('%s_dt', partition_name),\n format('%s_edt', partition_name),\n mindt,\n maxdt,\n minedt,\n maxedt,\n partition_name\n );\n ELSE\n cq := format($q$\n ALTER TABLE %5$I\n DROP CONSTRAINT IF EXISTS %1$I,\n DROP CONSTRAINT IF EXISTS %2$I,\n ADD CONSTRAINT %2$I\n CHECK ((end_datetime >= %3$L) AND (end_datetime <= %4$L)) NOT VALID\n ;\n ALTER TABLE %5$I\n VALIDATE CONSTRAINT %2$I;\n $q$,\n format('%s_dt', partition_name),\n format('%s_edt', partition_name),\n minedt,\n maxedt,\n partition_name\n );\n\n END IF;\n RAISE NOTICE 'Altering Constraints. %', cq;\n EXECUTE cq;\n END IF;\n ELSE\n NEW.datetime_range = NULL;\n NEW.end_datetime_range = NULL;\n\n cq := format($q$\n ALTER TABLE %3$I\n DROP CONSTRAINT IF EXISTS %1$I,\n DROP CONSTRAINT IF EXISTS %2$I,\n ADD CONSTRAINT %1$I\n CHECK ((datetime IS NULL AND end_datetime IS NULL)) NOT VALID\n ;\n ALTER TABLE %3$I\n VALIDATE CONSTRAINT %1$I;\n $q$,\n format('%s_dt', partition_name),\n format('%s_edt', partition_name),\n partition_name\n );\n EXECUTE cq;\n END IF;\n\n RETURN NEW;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER partitions_trigger BEFORE INSERT OR UPDATE ON partitions FOR EACH ROW\nEXECUTE FUNCTION partitions_trigger_func();\n\n\nCREATE OR REPLACE FUNCTION create_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection(data jsonb) RETURNS VOID AS $$\nDECLARE\n out collections%ROWTYPE;\nBEGIN\n UPDATE collections SET content=data WHERE id = data->>'id' RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ON CONFLICT (id) DO\n UPDATE\n SET content=EXCLUDED.content\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION delete_collection(_id text) RETURNS VOID AS $$\nDECLARE\n out collections%ROWTYPE;\nBEGIN\n DELETE FROM collections WHERE id = _id RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION get_collection(id text) RETURNS jsonb AS $$\n SELECT content FROM collections\n WHERE id=$1\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION all_collections() RETURNS jsonb AS $$\n SELECT jsonb_agg(content) FROM collections;\n;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\nCREATE TABLE queryables (\n id bigint GENERATED ALWAYS AS identity PRIMARY KEY,\n name text UNIQUE NOT NULL,\n collection_ids text[], -- used to determine what partitions to create indexes on\n definition jsonb,\n property_path text,\n property_wrapper text,\n property_index_type text\n);\nCREATE INDEX queryables_name_idx ON queryables (name);\nCREATE INDEX queryables_property_wrapper_idx ON queryables (property_wrapper);\n\n\nINSERT INTO queryables (name, definition) VALUES\n('id', '{\"title\": \"Item ID\",\"description\": \"Item identifier\",\"$ref\": \"https://schemas.stacspec.org/v1.0.0/item-spec/json-schema/item.json#/definitions/core/allOf/2/properties/id\"}'),\n('datetime','{\"description\": \"Datetime\",\"type\": \"string\",\"title\": \"Acquired\",\"format\": \"date-time\",\"pattern\": \"(\\\\+00:00|Z)$\"}')\nON CONFLICT DO NOTHING;\n\n\n\nINSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n('eo:cloud_cover','{\"$ref\": \"https://stac-extensions.github.io/eo/v1.0.0/schema.json#/definitions/fieldsproperties/eo:cloud_cover\"}','to_int','BTREE')\nON CONFLICT DO NOTHING;\n\nCREATE OR REPLACE FUNCTION array_to_path(arr text[]) RETURNS text AS $$\n SELECT string_agg(\n quote_literal(v),\n '->'\n ) FROM unnest(arr) v;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\n\n\nCREATE OR REPLACE FUNCTION queryable(\n IN dotpath text,\n OUT path text,\n OUT expression text,\n OUT wrapper text\n) AS $$\nDECLARE\n q RECORD;\n path_elements text[];\nBEGIN\n IF dotpath IN ('id', 'geometry', 'datetime', 'end_datetime', 'collection') THEN\n path := dotpath;\n expression := dotpath;\n wrapper := NULL;\n RETURN;\n END IF;\n SELECT * INTO q FROM queryables WHERE name=dotpath;\n IF q.property_wrapper IS NULL THEN\n IF q.definition->>'type' = 'number' THEN\n wrapper := 'to_float';\n ELSIF q.definition->>'format' = 'date-time' THEN\n wrapper := 'to_tstz';\n ELSE\n wrapper := 'to_text';\n END IF;\n ELSE\n wrapper := q.property_wrapper;\n END IF;\n IF q.property_path IS NOT NULL THEN\n path := q.property_path;\n ELSE\n path_elements := string_to_array(dotpath, '.');\n IF path_elements[1] IN ('links', 'assets', 'stac_version', 'stac_extensions') THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSIF path_elements[1] = 'properties' THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSE\n path := format($F$content->'properties'->%s$F$, array_to_path(path_elements));\n END IF;\n END IF;\n expression := format('%I(%s)', wrapper, path);\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\nCREATE OR REPLACE FUNCTION create_queryable_indexes() RETURNS VOID AS $$\nDECLARE\n queryable RECORD;\n q text;\nBEGIN\n FOR queryable IN\n SELECT\n queryables.id as qid,\n CASE WHEN collections.key IS NULL THEN 'items' ELSE format('_items_%s',collections.key) END AS part,\n property_index_type,\n expression\n FROM\n queryables\n LEFT JOIN collections ON (collections.id = ANY (queryables.collection_ids))\n JOIN LATERAL queryable(queryables.name) ON (queryables.property_index_type IS NOT NULL)\n LOOP\n q := format(\n $q$\n CREATE INDEX IF NOT EXISTS %I ON %I USING %s ((%s));\n $q$,\n format('%s_%s_idx', queryable.part, queryable.qid),\n queryable.part,\n COALESCE(queryable.property_index_type, 'to_text'),\n queryable.expression\n );\n RAISE NOTICE '%',q;\n EXECUTE q;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION queryables_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\nBEGIN\nPERFORM create_queryable_indexes();\nRETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER queryables_trigger AFTER INSERT OR UPDATE ON queryables\nFOR EACH STATEMENT EXECUTE PROCEDURE queryables_trigger_func();\n\nCREATE TRIGGER queryables_collection_trigger AFTER INSERT OR UPDATE ON collections\nFOR EACH STATEMENT EXECUTE PROCEDURE queryables_trigger_func();\nCREATE OR REPLACE FUNCTION parse_dtrange(\n _indate jsonb,\n relative_base timestamptz DEFAULT date_trunc('hour', CURRENT_TIMESTAMP)\n) RETURNS tstzrange AS $$\nDECLARE\n timestrs text[];\n s timestamptz;\n e timestamptz;\nBEGIN\n timestrs :=\n CASE\n WHEN _indate ? 'timestamp' THEN\n ARRAY[_indate->>'timestamp']\n WHEN _indate ? 'interval' THEN\n to_text_array(_indate->'interval')\n WHEN jsonb_typeof(_indate) = 'array' THEN\n to_text_array(_indate)\n ELSE\n regexp_split_to_array(\n _indate->>0,\n '/'\n )\n END;\n RAISE NOTICE 'TIMESTRS %', timestrs;\n IF cardinality(timestrs) = 1 THEN\n IF timestrs[1] ILIKE 'P%' THEN\n RETURN tstzrange(relative_base - upper(timestrs[1])::interval, relative_base, '[)');\n END IF;\n s := timestrs[1]::timestamptz;\n RETURN tstzrange(s, s, '[]');\n END IF;\n\n IF cardinality(timestrs) != 2 THEN\n RAISE EXCEPTION 'Timestamp cannot have more than 2 values';\n END IF;\n\n IF timestrs[1] = '..' THEN\n s := '-infinity'::timestamptz;\n e := timestrs[2]::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] = '..' THEN\n s := timestrs[1]::timestamptz;\n e := 'infinity'::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[1] ILIKE 'P%' AND timestrs[2] NOT ILIKE 'P%' THEN\n e := timestrs[2]::timestamptz;\n s := e - upper(timestrs[1])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] ILIKE 'P%' AND timestrs[1] NOT ILIKE 'P%' THEN\n s := timestrs[1]::timestamptz;\n e := s + upper(timestrs[2])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n s := timestrs[1]::timestamptz;\n e := timestrs[2]::timestamptz;\n\n RETURN tstzrange(s,e,'[)');\n\n RETURN NULL;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT PARALLEL SAFE SET TIME ZONE 'UTC';\n\nCREATE OR REPLACE FUNCTION parse_dtrange(\n _indate text,\n relative_base timestamptz DEFAULT CURRENT_TIMESTAMP\n) RETURNS tstzrange AS $$\n SELECT parse_dtrange(to_jsonb(_indate), relative_base);\n$$ LANGUAGE SQL STABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION temporal_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\n ll text := 'datetime';\n lh text := 'end_datetime';\n rrange tstzrange;\n rl text;\n rh text;\n outq text;\nBEGIN\n rrange := parse_dtrange(args->1);\n RAISE NOTICE 'Constructing temporal query OP: %, ARGS: %, RRANGE: %', op, args, rrange;\n op := lower(op);\n rl := format('%L::timestamptz', lower(rrange));\n rh := format('%L::timestamptz', upper(rrange));\n outq := CASE op\n WHEN 't_before' THEN 'lh < rl'\n WHEN 't_after' THEN 'll > rh'\n WHEN 't_meets' THEN 'lh = rl'\n WHEN 't_metby' THEN 'll = rh'\n WHEN 't_overlaps' THEN 'll < rl AND rl < lh < rh'\n WHEN 't_overlappedby' THEN 'rl < ll < rh AND lh > rh'\n WHEN 't_starts' THEN 'll = rl AND lh < rh'\n WHEN 't_startedby' THEN 'll = rl AND lh > rh'\n WHEN 't_during' THEN 'll > rl AND lh < rh'\n WHEN 't_contains' THEN 'll < rl AND lh > rh'\n WHEN 't_finishes' THEN 'll > rl AND lh = rh'\n WHEN 't_finishedby' THEN 'll < rl AND lh = rh'\n WHEN 't_equals' THEN 'll = rl AND lh = rh'\n WHEN 't_disjoint' THEN 'NOT (ll <= rh AND lh >= rl)'\n WHEN 't_intersects' THEN 'll <= rh AND lh >= rl'\n WHEN 'anyinteracts' THEN 'll <= rh AND lh >= rl'\n END;\n outq := regexp_replace(outq, '\\mll\\M', ll);\n outq := regexp_replace(outq, '\\mlh\\M', lh);\n outq := regexp_replace(outq, '\\mrl\\M', rl);\n outq := regexp_replace(outq, '\\mrh\\M', rh);\n outq := format('(%s)', outq);\n RETURN outq;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\n\n\nCREATE OR REPLACE FUNCTION spatial_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\n geom text;\n j jsonb := args->1;\nBEGIN\n op := lower(op);\n RAISE NOTICE 'Constructing spatial query OP: %, ARGS: %', op, args;\n IF op NOT IN ('s_equals','s_disjoint','s_touches','s_within','s_overlaps','s_crosses','s_intersects','intersects','s_contains') THEN\n RAISE EXCEPTION 'Spatial Operator % Not Supported', op;\n END IF;\n op := regexp_replace(op, '^s_', 'st_');\n IF op = 'intersects' THEN\n op := 'st_intersects';\n END IF;\n -- Convert geometry to WKB string\n IF j ? 'type' AND j ? 'coordinates' THEN\n geom := st_geomfromgeojson(j)::text;\n ELSIF jsonb_typeof(j) = 'array' THEN\n geom := bbox_geom(j)::text;\n END IF;\n\n RETURN format('%s(geometry, %L::geometry)', op, geom);\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION query_to_cql2(q jsonb) RETURNS jsonb AS $$\n-- Translates anything passed in through the deprecated \"query\" into equivalent CQL2\nWITH t AS (\n SELECT key as property, value as ops\n FROM jsonb_each(q)\n), t2 AS (\n SELECT property, (jsonb_each(ops)).*\n FROM t WHERE jsonb_typeof(ops) = 'object'\n UNION ALL\n SELECT property, 'eq', ops\n FROM t WHERE jsonb_typeof(ops) != 'object'\n)\nSELECT\n jsonb_strip_nulls(jsonb_build_object(\n 'op', 'and',\n 'args', jsonb_agg(\n jsonb_build_object(\n 'op', key,\n 'args', jsonb_build_array(\n jsonb_build_object('property',property),\n value\n )\n )\n )\n )\n) as qcql FROM t2\n;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\nCREATE OR REPLACE FUNCTION cql1_to_cql2(j jsonb) RETURNS jsonb AS $$\nDECLARE\n args jsonb;\n ret jsonb;\nBEGIN\n RAISE NOTICE 'CQL1_TO_CQL2: %', j;\n IF j ? 'filter' THEN\n RETURN cql1_to_cql2(j->'filter');\n END IF;\n IF j ? 'property' THEN\n RETURN j;\n END IF;\n IF jsonb_typeof(j) = 'array' THEN\n SELECT jsonb_agg(cql1_to_cql2(el)) INTO args FROM jsonb_array_elements(j) el;\n RETURN args;\n END IF;\n IF jsonb_typeof(j) = 'number' THEN\n RETURN j;\n END IF;\n IF jsonb_typeof(j) = 'string' THEN\n RETURN j;\n END IF;\n\n IF jsonb_typeof(j) = 'object' THEN\n SELECT jsonb_build_object(\n 'op', key,\n 'args', cql1_to_cql2(value)\n ) INTO ret\n FROM jsonb_each(j)\n WHERE j IS NOT NULL;\n RETURN ret;\n END IF;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE STRICT;\n\nCREATE TABLE cql2_ops (\n op text PRIMARY KEY,\n template text,\n types text[]\n);\nINSERT INTO cql2_ops (op, template, types) VALUES\n ('eq', '%s = %s', NULL),\n ('lt', '%s < %s', NULL),\n ('lte', '%s <= %s', NULL),\n ('gt', '%s > %s', NULL),\n ('gte', '%s >= %s', NULL),\n ('le', '%s <= %s', NULL),\n ('ge', '%s >= %s', NULL),\n ('=', '%s = %s', NULL),\n ('<', '%s < %s', NULL),\n ('<=', '%s <= %s', NULL),\n ('>', '%s > %s', NULL),\n ('>=', '%s >= %s', NULL),\n ('like', '%s LIKE %s', NULL),\n ('ilike', '%s ILIKE %s', NULL),\n ('+', '%s + %s', NULL),\n ('-', '%s - %s', NULL),\n ('*', '%s * %s', NULL),\n ('/', '%s / %s', NULL),\n ('in', '%s = ANY (%s)', NULL),\n ('not', 'NOT (%s)', NULL),\n ('between', '%s BETWEEN %s AND %s', NULL),\n ('isnull', '%s IS NULL', NULL),\n ('upper', 'upper(%s)', NULL),\n ('lower', 'lower(%s)', NULL)\nON CONFLICT (op) DO UPDATE\n SET\n template = EXCLUDED.template\n;\n\n\nCREATE OR REPLACE FUNCTION cql2_query(j jsonb, wrapper text DEFAULT NULL) RETURNS text AS $$\n#variable_conflict use_variable\nDECLARE\n args jsonb := j->'args';\n arg jsonb;\n op text := lower(j->>'op');\n cql2op RECORD;\n literal text;\n _wrapper text;\nBEGIN\n IF j IS NULL OR (op IS NOT NULL AND args IS NULL) THEN\n RETURN NULL;\n END IF;\n RAISE NOTICE 'CQL2_QUERY: %', j;\n IF j ? 'filter' THEN\n RETURN cql2_query(j->'filter');\n END IF;\n\n IF j ? 'upper' THEN\n RETURN cql2_query(jsonb_build_object('op', 'upper', 'args', j->'upper'));\n END IF;\n\n IF j ? 'lower' THEN\n RETURN cql2_query(jsonb_build_object('op', 'lower', 'args', j->'lower'));\n END IF;\n\n -- Temporal Query\n IF op ilike 't_%' or op = 'anyinteracts' THEN\n RETURN temporal_op_query(op, args);\n END IF;\n\n -- If property is a timestamp convert it to text to use with\n -- general operators\n IF j ? 'timestamp' THEN\n RETURN format('%L::timestamptz', to_tstz(j->'timestamp'));\n END IF;\n IF j ? 'interval' THEN\n RAISE EXCEPTION 'Please use temporal operators when using intervals.';\n RETURN NONE;\n END IF;\n\n -- Spatial Query\n IF op ilike 's_%' or op = 'intersects' THEN\n RETURN spatial_op_query(op, args);\n END IF;\n\n\n IF op = 'in' THEN\n RETURN format(\n '%s = ANY (%L)',\n cql2_query(args->0),\n to_text_array(args->1)\n );\n END IF;\n\n\n\n IF op = 'between' THEN\n args = jsonb_build_array(\n args->0,\n args->1->0,\n args->1->1\n );\n END IF;\n\n -- Make sure that args is an array and run cql2_query on\n -- each element of the array\n RAISE NOTICE 'ARGS PRE: %', args;\n IF j ? 'args' THEN\n IF jsonb_typeof(args) != 'array' THEN\n args := jsonb_build_array(args);\n END IF;\n\n IF jsonb_path_exists(args, '$[*] ? (@.property == \"id\" || @.property == \"datetime\" || @.property == \"end_datetime\" || @.property == \"collection\")') THEN\n wrapper := NULL;\n ELSE\n -- if any of the arguments are a property, try to get the property_wrapper\n FOR arg IN SELECT jsonb_path_query(args, '$[*] ? (@.property != null)') LOOP\n RAISE NOTICE 'Arg: %', arg;\n SELECT property_wrapper INTO wrapper\n FROM queryables\n WHERE name=(arg->>'property')\n LIMIT 1;\n RAISE NOTICE 'Property: %, Wrapper: %', arg, wrapper;\n IF wrapper IS NOT NULL THEN\n EXIT;\n END IF;\n END LOOP;\n\n -- if the property was not in queryables, see if any args were numbers\n IF\n wrapper IS NULL\n AND jsonb_path_exists(args, '$[*] ? (@.type()==\"number\")')\n THEN\n wrapper := 'to_float';\n END IF;\n wrapper := coalesce(wrapper, 'to_text');\n END IF;\n\n SELECT jsonb_agg(cql2_query(a, wrapper))\n INTO args\n FROM jsonb_array_elements(args) a;\n END IF;\n RAISE NOTICE 'ARGS: %', args;\n\n IF op IN ('and', 'or') THEN\n RETURN\n format(\n '(%s)',\n array_to_string(to_text_array(args), format(' %s ', upper(op)))\n );\n END IF;\n\n -- Look up template from cql2_ops\n IF j ? 'op' THEN\n SELECT * INTO cql2op FROM cql2_ops WHERE cql2_ops.op ilike op;\n IF FOUND THEN\n -- If specific index set in queryables for a property cast other arguments to that type\n RETURN format(\n cql2op.template,\n VARIADIC (to_text_array(args))\n );\n ELSE\n RAISE EXCEPTION 'Operator % Not Supported.', op;\n END IF;\n END IF;\n\n\n IF wrapper IS NOT NULL THEN\n RAISE NOTICE 'Wrapping % with %', j, wrapper;\n IF j ? 'property' THEN\n RETURN format('%I(%s)', wrapper, (queryable(j->>'property')).path);\n ELSE\n RETURN format('%I(%L)', wrapper, j);\n END IF;\n ELSIF j ? 'property' THEN\n RETURN quote_ident(j->>'property');\n END IF;\n\n RETURN quote_literal(to_text(j));\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION paging_dtrange(\n j jsonb\n) RETURNS tstzrange AS $$\nDECLARE\n op text;\n filter jsonb := j->'filter';\n dtrange tstzrange := tstzrange('-infinity'::timestamptz,'infinity'::timestamptz);\n sdate timestamptz := '-infinity'::timestamptz;\n edate timestamptz := 'infinity'::timestamptz;\n jpitem jsonb;\nBEGIN\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n END IF;\n IF NOT (filter @? '$.**.op ? (@ == \"or\" || @ == \"not\")') THEN\n FOR jpitem IN SELECT j FROM jsonb_path_query(filter,'strict $.** ? (@.args[*].property == \"datetime\")'::jsonpath) j LOOP\n op := lower(jpitem->>'op');\n dtrange := parse_dtrange(jpitem->'args'->1);\n IF op IN ('<=', 'lt', 'lte', '<', 'le', 't_before') THEN\n sdate := greatest(sdate,'-infinity');\n edate := least(edate, upper(dtrange));\n ELSIF op IN ('>=', '>', 'gt', 'gte', 'ge', 't_after') THEN\n edate := least(edate, 'infinity');\n sdate := greatest(sdate, lower(dtrange));\n ELSIF op IN ('=', 'eq') THEN\n edate := least(edate, upper(dtrange));\n sdate := greatest(sdate, lower(dtrange));\n END IF;\n RAISE NOTICE '2 OP: %, ARGS: %, DTRANGE: %, SDATE: %, EDATE: %', op, jpitem->'args'->1, dtrange, sdate, edate;\n END LOOP;\n END IF;\n IF sdate > edate THEN\n RETURN 'empty'::tstzrange;\n END IF;\n RETURN tstzrange(sdate,edate, '[]');\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT SET TIME ZONE 'UTC';\n\nCREATE OR REPLACE FUNCTION paging_collections(\n IN j jsonb\n) RETURNS text[] AS $$\nDECLARE\n filter jsonb := j->'filter';\n jpitem jsonb;\n op text;\n args jsonb;\n arg jsonb;\n collections text[];\nBEGIN\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n END IF;\n IF NOT (filter @? '$.**.op ? (@ == \"or\" || @ == \"not\")') THEN\n FOR jpitem IN SELECT j FROM jsonb_path_query(filter,'strict $.** ? (@.args[*].property == \"collection\")'::jsonpath) j LOOP\n RAISE NOTICE 'JPITEM: %', jpitem;\n op := jpitem->>'op';\n args := jpitem->'args';\n IF op IN ('=', 'eq', 'in') THEN\n FOR arg IN SELECT a FROM jsonb_array_elements(args) a LOOP\n IF jsonb_typeof(arg) IN ('string', 'array') THEN\n RAISE NOTICE 'arg: %, collections: %', arg, collections;\n IF collections IS NULL OR collections = '{}'::text[] THEN\n collections := to_text_array(arg);\n ELSE\n collections := array_intersection(collections, to_text_array(arg));\n END IF;\n END IF;\n END LOOP;\n END IF;\n END LOOP;\n END IF;\n IF collections = '{}'::text[] THEN\n RETURN NULL;\n END IF;\n RETURN collections;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\nCREATE TABLE items (\n id text NOT NULL,\n geometry geometry NOT NULL,\n collection text NOT NULL,\n datetime timestamptz NOT NULL,\n end_datetime timestamptz NOT NULL,\n content JSONB NOT NULL\n)\nPARTITION BY LIST (collection)\n;\n\nCREATE INDEX \"datetime_idx\" ON items USING BTREE (datetime DESC, end_datetime ASC);\nCREATE INDEX \"geometry_idx\" ON items USING GIST (geometry);\n\nCREATE STATISTICS datetime_stats (dependencies) on datetime, end_datetime from items;\n\n\nALTER TABLE items ADD CONSTRAINT items_collections_fk FOREIGN KEY (collection) REFERENCES collections(id) ON DELETE CASCADE DEFERRABLE;\n\n\nCREATE OR REPLACE FUNCTION content_slim(_item jsonb) RETURNS jsonb AS $$\n SELECT strip_jsonb(_item - '{id,geometry,collection,type}'::text[], collection_base_item(_item->>'collection')) - '{id,geometry,collection,type}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_dehydrate(content jsonb) RETURNS items AS $$\n SELECT\n content->>'id' as id,\n stac_geom(content) as geometry,\n content->>'collection' as collection,\n stac_datetime(content) as datetime,\n stac_end_datetime(content) as end_datetime,\n content_slim(content) as content\n ;\n$$ LANGUAGE SQL STABLE;\n\nCREATE OR REPLACE FUNCTION include_field(f text, fields jsonb DEFAULT '{}'::jsonb) RETURNS boolean AS $$\nDECLARE\n includes jsonb := fields->'include';\n excludes jsonb := fields->'exclude';\nBEGIN\n IF f IS NULL THEN\n RETURN NULL;\n END IF;\n\n\n IF\n jsonb_typeof(excludes) = 'array'\n AND jsonb_array_length(excludes)>0\n AND excludes ? f\n THEN\n RETURN FALSE;\n END IF;\n\n IF\n (\n jsonb_typeof(includes) = 'array'\n AND jsonb_array_length(includes) > 0\n AND includes ? f\n ) OR\n (\n includes IS NULL\n OR jsonb_typeof(includes) = 'null'\n OR jsonb_array_length(includes) = 0\n )\n THEN\n RETURN TRUE;\n END IF;\n\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nDROP FUNCTION IF EXISTS content_hydrate(jsonb, jsonb, jsonb);\nCREATE OR REPLACE FUNCTION content_hydrate(\n _item jsonb,\n _base_item jsonb,\n fields jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\n SELECT merge_jsonb(\n jsonb_fields(_item, fields),\n jsonb_fields(_base_item, fields)\n );\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION content_hydrate(_item items, _collection collections, fields jsonb DEFAULT '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\n content jsonb;\n base_item jsonb := _collection.base_item;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry, 20)::jsonb;\n END IF;\n output := content_hydrate(\n jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'collection', _item.collection,\n 'type', 'Feature'\n ) || _item.content,\n _collection.base_item,\n fields\n );\n\n RETURN output;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_nonhydrated(\n _item items,\n fields jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry, 20)::jsonb;\n END IF;\n output := jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'collection', _item.collection,\n 'type', 'Feature'\n ) || _item.content;\n RETURN output;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_hydrate(_item items, fields jsonb DEFAULT '{}'::jsonb) RETURNS jsonb AS $$\n SELECT content_hydrate(\n _item,\n (SELECT c FROM collections c WHERE id=_item.collection LIMIT 1),\n fields\n );\n$$ LANGUAGE SQL STABLE;\n\n\nCREATE UNLOGGED TABLE items_staging (\n content JSONB NOT NULL\n);\nCREATE UNLOGGED TABLE items_staging_ignore (\n content JSONB NOT NULL\n);\nCREATE UNLOGGED TABLE items_staging_upsert (\n content JSONB NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION items_staging_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p record;\n _partitions text[];\n ts timestamptz := clock_timestamp();\nBEGIN\n RAISE NOTICE 'Creating Partitions. %', clock_timestamp() - ts;\n WITH ranges AS (\n SELECT\n n.content->>'collection' as collection,\n stac_daterange(n.content->'properties') as dtr\n FROM newdata n\n ), p AS (\n SELECT\n collection,\n lower(dtr) as datetime,\n upper(dtr) as end_datetime,\n (partition_name(\n collection,\n lower(dtr)\n )).partition_name as name\n FROM ranges\n )\n INSERT INTO partitions (collection, datetime_range, end_datetime_range)\n SELECT\n collection,\n tstzrange(min(datetime), max(datetime), '[]') as datetime_range,\n tstzrange(min(end_datetime), max(end_datetime), '[]') as end_datetime_range\n FROM p\n GROUP BY collection, name\n ON CONFLICT (name) DO UPDATE SET\n datetime_range = EXCLUDED.datetime_range,\n end_datetime_range = EXCLUDED.end_datetime_range\n ;\n\n RAISE NOTICE 'Doing the insert. %', clock_timestamp() - ts;\n IF TG_TABLE_NAME = 'items_staging' THEN\n INSERT INTO items\n SELECT\n (content_dehydrate(content)).*\n FROM newdata;\n DELETE FROM items_staging;\n ELSIF TG_TABLE_NAME = 'items_staging_ignore' THEN\n INSERT INTO items\n SELECT\n (content_dehydrate(content)).*\n FROM newdata\n ON CONFLICT DO NOTHING;\n DELETE FROM items_staging_ignore;\n ELSIF TG_TABLE_NAME = 'items_staging_upsert' THEN\n WITH staging_formatted AS (\n SELECT (content_dehydrate(content)).* FROM newdata\n ), deletes AS (\n DELETE FROM items i USING staging_formatted s\n WHERE\n i.id = s.id\n AND i.collection = s.collection\n AND i IS DISTINCT FROM s\n RETURNING i.id, i.collection\n )\n INSERT INTO items\n SELECT s.* FROM\n staging_formatted s\n JOIN deletes d\n USING (id, collection);\n DELETE FROM items_staging_upsert;\n END IF;\n RAISE NOTICE 'Done. %', clock_timestamp() - ts;\n\n RETURN NULL;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER items_staging_insert_trigger AFTER INSERT ON items_staging REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\nCREATE TRIGGER items_staging_insert_ignore_trigger AFTER INSERT ON items_staging_ignore REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\nCREATE TRIGGER items_staging_insert_upsert_trigger AFTER INSERT ON items_staging_upsert REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\n\n\n\nCREATE OR REPLACE FUNCTION item_by_id(_id text, _collection text DEFAULT NULL) RETURNS items AS\n$$\nDECLARE\n i items%ROWTYPE;\nBEGIN\n SELECT * INTO i FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection) LIMIT 1;\n RETURN i;\nEND;\n$$ LANGUAGE PLPGSQL STABLE SECURITY DEFINER SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION get_item(_id text, _collection text DEFAULT NULL) RETURNS jsonb AS $$\n SELECT content_hydrate(items) FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection);\n$$ LANGUAGE SQL STABLE SECURITY DEFINER SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION delete_item(_id text, _collection text DEFAULT NULL) RETURNS VOID AS $$\nDECLARE\nout items%ROWTYPE;\nBEGIN\n DELETE FROM items WHERE id = _id AND (_collection IS NULL OR collection=_collection) RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n--/*\nCREATE OR REPLACE FUNCTION create_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION update_item(content jsonb) RETURNS VOID AS $$\nDECLARE\n old items %ROWTYPE;\n out items%ROWTYPE;\nBEGIN\n PERFORM delete_item(content->>'id', content->>'collection');\n PERFORM create_item(content);\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION create_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION collection_bbox(id text) RETURNS jsonb AS $$\n SELECT (replace(replace(replace(st_extent(geometry)::text,'BOX(','[['),')',']]'),' ',','))::jsonb\n FROM items WHERE collection=$1;\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION collection_temporal_extent(id text) RETURNS jsonb AS $$\n SELECT to_jsonb(array[array[min(datetime)::text, max(datetime)::text]])\n FROM items WHERE collection=$1;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection_extents() RETURNS VOID AS $$\nUPDATE collections SET\n content = content ||\n jsonb_build_object(\n 'extent', jsonb_build_object(\n 'spatial', jsonb_build_object(\n 'bbox', collection_bbox(collections.id)\n ),\n 'temporal', jsonb_build_object(\n 'interval', collection_temporal_extent(collections.id)\n )\n )\n )\n;\n$$ LANGUAGE SQL;\nCREATE VIEW partition_steps AS\nSELECT\n name,\n date_trunc('month',lower(datetime_range)) as sdate,\n date_trunc('month', upper(datetime_range)) + '1 month'::interval as edate\n FROM partitions WHERE datetime_range IS NOT NULL AND datetime_range != 'empty'::tstzrange\n ORDER BY datetime_range ASC\n;\n\nCREATE OR REPLACE FUNCTION chunker(\n IN _where text,\n OUT s timestamptz,\n OUT e timestamptz\n) RETURNS SETOF RECORD AS $$\nDECLARE\n explain jsonb;\nBEGIN\n IF _where IS NULL THEN\n _where := ' TRUE ';\n END IF;\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s;', _where)\n INTO explain;\n\n RETURN QUERY\n WITH t AS (\n SELECT j->>0 as p FROM\n jsonb_path_query(\n explain,\n 'strict $.**.\"Relation Name\" ? (@ != null)'\n ) j\n ),\n parts AS (\n SELECT sdate, edate FROM t JOIN partition_steps ON (t.p = name)\n ),\n times AS (\n SELECT sdate FROM parts\n UNION\n SELECT edate FROM parts\n ),\n uniq AS (\n SELECT DISTINCT sdate FROM times ORDER BY sdate\n ),\n last AS (\n SELECT sdate, lead(sdate, 1) over () as edate FROM uniq\n )\n SELECT sdate, edate FROM last WHERE edate IS NOT NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION partition_queries(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN partitions text[] DEFAULT NULL\n) RETURNS SETOF text AS $$\nDECLARE\n query text;\n sdate timestamptz;\n edate timestamptz;\nBEGIN\nIF _where IS NULL OR trim(_where) = '' THEN\n _where = ' TRUE ';\nEND IF;\nRAISE NOTICE 'Getting chunks for % %', _where, _orderby;\nIF _orderby ILIKE 'datetime d%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 DESC LOOP\n RETURN NEXT format($q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n $q$,\n sdate,\n edate,\n _where,\n _orderby\n );\n END LOOP;\nELSIF _orderby ILIKE 'datetime a%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 ASC LOOP\n RETURN NEXT format($q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n $q$,\n sdate,\n edate,\n _where,\n _orderby\n );\n END LOOP;\nELSE\n query := format($q$\n SELECT * FROM items\n WHERE %s\n ORDER BY %s\n $q$, _where, _orderby\n );\n\n RETURN NEXT query;\n RETURN;\nEND IF;\n\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION partition_query_view(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN _limit int DEFAULT 10\n) RETURNS text AS $$\n WITH p AS (\n SELECT * FROM partition_queries(_where, _orderby) p\n )\n SELECT\n CASE WHEN EXISTS (SELECT 1 FROM p) THEN\n (SELECT format($q$\n SELECT * FROM (\n %s\n ) total LIMIT %s\n $q$,\n string_agg(\n format($q$ SELECT * FROM ( %s ) AS sub $q$, p),\n '\n UNION ALL\n '\n ),\n _limit\n ))\n ELSE NULL\n END FROM p;\n$$ LANGUAGE SQL IMMUTABLE;\n\n\n\n\nCREATE OR REPLACE FUNCTION stac_search_to_where(j jsonb) RETURNS text AS $$\nDECLARE\n where_segments text[];\n _where text;\n dtrange tstzrange;\n collections text[];\n geom geometry;\n sdate timestamptz;\n edate timestamptz;\n filterlang text;\n filter jsonb := j->'filter';\nBEGIN\n IF j ? 'ids' THEN\n where_segments := where_segments || format('id = ANY (%L) ', to_text_array(j->'ids'));\n END IF;\n\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n where_segments := where_segments || format('collection = ANY (%L) ', collections);\n END IF;\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n\n where_segments := where_segments || format(' datetime <= %L::timestamptz AND end_datetime >= %L::timestamptz ',\n edate,\n sdate\n );\n END IF;\n\n geom := stac_geom(j);\n IF geom IS NOT NULL THEN\n where_segments := where_segments || format('st_intersects(geometry, %L)',geom);\n END IF;\n\n filterlang := COALESCE(\n j->>'filter-lang',\n get_setting('default-filter-lang', j->'conf')\n );\n IF NOT filter @? '$.**.op' THEN\n filterlang := 'cql-json';\n END IF;\n\n IF filterlang NOT IN ('cql-json','cql2-json') AND j ? 'filter' THEN\n RAISE EXCEPTION '% is not a supported filter-lang. Please use cql-json or cql2-json.', filterlang;\n END IF;\n\n IF j ? 'query' AND j ? 'filter' THEN\n RAISE EXCEPTION 'Can only use either query or filter at one time.';\n END IF;\n\n IF j ? 'query' THEN\n filter := query_to_cql2(j->'query');\n ELSIF filterlang = 'cql-json' THEN\n filter := cql1_to_cql2(filter);\n END IF;\n RAISE NOTICE 'FILTER: %', filter;\n where_segments := where_segments || cql2_query(filter);\n IF cardinality(where_segments) < 1 THEN\n RETURN ' TRUE ';\n END IF;\n\n _where := array_to_string(array_remove(where_segments, NULL), ' AND ');\n\n IF _where IS NULL OR BTRIM(_where) = '' THEN\n RETURN ' TRUE ';\n END IF;\n RETURN _where;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION parse_sort_dir(_dir text, reverse boolean default false) RETURNS text AS $$\n WITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n ) SELECT\n CASE\n WHEN NOT reverse THEN d\n WHEN d = 'ASC' THEN 'DESC'\n WHEN d = 'DESC' THEN 'ASC'\n END\n FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION sort_dir_to_op(_dir text, prev boolean default false) RETURNS text AS $$\n WITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n ) SELECT\n CASE\n WHEN d = 'ASC' AND prev THEN '<='\n WHEN d = 'DESC' AND prev THEN '>='\n WHEN d = 'ASC' THEN '>='\n WHEN d = 'DESC' THEN '<='\n END\n FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION sort_sqlorderby(\n _search jsonb DEFAULT NULL,\n reverse boolean DEFAULT FALSE\n) RETURNS text AS $$\n WITH sortby AS (\n SELECT coalesce(_search->'sortby','[{\"field\":\"datetime\", \"direction\":\"desc\"}]') as sort\n ), withid AS (\n SELECT CASE\n WHEN sort @? '$[*] ? (@.field == \"id\")' THEN sort\n ELSE sort || '[{\"field\":\"id\", \"direction\":\"desc\"}]'::jsonb\n END as sort\n FROM sortby\n ), withid_rows AS (\n SELECT jsonb_array_elements(sort) as value FROM withid\n ),sorts AS (\n SELECT\n coalesce(\n -- field_orderby((items_path(value->>'field')).path_txt),\n (queryable(value->>'field')).expression\n ) as key,\n parse_sort_dir(value->>'direction', reverse) as dir\n FROM withid_rows\n )\n SELECT array_to_string(\n array_agg(concat(key, ' ', dir)),\n ', '\n ) FROM sorts;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_sort_dir(sort_item jsonb) RETURNS text AS $$\n SELECT CASE WHEN sort_item->>'direction' ILIKE 'desc%' THEN 'DESC' ELSE 'ASC' END;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION get_token_filter(_search jsonb = '{}'::jsonb, token_rec jsonb DEFAULT NULL) RETURNS text AS $$\nDECLARE\n token_id text;\n filters text[] := '{}'::text[];\n prev boolean := TRUE;\n field text;\n dir text;\n sort record;\n orfilters text[] := '{}'::text[];\n andfilters text[] := '{}'::text[];\n output text;\n token_where text;\nBEGIN\n RAISE NOTICE 'Getting Token Filter. % %', _search, token_rec;\n -- If no token provided return NULL\n IF token_rec IS NULL THEN\n IF NOT (_search ? 'token' AND\n (\n (_search->>'token' ILIKE 'prev:%')\n OR\n (_search->>'token' ILIKE 'next:%')\n )\n ) THEN\n RETURN NULL;\n END IF;\n prev := (_search->>'token' ILIKE 'prev:%');\n token_id := substr(_search->>'token', 6);\n SELECT to_jsonb(items) INTO token_rec\n FROM items WHERE id=token_id;\n END IF;\n RAISE NOTICE 'TOKEN ID: % %', token_rec, token_rec->'id';\n\n CREATE TEMP TABLE sorts (\n _row int GENERATED ALWAYS AS IDENTITY NOT NULL,\n _field text PRIMARY KEY,\n _dir text NOT NULL,\n _val text\n ) ON COMMIT DROP;\n\n -- Make sure we only have distinct columns to sort with taking the first one we get\n INSERT INTO sorts (_field, _dir)\n SELECT\n (queryable(value->>'field')).expression,\n get_sort_dir(value)\n FROM\n jsonb_array_elements(coalesce(_search->'sortby','[{\"field\":\"datetime\",\"direction\":\"desc\"}]'))\n ON CONFLICT DO NOTHING\n ;\n RAISE NOTICE 'sorts 1: %', (SELECT jsonb_agg(to_json(sorts)) FROM sorts);\n -- Get the first sort direction provided. As the id is a primary key, if there are any\n -- sorts after id they won't do anything, so make sure that id is the last sort item.\n SELECT _dir INTO dir FROM sorts ORDER BY _row ASC LIMIT 1;\n IF EXISTS (SELECT 1 FROM sorts WHERE _field = 'id') THEN\n DELETE FROM sorts WHERE _row > (SELECT _row FROM sorts WHERE _field = 'id' ORDER BY _row ASC);\n ELSE\n INSERT INTO sorts (_field, _dir) VALUES ('id', dir);\n END IF;\n\n -- Add value from looked up item to the sorts table\n UPDATE sorts SET _val=quote_literal(token_rec->>_field);\n\n -- Check if all sorts are the same direction and use row comparison\n -- to filter\n RAISE NOTICE 'sorts 2: %', (SELECT jsonb_agg(to_json(sorts)) FROM sorts);\n\n IF (SELECT count(DISTINCT _dir) FROM sorts) = 1 THEN\n SELECT format(\n '(%s) %s (%s)',\n concat_ws(', ', VARIADIC array_agg(quote_ident(_field))),\n CASE WHEN (prev AND dir = 'ASC') OR (NOT prev AND dir = 'DESC') THEN '<' ELSE '>' END,\n concat_ws(', ', VARIADIC array_agg(_val))\n ) INTO output FROM sorts\n WHERE token_rec ? _field\n ;\n ELSE\n FOR sort IN SELECT * FROM sorts ORDER BY _row asc LOOP\n RAISE NOTICE 'SORT: %', sort;\n IF sort._row = 1 THEN\n orfilters := orfilters || format('(%s %s %s)',\n quote_ident(sort._field),\n CASE WHEN (prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC') THEN '<' ELSE '>' END,\n sort._val\n );\n ELSE\n orfilters := orfilters || format('(%s AND %s %s %s)',\n array_to_string(andfilters, ' AND '),\n quote_ident(sort._field),\n CASE WHEN (prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC') THEN '<' ELSE '>' END,\n sort._val\n );\n\n END IF;\n andfilters := andfilters || format('%s = %s',\n quote_ident(sort._field),\n sort._val\n );\n END LOOP;\n output := array_to_string(orfilters, ' OR ');\n END IF;\n DROP TABLE IF EXISTS sorts;\n token_where := concat('(',coalesce(output,'true'),')');\n IF trim(token_where) = '' THEN\n token_where := NULL;\n END IF;\n RAISE NOTICE 'TOKEN_WHERE: |%|',token_where;\n RETURN token_where;\n END;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION search_tohash(jsonb) RETURNS jsonb AS $$\n SELECT $1 - '{token,limit,context,includes,excludes}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION search_hash(jsonb, jsonb) RETURNS text AS $$\n SELECT md5(concat(search_tohash($1)::text,$2::text));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TABLE IF NOT EXISTS searches(\n hash text GENERATED ALWAYS AS (search_hash(search, metadata)) STORED PRIMARY KEY,\n search jsonb NOT NULL,\n _where text,\n orderby text,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n metadata jsonb DEFAULT '{}'::jsonb NOT NULL\n);\nCREATE TABLE IF NOT EXISTS search_wheres(\n id bigint generated always as identity primary key,\n _where text NOT NULL,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n statslastupdated timestamptz,\n estimated_count bigint,\n estimated_cost float,\n time_to_estimate float,\n total_count bigint,\n time_to_count float,\n partitions text[]\n);\n\nCREATE INDEX IF NOT EXISTS search_wheres_partitions ON search_wheres USING GIN (partitions);\nCREATE UNIQUE INDEX IF NOT EXISTS search_wheres_where ON search_wheres ((md5(_where)));\n\nCREATE OR REPLACE FUNCTION where_stats(inwhere text, updatestats boolean default false, conf jsonb default null) RETURNS search_wheres AS $$\nDECLARE\n t timestamptz;\n i interval;\n explain_json jsonb;\n partitions text[];\n sw search_wheres%ROWTYPE;\n inwhere_hash text := md5(inwhere);\n _context text := lower(context(conf));\n _stats_ttl interval := context_stats_ttl(conf);\n _estimated_cost float := context_estimated_cost(conf);\n _estimated_count int := context_estimated_count(conf);\nBEGIN\n IF _context = 'off' THEN\n sw._where := inwhere;\n return sw;\n END IF;\n\n SELECT * INTO sw FROM search_wheres WHERE md5(_where)=inwhere_hash FOR UPDATE;\n\n -- Update statistics if explicitly set, if statistics do not exist, or statistics ttl has expired\n IF NOT updatestats THEN\n RAISE NOTICE 'Checking if update is needed for: % .', inwhere;\n RAISE NOTICE 'Stats Last Updated: %', sw.statslastupdated;\n RAISE NOTICE 'TTL: %, Age: %', _stats_ttl, now() - sw.statslastupdated;\n RAISE NOTICE 'Context: %, Existing Total: %', _context, sw.total_count;\n IF\n sw.statslastupdated IS NULL\n OR (now() - sw.statslastupdated) > _stats_ttl\n OR (context(conf) != 'off' AND sw.total_count IS NULL)\n THEN\n updatestats := TRUE;\n END IF;\n END IF;\n\n sw._where := inwhere;\n sw.lastused := now();\n sw.usecount := coalesce(sw.usecount,0) + 1;\n\n IF NOT updatestats THEN\n UPDATE search_wheres SET\n lastused = sw.lastused,\n usecount = sw.usecount\n WHERE md5(_where) = inwhere_hash\n RETURNING * INTO sw\n ;\n RETURN sw;\n END IF;\n\n -- Use explain to get estimated count/cost and a list of the partitions that would be hit by the query\n t := clock_timestamp();\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s', inwhere)\n INTO explain_json;\n RAISE NOTICE 'Time for just the explain: %', clock_timestamp() - t;\n i := clock_timestamp() - t;\n\n sw.statslastupdated := now();\n sw.estimated_count := explain_json->0->'Plan'->'Plan Rows';\n sw.estimated_cost := explain_json->0->'Plan'->'Total Cost';\n sw.time_to_estimate := extract(epoch from i);\n\n RAISE NOTICE 'ESTIMATED_COUNT: % < %', sw.estimated_count, _estimated_count;\n RAISE NOTICE 'ESTIMATED_COST: % < %', sw.estimated_cost, _estimated_cost;\n\n -- Do a full count of rows if context is set to on or if auto is set and estimates are low enough\n IF\n _context = 'on'\n OR\n ( _context = 'auto' AND\n (\n sw.estimated_count < _estimated_count\n AND\n sw.estimated_cost < _estimated_cost\n )\n )\n THEN\n t := clock_timestamp();\n RAISE NOTICE 'Calculating actual count...';\n EXECUTE format(\n 'SELECT count(*) FROM items WHERE %s',\n inwhere\n ) INTO sw.total_count;\n i := clock_timestamp() - t;\n RAISE NOTICE 'Actual Count: % -- %', sw.total_count, i;\n sw.time_to_count := extract(epoch FROM i);\n ELSE\n sw.total_count := NULL;\n sw.time_to_count := NULL;\n END IF;\n\n\n INSERT INTO search_wheres\n (_where, lastused, usecount, statslastupdated, estimated_count, estimated_cost, time_to_estimate, partitions, total_count, time_to_count)\n SELECT sw._where, sw.lastused, sw.usecount, sw.statslastupdated, sw.estimated_count, sw.estimated_cost, sw.time_to_estimate, sw.partitions, sw.total_count, sw.time_to_count\n ON CONFLICT ((md5(_where)))\n DO UPDATE\n SET\n lastused = sw.lastused,\n usecount = sw.usecount,\n statslastupdated = sw.statslastupdated,\n estimated_count = sw.estimated_count,\n estimated_cost = sw.estimated_cost,\n time_to_estimate = sw.time_to_estimate,\n total_count = sw.total_count,\n time_to_count = sw.time_to_count\n ;\n RETURN sw;\nEND;\n$$ LANGUAGE PLPGSQL ;\n\n\n\nDROP FUNCTION IF EXISTS search_query;\nCREATE OR REPLACE FUNCTION search_query(\n _search jsonb = '{}'::jsonb,\n updatestats boolean = false,\n _metadata jsonb = '{}'::jsonb\n) RETURNS searches AS $$\nDECLARE\n search searches%ROWTYPE;\n pexplain jsonb;\n t timestamptz;\n i interval;\nBEGIN\n SELECT * INTO search FROM searches\n WHERE hash=search_hash(_search, _metadata) FOR UPDATE;\n\n -- Calculate the where clause if not already calculated\n IF search._where IS NULL THEN\n search._where := stac_search_to_where(_search);\n END IF;\n\n -- Calculate the order by clause if not already calculated\n IF search.orderby IS NULL THEN\n search.orderby := sort_sqlorderby(_search);\n END IF;\n\n PERFORM where_stats(search._where, updatestats, _search->'conf');\n\n search.lastused := now();\n search.usecount := coalesce(search.usecount, 0) + 1;\n INSERT INTO searches (search, _where, orderby, lastused, usecount, metadata)\n VALUES (_search, search._where, search.orderby, search.lastused, search.usecount, _metadata)\n ON CONFLICT (hash) DO\n UPDATE SET\n _where = EXCLUDED._where,\n orderby = EXCLUDED.orderby,\n lastused = EXCLUDED.lastused,\n usecount = EXCLUDED.usecount,\n metadata = EXCLUDED.metadata\n RETURNING * INTO search\n ;\n RETURN search;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION search(_search jsonb = '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n searches searches%ROWTYPE;\n _where text;\n token_where text;\n full_where text;\n orderby text;\n query text;\n token_type text := substr(_search->>'token',1,4);\n _limit int := coalesce((_search->>'limit')::int, 10);\n curs refcursor;\n cntr int := 0;\n iter_record items%ROWTYPE;\n first_record jsonb;\n first_item items%ROWTYPE;\n last_item items%ROWTYPE;\n last_record jsonb;\n out_records jsonb := '[]'::jsonb;\n prev_query text;\n next text;\n prev_id text;\n has_next boolean := false;\n has_prev boolean := false;\n prev text;\n total_count bigint;\n context jsonb;\n collection jsonb;\n includes text[];\n excludes text[];\n exit_flag boolean := FALSE;\n batches int := 0;\n timer timestamptz := clock_timestamp();\n pstart timestamptz;\n pend timestamptz;\n pcurs refcursor;\n search_where search_wheres%ROWTYPE;\n id text;\nBEGIN\nCREATE TEMP TABLE results (content jsonb) ON COMMIT DROP;\n-- if ids is set, short circuit and just use direct ids query for each id\n-- skip any paging or caching\n-- hard codes ordering in the same order as the array of ids\nIF _search ? 'ids' THEN\n INSERT INTO results\n SELECT\n CASE WHEN _search->'conf'->>'nohydrate' IS NOT NULL AND (_search->'conf'->>'nohydrate')::boolean = true THEN\n content_nonhydrated(items, _search->'fields')\n ELSE\n content_hydrate(items, _search->'fields')\n END\n FROM items WHERE\n items.id = ANY(to_text_array(_search->'ids'))\n AND\n CASE WHEN _search ? 'collections' THEN\n items.collection = ANY(to_text_array(_search->'collections'))\n ELSE TRUE\n END\n ORDER BY items.datetime desc, items.id desc\n ;\n SELECT INTO total_count count(*) FROM results;\nELSE\n searches := search_query(_search);\n _where := searches._where;\n orderby := searches.orderby;\n search_where := where_stats(_where);\n total_count := coalesce(search_where.total_count, search_where.estimated_count);\n\n IF token_type='prev' THEN\n token_where := get_token_filter(_search, null::jsonb);\n orderby := sort_sqlorderby(_search, TRUE);\n END IF;\n IF token_type='next' THEN\n token_where := get_token_filter(_search, null::jsonb);\n END IF;\n\n full_where := concat_ws(' AND ', _where, token_where);\n RAISE NOTICE 'FULL QUERY % %', full_where, clock_timestamp()-timer;\n timer := clock_timestamp();\n\n FOR query IN SELECT partition_queries(full_where, orderby, search_where.partitions) LOOP\n timer := clock_timestamp();\n query := format('%s LIMIT %s', query, _limit + 1);\n RAISE NOTICE 'Partition Query: %', query;\n batches := batches + 1;\n -- curs = create_cursor(query);\n OPEN curs FOR EXECUTE query;\n LOOP\n FETCH curs into iter_record;\n EXIT WHEN NOT FOUND;\n cntr := cntr + 1;\n\n IF _search->'conf'->>'nohydrate' IS NOT NULL AND (_search->'conf'->>'nohydrate')::boolean = true THEN\n last_record := content_nonhydrated(iter_record, _search->'fields');\n ELSE\n last_record := content_hydrate(iter_record, _search->'fields');\n END IF;\n last_item := iter_record;\n IF cntr = 1 THEN\n first_item := last_item;\n first_record := last_record;\n END IF;\n IF cntr <= _limit THEN\n INSERT INTO results (content) VALUES (last_record);\n ELSIF cntr > _limit THEN\n has_next := true;\n exit_flag := true;\n EXIT;\n END IF;\n END LOOP;\n CLOSE curs;\n RAISE NOTICE 'Query took %.', clock_timestamp()-timer;\n timer := clock_timestamp();\n EXIT WHEN exit_flag;\n END LOOP;\n RAISE NOTICE 'Scanned through % partitions.', batches;\nEND IF;\n\nSELECT jsonb_agg(content) INTO out_records FROM results WHERE content is not NULL;\n\nDROP TABLE results;\n\n\n-- Flip things around if this was the result of a prev token query\nIF token_type='prev' THEN\n out_records := flip_jsonb_array(out_records);\n first_record := last_record;\nEND IF;\n\n-- If this query has a token, see if there is data before the first record\nIF _search ? 'token' THEN\n prev_query := format(\n 'SELECT 1 FROM items WHERE %s LIMIT 1',\n concat_ws(\n ' AND ',\n _where,\n trim(get_token_filter(_search, to_jsonb(first_item)))\n )\n );\n RAISE NOTICE 'Query to get previous record: % --- %', prev_query, first_record;\n EXECUTE prev_query INTO has_prev;\n IF FOUND and has_prev IS NOT NULL THEN\n RAISE NOTICE 'Query results from prev query: %', has_prev;\n has_prev := TRUE;\n END IF;\nEND IF;\nhas_prev := COALESCE(has_prev, FALSE);\n\nIF has_prev THEN\n prev := out_records->0->>'id';\nEND IF;\nIF has_next OR token_type='prev' THEN\n next := out_records->-1->>'id';\nEND IF;\n\nIF context(_search->'conf') != 'off' THEN\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'matched', total_count,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\nELSE\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\nEND IF;\n\ncollection := jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb),\n 'next', next,\n 'prev', prev,\n 'context', context\n);\n\nRETURN collection;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER SET SEARCH_PATH TO pgstac, public;\n\n\nCREATE OR REPLACE FUNCTION search_cursor(_search jsonb = '{}'::jsonb) RETURNS refcursor AS $$\nDECLARE\n curs refcursor;\n searches searches%ROWTYPE;\n _where text;\n _orderby text;\n q text;\n\nBEGIN\n searches := search_query(_search);\n _where := searches._where;\n _orderby := searches.orderby;\n\n OPEN curs FOR\n WITH p AS (\n SELECT * FROM partition_queries(_where, _orderby) p\n )\n SELECT\n CASE WHEN EXISTS (SELECT 1 FROM p) THEN\n (SELECT format($q$\n SELECT * FROM (\n %s\n ) total\n $q$,\n string_agg(\n format($q$ SELECT * FROM ( %s ) AS sub $q$, p),\n '\n UNION ALL\n '\n )\n ))\n ELSE NULL\n END FROM p;\n RETURN curs;\nEND;\n$$ LANGUAGE PLPGSQL;\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION tileenvelope(zoom int, x int, y int) RETURNS geometry AS $$\nWITH t AS (\n SELECT\n 20037508.3427892 as merc_max,\n -20037508.3427892 as merc_min,\n (2 * 20037508.3427892) / (2 ^ zoom) as tile_size\n)\nSELECT st_makeenvelope(\n merc_min + (tile_size * x),\n merc_max - (tile_size * (y + 1)),\n merc_min + (tile_size * (x + 1)),\n merc_max - (tile_size * y),\n 3857\n) FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;DROP FUNCTION IF EXISTS mercgrid;\n\n\nCREATE OR REPLACE FUNCTION ftime() RETURNS interval as $$\nSELECT age(clock_timestamp(), transaction_timestamp());\n$$ LANGUAGE SQL;\nSET SEARCH_PATH to pgstac, public;\n\nDROP FUNCTION IF EXISTS geometrysearch;\nCREATE OR REPLACE FUNCTION geometrysearch(\n IN geom geometry,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE, -- Return as soon as the passed in geometry is full covered\n IN skipcovered boolean DEFAULT TRUE -- Skip any items that would show up completely under the previous items\n) RETURNS jsonb AS $$\nDECLARE\n search searches%ROWTYPE;\n curs refcursor;\n _where text;\n query text;\n iter_record items%ROWTYPE;\n out_records jsonb := '{}'::jsonb[];\n exit_flag boolean := FALSE;\n counter int := 1;\n scancounter int := 1;\n remaining_limit int := _scanlimit;\n tilearea float;\n unionedgeom geometry;\n clippedgeom geometry;\n unionedgeom_area float := 0;\n prev_area float := 0;\n excludes text[];\n includes text[];\n\nBEGIN\n DROP TABLE IF EXISTS pgstac_results;\n CREATE TEMP TABLE pgstac_results (content jsonb) ON COMMIT DROP;\n\n -- If skipcovered is true then you will always want to exit when the passed in geometry is full\n IF skipcovered THEN\n exitwhenfull := TRUE;\n END IF;\n\n SELECT * INTO search FROM searches WHERE hash=queryhash;\n\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Search with Query Hash % Not Found', queryhash;\n END IF;\n\n tilearea := st_area(geom);\n _where := format('%s AND st_intersects(geometry, %L::geometry)', search._where, geom);\n\n\n FOR query IN SELECT * FROM partition_queries(_where, search.orderby) LOOP\n query := format('%s LIMIT %L', query, remaining_limit);\n RAISE NOTICE '%', query;\n OPEN curs FOR EXECUTE query;\n LOOP\n FETCH curs INTO iter_record;\n EXIT WHEN NOT FOUND;\n IF exitwhenfull OR skipcovered THEN -- If we are not using exitwhenfull or skipcovered, we do not need to do expensive geometry operations\n clippedgeom := st_intersection(geom, iter_record.geometry);\n\n IF unionedgeom IS NULL THEN\n unionedgeom := clippedgeom;\n ELSE\n unionedgeom := st_union(unionedgeom, clippedgeom);\n END IF;\n\n unionedgeom_area := st_area(unionedgeom);\n\n IF skipcovered AND prev_area = unionedgeom_area THEN\n scancounter := scancounter + 1;\n CONTINUE;\n END IF;\n\n prev_area := unionedgeom_area;\n\n RAISE NOTICE '% % % %', unionedgeom_area/tilearea, counter, scancounter, ftime();\n END IF;\n RAISE NOTICE '% %', iter_record, content_hydrate(iter_record, fields);\n INSERT INTO pgstac_results (content) VALUES (content_hydrate(iter_record, fields));\n\n IF counter >= _limit\n OR scancounter > _scanlimit\n OR ftime() > _timelimit\n OR (exitwhenfull AND unionedgeom_area >= tilearea)\n THEN\n exit_flag := TRUE;\n EXIT;\n END IF;\n counter := counter + 1;\n scancounter := scancounter + 1;\n\n END LOOP;\n CLOSE curs;\n EXIT WHEN exit_flag;\n remaining_limit := _scanlimit - scancounter;\n END LOOP;\n\n SELECT jsonb_agg(content) INTO out_records FROM pgstac_results WHERE content IS NOT NULL;\n\n RETURN jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb)\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDROP FUNCTION IF EXISTS geojsonsearch;\nCREATE OR REPLACE FUNCTION geojsonsearch(\n IN geojson jsonb,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_geomfromgeojson(geojson),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS xyzsearch;\nCREATE OR REPLACE FUNCTION xyzsearch(\n IN _x int,\n IN _y int,\n IN _z int,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_transform(tileenvelope(_z, _x, _y), 4326),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nGRANT ALL ON SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac to pgstac_admin;\n\n-- pgstac_read role limited to using function apis\nGRANT EXECUTE ON FUNCTION search TO pgstac_read;\nGRANT EXECUTE ON FUNCTION search_query TO pgstac_read;\nGRANT EXECUTE ON FUNCTION item_by_id TO pgstac_read;\nGRANT EXECUTE ON FUNCTION get_item TO pgstac_read;\n\nGRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_ingest;\nGRANT USAGE ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_ingest;\nSELECT set_version('0.6.4');\n" }, { "path": "src/pgstac/migrations/pgstac.0.6.5-0.6.6.sql", "content": "SET SEARCH_PATH to pgstac, public;\nset check_function_bodies = off;\n\nCREATE OR REPLACE FUNCTION pgstac.cql2_query(j jsonb, wrapper text DEFAULT NULL::text)\n RETURNS text\n LANGUAGE plpgsql\n STABLE\nAS $function$\n#variable_conflict use_variable\nDECLARE\n args jsonb := j->'args';\n arg jsonb;\n op text := lower(j->>'op');\n cql2op RECORD;\n literal text;\n _wrapper text;\n leftarg text;\n rightarg text;\nBEGIN\n IF j IS NULL OR (op IS NOT NULL AND args IS NULL) THEN\n RETURN NULL;\n END IF;\n RAISE NOTICE 'CQL2_QUERY: %', j;\n IF j ? 'filter' THEN\n RETURN cql2_query(j->'filter');\n END IF;\n\n IF j ? 'upper' THEN\n RETURN cql2_query(jsonb_build_object('op', 'upper', 'args', j->'upper'));\n END IF;\n\n IF j ? 'lower' THEN\n RETURN cql2_query(jsonb_build_object('op', 'lower', 'args', j->'lower'));\n END IF;\n\n -- Temporal Query\n IF op ilike 't_%' or op = 'anyinteracts' THEN\n RETURN temporal_op_query(op, args);\n END IF;\n\n -- If property is a timestamp convert it to text to use with\n -- general operators\n IF j ? 'timestamp' THEN\n RETURN format('%L::timestamptz', to_tstz(j->'timestamp'));\n END IF;\n IF j ? 'interval' THEN\n RAISE EXCEPTION 'Please use temporal operators when using intervals.';\n RETURN NONE;\n END IF;\n\n -- Spatial Query\n IF op ilike 's_%' or op = 'intersects' THEN\n RETURN spatial_op_query(op, args);\n END IF;\n\n IF op IN ('a_equals','a_contains','a_contained_by','a_overlaps') THEN\n IF args->0 ? 'property' THEN\n leftarg := format('to_text_array(%s)', (queryable(args->0->>'property')).path);\n END IF;\n IF args->1 ? 'property' THEN\n rightarg := format('to_text_array(%s)', (queryable(args->1->>'property')).path);\n END IF;\n RETURN FORMAT(\n '%s %s %s',\n COALESCE(leftarg, quote_literal(to_text_array(args->0))),\n CASE op\n WHEN 'a_equals' THEN '='\n WHEN 'a_contains' THEN '@>'\n WHEN 'a_contained_by' THEN '<@'\n WHEN 'a_overlaps' THEN '&&'\n END,\n COALESCE(rightarg, quote_literal(to_text_array(args->1)))\n );\n END IF;\n\n IF op = 'in' THEN\n RAISE NOTICE 'IN : % % %', args, jsonb_build_array(args->0), args->1;\n args := jsonb_build_array(args->0) || (args->1);\n RAISE NOTICE 'IN2 : %', args;\n END IF;\n\n\n\n IF op = 'between' THEN\n args = jsonb_build_array(\n args->0,\n args->1->0,\n args->1->1\n );\n END IF;\n\n -- Make sure that args is an array and run cql2_query on\n -- each element of the array\n RAISE NOTICE 'ARGS PRE: %', args;\n IF j ? 'args' THEN\n IF jsonb_typeof(args) != 'array' THEN\n args := jsonb_build_array(args);\n END IF;\n\n IF jsonb_path_exists(args, '$[*] ? (@.property == \"id\" || @.property == \"datetime\" || @.property == \"end_datetime\" || @.property == \"collection\")') THEN\n wrapper := NULL;\n ELSE\n -- if any of the arguments are a property, try to get the property_wrapper\n FOR arg IN SELECT jsonb_path_query(args, '$[*] ? (@.property != null)') LOOP\n RAISE NOTICE 'Arg: %', arg;\n SELECT property_wrapper INTO wrapper\n FROM queryables\n WHERE name=(arg->>'property')\n LIMIT 1;\n RAISE NOTICE 'Property: %, Wrapper: %', arg, wrapper;\n IF wrapper IS NOT NULL THEN\n EXIT;\n END IF;\n END LOOP;\n\n -- if the property was not in queryables, see if any args were numbers\n IF\n wrapper IS NULL\n AND jsonb_path_exists(args, '$[*] ? (@.type()==\"number\")')\n THEN\n wrapper := 'to_float';\n END IF;\n wrapper := coalesce(wrapper, 'to_text');\n END IF;\n\n SELECT jsonb_agg(cql2_query(a, wrapper))\n INTO args\n FROM jsonb_array_elements(args) a;\n END IF;\n RAISE NOTICE 'ARGS: %', args;\n\n IF op IN ('and', 'or') THEN\n RETURN\n format(\n '(%s)',\n array_to_string(to_text_array(args), format(' %s ', upper(op)))\n );\n END IF;\n\n IF op = 'in' THEN\n RAISE NOTICE 'IN -- % %', args->0, to_text(args->0);\n RETURN format(\n '%s IN (%s)',\n to_text(args->0),\n array_to_string((to_text_array(args))[2:], ',')\n );\n END IF;\n\n -- Look up template from cql2_ops\n IF j ? 'op' THEN\n SELECT * INTO cql2op FROM cql2_ops WHERE cql2_ops.op ilike op;\n IF FOUND THEN\n -- If specific index set in queryables for a property cast other arguments to that type\n\n RETURN format(\n cql2op.template,\n VARIADIC (to_text_array(args))\n );\n ELSE\n RAISE EXCEPTION 'Operator % Not Supported.', op;\n END IF;\n END IF;\n\n\n IF wrapper IS NOT NULL THEN\n RAISE NOTICE 'Wrapping % with %', j, wrapper;\n IF j ? 'property' THEN\n RETURN format('%I(%s)', wrapper, (queryable(j->>'property')).path);\n ELSE\n RETURN format('%I(%L)', wrapper, j);\n END IF;\n ELSIF j ? 'property' THEN\n RETURN quote_ident(j->>'property');\n END IF;\n\n RETURN quote_literal(to_text(j));\nEND;\n$function$\n;\n\nTRUNCATE cql2_ops;\nINSERT INTO cql2_ops (op, template, types) VALUES\n ('eq', '%s = %s', NULL),\n ('neq', '%s != %s', NULL),\n ('ne', '%s != %s', NULL),\n ('!=', '%s != %s', NULL),\n ('<>', '%s != %s', NULL),\n ('lt', '%s < %s', NULL),\n ('lte', '%s <= %s', NULL),\n ('gt', '%s > %s', NULL),\n ('gte', '%s >= %s', NULL),\n ('le', '%s <= %s', NULL),\n ('ge', '%s >= %s', NULL),\n ('=', '%s = %s', NULL),\n ('<', '%s < %s', NULL),\n ('<=', '%s <= %s', NULL),\n ('>', '%s > %s', NULL),\n ('>=', '%s >= %s', NULL),\n ('like', '%s LIKE %s', NULL),\n ('ilike', '%s ILIKE %s', NULL),\n ('+', '%s + %s', NULL),\n ('-', '%s - %s', NULL),\n ('*', '%s * %s', NULL),\n ('/', '%s / %s', NULL),\n ('not', 'NOT (%s)', NULL),\n ('between', '%s BETWEEN %s AND %s', NULL),\n ('isnull', '%s IS NULL', NULL),\n ('upper', 'upper(%s)', NULL),\n ('lower', 'lower(%s)', NULL)\nON CONFLICT (op) DO UPDATE\n SET\n template = EXCLUDED.template\n;\n\n\nSELECT set_version('0.6.6');\n" }, { "path": "src/pgstac/migrations/pgstac.0.6.5.sql", "content": "CREATE EXTENSION IF NOT EXISTS postgis;\nCREATE EXTENSION IF NOT EXISTS btree_gist;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_admin;\n CREATE ROLE pgstac_read;\n CREATE ROLE pgstac_ingest;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nGRANT pgstac_admin TO current_user;\n\nCREATE SCHEMA IF NOT EXISTS pgstac AUTHORIZATION pgstac_admin;\n\nALTER ROLE pgstac_admin SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_read SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_ingest SET SEARCH_PATH TO pgstac, public;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\n\nGRANT pgstac_read TO pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\n\nSET ROLE pgstac_admin;\n\nSET SEARCH_PATH TO pgstac, public;\n\n\nCREATE TABLE IF NOT EXISTS migrations (\n version text PRIMARY KEY,\n datetime timestamptz DEFAULT clock_timestamp() NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION get_version() RETURNS text AS $$\n SELECT version FROM pgstac.migrations ORDER BY datetime DESC, version DESC LIMIT 1;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION set_version(text) RETURNS text AS $$\n INSERT INTO pgstac.migrations (version) VALUES ($1)\n ON CONFLICT DO NOTHING\n RETURNING version;\n$$ LANGUAGE SQL;\n\n\nCREATE TABLE IF NOT EXISTS pgstac_settings (\n name text PRIMARY KEY,\n value text NOT NULL\n);\n\nINSERT INTO pgstac_settings (name, value) VALUES\n ('context', 'off'),\n ('context_estimated_count', '100000'),\n ('context_estimated_cost', '100000'),\n ('context_stats_ttl', '1 day'),\n ('default-filter-lang', 'cql2-json'),\n ('additional_properties', 'true')\nON CONFLICT DO NOTHING\n;\n\n\nCREATE OR REPLACE FUNCTION get_setting(IN _setting text, IN conf jsonb DEFAULT NULL) RETURNS text AS $$\nSELECT COALESCE(\n conf->>_setting,\n current_setting(concat('pgstac.',_setting), TRUE),\n (SELECT value FROM pgstac.pgstac_settings WHERE name=_setting)\n);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context(conf jsonb DEFAULT NULL) RETURNS text AS $$\n SELECT pgstac.get_setting('context', conf);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context_estimated_count(conf jsonb DEFAULT NULL) RETURNS int AS $$\n SELECT pgstac.get_setting('context_estimated_count', conf)::int;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_estimated_cost();\nCREATE OR REPLACE FUNCTION context_estimated_cost(conf jsonb DEFAULT NULL) RETURNS float AS $$\n SELECT pgstac.get_setting('context_estimated_cost', conf)::float;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_stats_ttl();\nCREATE OR REPLACE FUNCTION context_stats_ttl(conf jsonb DEFAULT NULL) RETURNS interval AS $$\n SELECT pgstac.get_setting('context_stats_ttl', conf)::interval;\n$$ LANGUAGE SQL;\n\n\nCREATE OR REPLACE FUNCTION notice(VARIADIC text[]) RETURNS boolean AS $$\nDECLARE\ndebug boolean := current_setting('pgstac.debug', true);\nBEGIN\n IF debug THEN\n RAISE NOTICE 'NOTICE FROM FUNC: % >>>>> %', concat_ws(' | ', $1), clock_timestamp();\n RETURN TRUE;\n END IF;\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION empty_arr(ANYARRAY) RETURNS BOOLEAN AS $$\nSELECT CASE\n WHEN $1 IS NULL THEN TRUE\n WHEN cardinality($1)<1 THEN TRUE\nELSE FALSE\nEND;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_intersection(_a ANYARRAY, _b ANYARRAY) RETURNS ANYARRAY AS $$\n SELECT ARRAY ( SELECT unnest(_a) INTERSECT SELECT UNNEST(_b) );\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION array_map_ident(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_ident(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_map_literal(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_literal(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION array_reverse(anyarray) RETURNS anyarray AS $$\nSELECT ARRAY(\n SELECT $1[i]\n FROM generate_subscripts($1,1) AS s(i)\n ORDER BY i DESC\n);\n$$ LANGUAGE SQL STRICT IMMUTABLE;\nCREATE OR REPLACE FUNCTION to_int(jsonb) RETURNS int AS $$\n SELECT floor(($1->>0)::float)::int;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION to_float(jsonb) RETURNS float AS $$\n SELECT ($1->>0)::float;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION to_tstz(jsonb) RETURNS timestamptz AS $$\n SELECT ($1->>0)::timestamptz;\n$$ LANGUAGE SQL IMMUTABLE STRICT SET TIME ZONE 'UTC';\n\n\nCREATE OR REPLACE FUNCTION to_text(jsonb) RETURNS text AS $$\n SELECT CASE WHEN jsonb_typeof($1) IN ('array','object') THEN $1::text ELSE $1->>0 END;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION to_text_array(jsonb) RETURNS text[] AS $$\n SELECT\n CASE jsonb_typeof($1)\n WHEN 'array' THEN ARRAY(SELECT jsonb_array_elements_text($1))\n ELSE ARRAY[$1->>0]\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\nCREATE OR REPLACE FUNCTION bbox_geom(_bbox jsonb) RETURNS geometry AS $$\nSELECT CASE jsonb_array_length(_bbox)\n WHEN 4 THEN\n ST_SetSRID(ST_MakeEnvelope(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float,\n (_bbox->>3)::float\n ),4326)\n WHEN 6 THEN\n ST_SetSRID(ST_3DMakeBox(\n ST_MakePoint(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float\n ),\n ST_MakePoint(\n (_bbox->>3)::float,\n (_bbox->>4)::float,\n (_bbox->>5)::float\n )\n ),4326)\n ELSE null END;\n;\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION geom_bbox(_geom geometry) RETURNS jsonb AS $$\n SELECT jsonb_build_array(\n st_xmin(_geom),\n st_ymin(_geom),\n st_xmax(_geom),\n st_ymax(_geom)\n );\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION flip_jsonb_array(j jsonb) RETURNS jsonb AS $$\n SELECT jsonb_agg(value) FROM (SELECT value FROM jsonb_array_elements(j) WITH ORDINALITY ORDER BY ordinality DESC) as t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION explode_dotpaths(j jsonb) RETURNS SETOF text[] AS $$\n SELECT string_to_array(p, '.') as e FROM jsonb_array_elements_text(j) p;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION explode_dotpaths_recurse(IN j jsonb) RETURNS SETOF text[] AS $$\n WITH RECURSIVE t AS (\n SELECT e FROM explode_dotpaths(j) e\n UNION ALL\n SELECT e[1:cardinality(e)-1]\n FROM t\n WHERE cardinality(e)>1\n ) SELECT e FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION jsonb_set_nested(j jsonb, path text[], val jsonb) RETURNS jsonb AS $$\nDECLARE\nBEGIN\n IF cardinality(path) > 1 THEN\n FOR i IN 1..(cardinality(path)-1) LOOP\n IF j #> path[:i] IS NULL THEN\n j := jsonb_set_lax(j, path[:i], '{}', TRUE);\n END IF;\n END LOOP;\n END IF;\n RETURN jsonb_set_lax(j, path, val, true);\n\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\n\n\nCREATE OR REPLACE FUNCTION jsonb_include(j jsonb, f jsonb) RETURNS jsonb AS $$\nDECLARE\n includes jsonb := f-> 'include';\n outj jsonb := '{}'::jsonb;\n path text[];\nBEGIN\n IF\n includes IS NULL\n OR jsonb_array_length(includes) = 0\n THEN\n RETURN j;\n ELSE\n includes := includes || '[\"id\",\"collection\"]'::jsonb;\n FOR path IN SELECT explode_dotpaths(includes) LOOP\n outj := jsonb_set_nested(outj, path, j #> path);\n END LOOP;\n END IF;\n RETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_exclude(j jsonb, f jsonb) RETURNS jsonb AS $$\nDECLARE\n excludes jsonb := f-> 'exclude';\n outj jsonb := j;\n path text[];\nBEGIN\n IF\n excludes IS NULL\n OR jsonb_array_length(excludes) = 0\n THEN\n RETURN j;\n ELSE\n FOR path IN SELECT explode_dotpaths(excludes) LOOP\n outj := outj #- path;\n END LOOP;\n END IF;\n RETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_fields(j jsonb, f jsonb DEFAULT '{\"fields\":[]}') RETURNS jsonb AS $$\n SELECT jsonb_exclude(jsonb_include(j, f), f);\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION merge_jsonb(_a jsonb, _b jsonb) RETURNS jsonb AS $$\n SELECT\n CASE\n WHEN _a = '\"𒍟※\"'::jsonb THEN NULL\n WHEN _a IS NULL OR jsonb_typeof(_a) = 'null' THEN _b\n WHEN jsonb_typeof(_a) = 'object' AND jsonb_typeof(_b) = 'object' THEN\n (\n SELECT\n jsonb_strip_nulls(\n jsonb_object_agg(\n key,\n merge_jsonb(a.value, b.value)\n )\n )\n FROM\n jsonb_each(coalesce(_a,'{}'::jsonb)) as a\n FULL JOIN\n jsonb_each(coalesce(_b,'{}'::jsonb)) as b\n USING (key)\n )\n WHEN\n jsonb_typeof(_a) = 'array'\n AND jsonb_typeof(_b) = 'array'\n AND jsonb_array_length(_a) = jsonb_array_length(_b)\n THEN\n (\n SELECT jsonb_agg(m) FROM\n ( SELECT\n merge_jsonb(\n jsonb_array_elements(_a),\n jsonb_array_elements(_b)\n ) as m\n ) as l\n )\n ELSE _a\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION strip_jsonb(_a jsonb, _b jsonb) RETURNS jsonb AS $$\n SELECT\n CASE\n\n WHEN (_a IS NULL OR jsonb_typeof(_a) = 'null') AND _b IS NOT NULL AND jsonb_typeof(_b) != 'null' THEN '\"𒍟※\"'::jsonb\n WHEN _b IS NULL OR jsonb_typeof(_a) = 'null' THEN _a\n WHEN _a = _b AND jsonb_typeof(_a) = 'object' THEN '{}'::jsonb\n WHEN _a = _b THEN NULL\n WHEN jsonb_typeof(_a) = 'object' AND jsonb_typeof(_b) = 'object' THEN\n (\n SELECT\n jsonb_strip_nulls(\n jsonb_object_agg(\n key,\n strip_jsonb(a.value, b.value)\n )\n )\n FROM\n jsonb_each(_a) as a\n FULL JOIN\n jsonb_each(_b) as b\n USING (key)\n )\n WHEN\n jsonb_typeof(_a) = 'array'\n AND jsonb_typeof(_b) = 'array'\n AND jsonb_array_length(_a) = jsonb_array_length(_b)\n THEN\n (\n SELECT jsonb_agg(m) FROM\n ( SELECT\n strip_jsonb(\n jsonb_array_elements(_a),\n jsonb_array_elements(_b)\n ) as m\n ) as l\n )\n ELSE _a\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE;\n/* looks for a geometry in a stac item first from geometry and falling back to bbox */\nCREATE OR REPLACE FUNCTION stac_geom(value jsonb) RETURNS geometry AS $$\nSELECT\n CASE\n WHEN value ? 'intersects' THEN\n ST_GeomFromGeoJSON(value->>'intersects')\n WHEN value ? 'geometry' THEN\n ST_GeomFromGeoJSON(value->>'geometry')\n WHEN value ? 'bbox' THEN\n pgstac.bbox_geom(value->'bbox')\n ELSE NULL\n END as geometry\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION stac_daterange(\n value jsonb\n) RETURNS tstzrange AS $$\nDECLARE\n props jsonb := value;\n dt timestamptz;\n edt timestamptz;\nBEGIN\n IF props ? 'properties' THEN\n props := props->'properties';\n END IF;\n IF\n props ? 'start_datetime'\n AND props->>'start_datetime' IS NOT NULL\n AND props ? 'end_datetime'\n AND props->>'end_datetime' IS NOT NULL\n THEN\n dt := props->>'start_datetime';\n edt := props->>'end_datetime';\n IF dt > edt THEN\n RAISE EXCEPTION 'start_datetime must be < end_datetime';\n END IF;\n ELSE\n dt := props->>'datetime';\n edt := props->>'datetime';\n END IF;\n IF dt is NULL OR edt IS NULL THEN\n RAISE NOTICE 'DT: %, EDT: %', dt, edt;\n RAISE EXCEPTION 'Either datetime (%) or both start_datetime (%) and end_datetime (%) must be set.', props->>'datetime',props->>'start_datetime',props->>'end_datetime';\n END IF;\n RETURN tstzrange(dt, edt, '[]');\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_datetime(value jsonb) RETURNS timestamptz AS $$\n SELECT lower(stac_daterange(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_end_datetime(value jsonb) RETURNS timestamptz AS $$\n SELECT upper(stac_daterange(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\n\nCREATE TABLE IF NOT EXISTS stac_extensions(\n name text PRIMARY KEY,\n url text,\n enbabled_by_default boolean NOT NULL DEFAULT TRUE,\n enableable boolean NOT NULL DEFAULT TRUE\n);\n\nINSERT INTO stac_extensions (name, url) VALUES\n ('fields', 'https://api.stacspec.org/v1.0.0-beta.5/item-search#fields'),\n ('sort','https://api.stacspec.org/v1.0.0-beta.5/item-search#sort'),\n ('context','https://api.stacspec.org/v1.0.0-beta.5/item-search#context'),\n ('filter', 'https://api.stacspec.org/v1.0.0-beta.5/item-search#filter'),\n ('query', 'https://api.stacspec.org/v1.0.0-beta.5/item-search#query')\nON CONFLICT (name) DO UPDATE SET url=EXCLUDED.url;\n\n\n\nCREATE OR REPLACE FUNCTION collection_base_item(content jsonb) RETURNS jsonb AS $$\n SELECT jsonb_build_object(\n 'type', 'Feature',\n 'stac_version', content->'stac_version',\n 'assets', content->'item_assets',\n 'collection', content->'id'\n );\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TYPE partition_trunc_strategy AS ENUM ('year', 'month');\n\nCREATE TABLE IF NOT EXISTS collections (\n key bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,\n id text GENERATED ALWAYS AS (content->>'id') STORED UNIQUE,\n content JSONB NOT NULL,\n base_item jsonb GENERATED ALWAYS AS (pgstac.collection_base_item(content)) STORED,\n partition_trunc partition_trunc_strategy\n);\n\n\nCREATE OR REPLACE FUNCTION collection_base_item(cid text) RETURNS jsonb AS $$\n SELECT pgstac.collection_base_item(content) FROM pgstac.collections WHERE id = cid LIMIT 1;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION table_empty(text) RETURNS boolean AS $$\nDECLARE\n retval boolean;\nBEGIN\n EXECUTE format($q$\n SELECT NOT EXISTS (SELECT 1 FROM %I LIMIT 1)\n $q$,\n $1\n ) INTO retval;\n RETURN retval;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION collections_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n q text;\n partition_name text := format('_items_%s', NEW.key);\n partition_exists boolean := false;\n partition_empty boolean := true;\n err_context text;\n loadtemp boolean := FALSE;\nBEGIN\n RAISE NOTICE 'Collection Trigger. % %', NEW.id, NEW.key;\n SELECT relid::text INTO partition_name\n FROM pg_partition_tree('items')\n WHERE relid::text = partition_name;\n IF FOUND THEN\n partition_exists := true;\n partition_empty := table_empty(partition_name);\n ELSE\n partition_exists := false;\n partition_empty := true;\n partition_name := format('_items_%s', NEW.key);\n END IF;\n IF TG_OP = 'UPDATE' AND NEW.partition_trunc IS DISTINCT FROM OLD.partition_trunc AND partition_empty THEN\n q := format($q$\n DROP TABLE IF EXISTS %I CASCADE;\n $q$,\n partition_name\n );\n EXECUTE q;\n END IF;\n IF TG_OP = 'UPDATE' AND NEW.partition_trunc IS DISTINCT FROM OLD.partition_trunc AND partition_exists AND NOT partition_empty THEN\n q := format($q$\n CREATE TEMP TABLE changepartitionstaging ON COMMIT DROP AS SELECT * FROM %I;\n DROP TABLE IF EXISTS %I CASCADE;\n $q$,\n partition_name,\n partition_name\n );\n EXECUTE q;\n loadtemp := TRUE;\n partition_empty := TRUE;\n partition_exists := FALSE;\n END IF;\n IF TG_OP = 'UPDATE' AND NEW.partition_trunc IS NOT DISTINCT FROM OLD.partition_trunc THEN\n RETURN NEW;\n END IF;\n IF NEW.partition_trunc IS NULL AND partition_empty THEN\n RAISE NOTICE '% % % %',\n partition_name,\n NEW.id,\n concat(partition_name,'_id_idx'),\n partition_name\n ;\n q := format($q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n $q$,\n partition_name,\n NEW.id,\n concat(partition_name,'_id_idx'),\n partition_name\n );\n RAISE NOTICE 'q: %', q;\n BEGIN\n EXECUTE q;\n EXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', partition_name;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\n END;\n\n ALTER TABLE partitions DISABLE TRIGGER partitions_delete_trigger;\n DELETE FROM partitions WHERE collection=NEW.id AND name=partition_name;\n ALTER TABLE partitions ENABLE TRIGGER partitions_delete_trigger;\n\n INSERT INTO partitions (collection, name) VALUES (NEW.id, partition_name);\n ELSIF partition_empty THEN\n q := format($q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L)\n PARTITION BY RANGE (datetime);\n $q$,\n partition_name,\n NEW.id\n );\n RAISE NOTICE 'q: %', q;\n BEGIN\n EXECUTE q;\n EXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', partition_name;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\n END;\n ALTER TABLE partitions DISABLE TRIGGER partitions_delete_trigger;\n DELETE FROM partitions WHERE collection=NEW.id AND name=partition_name;\n ALTER TABLE partitions ENABLE TRIGGER partitions_delete_trigger;\n ELSE\n RAISE EXCEPTION 'Cannot modify partition % unless empty', partition_name;\n END IF;\n IF loadtemp THEN\n RAISE NOTICE 'Moving data into new partitions.';\n q := format($q$\n WITH p AS (\n SELECT\n collection,\n datetime as datetime,\n end_datetime as end_datetime,\n (partition_name(\n collection,\n datetime\n )).partition_name as name\n FROM changepartitionstaging\n )\n INSERT INTO partitions (collection, datetime_range, end_datetime_range)\n SELECT\n collection,\n tstzrange(min(datetime), max(datetime), '[]') as datetime_range,\n tstzrange(min(end_datetime), max(end_datetime), '[]') as end_datetime_range\n FROM p\n GROUP BY collection, name\n ON CONFLICT (name) DO UPDATE SET\n datetime_range = EXCLUDED.datetime_range,\n end_datetime_range = EXCLUDED.end_datetime_range\n ;\n INSERT INTO %I SELECT * FROM changepartitionstaging;\n DROP TABLE IF EXISTS changepartitionstaging;\n $q$,\n partition_name\n );\n EXECUTE q;\n END IF;\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE TRIGGER collections_trigger AFTER INSERT OR UPDATE ON collections FOR EACH ROW\nEXECUTE FUNCTION collections_trigger_func();\n\nCREATE OR REPLACE FUNCTION partition_collection(collection text, strategy partition_trunc_strategy) RETURNS text AS $$\n UPDATE collections SET partition_trunc=strategy WHERE id=collection RETURNING partition_trunc;\n$$ LANGUAGE SQL;\n\nCREATE TABLE IF NOT EXISTS partitions (\n collection text REFERENCES collections(id) ON DELETE CASCADE,\n name text PRIMARY KEY,\n partition_range tstzrange NOT NULL DEFAULT tstzrange('-infinity'::timestamptz,'infinity'::timestamptz, '[]'),\n datetime_range tstzrange,\n end_datetime_range tstzrange,\n CONSTRAINT prange EXCLUDE USING GIST (\n collection WITH =,\n partition_range WITH &&\n )\n) WITH (FILLFACTOR=90);\nCREATE INDEX partitions_range_idx ON partitions USING GIST(partition_range);\n\nCREATE OR REPLACE FUNCTION partitions_delete_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n q text;\nBEGIN\n RAISE NOTICE 'Partition Delete Trigger. %', OLD.name;\n EXECUTE format($q$\n DROP TABLE IF EXISTS %I CASCADE;\n $q$,\n OLD.name\n );\n RAISE NOTICE 'Dropped partition.';\n RETURN OLD;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER partitions_delete_trigger BEFORE DELETE ON partitions FOR EACH ROW\nEXECUTE FUNCTION partitions_delete_trigger_func();\n\nCREATE OR REPLACE FUNCTION partition_name(\n IN collection text,\n IN dt timestamptz,\n OUT partition_name text,\n OUT partition_range tstzrange\n) AS $$\nDECLARE\n c RECORD;\n parent_name text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n parent_name := format('_items_%s', c.key);\n\n\n IF c.partition_trunc = 'year' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYY'));\n ELSIF c.partition_trunc = 'month' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYYMM'));\n ELSE\n partition_name := parent_name;\n partition_range := tstzrange('-infinity'::timestamptz, 'infinity'::timestamptz, '[]');\n END IF;\n IF partition_range IS NULL THEN\n partition_range := tstzrange(\n date_trunc(c.partition_trunc::text, dt),\n date_trunc(c.partition_trunc::text, dt) + concat('1 ', c.partition_trunc)::interval\n );\n END IF;\n RETURN;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\n\nCREATE OR REPLACE FUNCTION partitions_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n q text;\n cq text;\n parent_name text;\n partition_trunc text;\n partition_name text := NEW.name;\n partition_exists boolean := false;\n partition_empty boolean := true;\n partition_range tstzrange;\n datetime_range tstzrange;\n end_datetime_range tstzrange;\n err_context text;\n mindt timestamptz := lower(NEW.datetime_range);\n maxdt timestamptz := upper(NEW.datetime_range);\n minedt timestamptz := lower(NEW.end_datetime_range);\n maxedt timestamptz := upper(NEW.end_datetime_range);\n t_mindt timestamptz;\n t_maxdt timestamptz;\n t_minedt timestamptz;\n t_maxedt timestamptz;\nBEGIN\n RAISE NOTICE 'Partitions Trigger. %', NEW;\n datetime_range := NEW.datetime_range;\n end_datetime_range := NEW.end_datetime_range;\n\n SELECT\n format('_items_%s', key),\n c.partition_trunc::text\n INTO\n parent_name,\n partition_trunc\n FROM pgstac.collections c\n WHERE c.id = NEW.collection;\n SELECT (pgstac.partition_name(NEW.collection, mindt)).* INTO partition_name, partition_range;\n NEW.name := partition_name;\n\n IF partition_range IS NULL OR partition_range = 'empty'::tstzrange THEN\n partition_range := tstzrange('-infinity'::timestamptz, 'infinity'::timestamptz, '[]');\n END IF;\n\n NEW.partition_range := partition_range;\n IF TG_OP = 'UPDATE' THEN\n mindt := least(mindt, lower(OLD.datetime_range));\n maxdt := greatest(maxdt, upper(OLD.datetime_range));\n minedt := least(minedt, lower(OLD.end_datetime_range));\n maxedt := greatest(maxedt, upper(OLD.end_datetime_range));\n NEW.datetime_range := tstzrange(mindt, maxdt, '[]');\n NEW.end_datetime_range := tstzrange(minedt, maxedt, '[]');\n END IF;\n IF TG_OP = 'INSERT' THEN\n\n IF partition_range != tstzrange('-infinity'::timestamptz, 'infinity'::timestamptz, '[]') THEN\n\n RAISE NOTICE '% % %', partition_name, parent_name, partition_range;\n q := format($q$\n CREATE TABLE IF NOT EXISTS %I partition OF %I FOR VALUES FROM (%L) TO (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n $q$,\n partition_name,\n parent_name,\n lower(partition_range),\n upper(partition_range),\n format('%s_pkey', partition_name),\n partition_name\n );\n BEGIN\n EXECUTE q;\n EXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', partition_name;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\n END;\n END IF;\n\n END IF;\n\n -- Update constraints\n EXECUTE format($q$\n SELECT\n min(datetime),\n max(datetime),\n min(end_datetime),\n max(end_datetime)\n FROM %I;\n $q$, partition_name)\n INTO t_mindt, t_maxdt, t_minedt, t_maxedt;\n mindt := least(mindt, t_mindt);\n maxdt := greatest(maxdt, t_maxdt);\n minedt := least(mindt, minedt, t_minedt);\n maxedt := greatest(maxdt, maxedt, t_maxedt);\n\n mindt := date_trunc(coalesce(partition_trunc, 'year'), mindt);\n maxdt := date_trunc(coalesce(partition_trunc, 'year'), maxdt - '1 second'::interval) + concat('1 ',coalesce(partition_trunc, 'year'))::interval;\n minedt := date_trunc(coalesce(partition_trunc, 'year'), minedt);\n maxedt := date_trunc(coalesce(partition_trunc, 'year'), maxedt - '1 second'::interval) + concat('1 ',coalesce(partition_trunc, 'year'))::interval;\n\n\n IF mindt IS NOT NULL AND maxdt IS NOT NULL AND minedt IS NOT NULL AND maxedt IS NOT NULL THEN\n NEW.datetime_range := tstzrange(mindt, maxdt, '[]');\n NEW.end_datetime_range := tstzrange(minedt, maxedt, '[]');\n IF\n TG_OP='UPDATE'\n AND OLD.datetime_range @> NEW.datetime_range\n AND OLD.end_datetime_range @> NEW.end_datetime_range\n THEN\n RAISE NOTICE 'Range unchanged, not updating constraints.';\n ELSE\n\n RAISE NOTICE '\n SETTING CONSTRAINTS\n mindt: %, maxdt: %\n minedt: %, maxedt: %\n ', mindt, maxdt, minedt, maxedt;\n IF partition_trunc IS NULL THEN\n cq := format($q$\n ALTER TABLE %7$I\n DROP CONSTRAINT IF EXISTS %1$I,\n DROP CONSTRAINT IF EXISTS %2$I,\n ADD CONSTRAINT %1$I\n CHECK (\n (datetime >= %3$L)\n AND (datetime <= %4$L)\n AND (end_datetime >= %5$L)\n AND (end_datetime <= %6$L)\n ) NOT VALID\n ;\n ALTER TABLE %7$I\n VALIDATE CONSTRAINT %1$I;\n $q$,\n format('%s_dt', partition_name),\n format('%s_edt', partition_name),\n mindt,\n maxdt,\n minedt,\n maxedt,\n partition_name\n );\n ELSE\n cq := format($q$\n ALTER TABLE %5$I\n DROP CONSTRAINT IF EXISTS %1$I,\n DROP CONSTRAINT IF EXISTS %2$I,\n ADD CONSTRAINT %2$I\n CHECK ((end_datetime >= %3$L) AND (end_datetime <= %4$L)) NOT VALID\n ;\n ALTER TABLE %5$I\n VALIDATE CONSTRAINT %2$I;\n $q$,\n format('%s_dt', partition_name),\n format('%s_edt', partition_name),\n minedt,\n maxedt,\n partition_name\n );\n\n END IF;\n RAISE NOTICE 'Altering Constraints. %', cq;\n EXECUTE cq;\n END IF;\n ELSE\n NEW.datetime_range = NULL;\n NEW.end_datetime_range = NULL;\n\n cq := format($q$\n ALTER TABLE %3$I\n DROP CONSTRAINT IF EXISTS %1$I,\n DROP CONSTRAINT IF EXISTS %2$I,\n ADD CONSTRAINT %1$I\n CHECK ((datetime IS NULL AND end_datetime IS NULL)) NOT VALID\n ;\n ALTER TABLE %3$I\n VALIDATE CONSTRAINT %1$I;\n $q$,\n format('%s_dt', partition_name),\n format('%s_edt', partition_name),\n partition_name\n );\n EXECUTE cq;\n END IF;\n\n RETURN NEW;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER partitions_trigger BEFORE INSERT OR UPDATE ON partitions FOR EACH ROW\nEXECUTE FUNCTION partitions_trigger_func();\n\n\nCREATE OR REPLACE FUNCTION create_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection(data jsonb) RETURNS VOID AS $$\nDECLARE\n out collections%ROWTYPE;\nBEGIN\n UPDATE collections SET content=data WHERE id = data->>'id' RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ON CONFLICT (id) DO\n UPDATE\n SET content=EXCLUDED.content\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION delete_collection(_id text) RETURNS VOID AS $$\nDECLARE\n out collections%ROWTYPE;\nBEGIN\n DELETE FROM collections WHERE id = _id RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION get_collection(id text) RETURNS jsonb AS $$\n SELECT content FROM collections\n WHERE id=$1\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION all_collections() RETURNS jsonb AS $$\n SELECT jsonb_agg(content) FROM collections;\n;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\nCREATE TABLE queryables (\n id bigint GENERATED ALWAYS AS identity PRIMARY KEY,\n name text UNIQUE NOT NULL,\n collection_ids text[], -- used to determine what partitions to create indexes on\n definition jsonb,\n property_path text,\n property_wrapper text,\n property_index_type text\n);\nCREATE INDEX queryables_name_idx ON queryables (name);\nCREATE INDEX queryables_property_wrapper_idx ON queryables (property_wrapper);\n\n\nINSERT INTO queryables (name, definition) VALUES\n('id', '{\"title\": \"Item ID\",\"description\": \"Item identifier\",\"$ref\": \"https://schemas.stacspec.org/v1.0.0/item-spec/json-schema/item.json#/definitions/core/allOf/2/properties/id\"}'),\n('datetime','{\"description\": \"Datetime\",\"type\": \"string\",\"title\": \"Acquired\",\"format\": \"date-time\",\"pattern\": \"(\\\\+00:00|Z)$\"}')\nON CONFLICT DO NOTHING;\n\n\n\nINSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n('eo:cloud_cover','{\"$ref\": \"https://stac-extensions.github.io/eo/v1.0.0/schema.json#/definitions/fieldsproperties/eo:cloud_cover\"}','to_int','BTREE')\nON CONFLICT DO NOTHING;\n\nCREATE OR REPLACE FUNCTION array_to_path(arr text[]) RETURNS text AS $$\n SELECT string_agg(\n quote_literal(v),\n '->'\n ) FROM unnest(arr) v;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\n\n\nCREATE OR REPLACE FUNCTION queryable(\n IN dotpath text,\n OUT path text,\n OUT expression text,\n OUT wrapper text\n) AS $$\nDECLARE\n q RECORD;\n path_elements text[];\nBEGIN\n IF dotpath IN ('id', 'geometry', 'datetime', 'end_datetime', 'collection') THEN\n path := dotpath;\n expression := dotpath;\n wrapper := NULL;\n RETURN;\n END IF;\n SELECT * INTO q FROM queryables WHERE name=dotpath;\n IF q.property_wrapper IS NULL THEN\n IF q.definition->>'type' = 'number' THEN\n wrapper := 'to_float';\n ELSIF q.definition->>'format' = 'date-time' THEN\n wrapper := 'to_tstz';\n ELSE\n wrapper := 'to_text';\n END IF;\n ELSE\n wrapper := q.property_wrapper;\n END IF;\n IF q.property_path IS NOT NULL THEN\n path := q.property_path;\n ELSE\n path_elements := string_to_array(dotpath, '.');\n IF path_elements[1] IN ('links', 'assets', 'stac_version', 'stac_extensions') THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSIF path_elements[1] = 'properties' THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSE\n path := format($F$content->'properties'->%s$F$, array_to_path(path_elements));\n END IF;\n END IF;\n expression := format('%I(%s)', wrapper, path);\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\nCREATE OR REPLACE FUNCTION create_queryable_indexes() RETURNS VOID AS $$\nDECLARE\n queryable RECORD;\n q text;\nBEGIN\n FOR queryable IN\n SELECT\n queryables.id as qid,\n CASE WHEN collections.key IS NULL THEN 'items' ELSE format('_items_%s',collections.key) END AS part,\n property_index_type,\n expression\n FROM\n queryables\n LEFT JOIN collections ON (collections.id = ANY (queryables.collection_ids))\n JOIN LATERAL queryable(queryables.name) ON (queryables.property_index_type IS NOT NULL)\n LOOP\n q := format(\n $q$\n CREATE INDEX IF NOT EXISTS %I ON %I USING %s ((%s));\n $q$,\n format('%s_%s_idx', queryable.part, queryable.qid),\n queryable.part,\n COALESCE(queryable.property_index_type, 'to_text'),\n queryable.expression\n );\n RAISE NOTICE '%',q;\n EXECUTE q;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION queryables_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\nBEGIN\nPERFORM create_queryable_indexes();\nRETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER queryables_trigger AFTER INSERT OR UPDATE ON queryables\nFOR EACH STATEMENT EXECUTE PROCEDURE queryables_trigger_func();\n\nCREATE TRIGGER queryables_collection_trigger AFTER INSERT OR UPDATE ON collections\nFOR EACH STATEMENT EXECUTE PROCEDURE queryables_trigger_func();\nCREATE OR REPLACE FUNCTION parse_dtrange(\n _indate jsonb,\n relative_base timestamptz DEFAULT date_trunc('hour', CURRENT_TIMESTAMP)\n) RETURNS tstzrange AS $$\nDECLARE\n timestrs text[];\n s timestamptz;\n e timestamptz;\nBEGIN\n timestrs :=\n CASE\n WHEN _indate ? 'timestamp' THEN\n ARRAY[_indate->>'timestamp']\n WHEN _indate ? 'interval' THEN\n to_text_array(_indate->'interval')\n WHEN jsonb_typeof(_indate) = 'array' THEN\n to_text_array(_indate)\n ELSE\n regexp_split_to_array(\n _indate->>0,\n '/'\n )\n END;\n RAISE NOTICE 'TIMESTRS %', timestrs;\n IF cardinality(timestrs) = 1 THEN\n IF timestrs[1] ILIKE 'P%' THEN\n RETURN tstzrange(relative_base - upper(timestrs[1])::interval, relative_base, '[)');\n END IF;\n s := timestrs[1]::timestamptz;\n RETURN tstzrange(s, s, '[]');\n END IF;\n\n IF cardinality(timestrs) != 2 THEN\n RAISE EXCEPTION 'Timestamp cannot have more than 2 values';\n END IF;\n\n IF timestrs[1] = '..' THEN\n s := '-infinity'::timestamptz;\n e := timestrs[2]::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] = '..' THEN\n s := timestrs[1]::timestamptz;\n e := 'infinity'::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[1] ILIKE 'P%' AND timestrs[2] NOT ILIKE 'P%' THEN\n e := timestrs[2]::timestamptz;\n s := e - upper(timestrs[1])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] ILIKE 'P%' AND timestrs[1] NOT ILIKE 'P%' THEN\n s := timestrs[1]::timestamptz;\n e := s + upper(timestrs[2])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n s := timestrs[1]::timestamptz;\n e := timestrs[2]::timestamptz;\n\n RETURN tstzrange(s,e,'[)');\n\n RETURN NULL;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT PARALLEL SAFE SET TIME ZONE 'UTC';\n\nCREATE OR REPLACE FUNCTION parse_dtrange(\n _indate text,\n relative_base timestamptz DEFAULT CURRENT_TIMESTAMP\n) RETURNS tstzrange AS $$\n SELECT parse_dtrange(to_jsonb(_indate), relative_base);\n$$ LANGUAGE SQL STABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION temporal_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\n ll text := 'datetime';\n lh text := 'end_datetime';\n rrange tstzrange;\n rl text;\n rh text;\n outq text;\nBEGIN\n rrange := parse_dtrange(args->1);\n RAISE NOTICE 'Constructing temporal query OP: %, ARGS: %, RRANGE: %', op, args, rrange;\n op := lower(op);\n rl := format('%L::timestamptz', lower(rrange));\n rh := format('%L::timestamptz', upper(rrange));\n outq := CASE op\n WHEN 't_before' THEN 'lh < rl'\n WHEN 't_after' THEN 'll > rh'\n WHEN 't_meets' THEN 'lh = rl'\n WHEN 't_metby' THEN 'll = rh'\n WHEN 't_overlaps' THEN 'll < rl AND rl < lh < rh'\n WHEN 't_overlappedby' THEN 'rl < ll < rh AND lh > rh'\n WHEN 't_starts' THEN 'll = rl AND lh < rh'\n WHEN 't_startedby' THEN 'll = rl AND lh > rh'\n WHEN 't_during' THEN 'll > rl AND lh < rh'\n WHEN 't_contains' THEN 'll < rl AND lh > rh'\n WHEN 't_finishes' THEN 'll > rl AND lh = rh'\n WHEN 't_finishedby' THEN 'll < rl AND lh = rh'\n WHEN 't_equals' THEN 'll = rl AND lh = rh'\n WHEN 't_disjoint' THEN 'NOT (ll <= rh AND lh >= rl)'\n WHEN 't_intersects' THEN 'll <= rh AND lh >= rl'\n WHEN 'anyinteracts' THEN 'll <= rh AND lh >= rl'\n END;\n outq := regexp_replace(outq, '\\mll\\M', ll);\n outq := regexp_replace(outq, '\\mlh\\M', lh);\n outq := regexp_replace(outq, '\\mrl\\M', rl);\n outq := regexp_replace(outq, '\\mrh\\M', rh);\n outq := format('(%s)', outq);\n RETURN outq;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\n\n\nCREATE OR REPLACE FUNCTION spatial_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\n geom text;\n j jsonb := args->1;\nBEGIN\n op := lower(op);\n RAISE NOTICE 'Constructing spatial query OP: %, ARGS: %', op, args;\n IF op NOT IN ('s_equals','s_disjoint','s_touches','s_within','s_overlaps','s_crosses','s_intersects','intersects','s_contains') THEN\n RAISE EXCEPTION 'Spatial Operator % Not Supported', op;\n END IF;\n op := regexp_replace(op, '^s_', 'st_');\n IF op = 'intersects' THEN\n op := 'st_intersects';\n END IF;\n -- Convert geometry to WKB string\n IF j ? 'type' AND j ? 'coordinates' THEN\n geom := st_geomfromgeojson(j)::text;\n ELSIF jsonb_typeof(j) = 'array' THEN\n geom := bbox_geom(j)::text;\n END IF;\n\n RETURN format('%s(geometry, %L::geometry)', op, geom);\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION query_to_cql2(q jsonb) RETURNS jsonb AS $$\n-- Translates anything passed in through the deprecated \"query\" into equivalent CQL2\nWITH t AS (\n SELECT key as property, value as ops\n FROM jsonb_each(q)\n), t2 AS (\n SELECT property, (jsonb_each(ops)).*\n FROM t WHERE jsonb_typeof(ops) = 'object'\n UNION ALL\n SELECT property, 'eq', ops\n FROM t WHERE jsonb_typeof(ops) != 'object'\n)\nSELECT\n jsonb_strip_nulls(jsonb_build_object(\n 'op', 'and',\n 'args', jsonb_agg(\n jsonb_build_object(\n 'op', key,\n 'args', jsonb_build_array(\n jsonb_build_object('property',property),\n value\n )\n )\n )\n )\n) as qcql FROM t2\n;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\nCREATE OR REPLACE FUNCTION cql1_to_cql2(j jsonb) RETURNS jsonb AS $$\nDECLARE\n args jsonb;\n ret jsonb;\nBEGIN\n RAISE NOTICE 'CQL1_TO_CQL2: %', j;\n IF j ? 'filter' THEN\n RETURN cql1_to_cql2(j->'filter');\n END IF;\n IF j ? 'property' THEN\n RETURN j;\n END IF;\n IF jsonb_typeof(j) = 'array' THEN\n SELECT jsonb_agg(cql1_to_cql2(el)) INTO args FROM jsonb_array_elements(j) el;\n RETURN args;\n END IF;\n IF jsonb_typeof(j) = 'number' THEN\n RETURN j;\n END IF;\n IF jsonb_typeof(j) = 'string' THEN\n RETURN j;\n END IF;\n\n IF jsonb_typeof(j) = 'object' THEN\n SELECT jsonb_build_object(\n 'op', key,\n 'args', cql1_to_cql2(value)\n ) INTO ret\n FROM jsonb_each(j)\n WHERE j IS NOT NULL;\n RETURN ret;\n END IF;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE STRICT;\n\nCREATE TABLE cql2_ops (\n op text PRIMARY KEY,\n template text,\n types text[]\n);\nINSERT INTO cql2_ops (op, template, types) VALUES\n ('eq', '%s = %s', NULL),\n ('neq', '%s != %s', NULL),\n ('ne', '%s != %s', NULL),\n ('!=', '%s != %s', NULL),\n ('<>', '%s != %s', NULL),\n ('lt', '%s < %s', NULL),\n ('lte', '%s <= %s', NULL),\n ('gt', '%s > %s', NULL),\n ('gte', '%s >= %s', NULL),\n ('le', '%s <= %s', NULL),\n ('ge', '%s >= %s', NULL),\n ('=', '%s = %s', NULL),\n ('<', '%s < %s', NULL),\n ('<=', '%s <= %s', NULL),\n ('>', '%s > %s', NULL),\n ('>=', '%s >= %s', NULL),\n ('like', '%s LIKE %s', NULL),\n ('ilike', '%s ILIKE %s', NULL),\n ('+', '%s + %s', NULL),\n ('-', '%s - %s', NULL),\n ('*', '%s * %s', NULL),\n ('/', '%s / %s', NULL),\n ('not', 'NOT (%s)', NULL),\n ('between', '%s BETWEEN %s AND %s', NULL),\n ('isnull', '%s IS NULL', NULL),\n ('upper', 'upper(%s)', NULL),\n ('lower', 'lower(%s)', NULL)\nON CONFLICT (op) DO UPDATE\n SET\n template = EXCLUDED.template\n;\n\n\nCREATE OR REPLACE FUNCTION cql2_query(j jsonb, wrapper text DEFAULT NULL) RETURNS text AS $$\n#variable_conflict use_variable\nDECLARE\n args jsonb := j->'args';\n arg jsonb;\n op text := lower(j->>'op');\n cql2op RECORD;\n literal text;\n _wrapper text;\nBEGIN\n IF j IS NULL OR (op IS NOT NULL AND args IS NULL) THEN\n RETURN NULL;\n END IF;\n RAISE NOTICE 'CQL2_QUERY: %', j;\n IF j ? 'filter' THEN\n RETURN cql2_query(j->'filter');\n END IF;\n\n IF j ? 'upper' THEN\n RETURN cql2_query(jsonb_build_object('op', 'upper', 'args', j->'upper'));\n END IF;\n\n IF j ? 'lower' THEN\n RETURN cql2_query(jsonb_build_object('op', 'lower', 'args', j->'lower'));\n END IF;\n\n -- Temporal Query\n IF op ilike 't_%' or op = 'anyinteracts' THEN\n RETURN temporal_op_query(op, args);\n END IF;\n\n -- If property is a timestamp convert it to text to use with\n -- general operators\n IF j ? 'timestamp' THEN\n RETURN format('%L::timestamptz', to_tstz(j->'timestamp'));\n END IF;\n IF j ? 'interval' THEN\n RAISE EXCEPTION 'Please use temporal operators when using intervals.';\n RETURN NONE;\n END IF;\n\n -- Spatial Query\n IF op ilike 's_%' or op = 'intersects' THEN\n RETURN spatial_op_query(op, args);\n END IF;\n\n\n IF op = 'in' THEN\n RAISE NOTICE 'IN : % % %', args, jsonb_build_array(args->0), args->1;\n args := jsonb_build_array(args->0) || (args->1);\n RAISE NOTICE 'IN2 : %', args;\n -- RETURN format(\n -- '%s = ANY (%L)',\n -- cql2_query(args->0),\n -- to_text_array(args->1)\n -- );\n END IF;\n\n\n\n IF op = 'between' THEN\n args = jsonb_build_array(\n args->0,\n args->1->0,\n args->1->1\n );\n END IF;\n\n -- Make sure that args is an array and run cql2_query on\n -- each element of the array\n RAISE NOTICE 'ARGS PRE: %', args;\n IF j ? 'args' THEN\n IF jsonb_typeof(args) != 'array' THEN\n args := jsonb_build_array(args);\n END IF;\n\n IF jsonb_path_exists(args, '$[*] ? (@.property == \"id\" || @.property == \"datetime\" || @.property == \"end_datetime\" || @.property == \"collection\")') THEN\n wrapper := NULL;\n ELSE\n -- if any of the arguments are a property, try to get the property_wrapper\n FOR arg IN SELECT jsonb_path_query(args, '$[*] ? (@.property != null)') LOOP\n RAISE NOTICE 'Arg: %', arg;\n SELECT property_wrapper INTO wrapper\n FROM queryables\n WHERE name=(arg->>'property')\n LIMIT 1;\n RAISE NOTICE 'Property: %, Wrapper: %', arg, wrapper;\n IF wrapper IS NOT NULL THEN\n EXIT;\n END IF;\n END LOOP;\n\n -- if the property was not in queryables, see if any args were numbers\n IF\n wrapper IS NULL\n AND jsonb_path_exists(args, '$[*] ? (@.type()==\"number\")')\n THEN\n wrapper := 'to_float';\n END IF;\n wrapper := coalesce(wrapper, 'to_text');\n END IF;\n\n SELECT jsonb_agg(cql2_query(a, wrapper))\n INTO args\n FROM jsonb_array_elements(args) a;\n END IF;\n RAISE NOTICE 'ARGS: %', args;\n\n IF op IN ('and', 'or') THEN\n RETURN\n format(\n '(%s)',\n array_to_string(to_text_array(args), format(' %s ', upper(op)))\n );\n END IF;\n\n IF op = 'in' THEN\n RAISE NOTICE 'IN -- % %', args->0, to_text(args->0);\n RETURN format(\n '%s IN (%s)',\n to_text(args->0),\n array_to_string((to_text_array(args))[2:], ',')\n );\n END IF;\n\n -- Look up template from cql2_ops\n IF j ? 'op' THEN\n SELECT * INTO cql2op FROM cql2_ops WHERE cql2_ops.op ilike op;\n IF FOUND THEN\n -- If specific index set in queryables for a property cast other arguments to that type\n\n RETURN format(\n cql2op.template,\n VARIADIC (to_text_array(args))\n );\n ELSE\n RAISE EXCEPTION 'Operator % Not Supported.', op;\n END IF;\n END IF;\n\n\n IF wrapper IS NOT NULL THEN\n RAISE NOTICE 'Wrapping % with %', j, wrapper;\n IF j ? 'property' THEN\n RETURN format('%I(%s)', wrapper, (queryable(j->>'property')).path);\n ELSE\n RETURN format('%I(%L)', wrapper, j);\n END IF;\n ELSIF j ? 'property' THEN\n RETURN quote_ident(j->>'property');\n END IF;\n\n RETURN quote_literal(to_text(j));\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION paging_dtrange(\n j jsonb\n) RETURNS tstzrange AS $$\nDECLARE\n op text;\n filter jsonb := j->'filter';\n dtrange tstzrange := tstzrange('-infinity'::timestamptz,'infinity'::timestamptz);\n sdate timestamptz := '-infinity'::timestamptz;\n edate timestamptz := 'infinity'::timestamptz;\n jpitem jsonb;\nBEGIN\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n END IF;\n IF NOT (filter @? '$.**.op ? (@ == \"or\" || @ == \"not\")') THEN\n FOR jpitem IN SELECT j FROM jsonb_path_query(filter,'strict $.** ? (@.args[*].property == \"datetime\")'::jsonpath) j LOOP\n op := lower(jpitem->>'op');\n dtrange := parse_dtrange(jpitem->'args'->1);\n IF op IN ('<=', 'lt', 'lte', '<', 'le', 't_before') THEN\n sdate := greatest(sdate,'-infinity');\n edate := least(edate, upper(dtrange));\n ELSIF op IN ('>=', '>', 'gt', 'gte', 'ge', 't_after') THEN\n edate := least(edate, 'infinity');\n sdate := greatest(sdate, lower(dtrange));\n ELSIF op IN ('=', 'eq') THEN\n edate := least(edate, upper(dtrange));\n sdate := greatest(sdate, lower(dtrange));\n END IF;\n RAISE NOTICE '2 OP: %, ARGS: %, DTRANGE: %, SDATE: %, EDATE: %', op, jpitem->'args'->1, dtrange, sdate, edate;\n END LOOP;\n END IF;\n IF sdate > edate THEN\n RETURN 'empty'::tstzrange;\n END IF;\n RETURN tstzrange(sdate,edate, '[]');\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT SET TIME ZONE 'UTC';\n\nCREATE OR REPLACE FUNCTION paging_collections(\n IN j jsonb\n) RETURNS text[] AS $$\nDECLARE\n filter jsonb := j->'filter';\n jpitem jsonb;\n op text;\n args jsonb;\n arg jsonb;\n collections text[];\nBEGIN\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n END IF;\n IF NOT (filter @? '$.**.op ? (@ == \"or\" || @ == \"not\")') THEN\n FOR jpitem IN SELECT j FROM jsonb_path_query(filter,'strict $.** ? (@.args[*].property == \"collection\")'::jsonpath) j LOOP\n RAISE NOTICE 'JPITEM: %', jpitem;\n op := jpitem->>'op';\n args := jpitem->'args';\n IF op IN ('=', 'eq', 'in') THEN\n FOR arg IN SELECT a FROM jsonb_array_elements(args) a LOOP\n IF jsonb_typeof(arg) IN ('string', 'array') THEN\n RAISE NOTICE 'arg: %, collections: %', arg, collections;\n IF collections IS NULL OR collections = '{}'::text[] THEN\n collections := to_text_array(arg);\n ELSE\n collections := array_intersection(collections, to_text_array(arg));\n END IF;\n END IF;\n END LOOP;\n END IF;\n END LOOP;\n END IF;\n IF collections = '{}'::text[] THEN\n RETURN NULL;\n END IF;\n RETURN collections;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\nCREATE TABLE items (\n id text NOT NULL,\n geometry geometry NOT NULL,\n collection text NOT NULL,\n datetime timestamptz NOT NULL,\n end_datetime timestamptz NOT NULL,\n content JSONB NOT NULL\n)\nPARTITION BY LIST (collection)\n;\n\nCREATE INDEX \"datetime_idx\" ON items USING BTREE (datetime DESC, end_datetime ASC);\nCREATE INDEX \"geometry_idx\" ON items USING GIST (geometry);\n\nCREATE STATISTICS datetime_stats (dependencies) on datetime, end_datetime from items;\n\n\nALTER TABLE items ADD CONSTRAINT items_collections_fk FOREIGN KEY (collection) REFERENCES collections(id) ON DELETE CASCADE DEFERRABLE;\n\n\nCREATE OR REPLACE FUNCTION content_slim(_item jsonb) RETURNS jsonb AS $$\n SELECT strip_jsonb(_item - '{id,geometry,collection,type}'::text[], collection_base_item(_item->>'collection')) - '{id,geometry,collection,type}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_dehydrate(content jsonb) RETURNS items AS $$\n SELECT\n content->>'id' as id,\n stac_geom(content) as geometry,\n content->>'collection' as collection,\n stac_datetime(content) as datetime,\n stac_end_datetime(content) as end_datetime,\n content_slim(content) as content\n ;\n$$ LANGUAGE SQL STABLE;\n\nCREATE OR REPLACE FUNCTION include_field(f text, fields jsonb DEFAULT '{}'::jsonb) RETURNS boolean AS $$\nDECLARE\n includes jsonb := fields->'include';\n excludes jsonb := fields->'exclude';\nBEGIN\n IF f IS NULL THEN\n RETURN NULL;\n END IF;\n\n\n IF\n jsonb_typeof(excludes) = 'array'\n AND jsonb_array_length(excludes)>0\n AND excludes ? f\n THEN\n RETURN FALSE;\n END IF;\n\n IF\n (\n jsonb_typeof(includes) = 'array'\n AND jsonb_array_length(includes) > 0\n AND includes ? f\n ) OR\n (\n includes IS NULL\n OR jsonb_typeof(includes) = 'null'\n OR jsonb_array_length(includes) = 0\n )\n THEN\n RETURN TRUE;\n END IF;\n\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nDROP FUNCTION IF EXISTS content_hydrate(jsonb, jsonb, jsonb);\nCREATE OR REPLACE FUNCTION content_hydrate(\n _item jsonb,\n _base_item jsonb,\n fields jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\n SELECT merge_jsonb(\n jsonb_fields(_item, fields),\n jsonb_fields(_base_item, fields)\n );\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION content_hydrate(_item items, _collection collections, fields jsonb DEFAULT '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\n content jsonb;\n base_item jsonb := _collection.base_item;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry, 20)::jsonb;\n END IF;\n output := content_hydrate(\n jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'collection', _item.collection,\n 'type', 'Feature'\n ) || _item.content,\n _collection.base_item,\n fields\n );\n\n RETURN output;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_nonhydrated(\n _item items,\n fields jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry, 20)::jsonb;\n END IF;\n output := jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'collection', _item.collection,\n 'type', 'Feature'\n ) || _item.content;\n RETURN output;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_hydrate(_item items, fields jsonb DEFAULT '{}'::jsonb) RETURNS jsonb AS $$\n SELECT content_hydrate(\n _item,\n (SELECT c FROM collections c WHERE id=_item.collection LIMIT 1),\n fields\n );\n$$ LANGUAGE SQL STABLE;\n\n\nCREATE UNLOGGED TABLE items_staging (\n content JSONB NOT NULL\n);\nCREATE UNLOGGED TABLE items_staging_ignore (\n content JSONB NOT NULL\n);\nCREATE UNLOGGED TABLE items_staging_upsert (\n content JSONB NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION items_staging_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p record;\n _partitions text[];\n ts timestamptz := clock_timestamp();\nBEGIN\n RAISE NOTICE 'Creating Partitions. %', clock_timestamp() - ts;\n WITH ranges AS (\n SELECT\n n.content->>'collection' as collection,\n stac_daterange(n.content->'properties') as dtr\n FROM newdata n\n ), p AS (\n SELECT\n collection,\n lower(dtr) as datetime,\n upper(dtr) as end_datetime,\n (partition_name(\n collection,\n lower(dtr)\n )).partition_name as name\n FROM ranges\n )\n INSERT INTO partitions (collection, datetime_range, end_datetime_range)\n SELECT\n collection,\n tstzrange(min(datetime), max(datetime), '[]') as datetime_range,\n tstzrange(min(end_datetime), max(end_datetime), '[]') as end_datetime_range\n FROM p\n GROUP BY collection, name\n ON CONFLICT (name) DO UPDATE SET\n datetime_range = EXCLUDED.datetime_range,\n end_datetime_range = EXCLUDED.end_datetime_range\n ;\n\n RAISE NOTICE 'Doing the insert. %', clock_timestamp() - ts;\n IF TG_TABLE_NAME = 'items_staging' THEN\n INSERT INTO items\n SELECT\n (content_dehydrate(content)).*\n FROM newdata;\n DELETE FROM items_staging;\n ELSIF TG_TABLE_NAME = 'items_staging_ignore' THEN\n INSERT INTO items\n SELECT\n (content_dehydrate(content)).*\n FROM newdata\n ON CONFLICT DO NOTHING;\n DELETE FROM items_staging_ignore;\n ELSIF TG_TABLE_NAME = 'items_staging_upsert' THEN\n WITH staging_formatted AS (\n SELECT (content_dehydrate(content)).* FROM newdata\n ), deletes AS (\n DELETE FROM items i USING staging_formatted s\n WHERE\n i.id = s.id\n AND i.collection = s.collection\n AND i IS DISTINCT FROM s\n RETURNING i.id, i.collection\n )\n INSERT INTO items\n SELECT s.* FROM\n staging_formatted s\n JOIN deletes d\n USING (id, collection);\n DELETE FROM items_staging_upsert;\n END IF;\n RAISE NOTICE 'Done. %', clock_timestamp() - ts;\n\n RETURN NULL;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER items_staging_insert_trigger AFTER INSERT ON items_staging REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\nCREATE TRIGGER items_staging_insert_ignore_trigger AFTER INSERT ON items_staging_ignore REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\nCREATE TRIGGER items_staging_insert_upsert_trigger AFTER INSERT ON items_staging_upsert REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\n\n\n\nCREATE OR REPLACE FUNCTION item_by_id(_id text, _collection text DEFAULT NULL) RETURNS items AS\n$$\nDECLARE\n i items%ROWTYPE;\nBEGIN\n SELECT * INTO i FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection) LIMIT 1;\n RETURN i;\nEND;\n$$ LANGUAGE PLPGSQL STABLE SECURITY DEFINER SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION get_item(_id text, _collection text DEFAULT NULL) RETURNS jsonb AS $$\n SELECT content_hydrate(items) FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection);\n$$ LANGUAGE SQL STABLE SECURITY DEFINER SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION delete_item(_id text, _collection text DEFAULT NULL) RETURNS VOID AS $$\nDECLARE\nout items%ROWTYPE;\nBEGIN\n DELETE FROM items WHERE id = _id AND (_collection IS NULL OR collection=_collection) RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n--/*\nCREATE OR REPLACE FUNCTION create_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION update_item(content jsonb) RETURNS VOID AS $$\nDECLARE\n old items %ROWTYPE;\n out items%ROWTYPE;\nBEGIN\n PERFORM delete_item(content->>'id', content->>'collection');\n PERFORM create_item(content);\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION create_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION collection_bbox(id text) RETURNS jsonb AS $$\n SELECT (replace(replace(replace(st_extent(geometry)::text,'BOX(','[['),')',']]'),' ',','))::jsonb\n FROM items WHERE collection=$1;\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION collection_temporal_extent(id text) RETURNS jsonb AS $$\n SELECT to_jsonb(array[array[min(datetime)::text, max(datetime)::text]])\n FROM items WHERE collection=$1;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection_extents() RETURNS VOID AS $$\nUPDATE collections SET\n content = content ||\n jsonb_build_object(\n 'extent', jsonb_build_object(\n 'spatial', jsonb_build_object(\n 'bbox', collection_bbox(collections.id)\n ),\n 'temporal', jsonb_build_object(\n 'interval', collection_temporal_extent(collections.id)\n )\n )\n )\n;\n$$ LANGUAGE SQL;\nCREATE VIEW partition_steps AS\nSELECT\n name,\n date_trunc('month',lower(datetime_range)) as sdate,\n date_trunc('month', upper(datetime_range)) + '1 month'::interval as edate\n FROM partitions WHERE datetime_range IS NOT NULL AND datetime_range != 'empty'::tstzrange\n ORDER BY datetime_range ASC\n;\n\nCREATE OR REPLACE FUNCTION chunker(\n IN _where text,\n OUT s timestamptz,\n OUT e timestamptz\n) RETURNS SETOF RECORD AS $$\nDECLARE\n explain jsonb;\nBEGIN\n IF _where IS NULL THEN\n _where := ' TRUE ';\n END IF;\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s;', _where)\n INTO explain;\n\n RETURN QUERY\n WITH t AS (\n SELECT j->>0 as p FROM\n jsonb_path_query(\n explain,\n 'strict $.**.\"Relation Name\" ? (@ != null)'\n ) j\n ),\n parts AS (\n SELECT sdate, edate FROM t JOIN partition_steps ON (t.p = name)\n ),\n times AS (\n SELECT sdate FROM parts\n UNION\n SELECT edate FROM parts\n ),\n uniq AS (\n SELECT DISTINCT sdate FROM times ORDER BY sdate\n ),\n last AS (\n SELECT sdate, lead(sdate, 1) over () as edate FROM uniq\n )\n SELECT sdate, edate FROM last WHERE edate IS NOT NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION partition_queries(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN partitions text[] DEFAULT NULL\n) RETURNS SETOF text AS $$\nDECLARE\n query text;\n sdate timestamptz;\n edate timestamptz;\nBEGIN\nIF _where IS NULL OR trim(_where) = '' THEN\n _where = ' TRUE ';\nEND IF;\nRAISE NOTICE 'Getting chunks for % %', _where, _orderby;\nIF _orderby ILIKE 'datetime d%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 DESC LOOP\n RETURN NEXT format($q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n $q$,\n sdate,\n edate,\n _where,\n _orderby\n );\n END LOOP;\nELSIF _orderby ILIKE 'datetime a%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 ASC LOOP\n RETURN NEXT format($q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n $q$,\n sdate,\n edate,\n _where,\n _orderby\n );\n END LOOP;\nELSE\n query := format($q$\n SELECT * FROM items\n WHERE %s\n ORDER BY %s\n $q$, _where, _orderby\n );\n\n RETURN NEXT query;\n RETURN;\nEND IF;\n\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION partition_query_view(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN _limit int DEFAULT 10\n) RETURNS text AS $$\n WITH p AS (\n SELECT * FROM partition_queries(_where, _orderby) p\n )\n SELECT\n CASE WHEN EXISTS (SELECT 1 FROM p) THEN\n (SELECT format($q$\n SELECT * FROM (\n %s\n ) total LIMIT %s\n $q$,\n string_agg(\n format($q$ SELECT * FROM ( %s ) AS sub $q$, p),\n '\n UNION ALL\n '\n ),\n _limit\n ))\n ELSE NULL\n END FROM p;\n$$ LANGUAGE SQL IMMUTABLE;\n\n\n\n\nCREATE OR REPLACE FUNCTION stac_search_to_where(j jsonb) RETURNS text AS $$\nDECLARE\n where_segments text[];\n _where text;\n dtrange tstzrange;\n collections text[];\n geom geometry;\n sdate timestamptz;\n edate timestamptz;\n filterlang text;\n filter jsonb := j->'filter';\nBEGIN\n IF j ? 'ids' THEN\n where_segments := where_segments || format('id = ANY (%L) ', to_text_array(j->'ids'));\n END IF;\n\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n where_segments := where_segments || format('collection = ANY (%L) ', collections);\n END IF;\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n\n where_segments := where_segments || format(' datetime <= %L::timestamptz AND end_datetime >= %L::timestamptz ',\n edate,\n sdate\n );\n END IF;\n\n geom := stac_geom(j);\n IF geom IS NOT NULL THEN\n where_segments := where_segments || format('st_intersects(geometry, %L)',geom);\n END IF;\n\n filterlang := COALESCE(\n j->>'filter-lang',\n get_setting('default-filter-lang', j->'conf')\n );\n IF NOT filter @? '$.**.op' THEN\n filterlang := 'cql-json';\n END IF;\n\n IF filterlang NOT IN ('cql-json','cql2-json') AND j ? 'filter' THEN\n RAISE EXCEPTION '% is not a supported filter-lang. Please use cql-json or cql2-json.', filterlang;\n END IF;\n\n IF j ? 'query' AND j ? 'filter' THEN\n RAISE EXCEPTION 'Can only use either query or filter at one time.';\n END IF;\n\n IF j ? 'query' THEN\n filter := query_to_cql2(j->'query');\n ELSIF filterlang = 'cql-json' THEN\n filter := cql1_to_cql2(filter);\n END IF;\n RAISE NOTICE 'FILTER: %', filter;\n where_segments := where_segments || cql2_query(filter);\n IF cardinality(where_segments) < 1 THEN\n RETURN ' TRUE ';\n END IF;\n\n _where := array_to_string(array_remove(where_segments, NULL), ' AND ');\n\n IF _where IS NULL OR BTRIM(_where) = '' THEN\n RETURN ' TRUE ';\n END IF;\n RETURN _where;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION parse_sort_dir(_dir text, reverse boolean default false) RETURNS text AS $$\n WITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n ) SELECT\n CASE\n WHEN NOT reverse THEN d\n WHEN d = 'ASC' THEN 'DESC'\n WHEN d = 'DESC' THEN 'ASC'\n END\n FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION sort_dir_to_op(_dir text, prev boolean default false) RETURNS text AS $$\n WITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n ) SELECT\n CASE\n WHEN d = 'ASC' AND prev THEN '<='\n WHEN d = 'DESC' AND prev THEN '>='\n WHEN d = 'ASC' THEN '>='\n WHEN d = 'DESC' THEN '<='\n END\n FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION sort_sqlorderby(\n _search jsonb DEFAULT NULL,\n reverse boolean DEFAULT FALSE\n) RETURNS text AS $$\n WITH sortby AS (\n SELECT coalesce(_search->'sortby','[{\"field\":\"datetime\", \"direction\":\"desc\"}]') as sort\n ), withid AS (\n SELECT CASE\n WHEN sort @? '$[*] ? (@.field == \"id\")' THEN sort\n ELSE sort || '[{\"field\":\"id\", \"direction\":\"desc\"}]'::jsonb\n END as sort\n FROM sortby\n ), withid_rows AS (\n SELECT jsonb_array_elements(sort) as value FROM withid\n ),sorts AS (\n SELECT\n coalesce(\n -- field_orderby((items_path(value->>'field')).path_txt),\n (queryable(value->>'field')).expression\n ) as key,\n parse_sort_dir(value->>'direction', reverse) as dir\n FROM withid_rows\n )\n SELECT array_to_string(\n array_agg(concat(key, ' ', dir)),\n ', '\n ) FROM sorts;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_sort_dir(sort_item jsonb) RETURNS text AS $$\n SELECT CASE WHEN sort_item->>'direction' ILIKE 'desc%' THEN 'DESC' ELSE 'ASC' END;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION get_token_filter(_search jsonb = '{}'::jsonb, token_rec jsonb DEFAULT NULL) RETURNS text AS $$\nDECLARE\n token_id text;\n filters text[] := '{}'::text[];\n prev boolean := TRUE;\n field text;\n dir text;\n sort record;\n orfilters text[] := '{}'::text[];\n andfilters text[] := '{}'::text[];\n output text;\n token_where text;\nBEGIN\n RAISE NOTICE 'Getting Token Filter. % %', _search, token_rec;\n -- If no token provided return NULL\n IF token_rec IS NULL THEN\n IF NOT (_search ? 'token' AND\n (\n (_search->>'token' ILIKE 'prev:%')\n OR\n (_search->>'token' ILIKE 'next:%')\n )\n ) THEN\n RETURN NULL;\n END IF;\n prev := (_search->>'token' ILIKE 'prev:%');\n token_id := substr(_search->>'token', 6);\n SELECT to_jsonb(items) INTO token_rec\n FROM items WHERE id=token_id;\n END IF;\n RAISE NOTICE 'TOKEN ID: % %', token_rec, token_rec->'id';\n\n CREATE TEMP TABLE sorts (\n _row int GENERATED ALWAYS AS IDENTITY NOT NULL,\n _field text PRIMARY KEY,\n _dir text NOT NULL,\n _val text\n ) ON COMMIT DROP;\n\n -- Make sure we only have distinct columns to sort with taking the first one we get\n INSERT INTO sorts (_field, _dir)\n SELECT\n (queryable(value->>'field')).expression,\n get_sort_dir(value)\n FROM\n jsonb_array_elements(coalesce(_search->'sortby','[{\"field\":\"datetime\",\"direction\":\"desc\"}]'))\n ON CONFLICT DO NOTHING\n ;\n RAISE NOTICE 'sorts 1: %', (SELECT jsonb_agg(to_json(sorts)) FROM sorts);\n -- Get the first sort direction provided. As the id is a primary key, if there are any\n -- sorts after id they won't do anything, so make sure that id is the last sort item.\n SELECT _dir INTO dir FROM sorts ORDER BY _row ASC LIMIT 1;\n IF EXISTS (SELECT 1 FROM sorts WHERE _field = 'id') THEN\n DELETE FROM sorts WHERE _row > (SELECT _row FROM sorts WHERE _field = 'id' ORDER BY _row ASC);\n ELSE\n INSERT INTO sorts (_field, _dir) VALUES ('id', dir);\n END IF;\n\n -- Add value from looked up item to the sorts table\n UPDATE sorts SET _val=quote_literal(token_rec->>_field);\n\n -- Check if all sorts are the same direction and use row comparison\n -- to filter\n RAISE NOTICE 'sorts 2: %', (SELECT jsonb_agg(to_json(sorts)) FROM sorts);\n\n IF (SELECT count(DISTINCT _dir) FROM sorts) = 1 THEN\n SELECT format(\n '(%s) %s (%s)',\n concat_ws(', ', VARIADIC array_agg(quote_ident(_field))),\n CASE WHEN (prev AND dir = 'ASC') OR (NOT prev AND dir = 'DESC') THEN '<' ELSE '>' END,\n concat_ws(', ', VARIADIC array_agg(_val))\n ) INTO output FROM sorts\n WHERE token_rec ? _field\n ;\n ELSE\n FOR sort IN SELECT * FROM sorts ORDER BY _row asc LOOP\n RAISE NOTICE 'SORT: %', sort;\n IF sort._row = 1 THEN\n orfilters := orfilters || format('(%s %s %s)',\n quote_ident(sort._field),\n CASE WHEN (prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC') THEN '<' ELSE '>' END,\n sort._val\n );\n ELSE\n orfilters := orfilters || format('(%s AND %s %s %s)',\n array_to_string(andfilters, ' AND '),\n quote_ident(sort._field),\n CASE WHEN (prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC') THEN '<' ELSE '>' END,\n sort._val\n );\n\n END IF;\n andfilters := andfilters || format('%s = %s',\n quote_ident(sort._field),\n sort._val\n );\n END LOOP;\n output := array_to_string(orfilters, ' OR ');\n END IF;\n DROP TABLE IF EXISTS sorts;\n token_where := concat('(',coalesce(output,'true'),')');\n IF trim(token_where) = '' THEN\n token_where := NULL;\n END IF;\n RAISE NOTICE 'TOKEN_WHERE: |%|',token_where;\n RETURN token_where;\n END;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION search_tohash(jsonb) RETURNS jsonb AS $$\n SELECT $1 - '{token,limit,context,includes,excludes}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION search_hash(jsonb, jsonb) RETURNS text AS $$\n SELECT md5(concat(search_tohash($1)::text,$2::text));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TABLE IF NOT EXISTS searches(\n hash text GENERATED ALWAYS AS (search_hash(search, metadata)) STORED PRIMARY KEY,\n search jsonb NOT NULL,\n _where text,\n orderby text,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n metadata jsonb DEFAULT '{}'::jsonb NOT NULL\n);\nCREATE TABLE IF NOT EXISTS search_wheres(\n id bigint generated always as identity primary key,\n _where text NOT NULL,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n statslastupdated timestamptz,\n estimated_count bigint,\n estimated_cost float,\n time_to_estimate float,\n total_count bigint,\n time_to_count float,\n partitions text[]\n);\n\nCREATE INDEX IF NOT EXISTS search_wheres_partitions ON search_wheres USING GIN (partitions);\nCREATE UNIQUE INDEX IF NOT EXISTS search_wheres_where ON search_wheres ((md5(_where)));\n\nCREATE OR REPLACE FUNCTION where_stats(inwhere text, updatestats boolean default false, conf jsonb default null) RETURNS search_wheres AS $$\nDECLARE\n t timestamptz;\n i interval;\n explain_json jsonb;\n partitions text[];\n sw search_wheres%ROWTYPE;\n inwhere_hash text := md5(inwhere);\n _context text := lower(context(conf));\n _stats_ttl interval := context_stats_ttl(conf);\n _estimated_cost float := context_estimated_cost(conf);\n _estimated_count int := context_estimated_count(conf);\nBEGIN\n IF _context = 'off' THEN\n sw._where := inwhere;\n return sw;\n END IF;\n\n SELECT * INTO sw FROM search_wheres WHERE md5(_where)=inwhere_hash FOR UPDATE;\n\n -- Update statistics if explicitly set, if statistics do not exist, or statistics ttl has expired\n IF NOT updatestats THEN\n RAISE NOTICE 'Checking if update is needed for: % .', inwhere;\n RAISE NOTICE 'Stats Last Updated: %', sw.statslastupdated;\n RAISE NOTICE 'TTL: %, Age: %', _stats_ttl, now() - sw.statslastupdated;\n RAISE NOTICE 'Context: %, Existing Total: %', _context, sw.total_count;\n IF\n sw.statslastupdated IS NULL\n OR (now() - sw.statslastupdated) > _stats_ttl\n OR (context(conf) != 'off' AND sw.total_count IS NULL)\n THEN\n updatestats := TRUE;\n END IF;\n END IF;\n\n sw._where := inwhere;\n sw.lastused := now();\n sw.usecount := coalesce(sw.usecount,0) + 1;\n\n IF NOT updatestats THEN\n UPDATE search_wheres SET\n lastused = sw.lastused,\n usecount = sw.usecount\n WHERE md5(_where) = inwhere_hash\n RETURNING * INTO sw\n ;\n RETURN sw;\n END IF;\n\n -- Use explain to get estimated count/cost and a list of the partitions that would be hit by the query\n t := clock_timestamp();\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s', inwhere)\n INTO explain_json;\n RAISE NOTICE 'Time for just the explain: %', clock_timestamp() - t;\n i := clock_timestamp() - t;\n\n sw.statslastupdated := now();\n sw.estimated_count := explain_json->0->'Plan'->'Plan Rows';\n sw.estimated_cost := explain_json->0->'Plan'->'Total Cost';\n sw.time_to_estimate := extract(epoch from i);\n\n RAISE NOTICE 'ESTIMATED_COUNT: % < %', sw.estimated_count, _estimated_count;\n RAISE NOTICE 'ESTIMATED_COST: % < %', sw.estimated_cost, _estimated_cost;\n\n -- Do a full count of rows if context is set to on or if auto is set and estimates are low enough\n IF\n _context = 'on'\n OR\n ( _context = 'auto' AND\n (\n sw.estimated_count < _estimated_count\n AND\n sw.estimated_cost < _estimated_cost\n )\n )\n THEN\n t := clock_timestamp();\n RAISE NOTICE 'Calculating actual count...';\n EXECUTE format(\n 'SELECT count(*) FROM items WHERE %s',\n inwhere\n ) INTO sw.total_count;\n i := clock_timestamp() - t;\n RAISE NOTICE 'Actual Count: % -- %', sw.total_count, i;\n sw.time_to_count := extract(epoch FROM i);\n ELSE\n sw.total_count := NULL;\n sw.time_to_count := NULL;\n END IF;\n\n\n INSERT INTO search_wheres\n (_where, lastused, usecount, statslastupdated, estimated_count, estimated_cost, time_to_estimate, partitions, total_count, time_to_count)\n SELECT sw._where, sw.lastused, sw.usecount, sw.statslastupdated, sw.estimated_count, sw.estimated_cost, sw.time_to_estimate, sw.partitions, sw.total_count, sw.time_to_count\n ON CONFLICT ((md5(_where)))\n DO UPDATE\n SET\n lastused = sw.lastused,\n usecount = sw.usecount,\n statslastupdated = sw.statslastupdated,\n estimated_count = sw.estimated_count,\n estimated_cost = sw.estimated_cost,\n time_to_estimate = sw.time_to_estimate,\n total_count = sw.total_count,\n time_to_count = sw.time_to_count\n ;\n RETURN sw;\nEND;\n$$ LANGUAGE PLPGSQL ;\n\n\n\nDROP FUNCTION IF EXISTS search_query;\nCREATE OR REPLACE FUNCTION search_query(\n _search jsonb = '{}'::jsonb,\n updatestats boolean = false,\n _metadata jsonb = '{}'::jsonb\n) RETURNS searches AS $$\nDECLARE\n search searches%ROWTYPE;\n pexplain jsonb;\n t timestamptz;\n i interval;\nBEGIN\n SELECT * INTO search FROM searches\n WHERE hash=search_hash(_search, _metadata) FOR UPDATE;\n\n -- Calculate the where clause if not already calculated\n IF search._where IS NULL THEN\n search._where := stac_search_to_where(_search);\n END IF;\n\n -- Calculate the order by clause if not already calculated\n IF search.orderby IS NULL THEN\n search.orderby := sort_sqlorderby(_search);\n END IF;\n\n PERFORM where_stats(search._where, updatestats, _search->'conf');\n\n search.lastused := now();\n search.usecount := coalesce(search.usecount, 0) + 1;\n INSERT INTO searches (search, _where, orderby, lastused, usecount, metadata)\n VALUES (_search, search._where, search.orderby, search.lastused, search.usecount, _metadata)\n ON CONFLICT (hash) DO\n UPDATE SET\n _where = EXCLUDED._where,\n orderby = EXCLUDED.orderby,\n lastused = EXCLUDED.lastused,\n usecount = EXCLUDED.usecount,\n metadata = EXCLUDED.metadata\n RETURNING * INTO search\n ;\n RETURN search;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION search(_search jsonb = '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n searches searches%ROWTYPE;\n _where text;\n token_where text;\n full_where text;\n orderby text;\n query text;\n token_type text := substr(_search->>'token',1,4);\n _limit int := coalesce((_search->>'limit')::int, 10);\n curs refcursor;\n cntr int := 0;\n iter_record items%ROWTYPE;\n first_record jsonb;\n first_item items%ROWTYPE;\n last_item items%ROWTYPE;\n last_record jsonb;\n out_records jsonb := '[]'::jsonb;\n prev_query text;\n next text;\n prev_id text;\n has_next boolean := false;\n has_prev boolean := false;\n prev text;\n total_count bigint;\n context jsonb;\n collection jsonb;\n includes text[];\n excludes text[];\n exit_flag boolean := FALSE;\n batches int := 0;\n timer timestamptz := clock_timestamp();\n pstart timestamptz;\n pend timestamptz;\n pcurs refcursor;\n search_where search_wheres%ROWTYPE;\n id text;\nBEGIN\nCREATE TEMP TABLE results (content jsonb) ON COMMIT DROP;\n-- if ids is set, short circuit and just use direct ids query for each id\n-- skip any paging or caching\n-- hard codes ordering in the same order as the array of ids\nIF _search ? 'ids' THEN\n INSERT INTO results\n SELECT\n CASE WHEN _search->'conf'->>'nohydrate' IS NOT NULL AND (_search->'conf'->>'nohydrate')::boolean = true THEN\n content_nonhydrated(items, _search->'fields')\n ELSE\n content_hydrate(items, _search->'fields')\n END\n FROM items WHERE\n items.id = ANY(to_text_array(_search->'ids'))\n AND\n CASE WHEN _search ? 'collections' THEN\n items.collection = ANY(to_text_array(_search->'collections'))\n ELSE TRUE\n END\n ORDER BY items.datetime desc, items.id desc\n ;\n SELECT INTO total_count count(*) FROM results;\nELSE\n searches := search_query(_search);\n _where := searches._where;\n orderby := searches.orderby;\n search_where := where_stats(_where);\n total_count := coalesce(search_where.total_count, search_where.estimated_count);\n\n IF token_type='prev' THEN\n token_where := get_token_filter(_search, null::jsonb);\n orderby := sort_sqlorderby(_search, TRUE);\n END IF;\n IF token_type='next' THEN\n token_where := get_token_filter(_search, null::jsonb);\n END IF;\n\n full_where := concat_ws(' AND ', _where, token_where);\n RAISE NOTICE 'FULL QUERY % %', full_where, clock_timestamp()-timer;\n timer := clock_timestamp();\n\n FOR query IN SELECT partition_queries(full_where, orderby, search_where.partitions) LOOP\n timer := clock_timestamp();\n query := format('%s LIMIT %s', query, _limit + 1);\n RAISE NOTICE 'Partition Query: %', query;\n batches := batches + 1;\n -- curs = create_cursor(query);\n OPEN curs FOR EXECUTE query;\n LOOP\n FETCH curs into iter_record;\n EXIT WHEN NOT FOUND;\n cntr := cntr + 1;\n\n IF _search->'conf'->>'nohydrate' IS NOT NULL AND (_search->'conf'->>'nohydrate')::boolean = true THEN\n last_record := content_nonhydrated(iter_record, _search->'fields');\n ELSE\n last_record := content_hydrate(iter_record, _search->'fields');\n END IF;\n last_item := iter_record;\n IF cntr = 1 THEN\n first_item := last_item;\n first_record := last_record;\n END IF;\n IF cntr <= _limit THEN\n INSERT INTO results (content) VALUES (last_record);\n ELSIF cntr > _limit THEN\n has_next := true;\n exit_flag := true;\n EXIT;\n END IF;\n END LOOP;\n CLOSE curs;\n RAISE NOTICE 'Query took %.', clock_timestamp()-timer;\n timer := clock_timestamp();\n EXIT WHEN exit_flag;\n END LOOP;\n RAISE NOTICE 'Scanned through % partitions.', batches;\nEND IF;\n\nSELECT jsonb_agg(content) INTO out_records FROM results WHERE content is not NULL;\n\nDROP TABLE results;\n\n\n-- Flip things around if this was the result of a prev token query\nIF token_type='prev' THEN\n out_records := flip_jsonb_array(out_records);\n first_record := last_record;\nEND IF;\n\n-- If this query has a token, see if there is data before the first record\nIF _search ? 'token' THEN\n prev_query := format(\n 'SELECT 1 FROM items WHERE %s LIMIT 1',\n concat_ws(\n ' AND ',\n _where,\n trim(get_token_filter(_search, to_jsonb(first_item)))\n )\n );\n RAISE NOTICE 'Query to get previous record: % --- %', prev_query, first_record;\n EXECUTE prev_query INTO has_prev;\n IF FOUND and has_prev IS NOT NULL THEN\n RAISE NOTICE 'Query results from prev query: %', has_prev;\n has_prev := TRUE;\n END IF;\nEND IF;\nhas_prev := COALESCE(has_prev, FALSE);\n\nIF has_prev THEN\n prev := out_records->0->>'id';\nEND IF;\nIF has_next OR token_type='prev' THEN\n next := out_records->-1->>'id';\nEND IF;\n\nIF context(_search->'conf') != 'off' THEN\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'matched', total_count,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\nELSE\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\nEND IF;\n\ncollection := jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb),\n 'next', next,\n 'prev', prev,\n 'context', context\n);\n\nRETURN collection;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER SET SEARCH_PATH TO pgstac, public;\n\n\nCREATE OR REPLACE FUNCTION search_cursor(_search jsonb = '{}'::jsonb) RETURNS refcursor AS $$\nDECLARE\n curs refcursor;\n searches searches%ROWTYPE;\n _where text;\n _orderby text;\n q text;\n\nBEGIN\n searches := search_query(_search);\n _where := searches._where;\n _orderby := searches.orderby;\n\n OPEN curs FOR\n WITH p AS (\n SELECT * FROM partition_queries(_where, _orderby) p\n )\n SELECT\n CASE WHEN EXISTS (SELECT 1 FROM p) THEN\n (SELECT format($q$\n SELECT * FROM (\n %s\n ) total\n $q$,\n string_agg(\n format($q$ SELECT * FROM ( %s ) AS sub $q$, p),\n '\n UNION ALL\n '\n )\n ))\n ELSE NULL\n END FROM p;\n RETURN curs;\nEND;\n$$ LANGUAGE PLPGSQL;\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION tileenvelope(zoom int, x int, y int) RETURNS geometry AS $$\nWITH t AS (\n SELECT\n 20037508.3427892 as merc_max,\n -20037508.3427892 as merc_min,\n (2 * 20037508.3427892) / (2 ^ zoom) as tile_size\n)\nSELECT st_makeenvelope(\n merc_min + (tile_size * x),\n merc_max - (tile_size * (y + 1)),\n merc_min + (tile_size * (x + 1)),\n merc_max - (tile_size * y),\n 3857\n) FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;DROP FUNCTION IF EXISTS mercgrid;\n\n\nCREATE OR REPLACE FUNCTION ftime() RETURNS interval as $$\nSELECT age(clock_timestamp(), transaction_timestamp());\n$$ LANGUAGE SQL;\nSET SEARCH_PATH to pgstac, public;\n\nDROP FUNCTION IF EXISTS geometrysearch;\nCREATE OR REPLACE FUNCTION geometrysearch(\n IN geom geometry,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE, -- Return as soon as the passed in geometry is full covered\n IN skipcovered boolean DEFAULT TRUE -- Skip any items that would show up completely under the previous items\n) RETURNS jsonb AS $$\nDECLARE\n search searches%ROWTYPE;\n curs refcursor;\n _where text;\n query text;\n iter_record items%ROWTYPE;\n out_records jsonb := '{}'::jsonb[];\n exit_flag boolean := FALSE;\n counter int := 1;\n scancounter int := 1;\n remaining_limit int := _scanlimit;\n tilearea float;\n unionedgeom geometry;\n clippedgeom geometry;\n unionedgeom_area float := 0;\n prev_area float := 0;\n excludes text[];\n includes text[];\n\nBEGIN\n DROP TABLE IF EXISTS pgstac_results;\n CREATE TEMP TABLE pgstac_results (content jsonb) ON COMMIT DROP;\n\n -- If skipcovered is true then you will always want to exit when the passed in geometry is full\n IF skipcovered THEN\n exitwhenfull := TRUE;\n END IF;\n\n SELECT * INTO search FROM searches WHERE hash=queryhash;\n\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Search with Query Hash % Not Found', queryhash;\n END IF;\n\n tilearea := st_area(geom);\n _where := format('%s AND st_intersects(geometry, %L::geometry)', search._where, geom);\n\n\n FOR query IN SELECT * FROM partition_queries(_where, search.orderby) LOOP\n query := format('%s LIMIT %L', query, remaining_limit);\n RAISE NOTICE '%', query;\n OPEN curs FOR EXECUTE query;\n LOOP\n FETCH curs INTO iter_record;\n EXIT WHEN NOT FOUND;\n IF exitwhenfull OR skipcovered THEN -- If we are not using exitwhenfull or skipcovered, we do not need to do expensive geometry operations\n clippedgeom := st_intersection(geom, iter_record.geometry);\n\n IF unionedgeom IS NULL THEN\n unionedgeom := clippedgeom;\n ELSE\n unionedgeom := st_union(unionedgeom, clippedgeom);\n END IF;\n\n unionedgeom_area := st_area(unionedgeom);\n\n IF skipcovered AND prev_area = unionedgeom_area THEN\n scancounter := scancounter + 1;\n CONTINUE;\n END IF;\n\n prev_area := unionedgeom_area;\n\n RAISE NOTICE '% % % %', unionedgeom_area/tilearea, counter, scancounter, ftime();\n END IF;\n RAISE NOTICE '% %', iter_record, content_hydrate(iter_record, fields);\n INSERT INTO pgstac_results (content) VALUES (content_hydrate(iter_record, fields));\n\n IF counter >= _limit\n OR scancounter > _scanlimit\n OR ftime() > _timelimit\n OR (exitwhenfull AND unionedgeom_area >= tilearea)\n THEN\n exit_flag := TRUE;\n EXIT;\n END IF;\n counter := counter + 1;\n scancounter := scancounter + 1;\n\n END LOOP;\n CLOSE curs;\n EXIT WHEN exit_flag;\n remaining_limit := _scanlimit - scancounter;\n END LOOP;\n\n SELECT jsonb_agg(content) INTO out_records FROM pgstac_results WHERE content IS NOT NULL;\n\n RETURN jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb)\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDROP FUNCTION IF EXISTS geojsonsearch;\nCREATE OR REPLACE FUNCTION geojsonsearch(\n IN geojson jsonb,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_geomfromgeojson(geojson),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS xyzsearch;\nCREATE OR REPLACE FUNCTION xyzsearch(\n IN _x int,\n IN _y int,\n IN _z int,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_transform(tileenvelope(_z, _x, _y), 4326),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nGRANT ALL ON SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac to pgstac_admin;\n\n-- pgstac_read role limited to using function apis\nGRANT EXECUTE ON FUNCTION search TO pgstac_read;\nGRANT EXECUTE ON FUNCTION search_query TO pgstac_read;\nGRANT EXECUTE ON FUNCTION item_by_id TO pgstac_read;\nGRANT EXECUTE ON FUNCTION get_item TO pgstac_read;\n\nGRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_ingest;\nGRANT USAGE ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_ingest;\nSELECT set_version('0.6.5');\n" }, { "path": "src/pgstac/migrations/pgstac.0.6.6-0.6.7.sql", "content": "SET SEARCH_PATH to pgstac, public;\nset check_function_bodies = off;\n\nCREATE OR REPLACE FUNCTION search(_search jsonb = '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n searches searches%ROWTYPE;\n _where text;\n token_where text;\n full_where text;\n orderby text;\n query text;\n token_type text := substr(_search->>'token',1,4);\n _limit int := coalesce((_search->>'limit')::int, 10);\n curs refcursor;\n cntr int := 0;\n iter_record items%ROWTYPE;\n first_record jsonb;\n first_item items%ROWTYPE;\n last_item items%ROWTYPE;\n last_record jsonb;\n out_records jsonb := '[]'::jsonb;\n prev_query text;\n next text;\n prev_id text;\n has_next boolean := false;\n has_prev boolean := false;\n prev text;\n total_count bigint;\n context jsonb;\n collection jsonb;\n includes text[];\n excludes text[];\n exit_flag boolean := FALSE;\n batches int := 0;\n timer timestamptz := clock_timestamp();\n pstart timestamptz;\n pend timestamptz;\n pcurs refcursor;\n search_where search_wheres%ROWTYPE;\n id text;\nBEGIN\nCREATE TEMP TABLE results (content jsonb) ON COMMIT DROP;\n-- if ids is set, short circuit and just use direct ids query for each id\n-- skip any paging or caching\n-- hard codes ordering in the same order as the array of ids\nIF _search ? 'ids' THEN\n INSERT INTO results\n SELECT\n CASE WHEN _search->'conf'->>'nohydrate' IS NOT NULL AND (_search->'conf'->>'nohydrate')::boolean = true THEN\n content_nonhydrated(items, _search->'fields')\n ELSE\n content_hydrate(items, _search->'fields')\n END\n FROM items WHERE\n items.id = ANY(to_text_array(_search->'ids'))\n AND\n CASE WHEN _search ? 'collections' THEN\n items.collection = ANY(to_text_array(_search->'collections'))\n ELSE TRUE\n END\n ORDER BY items.datetime desc, items.id desc\n ;\n SELECT INTO total_count count(*) FROM results;\nELSE\n searches := search_query(_search);\n _where := searches._where;\n orderby := searches.orderby;\n search_where := where_stats(_where);\n total_count := coalesce(search_where.total_count, search_where.estimated_count);\n\n IF token_type='prev' THEN\n token_where := get_token_filter(_search, null::jsonb);\n orderby := sort_sqlorderby(_search, TRUE);\n END IF;\n IF token_type='next' THEN\n token_where := get_token_filter(_search, null::jsonb);\n END IF;\n\n full_where := concat_ws(' AND ', _where, token_where);\n RAISE NOTICE 'FULL QUERY % %', full_where, clock_timestamp()-timer;\n timer := clock_timestamp();\n\n FOR query IN SELECT partition_queries(full_where, orderby, search_where.partitions) LOOP\n timer := clock_timestamp();\n query := format('%s LIMIT %s', query, _limit + 1);\n RAISE NOTICE 'Partition Query: %', query;\n batches := batches + 1;\n -- curs = create_cursor(query);\n OPEN curs FOR EXECUTE query;\n LOOP\n FETCH curs into iter_record;\n EXIT WHEN NOT FOUND;\n cntr := cntr + 1;\n\n IF _search->'conf'->>'nohydrate' IS NOT NULL AND (_search->'conf'->>'nohydrate')::boolean = true THEN\n last_record := content_nonhydrated(iter_record, _search->'fields');\n ELSE\n last_record := content_hydrate(iter_record, _search->'fields');\n END IF;\n last_item := iter_record;\n IF cntr = 1 THEN\n first_item := last_item;\n first_record := last_record;\n END IF;\n IF cntr <= _limit THEN\n INSERT INTO results (content) VALUES (last_record);\n ELSIF cntr > _limit THEN\n has_next := true;\n exit_flag := true;\n EXIT;\n END IF;\n END LOOP;\n CLOSE curs;\n RAISE NOTICE 'Query took %.', clock_timestamp()-timer;\n timer := clock_timestamp();\n EXIT WHEN exit_flag;\n END LOOP;\n RAISE NOTICE 'Scanned through % partitions.', batches;\nEND IF;\n\nSELECT jsonb_agg(content) INTO out_records FROM results WHERE content is not NULL;\n\nDROP TABLE results;\n\n\n-- Flip things around if this was the result of a prev token query\nIF token_type='prev' THEN\n out_records := flip_jsonb_array(out_records);\n first_item := last_item;\n first_record := last_record;\nEND IF;\n\n-- If this query has a token, see if there is data before the first record\nIF _search ? 'token' THEN\n prev_query := format(\n 'SELECT 1 FROM items WHERE %s LIMIT 1',\n concat_ws(\n ' AND ',\n _where,\n trim(get_token_filter(_search, to_jsonb(first_item)))\n )\n );\n RAISE NOTICE 'Query to get previous record: % --- %', prev_query, first_record;\n EXECUTE prev_query INTO has_prev;\n IF FOUND and has_prev IS NOT NULL THEN\n RAISE NOTICE 'Query results from prev query: %', has_prev;\n has_prev := TRUE;\n END IF;\nEND IF;\nhas_prev := COALESCE(has_prev, FALSE);\n\nIF has_prev THEN\n prev := out_records->0->>'id';\nEND IF;\nIF has_next OR token_type='prev' THEN\n next := out_records->-1->>'id';\nEND IF;\n\nIF context(_search->'conf') != 'off' THEN\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'matched', total_count,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\nELSE\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\nEND IF;\n\ncollection := jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb),\n 'next', next,\n 'prev', prev,\n 'context', context\n);\n\nRETURN collection;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER SET SEARCH_PATH TO pgstac, public;\n\n\nSELECT set_version('0.6.7');\n" }, { "path": "src/pgstac/migrations/pgstac.0.6.6.sql", "content": "CREATE EXTENSION IF NOT EXISTS postgis;\nCREATE EXTENSION IF NOT EXISTS btree_gist;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_admin;\n CREATE ROLE pgstac_read;\n CREATE ROLE pgstac_ingest;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nGRANT pgstac_admin TO current_user;\n\nCREATE SCHEMA IF NOT EXISTS pgstac AUTHORIZATION pgstac_admin;\n\nALTER ROLE pgstac_admin SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_read SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_ingest SET SEARCH_PATH TO pgstac, public;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\n\nGRANT pgstac_read TO pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\n\nSET ROLE pgstac_admin;\n\nSET SEARCH_PATH TO pgstac, public;\n\n\nCREATE TABLE IF NOT EXISTS migrations (\n version text PRIMARY KEY,\n datetime timestamptz DEFAULT clock_timestamp() NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION get_version() RETURNS text AS $$\n SELECT version FROM pgstac.migrations ORDER BY datetime DESC, version DESC LIMIT 1;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION set_version(text) RETURNS text AS $$\n INSERT INTO pgstac.migrations (version) VALUES ($1)\n ON CONFLICT DO NOTHING\n RETURNING version;\n$$ LANGUAGE SQL;\n\n\nCREATE TABLE IF NOT EXISTS pgstac_settings (\n name text PRIMARY KEY,\n value text NOT NULL\n);\n\nINSERT INTO pgstac_settings (name, value) VALUES\n ('context', 'off'),\n ('context_estimated_count', '100000'),\n ('context_estimated_cost', '100000'),\n ('context_stats_ttl', '1 day'),\n ('default-filter-lang', 'cql2-json'),\n ('additional_properties', 'true')\nON CONFLICT DO NOTHING\n;\n\n\nCREATE OR REPLACE FUNCTION get_setting(IN _setting text, IN conf jsonb DEFAULT NULL) RETURNS text AS $$\nSELECT COALESCE(\n conf->>_setting,\n current_setting(concat('pgstac.',_setting), TRUE),\n (SELECT value FROM pgstac.pgstac_settings WHERE name=_setting)\n);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context(conf jsonb DEFAULT NULL) RETURNS text AS $$\n SELECT pgstac.get_setting('context', conf);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context_estimated_count(conf jsonb DEFAULT NULL) RETURNS int AS $$\n SELECT pgstac.get_setting('context_estimated_count', conf)::int;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_estimated_cost();\nCREATE OR REPLACE FUNCTION context_estimated_cost(conf jsonb DEFAULT NULL) RETURNS float AS $$\n SELECT pgstac.get_setting('context_estimated_cost', conf)::float;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_stats_ttl();\nCREATE OR REPLACE FUNCTION context_stats_ttl(conf jsonb DEFAULT NULL) RETURNS interval AS $$\n SELECT pgstac.get_setting('context_stats_ttl', conf)::interval;\n$$ LANGUAGE SQL;\n\n\nCREATE OR REPLACE FUNCTION notice(VARIADIC text[]) RETURNS boolean AS $$\nDECLARE\ndebug boolean := current_setting('pgstac.debug', true);\nBEGIN\n IF debug THEN\n RAISE NOTICE 'NOTICE FROM FUNC: % >>>>> %', concat_ws(' | ', $1), clock_timestamp();\n RETURN TRUE;\n END IF;\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION empty_arr(ANYARRAY) RETURNS BOOLEAN AS $$\nSELECT CASE\n WHEN $1 IS NULL THEN TRUE\n WHEN cardinality($1)<1 THEN TRUE\nELSE FALSE\nEND;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_intersection(_a ANYARRAY, _b ANYARRAY) RETURNS ANYARRAY AS $$\n SELECT ARRAY ( SELECT unnest(_a) INTERSECT SELECT UNNEST(_b) );\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION array_map_ident(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_ident(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_map_literal(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_literal(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION array_reverse(anyarray) RETURNS anyarray AS $$\nSELECT ARRAY(\n SELECT $1[i]\n FROM generate_subscripts($1,1) AS s(i)\n ORDER BY i DESC\n);\n$$ LANGUAGE SQL STRICT IMMUTABLE;\nCREATE OR REPLACE FUNCTION to_int(jsonb) RETURNS int AS $$\n SELECT floor(($1->>0)::float)::int;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION to_float(jsonb) RETURNS float AS $$\n SELECT ($1->>0)::float;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION to_tstz(jsonb) RETURNS timestamptz AS $$\n SELECT ($1->>0)::timestamptz;\n$$ LANGUAGE SQL IMMUTABLE STRICT SET TIME ZONE 'UTC';\n\n\nCREATE OR REPLACE FUNCTION to_text(jsonb) RETURNS text AS $$\n SELECT CASE WHEN jsonb_typeof($1) IN ('array','object') THEN $1::text ELSE $1->>0 END;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION to_text_array(jsonb) RETURNS text[] AS $$\n SELECT\n CASE jsonb_typeof($1)\n WHEN 'array' THEN ARRAY(SELECT jsonb_array_elements_text($1))\n ELSE ARRAY[$1->>0]\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION bbox_geom(_bbox jsonb) RETURNS geometry AS $$\nSELECT CASE jsonb_array_length(_bbox)\n WHEN 4 THEN\n ST_SetSRID(ST_MakeEnvelope(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float,\n (_bbox->>3)::float\n ),4326)\n WHEN 6 THEN\n ST_SetSRID(ST_3DMakeBox(\n ST_MakePoint(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float\n ),\n ST_MakePoint(\n (_bbox->>3)::float,\n (_bbox->>4)::float,\n (_bbox->>5)::float\n )\n ),4326)\n ELSE null END;\n;\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION geom_bbox(_geom geometry) RETURNS jsonb AS $$\n SELECT jsonb_build_array(\n st_xmin(_geom),\n st_ymin(_geom),\n st_xmax(_geom),\n st_ymax(_geom)\n );\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION flip_jsonb_array(j jsonb) RETURNS jsonb AS $$\n SELECT jsonb_agg(value) FROM (SELECT value FROM jsonb_array_elements(j) WITH ORDINALITY ORDER BY ordinality DESC) as t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION explode_dotpaths(j jsonb) RETURNS SETOF text[] AS $$\n SELECT string_to_array(p, '.') as e FROM jsonb_array_elements_text(j) p;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION explode_dotpaths_recurse(IN j jsonb) RETURNS SETOF text[] AS $$\n WITH RECURSIVE t AS (\n SELECT e FROM explode_dotpaths(j) e\n UNION ALL\n SELECT e[1:cardinality(e)-1]\n FROM t\n WHERE cardinality(e)>1\n ) SELECT e FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION jsonb_set_nested(j jsonb, path text[], val jsonb) RETURNS jsonb AS $$\nDECLARE\nBEGIN\n IF cardinality(path) > 1 THEN\n FOR i IN 1..(cardinality(path)-1) LOOP\n IF j #> path[:i] IS NULL THEN\n j := jsonb_set_lax(j, path[:i], '{}', TRUE);\n END IF;\n END LOOP;\n END IF;\n RETURN jsonb_set_lax(j, path, val, true);\n\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\n\n\nCREATE OR REPLACE FUNCTION jsonb_include(j jsonb, f jsonb) RETURNS jsonb AS $$\nDECLARE\n includes jsonb := f-> 'include';\n outj jsonb := '{}'::jsonb;\n path text[];\nBEGIN\n IF\n includes IS NULL\n OR jsonb_array_length(includes) = 0\n THEN\n RETURN j;\n ELSE\n includes := includes || '[\"id\",\"collection\"]'::jsonb;\n FOR path IN SELECT explode_dotpaths(includes) LOOP\n outj := jsonb_set_nested(outj, path, j #> path);\n END LOOP;\n END IF;\n RETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_exclude(j jsonb, f jsonb) RETURNS jsonb AS $$\nDECLARE\n excludes jsonb := f-> 'exclude';\n outj jsonb := j;\n path text[];\nBEGIN\n IF\n excludes IS NULL\n OR jsonb_array_length(excludes) = 0\n THEN\n RETURN j;\n ELSE\n FOR path IN SELECT explode_dotpaths(excludes) LOOP\n outj := outj #- path;\n END LOOP;\n END IF;\n RETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_fields(j jsonb, f jsonb DEFAULT '{\"fields\":[]}') RETURNS jsonb AS $$\n SELECT jsonb_exclude(jsonb_include(j, f), f);\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION merge_jsonb(_a jsonb, _b jsonb) RETURNS jsonb AS $$\n SELECT\n CASE\n WHEN _a = '\"𒍟※\"'::jsonb THEN NULL\n WHEN _a IS NULL OR jsonb_typeof(_a) = 'null' THEN _b\n WHEN jsonb_typeof(_a) = 'object' AND jsonb_typeof(_b) = 'object' THEN\n (\n SELECT\n jsonb_strip_nulls(\n jsonb_object_agg(\n key,\n merge_jsonb(a.value, b.value)\n )\n )\n FROM\n jsonb_each(coalesce(_a,'{}'::jsonb)) as a\n FULL JOIN\n jsonb_each(coalesce(_b,'{}'::jsonb)) as b\n USING (key)\n )\n WHEN\n jsonb_typeof(_a) = 'array'\n AND jsonb_typeof(_b) = 'array'\n AND jsonb_array_length(_a) = jsonb_array_length(_b)\n THEN\n (\n SELECT jsonb_agg(m) FROM\n ( SELECT\n merge_jsonb(\n jsonb_array_elements(_a),\n jsonb_array_elements(_b)\n ) as m\n ) as l\n )\n ELSE _a\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION strip_jsonb(_a jsonb, _b jsonb) RETURNS jsonb AS $$\n SELECT\n CASE\n\n WHEN (_a IS NULL OR jsonb_typeof(_a) = 'null') AND _b IS NOT NULL AND jsonb_typeof(_b) != 'null' THEN '\"𒍟※\"'::jsonb\n WHEN _b IS NULL OR jsonb_typeof(_a) = 'null' THEN _a\n WHEN _a = _b AND jsonb_typeof(_a) = 'object' THEN '{}'::jsonb\n WHEN _a = _b THEN NULL\n WHEN jsonb_typeof(_a) = 'object' AND jsonb_typeof(_b) = 'object' THEN\n (\n SELECT\n jsonb_strip_nulls(\n jsonb_object_agg(\n key,\n strip_jsonb(a.value, b.value)\n )\n )\n FROM\n jsonb_each(_a) as a\n FULL JOIN\n jsonb_each(_b) as b\n USING (key)\n )\n WHEN\n jsonb_typeof(_a) = 'array'\n AND jsonb_typeof(_b) = 'array'\n AND jsonb_array_length(_a) = jsonb_array_length(_b)\n THEN\n (\n SELECT jsonb_agg(m) FROM\n ( SELECT\n strip_jsonb(\n jsonb_array_elements(_a),\n jsonb_array_elements(_b)\n ) as m\n ) as l\n )\n ELSE _a\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE;\n/* looks for a geometry in a stac item first from geometry and falling back to bbox */\nCREATE OR REPLACE FUNCTION stac_geom(value jsonb) RETURNS geometry AS $$\nSELECT\n CASE\n WHEN value ? 'intersects' THEN\n ST_GeomFromGeoJSON(value->>'intersects')\n WHEN value ? 'geometry' THEN\n ST_GeomFromGeoJSON(value->>'geometry')\n WHEN value ? 'bbox' THEN\n pgstac.bbox_geom(value->'bbox')\n ELSE NULL\n END as geometry\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION stac_daterange(\n value jsonb\n) RETURNS tstzrange AS $$\nDECLARE\n props jsonb := value;\n dt timestamptz;\n edt timestamptz;\nBEGIN\n IF props ? 'properties' THEN\n props := props->'properties';\n END IF;\n IF\n props ? 'start_datetime'\n AND props->>'start_datetime' IS NOT NULL\n AND props ? 'end_datetime'\n AND props->>'end_datetime' IS NOT NULL\n THEN\n dt := props->>'start_datetime';\n edt := props->>'end_datetime';\n IF dt > edt THEN\n RAISE EXCEPTION 'start_datetime must be < end_datetime';\n END IF;\n ELSE\n dt := props->>'datetime';\n edt := props->>'datetime';\n END IF;\n IF dt is NULL OR edt IS NULL THEN\n RAISE NOTICE 'DT: %, EDT: %', dt, edt;\n RAISE EXCEPTION 'Either datetime (%) or both start_datetime (%) and end_datetime (%) must be set.', props->>'datetime',props->>'start_datetime',props->>'end_datetime';\n END IF;\n RETURN tstzrange(dt, edt, '[]');\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_datetime(value jsonb) RETURNS timestamptz AS $$\n SELECT lower(stac_daterange(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_end_datetime(value jsonb) RETURNS timestamptz AS $$\n SELECT upper(stac_daterange(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\n\nCREATE TABLE IF NOT EXISTS stac_extensions(\n name text PRIMARY KEY,\n url text,\n enbabled_by_default boolean NOT NULL DEFAULT TRUE,\n enableable boolean NOT NULL DEFAULT TRUE\n);\n\nINSERT INTO stac_extensions (name, url) VALUES\n ('fields', 'https://api.stacspec.org/v1.0.0-beta.5/item-search#fields'),\n ('sort','https://api.stacspec.org/v1.0.0-beta.5/item-search#sort'),\n ('context','https://api.stacspec.org/v1.0.0-beta.5/item-search#context'),\n ('filter', 'https://api.stacspec.org/v1.0.0-beta.5/item-search#filter'),\n ('query', 'https://api.stacspec.org/v1.0.0-beta.5/item-search#query')\nON CONFLICT (name) DO UPDATE SET url=EXCLUDED.url;\n\n\n\nCREATE OR REPLACE FUNCTION collection_base_item(content jsonb) RETURNS jsonb AS $$\n SELECT jsonb_build_object(\n 'type', 'Feature',\n 'stac_version', content->'stac_version',\n 'assets', content->'item_assets',\n 'collection', content->'id'\n );\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TYPE partition_trunc_strategy AS ENUM ('year', 'month');\n\nCREATE TABLE IF NOT EXISTS collections (\n key bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,\n id text GENERATED ALWAYS AS (content->>'id') STORED UNIQUE,\n content JSONB NOT NULL,\n base_item jsonb GENERATED ALWAYS AS (pgstac.collection_base_item(content)) STORED,\n partition_trunc partition_trunc_strategy\n);\n\n\nCREATE OR REPLACE FUNCTION collection_base_item(cid text) RETURNS jsonb AS $$\n SELECT pgstac.collection_base_item(content) FROM pgstac.collections WHERE id = cid LIMIT 1;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION table_empty(text) RETURNS boolean AS $$\nDECLARE\n retval boolean;\nBEGIN\n EXECUTE format($q$\n SELECT NOT EXISTS (SELECT 1 FROM %I LIMIT 1)\n $q$,\n $1\n ) INTO retval;\n RETURN retval;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION collections_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n q text;\n partition_name text := format('_items_%s', NEW.key);\n partition_exists boolean := false;\n partition_empty boolean := true;\n err_context text;\n loadtemp boolean := FALSE;\nBEGIN\n RAISE NOTICE 'Collection Trigger. % %', NEW.id, NEW.key;\n SELECT relid::text INTO partition_name\n FROM pg_partition_tree('items')\n WHERE relid::text = partition_name;\n IF FOUND THEN\n partition_exists := true;\n partition_empty := table_empty(partition_name);\n ELSE\n partition_exists := false;\n partition_empty := true;\n partition_name := format('_items_%s', NEW.key);\n END IF;\n IF TG_OP = 'UPDATE' AND NEW.partition_trunc IS DISTINCT FROM OLD.partition_trunc AND partition_empty THEN\n q := format($q$\n DROP TABLE IF EXISTS %I CASCADE;\n $q$,\n partition_name\n );\n EXECUTE q;\n END IF;\n IF TG_OP = 'UPDATE' AND NEW.partition_trunc IS DISTINCT FROM OLD.partition_trunc AND partition_exists AND NOT partition_empty THEN\n q := format($q$\n CREATE TEMP TABLE changepartitionstaging ON COMMIT DROP AS SELECT * FROM %I;\n DROP TABLE IF EXISTS %I CASCADE;\n $q$,\n partition_name,\n partition_name\n );\n EXECUTE q;\n loadtemp := TRUE;\n partition_empty := TRUE;\n partition_exists := FALSE;\n END IF;\n IF TG_OP = 'UPDATE' AND NEW.partition_trunc IS NOT DISTINCT FROM OLD.partition_trunc THEN\n RETURN NEW;\n END IF;\n IF NEW.partition_trunc IS NULL AND partition_empty THEN\n RAISE NOTICE '% % % %',\n partition_name,\n NEW.id,\n concat(partition_name,'_id_idx'),\n partition_name\n ;\n q := format($q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n $q$,\n partition_name,\n NEW.id,\n concat(partition_name,'_id_idx'),\n partition_name\n );\n RAISE NOTICE 'q: %', q;\n BEGIN\n EXECUTE q;\n EXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', partition_name;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\n END;\n\n ALTER TABLE partitions DISABLE TRIGGER partitions_delete_trigger;\n DELETE FROM partitions WHERE collection=NEW.id AND name=partition_name;\n ALTER TABLE partitions ENABLE TRIGGER partitions_delete_trigger;\n\n INSERT INTO partitions (collection, name) VALUES (NEW.id, partition_name);\n ELSIF partition_empty THEN\n q := format($q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L)\n PARTITION BY RANGE (datetime);\n $q$,\n partition_name,\n NEW.id\n );\n RAISE NOTICE 'q: %', q;\n BEGIN\n EXECUTE q;\n EXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', partition_name;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\n END;\n ALTER TABLE partitions DISABLE TRIGGER partitions_delete_trigger;\n DELETE FROM partitions WHERE collection=NEW.id AND name=partition_name;\n ALTER TABLE partitions ENABLE TRIGGER partitions_delete_trigger;\n ELSE\n RAISE EXCEPTION 'Cannot modify partition % unless empty', partition_name;\n END IF;\n IF loadtemp THEN\n RAISE NOTICE 'Moving data into new partitions.';\n q := format($q$\n WITH p AS (\n SELECT\n collection,\n datetime as datetime,\n end_datetime as end_datetime,\n (partition_name(\n collection,\n datetime\n )).partition_name as name\n FROM changepartitionstaging\n )\n INSERT INTO partitions (collection, datetime_range, end_datetime_range)\n SELECT\n collection,\n tstzrange(min(datetime), max(datetime), '[]') as datetime_range,\n tstzrange(min(end_datetime), max(end_datetime), '[]') as end_datetime_range\n FROM p\n GROUP BY collection, name\n ON CONFLICT (name) DO UPDATE SET\n datetime_range = EXCLUDED.datetime_range,\n end_datetime_range = EXCLUDED.end_datetime_range\n ;\n INSERT INTO %I SELECT * FROM changepartitionstaging;\n DROP TABLE IF EXISTS changepartitionstaging;\n $q$,\n partition_name\n );\n EXECUTE q;\n END IF;\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE TRIGGER collections_trigger AFTER INSERT OR UPDATE ON collections FOR EACH ROW\nEXECUTE FUNCTION collections_trigger_func();\n\nCREATE OR REPLACE FUNCTION partition_collection(collection text, strategy partition_trunc_strategy) RETURNS text AS $$\n UPDATE collections SET partition_trunc=strategy WHERE id=collection RETURNING partition_trunc;\n$$ LANGUAGE SQL;\n\nCREATE TABLE IF NOT EXISTS partitions (\n collection text REFERENCES collections(id) ON DELETE CASCADE,\n name text PRIMARY KEY,\n partition_range tstzrange NOT NULL DEFAULT tstzrange('-infinity'::timestamptz,'infinity'::timestamptz, '[]'),\n datetime_range tstzrange,\n end_datetime_range tstzrange,\n CONSTRAINT prange EXCLUDE USING GIST (\n collection WITH =,\n partition_range WITH &&\n )\n) WITH (FILLFACTOR=90);\nCREATE INDEX partitions_range_idx ON partitions USING GIST(partition_range);\n\nCREATE OR REPLACE FUNCTION partitions_delete_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n q text;\nBEGIN\n RAISE NOTICE 'Partition Delete Trigger. %', OLD.name;\n EXECUTE format($q$\n DROP TABLE IF EXISTS %I CASCADE;\n $q$,\n OLD.name\n );\n RAISE NOTICE 'Dropped partition.';\n RETURN OLD;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER partitions_delete_trigger BEFORE DELETE ON partitions FOR EACH ROW\nEXECUTE FUNCTION partitions_delete_trigger_func();\n\nCREATE OR REPLACE FUNCTION partition_name(\n IN collection text,\n IN dt timestamptz,\n OUT partition_name text,\n OUT partition_range tstzrange\n) AS $$\nDECLARE\n c RECORD;\n parent_name text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n parent_name := format('_items_%s', c.key);\n\n\n IF c.partition_trunc = 'year' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYY'));\n ELSIF c.partition_trunc = 'month' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYYMM'));\n ELSE\n partition_name := parent_name;\n partition_range := tstzrange('-infinity'::timestamptz, 'infinity'::timestamptz, '[]');\n END IF;\n IF partition_range IS NULL THEN\n partition_range := tstzrange(\n date_trunc(c.partition_trunc::text, dt),\n date_trunc(c.partition_trunc::text, dt) + concat('1 ', c.partition_trunc)::interval\n );\n END IF;\n RETURN;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\n\nCREATE OR REPLACE FUNCTION partitions_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n q text;\n cq text;\n parent_name text;\n partition_trunc text;\n partition_name text := NEW.name;\n partition_exists boolean := false;\n partition_empty boolean := true;\n partition_range tstzrange;\n datetime_range tstzrange;\n end_datetime_range tstzrange;\n err_context text;\n mindt timestamptz := lower(NEW.datetime_range);\n maxdt timestamptz := upper(NEW.datetime_range);\n minedt timestamptz := lower(NEW.end_datetime_range);\n maxedt timestamptz := upper(NEW.end_datetime_range);\n t_mindt timestamptz;\n t_maxdt timestamptz;\n t_minedt timestamptz;\n t_maxedt timestamptz;\nBEGIN\n RAISE NOTICE 'Partitions Trigger. %', NEW;\n datetime_range := NEW.datetime_range;\n end_datetime_range := NEW.end_datetime_range;\n\n SELECT\n format('_items_%s', key),\n c.partition_trunc::text\n INTO\n parent_name,\n partition_trunc\n FROM pgstac.collections c\n WHERE c.id = NEW.collection;\n SELECT (pgstac.partition_name(NEW.collection, mindt)).* INTO partition_name, partition_range;\n NEW.name := partition_name;\n\n IF partition_range IS NULL OR partition_range = 'empty'::tstzrange THEN\n partition_range := tstzrange('-infinity'::timestamptz, 'infinity'::timestamptz, '[]');\n END IF;\n\n NEW.partition_range := partition_range;\n IF TG_OP = 'UPDATE' THEN\n mindt := least(mindt, lower(OLD.datetime_range));\n maxdt := greatest(maxdt, upper(OLD.datetime_range));\n minedt := least(minedt, lower(OLD.end_datetime_range));\n maxedt := greatest(maxedt, upper(OLD.end_datetime_range));\n NEW.datetime_range := tstzrange(mindt, maxdt, '[]');\n NEW.end_datetime_range := tstzrange(minedt, maxedt, '[]');\n END IF;\n IF TG_OP = 'INSERT' THEN\n\n IF partition_range != tstzrange('-infinity'::timestamptz, 'infinity'::timestamptz, '[]') THEN\n\n RAISE NOTICE '% % %', partition_name, parent_name, partition_range;\n q := format($q$\n CREATE TABLE IF NOT EXISTS %I partition OF %I FOR VALUES FROM (%L) TO (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n $q$,\n partition_name,\n parent_name,\n lower(partition_range),\n upper(partition_range),\n format('%s_pkey', partition_name),\n partition_name\n );\n BEGIN\n EXECUTE q;\n EXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', partition_name;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\n END;\n END IF;\n\n END IF;\n\n -- Update constraints\n EXECUTE format($q$\n SELECT\n min(datetime),\n max(datetime),\n min(end_datetime),\n max(end_datetime)\n FROM %I;\n $q$, partition_name)\n INTO t_mindt, t_maxdt, t_minedt, t_maxedt;\n mindt := least(mindt, t_mindt);\n maxdt := greatest(maxdt, t_maxdt);\n minedt := least(mindt, minedt, t_minedt);\n maxedt := greatest(maxdt, maxedt, t_maxedt);\n\n mindt := date_trunc(coalesce(partition_trunc, 'year'), mindt);\n maxdt := date_trunc(coalesce(partition_trunc, 'year'), maxdt - '1 second'::interval) + concat('1 ',coalesce(partition_trunc, 'year'))::interval;\n minedt := date_trunc(coalesce(partition_trunc, 'year'), minedt);\n maxedt := date_trunc(coalesce(partition_trunc, 'year'), maxedt - '1 second'::interval) + concat('1 ',coalesce(partition_trunc, 'year'))::interval;\n\n\n IF mindt IS NOT NULL AND maxdt IS NOT NULL AND minedt IS NOT NULL AND maxedt IS NOT NULL THEN\n NEW.datetime_range := tstzrange(mindt, maxdt, '[]');\n NEW.end_datetime_range := tstzrange(minedt, maxedt, '[]');\n IF\n TG_OP='UPDATE'\n AND OLD.datetime_range @> NEW.datetime_range\n AND OLD.end_datetime_range @> NEW.end_datetime_range\n THEN\n RAISE NOTICE 'Range unchanged, not updating constraints.';\n ELSE\n\n RAISE NOTICE '\n SETTING CONSTRAINTS\n mindt: %, maxdt: %\n minedt: %, maxedt: %\n ', mindt, maxdt, minedt, maxedt;\n IF partition_trunc IS NULL THEN\n cq := format($q$\n ALTER TABLE %7$I\n DROP CONSTRAINT IF EXISTS %1$I,\n DROP CONSTRAINT IF EXISTS %2$I,\n ADD CONSTRAINT %1$I\n CHECK (\n (datetime >= %3$L)\n AND (datetime <= %4$L)\n AND (end_datetime >= %5$L)\n AND (end_datetime <= %6$L)\n ) NOT VALID\n ;\n ALTER TABLE %7$I\n VALIDATE CONSTRAINT %1$I;\n $q$,\n format('%s_dt', partition_name),\n format('%s_edt', partition_name),\n mindt,\n maxdt,\n minedt,\n maxedt,\n partition_name\n );\n ELSE\n cq := format($q$\n ALTER TABLE %5$I\n DROP CONSTRAINT IF EXISTS %1$I,\n DROP CONSTRAINT IF EXISTS %2$I,\n ADD CONSTRAINT %2$I\n CHECK ((end_datetime >= %3$L) AND (end_datetime <= %4$L)) NOT VALID\n ;\n ALTER TABLE %5$I\n VALIDATE CONSTRAINT %2$I;\n $q$,\n format('%s_dt', partition_name),\n format('%s_edt', partition_name),\n minedt,\n maxedt,\n partition_name\n );\n\n END IF;\n RAISE NOTICE 'Altering Constraints. %', cq;\n EXECUTE cq;\n END IF;\n ELSE\n NEW.datetime_range = NULL;\n NEW.end_datetime_range = NULL;\n\n cq := format($q$\n ALTER TABLE %3$I\n DROP CONSTRAINT IF EXISTS %1$I,\n DROP CONSTRAINT IF EXISTS %2$I,\n ADD CONSTRAINT %1$I\n CHECK ((datetime IS NULL AND end_datetime IS NULL)) NOT VALID\n ;\n ALTER TABLE %3$I\n VALIDATE CONSTRAINT %1$I;\n $q$,\n format('%s_dt', partition_name),\n format('%s_edt', partition_name),\n partition_name\n );\n EXECUTE cq;\n END IF;\n\n RETURN NEW;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER partitions_trigger BEFORE INSERT OR UPDATE ON partitions FOR EACH ROW\nEXECUTE FUNCTION partitions_trigger_func();\n\n\nCREATE OR REPLACE FUNCTION create_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection(data jsonb) RETURNS VOID AS $$\nDECLARE\n out collections%ROWTYPE;\nBEGIN\n UPDATE collections SET content=data WHERE id = data->>'id' RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ON CONFLICT (id) DO\n UPDATE\n SET content=EXCLUDED.content\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION delete_collection(_id text) RETURNS VOID AS $$\nDECLARE\n out collections%ROWTYPE;\nBEGIN\n DELETE FROM collections WHERE id = _id RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION get_collection(id text) RETURNS jsonb AS $$\n SELECT content FROM collections\n WHERE id=$1\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION all_collections() RETURNS jsonb AS $$\n SELECT jsonb_agg(content) FROM collections;\n;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\nCREATE TABLE queryables (\n id bigint GENERATED ALWAYS AS identity PRIMARY KEY,\n name text UNIQUE NOT NULL,\n collection_ids text[], -- used to determine what partitions to create indexes on\n definition jsonb,\n property_path text,\n property_wrapper text,\n property_index_type text\n);\nCREATE INDEX queryables_name_idx ON queryables (name);\nCREATE INDEX queryables_property_wrapper_idx ON queryables (property_wrapper);\n\n\nINSERT INTO queryables (name, definition) VALUES\n('id', '{\"title\": \"Item ID\",\"description\": \"Item identifier\",\"$ref\": \"https://schemas.stacspec.org/v1.0.0/item-spec/json-schema/item.json#/definitions/core/allOf/2/properties/id\"}'),\n('datetime','{\"description\": \"Datetime\",\"type\": \"string\",\"title\": \"Acquired\",\"format\": \"date-time\",\"pattern\": \"(\\\\+00:00|Z)$\"}')\nON CONFLICT DO NOTHING;\n\n\n\nINSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n('eo:cloud_cover','{\"$ref\": \"https://stac-extensions.github.io/eo/v1.0.0/schema.json#/definitions/fieldsproperties/eo:cloud_cover\"}','to_int','BTREE')\nON CONFLICT DO NOTHING;\n\nCREATE OR REPLACE FUNCTION array_to_path(arr text[]) RETURNS text AS $$\n SELECT string_agg(\n quote_literal(v),\n '->'\n ) FROM unnest(arr) v;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\n\n\nCREATE OR REPLACE FUNCTION queryable(\n IN dotpath text,\n OUT path text,\n OUT expression text,\n OUT wrapper text\n) AS $$\nDECLARE\n q RECORD;\n path_elements text[];\nBEGIN\n IF dotpath IN ('id', 'geometry', 'datetime', 'end_datetime', 'collection') THEN\n path := dotpath;\n expression := dotpath;\n wrapper := NULL;\n RETURN;\n END IF;\n SELECT * INTO q FROM queryables WHERE name=dotpath;\n IF q.property_wrapper IS NULL THEN\n IF q.definition->>'type' = 'number' THEN\n wrapper := 'to_float';\n ELSIF q.definition->>'format' = 'date-time' THEN\n wrapper := 'to_tstz';\n ELSE\n wrapper := 'to_text';\n END IF;\n ELSE\n wrapper := q.property_wrapper;\n END IF;\n IF q.property_path IS NOT NULL THEN\n path := q.property_path;\n ELSE\n path_elements := string_to_array(dotpath, '.');\n IF path_elements[1] IN ('links', 'assets', 'stac_version', 'stac_extensions') THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSIF path_elements[1] = 'properties' THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSE\n path := format($F$content->'properties'->%s$F$, array_to_path(path_elements));\n END IF;\n END IF;\n expression := format('%I(%s)', wrapper, path);\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\nCREATE OR REPLACE FUNCTION create_queryable_indexes() RETURNS VOID AS $$\nDECLARE\n queryable RECORD;\n q text;\nBEGIN\n FOR queryable IN\n SELECT\n queryables.id as qid,\n CASE WHEN collections.key IS NULL THEN 'items' ELSE format('_items_%s',collections.key) END AS part,\n property_index_type,\n expression\n FROM\n queryables\n LEFT JOIN collections ON (collections.id = ANY (queryables.collection_ids))\n JOIN LATERAL queryable(queryables.name) ON (queryables.property_index_type IS NOT NULL)\n LOOP\n q := format(\n $q$\n CREATE INDEX IF NOT EXISTS %I ON %I USING %s ((%s));\n $q$,\n format('%s_%s_idx', queryable.part, queryable.qid),\n queryable.part,\n COALESCE(queryable.property_index_type, 'to_text'),\n queryable.expression\n );\n RAISE NOTICE '%',q;\n EXECUTE q;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION queryables_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\nBEGIN\nPERFORM create_queryable_indexes();\nRETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER queryables_trigger AFTER INSERT OR UPDATE ON queryables\nFOR EACH STATEMENT EXECUTE PROCEDURE queryables_trigger_func();\n\nCREATE TRIGGER queryables_collection_trigger AFTER INSERT OR UPDATE ON collections\nFOR EACH STATEMENT EXECUTE PROCEDURE queryables_trigger_func();\nCREATE OR REPLACE FUNCTION parse_dtrange(\n _indate jsonb,\n relative_base timestamptz DEFAULT date_trunc('hour', CURRENT_TIMESTAMP)\n) RETURNS tstzrange AS $$\nDECLARE\n timestrs text[];\n s timestamptz;\n e timestamptz;\nBEGIN\n timestrs :=\n CASE\n WHEN _indate ? 'timestamp' THEN\n ARRAY[_indate->>'timestamp']\n WHEN _indate ? 'interval' THEN\n to_text_array(_indate->'interval')\n WHEN jsonb_typeof(_indate) = 'array' THEN\n to_text_array(_indate)\n ELSE\n regexp_split_to_array(\n _indate->>0,\n '/'\n )\n END;\n RAISE NOTICE 'TIMESTRS %', timestrs;\n IF cardinality(timestrs) = 1 THEN\n IF timestrs[1] ILIKE 'P%' THEN\n RETURN tstzrange(relative_base - upper(timestrs[1])::interval, relative_base, '[)');\n END IF;\n s := timestrs[1]::timestamptz;\n RETURN tstzrange(s, s, '[]');\n END IF;\n\n IF cardinality(timestrs) != 2 THEN\n RAISE EXCEPTION 'Timestamp cannot have more than 2 values';\n END IF;\n\n IF timestrs[1] = '..' THEN\n s := '-infinity'::timestamptz;\n e := timestrs[2]::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] = '..' THEN\n s := timestrs[1]::timestamptz;\n e := 'infinity'::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[1] ILIKE 'P%' AND timestrs[2] NOT ILIKE 'P%' THEN\n e := timestrs[2]::timestamptz;\n s := e - upper(timestrs[1])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] ILIKE 'P%' AND timestrs[1] NOT ILIKE 'P%' THEN\n s := timestrs[1]::timestamptz;\n e := s + upper(timestrs[2])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n s := timestrs[1]::timestamptz;\n e := timestrs[2]::timestamptz;\n\n RETURN tstzrange(s,e,'[)');\n\n RETURN NULL;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT PARALLEL SAFE SET TIME ZONE 'UTC';\n\nCREATE OR REPLACE FUNCTION parse_dtrange(\n _indate text,\n relative_base timestamptz DEFAULT CURRENT_TIMESTAMP\n) RETURNS tstzrange AS $$\n SELECT parse_dtrange(to_jsonb(_indate), relative_base);\n$$ LANGUAGE SQL STABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION temporal_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\n ll text := 'datetime';\n lh text := 'end_datetime';\n rrange tstzrange;\n rl text;\n rh text;\n outq text;\nBEGIN\n rrange := parse_dtrange(args->1);\n RAISE NOTICE 'Constructing temporal query OP: %, ARGS: %, RRANGE: %', op, args, rrange;\n op := lower(op);\n rl := format('%L::timestamptz', lower(rrange));\n rh := format('%L::timestamptz', upper(rrange));\n outq := CASE op\n WHEN 't_before' THEN 'lh < rl'\n WHEN 't_after' THEN 'll > rh'\n WHEN 't_meets' THEN 'lh = rl'\n WHEN 't_metby' THEN 'll = rh'\n WHEN 't_overlaps' THEN 'll < rl AND rl < lh < rh'\n WHEN 't_overlappedby' THEN 'rl < ll < rh AND lh > rh'\n WHEN 't_starts' THEN 'll = rl AND lh < rh'\n WHEN 't_startedby' THEN 'll = rl AND lh > rh'\n WHEN 't_during' THEN 'll > rl AND lh < rh'\n WHEN 't_contains' THEN 'll < rl AND lh > rh'\n WHEN 't_finishes' THEN 'll > rl AND lh = rh'\n WHEN 't_finishedby' THEN 'll < rl AND lh = rh'\n WHEN 't_equals' THEN 'll = rl AND lh = rh'\n WHEN 't_disjoint' THEN 'NOT (ll <= rh AND lh >= rl)'\n WHEN 't_intersects' THEN 'll <= rh AND lh >= rl'\n WHEN 'anyinteracts' THEN 'll <= rh AND lh >= rl'\n END;\n outq := regexp_replace(outq, '\\mll\\M', ll);\n outq := regexp_replace(outq, '\\mlh\\M', lh);\n outq := regexp_replace(outq, '\\mrl\\M', rl);\n outq := regexp_replace(outq, '\\mrh\\M', rh);\n outq := format('(%s)', outq);\n RETURN outq;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\n\n\nCREATE OR REPLACE FUNCTION spatial_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\n geom text;\n j jsonb := args->1;\nBEGIN\n op := lower(op);\n RAISE NOTICE 'Constructing spatial query OP: %, ARGS: %', op, args;\n IF op NOT IN ('s_equals','s_disjoint','s_touches','s_within','s_overlaps','s_crosses','s_intersects','intersects','s_contains') THEN\n RAISE EXCEPTION 'Spatial Operator % Not Supported', op;\n END IF;\n op := regexp_replace(op, '^s_', 'st_');\n IF op = 'intersects' THEN\n op := 'st_intersects';\n END IF;\n -- Convert geometry to WKB string\n IF j ? 'type' AND j ? 'coordinates' THEN\n geom := st_geomfromgeojson(j)::text;\n ELSIF jsonb_typeof(j) = 'array' THEN\n geom := bbox_geom(j)::text;\n END IF;\n\n RETURN format('%s(geometry, %L::geometry)', op, geom);\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION query_to_cql2(q jsonb) RETURNS jsonb AS $$\n-- Translates anything passed in through the deprecated \"query\" into equivalent CQL2\nWITH t AS (\n SELECT key as property, value as ops\n FROM jsonb_each(q)\n), t2 AS (\n SELECT property, (jsonb_each(ops)).*\n FROM t WHERE jsonb_typeof(ops) = 'object'\n UNION ALL\n SELECT property, 'eq', ops\n FROM t WHERE jsonb_typeof(ops) != 'object'\n)\nSELECT\n jsonb_strip_nulls(jsonb_build_object(\n 'op', 'and',\n 'args', jsonb_agg(\n jsonb_build_object(\n 'op', key,\n 'args', jsonb_build_array(\n jsonb_build_object('property',property),\n value\n )\n )\n )\n )\n) as qcql FROM t2\n;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\nCREATE OR REPLACE FUNCTION cql1_to_cql2(j jsonb) RETURNS jsonb AS $$\nDECLARE\n args jsonb;\n ret jsonb;\nBEGIN\n RAISE NOTICE 'CQL1_TO_CQL2: %', j;\n IF j ? 'filter' THEN\n RETURN cql1_to_cql2(j->'filter');\n END IF;\n IF j ? 'property' THEN\n RETURN j;\n END IF;\n IF jsonb_typeof(j) = 'array' THEN\n SELECT jsonb_agg(cql1_to_cql2(el)) INTO args FROM jsonb_array_elements(j) el;\n RETURN args;\n END IF;\n IF jsonb_typeof(j) = 'number' THEN\n RETURN j;\n END IF;\n IF jsonb_typeof(j) = 'string' THEN\n RETURN j;\n END IF;\n\n IF jsonb_typeof(j) = 'object' THEN\n SELECT jsonb_build_object(\n 'op', key,\n 'args', cql1_to_cql2(value)\n ) INTO ret\n FROM jsonb_each(j)\n WHERE j IS NOT NULL;\n RETURN ret;\n END IF;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE STRICT;\n\nCREATE TABLE cql2_ops (\n op text PRIMARY KEY,\n template text,\n types text[]\n);\nINSERT INTO cql2_ops (op, template, types) VALUES\n ('eq', '%s = %s', NULL),\n ('neq', '%s != %s', NULL),\n ('ne', '%s != %s', NULL),\n ('!=', '%s != %s', NULL),\n ('<>', '%s != %s', NULL),\n ('lt', '%s < %s', NULL),\n ('lte', '%s <= %s', NULL),\n ('gt', '%s > %s', NULL),\n ('gte', '%s >= %s', NULL),\n ('le', '%s <= %s', NULL),\n ('ge', '%s >= %s', NULL),\n ('=', '%s = %s', NULL),\n ('<', '%s < %s', NULL),\n ('<=', '%s <= %s', NULL),\n ('>', '%s > %s', NULL),\n ('>=', '%s >= %s', NULL),\n ('like', '%s LIKE %s', NULL),\n ('ilike', '%s ILIKE %s', NULL),\n ('+', '%s + %s', NULL),\n ('-', '%s - %s', NULL),\n ('*', '%s * %s', NULL),\n ('/', '%s / %s', NULL),\n ('not', 'NOT (%s)', NULL),\n ('between', '%s BETWEEN %s AND %s', NULL),\n ('isnull', '%s IS NULL', NULL),\n ('upper', 'upper(%s)', NULL),\n ('lower', 'lower(%s)', NULL)\nON CONFLICT (op) DO UPDATE\n SET\n template = EXCLUDED.template\n;\n\n\nCREATE OR REPLACE FUNCTION cql2_query(j jsonb, wrapper text DEFAULT NULL) RETURNS text AS $$\n#variable_conflict use_variable\nDECLARE\n args jsonb := j->'args';\n arg jsonb;\n op text := lower(j->>'op');\n cql2op RECORD;\n literal text;\n _wrapper text;\n leftarg text;\n rightarg text;\nBEGIN\n IF j IS NULL OR (op IS NOT NULL AND args IS NULL) THEN\n RETURN NULL;\n END IF;\n RAISE NOTICE 'CQL2_QUERY: %', j;\n IF j ? 'filter' THEN\n RETURN cql2_query(j->'filter');\n END IF;\n\n IF j ? 'upper' THEN\n RETURN cql2_query(jsonb_build_object('op', 'upper', 'args', j->'upper'));\n END IF;\n\n IF j ? 'lower' THEN\n RETURN cql2_query(jsonb_build_object('op', 'lower', 'args', j->'lower'));\n END IF;\n\n -- Temporal Query\n IF op ilike 't_%' or op = 'anyinteracts' THEN\n RETURN temporal_op_query(op, args);\n END IF;\n\n -- If property is a timestamp convert it to text to use with\n -- general operators\n IF j ? 'timestamp' THEN\n RETURN format('%L::timestamptz', to_tstz(j->'timestamp'));\n END IF;\n IF j ? 'interval' THEN\n RAISE EXCEPTION 'Please use temporal operators when using intervals.';\n RETURN NONE;\n END IF;\n\n -- Spatial Query\n IF op ilike 's_%' or op = 'intersects' THEN\n RETURN spatial_op_query(op, args);\n END IF;\n\n IF op IN ('a_equals','a_contains','a_contained_by','a_overlaps') THEN\n IF args->0 ? 'property' THEN\n leftarg := format('to_text_array(%s)', (queryable(args->0->>'property')).path);\n END IF;\n IF args->1 ? 'property' THEN\n rightarg := format('to_text_array(%s)', (queryable(args->1->>'property')).path);\n END IF;\n RETURN FORMAT(\n '%s %s %s',\n COALESCE(leftarg, quote_literal(to_text_array(args->0))),\n CASE op\n WHEN 'a_equals' THEN '='\n WHEN 'a_contains' THEN '@>'\n WHEN 'a_contained_by' THEN '<@'\n WHEN 'a_overlaps' THEN '&&'\n END,\n COALESCE(rightarg, quote_literal(to_text_array(args->1)))\n );\n END IF;\n\n IF op = 'in' THEN\n RAISE NOTICE 'IN : % % %', args, jsonb_build_array(args->0), args->1;\n args := jsonb_build_array(args->0) || (args->1);\n RAISE NOTICE 'IN2 : %', args;\n END IF;\n\n\n\n IF op = 'between' THEN\n args = jsonb_build_array(\n args->0,\n args->1->0,\n args->1->1\n );\n END IF;\n\n -- Make sure that args is an array and run cql2_query on\n -- each element of the array\n RAISE NOTICE 'ARGS PRE: %', args;\n IF j ? 'args' THEN\n IF jsonb_typeof(args) != 'array' THEN\n args := jsonb_build_array(args);\n END IF;\n\n IF jsonb_path_exists(args, '$[*] ? (@.property == \"id\" || @.property == \"datetime\" || @.property == \"end_datetime\" || @.property == \"collection\")') THEN\n wrapper := NULL;\n ELSE\n -- if any of the arguments are a property, try to get the property_wrapper\n FOR arg IN SELECT jsonb_path_query(args, '$[*] ? (@.property != null)') LOOP\n RAISE NOTICE 'Arg: %', arg;\n SELECT property_wrapper INTO wrapper\n FROM queryables\n WHERE name=(arg->>'property')\n LIMIT 1;\n RAISE NOTICE 'Property: %, Wrapper: %', arg, wrapper;\n IF wrapper IS NOT NULL THEN\n EXIT;\n END IF;\n END LOOP;\n\n -- if the property was not in queryables, see if any args were numbers\n IF\n wrapper IS NULL\n AND jsonb_path_exists(args, '$[*] ? (@.type()==\"number\")')\n THEN\n wrapper := 'to_float';\n END IF;\n wrapper := coalesce(wrapper, 'to_text');\n END IF;\n\n SELECT jsonb_agg(cql2_query(a, wrapper))\n INTO args\n FROM jsonb_array_elements(args) a;\n END IF;\n RAISE NOTICE 'ARGS: %', args;\n\n IF op IN ('and', 'or') THEN\n RETURN\n format(\n '(%s)',\n array_to_string(to_text_array(args), format(' %s ', upper(op)))\n );\n END IF;\n\n IF op = 'in' THEN\n RAISE NOTICE 'IN -- % %', args->0, to_text(args->0);\n RETURN format(\n '%s IN (%s)',\n to_text(args->0),\n array_to_string((to_text_array(args))[2:], ',')\n );\n END IF;\n\n -- Look up template from cql2_ops\n IF j ? 'op' THEN\n SELECT * INTO cql2op FROM cql2_ops WHERE cql2_ops.op ilike op;\n IF FOUND THEN\n -- If specific index set in queryables for a property cast other arguments to that type\n\n RETURN format(\n cql2op.template,\n VARIADIC (to_text_array(args))\n );\n ELSE\n RAISE EXCEPTION 'Operator % Not Supported.', op;\n END IF;\n END IF;\n\n\n IF wrapper IS NOT NULL THEN\n RAISE NOTICE 'Wrapping % with %', j, wrapper;\n IF j ? 'property' THEN\n RETURN format('%I(%s)', wrapper, (queryable(j->>'property')).path);\n ELSE\n RETURN format('%I(%L)', wrapper, j);\n END IF;\n ELSIF j ? 'property' THEN\n RETURN quote_ident(j->>'property');\n END IF;\n\n RETURN quote_literal(to_text(j));\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION paging_dtrange(\n j jsonb\n) RETURNS tstzrange AS $$\nDECLARE\n op text;\n filter jsonb := j->'filter';\n dtrange tstzrange := tstzrange('-infinity'::timestamptz,'infinity'::timestamptz);\n sdate timestamptz := '-infinity'::timestamptz;\n edate timestamptz := 'infinity'::timestamptz;\n jpitem jsonb;\nBEGIN\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n END IF;\n IF NOT (filter @? '$.**.op ? (@ == \"or\" || @ == \"not\")') THEN\n FOR jpitem IN SELECT j FROM jsonb_path_query(filter,'strict $.** ? (@.args[*].property == \"datetime\")'::jsonpath) j LOOP\n op := lower(jpitem->>'op');\n dtrange := parse_dtrange(jpitem->'args'->1);\n IF op IN ('<=', 'lt', 'lte', '<', 'le', 't_before') THEN\n sdate := greatest(sdate,'-infinity');\n edate := least(edate, upper(dtrange));\n ELSIF op IN ('>=', '>', 'gt', 'gte', 'ge', 't_after') THEN\n edate := least(edate, 'infinity');\n sdate := greatest(sdate, lower(dtrange));\n ELSIF op IN ('=', 'eq') THEN\n edate := least(edate, upper(dtrange));\n sdate := greatest(sdate, lower(dtrange));\n END IF;\n RAISE NOTICE '2 OP: %, ARGS: %, DTRANGE: %, SDATE: %, EDATE: %', op, jpitem->'args'->1, dtrange, sdate, edate;\n END LOOP;\n END IF;\n IF sdate > edate THEN\n RETURN 'empty'::tstzrange;\n END IF;\n RETURN tstzrange(sdate,edate, '[]');\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT SET TIME ZONE 'UTC';\n\nCREATE OR REPLACE FUNCTION paging_collections(\n IN j jsonb\n) RETURNS text[] AS $$\nDECLARE\n filter jsonb := j->'filter';\n jpitem jsonb;\n op text;\n args jsonb;\n arg jsonb;\n collections text[];\nBEGIN\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n END IF;\n IF NOT (filter @? '$.**.op ? (@ == \"or\" || @ == \"not\")') THEN\n FOR jpitem IN SELECT j FROM jsonb_path_query(filter,'strict $.** ? (@.args[*].property == \"collection\")'::jsonpath) j LOOP\n RAISE NOTICE 'JPITEM: %', jpitem;\n op := jpitem->>'op';\n args := jpitem->'args';\n IF op IN ('=', 'eq', 'in') THEN\n FOR arg IN SELECT a FROM jsonb_array_elements(args) a LOOP\n IF jsonb_typeof(arg) IN ('string', 'array') THEN\n RAISE NOTICE 'arg: %, collections: %', arg, collections;\n IF collections IS NULL OR collections = '{}'::text[] THEN\n collections := to_text_array(arg);\n ELSE\n collections := array_intersection(collections, to_text_array(arg));\n END IF;\n END IF;\n END LOOP;\n END IF;\n END LOOP;\n END IF;\n IF collections = '{}'::text[] THEN\n RETURN NULL;\n END IF;\n RETURN collections;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\nCREATE TABLE items (\n id text NOT NULL,\n geometry geometry NOT NULL,\n collection text NOT NULL,\n datetime timestamptz NOT NULL,\n end_datetime timestamptz NOT NULL,\n content JSONB NOT NULL\n)\nPARTITION BY LIST (collection)\n;\n\nCREATE INDEX \"datetime_idx\" ON items USING BTREE (datetime DESC, end_datetime ASC);\nCREATE INDEX \"geometry_idx\" ON items USING GIST (geometry);\n\nCREATE STATISTICS datetime_stats (dependencies) on datetime, end_datetime from items;\n\n\nALTER TABLE items ADD CONSTRAINT items_collections_fk FOREIGN KEY (collection) REFERENCES collections(id) ON DELETE CASCADE DEFERRABLE;\n\n\nCREATE OR REPLACE FUNCTION content_slim(_item jsonb) RETURNS jsonb AS $$\n SELECT strip_jsonb(_item - '{id,geometry,collection,type}'::text[], collection_base_item(_item->>'collection')) - '{id,geometry,collection,type}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_dehydrate(content jsonb) RETURNS items AS $$\n SELECT\n content->>'id' as id,\n stac_geom(content) as geometry,\n content->>'collection' as collection,\n stac_datetime(content) as datetime,\n stac_end_datetime(content) as end_datetime,\n content_slim(content) as content\n ;\n$$ LANGUAGE SQL STABLE;\n\nCREATE OR REPLACE FUNCTION include_field(f text, fields jsonb DEFAULT '{}'::jsonb) RETURNS boolean AS $$\nDECLARE\n includes jsonb := fields->'include';\n excludes jsonb := fields->'exclude';\nBEGIN\n IF f IS NULL THEN\n RETURN NULL;\n END IF;\n\n\n IF\n jsonb_typeof(excludes) = 'array'\n AND jsonb_array_length(excludes)>0\n AND excludes ? f\n THEN\n RETURN FALSE;\n END IF;\n\n IF\n (\n jsonb_typeof(includes) = 'array'\n AND jsonb_array_length(includes) > 0\n AND includes ? f\n ) OR\n (\n includes IS NULL\n OR jsonb_typeof(includes) = 'null'\n OR jsonb_array_length(includes) = 0\n )\n THEN\n RETURN TRUE;\n END IF;\n\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nDROP FUNCTION IF EXISTS content_hydrate(jsonb, jsonb, jsonb);\nCREATE OR REPLACE FUNCTION content_hydrate(\n _item jsonb,\n _base_item jsonb,\n fields jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\n SELECT merge_jsonb(\n jsonb_fields(_item, fields),\n jsonb_fields(_base_item, fields)\n );\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION content_hydrate(_item items, _collection collections, fields jsonb DEFAULT '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\n content jsonb;\n base_item jsonb := _collection.base_item;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry, 20)::jsonb;\n END IF;\n output := content_hydrate(\n jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'collection', _item.collection,\n 'type', 'Feature'\n ) || _item.content,\n _collection.base_item,\n fields\n );\n\n RETURN output;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_nonhydrated(\n _item items,\n fields jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry, 20)::jsonb;\n END IF;\n output := jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'collection', _item.collection,\n 'type', 'Feature'\n ) || _item.content;\n RETURN output;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_hydrate(_item items, fields jsonb DEFAULT '{}'::jsonb) RETURNS jsonb AS $$\n SELECT content_hydrate(\n _item,\n (SELECT c FROM collections c WHERE id=_item.collection LIMIT 1),\n fields\n );\n$$ LANGUAGE SQL STABLE;\n\n\nCREATE UNLOGGED TABLE items_staging (\n content JSONB NOT NULL\n);\nCREATE UNLOGGED TABLE items_staging_ignore (\n content JSONB NOT NULL\n);\nCREATE UNLOGGED TABLE items_staging_upsert (\n content JSONB NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION items_staging_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p record;\n _partitions text[];\n ts timestamptz := clock_timestamp();\nBEGIN\n RAISE NOTICE 'Creating Partitions. %', clock_timestamp() - ts;\n WITH ranges AS (\n SELECT\n n.content->>'collection' as collection,\n stac_daterange(n.content->'properties') as dtr\n FROM newdata n\n ), p AS (\n SELECT\n collection,\n lower(dtr) as datetime,\n upper(dtr) as end_datetime,\n (partition_name(\n collection,\n lower(dtr)\n )).partition_name as name\n FROM ranges\n )\n INSERT INTO partitions (collection, datetime_range, end_datetime_range)\n SELECT\n collection,\n tstzrange(min(datetime), max(datetime), '[]') as datetime_range,\n tstzrange(min(end_datetime), max(end_datetime), '[]') as end_datetime_range\n FROM p\n GROUP BY collection, name\n ON CONFLICT (name) DO UPDATE SET\n datetime_range = EXCLUDED.datetime_range,\n end_datetime_range = EXCLUDED.end_datetime_range\n ;\n\n RAISE NOTICE 'Doing the insert. %', clock_timestamp() - ts;\n IF TG_TABLE_NAME = 'items_staging' THEN\n INSERT INTO items\n SELECT\n (content_dehydrate(content)).*\n FROM newdata;\n DELETE FROM items_staging;\n ELSIF TG_TABLE_NAME = 'items_staging_ignore' THEN\n INSERT INTO items\n SELECT\n (content_dehydrate(content)).*\n FROM newdata\n ON CONFLICT DO NOTHING;\n DELETE FROM items_staging_ignore;\n ELSIF TG_TABLE_NAME = 'items_staging_upsert' THEN\n WITH staging_formatted AS (\n SELECT (content_dehydrate(content)).* FROM newdata\n ), deletes AS (\n DELETE FROM items i USING staging_formatted s\n WHERE\n i.id = s.id\n AND i.collection = s.collection\n AND i IS DISTINCT FROM s\n RETURNING i.id, i.collection\n )\n INSERT INTO items\n SELECT s.* FROM\n staging_formatted s\n JOIN deletes d\n USING (id, collection);\n DELETE FROM items_staging_upsert;\n END IF;\n RAISE NOTICE 'Done. %', clock_timestamp() - ts;\n\n RETURN NULL;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER items_staging_insert_trigger AFTER INSERT ON items_staging REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\nCREATE TRIGGER items_staging_insert_ignore_trigger AFTER INSERT ON items_staging_ignore REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\nCREATE TRIGGER items_staging_insert_upsert_trigger AFTER INSERT ON items_staging_upsert REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\n\n\n\nCREATE OR REPLACE FUNCTION item_by_id(_id text, _collection text DEFAULT NULL) RETURNS items AS\n$$\nDECLARE\n i items%ROWTYPE;\nBEGIN\n SELECT * INTO i FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection) LIMIT 1;\n RETURN i;\nEND;\n$$ LANGUAGE PLPGSQL STABLE SECURITY DEFINER SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION get_item(_id text, _collection text DEFAULT NULL) RETURNS jsonb AS $$\n SELECT content_hydrate(items) FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection);\n$$ LANGUAGE SQL STABLE SECURITY DEFINER SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION delete_item(_id text, _collection text DEFAULT NULL) RETURNS VOID AS $$\nDECLARE\nout items%ROWTYPE;\nBEGIN\n DELETE FROM items WHERE id = _id AND (_collection IS NULL OR collection=_collection) RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n--/*\nCREATE OR REPLACE FUNCTION create_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION update_item(content jsonb) RETURNS VOID AS $$\nDECLARE\n old items %ROWTYPE;\n out items%ROWTYPE;\nBEGIN\n PERFORM delete_item(content->>'id', content->>'collection');\n PERFORM create_item(content);\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION create_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION collection_bbox(id text) RETURNS jsonb AS $$\n SELECT (replace(replace(replace(st_extent(geometry)::text,'BOX(','[['),')',']]'),' ',','))::jsonb\n FROM items WHERE collection=$1;\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION collection_temporal_extent(id text) RETURNS jsonb AS $$\n SELECT to_jsonb(array[array[min(datetime)::text, max(datetime)::text]])\n FROM items WHERE collection=$1;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection_extents() RETURNS VOID AS $$\nUPDATE collections SET\n content = content ||\n jsonb_build_object(\n 'extent', jsonb_build_object(\n 'spatial', jsonb_build_object(\n 'bbox', collection_bbox(collections.id)\n ),\n 'temporal', jsonb_build_object(\n 'interval', collection_temporal_extent(collections.id)\n )\n )\n )\n;\n$$ LANGUAGE SQL;\nCREATE VIEW partition_steps AS\nSELECT\n name,\n date_trunc('month',lower(datetime_range)) as sdate,\n date_trunc('month', upper(datetime_range)) + '1 month'::interval as edate\n FROM partitions WHERE datetime_range IS NOT NULL AND datetime_range != 'empty'::tstzrange\n ORDER BY datetime_range ASC\n;\n\nCREATE OR REPLACE FUNCTION chunker(\n IN _where text,\n OUT s timestamptz,\n OUT e timestamptz\n) RETURNS SETOF RECORD AS $$\nDECLARE\n explain jsonb;\nBEGIN\n IF _where IS NULL THEN\n _where := ' TRUE ';\n END IF;\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s;', _where)\n INTO explain;\n\n RETURN QUERY\n WITH t AS (\n SELECT j->>0 as p FROM\n jsonb_path_query(\n explain,\n 'strict $.**.\"Relation Name\" ? (@ != null)'\n ) j\n ),\n parts AS (\n SELECT sdate, edate FROM t JOIN partition_steps ON (t.p = name)\n ),\n times AS (\n SELECT sdate FROM parts\n UNION\n SELECT edate FROM parts\n ),\n uniq AS (\n SELECT DISTINCT sdate FROM times ORDER BY sdate\n ),\n last AS (\n SELECT sdate, lead(sdate, 1) over () as edate FROM uniq\n )\n SELECT sdate, edate FROM last WHERE edate IS NOT NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION partition_queries(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN partitions text[] DEFAULT NULL\n) RETURNS SETOF text AS $$\nDECLARE\n query text;\n sdate timestamptz;\n edate timestamptz;\nBEGIN\nIF _where IS NULL OR trim(_where) = '' THEN\n _where = ' TRUE ';\nEND IF;\nRAISE NOTICE 'Getting chunks for % %', _where, _orderby;\nIF _orderby ILIKE 'datetime d%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 DESC LOOP\n RETURN NEXT format($q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n $q$,\n sdate,\n edate,\n _where,\n _orderby\n );\n END LOOP;\nELSIF _orderby ILIKE 'datetime a%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 ASC LOOP\n RETURN NEXT format($q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n $q$,\n sdate,\n edate,\n _where,\n _orderby\n );\n END LOOP;\nELSE\n query := format($q$\n SELECT * FROM items\n WHERE %s\n ORDER BY %s\n $q$, _where, _orderby\n );\n\n RETURN NEXT query;\n RETURN;\nEND IF;\n\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION partition_query_view(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN _limit int DEFAULT 10\n) RETURNS text AS $$\n WITH p AS (\n SELECT * FROM partition_queries(_where, _orderby) p\n )\n SELECT\n CASE WHEN EXISTS (SELECT 1 FROM p) THEN\n (SELECT format($q$\n SELECT * FROM (\n %s\n ) total LIMIT %s\n $q$,\n string_agg(\n format($q$ SELECT * FROM ( %s ) AS sub $q$, p),\n '\n UNION ALL\n '\n ),\n _limit\n ))\n ELSE NULL\n END FROM p;\n$$ LANGUAGE SQL IMMUTABLE;\n\n\n\n\nCREATE OR REPLACE FUNCTION stac_search_to_where(j jsonb) RETURNS text AS $$\nDECLARE\n where_segments text[];\n _where text;\n dtrange tstzrange;\n collections text[];\n geom geometry;\n sdate timestamptz;\n edate timestamptz;\n filterlang text;\n filter jsonb := j->'filter';\nBEGIN\n IF j ? 'ids' THEN\n where_segments := where_segments || format('id = ANY (%L) ', to_text_array(j->'ids'));\n END IF;\n\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n where_segments := where_segments || format('collection = ANY (%L) ', collections);\n END IF;\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n\n where_segments := where_segments || format(' datetime <= %L::timestamptz AND end_datetime >= %L::timestamptz ',\n edate,\n sdate\n );\n END IF;\n\n geom := stac_geom(j);\n IF geom IS NOT NULL THEN\n where_segments := where_segments || format('st_intersects(geometry, %L)',geom);\n END IF;\n\n filterlang := COALESCE(\n j->>'filter-lang',\n get_setting('default-filter-lang', j->'conf')\n );\n IF NOT filter @? '$.**.op' THEN\n filterlang := 'cql-json';\n END IF;\n\n IF filterlang NOT IN ('cql-json','cql2-json') AND j ? 'filter' THEN\n RAISE EXCEPTION '% is not a supported filter-lang. Please use cql-json or cql2-json.', filterlang;\n END IF;\n\n IF j ? 'query' AND j ? 'filter' THEN\n RAISE EXCEPTION 'Can only use either query or filter at one time.';\n END IF;\n\n IF j ? 'query' THEN\n filter := query_to_cql2(j->'query');\n ELSIF filterlang = 'cql-json' THEN\n filter := cql1_to_cql2(filter);\n END IF;\n RAISE NOTICE 'FILTER: %', filter;\n where_segments := where_segments || cql2_query(filter);\n IF cardinality(where_segments) < 1 THEN\n RETURN ' TRUE ';\n END IF;\n\n _where := array_to_string(array_remove(where_segments, NULL), ' AND ');\n\n IF _where IS NULL OR BTRIM(_where) = '' THEN\n RETURN ' TRUE ';\n END IF;\n RETURN _where;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION parse_sort_dir(_dir text, reverse boolean default false) RETURNS text AS $$\n WITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n ) SELECT\n CASE\n WHEN NOT reverse THEN d\n WHEN d = 'ASC' THEN 'DESC'\n WHEN d = 'DESC' THEN 'ASC'\n END\n FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION sort_dir_to_op(_dir text, prev boolean default false) RETURNS text AS $$\n WITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n ) SELECT\n CASE\n WHEN d = 'ASC' AND prev THEN '<='\n WHEN d = 'DESC' AND prev THEN '>='\n WHEN d = 'ASC' THEN '>='\n WHEN d = 'DESC' THEN '<='\n END\n FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION sort_sqlorderby(\n _search jsonb DEFAULT NULL,\n reverse boolean DEFAULT FALSE\n) RETURNS text AS $$\n WITH sortby AS (\n SELECT coalesce(_search->'sortby','[{\"field\":\"datetime\", \"direction\":\"desc\"}]') as sort\n ), withid AS (\n SELECT CASE\n WHEN sort @? '$[*] ? (@.field == \"id\")' THEN sort\n ELSE sort || '[{\"field\":\"id\", \"direction\":\"desc\"}]'::jsonb\n END as sort\n FROM sortby\n ), withid_rows AS (\n SELECT jsonb_array_elements(sort) as value FROM withid\n ),sorts AS (\n SELECT\n coalesce(\n -- field_orderby((items_path(value->>'field')).path_txt),\n (queryable(value->>'field')).expression\n ) as key,\n parse_sort_dir(value->>'direction', reverse) as dir\n FROM withid_rows\n )\n SELECT array_to_string(\n array_agg(concat(key, ' ', dir)),\n ', '\n ) FROM sorts;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_sort_dir(sort_item jsonb) RETURNS text AS $$\n SELECT CASE WHEN sort_item->>'direction' ILIKE 'desc%' THEN 'DESC' ELSE 'ASC' END;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION get_token_filter(_search jsonb = '{}'::jsonb, token_rec jsonb DEFAULT NULL) RETURNS text AS $$\nDECLARE\n token_id text;\n filters text[] := '{}'::text[];\n prev boolean := TRUE;\n field text;\n dir text;\n sort record;\n orfilters text[] := '{}'::text[];\n andfilters text[] := '{}'::text[];\n output text;\n token_where text;\nBEGIN\n RAISE NOTICE 'Getting Token Filter. % %', _search, token_rec;\n -- If no token provided return NULL\n IF token_rec IS NULL THEN\n IF NOT (_search ? 'token' AND\n (\n (_search->>'token' ILIKE 'prev:%')\n OR\n (_search->>'token' ILIKE 'next:%')\n )\n ) THEN\n RETURN NULL;\n END IF;\n prev := (_search->>'token' ILIKE 'prev:%');\n token_id := substr(_search->>'token', 6);\n SELECT to_jsonb(items) INTO token_rec\n FROM items WHERE id=token_id;\n END IF;\n RAISE NOTICE 'TOKEN ID: % %', token_rec, token_rec->'id';\n\n CREATE TEMP TABLE sorts (\n _row int GENERATED ALWAYS AS IDENTITY NOT NULL,\n _field text PRIMARY KEY,\n _dir text NOT NULL,\n _val text\n ) ON COMMIT DROP;\n\n -- Make sure we only have distinct columns to sort with taking the first one we get\n INSERT INTO sorts (_field, _dir)\n SELECT\n (queryable(value->>'field')).expression,\n get_sort_dir(value)\n FROM\n jsonb_array_elements(coalesce(_search->'sortby','[{\"field\":\"datetime\",\"direction\":\"desc\"}]'))\n ON CONFLICT DO NOTHING\n ;\n RAISE NOTICE 'sorts 1: %', (SELECT jsonb_agg(to_json(sorts)) FROM sorts);\n -- Get the first sort direction provided. As the id is a primary key, if there are any\n -- sorts after id they won't do anything, so make sure that id is the last sort item.\n SELECT _dir INTO dir FROM sorts ORDER BY _row ASC LIMIT 1;\n IF EXISTS (SELECT 1 FROM sorts WHERE _field = 'id') THEN\n DELETE FROM sorts WHERE _row > (SELECT _row FROM sorts WHERE _field = 'id' ORDER BY _row ASC);\n ELSE\n INSERT INTO sorts (_field, _dir) VALUES ('id', dir);\n END IF;\n\n -- Add value from looked up item to the sorts table\n UPDATE sorts SET _val=quote_literal(token_rec->>_field);\n\n -- Check if all sorts are the same direction and use row comparison\n -- to filter\n RAISE NOTICE 'sorts 2: %', (SELECT jsonb_agg(to_json(sorts)) FROM sorts);\n\n IF (SELECT count(DISTINCT _dir) FROM sorts) = 1 THEN\n SELECT format(\n '(%s) %s (%s)',\n concat_ws(', ', VARIADIC array_agg(quote_ident(_field))),\n CASE WHEN (prev AND dir = 'ASC') OR (NOT prev AND dir = 'DESC') THEN '<' ELSE '>' END,\n concat_ws(', ', VARIADIC array_agg(_val))\n ) INTO output FROM sorts\n WHERE token_rec ? _field\n ;\n ELSE\n FOR sort IN SELECT * FROM sorts ORDER BY _row asc LOOP\n RAISE NOTICE 'SORT: %', sort;\n IF sort._row = 1 THEN\n orfilters := orfilters || format('(%s %s %s)',\n quote_ident(sort._field),\n CASE WHEN (prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC') THEN '<' ELSE '>' END,\n sort._val\n );\n ELSE\n orfilters := orfilters || format('(%s AND %s %s %s)',\n array_to_string(andfilters, ' AND '),\n quote_ident(sort._field),\n CASE WHEN (prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC') THEN '<' ELSE '>' END,\n sort._val\n );\n\n END IF;\n andfilters := andfilters || format('%s = %s',\n quote_ident(sort._field),\n sort._val\n );\n END LOOP;\n output := array_to_string(orfilters, ' OR ');\n END IF;\n DROP TABLE IF EXISTS sorts;\n token_where := concat('(',coalesce(output,'true'),')');\n IF trim(token_where) = '' THEN\n token_where := NULL;\n END IF;\n RAISE NOTICE 'TOKEN_WHERE: |%|',token_where;\n RETURN token_where;\n END;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION search_tohash(jsonb) RETURNS jsonb AS $$\n SELECT $1 - '{token,limit,context,includes,excludes}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION search_hash(jsonb, jsonb) RETURNS text AS $$\n SELECT md5(concat(search_tohash($1)::text,$2::text));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TABLE IF NOT EXISTS searches(\n hash text GENERATED ALWAYS AS (search_hash(search, metadata)) STORED PRIMARY KEY,\n search jsonb NOT NULL,\n _where text,\n orderby text,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n metadata jsonb DEFAULT '{}'::jsonb NOT NULL\n);\nCREATE TABLE IF NOT EXISTS search_wheres(\n id bigint generated always as identity primary key,\n _where text NOT NULL,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n statslastupdated timestamptz,\n estimated_count bigint,\n estimated_cost float,\n time_to_estimate float,\n total_count bigint,\n time_to_count float,\n partitions text[]\n);\n\nCREATE INDEX IF NOT EXISTS search_wheres_partitions ON search_wheres USING GIN (partitions);\nCREATE UNIQUE INDEX IF NOT EXISTS search_wheres_where ON search_wheres ((md5(_where)));\n\nCREATE OR REPLACE FUNCTION where_stats(inwhere text, updatestats boolean default false, conf jsonb default null) RETURNS search_wheres AS $$\nDECLARE\n t timestamptz;\n i interval;\n explain_json jsonb;\n partitions text[];\n sw search_wheres%ROWTYPE;\n inwhere_hash text := md5(inwhere);\n _context text := lower(context(conf));\n _stats_ttl interval := context_stats_ttl(conf);\n _estimated_cost float := context_estimated_cost(conf);\n _estimated_count int := context_estimated_count(conf);\nBEGIN\n IF _context = 'off' THEN\n sw._where := inwhere;\n return sw;\n END IF;\n\n SELECT * INTO sw FROM search_wheres WHERE md5(_where)=inwhere_hash FOR UPDATE;\n\n -- Update statistics if explicitly set, if statistics do not exist, or statistics ttl has expired\n IF NOT updatestats THEN\n RAISE NOTICE 'Checking if update is needed for: % .', inwhere;\n RAISE NOTICE 'Stats Last Updated: %', sw.statslastupdated;\n RAISE NOTICE 'TTL: %, Age: %', _stats_ttl, now() - sw.statslastupdated;\n RAISE NOTICE 'Context: %, Existing Total: %', _context, sw.total_count;\n IF\n sw.statslastupdated IS NULL\n OR (now() - sw.statslastupdated) > _stats_ttl\n OR (context(conf) != 'off' AND sw.total_count IS NULL)\n THEN\n updatestats := TRUE;\n END IF;\n END IF;\n\n sw._where := inwhere;\n sw.lastused := now();\n sw.usecount := coalesce(sw.usecount,0) + 1;\n\n IF NOT updatestats THEN\n UPDATE search_wheres SET\n lastused = sw.lastused,\n usecount = sw.usecount\n WHERE md5(_where) = inwhere_hash\n RETURNING * INTO sw\n ;\n RETURN sw;\n END IF;\n\n -- Use explain to get estimated count/cost and a list of the partitions that would be hit by the query\n t := clock_timestamp();\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s', inwhere)\n INTO explain_json;\n RAISE NOTICE 'Time for just the explain: %', clock_timestamp() - t;\n i := clock_timestamp() - t;\n\n sw.statslastupdated := now();\n sw.estimated_count := explain_json->0->'Plan'->'Plan Rows';\n sw.estimated_cost := explain_json->0->'Plan'->'Total Cost';\n sw.time_to_estimate := extract(epoch from i);\n\n RAISE NOTICE 'ESTIMATED_COUNT: % < %', sw.estimated_count, _estimated_count;\n RAISE NOTICE 'ESTIMATED_COST: % < %', sw.estimated_cost, _estimated_cost;\n\n -- Do a full count of rows if context is set to on or if auto is set and estimates are low enough\n IF\n _context = 'on'\n OR\n ( _context = 'auto' AND\n (\n sw.estimated_count < _estimated_count\n AND\n sw.estimated_cost < _estimated_cost\n )\n )\n THEN\n t := clock_timestamp();\n RAISE NOTICE 'Calculating actual count...';\n EXECUTE format(\n 'SELECT count(*) FROM items WHERE %s',\n inwhere\n ) INTO sw.total_count;\n i := clock_timestamp() - t;\n RAISE NOTICE 'Actual Count: % -- %', sw.total_count, i;\n sw.time_to_count := extract(epoch FROM i);\n ELSE\n sw.total_count := NULL;\n sw.time_to_count := NULL;\n END IF;\n\n\n INSERT INTO search_wheres\n (_where, lastused, usecount, statslastupdated, estimated_count, estimated_cost, time_to_estimate, partitions, total_count, time_to_count)\n SELECT sw._where, sw.lastused, sw.usecount, sw.statslastupdated, sw.estimated_count, sw.estimated_cost, sw.time_to_estimate, sw.partitions, sw.total_count, sw.time_to_count\n ON CONFLICT ((md5(_where)))\n DO UPDATE\n SET\n lastused = sw.lastused,\n usecount = sw.usecount,\n statslastupdated = sw.statslastupdated,\n estimated_count = sw.estimated_count,\n estimated_cost = sw.estimated_cost,\n time_to_estimate = sw.time_to_estimate,\n total_count = sw.total_count,\n time_to_count = sw.time_to_count\n ;\n RETURN sw;\nEND;\n$$ LANGUAGE PLPGSQL ;\n\n\n\nDROP FUNCTION IF EXISTS search_query;\nCREATE OR REPLACE FUNCTION search_query(\n _search jsonb = '{}'::jsonb,\n updatestats boolean = false,\n _metadata jsonb = '{}'::jsonb\n) RETURNS searches AS $$\nDECLARE\n search searches%ROWTYPE;\n pexplain jsonb;\n t timestamptz;\n i interval;\nBEGIN\n SELECT * INTO search FROM searches\n WHERE hash=search_hash(_search, _metadata) FOR UPDATE;\n\n -- Calculate the where clause if not already calculated\n IF search._where IS NULL THEN\n search._where := stac_search_to_where(_search);\n END IF;\n\n -- Calculate the order by clause if not already calculated\n IF search.orderby IS NULL THEN\n search.orderby := sort_sqlorderby(_search);\n END IF;\n\n PERFORM where_stats(search._where, updatestats, _search->'conf');\n\n search.lastused := now();\n search.usecount := coalesce(search.usecount, 0) + 1;\n INSERT INTO searches (search, _where, orderby, lastused, usecount, metadata)\n VALUES (_search, search._where, search.orderby, search.lastused, search.usecount, _metadata)\n ON CONFLICT (hash) DO\n UPDATE SET\n _where = EXCLUDED._where,\n orderby = EXCLUDED.orderby,\n lastused = EXCLUDED.lastused,\n usecount = EXCLUDED.usecount,\n metadata = EXCLUDED.metadata\n RETURNING * INTO search\n ;\n RETURN search;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION search(_search jsonb = '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n searches searches%ROWTYPE;\n _where text;\n token_where text;\n full_where text;\n orderby text;\n query text;\n token_type text := substr(_search->>'token',1,4);\n _limit int := coalesce((_search->>'limit')::int, 10);\n curs refcursor;\n cntr int := 0;\n iter_record items%ROWTYPE;\n first_record jsonb;\n first_item items%ROWTYPE;\n last_item items%ROWTYPE;\n last_record jsonb;\n out_records jsonb := '[]'::jsonb;\n prev_query text;\n next text;\n prev_id text;\n has_next boolean := false;\n has_prev boolean := false;\n prev text;\n total_count bigint;\n context jsonb;\n collection jsonb;\n includes text[];\n excludes text[];\n exit_flag boolean := FALSE;\n batches int := 0;\n timer timestamptz := clock_timestamp();\n pstart timestamptz;\n pend timestamptz;\n pcurs refcursor;\n search_where search_wheres%ROWTYPE;\n id text;\nBEGIN\nCREATE TEMP TABLE results (content jsonb) ON COMMIT DROP;\n-- if ids is set, short circuit and just use direct ids query for each id\n-- skip any paging or caching\n-- hard codes ordering in the same order as the array of ids\nIF _search ? 'ids' THEN\n INSERT INTO results\n SELECT\n CASE WHEN _search->'conf'->>'nohydrate' IS NOT NULL AND (_search->'conf'->>'nohydrate')::boolean = true THEN\n content_nonhydrated(items, _search->'fields')\n ELSE\n content_hydrate(items, _search->'fields')\n END\n FROM items WHERE\n items.id = ANY(to_text_array(_search->'ids'))\n AND\n CASE WHEN _search ? 'collections' THEN\n items.collection = ANY(to_text_array(_search->'collections'))\n ELSE TRUE\n END\n ORDER BY items.datetime desc, items.id desc\n ;\n SELECT INTO total_count count(*) FROM results;\nELSE\n searches := search_query(_search);\n _where := searches._where;\n orderby := searches.orderby;\n search_where := where_stats(_where);\n total_count := coalesce(search_where.total_count, search_where.estimated_count);\n\n IF token_type='prev' THEN\n token_where := get_token_filter(_search, null::jsonb);\n orderby := sort_sqlorderby(_search, TRUE);\n END IF;\n IF token_type='next' THEN\n token_where := get_token_filter(_search, null::jsonb);\n END IF;\n\n full_where := concat_ws(' AND ', _where, token_where);\n RAISE NOTICE 'FULL QUERY % %', full_where, clock_timestamp()-timer;\n timer := clock_timestamp();\n\n FOR query IN SELECT partition_queries(full_where, orderby, search_where.partitions) LOOP\n timer := clock_timestamp();\n query := format('%s LIMIT %s', query, _limit + 1);\n RAISE NOTICE 'Partition Query: %', query;\n batches := batches + 1;\n -- curs = create_cursor(query);\n OPEN curs FOR EXECUTE query;\n LOOP\n FETCH curs into iter_record;\n EXIT WHEN NOT FOUND;\n cntr := cntr + 1;\n\n IF _search->'conf'->>'nohydrate' IS NOT NULL AND (_search->'conf'->>'nohydrate')::boolean = true THEN\n last_record := content_nonhydrated(iter_record, _search->'fields');\n ELSE\n last_record := content_hydrate(iter_record, _search->'fields');\n END IF;\n last_item := iter_record;\n IF cntr = 1 THEN\n first_item := last_item;\n first_record := last_record;\n END IF;\n IF cntr <= _limit THEN\n INSERT INTO results (content) VALUES (last_record);\n ELSIF cntr > _limit THEN\n has_next := true;\n exit_flag := true;\n EXIT;\n END IF;\n END LOOP;\n CLOSE curs;\n RAISE NOTICE 'Query took %.', clock_timestamp()-timer;\n timer := clock_timestamp();\n EXIT WHEN exit_flag;\n END LOOP;\n RAISE NOTICE 'Scanned through % partitions.', batches;\nEND IF;\n\nSELECT jsonb_agg(content) INTO out_records FROM results WHERE content is not NULL;\n\nDROP TABLE results;\n\n\n-- Flip things around if this was the result of a prev token query\nIF token_type='prev' THEN\n out_records := flip_jsonb_array(out_records);\n first_record := last_record;\nEND IF;\n\n-- If this query has a token, see if there is data before the first record\nIF _search ? 'token' THEN\n prev_query := format(\n 'SELECT 1 FROM items WHERE %s LIMIT 1',\n concat_ws(\n ' AND ',\n _where,\n trim(get_token_filter(_search, to_jsonb(first_item)))\n )\n );\n RAISE NOTICE 'Query to get previous record: % --- %', prev_query, first_record;\n EXECUTE prev_query INTO has_prev;\n IF FOUND and has_prev IS NOT NULL THEN\n RAISE NOTICE 'Query results from prev query: %', has_prev;\n has_prev := TRUE;\n END IF;\nEND IF;\nhas_prev := COALESCE(has_prev, FALSE);\n\nIF has_prev THEN\n prev := out_records->0->>'id';\nEND IF;\nIF has_next OR token_type='prev' THEN\n next := out_records->-1->>'id';\nEND IF;\n\nIF context(_search->'conf') != 'off' THEN\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'matched', total_count,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\nELSE\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\nEND IF;\n\ncollection := jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb),\n 'next', next,\n 'prev', prev,\n 'context', context\n);\n\nRETURN collection;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER SET SEARCH_PATH TO pgstac, public;\n\n\nCREATE OR REPLACE FUNCTION search_cursor(_search jsonb = '{}'::jsonb) RETURNS refcursor AS $$\nDECLARE\n curs refcursor;\n searches searches%ROWTYPE;\n _where text;\n _orderby text;\n q text;\n\nBEGIN\n searches := search_query(_search);\n _where := searches._where;\n _orderby := searches.orderby;\n\n OPEN curs FOR\n WITH p AS (\n SELECT * FROM partition_queries(_where, _orderby) p\n )\n SELECT\n CASE WHEN EXISTS (SELECT 1 FROM p) THEN\n (SELECT format($q$\n SELECT * FROM (\n %s\n ) total\n $q$,\n string_agg(\n format($q$ SELECT * FROM ( %s ) AS sub $q$, p),\n '\n UNION ALL\n '\n )\n ))\n ELSE NULL\n END FROM p;\n RETURN curs;\nEND;\n$$ LANGUAGE PLPGSQL;\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION tileenvelope(zoom int, x int, y int) RETURNS geometry AS $$\nWITH t AS (\n SELECT\n 20037508.3427892 as merc_max,\n -20037508.3427892 as merc_min,\n (2 * 20037508.3427892) / (2 ^ zoom) as tile_size\n)\nSELECT st_makeenvelope(\n merc_min + (tile_size * x),\n merc_max - (tile_size * (y + 1)),\n merc_min + (tile_size * (x + 1)),\n merc_max - (tile_size * y),\n 3857\n) FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;DROP FUNCTION IF EXISTS mercgrid;\n\n\nCREATE OR REPLACE FUNCTION ftime() RETURNS interval as $$\nSELECT age(clock_timestamp(), transaction_timestamp());\n$$ LANGUAGE SQL;\nSET SEARCH_PATH to pgstac, public;\n\nDROP FUNCTION IF EXISTS geometrysearch;\nCREATE OR REPLACE FUNCTION geometrysearch(\n IN geom geometry,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE, -- Return as soon as the passed in geometry is full covered\n IN skipcovered boolean DEFAULT TRUE -- Skip any items that would show up completely under the previous items\n) RETURNS jsonb AS $$\nDECLARE\n search searches%ROWTYPE;\n curs refcursor;\n _where text;\n query text;\n iter_record items%ROWTYPE;\n out_records jsonb := '{}'::jsonb[];\n exit_flag boolean := FALSE;\n counter int := 1;\n scancounter int := 1;\n remaining_limit int := _scanlimit;\n tilearea float;\n unionedgeom geometry;\n clippedgeom geometry;\n unionedgeom_area float := 0;\n prev_area float := 0;\n excludes text[];\n includes text[];\n\nBEGIN\n DROP TABLE IF EXISTS pgstac_results;\n CREATE TEMP TABLE pgstac_results (content jsonb) ON COMMIT DROP;\n\n -- If skipcovered is true then you will always want to exit when the passed in geometry is full\n IF skipcovered THEN\n exitwhenfull := TRUE;\n END IF;\n\n SELECT * INTO search FROM searches WHERE hash=queryhash;\n\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Search with Query Hash % Not Found', queryhash;\n END IF;\n\n tilearea := st_area(geom);\n _where := format('%s AND st_intersects(geometry, %L::geometry)', search._where, geom);\n\n\n FOR query IN SELECT * FROM partition_queries(_where, search.orderby) LOOP\n query := format('%s LIMIT %L', query, remaining_limit);\n RAISE NOTICE '%', query;\n OPEN curs FOR EXECUTE query;\n LOOP\n FETCH curs INTO iter_record;\n EXIT WHEN NOT FOUND;\n IF exitwhenfull OR skipcovered THEN -- If we are not using exitwhenfull or skipcovered, we do not need to do expensive geometry operations\n clippedgeom := st_intersection(geom, iter_record.geometry);\n\n IF unionedgeom IS NULL THEN\n unionedgeom := clippedgeom;\n ELSE\n unionedgeom := st_union(unionedgeom, clippedgeom);\n END IF;\n\n unionedgeom_area := st_area(unionedgeom);\n\n IF skipcovered AND prev_area = unionedgeom_area THEN\n scancounter := scancounter + 1;\n CONTINUE;\n END IF;\n\n prev_area := unionedgeom_area;\n\n RAISE NOTICE '% % % %', unionedgeom_area/tilearea, counter, scancounter, ftime();\n END IF;\n RAISE NOTICE '% %', iter_record, content_hydrate(iter_record, fields);\n INSERT INTO pgstac_results (content) VALUES (content_hydrate(iter_record, fields));\n\n IF counter >= _limit\n OR scancounter > _scanlimit\n OR ftime() > _timelimit\n OR (exitwhenfull AND unionedgeom_area >= tilearea)\n THEN\n exit_flag := TRUE;\n EXIT;\n END IF;\n counter := counter + 1;\n scancounter := scancounter + 1;\n\n END LOOP;\n CLOSE curs;\n EXIT WHEN exit_flag;\n remaining_limit := _scanlimit - scancounter;\n END LOOP;\n\n SELECT jsonb_agg(content) INTO out_records FROM pgstac_results WHERE content IS NOT NULL;\n\n RETURN jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb)\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDROP FUNCTION IF EXISTS geojsonsearch;\nCREATE OR REPLACE FUNCTION geojsonsearch(\n IN geojson jsonb,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_geomfromgeojson(geojson),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS xyzsearch;\nCREATE OR REPLACE FUNCTION xyzsearch(\n IN _x int,\n IN _y int,\n IN _z int,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_transform(tileenvelope(_z, _x, _y), 4326),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nGRANT ALL ON SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac to pgstac_admin;\n\n-- pgstac_read role limited to using function apis\nGRANT EXECUTE ON FUNCTION search TO pgstac_read;\nGRANT EXECUTE ON FUNCTION search_query TO pgstac_read;\nGRANT EXECUTE ON FUNCTION item_by_id TO pgstac_read;\nGRANT EXECUTE ON FUNCTION get_item TO pgstac_read;\n\nGRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_ingest;\nGRANT USAGE ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_ingest;\nSELECT set_version('0.6.6');\n" }, { "path": "src/pgstac/migrations/pgstac.0.6.7-0.6.8.sql", "content": "SET SEARCH_PATH to pgstac, public;\ndrop function if exists \"pgstac\".\"queryable\"(dotpath text, OUT path text, OUT expression text, OUT wrapper text);\n\nset check_function_bodies = off;\n\nCREATE OR REPLACE FUNCTION pgstac.get_queryables(_collection text DEFAULT NULL::text)\n RETURNS jsonb\n LANGUAGE sql\nAS $function$\n SELECT\n CASE\n WHEN _collection IS NULL THEN get_queryables(NULL::text[])\n ELSE get_queryables(ARRAY[_collection])\n END\n ;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.get_queryables(_collection_ids text[] DEFAULT NULL::text[])\n RETURNS jsonb\n LANGUAGE sql\n STABLE\nAS $function$\n SELECT\n jsonb_build_object(\n '$schema', 'http://json-schema.org/draft-07/schema#',\n '$id', 'https://example.org/queryables',\n 'type', 'object',\n 'title', 'Stac Queryables.',\n 'properties', jsonb_object_agg(\n name,\n definition\n )\n )\n FROM queryables\n WHERE\n _collection_ids IS NULL OR\n cardinality(_collection_ids) = 0 OR\n collection_ids IS NULL OR\n _collection_ids && collection_ids\n ;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.missing_queryables(_collection text, _tablesample integer DEFAULT 5)\n RETURNS TABLE(collection text, name text, definition jsonb, property_wrapper text)\n LANGUAGE plpgsql\nAS $function$\nDECLARE\n q text;\n _partition text;\n explain_json json;\n psize bigint;\nBEGIN\n SELECT format('_items_%s', key) INTO _partition FROM collections WHERE id=_collection;\n\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM %I;', _partition)\n INTO explain_json;\n psize := explain_json->0->'Plan'->'Plan Rows';\n IF _tablesample * .01 * psize < 10 THEN\n _tablesample := 100;\n END IF;\n RAISE NOTICE 'Using tablesample % to find missing queryables from % % that has ~% rows', _tablesample, _collection, _partition, psize;\n\n q := format(\n $q$\n WITH q AS (\n SELECT * FROM queryables\n WHERE\n collection_ids IS NULL\n OR %L = ANY(collection_ids)\n ), t AS (\n SELECT\n content->'properties' AS properties\n FROM\n %I\n TABLESAMPLE SYSTEM(%L)\n ), p AS (\n SELECT DISTINCT ON (key)\n key,\n value\n FROM t\n JOIN LATERAL jsonb_each(properties) ON TRUE\n LEFT JOIN q ON (q.name=key)\n WHERE q.definition IS NULL\n )\n SELECT\n %L,\n key,\n jsonb_build_object('type',jsonb_typeof(value)) as definition,\n CASE jsonb_typeof(value)\n WHEN 'number' THEN 'to_float'\n WHEN 'array' THEN 'to_text_array'\n ELSE 'to_text'\n END\n FROM p;\n $q$,\n _collection,\n _partition,\n _tablesample,\n _collection\n );\n RETURN QUERY EXECUTE q;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.missing_queryables(_tablesample integer DEFAULT 5)\n RETURNS TABLE(collection_ids text[], name text, definition jsonb, property_wrapper text)\n LANGUAGE sql\nAS $function$\n SELECT\n array_agg(collection),\n name,\n definition,\n property_wrapper\n FROM\n collections\n JOIN LATERAL\n missing_queryables(id, _tablesample) c\n ON TRUE\n GROUP BY\n 2,3,4\n ORDER BY 2,1\n ;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.queryable(dotpath text, OUT path text, OUT expression text, OUT wrapper text, OUT nulled_wrapper text)\n RETURNS record\n LANGUAGE plpgsql\n STABLE STRICT\nAS $function$\nDECLARE\n q RECORD;\n path_elements text[];\nBEGIN\n IF dotpath IN ('id', 'geometry', 'datetime', 'end_datetime', 'collection') THEN\n path := dotpath;\n expression := dotpath;\n wrapper := NULL;\n RETURN;\n END IF;\n SELECT * INTO q FROM queryables\n WHERE\n name=dotpath\n OR name = 'properties.' || dotpath\n OR name = replace(dotpath, 'properties.', '')\n ;\n IF q.property_wrapper IS NULL THEN\n IF q.definition->>'type' = 'number' THEN\n wrapper := 'to_float';\n nulled_wrapper := wrapper;\n ELSIF q.definition->>'format' = 'date-time' THEN\n wrapper := 'to_tstz';\n nulled_wrapper := wrapper;\n ELSE\n nulled_wrapper := NULL;\n wrapper := 'to_text';\n END IF;\n ELSE\n wrapper := q.property_wrapper;\n nulled_wrapper := wrapper;\n END IF;\n IF q.property_path IS NOT NULL THEN\n path := q.property_path;\n ELSE\n path_elements := string_to_array(dotpath, '.');\n IF path_elements[1] IN ('links', 'assets', 'stac_version', 'stac_extensions') THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSIF path_elements[1] = 'properties' THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSE\n path := format($F$content->'properties'->%s$F$, array_to_path(path_elements));\n END IF;\n END IF;\n expression := format('%I(%s)', wrapper, path);\n RETURN;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.cql2_query(j jsonb, wrapper text DEFAULT NULL::text)\n RETURNS text\n LANGUAGE plpgsql\n STABLE\nAS $function$\n#variable_conflict use_variable\nDECLARE\n args jsonb := j->'args';\n arg jsonb;\n op text := lower(j->>'op');\n cql2op RECORD;\n literal text;\n _wrapper text;\n leftarg text;\n rightarg text;\nBEGIN\n IF j IS NULL OR (op IS NOT NULL AND args IS NULL) THEN\n RETURN NULL;\n END IF;\n RAISE NOTICE 'CQL2_QUERY: %', j;\n IF j ? 'filter' THEN\n RETURN cql2_query(j->'filter');\n END IF;\n\n IF j ? 'upper' THEN\n RETURN cql2_query(jsonb_build_object('op', 'upper', 'args', j->'upper'));\n END IF;\n\n IF j ? 'lower' THEN\n RETURN cql2_query(jsonb_build_object('op', 'lower', 'args', j->'lower'));\n END IF;\n\n -- Temporal Query\n IF op ilike 't_%' or op = 'anyinteracts' THEN\n RETURN temporal_op_query(op, args);\n END IF;\n\n -- If property is a timestamp convert it to text to use with\n -- general operators\n IF j ? 'timestamp' THEN\n RETURN format('%L::timestamptz', to_tstz(j->'timestamp'));\n END IF;\n IF j ? 'interval' THEN\n RAISE EXCEPTION 'Please use temporal operators when using intervals.';\n RETURN NONE;\n END IF;\n\n -- Spatial Query\n IF op ilike 's_%' or op = 'intersects' THEN\n RETURN spatial_op_query(op, args);\n END IF;\n\n IF op IN ('a_equals','a_contains','a_contained_by','a_overlaps') THEN\n IF args->0 ? 'property' THEN\n leftarg := format('to_text_array(%s)', (queryable(args->0->>'property')).path);\n END IF;\n IF args->1 ? 'property' THEN\n rightarg := format('to_text_array(%s)', (queryable(args->1->>'property')).path);\n END IF;\n RETURN FORMAT(\n '%s %s %s',\n COALESCE(leftarg, quote_literal(to_text_array(args->0))),\n CASE op\n WHEN 'a_equals' THEN '='\n WHEN 'a_contains' THEN '@>'\n WHEN 'a_contained_by' THEN '<@'\n WHEN 'a_overlaps' THEN '&&'\n END,\n COALESCE(rightarg, quote_literal(to_text_array(args->1)))\n );\n END IF;\n\n IF op = 'in' THEN\n RAISE NOTICE 'IN : % % %', args, jsonb_build_array(args->0), args->1;\n args := jsonb_build_array(args->0) || (args->1);\n RAISE NOTICE 'IN2 : %', args;\n END IF;\n\n\n\n IF op = 'between' THEN\n args = jsonb_build_array(\n args->0,\n args->1->0,\n args->1->1\n );\n END IF;\n\n -- Make sure that args is an array and run cql2_query on\n -- each element of the array\n RAISE NOTICE 'ARGS PRE: %', args;\n IF j ? 'args' THEN\n IF jsonb_typeof(args) != 'array' THEN\n args := jsonb_build_array(args);\n END IF;\n\n IF jsonb_path_exists(args, '$[*] ? (@.property == \"id\" || @.property == \"datetime\" || @.property == \"end_datetime\" || @.property == \"collection\")') THEN\n wrapper := NULL;\n ELSE\n -- if any of the arguments are a property, try to get the property_wrapper\n FOR arg IN SELECT jsonb_path_query(args, '$[*] ? (@.property != null)') LOOP\n RAISE NOTICE 'Arg: %', arg;\n wrapper := (queryable(arg->>'property')).nulled_wrapper;\n RAISE NOTICE 'Property: %, Wrapper: %', arg, wrapper;\n IF wrapper IS NOT NULL THEN\n EXIT;\n END IF;\n END LOOP;\n\n -- if the property was not in queryables, see if any args were numbers\n IF\n wrapper IS NULL\n AND jsonb_path_exists(args, '$[*] ? (@.type()==\"number\")')\n THEN\n wrapper := 'to_float';\n END IF;\n wrapper := coalesce(wrapper, 'to_text');\n END IF;\n\n SELECT jsonb_agg(cql2_query(a, wrapper))\n INTO args\n FROM jsonb_array_elements(args) a;\n END IF;\n RAISE NOTICE 'ARGS: %', args;\n\n IF op IN ('and', 'or') THEN\n RETURN\n format(\n '(%s)',\n array_to_string(to_text_array(args), format(' %s ', upper(op)))\n );\n END IF;\n\n IF op = 'in' THEN\n RAISE NOTICE 'IN -- % %', args->0, to_text(args->0);\n RETURN format(\n '%s IN (%s)',\n to_text(args->0),\n array_to_string((to_text_array(args))[2:], ',')\n );\n END IF;\n\n -- Look up template from cql2_ops\n IF j ? 'op' THEN\n SELECT * INTO cql2op FROM cql2_ops WHERE cql2_ops.op ilike op;\n IF FOUND THEN\n -- If specific index set in queryables for a property cast other arguments to that type\n\n RETURN format(\n cql2op.template,\n VARIADIC (to_text_array(args))\n );\n ELSE\n RAISE EXCEPTION 'Operator % Not Supported.', op;\n END IF;\n END IF;\n\n\n IF wrapper IS NOT NULL THEN\n RAISE NOTICE 'Wrapping % with %', j, wrapper;\n IF j ? 'property' THEN\n RETURN format('%I(%s)', wrapper, (queryable(j->>'property')).path);\n ELSE\n RETURN format('%I(%L)', wrapper, j);\n END IF;\n ELSIF j ? 'property' THEN\n RETURN quote_ident(j->>'property');\n END IF;\n\n RETURN quote_literal(to_text(j));\nEND;\n$function$\n;\n\n\n\nSELECT set_version('0.6.8');\n" }, { "path": "src/pgstac/migrations/pgstac.0.6.7.sql", "content": "CREATE EXTENSION IF NOT EXISTS postgis;\nCREATE EXTENSION IF NOT EXISTS btree_gist;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_admin;\n CREATE ROLE pgstac_read;\n CREATE ROLE pgstac_ingest;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nGRANT pgstac_admin TO current_user;\n\nCREATE SCHEMA IF NOT EXISTS pgstac AUTHORIZATION pgstac_admin;\n\nALTER ROLE pgstac_admin SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_read SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_ingest SET SEARCH_PATH TO pgstac, public;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\n\nGRANT pgstac_read TO pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\n\nSET ROLE pgstac_admin;\n\nSET SEARCH_PATH TO pgstac, public;\n\n\nCREATE TABLE IF NOT EXISTS migrations (\n version text PRIMARY KEY,\n datetime timestamptz DEFAULT clock_timestamp() NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION get_version() RETURNS text AS $$\n SELECT version FROM pgstac.migrations ORDER BY datetime DESC, version DESC LIMIT 1;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION set_version(text) RETURNS text AS $$\n INSERT INTO pgstac.migrations (version) VALUES ($1)\n ON CONFLICT DO NOTHING\n RETURNING version;\n$$ LANGUAGE SQL;\n\n\nCREATE TABLE IF NOT EXISTS pgstac_settings (\n name text PRIMARY KEY,\n value text NOT NULL\n);\n\nINSERT INTO pgstac_settings (name, value) VALUES\n ('context', 'off'),\n ('context_estimated_count', '100000'),\n ('context_estimated_cost', '100000'),\n ('context_stats_ttl', '1 day'),\n ('default-filter-lang', 'cql2-json'),\n ('additional_properties', 'true')\nON CONFLICT DO NOTHING\n;\n\n\nCREATE OR REPLACE FUNCTION get_setting(IN _setting text, IN conf jsonb DEFAULT NULL) RETURNS text AS $$\nSELECT COALESCE(\n conf->>_setting,\n current_setting(concat('pgstac.',_setting), TRUE),\n (SELECT value FROM pgstac.pgstac_settings WHERE name=_setting)\n);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context(conf jsonb DEFAULT NULL) RETURNS text AS $$\n SELECT pgstac.get_setting('context', conf);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context_estimated_count(conf jsonb DEFAULT NULL) RETURNS int AS $$\n SELECT pgstac.get_setting('context_estimated_count', conf)::int;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_estimated_cost();\nCREATE OR REPLACE FUNCTION context_estimated_cost(conf jsonb DEFAULT NULL) RETURNS float AS $$\n SELECT pgstac.get_setting('context_estimated_cost', conf)::float;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_stats_ttl();\nCREATE OR REPLACE FUNCTION context_stats_ttl(conf jsonb DEFAULT NULL) RETURNS interval AS $$\n SELECT pgstac.get_setting('context_stats_ttl', conf)::interval;\n$$ LANGUAGE SQL;\n\n\nCREATE OR REPLACE FUNCTION notice(VARIADIC text[]) RETURNS boolean AS $$\nDECLARE\ndebug boolean := current_setting('pgstac.debug', true);\nBEGIN\n IF debug THEN\n RAISE NOTICE 'NOTICE FROM FUNC: % >>>>> %', concat_ws(' | ', $1), clock_timestamp();\n RETURN TRUE;\n END IF;\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION empty_arr(ANYARRAY) RETURNS BOOLEAN AS $$\nSELECT CASE\n WHEN $1 IS NULL THEN TRUE\n WHEN cardinality($1)<1 THEN TRUE\nELSE FALSE\nEND;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_intersection(_a ANYARRAY, _b ANYARRAY) RETURNS ANYARRAY AS $$\n SELECT ARRAY ( SELECT unnest(_a) INTERSECT SELECT UNNEST(_b) );\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION array_map_ident(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_ident(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_map_literal(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_literal(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION array_reverse(anyarray) RETURNS anyarray AS $$\nSELECT ARRAY(\n SELECT $1[i]\n FROM generate_subscripts($1,1) AS s(i)\n ORDER BY i DESC\n);\n$$ LANGUAGE SQL STRICT IMMUTABLE;\nCREATE OR REPLACE FUNCTION to_int(jsonb) RETURNS int AS $$\n SELECT floor(($1->>0)::float)::int;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION to_float(jsonb) RETURNS float AS $$\n SELECT ($1->>0)::float;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION to_tstz(jsonb) RETURNS timestamptz AS $$\n SELECT ($1->>0)::timestamptz;\n$$ LANGUAGE SQL IMMUTABLE STRICT SET TIME ZONE 'UTC';\n\n\nCREATE OR REPLACE FUNCTION to_text(jsonb) RETURNS text AS $$\n SELECT CASE WHEN jsonb_typeof($1) IN ('array','object') THEN $1::text ELSE $1->>0 END;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION to_text_array(jsonb) RETURNS text[] AS $$\n SELECT\n CASE jsonb_typeof($1)\n WHEN 'array' THEN ARRAY(SELECT jsonb_array_elements_text($1))\n ELSE ARRAY[$1->>0]\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION bbox_geom(_bbox jsonb) RETURNS geometry AS $$\nSELECT CASE jsonb_array_length(_bbox)\n WHEN 4 THEN\n ST_SetSRID(ST_MakeEnvelope(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float,\n (_bbox->>3)::float\n ),4326)\n WHEN 6 THEN\n ST_SetSRID(ST_3DMakeBox(\n ST_MakePoint(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float\n ),\n ST_MakePoint(\n (_bbox->>3)::float,\n (_bbox->>4)::float,\n (_bbox->>5)::float\n )\n ),4326)\n ELSE null END;\n;\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION geom_bbox(_geom geometry) RETURNS jsonb AS $$\n SELECT jsonb_build_array(\n st_xmin(_geom),\n st_ymin(_geom),\n st_xmax(_geom),\n st_ymax(_geom)\n );\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION flip_jsonb_array(j jsonb) RETURNS jsonb AS $$\n SELECT jsonb_agg(value) FROM (SELECT value FROM jsonb_array_elements(j) WITH ORDINALITY ORDER BY ordinality DESC) as t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION explode_dotpaths(j jsonb) RETURNS SETOF text[] AS $$\n SELECT string_to_array(p, '.') as e FROM jsonb_array_elements_text(j) p;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION explode_dotpaths_recurse(IN j jsonb) RETURNS SETOF text[] AS $$\n WITH RECURSIVE t AS (\n SELECT e FROM explode_dotpaths(j) e\n UNION ALL\n SELECT e[1:cardinality(e)-1]\n FROM t\n WHERE cardinality(e)>1\n ) SELECT e FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION jsonb_set_nested(j jsonb, path text[], val jsonb) RETURNS jsonb AS $$\nDECLARE\nBEGIN\n IF cardinality(path) > 1 THEN\n FOR i IN 1..(cardinality(path)-1) LOOP\n IF j #> path[:i] IS NULL THEN\n j := jsonb_set_lax(j, path[:i], '{}', TRUE);\n END IF;\n END LOOP;\n END IF;\n RETURN jsonb_set_lax(j, path, val, true);\n\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\n\n\nCREATE OR REPLACE FUNCTION jsonb_include(j jsonb, f jsonb) RETURNS jsonb AS $$\nDECLARE\n includes jsonb := f-> 'include';\n outj jsonb := '{}'::jsonb;\n path text[];\nBEGIN\n IF\n includes IS NULL\n OR jsonb_array_length(includes) = 0\n THEN\n RETURN j;\n ELSE\n includes := includes || '[\"id\",\"collection\"]'::jsonb;\n FOR path IN SELECT explode_dotpaths(includes) LOOP\n outj := jsonb_set_nested(outj, path, j #> path);\n END LOOP;\n END IF;\n RETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_exclude(j jsonb, f jsonb) RETURNS jsonb AS $$\nDECLARE\n excludes jsonb := f-> 'exclude';\n outj jsonb := j;\n path text[];\nBEGIN\n IF\n excludes IS NULL\n OR jsonb_array_length(excludes) = 0\n THEN\n RETURN j;\n ELSE\n FOR path IN SELECT explode_dotpaths(excludes) LOOP\n outj := outj #- path;\n END LOOP;\n END IF;\n RETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_fields(j jsonb, f jsonb DEFAULT '{\"fields\":[]}') RETURNS jsonb AS $$\n SELECT jsonb_exclude(jsonb_include(j, f), f);\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION merge_jsonb(_a jsonb, _b jsonb) RETURNS jsonb AS $$\n SELECT\n CASE\n WHEN _a = '\"𒍟※\"'::jsonb THEN NULL\n WHEN _a IS NULL OR jsonb_typeof(_a) = 'null' THEN _b\n WHEN jsonb_typeof(_a) = 'object' AND jsonb_typeof(_b) = 'object' THEN\n (\n SELECT\n jsonb_strip_nulls(\n jsonb_object_agg(\n key,\n merge_jsonb(a.value, b.value)\n )\n )\n FROM\n jsonb_each(coalesce(_a,'{}'::jsonb)) as a\n FULL JOIN\n jsonb_each(coalesce(_b,'{}'::jsonb)) as b\n USING (key)\n )\n WHEN\n jsonb_typeof(_a) = 'array'\n AND jsonb_typeof(_b) = 'array'\n AND jsonb_array_length(_a) = jsonb_array_length(_b)\n THEN\n (\n SELECT jsonb_agg(m) FROM\n ( SELECT\n merge_jsonb(\n jsonb_array_elements(_a),\n jsonb_array_elements(_b)\n ) as m\n ) as l\n )\n ELSE _a\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION strip_jsonb(_a jsonb, _b jsonb) RETURNS jsonb AS $$\n SELECT\n CASE\n\n WHEN (_a IS NULL OR jsonb_typeof(_a) = 'null') AND _b IS NOT NULL AND jsonb_typeof(_b) != 'null' THEN '\"𒍟※\"'::jsonb\n WHEN _b IS NULL OR jsonb_typeof(_a) = 'null' THEN _a\n WHEN _a = _b AND jsonb_typeof(_a) = 'object' THEN '{}'::jsonb\n WHEN _a = _b THEN NULL\n WHEN jsonb_typeof(_a) = 'object' AND jsonb_typeof(_b) = 'object' THEN\n (\n SELECT\n jsonb_strip_nulls(\n jsonb_object_agg(\n key,\n strip_jsonb(a.value, b.value)\n )\n )\n FROM\n jsonb_each(_a) as a\n FULL JOIN\n jsonb_each(_b) as b\n USING (key)\n )\n WHEN\n jsonb_typeof(_a) = 'array'\n AND jsonb_typeof(_b) = 'array'\n AND jsonb_array_length(_a) = jsonb_array_length(_b)\n THEN\n (\n SELECT jsonb_agg(m) FROM\n ( SELECT\n strip_jsonb(\n jsonb_array_elements(_a),\n jsonb_array_elements(_b)\n ) as m\n ) as l\n )\n ELSE _a\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE;\n/* looks for a geometry in a stac item first from geometry and falling back to bbox */\nCREATE OR REPLACE FUNCTION stac_geom(value jsonb) RETURNS geometry AS $$\nSELECT\n CASE\n WHEN value ? 'intersects' THEN\n ST_GeomFromGeoJSON(value->>'intersects')\n WHEN value ? 'geometry' THEN\n ST_GeomFromGeoJSON(value->>'geometry')\n WHEN value ? 'bbox' THEN\n pgstac.bbox_geom(value->'bbox')\n ELSE NULL\n END as geometry\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION stac_daterange(\n value jsonb\n) RETURNS tstzrange AS $$\nDECLARE\n props jsonb := value;\n dt timestamptz;\n edt timestamptz;\nBEGIN\n IF props ? 'properties' THEN\n props := props->'properties';\n END IF;\n IF\n props ? 'start_datetime'\n AND props->>'start_datetime' IS NOT NULL\n AND props ? 'end_datetime'\n AND props->>'end_datetime' IS NOT NULL\n THEN\n dt := props->>'start_datetime';\n edt := props->>'end_datetime';\n IF dt > edt THEN\n RAISE EXCEPTION 'start_datetime must be < end_datetime';\n END IF;\n ELSE\n dt := props->>'datetime';\n edt := props->>'datetime';\n END IF;\n IF dt is NULL OR edt IS NULL THEN\n RAISE NOTICE 'DT: %, EDT: %', dt, edt;\n RAISE EXCEPTION 'Either datetime (%) or both start_datetime (%) and end_datetime (%) must be set.', props->>'datetime',props->>'start_datetime',props->>'end_datetime';\n END IF;\n RETURN tstzrange(dt, edt, '[]');\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_datetime(value jsonb) RETURNS timestamptz AS $$\n SELECT lower(stac_daterange(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_end_datetime(value jsonb) RETURNS timestamptz AS $$\n SELECT upper(stac_daterange(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\n\nCREATE TABLE IF NOT EXISTS stac_extensions(\n name text PRIMARY KEY,\n url text,\n enbabled_by_default boolean NOT NULL DEFAULT TRUE,\n enableable boolean NOT NULL DEFAULT TRUE\n);\n\nINSERT INTO stac_extensions (name, url) VALUES\n ('fields', 'https://api.stacspec.org/v1.0.0-beta.5/item-search#fields'),\n ('sort','https://api.stacspec.org/v1.0.0-beta.5/item-search#sort'),\n ('context','https://api.stacspec.org/v1.0.0-beta.5/item-search#context'),\n ('filter', 'https://api.stacspec.org/v1.0.0-beta.5/item-search#filter'),\n ('query', 'https://api.stacspec.org/v1.0.0-beta.5/item-search#query')\nON CONFLICT (name) DO UPDATE SET url=EXCLUDED.url;\n\n\n\nCREATE OR REPLACE FUNCTION collection_base_item(content jsonb) RETURNS jsonb AS $$\n SELECT jsonb_build_object(\n 'type', 'Feature',\n 'stac_version', content->'stac_version',\n 'assets', content->'item_assets',\n 'collection', content->'id'\n );\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TYPE partition_trunc_strategy AS ENUM ('year', 'month');\n\nCREATE TABLE IF NOT EXISTS collections (\n key bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,\n id text GENERATED ALWAYS AS (content->>'id') STORED UNIQUE,\n content JSONB NOT NULL,\n base_item jsonb GENERATED ALWAYS AS (pgstac.collection_base_item(content)) STORED,\n partition_trunc partition_trunc_strategy\n);\n\n\nCREATE OR REPLACE FUNCTION collection_base_item(cid text) RETURNS jsonb AS $$\n SELECT pgstac.collection_base_item(content) FROM pgstac.collections WHERE id = cid LIMIT 1;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION table_empty(text) RETURNS boolean AS $$\nDECLARE\n retval boolean;\nBEGIN\n EXECUTE format($q$\n SELECT NOT EXISTS (SELECT 1 FROM %I LIMIT 1)\n $q$,\n $1\n ) INTO retval;\n RETURN retval;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION collections_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n q text;\n partition_name text := format('_items_%s', NEW.key);\n partition_exists boolean := false;\n partition_empty boolean := true;\n err_context text;\n loadtemp boolean := FALSE;\nBEGIN\n RAISE NOTICE 'Collection Trigger. % %', NEW.id, NEW.key;\n SELECT relid::text INTO partition_name\n FROM pg_partition_tree('items')\n WHERE relid::text = partition_name;\n IF FOUND THEN\n partition_exists := true;\n partition_empty := table_empty(partition_name);\n ELSE\n partition_exists := false;\n partition_empty := true;\n partition_name := format('_items_%s', NEW.key);\n END IF;\n IF TG_OP = 'UPDATE' AND NEW.partition_trunc IS DISTINCT FROM OLD.partition_trunc AND partition_empty THEN\n q := format($q$\n DROP TABLE IF EXISTS %I CASCADE;\n $q$,\n partition_name\n );\n EXECUTE q;\n END IF;\n IF TG_OP = 'UPDATE' AND NEW.partition_trunc IS DISTINCT FROM OLD.partition_trunc AND partition_exists AND NOT partition_empty THEN\n q := format($q$\n CREATE TEMP TABLE changepartitionstaging ON COMMIT DROP AS SELECT * FROM %I;\n DROP TABLE IF EXISTS %I CASCADE;\n $q$,\n partition_name,\n partition_name\n );\n EXECUTE q;\n loadtemp := TRUE;\n partition_empty := TRUE;\n partition_exists := FALSE;\n END IF;\n IF TG_OP = 'UPDATE' AND NEW.partition_trunc IS NOT DISTINCT FROM OLD.partition_trunc THEN\n RETURN NEW;\n END IF;\n IF NEW.partition_trunc IS NULL AND partition_empty THEN\n RAISE NOTICE '% % % %',\n partition_name,\n NEW.id,\n concat(partition_name,'_id_idx'),\n partition_name\n ;\n q := format($q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n $q$,\n partition_name,\n NEW.id,\n concat(partition_name,'_id_idx'),\n partition_name\n );\n RAISE NOTICE 'q: %', q;\n BEGIN\n EXECUTE q;\n EXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', partition_name;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\n END;\n\n ALTER TABLE partitions DISABLE TRIGGER partitions_delete_trigger;\n DELETE FROM partitions WHERE collection=NEW.id AND name=partition_name;\n ALTER TABLE partitions ENABLE TRIGGER partitions_delete_trigger;\n\n INSERT INTO partitions (collection, name) VALUES (NEW.id, partition_name);\n ELSIF partition_empty THEN\n q := format($q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L)\n PARTITION BY RANGE (datetime);\n $q$,\n partition_name,\n NEW.id\n );\n RAISE NOTICE 'q: %', q;\n BEGIN\n EXECUTE q;\n EXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', partition_name;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\n END;\n ALTER TABLE partitions DISABLE TRIGGER partitions_delete_trigger;\n DELETE FROM partitions WHERE collection=NEW.id AND name=partition_name;\n ALTER TABLE partitions ENABLE TRIGGER partitions_delete_trigger;\n ELSE\n RAISE EXCEPTION 'Cannot modify partition % unless empty', partition_name;\n END IF;\n IF loadtemp THEN\n RAISE NOTICE 'Moving data into new partitions.';\n q := format($q$\n WITH p AS (\n SELECT\n collection,\n datetime as datetime,\n end_datetime as end_datetime,\n (partition_name(\n collection,\n datetime\n )).partition_name as name\n FROM changepartitionstaging\n )\n INSERT INTO partitions (collection, datetime_range, end_datetime_range)\n SELECT\n collection,\n tstzrange(min(datetime), max(datetime), '[]') as datetime_range,\n tstzrange(min(end_datetime), max(end_datetime), '[]') as end_datetime_range\n FROM p\n GROUP BY collection, name\n ON CONFLICT (name) DO UPDATE SET\n datetime_range = EXCLUDED.datetime_range,\n end_datetime_range = EXCLUDED.end_datetime_range\n ;\n INSERT INTO %I SELECT * FROM changepartitionstaging;\n DROP TABLE IF EXISTS changepartitionstaging;\n $q$,\n partition_name\n );\n EXECUTE q;\n END IF;\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE TRIGGER collections_trigger AFTER INSERT OR UPDATE ON collections FOR EACH ROW\nEXECUTE FUNCTION collections_trigger_func();\n\nCREATE OR REPLACE FUNCTION partition_collection(collection text, strategy partition_trunc_strategy) RETURNS text AS $$\n UPDATE collections SET partition_trunc=strategy WHERE id=collection RETURNING partition_trunc;\n$$ LANGUAGE SQL;\n\nCREATE TABLE IF NOT EXISTS partitions (\n collection text REFERENCES collections(id) ON DELETE CASCADE,\n name text PRIMARY KEY,\n partition_range tstzrange NOT NULL DEFAULT tstzrange('-infinity'::timestamptz,'infinity'::timestamptz, '[]'),\n datetime_range tstzrange,\n end_datetime_range tstzrange,\n CONSTRAINT prange EXCLUDE USING GIST (\n collection WITH =,\n partition_range WITH &&\n )\n) WITH (FILLFACTOR=90);\nCREATE INDEX partitions_range_idx ON partitions USING GIST(partition_range);\n\nCREATE OR REPLACE FUNCTION partitions_delete_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n q text;\nBEGIN\n RAISE NOTICE 'Partition Delete Trigger. %', OLD.name;\n EXECUTE format($q$\n DROP TABLE IF EXISTS %I CASCADE;\n $q$,\n OLD.name\n );\n RAISE NOTICE 'Dropped partition.';\n RETURN OLD;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER partitions_delete_trigger BEFORE DELETE ON partitions FOR EACH ROW\nEXECUTE FUNCTION partitions_delete_trigger_func();\n\nCREATE OR REPLACE FUNCTION partition_name(\n IN collection text,\n IN dt timestamptz,\n OUT partition_name text,\n OUT partition_range tstzrange\n) AS $$\nDECLARE\n c RECORD;\n parent_name text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n parent_name := format('_items_%s', c.key);\n\n\n IF c.partition_trunc = 'year' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYY'));\n ELSIF c.partition_trunc = 'month' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYYMM'));\n ELSE\n partition_name := parent_name;\n partition_range := tstzrange('-infinity'::timestamptz, 'infinity'::timestamptz, '[]');\n END IF;\n IF partition_range IS NULL THEN\n partition_range := tstzrange(\n date_trunc(c.partition_trunc::text, dt),\n date_trunc(c.partition_trunc::text, dt) + concat('1 ', c.partition_trunc)::interval\n );\n END IF;\n RETURN;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\n\nCREATE OR REPLACE FUNCTION partitions_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n q text;\n cq text;\n parent_name text;\n partition_trunc text;\n partition_name text := NEW.name;\n partition_exists boolean := false;\n partition_empty boolean := true;\n partition_range tstzrange;\n datetime_range tstzrange;\n end_datetime_range tstzrange;\n err_context text;\n mindt timestamptz := lower(NEW.datetime_range);\n maxdt timestamptz := upper(NEW.datetime_range);\n minedt timestamptz := lower(NEW.end_datetime_range);\n maxedt timestamptz := upper(NEW.end_datetime_range);\n t_mindt timestamptz;\n t_maxdt timestamptz;\n t_minedt timestamptz;\n t_maxedt timestamptz;\nBEGIN\n RAISE NOTICE 'Partitions Trigger. %', NEW;\n datetime_range := NEW.datetime_range;\n end_datetime_range := NEW.end_datetime_range;\n\n SELECT\n format('_items_%s', key),\n c.partition_trunc::text\n INTO\n parent_name,\n partition_trunc\n FROM pgstac.collections c\n WHERE c.id = NEW.collection;\n SELECT (pgstac.partition_name(NEW.collection, mindt)).* INTO partition_name, partition_range;\n NEW.name := partition_name;\n\n IF partition_range IS NULL OR partition_range = 'empty'::tstzrange THEN\n partition_range := tstzrange('-infinity'::timestamptz, 'infinity'::timestamptz, '[]');\n END IF;\n\n NEW.partition_range := partition_range;\n IF TG_OP = 'UPDATE' THEN\n mindt := least(mindt, lower(OLD.datetime_range));\n maxdt := greatest(maxdt, upper(OLD.datetime_range));\n minedt := least(minedt, lower(OLD.end_datetime_range));\n maxedt := greatest(maxedt, upper(OLD.end_datetime_range));\n NEW.datetime_range := tstzrange(mindt, maxdt, '[]');\n NEW.end_datetime_range := tstzrange(minedt, maxedt, '[]');\n END IF;\n IF TG_OP = 'INSERT' THEN\n\n IF partition_range != tstzrange('-infinity'::timestamptz, 'infinity'::timestamptz, '[]') THEN\n\n RAISE NOTICE '% % %', partition_name, parent_name, partition_range;\n q := format($q$\n CREATE TABLE IF NOT EXISTS %I partition OF %I FOR VALUES FROM (%L) TO (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n $q$,\n partition_name,\n parent_name,\n lower(partition_range),\n upper(partition_range),\n format('%s_pkey', partition_name),\n partition_name\n );\n BEGIN\n EXECUTE q;\n EXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', partition_name;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\n END;\n END IF;\n\n END IF;\n\n -- Update constraints\n EXECUTE format($q$\n SELECT\n min(datetime),\n max(datetime),\n min(end_datetime),\n max(end_datetime)\n FROM %I;\n $q$, partition_name)\n INTO t_mindt, t_maxdt, t_minedt, t_maxedt;\n mindt := least(mindt, t_mindt);\n maxdt := greatest(maxdt, t_maxdt);\n minedt := least(mindt, minedt, t_minedt);\n maxedt := greatest(maxdt, maxedt, t_maxedt);\n\n mindt := date_trunc(coalesce(partition_trunc, 'year'), mindt);\n maxdt := date_trunc(coalesce(partition_trunc, 'year'), maxdt - '1 second'::interval) + concat('1 ',coalesce(partition_trunc, 'year'))::interval;\n minedt := date_trunc(coalesce(partition_trunc, 'year'), minedt);\n maxedt := date_trunc(coalesce(partition_trunc, 'year'), maxedt - '1 second'::interval) + concat('1 ',coalesce(partition_trunc, 'year'))::interval;\n\n\n IF mindt IS NOT NULL AND maxdt IS NOT NULL AND minedt IS NOT NULL AND maxedt IS NOT NULL THEN\n NEW.datetime_range := tstzrange(mindt, maxdt, '[]');\n NEW.end_datetime_range := tstzrange(minedt, maxedt, '[]');\n IF\n TG_OP='UPDATE'\n AND OLD.datetime_range @> NEW.datetime_range\n AND OLD.end_datetime_range @> NEW.end_datetime_range\n THEN\n RAISE NOTICE 'Range unchanged, not updating constraints.';\n ELSE\n\n RAISE NOTICE '\n SETTING CONSTRAINTS\n mindt: %, maxdt: %\n minedt: %, maxedt: %\n ', mindt, maxdt, minedt, maxedt;\n IF partition_trunc IS NULL THEN\n cq := format($q$\n ALTER TABLE %7$I\n DROP CONSTRAINT IF EXISTS %1$I,\n DROP CONSTRAINT IF EXISTS %2$I,\n ADD CONSTRAINT %1$I\n CHECK (\n (datetime >= %3$L)\n AND (datetime <= %4$L)\n AND (end_datetime >= %5$L)\n AND (end_datetime <= %6$L)\n ) NOT VALID\n ;\n ALTER TABLE %7$I\n VALIDATE CONSTRAINT %1$I;\n $q$,\n format('%s_dt', partition_name),\n format('%s_edt', partition_name),\n mindt,\n maxdt,\n minedt,\n maxedt,\n partition_name\n );\n ELSE\n cq := format($q$\n ALTER TABLE %5$I\n DROP CONSTRAINT IF EXISTS %1$I,\n DROP CONSTRAINT IF EXISTS %2$I,\n ADD CONSTRAINT %2$I\n CHECK ((end_datetime >= %3$L) AND (end_datetime <= %4$L)) NOT VALID\n ;\n ALTER TABLE %5$I\n VALIDATE CONSTRAINT %2$I;\n $q$,\n format('%s_dt', partition_name),\n format('%s_edt', partition_name),\n minedt,\n maxedt,\n partition_name\n );\n\n END IF;\n RAISE NOTICE 'Altering Constraints. %', cq;\n EXECUTE cq;\n END IF;\n ELSE\n NEW.datetime_range = NULL;\n NEW.end_datetime_range = NULL;\n\n cq := format($q$\n ALTER TABLE %3$I\n DROP CONSTRAINT IF EXISTS %1$I,\n DROP CONSTRAINT IF EXISTS %2$I,\n ADD CONSTRAINT %1$I\n CHECK ((datetime IS NULL AND end_datetime IS NULL)) NOT VALID\n ;\n ALTER TABLE %3$I\n VALIDATE CONSTRAINT %1$I;\n $q$,\n format('%s_dt', partition_name),\n format('%s_edt', partition_name),\n partition_name\n );\n EXECUTE cq;\n END IF;\n\n RETURN NEW;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER partitions_trigger BEFORE INSERT OR UPDATE ON partitions FOR EACH ROW\nEXECUTE FUNCTION partitions_trigger_func();\n\n\nCREATE OR REPLACE FUNCTION create_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection(data jsonb) RETURNS VOID AS $$\nDECLARE\n out collections%ROWTYPE;\nBEGIN\n UPDATE collections SET content=data WHERE id = data->>'id' RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ON CONFLICT (id) DO\n UPDATE\n SET content=EXCLUDED.content\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION delete_collection(_id text) RETURNS VOID AS $$\nDECLARE\n out collections%ROWTYPE;\nBEGIN\n DELETE FROM collections WHERE id = _id RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION get_collection(id text) RETURNS jsonb AS $$\n SELECT content FROM collections\n WHERE id=$1\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION all_collections() RETURNS jsonb AS $$\n SELECT jsonb_agg(content) FROM collections;\n;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\nCREATE TABLE queryables (\n id bigint GENERATED ALWAYS AS identity PRIMARY KEY,\n name text UNIQUE NOT NULL,\n collection_ids text[], -- used to determine what partitions to create indexes on\n definition jsonb,\n property_path text,\n property_wrapper text,\n property_index_type text\n);\nCREATE INDEX queryables_name_idx ON queryables (name);\nCREATE INDEX queryables_property_wrapper_idx ON queryables (property_wrapper);\n\n\nINSERT INTO queryables (name, definition) VALUES\n('id', '{\"title\": \"Item ID\",\"description\": \"Item identifier\",\"$ref\": \"https://schemas.stacspec.org/v1.0.0/item-spec/json-schema/item.json#/definitions/core/allOf/2/properties/id\"}'),\n('datetime','{\"description\": \"Datetime\",\"type\": \"string\",\"title\": \"Acquired\",\"format\": \"date-time\",\"pattern\": \"(\\\\+00:00|Z)$\"}')\nON CONFLICT DO NOTHING;\n\n\n\nINSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n('eo:cloud_cover','{\"$ref\": \"https://stac-extensions.github.io/eo/v1.0.0/schema.json#/definitions/fieldsproperties/eo:cloud_cover\"}','to_int','BTREE')\nON CONFLICT DO NOTHING;\n\nCREATE OR REPLACE FUNCTION array_to_path(arr text[]) RETURNS text AS $$\n SELECT string_agg(\n quote_literal(v),\n '->'\n ) FROM unnest(arr) v;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\n\n\nCREATE OR REPLACE FUNCTION queryable(\n IN dotpath text,\n OUT path text,\n OUT expression text,\n OUT wrapper text\n) AS $$\nDECLARE\n q RECORD;\n path_elements text[];\nBEGIN\n IF dotpath IN ('id', 'geometry', 'datetime', 'end_datetime', 'collection') THEN\n path := dotpath;\n expression := dotpath;\n wrapper := NULL;\n RETURN;\n END IF;\n SELECT * INTO q FROM queryables WHERE name=dotpath;\n IF q.property_wrapper IS NULL THEN\n IF q.definition->>'type' = 'number' THEN\n wrapper := 'to_float';\n ELSIF q.definition->>'format' = 'date-time' THEN\n wrapper := 'to_tstz';\n ELSE\n wrapper := 'to_text';\n END IF;\n ELSE\n wrapper := q.property_wrapper;\n END IF;\n IF q.property_path IS NOT NULL THEN\n path := q.property_path;\n ELSE\n path_elements := string_to_array(dotpath, '.');\n IF path_elements[1] IN ('links', 'assets', 'stac_version', 'stac_extensions') THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSIF path_elements[1] = 'properties' THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSE\n path := format($F$content->'properties'->%s$F$, array_to_path(path_elements));\n END IF;\n END IF;\n expression := format('%I(%s)', wrapper, path);\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\nCREATE OR REPLACE FUNCTION create_queryable_indexes() RETURNS VOID AS $$\nDECLARE\n queryable RECORD;\n q text;\nBEGIN\n FOR queryable IN\n SELECT\n queryables.id as qid,\n CASE WHEN collections.key IS NULL THEN 'items' ELSE format('_items_%s',collections.key) END AS part,\n property_index_type,\n expression\n FROM\n queryables\n LEFT JOIN collections ON (collections.id = ANY (queryables.collection_ids))\n JOIN LATERAL queryable(queryables.name) ON (queryables.property_index_type IS NOT NULL)\n LOOP\n q := format(\n $q$\n CREATE INDEX IF NOT EXISTS %I ON %I USING %s ((%s));\n $q$,\n format('%s_%s_idx', queryable.part, queryable.qid),\n queryable.part,\n COALESCE(queryable.property_index_type, 'to_text'),\n queryable.expression\n );\n RAISE NOTICE '%',q;\n EXECUTE q;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION queryables_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\nBEGIN\nPERFORM create_queryable_indexes();\nRETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER queryables_trigger AFTER INSERT OR UPDATE ON queryables\nFOR EACH STATEMENT EXECUTE PROCEDURE queryables_trigger_func();\n\nCREATE TRIGGER queryables_collection_trigger AFTER INSERT OR UPDATE ON collections\nFOR EACH STATEMENT EXECUTE PROCEDURE queryables_trigger_func();\nCREATE OR REPLACE FUNCTION parse_dtrange(\n _indate jsonb,\n relative_base timestamptz DEFAULT date_trunc('hour', CURRENT_TIMESTAMP)\n) RETURNS tstzrange AS $$\nDECLARE\n timestrs text[];\n s timestamptz;\n e timestamptz;\nBEGIN\n timestrs :=\n CASE\n WHEN _indate ? 'timestamp' THEN\n ARRAY[_indate->>'timestamp']\n WHEN _indate ? 'interval' THEN\n to_text_array(_indate->'interval')\n WHEN jsonb_typeof(_indate) = 'array' THEN\n to_text_array(_indate)\n ELSE\n regexp_split_to_array(\n _indate->>0,\n '/'\n )\n END;\n RAISE NOTICE 'TIMESTRS %', timestrs;\n IF cardinality(timestrs) = 1 THEN\n IF timestrs[1] ILIKE 'P%' THEN\n RETURN tstzrange(relative_base - upper(timestrs[1])::interval, relative_base, '[)');\n END IF;\n s := timestrs[1]::timestamptz;\n RETURN tstzrange(s, s, '[]');\n END IF;\n\n IF cardinality(timestrs) != 2 THEN\n RAISE EXCEPTION 'Timestamp cannot have more than 2 values';\n END IF;\n\n IF timestrs[1] = '..' THEN\n s := '-infinity'::timestamptz;\n e := timestrs[2]::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] = '..' THEN\n s := timestrs[1]::timestamptz;\n e := 'infinity'::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[1] ILIKE 'P%' AND timestrs[2] NOT ILIKE 'P%' THEN\n e := timestrs[2]::timestamptz;\n s := e - upper(timestrs[1])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] ILIKE 'P%' AND timestrs[1] NOT ILIKE 'P%' THEN\n s := timestrs[1]::timestamptz;\n e := s + upper(timestrs[2])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n s := timestrs[1]::timestamptz;\n e := timestrs[2]::timestamptz;\n\n RETURN tstzrange(s,e,'[)');\n\n RETURN NULL;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT PARALLEL SAFE SET TIME ZONE 'UTC';\n\nCREATE OR REPLACE FUNCTION parse_dtrange(\n _indate text,\n relative_base timestamptz DEFAULT CURRENT_TIMESTAMP\n) RETURNS tstzrange AS $$\n SELECT parse_dtrange(to_jsonb(_indate), relative_base);\n$$ LANGUAGE SQL STABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION temporal_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\n ll text := 'datetime';\n lh text := 'end_datetime';\n rrange tstzrange;\n rl text;\n rh text;\n outq text;\nBEGIN\n rrange := parse_dtrange(args->1);\n RAISE NOTICE 'Constructing temporal query OP: %, ARGS: %, RRANGE: %', op, args, rrange;\n op := lower(op);\n rl := format('%L::timestamptz', lower(rrange));\n rh := format('%L::timestamptz', upper(rrange));\n outq := CASE op\n WHEN 't_before' THEN 'lh < rl'\n WHEN 't_after' THEN 'll > rh'\n WHEN 't_meets' THEN 'lh = rl'\n WHEN 't_metby' THEN 'll = rh'\n WHEN 't_overlaps' THEN 'll < rl AND rl < lh < rh'\n WHEN 't_overlappedby' THEN 'rl < ll < rh AND lh > rh'\n WHEN 't_starts' THEN 'll = rl AND lh < rh'\n WHEN 't_startedby' THEN 'll = rl AND lh > rh'\n WHEN 't_during' THEN 'll > rl AND lh < rh'\n WHEN 't_contains' THEN 'll < rl AND lh > rh'\n WHEN 't_finishes' THEN 'll > rl AND lh = rh'\n WHEN 't_finishedby' THEN 'll < rl AND lh = rh'\n WHEN 't_equals' THEN 'll = rl AND lh = rh'\n WHEN 't_disjoint' THEN 'NOT (ll <= rh AND lh >= rl)'\n WHEN 't_intersects' THEN 'll <= rh AND lh >= rl'\n WHEN 'anyinteracts' THEN 'll <= rh AND lh >= rl'\n END;\n outq := regexp_replace(outq, '\\mll\\M', ll);\n outq := regexp_replace(outq, '\\mlh\\M', lh);\n outq := regexp_replace(outq, '\\mrl\\M', rl);\n outq := regexp_replace(outq, '\\mrh\\M', rh);\n outq := format('(%s)', outq);\n RETURN outq;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\n\n\nCREATE OR REPLACE FUNCTION spatial_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\n geom text;\n j jsonb := args->1;\nBEGIN\n op := lower(op);\n RAISE NOTICE 'Constructing spatial query OP: %, ARGS: %', op, args;\n IF op NOT IN ('s_equals','s_disjoint','s_touches','s_within','s_overlaps','s_crosses','s_intersects','intersects','s_contains') THEN\n RAISE EXCEPTION 'Spatial Operator % Not Supported', op;\n END IF;\n op := regexp_replace(op, '^s_', 'st_');\n IF op = 'intersects' THEN\n op := 'st_intersects';\n END IF;\n -- Convert geometry to WKB string\n IF j ? 'type' AND j ? 'coordinates' THEN\n geom := st_geomfromgeojson(j)::text;\n ELSIF jsonb_typeof(j) = 'array' THEN\n geom := bbox_geom(j)::text;\n END IF;\n\n RETURN format('%s(geometry, %L::geometry)', op, geom);\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION query_to_cql2(q jsonb) RETURNS jsonb AS $$\n-- Translates anything passed in through the deprecated \"query\" into equivalent CQL2\nWITH t AS (\n SELECT key as property, value as ops\n FROM jsonb_each(q)\n), t2 AS (\n SELECT property, (jsonb_each(ops)).*\n FROM t WHERE jsonb_typeof(ops) = 'object'\n UNION ALL\n SELECT property, 'eq', ops\n FROM t WHERE jsonb_typeof(ops) != 'object'\n)\nSELECT\n jsonb_strip_nulls(jsonb_build_object(\n 'op', 'and',\n 'args', jsonb_agg(\n jsonb_build_object(\n 'op', key,\n 'args', jsonb_build_array(\n jsonb_build_object('property',property),\n value\n )\n )\n )\n )\n) as qcql FROM t2\n;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\nCREATE OR REPLACE FUNCTION cql1_to_cql2(j jsonb) RETURNS jsonb AS $$\nDECLARE\n args jsonb;\n ret jsonb;\nBEGIN\n RAISE NOTICE 'CQL1_TO_CQL2: %', j;\n IF j ? 'filter' THEN\n RETURN cql1_to_cql2(j->'filter');\n END IF;\n IF j ? 'property' THEN\n RETURN j;\n END IF;\n IF jsonb_typeof(j) = 'array' THEN\n SELECT jsonb_agg(cql1_to_cql2(el)) INTO args FROM jsonb_array_elements(j) el;\n RETURN args;\n END IF;\n IF jsonb_typeof(j) = 'number' THEN\n RETURN j;\n END IF;\n IF jsonb_typeof(j) = 'string' THEN\n RETURN j;\n END IF;\n\n IF jsonb_typeof(j) = 'object' THEN\n SELECT jsonb_build_object(\n 'op', key,\n 'args', cql1_to_cql2(value)\n ) INTO ret\n FROM jsonb_each(j)\n WHERE j IS NOT NULL;\n RETURN ret;\n END IF;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE STRICT;\n\nCREATE TABLE cql2_ops (\n op text PRIMARY KEY,\n template text,\n types text[]\n);\nINSERT INTO cql2_ops (op, template, types) VALUES\n ('eq', '%s = %s', NULL),\n ('neq', '%s != %s', NULL),\n ('ne', '%s != %s', NULL),\n ('!=', '%s != %s', NULL),\n ('<>', '%s != %s', NULL),\n ('lt', '%s < %s', NULL),\n ('lte', '%s <= %s', NULL),\n ('gt', '%s > %s', NULL),\n ('gte', '%s >= %s', NULL),\n ('le', '%s <= %s', NULL),\n ('ge', '%s >= %s', NULL),\n ('=', '%s = %s', NULL),\n ('<', '%s < %s', NULL),\n ('<=', '%s <= %s', NULL),\n ('>', '%s > %s', NULL),\n ('>=', '%s >= %s', NULL),\n ('like', '%s LIKE %s', NULL),\n ('ilike', '%s ILIKE %s', NULL),\n ('+', '%s + %s', NULL),\n ('-', '%s - %s', NULL),\n ('*', '%s * %s', NULL),\n ('/', '%s / %s', NULL),\n ('not', 'NOT (%s)', NULL),\n ('between', '%s BETWEEN %s AND %s', NULL),\n ('isnull', '%s IS NULL', NULL),\n ('upper', 'upper(%s)', NULL),\n ('lower', 'lower(%s)', NULL)\nON CONFLICT (op) DO UPDATE\n SET\n template = EXCLUDED.template\n;\n\n\nCREATE OR REPLACE FUNCTION cql2_query(j jsonb, wrapper text DEFAULT NULL) RETURNS text AS $$\n#variable_conflict use_variable\nDECLARE\n args jsonb := j->'args';\n arg jsonb;\n op text := lower(j->>'op');\n cql2op RECORD;\n literal text;\n _wrapper text;\n leftarg text;\n rightarg text;\nBEGIN\n IF j IS NULL OR (op IS NOT NULL AND args IS NULL) THEN\n RETURN NULL;\n END IF;\n RAISE NOTICE 'CQL2_QUERY: %', j;\n IF j ? 'filter' THEN\n RETURN cql2_query(j->'filter');\n END IF;\n\n IF j ? 'upper' THEN\n RETURN cql2_query(jsonb_build_object('op', 'upper', 'args', j->'upper'));\n END IF;\n\n IF j ? 'lower' THEN\n RETURN cql2_query(jsonb_build_object('op', 'lower', 'args', j->'lower'));\n END IF;\n\n -- Temporal Query\n IF op ilike 't_%' or op = 'anyinteracts' THEN\n RETURN temporal_op_query(op, args);\n END IF;\n\n -- If property is a timestamp convert it to text to use with\n -- general operators\n IF j ? 'timestamp' THEN\n RETURN format('%L::timestamptz', to_tstz(j->'timestamp'));\n END IF;\n IF j ? 'interval' THEN\n RAISE EXCEPTION 'Please use temporal operators when using intervals.';\n RETURN NONE;\n END IF;\n\n -- Spatial Query\n IF op ilike 's_%' or op = 'intersects' THEN\n RETURN spatial_op_query(op, args);\n END IF;\n\n IF op IN ('a_equals','a_contains','a_contained_by','a_overlaps') THEN\n IF args->0 ? 'property' THEN\n leftarg := format('to_text_array(%s)', (queryable(args->0->>'property')).path);\n END IF;\n IF args->1 ? 'property' THEN\n rightarg := format('to_text_array(%s)', (queryable(args->1->>'property')).path);\n END IF;\n RETURN FORMAT(\n '%s %s %s',\n COALESCE(leftarg, quote_literal(to_text_array(args->0))),\n CASE op\n WHEN 'a_equals' THEN '='\n WHEN 'a_contains' THEN '@>'\n WHEN 'a_contained_by' THEN '<@'\n WHEN 'a_overlaps' THEN '&&'\n END,\n COALESCE(rightarg, quote_literal(to_text_array(args->1)))\n );\n END IF;\n\n IF op = 'in' THEN\n RAISE NOTICE 'IN : % % %', args, jsonb_build_array(args->0), args->1;\n args := jsonb_build_array(args->0) || (args->1);\n RAISE NOTICE 'IN2 : %', args;\n END IF;\n\n\n\n IF op = 'between' THEN\n args = jsonb_build_array(\n args->0,\n args->1->0,\n args->1->1\n );\n END IF;\n\n -- Make sure that args is an array and run cql2_query on\n -- each element of the array\n RAISE NOTICE 'ARGS PRE: %', args;\n IF j ? 'args' THEN\n IF jsonb_typeof(args) != 'array' THEN\n args := jsonb_build_array(args);\n END IF;\n\n IF jsonb_path_exists(args, '$[*] ? (@.property == \"id\" || @.property == \"datetime\" || @.property == \"end_datetime\" || @.property == \"collection\")') THEN\n wrapper := NULL;\n ELSE\n -- if any of the arguments are a property, try to get the property_wrapper\n FOR arg IN SELECT jsonb_path_query(args, '$[*] ? (@.property != null)') LOOP\n RAISE NOTICE 'Arg: %', arg;\n SELECT property_wrapper INTO wrapper\n FROM queryables\n WHERE name=(arg->>'property')\n LIMIT 1;\n RAISE NOTICE 'Property: %, Wrapper: %', arg, wrapper;\n IF wrapper IS NOT NULL THEN\n EXIT;\n END IF;\n END LOOP;\n\n -- if the property was not in queryables, see if any args were numbers\n IF\n wrapper IS NULL\n AND jsonb_path_exists(args, '$[*] ? (@.type()==\"number\")')\n THEN\n wrapper := 'to_float';\n END IF;\n wrapper := coalesce(wrapper, 'to_text');\n END IF;\n\n SELECT jsonb_agg(cql2_query(a, wrapper))\n INTO args\n FROM jsonb_array_elements(args) a;\n END IF;\n RAISE NOTICE 'ARGS: %', args;\n\n IF op IN ('and', 'or') THEN\n RETURN\n format(\n '(%s)',\n array_to_string(to_text_array(args), format(' %s ', upper(op)))\n );\n END IF;\n\n IF op = 'in' THEN\n RAISE NOTICE 'IN -- % %', args->0, to_text(args->0);\n RETURN format(\n '%s IN (%s)',\n to_text(args->0),\n array_to_string((to_text_array(args))[2:], ',')\n );\n END IF;\n\n -- Look up template from cql2_ops\n IF j ? 'op' THEN\n SELECT * INTO cql2op FROM cql2_ops WHERE cql2_ops.op ilike op;\n IF FOUND THEN\n -- If specific index set in queryables for a property cast other arguments to that type\n\n RETURN format(\n cql2op.template,\n VARIADIC (to_text_array(args))\n );\n ELSE\n RAISE EXCEPTION 'Operator % Not Supported.', op;\n END IF;\n END IF;\n\n\n IF wrapper IS NOT NULL THEN\n RAISE NOTICE 'Wrapping % with %', j, wrapper;\n IF j ? 'property' THEN\n RETURN format('%I(%s)', wrapper, (queryable(j->>'property')).path);\n ELSE\n RETURN format('%I(%L)', wrapper, j);\n END IF;\n ELSIF j ? 'property' THEN\n RETURN quote_ident(j->>'property');\n END IF;\n\n RETURN quote_literal(to_text(j));\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION paging_dtrange(\n j jsonb\n) RETURNS tstzrange AS $$\nDECLARE\n op text;\n filter jsonb := j->'filter';\n dtrange tstzrange := tstzrange('-infinity'::timestamptz,'infinity'::timestamptz);\n sdate timestamptz := '-infinity'::timestamptz;\n edate timestamptz := 'infinity'::timestamptz;\n jpitem jsonb;\nBEGIN\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n END IF;\n IF NOT (filter @? '$.**.op ? (@ == \"or\" || @ == \"not\")') THEN\n FOR jpitem IN SELECT j FROM jsonb_path_query(filter,'strict $.** ? (@.args[*].property == \"datetime\")'::jsonpath) j LOOP\n op := lower(jpitem->>'op');\n dtrange := parse_dtrange(jpitem->'args'->1);\n IF op IN ('<=', 'lt', 'lte', '<', 'le', 't_before') THEN\n sdate := greatest(sdate,'-infinity');\n edate := least(edate, upper(dtrange));\n ELSIF op IN ('>=', '>', 'gt', 'gte', 'ge', 't_after') THEN\n edate := least(edate, 'infinity');\n sdate := greatest(sdate, lower(dtrange));\n ELSIF op IN ('=', 'eq') THEN\n edate := least(edate, upper(dtrange));\n sdate := greatest(sdate, lower(dtrange));\n END IF;\n RAISE NOTICE '2 OP: %, ARGS: %, DTRANGE: %, SDATE: %, EDATE: %', op, jpitem->'args'->1, dtrange, sdate, edate;\n END LOOP;\n END IF;\n IF sdate > edate THEN\n RETURN 'empty'::tstzrange;\n END IF;\n RETURN tstzrange(sdate,edate, '[]');\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT SET TIME ZONE 'UTC';\n\nCREATE OR REPLACE FUNCTION paging_collections(\n IN j jsonb\n) RETURNS text[] AS $$\nDECLARE\n filter jsonb := j->'filter';\n jpitem jsonb;\n op text;\n args jsonb;\n arg jsonb;\n collections text[];\nBEGIN\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n END IF;\n IF NOT (filter @? '$.**.op ? (@ == \"or\" || @ == \"not\")') THEN\n FOR jpitem IN SELECT j FROM jsonb_path_query(filter,'strict $.** ? (@.args[*].property == \"collection\")'::jsonpath) j LOOP\n RAISE NOTICE 'JPITEM: %', jpitem;\n op := jpitem->>'op';\n args := jpitem->'args';\n IF op IN ('=', 'eq', 'in') THEN\n FOR arg IN SELECT a FROM jsonb_array_elements(args) a LOOP\n IF jsonb_typeof(arg) IN ('string', 'array') THEN\n RAISE NOTICE 'arg: %, collections: %', arg, collections;\n IF collections IS NULL OR collections = '{}'::text[] THEN\n collections := to_text_array(arg);\n ELSE\n collections := array_intersection(collections, to_text_array(arg));\n END IF;\n END IF;\n END LOOP;\n END IF;\n END LOOP;\n END IF;\n IF collections = '{}'::text[] THEN\n RETURN NULL;\n END IF;\n RETURN collections;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\nCREATE TABLE items (\n id text NOT NULL,\n geometry geometry NOT NULL,\n collection text NOT NULL,\n datetime timestamptz NOT NULL,\n end_datetime timestamptz NOT NULL,\n content JSONB NOT NULL\n)\nPARTITION BY LIST (collection)\n;\n\nCREATE INDEX \"datetime_idx\" ON items USING BTREE (datetime DESC, end_datetime ASC);\nCREATE INDEX \"geometry_idx\" ON items USING GIST (geometry);\n\nCREATE STATISTICS datetime_stats (dependencies) on datetime, end_datetime from items;\n\n\nALTER TABLE items ADD CONSTRAINT items_collections_fk FOREIGN KEY (collection) REFERENCES collections(id) ON DELETE CASCADE DEFERRABLE;\n\n\nCREATE OR REPLACE FUNCTION content_slim(_item jsonb) RETURNS jsonb AS $$\n SELECT strip_jsonb(_item - '{id,geometry,collection,type}'::text[], collection_base_item(_item->>'collection')) - '{id,geometry,collection,type}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_dehydrate(content jsonb) RETURNS items AS $$\n SELECT\n content->>'id' as id,\n stac_geom(content) as geometry,\n content->>'collection' as collection,\n stac_datetime(content) as datetime,\n stac_end_datetime(content) as end_datetime,\n content_slim(content) as content\n ;\n$$ LANGUAGE SQL STABLE;\n\nCREATE OR REPLACE FUNCTION include_field(f text, fields jsonb DEFAULT '{}'::jsonb) RETURNS boolean AS $$\nDECLARE\n includes jsonb := fields->'include';\n excludes jsonb := fields->'exclude';\nBEGIN\n IF f IS NULL THEN\n RETURN NULL;\n END IF;\n\n\n IF\n jsonb_typeof(excludes) = 'array'\n AND jsonb_array_length(excludes)>0\n AND excludes ? f\n THEN\n RETURN FALSE;\n END IF;\n\n IF\n (\n jsonb_typeof(includes) = 'array'\n AND jsonb_array_length(includes) > 0\n AND includes ? f\n ) OR\n (\n includes IS NULL\n OR jsonb_typeof(includes) = 'null'\n OR jsonb_array_length(includes) = 0\n )\n THEN\n RETURN TRUE;\n END IF;\n\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nDROP FUNCTION IF EXISTS content_hydrate(jsonb, jsonb, jsonb);\nCREATE OR REPLACE FUNCTION content_hydrate(\n _item jsonb,\n _base_item jsonb,\n fields jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\n SELECT merge_jsonb(\n jsonb_fields(_item, fields),\n jsonb_fields(_base_item, fields)\n );\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION content_hydrate(_item items, _collection collections, fields jsonb DEFAULT '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\n content jsonb;\n base_item jsonb := _collection.base_item;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry, 20)::jsonb;\n END IF;\n output := content_hydrate(\n jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'collection', _item.collection,\n 'type', 'Feature'\n ) || _item.content,\n _collection.base_item,\n fields\n );\n\n RETURN output;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_nonhydrated(\n _item items,\n fields jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry, 20)::jsonb;\n END IF;\n output := jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'collection', _item.collection,\n 'type', 'Feature'\n ) || _item.content;\n RETURN output;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_hydrate(_item items, fields jsonb DEFAULT '{}'::jsonb) RETURNS jsonb AS $$\n SELECT content_hydrate(\n _item,\n (SELECT c FROM collections c WHERE id=_item.collection LIMIT 1),\n fields\n );\n$$ LANGUAGE SQL STABLE;\n\n\nCREATE UNLOGGED TABLE items_staging (\n content JSONB NOT NULL\n);\nCREATE UNLOGGED TABLE items_staging_ignore (\n content JSONB NOT NULL\n);\nCREATE UNLOGGED TABLE items_staging_upsert (\n content JSONB NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION items_staging_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p record;\n _partitions text[];\n ts timestamptz := clock_timestamp();\nBEGIN\n RAISE NOTICE 'Creating Partitions. %', clock_timestamp() - ts;\n WITH ranges AS (\n SELECT\n n.content->>'collection' as collection,\n stac_daterange(n.content->'properties') as dtr\n FROM newdata n\n ), p AS (\n SELECT\n collection,\n lower(dtr) as datetime,\n upper(dtr) as end_datetime,\n (partition_name(\n collection,\n lower(dtr)\n )).partition_name as name\n FROM ranges\n )\n INSERT INTO partitions (collection, datetime_range, end_datetime_range)\n SELECT\n collection,\n tstzrange(min(datetime), max(datetime), '[]') as datetime_range,\n tstzrange(min(end_datetime), max(end_datetime), '[]') as end_datetime_range\n FROM p\n GROUP BY collection, name\n ON CONFLICT (name) DO UPDATE SET\n datetime_range = EXCLUDED.datetime_range,\n end_datetime_range = EXCLUDED.end_datetime_range\n ;\n\n RAISE NOTICE 'Doing the insert. %', clock_timestamp() - ts;\n IF TG_TABLE_NAME = 'items_staging' THEN\n INSERT INTO items\n SELECT\n (content_dehydrate(content)).*\n FROM newdata;\n DELETE FROM items_staging;\n ELSIF TG_TABLE_NAME = 'items_staging_ignore' THEN\n INSERT INTO items\n SELECT\n (content_dehydrate(content)).*\n FROM newdata\n ON CONFLICT DO NOTHING;\n DELETE FROM items_staging_ignore;\n ELSIF TG_TABLE_NAME = 'items_staging_upsert' THEN\n WITH staging_formatted AS (\n SELECT (content_dehydrate(content)).* FROM newdata\n ), deletes AS (\n DELETE FROM items i USING staging_formatted s\n WHERE\n i.id = s.id\n AND i.collection = s.collection\n AND i IS DISTINCT FROM s\n RETURNING i.id, i.collection\n )\n INSERT INTO items\n SELECT s.* FROM\n staging_formatted s\n JOIN deletes d\n USING (id, collection);\n DELETE FROM items_staging_upsert;\n END IF;\n RAISE NOTICE 'Done. %', clock_timestamp() - ts;\n\n RETURN NULL;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER items_staging_insert_trigger AFTER INSERT ON items_staging REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\nCREATE TRIGGER items_staging_insert_ignore_trigger AFTER INSERT ON items_staging_ignore REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\nCREATE TRIGGER items_staging_insert_upsert_trigger AFTER INSERT ON items_staging_upsert REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\n\n\n\nCREATE OR REPLACE FUNCTION item_by_id(_id text, _collection text DEFAULT NULL) RETURNS items AS\n$$\nDECLARE\n i items%ROWTYPE;\nBEGIN\n SELECT * INTO i FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection) LIMIT 1;\n RETURN i;\nEND;\n$$ LANGUAGE PLPGSQL STABLE SECURITY DEFINER SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION get_item(_id text, _collection text DEFAULT NULL) RETURNS jsonb AS $$\n SELECT content_hydrate(items) FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection);\n$$ LANGUAGE SQL STABLE SECURITY DEFINER SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION delete_item(_id text, _collection text DEFAULT NULL) RETURNS VOID AS $$\nDECLARE\nout items%ROWTYPE;\nBEGIN\n DELETE FROM items WHERE id = _id AND (_collection IS NULL OR collection=_collection) RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n--/*\nCREATE OR REPLACE FUNCTION create_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION update_item(content jsonb) RETURNS VOID AS $$\nDECLARE\n old items %ROWTYPE;\n out items%ROWTYPE;\nBEGIN\n PERFORM delete_item(content->>'id', content->>'collection');\n PERFORM create_item(content);\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION create_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION collection_bbox(id text) RETURNS jsonb AS $$\n SELECT (replace(replace(replace(st_extent(geometry)::text,'BOX(','[['),')',']]'),' ',','))::jsonb\n FROM items WHERE collection=$1;\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION collection_temporal_extent(id text) RETURNS jsonb AS $$\n SELECT to_jsonb(array[array[min(datetime)::text, max(datetime)::text]])\n FROM items WHERE collection=$1;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection_extents() RETURNS VOID AS $$\nUPDATE collections SET\n content = content ||\n jsonb_build_object(\n 'extent', jsonb_build_object(\n 'spatial', jsonb_build_object(\n 'bbox', collection_bbox(collections.id)\n ),\n 'temporal', jsonb_build_object(\n 'interval', collection_temporal_extent(collections.id)\n )\n )\n )\n;\n$$ LANGUAGE SQL;\nCREATE VIEW partition_steps AS\nSELECT\n name,\n date_trunc('month',lower(datetime_range)) as sdate,\n date_trunc('month', upper(datetime_range)) + '1 month'::interval as edate\n FROM partitions WHERE datetime_range IS NOT NULL AND datetime_range != 'empty'::tstzrange\n ORDER BY datetime_range ASC\n;\n\nCREATE OR REPLACE FUNCTION chunker(\n IN _where text,\n OUT s timestamptz,\n OUT e timestamptz\n) RETURNS SETOF RECORD AS $$\nDECLARE\n explain jsonb;\nBEGIN\n IF _where IS NULL THEN\n _where := ' TRUE ';\n END IF;\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s;', _where)\n INTO explain;\n\n RETURN QUERY\n WITH t AS (\n SELECT j->>0 as p FROM\n jsonb_path_query(\n explain,\n 'strict $.**.\"Relation Name\" ? (@ != null)'\n ) j\n ),\n parts AS (\n SELECT sdate, edate FROM t JOIN partition_steps ON (t.p = name)\n ),\n times AS (\n SELECT sdate FROM parts\n UNION\n SELECT edate FROM parts\n ),\n uniq AS (\n SELECT DISTINCT sdate FROM times ORDER BY sdate\n ),\n last AS (\n SELECT sdate, lead(sdate, 1) over () as edate FROM uniq\n )\n SELECT sdate, edate FROM last WHERE edate IS NOT NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION partition_queries(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN partitions text[] DEFAULT NULL\n) RETURNS SETOF text AS $$\nDECLARE\n query text;\n sdate timestamptz;\n edate timestamptz;\nBEGIN\nIF _where IS NULL OR trim(_where) = '' THEN\n _where = ' TRUE ';\nEND IF;\nRAISE NOTICE 'Getting chunks for % %', _where, _orderby;\nIF _orderby ILIKE 'datetime d%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 DESC LOOP\n RETURN NEXT format($q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n $q$,\n sdate,\n edate,\n _where,\n _orderby\n );\n END LOOP;\nELSIF _orderby ILIKE 'datetime a%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 ASC LOOP\n RETURN NEXT format($q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n $q$,\n sdate,\n edate,\n _where,\n _orderby\n );\n END LOOP;\nELSE\n query := format($q$\n SELECT * FROM items\n WHERE %s\n ORDER BY %s\n $q$, _where, _orderby\n );\n\n RETURN NEXT query;\n RETURN;\nEND IF;\n\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION partition_query_view(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN _limit int DEFAULT 10\n) RETURNS text AS $$\n WITH p AS (\n SELECT * FROM partition_queries(_where, _orderby) p\n )\n SELECT\n CASE WHEN EXISTS (SELECT 1 FROM p) THEN\n (SELECT format($q$\n SELECT * FROM (\n %s\n ) total LIMIT %s\n $q$,\n string_agg(\n format($q$ SELECT * FROM ( %s ) AS sub $q$, p),\n '\n UNION ALL\n '\n ),\n _limit\n ))\n ELSE NULL\n END FROM p;\n$$ LANGUAGE SQL IMMUTABLE;\n\n\n\n\nCREATE OR REPLACE FUNCTION stac_search_to_where(j jsonb) RETURNS text AS $$\nDECLARE\n where_segments text[];\n _where text;\n dtrange tstzrange;\n collections text[];\n geom geometry;\n sdate timestamptz;\n edate timestamptz;\n filterlang text;\n filter jsonb := j->'filter';\nBEGIN\n IF j ? 'ids' THEN\n where_segments := where_segments || format('id = ANY (%L) ', to_text_array(j->'ids'));\n END IF;\n\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n where_segments := where_segments || format('collection = ANY (%L) ', collections);\n END IF;\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n\n where_segments := where_segments || format(' datetime <= %L::timestamptz AND end_datetime >= %L::timestamptz ',\n edate,\n sdate\n );\n END IF;\n\n geom := stac_geom(j);\n IF geom IS NOT NULL THEN\n where_segments := where_segments || format('st_intersects(geometry, %L)',geom);\n END IF;\n\n filterlang := COALESCE(\n j->>'filter-lang',\n get_setting('default-filter-lang', j->'conf')\n );\n IF NOT filter @? '$.**.op' THEN\n filterlang := 'cql-json';\n END IF;\n\n IF filterlang NOT IN ('cql-json','cql2-json') AND j ? 'filter' THEN\n RAISE EXCEPTION '% is not a supported filter-lang. Please use cql-json or cql2-json.', filterlang;\n END IF;\n\n IF j ? 'query' AND j ? 'filter' THEN\n RAISE EXCEPTION 'Can only use either query or filter at one time.';\n END IF;\n\n IF j ? 'query' THEN\n filter := query_to_cql2(j->'query');\n ELSIF filterlang = 'cql-json' THEN\n filter := cql1_to_cql2(filter);\n END IF;\n RAISE NOTICE 'FILTER: %', filter;\n where_segments := where_segments || cql2_query(filter);\n IF cardinality(where_segments) < 1 THEN\n RETURN ' TRUE ';\n END IF;\n\n _where := array_to_string(array_remove(where_segments, NULL), ' AND ');\n\n IF _where IS NULL OR BTRIM(_where) = '' THEN\n RETURN ' TRUE ';\n END IF;\n RETURN _where;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION parse_sort_dir(_dir text, reverse boolean default false) RETURNS text AS $$\n WITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n ) SELECT\n CASE\n WHEN NOT reverse THEN d\n WHEN d = 'ASC' THEN 'DESC'\n WHEN d = 'DESC' THEN 'ASC'\n END\n FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION sort_dir_to_op(_dir text, prev boolean default false) RETURNS text AS $$\n WITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n ) SELECT\n CASE\n WHEN d = 'ASC' AND prev THEN '<='\n WHEN d = 'DESC' AND prev THEN '>='\n WHEN d = 'ASC' THEN '>='\n WHEN d = 'DESC' THEN '<='\n END\n FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION sort_sqlorderby(\n _search jsonb DEFAULT NULL,\n reverse boolean DEFAULT FALSE\n) RETURNS text AS $$\n WITH sortby AS (\n SELECT coalesce(_search->'sortby','[{\"field\":\"datetime\", \"direction\":\"desc\"}]') as sort\n ), withid AS (\n SELECT CASE\n WHEN sort @? '$[*] ? (@.field == \"id\")' THEN sort\n ELSE sort || '[{\"field\":\"id\", \"direction\":\"desc\"}]'::jsonb\n END as sort\n FROM sortby\n ), withid_rows AS (\n SELECT jsonb_array_elements(sort) as value FROM withid\n ),sorts AS (\n SELECT\n coalesce(\n -- field_orderby((items_path(value->>'field')).path_txt),\n (queryable(value->>'field')).expression\n ) as key,\n parse_sort_dir(value->>'direction', reverse) as dir\n FROM withid_rows\n )\n SELECT array_to_string(\n array_agg(concat(key, ' ', dir)),\n ', '\n ) FROM sorts;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_sort_dir(sort_item jsonb) RETURNS text AS $$\n SELECT CASE WHEN sort_item->>'direction' ILIKE 'desc%' THEN 'DESC' ELSE 'ASC' END;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION get_token_filter(_search jsonb = '{}'::jsonb, token_rec jsonb DEFAULT NULL) RETURNS text AS $$\nDECLARE\n token_id text;\n filters text[] := '{}'::text[];\n prev boolean := TRUE;\n field text;\n dir text;\n sort record;\n orfilters text[] := '{}'::text[];\n andfilters text[] := '{}'::text[];\n output text;\n token_where text;\nBEGIN\n RAISE NOTICE 'Getting Token Filter. % %', _search, token_rec;\n -- If no token provided return NULL\n IF token_rec IS NULL THEN\n IF NOT (_search ? 'token' AND\n (\n (_search->>'token' ILIKE 'prev:%')\n OR\n (_search->>'token' ILIKE 'next:%')\n )\n ) THEN\n RETURN NULL;\n END IF;\n prev := (_search->>'token' ILIKE 'prev:%');\n token_id := substr(_search->>'token', 6);\n SELECT to_jsonb(items) INTO token_rec\n FROM items WHERE id=token_id;\n END IF;\n RAISE NOTICE 'TOKEN ID: % %', token_rec, token_rec->'id';\n\n CREATE TEMP TABLE sorts (\n _row int GENERATED ALWAYS AS IDENTITY NOT NULL,\n _field text PRIMARY KEY,\n _dir text NOT NULL,\n _val text\n ) ON COMMIT DROP;\n\n -- Make sure we only have distinct columns to sort with taking the first one we get\n INSERT INTO sorts (_field, _dir)\n SELECT\n (queryable(value->>'field')).expression,\n get_sort_dir(value)\n FROM\n jsonb_array_elements(coalesce(_search->'sortby','[{\"field\":\"datetime\",\"direction\":\"desc\"}]'))\n ON CONFLICT DO NOTHING\n ;\n RAISE NOTICE 'sorts 1: %', (SELECT jsonb_agg(to_json(sorts)) FROM sorts);\n -- Get the first sort direction provided. As the id is a primary key, if there are any\n -- sorts after id they won't do anything, so make sure that id is the last sort item.\n SELECT _dir INTO dir FROM sorts ORDER BY _row ASC LIMIT 1;\n IF EXISTS (SELECT 1 FROM sorts WHERE _field = 'id') THEN\n DELETE FROM sorts WHERE _row > (SELECT _row FROM sorts WHERE _field = 'id' ORDER BY _row ASC);\n ELSE\n INSERT INTO sorts (_field, _dir) VALUES ('id', dir);\n END IF;\n\n -- Add value from looked up item to the sorts table\n UPDATE sorts SET _val=quote_literal(token_rec->>_field);\n\n -- Check if all sorts are the same direction and use row comparison\n -- to filter\n RAISE NOTICE 'sorts 2: %', (SELECT jsonb_agg(to_json(sorts)) FROM sorts);\n\n IF (SELECT count(DISTINCT _dir) FROM sorts) = 1 THEN\n SELECT format(\n '(%s) %s (%s)',\n concat_ws(', ', VARIADIC array_agg(quote_ident(_field))),\n CASE WHEN (prev AND dir = 'ASC') OR (NOT prev AND dir = 'DESC') THEN '<' ELSE '>' END,\n concat_ws(', ', VARIADIC array_agg(_val))\n ) INTO output FROM sorts\n WHERE token_rec ? _field\n ;\n ELSE\n FOR sort IN SELECT * FROM sorts ORDER BY _row asc LOOP\n RAISE NOTICE 'SORT: %', sort;\n IF sort._row = 1 THEN\n orfilters := orfilters || format('(%s %s %s)',\n quote_ident(sort._field),\n CASE WHEN (prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC') THEN '<' ELSE '>' END,\n sort._val\n );\n ELSE\n orfilters := orfilters || format('(%s AND %s %s %s)',\n array_to_string(andfilters, ' AND '),\n quote_ident(sort._field),\n CASE WHEN (prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC') THEN '<' ELSE '>' END,\n sort._val\n );\n\n END IF;\n andfilters := andfilters || format('%s = %s',\n quote_ident(sort._field),\n sort._val\n );\n END LOOP;\n output := array_to_string(orfilters, ' OR ');\n END IF;\n DROP TABLE IF EXISTS sorts;\n token_where := concat('(',coalesce(output,'true'),')');\n IF trim(token_where) = '' THEN\n token_where := NULL;\n END IF;\n RAISE NOTICE 'TOKEN_WHERE: |%|',token_where;\n RETURN token_where;\n END;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION search_tohash(jsonb) RETURNS jsonb AS $$\n SELECT $1 - '{token,limit,context,includes,excludes}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION search_hash(jsonb, jsonb) RETURNS text AS $$\n SELECT md5(concat(search_tohash($1)::text,$2::text));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TABLE IF NOT EXISTS searches(\n hash text GENERATED ALWAYS AS (search_hash(search, metadata)) STORED PRIMARY KEY,\n search jsonb NOT NULL,\n _where text,\n orderby text,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n metadata jsonb DEFAULT '{}'::jsonb NOT NULL\n);\nCREATE TABLE IF NOT EXISTS search_wheres(\n id bigint generated always as identity primary key,\n _where text NOT NULL,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n statslastupdated timestamptz,\n estimated_count bigint,\n estimated_cost float,\n time_to_estimate float,\n total_count bigint,\n time_to_count float,\n partitions text[]\n);\n\nCREATE INDEX IF NOT EXISTS search_wheres_partitions ON search_wheres USING GIN (partitions);\nCREATE UNIQUE INDEX IF NOT EXISTS search_wheres_where ON search_wheres ((md5(_where)));\n\nCREATE OR REPLACE FUNCTION where_stats(inwhere text, updatestats boolean default false, conf jsonb default null) RETURNS search_wheres AS $$\nDECLARE\n t timestamptz;\n i interval;\n explain_json jsonb;\n partitions text[];\n sw search_wheres%ROWTYPE;\n inwhere_hash text := md5(inwhere);\n _context text := lower(context(conf));\n _stats_ttl interval := context_stats_ttl(conf);\n _estimated_cost float := context_estimated_cost(conf);\n _estimated_count int := context_estimated_count(conf);\nBEGIN\n IF _context = 'off' THEN\n sw._where := inwhere;\n return sw;\n END IF;\n\n SELECT * INTO sw FROM search_wheres WHERE md5(_where)=inwhere_hash FOR UPDATE;\n\n -- Update statistics if explicitly set, if statistics do not exist, or statistics ttl has expired\n IF NOT updatestats THEN\n RAISE NOTICE 'Checking if update is needed for: % .', inwhere;\n RAISE NOTICE 'Stats Last Updated: %', sw.statslastupdated;\n RAISE NOTICE 'TTL: %, Age: %', _stats_ttl, now() - sw.statslastupdated;\n RAISE NOTICE 'Context: %, Existing Total: %', _context, sw.total_count;\n IF\n sw.statslastupdated IS NULL\n OR (now() - sw.statslastupdated) > _stats_ttl\n OR (context(conf) != 'off' AND sw.total_count IS NULL)\n THEN\n updatestats := TRUE;\n END IF;\n END IF;\n\n sw._where := inwhere;\n sw.lastused := now();\n sw.usecount := coalesce(sw.usecount,0) + 1;\n\n IF NOT updatestats THEN\n UPDATE search_wheres SET\n lastused = sw.lastused,\n usecount = sw.usecount\n WHERE md5(_where) = inwhere_hash\n RETURNING * INTO sw\n ;\n RETURN sw;\n END IF;\n\n -- Use explain to get estimated count/cost and a list of the partitions that would be hit by the query\n t := clock_timestamp();\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s', inwhere)\n INTO explain_json;\n RAISE NOTICE 'Time for just the explain: %', clock_timestamp() - t;\n i := clock_timestamp() - t;\n\n sw.statslastupdated := now();\n sw.estimated_count := explain_json->0->'Plan'->'Plan Rows';\n sw.estimated_cost := explain_json->0->'Plan'->'Total Cost';\n sw.time_to_estimate := extract(epoch from i);\n\n RAISE NOTICE 'ESTIMATED_COUNT: % < %', sw.estimated_count, _estimated_count;\n RAISE NOTICE 'ESTIMATED_COST: % < %', sw.estimated_cost, _estimated_cost;\n\n -- Do a full count of rows if context is set to on or if auto is set and estimates are low enough\n IF\n _context = 'on'\n OR\n ( _context = 'auto' AND\n (\n sw.estimated_count < _estimated_count\n AND\n sw.estimated_cost < _estimated_cost\n )\n )\n THEN\n t := clock_timestamp();\n RAISE NOTICE 'Calculating actual count...';\n EXECUTE format(\n 'SELECT count(*) FROM items WHERE %s',\n inwhere\n ) INTO sw.total_count;\n i := clock_timestamp() - t;\n RAISE NOTICE 'Actual Count: % -- %', sw.total_count, i;\n sw.time_to_count := extract(epoch FROM i);\n ELSE\n sw.total_count := NULL;\n sw.time_to_count := NULL;\n END IF;\n\n\n INSERT INTO search_wheres\n (_where, lastused, usecount, statslastupdated, estimated_count, estimated_cost, time_to_estimate, partitions, total_count, time_to_count)\n SELECT sw._where, sw.lastused, sw.usecount, sw.statslastupdated, sw.estimated_count, sw.estimated_cost, sw.time_to_estimate, sw.partitions, sw.total_count, sw.time_to_count\n ON CONFLICT ((md5(_where)))\n DO UPDATE\n SET\n lastused = sw.lastused,\n usecount = sw.usecount,\n statslastupdated = sw.statslastupdated,\n estimated_count = sw.estimated_count,\n estimated_cost = sw.estimated_cost,\n time_to_estimate = sw.time_to_estimate,\n total_count = sw.total_count,\n time_to_count = sw.time_to_count\n ;\n RETURN sw;\nEND;\n$$ LANGUAGE PLPGSQL ;\n\n\n\nDROP FUNCTION IF EXISTS search_query;\nCREATE OR REPLACE FUNCTION search_query(\n _search jsonb = '{}'::jsonb,\n updatestats boolean = false,\n _metadata jsonb = '{}'::jsonb\n) RETURNS searches AS $$\nDECLARE\n search searches%ROWTYPE;\n pexplain jsonb;\n t timestamptz;\n i interval;\nBEGIN\n SELECT * INTO search FROM searches\n WHERE hash=search_hash(_search, _metadata) FOR UPDATE;\n\n -- Calculate the where clause if not already calculated\n IF search._where IS NULL THEN\n search._where := stac_search_to_where(_search);\n END IF;\n\n -- Calculate the order by clause if not already calculated\n IF search.orderby IS NULL THEN\n search.orderby := sort_sqlorderby(_search);\n END IF;\n\n PERFORM where_stats(search._where, updatestats, _search->'conf');\n\n search.lastused := now();\n search.usecount := coalesce(search.usecount, 0) + 1;\n INSERT INTO searches (search, _where, orderby, lastused, usecount, metadata)\n VALUES (_search, search._where, search.orderby, search.lastused, search.usecount, _metadata)\n ON CONFLICT (hash) DO\n UPDATE SET\n _where = EXCLUDED._where,\n orderby = EXCLUDED.orderby,\n lastused = EXCLUDED.lastused,\n usecount = EXCLUDED.usecount,\n metadata = EXCLUDED.metadata\n RETURNING * INTO search\n ;\n RETURN search;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION search(_search jsonb = '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n searches searches%ROWTYPE;\n _where text;\n token_where text;\n full_where text;\n orderby text;\n query text;\n token_type text := substr(_search->>'token',1,4);\n _limit int := coalesce((_search->>'limit')::int, 10);\n curs refcursor;\n cntr int := 0;\n iter_record items%ROWTYPE;\n first_record jsonb;\n first_item items%ROWTYPE;\n last_item items%ROWTYPE;\n last_record jsonb;\n out_records jsonb := '[]'::jsonb;\n prev_query text;\n next text;\n prev_id text;\n has_next boolean := false;\n has_prev boolean := false;\n prev text;\n total_count bigint;\n context jsonb;\n collection jsonb;\n includes text[];\n excludes text[];\n exit_flag boolean := FALSE;\n batches int := 0;\n timer timestamptz := clock_timestamp();\n pstart timestamptz;\n pend timestamptz;\n pcurs refcursor;\n search_where search_wheres%ROWTYPE;\n id text;\nBEGIN\nCREATE TEMP TABLE results (content jsonb) ON COMMIT DROP;\n-- if ids is set, short circuit and just use direct ids query for each id\n-- skip any paging or caching\n-- hard codes ordering in the same order as the array of ids\nIF _search ? 'ids' THEN\n INSERT INTO results\n SELECT\n CASE WHEN _search->'conf'->>'nohydrate' IS NOT NULL AND (_search->'conf'->>'nohydrate')::boolean = true THEN\n content_nonhydrated(items, _search->'fields')\n ELSE\n content_hydrate(items, _search->'fields')\n END\n FROM items WHERE\n items.id = ANY(to_text_array(_search->'ids'))\n AND\n CASE WHEN _search ? 'collections' THEN\n items.collection = ANY(to_text_array(_search->'collections'))\n ELSE TRUE\n END\n ORDER BY items.datetime desc, items.id desc\n ;\n SELECT INTO total_count count(*) FROM results;\nELSE\n searches := search_query(_search);\n _where := searches._where;\n orderby := searches.orderby;\n search_where := where_stats(_where);\n total_count := coalesce(search_where.total_count, search_where.estimated_count);\n\n IF token_type='prev' THEN\n token_where := get_token_filter(_search, null::jsonb);\n orderby := sort_sqlorderby(_search, TRUE);\n END IF;\n IF token_type='next' THEN\n token_where := get_token_filter(_search, null::jsonb);\n END IF;\n\n full_where := concat_ws(' AND ', _where, token_where);\n RAISE NOTICE 'FULL QUERY % %', full_where, clock_timestamp()-timer;\n timer := clock_timestamp();\n\n FOR query IN SELECT partition_queries(full_where, orderby, search_where.partitions) LOOP\n timer := clock_timestamp();\n query := format('%s LIMIT %s', query, _limit + 1);\n RAISE NOTICE 'Partition Query: %', query;\n batches := batches + 1;\n -- curs = create_cursor(query);\n OPEN curs FOR EXECUTE query;\n LOOP\n FETCH curs into iter_record;\n EXIT WHEN NOT FOUND;\n cntr := cntr + 1;\n\n IF _search->'conf'->>'nohydrate' IS NOT NULL AND (_search->'conf'->>'nohydrate')::boolean = true THEN\n last_record := content_nonhydrated(iter_record, _search->'fields');\n ELSE\n last_record := content_hydrate(iter_record, _search->'fields');\n END IF;\n last_item := iter_record;\n IF cntr = 1 THEN\n first_item := last_item;\n first_record := last_record;\n END IF;\n IF cntr <= _limit THEN\n INSERT INTO results (content) VALUES (last_record);\n ELSIF cntr > _limit THEN\n has_next := true;\n exit_flag := true;\n EXIT;\n END IF;\n END LOOP;\n CLOSE curs;\n RAISE NOTICE 'Query took %.', clock_timestamp()-timer;\n timer := clock_timestamp();\n EXIT WHEN exit_flag;\n END LOOP;\n RAISE NOTICE 'Scanned through % partitions.', batches;\nEND IF;\n\nSELECT jsonb_agg(content) INTO out_records FROM results WHERE content is not NULL;\n\nDROP TABLE results;\n\n\n-- Flip things around if this was the result of a prev token query\nIF token_type='prev' THEN\n out_records := flip_jsonb_array(out_records);\n first_item := last_item;\n first_record := last_record;\nEND IF;\n\n-- If this query has a token, see if there is data before the first record\nIF _search ? 'token' THEN\n prev_query := format(\n 'SELECT 1 FROM items WHERE %s LIMIT 1',\n concat_ws(\n ' AND ',\n _where,\n trim(get_token_filter(_search, to_jsonb(first_item)))\n )\n );\n RAISE NOTICE 'Query to get previous record: % --- %', prev_query, first_record;\n EXECUTE prev_query INTO has_prev;\n IF FOUND and has_prev IS NOT NULL THEN\n RAISE NOTICE 'Query results from prev query: %', has_prev;\n has_prev := TRUE;\n END IF;\nEND IF;\nhas_prev := COALESCE(has_prev, FALSE);\n\nIF has_prev THEN\n prev := out_records->0->>'id';\nEND IF;\nIF has_next OR token_type='prev' THEN\n next := out_records->-1->>'id';\nEND IF;\n\nIF context(_search->'conf') != 'off' THEN\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'matched', total_count,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\nELSE\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\nEND IF;\n\ncollection := jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb),\n 'next', next,\n 'prev', prev,\n 'context', context\n);\n\nRETURN collection;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER SET SEARCH_PATH TO pgstac, public;\n\n\nCREATE OR REPLACE FUNCTION search_cursor(_search jsonb = '{}'::jsonb) RETURNS refcursor AS $$\nDECLARE\n curs refcursor;\n searches searches%ROWTYPE;\n _where text;\n _orderby text;\n q text;\n\nBEGIN\n searches := search_query(_search);\n _where := searches._where;\n _orderby := searches.orderby;\n\n OPEN curs FOR\n WITH p AS (\n SELECT * FROM partition_queries(_where, _orderby) p\n )\n SELECT\n CASE WHEN EXISTS (SELECT 1 FROM p) THEN\n (SELECT format($q$\n SELECT * FROM (\n %s\n ) total\n $q$,\n string_agg(\n format($q$ SELECT * FROM ( %s ) AS sub $q$, p),\n '\n UNION ALL\n '\n )\n ))\n ELSE NULL\n END FROM p;\n RETURN curs;\nEND;\n$$ LANGUAGE PLPGSQL;\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION tileenvelope(zoom int, x int, y int) RETURNS geometry AS $$\nWITH t AS (\n SELECT\n 20037508.3427892 as merc_max,\n -20037508.3427892 as merc_min,\n (2 * 20037508.3427892) / (2 ^ zoom) as tile_size\n)\nSELECT st_makeenvelope(\n merc_min + (tile_size * x),\n merc_max - (tile_size * (y + 1)),\n merc_min + (tile_size * (x + 1)),\n merc_max - (tile_size * y),\n 3857\n) FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;DROP FUNCTION IF EXISTS mercgrid;\n\n\nCREATE OR REPLACE FUNCTION ftime() RETURNS interval as $$\nSELECT age(clock_timestamp(), transaction_timestamp());\n$$ LANGUAGE SQL;\nSET SEARCH_PATH to pgstac, public;\n\nDROP FUNCTION IF EXISTS geometrysearch;\nCREATE OR REPLACE FUNCTION geometrysearch(\n IN geom geometry,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE, -- Return as soon as the passed in geometry is full covered\n IN skipcovered boolean DEFAULT TRUE -- Skip any items that would show up completely under the previous items\n) RETURNS jsonb AS $$\nDECLARE\n search searches%ROWTYPE;\n curs refcursor;\n _where text;\n query text;\n iter_record items%ROWTYPE;\n out_records jsonb := '{}'::jsonb[];\n exit_flag boolean := FALSE;\n counter int := 1;\n scancounter int := 1;\n remaining_limit int := _scanlimit;\n tilearea float;\n unionedgeom geometry;\n clippedgeom geometry;\n unionedgeom_area float := 0;\n prev_area float := 0;\n excludes text[];\n includes text[];\n\nBEGIN\n DROP TABLE IF EXISTS pgstac_results;\n CREATE TEMP TABLE pgstac_results (content jsonb) ON COMMIT DROP;\n\n -- If skipcovered is true then you will always want to exit when the passed in geometry is full\n IF skipcovered THEN\n exitwhenfull := TRUE;\n END IF;\n\n SELECT * INTO search FROM searches WHERE hash=queryhash;\n\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Search with Query Hash % Not Found', queryhash;\n END IF;\n\n tilearea := st_area(geom);\n _where := format('%s AND st_intersects(geometry, %L::geometry)', search._where, geom);\n\n\n FOR query IN SELECT * FROM partition_queries(_where, search.orderby) LOOP\n query := format('%s LIMIT %L', query, remaining_limit);\n RAISE NOTICE '%', query;\n OPEN curs FOR EXECUTE query;\n LOOP\n FETCH curs INTO iter_record;\n EXIT WHEN NOT FOUND;\n IF exitwhenfull OR skipcovered THEN -- If we are not using exitwhenfull or skipcovered, we do not need to do expensive geometry operations\n clippedgeom := st_intersection(geom, iter_record.geometry);\n\n IF unionedgeom IS NULL THEN\n unionedgeom := clippedgeom;\n ELSE\n unionedgeom := st_union(unionedgeom, clippedgeom);\n END IF;\n\n unionedgeom_area := st_area(unionedgeom);\n\n IF skipcovered AND prev_area = unionedgeom_area THEN\n scancounter := scancounter + 1;\n CONTINUE;\n END IF;\n\n prev_area := unionedgeom_area;\n\n RAISE NOTICE '% % % %', unionedgeom_area/tilearea, counter, scancounter, ftime();\n END IF;\n RAISE NOTICE '% %', iter_record, content_hydrate(iter_record, fields);\n INSERT INTO pgstac_results (content) VALUES (content_hydrate(iter_record, fields));\n\n IF counter >= _limit\n OR scancounter > _scanlimit\n OR ftime() > _timelimit\n OR (exitwhenfull AND unionedgeom_area >= tilearea)\n THEN\n exit_flag := TRUE;\n EXIT;\n END IF;\n counter := counter + 1;\n scancounter := scancounter + 1;\n\n END LOOP;\n CLOSE curs;\n EXIT WHEN exit_flag;\n remaining_limit := _scanlimit - scancounter;\n END LOOP;\n\n SELECT jsonb_agg(content) INTO out_records FROM pgstac_results WHERE content IS NOT NULL;\n\n RETURN jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb)\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDROP FUNCTION IF EXISTS geojsonsearch;\nCREATE OR REPLACE FUNCTION geojsonsearch(\n IN geojson jsonb,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_geomfromgeojson(geojson),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS xyzsearch;\nCREATE OR REPLACE FUNCTION xyzsearch(\n IN _x int,\n IN _y int,\n IN _z int,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_transform(tileenvelope(_z, _x, _y), 4326),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nGRANT ALL ON SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac to pgstac_admin;\n\n-- pgstac_read role limited to using function apis\nGRANT EXECUTE ON FUNCTION search TO pgstac_read;\nGRANT EXECUTE ON FUNCTION search_query TO pgstac_read;\nGRANT EXECUTE ON FUNCTION item_by_id TO pgstac_read;\nGRANT EXECUTE ON FUNCTION get_item TO pgstac_read;\n\nGRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_ingest;\nGRANT USAGE ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_ingest;\nSELECT set_version('0.6.7');\n" }, { "path": "src/pgstac/migrations/pgstac.0.6.8-0.6.9.sql", "content": "SET SEARCH_PATH to pgstac, public;\nset check_function_bodies = off;\n\nCREATE OR REPLACE FUNCTION pgstac.search(_search jsonb DEFAULT '{}'::jsonb)\n RETURNS jsonb\n LANGUAGE plpgsql\n SECURITY DEFINER\n SET search_path TO 'pgstac', 'public'\n SET cursor_tuple_fraction TO '1'\nAS $function$\nDECLARE\n searches searches%ROWTYPE;\n _where text;\n token_where text;\n full_where text;\n orderby text;\n query text;\n token_type text := substr(_search->>'token',1,4);\n _limit int := coalesce((_search->>'limit')::int, 10);\n curs refcursor;\n cntr int := 0;\n iter_record items%ROWTYPE;\n first_record jsonb;\n first_item items%ROWTYPE;\n last_item items%ROWTYPE;\n last_record jsonb;\n out_records jsonb := '[]'::jsonb;\n prev_query text;\n next text;\n prev_id text;\n has_next boolean := false;\n has_prev boolean := false;\n prev text;\n total_count bigint;\n context jsonb;\n collection jsonb;\n includes text[];\n excludes text[];\n exit_flag boolean := FALSE;\n batches int := 0;\n timer timestamptz := clock_timestamp();\n pstart timestamptz;\n pend timestamptz;\n pcurs refcursor;\n search_where search_wheres%ROWTYPE;\n id text;\nBEGIN\nCREATE TEMP TABLE results (content jsonb) ON COMMIT DROP;\n-- if ids is set, short circuit and just use direct ids query for each id\n-- skip any paging or caching\n-- hard codes ordering in the same order as the array of ids\nIF _search ? 'ids' THEN\n INSERT INTO results\n SELECT\n CASE WHEN _search->'conf'->>'nohydrate' IS NOT NULL AND (_search->'conf'->>'nohydrate')::boolean = true THEN\n content_nonhydrated(items, _search->'fields')\n ELSE\n content_hydrate(items, _search->'fields')\n END\n FROM items WHERE\n items.id = ANY(to_text_array(_search->'ids'))\n AND\n CASE WHEN _search ? 'collections' THEN\n items.collection = ANY(to_text_array(_search->'collections'))\n ELSE TRUE\n END\n ORDER BY items.datetime desc, items.id desc\n ;\n SELECT INTO total_count count(*) FROM results;\nELSE\n searches := search_query(_search);\n _where := searches._where;\n orderby := searches.orderby;\n search_where := where_stats(_where);\n total_count := coalesce(search_where.total_count, search_where.estimated_count);\n\n IF token_type='prev' THEN\n token_where := get_token_filter(_search, null::jsonb);\n orderby := sort_sqlorderby(_search, TRUE);\n END IF;\n IF token_type='next' THEN\n token_where := get_token_filter(_search, null::jsonb);\n END IF;\n\n full_where := concat_ws(' AND ', _where, token_where);\n RAISE NOTICE 'FULL QUERY % %', full_where, clock_timestamp()-timer;\n timer := clock_timestamp();\n\n FOR query IN SELECT partition_queries(full_where, orderby, search_where.partitions) LOOP\n timer := clock_timestamp();\n query := format('%s LIMIT %s', query, _limit + 1);\n RAISE NOTICE 'Partition Query: %', query;\n batches := batches + 1;\n -- curs = create_cursor(query);\n RAISE NOTICE 'cursor_tuple_fraction: %', current_setting('cursor_tuple_fraction');\n OPEN curs FOR EXECUTE query;\n LOOP\n FETCH curs into iter_record;\n EXIT WHEN NOT FOUND;\n cntr := cntr + 1;\n\n IF _search->'conf'->>'nohydrate' IS NOT NULL AND (_search->'conf'->>'nohydrate')::boolean = true THEN\n last_record := content_nonhydrated(iter_record, _search->'fields');\n ELSE\n last_record := content_hydrate(iter_record, _search->'fields');\n END IF;\n last_item := iter_record;\n IF cntr = 1 THEN\n first_item := last_item;\n first_record := last_record;\n END IF;\n IF cntr <= _limit THEN\n INSERT INTO results (content) VALUES (last_record);\n ELSIF cntr > _limit THEN\n has_next := true;\n exit_flag := true;\n EXIT;\n END IF;\n END LOOP;\n CLOSE curs;\n RAISE NOTICE 'Query took %.', clock_timestamp()-timer;\n timer := clock_timestamp();\n EXIT WHEN exit_flag;\n END LOOP;\n RAISE NOTICE 'Scanned through % partitions.', batches;\nEND IF;\n\nSELECT jsonb_agg(content) INTO out_records FROM results WHERE content is not NULL;\n\nDROP TABLE results;\n\n\n-- Flip things around if this was the result of a prev token query\nIF token_type='prev' THEN\n out_records := flip_jsonb_array(out_records);\n first_item := last_item;\n first_record := last_record;\nEND IF;\n\n-- If this query has a token, see if there is data before the first record\nIF _search ? 'token' THEN\n prev_query := format(\n 'SELECT 1 FROM items WHERE %s LIMIT 1',\n concat_ws(\n ' AND ',\n _where,\n trim(get_token_filter(_search, to_jsonb(first_item)))\n )\n );\n RAISE NOTICE 'Query to get previous record: % --- %', prev_query, first_record;\n EXECUTE prev_query INTO has_prev;\n IF FOUND and has_prev IS NOT NULL THEN\n RAISE NOTICE 'Query results from prev query: %', has_prev;\n has_prev := TRUE;\n END IF;\nEND IF;\nhas_prev := COALESCE(has_prev, FALSE);\n\nIF has_prev THEN\n prev := out_records->0->>'id';\nEND IF;\nIF has_next OR token_type='prev' THEN\n next := out_records->-1->>'id';\nEND IF;\n\nIF context(_search->'conf') != 'off' THEN\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'matched', total_count,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\nELSE\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\nEND IF;\n\ncollection := jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb),\n 'next', next,\n 'prev', prev,\n 'context', context\n);\n\nRETURN collection;\nEND;\n$function$\n;\n\n\n\nSELECT set_version('0.6.9');\n" }, { "path": "src/pgstac/migrations/pgstac.0.6.8.sql", "content": "CREATE EXTENSION IF NOT EXISTS postgis;\nCREATE EXTENSION IF NOT EXISTS btree_gist;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_admin;\n CREATE ROLE pgstac_read;\n CREATE ROLE pgstac_ingest;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nGRANT pgstac_admin TO current_user;\n\nCREATE SCHEMA IF NOT EXISTS pgstac AUTHORIZATION pgstac_admin;\n\nALTER ROLE pgstac_admin SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_read SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_ingest SET SEARCH_PATH TO pgstac, public;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\n\nGRANT pgstac_read TO pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\n\nSET ROLE pgstac_admin;\n\nSET SEARCH_PATH TO pgstac, public;\n\n\nCREATE TABLE IF NOT EXISTS migrations (\n version text PRIMARY KEY,\n datetime timestamptz DEFAULT clock_timestamp() NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION get_version() RETURNS text AS $$\n SELECT version FROM pgstac.migrations ORDER BY datetime DESC, version DESC LIMIT 1;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION set_version(text) RETURNS text AS $$\n INSERT INTO pgstac.migrations (version) VALUES ($1)\n ON CONFLICT DO NOTHING\n RETURNING version;\n$$ LANGUAGE SQL;\n\n\nCREATE TABLE IF NOT EXISTS pgstac_settings (\n name text PRIMARY KEY,\n value text NOT NULL\n);\n\nINSERT INTO pgstac_settings (name, value) VALUES\n ('context', 'off'),\n ('context_estimated_count', '100000'),\n ('context_estimated_cost', '100000'),\n ('context_stats_ttl', '1 day'),\n ('default-filter-lang', 'cql2-json'),\n ('additional_properties', 'true')\nON CONFLICT DO NOTHING\n;\n\n\nCREATE OR REPLACE FUNCTION get_setting(IN _setting text, IN conf jsonb DEFAULT NULL) RETURNS text AS $$\nSELECT COALESCE(\n conf->>_setting,\n current_setting(concat('pgstac.',_setting), TRUE),\n (SELECT value FROM pgstac.pgstac_settings WHERE name=_setting)\n);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context(conf jsonb DEFAULT NULL) RETURNS text AS $$\n SELECT pgstac.get_setting('context', conf);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context_estimated_count(conf jsonb DEFAULT NULL) RETURNS int AS $$\n SELECT pgstac.get_setting('context_estimated_count', conf)::int;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_estimated_cost();\nCREATE OR REPLACE FUNCTION context_estimated_cost(conf jsonb DEFAULT NULL) RETURNS float AS $$\n SELECT pgstac.get_setting('context_estimated_cost', conf)::float;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_stats_ttl();\nCREATE OR REPLACE FUNCTION context_stats_ttl(conf jsonb DEFAULT NULL) RETURNS interval AS $$\n SELECT pgstac.get_setting('context_stats_ttl', conf)::interval;\n$$ LANGUAGE SQL;\n\n\nCREATE OR REPLACE FUNCTION notice(VARIADIC text[]) RETURNS boolean AS $$\nDECLARE\ndebug boolean := current_setting('pgstac.debug', true);\nBEGIN\n IF debug THEN\n RAISE NOTICE 'NOTICE FROM FUNC: % >>>>> %', concat_ws(' | ', $1), clock_timestamp();\n RETURN TRUE;\n END IF;\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION empty_arr(ANYARRAY) RETURNS BOOLEAN AS $$\nSELECT CASE\n WHEN $1 IS NULL THEN TRUE\n WHEN cardinality($1)<1 THEN TRUE\nELSE FALSE\nEND;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_intersection(_a ANYARRAY, _b ANYARRAY) RETURNS ANYARRAY AS $$\n SELECT ARRAY ( SELECT unnest(_a) INTERSECT SELECT UNNEST(_b) );\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION array_map_ident(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_ident(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_map_literal(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_literal(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION array_reverse(anyarray) RETURNS anyarray AS $$\nSELECT ARRAY(\n SELECT $1[i]\n FROM generate_subscripts($1,1) AS s(i)\n ORDER BY i DESC\n);\n$$ LANGUAGE SQL STRICT IMMUTABLE;\nCREATE OR REPLACE FUNCTION to_int(jsonb) RETURNS int AS $$\n SELECT floor(($1->>0)::float)::int;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION to_float(jsonb) RETURNS float AS $$\n SELECT ($1->>0)::float;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION to_tstz(jsonb) RETURNS timestamptz AS $$\n SELECT ($1->>0)::timestamptz;\n$$ LANGUAGE SQL IMMUTABLE STRICT SET TIME ZONE 'UTC';\n\n\nCREATE OR REPLACE FUNCTION to_text(jsonb) RETURNS text AS $$\n SELECT CASE WHEN jsonb_typeof($1) IN ('array','object') THEN $1::text ELSE $1->>0 END;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION to_text_array(jsonb) RETURNS text[] AS $$\n SELECT\n CASE jsonb_typeof($1)\n WHEN 'array' THEN ARRAY(SELECT jsonb_array_elements_text($1))\n ELSE ARRAY[$1->>0]\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION bbox_geom(_bbox jsonb) RETURNS geometry AS $$\nSELECT CASE jsonb_array_length(_bbox)\n WHEN 4 THEN\n ST_SetSRID(ST_MakeEnvelope(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float,\n (_bbox->>3)::float\n ),4326)\n WHEN 6 THEN\n ST_SetSRID(ST_3DMakeBox(\n ST_MakePoint(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float\n ),\n ST_MakePoint(\n (_bbox->>3)::float,\n (_bbox->>4)::float,\n (_bbox->>5)::float\n )\n ),4326)\n ELSE null END;\n;\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION geom_bbox(_geom geometry) RETURNS jsonb AS $$\n SELECT jsonb_build_array(\n st_xmin(_geom),\n st_ymin(_geom),\n st_xmax(_geom),\n st_ymax(_geom)\n );\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION flip_jsonb_array(j jsonb) RETURNS jsonb AS $$\n SELECT jsonb_agg(value) FROM (SELECT value FROM jsonb_array_elements(j) WITH ORDINALITY ORDER BY ordinality DESC) as t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION explode_dotpaths(j jsonb) RETURNS SETOF text[] AS $$\n SELECT string_to_array(p, '.') as e FROM jsonb_array_elements_text(j) p;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION explode_dotpaths_recurse(IN j jsonb) RETURNS SETOF text[] AS $$\n WITH RECURSIVE t AS (\n SELECT e FROM explode_dotpaths(j) e\n UNION ALL\n SELECT e[1:cardinality(e)-1]\n FROM t\n WHERE cardinality(e)>1\n ) SELECT e FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION jsonb_set_nested(j jsonb, path text[], val jsonb) RETURNS jsonb AS $$\nDECLARE\nBEGIN\n IF cardinality(path) > 1 THEN\n FOR i IN 1..(cardinality(path)-1) LOOP\n IF j #> path[:i] IS NULL THEN\n j := jsonb_set_lax(j, path[:i], '{}', TRUE);\n END IF;\n END LOOP;\n END IF;\n RETURN jsonb_set_lax(j, path, val, true);\n\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\n\n\nCREATE OR REPLACE FUNCTION jsonb_include(j jsonb, f jsonb) RETURNS jsonb AS $$\nDECLARE\n includes jsonb := f-> 'include';\n outj jsonb := '{}'::jsonb;\n path text[];\nBEGIN\n IF\n includes IS NULL\n OR jsonb_array_length(includes) = 0\n THEN\n RETURN j;\n ELSE\n includes := includes || '[\"id\",\"collection\"]'::jsonb;\n FOR path IN SELECT explode_dotpaths(includes) LOOP\n outj := jsonb_set_nested(outj, path, j #> path);\n END LOOP;\n END IF;\n RETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_exclude(j jsonb, f jsonb) RETURNS jsonb AS $$\nDECLARE\n excludes jsonb := f-> 'exclude';\n outj jsonb := j;\n path text[];\nBEGIN\n IF\n excludes IS NULL\n OR jsonb_array_length(excludes) = 0\n THEN\n RETURN j;\n ELSE\n FOR path IN SELECT explode_dotpaths(excludes) LOOP\n outj := outj #- path;\n END LOOP;\n END IF;\n RETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_fields(j jsonb, f jsonb DEFAULT '{\"fields\":[]}') RETURNS jsonb AS $$\n SELECT jsonb_exclude(jsonb_include(j, f), f);\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION merge_jsonb(_a jsonb, _b jsonb) RETURNS jsonb AS $$\n SELECT\n CASE\n WHEN _a = '\"𒍟※\"'::jsonb THEN NULL\n WHEN _a IS NULL OR jsonb_typeof(_a) = 'null' THEN _b\n WHEN jsonb_typeof(_a) = 'object' AND jsonb_typeof(_b) = 'object' THEN\n (\n SELECT\n jsonb_strip_nulls(\n jsonb_object_agg(\n key,\n merge_jsonb(a.value, b.value)\n )\n )\n FROM\n jsonb_each(coalesce(_a,'{}'::jsonb)) as a\n FULL JOIN\n jsonb_each(coalesce(_b,'{}'::jsonb)) as b\n USING (key)\n )\n WHEN\n jsonb_typeof(_a) = 'array'\n AND jsonb_typeof(_b) = 'array'\n AND jsonb_array_length(_a) = jsonb_array_length(_b)\n THEN\n (\n SELECT jsonb_agg(m) FROM\n ( SELECT\n merge_jsonb(\n jsonb_array_elements(_a),\n jsonb_array_elements(_b)\n ) as m\n ) as l\n )\n ELSE _a\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION strip_jsonb(_a jsonb, _b jsonb) RETURNS jsonb AS $$\n SELECT\n CASE\n\n WHEN (_a IS NULL OR jsonb_typeof(_a) = 'null') AND _b IS NOT NULL AND jsonb_typeof(_b) != 'null' THEN '\"𒍟※\"'::jsonb\n WHEN _b IS NULL OR jsonb_typeof(_a) = 'null' THEN _a\n WHEN _a = _b AND jsonb_typeof(_a) = 'object' THEN '{}'::jsonb\n WHEN _a = _b THEN NULL\n WHEN jsonb_typeof(_a) = 'object' AND jsonb_typeof(_b) = 'object' THEN\n (\n SELECT\n jsonb_strip_nulls(\n jsonb_object_agg(\n key,\n strip_jsonb(a.value, b.value)\n )\n )\n FROM\n jsonb_each(_a) as a\n FULL JOIN\n jsonb_each(_b) as b\n USING (key)\n )\n WHEN\n jsonb_typeof(_a) = 'array'\n AND jsonb_typeof(_b) = 'array'\n AND jsonb_array_length(_a) = jsonb_array_length(_b)\n THEN\n (\n SELECT jsonb_agg(m) FROM\n ( SELECT\n strip_jsonb(\n jsonb_array_elements(_a),\n jsonb_array_elements(_b)\n ) as m\n ) as l\n )\n ELSE _a\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE;\n/* looks for a geometry in a stac item first from geometry and falling back to bbox */\nCREATE OR REPLACE FUNCTION stac_geom(value jsonb) RETURNS geometry AS $$\nSELECT\n CASE\n WHEN value ? 'intersects' THEN\n ST_GeomFromGeoJSON(value->>'intersects')\n WHEN value ? 'geometry' THEN\n ST_GeomFromGeoJSON(value->>'geometry')\n WHEN value ? 'bbox' THEN\n pgstac.bbox_geom(value->'bbox')\n ELSE NULL\n END as geometry\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION stac_daterange(\n value jsonb\n) RETURNS tstzrange AS $$\nDECLARE\n props jsonb := value;\n dt timestamptz;\n edt timestamptz;\nBEGIN\n IF props ? 'properties' THEN\n props := props->'properties';\n END IF;\n IF\n props ? 'start_datetime'\n AND props->>'start_datetime' IS NOT NULL\n AND props ? 'end_datetime'\n AND props->>'end_datetime' IS NOT NULL\n THEN\n dt := props->>'start_datetime';\n edt := props->>'end_datetime';\n IF dt > edt THEN\n RAISE EXCEPTION 'start_datetime must be < end_datetime';\n END IF;\n ELSE\n dt := props->>'datetime';\n edt := props->>'datetime';\n END IF;\n IF dt is NULL OR edt IS NULL THEN\n RAISE NOTICE 'DT: %, EDT: %', dt, edt;\n RAISE EXCEPTION 'Either datetime (%) or both start_datetime (%) and end_datetime (%) must be set.', props->>'datetime',props->>'start_datetime',props->>'end_datetime';\n END IF;\n RETURN tstzrange(dt, edt, '[]');\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_datetime(value jsonb) RETURNS timestamptz AS $$\n SELECT lower(stac_daterange(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_end_datetime(value jsonb) RETURNS timestamptz AS $$\n SELECT upper(stac_daterange(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\n\nCREATE TABLE IF NOT EXISTS stac_extensions(\n name text PRIMARY KEY,\n url text,\n enbabled_by_default boolean NOT NULL DEFAULT TRUE,\n enableable boolean NOT NULL DEFAULT TRUE\n);\n\nINSERT INTO stac_extensions (name, url) VALUES\n ('fields', 'https://api.stacspec.org/v1.0.0-beta.5/item-search#fields'),\n ('sort','https://api.stacspec.org/v1.0.0-beta.5/item-search#sort'),\n ('context','https://api.stacspec.org/v1.0.0-beta.5/item-search#context'),\n ('filter', 'https://api.stacspec.org/v1.0.0-beta.5/item-search#filter'),\n ('query', 'https://api.stacspec.org/v1.0.0-beta.5/item-search#query')\nON CONFLICT (name) DO UPDATE SET url=EXCLUDED.url;\n\n\n\nCREATE OR REPLACE FUNCTION collection_base_item(content jsonb) RETURNS jsonb AS $$\n SELECT jsonb_build_object(\n 'type', 'Feature',\n 'stac_version', content->'stac_version',\n 'assets', content->'item_assets',\n 'collection', content->'id'\n );\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TYPE partition_trunc_strategy AS ENUM ('year', 'month');\n\nCREATE TABLE IF NOT EXISTS collections (\n key bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,\n id text GENERATED ALWAYS AS (content->>'id') STORED UNIQUE,\n content JSONB NOT NULL,\n base_item jsonb GENERATED ALWAYS AS (pgstac.collection_base_item(content)) STORED,\n partition_trunc partition_trunc_strategy\n);\n\n\nCREATE OR REPLACE FUNCTION collection_base_item(cid text) RETURNS jsonb AS $$\n SELECT pgstac.collection_base_item(content) FROM pgstac.collections WHERE id = cid LIMIT 1;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION table_empty(text) RETURNS boolean AS $$\nDECLARE\n retval boolean;\nBEGIN\n EXECUTE format($q$\n SELECT NOT EXISTS (SELECT 1 FROM %I LIMIT 1)\n $q$,\n $1\n ) INTO retval;\n RETURN retval;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION collections_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n q text;\n partition_name text := format('_items_%s', NEW.key);\n partition_exists boolean := false;\n partition_empty boolean := true;\n err_context text;\n loadtemp boolean := FALSE;\nBEGIN\n RAISE NOTICE 'Collection Trigger. % %', NEW.id, NEW.key;\n SELECT relid::text INTO partition_name\n FROM pg_partition_tree('items')\n WHERE relid::text = partition_name;\n IF FOUND THEN\n partition_exists := true;\n partition_empty := table_empty(partition_name);\n ELSE\n partition_exists := false;\n partition_empty := true;\n partition_name := format('_items_%s', NEW.key);\n END IF;\n IF TG_OP = 'UPDATE' AND NEW.partition_trunc IS DISTINCT FROM OLD.partition_trunc AND partition_empty THEN\n q := format($q$\n DROP TABLE IF EXISTS %I CASCADE;\n $q$,\n partition_name\n );\n EXECUTE q;\n END IF;\n IF TG_OP = 'UPDATE' AND NEW.partition_trunc IS DISTINCT FROM OLD.partition_trunc AND partition_exists AND NOT partition_empty THEN\n q := format($q$\n CREATE TEMP TABLE changepartitionstaging ON COMMIT DROP AS SELECT * FROM %I;\n DROP TABLE IF EXISTS %I CASCADE;\n $q$,\n partition_name,\n partition_name\n );\n EXECUTE q;\n loadtemp := TRUE;\n partition_empty := TRUE;\n partition_exists := FALSE;\n END IF;\n IF TG_OP = 'UPDATE' AND NEW.partition_trunc IS NOT DISTINCT FROM OLD.partition_trunc THEN\n RETURN NEW;\n END IF;\n IF NEW.partition_trunc IS NULL AND partition_empty THEN\n RAISE NOTICE '% % % %',\n partition_name,\n NEW.id,\n concat(partition_name,'_id_idx'),\n partition_name\n ;\n q := format($q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n $q$,\n partition_name,\n NEW.id,\n concat(partition_name,'_id_idx'),\n partition_name\n );\n RAISE NOTICE 'q: %', q;\n BEGIN\n EXECUTE q;\n EXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', partition_name;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\n END;\n\n ALTER TABLE partitions DISABLE TRIGGER partitions_delete_trigger;\n DELETE FROM partitions WHERE collection=NEW.id AND name=partition_name;\n ALTER TABLE partitions ENABLE TRIGGER partitions_delete_trigger;\n\n INSERT INTO partitions (collection, name) VALUES (NEW.id, partition_name);\n ELSIF partition_empty THEN\n q := format($q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L)\n PARTITION BY RANGE (datetime);\n $q$,\n partition_name,\n NEW.id\n );\n RAISE NOTICE 'q: %', q;\n BEGIN\n EXECUTE q;\n EXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', partition_name;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\n END;\n ALTER TABLE partitions DISABLE TRIGGER partitions_delete_trigger;\n DELETE FROM partitions WHERE collection=NEW.id AND name=partition_name;\n ALTER TABLE partitions ENABLE TRIGGER partitions_delete_trigger;\n ELSE\n RAISE EXCEPTION 'Cannot modify partition % unless empty', partition_name;\n END IF;\n IF loadtemp THEN\n RAISE NOTICE 'Moving data into new partitions.';\n q := format($q$\n WITH p AS (\n SELECT\n collection,\n datetime as datetime,\n end_datetime as end_datetime,\n (partition_name(\n collection,\n datetime\n )).partition_name as name\n FROM changepartitionstaging\n )\n INSERT INTO partitions (collection, datetime_range, end_datetime_range)\n SELECT\n collection,\n tstzrange(min(datetime), max(datetime), '[]') as datetime_range,\n tstzrange(min(end_datetime), max(end_datetime), '[]') as end_datetime_range\n FROM p\n GROUP BY collection, name\n ON CONFLICT (name) DO UPDATE SET\n datetime_range = EXCLUDED.datetime_range,\n end_datetime_range = EXCLUDED.end_datetime_range\n ;\n INSERT INTO %I SELECT * FROM changepartitionstaging;\n DROP TABLE IF EXISTS changepartitionstaging;\n $q$,\n partition_name\n );\n EXECUTE q;\n END IF;\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE TRIGGER collections_trigger AFTER INSERT OR UPDATE ON collections FOR EACH ROW\nEXECUTE FUNCTION collections_trigger_func();\n\nCREATE OR REPLACE FUNCTION partition_collection(collection text, strategy partition_trunc_strategy) RETURNS text AS $$\n UPDATE collections SET partition_trunc=strategy WHERE id=collection RETURNING partition_trunc;\n$$ LANGUAGE SQL;\n\nCREATE TABLE IF NOT EXISTS partitions (\n collection text REFERENCES collections(id) ON DELETE CASCADE,\n name text PRIMARY KEY,\n partition_range tstzrange NOT NULL DEFAULT tstzrange('-infinity'::timestamptz,'infinity'::timestamptz, '[]'),\n datetime_range tstzrange,\n end_datetime_range tstzrange,\n CONSTRAINT prange EXCLUDE USING GIST (\n collection WITH =,\n partition_range WITH &&\n )\n) WITH (FILLFACTOR=90);\nCREATE INDEX partitions_range_idx ON partitions USING GIST(partition_range);\n\nCREATE OR REPLACE FUNCTION partitions_delete_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n q text;\nBEGIN\n RAISE NOTICE 'Partition Delete Trigger. %', OLD.name;\n EXECUTE format($q$\n DROP TABLE IF EXISTS %I CASCADE;\n $q$,\n OLD.name\n );\n RAISE NOTICE 'Dropped partition.';\n RETURN OLD;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER partitions_delete_trigger BEFORE DELETE ON partitions FOR EACH ROW\nEXECUTE FUNCTION partitions_delete_trigger_func();\n\nCREATE OR REPLACE FUNCTION partition_name(\n IN collection text,\n IN dt timestamptz,\n OUT partition_name text,\n OUT partition_range tstzrange\n) AS $$\nDECLARE\n c RECORD;\n parent_name text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n parent_name := format('_items_%s', c.key);\n\n\n IF c.partition_trunc = 'year' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYY'));\n ELSIF c.partition_trunc = 'month' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYYMM'));\n ELSE\n partition_name := parent_name;\n partition_range := tstzrange('-infinity'::timestamptz, 'infinity'::timestamptz, '[]');\n END IF;\n IF partition_range IS NULL THEN\n partition_range := tstzrange(\n date_trunc(c.partition_trunc::text, dt),\n date_trunc(c.partition_trunc::text, dt) + concat('1 ', c.partition_trunc)::interval\n );\n END IF;\n RETURN;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\n\nCREATE OR REPLACE FUNCTION partitions_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n q text;\n cq text;\n parent_name text;\n partition_trunc text;\n partition_name text := NEW.name;\n partition_exists boolean := false;\n partition_empty boolean := true;\n partition_range tstzrange;\n datetime_range tstzrange;\n end_datetime_range tstzrange;\n err_context text;\n mindt timestamptz := lower(NEW.datetime_range);\n maxdt timestamptz := upper(NEW.datetime_range);\n minedt timestamptz := lower(NEW.end_datetime_range);\n maxedt timestamptz := upper(NEW.end_datetime_range);\n t_mindt timestamptz;\n t_maxdt timestamptz;\n t_minedt timestamptz;\n t_maxedt timestamptz;\nBEGIN\n RAISE NOTICE 'Partitions Trigger. %', NEW;\n datetime_range := NEW.datetime_range;\n end_datetime_range := NEW.end_datetime_range;\n\n SELECT\n format('_items_%s', key),\n c.partition_trunc::text\n INTO\n parent_name,\n partition_trunc\n FROM pgstac.collections c\n WHERE c.id = NEW.collection;\n SELECT (pgstac.partition_name(NEW.collection, mindt)).* INTO partition_name, partition_range;\n NEW.name := partition_name;\n\n IF partition_range IS NULL OR partition_range = 'empty'::tstzrange THEN\n partition_range := tstzrange('-infinity'::timestamptz, 'infinity'::timestamptz, '[]');\n END IF;\n\n NEW.partition_range := partition_range;\n IF TG_OP = 'UPDATE' THEN\n mindt := least(mindt, lower(OLD.datetime_range));\n maxdt := greatest(maxdt, upper(OLD.datetime_range));\n minedt := least(minedt, lower(OLD.end_datetime_range));\n maxedt := greatest(maxedt, upper(OLD.end_datetime_range));\n NEW.datetime_range := tstzrange(mindt, maxdt, '[]');\n NEW.end_datetime_range := tstzrange(minedt, maxedt, '[]');\n END IF;\n IF TG_OP = 'INSERT' THEN\n\n IF partition_range != tstzrange('-infinity'::timestamptz, 'infinity'::timestamptz, '[]') THEN\n\n RAISE NOTICE '% % %', partition_name, parent_name, partition_range;\n q := format($q$\n CREATE TABLE IF NOT EXISTS %I partition OF %I FOR VALUES FROM (%L) TO (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n $q$,\n partition_name,\n parent_name,\n lower(partition_range),\n upper(partition_range),\n format('%s_pkey', partition_name),\n partition_name\n );\n BEGIN\n EXECUTE q;\n EXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', partition_name;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\n END;\n END IF;\n\n END IF;\n\n -- Update constraints\n EXECUTE format($q$\n SELECT\n min(datetime),\n max(datetime),\n min(end_datetime),\n max(end_datetime)\n FROM %I;\n $q$, partition_name)\n INTO t_mindt, t_maxdt, t_minedt, t_maxedt;\n mindt := least(mindt, t_mindt);\n maxdt := greatest(maxdt, t_maxdt);\n minedt := least(mindt, minedt, t_minedt);\n maxedt := greatest(maxdt, maxedt, t_maxedt);\n\n mindt := date_trunc(coalesce(partition_trunc, 'year'), mindt);\n maxdt := date_trunc(coalesce(partition_trunc, 'year'), maxdt - '1 second'::interval) + concat('1 ',coalesce(partition_trunc, 'year'))::interval;\n minedt := date_trunc(coalesce(partition_trunc, 'year'), minedt);\n maxedt := date_trunc(coalesce(partition_trunc, 'year'), maxedt - '1 second'::interval) + concat('1 ',coalesce(partition_trunc, 'year'))::interval;\n\n\n IF mindt IS NOT NULL AND maxdt IS NOT NULL AND minedt IS NOT NULL AND maxedt IS NOT NULL THEN\n NEW.datetime_range := tstzrange(mindt, maxdt, '[]');\n NEW.end_datetime_range := tstzrange(minedt, maxedt, '[]');\n IF\n TG_OP='UPDATE'\n AND OLD.datetime_range @> NEW.datetime_range\n AND OLD.end_datetime_range @> NEW.end_datetime_range\n THEN\n RAISE NOTICE 'Range unchanged, not updating constraints.';\n ELSE\n\n RAISE NOTICE '\n SETTING CONSTRAINTS\n mindt: %, maxdt: %\n minedt: %, maxedt: %\n ', mindt, maxdt, minedt, maxedt;\n IF partition_trunc IS NULL THEN\n cq := format($q$\n ALTER TABLE %7$I\n DROP CONSTRAINT IF EXISTS %1$I,\n DROP CONSTRAINT IF EXISTS %2$I,\n ADD CONSTRAINT %1$I\n CHECK (\n (datetime >= %3$L)\n AND (datetime <= %4$L)\n AND (end_datetime >= %5$L)\n AND (end_datetime <= %6$L)\n ) NOT VALID\n ;\n ALTER TABLE %7$I\n VALIDATE CONSTRAINT %1$I;\n $q$,\n format('%s_dt', partition_name),\n format('%s_edt', partition_name),\n mindt,\n maxdt,\n minedt,\n maxedt,\n partition_name\n );\n ELSE\n cq := format($q$\n ALTER TABLE %5$I\n DROP CONSTRAINT IF EXISTS %1$I,\n DROP CONSTRAINT IF EXISTS %2$I,\n ADD CONSTRAINT %2$I\n CHECK ((end_datetime >= %3$L) AND (end_datetime <= %4$L)) NOT VALID\n ;\n ALTER TABLE %5$I\n VALIDATE CONSTRAINT %2$I;\n $q$,\n format('%s_dt', partition_name),\n format('%s_edt', partition_name),\n minedt,\n maxedt,\n partition_name\n );\n\n END IF;\n RAISE NOTICE 'Altering Constraints. %', cq;\n EXECUTE cq;\n END IF;\n ELSE\n NEW.datetime_range = NULL;\n NEW.end_datetime_range = NULL;\n\n cq := format($q$\n ALTER TABLE %3$I\n DROP CONSTRAINT IF EXISTS %1$I,\n DROP CONSTRAINT IF EXISTS %2$I,\n ADD CONSTRAINT %1$I\n CHECK ((datetime IS NULL AND end_datetime IS NULL)) NOT VALID\n ;\n ALTER TABLE %3$I\n VALIDATE CONSTRAINT %1$I;\n $q$,\n format('%s_dt', partition_name),\n format('%s_edt', partition_name),\n partition_name\n );\n EXECUTE cq;\n END IF;\n\n RETURN NEW;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER partitions_trigger BEFORE INSERT OR UPDATE ON partitions FOR EACH ROW\nEXECUTE FUNCTION partitions_trigger_func();\n\n\nCREATE OR REPLACE FUNCTION create_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection(data jsonb) RETURNS VOID AS $$\nDECLARE\n out collections%ROWTYPE;\nBEGIN\n UPDATE collections SET content=data WHERE id = data->>'id' RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ON CONFLICT (id) DO\n UPDATE\n SET content=EXCLUDED.content\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION delete_collection(_id text) RETURNS VOID AS $$\nDECLARE\n out collections%ROWTYPE;\nBEGIN\n DELETE FROM collections WHERE id = _id RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION get_collection(id text) RETURNS jsonb AS $$\n SELECT content FROM collections\n WHERE id=$1\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION all_collections() RETURNS jsonb AS $$\n SELECT jsonb_agg(content) FROM collections;\n;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\nCREATE TABLE queryables (\n id bigint GENERATED ALWAYS AS identity PRIMARY KEY,\n name text UNIQUE NOT NULL,\n collection_ids text[], -- used to determine what partitions to create indexes on\n definition jsonb,\n property_path text,\n property_wrapper text,\n property_index_type text\n);\nCREATE INDEX queryables_name_idx ON queryables (name);\nCREATE INDEX queryables_property_wrapper_idx ON queryables (property_wrapper);\n\n\nINSERT INTO queryables (name, definition) VALUES\n('id', '{\"title\": \"Item ID\",\"description\": \"Item identifier\",\"$ref\": \"https://schemas.stacspec.org/v1.0.0/item-spec/json-schema/item.json#/definitions/core/allOf/2/properties/id\"}'),\n('datetime','{\"description\": \"Datetime\",\"type\": \"string\",\"title\": \"Acquired\",\"format\": \"date-time\",\"pattern\": \"(\\\\+00:00|Z)$\"}'),\n('geometry', '{\"title\": \"Item Geometry\",\"description\": \"Item Geometry\",\"$ref\": \"https://geojson.org/schema/Feature.json\"}')\nON CONFLICT DO NOTHING;\n\nINSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n('eo:cloud_cover','{\"$ref\": \"https://stac-extensions.github.io/eo/v1.0.0/schema.json#/definitions/fieldsproperties/eo:cloud_cover\"}','to_int','BTREE')\nON CONFLICT DO NOTHING;\n\nCREATE OR REPLACE FUNCTION array_to_path(arr text[]) RETURNS text AS $$\n SELECT string_agg(\n quote_literal(v),\n '->'\n ) FROM unnest(arr) v;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\n\n\nCREATE OR REPLACE FUNCTION queryable(\n IN dotpath text,\n OUT path text,\n OUT expression text,\n OUT wrapper text,\n OUT nulled_wrapper text\n) AS $$\nDECLARE\n q RECORD;\n path_elements text[];\nBEGIN\n IF dotpath IN ('id', 'geometry', 'datetime', 'end_datetime', 'collection') THEN\n path := dotpath;\n expression := dotpath;\n wrapper := NULL;\n RETURN;\n END IF;\n SELECT * INTO q FROM queryables\n WHERE\n name=dotpath\n OR name = 'properties.' || dotpath\n OR name = replace(dotpath, 'properties.', '')\n ;\n IF q.property_wrapper IS NULL THEN\n IF q.definition->>'type' = 'number' THEN\n wrapper := 'to_float';\n nulled_wrapper := wrapper;\n ELSIF q.definition->>'format' = 'date-time' THEN\n wrapper := 'to_tstz';\n nulled_wrapper := wrapper;\n ELSE\n nulled_wrapper := NULL;\n wrapper := 'to_text';\n END IF;\n ELSE\n wrapper := q.property_wrapper;\n nulled_wrapper := wrapper;\n END IF;\n IF q.property_path IS NOT NULL THEN\n path := q.property_path;\n ELSE\n path_elements := string_to_array(dotpath, '.');\n IF path_elements[1] IN ('links', 'assets', 'stac_version', 'stac_extensions') THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSIF path_elements[1] = 'properties' THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSE\n path := format($F$content->'properties'->%s$F$, array_to_path(path_elements));\n END IF;\n END IF;\n expression := format('%I(%s)', wrapper, path);\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\nCREATE OR REPLACE FUNCTION create_queryable_indexes() RETURNS VOID AS $$\nDECLARE\n queryable RECORD;\n q text;\nBEGIN\n FOR queryable IN\n SELECT\n queryables.id as qid,\n CASE WHEN collections.key IS NULL THEN 'items' ELSE format('_items_%s',collections.key) END AS part,\n property_index_type,\n expression\n FROM\n queryables\n LEFT JOIN collections ON (collections.id = ANY (queryables.collection_ids))\n JOIN LATERAL queryable(queryables.name) ON (queryables.property_index_type IS NOT NULL)\n LOOP\n q := format(\n $q$\n CREATE INDEX IF NOT EXISTS %I ON %I USING %s ((%s));\n $q$,\n format('%s_%s_idx', queryable.part, queryable.qid),\n queryable.part,\n COALESCE(queryable.property_index_type, 'to_text'),\n queryable.expression\n );\n RAISE NOTICE '%',q;\n EXECUTE q;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION queryables_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\nBEGIN\nPERFORM create_queryable_indexes();\nRETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER queryables_trigger AFTER INSERT OR UPDATE ON queryables\nFOR EACH STATEMENT EXECUTE PROCEDURE queryables_trigger_func();\n\nCREATE TRIGGER queryables_collection_trigger AFTER INSERT OR UPDATE ON collections\nFOR EACH STATEMENT EXECUTE PROCEDURE queryables_trigger_func();\n\nCREATE OR REPLACE FUNCTION get_queryables(_collection_ids text[] DEFAULT NULL) RETURNS jsonb AS $$\n SELECT\n jsonb_build_object(\n '$schema', 'http://json-schema.org/draft-07/schema#',\n '$id', 'https://example.org/queryables',\n 'type', 'object',\n 'title', 'Stac Queryables.',\n 'properties', jsonb_object_agg(\n name,\n definition\n )\n )\n FROM queryables\n WHERE\n _collection_ids IS NULL OR\n cardinality(_collection_ids) = 0 OR\n collection_ids IS NULL OR\n _collection_ids && collection_ids\n ;\n$$ LANGUAGE SQL STABLE;\n\nCREATE OR REPLACE FUNCTION get_queryables(_collection text DEFAULT NULL) RETURNS jsonb AS $$\n SELECT\n CASE\n WHEN _collection IS NULL THEN get_queryables(NULL::text[])\n ELSE get_queryables(ARRAY[_collection])\n END\n ;\n$$ LANGUAGE SQL;\n\n\nCREATE OR REPLACE FUNCTION missing_queryables(_collection text, _tablesample int DEFAULT 5) RETURNS TABLE(collection text, name text, definition jsonb, property_wrapper text) AS $$\nDECLARE\n q text;\n _partition text;\n explain_json json;\n psize bigint;\nBEGIN\n SELECT format('_items_%s', key) INTO _partition FROM collections WHERE id=_collection;\n\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM %I;', _partition)\n INTO explain_json;\n psize := explain_json->0->'Plan'->'Plan Rows';\n IF _tablesample * .01 * psize < 10 THEN\n _tablesample := 100;\n END IF;\n RAISE NOTICE 'Using tablesample % to find missing queryables from % % that has ~% rows', _tablesample, _collection, _partition, psize;\n\n q := format(\n $q$\n WITH q AS (\n SELECT * FROM queryables\n WHERE\n collection_ids IS NULL\n OR %L = ANY(collection_ids)\n ), t AS (\n SELECT\n content->'properties' AS properties\n FROM\n %I\n TABLESAMPLE SYSTEM(%L)\n ), p AS (\n SELECT DISTINCT ON (key)\n key,\n value\n FROM t\n JOIN LATERAL jsonb_each(properties) ON TRUE\n LEFT JOIN q ON (q.name=key)\n WHERE q.definition IS NULL\n )\n SELECT\n %L,\n key,\n jsonb_build_object('type',jsonb_typeof(value)) as definition,\n CASE jsonb_typeof(value)\n WHEN 'number' THEN 'to_float'\n WHEN 'array' THEN 'to_text_array'\n ELSE 'to_text'\n END\n FROM p;\n $q$,\n _collection,\n _partition,\n _tablesample,\n _collection\n );\n RETURN QUERY EXECUTE q;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION missing_queryables(_tablesample int DEFAULT 5) RETURNS TABLE(collection_ids text[], name text, definition jsonb, property_wrapper text) AS $$\n SELECT\n array_agg(collection),\n name,\n definition,\n property_wrapper\n FROM\n collections\n JOIN LATERAL\n missing_queryables(id, _tablesample) c\n ON TRUE\n GROUP BY\n 2,3,4\n ORDER BY 2,1\n ;\n$$ LANGUAGE SQL;\nCREATE OR REPLACE FUNCTION parse_dtrange(\n _indate jsonb,\n relative_base timestamptz DEFAULT date_trunc('hour', CURRENT_TIMESTAMP)\n) RETURNS tstzrange AS $$\nDECLARE\n timestrs text[];\n s timestamptz;\n e timestamptz;\nBEGIN\n timestrs :=\n CASE\n WHEN _indate ? 'timestamp' THEN\n ARRAY[_indate->>'timestamp']\n WHEN _indate ? 'interval' THEN\n to_text_array(_indate->'interval')\n WHEN jsonb_typeof(_indate) = 'array' THEN\n to_text_array(_indate)\n ELSE\n regexp_split_to_array(\n _indate->>0,\n '/'\n )\n END;\n RAISE NOTICE 'TIMESTRS %', timestrs;\n IF cardinality(timestrs) = 1 THEN\n IF timestrs[1] ILIKE 'P%' THEN\n RETURN tstzrange(relative_base - upper(timestrs[1])::interval, relative_base, '[)');\n END IF;\n s := timestrs[1]::timestamptz;\n RETURN tstzrange(s, s, '[]');\n END IF;\n\n IF cardinality(timestrs) != 2 THEN\n RAISE EXCEPTION 'Timestamp cannot have more than 2 values';\n END IF;\n\n IF timestrs[1] = '..' THEN\n s := '-infinity'::timestamptz;\n e := timestrs[2]::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] = '..' THEN\n s := timestrs[1]::timestamptz;\n e := 'infinity'::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[1] ILIKE 'P%' AND timestrs[2] NOT ILIKE 'P%' THEN\n e := timestrs[2]::timestamptz;\n s := e - upper(timestrs[1])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] ILIKE 'P%' AND timestrs[1] NOT ILIKE 'P%' THEN\n s := timestrs[1]::timestamptz;\n e := s + upper(timestrs[2])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n s := timestrs[1]::timestamptz;\n e := timestrs[2]::timestamptz;\n\n RETURN tstzrange(s,e,'[)');\n\n RETURN NULL;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT PARALLEL SAFE SET TIME ZONE 'UTC';\n\nCREATE OR REPLACE FUNCTION parse_dtrange(\n _indate text,\n relative_base timestamptz DEFAULT CURRENT_TIMESTAMP\n) RETURNS tstzrange AS $$\n SELECT parse_dtrange(to_jsonb(_indate), relative_base);\n$$ LANGUAGE SQL STABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION temporal_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\n ll text := 'datetime';\n lh text := 'end_datetime';\n rrange tstzrange;\n rl text;\n rh text;\n outq text;\nBEGIN\n rrange := parse_dtrange(args->1);\n RAISE NOTICE 'Constructing temporal query OP: %, ARGS: %, RRANGE: %', op, args, rrange;\n op := lower(op);\n rl := format('%L::timestamptz', lower(rrange));\n rh := format('%L::timestamptz', upper(rrange));\n outq := CASE op\n WHEN 't_before' THEN 'lh < rl'\n WHEN 't_after' THEN 'll > rh'\n WHEN 't_meets' THEN 'lh = rl'\n WHEN 't_metby' THEN 'll = rh'\n WHEN 't_overlaps' THEN 'll < rl AND rl < lh < rh'\n WHEN 't_overlappedby' THEN 'rl < ll < rh AND lh > rh'\n WHEN 't_starts' THEN 'll = rl AND lh < rh'\n WHEN 't_startedby' THEN 'll = rl AND lh > rh'\n WHEN 't_during' THEN 'll > rl AND lh < rh'\n WHEN 't_contains' THEN 'll < rl AND lh > rh'\n WHEN 't_finishes' THEN 'll > rl AND lh = rh'\n WHEN 't_finishedby' THEN 'll < rl AND lh = rh'\n WHEN 't_equals' THEN 'll = rl AND lh = rh'\n WHEN 't_disjoint' THEN 'NOT (ll <= rh AND lh >= rl)'\n WHEN 't_intersects' THEN 'll <= rh AND lh >= rl'\n WHEN 'anyinteracts' THEN 'll <= rh AND lh >= rl'\n END;\n outq := regexp_replace(outq, '\\mll\\M', ll);\n outq := regexp_replace(outq, '\\mlh\\M', lh);\n outq := regexp_replace(outq, '\\mrl\\M', rl);\n outq := regexp_replace(outq, '\\mrh\\M', rh);\n outq := format('(%s)', outq);\n RETURN outq;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\n\n\nCREATE OR REPLACE FUNCTION spatial_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\n geom text;\n j jsonb := args->1;\nBEGIN\n op := lower(op);\n RAISE NOTICE 'Constructing spatial query OP: %, ARGS: %', op, args;\n IF op NOT IN ('s_equals','s_disjoint','s_touches','s_within','s_overlaps','s_crosses','s_intersects','intersects','s_contains') THEN\n RAISE EXCEPTION 'Spatial Operator % Not Supported', op;\n END IF;\n op := regexp_replace(op, '^s_', 'st_');\n IF op = 'intersects' THEN\n op := 'st_intersects';\n END IF;\n -- Convert geometry to WKB string\n IF j ? 'type' AND j ? 'coordinates' THEN\n geom := st_geomfromgeojson(j)::text;\n ELSIF jsonb_typeof(j) = 'array' THEN\n geom := bbox_geom(j)::text;\n END IF;\n\n RETURN format('%s(geometry, %L::geometry)', op, geom);\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION query_to_cql2(q jsonb) RETURNS jsonb AS $$\n-- Translates anything passed in through the deprecated \"query\" into equivalent CQL2\nWITH t AS (\n SELECT key as property, value as ops\n FROM jsonb_each(q)\n), t2 AS (\n SELECT property, (jsonb_each(ops)).*\n FROM t WHERE jsonb_typeof(ops) = 'object'\n UNION ALL\n SELECT property, 'eq', ops\n FROM t WHERE jsonb_typeof(ops) != 'object'\n)\nSELECT\n jsonb_strip_nulls(jsonb_build_object(\n 'op', 'and',\n 'args', jsonb_agg(\n jsonb_build_object(\n 'op', key,\n 'args', jsonb_build_array(\n jsonb_build_object('property',property),\n value\n )\n )\n )\n )\n) as qcql FROM t2\n;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\nCREATE OR REPLACE FUNCTION cql1_to_cql2(j jsonb) RETURNS jsonb AS $$\nDECLARE\n args jsonb;\n ret jsonb;\nBEGIN\n RAISE NOTICE 'CQL1_TO_CQL2: %', j;\n IF j ? 'filter' THEN\n RETURN cql1_to_cql2(j->'filter');\n END IF;\n IF j ? 'property' THEN\n RETURN j;\n END IF;\n IF jsonb_typeof(j) = 'array' THEN\n SELECT jsonb_agg(cql1_to_cql2(el)) INTO args FROM jsonb_array_elements(j) el;\n RETURN args;\n END IF;\n IF jsonb_typeof(j) = 'number' THEN\n RETURN j;\n END IF;\n IF jsonb_typeof(j) = 'string' THEN\n RETURN j;\n END IF;\n\n IF jsonb_typeof(j) = 'object' THEN\n SELECT jsonb_build_object(\n 'op', key,\n 'args', cql1_to_cql2(value)\n ) INTO ret\n FROM jsonb_each(j)\n WHERE j IS NOT NULL;\n RETURN ret;\n END IF;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE STRICT;\n\nCREATE TABLE cql2_ops (\n op text PRIMARY KEY,\n template text,\n types text[]\n);\nINSERT INTO cql2_ops (op, template, types) VALUES\n ('eq', '%s = %s', NULL),\n ('neq', '%s != %s', NULL),\n ('ne', '%s != %s', NULL),\n ('!=', '%s != %s', NULL),\n ('<>', '%s != %s', NULL),\n ('lt', '%s < %s', NULL),\n ('lte', '%s <= %s', NULL),\n ('gt', '%s > %s', NULL),\n ('gte', '%s >= %s', NULL),\n ('le', '%s <= %s', NULL),\n ('ge', '%s >= %s', NULL),\n ('=', '%s = %s', NULL),\n ('<', '%s < %s', NULL),\n ('<=', '%s <= %s', NULL),\n ('>', '%s > %s', NULL),\n ('>=', '%s >= %s', NULL),\n ('like', '%s LIKE %s', NULL),\n ('ilike', '%s ILIKE %s', NULL),\n ('+', '%s + %s', NULL),\n ('-', '%s - %s', NULL),\n ('*', '%s * %s', NULL),\n ('/', '%s / %s', NULL),\n ('not', 'NOT (%s)', NULL),\n ('between', '%s BETWEEN %s AND %s', NULL),\n ('isnull', '%s IS NULL', NULL),\n ('upper', 'upper(%s)', NULL),\n ('lower', 'lower(%s)', NULL)\nON CONFLICT (op) DO UPDATE\n SET\n template = EXCLUDED.template\n;\n\n\nCREATE OR REPLACE FUNCTION cql2_query(j jsonb, wrapper text DEFAULT NULL) RETURNS text AS $$\n#variable_conflict use_variable\nDECLARE\n args jsonb := j->'args';\n arg jsonb;\n op text := lower(j->>'op');\n cql2op RECORD;\n literal text;\n _wrapper text;\n leftarg text;\n rightarg text;\nBEGIN\n IF j IS NULL OR (op IS NOT NULL AND args IS NULL) THEN\n RETURN NULL;\n END IF;\n RAISE NOTICE 'CQL2_QUERY: %', j;\n IF j ? 'filter' THEN\n RETURN cql2_query(j->'filter');\n END IF;\n\n IF j ? 'upper' THEN\n RETURN cql2_query(jsonb_build_object('op', 'upper', 'args', j->'upper'));\n END IF;\n\n IF j ? 'lower' THEN\n RETURN cql2_query(jsonb_build_object('op', 'lower', 'args', j->'lower'));\n END IF;\n\n -- Temporal Query\n IF op ilike 't_%' or op = 'anyinteracts' THEN\n RETURN temporal_op_query(op, args);\n END IF;\n\n -- If property is a timestamp convert it to text to use with\n -- general operators\n IF j ? 'timestamp' THEN\n RETURN format('%L::timestamptz', to_tstz(j->'timestamp'));\n END IF;\n IF j ? 'interval' THEN\n RAISE EXCEPTION 'Please use temporal operators when using intervals.';\n RETURN NONE;\n END IF;\n\n -- Spatial Query\n IF op ilike 's_%' or op = 'intersects' THEN\n RETURN spatial_op_query(op, args);\n END IF;\n\n IF op IN ('a_equals','a_contains','a_contained_by','a_overlaps') THEN\n IF args->0 ? 'property' THEN\n leftarg := format('to_text_array(%s)', (queryable(args->0->>'property')).path);\n END IF;\n IF args->1 ? 'property' THEN\n rightarg := format('to_text_array(%s)', (queryable(args->1->>'property')).path);\n END IF;\n RETURN FORMAT(\n '%s %s %s',\n COALESCE(leftarg, quote_literal(to_text_array(args->0))),\n CASE op\n WHEN 'a_equals' THEN '='\n WHEN 'a_contains' THEN '@>'\n WHEN 'a_contained_by' THEN '<@'\n WHEN 'a_overlaps' THEN '&&'\n END,\n COALESCE(rightarg, quote_literal(to_text_array(args->1)))\n );\n END IF;\n\n IF op = 'in' THEN\n RAISE NOTICE 'IN : % % %', args, jsonb_build_array(args->0), args->1;\n args := jsonb_build_array(args->0) || (args->1);\n RAISE NOTICE 'IN2 : %', args;\n END IF;\n\n\n\n IF op = 'between' THEN\n args = jsonb_build_array(\n args->0,\n args->1->0,\n args->1->1\n );\n END IF;\n\n -- Make sure that args is an array and run cql2_query on\n -- each element of the array\n RAISE NOTICE 'ARGS PRE: %', args;\n IF j ? 'args' THEN\n IF jsonb_typeof(args) != 'array' THEN\n args := jsonb_build_array(args);\n END IF;\n\n IF jsonb_path_exists(args, '$[*] ? (@.property == \"id\" || @.property == \"datetime\" || @.property == \"end_datetime\" || @.property == \"collection\")') THEN\n wrapper := NULL;\n ELSE\n -- if any of the arguments are a property, try to get the property_wrapper\n FOR arg IN SELECT jsonb_path_query(args, '$[*] ? (@.property != null)') LOOP\n RAISE NOTICE 'Arg: %', arg;\n wrapper := (queryable(arg->>'property')).nulled_wrapper;\n RAISE NOTICE 'Property: %, Wrapper: %', arg, wrapper;\n IF wrapper IS NOT NULL THEN\n EXIT;\n END IF;\n END LOOP;\n\n -- if the property was not in queryables, see if any args were numbers\n IF\n wrapper IS NULL\n AND jsonb_path_exists(args, '$[*] ? (@.type()==\"number\")')\n THEN\n wrapper := 'to_float';\n END IF;\n wrapper := coalesce(wrapper, 'to_text');\n END IF;\n\n SELECT jsonb_agg(cql2_query(a, wrapper))\n INTO args\n FROM jsonb_array_elements(args) a;\n END IF;\n RAISE NOTICE 'ARGS: %', args;\n\n IF op IN ('and', 'or') THEN\n RETURN\n format(\n '(%s)',\n array_to_string(to_text_array(args), format(' %s ', upper(op)))\n );\n END IF;\n\n IF op = 'in' THEN\n RAISE NOTICE 'IN -- % %', args->0, to_text(args->0);\n RETURN format(\n '%s IN (%s)',\n to_text(args->0),\n array_to_string((to_text_array(args))[2:], ',')\n );\n END IF;\n\n -- Look up template from cql2_ops\n IF j ? 'op' THEN\n SELECT * INTO cql2op FROM cql2_ops WHERE cql2_ops.op ilike op;\n IF FOUND THEN\n -- If specific index set in queryables for a property cast other arguments to that type\n\n RETURN format(\n cql2op.template,\n VARIADIC (to_text_array(args))\n );\n ELSE\n RAISE EXCEPTION 'Operator % Not Supported.', op;\n END IF;\n END IF;\n\n\n IF wrapper IS NOT NULL THEN\n RAISE NOTICE 'Wrapping % with %', j, wrapper;\n IF j ? 'property' THEN\n RETURN format('%I(%s)', wrapper, (queryable(j->>'property')).path);\n ELSE\n RETURN format('%I(%L)', wrapper, j);\n END IF;\n ELSIF j ? 'property' THEN\n RETURN quote_ident(j->>'property');\n END IF;\n\n RETURN quote_literal(to_text(j));\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION paging_dtrange(\n j jsonb\n) RETURNS tstzrange AS $$\nDECLARE\n op text;\n filter jsonb := j->'filter';\n dtrange tstzrange := tstzrange('-infinity'::timestamptz,'infinity'::timestamptz);\n sdate timestamptz := '-infinity'::timestamptz;\n edate timestamptz := 'infinity'::timestamptz;\n jpitem jsonb;\nBEGIN\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n END IF;\n IF NOT (filter @? '$.**.op ? (@ == \"or\" || @ == \"not\")') THEN\n FOR jpitem IN SELECT j FROM jsonb_path_query(filter,'strict $.** ? (@.args[*].property == \"datetime\")'::jsonpath) j LOOP\n op := lower(jpitem->>'op');\n dtrange := parse_dtrange(jpitem->'args'->1);\n IF op IN ('<=', 'lt', 'lte', '<', 'le', 't_before') THEN\n sdate := greatest(sdate,'-infinity');\n edate := least(edate, upper(dtrange));\n ELSIF op IN ('>=', '>', 'gt', 'gte', 'ge', 't_after') THEN\n edate := least(edate, 'infinity');\n sdate := greatest(sdate, lower(dtrange));\n ELSIF op IN ('=', 'eq') THEN\n edate := least(edate, upper(dtrange));\n sdate := greatest(sdate, lower(dtrange));\n END IF;\n RAISE NOTICE '2 OP: %, ARGS: %, DTRANGE: %, SDATE: %, EDATE: %', op, jpitem->'args'->1, dtrange, sdate, edate;\n END LOOP;\n END IF;\n IF sdate > edate THEN\n RETURN 'empty'::tstzrange;\n END IF;\n RETURN tstzrange(sdate,edate, '[]');\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT SET TIME ZONE 'UTC';\n\nCREATE OR REPLACE FUNCTION paging_collections(\n IN j jsonb\n) RETURNS text[] AS $$\nDECLARE\n filter jsonb := j->'filter';\n jpitem jsonb;\n op text;\n args jsonb;\n arg jsonb;\n collections text[];\nBEGIN\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n END IF;\n IF NOT (filter @? '$.**.op ? (@ == \"or\" || @ == \"not\")') THEN\n FOR jpitem IN SELECT j FROM jsonb_path_query(filter,'strict $.** ? (@.args[*].property == \"collection\")'::jsonpath) j LOOP\n RAISE NOTICE 'JPITEM: %', jpitem;\n op := jpitem->>'op';\n args := jpitem->'args';\n IF op IN ('=', 'eq', 'in') THEN\n FOR arg IN SELECT a FROM jsonb_array_elements(args) a LOOP\n IF jsonb_typeof(arg) IN ('string', 'array') THEN\n RAISE NOTICE 'arg: %, collections: %', arg, collections;\n IF collections IS NULL OR collections = '{}'::text[] THEN\n collections := to_text_array(arg);\n ELSE\n collections := array_intersection(collections, to_text_array(arg));\n END IF;\n END IF;\n END LOOP;\n END IF;\n END LOOP;\n END IF;\n IF collections = '{}'::text[] THEN\n RETURN NULL;\n END IF;\n RETURN collections;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\nCREATE TABLE items (\n id text NOT NULL,\n geometry geometry NOT NULL,\n collection text NOT NULL,\n datetime timestamptz NOT NULL,\n end_datetime timestamptz NOT NULL,\n content JSONB NOT NULL\n)\nPARTITION BY LIST (collection)\n;\n\nCREATE INDEX \"datetime_idx\" ON items USING BTREE (datetime DESC, end_datetime ASC);\nCREATE INDEX \"geometry_idx\" ON items USING GIST (geometry);\n\nCREATE STATISTICS datetime_stats (dependencies) on datetime, end_datetime from items;\n\n\nALTER TABLE items ADD CONSTRAINT items_collections_fk FOREIGN KEY (collection) REFERENCES collections(id) ON DELETE CASCADE DEFERRABLE;\n\n\nCREATE OR REPLACE FUNCTION content_slim(_item jsonb) RETURNS jsonb AS $$\n SELECT strip_jsonb(_item - '{id,geometry,collection,type}'::text[], collection_base_item(_item->>'collection')) - '{id,geometry,collection,type}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_dehydrate(content jsonb) RETURNS items AS $$\n SELECT\n content->>'id' as id,\n stac_geom(content) as geometry,\n content->>'collection' as collection,\n stac_datetime(content) as datetime,\n stac_end_datetime(content) as end_datetime,\n content_slim(content) as content\n ;\n$$ LANGUAGE SQL STABLE;\n\nCREATE OR REPLACE FUNCTION include_field(f text, fields jsonb DEFAULT '{}'::jsonb) RETURNS boolean AS $$\nDECLARE\n includes jsonb := fields->'include';\n excludes jsonb := fields->'exclude';\nBEGIN\n IF f IS NULL THEN\n RETURN NULL;\n END IF;\n\n\n IF\n jsonb_typeof(excludes) = 'array'\n AND jsonb_array_length(excludes)>0\n AND excludes ? f\n THEN\n RETURN FALSE;\n END IF;\n\n IF\n (\n jsonb_typeof(includes) = 'array'\n AND jsonb_array_length(includes) > 0\n AND includes ? f\n ) OR\n (\n includes IS NULL\n OR jsonb_typeof(includes) = 'null'\n OR jsonb_array_length(includes) = 0\n )\n THEN\n RETURN TRUE;\n END IF;\n\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nDROP FUNCTION IF EXISTS content_hydrate(jsonb, jsonb, jsonb);\nCREATE OR REPLACE FUNCTION content_hydrate(\n _item jsonb,\n _base_item jsonb,\n fields jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\n SELECT merge_jsonb(\n jsonb_fields(_item, fields),\n jsonb_fields(_base_item, fields)\n );\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION content_hydrate(_item items, _collection collections, fields jsonb DEFAULT '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\n content jsonb;\n base_item jsonb := _collection.base_item;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry, 20)::jsonb;\n END IF;\n output := content_hydrate(\n jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'collection', _item.collection,\n 'type', 'Feature'\n ) || _item.content,\n _collection.base_item,\n fields\n );\n\n RETURN output;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_nonhydrated(\n _item items,\n fields jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry, 20)::jsonb;\n END IF;\n output := jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'collection', _item.collection,\n 'type', 'Feature'\n ) || _item.content;\n RETURN output;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_hydrate(_item items, fields jsonb DEFAULT '{}'::jsonb) RETURNS jsonb AS $$\n SELECT content_hydrate(\n _item,\n (SELECT c FROM collections c WHERE id=_item.collection LIMIT 1),\n fields\n );\n$$ LANGUAGE SQL STABLE;\n\n\nCREATE UNLOGGED TABLE items_staging (\n content JSONB NOT NULL\n);\nCREATE UNLOGGED TABLE items_staging_ignore (\n content JSONB NOT NULL\n);\nCREATE UNLOGGED TABLE items_staging_upsert (\n content JSONB NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION items_staging_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p record;\n _partitions text[];\n ts timestamptz := clock_timestamp();\nBEGIN\n RAISE NOTICE 'Creating Partitions. %', clock_timestamp() - ts;\n WITH ranges AS (\n SELECT\n n.content->>'collection' as collection,\n stac_daterange(n.content->'properties') as dtr\n FROM newdata n\n ), p AS (\n SELECT\n collection,\n lower(dtr) as datetime,\n upper(dtr) as end_datetime,\n (partition_name(\n collection,\n lower(dtr)\n )).partition_name as name\n FROM ranges\n )\n INSERT INTO partitions (collection, datetime_range, end_datetime_range)\n SELECT\n collection,\n tstzrange(min(datetime), max(datetime), '[]') as datetime_range,\n tstzrange(min(end_datetime), max(end_datetime), '[]') as end_datetime_range\n FROM p\n GROUP BY collection, name\n ON CONFLICT (name) DO UPDATE SET\n datetime_range = EXCLUDED.datetime_range,\n end_datetime_range = EXCLUDED.end_datetime_range\n ;\n\n RAISE NOTICE 'Doing the insert. %', clock_timestamp() - ts;\n IF TG_TABLE_NAME = 'items_staging' THEN\n INSERT INTO items\n SELECT\n (content_dehydrate(content)).*\n FROM newdata;\n DELETE FROM items_staging;\n ELSIF TG_TABLE_NAME = 'items_staging_ignore' THEN\n INSERT INTO items\n SELECT\n (content_dehydrate(content)).*\n FROM newdata\n ON CONFLICT DO NOTHING;\n DELETE FROM items_staging_ignore;\n ELSIF TG_TABLE_NAME = 'items_staging_upsert' THEN\n WITH staging_formatted AS (\n SELECT (content_dehydrate(content)).* FROM newdata\n ), deletes AS (\n DELETE FROM items i USING staging_formatted s\n WHERE\n i.id = s.id\n AND i.collection = s.collection\n AND i IS DISTINCT FROM s\n RETURNING i.id, i.collection\n )\n INSERT INTO items\n SELECT s.* FROM\n staging_formatted s\n JOIN deletes d\n USING (id, collection);\n DELETE FROM items_staging_upsert;\n END IF;\n RAISE NOTICE 'Done. %', clock_timestamp() - ts;\n\n RETURN NULL;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER items_staging_insert_trigger AFTER INSERT ON items_staging REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\nCREATE TRIGGER items_staging_insert_ignore_trigger AFTER INSERT ON items_staging_ignore REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\nCREATE TRIGGER items_staging_insert_upsert_trigger AFTER INSERT ON items_staging_upsert REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\n\n\n\nCREATE OR REPLACE FUNCTION item_by_id(_id text, _collection text DEFAULT NULL) RETURNS items AS\n$$\nDECLARE\n i items%ROWTYPE;\nBEGIN\n SELECT * INTO i FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection) LIMIT 1;\n RETURN i;\nEND;\n$$ LANGUAGE PLPGSQL STABLE SECURITY DEFINER SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION get_item(_id text, _collection text DEFAULT NULL) RETURNS jsonb AS $$\n SELECT content_hydrate(items) FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection);\n$$ LANGUAGE SQL STABLE SECURITY DEFINER SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION delete_item(_id text, _collection text DEFAULT NULL) RETURNS VOID AS $$\nDECLARE\nout items%ROWTYPE;\nBEGIN\n DELETE FROM items WHERE id = _id AND (_collection IS NULL OR collection=_collection) RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n--/*\nCREATE OR REPLACE FUNCTION create_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION update_item(content jsonb) RETURNS VOID AS $$\nDECLARE\n old items %ROWTYPE;\n out items%ROWTYPE;\nBEGIN\n PERFORM delete_item(content->>'id', content->>'collection');\n PERFORM create_item(content);\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION create_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION collection_bbox(id text) RETURNS jsonb AS $$\n SELECT (replace(replace(replace(st_extent(geometry)::text,'BOX(','[['),')',']]'),' ',','))::jsonb\n FROM items WHERE collection=$1;\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION collection_temporal_extent(id text) RETURNS jsonb AS $$\n SELECT to_jsonb(array[array[min(datetime)::text, max(datetime)::text]])\n FROM items WHERE collection=$1;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection_extents() RETURNS VOID AS $$\nUPDATE collections SET\n content = content ||\n jsonb_build_object(\n 'extent', jsonb_build_object(\n 'spatial', jsonb_build_object(\n 'bbox', collection_bbox(collections.id)\n ),\n 'temporal', jsonb_build_object(\n 'interval', collection_temporal_extent(collections.id)\n )\n )\n )\n;\n$$ LANGUAGE SQL;\nCREATE VIEW partition_steps AS\nSELECT\n name,\n date_trunc('month',lower(datetime_range)) as sdate,\n date_trunc('month', upper(datetime_range)) + '1 month'::interval as edate\n FROM partitions WHERE datetime_range IS NOT NULL AND datetime_range != 'empty'::tstzrange\n ORDER BY datetime_range ASC\n;\n\nCREATE OR REPLACE FUNCTION chunker(\n IN _where text,\n OUT s timestamptz,\n OUT e timestamptz\n) RETURNS SETOF RECORD AS $$\nDECLARE\n explain jsonb;\nBEGIN\n IF _where IS NULL THEN\n _where := ' TRUE ';\n END IF;\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s;', _where)\n INTO explain;\n\n RETURN QUERY\n WITH t AS (\n SELECT j->>0 as p FROM\n jsonb_path_query(\n explain,\n 'strict $.**.\"Relation Name\" ? (@ != null)'\n ) j\n ),\n parts AS (\n SELECT sdate, edate FROM t JOIN partition_steps ON (t.p = name)\n ),\n times AS (\n SELECT sdate FROM parts\n UNION\n SELECT edate FROM parts\n ),\n uniq AS (\n SELECT DISTINCT sdate FROM times ORDER BY sdate\n ),\n last AS (\n SELECT sdate, lead(sdate, 1) over () as edate FROM uniq\n )\n SELECT sdate, edate FROM last WHERE edate IS NOT NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION partition_queries(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN partitions text[] DEFAULT NULL\n) RETURNS SETOF text AS $$\nDECLARE\n query text;\n sdate timestamptz;\n edate timestamptz;\nBEGIN\nIF _where IS NULL OR trim(_where) = '' THEN\n _where = ' TRUE ';\nEND IF;\nRAISE NOTICE 'Getting chunks for % %', _where, _orderby;\nIF _orderby ILIKE 'datetime d%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 DESC LOOP\n RETURN NEXT format($q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n $q$,\n sdate,\n edate,\n _where,\n _orderby\n );\n END LOOP;\nELSIF _orderby ILIKE 'datetime a%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 ASC LOOP\n RETURN NEXT format($q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n $q$,\n sdate,\n edate,\n _where,\n _orderby\n );\n END LOOP;\nELSE\n query := format($q$\n SELECT * FROM items\n WHERE %s\n ORDER BY %s\n $q$, _where, _orderby\n );\n\n RETURN NEXT query;\n RETURN;\nEND IF;\n\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION partition_query_view(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN _limit int DEFAULT 10\n) RETURNS text AS $$\n WITH p AS (\n SELECT * FROM partition_queries(_where, _orderby) p\n )\n SELECT\n CASE WHEN EXISTS (SELECT 1 FROM p) THEN\n (SELECT format($q$\n SELECT * FROM (\n %s\n ) total LIMIT %s\n $q$,\n string_agg(\n format($q$ SELECT * FROM ( %s ) AS sub $q$, p),\n '\n UNION ALL\n '\n ),\n _limit\n ))\n ELSE NULL\n END FROM p;\n$$ LANGUAGE SQL IMMUTABLE;\n\n\n\n\nCREATE OR REPLACE FUNCTION stac_search_to_where(j jsonb) RETURNS text AS $$\nDECLARE\n where_segments text[];\n _where text;\n dtrange tstzrange;\n collections text[];\n geom geometry;\n sdate timestamptz;\n edate timestamptz;\n filterlang text;\n filter jsonb := j->'filter';\nBEGIN\n IF j ? 'ids' THEN\n where_segments := where_segments || format('id = ANY (%L) ', to_text_array(j->'ids'));\n END IF;\n\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n where_segments := where_segments || format('collection = ANY (%L) ', collections);\n END IF;\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n\n where_segments := where_segments || format(' datetime <= %L::timestamptz AND end_datetime >= %L::timestamptz ',\n edate,\n sdate\n );\n END IF;\n\n geom := stac_geom(j);\n IF geom IS NOT NULL THEN\n where_segments := where_segments || format('st_intersects(geometry, %L)',geom);\n END IF;\n\n filterlang := COALESCE(\n j->>'filter-lang',\n get_setting('default-filter-lang', j->'conf')\n );\n IF NOT filter @? '$.**.op' THEN\n filterlang := 'cql-json';\n END IF;\n\n IF filterlang NOT IN ('cql-json','cql2-json') AND j ? 'filter' THEN\n RAISE EXCEPTION '% is not a supported filter-lang. Please use cql-json or cql2-json.', filterlang;\n END IF;\n\n IF j ? 'query' AND j ? 'filter' THEN\n RAISE EXCEPTION 'Can only use either query or filter at one time.';\n END IF;\n\n IF j ? 'query' THEN\n filter := query_to_cql2(j->'query');\n ELSIF filterlang = 'cql-json' THEN\n filter := cql1_to_cql2(filter);\n END IF;\n RAISE NOTICE 'FILTER: %', filter;\n where_segments := where_segments || cql2_query(filter);\n IF cardinality(where_segments) < 1 THEN\n RETURN ' TRUE ';\n END IF;\n\n _where := array_to_string(array_remove(where_segments, NULL), ' AND ');\n\n IF _where IS NULL OR BTRIM(_where) = '' THEN\n RETURN ' TRUE ';\n END IF;\n RETURN _where;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION parse_sort_dir(_dir text, reverse boolean default false) RETURNS text AS $$\n WITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n ) SELECT\n CASE\n WHEN NOT reverse THEN d\n WHEN d = 'ASC' THEN 'DESC'\n WHEN d = 'DESC' THEN 'ASC'\n END\n FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION sort_dir_to_op(_dir text, prev boolean default false) RETURNS text AS $$\n WITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n ) SELECT\n CASE\n WHEN d = 'ASC' AND prev THEN '<='\n WHEN d = 'DESC' AND prev THEN '>='\n WHEN d = 'ASC' THEN '>='\n WHEN d = 'DESC' THEN '<='\n END\n FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION sort_sqlorderby(\n _search jsonb DEFAULT NULL,\n reverse boolean DEFAULT FALSE\n) RETURNS text AS $$\n WITH sortby AS (\n SELECT coalesce(_search->'sortby','[{\"field\":\"datetime\", \"direction\":\"desc\"}]') as sort\n ), withid AS (\n SELECT CASE\n WHEN sort @? '$[*] ? (@.field == \"id\")' THEN sort\n ELSE sort || '[{\"field\":\"id\", \"direction\":\"desc\"}]'::jsonb\n END as sort\n FROM sortby\n ), withid_rows AS (\n SELECT jsonb_array_elements(sort) as value FROM withid\n ),sorts AS (\n SELECT\n coalesce(\n -- field_orderby((items_path(value->>'field')).path_txt),\n (queryable(value->>'field')).expression\n ) as key,\n parse_sort_dir(value->>'direction', reverse) as dir\n FROM withid_rows\n )\n SELECT array_to_string(\n array_agg(concat(key, ' ', dir)),\n ', '\n ) FROM sorts;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_sort_dir(sort_item jsonb) RETURNS text AS $$\n SELECT CASE WHEN sort_item->>'direction' ILIKE 'desc%' THEN 'DESC' ELSE 'ASC' END;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION get_token_filter(_search jsonb = '{}'::jsonb, token_rec jsonb DEFAULT NULL) RETURNS text AS $$\nDECLARE\n token_id text;\n filters text[] := '{}'::text[];\n prev boolean := TRUE;\n field text;\n dir text;\n sort record;\n orfilters text[] := '{}'::text[];\n andfilters text[] := '{}'::text[];\n output text;\n token_where text;\nBEGIN\n RAISE NOTICE 'Getting Token Filter. % %', _search, token_rec;\n -- If no token provided return NULL\n IF token_rec IS NULL THEN\n IF NOT (_search ? 'token' AND\n (\n (_search->>'token' ILIKE 'prev:%')\n OR\n (_search->>'token' ILIKE 'next:%')\n )\n ) THEN\n RETURN NULL;\n END IF;\n prev := (_search->>'token' ILIKE 'prev:%');\n token_id := substr(_search->>'token', 6);\n SELECT to_jsonb(items) INTO token_rec\n FROM items WHERE id=token_id;\n END IF;\n RAISE NOTICE 'TOKEN ID: % %', token_rec, token_rec->'id';\n\n CREATE TEMP TABLE sorts (\n _row int GENERATED ALWAYS AS IDENTITY NOT NULL,\n _field text PRIMARY KEY,\n _dir text NOT NULL,\n _val text\n ) ON COMMIT DROP;\n\n -- Make sure we only have distinct columns to sort with taking the first one we get\n INSERT INTO sorts (_field, _dir)\n SELECT\n (queryable(value->>'field')).expression,\n get_sort_dir(value)\n FROM\n jsonb_array_elements(coalesce(_search->'sortby','[{\"field\":\"datetime\",\"direction\":\"desc\"}]'))\n ON CONFLICT DO NOTHING\n ;\n RAISE NOTICE 'sorts 1: %', (SELECT jsonb_agg(to_json(sorts)) FROM sorts);\n -- Get the first sort direction provided. As the id is a primary key, if there are any\n -- sorts after id they won't do anything, so make sure that id is the last sort item.\n SELECT _dir INTO dir FROM sorts ORDER BY _row ASC LIMIT 1;\n IF EXISTS (SELECT 1 FROM sorts WHERE _field = 'id') THEN\n DELETE FROM sorts WHERE _row > (SELECT _row FROM sorts WHERE _field = 'id' ORDER BY _row ASC);\n ELSE\n INSERT INTO sorts (_field, _dir) VALUES ('id', dir);\n END IF;\n\n -- Add value from looked up item to the sorts table\n UPDATE sorts SET _val=quote_literal(token_rec->>_field);\n\n -- Check if all sorts are the same direction and use row comparison\n -- to filter\n RAISE NOTICE 'sorts 2: %', (SELECT jsonb_agg(to_json(sorts)) FROM sorts);\n\n IF (SELECT count(DISTINCT _dir) FROM sorts) = 1 THEN\n SELECT format(\n '(%s) %s (%s)',\n concat_ws(', ', VARIADIC array_agg(quote_ident(_field))),\n CASE WHEN (prev AND dir = 'ASC') OR (NOT prev AND dir = 'DESC') THEN '<' ELSE '>' END,\n concat_ws(', ', VARIADIC array_agg(_val))\n ) INTO output FROM sorts\n WHERE token_rec ? _field\n ;\n ELSE\n FOR sort IN SELECT * FROM sorts ORDER BY _row asc LOOP\n RAISE NOTICE 'SORT: %', sort;\n IF sort._row = 1 THEN\n orfilters := orfilters || format('(%s %s %s)',\n quote_ident(sort._field),\n CASE WHEN (prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC') THEN '<' ELSE '>' END,\n sort._val\n );\n ELSE\n orfilters := orfilters || format('(%s AND %s %s %s)',\n array_to_string(andfilters, ' AND '),\n quote_ident(sort._field),\n CASE WHEN (prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC') THEN '<' ELSE '>' END,\n sort._val\n );\n\n END IF;\n andfilters := andfilters || format('%s = %s',\n quote_ident(sort._field),\n sort._val\n );\n END LOOP;\n output := array_to_string(orfilters, ' OR ');\n END IF;\n DROP TABLE IF EXISTS sorts;\n token_where := concat('(',coalesce(output,'true'),')');\n IF trim(token_where) = '' THEN\n token_where := NULL;\n END IF;\n RAISE NOTICE 'TOKEN_WHERE: |%|',token_where;\n RETURN token_where;\n END;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION search_tohash(jsonb) RETURNS jsonb AS $$\n SELECT $1 - '{token,limit,context,includes,excludes}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION search_hash(jsonb, jsonb) RETURNS text AS $$\n SELECT md5(concat(search_tohash($1)::text,$2::text));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TABLE IF NOT EXISTS searches(\n hash text GENERATED ALWAYS AS (search_hash(search, metadata)) STORED PRIMARY KEY,\n search jsonb NOT NULL,\n _where text,\n orderby text,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n metadata jsonb DEFAULT '{}'::jsonb NOT NULL\n);\nCREATE TABLE IF NOT EXISTS search_wheres(\n id bigint generated always as identity primary key,\n _where text NOT NULL,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n statslastupdated timestamptz,\n estimated_count bigint,\n estimated_cost float,\n time_to_estimate float,\n total_count bigint,\n time_to_count float,\n partitions text[]\n);\n\nCREATE INDEX IF NOT EXISTS search_wheres_partitions ON search_wheres USING GIN (partitions);\nCREATE UNIQUE INDEX IF NOT EXISTS search_wheres_where ON search_wheres ((md5(_where)));\n\nCREATE OR REPLACE FUNCTION where_stats(inwhere text, updatestats boolean default false, conf jsonb default null) RETURNS search_wheres AS $$\nDECLARE\n t timestamptz;\n i interval;\n explain_json jsonb;\n partitions text[];\n sw search_wheres%ROWTYPE;\n inwhere_hash text := md5(inwhere);\n _context text := lower(context(conf));\n _stats_ttl interval := context_stats_ttl(conf);\n _estimated_cost float := context_estimated_cost(conf);\n _estimated_count int := context_estimated_count(conf);\nBEGIN\n IF _context = 'off' THEN\n sw._where := inwhere;\n return sw;\n END IF;\n\n SELECT * INTO sw FROM search_wheres WHERE md5(_where)=inwhere_hash FOR UPDATE;\n\n -- Update statistics if explicitly set, if statistics do not exist, or statistics ttl has expired\n IF NOT updatestats THEN\n RAISE NOTICE 'Checking if update is needed for: % .', inwhere;\n RAISE NOTICE 'Stats Last Updated: %', sw.statslastupdated;\n RAISE NOTICE 'TTL: %, Age: %', _stats_ttl, now() - sw.statslastupdated;\n RAISE NOTICE 'Context: %, Existing Total: %', _context, sw.total_count;\n IF\n sw.statslastupdated IS NULL\n OR (now() - sw.statslastupdated) > _stats_ttl\n OR (context(conf) != 'off' AND sw.total_count IS NULL)\n THEN\n updatestats := TRUE;\n END IF;\n END IF;\n\n sw._where := inwhere;\n sw.lastused := now();\n sw.usecount := coalesce(sw.usecount,0) + 1;\n\n IF NOT updatestats THEN\n UPDATE search_wheres SET\n lastused = sw.lastused,\n usecount = sw.usecount\n WHERE md5(_where) = inwhere_hash\n RETURNING * INTO sw\n ;\n RETURN sw;\n END IF;\n\n -- Use explain to get estimated count/cost and a list of the partitions that would be hit by the query\n t := clock_timestamp();\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s', inwhere)\n INTO explain_json;\n RAISE NOTICE 'Time for just the explain: %', clock_timestamp() - t;\n i := clock_timestamp() - t;\n\n sw.statslastupdated := now();\n sw.estimated_count := explain_json->0->'Plan'->'Plan Rows';\n sw.estimated_cost := explain_json->0->'Plan'->'Total Cost';\n sw.time_to_estimate := extract(epoch from i);\n\n RAISE NOTICE 'ESTIMATED_COUNT: % < %', sw.estimated_count, _estimated_count;\n RAISE NOTICE 'ESTIMATED_COST: % < %', sw.estimated_cost, _estimated_cost;\n\n -- Do a full count of rows if context is set to on or if auto is set and estimates are low enough\n IF\n _context = 'on'\n OR\n ( _context = 'auto' AND\n (\n sw.estimated_count < _estimated_count\n AND\n sw.estimated_cost < _estimated_cost\n )\n )\n THEN\n t := clock_timestamp();\n RAISE NOTICE 'Calculating actual count...';\n EXECUTE format(\n 'SELECT count(*) FROM items WHERE %s',\n inwhere\n ) INTO sw.total_count;\n i := clock_timestamp() - t;\n RAISE NOTICE 'Actual Count: % -- %', sw.total_count, i;\n sw.time_to_count := extract(epoch FROM i);\n ELSE\n sw.total_count := NULL;\n sw.time_to_count := NULL;\n END IF;\n\n\n INSERT INTO search_wheres\n (_where, lastused, usecount, statslastupdated, estimated_count, estimated_cost, time_to_estimate, partitions, total_count, time_to_count)\n SELECT sw._where, sw.lastused, sw.usecount, sw.statslastupdated, sw.estimated_count, sw.estimated_cost, sw.time_to_estimate, sw.partitions, sw.total_count, sw.time_to_count\n ON CONFLICT ((md5(_where)))\n DO UPDATE\n SET\n lastused = sw.lastused,\n usecount = sw.usecount,\n statslastupdated = sw.statslastupdated,\n estimated_count = sw.estimated_count,\n estimated_cost = sw.estimated_cost,\n time_to_estimate = sw.time_to_estimate,\n total_count = sw.total_count,\n time_to_count = sw.time_to_count\n ;\n RETURN sw;\nEND;\n$$ LANGUAGE PLPGSQL ;\n\n\n\nDROP FUNCTION IF EXISTS search_query;\nCREATE OR REPLACE FUNCTION search_query(\n _search jsonb = '{}'::jsonb,\n updatestats boolean = false,\n _metadata jsonb = '{}'::jsonb\n) RETURNS searches AS $$\nDECLARE\n search searches%ROWTYPE;\n pexplain jsonb;\n t timestamptz;\n i interval;\nBEGIN\n SELECT * INTO search FROM searches\n WHERE hash=search_hash(_search, _metadata) FOR UPDATE;\n\n -- Calculate the where clause if not already calculated\n IF search._where IS NULL THEN\n search._where := stac_search_to_where(_search);\n END IF;\n\n -- Calculate the order by clause if not already calculated\n IF search.orderby IS NULL THEN\n search.orderby := sort_sqlorderby(_search);\n END IF;\n\n PERFORM where_stats(search._where, updatestats, _search->'conf');\n\n search.lastused := now();\n search.usecount := coalesce(search.usecount, 0) + 1;\n INSERT INTO searches (search, _where, orderby, lastused, usecount, metadata)\n VALUES (_search, search._where, search.orderby, search.lastused, search.usecount, _metadata)\n ON CONFLICT (hash) DO\n UPDATE SET\n _where = EXCLUDED._where,\n orderby = EXCLUDED.orderby,\n lastused = EXCLUDED.lastused,\n usecount = EXCLUDED.usecount,\n metadata = EXCLUDED.metadata\n RETURNING * INTO search\n ;\n RETURN search;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION search(_search jsonb = '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n searches searches%ROWTYPE;\n _where text;\n token_where text;\n full_where text;\n orderby text;\n query text;\n token_type text := substr(_search->>'token',1,4);\n _limit int := coalesce((_search->>'limit')::int, 10);\n curs refcursor;\n cntr int := 0;\n iter_record items%ROWTYPE;\n first_record jsonb;\n first_item items%ROWTYPE;\n last_item items%ROWTYPE;\n last_record jsonb;\n out_records jsonb := '[]'::jsonb;\n prev_query text;\n next text;\n prev_id text;\n has_next boolean := false;\n has_prev boolean := false;\n prev text;\n total_count bigint;\n context jsonb;\n collection jsonb;\n includes text[];\n excludes text[];\n exit_flag boolean := FALSE;\n batches int := 0;\n timer timestamptz := clock_timestamp();\n pstart timestamptz;\n pend timestamptz;\n pcurs refcursor;\n search_where search_wheres%ROWTYPE;\n id text;\nBEGIN\nCREATE TEMP TABLE results (content jsonb) ON COMMIT DROP;\n-- if ids is set, short circuit and just use direct ids query for each id\n-- skip any paging or caching\n-- hard codes ordering in the same order as the array of ids\nIF _search ? 'ids' THEN\n INSERT INTO results\n SELECT\n CASE WHEN _search->'conf'->>'nohydrate' IS NOT NULL AND (_search->'conf'->>'nohydrate')::boolean = true THEN\n content_nonhydrated(items, _search->'fields')\n ELSE\n content_hydrate(items, _search->'fields')\n END\n FROM items WHERE\n items.id = ANY(to_text_array(_search->'ids'))\n AND\n CASE WHEN _search ? 'collections' THEN\n items.collection = ANY(to_text_array(_search->'collections'))\n ELSE TRUE\n END\n ORDER BY items.datetime desc, items.id desc\n ;\n SELECT INTO total_count count(*) FROM results;\nELSE\n searches := search_query(_search);\n _where := searches._where;\n orderby := searches.orderby;\n search_where := where_stats(_where);\n total_count := coalesce(search_where.total_count, search_where.estimated_count);\n\n IF token_type='prev' THEN\n token_where := get_token_filter(_search, null::jsonb);\n orderby := sort_sqlorderby(_search, TRUE);\n END IF;\n IF token_type='next' THEN\n token_where := get_token_filter(_search, null::jsonb);\n END IF;\n\n full_where := concat_ws(' AND ', _where, token_where);\n RAISE NOTICE 'FULL QUERY % %', full_where, clock_timestamp()-timer;\n timer := clock_timestamp();\n\n FOR query IN SELECT partition_queries(full_where, orderby, search_where.partitions) LOOP\n timer := clock_timestamp();\n query := format('%s LIMIT %s', query, _limit + 1);\n RAISE NOTICE 'Partition Query: %', query;\n batches := batches + 1;\n -- curs = create_cursor(query);\n OPEN curs FOR EXECUTE query;\n LOOP\n FETCH curs into iter_record;\n EXIT WHEN NOT FOUND;\n cntr := cntr + 1;\n\n IF _search->'conf'->>'nohydrate' IS NOT NULL AND (_search->'conf'->>'nohydrate')::boolean = true THEN\n last_record := content_nonhydrated(iter_record, _search->'fields');\n ELSE\n last_record := content_hydrate(iter_record, _search->'fields');\n END IF;\n last_item := iter_record;\n IF cntr = 1 THEN\n first_item := last_item;\n first_record := last_record;\n END IF;\n IF cntr <= _limit THEN\n INSERT INTO results (content) VALUES (last_record);\n ELSIF cntr > _limit THEN\n has_next := true;\n exit_flag := true;\n EXIT;\n END IF;\n END LOOP;\n CLOSE curs;\n RAISE NOTICE 'Query took %.', clock_timestamp()-timer;\n timer := clock_timestamp();\n EXIT WHEN exit_flag;\n END LOOP;\n RAISE NOTICE 'Scanned through % partitions.', batches;\nEND IF;\n\nSELECT jsonb_agg(content) INTO out_records FROM results WHERE content is not NULL;\n\nDROP TABLE results;\n\n\n-- Flip things around if this was the result of a prev token query\nIF token_type='prev' THEN\n out_records := flip_jsonb_array(out_records);\n first_item := last_item;\n first_record := last_record;\nEND IF;\n\n-- If this query has a token, see if there is data before the first record\nIF _search ? 'token' THEN\n prev_query := format(\n 'SELECT 1 FROM items WHERE %s LIMIT 1',\n concat_ws(\n ' AND ',\n _where,\n trim(get_token_filter(_search, to_jsonb(first_item)))\n )\n );\n RAISE NOTICE 'Query to get previous record: % --- %', prev_query, first_record;\n EXECUTE prev_query INTO has_prev;\n IF FOUND and has_prev IS NOT NULL THEN\n RAISE NOTICE 'Query results from prev query: %', has_prev;\n has_prev := TRUE;\n END IF;\nEND IF;\nhas_prev := COALESCE(has_prev, FALSE);\n\nIF has_prev THEN\n prev := out_records->0->>'id';\nEND IF;\nIF has_next OR token_type='prev' THEN\n next := out_records->-1->>'id';\nEND IF;\n\nIF context(_search->'conf') != 'off' THEN\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'matched', total_count,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\nELSE\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\nEND IF;\n\ncollection := jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb),\n 'next', next,\n 'prev', prev,\n 'context', context\n);\n\nRETURN collection;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER SET SEARCH_PATH TO pgstac, public;\n\n\nCREATE OR REPLACE FUNCTION search_cursor(_search jsonb = '{}'::jsonb) RETURNS refcursor AS $$\nDECLARE\n curs refcursor;\n searches searches%ROWTYPE;\n _where text;\n _orderby text;\n q text;\n\nBEGIN\n searches := search_query(_search);\n _where := searches._where;\n _orderby := searches.orderby;\n\n OPEN curs FOR\n WITH p AS (\n SELECT * FROM partition_queries(_where, _orderby) p\n )\n SELECT\n CASE WHEN EXISTS (SELECT 1 FROM p) THEN\n (SELECT format($q$\n SELECT * FROM (\n %s\n ) total\n $q$,\n string_agg(\n format($q$ SELECT * FROM ( %s ) AS sub $q$, p),\n '\n UNION ALL\n '\n )\n ))\n ELSE NULL\n END FROM p;\n RETURN curs;\nEND;\n$$ LANGUAGE PLPGSQL;\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION tileenvelope(zoom int, x int, y int) RETURNS geometry AS $$\nWITH t AS (\n SELECT\n 20037508.3427892 as merc_max,\n -20037508.3427892 as merc_min,\n (2 * 20037508.3427892) / (2 ^ zoom) as tile_size\n)\nSELECT st_makeenvelope(\n merc_min + (tile_size * x),\n merc_max - (tile_size * (y + 1)),\n merc_min + (tile_size * (x + 1)),\n merc_max - (tile_size * y),\n 3857\n) FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;DROP FUNCTION IF EXISTS mercgrid;\n\n\nCREATE OR REPLACE FUNCTION ftime() RETURNS interval as $$\nSELECT age(clock_timestamp(), transaction_timestamp());\n$$ LANGUAGE SQL;\nSET SEARCH_PATH to pgstac, public;\n\nDROP FUNCTION IF EXISTS geometrysearch;\nCREATE OR REPLACE FUNCTION geometrysearch(\n IN geom geometry,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE, -- Return as soon as the passed in geometry is full covered\n IN skipcovered boolean DEFAULT TRUE -- Skip any items that would show up completely under the previous items\n) RETURNS jsonb AS $$\nDECLARE\n search searches%ROWTYPE;\n curs refcursor;\n _where text;\n query text;\n iter_record items%ROWTYPE;\n out_records jsonb := '{}'::jsonb[];\n exit_flag boolean := FALSE;\n counter int := 1;\n scancounter int := 1;\n remaining_limit int := _scanlimit;\n tilearea float;\n unionedgeom geometry;\n clippedgeom geometry;\n unionedgeom_area float := 0;\n prev_area float := 0;\n excludes text[];\n includes text[];\n\nBEGIN\n DROP TABLE IF EXISTS pgstac_results;\n CREATE TEMP TABLE pgstac_results (content jsonb) ON COMMIT DROP;\n\n -- If skipcovered is true then you will always want to exit when the passed in geometry is full\n IF skipcovered THEN\n exitwhenfull := TRUE;\n END IF;\n\n SELECT * INTO search FROM searches WHERE hash=queryhash;\n\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Search with Query Hash % Not Found', queryhash;\n END IF;\n\n tilearea := st_area(geom);\n _where := format('%s AND st_intersects(geometry, %L::geometry)', search._where, geom);\n\n\n FOR query IN SELECT * FROM partition_queries(_where, search.orderby) LOOP\n query := format('%s LIMIT %L', query, remaining_limit);\n RAISE NOTICE '%', query;\n OPEN curs FOR EXECUTE query;\n LOOP\n FETCH curs INTO iter_record;\n EXIT WHEN NOT FOUND;\n IF exitwhenfull OR skipcovered THEN -- If we are not using exitwhenfull or skipcovered, we do not need to do expensive geometry operations\n clippedgeom := st_intersection(geom, iter_record.geometry);\n\n IF unionedgeom IS NULL THEN\n unionedgeom := clippedgeom;\n ELSE\n unionedgeom := st_union(unionedgeom, clippedgeom);\n END IF;\n\n unionedgeom_area := st_area(unionedgeom);\n\n IF skipcovered AND prev_area = unionedgeom_area THEN\n scancounter := scancounter + 1;\n CONTINUE;\n END IF;\n\n prev_area := unionedgeom_area;\n\n RAISE NOTICE '% % % %', unionedgeom_area/tilearea, counter, scancounter, ftime();\n END IF;\n RAISE NOTICE '% %', iter_record, content_hydrate(iter_record, fields);\n INSERT INTO pgstac_results (content) VALUES (content_hydrate(iter_record, fields));\n\n IF counter >= _limit\n OR scancounter > _scanlimit\n OR ftime() > _timelimit\n OR (exitwhenfull AND unionedgeom_area >= tilearea)\n THEN\n exit_flag := TRUE;\n EXIT;\n END IF;\n counter := counter + 1;\n scancounter := scancounter + 1;\n\n END LOOP;\n CLOSE curs;\n EXIT WHEN exit_flag;\n remaining_limit := _scanlimit - scancounter;\n END LOOP;\n\n SELECT jsonb_agg(content) INTO out_records FROM pgstac_results WHERE content IS NOT NULL;\n\n RETURN jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb)\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDROP FUNCTION IF EXISTS geojsonsearch;\nCREATE OR REPLACE FUNCTION geojsonsearch(\n IN geojson jsonb,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_geomfromgeojson(geojson),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS xyzsearch;\nCREATE OR REPLACE FUNCTION xyzsearch(\n IN _x int,\n IN _y int,\n IN _z int,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_transform(tileenvelope(_z, _x, _y), 4326),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nGRANT ALL ON SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac to pgstac_admin;\n\n-- pgstac_read role limited to using function apis\nGRANT EXECUTE ON FUNCTION search TO pgstac_read;\nGRANT EXECUTE ON FUNCTION search_query TO pgstac_read;\nGRANT EXECUTE ON FUNCTION item_by_id TO pgstac_read;\nGRANT EXECUTE ON FUNCTION get_item TO pgstac_read;\n\nGRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_ingest;\nGRANT USAGE ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_ingest;\nSELECT set_version('0.6.8');\n" }, { "path": "src/pgstac/migrations/pgstac.0.6.9-0.6.10.sql", "content": "SET SEARCH_PATH to pgstac, public;\nset check_function_bodies = off;\n\nCREATE OR REPLACE FUNCTION pgstac.get_queryables(_collection_ids text[] DEFAULT NULL::text[])\n RETURNS jsonb\n LANGUAGE plpgsql\n STABLE\nAS $function$\nBEGIN\n -- Build up queryables if the input contains valid collection ids or is empty\n IF EXISTS (\n SELECT 1 FROM collections\n WHERE\n _collection_ids IS NULL\n OR cardinality(_collection_ids) = 0\n OR id = ANY(_collection_ids)\n )\n THEN\n RETURN (\n SELECT\n jsonb_build_object(\n '$schema', 'http://json-schema.org/draft-07/schema#',\n '$id', 'https://example.org/queryables',\n 'type', 'object',\n 'title', 'STAC Queryables.',\n 'properties', jsonb_object_agg(\n name,\n definition\n )\n )\n FROM queryables\n WHERE\n _collection_ids IS NULL OR\n cardinality(_collection_ids) = 0 OR\n collection_ids IS NULL OR\n _collection_ids && collection_ids\n );\n ELSE\n RETURN NULL;\n END IF;\nEND;\n\n$function$\n;\n\n\n\nSELECT set_version('0.6.10');\n" }, { "path": "src/pgstac/migrations/pgstac.0.6.9.sql", "content": "CREATE EXTENSION IF NOT EXISTS postgis;\nCREATE EXTENSION IF NOT EXISTS btree_gist;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_admin;\n CREATE ROLE pgstac_read;\n CREATE ROLE pgstac_ingest;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nGRANT pgstac_admin TO current_user;\n\nCREATE SCHEMA IF NOT EXISTS pgstac AUTHORIZATION pgstac_admin;\n\nALTER ROLE pgstac_admin SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_read SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_ingest SET SEARCH_PATH TO pgstac, public;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\n\nGRANT pgstac_read TO pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\n\nSET ROLE pgstac_admin;\n\nSET SEARCH_PATH TO pgstac, public;\n\n\nCREATE TABLE IF NOT EXISTS migrations (\n version text PRIMARY KEY,\n datetime timestamptz DEFAULT clock_timestamp() NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION get_version() RETURNS text AS $$\n SELECT version FROM pgstac.migrations ORDER BY datetime DESC, version DESC LIMIT 1;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION set_version(text) RETURNS text AS $$\n INSERT INTO pgstac.migrations (version) VALUES ($1)\n ON CONFLICT DO NOTHING\n RETURNING version;\n$$ LANGUAGE SQL;\n\n\nCREATE TABLE IF NOT EXISTS pgstac_settings (\n name text PRIMARY KEY,\n value text NOT NULL\n);\n\nINSERT INTO pgstac_settings (name, value) VALUES\n ('context', 'off'),\n ('context_estimated_count', '100000'),\n ('context_estimated_cost', '100000'),\n ('context_stats_ttl', '1 day'),\n ('default-filter-lang', 'cql2-json'),\n ('additional_properties', 'true')\nON CONFLICT DO NOTHING\n;\n\n\nCREATE OR REPLACE FUNCTION get_setting(IN _setting text, IN conf jsonb DEFAULT NULL) RETURNS text AS $$\nSELECT COALESCE(\n conf->>_setting,\n current_setting(concat('pgstac.',_setting), TRUE),\n (SELECT value FROM pgstac.pgstac_settings WHERE name=_setting)\n);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context(conf jsonb DEFAULT NULL) RETURNS text AS $$\n SELECT pgstac.get_setting('context', conf);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context_estimated_count(conf jsonb DEFAULT NULL) RETURNS int AS $$\n SELECT pgstac.get_setting('context_estimated_count', conf)::int;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_estimated_cost();\nCREATE OR REPLACE FUNCTION context_estimated_cost(conf jsonb DEFAULT NULL) RETURNS float AS $$\n SELECT pgstac.get_setting('context_estimated_cost', conf)::float;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_stats_ttl();\nCREATE OR REPLACE FUNCTION context_stats_ttl(conf jsonb DEFAULT NULL) RETURNS interval AS $$\n SELECT pgstac.get_setting('context_stats_ttl', conf)::interval;\n$$ LANGUAGE SQL;\n\n\nCREATE OR REPLACE FUNCTION notice(VARIADIC text[]) RETURNS boolean AS $$\nDECLARE\ndebug boolean := current_setting('pgstac.debug', true);\nBEGIN\n IF debug THEN\n RAISE NOTICE 'NOTICE FROM FUNC: % >>>>> %', concat_ws(' | ', $1), clock_timestamp();\n RETURN TRUE;\n END IF;\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION empty_arr(ANYARRAY) RETURNS BOOLEAN AS $$\nSELECT CASE\n WHEN $1 IS NULL THEN TRUE\n WHEN cardinality($1)<1 THEN TRUE\nELSE FALSE\nEND;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_intersection(_a ANYARRAY, _b ANYARRAY) RETURNS ANYARRAY AS $$\n SELECT ARRAY ( SELECT unnest(_a) INTERSECT SELECT UNNEST(_b) );\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION array_map_ident(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_ident(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_map_literal(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_literal(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION array_reverse(anyarray) RETURNS anyarray AS $$\nSELECT ARRAY(\n SELECT $1[i]\n FROM generate_subscripts($1,1) AS s(i)\n ORDER BY i DESC\n);\n$$ LANGUAGE SQL STRICT IMMUTABLE;\nCREATE OR REPLACE FUNCTION to_int(jsonb) RETURNS int AS $$\n SELECT floor(($1->>0)::float)::int;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION to_float(jsonb) RETURNS float AS $$\n SELECT ($1->>0)::float;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION to_tstz(jsonb) RETURNS timestamptz AS $$\n SELECT ($1->>0)::timestamptz;\n$$ LANGUAGE SQL IMMUTABLE STRICT SET TIME ZONE 'UTC';\n\n\nCREATE OR REPLACE FUNCTION to_text(jsonb) RETURNS text AS $$\n SELECT CASE WHEN jsonb_typeof($1) IN ('array','object') THEN $1::text ELSE $1->>0 END;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION to_text_array(jsonb) RETURNS text[] AS $$\n SELECT\n CASE jsonb_typeof($1)\n WHEN 'array' THEN ARRAY(SELECT jsonb_array_elements_text($1))\n ELSE ARRAY[$1->>0]\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION bbox_geom(_bbox jsonb) RETURNS geometry AS $$\nSELECT CASE jsonb_array_length(_bbox)\n WHEN 4 THEN\n ST_SetSRID(ST_MakeEnvelope(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float,\n (_bbox->>3)::float\n ),4326)\n WHEN 6 THEN\n ST_SetSRID(ST_3DMakeBox(\n ST_MakePoint(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float\n ),\n ST_MakePoint(\n (_bbox->>3)::float,\n (_bbox->>4)::float,\n (_bbox->>5)::float\n )\n ),4326)\n ELSE null END;\n;\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION geom_bbox(_geom geometry) RETURNS jsonb AS $$\n SELECT jsonb_build_array(\n st_xmin(_geom),\n st_ymin(_geom),\n st_xmax(_geom),\n st_ymax(_geom)\n );\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION flip_jsonb_array(j jsonb) RETURNS jsonb AS $$\n SELECT jsonb_agg(value) FROM (SELECT value FROM jsonb_array_elements(j) WITH ORDINALITY ORDER BY ordinality DESC) as t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION explode_dotpaths(j jsonb) RETURNS SETOF text[] AS $$\n SELECT string_to_array(p, '.') as e FROM jsonb_array_elements_text(j) p;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION explode_dotpaths_recurse(IN j jsonb) RETURNS SETOF text[] AS $$\n WITH RECURSIVE t AS (\n SELECT e FROM explode_dotpaths(j) e\n UNION ALL\n SELECT e[1:cardinality(e)-1]\n FROM t\n WHERE cardinality(e)>1\n ) SELECT e FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION jsonb_set_nested(j jsonb, path text[], val jsonb) RETURNS jsonb AS $$\nDECLARE\nBEGIN\n IF cardinality(path) > 1 THEN\n FOR i IN 1..(cardinality(path)-1) LOOP\n IF j #> path[:i] IS NULL THEN\n j := jsonb_set_lax(j, path[:i], '{}', TRUE);\n END IF;\n END LOOP;\n END IF;\n RETURN jsonb_set_lax(j, path, val, true);\n\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\n\n\nCREATE OR REPLACE FUNCTION jsonb_include(j jsonb, f jsonb) RETURNS jsonb AS $$\nDECLARE\n includes jsonb := f-> 'include';\n outj jsonb := '{}'::jsonb;\n path text[];\nBEGIN\n IF\n includes IS NULL\n OR jsonb_array_length(includes) = 0\n THEN\n RETURN j;\n ELSE\n includes := includes || '[\"id\",\"collection\"]'::jsonb;\n FOR path IN SELECT explode_dotpaths(includes) LOOP\n outj := jsonb_set_nested(outj, path, j #> path);\n END LOOP;\n END IF;\n RETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_exclude(j jsonb, f jsonb) RETURNS jsonb AS $$\nDECLARE\n excludes jsonb := f-> 'exclude';\n outj jsonb := j;\n path text[];\nBEGIN\n IF\n excludes IS NULL\n OR jsonb_array_length(excludes) = 0\n THEN\n RETURN j;\n ELSE\n FOR path IN SELECT explode_dotpaths(excludes) LOOP\n outj := outj #- path;\n END LOOP;\n END IF;\n RETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_fields(j jsonb, f jsonb DEFAULT '{\"fields\":[]}') RETURNS jsonb AS $$\n SELECT jsonb_exclude(jsonb_include(j, f), f);\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION merge_jsonb(_a jsonb, _b jsonb) RETURNS jsonb AS $$\n SELECT\n CASE\n WHEN _a = '\"𒍟※\"'::jsonb THEN NULL\n WHEN _a IS NULL OR jsonb_typeof(_a) = 'null' THEN _b\n WHEN jsonb_typeof(_a) = 'object' AND jsonb_typeof(_b) = 'object' THEN\n (\n SELECT\n jsonb_strip_nulls(\n jsonb_object_agg(\n key,\n merge_jsonb(a.value, b.value)\n )\n )\n FROM\n jsonb_each(coalesce(_a,'{}'::jsonb)) as a\n FULL JOIN\n jsonb_each(coalesce(_b,'{}'::jsonb)) as b\n USING (key)\n )\n WHEN\n jsonb_typeof(_a) = 'array'\n AND jsonb_typeof(_b) = 'array'\n AND jsonb_array_length(_a) = jsonb_array_length(_b)\n THEN\n (\n SELECT jsonb_agg(m) FROM\n ( SELECT\n merge_jsonb(\n jsonb_array_elements(_a),\n jsonb_array_elements(_b)\n ) as m\n ) as l\n )\n ELSE _a\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION strip_jsonb(_a jsonb, _b jsonb) RETURNS jsonb AS $$\n SELECT\n CASE\n\n WHEN (_a IS NULL OR jsonb_typeof(_a) = 'null') AND _b IS NOT NULL AND jsonb_typeof(_b) != 'null' THEN '\"𒍟※\"'::jsonb\n WHEN _b IS NULL OR jsonb_typeof(_a) = 'null' THEN _a\n WHEN _a = _b AND jsonb_typeof(_a) = 'object' THEN '{}'::jsonb\n WHEN _a = _b THEN NULL\n WHEN jsonb_typeof(_a) = 'object' AND jsonb_typeof(_b) = 'object' THEN\n (\n SELECT\n jsonb_strip_nulls(\n jsonb_object_agg(\n key,\n strip_jsonb(a.value, b.value)\n )\n )\n FROM\n jsonb_each(_a) as a\n FULL JOIN\n jsonb_each(_b) as b\n USING (key)\n )\n WHEN\n jsonb_typeof(_a) = 'array'\n AND jsonb_typeof(_b) = 'array'\n AND jsonb_array_length(_a) = jsonb_array_length(_b)\n THEN\n (\n SELECT jsonb_agg(m) FROM\n ( SELECT\n strip_jsonb(\n jsonb_array_elements(_a),\n jsonb_array_elements(_b)\n ) as m\n ) as l\n )\n ELSE _a\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE;\n/* looks for a geometry in a stac item first from geometry and falling back to bbox */\nCREATE OR REPLACE FUNCTION stac_geom(value jsonb) RETURNS geometry AS $$\nSELECT\n CASE\n WHEN value ? 'intersects' THEN\n ST_GeomFromGeoJSON(value->>'intersects')\n WHEN value ? 'geometry' THEN\n ST_GeomFromGeoJSON(value->>'geometry')\n WHEN value ? 'bbox' THEN\n pgstac.bbox_geom(value->'bbox')\n ELSE NULL\n END as geometry\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION stac_daterange(\n value jsonb\n) RETURNS tstzrange AS $$\nDECLARE\n props jsonb := value;\n dt timestamptz;\n edt timestamptz;\nBEGIN\n IF props ? 'properties' THEN\n props := props->'properties';\n END IF;\n IF\n props ? 'start_datetime'\n AND props->>'start_datetime' IS NOT NULL\n AND props ? 'end_datetime'\n AND props->>'end_datetime' IS NOT NULL\n THEN\n dt := props->>'start_datetime';\n edt := props->>'end_datetime';\n IF dt > edt THEN\n RAISE EXCEPTION 'start_datetime must be < end_datetime';\n END IF;\n ELSE\n dt := props->>'datetime';\n edt := props->>'datetime';\n END IF;\n IF dt is NULL OR edt IS NULL THEN\n RAISE NOTICE 'DT: %, EDT: %', dt, edt;\n RAISE EXCEPTION 'Either datetime (%) or both start_datetime (%) and end_datetime (%) must be set.', props->>'datetime',props->>'start_datetime',props->>'end_datetime';\n END IF;\n RETURN tstzrange(dt, edt, '[]');\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_datetime(value jsonb) RETURNS timestamptz AS $$\n SELECT lower(stac_daterange(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_end_datetime(value jsonb) RETURNS timestamptz AS $$\n SELECT upper(stac_daterange(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\n\nCREATE TABLE IF NOT EXISTS stac_extensions(\n name text PRIMARY KEY,\n url text,\n enbabled_by_default boolean NOT NULL DEFAULT TRUE,\n enableable boolean NOT NULL DEFAULT TRUE\n);\n\nINSERT INTO stac_extensions (name, url) VALUES\n ('fields', 'https://api.stacspec.org/v1.0.0-beta.5/item-search#fields'),\n ('sort','https://api.stacspec.org/v1.0.0-beta.5/item-search#sort'),\n ('context','https://api.stacspec.org/v1.0.0-beta.5/item-search#context'),\n ('filter', 'https://api.stacspec.org/v1.0.0-beta.5/item-search#filter'),\n ('query', 'https://api.stacspec.org/v1.0.0-beta.5/item-search#query')\nON CONFLICT (name) DO UPDATE SET url=EXCLUDED.url;\n\n\n\nCREATE OR REPLACE FUNCTION collection_base_item(content jsonb) RETURNS jsonb AS $$\n SELECT jsonb_build_object(\n 'type', 'Feature',\n 'stac_version', content->'stac_version',\n 'assets', content->'item_assets',\n 'collection', content->'id'\n );\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TYPE partition_trunc_strategy AS ENUM ('year', 'month');\n\nCREATE TABLE IF NOT EXISTS collections (\n key bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,\n id text GENERATED ALWAYS AS (content->>'id') STORED UNIQUE,\n content JSONB NOT NULL,\n base_item jsonb GENERATED ALWAYS AS (pgstac.collection_base_item(content)) STORED,\n partition_trunc partition_trunc_strategy\n);\n\n\nCREATE OR REPLACE FUNCTION collection_base_item(cid text) RETURNS jsonb AS $$\n SELECT pgstac.collection_base_item(content) FROM pgstac.collections WHERE id = cid LIMIT 1;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION table_empty(text) RETURNS boolean AS $$\nDECLARE\n retval boolean;\nBEGIN\n EXECUTE format($q$\n SELECT NOT EXISTS (SELECT 1 FROM %I LIMIT 1)\n $q$,\n $1\n ) INTO retval;\n RETURN retval;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION collections_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n q text;\n partition_name text := format('_items_%s', NEW.key);\n partition_exists boolean := false;\n partition_empty boolean := true;\n err_context text;\n loadtemp boolean := FALSE;\nBEGIN\n RAISE NOTICE 'Collection Trigger. % %', NEW.id, NEW.key;\n SELECT relid::text INTO partition_name\n FROM pg_partition_tree('items')\n WHERE relid::text = partition_name;\n IF FOUND THEN\n partition_exists := true;\n partition_empty := table_empty(partition_name);\n ELSE\n partition_exists := false;\n partition_empty := true;\n partition_name := format('_items_%s', NEW.key);\n END IF;\n IF TG_OP = 'UPDATE' AND NEW.partition_trunc IS DISTINCT FROM OLD.partition_trunc AND partition_empty THEN\n q := format($q$\n DROP TABLE IF EXISTS %I CASCADE;\n $q$,\n partition_name\n );\n EXECUTE q;\n END IF;\n IF TG_OP = 'UPDATE' AND NEW.partition_trunc IS DISTINCT FROM OLD.partition_trunc AND partition_exists AND NOT partition_empty THEN\n q := format($q$\n CREATE TEMP TABLE changepartitionstaging ON COMMIT DROP AS SELECT * FROM %I;\n DROP TABLE IF EXISTS %I CASCADE;\n $q$,\n partition_name,\n partition_name\n );\n EXECUTE q;\n loadtemp := TRUE;\n partition_empty := TRUE;\n partition_exists := FALSE;\n END IF;\n IF TG_OP = 'UPDATE' AND NEW.partition_trunc IS NOT DISTINCT FROM OLD.partition_trunc THEN\n RETURN NEW;\n END IF;\n IF NEW.partition_trunc IS NULL AND partition_empty THEN\n RAISE NOTICE '% % % %',\n partition_name,\n NEW.id,\n concat(partition_name,'_id_idx'),\n partition_name\n ;\n q := format($q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n $q$,\n partition_name,\n NEW.id,\n concat(partition_name,'_id_idx'),\n partition_name\n );\n RAISE NOTICE 'q: %', q;\n BEGIN\n EXECUTE q;\n EXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', partition_name;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\n END;\n\n ALTER TABLE partitions DISABLE TRIGGER partitions_delete_trigger;\n DELETE FROM partitions WHERE collection=NEW.id AND name=partition_name;\n ALTER TABLE partitions ENABLE TRIGGER partitions_delete_trigger;\n\n INSERT INTO partitions (collection, name) VALUES (NEW.id, partition_name);\n ELSIF partition_empty THEN\n q := format($q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L)\n PARTITION BY RANGE (datetime);\n $q$,\n partition_name,\n NEW.id\n );\n RAISE NOTICE 'q: %', q;\n BEGIN\n EXECUTE q;\n EXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', partition_name;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\n END;\n ALTER TABLE partitions DISABLE TRIGGER partitions_delete_trigger;\n DELETE FROM partitions WHERE collection=NEW.id AND name=partition_name;\n ALTER TABLE partitions ENABLE TRIGGER partitions_delete_trigger;\n ELSE\n RAISE EXCEPTION 'Cannot modify partition % unless empty', partition_name;\n END IF;\n IF loadtemp THEN\n RAISE NOTICE 'Moving data into new partitions.';\n q := format($q$\n WITH p AS (\n SELECT\n collection,\n datetime as datetime,\n end_datetime as end_datetime,\n (partition_name(\n collection,\n datetime\n )).partition_name as name\n FROM changepartitionstaging\n )\n INSERT INTO partitions (collection, datetime_range, end_datetime_range)\n SELECT\n collection,\n tstzrange(min(datetime), max(datetime), '[]') as datetime_range,\n tstzrange(min(end_datetime), max(end_datetime), '[]') as end_datetime_range\n FROM p\n GROUP BY collection, name\n ON CONFLICT (name) DO UPDATE SET\n datetime_range = EXCLUDED.datetime_range,\n end_datetime_range = EXCLUDED.end_datetime_range\n ;\n INSERT INTO %I SELECT * FROM changepartitionstaging;\n DROP TABLE IF EXISTS changepartitionstaging;\n $q$,\n partition_name\n );\n EXECUTE q;\n END IF;\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE TRIGGER collections_trigger AFTER INSERT OR UPDATE ON collections FOR EACH ROW\nEXECUTE FUNCTION collections_trigger_func();\n\nCREATE OR REPLACE FUNCTION partition_collection(collection text, strategy partition_trunc_strategy) RETURNS text AS $$\n UPDATE collections SET partition_trunc=strategy WHERE id=collection RETURNING partition_trunc;\n$$ LANGUAGE SQL;\n\nCREATE TABLE IF NOT EXISTS partitions (\n collection text REFERENCES collections(id) ON DELETE CASCADE,\n name text PRIMARY KEY,\n partition_range tstzrange NOT NULL DEFAULT tstzrange('-infinity'::timestamptz,'infinity'::timestamptz, '[]'),\n datetime_range tstzrange,\n end_datetime_range tstzrange,\n CONSTRAINT prange EXCLUDE USING GIST (\n collection WITH =,\n partition_range WITH &&\n )\n) WITH (FILLFACTOR=90);\nCREATE INDEX partitions_range_idx ON partitions USING GIST(partition_range);\n\nCREATE OR REPLACE FUNCTION partitions_delete_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n q text;\nBEGIN\n RAISE NOTICE 'Partition Delete Trigger. %', OLD.name;\n EXECUTE format($q$\n DROP TABLE IF EXISTS %I CASCADE;\n $q$,\n OLD.name\n );\n RAISE NOTICE 'Dropped partition.';\n RETURN OLD;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER partitions_delete_trigger BEFORE DELETE ON partitions FOR EACH ROW\nEXECUTE FUNCTION partitions_delete_trigger_func();\n\nCREATE OR REPLACE FUNCTION partition_name(\n IN collection text,\n IN dt timestamptz,\n OUT partition_name text,\n OUT partition_range tstzrange\n) AS $$\nDECLARE\n c RECORD;\n parent_name text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n parent_name := format('_items_%s', c.key);\n\n\n IF c.partition_trunc = 'year' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYY'));\n ELSIF c.partition_trunc = 'month' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYYMM'));\n ELSE\n partition_name := parent_name;\n partition_range := tstzrange('-infinity'::timestamptz, 'infinity'::timestamptz, '[]');\n END IF;\n IF partition_range IS NULL THEN\n partition_range := tstzrange(\n date_trunc(c.partition_trunc::text, dt),\n date_trunc(c.partition_trunc::text, dt) + concat('1 ', c.partition_trunc)::interval\n );\n END IF;\n RETURN;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\n\nCREATE OR REPLACE FUNCTION partitions_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n q text;\n cq text;\n parent_name text;\n partition_trunc text;\n partition_name text := NEW.name;\n partition_exists boolean := false;\n partition_empty boolean := true;\n partition_range tstzrange;\n datetime_range tstzrange;\n end_datetime_range tstzrange;\n err_context text;\n mindt timestamptz := lower(NEW.datetime_range);\n maxdt timestamptz := upper(NEW.datetime_range);\n minedt timestamptz := lower(NEW.end_datetime_range);\n maxedt timestamptz := upper(NEW.end_datetime_range);\n t_mindt timestamptz;\n t_maxdt timestamptz;\n t_minedt timestamptz;\n t_maxedt timestamptz;\nBEGIN\n RAISE NOTICE 'Partitions Trigger. %', NEW;\n datetime_range := NEW.datetime_range;\n end_datetime_range := NEW.end_datetime_range;\n\n SELECT\n format('_items_%s', key),\n c.partition_trunc::text\n INTO\n parent_name,\n partition_trunc\n FROM pgstac.collections c\n WHERE c.id = NEW.collection;\n SELECT (pgstac.partition_name(NEW.collection, mindt)).* INTO partition_name, partition_range;\n NEW.name := partition_name;\n\n IF partition_range IS NULL OR partition_range = 'empty'::tstzrange THEN\n partition_range := tstzrange('-infinity'::timestamptz, 'infinity'::timestamptz, '[]');\n END IF;\n\n NEW.partition_range := partition_range;\n IF TG_OP = 'UPDATE' THEN\n mindt := least(mindt, lower(OLD.datetime_range));\n maxdt := greatest(maxdt, upper(OLD.datetime_range));\n minedt := least(minedt, lower(OLD.end_datetime_range));\n maxedt := greatest(maxedt, upper(OLD.end_datetime_range));\n NEW.datetime_range := tstzrange(mindt, maxdt, '[]');\n NEW.end_datetime_range := tstzrange(minedt, maxedt, '[]');\n END IF;\n IF TG_OP = 'INSERT' THEN\n\n IF partition_range != tstzrange('-infinity'::timestamptz, 'infinity'::timestamptz, '[]') THEN\n\n RAISE NOTICE '% % %', partition_name, parent_name, partition_range;\n q := format($q$\n CREATE TABLE IF NOT EXISTS %I partition OF %I FOR VALUES FROM (%L) TO (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n $q$,\n partition_name,\n parent_name,\n lower(partition_range),\n upper(partition_range),\n format('%s_pkey', partition_name),\n partition_name\n );\n BEGIN\n EXECUTE q;\n EXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', partition_name;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\n END;\n END IF;\n\n END IF;\n\n -- Update constraints\n EXECUTE format($q$\n SELECT\n min(datetime),\n max(datetime),\n min(end_datetime),\n max(end_datetime)\n FROM %I;\n $q$, partition_name)\n INTO t_mindt, t_maxdt, t_minedt, t_maxedt;\n mindt := least(mindt, t_mindt);\n maxdt := greatest(maxdt, t_maxdt);\n minedt := least(mindt, minedt, t_minedt);\n maxedt := greatest(maxdt, maxedt, t_maxedt);\n\n mindt := date_trunc(coalesce(partition_trunc, 'year'), mindt);\n maxdt := date_trunc(coalesce(partition_trunc, 'year'), maxdt - '1 second'::interval) + concat('1 ',coalesce(partition_trunc, 'year'))::interval;\n minedt := date_trunc(coalesce(partition_trunc, 'year'), minedt);\n maxedt := date_trunc(coalesce(partition_trunc, 'year'), maxedt - '1 second'::interval) + concat('1 ',coalesce(partition_trunc, 'year'))::interval;\n\n\n IF mindt IS NOT NULL AND maxdt IS NOT NULL AND minedt IS NOT NULL AND maxedt IS NOT NULL THEN\n NEW.datetime_range := tstzrange(mindt, maxdt, '[]');\n NEW.end_datetime_range := tstzrange(minedt, maxedt, '[]');\n IF\n TG_OP='UPDATE'\n AND OLD.datetime_range @> NEW.datetime_range\n AND OLD.end_datetime_range @> NEW.end_datetime_range\n THEN\n RAISE NOTICE 'Range unchanged, not updating constraints.';\n ELSE\n\n RAISE NOTICE '\n SETTING CONSTRAINTS\n mindt: %, maxdt: %\n minedt: %, maxedt: %\n ', mindt, maxdt, minedt, maxedt;\n IF partition_trunc IS NULL THEN\n cq := format($q$\n ALTER TABLE %7$I\n DROP CONSTRAINT IF EXISTS %1$I,\n DROP CONSTRAINT IF EXISTS %2$I,\n ADD CONSTRAINT %1$I\n CHECK (\n (datetime >= %3$L)\n AND (datetime <= %4$L)\n AND (end_datetime >= %5$L)\n AND (end_datetime <= %6$L)\n ) NOT VALID\n ;\n ALTER TABLE %7$I\n VALIDATE CONSTRAINT %1$I;\n $q$,\n format('%s_dt', partition_name),\n format('%s_edt', partition_name),\n mindt,\n maxdt,\n minedt,\n maxedt,\n partition_name\n );\n ELSE\n cq := format($q$\n ALTER TABLE %5$I\n DROP CONSTRAINT IF EXISTS %1$I,\n DROP CONSTRAINT IF EXISTS %2$I,\n ADD CONSTRAINT %2$I\n CHECK ((end_datetime >= %3$L) AND (end_datetime <= %4$L)) NOT VALID\n ;\n ALTER TABLE %5$I\n VALIDATE CONSTRAINT %2$I;\n $q$,\n format('%s_dt', partition_name),\n format('%s_edt', partition_name),\n minedt,\n maxedt,\n partition_name\n );\n\n END IF;\n RAISE NOTICE 'Altering Constraints. %', cq;\n EXECUTE cq;\n END IF;\n ELSE\n NEW.datetime_range = NULL;\n NEW.end_datetime_range = NULL;\n\n cq := format($q$\n ALTER TABLE %3$I\n DROP CONSTRAINT IF EXISTS %1$I,\n DROP CONSTRAINT IF EXISTS %2$I,\n ADD CONSTRAINT %1$I\n CHECK ((datetime IS NULL AND end_datetime IS NULL)) NOT VALID\n ;\n ALTER TABLE %3$I\n VALIDATE CONSTRAINT %1$I;\n $q$,\n format('%s_dt', partition_name),\n format('%s_edt', partition_name),\n partition_name\n );\n EXECUTE cq;\n END IF;\n\n RETURN NEW;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER partitions_trigger BEFORE INSERT OR UPDATE ON partitions FOR EACH ROW\nEXECUTE FUNCTION partitions_trigger_func();\n\n\nCREATE OR REPLACE FUNCTION create_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection(data jsonb) RETURNS VOID AS $$\nDECLARE\n out collections%ROWTYPE;\nBEGIN\n UPDATE collections SET content=data WHERE id = data->>'id' RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ON CONFLICT (id) DO\n UPDATE\n SET content=EXCLUDED.content\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION delete_collection(_id text) RETURNS VOID AS $$\nDECLARE\n out collections%ROWTYPE;\nBEGIN\n DELETE FROM collections WHERE id = _id RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION get_collection(id text) RETURNS jsonb AS $$\n SELECT content FROM collections\n WHERE id=$1\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION all_collections() RETURNS jsonb AS $$\n SELECT jsonb_agg(content) FROM collections;\n;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac, public;\nCREATE TABLE queryables (\n id bigint GENERATED ALWAYS AS identity PRIMARY KEY,\n name text UNIQUE NOT NULL,\n collection_ids text[], -- used to determine what partitions to create indexes on\n definition jsonb,\n property_path text,\n property_wrapper text,\n property_index_type text\n);\nCREATE INDEX queryables_name_idx ON queryables (name);\nCREATE INDEX queryables_property_wrapper_idx ON queryables (property_wrapper);\n\n\nINSERT INTO queryables (name, definition) VALUES\n('id', '{\"title\": \"Item ID\",\"description\": \"Item identifier\",\"$ref\": \"https://schemas.stacspec.org/v1.0.0/item-spec/json-schema/item.json#/definitions/core/allOf/2/properties/id\"}'),\n('datetime','{\"description\": \"Datetime\",\"type\": \"string\",\"title\": \"Acquired\",\"format\": \"date-time\",\"pattern\": \"(\\\\+00:00|Z)$\"}'),\n('geometry', '{\"title\": \"Item Geometry\",\"description\": \"Item Geometry\",\"$ref\": \"https://geojson.org/schema/Feature.json\"}')\nON CONFLICT DO NOTHING;\n\nINSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n('eo:cloud_cover','{\"$ref\": \"https://stac-extensions.github.io/eo/v1.0.0/schema.json#/definitions/fieldsproperties/eo:cloud_cover\"}','to_int','BTREE')\nON CONFLICT DO NOTHING;\n\nCREATE OR REPLACE FUNCTION array_to_path(arr text[]) RETURNS text AS $$\n SELECT string_agg(\n quote_literal(v),\n '->'\n ) FROM unnest(arr) v;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\n\n\nCREATE OR REPLACE FUNCTION queryable(\n IN dotpath text,\n OUT path text,\n OUT expression text,\n OUT wrapper text,\n OUT nulled_wrapper text\n) AS $$\nDECLARE\n q RECORD;\n path_elements text[];\nBEGIN\n IF dotpath IN ('id', 'geometry', 'datetime', 'end_datetime', 'collection') THEN\n path := dotpath;\n expression := dotpath;\n wrapper := NULL;\n RETURN;\n END IF;\n SELECT * INTO q FROM queryables\n WHERE\n name=dotpath\n OR name = 'properties.' || dotpath\n OR name = replace(dotpath, 'properties.', '')\n ;\n IF q.property_wrapper IS NULL THEN\n IF q.definition->>'type' = 'number' THEN\n wrapper := 'to_float';\n nulled_wrapper := wrapper;\n ELSIF q.definition->>'format' = 'date-time' THEN\n wrapper := 'to_tstz';\n nulled_wrapper := wrapper;\n ELSE\n nulled_wrapper := NULL;\n wrapper := 'to_text';\n END IF;\n ELSE\n wrapper := q.property_wrapper;\n nulled_wrapper := wrapper;\n END IF;\n IF q.property_path IS NOT NULL THEN\n path := q.property_path;\n ELSE\n path_elements := string_to_array(dotpath, '.');\n IF path_elements[1] IN ('links', 'assets', 'stac_version', 'stac_extensions') THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSIF path_elements[1] = 'properties' THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSE\n path := format($F$content->'properties'->%s$F$, array_to_path(path_elements));\n END IF;\n END IF;\n expression := format('%I(%s)', wrapper, path);\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\nCREATE OR REPLACE FUNCTION create_queryable_indexes() RETURNS VOID AS $$\nDECLARE\n queryable RECORD;\n q text;\nBEGIN\n FOR queryable IN\n SELECT\n queryables.id as qid,\n CASE WHEN collections.key IS NULL THEN 'items' ELSE format('_items_%s',collections.key) END AS part,\n property_index_type,\n expression\n FROM\n queryables\n LEFT JOIN collections ON (collections.id = ANY (queryables.collection_ids))\n JOIN LATERAL queryable(queryables.name) ON (queryables.property_index_type IS NOT NULL)\n LOOP\n q := format(\n $q$\n CREATE INDEX IF NOT EXISTS %I ON %I USING %s ((%s));\n $q$,\n format('%s_%s_idx', queryable.part, queryable.qid),\n queryable.part,\n COALESCE(queryable.property_index_type, 'to_text'),\n queryable.expression\n );\n RAISE NOTICE '%',q;\n EXECUTE q;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION queryables_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\nBEGIN\nPERFORM create_queryable_indexes();\nRETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER queryables_trigger AFTER INSERT OR UPDATE ON queryables\nFOR EACH STATEMENT EXECUTE PROCEDURE queryables_trigger_func();\n\nCREATE TRIGGER queryables_collection_trigger AFTER INSERT OR UPDATE ON collections\nFOR EACH STATEMENT EXECUTE PROCEDURE queryables_trigger_func();\n\nCREATE OR REPLACE FUNCTION get_queryables(_collection_ids text[] DEFAULT NULL) RETURNS jsonb AS $$\n SELECT\n jsonb_build_object(\n '$schema', 'http://json-schema.org/draft-07/schema#',\n '$id', 'https://example.org/queryables',\n 'type', 'object',\n 'title', 'Stac Queryables.',\n 'properties', jsonb_object_agg(\n name,\n definition\n )\n )\n FROM queryables\n WHERE\n _collection_ids IS NULL OR\n cardinality(_collection_ids) = 0 OR\n collection_ids IS NULL OR\n _collection_ids && collection_ids\n ;\n$$ LANGUAGE SQL STABLE;\n\nCREATE OR REPLACE FUNCTION get_queryables(_collection text DEFAULT NULL) RETURNS jsonb AS $$\n SELECT\n CASE\n WHEN _collection IS NULL THEN get_queryables(NULL::text[])\n ELSE get_queryables(ARRAY[_collection])\n END\n ;\n$$ LANGUAGE SQL;\n\n\nCREATE OR REPLACE FUNCTION missing_queryables(_collection text, _tablesample int DEFAULT 5) RETURNS TABLE(collection text, name text, definition jsonb, property_wrapper text) AS $$\nDECLARE\n q text;\n _partition text;\n explain_json json;\n psize bigint;\nBEGIN\n SELECT format('_items_%s', key) INTO _partition FROM collections WHERE id=_collection;\n\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM %I;', _partition)\n INTO explain_json;\n psize := explain_json->0->'Plan'->'Plan Rows';\n IF _tablesample * .01 * psize < 10 THEN\n _tablesample := 100;\n END IF;\n RAISE NOTICE 'Using tablesample % to find missing queryables from % % that has ~% rows', _tablesample, _collection, _partition, psize;\n\n q := format(\n $q$\n WITH q AS (\n SELECT * FROM queryables\n WHERE\n collection_ids IS NULL\n OR %L = ANY(collection_ids)\n ), t AS (\n SELECT\n content->'properties' AS properties\n FROM\n %I\n TABLESAMPLE SYSTEM(%L)\n ), p AS (\n SELECT DISTINCT ON (key)\n key,\n value\n FROM t\n JOIN LATERAL jsonb_each(properties) ON TRUE\n LEFT JOIN q ON (q.name=key)\n WHERE q.definition IS NULL\n )\n SELECT\n %L,\n key,\n jsonb_build_object('type',jsonb_typeof(value)) as definition,\n CASE jsonb_typeof(value)\n WHEN 'number' THEN 'to_float'\n WHEN 'array' THEN 'to_text_array'\n ELSE 'to_text'\n END\n FROM p;\n $q$,\n _collection,\n _partition,\n _tablesample,\n _collection\n );\n RETURN QUERY EXECUTE q;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION missing_queryables(_tablesample int DEFAULT 5) RETURNS TABLE(collection_ids text[], name text, definition jsonb, property_wrapper text) AS $$\n SELECT\n array_agg(collection),\n name,\n definition,\n property_wrapper\n FROM\n collections\n JOIN LATERAL\n missing_queryables(id, _tablesample) c\n ON TRUE\n GROUP BY\n 2,3,4\n ORDER BY 2,1\n ;\n$$ LANGUAGE SQL;\nCREATE OR REPLACE FUNCTION parse_dtrange(\n _indate jsonb,\n relative_base timestamptz DEFAULT date_trunc('hour', CURRENT_TIMESTAMP)\n) RETURNS tstzrange AS $$\nDECLARE\n timestrs text[];\n s timestamptz;\n e timestamptz;\nBEGIN\n timestrs :=\n CASE\n WHEN _indate ? 'timestamp' THEN\n ARRAY[_indate->>'timestamp']\n WHEN _indate ? 'interval' THEN\n to_text_array(_indate->'interval')\n WHEN jsonb_typeof(_indate) = 'array' THEN\n to_text_array(_indate)\n ELSE\n regexp_split_to_array(\n _indate->>0,\n '/'\n )\n END;\n RAISE NOTICE 'TIMESTRS %', timestrs;\n IF cardinality(timestrs) = 1 THEN\n IF timestrs[1] ILIKE 'P%' THEN\n RETURN tstzrange(relative_base - upper(timestrs[1])::interval, relative_base, '[)');\n END IF;\n s := timestrs[1]::timestamptz;\n RETURN tstzrange(s, s, '[]');\n END IF;\n\n IF cardinality(timestrs) != 2 THEN\n RAISE EXCEPTION 'Timestamp cannot have more than 2 values';\n END IF;\n\n IF timestrs[1] = '..' THEN\n s := '-infinity'::timestamptz;\n e := timestrs[2]::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] = '..' THEN\n s := timestrs[1]::timestamptz;\n e := 'infinity'::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[1] ILIKE 'P%' AND timestrs[2] NOT ILIKE 'P%' THEN\n e := timestrs[2]::timestamptz;\n s := e - upper(timestrs[1])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] ILIKE 'P%' AND timestrs[1] NOT ILIKE 'P%' THEN\n s := timestrs[1]::timestamptz;\n e := s + upper(timestrs[2])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n s := timestrs[1]::timestamptz;\n e := timestrs[2]::timestamptz;\n\n RETURN tstzrange(s,e,'[)');\n\n RETURN NULL;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT PARALLEL SAFE SET TIME ZONE 'UTC';\n\nCREATE OR REPLACE FUNCTION parse_dtrange(\n _indate text,\n relative_base timestamptz DEFAULT CURRENT_TIMESTAMP\n) RETURNS tstzrange AS $$\n SELECT parse_dtrange(to_jsonb(_indate), relative_base);\n$$ LANGUAGE SQL STABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION temporal_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\n ll text := 'datetime';\n lh text := 'end_datetime';\n rrange tstzrange;\n rl text;\n rh text;\n outq text;\nBEGIN\n rrange := parse_dtrange(args->1);\n RAISE NOTICE 'Constructing temporal query OP: %, ARGS: %, RRANGE: %', op, args, rrange;\n op := lower(op);\n rl := format('%L::timestamptz', lower(rrange));\n rh := format('%L::timestamptz', upper(rrange));\n outq := CASE op\n WHEN 't_before' THEN 'lh < rl'\n WHEN 't_after' THEN 'll > rh'\n WHEN 't_meets' THEN 'lh = rl'\n WHEN 't_metby' THEN 'll = rh'\n WHEN 't_overlaps' THEN 'll < rl AND rl < lh < rh'\n WHEN 't_overlappedby' THEN 'rl < ll < rh AND lh > rh'\n WHEN 't_starts' THEN 'll = rl AND lh < rh'\n WHEN 't_startedby' THEN 'll = rl AND lh > rh'\n WHEN 't_during' THEN 'll > rl AND lh < rh'\n WHEN 't_contains' THEN 'll < rl AND lh > rh'\n WHEN 't_finishes' THEN 'll > rl AND lh = rh'\n WHEN 't_finishedby' THEN 'll < rl AND lh = rh'\n WHEN 't_equals' THEN 'll = rl AND lh = rh'\n WHEN 't_disjoint' THEN 'NOT (ll <= rh AND lh >= rl)'\n WHEN 't_intersects' THEN 'll <= rh AND lh >= rl'\n WHEN 'anyinteracts' THEN 'll <= rh AND lh >= rl'\n END;\n outq := regexp_replace(outq, '\\mll\\M', ll);\n outq := regexp_replace(outq, '\\mlh\\M', lh);\n outq := regexp_replace(outq, '\\mrl\\M', rl);\n outq := regexp_replace(outq, '\\mrh\\M', rh);\n outq := format('(%s)', outq);\n RETURN outq;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\n\n\nCREATE OR REPLACE FUNCTION spatial_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\n geom text;\n j jsonb := args->1;\nBEGIN\n op := lower(op);\n RAISE NOTICE 'Constructing spatial query OP: %, ARGS: %', op, args;\n IF op NOT IN ('s_equals','s_disjoint','s_touches','s_within','s_overlaps','s_crosses','s_intersects','intersects','s_contains') THEN\n RAISE EXCEPTION 'Spatial Operator % Not Supported', op;\n END IF;\n op := regexp_replace(op, '^s_', 'st_');\n IF op = 'intersects' THEN\n op := 'st_intersects';\n END IF;\n -- Convert geometry to WKB string\n IF j ? 'type' AND j ? 'coordinates' THEN\n geom := st_geomfromgeojson(j)::text;\n ELSIF jsonb_typeof(j) = 'array' THEN\n geom := bbox_geom(j)::text;\n END IF;\n\n RETURN format('%s(geometry, %L::geometry)', op, geom);\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION query_to_cql2(q jsonb) RETURNS jsonb AS $$\n-- Translates anything passed in through the deprecated \"query\" into equivalent CQL2\nWITH t AS (\n SELECT key as property, value as ops\n FROM jsonb_each(q)\n), t2 AS (\n SELECT property, (jsonb_each(ops)).*\n FROM t WHERE jsonb_typeof(ops) = 'object'\n UNION ALL\n SELECT property, 'eq', ops\n FROM t WHERE jsonb_typeof(ops) != 'object'\n)\nSELECT\n jsonb_strip_nulls(jsonb_build_object(\n 'op', 'and',\n 'args', jsonb_agg(\n jsonb_build_object(\n 'op', key,\n 'args', jsonb_build_array(\n jsonb_build_object('property',property),\n value\n )\n )\n )\n )\n) as qcql FROM t2\n;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\nCREATE OR REPLACE FUNCTION cql1_to_cql2(j jsonb) RETURNS jsonb AS $$\nDECLARE\n args jsonb;\n ret jsonb;\nBEGIN\n RAISE NOTICE 'CQL1_TO_CQL2: %', j;\n IF j ? 'filter' THEN\n RETURN cql1_to_cql2(j->'filter');\n END IF;\n IF j ? 'property' THEN\n RETURN j;\n END IF;\n IF jsonb_typeof(j) = 'array' THEN\n SELECT jsonb_agg(cql1_to_cql2(el)) INTO args FROM jsonb_array_elements(j) el;\n RETURN args;\n END IF;\n IF jsonb_typeof(j) = 'number' THEN\n RETURN j;\n END IF;\n IF jsonb_typeof(j) = 'string' THEN\n RETURN j;\n END IF;\n\n IF jsonb_typeof(j) = 'object' THEN\n SELECT jsonb_build_object(\n 'op', key,\n 'args', cql1_to_cql2(value)\n ) INTO ret\n FROM jsonb_each(j)\n WHERE j IS NOT NULL;\n RETURN ret;\n END IF;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE STRICT;\n\nCREATE TABLE cql2_ops (\n op text PRIMARY KEY,\n template text,\n types text[]\n);\nINSERT INTO cql2_ops (op, template, types) VALUES\n ('eq', '%s = %s', NULL),\n ('neq', '%s != %s', NULL),\n ('ne', '%s != %s', NULL),\n ('!=', '%s != %s', NULL),\n ('<>', '%s != %s', NULL),\n ('lt', '%s < %s', NULL),\n ('lte', '%s <= %s', NULL),\n ('gt', '%s > %s', NULL),\n ('gte', '%s >= %s', NULL),\n ('le', '%s <= %s', NULL),\n ('ge', '%s >= %s', NULL),\n ('=', '%s = %s', NULL),\n ('<', '%s < %s', NULL),\n ('<=', '%s <= %s', NULL),\n ('>', '%s > %s', NULL),\n ('>=', '%s >= %s', NULL),\n ('like', '%s LIKE %s', NULL),\n ('ilike', '%s ILIKE %s', NULL),\n ('+', '%s + %s', NULL),\n ('-', '%s - %s', NULL),\n ('*', '%s * %s', NULL),\n ('/', '%s / %s', NULL),\n ('not', 'NOT (%s)', NULL),\n ('between', '%s BETWEEN %s AND %s', NULL),\n ('isnull', '%s IS NULL', NULL),\n ('upper', 'upper(%s)', NULL),\n ('lower', 'lower(%s)', NULL)\nON CONFLICT (op) DO UPDATE\n SET\n template = EXCLUDED.template\n;\n\n\nCREATE OR REPLACE FUNCTION cql2_query(j jsonb, wrapper text DEFAULT NULL) RETURNS text AS $$\n#variable_conflict use_variable\nDECLARE\n args jsonb := j->'args';\n arg jsonb;\n op text := lower(j->>'op');\n cql2op RECORD;\n literal text;\n _wrapper text;\n leftarg text;\n rightarg text;\nBEGIN\n IF j IS NULL OR (op IS NOT NULL AND args IS NULL) THEN\n RETURN NULL;\n END IF;\n RAISE NOTICE 'CQL2_QUERY: %', j;\n IF j ? 'filter' THEN\n RETURN cql2_query(j->'filter');\n END IF;\n\n IF j ? 'upper' THEN\n RETURN cql2_query(jsonb_build_object('op', 'upper', 'args', j->'upper'));\n END IF;\n\n IF j ? 'lower' THEN\n RETURN cql2_query(jsonb_build_object('op', 'lower', 'args', j->'lower'));\n END IF;\n\n -- Temporal Query\n IF op ilike 't_%' or op = 'anyinteracts' THEN\n RETURN temporal_op_query(op, args);\n END IF;\n\n -- If property is a timestamp convert it to text to use with\n -- general operators\n IF j ? 'timestamp' THEN\n RETURN format('%L::timestamptz', to_tstz(j->'timestamp'));\n END IF;\n IF j ? 'interval' THEN\n RAISE EXCEPTION 'Please use temporal operators when using intervals.';\n RETURN NONE;\n END IF;\n\n -- Spatial Query\n IF op ilike 's_%' or op = 'intersects' THEN\n RETURN spatial_op_query(op, args);\n END IF;\n\n IF op IN ('a_equals','a_contains','a_contained_by','a_overlaps') THEN\n IF args->0 ? 'property' THEN\n leftarg := format('to_text_array(%s)', (queryable(args->0->>'property')).path);\n END IF;\n IF args->1 ? 'property' THEN\n rightarg := format('to_text_array(%s)', (queryable(args->1->>'property')).path);\n END IF;\n RETURN FORMAT(\n '%s %s %s',\n COALESCE(leftarg, quote_literal(to_text_array(args->0))),\n CASE op\n WHEN 'a_equals' THEN '='\n WHEN 'a_contains' THEN '@>'\n WHEN 'a_contained_by' THEN '<@'\n WHEN 'a_overlaps' THEN '&&'\n END,\n COALESCE(rightarg, quote_literal(to_text_array(args->1)))\n );\n END IF;\n\n IF op = 'in' THEN\n RAISE NOTICE 'IN : % % %', args, jsonb_build_array(args->0), args->1;\n args := jsonb_build_array(args->0) || (args->1);\n RAISE NOTICE 'IN2 : %', args;\n END IF;\n\n\n\n IF op = 'between' THEN\n args = jsonb_build_array(\n args->0,\n args->1->0,\n args->1->1\n );\n END IF;\n\n -- Make sure that args is an array and run cql2_query on\n -- each element of the array\n RAISE NOTICE 'ARGS PRE: %', args;\n IF j ? 'args' THEN\n IF jsonb_typeof(args) != 'array' THEN\n args := jsonb_build_array(args);\n END IF;\n\n IF jsonb_path_exists(args, '$[*] ? (@.property == \"id\" || @.property == \"datetime\" || @.property == \"end_datetime\" || @.property == \"collection\")') THEN\n wrapper := NULL;\n ELSE\n -- if any of the arguments are a property, try to get the property_wrapper\n FOR arg IN SELECT jsonb_path_query(args, '$[*] ? (@.property != null)') LOOP\n RAISE NOTICE 'Arg: %', arg;\n wrapper := (queryable(arg->>'property')).nulled_wrapper;\n RAISE NOTICE 'Property: %, Wrapper: %', arg, wrapper;\n IF wrapper IS NOT NULL THEN\n EXIT;\n END IF;\n END LOOP;\n\n -- if the property was not in queryables, see if any args were numbers\n IF\n wrapper IS NULL\n AND jsonb_path_exists(args, '$[*] ? (@.type()==\"number\")')\n THEN\n wrapper := 'to_float';\n END IF;\n wrapper := coalesce(wrapper, 'to_text');\n END IF;\n\n SELECT jsonb_agg(cql2_query(a, wrapper))\n INTO args\n FROM jsonb_array_elements(args) a;\n END IF;\n RAISE NOTICE 'ARGS: %', args;\n\n IF op IN ('and', 'or') THEN\n RETURN\n format(\n '(%s)',\n array_to_string(to_text_array(args), format(' %s ', upper(op)))\n );\n END IF;\n\n IF op = 'in' THEN\n RAISE NOTICE 'IN -- % %', args->0, to_text(args->0);\n RETURN format(\n '%s IN (%s)',\n to_text(args->0),\n array_to_string((to_text_array(args))[2:], ',')\n );\n END IF;\n\n -- Look up template from cql2_ops\n IF j ? 'op' THEN\n SELECT * INTO cql2op FROM cql2_ops WHERE cql2_ops.op ilike op;\n IF FOUND THEN\n -- If specific index set in queryables for a property cast other arguments to that type\n\n RETURN format(\n cql2op.template,\n VARIADIC (to_text_array(args))\n );\n ELSE\n RAISE EXCEPTION 'Operator % Not Supported.', op;\n END IF;\n END IF;\n\n\n IF wrapper IS NOT NULL THEN\n RAISE NOTICE 'Wrapping % with %', j, wrapper;\n IF j ? 'property' THEN\n RETURN format('%I(%s)', wrapper, (queryable(j->>'property')).path);\n ELSE\n RETURN format('%I(%L)', wrapper, j);\n END IF;\n ELSIF j ? 'property' THEN\n RETURN quote_ident(j->>'property');\n END IF;\n\n RETURN quote_literal(to_text(j));\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION paging_dtrange(\n j jsonb\n) RETURNS tstzrange AS $$\nDECLARE\n op text;\n filter jsonb := j->'filter';\n dtrange tstzrange := tstzrange('-infinity'::timestamptz,'infinity'::timestamptz);\n sdate timestamptz := '-infinity'::timestamptz;\n edate timestamptz := 'infinity'::timestamptz;\n jpitem jsonb;\nBEGIN\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n END IF;\n IF NOT (filter @? '$.**.op ? (@ == \"or\" || @ == \"not\")') THEN\n FOR jpitem IN SELECT j FROM jsonb_path_query(filter,'strict $.** ? (@.args[*].property == \"datetime\")'::jsonpath) j LOOP\n op := lower(jpitem->>'op');\n dtrange := parse_dtrange(jpitem->'args'->1);\n IF op IN ('<=', 'lt', 'lte', '<', 'le', 't_before') THEN\n sdate := greatest(sdate,'-infinity');\n edate := least(edate, upper(dtrange));\n ELSIF op IN ('>=', '>', 'gt', 'gte', 'ge', 't_after') THEN\n edate := least(edate, 'infinity');\n sdate := greatest(sdate, lower(dtrange));\n ELSIF op IN ('=', 'eq') THEN\n edate := least(edate, upper(dtrange));\n sdate := greatest(sdate, lower(dtrange));\n END IF;\n RAISE NOTICE '2 OP: %, ARGS: %, DTRANGE: %, SDATE: %, EDATE: %', op, jpitem->'args'->1, dtrange, sdate, edate;\n END LOOP;\n END IF;\n IF sdate > edate THEN\n RETURN 'empty'::tstzrange;\n END IF;\n RETURN tstzrange(sdate,edate, '[]');\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT SET TIME ZONE 'UTC';\n\nCREATE OR REPLACE FUNCTION paging_collections(\n IN j jsonb\n) RETURNS text[] AS $$\nDECLARE\n filter jsonb := j->'filter';\n jpitem jsonb;\n op text;\n args jsonb;\n arg jsonb;\n collections text[];\nBEGIN\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n END IF;\n IF NOT (filter @? '$.**.op ? (@ == \"or\" || @ == \"not\")') THEN\n FOR jpitem IN SELECT j FROM jsonb_path_query(filter,'strict $.** ? (@.args[*].property == \"collection\")'::jsonpath) j LOOP\n RAISE NOTICE 'JPITEM: %', jpitem;\n op := jpitem->>'op';\n args := jpitem->'args';\n IF op IN ('=', 'eq', 'in') THEN\n FOR arg IN SELECT a FROM jsonb_array_elements(args) a LOOP\n IF jsonb_typeof(arg) IN ('string', 'array') THEN\n RAISE NOTICE 'arg: %, collections: %', arg, collections;\n IF collections IS NULL OR collections = '{}'::text[] THEN\n collections := to_text_array(arg);\n ELSE\n collections := array_intersection(collections, to_text_array(arg));\n END IF;\n END IF;\n END LOOP;\n END IF;\n END LOOP;\n END IF;\n IF collections = '{}'::text[] THEN\n RETURN NULL;\n END IF;\n RETURN collections;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\nCREATE TABLE items (\n id text NOT NULL,\n geometry geometry NOT NULL,\n collection text NOT NULL,\n datetime timestamptz NOT NULL,\n end_datetime timestamptz NOT NULL,\n content JSONB NOT NULL\n)\nPARTITION BY LIST (collection)\n;\n\nCREATE INDEX \"datetime_idx\" ON items USING BTREE (datetime DESC, end_datetime ASC);\nCREATE INDEX \"geometry_idx\" ON items USING GIST (geometry);\n\nCREATE STATISTICS datetime_stats (dependencies) on datetime, end_datetime from items;\n\n\nALTER TABLE items ADD CONSTRAINT items_collections_fk FOREIGN KEY (collection) REFERENCES collections(id) ON DELETE CASCADE DEFERRABLE;\n\n\nCREATE OR REPLACE FUNCTION content_slim(_item jsonb) RETURNS jsonb AS $$\n SELECT strip_jsonb(_item - '{id,geometry,collection,type}'::text[], collection_base_item(_item->>'collection')) - '{id,geometry,collection,type}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_dehydrate(content jsonb) RETURNS items AS $$\n SELECT\n content->>'id' as id,\n stac_geom(content) as geometry,\n content->>'collection' as collection,\n stac_datetime(content) as datetime,\n stac_end_datetime(content) as end_datetime,\n content_slim(content) as content\n ;\n$$ LANGUAGE SQL STABLE;\n\nCREATE OR REPLACE FUNCTION include_field(f text, fields jsonb DEFAULT '{}'::jsonb) RETURNS boolean AS $$\nDECLARE\n includes jsonb := fields->'include';\n excludes jsonb := fields->'exclude';\nBEGIN\n IF f IS NULL THEN\n RETURN NULL;\n END IF;\n\n\n IF\n jsonb_typeof(excludes) = 'array'\n AND jsonb_array_length(excludes)>0\n AND excludes ? f\n THEN\n RETURN FALSE;\n END IF;\n\n IF\n (\n jsonb_typeof(includes) = 'array'\n AND jsonb_array_length(includes) > 0\n AND includes ? f\n ) OR\n (\n includes IS NULL\n OR jsonb_typeof(includes) = 'null'\n OR jsonb_array_length(includes) = 0\n )\n THEN\n RETURN TRUE;\n END IF;\n\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nDROP FUNCTION IF EXISTS content_hydrate(jsonb, jsonb, jsonb);\nCREATE OR REPLACE FUNCTION content_hydrate(\n _item jsonb,\n _base_item jsonb,\n fields jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\n SELECT merge_jsonb(\n jsonb_fields(_item, fields),\n jsonb_fields(_base_item, fields)\n );\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION content_hydrate(_item items, _collection collections, fields jsonb DEFAULT '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\n content jsonb;\n base_item jsonb := _collection.base_item;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry, 20)::jsonb;\n END IF;\n output := content_hydrate(\n jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'collection', _item.collection,\n 'type', 'Feature'\n ) || _item.content,\n _collection.base_item,\n fields\n );\n\n RETURN output;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_nonhydrated(\n _item items,\n fields jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry, 20)::jsonb;\n END IF;\n output := jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'collection', _item.collection,\n 'type', 'Feature'\n ) || _item.content;\n RETURN output;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_hydrate(_item items, fields jsonb DEFAULT '{}'::jsonb) RETURNS jsonb AS $$\n SELECT content_hydrate(\n _item,\n (SELECT c FROM collections c WHERE id=_item.collection LIMIT 1),\n fields\n );\n$$ LANGUAGE SQL STABLE;\n\n\nCREATE UNLOGGED TABLE items_staging (\n content JSONB NOT NULL\n);\nCREATE UNLOGGED TABLE items_staging_ignore (\n content JSONB NOT NULL\n);\nCREATE UNLOGGED TABLE items_staging_upsert (\n content JSONB NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION items_staging_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p record;\n _partitions text[];\n ts timestamptz := clock_timestamp();\nBEGIN\n RAISE NOTICE 'Creating Partitions. %', clock_timestamp() - ts;\n WITH ranges AS (\n SELECT\n n.content->>'collection' as collection,\n stac_daterange(n.content->'properties') as dtr\n FROM newdata n\n ), p AS (\n SELECT\n collection,\n lower(dtr) as datetime,\n upper(dtr) as end_datetime,\n (partition_name(\n collection,\n lower(dtr)\n )).partition_name as name\n FROM ranges\n )\n INSERT INTO partitions (collection, datetime_range, end_datetime_range)\n SELECT\n collection,\n tstzrange(min(datetime), max(datetime), '[]') as datetime_range,\n tstzrange(min(end_datetime), max(end_datetime), '[]') as end_datetime_range\n FROM p\n GROUP BY collection, name\n ON CONFLICT (name) DO UPDATE SET\n datetime_range = EXCLUDED.datetime_range,\n end_datetime_range = EXCLUDED.end_datetime_range\n ;\n\n RAISE NOTICE 'Doing the insert. %', clock_timestamp() - ts;\n IF TG_TABLE_NAME = 'items_staging' THEN\n INSERT INTO items\n SELECT\n (content_dehydrate(content)).*\n FROM newdata;\n DELETE FROM items_staging;\n ELSIF TG_TABLE_NAME = 'items_staging_ignore' THEN\n INSERT INTO items\n SELECT\n (content_dehydrate(content)).*\n FROM newdata\n ON CONFLICT DO NOTHING;\n DELETE FROM items_staging_ignore;\n ELSIF TG_TABLE_NAME = 'items_staging_upsert' THEN\n WITH staging_formatted AS (\n SELECT (content_dehydrate(content)).* FROM newdata\n ), deletes AS (\n DELETE FROM items i USING staging_formatted s\n WHERE\n i.id = s.id\n AND i.collection = s.collection\n AND i IS DISTINCT FROM s\n RETURNING i.id, i.collection\n )\n INSERT INTO items\n SELECT s.* FROM\n staging_formatted s\n JOIN deletes d\n USING (id, collection);\n DELETE FROM items_staging_upsert;\n END IF;\n RAISE NOTICE 'Done. %', clock_timestamp() - ts;\n\n RETURN NULL;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER items_staging_insert_trigger AFTER INSERT ON items_staging REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\nCREATE TRIGGER items_staging_insert_ignore_trigger AFTER INSERT ON items_staging_ignore REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\nCREATE TRIGGER items_staging_insert_upsert_trigger AFTER INSERT ON items_staging_upsert REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\n\n\n\nCREATE OR REPLACE FUNCTION item_by_id(_id text, _collection text DEFAULT NULL) RETURNS items AS\n$$\nDECLARE\n i items%ROWTYPE;\nBEGIN\n SELECT * INTO i FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection) LIMIT 1;\n RETURN i;\nEND;\n$$ LANGUAGE PLPGSQL STABLE SECURITY DEFINER SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION get_item(_id text, _collection text DEFAULT NULL) RETURNS jsonb AS $$\n SELECT content_hydrate(items) FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection);\n$$ LANGUAGE SQL STABLE SECURITY DEFINER SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION delete_item(_id text, _collection text DEFAULT NULL) RETURNS VOID AS $$\nDECLARE\nout items%ROWTYPE;\nBEGIN\n DELETE FROM items WHERE id = _id AND (_collection IS NULL OR collection=_collection) RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n--/*\nCREATE OR REPLACE FUNCTION create_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION update_item(content jsonb) RETURNS VOID AS $$\nDECLARE\n old items %ROWTYPE;\n out items%ROWTYPE;\nBEGIN\n PERFORM delete_item(content->>'id', content->>'collection');\n PERFORM create_item(content);\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION create_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION collection_bbox(id text) RETURNS jsonb AS $$\n SELECT (replace(replace(replace(st_extent(geometry)::text,'BOX(','[['),')',']]'),' ',','))::jsonb\n FROM items WHERE collection=$1;\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION collection_temporal_extent(id text) RETURNS jsonb AS $$\n SELECT to_jsonb(array[array[min(datetime)::text, max(datetime)::text]])\n FROM items WHERE collection=$1;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection_extents() RETURNS VOID AS $$\nUPDATE collections SET\n content = content ||\n jsonb_build_object(\n 'extent', jsonb_build_object(\n 'spatial', jsonb_build_object(\n 'bbox', collection_bbox(collections.id)\n ),\n 'temporal', jsonb_build_object(\n 'interval', collection_temporal_extent(collections.id)\n )\n )\n )\n;\n$$ LANGUAGE SQL;\nCREATE VIEW partition_steps AS\nSELECT\n name,\n date_trunc('month',lower(datetime_range)) as sdate,\n date_trunc('month', upper(datetime_range)) + '1 month'::interval as edate\n FROM partitions WHERE datetime_range IS NOT NULL AND datetime_range != 'empty'::tstzrange\n ORDER BY datetime_range ASC\n;\n\nCREATE OR REPLACE FUNCTION chunker(\n IN _where text,\n OUT s timestamptz,\n OUT e timestamptz\n) RETURNS SETOF RECORD AS $$\nDECLARE\n explain jsonb;\nBEGIN\n IF _where IS NULL THEN\n _where := ' TRUE ';\n END IF;\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s;', _where)\n INTO explain;\n\n RETURN QUERY\n WITH t AS (\n SELECT j->>0 as p FROM\n jsonb_path_query(\n explain,\n 'strict $.**.\"Relation Name\" ? (@ != null)'\n ) j\n ),\n parts AS (\n SELECT sdate, edate FROM t JOIN partition_steps ON (t.p = name)\n ),\n times AS (\n SELECT sdate FROM parts\n UNION\n SELECT edate FROM parts\n ),\n uniq AS (\n SELECT DISTINCT sdate FROM times ORDER BY sdate\n ),\n last AS (\n SELECT sdate, lead(sdate, 1) over () as edate FROM uniq\n )\n SELECT sdate, edate FROM last WHERE edate IS NOT NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION partition_queries(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN partitions text[] DEFAULT NULL\n) RETURNS SETOF text AS $$\nDECLARE\n query text;\n sdate timestamptz;\n edate timestamptz;\nBEGIN\nIF _where IS NULL OR trim(_where) = '' THEN\n _where = ' TRUE ';\nEND IF;\nRAISE NOTICE 'Getting chunks for % %', _where, _orderby;\nIF _orderby ILIKE 'datetime d%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 DESC LOOP\n RETURN NEXT format($q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n $q$,\n sdate,\n edate,\n _where,\n _orderby\n );\n END LOOP;\nELSIF _orderby ILIKE 'datetime a%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 ASC LOOP\n RETURN NEXT format($q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n $q$,\n sdate,\n edate,\n _where,\n _orderby\n );\n END LOOP;\nELSE\n query := format($q$\n SELECT * FROM items\n WHERE %s\n ORDER BY %s\n $q$, _where, _orderby\n );\n\n RETURN NEXT query;\n RETURN;\nEND IF;\n\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION partition_query_view(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN _limit int DEFAULT 10\n) RETURNS text AS $$\n WITH p AS (\n SELECT * FROM partition_queries(_where, _orderby) p\n )\n SELECT\n CASE WHEN EXISTS (SELECT 1 FROM p) THEN\n (SELECT format($q$\n SELECT * FROM (\n %s\n ) total LIMIT %s\n $q$,\n string_agg(\n format($q$ SELECT * FROM ( %s ) AS sub $q$, p),\n '\n UNION ALL\n '\n ),\n _limit\n ))\n ELSE NULL\n END FROM p;\n$$ LANGUAGE SQL IMMUTABLE;\n\n\n\n\nCREATE OR REPLACE FUNCTION stac_search_to_where(j jsonb) RETURNS text AS $$\nDECLARE\n where_segments text[];\n _where text;\n dtrange tstzrange;\n collections text[];\n geom geometry;\n sdate timestamptz;\n edate timestamptz;\n filterlang text;\n filter jsonb := j->'filter';\nBEGIN\n IF j ? 'ids' THEN\n where_segments := where_segments || format('id = ANY (%L) ', to_text_array(j->'ids'));\n END IF;\n\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n where_segments := where_segments || format('collection = ANY (%L) ', collections);\n END IF;\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n\n where_segments := where_segments || format(' datetime <= %L::timestamptz AND end_datetime >= %L::timestamptz ',\n edate,\n sdate\n );\n END IF;\n\n geom := stac_geom(j);\n IF geom IS NOT NULL THEN\n where_segments := where_segments || format('st_intersects(geometry, %L)',geom);\n END IF;\n\n filterlang := COALESCE(\n j->>'filter-lang',\n get_setting('default-filter-lang', j->'conf')\n );\n IF NOT filter @? '$.**.op' THEN\n filterlang := 'cql-json';\n END IF;\n\n IF filterlang NOT IN ('cql-json','cql2-json') AND j ? 'filter' THEN\n RAISE EXCEPTION '% is not a supported filter-lang. Please use cql-json or cql2-json.', filterlang;\n END IF;\n\n IF j ? 'query' AND j ? 'filter' THEN\n RAISE EXCEPTION 'Can only use either query or filter at one time.';\n END IF;\n\n IF j ? 'query' THEN\n filter := query_to_cql2(j->'query');\n ELSIF filterlang = 'cql-json' THEN\n filter := cql1_to_cql2(filter);\n END IF;\n RAISE NOTICE 'FILTER: %', filter;\n where_segments := where_segments || cql2_query(filter);\n IF cardinality(where_segments) < 1 THEN\n RETURN ' TRUE ';\n END IF;\n\n _where := array_to_string(array_remove(where_segments, NULL), ' AND ');\n\n IF _where IS NULL OR BTRIM(_where) = '' THEN\n RETURN ' TRUE ';\n END IF;\n RETURN _where;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION parse_sort_dir(_dir text, reverse boolean default false) RETURNS text AS $$\n WITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n ) SELECT\n CASE\n WHEN NOT reverse THEN d\n WHEN d = 'ASC' THEN 'DESC'\n WHEN d = 'DESC' THEN 'ASC'\n END\n FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION sort_dir_to_op(_dir text, prev boolean default false) RETURNS text AS $$\n WITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n ) SELECT\n CASE\n WHEN d = 'ASC' AND prev THEN '<='\n WHEN d = 'DESC' AND prev THEN '>='\n WHEN d = 'ASC' THEN '>='\n WHEN d = 'DESC' THEN '<='\n END\n FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION sort_sqlorderby(\n _search jsonb DEFAULT NULL,\n reverse boolean DEFAULT FALSE\n) RETURNS text AS $$\n WITH sortby AS (\n SELECT coalesce(_search->'sortby','[{\"field\":\"datetime\", \"direction\":\"desc\"}]') as sort\n ), withid AS (\n SELECT CASE\n WHEN sort @? '$[*] ? (@.field == \"id\")' THEN sort\n ELSE sort || '[{\"field\":\"id\", \"direction\":\"desc\"}]'::jsonb\n END as sort\n FROM sortby\n ), withid_rows AS (\n SELECT jsonb_array_elements(sort) as value FROM withid\n ),sorts AS (\n SELECT\n coalesce(\n -- field_orderby((items_path(value->>'field')).path_txt),\n (queryable(value->>'field')).expression\n ) as key,\n parse_sort_dir(value->>'direction', reverse) as dir\n FROM withid_rows\n )\n SELECT array_to_string(\n array_agg(concat(key, ' ', dir)),\n ', '\n ) FROM sorts;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_sort_dir(sort_item jsonb) RETURNS text AS $$\n SELECT CASE WHEN sort_item->>'direction' ILIKE 'desc%' THEN 'DESC' ELSE 'ASC' END;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION get_token_filter(_search jsonb = '{}'::jsonb, token_rec jsonb DEFAULT NULL) RETURNS text AS $$\nDECLARE\n token_id text;\n filters text[] := '{}'::text[];\n prev boolean := TRUE;\n field text;\n dir text;\n sort record;\n orfilters text[] := '{}'::text[];\n andfilters text[] := '{}'::text[];\n output text;\n token_where text;\nBEGIN\n RAISE NOTICE 'Getting Token Filter. % %', _search, token_rec;\n -- If no token provided return NULL\n IF token_rec IS NULL THEN\n IF NOT (_search ? 'token' AND\n (\n (_search->>'token' ILIKE 'prev:%')\n OR\n (_search->>'token' ILIKE 'next:%')\n )\n ) THEN\n RETURN NULL;\n END IF;\n prev := (_search->>'token' ILIKE 'prev:%');\n token_id := substr(_search->>'token', 6);\n SELECT to_jsonb(items) INTO token_rec\n FROM items WHERE id=token_id;\n END IF;\n RAISE NOTICE 'TOKEN ID: % %', token_rec, token_rec->'id';\n\n CREATE TEMP TABLE sorts (\n _row int GENERATED ALWAYS AS IDENTITY NOT NULL,\n _field text PRIMARY KEY,\n _dir text NOT NULL,\n _val text\n ) ON COMMIT DROP;\n\n -- Make sure we only have distinct columns to sort with taking the first one we get\n INSERT INTO sorts (_field, _dir)\n SELECT\n (queryable(value->>'field')).expression,\n get_sort_dir(value)\n FROM\n jsonb_array_elements(coalesce(_search->'sortby','[{\"field\":\"datetime\",\"direction\":\"desc\"}]'))\n ON CONFLICT DO NOTHING\n ;\n RAISE NOTICE 'sorts 1: %', (SELECT jsonb_agg(to_json(sorts)) FROM sorts);\n -- Get the first sort direction provided. As the id is a primary key, if there are any\n -- sorts after id they won't do anything, so make sure that id is the last sort item.\n SELECT _dir INTO dir FROM sorts ORDER BY _row ASC LIMIT 1;\n IF EXISTS (SELECT 1 FROM sorts WHERE _field = 'id') THEN\n DELETE FROM sorts WHERE _row > (SELECT _row FROM sorts WHERE _field = 'id' ORDER BY _row ASC);\n ELSE\n INSERT INTO sorts (_field, _dir) VALUES ('id', dir);\n END IF;\n\n -- Add value from looked up item to the sorts table\n UPDATE sorts SET _val=quote_literal(token_rec->>_field);\n\n -- Check if all sorts are the same direction and use row comparison\n -- to filter\n RAISE NOTICE 'sorts 2: %', (SELECT jsonb_agg(to_json(sorts)) FROM sorts);\n\n IF (SELECT count(DISTINCT _dir) FROM sorts) = 1 THEN\n SELECT format(\n '(%s) %s (%s)',\n concat_ws(', ', VARIADIC array_agg(quote_ident(_field))),\n CASE WHEN (prev AND dir = 'ASC') OR (NOT prev AND dir = 'DESC') THEN '<' ELSE '>' END,\n concat_ws(', ', VARIADIC array_agg(_val))\n ) INTO output FROM sorts\n WHERE token_rec ? _field\n ;\n ELSE\n FOR sort IN SELECT * FROM sorts ORDER BY _row asc LOOP\n RAISE NOTICE 'SORT: %', sort;\n IF sort._row = 1 THEN\n orfilters := orfilters || format('(%s %s %s)',\n quote_ident(sort._field),\n CASE WHEN (prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC') THEN '<' ELSE '>' END,\n sort._val\n );\n ELSE\n orfilters := orfilters || format('(%s AND %s %s %s)',\n array_to_string(andfilters, ' AND '),\n quote_ident(sort._field),\n CASE WHEN (prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC') THEN '<' ELSE '>' END,\n sort._val\n );\n\n END IF;\n andfilters := andfilters || format('%s = %s',\n quote_ident(sort._field),\n sort._val\n );\n END LOOP;\n output := array_to_string(orfilters, ' OR ');\n END IF;\n DROP TABLE IF EXISTS sorts;\n token_where := concat('(',coalesce(output,'true'),')');\n IF trim(token_where) = '' THEN\n token_where := NULL;\n END IF;\n RAISE NOTICE 'TOKEN_WHERE: |%|',token_where;\n RETURN token_where;\n END;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION search_tohash(jsonb) RETURNS jsonb AS $$\n SELECT $1 - '{token,limit,context,includes,excludes}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION search_hash(jsonb, jsonb) RETURNS text AS $$\n SELECT md5(concat(search_tohash($1)::text,$2::text));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TABLE IF NOT EXISTS searches(\n hash text GENERATED ALWAYS AS (search_hash(search, metadata)) STORED PRIMARY KEY,\n search jsonb NOT NULL,\n _where text,\n orderby text,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n metadata jsonb DEFAULT '{}'::jsonb NOT NULL\n);\nCREATE TABLE IF NOT EXISTS search_wheres(\n id bigint generated always as identity primary key,\n _where text NOT NULL,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n statslastupdated timestamptz,\n estimated_count bigint,\n estimated_cost float,\n time_to_estimate float,\n total_count bigint,\n time_to_count float,\n partitions text[]\n);\n\nCREATE INDEX IF NOT EXISTS search_wheres_partitions ON search_wheres USING GIN (partitions);\nCREATE UNIQUE INDEX IF NOT EXISTS search_wheres_where ON search_wheres ((md5(_where)));\n\nCREATE OR REPLACE FUNCTION where_stats(inwhere text, updatestats boolean default false, conf jsonb default null) RETURNS search_wheres AS $$\nDECLARE\n t timestamptz;\n i interval;\n explain_json jsonb;\n partitions text[];\n sw search_wheres%ROWTYPE;\n inwhere_hash text := md5(inwhere);\n _context text := lower(context(conf));\n _stats_ttl interval := context_stats_ttl(conf);\n _estimated_cost float := context_estimated_cost(conf);\n _estimated_count int := context_estimated_count(conf);\nBEGIN\n IF _context = 'off' THEN\n sw._where := inwhere;\n return sw;\n END IF;\n\n SELECT * INTO sw FROM search_wheres WHERE md5(_where)=inwhere_hash FOR UPDATE;\n\n -- Update statistics if explicitly set, if statistics do not exist, or statistics ttl has expired\n IF NOT updatestats THEN\n RAISE NOTICE 'Checking if update is needed for: % .', inwhere;\n RAISE NOTICE 'Stats Last Updated: %', sw.statslastupdated;\n RAISE NOTICE 'TTL: %, Age: %', _stats_ttl, now() - sw.statslastupdated;\n RAISE NOTICE 'Context: %, Existing Total: %', _context, sw.total_count;\n IF\n sw.statslastupdated IS NULL\n OR (now() - sw.statslastupdated) > _stats_ttl\n OR (context(conf) != 'off' AND sw.total_count IS NULL)\n THEN\n updatestats := TRUE;\n END IF;\n END IF;\n\n sw._where := inwhere;\n sw.lastused := now();\n sw.usecount := coalesce(sw.usecount,0) + 1;\n\n IF NOT updatestats THEN\n UPDATE search_wheres SET\n lastused = sw.lastused,\n usecount = sw.usecount\n WHERE md5(_where) = inwhere_hash\n RETURNING * INTO sw\n ;\n RETURN sw;\n END IF;\n\n -- Use explain to get estimated count/cost and a list of the partitions that would be hit by the query\n t := clock_timestamp();\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s', inwhere)\n INTO explain_json;\n RAISE NOTICE 'Time for just the explain: %', clock_timestamp() - t;\n i := clock_timestamp() - t;\n\n sw.statslastupdated := now();\n sw.estimated_count := explain_json->0->'Plan'->'Plan Rows';\n sw.estimated_cost := explain_json->0->'Plan'->'Total Cost';\n sw.time_to_estimate := extract(epoch from i);\n\n RAISE NOTICE 'ESTIMATED_COUNT: % < %', sw.estimated_count, _estimated_count;\n RAISE NOTICE 'ESTIMATED_COST: % < %', sw.estimated_cost, _estimated_cost;\n\n -- Do a full count of rows if context is set to on or if auto is set and estimates are low enough\n IF\n _context = 'on'\n OR\n ( _context = 'auto' AND\n (\n sw.estimated_count < _estimated_count\n AND\n sw.estimated_cost < _estimated_cost\n )\n )\n THEN\n t := clock_timestamp();\n RAISE NOTICE 'Calculating actual count...';\n EXECUTE format(\n 'SELECT count(*) FROM items WHERE %s',\n inwhere\n ) INTO sw.total_count;\n i := clock_timestamp() - t;\n RAISE NOTICE 'Actual Count: % -- %', sw.total_count, i;\n sw.time_to_count := extract(epoch FROM i);\n ELSE\n sw.total_count := NULL;\n sw.time_to_count := NULL;\n END IF;\n\n\n INSERT INTO search_wheres\n (_where, lastused, usecount, statslastupdated, estimated_count, estimated_cost, time_to_estimate, partitions, total_count, time_to_count)\n SELECT sw._where, sw.lastused, sw.usecount, sw.statslastupdated, sw.estimated_count, sw.estimated_cost, sw.time_to_estimate, sw.partitions, sw.total_count, sw.time_to_count\n ON CONFLICT ((md5(_where)))\n DO UPDATE\n SET\n lastused = sw.lastused,\n usecount = sw.usecount,\n statslastupdated = sw.statslastupdated,\n estimated_count = sw.estimated_count,\n estimated_cost = sw.estimated_cost,\n time_to_estimate = sw.time_to_estimate,\n total_count = sw.total_count,\n time_to_count = sw.time_to_count\n ;\n RETURN sw;\nEND;\n$$ LANGUAGE PLPGSQL ;\n\n\n\nDROP FUNCTION IF EXISTS search_query;\nCREATE OR REPLACE FUNCTION search_query(\n _search jsonb = '{}'::jsonb,\n updatestats boolean = false,\n _metadata jsonb = '{}'::jsonb\n) RETURNS searches AS $$\nDECLARE\n search searches%ROWTYPE;\n pexplain jsonb;\n t timestamptz;\n i interval;\nBEGIN\n SELECT * INTO search FROM searches\n WHERE hash=search_hash(_search, _metadata) FOR UPDATE;\n\n -- Calculate the where clause if not already calculated\n IF search._where IS NULL THEN\n search._where := stac_search_to_where(_search);\n END IF;\n\n -- Calculate the order by clause if not already calculated\n IF search.orderby IS NULL THEN\n search.orderby := sort_sqlorderby(_search);\n END IF;\n\n PERFORM where_stats(search._where, updatestats, _search->'conf');\n\n search.lastused := now();\n search.usecount := coalesce(search.usecount, 0) + 1;\n INSERT INTO searches (search, _where, orderby, lastused, usecount, metadata)\n VALUES (_search, search._where, search.orderby, search.lastused, search.usecount, _metadata)\n ON CONFLICT (hash) DO\n UPDATE SET\n _where = EXCLUDED._where,\n orderby = EXCLUDED.orderby,\n lastused = EXCLUDED.lastused,\n usecount = EXCLUDED.usecount,\n metadata = EXCLUDED.metadata\n RETURNING * INTO search\n ;\n RETURN search;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION search(_search jsonb = '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n searches searches%ROWTYPE;\n _where text;\n token_where text;\n full_where text;\n orderby text;\n query text;\n token_type text := substr(_search->>'token',1,4);\n _limit int := coalesce((_search->>'limit')::int, 10);\n curs refcursor;\n cntr int := 0;\n iter_record items%ROWTYPE;\n first_record jsonb;\n first_item items%ROWTYPE;\n last_item items%ROWTYPE;\n last_record jsonb;\n out_records jsonb := '[]'::jsonb;\n prev_query text;\n next text;\n prev_id text;\n has_next boolean := false;\n has_prev boolean := false;\n prev text;\n total_count bigint;\n context jsonb;\n collection jsonb;\n includes text[];\n excludes text[];\n exit_flag boolean := FALSE;\n batches int := 0;\n timer timestamptz := clock_timestamp();\n pstart timestamptz;\n pend timestamptz;\n pcurs refcursor;\n search_where search_wheres%ROWTYPE;\n id text;\nBEGIN\nCREATE TEMP TABLE results (content jsonb) ON COMMIT DROP;\n-- if ids is set, short circuit and just use direct ids query for each id\n-- skip any paging or caching\n-- hard codes ordering in the same order as the array of ids\nIF _search ? 'ids' THEN\n INSERT INTO results\n SELECT\n CASE WHEN _search->'conf'->>'nohydrate' IS NOT NULL AND (_search->'conf'->>'nohydrate')::boolean = true THEN\n content_nonhydrated(items, _search->'fields')\n ELSE\n content_hydrate(items, _search->'fields')\n END\n FROM items WHERE\n items.id = ANY(to_text_array(_search->'ids'))\n AND\n CASE WHEN _search ? 'collections' THEN\n items.collection = ANY(to_text_array(_search->'collections'))\n ELSE TRUE\n END\n ORDER BY items.datetime desc, items.id desc\n ;\n SELECT INTO total_count count(*) FROM results;\nELSE\n searches := search_query(_search);\n _where := searches._where;\n orderby := searches.orderby;\n search_where := where_stats(_where);\n total_count := coalesce(search_where.total_count, search_where.estimated_count);\n\n IF token_type='prev' THEN\n token_where := get_token_filter(_search, null::jsonb);\n orderby := sort_sqlorderby(_search, TRUE);\n END IF;\n IF token_type='next' THEN\n token_where := get_token_filter(_search, null::jsonb);\n END IF;\n\n full_where := concat_ws(' AND ', _where, token_where);\n RAISE NOTICE 'FULL QUERY % %', full_where, clock_timestamp()-timer;\n timer := clock_timestamp();\n\n FOR query IN SELECT partition_queries(full_where, orderby, search_where.partitions) LOOP\n timer := clock_timestamp();\n query := format('%s LIMIT %s', query, _limit + 1);\n RAISE NOTICE 'Partition Query: %', query;\n batches := batches + 1;\n -- curs = create_cursor(query);\n RAISE NOTICE 'cursor_tuple_fraction: %', current_setting('cursor_tuple_fraction');\n OPEN curs FOR EXECUTE query;\n LOOP\n FETCH curs into iter_record;\n EXIT WHEN NOT FOUND;\n cntr := cntr + 1;\n\n IF _search->'conf'->>'nohydrate' IS NOT NULL AND (_search->'conf'->>'nohydrate')::boolean = true THEN\n last_record := content_nonhydrated(iter_record, _search->'fields');\n ELSE\n last_record := content_hydrate(iter_record, _search->'fields');\n END IF;\n last_item := iter_record;\n IF cntr = 1 THEN\n first_item := last_item;\n first_record := last_record;\n END IF;\n IF cntr <= _limit THEN\n INSERT INTO results (content) VALUES (last_record);\n ELSIF cntr > _limit THEN\n has_next := true;\n exit_flag := true;\n EXIT;\n END IF;\n END LOOP;\n CLOSE curs;\n RAISE NOTICE 'Query took %.', clock_timestamp()-timer;\n timer := clock_timestamp();\n EXIT WHEN exit_flag;\n END LOOP;\n RAISE NOTICE 'Scanned through % partitions.', batches;\nEND IF;\n\nSELECT jsonb_agg(content) INTO out_records FROM results WHERE content is not NULL;\n\nDROP TABLE results;\n\n\n-- Flip things around if this was the result of a prev token query\nIF token_type='prev' THEN\n out_records := flip_jsonb_array(out_records);\n first_item := last_item;\n first_record := last_record;\nEND IF;\n\n-- If this query has a token, see if there is data before the first record\nIF _search ? 'token' THEN\n prev_query := format(\n 'SELECT 1 FROM items WHERE %s LIMIT 1',\n concat_ws(\n ' AND ',\n _where,\n trim(get_token_filter(_search, to_jsonb(first_item)))\n )\n );\n RAISE NOTICE 'Query to get previous record: % --- %', prev_query, first_record;\n EXECUTE prev_query INTO has_prev;\n IF FOUND and has_prev IS NOT NULL THEN\n RAISE NOTICE 'Query results from prev query: %', has_prev;\n has_prev := TRUE;\n END IF;\nEND IF;\nhas_prev := COALESCE(has_prev, FALSE);\n\nIF has_prev THEN\n prev := out_records->0->>'id';\nEND IF;\nIF has_next OR token_type='prev' THEN\n next := out_records->-1->>'id';\nEND IF;\n\nIF context(_search->'conf') != 'off' THEN\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'matched', total_count,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\nELSE\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\nEND IF;\n\ncollection := jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb),\n 'next', next,\n 'prev', prev,\n 'context', context\n);\n\nRETURN collection;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER SET SEARCH_PATH TO pgstac, public SET cursor_tuple_fraction TO 1;\n\n\nCREATE OR REPLACE FUNCTION search_cursor(_search jsonb = '{}'::jsonb) RETURNS refcursor AS $$\nDECLARE\n curs refcursor;\n searches searches%ROWTYPE;\n _where text;\n _orderby text;\n q text;\n\nBEGIN\n searches := search_query(_search);\n _where := searches._where;\n _orderby := searches.orderby;\n\n OPEN curs FOR\n WITH p AS (\n SELECT * FROM partition_queries(_where, _orderby) p\n )\n SELECT\n CASE WHEN EXISTS (SELECT 1 FROM p) THEN\n (SELECT format($q$\n SELECT * FROM (\n %s\n ) total\n $q$,\n string_agg(\n format($q$ SELECT * FROM ( %s ) AS sub $q$, p),\n '\n UNION ALL\n '\n )\n ))\n ELSE NULL\n END FROM p;\n RETURN curs;\nEND;\n$$ LANGUAGE PLPGSQL;\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION tileenvelope(zoom int, x int, y int) RETURNS geometry AS $$\nWITH t AS (\n SELECT\n 20037508.3427892 as merc_max,\n -20037508.3427892 as merc_min,\n (2 * 20037508.3427892) / (2 ^ zoom) as tile_size\n)\nSELECT st_makeenvelope(\n merc_min + (tile_size * x),\n merc_max - (tile_size * (y + 1)),\n merc_min + (tile_size * (x + 1)),\n merc_max - (tile_size * y),\n 3857\n) FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;DROP FUNCTION IF EXISTS mercgrid;\n\n\nCREATE OR REPLACE FUNCTION ftime() RETURNS interval as $$\nSELECT age(clock_timestamp(), transaction_timestamp());\n$$ LANGUAGE SQL;\nSET SEARCH_PATH to pgstac, public;\n\nDROP FUNCTION IF EXISTS geometrysearch;\nCREATE OR REPLACE FUNCTION geometrysearch(\n IN geom geometry,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE, -- Return as soon as the passed in geometry is full covered\n IN skipcovered boolean DEFAULT TRUE -- Skip any items that would show up completely under the previous items\n) RETURNS jsonb AS $$\nDECLARE\n search searches%ROWTYPE;\n curs refcursor;\n _where text;\n query text;\n iter_record items%ROWTYPE;\n out_records jsonb := '{}'::jsonb[];\n exit_flag boolean := FALSE;\n counter int := 1;\n scancounter int := 1;\n remaining_limit int := _scanlimit;\n tilearea float;\n unionedgeom geometry;\n clippedgeom geometry;\n unionedgeom_area float := 0;\n prev_area float := 0;\n excludes text[];\n includes text[];\n\nBEGIN\n DROP TABLE IF EXISTS pgstac_results;\n CREATE TEMP TABLE pgstac_results (content jsonb) ON COMMIT DROP;\n\n -- If skipcovered is true then you will always want to exit when the passed in geometry is full\n IF skipcovered THEN\n exitwhenfull := TRUE;\n END IF;\n\n SELECT * INTO search FROM searches WHERE hash=queryhash;\n\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Search with Query Hash % Not Found', queryhash;\n END IF;\n\n tilearea := st_area(geom);\n _where := format('%s AND st_intersects(geometry, %L::geometry)', search._where, geom);\n\n\n FOR query IN SELECT * FROM partition_queries(_where, search.orderby) LOOP\n query := format('%s LIMIT %L', query, remaining_limit);\n RAISE NOTICE '%', query;\n OPEN curs FOR EXECUTE query;\n LOOP\n FETCH curs INTO iter_record;\n EXIT WHEN NOT FOUND;\n IF exitwhenfull OR skipcovered THEN -- If we are not using exitwhenfull or skipcovered, we do not need to do expensive geometry operations\n clippedgeom := st_intersection(geom, iter_record.geometry);\n\n IF unionedgeom IS NULL THEN\n unionedgeom := clippedgeom;\n ELSE\n unionedgeom := st_union(unionedgeom, clippedgeom);\n END IF;\n\n unionedgeom_area := st_area(unionedgeom);\n\n IF skipcovered AND prev_area = unionedgeom_area THEN\n scancounter := scancounter + 1;\n CONTINUE;\n END IF;\n\n prev_area := unionedgeom_area;\n\n RAISE NOTICE '% % % %', unionedgeom_area/tilearea, counter, scancounter, ftime();\n END IF;\n RAISE NOTICE '% %', iter_record, content_hydrate(iter_record, fields);\n INSERT INTO pgstac_results (content) VALUES (content_hydrate(iter_record, fields));\n\n IF counter >= _limit\n OR scancounter > _scanlimit\n OR ftime() > _timelimit\n OR (exitwhenfull AND unionedgeom_area >= tilearea)\n THEN\n exit_flag := TRUE;\n EXIT;\n END IF;\n counter := counter + 1;\n scancounter := scancounter + 1;\n\n END LOOP;\n CLOSE curs;\n EXIT WHEN exit_flag;\n remaining_limit := _scanlimit - scancounter;\n END LOOP;\n\n SELECT jsonb_agg(content) INTO out_records FROM pgstac_results WHERE content IS NOT NULL;\n\n RETURN jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb)\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDROP FUNCTION IF EXISTS geojsonsearch;\nCREATE OR REPLACE FUNCTION geojsonsearch(\n IN geojson jsonb,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_geomfromgeojson(geojson),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS xyzsearch;\nCREATE OR REPLACE FUNCTION xyzsearch(\n IN _x int,\n IN _y int,\n IN _z int,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_transform(tileenvelope(_z, _x, _y), 4326),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nGRANT ALL ON SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac to pgstac_admin;\n\n-- pgstac_read role limited to using function apis\nGRANT EXECUTE ON FUNCTION search TO pgstac_read;\nGRANT EXECUTE ON FUNCTION search_query TO pgstac_read;\nGRANT EXECUTE ON FUNCTION item_by_id TO pgstac_read;\nGRANT EXECUTE ON FUNCTION get_item TO pgstac_read;\n\nGRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_ingest;\nGRANT USAGE ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_ingest;\nSELECT set_version('0.6.9');\n" }, { "path": "src/pgstac/migrations/pgstac.0.7.0-0.7.1.sql", "content": "RESET ROLE;\nDO $$\nDECLARE\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='postgis') THEN\n CREATE EXTENSION IF NOT EXISTS postgis;\n END IF;\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='btree_gist') THEN\n CREATE EXTENSION IF NOT EXISTS btree_gist;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_admin;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_read;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_ingest;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n\nGRANT pgstac_admin TO current_user;\n\n-- Function to make sure pgstac_admin is the owner of items\nCREATE OR REPLACE FUNCTION pgstac_admin_owns() RETURNS VOID AS $$\nDECLARE\n f RECORD;\nBEGIN\n FOR f IN (\n SELECT\n concat(\n oid::regproc::text,\n '(',\n coalesce(pg_get_function_identity_arguments(oid),''),\n ')'\n ) AS name,\n CASE prokind WHEN 'f' THEN 'FUNCTION' WHEN 'p' THEN 'PROCEDURE' WHEN 'a' THEN 'AGGREGATE' END as typ\n FROM pg_proc\n WHERE\n pronamespace=to_regnamespace('pgstac')\n AND proowner != to_regrole('pgstac_admin')\n AND proname NOT LIKE 'pg_stat%'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n FOR f IN (\n SELECT\n oid::regclass::text as name,\n CASE relkind\n WHEN 'i' THEN 'INDEX'\n WHEN 'I' THEN 'INDEX'\n WHEN 'p' THEN 'TABLE'\n WHEN 'r' THEN 'TABLE'\n WHEN 'v' THEN 'VIEW'\n WHEN 'S' THEN 'SEQUENCE'\n ELSE NULL\n END as typ\n FROM pg_class\n WHERE relnamespace=to_regnamespace('pgstac') and relowner != to_regrole('pgstac_admin') AND relkind IN ('r','p','v','S') AND relname NOT LIKE 'pg_stat'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\nSELECT pgstac_admin_owns();\n\nCREATE SCHEMA IF NOT EXISTS pgstac AUTHORIZATION pgstac_admin;\n\nGRANT ALL ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_admin;\n\nALTER ROLE pgstac_admin SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_read SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_ingest SET SEARCH_PATH TO pgstac, public;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\n\nGRANT pgstac_read TO pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\n\nSET ROLE pgstac_admin;\n\nSET SEARCH_PATH TO pgstac, public;\n\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS analyze_items;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS validate_constraints;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\nSET client_min_messages TO WARNING;\nSET SEARCH_PATH to pgstac, public;\n-- BEGIN migra calculated SQL\ndrop function if exists \"pgstac\".\"maintain_partition_queries\"(part text, dropindexes boolean, rebuildindexes boolean);\n\nCREATE INDEX queryables_collection_idx ON pgstac.queryables USING gin (collection_ids);\n\nset check_function_bodies = off;\n\nCREATE OR REPLACE FUNCTION pgstac.maintain_partition_queries(part text DEFAULT 'items'::text, dropindexes boolean DEFAULT false, rebuildindexes boolean DEFAULT false, idxconcurrently boolean DEFAULT false)\n RETURNS SETOF text\n LANGUAGE plpgsql\nAS $function$\nDECLARE\n parent text;\n level int;\n isleaf bool;\n collection collections%ROWTYPE;\n subpart text;\n baseidx text;\n queryable_name text;\n queryable_property_index_type text;\n queryable_property_wrapper text;\n queryable_parsed RECORD;\n deletedidx pg_indexes%ROWTYPE;\n q text;\n idx text;\n collection_partition bigint;\n _concurrently text := '';\nBEGIN\n RAISE NOTICE 'Maintaining partition: %', part;\n IF idxconcurrently THEN\n _concurrently='CONCURRENTLY';\n END IF;\n\n -- Get root partition\n SELECT parentrelid::text, pt.isleaf, pt.level\n INTO parent, isleaf, level\n FROM pg_partition_tree('items') pt\n WHERE relid::text = part;\n IF NOT FOUND THEN\n RAISE NOTICE 'Partition % Does Not Exist In Partition Tree', part;\n RETURN;\n END IF;\n\n -- If this is a parent partition, recurse to leaves\n IF NOT isleaf THEN\n FOR subpart IN\n SELECT relid::text\n FROM pg_partition_tree(part)\n WHERE relid::text != part\n LOOP\n RAISE NOTICE 'Recursing to %', subpart;\n RETURN QUERY SELECT * FROM maintain_partition_queries(subpart, dropindexes, rebuildindexes);\n END LOOP;\n RETURN; -- Don't continue since not an end leaf\n END IF;\n\n\n -- Get collection\n collection_partition := ((regexp_match(part, E'^_items_([0-9]+)'))[1])::bigint;\n RAISE NOTICE 'COLLECTION PARTITION: %', collection_partition;\n SELECT * INTO STRICT collection\n FROM collections\n WHERE key = collection_partition;\n RAISE NOTICE 'COLLECTION ID: %s', collection.id;\n\n\n -- Create temp table with existing indexes\n CREATE TEMP TABLE existing_indexes ON COMMIT DROP AS\n SELECT *\n FROM pg_indexes\n WHERE schemaname='pgstac' AND tablename=part;\n\n\n -- Check if index exists for each queryable.\n FOR\n queryable_name,\n queryable_property_index_type,\n queryable_property_wrapper\n IN\n SELECT\n name,\n COALESCE(property_index_type, 'BTREE'),\n COALESCE(property_wrapper, 'to_text')\n FROM queryables\n WHERE\n name NOT in ('id', 'datetime', 'geometry')\n AND (\n collection_ids IS NULL\n OR collection_ids = '{}'::text[]\n OR collection.id = ANY (collection_ids)\n )\n LOOP\n baseidx := format(\n $q$ ON %I USING %s (%s(((content -> 'properties'::text) -> %L::text)))$q$,\n part,\n queryable_property_index_type,\n queryable_property_wrapper,\n queryable_name\n );\n RAISE NOTICE 'BASEIDX: %', baseidx;\n RAISE NOTICE 'IDXSEARCH: %', format($q$[(']%s[')]$q$, queryable_name);\n -- If index already exists, delete it from existing indexes type table\n FOR deletedidx IN\n DELETE FROM existing_indexes\n WHERE indexdef ~* format($q$[(']%s[')]$q$, queryable_name)\n RETURNING *\n LOOP\n RAISE NOTICE 'EXISTING INDEX: %', deletedidx;\n IF NOT FOUND THEN -- index did not exist, create it\n RETURN NEXT format('CREATE INDEX %s %s;', _concurrently, baseidx);\n ELSIF rebuildindexes THEN\n RETURN NEXT format('REINDEX %I %s;', deletedidx.indexname, _concurrently);\n END IF;\n END LOOP;\n IF NOT FOUND THEN\n RAISE NOTICE 'CREATING INDEX for %', queryable_name;\n RETURN NEXT format('CREATE INDEX %s %s;', _concurrently, baseidx);\n END IF;\n END LOOP;\n IF NOT EXISTS (SELECT * FROM pg_indexes WHERE indexname::text = concat(part, '_datetime_end_datetime_idx')) THEN\n RETURN NEXT format(\n $f$CREATE INDEX IF NOT EXISTS %I ON %I USING BTREE (datetime DESC, end_datetime ASC);$f$,\n concat(part, '_datetime_end_datetime_idx'),\n part\n );\n END IF;\n\n IF NOT EXISTS (SELECT * FROM pg_indexes WHERE indexname::text = concat(part, '_geometry_idx')) THEN\n RETURN NEXT format(\n $f$CREATE INDEX IF NOT EXISTS %I ON %I USING GIST (geometry);$f$,\n concat(part, '_geometry_idx'),\n part\n );\n END IF;\n\n IF NOT EXISTS (SELECT * FROM pg_indexes WHERE indexname::text = concat(part, '_pk')) THEN\n RETURN NEXT format(\n $f$CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I USING BTREE(id);$f$,\n concat(part, '_pk'),\n part\n );\n END IF;\n DELETE FROM existing_indexes WHERE indexname::text IN (\n concat(part, '_datetime_end_datetime_idx'),\n concat(part, '_geometry_idx'),\n concat(part, '_pk')\n );\n -- Remove indexes that were not expected\n FOR idx IN SELECT indexname::text FROM existing_indexes\n LOOP\n RAISE WARNING 'Index: % is not defined by queryables.', idx;\n IF dropindexes THEN\n RETURN NEXT format('DROP INDEX IF EXISTS %I;', idx);\n END IF;\n END LOOP;\n\n DROP TABLE existing_indexes;\n RAISE NOTICE 'Returning from maintain_partition_queries.';\n RETURN;\n\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.queryable_signature(n text, c text[])\n RETURNS text\n LANGUAGE sql\n IMMUTABLE PARALLEL SAFE\nAS $function$\n SELECT concat(n, c);\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.queryables_constraint_triggerfunc()\n RETURNS trigger\n LANGUAGE plpgsql\nAS $function$\nDECLARE\n allcollections text[];\nBEGIN\n RAISE NOTICE 'Making sure that name/collection is unique for queryables';\n IF NEW.collection_ids IS NOT NULL THEN\n IF EXISTS (\n SELECT 1 FROM\n collections\n LEFT JOIN\n unnest(NEW.collection_ids) c\n ON (collections.id = c)\n WHERE c IS NULL\n ) THEN\n RAISE foreign_key_violation;\n RETURN NULL;\n END IF;\n END IF;\n IF TG_OP = 'INSERT' THEN\n IF EXISTS (\n SELECT 1 FROM queryables q\n WHERE\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n )\n ) THEN\n RAISE unique_violation;\n RETURN NULL;\n END IF;\n END IF;\n IF TG_OP = 'UPDATE' THEN\n IF EXISTS (\n SELECT 1 FROM queryables q\n WHERE\n q.id != NEW.id\n AND\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n )\n ) THEN\n RAISE unique_violation\n USING MESSAGE = format(\n 'There is already a queryable for %s for a collection in %s',\n NEW.name,\n NEW.collection_ids\n );\n RETURN NULL;\n END IF;\n END IF;\n\n RETURN NEW;\nEND;\n$function$\n;\n\nCREATE OR REPLACE PROCEDURE pgstac.analyze_items()\n LANGUAGE plpgsql\nAS $procedure$\nDECLARE\n q text;\n timeout_ts timestamptz;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n SELECT format('ANALYZE (VERBOSE, SKIP_LOCKED) %I;', relname) INTO q\n FROM pg_stat_user_tables\n WHERE relname like '_item%' AND (n_mod_since_analyze>0 OR last_analyze IS NULL) LIMIT 1;\n IF NOT FOUND THEN\n EXIT;\n END IF;\n RAISE NOTICE '%', q;\n EXECUTE q;\n COMMIT;\n END LOOP;\nEND;\n$procedure$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.get_token_filter(_search jsonb DEFAULT '{}'::jsonb, token_rec jsonb DEFAULT NULL::jsonb)\n RETURNS text\n LANGUAGE plpgsql\n SET transform_null_equals TO 'true'\nAS $function$\nDECLARE\n token_id text;\n filters text[] := '{}'::text[];\n prev boolean := TRUE;\n field text;\n dir text;\n sort record;\n orfilter text := '';\n orfilters text[] := '{}'::text[];\n andfilters text[] := '{}'::text[];\n output text;\n token_where text;\n token_item items%ROWTYPE;\nBEGIN\n RAISE NOTICE 'Getting Token Filter. % %', _search, token_rec;\n -- If no token provided return NULL\n IF token_rec IS NULL THEN\n IF NOT (_search ? 'token' AND\n (\n (_search->>'token' ILIKE 'prev:%')\n OR\n (_search->>'token' ILIKE 'next:%')\n )\n ) THEN\n RETURN NULL;\n END IF;\n prev := (_search->>'token' ILIKE 'prev:%');\n token_id := substr(_search->>'token', 6);\n IF token_id IS NULL OR token_id = '' THEN\n RAISE WARNING 'next or prev set, but no token id found';\n RETURN NULL;\n END IF;\n SELECT to_jsonb(items) INTO token_rec\n FROM items WHERE id=token_id;\n END IF;\n RAISE NOTICE 'TOKEN ID: % %', token_rec, token_rec->'id';\n\n\n RAISE NOTICE 'TOKEN ID: % %', token_rec, token_rec->'id';\n token_item := jsonb_populate_record(null::items, token_rec);\n RAISE NOTICE 'TOKEN ITEM ----- %', token_item;\n\n\n CREATE TEMP TABLE sorts (\n _row int GENERATED ALWAYS AS IDENTITY NOT NULL,\n _field text PRIMARY KEY,\n _dir text NOT NULL,\n _val text\n ) ON COMMIT DROP;\n\n -- Make sure we only have distinct columns to sort with taking the first one we get\n INSERT INTO sorts (_field, _dir)\n SELECT\n (queryable(value->>'field')).expression,\n get_sort_dir(value)\n FROM\n jsonb_array_elements(coalesce(_search->'sortby','[{\"field\":\"datetime\",\"direction\":\"desc\"}]'))\n ON CONFLICT DO NOTHING\n ;\n RAISE NOTICE 'sorts 1: %', (SELECT jsonb_agg(to_json(sorts)) FROM sorts);\n -- Get the first sort direction provided. As the id is a primary key, if there are any\n -- sorts after id they won't do anything, so make sure that id is the last sort item.\n SELECT _dir INTO dir FROM sorts ORDER BY _row ASC LIMIT 1;\n IF EXISTS (SELECT 1 FROM sorts WHERE _field = 'id') THEN\n DELETE FROM sorts WHERE _row > (SELECT _row FROM sorts WHERE _field = 'id' ORDER BY _row ASC);\n ELSE\n INSERT INTO sorts (_field, _dir) VALUES ('id', dir);\n END IF;\n\n -- Add value from looked up item to the sorts table\n UPDATE sorts SET _val=get_token_val_str(_field, token_item);\n\n -- Check if all sorts are the same direction and use row comparison\n -- to filter\n RAISE NOTICE 'sorts 2: %', (SELECT jsonb_agg(to_json(sorts)) FROM sorts);\n FOR sort IN SELECT * FROM sorts ORDER BY _row asc LOOP\n orfilter := NULL;\n RAISE NOTICE 'SORT: %', sort;\n IF sort._val IS NOT NULL AND ((prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC')) THEN\n orfilter := format($f$(\n (%s < %s) OR (%s IS NULL)\n )$f$,\n sort._field,\n sort._val,\n sort._val\n );\n ELSIF sort._val IS NULL AND ((prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC')) THEN\n RAISE NOTICE '< but null';\n orfilter := format('%s IS NOT NULL', sort._field);\n ELSIF sort._val IS NULL THEN\n RAISE NOTICE '> but null';\n --orfilter := format('%s IS NULL', sort._field);\n ELSE\n orfilter := format($f$(\n (%s > %s) OR (%s IS NULL)\n )$f$,\n sort._field,\n sort._val,\n sort._field\n );\n END IF;\n RAISE NOTICE 'ORFILTER: %', orfilter;\n\n IF orfilter IS NOT NULL THEN\n IF sort._row = 1 THEN\n orfilters := orfilters || orfilter;\n ELSE\n orfilters := orfilters || format('(%s AND %s)', array_to_string(andfilters, ' AND '), orfilter);\n END IF;\n END IF;\n IF sort._val IS NOT NULL THEN\n andfilters := andfilters || format('%s = %s', sort._field, sort._val);\n ELSE\n andfilters := andfilters || format('%s IS NULL', sort._field);\n END IF;\n END LOOP;\n output := array_to_string(orfilters, ' OR ');\n\n DROP TABLE IF EXISTS sorts;\n token_where := concat('(',coalesce(output,'true'),')');\n IF trim(token_where) = '' THEN\n token_where := NULL;\n END IF;\n RAISE NOTICE 'TOKEN_WHERE: %',token_where;\n RETURN token_where;\n END;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.queue_timeout()\n RETURNS interval\n LANGUAGE sql\nAS $function$\n SELECT t2s(coalesce(\n get_setting('queue_timeout'),\n '1h'\n ))::interval;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.search(_search jsonb DEFAULT '{}'::jsonb)\n RETURNS jsonb\n LANGUAGE plpgsql\n SET search_path TO 'pgstac', 'public'\n SET cursor_tuple_fraction TO '1'\nAS $function$\nDECLARE\n searches searches%ROWTYPE;\n _where text;\n token_where text;\n full_where text;\n orderby text;\n query text;\n token_type text := substr(_search->>'token',1,4);\n _limit int := coalesce((_search->>'limit')::int, 10);\n curs refcursor;\n cntr int := 0;\n iter_record items%ROWTYPE;\n first_record jsonb;\n first_item items%ROWTYPE;\n last_item items%ROWTYPE;\n last_record jsonb;\n out_records jsonb := '[]'::jsonb;\n prev_query text;\n next text;\n prev_id text;\n has_next boolean := false;\n has_prev boolean := false;\n prev text;\n total_count bigint;\n context jsonb;\n collection jsonb;\n includes text[];\n excludes text[];\n exit_flag boolean := FALSE;\n batches int := 0;\n timer timestamptz := clock_timestamp();\n pstart timestamptz;\n pend timestamptz;\n pcurs refcursor;\n search_where search_wheres%ROWTYPE;\n id text;\nBEGIN\nCREATE TEMP TABLE results (i int GENERATED ALWAYS AS IDENTITY, content jsonb) ON COMMIT DROP;\n searches := search_query(_search);\n _where := searches._where;\n orderby := searches.orderby;\n search_where := where_stats(_where);\n total_count := coalesce(search_where.total_count, search_where.estimated_count);\n\n IF token_type='prev' THEN\n token_where := get_token_filter(_search, null::jsonb);\n orderby := sort_sqlorderby(_search, TRUE);\n END IF;\n IF token_type='next' THEN\n token_where := get_token_filter(_search, null::jsonb);\n END IF;\n\n full_where := concat_ws(' AND ', _where, token_where);\n RAISE NOTICE 'FULL QUERY % %', full_where, clock_timestamp()-timer;\n timer := clock_timestamp();\n\n FOR query IN SELECT partition_queries(full_where, orderby, search_where.partitions) LOOP\n timer := clock_timestamp();\n query := format('%s LIMIT %s', query, _limit + 1);\n RAISE NOTICE 'Partition Query: %', query;\n batches := batches + 1;\n -- curs = create_cursor(query);\n RAISE NOTICE 'cursor_tuple_fraction: %', current_setting('cursor_tuple_fraction');\n OPEN curs FOR EXECUTE query;\n LOOP\n FETCH curs into iter_record;\n EXIT WHEN NOT FOUND;\n cntr := cntr + 1;\n\n IF _search->'conf'->>'nohydrate' IS NOT NULL AND (_search->'conf'->>'nohydrate')::boolean = true THEN\n last_record := content_nonhydrated(iter_record, _search->'fields');\n ELSE\n last_record := content_hydrate(iter_record, _search->'fields');\n END IF;\n last_item := iter_record;\n IF cntr = 1 THEN\n first_item := last_item;\n first_record := last_record;\n END IF;\n IF cntr <= _limit THEN\n INSERT INTO results (content) VALUES (last_record);\n ELSIF cntr > _limit THEN\n has_next := true;\n exit_flag := true;\n EXIT;\n END IF;\n END LOOP;\n CLOSE curs;\n RAISE NOTICE 'Query took %.', clock_timestamp()-timer;\n timer := clock_timestamp();\n EXIT WHEN exit_flag;\n END LOOP;\n RAISE NOTICE 'Scanned through % partitions.', batches;\n\nWITH ordered AS (SELECT * FROM results WHERE content IS NOT NULL ORDER BY i)\nSELECT jsonb_agg(content) INTO out_records FROM ordered;\n\nDROP TABLE results;\n\n\n-- Flip things around if this was the result of a prev token query\nIF token_type='prev' THEN\n out_records := flip_jsonb_array(out_records);\n first_item := last_item;\n first_record := last_record;\nEND IF;\n\n-- If this query has a token, see if there is data before the first record\nIF _search ? 'token' THEN\n prev_query := format(\n 'SELECT 1 FROM items WHERE %s LIMIT 1',\n concat_ws(\n ' AND ',\n _where,\n trim(get_token_filter(_search, to_jsonb(first_item)))\n )\n );\n RAISE NOTICE 'Query to get previous record: % --- %', prev_query, first_record;\n EXECUTE prev_query INTO has_prev;\n IF FOUND and has_prev IS NOT NULL THEN\n RAISE NOTICE 'Query results from prev query: %', has_prev;\n has_prev := TRUE;\n END IF;\nEND IF;\nhas_prev := COALESCE(has_prev, FALSE);\n\nIF has_prev THEN\n prev := out_records->0->>'id';\nEND IF;\nIF has_next OR token_type='prev' THEN\n next := out_records->-1->>'id';\nEND IF;\n\nIF context(_search->'conf') != 'off' THEN\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'matched', total_count,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\nELSE\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\nEND IF;\n\ncollection := jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb),\n 'next', next,\n 'prev', prev,\n 'context', context\n);\n\nRETURN collection;\nEND;\n$function$\n;\n\nCREATE TRIGGER queryables_constraint_insert_trigger BEFORE INSERT ON pgstac.queryables FOR EACH ROW EXECUTE FUNCTION pgstac.queryables_constraint_triggerfunc();\n\nCREATE TRIGGER queryables_constraint_update_trigger BEFORE UPDATE ON pgstac.queryables FOR EACH ROW WHEN (((new.name = old.name) AND (new.collection_ids IS DISTINCT FROM old.collection_ids))) EXECUTE FUNCTION pgstac.queryables_constraint_triggerfunc();\n\n\n\n-- END migra calculated SQL\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('id', '{\"title\": \"Item ID\",\"description\": \"Item identifier\",\"$ref\": \"https://schemas.stacspec.org/v1.0.0/item-spec/json-schema/item.json#/definitions/core/allOf/2/properties/id\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('geometry', '{\"title\": \"Item Geometry\",\"description\": \"Item Geometry\",\"$ref\": \"https://geojson.org/schema/Feature.json\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('datetime','{\"description\": \"Datetime\",\"type\": \"string\",\"title\": \"Acquired\",\"format\": \"date-time\",\"pattern\": \"(\\\\+00:00|Z)$\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('eo:cloud_cover','{\"$ref\": \"https://stac-extensions.github.io/eo/v1.0.0/schema.json#/definitions/fieldsproperties/eo:cloud_cover\"}','to_int','BTREE');\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDELETE FROM queryables a USING queryables b\n WHERE a.name = b.name AND a.collection_ids IS NOT DISTINCT FROM b.collection_ids AND a.id > b.id;\n\n\nINSERT INTO pgstac_settings (name, value) VALUES\n ('context', 'off'),\n ('context_estimated_count', '100000'),\n ('context_estimated_cost', '100000'),\n ('context_stats_ttl', '1 day'),\n ('default_filter_lang', 'cql2-json'),\n ('additional_properties', 'true'),\n ('use_queue', 'false'),\n ('queue_timeout', '10 minutes'),\n ('update_collection_extent', 'false')\nON CONFLICT DO NOTHING\n;\n\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nGRANT ALL ON SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac to pgstac_admin;\n\n-- pgstac_read role limited to using function apis\nGRANT EXECUTE ON FUNCTION search TO pgstac_read;\nGRANT EXECUTE ON FUNCTION search_query TO pgstac_read;\nGRANT EXECUTE ON FUNCTION item_by_id TO pgstac_read;\nGRANT EXECUTE ON FUNCTION get_item TO pgstac_read;\n\nGRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_ingest;\nGRANT USAGE ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_ingest;\n\nSELECT update_partition_stats_q(partition) FROM partitions;\nSELECT set_version('0.7.1');\n" }, { "path": "src/pgstac/migrations/pgstac.0.7.0.sql", "content": "DO $$\nDECLARE\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='postgis') THEN\n CREATE EXTENSION IF NOT EXISTS postgis;\n END IF;\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='btree_gist') THEN\n CREATE EXTENSION IF NOT EXISTS btree_gist;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_admin;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_read;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_ingest;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n\nGRANT pgstac_admin TO current_user;\n\n-- Function to make sure pgstac_admin is the owner of items\nCREATE OR REPLACE FUNCTION pgstac_admin_owns() RETURNS VOID AS $$\nDECLARE\n f RECORD;\nBEGIN\n FOR f IN (\n SELECT\n concat(\n oid::regproc::text,\n '(',\n coalesce(pg_get_function_identity_arguments(oid),''),\n ')'\n ) AS name,\n CASE prokind WHEN 'f' THEN 'FUNCTION' WHEN 'p' THEN 'PROCEDURE' WHEN 'a' THEN 'AGGREGATE' END as typ\n FROM pg_proc\n WHERE\n pronamespace=to_regnamespace('pgstac')\n AND proowner != to_regrole('pgstac_admin')\n AND proname NOT LIKE 'pg_stat%'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n FOR f IN (\n SELECT\n oid::regclass::text as name,\n CASE relkind\n WHEN 'i' THEN 'INDEX'\n WHEN 'I' THEN 'INDEX'\n WHEN 'p' THEN 'TABLE'\n WHEN 'r' THEN 'TABLE'\n WHEN 'v' THEN 'VIEW'\n WHEN 'S' THEN 'SEQUENCE'\n ELSE NULL\n END as typ\n FROM pg_class\n WHERE relnamespace=to_regnamespace('pgstac') and relowner != to_regrole('pgstac_admin') AND relkind IN ('r','p','v','S') AND relname NOT LIKE 'pg_stat'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\nSELECT pgstac_admin_owns();\n\nCREATE SCHEMA IF NOT EXISTS pgstac AUTHORIZATION pgstac_admin;\n\nGRANT ALL ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_admin;\n\nALTER ROLE pgstac_admin SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_read SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_ingest SET SEARCH_PATH TO pgstac, public;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\n\nGRANT pgstac_read TO pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\n\nSET ROLE pgstac_admin;\n\nSET SEARCH_PATH TO pgstac, public;\n\nDROP FUNCTION IF EXISTS analyze_items;\nDROP FUNCTION IF EXISTS validate_constraints;\nCREATE OR REPLACE FUNCTION to_int(jsonb) RETURNS int AS $$\n SELECT floor(($1->>0)::float)::int;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION to_float(jsonb) RETURNS float AS $$\n SELECT ($1->>0)::float;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION to_tstz(jsonb) RETURNS timestamptz AS $$\n SELECT ($1->>0)::timestamptz;\n$$ LANGUAGE SQL IMMUTABLE STRICT SET TIME ZONE 'UTC';\n\n\nCREATE OR REPLACE FUNCTION to_text(jsonb) RETURNS text AS $$\n SELECT CASE WHEN jsonb_typeof($1) IN ('array','object') THEN $1::text ELSE $1->>0 END;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION to_text_array(jsonb) RETURNS text[] AS $$\n SELECT\n CASE jsonb_typeof($1)\n WHEN 'array' THEN ARRAY(SELECT jsonb_array_elements_text($1))\n ELSE ARRAY[$1->>0]\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION bbox_geom(_bbox jsonb) RETURNS geometry AS $$\nSELECT CASE jsonb_array_length(_bbox)\n WHEN 4 THEN\n ST_SetSRID(ST_MakeEnvelope(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float,\n (_bbox->>3)::float\n ),4326)\n WHEN 6 THEN\n ST_SetSRID(ST_3DMakeBox(\n ST_MakePoint(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float\n ),\n ST_MakePoint(\n (_bbox->>3)::float,\n (_bbox->>4)::float,\n (_bbox->>5)::float\n )\n ),4326)\n ELSE null END;\n;\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION geom_bbox(_geom geometry) RETURNS jsonb AS $$\n SELECT jsonb_build_array(\n st_xmin(_geom),\n st_ymin(_geom),\n st_xmax(_geom),\n st_ymax(_geom)\n );\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION flip_jsonb_array(j jsonb) RETURNS jsonb AS $$\n SELECT jsonb_agg(value) FROM (SELECT value FROM jsonb_array_elements(j) WITH ORDINALITY ORDER BY ordinality DESC) as t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION explode_dotpaths(j jsonb) RETURNS SETOF text[] AS $$\n SELECT string_to_array(p, '.') as e FROM jsonb_array_elements_text(j) p;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION explode_dotpaths_recurse(IN j jsonb) RETURNS SETOF text[] AS $$\n WITH RECURSIVE t AS (\n SELECT e FROM explode_dotpaths(j) e\n UNION ALL\n SELECT e[1:cardinality(e)-1]\n FROM t\n WHERE cardinality(e)>1\n ) SELECT e FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION jsonb_set_nested(j jsonb, path text[], val jsonb) RETURNS jsonb AS $$\nDECLARE\nBEGIN\n IF cardinality(path) > 1 THEN\n FOR i IN 1..(cardinality(path)-1) LOOP\n IF j #> path[:i] IS NULL THEN\n j := jsonb_set_lax(j, path[:i], '{}', TRUE);\n END IF;\n END LOOP;\n END IF;\n RETURN jsonb_set_lax(j, path, val, true);\n\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\n\n\nCREATE OR REPLACE FUNCTION jsonb_include(j jsonb, f jsonb) RETURNS jsonb AS $$\nDECLARE\n includes jsonb := f-> 'include';\n outj jsonb := '{}'::jsonb;\n path text[];\nBEGIN\n IF\n includes IS NULL\n OR jsonb_array_length(includes) = 0\n THEN\n RETURN j;\n ELSE\n includes := includes || '[\"id\",\"collection\"]'::jsonb;\n FOR path IN SELECT explode_dotpaths(includes) LOOP\n outj := jsonb_set_nested(outj, path, j #> path);\n END LOOP;\n END IF;\n RETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_exclude(j jsonb, f jsonb) RETURNS jsonb AS $$\nDECLARE\n excludes jsonb := f-> 'exclude';\n outj jsonb := j;\n path text[];\nBEGIN\n IF\n excludes IS NULL\n OR jsonb_array_length(excludes) = 0\n THEN\n RETURN j;\n ELSE\n FOR path IN SELECT explode_dotpaths(excludes) LOOP\n outj := outj #- path;\n END LOOP;\n END IF;\n RETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_fields(j jsonb, f jsonb DEFAULT '{\"fields\":[]}') RETURNS jsonb AS $$\n SELECT jsonb_exclude(jsonb_include(j, f), f);\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION merge_jsonb(_a jsonb, _b jsonb) RETURNS jsonb AS $$\n SELECT\n CASE\n WHEN _a = '\"𒍟※\"'::jsonb THEN NULL\n WHEN _a IS NULL OR jsonb_typeof(_a) = 'null' THEN _b\n WHEN jsonb_typeof(_a) = 'object' AND jsonb_typeof(_b) = 'object' THEN\n (\n SELECT\n jsonb_strip_nulls(\n jsonb_object_agg(\n key,\n merge_jsonb(a.value, b.value)\n )\n )\n FROM\n jsonb_each(coalesce(_a,'{}'::jsonb)) as a\n FULL JOIN\n jsonb_each(coalesce(_b,'{}'::jsonb)) as b\n USING (key)\n )\n WHEN\n jsonb_typeof(_a) = 'array'\n AND jsonb_typeof(_b) = 'array'\n AND jsonb_array_length(_a) = jsonb_array_length(_b)\n THEN\n (\n SELECT jsonb_agg(m) FROM\n ( SELECT\n merge_jsonb(\n jsonb_array_elements(_a),\n jsonb_array_elements(_b)\n ) as m\n ) as l\n )\n ELSE _a\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION strip_jsonb(_a jsonb, _b jsonb) RETURNS jsonb AS $$\n SELECT\n CASE\n\n WHEN (_a IS NULL OR jsonb_typeof(_a) = 'null') AND _b IS NOT NULL AND jsonb_typeof(_b) != 'null' THEN '\"𒍟※\"'::jsonb\n WHEN _b IS NULL OR jsonb_typeof(_a) = 'null' THEN _a\n WHEN _a = _b AND jsonb_typeof(_a) = 'object' THEN '{}'::jsonb\n WHEN _a = _b THEN NULL\n WHEN jsonb_typeof(_a) = 'object' AND jsonb_typeof(_b) = 'object' THEN\n (\n SELECT\n jsonb_strip_nulls(\n jsonb_object_agg(\n key,\n strip_jsonb(a.value, b.value)\n )\n )\n FROM\n jsonb_each(_a) as a\n FULL JOIN\n jsonb_each(_b) as b\n USING (key)\n )\n WHEN\n jsonb_typeof(_a) = 'array'\n AND jsonb_typeof(_b) = 'array'\n AND jsonb_array_length(_a) = jsonb_array_length(_b)\n THEN\n (\n SELECT jsonb_agg(m) FROM\n ( SELECT\n strip_jsonb(\n jsonb_array_elements(_a),\n jsonb_array_elements(_b)\n ) as m\n ) as l\n )\n ELSE _a\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION nullif_jsonbnullempty(j jsonb) RETURNS jsonb AS $$\n SELECT nullif(nullif(nullif(j,'null'::jsonb),'{}'::jsonb),'[]'::jsonb);\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION jsonb_array_unique(j jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(jsonb_agg(DISTINCT a)) v FROM jsonb_array_elements(j) a;\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_concat_ignorenull(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT coalesce(a,'[]'::jsonb) || coalesce(b,'[]'::jsonb);\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_least(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(least(nullif_jsonbnullempty(a), nullif_jsonbnullempty(b)));\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_greatest(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(greatest(a, b));\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION first_notnull_sfunc(anyelement, anyelement) RETURNS anyelement AS $$\n SELECT COALESCE($1,$2);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE AGGREGATE first_notnull(anyelement)(\n SFUNC = first_notnull_sfunc,\n STYPE = anyelement\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_array_unique_merge(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_concat_ignorenull,\n FINALFUNC = jsonb_array_unique\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_min(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_least\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_max(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_greatest\n);\n\nCREATE TABLE IF NOT EXISTS migrations (\n version text PRIMARY KEY,\n datetime timestamptz DEFAULT clock_timestamp() NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION get_version() RETURNS text AS $$\n SELECT version FROM pgstac.migrations ORDER BY datetime DESC, version DESC LIMIT 1;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION set_version(text) RETURNS text AS $$\n INSERT INTO pgstac.migrations (version) VALUES ($1)\n ON CONFLICT DO NOTHING\n RETURNING version;\n$$ LANGUAGE SQL;\n\n\nCREATE TABLE IF NOT EXISTS pgstac_settings (\n name text PRIMARY KEY,\n value text NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION table_empty(text) RETURNS boolean AS $$\nDECLARE\n retval boolean;\nBEGIN\n EXECUTE format($q$\n SELECT NOT EXISTS (SELECT 1 FROM %I LIMIT 1)\n $q$,\n $1\n ) INTO retval;\n RETURN retval;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION get_setting(IN _setting text, IN conf jsonb DEFAULT NULL) RETURNS text AS $$\nSELECT COALESCE(\n nullif(conf->>_setting, ''),\n nullif(current_setting(concat('pgstac.',_setting), TRUE),''),\n nullif((SELECT value FROM pgstac.pgstac_settings WHERE name=_setting),'')\n);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_setting_bool(IN _setting text, IN conf jsonb DEFAULT NULL) RETURNS boolean AS $$\nSELECT COALESCE(\n conf->>_setting,\n current_setting(concat('pgstac.',_setting), TRUE),\n (SELECT value FROM pgstac.pgstac_settings WHERE name=_setting),\n 'FALSE'\n)::boolean;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context(conf jsonb DEFAULT NULL) RETURNS text AS $$\n SELECT pgstac.get_setting('context', conf);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context_estimated_count(conf jsonb DEFAULT NULL) RETURNS int AS $$\n SELECT pgstac.get_setting('context_estimated_count', conf)::int;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_estimated_cost();\nCREATE OR REPLACE FUNCTION context_estimated_cost(conf jsonb DEFAULT NULL) RETURNS float AS $$\n SELECT pgstac.get_setting('context_estimated_cost', conf)::float;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_stats_ttl();\nCREATE OR REPLACE FUNCTION context_stats_ttl(conf jsonb DEFAULT NULL) RETURNS interval AS $$\n SELECT pgstac.get_setting('context_stats_ttl', conf)::interval;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION t2s(text) RETURNS text AS $$\n SELECT extract(epoch FROM $1::interval)::text || ' s';\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE STRICT;\n\nCREATE OR REPLACE FUNCTION queue_timeout() RETURNS interval AS $$\n SELECT set_config(\n 'statement_timeout',\n t2s(coalesce(\n get_setting('queue_timeout'),\n '1h'\n )),\n false\n )::interval;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION notice(VARIADIC text[]) RETURNS boolean AS $$\nDECLARE\ndebug boolean := current_setting('pgstac.debug', true);\nBEGIN\n IF debug THEN\n RAISE NOTICE 'NOTICE FROM FUNC: % >>>>> %', concat_ws(' | ', $1), clock_timestamp();\n RETURN TRUE;\n END IF;\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION empty_arr(ANYARRAY) RETURNS BOOLEAN AS $$\nSELECT CASE\n WHEN $1 IS NULL THEN TRUE\n WHEN cardinality($1)<1 THEN TRUE\nELSE FALSE\nEND;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_intersection(_a ANYARRAY, _b ANYARRAY) RETURNS ANYARRAY AS $$\n SELECT ARRAY ( SELECT unnest(_a) INTERSECT SELECT UNNEST(_b) );\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION array_map_ident(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_ident(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_map_literal(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_literal(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION array_reverse(anyarray) RETURNS anyarray AS $$\nSELECT ARRAY(\n SELECT $1[i]\n FROM generate_subscripts($1,1) AS s(i)\n ORDER BY i DESC\n);\n$$ LANGUAGE SQL STRICT IMMUTABLE;\n\nDROP TABLE IF EXISTS query_queue;\nCREATE TABLE query_queue (\n query text PRIMARY KEY,\n added timestamptz DEFAULT now()\n);\n\nDROP TABLE IF EXISTS query_queue_history;\nCREATE TABLE query_queue_history(\n query text,\n added timestamptz NOT NULL,\n finished timestamptz NOT NULL DEFAULT now(),\n error text\n);\n\nCREATE OR REPLACE PROCEDURE run_queued_queries() AS $$\nDECLARE\n qitem query_queue%ROWTYPE;\n timeout_ts timestamptz;\n error text;\n cnt int := 0;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n DELETE FROM query_queue WHERE query = (SELECT query FROM query_queue ORDER BY added DESC LIMIT 1 FOR UPDATE SKIP LOCKED) RETURNING * INTO qitem;\n IF NOT FOUND THEN\n EXIT;\n END IF;\n cnt := cnt + 1;\n BEGIN\n RAISE NOTICE 'RUNNING QUERY: %', qitem.query;\n EXECUTE qitem.query;\n EXCEPTION WHEN others THEN\n error := format('%s | %s', SQLERRM, SQLSTATE);\n END;\n INSERT INTO query_queue_history (query, added, finished, error)\n VALUES (qitem.query, qitem.added, clock_timestamp(), error);\n COMMIT;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION run_queued_queries_intransaction() RETURNS int AS $$\nDECLARE\n qitem query_queue%ROWTYPE;\n timeout_ts timestamptz;\n error text;\n cnt int := 0;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n DELETE FROM query_queue WHERE query = (SELECT query FROM query_queue ORDER BY added DESC LIMIT 1 FOR UPDATE SKIP LOCKED) RETURNING * INTO qitem;\n IF NOT FOUND THEN\n RETURN cnt;\n END IF;\n cnt := cnt + 1;\n BEGIN\n qitem.query := regexp_replace(qitem.query, 'CONCURRENTLY', '');\n RAISE NOTICE 'RUNNING QUERY: %', qitem.query;\n\n EXECUTE qitem.query;\n EXCEPTION WHEN others THEN\n error := format('%s | %s', SQLERRM, SQLSTATE);\n RAISE WARNING '%', error;\n END;\n INSERT INTO query_queue_history (query, added, finished, error)\n VALUES (qitem.query, qitem.added, clock_timestamp(), error);\n END LOOP;\n RETURN cnt;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION run_or_queue(query text) RETURNS VOID AS $$\nDECLARE\n use_queue text := COALESCE(get_setting('use_queue'), 'FALSE')::boolean;\nBEGIN\n IF get_setting_bool('debug') THEN\n RAISE NOTICE '%', query;\n END IF;\n IF use_queue THEN\n INSERT INTO query_queue (query) VALUES (query) ON CONFLICT DO NOTHING;\n ELSE\n EXECUTE query;\n END IF;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n\nDROP FUNCTION IF EXISTS check_pgstac_settings;\nCREATE OR REPLACE FUNCTION check_pgstac_settings(_sysmem text DEFAULT NULL) RETURNS VOID AS $$\nDECLARE\n settingval text;\n sysmem bigint := pg_size_bytes(_sysmem);\n effective_cache_size bigint := pg_size_bytes(current_setting('effective_cache_size', TRUE));\n shared_buffers bigint := pg_size_bytes(current_setting('shared_buffers', TRUE));\n work_mem bigint := pg_size_bytes(current_setting('work_mem', TRUE));\n max_connections int := current_setting('max_connections', TRUE);\n maintenance_work_mem bigint := pg_size_bytes(current_setting('maintenance_work_mem', TRUE));\n seq_page_cost float := current_setting('seq_page_cost', TRUE);\n random_page_cost float := current_setting('random_page_cost', TRUE);\n temp_buffers bigint := pg_size_bytes(current_setting('temp_buffers', TRUE));\n r record;\nBEGIN\n IF _sysmem IS NULL THEN\n RAISE NOTICE 'Call function with the size of your system memory `SELECT check_pgstac_settings(''4GB'')` to get pg system setting recommendations.';\n ELSE\n IF effective_cache_size < (sysmem * 0.5) THEN\n RAISE WARNING 'effective_cache_size of % is set low for a system with %. Recomended value between % and %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.5), pg_size_pretty(sysmem * 0.75);\n ELSIF effective_cache_size > (sysmem * 0.75) THEN\n RAISE WARNING 'effective_cache_size of % is set high for a system with %. Recomended value between % and %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.5), pg_size_pretty(sysmem * 0.75);\n ELSE\n RAISE NOTICE 'effective_cache_size of % is set appropriately for a system with %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem);\n END IF;\n\n IF shared_buffers < (sysmem * 0.2) THEN\n RAISE WARNING 'shared_buffers of % is set low for a system with %. Recomended value between % and %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.2), pg_size_pretty(sysmem * 0.3);\n ELSIF shared_buffers > (sysmem * 0.3) THEN\n RAISE WARNING 'shared_buffers of % is set high for a system with %. Recomended value between % and %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.2), pg_size_pretty(sysmem * 0.3);\n ELSE\n RAISE NOTICE 'shared_buffers of % is set appropriately for a system with %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem);\n END IF;\n shared_buffers = sysmem * 0.3;\n IF maintenance_work_mem < (sysmem * 0.2) THEN\n RAISE WARNING 'maintenance_work_mem of % is set low for shared_buffers of %. Recomended value between % and %', pg_size_pretty(maintenance_work_mem), pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers * 0.2), pg_size_pretty(shared_buffers * 0.3);\n ELSIF maintenance_work_mem > (shared_buffers * 0.3) THEN\n RAISE WARNING 'maintenance_work_mem of % is set high for shared_buffers of %. Recomended value between % and %', pg_size_pretty(maintenance_work_mem), pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers * 0.2), pg_size_pretty(shared_buffers * 0.3);\n ELSE\n RAISE NOTICE 'maintenance_work_mem of % is set appropriately for shared_buffers of %', pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers);\n END IF;\n\n IF work_mem * max_connections > shared_buffers THEN\n RAISE WARNING 'work_mem setting of % is set high for % max_connections please reduce work_mem to % or decrease max_connections to %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers/max_connections), floor(shared_buffers/work_mem);\n ELSIF work_mem * max_connections < (shared_buffers * 0.75) THEN\n RAISE WARNING 'work_mem setting of % is set low for % max_connections you may consider raising work_mem to % or increasing max_connections to %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers/max_connections), floor(shared_buffers/work_mem);\n ELSE\n RAISE NOTICE 'work_mem setting of % and max_connections of % are adequate for shared_buffers of %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers);\n END IF;\n\n IF random_page_cost / seq_page_cost != 1.1 THEN\n RAISE WARNING 'random_page_cost (%) /seq_page_cost (%) should be set to 1.1 for SSD. Change random_page_cost to %', random_page_cost, seq_page_cost, 1.1 * seq_page_cost;\n ELSE\n RAISE NOTICE 'random_page_cost and seq_page_cost set appropriately for SSD';\n END IF;\n\n IF temp_buffers < greatest(pg_size_bytes('128MB'),(maintenance_work_mem / 2)) THEN\n RAISE WARNING 'pgstac makes heavy use of temp tables, consider raising temp_buffers from % to %', pg_size_pretty(temp_buffers), greatest('128MB', pg_size_pretty((shared_buffers / 16)));\n END IF;\n END IF;\n\n RAISE NOTICE 'VALUES FOR PGSTAC VARIABLES';\n RAISE NOTICE 'These can be set either as GUC system variables or by setting in the pgstac_settings table.';\n\n FOR r IN SELECT name, get_setting(name) as setting, CASE WHEN current_setting(concat('pgstac.',name), TRUE) IS NOT NULL THEN concat('pgstac.',name, ' GUC') WHEN value IS NOT NULL THEN 'pgstac_settings table' ELSE 'Not Set' END as loc FROM pgstac_settings LOOP\n RAISE NOTICE '% is set to % from the %', r.name, r.setting, r.loc;\n END LOOP;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pg_cron';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider intalling pg_cron which can be used to automate tasks';\n ELSE\n RAISE NOTICE 'pg_cron % is installed', settingval;\n END IF;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pgstattuple';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider installing the pgstattuple extension which can be used to help maintain tables and indexes.';\n ELSE\n RAISE NOTICE 'pgstattuple % is installed', settingval;\n END IF;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pg_stat_statements';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider installing the pg_stat_statements extension which is very helpful for tracking the types of queries on the system';\n ELSE\n RAISE NOTICE 'pg_stat_statements % is installed', settingval;\n IF current_setting('pg_stat_statements.track_statements', TRUE) IS DISTINCT FROM 'all' THEN\n RAISE WARNING 'SET pg_stat_statements.track_statements TO ''all''; --In order to track statements within functions.';\n END IF;\n END IF;\n\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac, public SET CLIENT_MIN_MESSAGES TO NOTICE;\n/* looks for a geometry in a stac item first from geometry and falling back to bbox */\nCREATE OR REPLACE FUNCTION stac_geom(value jsonb) RETURNS geometry AS $$\nSELECT\n CASE\n WHEN value ? 'intersects' THEN\n ST_GeomFromGeoJSON(value->>'intersects')\n WHEN value ? 'geometry' THEN\n ST_GeomFromGeoJSON(value->>'geometry')\n WHEN value ? 'bbox' THEN\n pgstac.bbox_geom(value->'bbox')\n ELSE NULL\n END as geometry\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION stac_daterange(\n value jsonb\n) RETURNS tstzrange AS $$\nDECLARE\n props jsonb := value;\n dt timestamptz;\n edt timestamptz;\nBEGIN\n IF props ? 'properties' THEN\n props := props->'properties';\n END IF;\n IF\n props ? 'start_datetime'\n AND props->>'start_datetime' IS NOT NULL\n AND props ? 'end_datetime'\n AND props->>'end_datetime' IS NOT NULL\n THEN\n dt := props->>'start_datetime';\n edt := props->>'end_datetime';\n IF dt > edt THEN\n RAISE EXCEPTION 'start_datetime must be < end_datetime';\n END IF;\n ELSE\n dt := props->>'datetime';\n edt := props->>'datetime';\n END IF;\n IF dt is NULL OR edt IS NULL THEN\n RAISE NOTICE 'DT: %, EDT: %', dt, edt;\n RAISE EXCEPTION 'Either datetime (%) or both start_datetime (%) and end_datetime (%) must be set.', props->>'datetime',props->>'start_datetime',props->>'end_datetime';\n END IF;\n RETURN tstzrange(dt, edt, '[]');\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_datetime(value jsonb) RETURNS timestamptz AS $$\n SELECT lower(stac_daterange(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_end_datetime(value jsonb) RETURNS timestamptz AS $$\n SELECT upper(stac_daterange(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE TABLE IF NOT EXISTS stac_extensions(\n url text PRIMARY KEY,\n content jsonb\n);\nCREATE TABLE queryables (\n id bigint GENERATED ALWAYS AS identity PRIMARY KEY,\n name text NOT NULL,\n collection_ids text[], -- used to determine what partitions to create indexes on\n definition jsonb,\n property_path text,\n property_wrapper text,\n property_index_type text\n);\nCREATE INDEX queryables_name_idx ON queryables (name);\nCREATE INDEX queryables_property_wrapper_idx ON queryables (property_wrapper);\n\n\n\n\nCREATE OR REPLACE FUNCTION array_to_path(arr text[]) RETURNS text AS $$\n SELECT string_agg(\n quote_literal(v),\n '->'\n ) FROM unnest(arr) v;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\n\n\nCREATE OR REPLACE FUNCTION queryable(\n IN dotpath text,\n OUT path text,\n OUT expression text,\n OUT wrapper text,\n OUT nulled_wrapper text\n) AS $$\nDECLARE\n q RECORD;\n path_elements text[];\nBEGIN\n IF dotpath IN ('id', 'geometry', 'datetime', 'end_datetime', 'collection') THEN\n path := dotpath;\n expression := dotpath;\n wrapper := NULL;\n RETURN;\n END IF;\n SELECT * INTO q FROM queryables\n WHERE\n name=dotpath\n OR name = 'properties.' || dotpath\n OR name = replace(dotpath, 'properties.', '')\n ;\n IF q.property_wrapper IS NULL THEN\n IF q.definition->>'type' = 'number' THEN\n wrapper := 'to_float';\n nulled_wrapper := wrapper;\n ELSIF q.definition->>'format' = 'date-time' THEN\n wrapper := 'to_tstz';\n nulled_wrapper := wrapper;\n ELSE\n nulled_wrapper := NULL;\n wrapper := 'to_text';\n END IF;\n ELSE\n wrapper := q.property_wrapper;\n nulled_wrapper := wrapper;\n END IF;\n IF q.property_path IS NOT NULL THEN\n path := q.property_path;\n ELSE\n path_elements := string_to_array(dotpath, '.');\n IF path_elements[1] IN ('links', 'assets', 'stac_version', 'stac_extensions') THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSIF path_elements[1] = 'properties' THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSE\n path := format($F$content->'properties'->%s$F$, array_to_path(path_elements));\n END IF;\n END IF;\n expression := format('%I(%s)', wrapper, path);\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\n\nDROP VIEW IF EXISTS pgstac_index;\nCREATE VIEW pgstac_indexes AS\nSELECT\n i.schemaname,\n i.tablename,\n i.indexname,\n indexdef,\n COALESCE(\n (regexp_match(indexdef, '\\(([a-zA-Z]+)\\)'))[1],\n (regexp_match(indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_]+)''::text'))[1],\n CASE WHEN indexdef ~* '\\(datetime desc, end_datetime\\)' THEN 'datetime_end_datetime' ELSE NULL END\n ) AS field,\n pg_table_size(i.indexname::text) as index_size,\n pg_size_pretty(pg_table_size(i.indexname::text)) as index_size_pretty\nFROM\n pg_indexes i\nWHERE i.schemaname='pgstac' and i.tablename ~ '_items_';\n\nDROP VIEW IF EXISTS pgstac_index_stats;\nCREATE VIEW pgstac_indexes_stats AS\nSELECT\n i.schemaname,\n i.tablename,\n i.indexname,\n indexdef,\n COALESCE(\n (regexp_match(indexdef, '\\(([a-zA-Z]+)\\)'))[1],\n (regexp_match(indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_]+)''::text'))[1],\n CASE WHEN indexdef ~* '\\(datetime desc, end_datetime\\)' THEN 'datetime_end_datetime' ELSE NULL END\n ) AS field,\n pg_table_size(i.indexname::text) as index_size,\n pg_size_pretty(pg_table_size(i.indexname::text)) as index_size_pretty,\n n_distinct,\n most_common_vals::text::text[],\n most_common_freqs::text::text[],\n histogram_bounds::text::text[],\n correlation\nFROM\n pg_indexes i\n LEFT JOIN pg_stats s ON (s.tablename = i.indexname)\nWHERE i.schemaname='pgstac' and i.tablename ~ '_items_';\n\nset check_function_bodies to off;\nCREATE OR REPLACE FUNCTION maintain_partition_queries(\n part text DEFAULT 'items',\n dropindexes boolean DEFAULT FALSE,\n rebuildindexes boolean DEFAULT FALSE\n) RETURNS SETOF text AS $$\nDECLARE\n parent text;\n level int;\n isleaf bool;\n collection collections%ROWTYPE;\n subpart text;\n baseidx text;\n queryable_name text;\n queryable_property_index_type text;\n queryable_property_wrapper text;\n queryable_parsed RECORD;\n deletedidx pg_indexes%ROWTYPE;\n q text;\n idx text;\n collection_partition bigint;\n _concurrently text := '';\nBEGIN\n RAISE NOTICE 'Maintaining partition: %', part;\n IF get_setting_bool('use_queue') THEN\n _concurrently='CONCURRENTLY';\n END IF;\n\n -- Get root partition\n SELECT parentrelid::text, pt.isleaf, pt.level\n INTO parent, isleaf, level\n FROM pg_partition_tree('items') pt\n WHERE relid::text = part;\n IF NOT FOUND THEN\n RAISE NOTICE 'Partition % Does Not Exist In Partition Tree', part;\n RETURN;\n END IF;\n\n -- If this is a parent partition, recurse to leaves\n IF NOT isleaf THEN\n FOR subpart IN\n SELECT relid::text\n FROM pg_partition_tree(part)\n WHERE relid::text != part\n LOOP\n RAISE NOTICE 'Recursing to %', subpart;\n RETURN QUERY SELECT * FROM maintain_partition_queries(subpart, dropindexes, rebuildindexes);\n END LOOP;\n RETURN; -- Don't continue since not an end leaf\n END IF;\n\n\n -- Get collection\n collection_partition := ((regexp_match(part, E'^_items_([0-9]+)'))[1])::bigint;\n RAISE NOTICE 'COLLECTION PARTITION: %', collection_partition;\n SELECT * INTO STRICT collection\n FROM collections\n WHERE key = collection_partition;\n RAISE NOTICE 'COLLECTION ID: %s', collection.id;\n\n\n -- Create temp table with existing indexes\n CREATE TEMP TABLE existing_indexes ON COMMIT DROP AS\n SELECT *\n FROM pg_indexes\n WHERE schemaname='pgstac' AND tablename=part;\n\n\n -- Check if index exists for each queryable.\n FOR\n queryable_name,\n queryable_property_index_type,\n queryable_property_wrapper\n IN\n SELECT\n name,\n COALESCE(property_index_type, 'BTREE'),\n COALESCE(property_wrapper, 'to_text')\n FROM queryables\n WHERE\n name NOT in ('id', 'datetime', 'geometry')\n AND (\n collection_ids IS NULL\n OR collection_ids = '{}'::text[]\n OR collection.id = ANY (collection_ids)\n )\n UNION ALL\n SELECT 'datetime desc, end_datetime', 'BTREE', ''\n UNION ALL\n SELECT 'geometry', 'GIST', ''\n UNION ALL\n SELECT 'id', 'BTREE', ''\n LOOP\n baseidx := format(\n $q$ ON %I USING %s (%s(((content -> 'properties'::text) -> %L::text)))$q$,\n part,\n queryable_property_index_type,\n queryable_property_wrapper,\n queryable_name\n );\n RAISE NOTICE 'BASEIDX: %', baseidx;\n RAISE NOTICE 'IDXSEARCH: %', format($q$[(']%s[')]$q$, queryable_name);\n -- If index already exists, delete it from existing indexes type table\n FOR deletedidx IN\n DELETE FROM existing_indexes\n WHERE indexdef ~* format($q$[(']%s[')]$q$, queryable_name)\n RETURNING *\n LOOP\n RAISE NOTICE 'EXISTING INDEX: %', deletedidx;\n IF NOT FOUND THEN -- index did not exist, create it\n RETURN NEXT format('CREATE INDEX %s %s;', _concurrently, baseidx);\n ELSIF rebuildindexes THEN\n RETURN NEXT format('REINDEX %I %s;', deletedidx.indexname, _concurrently);\n END IF;\n END LOOP;\n END LOOP;\n\n -- Remove indexes that were not expected\n FOR idx IN SELECT indexname::text FROM existing_indexes\n LOOP\n RAISE WARNING 'Index: % is not defined by queryables.', idx;\n IF dropindexes THEN\n RETURN NEXT format('DROP INDEX IF EXISTS %I;', idx);\n END IF;\n END LOOP;\n\n DROP TABLE existing_indexes;\n RAISE NOTICE 'Returning from maintain_partition_queries.';\n RETURN;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION maintain_partitions(\n part text DEFAULT 'items',\n dropindexes boolean DEFAULT FALSE,\n rebuildindexes boolean DEFAULT FALSE\n) RETURNS VOID AS $$\n WITH t AS (\n SELECT run_or_queue(q) FROM maintain_partition_queries(part, dropindexes, rebuildindexes) q\n ) SELECT count(*) FROM t;\n$$ LANGUAGE SQL;\n\n\nCREATE OR REPLACE FUNCTION queryables_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\nBEGIN\n PERFORM maintain_partitions();\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER queryables_trigger AFTER INSERT OR UPDATE ON queryables\nFOR EACH STATEMENT EXECUTE PROCEDURE queryables_trigger_func();\n\n\nCREATE OR REPLACE FUNCTION get_queryables(_collection_ids text[] DEFAULT NULL) RETURNS jsonb AS $$\nDECLARE\nBEGIN\n -- Build up queryables if the input contains valid collection ids or is empty\n IF EXISTS (\n SELECT 1 FROM collections\n WHERE\n _collection_ids IS NULL\n OR cardinality(_collection_ids) = 0\n OR id = ANY(_collection_ids)\n )\n THEN\n RETURN (\n WITH base AS (\n SELECT\n unnest(collection_ids) as collection_id,\n name,\n coalesce(definition, '{\"type\":\"string\"}'::jsonb) as definition\n FROM queryables\n WHERE\n _collection_ids IS NULL OR\n _collection_ids = '{}'::text[] OR\n _collection_ids && collection_ids\n UNION ALL\n SELECT null, name, coalesce(definition, '{\"type\":\"string\"}'::jsonb) as definition\n FROM queryables WHERE collection_ids IS NULL OR collection_ids = '{}'::text[]\n ), g AS (\n SELECT\n name,\n first_notnull(definition) as definition,\n jsonb_array_unique_merge(definition->'enum') as enum,\n jsonb_min(definition->'minimum') as minimum,\n jsonb_min(definition->'maxiumn') as maximum\n FROM base\n GROUP BY 1\n )\n SELECT\n jsonb_build_object(\n '$schema', 'http://json-schema.org/draft-07/schema#',\n '$id', '',\n 'type', 'object',\n 'title', 'STAC Queryables.',\n 'properties', jsonb_object_agg(\n name,\n definition\n ||\n jsonb_strip_nulls(jsonb_build_object(\n 'enum', enum,\n 'minimum', minimum,\n 'maximum', maximum\n ))\n )\n )\n FROM g\n );\n ELSE\n RETURN NULL;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\nCREATE OR REPLACE FUNCTION get_queryables(_collection text DEFAULT NULL) RETURNS jsonb AS $$\n SELECT\n CASE\n WHEN _collection IS NULL THEN get_queryables(NULL::text[])\n ELSE get_queryables(ARRAY[_collection])\n END\n ;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_queryables() RETURNS jsonb AS $$\n SELECT get_queryables(NULL::text[]);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION schema_qualify_refs(url text, j jsonb) returns jsonb as $$\n SELECT regexp_replace(j::text, '\"\\$ref\": \"#', concat('\"$ref\": \"', url, '#'), 'g')::jsonb;\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE VIEW stac_extension_queryables AS\nSELECT DISTINCT key as name, schema_qualify_refs(e.url, j.value) as definition FROM stac_extensions e, jsonb_each(e.content->'definitions'->'fields'->'properties') j;\n\n\nCREATE OR REPLACE FUNCTION missing_queryables(_collection text, _tablesample float DEFAULT 5, minrows float DEFAULT 10) RETURNS TABLE(collection text, name text, definition jsonb, property_wrapper text) AS $$\nDECLARE\n q text;\n _partition text;\n explain_json json;\n psize float;\n estrows float;\nBEGIN\n SELECT format('_items_%s', key) INTO _partition FROM collections WHERE id=_collection;\n\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM %I;', _partition)\n INTO explain_json;\n psize := explain_json->0->'Plan'->'Plan Rows';\n estrows := _tablesample * .01 * psize;\n IF estrows < minrows THEN\n _tablesample := least(100,greatest(_tablesample, (estrows / psize) / 100));\n RAISE NOTICE '%', (psize / estrows) / 100;\n END IF;\n RAISE NOTICE 'Using tablesample % to find missing queryables from % % that has ~% rows estrows: %', _tablesample, _collection, _partition, psize, estrows;\n\n q := format(\n $q$\n WITH q AS (\n SELECT * FROM queryables\n WHERE\n collection_ids IS NULL\n OR %L = ANY(collection_ids)\n ), t AS (\n SELECT\n content->'properties' AS properties\n FROM\n %I\n TABLESAMPLE SYSTEM(%L)\n ), p AS (\n SELECT DISTINCT ON (key)\n key,\n value,\n s.definition\n FROM t\n JOIN LATERAL jsonb_each(properties) ON TRUE\n LEFT JOIN q ON (q.name=key)\n LEFT JOIN stac_extension_queryables s ON (s.name=key)\n WHERE q.definition IS NULL\n )\n SELECT\n %L,\n key,\n COALESCE(definition, jsonb_build_object('type',jsonb_typeof(value))) as definition,\n CASE\n WHEN definition->>'type' = 'integer' THEN 'to_int'\n WHEN COALESCE(definition->>'type', jsonb_typeof(value)) = 'number' THEN 'to_float'\n WHEN COALESCE(definition->>'type', jsonb_typeof(value)) = 'array' THEN 'to_text_array'\n ELSE 'to_text'\n END\n FROM p;\n $q$,\n _collection,\n _partition,\n _tablesample,\n _collection\n );\n RETURN QUERY EXECUTE q;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION missing_queryables(_tablesample float DEFAULT 5) RETURNS TABLE(collection_ids text[], name text, definition jsonb, property_wrapper text) AS $$\n SELECT\n array_agg(collection),\n name,\n definition,\n property_wrapper\n FROM\n collections\n JOIN LATERAL\n missing_queryables(id, _tablesample) c\n ON TRUE\n GROUP BY\n 2,3,4\n ORDER BY 2,1\n ;\n$$ LANGUAGE SQL;\nCREATE OR REPLACE FUNCTION parse_dtrange(\n _indate jsonb,\n relative_base timestamptz DEFAULT date_trunc('hour', CURRENT_TIMESTAMP)\n) RETURNS tstzrange AS $$\nDECLARE\n timestrs text[];\n s timestamptz;\n e timestamptz;\nBEGIN\n timestrs :=\n CASE\n WHEN _indate ? 'timestamp' THEN\n ARRAY[_indate->>'timestamp']\n WHEN _indate ? 'interval' THEN\n to_text_array(_indate->'interval')\n WHEN jsonb_typeof(_indate) = 'array' THEN\n to_text_array(_indate)\n ELSE\n regexp_split_to_array(\n _indate->>0,\n '/'\n )\n END;\n RAISE NOTICE 'TIMESTRS %', timestrs;\n IF cardinality(timestrs) = 1 THEN\n IF timestrs[1] ILIKE 'P%' THEN\n RETURN tstzrange(relative_base - upper(timestrs[1])::interval, relative_base, '[)');\n END IF;\n s := timestrs[1]::timestamptz;\n RETURN tstzrange(s, s, '[]');\n END IF;\n\n IF cardinality(timestrs) != 2 THEN\n RAISE EXCEPTION 'Timestamp cannot have more than 2 values';\n END IF;\n\n IF timestrs[1] = '..' OR timestrs[1] = '' THEN\n s := '-infinity'::timestamptz;\n e := timestrs[2]::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] = '..' OR timestrs[2] = '' THEN\n s := timestrs[1]::timestamptz;\n e := 'infinity'::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[1] ILIKE 'P%' AND timestrs[2] NOT ILIKE 'P%' THEN\n e := timestrs[2]::timestamptz;\n s := e - upper(timestrs[1])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] ILIKE 'P%' AND timestrs[1] NOT ILIKE 'P%' THEN\n s := timestrs[1]::timestamptz;\n e := s + upper(timestrs[2])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n s := timestrs[1]::timestamptz;\n e := timestrs[2]::timestamptz;\n\n RETURN tstzrange(s,e,'[)');\n\n RETURN NULL;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT PARALLEL SAFE SET TIME ZONE 'UTC';\n\nCREATE OR REPLACE FUNCTION parse_dtrange(\n _indate text,\n relative_base timestamptz DEFAULT CURRENT_TIMESTAMP\n) RETURNS tstzrange AS $$\n SELECT parse_dtrange(to_jsonb(_indate), relative_base);\n$$ LANGUAGE SQL STABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION temporal_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\n ll text := 'datetime';\n lh text := 'end_datetime';\n rrange tstzrange;\n rl text;\n rh text;\n outq text;\nBEGIN\n rrange := parse_dtrange(args->1);\n RAISE NOTICE 'Constructing temporal query OP: %, ARGS: %, RRANGE: %', op, args, rrange;\n op := lower(op);\n rl := format('%L::timestamptz', lower(rrange));\n rh := format('%L::timestamptz', upper(rrange));\n outq := CASE op\n WHEN 't_before' THEN 'lh < rl'\n WHEN 't_after' THEN 'll > rh'\n WHEN 't_meets' THEN 'lh = rl'\n WHEN 't_metby' THEN 'll = rh'\n WHEN 't_overlaps' THEN 'll < rl AND rl < lh < rh'\n WHEN 't_overlappedby' THEN 'rl < ll < rh AND lh > rh'\n WHEN 't_starts' THEN 'll = rl AND lh < rh'\n WHEN 't_startedby' THEN 'll = rl AND lh > rh'\n WHEN 't_during' THEN 'll > rl AND lh < rh'\n WHEN 't_contains' THEN 'll < rl AND lh > rh'\n WHEN 't_finishes' THEN 'll > rl AND lh = rh'\n WHEN 't_finishedby' THEN 'll < rl AND lh = rh'\n WHEN 't_equals' THEN 'll = rl AND lh = rh'\n WHEN 't_disjoint' THEN 'NOT (ll <= rh AND lh >= rl)'\n WHEN 't_intersects' THEN 'll <= rh AND lh >= rl'\n WHEN 'anyinteracts' THEN 'll <= rh AND lh >= rl'\n END;\n outq := regexp_replace(outq, '\\mll\\M', ll);\n outq := regexp_replace(outq, '\\mlh\\M', lh);\n outq := regexp_replace(outq, '\\mrl\\M', rl);\n outq := regexp_replace(outq, '\\mrh\\M', rh);\n outq := format('(%s)', outq);\n RETURN outq;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\n\n\nCREATE OR REPLACE FUNCTION spatial_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\n geom text;\n j jsonb := args->1;\nBEGIN\n op := lower(op);\n RAISE NOTICE 'Constructing spatial query OP: %, ARGS: %', op, args;\n IF op NOT IN ('s_equals','s_disjoint','s_touches','s_within','s_overlaps','s_crosses','s_intersects','intersects','s_contains') THEN\n RAISE EXCEPTION 'Spatial Operator % Not Supported', op;\n END IF;\n op := regexp_replace(op, '^s_', 'st_');\n IF op = 'intersects' THEN\n op := 'st_intersects';\n END IF;\n -- Convert geometry to WKB string\n IF j ? 'type' AND j ? 'coordinates' THEN\n geom := st_geomfromgeojson(j)::text;\n ELSIF jsonb_typeof(j) = 'array' THEN\n geom := bbox_geom(j)::text;\n END IF;\n\n RETURN format('%s(geometry, %L::geometry)', op, geom);\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION query_to_cql2(q jsonb) RETURNS jsonb AS $$\n-- Translates anything passed in through the deprecated \"query\" into equivalent CQL2\nWITH t AS (\n SELECT key as property, value as ops\n FROM jsonb_each(q)\n), t2 AS (\n SELECT property, (jsonb_each(ops)).*\n FROM t WHERE jsonb_typeof(ops) = 'object'\n UNION ALL\n SELECT property, 'eq', ops\n FROM t WHERE jsonb_typeof(ops) != 'object'\n)\nSELECT\n jsonb_strip_nulls(jsonb_build_object(\n 'op', 'and',\n 'args', jsonb_agg(\n jsonb_build_object(\n 'op', key,\n 'args', jsonb_build_array(\n jsonb_build_object('property',property),\n value\n )\n )\n )\n )\n) as qcql FROM t2\n;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\nCREATE OR REPLACE FUNCTION cql1_to_cql2(j jsonb) RETURNS jsonb AS $$\nDECLARE\n args jsonb;\n ret jsonb;\nBEGIN\n RAISE NOTICE 'CQL1_TO_CQL2: %', j;\n IF j ? 'filter' THEN\n RETURN cql1_to_cql2(j->'filter');\n END IF;\n IF j ? 'property' THEN\n RETURN j;\n END IF;\n IF jsonb_typeof(j) = 'array' THEN\n SELECT jsonb_agg(cql1_to_cql2(el)) INTO args FROM jsonb_array_elements(j) el;\n RETURN args;\n END IF;\n IF jsonb_typeof(j) = 'number' THEN\n RETURN j;\n END IF;\n IF jsonb_typeof(j) = 'string' THEN\n RETURN j;\n END IF;\n\n IF jsonb_typeof(j) = 'object' THEN\n SELECT jsonb_build_object(\n 'op', key,\n 'args', cql1_to_cql2(value)\n ) INTO ret\n FROM jsonb_each(j)\n WHERE j IS NOT NULL;\n RETURN ret;\n END IF;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE STRICT;\n\nCREATE TABLE cql2_ops (\n op text PRIMARY KEY,\n template text,\n types text[]\n);\nINSERT INTO cql2_ops (op, template, types) VALUES\n ('eq', '%s = %s', NULL),\n ('neq', '%s != %s', NULL),\n ('ne', '%s != %s', NULL),\n ('!=', '%s != %s', NULL),\n ('<>', '%s != %s', NULL),\n ('lt', '%s < %s', NULL),\n ('lte', '%s <= %s', NULL),\n ('gt', '%s > %s', NULL),\n ('gte', '%s >= %s', NULL),\n ('le', '%s <= %s', NULL),\n ('ge', '%s >= %s', NULL),\n ('=', '%s = %s', NULL),\n ('<', '%s < %s', NULL),\n ('<=', '%s <= %s', NULL),\n ('>', '%s > %s', NULL),\n ('>=', '%s >= %s', NULL),\n ('like', '%s LIKE %s', NULL),\n ('ilike', '%s ILIKE %s', NULL),\n ('+', '%s + %s', NULL),\n ('-', '%s - %s', NULL),\n ('*', '%s * %s', NULL),\n ('/', '%s / %s', NULL),\n ('not', 'NOT (%s)', NULL),\n ('between', '%s BETWEEN %s AND %s', NULL),\n ('isnull', '%s IS NULL', NULL),\n ('upper', 'upper(%s)', NULL),\n ('lower', 'lower(%s)', NULL)\nON CONFLICT (op) DO UPDATE\n SET\n template = EXCLUDED.template\n;\n\n\nCREATE OR REPLACE FUNCTION cql2_query(j jsonb, wrapper text DEFAULT NULL) RETURNS text AS $$\n#variable_conflict use_variable\nDECLARE\n args jsonb := j->'args';\n arg jsonb;\n op text := lower(j->>'op');\n cql2op RECORD;\n literal text;\n _wrapper text;\n leftarg text;\n rightarg text;\nBEGIN\n IF j IS NULL OR (op IS NOT NULL AND args IS NULL) THEN\n RETURN NULL;\n END IF;\n RAISE NOTICE 'CQL2_QUERY: %', j;\n IF j ? 'filter' THEN\n RETURN cql2_query(j->'filter');\n END IF;\n\n IF j ? 'upper' THEN\n RETURN cql2_query(jsonb_build_object('op', 'upper', 'args', j->'upper'));\n END IF;\n\n IF j ? 'lower' THEN\n RETURN cql2_query(jsonb_build_object('op', 'lower', 'args', j->'lower'));\n END IF;\n\n -- Temporal Query\n IF op ilike 't_%' or op = 'anyinteracts' THEN\n RETURN temporal_op_query(op, args);\n END IF;\n\n -- If property is a timestamp convert it to text to use with\n -- general operators\n IF j ? 'timestamp' THEN\n RETURN format('%L::timestamptz', to_tstz(j->'timestamp'));\n END IF;\n IF j ? 'interval' THEN\n RAISE EXCEPTION 'Please use temporal operators when using intervals.';\n RETURN NONE;\n END IF;\n\n -- Spatial Query\n IF op ilike 's_%' or op = 'intersects' THEN\n RETURN spatial_op_query(op, args);\n END IF;\n\n IF op IN ('a_equals','a_contains','a_contained_by','a_overlaps') THEN\n IF args->0 ? 'property' THEN\n leftarg := format('to_text_array(%s)', (queryable(args->0->>'property')).path);\n END IF;\n IF args->1 ? 'property' THEN\n rightarg := format('to_text_array(%s)', (queryable(args->1->>'property')).path);\n END IF;\n RETURN FORMAT(\n '%s %s %s',\n COALESCE(leftarg, quote_literal(to_text_array(args->0))),\n CASE op\n WHEN 'a_equals' THEN '='\n WHEN 'a_contains' THEN '@>'\n WHEN 'a_contained_by' THEN '<@'\n WHEN 'a_overlaps' THEN '&&'\n END,\n COALESCE(rightarg, quote_literal(to_text_array(args->1)))\n );\n END IF;\n\n IF op = 'in' THEN\n RAISE NOTICE 'IN : % % %', args, jsonb_build_array(args->0), args->1;\n args := jsonb_build_array(args->0) || (args->1);\n RAISE NOTICE 'IN2 : %', args;\n END IF;\n\n\n\n IF op = 'between' THEN\n args = jsonb_build_array(\n args->0,\n args->1->0,\n args->1->1\n );\n END IF;\n\n -- Make sure that args is an array and run cql2_query on\n -- each element of the array\n RAISE NOTICE 'ARGS PRE: %', args;\n IF j ? 'args' THEN\n IF jsonb_typeof(args) != 'array' THEN\n args := jsonb_build_array(args);\n END IF;\n\n IF jsonb_path_exists(args, '$[*] ? (@.property == \"id\" || @.property == \"datetime\" || @.property == \"end_datetime\" || @.property == \"collection\")') THEN\n wrapper := NULL;\n ELSE\n -- if any of the arguments are a property, try to get the property_wrapper\n FOR arg IN SELECT jsonb_path_query(args, '$[*] ? (@.property != null)') LOOP\n RAISE NOTICE 'Arg: %', arg;\n wrapper := (queryable(arg->>'property')).nulled_wrapper;\n RAISE NOTICE 'Property: %, Wrapper: %', arg, wrapper;\n IF wrapper IS NOT NULL THEN\n EXIT;\n END IF;\n END LOOP;\n\n -- if the property was not in queryables, see if any args were numbers\n IF\n wrapper IS NULL\n AND jsonb_path_exists(args, '$[*] ? (@.type()==\"number\")')\n THEN\n wrapper := 'to_float';\n END IF;\n wrapper := coalesce(wrapper, 'to_text');\n END IF;\n\n SELECT jsonb_agg(cql2_query(a, wrapper))\n INTO args\n FROM jsonb_array_elements(args) a;\n END IF;\n RAISE NOTICE 'ARGS: %', args;\n\n IF op IN ('and', 'or') THEN\n RETURN\n format(\n '(%s)',\n array_to_string(to_text_array(args), format(' %s ', upper(op)))\n );\n END IF;\n\n IF op = 'in' THEN\n RAISE NOTICE 'IN -- % %', args->0, to_text(args->0);\n RETURN format(\n '%s IN (%s)',\n to_text(args->0),\n array_to_string((to_text_array(args))[2:], ',')\n );\n END IF;\n\n -- Look up template from cql2_ops\n IF j ? 'op' THEN\n SELECT * INTO cql2op FROM cql2_ops WHERE cql2_ops.op ilike op;\n IF FOUND THEN\n -- If specific index set in queryables for a property cast other arguments to that type\n\n RETURN format(\n cql2op.template,\n VARIADIC (to_text_array(args))\n );\n ELSE\n RAISE EXCEPTION 'Operator % Not Supported.', op;\n END IF;\n END IF;\n\n\n IF wrapper IS NOT NULL THEN\n RAISE NOTICE 'Wrapping % with %', j, wrapper;\n IF j ? 'property' THEN\n RETURN format('%I(%s)', wrapper, (queryable(j->>'property')).path);\n ELSE\n RETURN format('%I(%L)', wrapper, j);\n END IF;\n ELSIF j ? 'property' THEN\n RETURN quote_ident(j->>'property');\n END IF;\n\n RETURN quote_literal(to_text(j));\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION paging_dtrange(\n j jsonb\n) RETURNS tstzrange AS $$\nDECLARE\n op text;\n filter jsonb := j->'filter';\n dtrange tstzrange := tstzrange('-infinity'::timestamptz,'infinity'::timestamptz);\n sdate timestamptz := '-infinity'::timestamptz;\n edate timestamptz := 'infinity'::timestamptz;\n jpitem jsonb;\nBEGIN\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n END IF;\n IF NOT (filter @? '$.**.op ? (@ == \"or\" || @ == \"not\")') THEN\n FOR jpitem IN SELECT j FROM jsonb_path_query(filter,'strict $.** ? (@.args[*].property == \"datetime\")'::jsonpath) j LOOP\n op := lower(jpitem->>'op');\n dtrange := parse_dtrange(jpitem->'args'->1);\n IF op IN ('<=', 'lt', 'lte', '<', 'le', 't_before') THEN\n sdate := greatest(sdate,'-infinity');\n edate := least(edate, upper(dtrange));\n ELSIF op IN ('>=', '>', 'gt', 'gte', 'ge', 't_after') THEN\n edate := least(edate, 'infinity');\n sdate := greatest(sdate, lower(dtrange));\n ELSIF op IN ('=', 'eq') THEN\n edate := least(edate, upper(dtrange));\n sdate := greatest(sdate, lower(dtrange));\n END IF;\n RAISE NOTICE '2 OP: %, ARGS: %, DTRANGE: %, SDATE: %, EDATE: %', op, jpitem->'args'->1, dtrange, sdate, edate;\n END LOOP;\n END IF;\n IF sdate > edate THEN\n RETURN 'empty'::tstzrange;\n END IF;\n RETURN tstzrange(sdate,edate, '[]');\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT SET TIME ZONE 'UTC';\n\nCREATE OR REPLACE FUNCTION paging_collections(\n IN j jsonb\n) RETURNS text[] AS $$\nDECLARE\n filter jsonb := j->'filter';\n jpitem jsonb;\n op text;\n args jsonb;\n arg jsonb;\n collections text[];\nBEGIN\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n END IF;\n IF NOT (filter @? '$.**.op ? (@ == \"or\" || @ == \"not\")') THEN\n FOR jpitem IN SELECT j FROM jsonb_path_query(filter,'strict $.** ? (@.args[*].property == \"collection\")'::jsonpath) j LOOP\n RAISE NOTICE 'JPITEM: %', jpitem;\n op := jpitem->>'op';\n args := jpitem->'args';\n IF op IN ('=', 'eq', 'in') THEN\n FOR arg IN SELECT a FROM jsonb_array_elements(args) a LOOP\n IF jsonb_typeof(arg) IN ('string', 'array') THEN\n RAISE NOTICE 'arg: %, collections: %', arg, collections;\n IF collections IS NULL OR collections = '{}'::text[] THEN\n collections := to_text_array(arg);\n ELSE\n collections := array_intersection(collections, to_text_array(arg));\n END IF;\n END IF;\n END LOOP;\n END IF;\n END LOOP;\n END IF;\n IF collections = '{}'::text[] THEN\n RETURN NULL;\n END IF;\n RETURN collections;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\nCREATE OR REPLACE FUNCTION collection_base_item(content jsonb) RETURNS jsonb AS $$\n SELECT jsonb_build_object(\n 'type', 'Feature',\n 'stac_version', content->'stac_version',\n 'assets', content->'item_assets',\n 'collection', content->'id'\n );\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TABLE IF NOT EXISTS collections (\n key bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,\n id text GENERATED ALWAYS AS (content->>'id') STORED UNIQUE,\n content JSONB NOT NULL,\n base_item jsonb GENERATED ALWAYS AS (pgstac.collection_base_item(content)) STORED,\n partition_trunc text CHECK (partition_trunc IN ('year', 'month'))\n);\n\n\n\nCREATE OR REPLACE FUNCTION collection_base_item(cid text) RETURNS jsonb AS $$\n SELECT pgstac.collection_base_item(content) FROM pgstac.collections WHERE id = cid LIMIT 1;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION create_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION update_collection(data jsonb) RETURNS VOID AS $$\nDECLARE\n out collections%ROWTYPE;\nBEGIN\n UPDATE collections SET content=data WHERE id = data->>'id' RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ON CONFLICT (id) DO\n UPDATE\n SET content=EXCLUDED.content\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION delete_collection(_id text) RETURNS VOID AS $$\nDECLARE\n out collections%ROWTYPE;\nBEGIN\n DELETE FROM collections WHERE id = _id RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION get_collection(id text) RETURNS jsonb AS $$\n SELECT content FROM collections\n WHERE id=$1\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION all_collections() RETURNS jsonb AS $$\n SELECT jsonb_agg(content) FROM collections;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\nCREATE TABLE items (\n id text NOT NULL,\n geometry geometry NOT NULL,\n collection text NOT NULL,\n datetime timestamptz NOT NULL,\n end_datetime timestamptz NOT NULL,\n content JSONB NOT NULL\n)\nPARTITION BY LIST (collection)\n;\n\nCREATE INDEX \"datetime_idx\" ON items USING BTREE (datetime DESC, end_datetime ASC);\nCREATE INDEX \"geometry_idx\" ON items USING GIST (geometry);\n\nCREATE STATISTICS datetime_stats (dependencies) on datetime, end_datetime from items;\n\nALTER TABLE items ADD CONSTRAINT items_collections_fk FOREIGN KEY (collection) REFERENCES collections(id) ON DELETE CASCADE DEFERRABLE;\n\nCREATE OR REPLACE FUNCTION partition_after_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p text;\n t timestamptz := clock_timestamp();\nBEGIN\n RAISE NOTICE 'Updating partition stats %', t;\n FOR p IN SELECT DISTINCT partition\n FROM newdata n JOIN partition_sys_meta p\n ON (n.collection=p.collection AND n.datetime <@ p.partition_dtrange)\n LOOP\n PERFORM run_or_queue(format('SELECT update_partition_stats(%L, %L);', p, true));\n END LOOP;\n RAISE NOTICE 't: % %', t, clock_timestamp() - t;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE TRIGGER items_after_insert_trigger\nAFTER INSERT ON items\nREFERENCING NEW TABLE AS newdata\nFOR EACH STATEMENT\nEXECUTE FUNCTION partition_after_triggerfunc();\n\nCREATE TRIGGER items_after_update_trigger\nAFTER DELETE ON items\nREFERENCING OLD TABLE AS newdata\nFOR EACH STATEMENT\nEXECUTE FUNCTION partition_after_triggerfunc();\n\nCREATE TRIGGER items_after_delete_trigger\nAFTER UPDATE ON items\nREFERENCING NEW TABLE AS newdata\nFOR EACH STATEMENT\nEXECUTE FUNCTION partition_after_triggerfunc();\n\n\nCREATE OR REPLACE FUNCTION content_slim(_item jsonb) RETURNS jsonb AS $$\n SELECT strip_jsonb(_item - '{id,geometry,collection,type}'::text[], collection_base_item(_item->>'collection')) - '{id,geometry,collection,type}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_dehydrate(content jsonb) RETURNS items AS $$\n SELECT\n content->>'id' as id,\n stac_geom(content) as geometry,\n content->>'collection' as collection,\n stac_datetime(content) as datetime,\n stac_end_datetime(content) as end_datetime,\n content_slim(content) as content\n ;\n$$ LANGUAGE SQL STABLE;\n\nCREATE OR REPLACE FUNCTION include_field(f text, fields jsonb DEFAULT '{}'::jsonb) RETURNS boolean AS $$\nDECLARE\n includes jsonb := fields->'include';\n excludes jsonb := fields->'exclude';\nBEGIN\n IF f IS NULL THEN\n RETURN NULL;\n END IF;\n\n\n IF\n jsonb_typeof(excludes) = 'array'\n AND jsonb_array_length(excludes)>0\n AND excludes ? f\n THEN\n RETURN FALSE;\n END IF;\n\n IF\n (\n jsonb_typeof(includes) = 'array'\n AND jsonb_array_length(includes) > 0\n AND includes ? f\n ) OR\n (\n includes IS NULL\n OR jsonb_typeof(includes) = 'null'\n OR jsonb_array_length(includes) = 0\n )\n THEN\n RETURN TRUE;\n END IF;\n\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nDROP FUNCTION IF EXISTS content_hydrate(jsonb, jsonb, jsonb);\nCREATE OR REPLACE FUNCTION content_hydrate(\n _item jsonb,\n _base_item jsonb,\n fields jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\n SELECT merge_jsonb(\n jsonb_fields(_item, fields),\n jsonb_fields(_base_item, fields)\n );\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION content_hydrate(_item items, _collection collections, fields jsonb DEFAULT '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\n content jsonb;\n base_item jsonb := _collection.base_item;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry, 20)::jsonb;\n END IF;\n output := content_hydrate(\n jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'collection', _item.collection,\n 'type', 'Feature'\n ) || _item.content,\n _collection.base_item,\n fields\n );\n\n RETURN output;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_nonhydrated(\n _item items,\n fields jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry, 20)::jsonb;\n END IF;\n output := jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'collection', _item.collection,\n 'type', 'Feature'\n ) || _item.content;\n RETURN output;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_hydrate(_item items, fields jsonb DEFAULT '{}'::jsonb) RETURNS jsonb AS $$\n SELECT content_hydrate(\n _item,\n (SELECT c FROM collections c WHERE id=_item.collection LIMIT 1),\n fields\n );\n$$ LANGUAGE SQL STABLE;\n\n\nCREATE UNLOGGED TABLE items_staging (\n content JSONB NOT NULL\n);\nCREATE UNLOGGED TABLE items_staging_ignore (\n content JSONB NOT NULL\n);\nCREATE UNLOGGED TABLE items_staging_upsert (\n content JSONB NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION items_staging_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p record;\n _partitions text[];\n part text;\n ts timestamptz := clock_timestamp();\nBEGIN\n RAISE NOTICE 'Creating Partitions. %', clock_timestamp() - ts;\n\n FOR part IN WITH t AS (\n SELECT\n n.content->>'collection' as collection,\n stac_daterange(n.content->'properties') as dtr,\n partition_trunc\n FROM newdata n JOIN collections ON (n.content->>'collection'=collections.id)\n ), p AS (\n SELECT\n collection,\n COALESCE(date_trunc(partition_trunc::text, lower(dtr)),'-infinity') as d,\n tstzrange(min(lower(dtr)),max(lower(dtr)),'[]') as dtrange,\n tstzrange(min(upper(dtr)),max(upper(dtr)),'[]') as edtrange\n FROM t\n GROUP BY 1,2\n ) SELECT check_partition(collection, dtrange, edtrange) FROM p LOOP\n RAISE NOTICE 'Partition %', part;\n END LOOP;\n\n RAISE NOTICE 'Doing the insert. %', clock_timestamp() - ts;\n IF TG_TABLE_NAME = 'items_staging' THEN\n INSERT INTO items\n SELECT\n (content_dehydrate(content)).*\n FROM newdata;\n RAISE NOTICE 'Doing the delete. %', clock_timestamp() - ts;\n DELETE FROM items_staging;\n ELSIF TG_TABLE_NAME = 'items_staging_ignore' THEN\n INSERT INTO items\n SELECT\n (content_dehydrate(content)).*\n FROM newdata\n ON CONFLICT DO NOTHING;\n RAISE NOTICE 'Doing the delete. %', clock_timestamp() - ts;\n DELETE FROM items_staging_ignore;\n ELSIF TG_TABLE_NAME = 'items_staging_upsert' THEN\n WITH staging_formatted AS (\n SELECT (content_dehydrate(content)).* FROM newdata\n ), deletes AS (\n DELETE FROM items i USING staging_formatted s\n WHERE\n i.id = s.id\n AND i.collection = s.collection\n AND i IS DISTINCT FROM s\n RETURNING i.id, i.collection\n )\n INSERT INTO items\n SELECT s.* FROM\n staging_formatted s\n ON CONFLICT DO NOTHING;\n RAISE NOTICE 'Doing the delete. %', clock_timestamp() - ts;\n DELETE FROM items_staging_upsert;\n END IF;\n RAISE NOTICE 'Done. %', clock_timestamp() - ts;\n\n RETURN NULL;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER items_staging_insert_trigger AFTER INSERT ON items_staging REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\nCREATE TRIGGER items_staging_insert_ignore_trigger AFTER INSERT ON items_staging_ignore REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\nCREATE TRIGGER items_staging_insert_upsert_trigger AFTER INSERT ON items_staging_upsert REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\n\nCREATE OR REPLACE FUNCTION item_by_id(_id text, _collection text DEFAULT NULL) RETURNS items AS\n$$\nDECLARE\n i items%ROWTYPE;\nBEGIN\n SELECT * INTO i FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection) LIMIT 1;\n RETURN i;\nEND;\n$$ LANGUAGE PLPGSQL STABLE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION get_item(_id text, _collection text DEFAULT NULL) RETURNS jsonb AS $$\n SELECT content_hydrate(items) FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection);\n$$ LANGUAGE SQL STABLE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION delete_item(_id text, _collection text DEFAULT NULL) RETURNS VOID AS $$\nDECLARE\nout items%ROWTYPE;\nBEGIN\n DELETE FROM items WHERE id = _id AND (_collection IS NULL OR collection=_collection) RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n--/*\nCREATE OR REPLACE FUNCTION create_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION update_item(content jsonb) RETURNS VOID AS $$\nDECLARE\n old items %ROWTYPE;\n out items%ROWTYPE;\nBEGIN\n PERFORM delete_item(content->>'id', content->>'collection');\n PERFORM create_item(content);\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION create_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION collection_bbox(id text) RETURNS jsonb AS $$\n SELECT (replace(replace(replace(st_extent(geometry)::text,'BOX(','[['),')',']]'),' ',','))::jsonb\n FROM items WHERE collection=$1;\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION collection_temporal_extent(id text) RETURNS jsonb AS $$\n SELECT to_jsonb(array[array[min(datetime)::text, max(datetime)::text]])\n FROM items WHERE collection=$1;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection_extents() RETURNS VOID AS $$\nUPDATE collections SET\n content = content ||\n jsonb_build_object(\n 'extent', jsonb_build_object(\n 'spatial', jsonb_build_object(\n 'bbox', collection_bbox(collections.id)\n ),\n 'temporal', jsonb_build_object(\n 'interval', collection_temporal_extent(collections.id)\n )\n )\n )\n;\n$$ LANGUAGE SQL;\nCREATE TABLE partition_stats (\n partition text PRIMARY KEY,\n dtrange tstzrange,\n edtrange tstzrange,\n spatial geometry,\n last_updated timestamptz,\n keys text[]\n) WITH (FILLFACTOR=90);\n\nCREATE INDEX partitions_range_idx ON partition_stats USING GIST(dtrange);\n\n\nCREATE OR REPLACE FUNCTION constraint_tstzrange(expr text) RETURNS tstzrange AS $$\n WITH t AS (\n SELECT regexp_matches(\n expr,\n E'\\\\(''\\([0-9 :+-]*\\)''\\\\).*\\\\(''\\([0-9 :+-]*\\)''\\\\)'\n ) AS m\n ) SELECT tstzrange(m[1]::timestamptz, m[2]::timestamptz) FROM t\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE STRICT;\n\nCREATE OR REPLACE FUNCTION dt_constraint(coid oid, OUT dt tstzrange, OUT edt tstzrange) RETURNS RECORD AS $$\nDECLARE\n expr text := pg_get_constraintdef(coid);\n matches timestamptz[];\nBEGIN\n IF expr LIKE '%NULL%' THEN\n dt := tstzrange(null::timestamptz, null::timestamptz);\n edt := tstzrange(null::timestamptz, null::timestamptz);\n RETURN;\n END IF;\n WITH f AS (SELECT (regexp_matches(expr, E'([0-9]{4}-[0-1][0-9]-[0-3][0-9] [0-2][0-9]:[0-5][0-9]:[0-5][0-9]\\.?[0-9]*)', 'g'))[1] f)\n SELECT array_agg(f::timestamptz) INTO matches FROM f;\n IF cardinality(matches) = 4 THEN\n dt := tstzrange(matches[1], matches[2],'[]');\n edt := tstzrange(matches[3], matches[4], '[]');\n RETURN;\n ELSIF cardinality(matches) = 2 THEN\n edt := tstzrange(matches[1], matches[2],'[]');\n RETURN;\n END IF;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\nCREATE OR REPLACE VIEW partition_sys_meta AS\nSELECT\n relid::text as partition,\n replace(replace(CASE WHEN level = 1 THEN pg_get_expr(c.relpartbound, c.oid)\n ELSE pg_get_expr(parent.relpartbound, parent.oid)\n END, 'FOR VALUES IN (''',''), ''')','') AS collection,\n level,\n c.reltuples,\n c.relhastriggers,\n COALESCE(constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as partition_dtrange,\n COALESCE((dt_constraint(edt.oid)).dt, constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as constraint_dtrange,\n COALESCE((dt_constraint(edt.oid)).edt, tstzrange('-infinity', 'infinity','[]')) as constraint_edtrange\nFROM\n pg_partition_tree('items')\n JOIN pg_class c ON (relid::regclass = c.oid)\n JOIN pg_class parent ON (parentrelid::regclass = parent.oid AND isleaf)\n LEFT JOIN pg_constraint edt ON (conrelid=c.oid AND contype='c')\nWHERE isleaf\n;\n\nCREATE VIEW partitions AS\nSELECT * FROM partition_sys_meta LEFT JOIN partition_stats USING (partition);\n\nCREATE OR REPLACE FUNCTION update_partition_stats_q(_partition text, istrigger boolean default false) RETURNS VOID AS $$\nDECLARE\nBEGIN\n PERFORM run_or_queue(\n format('SELECT update_partition_stats(%L, %L);', _partition, istrigger)\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION update_partition_stats(_partition text, istrigger boolean default false) RETURNS VOID AS $$\nDECLARE\n dtrange tstzrange;\n edtrange tstzrange;\n cdtrange tstzrange;\n cedtrange tstzrange;\n extent geometry;\n collection text;\nBEGIN\n RAISE NOTICE 'Updating stats for %.', _partition;\n EXECUTE format(\n $q$\n SELECT\n tstzrange(min(datetime), max(datetime),'[]'),\n tstzrange(min(end_datetime), max(end_datetime), '[]')\n FROM %I\n $q$,\n _partition\n ) INTO dtrange, edtrange;\n extent := st_estimatedextent('pgstac', _partition, 'geometry');\n INSERT INTO partition_stats (partition, dtrange, edtrange, spatial, last_updated)\n SELECT _partition, dtrange, edtrange, extent, now()\n ON CONFLICT (partition) DO\n UPDATE SET\n dtrange=EXCLUDED.dtrange,\n edtrange=EXCLUDED.edtrange,\n spatial=EXCLUDED.spatial,\n last_updated=EXCLUDED.last_updated\n ;\n SELECT\n constraint_dtrange, constraint_edtrange, partitions.collection\n INTO cdtrange, cedtrange, collection\n FROM partitions WHERE partition = _partition;\n\n RAISE NOTICE 'Checking if we need to modify constraints.';\n IF\n (cdtrange IS DISTINCT FROM dtrange OR edtrange IS DISTINCT FROM cedtrange)\n AND NOT istrigger\n THEN\n RAISE NOTICE 'Modifying Constraints';\n RAISE NOTICE 'Existing % %', cdtrange, cedtrange;\n RAISE NOTICE 'New % %', dtrange, edtrange;\n PERFORM drop_table_constraints(_partition);\n PERFORM create_table_constraints(_partition, dtrange, edtrange);\n END IF;\n RAISE NOTICE 'Checking if we need to update collection extents.';\n IF get_setting_bool('update_collection_extent') THEN\n RAISE NOTICE 'updating collection extent for %', collection;\n PERFORM run_or_queue(format($q$\n UPDATE collections\n SET content = jsonb_set_lax(\n content,\n '{extent}'::text[],\n collection_extent(%L),\n true,\n 'use_json_null'\n ) WHERE id=%L\n ;\n $q$, collection, collection));\n ELSE\n RAISE NOTICE 'Not updating collection extent for %', collection;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL STRICT;\n\n\nCREATE OR REPLACE FUNCTION partition_name( IN collection text, IN dt timestamptz, OUT partition_name text, OUT partition_range tstzrange) AS $$\nDECLARE\n c RECORD;\n parent_name text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n parent_name := format('_items_%s', c.key);\n\n\n IF c.partition_trunc = 'year' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYY'));\n ELSIF c.partition_trunc = 'month' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYYMM'));\n ELSE\n partition_name := parent_name;\n partition_range := tstzrange('-infinity'::timestamptz, 'infinity'::timestamptz, '[]');\n END IF;\n IF partition_range IS NULL THEN\n partition_range := tstzrange(\n date_trunc(c.partition_trunc::text, dt),\n date_trunc(c.partition_trunc::text, dt) + concat('1 ', c.partition_trunc)::interval\n );\n END IF;\n RETURN;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION drop_table_constraints(t text) RETURNS text AS $$\nDECLARE\n q text;\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM partitions WHERE partition=t) THEN\n RETURN NULL;\n END IF;\n FOR q IN SELECT FORMAT(\n $q$\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n $q$,\n t,\n conname\n ) FROM pg_constraint\n WHERE conrelid=t::regclass::oid AND contype='c'\n LOOP\n EXECUTE q;\n END LOOP;\n RETURN t;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE OR REPLACE FUNCTION create_table_constraints(t text, _dtrange tstzrange, _edtrange tstzrange) RETURNS text AS $$\nDECLARE\n q text;\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM partitions WHERE partition=t) THEN\n RETURN NULL;\n END IF;\n RAISE NOTICE 'Creating Table Constraints for % % %', t, _dtrange, _edtrange;\n IF _dtrange = 'empty' AND _edtrange = 'empty' THEN\n q :=format(\n $q$\n ALTER TABLE %I\n ADD CONSTRAINT %I\n CHECK (((datetime IS NULL) AND (end_datetime IS NULL))) NOT VALID\n ;\n ALTER TABLE %I\n VALIDATE CONSTRAINT %I\n ;\n $q$,\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t)\n );\n ELSE\n q :=format(\n $q$\n ALTER TABLE %I\n ADD CONSTRAINT %I\n CHECK (\n (datetime >= %L)\n AND (datetime <= %L)\n AND (end_datetime >= %L)\n AND (end_datetime <= %L)\n ) NOT VALID\n ;\n ALTER TABLE %I\n VALIDATE CONSTRAINT %I\n ;\n $q$,\n t,\n format('%s_dt', t),\n lower(_dtrange),\n upper(_dtrange),\n lower(_edtrange),\n upper(_edtrange),\n t,\n format('%s_dt', t)\n );\n END IF;\n PERFORM run_or_queue(q);\n RETURN t;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION check_partition(\n _collection text,\n _dtrange tstzrange,\n _edtrange tstzrange\n) RETURNS text AS $$\nDECLARE\n c RECORD;\n pm RECORD;\n _partition_name text;\n _partition_dtrange tstzrange;\n _constraint_dtrange tstzrange;\n _constraint_edtrange tstzrange;\n q text;\n deferrable_q text;\n err_context text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=_collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', _collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n\n IF c.partition_trunc IS NOT NULL THEN\n _partition_dtrange := tstzrange(\n date_trunc(c.partition_trunc, lower(_dtrange)),\n date_trunc(c.partition_trunc, lower(_dtrange)) + (concat('1 ', c.partition_trunc))::interval,\n '[)'\n );\n ELSE\n _partition_dtrange := '[-infinity, infinity]'::tstzrange;\n END IF;\n\n IF NOT _partition_dtrange @> _dtrange THEN\n RAISE EXCEPTION 'dtrange % is greater than the partition size % for collection %', _dtrange, c.partition_trunc, _collection;\n END IF;\n\n\n IF c.partition_trunc = 'year' THEN\n _partition_name := format('_items_%s_%s', c.key, to_char(lower(_partition_dtrange),'YYYY'));\n ELSIF c.partition_trunc = 'month' THEN\n _partition_name := format('_items_%s_%s', c.key, to_char(lower(_partition_dtrange),'YYYYMM'));\n ELSE\n _partition_name := format('_items_%s', c.key);\n END IF;\n\n SELECT * INTO pm FROM partition_sys_meta WHERE collection=_collection AND partition_dtrange @> _dtrange;\n IF FOUND THEN\n RAISE NOTICE '% % %', _edtrange, _dtrange, pm;\n _constraint_edtrange :=\n tstzrange(\n least(\n lower(_edtrange),\n nullif(lower(pm.constraint_edtrange), '-infinity')\n ),\n greatest(\n upper(_edtrange),\n nullif(upper(pm.constraint_edtrange), 'infinity')\n ),\n '[]'\n );\n _constraint_dtrange :=\n tstzrange(\n least(\n lower(_dtrange),\n nullif(lower(pm.constraint_dtrange), '-infinity')\n ),\n greatest(\n upper(_dtrange),\n nullif(upper(pm.constraint_dtrange), 'infinity')\n ),\n '[]'\n );\n\n IF pm.constraint_edtrange @> _edtrange AND pm.constraint_dtrange @> _dtrange THEN\n RETURN pm.partition;\n ELSE\n PERFORM drop_table_constraints(_partition_name);\n END IF;\n ELSE\n _constraint_edtrange := _edtrange;\n _constraint_dtrange := _dtrange;\n END IF;\n RAISE NOTICE 'Creating partition % %', _partition_name, _partition_dtrange;\n IF c.partition_trunc IS NULL THEN\n q := format(\n $q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n $q$,\n _partition_name,\n _collection,\n concat(_partition_name,'_pk'),\n _partition_name\n );\n ELSE\n q := format(\n $q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L) PARTITION BY RANGE (datetime);\n CREATE TABLE IF NOT EXISTS %I partition OF %I FOR VALUES FROM (%L) TO (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n $q$,\n format('_items_%s', c.key),\n _collection,\n _partition_name,\n format('_items_%s', c.key),\n lower(_partition_dtrange),\n upper(_partition_dtrange),\n format('%s_pk', _partition_name),\n _partition_name\n );\n END IF;\n\n BEGIN\n EXECUTE q;\n EXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', _partition_name;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\n END;\n PERFORM create_table_constraints(_partition_name, _constraint_dtrange, _constraint_edtrange);\n PERFORM maintain_partitions(_partition_name);\n PERFORM update_partition_stats_q(_partition_name, true);\n RETURN _partition_name;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION repartition(_collection text, _partition_trunc text, triggered boolean DEFAULT FALSE) RETURNS text AS $$\nDECLARE\n c RECORD;\n q text;\n from_trunc text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=_collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', _collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n IF triggered THEN\n RAISE NOTICE 'Converting % to % partitioning via Trigger', _collection, _partition_trunc;\n ELSE\n RAISE NOTICE 'Converting % from using % to % partitioning', _collection, c.partition_trunc, _partition_trunc;\n IF c.partition_trunc IS NOT DISTINCT FROM _partition_trunc THEN\n RAISE NOTICE 'Collection % already set to use partition by %', _collection, _partition_trunc;\n RETURN _collection;\n END IF;\n END IF;\n\n IF EXISTS (SELECT 1 FROM partitions WHERE collection=_collection LIMIT 1) THEN\n EXECUTE format(\n $q$\n CREATE TEMP TABLE changepartitionstaging ON COMMIT DROP AS SELECT * FROM %I;\n DROP TABLE IF EXISTS %I CASCADE;\n WITH p AS (\n SELECT\n collection,\n CASE WHEN %L IS NULL THEN '-infinity'::timestamptz\n ELSE date_trunc(%L, datetime)\n END as d,\n tstzrange(min(datetime),max(datetime),'[]') as dtrange,\n tstzrange(min(datetime),max(datetime),'[]') as edtrange\n FROM changepartitionstaging\n GROUP BY 1,2\n ) SELECT check_partition(collection, dtrange, edtrange) FROM p;\n INSERT INTO items SELECT * FROM changepartitionstaging;\n DROP TABLE changepartitionstaging;\n $q$,\n concat('_items_', c.key),\n concat('_items_', c.key),\n c.partition_trunc,\n c.partition_trunc\n );\n END IF;\n RETURN _collection;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE OR REPLACE FUNCTION collections_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n q text;\n partition_name text := format('_items_%s', NEW.key);\n partition_exists boolean := false;\n partition_empty boolean := true;\n err_context text;\n loadtemp boolean := FALSE;\nBEGIN\n RAISE NOTICE 'Collection Trigger. % %', NEW.id, NEW.key;\n IF TG_OP = 'UPDATE' AND NEW.partition_trunc IS DISTINCT FROM OLD.partition_trunc THEN\n PERFORM repartition(NEW.id, NEW.partition_trunc, TRUE);\n END IF;\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER collections_trigger AFTER\nINSERT\nOR\nUPDATE ON collections\nFOR EACH ROW EXECUTE FUNCTION collections_trigger_func();\nCREATE OR REPLACE VIEW partition_steps AS\nSELECT\n partition as name,\n date_trunc('month',lower(partition_dtrange)) as sdate,\n date_trunc('month', upper(partition_dtrange)) + '1 month'::interval as edate\n FROM partitions WHERE partition_dtrange IS NOT NULL AND partition_dtrange != 'empty'::tstzrange\n ORDER BY dtrange ASC\n;\n\nCREATE OR REPLACE FUNCTION chunker(\n IN _where text,\n OUT s timestamptz,\n OUT e timestamptz\n) RETURNS SETOF RECORD AS $$\nDECLARE\n explain jsonb;\nBEGIN\n IF _where IS NULL THEN\n _where := ' TRUE ';\n END IF;\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s;', _where)\n INTO explain;\n\n RETURN QUERY\n WITH t AS (\n SELECT j->>0 as p FROM\n jsonb_path_query(\n explain,\n 'strict $.**.\"Relation Name\" ? (@ != null)'\n ) j\n ),\n parts AS (\n SELECT sdate, edate FROM t JOIN partition_steps ON (t.p = name)\n ),\n times AS (\n SELECT sdate FROM parts\n UNION\n SELECT edate FROM parts\n ),\n uniq AS (\n SELECT DISTINCT sdate FROM times ORDER BY sdate\n ),\n last AS (\n SELECT sdate, lead(sdate, 1) over () as edate FROM uniq\n )\n SELECT sdate, edate FROM last WHERE edate IS NOT NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION partition_queries(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN partitions text[] DEFAULT NULL\n) RETURNS SETOF text AS $$\nDECLARE\n query text;\n sdate timestamptz;\n edate timestamptz;\nBEGIN\nIF _where IS NULL OR trim(_where) = '' THEN\n _where = ' TRUE ';\nEND IF;\nRAISE NOTICE 'Getting chunks for % %', _where, _orderby;\nIF _orderby ILIKE 'datetime d%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 DESC LOOP\n RETURN NEXT format($q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n $q$,\n sdate,\n edate,\n _where,\n _orderby\n );\n END LOOP;\nELSIF _orderby ILIKE 'datetime a%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 ASC LOOP\n RETURN NEXT format($q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n $q$,\n sdate,\n edate,\n _where,\n _orderby\n );\n END LOOP;\nELSE\n query := format($q$\n SELECT * FROM items\n WHERE %s\n ORDER BY %s\n $q$, _where, _orderby\n );\n\n RETURN NEXT query;\n RETURN;\nEND IF;\n\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION partition_query_view(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN _limit int DEFAULT 10\n) RETURNS text AS $$\n WITH p AS (\n SELECT * FROM partition_queries(_where, _orderby) p\n )\n SELECT\n CASE WHEN EXISTS (SELECT 1 FROM p) THEN\n (SELECT format($q$\n SELECT * FROM (\n %s\n ) total LIMIT %s\n $q$,\n string_agg(\n format($q$ SELECT * FROM ( %s ) AS sub $q$, p),\n '\n UNION ALL\n '\n ),\n _limit\n ))\n ELSE NULL\n END FROM p;\n$$ LANGUAGE SQL IMMUTABLE;\n\n\n\n\nCREATE OR REPLACE FUNCTION stac_search_to_where(j jsonb) RETURNS text AS $$\nDECLARE\n where_segments text[];\n _where text;\n dtrange tstzrange;\n collections text[];\n geom geometry;\n sdate timestamptz;\n edate timestamptz;\n filterlang text;\n filter jsonb := j->'filter';\nBEGIN\n IF j ? 'ids' THEN\n where_segments := where_segments || format('id = ANY (%L) ', to_text_array(j->'ids'));\n END IF;\n\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n where_segments := where_segments || format('collection = ANY (%L) ', collections);\n END IF;\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n\n where_segments := where_segments || format(' datetime <= %L::timestamptz AND end_datetime >= %L::timestamptz ',\n edate,\n sdate\n );\n END IF;\n\n geom := stac_geom(j);\n IF geom IS NOT NULL THEN\n where_segments := where_segments || format('st_intersects(geometry, %L)',geom);\n END IF;\n\n filterlang := COALESCE(\n j->>'filter-lang',\n get_setting('default_filter_lang', j->'conf')\n );\n IF NOT filter @? '$.**.op' THEN\n filterlang := 'cql-json';\n END IF;\n\n IF filterlang NOT IN ('cql-json','cql2-json') AND j ? 'filter' THEN\n RAISE EXCEPTION '% is not a supported filter-lang. Please use cql-json or cql2-json.', filterlang;\n END IF;\n\n IF j ? 'query' AND j ? 'filter' THEN\n RAISE EXCEPTION 'Can only use either query or filter at one time.';\n END IF;\n\n IF j ? 'query' THEN\n filter := query_to_cql2(j->'query');\n ELSIF filterlang = 'cql-json' THEN\n filter := cql1_to_cql2(filter);\n END IF;\n RAISE NOTICE 'FILTER: %', filter;\n where_segments := where_segments || cql2_query(filter);\n IF cardinality(where_segments) < 1 THEN\n RETURN ' TRUE ';\n END IF;\n\n _where := array_to_string(array_remove(where_segments, NULL), ' AND ');\n\n IF _where IS NULL OR BTRIM(_where) = '' THEN\n RETURN ' TRUE ';\n END IF;\n RETURN _where;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION parse_sort_dir(_dir text, reverse boolean default false) RETURNS text AS $$\n WITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n ) SELECT\n CASE\n WHEN NOT reverse THEN d\n WHEN d = 'ASC' THEN 'DESC'\n WHEN d = 'DESC' THEN 'ASC'\n END\n FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION sort_dir_to_op(_dir text, prev boolean default false) RETURNS text AS $$\n WITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n ) SELECT\n CASE\n WHEN d = 'ASC' AND prev THEN '<='\n WHEN d = 'DESC' AND prev THEN '>='\n WHEN d = 'ASC' THEN '>='\n WHEN d = 'DESC' THEN '<='\n END\n FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION sort_sqlorderby(\n _search jsonb DEFAULT NULL,\n reverse boolean DEFAULT FALSE\n) RETURNS text AS $$\n WITH sortby AS (\n SELECT coalesce(_search->'sortby','[{\"field\":\"datetime\", \"direction\":\"desc\"}]') as sort\n ), withid AS (\n SELECT CASE\n WHEN sort @? '$[*] ? (@.field == \"id\")' THEN sort\n ELSE sort || '[{\"field\":\"id\", \"direction\":\"desc\"}]'::jsonb\n END as sort\n FROM sortby\n ), withid_rows AS (\n SELECT jsonb_array_elements(sort) as value FROM withid\n ),sorts AS (\n SELECT\n coalesce(\n -- field_orderby((items_path(value->>'field')).path_txt),\n (queryable(value->>'field')).expression\n ) as key,\n parse_sort_dir(value->>'direction', reverse) as dir\n FROM withid_rows\n )\n SELECT array_to_string(\n array_agg(concat(key, ' ', dir)),\n ', '\n ) FROM sorts;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_sort_dir(sort_item jsonb) RETURNS text AS $$\n SELECT CASE WHEN sort_item->>'direction' ILIKE 'desc%' THEN 'DESC' ELSE 'ASC' END;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION get_token_val_str(\n _field text,\n _item items\n) RETURNS text AS $$\nDECLARE\nliteral text;\nBEGIN\nRAISE NOTICE '% %', _field, _item;\nCREATE TEMP TABLE _token_item ON COMMIT DROP AS SELECT (_item).*;\nEXECUTE format($q$ SELECT quote_literal(%s) FROM _token_item $q$, _field) INTO literal;\nDROP TABLE IF EXISTS _token_item;\nRETURN literal;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION get_token_filter(_search jsonb = '{}'::jsonb, token_rec jsonb DEFAULT NULL) RETURNS text AS $$\nDECLARE\n token_id text;\n filters text[] := '{}'::text[];\n prev boolean := TRUE;\n field text;\n dir text;\n sort record;\n orfilters text[] := '{}'::text[];\n andfilters text[] := '{}'::text[];\n output text;\n token_where text;\n token_item items%ROWTYPE;\nBEGIN\n RAISE NOTICE 'Getting Token Filter. % %', _search, token_rec;\n -- If no token provided return NULL\n IF token_rec IS NULL THEN\n IF NOT (_search ? 'token' AND\n (\n (_search->>'token' ILIKE 'prev:%')\n OR\n (_search->>'token' ILIKE 'next:%')\n )\n ) THEN\n RETURN NULL;\n END IF;\n prev := (_search->>'token' ILIKE 'prev:%');\n token_id := substr(_search->>'token', 6);\n SELECT to_jsonb(items) INTO token_rec\n FROM items WHERE id=token_id;\n END IF;\n RAISE NOTICE 'TOKEN ID: % %', token_rec, token_rec->'id';\n\n\n RAISE NOTICE 'TOKEN ID: % %', token_rec, token_rec->'id';\n token_item := jsonb_populate_record(null::items, token_rec);\n RAISE NOTICE 'TOKEN ITEM ----- %', token_item;\n\n\n CREATE TEMP TABLE sorts (\n _row int GENERATED ALWAYS AS IDENTITY NOT NULL,\n _field text PRIMARY KEY,\n _dir text NOT NULL,\n _val text\n ) ON COMMIT DROP;\n\n -- Make sure we only have distinct columns to sort with taking the first one we get\n INSERT INTO sorts (_field, _dir)\n SELECT\n (queryable(value->>'field')).expression,\n get_sort_dir(value)\n FROM\n jsonb_array_elements(coalesce(_search->'sortby','[{\"field\":\"datetime\",\"direction\":\"desc\"}]'))\n ON CONFLICT DO NOTHING\n ;\n RAISE NOTICE 'sorts 1: %', (SELECT jsonb_agg(to_json(sorts)) FROM sorts);\n -- Get the first sort direction provided. As the id is a primary key, if there are any\n -- sorts after id they won't do anything, so make sure that id is the last sort item.\n SELECT _dir INTO dir FROM sorts ORDER BY _row ASC LIMIT 1;\n IF EXISTS (SELECT 1 FROM sorts WHERE _field = 'id') THEN\n DELETE FROM sorts WHERE _row > (SELECT _row FROM sorts WHERE _field = 'id' ORDER BY _row ASC);\n ELSE\n INSERT INTO sorts (_field, _dir) VALUES ('id', dir);\n END IF;\n\n -- Add value from looked up item to the sorts table\n UPDATE sorts SET _val=get_token_val_str(_field, token_item);\n\n -- Check if all sorts are the same direction and use row comparison\n -- to filter\n RAISE NOTICE 'sorts 2: %', (SELECT jsonb_agg(to_json(sorts)) FROM sorts);\n\n FOR sort IN SELECT * FROM sorts ORDER BY _row asc LOOP\n RAISE NOTICE 'SORT: %', sort;\n IF sort._row = 1 THEN\n IF sort._val IS NULL THEN\n orfilters := orfilters || format('(%s IS NOT NULL)', sort._field);\n ELSE\n orfilters := orfilters || format('(%s %s %s)',\n sort._field,\n CASE WHEN (prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC') THEN '<' ELSE '>' END,\n sort._val\n );\n END IF;\n ELSE\n IF sort._val IS NULL THEN\n orfilters := orfilters || format('(%s AND %s IS NOT NULL)',\n array_to_string(andfilters, ' AND '), sort._field);\n ELSE\n orfilters := orfilters || format('(%s AND %s %s %s)',\n array_to_string(andfilters, ' AND '),\n sort._field,\n CASE WHEN (prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC') THEN '<' ELSE '>' END,\n sort._val\n );\n END IF;\n END IF;\n IF sort._val IS NULL THEN\n andfilters := andfilters || format('%s IS NULL',\n sort._field\n );\n ELSE\n andfilters := andfilters || format('%s = %s',\n sort._field,\n sort._val\n );\n END IF;\n END LOOP;\n output := array_to_string(orfilters, ' OR ');\n\n DROP TABLE IF EXISTS sorts;\n token_where := concat('(',coalesce(output,'true'),')');\n IF trim(token_where) = '' THEN\n token_where := NULL;\n END IF;\n RAISE NOTICE 'TOKEN_WHERE: |%|',token_where;\n RETURN token_where;\n END;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION search_tohash(jsonb) RETURNS jsonb AS $$\n SELECT $1 - '{token,limit,context,includes,excludes}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION search_hash(jsonb, jsonb) RETURNS text AS $$\n SELECT md5(concat(search_tohash($1)::text,$2::text));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TABLE IF NOT EXISTS searches(\n hash text GENERATED ALWAYS AS (search_hash(search, metadata)) STORED PRIMARY KEY,\n search jsonb NOT NULL,\n _where text,\n orderby text,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n metadata jsonb DEFAULT '{}'::jsonb NOT NULL\n);\n\nCREATE TABLE IF NOT EXISTS search_wheres(\n id bigint generated always as identity primary key,\n _where text NOT NULL,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n statslastupdated timestamptz,\n estimated_count bigint,\n estimated_cost float,\n time_to_estimate float,\n total_count bigint,\n time_to_count float,\n partitions text[]\n);\n\nCREATE INDEX IF NOT EXISTS search_wheres_partitions ON search_wheres USING GIN (partitions);\nCREATE UNIQUE INDEX IF NOT EXISTS search_wheres_where ON search_wheres ((md5(_where)));\n\nCREATE OR REPLACE FUNCTION where_stats(inwhere text, updatestats boolean default false, conf jsonb default null) RETURNS search_wheres AS $$\nDECLARE\n t timestamptz;\n i interval;\n explain_json jsonb;\n partitions text[];\n sw search_wheres%ROWTYPE;\n inwhere_hash text := md5(inwhere);\n _context text := lower(context(conf));\n _stats_ttl interval := context_stats_ttl(conf);\n _estimated_cost float := context_estimated_cost(conf);\n _estimated_count int := context_estimated_count(conf);\nBEGIN\n IF _context = 'off' THEN\n sw._where := inwhere;\n return sw;\n END IF;\n\n SELECT * INTO sw FROM search_wheres WHERE md5(_where)=inwhere_hash FOR UPDATE;\n\n -- Update statistics if explicitly set, if statistics do not exist, or statistics ttl has expired\n IF NOT updatestats THEN\n RAISE NOTICE 'Checking if update is needed for: % .', inwhere;\n RAISE NOTICE 'Stats Last Updated: %', sw.statslastupdated;\n RAISE NOTICE 'TTL: %, Age: %', _stats_ttl, now() - sw.statslastupdated;\n RAISE NOTICE 'Context: %, Existing Total: %', _context, sw.total_count;\n IF\n sw.statslastupdated IS NULL\n OR (now() - sw.statslastupdated) > _stats_ttl\n OR (context(conf) != 'off' AND sw.total_count IS NULL)\n THEN\n updatestats := TRUE;\n END IF;\n END IF;\n\n sw._where := inwhere;\n sw.lastused := now();\n sw.usecount := coalesce(sw.usecount,0) + 1;\n\n IF NOT updatestats THEN\n UPDATE search_wheres SET\n lastused = sw.lastused,\n usecount = sw.usecount\n WHERE md5(_where) = inwhere_hash\n RETURNING * INTO sw\n ;\n RETURN sw;\n END IF;\n\n -- Use explain to get estimated count/cost and a list of the partitions that would be hit by the query\n t := clock_timestamp();\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s', inwhere)\n INTO explain_json;\n RAISE NOTICE 'Time for just the explain: %', clock_timestamp() - t;\n i := clock_timestamp() - t;\n\n sw.statslastupdated := now();\n sw.estimated_count := explain_json->0->'Plan'->'Plan Rows';\n sw.estimated_cost := explain_json->0->'Plan'->'Total Cost';\n sw.time_to_estimate := extract(epoch from i);\n\n RAISE NOTICE 'ESTIMATED_COUNT: % < %', sw.estimated_count, _estimated_count;\n RAISE NOTICE 'ESTIMATED_COST: % < %', sw.estimated_cost, _estimated_cost;\n\n -- Do a full count of rows if context is set to on or if auto is set and estimates are low enough\n IF\n _context = 'on'\n OR\n ( _context = 'auto' AND\n (\n sw.estimated_count < _estimated_count\n AND\n sw.estimated_cost < _estimated_cost\n )\n )\n THEN\n t := clock_timestamp();\n RAISE NOTICE 'Calculating actual count...';\n EXECUTE format(\n 'SELECT count(*) FROM items WHERE %s',\n inwhere\n ) INTO sw.total_count;\n i := clock_timestamp() - t;\n RAISE NOTICE 'Actual Count: % -- %', sw.total_count, i;\n sw.time_to_count := extract(epoch FROM i);\n ELSE\n sw.total_count := NULL;\n sw.time_to_count := NULL;\n END IF;\n\n\n INSERT INTO search_wheres\n (_where, lastused, usecount, statslastupdated, estimated_count, estimated_cost, time_to_estimate, partitions, total_count, time_to_count)\n SELECT sw._where, sw.lastused, sw.usecount, sw.statslastupdated, sw.estimated_count, sw.estimated_cost, sw.time_to_estimate, sw.partitions, sw.total_count, sw.time_to_count\n ON CONFLICT ((md5(_where)))\n DO UPDATE\n SET\n lastused = sw.lastused,\n usecount = sw.usecount,\n statslastupdated = sw.statslastupdated,\n estimated_count = sw.estimated_count,\n estimated_cost = sw.estimated_cost,\n time_to_estimate = sw.time_to_estimate,\n total_count = sw.total_count,\n time_to_count = sw.time_to_count\n ;\n RETURN sw;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION search_query(\n _search jsonb = '{}'::jsonb,\n updatestats boolean = false,\n _metadata jsonb = '{}'::jsonb\n) RETURNS searches AS $$\nDECLARE\n search searches%ROWTYPE;\n pexplain jsonb;\n t timestamptz;\n i interval;\nBEGIN\n SELECT * INTO search FROM searches\n WHERE hash=search_hash(_search, _metadata) FOR UPDATE;\n\n -- Calculate the where clause if not already calculated\n IF search._where IS NULL THEN\n search._where := stac_search_to_where(_search);\n END IF;\n\n -- Calculate the order by clause if not already calculated\n IF search.orderby IS NULL THEN\n search.orderby := sort_sqlorderby(_search);\n END IF;\n\n PERFORM where_stats(search._where, updatestats, _search->'conf');\n\n search.lastused := now();\n search.usecount := coalesce(search.usecount, 0) + 1;\n INSERT INTO searches (search, _where, orderby, lastused, usecount, metadata)\n VALUES (_search, search._where, search.orderby, search.lastused, search.usecount, _metadata)\n ON CONFLICT (hash) DO\n UPDATE SET\n _where = EXCLUDED._where,\n orderby = EXCLUDED.orderby,\n lastused = EXCLUDED.lastused,\n usecount = EXCLUDED.usecount,\n metadata = EXCLUDED.metadata\n RETURNING * INTO search\n ;\n RETURN search;\n\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE OR REPLACE FUNCTION search(_search jsonb = '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n searches searches%ROWTYPE;\n _where text;\n token_where text;\n full_where text;\n orderby text;\n query text;\n token_type text := substr(_search->>'token',1,4);\n _limit int := coalesce((_search->>'limit')::int, 10);\n curs refcursor;\n cntr int := 0;\n iter_record items%ROWTYPE;\n first_record jsonb;\n first_item items%ROWTYPE;\n last_item items%ROWTYPE;\n last_record jsonb;\n out_records jsonb := '[]'::jsonb;\n prev_query text;\n next text;\n prev_id text;\n has_next boolean := false;\n has_prev boolean := false;\n prev text;\n total_count bigint;\n context jsonb;\n collection jsonb;\n includes text[];\n excludes text[];\n exit_flag boolean := FALSE;\n batches int := 0;\n timer timestamptz := clock_timestamp();\n pstart timestamptz;\n pend timestamptz;\n pcurs refcursor;\n search_where search_wheres%ROWTYPE;\n id text;\nBEGIN\nCREATE TEMP TABLE results (i int GENERATED ALWAYS AS IDENTITY, content jsonb) ON COMMIT DROP;\n-- if ids is set, short circuit and just use direct ids query for each id\n-- skip any paging or caching\n-- hard codes ordering in the same order as the array of ids\nIF _search ? 'ids' THEN\n INSERT INTO results (content)\n SELECT\n CASE WHEN _search->'conf'->>'nohydrate' IS NOT NULL AND (_search->'conf'->>'nohydrate')::boolean = true THEN\n content_nonhydrated(items, _search->'fields')\n ELSE\n content_hydrate(items, _search->'fields')\n END\n FROM items WHERE\n items.id = ANY(to_text_array(_search->'ids'))\n AND\n CASE WHEN _search ? 'collections' THEN\n items.collection = ANY(to_text_array(_search->'collections'))\n ELSE TRUE\n END\n ORDER BY items.datetime desc, items.id desc\n ;\n SELECT INTO total_count count(*) FROM results;\nELSE\n searches := search_query(_search);\n _where := searches._where;\n orderby := searches.orderby;\n search_where := where_stats(_where);\n total_count := coalesce(search_where.total_count, search_where.estimated_count);\n\n IF token_type='prev' THEN\n token_where := get_token_filter(_search, null::jsonb);\n orderby := sort_sqlorderby(_search, TRUE);\n END IF;\n IF token_type='next' THEN\n token_where := get_token_filter(_search, null::jsonb);\n END IF;\n\n full_where := concat_ws(' AND ', _where, token_where);\n RAISE NOTICE 'FULL QUERY % %', full_where, clock_timestamp()-timer;\n timer := clock_timestamp();\n\n FOR query IN SELECT partition_queries(full_where, orderby, search_where.partitions) LOOP\n timer := clock_timestamp();\n query := format('%s LIMIT %s', query, _limit + 1);\n RAISE NOTICE 'Partition Query: %', query;\n batches := batches + 1;\n -- curs = create_cursor(query);\n RAISE NOTICE 'cursor_tuple_fraction: %', current_setting('cursor_tuple_fraction');\n OPEN curs FOR EXECUTE query;\n LOOP\n FETCH curs into iter_record;\n EXIT WHEN NOT FOUND;\n cntr := cntr + 1;\n\n IF _search->'conf'->>'nohydrate' IS NOT NULL AND (_search->'conf'->>'nohydrate')::boolean = true THEN\n last_record := content_nonhydrated(iter_record, _search->'fields');\n ELSE\n last_record := content_hydrate(iter_record, _search->'fields');\n END IF;\n last_item := iter_record;\n IF cntr = 1 THEN\n first_item := last_item;\n first_record := last_record;\n END IF;\n IF cntr <= _limit THEN\n INSERT INTO results (content) VALUES (last_record);\n ELSIF cntr > _limit THEN\n has_next := true;\n exit_flag := true;\n EXIT;\n END IF;\n END LOOP;\n CLOSE curs;\n RAISE NOTICE 'Query took %.', clock_timestamp()-timer;\n timer := clock_timestamp();\n EXIT WHEN exit_flag;\n END LOOP;\n RAISE NOTICE 'Scanned through % partitions.', batches;\nEND IF;\n\nWITH ordered AS (SELECT * FROM results WHERE content IS NOT NULL ORDER BY i)\nSELECT jsonb_agg(content) INTO out_records FROM ordered;\n\nDROP TABLE results;\n\n\n-- Flip things around if this was the result of a prev token query\nIF token_type='prev' THEN\n out_records := flip_jsonb_array(out_records);\n first_item := last_item;\n first_record := last_record;\nEND IF;\n\n-- If this query has a token, see if there is data before the first record\nIF _search ? 'token' THEN\n prev_query := format(\n 'SELECT 1 FROM items WHERE %s LIMIT 1',\n concat_ws(\n ' AND ',\n _where,\n trim(get_token_filter(_search, to_jsonb(first_item)))\n )\n );\n RAISE NOTICE 'Query to get previous record: % --- %', prev_query, first_record;\n EXECUTE prev_query INTO has_prev;\n IF FOUND and has_prev IS NOT NULL THEN\n RAISE NOTICE 'Query results from prev query: %', has_prev;\n has_prev := TRUE;\n END IF;\nEND IF;\nhas_prev := COALESCE(has_prev, FALSE);\n\nIF has_prev THEN\n prev := out_records->0->>'id';\nEND IF;\nIF has_next OR token_type='prev' THEN\n next := out_records->-1->>'id';\nEND IF;\n\nIF context(_search->'conf') != 'off' THEN\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'matched', total_count,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\nELSE\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\nEND IF;\n\ncollection := jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb),\n 'next', next,\n 'prev', prev,\n 'context', context\n);\n\nRETURN collection;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac, public SET cursor_tuple_fraction TO 1;\n\n\nCREATE OR REPLACE FUNCTION search_cursor(_search jsonb = '{}'::jsonb) RETURNS refcursor AS $$\nDECLARE\n curs refcursor;\n searches searches%ROWTYPE;\n _where text;\n _orderby text;\n q text;\n\nBEGIN\n searches := search_query(_search);\n _where := searches._where;\n _orderby := searches.orderby;\n\n OPEN curs FOR\n WITH p AS (\n SELECT * FROM partition_queries(_where, _orderby) p\n )\n SELECT\n CASE WHEN EXISTS (SELECT 1 FROM p) THEN\n (SELECT format($q$\n SELECT * FROM (\n %s\n ) total\n $q$,\n string_agg(\n format($q$ SELECT * FROM ( %s ) AS sub $q$, p),\n '\n UNION ALL\n '\n )\n ))\n ELSE NULL\n END FROM p;\n RETURN curs;\nEND;\n$$ LANGUAGE PLPGSQL;\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION tileenvelope(zoom int, x int, y int) RETURNS geometry AS $$\nWITH t AS (\n SELECT\n 20037508.3427892 as merc_max,\n -20037508.3427892 as merc_min,\n (2 * 20037508.3427892) / (2 ^ zoom) as tile_size\n)\nSELECT st_makeenvelope(\n merc_min + (tile_size * x),\n merc_max - (tile_size * (y + 1)),\n merc_min + (tile_size * (x + 1)),\n merc_max - (tile_size * y),\n 3857\n) FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;DROP FUNCTION IF EXISTS mercgrid;\n\n\nCREATE OR REPLACE FUNCTION ftime() RETURNS interval as $$\nSELECT age(clock_timestamp(), transaction_timestamp());\n$$ LANGUAGE SQL;\nSET SEARCH_PATH to pgstac, public;\n\nDROP FUNCTION IF EXISTS geometrysearch;\nCREATE OR REPLACE FUNCTION geometrysearch(\n IN geom geometry,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE, -- Return as soon as the passed in geometry is full covered\n IN skipcovered boolean DEFAULT TRUE -- Skip any items that would show up completely under the previous items\n) RETURNS jsonb AS $$\nDECLARE\n search searches%ROWTYPE;\n curs refcursor;\n _where text;\n query text;\n iter_record items%ROWTYPE;\n out_records jsonb := '{}'::jsonb[];\n exit_flag boolean := FALSE;\n counter int := 1;\n scancounter int := 1;\n remaining_limit int := _scanlimit;\n tilearea float;\n unionedgeom geometry;\n clippedgeom geometry;\n unionedgeom_area float := 0;\n prev_area float := 0;\n excludes text[];\n includes text[];\n\nBEGIN\n DROP TABLE IF EXISTS pgstac_results;\n CREATE TEMP TABLE pgstac_results (content jsonb) ON COMMIT DROP;\n\n -- If skipcovered is true then you will always want to exit when the passed in geometry is full\n IF skipcovered THEN\n exitwhenfull := TRUE;\n END IF;\n\n SELECT * INTO search FROM searches WHERE hash=queryhash;\n\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Search with Query Hash % Not Found', queryhash;\n END IF;\n\n tilearea := st_area(geom);\n _where := format('%s AND st_intersects(geometry, %L::geometry)', search._where, geom);\n\n\n FOR query IN SELECT * FROM partition_queries(_where, search.orderby) LOOP\n query := format('%s LIMIT %L', query, remaining_limit);\n RAISE NOTICE '%', query;\n OPEN curs FOR EXECUTE query;\n LOOP\n FETCH curs INTO iter_record;\n EXIT WHEN NOT FOUND;\n IF exitwhenfull OR skipcovered THEN -- If we are not using exitwhenfull or skipcovered, we do not need to do expensive geometry operations\n clippedgeom := st_intersection(geom, iter_record.geometry);\n\n IF unionedgeom IS NULL THEN\n unionedgeom := clippedgeom;\n ELSE\n unionedgeom := st_union(unionedgeom, clippedgeom);\n END IF;\n\n unionedgeom_area := st_area(unionedgeom);\n\n IF skipcovered AND prev_area = unionedgeom_area THEN\n scancounter := scancounter + 1;\n CONTINUE;\n END IF;\n\n prev_area := unionedgeom_area;\n\n RAISE NOTICE '% % % %', unionedgeom_area/tilearea, counter, scancounter, ftime();\n END IF;\n RAISE NOTICE '% %', iter_record, content_hydrate(iter_record, fields);\n INSERT INTO pgstac_results (content) VALUES (content_hydrate(iter_record, fields));\n\n IF counter >= _limit\n OR scancounter > _scanlimit\n OR ftime() > _timelimit\n OR (exitwhenfull AND unionedgeom_area >= tilearea)\n THEN\n exit_flag := TRUE;\n EXIT;\n END IF;\n counter := counter + 1;\n scancounter := scancounter + 1;\n\n END LOOP;\n CLOSE curs;\n EXIT WHEN exit_flag;\n remaining_limit := _scanlimit - scancounter;\n END LOOP;\n\n SELECT jsonb_agg(content) INTO out_records FROM pgstac_results WHERE content IS NOT NULL;\n\n RETURN jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb)\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDROP FUNCTION IF EXISTS geojsonsearch;\nCREATE OR REPLACE FUNCTION geojsonsearch(\n IN geojson jsonb,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_geomfromgeojson(geojson),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS xyzsearch;\nCREATE OR REPLACE FUNCTION xyzsearch(\n IN _x int,\n IN _y int,\n IN _z int,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_transform(tileenvelope(_z, _x, _y), 4326),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS analyze_items;\nCREATE OR REPLACE PROCEDURE analyze_items() AS $$\nDECLARE\n q text;\n timeout_ts timestamptz;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n RAISE NOTICE '% % %', clock_timestamp(), timeout_ts, current_setting('statement_timeout', TRUE);\n SELECT format('ANALYZE (VERBOSE, SKIP_LOCKED) %I;', relname) INTO q\n FROM pg_stat_user_tables\n WHERE relname like '_item%' AND (n_mod_since_analyze>0 OR last_analyze IS NULL) LIMIT 1;\n IF NOT FOUND THEN\n EXIT;\n END IF;\n RAISE NOTICE '%', q;\n EXECUTE q;\n COMMIT;\n RAISE NOTICE '%', queue_timeout();\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nDROP FUNCTION IF EXISTS validate_constraints;\nCREATE OR REPLACE PROCEDURE validate_constraints() AS $$\nDECLARE\n q text;\nBEGIN\n FOR q IN\n SELECT\n FORMAT(\n 'ALTER TABLE %I.%I VALIDATE CONSTRAINT %I;',\n nsp.nspname,\n cls.relname,\n con.conname\n )\n\n FROM pg_constraint AS con\n JOIN pg_class AS cls\n ON con.conrelid = cls.oid\n JOIN pg_namespace AS nsp\n ON cls.relnamespace = nsp.oid\n WHERE convalidated = FALSE AND contype in ('c','f')\n AND nsp.nspname = 'pgstac'\n LOOP\n RAISE NOTICE '%', q;\n PERFORM run_or_queue(q);\n COMMIT;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION collection_extent(_collection text, runupdate boolean default false) RETURNS jsonb AS $$\nDECLARE\n geom_extent geometry;\n mind timestamptz;\n maxd timestamptz;\n extent jsonb;\nBEGIN\n IF runupdate THEN\n PERFORM update_partition_stats_q(partition)\n FROM partitions WHERE collection=_collection;\n END IF;\n SELECT\n min(lower(dtrange)),\n max(upper(edtrange)),\n st_extent(spatial)\n INTO\n mind,\n maxd,\n geom_extent\n FROM partitions\n WHERE collection=_collection;\n\n IF geom_extent IS NOT NULL AND mind IS NOT NULL AND maxd IS NOT NULL THEN\n extent := jsonb_build_object(\n 'extent', jsonb_build_object(\n 'spatial', jsonb_build_object(\n 'bbox', to_jsonb(array[array[st_xmin(geom_extent), st_ymin(geom_extent), st_xmax(geom_extent), st_ymax(geom_extent)]])\n ),\n 'temporal', jsonb_build_object(\n 'interval', to_jsonb(array[array[mind, maxd]])\n )\n )\n );\n RETURN extent;\n END IF;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\nINSERT INTO queryables (name, definition) VALUES\n('id', '{\"title\": \"Item ID\",\"description\": \"Item identifier\",\"$ref\": \"https://schemas.stacspec.org/v1.0.0/item-spec/json-schema/item.json#/definitions/core/allOf/2/properties/id\"}'),\n('datetime','{\"description\": \"Datetime\",\"type\": \"string\",\"title\": \"Acquired\",\"format\": \"date-time\",\"pattern\": \"(\\\\+00:00|Z)$\"}'),\n('geometry', '{\"title\": \"Item Geometry\",\"description\": \"Item Geometry\",\"$ref\": \"https://geojson.org/schema/Feature.json\"}')\nON CONFLICT DO NOTHING;\n\nINSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n('eo:cloud_cover','{\"$ref\": \"https://stac-extensions.github.io/eo/v1.0.0/schema.json#/definitions/fieldsproperties/eo:cloud_cover\"}','to_int','BTREE')\nON CONFLICT DO NOTHING;\n\n\nINSERT INTO pgstac_settings (name, value) VALUES\n ('context', 'off'),\n ('context_estimated_count', '100000'),\n ('context_estimated_cost', '100000'),\n ('context_stats_ttl', '1 day'),\n ('default_filter_lang', 'cql2-json'),\n ('additional_properties', 'true'),\n ('use_queue', 'false'),\n ('queue_timeout', '10 minutes'),\n ('update_collection_extent', 'true')\nON CONFLICT DO NOTHING\n;\n\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nGRANT ALL ON SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac to pgstac_admin;\n\n-- pgstac_read role limited to using function apis\nGRANT EXECUTE ON FUNCTION search TO pgstac_read;\nGRANT EXECUTE ON FUNCTION search_query TO pgstac_read;\nGRANT EXECUTE ON FUNCTION item_by_id TO pgstac_read;\nGRANT EXECUTE ON FUNCTION get_item TO pgstac_read;\n\nGRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_ingest;\nGRANT USAGE ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_ingest;\n\nSELECT update_partition_stats_q(partition) FROM partitions;\nSELECT set_version('0.7.0');\n" }, { "path": "src/pgstac/migrations/pgstac.0.7.1-0.7.2.sql", "content": "RESET ROLE;\nDO $$\nDECLARE\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='postgis') THEN\n CREATE EXTENSION IF NOT EXISTS postgis;\n END IF;\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='btree_gist') THEN\n CREATE EXTENSION IF NOT EXISTS btree_gist;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_admin;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_read;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_ingest;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n\nGRANT pgstac_admin TO current_user;\n\n-- Function to make sure pgstac_admin is the owner of items\nCREATE OR REPLACE FUNCTION pgstac_admin_owns() RETURNS VOID AS $$\nDECLARE\n f RECORD;\nBEGIN\n FOR f IN (\n SELECT\n concat(\n oid::regproc::text,\n '(',\n coalesce(pg_get_function_identity_arguments(oid),''),\n ')'\n ) AS name,\n CASE prokind WHEN 'f' THEN 'FUNCTION' WHEN 'p' THEN 'PROCEDURE' WHEN 'a' THEN 'AGGREGATE' END as typ\n FROM pg_proc\n WHERE\n pronamespace=to_regnamespace('pgstac')\n AND proowner != to_regrole('pgstac_admin')\n AND proname NOT LIKE 'pg_stat%'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n FOR f IN (\n SELECT\n oid::regclass::text as name,\n CASE relkind\n WHEN 'i' THEN 'INDEX'\n WHEN 'I' THEN 'INDEX'\n WHEN 'p' THEN 'TABLE'\n WHEN 'r' THEN 'TABLE'\n WHEN 'v' THEN 'VIEW'\n WHEN 'S' THEN 'SEQUENCE'\n ELSE NULL\n END as typ\n FROM pg_class\n WHERE relnamespace=to_regnamespace('pgstac') and relowner != to_regrole('pgstac_admin') AND relkind IN ('r','p','v','S') AND relname NOT LIKE 'pg_stat'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\nSELECT pgstac_admin_owns();\n\nCREATE SCHEMA IF NOT EXISTS pgstac AUTHORIZATION pgstac_admin;\n\nGRANT ALL ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_admin;\n\nALTER ROLE pgstac_admin SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_read SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_ingest SET SEARCH_PATH TO pgstac, public;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\n\nGRANT pgstac_read TO pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\n\nSET ROLE pgstac_admin;\n\nSET SEARCH_PATH TO pgstac, public;\n\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS analyze_items;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS validate_constraints;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\nSET client_min_messages TO WARNING;\nSET SEARCH_PATH to pgstac, public;\n-- BEGIN migra calculated SQL\nset check_function_bodies = off;\n\nCREATE OR REPLACE FUNCTION pgstac.maintain_partition_queries(part text DEFAULT 'items'::text, dropindexes boolean DEFAULT false, rebuildindexes boolean DEFAULT false, idxconcurrently boolean DEFAULT false)\n RETURNS SETOF text\n LANGUAGE plpgsql\nAS $function$\nDECLARE\n parent text;\n level int;\n isleaf bool;\n collection collections%ROWTYPE;\n subpart text;\n baseidx text;\n queryable_name text;\n queryable_property_index_type text;\n queryable_property_wrapper text;\n queryable_parsed RECORD;\n deletedidx pg_indexes%ROWTYPE;\n q text;\n idx text;\n collection_partition bigint;\n _concurrently text := '';\n idxname text;\nBEGIN\n RAISE NOTICE 'Maintaining partition: %', part;\n IF idxconcurrently THEN\n _concurrently='CONCURRENTLY';\n END IF;\n\n -- Get root partition\n SELECT parentrelid::text, pt.isleaf, pt.level\n INTO parent, isleaf, level\n FROM pg_partition_tree('items') pt\n WHERE relid::text = part;\n IF NOT FOUND THEN\n RAISE NOTICE 'Partition % Does Not Exist In Partition Tree', part;\n RETURN;\n END IF;\n\n -- If this is a parent partition, recurse to leaves\n IF NOT isleaf THEN\n FOR subpart IN\n SELECT relid::text\n FROM pg_partition_tree(part)\n WHERE relid::text != part\n LOOP\n RAISE NOTICE 'Recursing to %', subpart;\n RETURN QUERY SELECT * FROM maintain_partition_queries(subpart, dropindexes, rebuildindexes);\n END LOOP;\n RETURN; -- Don't continue since not an end leaf\n END IF;\n\n\n -- Get collection\n collection_partition := ((regexp_match(part, E'^_items_([0-9]+)'))[1])::bigint;\n RAISE NOTICE 'COLLECTION PARTITION: %', collection_partition;\n SELECT * INTO STRICT collection\n FROM collections\n WHERE key = collection_partition;\n RAISE NOTICE 'COLLECTION ID: %s', collection.id;\n\n\n -- Create temp table with existing indexes\n CREATE TEMP TABLE existing_indexes ON COMMIT DROP AS\n SELECT *\n FROM pg_indexes\n WHERE schemaname='pgstac' AND tablename=part;\n\n\n -- Check if index exists for each queryable.\n FOR\n queryable_name,\n queryable_property_index_type,\n queryable_property_wrapper\n IN\n SELECT\n name,\n COALESCE(property_index_type, 'BTREE'),\n COALESCE(property_wrapper, 'to_text')\n FROM queryables\n WHERE\n name NOT in ('id', 'datetime', 'geometry')\n AND (\n collection_ids IS NULL\n OR collection_ids = '{}'::text[]\n OR collection.id = ANY (collection_ids)\n )\n LOOP\n baseidx := format(\n $q$ ON %I USING %s (%s(((content -> 'properties'::text) -> %L::text)))$q$,\n part,\n queryable_property_index_type,\n queryable_property_wrapper,\n queryable_name\n );\n RAISE NOTICE 'BASEIDX: %', baseidx;\n RAISE NOTICE 'IDXSEARCH: %', format($q$[(']%s[')]$q$, queryable_name);\n -- If index already exists, delete it from existing indexes type table\n FOR deletedidx IN\n DELETE FROM existing_indexes\n WHERE indexdef ~* format($q$[(']%s[')]$q$, queryable_name)\n RETURNING *\n LOOP\n RAISE NOTICE 'EXISTING INDEX: %', deletedidx;\n IF NOT FOUND THEN -- index did not exist, create it\n RETURN NEXT format('CREATE INDEX %s %s;', _concurrently, baseidx);\n ELSIF rebuildindexes THEN\n RETURN NEXT format('REINDEX %I %s;', deletedidx.indexname, _concurrently);\n END IF;\n END LOOP;\n IF NOT FOUND THEN\n RAISE NOTICE 'CREATING INDEX for %', queryable_name;\n RETURN NEXT format('CREATE INDEX %s %s;', _concurrently, baseidx);\n END IF;\n END LOOP;\n IF NOT EXISTS (SELECT * FROM pg_indexes WHERE indexname::text = concat(part, '_datetime_end_datetime_idx')) THEN\n RETURN NEXT format(\n $f$CREATE INDEX IF NOT EXISTS %I ON %I USING BTREE (datetime DESC, end_datetime ASC);$f$,\n concat(part, '_datetime_end_datetime_idx'),\n part\n );\n END IF;\n\n IF NOT EXISTS (SELECT * FROM pg_indexes WHERE indexname::text = concat(part, '_geometry_idx')) THEN\n RETURN NEXT format(\n $f$CREATE INDEX IF NOT EXISTS %I ON %I USING GIST (geometry);$f$,\n concat(part, '_geometry_idx'),\n part\n );\n END IF;\n\n FOR idxname IN\n SELECT indexname::text FROM pg_indexes\n WHERE tablename::text = part AND indexdef ILIKE 'CREATE UNIQUE INDEX % USING btree(id)'\n LOOP\n IF idxname != concat(part, '_pk') AND NOT EXISTS (SELECT * FROM pg_indexes WHERE indexname::text = concat(part,'_pk')) THEN\n RETURN NEXT format(\n $f$ALTER INDEX IF EXISTS %I RENAME TO %I;$f$,\n idxname,\n concat(part, '_pk')\n );\n ELSIF EXISTS (SELECT * FROM pg_indexes WHERE indexname::text = concat(part,'_pk')) AND dropindexes THEN\n RETURN NEXT format(\n $f$DROP INDEX IF EXISTS %I;$f$,\n idxname\n );\n END IF;\n END LOOP;\n\n IF NOT EXISTS (\n SELECT indexname::text FROM pg_indexes\n WHERE tablename::text = part AND indexdef ILIKE 'CREATE UNIQUE INDEX % USING btree (id)'\n ) THEN\n RETURN NEXT format(\n $f$CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I USING BTREE(id);$f$,\n concat(part, '_pk'),\n part\n );\n END IF;\n\n DELETE FROM existing_indexes WHERE indexname::text IN (\n concat(part, '_datetime_end_datetime_idx'),\n concat(part, '_geometry_idx'),\n concat(part, '_pk')\n );\n -- Remove indexes that were not expected\n FOR idx IN SELECT indexname::text FROM existing_indexes\n LOOP\n RAISE WARNING 'Index: % is not defined by queryables.', idx;\n IF dropindexes THEN\n RETURN NEXT format('DROP INDEX IF EXISTS %I;', idx);\n END IF;\n END LOOP;\n\n DROP TABLE existing_indexes;\n RAISE NOTICE 'Returning from maintain_partition_queries.';\n RETURN;\n\nEND;\n$function$\n;\n\n\n\n-- END migra calculated SQL\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('id', '{\"title\": \"Item ID\",\"description\": \"Item identifier\",\"$ref\": \"https://schemas.stacspec.org/v1.0.0/item-spec/json-schema/item.json#/definitions/core/allOf/2/properties/id\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('geometry', '{\"title\": \"Item Geometry\",\"description\": \"Item Geometry\",\"$ref\": \"https://geojson.org/schema/Feature.json\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('datetime','{\"description\": \"Datetime\",\"type\": \"string\",\"title\": \"Acquired\",\"format\": \"date-time\",\"pattern\": \"(\\\\+00:00|Z)$\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('eo:cloud_cover','{\"$ref\": \"https://stac-extensions.github.io/eo/v1.0.0/schema.json#/definitions/fieldsproperties/eo:cloud_cover\"}','to_int','BTREE');\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDELETE FROM queryables a USING queryables b\n WHERE a.name = b.name AND a.collection_ids IS NOT DISTINCT FROM b.collection_ids AND a.id > b.id;\n\n\nINSERT INTO pgstac_settings (name, value) VALUES\n ('context', 'off'),\n ('context_estimated_count', '100000'),\n ('context_estimated_cost', '100000'),\n ('context_stats_ttl', '1 day'),\n ('default_filter_lang', 'cql2-json'),\n ('additional_properties', 'true'),\n ('use_queue', 'false'),\n ('queue_timeout', '10 minutes'),\n ('update_collection_extent', 'false')\nON CONFLICT DO NOTHING\n;\n\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nGRANT ALL ON SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac to pgstac_admin;\n\n-- pgstac_read role limited to using function apis\nGRANT EXECUTE ON FUNCTION search TO pgstac_read;\nGRANT EXECUTE ON FUNCTION search_query TO pgstac_read;\nGRANT EXECUTE ON FUNCTION item_by_id TO pgstac_read;\nGRANT EXECUTE ON FUNCTION get_item TO pgstac_read;\nGRANT SELECT ON ALL TABLES IN SCHEMA pgstac TO pgstac_read;\n\n\nGRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_ingest;\nGRANT USAGE ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_ingest;\n\nSELECT update_partition_stats_q(partition) FROM partitions;\nSELECT set_version('0.7.2');\n" }, { "path": "src/pgstac/migrations/pgstac.0.7.1.sql", "content": "RESET ROLE;\nDO $$\nDECLARE\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='postgis') THEN\n CREATE EXTENSION IF NOT EXISTS postgis;\n END IF;\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='btree_gist') THEN\n CREATE EXTENSION IF NOT EXISTS btree_gist;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_admin;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_read;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_ingest;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n\nGRANT pgstac_admin TO current_user;\n\n-- Function to make sure pgstac_admin is the owner of items\nCREATE OR REPLACE FUNCTION pgstac_admin_owns() RETURNS VOID AS $$\nDECLARE\n f RECORD;\nBEGIN\n FOR f IN (\n SELECT\n concat(\n oid::regproc::text,\n '(',\n coalesce(pg_get_function_identity_arguments(oid),''),\n ')'\n ) AS name,\n CASE prokind WHEN 'f' THEN 'FUNCTION' WHEN 'p' THEN 'PROCEDURE' WHEN 'a' THEN 'AGGREGATE' END as typ\n FROM pg_proc\n WHERE\n pronamespace=to_regnamespace('pgstac')\n AND proowner != to_regrole('pgstac_admin')\n AND proname NOT LIKE 'pg_stat%'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n FOR f IN (\n SELECT\n oid::regclass::text as name,\n CASE relkind\n WHEN 'i' THEN 'INDEX'\n WHEN 'I' THEN 'INDEX'\n WHEN 'p' THEN 'TABLE'\n WHEN 'r' THEN 'TABLE'\n WHEN 'v' THEN 'VIEW'\n WHEN 'S' THEN 'SEQUENCE'\n ELSE NULL\n END as typ\n FROM pg_class\n WHERE relnamespace=to_regnamespace('pgstac') and relowner != to_regrole('pgstac_admin') AND relkind IN ('r','p','v','S') AND relname NOT LIKE 'pg_stat'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\nSELECT pgstac_admin_owns();\n\nCREATE SCHEMA IF NOT EXISTS pgstac AUTHORIZATION pgstac_admin;\n\nGRANT ALL ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_admin;\n\nALTER ROLE pgstac_admin SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_read SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_ingest SET SEARCH_PATH TO pgstac, public;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\n\nGRANT pgstac_read TO pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\n\nSET ROLE pgstac_admin;\n\nSET SEARCH_PATH TO pgstac, public;\n\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS analyze_items;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS validate_constraints;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\nCREATE OR REPLACE FUNCTION to_int(jsonb) RETURNS int AS $$\n SELECT floor(($1->>0)::float)::int;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION to_float(jsonb) RETURNS float AS $$\n SELECT ($1->>0)::float;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION to_tstz(jsonb) RETURNS timestamptz AS $$\n SELECT ($1->>0)::timestamptz;\n$$ LANGUAGE SQL IMMUTABLE STRICT SET TIME ZONE 'UTC';\n\n\nCREATE OR REPLACE FUNCTION to_text(jsonb) RETURNS text AS $$\n SELECT CASE WHEN jsonb_typeof($1) IN ('array','object') THEN $1::text ELSE $1->>0 END;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION to_text_array(jsonb) RETURNS text[] AS $$\n SELECT\n CASE jsonb_typeof($1)\n WHEN 'array' THEN ARRAY(SELECT jsonb_array_elements_text($1))\n ELSE ARRAY[$1->>0]\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION bbox_geom(_bbox jsonb) RETURNS geometry AS $$\nSELECT CASE jsonb_array_length(_bbox)\n WHEN 4 THEN\n ST_SetSRID(ST_MakeEnvelope(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float,\n (_bbox->>3)::float\n ),4326)\n WHEN 6 THEN\n ST_SetSRID(ST_3DMakeBox(\n ST_MakePoint(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float\n ),\n ST_MakePoint(\n (_bbox->>3)::float,\n (_bbox->>4)::float,\n (_bbox->>5)::float\n )\n ),4326)\n ELSE null END;\n;\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION geom_bbox(_geom geometry) RETURNS jsonb AS $$\n SELECT jsonb_build_array(\n st_xmin(_geom),\n st_ymin(_geom),\n st_xmax(_geom),\n st_ymax(_geom)\n );\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION flip_jsonb_array(j jsonb) RETURNS jsonb AS $$\n SELECT jsonb_agg(value) FROM (SELECT value FROM jsonb_array_elements(j) WITH ORDINALITY ORDER BY ordinality DESC) as t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION explode_dotpaths(j jsonb) RETURNS SETOF text[] AS $$\n SELECT string_to_array(p, '.') as e FROM jsonb_array_elements_text(j) p;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION explode_dotpaths_recurse(IN j jsonb) RETURNS SETOF text[] AS $$\n WITH RECURSIVE t AS (\n SELECT e FROM explode_dotpaths(j) e\n UNION ALL\n SELECT e[1:cardinality(e)-1]\n FROM t\n WHERE cardinality(e)>1\n ) SELECT e FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION jsonb_set_nested(j jsonb, path text[], val jsonb) RETURNS jsonb AS $$\nDECLARE\nBEGIN\n IF cardinality(path) > 1 THEN\n FOR i IN 1..(cardinality(path)-1) LOOP\n IF j #> path[:i] IS NULL THEN\n j := jsonb_set_lax(j, path[:i], '{}', TRUE);\n END IF;\n END LOOP;\n END IF;\n RETURN jsonb_set_lax(j, path, val, true);\n\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\n\n\nCREATE OR REPLACE FUNCTION jsonb_include(j jsonb, f jsonb) RETURNS jsonb AS $$\nDECLARE\n includes jsonb := f-> 'include';\n outj jsonb := '{}'::jsonb;\n path text[];\nBEGIN\n IF\n includes IS NULL\n OR jsonb_array_length(includes) = 0\n THEN\n RETURN j;\n ELSE\n includes := includes || '[\"id\",\"collection\"]'::jsonb;\n FOR path IN SELECT explode_dotpaths(includes) LOOP\n outj := jsonb_set_nested(outj, path, j #> path);\n END LOOP;\n END IF;\n RETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_exclude(j jsonb, f jsonb) RETURNS jsonb AS $$\nDECLARE\n excludes jsonb := f-> 'exclude';\n outj jsonb := j;\n path text[];\nBEGIN\n IF\n excludes IS NULL\n OR jsonb_array_length(excludes) = 0\n THEN\n RETURN j;\n ELSE\n FOR path IN SELECT explode_dotpaths(excludes) LOOP\n outj := outj #- path;\n END LOOP;\n END IF;\n RETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_fields(j jsonb, f jsonb DEFAULT '{\"fields\":[]}') RETURNS jsonb AS $$\n SELECT jsonb_exclude(jsonb_include(j, f), f);\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION merge_jsonb(_a jsonb, _b jsonb) RETURNS jsonb AS $$\n SELECT\n CASE\n WHEN _a = '\"𒍟※\"'::jsonb THEN NULL\n WHEN _a IS NULL OR jsonb_typeof(_a) = 'null' THEN _b\n WHEN jsonb_typeof(_a) = 'object' AND jsonb_typeof(_b) = 'object' THEN\n (\n SELECT\n jsonb_strip_nulls(\n jsonb_object_agg(\n key,\n merge_jsonb(a.value, b.value)\n )\n )\n FROM\n jsonb_each(coalesce(_a,'{}'::jsonb)) as a\n FULL JOIN\n jsonb_each(coalesce(_b,'{}'::jsonb)) as b\n USING (key)\n )\n WHEN\n jsonb_typeof(_a) = 'array'\n AND jsonb_typeof(_b) = 'array'\n AND jsonb_array_length(_a) = jsonb_array_length(_b)\n THEN\n (\n SELECT jsonb_agg(m) FROM\n ( SELECT\n merge_jsonb(\n jsonb_array_elements(_a),\n jsonb_array_elements(_b)\n ) as m\n ) as l\n )\n ELSE _a\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION strip_jsonb(_a jsonb, _b jsonb) RETURNS jsonb AS $$\n SELECT\n CASE\n\n WHEN (_a IS NULL OR jsonb_typeof(_a) = 'null') AND _b IS NOT NULL AND jsonb_typeof(_b) != 'null' THEN '\"𒍟※\"'::jsonb\n WHEN _b IS NULL OR jsonb_typeof(_a) = 'null' THEN _a\n WHEN _a = _b AND jsonb_typeof(_a) = 'object' THEN '{}'::jsonb\n WHEN _a = _b THEN NULL\n WHEN jsonb_typeof(_a) = 'object' AND jsonb_typeof(_b) = 'object' THEN\n (\n SELECT\n jsonb_strip_nulls(\n jsonb_object_agg(\n key,\n strip_jsonb(a.value, b.value)\n )\n )\n FROM\n jsonb_each(_a) as a\n FULL JOIN\n jsonb_each(_b) as b\n USING (key)\n )\n WHEN\n jsonb_typeof(_a) = 'array'\n AND jsonb_typeof(_b) = 'array'\n AND jsonb_array_length(_a) = jsonb_array_length(_b)\n THEN\n (\n SELECT jsonb_agg(m) FROM\n ( SELECT\n strip_jsonb(\n jsonb_array_elements(_a),\n jsonb_array_elements(_b)\n ) as m\n ) as l\n )\n ELSE _a\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION nullif_jsonbnullempty(j jsonb) RETURNS jsonb AS $$\n SELECT nullif(nullif(nullif(j,'null'::jsonb),'{}'::jsonb),'[]'::jsonb);\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION jsonb_array_unique(j jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(jsonb_agg(DISTINCT a)) v FROM jsonb_array_elements(j) a;\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_concat_ignorenull(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT coalesce(a,'[]'::jsonb) || coalesce(b,'[]'::jsonb);\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_least(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(least(nullif_jsonbnullempty(a), nullif_jsonbnullempty(b)));\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_greatest(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(greatest(a, b));\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION first_notnull_sfunc(anyelement, anyelement) RETURNS anyelement AS $$\n SELECT COALESCE($1,$2);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE AGGREGATE first_notnull(anyelement)(\n SFUNC = first_notnull_sfunc,\n STYPE = anyelement\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_array_unique_merge(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_concat_ignorenull,\n FINALFUNC = jsonb_array_unique\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_min(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_least\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_max(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_greatest\n);\n\nCREATE TABLE IF NOT EXISTS migrations (\n version text PRIMARY KEY,\n datetime timestamptz DEFAULT clock_timestamp() NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION get_version() RETURNS text AS $$\n SELECT version FROM pgstac.migrations ORDER BY datetime DESC, version DESC LIMIT 1;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION set_version(text) RETURNS text AS $$\n INSERT INTO pgstac.migrations (version) VALUES ($1)\n ON CONFLICT DO NOTHING\n RETURNING version;\n$$ LANGUAGE SQL;\n\n\nCREATE TABLE IF NOT EXISTS pgstac_settings (\n name text PRIMARY KEY,\n value text NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION table_empty(text) RETURNS boolean AS $$\nDECLARE\n retval boolean;\nBEGIN\n EXECUTE format($q$\n SELECT NOT EXISTS (SELECT 1 FROM %I LIMIT 1)\n $q$,\n $1\n ) INTO retval;\n RETURN retval;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION get_setting(IN _setting text, IN conf jsonb DEFAULT NULL) RETURNS text AS $$\nSELECT COALESCE(\n nullif(conf->>_setting, ''),\n nullif(current_setting(concat('pgstac.',_setting), TRUE),''),\n nullif((SELECT value FROM pgstac.pgstac_settings WHERE name=_setting),'')\n);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_setting_bool(IN _setting text, IN conf jsonb DEFAULT NULL) RETURNS boolean AS $$\nSELECT COALESCE(\n conf->>_setting,\n current_setting(concat('pgstac.',_setting), TRUE),\n (SELECT value FROM pgstac.pgstac_settings WHERE name=_setting),\n 'FALSE'\n)::boolean;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context(conf jsonb DEFAULT NULL) RETURNS text AS $$\n SELECT pgstac.get_setting('context', conf);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context_estimated_count(conf jsonb DEFAULT NULL) RETURNS int AS $$\n SELECT pgstac.get_setting('context_estimated_count', conf)::int;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_estimated_cost();\nCREATE OR REPLACE FUNCTION context_estimated_cost(conf jsonb DEFAULT NULL) RETURNS float AS $$\n SELECT pgstac.get_setting('context_estimated_cost', conf)::float;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_stats_ttl();\nCREATE OR REPLACE FUNCTION context_stats_ttl(conf jsonb DEFAULT NULL) RETURNS interval AS $$\n SELECT pgstac.get_setting('context_stats_ttl', conf)::interval;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION t2s(text) RETURNS text AS $$\n SELECT extract(epoch FROM $1::interval)::text || ' s';\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE STRICT;\n\nCREATE OR REPLACE FUNCTION queue_timeout() RETURNS interval AS $$\n SELECT t2s(coalesce(\n get_setting('queue_timeout'),\n '1h'\n ))::interval;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION notice(VARIADIC text[]) RETURNS boolean AS $$\nDECLARE\ndebug boolean := current_setting('pgstac.debug', true);\nBEGIN\n IF debug THEN\n RAISE NOTICE 'NOTICE FROM FUNC: % >>>>> %', concat_ws(' | ', $1), clock_timestamp();\n RETURN TRUE;\n END IF;\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION empty_arr(ANYARRAY) RETURNS BOOLEAN AS $$\nSELECT CASE\n WHEN $1 IS NULL THEN TRUE\n WHEN cardinality($1)<1 THEN TRUE\nELSE FALSE\nEND;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_intersection(_a ANYARRAY, _b ANYARRAY) RETURNS ANYARRAY AS $$\n SELECT ARRAY ( SELECT unnest(_a) INTERSECT SELECT UNNEST(_b) );\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION array_map_ident(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_ident(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_map_literal(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_literal(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION array_reverse(anyarray) RETURNS anyarray AS $$\nSELECT ARRAY(\n SELECT $1[i]\n FROM generate_subscripts($1,1) AS s(i)\n ORDER BY i DESC\n);\n$$ LANGUAGE SQL STRICT IMMUTABLE;\n\nDROP TABLE IF EXISTS query_queue;\nCREATE TABLE query_queue (\n query text PRIMARY KEY,\n added timestamptz DEFAULT now()\n);\n\nDROP TABLE IF EXISTS query_queue_history;\nCREATE TABLE query_queue_history(\n query text,\n added timestamptz NOT NULL,\n finished timestamptz NOT NULL DEFAULT now(),\n error text\n);\n\nCREATE OR REPLACE PROCEDURE run_queued_queries() AS $$\nDECLARE\n qitem query_queue%ROWTYPE;\n timeout_ts timestamptz;\n error text;\n cnt int := 0;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n DELETE FROM query_queue WHERE query = (SELECT query FROM query_queue ORDER BY added DESC LIMIT 1 FOR UPDATE SKIP LOCKED) RETURNING * INTO qitem;\n IF NOT FOUND THEN\n EXIT;\n END IF;\n cnt := cnt + 1;\n BEGIN\n RAISE NOTICE 'RUNNING QUERY: %', qitem.query;\n EXECUTE qitem.query;\n EXCEPTION WHEN others THEN\n error := format('%s | %s', SQLERRM, SQLSTATE);\n END;\n INSERT INTO query_queue_history (query, added, finished, error)\n VALUES (qitem.query, qitem.added, clock_timestamp(), error);\n COMMIT;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION run_queued_queries_intransaction() RETURNS int AS $$\nDECLARE\n qitem query_queue%ROWTYPE;\n timeout_ts timestamptz;\n error text;\n cnt int := 0;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n DELETE FROM query_queue WHERE query = (SELECT query FROM query_queue ORDER BY added DESC LIMIT 1 FOR UPDATE SKIP LOCKED) RETURNING * INTO qitem;\n IF NOT FOUND THEN\n RETURN cnt;\n END IF;\n cnt := cnt + 1;\n BEGIN\n qitem.query := regexp_replace(qitem.query, 'CONCURRENTLY', '');\n RAISE NOTICE 'RUNNING QUERY: %', qitem.query;\n\n EXECUTE qitem.query;\n EXCEPTION WHEN others THEN\n error := format('%s | %s', SQLERRM, SQLSTATE);\n RAISE WARNING '%', error;\n END;\n INSERT INTO query_queue_history (query, added, finished, error)\n VALUES (qitem.query, qitem.added, clock_timestamp(), error);\n END LOOP;\n RETURN cnt;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION run_or_queue(query text) RETURNS VOID AS $$\nDECLARE\n use_queue text := COALESCE(get_setting('use_queue'), 'FALSE')::boolean;\nBEGIN\n IF get_setting_bool('debug') THEN\n RAISE NOTICE '%', query;\n END IF;\n IF use_queue THEN\n INSERT INTO query_queue (query) VALUES (query) ON CONFLICT DO NOTHING;\n ELSE\n EXECUTE query;\n END IF;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n\nDROP FUNCTION IF EXISTS check_pgstac_settings;\nCREATE OR REPLACE FUNCTION check_pgstac_settings(_sysmem text DEFAULT NULL) RETURNS VOID AS $$\nDECLARE\n settingval text;\n sysmem bigint := pg_size_bytes(_sysmem);\n effective_cache_size bigint := pg_size_bytes(current_setting('effective_cache_size', TRUE));\n shared_buffers bigint := pg_size_bytes(current_setting('shared_buffers', TRUE));\n work_mem bigint := pg_size_bytes(current_setting('work_mem', TRUE));\n max_connections int := current_setting('max_connections', TRUE);\n maintenance_work_mem bigint := pg_size_bytes(current_setting('maintenance_work_mem', TRUE));\n seq_page_cost float := current_setting('seq_page_cost', TRUE);\n random_page_cost float := current_setting('random_page_cost', TRUE);\n temp_buffers bigint := pg_size_bytes(current_setting('temp_buffers', TRUE));\n r record;\nBEGIN\n IF _sysmem IS NULL THEN\n RAISE NOTICE 'Call function with the size of your system memory `SELECT check_pgstac_settings(''4GB'')` to get pg system setting recommendations.';\n ELSE\n IF effective_cache_size < (sysmem * 0.5) THEN\n RAISE WARNING 'effective_cache_size of % is set low for a system with %. Recomended value between % and %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.5), pg_size_pretty(sysmem * 0.75);\n ELSIF effective_cache_size > (sysmem * 0.75) THEN\n RAISE WARNING 'effective_cache_size of % is set high for a system with %. Recomended value between % and %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.5), pg_size_pretty(sysmem * 0.75);\n ELSE\n RAISE NOTICE 'effective_cache_size of % is set appropriately for a system with %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem);\n END IF;\n\n IF shared_buffers < (sysmem * 0.2) THEN\n RAISE WARNING 'shared_buffers of % is set low for a system with %. Recomended value between % and %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.2), pg_size_pretty(sysmem * 0.3);\n ELSIF shared_buffers > (sysmem * 0.3) THEN\n RAISE WARNING 'shared_buffers of % is set high for a system with %. Recomended value between % and %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.2), pg_size_pretty(sysmem * 0.3);\n ELSE\n RAISE NOTICE 'shared_buffers of % is set appropriately for a system with %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem);\n END IF;\n shared_buffers = sysmem * 0.3;\n IF maintenance_work_mem < (sysmem * 0.2) THEN\n RAISE WARNING 'maintenance_work_mem of % is set low for shared_buffers of %. Recomended value between % and %', pg_size_pretty(maintenance_work_mem), pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers * 0.2), pg_size_pretty(shared_buffers * 0.3);\n ELSIF maintenance_work_mem > (shared_buffers * 0.3) THEN\n RAISE WARNING 'maintenance_work_mem of % is set high for shared_buffers of %. Recomended value between % and %', pg_size_pretty(maintenance_work_mem), pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers * 0.2), pg_size_pretty(shared_buffers * 0.3);\n ELSE\n RAISE NOTICE 'maintenance_work_mem of % is set appropriately for shared_buffers of %', pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers);\n END IF;\n\n IF work_mem * max_connections > shared_buffers THEN\n RAISE WARNING 'work_mem setting of % is set high for % max_connections please reduce work_mem to % or decrease max_connections to %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers/max_connections), floor(shared_buffers/work_mem);\n ELSIF work_mem * max_connections < (shared_buffers * 0.75) THEN\n RAISE WARNING 'work_mem setting of % is set low for % max_connections you may consider raising work_mem to % or increasing max_connections to %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers/max_connections), floor(shared_buffers/work_mem);\n ELSE\n RAISE NOTICE 'work_mem setting of % and max_connections of % are adequate for shared_buffers of %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers);\n END IF;\n\n IF random_page_cost / seq_page_cost != 1.1 THEN\n RAISE WARNING 'random_page_cost (%) /seq_page_cost (%) should be set to 1.1 for SSD. Change random_page_cost to %', random_page_cost, seq_page_cost, 1.1 * seq_page_cost;\n ELSE\n RAISE NOTICE 'random_page_cost and seq_page_cost set appropriately for SSD';\n END IF;\n\n IF temp_buffers < greatest(pg_size_bytes('128MB'),(maintenance_work_mem / 2)) THEN\n RAISE WARNING 'pgstac makes heavy use of temp tables, consider raising temp_buffers from % to %', pg_size_pretty(temp_buffers), greatest('128MB', pg_size_pretty((shared_buffers / 16)));\n END IF;\n END IF;\n\n RAISE NOTICE 'VALUES FOR PGSTAC VARIABLES';\n RAISE NOTICE 'These can be set either as GUC system variables or by setting in the pgstac_settings table.';\n\n FOR r IN SELECT name, get_setting(name) as setting, CASE WHEN current_setting(concat('pgstac.',name), TRUE) IS NOT NULL THEN concat('pgstac.',name, ' GUC') WHEN value IS NOT NULL THEN 'pgstac_settings table' ELSE 'Not Set' END as loc FROM pgstac_settings LOOP\n RAISE NOTICE '% is set to % from the %', r.name, r.setting, r.loc;\n END LOOP;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pg_cron';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider intalling pg_cron which can be used to automate tasks';\n ELSE\n RAISE NOTICE 'pg_cron % is installed', settingval;\n END IF;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pgstattuple';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider installing the pgstattuple extension which can be used to help maintain tables and indexes.';\n ELSE\n RAISE NOTICE 'pgstattuple % is installed', settingval;\n END IF;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pg_stat_statements';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider installing the pg_stat_statements extension which is very helpful for tracking the types of queries on the system';\n ELSE\n RAISE NOTICE 'pg_stat_statements % is installed', settingval;\n IF current_setting('pg_stat_statements.track_statements', TRUE) IS DISTINCT FROM 'all' THEN\n RAISE WARNING 'SET pg_stat_statements.track_statements TO ''all''; --In order to track statements within functions.';\n END IF;\n END IF;\n\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac, public SET CLIENT_MIN_MESSAGES TO NOTICE;\n/* looks for a geometry in a stac item first from geometry and falling back to bbox */\nCREATE OR REPLACE FUNCTION stac_geom(value jsonb) RETURNS geometry AS $$\nSELECT\n CASE\n WHEN value ? 'intersects' THEN\n ST_GeomFromGeoJSON(value->>'intersects')\n WHEN value ? 'geometry' THEN\n ST_GeomFromGeoJSON(value->>'geometry')\n WHEN value ? 'bbox' THEN\n pgstac.bbox_geom(value->'bbox')\n ELSE NULL\n END as geometry\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION stac_daterange(\n value jsonb\n) RETURNS tstzrange AS $$\nDECLARE\n props jsonb := value;\n dt timestamptz;\n edt timestamptz;\nBEGIN\n IF props ? 'properties' THEN\n props := props->'properties';\n END IF;\n IF\n props ? 'start_datetime'\n AND props->>'start_datetime' IS NOT NULL\n AND props ? 'end_datetime'\n AND props->>'end_datetime' IS NOT NULL\n THEN\n dt := props->>'start_datetime';\n edt := props->>'end_datetime';\n IF dt > edt THEN\n RAISE EXCEPTION 'start_datetime must be < end_datetime';\n END IF;\n ELSE\n dt := props->>'datetime';\n edt := props->>'datetime';\n END IF;\n IF dt is NULL OR edt IS NULL THEN\n RAISE NOTICE 'DT: %, EDT: %', dt, edt;\n RAISE EXCEPTION 'Either datetime (%) or both start_datetime (%) and end_datetime (%) must be set.', props->>'datetime',props->>'start_datetime',props->>'end_datetime';\n END IF;\n RETURN tstzrange(dt, edt, '[]');\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_datetime(value jsonb) RETURNS timestamptz AS $$\n SELECT lower(stac_daterange(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_end_datetime(value jsonb) RETURNS timestamptz AS $$\n SELECT upper(stac_daterange(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE TABLE IF NOT EXISTS stac_extensions(\n url text PRIMARY KEY,\n content jsonb\n);\nCREATE OR REPLACE FUNCTION queryable_signature(n text, c text[]) RETURNS text AS $$\n SELECT concat(n, c);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TABLE queryables (\n id bigint GENERATED ALWAYS AS identity PRIMARY KEY,\n name text NOT NULL,\n collection_ids text[], -- used to determine what partitions to create indexes on\n definition jsonb,\n property_path text,\n property_wrapper text,\n property_index_type text\n);\nCREATE INDEX queryables_name_idx ON queryables (name);\nCREATE INDEX queryables_collection_idx ON queryables USING GIN (collection_ids);\nCREATE INDEX queryables_property_wrapper_idx ON queryables (property_wrapper);\n\nCREATE OR REPLACE FUNCTION queryables_constraint_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n allcollections text[];\nBEGIN\n RAISE NOTICE 'Making sure that name/collection is unique for queryables';\n IF NEW.collection_ids IS NOT NULL THEN\n IF EXISTS (\n SELECT 1 FROM\n collections\n LEFT JOIN\n unnest(NEW.collection_ids) c\n ON (collections.id = c)\n WHERE c IS NULL\n ) THEN\n RAISE foreign_key_violation;\n RETURN NULL;\n END IF;\n END IF;\n IF TG_OP = 'INSERT' THEN\n IF EXISTS (\n SELECT 1 FROM queryables q\n WHERE\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n )\n ) THEN\n RAISE unique_violation;\n RETURN NULL;\n END IF;\n END IF;\n IF TG_OP = 'UPDATE' THEN\n IF EXISTS (\n SELECT 1 FROM queryables q\n WHERE\n q.id != NEW.id\n AND\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n )\n ) THEN\n RAISE unique_violation\n USING MESSAGE = format(\n 'There is already a queryable for %s for a collection in %s',\n NEW.name,\n NEW.collection_ids\n );\n RETURN NULL;\n END IF;\n END IF;\n\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER queryables_constraint_insert_trigger\nBEFORE INSERT ON queryables\nFOR EACH ROW EXECUTE PROCEDURE queryables_constraint_triggerfunc();\n\nCREATE TRIGGER queryables_constraint_update_trigger\nBEFORE UPDATE ON queryables\nFOR EACH ROW\nWHEN (NEW.name = OLD.name AND NEW.collection_ids IS DISTINCT FROM OLD.collection_ids)\nEXECUTE PROCEDURE queryables_constraint_triggerfunc();\n\n\nCREATE OR REPLACE FUNCTION array_to_path(arr text[]) RETURNS text AS $$\n SELECT string_agg(\n quote_literal(v),\n '->'\n ) FROM unnest(arr) v;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\n\n\nCREATE OR REPLACE FUNCTION queryable(\n IN dotpath text,\n OUT path text,\n OUT expression text,\n OUT wrapper text,\n OUT nulled_wrapper text\n) AS $$\nDECLARE\n q RECORD;\n path_elements text[];\nBEGIN\n IF dotpath IN ('id', 'geometry', 'datetime', 'end_datetime', 'collection') THEN\n path := dotpath;\n expression := dotpath;\n wrapper := NULL;\n RETURN;\n END IF;\n SELECT * INTO q FROM queryables\n WHERE\n name=dotpath\n OR name = 'properties.' || dotpath\n OR name = replace(dotpath, 'properties.', '')\n ;\n IF q.property_wrapper IS NULL THEN\n IF q.definition->>'type' = 'number' THEN\n wrapper := 'to_float';\n nulled_wrapper := wrapper;\n ELSIF q.definition->>'format' = 'date-time' THEN\n wrapper := 'to_tstz';\n nulled_wrapper := wrapper;\n ELSE\n nulled_wrapper := NULL;\n wrapper := 'to_text';\n END IF;\n ELSE\n wrapper := q.property_wrapper;\n nulled_wrapper := wrapper;\n END IF;\n IF q.property_path IS NOT NULL THEN\n path := q.property_path;\n ELSE\n path_elements := string_to_array(dotpath, '.');\n IF path_elements[1] IN ('links', 'assets', 'stac_version', 'stac_extensions') THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSIF path_elements[1] = 'properties' THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSE\n path := format($F$content->'properties'->%s$F$, array_to_path(path_elements));\n END IF;\n END IF;\n expression := format('%I(%s)', wrapper, path);\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\n\nDROP VIEW IF EXISTS pgstac_index;\nCREATE VIEW pgstac_indexes AS\nSELECT\n i.schemaname,\n i.tablename,\n i.indexname,\n indexdef,\n COALESCE(\n (regexp_match(indexdef, '\\(([a-zA-Z]+)\\)'))[1],\n (regexp_match(indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_]+)''::text'))[1],\n CASE WHEN indexdef ~* '\\(datetime desc, end_datetime\\)' THEN 'datetime_end_datetime' ELSE NULL END\n ) AS field,\n pg_table_size(i.indexname::text) as index_size,\n pg_size_pretty(pg_table_size(i.indexname::text)) as index_size_pretty\nFROM\n pg_indexes i\nWHERE i.schemaname='pgstac' and i.tablename ~ '_items_';\n\nDROP VIEW IF EXISTS pgstac_index_stats;\nCREATE VIEW pgstac_indexes_stats AS\nSELECT\n i.schemaname,\n i.tablename,\n i.indexname,\n indexdef,\n COALESCE(\n (regexp_match(indexdef, '\\(([a-zA-Z]+)\\)'))[1],\n (regexp_match(indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_]+)''::text'))[1],\n CASE WHEN indexdef ~* '\\(datetime desc, end_datetime\\)' THEN 'datetime_end_datetime' ELSE NULL END\n ) AS field,\n pg_table_size(i.indexname::text) as index_size,\n pg_size_pretty(pg_table_size(i.indexname::text)) as index_size_pretty,\n n_distinct,\n most_common_vals::text::text[],\n most_common_freqs::text::text[],\n histogram_bounds::text::text[],\n correlation\nFROM\n pg_indexes i\n LEFT JOIN pg_stats s ON (s.tablename = i.indexname)\nWHERE i.schemaname='pgstac' and i.tablename ~ '_items_';\n\nset check_function_bodies to off;\nCREATE OR REPLACE FUNCTION maintain_partition_queries(\n part text DEFAULT 'items',\n dropindexes boolean DEFAULT FALSE,\n rebuildindexes boolean DEFAULT FALSE,\n idxconcurrently boolean DEFAULT FALSE\n) RETURNS SETOF text AS $$\nDECLARE\n parent text;\n level int;\n isleaf bool;\n collection collections%ROWTYPE;\n subpart text;\n baseidx text;\n queryable_name text;\n queryable_property_index_type text;\n queryable_property_wrapper text;\n queryable_parsed RECORD;\n deletedidx pg_indexes%ROWTYPE;\n q text;\n idx text;\n collection_partition bigint;\n _concurrently text := '';\nBEGIN\n RAISE NOTICE 'Maintaining partition: %', part;\n IF idxconcurrently THEN\n _concurrently='CONCURRENTLY';\n END IF;\n\n -- Get root partition\n SELECT parentrelid::text, pt.isleaf, pt.level\n INTO parent, isleaf, level\n FROM pg_partition_tree('items') pt\n WHERE relid::text = part;\n IF NOT FOUND THEN\n RAISE NOTICE 'Partition % Does Not Exist In Partition Tree', part;\n RETURN;\n END IF;\n\n -- If this is a parent partition, recurse to leaves\n IF NOT isleaf THEN\n FOR subpart IN\n SELECT relid::text\n FROM pg_partition_tree(part)\n WHERE relid::text != part\n LOOP\n RAISE NOTICE 'Recursing to %', subpart;\n RETURN QUERY SELECT * FROM maintain_partition_queries(subpart, dropindexes, rebuildindexes);\n END LOOP;\n RETURN; -- Don't continue since not an end leaf\n END IF;\n\n\n -- Get collection\n collection_partition := ((regexp_match(part, E'^_items_([0-9]+)'))[1])::bigint;\n RAISE NOTICE 'COLLECTION PARTITION: %', collection_partition;\n SELECT * INTO STRICT collection\n FROM collections\n WHERE key = collection_partition;\n RAISE NOTICE 'COLLECTION ID: %s', collection.id;\n\n\n -- Create temp table with existing indexes\n CREATE TEMP TABLE existing_indexes ON COMMIT DROP AS\n SELECT *\n FROM pg_indexes\n WHERE schemaname='pgstac' AND tablename=part;\n\n\n -- Check if index exists for each queryable.\n FOR\n queryable_name,\n queryable_property_index_type,\n queryable_property_wrapper\n IN\n SELECT\n name,\n COALESCE(property_index_type, 'BTREE'),\n COALESCE(property_wrapper, 'to_text')\n FROM queryables\n WHERE\n name NOT in ('id', 'datetime', 'geometry')\n AND (\n collection_ids IS NULL\n OR collection_ids = '{}'::text[]\n OR collection.id = ANY (collection_ids)\n )\n LOOP\n baseidx := format(\n $q$ ON %I USING %s (%s(((content -> 'properties'::text) -> %L::text)))$q$,\n part,\n queryable_property_index_type,\n queryable_property_wrapper,\n queryable_name\n );\n RAISE NOTICE 'BASEIDX: %', baseidx;\n RAISE NOTICE 'IDXSEARCH: %', format($q$[(']%s[')]$q$, queryable_name);\n -- If index already exists, delete it from existing indexes type table\n FOR deletedidx IN\n DELETE FROM existing_indexes\n WHERE indexdef ~* format($q$[(']%s[')]$q$, queryable_name)\n RETURNING *\n LOOP\n RAISE NOTICE 'EXISTING INDEX: %', deletedidx;\n IF NOT FOUND THEN -- index did not exist, create it\n RETURN NEXT format('CREATE INDEX %s %s;', _concurrently, baseidx);\n ELSIF rebuildindexes THEN\n RETURN NEXT format('REINDEX %I %s;', deletedidx.indexname, _concurrently);\n END IF;\n END LOOP;\n IF NOT FOUND THEN\n RAISE NOTICE 'CREATING INDEX for %', queryable_name;\n RETURN NEXT format('CREATE INDEX %s %s;', _concurrently, baseidx);\n END IF;\n END LOOP;\n IF NOT EXISTS (SELECT * FROM pg_indexes WHERE indexname::text = concat(part, '_datetime_end_datetime_idx')) THEN\n RETURN NEXT format(\n $f$CREATE INDEX IF NOT EXISTS %I ON %I USING BTREE (datetime DESC, end_datetime ASC);$f$,\n concat(part, '_datetime_end_datetime_idx'),\n part\n );\n END IF;\n\n IF NOT EXISTS (SELECT * FROM pg_indexes WHERE indexname::text = concat(part, '_geometry_idx')) THEN\n RETURN NEXT format(\n $f$CREATE INDEX IF NOT EXISTS %I ON %I USING GIST (geometry);$f$,\n concat(part, '_geometry_idx'),\n part\n );\n END IF;\n\n IF NOT EXISTS (SELECT * FROM pg_indexes WHERE indexname::text = concat(part, '_pk')) THEN\n RETURN NEXT format(\n $f$CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I USING BTREE(id);$f$,\n concat(part, '_pk'),\n part\n );\n END IF;\n DELETE FROM existing_indexes WHERE indexname::text IN (\n concat(part, '_datetime_end_datetime_idx'),\n concat(part, '_geometry_idx'),\n concat(part, '_pk')\n );\n -- Remove indexes that were not expected\n FOR idx IN SELECT indexname::text FROM existing_indexes\n LOOP\n RAISE WARNING 'Index: % is not defined by queryables.', idx;\n IF dropindexes THEN\n RETURN NEXT format('DROP INDEX IF EXISTS %I;', idx);\n END IF;\n END LOOP;\n\n DROP TABLE existing_indexes;\n RAISE NOTICE 'Returning from maintain_partition_queries.';\n RETURN;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION maintain_partitions(\n part text DEFAULT 'items',\n dropindexes boolean DEFAULT FALSE,\n rebuildindexes boolean DEFAULT FALSE\n) RETURNS VOID AS $$\n WITH t AS (\n SELECT run_or_queue(q) FROM maintain_partition_queries(part, dropindexes, rebuildindexes) q\n ) SELECT count(*) FROM t;\n$$ LANGUAGE SQL;\n\n\nCREATE OR REPLACE FUNCTION queryables_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\nBEGIN\n PERFORM maintain_partitions();\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER queryables_trigger AFTER INSERT OR UPDATE ON queryables\nFOR EACH STATEMENT EXECUTE PROCEDURE queryables_trigger_func();\n\n\nCREATE OR REPLACE FUNCTION get_queryables(_collection_ids text[] DEFAULT NULL) RETURNS jsonb AS $$\nDECLARE\nBEGIN\n -- Build up queryables if the input contains valid collection ids or is empty\n IF EXISTS (\n SELECT 1 FROM collections\n WHERE\n _collection_ids IS NULL\n OR cardinality(_collection_ids) = 0\n OR id = ANY(_collection_ids)\n )\n THEN\n RETURN (\n WITH base AS (\n SELECT\n unnest(collection_ids) as collection_id,\n name,\n coalesce(definition, '{\"type\":\"string\"}'::jsonb) as definition\n FROM queryables\n WHERE\n _collection_ids IS NULL OR\n _collection_ids = '{}'::text[] OR\n _collection_ids && collection_ids\n UNION ALL\n SELECT null, name, coalesce(definition, '{\"type\":\"string\"}'::jsonb) as definition\n FROM queryables WHERE collection_ids IS NULL OR collection_ids = '{}'::text[]\n ), g AS (\n SELECT\n name,\n first_notnull(definition) as definition,\n jsonb_array_unique_merge(definition->'enum') as enum,\n jsonb_min(definition->'minimum') as minimum,\n jsonb_min(definition->'maxiumn') as maximum\n FROM base\n GROUP BY 1\n )\n SELECT\n jsonb_build_object(\n '$schema', 'http://json-schema.org/draft-07/schema#',\n '$id', '',\n 'type', 'object',\n 'title', 'STAC Queryables.',\n 'properties', jsonb_object_agg(\n name,\n definition\n ||\n jsonb_strip_nulls(jsonb_build_object(\n 'enum', enum,\n 'minimum', minimum,\n 'maximum', maximum\n ))\n )\n )\n FROM g\n );\n ELSE\n RETURN NULL;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\nCREATE OR REPLACE FUNCTION get_queryables(_collection text DEFAULT NULL) RETURNS jsonb AS $$\n SELECT\n CASE\n WHEN _collection IS NULL THEN get_queryables(NULL::text[])\n ELSE get_queryables(ARRAY[_collection])\n END\n ;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_queryables() RETURNS jsonb AS $$\n SELECT get_queryables(NULL::text[]);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION schema_qualify_refs(url text, j jsonb) returns jsonb as $$\n SELECT regexp_replace(j::text, '\"\\$ref\": \"#', concat('\"$ref\": \"', url, '#'), 'g')::jsonb;\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE VIEW stac_extension_queryables AS\nSELECT DISTINCT key as name, schema_qualify_refs(e.url, j.value) as definition FROM stac_extensions e, jsonb_each(e.content->'definitions'->'fields'->'properties') j;\n\n\nCREATE OR REPLACE FUNCTION missing_queryables(_collection text, _tablesample float DEFAULT 5, minrows float DEFAULT 10) RETURNS TABLE(collection text, name text, definition jsonb, property_wrapper text) AS $$\nDECLARE\n q text;\n _partition text;\n explain_json json;\n psize float;\n estrows float;\nBEGIN\n SELECT format('_items_%s', key) INTO _partition FROM collections WHERE id=_collection;\n\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM %I;', _partition)\n INTO explain_json;\n psize := explain_json->0->'Plan'->'Plan Rows';\n estrows := _tablesample * .01 * psize;\n IF estrows < minrows THEN\n _tablesample := least(100,greatest(_tablesample, (estrows / psize) / 100));\n RAISE NOTICE '%', (psize / estrows) / 100;\n END IF;\n RAISE NOTICE 'Using tablesample % to find missing queryables from % % that has ~% rows estrows: %', _tablesample, _collection, _partition, psize, estrows;\n\n q := format(\n $q$\n WITH q AS (\n SELECT * FROM queryables\n WHERE\n collection_ids IS NULL\n OR %L = ANY(collection_ids)\n ), t AS (\n SELECT\n content->'properties' AS properties\n FROM\n %I\n TABLESAMPLE SYSTEM(%L)\n ), p AS (\n SELECT DISTINCT ON (key)\n key,\n value,\n s.definition\n FROM t\n JOIN LATERAL jsonb_each(properties) ON TRUE\n LEFT JOIN q ON (q.name=key)\n LEFT JOIN stac_extension_queryables s ON (s.name=key)\n WHERE q.definition IS NULL\n )\n SELECT\n %L,\n key,\n COALESCE(definition, jsonb_build_object('type',jsonb_typeof(value))) as definition,\n CASE\n WHEN definition->>'type' = 'integer' THEN 'to_int'\n WHEN COALESCE(definition->>'type', jsonb_typeof(value)) = 'number' THEN 'to_float'\n WHEN COALESCE(definition->>'type', jsonb_typeof(value)) = 'array' THEN 'to_text_array'\n ELSE 'to_text'\n END\n FROM p;\n $q$,\n _collection,\n _partition,\n _tablesample,\n _collection\n );\n RETURN QUERY EXECUTE q;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION missing_queryables(_tablesample float DEFAULT 5) RETURNS TABLE(collection_ids text[], name text, definition jsonb, property_wrapper text) AS $$\n SELECT\n array_agg(collection),\n name,\n definition,\n property_wrapper\n FROM\n collections\n JOIN LATERAL\n missing_queryables(id, _tablesample) c\n ON TRUE\n GROUP BY\n 2,3,4\n ORDER BY 2,1\n ;\n$$ LANGUAGE SQL;\nCREATE OR REPLACE FUNCTION parse_dtrange(\n _indate jsonb,\n relative_base timestamptz DEFAULT date_trunc('hour', CURRENT_TIMESTAMP)\n) RETURNS tstzrange AS $$\nDECLARE\n timestrs text[];\n s timestamptz;\n e timestamptz;\nBEGIN\n timestrs :=\n CASE\n WHEN _indate ? 'timestamp' THEN\n ARRAY[_indate->>'timestamp']\n WHEN _indate ? 'interval' THEN\n to_text_array(_indate->'interval')\n WHEN jsonb_typeof(_indate) = 'array' THEN\n to_text_array(_indate)\n ELSE\n regexp_split_to_array(\n _indate->>0,\n '/'\n )\n END;\n RAISE NOTICE 'TIMESTRS %', timestrs;\n IF cardinality(timestrs) = 1 THEN\n IF timestrs[1] ILIKE 'P%' THEN\n RETURN tstzrange(relative_base - upper(timestrs[1])::interval, relative_base, '[)');\n END IF;\n s := timestrs[1]::timestamptz;\n RETURN tstzrange(s, s, '[]');\n END IF;\n\n IF cardinality(timestrs) != 2 THEN\n RAISE EXCEPTION 'Timestamp cannot have more than 2 values';\n END IF;\n\n IF timestrs[1] = '..' OR timestrs[1] = '' THEN\n s := '-infinity'::timestamptz;\n e := timestrs[2]::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] = '..' OR timestrs[2] = '' THEN\n s := timestrs[1]::timestamptz;\n e := 'infinity'::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[1] ILIKE 'P%' AND timestrs[2] NOT ILIKE 'P%' THEN\n e := timestrs[2]::timestamptz;\n s := e - upper(timestrs[1])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] ILIKE 'P%' AND timestrs[1] NOT ILIKE 'P%' THEN\n s := timestrs[1]::timestamptz;\n e := s + upper(timestrs[2])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n s := timestrs[1]::timestamptz;\n e := timestrs[2]::timestamptz;\n\n RETURN tstzrange(s,e,'[)');\n\n RETURN NULL;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT PARALLEL SAFE SET TIME ZONE 'UTC';\n\nCREATE OR REPLACE FUNCTION parse_dtrange(\n _indate text,\n relative_base timestamptz DEFAULT CURRENT_TIMESTAMP\n) RETURNS tstzrange AS $$\n SELECT parse_dtrange(to_jsonb(_indate), relative_base);\n$$ LANGUAGE SQL STABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION temporal_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\n ll text := 'datetime';\n lh text := 'end_datetime';\n rrange tstzrange;\n rl text;\n rh text;\n outq text;\nBEGIN\n rrange := parse_dtrange(args->1);\n RAISE NOTICE 'Constructing temporal query OP: %, ARGS: %, RRANGE: %', op, args, rrange;\n op := lower(op);\n rl := format('%L::timestamptz', lower(rrange));\n rh := format('%L::timestamptz', upper(rrange));\n outq := CASE op\n WHEN 't_before' THEN 'lh < rl'\n WHEN 't_after' THEN 'll > rh'\n WHEN 't_meets' THEN 'lh = rl'\n WHEN 't_metby' THEN 'll = rh'\n WHEN 't_overlaps' THEN 'll < rl AND rl < lh < rh'\n WHEN 't_overlappedby' THEN 'rl < ll < rh AND lh > rh'\n WHEN 't_starts' THEN 'll = rl AND lh < rh'\n WHEN 't_startedby' THEN 'll = rl AND lh > rh'\n WHEN 't_during' THEN 'll > rl AND lh < rh'\n WHEN 't_contains' THEN 'll < rl AND lh > rh'\n WHEN 't_finishes' THEN 'll > rl AND lh = rh'\n WHEN 't_finishedby' THEN 'll < rl AND lh = rh'\n WHEN 't_equals' THEN 'll = rl AND lh = rh'\n WHEN 't_disjoint' THEN 'NOT (ll <= rh AND lh >= rl)'\n WHEN 't_intersects' THEN 'll <= rh AND lh >= rl'\n WHEN 'anyinteracts' THEN 'll <= rh AND lh >= rl'\n END;\n outq := regexp_replace(outq, '\\mll\\M', ll);\n outq := regexp_replace(outq, '\\mlh\\M', lh);\n outq := regexp_replace(outq, '\\mrl\\M', rl);\n outq := regexp_replace(outq, '\\mrh\\M', rh);\n outq := format('(%s)', outq);\n RETURN outq;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\n\n\nCREATE OR REPLACE FUNCTION spatial_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\n geom text;\n j jsonb := args->1;\nBEGIN\n op := lower(op);\n RAISE NOTICE 'Constructing spatial query OP: %, ARGS: %', op, args;\n IF op NOT IN ('s_equals','s_disjoint','s_touches','s_within','s_overlaps','s_crosses','s_intersects','intersects','s_contains') THEN\n RAISE EXCEPTION 'Spatial Operator % Not Supported', op;\n END IF;\n op := regexp_replace(op, '^s_', 'st_');\n IF op = 'intersects' THEN\n op := 'st_intersects';\n END IF;\n -- Convert geometry to WKB string\n IF j ? 'type' AND j ? 'coordinates' THEN\n geom := st_geomfromgeojson(j)::text;\n ELSIF jsonb_typeof(j) = 'array' THEN\n geom := bbox_geom(j)::text;\n END IF;\n\n RETURN format('%s(geometry, %L::geometry)', op, geom);\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION query_to_cql2(q jsonb) RETURNS jsonb AS $$\n-- Translates anything passed in through the deprecated \"query\" into equivalent CQL2\nWITH t AS (\n SELECT key as property, value as ops\n FROM jsonb_each(q)\n), t2 AS (\n SELECT property, (jsonb_each(ops)).*\n FROM t WHERE jsonb_typeof(ops) = 'object'\n UNION ALL\n SELECT property, 'eq', ops\n FROM t WHERE jsonb_typeof(ops) != 'object'\n)\nSELECT\n jsonb_strip_nulls(jsonb_build_object(\n 'op', 'and',\n 'args', jsonb_agg(\n jsonb_build_object(\n 'op', key,\n 'args', jsonb_build_array(\n jsonb_build_object('property',property),\n value\n )\n )\n )\n )\n) as qcql FROM t2\n;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\nCREATE OR REPLACE FUNCTION cql1_to_cql2(j jsonb) RETURNS jsonb AS $$\nDECLARE\n args jsonb;\n ret jsonb;\nBEGIN\n RAISE NOTICE 'CQL1_TO_CQL2: %', j;\n IF j ? 'filter' THEN\n RETURN cql1_to_cql2(j->'filter');\n END IF;\n IF j ? 'property' THEN\n RETURN j;\n END IF;\n IF jsonb_typeof(j) = 'array' THEN\n SELECT jsonb_agg(cql1_to_cql2(el)) INTO args FROM jsonb_array_elements(j) el;\n RETURN args;\n END IF;\n IF jsonb_typeof(j) = 'number' THEN\n RETURN j;\n END IF;\n IF jsonb_typeof(j) = 'string' THEN\n RETURN j;\n END IF;\n\n IF jsonb_typeof(j) = 'object' THEN\n SELECT jsonb_build_object(\n 'op', key,\n 'args', cql1_to_cql2(value)\n ) INTO ret\n FROM jsonb_each(j)\n WHERE j IS NOT NULL;\n RETURN ret;\n END IF;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE STRICT;\n\nCREATE TABLE cql2_ops (\n op text PRIMARY KEY,\n template text,\n types text[]\n);\nINSERT INTO cql2_ops (op, template, types) VALUES\n ('eq', '%s = %s', NULL),\n ('neq', '%s != %s', NULL),\n ('ne', '%s != %s', NULL),\n ('!=', '%s != %s', NULL),\n ('<>', '%s != %s', NULL),\n ('lt', '%s < %s', NULL),\n ('lte', '%s <= %s', NULL),\n ('gt', '%s > %s', NULL),\n ('gte', '%s >= %s', NULL),\n ('le', '%s <= %s', NULL),\n ('ge', '%s >= %s', NULL),\n ('=', '%s = %s', NULL),\n ('<', '%s < %s', NULL),\n ('<=', '%s <= %s', NULL),\n ('>', '%s > %s', NULL),\n ('>=', '%s >= %s', NULL),\n ('like', '%s LIKE %s', NULL),\n ('ilike', '%s ILIKE %s', NULL),\n ('+', '%s + %s', NULL),\n ('-', '%s - %s', NULL),\n ('*', '%s * %s', NULL),\n ('/', '%s / %s', NULL),\n ('not', 'NOT (%s)', NULL),\n ('between', '%s BETWEEN %s AND %s', NULL),\n ('isnull', '%s IS NULL', NULL),\n ('upper', 'upper(%s)', NULL),\n ('lower', 'lower(%s)', NULL)\nON CONFLICT (op) DO UPDATE\n SET\n template = EXCLUDED.template\n;\n\n\nCREATE OR REPLACE FUNCTION cql2_query(j jsonb, wrapper text DEFAULT NULL) RETURNS text AS $$\n#variable_conflict use_variable\nDECLARE\n args jsonb := j->'args';\n arg jsonb;\n op text := lower(j->>'op');\n cql2op RECORD;\n literal text;\n _wrapper text;\n leftarg text;\n rightarg text;\nBEGIN\n IF j IS NULL OR (op IS NOT NULL AND args IS NULL) THEN\n RETURN NULL;\n END IF;\n RAISE NOTICE 'CQL2_QUERY: %', j;\n IF j ? 'filter' THEN\n RETURN cql2_query(j->'filter');\n END IF;\n\n IF j ? 'upper' THEN\n RETURN cql2_query(jsonb_build_object('op', 'upper', 'args', j->'upper'));\n END IF;\n\n IF j ? 'lower' THEN\n RETURN cql2_query(jsonb_build_object('op', 'lower', 'args', j->'lower'));\n END IF;\n\n -- Temporal Query\n IF op ilike 't_%' or op = 'anyinteracts' THEN\n RETURN temporal_op_query(op, args);\n END IF;\n\n -- If property is a timestamp convert it to text to use with\n -- general operators\n IF j ? 'timestamp' THEN\n RETURN format('%L::timestamptz', to_tstz(j->'timestamp'));\n END IF;\n IF j ? 'interval' THEN\n RAISE EXCEPTION 'Please use temporal operators when using intervals.';\n RETURN NONE;\n END IF;\n\n -- Spatial Query\n IF op ilike 's_%' or op = 'intersects' THEN\n RETURN spatial_op_query(op, args);\n END IF;\n\n IF op IN ('a_equals','a_contains','a_contained_by','a_overlaps') THEN\n IF args->0 ? 'property' THEN\n leftarg := format('to_text_array(%s)', (queryable(args->0->>'property')).path);\n END IF;\n IF args->1 ? 'property' THEN\n rightarg := format('to_text_array(%s)', (queryable(args->1->>'property')).path);\n END IF;\n RETURN FORMAT(\n '%s %s %s',\n COALESCE(leftarg, quote_literal(to_text_array(args->0))),\n CASE op\n WHEN 'a_equals' THEN '='\n WHEN 'a_contains' THEN '@>'\n WHEN 'a_contained_by' THEN '<@'\n WHEN 'a_overlaps' THEN '&&'\n END,\n COALESCE(rightarg, quote_literal(to_text_array(args->1)))\n );\n END IF;\n\n IF op = 'in' THEN\n RAISE NOTICE 'IN : % % %', args, jsonb_build_array(args->0), args->1;\n args := jsonb_build_array(args->0) || (args->1);\n RAISE NOTICE 'IN2 : %', args;\n END IF;\n\n\n\n IF op = 'between' THEN\n args = jsonb_build_array(\n args->0,\n args->1->0,\n args->1->1\n );\n END IF;\n\n -- Make sure that args is an array and run cql2_query on\n -- each element of the array\n RAISE NOTICE 'ARGS PRE: %', args;\n IF j ? 'args' THEN\n IF jsonb_typeof(args) != 'array' THEN\n args := jsonb_build_array(args);\n END IF;\n\n IF jsonb_path_exists(args, '$[*] ? (@.property == \"id\" || @.property == \"datetime\" || @.property == \"end_datetime\" || @.property == \"collection\")') THEN\n wrapper := NULL;\n ELSE\n -- if any of the arguments are a property, try to get the property_wrapper\n FOR arg IN SELECT jsonb_path_query(args, '$[*] ? (@.property != null)') LOOP\n RAISE NOTICE 'Arg: %', arg;\n wrapper := (queryable(arg->>'property')).nulled_wrapper;\n RAISE NOTICE 'Property: %, Wrapper: %', arg, wrapper;\n IF wrapper IS NOT NULL THEN\n EXIT;\n END IF;\n END LOOP;\n\n -- if the property was not in queryables, see if any args were numbers\n IF\n wrapper IS NULL\n AND jsonb_path_exists(args, '$[*] ? (@.type()==\"number\")')\n THEN\n wrapper := 'to_float';\n END IF;\n wrapper := coalesce(wrapper, 'to_text');\n END IF;\n\n SELECT jsonb_agg(cql2_query(a, wrapper))\n INTO args\n FROM jsonb_array_elements(args) a;\n END IF;\n RAISE NOTICE 'ARGS: %', args;\n\n IF op IN ('and', 'or') THEN\n RETURN\n format(\n '(%s)',\n array_to_string(to_text_array(args), format(' %s ', upper(op)))\n );\n END IF;\n\n IF op = 'in' THEN\n RAISE NOTICE 'IN -- % %', args->0, to_text(args->0);\n RETURN format(\n '%s IN (%s)',\n to_text(args->0),\n array_to_string((to_text_array(args))[2:], ',')\n );\n END IF;\n\n -- Look up template from cql2_ops\n IF j ? 'op' THEN\n SELECT * INTO cql2op FROM cql2_ops WHERE cql2_ops.op ilike op;\n IF FOUND THEN\n -- If specific index set in queryables for a property cast other arguments to that type\n\n RETURN format(\n cql2op.template,\n VARIADIC (to_text_array(args))\n );\n ELSE\n RAISE EXCEPTION 'Operator % Not Supported.', op;\n END IF;\n END IF;\n\n\n IF wrapper IS NOT NULL THEN\n RAISE NOTICE 'Wrapping % with %', j, wrapper;\n IF j ? 'property' THEN\n RETURN format('%I(%s)', wrapper, (queryable(j->>'property')).path);\n ELSE\n RETURN format('%I(%L)', wrapper, j);\n END IF;\n ELSIF j ? 'property' THEN\n RETURN quote_ident(j->>'property');\n END IF;\n\n RETURN quote_literal(to_text(j));\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION paging_dtrange(\n j jsonb\n) RETURNS tstzrange AS $$\nDECLARE\n op text;\n filter jsonb := j->'filter';\n dtrange tstzrange := tstzrange('-infinity'::timestamptz,'infinity'::timestamptz);\n sdate timestamptz := '-infinity'::timestamptz;\n edate timestamptz := 'infinity'::timestamptz;\n jpitem jsonb;\nBEGIN\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n END IF;\n IF NOT (filter @? '$.**.op ? (@ == \"or\" || @ == \"not\")') THEN\n FOR jpitem IN SELECT j FROM jsonb_path_query(filter,'strict $.** ? (@.args[*].property == \"datetime\")'::jsonpath) j LOOP\n op := lower(jpitem->>'op');\n dtrange := parse_dtrange(jpitem->'args'->1);\n IF op IN ('<=', 'lt', 'lte', '<', 'le', 't_before') THEN\n sdate := greatest(sdate,'-infinity');\n edate := least(edate, upper(dtrange));\n ELSIF op IN ('>=', '>', 'gt', 'gte', 'ge', 't_after') THEN\n edate := least(edate, 'infinity');\n sdate := greatest(sdate, lower(dtrange));\n ELSIF op IN ('=', 'eq') THEN\n edate := least(edate, upper(dtrange));\n sdate := greatest(sdate, lower(dtrange));\n END IF;\n RAISE NOTICE '2 OP: %, ARGS: %, DTRANGE: %, SDATE: %, EDATE: %', op, jpitem->'args'->1, dtrange, sdate, edate;\n END LOOP;\n END IF;\n IF sdate > edate THEN\n RETURN 'empty'::tstzrange;\n END IF;\n RETURN tstzrange(sdate,edate, '[]');\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT SET TIME ZONE 'UTC';\n\nCREATE OR REPLACE FUNCTION paging_collections(\n IN j jsonb\n) RETURNS text[] AS $$\nDECLARE\n filter jsonb := j->'filter';\n jpitem jsonb;\n op text;\n args jsonb;\n arg jsonb;\n collections text[];\nBEGIN\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n END IF;\n IF NOT (filter @? '$.**.op ? (@ == \"or\" || @ == \"not\")') THEN\n FOR jpitem IN SELECT j FROM jsonb_path_query(filter,'strict $.** ? (@.args[*].property == \"collection\")'::jsonpath) j LOOP\n RAISE NOTICE 'JPITEM: %', jpitem;\n op := jpitem->>'op';\n args := jpitem->'args';\n IF op IN ('=', 'eq', 'in') THEN\n FOR arg IN SELECT a FROM jsonb_array_elements(args) a LOOP\n IF jsonb_typeof(arg) IN ('string', 'array') THEN\n RAISE NOTICE 'arg: %, collections: %', arg, collections;\n IF collections IS NULL OR collections = '{}'::text[] THEN\n collections := to_text_array(arg);\n ELSE\n collections := array_intersection(collections, to_text_array(arg));\n END IF;\n END IF;\n END LOOP;\n END IF;\n END LOOP;\n END IF;\n IF collections = '{}'::text[] THEN\n RETURN NULL;\n END IF;\n RETURN collections;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\nCREATE OR REPLACE FUNCTION collection_base_item(content jsonb) RETURNS jsonb AS $$\n SELECT jsonb_build_object(\n 'type', 'Feature',\n 'stac_version', content->'stac_version',\n 'assets', content->'item_assets',\n 'collection', content->'id'\n );\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TABLE IF NOT EXISTS collections (\n key bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,\n id text GENERATED ALWAYS AS (content->>'id') STORED UNIQUE,\n content JSONB NOT NULL,\n base_item jsonb GENERATED ALWAYS AS (pgstac.collection_base_item(content)) STORED,\n partition_trunc text CHECK (partition_trunc IN ('year', 'month'))\n);\n\n\n\nCREATE OR REPLACE FUNCTION collection_base_item(cid text) RETURNS jsonb AS $$\n SELECT pgstac.collection_base_item(content) FROM pgstac.collections WHERE id = cid LIMIT 1;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION create_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION update_collection(data jsonb) RETURNS VOID AS $$\nDECLARE\n out collections%ROWTYPE;\nBEGIN\n UPDATE collections SET content=data WHERE id = data->>'id' RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ON CONFLICT (id) DO\n UPDATE\n SET content=EXCLUDED.content\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION delete_collection(_id text) RETURNS VOID AS $$\nDECLARE\n out collections%ROWTYPE;\nBEGIN\n DELETE FROM collections WHERE id = _id RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION get_collection(id text) RETURNS jsonb AS $$\n SELECT content FROM collections\n WHERE id=$1\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION all_collections() RETURNS jsonb AS $$\n SELECT jsonb_agg(content) FROM collections;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\nCREATE TABLE items (\n id text NOT NULL,\n geometry geometry NOT NULL,\n collection text NOT NULL,\n datetime timestamptz NOT NULL,\n end_datetime timestamptz NOT NULL,\n content JSONB NOT NULL\n)\nPARTITION BY LIST (collection)\n;\n\nCREATE INDEX \"datetime_idx\" ON items USING BTREE (datetime DESC, end_datetime ASC);\nCREATE INDEX \"geometry_idx\" ON items USING GIST (geometry);\n\nCREATE STATISTICS datetime_stats (dependencies) on datetime, end_datetime from items;\n\nALTER TABLE items ADD CONSTRAINT items_collections_fk FOREIGN KEY (collection) REFERENCES collections(id) ON DELETE CASCADE DEFERRABLE;\n\nCREATE OR REPLACE FUNCTION partition_after_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p text;\n t timestamptz := clock_timestamp();\nBEGIN\n RAISE NOTICE 'Updating partition stats %', t;\n FOR p IN SELECT DISTINCT partition\n FROM newdata n JOIN partition_sys_meta p\n ON (n.collection=p.collection AND n.datetime <@ p.partition_dtrange)\n LOOP\n PERFORM run_or_queue(format('SELECT update_partition_stats(%L, %L);', p, true));\n END LOOP;\n RAISE NOTICE 't: % %', t, clock_timestamp() - t;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE TRIGGER items_after_insert_trigger\nAFTER INSERT ON items\nREFERENCING NEW TABLE AS newdata\nFOR EACH STATEMENT\nEXECUTE FUNCTION partition_after_triggerfunc();\n\nCREATE TRIGGER items_after_update_trigger\nAFTER DELETE ON items\nREFERENCING OLD TABLE AS newdata\nFOR EACH STATEMENT\nEXECUTE FUNCTION partition_after_triggerfunc();\n\nCREATE TRIGGER items_after_delete_trigger\nAFTER UPDATE ON items\nREFERENCING NEW TABLE AS newdata\nFOR EACH STATEMENT\nEXECUTE FUNCTION partition_after_triggerfunc();\n\n\nCREATE OR REPLACE FUNCTION content_slim(_item jsonb) RETURNS jsonb AS $$\n SELECT strip_jsonb(_item - '{id,geometry,collection,type}'::text[], collection_base_item(_item->>'collection')) - '{id,geometry,collection,type}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_dehydrate(content jsonb) RETURNS items AS $$\n SELECT\n content->>'id' as id,\n stac_geom(content) as geometry,\n content->>'collection' as collection,\n stac_datetime(content) as datetime,\n stac_end_datetime(content) as end_datetime,\n content_slim(content) as content\n ;\n$$ LANGUAGE SQL STABLE;\n\nCREATE OR REPLACE FUNCTION include_field(f text, fields jsonb DEFAULT '{}'::jsonb) RETURNS boolean AS $$\nDECLARE\n includes jsonb := fields->'include';\n excludes jsonb := fields->'exclude';\nBEGIN\n IF f IS NULL THEN\n RETURN NULL;\n END IF;\n\n\n IF\n jsonb_typeof(excludes) = 'array'\n AND jsonb_array_length(excludes)>0\n AND excludes ? f\n THEN\n RETURN FALSE;\n END IF;\n\n IF\n (\n jsonb_typeof(includes) = 'array'\n AND jsonb_array_length(includes) > 0\n AND includes ? f\n ) OR\n (\n includes IS NULL\n OR jsonb_typeof(includes) = 'null'\n OR jsonb_array_length(includes) = 0\n )\n THEN\n RETURN TRUE;\n END IF;\n\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nDROP FUNCTION IF EXISTS content_hydrate(jsonb, jsonb, jsonb);\nCREATE OR REPLACE FUNCTION content_hydrate(\n _item jsonb,\n _base_item jsonb,\n fields jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\n SELECT merge_jsonb(\n jsonb_fields(_item, fields),\n jsonb_fields(_base_item, fields)\n );\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION content_hydrate(_item items, _collection collections, fields jsonb DEFAULT '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\n content jsonb;\n base_item jsonb := _collection.base_item;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry, 20)::jsonb;\n END IF;\n output := content_hydrate(\n jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'collection', _item.collection,\n 'type', 'Feature'\n ) || _item.content,\n _collection.base_item,\n fields\n );\n\n RETURN output;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_nonhydrated(\n _item items,\n fields jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry, 20)::jsonb;\n END IF;\n output := jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'collection', _item.collection,\n 'type', 'Feature'\n ) || _item.content;\n RETURN output;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_hydrate(_item items, fields jsonb DEFAULT '{}'::jsonb) RETURNS jsonb AS $$\n SELECT content_hydrate(\n _item,\n (SELECT c FROM collections c WHERE id=_item.collection LIMIT 1),\n fields\n );\n$$ LANGUAGE SQL STABLE;\n\n\nCREATE UNLOGGED TABLE items_staging (\n content JSONB NOT NULL\n);\nCREATE UNLOGGED TABLE items_staging_ignore (\n content JSONB NOT NULL\n);\nCREATE UNLOGGED TABLE items_staging_upsert (\n content JSONB NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION items_staging_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p record;\n _partitions text[];\n part text;\n ts timestamptz := clock_timestamp();\nBEGIN\n RAISE NOTICE 'Creating Partitions. %', clock_timestamp() - ts;\n\n FOR part IN WITH t AS (\n SELECT\n n.content->>'collection' as collection,\n stac_daterange(n.content->'properties') as dtr,\n partition_trunc\n FROM newdata n JOIN collections ON (n.content->>'collection'=collections.id)\n ), p AS (\n SELECT\n collection,\n COALESCE(date_trunc(partition_trunc::text, lower(dtr)),'-infinity') as d,\n tstzrange(min(lower(dtr)),max(lower(dtr)),'[]') as dtrange,\n tstzrange(min(upper(dtr)),max(upper(dtr)),'[]') as edtrange\n FROM t\n GROUP BY 1,2\n ) SELECT check_partition(collection, dtrange, edtrange) FROM p LOOP\n RAISE NOTICE 'Partition %', part;\n END LOOP;\n\n RAISE NOTICE 'Doing the insert. %', clock_timestamp() - ts;\n IF TG_TABLE_NAME = 'items_staging' THEN\n INSERT INTO items\n SELECT\n (content_dehydrate(content)).*\n FROM newdata;\n RAISE NOTICE 'Doing the delete. %', clock_timestamp() - ts;\n DELETE FROM items_staging;\n ELSIF TG_TABLE_NAME = 'items_staging_ignore' THEN\n INSERT INTO items\n SELECT\n (content_dehydrate(content)).*\n FROM newdata\n ON CONFLICT DO NOTHING;\n RAISE NOTICE 'Doing the delete. %', clock_timestamp() - ts;\n DELETE FROM items_staging_ignore;\n ELSIF TG_TABLE_NAME = 'items_staging_upsert' THEN\n WITH staging_formatted AS (\n SELECT (content_dehydrate(content)).* FROM newdata\n ), deletes AS (\n DELETE FROM items i USING staging_formatted s\n WHERE\n i.id = s.id\n AND i.collection = s.collection\n AND i IS DISTINCT FROM s\n RETURNING i.id, i.collection\n )\n INSERT INTO items\n SELECT s.* FROM\n staging_formatted s\n ON CONFLICT DO NOTHING;\n RAISE NOTICE 'Doing the delete. %', clock_timestamp() - ts;\n DELETE FROM items_staging_upsert;\n END IF;\n RAISE NOTICE 'Done. %', clock_timestamp() - ts;\n\n RETURN NULL;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER items_staging_insert_trigger AFTER INSERT ON items_staging REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\nCREATE TRIGGER items_staging_insert_ignore_trigger AFTER INSERT ON items_staging_ignore REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\nCREATE TRIGGER items_staging_insert_upsert_trigger AFTER INSERT ON items_staging_upsert REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\n\nCREATE OR REPLACE FUNCTION item_by_id(_id text, _collection text DEFAULT NULL) RETURNS items AS\n$$\nDECLARE\n i items%ROWTYPE;\nBEGIN\n SELECT * INTO i FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection) LIMIT 1;\n RETURN i;\nEND;\n$$ LANGUAGE PLPGSQL STABLE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION get_item(_id text, _collection text DEFAULT NULL) RETURNS jsonb AS $$\n SELECT content_hydrate(items) FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection);\n$$ LANGUAGE SQL STABLE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION delete_item(_id text, _collection text DEFAULT NULL) RETURNS VOID AS $$\nDECLARE\nout items%ROWTYPE;\nBEGIN\n DELETE FROM items WHERE id = _id AND (_collection IS NULL OR collection=_collection) RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n--/*\nCREATE OR REPLACE FUNCTION create_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION update_item(content jsonb) RETURNS VOID AS $$\nDECLARE\n old items %ROWTYPE;\n out items%ROWTYPE;\nBEGIN\n PERFORM delete_item(content->>'id', content->>'collection');\n PERFORM create_item(content);\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION create_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION collection_bbox(id text) RETURNS jsonb AS $$\n SELECT (replace(replace(replace(st_extent(geometry)::text,'BOX(','[['),')',']]'),' ',','))::jsonb\n FROM items WHERE collection=$1;\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION collection_temporal_extent(id text) RETURNS jsonb AS $$\n SELECT to_jsonb(array[array[min(datetime)::text, max(datetime)::text]])\n FROM items WHERE collection=$1;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection_extents() RETURNS VOID AS $$\nUPDATE collections SET\n content = content ||\n jsonb_build_object(\n 'extent', jsonb_build_object(\n 'spatial', jsonb_build_object(\n 'bbox', collection_bbox(collections.id)\n ),\n 'temporal', jsonb_build_object(\n 'interval', collection_temporal_extent(collections.id)\n )\n )\n )\n;\n$$ LANGUAGE SQL;\nCREATE TABLE partition_stats (\n partition text PRIMARY KEY,\n dtrange tstzrange,\n edtrange tstzrange,\n spatial geometry,\n last_updated timestamptz,\n keys text[]\n) WITH (FILLFACTOR=90);\n\nCREATE INDEX partitions_range_idx ON partition_stats USING GIST(dtrange);\n\n\nCREATE OR REPLACE FUNCTION constraint_tstzrange(expr text) RETURNS tstzrange AS $$\n WITH t AS (\n SELECT regexp_matches(\n expr,\n E'\\\\(''\\([0-9 :+-]*\\)''\\\\).*\\\\(''\\([0-9 :+-]*\\)''\\\\)'\n ) AS m\n ) SELECT tstzrange(m[1]::timestamptz, m[2]::timestamptz) FROM t\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE STRICT;\n\nCREATE OR REPLACE FUNCTION dt_constraint(coid oid, OUT dt tstzrange, OUT edt tstzrange) RETURNS RECORD AS $$\nDECLARE\n expr text := pg_get_constraintdef(coid);\n matches timestamptz[];\nBEGIN\n IF expr LIKE '%NULL%' THEN\n dt := tstzrange(null::timestamptz, null::timestamptz);\n edt := tstzrange(null::timestamptz, null::timestamptz);\n RETURN;\n END IF;\n WITH f AS (SELECT (regexp_matches(expr, E'([0-9]{4}-[0-1][0-9]-[0-3][0-9] [0-2][0-9]:[0-5][0-9]:[0-5][0-9]\\.?[0-9]*)', 'g'))[1] f)\n SELECT array_agg(f::timestamptz) INTO matches FROM f;\n IF cardinality(matches) = 4 THEN\n dt := tstzrange(matches[1], matches[2],'[]');\n edt := tstzrange(matches[3], matches[4], '[]');\n RETURN;\n ELSIF cardinality(matches) = 2 THEN\n edt := tstzrange(matches[1], matches[2],'[]');\n RETURN;\n END IF;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\nCREATE OR REPLACE VIEW partition_sys_meta AS\nSELECT\n relid::text as partition,\n replace(replace(CASE WHEN level = 1 THEN pg_get_expr(c.relpartbound, c.oid)\n ELSE pg_get_expr(parent.relpartbound, parent.oid)\n END, 'FOR VALUES IN (''',''), ''')','') AS collection,\n level,\n c.reltuples,\n c.relhastriggers,\n COALESCE(constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as partition_dtrange,\n COALESCE((dt_constraint(edt.oid)).dt, constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as constraint_dtrange,\n COALESCE((dt_constraint(edt.oid)).edt, tstzrange('-infinity', 'infinity','[]')) as constraint_edtrange\nFROM\n pg_partition_tree('items')\n JOIN pg_class c ON (relid::regclass = c.oid)\n JOIN pg_class parent ON (parentrelid::regclass = parent.oid AND isleaf)\n LEFT JOIN pg_constraint edt ON (conrelid=c.oid AND contype='c')\nWHERE isleaf\n;\n\nCREATE VIEW partitions AS\nSELECT * FROM partition_sys_meta LEFT JOIN partition_stats USING (partition);\n\nCREATE OR REPLACE FUNCTION update_partition_stats_q(_partition text, istrigger boolean default false) RETURNS VOID AS $$\nDECLARE\nBEGIN\n PERFORM run_or_queue(\n format('SELECT update_partition_stats(%L, %L);', _partition, istrigger)\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION update_partition_stats(_partition text, istrigger boolean default false) RETURNS VOID AS $$\nDECLARE\n dtrange tstzrange;\n edtrange tstzrange;\n cdtrange tstzrange;\n cedtrange tstzrange;\n extent geometry;\n collection text;\nBEGIN\n RAISE NOTICE 'Updating stats for %.', _partition;\n EXECUTE format(\n $q$\n SELECT\n tstzrange(min(datetime), max(datetime),'[]'),\n tstzrange(min(end_datetime), max(end_datetime), '[]')\n FROM %I\n $q$,\n _partition\n ) INTO dtrange, edtrange;\n extent := st_estimatedextent('pgstac', _partition, 'geometry');\n INSERT INTO partition_stats (partition, dtrange, edtrange, spatial, last_updated)\n SELECT _partition, dtrange, edtrange, extent, now()\n ON CONFLICT (partition) DO\n UPDATE SET\n dtrange=EXCLUDED.dtrange,\n edtrange=EXCLUDED.edtrange,\n spatial=EXCLUDED.spatial,\n last_updated=EXCLUDED.last_updated\n ;\n SELECT\n constraint_dtrange, constraint_edtrange, partitions.collection\n INTO cdtrange, cedtrange, collection\n FROM partitions WHERE partition = _partition;\n\n RAISE NOTICE 'Checking if we need to modify constraints.';\n IF\n (cdtrange IS DISTINCT FROM dtrange OR edtrange IS DISTINCT FROM cedtrange)\n AND NOT istrigger\n THEN\n RAISE NOTICE 'Modifying Constraints';\n RAISE NOTICE 'Existing % %', cdtrange, cedtrange;\n RAISE NOTICE 'New % %', dtrange, edtrange;\n PERFORM drop_table_constraints(_partition);\n PERFORM create_table_constraints(_partition, dtrange, edtrange);\n END IF;\n RAISE NOTICE 'Checking if we need to update collection extents.';\n IF get_setting_bool('update_collection_extent') THEN\n RAISE NOTICE 'updating collection extent for %', collection;\n PERFORM run_or_queue(format($q$\n UPDATE collections\n SET content = jsonb_set_lax(\n content,\n '{extent}'::text[],\n collection_extent(%L),\n true,\n 'use_json_null'\n ) WHERE id=%L\n ;\n $q$, collection, collection));\n ELSE\n RAISE NOTICE 'Not updating collection extent for %', collection;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL STRICT;\n\n\nCREATE OR REPLACE FUNCTION partition_name( IN collection text, IN dt timestamptz, OUT partition_name text, OUT partition_range tstzrange) AS $$\nDECLARE\n c RECORD;\n parent_name text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n parent_name := format('_items_%s', c.key);\n\n\n IF c.partition_trunc = 'year' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYY'));\n ELSIF c.partition_trunc = 'month' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYYMM'));\n ELSE\n partition_name := parent_name;\n partition_range := tstzrange('-infinity'::timestamptz, 'infinity'::timestamptz, '[]');\n END IF;\n IF partition_range IS NULL THEN\n partition_range := tstzrange(\n date_trunc(c.partition_trunc::text, dt),\n date_trunc(c.partition_trunc::text, dt) + concat('1 ', c.partition_trunc)::interval\n );\n END IF;\n RETURN;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION drop_table_constraints(t text) RETURNS text AS $$\nDECLARE\n q text;\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM partitions WHERE partition=t) THEN\n RETURN NULL;\n END IF;\n FOR q IN SELECT FORMAT(\n $q$\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n $q$,\n t,\n conname\n ) FROM pg_constraint\n WHERE conrelid=t::regclass::oid AND contype='c'\n LOOP\n EXECUTE q;\n END LOOP;\n RETURN t;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE OR REPLACE FUNCTION create_table_constraints(t text, _dtrange tstzrange, _edtrange tstzrange) RETURNS text AS $$\nDECLARE\n q text;\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM partitions WHERE partition=t) THEN\n RETURN NULL;\n END IF;\n RAISE NOTICE 'Creating Table Constraints for % % %', t, _dtrange, _edtrange;\n IF _dtrange = 'empty' AND _edtrange = 'empty' THEN\n q :=format(\n $q$\n ALTER TABLE %I\n ADD CONSTRAINT %I\n CHECK (((datetime IS NULL) AND (end_datetime IS NULL))) NOT VALID\n ;\n ALTER TABLE %I\n VALIDATE CONSTRAINT %I\n ;\n $q$,\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t)\n );\n ELSE\n q :=format(\n $q$\n ALTER TABLE %I\n ADD CONSTRAINT %I\n CHECK (\n (datetime >= %L)\n AND (datetime <= %L)\n AND (end_datetime >= %L)\n AND (end_datetime <= %L)\n ) NOT VALID\n ;\n ALTER TABLE %I\n VALIDATE CONSTRAINT %I\n ;\n $q$,\n t,\n format('%s_dt', t),\n lower(_dtrange),\n upper(_dtrange),\n lower(_edtrange),\n upper(_edtrange),\n t,\n format('%s_dt', t)\n );\n END IF;\n PERFORM run_or_queue(q);\n RETURN t;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION check_partition(\n _collection text,\n _dtrange tstzrange,\n _edtrange tstzrange\n) RETURNS text AS $$\nDECLARE\n c RECORD;\n pm RECORD;\n _partition_name text;\n _partition_dtrange tstzrange;\n _constraint_dtrange tstzrange;\n _constraint_edtrange tstzrange;\n q text;\n deferrable_q text;\n err_context text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=_collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', _collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n\n IF c.partition_trunc IS NOT NULL THEN\n _partition_dtrange := tstzrange(\n date_trunc(c.partition_trunc, lower(_dtrange)),\n date_trunc(c.partition_trunc, lower(_dtrange)) + (concat('1 ', c.partition_trunc))::interval,\n '[)'\n );\n ELSE\n _partition_dtrange := '[-infinity, infinity]'::tstzrange;\n END IF;\n\n IF NOT _partition_dtrange @> _dtrange THEN\n RAISE EXCEPTION 'dtrange % is greater than the partition size % for collection %', _dtrange, c.partition_trunc, _collection;\n END IF;\n\n\n IF c.partition_trunc = 'year' THEN\n _partition_name := format('_items_%s_%s', c.key, to_char(lower(_partition_dtrange),'YYYY'));\n ELSIF c.partition_trunc = 'month' THEN\n _partition_name := format('_items_%s_%s', c.key, to_char(lower(_partition_dtrange),'YYYYMM'));\n ELSE\n _partition_name := format('_items_%s', c.key);\n END IF;\n\n SELECT * INTO pm FROM partition_sys_meta WHERE collection=_collection AND partition_dtrange @> _dtrange;\n IF FOUND THEN\n RAISE NOTICE '% % %', _edtrange, _dtrange, pm;\n _constraint_edtrange :=\n tstzrange(\n least(\n lower(_edtrange),\n nullif(lower(pm.constraint_edtrange), '-infinity')\n ),\n greatest(\n upper(_edtrange),\n nullif(upper(pm.constraint_edtrange), 'infinity')\n ),\n '[]'\n );\n _constraint_dtrange :=\n tstzrange(\n least(\n lower(_dtrange),\n nullif(lower(pm.constraint_dtrange), '-infinity')\n ),\n greatest(\n upper(_dtrange),\n nullif(upper(pm.constraint_dtrange), 'infinity')\n ),\n '[]'\n );\n\n IF pm.constraint_edtrange @> _edtrange AND pm.constraint_dtrange @> _dtrange THEN\n RETURN pm.partition;\n ELSE\n PERFORM drop_table_constraints(_partition_name);\n END IF;\n ELSE\n _constraint_edtrange := _edtrange;\n _constraint_dtrange := _dtrange;\n END IF;\n RAISE NOTICE 'Creating partition % %', _partition_name, _partition_dtrange;\n IF c.partition_trunc IS NULL THEN\n q := format(\n $q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n $q$,\n _partition_name,\n _collection,\n concat(_partition_name,'_pk'),\n _partition_name\n );\n ELSE\n q := format(\n $q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L) PARTITION BY RANGE (datetime);\n CREATE TABLE IF NOT EXISTS %I partition OF %I FOR VALUES FROM (%L) TO (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n $q$,\n format('_items_%s', c.key),\n _collection,\n _partition_name,\n format('_items_%s', c.key),\n lower(_partition_dtrange),\n upper(_partition_dtrange),\n format('%s_pk', _partition_name),\n _partition_name\n );\n END IF;\n\n BEGIN\n EXECUTE q;\n EXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', _partition_name;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\n END;\n PERFORM create_table_constraints(_partition_name, _constraint_dtrange, _constraint_edtrange);\n PERFORM maintain_partitions(_partition_name);\n PERFORM update_partition_stats_q(_partition_name, true);\n RETURN _partition_name;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION repartition(_collection text, _partition_trunc text, triggered boolean DEFAULT FALSE) RETURNS text AS $$\nDECLARE\n c RECORD;\n q text;\n from_trunc text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=_collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', _collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n IF triggered THEN\n RAISE NOTICE 'Converting % to % partitioning via Trigger', _collection, _partition_trunc;\n ELSE\n RAISE NOTICE 'Converting % from using % to % partitioning', _collection, c.partition_trunc, _partition_trunc;\n IF c.partition_trunc IS NOT DISTINCT FROM _partition_trunc THEN\n RAISE NOTICE 'Collection % already set to use partition by %', _collection, _partition_trunc;\n RETURN _collection;\n END IF;\n END IF;\n\n IF EXISTS (SELECT 1 FROM partitions WHERE collection=_collection LIMIT 1) THEN\n EXECUTE format(\n $q$\n CREATE TEMP TABLE changepartitionstaging ON COMMIT DROP AS SELECT * FROM %I;\n DROP TABLE IF EXISTS %I CASCADE;\n WITH p AS (\n SELECT\n collection,\n CASE WHEN %L IS NULL THEN '-infinity'::timestamptz\n ELSE date_trunc(%L, datetime)\n END as d,\n tstzrange(min(datetime),max(datetime),'[]') as dtrange,\n tstzrange(min(datetime),max(datetime),'[]') as edtrange\n FROM changepartitionstaging\n GROUP BY 1,2\n ) SELECT check_partition(collection, dtrange, edtrange) FROM p;\n INSERT INTO items SELECT * FROM changepartitionstaging;\n DROP TABLE changepartitionstaging;\n $q$,\n concat('_items_', c.key),\n concat('_items_', c.key),\n c.partition_trunc,\n c.partition_trunc\n );\n END IF;\n RETURN _collection;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE OR REPLACE FUNCTION collections_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n q text;\n partition_name text := format('_items_%s', NEW.key);\n partition_exists boolean := false;\n partition_empty boolean := true;\n err_context text;\n loadtemp boolean := FALSE;\nBEGIN\n RAISE NOTICE 'Collection Trigger. % %', NEW.id, NEW.key;\n IF TG_OP = 'UPDATE' AND NEW.partition_trunc IS DISTINCT FROM OLD.partition_trunc THEN\n PERFORM repartition(NEW.id, NEW.partition_trunc, TRUE);\n END IF;\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER collections_trigger AFTER\nINSERT\nOR\nUPDATE ON collections\nFOR EACH ROW EXECUTE FUNCTION collections_trigger_func();\nCREATE OR REPLACE VIEW partition_steps AS\nSELECT\n partition as name,\n date_trunc('month',lower(partition_dtrange)) as sdate,\n date_trunc('month', upper(partition_dtrange)) + '1 month'::interval as edate\n FROM partitions WHERE partition_dtrange IS NOT NULL AND partition_dtrange != 'empty'::tstzrange\n ORDER BY dtrange ASC\n;\n\nCREATE OR REPLACE FUNCTION chunker(\n IN _where text,\n OUT s timestamptz,\n OUT e timestamptz\n) RETURNS SETOF RECORD AS $$\nDECLARE\n explain jsonb;\nBEGIN\n IF _where IS NULL THEN\n _where := ' TRUE ';\n END IF;\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s;', _where)\n INTO explain;\n\n RETURN QUERY\n WITH t AS (\n SELECT j->>0 as p FROM\n jsonb_path_query(\n explain,\n 'strict $.**.\"Relation Name\" ? (@ != null)'\n ) j\n ),\n parts AS (\n SELECT sdate, edate FROM t JOIN partition_steps ON (t.p = name)\n ),\n times AS (\n SELECT sdate FROM parts\n UNION\n SELECT edate FROM parts\n ),\n uniq AS (\n SELECT DISTINCT sdate FROM times ORDER BY sdate\n ),\n last AS (\n SELECT sdate, lead(sdate, 1) over () as edate FROM uniq\n )\n SELECT sdate, edate FROM last WHERE edate IS NOT NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION partition_queries(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN partitions text[] DEFAULT NULL\n) RETURNS SETOF text AS $$\nDECLARE\n query text;\n sdate timestamptz;\n edate timestamptz;\nBEGIN\nIF _where IS NULL OR trim(_where) = '' THEN\n _where = ' TRUE ';\nEND IF;\nRAISE NOTICE 'Getting chunks for % %', _where, _orderby;\nIF _orderby ILIKE 'datetime d%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 DESC LOOP\n RETURN NEXT format($q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n $q$,\n sdate,\n edate,\n _where,\n _orderby\n );\n END LOOP;\nELSIF _orderby ILIKE 'datetime a%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 ASC LOOP\n RETURN NEXT format($q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n $q$,\n sdate,\n edate,\n _where,\n _orderby\n );\n END LOOP;\nELSE\n query := format($q$\n SELECT * FROM items\n WHERE %s\n ORDER BY %s\n $q$, _where, _orderby\n );\n\n RETURN NEXT query;\n RETURN;\nEND IF;\n\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION partition_query_view(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN _limit int DEFAULT 10\n) RETURNS text AS $$\n WITH p AS (\n SELECT * FROM partition_queries(_where, _orderby) p\n )\n SELECT\n CASE WHEN EXISTS (SELECT 1 FROM p) THEN\n (SELECT format($q$\n SELECT * FROM (\n %s\n ) total LIMIT %s\n $q$,\n string_agg(\n format($q$ SELECT * FROM ( %s ) AS sub $q$, p),\n '\n UNION ALL\n '\n ),\n _limit\n ))\n ELSE NULL\n END FROM p;\n$$ LANGUAGE SQL IMMUTABLE;\n\n\n\n\nCREATE OR REPLACE FUNCTION stac_search_to_where(j jsonb) RETURNS text AS $$\nDECLARE\n where_segments text[];\n _where text;\n dtrange tstzrange;\n collections text[];\n geom geometry;\n sdate timestamptz;\n edate timestamptz;\n filterlang text;\n filter jsonb := j->'filter';\nBEGIN\n IF j ? 'ids' THEN\n where_segments := where_segments || format('id = ANY (%L) ', to_text_array(j->'ids'));\n END IF;\n\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n where_segments := where_segments || format('collection = ANY (%L) ', collections);\n END IF;\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n\n where_segments := where_segments || format(' datetime <= %L::timestamptz AND end_datetime >= %L::timestamptz ',\n edate,\n sdate\n );\n END IF;\n\n geom := stac_geom(j);\n IF geom IS NOT NULL THEN\n where_segments := where_segments || format('st_intersects(geometry, %L)',geom);\n END IF;\n\n filterlang := COALESCE(\n j->>'filter-lang',\n get_setting('default_filter_lang', j->'conf')\n );\n IF NOT filter @? '$.**.op' THEN\n filterlang := 'cql-json';\n END IF;\n\n IF filterlang NOT IN ('cql-json','cql2-json') AND j ? 'filter' THEN\n RAISE EXCEPTION '% is not a supported filter-lang. Please use cql-json or cql2-json.', filterlang;\n END IF;\n\n IF j ? 'query' AND j ? 'filter' THEN\n RAISE EXCEPTION 'Can only use either query or filter at one time.';\n END IF;\n\n IF j ? 'query' THEN\n filter := query_to_cql2(j->'query');\n ELSIF filterlang = 'cql-json' THEN\n filter := cql1_to_cql2(filter);\n END IF;\n RAISE NOTICE 'FILTER: %', filter;\n where_segments := where_segments || cql2_query(filter);\n IF cardinality(where_segments) < 1 THEN\n RETURN ' TRUE ';\n END IF;\n\n _where := array_to_string(array_remove(where_segments, NULL), ' AND ');\n\n IF _where IS NULL OR BTRIM(_where) = '' THEN\n RETURN ' TRUE ';\n END IF;\n RETURN _where;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION parse_sort_dir(_dir text, reverse boolean default false) RETURNS text AS $$\n WITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n ) SELECT\n CASE\n WHEN NOT reverse THEN d\n WHEN d = 'ASC' THEN 'DESC'\n WHEN d = 'DESC' THEN 'ASC'\n END\n FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION sort_dir_to_op(_dir text, prev boolean default false) RETURNS text AS $$\n WITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n ) SELECT\n CASE\n WHEN d = 'ASC' AND prev THEN '<='\n WHEN d = 'DESC' AND prev THEN '>='\n WHEN d = 'ASC' THEN '>='\n WHEN d = 'DESC' THEN '<='\n END\n FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION sort_sqlorderby(\n _search jsonb DEFAULT NULL,\n reverse boolean DEFAULT FALSE\n) RETURNS text AS $$\n WITH sortby AS (\n SELECT coalesce(_search->'sortby','[{\"field\":\"datetime\", \"direction\":\"desc\"}]') as sort\n ), withid AS (\n SELECT CASE\n WHEN sort @? '$[*] ? (@.field == \"id\")' THEN sort\n ELSE sort || '[{\"field\":\"id\", \"direction\":\"desc\"}]'::jsonb\n END as sort\n FROM sortby\n ), withid_rows AS (\n SELECT jsonb_array_elements(sort) as value FROM withid\n ),sorts AS (\n SELECT\n coalesce(\n -- field_orderby((items_path(value->>'field')).path_txt),\n (queryable(value->>'field')).expression\n ) as key,\n parse_sort_dir(value->>'direction', reverse) as dir\n FROM withid_rows\n )\n SELECT array_to_string(\n array_agg(concat(key, ' ', dir)),\n ', '\n ) FROM sorts;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_sort_dir(sort_item jsonb) RETURNS text AS $$\n SELECT CASE WHEN sort_item->>'direction' ILIKE 'desc%' THEN 'DESC' ELSE 'ASC' END;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION get_token_val_str(\n _field text,\n _item items\n) RETURNS text AS $$\nDECLARE\nliteral text;\nBEGIN\nRAISE NOTICE '% %', _field, _item;\nCREATE TEMP TABLE _token_item ON COMMIT DROP AS SELECT (_item).*;\nEXECUTE format($q$ SELECT quote_literal(%s) FROM _token_item $q$, _field) INTO literal;\nDROP TABLE IF EXISTS _token_item;\nRETURN literal;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION get_token_filter(_search jsonb = '{}'::jsonb, token_rec jsonb DEFAULT NULL) RETURNS text AS $$\nDECLARE\n token_id text;\n filters text[] := '{}'::text[];\n prev boolean := TRUE;\n field text;\n dir text;\n sort record;\n orfilter text := '';\n orfilters text[] := '{}'::text[];\n andfilters text[] := '{}'::text[];\n output text;\n token_where text;\n token_item items%ROWTYPE;\nBEGIN\n RAISE NOTICE 'Getting Token Filter. % %', _search, token_rec;\n -- If no token provided return NULL\n IF token_rec IS NULL THEN\n IF NOT (_search ? 'token' AND\n (\n (_search->>'token' ILIKE 'prev:%')\n OR\n (_search->>'token' ILIKE 'next:%')\n )\n ) THEN\n RETURN NULL;\n END IF;\n prev := (_search->>'token' ILIKE 'prev:%');\n token_id := substr(_search->>'token', 6);\n IF token_id IS NULL OR token_id = '' THEN\n RAISE WARNING 'next or prev set, but no token id found';\n RETURN NULL;\n END IF;\n SELECT to_jsonb(items) INTO token_rec\n FROM items WHERE id=token_id;\n END IF;\n RAISE NOTICE 'TOKEN ID: % %', token_rec, token_rec->'id';\n\n\n RAISE NOTICE 'TOKEN ID: % %', token_rec, token_rec->'id';\n token_item := jsonb_populate_record(null::items, token_rec);\n RAISE NOTICE 'TOKEN ITEM ----- %', token_item;\n\n\n CREATE TEMP TABLE sorts (\n _row int GENERATED ALWAYS AS IDENTITY NOT NULL,\n _field text PRIMARY KEY,\n _dir text NOT NULL,\n _val text\n ) ON COMMIT DROP;\n\n -- Make sure we only have distinct columns to sort with taking the first one we get\n INSERT INTO sorts (_field, _dir)\n SELECT\n (queryable(value->>'field')).expression,\n get_sort_dir(value)\n FROM\n jsonb_array_elements(coalesce(_search->'sortby','[{\"field\":\"datetime\",\"direction\":\"desc\"}]'))\n ON CONFLICT DO NOTHING\n ;\n RAISE NOTICE 'sorts 1: %', (SELECT jsonb_agg(to_json(sorts)) FROM sorts);\n -- Get the first sort direction provided. As the id is a primary key, if there are any\n -- sorts after id they won't do anything, so make sure that id is the last sort item.\n SELECT _dir INTO dir FROM sorts ORDER BY _row ASC LIMIT 1;\n IF EXISTS (SELECT 1 FROM sorts WHERE _field = 'id') THEN\n DELETE FROM sorts WHERE _row > (SELECT _row FROM sorts WHERE _field = 'id' ORDER BY _row ASC);\n ELSE\n INSERT INTO sorts (_field, _dir) VALUES ('id', dir);\n END IF;\n\n -- Add value from looked up item to the sorts table\n UPDATE sorts SET _val=get_token_val_str(_field, token_item);\n\n -- Check if all sorts are the same direction and use row comparison\n -- to filter\n RAISE NOTICE 'sorts 2: %', (SELECT jsonb_agg(to_json(sorts)) FROM sorts);\n FOR sort IN SELECT * FROM sorts ORDER BY _row asc LOOP\n orfilter := NULL;\n RAISE NOTICE 'SORT: %', sort;\n IF sort._val IS NOT NULL AND ((prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC')) THEN\n orfilter := format($f$(\n (%s < %s) OR (%s IS NULL)\n )$f$,\n sort._field,\n sort._val,\n sort._val\n );\n ELSIF sort._val IS NULL AND ((prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC')) THEN\n RAISE NOTICE '< but null';\n orfilter := format('%s IS NOT NULL', sort._field);\n ELSIF sort._val IS NULL THEN\n RAISE NOTICE '> but null';\n --orfilter := format('%s IS NULL', sort._field);\n ELSE\n orfilter := format($f$(\n (%s > %s) OR (%s IS NULL)\n )$f$,\n sort._field,\n sort._val,\n sort._field\n );\n END IF;\n RAISE NOTICE 'ORFILTER: %', orfilter;\n\n IF orfilter IS NOT NULL THEN\n IF sort._row = 1 THEN\n orfilters := orfilters || orfilter;\n ELSE\n orfilters := orfilters || format('(%s AND %s)', array_to_string(andfilters, ' AND '), orfilter);\n END IF;\n END IF;\n IF sort._val IS NOT NULL THEN\n andfilters := andfilters || format('%s = %s', sort._field, sort._val);\n ELSE\n andfilters := andfilters || format('%s IS NULL', sort._field);\n END IF;\n END LOOP;\n output := array_to_string(orfilters, ' OR ');\n\n DROP TABLE IF EXISTS sorts;\n token_where := concat('(',coalesce(output,'true'),')');\n IF trim(token_where) = '' THEN\n token_where := NULL;\n END IF;\n RAISE NOTICE 'TOKEN_WHERE: %',token_where;\n RETURN token_where;\n END;\n$$ LANGUAGE PLPGSQL SET transform_null_equals TO TRUE\n;\n\nCREATE OR REPLACE FUNCTION search_tohash(jsonb) RETURNS jsonb AS $$\n SELECT $1 - '{token,limit,context,includes,excludes}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION search_hash(jsonb, jsonb) RETURNS text AS $$\n SELECT md5(concat(search_tohash($1)::text,$2::text));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TABLE IF NOT EXISTS searches(\n hash text GENERATED ALWAYS AS (search_hash(search, metadata)) STORED PRIMARY KEY,\n search jsonb NOT NULL,\n _where text,\n orderby text,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n metadata jsonb DEFAULT '{}'::jsonb NOT NULL\n);\n\nCREATE TABLE IF NOT EXISTS search_wheres(\n id bigint generated always as identity primary key,\n _where text NOT NULL,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n statslastupdated timestamptz,\n estimated_count bigint,\n estimated_cost float,\n time_to_estimate float,\n total_count bigint,\n time_to_count float,\n partitions text[]\n);\n\nCREATE INDEX IF NOT EXISTS search_wheres_partitions ON search_wheres USING GIN (partitions);\nCREATE UNIQUE INDEX IF NOT EXISTS search_wheres_where ON search_wheres ((md5(_where)));\n\nCREATE OR REPLACE FUNCTION where_stats(inwhere text, updatestats boolean default false, conf jsonb default null) RETURNS search_wheres AS $$\nDECLARE\n t timestamptz;\n i interval;\n explain_json jsonb;\n partitions text[];\n sw search_wheres%ROWTYPE;\n inwhere_hash text := md5(inwhere);\n _context text := lower(context(conf));\n _stats_ttl interval := context_stats_ttl(conf);\n _estimated_cost float := context_estimated_cost(conf);\n _estimated_count int := context_estimated_count(conf);\nBEGIN\n IF _context = 'off' THEN\n sw._where := inwhere;\n return sw;\n END IF;\n\n SELECT * INTO sw FROM search_wheres WHERE md5(_where)=inwhere_hash FOR UPDATE;\n\n -- Update statistics if explicitly set, if statistics do not exist, or statistics ttl has expired\n IF NOT updatestats THEN\n RAISE NOTICE 'Checking if update is needed for: % .', inwhere;\n RAISE NOTICE 'Stats Last Updated: %', sw.statslastupdated;\n RAISE NOTICE 'TTL: %, Age: %', _stats_ttl, now() - sw.statslastupdated;\n RAISE NOTICE 'Context: %, Existing Total: %', _context, sw.total_count;\n IF\n sw.statslastupdated IS NULL\n OR (now() - sw.statslastupdated) > _stats_ttl\n OR (context(conf) != 'off' AND sw.total_count IS NULL)\n THEN\n updatestats := TRUE;\n END IF;\n END IF;\n\n sw._where := inwhere;\n sw.lastused := now();\n sw.usecount := coalesce(sw.usecount,0) + 1;\n\n IF NOT updatestats THEN\n UPDATE search_wheres SET\n lastused = sw.lastused,\n usecount = sw.usecount\n WHERE md5(_where) = inwhere_hash\n RETURNING * INTO sw\n ;\n RETURN sw;\n END IF;\n\n -- Use explain to get estimated count/cost and a list of the partitions that would be hit by the query\n t := clock_timestamp();\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s', inwhere)\n INTO explain_json;\n RAISE NOTICE 'Time for just the explain: %', clock_timestamp() - t;\n i := clock_timestamp() - t;\n\n sw.statslastupdated := now();\n sw.estimated_count := explain_json->0->'Plan'->'Plan Rows';\n sw.estimated_cost := explain_json->0->'Plan'->'Total Cost';\n sw.time_to_estimate := extract(epoch from i);\n\n RAISE NOTICE 'ESTIMATED_COUNT: % < %', sw.estimated_count, _estimated_count;\n RAISE NOTICE 'ESTIMATED_COST: % < %', sw.estimated_cost, _estimated_cost;\n\n -- Do a full count of rows if context is set to on or if auto is set and estimates are low enough\n IF\n _context = 'on'\n OR\n ( _context = 'auto' AND\n (\n sw.estimated_count < _estimated_count\n AND\n sw.estimated_cost < _estimated_cost\n )\n )\n THEN\n t := clock_timestamp();\n RAISE NOTICE 'Calculating actual count...';\n EXECUTE format(\n 'SELECT count(*) FROM items WHERE %s',\n inwhere\n ) INTO sw.total_count;\n i := clock_timestamp() - t;\n RAISE NOTICE 'Actual Count: % -- %', sw.total_count, i;\n sw.time_to_count := extract(epoch FROM i);\n ELSE\n sw.total_count := NULL;\n sw.time_to_count := NULL;\n END IF;\n\n\n INSERT INTO search_wheres\n (_where, lastused, usecount, statslastupdated, estimated_count, estimated_cost, time_to_estimate, partitions, total_count, time_to_count)\n SELECT sw._where, sw.lastused, sw.usecount, sw.statslastupdated, sw.estimated_count, sw.estimated_cost, sw.time_to_estimate, sw.partitions, sw.total_count, sw.time_to_count\n ON CONFLICT ((md5(_where)))\n DO UPDATE\n SET\n lastused = sw.lastused,\n usecount = sw.usecount,\n statslastupdated = sw.statslastupdated,\n estimated_count = sw.estimated_count,\n estimated_cost = sw.estimated_cost,\n time_to_estimate = sw.time_to_estimate,\n total_count = sw.total_count,\n time_to_count = sw.time_to_count\n ;\n RETURN sw;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION search_query(\n _search jsonb = '{}'::jsonb,\n updatestats boolean = false,\n _metadata jsonb = '{}'::jsonb\n) RETURNS searches AS $$\nDECLARE\n search searches%ROWTYPE;\n pexplain jsonb;\n t timestamptz;\n i interval;\nBEGIN\n SELECT * INTO search FROM searches\n WHERE hash=search_hash(_search, _metadata) FOR UPDATE;\n\n -- Calculate the where clause if not already calculated\n IF search._where IS NULL THEN\n search._where := stac_search_to_where(_search);\n END IF;\n\n -- Calculate the order by clause if not already calculated\n IF search.orderby IS NULL THEN\n search.orderby := sort_sqlorderby(_search);\n END IF;\n\n PERFORM where_stats(search._where, updatestats, _search->'conf');\n\n search.lastused := now();\n search.usecount := coalesce(search.usecount, 0) + 1;\n INSERT INTO searches (search, _where, orderby, lastused, usecount, metadata)\n VALUES (_search, search._where, search.orderby, search.lastused, search.usecount, _metadata)\n ON CONFLICT (hash) DO\n UPDATE SET\n _where = EXCLUDED._where,\n orderby = EXCLUDED.orderby,\n lastused = EXCLUDED.lastused,\n usecount = EXCLUDED.usecount,\n metadata = EXCLUDED.metadata\n RETURNING * INTO search\n ;\n RETURN search;\n\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE OR REPLACE FUNCTION search(_search jsonb = '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n searches searches%ROWTYPE;\n _where text;\n token_where text;\n full_where text;\n orderby text;\n query text;\n token_type text := substr(_search->>'token',1,4);\n _limit int := coalesce((_search->>'limit')::int, 10);\n curs refcursor;\n cntr int := 0;\n iter_record items%ROWTYPE;\n first_record jsonb;\n first_item items%ROWTYPE;\n last_item items%ROWTYPE;\n last_record jsonb;\n out_records jsonb := '[]'::jsonb;\n prev_query text;\n next text;\n prev_id text;\n has_next boolean := false;\n has_prev boolean := false;\n prev text;\n total_count bigint;\n context jsonb;\n collection jsonb;\n includes text[];\n excludes text[];\n exit_flag boolean := FALSE;\n batches int := 0;\n timer timestamptz := clock_timestamp();\n pstart timestamptz;\n pend timestamptz;\n pcurs refcursor;\n search_where search_wheres%ROWTYPE;\n id text;\nBEGIN\nCREATE TEMP TABLE results (i int GENERATED ALWAYS AS IDENTITY, content jsonb) ON COMMIT DROP;\n searches := search_query(_search);\n _where := searches._where;\n orderby := searches.orderby;\n search_where := where_stats(_where);\n total_count := coalesce(search_where.total_count, search_where.estimated_count);\n\n IF token_type='prev' THEN\n token_where := get_token_filter(_search, null::jsonb);\n orderby := sort_sqlorderby(_search, TRUE);\n END IF;\n IF token_type='next' THEN\n token_where := get_token_filter(_search, null::jsonb);\n END IF;\n\n full_where := concat_ws(' AND ', _where, token_where);\n RAISE NOTICE 'FULL QUERY % %', full_where, clock_timestamp()-timer;\n timer := clock_timestamp();\n\n FOR query IN SELECT partition_queries(full_where, orderby, search_where.partitions) LOOP\n timer := clock_timestamp();\n query := format('%s LIMIT %s', query, _limit + 1);\n RAISE NOTICE 'Partition Query: %', query;\n batches := batches + 1;\n -- curs = create_cursor(query);\n RAISE NOTICE 'cursor_tuple_fraction: %', current_setting('cursor_tuple_fraction');\n OPEN curs FOR EXECUTE query;\n LOOP\n FETCH curs into iter_record;\n EXIT WHEN NOT FOUND;\n cntr := cntr + 1;\n\n IF _search->'conf'->>'nohydrate' IS NOT NULL AND (_search->'conf'->>'nohydrate')::boolean = true THEN\n last_record := content_nonhydrated(iter_record, _search->'fields');\n ELSE\n last_record := content_hydrate(iter_record, _search->'fields');\n END IF;\n last_item := iter_record;\n IF cntr = 1 THEN\n first_item := last_item;\n first_record := last_record;\n END IF;\n IF cntr <= _limit THEN\n INSERT INTO results (content) VALUES (last_record);\n ELSIF cntr > _limit THEN\n has_next := true;\n exit_flag := true;\n EXIT;\n END IF;\n END LOOP;\n CLOSE curs;\n RAISE NOTICE 'Query took %.', clock_timestamp()-timer;\n timer := clock_timestamp();\n EXIT WHEN exit_flag;\n END LOOP;\n RAISE NOTICE 'Scanned through % partitions.', batches;\n\nWITH ordered AS (SELECT * FROM results WHERE content IS NOT NULL ORDER BY i)\nSELECT jsonb_agg(content) INTO out_records FROM ordered;\n\nDROP TABLE results;\n\n\n-- Flip things around if this was the result of a prev token query\nIF token_type='prev' THEN\n out_records := flip_jsonb_array(out_records);\n first_item := last_item;\n first_record := last_record;\nEND IF;\n\n-- If this query has a token, see if there is data before the first record\nIF _search ? 'token' THEN\n prev_query := format(\n 'SELECT 1 FROM items WHERE %s LIMIT 1',\n concat_ws(\n ' AND ',\n _where,\n trim(get_token_filter(_search, to_jsonb(first_item)))\n )\n );\n RAISE NOTICE 'Query to get previous record: % --- %', prev_query, first_record;\n EXECUTE prev_query INTO has_prev;\n IF FOUND and has_prev IS NOT NULL THEN\n RAISE NOTICE 'Query results from prev query: %', has_prev;\n has_prev := TRUE;\n END IF;\nEND IF;\nhas_prev := COALESCE(has_prev, FALSE);\n\nIF has_prev THEN\n prev := out_records->0->>'id';\nEND IF;\nIF has_next OR token_type='prev' THEN\n next := out_records->-1->>'id';\nEND IF;\n\nIF context(_search->'conf') != 'off' THEN\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'matched', total_count,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\nELSE\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\nEND IF;\n\ncollection := jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb),\n 'next', next,\n 'prev', prev,\n 'context', context\n);\n\nRETURN collection;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac, public SET cursor_tuple_fraction TO 1;\n\n\nCREATE OR REPLACE FUNCTION search_cursor(_search jsonb = '{}'::jsonb) RETURNS refcursor AS $$\nDECLARE\n curs refcursor;\n searches searches%ROWTYPE;\n _where text;\n _orderby text;\n q text;\n\nBEGIN\n searches := search_query(_search);\n _where := searches._where;\n _orderby := searches.orderby;\n\n OPEN curs FOR\n WITH p AS (\n SELECT * FROM partition_queries(_where, _orderby) p\n )\n SELECT\n CASE WHEN EXISTS (SELECT 1 FROM p) THEN\n (SELECT format($q$\n SELECT * FROM (\n %s\n ) total\n $q$,\n string_agg(\n format($q$ SELECT * FROM ( %s ) AS sub $q$, p),\n '\n UNION ALL\n '\n )\n ))\n ELSE NULL\n END FROM p;\n RETURN curs;\nEND;\n$$ LANGUAGE PLPGSQL;\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION tileenvelope(zoom int, x int, y int) RETURNS geometry AS $$\nWITH t AS (\n SELECT\n 20037508.3427892 as merc_max,\n -20037508.3427892 as merc_min,\n (2 * 20037508.3427892) / (2 ^ zoom) as tile_size\n)\nSELECT st_makeenvelope(\n merc_min + (tile_size * x),\n merc_max - (tile_size * (y + 1)),\n merc_min + (tile_size * (x + 1)),\n merc_max - (tile_size * y),\n 3857\n) FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;DROP FUNCTION IF EXISTS mercgrid;\n\n\nCREATE OR REPLACE FUNCTION ftime() RETURNS interval as $$\nSELECT age(clock_timestamp(), transaction_timestamp());\n$$ LANGUAGE SQL;\nSET SEARCH_PATH to pgstac, public;\n\nDROP FUNCTION IF EXISTS geometrysearch;\nCREATE OR REPLACE FUNCTION geometrysearch(\n IN geom geometry,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE, -- Return as soon as the passed in geometry is full covered\n IN skipcovered boolean DEFAULT TRUE -- Skip any items that would show up completely under the previous items\n) RETURNS jsonb AS $$\nDECLARE\n search searches%ROWTYPE;\n curs refcursor;\n _where text;\n query text;\n iter_record items%ROWTYPE;\n out_records jsonb := '{}'::jsonb[];\n exit_flag boolean := FALSE;\n counter int := 1;\n scancounter int := 1;\n remaining_limit int := _scanlimit;\n tilearea float;\n unionedgeom geometry;\n clippedgeom geometry;\n unionedgeom_area float := 0;\n prev_area float := 0;\n excludes text[];\n includes text[];\n\nBEGIN\n DROP TABLE IF EXISTS pgstac_results;\n CREATE TEMP TABLE pgstac_results (content jsonb) ON COMMIT DROP;\n\n -- If skipcovered is true then you will always want to exit when the passed in geometry is full\n IF skipcovered THEN\n exitwhenfull := TRUE;\n END IF;\n\n SELECT * INTO search FROM searches WHERE hash=queryhash;\n\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Search with Query Hash % Not Found', queryhash;\n END IF;\n\n tilearea := st_area(geom);\n _where := format('%s AND st_intersects(geometry, %L::geometry)', search._where, geom);\n\n\n FOR query IN SELECT * FROM partition_queries(_where, search.orderby) LOOP\n query := format('%s LIMIT %L', query, remaining_limit);\n RAISE NOTICE '%', query;\n OPEN curs FOR EXECUTE query;\n LOOP\n FETCH curs INTO iter_record;\n EXIT WHEN NOT FOUND;\n IF exitwhenfull OR skipcovered THEN -- If we are not using exitwhenfull or skipcovered, we do not need to do expensive geometry operations\n clippedgeom := st_intersection(geom, iter_record.geometry);\n\n IF unionedgeom IS NULL THEN\n unionedgeom := clippedgeom;\n ELSE\n unionedgeom := st_union(unionedgeom, clippedgeom);\n END IF;\n\n unionedgeom_area := st_area(unionedgeom);\n\n IF skipcovered AND prev_area = unionedgeom_area THEN\n scancounter := scancounter + 1;\n CONTINUE;\n END IF;\n\n prev_area := unionedgeom_area;\n\n RAISE NOTICE '% % % %', unionedgeom_area/tilearea, counter, scancounter, ftime();\n END IF;\n RAISE NOTICE '% %', iter_record, content_hydrate(iter_record, fields);\n INSERT INTO pgstac_results (content) VALUES (content_hydrate(iter_record, fields));\n\n IF counter >= _limit\n OR scancounter > _scanlimit\n OR ftime() > _timelimit\n OR (exitwhenfull AND unionedgeom_area >= tilearea)\n THEN\n exit_flag := TRUE;\n EXIT;\n END IF;\n counter := counter + 1;\n scancounter := scancounter + 1;\n\n END LOOP;\n CLOSE curs;\n EXIT WHEN exit_flag;\n remaining_limit := _scanlimit - scancounter;\n END LOOP;\n\n SELECT jsonb_agg(content) INTO out_records FROM pgstac_results WHERE content IS NOT NULL;\n\n RETURN jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb)\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDROP FUNCTION IF EXISTS geojsonsearch;\nCREATE OR REPLACE FUNCTION geojsonsearch(\n IN geojson jsonb,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_geomfromgeojson(geojson),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS xyzsearch;\nCREATE OR REPLACE FUNCTION xyzsearch(\n IN _x int,\n IN _y int,\n IN _z int,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_transform(tileenvelope(_z, _x, _y), 4326),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE PROCEDURE analyze_items() AS $$\nDECLARE\n q text;\n timeout_ts timestamptz;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n SELECT format('ANALYZE (VERBOSE, SKIP_LOCKED) %I;', relname) INTO q\n FROM pg_stat_user_tables\n WHERE relname like '_item%' AND (n_mod_since_analyze>0 OR last_analyze IS NULL) LIMIT 1;\n IF NOT FOUND THEN\n EXIT;\n END IF;\n RAISE NOTICE '%', q;\n EXECUTE q;\n COMMIT;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE PROCEDURE validate_constraints() AS $$\nDECLARE\n q text;\nBEGIN\n FOR q IN\n SELECT\n FORMAT(\n 'ALTER TABLE %I.%I VALIDATE CONSTRAINT %I;',\n nsp.nspname,\n cls.relname,\n con.conname\n )\n\n FROM pg_constraint AS con\n JOIN pg_class AS cls\n ON con.conrelid = cls.oid\n JOIN pg_namespace AS nsp\n ON cls.relnamespace = nsp.oid\n WHERE convalidated = FALSE AND contype in ('c','f')\n AND nsp.nspname = 'pgstac'\n LOOP\n RAISE NOTICE '%', q;\n PERFORM run_or_queue(q);\n COMMIT;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION collection_extent(_collection text, runupdate boolean default false) RETURNS jsonb AS $$\nDECLARE\n geom_extent geometry;\n mind timestamptz;\n maxd timestamptz;\n extent jsonb;\nBEGIN\n IF runupdate THEN\n PERFORM update_partition_stats_q(partition)\n FROM partitions WHERE collection=_collection;\n END IF;\n SELECT\n min(lower(dtrange)),\n max(upper(edtrange)),\n st_extent(spatial)\n INTO\n mind,\n maxd,\n geom_extent\n FROM partitions\n WHERE collection=_collection;\n\n IF geom_extent IS NOT NULL AND mind IS NOT NULL AND maxd IS NOT NULL THEN\n extent := jsonb_build_object(\n 'extent', jsonb_build_object(\n 'spatial', jsonb_build_object(\n 'bbox', to_jsonb(array[array[st_xmin(geom_extent), st_ymin(geom_extent), st_xmax(geom_extent), st_ymax(geom_extent)]])\n ),\n 'temporal', jsonb_build_object(\n 'interval', to_jsonb(array[array[mind, maxd]])\n )\n )\n );\n RETURN extent;\n END IF;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('id', '{\"title\": \"Item ID\",\"description\": \"Item identifier\",\"$ref\": \"https://schemas.stacspec.org/v1.0.0/item-spec/json-schema/item.json#/definitions/core/allOf/2/properties/id\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('geometry', '{\"title\": \"Item Geometry\",\"description\": \"Item Geometry\",\"$ref\": \"https://geojson.org/schema/Feature.json\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('datetime','{\"description\": \"Datetime\",\"type\": \"string\",\"title\": \"Acquired\",\"format\": \"date-time\",\"pattern\": \"(\\\\+00:00|Z)$\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('eo:cloud_cover','{\"$ref\": \"https://stac-extensions.github.io/eo/v1.0.0/schema.json#/definitions/fieldsproperties/eo:cloud_cover\"}','to_int','BTREE');\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDELETE FROM queryables a USING queryables b\n WHERE a.name = b.name AND a.collection_ids IS NOT DISTINCT FROM b.collection_ids AND a.id > b.id;\n\n\nINSERT INTO pgstac_settings (name, value) VALUES\n ('context', 'off'),\n ('context_estimated_count', '100000'),\n ('context_estimated_cost', '100000'),\n ('context_stats_ttl', '1 day'),\n ('default_filter_lang', 'cql2-json'),\n ('additional_properties', 'true'),\n ('use_queue', 'false'),\n ('queue_timeout', '10 minutes'),\n ('update_collection_extent', 'false')\nON CONFLICT DO NOTHING\n;\n\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nGRANT ALL ON SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac to pgstac_admin;\n\n-- pgstac_read role limited to using function apis\nGRANT EXECUTE ON FUNCTION search TO pgstac_read;\nGRANT EXECUTE ON FUNCTION search_query TO pgstac_read;\nGRANT EXECUTE ON FUNCTION item_by_id TO pgstac_read;\nGRANT EXECUTE ON FUNCTION get_item TO pgstac_read;\n\nGRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_ingest;\nGRANT USAGE ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_ingest;\n\nSELECT update_partition_stats_q(partition) FROM partitions;\nSELECT set_version('0.7.1');\n" }, { "path": "src/pgstac/migrations/pgstac.0.7.10-0.8.0.sql", "content": "SET client_min_messages TO WARNING;\nSET SEARCH_PATH to pgstac, public;\nRESET ROLE;\nDO $$\nDECLARE\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='postgis') THEN\n CREATE EXTENSION IF NOT EXISTS postgis;\n END IF;\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='btree_gist') THEN\n CREATE EXTENSION IF NOT EXISTS btree_gist;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_admin;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_read;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_ingest;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n\nGRANT pgstac_admin TO current_user;\n\n-- Function to make sure pgstac_admin is the owner of items\nCREATE OR REPLACE FUNCTION pgstac_admin_owns() RETURNS VOID AS $$\nDECLARE\n f RECORD;\nBEGIN\n FOR f IN (\n SELECT\n concat(\n oid::regproc::text,\n '(',\n coalesce(pg_get_function_identity_arguments(oid),''),\n ')'\n ) AS name,\n CASE prokind WHEN 'f' THEN 'FUNCTION' WHEN 'p' THEN 'PROCEDURE' WHEN 'a' THEN 'AGGREGATE' END as typ\n FROM pg_proc\n WHERE\n pronamespace=to_regnamespace('pgstac')\n AND proowner != to_regrole('pgstac_admin')\n AND proname NOT LIKE 'pg_stat%'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n FOR f IN (\n SELECT\n oid::regclass::text as name,\n CASE relkind\n WHEN 'i' THEN 'INDEX'\n WHEN 'I' THEN 'INDEX'\n WHEN 'p' THEN 'TABLE'\n WHEN 'r' THEN 'TABLE'\n WHEN 'v' THEN 'VIEW'\n WHEN 'S' THEN 'SEQUENCE'\n ELSE NULL\n END as typ\n FROM pg_class\n WHERE relnamespace=to_regnamespace('pgstac') and relowner != to_regrole('pgstac_admin') AND relkind IN ('r','p','v','S') AND relname NOT LIKE 'pg_stat'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\nSELECT pgstac_admin_owns();\n\nCREATE SCHEMA IF NOT EXISTS pgstac AUTHORIZATION pgstac_admin;\n\nGRANT ALL ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_admin;\n\nALTER ROLE pgstac_admin SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_read SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_ingest SET SEARCH_PATH TO pgstac, public;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\n\nGRANT pgstac_read TO pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\n\nSET ROLE pgstac_admin;\n\nSET SEARCH_PATH TO pgstac, public;\n\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS analyze_items;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS validate_constraints;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n-- Install these idempotently as migrations do not put them before trying to modify the collections table\n\n\nCREATE OR REPLACE FUNCTION collection_geom(content jsonb)\nRETURNS geometry AS $$\n WITH box AS (SELECT content->'extent'->'spatial'->'bbox'->0 as box)\n SELECT\n st_makeenvelope(\n (box->>0)::float,\n (box->>1)::float,\n (box->>2)::float,\n (box->>3)::float,\n 4326\n )\n FROM box;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION collection_datetime(content jsonb)\nRETURNS timestamptz AS $$\n SELECT\n CASE\n WHEN\n (content->'extent'->'temporal'->'interval'->0->>0) IS NULL\n THEN '-infinity'::timestamptz\n ELSE\n (content->'extent'->'temporal'->'interval'->0->>0)::timestamptz\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION collection_enddatetime(content jsonb)\nRETURNS timestamptz AS $$\n SELECT\n CASE\n WHEN\n (content->'extent'->'temporal'->'interval'->0->>1) IS NULL\n THEN 'infinity'::timestamptz\n ELSE\n (content->'extent'->'temporal'->'interval'->0->>1)::timestamptz\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n-- BEGIN migra calculated SQL\nalter table \"pgstac\".\"collections\" add column \"datetime\" timestamp with time zone generated always as (collection_datetime(content)) stored;\n\nalter table \"pgstac\".\"collections\" add column \"end_datetime\" timestamp with time zone generated always as (collection_enddatetime(content)) stored;\n\nalter table \"pgstac\".\"collections\" add column \"geometry\" geometry generated always as (collection_geom(content)) stored;\n\nalter table \"pgstac\".\"collections\" add column \"private\" jsonb;\n\nalter table \"pgstac\".\"items\" add column \"private\" jsonb;\n\nset check_function_bodies = off;\n\nCREATE OR REPLACE FUNCTION pgstac.collection_datetime(content jsonb)\n RETURNS timestamp with time zone\n LANGUAGE sql\n IMMUTABLE STRICT\nAS $function$\n SELECT\n CASE\n WHEN\n (content->'extent'->'temporal'->'interval'->0->>0) IS NULL\n THEN '-infinity'::timestamptz\n ELSE\n (content->'extent'->'temporal'->'interval'->0->>0)::timestamptz\n END\n ;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.collection_enddatetime(content jsonb)\n RETURNS timestamp with time zone\n LANGUAGE sql\n IMMUTABLE STRICT\nAS $function$\n SELECT\n CASE\n WHEN\n (content->'extent'->'temporal'->'interval'->0->>1) IS NULL\n THEN 'infinity'::timestamptz\n ELSE\n (content->'extent'->'temporal'->'interval'->0->>1)::timestamptz\n END\n ;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.collection_geom(content jsonb)\n RETURNS geometry\n LANGUAGE sql\n IMMUTABLE STRICT\nAS $function$\n WITH box AS (SELECT content->'extent'->'spatial'->'bbox'->0 as box)\n SELECT\n st_makeenvelope(\n (box->>0)::float,\n (box->>1)::float,\n (box->>2)::float,\n (box->>3)::float,\n 4326\n )\n FROM box;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.content_dehydrate(content jsonb)\n RETURNS items\n LANGUAGE sql\n STABLE\nAS $function$\n SELECT\n content->>'id' as id,\n stac_geom(content) as geometry,\n content->>'collection' as collection,\n stac_datetime(content) as datetime,\n stac_end_datetime(content) as end_datetime,\n content_slim(content) as content,\n null::jsonb as private\n ;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.search(_search jsonb DEFAULT '{}'::jsonb)\n RETURNS jsonb\n LANGUAGE plpgsql\nAS $function$\nDECLARE\n searches searches%ROWTYPE;\n _where text;\n orderby text;\n search_where search_wheres%ROWTYPE;\n total_count bigint;\n token record;\n token_prev boolean;\n token_item items%ROWTYPE;\n token_where text;\n full_where text;\n init_ts timestamptz := clock_timestamp();\n timer timestamptz := clock_timestamp();\n hydrate bool := NOT (_search->'conf'->>'nohydrate' IS NOT NULL AND (_search->'conf'->>'nohydrate')::boolean = true);\n prev text;\n next text;\n context jsonb;\n collection jsonb;\n out_records jsonb;\n out_len int;\n _limit int := coalesce((_search->>'limit')::int, 10);\n _querylimit int;\n _fields jsonb := coalesce(_search->'fields', '{}'::jsonb);\n has_prev boolean := FALSE;\n has_next boolean := FALSE;\nBEGIN\n searches := search_query(_search);\n _where := searches._where;\n orderby := searches.orderby;\n search_where := where_stats(_where);\n total_count := coalesce(search_where.total_count, search_where.estimated_count);\n RAISE NOTICE 'SEARCH:TOKEN: %', _search->>'token';\n token := get_token_record(_search->>'token');\n RAISE NOTICE '***TOKEN: %', token;\n _querylimit := _limit + 1;\n IF token IS NOT NULL THEN\n token_prev := token.prev;\n token_item := token.item;\n token_where := get_token_filter(_search->'sortby', token_item, token_prev, FALSE);\n RAISE LOG 'TOKEN_WHERE: % (%ms from search start)', token_where, age_ms(timer);\n IF token_prev THEN -- if we are using a prev token, we know has_next is true\n RAISE LOG 'There is a previous token, so automatically setting has_next to true';\n has_next := TRUE;\n orderby := sort_sqlorderby(_search, TRUE);\n ELSE\n RAISE LOG 'There is a next token, so automatically setting has_prev to true';\n has_prev := TRUE;\n\n END IF;\n ELSE -- if there was no token, we know there is no prev\n RAISE LOG 'There is no token, so we know there is no prev. setting has_prev to false';\n has_prev := FALSE;\n END IF;\n\n full_where := concat_ws(' AND ', _where, token_where);\n RAISE NOTICE 'FULL WHERE CLAUSE: %', full_where;\n RAISE NOTICE 'Time to get counts and build query %', age_ms(timer);\n timer := clock_timestamp();\n\n IF hydrate THEN\n RAISE NOTICE 'Getting hydrated data.';\n ELSE\n RAISE NOTICE 'Getting non-hydrated data.';\n END IF;\n RAISE NOTICE 'CACHE SET TO %', get_setting_bool('format_cache');\n RAISE NOTICE 'Time to set hydration/formatting %', age_ms(timer);\n timer := clock_timestamp();\n SELECT jsonb_agg(format_item(i, _fields, hydrate)) INTO out_records\n FROM search_rows(\n full_where,\n orderby,\n search_where.partitions,\n _querylimit\n ) as i;\n\n RAISE NOTICE 'Time to fetch rows %', age_ms(timer);\n timer := clock_timestamp();\n\n\n IF token_prev THEN\n out_records := flip_jsonb_array(out_records);\n END IF;\n\n RAISE NOTICE 'Query returned % records.', jsonb_array_length(out_records);\n RAISE LOG 'TOKEN: % %', token_item.id, token_item.collection;\n RAISE LOG 'RECORD_1: % %', out_records->0->>'id', out_records->0->>'collection';\n RAISE LOG 'RECORD-1: % %', out_records->-1->>'id', out_records->-1->>'collection';\n\n -- REMOVE records that were from our token\n IF out_records->0->>'id' = token_item.id AND out_records->0->>'collection' = token_item.collection THEN\n out_records := out_records - 0;\n ELSIF out_records->-1->>'id' = token_item.id AND out_records->-1->>'collection' = token_item.collection THEN\n out_records := out_records - -1;\n END IF;\n\n out_len := jsonb_array_length(out_records);\n\n IF out_len = _limit + 1 THEN\n IF token_prev THEN\n has_prev := TRUE;\n out_records := out_records - 0;\n ELSE\n has_next := TRUE;\n out_records := out_records - -1;\n END IF;\n END IF;\n\n IF has_next THEN\n next := concat(out_records->-1->>'collection', ':', out_records->-1->>'id');\n RAISE NOTICE 'HAS NEXT | %', next;\n END IF;\n\n IF has_prev THEN\n prev := concat(out_records->0->>'collection', ':', out_records->0->>'id');\n RAISE NOTICE 'HAS PREV | %', prev;\n END IF;\n\n RAISE NOTICE 'Time to get prev/next %', age_ms(timer);\n timer := clock_timestamp();\n\n IF context(_search->'conf') != 'off' THEN\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'matched', total_count,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\n ELSE\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\n END IF;\n\n collection := jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb),\n 'next', next,\n 'prev', prev,\n 'context', context\n );\n\n IF get_setting_bool('timing', _search->'conf') THEN\n collection = collection || jsonb_build_object('timing', age_ms(init_ts));\n END IF;\n\n RAISE NOTICE 'Time to build final json %', age_ms(timer);\n timer := clock_timestamp();\n\n RAISE NOTICE 'Total Time: %', age_ms(current_timestamp);\n RAISE NOTICE 'RETURNING % records. NEXT: %. PREV: %', collection->'context'->>'returned', collection->>'next', collection->>'prev';\n RETURN collection;\nEND;\n$function$\n;\n\n\n-- END migra calculated SQL\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('id', '{\"title\": \"Item ID\",\"description\": \"Item identifier\",\"$ref\": \"https://schemas.stacspec.org/v1.0.0/item-spec/json-schema/item.json#/definitions/core/allOf/2/properties/id\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('geometry', '{\"title\": \"Item Geometry\",\"description\": \"Item Geometry\",\"$ref\": \"https://geojson.org/schema/Feature.json\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('datetime','{\"description\": \"Datetime\",\"type\": \"string\",\"title\": \"Acquired\",\"format\": \"date-time\",\"pattern\": \"(\\\\+00:00|Z)$\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDELETE FROM queryables a USING queryables b\n WHERE a.name = b.name AND a.collection_ids IS NOT DISTINCT FROM b.collection_ids AND a.id > b.id;\n\n\nINSERT INTO pgstac_settings (name, value) VALUES\n ('context', 'off'),\n ('context_estimated_count', '100000'),\n ('context_estimated_cost', '100000'),\n ('context_stats_ttl', '1 day'),\n ('default_filter_lang', 'cql2-json'),\n ('additional_properties', 'true'),\n ('use_queue', 'false'),\n ('queue_timeout', '10 minutes'),\n ('update_collection_extent', 'false'),\n ('format_cache', 'false')\nON CONFLICT DO NOTHING\n;\n\nALTER FUNCTION to_text COST 5000;\nALTER FUNCTION to_float COST 5000;\nALTER FUNCTION to_int COST 5000;\nALTER FUNCTION to_tstz COST 5000;\nALTER FUNCTION to_text_array COST 5000;\n\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nGRANT ALL ON SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac to pgstac_admin;\n\n-- pgstac_read role limited to using function apis\nGRANT EXECUTE ON FUNCTION search TO pgstac_read;\nGRANT EXECUTE ON FUNCTION search_query TO pgstac_read;\nGRANT EXECUTE ON FUNCTION item_by_id TO pgstac_read;\nGRANT EXECUTE ON FUNCTION get_item TO pgstac_read;\nGRANT SELECT ON ALL TABLES IN SCHEMA pgstac TO pgstac_read;\n\n\nGRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_ingest;\nGRANT USAGE ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_ingest;\n\nSELECT update_partition_stats_q(partition) FROM partitions_view;\nSELECT set_version('0.8.0');\n" }, { "path": "src/pgstac/migrations/pgstac.0.7.10.sql", "content": "RESET ROLE;\nDO $$\nDECLARE\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='postgis') THEN\n CREATE EXTENSION IF NOT EXISTS postgis;\n END IF;\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='btree_gist') THEN\n CREATE EXTENSION IF NOT EXISTS btree_gist;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_admin;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_read;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_ingest;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n\nGRANT pgstac_admin TO current_user;\n\n-- Function to make sure pgstac_admin is the owner of items\nCREATE OR REPLACE FUNCTION pgstac_admin_owns() RETURNS VOID AS $$\nDECLARE\n f RECORD;\nBEGIN\n FOR f IN (\n SELECT\n concat(\n oid::regproc::text,\n '(',\n coalesce(pg_get_function_identity_arguments(oid),''),\n ')'\n ) AS name,\n CASE prokind WHEN 'f' THEN 'FUNCTION' WHEN 'p' THEN 'PROCEDURE' WHEN 'a' THEN 'AGGREGATE' END as typ\n FROM pg_proc\n WHERE\n pronamespace=to_regnamespace('pgstac')\n AND proowner != to_regrole('pgstac_admin')\n AND proname NOT LIKE 'pg_stat%'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n FOR f IN (\n SELECT\n oid::regclass::text as name,\n CASE relkind\n WHEN 'i' THEN 'INDEX'\n WHEN 'I' THEN 'INDEX'\n WHEN 'p' THEN 'TABLE'\n WHEN 'r' THEN 'TABLE'\n WHEN 'v' THEN 'VIEW'\n WHEN 'S' THEN 'SEQUENCE'\n ELSE NULL\n END as typ\n FROM pg_class\n WHERE relnamespace=to_regnamespace('pgstac') and relowner != to_regrole('pgstac_admin') AND relkind IN ('r','p','v','S') AND relname NOT LIKE 'pg_stat'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\nSELECT pgstac_admin_owns();\n\nCREATE SCHEMA IF NOT EXISTS pgstac AUTHORIZATION pgstac_admin;\n\nGRANT ALL ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_admin;\n\nALTER ROLE pgstac_admin SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_read SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_ingest SET SEARCH_PATH TO pgstac, public;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\n\nGRANT pgstac_read TO pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\n\nSET ROLE pgstac_admin;\n\nSET SEARCH_PATH TO pgstac, public;\n\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS analyze_items;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS validate_constraints;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\nCREATE OR REPLACE FUNCTION to_int(jsonb) RETURNS int AS $$\n SELECT floor(($1->>0)::float)::int;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION to_float(jsonb) RETURNS float AS $$\n SELECT ($1->>0)::float;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION to_tstz(jsonb) RETURNS timestamptz AS $$\n SELECT ($1->>0)::timestamptz;\n$$ LANGUAGE SQL IMMUTABLE STRICT SET TIME ZONE 'UTC' COST 5000 PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION to_text(jsonb) RETURNS text AS $$\n SELECT CASE WHEN jsonb_typeof($1) IN ('array','object') THEN $1::text ELSE $1->>0 END;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION to_text_array(jsonb) RETURNS text[] AS $$\n SELECT\n CASE jsonb_typeof($1)\n WHEN 'array' THEN ARRAY(SELECT jsonb_array_elements_text($1))\n ELSE ARRAY[$1->>0]\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION bbox_geom(_bbox jsonb) RETURNS geometry AS $$\nSELECT CASE jsonb_array_length(_bbox)\n WHEN 4 THEN\n ST_SetSRID(ST_MakeEnvelope(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float,\n (_bbox->>3)::float\n ),4326)\n WHEN 6 THEN\n ST_SetSRID(ST_3DMakeBox(\n ST_MakePoint(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float\n ),\n ST_MakePoint(\n (_bbox->>3)::float,\n (_bbox->>4)::float,\n (_bbox->>5)::float\n )\n ),4326)\n ELSE null END;\n;\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION geom_bbox(_geom geometry) RETURNS jsonb AS $$\n SELECT jsonb_build_array(\n st_xmin(_geom),\n st_ymin(_geom),\n st_xmax(_geom),\n st_ymax(_geom)\n );\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION flip_jsonb_array(j jsonb) RETURNS jsonb AS $$\n SELECT jsonb_agg(value) FROM (SELECT value FROM jsonb_array_elements(j) WITH ORDINALITY ORDER BY ordinality DESC) as t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION explode_dotpaths(j jsonb) RETURNS SETOF text[] AS $$\n SELECT string_to_array(p, '.') as e FROM jsonb_array_elements_text(j) p;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION explode_dotpaths_recurse(IN j jsonb) RETURNS SETOF text[] AS $$\n WITH RECURSIVE t AS (\n SELECT e FROM explode_dotpaths(j) e\n UNION ALL\n SELECT e[1:cardinality(e)-1]\n FROM t\n WHERE cardinality(e)>1\n ) SELECT e FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION jsonb_set_nested(j jsonb, path text[], val jsonb) RETURNS jsonb AS $$\nDECLARE\nBEGIN\n IF cardinality(path) > 1 THEN\n FOR i IN 1..(cardinality(path)-1) LOOP\n IF j #> path[:i] IS NULL THEN\n j := jsonb_set_lax(j, path[:i], '{}', TRUE);\n END IF;\n END LOOP;\n END IF;\n RETURN jsonb_set_lax(j, path, val, true);\n\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\n\n\nCREATE OR REPLACE FUNCTION jsonb_include(j jsonb, f jsonb) RETURNS jsonb AS $$\nDECLARE\n includes jsonb := f-> 'include';\n outj jsonb := '{}'::jsonb;\n path text[];\nBEGIN\n IF\n includes IS NULL\n OR jsonb_array_length(includes) = 0\n THEN\n RETURN j;\n ELSE\n includes := includes || '[\"id\",\"collection\"]'::jsonb;\n FOR path IN SELECT explode_dotpaths(includes) LOOP\n outj := jsonb_set_nested(outj, path, j #> path);\n END LOOP;\n END IF;\n RETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_exclude(j jsonb, f jsonb) RETURNS jsonb AS $$\nDECLARE\n excludes jsonb := f-> 'exclude';\n outj jsonb := j;\n path text[];\nBEGIN\n IF\n excludes IS NULL\n OR jsonb_array_length(excludes) = 0\n THEN\n RETURN j;\n ELSE\n FOR path IN SELECT explode_dotpaths(excludes) LOOP\n outj := outj #- path;\n END LOOP;\n END IF;\n RETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_fields(j jsonb, f jsonb DEFAULT '{\"fields\":[]}') RETURNS jsonb AS $$\n SELECT jsonb_exclude(jsonb_include(j, f), f);\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION merge_jsonb(_a jsonb, _b jsonb) RETURNS jsonb AS $$\n SELECT\n CASE\n WHEN _a = '\"𒍟※\"'::jsonb THEN NULL\n WHEN _a IS NULL OR jsonb_typeof(_a) = 'null' THEN _b\n WHEN jsonb_typeof(_a) = 'object' AND jsonb_typeof(_b) = 'object' THEN\n (\n SELECT\n jsonb_strip_nulls(\n jsonb_object_agg(\n key,\n merge_jsonb(a.value, b.value)\n )\n )\n FROM\n jsonb_each(coalesce(_a,'{}'::jsonb)) as a\n FULL JOIN\n jsonb_each(coalesce(_b,'{}'::jsonb)) as b\n USING (key)\n )\n WHEN\n jsonb_typeof(_a) = 'array'\n AND jsonb_typeof(_b) = 'array'\n AND jsonb_array_length(_a) = jsonb_array_length(_b)\n THEN\n (\n SELECT jsonb_agg(m) FROM\n ( SELECT\n merge_jsonb(\n jsonb_array_elements(_a),\n jsonb_array_elements(_b)\n ) as m\n ) as l\n )\n ELSE _a\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION strip_jsonb(_a jsonb, _b jsonb) RETURNS jsonb AS $$\n SELECT\n CASE\n\n WHEN (_a IS NULL OR jsonb_typeof(_a) = 'null') AND _b IS NOT NULL AND jsonb_typeof(_b) != 'null' THEN '\"𒍟※\"'::jsonb\n WHEN _b IS NULL OR jsonb_typeof(_a) = 'null' THEN _a\n WHEN _a = _b AND jsonb_typeof(_a) = 'object' THEN '{}'::jsonb\n WHEN _a = _b THEN NULL\n WHEN jsonb_typeof(_a) = 'object' AND jsonb_typeof(_b) = 'object' THEN\n (\n SELECT\n jsonb_strip_nulls(\n jsonb_object_agg(\n key,\n strip_jsonb(a.value, b.value)\n )\n )\n FROM\n jsonb_each(_a) as a\n FULL JOIN\n jsonb_each(_b) as b\n USING (key)\n )\n WHEN\n jsonb_typeof(_a) = 'array'\n AND jsonb_typeof(_b) = 'array'\n AND jsonb_array_length(_a) = jsonb_array_length(_b)\n THEN\n (\n SELECT jsonb_agg(m) FROM\n ( SELECT\n strip_jsonb(\n jsonb_array_elements(_a),\n jsonb_array_elements(_b)\n ) as m\n ) as l\n )\n ELSE _a\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION nullif_jsonbnullempty(j jsonb) RETURNS jsonb AS $$\n SELECT nullif(nullif(nullif(j,'null'::jsonb),'{}'::jsonb),'[]'::jsonb);\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION jsonb_array_unique(j jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(jsonb_agg(DISTINCT a)) v FROM jsonb_array_elements(j) a;\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_concat_ignorenull(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT coalesce(a,'[]'::jsonb) || coalesce(b,'[]'::jsonb);\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_least(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(least(nullif_jsonbnullempty(a), nullif_jsonbnullempty(b)));\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_greatest(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(greatest(a, b));\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION first_notnull_sfunc(anyelement, anyelement) RETURNS anyelement AS $$\n SELECT COALESCE($1,$2);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE AGGREGATE first_notnull(anyelement)(\n SFUNC = first_notnull_sfunc,\n STYPE = anyelement\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_array_unique_merge(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_concat_ignorenull,\n FINALFUNC = jsonb_array_unique\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_min(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_least\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_max(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_greatest\n);\n\nCREATE TABLE IF NOT EXISTS migrations (\n version text PRIMARY KEY,\n datetime timestamptz DEFAULT clock_timestamp() NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION get_version() RETURNS text AS $$\n SELECT version FROM pgstac.migrations ORDER BY datetime DESC, version DESC LIMIT 1;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION set_version(text) RETURNS text AS $$\n INSERT INTO pgstac.migrations (version) VALUES ($1)\n ON CONFLICT DO NOTHING\n RETURNING version;\n$$ LANGUAGE SQL;\n\n\nCREATE TABLE IF NOT EXISTS pgstac_settings (\n name text PRIMARY KEY,\n value text NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION table_empty(text) RETURNS boolean AS $$\nDECLARE\n retval boolean;\nBEGIN\n EXECUTE format($q$\n SELECT NOT EXISTS (SELECT 1 FROM %I LIMIT 1)\n $q$,\n $1\n ) INTO retval;\n RETURN retval;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION get_setting(IN _setting text, IN conf jsonb DEFAULT NULL) RETURNS text AS $$\nSELECT COALESCE(\n nullif(conf->>_setting, ''),\n nullif(current_setting(concat('pgstac.',_setting), TRUE),''),\n nullif((SELECT value FROM pgstac.pgstac_settings WHERE name=_setting),'')\n);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_setting_bool(IN _setting text, IN conf jsonb DEFAULT NULL) RETURNS boolean AS $$\nSELECT COALESCE(\n nullif(conf->>_setting, ''),\n nullif(current_setting(concat('pgstac.',_setting), TRUE),''),\n nullif((SELECT value FROM pgstac.pgstac_settings WHERE name=_setting),''),\n 'FALSE'\n)::boolean;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context(conf jsonb DEFAULT NULL) RETURNS text AS $$\n SELECT pgstac.get_setting('context', conf);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context_estimated_count(conf jsonb DEFAULT NULL) RETURNS int AS $$\n SELECT pgstac.get_setting('context_estimated_count', conf)::int;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_estimated_cost();\nCREATE OR REPLACE FUNCTION context_estimated_cost(conf jsonb DEFAULT NULL) RETURNS float AS $$\n SELECT pgstac.get_setting('context_estimated_cost', conf)::float;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_stats_ttl();\nCREATE OR REPLACE FUNCTION context_stats_ttl(conf jsonb DEFAULT NULL) RETURNS interval AS $$\n SELECT pgstac.get_setting('context_stats_ttl', conf)::interval;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION t2s(text) RETURNS text AS $$\n SELECT extract(epoch FROM $1::interval)::text || ' s';\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE STRICT;\n\nCREATE OR REPLACE FUNCTION age_ms(a timestamptz, b timestamptz DEFAULT clock_timestamp()) RETURNS float AS $$\n SELECT abs(extract(epoch from age(a,b)) * 1000);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION queue_timeout() RETURNS interval AS $$\n SELECT t2s(coalesce(\n get_setting('queue_timeout'),\n '1h'\n ))::interval;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION notice(VARIADIC text[]) RETURNS boolean AS $$\nDECLARE\ndebug boolean := current_setting('pgstac.debug', true);\nBEGIN\n IF debug THEN\n RAISE NOTICE 'NOTICE FROM FUNC: % >>>>> %', concat_ws(' | ', $1), clock_timestamp();\n RETURN TRUE;\n END IF;\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION empty_arr(ANYARRAY) RETURNS BOOLEAN AS $$\nSELECT CASE\n WHEN $1 IS NULL THEN TRUE\n WHEN cardinality($1)<1 THEN TRUE\nELSE FALSE\nEND;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_intersection(_a ANYARRAY, _b ANYARRAY) RETURNS ANYARRAY AS $$\n SELECT ARRAY ( SELECT unnest(_a) INTERSECT SELECT UNNEST(_b) );\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION array_map_ident(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_ident(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_map_literal(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_literal(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION array_reverse(anyarray) RETURNS anyarray AS $$\nSELECT ARRAY(\n SELECT $1[i]\n FROM generate_subscripts($1,1) AS s(i)\n ORDER BY i DESC\n);\n$$ LANGUAGE SQL STRICT IMMUTABLE;\n\nDROP TABLE IF EXISTS query_queue;\nCREATE TABLE query_queue (\n query text PRIMARY KEY,\n added timestamptz DEFAULT now()\n);\n\nDROP TABLE IF EXISTS query_queue_history;\nCREATE TABLE query_queue_history(\n query text,\n added timestamptz NOT NULL,\n finished timestamptz NOT NULL DEFAULT now(),\n error text\n);\n\nCREATE OR REPLACE PROCEDURE run_queued_queries() AS $$\nDECLARE\n qitem query_queue%ROWTYPE;\n timeout_ts timestamptz;\n error text;\n cnt int := 0;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n DELETE FROM query_queue WHERE query = (SELECT query FROM query_queue ORDER BY added DESC LIMIT 1 FOR UPDATE SKIP LOCKED) RETURNING * INTO qitem;\n IF NOT FOUND THEN\n EXIT;\n END IF;\n cnt := cnt + 1;\n BEGIN\n RAISE NOTICE 'RUNNING QUERY: %', qitem.query;\n EXECUTE qitem.query;\n EXCEPTION WHEN others THEN\n error := format('%s | %s', SQLERRM, SQLSTATE);\n END;\n INSERT INTO query_queue_history (query, added, finished, error)\n VALUES (qitem.query, qitem.added, clock_timestamp(), error);\n COMMIT;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION run_queued_queries_intransaction() RETURNS int AS $$\nDECLARE\n qitem query_queue%ROWTYPE;\n timeout_ts timestamptz;\n error text;\n cnt int := 0;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n DELETE FROM query_queue WHERE query = (SELECT query FROM query_queue ORDER BY added DESC LIMIT 1 FOR UPDATE SKIP LOCKED) RETURNING * INTO qitem;\n IF NOT FOUND THEN\n RETURN cnt;\n END IF;\n cnt := cnt + 1;\n BEGIN\n qitem.query := regexp_replace(qitem.query, 'CONCURRENTLY', '');\n RAISE NOTICE 'RUNNING QUERY: %', qitem.query;\n\n EXECUTE qitem.query;\n EXCEPTION WHEN others THEN\n error := format('%s | %s', SQLERRM, SQLSTATE);\n RAISE WARNING '%', error;\n END;\n INSERT INTO query_queue_history (query, added, finished, error)\n VALUES (qitem.query, qitem.added, clock_timestamp(), error);\n END LOOP;\n RETURN cnt;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION run_or_queue(query text) RETURNS VOID AS $$\nDECLARE\n use_queue text := COALESCE(get_setting('use_queue'), 'FALSE')::boolean;\nBEGIN\n IF get_setting_bool('debug') THEN\n RAISE NOTICE '%', query;\n END IF;\n IF use_queue THEN\n INSERT INTO query_queue (query) VALUES (query) ON CONFLICT DO NOTHING;\n ELSE\n EXECUTE query;\n END IF;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n\nDROP FUNCTION IF EXISTS check_pgstac_settings;\nCREATE OR REPLACE FUNCTION check_pgstac_settings(_sysmem text DEFAULT NULL) RETURNS VOID AS $$\nDECLARE\n settingval text;\n sysmem bigint := pg_size_bytes(_sysmem);\n effective_cache_size bigint := pg_size_bytes(current_setting('effective_cache_size', TRUE));\n shared_buffers bigint := pg_size_bytes(current_setting('shared_buffers', TRUE));\n work_mem bigint := pg_size_bytes(current_setting('work_mem', TRUE));\n max_connections int := current_setting('max_connections', TRUE);\n maintenance_work_mem bigint := pg_size_bytes(current_setting('maintenance_work_mem', TRUE));\n seq_page_cost float := current_setting('seq_page_cost', TRUE);\n random_page_cost float := current_setting('random_page_cost', TRUE);\n temp_buffers bigint := pg_size_bytes(current_setting('temp_buffers', TRUE));\n r record;\nBEGIN\n IF _sysmem IS NULL THEN\n RAISE NOTICE 'Call function with the size of your system memory `SELECT check_pgstac_settings(''4GB'')` to get pg system setting recommendations.';\n ELSE\n IF effective_cache_size < (sysmem * 0.5) THEN\n RAISE WARNING 'effective_cache_size of % is set low for a system with %. Recomended value between % and %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.5), pg_size_pretty(sysmem * 0.75);\n ELSIF effective_cache_size > (sysmem * 0.75) THEN\n RAISE WARNING 'effective_cache_size of % is set high for a system with %. Recomended value between % and %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.5), pg_size_pretty(sysmem * 0.75);\n ELSE\n RAISE NOTICE 'effective_cache_size of % is set appropriately for a system with %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem);\n END IF;\n\n IF shared_buffers < (sysmem * 0.2) THEN\n RAISE WARNING 'shared_buffers of % is set low for a system with %. Recomended value between % and %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.2), pg_size_pretty(sysmem * 0.3);\n ELSIF shared_buffers > (sysmem * 0.3) THEN\n RAISE WARNING 'shared_buffers of % is set high for a system with %. Recomended value between % and %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.2), pg_size_pretty(sysmem * 0.3);\n ELSE\n RAISE NOTICE 'shared_buffers of % is set appropriately for a system with %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem);\n END IF;\n shared_buffers = sysmem * 0.3;\n IF maintenance_work_mem < (sysmem * 0.2) THEN\n RAISE WARNING 'maintenance_work_mem of % is set low for shared_buffers of %. Recomended value between % and %', pg_size_pretty(maintenance_work_mem), pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers * 0.2), pg_size_pretty(shared_buffers * 0.3);\n ELSIF maintenance_work_mem > (shared_buffers * 0.3) THEN\n RAISE WARNING 'maintenance_work_mem of % is set high for shared_buffers of %. Recomended value between % and %', pg_size_pretty(maintenance_work_mem), pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers * 0.2), pg_size_pretty(shared_buffers * 0.3);\n ELSE\n RAISE NOTICE 'maintenance_work_mem of % is set appropriately for shared_buffers of %', pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers);\n END IF;\n\n IF work_mem * max_connections > shared_buffers THEN\n RAISE WARNING 'work_mem setting of % is set high for % max_connections please reduce work_mem to % or decrease max_connections to %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers/max_connections), floor(shared_buffers/work_mem);\n ELSIF work_mem * max_connections < (shared_buffers * 0.75) THEN\n RAISE WARNING 'work_mem setting of % is set low for % max_connections you may consider raising work_mem to % or increasing max_connections to %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers/max_connections), floor(shared_buffers/work_mem);\n ELSE\n RAISE NOTICE 'work_mem setting of % and max_connections of % are adequate for shared_buffers of %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers);\n END IF;\n\n IF random_page_cost / seq_page_cost != 1.1 THEN\n RAISE WARNING 'random_page_cost (%) /seq_page_cost (%) should be set to 1.1 for SSD. Change random_page_cost to %', random_page_cost, seq_page_cost, 1.1 * seq_page_cost;\n ELSE\n RAISE NOTICE 'random_page_cost and seq_page_cost set appropriately for SSD';\n END IF;\n\n IF temp_buffers < greatest(pg_size_bytes('128MB'),(maintenance_work_mem / 2)) THEN\n RAISE WARNING 'pgstac makes heavy use of temp tables, consider raising temp_buffers from % to %', pg_size_pretty(temp_buffers), greatest('128MB', pg_size_pretty((shared_buffers / 16)));\n END IF;\n END IF;\n\n RAISE NOTICE 'VALUES FOR PGSTAC VARIABLES';\n RAISE NOTICE 'These can be set either as GUC system variables or by setting in the pgstac_settings table.';\n\n FOR r IN SELECT name, get_setting(name) as setting, CASE WHEN current_setting(concat('pgstac.',name), TRUE) IS NOT NULL THEN concat('pgstac.',name, ' GUC') WHEN value IS NOT NULL THEN 'pgstac_settings table' ELSE 'Not Set' END as loc FROM pgstac_settings LOOP\n RAISE NOTICE '% is set to % from the %', r.name, r.setting, r.loc;\n END LOOP;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pg_cron';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider intalling pg_cron which can be used to automate tasks';\n ELSE\n RAISE NOTICE 'pg_cron % is installed', settingval;\n END IF;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pgstattuple';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider installing the pgstattuple extension which can be used to help maintain tables and indexes.';\n ELSE\n RAISE NOTICE 'pgstattuple % is installed', settingval;\n END IF;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pg_stat_statements';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider installing the pg_stat_statements extension which is very helpful for tracking the types of queries on the system';\n ELSE\n RAISE NOTICE 'pg_stat_statements % is installed', settingval;\n IF current_setting('pg_stat_statements.track_statements', TRUE) IS DISTINCT FROM 'all' THEN\n RAISE WARNING 'SET pg_stat_statements.track_statements TO ''all''; --In order to track statements within functions.';\n END IF;\n END IF;\n\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac, public SET CLIENT_MIN_MESSAGES TO NOTICE;\n/* looks for a geometry in a stac item first from geometry and falling back to bbox */\nCREATE OR REPLACE FUNCTION stac_geom(value jsonb) RETURNS geometry AS $$\nSELECT\n CASE\n WHEN value ? 'intersects' THEN\n ST_GeomFromGeoJSON(value->>'intersects')\n WHEN value ? 'geometry' THEN\n ST_GeomFromGeoJSON(value->>'geometry')\n WHEN value ? 'bbox' THEN\n pgstac.bbox_geom(value->'bbox')\n ELSE NULL\n END as geometry\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION stac_daterange(\n value jsonb\n) RETURNS tstzrange AS $$\nDECLARE\n props jsonb := value;\n dt timestamptz;\n edt timestamptz;\nBEGIN\n IF props ? 'properties' THEN\n props := props->'properties';\n END IF;\n IF\n props ? 'start_datetime'\n AND props->>'start_datetime' IS NOT NULL\n AND props ? 'end_datetime'\n AND props->>'end_datetime' IS NOT NULL\n THEN\n dt := props->>'start_datetime';\n edt := props->>'end_datetime';\n IF dt > edt THEN\n RAISE EXCEPTION 'start_datetime must be < end_datetime';\n END IF;\n ELSE\n dt := props->>'datetime';\n edt := props->>'datetime';\n END IF;\n IF dt is NULL OR edt IS NULL THEN\n RAISE NOTICE 'DT: %, EDT: %', dt, edt;\n RAISE EXCEPTION 'Either datetime (%) or both start_datetime (%) and end_datetime (%) must be set.', props->>'datetime',props->>'start_datetime',props->>'end_datetime';\n END IF;\n RETURN tstzrange(dt, edt, '[]');\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_datetime(value jsonb) RETURNS timestamptz AS $$\n SELECT lower(stac_daterange(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_end_datetime(value jsonb) RETURNS timestamptz AS $$\n SELECT upper(stac_daterange(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE TABLE IF NOT EXISTS stac_extensions(\n url text PRIMARY KEY,\n content jsonb\n);\nCREATE OR REPLACE FUNCTION queryable_signature(n text, c text[]) RETURNS text AS $$\n SELECT concat(n, c);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TABLE queryables (\n id bigint GENERATED ALWAYS AS identity PRIMARY KEY,\n name text NOT NULL,\n collection_ids text[], -- used to determine what partitions to create indexes on\n definition jsonb,\n property_path text,\n property_wrapper text,\n property_index_type text\n);\nCREATE INDEX queryables_name_idx ON queryables (name);\nCREATE INDEX queryables_collection_idx ON queryables USING GIN (collection_ids);\nCREATE INDEX queryables_property_wrapper_idx ON queryables (property_wrapper);\n\nCREATE OR REPLACE FUNCTION pgstac.queryables_constraint_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n allcollections text[];\nBEGIN\n RAISE NOTICE 'Making sure that name/collection is unique for queryables %', NEW;\n IF NEW.collection_ids IS NOT NULL THEN\n IF EXISTS (\n SELECT 1\n FROM unnest(NEW.collection_ids) c\n LEFT JOIN\n collections\n ON (collections.id = c)\n WHERE collections.id IS NULL\n ) THEN\n RAISE foreign_key_violation USING MESSAGE = format(\n 'One or more collections in %s do not exist.', NEW.collection_ids\n );\n RETURN NULL;\n END IF;\n END IF;\n IF TG_OP = 'INSERT' THEN\n IF EXISTS (\n SELECT 1 FROM queryables q\n WHERE\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n )\n ) THEN\n RAISE unique_violation USING MESSAGE = format(\n 'There is already a queryable for %s for a collection in %s: %s',\n NEW.name,\n NEW.collection_ids,\n\t\t\t\t(SELECT json_agg(row_to_json(q)) FROM queryables q WHERE\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n ))\n );\n RETURN NULL;\n END IF;\n END IF;\n IF TG_OP = 'UPDATE' THEN\n IF EXISTS (\n SELECT 1 FROM queryables q\n WHERE\n q.id != NEW.id\n AND\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n )\n ) THEN\n RAISE unique_violation\n USING MESSAGE = format(\n 'There is already a queryable for %s for a collection in %s',\n NEW.name,\n NEW.collection_ids\n );\n RETURN NULL;\n END IF;\n END IF;\n\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER queryables_constraint_insert_trigger\nBEFORE INSERT ON queryables\nFOR EACH ROW EXECUTE PROCEDURE queryables_constraint_triggerfunc();\n\nCREATE TRIGGER queryables_constraint_update_trigger\nBEFORE UPDATE ON queryables\nFOR EACH ROW\nWHEN (NEW.name = OLD.name AND NEW.collection_ids IS DISTINCT FROM OLD.collection_ids)\nEXECUTE PROCEDURE queryables_constraint_triggerfunc();\n\n\nCREATE OR REPLACE FUNCTION array_to_path(arr text[]) RETURNS text AS $$\n SELECT string_agg(\n quote_literal(v),\n '->'\n ) FROM unnest(arr) v;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\n\n\nCREATE OR REPLACE FUNCTION queryable(\n IN dotpath text,\n OUT path text,\n OUT expression text,\n OUT wrapper text,\n OUT nulled_wrapper text\n) AS $$\nDECLARE\n q RECORD;\n path_elements text[];\nBEGIN\n IF dotpath IN ('id', 'geometry', 'datetime', 'end_datetime', 'collection') THEN\n path := dotpath;\n expression := dotpath;\n wrapper := NULL;\n RETURN;\n END IF;\n SELECT * INTO q FROM queryables\n WHERE\n name=dotpath\n OR name = 'properties.' || dotpath\n OR name = replace(dotpath, 'properties.', '')\n ;\n IF q.property_wrapper IS NULL THEN\n IF q.definition->>'type' = 'number' THEN\n wrapper := 'to_float';\n nulled_wrapper := wrapper;\n ELSIF q.definition->>'format' = 'date-time' THEN\n wrapper := 'to_tstz';\n nulled_wrapper := wrapper;\n ELSE\n nulled_wrapper := NULL;\n wrapper := 'to_text';\n END IF;\n ELSE\n wrapper := q.property_wrapper;\n nulled_wrapper := wrapper;\n END IF;\n IF q.property_path IS NOT NULL THEN\n path := q.property_path;\n ELSE\n path_elements := string_to_array(dotpath, '.');\n IF path_elements[1] IN ('links', 'assets', 'stac_version', 'stac_extensions') THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSIF path_elements[1] = 'properties' THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSE\n path := format($F$content->'properties'->%s$F$, array_to_path(path_elements));\n END IF;\n END IF;\n expression := format('%I(%s)', wrapper, path);\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\nCREATE OR REPLACE FUNCTION unnest_collection(collection_ids text[] DEFAULT NULL) RETURNS SETOF text AS $$\n DECLARE\n BEGIN\n IF collection_ids IS NULL THEN\n RETURN QUERY SELECT id FROM collections;\n END IF;\n RETURN QUERY SELECT unnest(collection_ids);\n END;\n$$ LANGUAGE PLPGSQL STABLE;\n\nCREATE OR REPLACE FUNCTION normalize_indexdef(def text) RETURNS text AS $$\nDECLARE\nBEGIN\n def := btrim(def, ' \\n\\t');\n\tdef := regexp_replace(def, '^CREATE (UNIQUE )?INDEX ([^ ]* )?ON (ONLY )?([^ ]* )?', '', 'i');\n RETURN def;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE STRICT PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION indexdef(q queryables) RETURNS text AS $$\n DECLARE\n out text;\n BEGIN\n IF q.name = 'id' THEN\n out := 'CREATE UNIQUE INDEX ON %I USING btree (id)';\n ELSIF q.name = 'datetime' THEN\n out := 'CREATE INDEX ON %I USING btree (datetime DESC, end_datetime)';\n ELSIF q.name = 'geometry' THEN\n out := 'CREATE INDEX ON %I USING gist (geometry)';\n ELSE\n out := format($q$CREATE INDEX ON %%I USING %s (%s(((content -> 'properties'::text) -> %L::text)))$q$,\n lower(COALESCE(q.property_index_type, 'BTREE')),\n lower(COALESCE(q.property_wrapper, 'to_text')),\n q.name\n );\n END IF;\n RETURN btrim(out, ' \\n\\t');\n END;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\n\nDROP VIEW IF EXISTS pgstac_indexes;\nCREATE VIEW pgstac_indexes AS\nSELECT\n i.schemaname,\n i.tablename,\n i.indexname,\n regexp_replace(btrim(replace(replace(indexdef, i.indexname, ''),'pgstac.',''),' \\t\\n'), '[ ]+', ' ', 'g') as idx,\n COALESCE(\n (regexp_match(indexdef, '\\(([a-zA-Z]+)\\)'))[1],\n (regexp_match(indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_-]+)''::text'))[1],\n CASE WHEN indexdef ~* '\\(datetime desc, end_datetime\\)' THEN 'datetime' ELSE NULL END\n ) AS field,\n pg_table_size(i.indexname::text) as index_size,\n pg_size_pretty(pg_table_size(i.indexname::text)) as index_size_pretty\nFROM\n pg_indexes i\nWHERE i.schemaname='pgstac' and i.tablename ~ '_items_' AND indexdef !~* ' only ';\n\nDROP VIEW IF EXISTS pgstac_index_stats;\nCREATE VIEW pgstac_indexes_stats AS\nSELECT\n i.schemaname,\n i.tablename,\n i.indexname,\n indexdef,\n COALESCE(\n (regexp_match(indexdef, '\\(([a-zA-Z]+)\\)'))[1],\n (regexp_match(indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_]+)''::text'))[1],\n CASE WHEN indexdef ~* '\\(datetime desc, end_datetime\\)' THEN 'datetime_end_datetime' ELSE NULL END\n ) AS field,\n pg_table_size(i.indexname::text) as index_size,\n pg_size_pretty(pg_table_size(i.indexname::text)) as index_size_pretty,\n n_distinct,\n most_common_vals::text::text[],\n most_common_freqs::text::text[],\n histogram_bounds::text::text[],\n correlation\nFROM\n pg_indexes i\n LEFT JOIN pg_stats s ON (s.tablename = i.indexname)\nWHERE i.schemaname='pgstac' and i.tablename ~ '_items_';\n\nset check_function_bodies to off;\nCREATE OR REPLACE FUNCTION maintain_partition_queries(\n part text DEFAULT 'items',\n dropindexes boolean DEFAULT FALSE,\n rebuildindexes boolean DEFAULT FALSE,\n idxconcurrently boolean DEFAULT FALSE\n) RETURNS SETOF text AS $$\nDECLARE\n rec record;\nBEGIN\n FOR rec IN (\n WITH p AS (\n SELECT\n relid::text as partition,\n replace(replace(\n CASE\n WHEN level = 1 THEN pg_get_expr(c.relpartbound, c.oid)\n ELSE pg_get_expr(parent.relpartbound, parent.oid)\n END,\n 'FOR VALUES IN (''',''), ''')',\n ''\n ) AS collection\n FROM pg_partition_tree('items')\n JOIN pg_class c ON (relid::regclass = c.oid)\n JOIN pg_class parent ON (parentrelid::regclass = parent.oid AND isleaf)\n ), i AS (\n SELECT\n partition,\n indexname,\n regexp_replace(btrim(replace(replace(indexdef, indexname, ''),'pgstac.',''),' \\t\\n'), '[ ]+', ' ', 'g') as iidx,\n COALESCE(\n (regexp_match(indexdef, '\\(([a-zA-Z]+)\\)'))[1],\n (regexp_match(indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_-]+)''::text'))[1],\n CASE WHEN indexdef ~* '\\(datetime desc, end_datetime\\)' THEN 'datetime' ELSE NULL END\n ) AS field\n FROM\n pg_indexes\n JOIN p ON (tablename=partition)\n ), q AS (\n SELECT\n name AS field,\n collection,\n partition,\n format(indexdef(queryables), partition) as qidx\n FROM queryables, unnest_collection(queryables.collection_ids) collection\n JOIN p USING (collection)\n WHERE property_index_type IS NOT NULL OR name IN ('datetime','geometry','id')\n )\n SELECT * FROM i FULL JOIN q USING (field, partition)\n WHERE lower(iidx) IS DISTINCT FROM lower(qidx)\n ) LOOP\n IF rec.iidx IS NULL THEN\n IF idxconcurrently THEN\n RETURN NEXT replace(rec.qidx, 'INDEX', 'INDEX CONCURRENTLY');\n ELSE\n RETURN NEXT rec.qidx;\n END IF;\n ELSIF rec.qidx IS NULL AND dropindexes THEN\n RETURN NEXT format('DROP INDEX IF EXISTS %I;', rec.indexname);\n ELSIF lower(rec.qidx) != lower(rec.iidx) THEN\n IF dropindexes THEN\n RETURN NEXT format('DROP INDEX IF EXISTS %I; %s;', rec.indexname, rec.qidx);\n ELSE\n IF idxconcurrently THEN\n RETURN NEXT replace(rec.qidx, 'INDEX', 'INDEX CONCURRENTLY');\n ELSE\n RETURN NEXT rec.qidx;\n END IF;\n END IF;\n ELSIF rebuildindexes and rec.indexname IS NOT NULL THEN\n IF idxconcurrently THEN\n RETURN NEXT format('REINDEX INDEX CONCURRENTLY %I;', rec.indexname);\n ELSE\n RETURN NEXT format('REINDEX INDEX %I;', rec.indexname);\n END IF;\n END IF;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION maintain_partitions(\n part text DEFAULT 'items',\n dropindexes boolean DEFAULT FALSE,\n rebuildindexes boolean DEFAULT FALSE\n) RETURNS VOID AS $$\n WITH t AS (\n SELECT run_or_queue(q) FROM maintain_partition_queries(part, dropindexes, rebuildindexes) q\n ) SELECT count(*) FROM t;\n$$ LANGUAGE SQL;\n\n\nCREATE OR REPLACE FUNCTION queryables_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\nBEGIN\n PERFORM maintain_partitions();\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER queryables_trigger AFTER INSERT OR UPDATE ON queryables\nFOR EACH STATEMENT EXECUTE PROCEDURE queryables_trigger_func();\n\n\nCREATE OR REPLACE FUNCTION get_queryables(_collection_ids text[] DEFAULT NULL) RETURNS jsonb AS $$\nDECLARE\nBEGIN\n -- Build up queryables if the input contains valid collection ids or is empty\n IF EXISTS (\n SELECT 1 FROM collections\n WHERE\n _collection_ids IS NULL\n OR cardinality(_collection_ids) = 0\n OR id = ANY(_collection_ids)\n )\n THEN\n RETURN (\n WITH base AS (\n SELECT\n unnest(collection_ids) as collection_id,\n name,\n coalesce(definition, '{\"type\":\"string\"}'::jsonb) as definition\n FROM queryables\n WHERE\n _collection_ids IS NULL OR\n _collection_ids = '{}'::text[] OR\n _collection_ids && collection_ids\n UNION ALL\n SELECT null, name, coalesce(definition, '{\"type\":\"string\"}'::jsonb) as definition\n FROM queryables WHERE collection_ids IS NULL OR collection_ids = '{}'::text[]\n ), g AS (\n SELECT\n name,\n first_notnull(definition) as definition,\n jsonb_array_unique_merge(definition->'enum') as enum,\n jsonb_min(definition->'minimum') as minimum,\n jsonb_min(definition->'maxiumn') as maximum\n FROM base\n GROUP BY 1\n )\n SELECT\n jsonb_build_object(\n '$schema', 'http://json-schema.org/draft-07/schema#',\n '$id', '',\n 'type', 'object',\n 'title', 'STAC Queryables.',\n 'properties', jsonb_object_agg(\n name,\n definition\n ||\n jsonb_strip_nulls(jsonb_build_object(\n 'enum', enum,\n 'minimum', minimum,\n 'maximum', maximum\n ))\n )\n )\n FROM g\n );\n ELSE\n RETURN NULL;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\nCREATE OR REPLACE FUNCTION get_queryables(_collection text DEFAULT NULL) RETURNS jsonb AS $$\n SELECT\n CASE\n WHEN _collection IS NULL THEN get_queryables(NULL::text[])\n ELSE get_queryables(ARRAY[_collection])\n END\n ;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_queryables() RETURNS jsonb AS $$\n SELECT get_queryables(NULL::text[]);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION schema_qualify_refs(url text, j jsonb) returns jsonb as $$\n SELECT regexp_replace(j::text, '\"\\$ref\": \"#', concat('\"$ref\": \"', url, '#'), 'g')::jsonb;\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE VIEW stac_extension_queryables AS\nSELECT DISTINCT key as name, schema_qualify_refs(e.url, j.value) as definition FROM stac_extensions e, jsonb_each(e.content->'definitions'->'fields'->'properties') j;\n\n\nCREATE OR REPLACE FUNCTION missing_queryables(_collection text, _tablesample float DEFAULT 5, minrows float DEFAULT 10) RETURNS TABLE(collection text, name text, definition jsonb, property_wrapper text) AS $$\nDECLARE\n q text;\n _partition text;\n explain_json json;\n psize float;\n estrows float;\nBEGIN\n SELECT format('_items_%s', key) INTO _partition FROM collections WHERE id=_collection;\n\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM %I;', _partition)\n INTO explain_json;\n psize := explain_json->0->'Plan'->'Plan Rows';\n estrows := _tablesample * .01 * psize;\n IF estrows < minrows THEN\n _tablesample := least(100,greatest(_tablesample, (estrows / psize) / 100));\n RAISE NOTICE '%', (psize / estrows) / 100;\n END IF;\n RAISE NOTICE 'Using tablesample % to find missing queryables from % % that has ~% rows estrows: %', _tablesample, _collection, _partition, psize, estrows;\n\n q := format(\n $q$\n WITH q AS (\n SELECT * FROM queryables\n WHERE\n collection_ids IS NULL\n OR %L = ANY(collection_ids)\n ), t AS (\n SELECT\n content->'properties' AS properties\n FROM\n %I\n TABLESAMPLE SYSTEM(%L)\n ), p AS (\n SELECT DISTINCT ON (key)\n key,\n value,\n s.definition\n FROM t\n JOIN LATERAL jsonb_each(properties) ON TRUE\n LEFT JOIN q ON (q.name=key)\n LEFT JOIN stac_extension_queryables s ON (s.name=key)\n WHERE q.definition IS NULL\n )\n SELECT\n %L,\n key,\n COALESCE(definition, jsonb_build_object('type',jsonb_typeof(value))) as definition,\n CASE\n WHEN definition->>'type' = 'integer' THEN 'to_int'\n WHEN COALESCE(definition->>'type', jsonb_typeof(value)) = 'number' THEN 'to_float'\n WHEN COALESCE(definition->>'type', jsonb_typeof(value)) = 'array' THEN 'to_text_array'\n ELSE 'to_text'\n END\n FROM p;\n $q$,\n _collection,\n _partition,\n _tablesample,\n _collection\n );\n RETURN QUERY EXECUTE q;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION missing_queryables(_tablesample float DEFAULT 5) RETURNS TABLE(collection_ids text[], name text, definition jsonb, property_wrapper text) AS $$\n SELECT\n array_agg(collection),\n name,\n definition,\n property_wrapper\n FROM\n collections\n JOIN LATERAL\n missing_queryables(id, _tablesample) c\n ON TRUE\n GROUP BY\n 2,3,4\n ORDER BY 2,1\n ;\n$$ LANGUAGE SQL;\nCREATE OR REPLACE FUNCTION parse_dtrange(\n _indate jsonb,\n relative_base timestamptz DEFAULT date_trunc('hour', CURRENT_TIMESTAMP)\n) RETURNS tstzrange AS $$\nDECLARE\n timestrs text[];\n s timestamptz;\n e timestamptz;\nBEGIN\n timestrs :=\n CASE\n WHEN _indate ? 'timestamp' THEN\n ARRAY[_indate->>'timestamp']\n WHEN _indate ? 'interval' THEN\n to_text_array(_indate->'interval')\n WHEN jsonb_typeof(_indate) = 'array' THEN\n to_text_array(_indate)\n ELSE\n regexp_split_to_array(\n _indate->>0,\n '/'\n )\n END;\n RAISE NOTICE 'TIMESTRS %', timestrs;\n IF cardinality(timestrs) = 1 THEN\n IF timestrs[1] ILIKE 'P%' THEN\n RETURN tstzrange(relative_base - upper(timestrs[1])::interval, relative_base, '[)');\n END IF;\n s := timestrs[1]::timestamptz;\n RETURN tstzrange(s, s, '[]');\n END IF;\n\n IF cardinality(timestrs) != 2 THEN\n RAISE EXCEPTION 'Timestamp cannot have more than 2 values';\n END IF;\n\n IF timestrs[1] = '..' OR timestrs[1] = '' THEN\n s := '-infinity'::timestamptz;\n e := timestrs[2]::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] = '..' OR timestrs[2] = '' THEN\n s := timestrs[1]::timestamptz;\n e := 'infinity'::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[1] ILIKE 'P%' AND timestrs[2] NOT ILIKE 'P%' THEN\n e := timestrs[2]::timestamptz;\n s := e - upper(timestrs[1])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] ILIKE 'P%' AND timestrs[1] NOT ILIKE 'P%' THEN\n s := timestrs[1]::timestamptz;\n e := s + upper(timestrs[2])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n s := timestrs[1]::timestamptz;\n e := timestrs[2]::timestamptz;\n\n RETURN tstzrange(s,e,'[)');\n\n RETURN NULL;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT PARALLEL SAFE SET TIME ZONE 'UTC';\n\nCREATE OR REPLACE FUNCTION parse_dtrange(\n _indate text,\n relative_base timestamptz DEFAULT CURRENT_TIMESTAMP\n) RETURNS tstzrange AS $$\n SELECT parse_dtrange(to_jsonb(_indate), relative_base);\n$$ LANGUAGE SQL STABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION temporal_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\n ll text := 'datetime';\n lh text := 'end_datetime';\n rrange tstzrange;\n rl text;\n rh text;\n outq text;\nBEGIN\n rrange := parse_dtrange(args->1);\n RAISE NOTICE 'Constructing temporal query OP: %, ARGS: %, RRANGE: %', op, args, rrange;\n op := lower(op);\n rl := format('%L::timestamptz', lower(rrange));\n rh := format('%L::timestamptz', upper(rrange));\n outq := CASE op\n WHEN 't_before' THEN 'lh < rl'\n WHEN 't_after' THEN 'll > rh'\n WHEN 't_meets' THEN 'lh = rl'\n WHEN 't_metby' THEN 'll = rh'\n WHEN 't_overlaps' THEN 'll < rl AND rl < lh < rh'\n WHEN 't_overlappedby' THEN 'rl < ll < rh AND lh > rh'\n WHEN 't_starts' THEN 'll = rl AND lh < rh'\n WHEN 't_startedby' THEN 'll = rl AND lh > rh'\n WHEN 't_during' THEN 'll > rl AND lh < rh'\n WHEN 't_contains' THEN 'll < rl AND lh > rh'\n WHEN 't_finishes' THEN 'll > rl AND lh = rh'\n WHEN 't_finishedby' THEN 'll < rl AND lh = rh'\n WHEN 't_equals' THEN 'll = rl AND lh = rh'\n WHEN 't_disjoint' THEN 'NOT (ll <= rh AND lh >= rl)'\n WHEN 't_intersects' THEN 'll <= rh AND lh >= rl'\n WHEN 'anyinteracts' THEN 'll <= rh AND lh >= rl'\n END;\n outq := regexp_replace(outq, '\\mll\\M', ll);\n outq := regexp_replace(outq, '\\mlh\\M', lh);\n outq := regexp_replace(outq, '\\mrl\\M', rl);\n outq := regexp_replace(outq, '\\mrh\\M', rh);\n outq := format('(%s)', outq);\n RETURN outq;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\n\n\nCREATE OR REPLACE FUNCTION spatial_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\n geom text;\n j jsonb := args->1;\nBEGIN\n op := lower(op);\n RAISE NOTICE 'Constructing spatial query OP: %, ARGS: %', op, args;\n IF op NOT IN ('s_equals','s_disjoint','s_touches','s_within','s_overlaps','s_crosses','s_intersects','intersects','s_contains') THEN\n RAISE EXCEPTION 'Spatial Operator % Not Supported', op;\n END IF;\n op := regexp_replace(op, '^s_', 'st_');\n IF op = 'intersects' THEN\n op := 'st_intersects';\n END IF;\n -- Convert geometry to WKB string\n IF j ? 'type' AND j ? 'coordinates' THEN\n geom := st_geomfromgeojson(j)::text;\n ELSIF jsonb_typeof(j) = 'array' THEN\n geom := bbox_geom(j)::text;\n END IF;\n\n RETURN format('%s(geometry, %L::geometry)', op, geom);\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION query_to_cql2(q jsonb) RETURNS jsonb AS $$\n-- Translates anything passed in through the deprecated \"query\" into equivalent CQL2\nWITH t AS (\n SELECT key as property, value as ops\n FROM jsonb_each(q)\n), t2 AS (\n SELECT property, (jsonb_each(ops)).*\n FROM t WHERE jsonb_typeof(ops) = 'object'\n UNION ALL\n SELECT property, 'eq', ops\n FROM t WHERE jsonb_typeof(ops) != 'object'\n)\nSELECT\n jsonb_strip_nulls(jsonb_build_object(\n 'op', 'and',\n 'args', jsonb_agg(\n jsonb_build_object(\n 'op', key,\n 'args', jsonb_build_array(\n jsonb_build_object('property',property),\n value\n )\n )\n )\n )\n) as qcql FROM t2\n;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\nCREATE OR REPLACE FUNCTION cql1_to_cql2(j jsonb) RETURNS jsonb AS $$\nDECLARE\n args jsonb;\n ret jsonb;\nBEGIN\n RAISE NOTICE 'CQL1_TO_CQL2: %', j;\n IF j ? 'filter' THEN\n RETURN cql1_to_cql2(j->'filter');\n END IF;\n IF j ? 'property' THEN\n RETURN j;\n END IF;\n IF jsonb_typeof(j) = 'array' THEN\n SELECT jsonb_agg(cql1_to_cql2(el)) INTO args FROM jsonb_array_elements(j) el;\n RETURN args;\n END IF;\n IF jsonb_typeof(j) = 'number' THEN\n RETURN j;\n END IF;\n IF jsonb_typeof(j) = 'string' THEN\n RETURN j;\n END IF;\n\n IF jsonb_typeof(j) = 'object' THEN\n SELECT jsonb_build_object(\n 'op', key,\n 'args', cql1_to_cql2(value)\n ) INTO ret\n FROM jsonb_each(j)\n WHERE j IS NOT NULL;\n RETURN ret;\n END IF;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE STRICT;\n\nCREATE TABLE cql2_ops (\n op text PRIMARY KEY,\n template text,\n types text[]\n);\nINSERT INTO cql2_ops (op, template, types) VALUES\n ('eq', '%s = %s', NULL),\n ('neq', '%s != %s', NULL),\n ('ne', '%s != %s', NULL),\n ('!=', '%s != %s', NULL),\n ('<>', '%s != %s', NULL),\n ('lt', '%s < %s', NULL),\n ('lte', '%s <= %s', NULL),\n ('gt', '%s > %s', NULL),\n ('gte', '%s >= %s', NULL),\n ('le', '%s <= %s', NULL),\n ('ge', '%s >= %s', NULL),\n ('=', '%s = %s', NULL),\n ('<', '%s < %s', NULL),\n ('<=', '%s <= %s', NULL),\n ('>', '%s > %s', NULL),\n ('>=', '%s >= %s', NULL),\n ('like', '%s LIKE %s', NULL),\n ('ilike', '%s ILIKE %s', NULL),\n ('+', '%s + %s', NULL),\n ('-', '%s - %s', NULL),\n ('*', '%s * %s', NULL),\n ('/', '%s / %s', NULL),\n ('not', 'NOT (%s)', NULL),\n ('between', '%s BETWEEN %s AND %s', NULL),\n ('isnull', '%s IS NULL', NULL),\n ('upper', 'upper(%s)', NULL),\n ('lower', 'lower(%s)', NULL)\nON CONFLICT (op) DO UPDATE\n SET\n template = EXCLUDED.template\n;\n\n\nCREATE OR REPLACE FUNCTION cql2_query(j jsonb, wrapper text DEFAULT NULL) RETURNS text AS $$\n#variable_conflict use_variable\nDECLARE\n args jsonb := j->'args';\n arg jsonb;\n op text := lower(j->>'op');\n cql2op RECORD;\n literal text;\n _wrapper text;\n leftarg text;\n rightarg text;\nBEGIN\n IF j IS NULL OR (op IS NOT NULL AND args IS NULL) THEN\n RETURN NULL;\n END IF;\n RAISE NOTICE 'CQL2_QUERY: %', j;\n IF j ? 'filter' THEN\n RETURN cql2_query(j->'filter');\n END IF;\n\n IF j ? 'upper' THEN\n RETURN cql2_query(jsonb_build_object('op', 'upper', 'args', j->'upper'));\n END IF;\n\n IF j ? 'lower' THEN\n RETURN cql2_query(jsonb_build_object('op', 'lower', 'args', j->'lower'));\n END IF;\n\n -- Temporal Query\n IF op ilike 't_%' or op = 'anyinteracts' THEN\n RETURN temporal_op_query(op, args);\n END IF;\n\n -- If property is a timestamp convert it to text to use with\n -- general operators\n IF j ? 'timestamp' THEN\n RETURN format('%L::timestamptz', to_tstz(j->'timestamp'));\n END IF;\n IF j ? 'interval' THEN\n RAISE EXCEPTION 'Please use temporal operators when using intervals.';\n RETURN NONE;\n END IF;\n\n -- Spatial Query\n IF op ilike 's_%' or op = 'intersects' THEN\n RETURN spatial_op_query(op, args);\n END IF;\n\n IF op IN ('a_equals','a_contains','a_contained_by','a_overlaps') THEN\n IF args->0 ? 'property' THEN\n leftarg := format('to_text_array(%s)', (queryable(args->0->>'property')).path);\n END IF;\n IF args->1 ? 'property' THEN\n rightarg := format('to_text_array(%s)', (queryable(args->1->>'property')).path);\n END IF;\n RETURN FORMAT(\n '%s %s %s',\n COALESCE(leftarg, quote_literal(to_text_array(args->0))),\n CASE op\n WHEN 'a_equals' THEN '='\n WHEN 'a_contains' THEN '@>'\n WHEN 'a_contained_by' THEN '<@'\n WHEN 'a_overlaps' THEN '&&'\n END,\n COALESCE(rightarg, quote_literal(to_text_array(args->1)))\n );\n END IF;\n\n IF op = 'in' THEN\n RAISE NOTICE 'IN : % % %', args, jsonb_build_array(args->0), args->1;\n args := jsonb_build_array(args->0) || (args->1);\n RAISE NOTICE 'IN2 : %', args;\n END IF;\n\n\n\n IF op = 'between' THEN\n args = jsonb_build_array(\n args->0,\n args->1->0,\n args->1->1\n );\n END IF;\n\n -- Make sure that args is an array and run cql2_query on\n -- each element of the array\n RAISE NOTICE 'ARGS PRE: %', args;\n IF j ? 'args' THEN\n IF jsonb_typeof(args) != 'array' THEN\n args := jsonb_build_array(args);\n END IF;\n\n IF jsonb_path_exists(args, '$[*] ? (@.property == \"id\" || @.property == \"datetime\" || @.property == \"end_datetime\" || @.property == \"collection\")') THEN\n wrapper := NULL;\n ELSE\n -- if any of the arguments are a property, try to get the property_wrapper\n FOR arg IN SELECT jsonb_path_query(args, '$[*] ? (@.property != null)') LOOP\n RAISE NOTICE 'Arg: %', arg;\n wrapper := (queryable(arg->>'property')).nulled_wrapper;\n RAISE NOTICE 'Property: %, Wrapper: %', arg, wrapper;\n IF wrapper IS NOT NULL THEN\n EXIT;\n END IF;\n END LOOP;\n\n -- if the property was not in queryables, see if any args were numbers\n IF\n wrapper IS NULL\n AND jsonb_path_exists(args, '$[*] ? (@.type()==\"number\")')\n THEN\n wrapper := 'to_float';\n END IF;\n wrapper := coalesce(wrapper, 'to_text');\n END IF;\n\n SELECT jsonb_agg(cql2_query(a, wrapper))\n INTO args\n FROM jsonb_array_elements(args) a;\n END IF;\n RAISE NOTICE 'ARGS: %', args;\n\n IF op IN ('and', 'or') THEN\n RETURN\n format(\n '(%s)',\n array_to_string(to_text_array(args), format(' %s ', upper(op)))\n );\n END IF;\n\n IF op = 'in' THEN\n RAISE NOTICE 'IN -- % %', args->0, to_text(args->0);\n RETURN format(\n '%s IN (%s)',\n to_text(args->0),\n array_to_string((to_text_array(args))[2:], ',')\n );\n END IF;\n\n -- Look up template from cql2_ops\n IF j ? 'op' THEN\n SELECT * INTO cql2op FROM cql2_ops WHERE cql2_ops.op ilike op;\n IF FOUND THEN\n -- If specific index set in queryables for a property cast other arguments to that type\n\n RETURN format(\n cql2op.template,\n VARIADIC (to_text_array(args))\n );\n ELSE\n RAISE EXCEPTION 'Operator % Not Supported.', op;\n END IF;\n END IF;\n\n\n IF wrapper IS NOT NULL THEN\n RAISE NOTICE 'Wrapping % with %', j, wrapper;\n IF j ? 'property' THEN\n RETURN format('%I(%s)', wrapper, (queryable(j->>'property')).path);\n ELSE\n RETURN format('%I(%L)', wrapper, j);\n END IF;\n ELSIF j ? 'property' THEN\n RETURN quote_ident(j->>'property');\n END IF;\n\n RETURN quote_literal(to_text(j));\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION paging_dtrange(\n j jsonb\n) RETURNS tstzrange AS $$\nDECLARE\n op text;\n filter jsonb := j->'filter';\n dtrange tstzrange := tstzrange('-infinity'::timestamptz,'infinity'::timestamptz);\n sdate timestamptz := '-infinity'::timestamptz;\n edate timestamptz := 'infinity'::timestamptz;\n jpitem jsonb;\nBEGIN\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n END IF;\n IF NOT (filter @? '$.**.op ? (@ == \"or\" || @ == \"not\")') THEN\n FOR jpitem IN SELECT j FROM jsonb_path_query(filter,'strict $.** ? (@.args[*].property == \"datetime\")'::jsonpath) j LOOP\n op := lower(jpitem->>'op');\n dtrange := parse_dtrange(jpitem->'args'->1);\n IF op IN ('<=', 'lt', 'lte', '<', 'le', 't_before') THEN\n sdate := greatest(sdate,'-infinity');\n edate := least(edate, upper(dtrange));\n ELSIF op IN ('>=', '>', 'gt', 'gte', 'ge', 't_after') THEN\n edate := least(edate, 'infinity');\n sdate := greatest(sdate, lower(dtrange));\n ELSIF op IN ('=', 'eq') THEN\n edate := least(edate, upper(dtrange));\n sdate := greatest(sdate, lower(dtrange));\n END IF;\n RAISE NOTICE '2 OP: %, ARGS: %, DTRANGE: %, SDATE: %, EDATE: %', op, jpitem->'args'->1, dtrange, sdate, edate;\n END LOOP;\n END IF;\n IF sdate > edate THEN\n RETURN 'empty'::tstzrange;\n END IF;\n RETURN tstzrange(sdate,edate, '[]');\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT SET TIME ZONE 'UTC';\n\nCREATE OR REPLACE FUNCTION paging_collections(\n IN j jsonb\n) RETURNS text[] AS $$\nDECLARE\n filter jsonb := j->'filter';\n jpitem jsonb;\n op text;\n args jsonb;\n arg jsonb;\n collections text[];\nBEGIN\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n END IF;\n IF NOT (filter @? '$.**.op ? (@ == \"or\" || @ == \"not\")') THEN\n FOR jpitem IN SELECT j FROM jsonb_path_query(filter,'strict $.** ? (@.args[*].property == \"collection\")'::jsonpath) j LOOP\n RAISE NOTICE 'JPITEM: %', jpitem;\n op := jpitem->>'op';\n args := jpitem->'args';\n IF op IN ('=', 'eq', 'in') THEN\n FOR arg IN SELECT a FROM jsonb_array_elements(args) a LOOP\n IF jsonb_typeof(arg) IN ('string', 'array') THEN\n RAISE NOTICE 'arg: %, collections: %', arg, collections;\n IF collections IS NULL OR collections = '{}'::text[] THEN\n collections := to_text_array(arg);\n ELSE\n collections := array_intersection(collections, to_text_array(arg));\n END IF;\n END IF;\n END LOOP;\n END IF;\n END LOOP;\n END IF;\n IF collections = '{}'::text[] THEN\n RETURN NULL;\n END IF;\n RETURN collections;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\nCREATE OR REPLACE FUNCTION collection_base_item(content jsonb) RETURNS jsonb AS $$\n SELECT jsonb_build_object(\n 'type', 'Feature',\n 'stac_version', content->'stac_version',\n 'assets', content->'item_assets',\n 'collection', content->'id'\n );\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TABLE IF NOT EXISTS collections (\n key bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,\n id text GENERATED ALWAYS AS (content->>'id') STORED UNIQUE,\n content JSONB NOT NULL,\n base_item jsonb GENERATED ALWAYS AS (pgstac.collection_base_item(content)) STORED,\n partition_trunc text CHECK (partition_trunc IN ('year', 'month'))\n);\n\n\n\nCREATE OR REPLACE FUNCTION collection_base_item(cid text) RETURNS jsonb AS $$\n SELECT pgstac.collection_base_item(content) FROM pgstac.collections WHERE id = cid LIMIT 1;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION create_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION update_collection(data jsonb) RETURNS VOID AS $$\nDECLARE\n out collections%ROWTYPE;\nBEGIN\n UPDATE collections SET content=data WHERE id = data->>'id' RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ON CONFLICT (id) DO\n UPDATE\n SET content=EXCLUDED.content\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION delete_collection(_id text) RETURNS VOID AS $$\nDECLARE\n out collections%ROWTYPE;\nBEGIN\n DELETE FROM collections WHERE id = _id RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION get_collection(id text) RETURNS jsonb AS $$\n SELECT content FROM collections\n WHERE id=$1\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION all_collections() RETURNS jsonb AS $$\n SELECT coalesce(jsonb_agg(content), '[]'::jsonb) FROM collections;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION collection_delete_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n collection_base_partition text := concat('_items_', OLD.key);\nBEGIN\n EXECUTE format($q$\n DELETE FROM partition_stats WHERE partition IN (\n SELECT partition FROM partition_sys_meta\n WHERE collection=%L\n );\n DROP TABLE IF EXISTS %I CASCADE;\n $q$,\n OLD.id,\n collection_base_partition\n );\n RETURN OLD;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER collection_delete_trigger BEFORE DELETE ON collections\nFOR EACH ROW EXECUTE FUNCTION collection_delete_trigger_func();\nCREATE TABLE items (\n id text NOT NULL,\n geometry geometry NOT NULL,\n collection text NOT NULL,\n datetime timestamptz NOT NULL,\n end_datetime timestamptz NOT NULL,\n content JSONB NOT NULL\n)\nPARTITION BY LIST (collection)\n;\n\nCREATE INDEX \"datetime_idx\" ON items USING BTREE (datetime DESC, end_datetime ASC);\nCREATE INDEX \"geometry_idx\" ON items USING GIST (geometry);\n\nCREATE STATISTICS datetime_stats (dependencies) on datetime, end_datetime from items;\n\nALTER TABLE items ADD CONSTRAINT items_collections_fk FOREIGN KEY (collection) REFERENCES collections(id) ON DELETE CASCADE DEFERRABLE;\n\nCREATE OR REPLACE FUNCTION partition_after_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p text;\n t timestamptz := clock_timestamp();\nBEGIN\n RAISE NOTICE 'Updating partition stats %', t;\n FOR p IN SELECT DISTINCT partition\n FROM newdata n JOIN partition_sys_meta p\n ON (n.collection=p.collection AND n.datetime <@ p.partition_dtrange)\n LOOP\n PERFORM run_or_queue(format('SELECT update_partition_stats(%L, %L);', p, true));\n END LOOP;\n IF TG_OP IN ('DELETE','UPDATE') THEN\n DELETE FROM format_item_cache c USING newdata n WHERE c.collection = n.collection AND c.id = n.id;\n END IF;\n RAISE NOTICE 't: % %', t, clock_timestamp() - t;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE TRIGGER items_after_insert_trigger\nAFTER INSERT ON items\nREFERENCING NEW TABLE AS newdata\nFOR EACH STATEMENT\nEXECUTE FUNCTION partition_after_triggerfunc();\n\nCREATE TRIGGER items_after_update_trigger\nAFTER DELETE ON items\nREFERENCING OLD TABLE AS newdata\nFOR EACH STATEMENT\nEXECUTE FUNCTION partition_after_triggerfunc();\n\nCREATE TRIGGER items_after_delete_trigger\nAFTER UPDATE ON items\nREFERENCING NEW TABLE AS newdata\nFOR EACH STATEMENT\nEXECUTE FUNCTION partition_after_triggerfunc();\n\n\nCREATE OR REPLACE FUNCTION content_slim(_item jsonb) RETURNS jsonb AS $$\n SELECT strip_jsonb(_item - '{id,geometry,collection,type}'::text[], collection_base_item(_item->>'collection')) - '{id,geometry,collection,type}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_dehydrate(content jsonb) RETURNS items AS $$\n SELECT\n content->>'id' as id,\n stac_geom(content) as geometry,\n content->>'collection' as collection,\n stac_datetime(content) as datetime,\n stac_end_datetime(content) as end_datetime,\n content_slim(content) as content\n ;\n$$ LANGUAGE SQL STABLE;\n\nCREATE OR REPLACE FUNCTION include_field(f text, fields jsonb DEFAULT '{}'::jsonb) RETURNS boolean AS $$\nDECLARE\n includes jsonb := fields->'include';\n excludes jsonb := fields->'exclude';\nBEGIN\n IF f IS NULL THEN\n RETURN NULL;\n END IF;\n\n\n IF\n jsonb_typeof(excludes) = 'array'\n AND jsonb_array_length(excludes)>0\n AND excludes ? f\n THEN\n RETURN FALSE;\n END IF;\n\n IF\n (\n jsonb_typeof(includes) = 'array'\n AND jsonb_array_length(includes) > 0\n AND includes ? f\n ) OR\n (\n includes IS NULL\n OR jsonb_typeof(includes) = 'null'\n OR jsonb_array_length(includes) = 0\n )\n THEN\n RETURN TRUE;\n END IF;\n\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nDROP FUNCTION IF EXISTS content_hydrate(jsonb, jsonb, jsonb);\nCREATE OR REPLACE FUNCTION content_hydrate(\n _item jsonb,\n _base_item jsonb,\n fields jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\n SELECT merge_jsonb(\n jsonb_fields(_item, fields),\n jsonb_fields(_base_item, fields)\n );\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION content_hydrate(_item items, _collection collections, fields jsonb DEFAULT '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\n content jsonb;\n base_item jsonb := _collection.base_item;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry, 20)::jsonb;\n END IF;\n output := content_hydrate(\n jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'collection', _item.collection,\n 'type', 'Feature'\n ) || _item.content,\n _collection.base_item,\n fields\n );\n\n RETURN output;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_nonhydrated(\n _item items,\n fields jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry, 20)::jsonb;\n END IF;\n output := jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'collection', _item.collection,\n 'type', 'Feature'\n ) || _item.content;\n RETURN output;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_hydrate(_item items, fields jsonb DEFAULT '{}'::jsonb) RETURNS jsonb AS $$\n SELECT content_hydrate(\n _item,\n (SELECT c FROM collections c WHERE id=_item.collection LIMIT 1),\n fields\n );\n$$ LANGUAGE SQL STABLE;\n\n\nCREATE UNLOGGED TABLE items_staging (\n content JSONB NOT NULL\n);\nCREATE UNLOGGED TABLE items_staging_ignore (\n content JSONB NOT NULL\n);\nCREATE UNLOGGED TABLE items_staging_upsert (\n content JSONB NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION items_staging_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p record;\n _partitions text[];\n part text;\n ts timestamptz := clock_timestamp();\nBEGIN\n RAISE NOTICE 'Creating Partitions. %', clock_timestamp() - ts;\n\n FOR part IN WITH t AS (\n SELECT\n n.content->>'collection' as collection,\n stac_daterange(n.content->'properties') as dtr,\n partition_trunc\n FROM newdata n JOIN collections ON (n.content->>'collection'=collections.id)\n ), p AS (\n SELECT\n collection,\n COALESCE(date_trunc(partition_trunc::text, lower(dtr)),'-infinity') as d,\n tstzrange(min(lower(dtr)),max(lower(dtr)),'[]') as dtrange,\n tstzrange(min(upper(dtr)),max(upper(dtr)),'[]') as edtrange\n FROM t\n GROUP BY 1,2\n ) SELECT check_partition(collection, dtrange, edtrange) FROM p LOOP\n RAISE NOTICE 'Partition %', part;\n END LOOP;\n\n RAISE NOTICE 'Doing the insert. %', clock_timestamp() - ts;\n IF TG_TABLE_NAME = 'items_staging' THEN\n INSERT INTO items\n SELECT\n (content_dehydrate(content)).*\n FROM newdata;\n RAISE NOTICE 'Doing the delete. %', clock_timestamp() - ts;\n DELETE FROM items_staging;\n ELSIF TG_TABLE_NAME = 'items_staging_ignore' THEN\n INSERT INTO items\n SELECT\n (content_dehydrate(content)).*\n FROM newdata\n ON CONFLICT DO NOTHING;\n RAISE NOTICE 'Doing the delete. %', clock_timestamp() - ts;\n DELETE FROM items_staging_ignore;\n ELSIF TG_TABLE_NAME = 'items_staging_upsert' THEN\n WITH staging_formatted AS (\n SELECT (content_dehydrate(content)).* FROM newdata\n ), deletes AS (\n DELETE FROM items i USING staging_formatted s\n WHERE\n i.id = s.id\n AND i.collection = s.collection\n AND i IS DISTINCT FROM s\n RETURNING i.id, i.collection\n )\n INSERT INTO items\n SELECT s.* FROM\n staging_formatted s\n ON CONFLICT DO NOTHING;\n RAISE NOTICE 'Doing the delete. %', clock_timestamp() - ts;\n DELETE FROM items_staging_upsert;\n END IF;\n RAISE NOTICE 'Done. %', clock_timestamp() - ts;\n\n RETURN NULL;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER items_staging_insert_trigger AFTER INSERT ON items_staging REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\nCREATE TRIGGER items_staging_insert_ignore_trigger AFTER INSERT ON items_staging_ignore REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\nCREATE TRIGGER items_staging_insert_upsert_trigger AFTER INSERT ON items_staging_upsert REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\n\nCREATE OR REPLACE FUNCTION item_by_id(_id text, _collection text DEFAULT NULL) RETURNS items AS\n$$\nDECLARE\n i items%ROWTYPE;\nBEGIN\n SELECT * INTO i FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection) LIMIT 1;\n RETURN i;\nEND;\n$$ LANGUAGE PLPGSQL STABLE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION get_item(_id text, _collection text DEFAULT NULL) RETURNS jsonb AS $$\n SELECT content_hydrate(items) FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection);\n$$ LANGUAGE SQL STABLE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION delete_item(_id text, _collection text DEFAULT NULL) RETURNS VOID AS $$\nDECLARE\nout items%ROWTYPE;\nBEGIN\n DELETE FROM items WHERE id = _id AND (_collection IS NULL OR collection=_collection) RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n--/*\nCREATE OR REPLACE FUNCTION create_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION update_item(content jsonb) RETURNS VOID AS $$\nDECLARE\n old items %ROWTYPE;\n out items%ROWTYPE;\nBEGIN\n PERFORM delete_item(content->>'id', content->>'collection');\n PERFORM create_item(content);\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION create_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION collection_bbox(id text) RETURNS jsonb AS $$\n SELECT (replace(replace(replace(st_extent(geometry)::text,'BOX(','[['),')',']]'),' ',','))::jsonb\n FROM items WHERE collection=$1;\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION collection_temporal_extent(id text) RETURNS jsonb AS $$\n SELECT to_jsonb(array[array[min(datetime)::text, max(datetime)::text]])\n FROM items WHERE collection=$1;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection_extents() RETURNS VOID AS $$\nUPDATE collections SET\n content = content ||\n jsonb_build_object(\n 'extent', jsonb_build_object(\n 'spatial', jsonb_build_object(\n 'bbox', collection_bbox(collections.id)\n ),\n 'temporal', jsonb_build_object(\n 'interval', collection_temporal_extent(collections.id)\n )\n )\n )\n;\n$$ LANGUAGE SQL;\nCREATE TABLE partition_stats (\n partition text PRIMARY KEY,\n dtrange tstzrange,\n edtrange tstzrange,\n spatial geometry,\n last_updated timestamptz,\n keys text[]\n) WITH (FILLFACTOR=90);\n\nCREATE INDEX partitions_range_idx ON partition_stats USING GIST(dtrange);\n\n\nCREATE OR REPLACE FUNCTION constraint_tstzrange(expr text) RETURNS tstzrange AS $$\n WITH t AS (\n SELECT regexp_matches(\n expr,\n E'\\\\(''\\([0-9 :+-]*\\)''\\\\).*\\\\(''\\([0-9 :+-]*\\)''\\\\)'\n ) AS m\n ) SELECT tstzrange(m[1]::timestamptz, m[2]::timestamptz) FROM t\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE STRICT;\n\nCREATE OR REPLACE FUNCTION dt_constraint(coid oid, OUT dt tstzrange, OUT edt tstzrange) RETURNS RECORD AS $$\nDECLARE\n expr text := pg_get_constraintdef(coid);\n matches timestamptz[];\nBEGIN\n IF expr LIKE '%NULL%' THEN\n dt := tstzrange(null::timestamptz, null::timestamptz);\n edt := tstzrange(null::timestamptz, null::timestamptz);\n RETURN;\n END IF;\n WITH f AS (SELECT (regexp_matches(expr, E'([0-9]{4}-[0-1][0-9]-[0-3][0-9] [0-2][0-9]:[0-5][0-9]:[0-5][0-9]\\.?[0-9]*)', 'g'))[1] f)\n SELECT array_agg(f::timestamptz) INTO matches FROM f;\n IF cardinality(matches) = 4 THEN\n dt := tstzrange(matches[1], matches[2],'[]');\n edt := tstzrange(matches[3], matches[4], '[]');\n RETURN;\n ELSIF cardinality(matches) = 2 THEN\n edt := tstzrange(matches[1], matches[2],'[]');\n RETURN;\n END IF;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\nCREATE OR REPLACE VIEW partition_sys_meta AS\nSELECT\n relid::text as partition,\n replace(replace(CASE WHEN level = 1 THEN pg_get_expr(c.relpartbound, c.oid)\n ELSE pg_get_expr(parent.relpartbound, parent.oid)\n END, 'FOR VALUES IN (''',''), ''')','') AS collection,\n level,\n c.reltuples,\n c.relhastriggers,\n COALESCE(constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as partition_dtrange,\n COALESCE((dt_constraint(edt.oid)).dt, constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as constraint_dtrange,\n COALESCE((dt_constraint(edt.oid)).edt, tstzrange('-infinity', 'infinity','[]')) as constraint_edtrange\nFROM\n pg_partition_tree('items')\n JOIN pg_class c ON (relid::regclass = c.oid)\n JOIN pg_class parent ON (parentrelid::regclass = parent.oid AND isleaf)\n LEFT JOIN pg_constraint edt ON (conrelid=c.oid AND contype='c')\nWHERE isleaf\n;\n\nCREATE VIEW partitions_view AS\nSELECT\n relid::text as partition,\n replace(replace(CASE WHEN level = 1 THEN pg_get_expr(c.relpartbound, c.oid)\n ELSE pg_get_expr(parent.relpartbound, parent.oid)\n END, 'FOR VALUES IN (''',''), ''')','') AS collection,\n level,\n c.reltuples,\n c.relhastriggers,\n COALESCE(constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as partition_dtrange,\n COALESCE((dt_constraint(edt.oid)).dt, constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as constraint_dtrange,\n COALESCE((dt_constraint(edt.oid)).edt, tstzrange('-infinity', 'infinity','[]')) as constraint_edtrange,\n dtrange,\n edtrange,\n spatial,\n last_updated\nFROM\n pg_partition_tree('items')\n JOIN pg_class c ON (relid::regclass = c.oid)\n JOIN pg_class parent ON (parentrelid::regclass = parent.oid AND isleaf)\n LEFT JOIN pg_constraint edt ON (conrelid=c.oid AND contype='c')\n LEFT JOIN partition_stats ON (relid::text=partition)\nWHERE isleaf\n;\n\nCREATE MATERIALIZED VIEW partitions AS\nSELECT * FROM partitions_view;\nCREATE UNIQUE INDEX ON partitions (partition);\n\nCREATE MATERIALIZED VIEW partition_steps AS\nSELECT\n partition as name,\n date_trunc('month',lower(partition_dtrange)) as sdate,\n date_trunc('month', upper(partition_dtrange)) + '1 month'::interval as edate\n FROM partitions_view WHERE partition_dtrange IS NOT NULL AND partition_dtrange != 'empty'::tstzrange\n ORDER BY dtrange ASC\n;\n\n\nCREATE OR REPLACE FUNCTION update_partition_stats_q(_partition text, istrigger boolean default false) RETURNS VOID AS $$\nDECLARE\nBEGIN\n PERFORM run_or_queue(\n format('SELECT update_partition_stats(%L, %L);', _partition, istrigger)\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION update_partition_stats(_partition text, istrigger boolean default false) RETURNS VOID AS $$\nDECLARE\n dtrange tstzrange;\n edtrange tstzrange;\n cdtrange tstzrange;\n cedtrange tstzrange;\n extent geometry;\n collection text;\nBEGIN\n RAISE NOTICE 'Updating stats for %.', _partition;\n EXECUTE format(\n $q$\n SELECT\n tstzrange(min(datetime), max(datetime),'[]'),\n tstzrange(min(end_datetime), max(end_datetime), '[]')\n FROM %I\n $q$,\n _partition\n ) INTO dtrange, edtrange;\n extent := st_estimatedextent('pgstac', _partition, 'geometry');\n INSERT INTO partition_stats (partition, dtrange, edtrange, spatial, last_updated)\n SELECT _partition, dtrange, edtrange, extent, now()\n ON CONFLICT (partition) DO\n UPDATE SET\n dtrange=EXCLUDED.dtrange,\n edtrange=EXCLUDED.edtrange,\n spatial=EXCLUDED.spatial,\n last_updated=EXCLUDED.last_updated\n ;\n\n SELECT\n constraint_dtrange, constraint_edtrange, pv.collection\n INTO cdtrange, cedtrange, collection\n FROM partitions_view pv WHERE partition = _partition;\n REFRESH MATERIALIZED VIEW partitions;\n REFRESH MATERIALIZED VIEW partition_steps;\n\n\n RAISE NOTICE 'Checking if we need to modify constraints.';\n IF\n (cdtrange IS DISTINCT FROM dtrange OR edtrange IS DISTINCT FROM cedtrange)\n AND NOT istrigger\n THEN\n RAISE NOTICE 'Modifying Constraints';\n RAISE NOTICE 'Existing % %', cdtrange, cedtrange;\n RAISE NOTICE 'New % %', dtrange, edtrange;\n PERFORM drop_table_constraints(_partition);\n PERFORM create_table_constraints(_partition, dtrange, edtrange);\n END IF;\n RAISE NOTICE 'Checking if we need to update collection extents.';\n IF get_setting_bool('update_collection_extent') THEN\n RAISE NOTICE 'updating collection extent for %', collection;\n PERFORM run_or_queue(format($q$\n UPDATE collections\n SET content = jsonb_set_lax(\n content,\n '{extent}'::text[],\n collection_extent(%L, FALSE),\n true,\n 'use_json_null'\n ) WHERE id=%L\n ;\n $q$, collection, collection));\n ELSE\n RAISE NOTICE 'Not updating collection extent for %', collection;\n END IF;\n\nEND;\n$$ LANGUAGE PLPGSQL STRICT;\n\n\nCREATE OR REPLACE FUNCTION partition_name( IN collection text, IN dt timestamptz, OUT partition_name text, OUT partition_range tstzrange) AS $$\nDECLARE\n c RECORD;\n parent_name text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n parent_name := format('_items_%s', c.key);\n\n\n IF c.partition_trunc = 'year' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYY'));\n ELSIF c.partition_trunc = 'month' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYYMM'));\n ELSE\n partition_name := parent_name;\n partition_range := tstzrange('-infinity'::timestamptz, 'infinity'::timestamptz, '[]');\n END IF;\n IF partition_range IS NULL THEN\n partition_range := tstzrange(\n date_trunc(c.partition_trunc::text, dt),\n date_trunc(c.partition_trunc::text, dt) + concat('1 ', c.partition_trunc)::interval\n );\n END IF;\n RETURN;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION drop_table_constraints(t text) RETURNS text AS $$\nDECLARE\n q text;\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM partitions_view WHERE partition=t) THEN\n RETURN NULL;\n END IF;\n FOR q IN SELECT FORMAT(\n $q$\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n $q$,\n t,\n conname\n ) FROM pg_constraint\n WHERE conrelid=t::regclass::oid AND contype='c'\n LOOP\n EXECUTE q;\n END LOOP;\n RETURN t;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE OR REPLACE FUNCTION create_table_constraints(t text, _dtrange tstzrange, _edtrange tstzrange) RETURNS text AS $$\nDECLARE\n q text;\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM partitions_view WHERE partition=t) THEN\n RETURN NULL;\n END IF;\n RAISE NOTICE 'Creating Table Constraints for % % %', t, _dtrange, _edtrange;\n IF _dtrange = 'empty' AND _edtrange = 'empty' THEN\n q :=format(\n $q$\n DO $block$\n BEGIN\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n ALTER TABLE %I\n ADD CONSTRAINT %I\n CHECK (((datetime IS NULL) AND (end_datetime IS NULL))) NOT VALID\n ;\n ALTER TABLE %I\n VALIDATE CONSTRAINT %I\n ;\n\n\n\n EXCEPTION WHEN others THEN\n RAISE WARNING '%%, Issue Altering Constraints. Please run update_partition_stats(%I)', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n $block$;\n $q$,\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t),\n t\n );\n ELSE\n q :=format(\n $q$\n DO $block$\n BEGIN\n\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n ALTER TABLE %I\n ADD CONSTRAINT %I\n CHECK (\n (datetime >= %L)\n AND (datetime <= %L)\n AND (end_datetime >= %L)\n AND (end_datetime <= %L)\n ) NOT VALID\n ;\n ALTER TABLE %I\n VALIDATE CONSTRAINT %I\n ;\n\n\n\n EXCEPTION WHEN others THEN\n RAISE WARNING '%%, Issue Altering Constraints. Please run update_partition_stats(%I)', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n $block$;\n $q$,\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t),\n lower(_dtrange),\n upper(_dtrange),\n lower(_edtrange),\n upper(_edtrange),\n t,\n format('%s_dt', t),\n t\n );\n END IF;\n PERFORM run_or_queue(q);\n RETURN t;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION check_partition(\n _collection text,\n _dtrange tstzrange,\n _edtrange tstzrange\n) RETURNS text AS $$\nDECLARE\n c RECORD;\n pm RECORD;\n _partition_name text;\n _partition_dtrange tstzrange;\n _constraint_dtrange tstzrange;\n _constraint_edtrange tstzrange;\n q text;\n deferrable_q text;\n err_context text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=_collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', _collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n\n IF c.partition_trunc IS NOT NULL THEN\n _partition_dtrange := tstzrange(\n date_trunc(c.partition_trunc, lower(_dtrange)),\n date_trunc(c.partition_trunc, lower(_dtrange)) + (concat('1 ', c.partition_trunc))::interval,\n '[)'\n );\n ELSE\n _partition_dtrange := '[-infinity, infinity]'::tstzrange;\n END IF;\n\n IF NOT _partition_dtrange @> _dtrange THEN\n RAISE EXCEPTION 'dtrange % is greater than the partition size % for collection %', _dtrange, c.partition_trunc, _collection;\n END IF;\n\n\n IF c.partition_trunc = 'year' THEN\n _partition_name := format('_items_%s_%s', c.key, to_char(lower(_partition_dtrange),'YYYY'));\n ELSIF c.partition_trunc = 'month' THEN\n _partition_name := format('_items_%s_%s', c.key, to_char(lower(_partition_dtrange),'YYYYMM'));\n ELSE\n _partition_name := format('_items_%s', c.key);\n END IF;\n\n SELECT * INTO pm FROM partition_sys_meta WHERE collection=_collection AND partition_dtrange @> _dtrange;\n IF FOUND THEN\n RAISE NOTICE '% % %', _edtrange, _dtrange, pm;\n _constraint_edtrange :=\n tstzrange(\n least(\n lower(_edtrange),\n nullif(lower(pm.constraint_edtrange), '-infinity')\n ),\n greatest(\n upper(_edtrange),\n nullif(upper(pm.constraint_edtrange), 'infinity')\n ),\n '[]'\n );\n _constraint_dtrange :=\n tstzrange(\n least(\n lower(_dtrange),\n nullif(lower(pm.constraint_dtrange), '-infinity')\n ),\n greatest(\n upper(_dtrange),\n nullif(upper(pm.constraint_dtrange), 'infinity')\n ),\n '[]'\n );\n\n IF pm.constraint_edtrange @> _edtrange AND pm.constraint_dtrange @> _dtrange THEN\n RETURN pm.partition;\n ELSE\n PERFORM drop_table_constraints(_partition_name);\n END IF;\n ELSE\n _constraint_edtrange := _edtrange;\n _constraint_dtrange := _dtrange;\n END IF;\n RAISE NOTICE 'Creating partition % %', _partition_name, _partition_dtrange;\n IF c.partition_trunc IS NULL THEN\n q := format(\n $q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n $q$,\n _partition_name,\n _collection,\n concat(_partition_name,'_pk'),\n _partition_name\n );\n ELSE\n q := format(\n $q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L) PARTITION BY RANGE (datetime);\n CREATE TABLE IF NOT EXISTS %I partition OF %I FOR VALUES FROM (%L) TO (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n $q$,\n format('_items_%s', c.key),\n _collection,\n _partition_name,\n format('_items_%s', c.key),\n lower(_partition_dtrange),\n upper(_partition_dtrange),\n format('%s_pk', _partition_name),\n _partition_name\n );\n END IF;\n\n BEGIN\n EXECUTE q;\n EXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', _partition_name;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\n END;\n PERFORM create_table_constraints(_partition_name, _constraint_dtrange, _constraint_edtrange);\n PERFORM maintain_partitions(_partition_name);\n PERFORM update_partition_stats_q(_partition_name, true);\n REFRESH MATERIALIZED VIEW partitions;\n REFRESH MATERIALIZED VIEW partition_steps;\n RETURN _partition_name;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION repartition(_collection text, _partition_trunc text, triggered boolean DEFAULT FALSE) RETURNS text AS $$\nDECLARE\n c RECORD;\n q text;\n from_trunc text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=_collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', _collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n IF triggered THEN\n RAISE NOTICE 'Converting % to % partitioning via Trigger', _collection, _partition_trunc;\n ELSE\n RAISE NOTICE 'Converting % from using % to % partitioning', _collection, c.partition_trunc, _partition_trunc;\n IF c.partition_trunc IS NOT DISTINCT FROM _partition_trunc THEN\n RAISE NOTICE 'Collection % already set to use partition by %', _collection, _partition_trunc;\n RETURN _collection;\n END IF;\n END IF;\n\n IF EXISTS (SELECT 1 FROM partitions_view WHERE collection=_collection LIMIT 1) THEN\n EXECUTE format(\n $q$\n CREATE TEMP TABLE changepartitionstaging ON COMMIT DROP AS SELECT * FROM %I;\n DROP TABLE IF EXISTS %I CASCADE;\n WITH p AS (\n SELECT\n collection,\n CASE WHEN %L IS NULL THEN '-infinity'::timestamptz\n ELSE date_trunc(%L, datetime)\n END as d,\n tstzrange(min(datetime),max(datetime),'[]') as dtrange,\n tstzrange(min(datetime),max(datetime),'[]') as edtrange\n FROM changepartitionstaging\n GROUP BY 1,2\n ) SELECT check_partition(collection, dtrange, edtrange) FROM p;\n INSERT INTO items SELECT * FROM changepartitionstaging;\n DROP TABLE changepartitionstaging;\n $q$,\n concat('_items_', c.key),\n concat('_items_', c.key),\n c.partition_trunc,\n c.partition_trunc\n );\n END IF;\n RETURN _collection;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE OR REPLACE FUNCTION collections_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n q text;\n partition_name text := format('_items_%s', NEW.key);\n partition_exists boolean := false;\n partition_empty boolean := true;\n err_context text;\n loadtemp boolean := FALSE;\nBEGIN\n RAISE NOTICE 'Collection Trigger. % %', NEW.id, NEW.key;\n IF TG_OP = 'UPDATE' AND NEW.partition_trunc IS DISTINCT FROM OLD.partition_trunc THEN\n PERFORM repartition(NEW.id, NEW.partition_trunc, TRUE);\n END IF;\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER collections_trigger AFTER\nINSERT\nOR\nUPDATE ON collections\nFOR EACH ROW EXECUTE FUNCTION collections_trigger_func();\n\nCREATE OR REPLACE FUNCTION chunker(\n IN _where text,\n OUT s timestamptz,\n OUT e timestamptz\n) RETURNS SETOF RECORD AS $$\nDECLARE\n explain jsonb;\nBEGIN\n IF _where IS NULL THEN\n _where := ' TRUE ';\n END IF;\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s;', _where)\n INTO explain;\n\n RETURN QUERY\n WITH t AS (\n SELECT j->>0 as p FROM\n jsonb_path_query(\n explain,\n 'strict $.**.\"Relation Name\" ? (@ != null)'\n ) j\n ),\n parts AS (\n SELECT sdate, edate FROM t JOIN partition_steps ON (t.p = name)\n ),\n times AS (\n SELECT sdate FROM parts\n UNION\n SELECT edate FROM parts\n ),\n uniq AS (\n SELECT DISTINCT sdate FROM times ORDER BY sdate\n ),\n last AS (\n SELECT sdate, lead(sdate, 1) over () as edate FROM uniq\n )\n SELECT sdate, edate FROM last WHERE edate IS NOT NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION partition_queries(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN partitions text[] DEFAULT NULL\n) RETURNS SETOF text AS $$\nDECLARE\n query text;\n sdate timestamptz;\n edate timestamptz;\nBEGIN\nIF _where IS NULL OR trim(_where) = '' THEN\n _where = ' TRUE ';\nEND IF;\nRAISE NOTICE 'Getting chunks for % %', _where, _orderby;\nIF _orderby ILIKE 'datetime d%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 DESC LOOP\n RETURN NEXT format($q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n $q$,\n sdate,\n edate,\n _where,\n _orderby\n );\n END LOOP;\nELSIF _orderby ILIKE 'datetime a%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 ASC LOOP\n RETURN NEXT format($q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n $q$,\n sdate,\n edate,\n _where,\n _orderby\n );\n END LOOP;\nELSE\n query := format($q$\n SELECT * FROM items\n WHERE %s\n ORDER BY %s\n $q$, _where, _orderby\n );\n\n RETURN NEXT query;\n RETURN;\nEND IF;\n\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION partition_query_view(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN _limit int DEFAULT 10\n) RETURNS text AS $$\n WITH p AS (\n SELECT * FROM partition_queries(_where, _orderby) p\n )\n SELECT\n CASE WHEN EXISTS (SELECT 1 FROM p) THEN\n (SELECT format($q$\n SELECT * FROM (\n %s\n ) total LIMIT %s\n $q$,\n string_agg(\n format($q$ SELECT * FROM ( %s ) AS sub $q$, p),\n '\n UNION ALL\n '\n ),\n _limit\n ))\n ELSE NULL\n END FROM p;\n$$ LANGUAGE SQL IMMUTABLE;\n\n\n\n\nCREATE OR REPLACE FUNCTION stac_search_to_where(j jsonb) RETURNS text AS $$\nDECLARE\n where_segments text[];\n _where text;\n dtrange tstzrange;\n collections text[];\n geom geometry;\n sdate timestamptz;\n edate timestamptz;\n filterlang text;\n filter jsonb := j->'filter';\nBEGIN\n IF j ? 'ids' THEN\n where_segments := where_segments || format('id = ANY (%L) ', to_text_array(j->'ids'));\n END IF;\n\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n where_segments := where_segments || format('collection = ANY (%L) ', collections);\n END IF;\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n\n where_segments := where_segments || format(' datetime <= %L::timestamptz AND end_datetime >= %L::timestamptz ',\n edate,\n sdate\n );\n END IF;\n\n geom := stac_geom(j);\n IF geom IS NOT NULL THEN\n where_segments := where_segments || format('st_intersects(geometry, %L)',geom);\n END IF;\n\n filterlang := COALESCE(\n j->>'filter-lang',\n get_setting('default_filter_lang', j->'conf')\n );\n IF NOT filter @? '$.**.op' THEN\n filterlang := 'cql-json';\n END IF;\n\n IF filterlang NOT IN ('cql-json','cql2-json') AND j ? 'filter' THEN\n RAISE EXCEPTION '% is not a supported filter-lang. Please use cql-json or cql2-json.', filterlang;\n END IF;\n\n IF j ? 'query' AND j ? 'filter' THEN\n RAISE EXCEPTION 'Can only use either query or filter at one time.';\n END IF;\n\n IF j ? 'query' THEN\n filter := query_to_cql2(j->'query');\n ELSIF filterlang = 'cql-json' THEN\n filter := cql1_to_cql2(filter);\n END IF;\n RAISE NOTICE 'FILTER: %', filter;\n where_segments := where_segments || cql2_query(filter);\n IF cardinality(where_segments) < 1 THEN\n RETURN ' TRUE ';\n END IF;\n\n _where := array_to_string(array_remove(where_segments, NULL), ' AND ');\n\n IF _where IS NULL OR BTRIM(_where) = '' THEN\n RETURN ' TRUE ';\n END IF;\n RETURN _where;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION parse_sort_dir(_dir text, reverse boolean default false) RETURNS text AS $$\n WITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n ) SELECT\n CASE\n WHEN NOT reverse THEN d\n WHEN d = 'ASC' THEN 'DESC'\n WHEN d = 'DESC' THEN 'ASC'\n END\n FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION sort_dir_to_op(_dir text, prev boolean default false) RETURNS text AS $$\n WITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n ) SELECT\n CASE\n WHEN d = 'ASC' AND prev THEN '<='\n WHEN d = 'DESC' AND prev THEN '>='\n WHEN d = 'ASC' THEN '>='\n WHEN d = 'DESC' THEN '<='\n END\n FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION sort_sqlorderby(\n _search jsonb DEFAULT NULL,\n reverse boolean DEFAULT FALSE\n) RETURNS text AS $$\n WITH sortby AS (\n SELECT coalesce(_search->'sortby','[{\"field\":\"datetime\", \"direction\":\"desc\"}]') as sort\n ), withid AS (\n SELECT CASE\n WHEN sort @? '$[*] ? (@.field == \"id\")' THEN sort\n ELSE sort || '[{\"field\":\"id\", \"direction\":\"desc\"}]'::jsonb\n END as sort\n FROM sortby\n ), withid_rows AS (\n SELECT jsonb_array_elements(sort) as value FROM withid\n ),sorts AS (\n SELECT\n coalesce(\n (queryable(value->>'field')).expression\n ) as key,\n parse_sort_dir(value->>'direction', reverse) as dir\n FROM withid_rows\n )\n SELECT array_to_string(\n array_agg(concat(key, ' ', dir)),\n ', '\n ) FROM sorts;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_sort_dir(sort_item jsonb) RETURNS text AS $$\n SELECT CASE WHEN sort_item->>'direction' ILIKE 'desc%' THEN 'DESC' ELSE 'ASC' END;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION get_token_val_str(\n _field text,\n _item items\n) RETURNS text AS $$\nDECLARE\n q text;\n literal text;\nBEGIN\n q := format($q$ SELECT quote_literal(%s) FROM (SELECT $1.*) as r;$q$, _field);\n EXECUTE q INTO literal USING _item;\n RETURN literal;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n\nCREATE OR REPLACE FUNCTION get_token_record(IN _token text, OUT prev BOOLEAN, OUT item items) RETURNS RECORD AS $$\nDECLARE\n _itemid text := _token;\n _collectionid text;\nBEGIN\n IF _token IS NULL THEN\n RETURN;\n END IF;\n RAISE NOTICE 'Looking for token: %', _token;\n prev := FALSE;\n IF _token ILIKE 'prev:%' THEN\n _itemid := replace(_token, 'prev:','');\n prev := TRUE;\n ELSIF _token ILIKE 'next:%' THEN\n _itemid := replace(_token, 'next:', '');\n END IF;\n SELECT id INTO _collectionid FROM collections WHERE _itemid LIKE concat(id,':%');\n IF FOUND THEN\n _itemid := replace(_itemid, concat(_collectionid,':'), '');\n SELECT * INTO item FROM items WHERE id=_itemid AND collection=_collectionid;\n ELSE\n SELECT * INTO item FROM items WHERE id=_itemid;\n END IF;\n IF item IS NULL THEN\n RAISE EXCEPTION 'Could not find item using token: % item: % collection: %', _token, _itemid, _collectionid;\n END IF;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\n\nCREATE OR REPLACE FUNCTION get_token_filter(\n _sortby jsonb DEFAULT '[{\"field\":\"datetime\",\"direction\":\"desc\"}]'::jsonb,\n token_item items DEFAULT NULL,\n prev boolean DEFAULT FALSE,\n inclusive boolean DEFAULT FALSE\n) RETURNS text AS $$\nDECLARE\n ltop text := '<';\n gtop text := '>';\n dir text;\n sort record;\n orfilter text := '';\n orfilters text[] := '{}'::text[];\n andfilters text[] := '{}'::text[];\n output text;\n token_where text;\nBEGIN\n IF _sortby IS NULL OR _sortby = '[]'::jsonb THEN\n _sortby := '[{\"field\":\"datetime\",\"direction\":\"desc\"}]'::jsonb;\n END IF;\n _sortby := _sortby || jsonb_build_object('field','id','direction',_sortby->0->>'direction');\n RAISE NOTICE 'Getting Token Filter. % %', _sortby, token_item;\n IF inclusive THEN\n orfilters := orfilters || format('( id=%L AND collection=%L )' , token_item.id, token_item.collection);\n END IF;\n\n FOR sort IN\n WITH s1 AS (\n SELECT\n _row,\n (queryable(value->>'field')).expression as _field,\n (value->>'field' = 'id') as _isid,\n get_sort_dir(value) as _dir\n FROM jsonb_array_elements(_sortby)\n WITH ORDINALITY AS t(value, _row)\n )\n SELECT\n _row,\n _field,\n _dir,\n get_token_val_str(_field, token_item) as _val\n FROM s1\n WHERE _row <= (SELECT min(_row) FROM s1 WHERE _isid)\n LOOP\n orfilter := NULL;\n RAISE NOTICE 'SORT: %', sort;\n IF sort._val IS NOT NULL AND ((prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC')) THEN\n orfilter := format($f$(\n (%s %s %s) OR (%s IS NULL)\n )$f$,\n sort._field,\n ltop,\n sort._val,\n sort._val\n );\n ELSIF sort._val IS NULL AND ((prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC')) THEN\n RAISE NOTICE '< but null';\n orfilter := format('%s IS NOT NULL', sort._field);\n ELSIF sort._val IS NULL THEN\n RAISE NOTICE '> but null';\n ELSE\n orfilter := format($f$(\n (%s %s %s) OR (%s IS NULL)\n )$f$,\n sort._field,\n gtop,\n sort._val,\n sort._field\n );\n END IF;\n RAISE NOTICE 'ORFILTER: %', orfilter;\n\n IF orfilter IS NOT NULL THEN\n IF sort._row = 1 THEN\n orfilters := orfilters || orfilter;\n ELSE\n orfilters := orfilters || format('(%s AND %s)', array_to_string(andfilters, ' AND '), orfilter);\n END IF;\n END IF;\n IF sort._val IS NOT NULL THEN\n andfilters := andfilters || format('%s = %s', sort._field, sort._val);\n ELSE\n andfilters := andfilters || format('%s IS NULL', sort._field);\n END IF;\n END LOOP;\n\n output := array_to_string(orfilters, ' OR ');\n\n token_where := concat('(',coalesce(output,'true'),')');\n IF trim(token_where) = '' THEN\n token_where := NULL;\n END IF;\n RAISE NOTICE 'TOKEN_WHERE: %',token_where;\n RETURN token_where;\n END;\n$$ LANGUAGE PLPGSQL SET transform_null_equals TO TRUE\n;\n\nCREATE OR REPLACE FUNCTION search_tohash(jsonb) RETURNS jsonb AS $$\n SELECT $1 - '{token,limit,context,includes,excludes}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION search_hash(jsonb, jsonb) RETURNS text AS $$\n SELECT md5(concat(search_tohash($1)::text,$2::text));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TABLE IF NOT EXISTS searches(\n hash text GENERATED ALWAYS AS (search_hash(search, metadata)) STORED PRIMARY KEY,\n search jsonb NOT NULL,\n _where text,\n orderby text,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n metadata jsonb DEFAULT '{}'::jsonb NOT NULL\n);\n\nCREATE TABLE IF NOT EXISTS search_wheres(\n id bigint generated always as identity primary key,\n _where text NOT NULL,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n statslastupdated timestamptz,\n estimated_count bigint,\n estimated_cost float,\n time_to_estimate float,\n total_count bigint,\n time_to_count float,\n partitions text[]\n);\n\nCREATE INDEX IF NOT EXISTS search_wheres_partitions ON search_wheres USING GIN (partitions);\nCREATE UNIQUE INDEX IF NOT EXISTS search_wheres_where ON search_wheres ((md5(_where)));\n\nCREATE OR REPLACE FUNCTION where_stats(inwhere text, updatestats boolean default false, conf jsonb default null) RETURNS search_wheres AS $$\nDECLARE\n t timestamptz;\n i interval;\n explain_json jsonb;\n partitions text[];\n sw search_wheres%ROWTYPE;\n inwhere_hash text := md5(inwhere);\n _context text := lower(context(conf));\n _stats_ttl interval := context_stats_ttl(conf);\n _estimated_cost float := context_estimated_cost(conf);\n _estimated_count int := context_estimated_count(conf);\nBEGIN\n IF _context = 'off' THEN\n sw._where := inwhere;\n return sw;\n END IF;\n\n SELECT * INTO sw FROM search_wheres WHERE md5(_where)=inwhere_hash FOR UPDATE;\n\n -- Update statistics if explicitly set, if statistics do not exist, or statistics ttl has expired\n IF NOT updatestats THEN\n RAISE NOTICE 'Checking if update is needed for: % .', inwhere;\n RAISE NOTICE 'Stats Last Updated: %', sw.statslastupdated;\n RAISE NOTICE 'TTL: %, Age: %', _stats_ttl, now() - sw.statslastupdated;\n RAISE NOTICE 'Context: %, Existing Total: %', _context, sw.total_count;\n IF\n sw.statslastupdated IS NULL\n OR (now() - sw.statslastupdated) > _stats_ttl\n OR (context(conf) != 'off' AND sw.total_count IS NULL)\n THEN\n updatestats := TRUE;\n END IF;\n END IF;\n\n sw._where := inwhere;\n sw.lastused := now();\n sw.usecount := coalesce(sw.usecount,0) + 1;\n\n IF NOT updatestats THEN\n UPDATE search_wheres SET\n lastused = sw.lastused,\n usecount = sw.usecount\n WHERE md5(_where) = inwhere_hash\n RETURNING * INTO sw\n ;\n RETURN sw;\n END IF;\n\n -- Use explain to get estimated count/cost and a list of the partitions that would be hit by the query\n t := clock_timestamp();\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s', inwhere)\n INTO explain_json;\n RAISE NOTICE 'Time for just the explain: %', clock_timestamp() - t;\n i := clock_timestamp() - t;\n\n sw.statslastupdated := now();\n sw.estimated_count := explain_json->0->'Plan'->'Plan Rows';\n sw.estimated_cost := explain_json->0->'Plan'->'Total Cost';\n sw.time_to_estimate := extract(epoch from i);\n\n RAISE NOTICE 'ESTIMATED_COUNT: % < %', sw.estimated_count, _estimated_count;\n RAISE NOTICE 'ESTIMATED_COST: % < %', sw.estimated_cost, _estimated_cost;\n\n -- Do a full count of rows if context is set to on or if auto is set and estimates are low enough\n IF\n _context = 'on'\n OR\n ( _context = 'auto' AND\n (\n sw.estimated_count < _estimated_count\n AND\n sw.estimated_cost < _estimated_cost\n )\n )\n THEN\n t := clock_timestamp();\n RAISE NOTICE 'Calculating actual count...';\n EXECUTE format(\n 'SELECT count(*) FROM items WHERE %s',\n inwhere\n ) INTO sw.total_count;\n i := clock_timestamp() - t;\n RAISE NOTICE 'Actual Count: % -- %', sw.total_count, i;\n sw.time_to_count := extract(epoch FROM i);\n ELSE\n sw.total_count := NULL;\n sw.time_to_count := NULL;\n END IF;\n\n\n INSERT INTO search_wheres\n (_where, lastused, usecount, statslastupdated, estimated_count, estimated_cost, time_to_estimate, partitions, total_count, time_to_count)\n SELECT sw._where, sw.lastused, sw.usecount, sw.statslastupdated, sw.estimated_count, sw.estimated_cost, sw.time_to_estimate, sw.partitions, sw.total_count, sw.time_to_count\n ON CONFLICT ((md5(_where)))\n DO UPDATE\n SET\n lastused = sw.lastused,\n usecount = sw.usecount,\n statslastupdated = sw.statslastupdated,\n estimated_count = sw.estimated_count,\n estimated_cost = sw.estimated_cost,\n time_to_estimate = sw.time_to_estimate,\n total_count = sw.total_count,\n time_to_count = sw.time_to_count\n ;\n RETURN sw;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION search_query(\n _search jsonb = '{}'::jsonb,\n updatestats boolean = false,\n _metadata jsonb = '{}'::jsonb\n) RETURNS searches AS $$\nDECLARE\n search searches%ROWTYPE;\n pexplain jsonb;\n t timestamptz;\n i interval;\n _hash text := search_hash(_search, _metadata);\n doupdate boolean := FALSE;\n insertfound boolean := FALSE;\nBEGIN\n SELECT * INTO search FROM searches\n WHERE hash=_hash;\n\n search.hash := _hash;\n\n -- Calculate the where clause if not already calculated\n IF search._where IS NULL THEN\n search._where := stac_search_to_where(_search);\n ELSE\n doupdate := TRUE;\n END IF;\n\n -- Calculate the order by clause if not already calculated\n IF search.orderby IS NULL THEN\n search.orderby := sort_sqlorderby(_search);\n ELSE\n doupdate := TRUE;\n END IF;\n\n PERFORM where_stats(search._where, updatestats, _search->'conf');\n\n IF NOT doupdate THEN\n INSERT INTO searches (search, _where, orderby, lastused, usecount, metadata)\n VALUES (_search, search._where, search.orderby, clock_timestamp(), 1, _metadata)\n ON CONFLICT (hash) DO NOTHING RETURNING * INTO search;\n IF FOUND THEN\n RETURN search;\n END IF;\n END IF;\n\n UPDATE searches\n SET\n lastused=clock_timestamp(),\n usecount=usecount+1\n WHERE hash=(\n SELECT hash FROM searches\n WHERE hash=_hash\n FOR UPDATE SKIP LOCKED\n );\n IF NOT FOUND THEN\n RAISE NOTICE 'Did not update stats for % due to lock. (This is generally OK)', _search;\n END IF;\n\n RETURN search;\n\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE OR REPLACE FUNCTION search_rows(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN partitions text[] DEFAULT NULL,\n IN _limit int DEFAULT 10\n) RETURNS SETOF items AS $$\nDECLARE\n base_query text;\n query text;\n sdate timestamptz;\n edate timestamptz;\n n int;\n records_left int := _limit;\n timer timestamptz := clock_timestamp();\n full_timer timestamptz := clock_timestamp();\nBEGIN\nIF _where IS NULL OR trim(_where) = '' THEN\n _where = ' TRUE ';\nEND IF;\nRAISE NOTICE 'Getting chunks for % %', _where, _orderby;\n\nbase_query := $q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n LIMIT %L\n$q$;\n\nIF _orderby ILIKE 'datetime d%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 DESC LOOP\n RAISE NOTICE 'Running Query for % to %. %', sdate, edate, age_ms(full_timer);\n query := format(\n base_query,\n sdate,\n edate,\n _where,\n _orderby,\n records_left\n );\n RAISE LOG 'QUERY: %', query;\n timer := clock_timestamp();\n RETURN QUERY EXECUTE query;\n\n GET DIAGNOSTICS n = ROW_COUNT;\n records_left := records_left - n;\n RAISE NOTICE 'Returned %/% Rows From % to %. % to go. Time: %ms', n, _limit, sdate, edate, records_left, age_ms(timer);\n timer := clock_timestamp();\n IF records_left <= 0 THEN\n RAISE NOTICE 'SEARCH_ROWS TOOK %ms', age_ms(full_timer);\n RETURN;\n END IF;\n END LOOP;\nELSIF _orderby ILIKE 'datetime a%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 ASC LOOP\n RAISE NOTICE 'Running Query for % to %. %', sdate, edate, age_ms(full_timer);\n query := format(\n base_query,\n sdate,\n edate,\n _where,\n _orderby,\n records_left\n );\n RAISE LOG 'QUERY: %', query;\n timer := clock_timestamp();\n RETURN QUERY EXECUTE query;\n\n GET DIAGNOSTICS n = ROW_COUNT;\n records_left := records_left - n;\n RAISE NOTICE 'Returned %/% Rows From % to %. % to go. Time: %ms', n, _limit, sdate, edate, records_left, age_ms(timer);\n timer := clock_timestamp();\n IF records_left <= 0 THEN\n RAISE NOTICE 'SEARCH_ROWS TOOK %ms', age_ms(full_timer);\n RETURN;\n END IF;\n END LOOP;\nELSE\n query := format($q$\n SELECT * FROM items\n WHERE %s\n ORDER BY %s\n LIMIT %L\n $q$, _where, _orderby, _limit\n );\n RAISE LOG 'QUERY: %', query;\n timer := clock_timestamp();\n RETURN QUERY EXECUTE query;\n RAISE NOTICE 'FULL QUERY TOOK %ms', age_ms(timer);\nEND IF;\nRAISE NOTICE 'SEARCH_ROWS TOOK %ms', age_ms(full_timer);\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE UNLOGGED TABLE format_item_cache(\n id text,\n collection text,\n fields text,\n hydrated bool,\n output jsonb,\n lastused timestamptz DEFAULT now(),\n usecount int DEFAULT 1,\n timetoformat float,\n PRIMARY KEY (collection, id, fields, hydrated)\n);\nCREATE INDEX ON format_item_cache (lastused);\n\nCREATE OR REPLACE FUNCTION format_item(_item items, _fields jsonb DEFAULT '{}', _hydrated bool DEFAULT TRUE) RETURNS jsonb AS $$\nDECLARE\n cache bool := get_setting_bool('format_cache');\n _output jsonb := null;\n t timestamptz := clock_timestamp();\nBEGIN\n IF cache THEN\n SELECT output INTO _output FROM format_item_cache\n WHERE id=_item.id AND collection=_item.collection AND fields=_fields::text AND hydrated=_hydrated;\n END IF;\n IF _output IS NULL THEN\n IF _hydrated THEN\n _output := content_hydrate(_item, _fields);\n ELSE\n _output := content_nonhydrated(_item, _fields);\n END IF;\n END IF;\n IF cache THEN\n INSERT INTO format_item_cache (id, collection, fields, hydrated, output, timetoformat)\n VALUES (_item.id, _item.collection, _fields::text, _hydrated, _output, age_ms(t))\n ON CONFLICT(collection, id, fields, hydrated) DO\n UPDATE\n SET lastused=now(), usecount = format_item_cache.usecount + 1\n ;\n END IF;\n RETURN _output;\n\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION search(_search jsonb = '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n searches searches%ROWTYPE;\n _where text;\n orderby text;\n search_where search_wheres%ROWTYPE;\n total_count bigint;\n token record;\n token_prev boolean;\n token_item items%ROWTYPE;\n token_where text;\n full_where text;\n init_ts timestamptz := clock_timestamp();\n timer timestamptz := clock_timestamp();\n hydrate bool := NOT (_search->'conf'->>'nohydrate' IS NOT NULL AND (_search->'conf'->>'nohydrate')::boolean = true);\n prev text;\n next text;\n context jsonb;\n collection jsonb;\n out_records jsonb;\n out_len int;\n _limit int := coalesce((_search->>'limit')::int, 10);\n init_limit int := _limit;\n _querylimit int;\n _fields jsonb := coalesce(_search->'fields', '{}'::jsonb);\n has_prev boolean := FALSE;\n has_next boolean := FALSE;\n items_cnt int := coalesce(jsonb_array_length(_search->'ids'),0);\nBEGIN\n RAISE NOTICE 'Items Count: %', items_cnt;\n IF items_cnt > 0 THEN\n IF items_cnt <= _limit THEN\n _limit := items_cnt - 1;\n ELSE\n _limit := items_cnt;\n END IF;\n RAISE NOTICE 'Items is set. Changing limit to %', items_cnt;\n END IF;\n searches := search_query(_search);\n _where := searches._where;\n orderby := searches.orderby;\n search_where := where_stats(_where);\n total_count := coalesce(search_where.total_count, search_where.estimated_count);\n RAISE NOTICE 'SEARCH:TOKEN: %', _search->>'token';\n token := get_token_record(_search->>'token');\n RAISE NOTICE '***TOKEN: %', token;\n _querylimit := _limit + 1;\n IF token IS NOT NULL THEN\n token_prev := token.prev;\n token_item := token.item;\n token_where := get_token_filter(_search->'sortby', token_item, token_prev, FALSE);\n RAISE LOG 'TOKEN_WHERE: % (%ms from search start)', token_where, age_ms(timer);\n IF token_prev THEN -- if we are using a prev token, we know has_next is true\n RAISE LOG 'There is a previous token, so automatically setting has_next to true';\n has_next := TRUE;\n orderby := sort_sqlorderby(_search, TRUE);\n ELSE\n RAISE LOG 'There is a next token, so automatically setting has_prev to true';\n has_prev := TRUE;\n\n END IF;\n ELSE -- if there was no token, we know there is no prev\n RAISE LOG 'There is no token, so we know there is no prev. setting has_prev to false';\n has_prev := FALSE;\n END IF;\n\n full_where := concat_ws(' AND ', _where, token_where);\n RAISE NOTICE 'FULL WHERE CLAUSE: %', full_where;\n RAISE NOTICE 'Time to get counts and build query %', age_ms(timer);\n timer := clock_timestamp();\n\n IF hydrate THEN\n RAISE NOTICE 'Getting hydrated data.';\n ELSE\n RAISE NOTICE 'Getting non-hydrated data.';\n END IF;\n RAISE NOTICE 'CACHE SET TO %', get_setting_bool('format_cache');\n RAISE NOTICE 'Time to set hydration/formatting %', age_ms(timer);\n timer := clock_timestamp();\n SELECT jsonb_agg(format_item(i, _fields, hydrate)) INTO out_records\n FROM search_rows(\n full_where,\n orderby,\n search_where.partitions,\n _querylimit\n ) as i;\n\n RAISE NOTICE 'Time to fetch rows %', age_ms(timer);\n timer := clock_timestamp();\n\n\n IF token_prev THEN\n out_records := flip_jsonb_array(out_records);\n END IF;\n\n RAISE NOTICE 'Query returned % records.', jsonb_array_length(out_records);\n RAISE LOG 'TOKEN: % %', token_item.id, token_item.collection;\n RAISE LOG 'RECORD_1: % %', out_records->0->>'id', out_records->0->>'collection';\n RAISE LOG 'RECORD-1: % %', out_records->-1->>'id', out_records->-1->>'collection';\n\n -- REMOVE records that were from our token\n IF out_records->0->>'id' = token_item.id AND out_records->0->>'collection' = token_item.collection THEN\n out_records := out_records - 0;\n ELSIF out_records->-1->>'id' = token_item.id AND out_records->-1->>'collection' = token_item.collection THEN\n out_records := out_records - -1;\n END IF;\n\n out_len := jsonb_array_length(out_records);\n\n IF out_len = init_limit + 1 THEN\n IF token_prev THEN\n has_prev := TRUE;\n out_records := out_records - 0;\n ELSE\n has_next := TRUE;\n out_records := out_records - -1;\n END IF;\n END IF;\n\n IF has_next THEN\n next := concat(out_records->-1->>'collection', ':', out_records->-1->>'id');\n RAISE NOTICE 'HAS NEXT | %', next;\n END IF;\n\n IF has_prev THEN\n prev := concat(out_records->0->>'collection', ':', out_records->0->>'id');\n RAISE NOTICE 'HAS PREV | %', prev;\n END IF;\n\n RAISE NOTICE 'Time to get prev/next %', age_ms(timer);\n timer := clock_timestamp();\n\n IF context(_search->'conf') != 'off' THEN\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', init_limit,\n 'matched', total_count,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\n ELSE\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', init_limit,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\n END IF;\n\n collection := jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb),\n 'next', next,\n 'prev', prev,\n 'context', context\n );\n\n IF get_setting_bool('timing', _search->'conf') THEN\n collection = collection || jsonb_build_object('timing', age_ms(init_ts));\n END IF;\n\n RAISE NOTICE 'Time to build final json %', age_ms(timer);\n timer := clock_timestamp();\n\n RAISE NOTICE 'Total Time: %', age_ms(current_timestamp);\n RAISE NOTICE 'RETURNING % records. NEXT: %. PREV: %', collection->'context'->>'returned', collection->>'next', collection->>'prev';\n RETURN collection;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION search_cursor(_search jsonb = '{}'::jsonb) RETURNS refcursor AS $$\nDECLARE\n curs refcursor;\n searches searches%ROWTYPE;\n _where text;\n _orderby text;\n q text;\n\nBEGIN\n searches := search_query(_search);\n _where := searches._where;\n _orderby := searches.orderby;\n\n OPEN curs FOR\n WITH p AS (\n SELECT * FROM partition_queries(_where, _orderby) p\n )\n SELECT\n CASE WHEN EXISTS (SELECT 1 FROM p) THEN\n (SELECT format($q$\n SELECT * FROM (\n %s\n ) total\n $q$,\n string_agg(\n format($q$ SELECT * FROM ( %s ) AS sub $q$, p),\n '\n UNION ALL\n '\n )\n ))\n ELSE NULL\n END FROM p;\n RETURN curs;\nEND;\n$$ LANGUAGE PLPGSQL;\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION tileenvelope(zoom int, x int, y int) RETURNS geometry AS $$\nWITH t AS (\n SELECT\n 20037508.3427892 as merc_max,\n -20037508.3427892 as merc_min,\n (2 * 20037508.3427892) / (2 ^ zoom) as tile_size\n)\nSELECT st_makeenvelope(\n merc_min + (tile_size * x),\n merc_max - (tile_size * (y + 1)),\n merc_min + (tile_size * (x + 1)),\n merc_max - (tile_size * y),\n 3857\n) FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;DROP FUNCTION IF EXISTS mercgrid;\n\n\nCREATE OR REPLACE FUNCTION ftime() RETURNS interval as $$\nSELECT age(clock_timestamp(), transaction_timestamp());\n$$ LANGUAGE SQL;\nSET SEARCH_PATH to pgstac, public;\n\nDROP FUNCTION IF EXISTS geometrysearch;\nCREATE OR REPLACE FUNCTION geometrysearch(\n IN geom geometry,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE, -- Return as soon as the passed in geometry is full covered\n IN skipcovered boolean DEFAULT TRUE -- Skip any items that would show up completely under the previous items\n) RETURNS jsonb AS $$\nDECLARE\n search searches%ROWTYPE;\n curs refcursor;\n _where text;\n query text;\n iter_record items%ROWTYPE;\n out_records jsonb := '{}'::jsonb[];\n exit_flag boolean := FALSE;\n counter int := 1;\n scancounter int := 1;\n remaining_limit int := _scanlimit;\n tilearea float;\n unionedgeom geometry;\n clippedgeom geometry;\n unionedgeom_area float := 0;\n prev_area float := 0;\n excludes text[];\n includes text[];\n\nBEGIN\n DROP TABLE IF EXISTS pgstac_results;\n CREATE TEMP TABLE pgstac_results (content jsonb) ON COMMIT DROP;\n\n -- If skipcovered is true then you will always want to exit when the passed in geometry is full\n IF skipcovered THEN\n exitwhenfull := TRUE;\n END IF;\n\n SELECT * INTO search FROM searches WHERE hash=queryhash;\n\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Search with Query Hash % Not Found', queryhash;\n END IF;\n\n tilearea := st_area(geom);\n _where := format('%s AND st_intersects(geometry, %L::geometry)', search._where, geom);\n\n\n FOR query IN SELECT * FROM partition_queries(_where, search.orderby) LOOP\n query := format('%s LIMIT %L', query, remaining_limit);\n RAISE NOTICE '%', query;\n OPEN curs FOR EXECUTE query;\n LOOP\n FETCH curs INTO iter_record;\n EXIT WHEN NOT FOUND;\n IF exitwhenfull OR skipcovered THEN -- If we are not using exitwhenfull or skipcovered, we do not need to do expensive geometry operations\n clippedgeom := st_intersection(geom, iter_record.geometry);\n\n IF unionedgeom IS NULL THEN\n unionedgeom := clippedgeom;\n ELSE\n unionedgeom := st_union(unionedgeom, clippedgeom);\n END IF;\n\n unionedgeom_area := st_area(unionedgeom);\n\n IF skipcovered AND prev_area = unionedgeom_area THEN\n scancounter := scancounter + 1;\n CONTINUE;\n END IF;\n\n prev_area := unionedgeom_area;\n\n RAISE NOTICE '% % % %', unionedgeom_area/tilearea, counter, scancounter, ftime();\n END IF;\n RAISE NOTICE '% %', iter_record, content_hydrate(iter_record, fields);\n INSERT INTO pgstac_results (content) VALUES (content_hydrate(iter_record, fields));\n\n IF counter >= _limit\n OR scancounter > _scanlimit\n OR ftime() > _timelimit\n OR (exitwhenfull AND unionedgeom_area >= tilearea)\n THEN\n exit_flag := TRUE;\n EXIT;\n END IF;\n counter := counter + 1;\n scancounter := scancounter + 1;\n\n END LOOP;\n CLOSE curs;\n EXIT WHEN exit_flag;\n remaining_limit := _scanlimit - scancounter;\n END LOOP;\n\n SELECT jsonb_agg(content) INTO out_records FROM pgstac_results WHERE content IS NOT NULL;\n\n RETURN jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb)\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDROP FUNCTION IF EXISTS geojsonsearch;\nCREATE OR REPLACE FUNCTION geojsonsearch(\n IN geojson jsonb,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_geomfromgeojson(geojson),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS xyzsearch;\nCREATE OR REPLACE FUNCTION xyzsearch(\n IN _x int,\n IN _y int,\n IN _z int,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_transform(tileenvelope(_z, _x, _y), 4326),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE PROCEDURE analyze_items() AS $$\nDECLARE\n q text;\n timeout_ts timestamptz;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n SELECT format('ANALYZE (VERBOSE, SKIP_LOCKED) %I;', relname) INTO q\n FROM pg_stat_user_tables\n WHERE relname like '_item%' AND (n_mod_since_analyze>0 OR last_analyze IS NULL) LIMIT 1;\n IF NOT FOUND THEN\n EXIT;\n END IF;\n RAISE NOTICE '%', q;\n EXECUTE q;\n COMMIT;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE PROCEDURE validate_constraints() AS $$\nDECLARE\n q text;\nBEGIN\n FOR q IN\n SELECT\n FORMAT(\n 'ALTER TABLE %I.%I VALIDATE CONSTRAINT %I;',\n nsp.nspname,\n cls.relname,\n con.conname\n )\n\n FROM pg_constraint AS con\n JOIN pg_class AS cls\n ON con.conrelid = cls.oid\n JOIN pg_namespace AS nsp\n ON cls.relnamespace = nsp.oid\n WHERE convalidated = FALSE AND contype in ('c','f')\n AND nsp.nspname = 'pgstac'\n LOOP\n RAISE NOTICE '%', q;\n PERFORM run_or_queue(q);\n COMMIT;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION collection_extent(_collection text, runupdate boolean default false) RETURNS jsonb AS $$\nDECLARE\n geom_extent geometry;\n mind timestamptz;\n maxd timestamptz;\n extent jsonb;\nBEGIN\n IF runupdate THEN\n PERFORM update_partition_stats_q(partition)\n FROM partitions_view WHERE collection=_collection;\n END IF;\n SELECT\n min(lower(dtrange)),\n max(upper(edtrange)),\n st_extent(spatial)\n INTO\n mind,\n maxd,\n geom_extent\n FROM partitions_view\n WHERE collection=_collection;\n\n IF geom_extent IS NOT NULL AND mind IS NOT NULL AND maxd IS NOT NULL THEN\n extent := jsonb_build_object(\n 'extent', jsonb_build_object(\n 'spatial', jsonb_build_object(\n 'bbox', to_jsonb(array[array[st_xmin(geom_extent), st_ymin(geom_extent), st_xmax(geom_extent), st_ymax(geom_extent)]])\n ),\n 'temporal', jsonb_build_object(\n 'interval', to_jsonb(array[array[mind, maxd]])\n )\n )\n );\n RETURN extent;\n END IF;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('id', '{\"title\": \"Item ID\",\"description\": \"Item identifier\",\"$ref\": \"https://schemas.stacspec.org/v1.0.0/item-spec/json-schema/item.json#/definitions/core/allOf/2/properties/id\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('geometry', '{\"title\": \"Item Geometry\",\"description\": \"Item Geometry\",\"$ref\": \"https://geojson.org/schema/Feature.json\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('datetime','{\"description\": \"Datetime\",\"type\": \"string\",\"title\": \"Acquired\",\"format\": \"date-time\",\"pattern\": \"(\\\\+00:00|Z)$\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('eo:cloud_cover','{\"$ref\": \"https://stac-extensions.github.io/eo/v1.0.0/schema.json#/definitions/fieldsproperties/eo:cloud_cover\"}','to_int','BTREE');\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDELETE FROM queryables a USING queryables b\n WHERE a.name = b.name AND a.collection_ids IS NOT DISTINCT FROM b.collection_ids AND a.id > b.id;\n\n\nINSERT INTO pgstac_settings (name, value) VALUES\n ('context', 'off'),\n ('context_estimated_count', '100000'),\n ('context_estimated_cost', '100000'),\n ('context_stats_ttl', '1 day'),\n ('default_filter_lang', 'cql2-json'),\n ('additional_properties', 'true'),\n ('use_queue', 'false'),\n ('queue_timeout', '10 minutes'),\n ('update_collection_extent', 'false'),\n ('format_cache', 'false')\nON CONFLICT DO NOTHING\n;\n\nALTER FUNCTION to_text COST 5000;\nALTER FUNCTION to_float COST 5000;\nALTER FUNCTION to_int COST 5000;\nALTER FUNCTION to_tstz COST 5000;\nALTER FUNCTION to_text_array COST 5000;\n\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nGRANT ALL ON SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac to pgstac_admin;\n\n-- pgstac_read role limited to using function apis\nGRANT EXECUTE ON FUNCTION search TO pgstac_read;\nGRANT EXECUTE ON FUNCTION search_query TO pgstac_read;\nGRANT EXECUTE ON FUNCTION item_by_id TO pgstac_read;\nGRANT EXECUTE ON FUNCTION get_item TO pgstac_read;\nGRANT SELECT ON ALL TABLES IN SCHEMA pgstac TO pgstac_read;\n\n\nGRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_ingest;\nGRANT USAGE ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_ingest;\n\nSELECT update_partition_stats_q(partition) FROM partitions_view;\nSELECT set_version('0.7.10');\n" }, { "path": "src/pgstac/migrations/pgstac.0.7.2-0.7.3.sql", "content": "RESET ROLE;\nDO $$\nDECLARE\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='postgis') THEN\n CREATE EXTENSION IF NOT EXISTS postgis;\n END IF;\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='btree_gist') THEN\n CREATE EXTENSION IF NOT EXISTS btree_gist;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_admin;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_read;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_ingest;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n\nGRANT pgstac_admin TO current_user;\n\n-- Function to make sure pgstac_admin is the owner of items\nCREATE OR REPLACE FUNCTION pgstac_admin_owns() RETURNS VOID AS $$\nDECLARE\n f RECORD;\nBEGIN\n FOR f IN (\n SELECT\n concat(\n oid::regproc::text,\n '(',\n coalesce(pg_get_function_identity_arguments(oid),''),\n ')'\n ) AS name,\n CASE prokind WHEN 'f' THEN 'FUNCTION' WHEN 'p' THEN 'PROCEDURE' WHEN 'a' THEN 'AGGREGATE' END as typ\n FROM pg_proc\n WHERE\n pronamespace=to_regnamespace('pgstac')\n AND proowner != to_regrole('pgstac_admin')\n AND proname NOT LIKE 'pg_stat%'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n FOR f IN (\n SELECT\n oid::regclass::text as name,\n CASE relkind\n WHEN 'i' THEN 'INDEX'\n WHEN 'I' THEN 'INDEX'\n WHEN 'p' THEN 'TABLE'\n WHEN 'r' THEN 'TABLE'\n WHEN 'v' THEN 'VIEW'\n WHEN 'S' THEN 'SEQUENCE'\n ELSE NULL\n END as typ\n FROM pg_class\n WHERE relnamespace=to_regnamespace('pgstac') and relowner != to_regrole('pgstac_admin') AND relkind IN ('r','p','v','S') AND relname NOT LIKE 'pg_stat'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\nSELECT pgstac_admin_owns();\n\nCREATE SCHEMA IF NOT EXISTS pgstac AUTHORIZATION pgstac_admin;\n\nGRANT ALL ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_admin;\n\nALTER ROLE pgstac_admin SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_read SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_ingest SET SEARCH_PATH TO pgstac, public;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\n\nGRANT pgstac_read TO pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\n\nSET ROLE pgstac_admin;\n\nSET SEARCH_PATH TO pgstac, public;\n\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS analyze_items;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS validate_constraints;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\nSET client_min_messages TO WARNING;\nSET SEARCH_PATH to pgstac, public;\n-- BEGIN migra calculated SQL\ndrop view if exists \"pgstac\".\"pgstac_indexes\";\n\nset check_function_bodies = off;\n\nCREATE OR REPLACE FUNCTION pgstac.indexdef(q pgstac.queryables)\n RETURNS text\n LANGUAGE plpgsql\n IMMUTABLE\nAS $function$\n DECLARE\n out text;\n BEGIN\n IF q.name = 'id' THEN\n out := 'CREATE UNIQUE INDEX ON %I USING btree (id)';\n ELSIF q.name = 'datetime' THEN\n out := 'CREATE INDEX ON %I USING btree (datetime DESC, end_datetime)';\n ELSIF q.name = 'geometry' THEN\n out := 'CREATE INDEX ON %I USING gist (geometry)';\n ELSE\n out := format($q$CREATE INDEX ON %%I USING %s (%s(((content -> 'properties'::text) -> %L::text)))$q$,\n lower(COALESCE(q.property_index_type, 'BTREE')),\n lower(COALESCE(q.property_wrapper, 'to_text')),\n q.name\n );\n END IF;\n RETURN btrim(out, ' \\n\\t');\n END;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.normalize_indexdef(def text)\n RETURNS text\n LANGUAGE plpgsql\n IMMUTABLE PARALLEL SAFE STRICT\nAS $function$\nDECLARE\nBEGIN\n def := btrim(def, ' \\n\\t');\n\tdef := regexp_replace(def, '^CREATE (UNIQUE )?INDEX ([^ ]* )?ON (ONLY )?([^ ]* )?', '', 'i');\n RETURN def;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.unnest_collection(collection_ids text[] DEFAULT NULL::text[])\n RETURNS SETOF text\n LANGUAGE plpgsql\n STABLE\nAS $function$\n DECLARE\n BEGIN\n IF collection_ids IS NULL THEN\n RETURN QUERY SELECT id FROM collections;\n END IF;\n RETURN QUERY SELECT unnest(collection_ids);\n END;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.create_table_constraints(t text, _dtrange tstzrange, _edtrange tstzrange)\n RETURNS text\n LANGUAGE plpgsql\n SECURITY DEFINER\nAS $function$\nDECLARE\n q text;\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM partitions WHERE partition=t) THEN\n RETURN NULL;\n END IF;\n RAISE NOTICE 'Creating Table Constraints for % % %', t, _dtrange, _edtrange;\n IF _dtrange = 'empty' AND _edtrange = 'empty' THEN\n q :=format(\n $q$\n DO $block$\n BEGIN\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n ALTER TABLE %I\n ADD CONSTRAINT %I\n CHECK (((datetime IS NULL) AND (end_datetime IS NULL))) NOT VALID\n ;\n ALTER TABLE %I\n VALIDATE CONSTRAINT %I\n ;\n\n EXCEPTION WHEN others THEN\n RAISE WARNING '%%, Issue Altering Constraints. Please run update_partition_stats(%I)', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n $block$;\n $q$,\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t),\n t\n );\n ELSE\n q :=format(\n $q$\n DO $block$\n BEGIN\n\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n ALTER TABLE %I\n ADD CONSTRAINT %I\n CHECK (\n (datetime >= %L)\n AND (datetime <= %L)\n AND (end_datetime >= %L)\n AND (end_datetime <= %L)\n ) NOT VALID\n ;\n ALTER TABLE %I\n VALIDATE CONSTRAINT %I\n ;\n\n EXCEPTION WHEN others THEN\n RAISE WARNING '%%, Issue Altering Constraints. Please run update_partition_stats(%I)', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n $block$;\n $q$,\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t),\n lower(_dtrange),\n upper(_dtrange),\n lower(_edtrange),\n upper(_edtrange),\n t,\n format('%s_dt', t),\n t\n );\n END IF;\n PERFORM run_or_queue(q);\n RETURN t;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.maintain_partition_queries(part text DEFAULT 'items'::text, dropindexes boolean DEFAULT false, rebuildindexes boolean DEFAULT false, idxconcurrently boolean DEFAULT false)\n RETURNS SETOF text\n LANGUAGE plpgsql\nAS $function$\nDECLARE\n rec record;\nBEGIN\n FOR rec IN (\n WITH p AS (\n SELECT\n relid::text as partition,\n replace(replace(\n CASE\n WHEN level = 1 THEN pg_get_expr(c.relpartbound, c.oid)\n ELSE pg_get_expr(parent.relpartbound, parent.oid)\n END,\n 'FOR VALUES IN (''',''), ''')',\n ''\n ) AS collection\n FROM pg_partition_tree('items')\n JOIN pg_class c ON (relid::regclass = c.oid)\n JOIN pg_class parent ON (parentrelid::regclass = parent.oid AND isleaf)\n ), i AS (\n SELECT\n partition,\n indexname,\n regexp_replace(btrim(replace(replace(indexdef, indexname, ''),'pgstac.',''),' \\t\\n'), '[ ]+', ' ', 'g') as iidx,\n COALESCE(\n (regexp_match(indexdef, '\\(([a-zA-Z]+)\\)'))[1],\n (regexp_match(indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_-]+)''::text'))[1],\n CASE WHEN indexdef ~* '\\(datetime desc, end_datetime\\)' THEN 'datetime' ELSE NULL END\n ) AS field\n FROM\n pg_indexes\n JOIN p ON (tablename=partition)\n ), q AS (\n SELECT\n name AS field,\n collection,\n partition,\n format(indexdef(queryables), partition) as qidx\n FROM queryables, unnest_collection(queryables.collection_ids) collection\n JOIN p USING (collection)\n WHERE property_index_type IS NOT NULL OR name IN ('datetime','geometry','id')\n )\n SELECT * FROM i FULL JOIN q USING (field, partition)\n WHERE lower(iidx) IS DISTINCT FROM lower(qidx)\n ) LOOP\n IF rec.iidx IS NULL THEN\n IF idxconcurrently THEN\n RETURN NEXT replace(rec.qidx, 'INDEX', 'INDEX CONCURRENTLY');\n ELSE\n RETURN NEXT rec.qidx;\n END IF;\n ELSIF rec.qidx IS NULL AND dropindexes THEN\n RETURN NEXT format('DROP INDEX IF EXISTS %I;', rec.indexname);\n ELSIF lower(rec.qidx) != lower(rec.iidx) THEN\n IF dropindexes THEN\n RETURN NEXT format('DROP INDEX IF EXISTS %I; %s;', rec.indexname, rec.qidx);\n ELSE\n IF idxconcurrently THEN\n RETURN NEXT replace(rec.qidx, 'INDEX', 'INDEX CONCURRENTLY');\n ELSE\n RETURN NEXT rec.qidx;\n END IF;\n END IF;\n ELSIF rebuildindexes and rec.indexname IS NOT NULL THEN\n IF idxconcurrently THEN\n RETURN NEXT format('REINDEX INDEX CONCURRENTLY %I;', rec.indexname);\n ELSE\n RETURN NEXT format('REINDEX INDEX %I;', rec.indexname);\n END IF;\n END IF;\n END LOOP;\n RETURN;\nEND;\n$function$\n;\n\ncreate or replace view \"pgstac\".\"pgstac_indexes\" as SELECT i.schemaname,\n i.tablename,\n i.indexname,\n regexp_replace(btrim(replace(replace(i.indexdef, (i.indexname)::text, ''::text), 'pgstac.'::text, ''::text), ' \\t\\n'::text), '[ ]+'::text, ' '::text, 'g'::text) AS idx,\n COALESCE((regexp_match(i.indexdef, '\\(([a-zA-Z]+)\\)'::text))[1], (regexp_match(i.indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_-]+)''::text'::text))[1],\n CASE\n WHEN (i.indexdef ~* '\\(datetime desc, end_datetime\\)'::text) THEN 'datetime'::text\n ELSE NULL::text\n END) AS field,\n pg_table_size(((i.indexname)::text)::regclass) AS index_size,\n pg_size_pretty(pg_table_size(((i.indexname)::text)::regclass)) AS index_size_pretty\n FROM pg_indexes i\n WHERE ((i.schemaname = 'pgstac'::name) AND (i.tablename ~ '_items_'::text) AND (i.indexdef !~* ' only '::text));\n\n\n\n\n-- END migra calculated SQL\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('id', '{\"title\": \"Item ID\",\"description\": \"Item identifier\",\"$ref\": \"https://schemas.stacspec.org/v1.0.0/item-spec/json-schema/item.json#/definitions/core/allOf/2/properties/id\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('geometry', '{\"title\": \"Item Geometry\",\"description\": \"Item Geometry\",\"$ref\": \"https://geojson.org/schema/Feature.json\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('datetime','{\"description\": \"Datetime\",\"type\": \"string\",\"title\": \"Acquired\",\"format\": \"date-time\",\"pattern\": \"(\\\\+00:00|Z)$\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('eo:cloud_cover','{\"$ref\": \"https://stac-extensions.github.io/eo/v1.0.0/schema.json#/definitions/fieldsproperties/eo:cloud_cover\"}','to_int','BTREE');\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDELETE FROM queryables a USING queryables b\n WHERE a.name = b.name AND a.collection_ids IS NOT DISTINCT FROM b.collection_ids AND a.id > b.id;\n\n\nINSERT INTO pgstac_settings (name, value) VALUES\n ('context', 'off'),\n ('context_estimated_count', '100000'),\n ('context_estimated_cost', '100000'),\n ('context_stats_ttl', '1 day'),\n ('default_filter_lang', 'cql2-json'),\n ('additional_properties', 'true'),\n ('use_queue', 'false'),\n ('queue_timeout', '10 minutes'),\n ('update_collection_extent', 'false')\nON CONFLICT DO NOTHING\n;\n\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nGRANT ALL ON SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac to pgstac_admin;\n\n-- pgstac_read role limited to using function apis\nGRANT EXECUTE ON FUNCTION search TO pgstac_read;\nGRANT EXECUTE ON FUNCTION search_query TO pgstac_read;\nGRANT EXECUTE ON FUNCTION item_by_id TO pgstac_read;\nGRANT EXECUTE ON FUNCTION get_item TO pgstac_read;\nGRANT SELECT ON ALL TABLES IN SCHEMA pgstac TO pgstac_read;\n\n\nGRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_ingest;\nGRANT USAGE ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_ingest;\n\nSELECT update_partition_stats_q(partition) FROM partitions;\nSELECT set_version('0.7.3');\n" }, { "path": "src/pgstac/migrations/pgstac.0.7.2.sql", "content": "RESET ROLE;\nDO $$\nDECLARE\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='postgis') THEN\n CREATE EXTENSION IF NOT EXISTS postgis;\n END IF;\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='btree_gist') THEN\n CREATE EXTENSION IF NOT EXISTS btree_gist;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_admin;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_read;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_ingest;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n\nGRANT pgstac_admin TO current_user;\n\n-- Function to make sure pgstac_admin is the owner of items\nCREATE OR REPLACE FUNCTION pgstac_admin_owns() RETURNS VOID AS $$\nDECLARE\n f RECORD;\nBEGIN\n FOR f IN (\n SELECT\n concat(\n oid::regproc::text,\n '(',\n coalesce(pg_get_function_identity_arguments(oid),''),\n ')'\n ) AS name,\n CASE prokind WHEN 'f' THEN 'FUNCTION' WHEN 'p' THEN 'PROCEDURE' WHEN 'a' THEN 'AGGREGATE' END as typ\n FROM pg_proc\n WHERE\n pronamespace=to_regnamespace('pgstac')\n AND proowner != to_regrole('pgstac_admin')\n AND proname NOT LIKE 'pg_stat%'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n FOR f IN (\n SELECT\n oid::regclass::text as name,\n CASE relkind\n WHEN 'i' THEN 'INDEX'\n WHEN 'I' THEN 'INDEX'\n WHEN 'p' THEN 'TABLE'\n WHEN 'r' THEN 'TABLE'\n WHEN 'v' THEN 'VIEW'\n WHEN 'S' THEN 'SEQUENCE'\n ELSE NULL\n END as typ\n FROM pg_class\n WHERE relnamespace=to_regnamespace('pgstac') and relowner != to_regrole('pgstac_admin') AND relkind IN ('r','p','v','S') AND relname NOT LIKE 'pg_stat'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\nSELECT pgstac_admin_owns();\n\nCREATE SCHEMA IF NOT EXISTS pgstac AUTHORIZATION pgstac_admin;\n\nGRANT ALL ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_admin;\n\nALTER ROLE pgstac_admin SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_read SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_ingest SET SEARCH_PATH TO pgstac, public;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\n\nGRANT pgstac_read TO pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\n\nSET ROLE pgstac_admin;\n\nSET SEARCH_PATH TO pgstac, public;\n\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS analyze_items;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS validate_constraints;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\nCREATE OR REPLACE FUNCTION to_int(jsonb) RETURNS int AS $$\n SELECT floor(($1->>0)::float)::int;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION to_float(jsonb) RETURNS float AS $$\n SELECT ($1->>0)::float;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION to_tstz(jsonb) RETURNS timestamptz AS $$\n SELECT ($1->>0)::timestamptz;\n$$ LANGUAGE SQL IMMUTABLE STRICT SET TIME ZONE 'UTC';\n\n\nCREATE OR REPLACE FUNCTION to_text(jsonb) RETURNS text AS $$\n SELECT CASE WHEN jsonb_typeof($1) IN ('array','object') THEN $1::text ELSE $1->>0 END;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION to_text_array(jsonb) RETURNS text[] AS $$\n SELECT\n CASE jsonb_typeof($1)\n WHEN 'array' THEN ARRAY(SELECT jsonb_array_elements_text($1))\n ELSE ARRAY[$1->>0]\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION bbox_geom(_bbox jsonb) RETURNS geometry AS $$\nSELECT CASE jsonb_array_length(_bbox)\n WHEN 4 THEN\n ST_SetSRID(ST_MakeEnvelope(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float,\n (_bbox->>3)::float\n ),4326)\n WHEN 6 THEN\n ST_SetSRID(ST_3DMakeBox(\n ST_MakePoint(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float\n ),\n ST_MakePoint(\n (_bbox->>3)::float,\n (_bbox->>4)::float,\n (_bbox->>5)::float\n )\n ),4326)\n ELSE null END;\n;\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION geom_bbox(_geom geometry) RETURNS jsonb AS $$\n SELECT jsonb_build_array(\n st_xmin(_geom),\n st_ymin(_geom),\n st_xmax(_geom),\n st_ymax(_geom)\n );\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION flip_jsonb_array(j jsonb) RETURNS jsonb AS $$\n SELECT jsonb_agg(value) FROM (SELECT value FROM jsonb_array_elements(j) WITH ORDINALITY ORDER BY ordinality DESC) as t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION explode_dotpaths(j jsonb) RETURNS SETOF text[] AS $$\n SELECT string_to_array(p, '.') as e FROM jsonb_array_elements_text(j) p;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION explode_dotpaths_recurse(IN j jsonb) RETURNS SETOF text[] AS $$\n WITH RECURSIVE t AS (\n SELECT e FROM explode_dotpaths(j) e\n UNION ALL\n SELECT e[1:cardinality(e)-1]\n FROM t\n WHERE cardinality(e)>1\n ) SELECT e FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION jsonb_set_nested(j jsonb, path text[], val jsonb) RETURNS jsonb AS $$\nDECLARE\nBEGIN\n IF cardinality(path) > 1 THEN\n FOR i IN 1..(cardinality(path)-1) LOOP\n IF j #> path[:i] IS NULL THEN\n j := jsonb_set_lax(j, path[:i], '{}', TRUE);\n END IF;\n END LOOP;\n END IF;\n RETURN jsonb_set_lax(j, path, val, true);\n\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\n\n\nCREATE OR REPLACE FUNCTION jsonb_include(j jsonb, f jsonb) RETURNS jsonb AS $$\nDECLARE\n includes jsonb := f-> 'include';\n outj jsonb := '{}'::jsonb;\n path text[];\nBEGIN\n IF\n includes IS NULL\n OR jsonb_array_length(includes) = 0\n THEN\n RETURN j;\n ELSE\n includes := includes || '[\"id\",\"collection\"]'::jsonb;\n FOR path IN SELECT explode_dotpaths(includes) LOOP\n outj := jsonb_set_nested(outj, path, j #> path);\n END LOOP;\n END IF;\n RETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_exclude(j jsonb, f jsonb) RETURNS jsonb AS $$\nDECLARE\n excludes jsonb := f-> 'exclude';\n outj jsonb := j;\n path text[];\nBEGIN\n IF\n excludes IS NULL\n OR jsonb_array_length(excludes) = 0\n THEN\n RETURN j;\n ELSE\n FOR path IN SELECT explode_dotpaths(excludes) LOOP\n outj := outj #- path;\n END LOOP;\n END IF;\n RETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_fields(j jsonb, f jsonb DEFAULT '{\"fields\":[]}') RETURNS jsonb AS $$\n SELECT jsonb_exclude(jsonb_include(j, f), f);\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION merge_jsonb(_a jsonb, _b jsonb) RETURNS jsonb AS $$\n SELECT\n CASE\n WHEN _a = '\"𒍟※\"'::jsonb THEN NULL\n WHEN _a IS NULL OR jsonb_typeof(_a) = 'null' THEN _b\n WHEN jsonb_typeof(_a) = 'object' AND jsonb_typeof(_b) = 'object' THEN\n (\n SELECT\n jsonb_strip_nulls(\n jsonb_object_agg(\n key,\n merge_jsonb(a.value, b.value)\n )\n )\n FROM\n jsonb_each(coalesce(_a,'{}'::jsonb)) as a\n FULL JOIN\n jsonb_each(coalesce(_b,'{}'::jsonb)) as b\n USING (key)\n )\n WHEN\n jsonb_typeof(_a) = 'array'\n AND jsonb_typeof(_b) = 'array'\n AND jsonb_array_length(_a) = jsonb_array_length(_b)\n THEN\n (\n SELECT jsonb_agg(m) FROM\n ( SELECT\n merge_jsonb(\n jsonb_array_elements(_a),\n jsonb_array_elements(_b)\n ) as m\n ) as l\n )\n ELSE _a\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION strip_jsonb(_a jsonb, _b jsonb) RETURNS jsonb AS $$\n SELECT\n CASE\n\n WHEN (_a IS NULL OR jsonb_typeof(_a) = 'null') AND _b IS NOT NULL AND jsonb_typeof(_b) != 'null' THEN '\"𒍟※\"'::jsonb\n WHEN _b IS NULL OR jsonb_typeof(_a) = 'null' THEN _a\n WHEN _a = _b AND jsonb_typeof(_a) = 'object' THEN '{}'::jsonb\n WHEN _a = _b THEN NULL\n WHEN jsonb_typeof(_a) = 'object' AND jsonb_typeof(_b) = 'object' THEN\n (\n SELECT\n jsonb_strip_nulls(\n jsonb_object_agg(\n key,\n strip_jsonb(a.value, b.value)\n )\n )\n FROM\n jsonb_each(_a) as a\n FULL JOIN\n jsonb_each(_b) as b\n USING (key)\n )\n WHEN\n jsonb_typeof(_a) = 'array'\n AND jsonb_typeof(_b) = 'array'\n AND jsonb_array_length(_a) = jsonb_array_length(_b)\n THEN\n (\n SELECT jsonb_agg(m) FROM\n ( SELECT\n strip_jsonb(\n jsonb_array_elements(_a),\n jsonb_array_elements(_b)\n ) as m\n ) as l\n )\n ELSE _a\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION nullif_jsonbnullempty(j jsonb) RETURNS jsonb AS $$\n SELECT nullif(nullif(nullif(j,'null'::jsonb),'{}'::jsonb),'[]'::jsonb);\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION jsonb_array_unique(j jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(jsonb_agg(DISTINCT a)) v FROM jsonb_array_elements(j) a;\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_concat_ignorenull(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT coalesce(a,'[]'::jsonb) || coalesce(b,'[]'::jsonb);\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_least(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(least(nullif_jsonbnullempty(a), nullif_jsonbnullempty(b)));\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_greatest(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(greatest(a, b));\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION first_notnull_sfunc(anyelement, anyelement) RETURNS anyelement AS $$\n SELECT COALESCE($1,$2);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE AGGREGATE first_notnull(anyelement)(\n SFUNC = first_notnull_sfunc,\n STYPE = anyelement\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_array_unique_merge(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_concat_ignorenull,\n FINALFUNC = jsonb_array_unique\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_min(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_least\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_max(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_greatest\n);\n\nCREATE TABLE IF NOT EXISTS migrations (\n version text PRIMARY KEY,\n datetime timestamptz DEFAULT clock_timestamp() NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION get_version() RETURNS text AS $$\n SELECT version FROM pgstac.migrations ORDER BY datetime DESC, version DESC LIMIT 1;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION set_version(text) RETURNS text AS $$\n INSERT INTO pgstac.migrations (version) VALUES ($1)\n ON CONFLICT DO NOTHING\n RETURNING version;\n$$ LANGUAGE SQL;\n\n\nCREATE TABLE IF NOT EXISTS pgstac_settings (\n name text PRIMARY KEY,\n value text NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION table_empty(text) RETURNS boolean AS $$\nDECLARE\n retval boolean;\nBEGIN\n EXECUTE format($q$\n SELECT NOT EXISTS (SELECT 1 FROM %I LIMIT 1)\n $q$,\n $1\n ) INTO retval;\n RETURN retval;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION get_setting(IN _setting text, IN conf jsonb DEFAULT NULL) RETURNS text AS $$\nSELECT COALESCE(\n nullif(conf->>_setting, ''),\n nullif(current_setting(concat('pgstac.',_setting), TRUE),''),\n nullif((SELECT value FROM pgstac.pgstac_settings WHERE name=_setting),'')\n);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_setting_bool(IN _setting text, IN conf jsonb DEFAULT NULL) RETURNS boolean AS $$\nSELECT COALESCE(\n conf->>_setting,\n current_setting(concat('pgstac.',_setting), TRUE),\n (SELECT value FROM pgstac.pgstac_settings WHERE name=_setting),\n 'FALSE'\n)::boolean;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context(conf jsonb DEFAULT NULL) RETURNS text AS $$\n SELECT pgstac.get_setting('context', conf);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context_estimated_count(conf jsonb DEFAULT NULL) RETURNS int AS $$\n SELECT pgstac.get_setting('context_estimated_count', conf)::int;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_estimated_cost();\nCREATE OR REPLACE FUNCTION context_estimated_cost(conf jsonb DEFAULT NULL) RETURNS float AS $$\n SELECT pgstac.get_setting('context_estimated_cost', conf)::float;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_stats_ttl();\nCREATE OR REPLACE FUNCTION context_stats_ttl(conf jsonb DEFAULT NULL) RETURNS interval AS $$\n SELECT pgstac.get_setting('context_stats_ttl', conf)::interval;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION t2s(text) RETURNS text AS $$\n SELECT extract(epoch FROM $1::interval)::text || ' s';\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE STRICT;\n\nCREATE OR REPLACE FUNCTION queue_timeout() RETURNS interval AS $$\n SELECT t2s(coalesce(\n get_setting('queue_timeout'),\n '1h'\n ))::interval;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION notice(VARIADIC text[]) RETURNS boolean AS $$\nDECLARE\ndebug boolean := current_setting('pgstac.debug', true);\nBEGIN\n IF debug THEN\n RAISE NOTICE 'NOTICE FROM FUNC: % >>>>> %', concat_ws(' | ', $1), clock_timestamp();\n RETURN TRUE;\n END IF;\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION empty_arr(ANYARRAY) RETURNS BOOLEAN AS $$\nSELECT CASE\n WHEN $1 IS NULL THEN TRUE\n WHEN cardinality($1)<1 THEN TRUE\nELSE FALSE\nEND;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_intersection(_a ANYARRAY, _b ANYARRAY) RETURNS ANYARRAY AS $$\n SELECT ARRAY ( SELECT unnest(_a) INTERSECT SELECT UNNEST(_b) );\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION array_map_ident(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_ident(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_map_literal(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_literal(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION array_reverse(anyarray) RETURNS anyarray AS $$\nSELECT ARRAY(\n SELECT $1[i]\n FROM generate_subscripts($1,1) AS s(i)\n ORDER BY i DESC\n);\n$$ LANGUAGE SQL STRICT IMMUTABLE;\n\nDROP TABLE IF EXISTS query_queue;\nCREATE TABLE query_queue (\n query text PRIMARY KEY,\n added timestamptz DEFAULT now()\n);\n\nDROP TABLE IF EXISTS query_queue_history;\nCREATE TABLE query_queue_history(\n query text,\n added timestamptz NOT NULL,\n finished timestamptz NOT NULL DEFAULT now(),\n error text\n);\n\nCREATE OR REPLACE PROCEDURE run_queued_queries() AS $$\nDECLARE\n qitem query_queue%ROWTYPE;\n timeout_ts timestamptz;\n error text;\n cnt int := 0;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n DELETE FROM query_queue WHERE query = (SELECT query FROM query_queue ORDER BY added DESC LIMIT 1 FOR UPDATE SKIP LOCKED) RETURNING * INTO qitem;\n IF NOT FOUND THEN\n EXIT;\n END IF;\n cnt := cnt + 1;\n BEGIN\n RAISE NOTICE 'RUNNING QUERY: %', qitem.query;\n EXECUTE qitem.query;\n EXCEPTION WHEN others THEN\n error := format('%s | %s', SQLERRM, SQLSTATE);\n END;\n INSERT INTO query_queue_history (query, added, finished, error)\n VALUES (qitem.query, qitem.added, clock_timestamp(), error);\n COMMIT;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION run_queued_queries_intransaction() RETURNS int AS $$\nDECLARE\n qitem query_queue%ROWTYPE;\n timeout_ts timestamptz;\n error text;\n cnt int := 0;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n DELETE FROM query_queue WHERE query = (SELECT query FROM query_queue ORDER BY added DESC LIMIT 1 FOR UPDATE SKIP LOCKED) RETURNING * INTO qitem;\n IF NOT FOUND THEN\n RETURN cnt;\n END IF;\n cnt := cnt + 1;\n BEGIN\n qitem.query := regexp_replace(qitem.query, 'CONCURRENTLY', '');\n RAISE NOTICE 'RUNNING QUERY: %', qitem.query;\n\n EXECUTE qitem.query;\n EXCEPTION WHEN others THEN\n error := format('%s | %s', SQLERRM, SQLSTATE);\n RAISE WARNING '%', error;\n END;\n INSERT INTO query_queue_history (query, added, finished, error)\n VALUES (qitem.query, qitem.added, clock_timestamp(), error);\n END LOOP;\n RETURN cnt;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION run_or_queue(query text) RETURNS VOID AS $$\nDECLARE\n use_queue text := COALESCE(get_setting('use_queue'), 'FALSE')::boolean;\nBEGIN\n IF get_setting_bool('debug') THEN\n RAISE NOTICE '%', query;\n END IF;\n IF use_queue THEN\n INSERT INTO query_queue (query) VALUES (query) ON CONFLICT DO NOTHING;\n ELSE\n EXECUTE query;\n END IF;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n\nDROP FUNCTION IF EXISTS check_pgstac_settings;\nCREATE OR REPLACE FUNCTION check_pgstac_settings(_sysmem text DEFAULT NULL) RETURNS VOID AS $$\nDECLARE\n settingval text;\n sysmem bigint := pg_size_bytes(_sysmem);\n effective_cache_size bigint := pg_size_bytes(current_setting('effective_cache_size', TRUE));\n shared_buffers bigint := pg_size_bytes(current_setting('shared_buffers', TRUE));\n work_mem bigint := pg_size_bytes(current_setting('work_mem', TRUE));\n max_connections int := current_setting('max_connections', TRUE);\n maintenance_work_mem bigint := pg_size_bytes(current_setting('maintenance_work_mem', TRUE));\n seq_page_cost float := current_setting('seq_page_cost', TRUE);\n random_page_cost float := current_setting('random_page_cost', TRUE);\n temp_buffers bigint := pg_size_bytes(current_setting('temp_buffers', TRUE));\n r record;\nBEGIN\n IF _sysmem IS NULL THEN\n RAISE NOTICE 'Call function with the size of your system memory `SELECT check_pgstac_settings(''4GB'')` to get pg system setting recommendations.';\n ELSE\n IF effective_cache_size < (sysmem * 0.5) THEN\n RAISE WARNING 'effective_cache_size of % is set low for a system with %. Recomended value between % and %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.5), pg_size_pretty(sysmem * 0.75);\n ELSIF effective_cache_size > (sysmem * 0.75) THEN\n RAISE WARNING 'effective_cache_size of % is set high for a system with %. Recomended value between % and %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.5), pg_size_pretty(sysmem * 0.75);\n ELSE\n RAISE NOTICE 'effective_cache_size of % is set appropriately for a system with %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem);\n END IF;\n\n IF shared_buffers < (sysmem * 0.2) THEN\n RAISE WARNING 'shared_buffers of % is set low for a system with %. Recomended value between % and %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.2), pg_size_pretty(sysmem * 0.3);\n ELSIF shared_buffers > (sysmem * 0.3) THEN\n RAISE WARNING 'shared_buffers of % is set high for a system with %. Recomended value between % and %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.2), pg_size_pretty(sysmem * 0.3);\n ELSE\n RAISE NOTICE 'shared_buffers of % is set appropriately for a system with %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem);\n END IF;\n shared_buffers = sysmem * 0.3;\n IF maintenance_work_mem < (sysmem * 0.2) THEN\n RAISE WARNING 'maintenance_work_mem of % is set low for shared_buffers of %. Recomended value between % and %', pg_size_pretty(maintenance_work_mem), pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers * 0.2), pg_size_pretty(shared_buffers * 0.3);\n ELSIF maintenance_work_mem > (shared_buffers * 0.3) THEN\n RAISE WARNING 'maintenance_work_mem of % is set high for shared_buffers of %. Recomended value between % and %', pg_size_pretty(maintenance_work_mem), pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers * 0.2), pg_size_pretty(shared_buffers * 0.3);\n ELSE\n RAISE NOTICE 'maintenance_work_mem of % is set appropriately for shared_buffers of %', pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers);\n END IF;\n\n IF work_mem * max_connections > shared_buffers THEN\n RAISE WARNING 'work_mem setting of % is set high for % max_connections please reduce work_mem to % or decrease max_connections to %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers/max_connections), floor(shared_buffers/work_mem);\n ELSIF work_mem * max_connections < (shared_buffers * 0.75) THEN\n RAISE WARNING 'work_mem setting of % is set low for % max_connections you may consider raising work_mem to % or increasing max_connections to %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers/max_connections), floor(shared_buffers/work_mem);\n ELSE\n RAISE NOTICE 'work_mem setting of % and max_connections of % are adequate for shared_buffers of %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers);\n END IF;\n\n IF random_page_cost / seq_page_cost != 1.1 THEN\n RAISE WARNING 'random_page_cost (%) /seq_page_cost (%) should be set to 1.1 for SSD. Change random_page_cost to %', random_page_cost, seq_page_cost, 1.1 * seq_page_cost;\n ELSE\n RAISE NOTICE 'random_page_cost and seq_page_cost set appropriately for SSD';\n END IF;\n\n IF temp_buffers < greatest(pg_size_bytes('128MB'),(maintenance_work_mem / 2)) THEN\n RAISE WARNING 'pgstac makes heavy use of temp tables, consider raising temp_buffers from % to %', pg_size_pretty(temp_buffers), greatest('128MB', pg_size_pretty((shared_buffers / 16)));\n END IF;\n END IF;\n\n RAISE NOTICE 'VALUES FOR PGSTAC VARIABLES';\n RAISE NOTICE 'These can be set either as GUC system variables or by setting in the pgstac_settings table.';\n\n FOR r IN SELECT name, get_setting(name) as setting, CASE WHEN current_setting(concat('pgstac.',name), TRUE) IS NOT NULL THEN concat('pgstac.',name, ' GUC') WHEN value IS NOT NULL THEN 'pgstac_settings table' ELSE 'Not Set' END as loc FROM pgstac_settings LOOP\n RAISE NOTICE '% is set to % from the %', r.name, r.setting, r.loc;\n END LOOP;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pg_cron';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider intalling pg_cron which can be used to automate tasks';\n ELSE\n RAISE NOTICE 'pg_cron % is installed', settingval;\n END IF;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pgstattuple';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider installing the pgstattuple extension which can be used to help maintain tables and indexes.';\n ELSE\n RAISE NOTICE 'pgstattuple % is installed', settingval;\n END IF;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pg_stat_statements';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider installing the pg_stat_statements extension which is very helpful for tracking the types of queries on the system';\n ELSE\n RAISE NOTICE 'pg_stat_statements % is installed', settingval;\n IF current_setting('pg_stat_statements.track_statements', TRUE) IS DISTINCT FROM 'all' THEN\n RAISE WARNING 'SET pg_stat_statements.track_statements TO ''all''; --In order to track statements within functions.';\n END IF;\n END IF;\n\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac, public SET CLIENT_MIN_MESSAGES TO NOTICE;\n/* looks for a geometry in a stac item first from geometry and falling back to bbox */\nCREATE OR REPLACE FUNCTION stac_geom(value jsonb) RETURNS geometry AS $$\nSELECT\n CASE\n WHEN value ? 'intersects' THEN\n ST_GeomFromGeoJSON(value->>'intersects')\n WHEN value ? 'geometry' THEN\n ST_GeomFromGeoJSON(value->>'geometry')\n WHEN value ? 'bbox' THEN\n pgstac.bbox_geom(value->'bbox')\n ELSE NULL\n END as geometry\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION stac_daterange(\n value jsonb\n) RETURNS tstzrange AS $$\nDECLARE\n props jsonb := value;\n dt timestamptz;\n edt timestamptz;\nBEGIN\n IF props ? 'properties' THEN\n props := props->'properties';\n END IF;\n IF\n props ? 'start_datetime'\n AND props->>'start_datetime' IS NOT NULL\n AND props ? 'end_datetime'\n AND props->>'end_datetime' IS NOT NULL\n THEN\n dt := props->>'start_datetime';\n edt := props->>'end_datetime';\n IF dt > edt THEN\n RAISE EXCEPTION 'start_datetime must be < end_datetime';\n END IF;\n ELSE\n dt := props->>'datetime';\n edt := props->>'datetime';\n END IF;\n IF dt is NULL OR edt IS NULL THEN\n RAISE NOTICE 'DT: %, EDT: %', dt, edt;\n RAISE EXCEPTION 'Either datetime (%) or both start_datetime (%) and end_datetime (%) must be set.', props->>'datetime',props->>'start_datetime',props->>'end_datetime';\n END IF;\n RETURN tstzrange(dt, edt, '[]');\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_datetime(value jsonb) RETURNS timestamptz AS $$\n SELECT lower(stac_daterange(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_end_datetime(value jsonb) RETURNS timestamptz AS $$\n SELECT upper(stac_daterange(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE TABLE IF NOT EXISTS stac_extensions(\n url text PRIMARY KEY,\n content jsonb\n);\nCREATE OR REPLACE FUNCTION queryable_signature(n text, c text[]) RETURNS text AS $$\n SELECT concat(n, c);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TABLE queryables (\n id bigint GENERATED ALWAYS AS identity PRIMARY KEY,\n name text NOT NULL,\n collection_ids text[], -- used to determine what partitions to create indexes on\n definition jsonb,\n property_path text,\n property_wrapper text,\n property_index_type text\n);\nCREATE INDEX queryables_name_idx ON queryables (name);\nCREATE INDEX queryables_collection_idx ON queryables USING GIN (collection_ids);\nCREATE INDEX queryables_property_wrapper_idx ON queryables (property_wrapper);\n\nCREATE OR REPLACE FUNCTION queryables_constraint_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n allcollections text[];\nBEGIN\n RAISE NOTICE 'Making sure that name/collection is unique for queryables';\n IF NEW.collection_ids IS NOT NULL THEN\n IF EXISTS (\n SELECT 1 FROM\n collections\n LEFT JOIN\n unnest(NEW.collection_ids) c\n ON (collections.id = c)\n WHERE c IS NULL\n ) THEN\n RAISE foreign_key_violation;\n RETURN NULL;\n END IF;\n END IF;\n IF TG_OP = 'INSERT' THEN\n IF EXISTS (\n SELECT 1 FROM queryables q\n WHERE\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n )\n ) THEN\n RAISE unique_violation;\n RETURN NULL;\n END IF;\n END IF;\n IF TG_OP = 'UPDATE' THEN\n IF EXISTS (\n SELECT 1 FROM queryables q\n WHERE\n q.id != NEW.id\n AND\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n )\n ) THEN\n RAISE unique_violation\n USING MESSAGE = format(\n 'There is already a queryable for %s for a collection in %s',\n NEW.name,\n NEW.collection_ids\n );\n RETURN NULL;\n END IF;\n END IF;\n\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER queryables_constraint_insert_trigger\nBEFORE INSERT ON queryables\nFOR EACH ROW EXECUTE PROCEDURE queryables_constraint_triggerfunc();\n\nCREATE TRIGGER queryables_constraint_update_trigger\nBEFORE UPDATE ON queryables\nFOR EACH ROW\nWHEN (NEW.name = OLD.name AND NEW.collection_ids IS DISTINCT FROM OLD.collection_ids)\nEXECUTE PROCEDURE queryables_constraint_triggerfunc();\n\n\nCREATE OR REPLACE FUNCTION array_to_path(arr text[]) RETURNS text AS $$\n SELECT string_agg(\n quote_literal(v),\n '->'\n ) FROM unnest(arr) v;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\n\n\nCREATE OR REPLACE FUNCTION queryable(\n IN dotpath text,\n OUT path text,\n OUT expression text,\n OUT wrapper text,\n OUT nulled_wrapper text\n) AS $$\nDECLARE\n q RECORD;\n path_elements text[];\nBEGIN\n IF dotpath IN ('id', 'geometry', 'datetime', 'end_datetime', 'collection') THEN\n path := dotpath;\n expression := dotpath;\n wrapper := NULL;\n RETURN;\n END IF;\n SELECT * INTO q FROM queryables\n WHERE\n name=dotpath\n OR name = 'properties.' || dotpath\n OR name = replace(dotpath, 'properties.', '')\n ;\n IF q.property_wrapper IS NULL THEN\n IF q.definition->>'type' = 'number' THEN\n wrapper := 'to_float';\n nulled_wrapper := wrapper;\n ELSIF q.definition->>'format' = 'date-time' THEN\n wrapper := 'to_tstz';\n nulled_wrapper := wrapper;\n ELSE\n nulled_wrapper := NULL;\n wrapper := 'to_text';\n END IF;\n ELSE\n wrapper := q.property_wrapper;\n nulled_wrapper := wrapper;\n END IF;\n IF q.property_path IS NOT NULL THEN\n path := q.property_path;\n ELSE\n path_elements := string_to_array(dotpath, '.');\n IF path_elements[1] IN ('links', 'assets', 'stac_version', 'stac_extensions') THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSIF path_elements[1] = 'properties' THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSE\n path := format($F$content->'properties'->%s$F$, array_to_path(path_elements));\n END IF;\n END IF;\n expression := format('%I(%s)', wrapper, path);\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\n\nDROP VIEW IF EXISTS pgstac_index;\nCREATE VIEW pgstac_indexes AS\nSELECT\n i.schemaname,\n i.tablename,\n i.indexname,\n indexdef,\n COALESCE(\n (regexp_match(indexdef, '\\(([a-zA-Z]+)\\)'))[1],\n (regexp_match(indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_]+)''::text'))[1],\n CASE WHEN indexdef ~* '\\(datetime desc, end_datetime\\)' THEN 'datetime_end_datetime' ELSE NULL END\n ) AS field,\n pg_table_size(i.indexname::text) as index_size,\n pg_size_pretty(pg_table_size(i.indexname::text)) as index_size_pretty\nFROM\n pg_indexes i\nWHERE i.schemaname='pgstac' and i.tablename ~ '_items_';\n\nDROP VIEW IF EXISTS pgstac_index_stats;\nCREATE VIEW pgstac_indexes_stats AS\nSELECT\n i.schemaname,\n i.tablename,\n i.indexname,\n indexdef,\n COALESCE(\n (regexp_match(indexdef, '\\(([a-zA-Z]+)\\)'))[1],\n (regexp_match(indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_]+)''::text'))[1],\n CASE WHEN indexdef ~* '\\(datetime desc, end_datetime\\)' THEN 'datetime_end_datetime' ELSE NULL END\n ) AS field,\n pg_table_size(i.indexname::text) as index_size,\n pg_size_pretty(pg_table_size(i.indexname::text)) as index_size_pretty,\n n_distinct,\n most_common_vals::text::text[],\n most_common_freqs::text::text[],\n histogram_bounds::text::text[],\n correlation\nFROM\n pg_indexes i\n LEFT JOIN pg_stats s ON (s.tablename = i.indexname)\nWHERE i.schemaname='pgstac' and i.tablename ~ '_items_';\n\nset check_function_bodies to off;\nCREATE OR REPLACE FUNCTION maintain_partition_queries(\n part text DEFAULT 'items',\n dropindexes boolean DEFAULT FALSE,\n rebuildindexes boolean DEFAULT FALSE,\n idxconcurrently boolean DEFAULT FALSE\n) RETURNS SETOF text AS $$\nDECLARE\n parent text;\n level int;\n isleaf bool;\n collection collections%ROWTYPE;\n subpart text;\n baseidx text;\n queryable_name text;\n queryable_property_index_type text;\n queryable_property_wrapper text;\n queryable_parsed RECORD;\n deletedidx pg_indexes%ROWTYPE;\n q text;\n idx text;\n collection_partition bigint;\n _concurrently text := '';\n idxname text;\nBEGIN\n RAISE NOTICE 'Maintaining partition: %', part;\n IF idxconcurrently THEN\n _concurrently='CONCURRENTLY';\n END IF;\n\n -- Get root partition\n SELECT parentrelid::text, pt.isleaf, pt.level\n INTO parent, isleaf, level\n FROM pg_partition_tree('items') pt\n WHERE relid::text = part;\n IF NOT FOUND THEN\n RAISE NOTICE 'Partition % Does Not Exist In Partition Tree', part;\n RETURN;\n END IF;\n\n -- If this is a parent partition, recurse to leaves\n IF NOT isleaf THEN\n FOR subpart IN\n SELECT relid::text\n FROM pg_partition_tree(part)\n WHERE relid::text != part\n LOOP\n RAISE NOTICE 'Recursing to %', subpart;\n RETURN QUERY SELECT * FROM maintain_partition_queries(subpart, dropindexes, rebuildindexes);\n END LOOP;\n RETURN; -- Don't continue since not an end leaf\n END IF;\n\n\n -- Get collection\n collection_partition := ((regexp_match(part, E'^_items_([0-9]+)'))[1])::bigint;\n RAISE NOTICE 'COLLECTION PARTITION: %', collection_partition;\n SELECT * INTO STRICT collection\n FROM collections\n WHERE key = collection_partition;\n RAISE NOTICE 'COLLECTION ID: %s', collection.id;\n\n\n -- Create temp table with existing indexes\n CREATE TEMP TABLE existing_indexes ON COMMIT DROP AS\n SELECT *\n FROM pg_indexes\n WHERE schemaname='pgstac' AND tablename=part;\n\n\n -- Check if index exists for each queryable.\n FOR\n queryable_name,\n queryable_property_index_type,\n queryable_property_wrapper\n IN\n SELECT\n name,\n COALESCE(property_index_type, 'BTREE'),\n COALESCE(property_wrapper, 'to_text')\n FROM queryables\n WHERE\n name NOT in ('id', 'datetime', 'geometry')\n AND (\n collection_ids IS NULL\n OR collection_ids = '{}'::text[]\n OR collection.id = ANY (collection_ids)\n )\n LOOP\n baseidx := format(\n $q$ ON %I USING %s (%s(((content -> 'properties'::text) -> %L::text)))$q$,\n part,\n queryable_property_index_type,\n queryable_property_wrapper,\n queryable_name\n );\n RAISE NOTICE 'BASEIDX: %', baseidx;\n RAISE NOTICE 'IDXSEARCH: %', format($q$[(']%s[')]$q$, queryable_name);\n -- If index already exists, delete it from existing indexes type table\n FOR deletedidx IN\n DELETE FROM existing_indexes\n WHERE indexdef ~* format($q$[(']%s[')]$q$, queryable_name)\n RETURNING *\n LOOP\n RAISE NOTICE 'EXISTING INDEX: %', deletedidx;\n IF NOT FOUND THEN -- index did not exist, create it\n RETURN NEXT format('CREATE INDEX %s %s;', _concurrently, baseidx);\n ELSIF rebuildindexes THEN\n RETURN NEXT format('REINDEX %I %s;', deletedidx.indexname, _concurrently);\n END IF;\n END LOOP;\n IF NOT FOUND THEN\n RAISE NOTICE 'CREATING INDEX for %', queryable_name;\n RETURN NEXT format('CREATE INDEX %s %s;', _concurrently, baseidx);\n END IF;\n END LOOP;\n IF NOT EXISTS (SELECT * FROM pg_indexes WHERE indexname::text = concat(part, '_datetime_end_datetime_idx')) THEN\n RETURN NEXT format(\n $f$CREATE INDEX IF NOT EXISTS %I ON %I USING BTREE (datetime DESC, end_datetime ASC);$f$,\n concat(part, '_datetime_end_datetime_idx'),\n part\n );\n END IF;\n\n IF NOT EXISTS (SELECT * FROM pg_indexes WHERE indexname::text = concat(part, '_geometry_idx')) THEN\n RETURN NEXT format(\n $f$CREATE INDEX IF NOT EXISTS %I ON %I USING GIST (geometry);$f$,\n concat(part, '_geometry_idx'),\n part\n );\n END IF;\n\n FOR idxname IN\n SELECT indexname::text FROM pg_indexes\n WHERE tablename::text = part AND indexdef ILIKE 'CREATE UNIQUE INDEX % USING btree(id)'\n LOOP\n IF idxname != concat(part, '_pk') AND NOT EXISTS (SELECT * FROM pg_indexes WHERE indexname::text = concat(part,'_pk')) THEN\n RETURN NEXT format(\n $f$ALTER INDEX IF EXISTS %I RENAME TO %I;$f$,\n idxname,\n concat(part, '_pk')\n );\n ELSIF EXISTS (SELECT * FROM pg_indexes WHERE indexname::text = concat(part,'_pk')) AND dropindexes THEN\n RETURN NEXT format(\n $f$DROP INDEX IF EXISTS %I;$f$,\n idxname\n );\n END IF;\n END LOOP;\n\n IF NOT EXISTS (\n SELECT indexname::text FROM pg_indexes\n WHERE tablename::text = part AND indexdef ILIKE 'CREATE UNIQUE INDEX % USING btree (id)'\n ) THEN\n RETURN NEXT format(\n $f$CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I USING BTREE(id);$f$,\n concat(part, '_pk'),\n part\n );\n END IF;\n\n DELETE FROM existing_indexes WHERE indexname::text IN (\n concat(part, '_datetime_end_datetime_idx'),\n concat(part, '_geometry_idx'),\n concat(part, '_pk')\n );\n -- Remove indexes that were not expected\n FOR idx IN SELECT indexname::text FROM existing_indexes\n LOOP\n RAISE WARNING 'Index: % is not defined by queryables.', idx;\n IF dropindexes THEN\n RETURN NEXT format('DROP INDEX IF EXISTS %I;', idx);\n END IF;\n END LOOP;\n\n DROP TABLE existing_indexes;\n RAISE NOTICE 'Returning from maintain_partition_queries.';\n RETURN;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION maintain_partitions(\n part text DEFAULT 'items',\n dropindexes boolean DEFAULT FALSE,\n rebuildindexes boolean DEFAULT FALSE\n) RETURNS VOID AS $$\n WITH t AS (\n SELECT run_or_queue(q) FROM maintain_partition_queries(part, dropindexes, rebuildindexes) q\n ) SELECT count(*) FROM t;\n$$ LANGUAGE SQL;\n\n\nCREATE OR REPLACE FUNCTION queryables_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\nBEGIN\n PERFORM maintain_partitions();\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER queryables_trigger AFTER INSERT OR UPDATE ON queryables\nFOR EACH STATEMENT EXECUTE PROCEDURE queryables_trigger_func();\n\n\nCREATE OR REPLACE FUNCTION get_queryables(_collection_ids text[] DEFAULT NULL) RETURNS jsonb AS $$\nDECLARE\nBEGIN\n -- Build up queryables if the input contains valid collection ids or is empty\n IF EXISTS (\n SELECT 1 FROM collections\n WHERE\n _collection_ids IS NULL\n OR cardinality(_collection_ids) = 0\n OR id = ANY(_collection_ids)\n )\n THEN\n RETURN (\n WITH base AS (\n SELECT\n unnest(collection_ids) as collection_id,\n name,\n coalesce(definition, '{\"type\":\"string\"}'::jsonb) as definition\n FROM queryables\n WHERE\n _collection_ids IS NULL OR\n _collection_ids = '{}'::text[] OR\n _collection_ids && collection_ids\n UNION ALL\n SELECT null, name, coalesce(definition, '{\"type\":\"string\"}'::jsonb) as definition\n FROM queryables WHERE collection_ids IS NULL OR collection_ids = '{}'::text[]\n ), g AS (\n SELECT\n name,\n first_notnull(definition) as definition,\n jsonb_array_unique_merge(definition->'enum') as enum,\n jsonb_min(definition->'minimum') as minimum,\n jsonb_min(definition->'maxiumn') as maximum\n FROM base\n GROUP BY 1\n )\n SELECT\n jsonb_build_object(\n '$schema', 'http://json-schema.org/draft-07/schema#',\n '$id', '',\n 'type', 'object',\n 'title', 'STAC Queryables.',\n 'properties', jsonb_object_agg(\n name,\n definition\n ||\n jsonb_strip_nulls(jsonb_build_object(\n 'enum', enum,\n 'minimum', minimum,\n 'maximum', maximum\n ))\n )\n )\n FROM g\n );\n ELSE\n RETURN NULL;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\nCREATE OR REPLACE FUNCTION get_queryables(_collection text DEFAULT NULL) RETURNS jsonb AS $$\n SELECT\n CASE\n WHEN _collection IS NULL THEN get_queryables(NULL::text[])\n ELSE get_queryables(ARRAY[_collection])\n END\n ;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_queryables() RETURNS jsonb AS $$\n SELECT get_queryables(NULL::text[]);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION schema_qualify_refs(url text, j jsonb) returns jsonb as $$\n SELECT regexp_replace(j::text, '\"\\$ref\": \"#', concat('\"$ref\": \"', url, '#'), 'g')::jsonb;\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE VIEW stac_extension_queryables AS\nSELECT DISTINCT key as name, schema_qualify_refs(e.url, j.value) as definition FROM stac_extensions e, jsonb_each(e.content->'definitions'->'fields'->'properties') j;\n\n\nCREATE OR REPLACE FUNCTION missing_queryables(_collection text, _tablesample float DEFAULT 5, minrows float DEFAULT 10) RETURNS TABLE(collection text, name text, definition jsonb, property_wrapper text) AS $$\nDECLARE\n q text;\n _partition text;\n explain_json json;\n psize float;\n estrows float;\nBEGIN\n SELECT format('_items_%s', key) INTO _partition FROM collections WHERE id=_collection;\n\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM %I;', _partition)\n INTO explain_json;\n psize := explain_json->0->'Plan'->'Plan Rows';\n estrows := _tablesample * .01 * psize;\n IF estrows < minrows THEN\n _tablesample := least(100,greatest(_tablesample, (estrows / psize) / 100));\n RAISE NOTICE '%', (psize / estrows) / 100;\n END IF;\n RAISE NOTICE 'Using tablesample % to find missing queryables from % % that has ~% rows estrows: %', _tablesample, _collection, _partition, psize, estrows;\n\n q := format(\n $q$\n WITH q AS (\n SELECT * FROM queryables\n WHERE\n collection_ids IS NULL\n OR %L = ANY(collection_ids)\n ), t AS (\n SELECT\n content->'properties' AS properties\n FROM\n %I\n TABLESAMPLE SYSTEM(%L)\n ), p AS (\n SELECT DISTINCT ON (key)\n key,\n value,\n s.definition\n FROM t\n JOIN LATERAL jsonb_each(properties) ON TRUE\n LEFT JOIN q ON (q.name=key)\n LEFT JOIN stac_extension_queryables s ON (s.name=key)\n WHERE q.definition IS NULL\n )\n SELECT\n %L,\n key,\n COALESCE(definition, jsonb_build_object('type',jsonb_typeof(value))) as definition,\n CASE\n WHEN definition->>'type' = 'integer' THEN 'to_int'\n WHEN COALESCE(definition->>'type', jsonb_typeof(value)) = 'number' THEN 'to_float'\n WHEN COALESCE(definition->>'type', jsonb_typeof(value)) = 'array' THEN 'to_text_array'\n ELSE 'to_text'\n END\n FROM p;\n $q$,\n _collection,\n _partition,\n _tablesample,\n _collection\n );\n RETURN QUERY EXECUTE q;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION missing_queryables(_tablesample float DEFAULT 5) RETURNS TABLE(collection_ids text[], name text, definition jsonb, property_wrapper text) AS $$\n SELECT\n array_agg(collection),\n name,\n definition,\n property_wrapper\n FROM\n collections\n JOIN LATERAL\n missing_queryables(id, _tablesample) c\n ON TRUE\n GROUP BY\n 2,3,4\n ORDER BY 2,1\n ;\n$$ LANGUAGE SQL;\nCREATE OR REPLACE FUNCTION parse_dtrange(\n _indate jsonb,\n relative_base timestamptz DEFAULT date_trunc('hour', CURRENT_TIMESTAMP)\n) RETURNS tstzrange AS $$\nDECLARE\n timestrs text[];\n s timestamptz;\n e timestamptz;\nBEGIN\n timestrs :=\n CASE\n WHEN _indate ? 'timestamp' THEN\n ARRAY[_indate->>'timestamp']\n WHEN _indate ? 'interval' THEN\n to_text_array(_indate->'interval')\n WHEN jsonb_typeof(_indate) = 'array' THEN\n to_text_array(_indate)\n ELSE\n regexp_split_to_array(\n _indate->>0,\n '/'\n )\n END;\n RAISE NOTICE 'TIMESTRS %', timestrs;\n IF cardinality(timestrs) = 1 THEN\n IF timestrs[1] ILIKE 'P%' THEN\n RETURN tstzrange(relative_base - upper(timestrs[1])::interval, relative_base, '[)');\n END IF;\n s := timestrs[1]::timestamptz;\n RETURN tstzrange(s, s, '[]');\n END IF;\n\n IF cardinality(timestrs) != 2 THEN\n RAISE EXCEPTION 'Timestamp cannot have more than 2 values';\n END IF;\n\n IF timestrs[1] = '..' OR timestrs[1] = '' THEN\n s := '-infinity'::timestamptz;\n e := timestrs[2]::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] = '..' OR timestrs[2] = '' THEN\n s := timestrs[1]::timestamptz;\n e := 'infinity'::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[1] ILIKE 'P%' AND timestrs[2] NOT ILIKE 'P%' THEN\n e := timestrs[2]::timestamptz;\n s := e - upper(timestrs[1])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] ILIKE 'P%' AND timestrs[1] NOT ILIKE 'P%' THEN\n s := timestrs[1]::timestamptz;\n e := s + upper(timestrs[2])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n s := timestrs[1]::timestamptz;\n e := timestrs[2]::timestamptz;\n\n RETURN tstzrange(s,e,'[)');\n\n RETURN NULL;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT PARALLEL SAFE SET TIME ZONE 'UTC';\n\nCREATE OR REPLACE FUNCTION parse_dtrange(\n _indate text,\n relative_base timestamptz DEFAULT CURRENT_TIMESTAMP\n) RETURNS tstzrange AS $$\n SELECT parse_dtrange(to_jsonb(_indate), relative_base);\n$$ LANGUAGE SQL STABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION temporal_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\n ll text := 'datetime';\n lh text := 'end_datetime';\n rrange tstzrange;\n rl text;\n rh text;\n outq text;\nBEGIN\n rrange := parse_dtrange(args->1);\n RAISE NOTICE 'Constructing temporal query OP: %, ARGS: %, RRANGE: %', op, args, rrange;\n op := lower(op);\n rl := format('%L::timestamptz', lower(rrange));\n rh := format('%L::timestamptz', upper(rrange));\n outq := CASE op\n WHEN 't_before' THEN 'lh < rl'\n WHEN 't_after' THEN 'll > rh'\n WHEN 't_meets' THEN 'lh = rl'\n WHEN 't_metby' THEN 'll = rh'\n WHEN 't_overlaps' THEN 'll < rl AND rl < lh < rh'\n WHEN 't_overlappedby' THEN 'rl < ll < rh AND lh > rh'\n WHEN 't_starts' THEN 'll = rl AND lh < rh'\n WHEN 't_startedby' THEN 'll = rl AND lh > rh'\n WHEN 't_during' THEN 'll > rl AND lh < rh'\n WHEN 't_contains' THEN 'll < rl AND lh > rh'\n WHEN 't_finishes' THEN 'll > rl AND lh = rh'\n WHEN 't_finishedby' THEN 'll < rl AND lh = rh'\n WHEN 't_equals' THEN 'll = rl AND lh = rh'\n WHEN 't_disjoint' THEN 'NOT (ll <= rh AND lh >= rl)'\n WHEN 't_intersects' THEN 'll <= rh AND lh >= rl'\n WHEN 'anyinteracts' THEN 'll <= rh AND lh >= rl'\n END;\n outq := regexp_replace(outq, '\\mll\\M', ll);\n outq := regexp_replace(outq, '\\mlh\\M', lh);\n outq := regexp_replace(outq, '\\mrl\\M', rl);\n outq := regexp_replace(outq, '\\mrh\\M', rh);\n outq := format('(%s)', outq);\n RETURN outq;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\n\n\nCREATE OR REPLACE FUNCTION spatial_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\n geom text;\n j jsonb := args->1;\nBEGIN\n op := lower(op);\n RAISE NOTICE 'Constructing spatial query OP: %, ARGS: %', op, args;\n IF op NOT IN ('s_equals','s_disjoint','s_touches','s_within','s_overlaps','s_crosses','s_intersects','intersects','s_contains') THEN\n RAISE EXCEPTION 'Spatial Operator % Not Supported', op;\n END IF;\n op := regexp_replace(op, '^s_', 'st_');\n IF op = 'intersects' THEN\n op := 'st_intersects';\n END IF;\n -- Convert geometry to WKB string\n IF j ? 'type' AND j ? 'coordinates' THEN\n geom := st_geomfromgeojson(j)::text;\n ELSIF jsonb_typeof(j) = 'array' THEN\n geom := bbox_geom(j)::text;\n END IF;\n\n RETURN format('%s(geometry, %L::geometry)', op, geom);\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION query_to_cql2(q jsonb) RETURNS jsonb AS $$\n-- Translates anything passed in through the deprecated \"query\" into equivalent CQL2\nWITH t AS (\n SELECT key as property, value as ops\n FROM jsonb_each(q)\n), t2 AS (\n SELECT property, (jsonb_each(ops)).*\n FROM t WHERE jsonb_typeof(ops) = 'object'\n UNION ALL\n SELECT property, 'eq', ops\n FROM t WHERE jsonb_typeof(ops) != 'object'\n)\nSELECT\n jsonb_strip_nulls(jsonb_build_object(\n 'op', 'and',\n 'args', jsonb_agg(\n jsonb_build_object(\n 'op', key,\n 'args', jsonb_build_array(\n jsonb_build_object('property',property),\n value\n )\n )\n )\n )\n) as qcql FROM t2\n;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\nCREATE OR REPLACE FUNCTION cql1_to_cql2(j jsonb) RETURNS jsonb AS $$\nDECLARE\n args jsonb;\n ret jsonb;\nBEGIN\n RAISE NOTICE 'CQL1_TO_CQL2: %', j;\n IF j ? 'filter' THEN\n RETURN cql1_to_cql2(j->'filter');\n END IF;\n IF j ? 'property' THEN\n RETURN j;\n END IF;\n IF jsonb_typeof(j) = 'array' THEN\n SELECT jsonb_agg(cql1_to_cql2(el)) INTO args FROM jsonb_array_elements(j) el;\n RETURN args;\n END IF;\n IF jsonb_typeof(j) = 'number' THEN\n RETURN j;\n END IF;\n IF jsonb_typeof(j) = 'string' THEN\n RETURN j;\n END IF;\n\n IF jsonb_typeof(j) = 'object' THEN\n SELECT jsonb_build_object(\n 'op', key,\n 'args', cql1_to_cql2(value)\n ) INTO ret\n FROM jsonb_each(j)\n WHERE j IS NOT NULL;\n RETURN ret;\n END IF;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE STRICT;\n\nCREATE TABLE cql2_ops (\n op text PRIMARY KEY,\n template text,\n types text[]\n);\nINSERT INTO cql2_ops (op, template, types) VALUES\n ('eq', '%s = %s', NULL),\n ('neq', '%s != %s', NULL),\n ('ne', '%s != %s', NULL),\n ('!=', '%s != %s', NULL),\n ('<>', '%s != %s', NULL),\n ('lt', '%s < %s', NULL),\n ('lte', '%s <= %s', NULL),\n ('gt', '%s > %s', NULL),\n ('gte', '%s >= %s', NULL),\n ('le', '%s <= %s', NULL),\n ('ge', '%s >= %s', NULL),\n ('=', '%s = %s', NULL),\n ('<', '%s < %s', NULL),\n ('<=', '%s <= %s', NULL),\n ('>', '%s > %s', NULL),\n ('>=', '%s >= %s', NULL),\n ('like', '%s LIKE %s', NULL),\n ('ilike', '%s ILIKE %s', NULL),\n ('+', '%s + %s', NULL),\n ('-', '%s - %s', NULL),\n ('*', '%s * %s', NULL),\n ('/', '%s / %s', NULL),\n ('not', 'NOT (%s)', NULL),\n ('between', '%s BETWEEN %s AND %s', NULL),\n ('isnull', '%s IS NULL', NULL),\n ('upper', 'upper(%s)', NULL),\n ('lower', 'lower(%s)', NULL)\nON CONFLICT (op) DO UPDATE\n SET\n template = EXCLUDED.template\n;\n\n\nCREATE OR REPLACE FUNCTION cql2_query(j jsonb, wrapper text DEFAULT NULL) RETURNS text AS $$\n#variable_conflict use_variable\nDECLARE\n args jsonb := j->'args';\n arg jsonb;\n op text := lower(j->>'op');\n cql2op RECORD;\n literal text;\n _wrapper text;\n leftarg text;\n rightarg text;\nBEGIN\n IF j IS NULL OR (op IS NOT NULL AND args IS NULL) THEN\n RETURN NULL;\n END IF;\n RAISE NOTICE 'CQL2_QUERY: %', j;\n IF j ? 'filter' THEN\n RETURN cql2_query(j->'filter');\n END IF;\n\n IF j ? 'upper' THEN\n RETURN cql2_query(jsonb_build_object('op', 'upper', 'args', j->'upper'));\n END IF;\n\n IF j ? 'lower' THEN\n RETURN cql2_query(jsonb_build_object('op', 'lower', 'args', j->'lower'));\n END IF;\n\n -- Temporal Query\n IF op ilike 't_%' or op = 'anyinteracts' THEN\n RETURN temporal_op_query(op, args);\n END IF;\n\n -- If property is a timestamp convert it to text to use with\n -- general operators\n IF j ? 'timestamp' THEN\n RETURN format('%L::timestamptz', to_tstz(j->'timestamp'));\n END IF;\n IF j ? 'interval' THEN\n RAISE EXCEPTION 'Please use temporal operators when using intervals.';\n RETURN NONE;\n END IF;\n\n -- Spatial Query\n IF op ilike 's_%' or op = 'intersects' THEN\n RETURN spatial_op_query(op, args);\n END IF;\n\n IF op IN ('a_equals','a_contains','a_contained_by','a_overlaps') THEN\n IF args->0 ? 'property' THEN\n leftarg := format('to_text_array(%s)', (queryable(args->0->>'property')).path);\n END IF;\n IF args->1 ? 'property' THEN\n rightarg := format('to_text_array(%s)', (queryable(args->1->>'property')).path);\n END IF;\n RETURN FORMAT(\n '%s %s %s',\n COALESCE(leftarg, quote_literal(to_text_array(args->0))),\n CASE op\n WHEN 'a_equals' THEN '='\n WHEN 'a_contains' THEN '@>'\n WHEN 'a_contained_by' THEN '<@'\n WHEN 'a_overlaps' THEN '&&'\n END,\n COALESCE(rightarg, quote_literal(to_text_array(args->1)))\n );\n END IF;\n\n IF op = 'in' THEN\n RAISE NOTICE 'IN : % % %', args, jsonb_build_array(args->0), args->1;\n args := jsonb_build_array(args->0) || (args->1);\n RAISE NOTICE 'IN2 : %', args;\n END IF;\n\n\n\n IF op = 'between' THEN\n args = jsonb_build_array(\n args->0,\n args->1->0,\n args->1->1\n );\n END IF;\n\n -- Make sure that args is an array and run cql2_query on\n -- each element of the array\n RAISE NOTICE 'ARGS PRE: %', args;\n IF j ? 'args' THEN\n IF jsonb_typeof(args) != 'array' THEN\n args := jsonb_build_array(args);\n END IF;\n\n IF jsonb_path_exists(args, '$[*] ? (@.property == \"id\" || @.property == \"datetime\" || @.property == \"end_datetime\" || @.property == \"collection\")') THEN\n wrapper := NULL;\n ELSE\n -- if any of the arguments are a property, try to get the property_wrapper\n FOR arg IN SELECT jsonb_path_query(args, '$[*] ? (@.property != null)') LOOP\n RAISE NOTICE 'Arg: %', arg;\n wrapper := (queryable(arg->>'property')).nulled_wrapper;\n RAISE NOTICE 'Property: %, Wrapper: %', arg, wrapper;\n IF wrapper IS NOT NULL THEN\n EXIT;\n END IF;\n END LOOP;\n\n -- if the property was not in queryables, see if any args were numbers\n IF\n wrapper IS NULL\n AND jsonb_path_exists(args, '$[*] ? (@.type()==\"number\")')\n THEN\n wrapper := 'to_float';\n END IF;\n wrapper := coalesce(wrapper, 'to_text');\n END IF;\n\n SELECT jsonb_agg(cql2_query(a, wrapper))\n INTO args\n FROM jsonb_array_elements(args) a;\n END IF;\n RAISE NOTICE 'ARGS: %', args;\n\n IF op IN ('and', 'or') THEN\n RETURN\n format(\n '(%s)',\n array_to_string(to_text_array(args), format(' %s ', upper(op)))\n );\n END IF;\n\n IF op = 'in' THEN\n RAISE NOTICE 'IN -- % %', args->0, to_text(args->0);\n RETURN format(\n '%s IN (%s)',\n to_text(args->0),\n array_to_string((to_text_array(args))[2:], ',')\n );\n END IF;\n\n -- Look up template from cql2_ops\n IF j ? 'op' THEN\n SELECT * INTO cql2op FROM cql2_ops WHERE cql2_ops.op ilike op;\n IF FOUND THEN\n -- If specific index set in queryables for a property cast other arguments to that type\n\n RETURN format(\n cql2op.template,\n VARIADIC (to_text_array(args))\n );\n ELSE\n RAISE EXCEPTION 'Operator % Not Supported.', op;\n END IF;\n END IF;\n\n\n IF wrapper IS NOT NULL THEN\n RAISE NOTICE 'Wrapping % with %', j, wrapper;\n IF j ? 'property' THEN\n RETURN format('%I(%s)', wrapper, (queryable(j->>'property')).path);\n ELSE\n RETURN format('%I(%L)', wrapper, j);\n END IF;\n ELSIF j ? 'property' THEN\n RETURN quote_ident(j->>'property');\n END IF;\n\n RETURN quote_literal(to_text(j));\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION paging_dtrange(\n j jsonb\n) RETURNS tstzrange AS $$\nDECLARE\n op text;\n filter jsonb := j->'filter';\n dtrange tstzrange := tstzrange('-infinity'::timestamptz,'infinity'::timestamptz);\n sdate timestamptz := '-infinity'::timestamptz;\n edate timestamptz := 'infinity'::timestamptz;\n jpitem jsonb;\nBEGIN\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n END IF;\n IF NOT (filter @? '$.**.op ? (@ == \"or\" || @ == \"not\")') THEN\n FOR jpitem IN SELECT j FROM jsonb_path_query(filter,'strict $.** ? (@.args[*].property == \"datetime\")'::jsonpath) j LOOP\n op := lower(jpitem->>'op');\n dtrange := parse_dtrange(jpitem->'args'->1);\n IF op IN ('<=', 'lt', 'lte', '<', 'le', 't_before') THEN\n sdate := greatest(sdate,'-infinity');\n edate := least(edate, upper(dtrange));\n ELSIF op IN ('>=', '>', 'gt', 'gte', 'ge', 't_after') THEN\n edate := least(edate, 'infinity');\n sdate := greatest(sdate, lower(dtrange));\n ELSIF op IN ('=', 'eq') THEN\n edate := least(edate, upper(dtrange));\n sdate := greatest(sdate, lower(dtrange));\n END IF;\n RAISE NOTICE '2 OP: %, ARGS: %, DTRANGE: %, SDATE: %, EDATE: %', op, jpitem->'args'->1, dtrange, sdate, edate;\n END LOOP;\n END IF;\n IF sdate > edate THEN\n RETURN 'empty'::tstzrange;\n END IF;\n RETURN tstzrange(sdate,edate, '[]');\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT SET TIME ZONE 'UTC';\n\nCREATE OR REPLACE FUNCTION paging_collections(\n IN j jsonb\n) RETURNS text[] AS $$\nDECLARE\n filter jsonb := j->'filter';\n jpitem jsonb;\n op text;\n args jsonb;\n arg jsonb;\n collections text[];\nBEGIN\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n END IF;\n IF NOT (filter @? '$.**.op ? (@ == \"or\" || @ == \"not\")') THEN\n FOR jpitem IN SELECT j FROM jsonb_path_query(filter,'strict $.** ? (@.args[*].property == \"collection\")'::jsonpath) j LOOP\n RAISE NOTICE 'JPITEM: %', jpitem;\n op := jpitem->>'op';\n args := jpitem->'args';\n IF op IN ('=', 'eq', 'in') THEN\n FOR arg IN SELECT a FROM jsonb_array_elements(args) a LOOP\n IF jsonb_typeof(arg) IN ('string', 'array') THEN\n RAISE NOTICE 'arg: %, collections: %', arg, collections;\n IF collections IS NULL OR collections = '{}'::text[] THEN\n collections := to_text_array(arg);\n ELSE\n collections := array_intersection(collections, to_text_array(arg));\n END IF;\n END IF;\n END LOOP;\n END IF;\n END LOOP;\n END IF;\n IF collections = '{}'::text[] THEN\n RETURN NULL;\n END IF;\n RETURN collections;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\nCREATE OR REPLACE FUNCTION collection_base_item(content jsonb) RETURNS jsonb AS $$\n SELECT jsonb_build_object(\n 'type', 'Feature',\n 'stac_version', content->'stac_version',\n 'assets', content->'item_assets',\n 'collection', content->'id'\n );\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TABLE IF NOT EXISTS collections (\n key bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,\n id text GENERATED ALWAYS AS (content->>'id') STORED UNIQUE,\n content JSONB NOT NULL,\n base_item jsonb GENERATED ALWAYS AS (pgstac.collection_base_item(content)) STORED,\n partition_trunc text CHECK (partition_trunc IN ('year', 'month'))\n);\n\n\n\nCREATE OR REPLACE FUNCTION collection_base_item(cid text) RETURNS jsonb AS $$\n SELECT pgstac.collection_base_item(content) FROM pgstac.collections WHERE id = cid LIMIT 1;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION create_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION update_collection(data jsonb) RETURNS VOID AS $$\nDECLARE\n out collections%ROWTYPE;\nBEGIN\n UPDATE collections SET content=data WHERE id = data->>'id' RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ON CONFLICT (id) DO\n UPDATE\n SET content=EXCLUDED.content\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION delete_collection(_id text) RETURNS VOID AS $$\nDECLARE\n out collections%ROWTYPE;\nBEGIN\n DELETE FROM collections WHERE id = _id RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION get_collection(id text) RETURNS jsonb AS $$\n SELECT content FROM collections\n WHERE id=$1\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION all_collections() RETURNS jsonb AS $$\n SELECT jsonb_agg(content) FROM collections;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\nCREATE TABLE items (\n id text NOT NULL,\n geometry geometry NOT NULL,\n collection text NOT NULL,\n datetime timestamptz NOT NULL,\n end_datetime timestamptz NOT NULL,\n content JSONB NOT NULL\n)\nPARTITION BY LIST (collection)\n;\n\nCREATE INDEX \"datetime_idx\" ON items USING BTREE (datetime DESC, end_datetime ASC);\nCREATE INDEX \"geometry_idx\" ON items USING GIST (geometry);\n\nCREATE STATISTICS datetime_stats (dependencies) on datetime, end_datetime from items;\n\nALTER TABLE items ADD CONSTRAINT items_collections_fk FOREIGN KEY (collection) REFERENCES collections(id) ON DELETE CASCADE DEFERRABLE;\n\nCREATE OR REPLACE FUNCTION partition_after_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p text;\n t timestamptz := clock_timestamp();\nBEGIN\n RAISE NOTICE 'Updating partition stats %', t;\n FOR p IN SELECT DISTINCT partition\n FROM newdata n JOIN partition_sys_meta p\n ON (n.collection=p.collection AND n.datetime <@ p.partition_dtrange)\n LOOP\n PERFORM run_or_queue(format('SELECT update_partition_stats(%L, %L);', p, true));\n END LOOP;\n RAISE NOTICE 't: % %', t, clock_timestamp() - t;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE TRIGGER items_after_insert_trigger\nAFTER INSERT ON items\nREFERENCING NEW TABLE AS newdata\nFOR EACH STATEMENT\nEXECUTE FUNCTION partition_after_triggerfunc();\n\nCREATE TRIGGER items_after_update_trigger\nAFTER DELETE ON items\nREFERENCING OLD TABLE AS newdata\nFOR EACH STATEMENT\nEXECUTE FUNCTION partition_after_triggerfunc();\n\nCREATE TRIGGER items_after_delete_trigger\nAFTER UPDATE ON items\nREFERENCING NEW TABLE AS newdata\nFOR EACH STATEMENT\nEXECUTE FUNCTION partition_after_triggerfunc();\n\n\nCREATE OR REPLACE FUNCTION content_slim(_item jsonb) RETURNS jsonb AS $$\n SELECT strip_jsonb(_item - '{id,geometry,collection,type}'::text[], collection_base_item(_item->>'collection')) - '{id,geometry,collection,type}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_dehydrate(content jsonb) RETURNS items AS $$\n SELECT\n content->>'id' as id,\n stac_geom(content) as geometry,\n content->>'collection' as collection,\n stac_datetime(content) as datetime,\n stac_end_datetime(content) as end_datetime,\n content_slim(content) as content\n ;\n$$ LANGUAGE SQL STABLE;\n\nCREATE OR REPLACE FUNCTION include_field(f text, fields jsonb DEFAULT '{}'::jsonb) RETURNS boolean AS $$\nDECLARE\n includes jsonb := fields->'include';\n excludes jsonb := fields->'exclude';\nBEGIN\n IF f IS NULL THEN\n RETURN NULL;\n END IF;\n\n\n IF\n jsonb_typeof(excludes) = 'array'\n AND jsonb_array_length(excludes)>0\n AND excludes ? f\n THEN\n RETURN FALSE;\n END IF;\n\n IF\n (\n jsonb_typeof(includes) = 'array'\n AND jsonb_array_length(includes) > 0\n AND includes ? f\n ) OR\n (\n includes IS NULL\n OR jsonb_typeof(includes) = 'null'\n OR jsonb_array_length(includes) = 0\n )\n THEN\n RETURN TRUE;\n END IF;\n\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nDROP FUNCTION IF EXISTS content_hydrate(jsonb, jsonb, jsonb);\nCREATE OR REPLACE FUNCTION content_hydrate(\n _item jsonb,\n _base_item jsonb,\n fields jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\n SELECT merge_jsonb(\n jsonb_fields(_item, fields),\n jsonb_fields(_base_item, fields)\n );\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION content_hydrate(_item items, _collection collections, fields jsonb DEFAULT '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\n content jsonb;\n base_item jsonb := _collection.base_item;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry, 20)::jsonb;\n END IF;\n output := content_hydrate(\n jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'collection', _item.collection,\n 'type', 'Feature'\n ) || _item.content,\n _collection.base_item,\n fields\n );\n\n RETURN output;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_nonhydrated(\n _item items,\n fields jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry, 20)::jsonb;\n END IF;\n output := jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'collection', _item.collection,\n 'type', 'Feature'\n ) || _item.content;\n RETURN output;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_hydrate(_item items, fields jsonb DEFAULT '{}'::jsonb) RETURNS jsonb AS $$\n SELECT content_hydrate(\n _item,\n (SELECT c FROM collections c WHERE id=_item.collection LIMIT 1),\n fields\n );\n$$ LANGUAGE SQL STABLE;\n\n\nCREATE UNLOGGED TABLE items_staging (\n content JSONB NOT NULL\n);\nCREATE UNLOGGED TABLE items_staging_ignore (\n content JSONB NOT NULL\n);\nCREATE UNLOGGED TABLE items_staging_upsert (\n content JSONB NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION items_staging_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p record;\n _partitions text[];\n part text;\n ts timestamptz := clock_timestamp();\nBEGIN\n RAISE NOTICE 'Creating Partitions. %', clock_timestamp() - ts;\n\n FOR part IN WITH t AS (\n SELECT\n n.content->>'collection' as collection,\n stac_daterange(n.content->'properties') as dtr,\n partition_trunc\n FROM newdata n JOIN collections ON (n.content->>'collection'=collections.id)\n ), p AS (\n SELECT\n collection,\n COALESCE(date_trunc(partition_trunc::text, lower(dtr)),'-infinity') as d,\n tstzrange(min(lower(dtr)),max(lower(dtr)),'[]') as dtrange,\n tstzrange(min(upper(dtr)),max(upper(dtr)),'[]') as edtrange\n FROM t\n GROUP BY 1,2\n ) SELECT check_partition(collection, dtrange, edtrange) FROM p LOOP\n RAISE NOTICE 'Partition %', part;\n END LOOP;\n\n RAISE NOTICE 'Doing the insert. %', clock_timestamp() - ts;\n IF TG_TABLE_NAME = 'items_staging' THEN\n INSERT INTO items\n SELECT\n (content_dehydrate(content)).*\n FROM newdata;\n RAISE NOTICE 'Doing the delete. %', clock_timestamp() - ts;\n DELETE FROM items_staging;\n ELSIF TG_TABLE_NAME = 'items_staging_ignore' THEN\n INSERT INTO items\n SELECT\n (content_dehydrate(content)).*\n FROM newdata\n ON CONFLICT DO NOTHING;\n RAISE NOTICE 'Doing the delete. %', clock_timestamp() - ts;\n DELETE FROM items_staging_ignore;\n ELSIF TG_TABLE_NAME = 'items_staging_upsert' THEN\n WITH staging_formatted AS (\n SELECT (content_dehydrate(content)).* FROM newdata\n ), deletes AS (\n DELETE FROM items i USING staging_formatted s\n WHERE\n i.id = s.id\n AND i.collection = s.collection\n AND i IS DISTINCT FROM s\n RETURNING i.id, i.collection\n )\n INSERT INTO items\n SELECT s.* FROM\n staging_formatted s\n ON CONFLICT DO NOTHING;\n RAISE NOTICE 'Doing the delete. %', clock_timestamp() - ts;\n DELETE FROM items_staging_upsert;\n END IF;\n RAISE NOTICE 'Done. %', clock_timestamp() - ts;\n\n RETURN NULL;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER items_staging_insert_trigger AFTER INSERT ON items_staging REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\nCREATE TRIGGER items_staging_insert_ignore_trigger AFTER INSERT ON items_staging_ignore REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\nCREATE TRIGGER items_staging_insert_upsert_trigger AFTER INSERT ON items_staging_upsert REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\n\nCREATE OR REPLACE FUNCTION item_by_id(_id text, _collection text DEFAULT NULL) RETURNS items AS\n$$\nDECLARE\n i items%ROWTYPE;\nBEGIN\n SELECT * INTO i FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection) LIMIT 1;\n RETURN i;\nEND;\n$$ LANGUAGE PLPGSQL STABLE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION get_item(_id text, _collection text DEFAULT NULL) RETURNS jsonb AS $$\n SELECT content_hydrate(items) FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection);\n$$ LANGUAGE SQL STABLE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION delete_item(_id text, _collection text DEFAULT NULL) RETURNS VOID AS $$\nDECLARE\nout items%ROWTYPE;\nBEGIN\n DELETE FROM items WHERE id = _id AND (_collection IS NULL OR collection=_collection) RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n--/*\nCREATE OR REPLACE FUNCTION create_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION update_item(content jsonb) RETURNS VOID AS $$\nDECLARE\n old items %ROWTYPE;\n out items%ROWTYPE;\nBEGIN\n PERFORM delete_item(content->>'id', content->>'collection');\n PERFORM create_item(content);\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION create_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION collection_bbox(id text) RETURNS jsonb AS $$\n SELECT (replace(replace(replace(st_extent(geometry)::text,'BOX(','[['),')',']]'),' ',','))::jsonb\n FROM items WHERE collection=$1;\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION collection_temporal_extent(id text) RETURNS jsonb AS $$\n SELECT to_jsonb(array[array[min(datetime)::text, max(datetime)::text]])\n FROM items WHERE collection=$1;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection_extents() RETURNS VOID AS $$\nUPDATE collections SET\n content = content ||\n jsonb_build_object(\n 'extent', jsonb_build_object(\n 'spatial', jsonb_build_object(\n 'bbox', collection_bbox(collections.id)\n ),\n 'temporal', jsonb_build_object(\n 'interval', collection_temporal_extent(collections.id)\n )\n )\n )\n;\n$$ LANGUAGE SQL;\nCREATE TABLE partition_stats (\n partition text PRIMARY KEY,\n dtrange tstzrange,\n edtrange tstzrange,\n spatial geometry,\n last_updated timestamptz,\n keys text[]\n) WITH (FILLFACTOR=90);\n\nCREATE INDEX partitions_range_idx ON partition_stats USING GIST(dtrange);\n\n\nCREATE OR REPLACE FUNCTION constraint_tstzrange(expr text) RETURNS tstzrange AS $$\n WITH t AS (\n SELECT regexp_matches(\n expr,\n E'\\\\(''\\([0-9 :+-]*\\)''\\\\).*\\\\(''\\([0-9 :+-]*\\)''\\\\)'\n ) AS m\n ) SELECT tstzrange(m[1]::timestamptz, m[2]::timestamptz) FROM t\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE STRICT;\n\nCREATE OR REPLACE FUNCTION dt_constraint(coid oid, OUT dt tstzrange, OUT edt tstzrange) RETURNS RECORD AS $$\nDECLARE\n expr text := pg_get_constraintdef(coid);\n matches timestamptz[];\nBEGIN\n IF expr LIKE '%NULL%' THEN\n dt := tstzrange(null::timestamptz, null::timestamptz);\n edt := tstzrange(null::timestamptz, null::timestamptz);\n RETURN;\n END IF;\n WITH f AS (SELECT (regexp_matches(expr, E'([0-9]{4}-[0-1][0-9]-[0-3][0-9] [0-2][0-9]:[0-5][0-9]:[0-5][0-9]\\.?[0-9]*)', 'g'))[1] f)\n SELECT array_agg(f::timestamptz) INTO matches FROM f;\n IF cardinality(matches) = 4 THEN\n dt := tstzrange(matches[1], matches[2],'[]');\n edt := tstzrange(matches[3], matches[4], '[]');\n RETURN;\n ELSIF cardinality(matches) = 2 THEN\n edt := tstzrange(matches[1], matches[2],'[]');\n RETURN;\n END IF;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\nCREATE OR REPLACE VIEW partition_sys_meta AS\nSELECT\n relid::text as partition,\n replace(replace(CASE WHEN level = 1 THEN pg_get_expr(c.relpartbound, c.oid)\n ELSE pg_get_expr(parent.relpartbound, parent.oid)\n END, 'FOR VALUES IN (''',''), ''')','') AS collection,\n level,\n c.reltuples,\n c.relhastriggers,\n COALESCE(constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as partition_dtrange,\n COALESCE((dt_constraint(edt.oid)).dt, constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as constraint_dtrange,\n COALESCE((dt_constraint(edt.oid)).edt, tstzrange('-infinity', 'infinity','[]')) as constraint_edtrange\nFROM\n pg_partition_tree('items')\n JOIN pg_class c ON (relid::regclass = c.oid)\n JOIN pg_class parent ON (parentrelid::regclass = parent.oid AND isleaf)\n LEFT JOIN pg_constraint edt ON (conrelid=c.oid AND contype='c')\nWHERE isleaf\n;\n\nCREATE VIEW partitions AS\nSELECT * FROM partition_sys_meta LEFT JOIN partition_stats USING (partition);\n\nCREATE OR REPLACE FUNCTION update_partition_stats_q(_partition text, istrigger boolean default false) RETURNS VOID AS $$\nDECLARE\nBEGIN\n PERFORM run_or_queue(\n format('SELECT update_partition_stats(%L, %L);', _partition, istrigger)\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION update_partition_stats(_partition text, istrigger boolean default false) RETURNS VOID AS $$\nDECLARE\n dtrange tstzrange;\n edtrange tstzrange;\n cdtrange tstzrange;\n cedtrange tstzrange;\n extent geometry;\n collection text;\nBEGIN\n RAISE NOTICE 'Updating stats for %.', _partition;\n EXECUTE format(\n $q$\n SELECT\n tstzrange(min(datetime), max(datetime),'[]'),\n tstzrange(min(end_datetime), max(end_datetime), '[]')\n FROM %I\n $q$,\n _partition\n ) INTO dtrange, edtrange;\n extent := st_estimatedextent('pgstac', _partition, 'geometry');\n INSERT INTO partition_stats (partition, dtrange, edtrange, spatial, last_updated)\n SELECT _partition, dtrange, edtrange, extent, now()\n ON CONFLICT (partition) DO\n UPDATE SET\n dtrange=EXCLUDED.dtrange,\n edtrange=EXCLUDED.edtrange,\n spatial=EXCLUDED.spatial,\n last_updated=EXCLUDED.last_updated\n ;\n SELECT\n constraint_dtrange, constraint_edtrange, partitions.collection\n INTO cdtrange, cedtrange, collection\n FROM partitions WHERE partition = _partition;\n\n RAISE NOTICE 'Checking if we need to modify constraints.';\n IF\n (cdtrange IS DISTINCT FROM dtrange OR edtrange IS DISTINCT FROM cedtrange)\n AND NOT istrigger\n THEN\n RAISE NOTICE 'Modifying Constraints';\n RAISE NOTICE 'Existing % %', cdtrange, cedtrange;\n RAISE NOTICE 'New % %', dtrange, edtrange;\n PERFORM drop_table_constraints(_partition);\n PERFORM create_table_constraints(_partition, dtrange, edtrange);\n END IF;\n RAISE NOTICE 'Checking if we need to update collection extents.';\n IF get_setting_bool('update_collection_extent') THEN\n RAISE NOTICE 'updating collection extent for %', collection;\n PERFORM run_or_queue(format($q$\n UPDATE collections\n SET content = jsonb_set_lax(\n content,\n '{extent}'::text[],\n collection_extent(%L),\n true,\n 'use_json_null'\n ) WHERE id=%L\n ;\n $q$, collection, collection));\n ELSE\n RAISE NOTICE 'Not updating collection extent for %', collection;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL STRICT;\n\n\nCREATE OR REPLACE FUNCTION partition_name( IN collection text, IN dt timestamptz, OUT partition_name text, OUT partition_range tstzrange) AS $$\nDECLARE\n c RECORD;\n parent_name text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n parent_name := format('_items_%s', c.key);\n\n\n IF c.partition_trunc = 'year' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYY'));\n ELSIF c.partition_trunc = 'month' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYYMM'));\n ELSE\n partition_name := parent_name;\n partition_range := tstzrange('-infinity'::timestamptz, 'infinity'::timestamptz, '[]');\n END IF;\n IF partition_range IS NULL THEN\n partition_range := tstzrange(\n date_trunc(c.partition_trunc::text, dt),\n date_trunc(c.partition_trunc::text, dt) + concat('1 ', c.partition_trunc)::interval\n );\n END IF;\n RETURN;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION drop_table_constraints(t text) RETURNS text AS $$\nDECLARE\n q text;\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM partitions WHERE partition=t) THEN\n RETURN NULL;\n END IF;\n FOR q IN SELECT FORMAT(\n $q$\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n $q$,\n t,\n conname\n ) FROM pg_constraint\n WHERE conrelid=t::regclass::oid AND contype='c'\n LOOP\n EXECUTE q;\n END LOOP;\n RETURN t;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE OR REPLACE FUNCTION create_table_constraints(t text, _dtrange tstzrange, _edtrange tstzrange) RETURNS text AS $$\nDECLARE\n q text;\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM partitions WHERE partition=t) THEN\n RETURN NULL;\n END IF;\n RAISE NOTICE 'Creating Table Constraints for % % %', t, _dtrange, _edtrange;\n IF _dtrange = 'empty' AND _edtrange = 'empty' THEN\n q :=format(\n $q$\n ALTER TABLE %I\n ADD CONSTRAINT %I\n CHECK (((datetime IS NULL) AND (end_datetime IS NULL))) NOT VALID\n ;\n ALTER TABLE %I\n VALIDATE CONSTRAINT %I\n ;\n $q$,\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t)\n );\n ELSE\n q :=format(\n $q$\n ALTER TABLE %I\n ADD CONSTRAINT %I\n CHECK (\n (datetime >= %L)\n AND (datetime <= %L)\n AND (end_datetime >= %L)\n AND (end_datetime <= %L)\n ) NOT VALID\n ;\n ALTER TABLE %I\n VALIDATE CONSTRAINT %I\n ;\n $q$,\n t,\n format('%s_dt', t),\n lower(_dtrange),\n upper(_dtrange),\n lower(_edtrange),\n upper(_edtrange),\n t,\n format('%s_dt', t)\n );\n END IF;\n PERFORM run_or_queue(q);\n RETURN t;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION check_partition(\n _collection text,\n _dtrange tstzrange,\n _edtrange tstzrange\n) RETURNS text AS $$\nDECLARE\n c RECORD;\n pm RECORD;\n _partition_name text;\n _partition_dtrange tstzrange;\n _constraint_dtrange tstzrange;\n _constraint_edtrange tstzrange;\n q text;\n deferrable_q text;\n err_context text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=_collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', _collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n\n IF c.partition_trunc IS NOT NULL THEN\n _partition_dtrange := tstzrange(\n date_trunc(c.partition_trunc, lower(_dtrange)),\n date_trunc(c.partition_trunc, lower(_dtrange)) + (concat('1 ', c.partition_trunc))::interval,\n '[)'\n );\n ELSE\n _partition_dtrange := '[-infinity, infinity]'::tstzrange;\n END IF;\n\n IF NOT _partition_dtrange @> _dtrange THEN\n RAISE EXCEPTION 'dtrange % is greater than the partition size % for collection %', _dtrange, c.partition_trunc, _collection;\n END IF;\n\n\n IF c.partition_trunc = 'year' THEN\n _partition_name := format('_items_%s_%s', c.key, to_char(lower(_partition_dtrange),'YYYY'));\n ELSIF c.partition_trunc = 'month' THEN\n _partition_name := format('_items_%s_%s', c.key, to_char(lower(_partition_dtrange),'YYYYMM'));\n ELSE\n _partition_name := format('_items_%s', c.key);\n END IF;\n\n SELECT * INTO pm FROM partition_sys_meta WHERE collection=_collection AND partition_dtrange @> _dtrange;\n IF FOUND THEN\n RAISE NOTICE '% % %', _edtrange, _dtrange, pm;\n _constraint_edtrange :=\n tstzrange(\n least(\n lower(_edtrange),\n nullif(lower(pm.constraint_edtrange), '-infinity')\n ),\n greatest(\n upper(_edtrange),\n nullif(upper(pm.constraint_edtrange), 'infinity')\n ),\n '[]'\n );\n _constraint_dtrange :=\n tstzrange(\n least(\n lower(_dtrange),\n nullif(lower(pm.constraint_dtrange), '-infinity')\n ),\n greatest(\n upper(_dtrange),\n nullif(upper(pm.constraint_dtrange), 'infinity')\n ),\n '[]'\n );\n\n IF pm.constraint_edtrange @> _edtrange AND pm.constraint_dtrange @> _dtrange THEN\n RETURN pm.partition;\n ELSE\n PERFORM drop_table_constraints(_partition_name);\n END IF;\n ELSE\n _constraint_edtrange := _edtrange;\n _constraint_dtrange := _dtrange;\n END IF;\n RAISE NOTICE 'Creating partition % %', _partition_name, _partition_dtrange;\n IF c.partition_trunc IS NULL THEN\n q := format(\n $q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n $q$,\n _partition_name,\n _collection,\n concat(_partition_name,'_pk'),\n _partition_name\n );\n ELSE\n q := format(\n $q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L) PARTITION BY RANGE (datetime);\n CREATE TABLE IF NOT EXISTS %I partition OF %I FOR VALUES FROM (%L) TO (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n $q$,\n format('_items_%s', c.key),\n _collection,\n _partition_name,\n format('_items_%s', c.key),\n lower(_partition_dtrange),\n upper(_partition_dtrange),\n format('%s_pk', _partition_name),\n _partition_name\n );\n END IF;\n\n BEGIN\n EXECUTE q;\n EXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', _partition_name;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\n END;\n PERFORM create_table_constraints(_partition_name, _constraint_dtrange, _constraint_edtrange);\n PERFORM maintain_partitions(_partition_name);\n PERFORM update_partition_stats_q(_partition_name, true);\n RETURN _partition_name;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION repartition(_collection text, _partition_trunc text, triggered boolean DEFAULT FALSE) RETURNS text AS $$\nDECLARE\n c RECORD;\n q text;\n from_trunc text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=_collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', _collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n IF triggered THEN\n RAISE NOTICE 'Converting % to % partitioning via Trigger', _collection, _partition_trunc;\n ELSE\n RAISE NOTICE 'Converting % from using % to % partitioning', _collection, c.partition_trunc, _partition_trunc;\n IF c.partition_trunc IS NOT DISTINCT FROM _partition_trunc THEN\n RAISE NOTICE 'Collection % already set to use partition by %', _collection, _partition_trunc;\n RETURN _collection;\n END IF;\n END IF;\n\n IF EXISTS (SELECT 1 FROM partitions WHERE collection=_collection LIMIT 1) THEN\n EXECUTE format(\n $q$\n CREATE TEMP TABLE changepartitionstaging ON COMMIT DROP AS SELECT * FROM %I;\n DROP TABLE IF EXISTS %I CASCADE;\n WITH p AS (\n SELECT\n collection,\n CASE WHEN %L IS NULL THEN '-infinity'::timestamptz\n ELSE date_trunc(%L, datetime)\n END as d,\n tstzrange(min(datetime),max(datetime),'[]') as dtrange,\n tstzrange(min(datetime),max(datetime),'[]') as edtrange\n FROM changepartitionstaging\n GROUP BY 1,2\n ) SELECT check_partition(collection, dtrange, edtrange) FROM p;\n INSERT INTO items SELECT * FROM changepartitionstaging;\n DROP TABLE changepartitionstaging;\n $q$,\n concat('_items_', c.key),\n concat('_items_', c.key),\n c.partition_trunc,\n c.partition_trunc\n );\n END IF;\n RETURN _collection;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE OR REPLACE FUNCTION collections_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n q text;\n partition_name text := format('_items_%s', NEW.key);\n partition_exists boolean := false;\n partition_empty boolean := true;\n err_context text;\n loadtemp boolean := FALSE;\nBEGIN\n RAISE NOTICE 'Collection Trigger. % %', NEW.id, NEW.key;\n IF TG_OP = 'UPDATE' AND NEW.partition_trunc IS DISTINCT FROM OLD.partition_trunc THEN\n PERFORM repartition(NEW.id, NEW.partition_trunc, TRUE);\n END IF;\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER collections_trigger AFTER\nINSERT\nOR\nUPDATE ON collections\nFOR EACH ROW EXECUTE FUNCTION collections_trigger_func();\nCREATE OR REPLACE VIEW partition_steps AS\nSELECT\n partition as name,\n date_trunc('month',lower(partition_dtrange)) as sdate,\n date_trunc('month', upper(partition_dtrange)) + '1 month'::interval as edate\n FROM partitions WHERE partition_dtrange IS NOT NULL AND partition_dtrange != 'empty'::tstzrange\n ORDER BY dtrange ASC\n;\n\nCREATE OR REPLACE FUNCTION chunker(\n IN _where text,\n OUT s timestamptz,\n OUT e timestamptz\n) RETURNS SETOF RECORD AS $$\nDECLARE\n explain jsonb;\nBEGIN\n IF _where IS NULL THEN\n _where := ' TRUE ';\n END IF;\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s;', _where)\n INTO explain;\n\n RETURN QUERY\n WITH t AS (\n SELECT j->>0 as p FROM\n jsonb_path_query(\n explain,\n 'strict $.**.\"Relation Name\" ? (@ != null)'\n ) j\n ),\n parts AS (\n SELECT sdate, edate FROM t JOIN partition_steps ON (t.p = name)\n ),\n times AS (\n SELECT sdate FROM parts\n UNION\n SELECT edate FROM parts\n ),\n uniq AS (\n SELECT DISTINCT sdate FROM times ORDER BY sdate\n ),\n last AS (\n SELECT sdate, lead(sdate, 1) over () as edate FROM uniq\n )\n SELECT sdate, edate FROM last WHERE edate IS NOT NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION partition_queries(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN partitions text[] DEFAULT NULL\n) RETURNS SETOF text AS $$\nDECLARE\n query text;\n sdate timestamptz;\n edate timestamptz;\nBEGIN\nIF _where IS NULL OR trim(_where) = '' THEN\n _where = ' TRUE ';\nEND IF;\nRAISE NOTICE 'Getting chunks for % %', _where, _orderby;\nIF _orderby ILIKE 'datetime d%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 DESC LOOP\n RETURN NEXT format($q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n $q$,\n sdate,\n edate,\n _where,\n _orderby\n );\n END LOOP;\nELSIF _orderby ILIKE 'datetime a%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 ASC LOOP\n RETURN NEXT format($q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n $q$,\n sdate,\n edate,\n _where,\n _orderby\n );\n END LOOP;\nELSE\n query := format($q$\n SELECT * FROM items\n WHERE %s\n ORDER BY %s\n $q$, _where, _orderby\n );\n\n RETURN NEXT query;\n RETURN;\nEND IF;\n\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION partition_query_view(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN _limit int DEFAULT 10\n) RETURNS text AS $$\n WITH p AS (\n SELECT * FROM partition_queries(_where, _orderby) p\n )\n SELECT\n CASE WHEN EXISTS (SELECT 1 FROM p) THEN\n (SELECT format($q$\n SELECT * FROM (\n %s\n ) total LIMIT %s\n $q$,\n string_agg(\n format($q$ SELECT * FROM ( %s ) AS sub $q$, p),\n '\n UNION ALL\n '\n ),\n _limit\n ))\n ELSE NULL\n END FROM p;\n$$ LANGUAGE SQL IMMUTABLE;\n\n\n\n\nCREATE OR REPLACE FUNCTION stac_search_to_where(j jsonb) RETURNS text AS $$\nDECLARE\n where_segments text[];\n _where text;\n dtrange tstzrange;\n collections text[];\n geom geometry;\n sdate timestamptz;\n edate timestamptz;\n filterlang text;\n filter jsonb := j->'filter';\nBEGIN\n IF j ? 'ids' THEN\n where_segments := where_segments || format('id = ANY (%L) ', to_text_array(j->'ids'));\n END IF;\n\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n where_segments := where_segments || format('collection = ANY (%L) ', collections);\n END IF;\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n\n where_segments := where_segments || format(' datetime <= %L::timestamptz AND end_datetime >= %L::timestamptz ',\n edate,\n sdate\n );\n END IF;\n\n geom := stac_geom(j);\n IF geom IS NOT NULL THEN\n where_segments := where_segments || format('st_intersects(geometry, %L)',geom);\n END IF;\n\n filterlang := COALESCE(\n j->>'filter-lang',\n get_setting('default_filter_lang', j->'conf')\n );\n IF NOT filter @? '$.**.op' THEN\n filterlang := 'cql-json';\n END IF;\n\n IF filterlang NOT IN ('cql-json','cql2-json') AND j ? 'filter' THEN\n RAISE EXCEPTION '% is not a supported filter-lang. Please use cql-json or cql2-json.', filterlang;\n END IF;\n\n IF j ? 'query' AND j ? 'filter' THEN\n RAISE EXCEPTION 'Can only use either query or filter at one time.';\n END IF;\n\n IF j ? 'query' THEN\n filter := query_to_cql2(j->'query');\n ELSIF filterlang = 'cql-json' THEN\n filter := cql1_to_cql2(filter);\n END IF;\n RAISE NOTICE 'FILTER: %', filter;\n where_segments := where_segments || cql2_query(filter);\n IF cardinality(where_segments) < 1 THEN\n RETURN ' TRUE ';\n END IF;\n\n _where := array_to_string(array_remove(where_segments, NULL), ' AND ');\n\n IF _where IS NULL OR BTRIM(_where) = '' THEN\n RETURN ' TRUE ';\n END IF;\n RETURN _where;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION parse_sort_dir(_dir text, reverse boolean default false) RETURNS text AS $$\n WITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n ) SELECT\n CASE\n WHEN NOT reverse THEN d\n WHEN d = 'ASC' THEN 'DESC'\n WHEN d = 'DESC' THEN 'ASC'\n END\n FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION sort_dir_to_op(_dir text, prev boolean default false) RETURNS text AS $$\n WITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n ) SELECT\n CASE\n WHEN d = 'ASC' AND prev THEN '<='\n WHEN d = 'DESC' AND prev THEN '>='\n WHEN d = 'ASC' THEN '>='\n WHEN d = 'DESC' THEN '<='\n END\n FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION sort_sqlorderby(\n _search jsonb DEFAULT NULL,\n reverse boolean DEFAULT FALSE\n) RETURNS text AS $$\n WITH sortby AS (\n SELECT coalesce(_search->'sortby','[{\"field\":\"datetime\", \"direction\":\"desc\"}]') as sort\n ), withid AS (\n SELECT CASE\n WHEN sort @? '$[*] ? (@.field == \"id\")' THEN sort\n ELSE sort || '[{\"field\":\"id\", \"direction\":\"desc\"}]'::jsonb\n END as sort\n FROM sortby\n ), withid_rows AS (\n SELECT jsonb_array_elements(sort) as value FROM withid\n ),sorts AS (\n SELECT\n coalesce(\n -- field_orderby((items_path(value->>'field')).path_txt),\n (queryable(value->>'field')).expression\n ) as key,\n parse_sort_dir(value->>'direction', reverse) as dir\n FROM withid_rows\n )\n SELECT array_to_string(\n array_agg(concat(key, ' ', dir)),\n ', '\n ) FROM sorts;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_sort_dir(sort_item jsonb) RETURNS text AS $$\n SELECT CASE WHEN sort_item->>'direction' ILIKE 'desc%' THEN 'DESC' ELSE 'ASC' END;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION get_token_val_str(\n _field text,\n _item items\n) RETURNS text AS $$\nDECLARE\nliteral text;\nBEGIN\nRAISE NOTICE '% %', _field, _item;\nCREATE TEMP TABLE _token_item ON COMMIT DROP AS SELECT (_item).*;\nEXECUTE format($q$ SELECT quote_literal(%s) FROM _token_item $q$, _field) INTO literal;\nDROP TABLE IF EXISTS _token_item;\nRETURN literal;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION get_token_filter(_search jsonb = '{}'::jsonb, token_rec jsonb DEFAULT NULL) RETURNS text AS $$\nDECLARE\n token_id text;\n filters text[] := '{}'::text[];\n prev boolean := TRUE;\n field text;\n dir text;\n sort record;\n orfilter text := '';\n orfilters text[] := '{}'::text[];\n andfilters text[] := '{}'::text[];\n output text;\n token_where text;\n token_item items%ROWTYPE;\nBEGIN\n RAISE NOTICE 'Getting Token Filter. % %', _search, token_rec;\n -- If no token provided return NULL\n IF token_rec IS NULL THEN\n IF NOT (_search ? 'token' AND\n (\n (_search->>'token' ILIKE 'prev:%')\n OR\n (_search->>'token' ILIKE 'next:%')\n )\n ) THEN\n RETURN NULL;\n END IF;\n prev := (_search->>'token' ILIKE 'prev:%');\n token_id := substr(_search->>'token', 6);\n IF token_id IS NULL OR token_id = '' THEN\n RAISE WARNING 'next or prev set, but no token id found';\n RETURN NULL;\n END IF;\n SELECT to_jsonb(items) INTO token_rec\n FROM items WHERE id=token_id;\n END IF;\n RAISE NOTICE 'TOKEN ID: % %', token_rec, token_rec->'id';\n\n\n RAISE NOTICE 'TOKEN ID: % %', token_rec, token_rec->'id';\n token_item := jsonb_populate_record(null::items, token_rec);\n RAISE NOTICE 'TOKEN ITEM ----- %', token_item;\n\n\n CREATE TEMP TABLE sorts (\n _row int GENERATED ALWAYS AS IDENTITY NOT NULL,\n _field text PRIMARY KEY,\n _dir text NOT NULL,\n _val text\n ) ON COMMIT DROP;\n\n -- Make sure we only have distinct columns to sort with taking the first one we get\n INSERT INTO sorts (_field, _dir)\n SELECT\n (queryable(value->>'field')).expression,\n get_sort_dir(value)\n FROM\n jsonb_array_elements(coalesce(_search->'sortby','[{\"field\":\"datetime\",\"direction\":\"desc\"}]'))\n ON CONFLICT DO NOTHING\n ;\n RAISE NOTICE 'sorts 1: %', (SELECT jsonb_agg(to_json(sorts)) FROM sorts);\n -- Get the first sort direction provided. As the id is a primary key, if there are any\n -- sorts after id they won't do anything, so make sure that id is the last sort item.\n SELECT _dir INTO dir FROM sorts ORDER BY _row ASC LIMIT 1;\n IF EXISTS (SELECT 1 FROM sorts WHERE _field = 'id') THEN\n DELETE FROM sorts WHERE _row > (SELECT _row FROM sorts WHERE _field = 'id' ORDER BY _row ASC);\n ELSE\n INSERT INTO sorts (_field, _dir) VALUES ('id', dir);\n END IF;\n\n -- Add value from looked up item to the sorts table\n UPDATE sorts SET _val=get_token_val_str(_field, token_item);\n\n -- Check if all sorts are the same direction and use row comparison\n -- to filter\n RAISE NOTICE 'sorts 2: %', (SELECT jsonb_agg(to_json(sorts)) FROM sorts);\n FOR sort IN SELECT * FROM sorts ORDER BY _row asc LOOP\n orfilter := NULL;\n RAISE NOTICE 'SORT: %', sort;\n IF sort._val IS NOT NULL AND ((prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC')) THEN\n orfilter := format($f$(\n (%s < %s) OR (%s IS NULL)\n )$f$,\n sort._field,\n sort._val,\n sort._val\n );\n ELSIF sort._val IS NULL AND ((prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC')) THEN\n RAISE NOTICE '< but null';\n orfilter := format('%s IS NOT NULL', sort._field);\n ELSIF sort._val IS NULL THEN\n RAISE NOTICE '> but null';\n --orfilter := format('%s IS NULL', sort._field);\n ELSE\n orfilter := format($f$(\n (%s > %s) OR (%s IS NULL)\n )$f$,\n sort._field,\n sort._val,\n sort._field\n );\n END IF;\n RAISE NOTICE 'ORFILTER: %', orfilter;\n\n IF orfilter IS NOT NULL THEN\n IF sort._row = 1 THEN\n orfilters := orfilters || orfilter;\n ELSE\n orfilters := orfilters || format('(%s AND %s)', array_to_string(andfilters, ' AND '), orfilter);\n END IF;\n END IF;\n IF sort._val IS NOT NULL THEN\n andfilters := andfilters || format('%s = %s', sort._field, sort._val);\n ELSE\n andfilters := andfilters || format('%s IS NULL', sort._field);\n END IF;\n END LOOP;\n output := array_to_string(orfilters, ' OR ');\n\n DROP TABLE IF EXISTS sorts;\n token_where := concat('(',coalesce(output,'true'),')');\n IF trim(token_where) = '' THEN\n token_where := NULL;\n END IF;\n RAISE NOTICE 'TOKEN_WHERE: %',token_where;\n RETURN token_where;\n END;\n$$ LANGUAGE PLPGSQL SET transform_null_equals TO TRUE\n;\n\nCREATE OR REPLACE FUNCTION search_tohash(jsonb) RETURNS jsonb AS $$\n SELECT $1 - '{token,limit,context,includes,excludes}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION search_hash(jsonb, jsonb) RETURNS text AS $$\n SELECT md5(concat(search_tohash($1)::text,$2::text));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TABLE IF NOT EXISTS searches(\n hash text GENERATED ALWAYS AS (search_hash(search, metadata)) STORED PRIMARY KEY,\n search jsonb NOT NULL,\n _where text,\n orderby text,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n metadata jsonb DEFAULT '{}'::jsonb NOT NULL\n);\n\nCREATE TABLE IF NOT EXISTS search_wheres(\n id bigint generated always as identity primary key,\n _where text NOT NULL,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n statslastupdated timestamptz,\n estimated_count bigint,\n estimated_cost float,\n time_to_estimate float,\n total_count bigint,\n time_to_count float,\n partitions text[]\n);\n\nCREATE INDEX IF NOT EXISTS search_wheres_partitions ON search_wheres USING GIN (partitions);\nCREATE UNIQUE INDEX IF NOT EXISTS search_wheres_where ON search_wheres ((md5(_where)));\n\nCREATE OR REPLACE FUNCTION where_stats(inwhere text, updatestats boolean default false, conf jsonb default null) RETURNS search_wheres AS $$\nDECLARE\n t timestamptz;\n i interval;\n explain_json jsonb;\n partitions text[];\n sw search_wheres%ROWTYPE;\n inwhere_hash text := md5(inwhere);\n _context text := lower(context(conf));\n _stats_ttl interval := context_stats_ttl(conf);\n _estimated_cost float := context_estimated_cost(conf);\n _estimated_count int := context_estimated_count(conf);\nBEGIN\n IF _context = 'off' THEN\n sw._where := inwhere;\n return sw;\n END IF;\n\n SELECT * INTO sw FROM search_wheres WHERE md5(_where)=inwhere_hash FOR UPDATE;\n\n -- Update statistics if explicitly set, if statistics do not exist, or statistics ttl has expired\n IF NOT updatestats THEN\n RAISE NOTICE 'Checking if update is needed for: % .', inwhere;\n RAISE NOTICE 'Stats Last Updated: %', sw.statslastupdated;\n RAISE NOTICE 'TTL: %, Age: %', _stats_ttl, now() - sw.statslastupdated;\n RAISE NOTICE 'Context: %, Existing Total: %', _context, sw.total_count;\n IF\n sw.statslastupdated IS NULL\n OR (now() - sw.statslastupdated) > _stats_ttl\n OR (context(conf) != 'off' AND sw.total_count IS NULL)\n THEN\n updatestats := TRUE;\n END IF;\n END IF;\n\n sw._where := inwhere;\n sw.lastused := now();\n sw.usecount := coalesce(sw.usecount,0) + 1;\n\n IF NOT updatestats THEN\n UPDATE search_wheres SET\n lastused = sw.lastused,\n usecount = sw.usecount\n WHERE md5(_where) = inwhere_hash\n RETURNING * INTO sw\n ;\n RETURN sw;\n END IF;\n\n -- Use explain to get estimated count/cost and a list of the partitions that would be hit by the query\n t := clock_timestamp();\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s', inwhere)\n INTO explain_json;\n RAISE NOTICE 'Time for just the explain: %', clock_timestamp() - t;\n i := clock_timestamp() - t;\n\n sw.statslastupdated := now();\n sw.estimated_count := explain_json->0->'Plan'->'Plan Rows';\n sw.estimated_cost := explain_json->0->'Plan'->'Total Cost';\n sw.time_to_estimate := extract(epoch from i);\n\n RAISE NOTICE 'ESTIMATED_COUNT: % < %', sw.estimated_count, _estimated_count;\n RAISE NOTICE 'ESTIMATED_COST: % < %', sw.estimated_cost, _estimated_cost;\n\n -- Do a full count of rows if context is set to on or if auto is set and estimates are low enough\n IF\n _context = 'on'\n OR\n ( _context = 'auto' AND\n (\n sw.estimated_count < _estimated_count\n AND\n sw.estimated_cost < _estimated_cost\n )\n )\n THEN\n t := clock_timestamp();\n RAISE NOTICE 'Calculating actual count...';\n EXECUTE format(\n 'SELECT count(*) FROM items WHERE %s',\n inwhere\n ) INTO sw.total_count;\n i := clock_timestamp() - t;\n RAISE NOTICE 'Actual Count: % -- %', sw.total_count, i;\n sw.time_to_count := extract(epoch FROM i);\n ELSE\n sw.total_count := NULL;\n sw.time_to_count := NULL;\n END IF;\n\n\n INSERT INTO search_wheres\n (_where, lastused, usecount, statslastupdated, estimated_count, estimated_cost, time_to_estimate, partitions, total_count, time_to_count)\n SELECT sw._where, sw.lastused, sw.usecount, sw.statslastupdated, sw.estimated_count, sw.estimated_cost, sw.time_to_estimate, sw.partitions, sw.total_count, sw.time_to_count\n ON CONFLICT ((md5(_where)))\n DO UPDATE\n SET\n lastused = sw.lastused,\n usecount = sw.usecount,\n statslastupdated = sw.statslastupdated,\n estimated_count = sw.estimated_count,\n estimated_cost = sw.estimated_cost,\n time_to_estimate = sw.time_to_estimate,\n total_count = sw.total_count,\n time_to_count = sw.time_to_count\n ;\n RETURN sw;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION search_query(\n _search jsonb = '{}'::jsonb,\n updatestats boolean = false,\n _metadata jsonb = '{}'::jsonb\n) RETURNS searches AS $$\nDECLARE\n search searches%ROWTYPE;\n pexplain jsonb;\n t timestamptz;\n i interval;\nBEGIN\n SELECT * INTO search FROM searches\n WHERE hash=search_hash(_search, _metadata) FOR UPDATE;\n\n -- Calculate the where clause if not already calculated\n IF search._where IS NULL THEN\n search._where := stac_search_to_where(_search);\n END IF;\n\n -- Calculate the order by clause if not already calculated\n IF search.orderby IS NULL THEN\n search.orderby := sort_sqlorderby(_search);\n END IF;\n\n PERFORM where_stats(search._where, updatestats, _search->'conf');\n\n search.lastused := now();\n search.usecount := coalesce(search.usecount, 0) + 1;\n INSERT INTO searches (search, _where, orderby, lastused, usecount, metadata)\n VALUES (_search, search._where, search.orderby, search.lastused, search.usecount, _metadata)\n ON CONFLICT (hash) DO\n UPDATE SET\n _where = EXCLUDED._where,\n orderby = EXCLUDED.orderby,\n lastused = EXCLUDED.lastused,\n usecount = EXCLUDED.usecount,\n metadata = EXCLUDED.metadata\n RETURNING * INTO search\n ;\n RETURN search;\n\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE OR REPLACE FUNCTION search(_search jsonb = '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n searches searches%ROWTYPE;\n _where text;\n token_where text;\n full_where text;\n orderby text;\n query text;\n token_type text := substr(_search->>'token',1,4);\n _limit int := coalesce((_search->>'limit')::int, 10);\n curs refcursor;\n cntr int := 0;\n iter_record items%ROWTYPE;\n first_record jsonb;\n first_item items%ROWTYPE;\n last_item items%ROWTYPE;\n last_record jsonb;\n out_records jsonb := '[]'::jsonb;\n prev_query text;\n next text;\n prev_id text;\n has_next boolean := false;\n has_prev boolean := false;\n prev text;\n total_count bigint;\n context jsonb;\n collection jsonb;\n includes text[];\n excludes text[];\n exit_flag boolean := FALSE;\n batches int := 0;\n timer timestamptz := clock_timestamp();\n pstart timestamptz;\n pend timestamptz;\n pcurs refcursor;\n search_where search_wheres%ROWTYPE;\n id text;\nBEGIN\nCREATE TEMP TABLE results (i int GENERATED ALWAYS AS IDENTITY, content jsonb) ON COMMIT DROP;\n searches := search_query(_search);\n _where := searches._where;\n orderby := searches.orderby;\n search_where := where_stats(_where);\n total_count := coalesce(search_where.total_count, search_where.estimated_count);\n\n IF token_type='prev' THEN\n token_where := get_token_filter(_search, null::jsonb);\n orderby := sort_sqlorderby(_search, TRUE);\n END IF;\n IF token_type='next' THEN\n token_where := get_token_filter(_search, null::jsonb);\n END IF;\n\n full_where := concat_ws(' AND ', _where, token_where);\n RAISE NOTICE 'FULL QUERY % %', full_where, clock_timestamp()-timer;\n timer := clock_timestamp();\n\n FOR query IN SELECT partition_queries(full_where, orderby, search_where.partitions) LOOP\n timer := clock_timestamp();\n query := format('%s LIMIT %s', query, _limit + 1);\n RAISE NOTICE 'Partition Query: %', query;\n batches := batches + 1;\n -- curs = create_cursor(query);\n RAISE NOTICE 'cursor_tuple_fraction: %', current_setting('cursor_tuple_fraction');\n OPEN curs FOR EXECUTE query;\n LOOP\n FETCH curs into iter_record;\n EXIT WHEN NOT FOUND;\n cntr := cntr + 1;\n\n IF _search->'conf'->>'nohydrate' IS NOT NULL AND (_search->'conf'->>'nohydrate')::boolean = true THEN\n last_record := content_nonhydrated(iter_record, _search->'fields');\n ELSE\n last_record := content_hydrate(iter_record, _search->'fields');\n END IF;\n last_item := iter_record;\n IF cntr = 1 THEN\n first_item := last_item;\n first_record := last_record;\n END IF;\n IF cntr <= _limit THEN\n INSERT INTO results (content) VALUES (last_record);\n ELSIF cntr > _limit THEN\n has_next := true;\n exit_flag := true;\n EXIT;\n END IF;\n END LOOP;\n CLOSE curs;\n RAISE NOTICE 'Query took %.', clock_timestamp()-timer;\n timer := clock_timestamp();\n EXIT WHEN exit_flag;\n END LOOP;\n RAISE NOTICE 'Scanned through % partitions.', batches;\n\nWITH ordered AS (SELECT * FROM results WHERE content IS NOT NULL ORDER BY i)\nSELECT jsonb_agg(content) INTO out_records FROM ordered;\n\nDROP TABLE results;\n\n\n-- Flip things around if this was the result of a prev token query\nIF token_type='prev' THEN\n out_records := flip_jsonb_array(out_records);\n first_item := last_item;\n first_record := last_record;\nEND IF;\n\n-- If this query has a token, see if there is data before the first record\nIF _search ? 'token' THEN\n prev_query := format(\n 'SELECT 1 FROM items WHERE %s LIMIT 1',\n concat_ws(\n ' AND ',\n _where,\n trim(get_token_filter(_search, to_jsonb(first_item)))\n )\n );\n RAISE NOTICE 'Query to get previous record: % --- %', prev_query, first_record;\n EXECUTE prev_query INTO has_prev;\n IF FOUND and has_prev IS NOT NULL THEN\n RAISE NOTICE 'Query results from prev query: %', has_prev;\n has_prev := TRUE;\n END IF;\nEND IF;\nhas_prev := COALESCE(has_prev, FALSE);\n\nIF has_prev THEN\n prev := out_records->0->>'id';\nEND IF;\nIF has_next OR token_type='prev' THEN\n next := out_records->-1->>'id';\nEND IF;\n\nIF context(_search->'conf') != 'off' THEN\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'matched', total_count,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\nELSE\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\nEND IF;\n\ncollection := jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb),\n 'next', next,\n 'prev', prev,\n 'context', context\n);\n\nRETURN collection;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac, public SET cursor_tuple_fraction TO 1;\n\n\nCREATE OR REPLACE FUNCTION search_cursor(_search jsonb = '{}'::jsonb) RETURNS refcursor AS $$\nDECLARE\n curs refcursor;\n searches searches%ROWTYPE;\n _where text;\n _orderby text;\n q text;\n\nBEGIN\n searches := search_query(_search);\n _where := searches._where;\n _orderby := searches.orderby;\n\n OPEN curs FOR\n WITH p AS (\n SELECT * FROM partition_queries(_where, _orderby) p\n )\n SELECT\n CASE WHEN EXISTS (SELECT 1 FROM p) THEN\n (SELECT format($q$\n SELECT * FROM (\n %s\n ) total\n $q$,\n string_agg(\n format($q$ SELECT * FROM ( %s ) AS sub $q$, p),\n '\n UNION ALL\n '\n )\n ))\n ELSE NULL\n END FROM p;\n RETURN curs;\nEND;\n$$ LANGUAGE PLPGSQL;\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION tileenvelope(zoom int, x int, y int) RETURNS geometry AS $$\nWITH t AS (\n SELECT\n 20037508.3427892 as merc_max,\n -20037508.3427892 as merc_min,\n (2 * 20037508.3427892) / (2 ^ zoom) as tile_size\n)\nSELECT st_makeenvelope(\n merc_min + (tile_size * x),\n merc_max - (tile_size * (y + 1)),\n merc_min + (tile_size * (x + 1)),\n merc_max - (tile_size * y),\n 3857\n) FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;DROP FUNCTION IF EXISTS mercgrid;\n\n\nCREATE OR REPLACE FUNCTION ftime() RETURNS interval as $$\nSELECT age(clock_timestamp(), transaction_timestamp());\n$$ LANGUAGE SQL;\nSET SEARCH_PATH to pgstac, public;\n\nDROP FUNCTION IF EXISTS geometrysearch;\nCREATE OR REPLACE FUNCTION geometrysearch(\n IN geom geometry,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE, -- Return as soon as the passed in geometry is full covered\n IN skipcovered boolean DEFAULT TRUE -- Skip any items that would show up completely under the previous items\n) RETURNS jsonb AS $$\nDECLARE\n search searches%ROWTYPE;\n curs refcursor;\n _where text;\n query text;\n iter_record items%ROWTYPE;\n out_records jsonb := '{}'::jsonb[];\n exit_flag boolean := FALSE;\n counter int := 1;\n scancounter int := 1;\n remaining_limit int := _scanlimit;\n tilearea float;\n unionedgeom geometry;\n clippedgeom geometry;\n unionedgeom_area float := 0;\n prev_area float := 0;\n excludes text[];\n includes text[];\n\nBEGIN\n DROP TABLE IF EXISTS pgstac_results;\n CREATE TEMP TABLE pgstac_results (content jsonb) ON COMMIT DROP;\n\n -- If skipcovered is true then you will always want to exit when the passed in geometry is full\n IF skipcovered THEN\n exitwhenfull := TRUE;\n END IF;\n\n SELECT * INTO search FROM searches WHERE hash=queryhash;\n\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Search with Query Hash % Not Found', queryhash;\n END IF;\n\n tilearea := st_area(geom);\n _where := format('%s AND st_intersects(geometry, %L::geometry)', search._where, geom);\n\n\n FOR query IN SELECT * FROM partition_queries(_where, search.orderby) LOOP\n query := format('%s LIMIT %L', query, remaining_limit);\n RAISE NOTICE '%', query;\n OPEN curs FOR EXECUTE query;\n LOOP\n FETCH curs INTO iter_record;\n EXIT WHEN NOT FOUND;\n IF exitwhenfull OR skipcovered THEN -- If we are not using exitwhenfull or skipcovered, we do not need to do expensive geometry operations\n clippedgeom := st_intersection(geom, iter_record.geometry);\n\n IF unionedgeom IS NULL THEN\n unionedgeom := clippedgeom;\n ELSE\n unionedgeom := st_union(unionedgeom, clippedgeom);\n END IF;\n\n unionedgeom_area := st_area(unionedgeom);\n\n IF skipcovered AND prev_area = unionedgeom_area THEN\n scancounter := scancounter + 1;\n CONTINUE;\n END IF;\n\n prev_area := unionedgeom_area;\n\n RAISE NOTICE '% % % %', unionedgeom_area/tilearea, counter, scancounter, ftime();\n END IF;\n RAISE NOTICE '% %', iter_record, content_hydrate(iter_record, fields);\n INSERT INTO pgstac_results (content) VALUES (content_hydrate(iter_record, fields));\n\n IF counter >= _limit\n OR scancounter > _scanlimit\n OR ftime() > _timelimit\n OR (exitwhenfull AND unionedgeom_area >= tilearea)\n THEN\n exit_flag := TRUE;\n EXIT;\n END IF;\n counter := counter + 1;\n scancounter := scancounter + 1;\n\n END LOOP;\n CLOSE curs;\n EXIT WHEN exit_flag;\n remaining_limit := _scanlimit - scancounter;\n END LOOP;\n\n SELECT jsonb_agg(content) INTO out_records FROM pgstac_results WHERE content IS NOT NULL;\n\n RETURN jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb)\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDROP FUNCTION IF EXISTS geojsonsearch;\nCREATE OR REPLACE FUNCTION geojsonsearch(\n IN geojson jsonb,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_geomfromgeojson(geojson),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS xyzsearch;\nCREATE OR REPLACE FUNCTION xyzsearch(\n IN _x int,\n IN _y int,\n IN _z int,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_transform(tileenvelope(_z, _x, _y), 4326),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE PROCEDURE analyze_items() AS $$\nDECLARE\n q text;\n timeout_ts timestamptz;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n SELECT format('ANALYZE (VERBOSE, SKIP_LOCKED) %I;', relname) INTO q\n FROM pg_stat_user_tables\n WHERE relname like '_item%' AND (n_mod_since_analyze>0 OR last_analyze IS NULL) LIMIT 1;\n IF NOT FOUND THEN\n EXIT;\n END IF;\n RAISE NOTICE '%', q;\n EXECUTE q;\n COMMIT;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE PROCEDURE validate_constraints() AS $$\nDECLARE\n q text;\nBEGIN\n FOR q IN\n SELECT\n FORMAT(\n 'ALTER TABLE %I.%I VALIDATE CONSTRAINT %I;',\n nsp.nspname,\n cls.relname,\n con.conname\n )\n\n FROM pg_constraint AS con\n JOIN pg_class AS cls\n ON con.conrelid = cls.oid\n JOIN pg_namespace AS nsp\n ON cls.relnamespace = nsp.oid\n WHERE convalidated = FALSE AND contype in ('c','f')\n AND nsp.nspname = 'pgstac'\n LOOP\n RAISE NOTICE '%', q;\n PERFORM run_or_queue(q);\n COMMIT;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION collection_extent(_collection text, runupdate boolean default false) RETURNS jsonb AS $$\nDECLARE\n geom_extent geometry;\n mind timestamptz;\n maxd timestamptz;\n extent jsonb;\nBEGIN\n IF runupdate THEN\n PERFORM update_partition_stats_q(partition)\n FROM partitions WHERE collection=_collection;\n END IF;\n SELECT\n min(lower(dtrange)),\n max(upper(edtrange)),\n st_extent(spatial)\n INTO\n mind,\n maxd,\n geom_extent\n FROM partitions\n WHERE collection=_collection;\n\n IF geom_extent IS NOT NULL AND mind IS NOT NULL AND maxd IS NOT NULL THEN\n extent := jsonb_build_object(\n 'extent', jsonb_build_object(\n 'spatial', jsonb_build_object(\n 'bbox', to_jsonb(array[array[st_xmin(geom_extent), st_ymin(geom_extent), st_xmax(geom_extent), st_ymax(geom_extent)]])\n ),\n 'temporal', jsonb_build_object(\n 'interval', to_jsonb(array[array[mind, maxd]])\n )\n )\n );\n RETURN extent;\n END IF;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('id', '{\"title\": \"Item ID\",\"description\": \"Item identifier\",\"$ref\": \"https://schemas.stacspec.org/v1.0.0/item-spec/json-schema/item.json#/definitions/core/allOf/2/properties/id\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('geometry', '{\"title\": \"Item Geometry\",\"description\": \"Item Geometry\",\"$ref\": \"https://geojson.org/schema/Feature.json\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('datetime','{\"description\": \"Datetime\",\"type\": \"string\",\"title\": \"Acquired\",\"format\": \"date-time\",\"pattern\": \"(\\\\+00:00|Z)$\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('eo:cloud_cover','{\"$ref\": \"https://stac-extensions.github.io/eo/v1.0.0/schema.json#/definitions/fieldsproperties/eo:cloud_cover\"}','to_int','BTREE');\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDELETE FROM queryables a USING queryables b\n WHERE a.name = b.name AND a.collection_ids IS NOT DISTINCT FROM b.collection_ids AND a.id > b.id;\n\n\nINSERT INTO pgstac_settings (name, value) VALUES\n ('context', 'off'),\n ('context_estimated_count', '100000'),\n ('context_estimated_cost', '100000'),\n ('context_stats_ttl', '1 day'),\n ('default_filter_lang', 'cql2-json'),\n ('additional_properties', 'true'),\n ('use_queue', 'false'),\n ('queue_timeout', '10 minutes'),\n ('update_collection_extent', 'false')\nON CONFLICT DO NOTHING\n;\n\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nGRANT ALL ON SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac to pgstac_admin;\n\n-- pgstac_read role limited to using function apis\nGRANT EXECUTE ON FUNCTION search TO pgstac_read;\nGRANT EXECUTE ON FUNCTION search_query TO pgstac_read;\nGRANT EXECUTE ON FUNCTION item_by_id TO pgstac_read;\nGRANT EXECUTE ON FUNCTION get_item TO pgstac_read;\nGRANT SELECT ON ALL TABLES IN SCHEMA pgstac TO pgstac_read;\n\n\nGRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_ingest;\nGRANT USAGE ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_ingest;\n\nSELECT update_partition_stats_q(partition) FROM partitions;\nSELECT set_version('0.7.2');\n" }, { "path": "src/pgstac/migrations/pgstac.0.7.3-0.7.4.sql", "content": "RESET ROLE;\nDO $$\nDECLARE\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='postgis') THEN\n CREATE EXTENSION IF NOT EXISTS postgis;\n END IF;\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='btree_gist') THEN\n CREATE EXTENSION IF NOT EXISTS btree_gist;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_admin;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_read;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_ingest;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n\nGRANT pgstac_admin TO current_user;\n\n-- Function to make sure pgstac_admin is the owner of items\nCREATE OR REPLACE FUNCTION pgstac_admin_owns() RETURNS VOID AS $$\nDECLARE\n f RECORD;\nBEGIN\n FOR f IN (\n SELECT\n concat(\n oid::regproc::text,\n '(',\n coalesce(pg_get_function_identity_arguments(oid),''),\n ')'\n ) AS name,\n CASE prokind WHEN 'f' THEN 'FUNCTION' WHEN 'p' THEN 'PROCEDURE' WHEN 'a' THEN 'AGGREGATE' END as typ\n FROM pg_proc\n WHERE\n pronamespace=to_regnamespace('pgstac')\n AND proowner != to_regrole('pgstac_admin')\n AND proname NOT LIKE 'pg_stat%'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n FOR f IN (\n SELECT\n oid::regclass::text as name,\n CASE relkind\n WHEN 'i' THEN 'INDEX'\n WHEN 'I' THEN 'INDEX'\n WHEN 'p' THEN 'TABLE'\n WHEN 'r' THEN 'TABLE'\n WHEN 'v' THEN 'VIEW'\n WHEN 'S' THEN 'SEQUENCE'\n ELSE NULL\n END as typ\n FROM pg_class\n WHERE relnamespace=to_regnamespace('pgstac') and relowner != to_regrole('pgstac_admin') AND relkind IN ('r','p','v','S') AND relname NOT LIKE 'pg_stat'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\nSELECT pgstac_admin_owns();\n\nCREATE SCHEMA IF NOT EXISTS pgstac AUTHORIZATION pgstac_admin;\n\nGRANT ALL ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_admin;\n\nALTER ROLE pgstac_admin SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_read SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_ingest SET SEARCH_PATH TO pgstac, public;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\n\nGRANT pgstac_read TO pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\n\nSET ROLE pgstac_admin;\n\nSET SEARCH_PATH TO pgstac, public;\n\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS analyze_items;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS validate_constraints;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\nSET client_min_messages TO WARNING;\nSET SEARCH_PATH to pgstac, public;\n-- BEGIN migra calculated SQL\ndrop function if exists \"pgstac\".\"get_token_filter\"(_search jsonb, token_rec jsonb);\n\ncreate unlogged table \"pgstac\".\"format_item_cache\" (\n \"id\" text not null,\n \"collection\" text not null,\n \"fields\" text not null,\n \"hydrated\" boolean not null,\n \"output\" jsonb,\n \"lastused\" timestamp with time zone default now(),\n \"usecount\" integer default 1,\n \"timetoformat\" double precision\n);\n\n\nCREATE INDEX format_item_cache_lastused_idx ON pgstac.format_item_cache USING btree (lastused);\n\nCREATE UNIQUE INDEX format_item_cache_pkey ON pgstac.format_item_cache USING btree (collection, id, fields, hydrated);\n\nalter table \"pgstac\".\"format_item_cache\" add constraint \"format_item_cache_pkey\" PRIMARY KEY using index \"format_item_cache_pkey\";\n\nset check_function_bodies = off;\n\nCREATE OR REPLACE FUNCTION pgstac.age_ms(a timestamp with time zone, b timestamp with time zone DEFAULT clock_timestamp())\n RETURNS double precision\n LANGUAGE sql\n IMMUTABLE PARALLEL SAFE\nAS $function$\n SELECT abs(extract(epoch from age(a,b)) * 1000);\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.format_item(_item pgstac.items, _fields jsonb DEFAULT '{}'::jsonb, _hydrated boolean DEFAULT true)\n RETURNS jsonb\n LANGUAGE plpgsql\n SECURITY DEFINER\nAS $function$\nDECLARE\n cache bool := get_setting_bool('format_cache');\n _output jsonb := null;\n t timestamptz := clock_timestamp();\nBEGIN\n IF cache THEN\n SELECT output INTO _output FROM format_item_cache\n WHERE id=_item.id AND collection=_item.collection AND fields=_fields::text AND hydrated=_hydrated;\n END IF;\n IF _output IS NULL THEN\n IF _hydrated THEN\n _output := content_hydrate(_item, _fields);\n ELSE\n _output := content_nonhydrated(_item, _fields);\n END IF;\n END IF;\n IF cache THEN\n INSERT INTO format_item_cache (id, collection, fields, hydrated, output, timetoformat)\n VALUES (_item.id, _item.collection, _fields::text, _hydrated, _output, age_ms(t))\n ON CONFLICT(collection, id, fields, hydrated) DO\n UPDATE\n SET lastused=now(), usecount = format_item_cache.usecount + 1\n ;\n END IF;\n RETURN _output;\n\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.get_token_filter(_sortby jsonb DEFAULT '[{\"field\": \"datetime\", \"direction\": \"desc\"}]'::jsonb, token_item pgstac.items DEFAULT NULL::pgstac.items, prev boolean DEFAULT false, inclusive boolean DEFAULT false)\n RETURNS text\n LANGUAGE plpgsql\n SET transform_null_equals TO 'true'\nAS $function$\nDECLARE\n ltop text := '<';\n gtop text := '>';\n dir text;\n sort record;\n orfilter text := '';\n orfilters text[] := '{}'::text[];\n andfilters text[] := '{}'::text[];\n output text;\n token_where text;\nBEGIN\n _sortby := _sortby || jsonb_build_object('field','id','direction',_sortby->0->>'direction');\n RAISE NOTICE 'Getting Token Filter. % %', _sortby, token_item;\n IF inclusive THEN\n orfilters := orfilters || format('( id=%L AND collection=%L )' , token_item.id, token_item.collection);\n END IF;\n\n FOR sort IN\n WITH s1 AS (\n SELECT\n _row,\n (queryable(value->>'field')).expression as _field,\n (value->>'field' = 'id') as _isid,\n get_sort_dir(value) as _dir\n FROM jsonb_array_elements(_sortby)\n WITH ORDINALITY AS t(value, _row)\n )\n SELECT\n _row,\n _field,\n _dir,\n get_token_val_str(_field, token_item) as _val\n FROM s1\n WHERE _row <= (SELECT min(_row) FROM s1 WHERE _isid)\n LOOP\n orfilter := NULL;\n RAISE NOTICE 'SORT: %', sort;\n IF sort._val IS NOT NULL AND ((prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC')) THEN\n orfilter := format($f$(\n (%s %s %s) OR (%s IS NULL)\n )$f$,\n sort._field,\n ltop,\n sort._val,\n sort._val\n );\n ELSIF sort._val IS NULL AND ((prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC')) THEN\n RAISE NOTICE '< but null';\n orfilter := format('%s IS NOT NULL', sort._field);\n ELSIF sort._val IS NULL THEN\n RAISE NOTICE '> but null';\n ELSE\n orfilter := format($f$(\n (%s %s %s) OR (%s IS NULL)\n )$f$,\n sort._field,\n gtop,\n sort._val,\n sort._field\n );\n END IF;\n RAISE NOTICE 'ORFILTER: %', orfilter;\n\n IF orfilter IS NOT NULL THEN\n IF sort._row = 1 THEN\n orfilters := orfilters || orfilter;\n ELSE\n orfilters := orfilters || format('(%s AND %s)', array_to_string(andfilters, ' AND '), orfilter);\n END IF;\n END IF;\n IF sort._val IS NOT NULL THEN\n andfilters := andfilters || format('%s = %s', sort._field, sort._val);\n ELSE\n andfilters := andfilters || format('%s IS NULL', sort._field);\n END IF;\n END LOOP;\n\n output := array_to_string(orfilters, ' OR ');\n\n token_where := concat('(',coalesce(output,'true'),')');\n IF trim(token_where) = '' THEN\n token_where := NULL;\n END IF;\n RAISE NOTICE 'TOKEN_WHERE: %',token_where;\n RETURN token_where;\n END;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.get_token_record(_token text, OUT prev boolean, OUT item pgstac.items)\n RETURNS record\n LANGUAGE plpgsql\n STABLE STRICT\nAS $function$\nDECLARE\n _itemid text := _token;\n _collectionid text;\nBEGIN\n IF _token IS NULL THEN\n RETURN;\n END IF;\n RAISE NOTICE 'Looking for token: %', _token;\n prev := FALSE;\n IF _token ILIKE 'prev:%' THEN\n _itemid := replace(_token, 'prev:','');\n prev := TRUE;\n ELSIF _token ILIKE 'next:%' THEN\n _itemid := replace(_token, 'next:', '');\n END IF;\n SELECT id INTO _collectionid FROM collections WHERE _itemid LIKE concat(id,':%');\n IF FOUND THEN\n _itemid := replace(_itemid, concat(_collectionid,':'), '');\n SELECT * INTO item FROM items WHERE id=_itemid AND collection=_collectionid;\n ELSE\n SELECT * INTO item FROM items WHERE id=_itemid;\n END IF;\n IF item IS NULL THEN\n RAISE EXCEPTION 'Could not find item using token: % item: % collection: %', _token, _itemid, _collectionid;\n END IF;\n RETURN;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.search_rows(_where text DEFAULT 'TRUE'::text, _orderby text DEFAULT 'datetime DESC, id DESC'::text, partitions text[] DEFAULT NULL::text[], _limit integer DEFAULT 10)\n RETURNS SETOF pgstac.items\n LANGUAGE plpgsql\n SET search_path TO 'pgstac', 'public'\nAS $function$\nDECLARE\n base_query text;\n query text;\n sdate timestamptz;\n edate timestamptz;\n n int;\n records_left int := _limit;\nBEGIN\nIF _where IS NULL OR trim(_where) = '' THEN\n _where = ' TRUE ';\nEND IF;\nRAISE NOTICE 'Getting chunks for % %', _where, _orderby;\n\nbase_query := $q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n LIMIT %L\n$q$;\n\nIF _orderby ILIKE 'datetime d%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 DESC LOOP\n RAISE NOTICE 'Running Query for % to %', sdate, edate;\n query := format(\n base_query,\n sdate,\n edate,\n _where,\n _orderby,\n records_left\n );\n RAISE LOG 'QUERY: %', query;\n RETURN QUERY EXECUTE query;\n\n GET DIAGNOSTICS n = ROW_COUNT;\n records_left := records_left - n;\n RAISE NOTICE 'Returned %/% Rows From % to %. % to go.', n, _limit, sdate, edate, records_left;\n IF records_left <= 0 THEN\n RETURN;\n END IF;\n END LOOP;\nELSIF _orderby ILIKE 'datetime a%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 ASC LOOP\n RAISE NOTICE 'Running Query for % to %', sdate, edate;\n query := format(\n base_query,\n sdate,\n edate,\n _where,\n _orderby,\n records_left\n );\n RAISE LOG 'QUERY: %', query;\n RETURN QUERY EXECUTE query;\n\n GET DIAGNOSTICS n = ROW_COUNT;\n records_left := records_left - n;\n RAISE NOTICE 'Returned %/% Rows From % to %. % to go.', n, _limit, sdate, edate, records_left;\n IF records_left <= 0 THEN\n RETURN;\n END IF;\n END LOOP;\nELSE\n query := format($q$\n SELECT * FROM items\n WHERE %s\n ORDER BY %s\n LIMIT %L\n $q$, _where, _orderby, _limit\n );\n RAISE LOG 'QUERY: %', query;\n RETURN QUERY EXECUTE query;\nEND IF;\nRETURN;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.get_token_val_str(_field text, _item pgstac.items)\n RETURNS text\n LANGUAGE plpgsql\nAS $function$\nDECLARE\n q text;\n literal text;\nBEGIN\n q := format($q$ SELECT quote_literal(%s) FROM (SELECT $1.*) as r;$q$, _field);\n EXECUTE q INTO literal USING _item;\n RETURN literal;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.partition_after_triggerfunc()\n RETURNS trigger\n LANGUAGE plpgsql\n SECURITY DEFINER\nAS $function$\nDECLARE\n p text;\n t timestamptz := clock_timestamp();\nBEGIN\n RAISE NOTICE 'Updating partition stats %', t;\n FOR p IN SELECT DISTINCT partition\n FROM newdata n JOIN partition_sys_meta p\n ON (n.collection=p.collection AND n.datetime <@ p.partition_dtrange)\n LOOP\n PERFORM run_or_queue(format('SELECT update_partition_stats(%L, %L);', p, true));\n END LOOP;\n IF TG_OP IN ('DELETE','UPDATE') THEN\n DELETE FROM format_item_cache c USING newdata n WHERE c.collection = n.collection AND c.id = n.id;\n END IF;\n RAISE NOTICE 't: % %', t, clock_timestamp() - t;\n RETURN NULL;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.queryables_constraint_triggerfunc()\n RETURNS trigger\n LANGUAGE plpgsql\nAS $function$\nDECLARE\n allcollections text[];\nBEGIN\n RAISE NOTICE 'Making sure that name/collection is unique for queryables %', NEW;\n IF NEW.collection_ids IS NOT NULL THEN\n IF EXISTS (\n SELECT 1 FROM\n collections\n LEFT JOIN\n unnest(NEW.collection_ids) c\n ON (collections.id = c)\n WHERE c IS NULL\n ) THEN\n RAISE foreign_key_violation USING MESSAGE = format(\n 'One or more collections in %s do not exist.', NEW.collection_ids\n );\n RETURN NULL;\n END IF;\n END IF;\n IF TG_OP = 'INSERT' THEN\n IF EXISTS (\n SELECT 1 FROM queryables q\n WHERE\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n )\n ) THEN\n RAISE unique_violation USING MESSAGE = format(\n 'There is already a queryable for %s for a collection in %s: %s',\n NEW.name,\n NEW.collection_ids,\n\t\t\t\t(SELECT json_agg(row_to_json(q)) FROM queryables q WHERE\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n ))\n );\n RETURN NULL;\n END IF;\n END IF;\n IF TG_OP = 'UPDATE' THEN\n IF EXISTS (\n SELECT 1 FROM queryables q\n WHERE\n q.id != NEW.id\n AND\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n )\n ) THEN\n RAISE unique_violation\n USING MESSAGE = format(\n 'There is already a queryable for %s for a collection in %s',\n NEW.name,\n NEW.collection_ids\n );\n RETURN NULL;\n END IF;\n END IF;\n\n RETURN NEW;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.search(_search jsonb DEFAULT '{}'::jsonb)\n RETURNS jsonb\n LANGUAGE plpgsql\nAS $function$\nDECLARE\n searches searches%ROWTYPE;\n _where text;\n orderby text;\n search_where search_wheres%ROWTYPE;\n total_count bigint;\n token record;\n token_prev boolean;\n token_item items%ROWTYPE;\n token_where text;\n full_where text;\n timer timestamptz := clock_timestamp();\n hydrate bool := NOT (_search->'conf'->>'nohydrate' IS NOT NULL AND (_search->'conf'->>'nohydrate')::boolean = true);\n prev text;\n next text;\n context jsonb;\n collection jsonb;\n out_records jsonb;\n out_len int;\n _limit int := coalesce((_search->>'limit')::int, 10);\n _querylimit int;\n _fields jsonb := coalesce(_search->'fields', '{}'::jsonb);\n has_prev boolean := FALSE;\n has_next boolean := FALSE;\nBEGIN\n searches := search_query(_search);\n _where := searches._where;\n orderby := searches.orderby;\n search_where := where_stats(_where);\n total_count := coalesce(search_where.total_count, search_where.estimated_count);\n RAISE NOTICE 'SEARCH:TOKEN: %', _search->>'token';\n token := get_token_record(_search->>'token');\n RAISE NOTICE '***TOKEN: %', token;\n _querylimit := _limit + 1;\n IF token IS NOT NULL THEN\n token_prev := token.prev;\n token_item := token.item;\n token_where := get_token_filter(_search->'sortby', token_item, token_prev, FALSE);\n RAISE LOG 'TOKEN_WHERE: %', token_where;\n IF token_prev THEN -- if we are using a prev token, we know has_next is true\n RAISE LOG 'There is a previous token, so automatically setting has_next to true';\n has_next := TRUE;\n orderby := sort_sqlorderby(_search, TRUE);\n ELSE\n RAISE LOG 'There is a next token, so automatically setting has_prev to true';\n has_prev := TRUE;\n\n END IF;\n ELSE -- if there was no token, we know there is no prev\n RAISE LOG 'There is no token, so we know there is no prev. setting has_prev to false';\n has_prev := FALSE;\n END IF;\n\n full_where := concat_ws(' AND ', _where, token_where);\n RAISE NOTICE 'FULL WHERE CLAUSE: %', full_where;\n RAISE NOTICE 'Time to get counts and build query %', age_ms(timer);\n timer := clock_timestamp();\n\n IF hydrate THEN\n RAISE NOTICE 'Getting hydrated data.';\n ELSE\n RAISE NOTICE 'Getting non-hydrated data.';\n END IF;\n RAISE NOTICE 'CACHE SET TO %', get_setting_bool('format_cache');\n SELECT jsonb_agg(format_item(i, _fields, hydrate)) INTO out_records\n FROM search_rows(\n full_where,\n orderby,\n search_where.partitions,\n _querylimit\n ) as i;\n\n RAISE NOTICE 'Time to fetch rows %', age_ms(timer);\n timer := clock_timestamp();\n\n\n IF token_prev THEN\n out_records := flip_jsonb_array(out_records);\n END IF;\n\n RAISE NOTICE 'Query returned % records.', jsonb_array_length(out_records);\n RAISE LOG 'TOKEN: % %', token_item.id, token_item.collection;\n RAISE LOG 'RECORD_1: % %', out_records->0->>'id', out_records->0->>'collection';\n RAISE LOG 'RECORD-1: % %', out_records->-1->>'id', out_records->-1->>'collection';\n\n -- REMOVE records that were from our token\n IF out_records->0->>'id' = token_item.id AND out_records->0->>'collection' = token_item.collection THEN\n out_records := out_records - 0;\n ELSIF out_records->-1->>'id' = token_item.id AND out_records->-1->>'collection' = token_item.collection THEN\n out_records := out_records - -1;\n END IF;\n\n out_len := jsonb_array_length(out_records);\n\n IF out_len = _limit + 1 THEN\n IF token_prev THEN\n has_prev := TRUE;\n out_records := out_records - 0;\n ELSE\n has_next := TRUE;\n out_records := out_records - -1;\n END IF;\n END IF;\n\n IF has_next THEN\n next := concat(out_records->-1->>'collection', ':', out_records->-1->>'id');\n RAISE NOTICE 'HAS NEXT | %', next;\n END IF;\n\n IF has_prev THEN\n prev := concat(out_records->0->>'collection', ':', out_records->0->>'id');\n RAISE NOTICE 'HAS PREV | %', prev;\n END IF;\n\n RAISE NOTICE 'Time to get prev/next %', age_ms(timer);\n timer := clock_timestamp();\n\n IF context(_search->'conf') != 'off' THEN\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'matched', total_count,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\n ELSE\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\n END IF;\n\n collection := jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb),\n 'next', next,\n 'prev', prev,\n 'context', context\n );\n\n RAISE NOTICE 'Time to build final json %', age_ms(timer);\n timer := clock_timestamp();\n\n RAISE NOTICE 'Total Time: %', age_ms(current_timestamp);\n RAISE NOTICE 'RETURNING % records. NEXT: %. PREV: %', collection->'context'->>'returned', collection->>'next', collection->>'prev';\n RETURN collection;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.sort_sqlorderby(_search jsonb DEFAULT NULL::jsonb, reverse boolean DEFAULT false)\n RETURNS text\n LANGUAGE sql\nAS $function$\n WITH sortby AS (\n SELECT coalesce(_search->'sortby','[{\"field\":\"datetime\", \"direction\":\"desc\"}]') as sort\n ), withid AS (\n SELECT CASE\n WHEN sort @? '$[*] ? (@.field == \"id\")' THEN sort\n ELSE sort || '[{\"field\":\"id\", \"direction\":\"desc\"}]'::jsonb\n END as sort\n FROM sortby\n ), withid_rows AS (\n SELECT jsonb_array_elements(sort) as value FROM withid\n ),sorts AS (\n SELECT\n coalesce(\n (queryable(value->>'field')).expression\n ) as key,\n parse_sort_dir(value->>'direction', reverse) as dir\n FROM withid_rows\n )\n SELECT array_to_string(\n array_agg(concat(key, ' ', dir)),\n ', '\n ) FROM sorts;\n$function$\n;\n\n\n\n-- END migra calculated SQL\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('id', '{\"title\": \"Item ID\",\"description\": \"Item identifier\",\"$ref\": \"https://schemas.stacspec.org/v1.0.0/item-spec/json-schema/item.json#/definitions/core/allOf/2/properties/id\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('geometry', '{\"title\": \"Item Geometry\",\"description\": \"Item Geometry\",\"$ref\": \"https://geojson.org/schema/Feature.json\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('datetime','{\"description\": \"Datetime\",\"type\": \"string\",\"title\": \"Acquired\",\"format\": \"date-time\",\"pattern\": \"(\\\\+00:00|Z)$\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('eo:cloud_cover','{\"$ref\": \"https://stac-extensions.github.io/eo/v1.0.0/schema.json#/definitions/fieldsproperties/eo:cloud_cover\"}','to_int','BTREE');\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDELETE FROM queryables a USING queryables b\n WHERE a.name = b.name AND a.collection_ids IS NOT DISTINCT FROM b.collection_ids AND a.id > b.id;\n\n\nINSERT INTO pgstac_settings (name, value) VALUES\n ('context', 'off'),\n ('context_estimated_count', '100000'),\n ('context_estimated_cost', '100000'),\n ('context_stats_ttl', '1 day'),\n ('default_filter_lang', 'cql2-json'),\n ('additional_properties', 'true'),\n ('use_queue', 'false'),\n ('queue_timeout', '10 minutes'),\n ('update_collection_extent', 'false'),\n ('format_cache', 'false')\nON CONFLICT DO NOTHING\n;\n\nALTER FUNCTION to_text COST 5000;\nALTER FUNCTION to_float COST 5000;\nALTER FUNCTION to_int COST 5000;\nALTER FUNCTION to_tstz COST 5000;\nALTER FUNCTION to_text_array COST 5000;\n\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nGRANT ALL ON SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac to pgstac_admin;\n\n-- pgstac_read role limited to using function apis\nGRANT EXECUTE ON FUNCTION search TO pgstac_read;\nGRANT EXECUTE ON FUNCTION search_query TO pgstac_read;\nGRANT EXECUTE ON FUNCTION item_by_id TO pgstac_read;\nGRANT EXECUTE ON FUNCTION get_item TO pgstac_read;\nGRANT SELECT ON ALL TABLES IN SCHEMA pgstac TO pgstac_read;\n\n\nGRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_ingest;\nGRANT USAGE ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_ingest;\n\nSELECT update_partition_stats_q(partition) FROM partitions;\nSELECT set_version('0.7.4');\n" }, { "path": "src/pgstac/migrations/pgstac.0.7.3.sql", "content": "RESET ROLE;\nDO $$\nDECLARE\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='postgis') THEN\n CREATE EXTENSION IF NOT EXISTS postgis;\n END IF;\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='btree_gist') THEN\n CREATE EXTENSION IF NOT EXISTS btree_gist;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_admin;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_read;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_ingest;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n\nGRANT pgstac_admin TO current_user;\n\n-- Function to make sure pgstac_admin is the owner of items\nCREATE OR REPLACE FUNCTION pgstac_admin_owns() RETURNS VOID AS $$\nDECLARE\n f RECORD;\nBEGIN\n FOR f IN (\n SELECT\n concat(\n oid::regproc::text,\n '(',\n coalesce(pg_get_function_identity_arguments(oid),''),\n ')'\n ) AS name,\n CASE prokind WHEN 'f' THEN 'FUNCTION' WHEN 'p' THEN 'PROCEDURE' WHEN 'a' THEN 'AGGREGATE' END as typ\n FROM pg_proc\n WHERE\n pronamespace=to_regnamespace('pgstac')\n AND proowner != to_regrole('pgstac_admin')\n AND proname NOT LIKE 'pg_stat%'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n FOR f IN (\n SELECT\n oid::regclass::text as name,\n CASE relkind\n WHEN 'i' THEN 'INDEX'\n WHEN 'I' THEN 'INDEX'\n WHEN 'p' THEN 'TABLE'\n WHEN 'r' THEN 'TABLE'\n WHEN 'v' THEN 'VIEW'\n WHEN 'S' THEN 'SEQUENCE'\n ELSE NULL\n END as typ\n FROM pg_class\n WHERE relnamespace=to_regnamespace('pgstac') and relowner != to_regrole('pgstac_admin') AND relkind IN ('r','p','v','S') AND relname NOT LIKE 'pg_stat'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\nSELECT pgstac_admin_owns();\n\nCREATE SCHEMA IF NOT EXISTS pgstac AUTHORIZATION pgstac_admin;\n\nGRANT ALL ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_admin;\n\nALTER ROLE pgstac_admin SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_read SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_ingest SET SEARCH_PATH TO pgstac, public;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\n\nGRANT pgstac_read TO pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\n\nSET ROLE pgstac_admin;\n\nSET SEARCH_PATH TO pgstac, public;\n\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS analyze_items;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS validate_constraints;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\nCREATE OR REPLACE FUNCTION to_int(jsonb) RETURNS int AS $$\n SELECT floor(($1->>0)::float)::int;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION to_float(jsonb) RETURNS float AS $$\n SELECT ($1->>0)::float;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION to_tstz(jsonb) RETURNS timestamptz AS $$\n SELECT ($1->>0)::timestamptz;\n$$ LANGUAGE SQL IMMUTABLE STRICT SET TIME ZONE 'UTC';\n\n\nCREATE OR REPLACE FUNCTION to_text(jsonb) RETURNS text AS $$\n SELECT CASE WHEN jsonb_typeof($1) IN ('array','object') THEN $1::text ELSE $1->>0 END;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION to_text_array(jsonb) RETURNS text[] AS $$\n SELECT\n CASE jsonb_typeof($1)\n WHEN 'array' THEN ARRAY(SELECT jsonb_array_elements_text($1))\n ELSE ARRAY[$1->>0]\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION bbox_geom(_bbox jsonb) RETURNS geometry AS $$\nSELECT CASE jsonb_array_length(_bbox)\n WHEN 4 THEN\n ST_SetSRID(ST_MakeEnvelope(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float,\n (_bbox->>3)::float\n ),4326)\n WHEN 6 THEN\n ST_SetSRID(ST_3DMakeBox(\n ST_MakePoint(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float\n ),\n ST_MakePoint(\n (_bbox->>3)::float,\n (_bbox->>4)::float,\n (_bbox->>5)::float\n )\n ),4326)\n ELSE null END;\n;\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION geom_bbox(_geom geometry) RETURNS jsonb AS $$\n SELECT jsonb_build_array(\n st_xmin(_geom),\n st_ymin(_geom),\n st_xmax(_geom),\n st_ymax(_geom)\n );\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION flip_jsonb_array(j jsonb) RETURNS jsonb AS $$\n SELECT jsonb_agg(value) FROM (SELECT value FROM jsonb_array_elements(j) WITH ORDINALITY ORDER BY ordinality DESC) as t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION explode_dotpaths(j jsonb) RETURNS SETOF text[] AS $$\n SELECT string_to_array(p, '.') as e FROM jsonb_array_elements_text(j) p;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION explode_dotpaths_recurse(IN j jsonb) RETURNS SETOF text[] AS $$\n WITH RECURSIVE t AS (\n SELECT e FROM explode_dotpaths(j) e\n UNION ALL\n SELECT e[1:cardinality(e)-1]\n FROM t\n WHERE cardinality(e)>1\n ) SELECT e FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION jsonb_set_nested(j jsonb, path text[], val jsonb) RETURNS jsonb AS $$\nDECLARE\nBEGIN\n IF cardinality(path) > 1 THEN\n FOR i IN 1..(cardinality(path)-1) LOOP\n IF j #> path[:i] IS NULL THEN\n j := jsonb_set_lax(j, path[:i], '{}', TRUE);\n END IF;\n END LOOP;\n END IF;\n RETURN jsonb_set_lax(j, path, val, true);\n\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\n\n\nCREATE OR REPLACE FUNCTION jsonb_include(j jsonb, f jsonb) RETURNS jsonb AS $$\nDECLARE\n includes jsonb := f-> 'include';\n outj jsonb := '{}'::jsonb;\n path text[];\nBEGIN\n IF\n includes IS NULL\n OR jsonb_array_length(includes) = 0\n THEN\n RETURN j;\n ELSE\n includes := includes || '[\"id\",\"collection\"]'::jsonb;\n FOR path IN SELECT explode_dotpaths(includes) LOOP\n outj := jsonb_set_nested(outj, path, j #> path);\n END LOOP;\n END IF;\n RETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_exclude(j jsonb, f jsonb) RETURNS jsonb AS $$\nDECLARE\n excludes jsonb := f-> 'exclude';\n outj jsonb := j;\n path text[];\nBEGIN\n IF\n excludes IS NULL\n OR jsonb_array_length(excludes) = 0\n THEN\n RETURN j;\n ELSE\n FOR path IN SELECT explode_dotpaths(excludes) LOOP\n outj := outj #- path;\n END LOOP;\n END IF;\n RETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_fields(j jsonb, f jsonb DEFAULT '{\"fields\":[]}') RETURNS jsonb AS $$\n SELECT jsonb_exclude(jsonb_include(j, f), f);\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION merge_jsonb(_a jsonb, _b jsonb) RETURNS jsonb AS $$\n SELECT\n CASE\n WHEN _a = '\"𒍟※\"'::jsonb THEN NULL\n WHEN _a IS NULL OR jsonb_typeof(_a) = 'null' THEN _b\n WHEN jsonb_typeof(_a) = 'object' AND jsonb_typeof(_b) = 'object' THEN\n (\n SELECT\n jsonb_strip_nulls(\n jsonb_object_agg(\n key,\n merge_jsonb(a.value, b.value)\n )\n )\n FROM\n jsonb_each(coalesce(_a,'{}'::jsonb)) as a\n FULL JOIN\n jsonb_each(coalesce(_b,'{}'::jsonb)) as b\n USING (key)\n )\n WHEN\n jsonb_typeof(_a) = 'array'\n AND jsonb_typeof(_b) = 'array'\n AND jsonb_array_length(_a) = jsonb_array_length(_b)\n THEN\n (\n SELECT jsonb_agg(m) FROM\n ( SELECT\n merge_jsonb(\n jsonb_array_elements(_a),\n jsonb_array_elements(_b)\n ) as m\n ) as l\n )\n ELSE _a\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION strip_jsonb(_a jsonb, _b jsonb) RETURNS jsonb AS $$\n SELECT\n CASE\n\n WHEN (_a IS NULL OR jsonb_typeof(_a) = 'null') AND _b IS NOT NULL AND jsonb_typeof(_b) != 'null' THEN '\"𒍟※\"'::jsonb\n WHEN _b IS NULL OR jsonb_typeof(_a) = 'null' THEN _a\n WHEN _a = _b AND jsonb_typeof(_a) = 'object' THEN '{}'::jsonb\n WHEN _a = _b THEN NULL\n WHEN jsonb_typeof(_a) = 'object' AND jsonb_typeof(_b) = 'object' THEN\n (\n SELECT\n jsonb_strip_nulls(\n jsonb_object_agg(\n key,\n strip_jsonb(a.value, b.value)\n )\n )\n FROM\n jsonb_each(_a) as a\n FULL JOIN\n jsonb_each(_b) as b\n USING (key)\n )\n WHEN\n jsonb_typeof(_a) = 'array'\n AND jsonb_typeof(_b) = 'array'\n AND jsonb_array_length(_a) = jsonb_array_length(_b)\n THEN\n (\n SELECT jsonb_agg(m) FROM\n ( SELECT\n strip_jsonb(\n jsonb_array_elements(_a),\n jsonb_array_elements(_b)\n ) as m\n ) as l\n )\n ELSE _a\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION nullif_jsonbnullempty(j jsonb) RETURNS jsonb AS $$\n SELECT nullif(nullif(nullif(j,'null'::jsonb),'{}'::jsonb),'[]'::jsonb);\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION jsonb_array_unique(j jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(jsonb_agg(DISTINCT a)) v FROM jsonb_array_elements(j) a;\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_concat_ignorenull(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT coalesce(a,'[]'::jsonb) || coalesce(b,'[]'::jsonb);\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_least(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(least(nullif_jsonbnullempty(a), nullif_jsonbnullempty(b)));\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_greatest(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(greatest(a, b));\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION first_notnull_sfunc(anyelement, anyelement) RETURNS anyelement AS $$\n SELECT COALESCE($1,$2);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE AGGREGATE first_notnull(anyelement)(\n SFUNC = first_notnull_sfunc,\n STYPE = anyelement\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_array_unique_merge(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_concat_ignorenull,\n FINALFUNC = jsonb_array_unique\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_min(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_least\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_max(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_greatest\n);\n\nCREATE TABLE IF NOT EXISTS migrations (\n version text PRIMARY KEY,\n datetime timestamptz DEFAULT clock_timestamp() NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION get_version() RETURNS text AS $$\n SELECT version FROM pgstac.migrations ORDER BY datetime DESC, version DESC LIMIT 1;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION set_version(text) RETURNS text AS $$\n INSERT INTO pgstac.migrations (version) VALUES ($1)\n ON CONFLICT DO NOTHING\n RETURNING version;\n$$ LANGUAGE SQL;\n\n\nCREATE TABLE IF NOT EXISTS pgstac_settings (\n name text PRIMARY KEY,\n value text NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION table_empty(text) RETURNS boolean AS $$\nDECLARE\n retval boolean;\nBEGIN\n EXECUTE format($q$\n SELECT NOT EXISTS (SELECT 1 FROM %I LIMIT 1)\n $q$,\n $1\n ) INTO retval;\n RETURN retval;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION get_setting(IN _setting text, IN conf jsonb DEFAULT NULL) RETURNS text AS $$\nSELECT COALESCE(\n nullif(conf->>_setting, ''),\n nullif(current_setting(concat('pgstac.',_setting), TRUE),''),\n nullif((SELECT value FROM pgstac.pgstac_settings WHERE name=_setting),'')\n);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_setting_bool(IN _setting text, IN conf jsonb DEFAULT NULL) RETURNS boolean AS $$\nSELECT COALESCE(\n conf->>_setting,\n current_setting(concat('pgstac.',_setting), TRUE),\n (SELECT value FROM pgstac.pgstac_settings WHERE name=_setting),\n 'FALSE'\n)::boolean;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context(conf jsonb DEFAULT NULL) RETURNS text AS $$\n SELECT pgstac.get_setting('context', conf);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context_estimated_count(conf jsonb DEFAULT NULL) RETURNS int AS $$\n SELECT pgstac.get_setting('context_estimated_count', conf)::int;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_estimated_cost();\nCREATE OR REPLACE FUNCTION context_estimated_cost(conf jsonb DEFAULT NULL) RETURNS float AS $$\n SELECT pgstac.get_setting('context_estimated_cost', conf)::float;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_stats_ttl();\nCREATE OR REPLACE FUNCTION context_stats_ttl(conf jsonb DEFAULT NULL) RETURNS interval AS $$\n SELECT pgstac.get_setting('context_stats_ttl', conf)::interval;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION t2s(text) RETURNS text AS $$\n SELECT extract(epoch FROM $1::interval)::text || ' s';\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE STRICT;\n\nCREATE OR REPLACE FUNCTION queue_timeout() RETURNS interval AS $$\n SELECT t2s(coalesce(\n get_setting('queue_timeout'),\n '1h'\n ))::interval;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION notice(VARIADIC text[]) RETURNS boolean AS $$\nDECLARE\ndebug boolean := current_setting('pgstac.debug', true);\nBEGIN\n IF debug THEN\n RAISE NOTICE 'NOTICE FROM FUNC: % >>>>> %', concat_ws(' | ', $1), clock_timestamp();\n RETURN TRUE;\n END IF;\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION empty_arr(ANYARRAY) RETURNS BOOLEAN AS $$\nSELECT CASE\n WHEN $1 IS NULL THEN TRUE\n WHEN cardinality($1)<1 THEN TRUE\nELSE FALSE\nEND;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_intersection(_a ANYARRAY, _b ANYARRAY) RETURNS ANYARRAY AS $$\n SELECT ARRAY ( SELECT unnest(_a) INTERSECT SELECT UNNEST(_b) );\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION array_map_ident(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_ident(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_map_literal(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_literal(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION array_reverse(anyarray) RETURNS anyarray AS $$\nSELECT ARRAY(\n SELECT $1[i]\n FROM generate_subscripts($1,1) AS s(i)\n ORDER BY i DESC\n);\n$$ LANGUAGE SQL STRICT IMMUTABLE;\n\nDROP TABLE IF EXISTS query_queue;\nCREATE TABLE query_queue (\n query text PRIMARY KEY,\n added timestamptz DEFAULT now()\n);\n\nDROP TABLE IF EXISTS query_queue_history;\nCREATE TABLE query_queue_history(\n query text,\n added timestamptz NOT NULL,\n finished timestamptz NOT NULL DEFAULT now(),\n error text\n);\n\nCREATE OR REPLACE PROCEDURE run_queued_queries() AS $$\nDECLARE\n qitem query_queue%ROWTYPE;\n timeout_ts timestamptz;\n error text;\n cnt int := 0;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n DELETE FROM query_queue WHERE query = (SELECT query FROM query_queue ORDER BY added DESC LIMIT 1 FOR UPDATE SKIP LOCKED) RETURNING * INTO qitem;\n IF NOT FOUND THEN\n EXIT;\n END IF;\n cnt := cnt + 1;\n BEGIN\n RAISE NOTICE 'RUNNING QUERY: %', qitem.query;\n EXECUTE qitem.query;\n EXCEPTION WHEN others THEN\n error := format('%s | %s', SQLERRM, SQLSTATE);\n END;\n INSERT INTO query_queue_history (query, added, finished, error)\n VALUES (qitem.query, qitem.added, clock_timestamp(), error);\n COMMIT;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION run_queued_queries_intransaction() RETURNS int AS $$\nDECLARE\n qitem query_queue%ROWTYPE;\n timeout_ts timestamptz;\n error text;\n cnt int := 0;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n DELETE FROM query_queue WHERE query = (SELECT query FROM query_queue ORDER BY added DESC LIMIT 1 FOR UPDATE SKIP LOCKED) RETURNING * INTO qitem;\n IF NOT FOUND THEN\n RETURN cnt;\n END IF;\n cnt := cnt + 1;\n BEGIN\n qitem.query := regexp_replace(qitem.query, 'CONCURRENTLY', '');\n RAISE NOTICE 'RUNNING QUERY: %', qitem.query;\n\n EXECUTE qitem.query;\n EXCEPTION WHEN others THEN\n error := format('%s | %s', SQLERRM, SQLSTATE);\n RAISE WARNING '%', error;\n END;\n INSERT INTO query_queue_history (query, added, finished, error)\n VALUES (qitem.query, qitem.added, clock_timestamp(), error);\n END LOOP;\n RETURN cnt;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION run_or_queue(query text) RETURNS VOID AS $$\nDECLARE\n use_queue text := COALESCE(get_setting('use_queue'), 'FALSE')::boolean;\nBEGIN\n IF get_setting_bool('debug') THEN\n RAISE NOTICE '%', query;\n END IF;\n IF use_queue THEN\n INSERT INTO query_queue (query) VALUES (query) ON CONFLICT DO NOTHING;\n ELSE\n EXECUTE query;\n END IF;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n\nDROP FUNCTION IF EXISTS check_pgstac_settings;\nCREATE OR REPLACE FUNCTION check_pgstac_settings(_sysmem text DEFAULT NULL) RETURNS VOID AS $$\nDECLARE\n settingval text;\n sysmem bigint := pg_size_bytes(_sysmem);\n effective_cache_size bigint := pg_size_bytes(current_setting('effective_cache_size', TRUE));\n shared_buffers bigint := pg_size_bytes(current_setting('shared_buffers', TRUE));\n work_mem bigint := pg_size_bytes(current_setting('work_mem', TRUE));\n max_connections int := current_setting('max_connections', TRUE);\n maintenance_work_mem bigint := pg_size_bytes(current_setting('maintenance_work_mem', TRUE));\n seq_page_cost float := current_setting('seq_page_cost', TRUE);\n random_page_cost float := current_setting('random_page_cost', TRUE);\n temp_buffers bigint := pg_size_bytes(current_setting('temp_buffers', TRUE));\n r record;\nBEGIN\n IF _sysmem IS NULL THEN\n RAISE NOTICE 'Call function with the size of your system memory `SELECT check_pgstac_settings(''4GB'')` to get pg system setting recommendations.';\n ELSE\n IF effective_cache_size < (sysmem * 0.5) THEN\n RAISE WARNING 'effective_cache_size of % is set low for a system with %. Recomended value between % and %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.5), pg_size_pretty(sysmem * 0.75);\n ELSIF effective_cache_size > (sysmem * 0.75) THEN\n RAISE WARNING 'effective_cache_size of % is set high for a system with %. Recomended value between % and %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.5), pg_size_pretty(sysmem * 0.75);\n ELSE\n RAISE NOTICE 'effective_cache_size of % is set appropriately for a system with %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem);\n END IF;\n\n IF shared_buffers < (sysmem * 0.2) THEN\n RAISE WARNING 'shared_buffers of % is set low for a system with %. Recomended value between % and %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.2), pg_size_pretty(sysmem * 0.3);\n ELSIF shared_buffers > (sysmem * 0.3) THEN\n RAISE WARNING 'shared_buffers of % is set high for a system with %. Recomended value between % and %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.2), pg_size_pretty(sysmem * 0.3);\n ELSE\n RAISE NOTICE 'shared_buffers of % is set appropriately for a system with %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem);\n END IF;\n shared_buffers = sysmem * 0.3;\n IF maintenance_work_mem < (sysmem * 0.2) THEN\n RAISE WARNING 'maintenance_work_mem of % is set low for shared_buffers of %. Recomended value between % and %', pg_size_pretty(maintenance_work_mem), pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers * 0.2), pg_size_pretty(shared_buffers * 0.3);\n ELSIF maintenance_work_mem > (shared_buffers * 0.3) THEN\n RAISE WARNING 'maintenance_work_mem of % is set high for shared_buffers of %. Recomended value between % and %', pg_size_pretty(maintenance_work_mem), pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers * 0.2), pg_size_pretty(shared_buffers * 0.3);\n ELSE\n RAISE NOTICE 'maintenance_work_mem of % is set appropriately for shared_buffers of %', pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers);\n END IF;\n\n IF work_mem * max_connections > shared_buffers THEN\n RAISE WARNING 'work_mem setting of % is set high for % max_connections please reduce work_mem to % or decrease max_connections to %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers/max_connections), floor(shared_buffers/work_mem);\n ELSIF work_mem * max_connections < (shared_buffers * 0.75) THEN\n RAISE WARNING 'work_mem setting of % is set low for % max_connections you may consider raising work_mem to % or increasing max_connections to %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers/max_connections), floor(shared_buffers/work_mem);\n ELSE\n RAISE NOTICE 'work_mem setting of % and max_connections of % are adequate for shared_buffers of %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers);\n END IF;\n\n IF random_page_cost / seq_page_cost != 1.1 THEN\n RAISE WARNING 'random_page_cost (%) /seq_page_cost (%) should be set to 1.1 for SSD. Change random_page_cost to %', random_page_cost, seq_page_cost, 1.1 * seq_page_cost;\n ELSE\n RAISE NOTICE 'random_page_cost and seq_page_cost set appropriately for SSD';\n END IF;\n\n IF temp_buffers < greatest(pg_size_bytes('128MB'),(maintenance_work_mem / 2)) THEN\n RAISE WARNING 'pgstac makes heavy use of temp tables, consider raising temp_buffers from % to %', pg_size_pretty(temp_buffers), greatest('128MB', pg_size_pretty((shared_buffers / 16)));\n END IF;\n END IF;\n\n RAISE NOTICE 'VALUES FOR PGSTAC VARIABLES';\n RAISE NOTICE 'These can be set either as GUC system variables or by setting in the pgstac_settings table.';\n\n FOR r IN SELECT name, get_setting(name) as setting, CASE WHEN current_setting(concat('pgstac.',name), TRUE) IS NOT NULL THEN concat('pgstac.',name, ' GUC') WHEN value IS NOT NULL THEN 'pgstac_settings table' ELSE 'Not Set' END as loc FROM pgstac_settings LOOP\n RAISE NOTICE '% is set to % from the %', r.name, r.setting, r.loc;\n END LOOP;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pg_cron';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider intalling pg_cron which can be used to automate tasks';\n ELSE\n RAISE NOTICE 'pg_cron % is installed', settingval;\n END IF;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pgstattuple';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider installing the pgstattuple extension which can be used to help maintain tables and indexes.';\n ELSE\n RAISE NOTICE 'pgstattuple % is installed', settingval;\n END IF;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pg_stat_statements';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider installing the pg_stat_statements extension which is very helpful for tracking the types of queries on the system';\n ELSE\n RAISE NOTICE 'pg_stat_statements % is installed', settingval;\n IF current_setting('pg_stat_statements.track_statements', TRUE) IS DISTINCT FROM 'all' THEN\n RAISE WARNING 'SET pg_stat_statements.track_statements TO ''all''; --In order to track statements within functions.';\n END IF;\n END IF;\n\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac, public SET CLIENT_MIN_MESSAGES TO NOTICE;\n/* looks for a geometry in a stac item first from geometry and falling back to bbox */\nCREATE OR REPLACE FUNCTION stac_geom(value jsonb) RETURNS geometry AS $$\nSELECT\n CASE\n WHEN value ? 'intersects' THEN\n ST_GeomFromGeoJSON(value->>'intersects')\n WHEN value ? 'geometry' THEN\n ST_GeomFromGeoJSON(value->>'geometry')\n WHEN value ? 'bbox' THEN\n pgstac.bbox_geom(value->'bbox')\n ELSE NULL\n END as geometry\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION stac_daterange(\n value jsonb\n) RETURNS tstzrange AS $$\nDECLARE\n props jsonb := value;\n dt timestamptz;\n edt timestamptz;\nBEGIN\n IF props ? 'properties' THEN\n props := props->'properties';\n END IF;\n IF\n props ? 'start_datetime'\n AND props->>'start_datetime' IS NOT NULL\n AND props ? 'end_datetime'\n AND props->>'end_datetime' IS NOT NULL\n THEN\n dt := props->>'start_datetime';\n edt := props->>'end_datetime';\n IF dt > edt THEN\n RAISE EXCEPTION 'start_datetime must be < end_datetime';\n END IF;\n ELSE\n dt := props->>'datetime';\n edt := props->>'datetime';\n END IF;\n IF dt is NULL OR edt IS NULL THEN\n RAISE NOTICE 'DT: %, EDT: %', dt, edt;\n RAISE EXCEPTION 'Either datetime (%) or both start_datetime (%) and end_datetime (%) must be set.', props->>'datetime',props->>'start_datetime',props->>'end_datetime';\n END IF;\n RETURN tstzrange(dt, edt, '[]');\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_datetime(value jsonb) RETURNS timestamptz AS $$\n SELECT lower(stac_daterange(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_end_datetime(value jsonb) RETURNS timestamptz AS $$\n SELECT upper(stac_daterange(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE TABLE IF NOT EXISTS stac_extensions(\n url text PRIMARY KEY,\n content jsonb\n);\nCREATE OR REPLACE FUNCTION queryable_signature(n text, c text[]) RETURNS text AS $$\n SELECT concat(n, c);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TABLE queryables (\n id bigint GENERATED ALWAYS AS identity PRIMARY KEY,\n name text NOT NULL,\n collection_ids text[], -- used to determine what partitions to create indexes on\n definition jsonb,\n property_path text,\n property_wrapper text,\n property_index_type text\n);\nCREATE INDEX queryables_name_idx ON queryables (name);\nCREATE INDEX queryables_collection_idx ON queryables USING GIN (collection_ids);\nCREATE INDEX queryables_property_wrapper_idx ON queryables (property_wrapper);\n\nCREATE OR REPLACE FUNCTION queryables_constraint_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n allcollections text[];\nBEGIN\n RAISE NOTICE 'Making sure that name/collection is unique for queryables';\n IF NEW.collection_ids IS NOT NULL THEN\n IF EXISTS (\n SELECT 1 FROM\n collections\n LEFT JOIN\n unnest(NEW.collection_ids) c\n ON (collections.id = c)\n WHERE c IS NULL\n ) THEN\n RAISE foreign_key_violation;\n RETURN NULL;\n END IF;\n END IF;\n IF TG_OP = 'INSERT' THEN\n IF EXISTS (\n SELECT 1 FROM queryables q\n WHERE\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n )\n ) THEN\n RAISE unique_violation;\n RETURN NULL;\n END IF;\n END IF;\n IF TG_OP = 'UPDATE' THEN\n IF EXISTS (\n SELECT 1 FROM queryables q\n WHERE\n q.id != NEW.id\n AND\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n )\n ) THEN\n RAISE unique_violation\n USING MESSAGE = format(\n 'There is already a queryable for %s for a collection in %s',\n NEW.name,\n NEW.collection_ids\n );\n RETURN NULL;\n END IF;\n END IF;\n\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER queryables_constraint_insert_trigger\nBEFORE INSERT ON queryables\nFOR EACH ROW EXECUTE PROCEDURE queryables_constraint_triggerfunc();\n\nCREATE TRIGGER queryables_constraint_update_trigger\nBEFORE UPDATE ON queryables\nFOR EACH ROW\nWHEN (NEW.name = OLD.name AND NEW.collection_ids IS DISTINCT FROM OLD.collection_ids)\nEXECUTE PROCEDURE queryables_constraint_triggerfunc();\n\n\nCREATE OR REPLACE FUNCTION array_to_path(arr text[]) RETURNS text AS $$\n SELECT string_agg(\n quote_literal(v),\n '->'\n ) FROM unnest(arr) v;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\n\n\nCREATE OR REPLACE FUNCTION queryable(\n IN dotpath text,\n OUT path text,\n OUT expression text,\n OUT wrapper text,\n OUT nulled_wrapper text\n) AS $$\nDECLARE\n q RECORD;\n path_elements text[];\nBEGIN\n IF dotpath IN ('id', 'geometry', 'datetime', 'end_datetime', 'collection') THEN\n path := dotpath;\n expression := dotpath;\n wrapper := NULL;\n RETURN;\n END IF;\n SELECT * INTO q FROM queryables\n WHERE\n name=dotpath\n OR name = 'properties.' || dotpath\n OR name = replace(dotpath, 'properties.', '')\n ;\n IF q.property_wrapper IS NULL THEN\n IF q.definition->>'type' = 'number' THEN\n wrapper := 'to_float';\n nulled_wrapper := wrapper;\n ELSIF q.definition->>'format' = 'date-time' THEN\n wrapper := 'to_tstz';\n nulled_wrapper := wrapper;\n ELSE\n nulled_wrapper := NULL;\n wrapper := 'to_text';\n END IF;\n ELSE\n wrapper := q.property_wrapper;\n nulled_wrapper := wrapper;\n END IF;\n IF q.property_path IS NOT NULL THEN\n path := q.property_path;\n ELSE\n path_elements := string_to_array(dotpath, '.');\n IF path_elements[1] IN ('links', 'assets', 'stac_version', 'stac_extensions') THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSIF path_elements[1] = 'properties' THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSE\n path := format($F$content->'properties'->%s$F$, array_to_path(path_elements));\n END IF;\n END IF;\n expression := format('%I(%s)', wrapper, path);\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\nCREATE OR REPLACE FUNCTION unnest_collection(collection_ids text[] DEFAULT NULL) RETURNS SETOF text AS $$\n DECLARE\n BEGIN\n IF collection_ids IS NULL THEN\n RETURN QUERY SELECT id FROM collections;\n END IF;\n RETURN QUERY SELECT unnest(collection_ids);\n END;\n$$ LANGUAGE PLPGSQL STABLE;\n\nCREATE OR REPLACE FUNCTION normalize_indexdef(def text) RETURNS text AS $$\nDECLARE\nBEGIN\n def := btrim(def, ' \\n\\t');\n\tdef := regexp_replace(def, '^CREATE (UNIQUE )?INDEX ([^ ]* )?ON (ONLY )?([^ ]* )?', '', 'i');\n RETURN def;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE STRICT PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION indexdef(q queryables) RETURNS text AS $$\n DECLARE\n out text;\n BEGIN\n IF q.name = 'id' THEN\n out := 'CREATE UNIQUE INDEX ON %I USING btree (id)';\n ELSIF q.name = 'datetime' THEN\n out := 'CREATE INDEX ON %I USING btree (datetime DESC, end_datetime)';\n ELSIF q.name = 'geometry' THEN\n out := 'CREATE INDEX ON %I USING gist (geometry)';\n ELSE\n out := format($q$CREATE INDEX ON %%I USING %s (%s(((content -> 'properties'::text) -> %L::text)))$q$,\n lower(COALESCE(q.property_index_type, 'BTREE')),\n lower(COALESCE(q.property_wrapper, 'to_text')),\n q.name\n );\n END IF;\n RETURN btrim(out, ' \\n\\t');\n END;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\n\nDROP VIEW IF EXISTS pgstac_indexes;\nCREATE VIEW pgstac_indexes AS\nSELECT\n i.schemaname,\n i.tablename,\n i.indexname,\n regexp_replace(btrim(replace(replace(indexdef, i.indexname, ''),'pgstac.',''),' \\t\\n'), '[ ]+', ' ', 'g') as idx,\n COALESCE(\n (regexp_match(indexdef, '\\(([a-zA-Z]+)\\)'))[1],\n (regexp_match(indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_-]+)''::text'))[1],\n CASE WHEN indexdef ~* '\\(datetime desc, end_datetime\\)' THEN 'datetime' ELSE NULL END\n ) AS field,\n pg_table_size(i.indexname::text) as index_size,\n pg_size_pretty(pg_table_size(i.indexname::text)) as index_size_pretty\nFROM\n pg_indexes i\nWHERE i.schemaname='pgstac' and i.tablename ~ '_items_' AND indexdef !~* ' only ';\n\nDROP VIEW IF EXISTS pgstac_index_stats;\nCREATE VIEW pgstac_indexes_stats AS\nSELECT\n i.schemaname,\n i.tablename,\n i.indexname,\n indexdef,\n COALESCE(\n (regexp_match(indexdef, '\\(([a-zA-Z]+)\\)'))[1],\n (regexp_match(indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_]+)''::text'))[1],\n CASE WHEN indexdef ~* '\\(datetime desc, end_datetime\\)' THEN 'datetime_end_datetime' ELSE NULL END\n ) AS field,\n pg_table_size(i.indexname::text) as index_size,\n pg_size_pretty(pg_table_size(i.indexname::text)) as index_size_pretty,\n n_distinct,\n most_common_vals::text::text[],\n most_common_freqs::text::text[],\n histogram_bounds::text::text[],\n correlation\nFROM\n pg_indexes i\n LEFT JOIN pg_stats s ON (s.tablename = i.indexname)\nWHERE i.schemaname='pgstac' and i.tablename ~ '_items_';\n\nset check_function_bodies to off;\nCREATE OR REPLACE FUNCTION maintain_partition_queries(\n part text DEFAULT 'items',\n dropindexes boolean DEFAULT FALSE,\n rebuildindexes boolean DEFAULT FALSE,\n idxconcurrently boolean DEFAULT FALSE\n) RETURNS SETOF text AS $$\nDECLARE\n rec record;\nBEGIN\n FOR rec IN (\n WITH p AS (\n SELECT\n relid::text as partition,\n replace(replace(\n CASE\n WHEN level = 1 THEN pg_get_expr(c.relpartbound, c.oid)\n ELSE pg_get_expr(parent.relpartbound, parent.oid)\n END,\n 'FOR VALUES IN (''',''), ''')',\n ''\n ) AS collection\n FROM pg_partition_tree('items')\n JOIN pg_class c ON (relid::regclass = c.oid)\n JOIN pg_class parent ON (parentrelid::regclass = parent.oid AND isleaf)\n ), i AS (\n SELECT\n partition,\n indexname,\n regexp_replace(btrim(replace(replace(indexdef, indexname, ''),'pgstac.',''),' \\t\\n'), '[ ]+', ' ', 'g') as iidx,\n COALESCE(\n (regexp_match(indexdef, '\\(([a-zA-Z]+)\\)'))[1],\n (regexp_match(indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_-]+)''::text'))[1],\n CASE WHEN indexdef ~* '\\(datetime desc, end_datetime\\)' THEN 'datetime' ELSE NULL END\n ) AS field\n FROM\n pg_indexes\n JOIN p ON (tablename=partition)\n ), q AS (\n SELECT\n name AS field,\n collection,\n partition,\n format(indexdef(queryables), partition) as qidx\n FROM queryables, unnest_collection(queryables.collection_ids) collection\n JOIN p USING (collection)\n WHERE property_index_type IS NOT NULL OR name IN ('datetime','geometry','id')\n )\n SELECT * FROM i FULL JOIN q USING (field, partition)\n WHERE lower(iidx) IS DISTINCT FROM lower(qidx)\n ) LOOP\n IF rec.iidx IS NULL THEN\n IF idxconcurrently THEN\n RETURN NEXT replace(rec.qidx, 'INDEX', 'INDEX CONCURRENTLY');\n ELSE\n RETURN NEXT rec.qidx;\n END IF;\n ELSIF rec.qidx IS NULL AND dropindexes THEN\n RETURN NEXT format('DROP INDEX IF EXISTS %I;', rec.indexname);\n ELSIF lower(rec.qidx) != lower(rec.iidx) THEN\n IF dropindexes THEN\n RETURN NEXT format('DROP INDEX IF EXISTS %I; %s;', rec.indexname, rec.qidx);\n ELSE\n IF idxconcurrently THEN\n RETURN NEXT replace(rec.qidx, 'INDEX', 'INDEX CONCURRENTLY');\n ELSE\n RETURN NEXT rec.qidx;\n END IF;\n END IF;\n ELSIF rebuildindexes and rec.indexname IS NOT NULL THEN\n IF idxconcurrently THEN\n RETURN NEXT format('REINDEX INDEX CONCURRENTLY %I;', rec.indexname);\n ELSE\n RETURN NEXT format('REINDEX INDEX %I;', rec.indexname);\n END IF;\n END IF;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION maintain_partitions(\n part text DEFAULT 'items',\n dropindexes boolean DEFAULT FALSE,\n rebuildindexes boolean DEFAULT FALSE\n) RETURNS VOID AS $$\n WITH t AS (\n SELECT run_or_queue(q) FROM maintain_partition_queries(part, dropindexes, rebuildindexes) q\n ) SELECT count(*) FROM t;\n$$ LANGUAGE SQL;\n\n\nCREATE OR REPLACE FUNCTION queryables_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\nBEGIN\n PERFORM maintain_partitions();\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER queryables_trigger AFTER INSERT OR UPDATE ON queryables\nFOR EACH STATEMENT EXECUTE PROCEDURE queryables_trigger_func();\n\n\nCREATE OR REPLACE FUNCTION get_queryables(_collection_ids text[] DEFAULT NULL) RETURNS jsonb AS $$\nDECLARE\nBEGIN\n -- Build up queryables if the input contains valid collection ids or is empty\n IF EXISTS (\n SELECT 1 FROM collections\n WHERE\n _collection_ids IS NULL\n OR cardinality(_collection_ids) = 0\n OR id = ANY(_collection_ids)\n )\n THEN\n RETURN (\n WITH base AS (\n SELECT\n unnest(collection_ids) as collection_id,\n name,\n coalesce(definition, '{\"type\":\"string\"}'::jsonb) as definition\n FROM queryables\n WHERE\n _collection_ids IS NULL OR\n _collection_ids = '{}'::text[] OR\n _collection_ids && collection_ids\n UNION ALL\n SELECT null, name, coalesce(definition, '{\"type\":\"string\"}'::jsonb) as definition\n FROM queryables WHERE collection_ids IS NULL OR collection_ids = '{}'::text[]\n ), g AS (\n SELECT\n name,\n first_notnull(definition) as definition,\n jsonb_array_unique_merge(definition->'enum') as enum,\n jsonb_min(definition->'minimum') as minimum,\n jsonb_min(definition->'maxiumn') as maximum\n FROM base\n GROUP BY 1\n )\n SELECT\n jsonb_build_object(\n '$schema', 'http://json-schema.org/draft-07/schema#',\n '$id', '',\n 'type', 'object',\n 'title', 'STAC Queryables.',\n 'properties', jsonb_object_agg(\n name,\n definition\n ||\n jsonb_strip_nulls(jsonb_build_object(\n 'enum', enum,\n 'minimum', minimum,\n 'maximum', maximum\n ))\n )\n )\n FROM g\n );\n ELSE\n RETURN NULL;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\nCREATE OR REPLACE FUNCTION get_queryables(_collection text DEFAULT NULL) RETURNS jsonb AS $$\n SELECT\n CASE\n WHEN _collection IS NULL THEN get_queryables(NULL::text[])\n ELSE get_queryables(ARRAY[_collection])\n END\n ;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_queryables() RETURNS jsonb AS $$\n SELECT get_queryables(NULL::text[]);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION schema_qualify_refs(url text, j jsonb) returns jsonb as $$\n SELECT regexp_replace(j::text, '\"\\$ref\": \"#', concat('\"$ref\": \"', url, '#'), 'g')::jsonb;\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE VIEW stac_extension_queryables AS\nSELECT DISTINCT key as name, schema_qualify_refs(e.url, j.value) as definition FROM stac_extensions e, jsonb_each(e.content->'definitions'->'fields'->'properties') j;\n\n\nCREATE OR REPLACE FUNCTION missing_queryables(_collection text, _tablesample float DEFAULT 5, minrows float DEFAULT 10) RETURNS TABLE(collection text, name text, definition jsonb, property_wrapper text) AS $$\nDECLARE\n q text;\n _partition text;\n explain_json json;\n psize float;\n estrows float;\nBEGIN\n SELECT format('_items_%s', key) INTO _partition FROM collections WHERE id=_collection;\n\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM %I;', _partition)\n INTO explain_json;\n psize := explain_json->0->'Plan'->'Plan Rows';\n estrows := _tablesample * .01 * psize;\n IF estrows < minrows THEN\n _tablesample := least(100,greatest(_tablesample, (estrows / psize) / 100));\n RAISE NOTICE '%', (psize / estrows) / 100;\n END IF;\n RAISE NOTICE 'Using tablesample % to find missing queryables from % % that has ~% rows estrows: %', _tablesample, _collection, _partition, psize, estrows;\n\n q := format(\n $q$\n WITH q AS (\n SELECT * FROM queryables\n WHERE\n collection_ids IS NULL\n OR %L = ANY(collection_ids)\n ), t AS (\n SELECT\n content->'properties' AS properties\n FROM\n %I\n TABLESAMPLE SYSTEM(%L)\n ), p AS (\n SELECT DISTINCT ON (key)\n key,\n value,\n s.definition\n FROM t\n JOIN LATERAL jsonb_each(properties) ON TRUE\n LEFT JOIN q ON (q.name=key)\n LEFT JOIN stac_extension_queryables s ON (s.name=key)\n WHERE q.definition IS NULL\n )\n SELECT\n %L,\n key,\n COALESCE(definition, jsonb_build_object('type',jsonb_typeof(value))) as definition,\n CASE\n WHEN definition->>'type' = 'integer' THEN 'to_int'\n WHEN COALESCE(definition->>'type', jsonb_typeof(value)) = 'number' THEN 'to_float'\n WHEN COALESCE(definition->>'type', jsonb_typeof(value)) = 'array' THEN 'to_text_array'\n ELSE 'to_text'\n END\n FROM p;\n $q$,\n _collection,\n _partition,\n _tablesample,\n _collection\n );\n RETURN QUERY EXECUTE q;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION missing_queryables(_tablesample float DEFAULT 5) RETURNS TABLE(collection_ids text[], name text, definition jsonb, property_wrapper text) AS $$\n SELECT\n array_agg(collection),\n name,\n definition,\n property_wrapper\n FROM\n collections\n JOIN LATERAL\n missing_queryables(id, _tablesample) c\n ON TRUE\n GROUP BY\n 2,3,4\n ORDER BY 2,1\n ;\n$$ LANGUAGE SQL;\nCREATE OR REPLACE FUNCTION parse_dtrange(\n _indate jsonb,\n relative_base timestamptz DEFAULT date_trunc('hour', CURRENT_TIMESTAMP)\n) RETURNS tstzrange AS $$\nDECLARE\n timestrs text[];\n s timestamptz;\n e timestamptz;\nBEGIN\n timestrs :=\n CASE\n WHEN _indate ? 'timestamp' THEN\n ARRAY[_indate->>'timestamp']\n WHEN _indate ? 'interval' THEN\n to_text_array(_indate->'interval')\n WHEN jsonb_typeof(_indate) = 'array' THEN\n to_text_array(_indate)\n ELSE\n regexp_split_to_array(\n _indate->>0,\n '/'\n )\n END;\n RAISE NOTICE 'TIMESTRS %', timestrs;\n IF cardinality(timestrs) = 1 THEN\n IF timestrs[1] ILIKE 'P%' THEN\n RETURN tstzrange(relative_base - upper(timestrs[1])::interval, relative_base, '[)');\n END IF;\n s := timestrs[1]::timestamptz;\n RETURN tstzrange(s, s, '[]');\n END IF;\n\n IF cardinality(timestrs) != 2 THEN\n RAISE EXCEPTION 'Timestamp cannot have more than 2 values';\n END IF;\n\n IF timestrs[1] = '..' OR timestrs[1] = '' THEN\n s := '-infinity'::timestamptz;\n e := timestrs[2]::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] = '..' OR timestrs[2] = '' THEN\n s := timestrs[1]::timestamptz;\n e := 'infinity'::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[1] ILIKE 'P%' AND timestrs[2] NOT ILIKE 'P%' THEN\n e := timestrs[2]::timestamptz;\n s := e - upper(timestrs[1])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] ILIKE 'P%' AND timestrs[1] NOT ILIKE 'P%' THEN\n s := timestrs[1]::timestamptz;\n e := s + upper(timestrs[2])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n s := timestrs[1]::timestamptz;\n e := timestrs[2]::timestamptz;\n\n RETURN tstzrange(s,e,'[)');\n\n RETURN NULL;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT PARALLEL SAFE SET TIME ZONE 'UTC';\n\nCREATE OR REPLACE FUNCTION parse_dtrange(\n _indate text,\n relative_base timestamptz DEFAULT CURRENT_TIMESTAMP\n) RETURNS tstzrange AS $$\n SELECT parse_dtrange(to_jsonb(_indate), relative_base);\n$$ LANGUAGE SQL STABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION temporal_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\n ll text := 'datetime';\n lh text := 'end_datetime';\n rrange tstzrange;\n rl text;\n rh text;\n outq text;\nBEGIN\n rrange := parse_dtrange(args->1);\n RAISE NOTICE 'Constructing temporal query OP: %, ARGS: %, RRANGE: %', op, args, rrange;\n op := lower(op);\n rl := format('%L::timestamptz', lower(rrange));\n rh := format('%L::timestamptz', upper(rrange));\n outq := CASE op\n WHEN 't_before' THEN 'lh < rl'\n WHEN 't_after' THEN 'll > rh'\n WHEN 't_meets' THEN 'lh = rl'\n WHEN 't_metby' THEN 'll = rh'\n WHEN 't_overlaps' THEN 'll < rl AND rl < lh < rh'\n WHEN 't_overlappedby' THEN 'rl < ll < rh AND lh > rh'\n WHEN 't_starts' THEN 'll = rl AND lh < rh'\n WHEN 't_startedby' THEN 'll = rl AND lh > rh'\n WHEN 't_during' THEN 'll > rl AND lh < rh'\n WHEN 't_contains' THEN 'll < rl AND lh > rh'\n WHEN 't_finishes' THEN 'll > rl AND lh = rh'\n WHEN 't_finishedby' THEN 'll < rl AND lh = rh'\n WHEN 't_equals' THEN 'll = rl AND lh = rh'\n WHEN 't_disjoint' THEN 'NOT (ll <= rh AND lh >= rl)'\n WHEN 't_intersects' THEN 'll <= rh AND lh >= rl'\n WHEN 'anyinteracts' THEN 'll <= rh AND lh >= rl'\n END;\n outq := regexp_replace(outq, '\\mll\\M', ll);\n outq := regexp_replace(outq, '\\mlh\\M', lh);\n outq := regexp_replace(outq, '\\mrl\\M', rl);\n outq := regexp_replace(outq, '\\mrh\\M', rh);\n outq := format('(%s)', outq);\n RETURN outq;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\n\n\nCREATE OR REPLACE FUNCTION spatial_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\n geom text;\n j jsonb := args->1;\nBEGIN\n op := lower(op);\n RAISE NOTICE 'Constructing spatial query OP: %, ARGS: %', op, args;\n IF op NOT IN ('s_equals','s_disjoint','s_touches','s_within','s_overlaps','s_crosses','s_intersects','intersects','s_contains') THEN\n RAISE EXCEPTION 'Spatial Operator % Not Supported', op;\n END IF;\n op := regexp_replace(op, '^s_', 'st_');\n IF op = 'intersects' THEN\n op := 'st_intersects';\n END IF;\n -- Convert geometry to WKB string\n IF j ? 'type' AND j ? 'coordinates' THEN\n geom := st_geomfromgeojson(j)::text;\n ELSIF jsonb_typeof(j) = 'array' THEN\n geom := bbox_geom(j)::text;\n END IF;\n\n RETURN format('%s(geometry, %L::geometry)', op, geom);\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION query_to_cql2(q jsonb) RETURNS jsonb AS $$\n-- Translates anything passed in through the deprecated \"query\" into equivalent CQL2\nWITH t AS (\n SELECT key as property, value as ops\n FROM jsonb_each(q)\n), t2 AS (\n SELECT property, (jsonb_each(ops)).*\n FROM t WHERE jsonb_typeof(ops) = 'object'\n UNION ALL\n SELECT property, 'eq', ops\n FROM t WHERE jsonb_typeof(ops) != 'object'\n)\nSELECT\n jsonb_strip_nulls(jsonb_build_object(\n 'op', 'and',\n 'args', jsonb_agg(\n jsonb_build_object(\n 'op', key,\n 'args', jsonb_build_array(\n jsonb_build_object('property',property),\n value\n )\n )\n )\n )\n) as qcql FROM t2\n;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\nCREATE OR REPLACE FUNCTION cql1_to_cql2(j jsonb) RETURNS jsonb AS $$\nDECLARE\n args jsonb;\n ret jsonb;\nBEGIN\n RAISE NOTICE 'CQL1_TO_CQL2: %', j;\n IF j ? 'filter' THEN\n RETURN cql1_to_cql2(j->'filter');\n END IF;\n IF j ? 'property' THEN\n RETURN j;\n END IF;\n IF jsonb_typeof(j) = 'array' THEN\n SELECT jsonb_agg(cql1_to_cql2(el)) INTO args FROM jsonb_array_elements(j) el;\n RETURN args;\n END IF;\n IF jsonb_typeof(j) = 'number' THEN\n RETURN j;\n END IF;\n IF jsonb_typeof(j) = 'string' THEN\n RETURN j;\n END IF;\n\n IF jsonb_typeof(j) = 'object' THEN\n SELECT jsonb_build_object(\n 'op', key,\n 'args', cql1_to_cql2(value)\n ) INTO ret\n FROM jsonb_each(j)\n WHERE j IS NOT NULL;\n RETURN ret;\n END IF;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE STRICT;\n\nCREATE TABLE cql2_ops (\n op text PRIMARY KEY,\n template text,\n types text[]\n);\nINSERT INTO cql2_ops (op, template, types) VALUES\n ('eq', '%s = %s', NULL),\n ('neq', '%s != %s', NULL),\n ('ne', '%s != %s', NULL),\n ('!=', '%s != %s', NULL),\n ('<>', '%s != %s', NULL),\n ('lt', '%s < %s', NULL),\n ('lte', '%s <= %s', NULL),\n ('gt', '%s > %s', NULL),\n ('gte', '%s >= %s', NULL),\n ('le', '%s <= %s', NULL),\n ('ge', '%s >= %s', NULL),\n ('=', '%s = %s', NULL),\n ('<', '%s < %s', NULL),\n ('<=', '%s <= %s', NULL),\n ('>', '%s > %s', NULL),\n ('>=', '%s >= %s', NULL),\n ('like', '%s LIKE %s', NULL),\n ('ilike', '%s ILIKE %s', NULL),\n ('+', '%s + %s', NULL),\n ('-', '%s - %s', NULL),\n ('*', '%s * %s', NULL),\n ('/', '%s / %s', NULL),\n ('not', 'NOT (%s)', NULL),\n ('between', '%s BETWEEN %s AND %s', NULL),\n ('isnull', '%s IS NULL', NULL),\n ('upper', 'upper(%s)', NULL),\n ('lower', 'lower(%s)', NULL)\nON CONFLICT (op) DO UPDATE\n SET\n template = EXCLUDED.template\n;\n\n\nCREATE OR REPLACE FUNCTION cql2_query(j jsonb, wrapper text DEFAULT NULL) RETURNS text AS $$\n#variable_conflict use_variable\nDECLARE\n args jsonb := j->'args';\n arg jsonb;\n op text := lower(j->>'op');\n cql2op RECORD;\n literal text;\n _wrapper text;\n leftarg text;\n rightarg text;\nBEGIN\n IF j IS NULL OR (op IS NOT NULL AND args IS NULL) THEN\n RETURN NULL;\n END IF;\n RAISE NOTICE 'CQL2_QUERY: %', j;\n IF j ? 'filter' THEN\n RETURN cql2_query(j->'filter');\n END IF;\n\n IF j ? 'upper' THEN\n RETURN cql2_query(jsonb_build_object('op', 'upper', 'args', j->'upper'));\n END IF;\n\n IF j ? 'lower' THEN\n RETURN cql2_query(jsonb_build_object('op', 'lower', 'args', j->'lower'));\n END IF;\n\n -- Temporal Query\n IF op ilike 't_%' or op = 'anyinteracts' THEN\n RETURN temporal_op_query(op, args);\n END IF;\n\n -- If property is a timestamp convert it to text to use with\n -- general operators\n IF j ? 'timestamp' THEN\n RETURN format('%L::timestamptz', to_tstz(j->'timestamp'));\n END IF;\n IF j ? 'interval' THEN\n RAISE EXCEPTION 'Please use temporal operators when using intervals.';\n RETURN NONE;\n END IF;\n\n -- Spatial Query\n IF op ilike 's_%' or op = 'intersects' THEN\n RETURN spatial_op_query(op, args);\n END IF;\n\n IF op IN ('a_equals','a_contains','a_contained_by','a_overlaps') THEN\n IF args->0 ? 'property' THEN\n leftarg := format('to_text_array(%s)', (queryable(args->0->>'property')).path);\n END IF;\n IF args->1 ? 'property' THEN\n rightarg := format('to_text_array(%s)', (queryable(args->1->>'property')).path);\n END IF;\n RETURN FORMAT(\n '%s %s %s',\n COALESCE(leftarg, quote_literal(to_text_array(args->0))),\n CASE op\n WHEN 'a_equals' THEN '='\n WHEN 'a_contains' THEN '@>'\n WHEN 'a_contained_by' THEN '<@'\n WHEN 'a_overlaps' THEN '&&'\n END,\n COALESCE(rightarg, quote_literal(to_text_array(args->1)))\n );\n END IF;\n\n IF op = 'in' THEN\n RAISE NOTICE 'IN : % % %', args, jsonb_build_array(args->0), args->1;\n args := jsonb_build_array(args->0) || (args->1);\n RAISE NOTICE 'IN2 : %', args;\n END IF;\n\n\n\n IF op = 'between' THEN\n args = jsonb_build_array(\n args->0,\n args->1->0,\n args->1->1\n );\n END IF;\n\n -- Make sure that args is an array and run cql2_query on\n -- each element of the array\n RAISE NOTICE 'ARGS PRE: %', args;\n IF j ? 'args' THEN\n IF jsonb_typeof(args) != 'array' THEN\n args := jsonb_build_array(args);\n END IF;\n\n IF jsonb_path_exists(args, '$[*] ? (@.property == \"id\" || @.property == \"datetime\" || @.property == \"end_datetime\" || @.property == \"collection\")') THEN\n wrapper := NULL;\n ELSE\n -- if any of the arguments are a property, try to get the property_wrapper\n FOR arg IN SELECT jsonb_path_query(args, '$[*] ? (@.property != null)') LOOP\n RAISE NOTICE 'Arg: %', arg;\n wrapper := (queryable(arg->>'property')).nulled_wrapper;\n RAISE NOTICE 'Property: %, Wrapper: %', arg, wrapper;\n IF wrapper IS NOT NULL THEN\n EXIT;\n END IF;\n END LOOP;\n\n -- if the property was not in queryables, see if any args were numbers\n IF\n wrapper IS NULL\n AND jsonb_path_exists(args, '$[*] ? (@.type()==\"number\")')\n THEN\n wrapper := 'to_float';\n END IF;\n wrapper := coalesce(wrapper, 'to_text');\n END IF;\n\n SELECT jsonb_agg(cql2_query(a, wrapper))\n INTO args\n FROM jsonb_array_elements(args) a;\n END IF;\n RAISE NOTICE 'ARGS: %', args;\n\n IF op IN ('and', 'or') THEN\n RETURN\n format(\n '(%s)',\n array_to_string(to_text_array(args), format(' %s ', upper(op)))\n );\n END IF;\n\n IF op = 'in' THEN\n RAISE NOTICE 'IN -- % %', args->0, to_text(args->0);\n RETURN format(\n '%s IN (%s)',\n to_text(args->0),\n array_to_string((to_text_array(args))[2:], ',')\n );\n END IF;\n\n -- Look up template from cql2_ops\n IF j ? 'op' THEN\n SELECT * INTO cql2op FROM cql2_ops WHERE cql2_ops.op ilike op;\n IF FOUND THEN\n -- If specific index set in queryables for a property cast other arguments to that type\n\n RETURN format(\n cql2op.template,\n VARIADIC (to_text_array(args))\n );\n ELSE\n RAISE EXCEPTION 'Operator % Not Supported.', op;\n END IF;\n END IF;\n\n\n IF wrapper IS NOT NULL THEN\n RAISE NOTICE 'Wrapping % with %', j, wrapper;\n IF j ? 'property' THEN\n RETURN format('%I(%s)', wrapper, (queryable(j->>'property')).path);\n ELSE\n RETURN format('%I(%L)', wrapper, j);\n END IF;\n ELSIF j ? 'property' THEN\n RETURN quote_ident(j->>'property');\n END IF;\n\n RETURN quote_literal(to_text(j));\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION paging_dtrange(\n j jsonb\n) RETURNS tstzrange AS $$\nDECLARE\n op text;\n filter jsonb := j->'filter';\n dtrange tstzrange := tstzrange('-infinity'::timestamptz,'infinity'::timestamptz);\n sdate timestamptz := '-infinity'::timestamptz;\n edate timestamptz := 'infinity'::timestamptz;\n jpitem jsonb;\nBEGIN\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n END IF;\n IF NOT (filter @? '$.**.op ? (@ == \"or\" || @ == \"not\")') THEN\n FOR jpitem IN SELECT j FROM jsonb_path_query(filter,'strict $.** ? (@.args[*].property == \"datetime\")'::jsonpath) j LOOP\n op := lower(jpitem->>'op');\n dtrange := parse_dtrange(jpitem->'args'->1);\n IF op IN ('<=', 'lt', 'lte', '<', 'le', 't_before') THEN\n sdate := greatest(sdate,'-infinity');\n edate := least(edate, upper(dtrange));\n ELSIF op IN ('>=', '>', 'gt', 'gte', 'ge', 't_after') THEN\n edate := least(edate, 'infinity');\n sdate := greatest(sdate, lower(dtrange));\n ELSIF op IN ('=', 'eq') THEN\n edate := least(edate, upper(dtrange));\n sdate := greatest(sdate, lower(dtrange));\n END IF;\n RAISE NOTICE '2 OP: %, ARGS: %, DTRANGE: %, SDATE: %, EDATE: %', op, jpitem->'args'->1, dtrange, sdate, edate;\n END LOOP;\n END IF;\n IF sdate > edate THEN\n RETURN 'empty'::tstzrange;\n END IF;\n RETURN tstzrange(sdate,edate, '[]');\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT SET TIME ZONE 'UTC';\n\nCREATE OR REPLACE FUNCTION paging_collections(\n IN j jsonb\n) RETURNS text[] AS $$\nDECLARE\n filter jsonb := j->'filter';\n jpitem jsonb;\n op text;\n args jsonb;\n arg jsonb;\n collections text[];\nBEGIN\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n END IF;\n IF NOT (filter @? '$.**.op ? (@ == \"or\" || @ == \"not\")') THEN\n FOR jpitem IN SELECT j FROM jsonb_path_query(filter,'strict $.** ? (@.args[*].property == \"collection\")'::jsonpath) j LOOP\n RAISE NOTICE 'JPITEM: %', jpitem;\n op := jpitem->>'op';\n args := jpitem->'args';\n IF op IN ('=', 'eq', 'in') THEN\n FOR arg IN SELECT a FROM jsonb_array_elements(args) a LOOP\n IF jsonb_typeof(arg) IN ('string', 'array') THEN\n RAISE NOTICE 'arg: %, collections: %', arg, collections;\n IF collections IS NULL OR collections = '{}'::text[] THEN\n collections := to_text_array(arg);\n ELSE\n collections := array_intersection(collections, to_text_array(arg));\n END IF;\n END IF;\n END LOOP;\n END IF;\n END LOOP;\n END IF;\n IF collections = '{}'::text[] THEN\n RETURN NULL;\n END IF;\n RETURN collections;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\nCREATE OR REPLACE FUNCTION collection_base_item(content jsonb) RETURNS jsonb AS $$\n SELECT jsonb_build_object(\n 'type', 'Feature',\n 'stac_version', content->'stac_version',\n 'assets', content->'item_assets',\n 'collection', content->'id'\n );\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TABLE IF NOT EXISTS collections (\n key bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,\n id text GENERATED ALWAYS AS (content->>'id') STORED UNIQUE,\n content JSONB NOT NULL,\n base_item jsonb GENERATED ALWAYS AS (pgstac.collection_base_item(content)) STORED,\n partition_trunc text CHECK (partition_trunc IN ('year', 'month'))\n);\n\n\n\nCREATE OR REPLACE FUNCTION collection_base_item(cid text) RETURNS jsonb AS $$\n SELECT pgstac.collection_base_item(content) FROM pgstac.collections WHERE id = cid LIMIT 1;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION create_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION update_collection(data jsonb) RETURNS VOID AS $$\nDECLARE\n out collections%ROWTYPE;\nBEGIN\n UPDATE collections SET content=data WHERE id = data->>'id' RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ON CONFLICT (id) DO\n UPDATE\n SET content=EXCLUDED.content\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION delete_collection(_id text) RETURNS VOID AS $$\nDECLARE\n out collections%ROWTYPE;\nBEGIN\n DELETE FROM collections WHERE id = _id RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION get_collection(id text) RETURNS jsonb AS $$\n SELECT content FROM collections\n WHERE id=$1\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION all_collections() RETURNS jsonb AS $$\n SELECT jsonb_agg(content) FROM collections;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\nCREATE TABLE items (\n id text NOT NULL,\n geometry geometry NOT NULL,\n collection text NOT NULL,\n datetime timestamptz NOT NULL,\n end_datetime timestamptz NOT NULL,\n content JSONB NOT NULL\n)\nPARTITION BY LIST (collection)\n;\n\nCREATE INDEX \"datetime_idx\" ON items USING BTREE (datetime DESC, end_datetime ASC);\nCREATE INDEX \"geometry_idx\" ON items USING GIST (geometry);\n\nCREATE STATISTICS datetime_stats (dependencies) on datetime, end_datetime from items;\n\nALTER TABLE items ADD CONSTRAINT items_collections_fk FOREIGN KEY (collection) REFERENCES collections(id) ON DELETE CASCADE DEFERRABLE;\n\nCREATE OR REPLACE FUNCTION partition_after_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p text;\n t timestamptz := clock_timestamp();\nBEGIN\n RAISE NOTICE 'Updating partition stats %', t;\n FOR p IN SELECT DISTINCT partition\n FROM newdata n JOIN partition_sys_meta p\n ON (n.collection=p.collection AND n.datetime <@ p.partition_dtrange)\n LOOP\n PERFORM run_or_queue(format('SELECT update_partition_stats(%L, %L);', p, true));\n END LOOP;\n RAISE NOTICE 't: % %', t, clock_timestamp() - t;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE TRIGGER items_after_insert_trigger\nAFTER INSERT ON items\nREFERENCING NEW TABLE AS newdata\nFOR EACH STATEMENT\nEXECUTE FUNCTION partition_after_triggerfunc();\n\nCREATE TRIGGER items_after_update_trigger\nAFTER DELETE ON items\nREFERENCING OLD TABLE AS newdata\nFOR EACH STATEMENT\nEXECUTE FUNCTION partition_after_triggerfunc();\n\nCREATE TRIGGER items_after_delete_trigger\nAFTER UPDATE ON items\nREFERENCING NEW TABLE AS newdata\nFOR EACH STATEMENT\nEXECUTE FUNCTION partition_after_triggerfunc();\n\n\nCREATE OR REPLACE FUNCTION content_slim(_item jsonb) RETURNS jsonb AS $$\n SELECT strip_jsonb(_item - '{id,geometry,collection,type}'::text[], collection_base_item(_item->>'collection')) - '{id,geometry,collection,type}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_dehydrate(content jsonb) RETURNS items AS $$\n SELECT\n content->>'id' as id,\n stac_geom(content) as geometry,\n content->>'collection' as collection,\n stac_datetime(content) as datetime,\n stac_end_datetime(content) as end_datetime,\n content_slim(content) as content\n ;\n$$ LANGUAGE SQL STABLE;\n\nCREATE OR REPLACE FUNCTION include_field(f text, fields jsonb DEFAULT '{}'::jsonb) RETURNS boolean AS $$\nDECLARE\n includes jsonb := fields->'include';\n excludes jsonb := fields->'exclude';\nBEGIN\n IF f IS NULL THEN\n RETURN NULL;\n END IF;\n\n\n IF\n jsonb_typeof(excludes) = 'array'\n AND jsonb_array_length(excludes)>0\n AND excludes ? f\n THEN\n RETURN FALSE;\n END IF;\n\n IF\n (\n jsonb_typeof(includes) = 'array'\n AND jsonb_array_length(includes) > 0\n AND includes ? f\n ) OR\n (\n includes IS NULL\n OR jsonb_typeof(includes) = 'null'\n OR jsonb_array_length(includes) = 0\n )\n THEN\n RETURN TRUE;\n END IF;\n\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nDROP FUNCTION IF EXISTS content_hydrate(jsonb, jsonb, jsonb);\nCREATE OR REPLACE FUNCTION content_hydrate(\n _item jsonb,\n _base_item jsonb,\n fields jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\n SELECT merge_jsonb(\n jsonb_fields(_item, fields),\n jsonb_fields(_base_item, fields)\n );\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION content_hydrate(_item items, _collection collections, fields jsonb DEFAULT '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\n content jsonb;\n base_item jsonb := _collection.base_item;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry, 20)::jsonb;\n END IF;\n output := content_hydrate(\n jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'collection', _item.collection,\n 'type', 'Feature'\n ) || _item.content,\n _collection.base_item,\n fields\n );\n\n RETURN output;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_nonhydrated(\n _item items,\n fields jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry, 20)::jsonb;\n END IF;\n output := jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'collection', _item.collection,\n 'type', 'Feature'\n ) || _item.content;\n RETURN output;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_hydrate(_item items, fields jsonb DEFAULT '{}'::jsonb) RETURNS jsonb AS $$\n SELECT content_hydrate(\n _item,\n (SELECT c FROM collections c WHERE id=_item.collection LIMIT 1),\n fields\n );\n$$ LANGUAGE SQL STABLE;\n\n\nCREATE UNLOGGED TABLE items_staging (\n content JSONB NOT NULL\n);\nCREATE UNLOGGED TABLE items_staging_ignore (\n content JSONB NOT NULL\n);\nCREATE UNLOGGED TABLE items_staging_upsert (\n content JSONB NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION items_staging_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p record;\n _partitions text[];\n part text;\n ts timestamptz := clock_timestamp();\nBEGIN\n RAISE NOTICE 'Creating Partitions. %', clock_timestamp() - ts;\n\n FOR part IN WITH t AS (\n SELECT\n n.content->>'collection' as collection,\n stac_daterange(n.content->'properties') as dtr,\n partition_trunc\n FROM newdata n JOIN collections ON (n.content->>'collection'=collections.id)\n ), p AS (\n SELECT\n collection,\n COALESCE(date_trunc(partition_trunc::text, lower(dtr)),'-infinity') as d,\n tstzrange(min(lower(dtr)),max(lower(dtr)),'[]') as dtrange,\n tstzrange(min(upper(dtr)),max(upper(dtr)),'[]') as edtrange\n FROM t\n GROUP BY 1,2\n ) SELECT check_partition(collection, dtrange, edtrange) FROM p LOOP\n RAISE NOTICE 'Partition %', part;\n END LOOP;\n\n RAISE NOTICE 'Doing the insert. %', clock_timestamp() - ts;\n IF TG_TABLE_NAME = 'items_staging' THEN\n INSERT INTO items\n SELECT\n (content_dehydrate(content)).*\n FROM newdata;\n RAISE NOTICE 'Doing the delete. %', clock_timestamp() - ts;\n DELETE FROM items_staging;\n ELSIF TG_TABLE_NAME = 'items_staging_ignore' THEN\n INSERT INTO items\n SELECT\n (content_dehydrate(content)).*\n FROM newdata\n ON CONFLICT DO NOTHING;\n RAISE NOTICE 'Doing the delete. %', clock_timestamp() - ts;\n DELETE FROM items_staging_ignore;\n ELSIF TG_TABLE_NAME = 'items_staging_upsert' THEN\n WITH staging_formatted AS (\n SELECT (content_dehydrate(content)).* FROM newdata\n ), deletes AS (\n DELETE FROM items i USING staging_formatted s\n WHERE\n i.id = s.id\n AND i.collection = s.collection\n AND i IS DISTINCT FROM s\n RETURNING i.id, i.collection\n )\n INSERT INTO items\n SELECT s.* FROM\n staging_formatted s\n ON CONFLICT DO NOTHING;\n RAISE NOTICE 'Doing the delete. %', clock_timestamp() - ts;\n DELETE FROM items_staging_upsert;\n END IF;\n RAISE NOTICE 'Done. %', clock_timestamp() - ts;\n\n RETURN NULL;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER items_staging_insert_trigger AFTER INSERT ON items_staging REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\nCREATE TRIGGER items_staging_insert_ignore_trigger AFTER INSERT ON items_staging_ignore REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\nCREATE TRIGGER items_staging_insert_upsert_trigger AFTER INSERT ON items_staging_upsert REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\n\nCREATE OR REPLACE FUNCTION item_by_id(_id text, _collection text DEFAULT NULL) RETURNS items AS\n$$\nDECLARE\n i items%ROWTYPE;\nBEGIN\n SELECT * INTO i FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection) LIMIT 1;\n RETURN i;\nEND;\n$$ LANGUAGE PLPGSQL STABLE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION get_item(_id text, _collection text DEFAULT NULL) RETURNS jsonb AS $$\n SELECT content_hydrate(items) FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection);\n$$ LANGUAGE SQL STABLE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION delete_item(_id text, _collection text DEFAULT NULL) RETURNS VOID AS $$\nDECLARE\nout items%ROWTYPE;\nBEGIN\n DELETE FROM items WHERE id = _id AND (_collection IS NULL OR collection=_collection) RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n--/*\nCREATE OR REPLACE FUNCTION create_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION update_item(content jsonb) RETURNS VOID AS $$\nDECLARE\n old items %ROWTYPE;\n out items%ROWTYPE;\nBEGIN\n PERFORM delete_item(content->>'id', content->>'collection');\n PERFORM create_item(content);\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION create_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION collection_bbox(id text) RETURNS jsonb AS $$\n SELECT (replace(replace(replace(st_extent(geometry)::text,'BOX(','[['),')',']]'),' ',','))::jsonb\n FROM items WHERE collection=$1;\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION collection_temporal_extent(id text) RETURNS jsonb AS $$\n SELECT to_jsonb(array[array[min(datetime)::text, max(datetime)::text]])\n FROM items WHERE collection=$1;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection_extents() RETURNS VOID AS $$\nUPDATE collections SET\n content = content ||\n jsonb_build_object(\n 'extent', jsonb_build_object(\n 'spatial', jsonb_build_object(\n 'bbox', collection_bbox(collections.id)\n ),\n 'temporal', jsonb_build_object(\n 'interval', collection_temporal_extent(collections.id)\n )\n )\n )\n;\n$$ LANGUAGE SQL;\nCREATE TABLE partition_stats (\n partition text PRIMARY KEY,\n dtrange tstzrange,\n edtrange tstzrange,\n spatial geometry,\n last_updated timestamptz,\n keys text[]\n) WITH (FILLFACTOR=90);\n\nCREATE INDEX partitions_range_idx ON partition_stats USING GIST(dtrange);\n\n\nCREATE OR REPLACE FUNCTION constraint_tstzrange(expr text) RETURNS tstzrange AS $$\n WITH t AS (\n SELECT regexp_matches(\n expr,\n E'\\\\(''\\([0-9 :+-]*\\)''\\\\).*\\\\(''\\([0-9 :+-]*\\)''\\\\)'\n ) AS m\n ) SELECT tstzrange(m[1]::timestamptz, m[2]::timestamptz) FROM t\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE STRICT;\n\nCREATE OR REPLACE FUNCTION dt_constraint(coid oid, OUT dt tstzrange, OUT edt tstzrange) RETURNS RECORD AS $$\nDECLARE\n expr text := pg_get_constraintdef(coid);\n matches timestamptz[];\nBEGIN\n IF expr LIKE '%NULL%' THEN\n dt := tstzrange(null::timestamptz, null::timestamptz);\n edt := tstzrange(null::timestamptz, null::timestamptz);\n RETURN;\n END IF;\n WITH f AS (SELECT (regexp_matches(expr, E'([0-9]{4}-[0-1][0-9]-[0-3][0-9] [0-2][0-9]:[0-5][0-9]:[0-5][0-9]\\.?[0-9]*)', 'g'))[1] f)\n SELECT array_agg(f::timestamptz) INTO matches FROM f;\n IF cardinality(matches) = 4 THEN\n dt := tstzrange(matches[1], matches[2],'[]');\n edt := tstzrange(matches[3], matches[4], '[]');\n RETURN;\n ELSIF cardinality(matches) = 2 THEN\n edt := tstzrange(matches[1], matches[2],'[]');\n RETURN;\n END IF;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\nCREATE OR REPLACE VIEW partition_sys_meta AS\nSELECT\n relid::text as partition,\n replace(replace(CASE WHEN level = 1 THEN pg_get_expr(c.relpartbound, c.oid)\n ELSE pg_get_expr(parent.relpartbound, parent.oid)\n END, 'FOR VALUES IN (''',''), ''')','') AS collection,\n level,\n c.reltuples,\n c.relhastriggers,\n COALESCE(constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as partition_dtrange,\n COALESCE((dt_constraint(edt.oid)).dt, constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as constraint_dtrange,\n COALESCE((dt_constraint(edt.oid)).edt, tstzrange('-infinity', 'infinity','[]')) as constraint_edtrange\nFROM\n pg_partition_tree('items')\n JOIN pg_class c ON (relid::regclass = c.oid)\n JOIN pg_class parent ON (parentrelid::regclass = parent.oid AND isleaf)\n LEFT JOIN pg_constraint edt ON (conrelid=c.oid AND contype='c')\nWHERE isleaf\n;\n\nCREATE VIEW partitions AS\nSELECT * FROM partition_sys_meta LEFT JOIN partition_stats USING (partition);\n\nCREATE OR REPLACE FUNCTION update_partition_stats_q(_partition text, istrigger boolean default false) RETURNS VOID AS $$\nDECLARE\nBEGIN\n PERFORM run_or_queue(\n format('SELECT update_partition_stats(%L, %L);', _partition, istrigger)\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION update_partition_stats(_partition text, istrigger boolean default false) RETURNS VOID AS $$\nDECLARE\n dtrange tstzrange;\n edtrange tstzrange;\n cdtrange tstzrange;\n cedtrange tstzrange;\n extent geometry;\n collection text;\nBEGIN\n RAISE NOTICE 'Updating stats for %.', _partition;\n EXECUTE format(\n $q$\n SELECT\n tstzrange(min(datetime), max(datetime),'[]'),\n tstzrange(min(end_datetime), max(end_datetime), '[]')\n FROM %I\n $q$,\n _partition\n ) INTO dtrange, edtrange;\n extent := st_estimatedextent('pgstac', _partition, 'geometry');\n INSERT INTO partition_stats (partition, dtrange, edtrange, spatial, last_updated)\n SELECT _partition, dtrange, edtrange, extent, now()\n ON CONFLICT (partition) DO\n UPDATE SET\n dtrange=EXCLUDED.dtrange,\n edtrange=EXCLUDED.edtrange,\n spatial=EXCLUDED.spatial,\n last_updated=EXCLUDED.last_updated\n ;\n SELECT\n constraint_dtrange, constraint_edtrange, partitions.collection\n INTO cdtrange, cedtrange, collection\n FROM partitions WHERE partition = _partition;\n\n RAISE NOTICE 'Checking if we need to modify constraints.';\n IF\n (cdtrange IS DISTINCT FROM dtrange OR edtrange IS DISTINCT FROM cedtrange)\n AND NOT istrigger\n THEN\n RAISE NOTICE 'Modifying Constraints';\n RAISE NOTICE 'Existing % %', cdtrange, cedtrange;\n RAISE NOTICE 'New % %', dtrange, edtrange;\n PERFORM drop_table_constraints(_partition);\n PERFORM create_table_constraints(_partition, dtrange, edtrange);\n END IF;\n RAISE NOTICE 'Checking if we need to update collection extents.';\n IF get_setting_bool('update_collection_extent') THEN\n RAISE NOTICE 'updating collection extent for %', collection;\n PERFORM run_or_queue(format($q$\n UPDATE collections\n SET content = jsonb_set_lax(\n content,\n '{extent}'::text[],\n collection_extent(%L),\n true,\n 'use_json_null'\n ) WHERE id=%L\n ;\n $q$, collection, collection));\n ELSE\n RAISE NOTICE 'Not updating collection extent for %', collection;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL STRICT;\n\n\nCREATE OR REPLACE FUNCTION partition_name( IN collection text, IN dt timestamptz, OUT partition_name text, OUT partition_range tstzrange) AS $$\nDECLARE\n c RECORD;\n parent_name text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n parent_name := format('_items_%s', c.key);\n\n\n IF c.partition_trunc = 'year' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYY'));\n ELSIF c.partition_trunc = 'month' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYYMM'));\n ELSE\n partition_name := parent_name;\n partition_range := tstzrange('-infinity'::timestamptz, 'infinity'::timestamptz, '[]');\n END IF;\n IF partition_range IS NULL THEN\n partition_range := tstzrange(\n date_trunc(c.partition_trunc::text, dt),\n date_trunc(c.partition_trunc::text, dt) + concat('1 ', c.partition_trunc)::interval\n );\n END IF;\n RETURN;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION drop_table_constraints(t text) RETURNS text AS $$\nDECLARE\n q text;\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM partitions WHERE partition=t) THEN\n RETURN NULL;\n END IF;\n FOR q IN SELECT FORMAT(\n $q$\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n $q$,\n t,\n conname\n ) FROM pg_constraint\n WHERE conrelid=t::regclass::oid AND contype='c'\n LOOP\n EXECUTE q;\n END LOOP;\n RETURN t;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE OR REPLACE FUNCTION create_table_constraints(t text, _dtrange tstzrange, _edtrange tstzrange) RETURNS text AS $$\nDECLARE\n q text;\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM partitions WHERE partition=t) THEN\n RETURN NULL;\n END IF;\n RAISE NOTICE 'Creating Table Constraints for % % %', t, _dtrange, _edtrange;\n IF _dtrange = 'empty' AND _edtrange = 'empty' THEN\n q :=format(\n $q$\n DO $block$\n BEGIN\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n ALTER TABLE %I\n ADD CONSTRAINT %I\n CHECK (((datetime IS NULL) AND (end_datetime IS NULL))) NOT VALID\n ;\n ALTER TABLE %I\n VALIDATE CONSTRAINT %I\n ;\n\n EXCEPTION WHEN others THEN\n RAISE WARNING '%%, Issue Altering Constraints. Please run update_partition_stats(%I)', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n $block$;\n $q$,\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t),\n t\n );\n ELSE\n q :=format(\n $q$\n DO $block$\n BEGIN\n\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n ALTER TABLE %I\n ADD CONSTRAINT %I\n CHECK (\n (datetime >= %L)\n AND (datetime <= %L)\n AND (end_datetime >= %L)\n AND (end_datetime <= %L)\n ) NOT VALID\n ;\n ALTER TABLE %I\n VALIDATE CONSTRAINT %I\n ;\n\n EXCEPTION WHEN others THEN\n RAISE WARNING '%%, Issue Altering Constraints. Please run update_partition_stats(%I)', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n $block$;\n $q$,\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t),\n lower(_dtrange),\n upper(_dtrange),\n lower(_edtrange),\n upper(_edtrange),\n t,\n format('%s_dt', t),\n t\n );\n END IF;\n PERFORM run_or_queue(q);\n RETURN t;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION check_partition(\n _collection text,\n _dtrange tstzrange,\n _edtrange tstzrange\n) RETURNS text AS $$\nDECLARE\n c RECORD;\n pm RECORD;\n _partition_name text;\n _partition_dtrange tstzrange;\n _constraint_dtrange tstzrange;\n _constraint_edtrange tstzrange;\n q text;\n deferrable_q text;\n err_context text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=_collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', _collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n\n IF c.partition_trunc IS NOT NULL THEN\n _partition_dtrange := tstzrange(\n date_trunc(c.partition_trunc, lower(_dtrange)),\n date_trunc(c.partition_trunc, lower(_dtrange)) + (concat('1 ', c.partition_trunc))::interval,\n '[)'\n );\n ELSE\n _partition_dtrange := '[-infinity, infinity]'::tstzrange;\n END IF;\n\n IF NOT _partition_dtrange @> _dtrange THEN\n RAISE EXCEPTION 'dtrange % is greater than the partition size % for collection %', _dtrange, c.partition_trunc, _collection;\n END IF;\n\n\n IF c.partition_trunc = 'year' THEN\n _partition_name := format('_items_%s_%s', c.key, to_char(lower(_partition_dtrange),'YYYY'));\n ELSIF c.partition_trunc = 'month' THEN\n _partition_name := format('_items_%s_%s', c.key, to_char(lower(_partition_dtrange),'YYYYMM'));\n ELSE\n _partition_name := format('_items_%s', c.key);\n END IF;\n\n SELECT * INTO pm FROM partition_sys_meta WHERE collection=_collection AND partition_dtrange @> _dtrange;\n IF FOUND THEN\n RAISE NOTICE '% % %', _edtrange, _dtrange, pm;\n _constraint_edtrange :=\n tstzrange(\n least(\n lower(_edtrange),\n nullif(lower(pm.constraint_edtrange), '-infinity')\n ),\n greatest(\n upper(_edtrange),\n nullif(upper(pm.constraint_edtrange), 'infinity')\n ),\n '[]'\n );\n _constraint_dtrange :=\n tstzrange(\n least(\n lower(_dtrange),\n nullif(lower(pm.constraint_dtrange), '-infinity')\n ),\n greatest(\n upper(_dtrange),\n nullif(upper(pm.constraint_dtrange), 'infinity')\n ),\n '[]'\n );\n\n IF pm.constraint_edtrange @> _edtrange AND pm.constraint_dtrange @> _dtrange THEN\n RETURN pm.partition;\n ELSE\n PERFORM drop_table_constraints(_partition_name);\n END IF;\n ELSE\n _constraint_edtrange := _edtrange;\n _constraint_dtrange := _dtrange;\n END IF;\n RAISE NOTICE 'Creating partition % %', _partition_name, _partition_dtrange;\n IF c.partition_trunc IS NULL THEN\n q := format(\n $q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n $q$,\n _partition_name,\n _collection,\n concat(_partition_name,'_pk'),\n _partition_name\n );\n ELSE\n q := format(\n $q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L) PARTITION BY RANGE (datetime);\n CREATE TABLE IF NOT EXISTS %I partition OF %I FOR VALUES FROM (%L) TO (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n $q$,\n format('_items_%s', c.key),\n _collection,\n _partition_name,\n format('_items_%s', c.key),\n lower(_partition_dtrange),\n upper(_partition_dtrange),\n format('%s_pk', _partition_name),\n _partition_name\n );\n END IF;\n\n BEGIN\n EXECUTE q;\n EXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', _partition_name;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\n END;\n PERFORM create_table_constraints(_partition_name, _constraint_dtrange, _constraint_edtrange);\n PERFORM maintain_partitions(_partition_name);\n PERFORM update_partition_stats_q(_partition_name, true);\n RETURN _partition_name;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION repartition(_collection text, _partition_trunc text, triggered boolean DEFAULT FALSE) RETURNS text AS $$\nDECLARE\n c RECORD;\n q text;\n from_trunc text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=_collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', _collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n IF triggered THEN\n RAISE NOTICE 'Converting % to % partitioning via Trigger', _collection, _partition_trunc;\n ELSE\n RAISE NOTICE 'Converting % from using % to % partitioning', _collection, c.partition_trunc, _partition_trunc;\n IF c.partition_trunc IS NOT DISTINCT FROM _partition_trunc THEN\n RAISE NOTICE 'Collection % already set to use partition by %', _collection, _partition_trunc;\n RETURN _collection;\n END IF;\n END IF;\n\n IF EXISTS (SELECT 1 FROM partitions WHERE collection=_collection LIMIT 1) THEN\n EXECUTE format(\n $q$\n CREATE TEMP TABLE changepartitionstaging ON COMMIT DROP AS SELECT * FROM %I;\n DROP TABLE IF EXISTS %I CASCADE;\n WITH p AS (\n SELECT\n collection,\n CASE WHEN %L IS NULL THEN '-infinity'::timestamptz\n ELSE date_trunc(%L, datetime)\n END as d,\n tstzrange(min(datetime),max(datetime),'[]') as dtrange,\n tstzrange(min(datetime),max(datetime),'[]') as edtrange\n FROM changepartitionstaging\n GROUP BY 1,2\n ) SELECT check_partition(collection, dtrange, edtrange) FROM p;\n INSERT INTO items SELECT * FROM changepartitionstaging;\n DROP TABLE changepartitionstaging;\n $q$,\n concat('_items_', c.key),\n concat('_items_', c.key),\n c.partition_trunc,\n c.partition_trunc\n );\n END IF;\n RETURN _collection;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE OR REPLACE FUNCTION collections_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n q text;\n partition_name text := format('_items_%s', NEW.key);\n partition_exists boolean := false;\n partition_empty boolean := true;\n err_context text;\n loadtemp boolean := FALSE;\nBEGIN\n RAISE NOTICE 'Collection Trigger. % %', NEW.id, NEW.key;\n IF TG_OP = 'UPDATE' AND NEW.partition_trunc IS DISTINCT FROM OLD.partition_trunc THEN\n PERFORM repartition(NEW.id, NEW.partition_trunc, TRUE);\n END IF;\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER collections_trigger AFTER\nINSERT\nOR\nUPDATE ON collections\nFOR EACH ROW EXECUTE FUNCTION collections_trigger_func();\nCREATE OR REPLACE VIEW partition_steps AS\nSELECT\n partition as name,\n date_trunc('month',lower(partition_dtrange)) as sdate,\n date_trunc('month', upper(partition_dtrange)) + '1 month'::interval as edate\n FROM partitions WHERE partition_dtrange IS NOT NULL AND partition_dtrange != 'empty'::tstzrange\n ORDER BY dtrange ASC\n;\n\nCREATE OR REPLACE FUNCTION chunker(\n IN _where text,\n OUT s timestamptz,\n OUT e timestamptz\n) RETURNS SETOF RECORD AS $$\nDECLARE\n explain jsonb;\nBEGIN\n IF _where IS NULL THEN\n _where := ' TRUE ';\n END IF;\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s;', _where)\n INTO explain;\n\n RETURN QUERY\n WITH t AS (\n SELECT j->>0 as p FROM\n jsonb_path_query(\n explain,\n 'strict $.**.\"Relation Name\" ? (@ != null)'\n ) j\n ),\n parts AS (\n SELECT sdate, edate FROM t JOIN partition_steps ON (t.p = name)\n ),\n times AS (\n SELECT sdate FROM parts\n UNION\n SELECT edate FROM parts\n ),\n uniq AS (\n SELECT DISTINCT sdate FROM times ORDER BY sdate\n ),\n last AS (\n SELECT sdate, lead(sdate, 1) over () as edate FROM uniq\n )\n SELECT sdate, edate FROM last WHERE edate IS NOT NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION partition_queries(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN partitions text[] DEFAULT NULL\n) RETURNS SETOF text AS $$\nDECLARE\n query text;\n sdate timestamptz;\n edate timestamptz;\nBEGIN\nIF _where IS NULL OR trim(_where) = '' THEN\n _where = ' TRUE ';\nEND IF;\nRAISE NOTICE 'Getting chunks for % %', _where, _orderby;\nIF _orderby ILIKE 'datetime d%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 DESC LOOP\n RETURN NEXT format($q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n $q$,\n sdate,\n edate,\n _where,\n _orderby\n );\n END LOOP;\nELSIF _orderby ILIKE 'datetime a%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 ASC LOOP\n RETURN NEXT format($q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n $q$,\n sdate,\n edate,\n _where,\n _orderby\n );\n END LOOP;\nELSE\n query := format($q$\n SELECT * FROM items\n WHERE %s\n ORDER BY %s\n $q$, _where, _orderby\n );\n\n RETURN NEXT query;\n RETURN;\nEND IF;\n\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION partition_query_view(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN _limit int DEFAULT 10\n) RETURNS text AS $$\n WITH p AS (\n SELECT * FROM partition_queries(_where, _orderby) p\n )\n SELECT\n CASE WHEN EXISTS (SELECT 1 FROM p) THEN\n (SELECT format($q$\n SELECT * FROM (\n %s\n ) total LIMIT %s\n $q$,\n string_agg(\n format($q$ SELECT * FROM ( %s ) AS sub $q$, p),\n '\n UNION ALL\n '\n ),\n _limit\n ))\n ELSE NULL\n END FROM p;\n$$ LANGUAGE SQL IMMUTABLE;\n\n\n\n\nCREATE OR REPLACE FUNCTION stac_search_to_where(j jsonb) RETURNS text AS $$\nDECLARE\n where_segments text[];\n _where text;\n dtrange tstzrange;\n collections text[];\n geom geometry;\n sdate timestamptz;\n edate timestamptz;\n filterlang text;\n filter jsonb := j->'filter';\nBEGIN\n IF j ? 'ids' THEN\n where_segments := where_segments || format('id = ANY (%L) ', to_text_array(j->'ids'));\n END IF;\n\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n where_segments := where_segments || format('collection = ANY (%L) ', collections);\n END IF;\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n\n where_segments := where_segments || format(' datetime <= %L::timestamptz AND end_datetime >= %L::timestamptz ',\n edate,\n sdate\n );\n END IF;\n\n geom := stac_geom(j);\n IF geom IS NOT NULL THEN\n where_segments := where_segments || format('st_intersects(geometry, %L)',geom);\n END IF;\n\n filterlang := COALESCE(\n j->>'filter-lang',\n get_setting('default_filter_lang', j->'conf')\n );\n IF NOT filter @? '$.**.op' THEN\n filterlang := 'cql-json';\n END IF;\n\n IF filterlang NOT IN ('cql-json','cql2-json') AND j ? 'filter' THEN\n RAISE EXCEPTION '% is not a supported filter-lang. Please use cql-json or cql2-json.', filterlang;\n END IF;\n\n IF j ? 'query' AND j ? 'filter' THEN\n RAISE EXCEPTION 'Can only use either query or filter at one time.';\n END IF;\n\n IF j ? 'query' THEN\n filter := query_to_cql2(j->'query');\n ELSIF filterlang = 'cql-json' THEN\n filter := cql1_to_cql2(filter);\n END IF;\n RAISE NOTICE 'FILTER: %', filter;\n where_segments := where_segments || cql2_query(filter);\n IF cardinality(where_segments) < 1 THEN\n RETURN ' TRUE ';\n END IF;\n\n _where := array_to_string(array_remove(where_segments, NULL), ' AND ');\n\n IF _where IS NULL OR BTRIM(_where) = '' THEN\n RETURN ' TRUE ';\n END IF;\n RETURN _where;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION parse_sort_dir(_dir text, reverse boolean default false) RETURNS text AS $$\n WITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n ) SELECT\n CASE\n WHEN NOT reverse THEN d\n WHEN d = 'ASC' THEN 'DESC'\n WHEN d = 'DESC' THEN 'ASC'\n END\n FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION sort_dir_to_op(_dir text, prev boolean default false) RETURNS text AS $$\n WITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n ) SELECT\n CASE\n WHEN d = 'ASC' AND prev THEN '<='\n WHEN d = 'DESC' AND prev THEN '>='\n WHEN d = 'ASC' THEN '>='\n WHEN d = 'DESC' THEN '<='\n END\n FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION sort_sqlorderby(\n _search jsonb DEFAULT NULL,\n reverse boolean DEFAULT FALSE\n) RETURNS text AS $$\n WITH sortby AS (\n SELECT coalesce(_search->'sortby','[{\"field\":\"datetime\", \"direction\":\"desc\"}]') as sort\n ), withid AS (\n SELECT CASE\n WHEN sort @? '$[*] ? (@.field == \"id\")' THEN sort\n ELSE sort || '[{\"field\":\"id\", \"direction\":\"desc\"}]'::jsonb\n END as sort\n FROM sortby\n ), withid_rows AS (\n SELECT jsonb_array_elements(sort) as value FROM withid\n ),sorts AS (\n SELECT\n coalesce(\n -- field_orderby((items_path(value->>'field')).path_txt),\n (queryable(value->>'field')).expression\n ) as key,\n parse_sort_dir(value->>'direction', reverse) as dir\n FROM withid_rows\n )\n SELECT array_to_string(\n array_agg(concat(key, ' ', dir)),\n ', '\n ) FROM sorts;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_sort_dir(sort_item jsonb) RETURNS text AS $$\n SELECT CASE WHEN sort_item->>'direction' ILIKE 'desc%' THEN 'DESC' ELSE 'ASC' END;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION get_token_val_str(\n _field text,\n _item items\n) RETURNS text AS $$\nDECLARE\nliteral text;\nBEGIN\nRAISE NOTICE '% %', _field, _item;\nCREATE TEMP TABLE _token_item ON COMMIT DROP AS SELECT (_item).*;\nEXECUTE format($q$ SELECT quote_literal(%s) FROM _token_item $q$, _field) INTO literal;\nDROP TABLE IF EXISTS _token_item;\nRETURN literal;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION get_token_filter(_search jsonb = '{}'::jsonb, token_rec jsonb DEFAULT NULL) RETURNS text AS $$\nDECLARE\n token_id text;\n filters text[] := '{}'::text[];\n prev boolean := TRUE;\n field text;\n dir text;\n sort record;\n orfilter text := '';\n orfilters text[] := '{}'::text[];\n andfilters text[] := '{}'::text[];\n output text;\n token_where text;\n token_item items%ROWTYPE;\nBEGIN\n RAISE NOTICE 'Getting Token Filter. % %', _search, token_rec;\n -- If no token provided return NULL\n IF token_rec IS NULL THEN\n IF NOT (_search ? 'token' AND\n (\n (_search->>'token' ILIKE 'prev:%')\n OR\n (_search->>'token' ILIKE 'next:%')\n )\n ) THEN\n RETURN NULL;\n END IF;\n prev := (_search->>'token' ILIKE 'prev:%');\n token_id := substr(_search->>'token', 6);\n IF token_id IS NULL OR token_id = '' THEN\n RAISE WARNING 'next or prev set, but no token id found';\n RETURN NULL;\n END IF;\n SELECT to_jsonb(items) INTO token_rec\n FROM items WHERE id=token_id;\n END IF;\n RAISE NOTICE 'TOKEN ID: % %', token_rec, token_rec->'id';\n\n\n RAISE NOTICE 'TOKEN ID: % %', token_rec, token_rec->'id';\n token_item := jsonb_populate_record(null::items, token_rec);\n RAISE NOTICE 'TOKEN ITEM ----- %', token_item;\n\n\n CREATE TEMP TABLE sorts (\n _row int GENERATED ALWAYS AS IDENTITY NOT NULL,\n _field text PRIMARY KEY,\n _dir text NOT NULL,\n _val text\n ) ON COMMIT DROP;\n\n -- Make sure we only have distinct columns to sort with taking the first one we get\n INSERT INTO sorts (_field, _dir)\n SELECT\n (queryable(value->>'field')).expression,\n get_sort_dir(value)\n FROM\n jsonb_array_elements(coalesce(_search->'sortby','[{\"field\":\"datetime\",\"direction\":\"desc\"}]'))\n ON CONFLICT DO NOTHING\n ;\n RAISE NOTICE 'sorts 1: %', (SELECT jsonb_agg(to_json(sorts)) FROM sorts);\n -- Get the first sort direction provided. As the id is a primary key, if there are any\n -- sorts after id they won't do anything, so make sure that id is the last sort item.\n SELECT _dir INTO dir FROM sorts ORDER BY _row ASC LIMIT 1;\n IF EXISTS (SELECT 1 FROM sorts WHERE _field = 'id') THEN\n DELETE FROM sorts WHERE _row > (SELECT _row FROM sorts WHERE _field = 'id' ORDER BY _row ASC);\n ELSE\n INSERT INTO sorts (_field, _dir) VALUES ('id', dir);\n END IF;\n\n -- Add value from looked up item to the sorts table\n UPDATE sorts SET _val=get_token_val_str(_field, token_item);\n\n -- Check if all sorts are the same direction and use row comparison\n -- to filter\n RAISE NOTICE 'sorts 2: %', (SELECT jsonb_agg(to_json(sorts)) FROM sorts);\n FOR sort IN SELECT * FROM sorts ORDER BY _row asc LOOP\n orfilter := NULL;\n RAISE NOTICE 'SORT: %', sort;\n IF sort._val IS NOT NULL AND ((prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC')) THEN\n orfilter := format($f$(\n (%s < %s) OR (%s IS NULL)\n )$f$,\n sort._field,\n sort._val,\n sort._val\n );\n ELSIF sort._val IS NULL AND ((prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC')) THEN\n RAISE NOTICE '< but null';\n orfilter := format('%s IS NOT NULL', sort._field);\n ELSIF sort._val IS NULL THEN\n RAISE NOTICE '> but null';\n --orfilter := format('%s IS NULL', sort._field);\n ELSE\n orfilter := format($f$(\n (%s > %s) OR (%s IS NULL)\n )$f$,\n sort._field,\n sort._val,\n sort._field\n );\n END IF;\n RAISE NOTICE 'ORFILTER: %', orfilter;\n\n IF orfilter IS NOT NULL THEN\n IF sort._row = 1 THEN\n orfilters := orfilters || orfilter;\n ELSE\n orfilters := orfilters || format('(%s AND %s)', array_to_string(andfilters, ' AND '), orfilter);\n END IF;\n END IF;\n IF sort._val IS NOT NULL THEN\n andfilters := andfilters || format('%s = %s', sort._field, sort._val);\n ELSE\n andfilters := andfilters || format('%s IS NULL', sort._field);\n END IF;\n END LOOP;\n output := array_to_string(orfilters, ' OR ');\n\n DROP TABLE IF EXISTS sorts;\n token_where := concat('(',coalesce(output,'true'),')');\n IF trim(token_where) = '' THEN\n token_where := NULL;\n END IF;\n RAISE NOTICE 'TOKEN_WHERE: %',token_where;\n RETURN token_where;\n END;\n$$ LANGUAGE PLPGSQL SET transform_null_equals TO TRUE\n;\n\nCREATE OR REPLACE FUNCTION search_tohash(jsonb) RETURNS jsonb AS $$\n SELECT $1 - '{token,limit,context,includes,excludes}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION search_hash(jsonb, jsonb) RETURNS text AS $$\n SELECT md5(concat(search_tohash($1)::text,$2::text));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TABLE IF NOT EXISTS searches(\n hash text GENERATED ALWAYS AS (search_hash(search, metadata)) STORED PRIMARY KEY,\n search jsonb NOT NULL,\n _where text,\n orderby text,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n metadata jsonb DEFAULT '{}'::jsonb NOT NULL\n);\n\nCREATE TABLE IF NOT EXISTS search_wheres(\n id bigint generated always as identity primary key,\n _where text NOT NULL,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n statslastupdated timestamptz,\n estimated_count bigint,\n estimated_cost float,\n time_to_estimate float,\n total_count bigint,\n time_to_count float,\n partitions text[]\n);\n\nCREATE INDEX IF NOT EXISTS search_wheres_partitions ON search_wheres USING GIN (partitions);\nCREATE UNIQUE INDEX IF NOT EXISTS search_wheres_where ON search_wheres ((md5(_where)));\n\nCREATE OR REPLACE FUNCTION where_stats(inwhere text, updatestats boolean default false, conf jsonb default null) RETURNS search_wheres AS $$\nDECLARE\n t timestamptz;\n i interval;\n explain_json jsonb;\n partitions text[];\n sw search_wheres%ROWTYPE;\n inwhere_hash text := md5(inwhere);\n _context text := lower(context(conf));\n _stats_ttl interval := context_stats_ttl(conf);\n _estimated_cost float := context_estimated_cost(conf);\n _estimated_count int := context_estimated_count(conf);\nBEGIN\n IF _context = 'off' THEN\n sw._where := inwhere;\n return sw;\n END IF;\n\n SELECT * INTO sw FROM search_wheres WHERE md5(_where)=inwhere_hash FOR UPDATE;\n\n -- Update statistics if explicitly set, if statistics do not exist, or statistics ttl has expired\n IF NOT updatestats THEN\n RAISE NOTICE 'Checking if update is needed for: % .', inwhere;\n RAISE NOTICE 'Stats Last Updated: %', sw.statslastupdated;\n RAISE NOTICE 'TTL: %, Age: %', _stats_ttl, now() - sw.statslastupdated;\n RAISE NOTICE 'Context: %, Existing Total: %', _context, sw.total_count;\n IF\n sw.statslastupdated IS NULL\n OR (now() - sw.statslastupdated) > _stats_ttl\n OR (context(conf) != 'off' AND sw.total_count IS NULL)\n THEN\n updatestats := TRUE;\n END IF;\n END IF;\n\n sw._where := inwhere;\n sw.lastused := now();\n sw.usecount := coalesce(sw.usecount,0) + 1;\n\n IF NOT updatestats THEN\n UPDATE search_wheres SET\n lastused = sw.lastused,\n usecount = sw.usecount\n WHERE md5(_where) = inwhere_hash\n RETURNING * INTO sw\n ;\n RETURN sw;\n END IF;\n\n -- Use explain to get estimated count/cost and a list of the partitions that would be hit by the query\n t := clock_timestamp();\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s', inwhere)\n INTO explain_json;\n RAISE NOTICE 'Time for just the explain: %', clock_timestamp() - t;\n i := clock_timestamp() - t;\n\n sw.statslastupdated := now();\n sw.estimated_count := explain_json->0->'Plan'->'Plan Rows';\n sw.estimated_cost := explain_json->0->'Plan'->'Total Cost';\n sw.time_to_estimate := extract(epoch from i);\n\n RAISE NOTICE 'ESTIMATED_COUNT: % < %', sw.estimated_count, _estimated_count;\n RAISE NOTICE 'ESTIMATED_COST: % < %', sw.estimated_cost, _estimated_cost;\n\n -- Do a full count of rows if context is set to on or if auto is set and estimates are low enough\n IF\n _context = 'on'\n OR\n ( _context = 'auto' AND\n (\n sw.estimated_count < _estimated_count\n AND\n sw.estimated_cost < _estimated_cost\n )\n )\n THEN\n t := clock_timestamp();\n RAISE NOTICE 'Calculating actual count...';\n EXECUTE format(\n 'SELECT count(*) FROM items WHERE %s',\n inwhere\n ) INTO sw.total_count;\n i := clock_timestamp() - t;\n RAISE NOTICE 'Actual Count: % -- %', sw.total_count, i;\n sw.time_to_count := extract(epoch FROM i);\n ELSE\n sw.total_count := NULL;\n sw.time_to_count := NULL;\n END IF;\n\n\n INSERT INTO search_wheres\n (_where, lastused, usecount, statslastupdated, estimated_count, estimated_cost, time_to_estimate, partitions, total_count, time_to_count)\n SELECT sw._where, sw.lastused, sw.usecount, sw.statslastupdated, sw.estimated_count, sw.estimated_cost, sw.time_to_estimate, sw.partitions, sw.total_count, sw.time_to_count\n ON CONFLICT ((md5(_where)))\n DO UPDATE\n SET\n lastused = sw.lastused,\n usecount = sw.usecount,\n statslastupdated = sw.statslastupdated,\n estimated_count = sw.estimated_count,\n estimated_cost = sw.estimated_cost,\n time_to_estimate = sw.time_to_estimate,\n total_count = sw.total_count,\n time_to_count = sw.time_to_count\n ;\n RETURN sw;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION search_query(\n _search jsonb = '{}'::jsonb,\n updatestats boolean = false,\n _metadata jsonb = '{}'::jsonb\n) RETURNS searches AS $$\nDECLARE\n search searches%ROWTYPE;\n pexplain jsonb;\n t timestamptz;\n i interval;\nBEGIN\n SELECT * INTO search FROM searches\n WHERE hash=search_hash(_search, _metadata) FOR UPDATE;\n\n -- Calculate the where clause if not already calculated\n IF search._where IS NULL THEN\n search._where := stac_search_to_where(_search);\n END IF;\n\n -- Calculate the order by clause if not already calculated\n IF search.orderby IS NULL THEN\n search.orderby := sort_sqlorderby(_search);\n END IF;\n\n PERFORM where_stats(search._where, updatestats, _search->'conf');\n\n search.lastused := now();\n search.usecount := coalesce(search.usecount, 0) + 1;\n INSERT INTO searches (search, _where, orderby, lastused, usecount, metadata)\n VALUES (_search, search._where, search.orderby, search.lastused, search.usecount, _metadata)\n ON CONFLICT (hash) DO\n UPDATE SET\n _where = EXCLUDED._where,\n orderby = EXCLUDED.orderby,\n lastused = EXCLUDED.lastused,\n usecount = EXCLUDED.usecount,\n metadata = EXCLUDED.metadata\n RETURNING * INTO search\n ;\n RETURN search;\n\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE OR REPLACE FUNCTION search(_search jsonb = '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n searches searches%ROWTYPE;\n _where text;\n token_where text;\n full_where text;\n orderby text;\n query text;\n token_type text := substr(_search->>'token',1,4);\n _limit int := coalesce((_search->>'limit')::int, 10);\n curs refcursor;\n cntr int := 0;\n iter_record items%ROWTYPE;\n first_record jsonb;\n first_item items%ROWTYPE;\n last_item items%ROWTYPE;\n last_record jsonb;\n out_records jsonb := '[]'::jsonb;\n prev_query text;\n next text;\n prev_id text;\n has_next boolean := false;\n has_prev boolean := false;\n prev text;\n total_count bigint;\n context jsonb;\n collection jsonb;\n includes text[];\n excludes text[];\n exit_flag boolean := FALSE;\n batches int := 0;\n timer timestamptz := clock_timestamp();\n pstart timestamptz;\n pend timestamptz;\n pcurs refcursor;\n search_where search_wheres%ROWTYPE;\n id text;\nBEGIN\nCREATE TEMP TABLE results (i int GENERATED ALWAYS AS IDENTITY, content jsonb) ON COMMIT DROP;\n searches := search_query(_search);\n _where := searches._where;\n orderby := searches.orderby;\n search_where := where_stats(_where);\n total_count := coalesce(search_where.total_count, search_where.estimated_count);\n\n IF token_type='prev' THEN\n token_where := get_token_filter(_search, null::jsonb);\n orderby := sort_sqlorderby(_search, TRUE);\n END IF;\n IF token_type='next' THEN\n token_where := get_token_filter(_search, null::jsonb);\n END IF;\n\n full_where := concat_ws(' AND ', _where, token_where);\n RAISE NOTICE 'FULL QUERY % %', full_where, clock_timestamp()-timer;\n timer := clock_timestamp();\n\n FOR query IN SELECT partition_queries(full_where, orderby, search_where.partitions) LOOP\n timer := clock_timestamp();\n query := format('%s LIMIT %s', query, _limit + 1);\n RAISE NOTICE 'Partition Query: %', query;\n batches := batches + 1;\n -- curs = create_cursor(query);\n RAISE NOTICE 'cursor_tuple_fraction: %', current_setting('cursor_tuple_fraction');\n OPEN curs FOR EXECUTE query;\n LOOP\n FETCH curs into iter_record;\n EXIT WHEN NOT FOUND;\n cntr := cntr + 1;\n\n IF _search->'conf'->>'nohydrate' IS NOT NULL AND (_search->'conf'->>'nohydrate')::boolean = true THEN\n last_record := content_nonhydrated(iter_record, _search->'fields');\n ELSE\n last_record := content_hydrate(iter_record, _search->'fields');\n END IF;\n last_item := iter_record;\n IF cntr = 1 THEN\n first_item := last_item;\n first_record := last_record;\n END IF;\n IF cntr <= _limit THEN\n INSERT INTO results (content) VALUES (last_record);\n ELSIF cntr > _limit THEN\n has_next := true;\n exit_flag := true;\n EXIT;\n END IF;\n END LOOP;\n CLOSE curs;\n RAISE NOTICE 'Query took %.', clock_timestamp()-timer;\n timer := clock_timestamp();\n EXIT WHEN exit_flag;\n END LOOP;\n RAISE NOTICE 'Scanned through % partitions.', batches;\n\nWITH ordered AS (SELECT * FROM results WHERE content IS NOT NULL ORDER BY i)\nSELECT jsonb_agg(content) INTO out_records FROM ordered;\n\nDROP TABLE results;\n\n\n-- Flip things around if this was the result of a prev token query\nIF token_type='prev' THEN\n out_records := flip_jsonb_array(out_records);\n first_item := last_item;\n first_record := last_record;\nEND IF;\n\n-- If this query has a token, see if there is data before the first record\nIF _search ? 'token' THEN\n prev_query := format(\n 'SELECT 1 FROM items WHERE %s LIMIT 1',\n concat_ws(\n ' AND ',\n _where,\n trim(get_token_filter(_search, to_jsonb(first_item)))\n )\n );\n RAISE NOTICE 'Query to get previous record: % --- %', prev_query, first_record;\n EXECUTE prev_query INTO has_prev;\n IF FOUND and has_prev IS NOT NULL THEN\n RAISE NOTICE 'Query results from prev query: %', has_prev;\n has_prev := TRUE;\n END IF;\nEND IF;\nhas_prev := COALESCE(has_prev, FALSE);\n\nIF has_prev THEN\n prev := out_records->0->>'id';\nEND IF;\nIF has_next OR token_type='prev' THEN\n next := out_records->-1->>'id';\nEND IF;\n\nIF context(_search->'conf') != 'off' THEN\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'matched', total_count,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\nELSE\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\nEND IF;\n\ncollection := jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb),\n 'next', next,\n 'prev', prev,\n 'context', context\n);\n\nRETURN collection;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac, public SET cursor_tuple_fraction TO 1;\n\n\nCREATE OR REPLACE FUNCTION search_cursor(_search jsonb = '{}'::jsonb) RETURNS refcursor AS $$\nDECLARE\n curs refcursor;\n searches searches%ROWTYPE;\n _where text;\n _orderby text;\n q text;\n\nBEGIN\n searches := search_query(_search);\n _where := searches._where;\n _orderby := searches.orderby;\n\n OPEN curs FOR\n WITH p AS (\n SELECT * FROM partition_queries(_where, _orderby) p\n )\n SELECT\n CASE WHEN EXISTS (SELECT 1 FROM p) THEN\n (SELECT format($q$\n SELECT * FROM (\n %s\n ) total\n $q$,\n string_agg(\n format($q$ SELECT * FROM ( %s ) AS sub $q$, p),\n '\n UNION ALL\n '\n )\n ))\n ELSE NULL\n END FROM p;\n RETURN curs;\nEND;\n$$ LANGUAGE PLPGSQL;\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION tileenvelope(zoom int, x int, y int) RETURNS geometry AS $$\nWITH t AS (\n SELECT\n 20037508.3427892 as merc_max,\n -20037508.3427892 as merc_min,\n (2 * 20037508.3427892) / (2 ^ zoom) as tile_size\n)\nSELECT st_makeenvelope(\n merc_min + (tile_size * x),\n merc_max - (tile_size * (y + 1)),\n merc_min + (tile_size * (x + 1)),\n merc_max - (tile_size * y),\n 3857\n) FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;DROP FUNCTION IF EXISTS mercgrid;\n\n\nCREATE OR REPLACE FUNCTION ftime() RETURNS interval as $$\nSELECT age(clock_timestamp(), transaction_timestamp());\n$$ LANGUAGE SQL;\nSET SEARCH_PATH to pgstac, public;\n\nDROP FUNCTION IF EXISTS geometrysearch;\nCREATE OR REPLACE FUNCTION geometrysearch(\n IN geom geometry,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE, -- Return as soon as the passed in geometry is full covered\n IN skipcovered boolean DEFAULT TRUE -- Skip any items that would show up completely under the previous items\n) RETURNS jsonb AS $$\nDECLARE\n search searches%ROWTYPE;\n curs refcursor;\n _where text;\n query text;\n iter_record items%ROWTYPE;\n out_records jsonb := '{}'::jsonb[];\n exit_flag boolean := FALSE;\n counter int := 1;\n scancounter int := 1;\n remaining_limit int := _scanlimit;\n tilearea float;\n unionedgeom geometry;\n clippedgeom geometry;\n unionedgeom_area float := 0;\n prev_area float := 0;\n excludes text[];\n includes text[];\n\nBEGIN\n DROP TABLE IF EXISTS pgstac_results;\n CREATE TEMP TABLE pgstac_results (content jsonb) ON COMMIT DROP;\n\n -- If skipcovered is true then you will always want to exit when the passed in geometry is full\n IF skipcovered THEN\n exitwhenfull := TRUE;\n END IF;\n\n SELECT * INTO search FROM searches WHERE hash=queryhash;\n\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Search with Query Hash % Not Found', queryhash;\n END IF;\n\n tilearea := st_area(geom);\n _where := format('%s AND st_intersects(geometry, %L::geometry)', search._where, geom);\n\n\n FOR query IN SELECT * FROM partition_queries(_where, search.orderby) LOOP\n query := format('%s LIMIT %L', query, remaining_limit);\n RAISE NOTICE '%', query;\n OPEN curs FOR EXECUTE query;\n LOOP\n FETCH curs INTO iter_record;\n EXIT WHEN NOT FOUND;\n IF exitwhenfull OR skipcovered THEN -- If we are not using exitwhenfull or skipcovered, we do not need to do expensive geometry operations\n clippedgeom := st_intersection(geom, iter_record.geometry);\n\n IF unionedgeom IS NULL THEN\n unionedgeom := clippedgeom;\n ELSE\n unionedgeom := st_union(unionedgeom, clippedgeom);\n END IF;\n\n unionedgeom_area := st_area(unionedgeom);\n\n IF skipcovered AND prev_area = unionedgeom_area THEN\n scancounter := scancounter + 1;\n CONTINUE;\n END IF;\n\n prev_area := unionedgeom_area;\n\n RAISE NOTICE '% % % %', unionedgeom_area/tilearea, counter, scancounter, ftime();\n END IF;\n RAISE NOTICE '% %', iter_record, content_hydrate(iter_record, fields);\n INSERT INTO pgstac_results (content) VALUES (content_hydrate(iter_record, fields));\n\n IF counter >= _limit\n OR scancounter > _scanlimit\n OR ftime() > _timelimit\n OR (exitwhenfull AND unionedgeom_area >= tilearea)\n THEN\n exit_flag := TRUE;\n EXIT;\n END IF;\n counter := counter + 1;\n scancounter := scancounter + 1;\n\n END LOOP;\n CLOSE curs;\n EXIT WHEN exit_flag;\n remaining_limit := _scanlimit - scancounter;\n END LOOP;\n\n SELECT jsonb_agg(content) INTO out_records FROM pgstac_results WHERE content IS NOT NULL;\n\n RETURN jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb)\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDROP FUNCTION IF EXISTS geojsonsearch;\nCREATE OR REPLACE FUNCTION geojsonsearch(\n IN geojson jsonb,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_geomfromgeojson(geojson),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS xyzsearch;\nCREATE OR REPLACE FUNCTION xyzsearch(\n IN _x int,\n IN _y int,\n IN _z int,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_transform(tileenvelope(_z, _x, _y), 4326),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE PROCEDURE analyze_items() AS $$\nDECLARE\n q text;\n timeout_ts timestamptz;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n SELECT format('ANALYZE (VERBOSE, SKIP_LOCKED) %I;', relname) INTO q\n FROM pg_stat_user_tables\n WHERE relname like '_item%' AND (n_mod_since_analyze>0 OR last_analyze IS NULL) LIMIT 1;\n IF NOT FOUND THEN\n EXIT;\n END IF;\n RAISE NOTICE '%', q;\n EXECUTE q;\n COMMIT;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE PROCEDURE validate_constraints() AS $$\nDECLARE\n q text;\nBEGIN\n FOR q IN\n SELECT\n FORMAT(\n 'ALTER TABLE %I.%I VALIDATE CONSTRAINT %I;',\n nsp.nspname,\n cls.relname,\n con.conname\n )\n\n FROM pg_constraint AS con\n JOIN pg_class AS cls\n ON con.conrelid = cls.oid\n JOIN pg_namespace AS nsp\n ON cls.relnamespace = nsp.oid\n WHERE convalidated = FALSE AND contype in ('c','f')\n AND nsp.nspname = 'pgstac'\n LOOP\n RAISE NOTICE '%', q;\n PERFORM run_or_queue(q);\n COMMIT;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION collection_extent(_collection text, runupdate boolean default false) RETURNS jsonb AS $$\nDECLARE\n geom_extent geometry;\n mind timestamptz;\n maxd timestamptz;\n extent jsonb;\nBEGIN\n IF runupdate THEN\n PERFORM update_partition_stats_q(partition)\n FROM partitions WHERE collection=_collection;\n END IF;\n SELECT\n min(lower(dtrange)),\n max(upper(edtrange)),\n st_extent(spatial)\n INTO\n mind,\n maxd,\n geom_extent\n FROM partitions\n WHERE collection=_collection;\n\n IF geom_extent IS NOT NULL AND mind IS NOT NULL AND maxd IS NOT NULL THEN\n extent := jsonb_build_object(\n 'extent', jsonb_build_object(\n 'spatial', jsonb_build_object(\n 'bbox', to_jsonb(array[array[st_xmin(geom_extent), st_ymin(geom_extent), st_xmax(geom_extent), st_ymax(geom_extent)]])\n ),\n 'temporal', jsonb_build_object(\n 'interval', to_jsonb(array[array[mind, maxd]])\n )\n )\n );\n RETURN extent;\n END IF;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('id', '{\"title\": \"Item ID\",\"description\": \"Item identifier\",\"$ref\": \"https://schemas.stacspec.org/v1.0.0/item-spec/json-schema/item.json#/definitions/core/allOf/2/properties/id\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('geometry', '{\"title\": \"Item Geometry\",\"description\": \"Item Geometry\",\"$ref\": \"https://geojson.org/schema/Feature.json\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('datetime','{\"description\": \"Datetime\",\"type\": \"string\",\"title\": \"Acquired\",\"format\": \"date-time\",\"pattern\": \"(\\\\+00:00|Z)$\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('eo:cloud_cover','{\"$ref\": \"https://stac-extensions.github.io/eo/v1.0.0/schema.json#/definitions/fieldsproperties/eo:cloud_cover\"}','to_int','BTREE');\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDELETE FROM queryables a USING queryables b\n WHERE a.name = b.name AND a.collection_ids IS NOT DISTINCT FROM b.collection_ids AND a.id > b.id;\n\n\nINSERT INTO pgstac_settings (name, value) VALUES\n ('context', 'off'),\n ('context_estimated_count', '100000'),\n ('context_estimated_cost', '100000'),\n ('context_stats_ttl', '1 day'),\n ('default_filter_lang', 'cql2-json'),\n ('additional_properties', 'true'),\n ('use_queue', 'false'),\n ('queue_timeout', '10 minutes'),\n ('update_collection_extent', 'false')\nON CONFLICT DO NOTHING\n;\n\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nGRANT ALL ON SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac to pgstac_admin;\n\n-- pgstac_read role limited to using function apis\nGRANT EXECUTE ON FUNCTION search TO pgstac_read;\nGRANT EXECUTE ON FUNCTION search_query TO pgstac_read;\nGRANT EXECUTE ON FUNCTION item_by_id TO pgstac_read;\nGRANT EXECUTE ON FUNCTION get_item TO pgstac_read;\nGRANT SELECT ON ALL TABLES IN SCHEMA pgstac TO pgstac_read;\n\n\nGRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_ingest;\nGRANT USAGE ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_ingest;\n\nSELECT update_partition_stats_q(partition) FROM partitions;\nSELECT set_version('0.7.3');\n" }, { "path": "src/pgstac/migrations/pgstac.0.7.4-0.7.5.sql", "content": "RESET ROLE;\nDO $$\nDECLARE\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='postgis') THEN\n CREATE EXTENSION IF NOT EXISTS postgis;\n END IF;\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='btree_gist') THEN\n CREATE EXTENSION IF NOT EXISTS btree_gist;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_admin;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_read;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_ingest;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n\nGRANT pgstac_admin TO current_user;\n\n-- Function to make sure pgstac_admin is the owner of items\nCREATE OR REPLACE FUNCTION pgstac_admin_owns() RETURNS VOID AS $$\nDECLARE\n f RECORD;\nBEGIN\n FOR f IN (\n SELECT\n concat(\n oid::regproc::text,\n '(',\n coalesce(pg_get_function_identity_arguments(oid),''),\n ')'\n ) AS name,\n CASE prokind WHEN 'f' THEN 'FUNCTION' WHEN 'p' THEN 'PROCEDURE' WHEN 'a' THEN 'AGGREGATE' END as typ\n FROM pg_proc\n WHERE\n pronamespace=to_regnamespace('pgstac')\n AND proowner != to_regrole('pgstac_admin')\n AND proname NOT LIKE 'pg_stat%'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n FOR f IN (\n SELECT\n oid::regclass::text as name,\n CASE relkind\n WHEN 'i' THEN 'INDEX'\n WHEN 'I' THEN 'INDEX'\n WHEN 'p' THEN 'TABLE'\n WHEN 'r' THEN 'TABLE'\n WHEN 'v' THEN 'VIEW'\n WHEN 'S' THEN 'SEQUENCE'\n ELSE NULL\n END as typ\n FROM pg_class\n WHERE relnamespace=to_regnamespace('pgstac') and relowner != to_regrole('pgstac_admin') AND relkind IN ('r','p','v','S') AND relname NOT LIKE 'pg_stat'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\nSELECT pgstac_admin_owns();\n\nCREATE SCHEMA IF NOT EXISTS pgstac AUTHORIZATION pgstac_admin;\n\nGRANT ALL ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_admin;\n\nALTER ROLE pgstac_admin SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_read SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_ingest SET SEARCH_PATH TO pgstac, public;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\n\nGRANT pgstac_read TO pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\n\nSET ROLE pgstac_admin;\n\nSET SEARCH_PATH TO pgstac, public;\n\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS analyze_items;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS validate_constraints;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\nSET client_min_messages TO WARNING;\nSET SEARCH_PATH to pgstac, public;\n-- BEGIN migra calculated SQL\ndrop view if exists \"pgstac\".\"partition_steps\";\n\ndrop view if exists \"pgstac\".\"partitions\";\n\nset check_function_bodies = off;\n\ncreate or replace view \"pgstac\".\"partitions_view\" as SELECT (pg_partition_tree.relid)::text AS partition,\n replace(replace(\n CASE\n WHEN (pg_partition_tree.level = 1) THEN pg_get_expr(c.relpartbound, c.oid)\n ELSE pg_get_expr(parent.relpartbound, parent.oid)\n END, 'FOR VALUES IN ('''::text, ''::text), ''')'::text, ''::text) AS collection,\n pg_partition_tree.level,\n c.reltuples,\n c.relhastriggers,\n COALESCE(pgstac.constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity'::timestamp with time zone, 'infinity'::timestamp with time zone, '[]'::text)) AS partition_dtrange,\n COALESCE((pgstac.dt_constraint(edt.oid)).dt, pgstac.constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity'::timestamp with time zone, 'infinity'::timestamp with time zone, '[]'::text)) AS constraint_dtrange,\n COALESCE((pgstac.dt_constraint(edt.oid)).edt, tstzrange('-infinity'::timestamp with time zone, 'infinity'::timestamp with time zone, '[]'::text)) AS constraint_edtrange,\n partition_stats.dtrange,\n partition_stats.edtrange,\n partition_stats.spatial,\n partition_stats.last_updated\n FROM ((((pg_partition_tree('pgstac.items'::regclass) pg_partition_tree(relid, parentrelid, isleaf, level)\n JOIN pg_class c ON (((pg_partition_tree.relid)::oid = c.oid)))\n JOIN pg_class parent ON ((((pg_partition_tree.parentrelid)::oid = parent.oid) AND pg_partition_tree.isleaf)))\n LEFT JOIN pg_constraint edt ON (((edt.conrelid = c.oid) AND (edt.contype = 'c'::\"char\"))))\n LEFT JOIN pgstac.partition_stats ON (((pg_partition_tree.relid)::text = partition_stats.partition)))\n WHERE pg_partition_tree.isleaf;\n\n\nCREATE OR REPLACE FUNCTION pgstac.check_partition(_collection text, _dtrange tstzrange, _edtrange tstzrange)\n RETURNS text\n LANGUAGE plpgsql\n SECURITY DEFINER\nAS $function$\nDECLARE\n c RECORD;\n pm RECORD;\n _partition_name text;\n _partition_dtrange tstzrange;\n _constraint_dtrange tstzrange;\n _constraint_edtrange tstzrange;\n q text;\n deferrable_q text;\n err_context text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=_collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', _collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n\n IF c.partition_trunc IS NOT NULL THEN\n _partition_dtrange := tstzrange(\n date_trunc(c.partition_trunc, lower(_dtrange)),\n date_trunc(c.partition_trunc, lower(_dtrange)) + (concat('1 ', c.partition_trunc))::interval,\n '[)'\n );\n ELSE\n _partition_dtrange := '[-infinity, infinity]'::tstzrange;\n END IF;\n\n IF NOT _partition_dtrange @> _dtrange THEN\n RAISE EXCEPTION 'dtrange % is greater than the partition size % for collection %', _dtrange, c.partition_trunc, _collection;\n END IF;\n\n\n IF c.partition_trunc = 'year' THEN\n _partition_name := format('_items_%s_%s', c.key, to_char(lower(_partition_dtrange),'YYYY'));\n ELSIF c.partition_trunc = 'month' THEN\n _partition_name := format('_items_%s_%s', c.key, to_char(lower(_partition_dtrange),'YYYYMM'));\n ELSE\n _partition_name := format('_items_%s', c.key);\n END IF;\n\n SELECT * INTO pm FROM partition_sys_meta WHERE collection=_collection AND partition_dtrange @> _dtrange;\n IF FOUND THEN\n RAISE NOTICE '% % %', _edtrange, _dtrange, pm;\n _constraint_edtrange :=\n tstzrange(\n least(\n lower(_edtrange),\n nullif(lower(pm.constraint_edtrange), '-infinity')\n ),\n greatest(\n upper(_edtrange),\n nullif(upper(pm.constraint_edtrange), 'infinity')\n ),\n '[]'\n );\n _constraint_dtrange :=\n tstzrange(\n least(\n lower(_dtrange),\n nullif(lower(pm.constraint_dtrange), '-infinity')\n ),\n greatest(\n upper(_dtrange),\n nullif(upper(pm.constraint_dtrange), 'infinity')\n ),\n '[]'\n );\n\n IF pm.constraint_edtrange @> _edtrange AND pm.constraint_dtrange @> _dtrange THEN\n RETURN pm.partition;\n ELSE\n PERFORM drop_table_constraints(_partition_name);\n END IF;\n ELSE\n _constraint_edtrange := _edtrange;\n _constraint_dtrange := _dtrange;\n END IF;\n RAISE NOTICE 'Creating partition % %', _partition_name, _partition_dtrange;\n IF c.partition_trunc IS NULL THEN\n q := format(\n $q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n $q$,\n _partition_name,\n _collection,\n concat(_partition_name,'_pk'),\n _partition_name\n );\n ELSE\n q := format(\n $q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L) PARTITION BY RANGE (datetime);\n CREATE TABLE IF NOT EXISTS %I partition OF %I FOR VALUES FROM (%L) TO (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n $q$,\n format('_items_%s', c.key),\n _collection,\n _partition_name,\n format('_items_%s', c.key),\n lower(_partition_dtrange),\n upper(_partition_dtrange),\n format('%s_pk', _partition_name),\n _partition_name\n );\n END IF;\n\n BEGIN\n EXECUTE q;\n EXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', _partition_name;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\n END;\n PERFORM create_table_constraints(_partition_name, _constraint_dtrange, _constraint_edtrange);\n PERFORM maintain_partitions(_partition_name);\n PERFORM update_partition_stats_q(_partition_name, true);\n REFRESH MATERIALIZED VIEW partitions;\n REFRESH MATERIALIZED VIEW partition_steps;\n RETURN _partition_name;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.collection_extent(_collection text, runupdate boolean DEFAULT false)\n RETURNS jsonb\n LANGUAGE plpgsql\nAS $function$\nDECLARE\n geom_extent geometry;\n mind timestamptz;\n maxd timestamptz;\n extent jsonb;\nBEGIN\n IF runupdate THEN\n PERFORM update_partition_stats_q(partition)\n FROM partitions_view WHERE collection=_collection;\n END IF;\n SELECT\n min(lower(dtrange)),\n max(upper(edtrange)),\n st_extent(spatial)\n INTO\n mind,\n maxd,\n geom_extent\n FROM partitions_view\n WHERE collection=_collection;\n\n IF geom_extent IS NOT NULL AND mind IS NOT NULL AND maxd IS NOT NULL THEN\n extent := jsonb_build_object(\n 'extent', jsonb_build_object(\n 'spatial', jsonb_build_object(\n 'bbox', to_jsonb(array[array[st_xmin(geom_extent), st_ymin(geom_extent), st_xmax(geom_extent), st_ymax(geom_extent)]])\n ),\n 'temporal', jsonb_build_object(\n 'interval', to_jsonb(array[array[mind, maxd]])\n )\n )\n );\n RETURN extent;\n END IF;\n RETURN NULL;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.create_table_constraints(t text, _dtrange tstzrange, _edtrange tstzrange)\n RETURNS text\n LANGUAGE plpgsql\n SECURITY DEFINER\nAS $function$\nDECLARE\n q text;\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM partitions_view WHERE partition=t) THEN\n RETURN NULL;\n END IF;\n RAISE NOTICE 'Creating Table Constraints for % % %', t, _dtrange, _edtrange;\n IF _dtrange = 'empty' AND _edtrange = 'empty' THEN\n q :=format(\n $q$\n DO $block$\n BEGIN\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n ALTER TABLE %I\n ADD CONSTRAINT %I\n CHECK (((datetime IS NULL) AND (end_datetime IS NULL))) NOT VALID\n ;\n ALTER TABLE %I\n VALIDATE CONSTRAINT %I\n ;\n\n\n\n EXCEPTION WHEN others THEN\n RAISE WARNING '%%, Issue Altering Constraints. Please run update_partition_stats(%I)', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n $block$;\n $q$,\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t),\n t\n );\n ELSE\n q :=format(\n $q$\n DO $block$\n BEGIN\n\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n ALTER TABLE %I\n ADD CONSTRAINT %I\n CHECK (\n (datetime >= %L)\n AND (datetime <= %L)\n AND (end_datetime >= %L)\n AND (end_datetime <= %L)\n ) NOT VALID\n ;\n ALTER TABLE %I\n VALIDATE CONSTRAINT %I\n ;\n\n\n\n EXCEPTION WHEN others THEN\n RAISE WARNING '%%, Issue Altering Constraints. Please run update_partition_stats(%I)', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n $block$;\n $q$,\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t),\n lower(_dtrange),\n upper(_dtrange),\n lower(_edtrange),\n upper(_edtrange),\n t,\n format('%s_dt', t),\n t\n );\n END IF;\n PERFORM run_or_queue(q);\n RETURN t;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.drop_table_constraints(t text)\n RETURNS text\n LANGUAGE plpgsql\n SECURITY DEFINER\nAS $function$\nDECLARE\n q text;\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM partitions_view WHERE partition=t) THEN\n RETURN NULL;\n END IF;\n FOR q IN SELECT FORMAT(\n $q$\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n $q$,\n t,\n conname\n ) FROM pg_constraint\n WHERE conrelid=t::regclass::oid AND contype='c'\n LOOP\n EXECUTE q;\n END LOOP;\n RETURN t;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.get_token_filter(_sortby jsonb DEFAULT '[{\"field\": \"datetime\", \"direction\": \"desc\"}]'::jsonb, token_item pgstac.items DEFAULT NULL::pgstac.items, prev boolean DEFAULT false, inclusive boolean DEFAULT false)\n RETURNS text\n LANGUAGE plpgsql\n SET transform_null_equals TO 'true'\nAS $function$\nDECLARE\n ltop text := '<';\n gtop text := '>';\n dir text;\n sort record;\n orfilter text := '';\n orfilters text[] := '{}'::text[];\n andfilters text[] := '{}'::text[];\n output text;\n token_where text;\nBEGIN\n IF _sortby IS NULL OR _sortby = '[]'::jsonb THEN\n _sortby := '[{\"field\":\"datetime\",\"direction\":\"desc\"}]'::jsonb;\n END IF;\n _sortby := _sortby || jsonb_build_object('field','id','direction',_sortby->0->>'direction');\n RAISE NOTICE 'Getting Token Filter. % %', _sortby, token_item;\n IF inclusive THEN\n orfilters := orfilters || format('( id=%L AND collection=%L )' , token_item.id, token_item.collection);\n END IF;\n\n FOR sort IN\n WITH s1 AS (\n SELECT\n _row,\n (queryable(value->>'field')).expression as _field,\n (value->>'field' = 'id') as _isid,\n get_sort_dir(value) as _dir\n FROM jsonb_array_elements(_sortby)\n WITH ORDINALITY AS t(value, _row)\n )\n SELECT\n _row,\n _field,\n _dir,\n get_token_val_str(_field, token_item) as _val\n FROM s1\n WHERE _row <= (SELECT min(_row) FROM s1 WHERE _isid)\n LOOP\n orfilter := NULL;\n RAISE NOTICE 'SORT: %', sort;\n IF sort._val IS NOT NULL AND ((prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC')) THEN\n orfilter := format($f$(\n (%s %s %s) OR (%s IS NULL)\n )$f$,\n sort._field,\n ltop,\n sort._val,\n sort._val\n );\n ELSIF sort._val IS NULL AND ((prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC')) THEN\n RAISE NOTICE '< but null';\n orfilter := format('%s IS NOT NULL', sort._field);\n ELSIF sort._val IS NULL THEN\n RAISE NOTICE '> but null';\n ELSE\n orfilter := format($f$(\n (%s %s %s) OR (%s IS NULL)\n )$f$,\n sort._field,\n gtop,\n sort._val,\n sort._field\n );\n END IF;\n RAISE NOTICE 'ORFILTER: %', orfilter;\n\n IF orfilter IS NOT NULL THEN\n IF sort._row = 1 THEN\n orfilters := orfilters || orfilter;\n ELSE\n orfilters := orfilters || format('(%s AND %s)', array_to_string(andfilters, ' AND '), orfilter);\n END IF;\n END IF;\n IF sort._val IS NOT NULL THEN\n andfilters := andfilters || format('%s = %s', sort._field, sort._val);\n ELSE\n andfilters := andfilters || format('%s IS NULL', sort._field);\n END IF;\n END LOOP;\n\n output := array_to_string(orfilters, ' OR ');\n\n token_where := concat('(',coalesce(output,'true'),')');\n IF trim(token_where) = '' THEN\n token_where := NULL;\n END IF;\n RAISE NOTICE 'TOKEN_WHERE: %',token_where;\n RETURN token_where;\n END;\n$function$\n;\n\ncreate materialized view \"pgstac\".\"partition_steps\" as SELECT partitions_view.partition AS name,\n date_trunc('month'::text, lower(partitions_view.partition_dtrange)) AS sdate,\n (date_trunc('month'::text, upper(partitions_view.partition_dtrange)) + '1 mon'::interval) AS edate\n FROM pgstac.partitions_view\n WHERE ((partitions_view.partition_dtrange IS NOT NULL) AND (partitions_view.partition_dtrange <> 'empty'::tstzrange))\n ORDER BY partitions_view.dtrange;\n\n\ncreate materialized view \"pgstac\".\"partitions\" as SELECT partitions_view.partition,\n partitions_view.collection,\n partitions_view.level,\n partitions_view.reltuples,\n partitions_view.relhastriggers,\n partitions_view.partition_dtrange,\n partitions_view.constraint_dtrange,\n partitions_view.constraint_edtrange,\n partitions_view.dtrange,\n partitions_view.edtrange,\n partitions_view.spatial,\n partitions_view.last_updated\n FROM pgstac.partitions_view;\n\n\nCREATE OR REPLACE FUNCTION pgstac.repartition(_collection text, _partition_trunc text, triggered boolean DEFAULT false)\n RETURNS text\n LANGUAGE plpgsql\n SECURITY DEFINER\nAS $function$\nDECLARE\n c RECORD;\n q text;\n from_trunc text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=_collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', _collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n IF triggered THEN\n RAISE NOTICE 'Converting % to % partitioning via Trigger', _collection, _partition_trunc;\n ELSE\n RAISE NOTICE 'Converting % from using % to % partitioning', _collection, c.partition_trunc, _partition_trunc;\n IF c.partition_trunc IS NOT DISTINCT FROM _partition_trunc THEN\n RAISE NOTICE 'Collection % already set to use partition by %', _collection, _partition_trunc;\n RETURN _collection;\n END IF;\n END IF;\n\n IF EXISTS (SELECT 1 FROM partitions_view WHERE collection=_collection LIMIT 1) THEN\n EXECUTE format(\n $q$\n CREATE TEMP TABLE changepartitionstaging ON COMMIT DROP AS SELECT * FROM %I;\n DROP TABLE IF EXISTS %I CASCADE;\n WITH p AS (\n SELECT\n collection,\n CASE WHEN %L IS NULL THEN '-infinity'::timestamptz\n ELSE date_trunc(%L, datetime)\n END as d,\n tstzrange(min(datetime),max(datetime),'[]') as dtrange,\n tstzrange(min(datetime),max(datetime),'[]') as edtrange\n FROM changepartitionstaging\n GROUP BY 1,2\n ) SELECT check_partition(collection, dtrange, edtrange) FROM p;\n INSERT INTO items SELECT * FROM changepartitionstaging;\n DROP TABLE changepartitionstaging;\n $q$,\n concat('_items_', c.key),\n concat('_items_', c.key),\n c.partition_trunc,\n c.partition_trunc\n );\n END IF;\n RETURN _collection;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.search(_search jsonb DEFAULT '{}'::jsonb)\n RETURNS jsonb\n LANGUAGE plpgsql\nAS $function$\nDECLARE\n searches searches%ROWTYPE;\n _where text;\n orderby text;\n search_where search_wheres%ROWTYPE;\n total_count bigint;\n token record;\n token_prev boolean;\n token_item items%ROWTYPE;\n token_where text;\n full_where text;\n timer timestamptz := clock_timestamp();\n hydrate bool := NOT (_search->'conf'->>'nohydrate' IS NOT NULL AND (_search->'conf'->>'nohydrate')::boolean = true);\n prev text;\n next text;\n context jsonb;\n collection jsonb;\n out_records jsonb;\n out_len int;\n _limit int := coalesce((_search->>'limit')::int, 10);\n _querylimit int;\n _fields jsonb := coalesce(_search->'fields', '{}'::jsonb);\n has_prev boolean := FALSE;\n has_next boolean := FALSE;\nBEGIN\n searches := search_query(_search);\n _where := searches._where;\n orderby := searches.orderby;\n search_where := where_stats(_where);\n total_count := coalesce(search_where.total_count, search_where.estimated_count);\n RAISE NOTICE 'SEARCH:TOKEN: %', _search->>'token';\n token := get_token_record(_search->>'token');\n RAISE NOTICE '***TOKEN: %', token;\n _querylimit := _limit + 1;\n IF token IS NOT NULL THEN\n token_prev := token.prev;\n token_item := token.item;\n token_where := get_token_filter(_search->'sortby', token_item, token_prev, FALSE);\n RAISE LOG 'TOKEN_WHERE: %', token_where;\n IF token_prev THEN -- if we are using a prev token, we know has_next is true\n RAISE LOG 'There is a previous token, so automatically setting has_next to true';\n has_next := TRUE;\n orderby := sort_sqlorderby(_search, TRUE);\n ELSE\n RAISE LOG 'There is a next token, so automatically setting has_prev to true';\n has_prev := TRUE;\n\n END IF;\n ELSE -- if there was no token, we know there is no prev\n RAISE LOG 'There is no token, so we know there is no prev. setting has_prev to false';\n has_prev := FALSE;\n END IF;\n\n full_where := concat_ws(' AND ', _where, token_where);\n RAISE NOTICE 'FULL WHERE CLAUSE: %', full_where;\n RAISE NOTICE 'Time to get counts and build query %', age_ms(timer);\n timer := clock_timestamp();\n\n IF hydrate THEN\n RAISE NOTICE 'Getting hydrated data.';\n ELSE\n RAISE NOTICE 'Getting non-hydrated data.';\n END IF;\n RAISE NOTICE 'CACHE SET TO %', get_setting_bool('format_cache');\n RAISE NOTICE 'Time to set hydration/formatting %', age_ms(timer);\n timer := clock_timestamp();\n SELECT jsonb_agg(format_item(i, _fields, hydrate)) INTO out_records\n FROM search_rows(\n full_where,\n orderby,\n search_where.partitions,\n _querylimit\n ) as i;\n\n RAISE NOTICE 'Time to fetch rows %', age_ms(timer);\n timer := clock_timestamp();\n\n\n IF token_prev THEN\n out_records := flip_jsonb_array(out_records);\n END IF;\n\n RAISE NOTICE 'Query returned % records.', jsonb_array_length(out_records);\n RAISE LOG 'TOKEN: % %', token_item.id, token_item.collection;\n RAISE LOG 'RECORD_1: % %', out_records->0->>'id', out_records->0->>'collection';\n RAISE LOG 'RECORD-1: % %', out_records->-1->>'id', out_records->-1->>'collection';\n\n -- REMOVE records that were from our token\n IF out_records->0->>'id' = token_item.id AND out_records->0->>'collection' = token_item.collection THEN\n out_records := out_records - 0;\n ELSIF out_records->-1->>'id' = token_item.id AND out_records->-1->>'collection' = token_item.collection THEN\n out_records := out_records - -1;\n END IF;\n\n out_len := jsonb_array_length(out_records);\n\n IF out_len = _limit + 1 THEN\n IF token_prev THEN\n has_prev := TRUE;\n out_records := out_records - 0;\n ELSE\n has_next := TRUE;\n out_records := out_records - -1;\n END IF;\n END IF;\n\n IF has_next THEN\n next := concat(out_records->-1->>'collection', ':', out_records->-1->>'id');\n RAISE NOTICE 'HAS NEXT | %', next;\n END IF;\n\n IF has_prev THEN\n prev := concat(out_records->0->>'collection', ':', out_records->0->>'id');\n RAISE NOTICE 'HAS PREV | %', prev;\n END IF;\n\n RAISE NOTICE 'Time to get prev/next %', age_ms(timer);\n timer := clock_timestamp();\n\n IF context(_search->'conf') != 'off' THEN\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'matched', total_count,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\n ELSE\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\n END IF;\n\n collection := jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb),\n 'next', next,\n 'prev', prev,\n 'context', context\n );\n\n RAISE NOTICE 'Time to build final json %', age_ms(timer);\n timer := clock_timestamp();\n\n RAISE NOTICE 'Total Time: %', age_ms(current_timestamp);\n RAISE NOTICE 'RETURNING % records. NEXT: %. PREV: %', collection->'context'->>'returned', collection->>'next', collection->>'prev';\n RETURN collection;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.search_rows(_where text DEFAULT 'TRUE'::text, _orderby text DEFAULT 'datetime DESC, id DESC'::text, partitions text[] DEFAULT NULL::text[], _limit integer DEFAULT 10)\n RETURNS SETOF pgstac.items\n LANGUAGE plpgsql\n SET search_path TO 'pgstac', 'public'\nAS $function$\nDECLARE\n base_query text;\n query text;\n sdate timestamptz;\n edate timestamptz;\n n int;\n records_left int := _limit;\n timer timestamptz := clock_timestamp();\n full_timer timestamptz := clock_timestamp();\nBEGIN\nIF _where IS NULL OR trim(_where) = '' THEN\n _where = ' TRUE ';\nEND IF;\nRAISE NOTICE 'Getting chunks for % %', _where, _orderby;\n\nbase_query := $q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n LIMIT %L\n$q$;\n\nIF _orderby ILIKE 'datetime d%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 DESC LOOP\n RAISE NOTICE 'Running Query for % to %. %', sdate, edate, age_ms(full_timer);\n query := format(\n base_query,\n sdate,\n edate,\n _where,\n _orderby,\n records_left\n );\n RAISE LOG 'QUERY: %', query;\n timer := clock_timestamp();\n RETURN QUERY EXECUTE query;\n\n GET DIAGNOSTICS n = ROW_COUNT;\n records_left := records_left - n;\n RAISE NOTICE 'Returned %/% Rows From % to %. % to go. Time: %ms', n, _limit, sdate, edate, records_left, age_ms(timer);\n timer := clock_timestamp();\n IF records_left <= 0 THEN\n RAISE NOTICE 'SEARCH_ROWS TOOK %ms', age_ms(full_timer);\n RETURN;\n END IF;\n END LOOP;\nELSIF _orderby ILIKE 'datetime a%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 ASC LOOP\n RAISE NOTICE 'Running Query for % to %. %', sdate, edate, age_ms(full_timer);\n query := format(\n base_query,\n sdate,\n edate,\n _where,\n _orderby,\n records_left\n );\n RAISE LOG 'QUERY: %', query;\n timer := clock_timestamp();\n RETURN QUERY EXECUTE query;\n\n GET DIAGNOSTICS n = ROW_COUNT;\n records_left := records_left - n;\n RAISE NOTICE 'Returned %/% Rows From % to %. % to go. Time: %ms', n, _limit, sdate, edate, records_left, age_ms(timer);\n timer := clock_timestamp();\n IF records_left <= 0 THEN\n RAISE NOTICE 'SEARCH_ROWS TOOK %ms', age_ms(full_timer);\n RETURN;\n END IF;\n END LOOP;\nELSE\n query := format($q$\n SELECT * FROM items\n WHERE %s\n ORDER BY %s\n LIMIT %L\n $q$, _where, _orderby, _limit\n );\n RAISE LOG 'QUERY: %', query;\n timer := clock_timestamp();\n RETURN QUERY EXECUTE query;\n RAISE NOTICE 'FULL QUERY TOOK %ms', age_ms(timer);\nEND IF;\nRAISE NOTICE 'SEARCH_ROWS TOOK %ms', age_ms(full_timer);\nRETURN;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.update_partition_stats(_partition text, istrigger boolean DEFAULT false)\n RETURNS void\n LANGUAGE plpgsql\n STRICT\nAS $function$\nDECLARE\n dtrange tstzrange;\n edtrange tstzrange;\n cdtrange tstzrange;\n cedtrange tstzrange;\n extent geometry;\n collection text;\nBEGIN\n RAISE NOTICE 'Updating stats for %.', _partition;\n EXECUTE format(\n $q$\n SELECT\n tstzrange(min(datetime), max(datetime),'[]'),\n tstzrange(min(end_datetime), max(end_datetime), '[]')\n FROM %I\n $q$,\n _partition\n ) INTO dtrange, edtrange;\n extent := st_estimatedextent('pgstac', _partition, 'geometry');\n INSERT INTO partition_stats (partition, dtrange, edtrange, spatial, last_updated)\n SELECT _partition, dtrange, edtrange, extent, now()\n ON CONFLICT (partition) DO\n UPDATE SET\n dtrange=EXCLUDED.dtrange,\n edtrange=EXCLUDED.edtrange,\n spatial=EXCLUDED.spatial,\n last_updated=EXCLUDED.last_updated\n ;\n\n SELECT\n constraint_dtrange, constraint_edtrange, pv.collection\n INTO cdtrange, cedtrange, collection\n FROM partitions_view pv WHERE partition = _partition;\n REFRESH MATERIALIZED VIEW partitions;\n REFRESH MATERIALIZED VIEW partition_steps;\n\n\n RAISE NOTICE 'Checking if we need to modify constraints.';\n IF\n (cdtrange IS DISTINCT FROM dtrange OR edtrange IS DISTINCT FROM cedtrange)\n AND NOT istrigger\n THEN\n RAISE NOTICE 'Modifying Constraints';\n RAISE NOTICE 'Existing % %', cdtrange, cedtrange;\n RAISE NOTICE 'New % %', dtrange, edtrange;\n PERFORM drop_table_constraints(_partition);\n PERFORM create_table_constraints(_partition, dtrange, edtrange);\n END IF;\n RAISE NOTICE 'Checking if we need to update collection extents.';\n IF get_setting_bool('update_collection_extent') THEN\n RAISE NOTICE 'updating collection extent for %', collection;\n PERFORM run_or_queue(format($q$\n UPDATE collections\n SET content = jsonb_set_lax(\n content,\n '{extent}'::text[],\n collection_extent(%L, FALSE),\n true,\n 'use_json_null'\n ) WHERE id=%L\n ;\n $q$, collection, collection));\n ELSE\n RAISE NOTICE 'Not updating collection extent for %', collection;\n END IF;\n\nEND;\n$function$\n;\n\nCREATE UNIQUE INDEX partitions_partition_idx ON pgstac.partitions USING btree (partition);\n\n\n\n-- END migra calculated SQL\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('id', '{\"title\": \"Item ID\",\"description\": \"Item identifier\",\"$ref\": \"https://schemas.stacspec.org/v1.0.0/item-spec/json-schema/item.json#/definitions/core/allOf/2/properties/id\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('geometry', '{\"title\": \"Item Geometry\",\"description\": \"Item Geometry\",\"$ref\": \"https://geojson.org/schema/Feature.json\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('datetime','{\"description\": \"Datetime\",\"type\": \"string\",\"title\": \"Acquired\",\"format\": \"date-time\",\"pattern\": \"(\\\\+00:00|Z)$\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('eo:cloud_cover','{\"$ref\": \"https://stac-extensions.github.io/eo/v1.0.0/schema.json#/definitions/fieldsproperties/eo:cloud_cover\"}','to_int','BTREE');\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDELETE FROM queryables a USING queryables b\n WHERE a.name = b.name AND a.collection_ids IS NOT DISTINCT FROM b.collection_ids AND a.id > b.id;\n\n\nINSERT INTO pgstac_settings (name, value) VALUES\n ('context', 'off'),\n ('context_estimated_count', '100000'),\n ('context_estimated_cost', '100000'),\n ('context_stats_ttl', '1 day'),\n ('default_filter_lang', 'cql2-json'),\n ('additional_properties', 'true'),\n ('use_queue', 'false'),\n ('queue_timeout', '10 minutes'),\n ('update_collection_extent', 'false'),\n ('format_cache', 'false')\nON CONFLICT DO NOTHING\n;\n\nALTER FUNCTION to_text COST 5000;\nALTER FUNCTION to_float COST 5000;\nALTER FUNCTION to_int COST 5000;\nALTER FUNCTION to_tstz COST 5000;\nALTER FUNCTION to_text_array COST 5000;\n\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nGRANT ALL ON SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac to pgstac_admin;\n\n-- pgstac_read role limited to using function apis\nGRANT EXECUTE ON FUNCTION search TO pgstac_read;\nGRANT EXECUTE ON FUNCTION search_query TO pgstac_read;\nGRANT EXECUTE ON FUNCTION item_by_id TO pgstac_read;\nGRANT EXECUTE ON FUNCTION get_item TO pgstac_read;\nGRANT SELECT ON ALL TABLES IN SCHEMA pgstac TO pgstac_read;\n\n\nGRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_ingest;\nGRANT USAGE ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_ingest;\n\nSELECT update_partition_stats_q(partition) FROM partitions_view;\nSELECT set_version('0.7.5');\n" }, { "path": "src/pgstac/migrations/pgstac.0.7.4.sql", "content": "RESET ROLE;\nDO $$\nDECLARE\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='postgis') THEN\n CREATE EXTENSION IF NOT EXISTS postgis;\n END IF;\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='btree_gist') THEN\n CREATE EXTENSION IF NOT EXISTS btree_gist;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_admin;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_read;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_ingest;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n\nGRANT pgstac_admin TO current_user;\n\n-- Function to make sure pgstac_admin is the owner of items\nCREATE OR REPLACE FUNCTION pgstac_admin_owns() RETURNS VOID AS $$\nDECLARE\n f RECORD;\nBEGIN\n FOR f IN (\n SELECT\n concat(\n oid::regproc::text,\n '(',\n coalesce(pg_get_function_identity_arguments(oid),''),\n ')'\n ) AS name,\n CASE prokind WHEN 'f' THEN 'FUNCTION' WHEN 'p' THEN 'PROCEDURE' WHEN 'a' THEN 'AGGREGATE' END as typ\n FROM pg_proc\n WHERE\n pronamespace=to_regnamespace('pgstac')\n AND proowner != to_regrole('pgstac_admin')\n AND proname NOT LIKE 'pg_stat%'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n FOR f IN (\n SELECT\n oid::regclass::text as name,\n CASE relkind\n WHEN 'i' THEN 'INDEX'\n WHEN 'I' THEN 'INDEX'\n WHEN 'p' THEN 'TABLE'\n WHEN 'r' THEN 'TABLE'\n WHEN 'v' THEN 'VIEW'\n WHEN 'S' THEN 'SEQUENCE'\n ELSE NULL\n END as typ\n FROM pg_class\n WHERE relnamespace=to_regnamespace('pgstac') and relowner != to_regrole('pgstac_admin') AND relkind IN ('r','p','v','S') AND relname NOT LIKE 'pg_stat'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\nSELECT pgstac_admin_owns();\n\nCREATE SCHEMA IF NOT EXISTS pgstac AUTHORIZATION pgstac_admin;\n\nGRANT ALL ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_admin;\n\nALTER ROLE pgstac_admin SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_read SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_ingest SET SEARCH_PATH TO pgstac, public;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\n\nGRANT pgstac_read TO pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\n\nSET ROLE pgstac_admin;\n\nSET SEARCH_PATH TO pgstac, public;\n\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS analyze_items;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS validate_constraints;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\nCREATE OR REPLACE FUNCTION to_int(jsonb) RETURNS int AS $$\n SELECT floor(($1->>0)::float)::int;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000;\n\nCREATE OR REPLACE FUNCTION to_float(jsonb) RETURNS float AS $$\n SELECT ($1->>0)::float;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000;\n\nCREATE OR REPLACE FUNCTION to_tstz(jsonb) RETURNS timestamptz AS $$\n SELECT ($1->>0)::timestamptz;\n$$ LANGUAGE SQL IMMUTABLE STRICT SET TIME ZONE 'UTC' COST 5000;\n\n\nCREATE OR REPLACE FUNCTION to_text(jsonb) RETURNS text AS $$\n SELECT CASE WHEN jsonb_typeof($1) IN ('array','object') THEN $1::text ELSE $1->>0 END;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000;\n\nCREATE OR REPLACE FUNCTION to_text_array(jsonb) RETURNS text[] AS $$\n SELECT\n CASE jsonb_typeof($1)\n WHEN 'array' THEN ARRAY(SELECT jsonb_array_elements_text($1))\n ELSE ARRAY[$1->>0]\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000;\n\nCREATE OR REPLACE FUNCTION bbox_geom(_bbox jsonb) RETURNS geometry AS $$\nSELECT CASE jsonb_array_length(_bbox)\n WHEN 4 THEN\n ST_SetSRID(ST_MakeEnvelope(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float,\n (_bbox->>3)::float\n ),4326)\n WHEN 6 THEN\n ST_SetSRID(ST_3DMakeBox(\n ST_MakePoint(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float\n ),\n ST_MakePoint(\n (_bbox->>3)::float,\n (_bbox->>4)::float,\n (_bbox->>5)::float\n )\n ),4326)\n ELSE null END;\n;\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION geom_bbox(_geom geometry) RETURNS jsonb AS $$\n SELECT jsonb_build_array(\n st_xmin(_geom),\n st_ymin(_geom),\n st_xmax(_geom),\n st_ymax(_geom)\n );\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION flip_jsonb_array(j jsonb) RETURNS jsonb AS $$\n SELECT jsonb_agg(value) FROM (SELECT value FROM jsonb_array_elements(j) WITH ORDINALITY ORDER BY ordinality DESC) as t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION explode_dotpaths(j jsonb) RETURNS SETOF text[] AS $$\n SELECT string_to_array(p, '.') as e FROM jsonb_array_elements_text(j) p;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION explode_dotpaths_recurse(IN j jsonb) RETURNS SETOF text[] AS $$\n WITH RECURSIVE t AS (\n SELECT e FROM explode_dotpaths(j) e\n UNION ALL\n SELECT e[1:cardinality(e)-1]\n FROM t\n WHERE cardinality(e)>1\n ) SELECT e FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION jsonb_set_nested(j jsonb, path text[], val jsonb) RETURNS jsonb AS $$\nDECLARE\nBEGIN\n IF cardinality(path) > 1 THEN\n FOR i IN 1..(cardinality(path)-1) LOOP\n IF j #> path[:i] IS NULL THEN\n j := jsonb_set_lax(j, path[:i], '{}', TRUE);\n END IF;\n END LOOP;\n END IF;\n RETURN jsonb_set_lax(j, path, val, true);\n\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\n\n\nCREATE OR REPLACE FUNCTION jsonb_include(j jsonb, f jsonb) RETURNS jsonb AS $$\nDECLARE\n includes jsonb := f-> 'include';\n outj jsonb := '{}'::jsonb;\n path text[];\nBEGIN\n IF\n includes IS NULL\n OR jsonb_array_length(includes) = 0\n THEN\n RETURN j;\n ELSE\n includes := includes || '[\"id\",\"collection\"]'::jsonb;\n FOR path IN SELECT explode_dotpaths(includes) LOOP\n outj := jsonb_set_nested(outj, path, j #> path);\n END LOOP;\n END IF;\n RETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_exclude(j jsonb, f jsonb) RETURNS jsonb AS $$\nDECLARE\n excludes jsonb := f-> 'exclude';\n outj jsonb := j;\n path text[];\nBEGIN\n IF\n excludes IS NULL\n OR jsonb_array_length(excludes) = 0\n THEN\n RETURN j;\n ELSE\n FOR path IN SELECT explode_dotpaths(excludes) LOOP\n outj := outj #- path;\n END LOOP;\n END IF;\n RETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_fields(j jsonb, f jsonb DEFAULT '{\"fields\":[]}') RETURNS jsonb AS $$\n SELECT jsonb_exclude(jsonb_include(j, f), f);\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION merge_jsonb(_a jsonb, _b jsonb) RETURNS jsonb AS $$\n SELECT\n CASE\n WHEN _a = '\"𒍟※\"'::jsonb THEN NULL\n WHEN _a IS NULL OR jsonb_typeof(_a) = 'null' THEN _b\n WHEN jsonb_typeof(_a) = 'object' AND jsonb_typeof(_b) = 'object' THEN\n (\n SELECT\n jsonb_strip_nulls(\n jsonb_object_agg(\n key,\n merge_jsonb(a.value, b.value)\n )\n )\n FROM\n jsonb_each(coalesce(_a,'{}'::jsonb)) as a\n FULL JOIN\n jsonb_each(coalesce(_b,'{}'::jsonb)) as b\n USING (key)\n )\n WHEN\n jsonb_typeof(_a) = 'array'\n AND jsonb_typeof(_b) = 'array'\n AND jsonb_array_length(_a) = jsonb_array_length(_b)\n THEN\n (\n SELECT jsonb_agg(m) FROM\n ( SELECT\n merge_jsonb(\n jsonb_array_elements(_a),\n jsonb_array_elements(_b)\n ) as m\n ) as l\n )\n ELSE _a\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION strip_jsonb(_a jsonb, _b jsonb) RETURNS jsonb AS $$\n SELECT\n CASE\n\n WHEN (_a IS NULL OR jsonb_typeof(_a) = 'null') AND _b IS NOT NULL AND jsonb_typeof(_b) != 'null' THEN '\"𒍟※\"'::jsonb\n WHEN _b IS NULL OR jsonb_typeof(_a) = 'null' THEN _a\n WHEN _a = _b AND jsonb_typeof(_a) = 'object' THEN '{}'::jsonb\n WHEN _a = _b THEN NULL\n WHEN jsonb_typeof(_a) = 'object' AND jsonb_typeof(_b) = 'object' THEN\n (\n SELECT\n jsonb_strip_nulls(\n jsonb_object_agg(\n key,\n strip_jsonb(a.value, b.value)\n )\n )\n FROM\n jsonb_each(_a) as a\n FULL JOIN\n jsonb_each(_b) as b\n USING (key)\n )\n WHEN\n jsonb_typeof(_a) = 'array'\n AND jsonb_typeof(_b) = 'array'\n AND jsonb_array_length(_a) = jsonb_array_length(_b)\n THEN\n (\n SELECT jsonb_agg(m) FROM\n ( SELECT\n strip_jsonb(\n jsonb_array_elements(_a),\n jsonb_array_elements(_b)\n ) as m\n ) as l\n )\n ELSE _a\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION nullif_jsonbnullempty(j jsonb) RETURNS jsonb AS $$\n SELECT nullif(nullif(nullif(j,'null'::jsonb),'{}'::jsonb),'[]'::jsonb);\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION jsonb_array_unique(j jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(jsonb_agg(DISTINCT a)) v FROM jsonb_array_elements(j) a;\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_concat_ignorenull(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT coalesce(a,'[]'::jsonb) || coalesce(b,'[]'::jsonb);\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_least(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(least(nullif_jsonbnullempty(a), nullif_jsonbnullempty(b)));\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_greatest(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(greatest(a, b));\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION first_notnull_sfunc(anyelement, anyelement) RETURNS anyelement AS $$\n SELECT COALESCE($1,$2);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE AGGREGATE first_notnull(anyelement)(\n SFUNC = first_notnull_sfunc,\n STYPE = anyelement\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_array_unique_merge(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_concat_ignorenull,\n FINALFUNC = jsonb_array_unique\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_min(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_least\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_max(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_greatest\n);\n\nCREATE TABLE IF NOT EXISTS migrations (\n version text PRIMARY KEY,\n datetime timestamptz DEFAULT clock_timestamp() NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION get_version() RETURNS text AS $$\n SELECT version FROM pgstac.migrations ORDER BY datetime DESC, version DESC LIMIT 1;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION set_version(text) RETURNS text AS $$\n INSERT INTO pgstac.migrations (version) VALUES ($1)\n ON CONFLICT DO NOTHING\n RETURNING version;\n$$ LANGUAGE SQL;\n\n\nCREATE TABLE IF NOT EXISTS pgstac_settings (\n name text PRIMARY KEY,\n value text NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION table_empty(text) RETURNS boolean AS $$\nDECLARE\n retval boolean;\nBEGIN\n EXECUTE format($q$\n SELECT NOT EXISTS (SELECT 1 FROM %I LIMIT 1)\n $q$,\n $1\n ) INTO retval;\n RETURN retval;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION get_setting(IN _setting text, IN conf jsonb DEFAULT NULL) RETURNS text AS $$\nSELECT COALESCE(\n nullif(conf->>_setting, ''),\n nullif(current_setting(concat('pgstac.',_setting), TRUE),''),\n nullif((SELECT value FROM pgstac.pgstac_settings WHERE name=_setting),'')\n);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_setting_bool(IN _setting text, IN conf jsonb DEFAULT NULL) RETURNS boolean AS $$\nSELECT COALESCE(\n conf->>_setting,\n current_setting(concat('pgstac.',_setting), TRUE),\n (SELECT value FROM pgstac.pgstac_settings WHERE name=_setting),\n 'FALSE'\n)::boolean;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context(conf jsonb DEFAULT NULL) RETURNS text AS $$\n SELECT pgstac.get_setting('context', conf);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context_estimated_count(conf jsonb DEFAULT NULL) RETURNS int AS $$\n SELECT pgstac.get_setting('context_estimated_count', conf)::int;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_estimated_cost();\nCREATE OR REPLACE FUNCTION context_estimated_cost(conf jsonb DEFAULT NULL) RETURNS float AS $$\n SELECT pgstac.get_setting('context_estimated_cost', conf)::float;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_stats_ttl();\nCREATE OR REPLACE FUNCTION context_stats_ttl(conf jsonb DEFAULT NULL) RETURNS interval AS $$\n SELECT pgstac.get_setting('context_stats_ttl', conf)::interval;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION t2s(text) RETURNS text AS $$\n SELECT extract(epoch FROM $1::interval)::text || ' s';\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE STRICT;\n\nCREATE OR REPLACE FUNCTION age_ms(a timestamptz, b timestamptz DEFAULT clock_timestamp()) RETURNS float AS $$\n SELECT abs(extract(epoch from age(a,b)) * 1000);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION queue_timeout() RETURNS interval AS $$\n SELECT t2s(coalesce(\n get_setting('queue_timeout'),\n '1h'\n ))::interval;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION notice(VARIADIC text[]) RETURNS boolean AS $$\nDECLARE\ndebug boolean := current_setting('pgstac.debug', true);\nBEGIN\n IF debug THEN\n RAISE NOTICE 'NOTICE FROM FUNC: % >>>>> %', concat_ws(' | ', $1), clock_timestamp();\n RETURN TRUE;\n END IF;\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION empty_arr(ANYARRAY) RETURNS BOOLEAN AS $$\nSELECT CASE\n WHEN $1 IS NULL THEN TRUE\n WHEN cardinality($1)<1 THEN TRUE\nELSE FALSE\nEND;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_intersection(_a ANYARRAY, _b ANYARRAY) RETURNS ANYARRAY AS $$\n SELECT ARRAY ( SELECT unnest(_a) INTERSECT SELECT UNNEST(_b) );\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION array_map_ident(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_ident(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_map_literal(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_literal(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION array_reverse(anyarray) RETURNS anyarray AS $$\nSELECT ARRAY(\n SELECT $1[i]\n FROM generate_subscripts($1,1) AS s(i)\n ORDER BY i DESC\n);\n$$ LANGUAGE SQL STRICT IMMUTABLE;\n\nDROP TABLE IF EXISTS query_queue;\nCREATE TABLE query_queue (\n query text PRIMARY KEY,\n added timestamptz DEFAULT now()\n);\n\nDROP TABLE IF EXISTS query_queue_history;\nCREATE TABLE query_queue_history(\n query text,\n added timestamptz NOT NULL,\n finished timestamptz NOT NULL DEFAULT now(),\n error text\n);\n\nCREATE OR REPLACE PROCEDURE run_queued_queries() AS $$\nDECLARE\n qitem query_queue%ROWTYPE;\n timeout_ts timestamptz;\n error text;\n cnt int := 0;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n DELETE FROM query_queue WHERE query = (SELECT query FROM query_queue ORDER BY added DESC LIMIT 1 FOR UPDATE SKIP LOCKED) RETURNING * INTO qitem;\n IF NOT FOUND THEN\n EXIT;\n END IF;\n cnt := cnt + 1;\n BEGIN\n RAISE NOTICE 'RUNNING QUERY: %', qitem.query;\n EXECUTE qitem.query;\n EXCEPTION WHEN others THEN\n error := format('%s | %s', SQLERRM, SQLSTATE);\n END;\n INSERT INTO query_queue_history (query, added, finished, error)\n VALUES (qitem.query, qitem.added, clock_timestamp(), error);\n COMMIT;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION run_queued_queries_intransaction() RETURNS int AS $$\nDECLARE\n qitem query_queue%ROWTYPE;\n timeout_ts timestamptz;\n error text;\n cnt int := 0;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n DELETE FROM query_queue WHERE query = (SELECT query FROM query_queue ORDER BY added DESC LIMIT 1 FOR UPDATE SKIP LOCKED) RETURNING * INTO qitem;\n IF NOT FOUND THEN\n RETURN cnt;\n END IF;\n cnt := cnt + 1;\n BEGIN\n qitem.query := regexp_replace(qitem.query, 'CONCURRENTLY', '');\n RAISE NOTICE 'RUNNING QUERY: %', qitem.query;\n\n EXECUTE qitem.query;\n EXCEPTION WHEN others THEN\n error := format('%s | %s', SQLERRM, SQLSTATE);\n RAISE WARNING '%', error;\n END;\n INSERT INTO query_queue_history (query, added, finished, error)\n VALUES (qitem.query, qitem.added, clock_timestamp(), error);\n END LOOP;\n RETURN cnt;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION run_or_queue(query text) RETURNS VOID AS $$\nDECLARE\n use_queue text := COALESCE(get_setting('use_queue'), 'FALSE')::boolean;\nBEGIN\n IF get_setting_bool('debug') THEN\n RAISE NOTICE '%', query;\n END IF;\n IF use_queue THEN\n INSERT INTO query_queue (query) VALUES (query) ON CONFLICT DO NOTHING;\n ELSE\n EXECUTE query;\n END IF;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n\nDROP FUNCTION IF EXISTS check_pgstac_settings;\nCREATE OR REPLACE FUNCTION check_pgstac_settings(_sysmem text DEFAULT NULL) RETURNS VOID AS $$\nDECLARE\n settingval text;\n sysmem bigint := pg_size_bytes(_sysmem);\n effective_cache_size bigint := pg_size_bytes(current_setting('effective_cache_size', TRUE));\n shared_buffers bigint := pg_size_bytes(current_setting('shared_buffers', TRUE));\n work_mem bigint := pg_size_bytes(current_setting('work_mem', TRUE));\n max_connections int := current_setting('max_connections', TRUE);\n maintenance_work_mem bigint := pg_size_bytes(current_setting('maintenance_work_mem', TRUE));\n seq_page_cost float := current_setting('seq_page_cost', TRUE);\n random_page_cost float := current_setting('random_page_cost', TRUE);\n temp_buffers bigint := pg_size_bytes(current_setting('temp_buffers', TRUE));\n r record;\nBEGIN\n IF _sysmem IS NULL THEN\n RAISE NOTICE 'Call function with the size of your system memory `SELECT check_pgstac_settings(''4GB'')` to get pg system setting recommendations.';\n ELSE\n IF effective_cache_size < (sysmem * 0.5) THEN\n RAISE WARNING 'effective_cache_size of % is set low for a system with %. Recomended value between % and %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.5), pg_size_pretty(sysmem * 0.75);\n ELSIF effective_cache_size > (sysmem * 0.75) THEN\n RAISE WARNING 'effective_cache_size of % is set high for a system with %. Recomended value between % and %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.5), pg_size_pretty(sysmem * 0.75);\n ELSE\n RAISE NOTICE 'effective_cache_size of % is set appropriately for a system with %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem);\n END IF;\n\n IF shared_buffers < (sysmem * 0.2) THEN\n RAISE WARNING 'shared_buffers of % is set low for a system with %. Recomended value between % and %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.2), pg_size_pretty(sysmem * 0.3);\n ELSIF shared_buffers > (sysmem * 0.3) THEN\n RAISE WARNING 'shared_buffers of % is set high for a system with %. Recomended value between % and %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.2), pg_size_pretty(sysmem * 0.3);\n ELSE\n RAISE NOTICE 'shared_buffers of % is set appropriately for a system with %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem);\n END IF;\n shared_buffers = sysmem * 0.3;\n IF maintenance_work_mem < (sysmem * 0.2) THEN\n RAISE WARNING 'maintenance_work_mem of % is set low for shared_buffers of %. Recomended value between % and %', pg_size_pretty(maintenance_work_mem), pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers * 0.2), pg_size_pretty(shared_buffers * 0.3);\n ELSIF maintenance_work_mem > (shared_buffers * 0.3) THEN\n RAISE WARNING 'maintenance_work_mem of % is set high for shared_buffers of %. Recomended value between % and %', pg_size_pretty(maintenance_work_mem), pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers * 0.2), pg_size_pretty(shared_buffers * 0.3);\n ELSE\n RAISE NOTICE 'maintenance_work_mem of % is set appropriately for shared_buffers of %', pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers);\n END IF;\n\n IF work_mem * max_connections > shared_buffers THEN\n RAISE WARNING 'work_mem setting of % is set high for % max_connections please reduce work_mem to % or decrease max_connections to %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers/max_connections), floor(shared_buffers/work_mem);\n ELSIF work_mem * max_connections < (shared_buffers * 0.75) THEN\n RAISE WARNING 'work_mem setting of % is set low for % max_connections you may consider raising work_mem to % or increasing max_connections to %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers/max_connections), floor(shared_buffers/work_mem);\n ELSE\n RAISE NOTICE 'work_mem setting of % and max_connections of % are adequate for shared_buffers of %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers);\n END IF;\n\n IF random_page_cost / seq_page_cost != 1.1 THEN\n RAISE WARNING 'random_page_cost (%) /seq_page_cost (%) should be set to 1.1 for SSD. Change random_page_cost to %', random_page_cost, seq_page_cost, 1.1 * seq_page_cost;\n ELSE\n RAISE NOTICE 'random_page_cost and seq_page_cost set appropriately for SSD';\n END IF;\n\n IF temp_buffers < greatest(pg_size_bytes('128MB'),(maintenance_work_mem / 2)) THEN\n RAISE WARNING 'pgstac makes heavy use of temp tables, consider raising temp_buffers from % to %', pg_size_pretty(temp_buffers), greatest('128MB', pg_size_pretty((shared_buffers / 16)));\n END IF;\n END IF;\n\n RAISE NOTICE 'VALUES FOR PGSTAC VARIABLES';\n RAISE NOTICE 'These can be set either as GUC system variables or by setting in the pgstac_settings table.';\n\n FOR r IN SELECT name, get_setting(name) as setting, CASE WHEN current_setting(concat('pgstac.',name), TRUE) IS NOT NULL THEN concat('pgstac.',name, ' GUC') WHEN value IS NOT NULL THEN 'pgstac_settings table' ELSE 'Not Set' END as loc FROM pgstac_settings LOOP\n RAISE NOTICE '% is set to % from the %', r.name, r.setting, r.loc;\n END LOOP;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pg_cron';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider intalling pg_cron which can be used to automate tasks';\n ELSE\n RAISE NOTICE 'pg_cron % is installed', settingval;\n END IF;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pgstattuple';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider installing the pgstattuple extension which can be used to help maintain tables and indexes.';\n ELSE\n RAISE NOTICE 'pgstattuple % is installed', settingval;\n END IF;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pg_stat_statements';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider installing the pg_stat_statements extension which is very helpful for tracking the types of queries on the system';\n ELSE\n RAISE NOTICE 'pg_stat_statements % is installed', settingval;\n IF current_setting('pg_stat_statements.track_statements', TRUE) IS DISTINCT FROM 'all' THEN\n RAISE WARNING 'SET pg_stat_statements.track_statements TO ''all''; --In order to track statements within functions.';\n END IF;\n END IF;\n\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac, public SET CLIENT_MIN_MESSAGES TO NOTICE;\n/* looks for a geometry in a stac item first from geometry and falling back to bbox */\nCREATE OR REPLACE FUNCTION stac_geom(value jsonb) RETURNS geometry AS $$\nSELECT\n CASE\n WHEN value ? 'intersects' THEN\n ST_GeomFromGeoJSON(value->>'intersects')\n WHEN value ? 'geometry' THEN\n ST_GeomFromGeoJSON(value->>'geometry')\n WHEN value ? 'bbox' THEN\n pgstac.bbox_geom(value->'bbox')\n ELSE NULL\n END as geometry\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION stac_daterange(\n value jsonb\n) RETURNS tstzrange AS $$\nDECLARE\n props jsonb := value;\n dt timestamptz;\n edt timestamptz;\nBEGIN\n IF props ? 'properties' THEN\n props := props->'properties';\n END IF;\n IF\n props ? 'start_datetime'\n AND props->>'start_datetime' IS NOT NULL\n AND props ? 'end_datetime'\n AND props->>'end_datetime' IS NOT NULL\n THEN\n dt := props->>'start_datetime';\n edt := props->>'end_datetime';\n IF dt > edt THEN\n RAISE EXCEPTION 'start_datetime must be < end_datetime';\n END IF;\n ELSE\n dt := props->>'datetime';\n edt := props->>'datetime';\n END IF;\n IF dt is NULL OR edt IS NULL THEN\n RAISE NOTICE 'DT: %, EDT: %', dt, edt;\n RAISE EXCEPTION 'Either datetime (%) or both start_datetime (%) and end_datetime (%) must be set.', props->>'datetime',props->>'start_datetime',props->>'end_datetime';\n END IF;\n RETURN tstzrange(dt, edt, '[]');\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_datetime(value jsonb) RETURNS timestamptz AS $$\n SELECT lower(stac_daterange(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_end_datetime(value jsonb) RETURNS timestamptz AS $$\n SELECT upper(stac_daterange(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE TABLE IF NOT EXISTS stac_extensions(\n url text PRIMARY KEY,\n content jsonb\n);\nCREATE OR REPLACE FUNCTION queryable_signature(n text, c text[]) RETURNS text AS $$\n SELECT concat(n, c);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TABLE queryables (\n id bigint GENERATED ALWAYS AS identity PRIMARY KEY,\n name text NOT NULL,\n collection_ids text[], -- used to determine what partitions to create indexes on\n definition jsonb,\n property_path text,\n property_wrapper text,\n property_index_type text\n);\nCREATE INDEX queryables_name_idx ON queryables (name);\nCREATE INDEX queryables_collection_idx ON queryables USING GIN (collection_ids);\nCREATE INDEX queryables_property_wrapper_idx ON queryables (property_wrapper);\n\nCREATE OR REPLACE FUNCTION pgstac.queryables_constraint_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n allcollections text[];\nBEGIN\n RAISE NOTICE 'Making sure that name/collection is unique for queryables %', NEW;\n IF NEW.collection_ids IS NOT NULL THEN\n IF EXISTS (\n SELECT 1 FROM\n collections\n LEFT JOIN\n unnest(NEW.collection_ids) c\n ON (collections.id = c)\n WHERE c IS NULL\n ) THEN\n RAISE foreign_key_violation USING MESSAGE = format(\n 'One or more collections in %s do not exist.', NEW.collection_ids\n );\n RETURN NULL;\n END IF;\n END IF;\n IF TG_OP = 'INSERT' THEN\n IF EXISTS (\n SELECT 1 FROM queryables q\n WHERE\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n )\n ) THEN\n RAISE unique_violation USING MESSAGE = format(\n 'There is already a queryable for %s for a collection in %s: %s',\n NEW.name,\n NEW.collection_ids,\n\t\t\t\t(SELECT json_agg(row_to_json(q)) FROM queryables q WHERE\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n ))\n );\n RETURN NULL;\n END IF;\n END IF;\n IF TG_OP = 'UPDATE' THEN\n IF EXISTS (\n SELECT 1 FROM queryables q\n WHERE\n q.id != NEW.id\n AND\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n )\n ) THEN\n RAISE unique_violation\n USING MESSAGE = format(\n 'There is already a queryable for %s for a collection in %s',\n NEW.name,\n NEW.collection_ids\n );\n RETURN NULL;\n END IF;\n END IF;\n\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER queryables_constraint_insert_trigger\nBEFORE INSERT ON queryables\nFOR EACH ROW EXECUTE PROCEDURE queryables_constraint_triggerfunc();\n\nCREATE TRIGGER queryables_constraint_update_trigger\nBEFORE UPDATE ON queryables\nFOR EACH ROW\nWHEN (NEW.name = OLD.name AND NEW.collection_ids IS DISTINCT FROM OLD.collection_ids)\nEXECUTE PROCEDURE queryables_constraint_triggerfunc();\n\n\nCREATE OR REPLACE FUNCTION array_to_path(arr text[]) RETURNS text AS $$\n SELECT string_agg(\n quote_literal(v),\n '->'\n ) FROM unnest(arr) v;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\n\n\nCREATE OR REPLACE FUNCTION queryable(\n IN dotpath text,\n OUT path text,\n OUT expression text,\n OUT wrapper text,\n OUT nulled_wrapper text\n) AS $$\nDECLARE\n q RECORD;\n path_elements text[];\nBEGIN\n IF dotpath IN ('id', 'geometry', 'datetime', 'end_datetime', 'collection') THEN\n path := dotpath;\n expression := dotpath;\n wrapper := NULL;\n RETURN;\n END IF;\n SELECT * INTO q FROM queryables\n WHERE\n name=dotpath\n OR name = 'properties.' || dotpath\n OR name = replace(dotpath, 'properties.', '')\n ;\n IF q.property_wrapper IS NULL THEN\n IF q.definition->>'type' = 'number' THEN\n wrapper := 'to_float';\n nulled_wrapper := wrapper;\n ELSIF q.definition->>'format' = 'date-time' THEN\n wrapper := 'to_tstz';\n nulled_wrapper := wrapper;\n ELSE\n nulled_wrapper := NULL;\n wrapper := 'to_text';\n END IF;\n ELSE\n wrapper := q.property_wrapper;\n nulled_wrapper := wrapper;\n END IF;\n IF q.property_path IS NOT NULL THEN\n path := q.property_path;\n ELSE\n path_elements := string_to_array(dotpath, '.');\n IF path_elements[1] IN ('links', 'assets', 'stac_version', 'stac_extensions') THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSIF path_elements[1] = 'properties' THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSE\n path := format($F$content->'properties'->%s$F$, array_to_path(path_elements));\n END IF;\n END IF;\n expression := format('%I(%s)', wrapper, path);\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\nCREATE OR REPLACE FUNCTION unnest_collection(collection_ids text[] DEFAULT NULL) RETURNS SETOF text AS $$\n DECLARE\n BEGIN\n IF collection_ids IS NULL THEN\n RETURN QUERY SELECT id FROM collections;\n END IF;\n RETURN QUERY SELECT unnest(collection_ids);\n END;\n$$ LANGUAGE PLPGSQL STABLE;\n\nCREATE OR REPLACE FUNCTION normalize_indexdef(def text) RETURNS text AS $$\nDECLARE\nBEGIN\n def := btrim(def, ' \\n\\t');\n\tdef := regexp_replace(def, '^CREATE (UNIQUE )?INDEX ([^ ]* )?ON (ONLY )?([^ ]* )?', '', 'i');\n RETURN def;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE STRICT PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION indexdef(q queryables) RETURNS text AS $$\n DECLARE\n out text;\n BEGIN\n IF q.name = 'id' THEN\n out := 'CREATE UNIQUE INDEX ON %I USING btree (id)';\n ELSIF q.name = 'datetime' THEN\n out := 'CREATE INDEX ON %I USING btree (datetime DESC, end_datetime)';\n ELSIF q.name = 'geometry' THEN\n out := 'CREATE INDEX ON %I USING gist (geometry)';\n ELSE\n out := format($q$CREATE INDEX ON %%I USING %s (%s(((content -> 'properties'::text) -> %L::text)))$q$,\n lower(COALESCE(q.property_index_type, 'BTREE')),\n lower(COALESCE(q.property_wrapper, 'to_text')),\n q.name\n );\n END IF;\n RETURN btrim(out, ' \\n\\t');\n END;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\n\nDROP VIEW IF EXISTS pgstac_indexes;\nCREATE VIEW pgstac_indexes AS\nSELECT\n i.schemaname,\n i.tablename,\n i.indexname,\n regexp_replace(btrim(replace(replace(indexdef, i.indexname, ''),'pgstac.',''),' \\t\\n'), '[ ]+', ' ', 'g') as idx,\n COALESCE(\n (regexp_match(indexdef, '\\(([a-zA-Z]+)\\)'))[1],\n (regexp_match(indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_-]+)''::text'))[1],\n CASE WHEN indexdef ~* '\\(datetime desc, end_datetime\\)' THEN 'datetime' ELSE NULL END\n ) AS field,\n pg_table_size(i.indexname::text) as index_size,\n pg_size_pretty(pg_table_size(i.indexname::text)) as index_size_pretty\nFROM\n pg_indexes i\nWHERE i.schemaname='pgstac' and i.tablename ~ '_items_' AND indexdef !~* ' only ';\n\nDROP VIEW IF EXISTS pgstac_index_stats;\nCREATE VIEW pgstac_indexes_stats AS\nSELECT\n i.schemaname,\n i.tablename,\n i.indexname,\n indexdef,\n COALESCE(\n (regexp_match(indexdef, '\\(([a-zA-Z]+)\\)'))[1],\n (regexp_match(indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_]+)''::text'))[1],\n CASE WHEN indexdef ~* '\\(datetime desc, end_datetime\\)' THEN 'datetime_end_datetime' ELSE NULL END\n ) AS field,\n pg_table_size(i.indexname::text) as index_size,\n pg_size_pretty(pg_table_size(i.indexname::text)) as index_size_pretty,\n n_distinct,\n most_common_vals::text::text[],\n most_common_freqs::text::text[],\n histogram_bounds::text::text[],\n correlation\nFROM\n pg_indexes i\n LEFT JOIN pg_stats s ON (s.tablename = i.indexname)\nWHERE i.schemaname='pgstac' and i.tablename ~ '_items_';\n\nset check_function_bodies to off;\nCREATE OR REPLACE FUNCTION maintain_partition_queries(\n part text DEFAULT 'items',\n dropindexes boolean DEFAULT FALSE,\n rebuildindexes boolean DEFAULT FALSE,\n idxconcurrently boolean DEFAULT FALSE\n) RETURNS SETOF text AS $$\nDECLARE\n rec record;\nBEGIN\n FOR rec IN (\n WITH p AS (\n SELECT\n relid::text as partition,\n replace(replace(\n CASE\n WHEN level = 1 THEN pg_get_expr(c.relpartbound, c.oid)\n ELSE pg_get_expr(parent.relpartbound, parent.oid)\n END,\n 'FOR VALUES IN (''',''), ''')',\n ''\n ) AS collection\n FROM pg_partition_tree('items')\n JOIN pg_class c ON (relid::regclass = c.oid)\n JOIN pg_class parent ON (parentrelid::regclass = parent.oid AND isleaf)\n ), i AS (\n SELECT\n partition,\n indexname,\n regexp_replace(btrim(replace(replace(indexdef, indexname, ''),'pgstac.',''),' \\t\\n'), '[ ]+', ' ', 'g') as iidx,\n COALESCE(\n (regexp_match(indexdef, '\\(([a-zA-Z]+)\\)'))[1],\n (regexp_match(indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_-]+)''::text'))[1],\n CASE WHEN indexdef ~* '\\(datetime desc, end_datetime\\)' THEN 'datetime' ELSE NULL END\n ) AS field\n FROM\n pg_indexes\n JOIN p ON (tablename=partition)\n ), q AS (\n SELECT\n name AS field,\n collection,\n partition,\n format(indexdef(queryables), partition) as qidx\n FROM queryables, unnest_collection(queryables.collection_ids) collection\n JOIN p USING (collection)\n WHERE property_index_type IS NOT NULL OR name IN ('datetime','geometry','id')\n )\n SELECT * FROM i FULL JOIN q USING (field, partition)\n WHERE lower(iidx) IS DISTINCT FROM lower(qidx)\n ) LOOP\n IF rec.iidx IS NULL THEN\n IF idxconcurrently THEN\n RETURN NEXT replace(rec.qidx, 'INDEX', 'INDEX CONCURRENTLY');\n ELSE\n RETURN NEXT rec.qidx;\n END IF;\n ELSIF rec.qidx IS NULL AND dropindexes THEN\n RETURN NEXT format('DROP INDEX IF EXISTS %I;', rec.indexname);\n ELSIF lower(rec.qidx) != lower(rec.iidx) THEN\n IF dropindexes THEN\n RETURN NEXT format('DROP INDEX IF EXISTS %I; %s;', rec.indexname, rec.qidx);\n ELSE\n IF idxconcurrently THEN\n RETURN NEXT replace(rec.qidx, 'INDEX', 'INDEX CONCURRENTLY');\n ELSE\n RETURN NEXT rec.qidx;\n END IF;\n END IF;\n ELSIF rebuildindexes and rec.indexname IS NOT NULL THEN\n IF idxconcurrently THEN\n RETURN NEXT format('REINDEX INDEX CONCURRENTLY %I;', rec.indexname);\n ELSE\n RETURN NEXT format('REINDEX INDEX %I;', rec.indexname);\n END IF;\n END IF;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION maintain_partitions(\n part text DEFAULT 'items',\n dropindexes boolean DEFAULT FALSE,\n rebuildindexes boolean DEFAULT FALSE\n) RETURNS VOID AS $$\n WITH t AS (\n SELECT run_or_queue(q) FROM maintain_partition_queries(part, dropindexes, rebuildindexes) q\n ) SELECT count(*) FROM t;\n$$ LANGUAGE SQL;\n\n\nCREATE OR REPLACE FUNCTION queryables_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\nBEGIN\n PERFORM maintain_partitions();\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER queryables_trigger AFTER INSERT OR UPDATE ON queryables\nFOR EACH STATEMENT EXECUTE PROCEDURE queryables_trigger_func();\n\n\nCREATE OR REPLACE FUNCTION get_queryables(_collection_ids text[] DEFAULT NULL) RETURNS jsonb AS $$\nDECLARE\nBEGIN\n -- Build up queryables if the input contains valid collection ids or is empty\n IF EXISTS (\n SELECT 1 FROM collections\n WHERE\n _collection_ids IS NULL\n OR cardinality(_collection_ids) = 0\n OR id = ANY(_collection_ids)\n )\n THEN\n RETURN (\n WITH base AS (\n SELECT\n unnest(collection_ids) as collection_id,\n name,\n coalesce(definition, '{\"type\":\"string\"}'::jsonb) as definition\n FROM queryables\n WHERE\n _collection_ids IS NULL OR\n _collection_ids = '{}'::text[] OR\n _collection_ids && collection_ids\n UNION ALL\n SELECT null, name, coalesce(definition, '{\"type\":\"string\"}'::jsonb) as definition\n FROM queryables WHERE collection_ids IS NULL OR collection_ids = '{}'::text[]\n ), g AS (\n SELECT\n name,\n first_notnull(definition) as definition,\n jsonb_array_unique_merge(definition->'enum') as enum,\n jsonb_min(definition->'minimum') as minimum,\n jsonb_min(definition->'maxiumn') as maximum\n FROM base\n GROUP BY 1\n )\n SELECT\n jsonb_build_object(\n '$schema', 'http://json-schema.org/draft-07/schema#',\n '$id', '',\n 'type', 'object',\n 'title', 'STAC Queryables.',\n 'properties', jsonb_object_agg(\n name,\n definition\n ||\n jsonb_strip_nulls(jsonb_build_object(\n 'enum', enum,\n 'minimum', minimum,\n 'maximum', maximum\n ))\n )\n )\n FROM g\n );\n ELSE\n RETURN NULL;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\nCREATE OR REPLACE FUNCTION get_queryables(_collection text DEFAULT NULL) RETURNS jsonb AS $$\n SELECT\n CASE\n WHEN _collection IS NULL THEN get_queryables(NULL::text[])\n ELSE get_queryables(ARRAY[_collection])\n END\n ;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_queryables() RETURNS jsonb AS $$\n SELECT get_queryables(NULL::text[]);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION schema_qualify_refs(url text, j jsonb) returns jsonb as $$\n SELECT regexp_replace(j::text, '\"\\$ref\": \"#', concat('\"$ref\": \"', url, '#'), 'g')::jsonb;\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE VIEW stac_extension_queryables AS\nSELECT DISTINCT key as name, schema_qualify_refs(e.url, j.value) as definition FROM stac_extensions e, jsonb_each(e.content->'definitions'->'fields'->'properties') j;\n\n\nCREATE OR REPLACE FUNCTION missing_queryables(_collection text, _tablesample float DEFAULT 5, minrows float DEFAULT 10) RETURNS TABLE(collection text, name text, definition jsonb, property_wrapper text) AS $$\nDECLARE\n q text;\n _partition text;\n explain_json json;\n psize float;\n estrows float;\nBEGIN\n SELECT format('_items_%s', key) INTO _partition FROM collections WHERE id=_collection;\n\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM %I;', _partition)\n INTO explain_json;\n psize := explain_json->0->'Plan'->'Plan Rows';\n estrows := _tablesample * .01 * psize;\n IF estrows < minrows THEN\n _tablesample := least(100,greatest(_tablesample, (estrows / psize) / 100));\n RAISE NOTICE '%', (psize / estrows) / 100;\n END IF;\n RAISE NOTICE 'Using tablesample % to find missing queryables from % % that has ~% rows estrows: %', _tablesample, _collection, _partition, psize, estrows;\n\n q := format(\n $q$\n WITH q AS (\n SELECT * FROM queryables\n WHERE\n collection_ids IS NULL\n OR %L = ANY(collection_ids)\n ), t AS (\n SELECT\n content->'properties' AS properties\n FROM\n %I\n TABLESAMPLE SYSTEM(%L)\n ), p AS (\n SELECT DISTINCT ON (key)\n key,\n value,\n s.definition\n FROM t\n JOIN LATERAL jsonb_each(properties) ON TRUE\n LEFT JOIN q ON (q.name=key)\n LEFT JOIN stac_extension_queryables s ON (s.name=key)\n WHERE q.definition IS NULL\n )\n SELECT\n %L,\n key,\n COALESCE(definition, jsonb_build_object('type',jsonb_typeof(value))) as definition,\n CASE\n WHEN definition->>'type' = 'integer' THEN 'to_int'\n WHEN COALESCE(definition->>'type', jsonb_typeof(value)) = 'number' THEN 'to_float'\n WHEN COALESCE(definition->>'type', jsonb_typeof(value)) = 'array' THEN 'to_text_array'\n ELSE 'to_text'\n END\n FROM p;\n $q$,\n _collection,\n _partition,\n _tablesample,\n _collection\n );\n RETURN QUERY EXECUTE q;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION missing_queryables(_tablesample float DEFAULT 5) RETURNS TABLE(collection_ids text[], name text, definition jsonb, property_wrapper text) AS $$\n SELECT\n array_agg(collection),\n name,\n definition,\n property_wrapper\n FROM\n collections\n JOIN LATERAL\n missing_queryables(id, _tablesample) c\n ON TRUE\n GROUP BY\n 2,3,4\n ORDER BY 2,1\n ;\n$$ LANGUAGE SQL;\nCREATE OR REPLACE FUNCTION parse_dtrange(\n _indate jsonb,\n relative_base timestamptz DEFAULT date_trunc('hour', CURRENT_TIMESTAMP)\n) RETURNS tstzrange AS $$\nDECLARE\n timestrs text[];\n s timestamptz;\n e timestamptz;\nBEGIN\n timestrs :=\n CASE\n WHEN _indate ? 'timestamp' THEN\n ARRAY[_indate->>'timestamp']\n WHEN _indate ? 'interval' THEN\n to_text_array(_indate->'interval')\n WHEN jsonb_typeof(_indate) = 'array' THEN\n to_text_array(_indate)\n ELSE\n regexp_split_to_array(\n _indate->>0,\n '/'\n )\n END;\n RAISE NOTICE 'TIMESTRS %', timestrs;\n IF cardinality(timestrs) = 1 THEN\n IF timestrs[1] ILIKE 'P%' THEN\n RETURN tstzrange(relative_base - upper(timestrs[1])::interval, relative_base, '[)');\n END IF;\n s := timestrs[1]::timestamptz;\n RETURN tstzrange(s, s, '[]');\n END IF;\n\n IF cardinality(timestrs) != 2 THEN\n RAISE EXCEPTION 'Timestamp cannot have more than 2 values';\n END IF;\n\n IF timestrs[1] = '..' OR timestrs[1] = '' THEN\n s := '-infinity'::timestamptz;\n e := timestrs[2]::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] = '..' OR timestrs[2] = '' THEN\n s := timestrs[1]::timestamptz;\n e := 'infinity'::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[1] ILIKE 'P%' AND timestrs[2] NOT ILIKE 'P%' THEN\n e := timestrs[2]::timestamptz;\n s := e - upper(timestrs[1])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] ILIKE 'P%' AND timestrs[1] NOT ILIKE 'P%' THEN\n s := timestrs[1]::timestamptz;\n e := s + upper(timestrs[2])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n s := timestrs[1]::timestamptz;\n e := timestrs[2]::timestamptz;\n\n RETURN tstzrange(s,e,'[)');\n\n RETURN NULL;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT PARALLEL SAFE SET TIME ZONE 'UTC';\n\nCREATE OR REPLACE FUNCTION parse_dtrange(\n _indate text,\n relative_base timestamptz DEFAULT CURRENT_TIMESTAMP\n) RETURNS tstzrange AS $$\n SELECT parse_dtrange(to_jsonb(_indate), relative_base);\n$$ LANGUAGE SQL STABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION temporal_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\n ll text := 'datetime';\n lh text := 'end_datetime';\n rrange tstzrange;\n rl text;\n rh text;\n outq text;\nBEGIN\n rrange := parse_dtrange(args->1);\n RAISE NOTICE 'Constructing temporal query OP: %, ARGS: %, RRANGE: %', op, args, rrange;\n op := lower(op);\n rl := format('%L::timestamptz', lower(rrange));\n rh := format('%L::timestamptz', upper(rrange));\n outq := CASE op\n WHEN 't_before' THEN 'lh < rl'\n WHEN 't_after' THEN 'll > rh'\n WHEN 't_meets' THEN 'lh = rl'\n WHEN 't_metby' THEN 'll = rh'\n WHEN 't_overlaps' THEN 'll < rl AND rl < lh < rh'\n WHEN 't_overlappedby' THEN 'rl < ll < rh AND lh > rh'\n WHEN 't_starts' THEN 'll = rl AND lh < rh'\n WHEN 't_startedby' THEN 'll = rl AND lh > rh'\n WHEN 't_during' THEN 'll > rl AND lh < rh'\n WHEN 't_contains' THEN 'll < rl AND lh > rh'\n WHEN 't_finishes' THEN 'll > rl AND lh = rh'\n WHEN 't_finishedby' THEN 'll < rl AND lh = rh'\n WHEN 't_equals' THEN 'll = rl AND lh = rh'\n WHEN 't_disjoint' THEN 'NOT (ll <= rh AND lh >= rl)'\n WHEN 't_intersects' THEN 'll <= rh AND lh >= rl'\n WHEN 'anyinteracts' THEN 'll <= rh AND lh >= rl'\n END;\n outq := regexp_replace(outq, '\\mll\\M', ll);\n outq := regexp_replace(outq, '\\mlh\\M', lh);\n outq := regexp_replace(outq, '\\mrl\\M', rl);\n outq := regexp_replace(outq, '\\mrh\\M', rh);\n outq := format('(%s)', outq);\n RETURN outq;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\n\n\nCREATE OR REPLACE FUNCTION spatial_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\n geom text;\n j jsonb := args->1;\nBEGIN\n op := lower(op);\n RAISE NOTICE 'Constructing spatial query OP: %, ARGS: %', op, args;\n IF op NOT IN ('s_equals','s_disjoint','s_touches','s_within','s_overlaps','s_crosses','s_intersects','intersects','s_contains') THEN\n RAISE EXCEPTION 'Spatial Operator % Not Supported', op;\n END IF;\n op := regexp_replace(op, '^s_', 'st_');\n IF op = 'intersects' THEN\n op := 'st_intersects';\n END IF;\n -- Convert geometry to WKB string\n IF j ? 'type' AND j ? 'coordinates' THEN\n geom := st_geomfromgeojson(j)::text;\n ELSIF jsonb_typeof(j) = 'array' THEN\n geom := bbox_geom(j)::text;\n END IF;\n\n RETURN format('%s(geometry, %L::geometry)', op, geom);\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION query_to_cql2(q jsonb) RETURNS jsonb AS $$\n-- Translates anything passed in through the deprecated \"query\" into equivalent CQL2\nWITH t AS (\n SELECT key as property, value as ops\n FROM jsonb_each(q)\n), t2 AS (\n SELECT property, (jsonb_each(ops)).*\n FROM t WHERE jsonb_typeof(ops) = 'object'\n UNION ALL\n SELECT property, 'eq', ops\n FROM t WHERE jsonb_typeof(ops) != 'object'\n)\nSELECT\n jsonb_strip_nulls(jsonb_build_object(\n 'op', 'and',\n 'args', jsonb_agg(\n jsonb_build_object(\n 'op', key,\n 'args', jsonb_build_array(\n jsonb_build_object('property',property),\n value\n )\n )\n )\n )\n) as qcql FROM t2\n;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\nCREATE OR REPLACE FUNCTION cql1_to_cql2(j jsonb) RETURNS jsonb AS $$\nDECLARE\n args jsonb;\n ret jsonb;\nBEGIN\n RAISE NOTICE 'CQL1_TO_CQL2: %', j;\n IF j ? 'filter' THEN\n RETURN cql1_to_cql2(j->'filter');\n END IF;\n IF j ? 'property' THEN\n RETURN j;\n END IF;\n IF jsonb_typeof(j) = 'array' THEN\n SELECT jsonb_agg(cql1_to_cql2(el)) INTO args FROM jsonb_array_elements(j) el;\n RETURN args;\n END IF;\n IF jsonb_typeof(j) = 'number' THEN\n RETURN j;\n END IF;\n IF jsonb_typeof(j) = 'string' THEN\n RETURN j;\n END IF;\n\n IF jsonb_typeof(j) = 'object' THEN\n SELECT jsonb_build_object(\n 'op', key,\n 'args', cql1_to_cql2(value)\n ) INTO ret\n FROM jsonb_each(j)\n WHERE j IS NOT NULL;\n RETURN ret;\n END IF;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE STRICT;\n\nCREATE TABLE cql2_ops (\n op text PRIMARY KEY,\n template text,\n types text[]\n);\nINSERT INTO cql2_ops (op, template, types) VALUES\n ('eq', '%s = %s', NULL),\n ('neq', '%s != %s', NULL),\n ('ne', '%s != %s', NULL),\n ('!=', '%s != %s', NULL),\n ('<>', '%s != %s', NULL),\n ('lt', '%s < %s', NULL),\n ('lte', '%s <= %s', NULL),\n ('gt', '%s > %s', NULL),\n ('gte', '%s >= %s', NULL),\n ('le', '%s <= %s', NULL),\n ('ge', '%s >= %s', NULL),\n ('=', '%s = %s', NULL),\n ('<', '%s < %s', NULL),\n ('<=', '%s <= %s', NULL),\n ('>', '%s > %s', NULL),\n ('>=', '%s >= %s', NULL),\n ('like', '%s LIKE %s', NULL),\n ('ilike', '%s ILIKE %s', NULL),\n ('+', '%s + %s', NULL),\n ('-', '%s - %s', NULL),\n ('*', '%s * %s', NULL),\n ('/', '%s / %s', NULL),\n ('not', 'NOT (%s)', NULL),\n ('between', '%s BETWEEN %s AND %s', NULL),\n ('isnull', '%s IS NULL', NULL),\n ('upper', 'upper(%s)', NULL),\n ('lower', 'lower(%s)', NULL)\nON CONFLICT (op) DO UPDATE\n SET\n template = EXCLUDED.template\n;\n\n\nCREATE OR REPLACE FUNCTION cql2_query(j jsonb, wrapper text DEFAULT NULL) RETURNS text AS $$\n#variable_conflict use_variable\nDECLARE\n args jsonb := j->'args';\n arg jsonb;\n op text := lower(j->>'op');\n cql2op RECORD;\n literal text;\n _wrapper text;\n leftarg text;\n rightarg text;\nBEGIN\n IF j IS NULL OR (op IS NOT NULL AND args IS NULL) THEN\n RETURN NULL;\n END IF;\n RAISE NOTICE 'CQL2_QUERY: %', j;\n IF j ? 'filter' THEN\n RETURN cql2_query(j->'filter');\n END IF;\n\n IF j ? 'upper' THEN\n RETURN cql2_query(jsonb_build_object('op', 'upper', 'args', j->'upper'));\n END IF;\n\n IF j ? 'lower' THEN\n RETURN cql2_query(jsonb_build_object('op', 'lower', 'args', j->'lower'));\n END IF;\n\n -- Temporal Query\n IF op ilike 't_%' or op = 'anyinteracts' THEN\n RETURN temporal_op_query(op, args);\n END IF;\n\n -- If property is a timestamp convert it to text to use with\n -- general operators\n IF j ? 'timestamp' THEN\n RETURN format('%L::timestamptz', to_tstz(j->'timestamp'));\n END IF;\n IF j ? 'interval' THEN\n RAISE EXCEPTION 'Please use temporal operators when using intervals.';\n RETURN NONE;\n END IF;\n\n -- Spatial Query\n IF op ilike 's_%' or op = 'intersects' THEN\n RETURN spatial_op_query(op, args);\n END IF;\n\n IF op IN ('a_equals','a_contains','a_contained_by','a_overlaps') THEN\n IF args->0 ? 'property' THEN\n leftarg := format('to_text_array(%s)', (queryable(args->0->>'property')).path);\n END IF;\n IF args->1 ? 'property' THEN\n rightarg := format('to_text_array(%s)', (queryable(args->1->>'property')).path);\n END IF;\n RETURN FORMAT(\n '%s %s %s',\n COALESCE(leftarg, quote_literal(to_text_array(args->0))),\n CASE op\n WHEN 'a_equals' THEN '='\n WHEN 'a_contains' THEN '@>'\n WHEN 'a_contained_by' THEN '<@'\n WHEN 'a_overlaps' THEN '&&'\n END,\n COALESCE(rightarg, quote_literal(to_text_array(args->1)))\n );\n END IF;\n\n IF op = 'in' THEN\n RAISE NOTICE 'IN : % % %', args, jsonb_build_array(args->0), args->1;\n args := jsonb_build_array(args->0) || (args->1);\n RAISE NOTICE 'IN2 : %', args;\n END IF;\n\n\n\n IF op = 'between' THEN\n args = jsonb_build_array(\n args->0,\n args->1->0,\n args->1->1\n );\n END IF;\n\n -- Make sure that args is an array and run cql2_query on\n -- each element of the array\n RAISE NOTICE 'ARGS PRE: %', args;\n IF j ? 'args' THEN\n IF jsonb_typeof(args) != 'array' THEN\n args := jsonb_build_array(args);\n END IF;\n\n IF jsonb_path_exists(args, '$[*] ? (@.property == \"id\" || @.property == \"datetime\" || @.property == \"end_datetime\" || @.property == \"collection\")') THEN\n wrapper := NULL;\n ELSE\n -- if any of the arguments are a property, try to get the property_wrapper\n FOR arg IN SELECT jsonb_path_query(args, '$[*] ? (@.property != null)') LOOP\n RAISE NOTICE 'Arg: %', arg;\n wrapper := (queryable(arg->>'property')).nulled_wrapper;\n RAISE NOTICE 'Property: %, Wrapper: %', arg, wrapper;\n IF wrapper IS NOT NULL THEN\n EXIT;\n END IF;\n END LOOP;\n\n -- if the property was not in queryables, see if any args were numbers\n IF\n wrapper IS NULL\n AND jsonb_path_exists(args, '$[*] ? (@.type()==\"number\")')\n THEN\n wrapper := 'to_float';\n END IF;\n wrapper := coalesce(wrapper, 'to_text');\n END IF;\n\n SELECT jsonb_agg(cql2_query(a, wrapper))\n INTO args\n FROM jsonb_array_elements(args) a;\n END IF;\n RAISE NOTICE 'ARGS: %', args;\n\n IF op IN ('and', 'or') THEN\n RETURN\n format(\n '(%s)',\n array_to_string(to_text_array(args), format(' %s ', upper(op)))\n );\n END IF;\n\n IF op = 'in' THEN\n RAISE NOTICE 'IN -- % %', args->0, to_text(args->0);\n RETURN format(\n '%s IN (%s)',\n to_text(args->0),\n array_to_string((to_text_array(args))[2:], ',')\n );\n END IF;\n\n -- Look up template from cql2_ops\n IF j ? 'op' THEN\n SELECT * INTO cql2op FROM cql2_ops WHERE cql2_ops.op ilike op;\n IF FOUND THEN\n -- If specific index set in queryables for a property cast other arguments to that type\n\n RETURN format(\n cql2op.template,\n VARIADIC (to_text_array(args))\n );\n ELSE\n RAISE EXCEPTION 'Operator % Not Supported.', op;\n END IF;\n END IF;\n\n\n IF wrapper IS NOT NULL THEN\n RAISE NOTICE 'Wrapping % with %', j, wrapper;\n IF j ? 'property' THEN\n RETURN format('%I(%s)', wrapper, (queryable(j->>'property')).path);\n ELSE\n RETURN format('%I(%L)', wrapper, j);\n END IF;\n ELSIF j ? 'property' THEN\n RETURN quote_ident(j->>'property');\n END IF;\n\n RETURN quote_literal(to_text(j));\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION paging_dtrange(\n j jsonb\n) RETURNS tstzrange AS $$\nDECLARE\n op text;\n filter jsonb := j->'filter';\n dtrange tstzrange := tstzrange('-infinity'::timestamptz,'infinity'::timestamptz);\n sdate timestamptz := '-infinity'::timestamptz;\n edate timestamptz := 'infinity'::timestamptz;\n jpitem jsonb;\nBEGIN\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n END IF;\n IF NOT (filter @? '$.**.op ? (@ == \"or\" || @ == \"not\")') THEN\n FOR jpitem IN SELECT j FROM jsonb_path_query(filter,'strict $.** ? (@.args[*].property == \"datetime\")'::jsonpath) j LOOP\n op := lower(jpitem->>'op');\n dtrange := parse_dtrange(jpitem->'args'->1);\n IF op IN ('<=', 'lt', 'lte', '<', 'le', 't_before') THEN\n sdate := greatest(sdate,'-infinity');\n edate := least(edate, upper(dtrange));\n ELSIF op IN ('>=', '>', 'gt', 'gte', 'ge', 't_after') THEN\n edate := least(edate, 'infinity');\n sdate := greatest(sdate, lower(dtrange));\n ELSIF op IN ('=', 'eq') THEN\n edate := least(edate, upper(dtrange));\n sdate := greatest(sdate, lower(dtrange));\n END IF;\n RAISE NOTICE '2 OP: %, ARGS: %, DTRANGE: %, SDATE: %, EDATE: %', op, jpitem->'args'->1, dtrange, sdate, edate;\n END LOOP;\n END IF;\n IF sdate > edate THEN\n RETURN 'empty'::tstzrange;\n END IF;\n RETURN tstzrange(sdate,edate, '[]');\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT SET TIME ZONE 'UTC';\n\nCREATE OR REPLACE FUNCTION paging_collections(\n IN j jsonb\n) RETURNS text[] AS $$\nDECLARE\n filter jsonb := j->'filter';\n jpitem jsonb;\n op text;\n args jsonb;\n arg jsonb;\n collections text[];\nBEGIN\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n END IF;\n IF NOT (filter @? '$.**.op ? (@ == \"or\" || @ == \"not\")') THEN\n FOR jpitem IN SELECT j FROM jsonb_path_query(filter,'strict $.** ? (@.args[*].property == \"collection\")'::jsonpath) j LOOP\n RAISE NOTICE 'JPITEM: %', jpitem;\n op := jpitem->>'op';\n args := jpitem->'args';\n IF op IN ('=', 'eq', 'in') THEN\n FOR arg IN SELECT a FROM jsonb_array_elements(args) a LOOP\n IF jsonb_typeof(arg) IN ('string', 'array') THEN\n RAISE NOTICE 'arg: %, collections: %', arg, collections;\n IF collections IS NULL OR collections = '{}'::text[] THEN\n collections := to_text_array(arg);\n ELSE\n collections := array_intersection(collections, to_text_array(arg));\n END IF;\n END IF;\n END LOOP;\n END IF;\n END LOOP;\n END IF;\n IF collections = '{}'::text[] THEN\n RETURN NULL;\n END IF;\n RETURN collections;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\nCREATE OR REPLACE FUNCTION collection_base_item(content jsonb) RETURNS jsonb AS $$\n SELECT jsonb_build_object(\n 'type', 'Feature',\n 'stac_version', content->'stac_version',\n 'assets', content->'item_assets',\n 'collection', content->'id'\n );\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TABLE IF NOT EXISTS collections (\n key bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,\n id text GENERATED ALWAYS AS (content->>'id') STORED UNIQUE,\n content JSONB NOT NULL,\n base_item jsonb GENERATED ALWAYS AS (pgstac.collection_base_item(content)) STORED,\n partition_trunc text CHECK (partition_trunc IN ('year', 'month'))\n);\n\n\n\nCREATE OR REPLACE FUNCTION collection_base_item(cid text) RETURNS jsonb AS $$\n SELECT pgstac.collection_base_item(content) FROM pgstac.collections WHERE id = cid LIMIT 1;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION create_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION update_collection(data jsonb) RETURNS VOID AS $$\nDECLARE\n out collections%ROWTYPE;\nBEGIN\n UPDATE collections SET content=data WHERE id = data->>'id' RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ON CONFLICT (id) DO\n UPDATE\n SET content=EXCLUDED.content\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION delete_collection(_id text) RETURNS VOID AS $$\nDECLARE\n out collections%ROWTYPE;\nBEGIN\n DELETE FROM collections WHERE id = _id RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION get_collection(id text) RETURNS jsonb AS $$\n SELECT content FROM collections\n WHERE id=$1\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION all_collections() RETURNS jsonb AS $$\n SELECT jsonb_agg(content) FROM collections;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\nCREATE TABLE items (\n id text NOT NULL,\n geometry geometry NOT NULL,\n collection text NOT NULL,\n datetime timestamptz NOT NULL,\n end_datetime timestamptz NOT NULL,\n content JSONB NOT NULL\n)\nPARTITION BY LIST (collection)\n;\n\nCREATE INDEX \"datetime_idx\" ON items USING BTREE (datetime DESC, end_datetime ASC);\nCREATE INDEX \"geometry_idx\" ON items USING GIST (geometry);\n\nCREATE STATISTICS datetime_stats (dependencies) on datetime, end_datetime from items;\n\nALTER TABLE items ADD CONSTRAINT items_collections_fk FOREIGN KEY (collection) REFERENCES collections(id) ON DELETE CASCADE DEFERRABLE;\n\nCREATE OR REPLACE FUNCTION partition_after_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p text;\n t timestamptz := clock_timestamp();\nBEGIN\n RAISE NOTICE 'Updating partition stats %', t;\n FOR p IN SELECT DISTINCT partition\n FROM newdata n JOIN partition_sys_meta p\n ON (n.collection=p.collection AND n.datetime <@ p.partition_dtrange)\n LOOP\n PERFORM run_or_queue(format('SELECT update_partition_stats(%L, %L);', p, true));\n END LOOP;\n IF TG_OP IN ('DELETE','UPDATE') THEN\n DELETE FROM format_item_cache c USING newdata n WHERE c.collection = n.collection AND c.id = n.id;\n END IF;\n RAISE NOTICE 't: % %', t, clock_timestamp() - t;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE TRIGGER items_after_insert_trigger\nAFTER INSERT ON items\nREFERENCING NEW TABLE AS newdata\nFOR EACH STATEMENT\nEXECUTE FUNCTION partition_after_triggerfunc();\n\nCREATE TRIGGER items_after_update_trigger\nAFTER DELETE ON items\nREFERENCING OLD TABLE AS newdata\nFOR EACH STATEMENT\nEXECUTE FUNCTION partition_after_triggerfunc();\n\nCREATE TRIGGER items_after_delete_trigger\nAFTER UPDATE ON items\nREFERENCING NEW TABLE AS newdata\nFOR EACH STATEMENT\nEXECUTE FUNCTION partition_after_triggerfunc();\n\n\nCREATE OR REPLACE FUNCTION content_slim(_item jsonb) RETURNS jsonb AS $$\n SELECT strip_jsonb(_item - '{id,geometry,collection,type}'::text[], collection_base_item(_item->>'collection')) - '{id,geometry,collection,type}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_dehydrate(content jsonb) RETURNS items AS $$\n SELECT\n content->>'id' as id,\n stac_geom(content) as geometry,\n content->>'collection' as collection,\n stac_datetime(content) as datetime,\n stac_end_datetime(content) as end_datetime,\n content_slim(content) as content\n ;\n$$ LANGUAGE SQL STABLE;\n\nCREATE OR REPLACE FUNCTION include_field(f text, fields jsonb DEFAULT '{}'::jsonb) RETURNS boolean AS $$\nDECLARE\n includes jsonb := fields->'include';\n excludes jsonb := fields->'exclude';\nBEGIN\n IF f IS NULL THEN\n RETURN NULL;\n END IF;\n\n\n IF\n jsonb_typeof(excludes) = 'array'\n AND jsonb_array_length(excludes)>0\n AND excludes ? f\n THEN\n RETURN FALSE;\n END IF;\n\n IF\n (\n jsonb_typeof(includes) = 'array'\n AND jsonb_array_length(includes) > 0\n AND includes ? f\n ) OR\n (\n includes IS NULL\n OR jsonb_typeof(includes) = 'null'\n OR jsonb_array_length(includes) = 0\n )\n THEN\n RETURN TRUE;\n END IF;\n\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nDROP FUNCTION IF EXISTS content_hydrate(jsonb, jsonb, jsonb);\nCREATE OR REPLACE FUNCTION content_hydrate(\n _item jsonb,\n _base_item jsonb,\n fields jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\n SELECT merge_jsonb(\n jsonb_fields(_item, fields),\n jsonb_fields(_base_item, fields)\n );\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION content_hydrate(_item items, _collection collections, fields jsonb DEFAULT '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\n content jsonb;\n base_item jsonb := _collection.base_item;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry, 20)::jsonb;\n END IF;\n output := content_hydrate(\n jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'collection', _item.collection,\n 'type', 'Feature'\n ) || _item.content,\n _collection.base_item,\n fields\n );\n\n RETURN output;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_nonhydrated(\n _item items,\n fields jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry, 20)::jsonb;\n END IF;\n output := jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'collection', _item.collection,\n 'type', 'Feature'\n ) || _item.content;\n RETURN output;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_hydrate(_item items, fields jsonb DEFAULT '{}'::jsonb) RETURNS jsonb AS $$\n SELECT content_hydrate(\n _item,\n (SELECT c FROM collections c WHERE id=_item.collection LIMIT 1),\n fields\n );\n$$ LANGUAGE SQL STABLE;\n\n\nCREATE UNLOGGED TABLE items_staging (\n content JSONB NOT NULL\n);\nCREATE UNLOGGED TABLE items_staging_ignore (\n content JSONB NOT NULL\n);\nCREATE UNLOGGED TABLE items_staging_upsert (\n content JSONB NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION items_staging_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p record;\n _partitions text[];\n part text;\n ts timestamptz := clock_timestamp();\nBEGIN\n RAISE NOTICE 'Creating Partitions. %', clock_timestamp() - ts;\n\n FOR part IN WITH t AS (\n SELECT\n n.content->>'collection' as collection,\n stac_daterange(n.content->'properties') as dtr,\n partition_trunc\n FROM newdata n JOIN collections ON (n.content->>'collection'=collections.id)\n ), p AS (\n SELECT\n collection,\n COALESCE(date_trunc(partition_trunc::text, lower(dtr)),'-infinity') as d,\n tstzrange(min(lower(dtr)),max(lower(dtr)),'[]') as dtrange,\n tstzrange(min(upper(dtr)),max(upper(dtr)),'[]') as edtrange\n FROM t\n GROUP BY 1,2\n ) SELECT check_partition(collection, dtrange, edtrange) FROM p LOOP\n RAISE NOTICE 'Partition %', part;\n END LOOP;\n\n RAISE NOTICE 'Doing the insert. %', clock_timestamp() - ts;\n IF TG_TABLE_NAME = 'items_staging' THEN\n INSERT INTO items\n SELECT\n (content_dehydrate(content)).*\n FROM newdata;\n RAISE NOTICE 'Doing the delete. %', clock_timestamp() - ts;\n DELETE FROM items_staging;\n ELSIF TG_TABLE_NAME = 'items_staging_ignore' THEN\n INSERT INTO items\n SELECT\n (content_dehydrate(content)).*\n FROM newdata\n ON CONFLICT DO NOTHING;\n RAISE NOTICE 'Doing the delete. %', clock_timestamp() - ts;\n DELETE FROM items_staging_ignore;\n ELSIF TG_TABLE_NAME = 'items_staging_upsert' THEN\n WITH staging_formatted AS (\n SELECT (content_dehydrate(content)).* FROM newdata\n ), deletes AS (\n DELETE FROM items i USING staging_formatted s\n WHERE\n i.id = s.id\n AND i.collection = s.collection\n AND i IS DISTINCT FROM s\n RETURNING i.id, i.collection\n )\n INSERT INTO items\n SELECT s.* FROM\n staging_formatted s\n ON CONFLICT DO NOTHING;\n RAISE NOTICE 'Doing the delete. %', clock_timestamp() - ts;\n DELETE FROM items_staging_upsert;\n END IF;\n RAISE NOTICE 'Done. %', clock_timestamp() - ts;\n\n RETURN NULL;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER items_staging_insert_trigger AFTER INSERT ON items_staging REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\nCREATE TRIGGER items_staging_insert_ignore_trigger AFTER INSERT ON items_staging_ignore REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\nCREATE TRIGGER items_staging_insert_upsert_trigger AFTER INSERT ON items_staging_upsert REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\n\nCREATE OR REPLACE FUNCTION item_by_id(_id text, _collection text DEFAULT NULL) RETURNS items AS\n$$\nDECLARE\n i items%ROWTYPE;\nBEGIN\n SELECT * INTO i FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection) LIMIT 1;\n RETURN i;\nEND;\n$$ LANGUAGE PLPGSQL STABLE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION get_item(_id text, _collection text DEFAULT NULL) RETURNS jsonb AS $$\n SELECT content_hydrate(items) FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection);\n$$ LANGUAGE SQL STABLE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION delete_item(_id text, _collection text DEFAULT NULL) RETURNS VOID AS $$\nDECLARE\nout items%ROWTYPE;\nBEGIN\n DELETE FROM items WHERE id = _id AND (_collection IS NULL OR collection=_collection) RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n--/*\nCREATE OR REPLACE FUNCTION create_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION update_item(content jsonb) RETURNS VOID AS $$\nDECLARE\n old items %ROWTYPE;\n out items%ROWTYPE;\nBEGIN\n PERFORM delete_item(content->>'id', content->>'collection');\n PERFORM create_item(content);\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION create_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION collection_bbox(id text) RETURNS jsonb AS $$\n SELECT (replace(replace(replace(st_extent(geometry)::text,'BOX(','[['),')',']]'),' ',','))::jsonb\n FROM items WHERE collection=$1;\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION collection_temporal_extent(id text) RETURNS jsonb AS $$\n SELECT to_jsonb(array[array[min(datetime)::text, max(datetime)::text]])\n FROM items WHERE collection=$1;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection_extents() RETURNS VOID AS $$\nUPDATE collections SET\n content = content ||\n jsonb_build_object(\n 'extent', jsonb_build_object(\n 'spatial', jsonb_build_object(\n 'bbox', collection_bbox(collections.id)\n ),\n 'temporal', jsonb_build_object(\n 'interval', collection_temporal_extent(collections.id)\n )\n )\n )\n;\n$$ LANGUAGE SQL;\nCREATE TABLE partition_stats (\n partition text PRIMARY KEY,\n dtrange tstzrange,\n edtrange tstzrange,\n spatial geometry,\n last_updated timestamptz,\n keys text[]\n) WITH (FILLFACTOR=90);\n\nCREATE INDEX partitions_range_idx ON partition_stats USING GIST(dtrange);\n\n\nCREATE OR REPLACE FUNCTION constraint_tstzrange(expr text) RETURNS tstzrange AS $$\n WITH t AS (\n SELECT regexp_matches(\n expr,\n E'\\\\(''\\([0-9 :+-]*\\)''\\\\).*\\\\(''\\([0-9 :+-]*\\)''\\\\)'\n ) AS m\n ) SELECT tstzrange(m[1]::timestamptz, m[2]::timestamptz) FROM t\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE STRICT;\n\nCREATE OR REPLACE FUNCTION dt_constraint(coid oid, OUT dt tstzrange, OUT edt tstzrange) RETURNS RECORD AS $$\nDECLARE\n expr text := pg_get_constraintdef(coid);\n matches timestamptz[];\nBEGIN\n IF expr LIKE '%NULL%' THEN\n dt := tstzrange(null::timestamptz, null::timestamptz);\n edt := tstzrange(null::timestamptz, null::timestamptz);\n RETURN;\n END IF;\n WITH f AS (SELECT (regexp_matches(expr, E'([0-9]{4}-[0-1][0-9]-[0-3][0-9] [0-2][0-9]:[0-5][0-9]:[0-5][0-9]\\.?[0-9]*)', 'g'))[1] f)\n SELECT array_agg(f::timestamptz) INTO matches FROM f;\n IF cardinality(matches) = 4 THEN\n dt := tstzrange(matches[1], matches[2],'[]');\n edt := tstzrange(matches[3], matches[4], '[]');\n RETURN;\n ELSIF cardinality(matches) = 2 THEN\n edt := tstzrange(matches[1], matches[2],'[]');\n RETURN;\n END IF;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\nCREATE OR REPLACE VIEW partition_sys_meta AS\nSELECT\n relid::text as partition,\n replace(replace(CASE WHEN level = 1 THEN pg_get_expr(c.relpartbound, c.oid)\n ELSE pg_get_expr(parent.relpartbound, parent.oid)\n END, 'FOR VALUES IN (''',''), ''')','') AS collection,\n level,\n c.reltuples,\n c.relhastriggers,\n COALESCE(constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as partition_dtrange,\n COALESCE((dt_constraint(edt.oid)).dt, constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as constraint_dtrange,\n COALESCE((dt_constraint(edt.oid)).edt, tstzrange('-infinity', 'infinity','[]')) as constraint_edtrange\nFROM\n pg_partition_tree('items')\n JOIN pg_class c ON (relid::regclass = c.oid)\n JOIN pg_class parent ON (parentrelid::regclass = parent.oid AND isleaf)\n LEFT JOIN pg_constraint edt ON (conrelid=c.oid AND contype='c')\nWHERE isleaf\n;\n\nCREATE VIEW partitions AS\nSELECT * FROM partition_sys_meta LEFT JOIN partition_stats USING (partition);\n\nCREATE OR REPLACE FUNCTION update_partition_stats_q(_partition text, istrigger boolean default false) RETURNS VOID AS $$\nDECLARE\nBEGIN\n PERFORM run_or_queue(\n format('SELECT update_partition_stats(%L, %L);', _partition, istrigger)\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION update_partition_stats(_partition text, istrigger boolean default false) RETURNS VOID AS $$\nDECLARE\n dtrange tstzrange;\n edtrange tstzrange;\n cdtrange tstzrange;\n cedtrange tstzrange;\n extent geometry;\n collection text;\nBEGIN\n RAISE NOTICE 'Updating stats for %.', _partition;\n EXECUTE format(\n $q$\n SELECT\n tstzrange(min(datetime), max(datetime),'[]'),\n tstzrange(min(end_datetime), max(end_datetime), '[]')\n FROM %I\n $q$,\n _partition\n ) INTO dtrange, edtrange;\n extent := st_estimatedextent('pgstac', _partition, 'geometry');\n INSERT INTO partition_stats (partition, dtrange, edtrange, spatial, last_updated)\n SELECT _partition, dtrange, edtrange, extent, now()\n ON CONFLICT (partition) DO\n UPDATE SET\n dtrange=EXCLUDED.dtrange,\n edtrange=EXCLUDED.edtrange,\n spatial=EXCLUDED.spatial,\n last_updated=EXCLUDED.last_updated\n ;\n SELECT\n constraint_dtrange, constraint_edtrange, partitions.collection\n INTO cdtrange, cedtrange, collection\n FROM partitions WHERE partition = _partition;\n\n RAISE NOTICE 'Checking if we need to modify constraints.';\n IF\n (cdtrange IS DISTINCT FROM dtrange OR edtrange IS DISTINCT FROM cedtrange)\n AND NOT istrigger\n THEN\n RAISE NOTICE 'Modifying Constraints';\n RAISE NOTICE 'Existing % %', cdtrange, cedtrange;\n RAISE NOTICE 'New % %', dtrange, edtrange;\n PERFORM drop_table_constraints(_partition);\n PERFORM create_table_constraints(_partition, dtrange, edtrange);\n END IF;\n RAISE NOTICE 'Checking if we need to update collection extents.';\n IF get_setting_bool('update_collection_extent') THEN\n RAISE NOTICE 'updating collection extent for %', collection;\n PERFORM run_or_queue(format($q$\n UPDATE collections\n SET content = jsonb_set_lax(\n content,\n '{extent}'::text[],\n collection_extent(%L),\n true,\n 'use_json_null'\n ) WHERE id=%L\n ;\n $q$, collection, collection));\n ELSE\n RAISE NOTICE 'Not updating collection extent for %', collection;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL STRICT;\n\n\nCREATE OR REPLACE FUNCTION partition_name( IN collection text, IN dt timestamptz, OUT partition_name text, OUT partition_range tstzrange) AS $$\nDECLARE\n c RECORD;\n parent_name text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n parent_name := format('_items_%s', c.key);\n\n\n IF c.partition_trunc = 'year' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYY'));\n ELSIF c.partition_trunc = 'month' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYYMM'));\n ELSE\n partition_name := parent_name;\n partition_range := tstzrange('-infinity'::timestamptz, 'infinity'::timestamptz, '[]');\n END IF;\n IF partition_range IS NULL THEN\n partition_range := tstzrange(\n date_trunc(c.partition_trunc::text, dt),\n date_trunc(c.partition_trunc::text, dt) + concat('1 ', c.partition_trunc)::interval\n );\n END IF;\n RETURN;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION drop_table_constraints(t text) RETURNS text AS $$\nDECLARE\n q text;\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM partitions WHERE partition=t) THEN\n RETURN NULL;\n END IF;\n FOR q IN SELECT FORMAT(\n $q$\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n $q$,\n t,\n conname\n ) FROM pg_constraint\n WHERE conrelid=t::regclass::oid AND contype='c'\n LOOP\n EXECUTE q;\n END LOOP;\n RETURN t;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE OR REPLACE FUNCTION create_table_constraints(t text, _dtrange tstzrange, _edtrange tstzrange) RETURNS text AS $$\nDECLARE\n q text;\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM partitions WHERE partition=t) THEN\n RETURN NULL;\n END IF;\n RAISE NOTICE 'Creating Table Constraints for % % %', t, _dtrange, _edtrange;\n IF _dtrange = 'empty' AND _edtrange = 'empty' THEN\n q :=format(\n $q$\n DO $block$\n BEGIN\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n ALTER TABLE %I\n ADD CONSTRAINT %I\n CHECK (((datetime IS NULL) AND (end_datetime IS NULL))) NOT VALID\n ;\n ALTER TABLE %I\n VALIDATE CONSTRAINT %I\n ;\n\n EXCEPTION WHEN others THEN\n RAISE WARNING '%%, Issue Altering Constraints. Please run update_partition_stats(%I)', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n $block$;\n $q$,\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t),\n t\n );\n ELSE\n q :=format(\n $q$\n DO $block$\n BEGIN\n\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n ALTER TABLE %I\n ADD CONSTRAINT %I\n CHECK (\n (datetime >= %L)\n AND (datetime <= %L)\n AND (end_datetime >= %L)\n AND (end_datetime <= %L)\n ) NOT VALID\n ;\n ALTER TABLE %I\n VALIDATE CONSTRAINT %I\n ;\n\n EXCEPTION WHEN others THEN\n RAISE WARNING '%%, Issue Altering Constraints. Please run update_partition_stats(%I)', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n $block$;\n $q$,\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t),\n lower(_dtrange),\n upper(_dtrange),\n lower(_edtrange),\n upper(_edtrange),\n t,\n format('%s_dt', t),\n t\n );\n END IF;\n PERFORM run_or_queue(q);\n RETURN t;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION check_partition(\n _collection text,\n _dtrange tstzrange,\n _edtrange tstzrange\n) RETURNS text AS $$\nDECLARE\n c RECORD;\n pm RECORD;\n _partition_name text;\n _partition_dtrange tstzrange;\n _constraint_dtrange tstzrange;\n _constraint_edtrange tstzrange;\n q text;\n deferrable_q text;\n err_context text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=_collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', _collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n\n IF c.partition_trunc IS NOT NULL THEN\n _partition_dtrange := tstzrange(\n date_trunc(c.partition_trunc, lower(_dtrange)),\n date_trunc(c.partition_trunc, lower(_dtrange)) + (concat('1 ', c.partition_trunc))::interval,\n '[)'\n );\n ELSE\n _partition_dtrange := '[-infinity, infinity]'::tstzrange;\n END IF;\n\n IF NOT _partition_dtrange @> _dtrange THEN\n RAISE EXCEPTION 'dtrange % is greater than the partition size % for collection %', _dtrange, c.partition_trunc, _collection;\n END IF;\n\n\n IF c.partition_trunc = 'year' THEN\n _partition_name := format('_items_%s_%s', c.key, to_char(lower(_partition_dtrange),'YYYY'));\n ELSIF c.partition_trunc = 'month' THEN\n _partition_name := format('_items_%s_%s', c.key, to_char(lower(_partition_dtrange),'YYYYMM'));\n ELSE\n _partition_name := format('_items_%s', c.key);\n END IF;\n\n SELECT * INTO pm FROM partition_sys_meta WHERE collection=_collection AND partition_dtrange @> _dtrange;\n IF FOUND THEN\n RAISE NOTICE '% % %', _edtrange, _dtrange, pm;\n _constraint_edtrange :=\n tstzrange(\n least(\n lower(_edtrange),\n nullif(lower(pm.constraint_edtrange), '-infinity')\n ),\n greatest(\n upper(_edtrange),\n nullif(upper(pm.constraint_edtrange), 'infinity')\n ),\n '[]'\n );\n _constraint_dtrange :=\n tstzrange(\n least(\n lower(_dtrange),\n nullif(lower(pm.constraint_dtrange), '-infinity')\n ),\n greatest(\n upper(_dtrange),\n nullif(upper(pm.constraint_dtrange), 'infinity')\n ),\n '[]'\n );\n\n IF pm.constraint_edtrange @> _edtrange AND pm.constraint_dtrange @> _dtrange THEN\n RETURN pm.partition;\n ELSE\n PERFORM drop_table_constraints(_partition_name);\n END IF;\n ELSE\n _constraint_edtrange := _edtrange;\n _constraint_dtrange := _dtrange;\n END IF;\n RAISE NOTICE 'Creating partition % %', _partition_name, _partition_dtrange;\n IF c.partition_trunc IS NULL THEN\n q := format(\n $q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n $q$,\n _partition_name,\n _collection,\n concat(_partition_name,'_pk'),\n _partition_name\n );\n ELSE\n q := format(\n $q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L) PARTITION BY RANGE (datetime);\n CREATE TABLE IF NOT EXISTS %I partition OF %I FOR VALUES FROM (%L) TO (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n $q$,\n format('_items_%s', c.key),\n _collection,\n _partition_name,\n format('_items_%s', c.key),\n lower(_partition_dtrange),\n upper(_partition_dtrange),\n format('%s_pk', _partition_name),\n _partition_name\n );\n END IF;\n\n BEGIN\n EXECUTE q;\n EXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', _partition_name;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\n END;\n PERFORM create_table_constraints(_partition_name, _constraint_dtrange, _constraint_edtrange);\n PERFORM maintain_partitions(_partition_name);\n PERFORM update_partition_stats_q(_partition_name, true);\n RETURN _partition_name;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION repartition(_collection text, _partition_trunc text, triggered boolean DEFAULT FALSE) RETURNS text AS $$\nDECLARE\n c RECORD;\n q text;\n from_trunc text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=_collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', _collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n IF triggered THEN\n RAISE NOTICE 'Converting % to % partitioning via Trigger', _collection, _partition_trunc;\n ELSE\n RAISE NOTICE 'Converting % from using % to % partitioning', _collection, c.partition_trunc, _partition_trunc;\n IF c.partition_trunc IS NOT DISTINCT FROM _partition_trunc THEN\n RAISE NOTICE 'Collection % already set to use partition by %', _collection, _partition_trunc;\n RETURN _collection;\n END IF;\n END IF;\n\n IF EXISTS (SELECT 1 FROM partitions WHERE collection=_collection LIMIT 1) THEN\n EXECUTE format(\n $q$\n CREATE TEMP TABLE changepartitionstaging ON COMMIT DROP AS SELECT * FROM %I;\n DROP TABLE IF EXISTS %I CASCADE;\n WITH p AS (\n SELECT\n collection,\n CASE WHEN %L IS NULL THEN '-infinity'::timestamptz\n ELSE date_trunc(%L, datetime)\n END as d,\n tstzrange(min(datetime),max(datetime),'[]') as dtrange,\n tstzrange(min(datetime),max(datetime),'[]') as edtrange\n FROM changepartitionstaging\n GROUP BY 1,2\n ) SELECT check_partition(collection, dtrange, edtrange) FROM p;\n INSERT INTO items SELECT * FROM changepartitionstaging;\n DROP TABLE changepartitionstaging;\n $q$,\n concat('_items_', c.key),\n concat('_items_', c.key),\n c.partition_trunc,\n c.partition_trunc\n );\n END IF;\n RETURN _collection;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE OR REPLACE FUNCTION collections_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n q text;\n partition_name text := format('_items_%s', NEW.key);\n partition_exists boolean := false;\n partition_empty boolean := true;\n err_context text;\n loadtemp boolean := FALSE;\nBEGIN\n RAISE NOTICE 'Collection Trigger. % %', NEW.id, NEW.key;\n IF TG_OP = 'UPDATE' AND NEW.partition_trunc IS DISTINCT FROM OLD.partition_trunc THEN\n PERFORM repartition(NEW.id, NEW.partition_trunc, TRUE);\n END IF;\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER collections_trigger AFTER\nINSERT\nOR\nUPDATE ON collections\nFOR EACH ROW EXECUTE FUNCTION collections_trigger_func();\nCREATE OR REPLACE VIEW partition_steps AS\nSELECT\n partition as name,\n date_trunc('month',lower(partition_dtrange)) as sdate,\n date_trunc('month', upper(partition_dtrange)) + '1 month'::interval as edate\n FROM partitions WHERE partition_dtrange IS NOT NULL AND partition_dtrange != 'empty'::tstzrange\n ORDER BY dtrange ASC\n;\n\nCREATE OR REPLACE FUNCTION chunker(\n IN _where text,\n OUT s timestamptz,\n OUT e timestamptz\n) RETURNS SETOF RECORD AS $$\nDECLARE\n explain jsonb;\nBEGIN\n IF _where IS NULL THEN\n _where := ' TRUE ';\n END IF;\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s;', _where)\n INTO explain;\n\n RETURN QUERY\n WITH t AS (\n SELECT j->>0 as p FROM\n jsonb_path_query(\n explain,\n 'strict $.**.\"Relation Name\" ? (@ != null)'\n ) j\n ),\n parts AS (\n SELECT sdate, edate FROM t JOIN partition_steps ON (t.p = name)\n ),\n times AS (\n SELECT sdate FROM parts\n UNION\n SELECT edate FROM parts\n ),\n uniq AS (\n SELECT DISTINCT sdate FROM times ORDER BY sdate\n ),\n last AS (\n SELECT sdate, lead(sdate, 1) over () as edate FROM uniq\n )\n SELECT sdate, edate FROM last WHERE edate IS NOT NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION partition_queries(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN partitions text[] DEFAULT NULL\n) RETURNS SETOF text AS $$\nDECLARE\n query text;\n sdate timestamptz;\n edate timestamptz;\nBEGIN\nIF _where IS NULL OR trim(_where) = '' THEN\n _where = ' TRUE ';\nEND IF;\nRAISE NOTICE 'Getting chunks for % %', _where, _orderby;\nIF _orderby ILIKE 'datetime d%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 DESC LOOP\n RETURN NEXT format($q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n $q$,\n sdate,\n edate,\n _where,\n _orderby\n );\n END LOOP;\nELSIF _orderby ILIKE 'datetime a%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 ASC LOOP\n RETURN NEXT format($q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n $q$,\n sdate,\n edate,\n _where,\n _orderby\n );\n END LOOP;\nELSE\n query := format($q$\n SELECT * FROM items\n WHERE %s\n ORDER BY %s\n $q$, _where, _orderby\n );\n\n RETURN NEXT query;\n RETURN;\nEND IF;\n\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION partition_query_view(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN _limit int DEFAULT 10\n) RETURNS text AS $$\n WITH p AS (\n SELECT * FROM partition_queries(_where, _orderby) p\n )\n SELECT\n CASE WHEN EXISTS (SELECT 1 FROM p) THEN\n (SELECT format($q$\n SELECT * FROM (\n %s\n ) total LIMIT %s\n $q$,\n string_agg(\n format($q$ SELECT * FROM ( %s ) AS sub $q$, p),\n '\n UNION ALL\n '\n ),\n _limit\n ))\n ELSE NULL\n END FROM p;\n$$ LANGUAGE SQL IMMUTABLE;\n\n\n\n\nCREATE OR REPLACE FUNCTION stac_search_to_where(j jsonb) RETURNS text AS $$\nDECLARE\n where_segments text[];\n _where text;\n dtrange tstzrange;\n collections text[];\n geom geometry;\n sdate timestamptz;\n edate timestamptz;\n filterlang text;\n filter jsonb := j->'filter';\nBEGIN\n IF j ? 'ids' THEN\n where_segments := where_segments || format('id = ANY (%L) ', to_text_array(j->'ids'));\n END IF;\n\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n where_segments := where_segments || format('collection = ANY (%L) ', collections);\n END IF;\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n\n where_segments := where_segments || format(' datetime <= %L::timestamptz AND end_datetime >= %L::timestamptz ',\n edate,\n sdate\n );\n END IF;\n\n geom := stac_geom(j);\n IF geom IS NOT NULL THEN\n where_segments := where_segments || format('st_intersects(geometry, %L)',geom);\n END IF;\n\n filterlang := COALESCE(\n j->>'filter-lang',\n get_setting('default_filter_lang', j->'conf')\n );\n IF NOT filter @? '$.**.op' THEN\n filterlang := 'cql-json';\n END IF;\n\n IF filterlang NOT IN ('cql-json','cql2-json') AND j ? 'filter' THEN\n RAISE EXCEPTION '% is not a supported filter-lang. Please use cql-json or cql2-json.', filterlang;\n END IF;\n\n IF j ? 'query' AND j ? 'filter' THEN\n RAISE EXCEPTION 'Can only use either query or filter at one time.';\n END IF;\n\n IF j ? 'query' THEN\n filter := query_to_cql2(j->'query');\n ELSIF filterlang = 'cql-json' THEN\n filter := cql1_to_cql2(filter);\n END IF;\n RAISE NOTICE 'FILTER: %', filter;\n where_segments := where_segments || cql2_query(filter);\n IF cardinality(where_segments) < 1 THEN\n RETURN ' TRUE ';\n END IF;\n\n _where := array_to_string(array_remove(where_segments, NULL), ' AND ');\n\n IF _where IS NULL OR BTRIM(_where) = '' THEN\n RETURN ' TRUE ';\n END IF;\n RETURN _where;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION parse_sort_dir(_dir text, reverse boolean default false) RETURNS text AS $$\n WITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n ) SELECT\n CASE\n WHEN NOT reverse THEN d\n WHEN d = 'ASC' THEN 'DESC'\n WHEN d = 'DESC' THEN 'ASC'\n END\n FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION sort_dir_to_op(_dir text, prev boolean default false) RETURNS text AS $$\n WITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n ) SELECT\n CASE\n WHEN d = 'ASC' AND prev THEN '<='\n WHEN d = 'DESC' AND prev THEN '>='\n WHEN d = 'ASC' THEN '>='\n WHEN d = 'DESC' THEN '<='\n END\n FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION sort_sqlorderby(\n _search jsonb DEFAULT NULL,\n reverse boolean DEFAULT FALSE\n) RETURNS text AS $$\n WITH sortby AS (\n SELECT coalesce(_search->'sortby','[{\"field\":\"datetime\", \"direction\":\"desc\"}]') as sort\n ), withid AS (\n SELECT CASE\n WHEN sort @? '$[*] ? (@.field == \"id\")' THEN sort\n ELSE sort || '[{\"field\":\"id\", \"direction\":\"desc\"}]'::jsonb\n END as sort\n FROM sortby\n ), withid_rows AS (\n SELECT jsonb_array_elements(sort) as value FROM withid\n ),sorts AS (\n SELECT\n coalesce(\n (queryable(value->>'field')).expression\n ) as key,\n parse_sort_dir(value->>'direction', reverse) as dir\n FROM withid_rows\n )\n SELECT array_to_string(\n array_agg(concat(key, ' ', dir)),\n ', '\n ) FROM sorts;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_sort_dir(sort_item jsonb) RETURNS text AS $$\n SELECT CASE WHEN sort_item->>'direction' ILIKE 'desc%' THEN 'DESC' ELSE 'ASC' END;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION get_token_val_str(\n _field text,\n _item items\n) RETURNS text AS $$\nDECLARE\n q text;\n literal text;\nBEGIN\n q := format($q$ SELECT quote_literal(%s) FROM (SELECT $1.*) as r;$q$, _field);\n EXECUTE q INTO literal USING _item;\n RETURN literal;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n\nCREATE OR REPLACE FUNCTION get_token_record(IN _token text, OUT prev BOOLEAN, OUT item items) RETURNS RECORD AS $$\nDECLARE\n _itemid text := _token;\n _collectionid text;\nBEGIN\n IF _token IS NULL THEN\n RETURN;\n END IF;\n RAISE NOTICE 'Looking for token: %', _token;\n prev := FALSE;\n IF _token ILIKE 'prev:%' THEN\n _itemid := replace(_token, 'prev:','');\n prev := TRUE;\n ELSIF _token ILIKE 'next:%' THEN\n _itemid := replace(_token, 'next:', '');\n END IF;\n SELECT id INTO _collectionid FROM collections WHERE _itemid LIKE concat(id,':%');\n IF FOUND THEN\n _itemid := replace(_itemid, concat(_collectionid,':'), '');\n SELECT * INTO item FROM items WHERE id=_itemid AND collection=_collectionid;\n ELSE\n SELECT * INTO item FROM items WHERE id=_itemid;\n END IF;\n IF item IS NULL THEN\n RAISE EXCEPTION 'Could not find item using token: % item: % collection: %', _token, _itemid, _collectionid;\n END IF;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\n\nCREATE OR REPLACE FUNCTION get_token_filter(\n _sortby jsonb DEFAULT '[{\"field\":\"datetime\",\"direction\":\"desc\"}]'::jsonb,\n token_item items DEFAULT NULL,\n prev boolean DEFAULT FALSE,\n inclusive boolean DEFAULT FALSE\n) RETURNS text AS $$\nDECLARE\n ltop text := '<';\n gtop text := '>';\n dir text;\n sort record;\n orfilter text := '';\n orfilters text[] := '{}'::text[];\n andfilters text[] := '{}'::text[];\n output text;\n token_where text;\nBEGIN\n _sortby := _sortby || jsonb_build_object('field','id','direction',_sortby->0->>'direction');\n RAISE NOTICE 'Getting Token Filter. % %', _sortby, token_item;\n IF inclusive THEN\n orfilters := orfilters || format('( id=%L AND collection=%L )' , token_item.id, token_item.collection);\n END IF;\n\n FOR sort IN\n WITH s1 AS (\n SELECT\n _row,\n (queryable(value->>'field')).expression as _field,\n (value->>'field' = 'id') as _isid,\n get_sort_dir(value) as _dir\n FROM jsonb_array_elements(_sortby)\n WITH ORDINALITY AS t(value, _row)\n )\n SELECT\n _row,\n _field,\n _dir,\n get_token_val_str(_field, token_item) as _val\n FROM s1\n WHERE _row <= (SELECT min(_row) FROM s1 WHERE _isid)\n LOOP\n orfilter := NULL;\n RAISE NOTICE 'SORT: %', sort;\n IF sort._val IS NOT NULL AND ((prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC')) THEN\n orfilter := format($f$(\n (%s %s %s) OR (%s IS NULL)\n )$f$,\n sort._field,\n ltop,\n sort._val,\n sort._val\n );\n ELSIF sort._val IS NULL AND ((prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC')) THEN\n RAISE NOTICE '< but null';\n orfilter := format('%s IS NOT NULL', sort._field);\n ELSIF sort._val IS NULL THEN\n RAISE NOTICE '> but null';\n ELSE\n orfilter := format($f$(\n (%s %s %s) OR (%s IS NULL)\n )$f$,\n sort._field,\n gtop,\n sort._val,\n sort._field\n );\n END IF;\n RAISE NOTICE 'ORFILTER: %', orfilter;\n\n IF orfilter IS NOT NULL THEN\n IF sort._row = 1 THEN\n orfilters := orfilters || orfilter;\n ELSE\n orfilters := orfilters || format('(%s AND %s)', array_to_string(andfilters, ' AND '), orfilter);\n END IF;\n END IF;\n IF sort._val IS NOT NULL THEN\n andfilters := andfilters || format('%s = %s', sort._field, sort._val);\n ELSE\n andfilters := andfilters || format('%s IS NULL', sort._field);\n END IF;\n END LOOP;\n\n output := array_to_string(orfilters, ' OR ');\n\n token_where := concat('(',coalesce(output,'true'),')');\n IF trim(token_where) = '' THEN\n token_where := NULL;\n END IF;\n RAISE NOTICE 'TOKEN_WHERE: %',token_where;\n RETURN token_where;\n END;\n$$ LANGUAGE PLPGSQL SET transform_null_equals TO TRUE\n;\n\nCREATE OR REPLACE FUNCTION search_tohash(jsonb) RETURNS jsonb AS $$\n SELECT $1 - '{token,limit,context,includes,excludes}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION search_hash(jsonb, jsonb) RETURNS text AS $$\n SELECT md5(concat(search_tohash($1)::text,$2::text));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TABLE IF NOT EXISTS searches(\n hash text GENERATED ALWAYS AS (search_hash(search, metadata)) STORED PRIMARY KEY,\n search jsonb NOT NULL,\n _where text,\n orderby text,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n metadata jsonb DEFAULT '{}'::jsonb NOT NULL\n);\n\nCREATE TABLE IF NOT EXISTS search_wheres(\n id bigint generated always as identity primary key,\n _where text NOT NULL,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n statslastupdated timestamptz,\n estimated_count bigint,\n estimated_cost float,\n time_to_estimate float,\n total_count bigint,\n time_to_count float,\n partitions text[]\n);\n\nCREATE INDEX IF NOT EXISTS search_wheres_partitions ON search_wheres USING GIN (partitions);\nCREATE UNIQUE INDEX IF NOT EXISTS search_wheres_where ON search_wheres ((md5(_where)));\n\nCREATE OR REPLACE FUNCTION where_stats(inwhere text, updatestats boolean default false, conf jsonb default null) RETURNS search_wheres AS $$\nDECLARE\n t timestamptz;\n i interval;\n explain_json jsonb;\n partitions text[];\n sw search_wheres%ROWTYPE;\n inwhere_hash text := md5(inwhere);\n _context text := lower(context(conf));\n _stats_ttl interval := context_stats_ttl(conf);\n _estimated_cost float := context_estimated_cost(conf);\n _estimated_count int := context_estimated_count(conf);\nBEGIN\n IF _context = 'off' THEN\n sw._where := inwhere;\n return sw;\n END IF;\n\n SELECT * INTO sw FROM search_wheres WHERE md5(_where)=inwhere_hash FOR UPDATE;\n\n -- Update statistics if explicitly set, if statistics do not exist, or statistics ttl has expired\n IF NOT updatestats THEN\n RAISE NOTICE 'Checking if update is needed for: % .', inwhere;\n RAISE NOTICE 'Stats Last Updated: %', sw.statslastupdated;\n RAISE NOTICE 'TTL: %, Age: %', _stats_ttl, now() - sw.statslastupdated;\n RAISE NOTICE 'Context: %, Existing Total: %', _context, sw.total_count;\n IF\n sw.statslastupdated IS NULL\n OR (now() - sw.statslastupdated) > _stats_ttl\n OR (context(conf) != 'off' AND sw.total_count IS NULL)\n THEN\n updatestats := TRUE;\n END IF;\n END IF;\n\n sw._where := inwhere;\n sw.lastused := now();\n sw.usecount := coalesce(sw.usecount,0) + 1;\n\n IF NOT updatestats THEN\n UPDATE search_wheres SET\n lastused = sw.lastused,\n usecount = sw.usecount\n WHERE md5(_where) = inwhere_hash\n RETURNING * INTO sw\n ;\n RETURN sw;\n END IF;\n\n -- Use explain to get estimated count/cost and a list of the partitions that would be hit by the query\n t := clock_timestamp();\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s', inwhere)\n INTO explain_json;\n RAISE NOTICE 'Time for just the explain: %', clock_timestamp() - t;\n i := clock_timestamp() - t;\n\n sw.statslastupdated := now();\n sw.estimated_count := explain_json->0->'Plan'->'Plan Rows';\n sw.estimated_cost := explain_json->0->'Plan'->'Total Cost';\n sw.time_to_estimate := extract(epoch from i);\n\n RAISE NOTICE 'ESTIMATED_COUNT: % < %', sw.estimated_count, _estimated_count;\n RAISE NOTICE 'ESTIMATED_COST: % < %', sw.estimated_cost, _estimated_cost;\n\n -- Do a full count of rows if context is set to on or if auto is set and estimates are low enough\n IF\n _context = 'on'\n OR\n ( _context = 'auto' AND\n (\n sw.estimated_count < _estimated_count\n AND\n sw.estimated_cost < _estimated_cost\n )\n )\n THEN\n t := clock_timestamp();\n RAISE NOTICE 'Calculating actual count...';\n EXECUTE format(\n 'SELECT count(*) FROM items WHERE %s',\n inwhere\n ) INTO sw.total_count;\n i := clock_timestamp() - t;\n RAISE NOTICE 'Actual Count: % -- %', sw.total_count, i;\n sw.time_to_count := extract(epoch FROM i);\n ELSE\n sw.total_count := NULL;\n sw.time_to_count := NULL;\n END IF;\n\n\n INSERT INTO search_wheres\n (_where, lastused, usecount, statslastupdated, estimated_count, estimated_cost, time_to_estimate, partitions, total_count, time_to_count)\n SELECT sw._where, sw.lastused, sw.usecount, sw.statslastupdated, sw.estimated_count, sw.estimated_cost, sw.time_to_estimate, sw.partitions, sw.total_count, sw.time_to_count\n ON CONFLICT ((md5(_where)))\n DO UPDATE\n SET\n lastused = sw.lastused,\n usecount = sw.usecount,\n statslastupdated = sw.statslastupdated,\n estimated_count = sw.estimated_count,\n estimated_cost = sw.estimated_cost,\n time_to_estimate = sw.time_to_estimate,\n total_count = sw.total_count,\n time_to_count = sw.time_to_count\n ;\n RETURN sw;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION search_query(\n _search jsonb = '{}'::jsonb,\n updatestats boolean = false,\n _metadata jsonb = '{}'::jsonb\n) RETURNS searches AS $$\nDECLARE\n search searches%ROWTYPE;\n pexplain jsonb;\n t timestamptz;\n i interval;\nBEGIN\n SELECT * INTO search FROM searches\n WHERE hash=search_hash(_search, _metadata) FOR UPDATE;\n\n -- Calculate the where clause if not already calculated\n IF search._where IS NULL THEN\n search._where := stac_search_to_where(_search);\n END IF;\n\n -- Calculate the order by clause if not already calculated\n IF search.orderby IS NULL THEN\n search.orderby := sort_sqlorderby(_search);\n END IF;\n\n PERFORM where_stats(search._where, updatestats, _search->'conf');\n\n search.lastused := now();\n search.usecount := coalesce(search.usecount, 0) + 1;\n INSERT INTO searches (search, _where, orderby, lastused, usecount, metadata)\n VALUES (_search, search._where, search.orderby, search.lastused, search.usecount, _metadata)\n ON CONFLICT (hash) DO\n UPDATE SET\n _where = EXCLUDED._where,\n orderby = EXCLUDED.orderby,\n lastused = EXCLUDED.lastused,\n usecount = EXCLUDED.usecount,\n metadata = EXCLUDED.metadata\n RETURNING * INTO search\n ;\n RETURN search;\n\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE OR REPLACE FUNCTION search_rows(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN partitions text[] DEFAULT NULL,\n IN _limit int DEFAULT 10\n) RETURNS SETOF items AS $$\nDECLARE\n base_query text;\n query text;\n sdate timestamptz;\n edate timestamptz;\n n int;\n records_left int := _limit;\nBEGIN\nIF _where IS NULL OR trim(_where) = '' THEN\n _where = ' TRUE ';\nEND IF;\nRAISE NOTICE 'Getting chunks for % %', _where, _orderby;\n\nbase_query := $q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n LIMIT %L\n$q$;\n\nIF _orderby ILIKE 'datetime d%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 DESC LOOP\n RAISE NOTICE 'Running Query for % to %', sdate, edate;\n query := format(\n base_query,\n sdate,\n edate,\n _where,\n _orderby,\n records_left\n );\n RAISE LOG 'QUERY: %', query;\n RETURN QUERY EXECUTE query;\n\n GET DIAGNOSTICS n = ROW_COUNT;\n records_left := records_left - n;\n RAISE NOTICE 'Returned %/% Rows From % to %. % to go.', n, _limit, sdate, edate, records_left;\n IF records_left <= 0 THEN\n RETURN;\n END IF;\n END LOOP;\nELSIF _orderby ILIKE 'datetime a%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 ASC LOOP\n RAISE NOTICE 'Running Query for % to %', sdate, edate;\n query := format(\n base_query,\n sdate,\n edate,\n _where,\n _orderby,\n records_left\n );\n RAISE LOG 'QUERY: %', query;\n RETURN QUERY EXECUTE query;\n\n GET DIAGNOSTICS n = ROW_COUNT;\n records_left := records_left - n;\n RAISE NOTICE 'Returned %/% Rows From % to %. % to go.', n, _limit, sdate, edate, records_left;\n IF records_left <= 0 THEN\n RETURN;\n END IF;\n END LOOP;\nELSE\n query := format($q$\n SELECT * FROM items\n WHERE %s\n ORDER BY %s\n LIMIT %L\n $q$, _where, _orderby, _limit\n );\n RAISE LOG 'QUERY: %', query;\n RETURN QUERY EXECUTE query;\nEND IF;\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE UNLOGGED TABLE format_item_cache(\n id text,\n collection text,\n fields text,\n hydrated bool,\n output jsonb,\n lastused timestamptz DEFAULT now(),\n usecount int DEFAULT 1,\n timetoformat float,\n PRIMARY KEY (collection, id, fields, hydrated)\n);\nCREATE INDEX ON format_item_cache (lastused);\n\nCREATE OR REPLACE FUNCTION format_item(_item items, _fields jsonb DEFAULT '{}', _hydrated bool DEFAULT TRUE) RETURNS jsonb AS $$\nDECLARE\n cache bool := get_setting_bool('format_cache');\n _output jsonb := null;\n t timestamptz := clock_timestamp();\nBEGIN\n IF cache THEN\n SELECT output INTO _output FROM format_item_cache\n WHERE id=_item.id AND collection=_item.collection AND fields=_fields::text AND hydrated=_hydrated;\n END IF;\n IF _output IS NULL THEN\n IF _hydrated THEN\n _output := content_hydrate(_item, _fields);\n ELSE\n _output := content_nonhydrated(_item, _fields);\n END IF;\n END IF;\n IF cache THEN\n INSERT INTO format_item_cache (id, collection, fields, hydrated, output, timetoformat)\n VALUES (_item.id, _item.collection, _fields::text, _hydrated, _output, age_ms(t))\n ON CONFLICT(collection, id, fields, hydrated) DO\n UPDATE\n SET lastused=now(), usecount = format_item_cache.usecount + 1\n ;\n END IF;\n RETURN _output;\n\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION search(_search jsonb = '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n searches searches%ROWTYPE;\n _where text;\n orderby text;\n search_where search_wheres%ROWTYPE;\n total_count bigint;\n token record;\n token_prev boolean;\n token_item items%ROWTYPE;\n token_where text;\n full_where text;\n timer timestamptz := clock_timestamp();\n hydrate bool := NOT (_search->'conf'->>'nohydrate' IS NOT NULL AND (_search->'conf'->>'nohydrate')::boolean = true);\n prev text;\n next text;\n context jsonb;\n collection jsonb;\n out_records jsonb;\n out_len int;\n _limit int := coalesce((_search->>'limit')::int, 10);\n _querylimit int;\n _fields jsonb := coalesce(_search->'fields', '{}'::jsonb);\n has_prev boolean := FALSE;\n has_next boolean := FALSE;\nBEGIN\n searches := search_query(_search);\n _where := searches._where;\n orderby := searches.orderby;\n search_where := where_stats(_where);\n total_count := coalesce(search_where.total_count, search_where.estimated_count);\n RAISE NOTICE 'SEARCH:TOKEN: %', _search->>'token';\n token := get_token_record(_search->>'token');\n RAISE NOTICE '***TOKEN: %', token;\n _querylimit := _limit + 1;\n IF token IS NOT NULL THEN\n token_prev := token.prev;\n token_item := token.item;\n token_where := get_token_filter(_search->'sortby', token_item, token_prev, FALSE);\n RAISE LOG 'TOKEN_WHERE: %', token_where;\n IF token_prev THEN -- if we are using a prev token, we know has_next is true\n RAISE LOG 'There is a previous token, so automatically setting has_next to true';\n has_next := TRUE;\n orderby := sort_sqlorderby(_search, TRUE);\n ELSE\n RAISE LOG 'There is a next token, so automatically setting has_prev to true';\n has_prev := TRUE;\n\n END IF;\n ELSE -- if there was no token, we know there is no prev\n RAISE LOG 'There is no token, so we know there is no prev. setting has_prev to false';\n has_prev := FALSE;\n END IF;\n\n full_where := concat_ws(' AND ', _where, token_where);\n RAISE NOTICE 'FULL WHERE CLAUSE: %', full_where;\n RAISE NOTICE 'Time to get counts and build query %', age_ms(timer);\n timer := clock_timestamp();\n\n IF hydrate THEN\n RAISE NOTICE 'Getting hydrated data.';\n ELSE\n RAISE NOTICE 'Getting non-hydrated data.';\n END IF;\n RAISE NOTICE 'CACHE SET TO %', get_setting_bool('format_cache');\n SELECT jsonb_agg(format_item(i, _fields, hydrate)) INTO out_records\n FROM search_rows(\n full_where,\n orderby,\n search_where.partitions,\n _querylimit\n ) as i;\n\n RAISE NOTICE 'Time to fetch rows %', age_ms(timer);\n timer := clock_timestamp();\n\n\n IF token_prev THEN\n out_records := flip_jsonb_array(out_records);\n END IF;\n\n RAISE NOTICE 'Query returned % records.', jsonb_array_length(out_records);\n RAISE LOG 'TOKEN: % %', token_item.id, token_item.collection;\n RAISE LOG 'RECORD_1: % %', out_records->0->>'id', out_records->0->>'collection';\n RAISE LOG 'RECORD-1: % %', out_records->-1->>'id', out_records->-1->>'collection';\n\n -- REMOVE records that were from our token\n IF out_records->0->>'id' = token_item.id AND out_records->0->>'collection' = token_item.collection THEN\n out_records := out_records - 0;\n ELSIF out_records->-1->>'id' = token_item.id AND out_records->-1->>'collection' = token_item.collection THEN\n out_records := out_records - -1;\n END IF;\n\n out_len := jsonb_array_length(out_records);\n\n IF out_len = _limit + 1 THEN\n IF token_prev THEN\n has_prev := TRUE;\n out_records := out_records - 0;\n ELSE\n has_next := TRUE;\n out_records := out_records - -1;\n END IF;\n END IF;\n\n IF has_next THEN\n next := concat(out_records->-1->>'collection', ':', out_records->-1->>'id');\n RAISE NOTICE 'HAS NEXT | %', next;\n END IF;\n\n IF has_prev THEN\n prev := concat(out_records->0->>'collection', ':', out_records->0->>'id');\n RAISE NOTICE 'HAS PREV | %', prev;\n END IF;\n\n RAISE NOTICE 'Time to get prev/next %', age_ms(timer);\n timer := clock_timestamp();\n\n IF context(_search->'conf') != 'off' THEN\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'matched', total_count,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\n ELSE\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\n END IF;\n\n collection := jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb),\n 'next', next,\n 'prev', prev,\n 'context', context\n );\n\n RAISE NOTICE 'Time to build final json %', age_ms(timer);\n timer := clock_timestamp();\n\n RAISE NOTICE 'Total Time: %', age_ms(current_timestamp);\n RAISE NOTICE 'RETURNING % records. NEXT: %. PREV: %', collection->'context'->>'returned', collection->>'next', collection->>'prev';\n RETURN collection;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION search_cursor(_search jsonb = '{}'::jsonb) RETURNS refcursor AS $$\nDECLARE\n curs refcursor;\n searches searches%ROWTYPE;\n _where text;\n _orderby text;\n q text;\n\nBEGIN\n searches := search_query(_search);\n _where := searches._where;\n _orderby := searches.orderby;\n\n OPEN curs FOR\n WITH p AS (\n SELECT * FROM partition_queries(_where, _orderby) p\n )\n SELECT\n CASE WHEN EXISTS (SELECT 1 FROM p) THEN\n (SELECT format($q$\n SELECT * FROM (\n %s\n ) total\n $q$,\n string_agg(\n format($q$ SELECT * FROM ( %s ) AS sub $q$, p),\n '\n UNION ALL\n '\n )\n ))\n ELSE NULL\n END FROM p;\n RETURN curs;\nEND;\n$$ LANGUAGE PLPGSQL;\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION tileenvelope(zoom int, x int, y int) RETURNS geometry AS $$\nWITH t AS (\n SELECT\n 20037508.3427892 as merc_max,\n -20037508.3427892 as merc_min,\n (2 * 20037508.3427892) / (2 ^ zoom) as tile_size\n)\nSELECT st_makeenvelope(\n merc_min + (tile_size * x),\n merc_max - (tile_size * (y + 1)),\n merc_min + (tile_size * (x + 1)),\n merc_max - (tile_size * y),\n 3857\n) FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;DROP FUNCTION IF EXISTS mercgrid;\n\n\nCREATE OR REPLACE FUNCTION ftime() RETURNS interval as $$\nSELECT age(clock_timestamp(), transaction_timestamp());\n$$ LANGUAGE SQL;\nSET SEARCH_PATH to pgstac, public;\n\nDROP FUNCTION IF EXISTS geometrysearch;\nCREATE OR REPLACE FUNCTION geometrysearch(\n IN geom geometry,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE, -- Return as soon as the passed in geometry is full covered\n IN skipcovered boolean DEFAULT TRUE -- Skip any items that would show up completely under the previous items\n) RETURNS jsonb AS $$\nDECLARE\n search searches%ROWTYPE;\n curs refcursor;\n _where text;\n query text;\n iter_record items%ROWTYPE;\n out_records jsonb := '{}'::jsonb[];\n exit_flag boolean := FALSE;\n counter int := 1;\n scancounter int := 1;\n remaining_limit int := _scanlimit;\n tilearea float;\n unionedgeom geometry;\n clippedgeom geometry;\n unionedgeom_area float := 0;\n prev_area float := 0;\n excludes text[];\n includes text[];\n\nBEGIN\n DROP TABLE IF EXISTS pgstac_results;\n CREATE TEMP TABLE pgstac_results (content jsonb) ON COMMIT DROP;\n\n -- If skipcovered is true then you will always want to exit when the passed in geometry is full\n IF skipcovered THEN\n exitwhenfull := TRUE;\n END IF;\n\n SELECT * INTO search FROM searches WHERE hash=queryhash;\n\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Search with Query Hash % Not Found', queryhash;\n END IF;\n\n tilearea := st_area(geom);\n _where := format('%s AND st_intersects(geometry, %L::geometry)', search._where, geom);\n\n\n FOR query IN SELECT * FROM partition_queries(_where, search.orderby) LOOP\n query := format('%s LIMIT %L', query, remaining_limit);\n RAISE NOTICE '%', query;\n OPEN curs FOR EXECUTE query;\n LOOP\n FETCH curs INTO iter_record;\n EXIT WHEN NOT FOUND;\n IF exitwhenfull OR skipcovered THEN -- If we are not using exitwhenfull or skipcovered, we do not need to do expensive geometry operations\n clippedgeom := st_intersection(geom, iter_record.geometry);\n\n IF unionedgeom IS NULL THEN\n unionedgeom := clippedgeom;\n ELSE\n unionedgeom := st_union(unionedgeom, clippedgeom);\n END IF;\n\n unionedgeom_area := st_area(unionedgeom);\n\n IF skipcovered AND prev_area = unionedgeom_area THEN\n scancounter := scancounter + 1;\n CONTINUE;\n END IF;\n\n prev_area := unionedgeom_area;\n\n RAISE NOTICE '% % % %', unionedgeom_area/tilearea, counter, scancounter, ftime();\n END IF;\n RAISE NOTICE '% %', iter_record, content_hydrate(iter_record, fields);\n INSERT INTO pgstac_results (content) VALUES (content_hydrate(iter_record, fields));\n\n IF counter >= _limit\n OR scancounter > _scanlimit\n OR ftime() > _timelimit\n OR (exitwhenfull AND unionedgeom_area >= tilearea)\n THEN\n exit_flag := TRUE;\n EXIT;\n END IF;\n counter := counter + 1;\n scancounter := scancounter + 1;\n\n END LOOP;\n CLOSE curs;\n EXIT WHEN exit_flag;\n remaining_limit := _scanlimit - scancounter;\n END LOOP;\n\n SELECT jsonb_agg(content) INTO out_records FROM pgstac_results WHERE content IS NOT NULL;\n\n RETURN jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb)\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDROP FUNCTION IF EXISTS geojsonsearch;\nCREATE OR REPLACE FUNCTION geojsonsearch(\n IN geojson jsonb,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_geomfromgeojson(geojson),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS xyzsearch;\nCREATE OR REPLACE FUNCTION xyzsearch(\n IN _x int,\n IN _y int,\n IN _z int,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_transform(tileenvelope(_z, _x, _y), 4326),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE PROCEDURE analyze_items() AS $$\nDECLARE\n q text;\n timeout_ts timestamptz;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n SELECT format('ANALYZE (VERBOSE, SKIP_LOCKED) %I;', relname) INTO q\n FROM pg_stat_user_tables\n WHERE relname like '_item%' AND (n_mod_since_analyze>0 OR last_analyze IS NULL) LIMIT 1;\n IF NOT FOUND THEN\n EXIT;\n END IF;\n RAISE NOTICE '%', q;\n EXECUTE q;\n COMMIT;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE PROCEDURE validate_constraints() AS $$\nDECLARE\n q text;\nBEGIN\n FOR q IN\n SELECT\n FORMAT(\n 'ALTER TABLE %I.%I VALIDATE CONSTRAINT %I;',\n nsp.nspname,\n cls.relname,\n con.conname\n )\n\n FROM pg_constraint AS con\n JOIN pg_class AS cls\n ON con.conrelid = cls.oid\n JOIN pg_namespace AS nsp\n ON cls.relnamespace = nsp.oid\n WHERE convalidated = FALSE AND contype in ('c','f')\n AND nsp.nspname = 'pgstac'\n LOOP\n RAISE NOTICE '%', q;\n PERFORM run_or_queue(q);\n COMMIT;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION collection_extent(_collection text, runupdate boolean default false) RETURNS jsonb AS $$\nDECLARE\n geom_extent geometry;\n mind timestamptz;\n maxd timestamptz;\n extent jsonb;\nBEGIN\n IF runupdate THEN\n PERFORM update_partition_stats_q(partition)\n FROM partitions WHERE collection=_collection;\n END IF;\n SELECT\n min(lower(dtrange)),\n max(upper(edtrange)),\n st_extent(spatial)\n INTO\n mind,\n maxd,\n geom_extent\n FROM partitions\n WHERE collection=_collection;\n\n IF geom_extent IS NOT NULL AND mind IS NOT NULL AND maxd IS NOT NULL THEN\n extent := jsonb_build_object(\n 'extent', jsonb_build_object(\n 'spatial', jsonb_build_object(\n 'bbox', to_jsonb(array[array[st_xmin(geom_extent), st_ymin(geom_extent), st_xmax(geom_extent), st_ymax(geom_extent)]])\n ),\n 'temporal', jsonb_build_object(\n 'interval', to_jsonb(array[array[mind, maxd]])\n )\n )\n );\n RETURN extent;\n END IF;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('id', '{\"title\": \"Item ID\",\"description\": \"Item identifier\",\"$ref\": \"https://schemas.stacspec.org/v1.0.0/item-spec/json-schema/item.json#/definitions/core/allOf/2/properties/id\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('geometry', '{\"title\": \"Item Geometry\",\"description\": \"Item Geometry\",\"$ref\": \"https://geojson.org/schema/Feature.json\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('datetime','{\"description\": \"Datetime\",\"type\": \"string\",\"title\": \"Acquired\",\"format\": \"date-time\",\"pattern\": \"(\\\\+00:00|Z)$\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('eo:cloud_cover','{\"$ref\": \"https://stac-extensions.github.io/eo/v1.0.0/schema.json#/definitions/fieldsproperties/eo:cloud_cover\"}','to_int','BTREE');\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDELETE FROM queryables a USING queryables b\n WHERE a.name = b.name AND a.collection_ids IS NOT DISTINCT FROM b.collection_ids AND a.id > b.id;\n\n\nINSERT INTO pgstac_settings (name, value) VALUES\n ('context', 'off'),\n ('context_estimated_count', '100000'),\n ('context_estimated_cost', '100000'),\n ('context_stats_ttl', '1 day'),\n ('default_filter_lang', 'cql2-json'),\n ('additional_properties', 'true'),\n ('use_queue', 'false'),\n ('queue_timeout', '10 minutes'),\n ('update_collection_extent', 'false'),\n ('format_cache', 'false')\nON CONFLICT DO NOTHING\n;\n\nALTER FUNCTION to_text COST 5000;\nALTER FUNCTION to_float COST 5000;\nALTER FUNCTION to_int COST 5000;\nALTER FUNCTION to_tstz COST 5000;\nALTER FUNCTION to_text_array COST 5000;\n\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nGRANT ALL ON SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac to pgstac_admin;\n\n-- pgstac_read role limited to using function apis\nGRANT EXECUTE ON FUNCTION search TO pgstac_read;\nGRANT EXECUTE ON FUNCTION search_query TO pgstac_read;\nGRANT EXECUTE ON FUNCTION item_by_id TO pgstac_read;\nGRANT EXECUTE ON FUNCTION get_item TO pgstac_read;\nGRANT SELECT ON ALL TABLES IN SCHEMA pgstac TO pgstac_read;\n\n\nGRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_ingest;\nGRANT USAGE ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_ingest;\n\nSELECT update_partition_stats_q(partition) FROM partitions;\nSELECT set_version('0.7.4');\n" }, { "path": "src/pgstac/migrations/pgstac.0.7.5-0.7.6.sql", "content": "RESET ROLE;\nDO $$\nDECLARE\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='postgis') THEN\n CREATE EXTENSION IF NOT EXISTS postgis;\n END IF;\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='btree_gist') THEN\n CREATE EXTENSION IF NOT EXISTS btree_gist;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_admin;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_read;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_ingest;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n\nGRANT pgstac_admin TO current_user;\n\n-- Function to make sure pgstac_admin is the owner of items\nCREATE OR REPLACE FUNCTION pgstac_admin_owns() RETURNS VOID AS $$\nDECLARE\n f RECORD;\nBEGIN\n FOR f IN (\n SELECT\n concat(\n oid::regproc::text,\n '(',\n coalesce(pg_get_function_identity_arguments(oid),''),\n ')'\n ) AS name,\n CASE prokind WHEN 'f' THEN 'FUNCTION' WHEN 'p' THEN 'PROCEDURE' WHEN 'a' THEN 'AGGREGATE' END as typ\n FROM pg_proc\n WHERE\n pronamespace=to_regnamespace('pgstac')\n AND proowner != to_regrole('pgstac_admin')\n AND proname NOT LIKE 'pg_stat%'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n FOR f IN (\n SELECT\n oid::regclass::text as name,\n CASE relkind\n WHEN 'i' THEN 'INDEX'\n WHEN 'I' THEN 'INDEX'\n WHEN 'p' THEN 'TABLE'\n WHEN 'r' THEN 'TABLE'\n WHEN 'v' THEN 'VIEW'\n WHEN 'S' THEN 'SEQUENCE'\n ELSE NULL\n END as typ\n FROM pg_class\n WHERE relnamespace=to_regnamespace('pgstac') and relowner != to_regrole('pgstac_admin') AND relkind IN ('r','p','v','S') AND relname NOT LIKE 'pg_stat'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\nSELECT pgstac_admin_owns();\n\nCREATE SCHEMA IF NOT EXISTS pgstac AUTHORIZATION pgstac_admin;\n\nGRANT ALL ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_admin;\n\nALTER ROLE pgstac_admin SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_read SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_ingest SET SEARCH_PATH TO pgstac, public;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\n\nGRANT pgstac_read TO pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\n\nSET ROLE pgstac_admin;\n\nSET SEARCH_PATH TO pgstac, public;\n\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS analyze_items;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS validate_constraints;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\nSET client_min_messages TO WARNING;\nSET SEARCH_PATH to pgstac, public;\n-- BEGIN migra calculated SQL\nset check_function_bodies = off;\n\nCREATE OR REPLACE FUNCTION pgstac.queryables_constraint_triggerfunc()\n RETURNS trigger\n LANGUAGE plpgsql\nAS $function$\nDECLARE\n allcollections text[];\nBEGIN\n RAISE NOTICE 'Making sure that name/collection is unique for queryables %', NEW;\n IF NEW.collection_ids IS NOT NULL THEN\n IF EXISTS (\n SELECT 1\n FROM unnest(NEW.collection_ids) c\n LEFT JOIN\n collections\n ON (collections.id = c)\n WHERE collections.id IS NULL\n ) THEN\n RAISE foreign_key_violation USING MESSAGE = format(\n 'One or more collections in %s do not exist.', NEW.collection_ids\n );\n RETURN NULL;\n END IF;\n END IF;\n IF TG_OP = 'INSERT' THEN\n IF EXISTS (\n SELECT 1 FROM queryables q\n WHERE\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n )\n ) THEN\n RAISE unique_violation USING MESSAGE = format(\n 'There is already a queryable for %s for a collection in %s: %s',\n NEW.name,\n NEW.collection_ids,\n\t\t\t\t(SELECT json_agg(row_to_json(q)) FROM queryables q WHERE\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n ))\n );\n RETURN NULL;\n END IF;\n END IF;\n IF TG_OP = 'UPDATE' THEN\n IF EXISTS (\n SELECT 1 FROM queryables q\n WHERE\n q.id != NEW.id\n AND\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n )\n ) THEN\n RAISE unique_violation\n USING MESSAGE = format(\n 'There is already a queryable for %s for a collection in %s',\n NEW.name,\n NEW.collection_ids\n );\n RETURN NULL;\n END IF;\n END IF;\n\n RETURN NEW;\nEND;\n$function$\n;\n\n\n\n-- END migra calculated SQL\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('id', '{\"title\": \"Item ID\",\"description\": \"Item identifier\",\"$ref\": \"https://schemas.stacspec.org/v1.0.0/item-spec/json-schema/item.json#/definitions/core/allOf/2/properties/id\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('geometry', '{\"title\": \"Item Geometry\",\"description\": \"Item Geometry\",\"$ref\": \"https://geojson.org/schema/Feature.json\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('datetime','{\"description\": \"Datetime\",\"type\": \"string\",\"title\": \"Acquired\",\"format\": \"date-time\",\"pattern\": \"(\\\\+00:00|Z)$\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('eo:cloud_cover','{\"$ref\": \"https://stac-extensions.github.io/eo/v1.0.0/schema.json#/definitions/fieldsproperties/eo:cloud_cover\"}','to_int','BTREE');\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDELETE FROM queryables a USING queryables b\n WHERE a.name = b.name AND a.collection_ids IS NOT DISTINCT FROM b.collection_ids AND a.id > b.id;\n\n\nINSERT INTO pgstac_settings (name, value) VALUES\n ('context', 'off'),\n ('context_estimated_count', '100000'),\n ('context_estimated_cost', '100000'),\n ('context_stats_ttl', '1 day'),\n ('default_filter_lang', 'cql2-json'),\n ('additional_properties', 'true'),\n ('use_queue', 'false'),\n ('queue_timeout', '10 minutes'),\n ('update_collection_extent', 'false'),\n ('format_cache', 'false')\nON CONFLICT DO NOTHING\n;\n\nALTER FUNCTION to_text COST 5000;\nALTER FUNCTION to_float COST 5000;\nALTER FUNCTION to_int COST 5000;\nALTER FUNCTION to_tstz COST 5000;\nALTER FUNCTION to_text_array COST 5000;\n\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nGRANT ALL ON SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac to pgstac_admin;\n\n-- pgstac_read role limited to using function apis\nGRANT EXECUTE ON FUNCTION search TO pgstac_read;\nGRANT EXECUTE ON FUNCTION search_query TO pgstac_read;\nGRANT EXECUTE ON FUNCTION item_by_id TO pgstac_read;\nGRANT EXECUTE ON FUNCTION get_item TO pgstac_read;\nGRANT SELECT ON ALL TABLES IN SCHEMA pgstac TO pgstac_read;\n\n\nGRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_ingest;\nGRANT USAGE ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_ingest;\n\nSELECT update_partition_stats_q(partition) FROM partitions_view;\nSELECT set_version('0.7.6');\n" }, { "path": "src/pgstac/migrations/pgstac.0.7.5.sql", "content": "RESET ROLE;\nDO $$\nDECLARE\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='postgis') THEN\n CREATE EXTENSION IF NOT EXISTS postgis;\n END IF;\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='btree_gist') THEN\n CREATE EXTENSION IF NOT EXISTS btree_gist;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_admin;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_read;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_ingest;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n\nGRANT pgstac_admin TO current_user;\n\n-- Function to make sure pgstac_admin is the owner of items\nCREATE OR REPLACE FUNCTION pgstac_admin_owns() RETURNS VOID AS $$\nDECLARE\n f RECORD;\nBEGIN\n FOR f IN (\n SELECT\n concat(\n oid::regproc::text,\n '(',\n coalesce(pg_get_function_identity_arguments(oid),''),\n ')'\n ) AS name,\n CASE prokind WHEN 'f' THEN 'FUNCTION' WHEN 'p' THEN 'PROCEDURE' WHEN 'a' THEN 'AGGREGATE' END as typ\n FROM pg_proc\n WHERE\n pronamespace=to_regnamespace('pgstac')\n AND proowner != to_regrole('pgstac_admin')\n AND proname NOT LIKE 'pg_stat%'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n FOR f IN (\n SELECT\n oid::regclass::text as name,\n CASE relkind\n WHEN 'i' THEN 'INDEX'\n WHEN 'I' THEN 'INDEX'\n WHEN 'p' THEN 'TABLE'\n WHEN 'r' THEN 'TABLE'\n WHEN 'v' THEN 'VIEW'\n WHEN 'S' THEN 'SEQUENCE'\n ELSE NULL\n END as typ\n FROM pg_class\n WHERE relnamespace=to_regnamespace('pgstac') and relowner != to_regrole('pgstac_admin') AND relkind IN ('r','p','v','S') AND relname NOT LIKE 'pg_stat'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\nSELECT pgstac_admin_owns();\n\nCREATE SCHEMA IF NOT EXISTS pgstac AUTHORIZATION pgstac_admin;\n\nGRANT ALL ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_admin;\n\nALTER ROLE pgstac_admin SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_read SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_ingest SET SEARCH_PATH TO pgstac, public;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\n\nGRANT pgstac_read TO pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\n\nSET ROLE pgstac_admin;\n\nSET SEARCH_PATH TO pgstac, public;\n\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS analyze_items;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS validate_constraints;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\nCREATE OR REPLACE FUNCTION to_int(jsonb) RETURNS int AS $$\n SELECT floor(($1->>0)::float)::int;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000;\n\nCREATE OR REPLACE FUNCTION to_float(jsonb) RETURNS float AS $$\n SELECT ($1->>0)::float;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000;\n\nCREATE OR REPLACE FUNCTION to_tstz(jsonb) RETURNS timestamptz AS $$\n SELECT ($1->>0)::timestamptz;\n$$ LANGUAGE SQL IMMUTABLE STRICT SET TIME ZONE 'UTC' COST 5000;\n\n\nCREATE OR REPLACE FUNCTION to_text(jsonb) RETURNS text AS $$\n SELECT CASE WHEN jsonb_typeof($1) IN ('array','object') THEN $1::text ELSE $1->>0 END;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000;\n\nCREATE OR REPLACE FUNCTION to_text_array(jsonb) RETURNS text[] AS $$\n SELECT\n CASE jsonb_typeof($1)\n WHEN 'array' THEN ARRAY(SELECT jsonb_array_elements_text($1))\n ELSE ARRAY[$1->>0]\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000;\n\nCREATE OR REPLACE FUNCTION bbox_geom(_bbox jsonb) RETURNS geometry AS $$\nSELECT CASE jsonb_array_length(_bbox)\n WHEN 4 THEN\n ST_SetSRID(ST_MakeEnvelope(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float,\n (_bbox->>3)::float\n ),4326)\n WHEN 6 THEN\n ST_SetSRID(ST_3DMakeBox(\n ST_MakePoint(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float\n ),\n ST_MakePoint(\n (_bbox->>3)::float,\n (_bbox->>4)::float,\n (_bbox->>5)::float\n )\n ),4326)\n ELSE null END;\n;\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION geom_bbox(_geom geometry) RETURNS jsonb AS $$\n SELECT jsonb_build_array(\n st_xmin(_geom),\n st_ymin(_geom),\n st_xmax(_geom),\n st_ymax(_geom)\n );\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION flip_jsonb_array(j jsonb) RETURNS jsonb AS $$\n SELECT jsonb_agg(value) FROM (SELECT value FROM jsonb_array_elements(j) WITH ORDINALITY ORDER BY ordinality DESC) as t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION explode_dotpaths(j jsonb) RETURNS SETOF text[] AS $$\n SELECT string_to_array(p, '.') as e FROM jsonb_array_elements_text(j) p;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION explode_dotpaths_recurse(IN j jsonb) RETURNS SETOF text[] AS $$\n WITH RECURSIVE t AS (\n SELECT e FROM explode_dotpaths(j) e\n UNION ALL\n SELECT e[1:cardinality(e)-1]\n FROM t\n WHERE cardinality(e)>1\n ) SELECT e FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION jsonb_set_nested(j jsonb, path text[], val jsonb) RETURNS jsonb AS $$\nDECLARE\nBEGIN\n IF cardinality(path) > 1 THEN\n FOR i IN 1..(cardinality(path)-1) LOOP\n IF j #> path[:i] IS NULL THEN\n j := jsonb_set_lax(j, path[:i], '{}', TRUE);\n END IF;\n END LOOP;\n END IF;\n RETURN jsonb_set_lax(j, path, val, true);\n\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\n\n\nCREATE OR REPLACE FUNCTION jsonb_include(j jsonb, f jsonb) RETURNS jsonb AS $$\nDECLARE\n includes jsonb := f-> 'include';\n outj jsonb := '{}'::jsonb;\n path text[];\nBEGIN\n IF\n includes IS NULL\n OR jsonb_array_length(includes) = 0\n THEN\n RETURN j;\n ELSE\n includes := includes || '[\"id\",\"collection\"]'::jsonb;\n FOR path IN SELECT explode_dotpaths(includes) LOOP\n outj := jsonb_set_nested(outj, path, j #> path);\n END LOOP;\n END IF;\n RETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_exclude(j jsonb, f jsonb) RETURNS jsonb AS $$\nDECLARE\n excludes jsonb := f-> 'exclude';\n outj jsonb := j;\n path text[];\nBEGIN\n IF\n excludes IS NULL\n OR jsonb_array_length(excludes) = 0\n THEN\n RETURN j;\n ELSE\n FOR path IN SELECT explode_dotpaths(excludes) LOOP\n outj := outj #- path;\n END LOOP;\n END IF;\n RETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_fields(j jsonb, f jsonb DEFAULT '{\"fields\":[]}') RETURNS jsonb AS $$\n SELECT jsonb_exclude(jsonb_include(j, f), f);\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION merge_jsonb(_a jsonb, _b jsonb) RETURNS jsonb AS $$\n SELECT\n CASE\n WHEN _a = '\"𒍟※\"'::jsonb THEN NULL\n WHEN _a IS NULL OR jsonb_typeof(_a) = 'null' THEN _b\n WHEN jsonb_typeof(_a) = 'object' AND jsonb_typeof(_b) = 'object' THEN\n (\n SELECT\n jsonb_strip_nulls(\n jsonb_object_agg(\n key,\n merge_jsonb(a.value, b.value)\n )\n )\n FROM\n jsonb_each(coalesce(_a,'{}'::jsonb)) as a\n FULL JOIN\n jsonb_each(coalesce(_b,'{}'::jsonb)) as b\n USING (key)\n )\n WHEN\n jsonb_typeof(_a) = 'array'\n AND jsonb_typeof(_b) = 'array'\n AND jsonb_array_length(_a) = jsonb_array_length(_b)\n THEN\n (\n SELECT jsonb_agg(m) FROM\n ( SELECT\n merge_jsonb(\n jsonb_array_elements(_a),\n jsonb_array_elements(_b)\n ) as m\n ) as l\n )\n ELSE _a\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION strip_jsonb(_a jsonb, _b jsonb) RETURNS jsonb AS $$\n SELECT\n CASE\n\n WHEN (_a IS NULL OR jsonb_typeof(_a) = 'null') AND _b IS NOT NULL AND jsonb_typeof(_b) != 'null' THEN '\"𒍟※\"'::jsonb\n WHEN _b IS NULL OR jsonb_typeof(_a) = 'null' THEN _a\n WHEN _a = _b AND jsonb_typeof(_a) = 'object' THEN '{}'::jsonb\n WHEN _a = _b THEN NULL\n WHEN jsonb_typeof(_a) = 'object' AND jsonb_typeof(_b) = 'object' THEN\n (\n SELECT\n jsonb_strip_nulls(\n jsonb_object_agg(\n key,\n strip_jsonb(a.value, b.value)\n )\n )\n FROM\n jsonb_each(_a) as a\n FULL JOIN\n jsonb_each(_b) as b\n USING (key)\n )\n WHEN\n jsonb_typeof(_a) = 'array'\n AND jsonb_typeof(_b) = 'array'\n AND jsonb_array_length(_a) = jsonb_array_length(_b)\n THEN\n (\n SELECT jsonb_agg(m) FROM\n ( SELECT\n strip_jsonb(\n jsonb_array_elements(_a),\n jsonb_array_elements(_b)\n ) as m\n ) as l\n )\n ELSE _a\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION nullif_jsonbnullempty(j jsonb) RETURNS jsonb AS $$\n SELECT nullif(nullif(nullif(j,'null'::jsonb),'{}'::jsonb),'[]'::jsonb);\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION jsonb_array_unique(j jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(jsonb_agg(DISTINCT a)) v FROM jsonb_array_elements(j) a;\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_concat_ignorenull(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT coalesce(a,'[]'::jsonb) || coalesce(b,'[]'::jsonb);\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_least(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(least(nullif_jsonbnullempty(a), nullif_jsonbnullempty(b)));\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_greatest(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(greatest(a, b));\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION first_notnull_sfunc(anyelement, anyelement) RETURNS anyelement AS $$\n SELECT COALESCE($1,$2);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE AGGREGATE first_notnull(anyelement)(\n SFUNC = first_notnull_sfunc,\n STYPE = anyelement\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_array_unique_merge(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_concat_ignorenull,\n FINALFUNC = jsonb_array_unique\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_min(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_least\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_max(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_greatest\n);\n\nCREATE TABLE IF NOT EXISTS migrations (\n version text PRIMARY KEY,\n datetime timestamptz DEFAULT clock_timestamp() NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION get_version() RETURNS text AS $$\n SELECT version FROM pgstac.migrations ORDER BY datetime DESC, version DESC LIMIT 1;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION set_version(text) RETURNS text AS $$\n INSERT INTO pgstac.migrations (version) VALUES ($1)\n ON CONFLICT DO NOTHING\n RETURNING version;\n$$ LANGUAGE SQL;\n\n\nCREATE TABLE IF NOT EXISTS pgstac_settings (\n name text PRIMARY KEY,\n value text NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION table_empty(text) RETURNS boolean AS $$\nDECLARE\n retval boolean;\nBEGIN\n EXECUTE format($q$\n SELECT NOT EXISTS (SELECT 1 FROM %I LIMIT 1)\n $q$,\n $1\n ) INTO retval;\n RETURN retval;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION get_setting(IN _setting text, IN conf jsonb DEFAULT NULL) RETURNS text AS $$\nSELECT COALESCE(\n nullif(conf->>_setting, ''),\n nullif(current_setting(concat('pgstac.',_setting), TRUE),''),\n nullif((SELECT value FROM pgstac.pgstac_settings WHERE name=_setting),'')\n);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_setting_bool(IN _setting text, IN conf jsonb DEFAULT NULL) RETURNS boolean AS $$\nSELECT COALESCE(\n conf->>_setting,\n current_setting(concat('pgstac.',_setting), TRUE),\n (SELECT value FROM pgstac.pgstac_settings WHERE name=_setting),\n 'FALSE'\n)::boolean;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context(conf jsonb DEFAULT NULL) RETURNS text AS $$\n SELECT pgstac.get_setting('context', conf);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context_estimated_count(conf jsonb DEFAULT NULL) RETURNS int AS $$\n SELECT pgstac.get_setting('context_estimated_count', conf)::int;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_estimated_cost();\nCREATE OR REPLACE FUNCTION context_estimated_cost(conf jsonb DEFAULT NULL) RETURNS float AS $$\n SELECT pgstac.get_setting('context_estimated_cost', conf)::float;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_stats_ttl();\nCREATE OR REPLACE FUNCTION context_stats_ttl(conf jsonb DEFAULT NULL) RETURNS interval AS $$\n SELECT pgstac.get_setting('context_stats_ttl', conf)::interval;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION t2s(text) RETURNS text AS $$\n SELECT extract(epoch FROM $1::interval)::text || ' s';\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE STRICT;\n\nCREATE OR REPLACE FUNCTION age_ms(a timestamptz, b timestamptz DEFAULT clock_timestamp()) RETURNS float AS $$\n SELECT abs(extract(epoch from age(a,b)) * 1000);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION queue_timeout() RETURNS interval AS $$\n SELECT t2s(coalesce(\n get_setting('queue_timeout'),\n '1h'\n ))::interval;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION notice(VARIADIC text[]) RETURNS boolean AS $$\nDECLARE\ndebug boolean := current_setting('pgstac.debug', true);\nBEGIN\n IF debug THEN\n RAISE NOTICE 'NOTICE FROM FUNC: % >>>>> %', concat_ws(' | ', $1), clock_timestamp();\n RETURN TRUE;\n END IF;\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION empty_arr(ANYARRAY) RETURNS BOOLEAN AS $$\nSELECT CASE\n WHEN $1 IS NULL THEN TRUE\n WHEN cardinality($1)<1 THEN TRUE\nELSE FALSE\nEND;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_intersection(_a ANYARRAY, _b ANYARRAY) RETURNS ANYARRAY AS $$\n SELECT ARRAY ( SELECT unnest(_a) INTERSECT SELECT UNNEST(_b) );\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION array_map_ident(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_ident(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_map_literal(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_literal(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION array_reverse(anyarray) RETURNS anyarray AS $$\nSELECT ARRAY(\n SELECT $1[i]\n FROM generate_subscripts($1,1) AS s(i)\n ORDER BY i DESC\n);\n$$ LANGUAGE SQL STRICT IMMUTABLE;\n\nDROP TABLE IF EXISTS query_queue;\nCREATE TABLE query_queue (\n query text PRIMARY KEY,\n added timestamptz DEFAULT now()\n);\n\nDROP TABLE IF EXISTS query_queue_history;\nCREATE TABLE query_queue_history(\n query text,\n added timestamptz NOT NULL,\n finished timestamptz NOT NULL DEFAULT now(),\n error text\n);\n\nCREATE OR REPLACE PROCEDURE run_queued_queries() AS $$\nDECLARE\n qitem query_queue%ROWTYPE;\n timeout_ts timestamptz;\n error text;\n cnt int := 0;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n DELETE FROM query_queue WHERE query = (SELECT query FROM query_queue ORDER BY added DESC LIMIT 1 FOR UPDATE SKIP LOCKED) RETURNING * INTO qitem;\n IF NOT FOUND THEN\n EXIT;\n END IF;\n cnt := cnt + 1;\n BEGIN\n RAISE NOTICE 'RUNNING QUERY: %', qitem.query;\n EXECUTE qitem.query;\n EXCEPTION WHEN others THEN\n error := format('%s | %s', SQLERRM, SQLSTATE);\n END;\n INSERT INTO query_queue_history (query, added, finished, error)\n VALUES (qitem.query, qitem.added, clock_timestamp(), error);\n COMMIT;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION run_queued_queries_intransaction() RETURNS int AS $$\nDECLARE\n qitem query_queue%ROWTYPE;\n timeout_ts timestamptz;\n error text;\n cnt int := 0;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n DELETE FROM query_queue WHERE query = (SELECT query FROM query_queue ORDER BY added DESC LIMIT 1 FOR UPDATE SKIP LOCKED) RETURNING * INTO qitem;\n IF NOT FOUND THEN\n RETURN cnt;\n END IF;\n cnt := cnt + 1;\n BEGIN\n qitem.query := regexp_replace(qitem.query, 'CONCURRENTLY', '');\n RAISE NOTICE 'RUNNING QUERY: %', qitem.query;\n\n EXECUTE qitem.query;\n EXCEPTION WHEN others THEN\n error := format('%s | %s', SQLERRM, SQLSTATE);\n RAISE WARNING '%', error;\n END;\n INSERT INTO query_queue_history (query, added, finished, error)\n VALUES (qitem.query, qitem.added, clock_timestamp(), error);\n END LOOP;\n RETURN cnt;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION run_or_queue(query text) RETURNS VOID AS $$\nDECLARE\n use_queue text := COALESCE(get_setting('use_queue'), 'FALSE')::boolean;\nBEGIN\n IF get_setting_bool('debug') THEN\n RAISE NOTICE '%', query;\n END IF;\n IF use_queue THEN\n INSERT INTO query_queue (query) VALUES (query) ON CONFLICT DO NOTHING;\n ELSE\n EXECUTE query;\n END IF;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n\nDROP FUNCTION IF EXISTS check_pgstac_settings;\nCREATE OR REPLACE FUNCTION check_pgstac_settings(_sysmem text DEFAULT NULL) RETURNS VOID AS $$\nDECLARE\n settingval text;\n sysmem bigint := pg_size_bytes(_sysmem);\n effective_cache_size bigint := pg_size_bytes(current_setting('effective_cache_size', TRUE));\n shared_buffers bigint := pg_size_bytes(current_setting('shared_buffers', TRUE));\n work_mem bigint := pg_size_bytes(current_setting('work_mem', TRUE));\n max_connections int := current_setting('max_connections', TRUE);\n maintenance_work_mem bigint := pg_size_bytes(current_setting('maintenance_work_mem', TRUE));\n seq_page_cost float := current_setting('seq_page_cost', TRUE);\n random_page_cost float := current_setting('random_page_cost', TRUE);\n temp_buffers bigint := pg_size_bytes(current_setting('temp_buffers', TRUE));\n r record;\nBEGIN\n IF _sysmem IS NULL THEN\n RAISE NOTICE 'Call function with the size of your system memory `SELECT check_pgstac_settings(''4GB'')` to get pg system setting recommendations.';\n ELSE\n IF effective_cache_size < (sysmem * 0.5) THEN\n RAISE WARNING 'effective_cache_size of % is set low for a system with %. Recomended value between % and %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.5), pg_size_pretty(sysmem * 0.75);\n ELSIF effective_cache_size > (sysmem * 0.75) THEN\n RAISE WARNING 'effective_cache_size of % is set high for a system with %. Recomended value between % and %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.5), pg_size_pretty(sysmem * 0.75);\n ELSE\n RAISE NOTICE 'effective_cache_size of % is set appropriately for a system with %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem);\n END IF;\n\n IF shared_buffers < (sysmem * 0.2) THEN\n RAISE WARNING 'shared_buffers of % is set low for a system with %. Recomended value between % and %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.2), pg_size_pretty(sysmem * 0.3);\n ELSIF shared_buffers > (sysmem * 0.3) THEN\n RAISE WARNING 'shared_buffers of % is set high for a system with %. Recomended value between % and %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.2), pg_size_pretty(sysmem * 0.3);\n ELSE\n RAISE NOTICE 'shared_buffers of % is set appropriately for a system with %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem);\n END IF;\n shared_buffers = sysmem * 0.3;\n IF maintenance_work_mem < (sysmem * 0.2) THEN\n RAISE WARNING 'maintenance_work_mem of % is set low for shared_buffers of %. Recomended value between % and %', pg_size_pretty(maintenance_work_mem), pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers * 0.2), pg_size_pretty(shared_buffers * 0.3);\n ELSIF maintenance_work_mem > (shared_buffers * 0.3) THEN\n RAISE WARNING 'maintenance_work_mem of % is set high for shared_buffers of %. Recomended value between % and %', pg_size_pretty(maintenance_work_mem), pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers * 0.2), pg_size_pretty(shared_buffers * 0.3);\n ELSE\n RAISE NOTICE 'maintenance_work_mem of % is set appropriately for shared_buffers of %', pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers);\n END IF;\n\n IF work_mem * max_connections > shared_buffers THEN\n RAISE WARNING 'work_mem setting of % is set high for % max_connections please reduce work_mem to % or decrease max_connections to %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers/max_connections), floor(shared_buffers/work_mem);\n ELSIF work_mem * max_connections < (shared_buffers * 0.75) THEN\n RAISE WARNING 'work_mem setting of % is set low for % max_connections you may consider raising work_mem to % or increasing max_connections to %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers/max_connections), floor(shared_buffers/work_mem);\n ELSE\n RAISE NOTICE 'work_mem setting of % and max_connections of % are adequate for shared_buffers of %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers);\n END IF;\n\n IF random_page_cost / seq_page_cost != 1.1 THEN\n RAISE WARNING 'random_page_cost (%) /seq_page_cost (%) should be set to 1.1 for SSD. Change random_page_cost to %', random_page_cost, seq_page_cost, 1.1 * seq_page_cost;\n ELSE\n RAISE NOTICE 'random_page_cost and seq_page_cost set appropriately for SSD';\n END IF;\n\n IF temp_buffers < greatest(pg_size_bytes('128MB'),(maintenance_work_mem / 2)) THEN\n RAISE WARNING 'pgstac makes heavy use of temp tables, consider raising temp_buffers from % to %', pg_size_pretty(temp_buffers), greatest('128MB', pg_size_pretty((shared_buffers / 16)));\n END IF;\n END IF;\n\n RAISE NOTICE 'VALUES FOR PGSTAC VARIABLES';\n RAISE NOTICE 'These can be set either as GUC system variables or by setting in the pgstac_settings table.';\n\n FOR r IN SELECT name, get_setting(name) as setting, CASE WHEN current_setting(concat('pgstac.',name), TRUE) IS NOT NULL THEN concat('pgstac.',name, ' GUC') WHEN value IS NOT NULL THEN 'pgstac_settings table' ELSE 'Not Set' END as loc FROM pgstac_settings LOOP\n RAISE NOTICE '% is set to % from the %', r.name, r.setting, r.loc;\n END LOOP;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pg_cron';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider intalling pg_cron which can be used to automate tasks';\n ELSE\n RAISE NOTICE 'pg_cron % is installed', settingval;\n END IF;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pgstattuple';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider installing the pgstattuple extension which can be used to help maintain tables and indexes.';\n ELSE\n RAISE NOTICE 'pgstattuple % is installed', settingval;\n END IF;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pg_stat_statements';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider installing the pg_stat_statements extension which is very helpful for tracking the types of queries on the system';\n ELSE\n RAISE NOTICE 'pg_stat_statements % is installed', settingval;\n IF current_setting('pg_stat_statements.track_statements', TRUE) IS DISTINCT FROM 'all' THEN\n RAISE WARNING 'SET pg_stat_statements.track_statements TO ''all''; --In order to track statements within functions.';\n END IF;\n END IF;\n\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac, public SET CLIENT_MIN_MESSAGES TO NOTICE;\n/* looks for a geometry in a stac item first from geometry and falling back to bbox */\nCREATE OR REPLACE FUNCTION stac_geom(value jsonb) RETURNS geometry AS $$\nSELECT\n CASE\n WHEN value ? 'intersects' THEN\n ST_GeomFromGeoJSON(value->>'intersects')\n WHEN value ? 'geometry' THEN\n ST_GeomFromGeoJSON(value->>'geometry')\n WHEN value ? 'bbox' THEN\n pgstac.bbox_geom(value->'bbox')\n ELSE NULL\n END as geometry\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION stac_daterange(\n value jsonb\n) RETURNS tstzrange AS $$\nDECLARE\n props jsonb := value;\n dt timestamptz;\n edt timestamptz;\nBEGIN\n IF props ? 'properties' THEN\n props := props->'properties';\n END IF;\n IF\n props ? 'start_datetime'\n AND props->>'start_datetime' IS NOT NULL\n AND props ? 'end_datetime'\n AND props->>'end_datetime' IS NOT NULL\n THEN\n dt := props->>'start_datetime';\n edt := props->>'end_datetime';\n IF dt > edt THEN\n RAISE EXCEPTION 'start_datetime must be < end_datetime';\n END IF;\n ELSE\n dt := props->>'datetime';\n edt := props->>'datetime';\n END IF;\n IF dt is NULL OR edt IS NULL THEN\n RAISE NOTICE 'DT: %, EDT: %', dt, edt;\n RAISE EXCEPTION 'Either datetime (%) or both start_datetime (%) and end_datetime (%) must be set.', props->>'datetime',props->>'start_datetime',props->>'end_datetime';\n END IF;\n RETURN tstzrange(dt, edt, '[]');\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_datetime(value jsonb) RETURNS timestamptz AS $$\n SELECT lower(stac_daterange(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_end_datetime(value jsonb) RETURNS timestamptz AS $$\n SELECT upper(stac_daterange(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE TABLE IF NOT EXISTS stac_extensions(\n url text PRIMARY KEY,\n content jsonb\n);\nCREATE OR REPLACE FUNCTION queryable_signature(n text, c text[]) RETURNS text AS $$\n SELECT concat(n, c);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TABLE queryables (\n id bigint GENERATED ALWAYS AS identity PRIMARY KEY,\n name text NOT NULL,\n collection_ids text[], -- used to determine what partitions to create indexes on\n definition jsonb,\n property_path text,\n property_wrapper text,\n property_index_type text\n);\nCREATE INDEX queryables_name_idx ON queryables (name);\nCREATE INDEX queryables_collection_idx ON queryables USING GIN (collection_ids);\nCREATE INDEX queryables_property_wrapper_idx ON queryables (property_wrapper);\n\nCREATE OR REPLACE FUNCTION pgstac.queryables_constraint_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n allcollections text[];\nBEGIN\n RAISE NOTICE 'Making sure that name/collection is unique for queryables %', NEW;\n IF NEW.collection_ids IS NOT NULL THEN\n IF EXISTS (\n SELECT 1 FROM\n collections\n LEFT JOIN\n unnest(NEW.collection_ids) c\n ON (collections.id = c)\n WHERE c IS NULL\n ) THEN\n RAISE foreign_key_violation USING MESSAGE = format(\n 'One or more collections in %s do not exist.', NEW.collection_ids\n );\n RETURN NULL;\n END IF;\n END IF;\n IF TG_OP = 'INSERT' THEN\n IF EXISTS (\n SELECT 1 FROM queryables q\n WHERE\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n )\n ) THEN\n RAISE unique_violation USING MESSAGE = format(\n 'There is already a queryable for %s for a collection in %s: %s',\n NEW.name,\n NEW.collection_ids,\n\t\t\t\t(SELECT json_agg(row_to_json(q)) FROM queryables q WHERE\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n ))\n );\n RETURN NULL;\n END IF;\n END IF;\n IF TG_OP = 'UPDATE' THEN\n IF EXISTS (\n SELECT 1 FROM queryables q\n WHERE\n q.id != NEW.id\n AND\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n )\n ) THEN\n RAISE unique_violation\n USING MESSAGE = format(\n 'There is already a queryable for %s for a collection in %s',\n NEW.name,\n NEW.collection_ids\n );\n RETURN NULL;\n END IF;\n END IF;\n\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER queryables_constraint_insert_trigger\nBEFORE INSERT ON queryables\nFOR EACH ROW EXECUTE PROCEDURE queryables_constraint_triggerfunc();\n\nCREATE TRIGGER queryables_constraint_update_trigger\nBEFORE UPDATE ON queryables\nFOR EACH ROW\nWHEN (NEW.name = OLD.name AND NEW.collection_ids IS DISTINCT FROM OLD.collection_ids)\nEXECUTE PROCEDURE queryables_constraint_triggerfunc();\n\n\nCREATE OR REPLACE FUNCTION array_to_path(arr text[]) RETURNS text AS $$\n SELECT string_agg(\n quote_literal(v),\n '->'\n ) FROM unnest(arr) v;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\n\n\nCREATE OR REPLACE FUNCTION queryable(\n IN dotpath text,\n OUT path text,\n OUT expression text,\n OUT wrapper text,\n OUT nulled_wrapper text\n) AS $$\nDECLARE\n q RECORD;\n path_elements text[];\nBEGIN\n IF dotpath IN ('id', 'geometry', 'datetime', 'end_datetime', 'collection') THEN\n path := dotpath;\n expression := dotpath;\n wrapper := NULL;\n RETURN;\n END IF;\n SELECT * INTO q FROM queryables\n WHERE\n name=dotpath\n OR name = 'properties.' || dotpath\n OR name = replace(dotpath, 'properties.', '')\n ;\n IF q.property_wrapper IS NULL THEN\n IF q.definition->>'type' = 'number' THEN\n wrapper := 'to_float';\n nulled_wrapper := wrapper;\n ELSIF q.definition->>'format' = 'date-time' THEN\n wrapper := 'to_tstz';\n nulled_wrapper := wrapper;\n ELSE\n nulled_wrapper := NULL;\n wrapper := 'to_text';\n END IF;\n ELSE\n wrapper := q.property_wrapper;\n nulled_wrapper := wrapper;\n END IF;\n IF q.property_path IS NOT NULL THEN\n path := q.property_path;\n ELSE\n path_elements := string_to_array(dotpath, '.');\n IF path_elements[1] IN ('links', 'assets', 'stac_version', 'stac_extensions') THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSIF path_elements[1] = 'properties' THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSE\n path := format($F$content->'properties'->%s$F$, array_to_path(path_elements));\n END IF;\n END IF;\n expression := format('%I(%s)', wrapper, path);\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\nCREATE OR REPLACE FUNCTION unnest_collection(collection_ids text[] DEFAULT NULL) RETURNS SETOF text AS $$\n DECLARE\n BEGIN\n IF collection_ids IS NULL THEN\n RETURN QUERY SELECT id FROM collections;\n END IF;\n RETURN QUERY SELECT unnest(collection_ids);\n END;\n$$ LANGUAGE PLPGSQL STABLE;\n\nCREATE OR REPLACE FUNCTION normalize_indexdef(def text) RETURNS text AS $$\nDECLARE\nBEGIN\n def := btrim(def, ' \\n\\t');\n\tdef := regexp_replace(def, '^CREATE (UNIQUE )?INDEX ([^ ]* )?ON (ONLY )?([^ ]* )?', '', 'i');\n RETURN def;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE STRICT PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION indexdef(q queryables) RETURNS text AS $$\n DECLARE\n out text;\n BEGIN\n IF q.name = 'id' THEN\n out := 'CREATE UNIQUE INDEX ON %I USING btree (id)';\n ELSIF q.name = 'datetime' THEN\n out := 'CREATE INDEX ON %I USING btree (datetime DESC, end_datetime)';\n ELSIF q.name = 'geometry' THEN\n out := 'CREATE INDEX ON %I USING gist (geometry)';\n ELSE\n out := format($q$CREATE INDEX ON %%I USING %s (%s(((content -> 'properties'::text) -> %L::text)))$q$,\n lower(COALESCE(q.property_index_type, 'BTREE')),\n lower(COALESCE(q.property_wrapper, 'to_text')),\n q.name\n );\n END IF;\n RETURN btrim(out, ' \\n\\t');\n END;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\n\nDROP VIEW IF EXISTS pgstac_indexes;\nCREATE VIEW pgstac_indexes AS\nSELECT\n i.schemaname,\n i.tablename,\n i.indexname,\n regexp_replace(btrim(replace(replace(indexdef, i.indexname, ''),'pgstac.',''),' \\t\\n'), '[ ]+', ' ', 'g') as idx,\n COALESCE(\n (regexp_match(indexdef, '\\(([a-zA-Z]+)\\)'))[1],\n (regexp_match(indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_-]+)''::text'))[1],\n CASE WHEN indexdef ~* '\\(datetime desc, end_datetime\\)' THEN 'datetime' ELSE NULL END\n ) AS field,\n pg_table_size(i.indexname::text) as index_size,\n pg_size_pretty(pg_table_size(i.indexname::text)) as index_size_pretty\nFROM\n pg_indexes i\nWHERE i.schemaname='pgstac' and i.tablename ~ '_items_' AND indexdef !~* ' only ';\n\nDROP VIEW IF EXISTS pgstac_index_stats;\nCREATE VIEW pgstac_indexes_stats AS\nSELECT\n i.schemaname,\n i.tablename,\n i.indexname,\n indexdef,\n COALESCE(\n (regexp_match(indexdef, '\\(([a-zA-Z]+)\\)'))[1],\n (regexp_match(indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_]+)''::text'))[1],\n CASE WHEN indexdef ~* '\\(datetime desc, end_datetime\\)' THEN 'datetime_end_datetime' ELSE NULL END\n ) AS field,\n pg_table_size(i.indexname::text) as index_size,\n pg_size_pretty(pg_table_size(i.indexname::text)) as index_size_pretty,\n n_distinct,\n most_common_vals::text::text[],\n most_common_freqs::text::text[],\n histogram_bounds::text::text[],\n correlation\nFROM\n pg_indexes i\n LEFT JOIN pg_stats s ON (s.tablename = i.indexname)\nWHERE i.schemaname='pgstac' and i.tablename ~ '_items_';\n\nset check_function_bodies to off;\nCREATE OR REPLACE FUNCTION maintain_partition_queries(\n part text DEFAULT 'items',\n dropindexes boolean DEFAULT FALSE,\n rebuildindexes boolean DEFAULT FALSE,\n idxconcurrently boolean DEFAULT FALSE\n) RETURNS SETOF text AS $$\nDECLARE\n rec record;\nBEGIN\n FOR rec IN (\n WITH p AS (\n SELECT\n relid::text as partition,\n replace(replace(\n CASE\n WHEN level = 1 THEN pg_get_expr(c.relpartbound, c.oid)\n ELSE pg_get_expr(parent.relpartbound, parent.oid)\n END,\n 'FOR VALUES IN (''',''), ''')',\n ''\n ) AS collection\n FROM pg_partition_tree('items')\n JOIN pg_class c ON (relid::regclass = c.oid)\n JOIN pg_class parent ON (parentrelid::regclass = parent.oid AND isleaf)\n ), i AS (\n SELECT\n partition,\n indexname,\n regexp_replace(btrim(replace(replace(indexdef, indexname, ''),'pgstac.',''),' \\t\\n'), '[ ]+', ' ', 'g') as iidx,\n COALESCE(\n (regexp_match(indexdef, '\\(([a-zA-Z]+)\\)'))[1],\n (regexp_match(indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_-]+)''::text'))[1],\n CASE WHEN indexdef ~* '\\(datetime desc, end_datetime\\)' THEN 'datetime' ELSE NULL END\n ) AS field\n FROM\n pg_indexes\n JOIN p ON (tablename=partition)\n ), q AS (\n SELECT\n name AS field,\n collection,\n partition,\n format(indexdef(queryables), partition) as qidx\n FROM queryables, unnest_collection(queryables.collection_ids) collection\n JOIN p USING (collection)\n WHERE property_index_type IS NOT NULL OR name IN ('datetime','geometry','id')\n )\n SELECT * FROM i FULL JOIN q USING (field, partition)\n WHERE lower(iidx) IS DISTINCT FROM lower(qidx)\n ) LOOP\n IF rec.iidx IS NULL THEN\n IF idxconcurrently THEN\n RETURN NEXT replace(rec.qidx, 'INDEX', 'INDEX CONCURRENTLY');\n ELSE\n RETURN NEXT rec.qidx;\n END IF;\n ELSIF rec.qidx IS NULL AND dropindexes THEN\n RETURN NEXT format('DROP INDEX IF EXISTS %I;', rec.indexname);\n ELSIF lower(rec.qidx) != lower(rec.iidx) THEN\n IF dropindexes THEN\n RETURN NEXT format('DROP INDEX IF EXISTS %I; %s;', rec.indexname, rec.qidx);\n ELSE\n IF idxconcurrently THEN\n RETURN NEXT replace(rec.qidx, 'INDEX', 'INDEX CONCURRENTLY');\n ELSE\n RETURN NEXT rec.qidx;\n END IF;\n END IF;\n ELSIF rebuildindexes and rec.indexname IS NOT NULL THEN\n IF idxconcurrently THEN\n RETURN NEXT format('REINDEX INDEX CONCURRENTLY %I;', rec.indexname);\n ELSE\n RETURN NEXT format('REINDEX INDEX %I;', rec.indexname);\n END IF;\n END IF;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION maintain_partitions(\n part text DEFAULT 'items',\n dropindexes boolean DEFAULT FALSE,\n rebuildindexes boolean DEFAULT FALSE\n) RETURNS VOID AS $$\n WITH t AS (\n SELECT run_or_queue(q) FROM maintain_partition_queries(part, dropindexes, rebuildindexes) q\n ) SELECT count(*) FROM t;\n$$ LANGUAGE SQL;\n\n\nCREATE OR REPLACE FUNCTION queryables_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\nBEGIN\n PERFORM maintain_partitions();\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER queryables_trigger AFTER INSERT OR UPDATE ON queryables\nFOR EACH STATEMENT EXECUTE PROCEDURE queryables_trigger_func();\n\n\nCREATE OR REPLACE FUNCTION get_queryables(_collection_ids text[] DEFAULT NULL) RETURNS jsonb AS $$\nDECLARE\nBEGIN\n -- Build up queryables if the input contains valid collection ids or is empty\n IF EXISTS (\n SELECT 1 FROM collections\n WHERE\n _collection_ids IS NULL\n OR cardinality(_collection_ids) = 0\n OR id = ANY(_collection_ids)\n )\n THEN\n RETURN (\n WITH base AS (\n SELECT\n unnest(collection_ids) as collection_id,\n name,\n coalesce(definition, '{\"type\":\"string\"}'::jsonb) as definition\n FROM queryables\n WHERE\n _collection_ids IS NULL OR\n _collection_ids = '{}'::text[] OR\n _collection_ids && collection_ids\n UNION ALL\n SELECT null, name, coalesce(definition, '{\"type\":\"string\"}'::jsonb) as definition\n FROM queryables WHERE collection_ids IS NULL OR collection_ids = '{}'::text[]\n ), g AS (\n SELECT\n name,\n first_notnull(definition) as definition,\n jsonb_array_unique_merge(definition->'enum') as enum,\n jsonb_min(definition->'minimum') as minimum,\n jsonb_min(definition->'maxiumn') as maximum\n FROM base\n GROUP BY 1\n )\n SELECT\n jsonb_build_object(\n '$schema', 'http://json-schema.org/draft-07/schema#',\n '$id', '',\n 'type', 'object',\n 'title', 'STAC Queryables.',\n 'properties', jsonb_object_agg(\n name,\n definition\n ||\n jsonb_strip_nulls(jsonb_build_object(\n 'enum', enum,\n 'minimum', minimum,\n 'maximum', maximum\n ))\n )\n )\n FROM g\n );\n ELSE\n RETURN NULL;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\nCREATE OR REPLACE FUNCTION get_queryables(_collection text DEFAULT NULL) RETURNS jsonb AS $$\n SELECT\n CASE\n WHEN _collection IS NULL THEN get_queryables(NULL::text[])\n ELSE get_queryables(ARRAY[_collection])\n END\n ;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_queryables() RETURNS jsonb AS $$\n SELECT get_queryables(NULL::text[]);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION schema_qualify_refs(url text, j jsonb) returns jsonb as $$\n SELECT regexp_replace(j::text, '\"\\$ref\": \"#', concat('\"$ref\": \"', url, '#'), 'g')::jsonb;\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE VIEW stac_extension_queryables AS\nSELECT DISTINCT key as name, schema_qualify_refs(e.url, j.value) as definition FROM stac_extensions e, jsonb_each(e.content->'definitions'->'fields'->'properties') j;\n\n\nCREATE OR REPLACE FUNCTION missing_queryables(_collection text, _tablesample float DEFAULT 5, minrows float DEFAULT 10) RETURNS TABLE(collection text, name text, definition jsonb, property_wrapper text) AS $$\nDECLARE\n q text;\n _partition text;\n explain_json json;\n psize float;\n estrows float;\nBEGIN\n SELECT format('_items_%s', key) INTO _partition FROM collections WHERE id=_collection;\n\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM %I;', _partition)\n INTO explain_json;\n psize := explain_json->0->'Plan'->'Plan Rows';\n estrows := _tablesample * .01 * psize;\n IF estrows < minrows THEN\n _tablesample := least(100,greatest(_tablesample, (estrows / psize) / 100));\n RAISE NOTICE '%', (psize / estrows) / 100;\n END IF;\n RAISE NOTICE 'Using tablesample % to find missing queryables from % % that has ~% rows estrows: %', _tablesample, _collection, _partition, psize, estrows;\n\n q := format(\n $q$\n WITH q AS (\n SELECT * FROM queryables\n WHERE\n collection_ids IS NULL\n OR %L = ANY(collection_ids)\n ), t AS (\n SELECT\n content->'properties' AS properties\n FROM\n %I\n TABLESAMPLE SYSTEM(%L)\n ), p AS (\n SELECT DISTINCT ON (key)\n key,\n value,\n s.definition\n FROM t\n JOIN LATERAL jsonb_each(properties) ON TRUE\n LEFT JOIN q ON (q.name=key)\n LEFT JOIN stac_extension_queryables s ON (s.name=key)\n WHERE q.definition IS NULL\n )\n SELECT\n %L,\n key,\n COALESCE(definition, jsonb_build_object('type',jsonb_typeof(value))) as definition,\n CASE\n WHEN definition->>'type' = 'integer' THEN 'to_int'\n WHEN COALESCE(definition->>'type', jsonb_typeof(value)) = 'number' THEN 'to_float'\n WHEN COALESCE(definition->>'type', jsonb_typeof(value)) = 'array' THEN 'to_text_array'\n ELSE 'to_text'\n END\n FROM p;\n $q$,\n _collection,\n _partition,\n _tablesample,\n _collection\n );\n RETURN QUERY EXECUTE q;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION missing_queryables(_tablesample float DEFAULT 5) RETURNS TABLE(collection_ids text[], name text, definition jsonb, property_wrapper text) AS $$\n SELECT\n array_agg(collection),\n name,\n definition,\n property_wrapper\n FROM\n collections\n JOIN LATERAL\n missing_queryables(id, _tablesample) c\n ON TRUE\n GROUP BY\n 2,3,4\n ORDER BY 2,1\n ;\n$$ LANGUAGE SQL;\nCREATE OR REPLACE FUNCTION parse_dtrange(\n _indate jsonb,\n relative_base timestamptz DEFAULT date_trunc('hour', CURRENT_TIMESTAMP)\n) RETURNS tstzrange AS $$\nDECLARE\n timestrs text[];\n s timestamptz;\n e timestamptz;\nBEGIN\n timestrs :=\n CASE\n WHEN _indate ? 'timestamp' THEN\n ARRAY[_indate->>'timestamp']\n WHEN _indate ? 'interval' THEN\n to_text_array(_indate->'interval')\n WHEN jsonb_typeof(_indate) = 'array' THEN\n to_text_array(_indate)\n ELSE\n regexp_split_to_array(\n _indate->>0,\n '/'\n )\n END;\n RAISE NOTICE 'TIMESTRS %', timestrs;\n IF cardinality(timestrs) = 1 THEN\n IF timestrs[1] ILIKE 'P%' THEN\n RETURN tstzrange(relative_base - upper(timestrs[1])::interval, relative_base, '[)');\n END IF;\n s := timestrs[1]::timestamptz;\n RETURN tstzrange(s, s, '[]');\n END IF;\n\n IF cardinality(timestrs) != 2 THEN\n RAISE EXCEPTION 'Timestamp cannot have more than 2 values';\n END IF;\n\n IF timestrs[1] = '..' OR timestrs[1] = '' THEN\n s := '-infinity'::timestamptz;\n e := timestrs[2]::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] = '..' OR timestrs[2] = '' THEN\n s := timestrs[1]::timestamptz;\n e := 'infinity'::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[1] ILIKE 'P%' AND timestrs[2] NOT ILIKE 'P%' THEN\n e := timestrs[2]::timestamptz;\n s := e - upper(timestrs[1])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] ILIKE 'P%' AND timestrs[1] NOT ILIKE 'P%' THEN\n s := timestrs[1]::timestamptz;\n e := s + upper(timestrs[2])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n s := timestrs[1]::timestamptz;\n e := timestrs[2]::timestamptz;\n\n RETURN tstzrange(s,e,'[)');\n\n RETURN NULL;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT PARALLEL SAFE SET TIME ZONE 'UTC';\n\nCREATE OR REPLACE FUNCTION parse_dtrange(\n _indate text,\n relative_base timestamptz DEFAULT CURRENT_TIMESTAMP\n) RETURNS tstzrange AS $$\n SELECT parse_dtrange(to_jsonb(_indate), relative_base);\n$$ LANGUAGE SQL STABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION temporal_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\n ll text := 'datetime';\n lh text := 'end_datetime';\n rrange tstzrange;\n rl text;\n rh text;\n outq text;\nBEGIN\n rrange := parse_dtrange(args->1);\n RAISE NOTICE 'Constructing temporal query OP: %, ARGS: %, RRANGE: %', op, args, rrange;\n op := lower(op);\n rl := format('%L::timestamptz', lower(rrange));\n rh := format('%L::timestamptz', upper(rrange));\n outq := CASE op\n WHEN 't_before' THEN 'lh < rl'\n WHEN 't_after' THEN 'll > rh'\n WHEN 't_meets' THEN 'lh = rl'\n WHEN 't_metby' THEN 'll = rh'\n WHEN 't_overlaps' THEN 'll < rl AND rl < lh < rh'\n WHEN 't_overlappedby' THEN 'rl < ll < rh AND lh > rh'\n WHEN 't_starts' THEN 'll = rl AND lh < rh'\n WHEN 't_startedby' THEN 'll = rl AND lh > rh'\n WHEN 't_during' THEN 'll > rl AND lh < rh'\n WHEN 't_contains' THEN 'll < rl AND lh > rh'\n WHEN 't_finishes' THEN 'll > rl AND lh = rh'\n WHEN 't_finishedby' THEN 'll < rl AND lh = rh'\n WHEN 't_equals' THEN 'll = rl AND lh = rh'\n WHEN 't_disjoint' THEN 'NOT (ll <= rh AND lh >= rl)'\n WHEN 't_intersects' THEN 'll <= rh AND lh >= rl'\n WHEN 'anyinteracts' THEN 'll <= rh AND lh >= rl'\n END;\n outq := regexp_replace(outq, '\\mll\\M', ll);\n outq := regexp_replace(outq, '\\mlh\\M', lh);\n outq := regexp_replace(outq, '\\mrl\\M', rl);\n outq := regexp_replace(outq, '\\mrh\\M', rh);\n outq := format('(%s)', outq);\n RETURN outq;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\n\n\nCREATE OR REPLACE FUNCTION spatial_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\n geom text;\n j jsonb := args->1;\nBEGIN\n op := lower(op);\n RAISE NOTICE 'Constructing spatial query OP: %, ARGS: %', op, args;\n IF op NOT IN ('s_equals','s_disjoint','s_touches','s_within','s_overlaps','s_crosses','s_intersects','intersects','s_contains') THEN\n RAISE EXCEPTION 'Spatial Operator % Not Supported', op;\n END IF;\n op := regexp_replace(op, '^s_', 'st_');\n IF op = 'intersects' THEN\n op := 'st_intersects';\n END IF;\n -- Convert geometry to WKB string\n IF j ? 'type' AND j ? 'coordinates' THEN\n geom := st_geomfromgeojson(j)::text;\n ELSIF jsonb_typeof(j) = 'array' THEN\n geom := bbox_geom(j)::text;\n END IF;\n\n RETURN format('%s(geometry, %L::geometry)', op, geom);\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION query_to_cql2(q jsonb) RETURNS jsonb AS $$\n-- Translates anything passed in through the deprecated \"query\" into equivalent CQL2\nWITH t AS (\n SELECT key as property, value as ops\n FROM jsonb_each(q)\n), t2 AS (\n SELECT property, (jsonb_each(ops)).*\n FROM t WHERE jsonb_typeof(ops) = 'object'\n UNION ALL\n SELECT property, 'eq', ops\n FROM t WHERE jsonb_typeof(ops) != 'object'\n)\nSELECT\n jsonb_strip_nulls(jsonb_build_object(\n 'op', 'and',\n 'args', jsonb_agg(\n jsonb_build_object(\n 'op', key,\n 'args', jsonb_build_array(\n jsonb_build_object('property',property),\n value\n )\n )\n )\n )\n) as qcql FROM t2\n;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\nCREATE OR REPLACE FUNCTION cql1_to_cql2(j jsonb) RETURNS jsonb AS $$\nDECLARE\n args jsonb;\n ret jsonb;\nBEGIN\n RAISE NOTICE 'CQL1_TO_CQL2: %', j;\n IF j ? 'filter' THEN\n RETURN cql1_to_cql2(j->'filter');\n END IF;\n IF j ? 'property' THEN\n RETURN j;\n END IF;\n IF jsonb_typeof(j) = 'array' THEN\n SELECT jsonb_agg(cql1_to_cql2(el)) INTO args FROM jsonb_array_elements(j) el;\n RETURN args;\n END IF;\n IF jsonb_typeof(j) = 'number' THEN\n RETURN j;\n END IF;\n IF jsonb_typeof(j) = 'string' THEN\n RETURN j;\n END IF;\n\n IF jsonb_typeof(j) = 'object' THEN\n SELECT jsonb_build_object(\n 'op', key,\n 'args', cql1_to_cql2(value)\n ) INTO ret\n FROM jsonb_each(j)\n WHERE j IS NOT NULL;\n RETURN ret;\n END IF;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE STRICT;\n\nCREATE TABLE cql2_ops (\n op text PRIMARY KEY,\n template text,\n types text[]\n);\nINSERT INTO cql2_ops (op, template, types) VALUES\n ('eq', '%s = %s', NULL),\n ('neq', '%s != %s', NULL),\n ('ne', '%s != %s', NULL),\n ('!=', '%s != %s', NULL),\n ('<>', '%s != %s', NULL),\n ('lt', '%s < %s', NULL),\n ('lte', '%s <= %s', NULL),\n ('gt', '%s > %s', NULL),\n ('gte', '%s >= %s', NULL),\n ('le', '%s <= %s', NULL),\n ('ge', '%s >= %s', NULL),\n ('=', '%s = %s', NULL),\n ('<', '%s < %s', NULL),\n ('<=', '%s <= %s', NULL),\n ('>', '%s > %s', NULL),\n ('>=', '%s >= %s', NULL),\n ('like', '%s LIKE %s', NULL),\n ('ilike', '%s ILIKE %s', NULL),\n ('+', '%s + %s', NULL),\n ('-', '%s - %s', NULL),\n ('*', '%s * %s', NULL),\n ('/', '%s / %s', NULL),\n ('not', 'NOT (%s)', NULL),\n ('between', '%s BETWEEN %s AND %s', NULL),\n ('isnull', '%s IS NULL', NULL),\n ('upper', 'upper(%s)', NULL),\n ('lower', 'lower(%s)', NULL)\nON CONFLICT (op) DO UPDATE\n SET\n template = EXCLUDED.template\n;\n\n\nCREATE OR REPLACE FUNCTION cql2_query(j jsonb, wrapper text DEFAULT NULL) RETURNS text AS $$\n#variable_conflict use_variable\nDECLARE\n args jsonb := j->'args';\n arg jsonb;\n op text := lower(j->>'op');\n cql2op RECORD;\n literal text;\n _wrapper text;\n leftarg text;\n rightarg text;\nBEGIN\n IF j IS NULL OR (op IS NOT NULL AND args IS NULL) THEN\n RETURN NULL;\n END IF;\n RAISE NOTICE 'CQL2_QUERY: %', j;\n IF j ? 'filter' THEN\n RETURN cql2_query(j->'filter');\n END IF;\n\n IF j ? 'upper' THEN\n RETURN cql2_query(jsonb_build_object('op', 'upper', 'args', j->'upper'));\n END IF;\n\n IF j ? 'lower' THEN\n RETURN cql2_query(jsonb_build_object('op', 'lower', 'args', j->'lower'));\n END IF;\n\n -- Temporal Query\n IF op ilike 't_%' or op = 'anyinteracts' THEN\n RETURN temporal_op_query(op, args);\n END IF;\n\n -- If property is a timestamp convert it to text to use with\n -- general operators\n IF j ? 'timestamp' THEN\n RETURN format('%L::timestamptz', to_tstz(j->'timestamp'));\n END IF;\n IF j ? 'interval' THEN\n RAISE EXCEPTION 'Please use temporal operators when using intervals.';\n RETURN NONE;\n END IF;\n\n -- Spatial Query\n IF op ilike 's_%' or op = 'intersects' THEN\n RETURN spatial_op_query(op, args);\n END IF;\n\n IF op IN ('a_equals','a_contains','a_contained_by','a_overlaps') THEN\n IF args->0 ? 'property' THEN\n leftarg := format('to_text_array(%s)', (queryable(args->0->>'property')).path);\n END IF;\n IF args->1 ? 'property' THEN\n rightarg := format('to_text_array(%s)', (queryable(args->1->>'property')).path);\n END IF;\n RETURN FORMAT(\n '%s %s %s',\n COALESCE(leftarg, quote_literal(to_text_array(args->0))),\n CASE op\n WHEN 'a_equals' THEN '='\n WHEN 'a_contains' THEN '@>'\n WHEN 'a_contained_by' THEN '<@'\n WHEN 'a_overlaps' THEN '&&'\n END,\n COALESCE(rightarg, quote_literal(to_text_array(args->1)))\n );\n END IF;\n\n IF op = 'in' THEN\n RAISE NOTICE 'IN : % % %', args, jsonb_build_array(args->0), args->1;\n args := jsonb_build_array(args->0) || (args->1);\n RAISE NOTICE 'IN2 : %', args;\n END IF;\n\n\n\n IF op = 'between' THEN\n args = jsonb_build_array(\n args->0,\n args->1->0,\n args->1->1\n );\n END IF;\n\n -- Make sure that args is an array and run cql2_query on\n -- each element of the array\n RAISE NOTICE 'ARGS PRE: %', args;\n IF j ? 'args' THEN\n IF jsonb_typeof(args) != 'array' THEN\n args := jsonb_build_array(args);\n END IF;\n\n IF jsonb_path_exists(args, '$[*] ? (@.property == \"id\" || @.property == \"datetime\" || @.property == \"end_datetime\" || @.property == \"collection\")') THEN\n wrapper := NULL;\n ELSE\n -- if any of the arguments are a property, try to get the property_wrapper\n FOR arg IN SELECT jsonb_path_query(args, '$[*] ? (@.property != null)') LOOP\n RAISE NOTICE 'Arg: %', arg;\n wrapper := (queryable(arg->>'property')).nulled_wrapper;\n RAISE NOTICE 'Property: %, Wrapper: %', arg, wrapper;\n IF wrapper IS NOT NULL THEN\n EXIT;\n END IF;\n END LOOP;\n\n -- if the property was not in queryables, see if any args were numbers\n IF\n wrapper IS NULL\n AND jsonb_path_exists(args, '$[*] ? (@.type()==\"number\")')\n THEN\n wrapper := 'to_float';\n END IF;\n wrapper := coalesce(wrapper, 'to_text');\n END IF;\n\n SELECT jsonb_agg(cql2_query(a, wrapper))\n INTO args\n FROM jsonb_array_elements(args) a;\n END IF;\n RAISE NOTICE 'ARGS: %', args;\n\n IF op IN ('and', 'or') THEN\n RETURN\n format(\n '(%s)',\n array_to_string(to_text_array(args), format(' %s ', upper(op)))\n );\n END IF;\n\n IF op = 'in' THEN\n RAISE NOTICE 'IN -- % %', args->0, to_text(args->0);\n RETURN format(\n '%s IN (%s)',\n to_text(args->0),\n array_to_string((to_text_array(args))[2:], ',')\n );\n END IF;\n\n -- Look up template from cql2_ops\n IF j ? 'op' THEN\n SELECT * INTO cql2op FROM cql2_ops WHERE cql2_ops.op ilike op;\n IF FOUND THEN\n -- If specific index set in queryables for a property cast other arguments to that type\n\n RETURN format(\n cql2op.template,\n VARIADIC (to_text_array(args))\n );\n ELSE\n RAISE EXCEPTION 'Operator % Not Supported.', op;\n END IF;\n END IF;\n\n\n IF wrapper IS NOT NULL THEN\n RAISE NOTICE 'Wrapping % with %', j, wrapper;\n IF j ? 'property' THEN\n RETURN format('%I(%s)', wrapper, (queryable(j->>'property')).path);\n ELSE\n RETURN format('%I(%L)', wrapper, j);\n END IF;\n ELSIF j ? 'property' THEN\n RETURN quote_ident(j->>'property');\n END IF;\n\n RETURN quote_literal(to_text(j));\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION paging_dtrange(\n j jsonb\n) RETURNS tstzrange AS $$\nDECLARE\n op text;\n filter jsonb := j->'filter';\n dtrange tstzrange := tstzrange('-infinity'::timestamptz,'infinity'::timestamptz);\n sdate timestamptz := '-infinity'::timestamptz;\n edate timestamptz := 'infinity'::timestamptz;\n jpitem jsonb;\nBEGIN\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n END IF;\n IF NOT (filter @? '$.**.op ? (@ == \"or\" || @ == \"not\")') THEN\n FOR jpitem IN SELECT j FROM jsonb_path_query(filter,'strict $.** ? (@.args[*].property == \"datetime\")'::jsonpath) j LOOP\n op := lower(jpitem->>'op');\n dtrange := parse_dtrange(jpitem->'args'->1);\n IF op IN ('<=', 'lt', 'lte', '<', 'le', 't_before') THEN\n sdate := greatest(sdate,'-infinity');\n edate := least(edate, upper(dtrange));\n ELSIF op IN ('>=', '>', 'gt', 'gte', 'ge', 't_after') THEN\n edate := least(edate, 'infinity');\n sdate := greatest(sdate, lower(dtrange));\n ELSIF op IN ('=', 'eq') THEN\n edate := least(edate, upper(dtrange));\n sdate := greatest(sdate, lower(dtrange));\n END IF;\n RAISE NOTICE '2 OP: %, ARGS: %, DTRANGE: %, SDATE: %, EDATE: %', op, jpitem->'args'->1, dtrange, sdate, edate;\n END LOOP;\n END IF;\n IF sdate > edate THEN\n RETURN 'empty'::tstzrange;\n END IF;\n RETURN tstzrange(sdate,edate, '[]');\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT SET TIME ZONE 'UTC';\n\nCREATE OR REPLACE FUNCTION paging_collections(\n IN j jsonb\n) RETURNS text[] AS $$\nDECLARE\n filter jsonb := j->'filter';\n jpitem jsonb;\n op text;\n args jsonb;\n arg jsonb;\n collections text[];\nBEGIN\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n END IF;\n IF NOT (filter @? '$.**.op ? (@ == \"or\" || @ == \"not\")') THEN\n FOR jpitem IN SELECT j FROM jsonb_path_query(filter,'strict $.** ? (@.args[*].property == \"collection\")'::jsonpath) j LOOP\n RAISE NOTICE 'JPITEM: %', jpitem;\n op := jpitem->>'op';\n args := jpitem->'args';\n IF op IN ('=', 'eq', 'in') THEN\n FOR arg IN SELECT a FROM jsonb_array_elements(args) a LOOP\n IF jsonb_typeof(arg) IN ('string', 'array') THEN\n RAISE NOTICE 'arg: %, collections: %', arg, collections;\n IF collections IS NULL OR collections = '{}'::text[] THEN\n collections := to_text_array(arg);\n ELSE\n collections := array_intersection(collections, to_text_array(arg));\n END IF;\n END IF;\n END LOOP;\n END IF;\n END LOOP;\n END IF;\n IF collections = '{}'::text[] THEN\n RETURN NULL;\n END IF;\n RETURN collections;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\nCREATE OR REPLACE FUNCTION collection_base_item(content jsonb) RETURNS jsonb AS $$\n SELECT jsonb_build_object(\n 'type', 'Feature',\n 'stac_version', content->'stac_version',\n 'assets', content->'item_assets',\n 'collection', content->'id'\n );\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TABLE IF NOT EXISTS collections (\n key bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,\n id text GENERATED ALWAYS AS (content->>'id') STORED UNIQUE,\n content JSONB NOT NULL,\n base_item jsonb GENERATED ALWAYS AS (pgstac.collection_base_item(content)) STORED,\n partition_trunc text CHECK (partition_trunc IN ('year', 'month'))\n);\n\n\n\nCREATE OR REPLACE FUNCTION collection_base_item(cid text) RETURNS jsonb AS $$\n SELECT pgstac.collection_base_item(content) FROM pgstac.collections WHERE id = cid LIMIT 1;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION create_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION update_collection(data jsonb) RETURNS VOID AS $$\nDECLARE\n out collections%ROWTYPE;\nBEGIN\n UPDATE collections SET content=data WHERE id = data->>'id' RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ON CONFLICT (id) DO\n UPDATE\n SET content=EXCLUDED.content\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION delete_collection(_id text) RETURNS VOID AS $$\nDECLARE\n out collections%ROWTYPE;\nBEGIN\n DELETE FROM collections WHERE id = _id RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION get_collection(id text) RETURNS jsonb AS $$\n SELECT content FROM collections\n WHERE id=$1\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION all_collections() RETURNS jsonb AS $$\n SELECT jsonb_agg(content) FROM collections;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\nCREATE TABLE items (\n id text NOT NULL,\n geometry geometry NOT NULL,\n collection text NOT NULL,\n datetime timestamptz NOT NULL,\n end_datetime timestamptz NOT NULL,\n content JSONB NOT NULL\n)\nPARTITION BY LIST (collection)\n;\n\nCREATE INDEX \"datetime_idx\" ON items USING BTREE (datetime DESC, end_datetime ASC);\nCREATE INDEX \"geometry_idx\" ON items USING GIST (geometry);\n\nCREATE STATISTICS datetime_stats (dependencies) on datetime, end_datetime from items;\n\nALTER TABLE items ADD CONSTRAINT items_collections_fk FOREIGN KEY (collection) REFERENCES collections(id) ON DELETE CASCADE DEFERRABLE;\n\nCREATE OR REPLACE FUNCTION partition_after_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p text;\n t timestamptz := clock_timestamp();\nBEGIN\n RAISE NOTICE 'Updating partition stats %', t;\n FOR p IN SELECT DISTINCT partition\n FROM newdata n JOIN partition_sys_meta p\n ON (n.collection=p.collection AND n.datetime <@ p.partition_dtrange)\n LOOP\n PERFORM run_or_queue(format('SELECT update_partition_stats(%L, %L);', p, true));\n END LOOP;\n IF TG_OP IN ('DELETE','UPDATE') THEN\n DELETE FROM format_item_cache c USING newdata n WHERE c.collection = n.collection AND c.id = n.id;\n END IF;\n RAISE NOTICE 't: % %', t, clock_timestamp() - t;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE TRIGGER items_after_insert_trigger\nAFTER INSERT ON items\nREFERENCING NEW TABLE AS newdata\nFOR EACH STATEMENT\nEXECUTE FUNCTION partition_after_triggerfunc();\n\nCREATE TRIGGER items_after_update_trigger\nAFTER DELETE ON items\nREFERENCING OLD TABLE AS newdata\nFOR EACH STATEMENT\nEXECUTE FUNCTION partition_after_triggerfunc();\n\nCREATE TRIGGER items_after_delete_trigger\nAFTER UPDATE ON items\nREFERENCING NEW TABLE AS newdata\nFOR EACH STATEMENT\nEXECUTE FUNCTION partition_after_triggerfunc();\n\n\nCREATE OR REPLACE FUNCTION content_slim(_item jsonb) RETURNS jsonb AS $$\n SELECT strip_jsonb(_item - '{id,geometry,collection,type}'::text[], collection_base_item(_item->>'collection')) - '{id,geometry,collection,type}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_dehydrate(content jsonb) RETURNS items AS $$\n SELECT\n content->>'id' as id,\n stac_geom(content) as geometry,\n content->>'collection' as collection,\n stac_datetime(content) as datetime,\n stac_end_datetime(content) as end_datetime,\n content_slim(content) as content\n ;\n$$ LANGUAGE SQL STABLE;\n\nCREATE OR REPLACE FUNCTION include_field(f text, fields jsonb DEFAULT '{}'::jsonb) RETURNS boolean AS $$\nDECLARE\n includes jsonb := fields->'include';\n excludes jsonb := fields->'exclude';\nBEGIN\n IF f IS NULL THEN\n RETURN NULL;\n END IF;\n\n\n IF\n jsonb_typeof(excludes) = 'array'\n AND jsonb_array_length(excludes)>0\n AND excludes ? f\n THEN\n RETURN FALSE;\n END IF;\n\n IF\n (\n jsonb_typeof(includes) = 'array'\n AND jsonb_array_length(includes) > 0\n AND includes ? f\n ) OR\n (\n includes IS NULL\n OR jsonb_typeof(includes) = 'null'\n OR jsonb_array_length(includes) = 0\n )\n THEN\n RETURN TRUE;\n END IF;\n\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nDROP FUNCTION IF EXISTS content_hydrate(jsonb, jsonb, jsonb);\nCREATE OR REPLACE FUNCTION content_hydrate(\n _item jsonb,\n _base_item jsonb,\n fields jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\n SELECT merge_jsonb(\n jsonb_fields(_item, fields),\n jsonb_fields(_base_item, fields)\n );\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION content_hydrate(_item items, _collection collections, fields jsonb DEFAULT '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\n content jsonb;\n base_item jsonb := _collection.base_item;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry, 20)::jsonb;\n END IF;\n output := content_hydrate(\n jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'collection', _item.collection,\n 'type', 'Feature'\n ) || _item.content,\n _collection.base_item,\n fields\n );\n\n RETURN output;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_nonhydrated(\n _item items,\n fields jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry, 20)::jsonb;\n END IF;\n output := jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'collection', _item.collection,\n 'type', 'Feature'\n ) || _item.content;\n RETURN output;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_hydrate(_item items, fields jsonb DEFAULT '{}'::jsonb) RETURNS jsonb AS $$\n SELECT content_hydrate(\n _item,\n (SELECT c FROM collections c WHERE id=_item.collection LIMIT 1),\n fields\n );\n$$ LANGUAGE SQL STABLE;\n\n\nCREATE UNLOGGED TABLE items_staging (\n content JSONB NOT NULL\n);\nCREATE UNLOGGED TABLE items_staging_ignore (\n content JSONB NOT NULL\n);\nCREATE UNLOGGED TABLE items_staging_upsert (\n content JSONB NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION items_staging_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p record;\n _partitions text[];\n part text;\n ts timestamptz := clock_timestamp();\nBEGIN\n RAISE NOTICE 'Creating Partitions. %', clock_timestamp() - ts;\n\n FOR part IN WITH t AS (\n SELECT\n n.content->>'collection' as collection,\n stac_daterange(n.content->'properties') as dtr,\n partition_trunc\n FROM newdata n JOIN collections ON (n.content->>'collection'=collections.id)\n ), p AS (\n SELECT\n collection,\n COALESCE(date_trunc(partition_trunc::text, lower(dtr)),'-infinity') as d,\n tstzrange(min(lower(dtr)),max(lower(dtr)),'[]') as dtrange,\n tstzrange(min(upper(dtr)),max(upper(dtr)),'[]') as edtrange\n FROM t\n GROUP BY 1,2\n ) SELECT check_partition(collection, dtrange, edtrange) FROM p LOOP\n RAISE NOTICE 'Partition %', part;\n END LOOP;\n\n RAISE NOTICE 'Doing the insert. %', clock_timestamp() - ts;\n IF TG_TABLE_NAME = 'items_staging' THEN\n INSERT INTO items\n SELECT\n (content_dehydrate(content)).*\n FROM newdata;\n RAISE NOTICE 'Doing the delete. %', clock_timestamp() - ts;\n DELETE FROM items_staging;\n ELSIF TG_TABLE_NAME = 'items_staging_ignore' THEN\n INSERT INTO items\n SELECT\n (content_dehydrate(content)).*\n FROM newdata\n ON CONFLICT DO NOTHING;\n RAISE NOTICE 'Doing the delete. %', clock_timestamp() - ts;\n DELETE FROM items_staging_ignore;\n ELSIF TG_TABLE_NAME = 'items_staging_upsert' THEN\n WITH staging_formatted AS (\n SELECT (content_dehydrate(content)).* FROM newdata\n ), deletes AS (\n DELETE FROM items i USING staging_formatted s\n WHERE\n i.id = s.id\n AND i.collection = s.collection\n AND i IS DISTINCT FROM s\n RETURNING i.id, i.collection\n )\n INSERT INTO items\n SELECT s.* FROM\n staging_formatted s\n ON CONFLICT DO NOTHING;\n RAISE NOTICE 'Doing the delete. %', clock_timestamp() - ts;\n DELETE FROM items_staging_upsert;\n END IF;\n RAISE NOTICE 'Done. %', clock_timestamp() - ts;\n\n RETURN NULL;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER items_staging_insert_trigger AFTER INSERT ON items_staging REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\nCREATE TRIGGER items_staging_insert_ignore_trigger AFTER INSERT ON items_staging_ignore REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\nCREATE TRIGGER items_staging_insert_upsert_trigger AFTER INSERT ON items_staging_upsert REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\n\nCREATE OR REPLACE FUNCTION item_by_id(_id text, _collection text DEFAULT NULL) RETURNS items AS\n$$\nDECLARE\n i items%ROWTYPE;\nBEGIN\n SELECT * INTO i FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection) LIMIT 1;\n RETURN i;\nEND;\n$$ LANGUAGE PLPGSQL STABLE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION get_item(_id text, _collection text DEFAULT NULL) RETURNS jsonb AS $$\n SELECT content_hydrate(items) FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection);\n$$ LANGUAGE SQL STABLE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION delete_item(_id text, _collection text DEFAULT NULL) RETURNS VOID AS $$\nDECLARE\nout items%ROWTYPE;\nBEGIN\n DELETE FROM items WHERE id = _id AND (_collection IS NULL OR collection=_collection) RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n--/*\nCREATE OR REPLACE FUNCTION create_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION update_item(content jsonb) RETURNS VOID AS $$\nDECLARE\n old items %ROWTYPE;\n out items%ROWTYPE;\nBEGIN\n PERFORM delete_item(content->>'id', content->>'collection');\n PERFORM create_item(content);\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION create_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION collection_bbox(id text) RETURNS jsonb AS $$\n SELECT (replace(replace(replace(st_extent(geometry)::text,'BOX(','[['),')',']]'),' ',','))::jsonb\n FROM items WHERE collection=$1;\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION collection_temporal_extent(id text) RETURNS jsonb AS $$\n SELECT to_jsonb(array[array[min(datetime)::text, max(datetime)::text]])\n FROM items WHERE collection=$1;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection_extents() RETURNS VOID AS $$\nUPDATE collections SET\n content = content ||\n jsonb_build_object(\n 'extent', jsonb_build_object(\n 'spatial', jsonb_build_object(\n 'bbox', collection_bbox(collections.id)\n ),\n 'temporal', jsonb_build_object(\n 'interval', collection_temporal_extent(collections.id)\n )\n )\n )\n;\n$$ LANGUAGE SQL;\nCREATE TABLE partition_stats (\n partition text PRIMARY KEY,\n dtrange tstzrange,\n edtrange tstzrange,\n spatial geometry,\n last_updated timestamptz,\n keys text[]\n) WITH (FILLFACTOR=90);\n\nCREATE INDEX partitions_range_idx ON partition_stats USING GIST(dtrange);\n\n\nCREATE OR REPLACE FUNCTION constraint_tstzrange(expr text) RETURNS tstzrange AS $$\n WITH t AS (\n SELECT regexp_matches(\n expr,\n E'\\\\(''\\([0-9 :+-]*\\)''\\\\).*\\\\(''\\([0-9 :+-]*\\)''\\\\)'\n ) AS m\n ) SELECT tstzrange(m[1]::timestamptz, m[2]::timestamptz) FROM t\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE STRICT;\n\nCREATE OR REPLACE FUNCTION dt_constraint(coid oid, OUT dt tstzrange, OUT edt tstzrange) RETURNS RECORD AS $$\nDECLARE\n expr text := pg_get_constraintdef(coid);\n matches timestamptz[];\nBEGIN\n IF expr LIKE '%NULL%' THEN\n dt := tstzrange(null::timestamptz, null::timestamptz);\n edt := tstzrange(null::timestamptz, null::timestamptz);\n RETURN;\n END IF;\n WITH f AS (SELECT (regexp_matches(expr, E'([0-9]{4}-[0-1][0-9]-[0-3][0-9] [0-2][0-9]:[0-5][0-9]:[0-5][0-9]\\.?[0-9]*)', 'g'))[1] f)\n SELECT array_agg(f::timestamptz) INTO matches FROM f;\n IF cardinality(matches) = 4 THEN\n dt := tstzrange(matches[1], matches[2],'[]');\n edt := tstzrange(matches[3], matches[4], '[]');\n RETURN;\n ELSIF cardinality(matches) = 2 THEN\n edt := tstzrange(matches[1], matches[2],'[]');\n RETURN;\n END IF;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\nCREATE OR REPLACE VIEW partition_sys_meta AS\nSELECT\n relid::text as partition,\n replace(replace(CASE WHEN level = 1 THEN pg_get_expr(c.relpartbound, c.oid)\n ELSE pg_get_expr(parent.relpartbound, parent.oid)\n END, 'FOR VALUES IN (''',''), ''')','') AS collection,\n level,\n c.reltuples,\n c.relhastriggers,\n COALESCE(constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as partition_dtrange,\n COALESCE((dt_constraint(edt.oid)).dt, constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as constraint_dtrange,\n COALESCE((dt_constraint(edt.oid)).edt, tstzrange('-infinity', 'infinity','[]')) as constraint_edtrange\nFROM\n pg_partition_tree('items')\n JOIN pg_class c ON (relid::regclass = c.oid)\n JOIN pg_class parent ON (parentrelid::regclass = parent.oid AND isleaf)\n LEFT JOIN pg_constraint edt ON (conrelid=c.oid AND contype='c')\nWHERE isleaf\n;\n\nCREATE VIEW partitions_view AS\nSELECT\n relid::text as partition,\n replace(replace(CASE WHEN level = 1 THEN pg_get_expr(c.relpartbound, c.oid)\n ELSE pg_get_expr(parent.relpartbound, parent.oid)\n END, 'FOR VALUES IN (''',''), ''')','') AS collection,\n level,\n c.reltuples,\n c.relhastriggers,\n COALESCE(constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as partition_dtrange,\n COALESCE((dt_constraint(edt.oid)).dt, constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as constraint_dtrange,\n COALESCE((dt_constraint(edt.oid)).edt, tstzrange('-infinity', 'infinity','[]')) as constraint_edtrange,\n dtrange,\n edtrange,\n spatial,\n last_updated\nFROM\n pg_partition_tree('items')\n JOIN pg_class c ON (relid::regclass = c.oid)\n JOIN pg_class parent ON (parentrelid::regclass = parent.oid AND isleaf)\n LEFT JOIN pg_constraint edt ON (conrelid=c.oid AND contype='c')\n LEFT JOIN partition_stats ON (relid::text=partition)\nWHERE isleaf\n;\n\nCREATE MATERIALIZED VIEW partitions AS\nSELECT * FROM partitions_view;\nCREATE UNIQUE INDEX ON partitions (partition);\n\nCREATE MATERIALIZED VIEW partition_steps AS\nSELECT\n partition as name,\n date_trunc('month',lower(partition_dtrange)) as sdate,\n date_trunc('month', upper(partition_dtrange)) + '1 month'::interval as edate\n FROM partitions_view WHERE partition_dtrange IS NOT NULL AND partition_dtrange != 'empty'::tstzrange\n ORDER BY dtrange ASC\n;\n\n\nCREATE OR REPLACE FUNCTION update_partition_stats_q(_partition text, istrigger boolean default false) RETURNS VOID AS $$\nDECLARE\nBEGIN\n PERFORM run_or_queue(\n format('SELECT update_partition_stats(%L, %L);', _partition, istrigger)\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION update_partition_stats(_partition text, istrigger boolean default false) RETURNS VOID AS $$\nDECLARE\n dtrange tstzrange;\n edtrange tstzrange;\n cdtrange tstzrange;\n cedtrange tstzrange;\n extent geometry;\n collection text;\nBEGIN\n RAISE NOTICE 'Updating stats for %.', _partition;\n EXECUTE format(\n $q$\n SELECT\n tstzrange(min(datetime), max(datetime),'[]'),\n tstzrange(min(end_datetime), max(end_datetime), '[]')\n FROM %I\n $q$,\n _partition\n ) INTO dtrange, edtrange;\n extent := st_estimatedextent('pgstac', _partition, 'geometry');\n INSERT INTO partition_stats (partition, dtrange, edtrange, spatial, last_updated)\n SELECT _partition, dtrange, edtrange, extent, now()\n ON CONFLICT (partition) DO\n UPDATE SET\n dtrange=EXCLUDED.dtrange,\n edtrange=EXCLUDED.edtrange,\n spatial=EXCLUDED.spatial,\n last_updated=EXCLUDED.last_updated\n ;\n\n SELECT\n constraint_dtrange, constraint_edtrange, pv.collection\n INTO cdtrange, cedtrange, collection\n FROM partitions_view pv WHERE partition = _partition;\n REFRESH MATERIALIZED VIEW partitions;\n REFRESH MATERIALIZED VIEW partition_steps;\n\n\n RAISE NOTICE 'Checking if we need to modify constraints.';\n IF\n (cdtrange IS DISTINCT FROM dtrange OR edtrange IS DISTINCT FROM cedtrange)\n AND NOT istrigger\n THEN\n RAISE NOTICE 'Modifying Constraints';\n RAISE NOTICE 'Existing % %', cdtrange, cedtrange;\n RAISE NOTICE 'New % %', dtrange, edtrange;\n PERFORM drop_table_constraints(_partition);\n PERFORM create_table_constraints(_partition, dtrange, edtrange);\n END IF;\n RAISE NOTICE 'Checking if we need to update collection extents.';\n IF get_setting_bool('update_collection_extent') THEN\n RAISE NOTICE 'updating collection extent for %', collection;\n PERFORM run_or_queue(format($q$\n UPDATE collections\n SET content = jsonb_set_lax(\n content,\n '{extent}'::text[],\n collection_extent(%L, FALSE),\n true,\n 'use_json_null'\n ) WHERE id=%L\n ;\n $q$, collection, collection));\n ELSE\n RAISE NOTICE 'Not updating collection extent for %', collection;\n END IF;\n\nEND;\n$$ LANGUAGE PLPGSQL STRICT;\n\n\nCREATE OR REPLACE FUNCTION partition_name( IN collection text, IN dt timestamptz, OUT partition_name text, OUT partition_range tstzrange) AS $$\nDECLARE\n c RECORD;\n parent_name text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n parent_name := format('_items_%s', c.key);\n\n\n IF c.partition_trunc = 'year' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYY'));\n ELSIF c.partition_trunc = 'month' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYYMM'));\n ELSE\n partition_name := parent_name;\n partition_range := tstzrange('-infinity'::timestamptz, 'infinity'::timestamptz, '[]');\n END IF;\n IF partition_range IS NULL THEN\n partition_range := tstzrange(\n date_trunc(c.partition_trunc::text, dt),\n date_trunc(c.partition_trunc::text, dt) + concat('1 ', c.partition_trunc)::interval\n );\n END IF;\n RETURN;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION drop_table_constraints(t text) RETURNS text AS $$\nDECLARE\n q text;\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM partitions_view WHERE partition=t) THEN\n RETURN NULL;\n END IF;\n FOR q IN SELECT FORMAT(\n $q$\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n $q$,\n t,\n conname\n ) FROM pg_constraint\n WHERE conrelid=t::regclass::oid AND contype='c'\n LOOP\n EXECUTE q;\n END LOOP;\n RETURN t;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE OR REPLACE FUNCTION create_table_constraints(t text, _dtrange tstzrange, _edtrange tstzrange) RETURNS text AS $$\nDECLARE\n q text;\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM partitions_view WHERE partition=t) THEN\n RETURN NULL;\n END IF;\n RAISE NOTICE 'Creating Table Constraints for % % %', t, _dtrange, _edtrange;\n IF _dtrange = 'empty' AND _edtrange = 'empty' THEN\n q :=format(\n $q$\n DO $block$\n BEGIN\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n ALTER TABLE %I\n ADD CONSTRAINT %I\n CHECK (((datetime IS NULL) AND (end_datetime IS NULL))) NOT VALID\n ;\n ALTER TABLE %I\n VALIDATE CONSTRAINT %I\n ;\n\n\n\n EXCEPTION WHEN others THEN\n RAISE WARNING '%%, Issue Altering Constraints. Please run update_partition_stats(%I)', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n $block$;\n $q$,\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t),\n t\n );\n ELSE\n q :=format(\n $q$\n DO $block$\n BEGIN\n\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n ALTER TABLE %I\n ADD CONSTRAINT %I\n CHECK (\n (datetime >= %L)\n AND (datetime <= %L)\n AND (end_datetime >= %L)\n AND (end_datetime <= %L)\n ) NOT VALID\n ;\n ALTER TABLE %I\n VALIDATE CONSTRAINT %I\n ;\n\n\n\n EXCEPTION WHEN others THEN\n RAISE WARNING '%%, Issue Altering Constraints. Please run update_partition_stats(%I)', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n $block$;\n $q$,\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t),\n lower(_dtrange),\n upper(_dtrange),\n lower(_edtrange),\n upper(_edtrange),\n t,\n format('%s_dt', t),\n t\n );\n END IF;\n PERFORM run_or_queue(q);\n RETURN t;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION check_partition(\n _collection text,\n _dtrange tstzrange,\n _edtrange tstzrange\n) RETURNS text AS $$\nDECLARE\n c RECORD;\n pm RECORD;\n _partition_name text;\n _partition_dtrange tstzrange;\n _constraint_dtrange tstzrange;\n _constraint_edtrange tstzrange;\n q text;\n deferrable_q text;\n err_context text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=_collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', _collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n\n IF c.partition_trunc IS NOT NULL THEN\n _partition_dtrange := tstzrange(\n date_trunc(c.partition_trunc, lower(_dtrange)),\n date_trunc(c.partition_trunc, lower(_dtrange)) + (concat('1 ', c.partition_trunc))::interval,\n '[)'\n );\n ELSE\n _partition_dtrange := '[-infinity, infinity]'::tstzrange;\n END IF;\n\n IF NOT _partition_dtrange @> _dtrange THEN\n RAISE EXCEPTION 'dtrange % is greater than the partition size % for collection %', _dtrange, c.partition_trunc, _collection;\n END IF;\n\n\n IF c.partition_trunc = 'year' THEN\n _partition_name := format('_items_%s_%s', c.key, to_char(lower(_partition_dtrange),'YYYY'));\n ELSIF c.partition_trunc = 'month' THEN\n _partition_name := format('_items_%s_%s', c.key, to_char(lower(_partition_dtrange),'YYYYMM'));\n ELSE\n _partition_name := format('_items_%s', c.key);\n END IF;\n\n SELECT * INTO pm FROM partition_sys_meta WHERE collection=_collection AND partition_dtrange @> _dtrange;\n IF FOUND THEN\n RAISE NOTICE '% % %', _edtrange, _dtrange, pm;\n _constraint_edtrange :=\n tstzrange(\n least(\n lower(_edtrange),\n nullif(lower(pm.constraint_edtrange), '-infinity')\n ),\n greatest(\n upper(_edtrange),\n nullif(upper(pm.constraint_edtrange), 'infinity')\n ),\n '[]'\n );\n _constraint_dtrange :=\n tstzrange(\n least(\n lower(_dtrange),\n nullif(lower(pm.constraint_dtrange), '-infinity')\n ),\n greatest(\n upper(_dtrange),\n nullif(upper(pm.constraint_dtrange), 'infinity')\n ),\n '[]'\n );\n\n IF pm.constraint_edtrange @> _edtrange AND pm.constraint_dtrange @> _dtrange THEN\n RETURN pm.partition;\n ELSE\n PERFORM drop_table_constraints(_partition_name);\n END IF;\n ELSE\n _constraint_edtrange := _edtrange;\n _constraint_dtrange := _dtrange;\n END IF;\n RAISE NOTICE 'Creating partition % %', _partition_name, _partition_dtrange;\n IF c.partition_trunc IS NULL THEN\n q := format(\n $q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n $q$,\n _partition_name,\n _collection,\n concat(_partition_name,'_pk'),\n _partition_name\n );\n ELSE\n q := format(\n $q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L) PARTITION BY RANGE (datetime);\n CREATE TABLE IF NOT EXISTS %I partition OF %I FOR VALUES FROM (%L) TO (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n $q$,\n format('_items_%s', c.key),\n _collection,\n _partition_name,\n format('_items_%s', c.key),\n lower(_partition_dtrange),\n upper(_partition_dtrange),\n format('%s_pk', _partition_name),\n _partition_name\n );\n END IF;\n\n BEGIN\n EXECUTE q;\n EXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', _partition_name;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\n END;\n PERFORM create_table_constraints(_partition_name, _constraint_dtrange, _constraint_edtrange);\n PERFORM maintain_partitions(_partition_name);\n PERFORM update_partition_stats_q(_partition_name, true);\n REFRESH MATERIALIZED VIEW partitions;\n REFRESH MATERIALIZED VIEW partition_steps;\n RETURN _partition_name;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION repartition(_collection text, _partition_trunc text, triggered boolean DEFAULT FALSE) RETURNS text AS $$\nDECLARE\n c RECORD;\n q text;\n from_trunc text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=_collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', _collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n IF triggered THEN\n RAISE NOTICE 'Converting % to % partitioning via Trigger', _collection, _partition_trunc;\n ELSE\n RAISE NOTICE 'Converting % from using % to % partitioning', _collection, c.partition_trunc, _partition_trunc;\n IF c.partition_trunc IS NOT DISTINCT FROM _partition_trunc THEN\n RAISE NOTICE 'Collection % already set to use partition by %', _collection, _partition_trunc;\n RETURN _collection;\n END IF;\n END IF;\n\n IF EXISTS (SELECT 1 FROM partitions_view WHERE collection=_collection LIMIT 1) THEN\n EXECUTE format(\n $q$\n CREATE TEMP TABLE changepartitionstaging ON COMMIT DROP AS SELECT * FROM %I;\n DROP TABLE IF EXISTS %I CASCADE;\n WITH p AS (\n SELECT\n collection,\n CASE WHEN %L IS NULL THEN '-infinity'::timestamptz\n ELSE date_trunc(%L, datetime)\n END as d,\n tstzrange(min(datetime),max(datetime),'[]') as dtrange,\n tstzrange(min(datetime),max(datetime),'[]') as edtrange\n FROM changepartitionstaging\n GROUP BY 1,2\n ) SELECT check_partition(collection, dtrange, edtrange) FROM p;\n INSERT INTO items SELECT * FROM changepartitionstaging;\n DROP TABLE changepartitionstaging;\n $q$,\n concat('_items_', c.key),\n concat('_items_', c.key),\n c.partition_trunc,\n c.partition_trunc\n );\n END IF;\n RETURN _collection;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE OR REPLACE FUNCTION collections_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n q text;\n partition_name text := format('_items_%s', NEW.key);\n partition_exists boolean := false;\n partition_empty boolean := true;\n err_context text;\n loadtemp boolean := FALSE;\nBEGIN\n RAISE NOTICE 'Collection Trigger. % %', NEW.id, NEW.key;\n IF TG_OP = 'UPDATE' AND NEW.partition_trunc IS DISTINCT FROM OLD.partition_trunc THEN\n PERFORM repartition(NEW.id, NEW.partition_trunc, TRUE);\n END IF;\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER collections_trigger AFTER\nINSERT\nOR\nUPDATE ON collections\nFOR EACH ROW EXECUTE FUNCTION collections_trigger_func();\n\nCREATE OR REPLACE FUNCTION chunker(\n IN _where text,\n OUT s timestamptz,\n OUT e timestamptz\n) RETURNS SETOF RECORD AS $$\nDECLARE\n explain jsonb;\nBEGIN\n IF _where IS NULL THEN\n _where := ' TRUE ';\n END IF;\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s;', _where)\n INTO explain;\n\n RETURN QUERY\n WITH t AS (\n SELECT j->>0 as p FROM\n jsonb_path_query(\n explain,\n 'strict $.**.\"Relation Name\" ? (@ != null)'\n ) j\n ),\n parts AS (\n SELECT sdate, edate FROM t JOIN partition_steps ON (t.p = name)\n ),\n times AS (\n SELECT sdate FROM parts\n UNION\n SELECT edate FROM parts\n ),\n uniq AS (\n SELECT DISTINCT sdate FROM times ORDER BY sdate\n ),\n last AS (\n SELECT sdate, lead(sdate, 1) over () as edate FROM uniq\n )\n SELECT sdate, edate FROM last WHERE edate IS NOT NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION partition_queries(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN partitions text[] DEFAULT NULL\n) RETURNS SETOF text AS $$\nDECLARE\n query text;\n sdate timestamptz;\n edate timestamptz;\nBEGIN\nIF _where IS NULL OR trim(_where) = '' THEN\n _where = ' TRUE ';\nEND IF;\nRAISE NOTICE 'Getting chunks for % %', _where, _orderby;\nIF _orderby ILIKE 'datetime d%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 DESC LOOP\n RETURN NEXT format($q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n $q$,\n sdate,\n edate,\n _where,\n _orderby\n );\n END LOOP;\nELSIF _orderby ILIKE 'datetime a%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 ASC LOOP\n RETURN NEXT format($q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n $q$,\n sdate,\n edate,\n _where,\n _orderby\n );\n END LOOP;\nELSE\n query := format($q$\n SELECT * FROM items\n WHERE %s\n ORDER BY %s\n $q$, _where, _orderby\n );\n\n RETURN NEXT query;\n RETURN;\nEND IF;\n\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION partition_query_view(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN _limit int DEFAULT 10\n) RETURNS text AS $$\n WITH p AS (\n SELECT * FROM partition_queries(_where, _orderby) p\n )\n SELECT\n CASE WHEN EXISTS (SELECT 1 FROM p) THEN\n (SELECT format($q$\n SELECT * FROM (\n %s\n ) total LIMIT %s\n $q$,\n string_agg(\n format($q$ SELECT * FROM ( %s ) AS sub $q$, p),\n '\n UNION ALL\n '\n ),\n _limit\n ))\n ELSE NULL\n END FROM p;\n$$ LANGUAGE SQL IMMUTABLE;\n\n\n\n\nCREATE OR REPLACE FUNCTION stac_search_to_where(j jsonb) RETURNS text AS $$\nDECLARE\n where_segments text[];\n _where text;\n dtrange tstzrange;\n collections text[];\n geom geometry;\n sdate timestamptz;\n edate timestamptz;\n filterlang text;\n filter jsonb := j->'filter';\nBEGIN\n IF j ? 'ids' THEN\n where_segments := where_segments || format('id = ANY (%L) ', to_text_array(j->'ids'));\n END IF;\n\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n where_segments := where_segments || format('collection = ANY (%L) ', collections);\n END IF;\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n\n where_segments := where_segments || format(' datetime <= %L::timestamptz AND end_datetime >= %L::timestamptz ',\n edate,\n sdate\n );\n END IF;\n\n geom := stac_geom(j);\n IF geom IS NOT NULL THEN\n where_segments := where_segments || format('st_intersects(geometry, %L)',geom);\n END IF;\n\n filterlang := COALESCE(\n j->>'filter-lang',\n get_setting('default_filter_lang', j->'conf')\n );\n IF NOT filter @? '$.**.op' THEN\n filterlang := 'cql-json';\n END IF;\n\n IF filterlang NOT IN ('cql-json','cql2-json') AND j ? 'filter' THEN\n RAISE EXCEPTION '% is not a supported filter-lang. Please use cql-json or cql2-json.', filterlang;\n END IF;\n\n IF j ? 'query' AND j ? 'filter' THEN\n RAISE EXCEPTION 'Can only use either query or filter at one time.';\n END IF;\n\n IF j ? 'query' THEN\n filter := query_to_cql2(j->'query');\n ELSIF filterlang = 'cql-json' THEN\n filter := cql1_to_cql2(filter);\n END IF;\n RAISE NOTICE 'FILTER: %', filter;\n where_segments := where_segments || cql2_query(filter);\n IF cardinality(where_segments) < 1 THEN\n RETURN ' TRUE ';\n END IF;\n\n _where := array_to_string(array_remove(where_segments, NULL), ' AND ');\n\n IF _where IS NULL OR BTRIM(_where) = '' THEN\n RETURN ' TRUE ';\n END IF;\n RETURN _where;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION parse_sort_dir(_dir text, reverse boolean default false) RETURNS text AS $$\n WITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n ) SELECT\n CASE\n WHEN NOT reverse THEN d\n WHEN d = 'ASC' THEN 'DESC'\n WHEN d = 'DESC' THEN 'ASC'\n END\n FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION sort_dir_to_op(_dir text, prev boolean default false) RETURNS text AS $$\n WITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n ) SELECT\n CASE\n WHEN d = 'ASC' AND prev THEN '<='\n WHEN d = 'DESC' AND prev THEN '>='\n WHEN d = 'ASC' THEN '>='\n WHEN d = 'DESC' THEN '<='\n END\n FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION sort_sqlorderby(\n _search jsonb DEFAULT NULL,\n reverse boolean DEFAULT FALSE\n) RETURNS text AS $$\n WITH sortby AS (\n SELECT coalesce(_search->'sortby','[{\"field\":\"datetime\", \"direction\":\"desc\"}]') as sort\n ), withid AS (\n SELECT CASE\n WHEN sort @? '$[*] ? (@.field == \"id\")' THEN sort\n ELSE sort || '[{\"field\":\"id\", \"direction\":\"desc\"}]'::jsonb\n END as sort\n FROM sortby\n ), withid_rows AS (\n SELECT jsonb_array_elements(sort) as value FROM withid\n ),sorts AS (\n SELECT\n coalesce(\n (queryable(value->>'field')).expression\n ) as key,\n parse_sort_dir(value->>'direction', reverse) as dir\n FROM withid_rows\n )\n SELECT array_to_string(\n array_agg(concat(key, ' ', dir)),\n ', '\n ) FROM sorts;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_sort_dir(sort_item jsonb) RETURNS text AS $$\n SELECT CASE WHEN sort_item->>'direction' ILIKE 'desc%' THEN 'DESC' ELSE 'ASC' END;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION get_token_val_str(\n _field text,\n _item items\n) RETURNS text AS $$\nDECLARE\n q text;\n literal text;\nBEGIN\n q := format($q$ SELECT quote_literal(%s) FROM (SELECT $1.*) as r;$q$, _field);\n EXECUTE q INTO literal USING _item;\n RETURN literal;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n\nCREATE OR REPLACE FUNCTION get_token_record(IN _token text, OUT prev BOOLEAN, OUT item items) RETURNS RECORD AS $$\nDECLARE\n _itemid text := _token;\n _collectionid text;\nBEGIN\n IF _token IS NULL THEN\n RETURN;\n END IF;\n RAISE NOTICE 'Looking for token: %', _token;\n prev := FALSE;\n IF _token ILIKE 'prev:%' THEN\n _itemid := replace(_token, 'prev:','');\n prev := TRUE;\n ELSIF _token ILIKE 'next:%' THEN\n _itemid := replace(_token, 'next:', '');\n END IF;\n SELECT id INTO _collectionid FROM collections WHERE _itemid LIKE concat(id,':%');\n IF FOUND THEN\n _itemid := replace(_itemid, concat(_collectionid,':'), '');\n SELECT * INTO item FROM items WHERE id=_itemid AND collection=_collectionid;\n ELSE\n SELECT * INTO item FROM items WHERE id=_itemid;\n END IF;\n IF item IS NULL THEN\n RAISE EXCEPTION 'Could not find item using token: % item: % collection: %', _token, _itemid, _collectionid;\n END IF;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\n\nCREATE OR REPLACE FUNCTION get_token_filter(\n _sortby jsonb DEFAULT '[{\"field\":\"datetime\",\"direction\":\"desc\"}]'::jsonb,\n token_item items DEFAULT NULL,\n prev boolean DEFAULT FALSE,\n inclusive boolean DEFAULT FALSE\n) RETURNS text AS $$\nDECLARE\n ltop text := '<';\n gtop text := '>';\n dir text;\n sort record;\n orfilter text := '';\n orfilters text[] := '{}'::text[];\n andfilters text[] := '{}'::text[];\n output text;\n token_where text;\nBEGIN\n IF _sortby IS NULL OR _sortby = '[]'::jsonb THEN\n _sortby := '[{\"field\":\"datetime\",\"direction\":\"desc\"}]'::jsonb;\n END IF;\n _sortby := _sortby || jsonb_build_object('field','id','direction',_sortby->0->>'direction');\n RAISE NOTICE 'Getting Token Filter. % %', _sortby, token_item;\n IF inclusive THEN\n orfilters := orfilters || format('( id=%L AND collection=%L )' , token_item.id, token_item.collection);\n END IF;\n\n FOR sort IN\n WITH s1 AS (\n SELECT\n _row,\n (queryable(value->>'field')).expression as _field,\n (value->>'field' = 'id') as _isid,\n get_sort_dir(value) as _dir\n FROM jsonb_array_elements(_sortby)\n WITH ORDINALITY AS t(value, _row)\n )\n SELECT\n _row,\n _field,\n _dir,\n get_token_val_str(_field, token_item) as _val\n FROM s1\n WHERE _row <= (SELECT min(_row) FROM s1 WHERE _isid)\n LOOP\n orfilter := NULL;\n RAISE NOTICE 'SORT: %', sort;\n IF sort._val IS NOT NULL AND ((prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC')) THEN\n orfilter := format($f$(\n (%s %s %s) OR (%s IS NULL)\n )$f$,\n sort._field,\n ltop,\n sort._val,\n sort._val\n );\n ELSIF sort._val IS NULL AND ((prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC')) THEN\n RAISE NOTICE '< but null';\n orfilter := format('%s IS NOT NULL', sort._field);\n ELSIF sort._val IS NULL THEN\n RAISE NOTICE '> but null';\n ELSE\n orfilter := format($f$(\n (%s %s %s) OR (%s IS NULL)\n )$f$,\n sort._field,\n gtop,\n sort._val,\n sort._field\n );\n END IF;\n RAISE NOTICE 'ORFILTER: %', orfilter;\n\n IF orfilter IS NOT NULL THEN\n IF sort._row = 1 THEN\n orfilters := orfilters || orfilter;\n ELSE\n orfilters := orfilters || format('(%s AND %s)', array_to_string(andfilters, ' AND '), orfilter);\n END IF;\n END IF;\n IF sort._val IS NOT NULL THEN\n andfilters := andfilters || format('%s = %s', sort._field, sort._val);\n ELSE\n andfilters := andfilters || format('%s IS NULL', sort._field);\n END IF;\n END LOOP;\n\n output := array_to_string(orfilters, ' OR ');\n\n token_where := concat('(',coalesce(output,'true'),')');\n IF trim(token_where) = '' THEN\n token_where := NULL;\n END IF;\n RAISE NOTICE 'TOKEN_WHERE: %',token_where;\n RETURN token_where;\n END;\n$$ LANGUAGE PLPGSQL SET transform_null_equals TO TRUE\n;\n\nCREATE OR REPLACE FUNCTION search_tohash(jsonb) RETURNS jsonb AS $$\n SELECT $1 - '{token,limit,context,includes,excludes}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION search_hash(jsonb, jsonb) RETURNS text AS $$\n SELECT md5(concat(search_tohash($1)::text,$2::text));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TABLE IF NOT EXISTS searches(\n hash text GENERATED ALWAYS AS (search_hash(search, metadata)) STORED PRIMARY KEY,\n search jsonb NOT NULL,\n _where text,\n orderby text,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n metadata jsonb DEFAULT '{}'::jsonb NOT NULL\n);\n\nCREATE TABLE IF NOT EXISTS search_wheres(\n id bigint generated always as identity primary key,\n _where text NOT NULL,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n statslastupdated timestamptz,\n estimated_count bigint,\n estimated_cost float,\n time_to_estimate float,\n total_count bigint,\n time_to_count float,\n partitions text[]\n);\n\nCREATE INDEX IF NOT EXISTS search_wheres_partitions ON search_wheres USING GIN (partitions);\nCREATE UNIQUE INDEX IF NOT EXISTS search_wheres_where ON search_wheres ((md5(_where)));\n\nCREATE OR REPLACE FUNCTION where_stats(inwhere text, updatestats boolean default false, conf jsonb default null) RETURNS search_wheres AS $$\nDECLARE\n t timestamptz;\n i interval;\n explain_json jsonb;\n partitions text[];\n sw search_wheres%ROWTYPE;\n inwhere_hash text := md5(inwhere);\n _context text := lower(context(conf));\n _stats_ttl interval := context_stats_ttl(conf);\n _estimated_cost float := context_estimated_cost(conf);\n _estimated_count int := context_estimated_count(conf);\nBEGIN\n IF _context = 'off' THEN\n sw._where := inwhere;\n return sw;\n END IF;\n\n SELECT * INTO sw FROM search_wheres WHERE md5(_where)=inwhere_hash FOR UPDATE;\n\n -- Update statistics if explicitly set, if statistics do not exist, or statistics ttl has expired\n IF NOT updatestats THEN\n RAISE NOTICE 'Checking if update is needed for: % .', inwhere;\n RAISE NOTICE 'Stats Last Updated: %', sw.statslastupdated;\n RAISE NOTICE 'TTL: %, Age: %', _stats_ttl, now() - sw.statslastupdated;\n RAISE NOTICE 'Context: %, Existing Total: %', _context, sw.total_count;\n IF\n sw.statslastupdated IS NULL\n OR (now() - sw.statslastupdated) > _stats_ttl\n OR (context(conf) != 'off' AND sw.total_count IS NULL)\n THEN\n updatestats := TRUE;\n END IF;\n END IF;\n\n sw._where := inwhere;\n sw.lastused := now();\n sw.usecount := coalesce(sw.usecount,0) + 1;\n\n IF NOT updatestats THEN\n UPDATE search_wheres SET\n lastused = sw.lastused,\n usecount = sw.usecount\n WHERE md5(_where) = inwhere_hash\n RETURNING * INTO sw\n ;\n RETURN sw;\n END IF;\n\n -- Use explain to get estimated count/cost and a list of the partitions that would be hit by the query\n t := clock_timestamp();\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s', inwhere)\n INTO explain_json;\n RAISE NOTICE 'Time for just the explain: %', clock_timestamp() - t;\n i := clock_timestamp() - t;\n\n sw.statslastupdated := now();\n sw.estimated_count := explain_json->0->'Plan'->'Plan Rows';\n sw.estimated_cost := explain_json->0->'Plan'->'Total Cost';\n sw.time_to_estimate := extract(epoch from i);\n\n RAISE NOTICE 'ESTIMATED_COUNT: % < %', sw.estimated_count, _estimated_count;\n RAISE NOTICE 'ESTIMATED_COST: % < %', sw.estimated_cost, _estimated_cost;\n\n -- Do a full count of rows if context is set to on or if auto is set and estimates are low enough\n IF\n _context = 'on'\n OR\n ( _context = 'auto' AND\n (\n sw.estimated_count < _estimated_count\n AND\n sw.estimated_cost < _estimated_cost\n )\n )\n THEN\n t := clock_timestamp();\n RAISE NOTICE 'Calculating actual count...';\n EXECUTE format(\n 'SELECT count(*) FROM items WHERE %s',\n inwhere\n ) INTO sw.total_count;\n i := clock_timestamp() - t;\n RAISE NOTICE 'Actual Count: % -- %', sw.total_count, i;\n sw.time_to_count := extract(epoch FROM i);\n ELSE\n sw.total_count := NULL;\n sw.time_to_count := NULL;\n END IF;\n\n\n INSERT INTO search_wheres\n (_where, lastused, usecount, statslastupdated, estimated_count, estimated_cost, time_to_estimate, partitions, total_count, time_to_count)\n SELECT sw._where, sw.lastused, sw.usecount, sw.statslastupdated, sw.estimated_count, sw.estimated_cost, sw.time_to_estimate, sw.partitions, sw.total_count, sw.time_to_count\n ON CONFLICT ((md5(_where)))\n DO UPDATE\n SET\n lastused = sw.lastused,\n usecount = sw.usecount,\n statslastupdated = sw.statslastupdated,\n estimated_count = sw.estimated_count,\n estimated_cost = sw.estimated_cost,\n time_to_estimate = sw.time_to_estimate,\n total_count = sw.total_count,\n time_to_count = sw.time_to_count\n ;\n RETURN sw;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION search_query(\n _search jsonb = '{}'::jsonb,\n updatestats boolean = false,\n _metadata jsonb = '{}'::jsonb\n) RETURNS searches AS $$\nDECLARE\n search searches%ROWTYPE;\n pexplain jsonb;\n t timestamptz;\n i interval;\nBEGIN\n SELECT * INTO search FROM searches\n WHERE hash=search_hash(_search, _metadata) FOR UPDATE;\n\n -- Calculate the where clause if not already calculated\n IF search._where IS NULL THEN\n search._where := stac_search_to_where(_search);\n END IF;\n\n -- Calculate the order by clause if not already calculated\n IF search.orderby IS NULL THEN\n search.orderby := sort_sqlorderby(_search);\n END IF;\n\n PERFORM where_stats(search._where, updatestats, _search->'conf');\n\n search.lastused := now();\n search.usecount := coalesce(search.usecount, 0) + 1;\n INSERT INTO searches (search, _where, orderby, lastused, usecount, metadata)\n VALUES (_search, search._where, search.orderby, search.lastused, search.usecount, _metadata)\n ON CONFLICT (hash) DO\n UPDATE SET\n _where = EXCLUDED._where,\n orderby = EXCLUDED.orderby,\n lastused = EXCLUDED.lastused,\n usecount = EXCLUDED.usecount,\n metadata = EXCLUDED.metadata\n RETURNING * INTO search\n ;\n RETURN search;\n\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE OR REPLACE FUNCTION search_rows(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN partitions text[] DEFAULT NULL,\n IN _limit int DEFAULT 10\n) RETURNS SETOF items AS $$\nDECLARE\n base_query text;\n query text;\n sdate timestamptz;\n edate timestamptz;\n n int;\n records_left int := _limit;\n timer timestamptz := clock_timestamp();\n full_timer timestamptz := clock_timestamp();\nBEGIN\nIF _where IS NULL OR trim(_where) = '' THEN\n _where = ' TRUE ';\nEND IF;\nRAISE NOTICE 'Getting chunks for % %', _where, _orderby;\n\nbase_query := $q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n LIMIT %L\n$q$;\n\nIF _orderby ILIKE 'datetime d%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 DESC LOOP\n RAISE NOTICE 'Running Query for % to %. %', sdate, edate, age_ms(full_timer);\n query := format(\n base_query,\n sdate,\n edate,\n _where,\n _orderby,\n records_left\n );\n RAISE LOG 'QUERY: %', query;\n timer := clock_timestamp();\n RETURN QUERY EXECUTE query;\n\n GET DIAGNOSTICS n = ROW_COUNT;\n records_left := records_left - n;\n RAISE NOTICE 'Returned %/% Rows From % to %. % to go. Time: %ms', n, _limit, sdate, edate, records_left, age_ms(timer);\n timer := clock_timestamp();\n IF records_left <= 0 THEN\n RAISE NOTICE 'SEARCH_ROWS TOOK %ms', age_ms(full_timer);\n RETURN;\n END IF;\n END LOOP;\nELSIF _orderby ILIKE 'datetime a%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 ASC LOOP\n RAISE NOTICE 'Running Query for % to %. %', sdate, edate, age_ms(full_timer);\n query := format(\n base_query,\n sdate,\n edate,\n _where,\n _orderby,\n records_left\n );\n RAISE LOG 'QUERY: %', query;\n timer := clock_timestamp();\n RETURN QUERY EXECUTE query;\n\n GET DIAGNOSTICS n = ROW_COUNT;\n records_left := records_left - n;\n RAISE NOTICE 'Returned %/% Rows From % to %. % to go. Time: %ms', n, _limit, sdate, edate, records_left, age_ms(timer);\n timer := clock_timestamp();\n IF records_left <= 0 THEN\n RAISE NOTICE 'SEARCH_ROWS TOOK %ms', age_ms(full_timer);\n RETURN;\n END IF;\n END LOOP;\nELSE\n query := format($q$\n SELECT * FROM items\n WHERE %s\n ORDER BY %s\n LIMIT %L\n $q$, _where, _orderby, _limit\n );\n RAISE LOG 'QUERY: %', query;\n timer := clock_timestamp();\n RETURN QUERY EXECUTE query;\n RAISE NOTICE 'FULL QUERY TOOK %ms', age_ms(timer);\nEND IF;\nRAISE NOTICE 'SEARCH_ROWS TOOK %ms', age_ms(full_timer);\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE UNLOGGED TABLE format_item_cache(\n id text,\n collection text,\n fields text,\n hydrated bool,\n output jsonb,\n lastused timestamptz DEFAULT now(),\n usecount int DEFAULT 1,\n timetoformat float,\n PRIMARY KEY (collection, id, fields, hydrated)\n);\nCREATE INDEX ON format_item_cache (lastused);\n\nCREATE OR REPLACE FUNCTION format_item(_item items, _fields jsonb DEFAULT '{}', _hydrated bool DEFAULT TRUE) RETURNS jsonb AS $$\nDECLARE\n cache bool := get_setting_bool('format_cache');\n _output jsonb := null;\n t timestamptz := clock_timestamp();\nBEGIN\n IF cache THEN\n SELECT output INTO _output FROM format_item_cache\n WHERE id=_item.id AND collection=_item.collection AND fields=_fields::text AND hydrated=_hydrated;\n END IF;\n IF _output IS NULL THEN\n IF _hydrated THEN\n _output := content_hydrate(_item, _fields);\n ELSE\n _output := content_nonhydrated(_item, _fields);\n END IF;\n END IF;\n IF cache THEN\n INSERT INTO format_item_cache (id, collection, fields, hydrated, output, timetoformat)\n VALUES (_item.id, _item.collection, _fields::text, _hydrated, _output, age_ms(t))\n ON CONFLICT(collection, id, fields, hydrated) DO\n UPDATE\n SET lastused=now(), usecount = format_item_cache.usecount + 1\n ;\n END IF;\n RETURN _output;\n\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION search(_search jsonb = '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n searches searches%ROWTYPE;\n _where text;\n orderby text;\n search_where search_wheres%ROWTYPE;\n total_count bigint;\n token record;\n token_prev boolean;\n token_item items%ROWTYPE;\n token_where text;\n full_where text;\n timer timestamptz := clock_timestamp();\n hydrate bool := NOT (_search->'conf'->>'nohydrate' IS NOT NULL AND (_search->'conf'->>'nohydrate')::boolean = true);\n prev text;\n next text;\n context jsonb;\n collection jsonb;\n out_records jsonb;\n out_len int;\n _limit int := coalesce((_search->>'limit')::int, 10);\n _querylimit int;\n _fields jsonb := coalesce(_search->'fields', '{}'::jsonb);\n has_prev boolean := FALSE;\n has_next boolean := FALSE;\nBEGIN\n searches := search_query(_search);\n _where := searches._where;\n orderby := searches.orderby;\n search_where := where_stats(_where);\n total_count := coalesce(search_where.total_count, search_where.estimated_count);\n RAISE NOTICE 'SEARCH:TOKEN: %', _search->>'token';\n token := get_token_record(_search->>'token');\n RAISE NOTICE '***TOKEN: %', token;\n _querylimit := _limit + 1;\n IF token IS NOT NULL THEN\n token_prev := token.prev;\n token_item := token.item;\n token_where := get_token_filter(_search->'sortby', token_item, token_prev, FALSE);\n RAISE LOG 'TOKEN_WHERE: %', token_where;\n IF token_prev THEN -- if we are using a prev token, we know has_next is true\n RAISE LOG 'There is a previous token, so automatically setting has_next to true';\n has_next := TRUE;\n orderby := sort_sqlorderby(_search, TRUE);\n ELSE\n RAISE LOG 'There is a next token, so automatically setting has_prev to true';\n has_prev := TRUE;\n\n END IF;\n ELSE -- if there was no token, we know there is no prev\n RAISE LOG 'There is no token, so we know there is no prev. setting has_prev to false';\n has_prev := FALSE;\n END IF;\n\n full_where := concat_ws(' AND ', _where, token_where);\n RAISE NOTICE 'FULL WHERE CLAUSE: %', full_where;\n RAISE NOTICE 'Time to get counts and build query %', age_ms(timer);\n timer := clock_timestamp();\n\n IF hydrate THEN\n RAISE NOTICE 'Getting hydrated data.';\n ELSE\n RAISE NOTICE 'Getting non-hydrated data.';\n END IF;\n RAISE NOTICE 'CACHE SET TO %', get_setting_bool('format_cache');\n RAISE NOTICE 'Time to set hydration/formatting %', age_ms(timer);\n timer := clock_timestamp();\n SELECT jsonb_agg(format_item(i, _fields, hydrate)) INTO out_records\n FROM search_rows(\n full_where,\n orderby,\n search_where.partitions,\n _querylimit\n ) as i;\n\n RAISE NOTICE 'Time to fetch rows %', age_ms(timer);\n timer := clock_timestamp();\n\n\n IF token_prev THEN\n out_records := flip_jsonb_array(out_records);\n END IF;\n\n RAISE NOTICE 'Query returned % records.', jsonb_array_length(out_records);\n RAISE LOG 'TOKEN: % %', token_item.id, token_item.collection;\n RAISE LOG 'RECORD_1: % %', out_records->0->>'id', out_records->0->>'collection';\n RAISE LOG 'RECORD-1: % %', out_records->-1->>'id', out_records->-1->>'collection';\n\n -- REMOVE records that were from our token\n IF out_records->0->>'id' = token_item.id AND out_records->0->>'collection' = token_item.collection THEN\n out_records := out_records - 0;\n ELSIF out_records->-1->>'id' = token_item.id AND out_records->-1->>'collection' = token_item.collection THEN\n out_records := out_records - -1;\n END IF;\n\n out_len := jsonb_array_length(out_records);\n\n IF out_len = _limit + 1 THEN\n IF token_prev THEN\n has_prev := TRUE;\n out_records := out_records - 0;\n ELSE\n has_next := TRUE;\n out_records := out_records - -1;\n END IF;\n END IF;\n\n IF has_next THEN\n next := concat(out_records->-1->>'collection', ':', out_records->-1->>'id');\n RAISE NOTICE 'HAS NEXT | %', next;\n END IF;\n\n IF has_prev THEN\n prev := concat(out_records->0->>'collection', ':', out_records->0->>'id');\n RAISE NOTICE 'HAS PREV | %', prev;\n END IF;\n\n RAISE NOTICE 'Time to get prev/next %', age_ms(timer);\n timer := clock_timestamp();\n\n IF context(_search->'conf') != 'off' THEN\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'matched', total_count,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\n ELSE\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\n END IF;\n\n collection := jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb),\n 'next', next,\n 'prev', prev,\n 'context', context\n );\n\n RAISE NOTICE 'Time to build final json %', age_ms(timer);\n timer := clock_timestamp();\n\n RAISE NOTICE 'Total Time: %', age_ms(current_timestamp);\n RAISE NOTICE 'RETURNING % records. NEXT: %. PREV: %', collection->'context'->>'returned', collection->>'next', collection->>'prev';\n RETURN collection;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION search_cursor(_search jsonb = '{}'::jsonb) RETURNS refcursor AS $$\nDECLARE\n curs refcursor;\n searches searches%ROWTYPE;\n _where text;\n _orderby text;\n q text;\n\nBEGIN\n searches := search_query(_search);\n _where := searches._where;\n _orderby := searches.orderby;\n\n OPEN curs FOR\n WITH p AS (\n SELECT * FROM partition_queries(_where, _orderby) p\n )\n SELECT\n CASE WHEN EXISTS (SELECT 1 FROM p) THEN\n (SELECT format($q$\n SELECT * FROM (\n %s\n ) total\n $q$,\n string_agg(\n format($q$ SELECT * FROM ( %s ) AS sub $q$, p),\n '\n UNION ALL\n '\n )\n ))\n ELSE NULL\n END FROM p;\n RETURN curs;\nEND;\n$$ LANGUAGE PLPGSQL;\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION tileenvelope(zoom int, x int, y int) RETURNS geometry AS $$\nWITH t AS (\n SELECT\n 20037508.3427892 as merc_max,\n -20037508.3427892 as merc_min,\n (2 * 20037508.3427892) / (2 ^ zoom) as tile_size\n)\nSELECT st_makeenvelope(\n merc_min + (tile_size * x),\n merc_max - (tile_size * (y + 1)),\n merc_min + (tile_size * (x + 1)),\n merc_max - (tile_size * y),\n 3857\n) FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;DROP FUNCTION IF EXISTS mercgrid;\n\n\nCREATE OR REPLACE FUNCTION ftime() RETURNS interval as $$\nSELECT age(clock_timestamp(), transaction_timestamp());\n$$ LANGUAGE SQL;\nSET SEARCH_PATH to pgstac, public;\n\nDROP FUNCTION IF EXISTS geometrysearch;\nCREATE OR REPLACE FUNCTION geometrysearch(\n IN geom geometry,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE, -- Return as soon as the passed in geometry is full covered\n IN skipcovered boolean DEFAULT TRUE -- Skip any items that would show up completely under the previous items\n) RETURNS jsonb AS $$\nDECLARE\n search searches%ROWTYPE;\n curs refcursor;\n _where text;\n query text;\n iter_record items%ROWTYPE;\n out_records jsonb := '{}'::jsonb[];\n exit_flag boolean := FALSE;\n counter int := 1;\n scancounter int := 1;\n remaining_limit int := _scanlimit;\n tilearea float;\n unionedgeom geometry;\n clippedgeom geometry;\n unionedgeom_area float := 0;\n prev_area float := 0;\n excludes text[];\n includes text[];\n\nBEGIN\n DROP TABLE IF EXISTS pgstac_results;\n CREATE TEMP TABLE pgstac_results (content jsonb) ON COMMIT DROP;\n\n -- If skipcovered is true then you will always want to exit when the passed in geometry is full\n IF skipcovered THEN\n exitwhenfull := TRUE;\n END IF;\n\n SELECT * INTO search FROM searches WHERE hash=queryhash;\n\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Search with Query Hash % Not Found', queryhash;\n END IF;\n\n tilearea := st_area(geom);\n _where := format('%s AND st_intersects(geometry, %L::geometry)', search._where, geom);\n\n\n FOR query IN SELECT * FROM partition_queries(_where, search.orderby) LOOP\n query := format('%s LIMIT %L', query, remaining_limit);\n RAISE NOTICE '%', query;\n OPEN curs FOR EXECUTE query;\n LOOP\n FETCH curs INTO iter_record;\n EXIT WHEN NOT FOUND;\n IF exitwhenfull OR skipcovered THEN -- If we are not using exitwhenfull or skipcovered, we do not need to do expensive geometry operations\n clippedgeom := st_intersection(geom, iter_record.geometry);\n\n IF unionedgeom IS NULL THEN\n unionedgeom := clippedgeom;\n ELSE\n unionedgeom := st_union(unionedgeom, clippedgeom);\n END IF;\n\n unionedgeom_area := st_area(unionedgeom);\n\n IF skipcovered AND prev_area = unionedgeom_area THEN\n scancounter := scancounter + 1;\n CONTINUE;\n END IF;\n\n prev_area := unionedgeom_area;\n\n RAISE NOTICE '% % % %', unionedgeom_area/tilearea, counter, scancounter, ftime();\n END IF;\n RAISE NOTICE '% %', iter_record, content_hydrate(iter_record, fields);\n INSERT INTO pgstac_results (content) VALUES (content_hydrate(iter_record, fields));\n\n IF counter >= _limit\n OR scancounter > _scanlimit\n OR ftime() > _timelimit\n OR (exitwhenfull AND unionedgeom_area >= tilearea)\n THEN\n exit_flag := TRUE;\n EXIT;\n END IF;\n counter := counter + 1;\n scancounter := scancounter + 1;\n\n END LOOP;\n CLOSE curs;\n EXIT WHEN exit_flag;\n remaining_limit := _scanlimit - scancounter;\n END LOOP;\n\n SELECT jsonb_agg(content) INTO out_records FROM pgstac_results WHERE content IS NOT NULL;\n\n RETURN jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb)\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDROP FUNCTION IF EXISTS geojsonsearch;\nCREATE OR REPLACE FUNCTION geojsonsearch(\n IN geojson jsonb,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_geomfromgeojson(geojson),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS xyzsearch;\nCREATE OR REPLACE FUNCTION xyzsearch(\n IN _x int,\n IN _y int,\n IN _z int,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_transform(tileenvelope(_z, _x, _y), 4326),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE PROCEDURE analyze_items() AS $$\nDECLARE\n q text;\n timeout_ts timestamptz;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n SELECT format('ANALYZE (VERBOSE, SKIP_LOCKED) %I;', relname) INTO q\n FROM pg_stat_user_tables\n WHERE relname like '_item%' AND (n_mod_since_analyze>0 OR last_analyze IS NULL) LIMIT 1;\n IF NOT FOUND THEN\n EXIT;\n END IF;\n RAISE NOTICE '%', q;\n EXECUTE q;\n COMMIT;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE PROCEDURE validate_constraints() AS $$\nDECLARE\n q text;\nBEGIN\n FOR q IN\n SELECT\n FORMAT(\n 'ALTER TABLE %I.%I VALIDATE CONSTRAINT %I;',\n nsp.nspname,\n cls.relname,\n con.conname\n )\n\n FROM pg_constraint AS con\n JOIN pg_class AS cls\n ON con.conrelid = cls.oid\n JOIN pg_namespace AS nsp\n ON cls.relnamespace = nsp.oid\n WHERE convalidated = FALSE AND contype in ('c','f')\n AND nsp.nspname = 'pgstac'\n LOOP\n RAISE NOTICE '%', q;\n PERFORM run_or_queue(q);\n COMMIT;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION collection_extent(_collection text, runupdate boolean default false) RETURNS jsonb AS $$\nDECLARE\n geom_extent geometry;\n mind timestamptz;\n maxd timestamptz;\n extent jsonb;\nBEGIN\n IF runupdate THEN\n PERFORM update_partition_stats_q(partition)\n FROM partitions_view WHERE collection=_collection;\n END IF;\n SELECT\n min(lower(dtrange)),\n max(upper(edtrange)),\n st_extent(spatial)\n INTO\n mind,\n maxd,\n geom_extent\n FROM partitions_view\n WHERE collection=_collection;\n\n IF geom_extent IS NOT NULL AND mind IS NOT NULL AND maxd IS NOT NULL THEN\n extent := jsonb_build_object(\n 'extent', jsonb_build_object(\n 'spatial', jsonb_build_object(\n 'bbox', to_jsonb(array[array[st_xmin(geom_extent), st_ymin(geom_extent), st_xmax(geom_extent), st_ymax(geom_extent)]])\n ),\n 'temporal', jsonb_build_object(\n 'interval', to_jsonb(array[array[mind, maxd]])\n )\n )\n );\n RETURN extent;\n END IF;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('id', '{\"title\": \"Item ID\",\"description\": \"Item identifier\",\"$ref\": \"https://schemas.stacspec.org/v1.0.0/item-spec/json-schema/item.json#/definitions/core/allOf/2/properties/id\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('geometry', '{\"title\": \"Item Geometry\",\"description\": \"Item Geometry\",\"$ref\": \"https://geojson.org/schema/Feature.json\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('datetime','{\"description\": \"Datetime\",\"type\": \"string\",\"title\": \"Acquired\",\"format\": \"date-time\",\"pattern\": \"(\\\\+00:00|Z)$\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('eo:cloud_cover','{\"$ref\": \"https://stac-extensions.github.io/eo/v1.0.0/schema.json#/definitions/fieldsproperties/eo:cloud_cover\"}','to_int','BTREE');\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDELETE FROM queryables a USING queryables b\n WHERE a.name = b.name AND a.collection_ids IS NOT DISTINCT FROM b.collection_ids AND a.id > b.id;\n\n\nINSERT INTO pgstac_settings (name, value) VALUES\n ('context', 'off'),\n ('context_estimated_count', '100000'),\n ('context_estimated_cost', '100000'),\n ('context_stats_ttl', '1 day'),\n ('default_filter_lang', 'cql2-json'),\n ('additional_properties', 'true'),\n ('use_queue', 'false'),\n ('queue_timeout', '10 minutes'),\n ('update_collection_extent', 'false'),\n ('format_cache', 'false')\nON CONFLICT DO NOTHING\n;\n\nALTER FUNCTION to_text COST 5000;\nALTER FUNCTION to_float COST 5000;\nALTER FUNCTION to_int COST 5000;\nALTER FUNCTION to_tstz COST 5000;\nALTER FUNCTION to_text_array COST 5000;\n\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nGRANT ALL ON SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac to pgstac_admin;\n\n-- pgstac_read role limited to using function apis\nGRANT EXECUTE ON FUNCTION search TO pgstac_read;\nGRANT EXECUTE ON FUNCTION search_query TO pgstac_read;\nGRANT EXECUTE ON FUNCTION item_by_id TO pgstac_read;\nGRANT EXECUTE ON FUNCTION get_item TO pgstac_read;\nGRANT SELECT ON ALL TABLES IN SCHEMA pgstac TO pgstac_read;\n\n\nGRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_ingest;\nGRANT USAGE ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_ingest;\n\nSELECT update_partition_stats_q(partition) FROM partitions_view;\nSELECT set_version('0.7.5');\n" }, { "path": "src/pgstac/migrations/pgstac.0.7.6-0.7.7.sql", "content": "RESET ROLE;\nDO $$\nDECLARE\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='postgis') THEN\n CREATE EXTENSION IF NOT EXISTS postgis;\n END IF;\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='btree_gist') THEN\n CREATE EXTENSION IF NOT EXISTS btree_gist;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_admin;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_read;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_ingest;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n\nGRANT pgstac_admin TO current_user;\n\n-- Function to make sure pgstac_admin is the owner of items\nCREATE OR REPLACE FUNCTION pgstac_admin_owns() RETURNS VOID AS $$\nDECLARE\n f RECORD;\nBEGIN\n FOR f IN (\n SELECT\n concat(\n oid::regproc::text,\n '(',\n coalesce(pg_get_function_identity_arguments(oid),''),\n ')'\n ) AS name,\n CASE prokind WHEN 'f' THEN 'FUNCTION' WHEN 'p' THEN 'PROCEDURE' WHEN 'a' THEN 'AGGREGATE' END as typ\n FROM pg_proc\n WHERE\n pronamespace=to_regnamespace('pgstac')\n AND proowner != to_regrole('pgstac_admin')\n AND proname NOT LIKE 'pg_stat%'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n FOR f IN (\n SELECT\n oid::regclass::text as name,\n CASE relkind\n WHEN 'i' THEN 'INDEX'\n WHEN 'I' THEN 'INDEX'\n WHEN 'p' THEN 'TABLE'\n WHEN 'r' THEN 'TABLE'\n WHEN 'v' THEN 'VIEW'\n WHEN 'S' THEN 'SEQUENCE'\n ELSE NULL\n END as typ\n FROM pg_class\n WHERE relnamespace=to_regnamespace('pgstac') and relowner != to_regrole('pgstac_admin') AND relkind IN ('r','p','v','S') AND relname NOT LIKE 'pg_stat'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\nSELECT pgstac_admin_owns();\n\nCREATE SCHEMA IF NOT EXISTS pgstac AUTHORIZATION pgstac_admin;\n\nGRANT ALL ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_admin;\n\nALTER ROLE pgstac_admin SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_read SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_ingest SET SEARCH_PATH TO pgstac, public;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\n\nGRANT pgstac_read TO pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\n\nSET ROLE pgstac_admin;\n\nSET SEARCH_PATH TO pgstac, public;\n\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS analyze_items;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS validate_constraints;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\nSET client_min_messages TO WARNING;\nSET SEARCH_PATH to pgstac, public;\n-- BEGIN migra calculated SQL\nset check_function_bodies = off;\n\nCREATE OR REPLACE FUNCTION pgstac.search(_search jsonb DEFAULT '{}'::jsonb)\n RETURNS jsonb\n LANGUAGE plpgsql\nAS $function$\nDECLARE\n searches searches%ROWTYPE;\n _where text;\n orderby text;\n search_where search_wheres%ROWTYPE;\n total_count bigint;\n token record;\n token_prev boolean;\n token_item items%ROWTYPE;\n token_where text;\n full_where text;\n init_ts timestamptz := clock_timestamp();\n timer timestamptz := clock_timestamp();\n hydrate bool := NOT (_search->'conf'->>'nohydrate' IS NOT NULL AND (_search->'conf'->>'nohydrate')::boolean = true);\n prev text;\n next text;\n context jsonb;\n collection jsonb;\n out_records jsonb;\n out_len int;\n _limit int := coalesce((_search->>'limit')::int, 10);\n init_limit int := _limit;\n _querylimit int;\n _fields jsonb := coalesce(_search->'fields', '{}'::jsonb);\n has_prev boolean := FALSE;\n has_next boolean := FALSE;\n items_cnt int := coalesce(jsonb_array_length(_search->'ids'),0);\nBEGIN\n RAISE NOTICE 'Items Count: %', items_cnt;\n IF items_cnt > 0 THEN\n IF items_cnt <= _limit THEN\n _limit := items_cnt - 1;\n ELSE\n _limit := items_cnt;\n END IF;\n RAISE NOTICE 'Items is set. Changing limit to %', items_cnt;\n END IF;\n searches := search_query(_search);\n _where := searches._where;\n orderby := searches.orderby;\n search_where := where_stats(_where);\n total_count := coalesce(search_where.total_count, search_where.estimated_count);\n RAISE NOTICE 'SEARCH:TOKEN: %', _search->>'token';\n token := get_token_record(_search->>'token');\n RAISE NOTICE '***TOKEN: %', token;\n _querylimit := _limit + 1;\n IF token IS NOT NULL THEN\n token_prev := token.prev;\n token_item := token.item;\n token_where := get_token_filter(_search->'sortby', token_item, token_prev, FALSE);\n RAISE LOG 'TOKEN_WHERE: % (%ms from search start)', token_where, age_ms(timer);\n IF token_prev THEN -- if we are using a prev token, we know has_next is true\n RAISE LOG 'There is a previous token, so automatically setting has_next to true';\n has_next := TRUE;\n orderby := sort_sqlorderby(_search, TRUE);\n ELSE\n RAISE LOG 'There is a next token, so automatically setting has_prev to true';\n has_prev := TRUE;\n\n END IF;\n ELSE -- if there was no token, we know there is no prev\n RAISE LOG 'There is no token, so we know there is no prev. setting has_prev to false';\n has_prev := FALSE;\n END IF;\n\n full_where := concat_ws(' AND ', _where, token_where);\n RAISE NOTICE 'FULL WHERE CLAUSE: %', full_where;\n RAISE NOTICE 'Time to get counts and build query %', age_ms(timer);\n timer := clock_timestamp();\n\n IF hydrate THEN\n RAISE NOTICE 'Getting hydrated data.';\n ELSE\n RAISE NOTICE 'Getting non-hydrated data.';\n END IF;\n RAISE NOTICE 'CACHE SET TO %', get_setting_bool('format_cache');\n RAISE NOTICE 'Time to set hydration/formatting %', age_ms(timer);\n timer := clock_timestamp();\n SELECT jsonb_agg(format_item(i, _fields, hydrate)) INTO out_records\n FROM search_rows(\n full_where,\n orderby,\n search_where.partitions,\n _querylimit\n ) as i;\n\n RAISE NOTICE 'Time to fetch rows %', age_ms(timer);\n timer := clock_timestamp();\n\n\n IF token_prev THEN\n out_records := flip_jsonb_array(out_records);\n END IF;\n\n RAISE NOTICE 'Query returned % records.', jsonb_array_length(out_records);\n RAISE LOG 'TOKEN: % %', token_item.id, token_item.collection;\n RAISE LOG 'RECORD_1: % %', out_records->0->>'id', out_records->0->>'collection';\n RAISE LOG 'RECORD-1: % %', out_records->-1->>'id', out_records->-1->>'collection';\n\n -- REMOVE records that were from our token\n IF out_records->0->>'id' = token_item.id AND out_records->0->>'collection' = token_item.collection THEN\n out_records := out_records - 0;\n ELSIF out_records->-1->>'id' = token_item.id AND out_records->-1->>'collection' = token_item.collection THEN\n out_records := out_records - -1;\n END IF;\n\n out_len := jsonb_array_length(out_records);\n\n IF out_len = init_limit + 1 THEN\n IF token_prev THEN\n has_prev := TRUE;\n out_records := out_records - 0;\n ELSE\n has_next := TRUE;\n out_records := out_records - -1;\n END IF;\n END IF;\n\n IF has_next THEN\n next := concat(out_records->-1->>'collection', ':', out_records->-1->>'id');\n RAISE NOTICE 'HAS NEXT | %', next;\n END IF;\n\n IF has_prev THEN\n prev := concat(out_records->0->>'collection', ':', out_records->0->>'id');\n RAISE NOTICE 'HAS PREV | %', prev;\n END IF;\n\n RAISE NOTICE 'Time to get prev/next %', age_ms(timer);\n timer := clock_timestamp();\n\n IF context(_search->'conf') != 'off' THEN\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', init_limit,\n 'matched', total_count,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\n ELSE\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', init_limit,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\n END IF;\n\n collection := jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb),\n 'next', next,\n 'prev', prev,\n 'context', context\n );\n\n IF get_setting_bool('timing', _search->'conf') THEN\n collection = collection || jsonb_build_object('timing', age_ms(init_ts));\n END IF;\n\n RAISE NOTICE 'Time to build final json %', age_ms(timer);\n timer := clock_timestamp();\n\n RAISE NOTICE 'Total Time: %', age_ms(current_timestamp);\n RAISE NOTICE 'RETURNING % records. NEXT: %. PREV: %', collection->'context'->>'returned', collection->>'next', collection->>'prev';\n RETURN collection;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.search_query(_search jsonb DEFAULT '{}'::jsonb, updatestats boolean DEFAULT false, _metadata jsonb DEFAULT '{}'::jsonb)\n RETURNS pgstac.searches\n LANGUAGE plpgsql\n SECURITY DEFINER\nAS $function$\nDECLARE\n search searches%ROWTYPE;\n pexplain jsonb;\n t timestamptz;\n i interval;\n _hash text := search_hash(_search, _metadata);\n doupdate boolean := FALSE;\n insertfound boolean := FALSE;\nBEGIN\n SELECT * INTO search FROM searches\n WHERE hash=_hash;\n\n search.hash := _hash;\n\n -- Calculate the where clause if not already calculated\n IF search._where IS NULL THEN\n search._where := stac_search_to_where(_search);\n ELSE\n doupdate := TRUE;\n END IF;\n\n -- Calculate the order by clause if not already calculated\n IF search.orderby IS NULL THEN\n search.orderby := sort_sqlorderby(_search);\n ELSE\n doupdate := TRUE;\n END IF;\n\n PERFORM where_stats(search._where, updatestats, _search->'conf');\n\n IF NOT doupdate THEN\n INSERT INTO searches (search, _where, orderby, lastused, usecount, metadata)\n VALUES (_search, search._where, search.orderby, clock_timestamp(), 1, _metadata)\n ON CONFLICT (hash) DO NOTHING;\n IF FOUND THEN\n RETURN search;\n END IF;\n END IF;\n\n UPDATE searches\n SET\n lastused=clock_timestamp(),\n usecount=usecount+1\n WHERE hash=(\n SELECT hash FROM searches\n WHERE hash=_hash\n FOR UPDATE SKIP LOCKED\n );\n IF NOT FOUND THEN\n RAISE NOTICE 'Did not update stats for % due to lock. (This is generally OK)', _search;\n END IF;\n\n RETURN search;\n\nEND;\n$function$\n;\n\n\n\n-- END migra calculated SQL\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('id', '{\"title\": \"Item ID\",\"description\": \"Item identifier\",\"$ref\": \"https://schemas.stacspec.org/v1.0.0/item-spec/json-schema/item.json#/definitions/core/allOf/2/properties/id\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('geometry', '{\"title\": \"Item Geometry\",\"description\": \"Item Geometry\",\"$ref\": \"https://geojson.org/schema/Feature.json\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('datetime','{\"description\": \"Datetime\",\"type\": \"string\",\"title\": \"Acquired\",\"format\": \"date-time\",\"pattern\": \"(\\\\+00:00|Z)$\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('eo:cloud_cover','{\"$ref\": \"https://stac-extensions.github.io/eo/v1.0.0/schema.json#/definitions/fieldsproperties/eo:cloud_cover\"}','to_int','BTREE');\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDELETE FROM queryables a USING queryables b\n WHERE a.name = b.name AND a.collection_ids IS NOT DISTINCT FROM b.collection_ids AND a.id > b.id;\n\n\nINSERT INTO pgstac_settings (name, value) VALUES\n ('context', 'off'),\n ('context_estimated_count', '100000'),\n ('context_estimated_cost', '100000'),\n ('context_stats_ttl', '1 day'),\n ('default_filter_lang', 'cql2-json'),\n ('additional_properties', 'true'),\n ('use_queue', 'false'),\n ('queue_timeout', '10 minutes'),\n ('update_collection_extent', 'false'),\n ('format_cache', 'false')\nON CONFLICT DO NOTHING\n;\n\nALTER FUNCTION to_text COST 5000;\nALTER FUNCTION to_float COST 5000;\nALTER FUNCTION to_int COST 5000;\nALTER FUNCTION to_tstz COST 5000;\nALTER FUNCTION to_text_array COST 5000;\n\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nGRANT ALL ON SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac to pgstac_admin;\n\n-- pgstac_read role limited to using function apis\nGRANT EXECUTE ON FUNCTION search TO pgstac_read;\nGRANT EXECUTE ON FUNCTION search_query TO pgstac_read;\nGRANT EXECUTE ON FUNCTION item_by_id TO pgstac_read;\nGRANT EXECUTE ON FUNCTION get_item TO pgstac_read;\nGRANT SELECT ON ALL TABLES IN SCHEMA pgstac TO pgstac_read;\n\n\nGRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_ingest;\nGRANT USAGE ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_ingest;\n\nSELECT update_partition_stats_q(partition) FROM partitions_view;\nSELECT set_version('0.7.7');\n" }, { "path": "src/pgstac/migrations/pgstac.0.7.6.sql", "content": "RESET ROLE;\nDO $$\nDECLARE\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='postgis') THEN\n CREATE EXTENSION IF NOT EXISTS postgis;\n END IF;\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='btree_gist') THEN\n CREATE EXTENSION IF NOT EXISTS btree_gist;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_admin;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_read;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_ingest;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n\nGRANT pgstac_admin TO current_user;\n\n-- Function to make sure pgstac_admin is the owner of items\nCREATE OR REPLACE FUNCTION pgstac_admin_owns() RETURNS VOID AS $$\nDECLARE\n f RECORD;\nBEGIN\n FOR f IN (\n SELECT\n concat(\n oid::regproc::text,\n '(',\n coalesce(pg_get_function_identity_arguments(oid),''),\n ')'\n ) AS name,\n CASE prokind WHEN 'f' THEN 'FUNCTION' WHEN 'p' THEN 'PROCEDURE' WHEN 'a' THEN 'AGGREGATE' END as typ\n FROM pg_proc\n WHERE\n pronamespace=to_regnamespace('pgstac')\n AND proowner != to_regrole('pgstac_admin')\n AND proname NOT LIKE 'pg_stat%'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n FOR f IN (\n SELECT\n oid::regclass::text as name,\n CASE relkind\n WHEN 'i' THEN 'INDEX'\n WHEN 'I' THEN 'INDEX'\n WHEN 'p' THEN 'TABLE'\n WHEN 'r' THEN 'TABLE'\n WHEN 'v' THEN 'VIEW'\n WHEN 'S' THEN 'SEQUENCE'\n ELSE NULL\n END as typ\n FROM pg_class\n WHERE relnamespace=to_regnamespace('pgstac') and relowner != to_regrole('pgstac_admin') AND relkind IN ('r','p','v','S') AND relname NOT LIKE 'pg_stat'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\nSELECT pgstac_admin_owns();\n\nCREATE SCHEMA IF NOT EXISTS pgstac AUTHORIZATION pgstac_admin;\n\nGRANT ALL ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_admin;\n\nALTER ROLE pgstac_admin SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_read SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_ingest SET SEARCH_PATH TO pgstac, public;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\n\nGRANT pgstac_read TO pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\n\nSET ROLE pgstac_admin;\n\nSET SEARCH_PATH TO pgstac, public;\n\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS analyze_items;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS validate_constraints;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\nCREATE OR REPLACE FUNCTION to_int(jsonb) RETURNS int AS $$\n SELECT floor(($1->>0)::float)::int;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000;\n\nCREATE OR REPLACE FUNCTION to_float(jsonb) RETURNS float AS $$\n SELECT ($1->>0)::float;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000;\n\nCREATE OR REPLACE FUNCTION to_tstz(jsonb) RETURNS timestamptz AS $$\n SELECT ($1->>0)::timestamptz;\n$$ LANGUAGE SQL IMMUTABLE STRICT SET TIME ZONE 'UTC' COST 5000;\n\n\nCREATE OR REPLACE FUNCTION to_text(jsonb) RETURNS text AS $$\n SELECT CASE WHEN jsonb_typeof($1) IN ('array','object') THEN $1::text ELSE $1->>0 END;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000;\n\nCREATE OR REPLACE FUNCTION to_text_array(jsonb) RETURNS text[] AS $$\n SELECT\n CASE jsonb_typeof($1)\n WHEN 'array' THEN ARRAY(SELECT jsonb_array_elements_text($1))\n ELSE ARRAY[$1->>0]\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000;\n\nCREATE OR REPLACE FUNCTION bbox_geom(_bbox jsonb) RETURNS geometry AS $$\nSELECT CASE jsonb_array_length(_bbox)\n WHEN 4 THEN\n ST_SetSRID(ST_MakeEnvelope(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float,\n (_bbox->>3)::float\n ),4326)\n WHEN 6 THEN\n ST_SetSRID(ST_3DMakeBox(\n ST_MakePoint(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float\n ),\n ST_MakePoint(\n (_bbox->>3)::float,\n (_bbox->>4)::float,\n (_bbox->>5)::float\n )\n ),4326)\n ELSE null END;\n;\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION geom_bbox(_geom geometry) RETURNS jsonb AS $$\n SELECT jsonb_build_array(\n st_xmin(_geom),\n st_ymin(_geom),\n st_xmax(_geom),\n st_ymax(_geom)\n );\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION flip_jsonb_array(j jsonb) RETURNS jsonb AS $$\n SELECT jsonb_agg(value) FROM (SELECT value FROM jsonb_array_elements(j) WITH ORDINALITY ORDER BY ordinality DESC) as t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION explode_dotpaths(j jsonb) RETURNS SETOF text[] AS $$\n SELECT string_to_array(p, '.') as e FROM jsonb_array_elements_text(j) p;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION explode_dotpaths_recurse(IN j jsonb) RETURNS SETOF text[] AS $$\n WITH RECURSIVE t AS (\n SELECT e FROM explode_dotpaths(j) e\n UNION ALL\n SELECT e[1:cardinality(e)-1]\n FROM t\n WHERE cardinality(e)>1\n ) SELECT e FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION jsonb_set_nested(j jsonb, path text[], val jsonb) RETURNS jsonb AS $$\nDECLARE\nBEGIN\n IF cardinality(path) > 1 THEN\n FOR i IN 1..(cardinality(path)-1) LOOP\n IF j #> path[:i] IS NULL THEN\n j := jsonb_set_lax(j, path[:i], '{}', TRUE);\n END IF;\n END LOOP;\n END IF;\n RETURN jsonb_set_lax(j, path, val, true);\n\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\n\n\nCREATE OR REPLACE FUNCTION jsonb_include(j jsonb, f jsonb) RETURNS jsonb AS $$\nDECLARE\n includes jsonb := f-> 'include';\n outj jsonb := '{}'::jsonb;\n path text[];\nBEGIN\n IF\n includes IS NULL\n OR jsonb_array_length(includes) = 0\n THEN\n RETURN j;\n ELSE\n includes := includes || '[\"id\",\"collection\"]'::jsonb;\n FOR path IN SELECT explode_dotpaths(includes) LOOP\n outj := jsonb_set_nested(outj, path, j #> path);\n END LOOP;\n END IF;\n RETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_exclude(j jsonb, f jsonb) RETURNS jsonb AS $$\nDECLARE\n excludes jsonb := f-> 'exclude';\n outj jsonb := j;\n path text[];\nBEGIN\n IF\n excludes IS NULL\n OR jsonb_array_length(excludes) = 0\n THEN\n RETURN j;\n ELSE\n FOR path IN SELECT explode_dotpaths(excludes) LOOP\n outj := outj #- path;\n END LOOP;\n END IF;\n RETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_fields(j jsonb, f jsonb DEFAULT '{\"fields\":[]}') RETURNS jsonb AS $$\n SELECT jsonb_exclude(jsonb_include(j, f), f);\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION merge_jsonb(_a jsonb, _b jsonb) RETURNS jsonb AS $$\n SELECT\n CASE\n WHEN _a = '\"𒍟※\"'::jsonb THEN NULL\n WHEN _a IS NULL OR jsonb_typeof(_a) = 'null' THEN _b\n WHEN jsonb_typeof(_a) = 'object' AND jsonb_typeof(_b) = 'object' THEN\n (\n SELECT\n jsonb_strip_nulls(\n jsonb_object_agg(\n key,\n merge_jsonb(a.value, b.value)\n )\n )\n FROM\n jsonb_each(coalesce(_a,'{}'::jsonb)) as a\n FULL JOIN\n jsonb_each(coalesce(_b,'{}'::jsonb)) as b\n USING (key)\n )\n WHEN\n jsonb_typeof(_a) = 'array'\n AND jsonb_typeof(_b) = 'array'\n AND jsonb_array_length(_a) = jsonb_array_length(_b)\n THEN\n (\n SELECT jsonb_agg(m) FROM\n ( SELECT\n merge_jsonb(\n jsonb_array_elements(_a),\n jsonb_array_elements(_b)\n ) as m\n ) as l\n )\n ELSE _a\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION strip_jsonb(_a jsonb, _b jsonb) RETURNS jsonb AS $$\n SELECT\n CASE\n\n WHEN (_a IS NULL OR jsonb_typeof(_a) = 'null') AND _b IS NOT NULL AND jsonb_typeof(_b) != 'null' THEN '\"𒍟※\"'::jsonb\n WHEN _b IS NULL OR jsonb_typeof(_a) = 'null' THEN _a\n WHEN _a = _b AND jsonb_typeof(_a) = 'object' THEN '{}'::jsonb\n WHEN _a = _b THEN NULL\n WHEN jsonb_typeof(_a) = 'object' AND jsonb_typeof(_b) = 'object' THEN\n (\n SELECT\n jsonb_strip_nulls(\n jsonb_object_agg(\n key,\n strip_jsonb(a.value, b.value)\n )\n )\n FROM\n jsonb_each(_a) as a\n FULL JOIN\n jsonb_each(_b) as b\n USING (key)\n )\n WHEN\n jsonb_typeof(_a) = 'array'\n AND jsonb_typeof(_b) = 'array'\n AND jsonb_array_length(_a) = jsonb_array_length(_b)\n THEN\n (\n SELECT jsonb_agg(m) FROM\n ( SELECT\n strip_jsonb(\n jsonb_array_elements(_a),\n jsonb_array_elements(_b)\n ) as m\n ) as l\n )\n ELSE _a\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION nullif_jsonbnullempty(j jsonb) RETURNS jsonb AS $$\n SELECT nullif(nullif(nullif(j,'null'::jsonb),'{}'::jsonb),'[]'::jsonb);\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION jsonb_array_unique(j jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(jsonb_agg(DISTINCT a)) v FROM jsonb_array_elements(j) a;\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_concat_ignorenull(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT coalesce(a,'[]'::jsonb) || coalesce(b,'[]'::jsonb);\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_least(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(least(nullif_jsonbnullempty(a), nullif_jsonbnullempty(b)));\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_greatest(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(greatest(a, b));\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION first_notnull_sfunc(anyelement, anyelement) RETURNS anyelement AS $$\n SELECT COALESCE($1,$2);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE AGGREGATE first_notnull(anyelement)(\n SFUNC = first_notnull_sfunc,\n STYPE = anyelement\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_array_unique_merge(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_concat_ignorenull,\n FINALFUNC = jsonb_array_unique\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_min(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_least\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_max(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_greatest\n);\n\nCREATE TABLE IF NOT EXISTS migrations (\n version text PRIMARY KEY,\n datetime timestamptz DEFAULT clock_timestamp() NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION get_version() RETURNS text AS $$\n SELECT version FROM pgstac.migrations ORDER BY datetime DESC, version DESC LIMIT 1;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION set_version(text) RETURNS text AS $$\n INSERT INTO pgstac.migrations (version) VALUES ($1)\n ON CONFLICT DO NOTHING\n RETURNING version;\n$$ LANGUAGE SQL;\n\n\nCREATE TABLE IF NOT EXISTS pgstac_settings (\n name text PRIMARY KEY,\n value text NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION table_empty(text) RETURNS boolean AS $$\nDECLARE\n retval boolean;\nBEGIN\n EXECUTE format($q$\n SELECT NOT EXISTS (SELECT 1 FROM %I LIMIT 1)\n $q$,\n $1\n ) INTO retval;\n RETURN retval;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION get_setting(IN _setting text, IN conf jsonb DEFAULT NULL) RETURNS text AS $$\nSELECT COALESCE(\n nullif(conf->>_setting, ''),\n nullif(current_setting(concat('pgstac.',_setting), TRUE),''),\n nullif((SELECT value FROM pgstac.pgstac_settings WHERE name=_setting),'')\n);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_setting_bool(IN _setting text, IN conf jsonb DEFAULT NULL) RETURNS boolean AS $$\nSELECT COALESCE(\n conf->>_setting,\n current_setting(concat('pgstac.',_setting), TRUE),\n (SELECT value FROM pgstac.pgstac_settings WHERE name=_setting),\n 'FALSE'\n)::boolean;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context(conf jsonb DEFAULT NULL) RETURNS text AS $$\n SELECT pgstac.get_setting('context', conf);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context_estimated_count(conf jsonb DEFAULT NULL) RETURNS int AS $$\n SELECT pgstac.get_setting('context_estimated_count', conf)::int;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_estimated_cost();\nCREATE OR REPLACE FUNCTION context_estimated_cost(conf jsonb DEFAULT NULL) RETURNS float AS $$\n SELECT pgstac.get_setting('context_estimated_cost', conf)::float;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_stats_ttl();\nCREATE OR REPLACE FUNCTION context_stats_ttl(conf jsonb DEFAULT NULL) RETURNS interval AS $$\n SELECT pgstac.get_setting('context_stats_ttl', conf)::interval;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION t2s(text) RETURNS text AS $$\n SELECT extract(epoch FROM $1::interval)::text || ' s';\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE STRICT;\n\nCREATE OR REPLACE FUNCTION age_ms(a timestamptz, b timestamptz DEFAULT clock_timestamp()) RETURNS float AS $$\n SELECT abs(extract(epoch from age(a,b)) * 1000);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION queue_timeout() RETURNS interval AS $$\n SELECT t2s(coalesce(\n get_setting('queue_timeout'),\n '1h'\n ))::interval;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION notice(VARIADIC text[]) RETURNS boolean AS $$\nDECLARE\ndebug boolean := current_setting('pgstac.debug', true);\nBEGIN\n IF debug THEN\n RAISE NOTICE 'NOTICE FROM FUNC: % >>>>> %', concat_ws(' | ', $1), clock_timestamp();\n RETURN TRUE;\n END IF;\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION empty_arr(ANYARRAY) RETURNS BOOLEAN AS $$\nSELECT CASE\n WHEN $1 IS NULL THEN TRUE\n WHEN cardinality($1)<1 THEN TRUE\nELSE FALSE\nEND;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_intersection(_a ANYARRAY, _b ANYARRAY) RETURNS ANYARRAY AS $$\n SELECT ARRAY ( SELECT unnest(_a) INTERSECT SELECT UNNEST(_b) );\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION array_map_ident(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_ident(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_map_literal(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_literal(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION array_reverse(anyarray) RETURNS anyarray AS $$\nSELECT ARRAY(\n SELECT $1[i]\n FROM generate_subscripts($1,1) AS s(i)\n ORDER BY i DESC\n);\n$$ LANGUAGE SQL STRICT IMMUTABLE;\n\nDROP TABLE IF EXISTS query_queue;\nCREATE TABLE query_queue (\n query text PRIMARY KEY,\n added timestamptz DEFAULT now()\n);\n\nDROP TABLE IF EXISTS query_queue_history;\nCREATE TABLE query_queue_history(\n query text,\n added timestamptz NOT NULL,\n finished timestamptz NOT NULL DEFAULT now(),\n error text\n);\n\nCREATE OR REPLACE PROCEDURE run_queued_queries() AS $$\nDECLARE\n qitem query_queue%ROWTYPE;\n timeout_ts timestamptz;\n error text;\n cnt int := 0;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n DELETE FROM query_queue WHERE query = (SELECT query FROM query_queue ORDER BY added DESC LIMIT 1 FOR UPDATE SKIP LOCKED) RETURNING * INTO qitem;\n IF NOT FOUND THEN\n EXIT;\n END IF;\n cnt := cnt + 1;\n BEGIN\n RAISE NOTICE 'RUNNING QUERY: %', qitem.query;\n EXECUTE qitem.query;\n EXCEPTION WHEN others THEN\n error := format('%s | %s', SQLERRM, SQLSTATE);\n END;\n INSERT INTO query_queue_history (query, added, finished, error)\n VALUES (qitem.query, qitem.added, clock_timestamp(), error);\n COMMIT;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION run_queued_queries_intransaction() RETURNS int AS $$\nDECLARE\n qitem query_queue%ROWTYPE;\n timeout_ts timestamptz;\n error text;\n cnt int := 0;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n DELETE FROM query_queue WHERE query = (SELECT query FROM query_queue ORDER BY added DESC LIMIT 1 FOR UPDATE SKIP LOCKED) RETURNING * INTO qitem;\n IF NOT FOUND THEN\n RETURN cnt;\n END IF;\n cnt := cnt + 1;\n BEGIN\n qitem.query := regexp_replace(qitem.query, 'CONCURRENTLY', '');\n RAISE NOTICE 'RUNNING QUERY: %', qitem.query;\n\n EXECUTE qitem.query;\n EXCEPTION WHEN others THEN\n error := format('%s | %s', SQLERRM, SQLSTATE);\n RAISE WARNING '%', error;\n END;\n INSERT INTO query_queue_history (query, added, finished, error)\n VALUES (qitem.query, qitem.added, clock_timestamp(), error);\n END LOOP;\n RETURN cnt;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION run_or_queue(query text) RETURNS VOID AS $$\nDECLARE\n use_queue text := COALESCE(get_setting('use_queue'), 'FALSE')::boolean;\nBEGIN\n IF get_setting_bool('debug') THEN\n RAISE NOTICE '%', query;\n END IF;\n IF use_queue THEN\n INSERT INTO query_queue (query) VALUES (query) ON CONFLICT DO NOTHING;\n ELSE\n EXECUTE query;\n END IF;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n\nDROP FUNCTION IF EXISTS check_pgstac_settings;\nCREATE OR REPLACE FUNCTION check_pgstac_settings(_sysmem text DEFAULT NULL) RETURNS VOID AS $$\nDECLARE\n settingval text;\n sysmem bigint := pg_size_bytes(_sysmem);\n effective_cache_size bigint := pg_size_bytes(current_setting('effective_cache_size', TRUE));\n shared_buffers bigint := pg_size_bytes(current_setting('shared_buffers', TRUE));\n work_mem bigint := pg_size_bytes(current_setting('work_mem', TRUE));\n max_connections int := current_setting('max_connections', TRUE);\n maintenance_work_mem bigint := pg_size_bytes(current_setting('maintenance_work_mem', TRUE));\n seq_page_cost float := current_setting('seq_page_cost', TRUE);\n random_page_cost float := current_setting('random_page_cost', TRUE);\n temp_buffers bigint := pg_size_bytes(current_setting('temp_buffers', TRUE));\n r record;\nBEGIN\n IF _sysmem IS NULL THEN\n RAISE NOTICE 'Call function with the size of your system memory `SELECT check_pgstac_settings(''4GB'')` to get pg system setting recommendations.';\n ELSE\n IF effective_cache_size < (sysmem * 0.5) THEN\n RAISE WARNING 'effective_cache_size of % is set low for a system with %. Recomended value between % and %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.5), pg_size_pretty(sysmem * 0.75);\n ELSIF effective_cache_size > (sysmem * 0.75) THEN\n RAISE WARNING 'effective_cache_size of % is set high for a system with %. Recomended value between % and %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.5), pg_size_pretty(sysmem * 0.75);\n ELSE\n RAISE NOTICE 'effective_cache_size of % is set appropriately for a system with %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem);\n END IF;\n\n IF shared_buffers < (sysmem * 0.2) THEN\n RAISE WARNING 'shared_buffers of % is set low for a system with %. Recomended value between % and %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.2), pg_size_pretty(sysmem * 0.3);\n ELSIF shared_buffers > (sysmem * 0.3) THEN\n RAISE WARNING 'shared_buffers of % is set high for a system with %. Recomended value between % and %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.2), pg_size_pretty(sysmem * 0.3);\n ELSE\n RAISE NOTICE 'shared_buffers of % is set appropriately for a system with %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem);\n END IF;\n shared_buffers = sysmem * 0.3;\n IF maintenance_work_mem < (sysmem * 0.2) THEN\n RAISE WARNING 'maintenance_work_mem of % is set low for shared_buffers of %. Recomended value between % and %', pg_size_pretty(maintenance_work_mem), pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers * 0.2), pg_size_pretty(shared_buffers * 0.3);\n ELSIF maintenance_work_mem > (shared_buffers * 0.3) THEN\n RAISE WARNING 'maintenance_work_mem of % is set high for shared_buffers of %. Recomended value between % and %', pg_size_pretty(maintenance_work_mem), pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers * 0.2), pg_size_pretty(shared_buffers * 0.3);\n ELSE\n RAISE NOTICE 'maintenance_work_mem of % is set appropriately for shared_buffers of %', pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers);\n END IF;\n\n IF work_mem * max_connections > shared_buffers THEN\n RAISE WARNING 'work_mem setting of % is set high for % max_connections please reduce work_mem to % or decrease max_connections to %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers/max_connections), floor(shared_buffers/work_mem);\n ELSIF work_mem * max_connections < (shared_buffers * 0.75) THEN\n RAISE WARNING 'work_mem setting of % is set low for % max_connections you may consider raising work_mem to % or increasing max_connections to %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers/max_connections), floor(shared_buffers/work_mem);\n ELSE\n RAISE NOTICE 'work_mem setting of % and max_connections of % are adequate for shared_buffers of %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers);\n END IF;\n\n IF random_page_cost / seq_page_cost != 1.1 THEN\n RAISE WARNING 'random_page_cost (%) /seq_page_cost (%) should be set to 1.1 for SSD. Change random_page_cost to %', random_page_cost, seq_page_cost, 1.1 * seq_page_cost;\n ELSE\n RAISE NOTICE 'random_page_cost and seq_page_cost set appropriately for SSD';\n END IF;\n\n IF temp_buffers < greatest(pg_size_bytes('128MB'),(maintenance_work_mem / 2)) THEN\n RAISE WARNING 'pgstac makes heavy use of temp tables, consider raising temp_buffers from % to %', pg_size_pretty(temp_buffers), greatest('128MB', pg_size_pretty((shared_buffers / 16)));\n END IF;\n END IF;\n\n RAISE NOTICE 'VALUES FOR PGSTAC VARIABLES';\n RAISE NOTICE 'These can be set either as GUC system variables or by setting in the pgstac_settings table.';\n\n FOR r IN SELECT name, get_setting(name) as setting, CASE WHEN current_setting(concat('pgstac.',name), TRUE) IS NOT NULL THEN concat('pgstac.',name, ' GUC') WHEN value IS NOT NULL THEN 'pgstac_settings table' ELSE 'Not Set' END as loc FROM pgstac_settings LOOP\n RAISE NOTICE '% is set to % from the %', r.name, r.setting, r.loc;\n END LOOP;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pg_cron';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider intalling pg_cron which can be used to automate tasks';\n ELSE\n RAISE NOTICE 'pg_cron % is installed', settingval;\n END IF;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pgstattuple';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider installing the pgstattuple extension which can be used to help maintain tables and indexes.';\n ELSE\n RAISE NOTICE 'pgstattuple % is installed', settingval;\n END IF;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pg_stat_statements';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider installing the pg_stat_statements extension which is very helpful for tracking the types of queries on the system';\n ELSE\n RAISE NOTICE 'pg_stat_statements % is installed', settingval;\n IF current_setting('pg_stat_statements.track_statements', TRUE) IS DISTINCT FROM 'all' THEN\n RAISE WARNING 'SET pg_stat_statements.track_statements TO ''all''; --In order to track statements within functions.';\n END IF;\n END IF;\n\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac, public SET CLIENT_MIN_MESSAGES TO NOTICE;\n/* looks for a geometry in a stac item first from geometry and falling back to bbox */\nCREATE OR REPLACE FUNCTION stac_geom(value jsonb) RETURNS geometry AS $$\nSELECT\n CASE\n WHEN value ? 'intersects' THEN\n ST_GeomFromGeoJSON(value->>'intersects')\n WHEN value ? 'geometry' THEN\n ST_GeomFromGeoJSON(value->>'geometry')\n WHEN value ? 'bbox' THEN\n pgstac.bbox_geom(value->'bbox')\n ELSE NULL\n END as geometry\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION stac_daterange(\n value jsonb\n) RETURNS tstzrange AS $$\nDECLARE\n props jsonb := value;\n dt timestamptz;\n edt timestamptz;\nBEGIN\n IF props ? 'properties' THEN\n props := props->'properties';\n END IF;\n IF\n props ? 'start_datetime'\n AND props->>'start_datetime' IS NOT NULL\n AND props ? 'end_datetime'\n AND props->>'end_datetime' IS NOT NULL\n THEN\n dt := props->>'start_datetime';\n edt := props->>'end_datetime';\n IF dt > edt THEN\n RAISE EXCEPTION 'start_datetime must be < end_datetime';\n END IF;\n ELSE\n dt := props->>'datetime';\n edt := props->>'datetime';\n END IF;\n IF dt is NULL OR edt IS NULL THEN\n RAISE NOTICE 'DT: %, EDT: %', dt, edt;\n RAISE EXCEPTION 'Either datetime (%) or both start_datetime (%) and end_datetime (%) must be set.', props->>'datetime',props->>'start_datetime',props->>'end_datetime';\n END IF;\n RETURN tstzrange(dt, edt, '[]');\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_datetime(value jsonb) RETURNS timestamptz AS $$\n SELECT lower(stac_daterange(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_end_datetime(value jsonb) RETURNS timestamptz AS $$\n SELECT upper(stac_daterange(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE TABLE IF NOT EXISTS stac_extensions(\n url text PRIMARY KEY,\n content jsonb\n);\nCREATE OR REPLACE FUNCTION queryable_signature(n text, c text[]) RETURNS text AS $$\n SELECT concat(n, c);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TABLE queryables (\n id bigint GENERATED ALWAYS AS identity PRIMARY KEY,\n name text NOT NULL,\n collection_ids text[], -- used to determine what partitions to create indexes on\n definition jsonb,\n property_path text,\n property_wrapper text,\n property_index_type text\n);\nCREATE INDEX queryables_name_idx ON queryables (name);\nCREATE INDEX queryables_collection_idx ON queryables USING GIN (collection_ids);\nCREATE INDEX queryables_property_wrapper_idx ON queryables (property_wrapper);\n\nCREATE OR REPLACE FUNCTION pgstac.queryables_constraint_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n allcollections text[];\nBEGIN\n RAISE NOTICE 'Making sure that name/collection is unique for queryables %', NEW;\n IF NEW.collection_ids IS NOT NULL THEN\n IF EXISTS (\n SELECT 1\n FROM unnest(NEW.collection_ids) c\n LEFT JOIN\n collections\n ON (collections.id = c)\n WHERE collections.id IS NULL\n ) THEN\n RAISE foreign_key_violation USING MESSAGE = format(\n 'One or more collections in %s do not exist.', NEW.collection_ids\n );\n RETURN NULL;\n END IF;\n END IF;\n IF TG_OP = 'INSERT' THEN\n IF EXISTS (\n SELECT 1 FROM queryables q\n WHERE\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n )\n ) THEN\n RAISE unique_violation USING MESSAGE = format(\n 'There is already a queryable for %s for a collection in %s: %s',\n NEW.name,\n NEW.collection_ids,\n\t\t\t\t(SELECT json_agg(row_to_json(q)) FROM queryables q WHERE\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n ))\n );\n RETURN NULL;\n END IF;\n END IF;\n IF TG_OP = 'UPDATE' THEN\n IF EXISTS (\n SELECT 1 FROM queryables q\n WHERE\n q.id != NEW.id\n AND\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n )\n ) THEN\n RAISE unique_violation\n USING MESSAGE = format(\n 'There is already a queryable for %s for a collection in %s',\n NEW.name,\n NEW.collection_ids\n );\n RETURN NULL;\n END IF;\n END IF;\n\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER queryables_constraint_insert_trigger\nBEFORE INSERT ON queryables\nFOR EACH ROW EXECUTE PROCEDURE queryables_constraint_triggerfunc();\n\nCREATE TRIGGER queryables_constraint_update_trigger\nBEFORE UPDATE ON queryables\nFOR EACH ROW\nWHEN (NEW.name = OLD.name AND NEW.collection_ids IS DISTINCT FROM OLD.collection_ids)\nEXECUTE PROCEDURE queryables_constraint_triggerfunc();\n\n\nCREATE OR REPLACE FUNCTION array_to_path(arr text[]) RETURNS text AS $$\n SELECT string_agg(\n quote_literal(v),\n '->'\n ) FROM unnest(arr) v;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\n\n\nCREATE OR REPLACE FUNCTION queryable(\n IN dotpath text,\n OUT path text,\n OUT expression text,\n OUT wrapper text,\n OUT nulled_wrapper text\n) AS $$\nDECLARE\n q RECORD;\n path_elements text[];\nBEGIN\n IF dotpath IN ('id', 'geometry', 'datetime', 'end_datetime', 'collection') THEN\n path := dotpath;\n expression := dotpath;\n wrapper := NULL;\n RETURN;\n END IF;\n SELECT * INTO q FROM queryables\n WHERE\n name=dotpath\n OR name = 'properties.' || dotpath\n OR name = replace(dotpath, 'properties.', '')\n ;\n IF q.property_wrapper IS NULL THEN\n IF q.definition->>'type' = 'number' THEN\n wrapper := 'to_float';\n nulled_wrapper := wrapper;\n ELSIF q.definition->>'format' = 'date-time' THEN\n wrapper := 'to_tstz';\n nulled_wrapper := wrapper;\n ELSE\n nulled_wrapper := NULL;\n wrapper := 'to_text';\n END IF;\n ELSE\n wrapper := q.property_wrapper;\n nulled_wrapper := wrapper;\n END IF;\n IF q.property_path IS NOT NULL THEN\n path := q.property_path;\n ELSE\n path_elements := string_to_array(dotpath, '.');\n IF path_elements[1] IN ('links', 'assets', 'stac_version', 'stac_extensions') THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSIF path_elements[1] = 'properties' THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSE\n path := format($F$content->'properties'->%s$F$, array_to_path(path_elements));\n END IF;\n END IF;\n expression := format('%I(%s)', wrapper, path);\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\nCREATE OR REPLACE FUNCTION unnest_collection(collection_ids text[] DEFAULT NULL) RETURNS SETOF text AS $$\n DECLARE\n BEGIN\n IF collection_ids IS NULL THEN\n RETURN QUERY SELECT id FROM collections;\n END IF;\n RETURN QUERY SELECT unnest(collection_ids);\n END;\n$$ LANGUAGE PLPGSQL STABLE;\n\nCREATE OR REPLACE FUNCTION normalize_indexdef(def text) RETURNS text AS $$\nDECLARE\nBEGIN\n def := btrim(def, ' \\n\\t');\n\tdef := regexp_replace(def, '^CREATE (UNIQUE )?INDEX ([^ ]* )?ON (ONLY )?([^ ]* )?', '', 'i');\n RETURN def;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE STRICT PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION indexdef(q queryables) RETURNS text AS $$\n DECLARE\n out text;\n BEGIN\n IF q.name = 'id' THEN\n out := 'CREATE UNIQUE INDEX ON %I USING btree (id)';\n ELSIF q.name = 'datetime' THEN\n out := 'CREATE INDEX ON %I USING btree (datetime DESC, end_datetime)';\n ELSIF q.name = 'geometry' THEN\n out := 'CREATE INDEX ON %I USING gist (geometry)';\n ELSE\n out := format($q$CREATE INDEX ON %%I USING %s (%s(((content -> 'properties'::text) -> %L::text)))$q$,\n lower(COALESCE(q.property_index_type, 'BTREE')),\n lower(COALESCE(q.property_wrapper, 'to_text')),\n q.name\n );\n END IF;\n RETURN btrim(out, ' \\n\\t');\n END;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\n\nDROP VIEW IF EXISTS pgstac_indexes;\nCREATE VIEW pgstac_indexes AS\nSELECT\n i.schemaname,\n i.tablename,\n i.indexname,\n regexp_replace(btrim(replace(replace(indexdef, i.indexname, ''),'pgstac.',''),' \\t\\n'), '[ ]+', ' ', 'g') as idx,\n COALESCE(\n (regexp_match(indexdef, '\\(([a-zA-Z]+)\\)'))[1],\n (regexp_match(indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_-]+)''::text'))[1],\n CASE WHEN indexdef ~* '\\(datetime desc, end_datetime\\)' THEN 'datetime' ELSE NULL END\n ) AS field,\n pg_table_size(i.indexname::text) as index_size,\n pg_size_pretty(pg_table_size(i.indexname::text)) as index_size_pretty\nFROM\n pg_indexes i\nWHERE i.schemaname='pgstac' and i.tablename ~ '_items_' AND indexdef !~* ' only ';\n\nDROP VIEW IF EXISTS pgstac_index_stats;\nCREATE VIEW pgstac_indexes_stats AS\nSELECT\n i.schemaname,\n i.tablename,\n i.indexname,\n indexdef,\n COALESCE(\n (regexp_match(indexdef, '\\(([a-zA-Z]+)\\)'))[1],\n (regexp_match(indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_]+)''::text'))[1],\n CASE WHEN indexdef ~* '\\(datetime desc, end_datetime\\)' THEN 'datetime_end_datetime' ELSE NULL END\n ) AS field,\n pg_table_size(i.indexname::text) as index_size,\n pg_size_pretty(pg_table_size(i.indexname::text)) as index_size_pretty,\n n_distinct,\n most_common_vals::text::text[],\n most_common_freqs::text::text[],\n histogram_bounds::text::text[],\n correlation\nFROM\n pg_indexes i\n LEFT JOIN pg_stats s ON (s.tablename = i.indexname)\nWHERE i.schemaname='pgstac' and i.tablename ~ '_items_';\n\nset check_function_bodies to off;\nCREATE OR REPLACE FUNCTION maintain_partition_queries(\n part text DEFAULT 'items',\n dropindexes boolean DEFAULT FALSE,\n rebuildindexes boolean DEFAULT FALSE,\n idxconcurrently boolean DEFAULT FALSE\n) RETURNS SETOF text AS $$\nDECLARE\n rec record;\nBEGIN\n FOR rec IN (\n WITH p AS (\n SELECT\n relid::text as partition,\n replace(replace(\n CASE\n WHEN level = 1 THEN pg_get_expr(c.relpartbound, c.oid)\n ELSE pg_get_expr(parent.relpartbound, parent.oid)\n END,\n 'FOR VALUES IN (''',''), ''')',\n ''\n ) AS collection\n FROM pg_partition_tree('items')\n JOIN pg_class c ON (relid::regclass = c.oid)\n JOIN pg_class parent ON (parentrelid::regclass = parent.oid AND isleaf)\n ), i AS (\n SELECT\n partition,\n indexname,\n regexp_replace(btrim(replace(replace(indexdef, indexname, ''),'pgstac.',''),' \\t\\n'), '[ ]+', ' ', 'g') as iidx,\n COALESCE(\n (regexp_match(indexdef, '\\(([a-zA-Z]+)\\)'))[1],\n (regexp_match(indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_-]+)''::text'))[1],\n CASE WHEN indexdef ~* '\\(datetime desc, end_datetime\\)' THEN 'datetime' ELSE NULL END\n ) AS field\n FROM\n pg_indexes\n JOIN p ON (tablename=partition)\n ), q AS (\n SELECT\n name AS field,\n collection,\n partition,\n format(indexdef(queryables), partition) as qidx\n FROM queryables, unnest_collection(queryables.collection_ids) collection\n JOIN p USING (collection)\n WHERE property_index_type IS NOT NULL OR name IN ('datetime','geometry','id')\n )\n SELECT * FROM i FULL JOIN q USING (field, partition)\n WHERE lower(iidx) IS DISTINCT FROM lower(qidx)\n ) LOOP\n IF rec.iidx IS NULL THEN\n IF idxconcurrently THEN\n RETURN NEXT replace(rec.qidx, 'INDEX', 'INDEX CONCURRENTLY');\n ELSE\n RETURN NEXT rec.qidx;\n END IF;\n ELSIF rec.qidx IS NULL AND dropindexes THEN\n RETURN NEXT format('DROP INDEX IF EXISTS %I;', rec.indexname);\n ELSIF lower(rec.qidx) != lower(rec.iidx) THEN\n IF dropindexes THEN\n RETURN NEXT format('DROP INDEX IF EXISTS %I; %s;', rec.indexname, rec.qidx);\n ELSE\n IF idxconcurrently THEN\n RETURN NEXT replace(rec.qidx, 'INDEX', 'INDEX CONCURRENTLY');\n ELSE\n RETURN NEXT rec.qidx;\n END IF;\n END IF;\n ELSIF rebuildindexes and rec.indexname IS NOT NULL THEN\n IF idxconcurrently THEN\n RETURN NEXT format('REINDEX INDEX CONCURRENTLY %I;', rec.indexname);\n ELSE\n RETURN NEXT format('REINDEX INDEX %I;', rec.indexname);\n END IF;\n END IF;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION maintain_partitions(\n part text DEFAULT 'items',\n dropindexes boolean DEFAULT FALSE,\n rebuildindexes boolean DEFAULT FALSE\n) RETURNS VOID AS $$\n WITH t AS (\n SELECT run_or_queue(q) FROM maintain_partition_queries(part, dropindexes, rebuildindexes) q\n ) SELECT count(*) FROM t;\n$$ LANGUAGE SQL;\n\n\nCREATE OR REPLACE FUNCTION queryables_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\nBEGIN\n PERFORM maintain_partitions();\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER queryables_trigger AFTER INSERT OR UPDATE ON queryables\nFOR EACH STATEMENT EXECUTE PROCEDURE queryables_trigger_func();\n\n\nCREATE OR REPLACE FUNCTION get_queryables(_collection_ids text[] DEFAULT NULL) RETURNS jsonb AS $$\nDECLARE\nBEGIN\n -- Build up queryables if the input contains valid collection ids or is empty\n IF EXISTS (\n SELECT 1 FROM collections\n WHERE\n _collection_ids IS NULL\n OR cardinality(_collection_ids) = 0\n OR id = ANY(_collection_ids)\n )\n THEN\n RETURN (\n WITH base AS (\n SELECT\n unnest(collection_ids) as collection_id,\n name,\n coalesce(definition, '{\"type\":\"string\"}'::jsonb) as definition\n FROM queryables\n WHERE\n _collection_ids IS NULL OR\n _collection_ids = '{}'::text[] OR\n _collection_ids && collection_ids\n UNION ALL\n SELECT null, name, coalesce(definition, '{\"type\":\"string\"}'::jsonb) as definition\n FROM queryables WHERE collection_ids IS NULL OR collection_ids = '{}'::text[]\n ), g AS (\n SELECT\n name,\n first_notnull(definition) as definition,\n jsonb_array_unique_merge(definition->'enum') as enum,\n jsonb_min(definition->'minimum') as minimum,\n jsonb_min(definition->'maxiumn') as maximum\n FROM base\n GROUP BY 1\n )\n SELECT\n jsonb_build_object(\n '$schema', 'http://json-schema.org/draft-07/schema#',\n '$id', '',\n 'type', 'object',\n 'title', 'STAC Queryables.',\n 'properties', jsonb_object_agg(\n name,\n definition\n ||\n jsonb_strip_nulls(jsonb_build_object(\n 'enum', enum,\n 'minimum', minimum,\n 'maximum', maximum\n ))\n )\n )\n FROM g\n );\n ELSE\n RETURN NULL;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\nCREATE OR REPLACE FUNCTION get_queryables(_collection text DEFAULT NULL) RETURNS jsonb AS $$\n SELECT\n CASE\n WHEN _collection IS NULL THEN get_queryables(NULL::text[])\n ELSE get_queryables(ARRAY[_collection])\n END\n ;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_queryables() RETURNS jsonb AS $$\n SELECT get_queryables(NULL::text[]);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION schema_qualify_refs(url text, j jsonb) returns jsonb as $$\n SELECT regexp_replace(j::text, '\"\\$ref\": \"#', concat('\"$ref\": \"', url, '#'), 'g')::jsonb;\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE VIEW stac_extension_queryables AS\nSELECT DISTINCT key as name, schema_qualify_refs(e.url, j.value) as definition FROM stac_extensions e, jsonb_each(e.content->'definitions'->'fields'->'properties') j;\n\n\nCREATE OR REPLACE FUNCTION missing_queryables(_collection text, _tablesample float DEFAULT 5, minrows float DEFAULT 10) RETURNS TABLE(collection text, name text, definition jsonb, property_wrapper text) AS $$\nDECLARE\n q text;\n _partition text;\n explain_json json;\n psize float;\n estrows float;\nBEGIN\n SELECT format('_items_%s', key) INTO _partition FROM collections WHERE id=_collection;\n\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM %I;', _partition)\n INTO explain_json;\n psize := explain_json->0->'Plan'->'Plan Rows';\n estrows := _tablesample * .01 * psize;\n IF estrows < minrows THEN\n _tablesample := least(100,greatest(_tablesample, (estrows / psize) / 100));\n RAISE NOTICE '%', (psize / estrows) / 100;\n END IF;\n RAISE NOTICE 'Using tablesample % to find missing queryables from % % that has ~% rows estrows: %', _tablesample, _collection, _partition, psize, estrows;\n\n q := format(\n $q$\n WITH q AS (\n SELECT * FROM queryables\n WHERE\n collection_ids IS NULL\n OR %L = ANY(collection_ids)\n ), t AS (\n SELECT\n content->'properties' AS properties\n FROM\n %I\n TABLESAMPLE SYSTEM(%L)\n ), p AS (\n SELECT DISTINCT ON (key)\n key,\n value,\n s.definition\n FROM t\n JOIN LATERAL jsonb_each(properties) ON TRUE\n LEFT JOIN q ON (q.name=key)\n LEFT JOIN stac_extension_queryables s ON (s.name=key)\n WHERE q.definition IS NULL\n )\n SELECT\n %L,\n key,\n COALESCE(definition, jsonb_build_object('type',jsonb_typeof(value))) as definition,\n CASE\n WHEN definition->>'type' = 'integer' THEN 'to_int'\n WHEN COALESCE(definition->>'type', jsonb_typeof(value)) = 'number' THEN 'to_float'\n WHEN COALESCE(definition->>'type', jsonb_typeof(value)) = 'array' THEN 'to_text_array'\n ELSE 'to_text'\n END\n FROM p;\n $q$,\n _collection,\n _partition,\n _tablesample,\n _collection\n );\n RETURN QUERY EXECUTE q;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION missing_queryables(_tablesample float DEFAULT 5) RETURNS TABLE(collection_ids text[], name text, definition jsonb, property_wrapper text) AS $$\n SELECT\n array_agg(collection),\n name,\n definition,\n property_wrapper\n FROM\n collections\n JOIN LATERAL\n missing_queryables(id, _tablesample) c\n ON TRUE\n GROUP BY\n 2,3,4\n ORDER BY 2,1\n ;\n$$ LANGUAGE SQL;\nCREATE OR REPLACE FUNCTION parse_dtrange(\n _indate jsonb,\n relative_base timestamptz DEFAULT date_trunc('hour', CURRENT_TIMESTAMP)\n) RETURNS tstzrange AS $$\nDECLARE\n timestrs text[];\n s timestamptz;\n e timestamptz;\nBEGIN\n timestrs :=\n CASE\n WHEN _indate ? 'timestamp' THEN\n ARRAY[_indate->>'timestamp']\n WHEN _indate ? 'interval' THEN\n to_text_array(_indate->'interval')\n WHEN jsonb_typeof(_indate) = 'array' THEN\n to_text_array(_indate)\n ELSE\n regexp_split_to_array(\n _indate->>0,\n '/'\n )\n END;\n RAISE NOTICE 'TIMESTRS %', timestrs;\n IF cardinality(timestrs) = 1 THEN\n IF timestrs[1] ILIKE 'P%' THEN\n RETURN tstzrange(relative_base - upper(timestrs[1])::interval, relative_base, '[)');\n END IF;\n s := timestrs[1]::timestamptz;\n RETURN tstzrange(s, s, '[]');\n END IF;\n\n IF cardinality(timestrs) != 2 THEN\n RAISE EXCEPTION 'Timestamp cannot have more than 2 values';\n END IF;\n\n IF timestrs[1] = '..' OR timestrs[1] = '' THEN\n s := '-infinity'::timestamptz;\n e := timestrs[2]::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] = '..' OR timestrs[2] = '' THEN\n s := timestrs[1]::timestamptz;\n e := 'infinity'::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[1] ILIKE 'P%' AND timestrs[2] NOT ILIKE 'P%' THEN\n e := timestrs[2]::timestamptz;\n s := e - upper(timestrs[1])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] ILIKE 'P%' AND timestrs[1] NOT ILIKE 'P%' THEN\n s := timestrs[1]::timestamptz;\n e := s + upper(timestrs[2])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n s := timestrs[1]::timestamptz;\n e := timestrs[2]::timestamptz;\n\n RETURN tstzrange(s,e,'[)');\n\n RETURN NULL;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT PARALLEL SAFE SET TIME ZONE 'UTC';\n\nCREATE OR REPLACE FUNCTION parse_dtrange(\n _indate text,\n relative_base timestamptz DEFAULT CURRENT_TIMESTAMP\n) RETURNS tstzrange AS $$\n SELECT parse_dtrange(to_jsonb(_indate), relative_base);\n$$ LANGUAGE SQL STABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION temporal_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\n ll text := 'datetime';\n lh text := 'end_datetime';\n rrange tstzrange;\n rl text;\n rh text;\n outq text;\nBEGIN\n rrange := parse_dtrange(args->1);\n RAISE NOTICE 'Constructing temporal query OP: %, ARGS: %, RRANGE: %', op, args, rrange;\n op := lower(op);\n rl := format('%L::timestamptz', lower(rrange));\n rh := format('%L::timestamptz', upper(rrange));\n outq := CASE op\n WHEN 't_before' THEN 'lh < rl'\n WHEN 't_after' THEN 'll > rh'\n WHEN 't_meets' THEN 'lh = rl'\n WHEN 't_metby' THEN 'll = rh'\n WHEN 't_overlaps' THEN 'll < rl AND rl < lh < rh'\n WHEN 't_overlappedby' THEN 'rl < ll < rh AND lh > rh'\n WHEN 't_starts' THEN 'll = rl AND lh < rh'\n WHEN 't_startedby' THEN 'll = rl AND lh > rh'\n WHEN 't_during' THEN 'll > rl AND lh < rh'\n WHEN 't_contains' THEN 'll < rl AND lh > rh'\n WHEN 't_finishes' THEN 'll > rl AND lh = rh'\n WHEN 't_finishedby' THEN 'll < rl AND lh = rh'\n WHEN 't_equals' THEN 'll = rl AND lh = rh'\n WHEN 't_disjoint' THEN 'NOT (ll <= rh AND lh >= rl)'\n WHEN 't_intersects' THEN 'll <= rh AND lh >= rl'\n WHEN 'anyinteracts' THEN 'll <= rh AND lh >= rl'\n END;\n outq := regexp_replace(outq, '\\mll\\M', ll);\n outq := regexp_replace(outq, '\\mlh\\M', lh);\n outq := regexp_replace(outq, '\\mrl\\M', rl);\n outq := regexp_replace(outq, '\\mrh\\M', rh);\n outq := format('(%s)', outq);\n RETURN outq;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\n\n\nCREATE OR REPLACE FUNCTION spatial_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\n geom text;\n j jsonb := args->1;\nBEGIN\n op := lower(op);\n RAISE NOTICE 'Constructing spatial query OP: %, ARGS: %', op, args;\n IF op NOT IN ('s_equals','s_disjoint','s_touches','s_within','s_overlaps','s_crosses','s_intersects','intersects','s_contains') THEN\n RAISE EXCEPTION 'Spatial Operator % Not Supported', op;\n END IF;\n op := regexp_replace(op, '^s_', 'st_');\n IF op = 'intersects' THEN\n op := 'st_intersects';\n END IF;\n -- Convert geometry to WKB string\n IF j ? 'type' AND j ? 'coordinates' THEN\n geom := st_geomfromgeojson(j)::text;\n ELSIF jsonb_typeof(j) = 'array' THEN\n geom := bbox_geom(j)::text;\n END IF;\n\n RETURN format('%s(geometry, %L::geometry)', op, geom);\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION query_to_cql2(q jsonb) RETURNS jsonb AS $$\n-- Translates anything passed in through the deprecated \"query\" into equivalent CQL2\nWITH t AS (\n SELECT key as property, value as ops\n FROM jsonb_each(q)\n), t2 AS (\n SELECT property, (jsonb_each(ops)).*\n FROM t WHERE jsonb_typeof(ops) = 'object'\n UNION ALL\n SELECT property, 'eq', ops\n FROM t WHERE jsonb_typeof(ops) != 'object'\n)\nSELECT\n jsonb_strip_nulls(jsonb_build_object(\n 'op', 'and',\n 'args', jsonb_agg(\n jsonb_build_object(\n 'op', key,\n 'args', jsonb_build_array(\n jsonb_build_object('property',property),\n value\n )\n )\n )\n )\n) as qcql FROM t2\n;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\nCREATE OR REPLACE FUNCTION cql1_to_cql2(j jsonb) RETURNS jsonb AS $$\nDECLARE\n args jsonb;\n ret jsonb;\nBEGIN\n RAISE NOTICE 'CQL1_TO_CQL2: %', j;\n IF j ? 'filter' THEN\n RETURN cql1_to_cql2(j->'filter');\n END IF;\n IF j ? 'property' THEN\n RETURN j;\n END IF;\n IF jsonb_typeof(j) = 'array' THEN\n SELECT jsonb_agg(cql1_to_cql2(el)) INTO args FROM jsonb_array_elements(j) el;\n RETURN args;\n END IF;\n IF jsonb_typeof(j) = 'number' THEN\n RETURN j;\n END IF;\n IF jsonb_typeof(j) = 'string' THEN\n RETURN j;\n END IF;\n\n IF jsonb_typeof(j) = 'object' THEN\n SELECT jsonb_build_object(\n 'op', key,\n 'args', cql1_to_cql2(value)\n ) INTO ret\n FROM jsonb_each(j)\n WHERE j IS NOT NULL;\n RETURN ret;\n END IF;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE STRICT;\n\nCREATE TABLE cql2_ops (\n op text PRIMARY KEY,\n template text,\n types text[]\n);\nINSERT INTO cql2_ops (op, template, types) VALUES\n ('eq', '%s = %s', NULL),\n ('neq', '%s != %s', NULL),\n ('ne', '%s != %s', NULL),\n ('!=', '%s != %s', NULL),\n ('<>', '%s != %s', NULL),\n ('lt', '%s < %s', NULL),\n ('lte', '%s <= %s', NULL),\n ('gt', '%s > %s', NULL),\n ('gte', '%s >= %s', NULL),\n ('le', '%s <= %s', NULL),\n ('ge', '%s >= %s', NULL),\n ('=', '%s = %s', NULL),\n ('<', '%s < %s', NULL),\n ('<=', '%s <= %s', NULL),\n ('>', '%s > %s', NULL),\n ('>=', '%s >= %s', NULL),\n ('like', '%s LIKE %s', NULL),\n ('ilike', '%s ILIKE %s', NULL),\n ('+', '%s + %s', NULL),\n ('-', '%s - %s', NULL),\n ('*', '%s * %s', NULL),\n ('/', '%s / %s', NULL),\n ('not', 'NOT (%s)', NULL),\n ('between', '%s BETWEEN %s AND %s', NULL),\n ('isnull', '%s IS NULL', NULL),\n ('upper', 'upper(%s)', NULL),\n ('lower', 'lower(%s)', NULL)\nON CONFLICT (op) DO UPDATE\n SET\n template = EXCLUDED.template\n;\n\n\nCREATE OR REPLACE FUNCTION cql2_query(j jsonb, wrapper text DEFAULT NULL) RETURNS text AS $$\n#variable_conflict use_variable\nDECLARE\n args jsonb := j->'args';\n arg jsonb;\n op text := lower(j->>'op');\n cql2op RECORD;\n literal text;\n _wrapper text;\n leftarg text;\n rightarg text;\nBEGIN\n IF j IS NULL OR (op IS NOT NULL AND args IS NULL) THEN\n RETURN NULL;\n END IF;\n RAISE NOTICE 'CQL2_QUERY: %', j;\n IF j ? 'filter' THEN\n RETURN cql2_query(j->'filter');\n END IF;\n\n IF j ? 'upper' THEN\n RETURN cql2_query(jsonb_build_object('op', 'upper', 'args', j->'upper'));\n END IF;\n\n IF j ? 'lower' THEN\n RETURN cql2_query(jsonb_build_object('op', 'lower', 'args', j->'lower'));\n END IF;\n\n -- Temporal Query\n IF op ilike 't_%' or op = 'anyinteracts' THEN\n RETURN temporal_op_query(op, args);\n END IF;\n\n -- If property is a timestamp convert it to text to use with\n -- general operators\n IF j ? 'timestamp' THEN\n RETURN format('%L::timestamptz', to_tstz(j->'timestamp'));\n END IF;\n IF j ? 'interval' THEN\n RAISE EXCEPTION 'Please use temporal operators when using intervals.';\n RETURN NONE;\n END IF;\n\n -- Spatial Query\n IF op ilike 's_%' or op = 'intersects' THEN\n RETURN spatial_op_query(op, args);\n END IF;\n\n IF op IN ('a_equals','a_contains','a_contained_by','a_overlaps') THEN\n IF args->0 ? 'property' THEN\n leftarg := format('to_text_array(%s)', (queryable(args->0->>'property')).path);\n END IF;\n IF args->1 ? 'property' THEN\n rightarg := format('to_text_array(%s)', (queryable(args->1->>'property')).path);\n END IF;\n RETURN FORMAT(\n '%s %s %s',\n COALESCE(leftarg, quote_literal(to_text_array(args->0))),\n CASE op\n WHEN 'a_equals' THEN '='\n WHEN 'a_contains' THEN '@>'\n WHEN 'a_contained_by' THEN '<@'\n WHEN 'a_overlaps' THEN '&&'\n END,\n COALESCE(rightarg, quote_literal(to_text_array(args->1)))\n );\n END IF;\n\n IF op = 'in' THEN\n RAISE NOTICE 'IN : % % %', args, jsonb_build_array(args->0), args->1;\n args := jsonb_build_array(args->0) || (args->1);\n RAISE NOTICE 'IN2 : %', args;\n END IF;\n\n\n\n IF op = 'between' THEN\n args = jsonb_build_array(\n args->0,\n args->1->0,\n args->1->1\n );\n END IF;\n\n -- Make sure that args is an array and run cql2_query on\n -- each element of the array\n RAISE NOTICE 'ARGS PRE: %', args;\n IF j ? 'args' THEN\n IF jsonb_typeof(args) != 'array' THEN\n args := jsonb_build_array(args);\n END IF;\n\n IF jsonb_path_exists(args, '$[*] ? (@.property == \"id\" || @.property == \"datetime\" || @.property == \"end_datetime\" || @.property == \"collection\")') THEN\n wrapper := NULL;\n ELSE\n -- if any of the arguments are a property, try to get the property_wrapper\n FOR arg IN SELECT jsonb_path_query(args, '$[*] ? (@.property != null)') LOOP\n RAISE NOTICE 'Arg: %', arg;\n wrapper := (queryable(arg->>'property')).nulled_wrapper;\n RAISE NOTICE 'Property: %, Wrapper: %', arg, wrapper;\n IF wrapper IS NOT NULL THEN\n EXIT;\n END IF;\n END LOOP;\n\n -- if the property was not in queryables, see if any args were numbers\n IF\n wrapper IS NULL\n AND jsonb_path_exists(args, '$[*] ? (@.type()==\"number\")')\n THEN\n wrapper := 'to_float';\n END IF;\n wrapper := coalesce(wrapper, 'to_text');\n END IF;\n\n SELECT jsonb_agg(cql2_query(a, wrapper))\n INTO args\n FROM jsonb_array_elements(args) a;\n END IF;\n RAISE NOTICE 'ARGS: %', args;\n\n IF op IN ('and', 'or') THEN\n RETURN\n format(\n '(%s)',\n array_to_string(to_text_array(args), format(' %s ', upper(op)))\n );\n END IF;\n\n IF op = 'in' THEN\n RAISE NOTICE 'IN -- % %', args->0, to_text(args->0);\n RETURN format(\n '%s IN (%s)',\n to_text(args->0),\n array_to_string((to_text_array(args))[2:], ',')\n );\n END IF;\n\n -- Look up template from cql2_ops\n IF j ? 'op' THEN\n SELECT * INTO cql2op FROM cql2_ops WHERE cql2_ops.op ilike op;\n IF FOUND THEN\n -- If specific index set in queryables for a property cast other arguments to that type\n\n RETURN format(\n cql2op.template,\n VARIADIC (to_text_array(args))\n );\n ELSE\n RAISE EXCEPTION 'Operator % Not Supported.', op;\n END IF;\n END IF;\n\n\n IF wrapper IS NOT NULL THEN\n RAISE NOTICE 'Wrapping % with %', j, wrapper;\n IF j ? 'property' THEN\n RETURN format('%I(%s)', wrapper, (queryable(j->>'property')).path);\n ELSE\n RETURN format('%I(%L)', wrapper, j);\n END IF;\n ELSIF j ? 'property' THEN\n RETURN quote_ident(j->>'property');\n END IF;\n\n RETURN quote_literal(to_text(j));\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION paging_dtrange(\n j jsonb\n) RETURNS tstzrange AS $$\nDECLARE\n op text;\n filter jsonb := j->'filter';\n dtrange tstzrange := tstzrange('-infinity'::timestamptz,'infinity'::timestamptz);\n sdate timestamptz := '-infinity'::timestamptz;\n edate timestamptz := 'infinity'::timestamptz;\n jpitem jsonb;\nBEGIN\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n END IF;\n IF NOT (filter @? '$.**.op ? (@ == \"or\" || @ == \"not\")') THEN\n FOR jpitem IN SELECT j FROM jsonb_path_query(filter,'strict $.** ? (@.args[*].property == \"datetime\")'::jsonpath) j LOOP\n op := lower(jpitem->>'op');\n dtrange := parse_dtrange(jpitem->'args'->1);\n IF op IN ('<=', 'lt', 'lte', '<', 'le', 't_before') THEN\n sdate := greatest(sdate,'-infinity');\n edate := least(edate, upper(dtrange));\n ELSIF op IN ('>=', '>', 'gt', 'gte', 'ge', 't_after') THEN\n edate := least(edate, 'infinity');\n sdate := greatest(sdate, lower(dtrange));\n ELSIF op IN ('=', 'eq') THEN\n edate := least(edate, upper(dtrange));\n sdate := greatest(sdate, lower(dtrange));\n END IF;\n RAISE NOTICE '2 OP: %, ARGS: %, DTRANGE: %, SDATE: %, EDATE: %', op, jpitem->'args'->1, dtrange, sdate, edate;\n END LOOP;\n END IF;\n IF sdate > edate THEN\n RETURN 'empty'::tstzrange;\n END IF;\n RETURN tstzrange(sdate,edate, '[]');\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT SET TIME ZONE 'UTC';\n\nCREATE OR REPLACE FUNCTION paging_collections(\n IN j jsonb\n) RETURNS text[] AS $$\nDECLARE\n filter jsonb := j->'filter';\n jpitem jsonb;\n op text;\n args jsonb;\n arg jsonb;\n collections text[];\nBEGIN\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n END IF;\n IF NOT (filter @? '$.**.op ? (@ == \"or\" || @ == \"not\")') THEN\n FOR jpitem IN SELECT j FROM jsonb_path_query(filter,'strict $.** ? (@.args[*].property == \"collection\")'::jsonpath) j LOOP\n RAISE NOTICE 'JPITEM: %', jpitem;\n op := jpitem->>'op';\n args := jpitem->'args';\n IF op IN ('=', 'eq', 'in') THEN\n FOR arg IN SELECT a FROM jsonb_array_elements(args) a LOOP\n IF jsonb_typeof(arg) IN ('string', 'array') THEN\n RAISE NOTICE 'arg: %, collections: %', arg, collections;\n IF collections IS NULL OR collections = '{}'::text[] THEN\n collections := to_text_array(arg);\n ELSE\n collections := array_intersection(collections, to_text_array(arg));\n END IF;\n END IF;\n END LOOP;\n END IF;\n END LOOP;\n END IF;\n IF collections = '{}'::text[] THEN\n RETURN NULL;\n END IF;\n RETURN collections;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\nCREATE OR REPLACE FUNCTION collection_base_item(content jsonb) RETURNS jsonb AS $$\n SELECT jsonb_build_object(\n 'type', 'Feature',\n 'stac_version', content->'stac_version',\n 'assets', content->'item_assets',\n 'collection', content->'id'\n );\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TABLE IF NOT EXISTS collections (\n key bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,\n id text GENERATED ALWAYS AS (content->>'id') STORED UNIQUE,\n content JSONB NOT NULL,\n base_item jsonb GENERATED ALWAYS AS (pgstac.collection_base_item(content)) STORED,\n partition_trunc text CHECK (partition_trunc IN ('year', 'month'))\n);\n\n\n\nCREATE OR REPLACE FUNCTION collection_base_item(cid text) RETURNS jsonb AS $$\n SELECT pgstac.collection_base_item(content) FROM pgstac.collections WHERE id = cid LIMIT 1;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION create_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION update_collection(data jsonb) RETURNS VOID AS $$\nDECLARE\n out collections%ROWTYPE;\nBEGIN\n UPDATE collections SET content=data WHERE id = data->>'id' RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ON CONFLICT (id) DO\n UPDATE\n SET content=EXCLUDED.content\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION delete_collection(_id text) RETURNS VOID AS $$\nDECLARE\n out collections%ROWTYPE;\nBEGIN\n DELETE FROM collections WHERE id = _id RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION get_collection(id text) RETURNS jsonb AS $$\n SELECT content FROM collections\n WHERE id=$1\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION all_collections() RETURNS jsonb AS $$\n SELECT jsonb_agg(content) FROM collections;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\nCREATE TABLE items (\n id text NOT NULL,\n geometry geometry NOT NULL,\n collection text NOT NULL,\n datetime timestamptz NOT NULL,\n end_datetime timestamptz NOT NULL,\n content JSONB NOT NULL\n)\nPARTITION BY LIST (collection)\n;\n\nCREATE INDEX \"datetime_idx\" ON items USING BTREE (datetime DESC, end_datetime ASC);\nCREATE INDEX \"geometry_idx\" ON items USING GIST (geometry);\n\nCREATE STATISTICS datetime_stats (dependencies) on datetime, end_datetime from items;\n\nALTER TABLE items ADD CONSTRAINT items_collections_fk FOREIGN KEY (collection) REFERENCES collections(id) ON DELETE CASCADE DEFERRABLE;\n\nCREATE OR REPLACE FUNCTION partition_after_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p text;\n t timestamptz := clock_timestamp();\nBEGIN\n RAISE NOTICE 'Updating partition stats %', t;\n FOR p IN SELECT DISTINCT partition\n FROM newdata n JOIN partition_sys_meta p\n ON (n.collection=p.collection AND n.datetime <@ p.partition_dtrange)\n LOOP\n PERFORM run_or_queue(format('SELECT update_partition_stats(%L, %L);', p, true));\n END LOOP;\n IF TG_OP IN ('DELETE','UPDATE') THEN\n DELETE FROM format_item_cache c USING newdata n WHERE c.collection = n.collection AND c.id = n.id;\n END IF;\n RAISE NOTICE 't: % %', t, clock_timestamp() - t;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE TRIGGER items_after_insert_trigger\nAFTER INSERT ON items\nREFERENCING NEW TABLE AS newdata\nFOR EACH STATEMENT\nEXECUTE FUNCTION partition_after_triggerfunc();\n\nCREATE TRIGGER items_after_update_trigger\nAFTER DELETE ON items\nREFERENCING OLD TABLE AS newdata\nFOR EACH STATEMENT\nEXECUTE FUNCTION partition_after_triggerfunc();\n\nCREATE TRIGGER items_after_delete_trigger\nAFTER UPDATE ON items\nREFERENCING NEW TABLE AS newdata\nFOR EACH STATEMENT\nEXECUTE FUNCTION partition_after_triggerfunc();\n\n\nCREATE OR REPLACE FUNCTION content_slim(_item jsonb) RETURNS jsonb AS $$\n SELECT strip_jsonb(_item - '{id,geometry,collection,type}'::text[], collection_base_item(_item->>'collection')) - '{id,geometry,collection,type}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_dehydrate(content jsonb) RETURNS items AS $$\n SELECT\n content->>'id' as id,\n stac_geom(content) as geometry,\n content->>'collection' as collection,\n stac_datetime(content) as datetime,\n stac_end_datetime(content) as end_datetime,\n content_slim(content) as content\n ;\n$$ LANGUAGE SQL STABLE;\n\nCREATE OR REPLACE FUNCTION include_field(f text, fields jsonb DEFAULT '{}'::jsonb) RETURNS boolean AS $$\nDECLARE\n includes jsonb := fields->'include';\n excludes jsonb := fields->'exclude';\nBEGIN\n IF f IS NULL THEN\n RETURN NULL;\n END IF;\n\n\n IF\n jsonb_typeof(excludes) = 'array'\n AND jsonb_array_length(excludes)>0\n AND excludes ? f\n THEN\n RETURN FALSE;\n END IF;\n\n IF\n (\n jsonb_typeof(includes) = 'array'\n AND jsonb_array_length(includes) > 0\n AND includes ? f\n ) OR\n (\n includes IS NULL\n OR jsonb_typeof(includes) = 'null'\n OR jsonb_array_length(includes) = 0\n )\n THEN\n RETURN TRUE;\n END IF;\n\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nDROP FUNCTION IF EXISTS content_hydrate(jsonb, jsonb, jsonb);\nCREATE OR REPLACE FUNCTION content_hydrate(\n _item jsonb,\n _base_item jsonb,\n fields jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\n SELECT merge_jsonb(\n jsonb_fields(_item, fields),\n jsonb_fields(_base_item, fields)\n );\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION content_hydrate(_item items, _collection collections, fields jsonb DEFAULT '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\n content jsonb;\n base_item jsonb := _collection.base_item;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry, 20)::jsonb;\n END IF;\n output := content_hydrate(\n jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'collection', _item.collection,\n 'type', 'Feature'\n ) || _item.content,\n _collection.base_item,\n fields\n );\n\n RETURN output;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_nonhydrated(\n _item items,\n fields jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry, 20)::jsonb;\n END IF;\n output := jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'collection', _item.collection,\n 'type', 'Feature'\n ) || _item.content;\n RETURN output;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_hydrate(_item items, fields jsonb DEFAULT '{}'::jsonb) RETURNS jsonb AS $$\n SELECT content_hydrate(\n _item,\n (SELECT c FROM collections c WHERE id=_item.collection LIMIT 1),\n fields\n );\n$$ LANGUAGE SQL STABLE;\n\n\nCREATE UNLOGGED TABLE items_staging (\n content JSONB NOT NULL\n);\nCREATE UNLOGGED TABLE items_staging_ignore (\n content JSONB NOT NULL\n);\nCREATE UNLOGGED TABLE items_staging_upsert (\n content JSONB NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION items_staging_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p record;\n _partitions text[];\n part text;\n ts timestamptz := clock_timestamp();\nBEGIN\n RAISE NOTICE 'Creating Partitions. %', clock_timestamp() - ts;\n\n FOR part IN WITH t AS (\n SELECT\n n.content->>'collection' as collection,\n stac_daterange(n.content->'properties') as dtr,\n partition_trunc\n FROM newdata n JOIN collections ON (n.content->>'collection'=collections.id)\n ), p AS (\n SELECT\n collection,\n COALESCE(date_trunc(partition_trunc::text, lower(dtr)),'-infinity') as d,\n tstzrange(min(lower(dtr)),max(lower(dtr)),'[]') as dtrange,\n tstzrange(min(upper(dtr)),max(upper(dtr)),'[]') as edtrange\n FROM t\n GROUP BY 1,2\n ) SELECT check_partition(collection, dtrange, edtrange) FROM p LOOP\n RAISE NOTICE 'Partition %', part;\n END LOOP;\n\n RAISE NOTICE 'Doing the insert. %', clock_timestamp() - ts;\n IF TG_TABLE_NAME = 'items_staging' THEN\n INSERT INTO items\n SELECT\n (content_dehydrate(content)).*\n FROM newdata;\n RAISE NOTICE 'Doing the delete. %', clock_timestamp() - ts;\n DELETE FROM items_staging;\n ELSIF TG_TABLE_NAME = 'items_staging_ignore' THEN\n INSERT INTO items\n SELECT\n (content_dehydrate(content)).*\n FROM newdata\n ON CONFLICT DO NOTHING;\n RAISE NOTICE 'Doing the delete. %', clock_timestamp() - ts;\n DELETE FROM items_staging_ignore;\n ELSIF TG_TABLE_NAME = 'items_staging_upsert' THEN\n WITH staging_formatted AS (\n SELECT (content_dehydrate(content)).* FROM newdata\n ), deletes AS (\n DELETE FROM items i USING staging_formatted s\n WHERE\n i.id = s.id\n AND i.collection = s.collection\n AND i IS DISTINCT FROM s\n RETURNING i.id, i.collection\n )\n INSERT INTO items\n SELECT s.* FROM\n staging_formatted s\n ON CONFLICT DO NOTHING;\n RAISE NOTICE 'Doing the delete. %', clock_timestamp() - ts;\n DELETE FROM items_staging_upsert;\n END IF;\n RAISE NOTICE 'Done. %', clock_timestamp() - ts;\n\n RETURN NULL;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER items_staging_insert_trigger AFTER INSERT ON items_staging REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\nCREATE TRIGGER items_staging_insert_ignore_trigger AFTER INSERT ON items_staging_ignore REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\nCREATE TRIGGER items_staging_insert_upsert_trigger AFTER INSERT ON items_staging_upsert REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\n\nCREATE OR REPLACE FUNCTION item_by_id(_id text, _collection text DEFAULT NULL) RETURNS items AS\n$$\nDECLARE\n i items%ROWTYPE;\nBEGIN\n SELECT * INTO i FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection) LIMIT 1;\n RETURN i;\nEND;\n$$ LANGUAGE PLPGSQL STABLE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION get_item(_id text, _collection text DEFAULT NULL) RETURNS jsonb AS $$\n SELECT content_hydrate(items) FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection);\n$$ LANGUAGE SQL STABLE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION delete_item(_id text, _collection text DEFAULT NULL) RETURNS VOID AS $$\nDECLARE\nout items%ROWTYPE;\nBEGIN\n DELETE FROM items WHERE id = _id AND (_collection IS NULL OR collection=_collection) RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n--/*\nCREATE OR REPLACE FUNCTION create_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION update_item(content jsonb) RETURNS VOID AS $$\nDECLARE\n old items %ROWTYPE;\n out items%ROWTYPE;\nBEGIN\n PERFORM delete_item(content->>'id', content->>'collection');\n PERFORM create_item(content);\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION create_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION collection_bbox(id text) RETURNS jsonb AS $$\n SELECT (replace(replace(replace(st_extent(geometry)::text,'BOX(','[['),')',']]'),' ',','))::jsonb\n FROM items WHERE collection=$1;\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION collection_temporal_extent(id text) RETURNS jsonb AS $$\n SELECT to_jsonb(array[array[min(datetime)::text, max(datetime)::text]])\n FROM items WHERE collection=$1;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection_extents() RETURNS VOID AS $$\nUPDATE collections SET\n content = content ||\n jsonb_build_object(\n 'extent', jsonb_build_object(\n 'spatial', jsonb_build_object(\n 'bbox', collection_bbox(collections.id)\n ),\n 'temporal', jsonb_build_object(\n 'interval', collection_temporal_extent(collections.id)\n )\n )\n )\n;\n$$ LANGUAGE SQL;\nCREATE TABLE partition_stats (\n partition text PRIMARY KEY,\n dtrange tstzrange,\n edtrange tstzrange,\n spatial geometry,\n last_updated timestamptz,\n keys text[]\n) WITH (FILLFACTOR=90);\n\nCREATE INDEX partitions_range_idx ON partition_stats USING GIST(dtrange);\n\n\nCREATE OR REPLACE FUNCTION constraint_tstzrange(expr text) RETURNS tstzrange AS $$\n WITH t AS (\n SELECT regexp_matches(\n expr,\n E'\\\\(''\\([0-9 :+-]*\\)''\\\\).*\\\\(''\\([0-9 :+-]*\\)''\\\\)'\n ) AS m\n ) SELECT tstzrange(m[1]::timestamptz, m[2]::timestamptz) FROM t\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE STRICT;\n\nCREATE OR REPLACE FUNCTION dt_constraint(coid oid, OUT dt tstzrange, OUT edt tstzrange) RETURNS RECORD AS $$\nDECLARE\n expr text := pg_get_constraintdef(coid);\n matches timestamptz[];\nBEGIN\n IF expr LIKE '%NULL%' THEN\n dt := tstzrange(null::timestamptz, null::timestamptz);\n edt := tstzrange(null::timestamptz, null::timestamptz);\n RETURN;\n END IF;\n WITH f AS (SELECT (regexp_matches(expr, E'([0-9]{4}-[0-1][0-9]-[0-3][0-9] [0-2][0-9]:[0-5][0-9]:[0-5][0-9]\\.?[0-9]*)', 'g'))[1] f)\n SELECT array_agg(f::timestamptz) INTO matches FROM f;\n IF cardinality(matches) = 4 THEN\n dt := tstzrange(matches[1], matches[2],'[]');\n edt := tstzrange(matches[3], matches[4], '[]');\n RETURN;\n ELSIF cardinality(matches) = 2 THEN\n edt := tstzrange(matches[1], matches[2],'[]');\n RETURN;\n END IF;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\nCREATE OR REPLACE VIEW partition_sys_meta AS\nSELECT\n relid::text as partition,\n replace(replace(CASE WHEN level = 1 THEN pg_get_expr(c.relpartbound, c.oid)\n ELSE pg_get_expr(parent.relpartbound, parent.oid)\n END, 'FOR VALUES IN (''',''), ''')','') AS collection,\n level,\n c.reltuples,\n c.relhastriggers,\n COALESCE(constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as partition_dtrange,\n COALESCE((dt_constraint(edt.oid)).dt, constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as constraint_dtrange,\n COALESCE((dt_constraint(edt.oid)).edt, tstzrange('-infinity', 'infinity','[]')) as constraint_edtrange\nFROM\n pg_partition_tree('items')\n JOIN pg_class c ON (relid::regclass = c.oid)\n JOIN pg_class parent ON (parentrelid::regclass = parent.oid AND isleaf)\n LEFT JOIN pg_constraint edt ON (conrelid=c.oid AND contype='c')\nWHERE isleaf\n;\n\nCREATE VIEW partitions_view AS\nSELECT\n relid::text as partition,\n replace(replace(CASE WHEN level = 1 THEN pg_get_expr(c.relpartbound, c.oid)\n ELSE pg_get_expr(parent.relpartbound, parent.oid)\n END, 'FOR VALUES IN (''',''), ''')','') AS collection,\n level,\n c.reltuples,\n c.relhastriggers,\n COALESCE(constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as partition_dtrange,\n COALESCE((dt_constraint(edt.oid)).dt, constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as constraint_dtrange,\n COALESCE((dt_constraint(edt.oid)).edt, tstzrange('-infinity', 'infinity','[]')) as constraint_edtrange,\n dtrange,\n edtrange,\n spatial,\n last_updated\nFROM\n pg_partition_tree('items')\n JOIN pg_class c ON (relid::regclass = c.oid)\n JOIN pg_class parent ON (parentrelid::regclass = parent.oid AND isleaf)\n LEFT JOIN pg_constraint edt ON (conrelid=c.oid AND contype='c')\n LEFT JOIN partition_stats ON (relid::text=partition)\nWHERE isleaf\n;\n\nCREATE MATERIALIZED VIEW partitions AS\nSELECT * FROM partitions_view;\nCREATE UNIQUE INDEX ON partitions (partition);\n\nCREATE MATERIALIZED VIEW partition_steps AS\nSELECT\n partition as name,\n date_trunc('month',lower(partition_dtrange)) as sdate,\n date_trunc('month', upper(partition_dtrange)) + '1 month'::interval as edate\n FROM partitions_view WHERE partition_dtrange IS NOT NULL AND partition_dtrange != 'empty'::tstzrange\n ORDER BY dtrange ASC\n;\n\n\nCREATE OR REPLACE FUNCTION update_partition_stats_q(_partition text, istrigger boolean default false) RETURNS VOID AS $$\nDECLARE\nBEGIN\n PERFORM run_or_queue(\n format('SELECT update_partition_stats(%L, %L);', _partition, istrigger)\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION update_partition_stats(_partition text, istrigger boolean default false) RETURNS VOID AS $$\nDECLARE\n dtrange tstzrange;\n edtrange tstzrange;\n cdtrange tstzrange;\n cedtrange tstzrange;\n extent geometry;\n collection text;\nBEGIN\n RAISE NOTICE 'Updating stats for %.', _partition;\n EXECUTE format(\n $q$\n SELECT\n tstzrange(min(datetime), max(datetime),'[]'),\n tstzrange(min(end_datetime), max(end_datetime), '[]')\n FROM %I\n $q$,\n _partition\n ) INTO dtrange, edtrange;\n extent := st_estimatedextent('pgstac', _partition, 'geometry');\n INSERT INTO partition_stats (partition, dtrange, edtrange, spatial, last_updated)\n SELECT _partition, dtrange, edtrange, extent, now()\n ON CONFLICT (partition) DO\n UPDATE SET\n dtrange=EXCLUDED.dtrange,\n edtrange=EXCLUDED.edtrange,\n spatial=EXCLUDED.spatial,\n last_updated=EXCLUDED.last_updated\n ;\n\n SELECT\n constraint_dtrange, constraint_edtrange, pv.collection\n INTO cdtrange, cedtrange, collection\n FROM partitions_view pv WHERE partition = _partition;\n REFRESH MATERIALIZED VIEW partitions;\n REFRESH MATERIALIZED VIEW partition_steps;\n\n\n RAISE NOTICE 'Checking if we need to modify constraints.';\n IF\n (cdtrange IS DISTINCT FROM dtrange OR edtrange IS DISTINCT FROM cedtrange)\n AND NOT istrigger\n THEN\n RAISE NOTICE 'Modifying Constraints';\n RAISE NOTICE 'Existing % %', cdtrange, cedtrange;\n RAISE NOTICE 'New % %', dtrange, edtrange;\n PERFORM drop_table_constraints(_partition);\n PERFORM create_table_constraints(_partition, dtrange, edtrange);\n END IF;\n RAISE NOTICE 'Checking if we need to update collection extents.';\n IF get_setting_bool('update_collection_extent') THEN\n RAISE NOTICE 'updating collection extent for %', collection;\n PERFORM run_or_queue(format($q$\n UPDATE collections\n SET content = jsonb_set_lax(\n content,\n '{extent}'::text[],\n collection_extent(%L, FALSE),\n true,\n 'use_json_null'\n ) WHERE id=%L\n ;\n $q$, collection, collection));\n ELSE\n RAISE NOTICE 'Not updating collection extent for %', collection;\n END IF;\n\nEND;\n$$ LANGUAGE PLPGSQL STRICT;\n\n\nCREATE OR REPLACE FUNCTION partition_name( IN collection text, IN dt timestamptz, OUT partition_name text, OUT partition_range tstzrange) AS $$\nDECLARE\n c RECORD;\n parent_name text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n parent_name := format('_items_%s', c.key);\n\n\n IF c.partition_trunc = 'year' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYY'));\n ELSIF c.partition_trunc = 'month' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYYMM'));\n ELSE\n partition_name := parent_name;\n partition_range := tstzrange('-infinity'::timestamptz, 'infinity'::timestamptz, '[]');\n END IF;\n IF partition_range IS NULL THEN\n partition_range := tstzrange(\n date_trunc(c.partition_trunc::text, dt),\n date_trunc(c.partition_trunc::text, dt) + concat('1 ', c.partition_trunc)::interval\n );\n END IF;\n RETURN;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION drop_table_constraints(t text) RETURNS text AS $$\nDECLARE\n q text;\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM partitions_view WHERE partition=t) THEN\n RETURN NULL;\n END IF;\n FOR q IN SELECT FORMAT(\n $q$\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n $q$,\n t,\n conname\n ) FROM pg_constraint\n WHERE conrelid=t::regclass::oid AND contype='c'\n LOOP\n EXECUTE q;\n END LOOP;\n RETURN t;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE OR REPLACE FUNCTION create_table_constraints(t text, _dtrange tstzrange, _edtrange tstzrange) RETURNS text AS $$\nDECLARE\n q text;\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM partitions_view WHERE partition=t) THEN\n RETURN NULL;\n END IF;\n RAISE NOTICE 'Creating Table Constraints for % % %', t, _dtrange, _edtrange;\n IF _dtrange = 'empty' AND _edtrange = 'empty' THEN\n q :=format(\n $q$\n DO $block$\n BEGIN\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n ALTER TABLE %I\n ADD CONSTRAINT %I\n CHECK (((datetime IS NULL) AND (end_datetime IS NULL))) NOT VALID\n ;\n ALTER TABLE %I\n VALIDATE CONSTRAINT %I\n ;\n\n\n\n EXCEPTION WHEN others THEN\n RAISE WARNING '%%, Issue Altering Constraints. Please run update_partition_stats(%I)', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n $block$;\n $q$,\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t),\n t\n );\n ELSE\n q :=format(\n $q$\n DO $block$\n BEGIN\n\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n ALTER TABLE %I\n ADD CONSTRAINT %I\n CHECK (\n (datetime >= %L)\n AND (datetime <= %L)\n AND (end_datetime >= %L)\n AND (end_datetime <= %L)\n ) NOT VALID\n ;\n ALTER TABLE %I\n VALIDATE CONSTRAINT %I\n ;\n\n\n\n EXCEPTION WHEN others THEN\n RAISE WARNING '%%, Issue Altering Constraints. Please run update_partition_stats(%I)', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n $block$;\n $q$,\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t),\n lower(_dtrange),\n upper(_dtrange),\n lower(_edtrange),\n upper(_edtrange),\n t,\n format('%s_dt', t),\n t\n );\n END IF;\n PERFORM run_or_queue(q);\n RETURN t;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION check_partition(\n _collection text,\n _dtrange tstzrange,\n _edtrange tstzrange\n) RETURNS text AS $$\nDECLARE\n c RECORD;\n pm RECORD;\n _partition_name text;\n _partition_dtrange tstzrange;\n _constraint_dtrange tstzrange;\n _constraint_edtrange tstzrange;\n q text;\n deferrable_q text;\n err_context text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=_collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', _collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n\n IF c.partition_trunc IS NOT NULL THEN\n _partition_dtrange := tstzrange(\n date_trunc(c.partition_trunc, lower(_dtrange)),\n date_trunc(c.partition_trunc, lower(_dtrange)) + (concat('1 ', c.partition_trunc))::interval,\n '[)'\n );\n ELSE\n _partition_dtrange := '[-infinity, infinity]'::tstzrange;\n END IF;\n\n IF NOT _partition_dtrange @> _dtrange THEN\n RAISE EXCEPTION 'dtrange % is greater than the partition size % for collection %', _dtrange, c.partition_trunc, _collection;\n END IF;\n\n\n IF c.partition_trunc = 'year' THEN\n _partition_name := format('_items_%s_%s', c.key, to_char(lower(_partition_dtrange),'YYYY'));\n ELSIF c.partition_trunc = 'month' THEN\n _partition_name := format('_items_%s_%s', c.key, to_char(lower(_partition_dtrange),'YYYYMM'));\n ELSE\n _partition_name := format('_items_%s', c.key);\n END IF;\n\n SELECT * INTO pm FROM partition_sys_meta WHERE collection=_collection AND partition_dtrange @> _dtrange;\n IF FOUND THEN\n RAISE NOTICE '% % %', _edtrange, _dtrange, pm;\n _constraint_edtrange :=\n tstzrange(\n least(\n lower(_edtrange),\n nullif(lower(pm.constraint_edtrange), '-infinity')\n ),\n greatest(\n upper(_edtrange),\n nullif(upper(pm.constraint_edtrange), 'infinity')\n ),\n '[]'\n );\n _constraint_dtrange :=\n tstzrange(\n least(\n lower(_dtrange),\n nullif(lower(pm.constraint_dtrange), '-infinity')\n ),\n greatest(\n upper(_dtrange),\n nullif(upper(pm.constraint_dtrange), 'infinity')\n ),\n '[]'\n );\n\n IF pm.constraint_edtrange @> _edtrange AND pm.constraint_dtrange @> _dtrange THEN\n RETURN pm.partition;\n ELSE\n PERFORM drop_table_constraints(_partition_name);\n END IF;\n ELSE\n _constraint_edtrange := _edtrange;\n _constraint_dtrange := _dtrange;\n END IF;\n RAISE NOTICE 'Creating partition % %', _partition_name, _partition_dtrange;\n IF c.partition_trunc IS NULL THEN\n q := format(\n $q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n $q$,\n _partition_name,\n _collection,\n concat(_partition_name,'_pk'),\n _partition_name\n );\n ELSE\n q := format(\n $q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L) PARTITION BY RANGE (datetime);\n CREATE TABLE IF NOT EXISTS %I partition OF %I FOR VALUES FROM (%L) TO (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n $q$,\n format('_items_%s', c.key),\n _collection,\n _partition_name,\n format('_items_%s', c.key),\n lower(_partition_dtrange),\n upper(_partition_dtrange),\n format('%s_pk', _partition_name),\n _partition_name\n );\n END IF;\n\n BEGIN\n EXECUTE q;\n EXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', _partition_name;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\n END;\n PERFORM create_table_constraints(_partition_name, _constraint_dtrange, _constraint_edtrange);\n PERFORM maintain_partitions(_partition_name);\n PERFORM update_partition_stats_q(_partition_name, true);\n REFRESH MATERIALIZED VIEW partitions;\n REFRESH MATERIALIZED VIEW partition_steps;\n RETURN _partition_name;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION repartition(_collection text, _partition_trunc text, triggered boolean DEFAULT FALSE) RETURNS text AS $$\nDECLARE\n c RECORD;\n q text;\n from_trunc text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=_collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', _collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n IF triggered THEN\n RAISE NOTICE 'Converting % to % partitioning via Trigger', _collection, _partition_trunc;\n ELSE\n RAISE NOTICE 'Converting % from using % to % partitioning', _collection, c.partition_trunc, _partition_trunc;\n IF c.partition_trunc IS NOT DISTINCT FROM _partition_trunc THEN\n RAISE NOTICE 'Collection % already set to use partition by %', _collection, _partition_trunc;\n RETURN _collection;\n END IF;\n END IF;\n\n IF EXISTS (SELECT 1 FROM partitions_view WHERE collection=_collection LIMIT 1) THEN\n EXECUTE format(\n $q$\n CREATE TEMP TABLE changepartitionstaging ON COMMIT DROP AS SELECT * FROM %I;\n DROP TABLE IF EXISTS %I CASCADE;\n WITH p AS (\n SELECT\n collection,\n CASE WHEN %L IS NULL THEN '-infinity'::timestamptz\n ELSE date_trunc(%L, datetime)\n END as d,\n tstzrange(min(datetime),max(datetime),'[]') as dtrange,\n tstzrange(min(datetime),max(datetime),'[]') as edtrange\n FROM changepartitionstaging\n GROUP BY 1,2\n ) SELECT check_partition(collection, dtrange, edtrange) FROM p;\n INSERT INTO items SELECT * FROM changepartitionstaging;\n DROP TABLE changepartitionstaging;\n $q$,\n concat('_items_', c.key),\n concat('_items_', c.key),\n c.partition_trunc,\n c.partition_trunc\n );\n END IF;\n RETURN _collection;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE OR REPLACE FUNCTION collections_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n q text;\n partition_name text := format('_items_%s', NEW.key);\n partition_exists boolean := false;\n partition_empty boolean := true;\n err_context text;\n loadtemp boolean := FALSE;\nBEGIN\n RAISE NOTICE 'Collection Trigger. % %', NEW.id, NEW.key;\n IF TG_OP = 'UPDATE' AND NEW.partition_trunc IS DISTINCT FROM OLD.partition_trunc THEN\n PERFORM repartition(NEW.id, NEW.partition_trunc, TRUE);\n END IF;\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER collections_trigger AFTER\nINSERT\nOR\nUPDATE ON collections\nFOR EACH ROW EXECUTE FUNCTION collections_trigger_func();\n\nCREATE OR REPLACE FUNCTION chunker(\n IN _where text,\n OUT s timestamptz,\n OUT e timestamptz\n) RETURNS SETOF RECORD AS $$\nDECLARE\n explain jsonb;\nBEGIN\n IF _where IS NULL THEN\n _where := ' TRUE ';\n END IF;\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s;', _where)\n INTO explain;\n\n RETURN QUERY\n WITH t AS (\n SELECT j->>0 as p FROM\n jsonb_path_query(\n explain,\n 'strict $.**.\"Relation Name\" ? (@ != null)'\n ) j\n ),\n parts AS (\n SELECT sdate, edate FROM t JOIN partition_steps ON (t.p = name)\n ),\n times AS (\n SELECT sdate FROM parts\n UNION\n SELECT edate FROM parts\n ),\n uniq AS (\n SELECT DISTINCT sdate FROM times ORDER BY sdate\n ),\n last AS (\n SELECT sdate, lead(sdate, 1) over () as edate FROM uniq\n )\n SELECT sdate, edate FROM last WHERE edate IS NOT NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION partition_queries(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN partitions text[] DEFAULT NULL\n) RETURNS SETOF text AS $$\nDECLARE\n query text;\n sdate timestamptz;\n edate timestamptz;\nBEGIN\nIF _where IS NULL OR trim(_where) = '' THEN\n _where = ' TRUE ';\nEND IF;\nRAISE NOTICE 'Getting chunks for % %', _where, _orderby;\nIF _orderby ILIKE 'datetime d%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 DESC LOOP\n RETURN NEXT format($q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n $q$,\n sdate,\n edate,\n _where,\n _orderby\n );\n END LOOP;\nELSIF _orderby ILIKE 'datetime a%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 ASC LOOP\n RETURN NEXT format($q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n $q$,\n sdate,\n edate,\n _where,\n _orderby\n );\n END LOOP;\nELSE\n query := format($q$\n SELECT * FROM items\n WHERE %s\n ORDER BY %s\n $q$, _where, _orderby\n );\n\n RETURN NEXT query;\n RETURN;\nEND IF;\n\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION partition_query_view(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN _limit int DEFAULT 10\n) RETURNS text AS $$\n WITH p AS (\n SELECT * FROM partition_queries(_where, _orderby) p\n )\n SELECT\n CASE WHEN EXISTS (SELECT 1 FROM p) THEN\n (SELECT format($q$\n SELECT * FROM (\n %s\n ) total LIMIT %s\n $q$,\n string_agg(\n format($q$ SELECT * FROM ( %s ) AS sub $q$, p),\n '\n UNION ALL\n '\n ),\n _limit\n ))\n ELSE NULL\n END FROM p;\n$$ LANGUAGE SQL IMMUTABLE;\n\n\n\n\nCREATE OR REPLACE FUNCTION stac_search_to_where(j jsonb) RETURNS text AS $$\nDECLARE\n where_segments text[];\n _where text;\n dtrange tstzrange;\n collections text[];\n geom geometry;\n sdate timestamptz;\n edate timestamptz;\n filterlang text;\n filter jsonb := j->'filter';\nBEGIN\n IF j ? 'ids' THEN\n where_segments := where_segments || format('id = ANY (%L) ', to_text_array(j->'ids'));\n END IF;\n\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n where_segments := where_segments || format('collection = ANY (%L) ', collections);\n END IF;\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n\n where_segments := where_segments || format(' datetime <= %L::timestamptz AND end_datetime >= %L::timestamptz ',\n edate,\n sdate\n );\n END IF;\n\n geom := stac_geom(j);\n IF geom IS NOT NULL THEN\n where_segments := where_segments || format('st_intersects(geometry, %L)',geom);\n END IF;\n\n filterlang := COALESCE(\n j->>'filter-lang',\n get_setting('default_filter_lang', j->'conf')\n );\n IF NOT filter @? '$.**.op' THEN\n filterlang := 'cql-json';\n END IF;\n\n IF filterlang NOT IN ('cql-json','cql2-json') AND j ? 'filter' THEN\n RAISE EXCEPTION '% is not a supported filter-lang. Please use cql-json or cql2-json.', filterlang;\n END IF;\n\n IF j ? 'query' AND j ? 'filter' THEN\n RAISE EXCEPTION 'Can only use either query or filter at one time.';\n END IF;\n\n IF j ? 'query' THEN\n filter := query_to_cql2(j->'query');\n ELSIF filterlang = 'cql-json' THEN\n filter := cql1_to_cql2(filter);\n END IF;\n RAISE NOTICE 'FILTER: %', filter;\n where_segments := where_segments || cql2_query(filter);\n IF cardinality(where_segments) < 1 THEN\n RETURN ' TRUE ';\n END IF;\n\n _where := array_to_string(array_remove(where_segments, NULL), ' AND ');\n\n IF _where IS NULL OR BTRIM(_where) = '' THEN\n RETURN ' TRUE ';\n END IF;\n RETURN _where;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION parse_sort_dir(_dir text, reverse boolean default false) RETURNS text AS $$\n WITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n ) SELECT\n CASE\n WHEN NOT reverse THEN d\n WHEN d = 'ASC' THEN 'DESC'\n WHEN d = 'DESC' THEN 'ASC'\n END\n FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION sort_dir_to_op(_dir text, prev boolean default false) RETURNS text AS $$\n WITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n ) SELECT\n CASE\n WHEN d = 'ASC' AND prev THEN '<='\n WHEN d = 'DESC' AND prev THEN '>='\n WHEN d = 'ASC' THEN '>='\n WHEN d = 'DESC' THEN '<='\n END\n FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION sort_sqlorderby(\n _search jsonb DEFAULT NULL,\n reverse boolean DEFAULT FALSE\n) RETURNS text AS $$\n WITH sortby AS (\n SELECT coalesce(_search->'sortby','[{\"field\":\"datetime\", \"direction\":\"desc\"}]') as sort\n ), withid AS (\n SELECT CASE\n WHEN sort @? '$[*] ? (@.field == \"id\")' THEN sort\n ELSE sort || '[{\"field\":\"id\", \"direction\":\"desc\"}]'::jsonb\n END as sort\n FROM sortby\n ), withid_rows AS (\n SELECT jsonb_array_elements(sort) as value FROM withid\n ),sorts AS (\n SELECT\n coalesce(\n (queryable(value->>'field')).expression\n ) as key,\n parse_sort_dir(value->>'direction', reverse) as dir\n FROM withid_rows\n )\n SELECT array_to_string(\n array_agg(concat(key, ' ', dir)),\n ', '\n ) FROM sorts;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_sort_dir(sort_item jsonb) RETURNS text AS $$\n SELECT CASE WHEN sort_item->>'direction' ILIKE 'desc%' THEN 'DESC' ELSE 'ASC' END;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION get_token_val_str(\n _field text,\n _item items\n) RETURNS text AS $$\nDECLARE\n q text;\n literal text;\nBEGIN\n q := format($q$ SELECT quote_literal(%s) FROM (SELECT $1.*) as r;$q$, _field);\n EXECUTE q INTO literal USING _item;\n RETURN literal;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n\nCREATE OR REPLACE FUNCTION get_token_record(IN _token text, OUT prev BOOLEAN, OUT item items) RETURNS RECORD AS $$\nDECLARE\n _itemid text := _token;\n _collectionid text;\nBEGIN\n IF _token IS NULL THEN\n RETURN;\n END IF;\n RAISE NOTICE 'Looking for token: %', _token;\n prev := FALSE;\n IF _token ILIKE 'prev:%' THEN\n _itemid := replace(_token, 'prev:','');\n prev := TRUE;\n ELSIF _token ILIKE 'next:%' THEN\n _itemid := replace(_token, 'next:', '');\n END IF;\n SELECT id INTO _collectionid FROM collections WHERE _itemid LIKE concat(id,':%');\n IF FOUND THEN\n _itemid := replace(_itemid, concat(_collectionid,':'), '');\n SELECT * INTO item FROM items WHERE id=_itemid AND collection=_collectionid;\n ELSE\n SELECT * INTO item FROM items WHERE id=_itemid;\n END IF;\n IF item IS NULL THEN\n RAISE EXCEPTION 'Could not find item using token: % item: % collection: %', _token, _itemid, _collectionid;\n END IF;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\n\nCREATE OR REPLACE FUNCTION get_token_filter(\n _sortby jsonb DEFAULT '[{\"field\":\"datetime\",\"direction\":\"desc\"}]'::jsonb,\n token_item items DEFAULT NULL,\n prev boolean DEFAULT FALSE,\n inclusive boolean DEFAULT FALSE\n) RETURNS text AS $$\nDECLARE\n ltop text := '<';\n gtop text := '>';\n dir text;\n sort record;\n orfilter text := '';\n orfilters text[] := '{}'::text[];\n andfilters text[] := '{}'::text[];\n output text;\n token_where text;\nBEGIN\n IF _sortby IS NULL OR _sortby = '[]'::jsonb THEN\n _sortby := '[{\"field\":\"datetime\",\"direction\":\"desc\"}]'::jsonb;\n END IF;\n _sortby := _sortby || jsonb_build_object('field','id','direction',_sortby->0->>'direction');\n RAISE NOTICE 'Getting Token Filter. % %', _sortby, token_item;\n IF inclusive THEN\n orfilters := orfilters || format('( id=%L AND collection=%L )' , token_item.id, token_item.collection);\n END IF;\n\n FOR sort IN\n WITH s1 AS (\n SELECT\n _row,\n (queryable(value->>'field')).expression as _field,\n (value->>'field' = 'id') as _isid,\n get_sort_dir(value) as _dir\n FROM jsonb_array_elements(_sortby)\n WITH ORDINALITY AS t(value, _row)\n )\n SELECT\n _row,\n _field,\n _dir,\n get_token_val_str(_field, token_item) as _val\n FROM s1\n WHERE _row <= (SELECT min(_row) FROM s1 WHERE _isid)\n LOOP\n orfilter := NULL;\n RAISE NOTICE 'SORT: %', sort;\n IF sort._val IS NOT NULL AND ((prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC')) THEN\n orfilter := format($f$(\n (%s %s %s) OR (%s IS NULL)\n )$f$,\n sort._field,\n ltop,\n sort._val,\n sort._val\n );\n ELSIF sort._val IS NULL AND ((prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC')) THEN\n RAISE NOTICE '< but null';\n orfilter := format('%s IS NOT NULL', sort._field);\n ELSIF sort._val IS NULL THEN\n RAISE NOTICE '> but null';\n ELSE\n orfilter := format($f$(\n (%s %s %s) OR (%s IS NULL)\n )$f$,\n sort._field,\n gtop,\n sort._val,\n sort._field\n );\n END IF;\n RAISE NOTICE 'ORFILTER: %', orfilter;\n\n IF orfilter IS NOT NULL THEN\n IF sort._row = 1 THEN\n orfilters := orfilters || orfilter;\n ELSE\n orfilters := orfilters || format('(%s AND %s)', array_to_string(andfilters, ' AND '), orfilter);\n END IF;\n END IF;\n IF sort._val IS NOT NULL THEN\n andfilters := andfilters || format('%s = %s', sort._field, sort._val);\n ELSE\n andfilters := andfilters || format('%s IS NULL', sort._field);\n END IF;\n END LOOP;\n\n output := array_to_string(orfilters, ' OR ');\n\n token_where := concat('(',coalesce(output,'true'),')');\n IF trim(token_where) = '' THEN\n token_where := NULL;\n END IF;\n RAISE NOTICE 'TOKEN_WHERE: %',token_where;\n RETURN token_where;\n END;\n$$ LANGUAGE PLPGSQL SET transform_null_equals TO TRUE\n;\n\nCREATE OR REPLACE FUNCTION search_tohash(jsonb) RETURNS jsonb AS $$\n SELECT $1 - '{token,limit,context,includes,excludes}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION search_hash(jsonb, jsonb) RETURNS text AS $$\n SELECT md5(concat(search_tohash($1)::text,$2::text));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TABLE IF NOT EXISTS searches(\n hash text GENERATED ALWAYS AS (search_hash(search, metadata)) STORED PRIMARY KEY,\n search jsonb NOT NULL,\n _where text,\n orderby text,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n metadata jsonb DEFAULT '{}'::jsonb NOT NULL\n);\n\nCREATE TABLE IF NOT EXISTS search_wheres(\n id bigint generated always as identity primary key,\n _where text NOT NULL,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n statslastupdated timestamptz,\n estimated_count bigint,\n estimated_cost float,\n time_to_estimate float,\n total_count bigint,\n time_to_count float,\n partitions text[]\n);\n\nCREATE INDEX IF NOT EXISTS search_wheres_partitions ON search_wheres USING GIN (partitions);\nCREATE UNIQUE INDEX IF NOT EXISTS search_wheres_where ON search_wheres ((md5(_where)));\n\nCREATE OR REPLACE FUNCTION where_stats(inwhere text, updatestats boolean default false, conf jsonb default null) RETURNS search_wheres AS $$\nDECLARE\n t timestamptz;\n i interval;\n explain_json jsonb;\n partitions text[];\n sw search_wheres%ROWTYPE;\n inwhere_hash text := md5(inwhere);\n _context text := lower(context(conf));\n _stats_ttl interval := context_stats_ttl(conf);\n _estimated_cost float := context_estimated_cost(conf);\n _estimated_count int := context_estimated_count(conf);\nBEGIN\n IF _context = 'off' THEN\n sw._where := inwhere;\n return sw;\n END IF;\n\n SELECT * INTO sw FROM search_wheres WHERE md5(_where)=inwhere_hash FOR UPDATE;\n\n -- Update statistics if explicitly set, if statistics do not exist, or statistics ttl has expired\n IF NOT updatestats THEN\n RAISE NOTICE 'Checking if update is needed for: % .', inwhere;\n RAISE NOTICE 'Stats Last Updated: %', sw.statslastupdated;\n RAISE NOTICE 'TTL: %, Age: %', _stats_ttl, now() - sw.statslastupdated;\n RAISE NOTICE 'Context: %, Existing Total: %', _context, sw.total_count;\n IF\n sw.statslastupdated IS NULL\n OR (now() - sw.statslastupdated) > _stats_ttl\n OR (context(conf) != 'off' AND sw.total_count IS NULL)\n THEN\n updatestats := TRUE;\n END IF;\n END IF;\n\n sw._where := inwhere;\n sw.lastused := now();\n sw.usecount := coalesce(sw.usecount,0) + 1;\n\n IF NOT updatestats THEN\n UPDATE search_wheres SET\n lastused = sw.lastused,\n usecount = sw.usecount\n WHERE md5(_where) = inwhere_hash\n RETURNING * INTO sw\n ;\n RETURN sw;\n END IF;\n\n -- Use explain to get estimated count/cost and a list of the partitions that would be hit by the query\n t := clock_timestamp();\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s', inwhere)\n INTO explain_json;\n RAISE NOTICE 'Time for just the explain: %', clock_timestamp() - t;\n i := clock_timestamp() - t;\n\n sw.statslastupdated := now();\n sw.estimated_count := explain_json->0->'Plan'->'Plan Rows';\n sw.estimated_cost := explain_json->0->'Plan'->'Total Cost';\n sw.time_to_estimate := extract(epoch from i);\n\n RAISE NOTICE 'ESTIMATED_COUNT: % < %', sw.estimated_count, _estimated_count;\n RAISE NOTICE 'ESTIMATED_COST: % < %', sw.estimated_cost, _estimated_cost;\n\n -- Do a full count of rows if context is set to on or if auto is set and estimates are low enough\n IF\n _context = 'on'\n OR\n ( _context = 'auto' AND\n (\n sw.estimated_count < _estimated_count\n AND\n sw.estimated_cost < _estimated_cost\n )\n )\n THEN\n t := clock_timestamp();\n RAISE NOTICE 'Calculating actual count...';\n EXECUTE format(\n 'SELECT count(*) FROM items WHERE %s',\n inwhere\n ) INTO sw.total_count;\n i := clock_timestamp() - t;\n RAISE NOTICE 'Actual Count: % -- %', sw.total_count, i;\n sw.time_to_count := extract(epoch FROM i);\n ELSE\n sw.total_count := NULL;\n sw.time_to_count := NULL;\n END IF;\n\n\n INSERT INTO search_wheres\n (_where, lastused, usecount, statslastupdated, estimated_count, estimated_cost, time_to_estimate, partitions, total_count, time_to_count)\n SELECT sw._where, sw.lastused, sw.usecount, sw.statslastupdated, sw.estimated_count, sw.estimated_cost, sw.time_to_estimate, sw.partitions, sw.total_count, sw.time_to_count\n ON CONFLICT ((md5(_where)))\n DO UPDATE\n SET\n lastused = sw.lastused,\n usecount = sw.usecount,\n statslastupdated = sw.statslastupdated,\n estimated_count = sw.estimated_count,\n estimated_cost = sw.estimated_cost,\n time_to_estimate = sw.time_to_estimate,\n total_count = sw.total_count,\n time_to_count = sw.time_to_count\n ;\n RETURN sw;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION search_query(\n _search jsonb = '{}'::jsonb,\n updatestats boolean = false,\n _metadata jsonb = '{}'::jsonb\n) RETURNS searches AS $$\nDECLARE\n search searches%ROWTYPE;\n pexplain jsonb;\n t timestamptz;\n i interval;\nBEGIN\n SELECT * INTO search FROM searches\n WHERE hash=search_hash(_search, _metadata) FOR UPDATE;\n\n -- Calculate the where clause if not already calculated\n IF search._where IS NULL THEN\n search._where := stac_search_to_where(_search);\n END IF;\n\n -- Calculate the order by clause if not already calculated\n IF search.orderby IS NULL THEN\n search.orderby := sort_sqlorderby(_search);\n END IF;\n\n PERFORM where_stats(search._where, updatestats, _search->'conf');\n\n search.lastused := now();\n search.usecount := coalesce(search.usecount, 0) + 1;\n INSERT INTO searches (search, _where, orderby, lastused, usecount, metadata)\n VALUES (_search, search._where, search.orderby, search.lastused, search.usecount, _metadata)\n ON CONFLICT (hash) DO\n UPDATE SET\n _where = EXCLUDED._where,\n orderby = EXCLUDED.orderby,\n lastused = EXCLUDED.lastused,\n usecount = EXCLUDED.usecount,\n metadata = EXCLUDED.metadata\n RETURNING * INTO search\n ;\n RETURN search;\n\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE OR REPLACE FUNCTION search_rows(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN partitions text[] DEFAULT NULL,\n IN _limit int DEFAULT 10\n) RETURNS SETOF items AS $$\nDECLARE\n base_query text;\n query text;\n sdate timestamptz;\n edate timestamptz;\n n int;\n records_left int := _limit;\n timer timestamptz := clock_timestamp();\n full_timer timestamptz := clock_timestamp();\nBEGIN\nIF _where IS NULL OR trim(_where) = '' THEN\n _where = ' TRUE ';\nEND IF;\nRAISE NOTICE 'Getting chunks for % %', _where, _orderby;\n\nbase_query := $q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n LIMIT %L\n$q$;\n\nIF _orderby ILIKE 'datetime d%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 DESC LOOP\n RAISE NOTICE 'Running Query for % to %. %', sdate, edate, age_ms(full_timer);\n query := format(\n base_query,\n sdate,\n edate,\n _where,\n _orderby,\n records_left\n );\n RAISE LOG 'QUERY: %', query;\n timer := clock_timestamp();\n RETURN QUERY EXECUTE query;\n\n GET DIAGNOSTICS n = ROW_COUNT;\n records_left := records_left - n;\n RAISE NOTICE 'Returned %/% Rows From % to %. % to go. Time: %ms', n, _limit, sdate, edate, records_left, age_ms(timer);\n timer := clock_timestamp();\n IF records_left <= 0 THEN\n RAISE NOTICE 'SEARCH_ROWS TOOK %ms', age_ms(full_timer);\n RETURN;\n END IF;\n END LOOP;\nELSIF _orderby ILIKE 'datetime a%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 ASC LOOP\n RAISE NOTICE 'Running Query for % to %. %', sdate, edate, age_ms(full_timer);\n query := format(\n base_query,\n sdate,\n edate,\n _where,\n _orderby,\n records_left\n );\n RAISE LOG 'QUERY: %', query;\n timer := clock_timestamp();\n RETURN QUERY EXECUTE query;\n\n GET DIAGNOSTICS n = ROW_COUNT;\n records_left := records_left - n;\n RAISE NOTICE 'Returned %/% Rows From % to %. % to go. Time: %ms', n, _limit, sdate, edate, records_left, age_ms(timer);\n timer := clock_timestamp();\n IF records_left <= 0 THEN\n RAISE NOTICE 'SEARCH_ROWS TOOK %ms', age_ms(full_timer);\n RETURN;\n END IF;\n END LOOP;\nELSE\n query := format($q$\n SELECT * FROM items\n WHERE %s\n ORDER BY %s\n LIMIT %L\n $q$, _where, _orderby, _limit\n );\n RAISE LOG 'QUERY: %', query;\n timer := clock_timestamp();\n RETURN QUERY EXECUTE query;\n RAISE NOTICE 'FULL QUERY TOOK %ms', age_ms(timer);\nEND IF;\nRAISE NOTICE 'SEARCH_ROWS TOOK %ms', age_ms(full_timer);\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE UNLOGGED TABLE format_item_cache(\n id text,\n collection text,\n fields text,\n hydrated bool,\n output jsonb,\n lastused timestamptz DEFAULT now(),\n usecount int DEFAULT 1,\n timetoformat float,\n PRIMARY KEY (collection, id, fields, hydrated)\n);\nCREATE INDEX ON format_item_cache (lastused);\n\nCREATE OR REPLACE FUNCTION format_item(_item items, _fields jsonb DEFAULT '{}', _hydrated bool DEFAULT TRUE) RETURNS jsonb AS $$\nDECLARE\n cache bool := get_setting_bool('format_cache');\n _output jsonb := null;\n t timestamptz := clock_timestamp();\nBEGIN\n IF cache THEN\n SELECT output INTO _output FROM format_item_cache\n WHERE id=_item.id AND collection=_item.collection AND fields=_fields::text AND hydrated=_hydrated;\n END IF;\n IF _output IS NULL THEN\n IF _hydrated THEN\n _output := content_hydrate(_item, _fields);\n ELSE\n _output := content_nonhydrated(_item, _fields);\n END IF;\n END IF;\n IF cache THEN\n INSERT INTO format_item_cache (id, collection, fields, hydrated, output, timetoformat)\n VALUES (_item.id, _item.collection, _fields::text, _hydrated, _output, age_ms(t))\n ON CONFLICT(collection, id, fields, hydrated) DO\n UPDATE\n SET lastused=now(), usecount = format_item_cache.usecount + 1\n ;\n END IF;\n RETURN _output;\n\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION search(_search jsonb = '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n searches searches%ROWTYPE;\n _where text;\n orderby text;\n search_where search_wheres%ROWTYPE;\n total_count bigint;\n token record;\n token_prev boolean;\n token_item items%ROWTYPE;\n token_where text;\n full_where text;\n timer timestamptz := clock_timestamp();\n hydrate bool := NOT (_search->'conf'->>'nohydrate' IS NOT NULL AND (_search->'conf'->>'nohydrate')::boolean = true);\n prev text;\n next text;\n context jsonb;\n collection jsonb;\n out_records jsonb;\n out_len int;\n _limit int := coalesce((_search->>'limit')::int, 10);\n _querylimit int;\n _fields jsonb := coalesce(_search->'fields', '{}'::jsonb);\n has_prev boolean := FALSE;\n has_next boolean := FALSE;\nBEGIN\n searches := search_query(_search);\n _where := searches._where;\n orderby := searches.orderby;\n search_where := where_stats(_where);\n total_count := coalesce(search_where.total_count, search_where.estimated_count);\n RAISE NOTICE 'SEARCH:TOKEN: %', _search->>'token';\n token := get_token_record(_search->>'token');\n RAISE NOTICE '***TOKEN: %', token;\n _querylimit := _limit + 1;\n IF token IS NOT NULL THEN\n token_prev := token.prev;\n token_item := token.item;\n token_where := get_token_filter(_search->'sortby', token_item, token_prev, FALSE);\n RAISE LOG 'TOKEN_WHERE: %', token_where;\n IF token_prev THEN -- if we are using a prev token, we know has_next is true\n RAISE LOG 'There is a previous token, so automatically setting has_next to true';\n has_next := TRUE;\n orderby := sort_sqlorderby(_search, TRUE);\n ELSE\n RAISE LOG 'There is a next token, so automatically setting has_prev to true';\n has_prev := TRUE;\n\n END IF;\n ELSE -- if there was no token, we know there is no prev\n RAISE LOG 'There is no token, so we know there is no prev. setting has_prev to false';\n has_prev := FALSE;\n END IF;\n\n full_where := concat_ws(' AND ', _where, token_where);\n RAISE NOTICE 'FULL WHERE CLAUSE: %', full_where;\n RAISE NOTICE 'Time to get counts and build query %', age_ms(timer);\n timer := clock_timestamp();\n\n IF hydrate THEN\n RAISE NOTICE 'Getting hydrated data.';\n ELSE\n RAISE NOTICE 'Getting non-hydrated data.';\n END IF;\n RAISE NOTICE 'CACHE SET TO %', get_setting_bool('format_cache');\n RAISE NOTICE 'Time to set hydration/formatting %', age_ms(timer);\n timer := clock_timestamp();\n SELECT jsonb_agg(format_item(i, _fields, hydrate)) INTO out_records\n FROM search_rows(\n full_where,\n orderby,\n search_where.partitions,\n _querylimit\n ) as i;\n\n RAISE NOTICE 'Time to fetch rows %', age_ms(timer);\n timer := clock_timestamp();\n\n\n IF token_prev THEN\n out_records := flip_jsonb_array(out_records);\n END IF;\n\n RAISE NOTICE 'Query returned % records.', jsonb_array_length(out_records);\n RAISE LOG 'TOKEN: % %', token_item.id, token_item.collection;\n RAISE LOG 'RECORD_1: % %', out_records->0->>'id', out_records->0->>'collection';\n RAISE LOG 'RECORD-1: % %', out_records->-1->>'id', out_records->-1->>'collection';\n\n -- REMOVE records that were from our token\n IF out_records->0->>'id' = token_item.id AND out_records->0->>'collection' = token_item.collection THEN\n out_records := out_records - 0;\n ELSIF out_records->-1->>'id' = token_item.id AND out_records->-1->>'collection' = token_item.collection THEN\n out_records := out_records - -1;\n END IF;\n\n out_len := jsonb_array_length(out_records);\n\n IF out_len = _limit + 1 THEN\n IF token_prev THEN\n has_prev := TRUE;\n out_records := out_records - 0;\n ELSE\n has_next := TRUE;\n out_records := out_records - -1;\n END IF;\n END IF;\n\n IF has_next THEN\n next := concat(out_records->-1->>'collection', ':', out_records->-1->>'id');\n RAISE NOTICE 'HAS NEXT | %', next;\n END IF;\n\n IF has_prev THEN\n prev := concat(out_records->0->>'collection', ':', out_records->0->>'id');\n RAISE NOTICE 'HAS PREV | %', prev;\n END IF;\n\n RAISE NOTICE 'Time to get prev/next %', age_ms(timer);\n timer := clock_timestamp();\n\n IF context(_search->'conf') != 'off' THEN\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'matched', total_count,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\n ELSE\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\n END IF;\n\n collection := jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb),\n 'next', next,\n 'prev', prev,\n 'context', context\n );\n\n RAISE NOTICE 'Time to build final json %', age_ms(timer);\n timer := clock_timestamp();\n\n RAISE NOTICE 'Total Time: %', age_ms(current_timestamp);\n RAISE NOTICE 'RETURNING % records. NEXT: %. PREV: %', collection->'context'->>'returned', collection->>'next', collection->>'prev';\n RETURN collection;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION search_cursor(_search jsonb = '{}'::jsonb) RETURNS refcursor AS $$\nDECLARE\n curs refcursor;\n searches searches%ROWTYPE;\n _where text;\n _orderby text;\n q text;\n\nBEGIN\n searches := search_query(_search);\n _where := searches._where;\n _orderby := searches.orderby;\n\n OPEN curs FOR\n WITH p AS (\n SELECT * FROM partition_queries(_where, _orderby) p\n )\n SELECT\n CASE WHEN EXISTS (SELECT 1 FROM p) THEN\n (SELECT format($q$\n SELECT * FROM (\n %s\n ) total\n $q$,\n string_agg(\n format($q$ SELECT * FROM ( %s ) AS sub $q$, p),\n '\n UNION ALL\n '\n )\n ))\n ELSE NULL\n END FROM p;\n RETURN curs;\nEND;\n$$ LANGUAGE PLPGSQL;\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION tileenvelope(zoom int, x int, y int) RETURNS geometry AS $$\nWITH t AS (\n SELECT\n 20037508.3427892 as merc_max,\n -20037508.3427892 as merc_min,\n (2 * 20037508.3427892) / (2 ^ zoom) as tile_size\n)\nSELECT st_makeenvelope(\n merc_min + (tile_size * x),\n merc_max - (tile_size * (y + 1)),\n merc_min + (tile_size * (x + 1)),\n merc_max - (tile_size * y),\n 3857\n) FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;DROP FUNCTION IF EXISTS mercgrid;\n\n\nCREATE OR REPLACE FUNCTION ftime() RETURNS interval as $$\nSELECT age(clock_timestamp(), transaction_timestamp());\n$$ LANGUAGE SQL;\nSET SEARCH_PATH to pgstac, public;\n\nDROP FUNCTION IF EXISTS geometrysearch;\nCREATE OR REPLACE FUNCTION geometrysearch(\n IN geom geometry,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE, -- Return as soon as the passed in geometry is full covered\n IN skipcovered boolean DEFAULT TRUE -- Skip any items that would show up completely under the previous items\n) RETURNS jsonb AS $$\nDECLARE\n search searches%ROWTYPE;\n curs refcursor;\n _where text;\n query text;\n iter_record items%ROWTYPE;\n out_records jsonb := '{}'::jsonb[];\n exit_flag boolean := FALSE;\n counter int := 1;\n scancounter int := 1;\n remaining_limit int := _scanlimit;\n tilearea float;\n unionedgeom geometry;\n clippedgeom geometry;\n unionedgeom_area float := 0;\n prev_area float := 0;\n excludes text[];\n includes text[];\n\nBEGIN\n DROP TABLE IF EXISTS pgstac_results;\n CREATE TEMP TABLE pgstac_results (content jsonb) ON COMMIT DROP;\n\n -- If skipcovered is true then you will always want to exit when the passed in geometry is full\n IF skipcovered THEN\n exitwhenfull := TRUE;\n END IF;\n\n SELECT * INTO search FROM searches WHERE hash=queryhash;\n\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Search with Query Hash % Not Found', queryhash;\n END IF;\n\n tilearea := st_area(geom);\n _where := format('%s AND st_intersects(geometry, %L::geometry)', search._where, geom);\n\n\n FOR query IN SELECT * FROM partition_queries(_where, search.orderby) LOOP\n query := format('%s LIMIT %L', query, remaining_limit);\n RAISE NOTICE '%', query;\n OPEN curs FOR EXECUTE query;\n LOOP\n FETCH curs INTO iter_record;\n EXIT WHEN NOT FOUND;\n IF exitwhenfull OR skipcovered THEN -- If we are not using exitwhenfull or skipcovered, we do not need to do expensive geometry operations\n clippedgeom := st_intersection(geom, iter_record.geometry);\n\n IF unionedgeom IS NULL THEN\n unionedgeom := clippedgeom;\n ELSE\n unionedgeom := st_union(unionedgeom, clippedgeom);\n END IF;\n\n unionedgeom_area := st_area(unionedgeom);\n\n IF skipcovered AND prev_area = unionedgeom_area THEN\n scancounter := scancounter + 1;\n CONTINUE;\n END IF;\n\n prev_area := unionedgeom_area;\n\n RAISE NOTICE '% % % %', unionedgeom_area/tilearea, counter, scancounter, ftime();\n END IF;\n RAISE NOTICE '% %', iter_record, content_hydrate(iter_record, fields);\n INSERT INTO pgstac_results (content) VALUES (content_hydrate(iter_record, fields));\n\n IF counter >= _limit\n OR scancounter > _scanlimit\n OR ftime() > _timelimit\n OR (exitwhenfull AND unionedgeom_area >= tilearea)\n THEN\n exit_flag := TRUE;\n EXIT;\n END IF;\n counter := counter + 1;\n scancounter := scancounter + 1;\n\n END LOOP;\n CLOSE curs;\n EXIT WHEN exit_flag;\n remaining_limit := _scanlimit - scancounter;\n END LOOP;\n\n SELECT jsonb_agg(content) INTO out_records FROM pgstac_results WHERE content IS NOT NULL;\n\n RETURN jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb)\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDROP FUNCTION IF EXISTS geojsonsearch;\nCREATE OR REPLACE FUNCTION geojsonsearch(\n IN geojson jsonb,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_geomfromgeojson(geojson),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS xyzsearch;\nCREATE OR REPLACE FUNCTION xyzsearch(\n IN _x int,\n IN _y int,\n IN _z int,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_transform(tileenvelope(_z, _x, _y), 4326),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE PROCEDURE analyze_items() AS $$\nDECLARE\n q text;\n timeout_ts timestamptz;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n SELECT format('ANALYZE (VERBOSE, SKIP_LOCKED) %I;', relname) INTO q\n FROM pg_stat_user_tables\n WHERE relname like '_item%' AND (n_mod_since_analyze>0 OR last_analyze IS NULL) LIMIT 1;\n IF NOT FOUND THEN\n EXIT;\n END IF;\n RAISE NOTICE '%', q;\n EXECUTE q;\n COMMIT;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE PROCEDURE validate_constraints() AS $$\nDECLARE\n q text;\nBEGIN\n FOR q IN\n SELECT\n FORMAT(\n 'ALTER TABLE %I.%I VALIDATE CONSTRAINT %I;',\n nsp.nspname,\n cls.relname,\n con.conname\n )\n\n FROM pg_constraint AS con\n JOIN pg_class AS cls\n ON con.conrelid = cls.oid\n JOIN pg_namespace AS nsp\n ON cls.relnamespace = nsp.oid\n WHERE convalidated = FALSE AND contype in ('c','f')\n AND nsp.nspname = 'pgstac'\n LOOP\n RAISE NOTICE '%', q;\n PERFORM run_or_queue(q);\n COMMIT;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION collection_extent(_collection text, runupdate boolean default false) RETURNS jsonb AS $$\nDECLARE\n geom_extent geometry;\n mind timestamptz;\n maxd timestamptz;\n extent jsonb;\nBEGIN\n IF runupdate THEN\n PERFORM update_partition_stats_q(partition)\n FROM partitions_view WHERE collection=_collection;\n END IF;\n SELECT\n min(lower(dtrange)),\n max(upper(edtrange)),\n st_extent(spatial)\n INTO\n mind,\n maxd,\n geom_extent\n FROM partitions_view\n WHERE collection=_collection;\n\n IF geom_extent IS NOT NULL AND mind IS NOT NULL AND maxd IS NOT NULL THEN\n extent := jsonb_build_object(\n 'extent', jsonb_build_object(\n 'spatial', jsonb_build_object(\n 'bbox', to_jsonb(array[array[st_xmin(geom_extent), st_ymin(geom_extent), st_xmax(geom_extent), st_ymax(geom_extent)]])\n ),\n 'temporal', jsonb_build_object(\n 'interval', to_jsonb(array[array[mind, maxd]])\n )\n )\n );\n RETURN extent;\n END IF;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('id', '{\"title\": \"Item ID\",\"description\": \"Item identifier\",\"$ref\": \"https://schemas.stacspec.org/v1.0.0/item-spec/json-schema/item.json#/definitions/core/allOf/2/properties/id\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('geometry', '{\"title\": \"Item Geometry\",\"description\": \"Item Geometry\",\"$ref\": \"https://geojson.org/schema/Feature.json\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('datetime','{\"description\": \"Datetime\",\"type\": \"string\",\"title\": \"Acquired\",\"format\": \"date-time\",\"pattern\": \"(\\\\+00:00|Z)$\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('eo:cloud_cover','{\"$ref\": \"https://stac-extensions.github.io/eo/v1.0.0/schema.json#/definitions/fieldsproperties/eo:cloud_cover\"}','to_int','BTREE');\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDELETE FROM queryables a USING queryables b\n WHERE a.name = b.name AND a.collection_ids IS NOT DISTINCT FROM b.collection_ids AND a.id > b.id;\n\n\nINSERT INTO pgstac_settings (name, value) VALUES\n ('context', 'off'),\n ('context_estimated_count', '100000'),\n ('context_estimated_cost', '100000'),\n ('context_stats_ttl', '1 day'),\n ('default_filter_lang', 'cql2-json'),\n ('additional_properties', 'true'),\n ('use_queue', 'false'),\n ('queue_timeout', '10 minutes'),\n ('update_collection_extent', 'false'),\n ('format_cache', 'false')\nON CONFLICT DO NOTHING\n;\n\nALTER FUNCTION to_text COST 5000;\nALTER FUNCTION to_float COST 5000;\nALTER FUNCTION to_int COST 5000;\nALTER FUNCTION to_tstz COST 5000;\nALTER FUNCTION to_text_array COST 5000;\n\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nGRANT ALL ON SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac to pgstac_admin;\n\n-- pgstac_read role limited to using function apis\nGRANT EXECUTE ON FUNCTION search TO pgstac_read;\nGRANT EXECUTE ON FUNCTION search_query TO pgstac_read;\nGRANT EXECUTE ON FUNCTION item_by_id TO pgstac_read;\nGRANT EXECUTE ON FUNCTION get_item TO pgstac_read;\nGRANT SELECT ON ALL TABLES IN SCHEMA pgstac TO pgstac_read;\n\n\nGRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_ingest;\nGRANT USAGE ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_ingest;\n\nSELECT update_partition_stats_q(partition) FROM partitions_view;\nSELECT set_version('0.7.6');\n" }, { "path": "src/pgstac/migrations/pgstac.0.7.7-0.7.8.sql", "content": "RESET ROLE;\nDO $$\nDECLARE\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='postgis') THEN\n CREATE EXTENSION IF NOT EXISTS postgis;\n END IF;\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='btree_gist') THEN\n CREATE EXTENSION IF NOT EXISTS btree_gist;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_admin;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_read;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_ingest;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n\nGRANT pgstac_admin TO current_user;\n\n-- Function to make sure pgstac_admin is the owner of items\nCREATE OR REPLACE FUNCTION pgstac_admin_owns() RETURNS VOID AS $$\nDECLARE\n f RECORD;\nBEGIN\n FOR f IN (\n SELECT\n concat(\n oid::regproc::text,\n '(',\n coalesce(pg_get_function_identity_arguments(oid),''),\n ')'\n ) AS name,\n CASE prokind WHEN 'f' THEN 'FUNCTION' WHEN 'p' THEN 'PROCEDURE' WHEN 'a' THEN 'AGGREGATE' END as typ\n FROM pg_proc\n WHERE\n pronamespace=to_regnamespace('pgstac')\n AND proowner != to_regrole('pgstac_admin')\n AND proname NOT LIKE 'pg_stat%'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n FOR f IN (\n SELECT\n oid::regclass::text as name,\n CASE relkind\n WHEN 'i' THEN 'INDEX'\n WHEN 'I' THEN 'INDEX'\n WHEN 'p' THEN 'TABLE'\n WHEN 'r' THEN 'TABLE'\n WHEN 'v' THEN 'VIEW'\n WHEN 'S' THEN 'SEQUENCE'\n ELSE NULL\n END as typ\n FROM pg_class\n WHERE relnamespace=to_regnamespace('pgstac') and relowner != to_regrole('pgstac_admin') AND relkind IN ('r','p','v','S') AND relname NOT LIKE 'pg_stat'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\nSELECT pgstac_admin_owns();\n\nCREATE SCHEMA IF NOT EXISTS pgstac AUTHORIZATION pgstac_admin;\n\nGRANT ALL ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_admin;\n\nALTER ROLE pgstac_admin SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_read SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_ingest SET SEARCH_PATH TO pgstac, public;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\n\nGRANT pgstac_read TO pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\n\nSET ROLE pgstac_admin;\n\nSET SEARCH_PATH TO pgstac, public;\n\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS analyze_items;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS validate_constraints;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\nSET client_min_messages TO WARNING;\nSET SEARCH_PATH to pgstac, public;\n-- BEGIN migra calculated SQL\nset check_function_bodies = off;\n\nCREATE OR REPLACE FUNCTION pgstac.search_query(_search jsonb DEFAULT '{}'::jsonb, updatestats boolean DEFAULT false, _metadata jsonb DEFAULT '{}'::jsonb)\n RETURNS pgstac.searches\n LANGUAGE plpgsql\n SECURITY DEFINER\nAS $function$\nDECLARE\n search searches%ROWTYPE;\n pexplain jsonb;\n t timestamptz;\n i interval;\n _hash text := search_hash(_search, _metadata);\n doupdate boolean := FALSE;\n insertfound boolean := FALSE;\nBEGIN\n SELECT * INTO search FROM searches\n WHERE hash=_hash;\n\n search.hash := _hash;\n\n -- Calculate the where clause if not already calculated\n IF search._where IS NULL THEN\n search._where := stac_search_to_where(_search);\n ELSE\n doupdate := TRUE;\n END IF;\n\n -- Calculate the order by clause if not already calculated\n IF search.orderby IS NULL THEN\n search.orderby := sort_sqlorderby(_search);\n ELSE\n doupdate := TRUE;\n END IF;\n\n PERFORM where_stats(search._where, updatestats, _search->'conf');\n\n IF NOT doupdate THEN\n INSERT INTO searches (search, _where, orderby, lastused, usecount, metadata)\n VALUES (_search, search._where, search.orderby, clock_timestamp(), 1, _metadata)\n ON CONFLICT (hash) DO NOTHING RETURNING * INTO search;\n IF FOUND THEN\n RETURN search;\n END IF;\n END IF;\n\n UPDATE searches\n SET\n lastused=clock_timestamp(),\n usecount=usecount+1\n WHERE hash=(\n SELECT hash FROM searches\n WHERE hash=_hash\n FOR UPDATE SKIP LOCKED\n );\n IF NOT FOUND THEN\n RAISE NOTICE 'Did not update stats for % due to lock. (This is generally OK)', _search;\n END IF;\n\n RETURN search;\n\nEND;\n$function$\n;\n\n\n\n-- END migra calculated SQL\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('id', '{\"title\": \"Item ID\",\"description\": \"Item identifier\",\"$ref\": \"https://schemas.stacspec.org/v1.0.0/item-spec/json-schema/item.json#/definitions/core/allOf/2/properties/id\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('geometry', '{\"title\": \"Item Geometry\",\"description\": \"Item Geometry\",\"$ref\": \"https://geojson.org/schema/Feature.json\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('datetime','{\"description\": \"Datetime\",\"type\": \"string\",\"title\": \"Acquired\",\"format\": \"date-time\",\"pattern\": \"(\\\\+00:00|Z)$\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('eo:cloud_cover','{\"$ref\": \"https://stac-extensions.github.io/eo/v1.0.0/schema.json#/definitions/fieldsproperties/eo:cloud_cover\"}','to_int','BTREE');\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDELETE FROM queryables a USING queryables b\n WHERE a.name = b.name AND a.collection_ids IS NOT DISTINCT FROM b.collection_ids AND a.id > b.id;\n\n\nINSERT INTO pgstac_settings (name, value) VALUES\n ('context', 'off'),\n ('context_estimated_count', '100000'),\n ('context_estimated_cost', '100000'),\n ('context_stats_ttl', '1 day'),\n ('default_filter_lang', 'cql2-json'),\n ('additional_properties', 'true'),\n ('use_queue', 'false'),\n ('queue_timeout', '10 minutes'),\n ('update_collection_extent', 'false'),\n ('format_cache', 'false')\nON CONFLICT DO NOTHING\n;\n\nALTER FUNCTION to_text COST 5000;\nALTER FUNCTION to_float COST 5000;\nALTER FUNCTION to_int COST 5000;\nALTER FUNCTION to_tstz COST 5000;\nALTER FUNCTION to_text_array COST 5000;\n\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nGRANT ALL ON SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac to pgstac_admin;\n\n-- pgstac_read role limited to using function apis\nGRANT EXECUTE ON FUNCTION search TO pgstac_read;\nGRANT EXECUTE ON FUNCTION search_query TO pgstac_read;\nGRANT EXECUTE ON FUNCTION item_by_id TO pgstac_read;\nGRANT EXECUTE ON FUNCTION get_item TO pgstac_read;\nGRANT SELECT ON ALL TABLES IN SCHEMA pgstac TO pgstac_read;\n\n\nGRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_ingest;\nGRANT USAGE ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_ingest;\n\nSELECT update_partition_stats_q(partition) FROM partitions_view;\nSELECT set_version('0.7.8');\n" }, { "path": "src/pgstac/migrations/pgstac.0.7.7.sql", "content": "RESET ROLE;\nDO $$\nDECLARE\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='postgis') THEN\n CREATE EXTENSION IF NOT EXISTS postgis;\n END IF;\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='btree_gist') THEN\n CREATE EXTENSION IF NOT EXISTS btree_gist;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_admin;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_read;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_ingest;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n\nGRANT pgstac_admin TO current_user;\n\n-- Function to make sure pgstac_admin is the owner of items\nCREATE OR REPLACE FUNCTION pgstac_admin_owns() RETURNS VOID AS $$\nDECLARE\n f RECORD;\nBEGIN\n FOR f IN (\n SELECT\n concat(\n oid::regproc::text,\n '(',\n coalesce(pg_get_function_identity_arguments(oid),''),\n ')'\n ) AS name,\n CASE prokind WHEN 'f' THEN 'FUNCTION' WHEN 'p' THEN 'PROCEDURE' WHEN 'a' THEN 'AGGREGATE' END as typ\n FROM pg_proc\n WHERE\n pronamespace=to_regnamespace('pgstac')\n AND proowner != to_regrole('pgstac_admin')\n AND proname NOT LIKE 'pg_stat%'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n FOR f IN (\n SELECT\n oid::regclass::text as name,\n CASE relkind\n WHEN 'i' THEN 'INDEX'\n WHEN 'I' THEN 'INDEX'\n WHEN 'p' THEN 'TABLE'\n WHEN 'r' THEN 'TABLE'\n WHEN 'v' THEN 'VIEW'\n WHEN 'S' THEN 'SEQUENCE'\n ELSE NULL\n END as typ\n FROM pg_class\n WHERE relnamespace=to_regnamespace('pgstac') and relowner != to_regrole('pgstac_admin') AND relkind IN ('r','p','v','S') AND relname NOT LIKE 'pg_stat'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\nSELECT pgstac_admin_owns();\n\nCREATE SCHEMA IF NOT EXISTS pgstac AUTHORIZATION pgstac_admin;\n\nGRANT ALL ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_admin;\n\nALTER ROLE pgstac_admin SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_read SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_ingest SET SEARCH_PATH TO pgstac, public;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\n\nGRANT pgstac_read TO pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\n\nSET ROLE pgstac_admin;\n\nSET SEARCH_PATH TO pgstac, public;\n\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS analyze_items;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS validate_constraints;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\nCREATE OR REPLACE FUNCTION to_int(jsonb) RETURNS int AS $$\n SELECT floor(($1->>0)::float)::int;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION to_float(jsonb) RETURNS float AS $$\n SELECT ($1->>0)::float;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION to_tstz(jsonb) RETURNS timestamptz AS $$\n SELECT ($1->>0)::timestamptz;\n$$ LANGUAGE SQL IMMUTABLE STRICT SET TIME ZONE 'UTC' COST 5000 PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION to_text(jsonb) RETURNS text AS $$\n SELECT CASE WHEN jsonb_typeof($1) IN ('array','object') THEN $1::text ELSE $1->>0 END;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION to_text_array(jsonb) RETURNS text[] AS $$\n SELECT\n CASE jsonb_typeof($1)\n WHEN 'array' THEN ARRAY(SELECT jsonb_array_elements_text($1))\n ELSE ARRAY[$1->>0]\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION bbox_geom(_bbox jsonb) RETURNS geometry AS $$\nSELECT CASE jsonb_array_length(_bbox)\n WHEN 4 THEN\n ST_SetSRID(ST_MakeEnvelope(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float,\n (_bbox->>3)::float\n ),4326)\n WHEN 6 THEN\n ST_SetSRID(ST_3DMakeBox(\n ST_MakePoint(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float\n ),\n ST_MakePoint(\n (_bbox->>3)::float,\n (_bbox->>4)::float,\n (_bbox->>5)::float\n )\n ),4326)\n ELSE null END;\n;\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION geom_bbox(_geom geometry) RETURNS jsonb AS $$\n SELECT jsonb_build_array(\n st_xmin(_geom),\n st_ymin(_geom),\n st_xmax(_geom),\n st_ymax(_geom)\n );\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION flip_jsonb_array(j jsonb) RETURNS jsonb AS $$\n SELECT jsonb_agg(value) FROM (SELECT value FROM jsonb_array_elements(j) WITH ORDINALITY ORDER BY ordinality DESC) as t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION explode_dotpaths(j jsonb) RETURNS SETOF text[] AS $$\n SELECT string_to_array(p, '.') as e FROM jsonb_array_elements_text(j) p;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION explode_dotpaths_recurse(IN j jsonb) RETURNS SETOF text[] AS $$\n WITH RECURSIVE t AS (\n SELECT e FROM explode_dotpaths(j) e\n UNION ALL\n SELECT e[1:cardinality(e)-1]\n FROM t\n WHERE cardinality(e)>1\n ) SELECT e FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION jsonb_set_nested(j jsonb, path text[], val jsonb) RETURNS jsonb AS $$\nDECLARE\nBEGIN\n IF cardinality(path) > 1 THEN\n FOR i IN 1..(cardinality(path)-1) LOOP\n IF j #> path[:i] IS NULL THEN\n j := jsonb_set_lax(j, path[:i], '{}', TRUE);\n END IF;\n END LOOP;\n END IF;\n RETURN jsonb_set_lax(j, path, val, true);\n\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\n\n\nCREATE OR REPLACE FUNCTION jsonb_include(j jsonb, f jsonb) RETURNS jsonb AS $$\nDECLARE\n includes jsonb := f-> 'include';\n outj jsonb := '{}'::jsonb;\n path text[];\nBEGIN\n IF\n includes IS NULL\n OR jsonb_array_length(includes) = 0\n THEN\n RETURN j;\n ELSE\n includes := includes || '[\"id\",\"collection\"]'::jsonb;\n FOR path IN SELECT explode_dotpaths(includes) LOOP\n outj := jsonb_set_nested(outj, path, j #> path);\n END LOOP;\n END IF;\n RETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_exclude(j jsonb, f jsonb) RETURNS jsonb AS $$\nDECLARE\n excludes jsonb := f-> 'exclude';\n outj jsonb := j;\n path text[];\nBEGIN\n IF\n excludes IS NULL\n OR jsonb_array_length(excludes) = 0\n THEN\n RETURN j;\n ELSE\n FOR path IN SELECT explode_dotpaths(excludes) LOOP\n outj := outj #- path;\n END LOOP;\n END IF;\n RETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_fields(j jsonb, f jsonb DEFAULT '{\"fields\":[]}') RETURNS jsonb AS $$\n SELECT jsonb_exclude(jsonb_include(j, f), f);\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION merge_jsonb(_a jsonb, _b jsonb) RETURNS jsonb AS $$\n SELECT\n CASE\n WHEN _a = '\"𒍟※\"'::jsonb THEN NULL\n WHEN _a IS NULL OR jsonb_typeof(_a) = 'null' THEN _b\n WHEN jsonb_typeof(_a) = 'object' AND jsonb_typeof(_b) = 'object' THEN\n (\n SELECT\n jsonb_strip_nulls(\n jsonb_object_agg(\n key,\n merge_jsonb(a.value, b.value)\n )\n )\n FROM\n jsonb_each(coalesce(_a,'{}'::jsonb)) as a\n FULL JOIN\n jsonb_each(coalesce(_b,'{}'::jsonb)) as b\n USING (key)\n )\n WHEN\n jsonb_typeof(_a) = 'array'\n AND jsonb_typeof(_b) = 'array'\n AND jsonb_array_length(_a) = jsonb_array_length(_b)\n THEN\n (\n SELECT jsonb_agg(m) FROM\n ( SELECT\n merge_jsonb(\n jsonb_array_elements(_a),\n jsonb_array_elements(_b)\n ) as m\n ) as l\n )\n ELSE _a\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION strip_jsonb(_a jsonb, _b jsonb) RETURNS jsonb AS $$\n SELECT\n CASE\n\n WHEN (_a IS NULL OR jsonb_typeof(_a) = 'null') AND _b IS NOT NULL AND jsonb_typeof(_b) != 'null' THEN '\"𒍟※\"'::jsonb\n WHEN _b IS NULL OR jsonb_typeof(_a) = 'null' THEN _a\n WHEN _a = _b AND jsonb_typeof(_a) = 'object' THEN '{}'::jsonb\n WHEN _a = _b THEN NULL\n WHEN jsonb_typeof(_a) = 'object' AND jsonb_typeof(_b) = 'object' THEN\n (\n SELECT\n jsonb_strip_nulls(\n jsonb_object_agg(\n key,\n strip_jsonb(a.value, b.value)\n )\n )\n FROM\n jsonb_each(_a) as a\n FULL JOIN\n jsonb_each(_b) as b\n USING (key)\n )\n WHEN\n jsonb_typeof(_a) = 'array'\n AND jsonb_typeof(_b) = 'array'\n AND jsonb_array_length(_a) = jsonb_array_length(_b)\n THEN\n (\n SELECT jsonb_agg(m) FROM\n ( SELECT\n strip_jsonb(\n jsonb_array_elements(_a),\n jsonb_array_elements(_b)\n ) as m\n ) as l\n )\n ELSE _a\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION nullif_jsonbnullempty(j jsonb) RETURNS jsonb AS $$\n SELECT nullif(nullif(nullif(j,'null'::jsonb),'{}'::jsonb),'[]'::jsonb);\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION jsonb_array_unique(j jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(jsonb_agg(DISTINCT a)) v FROM jsonb_array_elements(j) a;\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_concat_ignorenull(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT coalesce(a,'[]'::jsonb) || coalesce(b,'[]'::jsonb);\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_least(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(least(nullif_jsonbnullempty(a), nullif_jsonbnullempty(b)));\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_greatest(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(greatest(a, b));\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION first_notnull_sfunc(anyelement, anyelement) RETURNS anyelement AS $$\n SELECT COALESCE($1,$2);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE AGGREGATE first_notnull(anyelement)(\n SFUNC = first_notnull_sfunc,\n STYPE = anyelement\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_array_unique_merge(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_concat_ignorenull,\n FINALFUNC = jsonb_array_unique\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_min(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_least\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_max(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_greatest\n);\n\nCREATE TABLE IF NOT EXISTS migrations (\n version text PRIMARY KEY,\n datetime timestamptz DEFAULT clock_timestamp() NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION get_version() RETURNS text AS $$\n SELECT version FROM pgstac.migrations ORDER BY datetime DESC, version DESC LIMIT 1;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION set_version(text) RETURNS text AS $$\n INSERT INTO pgstac.migrations (version) VALUES ($1)\n ON CONFLICT DO NOTHING\n RETURNING version;\n$$ LANGUAGE SQL;\n\n\nCREATE TABLE IF NOT EXISTS pgstac_settings (\n name text PRIMARY KEY,\n value text NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION table_empty(text) RETURNS boolean AS $$\nDECLARE\n retval boolean;\nBEGIN\n EXECUTE format($q$\n SELECT NOT EXISTS (SELECT 1 FROM %I LIMIT 1)\n $q$,\n $1\n ) INTO retval;\n RETURN retval;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION get_setting(IN _setting text, IN conf jsonb DEFAULT NULL) RETURNS text AS $$\nSELECT COALESCE(\n nullif(conf->>_setting, ''),\n nullif(current_setting(concat('pgstac.',_setting), TRUE),''),\n nullif((SELECT value FROM pgstac.pgstac_settings WHERE name=_setting),'')\n);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_setting_bool(IN _setting text, IN conf jsonb DEFAULT NULL) RETURNS boolean AS $$\nSELECT COALESCE(\n conf->>_setting,\n current_setting(concat('pgstac.',_setting), TRUE),\n (SELECT value FROM pgstac.pgstac_settings WHERE name=_setting),\n 'FALSE'\n)::boolean;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context(conf jsonb DEFAULT NULL) RETURNS text AS $$\n SELECT pgstac.get_setting('context', conf);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context_estimated_count(conf jsonb DEFAULT NULL) RETURNS int AS $$\n SELECT pgstac.get_setting('context_estimated_count', conf)::int;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_estimated_cost();\nCREATE OR REPLACE FUNCTION context_estimated_cost(conf jsonb DEFAULT NULL) RETURNS float AS $$\n SELECT pgstac.get_setting('context_estimated_cost', conf)::float;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_stats_ttl();\nCREATE OR REPLACE FUNCTION context_stats_ttl(conf jsonb DEFAULT NULL) RETURNS interval AS $$\n SELECT pgstac.get_setting('context_stats_ttl', conf)::interval;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION t2s(text) RETURNS text AS $$\n SELECT extract(epoch FROM $1::interval)::text || ' s';\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE STRICT;\n\nCREATE OR REPLACE FUNCTION age_ms(a timestamptz, b timestamptz DEFAULT clock_timestamp()) RETURNS float AS $$\n SELECT abs(extract(epoch from age(a,b)) * 1000);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION queue_timeout() RETURNS interval AS $$\n SELECT t2s(coalesce(\n get_setting('queue_timeout'),\n '1h'\n ))::interval;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION notice(VARIADIC text[]) RETURNS boolean AS $$\nDECLARE\ndebug boolean := current_setting('pgstac.debug', true);\nBEGIN\n IF debug THEN\n RAISE NOTICE 'NOTICE FROM FUNC: % >>>>> %', concat_ws(' | ', $1), clock_timestamp();\n RETURN TRUE;\n END IF;\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION empty_arr(ANYARRAY) RETURNS BOOLEAN AS $$\nSELECT CASE\n WHEN $1 IS NULL THEN TRUE\n WHEN cardinality($1)<1 THEN TRUE\nELSE FALSE\nEND;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_intersection(_a ANYARRAY, _b ANYARRAY) RETURNS ANYARRAY AS $$\n SELECT ARRAY ( SELECT unnest(_a) INTERSECT SELECT UNNEST(_b) );\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION array_map_ident(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_ident(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_map_literal(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_literal(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION array_reverse(anyarray) RETURNS anyarray AS $$\nSELECT ARRAY(\n SELECT $1[i]\n FROM generate_subscripts($1,1) AS s(i)\n ORDER BY i DESC\n);\n$$ LANGUAGE SQL STRICT IMMUTABLE;\n\nDROP TABLE IF EXISTS query_queue;\nCREATE TABLE query_queue (\n query text PRIMARY KEY,\n added timestamptz DEFAULT now()\n);\n\nDROP TABLE IF EXISTS query_queue_history;\nCREATE TABLE query_queue_history(\n query text,\n added timestamptz NOT NULL,\n finished timestamptz NOT NULL DEFAULT now(),\n error text\n);\n\nCREATE OR REPLACE PROCEDURE run_queued_queries() AS $$\nDECLARE\n qitem query_queue%ROWTYPE;\n timeout_ts timestamptz;\n error text;\n cnt int := 0;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n DELETE FROM query_queue WHERE query = (SELECT query FROM query_queue ORDER BY added DESC LIMIT 1 FOR UPDATE SKIP LOCKED) RETURNING * INTO qitem;\n IF NOT FOUND THEN\n EXIT;\n END IF;\n cnt := cnt + 1;\n BEGIN\n RAISE NOTICE 'RUNNING QUERY: %', qitem.query;\n EXECUTE qitem.query;\n EXCEPTION WHEN others THEN\n error := format('%s | %s', SQLERRM, SQLSTATE);\n END;\n INSERT INTO query_queue_history (query, added, finished, error)\n VALUES (qitem.query, qitem.added, clock_timestamp(), error);\n COMMIT;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION run_queued_queries_intransaction() RETURNS int AS $$\nDECLARE\n qitem query_queue%ROWTYPE;\n timeout_ts timestamptz;\n error text;\n cnt int := 0;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n DELETE FROM query_queue WHERE query = (SELECT query FROM query_queue ORDER BY added DESC LIMIT 1 FOR UPDATE SKIP LOCKED) RETURNING * INTO qitem;\n IF NOT FOUND THEN\n RETURN cnt;\n END IF;\n cnt := cnt + 1;\n BEGIN\n qitem.query := regexp_replace(qitem.query, 'CONCURRENTLY', '');\n RAISE NOTICE 'RUNNING QUERY: %', qitem.query;\n\n EXECUTE qitem.query;\n EXCEPTION WHEN others THEN\n error := format('%s | %s', SQLERRM, SQLSTATE);\n RAISE WARNING '%', error;\n END;\n INSERT INTO query_queue_history (query, added, finished, error)\n VALUES (qitem.query, qitem.added, clock_timestamp(), error);\n END LOOP;\n RETURN cnt;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION run_or_queue(query text) RETURNS VOID AS $$\nDECLARE\n use_queue text := COALESCE(get_setting('use_queue'), 'FALSE')::boolean;\nBEGIN\n IF get_setting_bool('debug') THEN\n RAISE NOTICE '%', query;\n END IF;\n IF use_queue THEN\n INSERT INTO query_queue (query) VALUES (query) ON CONFLICT DO NOTHING;\n ELSE\n EXECUTE query;\n END IF;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n\nDROP FUNCTION IF EXISTS check_pgstac_settings;\nCREATE OR REPLACE FUNCTION check_pgstac_settings(_sysmem text DEFAULT NULL) RETURNS VOID AS $$\nDECLARE\n settingval text;\n sysmem bigint := pg_size_bytes(_sysmem);\n effective_cache_size bigint := pg_size_bytes(current_setting('effective_cache_size', TRUE));\n shared_buffers bigint := pg_size_bytes(current_setting('shared_buffers', TRUE));\n work_mem bigint := pg_size_bytes(current_setting('work_mem', TRUE));\n max_connections int := current_setting('max_connections', TRUE);\n maintenance_work_mem bigint := pg_size_bytes(current_setting('maintenance_work_mem', TRUE));\n seq_page_cost float := current_setting('seq_page_cost', TRUE);\n random_page_cost float := current_setting('random_page_cost', TRUE);\n temp_buffers bigint := pg_size_bytes(current_setting('temp_buffers', TRUE));\n r record;\nBEGIN\n IF _sysmem IS NULL THEN\n RAISE NOTICE 'Call function with the size of your system memory `SELECT check_pgstac_settings(''4GB'')` to get pg system setting recommendations.';\n ELSE\n IF effective_cache_size < (sysmem * 0.5) THEN\n RAISE WARNING 'effective_cache_size of % is set low for a system with %. Recomended value between % and %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.5), pg_size_pretty(sysmem * 0.75);\n ELSIF effective_cache_size > (sysmem * 0.75) THEN\n RAISE WARNING 'effective_cache_size of % is set high for a system with %. Recomended value between % and %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.5), pg_size_pretty(sysmem * 0.75);\n ELSE\n RAISE NOTICE 'effective_cache_size of % is set appropriately for a system with %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem);\n END IF;\n\n IF shared_buffers < (sysmem * 0.2) THEN\n RAISE WARNING 'shared_buffers of % is set low for a system with %. Recomended value between % and %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.2), pg_size_pretty(sysmem * 0.3);\n ELSIF shared_buffers > (sysmem * 0.3) THEN\n RAISE WARNING 'shared_buffers of % is set high for a system with %. Recomended value between % and %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.2), pg_size_pretty(sysmem * 0.3);\n ELSE\n RAISE NOTICE 'shared_buffers of % is set appropriately for a system with %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem);\n END IF;\n shared_buffers = sysmem * 0.3;\n IF maintenance_work_mem < (sysmem * 0.2) THEN\n RAISE WARNING 'maintenance_work_mem of % is set low for shared_buffers of %. Recomended value between % and %', pg_size_pretty(maintenance_work_mem), pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers * 0.2), pg_size_pretty(shared_buffers * 0.3);\n ELSIF maintenance_work_mem > (shared_buffers * 0.3) THEN\n RAISE WARNING 'maintenance_work_mem of % is set high for shared_buffers of %. Recomended value between % and %', pg_size_pretty(maintenance_work_mem), pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers * 0.2), pg_size_pretty(shared_buffers * 0.3);\n ELSE\n RAISE NOTICE 'maintenance_work_mem of % is set appropriately for shared_buffers of %', pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers);\n END IF;\n\n IF work_mem * max_connections > shared_buffers THEN\n RAISE WARNING 'work_mem setting of % is set high for % max_connections please reduce work_mem to % or decrease max_connections to %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers/max_connections), floor(shared_buffers/work_mem);\n ELSIF work_mem * max_connections < (shared_buffers * 0.75) THEN\n RAISE WARNING 'work_mem setting of % is set low for % max_connections you may consider raising work_mem to % or increasing max_connections to %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers/max_connections), floor(shared_buffers/work_mem);\n ELSE\n RAISE NOTICE 'work_mem setting of % and max_connections of % are adequate for shared_buffers of %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers);\n END IF;\n\n IF random_page_cost / seq_page_cost != 1.1 THEN\n RAISE WARNING 'random_page_cost (%) /seq_page_cost (%) should be set to 1.1 for SSD. Change random_page_cost to %', random_page_cost, seq_page_cost, 1.1 * seq_page_cost;\n ELSE\n RAISE NOTICE 'random_page_cost and seq_page_cost set appropriately for SSD';\n END IF;\n\n IF temp_buffers < greatest(pg_size_bytes('128MB'),(maintenance_work_mem / 2)) THEN\n RAISE WARNING 'pgstac makes heavy use of temp tables, consider raising temp_buffers from % to %', pg_size_pretty(temp_buffers), greatest('128MB', pg_size_pretty((shared_buffers / 16)));\n END IF;\n END IF;\n\n RAISE NOTICE 'VALUES FOR PGSTAC VARIABLES';\n RAISE NOTICE 'These can be set either as GUC system variables or by setting in the pgstac_settings table.';\n\n FOR r IN SELECT name, get_setting(name) as setting, CASE WHEN current_setting(concat('pgstac.',name), TRUE) IS NOT NULL THEN concat('pgstac.',name, ' GUC') WHEN value IS NOT NULL THEN 'pgstac_settings table' ELSE 'Not Set' END as loc FROM pgstac_settings LOOP\n RAISE NOTICE '% is set to % from the %', r.name, r.setting, r.loc;\n END LOOP;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pg_cron';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider intalling pg_cron which can be used to automate tasks';\n ELSE\n RAISE NOTICE 'pg_cron % is installed', settingval;\n END IF;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pgstattuple';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider installing the pgstattuple extension which can be used to help maintain tables and indexes.';\n ELSE\n RAISE NOTICE 'pgstattuple % is installed', settingval;\n END IF;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pg_stat_statements';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider installing the pg_stat_statements extension which is very helpful for tracking the types of queries on the system';\n ELSE\n RAISE NOTICE 'pg_stat_statements % is installed', settingval;\n IF current_setting('pg_stat_statements.track_statements', TRUE) IS DISTINCT FROM 'all' THEN\n RAISE WARNING 'SET pg_stat_statements.track_statements TO ''all''; --In order to track statements within functions.';\n END IF;\n END IF;\n\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac, public SET CLIENT_MIN_MESSAGES TO NOTICE;\n/* looks for a geometry in a stac item first from geometry and falling back to bbox */\nCREATE OR REPLACE FUNCTION stac_geom(value jsonb) RETURNS geometry AS $$\nSELECT\n CASE\n WHEN value ? 'intersects' THEN\n ST_GeomFromGeoJSON(value->>'intersects')\n WHEN value ? 'geometry' THEN\n ST_GeomFromGeoJSON(value->>'geometry')\n WHEN value ? 'bbox' THEN\n pgstac.bbox_geom(value->'bbox')\n ELSE NULL\n END as geometry\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION stac_daterange(\n value jsonb\n) RETURNS tstzrange AS $$\nDECLARE\n props jsonb := value;\n dt timestamptz;\n edt timestamptz;\nBEGIN\n IF props ? 'properties' THEN\n props := props->'properties';\n END IF;\n IF\n props ? 'start_datetime'\n AND props->>'start_datetime' IS NOT NULL\n AND props ? 'end_datetime'\n AND props->>'end_datetime' IS NOT NULL\n THEN\n dt := props->>'start_datetime';\n edt := props->>'end_datetime';\n IF dt > edt THEN\n RAISE EXCEPTION 'start_datetime must be < end_datetime';\n END IF;\n ELSE\n dt := props->>'datetime';\n edt := props->>'datetime';\n END IF;\n IF dt is NULL OR edt IS NULL THEN\n RAISE NOTICE 'DT: %, EDT: %', dt, edt;\n RAISE EXCEPTION 'Either datetime (%) or both start_datetime (%) and end_datetime (%) must be set.', props->>'datetime',props->>'start_datetime',props->>'end_datetime';\n END IF;\n RETURN tstzrange(dt, edt, '[]');\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_datetime(value jsonb) RETURNS timestamptz AS $$\n SELECT lower(stac_daterange(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_end_datetime(value jsonb) RETURNS timestamptz AS $$\n SELECT upper(stac_daterange(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE TABLE IF NOT EXISTS stac_extensions(\n url text PRIMARY KEY,\n content jsonb\n);\nCREATE OR REPLACE FUNCTION queryable_signature(n text, c text[]) RETURNS text AS $$\n SELECT concat(n, c);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TABLE queryables (\n id bigint GENERATED ALWAYS AS identity PRIMARY KEY,\n name text NOT NULL,\n collection_ids text[], -- used to determine what partitions to create indexes on\n definition jsonb,\n property_path text,\n property_wrapper text,\n property_index_type text\n);\nCREATE INDEX queryables_name_idx ON queryables (name);\nCREATE INDEX queryables_collection_idx ON queryables USING GIN (collection_ids);\nCREATE INDEX queryables_property_wrapper_idx ON queryables (property_wrapper);\n\nCREATE OR REPLACE FUNCTION pgstac.queryables_constraint_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n allcollections text[];\nBEGIN\n RAISE NOTICE 'Making sure that name/collection is unique for queryables %', NEW;\n IF NEW.collection_ids IS NOT NULL THEN\n IF EXISTS (\n SELECT 1\n FROM unnest(NEW.collection_ids) c\n LEFT JOIN\n collections\n ON (collections.id = c)\n WHERE collections.id IS NULL\n ) THEN\n RAISE foreign_key_violation USING MESSAGE = format(\n 'One or more collections in %s do not exist.', NEW.collection_ids\n );\n RETURN NULL;\n END IF;\n END IF;\n IF TG_OP = 'INSERT' THEN\n IF EXISTS (\n SELECT 1 FROM queryables q\n WHERE\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n )\n ) THEN\n RAISE unique_violation USING MESSAGE = format(\n 'There is already a queryable for %s for a collection in %s: %s',\n NEW.name,\n NEW.collection_ids,\n\t\t\t\t(SELECT json_agg(row_to_json(q)) FROM queryables q WHERE\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n ))\n );\n RETURN NULL;\n END IF;\n END IF;\n IF TG_OP = 'UPDATE' THEN\n IF EXISTS (\n SELECT 1 FROM queryables q\n WHERE\n q.id != NEW.id\n AND\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n )\n ) THEN\n RAISE unique_violation\n USING MESSAGE = format(\n 'There is already a queryable for %s for a collection in %s',\n NEW.name,\n NEW.collection_ids\n );\n RETURN NULL;\n END IF;\n END IF;\n\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER queryables_constraint_insert_trigger\nBEFORE INSERT ON queryables\nFOR EACH ROW EXECUTE PROCEDURE queryables_constraint_triggerfunc();\n\nCREATE TRIGGER queryables_constraint_update_trigger\nBEFORE UPDATE ON queryables\nFOR EACH ROW\nWHEN (NEW.name = OLD.name AND NEW.collection_ids IS DISTINCT FROM OLD.collection_ids)\nEXECUTE PROCEDURE queryables_constraint_triggerfunc();\n\n\nCREATE OR REPLACE FUNCTION array_to_path(arr text[]) RETURNS text AS $$\n SELECT string_agg(\n quote_literal(v),\n '->'\n ) FROM unnest(arr) v;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\n\n\nCREATE OR REPLACE FUNCTION queryable(\n IN dotpath text,\n OUT path text,\n OUT expression text,\n OUT wrapper text,\n OUT nulled_wrapper text\n) AS $$\nDECLARE\n q RECORD;\n path_elements text[];\nBEGIN\n IF dotpath IN ('id', 'geometry', 'datetime', 'end_datetime', 'collection') THEN\n path := dotpath;\n expression := dotpath;\n wrapper := NULL;\n RETURN;\n END IF;\n SELECT * INTO q FROM queryables\n WHERE\n name=dotpath\n OR name = 'properties.' || dotpath\n OR name = replace(dotpath, 'properties.', '')\n ;\n IF q.property_wrapper IS NULL THEN\n IF q.definition->>'type' = 'number' THEN\n wrapper := 'to_float';\n nulled_wrapper := wrapper;\n ELSIF q.definition->>'format' = 'date-time' THEN\n wrapper := 'to_tstz';\n nulled_wrapper := wrapper;\n ELSE\n nulled_wrapper := NULL;\n wrapper := 'to_text';\n END IF;\n ELSE\n wrapper := q.property_wrapper;\n nulled_wrapper := wrapper;\n END IF;\n IF q.property_path IS NOT NULL THEN\n path := q.property_path;\n ELSE\n path_elements := string_to_array(dotpath, '.');\n IF path_elements[1] IN ('links', 'assets', 'stac_version', 'stac_extensions') THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSIF path_elements[1] = 'properties' THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSE\n path := format($F$content->'properties'->%s$F$, array_to_path(path_elements));\n END IF;\n END IF;\n expression := format('%I(%s)', wrapper, path);\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\nCREATE OR REPLACE FUNCTION unnest_collection(collection_ids text[] DEFAULT NULL) RETURNS SETOF text AS $$\n DECLARE\n BEGIN\n IF collection_ids IS NULL THEN\n RETURN QUERY SELECT id FROM collections;\n END IF;\n RETURN QUERY SELECT unnest(collection_ids);\n END;\n$$ LANGUAGE PLPGSQL STABLE;\n\nCREATE OR REPLACE FUNCTION normalize_indexdef(def text) RETURNS text AS $$\nDECLARE\nBEGIN\n def := btrim(def, ' \\n\\t');\n\tdef := regexp_replace(def, '^CREATE (UNIQUE )?INDEX ([^ ]* )?ON (ONLY )?([^ ]* )?', '', 'i');\n RETURN def;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE STRICT PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION indexdef(q queryables) RETURNS text AS $$\n DECLARE\n out text;\n BEGIN\n IF q.name = 'id' THEN\n out := 'CREATE UNIQUE INDEX ON %I USING btree (id)';\n ELSIF q.name = 'datetime' THEN\n out := 'CREATE INDEX ON %I USING btree (datetime DESC, end_datetime)';\n ELSIF q.name = 'geometry' THEN\n out := 'CREATE INDEX ON %I USING gist (geometry)';\n ELSE\n out := format($q$CREATE INDEX ON %%I USING %s (%s(((content -> 'properties'::text) -> %L::text)))$q$,\n lower(COALESCE(q.property_index_type, 'BTREE')),\n lower(COALESCE(q.property_wrapper, 'to_text')),\n q.name\n );\n END IF;\n RETURN btrim(out, ' \\n\\t');\n END;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\n\nDROP VIEW IF EXISTS pgstac_indexes;\nCREATE VIEW pgstac_indexes AS\nSELECT\n i.schemaname,\n i.tablename,\n i.indexname,\n regexp_replace(btrim(replace(replace(indexdef, i.indexname, ''),'pgstac.',''),' \\t\\n'), '[ ]+', ' ', 'g') as idx,\n COALESCE(\n (regexp_match(indexdef, '\\(([a-zA-Z]+)\\)'))[1],\n (regexp_match(indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_-]+)''::text'))[1],\n CASE WHEN indexdef ~* '\\(datetime desc, end_datetime\\)' THEN 'datetime' ELSE NULL END\n ) AS field,\n pg_table_size(i.indexname::text) as index_size,\n pg_size_pretty(pg_table_size(i.indexname::text)) as index_size_pretty\nFROM\n pg_indexes i\nWHERE i.schemaname='pgstac' and i.tablename ~ '_items_' AND indexdef !~* ' only ';\n\nDROP VIEW IF EXISTS pgstac_index_stats;\nCREATE VIEW pgstac_indexes_stats AS\nSELECT\n i.schemaname,\n i.tablename,\n i.indexname,\n indexdef,\n COALESCE(\n (regexp_match(indexdef, '\\(([a-zA-Z]+)\\)'))[1],\n (regexp_match(indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_]+)''::text'))[1],\n CASE WHEN indexdef ~* '\\(datetime desc, end_datetime\\)' THEN 'datetime_end_datetime' ELSE NULL END\n ) AS field,\n pg_table_size(i.indexname::text) as index_size,\n pg_size_pretty(pg_table_size(i.indexname::text)) as index_size_pretty,\n n_distinct,\n most_common_vals::text::text[],\n most_common_freqs::text::text[],\n histogram_bounds::text::text[],\n correlation\nFROM\n pg_indexes i\n LEFT JOIN pg_stats s ON (s.tablename = i.indexname)\nWHERE i.schemaname='pgstac' and i.tablename ~ '_items_';\n\nset check_function_bodies to off;\nCREATE OR REPLACE FUNCTION maintain_partition_queries(\n part text DEFAULT 'items',\n dropindexes boolean DEFAULT FALSE,\n rebuildindexes boolean DEFAULT FALSE,\n idxconcurrently boolean DEFAULT FALSE\n) RETURNS SETOF text AS $$\nDECLARE\n rec record;\nBEGIN\n FOR rec IN (\n WITH p AS (\n SELECT\n relid::text as partition,\n replace(replace(\n CASE\n WHEN level = 1 THEN pg_get_expr(c.relpartbound, c.oid)\n ELSE pg_get_expr(parent.relpartbound, parent.oid)\n END,\n 'FOR VALUES IN (''',''), ''')',\n ''\n ) AS collection\n FROM pg_partition_tree('items')\n JOIN pg_class c ON (relid::regclass = c.oid)\n JOIN pg_class parent ON (parentrelid::regclass = parent.oid AND isleaf)\n ), i AS (\n SELECT\n partition,\n indexname,\n regexp_replace(btrim(replace(replace(indexdef, indexname, ''),'pgstac.',''),' \\t\\n'), '[ ]+', ' ', 'g') as iidx,\n COALESCE(\n (regexp_match(indexdef, '\\(([a-zA-Z]+)\\)'))[1],\n (regexp_match(indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_-]+)''::text'))[1],\n CASE WHEN indexdef ~* '\\(datetime desc, end_datetime\\)' THEN 'datetime' ELSE NULL END\n ) AS field\n FROM\n pg_indexes\n JOIN p ON (tablename=partition)\n ), q AS (\n SELECT\n name AS field,\n collection,\n partition,\n format(indexdef(queryables), partition) as qidx\n FROM queryables, unnest_collection(queryables.collection_ids) collection\n JOIN p USING (collection)\n WHERE property_index_type IS NOT NULL OR name IN ('datetime','geometry','id')\n )\n SELECT * FROM i FULL JOIN q USING (field, partition)\n WHERE lower(iidx) IS DISTINCT FROM lower(qidx)\n ) LOOP\n IF rec.iidx IS NULL THEN\n IF idxconcurrently THEN\n RETURN NEXT replace(rec.qidx, 'INDEX', 'INDEX CONCURRENTLY');\n ELSE\n RETURN NEXT rec.qidx;\n END IF;\n ELSIF rec.qidx IS NULL AND dropindexes THEN\n RETURN NEXT format('DROP INDEX IF EXISTS %I;', rec.indexname);\n ELSIF lower(rec.qidx) != lower(rec.iidx) THEN\n IF dropindexes THEN\n RETURN NEXT format('DROP INDEX IF EXISTS %I; %s;', rec.indexname, rec.qidx);\n ELSE\n IF idxconcurrently THEN\n RETURN NEXT replace(rec.qidx, 'INDEX', 'INDEX CONCURRENTLY');\n ELSE\n RETURN NEXT rec.qidx;\n END IF;\n END IF;\n ELSIF rebuildindexes and rec.indexname IS NOT NULL THEN\n IF idxconcurrently THEN\n RETURN NEXT format('REINDEX INDEX CONCURRENTLY %I;', rec.indexname);\n ELSE\n RETURN NEXT format('REINDEX INDEX %I;', rec.indexname);\n END IF;\n END IF;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION maintain_partitions(\n part text DEFAULT 'items',\n dropindexes boolean DEFAULT FALSE,\n rebuildindexes boolean DEFAULT FALSE\n) RETURNS VOID AS $$\n WITH t AS (\n SELECT run_or_queue(q) FROM maintain_partition_queries(part, dropindexes, rebuildindexes) q\n ) SELECT count(*) FROM t;\n$$ LANGUAGE SQL;\n\n\nCREATE OR REPLACE FUNCTION queryables_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\nBEGIN\n PERFORM maintain_partitions();\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER queryables_trigger AFTER INSERT OR UPDATE ON queryables\nFOR EACH STATEMENT EXECUTE PROCEDURE queryables_trigger_func();\n\n\nCREATE OR REPLACE FUNCTION get_queryables(_collection_ids text[] DEFAULT NULL) RETURNS jsonb AS $$\nDECLARE\nBEGIN\n -- Build up queryables if the input contains valid collection ids or is empty\n IF EXISTS (\n SELECT 1 FROM collections\n WHERE\n _collection_ids IS NULL\n OR cardinality(_collection_ids) = 0\n OR id = ANY(_collection_ids)\n )\n THEN\n RETURN (\n WITH base AS (\n SELECT\n unnest(collection_ids) as collection_id,\n name,\n coalesce(definition, '{\"type\":\"string\"}'::jsonb) as definition\n FROM queryables\n WHERE\n _collection_ids IS NULL OR\n _collection_ids = '{}'::text[] OR\n _collection_ids && collection_ids\n UNION ALL\n SELECT null, name, coalesce(definition, '{\"type\":\"string\"}'::jsonb) as definition\n FROM queryables WHERE collection_ids IS NULL OR collection_ids = '{}'::text[]\n ), g AS (\n SELECT\n name,\n first_notnull(definition) as definition,\n jsonb_array_unique_merge(definition->'enum') as enum,\n jsonb_min(definition->'minimum') as minimum,\n jsonb_min(definition->'maxiumn') as maximum\n FROM base\n GROUP BY 1\n )\n SELECT\n jsonb_build_object(\n '$schema', 'http://json-schema.org/draft-07/schema#',\n '$id', '',\n 'type', 'object',\n 'title', 'STAC Queryables.',\n 'properties', jsonb_object_agg(\n name,\n definition\n ||\n jsonb_strip_nulls(jsonb_build_object(\n 'enum', enum,\n 'minimum', minimum,\n 'maximum', maximum\n ))\n )\n )\n FROM g\n );\n ELSE\n RETURN NULL;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\nCREATE OR REPLACE FUNCTION get_queryables(_collection text DEFAULT NULL) RETURNS jsonb AS $$\n SELECT\n CASE\n WHEN _collection IS NULL THEN get_queryables(NULL::text[])\n ELSE get_queryables(ARRAY[_collection])\n END\n ;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_queryables() RETURNS jsonb AS $$\n SELECT get_queryables(NULL::text[]);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION schema_qualify_refs(url text, j jsonb) returns jsonb as $$\n SELECT regexp_replace(j::text, '\"\\$ref\": \"#', concat('\"$ref\": \"', url, '#'), 'g')::jsonb;\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE VIEW stac_extension_queryables AS\nSELECT DISTINCT key as name, schema_qualify_refs(e.url, j.value) as definition FROM stac_extensions e, jsonb_each(e.content->'definitions'->'fields'->'properties') j;\n\n\nCREATE OR REPLACE FUNCTION missing_queryables(_collection text, _tablesample float DEFAULT 5, minrows float DEFAULT 10) RETURNS TABLE(collection text, name text, definition jsonb, property_wrapper text) AS $$\nDECLARE\n q text;\n _partition text;\n explain_json json;\n psize float;\n estrows float;\nBEGIN\n SELECT format('_items_%s', key) INTO _partition FROM collections WHERE id=_collection;\n\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM %I;', _partition)\n INTO explain_json;\n psize := explain_json->0->'Plan'->'Plan Rows';\n estrows := _tablesample * .01 * psize;\n IF estrows < minrows THEN\n _tablesample := least(100,greatest(_tablesample, (estrows / psize) / 100));\n RAISE NOTICE '%', (psize / estrows) / 100;\n END IF;\n RAISE NOTICE 'Using tablesample % to find missing queryables from % % that has ~% rows estrows: %', _tablesample, _collection, _partition, psize, estrows;\n\n q := format(\n $q$\n WITH q AS (\n SELECT * FROM queryables\n WHERE\n collection_ids IS NULL\n OR %L = ANY(collection_ids)\n ), t AS (\n SELECT\n content->'properties' AS properties\n FROM\n %I\n TABLESAMPLE SYSTEM(%L)\n ), p AS (\n SELECT DISTINCT ON (key)\n key,\n value,\n s.definition\n FROM t\n JOIN LATERAL jsonb_each(properties) ON TRUE\n LEFT JOIN q ON (q.name=key)\n LEFT JOIN stac_extension_queryables s ON (s.name=key)\n WHERE q.definition IS NULL\n )\n SELECT\n %L,\n key,\n COALESCE(definition, jsonb_build_object('type',jsonb_typeof(value))) as definition,\n CASE\n WHEN definition->>'type' = 'integer' THEN 'to_int'\n WHEN COALESCE(definition->>'type', jsonb_typeof(value)) = 'number' THEN 'to_float'\n WHEN COALESCE(definition->>'type', jsonb_typeof(value)) = 'array' THEN 'to_text_array'\n ELSE 'to_text'\n END\n FROM p;\n $q$,\n _collection,\n _partition,\n _tablesample,\n _collection\n );\n RETURN QUERY EXECUTE q;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION missing_queryables(_tablesample float DEFAULT 5) RETURNS TABLE(collection_ids text[], name text, definition jsonb, property_wrapper text) AS $$\n SELECT\n array_agg(collection),\n name,\n definition,\n property_wrapper\n FROM\n collections\n JOIN LATERAL\n missing_queryables(id, _tablesample) c\n ON TRUE\n GROUP BY\n 2,3,4\n ORDER BY 2,1\n ;\n$$ LANGUAGE SQL;\nCREATE OR REPLACE FUNCTION parse_dtrange(\n _indate jsonb,\n relative_base timestamptz DEFAULT date_trunc('hour', CURRENT_TIMESTAMP)\n) RETURNS tstzrange AS $$\nDECLARE\n timestrs text[];\n s timestamptz;\n e timestamptz;\nBEGIN\n timestrs :=\n CASE\n WHEN _indate ? 'timestamp' THEN\n ARRAY[_indate->>'timestamp']\n WHEN _indate ? 'interval' THEN\n to_text_array(_indate->'interval')\n WHEN jsonb_typeof(_indate) = 'array' THEN\n to_text_array(_indate)\n ELSE\n regexp_split_to_array(\n _indate->>0,\n '/'\n )\n END;\n RAISE NOTICE 'TIMESTRS %', timestrs;\n IF cardinality(timestrs) = 1 THEN\n IF timestrs[1] ILIKE 'P%' THEN\n RETURN tstzrange(relative_base - upper(timestrs[1])::interval, relative_base, '[)');\n END IF;\n s := timestrs[1]::timestamptz;\n RETURN tstzrange(s, s, '[]');\n END IF;\n\n IF cardinality(timestrs) != 2 THEN\n RAISE EXCEPTION 'Timestamp cannot have more than 2 values';\n END IF;\n\n IF timestrs[1] = '..' OR timestrs[1] = '' THEN\n s := '-infinity'::timestamptz;\n e := timestrs[2]::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] = '..' OR timestrs[2] = '' THEN\n s := timestrs[1]::timestamptz;\n e := 'infinity'::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[1] ILIKE 'P%' AND timestrs[2] NOT ILIKE 'P%' THEN\n e := timestrs[2]::timestamptz;\n s := e - upper(timestrs[1])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] ILIKE 'P%' AND timestrs[1] NOT ILIKE 'P%' THEN\n s := timestrs[1]::timestamptz;\n e := s + upper(timestrs[2])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n s := timestrs[1]::timestamptz;\n e := timestrs[2]::timestamptz;\n\n RETURN tstzrange(s,e,'[)');\n\n RETURN NULL;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT PARALLEL SAFE SET TIME ZONE 'UTC';\n\nCREATE OR REPLACE FUNCTION parse_dtrange(\n _indate text,\n relative_base timestamptz DEFAULT CURRENT_TIMESTAMP\n) RETURNS tstzrange AS $$\n SELECT parse_dtrange(to_jsonb(_indate), relative_base);\n$$ LANGUAGE SQL STABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION temporal_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\n ll text := 'datetime';\n lh text := 'end_datetime';\n rrange tstzrange;\n rl text;\n rh text;\n outq text;\nBEGIN\n rrange := parse_dtrange(args->1);\n RAISE NOTICE 'Constructing temporal query OP: %, ARGS: %, RRANGE: %', op, args, rrange;\n op := lower(op);\n rl := format('%L::timestamptz', lower(rrange));\n rh := format('%L::timestamptz', upper(rrange));\n outq := CASE op\n WHEN 't_before' THEN 'lh < rl'\n WHEN 't_after' THEN 'll > rh'\n WHEN 't_meets' THEN 'lh = rl'\n WHEN 't_metby' THEN 'll = rh'\n WHEN 't_overlaps' THEN 'll < rl AND rl < lh < rh'\n WHEN 't_overlappedby' THEN 'rl < ll < rh AND lh > rh'\n WHEN 't_starts' THEN 'll = rl AND lh < rh'\n WHEN 't_startedby' THEN 'll = rl AND lh > rh'\n WHEN 't_during' THEN 'll > rl AND lh < rh'\n WHEN 't_contains' THEN 'll < rl AND lh > rh'\n WHEN 't_finishes' THEN 'll > rl AND lh = rh'\n WHEN 't_finishedby' THEN 'll < rl AND lh = rh'\n WHEN 't_equals' THEN 'll = rl AND lh = rh'\n WHEN 't_disjoint' THEN 'NOT (ll <= rh AND lh >= rl)'\n WHEN 't_intersects' THEN 'll <= rh AND lh >= rl'\n WHEN 'anyinteracts' THEN 'll <= rh AND lh >= rl'\n END;\n outq := regexp_replace(outq, '\\mll\\M', ll);\n outq := regexp_replace(outq, '\\mlh\\M', lh);\n outq := regexp_replace(outq, '\\mrl\\M', rl);\n outq := regexp_replace(outq, '\\mrh\\M', rh);\n outq := format('(%s)', outq);\n RETURN outq;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\n\n\nCREATE OR REPLACE FUNCTION spatial_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\n geom text;\n j jsonb := args->1;\nBEGIN\n op := lower(op);\n RAISE NOTICE 'Constructing spatial query OP: %, ARGS: %', op, args;\n IF op NOT IN ('s_equals','s_disjoint','s_touches','s_within','s_overlaps','s_crosses','s_intersects','intersects','s_contains') THEN\n RAISE EXCEPTION 'Spatial Operator % Not Supported', op;\n END IF;\n op := regexp_replace(op, '^s_', 'st_');\n IF op = 'intersects' THEN\n op := 'st_intersects';\n END IF;\n -- Convert geometry to WKB string\n IF j ? 'type' AND j ? 'coordinates' THEN\n geom := st_geomfromgeojson(j)::text;\n ELSIF jsonb_typeof(j) = 'array' THEN\n geom := bbox_geom(j)::text;\n END IF;\n\n RETURN format('%s(geometry, %L::geometry)', op, geom);\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION query_to_cql2(q jsonb) RETURNS jsonb AS $$\n-- Translates anything passed in through the deprecated \"query\" into equivalent CQL2\nWITH t AS (\n SELECT key as property, value as ops\n FROM jsonb_each(q)\n), t2 AS (\n SELECT property, (jsonb_each(ops)).*\n FROM t WHERE jsonb_typeof(ops) = 'object'\n UNION ALL\n SELECT property, 'eq', ops\n FROM t WHERE jsonb_typeof(ops) != 'object'\n)\nSELECT\n jsonb_strip_nulls(jsonb_build_object(\n 'op', 'and',\n 'args', jsonb_agg(\n jsonb_build_object(\n 'op', key,\n 'args', jsonb_build_array(\n jsonb_build_object('property',property),\n value\n )\n )\n )\n )\n) as qcql FROM t2\n;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\nCREATE OR REPLACE FUNCTION cql1_to_cql2(j jsonb) RETURNS jsonb AS $$\nDECLARE\n args jsonb;\n ret jsonb;\nBEGIN\n RAISE NOTICE 'CQL1_TO_CQL2: %', j;\n IF j ? 'filter' THEN\n RETURN cql1_to_cql2(j->'filter');\n END IF;\n IF j ? 'property' THEN\n RETURN j;\n END IF;\n IF jsonb_typeof(j) = 'array' THEN\n SELECT jsonb_agg(cql1_to_cql2(el)) INTO args FROM jsonb_array_elements(j) el;\n RETURN args;\n END IF;\n IF jsonb_typeof(j) = 'number' THEN\n RETURN j;\n END IF;\n IF jsonb_typeof(j) = 'string' THEN\n RETURN j;\n END IF;\n\n IF jsonb_typeof(j) = 'object' THEN\n SELECT jsonb_build_object(\n 'op', key,\n 'args', cql1_to_cql2(value)\n ) INTO ret\n FROM jsonb_each(j)\n WHERE j IS NOT NULL;\n RETURN ret;\n END IF;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE STRICT;\n\nCREATE TABLE cql2_ops (\n op text PRIMARY KEY,\n template text,\n types text[]\n);\nINSERT INTO cql2_ops (op, template, types) VALUES\n ('eq', '%s = %s', NULL),\n ('neq', '%s != %s', NULL),\n ('ne', '%s != %s', NULL),\n ('!=', '%s != %s', NULL),\n ('<>', '%s != %s', NULL),\n ('lt', '%s < %s', NULL),\n ('lte', '%s <= %s', NULL),\n ('gt', '%s > %s', NULL),\n ('gte', '%s >= %s', NULL),\n ('le', '%s <= %s', NULL),\n ('ge', '%s >= %s', NULL),\n ('=', '%s = %s', NULL),\n ('<', '%s < %s', NULL),\n ('<=', '%s <= %s', NULL),\n ('>', '%s > %s', NULL),\n ('>=', '%s >= %s', NULL),\n ('like', '%s LIKE %s', NULL),\n ('ilike', '%s ILIKE %s', NULL),\n ('+', '%s + %s', NULL),\n ('-', '%s - %s', NULL),\n ('*', '%s * %s', NULL),\n ('/', '%s / %s', NULL),\n ('not', 'NOT (%s)', NULL),\n ('between', '%s BETWEEN %s AND %s', NULL),\n ('isnull', '%s IS NULL', NULL),\n ('upper', 'upper(%s)', NULL),\n ('lower', 'lower(%s)', NULL)\nON CONFLICT (op) DO UPDATE\n SET\n template = EXCLUDED.template\n;\n\n\nCREATE OR REPLACE FUNCTION cql2_query(j jsonb, wrapper text DEFAULT NULL) RETURNS text AS $$\n#variable_conflict use_variable\nDECLARE\n args jsonb := j->'args';\n arg jsonb;\n op text := lower(j->>'op');\n cql2op RECORD;\n literal text;\n _wrapper text;\n leftarg text;\n rightarg text;\nBEGIN\n IF j IS NULL OR (op IS NOT NULL AND args IS NULL) THEN\n RETURN NULL;\n END IF;\n RAISE NOTICE 'CQL2_QUERY: %', j;\n IF j ? 'filter' THEN\n RETURN cql2_query(j->'filter');\n END IF;\n\n IF j ? 'upper' THEN\n RETURN cql2_query(jsonb_build_object('op', 'upper', 'args', j->'upper'));\n END IF;\n\n IF j ? 'lower' THEN\n RETURN cql2_query(jsonb_build_object('op', 'lower', 'args', j->'lower'));\n END IF;\n\n -- Temporal Query\n IF op ilike 't_%' or op = 'anyinteracts' THEN\n RETURN temporal_op_query(op, args);\n END IF;\n\n -- If property is a timestamp convert it to text to use with\n -- general operators\n IF j ? 'timestamp' THEN\n RETURN format('%L::timestamptz', to_tstz(j->'timestamp'));\n END IF;\n IF j ? 'interval' THEN\n RAISE EXCEPTION 'Please use temporal operators when using intervals.';\n RETURN NONE;\n END IF;\n\n -- Spatial Query\n IF op ilike 's_%' or op = 'intersects' THEN\n RETURN spatial_op_query(op, args);\n END IF;\n\n IF op IN ('a_equals','a_contains','a_contained_by','a_overlaps') THEN\n IF args->0 ? 'property' THEN\n leftarg := format('to_text_array(%s)', (queryable(args->0->>'property')).path);\n END IF;\n IF args->1 ? 'property' THEN\n rightarg := format('to_text_array(%s)', (queryable(args->1->>'property')).path);\n END IF;\n RETURN FORMAT(\n '%s %s %s',\n COALESCE(leftarg, quote_literal(to_text_array(args->0))),\n CASE op\n WHEN 'a_equals' THEN '='\n WHEN 'a_contains' THEN '@>'\n WHEN 'a_contained_by' THEN '<@'\n WHEN 'a_overlaps' THEN '&&'\n END,\n COALESCE(rightarg, quote_literal(to_text_array(args->1)))\n );\n END IF;\n\n IF op = 'in' THEN\n RAISE NOTICE 'IN : % % %', args, jsonb_build_array(args->0), args->1;\n args := jsonb_build_array(args->0) || (args->1);\n RAISE NOTICE 'IN2 : %', args;\n END IF;\n\n\n\n IF op = 'between' THEN\n args = jsonb_build_array(\n args->0,\n args->1->0,\n args->1->1\n );\n END IF;\n\n -- Make sure that args is an array and run cql2_query on\n -- each element of the array\n RAISE NOTICE 'ARGS PRE: %', args;\n IF j ? 'args' THEN\n IF jsonb_typeof(args) != 'array' THEN\n args := jsonb_build_array(args);\n END IF;\n\n IF jsonb_path_exists(args, '$[*] ? (@.property == \"id\" || @.property == \"datetime\" || @.property == \"end_datetime\" || @.property == \"collection\")') THEN\n wrapper := NULL;\n ELSE\n -- if any of the arguments are a property, try to get the property_wrapper\n FOR arg IN SELECT jsonb_path_query(args, '$[*] ? (@.property != null)') LOOP\n RAISE NOTICE 'Arg: %', arg;\n wrapper := (queryable(arg->>'property')).nulled_wrapper;\n RAISE NOTICE 'Property: %, Wrapper: %', arg, wrapper;\n IF wrapper IS NOT NULL THEN\n EXIT;\n END IF;\n END LOOP;\n\n -- if the property was not in queryables, see if any args were numbers\n IF\n wrapper IS NULL\n AND jsonb_path_exists(args, '$[*] ? (@.type()==\"number\")')\n THEN\n wrapper := 'to_float';\n END IF;\n wrapper := coalesce(wrapper, 'to_text');\n END IF;\n\n SELECT jsonb_agg(cql2_query(a, wrapper))\n INTO args\n FROM jsonb_array_elements(args) a;\n END IF;\n RAISE NOTICE 'ARGS: %', args;\n\n IF op IN ('and', 'or') THEN\n RETURN\n format(\n '(%s)',\n array_to_string(to_text_array(args), format(' %s ', upper(op)))\n );\n END IF;\n\n IF op = 'in' THEN\n RAISE NOTICE 'IN -- % %', args->0, to_text(args->0);\n RETURN format(\n '%s IN (%s)',\n to_text(args->0),\n array_to_string((to_text_array(args))[2:], ',')\n );\n END IF;\n\n -- Look up template from cql2_ops\n IF j ? 'op' THEN\n SELECT * INTO cql2op FROM cql2_ops WHERE cql2_ops.op ilike op;\n IF FOUND THEN\n -- If specific index set in queryables for a property cast other arguments to that type\n\n RETURN format(\n cql2op.template,\n VARIADIC (to_text_array(args))\n );\n ELSE\n RAISE EXCEPTION 'Operator % Not Supported.', op;\n END IF;\n END IF;\n\n\n IF wrapper IS NOT NULL THEN\n RAISE NOTICE 'Wrapping % with %', j, wrapper;\n IF j ? 'property' THEN\n RETURN format('%I(%s)', wrapper, (queryable(j->>'property')).path);\n ELSE\n RETURN format('%I(%L)', wrapper, j);\n END IF;\n ELSIF j ? 'property' THEN\n RETURN quote_ident(j->>'property');\n END IF;\n\n RETURN quote_literal(to_text(j));\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION paging_dtrange(\n j jsonb\n) RETURNS tstzrange AS $$\nDECLARE\n op text;\n filter jsonb := j->'filter';\n dtrange tstzrange := tstzrange('-infinity'::timestamptz,'infinity'::timestamptz);\n sdate timestamptz := '-infinity'::timestamptz;\n edate timestamptz := 'infinity'::timestamptz;\n jpitem jsonb;\nBEGIN\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n END IF;\n IF NOT (filter @? '$.**.op ? (@ == \"or\" || @ == \"not\")') THEN\n FOR jpitem IN SELECT j FROM jsonb_path_query(filter,'strict $.** ? (@.args[*].property == \"datetime\")'::jsonpath) j LOOP\n op := lower(jpitem->>'op');\n dtrange := parse_dtrange(jpitem->'args'->1);\n IF op IN ('<=', 'lt', 'lte', '<', 'le', 't_before') THEN\n sdate := greatest(sdate,'-infinity');\n edate := least(edate, upper(dtrange));\n ELSIF op IN ('>=', '>', 'gt', 'gte', 'ge', 't_after') THEN\n edate := least(edate, 'infinity');\n sdate := greatest(sdate, lower(dtrange));\n ELSIF op IN ('=', 'eq') THEN\n edate := least(edate, upper(dtrange));\n sdate := greatest(sdate, lower(dtrange));\n END IF;\n RAISE NOTICE '2 OP: %, ARGS: %, DTRANGE: %, SDATE: %, EDATE: %', op, jpitem->'args'->1, dtrange, sdate, edate;\n END LOOP;\n END IF;\n IF sdate > edate THEN\n RETURN 'empty'::tstzrange;\n END IF;\n RETURN tstzrange(sdate,edate, '[]');\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT SET TIME ZONE 'UTC';\n\nCREATE OR REPLACE FUNCTION paging_collections(\n IN j jsonb\n) RETURNS text[] AS $$\nDECLARE\n filter jsonb := j->'filter';\n jpitem jsonb;\n op text;\n args jsonb;\n arg jsonb;\n collections text[];\nBEGIN\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n END IF;\n IF NOT (filter @? '$.**.op ? (@ == \"or\" || @ == \"not\")') THEN\n FOR jpitem IN SELECT j FROM jsonb_path_query(filter,'strict $.** ? (@.args[*].property == \"collection\")'::jsonpath) j LOOP\n RAISE NOTICE 'JPITEM: %', jpitem;\n op := jpitem->>'op';\n args := jpitem->'args';\n IF op IN ('=', 'eq', 'in') THEN\n FOR arg IN SELECT a FROM jsonb_array_elements(args) a LOOP\n IF jsonb_typeof(arg) IN ('string', 'array') THEN\n RAISE NOTICE 'arg: %, collections: %', arg, collections;\n IF collections IS NULL OR collections = '{}'::text[] THEN\n collections := to_text_array(arg);\n ELSE\n collections := array_intersection(collections, to_text_array(arg));\n END IF;\n END IF;\n END LOOP;\n END IF;\n END LOOP;\n END IF;\n IF collections = '{}'::text[] THEN\n RETURN NULL;\n END IF;\n RETURN collections;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\nCREATE OR REPLACE FUNCTION collection_base_item(content jsonb) RETURNS jsonb AS $$\n SELECT jsonb_build_object(\n 'type', 'Feature',\n 'stac_version', content->'stac_version',\n 'assets', content->'item_assets',\n 'collection', content->'id'\n );\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TABLE IF NOT EXISTS collections (\n key bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,\n id text GENERATED ALWAYS AS (content->>'id') STORED UNIQUE,\n content JSONB NOT NULL,\n base_item jsonb GENERATED ALWAYS AS (pgstac.collection_base_item(content)) STORED,\n partition_trunc text CHECK (partition_trunc IN ('year', 'month'))\n);\n\n\n\nCREATE OR REPLACE FUNCTION collection_base_item(cid text) RETURNS jsonb AS $$\n SELECT pgstac.collection_base_item(content) FROM pgstac.collections WHERE id = cid LIMIT 1;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION create_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION update_collection(data jsonb) RETURNS VOID AS $$\nDECLARE\n out collections%ROWTYPE;\nBEGIN\n UPDATE collections SET content=data WHERE id = data->>'id' RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ON CONFLICT (id) DO\n UPDATE\n SET content=EXCLUDED.content\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION delete_collection(_id text) RETURNS VOID AS $$\nDECLARE\n out collections%ROWTYPE;\nBEGIN\n DELETE FROM collections WHERE id = _id RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION get_collection(id text) RETURNS jsonb AS $$\n SELECT content FROM collections\n WHERE id=$1\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION all_collections() RETURNS jsonb AS $$\n SELECT jsonb_agg(content) FROM collections;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\nCREATE TABLE items (\n id text NOT NULL,\n geometry geometry NOT NULL,\n collection text NOT NULL,\n datetime timestamptz NOT NULL,\n end_datetime timestamptz NOT NULL,\n content JSONB NOT NULL\n)\nPARTITION BY LIST (collection)\n;\n\nCREATE INDEX \"datetime_idx\" ON items USING BTREE (datetime DESC, end_datetime ASC);\nCREATE INDEX \"geometry_idx\" ON items USING GIST (geometry);\n\nCREATE STATISTICS datetime_stats (dependencies) on datetime, end_datetime from items;\n\nALTER TABLE items ADD CONSTRAINT items_collections_fk FOREIGN KEY (collection) REFERENCES collections(id) ON DELETE CASCADE DEFERRABLE;\n\nCREATE OR REPLACE FUNCTION partition_after_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p text;\n t timestamptz := clock_timestamp();\nBEGIN\n RAISE NOTICE 'Updating partition stats %', t;\n FOR p IN SELECT DISTINCT partition\n FROM newdata n JOIN partition_sys_meta p\n ON (n.collection=p.collection AND n.datetime <@ p.partition_dtrange)\n LOOP\n PERFORM run_or_queue(format('SELECT update_partition_stats(%L, %L);', p, true));\n END LOOP;\n IF TG_OP IN ('DELETE','UPDATE') THEN\n DELETE FROM format_item_cache c USING newdata n WHERE c.collection = n.collection AND c.id = n.id;\n END IF;\n RAISE NOTICE 't: % %', t, clock_timestamp() - t;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE TRIGGER items_after_insert_trigger\nAFTER INSERT ON items\nREFERENCING NEW TABLE AS newdata\nFOR EACH STATEMENT\nEXECUTE FUNCTION partition_after_triggerfunc();\n\nCREATE TRIGGER items_after_update_trigger\nAFTER DELETE ON items\nREFERENCING OLD TABLE AS newdata\nFOR EACH STATEMENT\nEXECUTE FUNCTION partition_after_triggerfunc();\n\nCREATE TRIGGER items_after_delete_trigger\nAFTER UPDATE ON items\nREFERENCING NEW TABLE AS newdata\nFOR EACH STATEMENT\nEXECUTE FUNCTION partition_after_triggerfunc();\n\n\nCREATE OR REPLACE FUNCTION content_slim(_item jsonb) RETURNS jsonb AS $$\n SELECT strip_jsonb(_item - '{id,geometry,collection,type}'::text[], collection_base_item(_item->>'collection')) - '{id,geometry,collection,type}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_dehydrate(content jsonb) RETURNS items AS $$\n SELECT\n content->>'id' as id,\n stac_geom(content) as geometry,\n content->>'collection' as collection,\n stac_datetime(content) as datetime,\n stac_end_datetime(content) as end_datetime,\n content_slim(content) as content\n ;\n$$ LANGUAGE SQL STABLE;\n\nCREATE OR REPLACE FUNCTION include_field(f text, fields jsonb DEFAULT '{}'::jsonb) RETURNS boolean AS $$\nDECLARE\n includes jsonb := fields->'include';\n excludes jsonb := fields->'exclude';\nBEGIN\n IF f IS NULL THEN\n RETURN NULL;\n END IF;\n\n\n IF\n jsonb_typeof(excludes) = 'array'\n AND jsonb_array_length(excludes)>0\n AND excludes ? f\n THEN\n RETURN FALSE;\n END IF;\n\n IF\n (\n jsonb_typeof(includes) = 'array'\n AND jsonb_array_length(includes) > 0\n AND includes ? f\n ) OR\n (\n includes IS NULL\n OR jsonb_typeof(includes) = 'null'\n OR jsonb_array_length(includes) = 0\n )\n THEN\n RETURN TRUE;\n END IF;\n\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nDROP FUNCTION IF EXISTS content_hydrate(jsonb, jsonb, jsonb);\nCREATE OR REPLACE FUNCTION content_hydrate(\n _item jsonb,\n _base_item jsonb,\n fields jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\n SELECT merge_jsonb(\n jsonb_fields(_item, fields),\n jsonb_fields(_base_item, fields)\n );\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION content_hydrate(_item items, _collection collections, fields jsonb DEFAULT '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\n content jsonb;\n base_item jsonb := _collection.base_item;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry, 20)::jsonb;\n END IF;\n output := content_hydrate(\n jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'collection', _item.collection,\n 'type', 'Feature'\n ) || _item.content,\n _collection.base_item,\n fields\n );\n\n RETURN output;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_nonhydrated(\n _item items,\n fields jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry, 20)::jsonb;\n END IF;\n output := jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'collection', _item.collection,\n 'type', 'Feature'\n ) || _item.content;\n RETURN output;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_hydrate(_item items, fields jsonb DEFAULT '{}'::jsonb) RETURNS jsonb AS $$\n SELECT content_hydrate(\n _item,\n (SELECT c FROM collections c WHERE id=_item.collection LIMIT 1),\n fields\n );\n$$ LANGUAGE SQL STABLE;\n\n\nCREATE UNLOGGED TABLE items_staging (\n content JSONB NOT NULL\n);\nCREATE UNLOGGED TABLE items_staging_ignore (\n content JSONB NOT NULL\n);\nCREATE UNLOGGED TABLE items_staging_upsert (\n content JSONB NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION items_staging_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p record;\n _partitions text[];\n part text;\n ts timestamptz := clock_timestamp();\nBEGIN\n RAISE NOTICE 'Creating Partitions. %', clock_timestamp() - ts;\n\n FOR part IN WITH t AS (\n SELECT\n n.content->>'collection' as collection,\n stac_daterange(n.content->'properties') as dtr,\n partition_trunc\n FROM newdata n JOIN collections ON (n.content->>'collection'=collections.id)\n ), p AS (\n SELECT\n collection,\n COALESCE(date_trunc(partition_trunc::text, lower(dtr)),'-infinity') as d,\n tstzrange(min(lower(dtr)),max(lower(dtr)),'[]') as dtrange,\n tstzrange(min(upper(dtr)),max(upper(dtr)),'[]') as edtrange\n FROM t\n GROUP BY 1,2\n ) SELECT check_partition(collection, dtrange, edtrange) FROM p LOOP\n RAISE NOTICE 'Partition %', part;\n END LOOP;\n\n RAISE NOTICE 'Doing the insert. %', clock_timestamp() - ts;\n IF TG_TABLE_NAME = 'items_staging' THEN\n INSERT INTO items\n SELECT\n (content_dehydrate(content)).*\n FROM newdata;\n RAISE NOTICE 'Doing the delete. %', clock_timestamp() - ts;\n DELETE FROM items_staging;\n ELSIF TG_TABLE_NAME = 'items_staging_ignore' THEN\n INSERT INTO items\n SELECT\n (content_dehydrate(content)).*\n FROM newdata\n ON CONFLICT DO NOTHING;\n RAISE NOTICE 'Doing the delete. %', clock_timestamp() - ts;\n DELETE FROM items_staging_ignore;\n ELSIF TG_TABLE_NAME = 'items_staging_upsert' THEN\n WITH staging_formatted AS (\n SELECT (content_dehydrate(content)).* FROM newdata\n ), deletes AS (\n DELETE FROM items i USING staging_formatted s\n WHERE\n i.id = s.id\n AND i.collection = s.collection\n AND i IS DISTINCT FROM s\n RETURNING i.id, i.collection\n )\n INSERT INTO items\n SELECT s.* FROM\n staging_formatted s\n ON CONFLICT DO NOTHING;\n RAISE NOTICE 'Doing the delete. %', clock_timestamp() - ts;\n DELETE FROM items_staging_upsert;\n END IF;\n RAISE NOTICE 'Done. %', clock_timestamp() - ts;\n\n RETURN NULL;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER items_staging_insert_trigger AFTER INSERT ON items_staging REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\nCREATE TRIGGER items_staging_insert_ignore_trigger AFTER INSERT ON items_staging_ignore REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\nCREATE TRIGGER items_staging_insert_upsert_trigger AFTER INSERT ON items_staging_upsert REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\n\nCREATE OR REPLACE FUNCTION item_by_id(_id text, _collection text DEFAULT NULL) RETURNS items AS\n$$\nDECLARE\n i items%ROWTYPE;\nBEGIN\n SELECT * INTO i FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection) LIMIT 1;\n RETURN i;\nEND;\n$$ LANGUAGE PLPGSQL STABLE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION get_item(_id text, _collection text DEFAULT NULL) RETURNS jsonb AS $$\n SELECT content_hydrate(items) FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection);\n$$ LANGUAGE SQL STABLE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION delete_item(_id text, _collection text DEFAULT NULL) RETURNS VOID AS $$\nDECLARE\nout items%ROWTYPE;\nBEGIN\n DELETE FROM items WHERE id = _id AND (_collection IS NULL OR collection=_collection) RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n--/*\nCREATE OR REPLACE FUNCTION create_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION update_item(content jsonb) RETURNS VOID AS $$\nDECLARE\n old items %ROWTYPE;\n out items%ROWTYPE;\nBEGIN\n PERFORM delete_item(content->>'id', content->>'collection');\n PERFORM create_item(content);\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION create_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION collection_bbox(id text) RETURNS jsonb AS $$\n SELECT (replace(replace(replace(st_extent(geometry)::text,'BOX(','[['),')',']]'),' ',','))::jsonb\n FROM items WHERE collection=$1;\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION collection_temporal_extent(id text) RETURNS jsonb AS $$\n SELECT to_jsonb(array[array[min(datetime)::text, max(datetime)::text]])\n FROM items WHERE collection=$1;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection_extents() RETURNS VOID AS $$\nUPDATE collections SET\n content = content ||\n jsonb_build_object(\n 'extent', jsonb_build_object(\n 'spatial', jsonb_build_object(\n 'bbox', collection_bbox(collections.id)\n ),\n 'temporal', jsonb_build_object(\n 'interval', collection_temporal_extent(collections.id)\n )\n )\n )\n;\n$$ LANGUAGE SQL;\nCREATE TABLE partition_stats (\n partition text PRIMARY KEY,\n dtrange tstzrange,\n edtrange tstzrange,\n spatial geometry,\n last_updated timestamptz,\n keys text[]\n) WITH (FILLFACTOR=90);\n\nCREATE INDEX partitions_range_idx ON partition_stats USING GIST(dtrange);\n\n\nCREATE OR REPLACE FUNCTION constraint_tstzrange(expr text) RETURNS tstzrange AS $$\n WITH t AS (\n SELECT regexp_matches(\n expr,\n E'\\\\(''\\([0-9 :+-]*\\)''\\\\).*\\\\(''\\([0-9 :+-]*\\)''\\\\)'\n ) AS m\n ) SELECT tstzrange(m[1]::timestamptz, m[2]::timestamptz) FROM t\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE STRICT;\n\nCREATE OR REPLACE FUNCTION dt_constraint(coid oid, OUT dt tstzrange, OUT edt tstzrange) RETURNS RECORD AS $$\nDECLARE\n expr text := pg_get_constraintdef(coid);\n matches timestamptz[];\nBEGIN\n IF expr LIKE '%NULL%' THEN\n dt := tstzrange(null::timestamptz, null::timestamptz);\n edt := tstzrange(null::timestamptz, null::timestamptz);\n RETURN;\n END IF;\n WITH f AS (SELECT (regexp_matches(expr, E'([0-9]{4}-[0-1][0-9]-[0-3][0-9] [0-2][0-9]:[0-5][0-9]:[0-5][0-9]\\.?[0-9]*)', 'g'))[1] f)\n SELECT array_agg(f::timestamptz) INTO matches FROM f;\n IF cardinality(matches) = 4 THEN\n dt := tstzrange(matches[1], matches[2],'[]');\n edt := tstzrange(matches[3], matches[4], '[]');\n RETURN;\n ELSIF cardinality(matches) = 2 THEN\n edt := tstzrange(matches[1], matches[2],'[]');\n RETURN;\n END IF;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\nCREATE OR REPLACE VIEW partition_sys_meta AS\nSELECT\n relid::text as partition,\n replace(replace(CASE WHEN level = 1 THEN pg_get_expr(c.relpartbound, c.oid)\n ELSE pg_get_expr(parent.relpartbound, parent.oid)\n END, 'FOR VALUES IN (''',''), ''')','') AS collection,\n level,\n c.reltuples,\n c.relhastriggers,\n COALESCE(constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as partition_dtrange,\n COALESCE((dt_constraint(edt.oid)).dt, constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as constraint_dtrange,\n COALESCE((dt_constraint(edt.oid)).edt, tstzrange('-infinity', 'infinity','[]')) as constraint_edtrange\nFROM\n pg_partition_tree('items')\n JOIN pg_class c ON (relid::regclass = c.oid)\n JOIN pg_class parent ON (parentrelid::regclass = parent.oid AND isleaf)\n LEFT JOIN pg_constraint edt ON (conrelid=c.oid AND contype='c')\nWHERE isleaf\n;\n\nCREATE VIEW partitions_view AS\nSELECT\n relid::text as partition,\n replace(replace(CASE WHEN level = 1 THEN pg_get_expr(c.relpartbound, c.oid)\n ELSE pg_get_expr(parent.relpartbound, parent.oid)\n END, 'FOR VALUES IN (''',''), ''')','') AS collection,\n level,\n c.reltuples,\n c.relhastriggers,\n COALESCE(constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as partition_dtrange,\n COALESCE((dt_constraint(edt.oid)).dt, constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as constraint_dtrange,\n COALESCE((dt_constraint(edt.oid)).edt, tstzrange('-infinity', 'infinity','[]')) as constraint_edtrange,\n dtrange,\n edtrange,\n spatial,\n last_updated\nFROM\n pg_partition_tree('items')\n JOIN pg_class c ON (relid::regclass = c.oid)\n JOIN pg_class parent ON (parentrelid::regclass = parent.oid AND isleaf)\n LEFT JOIN pg_constraint edt ON (conrelid=c.oid AND contype='c')\n LEFT JOIN partition_stats ON (relid::text=partition)\nWHERE isleaf\n;\n\nCREATE MATERIALIZED VIEW partitions AS\nSELECT * FROM partitions_view;\nCREATE UNIQUE INDEX ON partitions (partition);\n\nCREATE MATERIALIZED VIEW partition_steps AS\nSELECT\n partition as name,\n date_trunc('month',lower(partition_dtrange)) as sdate,\n date_trunc('month', upper(partition_dtrange)) + '1 month'::interval as edate\n FROM partitions_view WHERE partition_dtrange IS NOT NULL AND partition_dtrange != 'empty'::tstzrange\n ORDER BY dtrange ASC\n;\n\n\nCREATE OR REPLACE FUNCTION update_partition_stats_q(_partition text, istrigger boolean default false) RETURNS VOID AS $$\nDECLARE\nBEGIN\n PERFORM run_or_queue(\n format('SELECT update_partition_stats(%L, %L);', _partition, istrigger)\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION update_partition_stats(_partition text, istrigger boolean default false) RETURNS VOID AS $$\nDECLARE\n dtrange tstzrange;\n edtrange tstzrange;\n cdtrange tstzrange;\n cedtrange tstzrange;\n extent geometry;\n collection text;\nBEGIN\n RAISE NOTICE 'Updating stats for %.', _partition;\n EXECUTE format(\n $q$\n SELECT\n tstzrange(min(datetime), max(datetime),'[]'),\n tstzrange(min(end_datetime), max(end_datetime), '[]')\n FROM %I\n $q$,\n _partition\n ) INTO dtrange, edtrange;\n extent := st_estimatedextent('pgstac', _partition, 'geometry');\n INSERT INTO partition_stats (partition, dtrange, edtrange, spatial, last_updated)\n SELECT _partition, dtrange, edtrange, extent, now()\n ON CONFLICT (partition) DO\n UPDATE SET\n dtrange=EXCLUDED.dtrange,\n edtrange=EXCLUDED.edtrange,\n spatial=EXCLUDED.spatial,\n last_updated=EXCLUDED.last_updated\n ;\n\n SELECT\n constraint_dtrange, constraint_edtrange, pv.collection\n INTO cdtrange, cedtrange, collection\n FROM partitions_view pv WHERE partition = _partition;\n REFRESH MATERIALIZED VIEW partitions;\n REFRESH MATERIALIZED VIEW partition_steps;\n\n\n RAISE NOTICE 'Checking if we need to modify constraints.';\n IF\n (cdtrange IS DISTINCT FROM dtrange OR edtrange IS DISTINCT FROM cedtrange)\n AND NOT istrigger\n THEN\n RAISE NOTICE 'Modifying Constraints';\n RAISE NOTICE 'Existing % %', cdtrange, cedtrange;\n RAISE NOTICE 'New % %', dtrange, edtrange;\n PERFORM drop_table_constraints(_partition);\n PERFORM create_table_constraints(_partition, dtrange, edtrange);\n END IF;\n RAISE NOTICE 'Checking if we need to update collection extents.';\n IF get_setting_bool('update_collection_extent') THEN\n RAISE NOTICE 'updating collection extent for %', collection;\n PERFORM run_or_queue(format($q$\n UPDATE collections\n SET content = jsonb_set_lax(\n content,\n '{extent}'::text[],\n collection_extent(%L, FALSE),\n true,\n 'use_json_null'\n ) WHERE id=%L\n ;\n $q$, collection, collection));\n ELSE\n RAISE NOTICE 'Not updating collection extent for %', collection;\n END IF;\n\nEND;\n$$ LANGUAGE PLPGSQL STRICT;\n\n\nCREATE OR REPLACE FUNCTION partition_name( IN collection text, IN dt timestamptz, OUT partition_name text, OUT partition_range tstzrange) AS $$\nDECLARE\n c RECORD;\n parent_name text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n parent_name := format('_items_%s', c.key);\n\n\n IF c.partition_trunc = 'year' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYY'));\n ELSIF c.partition_trunc = 'month' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYYMM'));\n ELSE\n partition_name := parent_name;\n partition_range := tstzrange('-infinity'::timestamptz, 'infinity'::timestamptz, '[]');\n END IF;\n IF partition_range IS NULL THEN\n partition_range := tstzrange(\n date_trunc(c.partition_trunc::text, dt),\n date_trunc(c.partition_trunc::text, dt) + concat('1 ', c.partition_trunc)::interval\n );\n END IF;\n RETURN;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION drop_table_constraints(t text) RETURNS text AS $$\nDECLARE\n q text;\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM partitions_view WHERE partition=t) THEN\n RETURN NULL;\n END IF;\n FOR q IN SELECT FORMAT(\n $q$\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n $q$,\n t,\n conname\n ) FROM pg_constraint\n WHERE conrelid=t::regclass::oid AND contype='c'\n LOOP\n EXECUTE q;\n END LOOP;\n RETURN t;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE OR REPLACE FUNCTION create_table_constraints(t text, _dtrange tstzrange, _edtrange tstzrange) RETURNS text AS $$\nDECLARE\n q text;\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM partitions_view WHERE partition=t) THEN\n RETURN NULL;\n END IF;\n RAISE NOTICE 'Creating Table Constraints for % % %', t, _dtrange, _edtrange;\n IF _dtrange = 'empty' AND _edtrange = 'empty' THEN\n q :=format(\n $q$\n DO $block$\n BEGIN\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n ALTER TABLE %I\n ADD CONSTRAINT %I\n CHECK (((datetime IS NULL) AND (end_datetime IS NULL))) NOT VALID\n ;\n ALTER TABLE %I\n VALIDATE CONSTRAINT %I\n ;\n\n\n\n EXCEPTION WHEN others THEN\n RAISE WARNING '%%, Issue Altering Constraints. Please run update_partition_stats(%I)', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n $block$;\n $q$,\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t),\n t\n );\n ELSE\n q :=format(\n $q$\n DO $block$\n BEGIN\n\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n ALTER TABLE %I\n ADD CONSTRAINT %I\n CHECK (\n (datetime >= %L)\n AND (datetime <= %L)\n AND (end_datetime >= %L)\n AND (end_datetime <= %L)\n ) NOT VALID\n ;\n ALTER TABLE %I\n VALIDATE CONSTRAINT %I\n ;\n\n\n\n EXCEPTION WHEN others THEN\n RAISE WARNING '%%, Issue Altering Constraints. Please run update_partition_stats(%I)', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n $block$;\n $q$,\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t),\n lower(_dtrange),\n upper(_dtrange),\n lower(_edtrange),\n upper(_edtrange),\n t,\n format('%s_dt', t),\n t\n );\n END IF;\n PERFORM run_or_queue(q);\n RETURN t;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION check_partition(\n _collection text,\n _dtrange tstzrange,\n _edtrange tstzrange\n) RETURNS text AS $$\nDECLARE\n c RECORD;\n pm RECORD;\n _partition_name text;\n _partition_dtrange tstzrange;\n _constraint_dtrange tstzrange;\n _constraint_edtrange tstzrange;\n q text;\n deferrable_q text;\n err_context text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=_collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', _collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n\n IF c.partition_trunc IS NOT NULL THEN\n _partition_dtrange := tstzrange(\n date_trunc(c.partition_trunc, lower(_dtrange)),\n date_trunc(c.partition_trunc, lower(_dtrange)) + (concat('1 ', c.partition_trunc))::interval,\n '[)'\n );\n ELSE\n _partition_dtrange := '[-infinity, infinity]'::tstzrange;\n END IF;\n\n IF NOT _partition_dtrange @> _dtrange THEN\n RAISE EXCEPTION 'dtrange % is greater than the partition size % for collection %', _dtrange, c.partition_trunc, _collection;\n END IF;\n\n\n IF c.partition_trunc = 'year' THEN\n _partition_name := format('_items_%s_%s', c.key, to_char(lower(_partition_dtrange),'YYYY'));\n ELSIF c.partition_trunc = 'month' THEN\n _partition_name := format('_items_%s_%s', c.key, to_char(lower(_partition_dtrange),'YYYYMM'));\n ELSE\n _partition_name := format('_items_%s', c.key);\n END IF;\n\n SELECT * INTO pm FROM partition_sys_meta WHERE collection=_collection AND partition_dtrange @> _dtrange;\n IF FOUND THEN\n RAISE NOTICE '% % %', _edtrange, _dtrange, pm;\n _constraint_edtrange :=\n tstzrange(\n least(\n lower(_edtrange),\n nullif(lower(pm.constraint_edtrange), '-infinity')\n ),\n greatest(\n upper(_edtrange),\n nullif(upper(pm.constraint_edtrange), 'infinity')\n ),\n '[]'\n );\n _constraint_dtrange :=\n tstzrange(\n least(\n lower(_dtrange),\n nullif(lower(pm.constraint_dtrange), '-infinity')\n ),\n greatest(\n upper(_dtrange),\n nullif(upper(pm.constraint_dtrange), 'infinity')\n ),\n '[]'\n );\n\n IF pm.constraint_edtrange @> _edtrange AND pm.constraint_dtrange @> _dtrange THEN\n RETURN pm.partition;\n ELSE\n PERFORM drop_table_constraints(_partition_name);\n END IF;\n ELSE\n _constraint_edtrange := _edtrange;\n _constraint_dtrange := _dtrange;\n END IF;\n RAISE NOTICE 'Creating partition % %', _partition_name, _partition_dtrange;\n IF c.partition_trunc IS NULL THEN\n q := format(\n $q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n $q$,\n _partition_name,\n _collection,\n concat(_partition_name,'_pk'),\n _partition_name\n );\n ELSE\n q := format(\n $q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L) PARTITION BY RANGE (datetime);\n CREATE TABLE IF NOT EXISTS %I partition OF %I FOR VALUES FROM (%L) TO (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n $q$,\n format('_items_%s', c.key),\n _collection,\n _partition_name,\n format('_items_%s', c.key),\n lower(_partition_dtrange),\n upper(_partition_dtrange),\n format('%s_pk', _partition_name),\n _partition_name\n );\n END IF;\n\n BEGIN\n EXECUTE q;\n EXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', _partition_name;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\n END;\n PERFORM create_table_constraints(_partition_name, _constraint_dtrange, _constraint_edtrange);\n PERFORM maintain_partitions(_partition_name);\n PERFORM update_partition_stats_q(_partition_name, true);\n REFRESH MATERIALIZED VIEW partitions;\n REFRESH MATERIALIZED VIEW partition_steps;\n RETURN _partition_name;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION repartition(_collection text, _partition_trunc text, triggered boolean DEFAULT FALSE) RETURNS text AS $$\nDECLARE\n c RECORD;\n q text;\n from_trunc text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=_collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', _collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n IF triggered THEN\n RAISE NOTICE 'Converting % to % partitioning via Trigger', _collection, _partition_trunc;\n ELSE\n RAISE NOTICE 'Converting % from using % to % partitioning', _collection, c.partition_trunc, _partition_trunc;\n IF c.partition_trunc IS NOT DISTINCT FROM _partition_trunc THEN\n RAISE NOTICE 'Collection % already set to use partition by %', _collection, _partition_trunc;\n RETURN _collection;\n END IF;\n END IF;\n\n IF EXISTS (SELECT 1 FROM partitions_view WHERE collection=_collection LIMIT 1) THEN\n EXECUTE format(\n $q$\n CREATE TEMP TABLE changepartitionstaging ON COMMIT DROP AS SELECT * FROM %I;\n DROP TABLE IF EXISTS %I CASCADE;\n WITH p AS (\n SELECT\n collection,\n CASE WHEN %L IS NULL THEN '-infinity'::timestamptz\n ELSE date_trunc(%L, datetime)\n END as d,\n tstzrange(min(datetime),max(datetime),'[]') as dtrange,\n tstzrange(min(datetime),max(datetime),'[]') as edtrange\n FROM changepartitionstaging\n GROUP BY 1,2\n ) SELECT check_partition(collection, dtrange, edtrange) FROM p;\n INSERT INTO items SELECT * FROM changepartitionstaging;\n DROP TABLE changepartitionstaging;\n $q$,\n concat('_items_', c.key),\n concat('_items_', c.key),\n c.partition_trunc,\n c.partition_trunc\n );\n END IF;\n RETURN _collection;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE OR REPLACE FUNCTION collections_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n q text;\n partition_name text := format('_items_%s', NEW.key);\n partition_exists boolean := false;\n partition_empty boolean := true;\n err_context text;\n loadtemp boolean := FALSE;\nBEGIN\n RAISE NOTICE 'Collection Trigger. % %', NEW.id, NEW.key;\n IF TG_OP = 'UPDATE' AND NEW.partition_trunc IS DISTINCT FROM OLD.partition_trunc THEN\n PERFORM repartition(NEW.id, NEW.partition_trunc, TRUE);\n END IF;\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER collections_trigger AFTER\nINSERT\nOR\nUPDATE ON collections\nFOR EACH ROW EXECUTE FUNCTION collections_trigger_func();\n\nCREATE OR REPLACE FUNCTION chunker(\n IN _where text,\n OUT s timestamptz,\n OUT e timestamptz\n) RETURNS SETOF RECORD AS $$\nDECLARE\n explain jsonb;\nBEGIN\n IF _where IS NULL THEN\n _where := ' TRUE ';\n END IF;\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s;', _where)\n INTO explain;\n\n RETURN QUERY\n WITH t AS (\n SELECT j->>0 as p FROM\n jsonb_path_query(\n explain,\n 'strict $.**.\"Relation Name\" ? (@ != null)'\n ) j\n ),\n parts AS (\n SELECT sdate, edate FROM t JOIN partition_steps ON (t.p = name)\n ),\n times AS (\n SELECT sdate FROM parts\n UNION\n SELECT edate FROM parts\n ),\n uniq AS (\n SELECT DISTINCT sdate FROM times ORDER BY sdate\n ),\n last AS (\n SELECT sdate, lead(sdate, 1) over () as edate FROM uniq\n )\n SELECT sdate, edate FROM last WHERE edate IS NOT NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION partition_queries(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN partitions text[] DEFAULT NULL\n) RETURNS SETOF text AS $$\nDECLARE\n query text;\n sdate timestamptz;\n edate timestamptz;\nBEGIN\nIF _where IS NULL OR trim(_where) = '' THEN\n _where = ' TRUE ';\nEND IF;\nRAISE NOTICE 'Getting chunks for % %', _where, _orderby;\nIF _orderby ILIKE 'datetime d%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 DESC LOOP\n RETURN NEXT format($q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n $q$,\n sdate,\n edate,\n _where,\n _orderby\n );\n END LOOP;\nELSIF _orderby ILIKE 'datetime a%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 ASC LOOP\n RETURN NEXT format($q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n $q$,\n sdate,\n edate,\n _where,\n _orderby\n );\n END LOOP;\nELSE\n query := format($q$\n SELECT * FROM items\n WHERE %s\n ORDER BY %s\n $q$, _where, _orderby\n );\n\n RETURN NEXT query;\n RETURN;\nEND IF;\n\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION partition_query_view(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN _limit int DEFAULT 10\n) RETURNS text AS $$\n WITH p AS (\n SELECT * FROM partition_queries(_where, _orderby) p\n )\n SELECT\n CASE WHEN EXISTS (SELECT 1 FROM p) THEN\n (SELECT format($q$\n SELECT * FROM (\n %s\n ) total LIMIT %s\n $q$,\n string_agg(\n format($q$ SELECT * FROM ( %s ) AS sub $q$, p),\n '\n UNION ALL\n '\n ),\n _limit\n ))\n ELSE NULL\n END FROM p;\n$$ LANGUAGE SQL IMMUTABLE;\n\n\n\n\nCREATE OR REPLACE FUNCTION stac_search_to_where(j jsonb) RETURNS text AS $$\nDECLARE\n where_segments text[];\n _where text;\n dtrange tstzrange;\n collections text[];\n geom geometry;\n sdate timestamptz;\n edate timestamptz;\n filterlang text;\n filter jsonb := j->'filter';\nBEGIN\n IF j ? 'ids' THEN\n where_segments := where_segments || format('id = ANY (%L) ', to_text_array(j->'ids'));\n END IF;\n\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n where_segments := where_segments || format('collection = ANY (%L) ', collections);\n END IF;\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n\n where_segments := where_segments || format(' datetime <= %L::timestamptz AND end_datetime >= %L::timestamptz ',\n edate,\n sdate\n );\n END IF;\n\n geom := stac_geom(j);\n IF geom IS NOT NULL THEN\n where_segments := where_segments || format('st_intersects(geometry, %L)',geom);\n END IF;\n\n filterlang := COALESCE(\n j->>'filter-lang',\n get_setting('default_filter_lang', j->'conf')\n );\n IF NOT filter @? '$.**.op' THEN\n filterlang := 'cql-json';\n END IF;\n\n IF filterlang NOT IN ('cql-json','cql2-json') AND j ? 'filter' THEN\n RAISE EXCEPTION '% is not a supported filter-lang. Please use cql-json or cql2-json.', filterlang;\n END IF;\n\n IF j ? 'query' AND j ? 'filter' THEN\n RAISE EXCEPTION 'Can only use either query or filter at one time.';\n END IF;\n\n IF j ? 'query' THEN\n filter := query_to_cql2(j->'query');\n ELSIF filterlang = 'cql-json' THEN\n filter := cql1_to_cql2(filter);\n END IF;\n RAISE NOTICE 'FILTER: %', filter;\n where_segments := where_segments || cql2_query(filter);\n IF cardinality(where_segments) < 1 THEN\n RETURN ' TRUE ';\n END IF;\n\n _where := array_to_string(array_remove(where_segments, NULL), ' AND ');\n\n IF _where IS NULL OR BTRIM(_where) = '' THEN\n RETURN ' TRUE ';\n END IF;\n RETURN _where;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION parse_sort_dir(_dir text, reverse boolean default false) RETURNS text AS $$\n WITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n ) SELECT\n CASE\n WHEN NOT reverse THEN d\n WHEN d = 'ASC' THEN 'DESC'\n WHEN d = 'DESC' THEN 'ASC'\n END\n FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION sort_dir_to_op(_dir text, prev boolean default false) RETURNS text AS $$\n WITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n ) SELECT\n CASE\n WHEN d = 'ASC' AND prev THEN '<='\n WHEN d = 'DESC' AND prev THEN '>='\n WHEN d = 'ASC' THEN '>='\n WHEN d = 'DESC' THEN '<='\n END\n FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION sort_sqlorderby(\n _search jsonb DEFAULT NULL,\n reverse boolean DEFAULT FALSE\n) RETURNS text AS $$\n WITH sortby AS (\n SELECT coalesce(_search->'sortby','[{\"field\":\"datetime\", \"direction\":\"desc\"}]') as sort\n ), withid AS (\n SELECT CASE\n WHEN sort @? '$[*] ? (@.field == \"id\")' THEN sort\n ELSE sort || '[{\"field\":\"id\", \"direction\":\"desc\"}]'::jsonb\n END as sort\n FROM sortby\n ), withid_rows AS (\n SELECT jsonb_array_elements(sort) as value FROM withid\n ),sorts AS (\n SELECT\n coalesce(\n (queryable(value->>'field')).expression\n ) as key,\n parse_sort_dir(value->>'direction', reverse) as dir\n FROM withid_rows\n )\n SELECT array_to_string(\n array_agg(concat(key, ' ', dir)),\n ', '\n ) FROM sorts;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_sort_dir(sort_item jsonb) RETURNS text AS $$\n SELECT CASE WHEN sort_item->>'direction' ILIKE 'desc%' THEN 'DESC' ELSE 'ASC' END;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION get_token_val_str(\n _field text,\n _item items\n) RETURNS text AS $$\nDECLARE\n q text;\n literal text;\nBEGIN\n q := format($q$ SELECT quote_literal(%s) FROM (SELECT $1.*) as r;$q$, _field);\n EXECUTE q INTO literal USING _item;\n RETURN literal;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n\nCREATE OR REPLACE FUNCTION get_token_record(IN _token text, OUT prev BOOLEAN, OUT item items) RETURNS RECORD AS $$\nDECLARE\n _itemid text := _token;\n _collectionid text;\nBEGIN\n IF _token IS NULL THEN\n RETURN;\n END IF;\n RAISE NOTICE 'Looking for token: %', _token;\n prev := FALSE;\n IF _token ILIKE 'prev:%' THEN\n _itemid := replace(_token, 'prev:','');\n prev := TRUE;\n ELSIF _token ILIKE 'next:%' THEN\n _itemid := replace(_token, 'next:', '');\n END IF;\n SELECT id INTO _collectionid FROM collections WHERE _itemid LIKE concat(id,':%');\n IF FOUND THEN\n _itemid := replace(_itemid, concat(_collectionid,':'), '');\n SELECT * INTO item FROM items WHERE id=_itemid AND collection=_collectionid;\n ELSE\n SELECT * INTO item FROM items WHERE id=_itemid;\n END IF;\n IF item IS NULL THEN\n RAISE EXCEPTION 'Could not find item using token: % item: % collection: %', _token, _itemid, _collectionid;\n END IF;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\n\nCREATE OR REPLACE FUNCTION get_token_filter(\n _sortby jsonb DEFAULT '[{\"field\":\"datetime\",\"direction\":\"desc\"}]'::jsonb,\n token_item items DEFAULT NULL,\n prev boolean DEFAULT FALSE,\n inclusive boolean DEFAULT FALSE\n) RETURNS text AS $$\nDECLARE\n ltop text := '<';\n gtop text := '>';\n dir text;\n sort record;\n orfilter text := '';\n orfilters text[] := '{}'::text[];\n andfilters text[] := '{}'::text[];\n output text;\n token_where text;\nBEGIN\n IF _sortby IS NULL OR _sortby = '[]'::jsonb THEN\n _sortby := '[{\"field\":\"datetime\",\"direction\":\"desc\"}]'::jsonb;\n END IF;\n _sortby := _sortby || jsonb_build_object('field','id','direction',_sortby->0->>'direction');\n RAISE NOTICE 'Getting Token Filter. % %', _sortby, token_item;\n IF inclusive THEN\n orfilters := orfilters || format('( id=%L AND collection=%L )' , token_item.id, token_item.collection);\n END IF;\n\n FOR sort IN\n WITH s1 AS (\n SELECT\n _row,\n (queryable(value->>'field')).expression as _field,\n (value->>'field' = 'id') as _isid,\n get_sort_dir(value) as _dir\n FROM jsonb_array_elements(_sortby)\n WITH ORDINALITY AS t(value, _row)\n )\n SELECT\n _row,\n _field,\n _dir,\n get_token_val_str(_field, token_item) as _val\n FROM s1\n WHERE _row <= (SELECT min(_row) FROM s1 WHERE _isid)\n LOOP\n orfilter := NULL;\n RAISE NOTICE 'SORT: %', sort;\n IF sort._val IS NOT NULL AND ((prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC')) THEN\n orfilter := format($f$(\n (%s %s %s) OR (%s IS NULL)\n )$f$,\n sort._field,\n ltop,\n sort._val,\n sort._val\n );\n ELSIF sort._val IS NULL AND ((prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC')) THEN\n RAISE NOTICE '< but null';\n orfilter := format('%s IS NOT NULL', sort._field);\n ELSIF sort._val IS NULL THEN\n RAISE NOTICE '> but null';\n ELSE\n orfilter := format($f$(\n (%s %s %s) OR (%s IS NULL)\n )$f$,\n sort._field,\n gtop,\n sort._val,\n sort._field\n );\n END IF;\n RAISE NOTICE 'ORFILTER: %', orfilter;\n\n IF orfilter IS NOT NULL THEN\n IF sort._row = 1 THEN\n orfilters := orfilters || orfilter;\n ELSE\n orfilters := orfilters || format('(%s AND %s)', array_to_string(andfilters, ' AND '), orfilter);\n END IF;\n END IF;\n IF sort._val IS NOT NULL THEN\n andfilters := andfilters || format('%s = %s', sort._field, sort._val);\n ELSE\n andfilters := andfilters || format('%s IS NULL', sort._field);\n END IF;\n END LOOP;\n\n output := array_to_string(orfilters, ' OR ');\n\n token_where := concat('(',coalesce(output,'true'),')');\n IF trim(token_where) = '' THEN\n token_where := NULL;\n END IF;\n RAISE NOTICE 'TOKEN_WHERE: %',token_where;\n RETURN token_where;\n END;\n$$ LANGUAGE PLPGSQL SET transform_null_equals TO TRUE\n;\n\nCREATE OR REPLACE FUNCTION search_tohash(jsonb) RETURNS jsonb AS $$\n SELECT $1 - '{token,limit,context,includes,excludes}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION search_hash(jsonb, jsonb) RETURNS text AS $$\n SELECT md5(concat(search_tohash($1)::text,$2::text));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TABLE IF NOT EXISTS searches(\n hash text GENERATED ALWAYS AS (search_hash(search, metadata)) STORED PRIMARY KEY,\n search jsonb NOT NULL,\n _where text,\n orderby text,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n metadata jsonb DEFAULT '{}'::jsonb NOT NULL\n);\n\nCREATE TABLE IF NOT EXISTS search_wheres(\n id bigint generated always as identity primary key,\n _where text NOT NULL,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n statslastupdated timestamptz,\n estimated_count bigint,\n estimated_cost float,\n time_to_estimate float,\n total_count bigint,\n time_to_count float,\n partitions text[]\n);\n\nCREATE INDEX IF NOT EXISTS search_wheres_partitions ON search_wheres USING GIN (partitions);\nCREATE UNIQUE INDEX IF NOT EXISTS search_wheres_where ON search_wheres ((md5(_where)));\n\nCREATE OR REPLACE FUNCTION where_stats(inwhere text, updatestats boolean default false, conf jsonb default null) RETURNS search_wheres AS $$\nDECLARE\n t timestamptz;\n i interval;\n explain_json jsonb;\n partitions text[];\n sw search_wheres%ROWTYPE;\n inwhere_hash text := md5(inwhere);\n _context text := lower(context(conf));\n _stats_ttl interval := context_stats_ttl(conf);\n _estimated_cost float := context_estimated_cost(conf);\n _estimated_count int := context_estimated_count(conf);\nBEGIN\n IF _context = 'off' THEN\n sw._where := inwhere;\n return sw;\n END IF;\n\n SELECT * INTO sw FROM search_wheres WHERE md5(_where)=inwhere_hash FOR UPDATE;\n\n -- Update statistics if explicitly set, if statistics do not exist, or statistics ttl has expired\n IF NOT updatestats THEN\n RAISE NOTICE 'Checking if update is needed for: % .', inwhere;\n RAISE NOTICE 'Stats Last Updated: %', sw.statslastupdated;\n RAISE NOTICE 'TTL: %, Age: %', _stats_ttl, now() - sw.statslastupdated;\n RAISE NOTICE 'Context: %, Existing Total: %', _context, sw.total_count;\n IF\n sw.statslastupdated IS NULL\n OR (now() - sw.statslastupdated) > _stats_ttl\n OR (context(conf) != 'off' AND sw.total_count IS NULL)\n THEN\n updatestats := TRUE;\n END IF;\n END IF;\n\n sw._where := inwhere;\n sw.lastused := now();\n sw.usecount := coalesce(sw.usecount,0) + 1;\n\n IF NOT updatestats THEN\n UPDATE search_wheres SET\n lastused = sw.lastused,\n usecount = sw.usecount\n WHERE md5(_where) = inwhere_hash\n RETURNING * INTO sw\n ;\n RETURN sw;\n END IF;\n\n -- Use explain to get estimated count/cost and a list of the partitions that would be hit by the query\n t := clock_timestamp();\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s', inwhere)\n INTO explain_json;\n RAISE NOTICE 'Time for just the explain: %', clock_timestamp() - t;\n i := clock_timestamp() - t;\n\n sw.statslastupdated := now();\n sw.estimated_count := explain_json->0->'Plan'->'Plan Rows';\n sw.estimated_cost := explain_json->0->'Plan'->'Total Cost';\n sw.time_to_estimate := extract(epoch from i);\n\n RAISE NOTICE 'ESTIMATED_COUNT: % < %', sw.estimated_count, _estimated_count;\n RAISE NOTICE 'ESTIMATED_COST: % < %', sw.estimated_cost, _estimated_cost;\n\n -- Do a full count of rows if context is set to on or if auto is set and estimates are low enough\n IF\n _context = 'on'\n OR\n ( _context = 'auto' AND\n (\n sw.estimated_count < _estimated_count\n AND\n sw.estimated_cost < _estimated_cost\n )\n )\n THEN\n t := clock_timestamp();\n RAISE NOTICE 'Calculating actual count...';\n EXECUTE format(\n 'SELECT count(*) FROM items WHERE %s',\n inwhere\n ) INTO sw.total_count;\n i := clock_timestamp() - t;\n RAISE NOTICE 'Actual Count: % -- %', sw.total_count, i;\n sw.time_to_count := extract(epoch FROM i);\n ELSE\n sw.total_count := NULL;\n sw.time_to_count := NULL;\n END IF;\n\n\n INSERT INTO search_wheres\n (_where, lastused, usecount, statslastupdated, estimated_count, estimated_cost, time_to_estimate, partitions, total_count, time_to_count)\n SELECT sw._where, sw.lastused, sw.usecount, sw.statslastupdated, sw.estimated_count, sw.estimated_cost, sw.time_to_estimate, sw.partitions, sw.total_count, sw.time_to_count\n ON CONFLICT ((md5(_where)))\n DO UPDATE\n SET\n lastused = sw.lastused,\n usecount = sw.usecount,\n statslastupdated = sw.statslastupdated,\n estimated_count = sw.estimated_count,\n estimated_cost = sw.estimated_cost,\n time_to_estimate = sw.time_to_estimate,\n total_count = sw.total_count,\n time_to_count = sw.time_to_count\n ;\n RETURN sw;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION search_query(\n _search jsonb = '{}'::jsonb,\n updatestats boolean = false,\n _metadata jsonb = '{}'::jsonb\n) RETURNS searches AS $$\nDECLARE\n search searches%ROWTYPE;\n pexplain jsonb;\n t timestamptz;\n i interval;\n _hash text := search_hash(_search, _metadata);\n doupdate boolean := FALSE;\n insertfound boolean := FALSE;\nBEGIN\n SELECT * INTO search FROM searches\n WHERE hash=_hash;\n\n search.hash := _hash;\n\n -- Calculate the where clause if not already calculated\n IF search._where IS NULL THEN\n search._where := stac_search_to_where(_search);\n ELSE\n doupdate := TRUE;\n END IF;\n\n -- Calculate the order by clause if not already calculated\n IF search.orderby IS NULL THEN\n search.orderby := sort_sqlorderby(_search);\n ELSE\n doupdate := TRUE;\n END IF;\n\n PERFORM where_stats(search._where, updatestats, _search->'conf');\n\n IF NOT doupdate THEN\n INSERT INTO searches (search, _where, orderby, lastused, usecount, metadata)\n VALUES (_search, search._where, search.orderby, clock_timestamp(), 1, _metadata)\n ON CONFLICT (hash) DO NOTHING;\n IF FOUND THEN\n RETURN search;\n END IF;\n END IF;\n\n UPDATE searches\n SET\n lastused=clock_timestamp(),\n usecount=usecount+1\n WHERE hash=(\n SELECT hash FROM searches\n WHERE hash=_hash\n FOR UPDATE SKIP LOCKED\n );\n IF NOT FOUND THEN\n RAISE NOTICE 'Did not update stats for % due to lock. (This is generally OK)', _search;\n END IF;\n\n RETURN search;\n\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE OR REPLACE FUNCTION search_rows(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN partitions text[] DEFAULT NULL,\n IN _limit int DEFAULT 10\n) RETURNS SETOF items AS $$\nDECLARE\n base_query text;\n query text;\n sdate timestamptz;\n edate timestamptz;\n n int;\n records_left int := _limit;\n timer timestamptz := clock_timestamp();\n full_timer timestamptz := clock_timestamp();\nBEGIN\nIF _where IS NULL OR trim(_where) = '' THEN\n _where = ' TRUE ';\nEND IF;\nRAISE NOTICE 'Getting chunks for % %', _where, _orderby;\n\nbase_query := $q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n LIMIT %L\n$q$;\n\nIF _orderby ILIKE 'datetime d%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 DESC LOOP\n RAISE NOTICE 'Running Query for % to %. %', sdate, edate, age_ms(full_timer);\n query := format(\n base_query,\n sdate,\n edate,\n _where,\n _orderby,\n records_left\n );\n RAISE LOG 'QUERY: %', query;\n timer := clock_timestamp();\n RETURN QUERY EXECUTE query;\n\n GET DIAGNOSTICS n = ROW_COUNT;\n records_left := records_left - n;\n RAISE NOTICE 'Returned %/% Rows From % to %. % to go. Time: %ms', n, _limit, sdate, edate, records_left, age_ms(timer);\n timer := clock_timestamp();\n IF records_left <= 0 THEN\n RAISE NOTICE 'SEARCH_ROWS TOOK %ms', age_ms(full_timer);\n RETURN;\n END IF;\n END LOOP;\nELSIF _orderby ILIKE 'datetime a%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 ASC LOOP\n RAISE NOTICE 'Running Query for % to %. %', sdate, edate, age_ms(full_timer);\n query := format(\n base_query,\n sdate,\n edate,\n _where,\n _orderby,\n records_left\n );\n RAISE LOG 'QUERY: %', query;\n timer := clock_timestamp();\n RETURN QUERY EXECUTE query;\n\n GET DIAGNOSTICS n = ROW_COUNT;\n records_left := records_left - n;\n RAISE NOTICE 'Returned %/% Rows From % to %. % to go. Time: %ms', n, _limit, sdate, edate, records_left, age_ms(timer);\n timer := clock_timestamp();\n IF records_left <= 0 THEN\n RAISE NOTICE 'SEARCH_ROWS TOOK %ms', age_ms(full_timer);\n RETURN;\n END IF;\n END LOOP;\nELSE\n query := format($q$\n SELECT * FROM items\n WHERE %s\n ORDER BY %s\n LIMIT %L\n $q$, _where, _orderby, _limit\n );\n RAISE LOG 'QUERY: %', query;\n timer := clock_timestamp();\n RETURN QUERY EXECUTE query;\n RAISE NOTICE 'FULL QUERY TOOK %ms', age_ms(timer);\nEND IF;\nRAISE NOTICE 'SEARCH_ROWS TOOK %ms', age_ms(full_timer);\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE UNLOGGED TABLE format_item_cache(\n id text,\n collection text,\n fields text,\n hydrated bool,\n output jsonb,\n lastused timestamptz DEFAULT now(),\n usecount int DEFAULT 1,\n timetoformat float,\n PRIMARY KEY (collection, id, fields, hydrated)\n);\nCREATE INDEX ON format_item_cache (lastused);\n\nCREATE OR REPLACE FUNCTION format_item(_item items, _fields jsonb DEFAULT '{}', _hydrated bool DEFAULT TRUE) RETURNS jsonb AS $$\nDECLARE\n cache bool := get_setting_bool('format_cache');\n _output jsonb := null;\n t timestamptz := clock_timestamp();\nBEGIN\n IF cache THEN\n SELECT output INTO _output FROM format_item_cache\n WHERE id=_item.id AND collection=_item.collection AND fields=_fields::text AND hydrated=_hydrated;\n END IF;\n IF _output IS NULL THEN\n IF _hydrated THEN\n _output := content_hydrate(_item, _fields);\n ELSE\n _output := content_nonhydrated(_item, _fields);\n END IF;\n END IF;\n IF cache THEN\n INSERT INTO format_item_cache (id, collection, fields, hydrated, output, timetoformat)\n VALUES (_item.id, _item.collection, _fields::text, _hydrated, _output, age_ms(t))\n ON CONFLICT(collection, id, fields, hydrated) DO\n UPDATE\n SET lastused=now(), usecount = format_item_cache.usecount + 1\n ;\n END IF;\n RETURN _output;\n\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION search(_search jsonb = '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n searches searches%ROWTYPE;\n _where text;\n orderby text;\n search_where search_wheres%ROWTYPE;\n total_count bigint;\n token record;\n token_prev boolean;\n token_item items%ROWTYPE;\n token_where text;\n full_where text;\n init_ts timestamptz := clock_timestamp();\n timer timestamptz := clock_timestamp();\n hydrate bool := NOT (_search->'conf'->>'nohydrate' IS NOT NULL AND (_search->'conf'->>'nohydrate')::boolean = true);\n prev text;\n next text;\n context jsonb;\n collection jsonb;\n out_records jsonb;\n out_len int;\n _limit int := coalesce((_search->>'limit')::int, 10);\n init_limit int := _limit;\n _querylimit int;\n _fields jsonb := coalesce(_search->'fields', '{}'::jsonb);\n has_prev boolean := FALSE;\n has_next boolean := FALSE;\n items_cnt int := coalesce(jsonb_array_length(_search->'ids'),0);\nBEGIN\n RAISE NOTICE 'Items Count: %', items_cnt;\n IF items_cnt > 0 THEN\n IF items_cnt <= _limit THEN\n _limit := items_cnt - 1;\n ELSE\n _limit := items_cnt;\n END IF;\n RAISE NOTICE 'Items is set. Changing limit to %', items_cnt;\n END IF;\n searches := search_query(_search);\n _where := searches._where;\n orderby := searches.orderby;\n search_where := where_stats(_where);\n total_count := coalesce(search_where.total_count, search_where.estimated_count);\n RAISE NOTICE 'SEARCH:TOKEN: %', _search->>'token';\n token := get_token_record(_search->>'token');\n RAISE NOTICE '***TOKEN: %', token;\n _querylimit := _limit + 1;\n IF token IS NOT NULL THEN\n token_prev := token.prev;\n token_item := token.item;\n token_where := get_token_filter(_search->'sortby', token_item, token_prev, FALSE);\n RAISE LOG 'TOKEN_WHERE: % (%ms from search start)', token_where, age_ms(timer);\n IF token_prev THEN -- if we are using a prev token, we know has_next is true\n RAISE LOG 'There is a previous token, so automatically setting has_next to true';\n has_next := TRUE;\n orderby := sort_sqlorderby(_search, TRUE);\n ELSE\n RAISE LOG 'There is a next token, so automatically setting has_prev to true';\n has_prev := TRUE;\n\n END IF;\n ELSE -- if there was no token, we know there is no prev\n RAISE LOG 'There is no token, so we know there is no prev. setting has_prev to false';\n has_prev := FALSE;\n END IF;\n\n full_where := concat_ws(' AND ', _where, token_where);\n RAISE NOTICE 'FULL WHERE CLAUSE: %', full_where;\n RAISE NOTICE 'Time to get counts and build query %', age_ms(timer);\n timer := clock_timestamp();\n\n IF hydrate THEN\n RAISE NOTICE 'Getting hydrated data.';\n ELSE\n RAISE NOTICE 'Getting non-hydrated data.';\n END IF;\n RAISE NOTICE 'CACHE SET TO %', get_setting_bool('format_cache');\n RAISE NOTICE 'Time to set hydration/formatting %', age_ms(timer);\n timer := clock_timestamp();\n SELECT jsonb_agg(format_item(i, _fields, hydrate)) INTO out_records\n FROM search_rows(\n full_where,\n orderby,\n search_where.partitions,\n _querylimit\n ) as i;\n\n RAISE NOTICE 'Time to fetch rows %', age_ms(timer);\n timer := clock_timestamp();\n\n\n IF token_prev THEN\n out_records := flip_jsonb_array(out_records);\n END IF;\n\n RAISE NOTICE 'Query returned % records.', jsonb_array_length(out_records);\n RAISE LOG 'TOKEN: % %', token_item.id, token_item.collection;\n RAISE LOG 'RECORD_1: % %', out_records->0->>'id', out_records->0->>'collection';\n RAISE LOG 'RECORD-1: % %', out_records->-1->>'id', out_records->-1->>'collection';\n\n -- REMOVE records that were from our token\n IF out_records->0->>'id' = token_item.id AND out_records->0->>'collection' = token_item.collection THEN\n out_records := out_records - 0;\n ELSIF out_records->-1->>'id' = token_item.id AND out_records->-1->>'collection' = token_item.collection THEN\n out_records := out_records - -1;\n END IF;\n\n out_len := jsonb_array_length(out_records);\n\n IF out_len = init_limit + 1 THEN\n IF token_prev THEN\n has_prev := TRUE;\n out_records := out_records - 0;\n ELSE\n has_next := TRUE;\n out_records := out_records - -1;\n END IF;\n END IF;\n\n IF has_next THEN\n next := concat(out_records->-1->>'collection', ':', out_records->-1->>'id');\n RAISE NOTICE 'HAS NEXT | %', next;\n END IF;\n\n IF has_prev THEN\n prev := concat(out_records->0->>'collection', ':', out_records->0->>'id');\n RAISE NOTICE 'HAS PREV | %', prev;\n END IF;\n\n RAISE NOTICE 'Time to get prev/next %', age_ms(timer);\n timer := clock_timestamp();\n\n IF context(_search->'conf') != 'off' THEN\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', init_limit,\n 'matched', total_count,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\n ELSE\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', init_limit,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\n END IF;\n\n collection := jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb),\n 'next', next,\n 'prev', prev,\n 'context', context\n );\n\n IF get_setting_bool('timing', _search->'conf') THEN\n collection = collection || jsonb_build_object('timing', age_ms(init_ts));\n END IF;\n\n RAISE NOTICE 'Time to build final json %', age_ms(timer);\n timer := clock_timestamp();\n\n RAISE NOTICE 'Total Time: %', age_ms(current_timestamp);\n RAISE NOTICE 'RETURNING % records. NEXT: %. PREV: %', collection->'context'->>'returned', collection->>'next', collection->>'prev';\n RETURN collection;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION search_cursor(_search jsonb = '{}'::jsonb) RETURNS refcursor AS $$\nDECLARE\n curs refcursor;\n searches searches%ROWTYPE;\n _where text;\n _orderby text;\n q text;\n\nBEGIN\n searches := search_query(_search);\n _where := searches._where;\n _orderby := searches.orderby;\n\n OPEN curs FOR\n WITH p AS (\n SELECT * FROM partition_queries(_where, _orderby) p\n )\n SELECT\n CASE WHEN EXISTS (SELECT 1 FROM p) THEN\n (SELECT format($q$\n SELECT * FROM (\n %s\n ) total\n $q$,\n string_agg(\n format($q$ SELECT * FROM ( %s ) AS sub $q$, p),\n '\n UNION ALL\n '\n )\n ))\n ELSE NULL\n END FROM p;\n RETURN curs;\nEND;\n$$ LANGUAGE PLPGSQL;\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION tileenvelope(zoom int, x int, y int) RETURNS geometry AS $$\nWITH t AS (\n SELECT\n 20037508.3427892 as merc_max,\n -20037508.3427892 as merc_min,\n (2 * 20037508.3427892) / (2 ^ zoom) as tile_size\n)\nSELECT st_makeenvelope(\n merc_min + (tile_size * x),\n merc_max - (tile_size * (y + 1)),\n merc_min + (tile_size * (x + 1)),\n merc_max - (tile_size * y),\n 3857\n) FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;DROP FUNCTION IF EXISTS mercgrid;\n\n\nCREATE OR REPLACE FUNCTION ftime() RETURNS interval as $$\nSELECT age(clock_timestamp(), transaction_timestamp());\n$$ LANGUAGE SQL;\nSET SEARCH_PATH to pgstac, public;\n\nDROP FUNCTION IF EXISTS geometrysearch;\nCREATE OR REPLACE FUNCTION geometrysearch(\n IN geom geometry,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE, -- Return as soon as the passed in geometry is full covered\n IN skipcovered boolean DEFAULT TRUE -- Skip any items that would show up completely under the previous items\n) RETURNS jsonb AS $$\nDECLARE\n search searches%ROWTYPE;\n curs refcursor;\n _where text;\n query text;\n iter_record items%ROWTYPE;\n out_records jsonb := '{}'::jsonb[];\n exit_flag boolean := FALSE;\n counter int := 1;\n scancounter int := 1;\n remaining_limit int := _scanlimit;\n tilearea float;\n unionedgeom geometry;\n clippedgeom geometry;\n unionedgeom_area float := 0;\n prev_area float := 0;\n excludes text[];\n includes text[];\n\nBEGIN\n DROP TABLE IF EXISTS pgstac_results;\n CREATE TEMP TABLE pgstac_results (content jsonb) ON COMMIT DROP;\n\n -- If skipcovered is true then you will always want to exit when the passed in geometry is full\n IF skipcovered THEN\n exitwhenfull := TRUE;\n END IF;\n\n SELECT * INTO search FROM searches WHERE hash=queryhash;\n\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Search with Query Hash % Not Found', queryhash;\n END IF;\n\n tilearea := st_area(geom);\n _where := format('%s AND st_intersects(geometry, %L::geometry)', search._where, geom);\n\n\n FOR query IN SELECT * FROM partition_queries(_where, search.orderby) LOOP\n query := format('%s LIMIT %L', query, remaining_limit);\n RAISE NOTICE '%', query;\n OPEN curs FOR EXECUTE query;\n LOOP\n FETCH curs INTO iter_record;\n EXIT WHEN NOT FOUND;\n IF exitwhenfull OR skipcovered THEN -- If we are not using exitwhenfull or skipcovered, we do not need to do expensive geometry operations\n clippedgeom := st_intersection(geom, iter_record.geometry);\n\n IF unionedgeom IS NULL THEN\n unionedgeom := clippedgeom;\n ELSE\n unionedgeom := st_union(unionedgeom, clippedgeom);\n END IF;\n\n unionedgeom_area := st_area(unionedgeom);\n\n IF skipcovered AND prev_area = unionedgeom_area THEN\n scancounter := scancounter + 1;\n CONTINUE;\n END IF;\n\n prev_area := unionedgeom_area;\n\n RAISE NOTICE '% % % %', unionedgeom_area/tilearea, counter, scancounter, ftime();\n END IF;\n RAISE NOTICE '% %', iter_record, content_hydrate(iter_record, fields);\n INSERT INTO pgstac_results (content) VALUES (content_hydrate(iter_record, fields));\n\n IF counter >= _limit\n OR scancounter > _scanlimit\n OR ftime() > _timelimit\n OR (exitwhenfull AND unionedgeom_area >= tilearea)\n THEN\n exit_flag := TRUE;\n EXIT;\n END IF;\n counter := counter + 1;\n scancounter := scancounter + 1;\n\n END LOOP;\n CLOSE curs;\n EXIT WHEN exit_flag;\n remaining_limit := _scanlimit - scancounter;\n END LOOP;\n\n SELECT jsonb_agg(content) INTO out_records FROM pgstac_results WHERE content IS NOT NULL;\n\n RETURN jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb)\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDROP FUNCTION IF EXISTS geojsonsearch;\nCREATE OR REPLACE FUNCTION geojsonsearch(\n IN geojson jsonb,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_geomfromgeojson(geojson),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS xyzsearch;\nCREATE OR REPLACE FUNCTION xyzsearch(\n IN _x int,\n IN _y int,\n IN _z int,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_transform(tileenvelope(_z, _x, _y), 4326),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE PROCEDURE analyze_items() AS $$\nDECLARE\n q text;\n timeout_ts timestamptz;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n SELECT format('ANALYZE (VERBOSE, SKIP_LOCKED) %I;', relname) INTO q\n FROM pg_stat_user_tables\n WHERE relname like '_item%' AND (n_mod_since_analyze>0 OR last_analyze IS NULL) LIMIT 1;\n IF NOT FOUND THEN\n EXIT;\n END IF;\n RAISE NOTICE '%', q;\n EXECUTE q;\n COMMIT;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE PROCEDURE validate_constraints() AS $$\nDECLARE\n q text;\nBEGIN\n FOR q IN\n SELECT\n FORMAT(\n 'ALTER TABLE %I.%I VALIDATE CONSTRAINT %I;',\n nsp.nspname,\n cls.relname,\n con.conname\n )\n\n FROM pg_constraint AS con\n JOIN pg_class AS cls\n ON con.conrelid = cls.oid\n JOIN pg_namespace AS nsp\n ON cls.relnamespace = nsp.oid\n WHERE convalidated = FALSE AND contype in ('c','f')\n AND nsp.nspname = 'pgstac'\n LOOP\n RAISE NOTICE '%', q;\n PERFORM run_or_queue(q);\n COMMIT;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION collection_extent(_collection text, runupdate boolean default false) RETURNS jsonb AS $$\nDECLARE\n geom_extent geometry;\n mind timestamptz;\n maxd timestamptz;\n extent jsonb;\nBEGIN\n IF runupdate THEN\n PERFORM update_partition_stats_q(partition)\n FROM partitions_view WHERE collection=_collection;\n END IF;\n SELECT\n min(lower(dtrange)),\n max(upper(edtrange)),\n st_extent(spatial)\n INTO\n mind,\n maxd,\n geom_extent\n FROM partitions_view\n WHERE collection=_collection;\n\n IF geom_extent IS NOT NULL AND mind IS NOT NULL AND maxd IS NOT NULL THEN\n extent := jsonb_build_object(\n 'extent', jsonb_build_object(\n 'spatial', jsonb_build_object(\n 'bbox', to_jsonb(array[array[st_xmin(geom_extent), st_ymin(geom_extent), st_xmax(geom_extent), st_ymax(geom_extent)]])\n ),\n 'temporal', jsonb_build_object(\n 'interval', to_jsonb(array[array[mind, maxd]])\n )\n )\n );\n RETURN extent;\n END IF;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('id', '{\"title\": \"Item ID\",\"description\": \"Item identifier\",\"$ref\": \"https://schemas.stacspec.org/v1.0.0/item-spec/json-schema/item.json#/definitions/core/allOf/2/properties/id\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('geometry', '{\"title\": \"Item Geometry\",\"description\": \"Item Geometry\",\"$ref\": \"https://geojson.org/schema/Feature.json\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('datetime','{\"description\": \"Datetime\",\"type\": \"string\",\"title\": \"Acquired\",\"format\": \"date-time\",\"pattern\": \"(\\\\+00:00|Z)$\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('eo:cloud_cover','{\"$ref\": \"https://stac-extensions.github.io/eo/v1.0.0/schema.json#/definitions/fieldsproperties/eo:cloud_cover\"}','to_int','BTREE');\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDELETE FROM queryables a USING queryables b\n WHERE a.name = b.name AND a.collection_ids IS NOT DISTINCT FROM b.collection_ids AND a.id > b.id;\n\n\nINSERT INTO pgstac_settings (name, value) VALUES\n ('context', 'off'),\n ('context_estimated_count', '100000'),\n ('context_estimated_cost', '100000'),\n ('context_stats_ttl', '1 day'),\n ('default_filter_lang', 'cql2-json'),\n ('additional_properties', 'true'),\n ('use_queue', 'false'),\n ('queue_timeout', '10 minutes'),\n ('update_collection_extent', 'false'),\n ('format_cache', 'false')\nON CONFLICT DO NOTHING\n;\n\nALTER FUNCTION to_text COST 5000;\nALTER FUNCTION to_float COST 5000;\nALTER FUNCTION to_int COST 5000;\nALTER FUNCTION to_tstz COST 5000;\nALTER FUNCTION to_text_array COST 5000;\n\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nGRANT ALL ON SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac to pgstac_admin;\n\n-- pgstac_read role limited to using function apis\nGRANT EXECUTE ON FUNCTION search TO pgstac_read;\nGRANT EXECUTE ON FUNCTION search_query TO pgstac_read;\nGRANT EXECUTE ON FUNCTION item_by_id TO pgstac_read;\nGRANT EXECUTE ON FUNCTION get_item TO pgstac_read;\nGRANT SELECT ON ALL TABLES IN SCHEMA pgstac TO pgstac_read;\n\n\nGRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_ingest;\nGRANT USAGE ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_ingest;\n\nSELECT update_partition_stats_q(partition) FROM partitions_view;\nSELECT set_version('0.7.7');\n" }, { "path": "src/pgstac/migrations/pgstac.0.7.8-0.7.9.sql", "content": "RESET ROLE;\nDO $$\nDECLARE\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='postgis') THEN\n CREATE EXTENSION IF NOT EXISTS postgis;\n END IF;\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='btree_gist') THEN\n CREATE EXTENSION IF NOT EXISTS btree_gist;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_admin;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_read;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_ingest;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n\nGRANT pgstac_admin TO current_user;\n\n-- Function to make sure pgstac_admin is the owner of items\nCREATE OR REPLACE FUNCTION pgstac_admin_owns() RETURNS VOID AS $$\nDECLARE\n f RECORD;\nBEGIN\n FOR f IN (\n SELECT\n concat(\n oid::regproc::text,\n '(',\n coalesce(pg_get_function_identity_arguments(oid),''),\n ')'\n ) AS name,\n CASE prokind WHEN 'f' THEN 'FUNCTION' WHEN 'p' THEN 'PROCEDURE' WHEN 'a' THEN 'AGGREGATE' END as typ\n FROM pg_proc\n WHERE\n pronamespace=to_regnamespace('pgstac')\n AND proowner != to_regrole('pgstac_admin')\n AND proname NOT LIKE 'pg_stat%'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n FOR f IN (\n SELECT\n oid::regclass::text as name,\n CASE relkind\n WHEN 'i' THEN 'INDEX'\n WHEN 'I' THEN 'INDEX'\n WHEN 'p' THEN 'TABLE'\n WHEN 'r' THEN 'TABLE'\n WHEN 'v' THEN 'VIEW'\n WHEN 'S' THEN 'SEQUENCE'\n ELSE NULL\n END as typ\n FROM pg_class\n WHERE relnamespace=to_regnamespace('pgstac') and relowner != to_regrole('pgstac_admin') AND relkind IN ('r','p','v','S') AND relname NOT LIKE 'pg_stat'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\nSELECT pgstac_admin_owns();\n\nCREATE SCHEMA IF NOT EXISTS pgstac AUTHORIZATION pgstac_admin;\n\nGRANT ALL ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_admin;\n\nALTER ROLE pgstac_admin SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_read SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_ingest SET SEARCH_PATH TO pgstac, public;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\n\nGRANT pgstac_read TO pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\n\nSET ROLE pgstac_admin;\n\nSET SEARCH_PATH TO pgstac, public;\n\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS analyze_items;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS validate_constraints;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\nSET client_min_messages TO WARNING;\nSET SEARCH_PATH to pgstac, public;\n-- BEGIN migra calculated SQL\n-- END migra calculated SQL\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('id', '{\"title\": \"Item ID\",\"description\": \"Item identifier\",\"$ref\": \"https://schemas.stacspec.org/v1.0.0/item-spec/json-schema/item.json#/definitions/core/allOf/2/properties/id\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('geometry', '{\"title\": \"Item Geometry\",\"description\": \"Item Geometry\",\"$ref\": \"https://geojson.org/schema/Feature.json\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('datetime','{\"description\": \"Datetime\",\"type\": \"string\",\"title\": \"Acquired\",\"format\": \"date-time\",\"pattern\": \"(\\\\+00:00|Z)$\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('eo:cloud_cover','{\"$ref\": \"https://stac-extensions.github.io/eo/v1.0.0/schema.json#/definitions/fieldsproperties/eo:cloud_cover\"}','to_int','BTREE');\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDELETE FROM queryables a USING queryables b\n WHERE a.name = b.name AND a.collection_ids IS NOT DISTINCT FROM b.collection_ids AND a.id > b.id;\n\n\nINSERT INTO pgstac_settings (name, value) VALUES\n ('context', 'off'),\n ('context_estimated_count', '100000'),\n ('context_estimated_cost', '100000'),\n ('context_stats_ttl', '1 day'),\n ('default_filter_lang', 'cql2-json'),\n ('additional_properties', 'true'),\n ('use_queue', 'false'),\n ('queue_timeout', '10 minutes'),\n ('update_collection_extent', 'false'),\n ('format_cache', 'false')\nON CONFLICT DO NOTHING\n;\n\nALTER FUNCTION to_text COST 5000;\nALTER FUNCTION to_float COST 5000;\nALTER FUNCTION to_int COST 5000;\nALTER FUNCTION to_tstz COST 5000;\nALTER FUNCTION to_text_array COST 5000;\n\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nGRANT ALL ON SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac to pgstac_admin;\n\n-- pgstac_read role limited to using function apis\nGRANT EXECUTE ON FUNCTION search TO pgstac_read;\nGRANT EXECUTE ON FUNCTION search_query TO pgstac_read;\nGRANT EXECUTE ON FUNCTION item_by_id TO pgstac_read;\nGRANT EXECUTE ON FUNCTION get_item TO pgstac_read;\nGRANT SELECT ON ALL TABLES IN SCHEMA pgstac TO pgstac_read;\n\n\nGRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_ingest;\nGRANT USAGE ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_ingest;\n\nSELECT update_partition_stats_q(partition) FROM partitions_view;\nSELECT set_version('0.7.9');\n" }, { "path": "src/pgstac/migrations/pgstac.0.7.8.sql", "content": "RESET ROLE;\nDO $$\nDECLARE\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='postgis') THEN\n CREATE EXTENSION IF NOT EXISTS postgis;\n END IF;\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='btree_gist') THEN\n CREATE EXTENSION IF NOT EXISTS btree_gist;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_admin;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_read;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_ingest;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n\nGRANT pgstac_admin TO current_user;\n\n-- Function to make sure pgstac_admin is the owner of items\nCREATE OR REPLACE FUNCTION pgstac_admin_owns() RETURNS VOID AS $$\nDECLARE\n f RECORD;\nBEGIN\n FOR f IN (\n SELECT\n concat(\n oid::regproc::text,\n '(',\n coalesce(pg_get_function_identity_arguments(oid),''),\n ')'\n ) AS name,\n CASE prokind WHEN 'f' THEN 'FUNCTION' WHEN 'p' THEN 'PROCEDURE' WHEN 'a' THEN 'AGGREGATE' END as typ\n FROM pg_proc\n WHERE\n pronamespace=to_regnamespace('pgstac')\n AND proowner != to_regrole('pgstac_admin')\n AND proname NOT LIKE 'pg_stat%'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n FOR f IN (\n SELECT\n oid::regclass::text as name,\n CASE relkind\n WHEN 'i' THEN 'INDEX'\n WHEN 'I' THEN 'INDEX'\n WHEN 'p' THEN 'TABLE'\n WHEN 'r' THEN 'TABLE'\n WHEN 'v' THEN 'VIEW'\n WHEN 'S' THEN 'SEQUENCE'\n ELSE NULL\n END as typ\n FROM pg_class\n WHERE relnamespace=to_regnamespace('pgstac') and relowner != to_regrole('pgstac_admin') AND relkind IN ('r','p','v','S') AND relname NOT LIKE 'pg_stat'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\nSELECT pgstac_admin_owns();\n\nCREATE SCHEMA IF NOT EXISTS pgstac AUTHORIZATION pgstac_admin;\n\nGRANT ALL ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_admin;\n\nALTER ROLE pgstac_admin SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_read SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_ingest SET SEARCH_PATH TO pgstac, public;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\n\nGRANT pgstac_read TO pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\n\nSET ROLE pgstac_admin;\n\nSET SEARCH_PATH TO pgstac, public;\n\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS analyze_items;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS validate_constraints;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\nCREATE OR REPLACE FUNCTION to_int(jsonb) RETURNS int AS $$\n SELECT floor(($1->>0)::float)::int;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION to_float(jsonb) RETURNS float AS $$\n SELECT ($1->>0)::float;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION to_tstz(jsonb) RETURNS timestamptz AS $$\n SELECT ($1->>0)::timestamptz;\n$$ LANGUAGE SQL IMMUTABLE STRICT SET TIME ZONE 'UTC' COST 5000 PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION to_text(jsonb) RETURNS text AS $$\n SELECT CASE WHEN jsonb_typeof($1) IN ('array','object') THEN $1::text ELSE $1->>0 END;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION to_text_array(jsonb) RETURNS text[] AS $$\n SELECT\n CASE jsonb_typeof($1)\n WHEN 'array' THEN ARRAY(SELECT jsonb_array_elements_text($1))\n ELSE ARRAY[$1->>0]\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION bbox_geom(_bbox jsonb) RETURNS geometry AS $$\nSELECT CASE jsonb_array_length(_bbox)\n WHEN 4 THEN\n ST_SetSRID(ST_MakeEnvelope(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float,\n (_bbox->>3)::float\n ),4326)\n WHEN 6 THEN\n ST_SetSRID(ST_3DMakeBox(\n ST_MakePoint(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float\n ),\n ST_MakePoint(\n (_bbox->>3)::float,\n (_bbox->>4)::float,\n (_bbox->>5)::float\n )\n ),4326)\n ELSE null END;\n;\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION geom_bbox(_geom geometry) RETURNS jsonb AS $$\n SELECT jsonb_build_array(\n st_xmin(_geom),\n st_ymin(_geom),\n st_xmax(_geom),\n st_ymax(_geom)\n );\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION flip_jsonb_array(j jsonb) RETURNS jsonb AS $$\n SELECT jsonb_agg(value) FROM (SELECT value FROM jsonb_array_elements(j) WITH ORDINALITY ORDER BY ordinality DESC) as t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION explode_dotpaths(j jsonb) RETURNS SETOF text[] AS $$\n SELECT string_to_array(p, '.') as e FROM jsonb_array_elements_text(j) p;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION explode_dotpaths_recurse(IN j jsonb) RETURNS SETOF text[] AS $$\n WITH RECURSIVE t AS (\n SELECT e FROM explode_dotpaths(j) e\n UNION ALL\n SELECT e[1:cardinality(e)-1]\n FROM t\n WHERE cardinality(e)>1\n ) SELECT e FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION jsonb_set_nested(j jsonb, path text[], val jsonb) RETURNS jsonb AS $$\nDECLARE\nBEGIN\n IF cardinality(path) > 1 THEN\n FOR i IN 1..(cardinality(path)-1) LOOP\n IF j #> path[:i] IS NULL THEN\n j := jsonb_set_lax(j, path[:i], '{}', TRUE);\n END IF;\n END LOOP;\n END IF;\n RETURN jsonb_set_lax(j, path, val, true);\n\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\n\n\nCREATE OR REPLACE FUNCTION jsonb_include(j jsonb, f jsonb) RETURNS jsonb AS $$\nDECLARE\n includes jsonb := f-> 'include';\n outj jsonb := '{}'::jsonb;\n path text[];\nBEGIN\n IF\n includes IS NULL\n OR jsonb_array_length(includes) = 0\n THEN\n RETURN j;\n ELSE\n includes := includes || '[\"id\",\"collection\"]'::jsonb;\n FOR path IN SELECT explode_dotpaths(includes) LOOP\n outj := jsonb_set_nested(outj, path, j #> path);\n END LOOP;\n END IF;\n RETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_exclude(j jsonb, f jsonb) RETURNS jsonb AS $$\nDECLARE\n excludes jsonb := f-> 'exclude';\n outj jsonb := j;\n path text[];\nBEGIN\n IF\n excludes IS NULL\n OR jsonb_array_length(excludes) = 0\n THEN\n RETURN j;\n ELSE\n FOR path IN SELECT explode_dotpaths(excludes) LOOP\n outj := outj #- path;\n END LOOP;\n END IF;\n RETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_fields(j jsonb, f jsonb DEFAULT '{\"fields\":[]}') RETURNS jsonb AS $$\n SELECT jsonb_exclude(jsonb_include(j, f), f);\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION merge_jsonb(_a jsonb, _b jsonb) RETURNS jsonb AS $$\n SELECT\n CASE\n WHEN _a = '\"𒍟※\"'::jsonb THEN NULL\n WHEN _a IS NULL OR jsonb_typeof(_a) = 'null' THEN _b\n WHEN jsonb_typeof(_a) = 'object' AND jsonb_typeof(_b) = 'object' THEN\n (\n SELECT\n jsonb_strip_nulls(\n jsonb_object_agg(\n key,\n merge_jsonb(a.value, b.value)\n )\n )\n FROM\n jsonb_each(coalesce(_a,'{}'::jsonb)) as a\n FULL JOIN\n jsonb_each(coalesce(_b,'{}'::jsonb)) as b\n USING (key)\n )\n WHEN\n jsonb_typeof(_a) = 'array'\n AND jsonb_typeof(_b) = 'array'\n AND jsonb_array_length(_a) = jsonb_array_length(_b)\n THEN\n (\n SELECT jsonb_agg(m) FROM\n ( SELECT\n merge_jsonb(\n jsonb_array_elements(_a),\n jsonb_array_elements(_b)\n ) as m\n ) as l\n )\n ELSE _a\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION strip_jsonb(_a jsonb, _b jsonb) RETURNS jsonb AS $$\n SELECT\n CASE\n\n WHEN (_a IS NULL OR jsonb_typeof(_a) = 'null') AND _b IS NOT NULL AND jsonb_typeof(_b) != 'null' THEN '\"𒍟※\"'::jsonb\n WHEN _b IS NULL OR jsonb_typeof(_a) = 'null' THEN _a\n WHEN _a = _b AND jsonb_typeof(_a) = 'object' THEN '{}'::jsonb\n WHEN _a = _b THEN NULL\n WHEN jsonb_typeof(_a) = 'object' AND jsonb_typeof(_b) = 'object' THEN\n (\n SELECT\n jsonb_strip_nulls(\n jsonb_object_agg(\n key,\n strip_jsonb(a.value, b.value)\n )\n )\n FROM\n jsonb_each(_a) as a\n FULL JOIN\n jsonb_each(_b) as b\n USING (key)\n )\n WHEN\n jsonb_typeof(_a) = 'array'\n AND jsonb_typeof(_b) = 'array'\n AND jsonb_array_length(_a) = jsonb_array_length(_b)\n THEN\n (\n SELECT jsonb_agg(m) FROM\n ( SELECT\n strip_jsonb(\n jsonb_array_elements(_a),\n jsonb_array_elements(_b)\n ) as m\n ) as l\n )\n ELSE _a\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION nullif_jsonbnullempty(j jsonb) RETURNS jsonb AS $$\n SELECT nullif(nullif(nullif(j,'null'::jsonb),'{}'::jsonb),'[]'::jsonb);\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION jsonb_array_unique(j jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(jsonb_agg(DISTINCT a)) v FROM jsonb_array_elements(j) a;\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_concat_ignorenull(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT coalesce(a,'[]'::jsonb) || coalesce(b,'[]'::jsonb);\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_least(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(least(nullif_jsonbnullempty(a), nullif_jsonbnullempty(b)));\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_greatest(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(greatest(a, b));\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION first_notnull_sfunc(anyelement, anyelement) RETURNS anyelement AS $$\n SELECT COALESCE($1,$2);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE AGGREGATE first_notnull(anyelement)(\n SFUNC = first_notnull_sfunc,\n STYPE = anyelement\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_array_unique_merge(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_concat_ignorenull,\n FINALFUNC = jsonb_array_unique\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_min(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_least\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_max(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_greatest\n);\n\nCREATE TABLE IF NOT EXISTS migrations (\n version text PRIMARY KEY,\n datetime timestamptz DEFAULT clock_timestamp() NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION get_version() RETURNS text AS $$\n SELECT version FROM pgstac.migrations ORDER BY datetime DESC, version DESC LIMIT 1;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION set_version(text) RETURNS text AS $$\n INSERT INTO pgstac.migrations (version) VALUES ($1)\n ON CONFLICT DO NOTHING\n RETURNING version;\n$$ LANGUAGE SQL;\n\n\nCREATE TABLE IF NOT EXISTS pgstac_settings (\n name text PRIMARY KEY,\n value text NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION table_empty(text) RETURNS boolean AS $$\nDECLARE\n retval boolean;\nBEGIN\n EXECUTE format($q$\n SELECT NOT EXISTS (SELECT 1 FROM %I LIMIT 1)\n $q$,\n $1\n ) INTO retval;\n RETURN retval;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION get_setting(IN _setting text, IN conf jsonb DEFAULT NULL) RETURNS text AS $$\nSELECT COALESCE(\n nullif(conf->>_setting, ''),\n nullif(current_setting(concat('pgstac.',_setting), TRUE),''),\n nullif((SELECT value FROM pgstac.pgstac_settings WHERE name=_setting),'')\n);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_setting_bool(IN _setting text, IN conf jsonb DEFAULT NULL) RETURNS boolean AS $$\nSELECT COALESCE(\n conf->>_setting,\n current_setting(concat('pgstac.',_setting), TRUE),\n (SELECT value FROM pgstac.pgstac_settings WHERE name=_setting),\n 'FALSE'\n)::boolean;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context(conf jsonb DEFAULT NULL) RETURNS text AS $$\n SELECT pgstac.get_setting('context', conf);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context_estimated_count(conf jsonb DEFAULT NULL) RETURNS int AS $$\n SELECT pgstac.get_setting('context_estimated_count', conf)::int;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_estimated_cost();\nCREATE OR REPLACE FUNCTION context_estimated_cost(conf jsonb DEFAULT NULL) RETURNS float AS $$\n SELECT pgstac.get_setting('context_estimated_cost', conf)::float;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_stats_ttl();\nCREATE OR REPLACE FUNCTION context_stats_ttl(conf jsonb DEFAULT NULL) RETURNS interval AS $$\n SELECT pgstac.get_setting('context_stats_ttl', conf)::interval;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION t2s(text) RETURNS text AS $$\n SELECT extract(epoch FROM $1::interval)::text || ' s';\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE STRICT;\n\nCREATE OR REPLACE FUNCTION age_ms(a timestamptz, b timestamptz DEFAULT clock_timestamp()) RETURNS float AS $$\n SELECT abs(extract(epoch from age(a,b)) * 1000);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION queue_timeout() RETURNS interval AS $$\n SELECT t2s(coalesce(\n get_setting('queue_timeout'),\n '1h'\n ))::interval;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION notice(VARIADIC text[]) RETURNS boolean AS $$\nDECLARE\ndebug boolean := current_setting('pgstac.debug', true);\nBEGIN\n IF debug THEN\n RAISE NOTICE 'NOTICE FROM FUNC: % >>>>> %', concat_ws(' | ', $1), clock_timestamp();\n RETURN TRUE;\n END IF;\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION empty_arr(ANYARRAY) RETURNS BOOLEAN AS $$\nSELECT CASE\n WHEN $1 IS NULL THEN TRUE\n WHEN cardinality($1)<1 THEN TRUE\nELSE FALSE\nEND;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_intersection(_a ANYARRAY, _b ANYARRAY) RETURNS ANYARRAY AS $$\n SELECT ARRAY ( SELECT unnest(_a) INTERSECT SELECT UNNEST(_b) );\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION array_map_ident(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_ident(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_map_literal(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_literal(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION array_reverse(anyarray) RETURNS anyarray AS $$\nSELECT ARRAY(\n SELECT $1[i]\n FROM generate_subscripts($1,1) AS s(i)\n ORDER BY i DESC\n);\n$$ LANGUAGE SQL STRICT IMMUTABLE;\n\nDROP TABLE IF EXISTS query_queue;\nCREATE TABLE query_queue (\n query text PRIMARY KEY,\n added timestamptz DEFAULT now()\n);\n\nDROP TABLE IF EXISTS query_queue_history;\nCREATE TABLE query_queue_history(\n query text,\n added timestamptz NOT NULL,\n finished timestamptz NOT NULL DEFAULT now(),\n error text\n);\n\nCREATE OR REPLACE PROCEDURE run_queued_queries() AS $$\nDECLARE\n qitem query_queue%ROWTYPE;\n timeout_ts timestamptz;\n error text;\n cnt int := 0;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n DELETE FROM query_queue WHERE query = (SELECT query FROM query_queue ORDER BY added DESC LIMIT 1 FOR UPDATE SKIP LOCKED) RETURNING * INTO qitem;\n IF NOT FOUND THEN\n EXIT;\n END IF;\n cnt := cnt + 1;\n BEGIN\n RAISE NOTICE 'RUNNING QUERY: %', qitem.query;\n EXECUTE qitem.query;\n EXCEPTION WHEN others THEN\n error := format('%s | %s', SQLERRM, SQLSTATE);\n END;\n INSERT INTO query_queue_history (query, added, finished, error)\n VALUES (qitem.query, qitem.added, clock_timestamp(), error);\n COMMIT;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION run_queued_queries_intransaction() RETURNS int AS $$\nDECLARE\n qitem query_queue%ROWTYPE;\n timeout_ts timestamptz;\n error text;\n cnt int := 0;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n DELETE FROM query_queue WHERE query = (SELECT query FROM query_queue ORDER BY added DESC LIMIT 1 FOR UPDATE SKIP LOCKED) RETURNING * INTO qitem;\n IF NOT FOUND THEN\n RETURN cnt;\n END IF;\n cnt := cnt + 1;\n BEGIN\n qitem.query := regexp_replace(qitem.query, 'CONCURRENTLY', '');\n RAISE NOTICE 'RUNNING QUERY: %', qitem.query;\n\n EXECUTE qitem.query;\n EXCEPTION WHEN others THEN\n error := format('%s | %s', SQLERRM, SQLSTATE);\n RAISE WARNING '%', error;\n END;\n INSERT INTO query_queue_history (query, added, finished, error)\n VALUES (qitem.query, qitem.added, clock_timestamp(), error);\n END LOOP;\n RETURN cnt;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION run_or_queue(query text) RETURNS VOID AS $$\nDECLARE\n use_queue text := COALESCE(get_setting('use_queue'), 'FALSE')::boolean;\nBEGIN\n IF get_setting_bool('debug') THEN\n RAISE NOTICE '%', query;\n END IF;\n IF use_queue THEN\n INSERT INTO query_queue (query) VALUES (query) ON CONFLICT DO NOTHING;\n ELSE\n EXECUTE query;\n END IF;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n\nDROP FUNCTION IF EXISTS check_pgstac_settings;\nCREATE OR REPLACE FUNCTION check_pgstac_settings(_sysmem text DEFAULT NULL) RETURNS VOID AS $$\nDECLARE\n settingval text;\n sysmem bigint := pg_size_bytes(_sysmem);\n effective_cache_size bigint := pg_size_bytes(current_setting('effective_cache_size', TRUE));\n shared_buffers bigint := pg_size_bytes(current_setting('shared_buffers', TRUE));\n work_mem bigint := pg_size_bytes(current_setting('work_mem', TRUE));\n max_connections int := current_setting('max_connections', TRUE);\n maintenance_work_mem bigint := pg_size_bytes(current_setting('maintenance_work_mem', TRUE));\n seq_page_cost float := current_setting('seq_page_cost', TRUE);\n random_page_cost float := current_setting('random_page_cost', TRUE);\n temp_buffers bigint := pg_size_bytes(current_setting('temp_buffers', TRUE));\n r record;\nBEGIN\n IF _sysmem IS NULL THEN\n RAISE NOTICE 'Call function with the size of your system memory `SELECT check_pgstac_settings(''4GB'')` to get pg system setting recommendations.';\n ELSE\n IF effective_cache_size < (sysmem * 0.5) THEN\n RAISE WARNING 'effective_cache_size of % is set low for a system with %. Recomended value between % and %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.5), pg_size_pretty(sysmem * 0.75);\n ELSIF effective_cache_size > (sysmem * 0.75) THEN\n RAISE WARNING 'effective_cache_size of % is set high for a system with %. Recomended value between % and %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.5), pg_size_pretty(sysmem * 0.75);\n ELSE\n RAISE NOTICE 'effective_cache_size of % is set appropriately for a system with %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem);\n END IF;\n\n IF shared_buffers < (sysmem * 0.2) THEN\n RAISE WARNING 'shared_buffers of % is set low for a system with %. Recomended value between % and %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.2), pg_size_pretty(sysmem * 0.3);\n ELSIF shared_buffers > (sysmem * 0.3) THEN\n RAISE WARNING 'shared_buffers of % is set high for a system with %. Recomended value between % and %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.2), pg_size_pretty(sysmem * 0.3);\n ELSE\n RAISE NOTICE 'shared_buffers of % is set appropriately for a system with %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem);\n END IF;\n shared_buffers = sysmem * 0.3;\n IF maintenance_work_mem < (sysmem * 0.2) THEN\n RAISE WARNING 'maintenance_work_mem of % is set low for shared_buffers of %. Recomended value between % and %', pg_size_pretty(maintenance_work_mem), pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers * 0.2), pg_size_pretty(shared_buffers * 0.3);\n ELSIF maintenance_work_mem > (shared_buffers * 0.3) THEN\n RAISE WARNING 'maintenance_work_mem of % is set high for shared_buffers of %. Recomended value between % and %', pg_size_pretty(maintenance_work_mem), pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers * 0.2), pg_size_pretty(shared_buffers * 0.3);\n ELSE\n RAISE NOTICE 'maintenance_work_mem of % is set appropriately for shared_buffers of %', pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers);\n END IF;\n\n IF work_mem * max_connections > shared_buffers THEN\n RAISE WARNING 'work_mem setting of % is set high for % max_connections please reduce work_mem to % or decrease max_connections to %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers/max_connections), floor(shared_buffers/work_mem);\n ELSIF work_mem * max_connections < (shared_buffers * 0.75) THEN\n RAISE WARNING 'work_mem setting of % is set low for % max_connections you may consider raising work_mem to % or increasing max_connections to %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers/max_connections), floor(shared_buffers/work_mem);\n ELSE\n RAISE NOTICE 'work_mem setting of % and max_connections of % are adequate for shared_buffers of %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers);\n END IF;\n\n IF random_page_cost / seq_page_cost != 1.1 THEN\n RAISE WARNING 'random_page_cost (%) /seq_page_cost (%) should be set to 1.1 for SSD. Change random_page_cost to %', random_page_cost, seq_page_cost, 1.1 * seq_page_cost;\n ELSE\n RAISE NOTICE 'random_page_cost and seq_page_cost set appropriately for SSD';\n END IF;\n\n IF temp_buffers < greatest(pg_size_bytes('128MB'),(maintenance_work_mem / 2)) THEN\n RAISE WARNING 'pgstac makes heavy use of temp tables, consider raising temp_buffers from % to %', pg_size_pretty(temp_buffers), greatest('128MB', pg_size_pretty((shared_buffers / 16)));\n END IF;\n END IF;\n\n RAISE NOTICE 'VALUES FOR PGSTAC VARIABLES';\n RAISE NOTICE 'These can be set either as GUC system variables or by setting in the pgstac_settings table.';\n\n FOR r IN SELECT name, get_setting(name) as setting, CASE WHEN current_setting(concat('pgstac.',name), TRUE) IS NOT NULL THEN concat('pgstac.',name, ' GUC') WHEN value IS NOT NULL THEN 'pgstac_settings table' ELSE 'Not Set' END as loc FROM pgstac_settings LOOP\n RAISE NOTICE '% is set to % from the %', r.name, r.setting, r.loc;\n END LOOP;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pg_cron';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider intalling pg_cron which can be used to automate tasks';\n ELSE\n RAISE NOTICE 'pg_cron % is installed', settingval;\n END IF;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pgstattuple';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider installing the pgstattuple extension which can be used to help maintain tables and indexes.';\n ELSE\n RAISE NOTICE 'pgstattuple % is installed', settingval;\n END IF;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pg_stat_statements';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider installing the pg_stat_statements extension which is very helpful for tracking the types of queries on the system';\n ELSE\n RAISE NOTICE 'pg_stat_statements % is installed', settingval;\n IF current_setting('pg_stat_statements.track_statements', TRUE) IS DISTINCT FROM 'all' THEN\n RAISE WARNING 'SET pg_stat_statements.track_statements TO ''all''; --In order to track statements within functions.';\n END IF;\n END IF;\n\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac, public SET CLIENT_MIN_MESSAGES TO NOTICE;\n/* looks for a geometry in a stac item first from geometry and falling back to bbox */\nCREATE OR REPLACE FUNCTION stac_geom(value jsonb) RETURNS geometry AS $$\nSELECT\n CASE\n WHEN value ? 'intersects' THEN\n ST_GeomFromGeoJSON(value->>'intersects')\n WHEN value ? 'geometry' THEN\n ST_GeomFromGeoJSON(value->>'geometry')\n WHEN value ? 'bbox' THEN\n pgstac.bbox_geom(value->'bbox')\n ELSE NULL\n END as geometry\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION stac_daterange(\n value jsonb\n) RETURNS tstzrange AS $$\nDECLARE\n props jsonb := value;\n dt timestamptz;\n edt timestamptz;\nBEGIN\n IF props ? 'properties' THEN\n props := props->'properties';\n END IF;\n IF\n props ? 'start_datetime'\n AND props->>'start_datetime' IS NOT NULL\n AND props ? 'end_datetime'\n AND props->>'end_datetime' IS NOT NULL\n THEN\n dt := props->>'start_datetime';\n edt := props->>'end_datetime';\n IF dt > edt THEN\n RAISE EXCEPTION 'start_datetime must be < end_datetime';\n END IF;\n ELSE\n dt := props->>'datetime';\n edt := props->>'datetime';\n END IF;\n IF dt is NULL OR edt IS NULL THEN\n RAISE NOTICE 'DT: %, EDT: %', dt, edt;\n RAISE EXCEPTION 'Either datetime (%) or both start_datetime (%) and end_datetime (%) must be set.', props->>'datetime',props->>'start_datetime',props->>'end_datetime';\n END IF;\n RETURN tstzrange(dt, edt, '[]');\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_datetime(value jsonb) RETURNS timestamptz AS $$\n SELECT lower(stac_daterange(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_end_datetime(value jsonb) RETURNS timestamptz AS $$\n SELECT upper(stac_daterange(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE TABLE IF NOT EXISTS stac_extensions(\n url text PRIMARY KEY,\n content jsonb\n);\nCREATE OR REPLACE FUNCTION queryable_signature(n text, c text[]) RETURNS text AS $$\n SELECT concat(n, c);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TABLE queryables (\n id bigint GENERATED ALWAYS AS identity PRIMARY KEY,\n name text NOT NULL,\n collection_ids text[], -- used to determine what partitions to create indexes on\n definition jsonb,\n property_path text,\n property_wrapper text,\n property_index_type text\n);\nCREATE INDEX queryables_name_idx ON queryables (name);\nCREATE INDEX queryables_collection_idx ON queryables USING GIN (collection_ids);\nCREATE INDEX queryables_property_wrapper_idx ON queryables (property_wrapper);\n\nCREATE OR REPLACE FUNCTION pgstac.queryables_constraint_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n allcollections text[];\nBEGIN\n RAISE NOTICE 'Making sure that name/collection is unique for queryables %', NEW;\n IF NEW.collection_ids IS NOT NULL THEN\n IF EXISTS (\n SELECT 1\n FROM unnest(NEW.collection_ids) c\n LEFT JOIN\n collections\n ON (collections.id = c)\n WHERE collections.id IS NULL\n ) THEN\n RAISE foreign_key_violation USING MESSAGE = format(\n 'One or more collections in %s do not exist.', NEW.collection_ids\n );\n RETURN NULL;\n END IF;\n END IF;\n IF TG_OP = 'INSERT' THEN\n IF EXISTS (\n SELECT 1 FROM queryables q\n WHERE\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n )\n ) THEN\n RAISE unique_violation USING MESSAGE = format(\n 'There is already a queryable for %s for a collection in %s: %s',\n NEW.name,\n NEW.collection_ids,\n\t\t\t\t(SELECT json_agg(row_to_json(q)) FROM queryables q WHERE\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n ))\n );\n RETURN NULL;\n END IF;\n END IF;\n IF TG_OP = 'UPDATE' THEN\n IF EXISTS (\n SELECT 1 FROM queryables q\n WHERE\n q.id != NEW.id\n AND\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n )\n ) THEN\n RAISE unique_violation\n USING MESSAGE = format(\n 'There is already a queryable for %s for a collection in %s',\n NEW.name,\n NEW.collection_ids\n );\n RETURN NULL;\n END IF;\n END IF;\n\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER queryables_constraint_insert_trigger\nBEFORE INSERT ON queryables\nFOR EACH ROW EXECUTE PROCEDURE queryables_constraint_triggerfunc();\n\nCREATE TRIGGER queryables_constraint_update_trigger\nBEFORE UPDATE ON queryables\nFOR EACH ROW\nWHEN (NEW.name = OLD.name AND NEW.collection_ids IS DISTINCT FROM OLD.collection_ids)\nEXECUTE PROCEDURE queryables_constraint_triggerfunc();\n\n\nCREATE OR REPLACE FUNCTION array_to_path(arr text[]) RETURNS text AS $$\n SELECT string_agg(\n quote_literal(v),\n '->'\n ) FROM unnest(arr) v;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\n\n\nCREATE OR REPLACE FUNCTION queryable(\n IN dotpath text,\n OUT path text,\n OUT expression text,\n OUT wrapper text,\n OUT nulled_wrapper text\n) AS $$\nDECLARE\n q RECORD;\n path_elements text[];\nBEGIN\n IF dotpath IN ('id', 'geometry', 'datetime', 'end_datetime', 'collection') THEN\n path := dotpath;\n expression := dotpath;\n wrapper := NULL;\n RETURN;\n END IF;\n SELECT * INTO q FROM queryables\n WHERE\n name=dotpath\n OR name = 'properties.' || dotpath\n OR name = replace(dotpath, 'properties.', '')\n ;\n IF q.property_wrapper IS NULL THEN\n IF q.definition->>'type' = 'number' THEN\n wrapper := 'to_float';\n nulled_wrapper := wrapper;\n ELSIF q.definition->>'format' = 'date-time' THEN\n wrapper := 'to_tstz';\n nulled_wrapper := wrapper;\n ELSE\n nulled_wrapper := NULL;\n wrapper := 'to_text';\n END IF;\n ELSE\n wrapper := q.property_wrapper;\n nulled_wrapper := wrapper;\n END IF;\n IF q.property_path IS NOT NULL THEN\n path := q.property_path;\n ELSE\n path_elements := string_to_array(dotpath, '.');\n IF path_elements[1] IN ('links', 'assets', 'stac_version', 'stac_extensions') THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSIF path_elements[1] = 'properties' THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSE\n path := format($F$content->'properties'->%s$F$, array_to_path(path_elements));\n END IF;\n END IF;\n expression := format('%I(%s)', wrapper, path);\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\nCREATE OR REPLACE FUNCTION unnest_collection(collection_ids text[] DEFAULT NULL) RETURNS SETOF text AS $$\n DECLARE\n BEGIN\n IF collection_ids IS NULL THEN\n RETURN QUERY SELECT id FROM collections;\n END IF;\n RETURN QUERY SELECT unnest(collection_ids);\n END;\n$$ LANGUAGE PLPGSQL STABLE;\n\nCREATE OR REPLACE FUNCTION normalize_indexdef(def text) RETURNS text AS $$\nDECLARE\nBEGIN\n def := btrim(def, ' \\n\\t');\n\tdef := regexp_replace(def, '^CREATE (UNIQUE )?INDEX ([^ ]* )?ON (ONLY )?([^ ]* )?', '', 'i');\n RETURN def;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE STRICT PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION indexdef(q queryables) RETURNS text AS $$\n DECLARE\n out text;\n BEGIN\n IF q.name = 'id' THEN\n out := 'CREATE UNIQUE INDEX ON %I USING btree (id)';\n ELSIF q.name = 'datetime' THEN\n out := 'CREATE INDEX ON %I USING btree (datetime DESC, end_datetime)';\n ELSIF q.name = 'geometry' THEN\n out := 'CREATE INDEX ON %I USING gist (geometry)';\n ELSE\n out := format($q$CREATE INDEX ON %%I USING %s (%s(((content -> 'properties'::text) -> %L::text)))$q$,\n lower(COALESCE(q.property_index_type, 'BTREE')),\n lower(COALESCE(q.property_wrapper, 'to_text')),\n q.name\n );\n END IF;\n RETURN btrim(out, ' \\n\\t');\n END;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\n\nDROP VIEW IF EXISTS pgstac_indexes;\nCREATE VIEW pgstac_indexes AS\nSELECT\n i.schemaname,\n i.tablename,\n i.indexname,\n regexp_replace(btrim(replace(replace(indexdef, i.indexname, ''),'pgstac.',''),' \\t\\n'), '[ ]+', ' ', 'g') as idx,\n COALESCE(\n (regexp_match(indexdef, '\\(([a-zA-Z]+)\\)'))[1],\n (regexp_match(indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_-]+)''::text'))[1],\n CASE WHEN indexdef ~* '\\(datetime desc, end_datetime\\)' THEN 'datetime' ELSE NULL END\n ) AS field,\n pg_table_size(i.indexname::text) as index_size,\n pg_size_pretty(pg_table_size(i.indexname::text)) as index_size_pretty\nFROM\n pg_indexes i\nWHERE i.schemaname='pgstac' and i.tablename ~ '_items_' AND indexdef !~* ' only ';\n\nDROP VIEW IF EXISTS pgstac_index_stats;\nCREATE VIEW pgstac_indexes_stats AS\nSELECT\n i.schemaname,\n i.tablename,\n i.indexname,\n indexdef,\n COALESCE(\n (regexp_match(indexdef, '\\(([a-zA-Z]+)\\)'))[1],\n (regexp_match(indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_]+)''::text'))[1],\n CASE WHEN indexdef ~* '\\(datetime desc, end_datetime\\)' THEN 'datetime_end_datetime' ELSE NULL END\n ) AS field,\n pg_table_size(i.indexname::text) as index_size,\n pg_size_pretty(pg_table_size(i.indexname::text)) as index_size_pretty,\n n_distinct,\n most_common_vals::text::text[],\n most_common_freqs::text::text[],\n histogram_bounds::text::text[],\n correlation\nFROM\n pg_indexes i\n LEFT JOIN pg_stats s ON (s.tablename = i.indexname)\nWHERE i.schemaname='pgstac' and i.tablename ~ '_items_';\n\nset check_function_bodies to off;\nCREATE OR REPLACE FUNCTION maintain_partition_queries(\n part text DEFAULT 'items',\n dropindexes boolean DEFAULT FALSE,\n rebuildindexes boolean DEFAULT FALSE,\n idxconcurrently boolean DEFAULT FALSE\n) RETURNS SETOF text AS $$\nDECLARE\n rec record;\nBEGIN\n FOR rec IN (\n WITH p AS (\n SELECT\n relid::text as partition,\n replace(replace(\n CASE\n WHEN level = 1 THEN pg_get_expr(c.relpartbound, c.oid)\n ELSE pg_get_expr(parent.relpartbound, parent.oid)\n END,\n 'FOR VALUES IN (''',''), ''')',\n ''\n ) AS collection\n FROM pg_partition_tree('items')\n JOIN pg_class c ON (relid::regclass = c.oid)\n JOIN pg_class parent ON (parentrelid::regclass = parent.oid AND isleaf)\n ), i AS (\n SELECT\n partition,\n indexname,\n regexp_replace(btrim(replace(replace(indexdef, indexname, ''),'pgstac.',''),' \\t\\n'), '[ ]+', ' ', 'g') as iidx,\n COALESCE(\n (regexp_match(indexdef, '\\(([a-zA-Z]+)\\)'))[1],\n (regexp_match(indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_-]+)''::text'))[1],\n CASE WHEN indexdef ~* '\\(datetime desc, end_datetime\\)' THEN 'datetime' ELSE NULL END\n ) AS field\n FROM\n pg_indexes\n JOIN p ON (tablename=partition)\n ), q AS (\n SELECT\n name AS field,\n collection,\n partition,\n format(indexdef(queryables), partition) as qidx\n FROM queryables, unnest_collection(queryables.collection_ids) collection\n JOIN p USING (collection)\n WHERE property_index_type IS NOT NULL OR name IN ('datetime','geometry','id')\n )\n SELECT * FROM i FULL JOIN q USING (field, partition)\n WHERE lower(iidx) IS DISTINCT FROM lower(qidx)\n ) LOOP\n IF rec.iidx IS NULL THEN\n IF idxconcurrently THEN\n RETURN NEXT replace(rec.qidx, 'INDEX', 'INDEX CONCURRENTLY');\n ELSE\n RETURN NEXT rec.qidx;\n END IF;\n ELSIF rec.qidx IS NULL AND dropindexes THEN\n RETURN NEXT format('DROP INDEX IF EXISTS %I;', rec.indexname);\n ELSIF lower(rec.qidx) != lower(rec.iidx) THEN\n IF dropindexes THEN\n RETURN NEXT format('DROP INDEX IF EXISTS %I; %s;', rec.indexname, rec.qidx);\n ELSE\n IF idxconcurrently THEN\n RETURN NEXT replace(rec.qidx, 'INDEX', 'INDEX CONCURRENTLY');\n ELSE\n RETURN NEXT rec.qidx;\n END IF;\n END IF;\n ELSIF rebuildindexes and rec.indexname IS NOT NULL THEN\n IF idxconcurrently THEN\n RETURN NEXT format('REINDEX INDEX CONCURRENTLY %I;', rec.indexname);\n ELSE\n RETURN NEXT format('REINDEX INDEX %I;', rec.indexname);\n END IF;\n END IF;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION maintain_partitions(\n part text DEFAULT 'items',\n dropindexes boolean DEFAULT FALSE,\n rebuildindexes boolean DEFAULT FALSE\n) RETURNS VOID AS $$\n WITH t AS (\n SELECT run_or_queue(q) FROM maintain_partition_queries(part, dropindexes, rebuildindexes) q\n ) SELECT count(*) FROM t;\n$$ LANGUAGE SQL;\n\n\nCREATE OR REPLACE FUNCTION queryables_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\nBEGIN\n PERFORM maintain_partitions();\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER queryables_trigger AFTER INSERT OR UPDATE ON queryables\nFOR EACH STATEMENT EXECUTE PROCEDURE queryables_trigger_func();\n\n\nCREATE OR REPLACE FUNCTION get_queryables(_collection_ids text[] DEFAULT NULL) RETURNS jsonb AS $$\nDECLARE\nBEGIN\n -- Build up queryables if the input contains valid collection ids or is empty\n IF EXISTS (\n SELECT 1 FROM collections\n WHERE\n _collection_ids IS NULL\n OR cardinality(_collection_ids) = 0\n OR id = ANY(_collection_ids)\n )\n THEN\n RETURN (\n WITH base AS (\n SELECT\n unnest(collection_ids) as collection_id,\n name,\n coalesce(definition, '{\"type\":\"string\"}'::jsonb) as definition\n FROM queryables\n WHERE\n _collection_ids IS NULL OR\n _collection_ids = '{}'::text[] OR\n _collection_ids && collection_ids\n UNION ALL\n SELECT null, name, coalesce(definition, '{\"type\":\"string\"}'::jsonb) as definition\n FROM queryables WHERE collection_ids IS NULL OR collection_ids = '{}'::text[]\n ), g AS (\n SELECT\n name,\n first_notnull(definition) as definition,\n jsonb_array_unique_merge(definition->'enum') as enum,\n jsonb_min(definition->'minimum') as minimum,\n jsonb_min(definition->'maxiumn') as maximum\n FROM base\n GROUP BY 1\n )\n SELECT\n jsonb_build_object(\n '$schema', 'http://json-schema.org/draft-07/schema#',\n '$id', '',\n 'type', 'object',\n 'title', 'STAC Queryables.',\n 'properties', jsonb_object_agg(\n name,\n definition\n ||\n jsonb_strip_nulls(jsonb_build_object(\n 'enum', enum,\n 'minimum', minimum,\n 'maximum', maximum\n ))\n )\n )\n FROM g\n );\n ELSE\n RETURN NULL;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\nCREATE OR REPLACE FUNCTION get_queryables(_collection text DEFAULT NULL) RETURNS jsonb AS $$\n SELECT\n CASE\n WHEN _collection IS NULL THEN get_queryables(NULL::text[])\n ELSE get_queryables(ARRAY[_collection])\n END\n ;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_queryables() RETURNS jsonb AS $$\n SELECT get_queryables(NULL::text[]);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION schema_qualify_refs(url text, j jsonb) returns jsonb as $$\n SELECT regexp_replace(j::text, '\"\\$ref\": \"#', concat('\"$ref\": \"', url, '#'), 'g')::jsonb;\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE VIEW stac_extension_queryables AS\nSELECT DISTINCT key as name, schema_qualify_refs(e.url, j.value) as definition FROM stac_extensions e, jsonb_each(e.content->'definitions'->'fields'->'properties') j;\n\n\nCREATE OR REPLACE FUNCTION missing_queryables(_collection text, _tablesample float DEFAULT 5, minrows float DEFAULT 10) RETURNS TABLE(collection text, name text, definition jsonb, property_wrapper text) AS $$\nDECLARE\n q text;\n _partition text;\n explain_json json;\n psize float;\n estrows float;\nBEGIN\n SELECT format('_items_%s', key) INTO _partition FROM collections WHERE id=_collection;\n\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM %I;', _partition)\n INTO explain_json;\n psize := explain_json->0->'Plan'->'Plan Rows';\n estrows := _tablesample * .01 * psize;\n IF estrows < minrows THEN\n _tablesample := least(100,greatest(_tablesample, (estrows / psize) / 100));\n RAISE NOTICE '%', (psize / estrows) / 100;\n END IF;\n RAISE NOTICE 'Using tablesample % to find missing queryables from % % that has ~% rows estrows: %', _tablesample, _collection, _partition, psize, estrows;\n\n q := format(\n $q$\n WITH q AS (\n SELECT * FROM queryables\n WHERE\n collection_ids IS NULL\n OR %L = ANY(collection_ids)\n ), t AS (\n SELECT\n content->'properties' AS properties\n FROM\n %I\n TABLESAMPLE SYSTEM(%L)\n ), p AS (\n SELECT DISTINCT ON (key)\n key,\n value,\n s.definition\n FROM t\n JOIN LATERAL jsonb_each(properties) ON TRUE\n LEFT JOIN q ON (q.name=key)\n LEFT JOIN stac_extension_queryables s ON (s.name=key)\n WHERE q.definition IS NULL\n )\n SELECT\n %L,\n key,\n COALESCE(definition, jsonb_build_object('type',jsonb_typeof(value))) as definition,\n CASE\n WHEN definition->>'type' = 'integer' THEN 'to_int'\n WHEN COALESCE(definition->>'type', jsonb_typeof(value)) = 'number' THEN 'to_float'\n WHEN COALESCE(definition->>'type', jsonb_typeof(value)) = 'array' THEN 'to_text_array'\n ELSE 'to_text'\n END\n FROM p;\n $q$,\n _collection,\n _partition,\n _tablesample,\n _collection\n );\n RETURN QUERY EXECUTE q;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION missing_queryables(_tablesample float DEFAULT 5) RETURNS TABLE(collection_ids text[], name text, definition jsonb, property_wrapper text) AS $$\n SELECT\n array_agg(collection),\n name,\n definition,\n property_wrapper\n FROM\n collections\n JOIN LATERAL\n missing_queryables(id, _tablesample) c\n ON TRUE\n GROUP BY\n 2,3,4\n ORDER BY 2,1\n ;\n$$ LANGUAGE SQL;\nCREATE OR REPLACE FUNCTION parse_dtrange(\n _indate jsonb,\n relative_base timestamptz DEFAULT date_trunc('hour', CURRENT_TIMESTAMP)\n) RETURNS tstzrange AS $$\nDECLARE\n timestrs text[];\n s timestamptz;\n e timestamptz;\nBEGIN\n timestrs :=\n CASE\n WHEN _indate ? 'timestamp' THEN\n ARRAY[_indate->>'timestamp']\n WHEN _indate ? 'interval' THEN\n to_text_array(_indate->'interval')\n WHEN jsonb_typeof(_indate) = 'array' THEN\n to_text_array(_indate)\n ELSE\n regexp_split_to_array(\n _indate->>0,\n '/'\n )\n END;\n RAISE NOTICE 'TIMESTRS %', timestrs;\n IF cardinality(timestrs) = 1 THEN\n IF timestrs[1] ILIKE 'P%' THEN\n RETURN tstzrange(relative_base - upper(timestrs[1])::interval, relative_base, '[)');\n END IF;\n s := timestrs[1]::timestamptz;\n RETURN tstzrange(s, s, '[]');\n END IF;\n\n IF cardinality(timestrs) != 2 THEN\n RAISE EXCEPTION 'Timestamp cannot have more than 2 values';\n END IF;\n\n IF timestrs[1] = '..' OR timestrs[1] = '' THEN\n s := '-infinity'::timestamptz;\n e := timestrs[2]::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] = '..' OR timestrs[2] = '' THEN\n s := timestrs[1]::timestamptz;\n e := 'infinity'::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[1] ILIKE 'P%' AND timestrs[2] NOT ILIKE 'P%' THEN\n e := timestrs[2]::timestamptz;\n s := e - upper(timestrs[1])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] ILIKE 'P%' AND timestrs[1] NOT ILIKE 'P%' THEN\n s := timestrs[1]::timestamptz;\n e := s + upper(timestrs[2])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n s := timestrs[1]::timestamptz;\n e := timestrs[2]::timestamptz;\n\n RETURN tstzrange(s,e,'[)');\n\n RETURN NULL;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT PARALLEL SAFE SET TIME ZONE 'UTC';\n\nCREATE OR REPLACE FUNCTION parse_dtrange(\n _indate text,\n relative_base timestamptz DEFAULT CURRENT_TIMESTAMP\n) RETURNS tstzrange AS $$\n SELECT parse_dtrange(to_jsonb(_indate), relative_base);\n$$ LANGUAGE SQL STABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION temporal_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\n ll text := 'datetime';\n lh text := 'end_datetime';\n rrange tstzrange;\n rl text;\n rh text;\n outq text;\nBEGIN\n rrange := parse_dtrange(args->1);\n RAISE NOTICE 'Constructing temporal query OP: %, ARGS: %, RRANGE: %', op, args, rrange;\n op := lower(op);\n rl := format('%L::timestamptz', lower(rrange));\n rh := format('%L::timestamptz', upper(rrange));\n outq := CASE op\n WHEN 't_before' THEN 'lh < rl'\n WHEN 't_after' THEN 'll > rh'\n WHEN 't_meets' THEN 'lh = rl'\n WHEN 't_metby' THEN 'll = rh'\n WHEN 't_overlaps' THEN 'll < rl AND rl < lh < rh'\n WHEN 't_overlappedby' THEN 'rl < ll < rh AND lh > rh'\n WHEN 't_starts' THEN 'll = rl AND lh < rh'\n WHEN 't_startedby' THEN 'll = rl AND lh > rh'\n WHEN 't_during' THEN 'll > rl AND lh < rh'\n WHEN 't_contains' THEN 'll < rl AND lh > rh'\n WHEN 't_finishes' THEN 'll > rl AND lh = rh'\n WHEN 't_finishedby' THEN 'll < rl AND lh = rh'\n WHEN 't_equals' THEN 'll = rl AND lh = rh'\n WHEN 't_disjoint' THEN 'NOT (ll <= rh AND lh >= rl)'\n WHEN 't_intersects' THEN 'll <= rh AND lh >= rl'\n WHEN 'anyinteracts' THEN 'll <= rh AND lh >= rl'\n END;\n outq := regexp_replace(outq, '\\mll\\M', ll);\n outq := regexp_replace(outq, '\\mlh\\M', lh);\n outq := regexp_replace(outq, '\\mrl\\M', rl);\n outq := regexp_replace(outq, '\\mrh\\M', rh);\n outq := format('(%s)', outq);\n RETURN outq;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\n\n\nCREATE OR REPLACE FUNCTION spatial_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\n geom text;\n j jsonb := args->1;\nBEGIN\n op := lower(op);\n RAISE NOTICE 'Constructing spatial query OP: %, ARGS: %', op, args;\n IF op NOT IN ('s_equals','s_disjoint','s_touches','s_within','s_overlaps','s_crosses','s_intersects','intersects','s_contains') THEN\n RAISE EXCEPTION 'Spatial Operator % Not Supported', op;\n END IF;\n op := regexp_replace(op, '^s_', 'st_');\n IF op = 'intersects' THEN\n op := 'st_intersects';\n END IF;\n -- Convert geometry to WKB string\n IF j ? 'type' AND j ? 'coordinates' THEN\n geom := st_geomfromgeojson(j)::text;\n ELSIF jsonb_typeof(j) = 'array' THEN\n geom := bbox_geom(j)::text;\n END IF;\n\n RETURN format('%s(geometry, %L::geometry)', op, geom);\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION query_to_cql2(q jsonb) RETURNS jsonb AS $$\n-- Translates anything passed in through the deprecated \"query\" into equivalent CQL2\nWITH t AS (\n SELECT key as property, value as ops\n FROM jsonb_each(q)\n), t2 AS (\n SELECT property, (jsonb_each(ops)).*\n FROM t WHERE jsonb_typeof(ops) = 'object'\n UNION ALL\n SELECT property, 'eq', ops\n FROM t WHERE jsonb_typeof(ops) != 'object'\n)\nSELECT\n jsonb_strip_nulls(jsonb_build_object(\n 'op', 'and',\n 'args', jsonb_agg(\n jsonb_build_object(\n 'op', key,\n 'args', jsonb_build_array(\n jsonb_build_object('property',property),\n value\n )\n )\n )\n )\n) as qcql FROM t2\n;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\nCREATE OR REPLACE FUNCTION cql1_to_cql2(j jsonb) RETURNS jsonb AS $$\nDECLARE\n args jsonb;\n ret jsonb;\nBEGIN\n RAISE NOTICE 'CQL1_TO_CQL2: %', j;\n IF j ? 'filter' THEN\n RETURN cql1_to_cql2(j->'filter');\n END IF;\n IF j ? 'property' THEN\n RETURN j;\n END IF;\n IF jsonb_typeof(j) = 'array' THEN\n SELECT jsonb_agg(cql1_to_cql2(el)) INTO args FROM jsonb_array_elements(j) el;\n RETURN args;\n END IF;\n IF jsonb_typeof(j) = 'number' THEN\n RETURN j;\n END IF;\n IF jsonb_typeof(j) = 'string' THEN\n RETURN j;\n END IF;\n\n IF jsonb_typeof(j) = 'object' THEN\n SELECT jsonb_build_object(\n 'op', key,\n 'args', cql1_to_cql2(value)\n ) INTO ret\n FROM jsonb_each(j)\n WHERE j IS NOT NULL;\n RETURN ret;\n END IF;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE STRICT;\n\nCREATE TABLE cql2_ops (\n op text PRIMARY KEY,\n template text,\n types text[]\n);\nINSERT INTO cql2_ops (op, template, types) VALUES\n ('eq', '%s = %s', NULL),\n ('neq', '%s != %s', NULL),\n ('ne', '%s != %s', NULL),\n ('!=', '%s != %s', NULL),\n ('<>', '%s != %s', NULL),\n ('lt', '%s < %s', NULL),\n ('lte', '%s <= %s', NULL),\n ('gt', '%s > %s', NULL),\n ('gte', '%s >= %s', NULL),\n ('le', '%s <= %s', NULL),\n ('ge', '%s >= %s', NULL),\n ('=', '%s = %s', NULL),\n ('<', '%s < %s', NULL),\n ('<=', '%s <= %s', NULL),\n ('>', '%s > %s', NULL),\n ('>=', '%s >= %s', NULL),\n ('like', '%s LIKE %s', NULL),\n ('ilike', '%s ILIKE %s', NULL),\n ('+', '%s + %s', NULL),\n ('-', '%s - %s', NULL),\n ('*', '%s * %s', NULL),\n ('/', '%s / %s', NULL),\n ('not', 'NOT (%s)', NULL),\n ('between', '%s BETWEEN %s AND %s', NULL),\n ('isnull', '%s IS NULL', NULL),\n ('upper', 'upper(%s)', NULL),\n ('lower', 'lower(%s)', NULL)\nON CONFLICT (op) DO UPDATE\n SET\n template = EXCLUDED.template\n;\n\n\nCREATE OR REPLACE FUNCTION cql2_query(j jsonb, wrapper text DEFAULT NULL) RETURNS text AS $$\n#variable_conflict use_variable\nDECLARE\n args jsonb := j->'args';\n arg jsonb;\n op text := lower(j->>'op');\n cql2op RECORD;\n literal text;\n _wrapper text;\n leftarg text;\n rightarg text;\nBEGIN\n IF j IS NULL OR (op IS NOT NULL AND args IS NULL) THEN\n RETURN NULL;\n END IF;\n RAISE NOTICE 'CQL2_QUERY: %', j;\n IF j ? 'filter' THEN\n RETURN cql2_query(j->'filter');\n END IF;\n\n IF j ? 'upper' THEN\n RETURN cql2_query(jsonb_build_object('op', 'upper', 'args', j->'upper'));\n END IF;\n\n IF j ? 'lower' THEN\n RETURN cql2_query(jsonb_build_object('op', 'lower', 'args', j->'lower'));\n END IF;\n\n -- Temporal Query\n IF op ilike 't_%' or op = 'anyinteracts' THEN\n RETURN temporal_op_query(op, args);\n END IF;\n\n -- If property is a timestamp convert it to text to use with\n -- general operators\n IF j ? 'timestamp' THEN\n RETURN format('%L::timestamptz', to_tstz(j->'timestamp'));\n END IF;\n IF j ? 'interval' THEN\n RAISE EXCEPTION 'Please use temporal operators when using intervals.';\n RETURN NONE;\n END IF;\n\n -- Spatial Query\n IF op ilike 's_%' or op = 'intersects' THEN\n RETURN spatial_op_query(op, args);\n END IF;\n\n IF op IN ('a_equals','a_contains','a_contained_by','a_overlaps') THEN\n IF args->0 ? 'property' THEN\n leftarg := format('to_text_array(%s)', (queryable(args->0->>'property')).path);\n END IF;\n IF args->1 ? 'property' THEN\n rightarg := format('to_text_array(%s)', (queryable(args->1->>'property')).path);\n END IF;\n RETURN FORMAT(\n '%s %s %s',\n COALESCE(leftarg, quote_literal(to_text_array(args->0))),\n CASE op\n WHEN 'a_equals' THEN '='\n WHEN 'a_contains' THEN '@>'\n WHEN 'a_contained_by' THEN '<@'\n WHEN 'a_overlaps' THEN '&&'\n END,\n COALESCE(rightarg, quote_literal(to_text_array(args->1)))\n );\n END IF;\n\n IF op = 'in' THEN\n RAISE NOTICE 'IN : % % %', args, jsonb_build_array(args->0), args->1;\n args := jsonb_build_array(args->0) || (args->1);\n RAISE NOTICE 'IN2 : %', args;\n END IF;\n\n\n\n IF op = 'between' THEN\n args = jsonb_build_array(\n args->0,\n args->1->0,\n args->1->1\n );\n END IF;\n\n -- Make sure that args is an array and run cql2_query on\n -- each element of the array\n RAISE NOTICE 'ARGS PRE: %', args;\n IF j ? 'args' THEN\n IF jsonb_typeof(args) != 'array' THEN\n args := jsonb_build_array(args);\n END IF;\n\n IF jsonb_path_exists(args, '$[*] ? (@.property == \"id\" || @.property == \"datetime\" || @.property == \"end_datetime\" || @.property == \"collection\")') THEN\n wrapper := NULL;\n ELSE\n -- if any of the arguments are a property, try to get the property_wrapper\n FOR arg IN SELECT jsonb_path_query(args, '$[*] ? (@.property != null)') LOOP\n RAISE NOTICE 'Arg: %', arg;\n wrapper := (queryable(arg->>'property')).nulled_wrapper;\n RAISE NOTICE 'Property: %, Wrapper: %', arg, wrapper;\n IF wrapper IS NOT NULL THEN\n EXIT;\n END IF;\n END LOOP;\n\n -- if the property was not in queryables, see if any args were numbers\n IF\n wrapper IS NULL\n AND jsonb_path_exists(args, '$[*] ? (@.type()==\"number\")')\n THEN\n wrapper := 'to_float';\n END IF;\n wrapper := coalesce(wrapper, 'to_text');\n END IF;\n\n SELECT jsonb_agg(cql2_query(a, wrapper))\n INTO args\n FROM jsonb_array_elements(args) a;\n END IF;\n RAISE NOTICE 'ARGS: %', args;\n\n IF op IN ('and', 'or') THEN\n RETURN\n format(\n '(%s)',\n array_to_string(to_text_array(args), format(' %s ', upper(op)))\n );\n END IF;\n\n IF op = 'in' THEN\n RAISE NOTICE 'IN -- % %', args->0, to_text(args->0);\n RETURN format(\n '%s IN (%s)',\n to_text(args->0),\n array_to_string((to_text_array(args))[2:], ',')\n );\n END IF;\n\n -- Look up template from cql2_ops\n IF j ? 'op' THEN\n SELECT * INTO cql2op FROM cql2_ops WHERE cql2_ops.op ilike op;\n IF FOUND THEN\n -- If specific index set in queryables for a property cast other arguments to that type\n\n RETURN format(\n cql2op.template,\n VARIADIC (to_text_array(args))\n );\n ELSE\n RAISE EXCEPTION 'Operator % Not Supported.', op;\n END IF;\n END IF;\n\n\n IF wrapper IS NOT NULL THEN\n RAISE NOTICE 'Wrapping % with %', j, wrapper;\n IF j ? 'property' THEN\n RETURN format('%I(%s)', wrapper, (queryable(j->>'property')).path);\n ELSE\n RETURN format('%I(%L)', wrapper, j);\n END IF;\n ELSIF j ? 'property' THEN\n RETURN quote_ident(j->>'property');\n END IF;\n\n RETURN quote_literal(to_text(j));\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION paging_dtrange(\n j jsonb\n) RETURNS tstzrange AS $$\nDECLARE\n op text;\n filter jsonb := j->'filter';\n dtrange tstzrange := tstzrange('-infinity'::timestamptz,'infinity'::timestamptz);\n sdate timestamptz := '-infinity'::timestamptz;\n edate timestamptz := 'infinity'::timestamptz;\n jpitem jsonb;\nBEGIN\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n END IF;\n IF NOT (filter @? '$.**.op ? (@ == \"or\" || @ == \"not\")') THEN\n FOR jpitem IN SELECT j FROM jsonb_path_query(filter,'strict $.** ? (@.args[*].property == \"datetime\")'::jsonpath) j LOOP\n op := lower(jpitem->>'op');\n dtrange := parse_dtrange(jpitem->'args'->1);\n IF op IN ('<=', 'lt', 'lte', '<', 'le', 't_before') THEN\n sdate := greatest(sdate,'-infinity');\n edate := least(edate, upper(dtrange));\n ELSIF op IN ('>=', '>', 'gt', 'gte', 'ge', 't_after') THEN\n edate := least(edate, 'infinity');\n sdate := greatest(sdate, lower(dtrange));\n ELSIF op IN ('=', 'eq') THEN\n edate := least(edate, upper(dtrange));\n sdate := greatest(sdate, lower(dtrange));\n END IF;\n RAISE NOTICE '2 OP: %, ARGS: %, DTRANGE: %, SDATE: %, EDATE: %', op, jpitem->'args'->1, dtrange, sdate, edate;\n END LOOP;\n END IF;\n IF sdate > edate THEN\n RETURN 'empty'::tstzrange;\n END IF;\n RETURN tstzrange(sdate,edate, '[]');\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT SET TIME ZONE 'UTC';\n\nCREATE OR REPLACE FUNCTION paging_collections(\n IN j jsonb\n) RETURNS text[] AS $$\nDECLARE\n filter jsonb := j->'filter';\n jpitem jsonb;\n op text;\n args jsonb;\n arg jsonb;\n collections text[];\nBEGIN\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n END IF;\n IF NOT (filter @? '$.**.op ? (@ == \"or\" || @ == \"not\")') THEN\n FOR jpitem IN SELECT j FROM jsonb_path_query(filter,'strict $.** ? (@.args[*].property == \"collection\")'::jsonpath) j LOOP\n RAISE NOTICE 'JPITEM: %', jpitem;\n op := jpitem->>'op';\n args := jpitem->'args';\n IF op IN ('=', 'eq', 'in') THEN\n FOR arg IN SELECT a FROM jsonb_array_elements(args) a LOOP\n IF jsonb_typeof(arg) IN ('string', 'array') THEN\n RAISE NOTICE 'arg: %, collections: %', arg, collections;\n IF collections IS NULL OR collections = '{}'::text[] THEN\n collections := to_text_array(arg);\n ELSE\n collections := array_intersection(collections, to_text_array(arg));\n END IF;\n END IF;\n END LOOP;\n END IF;\n END LOOP;\n END IF;\n IF collections = '{}'::text[] THEN\n RETURN NULL;\n END IF;\n RETURN collections;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\nCREATE OR REPLACE FUNCTION collection_base_item(content jsonb) RETURNS jsonb AS $$\n SELECT jsonb_build_object(\n 'type', 'Feature',\n 'stac_version', content->'stac_version',\n 'assets', content->'item_assets',\n 'collection', content->'id'\n );\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TABLE IF NOT EXISTS collections (\n key bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,\n id text GENERATED ALWAYS AS (content->>'id') STORED UNIQUE,\n content JSONB NOT NULL,\n base_item jsonb GENERATED ALWAYS AS (pgstac.collection_base_item(content)) STORED,\n partition_trunc text CHECK (partition_trunc IN ('year', 'month'))\n);\n\n\n\nCREATE OR REPLACE FUNCTION collection_base_item(cid text) RETURNS jsonb AS $$\n SELECT pgstac.collection_base_item(content) FROM pgstac.collections WHERE id = cid LIMIT 1;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION create_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION update_collection(data jsonb) RETURNS VOID AS $$\nDECLARE\n out collections%ROWTYPE;\nBEGIN\n UPDATE collections SET content=data WHERE id = data->>'id' RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ON CONFLICT (id) DO\n UPDATE\n SET content=EXCLUDED.content\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION delete_collection(_id text) RETURNS VOID AS $$\nDECLARE\n out collections%ROWTYPE;\nBEGIN\n DELETE FROM collections WHERE id = _id RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION get_collection(id text) RETURNS jsonb AS $$\n SELECT content FROM collections\n WHERE id=$1\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION all_collections() RETURNS jsonb AS $$\n SELECT jsonb_agg(content) FROM collections;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\nCREATE TABLE items (\n id text NOT NULL,\n geometry geometry NOT NULL,\n collection text NOT NULL,\n datetime timestamptz NOT NULL,\n end_datetime timestamptz NOT NULL,\n content JSONB NOT NULL\n)\nPARTITION BY LIST (collection)\n;\n\nCREATE INDEX \"datetime_idx\" ON items USING BTREE (datetime DESC, end_datetime ASC);\nCREATE INDEX \"geometry_idx\" ON items USING GIST (geometry);\n\nCREATE STATISTICS datetime_stats (dependencies) on datetime, end_datetime from items;\n\nALTER TABLE items ADD CONSTRAINT items_collections_fk FOREIGN KEY (collection) REFERENCES collections(id) ON DELETE CASCADE DEFERRABLE;\n\nCREATE OR REPLACE FUNCTION partition_after_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p text;\n t timestamptz := clock_timestamp();\nBEGIN\n RAISE NOTICE 'Updating partition stats %', t;\n FOR p IN SELECT DISTINCT partition\n FROM newdata n JOIN partition_sys_meta p\n ON (n.collection=p.collection AND n.datetime <@ p.partition_dtrange)\n LOOP\n PERFORM run_or_queue(format('SELECT update_partition_stats(%L, %L);', p, true));\n END LOOP;\n IF TG_OP IN ('DELETE','UPDATE') THEN\n DELETE FROM format_item_cache c USING newdata n WHERE c.collection = n.collection AND c.id = n.id;\n END IF;\n RAISE NOTICE 't: % %', t, clock_timestamp() - t;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE TRIGGER items_after_insert_trigger\nAFTER INSERT ON items\nREFERENCING NEW TABLE AS newdata\nFOR EACH STATEMENT\nEXECUTE FUNCTION partition_after_triggerfunc();\n\nCREATE TRIGGER items_after_update_trigger\nAFTER DELETE ON items\nREFERENCING OLD TABLE AS newdata\nFOR EACH STATEMENT\nEXECUTE FUNCTION partition_after_triggerfunc();\n\nCREATE TRIGGER items_after_delete_trigger\nAFTER UPDATE ON items\nREFERENCING NEW TABLE AS newdata\nFOR EACH STATEMENT\nEXECUTE FUNCTION partition_after_triggerfunc();\n\n\nCREATE OR REPLACE FUNCTION content_slim(_item jsonb) RETURNS jsonb AS $$\n SELECT strip_jsonb(_item - '{id,geometry,collection,type}'::text[], collection_base_item(_item->>'collection')) - '{id,geometry,collection,type}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_dehydrate(content jsonb) RETURNS items AS $$\n SELECT\n content->>'id' as id,\n stac_geom(content) as geometry,\n content->>'collection' as collection,\n stac_datetime(content) as datetime,\n stac_end_datetime(content) as end_datetime,\n content_slim(content) as content\n ;\n$$ LANGUAGE SQL STABLE;\n\nCREATE OR REPLACE FUNCTION include_field(f text, fields jsonb DEFAULT '{}'::jsonb) RETURNS boolean AS $$\nDECLARE\n includes jsonb := fields->'include';\n excludes jsonb := fields->'exclude';\nBEGIN\n IF f IS NULL THEN\n RETURN NULL;\n END IF;\n\n\n IF\n jsonb_typeof(excludes) = 'array'\n AND jsonb_array_length(excludes)>0\n AND excludes ? f\n THEN\n RETURN FALSE;\n END IF;\n\n IF\n (\n jsonb_typeof(includes) = 'array'\n AND jsonb_array_length(includes) > 0\n AND includes ? f\n ) OR\n (\n includes IS NULL\n OR jsonb_typeof(includes) = 'null'\n OR jsonb_array_length(includes) = 0\n )\n THEN\n RETURN TRUE;\n END IF;\n\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nDROP FUNCTION IF EXISTS content_hydrate(jsonb, jsonb, jsonb);\nCREATE OR REPLACE FUNCTION content_hydrate(\n _item jsonb,\n _base_item jsonb,\n fields jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\n SELECT merge_jsonb(\n jsonb_fields(_item, fields),\n jsonb_fields(_base_item, fields)\n );\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION content_hydrate(_item items, _collection collections, fields jsonb DEFAULT '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\n content jsonb;\n base_item jsonb := _collection.base_item;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry, 20)::jsonb;\n END IF;\n output := content_hydrate(\n jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'collection', _item.collection,\n 'type', 'Feature'\n ) || _item.content,\n _collection.base_item,\n fields\n );\n\n RETURN output;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_nonhydrated(\n _item items,\n fields jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry, 20)::jsonb;\n END IF;\n output := jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'collection', _item.collection,\n 'type', 'Feature'\n ) || _item.content;\n RETURN output;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_hydrate(_item items, fields jsonb DEFAULT '{}'::jsonb) RETURNS jsonb AS $$\n SELECT content_hydrate(\n _item,\n (SELECT c FROM collections c WHERE id=_item.collection LIMIT 1),\n fields\n );\n$$ LANGUAGE SQL STABLE;\n\n\nCREATE UNLOGGED TABLE items_staging (\n content JSONB NOT NULL\n);\nCREATE UNLOGGED TABLE items_staging_ignore (\n content JSONB NOT NULL\n);\nCREATE UNLOGGED TABLE items_staging_upsert (\n content JSONB NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION items_staging_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p record;\n _partitions text[];\n part text;\n ts timestamptz := clock_timestamp();\nBEGIN\n RAISE NOTICE 'Creating Partitions. %', clock_timestamp() - ts;\n\n FOR part IN WITH t AS (\n SELECT\n n.content->>'collection' as collection,\n stac_daterange(n.content->'properties') as dtr,\n partition_trunc\n FROM newdata n JOIN collections ON (n.content->>'collection'=collections.id)\n ), p AS (\n SELECT\n collection,\n COALESCE(date_trunc(partition_trunc::text, lower(dtr)),'-infinity') as d,\n tstzrange(min(lower(dtr)),max(lower(dtr)),'[]') as dtrange,\n tstzrange(min(upper(dtr)),max(upper(dtr)),'[]') as edtrange\n FROM t\n GROUP BY 1,2\n ) SELECT check_partition(collection, dtrange, edtrange) FROM p LOOP\n RAISE NOTICE 'Partition %', part;\n END LOOP;\n\n RAISE NOTICE 'Doing the insert. %', clock_timestamp() - ts;\n IF TG_TABLE_NAME = 'items_staging' THEN\n INSERT INTO items\n SELECT\n (content_dehydrate(content)).*\n FROM newdata;\n RAISE NOTICE 'Doing the delete. %', clock_timestamp() - ts;\n DELETE FROM items_staging;\n ELSIF TG_TABLE_NAME = 'items_staging_ignore' THEN\n INSERT INTO items\n SELECT\n (content_dehydrate(content)).*\n FROM newdata\n ON CONFLICT DO NOTHING;\n RAISE NOTICE 'Doing the delete. %', clock_timestamp() - ts;\n DELETE FROM items_staging_ignore;\n ELSIF TG_TABLE_NAME = 'items_staging_upsert' THEN\n WITH staging_formatted AS (\n SELECT (content_dehydrate(content)).* FROM newdata\n ), deletes AS (\n DELETE FROM items i USING staging_formatted s\n WHERE\n i.id = s.id\n AND i.collection = s.collection\n AND i IS DISTINCT FROM s\n RETURNING i.id, i.collection\n )\n INSERT INTO items\n SELECT s.* FROM\n staging_formatted s\n ON CONFLICT DO NOTHING;\n RAISE NOTICE 'Doing the delete. %', clock_timestamp() - ts;\n DELETE FROM items_staging_upsert;\n END IF;\n RAISE NOTICE 'Done. %', clock_timestamp() - ts;\n\n RETURN NULL;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER items_staging_insert_trigger AFTER INSERT ON items_staging REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\nCREATE TRIGGER items_staging_insert_ignore_trigger AFTER INSERT ON items_staging_ignore REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\nCREATE TRIGGER items_staging_insert_upsert_trigger AFTER INSERT ON items_staging_upsert REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\n\nCREATE OR REPLACE FUNCTION item_by_id(_id text, _collection text DEFAULT NULL) RETURNS items AS\n$$\nDECLARE\n i items%ROWTYPE;\nBEGIN\n SELECT * INTO i FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection) LIMIT 1;\n RETURN i;\nEND;\n$$ LANGUAGE PLPGSQL STABLE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION get_item(_id text, _collection text DEFAULT NULL) RETURNS jsonb AS $$\n SELECT content_hydrate(items) FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection);\n$$ LANGUAGE SQL STABLE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION delete_item(_id text, _collection text DEFAULT NULL) RETURNS VOID AS $$\nDECLARE\nout items%ROWTYPE;\nBEGIN\n DELETE FROM items WHERE id = _id AND (_collection IS NULL OR collection=_collection) RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n--/*\nCREATE OR REPLACE FUNCTION create_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION update_item(content jsonb) RETURNS VOID AS $$\nDECLARE\n old items %ROWTYPE;\n out items%ROWTYPE;\nBEGIN\n PERFORM delete_item(content->>'id', content->>'collection');\n PERFORM create_item(content);\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION create_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION collection_bbox(id text) RETURNS jsonb AS $$\n SELECT (replace(replace(replace(st_extent(geometry)::text,'BOX(','[['),')',']]'),' ',','))::jsonb\n FROM items WHERE collection=$1;\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION collection_temporal_extent(id text) RETURNS jsonb AS $$\n SELECT to_jsonb(array[array[min(datetime)::text, max(datetime)::text]])\n FROM items WHERE collection=$1;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection_extents() RETURNS VOID AS $$\nUPDATE collections SET\n content = content ||\n jsonb_build_object(\n 'extent', jsonb_build_object(\n 'spatial', jsonb_build_object(\n 'bbox', collection_bbox(collections.id)\n ),\n 'temporal', jsonb_build_object(\n 'interval', collection_temporal_extent(collections.id)\n )\n )\n )\n;\n$$ LANGUAGE SQL;\nCREATE TABLE partition_stats (\n partition text PRIMARY KEY,\n dtrange tstzrange,\n edtrange tstzrange,\n spatial geometry,\n last_updated timestamptz,\n keys text[]\n) WITH (FILLFACTOR=90);\n\nCREATE INDEX partitions_range_idx ON partition_stats USING GIST(dtrange);\n\n\nCREATE OR REPLACE FUNCTION constraint_tstzrange(expr text) RETURNS tstzrange AS $$\n WITH t AS (\n SELECT regexp_matches(\n expr,\n E'\\\\(''\\([0-9 :+-]*\\)''\\\\).*\\\\(''\\([0-9 :+-]*\\)''\\\\)'\n ) AS m\n ) SELECT tstzrange(m[1]::timestamptz, m[2]::timestamptz) FROM t\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE STRICT;\n\nCREATE OR REPLACE FUNCTION dt_constraint(coid oid, OUT dt tstzrange, OUT edt tstzrange) RETURNS RECORD AS $$\nDECLARE\n expr text := pg_get_constraintdef(coid);\n matches timestamptz[];\nBEGIN\n IF expr LIKE '%NULL%' THEN\n dt := tstzrange(null::timestamptz, null::timestamptz);\n edt := tstzrange(null::timestamptz, null::timestamptz);\n RETURN;\n END IF;\n WITH f AS (SELECT (regexp_matches(expr, E'([0-9]{4}-[0-1][0-9]-[0-3][0-9] [0-2][0-9]:[0-5][0-9]:[0-5][0-9]\\.?[0-9]*)', 'g'))[1] f)\n SELECT array_agg(f::timestamptz) INTO matches FROM f;\n IF cardinality(matches) = 4 THEN\n dt := tstzrange(matches[1], matches[2],'[]');\n edt := tstzrange(matches[3], matches[4], '[]');\n RETURN;\n ELSIF cardinality(matches) = 2 THEN\n edt := tstzrange(matches[1], matches[2],'[]');\n RETURN;\n END IF;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\nCREATE OR REPLACE VIEW partition_sys_meta AS\nSELECT\n relid::text as partition,\n replace(replace(CASE WHEN level = 1 THEN pg_get_expr(c.relpartbound, c.oid)\n ELSE pg_get_expr(parent.relpartbound, parent.oid)\n END, 'FOR VALUES IN (''',''), ''')','') AS collection,\n level,\n c.reltuples,\n c.relhastriggers,\n COALESCE(constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as partition_dtrange,\n COALESCE((dt_constraint(edt.oid)).dt, constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as constraint_dtrange,\n COALESCE((dt_constraint(edt.oid)).edt, tstzrange('-infinity', 'infinity','[]')) as constraint_edtrange\nFROM\n pg_partition_tree('items')\n JOIN pg_class c ON (relid::regclass = c.oid)\n JOIN pg_class parent ON (parentrelid::regclass = parent.oid AND isleaf)\n LEFT JOIN pg_constraint edt ON (conrelid=c.oid AND contype='c')\nWHERE isleaf\n;\n\nCREATE VIEW partitions_view AS\nSELECT\n relid::text as partition,\n replace(replace(CASE WHEN level = 1 THEN pg_get_expr(c.relpartbound, c.oid)\n ELSE pg_get_expr(parent.relpartbound, parent.oid)\n END, 'FOR VALUES IN (''',''), ''')','') AS collection,\n level,\n c.reltuples,\n c.relhastriggers,\n COALESCE(constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as partition_dtrange,\n COALESCE((dt_constraint(edt.oid)).dt, constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as constraint_dtrange,\n COALESCE((dt_constraint(edt.oid)).edt, tstzrange('-infinity', 'infinity','[]')) as constraint_edtrange,\n dtrange,\n edtrange,\n spatial,\n last_updated\nFROM\n pg_partition_tree('items')\n JOIN pg_class c ON (relid::regclass = c.oid)\n JOIN pg_class parent ON (parentrelid::regclass = parent.oid AND isleaf)\n LEFT JOIN pg_constraint edt ON (conrelid=c.oid AND contype='c')\n LEFT JOIN partition_stats ON (relid::text=partition)\nWHERE isleaf\n;\n\nCREATE MATERIALIZED VIEW partitions AS\nSELECT * FROM partitions_view;\nCREATE UNIQUE INDEX ON partitions (partition);\n\nCREATE MATERIALIZED VIEW partition_steps AS\nSELECT\n partition as name,\n date_trunc('month',lower(partition_dtrange)) as sdate,\n date_trunc('month', upper(partition_dtrange)) + '1 month'::interval as edate\n FROM partitions_view WHERE partition_dtrange IS NOT NULL AND partition_dtrange != 'empty'::tstzrange\n ORDER BY dtrange ASC\n;\n\n\nCREATE OR REPLACE FUNCTION update_partition_stats_q(_partition text, istrigger boolean default false) RETURNS VOID AS $$\nDECLARE\nBEGIN\n PERFORM run_or_queue(\n format('SELECT update_partition_stats(%L, %L);', _partition, istrigger)\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION update_partition_stats(_partition text, istrigger boolean default false) RETURNS VOID AS $$\nDECLARE\n dtrange tstzrange;\n edtrange tstzrange;\n cdtrange tstzrange;\n cedtrange tstzrange;\n extent geometry;\n collection text;\nBEGIN\n RAISE NOTICE 'Updating stats for %.', _partition;\n EXECUTE format(\n $q$\n SELECT\n tstzrange(min(datetime), max(datetime),'[]'),\n tstzrange(min(end_datetime), max(end_datetime), '[]')\n FROM %I\n $q$,\n _partition\n ) INTO dtrange, edtrange;\n extent := st_estimatedextent('pgstac', _partition, 'geometry');\n INSERT INTO partition_stats (partition, dtrange, edtrange, spatial, last_updated)\n SELECT _partition, dtrange, edtrange, extent, now()\n ON CONFLICT (partition) DO\n UPDATE SET\n dtrange=EXCLUDED.dtrange,\n edtrange=EXCLUDED.edtrange,\n spatial=EXCLUDED.spatial,\n last_updated=EXCLUDED.last_updated\n ;\n\n SELECT\n constraint_dtrange, constraint_edtrange, pv.collection\n INTO cdtrange, cedtrange, collection\n FROM partitions_view pv WHERE partition = _partition;\n REFRESH MATERIALIZED VIEW partitions;\n REFRESH MATERIALIZED VIEW partition_steps;\n\n\n RAISE NOTICE 'Checking if we need to modify constraints.';\n IF\n (cdtrange IS DISTINCT FROM dtrange OR edtrange IS DISTINCT FROM cedtrange)\n AND NOT istrigger\n THEN\n RAISE NOTICE 'Modifying Constraints';\n RAISE NOTICE 'Existing % %', cdtrange, cedtrange;\n RAISE NOTICE 'New % %', dtrange, edtrange;\n PERFORM drop_table_constraints(_partition);\n PERFORM create_table_constraints(_partition, dtrange, edtrange);\n END IF;\n RAISE NOTICE 'Checking if we need to update collection extents.';\n IF get_setting_bool('update_collection_extent') THEN\n RAISE NOTICE 'updating collection extent for %', collection;\n PERFORM run_or_queue(format($q$\n UPDATE collections\n SET content = jsonb_set_lax(\n content,\n '{extent}'::text[],\n collection_extent(%L, FALSE),\n true,\n 'use_json_null'\n ) WHERE id=%L\n ;\n $q$, collection, collection));\n ELSE\n RAISE NOTICE 'Not updating collection extent for %', collection;\n END IF;\n\nEND;\n$$ LANGUAGE PLPGSQL STRICT;\n\n\nCREATE OR REPLACE FUNCTION partition_name( IN collection text, IN dt timestamptz, OUT partition_name text, OUT partition_range tstzrange) AS $$\nDECLARE\n c RECORD;\n parent_name text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n parent_name := format('_items_%s', c.key);\n\n\n IF c.partition_trunc = 'year' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYY'));\n ELSIF c.partition_trunc = 'month' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYYMM'));\n ELSE\n partition_name := parent_name;\n partition_range := tstzrange('-infinity'::timestamptz, 'infinity'::timestamptz, '[]');\n END IF;\n IF partition_range IS NULL THEN\n partition_range := tstzrange(\n date_trunc(c.partition_trunc::text, dt),\n date_trunc(c.partition_trunc::text, dt) + concat('1 ', c.partition_trunc)::interval\n );\n END IF;\n RETURN;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION drop_table_constraints(t text) RETURNS text AS $$\nDECLARE\n q text;\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM partitions_view WHERE partition=t) THEN\n RETURN NULL;\n END IF;\n FOR q IN SELECT FORMAT(\n $q$\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n $q$,\n t,\n conname\n ) FROM pg_constraint\n WHERE conrelid=t::regclass::oid AND contype='c'\n LOOP\n EXECUTE q;\n END LOOP;\n RETURN t;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE OR REPLACE FUNCTION create_table_constraints(t text, _dtrange tstzrange, _edtrange tstzrange) RETURNS text AS $$\nDECLARE\n q text;\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM partitions_view WHERE partition=t) THEN\n RETURN NULL;\n END IF;\n RAISE NOTICE 'Creating Table Constraints for % % %', t, _dtrange, _edtrange;\n IF _dtrange = 'empty' AND _edtrange = 'empty' THEN\n q :=format(\n $q$\n DO $block$\n BEGIN\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n ALTER TABLE %I\n ADD CONSTRAINT %I\n CHECK (((datetime IS NULL) AND (end_datetime IS NULL))) NOT VALID\n ;\n ALTER TABLE %I\n VALIDATE CONSTRAINT %I\n ;\n\n\n\n EXCEPTION WHEN others THEN\n RAISE WARNING '%%, Issue Altering Constraints. Please run update_partition_stats(%I)', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n $block$;\n $q$,\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t),\n t\n );\n ELSE\n q :=format(\n $q$\n DO $block$\n BEGIN\n\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n ALTER TABLE %I\n ADD CONSTRAINT %I\n CHECK (\n (datetime >= %L)\n AND (datetime <= %L)\n AND (end_datetime >= %L)\n AND (end_datetime <= %L)\n ) NOT VALID\n ;\n ALTER TABLE %I\n VALIDATE CONSTRAINT %I\n ;\n\n\n\n EXCEPTION WHEN others THEN\n RAISE WARNING '%%, Issue Altering Constraints. Please run update_partition_stats(%I)', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n $block$;\n $q$,\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t),\n lower(_dtrange),\n upper(_dtrange),\n lower(_edtrange),\n upper(_edtrange),\n t,\n format('%s_dt', t),\n t\n );\n END IF;\n PERFORM run_or_queue(q);\n RETURN t;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION check_partition(\n _collection text,\n _dtrange tstzrange,\n _edtrange tstzrange\n) RETURNS text AS $$\nDECLARE\n c RECORD;\n pm RECORD;\n _partition_name text;\n _partition_dtrange tstzrange;\n _constraint_dtrange tstzrange;\n _constraint_edtrange tstzrange;\n q text;\n deferrable_q text;\n err_context text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=_collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', _collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n\n IF c.partition_trunc IS NOT NULL THEN\n _partition_dtrange := tstzrange(\n date_trunc(c.partition_trunc, lower(_dtrange)),\n date_trunc(c.partition_trunc, lower(_dtrange)) + (concat('1 ', c.partition_trunc))::interval,\n '[)'\n );\n ELSE\n _partition_dtrange := '[-infinity, infinity]'::tstzrange;\n END IF;\n\n IF NOT _partition_dtrange @> _dtrange THEN\n RAISE EXCEPTION 'dtrange % is greater than the partition size % for collection %', _dtrange, c.partition_trunc, _collection;\n END IF;\n\n\n IF c.partition_trunc = 'year' THEN\n _partition_name := format('_items_%s_%s', c.key, to_char(lower(_partition_dtrange),'YYYY'));\n ELSIF c.partition_trunc = 'month' THEN\n _partition_name := format('_items_%s_%s', c.key, to_char(lower(_partition_dtrange),'YYYYMM'));\n ELSE\n _partition_name := format('_items_%s', c.key);\n END IF;\n\n SELECT * INTO pm FROM partition_sys_meta WHERE collection=_collection AND partition_dtrange @> _dtrange;\n IF FOUND THEN\n RAISE NOTICE '% % %', _edtrange, _dtrange, pm;\n _constraint_edtrange :=\n tstzrange(\n least(\n lower(_edtrange),\n nullif(lower(pm.constraint_edtrange), '-infinity')\n ),\n greatest(\n upper(_edtrange),\n nullif(upper(pm.constraint_edtrange), 'infinity')\n ),\n '[]'\n );\n _constraint_dtrange :=\n tstzrange(\n least(\n lower(_dtrange),\n nullif(lower(pm.constraint_dtrange), '-infinity')\n ),\n greatest(\n upper(_dtrange),\n nullif(upper(pm.constraint_dtrange), 'infinity')\n ),\n '[]'\n );\n\n IF pm.constraint_edtrange @> _edtrange AND pm.constraint_dtrange @> _dtrange THEN\n RETURN pm.partition;\n ELSE\n PERFORM drop_table_constraints(_partition_name);\n END IF;\n ELSE\n _constraint_edtrange := _edtrange;\n _constraint_dtrange := _dtrange;\n END IF;\n RAISE NOTICE 'Creating partition % %', _partition_name, _partition_dtrange;\n IF c.partition_trunc IS NULL THEN\n q := format(\n $q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n $q$,\n _partition_name,\n _collection,\n concat(_partition_name,'_pk'),\n _partition_name\n );\n ELSE\n q := format(\n $q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L) PARTITION BY RANGE (datetime);\n CREATE TABLE IF NOT EXISTS %I partition OF %I FOR VALUES FROM (%L) TO (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n $q$,\n format('_items_%s', c.key),\n _collection,\n _partition_name,\n format('_items_%s', c.key),\n lower(_partition_dtrange),\n upper(_partition_dtrange),\n format('%s_pk', _partition_name),\n _partition_name\n );\n END IF;\n\n BEGIN\n EXECUTE q;\n EXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', _partition_name;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\n END;\n PERFORM create_table_constraints(_partition_name, _constraint_dtrange, _constraint_edtrange);\n PERFORM maintain_partitions(_partition_name);\n PERFORM update_partition_stats_q(_partition_name, true);\n REFRESH MATERIALIZED VIEW partitions;\n REFRESH MATERIALIZED VIEW partition_steps;\n RETURN _partition_name;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION repartition(_collection text, _partition_trunc text, triggered boolean DEFAULT FALSE) RETURNS text AS $$\nDECLARE\n c RECORD;\n q text;\n from_trunc text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=_collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', _collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n IF triggered THEN\n RAISE NOTICE 'Converting % to % partitioning via Trigger', _collection, _partition_trunc;\n ELSE\n RAISE NOTICE 'Converting % from using % to % partitioning', _collection, c.partition_trunc, _partition_trunc;\n IF c.partition_trunc IS NOT DISTINCT FROM _partition_trunc THEN\n RAISE NOTICE 'Collection % already set to use partition by %', _collection, _partition_trunc;\n RETURN _collection;\n END IF;\n END IF;\n\n IF EXISTS (SELECT 1 FROM partitions_view WHERE collection=_collection LIMIT 1) THEN\n EXECUTE format(\n $q$\n CREATE TEMP TABLE changepartitionstaging ON COMMIT DROP AS SELECT * FROM %I;\n DROP TABLE IF EXISTS %I CASCADE;\n WITH p AS (\n SELECT\n collection,\n CASE WHEN %L IS NULL THEN '-infinity'::timestamptz\n ELSE date_trunc(%L, datetime)\n END as d,\n tstzrange(min(datetime),max(datetime),'[]') as dtrange,\n tstzrange(min(datetime),max(datetime),'[]') as edtrange\n FROM changepartitionstaging\n GROUP BY 1,2\n ) SELECT check_partition(collection, dtrange, edtrange) FROM p;\n INSERT INTO items SELECT * FROM changepartitionstaging;\n DROP TABLE changepartitionstaging;\n $q$,\n concat('_items_', c.key),\n concat('_items_', c.key),\n c.partition_trunc,\n c.partition_trunc\n );\n END IF;\n RETURN _collection;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE OR REPLACE FUNCTION collections_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n q text;\n partition_name text := format('_items_%s', NEW.key);\n partition_exists boolean := false;\n partition_empty boolean := true;\n err_context text;\n loadtemp boolean := FALSE;\nBEGIN\n RAISE NOTICE 'Collection Trigger. % %', NEW.id, NEW.key;\n IF TG_OP = 'UPDATE' AND NEW.partition_trunc IS DISTINCT FROM OLD.partition_trunc THEN\n PERFORM repartition(NEW.id, NEW.partition_trunc, TRUE);\n END IF;\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER collections_trigger AFTER\nINSERT\nOR\nUPDATE ON collections\nFOR EACH ROW EXECUTE FUNCTION collections_trigger_func();\n\nCREATE OR REPLACE FUNCTION chunker(\n IN _where text,\n OUT s timestamptz,\n OUT e timestamptz\n) RETURNS SETOF RECORD AS $$\nDECLARE\n explain jsonb;\nBEGIN\n IF _where IS NULL THEN\n _where := ' TRUE ';\n END IF;\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s;', _where)\n INTO explain;\n\n RETURN QUERY\n WITH t AS (\n SELECT j->>0 as p FROM\n jsonb_path_query(\n explain,\n 'strict $.**.\"Relation Name\" ? (@ != null)'\n ) j\n ),\n parts AS (\n SELECT sdate, edate FROM t JOIN partition_steps ON (t.p = name)\n ),\n times AS (\n SELECT sdate FROM parts\n UNION\n SELECT edate FROM parts\n ),\n uniq AS (\n SELECT DISTINCT sdate FROM times ORDER BY sdate\n ),\n last AS (\n SELECT sdate, lead(sdate, 1) over () as edate FROM uniq\n )\n SELECT sdate, edate FROM last WHERE edate IS NOT NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION partition_queries(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN partitions text[] DEFAULT NULL\n) RETURNS SETOF text AS $$\nDECLARE\n query text;\n sdate timestamptz;\n edate timestamptz;\nBEGIN\nIF _where IS NULL OR trim(_where) = '' THEN\n _where = ' TRUE ';\nEND IF;\nRAISE NOTICE 'Getting chunks for % %', _where, _orderby;\nIF _orderby ILIKE 'datetime d%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 DESC LOOP\n RETURN NEXT format($q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n $q$,\n sdate,\n edate,\n _where,\n _orderby\n );\n END LOOP;\nELSIF _orderby ILIKE 'datetime a%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 ASC LOOP\n RETURN NEXT format($q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n $q$,\n sdate,\n edate,\n _where,\n _orderby\n );\n END LOOP;\nELSE\n query := format($q$\n SELECT * FROM items\n WHERE %s\n ORDER BY %s\n $q$, _where, _orderby\n );\n\n RETURN NEXT query;\n RETURN;\nEND IF;\n\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION partition_query_view(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN _limit int DEFAULT 10\n) RETURNS text AS $$\n WITH p AS (\n SELECT * FROM partition_queries(_where, _orderby) p\n )\n SELECT\n CASE WHEN EXISTS (SELECT 1 FROM p) THEN\n (SELECT format($q$\n SELECT * FROM (\n %s\n ) total LIMIT %s\n $q$,\n string_agg(\n format($q$ SELECT * FROM ( %s ) AS sub $q$, p),\n '\n UNION ALL\n '\n ),\n _limit\n ))\n ELSE NULL\n END FROM p;\n$$ LANGUAGE SQL IMMUTABLE;\n\n\n\n\nCREATE OR REPLACE FUNCTION stac_search_to_where(j jsonb) RETURNS text AS $$\nDECLARE\n where_segments text[];\n _where text;\n dtrange tstzrange;\n collections text[];\n geom geometry;\n sdate timestamptz;\n edate timestamptz;\n filterlang text;\n filter jsonb := j->'filter';\nBEGIN\n IF j ? 'ids' THEN\n where_segments := where_segments || format('id = ANY (%L) ', to_text_array(j->'ids'));\n END IF;\n\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n where_segments := where_segments || format('collection = ANY (%L) ', collections);\n END IF;\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n\n where_segments := where_segments || format(' datetime <= %L::timestamptz AND end_datetime >= %L::timestamptz ',\n edate,\n sdate\n );\n END IF;\n\n geom := stac_geom(j);\n IF geom IS NOT NULL THEN\n where_segments := where_segments || format('st_intersects(geometry, %L)',geom);\n END IF;\n\n filterlang := COALESCE(\n j->>'filter-lang',\n get_setting('default_filter_lang', j->'conf')\n );\n IF NOT filter @? '$.**.op' THEN\n filterlang := 'cql-json';\n END IF;\n\n IF filterlang NOT IN ('cql-json','cql2-json') AND j ? 'filter' THEN\n RAISE EXCEPTION '% is not a supported filter-lang. Please use cql-json or cql2-json.', filterlang;\n END IF;\n\n IF j ? 'query' AND j ? 'filter' THEN\n RAISE EXCEPTION 'Can only use either query or filter at one time.';\n END IF;\n\n IF j ? 'query' THEN\n filter := query_to_cql2(j->'query');\n ELSIF filterlang = 'cql-json' THEN\n filter := cql1_to_cql2(filter);\n END IF;\n RAISE NOTICE 'FILTER: %', filter;\n where_segments := where_segments || cql2_query(filter);\n IF cardinality(where_segments) < 1 THEN\n RETURN ' TRUE ';\n END IF;\n\n _where := array_to_string(array_remove(where_segments, NULL), ' AND ');\n\n IF _where IS NULL OR BTRIM(_where) = '' THEN\n RETURN ' TRUE ';\n END IF;\n RETURN _where;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION parse_sort_dir(_dir text, reverse boolean default false) RETURNS text AS $$\n WITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n ) SELECT\n CASE\n WHEN NOT reverse THEN d\n WHEN d = 'ASC' THEN 'DESC'\n WHEN d = 'DESC' THEN 'ASC'\n END\n FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION sort_dir_to_op(_dir text, prev boolean default false) RETURNS text AS $$\n WITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n ) SELECT\n CASE\n WHEN d = 'ASC' AND prev THEN '<='\n WHEN d = 'DESC' AND prev THEN '>='\n WHEN d = 'ASC' THEN '>='\n WHEN d = 'DESC' THEN '<='\n END\n FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION sort_sqlorderby(\n _search jsonb DEFAULT NULL,\n reverse boolean DEFAULT FALSE\n) RETURNS text AS $$\n WITH sortby AS (\n SELECT coalesce(_search->'sortby','[{\"field\":\"datetime\", \"direction\":\"desc\"}]') as sort\n ), withid AS (\n SELECT CASE\n WHEN sort @? '$[*] ? (@.field == \"id\")' THEN sort\n ELSE sort || '[{\"field\":\"id\", \"direction\":\"desc\"}]'::jsonb\n END as sort\n FROM sortby\n ), withid_rows AS (\n SELECT jsonb_array_elements(sort) as value FROM withid\n ),sorts AS (\n SELECT\n coalesce(\n (queryable(value->>'field')).expression\n ) as key,\n parse_sort_dir(value->>'direction', reverse) as dir\n FROM withid_rows\n )\n SELECT array_to_string(\n array_agg(concat(key, ' ', dir)),\n ', '\n ) FROM sorts;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_sort_dir(sort_item jsonb) RETURNS text AS $$\n SELECT CASE WHEN sort_item->>'direction' ILIKE 'desc%' THEN 'DESC' ELSE 'ASC' END;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION get_token_val_str(\n _field text,\n _item items\n) RETURNS text AS $$\nDECLARE\n q text;\n literal text;\nBEGIN\n q := format($q$ SELECT quote_literal(%s) FROM (SELECT $1.*) as r;$q$, _field);\n EXECUTE q INTO literal USING _item;\n RETURN literal;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n\nCREATE OR REPLACE FUNCTION get_token_record(IN _token text, OUT prev BOOLEAN, OUT item items) RETURNS RECORD AS $$\nDECLARE\n _itemid text := _token;\n _collectionid text;\nBEGIN\n IF _token IS NULL THEN\n RETURN;\n END IF;\n RAISE NOTICE 'Looking for token: %', _token;\n prev := FALSE;\n IF _token ILIKE 'prev:%' THEN\n _itemid := replace(_token, 'prev:','');\n prev := TRUE;\n ELSIF _token ILIKE 'next:%' THEN\n _itemid := replace(_token, 'next:', '');\n END IF;\n SELECT id INTO _collectionid FROM collections WHERE _itemid LIKE concat(id,':%');\n IF FOUND THEN\n _itemid := replace(_itemid, concat(_collectionid,':'), '');\n SELECT * INTO item FROM items WHERE id=_itemid AND collection=_collectionid;\n ELSE\n SELECT * INTO item FROM items WHERE id=_itemid;\n END IF;\n IF item IS NULL THEN\n RAISE EXCEPTION 'Could not find item using token: % item: % collection: %', _token, _itemid, _collectionid;\n END IF;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\n\nCREATE OR REPLACE FUNCTION get_token_filter(\n _sortby jsonb DEFAULT '[{\"field\":\"datetime\",\"direction\":\"desc\"}]'::jsonb,\n token_item items DEFAULT NULL,\n prev boolean DEFAULT FALSE,\n inclusive boolean DEFAULT FALSE\n) RETURNS text AS $$\nDECLARE\n ltop text := '<';\n gtop text := '>';\n dir text;\n sort record;\n orfilter text := '';\n orfilters text[] := '{}'::text[];\n andfilters text[] := '{}'::text[];\n output text;\n token_where text;\nBEGIN\n IF _sortby IS NULL OR _sortby = '[]'::jsonb THEN\n _sortby := '[{\"field\":\"datetime\",\"direction\":\"desc\"}]'::jsonb;\n END IF;\n _sortby := _sortby || jsonb_build_object('field','id','direction',_sortby->0->>'direction');\n RAISE NOTICE 'Getting Token Filter. % %', _sortby, token_item;\n IF inclusive THEN\n orfilters := orfilters || format('( id=%L AND collection=%L )' , token_item.id, token_item.collection);\n END IF;\n\n FOR sort IN\n WITH s1 AS (\n SELECT\n _row,\n (queryable(value->>'field')).expression as _field,\n (value->>'field' = 'id') as _isid,\n get_sort_dir(value) as _dir\n FROM jsonb_array_elements(_sortby)\n WITH ORDINALITY AS t(value, _row)\n )\n SELECT\n _row,\n _field,\n _dir,\n get_token_val_str(_field, token_item) as _val\n FROM s1\n WHERE _row <= (SELECT min(_row) FROM s1 WHERE _isid)\n LOOP\n orfilter := NULL;\n RAISE NOTICE 'SORT: %', sort;\n IF sort._val IS NOT NULL AND ((prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC')) THEN\n orfilter := format($f$(\n (%s %s %s) OR (%s IS NULL)\n )$f$,\n sort._field,\n ltop,\n sort._val,\n sort._val\n );\n ELSIF sort._val IS NULL AND ((prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC')) THEN\n RAISE NOTICE '< but null';\n orfilter := format('%s IS NOT NULL', sort._field);\n ELSIF sort._val IS NULL THEN\n RAISE NOTICE '> but null';\n ELSE\n orfilter := format($f$(\n (%s %s %s) OR (%s IS NULL)\n )$f$,\n sort._field,\n gtop,\n sort._val,\n sort._field\n );\n END IF;\n RAISE NOTICE 'ORFILTER: %', orfilter;\n\n IF orfilter IS NOT NULL THEN\n IF sort._row = 1 THEN\n orfilters := orfilters || orfilter;\n ELSE\n orfilters := orfilters || format('(%s AND %s)', array_to_string(andfilters, ' AND '), orfilter);\n END IF;\n END IF;\n IF sort._val IS NOT NULL THEN\n andfilters := andfilters || format('%s = %s', sort._field, sort._val);\n ELSE\n andfilters := andfilters || format('%s IS NULL', sort._field);\n END IF;\n END LOOP;\n\n output := array_to_string(orfilters, ' OR ');\n\n token_where := concat('(',coalesce(output,'true'),')');\n IF trim(token_where) = '' THEN\n token_where := NULL;\n END IF;\n RAISE NOTICE 'TOKEN_WHERE: %',token_where;\n RETURN token_where;\n END;\n$$ LANGUAGE PLPGSQL SET transform_null_equals TO TRUE\n;\n\nCREATE OR REPLACE FUNCTION search_tohash(jsonb) RETURNS jsonb AS $$\n SELECT $1 - '{token,limit,context,includes,excludes}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION search_hash(jsonb, jsonb) RETURNS text AS $$\n SELECT md5(concat(search_tohash($1)::text,$2::text));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TABLE IF NOT EXISTS searches(\n hash text GENERATED ALWAYS AS (search_hash(search, metadata)) STORED PRIMARY KEY,\n search jsonb NOT NULL,\n _where text,\n orderby text,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n metadata jsonb DEFAULT '{}'::jsonb NOT NULL\n);\n\nCREATE TABLE IF NOT EXISTS search_wheres(\n id bigint generated always as identity primary key,\n _where text NOT NULL,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n statslastupdated timestamptz,\n estimated_count bigint,\n estimated_cost float,\n time_to_estimate float,\n total_count bigint,\n time_to_count float,\n partitions text[]\n);\n\nCREATE INDEX IF NOT EXISTS search_wheres_partitions ON search_wheres USING GIN (partitions);\nCREATE UNIQUE INDEX IF NOT EXISTS search_wheres_where ON search_wheres ((md5(_where)));\n\nCREATE OR REPLACE FUNCTION where_stats(inwhere text, updatestats boolean default false, conf jsonb default null) RETURNS search_wheres AS $$\nDECLARE\n t timestamptz;\n i interval;\n explain_json jsonb;\n partitions text[];\n sw search_wheres%ROWTYPE;\n inwhere_hash text := md5(inwhere);\n _context text := lower(context(conf));\n _stats_ttl interval := context_stats_ttl(conf);\n _estimated_cost float := context_estimated_cost(conf);\n _estimated_count int := context_estimated_count(conf);\nBEGIN\n IF _context = 'off' THEN\n sw._where := inwhere;\n return sw;\n END IF;\n\n SELECT * INTO sw FROM search_wheres WHERE md5(_where)=inwhere_hash FOR UPDATE;\n\n -- Update statistics if explicitly set, if statistics do not exist, or statistics ttl has expired\n IF NOT updatestats THEN\n RAISE NOTICE 'Checking if update is needed for: % .', inwhere;\n RAISE NOTICE 'Stats Last Updated: %', sw.statslastupdated;\n RAISE NOTICE 'TTL: %, Age: %', _stats_ttl, now() - sw.statslastupdated;\n RAISE NOTICE 'Context: %, Existing Total: %', _context, sw.total_count;\n IF\n sw.statslastupdated IS NULL\n OR (now() - sw.statslastupdated) > _stats_ttl\n OR (context(conf) != 'off' AND sw.total_count IS NULL)\n THEN\n updatestats := TRUE;\n END IF;\n END IF;\n\n sw._where := inwhere;\n sw.lastused := now();\n sw.usecount := coalesce(sw.usecount,0) + 1;\n\n IF NOT updatestats THEN\n UPDATE search_wheres SET\n lastused = sw.lastused,\n usecount = sw.usecount\n WHERE md5(_where) = inwhere_hash\n RETURNING * INTO sw\n ;\n RETURN sw;\n END IF;\n\n -- Use explain to get estimated count/cost and a list of the partitions that would be hit by the query\n t := clock_timestamp();\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s', inwhere)\n INTO explain_json;\n RAISE NOTICE 'Time for just the explain: %', clock_timestamp() - t;\n i := clock_timestamp() - t;\n\n sw.statslastupdated := now();\n sw.estimated_count := explain_json->0->'Plan'->'Plan Rows';\n sw.estimated_cost := explain_json->0->'Plan'->'Total Cost';\n sw.time_to_estimate := extract(epoch from i);\n\n RAISE NOTICE 'ESTIMATED_COUNT: % < %', sw.estimated_count, _estimated_count;\n RAISE NOTICE 'ESTIMATED_COST: % < %', sw.estimated_cost, _estimated_cost;\n\n -- Do a full count of rows if context is set to on or if auto is set and estimates are low enough\n IF\n _context = 'on'\n OR\n ( _context = 'auto' AND\n (\n sw.estimated_count < _estimated_count\n AND\n sw.estimated_cost < _estimated_cost\n )\n )\n THEN\n t := clock_timestamp();\n RAISE NOTICE 'Calculating actual count...';\n EXECUTE format(\n 'SELECT count(*) FROM items WHERE %s',\n inwhere\n ) INTO sw.total_count;\n i := clock_timestamp() - t;\n RAISE NOTICE 'Actual Count: % -- %', sw.total_count, i;\n sw.time_to_count := extract(epoch FROM i);\n ELSE\n sw.total_count := NULL;\n sw.time_to_count := NULL;\n END IF;\n\n\n INSERT INTO search_wheres\n (_where, lastused, usecount, statslastupdated, estimated_count, estimated_cost, time_to_estimate, partitions, total_count, time_to_count)\n SELECT sw._where, sw.lastused, sw.usecount, sw.statslastupdated, sw.estimated_count, sw.estimated_cost, sw.time_to_estimate, sw.partitions, sw.total_count, sw.time_to_count\n ON CONFLICT ((md5(_where)))\n DO UPDATE\n SET\n lastused = sw.lastused,\n usecount = sw.usecount,\n statslastupdated = sw.statslastupdated,\n estimated_count = sw.estimated_count,\n estimated_cost = sw.estimated_cost,\n time_to_estimate = sw.time_to_estimate,\n total_count = sw.total_count,\n time_to_count = sw.time_to_count\n ;\n RETURN sw;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION search_query(\n _search jsonb = '{}'::jsonb,\n updatestats boolean = false,\n _metadata jsonb = '{}'::jsonb\n) RETURNS searches AS $$\nDECLARE\n search searches%ROWTYPE;\n pexplain jsonb;\n t timestamptz;\n i interval;\n _hash text := search_hash(_search, _metadata);\n doupdate boolean := FALSE;\n insertfound boolean := FALSE;\nBEGIN\n SELECT * INTO search FROM searches\n WHERE hash=_hash;\n\n search.hash := _hash;\n\n -- Calculate the where clause if not already calculated\n IF search._where IS NULL THEN\n search._where := stac_search_to_where(_search);\n ELSE\n doupdate := TRUE;\n END IF;\n\n -- Calculate the order by clause if not already calculated\n IF search.orderby IS NULL THEN\n search.orderby := sort_sqlorderby(_search);\n ELSE\n doupdate := TRUE;\n END IF;\n\n PERFORM where_stats(search._where, updatestats, _search->'conf');\n\n IF NOT doupdate THEN\n INSERT INTO searches (search, _where, orderby, lastused, usecount, metadata)\n VALUES (_search, search._where, search.orderby, clock_timestamp(), 1, _metadata)\n ON CONFLICT (hash) DO NOTHING RETURNING * INTO search;\n IF FOUND THEN\n RETURN search;\n END IF;\n END IF;\n\n UPDATE searches\n SET\n lastused=clock_timestamp(),\n usecount=usecount+1\n WHERE hash=(\n SELECT hash FROM searches\n WHERE hash=_hash\n FOR UPDATE SKIP LOCKED\n );\n IF NOT FOUND THEN\n RAISE NOTICE 'Did not update stats for % due to lock. (This is generally OK)', _search;\n END IF;\n\n RETURN search;\n\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE OR REPLACE FUNCTION search_rows(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN partitions text[] DEFAULT NULL,\n IN _limit int DEFAULT 10\n) RETURNS SETOF items AS $$\nDECLARE\n base_query text;\n query text;\n sdate timestamptz;\n edate timestamptz;\n n int;\n records_left int := _limit;\n timer timestamptz := clock_timestamp();\n full_timer timestamptz := clock_timestamp();\nBEGIN\nIF _where IS NULL OR trim(_where) = '' THEN\n _where = ' TRUE ';\nEND IF;\nRAISE NOTICE 'Getting chunks for % %', _where, _orderby;\n\nbase_query := $q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n LIMIT %L\n$q$;\n\nIF _orderby ILIKE 'datetime d%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 DESC LOOP\n RAISE NOTICE 'Running Query for % to %. %', sdate, edate, age_ms(full_timer);\n query := format(\n base_query,\n sdate,\n edate,\n _where,\n _orderby,\n records_left\n );\n RAISE LOG 'QUERY: %', query;\n timer := clock_timestamp();\n RETURN QUERY EXECUTE query;\n\n GET DIAGNOSTICS n = ROW_COUNT;\n records_left := records_left - n;\n RAISE NOTICE 'Returned %/% Rows From % to %. % to go. Time: %ms', n, _limit, sdate, edate, records_left, age_ms(timer);\n timer := clock_timestamp();\n IF records_left <= 0 THEN\n RAISE NOTICE 'SEARCH_ROWS TOOK %ms', age_ms(full_timer);\n RETURN;\n END IF;\n END LOOP;\nELSIF _orderby ILIKE 'datetime a%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 ASC LOOP\n RAISE NOTICE 'Running Query for % to %. %', sdate, edate, age_ms(full_timer);\n query := format(\n base_query,\n sdate,\n edate,\n _where,\n _orderby,\n records_left\n );\n RAISE LOG 'QUERY: %', query;\n timer := clock_timestamp();\n RETURN QUERY EXECUTE query;\n\n GET DIAGNOSTICS n = ROW_COUNT;\n records_left := records_left - n;\n RAISE NOTICE 'Returned %/% Rows From % to %. % to go. Time: %ms', n, _limit, sdate, edate, records_left, age_ms(timer);\n timer := clock_timestamp();\n IF records_left <= 0 THEN\n RAISE NOTICE 'SEARCH_ROWS TOOK %ms', age_ms(full_timer);\n RETURN;\n END IF;\n END LOOP;\nELSE\n query := format($q$\n SELECT * FROM items\n WHERE %s\n ORDER BY %s\n LIMIT %L\n $q$, _where, _orderby, _limit\n );\n RAISE LOG 'QUERY: %', query;\n timer := clock_timestamp();\n RETURN QUERY EXECUTE query;\n RAISE NOTICE 'FULL QUERY TOOK %ms', age_ms(timer);\nEND IF;\nRAISE NOTICE 'SEARCH_ROWS TOOK %ms', age_ms(full_timer);\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE UNLOGGED TABLE format_item_cache(\n id text,\n collection text,\n fields text,\n hydrated bool,\n output jsonb,\n lastused timestamptz DEFAULT now(),\n usecount int DEFAULT 1,\n timetoformat float,\n PRIMARY KEY (collection, id, fields, hydrated)\n);\nCREATE INDEX ON format_item_cache (lastused);\n\nCREATE OR REPLACE FUNCTION format_item(_item items, _fields jsonb DEFAULT '{}', _hydrated bool DEFAULT TRUE) RETURNS jsonb AS $$\nDECLARE\n cache bool := get_setting_bool('format_cache');\n _output jsonb := null;\n t timestamptz := clock_timestamp();\nBEGIN\n IF cache THEN\n SELECT output INTO _output FROM format_item_cache\n WHERE id=_item.id AND collection=_item.collection AND fields=_fields::text AND hydrated=_hydrated;\n END IF;\n IF _output IS NULL THEN\n IF _hydrated THEN\n _output := content_hydrate(_item, _fields);\n ELSE\n _output := content_nonhydrated(_item, _fields);\n END IF;\n END IF;\n IF cache THEN\n INSERT INTO format_item_cache (id, collection, fields, hydrated, output, timetoformat)\n VALUES (_item.id, _item.collection, _fields::text, _hydrated, _output, age_ms(t))\n ON CONFLICT(collection, id, fields, hydrated) DO\n UPDATE\n SET lastused=now(), usecount = format_item_cache.usecount + 1\n ;\n END IF;\n RETURN _output;\n\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION search(_search jsonb = '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n searches searches%ROWTYPE;\n _where text;\n orderby text;\n search_where search_wheres%ROWTYPE;\n total_count bigint;\n token record;\n token_prev boolean;\n token_item items%ROWTYPE;\n token_where text;\n full_where text;\n init_ts timestamptz := clock_timestamp();\n timer timestamptz := clock_timestamp();\n hydrate bool := NOT (_search->'conf'->>'nohydrate' IS NOT NULL AND (_search->'conf'->>'nohydrate')::boolean = true);\n prev text;\n next text;\n context jsonb;\n collection jsonb;\n out_records jsonb;\n out_len int;\n _limit int := coalesce((_search->>'limit')::int, 10);\n init_limit int := _limit;\n _querylimit int;\n _fields jsonb := coalesce(_search->'fields', '{}'::jsonb);\n has_prev boolean := FALSE;\n has_next boolean := FALSE;\n items_cnt int := coalesce(jsonb_array_length(_search->'ids'),0);\nBEGIN\n RAISE NOTICE 'Items Count: %', items_cnt;\n IF items_cnt > 0 THEN\n IF items_cnt <= _limit THEN\n _limit := items_cnt - 1;\n ELSE\n _limit := items_cnt;\n END IF;\n RAISE NOTICE 'Items is set. Changing limit to %', items_cnt;\n END IF;\n searches := search_query(_search);\n _where := searches._where;\n orderby := searches.orderby;\n search_where := where_stats(_where);\n total_count := coalesce(search_where.total_count, search_where.estimated_count);\n RAISE NOTICE 'SEARCH:TOKEN: %', _search->>'token';\n token := get_token_record(_search->>'token');\n RAISE NOTICE '***TOKEN: %', token;\n _querylimit := _limit + 1;\n IF token IS NOT NULL THEN\n token_prev := token.prev;\n token_item := token.item;\n token_where := get_token_filter(_search->'sortby', token_item, token_prev, FALSE);\n RAISE LOG 'TOKEN_WHERE: % (%ms from search start)', token_where, age_ms(timer);\n IF token_prev THEN -- if we are using a prev token, we know has_next is true\n RAISE LOG 'There is a previous token, so automatically setting has_next to true';\n has_next := TRUE;\n orderby := sort_sqlorderby(_search, TRUE);\n ELSE\n RAISE LOG 'There is a next token, so automatically setting has_prev to true';\n has_prev := TRUE;\n\n END IF;\n ELSE -- if there was no token, we know there is no prev\n RAISE LOG 'There is no token, so we know there is no prev. setting has_prev to false';\n has_prev := FALSE;\n END IF;\n\n full_where := concat_ws(' AND ', _where, token_where);\n RAISE NOTICE 'FULL WHERE CLAUSE: %', full_where;\n RAISE NOTICE 'Time to get counts and build query %', age_ms(timer);\n timer := clock_timestamp();\n\n IF hydrate THEN\n RAISE NOTICE 'Getting hydrated data.';\n ELSE\n RAISE NOTICE 'Getting non-hydrated data.';\n END IF;\n RAISE NOTICE 'CACHE SET TO %', get_setting_bool('format_cache');\n RAISE NOTICE 'Time to set hydration/formatting %', age_ms(timer);\n timer := clock_timestamp();\n SELECT jsonb_agg(format_item(i, _fields, hydrate)) INTO out_records\n FROM search_rows(\n full_where,\n orderby,\n search_where.partitions,\n _querylimit\n ) as i;\n\n RAISE NOTICE 'Time to fetch rows %', age_ms(timer);\n timer := clock_timestamp();\n\n\n IF token_prev THEN\n out_records := flip_jsonb_array(out_records);\n END IF;\n\n RAISE NOTICE 'Query returned % records.', jsonb_array_length(out_records);\n RAISE LOG 'TOKEN: % %', token_item.id, token_item.collection;\n RAISE LOG 'RECORD_1: % %', out_records->0->>'id', out_records->0->>'collection';\n RAISE LOG 'RECORD-1: % %', out_records->-1->>'id', out_records->-1->>'collection';\n\n -- REMOVE records that were from our token\n IF out_records->0->>'id' = token_item.id AND out_records->0->>'collection' = token_item.collection THEN\n out_records := out_records - 0;\n ELSIF out_records->-1->>'id' = token_item.id AND out_records->-1->>'collection' = token_item.collection THEN\n out_records := out_records - -1;\n END IF;\n\n out_len := jsonb_array_length(out_records);\n\n IF out_len = init_limit + 1 THEN\n IF token_prev THEN\n has_prev := TRUE;\n out_records := out_records - 0;\n ELSE\n has_next := TRUE;\n out_records := out_records - -1;\n END IF;\n END IF;\n\n IF has_next THEN\n next := concat(out_records->-1->>'collection', ':', out_records->-1->>'id');\n RAISE NOTICE 'HAS NEXT | %', next;\n END IF;\n\n IF has_prev THEN\n prev := concat(out_records->0->>'collection', ':', out_records->0->>'id');\n RAISE NOTICE 'HAS PREV | %', prev;\n END IF;\n\n RAISE NOTICE 'Time to get prev/next %', age_ms(timer);\n timer := clock_timestamp();\n\n IF context(_search->'conf') != 'off' THEN\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', init_limit,\n 'matched', total_count,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\n ELSE\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', init_limit,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\n END IF;\n\n collection := jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb),\n 'next', next,\n 'prev', prev,\n 'context', context\n );\n\n IF get_setting_bool('timing', _search->'conf') THEN\n collection = collection || jsonb_build_object('timing', age_ms(init_ts));\n END IF;\n\n RAISE NOTICE 'Time to build final json %', age_ms(timer);\n timer := clock_timestamp();\n\n RAISE NOTICE 'Total Time: %', age_ms(current_timestamp);\n RAISE NOTICE 'RETURNING % records. NEXT: %. PREV: %', collection->'context'->>'returned', collection->>'next', collection->>'prev';\n RETURN collection;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION search_cursor(_search jsonb = '{}'::jsonb) RETURNS refcursor AS $$\nDECLARE\n curs refcursor;\n searches searches%ROWTYPE;\n _where text;\n _orderby text;\n q text;\n\nBEGIN\n searches := search_query(_search);\n _where := searches._where;\n _orderby := searches.orderby;\n\n OPEN curs FOR\n WITH p AS (\n SELECT * FROM partition_queries(_where, _orderby) p\n )\n SELECT\n CASE WHEN EXISTS (SELECT 1 FROM p) THEN\n (SELECT format($q$\n SELECT * FROM (\n %s\n ) total\n $q$,\n string_agg(\n format($q$ SELECT * FROM ( %s ) AS sub $q$, p),\n '\n UNION ALL\n '\n )\n ))\n ELSE NULL\n END FROM p;\n RETURN curs;\nEND;\n$$ LANGUAGE PLPGSQL;\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION tileenvelope(zoom int, x int, y int) RETURNS geometry AS $$\nWITH t AS (\n SELECT\n 20037508.3427892 as merc_max,\n -20037508.3427892 as merc_min,\n (2 * 20037508.3427892) / (2 ^ zoom) as tile_size\n)\nSELECT st_makeenvelope(\n merc_min + (tile_size * x),\n merc_max - (tile_size * (y + 1)),\n merc_min + (tile_size * (x + 1)),\n merc_max - (tile_size * y),\n 3857\n) FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;DROP FUNCTION IF EXISTS mercgrid;\n\n\nCREATE OR REPLACE FUNCTION ftime() RETURNS interval as $$\nSELECT age(clock_timestamp(), transaction_timestamp());\n$$ LANGUAGE SQL;\nSET SEARCH_PATH to pgstac, public;\n\nDROP FUNCTION IF EXISTS geometrysearch;\nCREATE OR REPLACE FUNCTION geometrysearch(\n IN geom geometry,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE, -- Return as soon as the passed in geometry is full covered\n IN skipcovered boolean DEFAULT TRUE -- Skip any items that would show up completely under the previous items\n) RETURNS jsonb AS $$\nDECLARE\n search searches%ROWTYPE;\n curs refcursor;\n _where text;\n query text;\n iter_record items%ROWTYPE;\n out_records jsonb := '{}'::jsonb[];\n exit_flag boolean := FALSE;\n counter int := 1;\n scancounter int := 1;\n remaining_limit int := _scanlimit;\n tilearea float;\n unionedgeom geometry;\n clippedgeom geometry;\n unionedgeom_area float := 0;\n prev_area float := 0;\n excludes text[];\n includes text[];\n\nBEGIN\n DROP TABLE IF EXISTS pgstac_results;\n CREATE TEMP TABLE pgstac_results (content jsonb) ON COMMIT DROP;\n\n -- If skipcovered is true then you will always want to exit when the passed in geometry is full\n IF skipcovered THEN\n exitwhenfull := TRUE;\n END IF;\n\n SELECT * INTO search FROM searches WHERE hash=queryhash;\n\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Search with Query Hash % Not Found', queryhash;\n END IF;\n\n tilearea := st_area(geom);\n _where := format('%s AND st_intersects(geometry, %L::geometry)', search._where, geom);\n\n\n FOR query IN SELECT * FROM partition_queries(_where, search.orderby) LOOP\n query := format('%s LIMIT %L', query, remaining_limit);\n RAISE NOTICE '%', query;\n OPEN curs FOR EXECUTE query;\n LOOP\n FETCH curs INTO iter_record;\n EXIT WHEN NOT FOUND;\n IF exitwhenfull OR skipcovered THEN -- If we are not using exitwhenfull or skipcovered, we do not need to do expensive geometry operations\n clippedgeom := st_intersection(geom, iter_record.geometry);\n\n IF unionedgeom IS NULL THEN\n unionedgeom := clippedgeom;\n ELSE\n unionedgeom := st_union(unionedgeom, clippedgeom);\n END IF;\n\n unionedgeom_area := st_area(unionedgeom);\n\n IF skipcovered AND prev_area = unionedgeom_area THEN\n scancounter := scancounter + 1;\n CONTINUE;\n END IF;\n\n prev_area := unionedgeom_area;\n\n RAISE NOTICE '% % % %', unionedgeom_area/tilearea, counter, scancounter, ftime();\n END IF;\n RAISE NOTICE '% %', iter_record, content_hydrate(iter_record, fields);\n INSERT INTO pgstac_results (content) VALUES (content_hydrate(iter_record, fields));\n\n IF counter >= _limit\n OR scancounter > _scanlimit\n OR ftime() > _timelimit\n OR (exitwhenfull AND unionedgeom_area >= tilearea)\n THEN\n exit_flag := TRUE;\n EXIT;\n END IF;\n counter := counter + 1;\n scancounter := scancounter + 1;\n\n END LOOP;\n CLOSE curs;\n EXIT WHEN exit_flag;\n remaining_limit := _scanlimit - scancounter;\n END LOOP;\n\n SELECT jsonb_agg(content) INTO out_records FROM pgstac_results WHERE content IS NOT NULL;\n\n RETURN jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb)\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDROP FUNCTION IF EXISTS geojsonsearch;\nCREATE OR REPLACE FUNCTION geojsonsearch(\n IN geojson jsonb,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_geomfromgeojson(geojson),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS xyzsearch;\nCREATE OR REPLACE FUNCTION xyzsearch(\n IN _x int,\n IN _y int,\n IN _z int,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_transform(tileenvelope(_z, _x, _y), 4326),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE PROCEDURE analyze_items() AS $$\nDECLARE\n q text;\n timeout_ts timestamptz;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n SELECT format('ANALYZE (VERBOSE, SKIP_LOCKED) %I;', relname) INTO q\n FROM pg_stat_user_tables\n WHERE relname like '_item%' AND (n_mod_since_analyze>0 OR last_analyze IS NULL) LIMIT 1;\n IF NOT FOUND THEN\n EXIT;\n END IF;\n RAISE NOTICE '%', q;\n EXECUTE q;\n COMMIT;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE PROCEDURE validate_constraints() AS $$\nDECLARE\n q text;\nBEGIN\n FOR q IN\n SELECT\n FORMAT(\n 'ALTER TABLE %I.%I VALIDATE CONSTRAINT %I;',\n nsp.nspname,\n cls.relname,\n con.conname\n )\n\n FROM pg_constraint AS con\n JOIN pg_class AS cls\n ON con.conrelid = cls.oid\n JOIN pg_namespace AS nsp\n ON cls.relnamespace = nsp.oid\n WHERE convalidated = FALSE AND contype in ('c','f')\n AND nsp.nspname = 'pgstac'\n LOOP\n RAISE NOTICE '%', q;\n PERFORM run_or_queue(q);\n COMMIT;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION collection_extent(_collection text, runupdate boolean default false) RETURNS jsonb AS $$\nDECLARE\n geom_extent geometry;\n mind timestamptz;\n maxd timestamptz;\n extent jsonb;\nBEGIN\n IF runupdate THEN\n PERFORM update_partition_stats_q(partition)\n FROM partitions_view WHERE collection=_collection;\n END IF;\n SELECT\n min(lower(dtrange)),\n max(upper(edtrange)),\n st_extent(spatial)\n INTO\n mind,\n maxd,\n geom_extent\n FROM partitions_view\n WHERE collection=_collection;\n\n IF geom_extent IS NOT NULL AND mind IS NOT NULL AND maxd IS NOT NULL THEN\n extent := jsonb_build_object(\n 'extent', jsonb_build_object(\n 'spatial', jsonb_build_object(\n 'bbox', to_jsonb(array[array[st_xmin(geom_extent), st_ymin(geom_extent), st_xmax(geom_extent), st_ymax(geom_extent)]])\n ),\n 'temporal', jsonb_build_object(\n 'interval', to_jsonb(array[array[mind, maxd]])\n )\n )\n );\n RETURN extent;\n END IF;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('id', '{\"title\": \"Item ID\",\"description\": \"Item identifier\",\"$ref\": \"https://schemas.stacspec.org/v1.0.0/item-spec/json-schema/item.json#/definitions/core/allOf/2/properties/id\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('geometry', '{\"title\": \"Item Geometry\",\"description\": \"Item Geometry\",\"$ref\": \"https://geojson.org/schema/Feature.json\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('datetime','{\"description\": \"Datetime\",\"type\": \"string\",\"title\": \"Acquired\",\"format\": \"date-time\",\"pattern\": \"(\\\\+00:00|Z)$\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('eo:cloud_cover','{\"$ref\": \"https://stac-extensions.github.io/eo/v1.0.0/schema.json#/definitions/fieldsproperties/eo:cloud_cover\"}','to_int','BTREE');\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDELETE FROM queryables a USING queryables b\n WHERE a.name = b.name AND a.collection_ids IS NOT DISTINCT FROM b.collection_ids AND a.id > b.id;\n\n\nINSERT INTO pgstac_settings (name, value) VALUES\n ('context', 'off'),\n ('context_estimated_count', '100000'),\n ('context_estimated_cost', '100000'),\n ('context_stats_ttl', '1 day'),\n ('default_filter_lang', 'cql2-json'),\n ('additional_properties', 'true'),\n ('use_queue', 'false'),\n ('queue_timeout', '10 minutes'),\n ('update_collection_extent', 'false'),\n ('format_cache', 'false')\nON CONFLICT DO NOTHING\n;\n\nALTER FUNCTION to_text COST 5000;\nALTER FUNCTION to_float COST 5000;\nALTER FUNCTION to_int COST 5000;\nALTER FUNCTION to_tstz COST 5000;\nALTER FUNCTION to_text_array COST 5000;\n\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nGRANT ALL ON SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac to pgstac_admin;\n\n-- pgstac_read role limited to using function apis\nGRANT EXECUTE ON FUNCTION search TO pgstac_read;\nGRANT EXECUTE ON FUNCTION search_query TO pgstac_read;\nGRANT EXECUTE ON FUNCTION item_by_id TO pgstac_read;\nGRANT EXECUTE ON FUNCTION get_item TO pgstac_read;\nGRANT SELECT ON ALL TABLES IN SCHEMA pgstac TO pgstac_read;\n\n\nGRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_ingest;\nGRANT USAGE ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_ingest;\n\nSELECT update_partition_stats_q(partition) FROM partitions_view;\nSELECT set_version('0.7.8');\n" }, { "path": "src/pgstac/migrations/pgstac.0.7.9-0.7.10.sql", "content": "RESET ROLE;\nDO $$\nDECLARE\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='postgis') THEN\n CREATE EXTENSION IF NOT EXISTS postgis;\n END IF;\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='btree_gist') THEN\n CREATE EXTENSION IF NOT EXISTS btree_gist;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_admin;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_read;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_ingest;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n\nGRANT pgstac_admin TO current_user;\n\n-- Function to make sure pgstac_admin is the owner of items\nCREATE OR REPLACE FUNCTION pgstac_admin_owns() RETURNS VOID AS $$\nDECLARE\n f RECORD;\nBEGIN\n FOR f IN (\n SELECT\n concat(\n oid::regproc::text,\n '(',\n coalesce(pg_get_function_identity_arguments(oid),''),\n ')'\n ) AS name,\n CASE prokind WHEN 'f' THEN 'FUNCTION' WHEN 'p' THEN 'PROCEDURE' WHEN 'a' THEN 'AGGREGATE' END as typ\n FROM pg_proc\n WHERE\n pronamespace=to_regnamespace('pgstac')\n AND proowner != to_regrole('pgstac_admin')\n AND proname NOT LIKE 'pg_stat%'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n FOR f IN (\n SELECT\n oid::regclass::text as name,\n CASE relkind\n WHEN 'i' THEN 'INDEX'\n WHEN 'I' THEN 'INDEX'\n WHEN 'p' THEN 'TABLE'\n WHEN 'r' THEN 'TABLE'\n WHEN 'v' THEN 'VIEW'\n WHEN 'S' THEN 'SEQUENCE'\n ELSE NULL\n END as typ\n FROM pg_class\n WHERE relnamespace=to_regnamespace('pgstac') and relowner != to_regrole('pgstac_admin') AND relkind IN ('r','p','v','S') AND relname NOT LIKE 'pg_stat'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\nSELECT pgstac_admin_owns();\n\nCREATE SCHEMA IF NOT EXISTS pgstac AUTHORIZATION pgstac_admin;\n\nGRANT ALL ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_admin;\n\nALTER ROLE pgstac_admin SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_read SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_ingest SET SEARCH_PATH TO pgstac, public;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\n\nGRANT pgstac_read TO pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\n\nSET ROLE pgstac_admin;\n\nSET SEARCH_PATH TO pgstac, public;\n\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS analyze_items;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS validate_constraints;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\nSET client_min_messages TO WARNING;\nSET SEARCH_PATH to pgstac, public;\n-- BEGIN migra calculated SQL\nset check_function_bodies = off;\n\nCREATE OR REPLACE FUNCTION pgstac.collection_delete_trigger_func()\n RETURNS trigger\n LANGUAGE plpgsql\nAS $function$\nDECLARE\n collection_base_partition text := concat('_items_', OLD.key);\nBEGIN\n EXECUTE format($q$\n DELETE FROM partition_stats WHERE partition IN (\n SELECT partition FROM partition_sys_meta\n WHERE collection=%L\n );\n DROP TABLE IF EXISTS %I CASCADE;\n $q$,\n OLD.id,\n collection_base_partition\n );\n RETURN OLD;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.all_collections()\n RETURNS jsonb\n LANGUAGE sql\n SET search_path TO 'pgstac', 'public'\nAS $function$\n SELECT coalesce(jsonb_agg(content), '[]'::jsonb) FROM collections;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.get_setting_bool(_setting text, conf jsonb DEFAULT NULL::jsonb)\n RETURNS boolean\n LANGUAGE sql\nAS $function$\nSELECT COALESCE(\n nullif(conf->>_setting, ''),\n nullif(current_setting(concat('pgstac.',_setting), TRUE),''),\n nullif((SELECT value FROM pgstac.pgstac_settings WHERE name=_setting),''),\n 'FALSE'\n)::boolean;\n$function$\n;\n\nCREATE TRIGGER collection_delete_trigger BEFORE DELETE ON pgstac.collections FOR EACH ROW EXECUTE FUNCTION pgstac.collection_delete_trigger_func();\n\n\n\n-- END migra calculated SQL\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('id', '{\"title\": \"Item ID\",\"description\": \"Item identifier\",\"$ref\": \"https://schemas.stacspec.org/v1.0.0/item-spec/json-schema/item.json#/definitions/core/allOf/2/properties/id\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('geometry', '{\"title\": \"Item Geometry\",\"description\": \"Item Geometry\",\"$ref\": \"https://geojson.org/schema/Feature.json\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('datetime','{\"description\": \"Datetime\",\"type\": \"string\",\"title\": \"Acquired\",\"format\": \"date-time\",\"pattern\": \"(\\\\+00:00|Z)$\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('eo:cloud_cover','{\"$ref\": \"https://stac-extensions.github.io/eo/v1.0.0/schema.json#/definitions/fieldsproperties/eo:cloud_cover\"}','to_int','BTREE');\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDELETE FROM queryables a USING queryables b\n WHERE a.name = b.name AND a.collection_ids IS NOT DISTINCT FROM b.collection_ids AND a.id > b.id;\n\n\nINSERT INTO pgstac_settings (name, value) VALUES\n ('context', 'off'),\n ('context_estimated_count', '100000'),\n ('context_estimated_cost', '100000'),\n ('context_stats_ttl', '1 day'),\n ('default_filter_lang', 'cql2-json'),\n ('additional_properties', 'true'),\n ('use_queue', 'false'),\n ('queue_timeout', '10 minutes'),\n ('update_collection_extent', 'false'),\n ('format_cache', 'false')\nON CONFLICT DO NOTHING\n;\n\nALTER FUNCTION to_text COST 5000;\nALTER FUNCTION to_float COST 5000;\nALTER FUNCTION to_int COST 5000;\nALTER FUNCTION to_tstz COST 5000;\nALTER FUNCTION to_text_array COST 5000;\n\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nGRANT ALL ON SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac to pgstac_admin;\n\n-- pgstac_read role limited to using function apis\nGRANT EXECUTE ON FUNCTION search TO pgstac_read;\nGRANT EXECUTE ON FUNCTION search_query TO pgstac_read;\nGRANT EXECUTE ON FUNCTION item_by_id TO pgstac_read;\nGRANT EXECUTE ON FUNCTION get_item TO pgstac_read;\nGRANT SELECT ON ALL TABLES IN SCHEMA pgstac TO pgstac_read;\n\n\nGRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_ingest;\nGRANT USAGE ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_ingest;\n\nSELECT update_partition_stats_q(partition) FROM partitions_view;\nSELECT set_version('0.7.10');\n" }, { "path": "src/pgstac/migrations/pgstac.0.7.9.sql", "content": "RESET ROLE;\nDO $$\nDECLARE\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='postgis') THEN\n CREATE EXTENSION IF NOT EXISTS postgis;\n END IF;\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='btree_gist') THEN\n CREATE EXTENSION IF NOT EXISTS btree_gist;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_admin;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_read;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_ingest;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n\nGRANT pgstac_admin TO current_user;\n\n-- Function to make sure pgstac_admin is the owner of items\nCREATE OR REPLACE FUNCTION pgstac_admin_owns() RETURNS VOID AS $$\nDECLARE\n f RECORD;\nBEGIN\n FOR f IN (\n SELECT\n concat(\n oid::regproc::text,\n '(',\n coalesce(pg_get_function_identity_arguments(oid),''),\n ')'\n ) AS name,\n CASE prokind WHEN 'f' THEN 'FUNCTION' WHEN 'p' THEN 'PROCEDURE' WHEN 'a' THEN 'AGGREGATE' END as typ\n FROM pg_proc\n WHERE\n pronamespace=to_regnamespace('pgstac')\n AND proowner != to_regrole('pgstac_admin')\n AND proname NOT LIKE 'pg_stat%'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n FOR f IN (\n SELECT\n oid::regclass::text as name,\n CASE relkind\n WHEN 'i' THEN 'INDEX'\n WHEN 'I' THEN 'INDEX'\n WHEN 'p' THEN 'TABLE'\n WHEN 'r' THEN 'TABLE'\n WHEN 'v' THEN 'VIEW'\n WHEN 'S' THEN 'SEQUENCE'\n ELSE NULL\n END as typ\n FROM pg_class\n WHERE relnamespace=to_regnamespace('pgstac') and relowner != to_regrole('pgstac_admin') AND relkind IN ('r','p','v','S') AND relname NOT LIKE 'pg_stat'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\nSELECT pgstac_admin_owns();\n\nCREATE SCHEMA IF NOT EXISTS pgstac AUTHORIZATION pgstac_admin;\n\nGRANT ALL ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_admin;\n\nALTER ROLE pgstac_admin SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_read SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_ingest SET SEARCH_PATH TO pgstac, public;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\n\nGRANT pgstac_read TO pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\n\nSET ROLE pgstac_admin;\n\nSET SEARCH_PATH TO pgstac, public;\n\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS analyze_items;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS validate_constraints;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\nCREATE OR REPLACE FUNCTION to_int(jsonb) RETURNS int AS $$\n SELECT floor(($1->>0)::float)::int;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION to_float(jsonb) RETURNS float AS $$\n SELECT ($1->>0)::float;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION to_tstz(jsonb) RETURNS timestamptz AS $$\n SELECT ($1->>0)::timestamptz;\n$$ LANGUAGE SQL IMMUTABLE STRICT SET TIME ZONE 'UTC' COST 5000 PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION to_text(jsonb) RETURNS text AS $$\n SELECT CASE WHEN jsonb_typeof($1) IN ('array','object') THEN $1::text ELSE $1->>0 END;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION to_text_array(jsonb) RETURNS text[] AS $$\n SELECT\n CASE jsonb_typeof($1)\n WHEN 'array' THEN ARRAY(SELECT jsonb_array_elements_text($1))\n ELSE ARRAY[$1->>0]\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION bbox_geom(_bbox jsonb) RETURNS geometry AS $$\nSELECT CASE jsonb_array_length(_bbox)\n WHEN 4 THEN\n ST_SetSRID(ST_MakeEnvelope(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float,\n (_bbox->>3)::float\n ),4326)\n WHEN 6 THEN\n ST_SetSRID(ST_3DMakeBox(\n ST_MakePoint(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float\n ),\n ST_MakePoint(\n (_bbox->>3)::float,\n (_bbox->>4)::float,\n (_bbox->>5)::float\n )\n ),4326)\n ELSE null END;\n;\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION geom_bbox(_geom geometry) RETURNS jsonb AS $$\n SELECT jsonb_build_array(\n st_xmin(_geom),\n st_ymin(_geom),\n st_xmax(_geom),\n st_ymax(_geom)\n );\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION flip_jsonb_array(j jsonb) RETURNS jsonb AS $$\n SELECT jsonb_agg(value) FROM (SELECT value FROM jsonb_array_elements(j) WITH ORDINALITY ORDER BY ordinality DESC) as t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION explode_dotpaths(j jsonb) RETURNS SETOF text[] AS $$\n SELECT string_to_array(p, '.') as e FROM jsonb_array_elements_text(j) p;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION explode_dotpaths_recurse(IN j jsonb) RETURNS SETOF text[] AS $$\n WITH RECURSIVE t AS (\n SELECT e FROM explode_dotpaths(j) e\n UNION ALL\n SELECT e[1:cardinality(e)-1]\n FROM t\n WHERE cardinality(e)>1\n ) SELECT e FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION jsonb_set_nested(j jsonb, path text[], val jsonb) RETURNS jsonb AS $$\nDECLARE\nBEGIN\n IF cardinality(path) > 1 THEN\n FOR i IN 1..(cardinality(path)-1) LOOP\n IF j #> path[:i] IS NULL THEN\n j := jsonb_set_lax(j, path[:i], '{}', TRUE);\n END IF;\n END LOOP;\n END IF;\n RETURN jsonb_set_lax(j, path, val, true);\n\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\n\n\nCREATE OR REPLACE FUNCTION jsonb_include(j jsonb, f jsonb) RETURNS jsonb AS $$\nDECLARE\n includes jsonb := f-> 'include';\n outj jsonb := '{}'::jsonb;\n path text[];\nBEGIN\n IF\n includes IS NULL\n OR jsonb_array_length(includes) = 0\n THEN\n RETURN j;\n ELSE\n includes := includes || '[\"id\",\"collection\"]'::jsonb;\n FOR path IN SELECT explode_dotpaths(includes) LOOP\n outj := jsonb_set_nested(outj, path, j #> path);\n END LOOP;\n END IF;\n RETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_exclude(j jsonb, f jsonb) RETURNS jsonb AS $$\nDECLARE\n excludes jsonb := f-> 'exclude';\n outj jsonb := j;\n path text[];\nBEGIN\n IF\n excludes IS NULL\n OR jsonb_array_length(excludes) = 0\n THEN\n RETURN j;\n ELSE\n FOR path IN SELECT explode_dotpaths(excludes) LOOP\n outj := outj #- path;\n END LOOP;\n END IF;\n RETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_fields(j jsonb, f jsonb DEFAULT '{\"fields\":[]}') RETURNS jsonb AS $$\n SELECT jsonb_exclude(jsonb_include(j, f), f);\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION merge_jsonb(_a jsonb, _b jsonb) RETURNS jsonb AS $$\n SELECT\n CASE\n WHEN _a = '\"𒍟※\"'::jsonb THEN NULL\n WHEN _a IS NULL OR jsonb_typeof(_a) = 'null' THEN _b\n WHEN jsonb_typeof(_a) = 'object' AND jsonb_typeof(_b) = 'object' THEN\n (\n SELECT\n jsonb_strip_nulls(\n jsonb_object_agg(\n key,\n merge_jsonb(a.value, b.value)\n )\n )\n FROM\n jsonb_each(coalesce(_a,'{}'::jsonb)) as a\n FULL JOIN\n jsonb_each(coalesce(_b,'{}'::jsonb)) as b\n USING (key)\n )\n WHEN\n jsonb_typeof(_a) = 'array'\n AND jsonb_typeof(_b) = 'array'\n AND jsonb_array_length(_a) = jsonb_array_length(_b)\n THEN\n (\n SELECT jsonb_agg(m) FROM\n ( SELECT\n merge_jsonb(\n jsonb_array_elements(_a),\n jsonb_array_elements(_b)\n ) as m\n ) as l\n )\n ELSE _a\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION strip_jsonb(_a jsonb, _b jsonb) RETURNS jsonb AS $$\n SELECT\n CASE\n\n WHEN (_a IS NULL OR jsonb_typeof(_a) = 'null') AND _b IS NOT NULL AND jsonb_typeof(_b) != 'null' THEN '\"𒍟※\"'::jsonb\n WHEN _b IS NULL OR jsonb_typeof(_a) = 'null' THEN _a\n WHEN _a = _b AND jsonb_typeof(_a) = 'object' THEN '{}'::jsonb\n WHEN _a = _b THEN NULL\n WHEN jsonb_typeof(_a) = 'object' AND jsonb_typeof(_b) = 'object' THEN\n (\n SELECT\n jsonb_strip_nulls(\n jsonb_object_agg(\n key,\n strip_jsonb(a.value, b.value)\n )\n )\n FROM\n jsonb_each(_a) as a\n FULL JOIN\n jsonb_each(_b) as b\n USING (key)\n )\n WHEN\n jsonb_typeof(_a) = 'array'\n AND jsonb_typeof(_b) = 'array'\n AND jsonb_array_length(_a) = jsonb_array_length(_b)\n THEN\n (\n SELECT jsonb_agg(m) FROM\n ( SELECT\n strip_jsonb(\n jsonb_array_elements(_a),\n jsonb_array_elements(_b)\n ) as m\n ) as l\n )\n ELSE _a\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION nullif_jsonbnullempty(j jsonb) RETURNS jsonb AS $$\n SELECT nullif(nullif(nullif(j,'null'::jsonb),'{}'::jsonb),'[]'::jsonb);\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION jsonb_array_unique(j jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(jsonb_agg(DISTINCT a)) v FROM jsonb_array_elements(j) a;\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_concat_ignorenull(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT coalesce(a,'[]'::jsonb) || coalesce(b,'[]'::jsonb);\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_least(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(least(nullif_jsonbnullempty(a), nullif_jsonbnullempty(b)));\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_greatest(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(greatest(a, b));\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION first_notnull_sfunc(anyelement, anyelement) RETURNS anyelement AS $$\n SELECT COALESCE($1,$2);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE AGGREGATE first_notnull(anyelement)(\n SFUNC = first_notnull_sfunc,\n STYPE = anyelement\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_array_unique_merge(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_concat_ignorenull,\n FINALFUNC = jsonb_array_unique\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_min(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_least\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_max(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_greatest\n);\n\nCREATE TABLE IF NOT EXISTS migrations (\n version text PRIMARY KEY,\n datetime timestamptz DEFAULT clock_timestamp() NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION get_version() RETURNS text AS $$\n SELECT version FROM pgstac.migrations ORDER BY datetime DESC, version DESC LIMIT 1;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION set_version(text) RETURNS text AS $$\n INSERT INTO pgstac.migrations (version) VALUES ($1)\n ON CONFLICT DO NOTHING\n RETURNING version;\n$$ LANGUAGE SQL;\n\n\nCREATE TABLE IF NOT EXISTS pgstac_settings (\n name text PRIMARY KEY,\n value text NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION table_empty(text) RETURNS boolean AS $$\nDECLARE\n retval boolean;\nBEGIN\n EXECUTE format($q$\n SELECT NOT EXISTS (SELECT 1 FROM %I LIMIT 1)\n $q$,\n $1\n ) INTO retval;\n RETURN retval;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION get_setting(IN _setting text, IN conf jsonb DEFAULT NULL) RETURNS text AS $$\nSELECT COALESCE(\n nullif(conf->>_setting, ''),\n nullif(current_setting(concat('pgstac.',_setting), TRUE),''),\n nullif((SELECT value FROM pgstac.pgstac_settings WHERE name=_setting),'')\n);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_setting_bool(IN _setting text, IN conf jsonb DEFAULT NULL) RETURNS boolean AS $$\nSELECT COALESCE(\n conf->>_setting,\n current_setting(concat('pgstac.',_setting), TRUE),\n (SELECT value FROM pgstac.pgstac_settings WHERE name=_setting),\n 'FALSE'\n)::boolean;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context(conf jsonb DEFAULT NULL) RETURNS text AS $$\n SELECT pgstac.get_setting('context', conf);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context_estimated_count(conf jsonb DEFAULT NULL) RETURNS int AS $$\n SELECT pgstac.get_setting('context_estimated_count', conf)::int;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_estimated_cost();\nCREATE OR REPLACE FUNCTION context_estimated_cost(conf jsonb DEFAULT NULL) RETURNS float AS $$\n SELECT pgstac.get_setting('context_estimated_cost', conf)::float;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_stats_ttl();\nCREATE OR REPLACE FUNCTION context_stats_ttl(conf jsonb DEFAULT NULL) RETURNS interval AS $$\n SELECT pgstac.get_setting('context_stats_ttl', conf)::interval;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION t2s(text) RETURNS text AS $$\n SELECT extract(epoch FROM $1::interval)::text || ' s';\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE STRICT;\n\nCREATE OR REPLACE FUNCTION age_ms(a timestamptz, b timestamptz DEFAULT clock_timestamp()) RETURNS float AS $$\n SELECT abs(extract(epoch from age(a,b)) * 1000);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION queue_timeout() RETURNS interval AS $$\n SELECT t2s(coalesce(\n get_setting('queue_timeout'),\n '1h'\n ))::interval;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION notice(VARIADIC text[]) RETURNS boolean AS $$\nDECLARE\ndebug boolean := current_setting('pgstac.debug', true);\nBEGIN\n IF debug THEN\n RAISE NOTICE 'NOTICE FROM FUNC: % >>>>> %', concat_ws(' | ', $1), clock_timestamp();\n RETURN TRUE;\n END IF;\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION empty_arr(ANYARRAY) RETURNS BOOLEAN AS $$\nSELECT CASE\n WHEN $1 IS NULL THEN TRUE\n WHEN cardinality($1)<1 THEN TRUE\nELSE FALSE\nEND;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_intersection(_a ANYARRAY, _b ANYARRAY) RETURNS ANYARRAY AS $$\n SELECT ARRAY ( SELECT unnest(_a) INTERSECT SELECT UNNEST(_b) );\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION array_map_ident(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_ident(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_map_literal(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_literal(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION array_reverse(anyarray) RETURNS anyarray AS $$\nSELECT ARRAY(\n SELECT $1[i]\n FROM generate_subscripts($1,1) AS s(i)\n ORDER BY i DESC\n);\n$$ LANGUAGE SQL STRICT IMMUTABLE;\n\nDROP TABLE IF EXISTS query_queue;\nCREATE TABLE query_queue (\n query text PRIMARY KEY,\n added timestamptz DEFAULT now()\n);\n\nDROP TABLE IF EXISTS query_queue_history;\nCREATE TABLE query_queue_history(\n query text,\n added timestamptz NOT NULL,\n finished timestamptz NOT NULL DEFAULT now(),\n error text\n);\n\nCREATE OR REPLACE PROCEDURE run_queued_queries() AS $$\nDECLARE\n qitem query_queue%ROWTYPE;\n timeout_ts timestamptz;\n error text;\n cnt int := 0;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n DELETE FROM query_queue WHERE query = (SELECT query FROM query_queue ORDER BY added DESC LIMIT 1 FOR UPDATE SKIP LOCKED) RETURNING * INTO qitem;\n IF NOT FOUND THEN\n EXIT;\n END IF;\n cnt := cnt + 1;\n BEGIN\n RAISE NOTICE 'RUNNING QUERY: %', qitem.query;\n EXECUTE qitem.query;\n EXCEPTION WHEN others THEN\n error := format('%s | %s', SQLERRM, SQLSTATE);\n END;\n INSERT INTO query_queue_history (query, added, finished, error)\n VALUES (qitem.query, qitem.added, clock_timestamp(), error);\n COMMIT;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION run_queued_queries_intransaction() RETURNS int AS $$\nDECLARE\n qitem query_queue%ROWTYPE;\n timeout_ts timestamptz;\n error text;\n cnt int := 0;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n DELETE FROM query_queue WHERE query = (SELECT query FROM query_queue ORDER BY added DESC LIMIT 1 FOR UPDATE SKIP LOCKED) RETURNING * INTO qitem;\n IF NOT FOUND THEN\n RETURN cnt;\n END IF;\n cnt := cnt + 1;\n BEGIN\n qitem.query := regexp_replace(qitem.query, 'CONCURRENTLY', '');\n RAISE NOTICE 'RUNNING QUERY: %', qitem.query;\n\n EXECUTE qitem.query;\n EXCEPTION WHEN others THEN\n error := format('%s | %s', SQLERRM, SQLSTATE);\n RAISE WARNING '%', error;\n END;\n INSERT INTO query_queue_history (query, added, finished, error)\n VALUES (qitem.query, qitem.added, clock_timestamp(), error);\n END LOOP;\n RETURN cnt;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION run_or_queue(query text) RETURNS VOID AS $$\nDECLARE\n use_queue text := COALESCE(get_setting('use_queue'), 'FALSE')::boolean;\nBEGIN\n IF get_setting_bool('debug') THEN\n RAISE NOTICE '%', query;\n END IF;\n IF use_queue THEN\n INSERT INTO query_queue (query) VALUES (query) ON CONFLICT DO NOTHING;\n ELSE\n EXECUTE query;\n END IF;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n\nDROP FUNCTION IF EXISTS check_pgstac_settings;\nCREATE OR REPLACE FUNCTION check_pgstac_settings(_sysmem text DEFAULT NULL) RETURNS VOID AS $$\nDECLARE\n settingval text;\n sysmem bigint := pg_size_bytes(_sysmem);\n effective_cache_size bigint := pg_size_bytes(current_setting('effective_cache_size', TRUE));\n shared_buffers bigint := pg_size_bytes(current_setting('shared_buffers', TRUE));\n work_mem bigint := pg_size_bytes(current_setting('work_mem', TRUE));\n max_connections int := current_setting('max_connections', TRUE);\n maintenance_work_mem bigint := pg_size_bytes(current_setting('maintenance_work_mem', TRUE));\n seq_page_cost float := current_setting('seq_page_cost', TRUE);\n random_page_cost float := current_setting('random_page_cost', TRUE);\n temp_buffers bigint := pg_size_bytes(current_setting('temp_buffers', TRUE));\n r record;\nBEGIN\n IF _sysmem IS NULL THEN\n RAISE NOTICE 'Call function with the size of your system memory `SELECT check_pgstac_settings(''4GB'')` to get pg system setting recommendations.';\n ELSE\n IF effective_cache_size < (sysmem * 0.5) THEN\n RAISE WARNING 'effective_cache_size of % is set low for a system with %. Recomended value between % and %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.5), pg_size_pretty(sysmem * 0.75);\n ELSIF effective_cache_size > (sysmem * 0.75) THEN\n RAISE WARNING 'effective_cache_size of % is set high for a system with %. Recomended value between % and %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.5), pg_size_pretty(sysmem * 0.75);\n ELSE\n RAISE NOTICE 'effective_cache_size of % is set appropriately for a system with %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem);\n END IF;\n\n IF shared_buffers < (sysmem * 0.2) THEN\n RAISE WARNING 'shared_buffers of % is set low for a system with %. Recomended value between % and %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.2), pg_size_pretty(sysmem * 0.3);\n ELSIF shared_buffers > (sysmem * 0.3) THEN\n RAISE WARNING 'shared_buffers of % is set high for a system with %. Recomended value between % and %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.2), pg_size_pretty(sysmem * 0.3);\n ELSE\n RAISE NOTICE 'shared_buffers of % is set appropriately for a system with %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem);\n END IF;\n shared_buffers = sysmem * 0.3;\n IF maintenance_work_mem < (sysmem * 0.2) THEN\n RAISE WARNING 'maintenance_work_mem of % is set low for shared_buffers of %. Recomended value between % and %', pg_size_pretty(maintenance_work_mem), pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers * 0.2), pg_size_pretty(shared_buffers * 0.3);\n ELSIF maintenance_work_mem > (shared_buffers * 0.3) THEN\n RAISE WARNING 'maintenance_work_mem of % is set high for shared_buffers of %. Recomended value between % and %', pg_size_pretty(maintenance_work_mem), pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers * 0.2), pg_size_pretty(shared_buffers * 0.3);\n ELSE\n RAISE NOTICE 'maintenance_work_mem of % is set appropriately for shared_buffers of %', pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers);\n END IF;\n\n IF work_mem * max_connections > shared_buffers THEN\n RAISE WARNING 'work_mem setting of % is set high for % max_connections please reduce work_mem to % or decrease max_connections to %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers/max_connections), floor(shared_buffers/work_mem);\n ELSIF work_mem * max_connections < (shared_buffers * 0.75) THEN\n RAISE WARNING 'work_mem setting of % is set low for % max_connections you may consider raising work_mem to % or increasing max_connections to %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers/max_connections), floor(shared_buffers/work_mem);\n ELSE\n RAISE NOTICE 'work_mem setting of % and max_connections of % are adequate for shared_buffers of %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers);\n END IF;\n\n IF random_page_cost / seq_page_cost != 1.1 THEN\n RAISE WARNING 'random_page_cost (%) /seq_page_cost (%) should be set to 1.1 for SSD. Change random_page_cost to %', random_page_cost, seq_page_cost, 1.1 * seq_page_cost;\n ELSE\n RAISE NOTICE 'random_page_cost and seq_page_cost set appropriately for SSD';\n END IF;\n\n IF temp_buffers < greatest(pg_size_bytes('128MB'),(maintenance_work_mem / 2)) THEN\n RAISE WARNING 'pgstac makes heavy use of temp tables, consider raising temp_buffers from % to %', pg_size_pretty(temp_buffers), greatest('128MB', pg_size_pretty((shared_buffers / 16)));\n END IF;\n END IF;\n\n RAISE NOTICE 'VALUES FOR PGSTAC VARIABLES';\n RAISE NOTICE 'These can be set either as GUC system variables or by setting in the pgstac_settings table.';\n\n FOR r IN SELECT name, get_setting(name) as setting, CASE WHEN current_setting(concat('pgstac.',name), TRUE) IS NOT NULL THEN concat('pgstac.',name, ' GUC') WHEN value IS NOT NULL THEN 'pgstac_settings table' ELSE 'Not Set' END as loc FROM pgstac_settings LOOP\n RAISE NOTICE '% is set to % from the %', r.name, r.setting, r.loc;\n END LOOP;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pg_cron';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider intalling pg_cron which can be used to automate tasks';\n ELSE\n RAISE NOTICE 'pg_cron % is installed', settingval;\n END IF;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pgstattuple';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider installing the pgstattuple extension which can be used to help maintain tables and indexes.';\n ELSE\n RAISE NOTICE 'pgstattuple % is installed', settingval;\n END IF;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pg_stat_statements';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider installing the pg_stat_statements extension which is very helpful for tracking the types of queries on the system';\n ELSE\n RAISE NOTICE 'pg_stat_statements % is installed', settingval;\n IF current_setting('pg_stat_statements.track_statements', TRUE) IS DISTINCT FROM 'all' THEN\n RAISE WARNING 'SET pg_stat_statements.track_statements TO ''all''; --In order to track statements within functions.';\n END IF;\n END IF;\n\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac, public SET CLIENT_MIN_MESSAGES TO NOTICE;\n/* looks for a geometry in a stac item first from geometry and falling back to bbox */\nCREATE OR REPLACE FUNCTION stac_geom(value jsonb) RETURNS geometry AS $$\nSELECT\n CASE\n WHEN value ? 'intersects' THEN\n ST_GeomFromGeoJSON(value->>'intersects')\n WHEN value ? 'geometry' THEN\n ST_GeomFromGeoJSON(value->>'geometry')\n WHEN value ? 'bbox' THEN\n pgstac.bbox_geom(value->'bbox')\n ELSE NULL\n END as geometry\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION stac_daterange(\n value jsonb\n) RETURNS tstzrange AS $$\nDECLARE\n props jsonb := value;\n dt timestamptz;\n edt timestamptz;\nBEGIN\n IF props ? 'properties' THEN\n props := props->'properties';\n END IF;\n IF\n props ? 'start_datetime'\n AND props->>'start_datetime' IS NOT NULL\n AND props ? 'end_datetime'\n AND props->>'end_datetime' IS NOT NULL\n THEN\n dt := props->>'start_datetime';\n edt := props->>'end_datetime';\n IF dt > edt THEN\n RAISE EXCEPTION 'start_datetime must be < end_datetime';\n END IF;\n ELSE\n dt := props->>'datetime';\n edt := props->>'datetime';\n END IF;\n IF dt is NULL OR edt IS NULL THEN\n RAISE NOTICE 'DT: %, EDT: %', dt, edt;\n RAISE EXCEPTION 'Either datetime (%) or both start_datetime (%) and end_datetime (%) must be set.', props->>'datetime',props->>'start_datetime',props->>'end_datetime';\n END IF;\n RETURN tstzrange(dt, edt, '[]');\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_datetime(value jsonb) RETURNS timestamptz AS $$\n SELECT lower(stac_daterange(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_end_datetime(value jsonb) RETURNS timestamptz AS $$\n SELECT upper(stac_daterange(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE TABLE IF NOT EXISTS stac_extensions(\n url text PRIMARY KEY,\n content jsonb\n);\nCREATE OR REPLACE FUNCTION queryable_signature(n text, c text[]) RETURNS text AS $$\n SELECT concat(n, c);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TABLE queryables (\n id bigint GENERATED ALWAYS AS identity PRIMARY KEY,\n name text NOT NULL,\n collection_ids text[], -- used to determine what partitions to create indexes on\n definition jsonb,\n property_path text,\n property_wrapper text,\n property_index_type text\n);\nCREATE INDEX queryables_name_idx ON queryables (name);\nCREATE INDEX queryables_collection_idx ON queryables USING GIN (collection_ids);\nCREATE INDEX queryables_property_wrapper_idx ON queryables (property_wrapper);\n\nCREATE OR REPLACE FUNCTION pgstac.queryables_constraint_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n allcollections text[];\nBEGIN\n RAISE NOTICE 'Making sure that name/collection is unique for queryables %', NEW;\n IF NEW.collection_ids IS NOT NULL THEN\n IF EXISTS (\n SELECT 1\n FROM unnest(NEW.collection_ids) c\n LEFT JOIN\n collections\n ON (collections.id = c)\n WHERE collections.id IS NULL\n ) THEN\n RAISE foreign_key_violation USING MESSAGE = format(\n 'One or more collections in %s do not exist.', NEW.collection_ids\n );\n RETURN NULL;\n END IF;\n END IF;\n IF TG_OP = 'INSERT' THEN\n IF EXISTS (\n SELECT 1 FROM queryables q\n WHERE\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n )\n ) THEN\n RAISE unique_violation USING MESSAGE = format(\n 'There is already a queryable for %s for a collection in %s: %s',\n NEW.name,\n NEW.collection_ids,\n\t\t\t\t(SELECT json_agg(row_to_json(q)) FROM queryables q WHERE\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n ))\n );\n RETURN NULL;\n END IF;\n END IF;\n IF TG_OP = 'UPDATE' THEN\n IF EXISTS (\n SELECT 1 FROM queryables q\n WHERE\n q.id != NEW.id\n AND\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n )\n ) THEN\n RAISE unique_violation\n USING MESSAGE = format(\n 'There is already a queryable for %s for a collection in %s',\n NEW.name,\n NEW.collection_ids\n );\n RETURN NULL;\n END IF;\n END IF;\n\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER queryables_constraint_insert_trigger\nBEFORE INSERT ON queryables\nFOR EACH ROW EXECUTE PROCEDURE queryables_constraint_triggerfunc();\n\nCREATE TRIGGER queryables_constraint_update_trigger\nBEFORE UPDATE ON queryables\nFOR EACH ROW\nWHEN (NEW.name = OLD.name AND NEW.collection_ids IS DISTINCT FROM OLD.collection_ids)\nEXECUTE PROCEDURE queryables_constraint_triggerfunc();\n\n\nCREATE OR REPLACE FUNCTION array_to_path(arr text[]) RETURNS text AS $$\n SELECT string_agg(\n quote_literal(v),\n '->'\n ) FROM unnest(arr) v;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\n\n\nCREATE OR REPLACE FUNCTION queryable(\n IN dotpath text,\n OUT path text,\n OUT expression text,\n OUT wrapper text,\n OUT nulled_wrapper text\n) AS $$\nDECLARE\n q RECORD;\n path_elements text[];\nBEGIN\n IF dotpath IN ('id', 'geometry', 'datetime', 'end_datetime', 'collection') THEN\n path := dotpath;\n expression := dotpath;\n wrapper := NULL;\n RETURN;\n END IF;\n SELECT * INTO q FROM queryables\n WHERE\n name=dotpath\n OR name = 'properties.' || dotpath\n OR name = replace(dotpath, 'properties.', '')\n ;\n IF q.property_wrapper IS NULL THEN\n IF q.definition->>'type' = 'number' THEN\n wrapper := 'to_float';\n nulled_wrapper := wrapper;\n ELSIF q.definition->>'format' = 'date-time' THEN\n wrapper := 'to_tstz';\n nulled_wrapper := wrapper;\n ELSE\n nulled_wrapper := NULL;\n wrapper := 'to_text';\n END IF;\n ELSE\n wrapper := q.property_wrapper;\n nulled_wrapper := wrapper;\n END IF;\n IF q.property_path IS NOT NULL THEN\n path := q.property_path;\n ELSE\n path_elements := string_to_array(dotpath, '.');\n IF path_elements[1] IN ('links', 'assets', 'stac_version', 'stac_extensions') THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSIF path_elements[1] = 'properties' THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSE\n path := format($F$content->'properties'->%s$F$, array_to_path(path_elements));\n END IF;\n END IF;\n expression := format('%I(%s)', wrapper, path);\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\nCREATE OR REPLACE FUNCTION unnest_collection(collection_ids text[] DEFAULT NULL) RETURNS SETOF text AS $$\n DECLARE\n BEGIN\n IF collection_ids IS NULL THEN\n RETURN QUERY SELECT id FROM collections;\n END IF;\n RETURN QUERY SELECT unnest(collection_ids);\n END;\n$$ LANGUAGE PLPGSQL STABLE;\n\nCREATE OR REPLACE FUNCTION normalize_indexdef(def text) RETURNS text AS $$\nDECLARE\nBEGIN\n def := btrim(def, ' \\n\\t');\n\tdef := regexp_replace(def, '^CREATE (UNIQUE )?INDEX ([^ ]* )?ON (ONLY )?([^ ]* )?', '', 'i');\n RETURN def;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE STRICT PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION indexdef(q queryables) RETURNS text AS $$\n DECLARE\n out text;\n BEGIN\n IF q.name = 'id' THEN\n out := 'CREATE UNIQUE INDEX ON %I USING btree (id)';\n ELSIF q.name = 'datetime' THEN\n out := 'CREATE INDEX ON %I USING btree (datetime DESC, end_datetime)';\n ELSIF q.name = 'geometry' THEN\n out := 'CREATE INDEX ON %I USING gist (geometry)';\n ELSE\n out := format($q$CREATE INDEX ON %%I USING %s (%s(((content -> 'properties'::text) -> %L::text)))$q$,\n lower(COALESCE(q.property_index_type, 'BTREE')),\n lower(COALESCE(q.property_wrapper, 'to_text')),\n q.name\n );\n END IF;\n RETURN btrim(out, ' \\n\\t');\n END;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\n\nDROP VIEW IF EXISTS pgstac_indexes;\nCREATE VIEW pgstac_indexes AS\nSELECT\n i.schemaname,\n i.tablename,\n i.indexname,\n regexp_replace(btrim(replace(replace(indexdef, i.indexname, ''),'pgstac.',''),' \\t\\n'), '[ ]+', ' ', 'g') as idx,\n COALESCE(\n (regexp_match(indexdef, '\\(([a-zA-Z]+)\\)'))[1],\n (regexp_match(indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_-]+)''::text'))[1],\n CASE WHEN indexdef ~* '\\(datetime desc, end_datetime\\)' THEN 'datetime' ELSE NULL END\n ) AS field,\n pg_table_size(i.indexname::text) as index_size,\n pg_size_pretty(pg_table_size(i.indexname::text)) as index_size_pretty\nFROM\n pg_indexes i\nWHERE i.schemaname='pgstac' and i.tablename ~ '_items_' AND indexdef !~* ' only ';\n\nDROP VIEW IF EXISTS pgstac_index_stats;\nCREATE VIEW pgstac_indexes_stats AS\nSELECT\n i.schemaname,\n i.tablename,\n i.indexname,\n indexdef,\n COALESCE(\n (regexp_match(indexdef, '\\(([a-zA-Z]+)\\)'))[1],\n (regexp_match(indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_]+)''::text'))[1],\n CASE WHEN indexdef ~* '\\(datetime desc, end_datetime\\)' THEN 'datetime_end_datetime' ELSE NULL END\n ) AS field,\n pg_table_size(i.indexname::text) as index_size,\n pg_size_pretty(pg_table_size(i.indexname::text)) as index_size_pretty,\n n_distinct,\n most_common_vals::text::text[],\n most_common_freqs::text::text[],\n histogram_bounds::text::text[],\n correlation\nFROM\n pg_indexes i\n LEFT JOIN pg_stats s ON (s.tablename = i.indexname)\nWHERE i.schemaname='pgstac' and i.tablename ~ '_items_';\n\nset check_function_bodies to off;\nCREATE OR REPLACE FUNCTION maintain_partition_queries(\n part text DEFAULT 'items',\n dropindexes boolean DEFAULT FALSE,\n rebuildindexes boolean DEFAULT FALSE,\n idxconcurrently boolean DEFAULT FALSE\n) RETURNS SETOF text AS $$\nDECLARE\n rec record;\nBEGIN\n FOR rec IN (\n WITH p AS (\n SELECT\n relid::text as partition,\n replace(replace(\n CASE\n WHEN level = 1 THEN pg_get_expr(c.relpartbound, c.oid)\n ELSE pg_get_expr(parent.relpartbound, parent.oid)\n END,\n 'FOR VALUES IN (''',''), ''')',\n ''\n ) AS collection\n FROM pg_partition_tree('items')\n JOIN pg_class c ON (relid::regclass = c.oid)\n JOIN pg_class parent ON (parentrelid::regclass = parent.oid AND isleaf)\n ), i AS (\n SELECT\n partition,\n indexname,\n regexp_replace(btrim(replace(replace(indexdef, indexname, ''),'pgstac.',''),' \\t\\n'), '[ ]+', ' ', 'g') as iidx,\n COALESCE(\n (regexp_match(indexdef, '\\(([a-zA-Z]+)\\)'))[1],\n (regexp_match(indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_-]+)''::text'))[1],\n CASE WHEN indexdef ~* '\\(datetime desc, end_datetime\\)' THEN 'datetime' ELSE NULL END\n ) AS field\n FROM\n pg_indexes\n JOIN p ON (tablename=partition)\n ), q AS (\n SELECT\n name AS field,\n collection,\n partition,\n format(indexdef(queryables), partition) as qidx\n FROM queryables, unnest_collection(queryables.collection_ids) collection\n JOIN p USING (collection)\n WHERE property_index_type IS NOT NULL OR name IN ('datetime','geometry','id')\n )\n SELECT * FROM i FULL JOIN q USING (field, partition)\n WHERE lower(iidx) IS DISTINCT FROM lower(qidx)\n ) LOOP\n IF rec.iidx IS NULL THEN\n IF idxconcurrently THEN\n RETURN NEXT replace(rec.qidx, 'INDEX', 'INDEX CONCURRENTLY');\n ELSE\n RETURN NEXT rec.qidx;\n END IF;\n ELSIF rec.qidx IS NULL AND dropindexes THEN\n RETURN NEXT format('DROP INDEX IF EXISTS %I;', rec.indexname);\n ELSIF lower(rec.qidx) != lower(rec.iidx) THEN\n IF dropindexes THEN\n RETURN NEXT format('DROP INDEX IF EXISTS %I; %s;', rec.indexname, rec.qidx);\n ELSE\n IF idxconcurrently THEN\n RETURN NEXT replace(rec.qidx, 'INDEX', 'INDEX CONCURRENTLY');\n ELSE\n RETURN NEXT rec.qidx;\n END IF;\n END IF;\n ELSIF rebuildindexes and rec.indexname IS NOT NULL THEN\n IF idxconcurrently THEN\n RETURN NEXT format('REINDEX INDEX CONCURRENTLY %I;', rec.indexname);\n ELSE\n RETURN NEXT format('REINDEX INDEX %I;', rec.indexname);\n END IF;\n END IF;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION maintain_partitions(\n part text DEFAULT 'items',\n dropindexes boolean DEFAULT FALSE,\n rebuildindexes boolean DEFAULT FALSE\n) RETURNS VOID AS $$\n WITH t AS (\n SELECT run_or_queue(q) FROM maintain_partition_queries(part, dropindexes, rebuildindexes) q\n ) SELECT count(*) FROM t;\n$$ LANGUAGE SQL;\n\n\nCREATE OR REPLACE FUNCTION queryables_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\nBEGIN\n PERFORM maintain_partitions();\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER queryables_trigger AFTER INSERT OR UPDATE ON queryables\nFOR EACH STATEMENT EXECUTE PROCEDURE queryables_trigger_func();\n\n\nCREATE OR REPLACE FUNCTION get_queryables(_collection_ids text[] DEFAULT NULL) RETURNS jsonb AS $$\nDECLARE\nBEGIN\n -- Build up queryables if the input contains valid collection ids or is empty\n IF EXISTS (\n SELECT 1 FROM collections\n WHERE\n _collection_ids IS NULL\n OR cardinality(_collection_ids) = 0\n OR id = ANY(_collection_ids)\n )\n THEN\n RETURN (\n WITH base AS (\n SELECT\n unnest(collection_ids) as collection_id,\n name,\n coalesce(definition, '{\"type\":\"string\"}'::jsonb) as definition\n FROM queryables\n WHERE\n _collection_ids IS NULL OR\n _collection_ids = '{}'::text[] OR\n _collection_ids && collection_ids\n UNION ALL\n SELECT null, name, coalesce(definition, '{\"type\":\"string\"}'::jsonb) as definition\n FROM queryables WHERE collection_ids IS NULL OR collection_ids = '{}'::text[]\n ), g AS (\n SELECT\n name,\n first_notnull(definition) as definition,\n jsonb_array_unique_merge(definition->'enum') as enum,\n jsonb_min(definition->'minimum') as minimum,\n jsonb_min(definition->'maxiumn') as maximum\n FROM base\n GROUP BY 1\n )\n SELECT\n jsonb_build_object(\n '$schema', 'http://json-schema.org/draft-07/schema#',\n '$id', '',\n 'type', 'object',\n 'title', 'STAC Queryables.',\n 'properties', jsonb_object_agg(\n name,\n definition\n ||\n jsonb_strip_nulls(jsonb_build_object(\n 'enum', enum,\n 'minimum', minimum,\n 'maximum', maximum\n ))\n )\n )\n FROM g\n );\n ELSE\n RETURN NULL;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\nCREATE OR REPLACE FUNCTION get_queryables(_collection text DEFAULT NULL) RETURNS jsonb AS $$\n SELECT\n CASE\n WHEN _collection IS NULL THEN get_queryables(NULL::text[])\n ELSE get_queryables(ARRAY[_collection])\n END\n ;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_queryables() RETURNS jsonb AS $$\n SELECT get_queryables(NULL::text[]);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION schema_qualify_refs(url text, j jsonb) returns jsonb as $$\n SELECT regexp_replace(j::text, '\"\\$ref\": \"#', concat('\"$ref\": \"', url, '#'), 'g')::jsonb;\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE VIEW stac_extension_queryables AS\nSELECT DISTINCT key as name, schema_qualify_refs(e.url, j.value) as definition FROM stac_extensions e, jsonb_each(e.content->'definitions'->'fields'->'properties') j;\n\n\nCREATE OR REPLACE FUNCTION missing_queryables(_collection text, _tablesample float DEFAULT 5, minrows float DEFAULT 10) RETURNS TABLE(collection text, name text, definition jsonb, property_wrapper text) AS $$\nDECLARE\n q text;\n _partition text;\n explain_json json;\n psize float;\n estrows float;\nBEGIN\n SELECT format('_items_%s', key) INTO _partition FROM collections WHERE id=_collection;\n\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM %I;', _partition)\n INTO explain_json;\n psize := explain_json->0->'Plan'->'Plan Rows';\n estrows := _tablesample * .01 * psize;\n IF estrows < minrows THEN\n _tablesample := least(100,greatest(_tablesample, (estrows / psize) / 100));\n RAISE NOTICE '%', (psize / estrows) / 100;\n END IF;\n RAISE NOTICE 'Using tablesample % to find missing queryables from % % that has ~% rows estrows: %', _tablesample, _collection, _partition, psize, estrows;\n\n q := format(\n $q$\n WITH q AS (\n SELECT * FROM queryables\n WHERE\n collection_ids IS NULL\n OR %L = ANY(collection_ids)\n ), t AS (\n SELECT\n content->'properties' AS properties\n FROM\n %I\n TABLESAMPLE SYSTEM(%L)\n ), p AS (\n SELECT DISTINCT ON (key)\n key,\n value,\n s.definition\n FROM t\n JOIN LATERAL jsonb_each(properties) ON TRUE\n LEFT JOIN q ON (q.name=key)\n LEFT JOIN stac_extension_queryables s ON (s.name=key)\n WHERE q.definition IS NULL\n )\n SELECT\n %L,\n key,\n COALESCE(definition, jsonb_build_object('type',jsonb_typeof(value))) as definition,\n CASE\n WHEN definition->>'type' = 'integer' THEN 'to_int'\n WHEN COALESCE(definition->>'type', jsonb_typeof(value)) = 'number' THEN 'to_float'\n WHEN COALESCE(definition->>'type', jsonb_typeof(value)) = 'array' THEN 'to_text_array'\n ELSE 'to_text'\n END\n FROM p;\n $q$,\n _collection,\n _partition,\n _tablesample,\n _collection\n );\n RETURN QUERY EXECUTE q;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION missing_queryables(_tablesample float DEFAULT 5) RETURNS TABLE(collection_ids text[], name text, definition jsonb, property_wrapper text) AS $$\n SELECT\n array_agg(collection),\n name,\n definition,\n property_wrapper\n FROM\n collections\n JOIN LATERAL\n missing_queryables(id, _tablesample) c\n ON TRUE\n GROUP BY\n 2,3,4\n ORDER BY 2,1\n ;\n$$ LANGUAGE SQL;\nCREATE OR REPLACE FUNCTION parse_dtrange(\n _indate jsonb,\n relative_base timestamptz DEFAULT date_trunc('hour', CURRENT_TIMESTAMP)\n) RETURNS tstzrange AS $$\nDECLARE\n timestrs text[];\n s timestamptz;\n e timestamptz;\nBEGIN\n timestrs :=\n CASE\n WHEN _indate ? 'timestamp' THEN\n ARRAY[_indate->>'timestamp']\n WHEN _indate ? 'interval' THEN\n to_text_array(_indate->'interval')\n WHEN jsonb_typeof(_indate) = 'array' THEN\n to_text_array(_indate)\n ELSE\n regexp_split_to_array(\n _indate->>0,\n '/'\n )\n END;\n RAISE NOTICE 'TIMESTRS %', timestrs;\n IF cardinality(timestrs) = 1 THEN\n IF timestrs[1] ILIKE 'P%' THEN\n RETURN tstzrange(relative_base - upper(timestrs[1])::interval, relative_base, '[)');\n END IF;\n s := timestrs[1]::timestamptz;\n RETURN tstzrange(s, s, '[]');\n END IF;\n\n IF cardinality(timestrs) != 2 THEN\n RAISE EXCEPTION 'Timestamp cannot have more than 2 values';\n END IF;\n\n IF timestrs[1] = '..' OR timestrs[1] = '' THEN\n s := '-infinity'::timestamptz;\n e := timestrs[2]::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] = '..' OR timestrs[2] = '' THEN\n s := timestrs[1]::timestamptz;\n e := 'infinity'::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[1] ILIKE 'P%' AND timestrs[2] NOT ILIKE 'P%' THEN\n e := timestrs[2]::timestamptz;\n s := e - upper(timestrs[1])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] ILIKE 'P%' AND timestrs[1] NOT ILIKE 'P%' THEN\n s := timestrs[1]::timestamptz;\n e := s + upper(timestrs[2])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n s := timestrs[1]::timestamptz;\n e := timestrs[2]::timestamptz;\n\n RETURN tstzrange(s,e,'[)');\n\n RETURN NULL;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT PARALLEL SAFE SET TIME ZONE 'UTC';\n\nCREATE OR REPLACE FUNCTION parse_dtrange(\n _indate text,\n relative_base timestamptz DEFAULT CURRENT_TIMESTAMP\n) RETURNS tstzrange AS $$\n SELECT parse_dtrange(to_jsonb(_indate), relative_base);\n$$ LANGUAGE SQL STABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION temporal_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\n ll text := 'datetime';\n lh text := 'end_datetime';\n rrange tstzrange;\n rl text;\n rh text;\n outq text;\nBEGIN\n rrange := parse_dtrange(args->1);\n RAISE NOTICE 'Constructing temporal query OP: %, ARGS: %, RRANGE: %', op, args, rrange;\n op := lower(op);\n rl := format('%L::timestamptz', lower(rrange));\n rh := format('%L::timestamptz', upper(rrange));\n outq := CASE op\n WHEN 't_before' THEN 'lh < rl'\n WHEN 't_after' THEN 'll > rh'\n WHEN 't_meets' THEN 'lh = rl'\n WHEN 't_metby' THEN 'll = rh'\n WHEN 't_overlaps' THEN 'll < rl AND rl < lh < rh'\n WHEN 't_overlappedby' THEN 'rl < ll < rh AND lh > rh'\n WHEN 't_starts' THEN 'll = rl AND lh < rh'\n WHEN 't_startedby' THEN 'll = rl AND lh > rh'\n WHEN 't_during' THEN 'll > rl AND lh < rh'\n WHEN 't_contains' THEN 'll < rl AND lh > rh'\n WHEN 't_finishes' THEN 'll > rl AND lh = rh'\n WHEN 't_finishedby' THEN 'll < rl AND lh = rh'\n WHEN 't_equals' THEN 'll = rl AND lh = rh'\n WHEN 't_disjoint' THEN 'NOT (ll <= rh AND lh >= rl)'\n WHEN 't_intersects' THEN 'll <= rh AND lh >= rl'\n WHEN 'anyinteracts' THEN 'll <= rh AND lh >= rl'\n END;\n outq := regexp_replace(outq, '\\mll\\M', ll);\n outq := regexp_replace(outq, '\\mlh\\M', lh);\n outq := regexp_replace(outq, '\\mrl\\M', rl);\n outq := regexp_replace(outq, '\\mrh\\M', rh);\n outq := format('(%s)', outq);\n RETURN outq;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\n\n\nCREATE OR REPLACE FUNCTION spatial_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\n geom text;\n j jsonb := args->1;\nBEGIN\n op := lower(op);\n RAISE NOTICE 'Constructing spatial query OP: %, ARGS: %', op, args;\n IF op NOT IN ('s_equals','s_disjoint','s_touches','s_within','s_overlaps','s_crosses','s_intersects','intersects','s_contains') THEN\n RAISE EXCEPTION 'Spatial Operator % Not Supported', op;\n END IF;\n op := regexp_replace(op, '^s_', 'st_');\n IF op = 'intersects' THEN\n op := 'st_intersects';\n END IF;\n -- Convert geometry to WKB string\n IF j ? 'type' AND j ? 'coordinates' THEN\n geom := st_geomfromgeojson(j)::text;\n ELSIF jsonb_typeof(j) = 'array' THEN\n geom := bbox_geom(j)::text;\n END IF;\n\n RETURN format('%s(geometry, %L::geometry)', op, geom);\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION query_to_cql2(q jsonb) RETURNS jsonb AS $$\n-- Translates anything passed in through the deprecated \"query\" into equivalent CQL2\nWITH t AS (\n SELECT key as property, value as ops\n FROM jsonb_each(q)\n), t2 AS (\n SELECT property, (jsonb_each(ops)).*\n FROM t WHERE jsonb_typeof(ops) = 'object'\n UNION ALL\n SELECT property, 'eq', ops\n FROM t WHERE jsonb_typeof(ops) != 'object'\n)\nSELECT\n jsonb_strip_nulls(jsonb_build_object(\n 'op', 'and',\n 'args', jsonb_agg(\n jsonb_build_object(\n 'op', key,\n 'args', jsonb_build_array(\n jsonb_build_object('property',property),\n value\n )\n )\n )\n )\n) as qcql FROM t2\n;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\nCREATE OR REPLACE FUNCTION cql1_to_cql2(j jsonb) RETURNS jsonb AS $$\nDECLARE\n args jsonb;\n ret jsonb;\nBEGIN\n RAISE NOTICE 'CQL1_TO_CQL2: %', j;\n IF j ? 'filter' THEN\n RETURN cql1_to_cql2(j->'filter');\n END IF;\n IF j ? 'property' THEN\n RETURN j;\n END IF;\n IF jsonb_typeof(j) = 'array' THEN\n SELECT jsonb_agg(cql1_to_cql2(el)) INTO args FROM jsonb_array_elements(j) el;\n RETURN args;\n END IF;\n IF jsonb_typeof(j) = 'number' THEN\n RETURN j;\n END IF;\n IF jsonb_typeof(j) = 'string' THEN\n RETURN j;\n END IF;\n\n IF jsonb_typeof(j) = 'object' THEN\n SELECT jsonb_build_object(\n 'op', key,\n 'args', cql1_to_cql2(value)\n ) INTO ret\n FROM jsonb_each(j)\n WHERE j IS NOT NULL;\n RETURN ret;\n END IF;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE STRICT;\n\nCREATE TABLE cql2_ops (\n op text PRIMARY KEY,\n template text,\n types text[]\n);\nINSERT INTO cql2_ops (op, template, types) VALUES\n ('eq', '%s = %s', NULL),\n ('neq', '%s != %s', NULL),\n ('ne', '%s != %s', NULL),\n ('!=', '%s != %s', NULL),\n ('<>', '%s != %s', NULL),\n ('lt', '%s < %s', NULL),\n ('lte', '%s <= %s', NULL),\n ('gt', '%s > %s', NULL),\n ('gte', '%s >= %s', NULL),\n ('le', '%s <= %s', NULL),\n ('ge', '%s >= %s', NULL),\n ('=', '%s = %s', NULL),\n ('<', '%s < %s', NULL),\n ('<=', '%s <= %s', NULL),\n ('>', '%s > %s', NULL),\n ('>=', '%s >= %s', NULL),\n ('like', '%s LIKE %s', NULL),\n ('ilike', '%s ILIKE %s', NULL),\n ('+', '%s + %s', NULL),\n ('-', '%s - %s', NULL),\n ('*', '%s * %s', NULL),\n ('/', '%s / %s', NULL),\n ('not', 'NOT (%s)', NULL),\n ('between', '%s BETWEEN %s AND %s', NULL),\n ('isnull', '%s IS NULL', NULL),\n ('upper', 'upper(%s)', NULL),\n ('lower', 'lower(%s)', NULL)\nON CONFLICT (op) DO UPDATE\n SET\n template = EXCLUDED.template\n;\n\n\nCREATE OR REPLACE FUNCTION cql2_query(j jsonb, wrapper text DEFAULT NULL) RETURNS text AS $$\n#variable_conflict use_variable\nDECLARE\n args jsonb := j->'args';\n arg jsonb;\n op text := lower(j->>'op');\n cql2op RECORD;\n literal text;\n _wrapper text;\n leftarg text;\n rightarg text;\nBEGIN\n IF j IS NULL OR (op IS NOT NULL AND args IS NULL) THEN\n RETURN NULL;\n END IF;\n RAISE NOTICE 'CQL2_QUERY: %', j;\n IF j ? 'filter' THEN\n RETURN cql2_query(j->'filter');\n END IF;\n\n IF j ? 'upper' THEN\n RETURN cql2_query(jsonb_build_object('op', 'upper', 'args', j->'upper'));\n END IF;\n\n IF j ? 'lower' THEN\n RETURN cql2_query(jsonb_build_object('op', 'lower', 'args', j->'lower'));\n END IF;\n\n -- Temporal Query\n IF op ilike 't_%' or op = 'anyinteracts' THEN\n RETURN temporal_op_query(op, args);\n END IF;\n\n -- If property is a timestamp convert it to text to use with\n -- general operators\n IF j ? 'timestamp' THEN\n RETURN format('%L::timestamptz', to_tstz(j->'timestamp'));\n END IF;\n IF j ? 'interval' THEN\n RAISE EXCEPTION 'Please use temporal operators when using intervals.';\n RETURN NONE;\n END IF;\n\n -- Spatial Query\n IF op ilike 's_%' or op = 'intersects' THEN\n RETURN spatial_op_query(op, args);\n END IF;\n\n IF op IN ('a_equals','a_contains','a_contained_by','a_overlaps') THEN\n IF args->0 ? 'property' THEN\n leftarg := format('to_text_array(%s)', (queryable(args->0->>'property')).path);\n END IF;\n IF args->1 ? 'property' THEN\n rightarg := format('to_text_array(%s)', (queryable(args->1->>'property')).path);\n END IF;\n RETURN FORMAT(\n '%s %s %s',\n COALESCE(leftarg, quote_literal(to_text_array(args->0))),\n CASE op\n WHEN 'a_equals' THEN '='\n WHEN 'a_contains' THEN '@>'\n WHEN 'a_contained_by' THEN '<@'\n WHEN 'a_overlaps' THEN '&&'\n END,\n COALESCE(rightarg, quote_literal(to_text_array(args->1)))\n );\n END IF;\n\n IF op = 'in' THEN\n RAISE NOTICE 'IN : % % %', args, jsonb_build_array(args->0), args->1;\n args := jsonb_build_array(args->0) || (args->1);\n RAISE NOTICE 'IN2 : %', args;\n END IF;\n\n\n\n IF op = 'between' THEN\n args = jsonb_build_array(\n args->0,\n args->1->0,\n args->1->1\n );\n END IF;\n\n -- Make sure that args is an array and run cql2_query on\n -- each element of the array\n RAISE NOTICE 'ARGS PRE: %', args;\n IF j ? 'args' THEN\n IF jsonb_typeof(args) != 'array' THEN\n args := jsonb_build_array(args);\n END IF;\n\n IF jsonb_path_exists(args, '$[*] ? (@.property == \"id\" || @.property == \"datetime\" || @.property == \"end_datetime\" || @.property == \"collection\")') THEN\n wrapper := NULL;\n ELSE\n -- if any of the arguments are a property, try to get the property_wrapper\n FOR arg IN SELECT jsonb_path_query(args, '$[*] ? (@.property != null)') LOOP\n RAISE NOTICE 'Arg: %', arg;\n wrapper := (queryable(arg->>'property')).nulled_wrapper;\n RAISE NOTICE 'Property: %, Wrapper: %', arg, wrapper;\n IF wrapper IS NOT NULL THEN\n EXIT;\n END IF;\n END LOOP;\n\n -- if the property was not in queryables, see if any args were numbers\n IF\n wrapper IS NULL\n AND jsonb_path_exists(args, '$[*] ? (@.type()==\"number\")')\n THEN\n wrapper := 'to_float';\n END IF;\n wrapper := coalesce(wrapper, 'to_text');\n END IF;\n\n SELECT jsonb_agg(cql2_query(a, wrapper))\n INTO args\n FROM jsonb_array_elements(args) a;\n END IF;\n RAISE NOTICE 'ARGS: %', args;\n\n IF op IN ('and', 'or') THEN\n RETURN\n format(\n '(%s)',\n array_to_string(to_text_array(args), format(' %s ', upper(op)))\n );\n END IF;\n\n IF op = 'in' THEN\n RAISE NOTICE 'IN -- % %', args->0, to_text(args->0);\n RETURN format(\n '%s IN (%s)',\n to_text(args->0),\n array_to_string((to_text_array(args))[2:], ',')\n );\n END IF;\n\n -- Look up template from cql2_ops\n IF j ? 'op' THEN\n SELECT * INTO cql2op FROM cql2_ops WHERE cql2_ops.op ilike op;\n IF FOUND THEN\n -- If specific index set in queryables for a property cast other arguments to that type\n\n RETURN format(\n cql2op.template,\n VARIADIC (to_text_array(args))\n );\n ELSE\n RAISE EXCEPTION 'Operator % Not Supported.', op;\n END IF;\n END IF;\n\n\n IF wrapper IS NOT NULL THEN\n RAISE NOTICE 'Wrapping % with %', j, wrapper;\n IF j ? 'property' THEN\n RETURN format('%I(%s)', wrapper, (queryable(j->>'property')).path);\n ELSE\n RETURN format('%I(%L)', wrapper, j);\n END IF;\n ELSIF j ? 'property' THEN\n RETURN quote_ident(j->>'property');\n END IF;\n\n RETURN quote_literal(to_text(j));\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION paging_dtrange(\n j jsonb\n) RETURNS tstzrange AS $$\nDECLARE\n op text;\n filter jsonb := j->'filter';\n dtrange tstzrange := tstzrange('-infinity'::timestamptz,'infinity'::timestamptz);\n sdate timestamptz := '-infinity'::timestamptz;\n edate timestamptz := 'infinity'::timestamptz;\n jpitem jsonb;\nBEGIN\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n END IF;\n IF NOT (filter @? '$.**.op ? (@ == \"or\" || @ == \"not\")') THEN\n FOR jpitem IN SELECT j FROM jsonb_path_query(filter,'strict $.** ? (@.args[*].property == \"datetime\")'::jsonpath) j LOOP\n op := lower(jpitem->>'op');\n dtrange := parse_dtrange(jpitem->'args'->1);\n IF op IN ('<=', 'lt', 'lte', '<', 'le', 't_before') THEN\n sdate := greatest(sdate,'-infinity');\n edate := least(edate, upper(dtrange));\n ELSIF op IN ('>=', '>', 'gt', 'gte', 'ge', 't_after') THEN\n edate := least(edate, 'infinity');\n sdate := greatest(sdate, lower(dtrange));\n ELSIF op IN ('=', 'eq') THEN\n edate := least(edate, upper(dtrange));\n sdate := greatest(sdate, lower(dtrange));\n END IF;\n RAISE NOTICE '2 OP: %, ARGS: %, DTRANGE: %, SDATE: %, EDATE: %', op, jpitem->'args'->1, dtrange, sdate, edate;\n END LOOP;\n END IF;\n IF sdate > edate THEN\n RETURN 'empty'::tstzrange;\n END IF;\n RETURN tstzrange(sdate,edate, '[]');\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT SET TIME ZONE 'UTC';\n\nCREATE OR REPLACE FUNCTION paging_collections(\n IN j jsonb\n) RETURNS text[] AS $$\nDECLARE\n filter jsonb := j->'filter';\n jpitem jsonb;\n op text;\n args jsonb;\n arg jsonb;\n collections text[];\nBEGIN\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n END IF;\n IF NOT (filter @? '$.**.op ? (@ == \"or\" || @ == \"not\")') THEN\n FOR jpitem IN SELECT j FROM jsonb_path_query(filter,'strict $.** ? (@.args[*].property == \"collection\")'::jsonpath) j LOOP\n RAISE NOTICE 'JPITEM: %', jpitem;\n op := jpitem->>'op';\n args := jpitem->'args';\n IF op IN ('=', 'eq', 'in') THEN\n FOR arg IN SELECT a FROM jsonb_array_elements(args) a LOOP\n IF jsonb_typeof(arg) IN ('string', 'array') THEN\n RAISE NOTICE 'arg: %, collections: %', arg, collections;\n IF collections IS NULL OR collections = '{}'::text[] THEN\n collections := to_text_array(arg);\n ELSE\n collections := array_intersection(collections, to_text_array(arg));\n END IF;\n END IF;\n END LOOP;\n END IF;\n END LOOP;\n END IF;\n IF collections = '{}'::text[] THEN\n RETURN NULL;\n END IF;\n RETURN collections;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\nCREATE OR REPLACE FUNCTION collection_base_item(content jsonb) RETURNS jsonb AS $$\n SELECT jsonb_build_object(\n 'type', 'Feature',\n 'stac_version', content->'stac_version',\n 'assets', content->'item_assets',\n 'collection', content->'id'\n );\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TABLE IF NOT EXISTS collections (\n key bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,\n id text GENERATED ALWAYS AS (content->>'id') STORED UNIQUE,\n content JSONB NOT NULL,\n base_item jsonb GENERATED ALWAYS AS (pgstac.collection_base_item(content)) STORED,\n partition_trunc text CHECK (partition_trunc IN ('year', 'month'))\n);\n\n\n\nCREATE OR REPLACE FUNCTION collection_base_item(cid text) RETURNS jsonb AS $$\n SELECT pgstac.collection_base_item(content) FROM pgstac.collections WHERE id = cid LIMIT 1;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION create_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION update_collection(data jsonb) RETURNS VOID AS $$\nDECLARE\n out collections%ROWTYPE;\nBEGIN\n UPDATE collections SET content=data WHERE id = data->>'id' RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ON CONFLICT (id) DO\n UPDATE\n SET content=EXCLUDED.content\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION delete_collection(_id text) RETURNS VOID AS $$\nDECLARE\n out collections%ROWTYPE;\nBEGIN\n DELETE FROM collections WHERE id = _id RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION get_collection(id text) RETURNS jsonb AS $$\n SELECT content FROM collections\n WHERE id=$1\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION all_collections() RETURNS jsonb AS $$\n SELECT jsonb_agg(content) FROM collections;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\nCREATE TABLE items (\n id text NOT NULL,\n geometry geometry NOT NULL,\n collection text NOT NULL,\n datetime timestamptz NOT NULL,\n end_datetime timestamptz NOT NULL,\n content JSONB NOT NULL\n)\nPARTITION BY LIST (collection)\n;\n\nCREATE INDEX \"datetime_idx\" ON items USING BTREE (datetime DESC, end_datetime ASC);\nCREATE INDEX \"geometry_idx\" ON items USING GIST (geometry);\n\nCREATE STATISTICS datetime_stats (dependencies) on datetime, end_datetime from items;\n\nALTER TABLE items ADD CONSTRAINT items_collections_fk FOREIGN KEY (collection) REFERENCES collections(id) ON DELETE CASCADE DEFERRABLE;\n\nCREATE OR REPLACE FUNCTION partition_after_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p text;\n t timestamptz := clock_timestamp();\nBEGIN\n RAISE NOTICE 'Updating partition stats %', t;\n FOR p IN SELECT DISTINCT partition\n FROM newdata n JOIN partition_sys_meta p\n ON (n.collection=p.collection AND n.datetime <@ p.partition_dtrange)\n LOOP\n PERFORM run_or_queue(format('SELECT update_partition_stats(%L, %L);', p, true));\n END LOOP;\n IF TG_OP IN ('DELETE','UPDATE') THEN\n DELETE FROM format_item_cache c USING newdata n WHERE c.collection = n.collection AND c.id = n.id;\n END IF;\n RAISE NOTICE 't: % %', t, clock_timestamp() - t;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE TRIGGER items_after_insert_trigger\nAFTER INSERT ON items\nREFERENCING NEW TABLE AS newdata\nFOR EACH STATEMENT\nEXECUTE FUNCTION partition_after_triggerfunc();\n\nCREATE TRIGGER items_after_update_trigger\nAFTER DELETE ON items\nREFERENCING OLD TABLE AS newdata\nFOR EACH STATEMENT\nEXECUTE FUNCTION partition_after_triggerfunc();\n\nCREATE TRIGGER items_after_delete_trigger\nAFTER UPDATE ON items\nREFERENCING NEW TABLE AS newdata\nFOR EACH STATEMENT\nEXECUTE FUNCTION partition_after_triggerfunc();\n\n\nCREATE OR REPLACE FUNCTION content_slim(_item jsonb) RETURNS jsonb AS $$\n SELECT strip_jsonb(_item - '{id,geometry,collection,type}'::text[], collection_base_item(_item->>'collection')) - '{id,geometry,collection,type}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_dehydrate(content jsonb) RETURNS items AS $$\n SELECT\n content->>'id' as id,\n stac_geom(content) as geometry,\n content->>'collection' as collection,\n stac_datetime(content) as datetime,\n stac_end_datetime(content) as end_datetime,\n content_slim(content) as content\n ;\n$$ LANGUAGE SQL STABLE;\n\nCREATE OR REPLACE FUNCTION include_field(f text, fields jsonb DEFAULT '{}'::jsonb) RETURNS boolean AS $$\nDECLARE\n includes jsonb := fields->'include';\n excludes jsonb := fields->'exclude';\nBEGIN\n IF f IS NULL THEN\n RETURN NULL;\n END IF;\n\n\n IF\n jsonb_typeof(excludes) = 'array'\n AND jsonb_array_length(excludes)>0\n AND excludes ? f\n THEN\n RETURN FALSE;\n END IF;\n\n IF\n (\n jsonb_typeof(includes) = 'array'\n AND jsonb_array_length(includes) > 0\n AND includes ? f\n ) OR\n (\n includes IS NULL\n OR jsonb_typeof(includes) = 'null'\n OR jsonb_array_length(includes) = 0\n )\n THEN\n RETURN TRUE;\n END IF;\n\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nDROP FUNCTION IF EXISTS content_hydrate(jsonb, jsonb, jsonb);\nCREATE OR REPLACE FUNCTION content_hydrate(\n _item jsonb,\n _base_item jsonb,\n fields jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\n SELECT merge_jsonb(\n jsonb_fields(_item, fields),\n jsonb_fields(_base_item, fields)\n );\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION content_hydrate(_item items, _collection collections, fields jsonb DEFAULT '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\n content jsonb;\n base_item jsonb := _collection.base_item;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry, 20)::jsonb;\n END IF;\n output := content_hydrate(\n jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'collection', _item.collection,\n 'type', 'Feature'\n ) || _item.content,\n _collection.base_item,\n fields\n );\n\n RETURN output;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_nonhydrated(\n _item items,\n fields jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry, 20)::jsonb;\n END IF;\n output := jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'collection', _item.collection,\n 'type', 'Feature'\n ) || _item.content;\n RETURN output;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_hydrate(_item items, fields jsonb DEFAULT '{}'::jsonb) RETURNS jsonb AS $$\n SELECT content_hydrate(\n _item,\n (SELECT c FROM collections c WHERE id=_item.collection LIMIT 1),\n fields\n );\n$$ LANGUAGE SQL STABLE;\n\n\nCREATE UNLOGGED TABLE items_staging (\n content JSONB NOT NULL\n);\nCREATE UNLOGGED TABLE items_staging_ignore (\n content JSONB NOT NULL\n);\nCREATE UNLOGGED TABLE items_staging_upsert (\n content JSONB NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION items_staging_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p record;\n _partitions text[];\n part text;\n ts timestamptz := clock_timestamp();\nBEGIN\n RAISE NOTICE 'Creating Partitions. %', clock_timestamp() - ts;\n\n FOR part IN WITH t AS (\n SELECT\n n.content->>'collection' as collection,\n stac_daterange(n.content->'properties') as dtr,\n partition_trunc\n FROM newdata n JOIN collections ON (n.content->>'collection'=collections.id)\n ), p AS (\n SELECT\n collection,\n COALESCE(date_trunc(partition_trunc::text, lower(dtr)),'-infinity') as d,\n tstzrange(min(lower(dtr)),max(lower(dtr)),'[]') as dtrange,\n tstzrange(min(upper(dtr)),max(upper(dtr)),'[]') as edtrange\n FROM t\n GROUP BY 1,2\n ) SELECT check_partition(collection, dtrange, edtrange) FROM p LOOP\n RAISE NOTICE 'Partition %', part;\n END LOOP;\n\n RAISE NOTICE 'Doing the insert. %', clock_timestamp() - ts;\n IF TG_TABLE_NAME = 'items_staging' THEN\n INSERT INTO items\n SELECT\n (content_dehydrate(content)).*\n FROM newdata;\n RAISE NOTICE 'Doing the delete. %', clock_timestamp() - ts;\n DELETE FROM items_staging;\n ELSIF TG_TABLE_NAME = 'items_staging_ignore' THEN\n INSERT INTO items\n SELECT\n (content_dehydrate(content)).*\n FROM newdata\n ON CONFLICT DO NOTHING;\n RAISE NOTICE 'Doing the delete. %', clock_timestamp() - ts;\n DELETE FROM items_staging_ignore;\n ELSIF TG_TABLE_NAME = 'items_staging_upsert' THEN\n WITH staging_formatted AS (\n SELECT (content_dehydrate(content)).* FROM newdata\n ), deletes AS (\n DELETE FROM items i USING staging_formatted s\n WHERE\n i.id = s.id\n AND i.collection = s.collection\n AND i IS DISTINCT FROM s\n RETURNING i.id, i.collection\n )\n INSERT INTO items\n SELECT s.* FROM\n staging_formatted s\n ON CONFLICT DO NOTHING;\n RAISE NOTICE 'Doing the delete. %', clock_timestamp() - ts;\n DELETE FROM items_staging_upsert;\n END IF;\n RAISE NOTICE 'Done. %', clock_timestamp() - ts;\n\n RETURN NULL;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER items_staging_insert_trigger AFTER INSERT ON items_staging REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\nCREATE TRIGGER items_staging_insert_ignore_trigger AFTER INSERT ON items_staging_ignore REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\nCREATE TRIGGER items_staging_insert_upsert_trigger AFTER INSERT ON items_staging_upsert REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\n\nCREATE OR REPLACE FUNCTION item_by_id(_id text, _collection text DEFAULT NULL) RETURNS items AS\n$$\nDECLARE\n i items%ROWTYPE;\nBEGIN\n SELECT * INTO i FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection) LIMIT 1;\n RETURN i;\nEND;\n$$ LANGUAGE PLPGSQL STABLE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION get_item(_id text, _collection text DEFAULT NULL) RETURNS jsonb AS $$\n SELECT content_hydrate(items) FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection);\n$$ LANGUAGE SQL STABLE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION delete_item(_id text, _collection text DEFAULT NULL) RETURNS VOID AS $$\nDECLARE\nout items%ROWTYPE;\nBEGIN\n DELETE FROM items WHERE id = _id AND (_collection IS NULL OR collection=_collection) RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n--/*\nCREATE OR REPLACE FUNCTION create_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION update_item(content jsonb) RETURNS VOID AS $$\nDECLARE\n old items %ROWTYPE;\n out items%ROWTYPE;\nBEGIN\n PERFORM delete_item(content->>'id', content->>'collection');\n PERFORM create_item(content);\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION create_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION collection_bbox(id text) RETURNS jsonb AS $$\n SELECT (replace(replace(replace(st_extent(geometry)::text,'BOX(','[['),')',']]'),' ',','))::jsonb\n FROM items WHERE collection=$1;\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION collection_temporal_extent(id text) RETURNS jsonb AS $$\n SELECT to_jsonb(array[array[min(datetime)::text, max(datetime)::text]])\n FROM items WHERE collection=$1;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection_extents() RETURNS VOID AS $$\nUPDATE collections SET\n content = content ||\n jsonb_build_object(\n 'extent', jsonb_build_object(\n 'spatial', jsonb_build_object(\n 'bbox', collection_bbox(collections.id)\n ),\n 'temporal', jsonb_build_object(\n 'interval', collection_temporal_extent(collections.id)\n )\n )\n )\n;\n$$ LANGUAGE SQL;\nCREATE TABLE partition_stats (\n partition text PRIMARY KEY,\n dtrange tstzrange,\n edtrange tstzrange,\n spatial geometry,\n last_updated timestamptz,\n keys text[]\n) WITH (FILLFACTOR=90);\n\nCREATE INDEX partitions_range_idx ON partition_stats USING GIST(dtrange);\n\n\nCREATE OR REPLACE FUNCTION constraint_tstzrange(expr text) RETURNS tstzrange AS $$\n WITH t AS (\n SELECT regexp_matches(\n expr,\n E'\\\\(''\\([0-9 :+-]*\\)''\\\\).*\\\\(''\\([0-9 :+-]*\\)''\\\\)'\n ) AS m\n ) SELECT tstzrange(m[1]::timestamptz, m[2]::timestamptz) FROM t\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE STRICT;\n\nCREATE OR REPLACE FUNCTION dt_constraint(coid oid, OUT dt tstzrange, OUT edt tstzrange) RETURNS RECORD AS $$\nDECLARE\n expr text := pg_get_constraintdef(coid);\n matches timestamptz[];\nBEGIN\n IF expr LIKE '%NULL%' THEN\n dt := tstzrange(null::timestamptz, null::timestamptz);\n edt := tstzrange(null::timestamptz, null::timestamptz);\n RETURN;\n END IF;\n WITH f AS (SELECT (regexp_matches(expr, E'([0-9]{4}-[0-1][0-9]-[0-3][0-9] [0-2][0-9]:[0-5][0-9]:[0-5][0-9]\\.?[0-9]*)', 'g'))[1] f)\n SELECT array_agg(f::timestamptz) INTO matches FROM f;\n IF cardinality(matches) = 4 THEN\n dt := tstzrange(matches[1], matches[2],'[]');\n edt := tstzrange(matches[3], matches[4], '[]');\n RETURN;\n ELSIF cardinality(matches) = 2 THEN\n edt := tstzrange(matches[1], matches[2],'[]');\n RETURN;\n END IF;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\nCREATE OR REPLACE VIEW partition_sys_meta AS\nSELECT\n relid::text as partition,\n replace(replace(CASE WHEN level = 1 THEN pg_get_expr(c.relpartbound, c.oid)\n ELSE pg_get_expr(parent.relpartbound, parent.oid)\n END, 'FOR VALUES IN (''',''), ''')','') AS collection,\n level,\n c.reltuples,\n c.relhastriggers,\n COALESCE(constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as partition_dtrange,\n COALESCE((dt_constraint(edt.oid)).dt, constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as constraint_dtrange,\n COALESCE((dt_constraint(edt.oid)).edt, tstzrange('-infinity', 'infinity','[]')) as constraint_edtrange\nFROM\n pg_partition_tree('items')\n JOIN pg_class c ON (relid::regclass = c.oid)\n JOIN pg_class parent ON (parentrelid::regclass = parent.oid AND isleaf)\n LEFT JOIN pg_constraint edt ON (conrelid=c.oid AND contype='c')\nWHERE isleaf\n;\n\nCREATE VIEW partitions_view AS\nSELECT\n relid::text as partition,\n replace(replace(CASE WHEN level = 1 THEN pg_get_expr(c.relpartbound, c.oid)\n ELSE pg_get_expr(parent.relpartbound, parent.oid)\n END, 'FOR VALUES IN (''',''), ''')','') AS collection,\n level,\n c.reltuples,\n c.relhastriggers,\n COALESCE(constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as partition_dtrange,\n COALESCE((dt_constraint(edt.oid)).dt, constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as constraint_dtrange,\n COALESCE((dt_constraint(edt.oid)).edt, tstzrange('-infinity', 'infinity','[]')) as constraint_edtrange,\n dtrange,\n edtrange,\n spatial,\n last_updated\nFROM\n pg_partition_tree('items')\n JOIN pg_class c ON (relid::regclass = c.oid)\n JOIN pg_class parent ON (parentrelid::regclass = parent.oid AND isleaf)\n LEFT JOIN pg_constraint edt ON (conrelid=c.oid AND contype='c')\n LEFT JOIN partition_stats ON (relid::text=partition)\nWHERE isleaf\n;\n\nCREATE MATERIALIZED VIEW partitions AS\nSELECT * FROM partitions_view;\nCREATE UNIQUE INDEX ON partitions (partition);\n\nCREATE MATERIALIZED VIEW partition_steps AS\nSELECT\n partition as name,\n date_trunc('month',lower(partition_dtrange)) as sdate,\n date_trunc('month', upper(partition_dtrange)) + '1 month'::interval as edate\n FROM partitions_view WHERE partition_dtrange IS NOT NULL AND partition_dtrange != 'empty'::tstzrange\n ORDER BY dtrange ASC\n;\n\n\nCREATE OR REPLACE FUNCTION update_partition_stats_q(_partition text, istrigger boolean default false) RETURNS VOID AS $$\nDECLARE\nBEGIN\n PERFORM run_or_queue(\n format('SELECT update_partition_stats(%L, %L);', _partition, istrigger)\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION update_partition_stats(_partition text, istrigger boolean default false) RETURNS VOID AS $$\nDECLARE\n dtrange tstzrange;\n edtrange tstzrange;\n cdtrange tstzrange;\n cedtrange tstzrange;\n extent geometry;\n collection text;\nBEGIN\n RAISE NOTICE 'Updating stats for %.', _partition;\n EXECUTE format(\n $q$\n SELECT\n tstzrange(min(datetime), max(datetime),'[]'),\n tstzrange(min(end_datetime), max(end_datetime), '[]')\n FROM %I\n $q$,\n _partition\n ) INTO dtrange, edtrange;\n extent := st_estimatedextent('pgstac', _partition, 'geometry');\n INSERT INTO partition_stats (partition, dtrange, edtrange, spatial, last_updated)\n SELECT _partition, dtrange, edtrange, extent, now()\n ON CONFLICT (partition) DO\n UPDATE SET\n dtrange=EXCLUDED.dtrange,\n edtrange=EXCLUDED.edtrange,\n spatial=EXCLUDED.spatial,\n last_updated=EXCLUDED.last_updated\n ;\n\n SELECT\n constraint_dtrange, constraint_edtrange, pv.collection\n INTO cdtrange, cedtrange, collection\n FROM partitions_view pv WHERE partition = _partition;\n REFRESH MATERIALIZED VIEW partitions;\n REFRESH MATERIALIZED VIEW partition_steps;\n\n\n RAISE NOTICE 'Checking if we need to modify constraints.';\n IF\n (cdtrange IS DISTINCT FROM dtrange OR edtrange IS DISTINCT FROM cedtrange)\n AND NOT istrigger\n THEN\n RAISE NOTICE 'Modifying Constraints';\n RAISE NOTICE 'Existing % %', cdtrange, cedtrange;\n RAISE NOTICE 'New % %', dtrange, edtrange;\n PERFORM drop_table_constraints(_partition);\n PERFORM create_table_constraints(_partition, dtrange, edtrange);\n END IF;\n RAISE NOTICE 'Checking if we need to update collection extents.';\n IF get_setting_bool('update_collection_extent') THEN\n RAISE NOTICE 'updating collection extent for %', collection;\n PERFORM run_or_queue(format($q$\n UPDATE collections\n SET content = jsonb_set_lax(\n content,\n '{extent}'::text[],\n collection_extent(%L, FALSE),\n true,\n 'use_json_null'\n ) WHERE id=%L\n ;\n $q$, collection, collection));\n ELSE\n RAISE NOTICE 'Not updating collection extent for %', collection;\n END IF;\n\nEND;\n$$ LANGUAGE PLPGSQL STRICT;\n\n\nCREATE OR REPLACE FUNCTION partition_name( IN collection text, IN dt timestamptz, OUT partition_name text, OUT partition_range tstzrange) AS $$\nDECLARE\n c RECORD;\n parent_name text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n parent_name := format('_items_%s', c.key);\n\n\n IF c.partition_trunc = 'year' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYY'));\n ELSIF c.partition_trunc = 'month' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYYMM'));\n ELSE\n partition_name := parent_name;\n partition_range := tstzrange('-infinity'::timestamptz, 'infinity'::timestamptz, '[]');\n END IF;\n IF partition_range IS NULL THEN\n partition_range := tstzrange(\n date_trunc(c.partition_trunc::text, dt),\n date_trunc(c.partition_trunc::text, dt) + concat('1 ', c.partition_trunc)::interval\n );\n END IF;\n RETURN;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION drop_table_constraints(t text) RETURNS text AS $$\nDECLARE\n q text;\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM partitions_view WHERE partition=t) THEN\n RETURN NULL;\n END IF;\n FOR q IN SELECT FORMAT(\n $q$\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n $q$,\n t,\n conname\n ) FROM pg_constraint\n WHERE conrelid=t::regclass::oid AND contype='c'\n LOOP\n EXECUTE q;\n END LOOP;\n RETURN t;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE OR REPLACE FUNCTION create_table_constraints(t text, _dtrange tstzrange, _edtrange tstzrange) RETURNS text AS $$\nDECLARE\n q text;\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM partitions_view WHERE partition=t) THEN\n RETURN NULL;\n END IF;\n RAISE NOTICE 'Creating Table Constraints for % % %', t, _dtrange, _edtrange;\n IF _dtrange = 'empty' AND _edtrange = 'empty' THEN\n q :=format(\n $q$\n DO $block$\n BEGIN\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n ALTER TABLE %I\n ADD CONSTRAINT %I\n CHECK (((datetime IS NULL) AND (end_datetime IS NULL))) NOT VALID\n ;\n ALTER TABLE %I\n VALIDATE CONSTRAINT %I\n ;\n\n\n\n EXCEPTION WHEN others THEN\n RAISE WARNING '%%, Issue Altering Constraints. Please run update_partition_stats(%I)', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n $block$;\n $q$,\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t),\n t\n );\n ELSE\n q :=format(\n $q$\n DO $block$\n BEGIN\n\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n ALTER TABLE %I\n ADD CONSTRAINT %I\n CHECK (\n (datetime >= %L)\n AND (datetime <= %L)\n AND (end_datetime >= %L)\n AND (end_datetime <= %L)\n ) NOT VALID\n ;\n ALTER TABLE %I\n VALIDATE CONSTRAINT %I\n ;\n\n\n\n EXCEPTION WHEN others THEN\n RAISE WARNING '%%, Issue Altering Constraints. Please run update_partition_stats(%I)', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n $block$;\n $q$,\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t),\n lower(_dtrange),\n upper(_dtrange),\n lower(_edtrange),\n upper(_edtrange),\n t,\n format('%s_dt', t),\n t\n );\n END IF;\n PERFORM run_or_queue(q);\n RETURN t;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION check_partition(\n _collection text,\n _dtrange tstzrange,\n _edtrange tstzrange\n) RETURNS text AS $$\nDECLARE\n c RECORD;\n pm RECORD;\n _partition_name text;\n _partition_dtrange tstzrange;\n _constraint_dtrange tstzrange;\n _constraint_edtrange tstzrange;\n q text;\n deferrable_q text;\n err_context text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=_collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', _collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n\n IF c.partition_trunc IS NOT NULL THEN\n _partition_dtrange := tstzrange(\n date_trunc(c.partition_trunc, lower(_dtrange)),\n date_trunc(c.partition_trunc, lower(_dtrange)) + (concat('1 ', c.partition_trunc))::interval,\n '[)'\n );\n ELSE\n _partition_dtrange := '[-infinity, infinity]'::tstzrange;\n END IF;\n\n IF NOT _partition_dtrange @> _dtrange THEN\n RAISE EXCEPTION 'dtrange % is greater than the partition size % for collection %', _dtrange, c.partition_trunc, _collection;\n END IF;\n\n\n IF c.partition_trunc = 'year' THEN\n _partition_name := format('_items_%s_%s', c.key, to_char(lower(_partition_dtrange),'YYYY'));\n ELSIF c.partition_trunc = 'month' THEN\n _partition_name := format('_items_%s_%s', c.key, to_char(lower(_partition_dtrange),'YYYYMM'));\n ELSE\n _partition_name := format('_items_%s', c.key);\n END IF;\n\n SELECT * INTO pm FROM partition_sys_meta WHERE collection=_collection AND partition_dtrange @> _dtrange;\n IF FOUND THEN\n RAISE NOTICE '% % %', _edtrange, _dtrange, pm;\n _constraint_edtrange :=\n tstzrange(\n least(\n lower(_edtrange),\n nullif(lower(pm.constraint_edtrange), '-infinity')\n ),\n greatest(\n upper(_edtrange),\n nullif(upper(pm.constraint_edtrange), 'infinity')\n ),\n '[]'\n );\n _constraint_dtrange :=\n tstzrange(\n least(\n lower(_dtrange),\n nullif(lower(pm.constraint_dtrange), '-infinity')\n ),\n greatest(\n upper(_dtrange),\n nullif(upper(pm.constraint_dtrange), 'infinity')\n ),\n '[]'\n );\n\n IF pm.constraint_edtrange @> _edtrange AND pm.constraint_dtrange @> _dtrange THEN\n RETURN pm.partition;\n ELSE\n PERFORM drop_table_constraints(_partition_name);\n END IF;\n ELSE\n _constraint_edtrange := _edtrange;\n _constraint_dtrange := _dtrange;\n END IF;\n RAISE NOTICE 'Creating partition % %', _partition_name, _partition_dtrange;\n IF c.partition_trunc IS NULL THEN\n q := format(\n $q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n $q$,\n _partition_name,\n _collection,\n concat(_partition_name,'_pk'),\n _partition_name\n );\n ELSE\n q := format(\n $q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L) PARTITION BY RANGE (datetime);\n CREATE TABLE IF NOT EXISTS %I partition OF %I FOR VALUES FROM (%L) TO (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n $q$,\n format('_items_%s', c.key),\n _collection,\n _partition_name,\n format('_items_%s', c.key),\n lower(_partition_dtrange),\n upper(_partition_dtrange),\n format('%s_pk', _partition_name),\n _partition_name\n );\n END IF;\n\n BEGIN\n EXECUTE q;\n EXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', _partition_name;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\n END;\n PERFORM create_table_constraints(_partition_name, _constraint_dtrange, _constraint_edtrange);\n PERFORM maintain_partitions(_partition_name);\n PERFORM update_partition_stats_q(_partition_name, true);\n REFRESH MATERIALIZED VIEW partitions;\n REFRESH MATERIALIZED VIEW partition_steps;\n RETURN _partition_name;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION repartition(_collection text, _partition_trunc text, triggered boolean DEFAULT FALSE) RETURNS text AS $$\nDECLARE\n c RECORD;\n q text;\n from_trunc text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=_collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', _collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n IF triggered THEN\n RAISE NOTICE 'Converting % to % partitioning via Trigger', _collection, _partition_trunc;\n ELSE\n RAISE NOTICE 'Converting % from using % to % partitioning', _collection, c.partition_trunc, _partition_trunc;\n IF c.partition_trunc IS NOT DISTINCT FROM _partition_trunc THEN\n RAISE NOTICE 'Collection % already set to use partition by %', _collection, _partition_trunc;\n RETURN _collection;\n END IF;\n END IF;\n\n IF EXISTS (SELECT 1 FROM partitions_view WHERE collection=_collection LIMIT 1) THEN\n EXECUTE format(\n $q$\n CREATE TEMP TABLE changepartitionstaging ON COMMIT DROP AS SELECT * FROM %I;\n DROP TABLE IF EXISTS %I CASCADE;\n WITH p AS (\n SELECT\n collection,\n CASE WHEN %L IS NULL THEN '-infinity'::timestamptz\n ELSE date_trunc(%L, datetime)\n END as d,\n tstzrange(min(datetime),max(datetime),'[]') as dtrange,\n tstzrange(min(datetime),max(datetime),'[]') as edtrange\n FROM changepartitionstaging\n GROUP BY 1,2\n ) SELECT check_partition(collection, dtrange, edtrange) FROM p;\n INSERT INTO items SELECT * FROM changepartitionstaging;\n DROP TABLE changepartitionstaging;\n $q$,\n concat('_items_', c.key),\n concat('_items_', c.key),\n c.partition_trunc,\n c.partition_trunc\n );\n END IF;\n RETURN _collection;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE OR REPLACE FUNCTION collections_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n q text;\n partition_name text := format('_items_%s', NEW.key);\n partition_exists boolean := false;\n partition_empty boolean := true;\n err_context text;\n loadtemp boolean := FALSE;\nBEGIN\n RAISE NOTICE 'Collection Trigger. % %', NEW.id, NEW.key;\n IF TG_OP = 'UPDATE' AND NEW.partition_trunc IS DISTINCT FROM OLD.partition_trunc THEN\n PERFORM repartition(NEW.id, NEW.partition_trunc, TRUE);\n END IF;\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER collections_trigger AFTER\nINSERT\nOR\nUPDATE ON collections\nFOR EACH ROW EXECUTE FUNCTION collections_trigger_func();\n\nCREATE OR REPLACE FUNCTION chunker(\n IN _where text,\n OUT s timestamptz,\n OUT e timestamptz\n) RETURNS SETOF RECORD AS $$\nDECLARE\n explain jsonb;\nBEGIN\n IF _where IS NULL THEN\n _where := ' TRUE ';\n END IF;\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s;', _where)\n INTO explain;\n\n RETURN QUERY\n WITH t AS (\n SELECT j->>0 as p FROM\n jsonb_path_query(\n explain,\n 'strict $.**.\"Relation Name\" ? (@ != null)'\n ) j\n ),\n parts AS (\n SELECT sdate, edate FROM t JOIN partition_steps ON (t.p = name)\n ),\n times AS (\n SELECT sdate FROM parts\n UNION\n SELECT edate FROM parts\n ),\n uniq AS (\n SELECT DISTINCT sdate FROM times ORDER BY sdate\n ),\n last AS (\n SELECT sdate, lead(sdate, 1) over () as edate FROM uniq\n )\n SELECT sdate, edate FROM last WHERE edate IS NOT NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION partition_queries(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN partitions text[] DEFAULT NULL\n) RETURNS SETOF text AS $$\nDECLARE\n query text;\n sdate timestamptz;\n edate timestamptz;\nBEGIN\nIF _where IS NULL OR trim(_where) = '' THEN\n _where = ' TRUE ';\nEND IF;\nRAISE NOTICE 'Getting chunks for % %', _where, _orderby;\nIF _orderby ILIKE 'datetime d%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 DESC LOOP\n RETURN NEXT format($q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n $q$,\n sdate,\n edate,\n _where,\n _orderby\n );\n END LOOP;\nELSIF _orderby ILIKE 'datetime a%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 ASC LOOP\n RETURN NEXT format($q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n $q$,\n sdate,\n edate,\n _where,\n _orderby\n );\n END LOOP;\nELSE\n query := format($q$\n SELECT * FROM items\n WHERE %s\n ORDER BY %s\n $q$, _where, _orderby\n );\n\n RETURN NEXT query;\n RETURN;\nEND IF;\n\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION partition_query_view(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN _limit int DEFAULT 10\n) RETURNS text AS $$\n WITH p AS (\n SELECT * FROM partition_queries(_where, _orderby) p\n )\n SELECT\n CASE WHEN EXISTS (SELECT 1 FROM p) THEN\n (SELECT format($q$\n SELECT * FROM (\n %s\n ) total LIMIT %s\n $q$,\n string_agg(\n format($q$ SELECT * FROM ( %s ) AS sub $q$, p),\n '\n UNION ALL\n '\n ),\n _limit\n ))\n ELSE NULL\n END FROM p;\n$$ LANGUAGE SQL IMMUTABLE;\n\n\n\n\nCREATE OR REPLACE FUNCTION stac_search_to_where(j jsonb) RETURNS text AS $$\nDECLARE\n where_segments text[];\n _where text;\n dtrange tstzrange;\n collections text[];\n geom geometry;\n sdate timestamptz;\n edate timestamptz;\n filterlang text;\n filter jsonb := j->'filter';\nBEGIN\n IF j ? 'ids' THEN\n where_segments := where_segments || format('id = ANY (%L) ', to_text_array(j->'ids'));\n END IF;\n\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n where_segments := where_segments || format('collection = ANY (%L) ', collections);\n END IF;\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n\n where_segments := where_segments || format(' datetime <= %L::timestamptz AND end_datetime >= %L::timestamptz ',\n edate,\n sdate\n );\n END IF;\n\n geom := stac_geom(j);\n IF geom IS NOT NULL THEN\n where_segments := where_segments || format('st_intersects(geometry, %L)',geom);\n END IF;\n\n filterlang := COALESCE(\n j->>'filter-lang',\n get_setting('default_filter_lang', j->'conf')\n );\n IF NOT filter @? '$.**.op' THEN\n filterlang := 'cql-json';\n END IF;\n\n IF filterlang NOT IN ('cql-json','cql2-json') AND j ? 'filter' THEN\n RAISE EXCEPTION '% is not a supported filter-lang. Please use cql-json or cql2-json.', filterlang;\n END IF;\n\n IF j ? 'query' AND j ? 'filter' THEN\n RAISE EXCEPTION 'Can only use either query or filter at one time.';\n END IF;\n\n IF j ? 'query' THEN\n filter := query_to_cql2(j->'query');\n ELSIF filterlang = 'cql-json' THEN\n filter := cql1_to_cql2(filter);\n END IF;\n RAISE NOTICE 'FILTER: %', filter;\n where_segments := where_segments || cql2_query(filter);\n IF cardinality(where_segments) < 1 THEN\n RETURN ' TRUE ';\n END IF;\n\n _where := array_to_string(array_remove(where_segments, NULL), ' AND ');\n\n IF _where IS NULL OR BTRIM(_where) = '' THEN\n RETURN ' TRUE ';\n END IF;\n RETURN _where;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION parse_sort_dir(_dir text, reverse boolean default false) RETURNS text AS $$\n WITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n ) SELECT\n CASE\n WHEN NOT reverse THEN d\n WHEN d = 'ASC' THEN 'DESC'\n WHEN d = 'DESC' THEN 'ASC'\n END\n FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION sort_dir_to_op(_dir text, prev boolean default false) RETURNS text AS $$\n WITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n ) SELECT\n CASE\n WHEN d = 'ASC' AND prev THEN '<='\n WHEN d = 'DESC' AND prev THEN '>='\n WHEN d = 'ASC' THEN '>='\n WHEN d = 'DESC' THEN '<='\n END\n FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION sort_sqlorderby(\n _search jsonb DEFAULT NULL,\n reverse boolean DEFAULT FALSE\n) RETURNS text AS $$\n WITH sortby AS (\n SELECT coalesce(_search->'sortby','[{\"field\":\"datetime\", \"direction\":\"desc\"}]') as sort\n ), withid AS (\n SELECT CASE\n WHEN sort @? '$[*] ? (@.field == \"id\")' THEN sort\n ELSE sort || '[{\"field\":\"id\", \"direction\":\"desc\"}]'::jsonb\n END as sort\n FROM sortby\n ), withid_rows AS (\n SELECT jsonb_array_elements(sort) as value FROM withid\n ),sorts AS (\n SELECT\n coalesce(\n (queryable(value->>'field')).expression\n ) as key,\n parse_sort_dir(value->>'direction', reverse) as dir\n FROM withid_rows\n )\n SELECT array_to_string(\n array_agg(concat(key, ' ', dir)),\n ', '\n ) FROM sorts;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_sort_dir(sort_item jsonb) RETURNS text AS $$\n SELECT CASE WHEN sort_item->>'direction' ILIKE 'desc%' THEN 'DESC' ELSE 'ASC' END;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION get_token_val_str(\n _field text,\n _item items\n) RETURNS text AS $$\nDECLARE\n q text;\n literal text;\nBEGIN\n q := format($q$ SELECT quote_literal(%s) FROM (SELECT $1.*) as r;$q$, _field);\n EXECUTE q INTO literal USING _item;\n RETURN literal;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n\nCREATE OR REPLACE FUNCTION get_token_record(IN _token text, OUT prev BOOLEAN, OUT item items) RETURNS RECORD AS $$\nDECLARE\n _itemid text := _token;\n _collectionid text;\nBEGIN\n IF _token IS NULL THEN\n RETURN;\n END IF;\n RAISE NOTICE 'Looking for token: %', _token;\n prev := FALSE;\n IF _token ILIKE 'prev:%' THEN\n _itemid := replace(_token, 'prev:','');\n prev := TRUE;\n ELSIF _token ILIKE 'next:%' THEN\n _itemid := replace(_token, 'next:', '');\n END IF;\n SELECT id INTO _collectionid FROM collections WHERE _itemid LIKE concat(id,':%');\n IF FOUND THEN\n _itemid := replace(_itemid, concat(_collectionid,':'), '');\n SELECT * INTO item FROM items WHERE id=_itemid AND collection=_collectionid;\n ELSE\n SELECT * INTO item FROM items WHERE id=_itemid;\n END IF;\n IF item IS NULL THEN\n RAISE EXCEPTION 'Could not find item using token: % item: % collection: %', _token, _itemid, _collectionid;\n END IF;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\n\nCREATE OR REPLACE FUNCTION get_token_filter(\n _sortby jsonb DEFAULT '[{\"field\":\"datetime\",\"direction\":\"desc\"}]'::jsonb,\n token_item items DEFAULT NULL,\n prev boolean DEFAULT FALSE,\n inclusive boolean DEFAULT FALSE\n) RETURNS text AS $$\nDECLARE\n ltop text := '<';\n gtop text := '>';\n dir text;\n sort record;\n orfilter text := '';\n orfilters text[] := '{}'::text[];\n andfilters text[] := '{}'::text[];\n output text;\n token_where text;\nBEGIN\n IF _sortby IS NULL OR _sortby = '[]'::jsonb THEN\n _sortby := '[{\"field\":\"datetime\",\"direction\":\"desc\"}]'::jsonb;\n END IF;\n _sortby := _sortby || jsonb_build_object('field','id','direction',_sortby->0->>'direction');\n RAISE NOTICE 'Getting Token Filter. % %', _sortby, token_item;\n IF inclusive THEN\n orfilters := orfilters || format('( id=%L AND collection=%L )' , token_item.id, token_item.collection);\n END IF;\n\n FOR sort IN\n WITH s1 AS (\n SELECT\n _row,\n (queryable(value->>'field')).expression as _field,\n (value->>'field' = 'id') as _isid,\n get_sort_dir(value) as _dir\n FROM jsonb_array_elements(_sortby)\n WITH ORDINALITY AS t(value, _row)\n )\n SELECT\n _row,\n _field,\n _dir,\n get_token_val_str(_field, token_item) as _val\n FROM s1\n WHERE _row <= (SELECT min(_row) FROM s1 WHERE _isid)\n LOOP\n orfilter := NULL;\n RAISE NOTICE 'SORT: %', sort;\n IF sort._val IS NOT NULL AND ((prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC')) THEN\n orfilter := format($f$(\n (%s %s %s) OR (%s IS NULL)\n )$f$,\n sort._field,\n ltop,\n sort._val,\n sort._val\n );\n ELSIF sort._val IS NULL AND ((prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC')) THEN\n RAISE NOTICE '< but null';\n orfilter := format('%s IS NOT NULL', sort._field);\n ELSIF sort._val IS NULL THEN\n RAISE NOTICE '> but null';\n ELSE\n orfilter := format($f$(\n (%s %s %s) OR (%s IS NULL)\n )$f$,\n sort._field,\n gtop,\n sort._val,\n sort._field\n );\n END IF;\n RAISE NOTICE 'ORFILTER: %', orfilter;\n\n IF orfilter IS NOT NULL THEN\n IF sort._row = 1 THEN\n orfilters := orfilters || orfilter;\n ELSE\n orfilters := orfilters || format('(%s AND %s)', array_to_string(andfilters, ' AND '), orfilter);\n END IF;\n END IF;\n IF sort._val IS NOT NULL THEN\n andfilters := andfilters || format('%s = %s', sort._field, sort._val);\n ELSE\n andfilters := andfilters || format('%s IS NULL', sort._field);\n END IF;\n END LOOP;\n\n output := array_to_string(orfilters, ' OR ');\n\n token_where := concat('(',coalesce(output,'true'),')');\n IF trim(token_where) = '' THEN\n token_where := NULL;\n END IF;\n RAISE NOTICE 'TOKEN_WHERE: %',token_where;\n RETURN token_where;\n END;\n$$ LANGUAGE PLPGSQL SET transform_null_equals TO TRUE\n;\n\nCREATE OR REPLACE FUNCTION search_tohash(jsonb) RETURNS jsonb AS $$\n SELECT $1 - '{token,limit,context,includes,excludes}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION search_hash(jsonb, jsonb) RETURNS text AS $$\n SELECT md5(concat(search_tohash($1)::text,$2::text));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TABLE IF NOT EXISTS searches(\n hash text GENERATED ALWAYS AS (search_hash(search, metadata)) STORED PRIMARY KEY,\n search jsonb NOT NULL,\n _where text,\n orderby text,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n metadata jsonb DEFAULT '{}'::jsonb NOT NULL\n);\n\nCREATE TABLE IF NOT EXISTS search_wheres(\n id bigint generated always as identity primary key,\n _where text NOT NULL,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n statslastupdated timestamptz,\n estimated_count bigint,\n estimated_cost float,\n time_to_estimate float,\n total_count bigint,\n time_to_count float,\n partitions text[]\n);\n\nCREATE INDEX IF NOT EXISTS search_wheres_partitions ON search_wheres USING GIN (partitions);\nCREATE UNIQUE INDEX IF NOT EXISTS search_wheres_where ON search_wheres ((md5(_where)));\n\nCREATE OR REPLACE FUNCTION where_stats(inwhere text, updatestats boolean default false, conf jsonb default null) RETURNS search_wheres AS $$\nDECLARE\n t timestamptz;\n i interval;\n explain_json jsonb;\n partitions text[];\n sw search_wheres%ROWTYPE;\n inwhere_hash text := md5(inwhere);\n _context text := lower(context(conf));\n _stats_ttl interval := context_stats_ttl(conf);\n _estimated_cost float := context_estimated_cost(conf);\n _estimated_count int := context_estimated_count(conf);\nBEGIN\n IF _context = 'off' THEN\n sw._where := inwhere;\n return sw;\n END IF;\n\n SELECT * INTO sw FROM search_wheres WHERE md5(_where)=inwhere_hash FOR UPDATE;\n\n -- Update statistics if explicitly set, if statistics do not exist, or statistics ttl has expired\n IF NOT updatestats THEN\n RAISE NOTICE 'Checking if update is needed for: % .', inwhere;\n RAISE NOTICE 'Stats Last Updated: %', sw.statslastupdated;\n RAISE NOTICE 'TTL: %, Age: %', _stats_ttl, now() - sw.statslastupdated;\n RAISE NOTICE 'Context: %, Existing Total: %', _context, sw.total_count;\n IF\n sw.statslastupdated IS NULL\n OR (now() - sw.statslastupdated) > _stats_ttl\n OR (context(conf) != 'off' AND sw.total_count IS NULL)\n THEN\n updatestats := TRUE;\n END IF;\n END IF;\n\n sw._where := inwhere;\n sw.lastused := now();\n sw.usecount := coalesce(sw.usecount,0) + 1;\n\n IF NOT updatestats THEN\n UPDATE search_wheres SET\n lastused = sw.lastused,\n usecount = sw.usecount\n WHERE md5(_where) = inwhere_hash\n RETURNING * INTO sw\n ;\n RETURN sw;\n END IF;\n\n -- Use explain to get estimated count/cost and a list of the partitions that would be hit by the query\n t := clock_timestamp();\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s', inwhere)\n INTO explain_json;\n RAISE NOTICE 'Time for just the explain: %', clock_timestamp() - t;\n i := clock_timestamp() - t;\n\n sw.statslastupdated := now();\n sw.estimated_count := explain_json->0->'Plan'->'Plan Rows';\n sw.estimated_cost := explain_json->0->'Plan'->'Total Cost';\n sw.time_to_estimate := extract(epoch from i);\n\n RAISE NOTICE 'ESTIMATED_COUNT: % < %', sw.estimated_count, _estimated_count;\n RAISE NOTICE 'ESTIMATED_COST: % < %', sw.estimated_cost, _estimated_cost;\n\n -- Do a full count of rows if context is set to on or if auto is set and estimates are low enough\n IF\n _context = 'on'\n OR\n ( _context = 'auto' AND\n (\n sw.estimated_count < _estimated_count\n AND\n sw.estimated_cost < _estimated_cost\n )\n )\n THEN\n t := clock_timestamp();\n RAISE NOTICE 'Calculating actual count...';\n EXECUTE format(\n 'SELECT count(*) FROM items WHERE %s',\n inwhere\n ) INTO sw.total_count;\n i := clock_timestamp() - t;\n RAISE NOTICE 'Actual Count: % -- %', sw.total_count, i;\n sw.time_to_count := extract(epoch FROM i);\n ELSE\n sw.total_count := NULL;\n sw.time_to_count := NULL;\n END IF;\n\n\n INSERT INTO search_wheres\n (_where, lastused, usecount, statslastupdated, estimated_count, estimated_cost, time_to_estimate, partitions, total_count, time_to_count)\n SELECT sw._where, sw.lastused, sw.usecount, sw.statslastupdated, sw.estimated_count, sw.estimated_cost, sw.time_to_estimate, sw.partitions, sw.total_count, sw.time_to_count\n ON CONFLICT ((md5(_where)))\n DO UPDATE\n SET\n lastused = sw.lastused,\n usecount = sw.usecount,\n statslastupdated = sw.statslastupdated,\n estimated_count = sw.estimated_count,\n estimated_cost = sw.estimated_cost,\n time_to_estimate = sw.time_to_estimate,\n total_count = sw.total_count,\n time_to_count = sw.time_to_count\n ;\n RETURN sw;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION search_query(\n _search jsonb = '{}'::jsonb,\n updatestats boolean = false,\n _metadata jsonb = '{}'::jsonb\n) RETURNS searches AS $$\nDECLARE\n search searches%ROWTYPE;\n pexplain jsonb;\n t timestamptz;\n i interval;\n _hash text := search_hash(_search, _metadata);\n doupdate boolean := FALSE;\n insertfound boolean := FALSE;\nBEGIN\n SELECT * INTO search FROM searches\n WHERE hash=_hash;\n\n search.hash := _hash;\n\n -- Calculate the where clause if not already calculated\n IF search._where IS NULL THEN\n search._where := stac_search_to_where(_search);\n ELSE\n doupdate := TRUE;\n END IF;\n\n -- Calculate the order by clause if not already calculated\n IF search.orderby IS NULL THEN\n search.orderby := sort_sqlorderby(_search);\n ELSE\n doupdate := TRUE;\n END IF;\n\n PERFORM where_stats(search._where, updatestats, _search->'conf');\n\n IF NOT doupdate THEN\n INSERT INTO searches (search, _where, orderby, lastused, usecount, metadata)\n VALUES (_search, search._where, search.orderby, clock_timestamp(), 1, _metadata)\n ON CONFLICT (hash) DO NOTHING RETURNING * INTO search;\n IF FOUND THEN\n RETURN search;\n END IF;\n END IF;\n\n UPDATE searches\n SET\n lastused=clock_timestamp(),\n usecount=usecount+1\n WHERE hash=(\n SELECT hash FROM searches\n WHERE hash=_hash\n FOR UPDATE SKIP LOCKED\n );\n IF NOT FOUND THEN\n RAISE NOTICE 'Did not update stats for % due to lock. (This is generally OK)', _search;\n END IF;\n\n RETURN search;\n\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE OR REPLACE FUNCTION search_rows(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN partitions text[] DEFAULT NULL,\n IN _limit int DEFAULT 10\n) RETURNS SETOF items AS $$\nDECLARE\n base_query text;\n query text;\n sdate timestamptz;\n edate timestamptz;\n n int;\n records_left int := _limit;\n timer timestamptz := clock_timestamp();\n full_timer timestamptz := clock_timestamp();\nBEGIN\nIF _where IS NULL OR trim(_where) = '' THEN\n _where = ' TRUE ';\nEND IF;\nRAISE NOTICE 'Getting chunks for % %', _where, _orderby;\n\nbase_query := $q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n LIMIT %L\n$q$;\n\nIF _orderby ILIKE 'datetime d%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 DESC LOOP\n RAISE NOTICE 'Running Query for % to %. %', sdate, edate, age_ms(full_timer);\n query := format(\n base_query,\n sdate,\n edate,\n _where,\n _orderby,\n records_left\n );\n RAISE LOG 'QUERY: %', query;\n timer := clock_timestamp();\n RETURN QUERY EXECUTE query;\n\n GET DIAGNOSTICS n = ROW_COUNT;\n records_left := records_left - n;\n RAISE NOTICE 'Returned %/% Rows From % to %. % to go. Time: %ms', n, _limit, sdate, edate, records_left, age_ms(timer);\n timer := clock_timestamp();\n IF records_left <= 0 THEN\n RAISE NOTICE 'SEARCH_ROWS TOOK %ms', age_ms(full_timer);\n RETURN;\n END IF;\n END LOOP;\nELSIF _orderby ILIKE 'datetime a%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 ASC LOOP\n RAISE NOTICE 'Running Query for % to %. %', sdate, edate, age_ms(full_timer);\n query := format(\n base_query,\n sdate,\n edate,\n _where,\n _orderby,\n records_left\n );\n RAISE LOG 'QUERY: %', query;\n timer := clock_timestamp();\n RETURN QUERY EXECUTE query;\n\n GET DIAGNOSTICS n = ROW_COUNT;\n records_left := records_left - n;\n RAISE NOTICE 'Returned %/% Rows From % to %. % to go. Time: %ms', n, _limit, sdate, edate, records_left, age_ms(timer);\n timer := clock_timestamp();\n IF records_left <= 0 THEN\n RAISE NOTICE 'SEARCH_ROWS TOOK %ms', age_ms(full_timer);\n RETURN;\n END IF;\n END LOOP;\nELSE\n query := format($q$\n SELECT * FROM items\n WHERE %s\n ORDER BY %s\n LIMIT %L\n $q$, _where, _orderby, _limit\n );\n RAISE LOG 'QUERY: %', query;\n timer := clock_timestamp();\n RETURN QUERY EXECUTE query;\n RAISE NOTICE 'FULL QUERY TOOK %ms', age_ms(timer);\nEND IF;\nRAISE NOTICE 'SEARCH_ROWS TOOK %ms', age_ms(full_timer);\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE UNLOGGED TABLE format_item_cache(\n id text,\n collection text,\n fields text,\n hydrated bool,\n output jsonb,\n lastused timestamptz DEFAULT now(),\n usecount int DEFAULT 1,\n timetoformat float,\n PRIMARY KEY (collection, id, fields, hydrated)\n);\nCREATE INDEX ON format_item_cache (lastused);\n\nCREATE OR REPLACE FUNCTION format_item(_item items, _fields jsonb DEFAULT '{}', _hydrated bool DEFAULT TRUE) RETURNS jsonb AS $$\nDECLARE\n cache bool := get_setting_bool('format_cache');\n _output jsonb := null;\n t timestamptz := clock_timestamp();\nBEGIN\n IF cache THEN\n SELECT output INTO _output FROM format_item_cache\n WHERE id=_item.id AND collection=_item.collection AND fields=_fields::text AND hydrated=_hydrated;\n END IF;\n IF _output IS NULL THEN\n IF _hydrated THEN\n _output := content_hydrate(_item, _fields);\n ELSE\n _output := content_nonhydrated(_item, _fields);\n END IF;\n END IF;\n IF cache THEN\n INSERT INTO format_item_cache (id, collection, fields, hydrated, output, timetoformat)\n VALUES (_item.id, _item.collection, _fields::text, _hydrated, _output, age_ms(t))\n ON CONFLICT(collection, id, fields, hydrated) DO\n UPDATE\n SET lastused=now(), usecount = format_item_cache.usecount + 1\n ;\n END IF;\n RETURN _output;\n\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION search(_search jsonb = '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n searches searches%ROWTYPE;\n _where text;\n orderby text;\n search_where search_wheres%ROWTYPE;\n total_count bigint;\n token record;\n token_prev boolean;\n token_item items%ROWTYPE;\n token_where text;\n full_where text;\n init_ts timestamptz := clock_timestamp();\n timer timestamptz := clock_timestamp();\n hydrate bool := NOT (_search->'conf'->>'nohydrate' IS NOT NULL AND (_search->'conf'->>'nohydrate')::boolean = true);\n prev text;\n next text;\n context jsonb;\n collection jsonb;\n out_records jsonb;\n out_len int;\n _limit int := coalesce((_search->>'limit')::int, 10);\n init_limit int := _limit;\n _querylimit int;\n _fields jsonb := coalesce(_search->'fields', '{}'::jsonb);\n has_prev boolean := FALSE;\n has_next boolean := FALSE;\n items_cnt int := coalesce(jsonb_array_length(_search->'ids'),0);\nBEGIN\n RAISE NOTICE 'Items Count: %', items_cnt;\n IF items_cnt > 0 THEN\n IF items_cnt <= _limit THEN\n _limit := items_cnt - 1;\n ELSE\n _limit := items_cnt;\n END IF;\n RAISE NOTICE 'Items is set. Changing limit to %', items_cnt;\n END IF;\n searches := search_query(_search);\n _where := searches._where;\n orderby := searches.orderby;\n search_where := where_stats(_where);\n total_count := coalesce(search_where.total_count, search_where.estimated_count);\n RAISE NOTICE 'SEARCH:TOKEN: %', _search->>'token';\n token := get_token_record(_search->>'token');\n RAISE NOTICE '***TOKEN: %', token;\n _querylimit := _limit + 1;\n IF token IS NOT NULL THEN\n token_prev := token.prev;\n token_item := token.item;\n token_where := get_token_filter(_search->'sortby', token_item, token_prev, FALSE);\n RAISE LOG 'TOKEN_WHERE: % (%ms from search start)', token_where, age_ms(timer);\n IF token_prev THEN -- if we are using a prev token, we know has_next is true\n RAISE LOG 'There is a previous token, so automatically setting has_next to true';\n has_next := TRUE;\n orderby := sort_sqlorderby(_search, TRUE);\n ELSE\n RAISE LOG 'There is a next token, so automatically setting has_prev to true';\n has_prev := TRUE;\n\n END IF;\n ELSE -- if there was no token, we know there is no prev\n RAISE LOG 'There is no token, so we know there is no prev. setting has_prev to false';\n has_prev := FALSE;\n END IF;\n\n full_where := concat_ws(' AND ', _where, token_where);\n RAISE NOTICE 'FULL WHERE CLAUSE: %', full_where;\n RAISE NOTICE 'Time to get counts and build query %', age_ms(timer);\n timer := clock_timestamp();\n\n IF hydrate THEN\n RAISE NOTICE 'Getting hydrated data.';\n ELSE\n RAISE NOTICE 'Getting non-hydrated data.';\n END IF;\n RAISE NOTICE 'CACHE SET TO %', get_setting_bool('format_cache');\n RAISE NOTICE 'Time to set hydration/formatting %', age_ms(timer);\n timer := clock_timestamp();\n SELECT jsonb_agg(format_item(i, _fields, hydrate)) INTO out_records\n FROM search_rows(\n full_where,\n orderby,\n search_where.partitions,\n _querylimit\n ) as i;\n\n RAISE NOTICE 'Time to fetch rows %', age_ms(timer);\n timer := clock_timestamp();\n\n\n IF token_prev THEN\n out_records := flip_jsonb_array(out_records);\n END IF;\n\n RAISE NOTICE 'Query returned % records.', jsonb_array_length(out_records);\n RAISE LOG 'TOKEN: % %', token_item.id, token_item.collection;\n RAISE LOG 'RECORD_1: % %', out_records->0->>'id', out_records->0->>'collection';\n RAISE LOG 'RECORD-1: % %', out_records->-1->>'id', out_records->-1->>'collection';\n\n -- REMOVE records that were from our token\n IF out_records->0->>'id' = token_item.id AND out_records->0->>'collection' = token_item.collection THEN\n out_records := out_records - 0;\n ELSIF out_records->-1->>'id' = token_item.id AND out_records->-1->>'collection' = token_item.collection THEN\n out_records := out_records - -1;\n END IF;\n\n out_len := jsonb_array_length(out_records);\n\n IF out_len = init_limit + 1 THEN\n IF token_prev THEN\n has_prev := TRUE;\n out_records := out_records - 0;\n ELSE\n has_next := TRUE;\n out_records := out_records - -1;\n END IF;\n END IF;\n\n IF has_next THEN\n next := concat(out_records->-1->>'collection', ':', out_records->-1->>'id');\n RAISE NOTICE 'HAS NEXT | %', next;\n END IF;\n\n IF has_prev THEN\n prev := concat(out_records->0->>'collection', ':', out_records->0->>'id');\n RAISE NOTICE 'HAS PREV | %', prev;\n END IF;\n\n RAISE NOTICE 'Time to get prev/next %', age_ms(timer);\n timer := clock_timestamp();\n\n IF context(_search->'conf') != 'off' THEN\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', init_limit,\n 'matched', total_count,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\n ELSE\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', init_limit,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\n END IF;\n\n collection := jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb),\n 'next', next,\n 'prev', prev,\n 'context', context\n );\n\n IF get_setting_bool('timing', _search->'conf') THEN\n collection = collection || jsonb_build_object('timing', age_ms(init_ts));\n END IF;\n\n RAISE NOTICE 'Time to build final json %', age_ms(timer);\n timer := clock_timestamp();\n\n RAISE NOTICE 'Total Time: %', age_ms(current_timestamp);\n RAISE NOTICE 'RETURNING % records. NEXT: %. PREV: %', collection->'context'->>'returned', collection->>'next', collection->>'prev';\n RETURN collection;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION search_cursor(_search jsonb = '{}'::jsonb) RETURNS refcursor AS $$\nDECLARE\n curs refcursor;\n searches searches%ROWTYPE;\n _where text;\n _orderby text;\n q text;\n\nBEGIN\n searches := search_query(_search);\n _where := searches._where;\n _orderby := searches.orderby;\n\n OPEN curs FOR\n WITH p AS (\n SELECT * FROM partition_queries(_where, _orderby) p\n )\n SELECT\n CASE WHEN EXISTS (SELECT 1 FROM p) THEN\n (SELECT format($q$\n SELECT * FROM (\n %s\n ) total\n $q$,\n string_agg(\n format($q$ SELECT * FROM ( %s ) AS sub $q$, p),\n '\n UNION ALL\n '\n )\n ))\n ELSE NULL\n END FROM p;\n RETURN curs;\nEND;\n$$ LANGUAGE PLPGSQL;\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION tileenvelope(zoom int, x int, y int) RETURNS geometry AS $$\nWITH t AS (\n SELECT\n 20037508.3427892 as merc_max,\n -20037508.3427892 as merc_min,\n (2 * 20037508.3427892) / (2 ^ zoom) as tile_size\n)\nSELECT st_makeenvelope(\n merc_min + (tile_size * x),\n merc_max - (tile_size * (y + 1)),\n merc_min + (tile_size * (x + 1)),\n merc_max - (tile_size * y),\n 3857\n) FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;DROP FUNCTION IF EXISTS mercgrid;\n\n\nCREATE OR REPLACE FUNCTION ftime() RETURNS interval as $$\nSELECT age(clock_timestamp(), transaction_timestamp());\n$$ LANGUAGE SQL;\nSET SEARCH_PATH to pgstac, public;\n\nDROP FUNCTION IF EXISTS geometrysearch;\nCREATE OR REPLACE FUNCTION geometrysearch(\n IN geom geometry,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE, -- Return as soon as the passed in geometry is full covered\n IN skipcovered boolean DEFAULT TRUE -- Skip any items that would show up completely under the previous items\n) RETURNS jsonb AS $$\nDECLARE\n search searches%ROWTYPE;\n curs refcursor;\n _where text;\n query text;\n iter_record items%ROWTYPE;\n out_records jsonb := '{}'::jsonb[];\n exit_flag boolean := FALSE;\n counter int := 1;\n scancounter int := 1;\n remaining_limit int := _scanlimit;\n tilearea float;\n unionedgeom geometry;\n clippedgeom geometry;\n unionedgeom_area float := 0;\n prev_area float := 0;\n excludes text[];\n includes text[];\n\nBEGIN\n DROP TABLE IF EXISTS pgstac_results;\n CREATE TEMP TABLE pgstac_results (content jsonb) ON COMMIT DROP;\n\n -- If skipcovered is true then you will always want to exit when the passed in geometry is full\n IF skipcovered THEN\n exitwhenfull := TRUE;\n END IF;\n\n SELECT * INTO search FROM searches WHERE hash=queryhash;\n\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Search with Query Hash % Not Found', queryhash;\n END IF;\n\n tilearea := st_area(geom);\n _where := format('%s AND st_intersects(geometry, %L::geometry)', search._where, geom);\n\n\n FOR query IN SELECT * FROM partition_queries(_where, search.orderby) LOOP\n query := format('%s LIMIT %L', query, remaining_limit);\n RAISE NOTICE '%', query;\n OPEN curs FOR EXECUTE query;\n LOOP\n FETCH curs INTO iter_record;\n EXIT WHEN NOT FOUND;\n IF exitwhenfull OR skipcovered THEN -- If we are not using exitwhenfull or skipcovered, we do not need to do expensive geometry operations\n clippedgeom := st_intersection(geom, iter_record.geometry);\n\n IF unionedgeom IS NULL THEN\n unionedgeom := clippedgeom;\n ELSE\n unionedgeom := st_union(unionedgeom, clippedgeom);\n END IF;\n\n unionedgeom_area := st_area(unionedgeom);\n\n IF skipcovered AND prev_area = unionedgeom_area THEN\n scancounter := scancounter + 1;\n CONTINUE;\n END IF;\n\n prev_area := unionedgeom_area;\n\n RAISE NOTICE '% % % %', unionedgeom_area/tilearea, counter, scancounter, ftime();\n END IF;\n RAISE NOTICE '% %', iter_record, content_hydrate(iter_record, fields);\n INSERT INTO pgstac_results (content) VALUES (content_hydrate(iter_record, fields));\n\n IF counter >= _limit\n OR scancounter > _scanlimit\n OR ftime() > _timelimit\n OR (exitwhenfull AND unionedgeom_area >= tilearea)\n THEN\n exit_flag := TRUE;\n EXIT;\n END IF;\n counter := counter + 1;\n scancounter := scancounter + 1;\n\n END LOOP;\n CLOSE curs;\n EXIT WHEN exit_flag;\n remaining_limit := _scanlimit - scancounter;\n END LOOP;\n\n SELECT jsonb_agg(content) INTO out_records FROM pgstac_results WHERE content IS NOT NULL;\n\n RETURN jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb)\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDROP FUNCTION IF EXISTS geojsonsearch;\nCREATE OR REPLACE FUNCTION geojsonsearch(\n IN geojson jsonb,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_geomfromgeojson(geojson),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS xyzsearch;\nCREATE OR REPLACE FUNCTION xyzsearch(\n IN _x int,\n IN _y int,\n IN _z int,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_transform(tileenvelope(_z, _x, _y), 4326),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE PROCEDURE analyze_items() AS $$\nDECLARE\n q text;\n timeout_ts timestamptz;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n SELECT format('ANALYZE (VERBOSE, SKIP_LOCKED) %I;', relname) INTO q\n FROM pg_stat_user_tables\n WHERE relname like '_item%' AND (n_mod_since_analyze>0 OR last_analyze IS NULL) LIMIT 1;\n IF NOT FOUND THEN\n EXIT;\n END IF;\n RAISE NOTICE '%', q;\n EXECUTE q;\n COMMIT;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE PROCEDURE validate_constraints() AS $$\nDECLARE\n q text;\nBEGIN\n FOR q IN\n SELECT\n FORMAT(\n 'ALTER TABLE %I.%I VALIDATE CONSTRAINT %I;',\n nsp.nspname,\n cls.relname,\n con.conname\n )\n\n FROM pg_constraint AS con\n JOIN pg_class AS cls\n ON con.conrelid = cls.oid\n JOIN pg_namespace AS nsp\n ON cls.relnamespace = nsp.oid\n WHERE convalidated = FALSE AND contype in ('c','f')\n AND nsp.nspname = 'pgstac'\n LOOP\n RAISE NOTICE '%', q;\n PERFORM run_or_queue(q);\n COMMIT;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION collection_extent(_collection text, runupdate boolean default false) RETURNS jsonb AS $$\nDECLARE\n geom_extent geometry;\n mind timestamptz;\n maxd timestamptz;\n extent jsonb;\nBEGIN\n IF runupdate THEN\n PERFORM update_partition_stats_q(partition)\n FROM partitions_view WHERE collection=_collection;\n END IF;\n SELECT\n min(lower(dtrange)),\n max(upper(edtrange)),\n st_extent(spatial)\n INTO\n mind,\n maxd,\n geom_extent\n FROM partitions_view\n WHERE collection=_collection;\n\n IF geom_extent IS NOT NULL AND mind IS NOT NULL AND maxd IS NOT NULL THEN\n extent := jsonb_build_object(\n 'extent', jsonb_build_object(\n 'spatial', jsonb_build_object(\n 'bbox', to_jsonb(array[array[st_xmin(geom_extent), st_ymin(geom_extent), st_xmax(geom_extent), st_ymax(geom_extent)]])\n ),\n 'temporal', jsonb_build_object(\n 'interval', to_jsonb(array[array[mind, maxd]])\n )\n )\n );\n RETURN extent;\n END IF;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('id', '{\"title\": \"Item ID\",\"description\": \"Item identifier\",\"$ref\": \"https://schemas.stacspec.org/v1.0.0/item-spec/json-schema/item.json#/definitions/core/allOf/2/properties/id\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('geometry', '{\"title\": \"Item Geometry\",\"description\": \"Item Geometry\",\"$ref\": \"https://geojson.org/schema/Feature.json\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('datetime','{\"description\": \"Datetime\",\"type\": \"string\",\"title\": \"Acquired\",\"format\": \"date-time\",\"pattern\": \"(\\\\+00:00|Z)$\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('eo:cloud_cover','{\"$ref\": \"https://stac-extensions.github.io/eo/v1.0.0/schema.json#/definitions/fieldsproperties/eo:cloud_cover\"}','to_int','BTREE');\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDELETE FROM queryables a USING queryables b\n WHERE a.name = b.name AND a.collection_ids IS NOT DISTINCT FROM b.collection_ids AND a.id > b.id;\n\n\nINSERT INTO pgstac_settings (name, value) VALUES\n ('context', 'off'),\n ('context_estimated_count', '100000'),\n ('context_estimated_cost', '100000'),\n ('context_stats_ttl', '1 day'),\n ('default_filter_lang', 'cql2-json'),\n ('additional_properties', 'true'),\n ('use_queue', 'false'),\n ('queue_timeout', '10 minutes'),\n ('update_collection_extent', 'false'),\n ('format_cache', 'false')\nON CONFLICT DO NOTHING\n;\n\nALTER FUNCTION to_text COST 5000;\nALTER FUNCTION to_float COST 5000;\nALTER FUNCTION to_int COST 5000;\nALTER FUNCTION to_tstz COST 5000;\nALTER FUNCTION to_text_array COST 5000;\n\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nGRANT ALL ON SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac to pgstac_admin;\n\n-- pgstac_read role limited to using function apis\nGRANT EXECUTE ON FUNCTION search TO pgstac_read;\nGRANT EXECUTE ON FUNCTION search_query TO pgstac_read;\nGRANT EXECUTE ON FUNCTION item_by_id TO pgstac_read;\nGRANT EXECUTE ON FUNCTION get_item TO pgstac_read;\nGRANT SELECT ON ALL TABLES IN SCHEMA pgstac TO pgstac_read;\n\n\nGRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_ingest;\nGRANT USAGE ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_ingest;\n\nSELECT update_partition_stats_q(partition) FROM partitions_view;\nSELECT set_version('0.7.9');\n" }, { "path": "src/pgstac/migrations/pgstac.0.8.0-0.8.1.sql", "content": "SET client_min_messages TO WARNING;\nSET SEARCH_PATH to pgstac, public;\nRESET ROLE;\nDO $$\nDECLARE\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='postgis') THEN\n CREATE EXTENSION IF NOT EXISTS postgis;\n END IF;\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='btree_gist') THEN\n CREATE EXTENSION IF NOT EXISTS btree_gist;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_admin;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_read;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_ingest;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n\nGRANT pgstac_admin TO current_user;\n\n-- Function to make sure pgstac_admin is the owner of items\nCREATE OR REPLACE FUNCTION pgstac_admin_owns() RETURNS VOID AS $$\nDECLARE\n f RECORD;\nBEGIN\n FOR f IN (\n SELECT\n concat(\n oid::regproc::text,\n '(',\n coalesce(pg_get_function_identity_arguments(oid),''),\n ')'\n ) AS name,\n CASE prokind WHEN 'f' THEN 'FUNCTION' WHEN 'p' THEN 'PROCEDURE' WHEN 'a' THEN 'AGGREGATE' END as typ\n FROM pg_proc\n WHERE\n pronamespace=to_regnamespace('pgstac')\n AND proowner != to_regrole('pgstac_admin')\n AND proname NOT LIKE 'pg_stat%'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n FOR f IN (\n SELECT\n oid::regclass::text as name,\n CASE relkind\n WHEN 'i' THEN 'INDEX'\n WHEN 'I' THEN 'INDEX'\n WHEN 'p' THEN 'TABLE'\n WHEN 'r' THEN 'TABLE'\n WHEN 'v' THEN 'VIEW'\n WHEN 'S' THEN 'SEQUENCE'\n ELSE NULL\n END as typ\n FROM pg_class\n WHERE relnamespace=to_regnamespace('pgstac') and relowner != to_regrole('pgstac_admin') AND relkind IN ('r','p','v','S') AND relname NOT LIKE 'pg_stat'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\nSELECT pgstac_admin_owns();\n\nCREATE SCHEMA IF NOT EXISTS pgstac AUTHORIZATION pgstac_admin;\n\nGRANT ALL ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_admin;\n\nALTER ROLE pgstac_admin SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_read SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_ingest SET SEARCH_PATH TO pgstac, public;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\n\nGRANT pgstac_read TO pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\n\nSET ROLE pgstac_admin;\n\nSET SEARCH_PATH TO pgstac, public;\n\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS analyze_items;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS validate_constraints;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n-- Install these idempotently as migrations do not put them before trying to modify the collections table\n\n\nCREATE OR REPLACE FUNCTION collection_geom(content jsonb)\nRETURNS geometry AS $$\n WITH box AS (SELECT content->'extent'->'spatial'->'bbox'->0 as box)\n SELECT\n st_makeenvelope(\n (box->>0)::float,\n (box->>1)::float,\n (box->>2)::float,\n (box->>3)::float,\n 4326\n )\n FROM box;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION collection_datetime(content jsonb)\nRETURNS timestamptz AS $$\n SELECT\n CASE\n WHEN\n (content->'extent'->'temporal'->'interval'->0->>0) IS NULL\n THEN '-infinity'::timestamptz\n ELSE\n (content->'extent'->'temporal'->'interval'->0->>0)::timestamptz\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION collection_enddatetime(content jsonb)\nRETURNS timestamptz AS $$\n SELECT\n CASE\n WHEN\n (content->'extent'->'temporal'->'interval'->0->>1) IS NULL\n THEN 'infinity'::timestamptz\n ELSE\n (content->'extent'->'temporal'->'interval'->0->>1)::timestamptz\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n-- BEGIN migra calculated SQL\n-- END migra calculated SQL\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('id', '{\"title\": \"Item ID\",\"description\": \"Item identifier\",\"$ref\": \"https://schemas.stacspec.org/v1.0.0/item-spec/json-schema/item.json#/definitions/core/allOf/2/properties/id\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('geometry', '{\"title\": \"Item Geometry\",\"description\": \"Item Geometry\",\"$ref\": \"https://geojson.org/schema/Feature.json\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('datetime','{\"description\": \"Datetime\",\"type\": \"string\",\"title\": \"Acquired\",\"format\": \"date-time\",\"pattern\": \"(\\\\+00:00|Z)$\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDELETE FROM queryables a USING queryables b\n WHERE a.name = b.name AND a.collection_ids IS NOT DISTINCT FROM b.collection_ids AND a.id > b.id;\n\n\nINSERT INTO pgstac_settings (name, value) VALUES\n ('context', 'off'),\n ('context_estimated_count', '100000'),\n ('context_estimated_cost', '100000'),\n ('context_stats_ttl', '1 day'),\n ('default_filter_lang', 'cql2-json'),\n ('additional_properties', 'true'),\n ('use_queue', 'false'),\n ('queue_timeout', '10 minutes'),\n ('update_collection_extent', 'false'),\n ('format_cache', 'false')\nON CONFLICT DO NOTHING\n;\n\nALTER FUNCTION to_text COST 5000;\nALTER FUNCTION to_float COST 5000;\nALTER FUNCTION to_int COST 5000;\nALTER FUNCTION to_tstz COST 5000;\nALTER FUNCTION to_text_array COST 5000;\n\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nGRANT ALL ON SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac to pgstac_admin;\n\n-- pgstac_read role limited to using function apis\nGRANT EXECUTE ON FUNCTION search TO pgstac_read;\nGRANT EXECUTE ON FUNCTION search_query TO pgstac_read;\nGRANT EXECUTE ON FUNCTION item_by_id TO pgstac_read;\nGRANT EXECUTE ON FUNCTION get_item TO pgstac_read;\nGRANT SELECT ON ALL TABLES IN SCHEMA pgstac TO pgstac_read;\n\n\nGRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_ingest;\nGRANT USAGE ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_ingest;\n\nSELECT update_partition_stats_q(partition) FROM partitions_view;\nSELECT set_version('0.8.1');\n" }, { "path": "src/pgstac/migrations/pgstac.0.8.0.sql", "content": "RESET ROLE;\nDO $$\nDECLARE\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='postgis') THEN\n CREATE EXTENSION IF NOT EXISTS postgis;\n END IF;\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='btree_gist') THEN\n CREATE EXTENSION IF NOT EXISTS btree_gist;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_admin;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_read;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_ingest;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n\nGRANT pgstac_admin TO current_user;\n\n-- Function to make sure pgstac_admin is the owner of items\nCREATE OR REPLACE FUNCTION pgstac_admin_owns() RETURNS VOID AS $$\nDECLARE\n f RECORD;\nBEGIN\n FOR f IN (\n SELECT\n concat(\n oid::regproc::text,\n '(',\n coalesce(pg_get_function_identity_arguments(oid),''),\n ')'\n ) AS name,\n CASE prokind WHEN 'f' THEN 'FUNCTION' WHEN 'p' THEN 'PROCEDURE' WHEN 'a' THEN 'AGGREGATE' END as typ\n FROM pg_proc\n WHERE\n pronamespace=to_regnamespace('pgstac')\n AND proowner != to_regrole('pgstac_admin')\n AND proname NOT LIKE 'pg_stat%'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n FOR f IN (\n SELECT\n oid::regclass::text as name,\n CASE relkind\n WHEN 'i' THEN 'INDEX'\n WHEN 'I' THEN 'INDEX'\n WHEN 'p' THEN 'TABLE'\n WHEN 'r' THEN 'TABLE'\n WHEN 'v' THEN 'VIEW'\n WHEN 'S' THEN 'SEQUENCE'\n ELSE NULL\n END as typ\n FROM pg_class\n WHERE relnamespace=to_regnamespace('pgstac') and relowner != to_regrole('pgstac_admin') AND relkind IN ('r','p','v','S') AND relname NOT LIKE 'pg_stat'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\nSELECT pgstac_admin_owns();\n\nCREATE SCHEMA IF NOT EXISTS pgstac AUTHORIZATION pgstac_admin;\n\nGRANT ALL ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_admin;\n\nALTER ROLE pgstac_admin SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_read SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_ingest SET SEARCH_PATH TO pgstac, public;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\n\nGRANT pgstac_read TO pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\n\nSET ROLE pgstac_admin;\n\nSET SEARCH_PATH TO pgstac, public;\n\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS analyze_items;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS validate_constraints;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n-- Install these idempotently as migrations do not put them before trying to modify the collections table\n\n\nCREATE OR REPLACE FUNCTION collection_geom(content jsonb)\nRETURNS geometry AS $$\n WITH box AS (SELECT content->'extent'->'spatial'->'bbox'->0 as box)\n SELECT\n st_makeenvelope(\n (box->>0)::float,\n (box->>1)::float,\n (box->>2)::float,\n (box->>3)::float,\n 4326\n )\n FROM box;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION collection_datetime(content jsonb)\nRETURNS timestamptz AS $$\n SELECT\n CASE\n WHEN\n (content->'extent'->'temporal'->'interval'->0->>0) IS NULL\n THEN '-infinity'::timestamptz\n ELSE\n (content->'extent'->'temporal'->'interval'->0->>0)::timestamptz\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION collection_enddatetime(content jsonb)\nRETURNS timestamptz AS $$\n SELECT\n CASE\n WHEN\n (content->'extent'->'temporal'->'interval'->0->>1) IS NULL\n THEN 'infinity'::timestamptz\n ELSE\n (content->'extent'->'temporal'->'interval'->0->>1)::timestamptz\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE TABLE IF NOT EXISTS migrations (\n version text PRIMARY KEY,\n datetime timestamptz DEFAULT clock_timestamp() NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION get_version() RETURNS text AS $$\n SELECT version FROM pgstac.migrations ORDER BY datetime DESC, version DESC LIMIT 1;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION set_version(text) RETURNS text AS $$\n INSERT INTO pgstac.migrations (version) VALUES ($1)\n ON CONFLICT DO NOTHING\n RETURNING version;\n$$ LANGUAGE SQL;\n\n\nCREATE TABLE IF NOT EXISTS pgstac_settings (\n name text PRIMARY KEY,\n value text NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION table_empty(text) RETURNS boolean AS $$\nDECLARE\n retval boolean;\nBEGIN\n EXECUTE format($q$\n SELECT NOT EXISTS (SELECT 1 FROM %I LIMIT 1)\n $q$,\n $1\n ) INTO retval;\n RETURN retval;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION get_setting(IN _setting text, IN conf jsonb DEFAULT NULL) RETURNS text AS $$\nSELECT COALESCE(\n nullif(conf->>_setting, ''),\n nullif(current_setting(concat('pgstac.',_setting), TRUE),''),\n nullif((SELECT value FROM pgstac.pgstac_settings WHERE name=_setting),'')\n);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_setting_bool(IN _setting text, IN conf jsonb DEFAULT NULL) RETURNS boolean AS $$\nSELECT COALESCE(\n nullif(conf->>_setting, ''),\n nullif(current_setting(concat('pgstac.',_setting), TRUE),''),\n nullif((SELECT value FROM pgstac.pgstac_settings WHERE name=_setting),''),\n 'FALSE'\n)::boolean;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context(conf jsonb DEFAULT NULL) RETURNS text AS $$\n SELECT pgstac.get_setting('context', conf);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context_estimated_count(conf jsonb DEFAULT NULL) RETURNS int AS $$\n SELECT pgstac.get_setting('context_estimated_count', conf)::int;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_estimated_cost();\nCREATE OR REPLACE FUNCTION context_estimated_cost(conf jsonb DEFAULT NULL) RETURNS float AS $$\n SELECT pgstac.get_setting('context_estimated_cost', conf)::float;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_stats_ttl();\nCREATE OR REPLACE FUNCTION context_stats_ttl(conf jsonb DEFAULT NULL) RETURNS interval AS $$\n SELECT pgstac.get_setting('context_stats_ttl', conf)::interval;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION t2s(text) RETURNS text AS $$\n SELECT extract(epoch FROM $1::interval)::text || ' s';\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE STRICT;\n\nCREATE OR REPLACE FUNCTION age_ms(a timestamptz, b timestamptz DEFAULT clock_timestamp()) RETURNS float AS $$\n SELECT abs(extract(epoch from age(a,b)) * 1000);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION queue_timeout() RETURNS interval AS $$\n SELECT t2s(coalesce(\n get_setting('queue_timeout'),\n '1h'\n ))::interval;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION notice(VARIADIC text[]) RETURNS boolean AS $$\nDECLARE\ndebug boolean := current_setting('pgstac.debug', true);\nBEGIN\n IF debug THEN\n RAISE NOTICE 'NOTICE FROM FUNC: % >>>>> %', concat_ws(' | ', $1), clock_timestamp();\n RETURN TRUE;\n END IF;\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION empty_arr(ANYARRAY) RETURNS BOOLEAN AS $$\nSELECT CASE\n WHEN $1 IS NULL THEN TRUE\n WHEN cardinality($1)<1 THEN TRUE\nELSE FALSE\nEND;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_intersection(_a ANYARRAY, _b ANYARRAY) RETURNS ANYARRAY AS $$\n SELECT ARRAY ( SELECT unnest(_a) INTERSECT SELECT UNNEST(_b) );\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION array_map_ident(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_ident(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_map_literal(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_literal(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION array_reverse(anyarray) RETURNS anyarray AS $$\nSELECT ARRAY(\n SELECT $1[i]\n FROM generate_subscripts($1,1) AS s(i)\n ORDER BY i DESC\n);\n$$ LANGUAGE SQL STRICT IMMUTABLE;\n\nDROP TABLE IF EXISTS query_queue;\nCREATE TABLE query_queue (\n query text PRIMARY KEY,\n added timestamptz DEFAULT now()\n);\n\nDROP TABLE IF EXISTS query_queue_history;\nCREATE TABLE query_queue_history(\n query text,\n added timestamptz NOT NULL,\n finished timestamptz NOT NULL DEFAULT now(),\n error text\n);\n\nCREATE OR REPLACE PROCEDURE run_queued_queries() AS $$\nDECLARE\n qitem query_queue%ROWTYPE;\n timeout_ts timestamptz;\n error text;\n cnt int := 0;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n DELETE FROM query_queue WHERE query = (SELECT query FROM query_queue ORDER BY added DESC LIMIT 1 FOR UPDATE SKIP LOCKED) RETURNING * INTO qitem;\n IF NOT FOUND THEN\n EXIT;\n END IF;\n cnt := cnt + 1;\n BEGIN\n RAISE NOTICE 'RUNNING QUERY: %', qitem.query;\n EXECUTE qitem.query;\n EXCEPTION WHEN others THEN\n error := format('%s | %s', SQLERRM, SQLSTATE);\n END;\n INSERT INTO query_queue_history (query, added, finished, error)\n VALUES (qitem.query, qitem.added, clock_timestamp(), error);\n COMMIT;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION run_queued_queries_intransaction() RETURNS int AS $$\nDECLARE\n qitem query_queue%ROWTYPE;\n timeout_ts timestamptz;\n error text;\n cnt int := 0;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n DELETE FROM query_queue WHERE query = (SELECT query FROM query_queue ORDER BY added DESC LIMIT 1 FOR UPDATE SKIP LOCKED) RETURNING * INTO qitem;\n IF NOT FOUND THEN\n RETURN cnt;\n END IF;\n cnt := cnt + 1;\n BEGIN\n qitem.query := regexp_replace(qitem.query, 'CONCURRENTLY', '');\n RAISE NOTICE 'RUNNING QUERY: %', qitem.query;\n\n EXECUTE qitem.query;\n EXCEPTION WHEN others THEN\n error := format('%s | %s', SQLERRM, SQLSTATE);\n RAISE WARNING '%', error;\n END;\n INSERT INTO query_queue_history (query, added, finished, error)\n VALUES (qitem.query, qitem.added, clock_timestamp(), error);\n END LOOP;\n RETURN cnt;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION run_or_queue(query text) RETURNS VOID AS $$\nDECLARE\n use_queue text := COALESCE(get_setting('use_queue'), 'FALSE')::boolean;\nBEGIN\n IF get_setting_bool('debug') THEN\n RAISE NOTICE '%', query;\n END IF;\n IF use_queue THEN\n INSERT INTO query_queue (query) VALUES (query) ON CONFLICT DO NOTHING;\n ELSE\n EXECUTE query;\n END IF;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n\nDROP FUNCTION IF EXISTS check_pgstac_settings;\nCREATE OR REPLACE FUNCTION check_pgstac_settings(_sysmem text DEFAULT NULL) RETURNS VOID AS $$\nDECLARE\n settingval text;\n sysmem bigint := pg_size_bytes(_sysmem);\n effective_cache_size bigint := pg_size_bytes(current_setting('effective_cache_size', TRUE));\n shared_buffers bigint := pg_size_bytes(current_setting('shared_buffers', TRUE));\n work_mem bigint := pg_size_bytes(current_setting('work_mem', TRUE));\n max_connections int := current_setting('max_connections', TRUE);\n maintenance_work_mem bigint := pg_size_bytes(current_setting('maintenance_work_mem', TRUE));\n seq_page_cost float := current_setting('seq_page_cost', TRUE);\n random_page_cost float := current_setting('random_page_cost', TRUE);\n temp_buffers bigint := pg_size_bytes(current_setting('temp_buffers', TRUE));\n r record;\nBEGIN\n IF _sysmem IS NULL THEN\n RAISE NOTICE 'Call function with the size of your system memory `SELECT check_pgstac_settings(''4GB'')` to get pg system setting recommendations.';\n ELSE\n IF effective_cache_size < (sysmem * 0.5) THEN\n RAISE WARNING 'effective_cache_size of % is set low for a system with %. Recomended value between % and %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.5), pg_size_pretty(sysmem * 0.75);\n ELSIF effective_cache_size > (sysmem * 0.75) THEN\n RAISE WARNING 'effective_cache_size of % is set high for a system with %. Recomended value between % and %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.5), pg_size_pretty(sysmem * 0.75);\n ELSE\n RAISE NOTICE 'effective_cache_size of % is set appropriately for a system with %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem);\n END IF;\n\n IF shared_buffers < (sysmem * 0.2) THEN\n RAISE WARNING 'shared_buffers of % is set low for a system with %. Recomended value between % and %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.2), pg_size_pretty(sysmem * 0.3);\n ELSIF shared_buffers > (sysmem * 0.3) THEN\n RAISE WARNING 'shared_buffers of % is set high for a system with %. Recomended value between % and %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.2), pg_size_pretty(sysmem * 0.3);\n ELSE\n RAISE NOTICE 'shared_buffers of % is set appropriately for a system with %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem);\n END IF;\n shared_buffers = sysmem * 0.3;\n IF maintenance_work_mem < (sysmem * 0.2) THEN\n RAISE WARNING 'maintenance_work_mem of % is set low for shared_buffers of %. Recomended value between % and %', pg_size_pretty(maintenance_work_mem), pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers * 0.2), pg_size_pretty(shared_buffers * 0.3);\n ELSIF maintenance_work_mem > (shared_buffers * 0.3) THEN\n RAISE WARNING 'maintenance_work_mem of % is set high for shared_buffers of %. Recomended value between % and %', pg_size_pretty(maintenance_work_mem), pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers * 0.2), pg_size_pretty(shared_buffers * 0.3);\n ELSE\n RAISE NOTICE 'maintenance_work_mem of % is set appropriately for shared_buffers of %', pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers);\n END IF;\n\n IF work_mem * max_connections > shared_buffers THEN\n RAISE WARNING 'work_mem setting of % is set high for % max_connections please reduce work_mem to % or decrease max_connections to %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers/max_connections), floor(shared_buffers/work_mem);\n ELSIF work_mem * max_connections < (shared_buffers * 0.75) THEN\n RAISE WARNING 'work_mem setting of % is set low for % max_connections you may consider raising work_mem to % or increasing max_connections to %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers/max_connections), floor(shared_buffers/work_mem);\n ELSE\n RAISE NOTICE 'work_mem setting of % and max_connections of % are adequate for shared_buffers of %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers);\n END IF;\n\n IF random_page_cost / seq_page_cost != 1.1 THEN\n RAISE WARNING 'random_page_cost (%) /seq_page_cost (%) should be set to 1.1 for SSD. Change random_page_cost to %', random_page_cost, seq_page_cost, 1.1 * seq_page_cost;\n ELSE\n RAISE NOTICE 'random_page_cost and seq_page_cost set appropriately for SSD';\n END IF;\n\n IF temp_buffers < greatest(pg_size_bytes('128MB'),(maintenance_work_mem / 2)) THEN\n RAISE WARNING 'pgstac makes heavy use of temp tables, consider raising temp_buffers from % to %', pg_size_pretty(temp_buffers), greatest('128MB', pg_size_pretty((shared_buffers / 16)));\n END IF;\n END IF;\n\n RAISE NOTICE 'VALUES FOR PGSTAC VARIABLES';\n RAISE NOTICE 'These can be set either as GUC system variables or by setting in the pgstac_settings table.';\n\n FOR r IN SELECT name, get_setting(name) as setting, CASE WHEN current_setting(concat('pgstac.',name), TRUE) IS NOT NULL THEN concat('pgstac.',name, ' GUC') WHEN value IS NOT NULL THEN 'pgstac_settings table' ELSE 'Not Set' END as loc FROM pgstac_settings LOOP\n RAISE NOTICE '% is set to % from the %', r.name, r.setting, r.loc;\n END LOOP;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pg_cron';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider intalling pg_cron which can be used to automate tasks';\n ELSE\n RAISE NOTICE 'pg_cron % is installed', settingval;\n END IF;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pgstattuple';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider installing the pgstattuple extension which can be used to help maintain tables and indexes.';\n ELSE\n RAISE NOTICE 'pgstattuple % is installed', settingval;\n END IF;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pg_stat_statements';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider installing the pg_stat_statements extension which is very helpful for tracking the types of queries on the system';\n ELSE\n RAISE NOTICE 'pg_stat_statements % is installed', settingval;\n IF current_setting('pg_stat_statements.track_statements', TRUE) IS DISTINCT FROM 'all' THEN\n RAISE WARNING 'SET pg_stat_statements.track_statements TO ''all''; --In order to track statements within functions.';\n END IF;\n END IF;\n\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac, public SET CLIENT_MIN_MESSAGES TO NOTICE;\nCREATE OR REPLACE FUNCTION to_int(jsonb) RETURNS int AS $$\n SELECT floor(($1->>0)::float)::int;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION to_float(jsonb) RETURNS float AS $$\n SELECT ($1->>0)::float;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION to_tstz(jsonb) RETURNS timestamptz AS $$\n SELECT ($1->>0)::timestamptz;\n$$ LANGUAGE SQL IMMUTABLE STRICT SET TIME ZONE 'UTC' COST 5000 PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION to_text(jsonb) RETURNS text AS $$\n SELECT CASE WHEN jsonb_typeof($1) IN ('array','object') THEN $1::text ELSE $1->>0 END;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION to_text_array(jsonb) RETURNS text[] AS $$\n SELECT\n CASE jsonb_typeof($1)\n WHEN 'array' THEN ARRAY(SELECT jsonb_array_elements_text($1))\n ELSE ARRAY[$1->>0]\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION bbox_geom(_bbox jsonb) RETURNS geometry AS $$\nSELECT CASE jsonb_array_length(_bbox)\n WHEN 4 THEN\n ST_SetSRID(ST_MakeEnvelope(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float,\n (_bbox->>3)::float\n ),4326)\n WHEN 6 THEN\n ST_SetSRID(ST_3DMakeBox(\n ST_MakePoint(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float\n ),\n ST_MakePoint(\n (_bbox->>3)::float,\n (_bbox->>4)::float,\n (_bbox->>5)::float\n )\n ),4326)\n ELSE null END;\n;\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION geom_bbox(_geom geometry) RETURNS jsonb AS $$\n SELECT jsonb_build_array(\n st_xmin(_geom),\n st_ymin(_geom),\n st_xmax(_geom),\n st_ymax(_geom)\n );\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION flip_jsonb_array(j jsonb) RETURNS jsonb AS $$\n SELECT jsonb_agg(value) FROM (SELECT value FROM jsonb_array_elements(j) WITH ORDINALITY ORDER BY ordinality DESC) as t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION explode_dotpaths(j jsonb) RETURNS SETOF text[] AS $$\n SELECT string_to_array(p, '.') as e FROM jsonb_array_elements_text(j) p;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION explode_dotpaths_recurse(IN j jsonb) RETURNS SETOF text[] AS $$\n WITH RECURSIVE t AS (\n SELECT e FROM explode_dotpaths(j) e\n UNION ALL\n SELECT e[1:cardinality(e)-1]\n FROM t\n WHERE cardinality(e)>1\n ) SELECT e FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION jsonb_set_nested(j jsonb, path text[], val jsonb) RETURNS jsonb AS $$\nDECLARE\nBEGIN\n IF cardinality(path) > 1 THEN\n FOR i IN 1..(cardinality(path)-1) LOOP\n IF j #> path[:i] IS NULL THEN\n j := jsonb_set_lax(j, path[:i], '{}', TRUE);\n END IF;\n END LOOP;\n END IF;\n RETURN jsonb_set_lax(j, path, val, true);\n\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\n\n\nCREATE OR REPLACE FUNCTION jsonb_include(j jsonb, f jsonb) RETURNS jsonb AS $$\nDECLARE\n includes jsonb := f-> 'include';\n outj jsonb := '{}'::jsonb;\n path text[];\nBEGIN\n IF\n includes IS NULL\n OR jsonb_array_length(includes) = 0\n THEN\n RETURN j;\n ELSE\n includes := includes || '[\"id\",\"collection\"]'::jsonb;\n FOR path IN SELECT explode_dotpaths(includes) LOOP\n outj := jsonb_set_nested(outj, path, j #> path);\n END LOOP;\n END IF;\n RETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_exclude(j jsonb, f jsonb) RETURNS jsonb AS $$\nDECLARE\n excludes jsonb := f-> 'exclude';\n outj jsonb := j;\n path text[];\nBEGIN\n IF\n excludes IS NULL\n OR jsonb_array_length(excludes) = 0\n THEN\n RETURN j;\n ELSE\n FOR path IN SELECT explode_dotpaths(excludes) LOOP\n outj := outj #- path;\n END LOOP;\n END IF;\n RETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_fields(j jsonb, f jsonb DEFAULT '{\"fields\":[]}') RETURNS jsonb AS $$\n SELECT jsonb_exclude(jsonb_include(j, f), f);\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION merge_jsonb(_a jsonb, _b jsonb) RETURNS jsonb AS $$\n SELECT\n CASE\n WHEN _a = '\"𒍟※\"'::jsonb THEN NULL\n WHEN _a IS NULL OR jsonb_typeof(_a) = 'null' THEN _b\n WHEN jsonb_typeof(_a) = 'object' AND jsonb_typeof(_b) = 'object' THEN\n (\n SELECT\n jsonb_strip_nulls(\n jsonb_object_agg(\n key,\n merge_jsonb(a.value, b.value)\n )\n )\n FROM\n jsonb_each(coalesce(_a,'{}'::jsonb)) as a\n FULL JOIN\n jsonb_each(coalesce(_b,'{}'::jsonb)) as b\n USING (key)\n )\n WHEN\n jsonb_typeof(_a) = 'array'\n AND jsonb_typeof(_b) = 'array'\n AND jsonb_array_length(_a) = jsonb_array_length(_b)\n THEN\n (\n SELECT jsonb_agg(m) FROM\n ( SELECT\n merge_jsonb(\n jsonb_array_elements(_a),\n jsonb_array_elements(_b)\n ) as m\n ) as l\n )\n ELSE _a\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION strip_jsonb(_a jsonb, _b jsonb) RETURNS jsonb AS $$\n SELECT\n CASE\n\n WHEN (_a IS NULL OR jsonb_typeof(_a) = 'null') AND _b IS NOT NULL AND jsonb_typeof(_b) != 'null' THEN '\"𒍟※\"'::jsonb\n WHEN _b IS NULL OR jsonb_typeof(_a) = 'null' THEN _a\n WHEN _a = _b AND jsonb_typeof(_a) = 'object' THEN '{}'::jsonb\n WHEN _a = _b THEN NULL\n WHEN jsonb_typeof(_a) = 'object' AND jsonb_typeof(_b) = 'object' THEN\n (\n SELECT\n jsonb_strip_nulls(\n jsonb_object_agg(\n key,\n strip_jsonb(a.value, b.value)\n )\n )\n FROM\n jsonb_each(_a) as a\n FULL JOIN\n jsonb_each(_b) as b\n USING (key)\n )\n WHEN\n jsonb_typeof(_a) = 'array'\n AND jsonb_typeof(_b) = 'array'\n AND jsonb_array_length(_a) = jsonb_array_length(_b)\n THEN\n (\n SELECT jsonb_agg(m) FROM\n ( SELECT\n strip_jsonb(\n jsonb_array_elements(_a),\n jsonb_array_elements(_b)\n ) as m\n ) as l\n )\n ELSE _a\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION nullif_jsonbnullempty(j jsonb) RETURNS jsonb AS $$\n SELECT nullif(nullif(nullif(j,'null'::jsonb),'{}'::jsonb),'[]'::jsonb);\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION jsonb_array_unique(j jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(jsonb_agg(DISTINCT a)) v FROM jsonb_array_elements(j) a;\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_concat_ignorenull(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT coalesce(a,'[]'::jsonb) || coalesce(b,'[]'::jsonb);\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_least(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(least(nullif_jsonbnullempty(a), nullif_jsonbnullempty(b)));\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_greatest(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(greatest(a, b));\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION first_notnull_sfunc(anyelement, anyelement) RETURNS anyelement AS $$\n SELECT COALESCE($1,$2);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE AGGREGATE first_notnull(anyelement)(\n SFUNC = first_notnull_sfunc,\n STYPE = anyelement\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_array_unique_merge(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_concat_ignorenull,\n FINALFUNC = jsonb_array_unique\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_min(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_least\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_max(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_greatest\n);\n/* looks for a geometry in a stac item first from geometry and falling back to bbox */\nCREATE OR REPLACE FUNCTION stac_geom(value jsonb) RETURNS geometry AS $$\nSELECT\n CASE\n WHEN value ? 'intersects' THEN\n ST_GeomFromGeoJSON(value->>'intersects')\n WHEN value ? 'geometry' THEN\n ST_GeomFromGeoJSON(value->>'geometry')\n WHEN value ? 'bbox' THEN\n pgstac.bbox_geom(value->'bbox')\n ELSE NULL\n END as geometry\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION stac_daterange(\n value jsonb\n) RETURNS tstzrange AS $$\nDECLARE\n props jsonb := value;\n dt timestamptz;\n edt timestamptz;\nBEGIN\n IF props ? 'properties' THEN\n props := props->'properties';\n END IF;\n IF\n props ? 'start_datetime'\n AND props->>'start_datetime' IS NOT NULL\n AND props ? 'end_datetime'\n AND props->>'end_datetime' IS NOT NULL\n THEN\n dt := props->>'start_datetime';\n edt := props->>'end_datetime';\n IF dt > edt THEN\n RAISE EXCEPTION 'start_datetime must be < end_datetime';\n END IF;\n ELSE\n dt := props->>'datetime';\n edt := props->>'datetime';\n END IF;\n IF dt is NULL OR edt IS NULL THEN\n RAISE NOTICE 'DT: %, EDT: %', dt, edt;\n RAISE EXCEPTION 'Either datetime (%) or both start_datetime (%) and end_datetime (%) must be set.', props->>'datetime',props->>'start_datetime',props->>'end_datetime';\n END IF;\n RETURN tstzrange(dt, edt, '[]');\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_datetime(value jsonb) RETURNS timestamptz AS $$\n SELECT lower(stac_daterange(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_end_datetime(value jsonb) RETURNS timestamptz AS $$\n SELECT upper(stac_daterange(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE TABLE IF NOT EXISTS stac_extensions(\n url text PRIMARY KEY,\n content jsonb\n);\nCREATE OR REPLACE FUNCTION collection_base_item(content jsonb) RETURNS jsonb AS $$\n SELECT jsonb_build_object(\n 'type', 'Feature',\n 'stac_version', content->'stac_version',\n 'assets', content->'item_assets',\n 'collection', content->'id'\n );\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE TABLE IF NOT EXISTS collections (\n key bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,\n id text GENERATED ALWAYS AS (content->>'id') STORED UNIQUE,\n content JSONB NOT NULL,\n base_item jsonb GENERATED ALWAYS AS (pgstac.collection_base_item(content)) STORED,\n geometry geometry GENERATED ALWAYS AS (pgstac.collection_geom(content)) STORED,\n datetime timestamptz GENERATED ALWAYS AS (pgstac.collection_datetime(content)) STORED,\n end_datetime timestamptz GENERATED ALWAYS AS (pgstac.collection_enddatetime(content)) STORED,\n private jsonb,\n partition_trunc text CHECK (partition_trunc IN ('year', 'month'))\n);\n\n\n\nCREATE OR REPLACE FUNCTION collection_base_item(cid text) RETURNS jsonb AS $$\n SELECT pgstac.collection_base_item(content) FROM pgstac.collections WHERE id = cid LIMIT 1;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION create_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION update_collection(data jsonb) RETURNS VOID AS $$\nDECLARE\n out collections%ROWTYPE;\nBEGIN\n UPDATE collections SET content=data WHERE id = data->>'id' RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ON CONFLICT (id) DO\n UPDATE\n SET content=EXCLUDED.content\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION delete_collection(_id text) RETURNS VOID AS $$\nDECLARE\n out collections%ROWTYPE;\nBEGIN\n DELETE FROM collections WHERE id = _id RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION get_collection(id text) RETURNS jsonb AS $$\n SELECT content FROM collections\n WHERE id=$1\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION all_collections() RETURNS jsonb AS $$\n SELECT coalesce(jsonb_agg(content), '[]'::jsonb) FROM collections;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION collection_delete_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n collection_base_partition text := concat('_items_', OLD.key);\nBEGIN\n EXECUTE format($q$\n DELETE FROM partition_stats WHERE partition IN (\n SELECT partition FROM partition_sys_meta\n WHERE collection=%L\n );\n DROP TABLE IF EXISTS %I CASCADE;\n $q$,\n OLD.id,\n collection_base_partition\n );\n RETURN OLD;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER collection_delete_trigger BEFORE DELETE ON collections\nFOR EACH ROW EXECUTE FUNCTION collection_delete_trigger_func();\nCREATE OR REPLACE FUNCTION queryable_signature(n text, c text[]) RETURNS text AS $$\n SELECT concat(n, c);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TABLE queryables (\n id bigint GENERATED ALWAYS AS identity PRIMARY KEY,\n name text NOT NULL,\n collection_ids text[], -- used to determine what partitions to create indexes on\n definition jsonb,\n property_path text,\n property_wrapper text,\n property_index_type text\n);\nCREATE INDEX queryables_name_idx ON queryables (name);\nCREATE INDEX queryables_collection_idx ON queryables USING GIN (collection_ids);\nCREATE INDEX queryables_property_wrapper_idx ON queryables (property_wrapper);\n\nCREATE OR REPLACE FUNCTION pgstac.queryables_constraint_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n allcollections text[];\nBEGIN\n RAISE NOTICE 'Making sure that name/collection is unique for queryables %', NEW;\n IF NEW.collection_ids IS NOT NULL THEN\n IF EXISTS (\n SELECT 1\n FROM unnest(NEW.collection_ids) c\n LEFT JOIN\n collections\n ON (collections.id = c)\n WHERE collections.id IS NULL\n ) THEN\n RAISE foreign_key_violation USING MESSAGE = format(\n 'One or more collections in %s do not exist.', NEW.collection_ids\n );\n RETURN NULL;\n END IF;\n END IF;\n IF TG_OP = 'INSERT' THEN\n IF EXISTS (\n SELECT 1 FROM queryables q\n WHERE\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n )\n ) THEN\n RAISE unique_violation USING MESSAGE = format(\n 'There is already a queryable for %s for a collection in %s: %s',\n NEW.name,\n NEW.collection_ids,\n\t\t\t\t(SELECT json_agg(row_to_json(q)) FROM queryables q WHERE\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n ))\n );\n RETURN NULL;\n END IF;\n END IF;\n IF TG_OP = 'UPDATE' THEN\n IF EXISTS (\n SELECT 1 FROM queryables q\n WHERE\n q.id != NEW.id\n AND\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n )\n ) THEN\n RAISE unique_violation\n USING MESSAGE = format(\n 'There is already a queryable for %s for a collection in %s',\n NEW.name,\n NEW.collection_ids\n );\n RETURN NULL;\n END IF;\n END IF;\n\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER queryables_constraint_insert_trigger\nBEFORE INSERT ON queryables\nFOR EACH ROW EXECUTE PROCEDURE queryables_constraint_triggerfunc();\n\nCREATE TRIGGER queryables_constraint_update_trigger\nBEFORE UPDATE ON queryables\nFOR EACH ROW\nWHEN (NEW.name = OLD.name AND NEW.collection_ids IS DISTINCT FROM OLD.collection_ids)\nEXECUTE PROCEDURE queryables_constraint_triggerfunc();\n\n\nCREATE OR REPLACE FUNCTION array_to_path(arr text[]) RETURNS text AS $$\n SELECT string_agg(\n quote_literal(v),\n '->'\n ) FROM unnest(arr) v;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\n\n\nCREATE OR REPLACE FUNCTION queryable(\n IN dotpath text,\n OUT path text,\n OUT expression text,\n OUT wrapper text,\n OUT nulled_wrapper text\n) AS $$\nDECLARE\n q RECORD;\n path_elements text[];\nBEGIN\n IF dotpath IN ('id', 'geometry', 'datetime', 'end_datetime', 'collection') THEN\n path := dotpath;\n expression := dotpath;\n wrapper := NULL;\n RETURN;\n END IF;\n SELECT * INTO q FROM queryables\n WHERE\n name=dotpath\n OR name = 'properties.' || dotpath\n OR name = replace(dotpath, 'properties.', '')\n ;\n IF q.property_wrapper IS NULL THEN\n IF q.definition->>'type' = 'number' THEN\n wrapper := 'to_float';\n nulled_wrapper := wrapper;\n ELSIF q.definition->>'format' = 'date-time' THEN\n wrapper := 'to_tstz';\n nulled_wrapper := wrapper;\n ELSE\n nulled_wrapper := NULL;\n wrapper := 'to_text';\n END IF;\n ELSE\n wrapper := q.property_wrapper;\n nulled_wrapper := wrapper;\n END IF;\n IF q.property_path IS NOT NULL THEN\n path := q.property_path;\n ELSE\n path_elements := string_to_array(dotpath, '.');\n IF path_elements[1] IN ('links', 'assets', 'stac_version', 'stac_extensions') THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSIF path_elements[1] = 'properties' THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSE\n path := format($F$content->'properties'->%s$F$, array_to_path(path_elements));\n END IF;\n END IF;\n expression := format('%I(%s)', wrapper, path);\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\nCREATE OR REPLACE FUNCTION unnest_collection(collection_ids text[] DEFAULT NULL) RETURNS SETOF text AS $$\n DECLARE\n BEGIN\n IF collection_ids IS NULL THEN\n RETURN QUERY SELECT id FROM collections;\n END IF;\n RETURN QUERY SELECT unnest(collection_ids);\n END;\n$$ LANGUAGE PLPGSQL STABLE;\n\nCREATE OR REPLACE FUNCTION normalize_indexdef(def text) RETURNS text AS $$\nDECLARE\nBEGIN\n def := btrim(def, ' \\n\\t');\n\tdef := regexp_replace(def, '^CREATE (UNIQUE )?INDEX ([^ ]* )?ON (ONLY )?([^ ]* )?', '', 'i');\n RETURN def;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE STRICT PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION indexdef(q queryables) RETURNS text AS $$\n DECLARE\n out text;\n BEGIN\n IF q.name = 'id' THEN\n out := 'CREATE UNIQUE INDEX ON %I USING btree (id)';\n ELSIF q.name = 'datetime' THEN\n out := 'CREATE INDEX ON %I USING btree (datetime DESC, end_datetime)';\n ELSIF q.name = 'geometry' THEN\n out := 'CREATE INDEX ON %I USING gist (geometry)';\n ELSE\n out := format($q$CREATE INDEX ON %%I USING %s (%s(((content -> 'properties'::text) -> %L::text)))$q$,\n lower(COALESCE(q.property_index_type, 'BTREE')),\n lower(COALESCE(q.property_wrapper, 'to_text')),\n q.name\n );\n END IF;\n RETURN btrim(out, ' \\n\\t');\n END;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\n\nDROP VIEW IF EXISTS pgstac_indexes;\nCREATE VIEW pgstac_indexes AS\nSELECT\n i.schemaname,\n i.tablename,\n i.indexname,\n regexp_replace(btrim(replace(replace(indexdef, i.indexname, ''),'pgstac.',''),' \\t\\n'), '[ ]+', ' ', 'g') as idx,\n COALESCE(\n (regexp_match(indexdef, '\\(([a-zA-Z]+)\\)'))[1],\n (regexp_match(indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_-]+)''::text'))[1],\n CASE WHEN indexdef ~* '\\(datetime desc, end_datetime\\)' THEN 'datetime' ELSE NULL END\n ) AS field,\n pg_table_size(i.indexname::text) as index_size,\n pg_size_pretty(pg_table_size(i.indexname::text)) as index_size_pretty\nFROM\n pg_indexes i\nWHERE i.schemaname='pgstac' and i.tablename ~ '_items_' AND indexdef !~* ' only ';\n\nDROP VIEW IF EXISTS pgstac_index_stats;\nCREATE VIEW pgstac_indexes_stats AS\nSELECT\n i.schemaname,\n i.tablename,\n i.indexname,\n indexdef,\n COALESCE(\n (regexp_match(indexdef, '\\(([a-zA-Z]+)\\)'))[1],\n (regexp_match(indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_]+)''::text'))[1],\n CASE WHEN indexdef ~* '\\(datetime desc, end_datetime\\)' THEN 'datetime_end_datetime' ELSE NULL END\n ) AS field,\n pg_table_size(i.indexname::text) as index_size,\n pg_size_pretty(pg_table_size(i.indexname::text)) as index_size_pretty,\n n_distinct,\n most_common_vals::text::text[],\n most_common_freqs::text::text[],\n histogram_bounds::text::text[],\n correlation\nFROM\n pg_indexes i\n LEFT JOIN pg_stats s ON (s.tablename = i.indexname)\nWHERE i.schemaname='pgstac' and i.tablename ~ '_items_';\n\nset check_function_bodies to off;\nCREATE OR REPLACE FUNCTION maintain_partition_queries(\n part text DEFAULT 'items',\n dropindexes boolean DEFAULT FALSE,\n rebuildindexes boolean DEFAULT FALSE,\n idxconcurrently boolean DEFAULT FALSE\n) RETURNS SETOF text AS $$\nDECLARE\n rec record;\nBEGIN\n FOR rec IN (\n WITH p AS (\n SELECT\n relid::text as partition,\n replace(replace(\n CASE\n WHEN level = 1 THEN pg_get_expr(c.relpartbound, c.oid)\n ELSE pg_get_expr(parent.relpartbound, parent.oid)\n END,\n 'FOR VALUES IN (''',''), ''')',\n ''\n ) AS collection\n FROM pg_partition_tree('items')\n JOIN pg_class c ON (relid::regclass = c.oid)\n JOIN pg_class parent ON (parentrelid::regclass = parent.oid AND isleaf)\n ), i AS (\n SELECT\n partition,\n indexname,\n regexp_replace(btrim(replace(replace(indexdef, indexname, ''),'pgstac.',''),' \\t\\n'), '[ ]+', ' ', 'g') as iidx,\n COALESCE(\n (regexp_match(indexdef, '\\(([a-zA-Z]+)\\)'))[1],\n (regexp_match(indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_-]+)''::text'))[1],\n CASE WHEN indexdef ~* '\\(datetime desc, end_datetime\\)' THEN 'datetime' ELSE NULL END\n ) AS field\n FROM\n pg_indexes\n JOIN p ON (tablename=partition)\n ), q AS (\n SELECT\n name AS field,\n collection,\n partition,\n format(indexdef(queryables), partition) as qidx\n FROM queryables, unnest_collection(queryables.collection_ids) collection\n JOIN p USING (collection)\n WHERE property_index_type IS NOT NULL OR name IN ('datetime','geometry','id')\n )\n SELECT * FROM i FULL JOIN q USING (field, partition)\n WHERE lower(iidx) IS DISTINCT FROM lower(qidx)\n ) LOOP\n IF rec.iidx IS NULL THEN\n IF idxconcurrently THEN\n RETURN NEXT replace(rec.qidx, 'INDEX', 'INDEX CONCURRENTLY');\n ELSE\n RETURN NEXT rec.qidx;\n END IF;\n ELSIF rec.qidx IS NULL AND dropindexes THEN\n RETURN NEXT format('DROP INDEX IF EXISTS %I;', rec.indexname);\n ELSIF lower(rec.qidx) != lower(rec.iidx) THEN\n IF dropindexes THEN\n RETURN NEXT format('DROP INDEX IF EXISTS %I; %s;', rec.indexname, rec.qidx);\n ELSE\n IF idxconcurrently THEN\n RETURN NEXT replace(rec.qidx, 'INDEX', 'INDEX CONCURRENTLY');\n ELSE\n RETURN NEXT rec.qidx;\n END IF;\n END IF;\n ELSIF rebuildindexes and rec.indexname IS NOT NULL THEN\n IF idxconcurrently THEN\n RETURN NEXT format('REINDEX INDEX CONCURRENTLY %I;', rec.indexname);\n ELSE\n RETURN NEXT format('REINDEX INDEX %I;', rec.indexname);\n END IF;\n END IF;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION maintain_partitions(\n part text DEFAULT 'items',\n dropindexes boolean DEFAULT FALSE,\n rebuildindexes boolean DEFAULT FALSE\n) RETURNS VOID AS $$\n WITH t AS (\n SELECT run_or_queue(q) FROM maintain_partition_queries(part, dropindexes, rebuildindexes) q\n ) SELECT count(*) FROM t;\n$$ LANGUAGE SQL;\n\n\nCREATE OR REPLACE FUNCTION queryables_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\nBEGIN\n PERFORM maintain_partitions();\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER queryables_trigger AFTER INSERT OR UPDATE ON queryables\nFOR EACH STATEMENT EXECUTE PROCEDURE queryables_trigger_func();\n\n\nCREATE OR REPLACE FUNCTION get_queryables(_collection_ids text[] DEFAULT NULL) RETURNS jsonb AS $$\nDECLARE\nBEGIN\n -- Build up queryables if the input contains valid collection ids or is empty\n IF EXISTS (\n SELECT 1 FROM collections\n WHERE\n _collection_ids IS NULL\n OR cardinality(_collection_ids) = 0\n OR id = ANY(_collection_ids)\n )\n THEN\n RETURN (\n WITH base AS (\n SELECT\n unnest(collection_ids) as collection_id,\n name,\n coalesce(definition, '{\"type\":\"string\"}'::jsonb) as definition\n FROM queryables\n WHERE\n _collection_ids IS NULL OR\n _collection_ids = '{}'::text[] OR\n _collection_ids && collection_ids\n UNION ALL\n SELECT null, name, coalesce(definition, '{\"type\":\"string\"}'::jsonb) as definition\n FROM queryables WHERE collection_ids IS NULL OR collection_ids = '{}'::text[]\n ), g AS (\n SELECT\n name,\n first_notnull(definition) as definition,\n jsonb_array_unique_merge(definition->'enum') as enum,\n jsonb_min(definition->'minimum') as minimum,\n jsonb_min(definition->'maxiumn') as maximum\n FROM base\n GROUP BY 1\n )\n SELECT\n jsonb_build_object(\n '$schema', 'http://json-schema.org/draft-07/schema#',\n '$id', '',\n 'type', 'object',\n 'title', 'STAC Queryables.',\n 'properties', jsonb_object_agg(\n name,\n definition\n ||\n jsonb_strip_nulls(jsonb_build_object(\n 'enum', enum,\n 'minimum', minimum,\n 'maximum', maximum\n ))\n )\n )\n FROM g\n );\n ELSE\n RETURN NULL;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\nCREATE OR REPLACE FUNCTION get_queryables(_collection text DEFAULT NULL) RETURNS jsonb AS $$\n SELECT\n CASE\n WHEN _collection IS NULL THEN get_queryables(NULL::text[])\n ELSE get_queryables(ARRAY[_collection])\n END\n ;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_queryables() RETURNS jsonb AS $$\n SELECT get_queryables(NULL::text[]);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION schema_qualify_refs(url text, j jsonb) returns jsonb as $$\n SELECT regexp_replace(j::text, '\"\\$ref\": \"#', concat('\"$ref\": \"', url, '#'), 'g')::jsonb;\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE VIEW stac_extension_queryables AS\nSELECT DISTINCT key as name, schema_qualify_refs(e.url, j.value) as definition FROM stac_extensions e, jsonb_each(e.content->'definitions'->'fields'->'properties') j;\n\n\nCREATE OR REPLACE FUNCTION missing_queryables(_collection text, _tablesample float DEFAULT 5, minrows float DEFAULT 10) RETURNS TABLE(collection text, name text, definition jsonb, property_wrapper text) AS $$\nDECLARE\n q text;\n _partition text;\n explain_json json;\n psize float;\n estrows float;\nBEGIN\n SELECT format('_items_%s', key) INTO _partition FROM collections WHERE id=_collection;\n\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM %I;', _partition)\n INTO explain_json;\n psize := explain_json->0->'Plan'->'Plan Rows';\n estrows := _tablesample * .01 * psize;\n IF estrows < minrows THEN\n _tablesample := least(100,greatest(_tablesample, (estrows / psize) / 100));\n RAISE NOTICE '%', (psize / estrows) / 100;\n END IF;\n RAISE NOTICE 'Using tablesample % to find missing queryables from % % that has ~% rows estrows: %', _tablesample, _collection, _partition, psize, estrows;\n\n q := format(\n $q$\n WITH q AS (\n SELECT * FROM queryables\n WHERE\n collection_ids IS NULL\n OR %L = ANY(collection_ids)\n ), t AS (\n SELECT\n content->'properties' AS properties\n FROM\n %I\n TABLESAMPLE SYSTEM(%L)\n ), p AS (\n SELECT DISTINCT ON (key)\n key,\n value,\n s.definition\n FROM t\n JOIN LATERAL jsonb_each(properties) ON TRUE\n LEFT JOIN q ON (q.name=key)\n LEFT JOIN stac_extension_queryables s ON (s.name=key)\n WHERE q.definition IS NULL\n )\n SELECT\n %L,\n key,\n COALESCE(definition, jsonb_build_object('type',jsonb_typeof(value))) as definition,\n CASE\n WHEN definition->>'type' = 'integer' THEN 'to_int'\n WHEN COALESCE(definition->>'type', jsonb_typeof(value)) = 'number' THEN 'to_float'\n WHEN COALESCE(definition->>'type', jsonb_typeof(value)) = 'array' THEN 'to_text_array'\n ELSE 'to_text'\n END\n FROM p;\n $q$,\n _collection,\n _partition,\n _tablesample,\n _collection\n );\n RETURN QUERY EXECUTE q;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION missing_queryables(_tablesample float DEFAULT 5) RETURNS TABLE(collection_ids text[], name text, definition jsonb, property_wrapper text) AS $$\n SELECT\n array_agg(collection),\n name,\n definition,\n property_wrapper\n FROM\n collections\n JOIN LATERAL\n missing_queryables(id, _tablesample) c\n ON TRUE\n GROUP BY\n 2,3,4\n ORDER BY 2,1\n ;\n$$ LANGUAGE SQL;\nCREATE OR REPLACE FUNCTION parse_dtrange(\n _indate jsonb,\n relative_base timestamptz DEFAULT date_trunc('hour', CURRENT_TIMESTAMP)\n) RETURNS tstzrange AS $$\nDECLARE\n timestrs text[];\n s timestamptz;\n e timestamptz;\nBEGIN\n timestrs :=\n CASE\n WHEN _indate ? 'timestamp' THEN\n ARRAY[_indate->>'timestamp']\n WHEN _indate ? 'interval' THEN\n to_text_array(_indate->'interval')\n WHEN jsonb_typeof(_indate) = 'array' THEN\n to_text_array(_indate)\n ELSE\n regexp_split_to_array(\n _indate->>0,\n '/'\n )\n END;\n RAISE NOTICE 'TIMESTRS %', timestrs;\n IF cardinality(timestrs) = 1 THEN\n IF timestrs[1] ILIKE 'P%' THEN\n RETURN tstzrange(relative_base - upper(timestrs[1])::interval, relative_base, '[)');\n END IF;\n s := timestrs[1]::timestamptz;\n RETURN tstzrange(s, s, '[]');\n END IF;\n\n IF cardinality(timestrs) != 2 THEN\n RAISE EXCEPTION 'Timestamp cannot have more than 2 values';\n END IF;\n\n IF timestrs[1] = '..' OR timestrs[1] = '' THEN\n s := '-infinity'::timestamptz;\n e := timestrs[2]::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] = '..' OR timestrs[2] = '' THEN\n s := timestrs[1]::timestamptz;\n e := 'infinity'::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[1] ILIKE 'P%' AND timestrs[2] NOT ILIKE 'P%' THEN\n e := timestrs[2]::timestamptz;\n s := e - upper(timestrs[1])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] ILIKE 'P%' AND timestrs[1] NOT ILIKE 'P%' THEN\n s := timestrs[1]::timestamptz;\n e := s + upper(timestrs[2])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n s := timestrs[1]::timestamptz;\n e := timestrs[2]::timestamptz;\n\n RETURN tstzrange(s,e,'[)');\n\n RETURN NULL;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT PARALLEL SAFE SET TIME ZONE 'UTC';\n\nCREATE OR REPLACE FUNCTION parse_dtrange(\n _indate text,\n relative_base timestamptz DEFAULT CURRENT_TIMESTAMP\n) RETURNS tstzrange AS $$\n SELECT parse_dtrange(to_jsonb(_indate), relative_base);\n$$ LANGUAGE SQL STABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION temporal_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\n ll text := 'datetime';\n lh text := 'end_datetime';\n rrange tstzrange;\n rl text;\n rh text;\n outq text;\nBEGIN\n rrange := parse_dtrange(args->1);\n RAISE NOTICE 'Constructing temporal query OP: %, ARGS: %, RRANGE: %', op, args, rrange;\n op := lower(op);\n rl := format('%L::timestamptz', lower(rrange));\n rh := format('%L::timestamptz', upper(rrange));\n outq := CASE op\n WHEN 't_before' THEN 'lh < rl'\n WHEN 't_after' THEN 'll > rh'\n WHEN 't_meets' THEN 'lh = rl'\n WHEN 't_metby' THEN 'll = rh'\n WHEN 't_overlaps' THEN 'll < rl AND rl < lh < rh'\n WHEN 't_overlappedby' THEN 'rl < ll < rh AND lh > rh'\n WHEN 't_starts' THEN 'll = rl AND lh < rh'\n WHEN 't_startedby' THEN 'll = rl AND lh > rh'\n WHEN 't_during' THEN 'll > rl AND lh < rh'\n WHEN 't_contains' THEN 'll < rl AND lh > rh'\n WHEN 't_finishes' THEN 'll > rl AND lh = rh'\n WHEN 't_finishedby' THEN 'll < rl AND lh = rh'\n WHEN 't_equals' THEN 'll = rl AND lh = rh'\n WHEN 't_disjoint' THEN 'NOT (ll <= rh AND lh >= rl)'\n WHEN 't_intersects' THEN 'll <= rh AND lh >= rl'\n WHEN 'anyinteracts' THEN 'll <= rh AND lh >= rl'\n END;\n outq := regexp_replace(outq, '\\mll\\M', ll);\n outq := regexp_replace(outq, '\\mlh\\M', lh);\n outq := regexp_replace(outq, '\\mrl\\M', rl);\n outq := regexp_replace(outq, '\\mrh\\M', rh);\n outq := format('(%s)', outq);\n RETURN outq;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\n\n\nCREATE OR REPLACE FUNCTION spatial_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\n geom text;\n j jsonb := args->1;\nBEGIN\n op := lower(op);\n RAISE NOTICE 'Constructing spatial query OP: %, ARGS: %', op, args;\n IF op NOT IN ('s_equals','s_disjoint','s_touches','s_within','s_overlaps','s_crosses','s_intersects','intersects','s_contains') THEN\n RAISE EXCEPTION 'Spatial Operator % Not Supported', op;\n END IF;\n op := regexp_replace(op, '^s_', 'st_');\n IF op = 'intersects' THEN\n op := 'st_intersects';\n END IF;\n -- Convert geometry to WKB string\n IF j ? 'type' AND j ? 'coordinates' THEN\n geom := st_geomfromgeojson(j)::text;\n ELSIF jsonb_typeof(j) = 'array' THEN\n geom := bbox_geom(j)::text;\n END IF;\n\n RETURN format('%s(geometry, %L::geometry)', op, geom);\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION query_to_cql2(q jsonb) RETURNS jsonb AS $$\n-- Translates anything passed in through the deprecated \"query\" into equivalent CQL2\nWITH t AS (\n SELECT key as property, value as ops\n FROM jsonb_each(q)\n), t2 AS (\n SELECT property, (jsonb_each(ops)).*\n FROM t WHERE jsonb_typeof(ops) = 'object'\n UNION ALL\n SELECT property, 'eq', ops\n FROM t WHERE jsonb_typeof(ops) != 'object'\n)\nSELECT\n jsonb_strip_nulls(jsonb_build_object(\n 'op', 'and',\n 'args', jsonb_agg(\n jsonb_build_object(\n 'op', key,\n 'args', jsonb_build_array(\n jsonb_build_object('property',property),\n value\n )\n )\n )\n )\n) as qcql FROM t2\n;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\nCREATE OR REPLACE FUNCTION cql1_to_cql2(j jsonb) RETURNS jsonb AS $$\nDECLARE\n args jsonb;\n ret jsonb;\nBEGIN\n RAISE NOTICE 'CQL1_TO_CQL2: %', j;\n IF j ? 'filter' THEN\n RETURN cql1_to_cql2(j->'filter');\n END IF;\n IF j ? 'property' THEN\n RETURN j;\n END IF;\n IF jsonb_typeof(j) = 'array' THEN\n SELECT jsonb_agg(cql1_to_cql2(el)) INTO args FROM jsonb_array_elements(j) el;\n RETURN args;\n END IF;\n IF jsonb_typeof(j) = 'number' THEN\n RETURN j;\n END IF;\n IF jsonb_typeof(j) = 'string' THEN\n RETURN j;\n END IF;\n\n IF jsonb_typeof(j) = 'object' THEN\n SELECT jsonb_build_object(\n 'op', key,\n 'args', cql1_to_cql2(value)\n ) INTO ret\n FROM jsonb_each(j)\n WHERE j IS NOT NULL;\n RETURN ret;\n END IF;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE STRICT;\n\nCREATE TABLE cql2_ops (\n op text PRIMARY KEY,\n template text,\n types text[]\n);\nINSERT INTO cql2_ops (op, template, types) VALUES\n ('eq', '%s = %s', NULL),\n ('neq', '%s != %s', NULL),\n ('ne', '%s != %s', NULL),\n ('!=', '%s != %s', NULL),\n ('<>', '%s != %s', NULL),\n ('lt', '%s < %s', NULL),\n ('lte', '%s <= %s', NULL),\n ('gt', '%s > %s', NULL),\n ('gte', '%s >= %s', NULL),\n ('le', '%s <= %s', NULL),\n ('ge', '%s >= %s', NULL),\n ('=', '%s = %s', NULL),\n ('<', '%s < %s', NULL),\n ('<=', '%s <= %s', NULL),\n ('>', '%s > %s', NULL),\n ('>=', '%s >= %s', NULL),\n ('like', '%s LIKE %s', NULL),\n ('ilike', '%s ILIKE %s', NULL),\n ('+', '%s + %s', NULL),\n ('-', '%s - %s', NULL),\n ('*', '%s * %s', NULL),\n ('/', '%s / %s', NULL),\n ('not', 'NOT (%s)', NULL),\n ('between', '%s BETWEEN %s AND %s', NULL),\n ('isnull', '%s IS NULL', NULL),\n ('upper', 'upper(%s)', NULL),\n ('lower', 'lower(%s)', NULL)\nON CONFLICT (op) DO UPDATE\n SET\n template = EXCLUDED.template\n;\n\n\nCREATE OR REPLACE FUNCTION cql2_query(j jsonb, wrapper text DEFAULT NULL) RETURNS text AS $$\n#variable_conflict use_variable\nDECLARE\n args jsonb := j->'args';\n arg jsonb;\n op text := lower(j->>'op');\n cql2op RECORD;\n literal text;\n _wrapper text;\n leftarg text;\n rightarg text;\nBEGIN\n IF j IS NULL OR (op IS NOT NULL AND args IS NULL) THEN\n RETURN NULL;\n END IF;\n RAISE NOTICE 'CQL2_QUERY: %', j;\n IF j ? 'filter' THEN\n RETURN cql2_query(j->'filter');\n END IF;\n\n IF j ? 'upper' THEN\n RETURN cql2_query(jsonb_build_object('op', 'upper', 'args', j->'upper'));\n END IF;\n\n IF j ? 'lower' THEN\n RETURN cql2_query(jsonb_build_object('op', 'lower', 'args', j->'lower'));\n END IF;\n\n -- Temporal Query\n IF op ilike 't_%' or op = 'anyinteracts' THEN\n RETURN temporal_op_query(op, args);\n END IF;\n\n -- If property is a timestamp convert it to text to use with\n -- general operators\n IF j ? 'timestamp' THEN\n RETURN format('%L::timestamptz', to_tstz(j->'timestamp'));\n END IF;\n IF j ? 'interval' THEN\n RAISE EXCEPTION 'Please use temporal operators when using intervals.';\n RETURN NONE;\n END IF;\n\n -- Spatial Query\n IF op ilike 's_%' or op = 'intersects' THEN\n RETURN spatial_op_query(op, args);\n END IF;\n\n IF op IN ('a_equals','a_contains','a_contained_by','a_overlaps') THEN\n IF args->0 ? 'property' THEN\n leftarg := format('to_text_array(%s)', (queryable(args->0->>'property')).path);\n END IF;\n IF args->1 ? 'property' THEN\n rightarg := format('to_text_array(%s)', (queryable(args->1->>'property')).path);\n END IF;\n RETURN FORMAT(\n '%s %s %s',\n COALESCE(leftarg, quote_literal(to_text_array(args->0))),\n CASE op\n WHEN 'a_equals' THEN '='\n WHEN 'a_contains' THEN '@>'\n WHEN 'a_contained_by' THEN '<@'\n WHEN 'a_overlaps' THEN '&&'\n END,\n COALESCE(rightarg, quote_literal(to_text_array(args->1)))\n );\n END IF;\n\n IF op = 'in' THEN\n RAISE NOTICE 'IN : % % %', args, jsonb_build_array(args->0), args->1;\n args := jsonb_build_array(args->0) || (args->1);\n RAISE NOTICE 'IN2 : %', args;\n END IF;\n\n\n\n IF op = 'between' THEN\n args = jsonb_build_array(\n args->0,\n args->1->0,\n args->1->1\n );\n END IF;\n\n -- Make sure that args is an array and run cql2_query on\n -- each element of the array\n RAISE NOTICE 'ARGS PRE: %', args;\n IF j ? 'args' THEN\n IF jsonb_typeof(args) != 'array' THEN\n args := jsonb_build_array(args);\n END IF;\n\n IF jsonb_path_exists(args, '$[*] ? (@.property == \"id\" || @.property == \"datetime\" || @.property == \"end_datetime\" || @.property == \"collection\")') THEN\n wrapper := NULL;\n ELSE\n -- if any of the arguments are a property, try to get the property_wrapper\n FOR arg IN SELECT jsonb_path_query(args, '$[*] ? (@.property != null)') LOOP\n RAISE NOTICE 'Arg: %', arg;\n wrapper := (queryable(arg->>'property')).nulled_wrapper;\n RAISE NOTICE 'Property: %, Wrapper: %', arg, wrapper;\n IF wrapper IS NOT NULL THEN\n EXIT;\n END IF;\n END LOOP;\n\n -- if the property was not in queryables, see if any args were numbers\n IF\n wrapper IS NULL\n AND jsonb_path_exists(args, '$[*] ? (@.type()==\"number\")')\n THEN\n wrapper := 'to_float';\n END IF;\n wrapper := coalesce(wrapper, 'to_text');\n END IF;\n\n SELECT jsonb_agg(cql2_query(a, wrapper))\n INTO args\n FROM jsonb_array_elements(args) a;\n END IF;\n RAISE NOTICE 'ARGS: %', args;\n\n IF op IN ('and', 'or') THEN\n RETURN\n format(\n '(%s)',\n array_to_string(to_text_array(args), format(' %s ', upper(op)))\n );\n END IF;\n\n IF op = 'in' THEN\n RAISE NOTICE 'IN -- % %', args->0, to_text(args->0);\n RETURN format(\n '%s IN (%s)',\n to_text(args->0),\n array_to_string((to_text_array(args))[2:], ',')\n );\n END IF;\n\n -- Look up template from cql2_ops\n IF j ? 'op' THEN\n SELECT * INTO cql2op FROM cql2_ops WHERE cql2_ops.op ilike op;\n IF FOUND THEN\n -- If specific index set in queryables for a property cast other arguments to that type\n\n RETURN format(\n cql2op.template,\n VARIADIC (to_text_array(args))\n );\n ELSE\n RAISE EXCEPTION 'Operator % Not Supported.', op;\n END IF;\n END IF;\n\n\n IF wrapper IS NOT NULL THEN\n RAISE NOTICE 'Wrapping % with %', j, wrapper;\n IF j ? 'property' THEN\n RETURN format('%I(%s)', wrapper, (queryable(j->>'property')).path);\n ELSE\n RETURN format('%I(%L)', wrapper, j);\n END IF;\n ELSIF j ? 'property' THEN\n RETURN quote_ident(j->>'property');\n END IF;\n\n RETURN quote_literal(to_text(j));\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION paging_dtrange(\n j jsonb\n) RETURNS tstzrange AS $$\nDECLARE\n op text;\n filter jsonb := j->'filter';\n dtrange tstzrange := tstzrange('-infinity'::timestamptz,'infinity'::timestamptz);\n sdate timestamptz := '-infinity'::timestamptz;\n edate timestamptz := 'infinity'::timestamptz;\n jpitem jsonb;\nBEGIN\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n END IF;\n IF NOT (filter @? '$.**.op ? (@ == \"or\" || @ == \"not\")') THEN\n FOR jpitem IN SELECT j FROM jsonb_path_query(filter,'strict $.** ? (@.args[*].property == \"datetime\")'::jsonpath) j LOOP\n op := lower(jpitem->>'op');\n dtrange := parse_dtrange(jpitem->'args'->1);\n IF op IN ('<=', 'lt', 'lte', '<', 'le', 't_before') THEN\n sdate := greatest(sdate,'-infinity');\n edate := least(edate, upper(dtrange));\n ELSIF op IN ('>=', '>', 'gt', 'gte', 'ge', 't_after') THEN\n edate := least(edate, 'infinity');\n sdate := greatest(sdate, lower(dtrange));\n ELSIF op IN ('=', 'eq') THEN\n edate := least(edate, upper(dtrange));\n sdate := greatest(sdate, lower(dtrange));\n END IF;\n RAISE NOTICE '2 OP: %, ARGS: %, DTRANGE: %, SDATE: %, EDATE: %', op, jpitem->'args'->1, dtrange, sdate, edate;\n END LOOP;\n END IF;\n IF sdate > edate THEN\n RETURN 'empty'::tstzrange;\n END IF;\n RETURN tstzrange(sdate,edate, '[]');\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT SET TIME ZONE 'UTC';\n\nCREATE OR REPLACE FUNCTION paging_collections(\n IN j jsonb\n) RETURNS text[] AS $$\nDECLARE\n filter jsonb := j->'filter';\n jpitem jsonb;\n op text;\n args jsonb;\n arg jsonb;\n collections text[];\nBEGIN\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n END IF;\n IF NOT (filter @? '$.**.op ? (@ == \"or\" || @ == \"not\")') THEN\n FOR jpitem IN SELECT j FROM jsonb_path_query(filter,'strict $.** ? (@.args[*].property == \"collection\")'::jsonpath) j LOOP\n RAISE NOTICE 'JPITEM: %', jpitem;\n op := jpitem->>'op';\n args := jpitem->'args';\n IF op IN ('=', 'eq', 'in') THEN\n FOR arg IN SELECT a FROM jsonb_array_elements(args) a LOOP\n IF jsonb_typeof(arg) IN ('string', 'array') THEN\n RAISE NOTICE 'arg: %, collections: %', arg, collections;\n IF collections IS NULL OR collections = '{}'::text[] THEN\n collections := to_text_array(arg);\n ELSE\n collections := array_intersection(collections, to_text_array(arg));\n END IF;\n END IF;\n END LOOP;\n END IF;\n END LOOP;\n END IF;\n IF collections = '{}'::text[] THEN\n RETURN NULL;\n END IF;\n RETURN collections;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\nCREATE TABLE items (\n id text NOT NULL,\n geometry geometry NOT NULL,\n collection text NOT NULL,\n datetime timestamptz NOT NULL,\n end_datetime timestamptz NOT NULL,\n content JSONB NOT NULL,\n private jsonb\n)\nPARTITION BY LIST (collection)\n;\n\nCREATE INDEX \"datetime_idx\" ON items USING BTREE (datetime DESC, end_datetime ASC);\nCREATE INDEX \"geometry_idx\" ON items USING GIST (geometry);\n\nCREATE STATISTICS datetime_stats (dependencies) on datetime, end_datetime from items;\n\nALTER TABLE items ADD CONSTRAINT items_collections_fk FOREIGN KEY (collection) REFERENCES collections(id) ON DELETE CASCADE DEFERRABLE;\n\nCREATE OR REPLACE FUNCTION partition_after_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p text;\n t timestamptz := clock_timestamp();\nBEGIN\n RAISE NOTICE 'Updating partition stats %', t;\n FOR p IN SELECT DISTINCT partition\n FROM newdata n JOIN partition_sys_meta p\n ON (n.collection=p.collection AND n.datetime <@ p.partition_dtrange)\n LOOP\n PERFORM run_or_queue(format('SELECT update_partition_stats(%L, %L);', p, true));\n END LOOP;\n IF TG_OP IN ('DELETE','UPDATE') THEN\n DELETE FROM format_item_cache c USING newdata n WHERE c.collection = n.collection AND c.id = n.id;\n END IF;\n RAISE NOTICE 't: % %', t, clock_timestamp() - t;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE TRIGGER items_after_insert_trigger\nAFTER INSERT ON items\nREFERENCING NEW TABLE AS newdata\nFOR EACH STATEMENT\nEXECUTE FUNCTION partition_after_triggerfunc();\n\nCREATE TRIGGER items_after_update_trigger\nAFTER DELETE ON items\nREFERENCING OLD TABLE AS newdata\nFOR EACH STATEMENT\nEXECUTE FUNCTION partition_after_triggerfunc();\n\nCREATE TRIGGER items_after_delete_trigger\nAFTER UPDATE ON items\nREFERENCING NEW TABLE AS newdata\nFOR EACH STATEMENT\nEXECUTE FUNCTION partition_after_triggerfunc();\n\n\nCREATE OR REPLACE FUNCTION content_slim(_item jsonb) RETURNS jsonb AS $$\n SELECT strip_jsonb(_item - '{id,geometry,collection,type}'::text[], collection_base_item(_item->>'collection')) - '{id,geometry,collection,type}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_dehydrate(content jsonb) RETURNS items AS $$\n SELECT\n content->>'id' as id,\n stac_geom(content) as geometry,\n content->>'collection' as collection,\n stac_datetime(content) as datetime,\n stac_end_datetime(content) as end_datetime,\n content_slim(content) as content,\n null::jsonb as private\n ;\n$$ LANGUAGE SQL STABLE;\n\nCREATE OR REPLACE FUNCTION include_field(f text, fields jsonb DEFAULT '{}'::jsonb) RETURNS boolean AS $$\nDECLARE\n includes jsonb := fields->'include';\n excludes jsonb := fields->'exclude';\nBEGIN\n IF f IS NULL THEN\n RETURN NULL;\n END IF;\n\n\n IF\n jsonb_typeof(excludes) = 'array'\n AND jsonb_array_length(excludes)>0\n AND excludes ? f\n THEN\n RETURN FALSE;\n END IF;\n\n IF\n (\n jsonb_typeof(includes) = 'array'\n AND jsonb_array_length(includes) > 0\n AND includes ? f\n ) OR\n (\n includes IS NULL\n OR jsonb_typeof(includes) = 'null'\n OR jsonb_array_length(includes) = 0\n )\n THEN\n RETURN TRUE;\n END IF;\n\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nDROP FUNCTION IF EXISTS content_hydrate(jsonb, jsonb, jsonb);\nCREATE OR REPLACE FUNCTION content_hydrate(\n _item jsonb,\n _base_item jsonb,\n fields jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\n SELECT merge_jsonb(\n jsonb_fields(_item, fields),\n jsonb_fields(_base_item, fields)\n );\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION content_hydrate(_item items, _collection collections, fields jsonb DEFAULT '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\n content jsonb;\n base_item jsonb := _collection.base_item;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry, 20)::jsonb;\n END IF;\n output := content_hydrate(\n jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'collection', _item.collection,\n 'type', 'Feature'\n ) || _item.content,\n _collection.base_item,\n fields\n );\n\n RETURN output;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_nonhydrated(\n _item items,\n fields jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry, 20)::jsonb;\n END IF;\n output := jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'collection', _item.collection,\n 'type', 'Feature'\n ) || _item.content;\n RETURN output;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_hydrate(_item items, fields jsonb DEFAULT '{}'::jsonb) RETURNS jsonb AS $$\n SELECT content_hydrate(\n _item,\n (SELECT c FROM collections c WHERE id=_item.collection LIMIT 1),\n fields\n );\n$$ LANGUAGE SQL STABLE;\n\n\nCREATE UNLOGGED TABLE items_staging (\n content JSONB NOT NULL\n);\nCREATE UNLOGGED TABLE items_staging_ignore (\n content JSONB NOT NULL\n);\nCREATE UNLOGGED TABLE items_staging_upsert (\n content JSONB NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION items_staging_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p record;\n _partitions text[];\n part text;\n ts timestamptz := clock_timestamp();\nBEGIN\n RAISE NOTICE 'Creating Partitions. %', clock_timestamp() - ts;\n\n FOR part IN WITH t AS (\n SELECT\n n.content->>'collection' as collection,\n stac_daterange(n.content->'properties') as dtr,\n partition_trunc\n FROM newdata n JOIN collections ON (n.content->>'collection'=collections.id)\n ), p AS (\n SELECT\n collection,\n COALESCE(date_trunc(partition_trunc::text, lower(dtr)),'-infinity') as d,\n tstzrange(min(lower(dtr)),max(lower(dtr)),'[]') as dtrange,\n tstzrange(min(upper(dtr)),max(upper(dtr)),'[]') as edtrange\n FROM t\n GROUP BY 1,2\n ) SELECT check_partition(collection, dtrange, edtrange) FROM p LOOP\n RAISE NOTICE 'Partition %', part;\n END LOOP;\n\n RAISE NOTICE 'Doing the insert. %', clock_timestamp() - ts;\n IF TG_TABLE_NAME = 'items_staging' THEN\n INSERT INTO items\n SELECT\n (content_dehydrate(content)).*\n FROM newdata;\n RAISE NOTICE 'Doing the delete. %', clock_timestamp() - ts;\n DELETE FROM items_staging;\n ELSIF TG_TABLE_NAME = 'items_staging_ignore' THEN\n INSERT INTO items\n SELECT\n (content_dehydrate(content)).*\n FROM newdata\n ON CONFLICT DO NOTHING;\n RAISE NOTICE 'Doing the delete. %', clock_timestamp() - ts;\n DELETE FROM items_staging_ignore;\n ELSIF TG_TABLE_NAME = 'items_staging_upsert' THEN\n WITH staging_formatted AS (\n SELECT (content_dehydrate(content)).* FROM newdata\n ), deletes AS (\n DELETE FROM items i USING staging_formatted s\n WHERE\n i.id = s.id\n AND i.collection = s.collection\n AND i IS DISTINCT FROM s\n RETURNING i.id, i.collection\n )\n INSERT INTO items\n SELECT s.* FROM\n staging_formatted s\n ON CONFLICT DO NOTHING;\n RAISE NOTICE 'Doing the delete. %', clock_timestamp() - ts;\n DELETE FROM items_staging_upsert;\n END IF;\n RAISE NOTICE 'Done. %', clock_timestamp() - ts;\n\n RETURN NULL;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER items_staging_insert_trigger AFTER INSERT ON items_staging REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\nCREATE TRIGGER items_staging_insert_ignore_trigger AFTER INSERT ON items_staging_ignore REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\nCREATE TRIGGER items_staging_insert_upsert_trigger AFTER INSERT ON items_staging_upsert REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\n\nCREATE OR REPLACE FUNCTION item_by_id(_id text, _collection text DEFAULT NULL) RETURNS items AS\n$$\nDECLARE\n i items%ROWTYPE;\nBEGIN\n SELECT * INTO i FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection) LIMIT 1;\n RETURN i;\nEND;\n$$ LANGUAGE PLPGSQL STABLE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION get_item(_id text, _collection text DEFAULT NULL) RETURNS jsonb AS $$\n SELECT content_hydrate(items) FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection);\n$$ LANGUAGE SQL STABLE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION delete_item(_id text, _collection text DEFAULT NULL) RETURNS VOID AS $$\nDECLARE\nout items%ROWTYPE;\nBEGIN\n DELETE FROM items WHERE id = _id AND (_collection IS NULL OR collection=_collection) RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n--/*\nCREATE OR REPLACE FUNCTION create_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION update_item(content jsonb) RETURNS VOID AS $$\nDECLARE\n old items %ROWTYPE;\n out items%ROWTYPE;\nBEGIN\n PERFORM delete_item(content->>'id', content->>'collection');\n PERFORM create_item(content);\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION create_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION collection_bbox(id text) RETURNS jsonb AS $$\n SELECT (replace(replace(replace(st_extent(geometry)::text,'BOX(','[['),')',']]'),' ',','))::jsonb\n FROM items WHERE collection=$1;\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION collection_temporal_extent(id text) RETURNS jsonb AS $$\n SELECT to_jsonb(array[array[min(datetime)::text, max(datetime)::text]])\n FROM items WHERE collection=$1;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection_extents() RETURNS VOID AS $$\nUPDATE collections SET\n content = content ||\n jsonb_build_object(\n 'extent', jsonb_build_object(\n 'spatial', jsonb_build_object(\n 'bbox', collection_bbox(collections.id)\n ),\n 'temporal', jsonb_build_object(\n 'interval', collection_temporal_extent(collections.id)\n )\n )\n )\n;\n$$ LANGUAGE SQL;\nCREATE TABLE partition_stats (\n partition text PRIMARY KEY,\n dtrange tstzrange,\n edtrange tstzrange,\n spatial geometry,\n last_updated timestamptz,\n keys text[]\n) WITH (FILLFACTOR=90);\n\nCREATE INDEX partitions_range_idx ON partition_stats USING GIST(dtrange);\n\n\nCREATE OR REPLACE FUNCTION constraint_tstzrange(expr text) RETURNS tstzrange AS $$\n WITH t AS (\n SELECT regexp_matches(\n expr,\n E'\\\\(''\\([0-9 :+-]*\\)''\\\\).*\\\\(''\\([0-9 :+-]*\\)''\\\\)'\n ) AS m\n ) SELECT tstzrange(m[1]::timestamptz, m[2]::timestamptz) FROM t\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE STRICT;\n\nCREATE OR REPLACE FUNCTION dt_constraint(coid oid, OUT dt tstzrange, OUT edt tstzrange) RETURNS RECORD AS $$\nDECLARE\n expr text := pg_get_constraintdef(coid);\n matches timestamptz[];\nBEGIN\n IF expr LIKE '%NULL%' THEN\n dt := tstzrange(null::timestamptz, null::timestamptz);\n edt := tstzrange(null::timestamptz, null::timestamptz);\n RETURN;\n END IF;\n WITH f AS (SELECT (regexp_matches(expr, E'([0-9]{4}-[0-1][0-9]-[0-3][0-9] [0-2][0-9]:[0-5][0-9]:[0-5][0-9]\\.?[0-9]*)', 'g'))[1] f)\n SELECT array_agg(f::timestamptz) INTO matches FROM f;\n IF cardinality(matches) = 4 THEN\n dt := tstzrange(matches[1], matches[2],'[]');\n edt := tstzrange(matches[3], matches[4], '[]');\n RETURN;\n ELSIF cardinality(matches) = 2 THEN\n edt := tstzrange(matches[1], matches[2],'[]');\n RETURN;\n END IF;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\nCREATE OR REPLACE VIEW partition_sys_meta AS\nSELECT\n relid::text as partition,\n replace(replace(CASE WHEN level = 1 THEN pg_get_expr(c.relpartbound, c.oid)\n ELSE pg_get_expr(parent.relpartbound, parent.oid)\n END, 'FOR VALUES IN (''',''), ''')','') AS collection,\n level,\n c.reltuples,\n c.relhastriggers,\n COALESCE(constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as partition_dtrange,\n COALESCE((dt_constraint(edt.oid)).dt, constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as constraint_dtrange,\n COALESCE((dt_constraint(edt.oid)).edt, tstzrange('-infinity', 'infinity','[]')) as constraint_edtrange\nFROM\n pg_partition_tree('items')\n JOIN pg_class c ON (relid::regclass = c.oid)\n JOIN pg_class parent ON (parentrelid::regclass = parent.oid AND isleaf)\n LEFT JOIN pg_constraint edt ON (conrelid=c.oid AND contype='c')\nWHERE isleaf\n;\n\nCREATE VIEW partitions_view AS\nSELECT\n relid::text as partition,\n replace(replace(CASE WHEN level = 1 THEN pg_get_expr(c.relpartbound, c.oid)\n ELSE pg_get_expr(parent.relpartbound, parent.oid)\n END, 'FOR VALUES IN (''',''), ''')','') AS collection,\n level,\n c.reltuples,\n c.relhastriggers,\n COALESCE(constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as partition_dtrange,\n COALESCE((dt_constraint(edt.oid)).dt, constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as constraint_dtrange,\n COALESCE((dt_constraint(edt.oid)).edt, tstzrange('-infinity', 'infinity','[]')) as constraint_edtrange,\n dtrange,\n edtrange,\n spatial,\n last_updated\nFROM\n pg_partition_tree('items')\n JOIN pg_class c ON (relid::regclass = c.oid)\n JOIN pg_class parent ON (parentrelid::regclass = parent.oid AND isleaf)\n LEFT JOIN pg_constraint edt ON (conrelid=c.oid AND contype='c')\n LEFT JOIN partition_stats ON (relid::text=partition)\nWHERE isleaf\n;\n\nCREATE MATERIALIZED VIEW partitions AS\nSELECT * FROM partitions_view;\nCREATE UNIQUE INDEX ON partitions (partition);\n\nCREATE MATERIALIZED VIEW partition_steps AS\nSELECT\n partition as name,\n date_trunc('month',lower(partition_dtrange)) as sdate,\n date_trunc('month', upper(partition_dtrange)) + '1 month'::interval as edate\n FROM partitions_view WHERE partition_dtrange IS NOT NULL AND partition_dtrange != 'empty'::tstzrange\n ORDER BY dtrange ASC\n;\n\n\nCREATE OR REPLACE FUNCTION update_partition_stats_q(_partition text, istrigger boolean default false) RETURNS VOID AS $$\nDECLARE\nBEGIN\n PERFORM run_or_queue(\n format('SELECT update_partition_stats(%L, %L);', _partition, istrigger)\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION update_partition_stats(_partition text, istrigger boolean default false) RETURNS VOID AS $$\nDECLARE\n dtrange tstzrange;\n edtrange tstzrange;\n cdtrange tstzrange;\n cedtrange tstzrange;\n extent geometry;\n collection text;\nBEGIN\n RAISE NOTICE 'Updating stats for %.', _partition;\n EXECUTE format(\n $q$\n SELECT\n tstzrange(min(datetime), max(datetime),'[]'),\n tstzrange(min(end_datetime), max(end_datetime), '[]')\n FROM %I\n $q$,\n _partition\n ) INTO dtrange, edtrange;\n extent := st_estimatedextent('pgstac', _partition, 'geometry');\n INSERT INTO partition_stats (partition, dtrange, edtrange, spatial, last_updated)\n SELECT _partition, dtrange, edtrange, extent, now()\n ON CONFLICT (partition) DO\n UPDATE SET\n dtrange=EXCLUDED.dtrange,\n edtrange=EXCLUDED.edtrange,\n spatial=EXCLUDED.spatial,\n last_updated=EXCLUDED.last_updated\n ;\n\n SELECT\n constraint_dtrange, constraint_edtrange, pv.collection\n INTO cdtrange, cedtrange, collection\n FROM partitions_view pv WHERE partition = _partition;\n REFRESH MATERIALIZED VIEW partitions;\n REFRESH MATERIALIZED VIEW partition_steps;\n\n\n RAISE NOTICE 'Checking if we need to modify constraints.';\n IF\n (cdtrange IS DISTINCT FROM dtrange OR edtrange IS DISTINCT FROM cedtrange)\n AND NOT istrigger\n THEN\n RAISE NOTICE 'Modifying Constraints';\n RAISE NOTICE 'Existing % %', cdtrange, cedtrange;\n RAISE NOTICE 'New % %', dtrange, edtrange;\n PERFORM drop_table_constraints(_partition);\n PERFORM create_table_constraints(_partition, dtrange, edtrange);\n END IF;\n RAISE NOTICE 'Checking if we need to update collection extents.';\n IF get_setting_bool('update_collection_extent') THEN\n RAISE NOTICE 'updating collection extent for %', collection;\n PERFORM run_or_queue(format($q$\n UPDATE collections\n SET content = jsonb_set_lax(\n content,\n '{extent}'::text[],\n collection_extent(%L, FALSE),\n true,\n 'use_json_null'\n ) WHERE id=%L\n ;\n $q$, collection, collection));\n ELSE\n RAISE NOTICE 'Not updating collection extent for %', collection;\n END IF;\n\nEND;\n$$ LANGUAGE PLPGSQL STRICT;\n\n\nCREATE OR REPLACE FUNCTION partition_name( IN collection text, IN dt timestamptz, OUT partition_name text, OUT partition_range tstzrange) AS $$\nDECLARE\n c RECORD;\n parent_name text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n parent_name := format('_items_%s', c.key);\n\n\n IF c.partition_trunc = 'year' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYY'));\n ELSIF c.partition_trunc = 'month' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYYMM'));\n ELSE\n partition_name := parent_name;\n partition_range := tstzrange('-infinity'::timestamptz, 'infinity'::timestamptz, '[]');\n END IF;\n IF partition_range IS NULL THEN\n partition_range := tstzrange(\n date_trunc(c.partition_trunc::text, dt),\n date_trunc(c.partition_trunc::text, dt) + concat('1 ', c.partition_trunc)::interval\n );\n END IF;\n RETURN;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION drop_table_constraints(t text) RETURNS text AS $$\nDECLARE\n q text;\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM partitions_view WHERE partition=t) THEN\n RETURN NULL;\n END IF;\n FOR q IN SELECT FORMAT(\n $q$\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n $q$,\n t,\n conname\n ) FROM pg_constraint\n WHERE conrelid=t::regclass::oid AND contype='c'\n LOOP\n EXECUTE q;\n END LOOP;\n RETURN t;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE OR REPLACE FUNCTION create_table_constraints(t text, _dtrange tstzrange, _edtrange tstzrange) RETURNS text AS $$\nDECLARE\n q text;\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM partitions_view WHERE partition=t) THEN\n RETURN NULL;\n END IF;\n RAISE NOTICE 'Creating Table Constraints for % % %', t, _dtrange, _edtrange;\n IF _dtrange = 'empty' AND _edtrange = 'empty' THEN\n q :=format(\n $q$\n DO $block$\n BEGIN\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n ALTER TABLE %I\n ADD CONSTRAINT %I\n CHECK (((datetime IS NULL) AND (end_datetime IS NULL))) NOT VALID\n ;\n ALTER TABLE %I\n VALIDATE CONSTRAINT %I\n ;\n\n\n\n EXCEPTION WHEN others THEN\n RAISE WARNING '%%, Issue Altering Constraints. Please run update_partition_stats(%I)', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n $block$;\n $q$,\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t),\n t\n );\n ELSE\n q :=format(\n $q$\n DO $block$\n BEGIN\n\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n ALTER TABLE %I\n ADD CONSTRAINT %I\n CHECK (\n (datetime >= %L)\n AND (datetime <= %L)\n AND (end_datetime >= %L)\n AND (end_datetime <= %L)\n ) NOT VALID\n ;\n ALTER TABLE %I\n VALIDATE CONSTRAINT %I\n ;\n\n\n\n EXCEPTION WHEN others THEN\n RAISE WARNING '%%, Issue Altering Constraints. Please run update_partition_stats(%I)', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n $block$;\n $q$,\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t),\n lower(_dtrange),\n upper(_dtrange),\n lower(_edtrange),\n upper(_edtrange),\n t,\n format('%s_dt', t),\n t\n );\n END IF;\n PERFORM run_or_queue(q);\n RETURN t;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION check_partition(\n _collection text,\n _dtrange tstzrange,\n _edtrange tstzrange\n) RETURNS text AS $$\nDECLARE\n c RECORD;\n pm RECORD;\n _partition_name text;\n _partition_dtrange tstzrange;\n _constraint_dtrange tstzrange;\n _constraint_edtrange tstzrange;\n q text;\n deferrable_q text;\n err_context text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=_collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', _collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n\n IF c.partition_trunc IS NOT NULL THEN\n _partition_dtrange := tstzrange(\n date_trunc(c.partition_trunc, lower(_dtrange)),\n date_trunc(c.partition_trunc, lower(_dtrange)) + (concat('1 ', c.partition_trunc))::interval,\n '[)'\n );\n ELSE\n _partition_dtrange := '[-infinity, infinity]'::tstzrange;\n END IF;\n\n IF NOT _partition_dtrange @> _dtrange THEN\n RAISE EXCEPTION 'dtrange % is greater than the partition size % for collection %', _dtrange, c.partition_trunc, _collection;\n END IF;\n\n\n IF c.partition_trunc = 'year' THEN\n _partition_name := format('_items_%s_%s', c.key, to_char(lower(_partition_dtrange),'YYYY'));\n ELSIF c.partition_trunc = 'month' THEN\n _partition_name := format('_items_%s_%s', c.key, to_char(lower(_partition_dtrange),'YYYYMM'));\n ELSE\n _partition_name := format('_items_%s', c.key);\n END IF;\n\n SELECT * INTO pm FROM partition_sys_meta WHERE collection=_collection AND partition_dtrange @> _dtrange;\n IF FOUND THEN\n RAISE NOTICE '% % %', _edtrange, _dtrange, pm;\n _constraint_edtrange :=\n tstzrange(\n least(\n lower(_edtrange),\n nullif(lower(pm.constraint_edtrange), '-infinity')\n ),\n greatest(\n upper(_edtrange),\n nullif(upper(pm.constraint_edtrange), 'infinity')\n ),\n '[]'\n );\n _constraint_dtrange :=\n tstzrange(\n least(\n lower(_dtrange),\n nullif(lower(pm.constraint_dtrange), '-infinity')\n ),\n greatest(\n upper(_dtrange),\n nullif(upper(pm.constraint_dtrange), 'infinity')\n ),\n '[]'\n );\n\n IF pm.constraint_edtrange @> _edtrange AND pm.constraint_dtrange @> _dtrange THEN\n RETURN pm.partition;\n ELSE\n PERFORM drop_table_constraints(_partition_name);\n END IF;\n ELSE\n _constraint_edtrange := _edtrange;\n _constraint_dtrange := _dtrange;\n END IF;\n RAISE NOTICE 'Creating partition % %', _partition_name, _partition_dtrange;\n IF c.partition_trunc IS NULL THEN\n q := format(\n $q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n $q$,\n _partition_name,\n _collection,\n concat(_partition_name,'_pk'),\n _partition_name\n );\n ELSE\n q := format(\n $q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L) PARTITION BY RANGE (datetime);\n CREATE TABLE IF NOT EXISTS %I partition OF %I FOR VALUES FROM (%L) TO (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n $q$,\n format('_items_%s', c.key),\n _collection,\n _partition_name,\n format('_items_%s', c.key),\n lower(_partition_dtrange),\n upper(_partition_dtrange),\n format('%s_pk', _partition_name),\n _partition_name\n );\n END IF;\n\n BEGIN\n EXECUTE q;\n EXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', _partition_name;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\n END;\n PERFORM create_table_constraints(_partition_name, _constraint_dtrange, _constraint_edtrange);\n PERFORM maintain_partitions(_partition_name);\n PERFORM update_partition_stats_q(_partition_name, true);\n REFRESH MATERIALIZED VIEW partitions;\n REFRESH MATERIALIZED VIEW partition_steps;\n RETURN _partition_name;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION repartition(_collection text, _partition_trunc text, triggered boolean DEFAULT FALSE) RETURNS text AS $$\nDECLARE\n c RECORD;\n q text;\n from_trunc text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=_collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', _collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n IF triggered THEN\n RAISE NOTICE 'Converting % to % partitioning via Trigger', _collection, _partition_trunc;\n ELSE\n RAISE NOTICE 'Converting % from using % to % partitioning', _collection, c.partition_trunc, _partition_trunc;\n IF c.partition_trunc IS NOT DISTINCT FROM _partition_trunc THEN\n RAISE NOTICE 'Collection % already set to use partition by %', _collection, _partition_trunc;\n RETURN _collection;\n END IF;\n END IF;\n\n IF EXISTS (SELECT 1 FROM partitions_view WHERE collection=_collection LIMIT 1) THEN\n EXECUTE format(\n $q$\n CREATE TEMP TABLE changepartitionstaging ON COMMIT DROP AS SELECT * FROM %I;\n DROP TABLE IF EXISTS %I CASCADE;\n WITH p AS (\n SELECT\n collection,\n CASE WHEN %L IS NULL THEN '-infinity'::timestamptz\n ELSE date_trunc(%L, datetime)\n END as d,\n tstzrange(min(datetime),max(datetime),'[]') as dtrange,\n tstzrange(min(datetime),max(datetime),'[]') as edtrange\n FROM changepartitionstaging\n GROUP BY 1,2\n ) SELECT check_partition(collection, dtrange, edtrange) FROM p;\n INSERT INTO items SELECT * FROM changepartitionstaging;\n DROP TABLE changepartitionstaging;\n $q$,\n concat('_items_', c.key),\n concat('_items_', c.key),\n c.partition_trunc,\n c.partition_trunc\n );\n END IF;\n RETURN _collection;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE OR REPLACE FUNCTION collections_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n q text;\n partition_name text := format('_items_%s', NEW.key);\n partition_exists boolean := false;\n partition_empty boolean := true;\n err_context text;\n loadtemp boolean := FALSE;\nBEGIN\n RAISE NOTICE 'Collection Trigger. % %', NEW.id, NEW.key;\n IF TG_OP = 'UPDATE' AND NEW.partition_trunc IS DISTINCT FROM OLD.partition_trunc THEN\n PERFORM repartition(NEW.id, NEW.partition_trunc, TRUE);\n END IF;\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER collections_trigger AFTER\nINSERT\nOR\nUPDATE ON collections\nFOR EACH ROW EXECUTE FUNCTION collections_trigger_func();\n\nCREATE OR REPLACE FUNCTION chunker(\n IN _where text,\n OUT s timestamptz,\n OUT e timestamptz\n) RETURNS SETOF RECORD AS $$\nDECLARE\n explain jsonb;\nBEGIN\n IF _where IS NULL THEN\n _where := ' TRUE ';\n END IF;\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s;', _where)\n INTO explain;\n\n RETURN QUERY\n WITH t AS (\n SELECT j->>0 as p FROM\n jsonb_path_query(\n explain,\n 'strict $.**.\"Relation Name\" ? (@ != null)'\n ) j\n ),\n parts AS (\n SELECT sdate, edate FROM t JOIN partition_steps ON (t.p = name)\n ),\n times AS (\n SELECT sdate FROM parts\n UNION\n SELECT edate FROM parts\n ),\n uniq AS (\n SELECT DISTINCT sdate FROM times ORDER BY sdate\n ),\n last AS (\n SELECT sdate, lead(sdate, 1) over () as edate FROM uniq\n )\n SELECT sdate, edate FROM last WHERE edate IS NOT NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION partition_queries(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN partitions text[] DEFAULT NULL\n) RETURNS SETOF text AS $$\nDECLARE\n query text;\n sdate timestamptz;\n edate timestamptz;\nBEGIN\nIF _where IS NULL OR trim(_where) = '' THEN\n _where = ' TRUE ';\nEND IF;\nRAISE NOTICE 'Getting chunks for % %', _where, _orderby;\nIF _orderby ILIKE 'datetime d%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 DESC LOOP\n RETURN NEXT format($q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n $q$,\n sdate,\n edate,\n _where,\n _orderby\n );\n END LOOP;\nELSIF _orderby ILIKE 'datetime a%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 ASC LOOP\n RETURN NEXT format($q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n $q$,\n sdate,\n edate,\n _where,\n _orderby\n );\n END LOOP;\nELSE\n query := format($q$\n SELECT * FROM items\n WHERE %s\n ORDER BY %s\n $q$, _where, _orderby\n );\n\n RETURN NEXT query;\n RETURN;\nEND IF;\n\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION partition_query_view(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN _limit int DEFAULT 10\n) RETURNS text AS $$\n WITH p AS (\n SELECT * FROM partition_queries(_where, _orderby) p\n )\n SELECT\n CASE WHEN EXISTS (SELECT 1 FROM p) THEN\n (SELECT format($q$\n SELECT * FROM (\n %s\n ) total LIMIT %s\n $q$,\n string_agg(\n format($q$ SELECT * FROM ( %s ) AS sub $q$, p),\n '\n UNION ALL\n '\n ),\n _limit\n ))\n ELSE NULL\n END FROM p;\n$$ LANGUAGE SQL IMMUTABLE;\n\n\n\n\nCREATE OR REPLACE FUNCTION stac_search_to_where(j jsonb) RETURNS text AS $$\nDECLARE\n where_segments text[];\n _where text;\n dtrange tstzrange;\n collections text[];\n geom geometry;\n sdate timestamptz;\n edate timestamptz;\n filterlang text;\n filter jsonb := j->'filter';\nBEGIN\n IF j ? 'ids' THEN\n where_segments := where_segments || format('id = ANY (%L) ', to_text_array(j->'ids'));\n END IF;\n\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n where_segments := where_segments || format('collection = ANY (%L) ', collections);\n END IF;\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n\n where_segments := where_segments || format(' datetime <= %L::timestamptz AND end_datetime >= %L::timestamptz ',\n edate,\n sdate\n );\n END IF;\n\n geom := stac_geom(j);\n IF geom IS NOT NULL THEN\n where_segments := where_segments || format('st_intersects(geometry, %L)',geom);\n END IF;\n\n filterlang := COALESCE(\n j->>'filter-lang',\n get_setting('default_filter_lang', j->'conf')\n );\n IF NOT filter @? '$.**.op' THEN\n filterlang := 'cql-json';\n END IF;\n\n IF filterlang NOT IN ('cql-json','cql2-json') AND j ? 'filter' THEN\n RAISE EXCEPTION '% is not a supported filter-lang. Please use cql-json or cql2-json.', filterlang;\n END IF;\n\n IF j ? 'query' AND j ? 'filter' THEN\n RAISE EXCEPTION 'Can only use either query or filter at one time.';\n END IF;\n\n IF j ? 'query' THEN\n filter := query_to_cql2(j->'query');\n ELSIF filterlang = 'cql-json' THEN\n filter := cql1_to_cql2(filter);\n END IF;\n RAISE NOTICE 'FILTER: %', filter;\n where_segments := where_segments || cql2_query(filter);\n IF cardinality(where_segments) < 1 THEN\n RETURN ' TRUE ';\n END IF;\n\n _where := array_to_string(array_remove(where_segments, NULL), ' AND ');\n\n IF _where IS NULL OR BTRIM(_where) = '' THEN\n RETURN ' TRUE ';\n END IF;\n RETURN _where;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION parse_sort_dir(_dir text, reverse boolean default false) RETURNS text AS $$\n WITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n ) SELECT\n CASE\n WHEN NOT reverse THEN d\n WHEN d = 'ASC' THEN 'DESC'\n WHEN d = 'DESC' THEN 'ASC'\n END\n FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION sort_dir_to_op(_dir text, prev boolean default false) RETURNS text AS $$\n WITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n ) SELECT\n CASE\n WHEN d = 'ASC' AND prev THEN '<='\n WHEN d = 'DESC' AND prev THEN '>='\n WHEN d = 'ASC' THEN '>='\n WHEN d = 'DESC' THEN '<='\n END\n FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION sort_sqlorderby(\n _search jsonb DEFAULT NULL,\n reverse boolean DEFAULT FALSE\n) RETURNS text AS $$\n WITH sortby AS (\n SELECT coalesce(_search->'sortby','[{\"field\":\"datetime\", \"direction\":\"desc\"}]') as sort\n ), withid AS (\n SELECT CASE\n WHEN sort @? '$[*] ? (@.field == \"id\")' THEN sort\n ELSE sort || '[{\"field\":\"id\", \"direction\":\"desc\"}]'::jsonb\n END as sort\n FROM sortby\n ), withid_rows AS (\n SELECT jsonb_array_elements(sort) as value FROM withid\n ),sorts AS (\n SELECT\n coalesce(\n (queryable(value->>'field')).expression\n ) as key,\n parse_sort_dir(value->>'direction', reverse) as dir\n FROM withid_rows\n )\n SELECT array_to_string(\n array_agg(concat(key, ' ', dir)),\n ', '\n ) FROM sorts;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_sort_dir(sort_item jsonb) RETURNS text AS $$\n SELECT CASE WHEN sort_item->>'direction' ILIKE 'desc%' THEN 'DESC' ELSE 'ASC' END;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION get_token_val_str(\n _field text,\n _item items\n) RETURNS text AS $$\nDECLARE\n q text;\n literal text;\nBEGIN\n q := format($q$ SELECT quote_literal(%s) FROM (SELECT $1.*) as r;$q$, _field);\n EXECUTE q INTO literal USING _item;\n RETURN literal;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n\nCREATE OR REPLACE FUNCTION get_token_record(IN _token text, OUT prev BOOLEAN, OUT item items) RETURNS RECORD AS $$\nDECLARE\n _itemid text := _token;\n _collectionid text;\nBEGIN\n IF _token IS NULL THEN\n RETURN;\n END IF;\n RAISE NOTICE 'Looking for token: %', _token;\n prev := FALSE;\n IF _token ILIKE 'prev:%' THEN\n _itemid := replace(_token, 'prev:','');\n prev := TRUE;\n ELSIF _token ILIKE 'next:%' THEN\n _itemid := replace(_token, 'next:', '');\n END IF;\n SELECT id INTO _collectionid FROM collections WHERE _itemid LIKE concat(id,':%');\n IF FOUND THEN\n _itemid := replace(_itemid, concat(_collectionid,':'), '');\n SELECT * INTO item FROM items WHERE id=_itemid AND collection=_collectionid;\n ELSE\n SELECT * INTO item FROM items WHERE id=_itemid;\n END IF;\n IF item IS NULL THEN\n RAISE EXCEPTION 'Could not find item using token: % item: % collection: %', _token, _itemid, _collectionid;\n END IF;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\n\nCREATE OR REPLACE FUNCTION get_token_filter(\n _sortby jsonb DEFAULT '[{\"field\":\"datetime\",\"direction\":\"desc\"}]'::jsonb,\n token_item items DEFAULT NULL,\n prev boolean DEFAULT FALSE,\n inclusive boolean DEFAULT FALSE\n) RETURNS text AS $$\nDECLARE\n ltop text := '<';\n gtop text := '>';\n dir text;\n sort record;\n orfilter text := '';\n orfilters text[] := '{}'::text[];\n andfilters text[] := '{}'::text[];\n output text;\n token_where text;\nBEGIN\n IF _sortby IS NULL OR _sortby = '[]'::jsonb THEN\n _sortby := '[{\"field\":\"datetime\",\"direction\":\"desc\"}]'::jsonb;\n END IF;\n _sortby := _sortby || jsonb_build_object('field','id','direction',_sortby->0->>'direction');\n RAISE NOTICE 'Getting Token Filter. % %', _sortby, token_item;\n IF inclusive THEN\n orfilters := orfilters || format('( id=%L AND collection=%L )' , token_item.id, token_item.collection);\n END IF;\n\n FOR sort IN\n WITH s1 AS (\n SELECT\n _row,\n (queryable(value->>'field')).expression as _field,\n (value->>'field' = 'id') as _isid,\n get_sort_dir(value) as _dir\n FROM jsonb_array_elements(_sortby)\n WITH ORDINALITY AS t(value, _row)\n )\n SELECT\n _row,\n _field,\n _dir,\n get_token_val_str(_field, token_item) as _val\n FROM s1\n WHERE _row <= (SELECT min(_row) FROM s1 WHERE _isid)\n LOOP\n orfilter := NULL;\n RAISE NOTICE 'SORT: %', sort;\n IF sort._val IS NOT NULL AND ((prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC')) THEN\n orfilter := format($f$(\n (%s %s %s) OR (%s IS NULL)\n )$f$,\n sort._field,\n ltop,\n sort._val,\n sort._val\n );\n ELSIF sort._val IS NULL AND ((prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC')) THEN\n RAISE NOTICE '< but null';\n orfilter := format('%s IS NOT NULL', sort._field);\n ELSIF sort._val IS NULL THEN\n RAISE NOTICE '> but null';\n ELSE\n orfilter := format($f$(\n (%s %s %s) OR (%s IS NULL)\n )$f$,\n sort._field,\n gtop,\n sort._val,\n sort._field\n );\n END IF;\n RAISE NOTICE 'ORFILTER: %', orfilter;\n\n IF orfilter IS NOT NULL THEN\n IF sort._row = 1 THEN\n orfilters := orfilters || orfilter;\n ELSE\n orfilters := orfilters || format('(%s AND %s)', array_to_string(andfilters, ' AND '), orfilter);\n END IF;\n END IF;\n IF sort._val IS NOT NULL THEN\n andfilters := andfilters || format('%s = %s', sort._field, sort._val);\n ELSE\n andfilters := andfilters || format('%s IS NULL', sort._field);\n END IF;\n END LOOP;\n\n output := array_to_string(orfilters, ' OR ');\n\n token_where := concat('(',coalesce(output,'true'),')');\n IF trim(token_where) = '' THEN\n token_where := NULL;\n END IF;\n RAISE NOTICE 'TOKEN_WHERE: %',token_where;\n RETURN token_where;\n END;\n$$ LANGUAGE PLPGSQL SET transform_null_equals TO TRUE\n;\n\nCREATE OR REPLACE FUNCTION search_tohash(jsonb) RETURNS jsonb AS $$\n SELECT $1 - '{token,limit,context,includes,excludes}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION search_hash(jsonb, jsonb) RETURNS text AS $$\n SELECT md5(concat(search_tohash($1)::text,$2::text));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TABLE IF NOT EXISTS searches(\n hash text GENERATED ALWAYS AS (search_hash(search, metadata)) STORED PRIMARY KEY,\n search jsonb NOT NULL,\n _where text,\n orderby text,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n metadata jsonb DEFAULT '{}'::jsonb NOT NULL\n);\n\nCREATE TABLE IF NOT EXISTS search_wheres(\n id bigint generated always as identity primary key,\n _where text NOT NULL,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n statslastupdated timestamptz,\n estimated_count bigint,\n estimated_cost float,\n time_to_estimate float,\n total_count bigint,\n time_to_count float,\n partitions text[]\n);\n\nCREATE INDEX IF NOT EXISTS search_wheres_partitions ON search_wheres USING GIN (partitions);\nCREATE UNIQUE INDEX IF NOT EXISTS search_wheres_where ON search_wheres ((md5(_where)));\n\nCREATE OR REPLACE FUNCTION where_stats(inwhere text, updatestats boolean default false, conf jsonb default null) RETURNS search_wheres AS $$\nDECLARE\n t timestamptz;\n i interval;\n explain_json jsonb;\n partitions text[];\n sw search_wheres%ROWTYPE;\n inwhere_hash text := md5(inwhere);\n _context text := lower(context(conf));\n _stats_ttl interval := context_stats_ttl(conf);\n _estimated_cost float := context_estimated_cost(conf);\n _estimated_count int := context_estimated_count(conf);\nBEGIN\n IF _context = 'off' THEN\n sw._where := inwhere;\n return sw;\n END IF;\n\n SELECT * INTO sw FROM search_wheres WHERE md5(_where)=inwhere_hash FOR UPDATE;\n\n -- Update statistics if explicitly set, if statistics do not exist, or statistics ttl has expired\n IF NOT updatestats THEN\n RAISE NOTICE 'Checking if update is needed for: % .', inwhere;\n RAISE NOTICE 'Stats Last Updated: %', sw.statslastupdated;\n RAISE NOTICE 'TTL: %, Age: %', _stats_ttl, now() - sw.statslastupdated;\n RAISE NOTICE 'Context: %, Existing Total: %', _context, sw.total_count;\n IF\n sw.statslastupdated IS NULL\n OR (now() - sw.statslastupdated) > _stats_ttl\n OR (context(conf) != 'off' AND sw.total_count IS NULL)\n THEN\n updatestats := TRUE;\n END IF;\n END IF;\n\n sw._where := inwhere;\n sw.lastused := now();\n sw.usecount := coalesce(sw.usecount,0) + 1;\n\n IF NOT updatestats THEN\n UPDATE search_wheres SET\n lastused = sw.lastused,\n usecount = sw.usecount\n WHERE md5(_where) = inwhere_hash\n RETURNING * INTO sw\n ;\n RETURN sw;\n END IF;\n\n -- Use explain to get estimated count/cost and a list of the partitions that would be hit by the query\n t := clock_timestamp();\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s', inwhere)\n INTO explain_json;\n RAISE NOTICE 'Time for just the explain: %', clock_timestamp() - t;\n i := clock_timestamp() - t;\n\n sw.statslastupdated := now();\n sw.estimated_count := explain_json->0->'Plan'->'Plan Rows';\n sw.estimated_cost := explain_json->0->'Plan'->'Total Cost';\n sw.time_to_estimate := extract(epoch from i);\n\n RAISE NOTICE 'ESTIMATED_COUNT: % < %', sw.estimated_count, _estimated_count;\n RAISE NOTICE 'ESTIMATED_COST: % < %', sw.estimated_cost, _estimated_cost;\n\n -- Do a full count of rows if context is set to on or if auto is set and estimates are low enough\n IF\n _context = 'on'\n OR\n ( _context = 'auto' AND\n (\n sw.estimated_count < _estimated_count\n AND\n sw.estimated_cost < _estimated_cost\n )\n )\n THEN\n t := clock_timestamp();\n RAISE NOTICE 'Calculating actual count...';\n EXECUTE format(\n 'SELECT count(*) FROM items WHERE %s',\n inwhere\n ) INTO sw.total_count;\n i := clock_timestamp() - t;\n RAISE NOTICE 'Actual Count: % -- %', sw.total_count, i;\n sw.time_to_count := extract(epoch FROM i);\n ELSE\n sw.total_count := NULL;\n sw.time_to_count := NULL;\n END IF;\n\n\n INSERT INTO search_wheres\n (_where, lastused, usecount, statslastupdated, estimated_count, estimated_cost, time_to_estimate, partitions, total_count, time_to_count)\n SELECT sw._where, sw.lastused, sw.usecount, sw.statslastupdated, sw.estimated_count, sw.estimated_cost, sw.time_to_estimate, sw.partitions, sw.total_count, sw.time_to_count\n ON CONFLICT ((md5(_where)))\n DO UPDATE\n SET\n lastused = sw.lastused,\n usecount = sw.usecount,\n statslastupdated = sw.statslastupdated,\n estimated_count = sw.estimated_count,\n estimated_cost = sw.estimated_cost,\n time_to_estimate = sw.time_to_estimate,\n total_count = sw.total_count,\n time_to_count = sw.time_to_count\n ;\n RETURN sw;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION search_query(\n _search jsonb = '{}'::jsonb,\n updatestats boolean = false,\n _metadata jsonb = '{}'::jsonb\n) RETURNS searches AS $$\nDECLARE\n search searches%ROWTYPE;\n pexplain jsonb;\n t timestamptz;\n i interval;\n _hash text := search_hash(_search, _metadata);\n doupdate boolean := FALSE;\n insertfound boolean := FALSE;\nBEGIN\n SELECT * INTO search FROM searches\n WHERE hash=_hash;\n\n search.hash := _hash;\n\n -- Calculate the where clause if not already calculated\n IF search._where IS NULL THEN\n search._where := stac_search_to_where(_search);\n ELSE\n doupdate := TRUE;\n END IF;\n\n -- Calculate the order by clause if not already calculated\n IF search.orderby IS NULL THEN\n search.orderby := sort_sqlorderby(_search);\n ELSE\n doupdate := TRUE;\n END IF;\n\n PERFORM where_stats(search._where, updatestats, _search->'conf');\n\n IF NOT doupdate THEN\n INSERT INTO searches (search, _where, orderby, lastused, usecount, metadata)\n VALUES (_search, search._where, search.orderby, clock_timestamp(), 1, _metadata)\n ON CONFLICT (hash) DO NOTHING RETURNING * INTO search;\n IF FOUND THEN\n RETURN search;\n END IF;\n END IF;\n\n UPDATE searches\n SET\n lastused=clock_timestamp(),\n usecount=usecount+1\n WHERE hash=(\n SELECT hash FROM searches\n WHERE hash=_hash\n FOR UPDATE SKIP LOCKED\n );\n IF NOT FOUND THEN\n RAISE NOTICE 'Did not update stats for % due to lock. (This is generally OK)', _search;\n END IF;\n\n RETURN search;\n\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE OR REPLACE FUNCTION search_rows(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN partitions text[] DEFAULT NULL,\n IN _limit int DEFAULT 10\n) RETURNS SETOF items AS $$\nDECLARE\n base_query text;\n query text;\n sdate timestamptz;\n edate timestamptz;\n n int;\n records_left int := _limit;\n timer timestamptz := clock_timestamp();\n full_timer timestamptz := clock_timestamp();\nBEGIN\nIF _where IS NULL OR trim(_where) = '' THEN\n _where = ' TRUE ';\nEND IF;\nRAISE NOTICE 'Getting chunks for % %', _where, _orderby;\n\nbase_query := $q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n LIMIT %L\n$q$;\n\nIF _orderby ILIKE 'datetime d%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 DESC LOOP\n RAISE NOTICE 'Running Query for % to %. %', sdate, edate, age_ms(full_timer);\n query := format(\n base_query,\n sdate,\n edate,\n _where,\n _orderby,\n records_left\n );\n RAISE LOG 'QUERY: %', query;\n timer := clock_timestamp();\n RETURN QUERY EXECUTE query;\n\n GET DIAGNOSTICS n = ROW_COUNT;\n records_left := records_left - n;\n RAISE NOTICE 'Returned %/% Rows From % to %. % to go. Time: %ms', n, _limit, sdate, edate, records_left, age_ms(timer);\n timer := clock_timestamp();\n IF records_left <= 0 THEN\n RAISE NOTICE 'SEARCH_ROWS TOOK %ms', age_ms(full_timer);\n RETURN;\n END IF;\n END LOOP;\nELSIF _orderby ILIKE 'datetime a%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 ASC LOOP\n RAISE NOTICE 'Running Query for % to %. %', sdate, edate, age_ms(full_timer);\n query := format(\n base_query,\n sdate,\n edate,\n _where,\n _orderby,\n records_left\n );\n RAISE LOG 'QUERY: %', query;\n timer := clock_timestamp();\n RETURN QUERY EXECUTE query;\n\n GET DIAGNOSTICS n = ROW_COUNT;\n records_left := records_left - n;\n RAISE NOTICE 'Returned %/% Rows From % to %. % to go. Time: %ms', n, _limit, sdate, edate, records_left, age_ms(timer);\n timer := clock_timestamp();\n IF records_left <= 0 THEN\n RAISE NOTICE 'SEARCH_ROWS TOOK %ms', age_ms(full_timer);\n RETURN;\n END IF;\n END LOOP;\nELSE\n query := format($q$\n SELECT * FROM items\n WHERE %s\n ORDER BY %s\n LIMIT %L\n $q$, _where, _orderby, _limit\n );\n RAISE LOG 'QUERY: %', query;\n timer := clock_timestamp();\n RETURN QUERY EXECUTE query;\n RAISE NOTICE 'FULL QUERY TOOK %ms', age_ms(timer);\nEND IF;\nRAISE NOTICE 'SEARCH_ROWS TOOK %ms', age_ms(full_timer);\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE UNLOGGED TABLE format_item_cache(\n id text,\n collection text,\n fields text,\n hydrated bool,\n output jsonb,\n lastused timestamptz DEFAULT now(),\n usecount int DEFAULT 1,\n timetoformat float,\n PRIMARY KEY (collection, id, fields, hydrated)\n);\nCREATE INDEX ON format_item_cache (lastused);\n\nCREATE OR REPLACE FUNCTION format_item(_item items, _fields jsonb DEFAULT '{}', _hydrated bool DEFAULT TRUE) RETURNS jsonb AS $$\nDECLARE\n cache bool := get_setting_bool('format_cache');\n _output jsonb := null;\n t timestamptz := clock_timestamp();\nBEGIN\n IF cache THEN\n SELECT output INTO _output FROM format_item_cache\n WHERE id=_item.id AND collection=_item.collection AND fields=_fields::text AND hydrated=_hydrated;\n END IF;\n IF _output IS NULL THEN\n IF _hydrated THEN\n _output := content_hydrate(_item, _fields);\n ELSE\n _output := content_nonhydrated(_item, _fields);\n END IF;\n END IF;\n IF cache THEN\n INSERT INTO format_item_cache (id, collection, fields, hydrated, output, timetoformat)\n VALUES (_item.id, _item.collection, _fields::text, _hydrated, _output, age_ms(t))\n ON CONFLICT(collection, id, fields, hydrated) DO\n UPDATE\n SET lastused=now(), usecount = format_item_cache.usecount + 1\n ;\n END IF;\n RETURN _output;\n\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION search(_search jsonb = '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n searches searches%ROWTYPE;\n _where text;\n orderby text;\n search_where search_wheres%ROWTYPE;\n total_count bigint;\n token record;\n token_prev boolean;\n token_item items%ROWTYPE;\n token_where text;\n full_where text;\n init_ts timestamptz := clock_timestamp();\n timer timestamptz := clock_timestamp();\n hydrate bool := NOT (_search->'conf'->>'nohydrate' IS NOT NULL AND (_search->'conf'->>'nohydrate')::boolean = true);\n prev text;\n next text;\n context jsonb;\n collection jsonb;\n out_records jsonb;\n out_len int;\n _limit int := coalesce((_search->>'limit')::int, 10);\n _querylimit int;\n _fields jsonb := coalesce(_search->'fields', '{}'::jsonb);\n has_prev boolean := FALSE;\n has_next boolean := FALSE;\nBEGIN\n searches := search_query(_search);\n _where := searches._where;\n orderby := searches.orderby;\n search_where := where_stats(_where);\n total_count := coalesce(search_where.total_count, search_where.estimated_count);\n RAISE NOTICE 'SEARCH:TOKEN: %', _search->>'token';\n token := get_token_record(_search->>'token');\n RAISE NOTICE '***TOKEN: %', token;\n _querylimit := _limit + 1;\n IF token IS NOT NULL THEN\n token_prev := token.prev;\n token_item := token.item;\n token_where := get_token_filter(_search->'sortby', token_item, token_prev, FALSE);\n RAISE LOG 'TOKEN_WHERE: % (%ms from search start)', token_where, age_ms(timer);\n IF token_prev THEN -- if we are using a prev token, we know has_next is true\n RAISE LOG 'There is a previous token, so automatically setting has_next to true';\n has_next := TRUE;\n orderby := sort_sqlorderby(_search, TRUE);\n ELSE\n RAISE LOG 'There is a next token, so automatically setting has_prev to true';\n has_prev := TRUE;\n\n END IF;\n ELSE -- if there was no token, we know there is no prev\n RAISE LOG 'There is no token, so we know there is no prev. setting has_prev to false';\n has_prev := FALSE;\n END IF;\n\n full_where := concat_ws(' AND ', _where, token_where);\n RAISE NOTICE 'FULL WHERE CLAUSE: %', full_where;\n RAISE NOTICE 'Time to get counts and build query %', age_ms(timer);\n timer := clock_timestamp();\n\n IF hydrate THEN\n RAISE NOTICE 'Getting hydrated data.';\n ELSE\n RAISE NOTICE 'Getting non-hydrated data.';\n END IF;\n RAISE NOTICE 'CACHE SET TO %', get_setting_bool('format_cache');\n RAISE NOTICE 'Time to set hydration/formatting %', age_ms(timer);\n timer := clock_timestamp();\n SELECT jsonb_agg(format_item(i, _fields, hydrate)) INTO out_records\n FROM search_rows(\n full_where,\n orderby,\n search_where.partitions,\n _querylimit\n ) as i;\n\n RAISE NOTICE 'Time to fetch rows %', age_ms(timer);\n timer := clock_timestamp();\n\n\n IF token_prev THEN\n out_records := flip_jsonb_array(out_records);\n END IF;\n\n RAISE NOTICE 'Query returned % records.', jsonb_array_length(out_records);\n RAISE LOG 'TOKEN: % %', token_item.id, token_item.collection;\n RAISE LOG 'RECORD_1: % %', out_records->0->>'id', out_records->0->>'collection';\n RAISE LOG 'RECORD-1: % %', out_records->-1->>'id', out_records->-1->>'collection';\n\n -- REMOVE records that were from our token\n IF out_records->0->>'id' = token_item.id AND out_records->0->>'collection' = token_item.collection THEN\n out_records := out_records - 0;\n ELSIF out_records->-1->>'id' = token_item.id AND out_records->-1->>'collection' = token_item.collection THEN\n out_records := out_records - -1;\n END IF;\n\n out_len := jsonb_array_length(out_records);\n\n IF out_len = _limit + 1 THEN\n IF token_prev THEN\n has_prev := TRUE;\n out_records := out_records - 0;\n ELSE\n has_next := TRUE;\n out_records := out_records - -1;\n END IF;\n END IF;\n\n IF has_next THEN\n next := concat(out_records->-1->>'collection', ':', out_records->-1->>'id');\n RAISE NOTICE 'HAS NEXT | %', next;\n END IF;\n\n IF has_prev THEN\n prev := concat(out_records->0->>'collection', ':', out_records->0->>'id');\n RAISE NOTICE 'HAS PREV | %', prev;\n END IF;\n\n RAISE NOTICE 'Time to get prev/next %', age_ms(timer);\n timer := clock_timestamp();\n\n IF context(_search->'conf') != 'off' THEN\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'matched', total_count,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\n ELSE\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\n END IF;\n\n collection := jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb),\n 'next', next,\n 'prev', prev,\n 'context', context\n );\n\n IF get_setting_bool('timing', _search->'conf') THEN\n collection = collection || jsonb_build_object('timing', age_ms(init_ts));\n END IF;\n\n RAISE NOTICE 'Time to build final json %', age_ms(timer);\n timer := clock_timestamp();\n\n RAISE NOTICE 'Total Time: %', age_ms(current_timestamp);\n RAISE NOTICE 'RETURNING % records. NEXT: %. PREV: %', collection->'context'->>'returned', collection->>'next', collection->>'prev';\n RETURN collection;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION search_cursor(_search jsonb = '{}'::jsonb) RETURNS refcursor AS $$\nDECLARE\n curs refcursor;\n searches searches%ROWTYPE;\n _where text;\n _orderby text;\n q text;\n\nBEGIN\n searches := search_query(_search);\n _where := searches._where;\n _orderby := searches.orderby;\n\n OPEN curs FOR\n WITH p AS (\n SELECT * FROM partition_queries(_where, _orderby) p\n )\n SELECT\n CASE WHEN EXISTS (SELECT 1 FROM p) THEN\n (SELECT format($q$\n SELECT * FROM (\n %s\n ) total\n $q$,\n string_agg(\n format($q$ SELECT * FROM ( %s ) AS sub $q$, p),\n '\n UNION ALL\n '\n )\n ))\n ELSE NULL\n END FROM p;\n RETURN curs;\nEND;\n$$ LANGUAGE PLPGSQL;\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION tileenvelope(zoom int, x int, y int) RETURNS geometry AS $$\nWITH t AS (\n SELECT\n 20037508.3427892 as merc_max,\n -20037508.3427892 as merc_min,\n (2 * 20037508.3427892) / (2 ^ zoom) as tile_size\n)\nSELECT st_makeenvelope(\n merc_min + (tile_size * x),\n merc_max - (tile_size * (y + 1)),\n merc_min + (tile_size * (x + 1)),\n merc_max - (tile_size * y),\n 3857\n) FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;DROP FUNCTION IF EXISTS mercgrid;\n\n\nCREATE OR REPLACE FUNCTION ftime() RETURNS interval as $$\nSELECT age(clock_timestamp(), transaction_timestamp());\n$$ LANGUAGE SQL;\nSET SEARCH_PATH to pgstac, public;\n\nDROP FUNCTION IF EXISTS geometrysearch;\nCREATE OR REPLACE FUNCTION geometrysearch(\n IN geom geometry,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE, -- Return as soon as the passed in geometry is full covered\n IN skipcovered boolean DEFAULT TRUE -- Skip any items that would show up completely under the previous items\n) RETURNS jsonb AS $$\nDECLARE\n search searches%ROWTYPE;\n curs refcursor;\n _where text;\n query text;\n iter_record items%ROWTYPE;\n out_records jsonb := '{}'::jsonb[];\n exit_flag boolean := FALSE;\n counter int := 1;\n scancounter int := 1;\n remaining_limit int := _scanlimit;\n tilearea float;\n unionedgeom geometry;\n clippedgeom geometry;\n unionedgeom_area float := 0;\n prev_area float := 0;\n excludes text[];\n includes text[];\n\nBEGIN\n DROP TABLE IF EXISTS pgstac_results;\n CREATE TEMP TABLE pgstac_results (content jsonb) ON COMMIT DROP;\n\n -- If skipcovered is true then you will always want to exit when the passed in geometry is full\n IF skipcovered THEN\n exitwhenfull := TRUE;\n END IF;\n\n SELECT * INTO search FROM searches WHERE hash=queryhash;\n\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Search with Query Hash % Not Found', queryhash;\n END IF;\n\n tilearea := st_area(geom);\n _where := format('%s AND st_intersects(geometry, %L::geometry)', search._where, geom);\n\n\n FOR query IN SELECT * FROM partition_queries(_where, search.orderby) LOOP\n query := format('%s LIMIT %L', query, remaining_limit);\n RAISE NOTICE '%', query;\n OPEN curs FOR EXECUTE query;\n LOOP\n FETCH curs INTO iter_record;\n EXIT WHEN NOT FOUND;\n IF exitwhenfull OR skipcovered THEN -- If we are not using exitwhenfull or skipcovered, we do not need to do expensive geometry operations\n clippedgeom := st_intersection(geom, iter_record.geometry);\n\n IF unionedgeom IS NULL THEN\n unionedgeom := clippedgeom;\n ELSE\n unionedgeom := st_union(unionedgeom, clippedgeom);\n END IF;\n\n unionedgeom_area := st_area(unionedgeom);\n\n IF skipcovered AND prev_area = unionedgeom_area THEN\n scancounter := scancounter + 1;\n CONTINUE;\n END IF;\n\n prev_area := unionedgeom_area;\n\n RAISE NOTICE '% % % %', unionedgeom_area/tilearea, counter, scancounter, ftime();\n END IF;\n RAISE NOTICE '% %', iter_record, content_hydrate(iter_record, fields);\n INSERT INTO pgstac_results (content) VALUES (content_hydrate(iter_record, fields));\n\n IF counter >= _limit\n OR scancounter > _scanlimit\n OR ftime() > _timelimit\n OR (exitwhenfull AND unionedgeom_area >= tilearea)\n THEN\n exit_flag := TRUE;\n EXIT;\n END IF;\n counter := counter + 1;\n scancounter := scancounter + 1;\n\n END LOOP;\n CLOSE curs;\n EXIT WHEN exit_flag;\n remaining_limit := _scanlimit - scancounter;\n END LOOP;\n\n SELECT jsonb_agg(content) INTO out_records FROM pgstac_results WHERE content IS NOT NULL;\n\n RETURN jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb)\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDROP FUNCTION IF EXISTS geojsonsearch;\nCREATE OR REPLACE FUNCTION geojsonsearch(\n IN geojson jsonb,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_geomfromgeojson(geojson),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS xyzsearch;\nCREATE OR REPLACE FUNCTION xyzsearch(\n IN _x int,\n IN _y int,\n IN _z int,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_transform(tileenvelope(_z, _x, _y), 4326),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE PROCEDURE analyze_items() AS $$\nDECLARE\n q text;\n timeout_ts timestamptz;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n SELECT format('ANALYZE (VERBOSE, SKIP_LOCKED) %I;', relname) INTO q\n FROM pg_stat_user_tables\n WHERE relname like '_item%' AND (n_mod_since_analyze>0 OR last_analyze IS NULL) LIMIT 1;\n IF NOT FOUND THEN\n EXIT;\n END IF;\n RAISE NOTICE '%', q;\n EXECUTE q;\n COMMIT;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE PROCEDURE validate_constraints() AS $$\nDECLARE\n q text;\nBEGIN\n FOR q IN\n SELECT\n FORMAT(\n 'ALTER TABLE %I.%I VALIDATE CONSTRAINT %I;',\n nsp.nspname,\n cls.relname,\n con.conname\n )\n\n FROM pg_constraint AS con\n JOIN pg_class AS cls\n ON con.conrelid = cls.oid\n JOIN pg_namespace AS nsp\n ON cls.relnamespace = nsp.oid\n WHERE convalidated = FALSE AND contype in ('c','f')\n AND nsp.nspname = 'pgstac'\n LOOP\n RAISE NOTICE '%', q;\n PERFORM run_or_queue(q);\n COMMIT;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION collection_extent(_collection text, runupdate boolean default false) RETURNS jsonb AS $$\nDECLARE\n geom_extent geometry;\n mind timestamptz;\n maxd timestamptz;\n extent jsonb;\nBEGIN\n IF runupdate THEN\n PERFORM update_partition_stats_q(partition)\n FROM partitions_view WHERE collection=_collection;\n END IF;\n SELECT\n min(lower(dtrange)),\n max(upper(edtrange)),\n st_extent(spatial)\n INTO\n mind,\n maxd,\n geom_extent\n FROM partitions_view\n WHERE collection=_collection;\n\n IF geom_extent IS NOT NULL AND mind IS NOT NULL AND maxd IS NOT NULL THEN\n extent := jsonb_build_object(\n 'extent', jsonb_build_object(\n 'spatial', jsonb_build_object(\n 'bbox', to_jsonb(array[array[st_xmin(geom_extent), st_ymin(geom_extent), st_xmax(geom_extent), st_ymax(geom_extent)]])\n ),\n 'temporal', jsonb_build_object(\n 'interval', to_jsonb(array[array[mind, maxd]])\n )\n )\n );\n RETURN extent;\n END IF;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('id', '{\"title\": \"Item ID\",\"description\": \"Item identifier\",\"$ref\": \"https://schemas.stacspec.org/v1.0.0/item-spec/json-schema/item.json#/definitions/core/allOf/2/properties/id\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('geometry', '{\"title\": \"Item Geometry\",\"description\": \"Item Geometry\",\"$ref\": \"https://geojson.org/schema/Feature.json\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('datetime','{\"description\": \"Datetime\",\"type\": \"string\",\"title\": \"Acquired\",\"format\": \"date-time\",\"pattern\": \"(\\\\+00:00|Z)$\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDELETE FROM queryables a USING queryables b\n WHERE a.name = b.name AND a.collection_ids IS NOT DISTINCT FROM b.collection_ids AND a.id > b.id;\n\n\nINSERT INTO pgstac_settings (name, value) VALUES\n ('context', 'off'),\n ('context_estimated_count', '100000'),\n ('context_estimated_cost', '100000'),\n ('context_stats_ttl', '1 day'),\n ('default_filter_lang', 'cql2-json'),\n ('additional_properties', 'true'),\n ('use_queue', 'false'),\n ('queue_timeout', '10 minutes'),\n ('update_collection_extent', 'false'),\n ('format_cache', 'false')\nON CONFLICT DO NOTHING\n;\n\nALTER FUNCTION to_text COST 5000;\nALTER FUNCTION to_float COST 5000;\nALTER FUNCTION to_int COST 5000;\nALTER FUNCTION to_tstz COST 5000;\nALTER FUNCTION to_text_array COST 5000;\n\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nGRANT ALL ON SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac to pgstac_admin;\n\n-- pgstac_read role limited to using function apis\nGRANT EXECUTE ON FUNCTION search TO pgstac_read;\nGRANT EXECUTE ON FUNCTION search_query TO pgstac_read;\nGRANT EXECUTE ON FUNCTION item_by_id TO pgstac_read;\nGRANT EXECUTE ON FUNCTION get_item TO pgstac_read;\nGRANT SELECT ON ALL TABLES IN SCHEMA pgstac TO pgstac_read;\n\n\nGRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_ingest;\nGRANT USAGE ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_ingest;\n\nSELECT update_partition_stats_q(partition) FROM partitions_view;\nSELECT set_version('0.8.0');\n" }, { "path": "src/pgstac/migrations/pgstac.0.8.1-0.8.2.sql", "content": "SET client_min_messages TO WARNING;\nSET SEARCH_PATH to pgstac, public;\nRESET ROLE;\nDO $$\nDECLARE\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='postgis') THEN\n CREATE EXTENSION IF NOT EXISTS postgis;\n END IF;\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='btree_gist') THEN\n CREATE EXTENSION IF NOT EXISTS btree_gist;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_admin;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_read;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_ingest;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n\nGRANT pgstac_admin TO current_user;\n\n-- Function to make sure pgstac_admin is the owner of items\nCREATE OR REPLACE FUNCTION pgstac_admin_owns() RETURNS VOID AS $$\nDECLARE\n f RECORD;\nBEGIN\n FOR f IN (\n SELECT\n concat(\n oid::regproc::text,\n '(',\n coalesce(pg_get_function_identity_arguments(oid),''),\n ')'\n ) AS name,\n CASE prokind WHEN 'f' THEN 'FUNCTION' WHEN 'p' THEN 'PROCEDURE' WHEN 'a' THEN 'AGGREGATE' END as typ\n FROM pg_proc\n WHERE\n pronamespace=to_regnamespace('pgstac')\n AND proowner != to_regrole('pgstac_admin')\n AND proname NOT LIKE 'pg_stat%'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n FOR f IN (\n SELECT\n oid::regclass::text as name,\n CASE relkind\n WHEN 'i' THEN 'INDEX'\n WHEN 'I' THEN 'INDEX'\n WHEN 'p' THEN 'TABLE'\n WHEN 'r' THEN 'TABLE'\n WHEN 'v' THEN 'VIEW'\n WHEN 'S' THEN 'SEQUENCE'\n ELSE NULL\n END as typ\n FROM pg_class\n WHERE relnamespace=to_regnamespace('pgstac') and relowner != to_regrole('pgstac_admin') AND relkind IN ('r','p','v','S') AND relname NOT LIKE 'pg_stat'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\nSELECT pgstac_admin_owns();\n\nCREATE SCHEMA IF NOT EXISTS pgstac AUTHORIZATION pgstac_admin;\n\nGRANT ALL ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_admin;\n\nALTER ROLE pgstac_admin SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_read SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_ingest SET SEARCH_PATH TO pgstac, public;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\n\nGRANT pgstac_read TO pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\n\nSET ROLE pgstac_admin;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\nRESET ROLE;\n\nSET ROLE pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\nRESET ROLE;\n\nSET SEARCH_PATH TO pgstac, public;\nSET ROLE pgstac_admin;\n\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS analyze_items;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS validate_constraints;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n-- Install these idempotently as migrations do not put them before trying to modify the collections table\n\n\nCREATE OR REPLACE FUNCTION collection_geom(content jsonb)\nRETURNS geometry AS $$\n WITH box AS (SELECT content->'extent'->'spatial'->'bbox'->0 as box)\n SELECT\n st_makeenvelope(\n (box->>0)::float,\n (box->>1)::float,\n (box->>2)::float,\n (box->>3)::float,\n 4326\n )\n FROM box;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION collection_datetime(content jsonb)\nRETURNS timestamptz AS $$\n SELECT\n CASE\n WHEN\n (content->'extent'->'temporal'->'interval'->0->>0) IS NULL\n THEN '-infinity'::timestamptz\n ELSE\n (content->'extent'->'temporal'->'interval'->0->>0)::timestamptz\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION collection_enddatetime(content jsonb)\nRETURNS timestamptz AS $$\n SELECT\n CASE\n WHEN\n (content->'extent'->'temporal'->'interval'->0->>1) IS NULL\n THEN 'infinity'::timestamptz\n ELSE\n (content->'extent'->'temporal'->'interval'->0->>1)::timestamptz\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n-- BEGIN migra calculated SQL\nalter table \"pgstac\".\"collections\" alter column \"id\" set not null;\n\nset check_function_bodies = off;\n\nCREATE OR REPLACE FUNCTION pgstac.additional_properties()\n RETURNS boolean\n LANGUAGE sql\nAS $function$\n SELECT pgstac.get_setting_bool('additional_properties');\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.base_url(conf jsonb DEFAULT NULL::jsonb)\n RETURNS text\n LANGUAGE sql\nAS $function$\n SELECT COALESCE(pgstac.get_setting('base_url', conf), '.');\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.collection_search(_search jsonb DEFAULT '{}'::jsonb)\n RETURNS jsonb\n LANGUAGE plpgsql\n STABLE PARALLEL SAFE\nAS $function$\nDECLARE\n out_records jsonb;\n number_matched bigint := collection_search_matched(_search);\n number_returned bigint;\n _limit int := coalesce((_search->>'limit')::float::int, 10);\n _offset int := coalesce((_search->>'offset')::float::int, 0);\n links jsonb := '[]';\n ret jsonb;\n base_url text:= concat(rtrim(base_url(_search->'conf'),'/'), '/collections');\n prevoffset int;\n nextoffset int;\nBEGIN\n SELECT\n coalesce(jsonb_agg(c), '[]')\n INTO out_records\n FROM collection_search_rows(_search) c;\n\n number_returned := jsonb_array_length(out_records);\n\n IF _limit <= number_matched THEN --need to have paging links\n nextoffset := least(_offset + _limit, number_matched - 1);\n prevoffset := greatest(_offset - _limit, 0);\n IF _offset = 0 THEN -- no previous paging\n\n links := jsonb_build_array(\n jsonb_build_object(\n 'rel', 'next',\n 'type', 'application/json',\n 'method', 'GET' ,\n 'href', base_url,\n 'body', jsonb_build_object('offset', nextoffset),\n 'merge', TRUE\n )\n );\n ELSE\n links := jsonb_build_array(\n jsonb_build_object(\n 'rel', 'prev',\n 'type', 'application/json',\n 'method', 'GET' ,\n 'href', base_url,\n 'body', jsonb_build_object('offset', prevoffset),\n 'merge', TRUE\n ),\n jsonb_build_object(\n 'rel', 'next',\n 'type', 'application/json',\n 'method', 'GET' ,\n 'href', base_url,\n 'body', jsonb_build_object('offset', nextoffset),\n 'merge', TRUE\n )\n );\n END IF;\n END IF;\n\n ret := jsonb_build_object(\n 'collections', out_records,\n 'context', jsonb_build_object(\n 'limit', _limit,\n 'matched', number_matched,\n 'returned', number_returned\n ),\n 'links', links\n );\n RETURN ret;\n\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.collection_search_matched(_search jsonb DEFAULT '{}'::jsonb, OUT matched bigint)\n RETURNS bigint\n LANGUAGE plpgsql\n STABLE PARALLEL SAFE\nAS $function$\nDECLARE\n _where text := stac_search_to_where(_search);\nBEGIN\n EXECUTE format(\n $query$\n SELECT\n count(*)\n FROM\n collections_asitems\n WHERE %s\n ;\n $query$,\n _where\n ) INTO matched;\n RETURN;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.collection_search_rows(_search jsonb DEFAULT '{}'::jsonb)\n RETURNS SETOF jsonb\n LANGUAGE plpgsql\nAS $function$\nDECLARE\n _where text := stac_search_to_where(_search);\n _limit int := coalesce((_search->>'limit')::int, 10);\n _fields jsonb := coalesce(_search->'fields', '{}'::jsonb);\n _orderby text;\n _offset int := COALESCE((_search->>'offset')::int, 0);\nBEGIN\n _orderby := sort_sqlorderby(\n jsonb_build_object(\n 'sortby',\n coalesce(\n _search->'sortby',\n '[{\"field\": \"id\", \"direction\": \"asc\"}]'::jsonb\n )\n )\n );\n RETURN QUERY EXECUTE format(\n $query$\n SELECT\n jsonb_fields(collectionjson, %L) as c\n FROM\n collections_asitems\n WHERE %s\n ORDER BY %s\n LIMIT %L\n OFFSET %L\n ;\n $query$,\n _fields,\n _where,\n _orderby,\n _limit,\n _offset\n );\nEND;\n$function$\n;\n\ncreate or replace view \"pgstac\".\"collections_asitems\" as SELECT collections.id,\n collections.geometry,\n 'collections'::text AS collection,\n collections.datetime,\n collections.end_datetime,\n jsonb_build_object('properties', (collections.content - '{links,assets,stac_version,stac_extensions}'::text), 'links', (collections.content -> 'links'::text), 'assets', (collections.content -> 'assets'::text), 'stac_version', (collections.content -> 'stac_version'::text), 'stac_extensions', (collections.content -> 'stac_extensions'::text)) AS content,\n collections.content AS collectionjson\n FROM collections;\n\n\nCREATE OR REPLACE FUNCTION pgstac.maintain_index(indexname text, queryable_idx text, dropindexes boolean DEFAULT false, rebuildindexes boolean DEFAULT false, idxconcurrently boolean DEFAULT false)\n RETURNS void\n LANGUAGE plpgsql\n SECURITY DEFINER\nAS $function$\nDECLARE\nBEGIN\n IF indexname IS NOT NULL THEN\n IF dropindexes OR queryable_idx IS NOT NULL THEN\n EXECUTE format('DROP INDEX IF EXISTS %I;', indexname);\n ELSIF rebuildindexes THEN\n IF idxconcurrently THEN\n EXECUTE format('REINDEX INDEX CONCURRENTLY %I;', indexname);\n ELSE\n EXECUTE format('REINDEX INDEX CONCURRENTLY %I;', indexname);\n END IF;\n END IF;\n END IF;\n IF queryable_idx IS NOT NULL THEN\n IF idxconcurrently THEN\n EXECUTE replace(queryable_idx, 'INDEX', 'INDEX CONCURRENTLY');\n ELSE EXECUTE queryable_idx;\n END IF;\n END IF;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.queryable_indexes(treeroot text DEFAULT 'items'::text, changes boolean DEFAULT false, OUT collection text, OUT partition text, OUT field text, OUT indexname text, OUT existing_idx text, OUT queryable_idx text)\n RETURNS SETOF record\n LANGUAGE sql\nAS $function$\nWITH p AS (\n SELECT\n relid::text as partition,\n replace(replace(\n CASE\n WHEN parentrelid::regclass::text='items' THEN pg_get_expr(c.relpartbound, c.oid)\n ELSE pg_get_expr(parent.relpartbound, parent.oid)\n END,\n 'FOR VALUES IN (''',''), ''')',\n ''\n ) AS collection\n FROM pg_partition_tree(treeroot)\n JOIN pg_class c ON (relid::regclass = c.oid)\n JOIN pg_class parent ON (parentrelid::regclass = parent.oid AND isleaf)\n ), i AS (\n SELECT\n partition,\n indexname,\n regexp_replace(btrim(replace(replace(indexdef, indexname, ''),'pgstac.',''),' \\t\\n'), '[ ]+', ' ', 'g') as iidx,\n COALESCE(\n (regexp_match(indexdef, '\\(([a-zA-Z]+)\\)'))[1],\n (regexp_match(indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_-]+)''::text'))[1],\n CASE WHEN indexdef ~* '\\(datetime desc, end_datetime\\)' THEN 'datetime' ELSE NULL END\n ) AS field\n FROM\n pg_indexes\n JOIN p ON (tablename=partition)\n ), q AS (\n SELECT\n name AS field,\n collection,\n partition,\n format(indexdef(queryables), partition) as qidx\n FROM queryables, unnest_collection(queryables.collection_ids) collection\n JOIN p USING (collection)\n WHERE property_index_type IS NOT NULL OR name IN ('datetime','geometry','id')\n )\n SELECT\n collection,\n partition,\n field,\n indexname,\n iidx as existing_idx,\n qidx as queryable_idx\n FROM i FULL JOIN q USING (field, partition)\n WHERE CASE WHEN changes THEN lower(iidx) IS DISTINCT FROM lower(qidx) ELSE TRUE END;\n;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.readonly(conf jsonb DEFAULT NULL::jsonb)\n RETURNS boolean\n LANGUAGE sql\nAS $function$\n SELECT pgstac.get_setting_bool('readonly', conf);\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.check_partition(_collection text, _dtrange tstzrange, _edtrange tstzrange)\n RETURNS text\n LANGUAGE plpgsql\n SECURITY DEFINER\nAS $function$\nDECLARE\n c RECORD;\n pm RECORD;\n _partition_name text;\n _partition_dtrange tstzrange;\n _constraint_dtrange tstzrange;\n _constraint_edtrange tstzrange;\n q text;\n deferrable_q text;\n err_context text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=_collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', _collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n\n IF c.partition_trunc IS NOT NULL THEN\n _partition_dtrange := tstzrange(\n date_trunc(c.partition_trunc, lower(_dtrange)),\n date_trunc(c.partition_trunc, lower(_dtrange)) + (concat('1 ', c.partition_trunc))::interval,\n '[)'\n );\n ELSE\n _partition_dtrange := '[-infinity, infinity]'::tstzrange;\n END IF;\n\n IF NOT _partition_dtrange @> _dtrange THEN\n RAISE EXCEPTION 'dtrange % is greater than the partition size % for collection %', _dtrange, c.partition_trunc, _collection;\n END IF;\n\n\n IF c.partition_trunc = 'year' THEN\n _partition_name := format('_items_%s_%s', c.key, to_char(lower(_partition_dtrange),'YYYY'));\n ELSIF c.partition_trunc = 'month' THEN\n _partition_name := format('_items_%s_%s', c.key, to_char(lower(_partition_dtrange),'YYYYMM'));\n ELSE\n _partition_name := format('_items_%s', c.key);\n END IF;\n\n SELECT * INTO pm FROM partition_sys_meta WHERE collection=_collection AND partition_dtrange @> _dtrange;\n IF FOUND THEN\n RAISE NOTICE '% % %', _edtrange, _dtrange, pm;\n _constraint_edtrange :=\n tstzrange(\n least(\n lower(_edtrange),\n nullif(lower(pm.constraint_edtrange), '-infinity')\n ),\n greatest(\n upper(_edtrange),\n nullif(upper(pm.constraint_edtrange), 'infinity')\n ),\n '[]'\n );\n _constraint_dtrange :=\n tstzrange(\n least(\n lower(_dtrange),\n nullif(lower(pm.constraint_dtrange), '-infinity')\n ),\n greatest(\n upper(_dtrange),\n nullif(upper(pm.constraint_dtrange), 'infinity')\n ),\n '[]'\n );\n\n IF pm.constraint_edtrange @> _edtrange AND pm.constraint_dtrange @> _dtrange THEN\n RETURN pm.partition;\n ELSE\n PERFORM drop_table_constraints(_partition_name);\n END IF;\n ELSE\n _constraint_edtrange := _edtrange;\n _constraint_dtrange := _dtrange;\n END IF;\n RAISE NOTICE 'EXISTING CONSTRAINTS % %, NEW % %', pm.constraint_dtrange, pm.constraint_edtrange, _constraint_dtrange, _constraint_edtrange;\n RAISE NOTICE 'Creating partition % %', _partition_name, _partition_dtrange;\n IF c.partition_trunc IS NULL THEN\n q := format(\n $q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n GRANT ALL ON %I to pgstac_ingest;\n $q$,\n _partition_name,\n _collection,\n concat(_partition_name,'_pk'),\n _partition_name,\n _partition_name\n );\n ELSE\n q := format(\n $q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L) PARTITION BY RANGE (datetime);\n CREATE TABLE IF NOT EXISTS %I partition OF %I FOR VALUES FROM (%L) TO (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n GRANT ALL ON %I TO pgstac_ingest;\n $q$,\n format('_items_%s', c.key),\n _collection,\n _partition_name,\n format('_items_%s', c.key),\n lower(_partition_dtrange),\n upper(_partition_dtrange),\n format('%s_pk', _partition_name),\n _partition_name,\n _partition_name\n );\n END IF;\n\n BEGIN\n EXECUTE q;\n EXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', _partition_name;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\n END;\n PERFORM maintain_partitions(_partition_name);\n PERFORM update_partition_stats_q(_partition_name, true);\n REFRESH MATERIALIZED VIEW partitions;\n REFRESH MATERIALIZED VIEW partition_steps;\n RETURN _partition_name;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.cql2_query(j jsonb, wrapper text DEFAULT NULL::text)\n RETURNS text\n LANGUAGE plpgsql\n STABLE\nAS $function$\n#variable_conflict use_variable\nDECLARE\n args jsonb := j->'args';\n arg jsonb;\n op text := lower(j->>'op');\n cql2op RECORD;\n literal text;\n _wrapper text;\n leftarg text;\n rightarg text;\n prop text;\n extra_props bool := pgstac.additional_properties();\nBEGIN\n IF j IS NULL OR (op IS NOT NULL AND args IS NULL) THEN\n RETURN NULL;\n END IF;\n RAISE NOTICE 'CQL2_QUERY: %', j;\n\n -- check if all properties are represented in the queryables\n IF NOT extra_props THEN\n FOR prop IN\n SELECT DISTINCT p->>0\n FROM jsonb_path_query(j, 'strict $.**.property') p\n WHERE p->>0 NOT IN ('id', 'datetime', 'end_datetime', 'collection')\n LOOP\n IF (queryable(prop)).nulled_wrapper IS NULL THEN\n RAISE EXCEPTION 'Term % is not found in queryables.', prop;\n END IF;\n END LOOP;\n END IF;\n\n IF j ? 'filter' THEN\n RETURN cql2_query(j->'filter');\n END IF;\n\n IF j ? 'upper' THEN\n RETURN cql2_query(jsonb_build_object('op', 'upper', 'args', j->'upper'));\n END IF;\n\n IF j ? 'lower' THEN\n RETURN cql2_query(jsonb_build_object('op', 'lower', 'args', j->'lower'));\n END IF;\n\n -- Temporal Query\n IF op ilike 't_%' or op = 'anyinteracts' THEN\n RETURN temporal_op_query(op, args);\n END IF;\n\n -- If property is a timestamp convert it to text to use with\n -- general operators\n IF j ? 'timestamp' THEN\n RETURN format('%L::timestamptz', to_tstz(j->'timestamp'));\n END IF;\n IF j ? 'interval' THEN\n RAISE EXCEPTION 'Please use temporal operators when using intervals.';\n RETURN NONE;\n END IF;\n\n -- Spatial Query\n IF op ilike 's_%' or op = 'intersects' THEN\n RETURN spatial_op_query(op, args);\n END IF;\n\n IF op IN ('a_equals','a_contains','a_contained_by','a_overlaps') THEN\n IF args->0 ? 'property' THEN\n leftarg := format('to_text_array(%s)', (queryable(args->0->>'property')).path);\n END IF;\n IF args->1 ? 'property' THEN\n rightarg := format('to_text_array(%s)', (queryable(args->1->>'property')).path);\n END IF;\n RETURN FORMAT(\n '%s %s %s',\n COALESCE(leftarg, quote_literal(to_text_array(args->0))),\n CASE op\n WHEN 'a_equals' THEN '='\n WHEN 'a_contains' THEN '@>'\n WHEN 'a_contained_by' THEN '<@'\n WHEN 'a_overlaps' THEN '&&'\n END,\n COALESCE(rightarg, quote_literal(to_text_array(args->1)))\n );\n END IF;\n\n IF op = 'in' THEN\n RAISE NOTICE 'IN : % % %', args, jsonb_build_array(args->0), args->1;\n args := jsonb_build_array(args->0) || (args->1);\n RAISE NOTICE 'IN2 : %', args;\n END IF;\n\n\n\n IF op = 'between' THEN\n args = jsonb_build_array(\n args->0,\n args->1->0,\n args->1->1\n );\n END IF;\n\n -- Make sure that args is an array and run cql2_query on\n -- each element of the array\n RAISE NOTICE 'ARGS PRE: %', args;\n IF j ? 'args' THEN\n IF jsonb_typeof(args) != 'array' THEN\n args := jsonb_build_array(args);\n END IF;\n\n IF jsonb_path_exists(args, '$[*] ? (@.property == \"id\" || @.property == \"datetime\" || @.property == \"end_datetime\" || @.property == \"collection\")') THEN\n wrapper := NULL;\n ELSE\n -- if any of the arguments are a property, try to get the property_wrapper\n FOR arg IN SELECT jsonb_path_query(args, '$[*] ? (@.property != null)') LOOP\n RAISE NOTICE 'Arg: %', arg;\n wrapper := (queryable(arg->>'property')).nulled_wrapper;\n RAISE NOTICE 'Property: %, Wrapper: %', arg, wrapper;\n IF wrapper IS NOT NULL THEN\n EXIT;\n END IF;\n END LOOP;\n\n -- if the property was not in queryables, see if any args were numbers\n IF\n wrapper IS NULL\n AND jsonb_path_exists(args, '$[*] ? (@.type()==\"number\")')\n THEN\n wrapper := 'to_float';\n END IF;\n wrapper := coalesce(wrapper, 'to_text');\n END IF;\n\n SELECT jsonb_agg(cql2_query(a, wrapper))\n INTO args\n FROM jsonb_array_elements(args) a;\n END IF;\n RAISE NOTICE 'ARGS: %', args;\n\n IF op IN ('and', 'or') THEN\n RETURN\n format(\n '(%s)',\n array_to_string(to_text_array(args), format(' %s ', upper(op)))\n );\n END IF;\n\n IF op = 'in' THEN\n RAISE NOTICE 'IN -- % %', args->0, to_text(args->0);\n RETURN format(\n '%s IN (%s)',\n to_text(args->0),\n array_to_string((to_text_array(args))[2:], ',')\n );\n END IF;\n\n -- Look up template from cql2_ops\n IF j ? 'op' THEN\n SELECT * INTO cql2op FROM cql2_ops WHERE cql2_ops.op ilike op;\n IF FOUND THEN\n -- If specific index set in queryables for a property cast other arguments to that type\n\n RETURN format(\n cql2op.template,\n VARIADIC (to_text_array(args))\n );\n ELSE\n RAISE EXCEPTION 'Operator % Not Supported.', op;\n END IF;\n END IF;\n\n\n IF wrapper IS NOT NULL THEN\n RAISE NOTICE 'Wrapping % with %', j, wrapper;\n IF j ? 'property' THEN\n RETURN format('%I(%s)', wrapper, (queryable(j->>'property')).path);\n ELSE\n RETURN format('%I(%L)', wrapper, j);\n END IF;\n ELSIF j ? 'property' THEN\n RETURN quote_ident(j->>'property');\n END IF;\n\n RETURN quote_literal(to_text(j));\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.geometrysearch(geom geometry, queryhash text, fields jsonb DEFAULT NULL::jsonb, _scanlimit integer DEFAULT 10000, _limit integer DEFAULT 100, _timelimit interval DEFAULT '00:00:05'::interval, exitwhenfull boolean DEFAULT true, skipcovered boolean DEFAULT true)\n RETURNS jsonb\n LANGUAGE plpgsql\nAS $function$\nDECLARE\n search searches%ROWTYPE;\n curs refcursor;\n _where text;\n query text;\n iter_record items%ROWTYPE;\n out_records jsonb := '{}'::jsonb[];\n exit_flag boolean := FALSE;\n counter int := 1;\n scancounter int := 1;\n remaining_limit int := _scanlimit;\n tilearea float;\n unionedgeom geometry;\n clippedgeom geometry;\n unionedgeom_area float := 0;\n prev_area float := 0;\n excludes text[];\n includes text[];\n\nBEGIN\n DROP TABLE IF EXISTS pgstac_results;\n CREATE TEMP TABLE pgstac_results (content jsonb) ON COMMIT DROP;\n\n -- If the passed in geometry is not an area set exitwhenfull and skipcovered to false\n IF ST_GeometryType(geom) !~* 'polygon' THEN\n RAISE NOTICE 'GEOMETRY IS NOT AN AREA';\n skipcovered = FALSE;\n exitwhenfull = FALSE;\n END IF;\n\n -- If skipcovered is true then you will always want to exit when the passed in geometry is full\n IF skipcovered THEN\n exitwhenfull := TRUE;\n END IF;\n\n SELECT * INTO search FROM searches WHERE hash=queryhash;\n\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Search with Query Hash % Not Found', queryhash;\n END IF;\n\n tilearea := st_area(geom);\n _where := format('%s AND st_intersects(geometry, %L::geometry)', search._where, geom);\n\n\n FOR query IN SELECT * FROM partition_queries(_where, search.orderby) LOOP\n query := format('%s LIMIT %L', query, remaining_limit);\n RAISE NOTICE '%', query;\n OPEN curs FOR EXECUTE query;\n LOOP\n FETCH curs INTO iter_record;\n EXIT WHEN NOT FOUND;\n IF exitwhenfull OR skipcovered THEN -- If we are not using exitwhenfull or skipcovered, we do not need to do expensive geometry operations\n clippedgeom := st_intersection(geom, iter_record.geometry);\n\n IF unionedgeom IS NULL THEN\n unionedgeom := clippedgeom;\n ELSE\n unionedgeom := st_union(unionedgeom, clippedgeom);\n END IF;\n\n unionedgeom_area := st_area(unionedgeom);\n\n IF skipcovered AND prev_area = unionedgeom_area THEN\n scancounter := scancounter + 1;\n CONTINUE;\n END IF;\n\n prev_area := unionedgeom_area;\n\n RAISE NOTICE '% % % %', unionedgeom_area/tilearea, counter, scancounter, ftime();\n END IF;\n RAISE NOTICE '% %', iter_record, content_hydrate(iter_record, fields);\n INSERT INTO pgstac_results (content) VALUES (content_hydrate(iter_record, fields));\n\n IF counter >= _limit\n OR scancounter > _scanlimit\n OR ftime() > _timelimit\n OR (exitwhenfull AND unionedgeom_area >= tilearea)\n THEN\n exit_flag := TRUE;\n EXIT;\n END IF;\n counter := counter + 1;\n scancounter := scancounter + 1;\n\n END LOOP;\n CLOSE curs;\n EXIT WHEN exit_flag;\n remaining_limit := _scanlimit - scancounter;\n END LOOP;\n\n SELECT jsonb_agg(content) INTO out_records FROM pgstac_results WHERE content IS NOT NULL;\n\n RETURN jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb)\n );\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.get_queryables(_collection_ids text[] DEFAULT NULL::text[])\n RETURNS jsonb\n LANGUAGE plpgsql\n STABLE\nAS $function$\nDECLARE\nBEGIN\n -- Build up queryables if the input contains valid collection ids or is empty\n IF EXISTS (\n SELECT 1 FROM collections\n WHERE\n _collection_ids IS NULL\n OR cardinality(_collection_ids) = 0\n OR id = ANY(_collection_ids)\n )\n THEN\n RETURN (\n WITH base AS (\n SELECT\n unnest(collection_ids) as collection_id,\n name,\n coalesce(definition, '{\"type\":\"string\"}'::jsonb) as definition\n FROM queryables\n WHERE\n _collection_ids IS NULL OR\n _collection_ids = '{}'::text[] OR\n _collection_ids && collection_ids\n UNION ALL\n SELECT null, name, coalesce(definition, '{\"type\":\"string\"}'::jsonb) as definition\n FROM queryables WHERE collection_ids IS NULL OR collection_ids = '{}'::text[]\n ), g AS (\n SELECT\n name,\n first_notnull(definition) as definition,\n jsonb_array_unique_merge(definition->'enum') as enum,\n jsonb_min(definition->'minimum') as minimum,\n jsonb_min(definition->'maxiumn') as maximum\n FROM base\n GROUP BY 1\n )\n SELECT\n jsonb_build_object(\n '$schema', 'http://json-schema.org/draft-07/schema#',\n '$id', '',\n 'type', 'object',\n 'title', 'STAC Queryables.',\n 'properties', jsonb_object_agg(\n name,\n definition\n ||\n jsonb_strip_nulls(jsonb_build_object(\n 'enum', enum,\n 'minimum', minimum,\n 'maximum', maximum\n ))\n ),\n 'additionalProperties', pgstac.additional_properties()\n )\n FROM g\n );\n ELSE\n RETURN NULL;\n END IF;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.maintain_partition_queries(part text DEFAULT 'items'::text, dropindexes boolean DEFAULT false, rebuildindexes boolean DEFAULT false, idxconcurrently boolean DEFAULT false)\n RETURNS SETOF text\n LANGUAGE plpgsql\nAS $function$\nDECLARE\n rec record;\n q text;\nBEGIN\n FOR rec IN (\n SELECT * FROM queryable_indexes(part,true)\n ) LOOP\n q := format(\n 'SELECT maintain_index(\n %L,%L,%L,%L,%L\n );',\n rec.indexname,\n rec.queryable_idx,\n dropindexes,\n rebuildindexes,\n idxconcurrently\n );\n RAISE NOTICE 'Q: %s', q;\n RETURN NEXT q;\n END LOOP;\n RETURN;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.repartition(_collection text, _partition_trunc text, triggered boolean DEFAULT false)\n RETURNS text\n LANGUAGE plpgsql\n SECURITY DEFINER\nAS $function$\nDECLARE\n c RECORD;\n q text;\n from_trunc text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=_collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', _collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n IF triggered THEN\n RAISE NOTICE 'Converting % to % partitioning via Trigger', _collection, _partition_trunc;\n ELSE\n RAISE NOTICE 'Converting % from using % to % partitioning', _collection, c.partition_trunc, _partition_trunc;\n IF c.partition_trunc IS NOT DISTINCT FROM _partition_trunc THEN\n RAISE NOTICE 'Collection % already set to use partition by %', _collection, _partition_trunc;\n RETURN _collection;\n END IF;\n END IF;\n\n IF EXISTS (SELECT 1 FROM partitions_view WHERE collection=_collection LIMIT 1) THEN\n EXECUTE format(\n $q$\n CREATE TEMP TABLE changepartitionstaging ON COMMIT DROP AS SELECT * FROM %I;\n DROP TABLE IF EXISTS %I CASCADE;\n WITH p AS (\n SELECT\n collection,\n CASE\n WHEN %L IS NULL THEN '-infinity'::timestamptz\n ELSE date_trunc(%L, datetime)\n END as d,\n tstzrange(min(datetime),max(datetime),'[]') as dtrange,\n tstzrange(min(end_datetime),max(end_datetime),'[]') as edtrange\n FROM changepartitionstaging\n GROUP BY 1,2\n ) SELECT check_partition(collection, dtrange, edtrange) FROM p;\n INSERT INTO items SELECT * FROM changepartitionstaging;\n DROP TABLE changepartitionstaging;\n $q$,\n concat('_items_', c.key),\n concat('_items_', c.key),\n c.partition_trunc,\n c.partition_trunc\n );\n END IF;\n RETURN _collection;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.search_query(_search jsonb DEFAULT '{}'::jsonb, updatestats boolean DEFAULT false, _metadata jsonb DEFAULT '{}'::jsonb)\n RETURNS searches\n LANGUAGE plpgsql\n SECURITY DEFINER\nAS $function$\nDECLARE\n search searches%ROWTYPE;\n pexplain jsonb;\n t timestamptz;\n i interval;\n _hash text := search_hash(_search, _metadata);\n doupdate boolean := FALSE;\n insertfound boolean := FALSE;\n ro boolean := pgstac.readonly();\nBEGIN\n IF ro THEN\n updatestats := FALSE;\n END IF;\n\n SELECT * INTO search FROM searches\n WHERE hash=_hash;\n\n search.hash := _hash;\n\n -- Calculate the where clause if not already calculated\n IF search._where IS NULL THEN\n search._where := stac_search_to_where(_search);\n ELSE\n doupdate := TRUE;\n END IF;\n\n -- Calculate the order by clause if not already calculated\n IF search.orderby IS NULL THEN\n search.orderby := sort_sqlorderby(_search);\n ELSE\n doupdate := TRUE;\n END IF;\n\n PERFORM where_stats(search._where, updatestats, _search->'conf');\n\n IF NOT ro THEN\n IF NOT doupdate THEN\n INSERT INTO searches (search, _where, orderby, lastused, usecount, metadata)\n VALUES (_search, search._where, search.orderby, clock_timestamp(), 1, _metadata)\n ON CONFLICT (hash) DO NOTHING RETURNING * INTO search;\n IF FOUND THEN\n RETURN search;\n END IF;\n END IF;\n\n UPDATE searches\n SET\n lastused=clock_timestamp(),\n usecount=usecount+1\n WHERE hash=(\n SELECT hash FROM searches\n WHERE hash=_hash\n FOR UPDATE SKIP LOCKED\n );\n IF NOT FOUND THEN\n RAISE NOTICE 'Did not update stats for % due to lock. (This is generally OK)', _search;\n END IF;\n END IF;\n\n RETURN search;\n\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.update_partition_stats(_partition text, istrigger boolean DEFAULT false)\n RETURNS void\n LANGUAGE plpgsql\n STRICT SECURITY DEFINER\nAS $function$\nDECLARE\n dtrange tstzrange;\n edtrange tstzrange;\n cdtrange tstzrange;\n cedtrange tstzrange;\n extent geometry;\n collection text;\nBEGIN\n RAISE NOTICE 'Updating stats for %.', _partition;\n EXECUTE format(\n $q$\n SELECT\n tstzrange(min(datetime), max(datetime),'[]'),\n tstzrange(min(end_datetime), max(end_datetime), '[]')\n FROM %I\n $q$,\n _partition\n ) INTO dtrange, edtrange;\n extent := st_estimatedextent('pgstac', _partition, 'geometry');\n INSERT INTO partition_stats (partition, dtrange, edtrange, spatial, last_updated)\n SELECT _partition, dtrange, edtrange, extent, now()\n ON CONFLICT (partition) DO\n UPDATE SET\n dtrange=EXCLUDED.dtrange,\n edtrange=EXCLUDED.edtrange,\n spatial=EXCLUDED.spatial,\n last_updated=EXCLUDED.last_updated\n ;\n\n SELECT\n constraint_dtrange, constraint_edtrange, pv.collection\n INTO cdtrange, cedtrange, collection\n FROM partitions_view pv WHERE partition = _partition;\n REFRESH MATERIALIZED VIEW partitions;\n REFRESH MATERIALIZED VIEW partition_steps;\n\n\n RAISE NOTICE 'Checking if we need to modify constraints...';\n RAISE NOTICE 'cdtrange: % dtrange: % cedtrange: % edtrange: %',cdtrange, dtrange, cedtrange, edtrange;\n IF\n (cdtrange IS DISTINCT FROM dtrange OR edtrange IS DISTINCT FROM cedtrange)\n AND NOT istrigger\n THEN\n RAISE NOTICE 'Modifying Constraints';\n RAISE NOTICE 'Existing % %', cdtrange, cedtrange;\n RAISE NOTICE 'New % %', dtrange, edtrange;\n PERFORM drop_table_constraints(_partition);\n PERFORM create_table_constraints(_partition, dtrange, edtrange);\n REFRESH MATERIALIZED VIEW partitions;\n REFRESH MATERIALIZED VIEW partition_steps;\n END IF;\n RAISE NOTICE 'Checking if we need to update collection extents.';\n IF get_setting_bool('update_collection_extent') THEN\n RAISE NOTICE 'updating collection extent for %', collection;\n PERFORM run_or_queue(format($q$\n UPDATE collections\n SET content = jsonb_set_lax(\n content,\n '{extent}'::text[],\n collection_extent(%L, FALSE),\n true,\n 'use_json_null'\n ) WHERE id=%L\n ;\n $q$, collection, collection));\n ELSE\n RAISE NOTICE 'Not updating collection extent for %', collection;\n END IF;\n\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.where_stats(inwhere text, updatestats boolean DEFAULT false, conf jsonb DEFAULT NULL::jsonb)\n RETURNS search_wheres\n LANGUAGE plpgsql\n SECURITY DEFINER\nAS $function$\nDECLARE\n t timestamptz;\n i interval;\n explain_json jsonb;\n partitions text[];\n sw search_wheres%ROWTYPE;\n inwhere_hash text := md5(inwhere);\n _context text := lower(context(conf));\n _stats_ttl interval := context_stats_ttl(conf);\n _estimated_cost float := context_estimated_cost(conf);\n _estimated_count int := context_estimated_count(conf);\n ro bool := pgstac.readonly(conf);\nBEGIN\n IF ro THEN\n updatestats := FALSE;\n END IF;\n\n IF _context = 'off' THEN\n sw._where := inwhere;\n return sw;\n END IF;\n\n SELECT * INTO sw FROM search_wheres WHERE md5(_where)=inwhere_hash FOR UPDATE;\n\n -- Update statistics if explicitly set, if statistics do not exist, or statistics ttl has expired\n IF NOT updatestats THEN\n RAISE NOTICE 'Checking if update is needed for: % .', inwhere;\n RAISE NOTICE 'Stats Last Updated: %', sw.statslastupdated;\n RAISE NOTICE 'TTL: %, Age: %', _stats_ttl, now() - sw.statslastupdated;\n RAISE NOTICE 'Context: %, Existing Total: %', _context, sw.total_count;\n IF\n (\n sw.statslastupdated IS NULL\n OR (now() - sw.statslastupdated) > _stats_ttl\n OR (context(conf) != 'off' AND sw.total_count IS NULL)\n ) AND NOT ro\n THEN\n updatestats := TRUE;\n END IF;\n END IF;\n\n sw._where := inwhere;\n sw.lastused := now();\n sw.usecount := coalesce(sw.usecount,0) + 1;\n\n IF NOT updatestats THEN\n UPDATE search_wheres SET\n lastused = sw.lastused,\n usecount = sw.usecount\n WHERE md5(_where) = inwhere_hash\n RETURNING * INTO sw\n ;\n RETURN sw;\n END IF;\n\n -- Use explain to get estimated count/cost and a list of the partitions that would be hit by the query\n t := clock_timestamp();\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s', inwhere)\n INTO explain_json;\n RAISE NOTICE 'Time for just the explain: %', clock_timestamp() - t;\n i := clock_timestamp() - t;\n\n sw.statslastupdated := now();\n sw.estimated_count := explain_json->0->'Plan'->'Plan Rows';\n sw.estimated_cost := explain_json->0->'Plan'->'Total Cost';\n sw.time_to_estimate := extract(epoch from i);\n\n RAISE NOTICE 'ESTIMATED_COUNT: % < %', sw.estimated_count, _estimated_count;\n RAISE NOTICE 'ESTIMATED_COST: % < %', sw.estimated_cost, _estimated_cost;\n\n -- Do a full count of rows if context is set to on or if auto is set and estimates are low enough\n IF\n _context = 'on'\n OR\n ( _context = 'auto' AND\n (\n sw.estimated_count < _estimated_count\n AND\n sw.estimated_cost < _estimated_cost\n )\n )\n THEN\n t := clock_timestamp();\n RAISE NOTICE 'Calculating actual count...';\n EXECUTE format(\n 'SELECT count(*) FROM items WHERE %s',\n inwhere\n ) INTO sw.total_count;\n i := clock_timestamp() - t;\n RAISE NOTICE 'Actual Count: % -- %', sw.total_count, i;\n sw.time_to_count := extract(epoch FROM i);\n ELSE\n sw.total_count := NULL;\n sw.time_to_count := NULL;\n END IF;\n\n IF NOT ro THEN\n INSERT INTO search_wheres\n (_where, lastused, usecount, statslastupdated, estimated_count, estimated_cost, time_to_estimate, partitions, total_count, time_to_count)\n SELECT sw._where, sw.lastused, sw.usecount, sw.statslastupdated, sw.estimated_count, sw.estimated_cost, sw.time_to_estimate, sw.partitions, sw.total_count, sw.time_to_count\n ON CONFLICT ((md5(_where)))\n DO UPDATE\n SET\n lastused = sw.lastused,\n usecount = sw.usecount,\n statslastupdated = sw.statslastupdated,\n estimated_count = sw.estimated_count,\n estimated_cost = sw.estimated_cost,\n time_to_estimate = sw.time_to_estimate,\n total_count = sw.total_count,\n time_to_count = sw.time_to_count\n ;\n END IF;\n RETURN sw;\nEND;\n$function$\n;\n\n\n-- END migra calculated SQL\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('id', '{\"title\": \"Item ID\",\"description\": \"Item identifier\",\"$ref\": \"https://schemas.stacspec.org/v1.0.0/item-spec/json-schema/item.json#/definitions/core/allOf/2/properties/id\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('geometry', '{\"title\": \"Item Geometry\",\"description\": \"Item Geometry\",\"$ref\": \"https://geojson.org/schema/Feature.json\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('datetime','{\"description\": \"Datetime\",\"type\": \"string\",\"title\": \"Acquired\",\"format\": \"date-time\",\"pattern\": \"(\\\\+00:00|Z)$\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDELETE FROM queryables a USING queryables b\n WHERE a.name = b.name AND a.collection_ids IS NOT DISTINCT FROM b.collection_ids AND a.id > b.id;\n\n\nINSERT INTO pgstac_settings (name, value) VALUES\n ('context', 'off'),\n ('context_estimated_count', '100000'),\n ('context_estimated_cost', '100000'),\n ('context_stats_ttl', '1 day'),\n ('default_filter_lang', 'cql2-json'),\n ('additional_properties', 'true'),\n ('use_queue', 'false'),\n ('queue_timeout', '10 minutes'),\n ('update_collection_extent', 'false'),\n ('format_cache', 'false'),\n ('readonly', 'false')\nON CONFLICT DO NOTHING\n;\n\nALTER FUNCTION to_text COST 5000;\nALTER FUNCTION to_float COST 5000;\nALTER FUNCTION to_int COST 5000;\nALTER FUNCTION to_tstz COST 5000;\nALTER FUNCTION to_text_array COST 5000;\n\nALTER FUNCTION update_partition_stats SECURITY DEFINER;\nALTER FUNCTION partition_after_triggerfunc SECURITY DEFINER;\nALTER FUNCTION drop_table_constraints SECURITY DEFINER;\nALTER FUNCTION create_table_constraints SECURITY DEFINER;\nALTER FUNCTION check_partition SECURITY DEFINER;\nALTER FUNCTION repartition SECURITY DEFINER;\nALTER FUNCTION where_stats SECURITY DEFINER;\nALTER FUNCTION search_query SECURITY DEFINER;\nALTER FUNCTION format_item SECURITY DEFINER;\nALTER FUNCTION maintain_index SECURITY DEFINER;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nGRANT ALL ON SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac to pgstac_admin;\n\n-- pgstac_read role limited to using function apis\nGRANT EXECUTE ON FUNCTION search TO pgstac_read;\nGRANT EXECUTE ON FUNCTION search_query TO pgstac_read;\nGRANT EXECUTE ON FUNCTION item_by_id TO pgstac_read;\nGRANT EXECUTE ON FUNCTION get_item TO pgstac_read;\nGRANT SELECT ON ALL TABLES IN SCHEMA pgstac TO pgstac_read;\n\n\nGRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_ingest;\nGRANT USAGE ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_ingest;\n\nREVOKE ALL PRIVILEGES ON PROCEDURE run_queued_queries FROM public;\nGRANT ALL ON PROCEDURE run_queued_queries TO pgstac_admin;\n\nREVOKE ALL PRIVILEGES ON FUNCTION run_queued_queries_intransaction FROM public;\nGRANT ALL ON FUNCTION run_queued_queries_intransaction TO pgstac_admin;\n\nRESET ROLE;\n\nSET ROLE pgstac_ingest;\nSELECT update_partition_stats_q(partition) FROM partitions_view;\nSELECT set_version('0.8.2');\n" }, { "path": "src/pgstac/migrations/pgstac.0.8.1.sql", "content": "RESET ROLE;\nDO $$\nDECLARE\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='postgis') THEN\n CREATE EXTENSION IF NOT EXISTS postgis;\n END IF;\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='btree_gist') THEN\n CREATE EXTENSION IF NOT EXISTS btree_gist;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_admin;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_read;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_ingest;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n\nGRANT pgstac_admin TO current_user;\n\n-- Function to make sure pgstac_admin is the owner of items\nCREATE OR REPLACE FUNCTION pgstac_admin_owns() RETURNS VOID AS $$\nDECLARE\n f RECORD;\nBEGIN\n FOR f IN (\n SELECT\n concat(\n oid::regproc::text,\n '(',\n coalesce(pg_get_function_identity_arguments(oid),''),\n ')'\n ) AS name,\n CASE prokind WHEN 'f' THEN 'FUNCTION' WHEN 'p' THEN 'PROCEDURE' WHEN 'a' THEN 'AGGREGATE' END as typ\n FROM pg_proc\n WHERE\n pronamespace=to_regnamespace('pgstac')\n AND proowner != to_regrole('pgstac_admin')\n AND proname NOT LIKE 'pg_stat%'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n FOR f IN (\n SELECT\n oid::regclass::text as name,\n CASE relkind\n WHEN 'i' THEN 'INDEX'\n WHEN 'I' THEN 'INDEX'\n WHEN 'p' THEN 'TABLE'\n WHEN 'r' THEN 'TABLE'\n WHEN 'v' THEN 'VIEW'\n WHEN 'S' THEN 'SEQUENCE'\n ELSE NULL\n END as typ\n FROM pg_class\n WHERE relnamespace=to_regnamespace('pgstac') and relowner != to_regrole('pgstac_admin') AND relkind IN ('r','p','v','S') AND relname NOT LIKE 'pg_stat'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\nSELECT pgstac_admin_owns();\n\nCREATE SCHEMA IF NOT EXISTS pgstac AUTHORIZATION pgstac_admin;\n\nGRANT ALL ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_admin;\n\nALTER ROLE pgstac_admin SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_read SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_ingest SET SEARCH_PATH TO pgstac, public;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\n\nGRANT pgstac_read TO pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\n\nSET ROLE pgstac_admin;\n\nSET SEARCH_PATH TO pgstac, public;\n\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS analyze_items;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS validate_constraints;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n-- Install these idempotently as migrations do not put them before trying to modify the collections table\n\n\nCREATE OR REPLACE FUNCTION collection_geom(content jsonb)\nRETURNS geometry AS $$\n WITH box AS (SELECT content->'extent'->'spatial'->'bbox'->0 as box)\n SELECT\n st_makeenvelope(\n (box->>0)::float,\n (box->>1)::float,\n (box->>2)::float,\n (box->>3)::float,\n 4326\n )\n FROM box;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION collection_datetime(content jsonb)\nRETURNS timestamptz AS $$\n SELECT\n CASE\n WHEN\n (content->'extent'->'temporal'->'interval'->0->>0) IS NULL\n THEN '-infinity'::timestamptz\n ELSE\n (content->'extent'->'temporal'->'interval'->0->>0)::timestamptz\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION collection_enddatetime(content jsonb)\nRETURNS timestamptz AS $$\n SELECT\n CASE\n WHEN\n (content->'extent'->'temporal'->'interval'->0->>1) IS NULL\n THEN 'infinity'::timestamptz\n ELSE\n (content->'extent'->'temporal'->'interval'->0->>1)::timestamptz\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE TABLE IF NOT EXISTS migrations (\n version text PRIMARY KEY,\n datetime timestamptz DEFAULT clock_timestamp() NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION get_version() RETURNS text AS $$\n SELECT version FROM pgstac.migrations ORDER BY datetime DESC, version DESC LIMIT 1;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION set_version(text) RETURNS text AS $$\n INSERT INTO pgstac.migrations (version) VALUES ($1)\n ON CONFLICT DO NOTHING\n RETURNING version;\n$$ LANGUAGE SQL;\n\n\nCREATE TABLE IF NOT EXISTS pgstac_settings (\n name text PRIMARY KEY,\n value text NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION table_empty(text) RETURNS boolean AS $$\nDECLARE\n retval boolean;\nBEGIN\n EXECUTE format($q$\n SELECT NOT EXISTS (SELECT 1 FROM %I LIMIT 1)\n $q$,\n $1\n ) INTO retval;\n RETURN retval;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION get_setting(IN _setting text, IN conf jsonb DEFAULT NULL) RETURNS text AS $$\nSELECT COALESCE(\n nullif(conf->>_setting, ''),\n nullif(current_setting(concat('pgstac.',_setting), TRUE),''),\n nullif((SELECT value FROM pgstac.pgstac_settings WHERE name=_setting),'')\n);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_setting_bool(IN _setting text, IN conf jsonb DEFAULT NULL) RETURNS boolean AS $$\nSELECT COALESCE(\n nullif(conf->>_setting, ''),\n nullif(current_setting(concat('pgstac.',_setting), TRUE),''),\n nullif((SELECT value FROM pgstac.pgstac_settings WHERE name=_setting),''),\n 'FALSE'\n)::boolean;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context(conf jsonb DEFAULT NULL) RETURNS text AS $$\n SELECT pgstac.get_setting('context', conf);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context_estimated_count(conf jsonb DEFAULT NULL) RETURNS int AS $$\n SELECT pgstac.get_setting('context_estimated_count', conf)::int;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_estimated_cost();\nCREATE OR REPLACE FUNCTION context_estimated_cost(conf jsonb DEFAULT NULL) RETURNS float AS $$\n SELECT pgstac.get_setting('context_estimated_cost', conf)::float;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_stats_ttl();\nCREATE OR REPLACE FUNCTION context_stats_ttl(conf jsonb DEFAULT NULL) RETURNS interval AS $$\n SELECT pgstac.get_setting('context_stats_ttl', conf)::interval;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION t2s(text) RETURNS text AS $$\n SELECT extract(epoch FROM $1::interval)::text || ' s';\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE STRICT;\n\nCREATE OR REPLACE FUNCTION age_ms(a timestamptz, b timestamptz DEFAULT clock_timestamp()) RETURNS float AS $$\n SELECT abs(extract(epoch from age(a,b)) * 1000);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION queue_timeout() RETURNS interval AS $$\n SELECT t2s(coalesce(\n get_setting('queue_timeout'),\n '1h'\n ))::interval;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION notice(VARIADIC text[]) RETURNS boolean AS $$\nDECLARE\ndebug boolean := current_setting('pgstac.debug', true);\nBEGIN\n IF debug THEN\n RAISE NOTICE 'NOTICE FROM FUNC: % >>>>> %', concat_ws(' | ', $1), clock_timestamp();\n RETURN TRUE;\n END IF;\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION empty_arr(ANYARRAY) RETURNS BOOLEAN AS $$\nSELECT CASE\n WHEN $1 IS NULL THEN TRUE\n WHEN cardinality($1)<1 THEN TRUE\nELSE FALSE\nEND;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_intersection(_a ANYARRAY, _b ANYARRAY) RETURNS ANYARRAY AS $$\n SELECT ARRAY ( SELECT unnest(_a) INTERSECT SELECT UNNEST(_b) );\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION array_map_ident(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_ident(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_map_literal(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_literal(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION array_reverse(anyarray) RETURNS anyarray AS $$\nSELECT ARRAY(\n SELECT $1[i]\n FROM generate_subscripts($1,1) AS s(i)\n ORDER BY i DESC\n);\n$$ LANGUAGE SQL STRICT IMMUTABLE;\n\nDROP TABLE IF EXISTS query_queue;\nCREATE TABLE query_queue (\n query text PRIMARY KEY,\n added timestamptz DEFAULT now()\n);\n\nDROP TABLE IF EXISTS query_queue_history;\nCREATE TABLE query_queue_history(\n query text,\n added timestamptz NOT NULL,\n finished timestamptz NOT NULL DEFAULT now(),\n error text\n);\n\nCREATE OR REPLACE PROCEDURE run_queued_queries() AS $$\nDECLARE\n qitem query_queue%ROWTYPE;\n timeout_ts timestamptz;\n error text;\n cnt int := 0;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n DELETE FROM query_queue WHERE query = (SELECT query FROM query_queue ORDER BY added DESC LIMIT 1 FOR UPDATE SKIP LOCKED) RETURNING * INTO qitem;\n IF NOT FOUND THEN\n EXIT;\n END IF;\n cnt := cnt + 1;\n BEGIN\n RAISE NOTICE 'RUNNING QUERY: %', qitem.query;\n EXECUTE qitem.query;\n EXCEPTION WHEN others THEN\n error := format('%s | %s', SQLERRM, SQLSTATE);\n END;\n INSERT INTO query_queue_history (query, added, finished, error)\n VALUES (qitem.query, qitem.added, clock_timestamp(), error);\n COMMIT;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION run_queued_queries_intransaction() RETURNS int AS $$\nDECLARE\n qitem query_queue%ROWTYPE;\n timeout_ts timestamptz;\n error text;\n cnt int := 0;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n DELETE FROM query_queue WHERE query = (SELECT query FROM query_queue ORDER BY added DESC LIMIT 1 FOR UPDATE SKIP LOCKED) RETURNING * INTO qitem;\n IF NOT FOUND THEN\n RETURN cnt;\n END IF;\n cnt := cnt + 1;\n BEGIN\n qitem.query := regexp_replace(qitem.query, 'CONCURRENTLY', '');\n RAISE NOTICE 'RUNNING QUERY: %', qitem.query;\n\n EXECUTE qitem.query;\n EXCEPTION WHEN others THEN\n error := format('%s | %s', SQLERRM, SQLSTATE);\n RAISE WARNING '%', error;\n END;\n INSERT INTO query_queue_history (query, added, finished, error)\n VALUES (qitem.query, qitem.added, clock_timestamp(), error);\n END LOOP;\n RETURN cnt;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION run_or_queue(query text) RETURNS VOID AS $$\nDECLARE\n use_queue text := COALESCE(get_setting('use_queue'), 'FALSE')::boolean;\nBEGIN\n IF get_setting_bool('debug') THEN\n RAISE NOTICE '%', query;\n END IF;\n IF use_queue THEN\n INSERT INTO query_queue (query) VALUES (query) ON CONFLICT DO NOTHING;\n ELSE\n EXECUTE query;\n END IF;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n\nDROP FUNCTION IF EXISTS check_pgstac_settings;\nCREATE OR REPLACE FUNCTION check_pgstac_settings(_sysmem text DEFAULT NULL) RETURNS VOID AS $$\nDECLARE\n settingval text;\n sysmem bigint := pg_size_bytes(_sysmem);\n effective_cache_size bigint := pg_size_bytes(current_setting('effective_cache_size', TRUE));\n shared_buffers bigint := pg_size_bytes(current_setting('shared_buffers', TRUE));\n work_mem bigint := pg_size_bytes(current_setting('work_mem', TRUE));\n max_connections int := current_setting('max_connections', TRUE);\n maintenance_work_mem bigint := pg_size_bytes(current_setting('maintenance_work_mem', TRUE));\n seq_page_cost float := current_setting('seq_page_cost', TRUE);\n random_page_cost float := current_setting('random_page_cost', TRUE);\n temp_buffers bigint := pg_size_bytes(current_setting('temp_buffers', TRUE));\n r record;\nBEGIN\n IF _sysmem IS NULL THEN\n RAISE NOTICE 'Call function with the size of your system memory `SELECT check_pgstac_settings(''4GB'')` to get pg system setting recommendations.';\n ELSE\n IF effective_cache_size < (sysmem * 0.5) THEN\n RAISE WARNING 'effective_cache_size of % is set low for a system with %. Recomended value between % and %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.5), pg_size_pretty(sysmem * 0.75);\n ELSIF effective_cache_size > (sysmem * 0.75) THEN\n RAISE WARNING 'effective_cache_size of % is set high for a system with %. Recomended value between % and %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.5), pg_size_pretty(sysmem * 0.75);\n ELSE\n RAISE NOTICE 'effective_cache_size of % is set appropriately for a system with %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem);\n END IF;\n\n IF shared_buffers < (sysmem * 0.2) THEN\n RAISE WARNING 'shared_buffers of % is set low for a system with %. Recomended value between % and %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.2), pg_size_pretty(sysmem * 0.3);\n ELSIF shared_buffers > (sysmem * 0.3) THEN\n RAISE WARNING 'shared_buffers of % is set high for a system with %. Recomended value between % and %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.2), pg_size_pretty(sysmem * 0.3);\n ELSE\n RAISE NOTICE 'shared_buffers of % is set appropriately for a system with %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem);\n END IF;\n shared_buffers = sysmem * 0.3;\n IF maintenance_work_mem < (sysmem * 0.2) THEN\n RAISE WARNING 'maintenance_work_mem of % is set low for shared_buffers of %. Recomended value between % and %', pg_size_pretty(maintenance_work_mem), pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers * 0.2), pg_size_pretty(shared_buffers * 0.3);\n ELSIF maintenance_work_mem > (shared_buffers * 0.3) THEN\n RAISE WARNING 'maintenance_work_mem of % is set high for shared_buffers of %. Recomended value between % and %', pg_size_pretty(maintenance_work_mem), pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers * 0.2), pg_size_pretty(shared_buffers * 0.3);\n ELSE\n RAISE NOTICE 'maintenance_work_mem of % is set appropriately for shared_buffers of %', pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers);\n END IF;\n\n IF work_mem * max_connections > shared_buffers THEN\n RAISE WARNING 'work_mem setting of % is set high for % max_connections please reduce work_mem to % or decrease max_connections to %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers/max_connections), floor(shared_buffers/work_mem);\n ELSIF work_mem * max_connections < (shared_buffers * 0.75) THEN\n RAISE WARNING 'work_mem setting of % is set low for % max_connections you may consider raising work_mem to % or increasing max_connections to %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers/max_connections), floor(shared_buffers/work_mem);\n ELSE\n RAISE NOTICE 'work_mem setting of % and max_connections of % are adequate for shared_buffers of %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers);\n END IF;\n\n IF random_page_cost / seq_page_cost != 1.1 THEN\n RAISE WARNING 'random_page_cost (%) /seq_page_cost (%) should be set to 1.1 for SSD. Change random_page_cost to %', random_page_cost, seq_page_cost, 1.1 * seq_page_cost;\n ELSE\n RAISE NOTICE 'random_page_cost and seq_page_cost set appropriately for SSD';\n END IF;\n\n IF temp_buffers < greatest(pg_size_bytes('128MB'),(maintenance_work_mem / 2)) THEN\n RAISE WARNING 'pgstac makes heavy use of temp tables, consider raising temp_buffers from % to %', pg_size_pretty(temp_buffers), greatest('128MB', pg_size_pretty((shared_buffers / 16)));\n END IF;\n END IF;\n\n RAISE NOTICE 'VALUES FOR PGSTAC VARIABLES';\n RAISE NOTICE 'These can be set either as GUC system variables or by setting in the pgstac_settings table.';\n\n FOR r IN SELECT name, get_setting(name) as setting, CASE WHEN current_setting(concat('pgstac.',name), TRUE) IS NOT NULL THEN concat('pgstac.',name, ' GUC') WHEN value IS NOT NULL THEN 'pgstac_settings table' ELSE 'Not Set' END as loc FROM pgstac_settings LOOP\n RAISE NOTICE '% is set to % from the %', r.name, r.setting, r.loc;\n END LOOP;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pg_cron';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider intalling pg_cron which can be used to automate tasks';\n ELSE\n RAISE NOTICE 'pg_cron % is installed', settingval;\n END IF;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pgstattuple';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider installing the pgstattuple extension which can be used to help maintain tables and indexes.';\n ELSE\n RAISE NOTICE 'pgstattuple % is installed', settingval;\n END IF;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pg_stat_statements';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider installing the pg_stat_statements extension which is very helpful for tracking the types of queries on the system';\n ELSE\n RAISE NOTICE 'pg_stat_statements % is installed', settingval;\n IF current_setting('pg_stat_statements.track_statements', TRUE) IS DISTINCT FROM 'all' THEN\n RAISE WARNING 'SET pg_stat_statements.track_statements TO ''all''; --In order to track statements within functions.';\n END IF;\n END IF;\n\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac, public SET CLIENT_MIN_MESSAGES TO NOTICE;\nCREATE OR REPLACE FUNCTION to_int(jsonb) RETURNS int AS $$\n SELECT floor(($1->>0)::float)::int;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION to_float(jsonb) RETURNS float AS $$\n SELECT ($1->>0)::float;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION to_tstz(jsonb) RETURNS timestamptz AS $$\n SELECT ($1->>0)::timestamptz;\n$$ LANGUAGE SQL IMMUTABLE STRICT SET TIME ZONE 'UTC' COST 5000 PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION to_text(jsonb) RETURNS text AS $$\n SELECT CASE WHEN jsonb_typeof($1) IN ('array','object') THEN $1::text ELSE $1->>0 END;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION to_text_array(jsonb) RETURNS text[] AS $$\n SELECT\n CASE jsonb_typeof($1)\n WHEN 'array' THEN ARRAY(SELECT jsonb_array_elements_text($1))\n ELSE ARRAY[$1->>0]\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION bbox_geom(_bbox jsonb) RETURNS geometry AS $$\nSELECT CASE jsonb_array_length(_bbox)\n WHEN 4 THEN\n ST_SetSRID(ST_MakeEnvelope(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float,\n (_bbox->>3)::float\n ),4326)\n WHEN 6 THEN\n ST_SetSRID(ST_3DMakeBox(\n ST_MakePoint(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float\n ),\n ST_MakePoint(\n (_bbox->>3)::float,\n (_bbox->>4)::float,\n (_bbox->>5)::float\n )\n ),4326)\n ELSE null END;\n;\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION geom_bbox(_geom geometry) RETURNS jsonb AS $$\n SELECT jsonb_build_array(\n st_xmin(_geom),\n st_ymin(_geom),\n st_xmax(_geom),\n st_ymax(_geom)\n );\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION flip_jsonb_array(j jsonb) RETURNS jsonb AS $$\n SELECT jsonb_agg(value) FROM (SELECT value FROM jsonb_array_elements(j) WITH ORDINALITY ORDER BY ordinality DESC) as t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION explode_dotpaths(j jsonb) RETURNS SETOF text[] AS $$\n SELECT string_to_array(p, '.') as e FROM jsonb_array_elements_text(j) p;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION explode_dotpaths_recurse(IN j jsonb) RETURNS SETOF text[] AS $$\n WITH RECURSIVE t AS (\n SELECT e FROM explode_dotpaths(j) e\n UNION ALL\n SELECT e[1:cardinality(e)-1]\n FROM t\n WHERE cardinality(e)>1\n ) SELECT e FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION jsonb_set_nested(j jsonb, path text[], val jsonb) RETURNS jsonb AS $$\nDECLARE\nBEGIN\n IF cardinality(path) > 1 THEN\n FOR i IN 1..(cardinality(path)-1) LOOP\n IF j #> path[:i] IS NULL THEN\n j := jsonb_set_lax(j, path[:i], '{}', TRUE);\n END IF;\n END LOOP;\n END IF;\n RETURN jsonb_set_lax(j, path, val, true);\n\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\n\n\nCREATE OR REPLACE FUNCTION jsonb_include(j jsonb, f jsonb) RETURNS jsonb AS $$\nDECLARE\n includes jsonb := f-> 'include';\n outj jsonb := '{}'::jsonb;\n path text[];\nBEGIN\n IF\n includes IS NULL\n OR jsonb_array_length(includes) = 0\n THEN\n RETURN j;\n ELSE\n includes := includes || '[\"id\",\"collection\"]'::jsonb;\n FOR path IN SELECT explode_dotpaths(includes) LOOP\n outj := jsonb_set_nested(outj, path, j #> path);\n END LOOP;\n END IF;\n RETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_exclude(j jsonb, f jsonb) RETURNS jsonb AS $$\nDECLARE\n excludes jsonb := f-> 'exclude';\n outj jsonb := j;\n path text[];\nBEGIN\n IF\n excludes IS NULL\n OR jsonb_array_length(excludes) = 0\n THEN\n RETURN j;\n ELSE\n FOR path IN SELECT explode_dotpaths(excludes) LOOP\n outj := outj #- path;\n END LOOP;\n END IF;\n RETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_fields(j jsonb, f jsonb DEFAULT '{\"fields\":[]}') RETURNS jsonb AS $$\n SELECT jsonb_exclude(jsonb_include(j, f), f);\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION merge_jsonb(_a jsonb, _b jsonb) RETURNS jsonb AS $$\n SELECT\n CASE\n WHEN _a = '\"𒍟※\"'::jsonb THEN NULL\n WHEN _a IS NULL OR jsonb_typeof(_a) = 'null' THEN _b\n WHEN jsonb_typeof(_a) = 'object' AND jsonb_typeof(_b) = 'object' THEN\n (\n SELECT\n jsonb_strip_nulls(\n jsonb_object_agg(\n key,\n merge_jsonb(a.value, b.value)\n )\n )\n FROM\n jsonb_each(coalesce(_a,'{}'::jsonb)) as a\n FULL JOIN\n jsonb_each(coalesce(_b,'{}'::jsonb)) as b\n USING (key)\n )\n WHEN\n jsonb_typeof(_a) = 'array'\n AND jsonb_typeof(_b) = 'array'\n AND jsonb_array_length(_a) = jsonb_array_length(_b)\n THEN\n (\n SELECT jsonb_agg(m) FROM\n ( SELECT\n merge_jsonb(\n jsonb_array_elements(_a),\n jsonb_array_elements(_b)\n ) as m\n ) as l\n )\n ELSE _a\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION strip_jsonb(_a jsonb, _b jsonb) RETURNS jsonb AS $$\n SELECT\n CASE\n\n WHEN (_a IS NULL OR jsonb_typeof(_a) = 'null') AND _b IS NOT NULL AND jsonb_typeof(_b) != 'null' THEN '\"𒍟※\"'::jsonb\n WHEN _b IS NULL OR jsonb_typeof(_a) = 'null' THEN _a\n WHEN _a = _b AND jsonb_typeof(_a) = 'object' THEN '{}'::jsonb\n WHEN _a = _b THEN NULL\n WHEN jsonb_typeof(_a) = 'object' AND jsonb_typeof(_b) = 'object' THEN\n (\n SELECT\n jsonb_strip_nulls(\n jsonb_object_agg(\n key,\n strip_jsonb(a.value, b.value)\n )\n )\n FROM\n jsonb_each(_a) as a\n FULL JOIN\n jsonb_each(_b) as b\n USING (key)\n )\n WHEN\n jsonb_typeof(_a) = 'array'\n AND jsonb_typeof(_b) = 'array'\n AND jsonb_array_length(_a) = jsonb_array_length(_b)\n THEN\n (\n SELECT jsonb_agg(m) FROM\n ( SELECT\n strip_jsonb(\n jsonb_array_elements(_a),\n jsonb_array_elements(_b)\n ) as m\n ) as l\n )\n ELSE _a\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION nullif_jsonbnullempty(j jsonb) RETURNS jsonb AS $$\n SELECT nullif(nullif(nullif(j,'null'::jsonb),'{}'::jsonb),'[]'::jsonb);\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION jsonb_array_unique(j jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(jsonb_agg(DISTINCT a)) v FROM jsonb_array_elements(j) a;\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_concat_ignorenull(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT coalesce(a,'[]'::jsonb) || coalesce(b,'[]'::jsonb);\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_least(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(least(nullif_jsonbnullempty(a), nullif_jsonbnullempty(b)));\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_greatest(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(greatest(a, b));\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION first_notnull_sfunc(anyelement, anyelement) RETURNS anyelement AS $$\n SELECT COALESCE($1,$2);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE AGGREGATE first_notnull(anyelement)(\n SFUNC = first_notnull_sfunc,\n STYPE = anyelement\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_array_unique_merge(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_concat_ignorenull,\n FINALFUNC = jsonb_array_unique\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_min(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_least\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_max(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_greatest\n);\n/* looks for a geometry in a stac item first from geometry and falling back to bbox */\nCREATE OR REPLACE FUNCTION stac_geom(value jsonb) RETURNS geometry AS $$\nSELECT\n CASE\n WHEN value ? 'intersects' THEN\n ST_GeomFromGeoJSON(value->>'intersects')\n WHEN value ? 'geometry' THEN\n ST_GeomFromGeoJSON(value->>'geometry')\n WHEN value ? 'bbox' THEN\n pgstac.bbox_geom(value->'bbox')\n ELSE NULL\n END as geometry\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION stac_daterange(\n value jsonb\n) RETURNS tstzrange AS $$\nDECLARE\n props jsonb := value;\n dt timestamptz;\n edt timestamptz;\nBEGIN\n IF props ? 'properties' THEN\n props := props->'properties';\n END IF;\n IF\n props ? 'start_datetime'\n AND props->>'start_datetime' IS NOT NULL\n AND props ? 'end_datetime'\n AND props->>'end_datetime' IS NOT NULL\n THEN\n dt := props->>'start_datetime';\n edt := props->>'end_datetime';\n IF dt > edt THEN\n RAISE EXCEPTION 'start_datetime must be < end_datetime';\n END IF;\n ELSE\n dt := props->>'datetime';\n edt := props->>'datetime';\n END IF;\n IF dt is NULL OR edt IS NULL THEN\n RAISE NOTICE 'DT: %, EDT: %', dt, edt;\n RAISE EXCEPTION 'Either datetime (%) or both start_datetime (%) and end_datetime (%) must be set.', props->>'datetime',props->>'start_datetime',props->>'end_datetime';\n END IF;\n RETURN tstzrange(dt, edt, '[]');\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_datetime(value jsonb) RETURNS timestamptz AS $$\n SELECT lower(stac_daterange(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_end_datetime(value jsonb) RETURNS timestamptz AS $$\n SELECT upper(stac_daterange(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE TABLE IF NOT EXISTS stac_extensions(\n url text PRIMARY KEY,\n content jsonb\n);\nCREATE OR REPLACE FUNCTION collection_base_item(content jsonb) RETURNS jsonb AS $$\n SELECT jsonb_build_object(\n 'type', 'Feature',\n 'stac_version', content->'stac_version',\n 'assets', content->'item_assets',\n 'collection', content->'id'\n );\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE TABLE IF NOT EXISTS collections (\n key bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,\n id text GENERATED ALWAYS AS (content->>'id') STORED UNIQUE,\n content JSONB NOT NULL,\n base_item jsonb GENERATED ALWAYS AS (pgstac.collection_base_item(content)) STORED,\n geometry geometry GENERATED ALWAYS AS (pgstac.collection_geom(content)) STORED,\n datetime timestamptz GENERATED ALWAYS AS (pgstac.collection_datetime(content)) STORED,\n end_datetime timestamptz GENERATED ALWAYS AS (pgstac.collection_enddatetime(content)) STORED,\n private jsonb,\n partition_trunc text CHECK (partition_trunc IN ('year', 'month'))\n);\n\n\n\nCREATE OR REPLACE FUNCTION collection_base_item(cid text) RETURNS jsonb AS $$\n SELECT pgstac.collection_base_item(content) FROM pgstac.collections WHERE id = cid LIMIT 1;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION create_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION update_collection(data jsonb) RETURNS VOID AS $$\nDECLARE\n out collections%ROWTYPE;\nBEGIN\n UPDATE collections SET content=data WHERE id = data->>'id' RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ON CONFLICT (id) DO\n UPDATE\n SET content=EXCLUDED.content\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION delete_collection(_id text) RETURNS VOID AS $$\nDECLARE\n out collections%ROWTYPE;\nBEGIN\n DELETE FROM collections WHERE id = _id RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION get_collection(id text) RETURNS jsonb AS $$\n SELECT content FROM collections\n WHERE id=$1\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION all_collections() RETURNS jsonb AS $$\n SELECT coalesce(jsonb_agg(content), '[]'::jsonb) FROM collections;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION collection_delete_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n collection_base_partition text := concat('_items_', OLD.key);\nBEGIN\n EXECUTE format($q$\n DELETE FROM partition_stats WHERE partition IN (\n SELECT partition FROM partition_sys_meta\n WHERE collection=%L\n );\n DROP TABLE IF EXISTS %I CASCADE;\n $q$,\n OLD.id,\n collection_base_partition\n );\n RETURN OLD;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER collection_delete_trigger BEFORE DELETE ON collections\nFOR EACH ROW EXECUTE FUNCTION collection_delete_trigger_func();\nCREATE OR REPLACE FUNCTION queryable_signature(n text, c text[]) RETURNS text AS $$\n SELECT concat(n, c);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TABLE queryables (\n id bigint GENERATED ALWAYS AS identity PRIMARY KEY,\n name text NOT NULL,\n collection_ids text[], -- used to determine what partitions to create indexes on\n definition jsonb,\n property_path text,\n property_wrapper text,\n property_index_type text\n);\nCREATE INDEX queryables_name_idx ON queryables (name);\nCREATE INDEX queryables_collection_idx ON queryables USING GIN (collection_ids);\nCREATE INDEX queryables_property_wrapper_idx ON queryables (property_wrapper);\n\nCREATE OR REPLACE FUNCTION pgstac.queryables_constraint_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n allcollections text[];\nBEGIN\n RAISE NOTICE 'Making sure that name/collection is unique for queryables %', NEW;\n IF NEW.collection_ids IS NOT NULL THEN\n IF EXISTS (\n SELECT 1\n FROM unnest(NEW.collection_ids) c\n LEFT JOIN\n collections\n ON (collections.id = c)\n WHERE collections.id IS NULL\n ) THEN\n RAISE foreign_key_violation USING MESSAGE = format(\n 'One or more collections in %s do not exist.', NEW.collection_ids\n );\n RETURN NULL;\n END IF;\n END IF;\n IF TG_OP = 'INSERT' THEN\n IF EXISTS (\n SELECT 1 FROM queryables q\n WHERE\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n )\n ) THEN\n RAISE unique_violation USING MESSAGE = format(\n 'There is already a queryable for %s for a collection in %s: %s',\n NEW.name,\n NEW.collection_ids,\n\t\t\t\t(SELECT json_agg(row_to_json(q)) FROM queryables q WHERE\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n ))\n );\n RETURN NULL;\n END IF;\n END IF;\n IF TG_OP = 'UPDATE' THEN\n IF EXISTS (\n SELECT 1 FROM queryables q\n WHERE\n q.id != NEW.id\n AND\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n )\n ) THEN\n RAISE unique_violation\n USING MESSAGE = format(\n 'There is already a queryable for %s for a collection in %s',\n NEW.name,\n NEW.collection_ids\n );\n RETURN NULL;\n END IF;\n END IF;\n\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER queryables_constraint_insert_trigger\nBEFORE INSERT ON queryables\nFOR EACH ROW EXECUTE PROCEDURE queryables_constraint_triggerfunc();\n\nCREATE TRIGGER queryables_constraint_update_trigger\nBEFORE UPDATE ON queryables\nFOR EACH ROW\nWHEN (NEW.name = OLD.name AND NEW.collection_ids IS DISTINCT FROM OLD.collection_ids)\nEXECUTE PROCEDURE queryables_constraint_triggerfunc();\n\n\nCREATE OR REPLACE FUNCTION array_to_path(arr text[]) RETURNS text AS $$\n SELECT string_agg(\n quote_literal(v),\n '->'\n ) FROM unnest(arr) v;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\n\n\nCREATE OR REPLACE FUNCTION queryable(\n IN dotpath text,\n OUT path text,\n OUT expression text,\n OUT wrapper text,\n OUT nulled_wrapper text\n) AS $$\nDECLARE\n q RECORD;\n path_elements text[];\nBEGIN\n IF dotpath IN ('id', 'geometry', 'datetime', 'end_datetime', 'collection') THEN\n path := dotpath;\n expression := dotpath;\n wrapper := NULL;\n RETURN;\n END IF;\n SELECT * INTO q FROM queryables\n WHERE\n name=dotpath\n OR name = 'properties.' || dotpath\n OR name = replace(dotpath, 'properties.', '')\n ;\n IF q.property_wrapper IS NULL THEN\n IF q.definition->>'type' = 'number' THEN\n wrapper := 'to_float';\n nulled_wrapper := wrapper;\n ELSIF q.definition->>'format' = 'date-time' THEN\n wrapper := 'to_tstz';\n nulled_wrapper := wrapper;\n ELSE\n nulled_wrapper := NULL;\n wrapper := 'to_text';\n END IF;\n ELSE\n wrapper := q.property_wrapper;\n nulled_wrapper := wrapper;\n END IF;\n IF q.property_path IS NOT NULL THEN\n path := q.property_path;\n ELSE\n path_elements := string_to_array(dotpath, '.');\n IF path_elements[1] IN ('links', 'assets', 'stac_version', 'stac_extensions') THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSIF path_elements[1] = 'properties' THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSE\n path := format($F$content->'properties'->%s$F$, array_to_path(path_elements));\n END IF;\n END IF;\n expression := format('%I(%s)', wrapper, path);\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\nCREATE OR REPLACE FUNCTION unnest_collection(collection_ids text[] DEFAULT NULL) RETURNS SETOF text AS $$\n DECLARE\n BEGIN\n IF collection_ids IS NULL THEN\n RETURN QUERY SELECT id FROM collections;\n END IF;\n RETURN QUERY SELECT unnest(collection_ids);\n END;\n$$ LANGUAGE PLPGSQL STABLE;\n\nCREATE OR REPLACE FUNCTION normalize_indexdef(def text) RETURNS text AS $$\nDECLARE\nBEGIN\n def := btrim(def, ' \\n\\t');\n\tdef := regexp_replace(def, '^CREATE (UNIQUE )?INDEX ([^ ]* )?ON (ONLY )?([^ ]* )?', '', 'i');\n RETURN def;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE STRICT PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION indexdef(q queryables) RETURNS text AS $$\n DECLARE\n out text;\n BEGIN\n IF q.name = 'id' THEN\n out := 'CREATE UNIQUE INDEX ON %I USING btree (id)';\n ELSIF q.name = 'datetime' THEN\n out := 'CREATE INDEX ON %I USING btree (datetime DESC, end_datetime)';\n ELSIF q.name = 'geometry' THEN\n out := 'CREATE INDEX ON %I USING gist (geometry)';\n ELSE\n out := format($q$CREATE INDEX ON %%I USING %s (%s(((content -> 'properties'::text) -> %L::text)))$q$,\n lower(COALESCE(q.property_index_type, 'BTREE')),\n lower(COALESCE(q.property_wrapper, 'to_text')),\n q.name\n );\n END IF;\n RETURN btrim(out, ' \\n\\t');\n END;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\n\nDROP VIEW IF EXISTS pgstac_indexes;\nCREATE VIEW pgstac_indexes AS\nSELECT\n i.schemaname,\n i.tablename,\n i.indexname,\n regexp_replace(btrim(replace(replace(indexdef, i.indexname, ''),'pgstac.',''),' \\t\\n'), '[ ]+', ' ', 'g') as idx,\n COALESCE(\n (regexp_match(indexdef, '\\(([a-zA-Z]+)\\)'))[1],\n (regexp_match(indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_-]+)''::text'))[1],\n CASE WHEN indexdef ~* '\\(datetime desc, end_datetime\\)' THEN 'datetime' ELSE NULL END\n ) AS field,\n pg_table_size(i.indexname::text) as index_size,\n pg_size_pretty(pg_table_size(i.indexname::text)) as index_size_pretty\nFROM\n pg_indexes i\nWHERE i.schemaname='pgstac' and i.tablename ~ '_items_' AND indexdef !~* ' only ';\n\nDROP VIEW IF EXISTS pgstac_index_stats;\nCREATE VIEW pgstac_indexes_stats AS\nSELECT\n i.schemaname,\n i.tablename,\n i.indexname,\n indexdef,\n COALESCE(\n (regexp_match(indexdef, '\\(([a-zA-Z]+)\\)'))[1],\n (regexp_match(indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_]+)''::text'))[1],\n CASE WHEN indexdef ~* '\\(datetime desc, end_datetime\\)' THEN 'datetime_end_datetime' ELSE NULL END\n ) AS field,\n pg_table_size(i.indexname::text) as index_size,\n pg_size_pretty(pg_table_size(i.indexname::text)) as index_size_pretty,\n n_distinct,\n most_common_vals::text::text[],\n most_common_freqs::text::text[],\n histogram_bounds::text::text[],\n correlation\nFROM\n pg_indexes i\n LEFT JOIN pg_stats s ON (s.tablename = i.indexname)\nWHERE i.schemaname='pgstac' and i.tablename ~ '_items_';\n\nset check_function_bodies to off;\nCREATE OR REPLACE FUNCTION maintain_partition_queries(\n part text DEFAULT 'items',\n dropindexes boolean DEFAULT FALSE,\n rebuildindexes boolean DEFAULT FALSE,\n idxconcurrently boolean DEFAULT FALSE\n) RETURNS SETOF text AS $$\nDECLARE\n rec record;\nBEGIN\n FOR rec IN (\n WITH p AS (\n SELECT\n relid::text as partition,\n replace(replace(\n CASE\n WHEN level = 1 THEN pg_get_expr(c.relpartbound, c.oid)\n ELSE pg_get_expr(parent.relpartbound, parent.oid)\n END,\n 'FOR VALUES IN (''',''), ''')',\n ''\n ) AS collection\n FROM pg_partition_tree('items')\n JOIN pg_class c ON (relid::regclass = c.oid)\n JOIN pg_class parent ON (parentrelid::regclass = parent.oid AND isleaf)\n ), i AS (\n SELECT\n partition,\n indexname,\n regexp_replace(btrim(replace(replace(indexdef, indexname, ''),'pgstac.',''),' \\t\\n'), '[ ]+', ' ', 'g') as iidx,\n COALESCE(\n (regexp_match(indexdef, '\\(([a-zA-Z]+)\\)'))[1],\n (regexp_match(indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_-]+)''::text'))[1],\n CASE WHEN indexdef ~* '\\(datetime desc, end_datetime\\)' THEN 'datetime' ELSE NULL END\n ) AS field\n FROM\n pg_indexes\n JOIN p ON (tablename=partition)\n ), q AS (\n SELECT\n name AS field,\n collection,\n partition,\n format(indexdef(queryables), partition) as qidx\n FROM queryables, unnest_collection(queryables.collection_ids) collection\n JOIN p USING (collection)\n WHERE property_index_type IS NOT NULL OR name IN ('datetime','geometry','id')\n )\n SELECT * FROM i FULL JOIN q USING (field, partition)\n WHERE lower(iidx) IS DISTINCT FROM lower(qidx)\n ) LOOP\n IF rec.iidx IS NULL THEN\n IF idxconcurrently THEN\n RETURN NEXT replace(rec.qidx, 'INDEX', 'INDEX CONCURRENTLY');\n ELSE\n RETURN NEXT rec.qidx;\n END IF;\n ELSIF rec.qidx IS NULL AND dropindexes THEN\n RETURN NEXT format('DROP INDEX IF EXISTS %I;', rec.indexname);\n ELSIF lower(rec.qidx) != lower(rec.iidx) THEN\n IF dropindexes THEN\n RETURN NEXT format('DROP INDEX IF EXISTS %I; %s;', rec.indexname, rec.qidx);\n ELSE\n IF idxconcurrently THEN\n RETURN NEXT replace(rec.qidx, 'INDEX', 'INDEX CONCURRENTLY');\n ELSE\n RETURN NEXT rec.qidx;\n END IF;\n END IF;\n ELSIF rebuildindexes and rec.indexname IS NOT NULL THEN\n IF idxconcurrently THEN\n RETURN NEXT format('REINDEX INDEX CONCURRENTLY %I;', rec.indexname);\n ELSE\n RETURN NEXT format('REINDEX INDEX %I;', rec.indexname);\n END IF;\n END IF;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION maintain_partitions(\n part text DEFAULT 'items',\n dropindexes boolean DEFAULT FALSE,\n rebuildindexes boolean DEFAULT FALSE\n) RETURNS VOID AS $$\n WITH t AS (\n SELECT run_or_queue(q) FROM maintain_partition_queries(part, dropindexes, rebuildindexes) q\n ) SELECT count(*) FROM t;\n$$ LANGUAGE SQL;\n\n\nCREATE OR REPLACE FUNCTION queryables_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\nBEGIN\n PERFORM maintain_partitions();\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER queryables_trigger AFTER INSERT OR UPDATE ON queryables\nFOR EACH STATEMENT EXECUTE PROCEDURE queryables_trigger_func();\n\n\nCREATE OR REPLACE FUNCTION get_queryables(_collection_ids text[] DEFAULT NULL) RETURNS jsonb AS $$\nDECLARE\nBEGIN\n -- Build up queryables if the input contains valid collection ids or is empty\n IF EXISTS (\n SELECT 1 FROM collections\n WHERE\n _collection_ids IS NULL\n OR cardinality(_collection_ids) = 0\n OR id = ANY(_collection_ids)\n )\n THEN\n RETURN (\n WITH base AS (\n SELECT\n unnest(collection_ids) as collection_id,\n name,\n coalesce(definition, '{\"type\":\"string\"}'::jsonb) as definition\n FROM queryables\n WHERE\n _collection_ids IS NULL OR\n _collection_ids = '{}'::text[] OR\n _collection_ids && collection_ids\n UNION ALL\n SELECT null, name, coalesce(definition, '{\"type\":\"string\"}'::jsonb) as definition\n FROM queryables WHERE collection_ids IS NULL OR collection_ids = '{}'::text[]\n ), g AS (\n SELECT\n name,\n first_notnull(definition) as definition,\n jsonb_array_unique_merge(definition->'enum') as enum,\n jsonb_min(definition->'minimum') as minimum,\n jsonb_min(definition->'maxiumn') as maximum\n FROM base\n GROUP BY 1\n )\n SELECT\n jsonb_build_object(\n '$schema', 'http://json-schema.org/draft-07/schema#',\n '$id', '',\n 'type', 'object',\n 'title', 'STAC Queryables.',\n 'properties', jsonb_object_agg(\n name,\n definition\n ||\n jsonb_strip_nulls(jsonb_build_object(\n 'enum', enum,\n 'minimum', minimum,\n 'maximum', maximum\n ))\n )\n )\n FROM g\n );\n ELSE\n RETURN NULL;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\nCREATE OR REPLACE FUNCTION get_queryables(_collection text DEFAULT NULL) RETURNS jsonb AS $$\n SELECT\n CASE\n WHEN _collection IS NULL THEN get_queryables(NULL::text[])\n ELSE get_queryables(ARRAY[_collection])\n END\n ;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_queryables() RETURNS jsonb AS $$\n SELECT get_queryables(NULL::text[]);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION schema_qualify_refs(url text, j jsonb) returns jsonb as $$\n SELECT regexp_replace(j::text, '\"\\$ref\": \"#', concat('\"$ref\": \"', url, '#'), 'g')::jsonb;\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE VIEW stac_extension_queryables AS\nSELECT DISTINCT key as name, schema_qualify_refs(e.url, j.value) as definition FROM stac_extensions e, jsonb_each(e.content->'definitions'->'fields'->'properties') j;\n\n\nCREATE OR REPLACE FUNCTION missing_queryables(_collection text, _tablesample float DEFAULT 5, minrows float DEFAULT 10) RETURNS TABLE(collection text, name text, definition jsonb, property_wrapper text) AS $$\nDECLARE\n q text;\n _partition text;\n explain_json json;\n psize float;\n estrows float;\nBEGIN\n SELECT format('_items_%s', key) INTO _partition FROM collections WHERE id=_collection;\n\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM %I;', _partition)\n INTO explain_json;\n psize := explain_json->0->'Plan'->'Plan Rows';\n estrows := _tablesample * .01 * psize;\n IF estrows < minrows THEN\n _tablesample := least(100,greatest(_tablesample, (estrows / psize) / 100));\n RAISE NOTICE '%', (psize / estrows) / 100;\n END IF;\n RAISE NOTICE 'Using tablesample % to find missing queryables from % % that has ~% rows estrows: %', _tablesample, _collection, _partition, psize, estrows;\n\n q := format(\n $q$\n WITH q AS (\n SELECT * FROM queryables\n WHERE\n collection_ids IS NULL\n OR %L = ANY(collection_ids)\n ), t AS (\n SELECT\n content->'properties' AS properties\n FROM\n %I\n TABLESAMPLE SYSTEM(%L)\n ), p AS (\n SELECT DISTINCT ON (key)\n key,\n value,\n s.definition\n FROM t\n JOIN LATERAL jsonb_each(properties) ON TRUE\n LEFT JOIN q ON (q.name=key)\n LEFT JOIN stac_extension_queryables s ON (s.name=key)\n WHERE q.definition IS NULL\n )\n SELECT\n %L,\n key,\n COALESCE(definition, jsonb_build_object('type',jsonb_typeof(value))) as definition,\n CASE\n WHEN definition->>'type' = 'integer' THEN 'to_int'\n WHEN COALESCE(definition->>'type', jsonb_typeof(value)) = 'number' THEN 'to_float'\n WHEN COALESCE(definition->>'type', jsonb_typeof(value)) = 'array' THEN 'to_text_array'\n ELSE 'to_text'\n END\n FROM p;\n $q$,\n _collection,\n _partition,\n _tablesample,\n _collection\n );\n RETURN QUERY EXECUTE q;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION missing_queryables(_tablesample float DEFAULT 5) RETURNS TABLE(collection_ids text[], name text, definition jsonb, property_wrapper text) AS $$\n SELECT\n array_agg(collection),\n name,\n definition,\n property_wrapper\n FROM\n collections\n JOIN LATERAL\n missing_queryables(id, _tablesample) c\n ON TRUE\n GROUP BY\n 2,3,4\n ORDER BY 2,1\n ;\n$$ LANGUAGE SQL;\nCREATE OR REPLACE FUNCTION parse_dtrange(\n _indate jsonb,\n relative_base timestamptz DEFAULT date_trunc('hour', CURRENT_TIMESTAMP)\n) RETURNS tstzrange AS $$\nDECLARE\n timestrs text[];\n s timestamptz;\n e timestamptz;\nBEGIN\n timestrs :=\n CASE\n WHEN _indate ? 'timestamp' THEN\n ARRAY[_indate->>'timestamp']\n WHEN _indate ? 'interval' THEN\n to_text_array(_indate->'interval')\n WHEN jsonb_typeof(_indate) = 'array' THEN\n to_text_array(_indate)\n ELSE\n regexp_split_to_array(\n _indate->>0,\n '/'\n )\n END;\n RAISE NOTICE 'TIMESTRS %', timestrs;\n IF cardinality(timestrs) = 1 THEN\n IF timestrs[1] ILIKE 'P%' THEN\n RETURN tstzrange(relative_base - upper(timestrs[1])::interval, relative_base, '[)');\n END IF;\n s := timestrs[1]::timestamptz;\n RETURN tstzrange(s, s, '[]');\n END IF;\n\n IF cardinality(timestrs) != 2 THEN\n RAISE EXCEPTION 'Timestamp cannot have more than 2 values';\n END IF;\n\n IF timestrs[1] = '..' OR timestrs[1] = '' THEN\n s := '-infinity'::timestamptz;\n e := timestrs[2]::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] = '..' OR timestrs[2] = '' THEN\n s := timestrs[1]::timestamptz;\n e := 'infinity'::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[1] ILIKE 'P%' AND timestrs[2] NOT ILIKE 'P%' THEN\n e := timestrs[2]::timestamptz;\n s := e - upper(timestrs[1])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] ILIKE 'P%' AND timestrs[1] NOT ILIKE 'P%' THEN\n s := timestrs[1]::timestamptz;\n e := s + upper(timestrs[2])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n s := timestrs[1]::timestamptz;\n e := timestrs[2]::timestamptz;\n\n RETURN tstzrange(s,e,'[)');\n\n RETURN NULL;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT PARALLEL SAFE SET TIME ZONE 'UTC';\n\nCREATE OR REPLACE FUNCTION parse_dtrange(\n _indate text,\n relative_base timestamptz DEFAULT CURRENT_TIMESTAMP\n) RETURNS tstzrange AS $$\n SELECT parse_dtrange(to_jsonb(_indate), relative_base);\n$$ LANGUAGE SQL STABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION temporal_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\n ll text := 'datetime';\n lh text := 'end_datetime';\n rrange tstzrange;\n rl text;\n rh text;\n outq text;\nBEGIN\n rrange := parse_dtrange(args->1);\n RAISE NOTICE 'Constructing temporal query OP: %, ARGS: %, RRANGE: %', op, args, rrange;\n op := lower(op);\n rl := format('%L::timestamptz', lower(rrange));\n rh := format('%L::timestamptz', upper(rrange));\n outq := CASE op\n WHEN 't_before' THEN 'lh < rl'\n WHEN 't_after' THEN 'll > rh'\n WHEN 't_meets' THEN 'lh = rl'\n WHEN 't_metby' THEN 'll = rh'\n WHEN 't_overlaps' THEN 'll < rl AND rl < lh < rh'\n WHEN 't_overlappedby' THEN 'rl < ll < rh AND lh > rh'\n WHEN 't_starts' THEN 'll = rl AND lh < rh'\n WHEN 't_startedby' THEN 'll = rl AND lh > rh'\n WHEN 't_during' THEN 'll > rl AND lh < rh'\n WHEN 't_contains' THEN 'll < rl AND lh > rh'\n WHEN 't_finishes' THEN 'll > rl AND lh = rh'\n WHEN 't_finishedby' THEN 'll < rl AND lh = rh'\n WHEN 't_equals' THEN 'll = rl AND lh = rh'\n WHEN 't_disjoint' THEN 'NOT (ll <= rh AND lh >= rl)'\n WHEN 't_intersects' THEN 'll <= rh AND lh >= rl'\n WHEN 'anyinteracts' THEN 'll <= rh AND lh >= rl'\n END;\n outq := regexp_replace(outq, '\\mll\\M', ll);\n outq := regexp_replace(outq, '\\mlh\\M', lh);\n outq := regexp_replace(outq, '\\mrl\\M', rl);\n outq := regexp_replace(outq, '\\mrh\\M', rh);\n outq := format('(%s)', outq);\n RETURN outq;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\n\n\nCREATE OR REPLACE FUNCTION spatial_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\n geom text;\n j jsonb := args->1;\nBEGIN\n op := lower(op);\n RAISE NOTICE 'Constructing spatial query OP: %, ARGS: %', op, args;\n IF op NOT IN ('s_equals','s_disjoint','s_touches','s_within','s_overlaps','s_crosses','s_intersects','intersects','s_contains') THEN\n RAISE EXCEPTION 'Spatial Operator % Not Supported', op;\n END IF;\n op := regexp_replace(op, '^s_', 'st_');\n IF op = 'intersects' THEN\n op := 'st_intersects';\n END IF;\n -- Convert geometry to WKB string\n IF j ? 'type' AND j ? 'coordinates' THEN\n geom := st_geomfromgeojson(j)::text;\n ELSIF jsonb_typeof(j) = 'array' THEN\n geom := bbox_geom(j)::text;\n END IF;\n\n RETURN format('%s(geometry, %L::geometry)', op, geom);\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION query_to_cql2(q jsonb) RETURNS jsonb AS $$\n-- Translates anything passed in through the deprecated \"query\" into equivalent CQL2\nWITH t AS (\n SELECT key as property, value as ops\n FROM jsonb_each(q)\n), t2 AS (\n SELECT property, (jsonb_each(ops)).*\n FROM t WHERE jsonb_typeof(ops) = 'object'\n UNION ALL\n SELECT property, 'eq', ops\n FROM t WHERE jsonb_typeof(ops) != 'object'\n)\nSELECT\n jsonb_strip_nulls(jsonb_build_object(\n 'op', 'and',\n 'args', jsonb_agg(\n jsonb_build_object(\n 'op', key,\n 'args', jsonb_build_array(\n jsonb_build_object('property',property),\n value\n )\n )\n )\n )\n) as qcql FROM t2\n;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\nCREATE OR REPLACE FUNCTION cql1_to_cql2(j jsonb) RETURNS jsonb AS $$\nDECLARE\n args jsonb;\n ret jsonb;\nBEGIN\n RAISE NOTICE 'CQL1_TO_CQL2: %', j;\n IF j ? 'filter' THEN\n RETURN cql1_to_cql2(j->'filter');\n END IF;\n IF j ? 'property' THEN\n RETURN j;\n END IF;\n IF jsonb_typeof(j) = 'array' THEN\n SELECT jsonb_agg(cql1_to_cql2(el)) INTO args FROM jsonb_array_elements(j) el;\n RETURN args;\n END IF;\n IF jsonb_typeof(j) = 'number' THEN\n RETURN j;\n END IF;\n IF jsonb_typeof(j) = 'string' THEN\n RETURN j;\n END IF;\n\n IF jsonb_typeof(j) = 'object' THEN\n SELECT jsonb_build_object(\n 'op', key,\n 'args', cql1_to_cql2(value)\n ) INTO ret\n FROM jsonb_each(j)\n WHERE j IS NOT NULL;\n RETURN ret;\n END IF;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE STRICT;\n\nCREATE TABLE cql2_ops (\n op text PRIMARY KEY,\n template text,\n types text[]\n);\nINSERT INTO cql2_ops (op, template, types) VALUES\n ('eq', '%s = %s', NULL),\n ('neq', '%s != %s', NULL),\n ('ne', '%s != %s', NULL),\n ('!=', '%s != %s', NULL),\n ('<>', '%s != %s', NULL),\n ('lt', '%s < %s', NULL),\n ('lte', '%s <= %s', NULL),\n ('gt', '%s > %s', NULL),\n ('gte', '%s >= %s', NULL),\n ('le', '%s <= %s', NULL),\n ('ge', '%s >= %s', NULL),\n ('=', '%s = %s', NULL),\n ('<', '%s < %s', NULL),\n ('<=', '%s <= %s', NULL),\n ('>', '%s > %s', NULL),\n ('>=', '%s >= %s', NULL),\n ('like', '%s LIKE %s', NULL),\n ('ilike', '%s ILIKE %s', NULL),\n ('+', '%s + %s', NULL),\n ('-', '%s - %s', NULL),\n ('*', '%s * %s', NULL),\n ('/', '%s / %s', NULL),\n ('not', 'NOT (%s)', NULL),\n ('between', '%s BETWEEN %s AND %s', NULL),\n ('isnull', '%s IS NULL', NULL),\n ('upper', 'upper(%s)', NULL),\n ('lower', 'lower(%s)', NULL)\nON CONFLICT (op) DO UPDATE\n SET\n template = EXCLUDED.template\n;\n\n\nCREATE OR REPLACE FUNCTION cql2_query(j jsonb, wrapper text DEFAULT NULL) RETURNS text AS $$\n#variable_conflict use_variable\nDECLARE\n args jsonb := j->'args';\n arg jsonb;\n op text := lower(j->>'op');\n cql2op RECORD;\n literal text;\n _wrapper text;\n leftarg text;\n rightarg text;\nBEGIN\n IF j IS NULL OR (op IS NOT NULL AND args IS NULL) THEN\n RETURN NULL;\n END IF;\n RAISE NOTICE 'CQL2_QUERY: %', j;\n IF j ? 'filter' THEN\n RETURN cql2_query(j->'filter');\n END IF;\n\n IF j ? 'upper' THEN\n RETURN cql2_query(jsonb_build_object('op', 'upper', 'args', j->'upper'));\n END IF;\n\n IF j ? 'lower' THEN\n RETURN cql2_query(jsonb_build_object('op', 'lower', 'args', j->'lower'));\n END IF;\n\n -- Temporal Query\n IF op ilike 't_%' or op = 'anyinteracts' THEN\n RETURN temporal_op_query(op, args);\n END IF;\n\n -- If property is a timestamp convert it to text to use with\n -- general operators\n IF j ? 'timestamp' THEN\n RETURN format('%L::timestamptz', to_tstz(j->'timestamp'));\n END IF;\n IF j ? 'interval' THEN\n RAISE EXCEPTION 'Please use temporal operators when using intervals.';\n RETURN NONE;\n END IF;\n\n -- Spatial Query\n IF op ilike 's_%' or op = 'intersects' THEN\n RETURN spatial_op_query(op, args);\n END IF;\n\n IF op IN ('a_equals','a_contains','a_contained_by','a_overlaps') THEN\n IF args->0 ? 'property' THEN\n leftarg := format('to_text_array(%s)', (queryable(args->0->>'property')).path);\n END IF;\n IF args->1 ? 'property' THEN\n rightarg := format('to_text_array(%s)', (queryable(args->1->>'property')).path);\n END IF;\n RETURN FORMAT(\n '%s %s %s',\n COALESCE(leftarg, quote_literal(to_text_array(args->0))),\n CASE op\n WHEN 'a_equals' THEN '='\n WHEN 'a_contains' THEN '@>'\n WHEN 'a_contained_by' THEN '<@'\n WHEN 'a_overlaps' THEN '&&'\n END,\n COALESCE(rightarg, quote_literal(to_text_array(args->1)))\n );\n END IF;\n\n IF op = 'in' THEN\n RAISE NOTICE 'IN : % % %', args, jsonb_build_array(args->0), args->1;\n args := jsonb_build_array(args->0) || (args->1);\n RAISE NOTICE 'IN2 : %', args;\n END IF;\n\n\n\n IF op = 'between' THEN\n args = jsonb_build_array(\n args->0,\n args->1->0,\n args->1->1\n );\n END IF;\n\n -- Make sure that args is an array and run cql2_query on\n -- each element of the array\n RAISE NOTICE 'ARGS PRE: %', args;\n IF j ? 'args' THEN\n IF jsonb_typeof(args) != 'array' THEN\n args := jsonb_build_array(args);\n END IF;\n\n IF jsonb_path_exists(args, '$[*] ? (@.property == \"id\" || @.property == \"datetime\" || @.property == \"end_datetime\" || @.property == \"collection\")') THEN\n wrapper := NULL;\n ELSE\n -- if any of the arguments are a property, try to get the property_wrapper\n FOR arg IN SELECT jsonb_path_query(args, '$[*] ? (@.property != null)') LOOP\n RAISE NOTICE 'Arg: %', arg;\n wrapper := (queryable(arg->>'property')).nulled_wrapper;\n RAISE NOTICE 'Property: %, Wrapper: %', arg, wrapper;\n IF wrapper IS NOT NULL THEN\n EXIT;\n END IF;\n END LOOP;\n\n -- if the property was not in queryables, see if any args were numbers\n IF\n wrapper IS NULL\n AND jsonb_path_exists(args, '$[*] ? (@.type()==\"number\")')\n THEN\n wrapper := 'to_float';\n END IF;\n wrapper := coalesce(wrapper, 'to_text');\n END IF;\n\n SELECT jsonb_agg(cql2_query(a, wrapper))\n INTO args\n FROM jsonb_array_elements(args) a;\n END IF;\n RAISE NOTICE 'ARGS: %', args;\n\n IF op IN ('and', 'or') THEN\n RETURN\n format(\n '(%s)',\n array_to_string(to_text_array(args), format(' %s ', upper(op)))\n );\n END IF;\n\n IF op = 'in' THEN\n RAISE NOTICE 'IN -- % %', args->0, to_text(args->0);\n RETURN format(\n '%s IN (%s)',\n to_text(args->0),\n array_to_string((to_text_array(args))[2:], ',')\n );\n END IF;\n\n -- Look up template from cql2_ops\n IF j ? 'op' THEN\n SELECT * INTO cql2op FROM cql2_ops WHERE cql2_ops.op ilike op;\n IF FOUND THEN\n -- If specific index set in queryables for a property cast other arguments to that type\n\n RETURN format(\n cql2op.template,\n VARIADIC (to_text_array(args))\n );\n ELSE\n RAISE EXCEPTION 'Operator % Not Supported.', op;\n END IF;\n END IF;\n\n\n IF wrapper IS NOT NULL THEN\n RAISE NOTICE 'Wrapping % with %', j, wrapper;\n IF j ? 'property' THEN\n RETURN format('%I(%s)', wrapper, (queryable(j->>'property')).path);\n ELSE\n RETURN format('%I(%L)', wrapper, j);\n END IF;\n ELSIF j ? 'property' THEN\n RETURN quote_ident(j->>'property');\n END IF;\n\n RETURN quote_literal(to_text(j));\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION paging_dtrange(\n j jsonb\n) RETURNS tstzrange AS $$\nDECLARE\n op text;\n filter jsonb := j->'filter';\n dtrange tstzrange := tstzrange('-infinity'::timestamptz,'infinity'::timestamptz);\n sdate timestamptz := '-infinity'::timestamptz;\n edate timestamptz := 'infinity'::timestamptz;\n jpitem jsonb;\nBEGIN\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n END IF;\n IF NOT (filter @? '$.**.op ? (@ == \"or\" || @ == \"not\")') THEN\n FOR jpitem IN SELECT j FROM jsonb_path_query(filter,'strict $.** ? (@.args[*].property == \"datetime\")'::jsonpath) j LOOP\n op := lower(jpitem->>'op');\n dtrange := parse_dtrange(jpitem->'args'->1);\n IF op IN ('<=', 'lt', 'lte', '<', 'le', 't_before') THEN\n sdate := greatest(sdate,'-infinity');\n edate := least(edate, upper(dtrange));\n ELSIF op IN ('>=', '>', 'gt', 'gte', 'ge', 't_after') THEN\n edate := least(edate, 'infinity');\n sdate := greatest(sdate, lower(dtrange));\n ELSIF op IN ('=', 'eq') THEN\n edate := least(edate, upper(dtrange));\n sdate := greatest(sdate, lower(dtrange));\n END IF;\n RAISE NOTICE '2 OP: %, ARGS: %, DTRANGE: %, SDATE: %, EDATE: %', op, jpitem->'args'->1, dtrange, sdate, edate;\n END LOOP;\n END IF;\n IF sdate > edate THEN\n RETURN 'empty'::tstzrange;\n END IF;\n RETURN tstzrange(sdate,edate, '[]');\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT SET TIME ZONE 'UTC';\n\nCREATE OR REPLACE FUNCTION paging_collections(\n IN j jsonb\n) RETURNS text[] AS $$\nDECLARE\n filter jsonb := j->'filter';\n jpitem jsonb;\n op text;\n args jsonb;\n arg jsonb;\n collections text[];\nBEGIN\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n END IF;\n IF NOT (filter @? '$.**.op ? (@ == \"or\" || @ == \"not\")') THEN\n FOR jpitem IN SELECT j FROM jsonb_path_query(filter,'strict $.** ? (@.args[*].property == \"collection\")'::jsonpath) j LOOP\n RAISE NOTICE 'JPITEM: %', jpitem;\n op := jpitem->>'op';\n args := jpitem->'args';\n IF op IN ('=', 'eq', 'in') THEN\n FOR arg IN SELECT a FROM jsonb_array_elements(args) a LOOP\n IF jsonb_typeof(arg) IN ('string', 'array') THEN\n RAISE NOTICE 'arg: %, collections: %', arg, collections;\n IF collections IS NULL OR collections = '{}'::text[] THEN\n collections := to_text_array(arg);\n ELSE\n collections := array_intersection(collections, to_text_array(arg));\n END IF;\n END IF;\n END LOOP;\n END IF;\n END LOOP;\n END IF;\n IF collections = '{}'::text[] THEN\n RETURN NULL;\n END IF;\n RETURN collections;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\nCREATE TABLE items (\n id text NOT NULL,\n geometry geometry NOT NULL,\n collection text NOT NULL,\n datetime timestamptz NOT NULL,\n end_datetime timestamptz NOT NULL,\n content JSONB NOT NULL,\n private jsonb\n)\nPARTITION BY LIST (collection)\n;\n\nCREATE INDEX \"datetime_idx\" ON items USING BTREE (datetime DESC, end_datetime ASC);\nCREATE INDEX \"geometry_idx\" ON items USING GIST (geometry);\n\nCREATE STATISTICS datetime_stats (dependencies) on datetime, end_datetime from items;\n\nALTER TABLE items ADD CONSTRAINT items_collections_fk FOREIGN KEY (collection) REFERENCES collections(id) ON DELETE CASCADE DEFERRABLE;\n\nCREATE OR REPLACE FUNCTION partition_after_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p text;\n t timestamptz := clock_timestamp();\nBEGIN\n RAISE NOTICE 'Updating partition stats %', t;\n FOR p IN SELECT DISTINCT partition\n FROM newdata n JOIN partition_sys_meta p\n ON (n.collection=p.collection AND n.datetime <@ p.partition_dtrange)\n LOOP\n PERFORM run_or_queue(format('SELECT update_partition_stats(%L, %L);', p, true));\n END LOOP;\n IF TG_OP IN ('DELETE','UPDATE') THEN\n DELETE FROM format_item_cache c USING newdata n WHERE c.collection = n.collection AND c.id = n.id;\n END IF;\n RAISE NOTICE 't: % %', t, clock_timestamp() - t;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE TRIGGER items_after_insert_trigger\nAFTER INSERT ON items\nREFERENCING NEW TABLE AS newdata\nFOR EACH STATEMENT\nEXECUTE FUNCTION partition_after_triggerfunc();\n\nCREATE TRIGGER items_after_update_trigger\nAFTER DELETE ON items\nREFERENCING OLD TABLE AS newdata\nFOR EACH STATEMENT\nEXECUTE FUNCTION partition_after_triggerfunc();\n\nCREATE TRIGGER items_after_delete_trigger\nAFTER UPDATE ON items\nREFERENCING NEW TABLE AS newdata\nFOR EACH STATEMENT\nEXECUTE FUNCTION partition_after_triggerfunc();\n\n\nCREATE OR REPLACE FUNCTION content_slim(_item jsonb) RETURNS jsonb AS $$\n SELECT strip_jsonb(_item - '{id,geometry,collection,type}'::text[], collection_base_item(_item->>'collection')) - '{id,geometry,collection,type}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_dehydrate(content jsonb) RETURNS items AS $$\n SELECT\n content->>'id' as id,\n stac_geom(content) as geometry,\n content->>'collection' as collection,\n stac_datetime(content) as datetime,\n stac_end_datetime(content) as end_datetime,\n content_slim(content) as content,\n null::jsonb as private\n ;\n$$ LANGUAGE SQL STABLE;\n\nCREATE OR REPLACE FUNCTION include_field(f text, fields jsonb DEFAULT '{}'::jsonb) RETURNS boolean AS $$\nDECLARE\n includes jsonb := fields->'include';\n excludes jsonb := fields->'exclude';\nBEGIN\n IF f IS NULL THEN\n RETURN NULL;\n END IF;\n\n\n IF\n jsonb_typeof(excludes) = 'array'\n AND jsonb_array_length(excludes)>0\n AND excludes ? f\n THEN\n RETURN FALSE;\n END IF;\n\n IF\n (\n jsonb_typeof(includes) = 'array'\n AND jsonb_array_length(includes) > 0\n AND includes ? f\n ) OR\n (\n includes IS NULL\n OR jsonb_typeof(includes) = 'null'\n OR jsonb_array_length(includes) = 0\n )\n THEN\n RETURN TRUE;\n END IF;\n\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nDROP FUNCTION IF EXISTS content_hydrate(jsonb, jsonb, jsonb);\nCREATE OR REPLACE FUNCTION content_hydrate(\n _item jsonb,\n _base_item jsonb,\n fields jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\n SELECT merge_jsonb(\n jsonb_fields(_item, fields),\n jsonb_fields(_base_item, fields)\n );\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION content_hydrate(_item items, _collection collections, fields jsonb DEFAULT '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\n content jsonb;\n base_item jsonb := _collection.base_item;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry, 20)::jsonb;\n END IF;\n output := content_hydrate(\n jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'collection', _item.collection,\n 'type', 'Feature'\n ) || _item.content,\n _collection.base_item,\n fields\n );\n\n RETURN output;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_nonhydrated(\n _item items,\n fields jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry, 20)::jsonb;\n END IF;\n output := jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'collection', _item.collection,\n 'type', 'Feature'\n ) || _item.content;\n RETURN output;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_hydrate(_item items, fields jsonb DEFAULT '{}'::jsonb) RETURNS jsonb AS $$\n SELECT content_hydrate(\n _item,\n (SELECT c FROM collections c WHERE id=_item.collection LIMIT 1),\n fields\n );\n$$ LANGUAGE SQL STABLE;\n\n\nCREATE UNLOGGED TABLE items_staging (\n content JSONB NOT NULL\n);\nCREATE UNLOGGED TABLE items_staging_ignore (\n content JSONB NOT NULL\n);\nCREATE UNLOGGED TABLE items_staging_upsert (\n content JSONB NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION items_staging_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p record;\n _partitions text[];\n part text;\n ts timestamptz := clock_timestamp();\nBEGIN\n RAISE NOTICE 'Creating Partitions. %', clock_timestamp() - ts;\n\n FOR part IN WITH t AS (\n SELECT\n n.content->>'collection' as collection,\n stac_daterange(n.content->'properties') as dtr,\n partition_trunc\n FROM newdata n JOIN collections ON (n.content->>'collection'=collections.id)\n ), p AS (\n SELECT\n collection,\n COALESCE(date_trunc(partition_trunc::text, lower(dtr)),'-infinity') as d,\n tstzrange(min(lower(dtr)),max(lower(dtr)),'[]') as dtrange,\n tstzrange(min(upper(dtr)),max(upper(dtr)),'[]') as edtrange\n FROM t\n GROUP BY 1,2\n ) SELECT check_partition(collection, dtrange, edtrange) FROM p LOOP\n RAISE NOTICE 'Partition %', part;\n END LOOP;\n\n RAISE NOTICE 'Doing the insert. %', clock_timestamp() - ts;\n IF TG_TABLE_NAME = 'items_staging' THEN\n INSERT INTO items\n SELECT\n (content_dehydrate(content)).*\n FROM newdata;\n RAISE NOTICE 'Doing the delete. %', clock_timestamp() - ts;\n DELETE FROM items_staging;\n ELSIF TG_TABLE_NAME = 'items_staging_ignore' THEN\n INSERT INTO items\n SELECT\n (content_dehydrate(content)).*\n FROM newdata\n ON CONFLICT DO NOTHING;\n RAISE NOTICE 'Doing the delete. %', clock_timestamp() - ts;\n DELETE FROM items_staging_ignore;\n ELSIF TG_TABLE_NAME = 'items_staging_upsert' THEN\n WITH staging_formatted AS (\n SELECT (content_dehydrate(content)).* FROM newdata\n ), deletes AS (\n DELETE FROM items i USING staging_formatted s\n WHERE\n i.id = s.id\n AND i.collection = s.collection\n AND i IS DISTINCT FROM s\n RETURNING i.id, i.collection\n )\n INSERT INTO items\n SELECT s.* FROM\n staging_formatted s\n ON CONFLICT DO NOTHING;\n RAISE NOTICE 'Doing the delete. %', clock_timestamp() - ts;\n DELETE FROM items_staging_upsert;\n END IF;\n RAISE NOTICE 'Done. %', clock_timestamp() - ts;\n\n RETURN NULL;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER items_staging_insert_trigger AFTER INSERT ON items_staging REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\nCREATE TRIGGER items_staging_insert_ignore_trigger AFTER INSERT ON items_staging_ignore REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\nCREATE TRIGGER items_staging_insert_upsert_trigger AFTER INSERT ON items_staging_upsert REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\n\nCREATE OR REPLACE FUNCTION item_by_id(_id text, _collection text DEFAULT NULL) RETURNS items AS\n$$\nDECLARE\n i items%ROWTYPE;\nBEGIN\n SELECT * INTO i FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection) LIMIT 1;\n RETURN i;\nEND;\n$$ LANGUAGE PLPGSQL STABLE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION get_item(_id text, _collection text DEFAULT NULL) RETURNS jsonb AS $$\n SELECT content_hydrate(items) FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection);\n$$ LANGUAGE SQL STABLE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION delete_item(_id text, _collection text DEFAULT NULL) RETURNS VOID AS $$\nDECLARE\nout items%ROWTYPE;\nBEGIN\n DELETE FROM items WHERE id = _id AND (_collection IS NULL OR collection=_collection) RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n--/*\nCREATE OR REPLACE FUNCTION create_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION update_item(content jsonb) RETURNS VOID AS $$\nDECLARE\n old items %ROWTYPE;\n out items%ROWTYPE;\nBEGIN\n PERFORM delete_item(content->>'id', content->>'collection');\n PERFORM create_item(content);\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION create_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION collection_bbox(id text) RETURNS jsonb AS $$\n SELECT (replace(replace(replace(st_extent(geometry)::text,'BOX(','[['),')',']]'),' ',','))::jsonb\n FROM items WHERE collection=$1;\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION collection_temporal_extent(id text) RETURNS jsonb AS $$\n SELECT to_jsonb(array[array[min(datetime)::text, max(datetime)::text]])\n FROM items WHERE collection=$1;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection_extents() RETURNS VOID AS $$\nUPDATE collections SET\n content = content ||\n jsonb_build_object(\n 'extent', jsonb_build_object(\n 'spatial', jsonb_build_object(\n 'bbox', collection_bbox(collections.id)\n ),\n 'temporal', jsonb_build_object(\n 'interval', collection_temporal_extent(collections.id)\n )\n )\n )\n;\n$$ LANGUAGE SQL;\nCREATE TABLE partition_stats (\n partition text PRIMARY KEY,\n dtrange tstzrange,\n edtrange tstzrange,\n spatial geometry,\n last_updated timestamptz,\n keys text[]\n) WITH (FILLFACTOR=90);\n\nCREATE INDEX partitions_range_idx ON partition_stats USING GIST(dtrange);\n\n\nCREATE OR REPLACE FUNCTION constraint_tstzrange(expr text) RETURNS tstzrange AS $$\n WITH t AS (\n SELECT regexp_matches(\n expr,\n E'\\\\(''\\([0-9 :+-]*\\)''\\\\).*\\\\(''\\([0-9 :+-]*\\)''\\\\)'\n ) AS m\n ) SELECT tstzrange(m[1]::timestamptz, m[2]::timestamptz) FROM t\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE STRICT;\n\nCREATE OR REPLACE FUNCTION dt_constraint(coid oid, OUT dt tstzrange, OUT edt tstzrange) RETURNS RECORD AS $$\nDECLARE\n expr text := pg_get_constraintdef(coid);\n matches timestamptz[];\nBEGIN\n IF expr LIKE '%NULL%' THEN\n dt := tstzrange(null::timestamptz, null::timestamptz);\n edt := tstzrange(null::timestamptz, null::timestamptz);\n RETURN;\n END IF;\n WITH f AS (SELECT (regexp_matches(expr, E'([0-9]{4}-[0-1][0-9]-[0-3][0-9] [0-2][0-9]:[0-5][0-9]:[0-5][0-9]\\.?[0-9]*)', 'g'))[1] f)\n SELECT array_agg(f::timestamptz) INTO matches FROM f;\n IF cardinality(matches) = 4 THEN\n dt := tstzrange(matches[1], matches[2],'[]');\n edt := tstzrange(matches[3], matches[4], '[]');\n RETURN;\n ELSIF cardinality(matches) = 2 THEN\n edt := tstzrange(matches[1], matches[2],'[]');\n RETURN;\n END IF;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\nCREATE OR REPLACE VIEW partition_sys_meta AS\nSELECT\n relid::text as partition,\n replace(replace(CASE WHEN level = 1 THEN pg_get_expr(c.relpartbound, c.oid)\n ELSE pg_get_expr(parent.relpartbound, parent.oid)\n END, 'FOR VALUES IN (''',''), ''')','') AS collection,\n level,\n c.reltuples,\n c.relhastriggers,\n COALESCE(constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as partition_dtrange,\n COALESCE((dt_constraint(edt.oid)).dt, constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as constraint_dtrange,\n COALESCE((dt_constraint(edt.oid)).edt, tstzrange('-infinity', 'infinity','[]')) as constraint_edtrange\nFROM\n pg_partition_tree('items')\n JOIN pg_class c ON (relid::regclass = c.oid)\n JOIN pg_class parent ON (parentrelid::regclass = parent.oid AND isleaf)\n LEFT JOIN pg_constraint edt ON (conrelid=c.oid AND contype='c')\nWHERE isleaf\n;\n\nCREATE VIEW partitions_view AS\nSELECT\n relid::text as partition,\n replace(replace(CASE WHEN level = 1 THEN pg_get_expr(c.relpartbound, c.oid)\n ELSE pg_get_expr(parent.relpartbound, parent.oid)\n END, 'FOR VALUES IN (''',''), ''')','') AS collection,\n level,\n c.reltuples,\n c.relhastriggers,\n COALESCE(constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as partition_dtrange,\n COALESCE((dt_constraint(edt.oid)).dt, constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as constraint_dtrange,\n COALESCE((dt_constraint(edt.oid)).edt, tstzrange('-infinity', 'infinity','[]')) as constraint_edtrange,\n dtrange,\n edtrange,\n spatial,\n last_updated\nFROM\n pg_partition_tree('items')\n JOIN pg_class c ON (relid::regclass = c.oid)\n JOIN pg_class parent ON (parentrelid::regclass = parent.oid AND isleaf)\n LEFT JOIN pg_constraint edt ON (conrelid=c.oid AND contype='c')\n LEFT JOIN partition_stats ON (relid::text=partition)\nWHERE isleaf\n;\n\nCREATE MATERIALIZED VIEW partitions AS\nSELECT * FROM partitions_view;\nCREATE UNIQUE INDEX ON partitions (partition);\n\nCREATE MATERIALIZED VIEW partition_steps AS\nSELECT\n partition as name,\n date_trunc('month',lower(partition_dtrange)) as sdate,\n date_trunc('month', upper(partition_dtrange)) + '1 month'::interval as edate\n FROM partitions_view WHERE partition_dtrange IS NOT NULL AND partition_dtrange != 'empty'::tstzrange\n ORDER BY dtrange ASC\n;\n\n\nCREATE OR REPLACE FUNCTION update_partition_stats_q(_partition text, istrigger boolean default false) RETURNS VOID AS $$\nDECLARE\nBEGIN\n PERFORM run_or_queue(\n format('SELECT update_partition_stats(%L, %L);', _partition, istrigger)\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION update_partition_stats(_partition text, istrigger boolean default false) RETURNS VOID AS $$\nDECLARE\n dtrange tstzrange;\n edtrange tstzrange;\n cdtrange tstzrange;\n cedtrange tstzrange;\n extent geometry;\n collection text;\nBEGIN\n RAISE NOTICE 'Updating stats for %.', _partition;\n EXECUTE format(\n $q$\n SELECT\n tstzrange(min(datetime), max(datetime),'[]'),\n tstzrange(min(end_datetime), max(end_datetime), '[]')\n FROM %I\n $q$,\n _partition\n ) INTO dtrange, edtrange;\n extent := st_estimatedextent('pgstac', _partition, 'geometry');\n INSERT INTO partition_stats (partition, dtrange, edtrange, spatial, last_updated)\n SELECT _partition, dtrange, edtrange, extent, now()\n ON CONFLICT (partition) DO\n UPDATE SET\n dtrange=EXCLUDED.dtrange,\n edtrange=EXCLUDED.edtrange,\n spatial=EXCLUDED.spatial,\n last_updated=EXCLUDED.last_updated\n ;\n\n SELECT\n constraint_dtrange, constraint_edtrange, pv.collection\n INTO cdtrange, cedtrange, collection\n FROM partitions_view pv WHERE partition = _partition;\n REFRESH MATERIALIZED VIEW partitions;\n REFRESH MATERIALIZED VIEW partition_steps;\n\n\n RAISE NOTICE 'Checking if we need to modify constraints.';\n IF\n (cdtrange IS DISTINCT FROM dtrange OR edtrange IS DISTINCT FROM cedtrange)\n AND NOT istrigger\n THEN\n RAISE NOTICE 'Modifying Constraints';\n RAISE NOTICE 'Existing % %', cdtrange, cedtrange;\n RAISE NOTICE 'New % %', dtrange, edtrange;\n PERFORM drop_table_constraints(_partition);\n PERFORM create_table_constraints(_partition, dtrange, edtrange);\n END IF;\n RAISE NOTICE 'Checking if we need to update collection extents.';\n IF get_setting_bool('update_collection_extent') THEN\n RAISE NOTICE 'updating collection extent for %', collection;\n PERFORM run_or_queue(format($q$\n UPDATE collections\n SET content = jsonb_set_lax(\n content,\n '{extent}'::text[],\n collection_extent(%L, FALSE),\n true,\n 'use_json_null'\n ) WHERE id=%L\n ;\n $q$, collection, collection));\n ELSE\n RAISE NOTICE 'Not updating collection extent for %', collection;\n END IF;\n\nEND;\n$$ LANGUAGE PLPGSQL STRICT;\n\n\nCREATE OR REPLACE FUNCTION partition_name( IN collection text, IN dt timestamptz, OUT partition_name text, OUT partition_range tstzrange) AS $$\nDECLARE\n c RECORD;\n parent_name text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n parent_name := format('_items_%s', c.key);\n\n\n IF c.partition_trunc = 'year' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYY'));\n ELSIF c.partition_trunc = 'month' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYYMM'));\n ELSE\n partition_name := parent_name;\n partition_range := tstzrange('-infinity'::timestamptz, 'infinity'::timestamptz, '[]');\n END IF;\n IF partition_range IS NULL THEN\n partition_range := tstzrange(\n date_trunc(c.partition_trunc::text, dt),\n date_trunc(c.partition_trunc::text, dt) + concat('1 ', c.partition_trunc)::interval\n );\n END IF;\n RETURN;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION drop_table_constraints(t text) RETURNS text AS $$\nDECLARE\n q text;\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM partitions_view WHERE partition=t) THEN\n RETURN NULL;\n END IF;\n FOR q IN SELECT FORMAT(\n $q$\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n $q$,\n t,\n conname\n ) FROM pg_constraint\n WHERE conrelid=t::regclass::oid AND contype='c'\n LOOP\n EXECUTE q;\n END LOOP;\n RETURN t;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE OR REPLACE FUNCTION create_table_constraints(t text, _dtrange tstzrange, _edtrange tstzrange) RETURNS text AS $$\nDECLARE\n q text;\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM partitions_view WHERE partition=t) THEN\n RETURN NULL;\n END IF;\n RAISE NOTICE 'Creating Table Constraints for % % %', t, _dtrange, _edtrange;\n IF _dtrange = 'empty' AND _edtrange = 'empty' THEN\n q :=format(\n $q$\n DO $block$\n BEGIN\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n ALTER TABLE %I\n ADD CONSTRAINT %I\n CHECK (((datetime IS NULL) AND (end_datetime IS NULL))) NOT VALID\n ;\n ALTER TABLE %I\n VALIDATE CONSTRAINT %I\n ;\n\n\n\n EXCEPTION WHEN others THEN\n RAISE WARNING '%%, Issue Altering Constraints. Please run update_partition_stats(%I)', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n $block$;\n $q$,\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t),\n t\n );\n ELSE\n q :=format(\n $q$\n DO $block$\n BEGIN\n\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n ALTER TABLE %I\n ADD CONSTRAINT %I\n CHECK (\n (datetime >= %L)\n AND (datetime <= %L)\n AND (end_datetime >= %L)\n AND (end_datetime <= %L)\n ) NOT VALID\n ;\n ALTER TABLE %I\n VALIDATE CONSTRAINT %I\n ;\n\n\n\n EXCEPTION WHEN others THEN\n RAISE WARNING '%%, Issue Altering Constraints. Please run update_partition_stats(%I)', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n $block$;\n $q$,\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t),\n lower(_dtrange),\n upper(_dtrange),\n lower(_edtrange),\n upper(_edtrange),\n t,\n format('%s_dt', t),\n t\n );\n END IF;\n PERFORM run_or_queue(q);\n RETURN t;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION check_partition(\n _collection text,\n _dtrange tstzrange,\n _edtrange tstzrange\n) RETURNS text AS $$\nDECLARE\n c RECORD;\n pm RECORD;\n _partition_name text;\n _partition_dtrange tstzrange;\n _constraint_dtrange tstzrange;\n _constraint_edtrange tstzrange;\n q text;\n deferrable_q text;\n err_context text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=_collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', _collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n\n IF c.partition_trunc IS NOT NULL THEN\n _partition_dtrange := tstzrange(\n date_trunc(c.partition_trunc, lower(_dtrange)),\n date_trunc(c.partition_trunc, lower(_dtrange)) + (concat('1 ', c.partition_trunc))::interval,\n '[)'\n );\n ELSE\n _partition_dtrange := '[-infinity, infinity]'::tstzrange;\n END IF;\n\n IF NOT _partition_dtrange @> _dtrange THEN\n RAISE EXCEPTION 'dtrange % is greater than the partition size % for collection %', _dtrange, c.partition_trunc, _collection;\n END IF;\n\n\n IF c.partition_trunc = 'year' THEN\n _partition_name := format('_items_%s_%s', c.key, to_char(lower(_partition_dtrange),'YYYY'));\n ELSIF c.partition_trunc = 'month' THEN\n _partition_name := format('_items_%s_%s', c.key, to_char(lower(_partition_dtrange),'YYYYMM'));\n ELSE\n _partition_name := format('_items_%s', c.key);\n END IF;\n\n SELECT * INTO pm FROM partition_sys_meta WHERE collection=_collection AND partition_dtrange @> _dtrange;\n IF FOUND THEN\n RAISE NOTICE '% % %', _edtrange, _dtrange, pm;\n _constraint_edtrange :=\n tstzrange(\n least(\n lower(_edtrange),\n nullif(lower(pm.constraint_edtrange), '-infinity')\n ),\n greatest(\n upper(_edtrange),\n nullif(upper(pm.constraint_edtrange), 'infinity')\n ),\n '[]'\n );\n _constraint_dtrange :=\n tstzrange(\n least(\n lower(_dtrange),\n nullif(lower(pm.constraint_dtrange), '-infinity')\n ),\n greatest(\n upper(_dtrange),\n nullif(upper(pm.constraint_dtrange), 'infinity')\n ),\n '[]'\n );\n\n IF pm.constraint_edtrange @> _edtrange AND pm.constraint_dtrange @> _dtrange THEN\n RETURN pm.partition;\n ELSE\n PERFORM drop_table_constraints(_partition_name);\n END IF;\n ELSE\n _constraint_edtrange := _edtrange;\n _constraint_dtrange := _dtrange;\n END IF;\n RAISE NOTICE 'Creating partition % %', _partition_name, _partition_dtrange;\n IF c.partition_trunc IS NULL THEN\n q := format(\n $q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n $q$,\n _partition_name,\n _collection,\n concat(_partition_name,'_pk'),\n _partition_name\n );\n ELSE\n q := format(\n $q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L) PARTITION BY RANGE (datetime);\n CREATE TABLE IF NOT EXISTS %I partition OF %I FOR VALUES FROM (%L) TO (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n $q$,\n format('_items_%s', c.key),\n _collection,\n _partition_name,\n format('_items_%s', c.key),\n lower(_partition_dtrange),\n upper(_partition_dtrange),\n format('%s_pk', _partition_name),\n _partition_name\n );\n END IF;\n\n BEGIN\n EXECUTE q;\n EXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', _partition_name;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\n END;\n PERFORM create_table_constraints(_partition_name, _constraint_dtrange, _constraint_edtrange);\n PERFORM maintain_partitions(_partition_name);\n PERFORM update_partition_stats_q(_partition_name, true);\n REFRESH MATERIALIZED VIEW partitions;\n REFRESH MATERIALIZED VIEW partition_steps;\n RETURN _partition_name;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION repartition(_collection text, _partition_trunc text, triggered boolean DEFAULT FALSE) RETURNS text AS $$\nDECLARE\n c RECORD;\n q text;\n from_trunc text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=_collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', _collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n IF triggered THEN\n RAISE NOTICE 'Converting % to % partitioning via Trigger', _collection, _partition_trunc;\n ELSE\n RAISE NOTICE 'Converting % from using % to % partitioning', _collection, c.partition_trunc, _partition_trunc;\n IF c.partition_trunc IS NOT DISTINCT FROM _partition_trunc THEN\n RAISE NOTICE 'Collection % already set to use partition by %', _collection, _partition_trunc;\n RETURN _collection;\n END IF;\n END IF;\n\n IF EXISTS (SELECT 1 FROM partitions_view WHERE collection=_collection LIMIT 1) THEN\n EXECUTE format(\n $q$\n CREATE TEMP TABLE changepartitionstaging ON COMMIT DROP AS SELECT * FROM %I;\n DROP TABLE IF EXISTS %I CASCADE;\n WITH p AS (\n SELECT\n collection,\n CASE WHEN %L IS NULL THEN '-infinity'::timestamptz\n ELSE date_trunc(%L, datetime)\n END as d,\n tstzrange(min(datetime),max(datetime),'[]') as dtrange,\n tstzrange(min(datetime),max(datetime),'[]') as edtrange\n FROM changepartitionstaging\n GROUP BY 1,2\n ) SELECT check_partition(collection, dtrange, edtrange) FROM p;\n INSERT INTO items SELECT * FROM changepartitionstaging;\n DROP TABLE changepartitionstaging;\n $q$,\n concat('_items_', c.key),\n concat('_items_', c.key),\n c.partition_trunc,\n c.partition_trunc\n );\n END IF;\n RETURN _collection;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE OR REPLACE FUNCTION collections_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n q text;\n partition_name text := format('_items_%s', NEW.key);\n partition_exists boolean := false;\n partition_empty boolean := true;\n err_context text;\n loadtemp boolean := FALSE;\nBEGIN\n RAISE NOTICE 'Collection Trigger. % %', NEW.id, NEW.key;\n IF TG_OP = 'UPDATE' AND NEW.partition_trunc IS DISTINCT FROM OLD.partition_trunc THEN\n PERFORM repartition(NEW.id, NEW.partition_trunc, TRUE);\n END IF;\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER collections_trigger AFTER\nINSERT\nOR\nUPDATE ON collections\nFOR EACH ROW EXECUTE FUNCTION collections_trigger_func();\n\nCREATE OR REPLACE FUNCTION chunker(\n IN _where text,\n OUT s timestamptz,\n OUT e timestamptz\n) RETURNS SETOF RECORD AS $$\nDECLARE\n explain jsonb;\nBEGIN\n IF _where IS NULL THEN\n _where := ' TRUE ';\n END IF;\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s;', _where)\n INTO explain;\n\n RETURN QUERY\n WITH t AS (\n SELECT j->>0 as p FROM\n jsonb_path_query(\n explain,\n 'strict $.**.\"Relation Name\" ? (@ != null)'\n ) j\n ),\n parts AS (\n SELECT sdate, edate FROM t JOIN partition_steps ON (t.p = name)\n ),\n times AS (\n SELECT sdate FROM parts\n UNION\n SELECT edate FROM parts\n ),\n uniq AS (\n SELECT DISTINCT sdate FROM times ORDER BY sdate\n ),\n last AS (\n SELECT sdate, lead(sdate, 1) over () as edate FROM uniq\n )\n SELECT sdate, edate FROM last WHERE edate IS NOT NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION partition_queries(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN partitions text[] DEFAULT NULL\n) RETURNS SETOF text AS $$\nDECLARE\n query text;\n sdate timestamptz;\n edate timestamptz;\nBEGIN\nIF _where IS NULL OR trim(_where) = '' THEN\n _where = ' TRUE ';\nEND IF;\nRAISE NOTICE 'Getting chunks for % %', _where, _orderby;\nIF _orderby ILIKE 'datetime d%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 DESC LOOP\n RETURN NEXT format($q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n $q$,\n sdate,\n edate,\n _where,\n _orderby\n );\n END LOOP;\nELSIF _orderby ILIKE 'datetime a%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 ASC LOOP\n RETURN NEXT format($q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n $q$,\n sdate,\n edate,\n _where,\n _orderby\n );\n END LOOP;\nELSE\n query := format($q$\n SELECT * FROM items\n WHERE %s\n ORDER BY %s\n $q$, _where, _orderby\n );\n\n RETURN NEXT query;\n RETURN;\nEND IF;\n\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION partition_query_view(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN _limit int DEFAULT 10\n) RETURNS text AS $$\n WITH p AS (\n SELECT * FROM partition_queries(_where, _orderby) p\n )\n SELECT\n CASE WHEN EXISTS (SELECT 1 FROM p) THEN\n (SELECT format($q$\n SELECT * FROM (\n %s\n ) total LIMIT %s\n $q$,\n string_agg(\n format($q$ SELECT * FROM ( %s ) AS sub $q$, p),\n '\n UNION ALL\n '\n ),\n _limit\n ))\n ELSE NULL\n END FROM p;\n$$ LANGUAGE SQL IMMUTABLE;\n\n\n\n\nCREATE OR REPLACE FUNCTION stac_search_to_where(j jsonb) RETURNS text AS $$\nDECLARE\n where_segments text[];\n _where text;\n dtrange tstzrange;\n collections text[];\n geom geometry;\n sdate timestamptz;\n edate timestamptz;\n filterlang text;\n filter jsonb := j->'filter';\nBEGIN\n IF j ? 'ids' THEN\n where_segments := where_segments || format('id = ANY (%L) ', to_text_array(j->'ids'));\n END IF;\n\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n where_segments := where_segments || format('collection = ANY (%L) ', collections);\n END IF;\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n\n where_segments := where_segments || format(' datetime <= %L::timestamptz AND end_datetime >= %L::timestamptz ',\n edate,\n sdate\n );\n END IF;\n\n geom := stac_geom(j);\n IF geom IS NOT NULL THEN\n where_segments := where_segments || format('st_intersects(geometry, %L)',geom);\n END IF;\n\n filterlang := COALESCE(\n j->>'filter-lang',\n get_setting('default_filter_lang', j->'conf')\n );\n IF NOT filter @? '$.**.op' THEN\n filterlang := 'cql-json';\n END IF;\n\n IF filterlang NOT IN ('cql-json','cql2-json') AND j ? 'filter' THEN\n RAISE EXCEPTION '% is not a supported filter-lang. Please use cql-json or cql2-json.', filterlang;\n END IF;\n\n IF j ? 'query' AND j ? 'filter' THEN\n RAISE EXCEPTION 'Can only use either query or filter at one time.';\n END IF;\n\n IF j ? 'query' THEN\n filter := query_to_cql2(j->'query');\n ELSIF filterlang = 'cql-json' THEN\n filter := cql1_to_cql2(filter);\n END IF;\n RAISE NOTICE 'FILTER: %', filter;\n where_segments := where_segments || cql2_query(filter);\n IF cardinality(where_segments) < 1 THEN\n RETURN ' TRUE ';\n END IF;\n\n _where := array_to_string(array_remove(where_segments, NULL), ' AND ');\n\n IF _where IS NULL OR BTRIM(_where) = '' THEN\n RETURN ' TRUE ';\n END IF;\n RETURN _where;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION parse_sort_dir(_dir text, reverse boolean default false) RETURNS text AS $$\n WITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n ) SELECT\n CASE\n WHEN NOT reverse THEN d\n WHEN d = 'ASC' THEN 'DESC'\n WHEN d = 'DESC' THEN 'ASC'\n END\n FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION sort_dir_to_op(_dir text, prev boolean default false) RETURNS text AS $$\n WITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n ) SELECT\n CASE\n WHEN d = 'ASC' AND prev THEN '<='\n WHEN d = 'DESC' AND prev THEN '>='\n WHEN d = 'ASC' THEN '>='\n WHEN d = 'DESC' THEN '<='\n END\n FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION sort_sqlorderby(\n _search jsonb DEFAULT NULL,\n reverse boolean DEFAULT FALSE\n) RETURNS text AS $$\n WITH sortby AS (\n SELECT coalesce(_search->'sortby','[{\"field\":\"datetime\", \"direction\":\"desc\"}]') as sort\n ), withid AS (\n SELECT CASE\n WHEN sort @? '$[*] ? (@.field == \"id\")' THEN sort\n ELSE sort || '[{\"field\":\"id\", \"direction\":\"desc\"}]'::jsonb\n END as sort\n FROM sortby\n ), withid_rows AS (\n SELECT jsonb_array_elements(sort) as value FROM withid\n ),sorts AS (\n SELECT\n coalesce(\n (queryable(value->>'field')).expression\n ) as key,\n parse_sort_dir(value->>'direction', reverse) as dir\n FROM withid_rows\n )\n SELECT array_to_string(\n array_agg(concat(key, ' ', dir)),\n ', '\n ) FROM sorts;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_sort_dir(sort_item jsonb) RETURNS text AS $$\n SELECT CASE WHEN sort_item->>'direction' ILIKE 'desc%' THEN 'DESC' ELSE 'ASC' END;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION get_token_val_str(\n _field text,\n _item items\n) RETURNS text AS $$\nDECLARE\n q text;\n literal text;\nBEGIN\n q := format($q$ SELECT quote_literal(%s) FROM (SELECT $1.*) as r;$q$, _field);\n EXECUTE q INTO literal USING _item;\n RETURN literal;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n\nCREATE OR REPLACE FUNCTION get_token_record(IN _token text, OUT prev BOOLEAN, OUT item items) RETURNS RECORD AS $$\nDECLARE\n _itemid text := _token;\n _collectionid text;\nBEGIN\n IF _token IS NULL THEN\n RETURN;\n END IF;\n RAISE NOTICE 'Looking for token: %', _token;\n prev := FALSE;\n IF _token ILIKE 'prev:%' THEN\n _itemid := replace(_token, 'prev:','');\n prev := TRUE;\n ELSIF _token ILIKE 'next:%' THEN\n _itemid := replace(_token, 'next:', '');\n END IF;\n SELECT id INTO _collectionid FROM collections WHERE _itemid LIKE concat(id,':%');\n IF FOUND THEN\n _itemid := replace(_itemid, concat(_collectionid,':'), '');\n SELECT * INTO item FROM items WHERE id=_itemid AND collection=_collectionid;\n ELSE\n SELECT * INTO item FROM items WHERE id=_itemid;\n END IF;\n IF item IS NULL THEN\n RAISE EXCEPTION 'Could not find item using token: % item: % collection: %', _token, _itemid, _collectionid;\n END IF;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\n\nCREATE OR REPLACE FUNCTION get_token_filter(\n _sortby jsonb DEFAULT '[{\"field\":\"datetime\",\"direction\":\"desc\"}]'::jsonb,\n token_item items DEFAULT NULL,\n prev boolean DEFAULT FALSE,\n inclusive boolean DEFAULT FALSE\n) RETURNS text AS $$\nDECLARE\n ltop text := '<';\n gtop text := '>';\n dir text;\n sort record;\n orfilter text := '';\n orfilters text[] := '{}'::text[];\n andfilters text[] := '{}'::text[];\n output text;\n token_where text;\nBEGIN\n IF _sortby IS NULL OR _sortby = '[]'::jsonb THEN\n _sortby := '[{\"field\":\"datetime\",\"direction\":\"desc\"}]'::jsonb;\n END IF;\n _sortby := _sortby || jsonb_build_object('field','id','direction',_sortby->0->>'direction');\n RAISE NOTICE 'Getting Token Filter. % %', _sortby, token_item;\n IF inclusive THEN\n orfilters := orfilters || format('( id=%L AND collection=%L )' , token_item.id, token_item.collection);\n END IF;\n\n FOR sort IN\n WITH s1 AS (\n SELECT\n _row,\n (queryable(value->>'field')).expression as _field,\n (value->>'field' = 'id') as _isid,\n get_sort_dir(value) as _dir\n FROM jsonb_array_elements(_sortby)\n WITH ORDINALITY AS t(value, _row)\n )\n SELECT\n _row,\n _field,\n _dir,\n get_token_val_str(_field, token_item) as _val\n FROM s1\n WHERE _row <= (SELECT min(_row) FROM s1 WHERE _isid)\n LOOP\n orfilter := NULL;\n RAISE NOTICE 'SORT: %', sort;\n IF sort._val IS NOT NULL AND ((prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC')) THEN\n orfilter := format($f$(\n (%s %s %s) OR (%s IS NULL)\n )$f$,\n sort._field,\n ltop,\n sort._val,\n sort._val\n );\n ELSIF sort._val IS NULL AND ((prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC')) THEN\n RAISE NOTICE '< but null';\n orfilter := format('%s IS NOT NULL', sort._field);\n ELSIF sort._val IS NULL THEN\n RAISE NOTICE '> but null';\n ELSE\n orfilter := format($f$(\n (%s %s %s) OR (%s IS NULL)\n )$f$,\n sort._field,\n gtop,\n sort._val,\n sort._field\n );\n END IF;\n RAISE NOTICE 'ORFILTER: %', orfilter;\n\n IF orfilter IS NOT NULL THEN\n IF sort._row = 1 THEN\n orfilters := orfilters || orfilter;\n ELSE\n orfilters := orfilters || format('(%s AND %s)', array_to_string(andfilters, ' AND '), orfilter);\n END IF;\n END IF;\n IF sort._val IS NOT NULL THEN\n andfilters := andfilters || format('%s = %s', sort._field, sort._val);\n ELSE\n andfilters := andfilters || format('%s IS NULL', sort._field);\n END IF;\n END LOOP;\n\n output := array_to_string(orfilters, ' OR ');\n\n token_where := concat('(',coalesce(output,'true'),')');\n IF trim(token_where) = '' THEN\n token_where := NULL;\n END IF;\n RAISE NOTICE 'TOKEN_WHERE: %',token_where;\n RETURN token_where;\n END;\n$$ LANGUAGE PLPGSQL SET transform_null_equals TO TRUE\n;\n\nCREATE OR REPLACE FUNCTION search_tohash(jsonb) RETURNS jsonb AS $$\n SELECT $1 - '{token,limit,context,includes,excludes}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION search_hash(jsonb, jsonb) RETURNS text AS $$\n SELECT md5(concat(search_tohash($1)::text,$2::text));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TABLE IF NOT EXISTS searches(\n hash text GENERATED ALWAYS AS (search_hash(search, metadata)) STORED PRIMARY KEY,\n search jsonb NOT NULL,\n _where text,\n orderby text,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n metadata jsonb DEFAULT '{}'::jsonb NOT NULL\n);\n\nCREATE TABLE IF NOT EXISTS search_wheres(\n id bigint generated always as identity primary key,\n _where text NOT NULL,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n statslastupdated timestamptz,\n estimated_count bigint,\n estimated_cost float,\n time_to_estimate float,\n total_count bigint,\n time_to_count float,\n partitions text[]\n);\n\nCREATE INDEX IF NOT EXISTS search_wheres_partitions ON search_wheres USING GIN (partitions);\nCREATE UNIQUE INDEX IF NOT EXISTS search_wheres_where ON search_wheres ((md5(_where)));\n\nCREATE OR REPLACE FUNCTION where_stats(inwhere text, updatestats boolean default false, conf jsonb default null) RETURNS search_wheres AS $$\nDECLARE\n t timestamptz;\n i interval;\n explain_json jsonb;\n partitions text[];\n sw search_wheres%ROWTYPE;\n inwhere_hash text := md5(inwhere);\n _context text := lower(context(conf));\n _stats_ttl interval := context_stats_ttl(conf);\n _estimated_cost float := context_estimated_cost(conf);\n _estimated_count int := context_estimated_count(conf);\nBEGIN\n IF _context = 'off' THEN\n sw._where := inwhere;\n return sw;\n END IF;\n\n SELECT * INTO sw FROM search_wheres WHERE md5(_where)=inwhere_hash FOR UPDATE;\n\n -- Update statistics if explicitly set, if statistics do not exist, or statistics ttl has expired\n IF NOT updatestats THEN\n RAISE NOTICE 'Checking if update is needed for: % .', inwhere;\n RAISE NOTICE 'Stats Last Updated: %', sw.statslastupdated;\n RAISE NOTICE 'TTL: %, Age: %', _stats_ttl, now() - sw.statslastupdated;\n RAISE NOTICE 'Context: %, Existing Total: %', _context, sw.total_count;\n IF\n sw.statslastupdated IS NULL\n OR (now() - sw.statslastupdated) > _stats_ttl\n OR (context(conf) != 'off' AND sw.total_count IS NULL)\n THEN\n updatestats := TRUE;\n END IF;\n END IF;\n\n sw._where := inwhere;\n sw.lastused := now();\n sw.usecount := coalesce(sw.usecount,0) + 1;\n\n IF NOT updatestats THEN\n UPDATE search_wheres SET\n lastused = sw.lastused,\n usecount = sw.usecount\n WHERE md5(_where) = inwhere_hash\n RETURNING * INTO sw\n ;\n RETURN sw;\n END IF;\n\n -- Use explain to get estimated count/cost and a list of the partitions that would be hit by the query\n t := clock_timestamp();\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s', inwhere)\n INTO explain_json;\n RAISE NOTICE 'Time for just the explain: %', clock_timestamp() - t;\n i := clock_timestamp() - t;\n\n sw.statslastupdated := now();\n sw.estimated_count := explain_json->0->'Plan'->'Plan Rows';\n sw.estimated_cost := explain_json->0->'Plan'->'Total Cost';\n sw.time_to_estimate := extract(epoch from i);\n\n RAISE NOTICE 'ESTIMATED_COUNT: % < %', sw.estimated_count, _estimated_count;\n RAISE NOTICE 'ESTIMATED_COST: % < %', sw.estimated_cost, _estimated_cost;\n\n -- Do a full count of rows if context is set to on or if auto is set and estimates are low enough\n IF\n _context = 'on'\n OR\n ( _context = 'auto' AND\n (\n sw.estimated_count < _estimated_count\n AND\n sw.estimated_cost < _estimated_cost\n )\n )\n THEN\n t := clock_timestamp();\n RAISE NOTICE 'Calculating actual count...';\n EXECUTE format(\n 'SELECT count(*) FROM items WHERE %s',\n inwhere\n ) INTO sw.total_count;\n i := clock_timestamp() - t;\n RAISE NOTICE 'Actual Count: % -- %', sw.total_count, i;\n sw.time_to_count := extract(epoch FROM i);\n ELSE\n sw.total_count := NULL;\n sw.time_to_count := NULL;\n END IF;\n\n\n INSERT INTO search_wheres\n (_where, lastused, usecount, statslastupdated, estimated_count, estimated_cost, time_to_estimate, partitions, total_count, time_to_count)\n SELECT sw._where, sw.lastused, sw.usecount, sw.statslastupdated, sw.estimated_count, sw.estimated_cost, sw.time_to_estimate, sw.partitions, sw.total_count, sw.time_to_count\n ON CONFLICT ((md5(_where)))\n DO UPDATE\n SET\n lastused = sw.lastused,\n usecount = sw.usecount,\n statslastupdated = sw.statslastupdated,\n estimated_count = sw.estimated_count,\n estimated_cost = sw.estimated_cost,\n time_to_estimate = sw.time_to_estimate,\n total_count = sw.total_count,\n time_to_count = sw.time_to_count\n ;\n RETURN sw;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION search_query(\n _search jsonb = '{}'::jsonb,\n updatestats boolean = false,\n _metadata jsonb = '{}'::jsonb\n) RETURNS searches AS $$\nDECLARE\n search searches%ROWTYPE;\n pexplain jsonb;\n t timestamptz;\n i interval;\n _hash text := search_hash(_search, _metadata);\n doupdate boolean := FALSE;\n insertfound boolean := FALSE;\nBEGIN\n SELECT * INTO search FROM searches\n WHERE hash=_hash;\n\n search.hash := _hash;\n\n -- Calculate the where clause if not already calculated\n IF search._where IS NULL THEN\n search._where := stac_search_to_where(_search);\n ELSE\n doupdate := TRUE;\n END IF;\n\n -- Calculate the order by clause if not already calculated\n IF search.orderby IS NULL THEN\n search.orderby := sort_sqlorderby(_search);\n ELSE\n doupdate := TRUE;\n END IF;\n\n PERFORM where_stats(search._where, updatestats, _search->'conf');\n\n IF NOT doupdate THEN\n INSERT INTO searches (search, _where, orderby, lastused, usecount, metadata)\n VALUES (_search, search._where, search.orderby, clock_timestamp(), 1, _metadata)\n ON CONFLICT (hash) DO NOTHING RETURNING * INTO search;\n IF FOUND THEN\n RETURN search;\n END IF;\n END IF;\n\n UPDATE searches\n SET\n lastused=clock_timestamp(),\n usecount=usecount+1\n WHERE hash=(\n SELECT hash FROM searches\n WHERE hash=_hash\n FOR UPDATE SKIP LOCKED\n );\n IF NOT FOUND THEN\n RAISE NOTICE 'Did not update stats for % due to lock. (This is generally OK)', _search;\n END IF;\n\n RETURN search;\n\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE OR REPLACE FUNCTION search_rows(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN partitions text[] DEFAULT NULL,\n IN _limit int DEFAULT 10\n) RETURNS SETOF items AS $$\nDECLARE\n base_query text;\n query text;\n sdate timestamptz;\n edate timestamptz;\n n int;\n records_left int := _limit;\n timer timestamptz := clock_timestamp();\n full_timer timestamptz := clock_timestamp();\nBEGIN\nIF _where IS NULL OR trim(_where) = '' THEN\n _where = ' TRUE ';\nEND IF;\nRAISE NOTICE 'Getting chunks for % %', _where, _orderby;\n\nbase_query := $q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n LIMIT %L\n$q$;\n\nIF _orderby ILIKE 'datetime d%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 DESC LOOP\n RAISE NOTICE 'Running Query for % to %. %', sdate, edate, age_ms(full_timer);\n query := format(\n base_query,\n sdate,\n edate,\n _where,\n _orderby,\n records_left\n );\n RAISE LOG 'QUERY: %', query;\n timer := clock_timestamp();\n RETURN QUERY EXECUTE query;\n\n GET DIAGNOSTICS n = ROW_COUNT;\n records_left := records_left - n;\n RAISE NOTICE 'Returned %/% Rows From % to %. % to go. Time: %ms', n, _limit, sdate, edate, records_left, age_ms(timer);\n timer := clock_timestamp();\n IF records_left <= 0 THEN\n RAISE NOTICE 'SEARCH_ROWS TOOK %ms', age_ms(full_timer);\n RETURN;\n END IF;\n END LOOP;\nELSIF _orderby ILIKE 'datetime a%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 ASC LOOP\n RAISE NOTICE 'Running Query for % to %. %', sdate, edate, age_ms(full_timer);\n query := format(\n base_query,\n sdate,\n edate,\n _where,\n _orderby,\n records_left\n );\n RAISE LOG 'QUERY: %', query;\n timer := clock_timestamp();\n RETURN QUERY EXECUTE query;\n\n GET DIAGNOSTICS n = ROW_COUNT;\n records_left := records_left - n;\n RAISE NOTICE 'Returned %/% Rows From % to %. % to go. Time: %ms', n, _limit, sdate, edate, records_left, age_ms(timer);\n timer := clock_timestamp();\n IF records_left <= 0 THEN\n RAISE NOTICE 'SEARCH_ROWS TOOK %ms', age_ms(full_timer);\n RETURN;\n END IF;\n END LOOP;\nELSE\n query := format($q$\n SELECT * FROM items\n WHERE %s\n ORDER BY %s\n LIMIT %L\n $q$, _where, _orderby, _limit\n );\n RAISE LOG 'QUERY: %', query;\n timer := clock_timestamp();\n RETURN QUERY EXECUTE query;\n RAISE NOTICE 'FULL QUERY TOOK %ms', age_ms(timer);\nEND IF;\nRAISE NOTICE 'SEARCH_ROWS TOOK %ms', age_ms(full_timer);\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE UNLOGGED TABLE format_item_cache(\n id text,\n collection text,\n fields text,\n hydrated bool,\n output jsonb,\n lastused timestamptz DEFAULT now(),\n usecount int DEFAULT 1,\n timetoformat float,\n PRIMARY KEY (collection, id, fields, hydrated)\n);\nCREATE INDEX ON format_item_cache (lastused);\n\nCREATE OR REPLACE FUNCTION format_item(_item items, _fields jsonb DEFAULT '{}', _hydrated bool DEFAULT TRUE) RETURNS jsonb AS $$\nDECLARE\n cache bool := get_setting_bool('format_cache');\n _output jsonb := null;\n t timestamptz := clock_timestamp();\nBEGIN\n IF cache THEN\n SELECT output INTO _output FROM format_item_cache\n WHERE id=_item.id AND collection=_item.collection AND fields=_fields::text AND hydrated=_hydrated;\n END IF;\n IF _output IS NULL THEN\n IF _hydrated THEN\n _output := content_hydrate(_item, _fields);\n ELSE\n _output := content_nonhydrated(_item, _fields);\n END IF;\n END IF;\n IF cache THEN\n INSERT INTO format_item_cache (id, collection, fields, hydrated, output, timetoformat)\n VALUES (_item.id, _item.collection, _fields::text, _hydrated, _output, age_ms(t))\n ON CONFLICT(collection, id, fields, hydrated) DO\n UPDATE\n SET lastused=now(), usecount = format_item_cache.usecount + 1\n ;\n END IF;\n RETURN _output;\n\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION search(_search jsonb = '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n searches searches%ROWTYPE;\n _where text;\n orderby text;\n search_where search_wheres%ROWTYPE;\n total_count bigint;\n token record;\n token_prev boolean;\n token_item items%ROWTYPE;\n token_where text;\n full_where text;\n init_ts timestamptz := clock_timestamp();\n timer timestamptz := clock_timestamp();\n hydrate bool := NOT (_search->'conf'->>'nohydrate' IS NOT NULL AND (_search->'conf'->>'nohydrate')::boolean = true);\n prev text;\n next text;\n context jsonb;\n collection jsonb;\n out_records jsonb;\n out_len int;\n _limit int := coalesce((_search->>'limit')::int, 10);\n _querylimit int;\n _fields jsonb := coalesce(_search->'fields', '{}'::jsonb);\n has_prev boolean := FALSE;\n has_next boolean := FALSE;\nBEGIN\n searches := search_query(_search);\n _where := searches._where;\n orderby := searches.orderby;\n search_where := where_stats(_where);\n total_count := coalesce(search_where.total_count, search_where.estimated_count);\n RAISE NOTICE 'SEARCH:TOKEN: %', _search->>'token';\n token := get_token_record(_search->>'token');\n RAISE NOTICE '***TOKEN: %', token;\n _querylimit := _limit + 1;\n IF token IS NOT NULL THEN\n token_prev := token.prev;\n token_item := token.item;\n token_where := get_token_filter(_search->'sortby', token_item, token_prev, FALSE);\n RAISE LOG 'TOKEN_WHERE: % (%ms from search start)', token_where, age_ms(timer);\n IF token_prev THEN -- if we are using a prev token, we know has_next is true\n RAISE LOG 'There is a previous token, so automatically setting has_next to true';\n has_next := TRUE;\n orderby := sort_sqlorderby(_search, TRUE);\n ELSE\n RAISE LOG 'There is a next token, so automatically setting has_prev to true';\n has_prev := TRUE;\n\n END IF;\n ELSE -- if there was no token, we know there is no prev\n RAISE LOG 'There is no token, so we know there is no prev. setting has_prev to false';\n has_prev := FALSE;\n END IF;\n\n full_where := concat_ws(' AND ', _where, token_where);\n RAISE NOTICE 'FULL WHERE CLAUSE: %', full_where;\n RAISE NOTICE 'Time to get counts and build query %', age_ms(timer);\n timer := clock_timestamp();\n\n IF hydrate THEN\n RAISE NOTICE 'Getting hydrated data.';\n ELSE\n RAISE NOTICE 'Getting non-hydrated data.';\n END IF;\n RAISE NOTICE 'CACHE SET TO %', get_setting_bool('format_cache');\n RAISE NOTICE 'Time to set hydration/formatting %', age_ms(timer);\n timer := clock_timestamp();\n SELECT jsonb_agg(format_item(i, _fields, hydrate)) INTO out_records\n FROM search_rows(\n full_where,\n orderby,\n search_where.partitions,\n _querylimit\n ) as i;\n\n RAISE NOTICE 'Time to fetch rows %', age_ms(timer);\n timer := clock_timestamp();\n\n\n IF token_prev THEN\n out_records := flip_jsonb_array(out_records);\n END IF;\n\n RAISE NOTICE 'Query returned % records.', jsonb_array_length(out_records);\n RAISE LOG 'TOKEN: % %', token_item.id, token_item.collection;\n RAISE LOG 'RECORD_1: % %', out_records->0->>'id', out_records->0->>'collection';\n RAISE LOG 'RECORD-1: % %', out_records->-1->>'id', out_records->-1->>'collection';\n\n -- REMOVE records that were from our token\n IF out_records->0->>'id' = token_item.id AND out_records->0->>'collection' = token_item.collection THEN\n out_records := out_records - 0;\n ELSIF out_records->-1->>'id' = token_item.id AND out_records->-1->>'collection' = token_item.collection THEN\n out_records := out_records - -1;\n END IF;\n\n out_len := jsonb_array_length(out_records);\n\n IF out_len = _limit + 1 THEN\n IF token_prev THEN\n has_prev := TRUE;\n out_records := out_records - 0;\n ELSE\n has_next := TRUE;\n out_records := out_records - -1;\n END IF;\n END IF;\n\n IF has_next THEN\n next := concat(out_records->-1->>'collection', ':', out_records->-1->>'id');\n RAISE NOTICE 'HAS NEXT | %', next;\n END IF;\n\n IF has_prev THEN\n prev := concat(out_records->0->>'collection', ':', out_records->0->>'id');\n RAISE NOTICE 'HAS PREV | %', prev;\n END IF;\n\n RAISE NOTICE 'Time to get prev/next %', age_ms(timer);\n timer := clock_timestamp();\n\n IF context(_search->'conf') != 'off' THEN\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'matched', total_count,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\n ELSE\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\n END IF;\n\n collection := jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb),\n 'next', next,\n 'prev', prev,\n 'context', context\n );\n\n IF get_setting_bool('timing', _search->'conf') THEN\n collection = collection || jsonb_build_object('timing', age_ms(init_ts));\n END IF;\n\n RAISE NOTICE 'Time to build final json %', age_ms(timer);\n timer := clock_timestamp();\n\n RAISE NOTICE 'Total Time: %', age_ms(current_timestamp);\n RAISE NOTICE 'RETURNING % records. NEXT: %. PREV: %', collection->'context'->>'returned', collection->>'next', collection->>'prev';\n RETURN collection;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION search_cursor(_search jsonb = '{}'::jsonb) RETURNS refcursor AS $$\nDECLARE\n curs refcursor;\n searches searches%ROWTYPE;\n _where text;\n _orderby text;\n q text;\n\nBEGIN\n searches := search_query(_search);\n _where := searches._where;\n _orderby := searches.orderby;\n\n OPEN curs FOR\n WITH p AS (\n SELECT * FROM partition_queries(_where, _orderby) p\n )\n SELECT\n CASE WHEN EXISTS (SELECT 1 FROM p) THEN\n (SELECT format($q$\n SELECT * FROM (\n %s\n ) total\n $q$,\n string_agg(\n format($q$ SELECT * FROM ( %s ) AS sub $q$, p),\n '\n UNION ALL\n '\n )\n ))\n ELSE NULL\n END FROM p;\n RETURN curs;\nEND;\n$$ LANGUAGE PLPGSQL;\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION tileenvelope(zoom int, x int, y int) RETURNS geometry AS $$\nWITH t AS (\n SELECT\n 20037508.3427892 as merc_max,\n -20037508.3427892 as merc_min,\n (2 * 20037508.3427892) / (2 ^ zoom) as tile_size\n)\nSELECT st_makeenvelope(\n merc_min + (tile_size * x),\n merc_max - (tile_size * (y + 1)),\n merc_min + (tile_size * (x + 1)),\n merc_max - (tile_size * y),\n 3857\n) FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;DROP FUNCTION IF EXISTS mercgrid;\n\n\nCREATE OR REPLACE FUNCTION ftime() RETURNS interval as $$\nSELECT age(clock_timestamp(), transaction_timestamp());\n$$ LANGUAGE SQL;\nSET SEARCH_PATH to pgstac, public;\n\nDROP FUNCTION IF EXISTS geometrysearch;\nCREATE OR REPLACE FUNCTION geometrysearch(\n IN geom geometry,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE, -- Return as soon as the passed in geometry is full covered\n IN skipcovered boolean DEFAULT TRUE -- Skip any items that would show up completely under the previous items\n) RETURNS jsonb AS $$\nDECLARE\n search searches%ROWTYPE;\n curs refcursor;\n _where text;\n query text;\n iter_record items%ROWTYPE;\n out_records jsonb := '{}'::jsonb[];\n exit_flag boolean := FALSE;\n counter int := 1;\n scancounter int := 1;\n remaining_limit int := _scanlimit;\n tilearea float;\n unionedgeom geometry;\n clippedgeom geometry;\n unionedgeom_area float := 0;\n prev_area float := 0;\n excludes text[];\n includes text[];\n\nBEGIN\n DROP TABLE IF EXISTS pgstac_results;\n CREATE TEMP TABLE pgstac_results (content jsonb) ON COMMIT DROP;\n\n -- If skipcovered is true then you will always want to exit when the passed in geometry is full\n IF skipcovered THEN\n exitwhenfull := TRUE;\n END IF;\n\n SELECT * INTO search FROM searches WHERE hash=queryhash;\n\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Search with Query Hash % Not Found', queryhash;\n END IF;\n\n tilearea := st_area(geom);\n _where := format('%s AND st_intersects(geometry, %L::geometry)', search._where, geom);\n\n\n FOR query IN SELECT * FROM partition_queries(_where, search.orderby) LOOP\n query := format('%s LIMIT %L', query, remaining_limit);\n RAISE NOTICE '%', query;\n OPEN curs FOR EXECUTE query;\n LOOP\n FETCH curs INTO iter_record;\n EXIT WHEN NOT FOUND;\n IF exitwhenfull OR skipcovered THEN -- If we are not using exitwhenfull or skipcovered, we do not need to do expensive geometry operations\n clippedgeom := st_intersection(geom, iter_record.geometry);\n\n IF unionedgeom IS NULL THEN\n unionedgeom := clippedgeom;\n ELSE\n unionedgeom := st_union(unionedgeom, clippedgeom);\n END IF;\n\n unionedgeom_area := st_area(unionedgeom);\n\n IF skipcovered AND prev_area = unionedgeom_area THEN\n scancounter := scancounter + 1;\n CONTINUE;\n END IF;\n\n prev_area := unionedgeom_area;\n\n RAISE NOTICE '% % % %', unionedgeom_area/tilearea, counter, scancounter, ftime();\n END IF;\n RAISE NOTICE '% %', iter_record, content_hydrate(iter_record, fields);\n INSERT INTO pgstac_results (content) VALUES (content_hydrate(iter_record, fields));\n\n IF counter >= _limit\n OR scancounter > _scanlimit\n OR ftime() > _timelimit\n OR (exitwhenfull AND unionedgeom_area >= tilearea)\n THEN\n exit_flag := TRUE;\n EXIT;\n END IF;\n counter := counter + 1;\n scancounter := scancounter + 1;\n\n END LOOP;\n CLOSE curs;\n EXIT WHEN exit_flag;\n remaining_limit := _scanlimit - scancounter;\n END LOOP;\n\n SELECT jsonb_agg(content) INTO out_records FROM pgstac_results WHERE content IS NOT NULL;\n\n RETURN jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb)\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDROP FUNCTION IF EXISTS geojsonsearch;\nCREATE OR REPLACE FUNCTION geojsonsearch(\n IN geojson jsonb,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_geomfromgeojson(geojson),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS xyzsearch;\nCREATE OR REPLACE FUNCTION xyzsearch(\n IN _x int,\n IN _y int,\n IN _z int,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_transform(tileenvelope(_z, _x, _y), 4326),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE PROCEDURE analyze_items() AS $$\nDECLARE\n q text;\n timeout_ts timestamptz;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n SELECT format('ANALYZE (VERBOSE, SKIP_LOCKED) %I;', relname) INTO q\n FROM pg_stat_user_tables\n WHERE relname like '_item%' AND (n_mod_since_analyze>0 OR last_analyze IS NULL) LIMIT 1;\n IF NOT FOUND THEN\n EXIT;\n END IF;\n RAISE NOTICE '%', q;\n EXECUTE q;\n COMMIT;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE PROCEDURE validate_constraints() AS $$\nDECLARE\n q text;\nBEGIN\n FOR q IN\n SELECT\n FORMAT(\n 'ALTER TABLE %I.%I VALIDATE CONSTRAINT %I;',\n nsp.nspname,\n cls.relname,\n con.conname\n )\n\n FROM pg_constraint AS con\n JOIN pg_class AS cls\n ON con.conrelid = cls.oid\n JOIN pg_namespace AS nsp\n ON cls.relnamespace = nsp.oid\n WHERE convalidated = FALSE AND contype in ('c','f')\n AND nsp.nspname = 'pgstac'\n LOOP\n RAISE NOTICE '%', q;\n PERFORM run_or_queue(q);\n COMMIT;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION collection_extent(_collection text, runupdate boolean default false) RETURNS jsonb AS $$\nDECLARE\n geom_extent geometry;\n mind timestamptz;\n maxd timestamptz;\n extent jsonb;\nBEGIN\n IF runupdate THEN\n PERFORM update_partition_stats_q(partition)\n FROM partitions_view WHERE collection=_collection;\n END IF;\n SELECT\n min(lower(dtrange)),\n max(upper(edtrange)),\n st_extent(spatial)\n INTO\n mind,\n maxd,\n geom_extent\n FROM partitions_view\n WHERE collection=_collection;\n\n IF geom_extent IS NOT NULL AND mind IS NOT NULL AND maxd IS NOT NULL THEN\n extent := jsonb_build_object(\n 'extent', jsonb_build_object(\n 'spatial', jsonb_build_object(\n 'bbox', to_jsonb(array[array[st_xmin(geom_extent), st_ymin(geom_extent), st_xmax(geom_extent), st_ymax(geom_extent)]])\n ),\n 'temporal', jsonb_build_object(\n 'interval', to_jsonb(array[array[mind, maxd]])\n )\n )\n );\n RETURN extent;\n END IF;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('id', '{\"title\": \"Item ID\",\"description\": \"Item identifier\",\"$ref\": \"https://schemas.stacspec.org/v1.0.0/item-spec/json-schema/item.json#/definitions/core/allOf/2/properties/id\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('geometry', '{\"title\": \"Item Geometry\",\"description\": \"Item Geometry\",\"$ref\": \"https://geojson.org/schema/Feature.json\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('datetime','{\"description\": \"Datetime\",\"type\": \"string\",\"title\": \"Acquired\",\"format\": \"date-time\",\"pattern\": \"(\\\\+00:00|Z)$\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDELETE FROM queryables a USING queryables b\n WHERE a.name = b.name AND a.collection_ids IS NOT DISTINCT FROM b.collection_ids AND a.id > b.id;\n\n\nINSERT INTO pgstac_settings (name, value) VALUES\n ('context', 'off'),\n ('context_estimated_count', '100000'),\n ('context_estimated_cost', '100000'),\n ('context_stats_ttl', '1 day'),\n ('default_filter_lang', 'cql2-json'),\n ('additional_properties', 'true'),\n ('use_queue', 'false'),\n ('queue_timeout', '10 minutes'),\n ('update_collection_extent', 'false'),\n ('format_cache', 'false')\nON CONFLICT DO NOTHING\n;\n\nALTER FUNCTION to_text COST 5000;\nALTER FUNCTION to_float COST 5000;\nALTER FUNCTION to_int COST 5000;\nALTER FUNCTION to_tstz COST 5000;\nALTER FUNCTION to_text_array COST 5000;\n\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nGRANT ALL ON SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac to pgstac_admin;\n\n-- pgstac_read role limited to using function apis\nGRANT EXECUTE ON FUNCTION search TO pgstac_read;\nGRANT EXECUTE ON FUNCTION search_query TO pgstac_read;\nGRANT EXECUTE ON FUNCTION item_by_id TO pgstac_read;\nGRANT EXECUTE ON FUNCTION get_item TO pgstac_read;\nGRANT SELECT ON ALL TABLES IN SCHEMA pgstac TO pgstac_read;\n\n\nGRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_ingest;\nGRANT USAGE ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_ingest;\n\nSELECT update_partition_stats_q(partition) FROM partitions_view;\nSELECT set_version('0.8.1');\n" }, { "path": "src/pgstac/migrations/pgstac.0.8.2-0.8.3.sql", "content": "SET client_min_messages TO WARNING;\nSET SEARCH_PATH to pgstac, public;\nRESET ROLE;\nDO $$\nDECLARE\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='postgis') THEN\n CREATE EXTENSION IF NOT EXISTS postgis;\n END IF;\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='btree_gist') THEN\n CREATE EXTENSION IF NOT EXISTS btree_gist;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_admin;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_read;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_ingest;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n\nGRANT pgstac_admin TO current_user;\n\n-- Function to make sure pgstac_admin is the owner of items\nCREATE OR REPLACE FUNCTION pgstac_admin_owns() RETURNS VOID AS $$\nDECLARE\n f RECORD;\nBEGIN\n FOR f IN (\n SELECT\n concat(\n oid::regproc::text,\n '(',\n coalesce(pg_get_function_identity_arguments(oid),''),\n ')'\n ) AS name,\n CASE prokind WHEN 'f' THEN 'FUNCTION' WHEN 'p' THEN 'PROCEDURE' WHEN 'a' THEN 'AGGREGATE' END as typ\n FROM pg_proc\n WHERE\n pronamespace=to_regnamespace('pgstac')\n AND proowner != to_regrole('pgstac_admin')\n AND proname NOT LIKE 'pg_stat%'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n FOR f IN (\n SELECT\n oid::regclass::text as name,\n CASE relkind\n WHEN 'i' THEN 'INDEX'\n WHEN 'I' THEN 'INDEX'\n WHEN 'p' THEN 'TABLE'\n WHEN 'r' THEN 'TABLE'\n WHEN 'v' THEN 'VIEW'\n WHEN 'S' THEN 'SEQUENCE'\n ELSE NULL\n END as typ\n FROM pg_class\n WHERE relnamespace=to_regnamespace('pgstac') and relowner != to_regrole('pgstac_admin') AND relkind IN ('r','p','v','S') AND relname NOT LIKE 'pg_stat'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\nSELECT pgstac_admin_owns();\n\nCREATE SCHEMA IF NOT EXISTS pgstac AUTHORIZATION pgstac_admin;\n\nGRANT ALL ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_admin;\n\nALTER ROLE pgstac_admin SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_read SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_ingest SET SEARCH_PATH TO pgstac, public;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\n\nGRANT pgstac_read TO pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\n\nSET ROLE pgstac_admin;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\nRESET ROLE;\n\nSET ROLE pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\nRESET ROLE;\n\nSET SEARCH_PATH TO pgstac, public;\nSET ROLE pgstac_admin;\n\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS analyze_items;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS validate_constraints;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n-- Install these idempotently as migrations do not put them before trying to modify the collections table\n\n\nCREATE OR REPLACE FUNCTION collection_geom(content jsonb)\nRETURNS geometry AS $$\n WITH box AS (SELECT content->'extent'->'spatial'->'bbox'->0 as box)\n SELECT\n st_makeenvelope(\n (box->>0)::float,\n (box->>1)::float,\n (box->>2)::float,\n (box->>3)::float,\n 4326\n )\n FROM box;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION collection_datetime(content jsonb)\nRETURNS timestamptz AS $$\n SELECT\n CASE\n WHEN\n (content->'extent'->'temporal'->'interval'->0->>0) IS NULL\n THEN '-infinity'::timestamptz\n ELSE\n (content->'extent'->'temporal'->'interval'->0->>0)::timestamptz\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION collection_enddatetime(content jsonb)\nRETURNS timestamptz AS $$\n SELECT\n CASE\n WHEN\n (content->'extent'->'temporal'->'interval'->0->>1) IS NULL\n THEN 'infinity'::timestamptz\n ELSE\n (content->'extent'->'temporal'->'interval'->0->>1)::timestamptz\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n-- BEGIN migra calculated SQL\nset check_function_bodies = off;\n\nCREATE OR REPLACE FUNCTION pgstac.items_staging_triggerfunc()\n RETURNS trigger\n LANGUAGE plpgsql\nAS $function$\nDECLARE\n p record;\n _partitions text[];\n part text;\n ts timestamptz := clock_timestamp();\n nrows int;\nBEGIN\n RAISE NOTICE 'Creating Partitions. %', clock_timestamp() - ts;\n\n FOR part IN WITH t AS (\n SELECT\n n.content->>'collection' as collection,\n stac_daterange(n.content->'properties') as dtr,\n partition_trunc\n FROM newdata n JOIN collections ON (n.content->>'collection'=collections.id)\n ), p AS (\n SELECT\n collection,\n COALESCE(date_trunc(partition_trunc::text, lower(dtr)),'-infinity') as d,\n tstzrange(min(lower(dtr)),max(lower(dtr)),'[]') as dtrange,\n tstzrange(min(upper(dtr)),max(upper(dtr)),'[]') as edtrange\n FROM t\n GROUP BY 1,2\n ) SELECT check_partition(collection, dtrange, edtrange) FROM p LOOP\n RAISE NOTICE 'Partition %', part;\n END LOOP;\n\n RAISE NOTICE 'Creating temp table with data to be added. %', clock_timestamp() - ts;\n DROP TABLE IF EXISTS tmpdata;\n CREATE TEMP TABLE tmpdata ON COMMIT DROP AS\n SELECT\n (content_dehydrate(content)).*\n FROM newdata;\n GET DIAGNOSTICS nrows = ROW_COUNT;\n RAISE NOTICE 'Added % rows to tmpdata. %', nrows, clock_timestamp() - ts;\n\n RAISE NOTICE 'Doing the insert. %', clock_timestamp() - ts;\n IF TG_TABLE_NAME = 'items_staging' THEN\n INSERT INTO items\n SELECT * FROM tmpdata;\n GET DIAGNOSTICS nrows = ROW_COUNT;\n RAISE NOTICE 'Inserted % rows to items. %', nrows, clock_timestamp() - ts;\n ELSIF TG_TABLE_NAME = 'items_staging_ignore' THEN\n INSERT INTO items\n SELECT * FROM tmpdata\n ON CONFLICT DO NOTHING;\n GET DIAGNOSTICS nrows = ROW_COUNT;\n RAISE NOTICE 'Inserted % rows to items. %', nrows, clock_timestamp() - ts;\n ELSIF TG_TABLE_NAME = 'items_staging_upsert' THEN\n DELETE FROM items i USING tmpdata s\n WHERE\n i.id = s.id\n AND i.collection = s.collection\n AND i IS DISTINCT FROM s\n ;\n GET DIAGNOSTICS nrows = ROW_COUNT;\n RAISE NOTICE 'Deleted % rows from items. %', nrows, clock_timestamp() - ts;\n INSERT INTO items AS t\n SELECT * FROM tmpdata\n ON CONFLICT DO NOTHING;\n GET DIAGNOSTICS nrows = ROW_COUNT;\n RAISE NOTICE 'Inserted % rows to items. %', nrows, clock_timestamp() - ts;\n END IF;\n\n RAISE NOTICE 'Deleting data from staging table. %', clock_timestamp() - ts;\n DELETE FROM items_staging;\n RAISE NOTICE 'Done. %', clock_timestamp() - ts;\n\n RETURN NULL;\n\nEND;\n$function$\n;\n\n\n-- END migra calculated SQL\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('id', '{\"title\": \"Item ID\",\"description\": \"Item identifier\",\"$ref\": \"https://schemas.stacspec.org/v1.0.0/item-spec/json-schema/item.json#/definitions/core/allOf/2/properties/id\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('geometry', '{\"title\": \"Item Geometry\",\"description\": \"Item Geometry\",\"$ref\": \"https://geojson.org/schema/Feature.json\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('datetime','{\"description\": \"Datetime\",\"type\": \"string\",\"title\": \"Acquired\",\"format\": \"date-time\",\"pattern\": \"(\\\\+00:00|Z)$\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDELETE FROM queryables a USING queryables b\n WHERE a.name = b.name AND a.collection_ids IS NOT DISTINCT FROM b.collection_ids AND a.id > b.id;\n\n\nINSERT INTO pgstac_settings (name, value) VALUES\n ('context', 'off'),\n ('context_estimated_count', '100000'),\n ('context_estimated_cost', '100000'),\n ('context_stats_ttl', '1 day'),\n ('default_filter_lang', 'cql2-json'),\n ('additional_properties', 'true'),\n ('use_queue', 'false'),\n ('queue_timeout', '10 minutes'),\n ('update_collection_extent', 'false'),\n ('format_cache', 'false'),\n ('readonly', 'false')\nON CONFLICT DO NOTHING\n;\n\nALTER FUNCTION to_text COST 5000;\nALTER FUNCTION to_float COST 5000;\nALTER FUNCTION to_int COST 5000;\nALTER FUNCTION to_tstz COST 5000;\nALTER FUNCTION to_text_array COST 5000;\n\nALTER FUNCTION update_partition_stats SECURITY DEFINER;\nALTER FUNCTION partition_after_triggerfunc SECURITY DEFINER;\nALTER FUNCTION drop_table_constraints SECURITY DEFINER;\nALTER FUNCTION create_table_constraints SECURITY DEFINER;\nALTER FUNCTION check_partition SECURITY DEFINER;\nALTER FUNCTION repartition SECURITY DEFINER;\nALTER FUNCTION where_stats SECURITY DEFINER;\nALTER FUNCTION search_query SECURITY DEFINER;\nALTER FUNCTION format_item SECURITY DEFINER;\nALTER FUNCTION maintain_index SECURITY DEFINER;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nGRANT ALL ON SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac to pgstac_admin;\n\n-- pgstac_read role limited to using function apis\nGRANT EXECUTE ON FUNCTION search TO pgstac_read;\nGRANT EXECUTE ON FUNCTION search_query TO pgstac_read;\nGRANT EXECUTE ON FUNCTION item_by_id TO pgstac_read;\nGRANT EXECUTE ON FUNCTION get_item TO pgstac_read;\nGRANT SELECT ON ALL TABLES IN SCHEMA pgstac TO pgstac_read;\n\n\nGRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_ingest;\nGRANT USAGE ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_ingest;\n\nREVOKE ALL PRIVILEGES ON PROCEDURE run_queued_queries FROM public;\nGRANT ALL ON PROCEDURE run_queued_queries TO pgstac_admin;\n\nREVOKE ALL PRIVILEGES ON FUNCTION run_queued_queries_intransaction FROM public;\nGRANT ALL ON FUNCTION run_queued_queries_intransaction TO pgstac_admin;\n\nRESET ROLE;\n\nSET ROLE pgstac_ingest;\nSELECT update_partition_stats_q(partition) FROM partitions_view;\nSELECT set_version('0.8.3');\n" }, { "path": "src/pgstac/migrations/pgstac.0.8.2.sql", "content": "RESET ROLE;\nDO $$\nDECLARE\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='postgis') THEN\n CREATE EXTENSION IF NOT EXISTS postgis;\n END IF;\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='btree_gist') THEN\n CREATE EXTENSION IF NOT EXISTS btree_gist;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_admin;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_read;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_ingest;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n\nGRANT pgstac_admin TO current_user;\n\n-- Function to make sure pgstac_admin is the owner of items\nCREATE OR REPLACE FUNCTION pgstac_admin_owns() RETURNS VOID AS $$\nDECLARE\n f RECORD;\nBEGIN\n FOR f IN (\n SELECT\n concat(\n oid::regproc::text,\n '(',\n coalesce(pg_get_function_identity_arguments(oid),''),\n ')'\n ) AS name,\n CASE prokind WHEN 'f' THEN 'FUNCTION' WHEN 'p' THEN 'PROCEDURE' WHEN 'a' THEN 'AGGREGATE' END as typ\n FROM pg_proc\n WHERE\n pronamespace=to_regnamespace('pgstac')\n AND proowner != to_regrole('pgstac_admin')\n AND proname NOT LIKE 'pg_stat%'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n FOR f IN (\n SELECT\n oid::regclass::text as name,\n CASE relkind\n WHEN 'i' THEN 'INDEX'\n WHEN 'I' THEN 'INDEX'\n WHEN 'p' THEN 'TABLE'\n WHEN 'r' THEN 'TABLE'\n WHEN 'v' THEN 'VIEW'\n WHEN 'S' THEN 'SEQUENCE'\n ELSE NULL\n END as typ\n FROM pg_class\n WHERE relnamespace=to_regnamespace('pgstac') and relowner != to_regrole('pgstac_admin') AND relkind IN ('r','p','v','S') AND relname NOT LIKE 'pg_stat'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\nSELECT pgstac_admin_owns();\n\nCREATE SCHEMA IF NOT EXISTS pgstac AUTHORIZATION pgstac_admin;\n\nGRANT ALL ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_admin;\n\nALTER ROLE pgstac_admin SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_read SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_ingest SET SEARCH_PATH TO pgstac, public;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\n\nGRANT pgstac_read TO pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\n\nSET ROLE pgstac_admin;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\nRESET ROLE;\n\nSET ROLE pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\nRESET ROLE;\n\nSET SEARCH_PATH TO pgstac, public;\nSET ROLE pgstac_admin;\n\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS analyze_items;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS validate_constraints;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n-- Install these idempotently as migrations do not put them before trying to modify the collections table\n\n\nCREATE OR REPLACE FUNCTION collection_geom(content jsonb)\nRETURNS geometry AS $$\n WITH box AS (SELECT content->'extent'->'spatial'->'bbox'->0 as box)\n SELECT\n st_makeenvelope(\n (box->>0)::float,\n (box->>1)::float,\n (box->>2)::float,\n (box->>3)::float,\n 4326\n )\n FROM box;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION collection_datetime(content jsonb)\nRETURNS timestamptz AS $$\n SELECT\n CASE\n WHEN\n (content->'extent'->'temporal'->'interval'->0->>0) IS NULL\n THEN '-infinity'::timestamptz\n ELSE\n (content->'extent'->'temporal'->'interval'->0->>0)::timestamptz\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION collection_enddatetime(content jsonb)\nRETURNS timestamptz AS $$\n SELECT\n CASE\n WHEN\n (content->'extent'->'temporal'->'interval'->0->>1) IS NULL\n THEN 'infinity'::timestamptz\n ELSE\n (content->'extent'->'temporal'->'interval'->0->>1)::timestamptz\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE TABLE IF NOT EXISTS migrations (\n version text PRIMARY KEY,\n datetime timestamptz DEFAULT clock_timestamp() NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION get_version() RETURNS text AS $$\n SELECT version FROM pgstac.migrations ORDER BY datetime DESC, version DESC LIMIT 1;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION set_version(text) RETURNS text AS $$\n INSERT INTO pgstac.migrations (version) VALUES ($1)\n ON CONFLICT DO NOTHING\n RETURNING version;\n$$ LANGUAGE SQL;\n\n\nCREATE TABLE IF NOT EXISTS pgstac_settings (\n name text PRIMARY KEY,\n value text NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION table_empty(text) RETURNS boolean AS $$\nDECLARE\n retval boolean;\nBEGIN\n EXECUTE format($q$\n SELECT NOT EXISTS (SELECT 1 FROM %I LIMIT 1)\n $q$,\n $1\n ) INTO retval;\n RETURN retval;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION get_setting(IN _setting text, IN conf jsonb DEFAULT NULL) RETURNS text AS $$\nSELECT COALESCE(\n nullif(conf->>_setting, ''),\n nullif(current_setting(concat('pgstac.',_setting), TRUE),''),\n nullif((SELECT value FROM pgstac.pgstac_settings WHERE name=_setting),'')\n);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_setting_bool(IN _setting text, IN conf jsonb DEFAULT NULL) RETURNS boolean AS $$\nSELECT COALESCE(\n nullif(conf->>_setting, ''),\n nullif(current_setting(concat('pgstac.',_setting), TRUE),''),\n nullif((SELECT value FROM pgstac.pgstac_settings WHERE name=_setting),''),\n 'FALSE'\n)::boolean;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION base_url(conf jsonb DEFAULT NULL) RETURNS text AS $$\n SELECT COALESCE(pgstac.get_setting('base_url', conf), '.');\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION additional_properties() RETURNS boolean AS $$\n SELECT pgstac.get_setting_bool('additional_properties');\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION readonly(conf jsonb DEFAULT NULL) RETURNS boolean AS $$\n SELECT pgstac.get_setting_bool('readonly', conf);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context(conf jsonb DEFAULT NULL) RETURNS text AS $$\n SELECT pgstac.get_setting('context', conf);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context_estimated_count(conf jsonb DEFAULT NULL) RETURNS int AS $$\n SELECT pgstac.get_setting('context_estimated_count', conf)::int;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_estimated_cost();\nCREATE OR REPLACE FUNCTION context_estimated_cost(conf jsonb DEFAULT NULL) RETURNS float AS $$\n SELECT pgstac.get_setting('context_estimated_cost', conf)::float;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_stats_ttl();\nCREATE OR REPLACE FUNCTION context_stats_ttl(conf jsonb DEFAULT NULL) RETURNS interval AS $$\n SELECT pgstac.get_setting('context_stats_ttl', conf)::interval;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION t2s(text) RETURNS text AS $$\n SELECT extract(epoch FROM $1::interval)::text || ' s';\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE STRICT;\n\nCREATE OR REPLACE FUNCTION age_ms(a timestamptz, b timestamptz DEFAULT clock_timestamp()) RETURNS float AS $$\n SELECT abs(extract(epoch from age(a,b)) * 1000);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION queue_timeout() RETURNS interval AS $$\n SELECT t2s(coalesce(\n get_setting('queue_timeout'),\n '1h'\n ))::interval;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION notice(VARIADIC text[]) RETURNS boolean AS $$\nDECLARE\ndebug boolean := current_setting('pgstac.debug', true);\nBEGIN\n IF debug THEN\n RAISE NOTICE 'NOTICE FROM FUNC: % >>>>> %', concat_ws(' | ', $1), clock_timestamp();\n RETURN TRUE;\n END IF;\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION empty_arr(ANYARRAY) RETURNS BOOLEAN AS $$\nSELECT CASE\n WHEN $1 IS NULL THEN TRUE\n WHEN cardinality($1)<1 THEN TRUE\nELSE FALSE\nEND;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_intersection(_a ANYARRAY, _b ANYARRAY) RETURNS ANYARRAY AS $$\n SELECT ARRAY ( SELECT unnest(_a) INTERSECT SELECT UNNEST(_b) );\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION array_map_ident(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_ident(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_map_literal(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_literal(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION array_reverse(anyarray) RETURNS anyarray AS $$\nSELECT ARRAY(\n SELECT $1[i]\n FROM generate_subscripts($1,1) AS s(i)\n ORDER BY i DESC\n);\n$$ LANGUAGE SQL STRICT IMMUTABLE;\n\nDROP TABLE IF EXISTS query_queue;\nCREATE TABLE query_queue (\n query text PRIMARY KEY,\n added timestamptz DEFAULT now()\n);\n\nDROP TABLE IF EXISTS query_queue_history;\nCREATE TABLE query_queue_history(\n query text,\n added timestamptz NOT NULL,\n finished timestamptz NOT NULL DEFAULT now(),\n error text\n);\n\nCREATE OR REPLACE PROCEDURE run_queued_queries() AS $$\nDECLARE\n qitem query_queue%ROWTYPE;\n timeout_ts timestamptz;\n error text;\n cnt int := 0;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n DELETE FROM query_queue WHERE query = (SELECT query FROM query_queue ORDER BY added DESC LIMIT 1 FOR UPDATE SKIP LOCKED) RETURNING * INTO qitem;\n IF NOT FOUND THEN\n EXIT;\n END IF;\n cnt := cnt + 1;\n BEGIN\n RAISE NOTICE 'RUNNING QUERY: %', qitem.query;\n EXECUTE qitem.query;\n EXCEPTION WHEN others THEN\n error := format('%s | %s', SQLERRM, SQLSTATE);\n END;\n INSERT INTO query_queue_history (query, added, finished, error)\n VALUES (qitem.query, qitem.added, clock_timestamp(), error);\n COMMIT;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION run_queued_queries_intransaction() RETURNS int AS $$\nDECLARE\n qitem query_queue%ROWTYPE;\n timeout_ts timestamptz;\n error text;\n cnt int := 0;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n DELETE FROM query_queue WHERE query = (SELECT query FROM query_queue ORDER BY added DESC LIMIT 1 FOR UPDATE SKIP LOCKED) RETURNING * INTO qitem;\n IF NOT FOUND THEN\n RETURN cnt;\n END IF;\n cnt := cnt + 1;\n BEGIN\n qitem.query := regexp_replace(qitem.query, 'CONCURRENTLY', '');\n RAISE NOTICE 'RUNNING QUERY: %', qitem.query;\n\n EXECUTE qitem.query;\n EXCEPTION WHEN others THEN\n error := format('%s | %s', SQLERRM, SQLSTATE);\n RAISE WARNING '%', error;\n END;\n INSERT INTO query_queue_history (query, added, finished, error)\n VALUES (qitem.query, qitem.added, clock_timestamp(), error);\n END LOOP;\n RETURN cnt;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION run_or_queue(query text) RETURNS VOID AS $$\nDECLARE\n use_queue text := COALESCE(get_setting('use_queue'), 'FALSE')::boolean;\nBEGIN\n IF get_setting_bool('debug') THEN\n RAISE NOTICE '%', query;\n END IF;\n IF use_queue THEN\n INSERT INTO query_queue (query) VALUES (query) ON CONFLICT DO NOTHING;\n ELSE\n EXECUTE query;\n END IF;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n\nDROP FUNCTION IF EXISTS check_pgstac_settings;\nCREATE OR REPLACE FUNCTION check_pgstac_settings(_sysmem text DEFAULT NULL) RETURNS VOID AS $$\nDECLARE\n settingval text;\n sysmem bigint := pg_size_bytes(_sysmem);\n effective_cache_size bigint := pg_size_bytes(current_setting('effective_cache_size', TRUE));\n shared_buffers bigint := pg_size_bytes(current_setting('shared_buffers', TRUE));\n work_mem bigint := pg_size_bytes(current_setting('work_mem', TRUE));\n max_connections int := current_setting('max_connections', TRUE);\n maintenance_work_mem bigint := pg_size_bytes(current_setting('maintenance_work_mem', TRUE));\n seq_page_cost float := current_setting('seq_page_cost', TRUE);\n random_page_cost float := current_setting('random_page_cost', TRUE);\n temp_buffers bigint := pg_size_bytes(current_setting('temp_buffers', TRUE));\n r record;\nBEGIN\n IF _sysmem IS NULL THEN\n RAISE NOTICE 'Call function with the size of your system memory `SELECT check_pgstac_settings(''4GB'')` to get pg system setting recommendations.';\n ELSE\n IF effective_cache_size < (sysmem * 0.5) THEN\n RAISE WARNING 'effective_cache_size of % is set low for a system with %. Recomended value between % and %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.5), pg_size_pretty(sysmem * 0.75);\n ELSIF effective_cache_size > (sysmem * 0.75) THEN\n RAISE WARNING 'effective_cache_size of % is set high for a system with %. Recomended value between % and %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.5), pg_size_pretty(sysmem * 0.75);\n ELSE\n RAISE NOTICE 'effective_cache_size of % is set appropriately for a system with %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem);\n END IF;\n\n IF shared_buffers < (sysmem * 0.2) THEN\n RAISE WARNING 'shared_buffers of % is set low for a system with %. Recomended value between % and %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.2), pg_size_pretty(sysmem * 0.3);\n ELSIF shared_buffers > (sysmem * 0.3) THEN\n RAISE WARNING 'shared_buffers of % is set high for a system with %. Recomended value between % and %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.2), pg_size_pretty(sysmem * 0.3);\n ELSE\n RAISE NOTICE 'shared_buffers of % is set appropriately for a system with %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem);\n END IF;\n shared_buffers = sysmem * 0.3;\n IF maintenance_work_mem < (sysmem * 0.2) THEN\n RAISE WARNING 'maintenance_work_mem of % is set low for shared_buffers of %. Recomended value between % and %', pg_size_pretty(maintenance_work_mem), pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers * 0.2), pg_size_pretty(shared_buffers * 0.3);\n ELSIF maintenance_work_mem > (shared_buffers * 0.3) THEN\n RAISE WARNING 'maintenance_work_mem of % is set high for shared_buffers of %. Recomended value between % and %', pg_size_pretty(maintenance_work_mem), pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers * 0.2), pg_size_pretty(shared_buffers * 0.3);\n ELSE\n RAISE NOTICE 'maintenance_work_mem of % is set appropriately for shared_buffers of %', pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers);\n END IF;\n\n IF work_mem * max_connections > shared_buffers THEN\n RAISE WARNING 'work_mem setting of % is set high for % max_connections please reduce work_mem to % or decrease max_connections to %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers/max_connections), floor(shared_buffers/work_mem);\n ELSIF work_mem * max_connections < (shared_buffers * 0.75) THEN\n RAISE WARNING 'work_mem setting of % is set low for % max_connections you may consider raising work_mem to % or increasing max_connections to %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers/max_connections), floor(shared_buffers/work_mem);\n ELSE\n RAISE NOTICE 'work_mem setting of % and max_connections of % are adequate for shared_buffers of %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers);\n END IF;\n\n IF random_page_cost / seq_page_cost != 1.1 THEN\n RAISE WARNING 'random_page_cost (%) /seq_page_cost (%) should be set to 1.1 for SSD. Change random_page_cost to %', random_page_cost, seq_page_cost, 1.1 * seq_page_cost;\n ELSE\n RAISE NOTICE 'random_page_cost and seq_page_cost set appropriately for SSD';\n END IF;\n\n IF temp_buffers < greatest(pg_size_bytes('128MB'),(maintenance_work_mem / 2)) THEN\n RAISE WARNING 'pgstac makes heavy use of temp tables, consider raising temp_buffers from % to %', pg_size_pretty(temp_buffers), greatest('128MB', pg_size_pretty((shared_buffers / 16)));\n END IF;\n END IF;\n\n RAISE NOTICE 'VALUES FOR PGSTAC VARIABLES';\n RAISE NOTICE 'These can be set either as GUC system variables or by setting in the pgstac_settings table.';\n\n FOR r IN SELECT name, get_setting(name) as setting, CASE WHEN current_setting(concat('pgstac.',name), TRUE) IS NOT NULL THEN concat('pgstac.',name, ' GUC') WHEN value IS NOT NULL THEN 'pgstac_settings table' ELSE 'Not Set' END as loc FROM pgstac_settings LOOP\n RAISE NOTICE '% is set to % from the %', r.name, r.setting, r.loc;\n END LOOP;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pg_cron';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider intalling pg_cron which can be used to automate tasks';\n ELSE\n RAISE NOTICE 'pg_cron % is installed', settingval;\n END IF;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pgstattuple';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider installing the pgstattuple extension which can be used to help maintain tables and indexes.';\n ELSE\n RAISE NOTICE 'pgstattuple % is installed', settingval;\n END IF;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pg_stat_statements';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider installing the pg_stat_statements extension which is very helpful for tracking the types of queries on the system';\n ELSE\n RAISE NOTICE 'pg_stat_statements % is installed', settingval;\n IF current_setting('pg_stat_statements.track_statements', TRUE) IS DISTINCT FROM 'all' THEN\n RAISE WARNING 'SET pg_stat_statements.track_statements TO ''all''; --In order to track statements within functions.';\n END IF;\n END IF;\n\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac, public SET CLIENT_MIN_MESSAGES TO NOTICE;\nCREATE OR REPLACE FUNCTION to_int(jsonb) RETURNS int AS $$\n SELECT floor(($1->>0)::float)::int;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION to_float(jsonb) RETURNS float AS $$\n SELECT ($1->>0)::float;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION to_tstz(jsonb) RETURNS timestamptz AS $$\n SELECT ($1->>0)::timestamptz;\n$$ LANGUAGE SQL IMMUTABLE STRICT SET TIME ZONE 'UTC' COST 5000 PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION to_text(jsonb) RETURNS text AS $$\n SELECT CASE WHEN jsonb_typeof($1) IN ('array','object') THEN $1::text ELSE $1->>0 END;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION to_text_array(jsonb) RETURNS text[] AS $$\n SELECT\n CASE jsonb_typeof($1)\n WHEN 'array' THEN ARRAY(SELECT jsonb_array_elements_text($1))\n ELSE ARRAY[$1->>0]\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION bbox_geom(_bbox jsonb) RETURNS geometry AS $$\nSELECT CASE jsonb_array_length(_bbox)\n WHEN 4 THEN\n ST_SetSRID(ST_MakeEnvelope(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float,\n (_bbox->>3)::float\n ),4326)\n WHEN 6 THEN\n ST_SetSRID(ST_3DMakeBox(\n ST_MakePoint(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float\n ),\n ST_MakePoint(\n (_bbox->>3)::float,\n (_bbox->>4)::float,\n (_bbox->>5)::float\n )\n ),4326)\n ELSE null END;\n;\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION geom_bbox(_geom geometry) RETURNS jsonb AS $$\n SELECT jsonb_build_array(\n st_xmin(_geom),\n st_ymin(_geom),\n st_xmax(_geom),\n st_ymax(_geom)\n );\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION flip_jsonb_array(j jsonb) RETURNS jsonb AS $$\n SELECT jsonb_agg(value) FROM (SELECT value FROM jsonb_array_elements(j) WITH ORDINALITY ORDER BY ordinality DESC) as t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION explode_dotpaths(j jsonb) RETURNS SETOF text[] AS $$\n SELECT string_to_array(p, '.') as e FROM jsonb_array_elements_text(j) p;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION explode_dotpaths_recurse(IN j jsonb) RETURNS SETOF text[] AS $$\n WITH RECURSIVE t AS (\n SELECT e FROM explode_dotpaths(j) e\n UNION ALL\n SELECT e[1:cardinality(e)-1]\n FROM t\n WHERE cardinality(e)>1\n ) SELECT e FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION jsonb_set_nested(j jsonb, path text[], val jsonb) RETURNS jsonb AS $$\nDECLARE\nBEGIN\n IF cardinality(path) > 1 THEN\n FOR i IN 1..(cardinality(path)-1) LOOP\n IF j #> path[:i] IS NULL THEN\n j := jsonb_set_lax(j, path[:i], '{}', TRUE);\n END IF;\n END LOOP;\n END IF;\n RETURN jsonb_set_lax(j, path, val, true);\n\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\n\n\nCREATE OR REPLACE FUNCTION jsonb_include(j jsonb, f jsonb) RETURNS jsonb AS $$\nDECLARE\n includes jsonb := f-> 'include';\n outj jsonb := '{}'::jsonb;\n path text[];\nBEGIN\n IF\n includes IS NULL\n OR jsonb_array_length(includes) = 0\n THEN\n RETURN j;\n ELSE\n includes := includes || '[\"id\",\"collection\"]'::jsonb;\n FOR path IN SELECT explode_dotpaths(includes) LOOP\n outj := jsonb_set_nested(outj, path, j #> path);\n END LOOP;\n END IF;\n RETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_exclude(j jsonb, f jsonb) RETURNS jsonb AS $$\nDECLARE\n excludes jsonb := f-> 'exclude';\n outj jsonb := j;\n path text[];\nBEGIN\n IF\n excludes IS NULL\n OR jsonb_array_length(excludes) = 0\n THEN\n RETURN j;\n ELSE\n FOR path IN SELECT explode_dotpaths(excludes) LOOP\n outj := outj #- path;\n END LOOP;\n END IF;\n RETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_fields(j jsonb, f jsonb DEFAULT '{\"fields\":[]}') RETURNS jsonb AS $$\n SELECT jsonb_exclude(jsonb_include(j, f), f);\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION merge_jsonb(_a jsonb, _b jsonb) RETURNS jsonb AS $$\n SELECT\n CASE\n WHEN _a = '\"𒍟※\"'::jsonb THEN NULL\n WHEN _a IS NULL OR jsonb_typeof(_a) = 'null' THEN _b\n WHEN jsonb_typeof(_a) = 'object' AND jsonb_typeof(_b) = 'object' THEN\n (\n SELECT\n jsonb_strip_nulls(\n jsonb_object_agg(\n key,\n merge_jsonb(a.value, b.value)\n )\n )\n FROM\n jsonb_each(coalesce(_a,'{}'::jsonb)) as a\n FULL JOIN\n jsonb_each(coalesce(_b,'{}'::jsonb)) as b\n USING (key)\n )\n WHEN\n jsonb_typeof(_a) = 'array'\n AND jsonb_typeof(_b) = 'array'\n AND jsonb_array_length(_a) = jsonb_array_length(_b)\n THEN\n (\n SELECT jsonb_agg(m) FROM\n ( SELECT\n merge_jsonb(\n jsonb_array_elements(_a),\n jsonb_array_elements(_b)\n ) as m\n ) as l\n )\n ELSE _a\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION strip_jsonb(_a jsonb, _b jsonb) RETURNS jsonb AS $$\n SELECT\n CASE\n\n WHEN (_a IS NULL OR jsonb_typeof(_a) = 'null') AND _b IS NOT NULL AND jsonb_typeof(_b) != 'null' THEN '\"𒍟※\"'::jsonb\n WHEN _b IS NULL OR jsonb_typeof(_a) = 'null' THEN _a\n WHEN _a = _b AND jsonb_typeof(_a) = 'object' THEN '{}'::jsonb\n WHEN _a = _b THEN NULL\n WHEN jsonb_typeof(_a) = 'object' AND jsonb_typeof(_b) = 'object' THEN\n (\n SELECT\n jsonb_strip_nulls(\n jsonb_object_agg(\n key,\n strip_jsonb(a.value, b.value)\n )\n )\n FROM\n jsonb_each(_a) as a\n FULL JOIN\n jsonb_each(_b) as b\n USING (key)\n )\n WHEN\n jsonb_typeof(_a) = 'array'\n AND jsonb_typeof(_b) = 'array'\n AND jsonb_array_length(_a) = jsonb_array_length(_b)\n THEN\n (\n SELECT jsonb_agg(m) FROM\n ( SELECT\n strip_jsonb(\n jsonb_array_elements(_a),\n jsonb_array_elements(_b)\n ) as m\n ) as l\n )\n ELSE _a\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION nullif_jsonbnullempty(j jsonb) RETURNS jsonb AS $$\n SELECT nullif(nullif(nullif(j,'null'::jsonb),'{}'::jsonb),'[]'::jsonb);\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION jsonb_array_unique(j jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(jsonb_agg(DISTINCT a)) v FROM jsonb_array_elements(j) a;\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_concat_ignorenull(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT coalesce(a,'[]'::jsonb) || coalesce(b,'[]'::jsonb);\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_least(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(least(nullif_jsonbnullempty(a), nullif_jsonbnullempty(b)));\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_greatest(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(greatest(a, b));\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION first_notnull_sfunc(anyelement, anyelement) RETURNS anyelement AS $$\n SELECT COALESCE($1,$2);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE AGGREGATE first_notnull(anyelement)(\n SFUNC = first_notnull_sfunc,\n STYPE = anyelement\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_array_unique_merge(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_concat_ignorenull,\n FINALFUNC = jsonb_array_unique\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_min(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_least\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_max(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_greatest\n);\n/* looks for a geometry in a stac item first from geometry and falling back to bbox */\nCREATE OR REPLACE FUNCTION stac_geom(value jsonb) RETURNS geometry AS $$\nSELECT\n CASE\n WHEN value ? 'intersects' THEN\n ST_GeomFromGeoJSON(value->>'intersects')\n WHEN value ? 'geometry' THEN\n ST_GeomFromGeoJSON(value->>'geometry')\n WHEN value ? 'bbox' THEN\n pgstac.bbox_geom(value->'bbox')\n ELSE NULL\n END as geometry\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION stac_daterange(\n value jsonb\n) RETURNS tstzrange AS $$\nDECLARE\n props jsonb := value;\n dt timestamptz;\n edt timestamptz;\nBEGIN\n IF props ? 'properties' THEN\n props := props->'properties';\n END IF;\n IF\n props ? 'start_datetime'\n AND props->>'start_datetime' IS NOT NULL\n AND props ? 'end_datetime'\n AND props->>'end_datetime' IS NOT NULL\n THEN\n dt := props->>'start_datetime';\n edt := props->>'end_datetime';\n IF dt > edt THEN\n RAISE EXCEPTION 'start_datetime must be < end_datetime';\n END IF;\n ELSE\n dt := props->>'datetime';\n edt := props->>'datetime';\n END IF;\n IF dt is NULL OR edt IS NULL THEN\n RAISE NOTICE 'DT: %, EDT: %', dt, edt;\n RAISE EXCEPTION 'Either datetime (%) or both start_datetime (%) and end_datetime (%) must be set.', props->>'datetime',props->>'start_datetime',props->>'end_datetime';\n END IF;\n RETURN tstzrange(dt, edt, '[]');\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_datetime(value jsonb) RETURNS timestamptz AS $$\n SELECT lower(stac_daterange(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_end_datetime(value jsonb) RETURNS timestamptz AS $$\n SELECT upper(stac_daterange(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE TABLE IF NOT EXISTS stac_extensions(\n url text PRIMARY KEY,\n content jsonb\n);\nCREATE OR REPLACE FUNCTION collection_base_item(content jsonb) RETURNS jsonb AS $$\n SELECT jsonb_build_object(\n 'type', 'Feature',\n 'stac_version', content->'stac_version',\n 'assets', content->'item_assets',\n 'collection', content->'id'\n );\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE TABLE IF NOT EXISTS collections (\n key bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,\n id text GENERATED ALWAYS AS (content->>'id') STORED UNIQUE NOT NULL,\n content JSONB NOT NULL,\n base_item jsonb GENERATED ALWAYS AS (pgstac.collection_base_item(content)) STORED,\n geometry geometry GENERATED ALWAYS AS (pgstac.collection_geom(content)) STORED,\n datetime timestamptz GENERATED ALWAYS AS (pgstac.collection_datetime(content)) STORED,\n end_datetime timestamptz GENERATED ALWAYS AS (pgstac.collection_enddatetime(content)) STORED,\n private jsonb,\n partition_trunc text CHECK (partition_trunc IN ('year', 'month'))\n);\n\nCREATE OR REPLACE FUNCTION collection_base_item(cid text) RETURNS jsonb AS $$\n SELECT pgstac.collection_base_item(content) FROM pgstac.collections WHERE id = cid LIMIT 1;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION create_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION update_collection(data jsonb) RETURNS VOID AS $$\nDECLARE\n out collections%ROWTYPE;\nBEGIN\n UPDATE collections SET content=data WHERE id = data->>'id' RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ON CONFLICT (id) DO\n UPDATE\n SET content=EXCLUDED.content\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION delete_collection(_id text) RETURNS VOID AS $$\nDECLARE\n out collections%ROWTYPE;\nBEGIN\n DELETE FROM collections WHERE id = _id RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION get_collection(id text) RETURNS jsonb AS $$\n SELECT content FROM collections\n WHERE id=$1\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION all_collections() RETURNS jsonb AS $$\n SELECT coalesce(jsonb_agg(content), '[]'::jsonb) FROM collections;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION collection_delete_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n collection_base_partition text := concat('_items_', OLD.key);\nBEGIN\n EXECUTE format($q$\n DELETE FROM partition_stats WHERE partition IN (\n SELECT partition FROM partition_sys_meta\n WHERE collection=%L\n );\n DROP TABLE IF EXISTS %I CASCADE;\n $q$,\n OLD.id,\n collection_base_partition\n );\n RETURN OLD;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER collection_delete_trigger BEFORE DELETE ON collections\nFOR EACH ROW EXECUTE FUNCTION collection_delete_trigger_func();\nCREATE OR REPLACE FUNCTION queryable_signature(n text, c text[]) RETURNS text AS $$\n SELECT concat(n, c);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TABLE queryables (\n id bigint GENERATED ALWAYS AS identity PRIMARY KEY,\n name text NOT NULL,\n collection_ids text[], -- used to determine what partitions to create indexes on\n definition jsonb,\n property_path text,\n property_wrapper text,\n property_index_type text\n);\nCREATE INDEX queryables_name_idx ON queryables (name);\nCREATE INDEX queryables_collection_idx ON queryables USING GIN (collection_ids);\nCREATE INDEX queryables_property_wrapper_idx ON queryables (property_wrapper);\n\nCREATE OR REPLACE FUNCTION pgstac.queryables_constraint_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n allcollections text[];\nBEGIN\n RAISE NOTICE 'Making sure that name/collection is unique for queryables %', NEW;\n IF NEW.collection_ids IS NOT NULL THEN\n IF EXISTS (\n SELECT 1\n FROM unnest(NEW.collection_ids) c\n LEFT JOIN\n collections\n ON (collections.id = c)\n WHERE collections.id IS NULL\n ) THEN\n RAISE foreign_key_violation USING MESSAGE = format(\n 'One or more collections in %s do not exist.', NEW.collection_ids\n );\n RETURN NULL;\n END IF;\n END IF;\n IF TG_OP = 'INSERT' THEN\n IF EXISTS (\n SELECT 1 FROM queryables q\n WHERE\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n )\n ) THEN\n RAISE unique_violation USING MESSAGE = format(\n 'There is already a queryable for %s for a collection in %s: %s',\n NEW.name,\n NEW.collection_ids,\n\t\t\t\t(SELECT json_agg(row_to_json(q)) FROM queryables q WHERE\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n ))\n );\n RETURN NULL;\n END IF;\n END IF;\n IF TG_OP = 'UPDATE' THEN\n IF EXISTS (\n SELECT 1 FROM queryables q\n WHERE\n q.id != NEW.id\n AND\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n )\n ) THEN\n RAISE unique_violation\n USING MESSAGE = format(\n 'There is already a queryable for %s for a collection in %s',\n NEW.name,\n NEW.collection_ids\n );\n RETURN NULL;\n END IF;\n END IF;\n\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER queryables_constraint_insert_trigger\nBEFORE INSERT ON queryables\nFOR EACH ROW EXECUTE PROCEDURE queryables_constraint_triggerfunc();\n\nCREATE TRIGGER queryables_constraint_update_trigger\nBEFORE UPDATE ON queryables\nFOR EACH ROW\nWHEN (NEW.name = OLD.name AND NEW.collection_ids IS DISTINCT FROM OLD.collection_ids)\nEXECUTE PROCEDURE queryables_constraint_triggerfunc();\n\n\nCREATE OR REPLACE FUNCTION array_to_path(arr text[]) RETURNS text AS $$\n SELECT string_agg(\n quote_literal(v),\n '->'\n ) FROM unnest(arr) v;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\n\n\nCREATE OR REPLACE FUNCTION queryable(\n IN dotpath text,\n OUT path text,\n OUT expression text,\n OUT wrapper text,\n OUT nulled_wrapper text\n) AS $$\nDECLARE\n q RECORD;\n path_elements text[];\nBEGIN\n IF dotpath IN ('id', 'geometry', 'datetime', 'end_datetime', 'collection') THEN\n path := dotpath;\n expression := dotpath;\n wrapper := NULL;\n RETURN;\n END IF;\n SELECT * INTO q FROM queryables\n WHERE\n name=dotpath\n OR name = 'properties.' || dotpath\n OR name = replace(dotpath, 'properties.', '')\n ;\n IF q.property_wrapper IS NULL THEN\n IF q.definition->>'type' = 'number' THEN\n wrapper := 'to_float';\n nulled_wrapper := wrapper;\n ELSIF q.definition->>'format' = 'date-time' THEN\n wrapper := 'to_tstz';\n nulled_wrapper := wrapper;\n ELSE\n nulled_wrapper := NULL;\n wrapper := 'to_text';\n END IF;\n ELSE\n wrapper := q.property_wrapper;\n nulled_wrapper := wrapper;\n END IF;\n IF q.property_path IS NOT NULL THEN\n path := q.property_path;\n ELSE\n path_elements := string_to_array(dotpath, '.');\n IF path_elements[1] IN ('links', 'assets', 'stac_version', 'stac_extensions') THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSIF path_elements[1] = 'properties' THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSE\n path := format($F$content->'properties'->%s$F$, array_to_path(path_elements));\n END IF;\n END IF;\n expression := format('%I(%s)', wrapper, path);\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\nCREATE OR REPLACE FUNCTION unnest_collection(collection_ids text[] DEFAULT NULL) RETURNS SETOF text AS $$\n DECLARE\n BEGIN\n IF collection_ids IS NULL THEN\n RETURN QUERY SELECT id FROM collections;\n END IF;\n RETURN QUERY SELECT unnest(collection_ids);\n END;\n$$ LANGUAGE PLPGSQL STABLE;\n\nCREATE OR REPLACE FUNCTION normalize_indexdef(def text) RETURNS text AS $$\nDECLARE\nBEGIN\n def := btrim(def, ' \\n\\t');\n\tdef := regexp_replace(def, '^CREATE (UNIQUE )?INDEX ([^ ]* )?ON (ONLY )?([^ ]* )?', '', 'i');\n RETURN def;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION indexdef(q queryables) RETURNS text AS $$\n DECLARE\n out text;\n BEGIN\n IF q.name = 'id' THEN\n out := 'CREATE UNIQUE INDEX ON %I USING btree (id)';\n ELSIF q.name = 'datetime' THEN\n out := 'CREATE INDEX ON %I USING btree (datetime DESC, end_datetime)';\n ELSIF q.name = 'geometry' THEN\n out := 'CREATE INDEX ON %I USING gist (geometry)';\n ELSE\n out := format($q$CREATE INDEX ON %%I USING %s (%s(((content -> 'properties'::text) -> %L::text)))$q$,\n lower(COALESCE(q.property_index_type, 'BTREE')),\n lower(COALESCE(q.property_wrapper, 'to_text')),\n q.name\n );\n END IF;\n RETURN btrim(out, ' \\n\\t');\n END;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nDROP VIEW IF EXISTS pgstac_indexes;\nCREATE VIEW pgstac_indexes AS\nSELECT\n i.schemaname,\n i.tablename,\n i.indexname,\n regexp_replace(btrim(replace(replace(indexdef, i.indexname, ''),'pgstac.',''),' \\t\\n'), '[ ]+', ' ', 'g') as idx,\n COALESCE(\n (regexp_match(indexdef, '\\(([a-zA-Z]+)\\)'))[1],\n (regexp_match(indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_-]+)''::text'))[1],\n CASE WHEN indexdef ~* '\\(datetime desc, end_datetime\\)' THEN 'datetime' ELSE NULL END\n ) AS field,\n pg_table_size(i.indexname::text) as index_size,\n pg_size_pretty(pg_table_size(i.indexname::text)) as index_size_pretty\nFROM\n pg_indexes i\nWHERE i.schemaname='pgstac' and i.tablename ~ '_items_' AND indexdef !~* ' only ';\n\nDROP VIEW IF EXISTS pgstac_index_stats;\nCREATE VIEW pgstac_indexes_stats AS\nSELECT\n i.schemaname,\n i.tablename,\n i.indexname,\n indexdef,\n COALESCE(\n (regexp_match(indexdef, '\\(([a-zA-Z]+)\\)'))[1],\n (regexp_match(indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_]+)''::text'))[1],\n CASE WHEN indexdef ~* '\\(datetime desc, end_datetime\\)' THEN 'datetime_end_datetime' ELSE NULL END\n ) AS field,\n pg_table_size(i.indexname::text) as index_size,\n pg_size_pretty(pg_table_size(i.indexname::text)) as index_size_pretty,\n n_distinct,\n most_common_vals::text::text[],\n most_common_freqs::text::text[],\n histogram_bounds::text::text[],\n correlation\nFROM\n pg_indexes i\n LEFT JOIN pg_stats s ON (s.tablename = i.indexname)\nWHERE i.schemaname='pgstac' and i.tablename ~ '_items_';\n\nCREATE OR REPLACE FUNCTION queryable_indexes(\n IN treeroot text DEFAULT 'items',\n IN changes boolean DEFAULT FALSE,\n OUT collection text,\n OUT partition text,\n OUT field text,\n OUT indexname text,\n OUT existing_idx text,\n OUT queryable_idx text\n) RETURNS SETOF RECORD AS $$\nWITH p AS (\n SELECT\n relid::text as partition,\n replace(replace(\n CASE\n WHEN parentrelid::regclass::text='items' THEN pg_get_expr(c.relpartbound, c.oid)\n ELSE pg_get_expr(parent.relpartbound, parent.oid)\n END,\n 'FOR VALUES IN (''',''), ''')',\n ''\n ) AS collection\n FROM pg_partition_tree(treeroot)\n JOIN pg_class c ON (relid::regclass = c.oid)\n JOIN pg_class parent ON (parentrelid::regclass = parent.oid AND isleaf)\n ), i AS (\n SELECT\n partition,\n indexname,\n regexp_replace(btrim(replace(replace(indexdef, indexname, ''),'pgstac.',''),' \\t\\n'), '[ ]+', ' ', 'g') as iidx,\n COALESCE(\n (regexp_match(indexdef, '\\(([a-zA-Z]+)\\)'))[1],\n (regexp_match(indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_-]+)''::text'))[1],\n CASE WHEN indexdef ~* '\\(datetime desc, end_datetime\\)' THEN 'datetime' ELSE NULL END\n ) AS field\n FROM\n pg_indexes\n JOIN p ON (tablename=partition)\n ), q AS (\n SELECT\n name AS field,\n collection,\n partition,\n format(indexdef(queryables), partition) as qidx\n FROM queryables, unnest_collection(queryables.collection_ids) collection\n JOIN p USING (collection)\n WHERE property_index_type IS NOT NULL OR name IN ('datetime','geometry','id')\n )\n SELECT\n collection,\n partition,\n field,\n indexname,\n iidx as existing_idx,\n qidx as queryable_idx\n FROM i FULL JOIN q USING (field, partition)\n WHERE CASE WHEN changes THEN lower(iidx) IS DISTINCT FROM lower(qidx) ELSE TRUE END;\n;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION maintain_index(\n indexname text,\n queryable_idx text,\n dropindexes boolean DEFAULT FALSE,\n rebuildindexes boolean DEFAULT FALSE,\n idxconcurrently boolean DEFAULT FALSE\n) RETURNS VOID AS $$\nDECLARE\nBEGIN\n IF indexname IS NOT NULL THEN\n IF dropindexes OR queryable_idx IS NOT NULL THEN\n EXECUTE format('DROP INDEX IF EXISTS %I;', indexname);\n ELSIF rebuildindexes THEN\n IF idxconcurrently THEN\n EXECUTE format('REINDEX INDEX CONCURRENTLY %I;', indexname);\n ELSE\n EXECUTE format('REINDEX INDEX CONCURRENTLY %I;', indexname);\n END IF;\n END IF;\n END IF;\n IF queryable_idx IS NOT NULL THEN\n IF idxconcurrently THEN\n EXECUTE replace(queryable_idx, 'INDEX', 'INDEX CONCURRENTLY');\n ELSE EXECUTE queryable_idx;\n END IF;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\n\nset check_function_bodies to off;\nCREATE OR REPLACE FUNCTION maintain_partition_queries(\n part text DEFAULT 'items',\n dropindexes boolean DEFAULT FALSE,\n rebuildindexes boolean DEFAULT FALSE,\n idxconcurrently boolean DEFAULT FALSE\n) RETURNS SETOF text AS $$\nDECLARE\n rec record;\n q text;\nBEGIN\n FOR rec IN (\n SELECT * FROM queryable_indexes(part,true)\n ) LOOP\n q := format(\n 'SELECT maintain_index(\n %L,%L,%L,%L,%L\n );',\n rec.indexname,\n rec.queryable_idx,\n dropindexes,\n rebuildindexes,\n idxconcurrently\n );\n RAISE NOTICE 'Q: %s', q;\n RETURN NEXT q;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION maintain_partitions(\n part text DEFAULT 'items',\n dropindexes boolean DEFAULT FALSE,\n rebuildindexes boolean DEFAULT FALSE\n) RETURNS VOID AS $$\n WITH t AS (\n SELECT run_or_queue(q) FROM maintain_partition_queries(part, dropindexes, rebuildindexes) q\n ) SELECT count(*) FROM t;\n$$ LANGUAGE SQL;\n\n\nCREATE OR REPLACE FUNCTION queryables_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\nBEGIN\n PERFORM maintain_partitions();\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER queryables_trigger AFTER INSERT OR UPDATE ON queryables\nFOR EACH STATEMENT EXECUTE PROCEDURE queryables_trigger_func();\n\n\nCREATE OR REPLACE FUNCTION get_queryables(_collection_ids text[] DEFAULT NULL) RETURNS jsonb AS $$\nDECLARE\nBEGIN\n -- Build up queryables if the input contains valid collection ids or is empty\n IF EXISTS (\n SELECT 1 FROM collections\n WHERE\n _collection_ids IS NULL\n OR cardinality(_collection_ids) = 0\n OR id = ANY(_collection_ids)\n )\n THEN\n RETURN (\n WITH base AS (\n SELECT\n unnest(collection_ids) as collection_id,\n name,\n coalesce(definition, '{\"type\":\"string\"}'::jsonb) as definition\n FROM queryables\n WHERE\n _collection_ids IS NULL OR\n _collection_ids = '{}'::text[] OR\n _collection_ids && collection_ids\n UNION ALL\n SELECT null, name, coalesce(definition, '{\"type\":\"string\"}'::jsonb) as definition\n FROM queryables WHERE collection_ids IS NULL OR collection_ids = '{}'::text[]\n ), g AS (\n SELECT\n name,\n first_notnull(definition) as definition,\n jsonb_array_unique_merge(definition->'enum') as enum,\n jsonb_min(definition->'minimum') as minimum,\n jsonb_min(definition->'maxiumn') as maximum\n FROM base\n GROUP BY 1\n )\n SELECT\n jsonb_build_object(\n '$schema', 'http://json-schema.org/draft-07/schema#',\n '$id', '',\n 'type', 'object',\n 'title', 'STAC Queryables.',\n 'properties', jsonb_object_agg(\n name,\n definition\n ||\n jsonb_strip_nulls(jsonb_build_object(\n 'enum', enum,\n 'minimum', minimum,\n 'maximum', maximum\n ))\n ),\n 'additionalProperties', pgstac.additional_properties()\n )\n FROM g\n );\n ELSE\n RETURN NULL;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\nCREATE OR REPLACE FUNCTION get_queryables(_collection text DEFAULT NULL) RETURNS jsonb AS $$\n SELECT\n CASE\n WHEN _collection IS NULL THEN get_queryables(NULL::text[])\n ELSE get_queryables(ARRAY[_collection])\n END\n ;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_queryables() RETURNS jsonb AS $$\n SELECT get_queryables(NULL::text[]);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION schema_qualify_refs(url text, j jsonb) returns jsonb as $$\n SELECT regexp_replace(j::text, '\"\\$ref\": \"#', concat('\"$ref\": \"', url, '#'), 'g')::jsonb;\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE VIEW stac_extension_queryables AS\nSELECT DISTINCT key as name, schema_qualify_refs(e.url, j.value) as definition FROM stac_extensions e, jsonb_each(e.content->'definitions'->'fields'->'properties') j;\n\n\nCREATE OR REPLACE FUNCTION missing_queryables(_collection text, _tablesample float DEFAULT 5, minrows float DEFAULT 10) RETURNS TABLE(collection text, name text, definition jsonb, property_wrapper text) AS $$\nDECLARE\n q text;\n _partition text;\n explain_json json;\n psize float;\n estrows float;\nBEGIN\n SELECT format('_items_%s', key) INTO _partition FROM collections WHERE id=_collection;\n\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM %I;', _partition)\n INTO explain_json;\n psize := explain_json->0->'Plan'->'Plan Rows';\n estrows := _tablesample * .01 * psize;\n IF estrows < minrows THEN\n _tablesample := least(100,greatest(_tablesample, (estrows / psize) / 100));\n RAISE NOTICE '%', (psize / estrows) / 100;\n END IF;\n RAISE NOTICE 'Using tablesample % to find missing queryables from % % that has ~% rows estrows: %', _tablesample, _collection, _partition, psize, estrows;\n\n q := format(\n $q$\n WITH q AS (\n SELECT * FROM queryables\n WHERE\n collection_ids IS NULL\n OR %L = ANY(collection_ids)\n ), t AS (\n SELECT\n content->'properties' AS properties\n FROM\n %I\n TABLESAMPLE SYSTEM(%L)\n ), p AS (\n SELECT DISTINCT ON (key)\n key,\n value,\n s.definition\n FROM t\n JOIN LATERAL jsonb_each(properties) ON TRUE\n LEFT JOIN q ON (q.name=key)\n LEFT JOIN stac_extension_queryables s ON (s.name=key)\n WHERE q.definition IS NULL\n )\n SELECT\n %L,\n key,\n COALESCE(definition, jsonb_build_object('type',jsonb_typeof(value))) as definition,\n CASE\n WHEN definition->>'type' = 'integer' THEN 'to_int'\n WHEN COALESCE(definition->>'type', jsonb_typeof(value)) = 'number' THEN 'to_float'\n WHEN COALESCE(definition->>'type', jsonb_typeof(value)) = 'array' THEN 'to_text_array'\n ELSE 'to_text'\n END\n FROM p;\n $q$,\n _collection,\n _partition,\n _tablesample,\n _collection\n );\n RETURN QUERY EXECUTE q;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION missing_queryables(_tablesample float DEFAULT 5) RETURNS TABLE(collection_ids text[], name text, definition jsonb, property_wrapper text) AS $$\n SELECT\n array_agg(collection),\n name,\n definition,\n property_wrapper\n FROM\n collections\n JOIN LATERAL\n missing_queryables(id, _tablesample) c\n ON TRUE\n GROUP BY\n 2,3,4\n ORDER BY 2,1\n ;\n$$ LANGUAGE SQL;\nCREATE OR REPLACE FUNCTION parse_dtrange(\n _indate jsonb,\n relative_base timestamptz DEFAULT date_trunc('hour', CURRENT_TIMESTAMP)\n) RETURNS tstzrange AS $$\nDECLARE\n timestrs text[];\n s timestamptz;\n e timestamptz;\nBEGIN\n timestrs :=\n CASE\n WHEN _indate ? 'timestamp' THEN\n ARRAY[_indate->>'timestamp']\n WHEN _indate ? 'interval' THEN\n to_text_array(_indate->'interval')\n WHEN jsonb_typeof(_indate) = 'array' THEN\n to_text_array(_indate)\n ELSE\n regexp_split_to_array(\n _indate->>0,\n '/'\n )\n END;\n RAISE NOTICE 'TIMESTRS %', timestrs;\n IF cardinality(timestrs) = 1 THEN\n IF timestrs[1] ILIKE 'P%' THEN\n RETURN tstzrange(relative_base - upper(timestrs[1])::interval, relative_base, '[)');\n END IF;\n s := timestrs[1]::timestamptz;\n RETURN tstzrange(s, s, '[]');\n END IF;\n\n IF cardinality(timestrs) != 2 THEN\n RAISE EXCEPTION 'Timestamp cannot have more than 2 values';\n END IF;\n\n IF timestrs[1] = '..' OR timestrs[1] = '' THEN\n s := '-infinity'::timestamptz;\n e := timestrs[2]::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] = '..' OR timestrs[2] = '' THEN\n s := timestrs[1]::timestamptz;\n e := 'infinity'::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[1] ILIKE 'P%' AND timestrs[2] NOT ILIKE 'P%' THEN\n e := timestrs[2]::timestamptz;\n s := e - upper(timestrs[1])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] ILIKE 'P%' AND timestrs[1] NOT ILIKE 'P%' THEN\n s := timestrs[1]::timestamptz;\n e := s + upper(timestrs[2])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n s := timestrs[1]::timestamptz;\n e := timestrs[2]::timestamptz;\n\n RETURN tstzrange(s,e,'[)');\n\n RETURN NULL;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT PARALLEL SAFE SET TIME ZONE 'UTC';\n\nCREATE OR REPLACE FUNCTION parse_dtrange(\n _indate text,\n relative_base timestamptz DEFAULT CURRENT_TIMESTAMP\n) RETURNS tstzrange AS $$\n SELECT parse_dtrange(to_jsonb(_indate), relative_base);\n$$ LANGUAGE SQL STABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION temporal_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\n ll text := 'datetime';\n lh text := 'end_datetime';\n rrange tstzrange;\n rl text;\n rh text;\n outq text;\nBEGIN\n rrange := parse_dtrange(args->1);\n RAISE NOTICE 'Constructing temporal query OP: %, ARGS: %, RRANGE: %', op, args, rrange;\n op := lower(op);\n rl := format('%L::timestamptz', lower(rrange));\n rh := format('%L::timestamptz', upper(rrange));\n outq := CASE op\n WHEN 't_before' THEN 'lh < rl'\n WHEN 't_after' THEN 'll > rh'\n WHEN 't_meets' THEN 'lh = rl'\n WHEN 't_metby' THEN 'll = rh'\n WHEN 't_overlaps' THEN 'll < rl AND rl < lh < rh'\n WHEN 't_overlappedby' THEN 'rl < ll < rh AND lh > rh'\n WHEN 't_starts' THEN 'll = rl AND lh < rh'\n WHEN 't_startedby' THEN 'll = rl AND lh > rh'\n WHEN 't_during' THEN 'll > rl AND lh < rh'\n WHEN 't_contains' THEN 'll < rl AND lh > rh'\n WHEN 't_finishes' THEN 'll > rl AND lh = rh'\n WHEN 't_finishedby' THEN 'll < rl AND lh = rh'\n WHEN 't_equals' THEN 'll = rl AND lh = rh'\n WHEN 't_disjoint' THEN 'NOT (ll <= rh AND lh >= rl)'\n WHEN 't_intersects' THEN 'll <= rh AND lh >= rl'\n WHEN 'anyinteracts' THEN 'll <= rh AND lh >= rl'\n END;\n outq := regexp_replace(outq, '\\mll\\M', ll);\n outq := regexp_replace(outq, '\\mlh\\M', lh);\n outq := regexp_replace(outq, '\\mrl\\M', rl);\n outq := regexp_replace(outq, '\\mrh\\M', rh);\n outq := format('(%s)', outq);\n RETURN outq;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\n\n\nCREATE OR REPLACE FUNCTION spatial_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\n geom text;\n j jsonb := args->1;\nBEGIN\n op := lower(op);\n RAISE NOTICE 'Constructing spatial query OP: %, ARGS: %', op, args;\n IF op NOT IN ('s_equals','s_disjoint','s_touches','s_within','s_overlaps','s_crosses','s_intersects','intersects','s_contains') THEN\n RAISE EXCEPTION 'Spatial Operator % Not Supported', op;\n END IF;\n op := regexp_replace(op, '^s_', 'st_');\n IF op = 'intersects' THEN\n op := 'st_intersects';\n END IF;\n -- Convert geometry to WKB string\n IF j ? 'type' AND j ? 'coordinates' THEN\n geom := st_geomfromgeojson(j)::text;\n ELSIF jsonb_typeof(j) = 'array' THEN\n geom := bbox_geom(j)::text;\n END IF;\n\n RETURN format('%s(geometry, %L::geometry)', op, geom);\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION query_to_cql2(q jsonb) RETURNS jsonb AS $$\n-- Translates anything passed in through the deprecated \"query\" into equivalent CQL2\nWITH t AS (\n SELECT key as property, value as ops\n FROM jsonb_each(q)\n), t2 AS (\n SELECT property, (jsonb_each(ops)).*\n FROM t WHERE jsonb_typeof(ops) = 'object'\n UNION ALL\n SELECT property, 'eq', ops\n FROM t WHERE jsonb_typeof(ops) != 'object'\n)\nSELECT\n jsonb_strip_nulls(jsonb_build_object(\n 'op', 'and',\n 'args', jsonb_agg(\n jsonb_build_object(\n 'op', key,\n 'args', jsonb_build_array(\n jsonb_build_object('property',property),\n value\n )\n )\n )\n )\n) as qcql FROM t2\n;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\nCREATE OR REPLACE FUNCTION cql1_to_cql2(j jsonb) RETURNS jsonb AS $$\nDECLARE\n args jsonb;\n ret jsonb;\nBEGIN\n RAISE NOTICE 'CQL1_TO_CQL2: %', j;\n IF j ? 'filter' THEN\n RETURN cql1_to_cql2(j->'filter');\n END IF;\n IF j ? 'property' THEN\n RETURN j;\n END IF;\n IF jsonb_typeof(j) = 'array' THEN\n SELECT jsonb_agg(cql1_to_cql2(el)) INTO args FROM jsonb_array_elements(j) el;\n RETURN args;\n END IF;\n IF jsonb_typeof(j) = 'number' THEN\n RETURN j;\n END IF;\n IF jsonb_typeof(j) = 'string' THEN\n RETURN j;\n END IF;\n\n IF jsonb_typeof(j) = 'object' THEN\n SELECT jsonb_build_object(\n 'op', key,\n 'args', cql1_to_cql2(value)\n ) INTO ret\n FROM jsonb_each(j)\n WHERE j IS NOT NULL;\n RETURN ret;\n END IF;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE STRICT;\n\nCREATE TABLE cql2_ops (\n op text PRIMARY KEY,\n template text,\n types text[]\n);\nINSERT INTO cql2_ops (op, template, types) VALUES\n ('eq', '%s = %s', NULL),\n ('neq', '%s != %s', NULL),\n ('ne', '%s != %s', NULL),\n ('!=', '%s != %s', NULL),\n ('<>', '%s != %s', NULL),\n ('lt', '%s < %s', NULL),\n ('lte', '%s <= %s', NULL),\n ('gt', '%s > %s', NULL),\n ('gte', '%s >= %s', NULL),\n ('le', '%s <= %s', NULL),\n ('ge', '%s >= %s', NULL),\n ('=', '%s = %s', NULL),\n ('<', '%s < %s', NULL),\n ('<=', '%s <= %s', NULL),\n ('>', '%s > %s', NULL),\n ('>=', '%s >= %s', NULL),\n ('like', '%s LIKE %s', NULL),\n ('ilike', '%s ILIKE %s', NULL),\n ('+', '%s + %s', NULL),\n ('-', '%s - %s', NULL),\n ('*', '%s * %s', NULL),\n ('/', '%s / %s', NULL),\n ('not', 'NOT (%s)', NULL),\n ('between', '%s BETWEEN %s AND %s', NULL),\n ('isnull', '%s IS NULL', NULL),\n ('upper', 'upper(%s)', NULL),\n ('lower', 'lower(%s)', NULL)\nON CONFLICT (op) DO UPDATE\n SET\n template = EXCLUDED.template\n;\n\n\nCREATE OR REPLACE FUNCTION cql2_query(j jsonb, wrapper text DEFAULT NULL) RETURNS text AS $$\n#variable_conflict use_variable\nDECLARE\n args jsonb := j->'args';\n arg jsonb;\n op text := lower(j->>'op');\n cql2op RECORD;\n literal text;\n _wrapper text;\n leftarg text;\n rightarg text;\n prop text;\n extra_props bool := pgstac.additional_properties();\nBEGIN\n IF j IS NULL OR (op IS NOT NULL AND args IS NULL) THEN\n RETURN NULL;\n END IF;\n RAISE NOTICE 'CQL2_QUERY: %', j;\n\n -- check if all properties are represented in the queryables\n IF NOT extra_props THEN\n FOR prop IN\n SELECT DISTINCT p->>0\n FROM jsonb_path_query(j, 'strict $.**.property') p\n WHERE p->>0 NOT IN ('id', 'datetime', 'end_datetime', 'collection')\n LOOP\n IF (queryable(prop)).nulled_wrapper IS NULL THEN\n RAISE EXCEPTION 'Term % is not found in queryables.', prop;\n END IF;\n END LOOP;\n END IF;\n\n IF j ? 'filter' THEN\n RETURN cql2_query(j->'filter');\n END IF;\n\n IF j ? 'upper' THEN\n RETURN cql2_query(jsonb_build_object('op', 'upper', 'args', j->'upper'));\n END IF;\n\n IF j ? 'lower' THEN\n RETURN cql2_query(jsonb_build_object('op', 'lower', 'args', j->'lower'));\n END IF;\n\n -- Temporal Query\n IF op ilike 't_%' or op = 'anyinteracts' THEN\n RETURN temporal_op_query(op, args);\n END IF;\n\n -- If property is a timestamp convert it to text to use with\n -- general operators\n IF j ? 'timestamp' THEN\n RETURN format('%L::timestamptz', to_tstz(j->'timestamp'));\n END IF;\n IF j ? 'interval' THEN\n RAISE EXCEPTION 'Please use temporal operators when using intervals.';\n RETURN NONE;\n END IF;\n\n -- Spatial Query\n IF op ilike 's_%' or op = 'intersects' THEN\n RETURN spatial_op_query(op, args);\n END IF;\n\n IF op IN ('a_equals','a_contains','a_contained_by','a_overlaps') THEN\n IF args->0 ? 'property' THEN\n leftarg := format('to_text_array(%s)', (queryable(args->0->>'property')).path);\n END IF;\n IF args->1 ? 'property' THEN\n rightarg := format('to_text_array(%s)', (queryable(args->1->>'property')).path);\n END IF;\n RETURN FORMAT(\n '%s %s %s',\n COALESCE(leftarg, quote_literal(to_text_array(args->0))),\n CASE op\n WHEN 'a_equals' THEN '='\n WHEN 'a_contains' THEN '@>'\n WHEN 'a_contained_by' THEN '<@'\n WHEN 'a_overlaps' THEN '&&'\n END,\n COALESCE(rightarg, quote_literal(to_text_array(args->1)))\n );\n END IF;\n\n IF op = 'in' THEN\n RAISE NOTICE 'IN : % % %', args, jsonb_build_array(args->0), args->1;\n args := jsonb_build_array(args->0) || (args->1);\n RAISE NOTICE 'IN2 : %', args;\n END IF;\n\n\n\n IF op = 'between' THEN\n args = jsonb_build_array(\n args->0,\n args->1->0,\n args->1->1\n );\n END IF;\n\n -- Make sure that args is an array and run cql2_query on\n -- each element of the array\n RAISE NOTICE 'ARGS PRE: %', args;\n IF j ? 'args' THEN\n IF jsonb_typeof(args) != 'array' THEN\n args := jsonb_build_array(args);\n END IF;\n\n IF jsonb_path_exists(args, '$[*] ? (@.property == \"id\" || @.property == \"datetime\" || @.property == \"end_datetime\" || @.property == \"collection\")') THEN\n wrapper := NULL;\n ELSE\n -- if any of the arguments are a property, try to get the property_wrapper\n FOR arg IN SELECT jsonb_path_query(args, '$[*] ? (@.property != null)') LOOP\n RAISE NOTICE 'Arg: %', arg;\n wrapper := (queryable(arg->>'property')).nulled_wrapper;\n RAISE NOTICE 'Property: %, Wrapper: %', arg, wrapper;\n IF wrapper IS NOT NULL THEN\n EXIT;\n END IF;\n END LOOP;\n\n -- if the property was not in queryables, see if any args were numbers\n IF\n wrapper IS NULL\n AND jsonb_path_exists(args, '$[*] ? (@.type()==\"number\")')\n THEN\n wrapper := 'to_float';\n END IF;\n wrapper := coalesce(wrapper, 'to_text');\n END IF;\n\n SELECT jsonb_agg(cql2_query(a, wrapper))\n INTO args\n FROM jsonb_array_elements(args) a;\n END IF;\n RAISE NOTICE 'ARGS: %', args;\n\n IF op IN ('and', 'or') THEN\n RETURN\n format(\n '(%s)',\n array_to_string(to_text_array(args), format(' %s ', upper(op)))\n );\n END IF;\n\n IF op = 'in' THEN\n RAISE NOTICE 'IN -- % %', args->0, to_text(args->0);\n RETURN format(\n '%s IN (%s)',\n to_text(args->0),\n array_to_string((to_text_array(args))[2:], ',')\n );\n END IF;\n\n -- Look up template from cql2_ops\n IF j ? 'op' THEN\n SELECT * INTO cql2op FROM cql2_ops WHERE cql2_ops.op ilike op;\n IF FOUND THEN\n -- If specific index set in queryables for a property cast other arguments to that type\n\n RETURN format(\n cql2op.template,\n VARIADIC (to_text_array(args))\n );\n ELSE\n RAISE EXCEPTION 'Operator % Not Supported.', op;\n END IF;\n END IF;\n\n\n IF wrapper IS NOT NULL THEN\n RAISE NOTICE 'Wrapping % with %', j, wrapper;\n IF j ? 'property' THEN\n RETURN format('%I(%s)', wrapper, (queryable(j->>'property')).path);\n ELSE\n RETURN format('%I(%L)', wrapper, j);\n END IF;\n ELSIF j ? 'property' THEN\n RETURN quote_ident(j->>'property');\n END IF;\n\n RETURN quote_literal(to_text(j));\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION paging_dtrange(\n j jsonb\n) RETURNS tstzrange AS $$\nDECLARE\n op text;\n filter jsonb := j->'filter';\n dtrange tstzrange := tstzrange('-infinity'::timestamptz,'infinity'::timestamptz);\n sdate timestamptz := '-infinity'::timestamptz;\n edate timestamptz := 'infinity'::timestamptz;\n jpitem jsonb;\nBEGIN\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n END IF;\n IF NOT (filter @? '$.**.op ? (@ == \"or\" || @ == \"not\")') THEN\n FOR jpitem IN SELECT j FROM jsonb_path_query(filter,'strict $.** ? (@.args[*].property == \"datetime\")'::jsonpath) j LOOP\n op := lower(jpitem->>'op');\n dtrange := parse_dtrange(jpitem->'args'->1);\n IF op IN ('<=', 'lt', 'lte', '<', 'le', 't_before') THEN\n sdate := greatest(sdate,'-infinity');\n edate := least(edate, upper(dtrange));\n ELSIF op IN ('>=', '>', 'gt', 'gte', 'ge', 't_after') THEN\n edate := least(edate, 'infinity');\n sdate := greatest(sdate, lower(dtrange));\n ELSIF op IN ('=', 'eq') THEN\n edate := least(edate, upper(dtrange));\n sdate := greatest(sdate, lower(dtrange));\n END IF;\n RAISE NOTICE '2 OP: %, ARGS: %, DTRANGE: %, SDATE: %, EDATE: %', op, jpitem->'args'->1, dtrange, sdate, edate;\n END LOOP;\n END IF;\n IF sdate > edate THEN\n RETURN 'empty'::tstzrange;\n END IF;\n RETURN tstzrange(sdate,edate, '[]');\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT SET TIME ZONE 'UTC';\n\nCREATE OR REPLACE FUNCTION paging_collections(\n IN j jsonb\n) RETURNS text[] AS $$\nDECLARE\n filter jsonb := j->'filter';\n jpitem jsonb;\n op text;\n args jsonb;\n arg jsonb;\n collections text[];\nBEGIN\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n END IF;\n IF NOT (filter @? '$.**.op ? (@ == \"or\" || @ == \"not\")') THEN\n FOR jpitem IN SELECT j FROM jsonb_path_query(filter,'strict $.** ? (@.args[*].property == \"collection\")'::jsonpath) j LOOP\n RAISE NOTICE 'JPITEM: %', jpitem;\n op := jpitem->>'op';\n args := jpitem->'args';\n IF op IN ('=', 'eq', 'in') THEN\n FOR arg IN SELECT a FROM jsonb_array_elements(args) a LOOP\n IF jsonb_typeof(arg) IN ('string', 'array') THEN\n RAISE NOTICE 'arg: %, collections: %', arg, collections;\n IF collections IS NULL OR collections = '{}'::text[] THEN\n collections := to_text_array(arg);\n ELSE\n collections := array_intersection(collections, to_text_array(arg));\n END IF;\n END IF;\n END LOOP;\n END IF;\n END LOOP;\n END IF;\n IF collections = '{}'::text[] THEN\n RETURN NULL;\n END IF;\n RETURN collections;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\nCREATE TABLE items (\n id text NOT NULL,\n geometry geometry NOT NULL,\n collection text NOT NULL,\n datetime timestamptz NOT NULL,\n end_datetime timestamptz NOT NULL,\n content JSONB NOT NULL,\n private jsonb\n)\nPARTITION BY LIST (collection)\n;\n\nCREATE INDEX \"datetime_idx\" ON items USING BTREE (datetime DESC, end_datetime ASC);\nCREATE INDEX \"geometry_idx\" ON items USING GIST (geometry);\n\nCREATE STATISTICS datetime_stats (dependencies) on datetime, end_datetime from items;\n\nALTER TABLE items ADD CONSTRAINT items_collections_fk FOREIGN KEY (collection) REFERENCES collections(id) ON DELETE CASCADE DEFERRABLE;\n\nCREATE OR REPLACE FUNCTION partition_after_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p text;\n t timestamptz := clock_timestamp();\nBEGIN\n RAISE NOTICE 'Updating partition stats %', t;\n FOR p IN SELECT DISTINCT partition\n FROM newdata n JOIN partition_sys_meta p\n ON (n.collection=p.collection AND n.datetime <@ p.partition_dtrange)\n LOOP\n PERFORM run_or_queue(format('SELECT update_partition_stats(%L, %L);', p, true));\n END LOOP;\n IF TG_OP IN ('DELETE','UPDATE') THEN\n DELETE FROM format_item_cache c USING newdata n WHERE c.collection = n.collection AND c.id = n.id;\n END IF;\n RAISE NOTICE 't: % %', t, clock_timestamp() - t;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE TRIGGER items_after_insert_trigger\nAFTER INSERT ON items\nREFERENCING NEW TABLE AS newdata\nFOR EACH STATEMENT\nEXECUTE FUNCTION partition_after_triggerfunc();\n\nCREATE TRIGGER items_after_update_trigger\nAFTER DELETE ON items\nREFERENCING OLD TABLE AS newdata\nFOR EACH STATEMENT\nEXECUTE FUNCTION partition_after_triggerfunc();\n\nCREATE TRIGGER items_after_delete_trigger\nAFTER UPDATE ON items\nREFERENCING NEW TABLE AS newdata\nFOR EACH STATEMENT\nEXECUTE FUNCTION partition_after_triggerfunc();\n\n\nCREATE OR REPLACE FUNCTION content_slim(_item jsonb) RETURNS jsonb AS $$\n SELECT strip_jsonb(_item - '{id,geometry,collection,type}'::text[], collection_base_item(_item->>'collection')) - '{id,geometry,collection,type}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_dehydrate(content jsonb) RETURNS items AS $$\n SELECT\n content->>'id' as id,\n stac_geom(content) as geometry,\n content->>'collection' as collection,\n stac_datetime(content) as datetime,\n stac_end_datetime(content) as end_datetime,\n content_slim(content) as content,\n null::jsonb as private\n ;\n$$ LANGUAGE SQL STABLE;\n\nCREATE OR REPLACE FUNCTION include_field(f text, fields jsonb DEFAULT '{}'::jsonb) RETURNS boolean AS $$\nDECLARE\n includes jsonb := fields->'include';\n excludes jsonb := fields->'exclude';\nBEGIN\n IF f IS NULL THEN\n RETURN NULL;\n END IF;\n\n\n IF\n jsonb_typeof(excludes) = 'array'\n AND jsonb_array_length(excludes)>0\n AND excludes ? f\n THEN\n RETURN FALSE;\n END IF;\n\n IF\n (\n jsonb_typeof(includes) = 'array'\n AND jsonb_array_length(includes) > 0\n AND includes ? f\n ) OR\n (\n includes IS NULL\n OR jsonb_typeof(includes) = 'null'\n OR jsonb_array_length(includes) = 0\n )\n THEN\n RETURN TRUE;\n END IF;\n\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nDROP FUNCTION IF EXISTS content_hydrate(jsonb, jsonb, jsonb);\nCREATE OR REPLACE FUNCTION content_hydrate(\n _item jsonb,\n _base_item jsonb,\n fields jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\n SELECT merge_jsonb(\n jsonb_fields(_item, fields),\n jsonb_fields(_base_item, fields)\n );\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION content_hydrate(_item items, _collection collections, fields jsonb DEFAULT '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\n content jsonb;\n base_item jsonb := _collection.base_item;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry, 20)::jsonb;\n END IF;\n output := content_hydrate(\n jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'collection', _item.collection,\n 'type', 'Feature'\n ) || _item.content,\n _collection.base_item,\n fields\n );\n\n RETURN output;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_nonhydrated(\n _item items,\n fields jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry, 20)::jsonb;\n END IF;\n output := jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'collection', _item.collection,\n 'type', 'Feature'\n ) || _item.content;\n RETURN output;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_hydrate(_item items, fields jsonb DEFAULT '{}'::jsonb) RETURNS jsonb AS $$\n SELECT content_hydrate(\n _item,\n (SELECT c FROM collections c WHERE id=_item.collection LIMIT 1),\n fields\n );\n$$ LANGUAGE SQL STABLE;\n\n\nCREATE UNLOGGED TABLE items_staging (\n content JSONB NOT NULL\n);\nCREATE UNLOGGED TABLE items_staging_ignore (\n content JSONB NOT NULL\n);\nCREATE UNLOGGED TABLE items_staging_upsert (\n content JSONB NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION items_staging_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p record;\n _partitions text[];\n part text;\n ts timestamptz := clock_timestamp();\nBEGIN\n RAISE NOTICE 'Creating Partitions. %', clock_timestamp() - ts;\n\n FOR part IN WITH t AS (\n SELECT\n n.content->>'collection' as collection,\n stac_daterange(n.content->'properties') as dtr,\n partition_trunc\n FROM newdata n JOIN collections ON (n.content->>'collection'=collections.id)\n ), p AS (\n SELECT\n collection,\n COALESCE(date_trunc(partition_trunc::text, lower(dtr)),'-infinity') as d,\n tstzrange(min(lower(dtr)),max(lower(dtr)),'[]') as dtrange,\n tstzrange(min(upper(dtr)),max(upper(dtr)),'[]') as edtrange\n FROM t\n GROUP BY 1,2\n ) SELECT check_partition(collection, dtrange, edtrange) FROM p LOOP\n RAISE NOTICE 'Partition %', part;\n END LOOP;\n\n RAISE NOTICE 'Doing the insert. %', clock_timestamp() - ts;\n IF TG_TABLE_NAME = 'items_staging' THEN\n INSERT INTO items\n SELECT\n (content_dehydrate(content)).*\n FROM newdata;\n RAISE NOTICE 'Doing the delete. %', clock_timestamp() - ts;\n DELETE FROM items_staging;\n ELSIF TG_TABLE_NAME = 'items_staging_ignore' THEN\n INSERT INTO items\n SELECT\n (content_dehydrate(content)).*\n FROM newdata\n ON CONFLICT DO NOTHING;\n RAISE NOTICE 'Doing the delete. %', clock_timestamp() - ts;\n DELETE FROM items_staging_ignore;\n ELSIF TG_TABLE_NAME = 'items_staging_upsert' THEN\n WITH staging_formatted AS (\n SELECT (content_dehydrate(content)).* FROM newdata\n ), deletes AS (\n DELETE FROM items i USING staging_formatted s\n WHERE\n i.id = s.id\n AND i.collection = s.collection\n AND i IS DISTINCT FROM s\n RETURNING i.id, i.collection\n )\n INSERT INTO items\n SELECT s.* FROM\n staging_formatted s\n ON CONFLICT DO NOTHING;\n RAISE NOTICE 'Doing the delete. %', clock_timestamp() - ts;\n DELETE FROM items_staging_upsert;\n END IF;\n RAISE NOTICE 'Done. %', clock_timestamp() - ts;\n\n RETURN NULL;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER items_staging_insert_trigger AFTER INSERT ON items_staging REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\nCREATE TRIGGER items_staging_insert_ignore_trigger AFTER INSERT ON items_staging_ignore REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\nCREATE TRIGGER items_staging_insert_upsert_trigger AFTER INSERT ON items_staging_upsert REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\n\nCREATE OR REPLACE FUNCTION item_by_id(_id text, _collection text DEFAULT NULL) RETURNS items AS\n$$\nDECLARE\n i items%ROWTYPE;\nBEGIN\n SELECT * INTO i FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection) LIMIT 1;\n RETURN i;\nEND;\n$$ LANGUAGE PLPGSQL STABLE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION get_item(_id text, _collection text DEFAULT NULL) RETURNS jsonb AS $$\n SELECT content_hydrate(items) FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection);\n$$ LANGUAGE SQL STABLE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION delete_item(_id text, _collection text DEFAULT NULL) RETURNS VOID AS $$\nDECLARE\nout items%ROWTYPE;\nBEGIN\n DELETE FROM items WHERE id = _id AND (_collection IS NULL OR collection=_collection) RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n--/*\nCREATE OR REPLACE FUNCTION create_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION update_item(content jsonb) RETURNS VOID AS $$\nDECLARE\n old items %ROWTYPE;\n out items%ROWTYPE;\nBEGIN\n PERFORM delete_item(content->>'id', content->>'collection');\n PERFORM create_item(content);\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION create_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION collection_bbox(id text) RETURNS jsonb AS $$\n SELECT (replace(replace(replace(st_extent(geometry)::text,'BOX(','[['),')',']]'),' ',','))::jsonb\n FROM items WHERE collection=$1;\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION collection_temporal_extent(id text) RETURNS jsonb AS $$\n SELECT to_jsonb(array[array[min(datetime)::text, max(datetime)::text]])\n FROM items WHERE collection=$1;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection_extents() RETURNS VOID AS $$\nUPDATE collections SET\n content = content ||\n jsonb_build_object(\n 'extent', jsonb_build_object(\n 'spatial', jsonb_build_object(\n 'bbox', collection_bbox(collections.id)\n ),\n 'temporal', jsonb_build_object(\n 'interval', collection_temporal_extent(collections.id)\n )\n )\n )\n;\n$$ LANGUAGE SQL;\nCREATE TABLE partition_stats (\n partition text PRIMARY KEY,\n dtrange tstzrange,\n edtrange tstzrange,\n spatial geometry,\n last_updated timestamptz,\n keys text[]\n) WITH (FILLFACTOR=90);\n\nCREATE INDEX partitions_range_idx ON partition_stats USING GIST(dtrange);\n\n\nCREATE OR REPLACE FUNCTION constraint_tstzrange(expr text) RETURNS tstzrange AS $$\n WITH t AS (\n SELECT regexp_matches(\n expr,\n E'\\\\(''\\([0-9 :+-]*\\)''\\\\).*\\\\(''\\([0-9 :+-]*\\)''\\\\)'\n ) AS m\n ) SELECT tstzrange(m[1]::timestamptz, m[2]::timestamptz) FROM t\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE STRICT;\n\nCREATE OR REPLACE FUNCTION dt_constraint(coid oid, OUT dt tstzrange, OUT edt tstzrange) RETURNS RECORD AS $$\nDECLARE\n expr text := pg_get_constraintdef(coid);\n matches timestamptz[];\nBEGIN\n IF expr LIKE '%NULL%' THEN\n dt := tstzrange(null::timestamptz, null::timestamptz);\n edt := tstzrange(null::timestamptz, null::timestamptz);\n RETURN;\n END IF;\n WITH f AS (SELECT (regexp_matches(expr, E'([0-9]{4}-[0-1][0-9]-[0-3][0-9] [0-2][0-9]:[0-5][0-9]:[0-5][0-9]\\.?[0-9]*)', 'g'))[1] f)\n SELECT array_agg(f::timestamptz) INTO matches FROM f;\n IF cardinality(matches) = 4 THEN\n dt := tstzrange(matches[1], matches[2],'[]');\n edt := tstzrange(matches[3], matches[4], '[]');\n RETURN;\n ELSIF cardinality(matches) = 2 THEN\n edt := tstzrange(matches[1], matches[2],'[]');\n RETURN;\n END IF;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\nCREATE OR REPLACE VIEW partition_sys_meta AS\nSELECT\n relid::text as partition,\n replace(replace(CASE WHEN level = 1 THEN pg_get_expr(c.relpartbound, c.oid)\n ELSE pg_get_expr(parent.relpartbound, parent.oid)\n END, 'FOR VALUES IN (''',''), ''')','') AS collection,\n level,\n c.reltuples,\n c.relhastriggers,\n COALESCE(constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as partition_dtrange,\n COALESCE((dt_constraint(edt.oid)).dt, constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as constraint_dtrange,\n COALESCE((dt_constraint(edt.oid)).edt, tstzrange('-infinity', 'infinity','[]')) as constraint_edtrange\nFROM\n pg_partition_tree('items')\n JOIN pg_class c ON (relid::regclass = c.oid)\n JOIN pg_class parent ON (parentrelid::regclass = parent.oid AND isleaf)\n LEFT JOIN pg_constraint edt ON (conrelid=c.oid AND contype='c')\nWHERE isleaf\n;\n\nCREATE VIEW partitions_view AS\nSELECT\n relid::text as partition,\n replace(replace(CASE WHEN level = 1 THEN pg_get_expr(c.relpartbound, c.oid)\n ELSE pg_get_expr(parent.relpartbound, parent.oid)\n END, 'FOR VALUES IN (''',''), ''')','') AS collection,\n level,\n c.reltuples,\n c.relhastriggers,\n COALESCE(constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as partition_dtrange,\n COALESCE((dt_constraint(edt.oid)).dt, constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as constraint_dtrange,\n COALESCE((dt_constraint(edt.oid)).edt, tstzrange('-infinity', 'infinity','[]')) as constraint_edtrange,\n dtrange,\n edtrange,\n spatial,\n last_updated\nFROM\n pg_partition_tree('items')\n JOIN pg_class c ON (relid::regclass = c.oid)\n JOIN pg_class parent ON (parentrelid::regclass = parent.oid AND isleaf)\n LEFT JOIN pg_constraint edt ON (conrelid=c.oid AND contype='c')\n LEFT JOIN partition_stats ON (relid::text=partition)\nWHERE isleaf\n;\n\nCREATE MATERIALIZED VIEW partitions AS\nSELECT * FROM partitions_view;\nCREATE UNIQUE INDEX ON partitions (partition);\n\nCREATE MATERIALIZED VIEW partition_steps AS\nSELECT\n partition as name,\n date_trunc('month',lower(partition_dtrange)) as sdate,\n date_trunc('month', upper(partition_dtrange)) + '1 month'::interval as edate\n FROM partitions_view WHERE partition_dtrange IS NOT NULL AND partition_dtrange != 'empty'::tstzrange\n ORDER BY dtrange ASC\n;\n\n\nCREATE OR REPLACE FUNCTION update_partition_stats_q(_partition text, istrigger boolean default false) RETURNS VOID AS $$\nDECLARE\nBEGIN\n PERFORM run_or_queue(\n format('SELECT update_partition_stats(%L, %L);', _partition, istrigger)\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION update_partition_stats(_partition text, istrigger boolean default false) RETURNS VOID AS $$\nDECLARE\n dtrange tstzrange;\n edtrange tstzrange;\n cdtrange tstzrange;\n cedtrange tstzrange;\n extent geometry;\n collection text;\nBEGIN\n RAISE NOTICE 'Updating stats for %.', _partition;\n EXECUTE format(\n $q$\n SELECT\n tstzrange(min(datetime), max(datetime),'[]'),\n tstzrange(min(end_datetime), max(end_datetime), '[]')\n FROM %I\n $q$,\n _partition\n ) INTO dtrange, edtrange;\n extent := st_estimatedextent('pgstac', _partition, 'geometry');\n INSERT INTO partition_stats (partition, dtrange, edtrange, spatial, last_updated)\n SELECT _partition, dtrange, edtrange, extent, now()\n ON CONFLICT (partition) DO\n UPDATE SET\n dtrange=EXCLUDED.dtrange,\n edtrange=EXCLUDED.edtrange,\n spatial=EXCLUDED.spatial,\n last_updated=EXCLUDED.last_updated\n ;\n\n SELECT\n constraint_dtrange, constraint_edtrange, pv.collection\n INTO cdtrange, cedtrange, collection\n FROM partitions_view pv WHERE partition = _partition;\n REFRESH MATERIALIZED VIEW partitions;\n REFRESH MATERIALIZED VIEW partition_steps;\n\n\n RAISE NOTICE 'Checking if we need to modify constraints...';\n RAISE NOTICE 'cdtrange: % dtrange: % cedtrange: % edtrange: %',cdtrange, dtrange, cedtrange, edtrange;\n IF\n (cdtrange IS DISTINCT FROM dtrange OR edtrange IS DISTINCT FROM cedtrange)\n AND NOT istrigger\n THEN\n RAISE NOTICE 'Modifying Constraints';\n RAISE NOTICE 'Existing % %', cdtrange, cedtrange;\n RAISE NOTICE 'New % %', dtrange, edtrange;\n PERFORM drop_table_constraints(_partition);\n PERFORM create_table_constraints(_partition, dtrange, edtrange);\n REFRESH MATERIALIZED VIEW partitions;\n REFRESH MATERIALIZED VIEW partition_steps;\n END IF;\n RAISE NOTICE 'Checking if we need to update collection extents.';\n IF get_setting_bool('update_collection_extent') THEN\n RAISE NOTICE 'updating collection extent for %', collection;\n PERFORM run_or_queue(format($q$\n UPDATE collections\n SET content = jsonb_set_lax(\n content,\n '{extent}'::text[],\n collection_extent(%L, FALSE),\n true,\n 'use_json_null'\n ) WHERE id=%L\n ;\n $q$, collection, collection));\n ELSE\n RAISE NOTICE 'Not updating collection extent for %', collection;\n END IF;\n\nEND;\n$$ LANGUAGE PLPGSQL STRICT SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION partition_name( IN collection text, IN dt timestamptz, OUT partition_name text, OUT partition_range tstzrange) AS $$\nDECLARE\n c RECORD;\n parent_name text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n parent_name := format('_items_%s', c.key);\n\n\n IF c.partition_trunc = 'year' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYY'));\n ELSIF c.partition_trunc = 'month' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYYMM'));\n ELSE\n partition_name := parent_name;\n partition_range := tstzrange('-infinity'::timestamptz, 'infinity'::timestamptz, '[]');\n END IF;\n IF partition_range IS NULL THEN\n partition_range := tstzrange(\n date_trunc(c.partition_trunc::text, dt),\n date_trunc(c.partition_trunc::text, dt) + concat('1 ', c.partition_trunc)::interval\n );\n END IF;\n RETURN;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION drop_table_constraints(t text) RETURNS text AS $$\nDECLARE\n q text;\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM partitions_view WHERE partition=t) THEN\n RETURN NULL;\n END IF;\n FOR q IN SELECT FORMAT(\n $q$\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n $q$,\n t,\n conname\n ) FROM pg_constraint\n WHERE conrelid=t::regclass::oid AND contype='c'\n LOOP\n EXECUTE q;\n END LOOP;\n RETURN t;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE OR REPLACE FUNCTION create_table_constraints(t text, _dtrange tstzrange, _edtrange tstzrange) RETURNS text AS $$\nDECLARE\n q text;\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM partitions_view WHERE partition=t) THEN\n RETURN NULL;\n END IF;\n RAISE NOTICE 'Creating Table Constraints for % % %', t, _dtrange, _edtrange;\n IF _dtrange = 'empty' AND _edtrange = 'empty' THEN\n q :=format(\n $q$\n DO $block$\n BEGIN\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n ALTER TABLE %I\n ADD CONSTRAINT %I\n CHECK (((datetime IS NULL) AND (end_datetime IS NULL))) NOT VALID\n ;\n ALTER TABLE %I\n VALIDATE CONSTRAINT %I\n ;\n\n\n\n EXCEPTION WHEN others THEN\n RAISE WARNING '%%, Issue Altering Constraints. Please run update_partition_stats(%I)', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n $block$;\n $q$,\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t),\n t\n );\n ELSE\n q :=format(\n $q$\n DO $block$\n BEGIN\n\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n ALTER TABLE %I\n ADD CONSTRAINT %I\n CHECK (\n (datetime >= %L)\n AND (datetime <= %L)\n AND (end_datetime >= %L)\n AND (end_datetime <= %L)\n ) NOT VALID\n ;\n ALTER TABLE %I\n VALIDATE CONSTRAINT %I\n ;\n\n\n\n EXCEPTION WHEN others THEN\n RAISE WARNING '%%, Issue Altering Constraints. Please run update_partition_stats(%I)', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n $block$;\n $q$,\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t),\n lower(_dtrange),\n upper(_dtrange),\n lower(_edtrange),\n upper(_edtrange),\n t,\n format('%s_dt', t),\n t\n );\n END IF;\n PERFORM run_or_queue(q);\n RETURN t;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION check_partition(\n _collection text,\n _dtrange tstzrange,\n _edtrange tstzrange\n) RETURNS text AS $$\nDECLARE\n c RECORD;\n pm RECORD;\n _partition_name text;\n _partition_dtrange tstzrange;\n _constraint_dtrange tstzrange;\n _constraint_edtrange tstzrange;\n q text;\n deferrable_q text;\n err_context text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=_collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', _collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n\n IF c.partition_trunc IS NOT NULL THEN\n _partition_dtrange := tstzrange(\n date_trunc(c.partition_trunc, lower(_dtrange)),\n date_trunc(c.partition_trunc, lower(_dtrange)) + (concat('1 ', c.partition_trunc))::interval,\n '[)'\n );\n ELSE\n _partition_dtrange := '[-infinity, infinity]'::tstzrange;\n END IF;\n\n IF NOT _partition_dtrange @> _dtrange THEN\n RAISE EXCEPTION 'dtrange % is greater than the partition size % for collection %', _dtrange, c.partition_trunc, _collection;\n END IF;\n\n\n IF c.partition_trunc = 'year' THEN\n _partition_name := format('_items_%s_%s', c.key, to_char(lower(_partition_dtrange),'YYYY'));\n ELSIF c.partition_trunc = 'month' THEN\n _partition_name := format('_items_%s_%s', c.key, to_char(lower(_partition_dtrange),'YYYYMM'));\n ELSE\n _partition_name := format('_items_%s', c.key);\n END IF;\n\n SELECT * INTO pm FROM partition_sys_meta WHERE collection=_collection AND partition_dtrange @> _dtrange;\n IF FOUND THEN\n RAISE NOTICE '% % %', _edtrange, _dtrange, pm;\n _constraint_edtrange :=\n tstzrange(\n least(\n lower(_edtrange),\n nullif(lower(pm.constraint_edtrange), '-infinity')\n ),\n greatest(\n upper(_edtrange),\n nullif(upper(pm.constraint_edtrange), 'infinity')\n ),\n '[]'\n );\n _constraint_dtrange :=\n tstzrange(\n least(\n lower(_dtrange),\n nullif(lower(pm.constraint_dtrange), '-infinity')\n ),\n greatest(\n upper(_dtrange),\n nullif(upper(pm.constraint_dtrange), 'infinity')\n ),\n '[]'\n );\n\n IF pm.constraint_edtrange @> _edtrange AND pm.constraint_dtrange @> _dtrange THEN\n RETURN pm.partition;\n ELSE\n PERFORM drop_table_constraints(_partition_name);\n END IF;\n ELSE\n _constraint_edtrange := _edtrange;\n _constraint_dtrange := _dtrange;\n END IF;\n RAISE NOTICE 'EXISTING CONSTRAINTS % %, NEW % %', pm.constraint_dtrange, pm.constraint_edtrange, _constraint_dtrange, _constraint_edtrange;\n RAISE NOTICE 'Creating partition % %', _partition_name, _partition_dtrange;\n IF c.partition_trunc IS NULL THEN\n q := format(\n $q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n GRANT ALL ON %I to pgstac_ingest;\n $q$,\n _partition_name,\n _collection,\n concat(_partition_name,'_pk'),\n _partition_name,\n _partition_name\n );\n ELSE\n q := format(\n $q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L) PARTITION BY RANGE (datetime);\n CREATE TABLE IF NOT EXISTS %I partition OF %I FOR VALUES FROM (%L) TO (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n GRANT ALL ON %I TO pgstac_ingest;\n $q$,\n format('_items_%s', c.key),\n _collection,\n _partition_name,\n format('_items_%s', c.key),\n lower(_partition_dtrange),\n upper(_partition_dtrange),\n format('%s_pk', _partition_name),\n _partition_name,\n _partition_name\n );\n END IF;\n\n BEGIN\n EXECUTE q;\n EXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', _partition_name;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\n END;\n PERFORM maintain_partitions(_partition_name);\n PERFORM update_partition_stats_q(_partition_name, true);\n REFRESH MATERIALIZED VIEW partitions;\n REFRESH MATERIALIZED VIEW partition_steps;\n RETURN _partition_name;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION repartition(_collection text, _partition_trunc text, triggered boolean DEFAULT FALSE) RETURNS text AS $$\nDECLARE\n c RECORD;\n q text;\n from_trunc text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=_collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', _collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n IF triggered THEN\n RAISE NOTICE 'Converting % to % partitioning via Trigger', _collection, _partition_trunc;\n ELSE\n RAISE NOTICE 'Converting % from using % to % partitioning', _collection, c.partition_trunc, _partition_trunc;\n IF c.partition_trunc IS NOT DISTINCT FROM _partition_trunc THEN\n RAISE NOTICE 'Collection % already set to use partition by %', _collection, _partition_trunc;\n RETURN _collection;\n END IF;\n END IF;\n\n IF EXISTS (SELECT 1 FROM partitions_view WHERE collection=_collection LIMIT 1) THEN\n EXECUTE format(\n $q$\n CREATE TEMP TABLE changepartitionstaging ON COMMIT DROP AS SELECT * FROM %I;\n DROP TABLE IF EXISTS %I CASCADE;\n WITH p AS (\n SELECT\n collection,\n CASE\n WHEN %L IS NULL THEN '-infinity'::timestamptz\n ELSE date_trunc(%L, datetime)\n END as d,\n tstzrange(min(datetime),max(datetime),'[]') as dtrange,\n tstzrange(min(end_datetime),max(end_datetime),'[]') as edtrange\n FROM changepartitionstaging\n GROUP BY 1,2\n ) SELECT check_partition(collection, dtrange, edtrange) FROM p;\n INSERT INTO items SELECT * FROM changepartitionstaging;\n DROP TABLE changepartitionstaging;\n $q$,\n concat('_items_', c.key),\n concat('_items_', c.key),\n c.partition_trunc,\n c.partition_trunc\n );\n END IF;\n RETURN _collection;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE OR REPLACE FUNCTION collections_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n q text;\n partition_name text := format('_items_%s', NEW.key);\n partition_exists boolean := false;\n partition_empty boolean := true;\n err_context text;\n loadtemp boolean := FALSE;\nBEGIN\n RAISE NOTICE 'Collection Trigger. % %', NEW.id, NEW.key;\n IF TG_OP = 'UPDATE' AND NEW.partition_trunc IS DISTINCT FROM OLD.partition_trunc THEN\n PERFORM repartition(NEW.id, NEW.partition_trunc, TRUE);\n END IF;\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER collections_trigger AFTER\nINSERT\nOR\nUPDATE ON collections\nFOR EACH ROW EXECUTE FUNCTION collections_trigger_func();\n\nCREATE OR REPLACE FUNCTION chunker(\n IN _where text,\n OUT s timestamptz,\n OUT e timestamptz\n) RETURNS SETOF RECORD AS $$\nDECLARE\n explain jsonb;\nBEGIN\n IF _where IS NULL THEN\n _where := ' TRUE ';\n END IF;\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s;', _where)\n INTO explain;\n\n RETURN QUERY\n WITH t AS (\n SELECT j->>0 as p FROM\n jsonb_path_query(\n explain,\n 'strict $.**.\"Relation Name\" ? (@ != null)'\n ) j\n ),\n parts AS (\n SELECT sdate, edate FROM t JOIN partition_steps ON (t.p = name)\n ),\n times AS (\n SELECT sdate FROM parts\n UNION\n SELECT edate FROM parts\n ),\n uniq AS (\n SELECT DISTINCT sdate FROM times ORDER BY sdate\n ),\n last AS (\n SELECT sdate, lead(sdate, 1) over () as edate FROM uniq\n )\n SELECT sdate, edate FROM last WHERE edate IS NOT NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION partition_queries(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN partitions text[] DEFAULT NULL\n) RETURNS SETOF text AS $$\nDECLARE\n query text;\n sdate timestamptz;\n edate timestamptz;\nBEGIN\nIF _where IS NULL OR trim(_where) = '' THEN\n _where = ' TRUE ';\nEND IF;\nRAISE NOTICE 'Getting chunks for % %', _where, _orderby;\nIF _orderby ILIKE 'datetime d%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 DESC LOOP\n RETURN NEXT format($q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n $q$,\n sdate,\n edate,\n _where,\n _orderby\n );\n END LOOP;\nELSIF _orderby ILIKE 'datetime a%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 ASC LOOP\n RETURN NEXT format($q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n $q$,\n sdate,\n edate,\n _where,\n _orderby\n );\n END LOOP;\nELSE\n query := format($q$\n SELECT * FROM items\n WHERE %s\n ORDER BY %s\n $q$, _where, _orderby\n );\n\n RETURN NEXT query;\n RETURN;\nEND IF;\n\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION partition_query_view(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN _limit int DEFAULT 10\n) RETURNS text AS $$\n WITH p AS (\n SELECT * FROM partition_queries(_where, _orderby) p\n )\n SELECT\n CASE WHEN EXISTS (SELECT 1 FROM p) THEN\n (SELECT format($q$\n SELECT * FROM (\n %s\n ) total LIMIT %s\n $q$,\n string_agg(\n format($q$ SELECT * FROM ( %s ) AS sub $q$, p),\n '\n UNION ALL\n '\n ),\n _limit\n ))\n ELSE NULL\n END FROM p;\n$$ LANGUAGE SQL IMMUTABLE;\n\n\n\n\nCREATE OR REPLACE FUNCTION stac_search_to_where(j jsonb) RETURNS text AS $$\nDECLARE\n where_segments text[];\n _where text;\n dtrange tstzrange;\n collections text[];\n geom geometry;\n sdate timestamptz;\n edate timestamptz;\n filterlang text;\n filter jsonb := j->'filter';\nBEGIN\n IF j ? 'ids' THEN\n where_segments := where_segments || format('id = ANY (%L) ', to_text_array(j->'ids'));\n END IF;\n\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n where_segments := where_segments || format('collection = ANY (%L) ', collections);\n END IF;\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n\n where_segments := where_segments || format(' datetime <= %L::timestamptz AND end_datetime >= %L::timestamptz ',\n edate,\n sdate\n );\n END IF;\n\n geom := stac_geom(j);\n IF geom IS NOT NULL THEN\n where_segments := where_segments || format('st_intersects(geometry, %L)',geom);\n END IF;\n\n filterlang := COALESCE(\n j->>'filter-lang',\n get_setting('default_filter_lang', j->'conf')\n );\n IF NOT filter @? '$.**.op' THEN\n filterlang := 'cql-json';\n END IF;\n\n IF filterlang NOT IN ('cql-json','cql2-json') AND j ? 'filter' THEN\n RAISE EXCEPTION '% is not a supported filter-lang. Please use cql-json or cql2-json.', filterlang;\n END IF;\n\n IF j ? 'query' AND j ? 'filter' THEN\n RAISE EXCEPTION 'Can only use either query or filter at one time.';\n END IF;\n\n IF j ? 'query' THEN\n filter := query_to_cql2(j->'query');\n ELSIF filterlang = 'cql-json' THEN\n filter := cql1_to_cql2(filter);\n END IF;\n RAISE NOTICE 'FILTER: %', filter;\n where_segments := where_segments || cql2_query(filter);\n IF cardinality(where_segments) < 1 THEN\n RETURN ' TRUE ';\n END IF;\n\n _where := array_to_string(array_remove(where_segments, NULL), ' AND ');\n\n IF _where IS NULL OR BTRIM(_where) = '' THEN\n RETURN ' TRUE ';\n END IF;\n RETURN _where;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION parse_sort_dir(_dir text, reverse boolean default false) RETURNS text AS $$\n WITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n ) SELECT\n CASE\n WHEN NOT reverse THEN d\n WHEN d = 'ASC' THEN 'DESC'\n WHEN d = 'DESC' THEN 'ASC'\n END\n FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION sort_dir_to_op(_dir text, prev boolean default false) RETURNS text AS $$\n WITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n ) SELECT\n CASE\n WHEN d = 'ASC' AND prev THEN '<='\n WHEN d = 'DESC' AND prev THEN '>='\n WHEN d = 'ASC' THEN '>='\n WHEN d = 'DESC' THEN '<='\n END\n FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION sort_sqlorderby(\n _search jsonb DEFAULT NULL,\n reverse boolean DEFAULT FALSE\n) RETURNS text AS $$\n WITH sortby AS (\n SELECT coalesce(_search->'sortby','[{\"field\":\"datetime\", \"direction\":\"desc\"}]') as sort\n ), withid AS (\n SELECT CASE\n WHEN sort @? '$[*] ? (@.field == \"id\")' THEN sort\n ELSE sort || '[{\"field\":\"id\", \"direction\":\"desc\"}]'::jsonb\n END as sort\n FROM sortby\n ), withid_rows AS (\n SELECT jsonb_array_elements(sort) as value FROM withid\n ),sorts AS (\n SELECT\n coalesce(\n (queryable(value->>'field')).expression\n ) as key,\n parse_sort_dir(value->>'direction', reverse) as dir\n FROM withid_rows\n )\n SELECT array_to_string(\n array_agg(concat(key, ' ', dir)),\n ', '\n ) FROM sorts;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_sort_dir(sort_item jsonb) RETURNS text AS $$\n SELECT CASE WHEN sort_item->>'direction' ILIKE 'desc%' THEN 'DESC' ELSE 'ASC' END;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION get_token_val_str(\n _field text,\n _item items\n) RETURNS text AS $$\nDECLARE\n q text;\n literal text;\nBEGIN\n q := format($q$ SELECT quote_literal(%s) FROM (SELECT $1.*) as r;$q$, _field);\n EXECUTE q INTO literal USING _item;\n RETURN literal;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n\nCREATE OR REPLACE FUNCTION get_token_record(IN _token text, OUT prev BOOLEAN, OUT item items) RETURNS RECORD AS $$\nDECLARE\n _itemid text := _token;\n _collectionid text;\nBEGIN\n IF _token IS NULL THEN\n RETURN;\n END IF;\n RAISE NOTICE 'Looking for token: %', _token;\n prev := FALSE;\n IF _token ILIKE 'prev:%' THEN\n _itemid := replace(_token, 'prev:','');\n prev := TRUE;\n ELSIF _token ILIKE 'next:%' THEN\n _itemid := replace(_token, 'next:', '');\n END IF;\n SELECT id INTO _collectionid FROM collections WHERE _itemid LIKE concat(id,':%');\n IF FOUND THEN\n _itemid := replace(_itemid, concat(_collectionid,':'), '');\n SELECT * INTO item FROM items WHERE id=_itemid AND collection=_collectionid;\n ELSE\n SELECT * INTO item FROM items WHERE id=_itemid;\n END IF;\n IF item IS NULL THEN\n RAISE EXCEPTION 'Could not find item using token: % item: % collection: %', _token, _itemid, _collectionid;\n END IF;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\n\nCREATE OR REPLACE FUNCTION get_token_filter(\n _sortby jsonb DEFAULT '[{\"field\":\"datetime\",\"direction\":\"desc\"}]'::jsonb,\n token_item items DEFAULT NULL,\n prev boolean DEFAULT FALSE,\n inclusive boolean DEFAULT FALSE\n) RETURNS text AS $$\nDECLARE\n ltop text := '<';\n gtop text := '>';\n dir text;\n sort record;\n orfilter text := '';\n orfilters text[] := '{}'::text[];\n andfilters text[] := '{}'::text[];\n output text;\n token_where text;\nBEGIN\n IF _sortby IS NULL OR _sortby = '[]'::jsonb THEN\n _sortby := '[{\"field\":\"datetime\",\"direction\":\"desc\"}]'::jsonb;\n END IF;\n _sortby := _sortby || jsonb_build_object('field','id','direction',_sortby->0->>'direction');\n RAISE NOTICE 'Getting Token Filter. % %', _sortby, token_item;\n IF inclusive THEN\n orfilters := orfilters || format('( id=%L AND collection=%L )' , token_item.id, token_item.collection);\n END IF;\n\n FOR sort IN\n WITH s1 AS (\n SELECT\n _row,\n (queryable(value->>'field')).expression as _field,\n (value->>'field' = 'id') as _isid,\n get_sort_dir(value) as _dir\n FROM jsonb_array_elements(_sortby)\n WITH ORDINALITY AS t(value, _row)\n )\n SELECT\n _row,\n _field,\n _dir,\n get_token_val_str(_field, token_item) as _val\n FROM s1\n WHERE _row <= (SELECT min(_row) FROM s1 WHERE _isid)\n LOOP\n orfilter := NULL;\n RAISE NOTICE 'SORT: %', sort;\n IF sort._val IS NOT NULL AND ((prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC')) THEN\n orfilter := format($f$(\n (%s %s %s) OR (%s IS NULL)\n )$f$,\n sort._field,\n ltop,\n sort._val,\n sort._val\n );\n ELSIF sort._val IS NULL AND ((prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC')) THEN\n RAISE NOTICE '< but null';\n orfilter := format('%s IS NOT NULL', sort._field);\n ELSIF sort._val IS NULL THEN\n RAISE NOTICE '> but null';\n ELSE\n orfilter := format($f$(\n (%s %s %s) OR (%s IS NULL)\n )$f$,\n sort._field,\n gtop,\n sort._val,\n sort._field\n );\n END IF;\n RAISE NOTICE 'ORFILTER: %', orfilter;\n\n IF orfilter IS NOT NULL THEN\n IF sort._row = 1 THEN\n orfilters := orfilters || orfilter;\n ELSE\n orfilters := orfilters || format('(%s AND %s)', array_to_string(andfilters, ' AND '), orfilter);\n END IF;\n END IF;\n IF sort._val IS NOT NULL THEN\n andfilters := andfilters || format('%s = %s', sort._field, sort._val);\n ELSE\n andfilters := andfilters || format('%s IS NULL', sort._field);\n END IF;\n END LOOP;\n\n output := array_to_string(orfilters, ' OR ');\n\n token_where := concat('(',coalesce(output,'true'),')');\n IF trim(token_where) = '' THEN\n token_where := NULL;\n END IF;\n RAISE NOTICE 'TOKEN_WHERE: %',token_where;\n RETURN token_where;\n END;\n$$ LANGUAGE PLPGSQL SET transform_null_equals TO TRUE\n;\n\nCREATE OR REPLACE FUNCTION search_tohash(jsonb) RETURNS jsonb AS $$\n SELECT $1 - '{token,limit,context,includes,excludes}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION search_hash(jsonb, jsonb) RETURNS text AS $$\n SELECT md5(concat(search_tohash($1)::text,$2::text));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TABLE IF NOT EXISTS searches(\n hash text GENERATED ALWAYS AS (search_hash(search, metadata)) STORED PRIMARY KEY,\n search jsonb NOT NULL,\n _where text,\n orderby text,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n metadata jsonb DEFAULT '{}'::jsonb NOT NULL\n);\n\nCREATE TABLE IF NOT EXISTS search_wheres(\n id bigint generated always as identity primary key,\n _where text NOT NULL,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n statslastupdated timestamptz,\n estimated_count bigint,\n estimated_cost float,\n time_to_estimate float,\n total_count bigint,\n time_to_count float,\n partitions text[]\n);\n\nCREATE INDEX IF NOT EXISTS search_wheres_partitions ON search_wheres USING GIN (partitions);\nCREATE UNIQUE INDEX IF NOT EXISTS search_wheres_where ON search_wheres ((md5(_where)));\n\nCREATE OR REPLACE FUNCTION where_stats(inwhere text, updatestats boolean default false, conf jsonb default null) RETURNS search_wheres AS $$\nDECLARE\n t timestamptz;\n i interval;\n explain_json jsonb;\n partitions text[];\n sw search_wheres%ROWTYPE;\n inwhere_hash text := md5(inwhere);\n _context text := lower(context(conf));\n _stats_ttl interval := context_stats_ttl(conf);\n _estimated_cost float := context_estimated_cost(conf);\n _estimated_count int := context_estimated_count(conf);\n ro bool := pgstac.readonly(conf);\nBEGIN\n IF ro THEN\n updatestats := FALSE;\n END IF;\n\n IF _context = 'off' THEN\n sw._where := inwhere;\n return sw;\n END IF;\n\n SELECT * INTO sw FROM search_wheres WHERE md5(_where)=inwhere_hash FOR UPDATE;\n\n -- Update statistics if explicitly set, if statistics do not exist, or statistics ttl has expired\n IF NOT updatestats THEN\n RAISE NOTICE 'Checking if update is needed for: % .', inwhere;\n RAISE NOTICE 'Stats Last Updated: %', sw.statslastupdated;\n RAISE NOTICE 'TTL: %, Age: %', _stats_ttl, now() - sw.statslastupdated;\n RAISE NOTICE 'Context: %, Existing Total: %', _context, sw.total_count;\n IF\n (\n sw.statslastupdated IS NULL\n OR (now() - sw.statslastupdated) > _stats_ttl\n OR (context(conf) != 'off' AND sw.total_count IS NULL)\n ) AND NOT ro\n THEN\n updatestats := TRUE;\n END IF;\n END IF;\n\n sw._where := inwhere;\n sw.lastused := now();\n sw.usecount := coalesce(sw.usecount,0) + 1;\n\n IF NOT updatestats THEN\n UPDATE search_wheres SET\n lastused = sw.lastused,\n usecount = sw.usecount\n WHERE md5(_where) = inwhere_hash\n RETURNING * INTO sw\n ;\n RETURN sw;\n END IF;\n\n -- Use explain to get estimated count/cost and a list of the partitions that would be hit by the query\n t := clock_timestamp();\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s', inwhere)\n INTO explain_json;\n RAISE NOTICE 'Time for just the explain: %', clock_timestamp() - t;\n i := clock_timestamp() - t;\n\n sw.statslastupdated := now();\n sw.estimated_count := explain_json->0->'Plan'->'Plan Rows';\n sw.estimated_cost := explain_json->0->'Plan'->'Total Cost';\n sw.time_to_estimate := extract(epoch from i);\n\n RAISE NOTICE 'ESTIMATED_COUNT: % < %', sw.estimated_count, _estimated_count;\n RAISE NOTICE 'ESTIMATED_COST: % < %', sw.estimated_cost, _estimated_cost;\n\n -- Do a full count of rows if context is set to on or if auto is set and estimates are low enough\n IF\n _context = 'on'\n OR\n ( _context = 'auto' AND\n (\n sw.estimated_count < _estimated_count\n AND\n sw.estimated_cost < _estimated_cost\n )\n )\n THEN\n t := clock_timestamp();\n RAISE NOTICE 'Calculating actual count...';\n EXECUTE format(\n 'SELECT count(*) FROM items WHERE %s',\n inwhere\n ) INTO sw.total_count;\n i := clock_timestamp() - t;\n RAISE NOTICE 'Actual Count: % -- %', sw.total_count, i;\n sw.time_to_count := extract(epoch FROM i);\n ELSE\n sw.total_count := NULL;\n sw.time_to_count := NULL;\n END IF;\n\n IF NOT ro THEN\n INSERT INTO search_wheres\n (_where, lastused, usecount, statslastupdated, estimated_count, estimated_cost, time_to_estimate, partitions, total_count, time_to_count)\n SELECT sw._where, sw.lastused, sw.usecount, sw.statslastupdated, sw.estimated_count, sw.estimated_cost, sw.time_to_estimate, sw.partitions, sw.total_count, sw.time_to_count\n ON CONFLICT ((md5(_where)))\n DO UPDATE\n SET\n lastused = sw.lastused,\n usecount = sw.usecount,\n statslastupdated = sw.statslastupdated,\n estimated_count = sw.estimated_count,\n estimated_cost = sw.estimated_cost,\n time_to_estimate = sw.time_to_estimate,\n total_count = sw.total_count,\n time_to_count = sw.time_to_count\n ;\n END IF;\n RETURN sw;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION search_query(\n _search jsonb = '{}'::jsonb,\n updatestats boolean = false,\n _metadata jsonb = '{}'::jsonb\n) RETURNS searches AS $$\nDECLARE\n search searches%ROWTYPE;\n pexplain jsonb;\n t timestamptz;\n i interval;\n _hash text := search_hash(_search, _metadata);\n doupdate boolean := FALSE;\n insertfound boolean := FALSE;\n ro boolean := pgstac.readonly();\nBEGIN\n IF ro THEN\n updatestats := FALSE;\n END IF;\n\n SELECT * INTO search FROM searches\n WHERE hash=_hash;\n\n search.hash := _hash;\n\n -- Calculate the where clause if not already calculated\n IF search._where IS NULL THEN\n search._where := stac_search_to_where(_search);\n ELSE\n doupdate := TRUE;\n END IF;\n\n -- Calculate the order by clause if not already calculated\n IF search.orderby IS NULL THEN\n search.orderby := sort_sqlorderby(_search);\n ELSE\n doupdate := TRUE;\n END IF;\n\n PERFORM where_stats(search._where, updatestats, _search->'conf');\n\n IF NOT ro THEN\n IF NOT doupdate THEN\n INSERT INTO searches (search, _where, orderby, lastused, usecount, metadata)\n VALUES (_search, search._where, search.orderby, clock_timestamp(), 1, _metadata)\n ON CONFLICT (hash) DO NOTHING RETURNING * INTO search;\n IF FOUND THEN\n RETURN search;\n END IF;\n END IF;\n\n UPDATE searches\n SET\n lastused=clock_timestamp(),\n usecount=usecount+1\n WHERE hash=(\n SELECT hash FROM searches\n WHERE hash=_hash\n FOR UPDATE SKIP LOCKED\n );\n IF NOT FOUND THEN\n RAISE NOTICE 'Did not update stats for % due to lock. (This is generally OK)', _search;\n END IF;\n END IF;\n\n RETURN search;\n\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE OR REPLACE FUNCTION search_rows(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN partitions text[] DEFAULT NULL,\n IN _limit int DEFAULT 10\n) RETURNS SETOF items AS $$\nDECLARE\n base_query text;\n query text;\n sdate timestamptz;\n edate timestamptz;\n n int;\n records_left int := _limit;\n timer timestamptz := clock_timestamp();\n full_timer timestamptz := clock_timestamp();\nBEGIN\nIF _where IS NULL OR trim(_where) = '' THEN\n _where = ' TRUE ';\nEND IF;\nRAISE NOTICE 'Getting chunks for % %', _where, _orderby;\n\nbase_query := $q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n LIMIT %L\n$q$;\n\nIF _orderby ILIKE 'datetime d%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 DESC LOOP\n RAISE NOTICE 'Running Query for % to %. %', sdate, edate, age_ms(full_timer);\n query := format(\n base_query,\n sdate,\n edate,\n _where,\n _orderby,\n records_left\n );\n RAISE LOG 'QUERY: %', query;\n timer := clock_timestamp();\n RETURN QUERY EXECUTE query;\n\n GET DIAGNOSTICS n = ROW_COUNT;\n records_left := records_left - n;\n RAISE NOTICE 'Returned %/% Rows From % to %. % to go. Time: %ms', n, _limit, sdate, edate, records_left, age_ms(timer);\n timer := clock_timestamp();\n IF records_left <= 0 THEN\n RAISE NOTICE 'SEARCH_ROWS TOOK %ms', age_ms(full_timer);\n RETURN;\n END IF;\n END LOOP;\nELSIF _orderby ILIKE 'datetime a%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 ASC LOOP\n RAISE NOTICE 'Running Query for % to %. %', sdate, edate, age_ms(full_timer);\n query := format(\n base_query,\n sdate,\n edate,\n _where,\n _orderby,\n records_left\n );\n RAISE LOG 'QUERY: %', query;\n timer := clock_timestamp();\n RETURN QUERY EXECUTE query;\n\n GET DIAGNOSTICS n = ROW_COUNT;\n records_left := records_left - n;\n RAISE NOTICE 'Returned %/% Rows From % to %. % to go. Time: %ms', n, _limit, sdate, edate, records_left, age_ms(timer);\n timer := clock_timestamp();\n IF records_left <= 0 THEN\n RAISE NOTICE 'SEARCH_ROWS TOOK %ms', age_ms(full_timer);\n RETURN;\n END IF;\n END LOOP;\nELSE\n query := format($q$\n SELECT * FROM items\n WHERE %s\n ORDER BY %s\n LIMIT %L\n $q$, _where, _orderby, _limit\n );\n RAISE LOG 'QUERY: %', query;\n timer := clock_timestamp();\n RETURN QUERY EXECUTE query;\n RAISE NOTICE 'FULL QUERY TOOK %ms', age_ms(timer);\nEND IF;\nRAISE NOTICE 'SEARCH_ROWS TOOK %ms', age_ms(full_timer);\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE UNLOGGED TABLE format_item_cache(\n id text,\n collection text,\n fields text,\n hydrated bool,\n output jsonb,\n lastused timestamptz DEFAULT now(),\n usecount int DEFAULT 1,\n timetoformat float,\n PRIMARY KEY (collection, id, fields, hydrated)\n);\nCREATE INDEX ON format_item_cache (lastused);\n\nCREATE OR REPLACE FUNCTION format_item(_item items, _fields jsonb DEFAULT '{}', _hydrated bool DEFAULT TRUE) RETURNS jsonb AS $$\nDECLARE\n cache bool := get_setting_bool('format_cache');\n _output jsonb := null;\n t timestamptz := clock_timestamp();\nBEGIN\n IF cache THEN\n SELECT output INTO _output FROM format_item_cache\n WHERE id=_item.id AND collection=_item.collection AND fields=_fields::text AND hydrated=_hydrated;\n END IF;\n IF _output IS NULL THEN\n IF _hydrated THEN\n _output := content_hydrate(_item, _fields);\n ELSE\n _output := content_nonhydrated(_item, _fields);\n END IF;\n END IF;\n IF cache THEN\n INSERT INTO format_item_cache (id, collection, fields, hydrated, output, timetoformat)\n VALUES (_item.id, _item.collection, _fields::text, _hydrated, _output, age_ms(t))\n ON CONFLICT(collection, id, fields, hydrated) DO\n UPDATE\n SET lastused=now(), usecount = format_item_cache.usecount + 1\n ;\n END IF;\n RETURN _output;\n\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION search(_search jsonb = '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n searches searches%ROWTYPE;\n _where text;\n orderby text;\n search_where search_wheres%ROWTYPE;\n total_count bigint;\n token record;\n token_prev boolean;\n token_item items%ROWTYPE;\n token_where text;\n full_where text;\n init_ts timestamptz := clock_timestamp();\n timer timestamptz := clock_timestamp();\n hydrate bool := NOT (_search->'conf'->>'nohydrate' IS NOT NULL AND (_search->'conf'->>'nohydrate')::boolean = true);\n prev text;\n next text;\n context jsonb;\n collection jsonb;\n out_records jsonb;\n out_len int;\n _limit int := coalesce((_search->>'limit')::int, 10);\n _querylimit int;\n _fields jsonb := coalesce(_search->'fields', '{}'::jsonb);\n has_prev boolean := FALSE;\n has_next boolean := FALSE;\nBEGIN\n searches := search_query(_search);\n _where := searches._where;\n orderby := searches.orderby;\n search_where := where_stats(_where);\n total_count := coalesce(search_where.total_count, search_where.estimated_count);\n RAISE NOTICE 'SEARCH:TOKEN: %', _search->>'token';\n token := get_token_record(_search->>'token');\n RAISE NOTICE '***TOKEN: %', token;\n _querylimit := _limit + 1;\n IF token IS NOT NULL THEN\n token_prev := token.prev;\n token_item := token.item;\n token_where := get_token_filter(_search->'sortby', token_item, token_prev, FALSE);\n RAISE LOG 'TOKEN_WHERE: % (%ms from search start)', token_where, age_ms(timer);\n IF token_prev THEN -- if we are using a prev token, we know has_next is true\n RAISE LOG 'There is a previous token, so automatically setting has_next to true';\n has_next := TRUE;\n orderby := sort_sqlorderby(_search, TRUE);\n ELSE\n RAISE LOG 'There is a next token, so automatically setting has_prev to true';\n has_prev := TRUE;\n\n END IF;\n ELSE -- if there was no token, we know there is no prev\n RAISE LOG 'There is no token, so we know there is no prev. setting has_prev to false';\n has_prev := FALSE;\n END IF;\n\n full_where := concat_ws(' AND ', _where, token_where);\n RAISE NOTICE 'FULL WHERE CLAUSE: %', full_where;\n RAISE NOTICE 'Time to get counts and build query %', age_ms(timer);\n timer := clock_timestamp();\n\n IF hydrate THEN\n RAISE NOTICE 'Getting hydrated data.';\n ELSE\n RAISE NOTICE 'Getting non-hydrated data.';\n END IF;\n RAISE NOTICE 'CACHE SET TO %', get_setting_bool('format_cache');\n RAISE NOTICE 'Time to set hydration/formatting %', age_ms(timer);\n timer := clock_timestamp();\n SELECT jsonb_agg(format_item(i, _fields, hydrate)) INTO out_records\n FROM search_rows(\n full_where,\n orderby,\n search_where.partitions,\n _querylimit\n ) as i;\n\n RAISE NOTICE 'Time to fetch rows %', age_ms(timer);\n timer := clock_timestamp();\n\n\n IF token_prev THEN\n out_records := flip_jsonb_array(out_records);\n END IF;\n\n RAISE NOTICE 'Query returned % records.', jsonb_array_length(out_records);\n RAISE LOG 'TOKEN: % %', token_item.id, token_item.collection;\n RAISE LOG 'RECORD_1: % %', out_records->0->>'id', out_records->0->>'collection';\n RAISE LOG 'RECORD-1: % %', out_records->-1->>'id', out_records->-1->>'collection';\n\n -- REMOVE records that were from our token\n IF out_records->0->>'id' = token_item.id AND out_records->0->>'collection' = token_item.collection THEN\n out_records := out_records - 0;\n ELSIF out_records->-1->>'id' = token_item.id AND out_records->-1->>'collection' = token_item.collection THEN\n out_records := out_records - -1;\n END IF;\n\n out_len := jsonb_array_length(out_records);\n\n IF out_len = _limit + 1 THEN\n IF token_prev THEN\n has_prev := TRUE;\n out_records := out_records - 0;\n ELSE\n has_next := TRUE;\n out_records := out_records - -1;\n END IF;\n END IF;\n\n IF has_next THEN\n next := concat(out_records->-1->>'collection', ':', out_records->-1->>'id');\n RAISE NOTICE 'HAS NEXT | %', next;\n END IF;\n\n IF has_prev THEN\n prev := concat(out_records->0->>'collection', ':', out_records->0->>'id');\n RAISE NOTICE 'HAS PREV | %', prev;\n END IF;\n\n RAISE NOTICE 'Time to get prev/next %', age_ms(timer);\n timer := clock_timestamp();\n\n IF context(_search->'conf') != 'off' THEN\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'matched', total_count,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\n ELSE\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\n END IF;\n\n collection := jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb),\n 'next', next,\n 'prev', prev,\n 'context', context\n );\n\n IF get_setting_bool('timing', _search->'conf') THEN\n collection = collection || jsonb_build_object('timing', age_ms(init_ts));\n END IF;\n\n RAISE NOTICE 'Time to build final json %', age_ms(timer);\n timer := clock_timestamp();\n\n RAISE NOTICE 'Total Time: %', age_ms(current_timestamp);\n RAISE NOTICE 'RETURNING % records. NEXT: %. PREV: %', collection->'context'->>'returned', collection->>'next', collection->>'prev';\n RETURN collection;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION search_cursor(_search jsonb = '{}'::jsonb) RETURNS refcursor AS $$\nDECLARE\n curs refcursor;\n searches searches%ROWTYPE;\n _where text;\n _orderby text;\n q text;\n\nBEGIN\n searches := search_query(_search);\n _where := searches._where;\n _orderby := searches.orderby;\n\n OPEN curs FOR\n WITH p AS (\n SELECT * FROM partition_queries(_where, _orderby) p\n )\n SELECT\n CASE WHEN EXISTS (SELECT 1 FROM p) THEN\n (SELECT format($q$\n SELECT * FROM (\n %s\n ) total\n $q$,\n string_agg(\n format($q$ SELECT * FROM ( %s ) AS sub $q$, p),\n '\n UNION ALL\n '\n )\n ))\n ELSE NULL\n END FROM p;\n RETURN curs;\nEND;\n$$ LANGUAGE PLPGSQL;\nCREATE OR REPLACE VIEW collections_asitems AS\nSELECT\n id,\n geometry,\n 'collections' AS collection,\n datetime,\n end_datetime,\n jsonb_build_object(\n 'properties', content - '{links,assets,stac_version,stac_extensions}',\n 'links', content->'links',\n 'assets', content->'assets',\n 'stac_version', content->'stac_version',\n 'stac_extensions', content->'stac_extensions'\n ) AS content,\n content as collectionjson\nFROM collections;\n\n\nCREATE OR REPLACE FUNCTION collection_search_matched(\n IN _search jsonb DEFAULT '{}'::jsonb,\n OUT matched bigint\n) RETURNS bigint AS $$\nDECLARE\n _where text := stac_search_to_where(_search);\nBEGIN\n EXECUTE format(\n $query$\n SELECT\n count(*)\n FROM\n collections_asitems\n WHERE %s\n ;\n $query$,\n _where\n ) INTO matched;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION collection_search_rows(\n _search jsonb DEFAULT '{}'::jsonb\n) RETURNS SETOF jsonb AS $$\nDECLARE\n _where text := stac_search_to_where(_search);\n _limit int := coalesce((_search->>'limit')::int, 10);\n _fields jsonb := coalesce(_search->'fields', '{}'::jsonb);\n _orderby text;\n _offset int := COALESCE((_search->>'offset')::int, 0);\nBEGIN\n _orderby := sort_sqlorderby(\n jsonb_build_object(\n 'sortby',\n coalesce(\n _search->'sortby',\n '[{\"field\": \"id\", \"direction\": \"asc\"}]'::jsonb\n )\n )\n );\n RETURN QUERY EXECUTE format(\n $query$\n SELECT\n jsonb_fields(collectionjson, %L) as c\n FROM\n collections_asitems\n WHERE %s\n ORDER BY %s\n LIMIT %L\n OFFSET %L\n ;\n $query$,\n _fields,\n _where,\n _orderby,\n _limit,\n _offset\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION collection_search(\n _search jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\nDECLARE\n out_records jsonb;\n number_matched bigint := collection_search_matched(_search);\n number_returned bigint;\n _limit int := coalesce((_search->>'limit')::float::int, 10);\n _offset int := coalesce((_search->>'offset')::float::int, 0);\n links jsonb := '[]';\n ret jsonb;\n base_url text:= concat(rtrim(base_url(_search->'conf'),'/'), '/collections');\n prevoffset int;\n nextoffset int;\nBEGIN\n SELECT\n coalesce(jsonb_agg(c), '[]')\n INTO out_records\n FROM collection_search_rows(_search) c;\n\n number_returned := jsonb_array_length(out_records);\n\n IF _limit <= number_matched THEN --need to have paging links\n nextoffset := least(_offset + _limit, number_matched - 1);\n prevoffset := greatest(_offset - _limit, 0);\n IF _offset = 0 THEN -- no previous paging\n\n links := jsonb_build_array(\n jsonb_build_object(\n 'rel', 'next',\n 'type', 'application/json',\n 'method', 'GET' ,\n 'href', base_url,\n 'body', jsonb_build_object('offset', nextoffset),\n 'merge', TRUE\n )\n );\n ELSE\n links := jsonb_build_array(\n jsonb_build_object(\n 'rel', 'prev',\n 'type', 'application/json',\n 'method', 'GET' ,\n 'href', base_url,\n 'body', jsonb_build_object('offset', prevoffset),\n 'merge', TRUE\n ),\n jsonb_build_object(\n 'rel', 'next',\n 'type', 'application/json',\n 'method', 'GET' ,\n 'href', base_url,\n 'body', jsonb_build_object('offset', nextoffset),\n 'merge', TRUE\n )\n );\n END IF;\n END IF;\n\n ret := jsonb_build_object(\n 'collections', out_records,\n 'context', jsonb_build_object(\n 'limit', _limit,\n 'matched', number_matched,\n 'returned', number_returned\n ),\n 'links', links\n );\n RETURN ret;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION tileenvelope(zoom int, x int, y int) RETURNS geometry AS $$\nWITH t AS (\n SELECT\n 20037508.3427892 as merc_max,\n -20037508.3427892 as merc_min,\n (2 * 20037508.3427892) / (2 ^ zoom) as tile_size\n)\nSELECT st_makeenvelope(\n merc_min + (tile_size * x),\n merc_max - (tile_size * (y + 1)),\n merc_min + (tile_size * (x + 1)),\n merc_max - (tile_size * y),\n 3857\n) FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;DROP FUNCTION IF EXISTS mercgrid;\n\n\nCREATE OR REPLACE FUNCTION ftime() RETURNS interval as $$\nSELECT age(clock_timestamp(), transaction_timestamp());\n$$ LANGUAGE SQL;\nSET SEARCH_PATH to pgstac, public;\n\nDROP FUNCTION IF EXISTS geometrysearch;\nCREATE OR REPLACE FUNCTION geometrysearch(\n IN geom geometry,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE, -- Return as soon as the passed in geometry is full covered\n IN skipcovered boolean DEFAULT TRUE -- Skip any items that would show up completely under the previous items\n) RETURNS jsonb AS $$\nDECLARE\n search searches%ROWTYPE;\n curs refcursor;\n _where text;\n query text;\n iter_record items%ROWTYPE;\n out_records jsonb := '{}'::jsonb[];\n exit_flag boolean := FALSE;\n counter int := 1;\n scancounter int := 1;\n remaining_limit int := _scanlimit;\n tilearea float;\n unionedgeom geometry;\n clippedgeom geometry;\n unionedgeom_area float := 0;\n prev_area float := 0;\n excludes text[];\n includes text[];\n\nBEGIN\n DROP TABLE IF EXISTS pgstac_results;\n CREATE TEMP TABLE pgstac_results (content jsonb) ON COMMIT DROP;\n\n -- If the passed in geometry is not an area set exitwhenfull and skipcovered to false\n IF ST_GeometryType(geom) !~* 'polygon' THEN\n RAISE NOTICE 'GEOMETRY IS NOT AN AREA';\n skipcovered = FALSE;\n exitwhenfull = FALSE;\n END IF;\n\n -- If skipcovered is true then you will always want to exit when the passed in geometry is full\n IF skipcovered THEN\n exitwhenfull := TRUE;\n END IF;\n\n SELECT * INTO search FROM searches WHERE hash=queryhash;\n\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Search with Query Hash % Not Found', queryhash;\n END IF;\n\n tilearea := st_area(geom);\n _where := format('%s AND st_intersects(geometry, %L::geometry)', search._where, geom);\n\n\n FOR query IN SELECT * FROM partition_queries(_where, search.orderby) LOOP\n query := format('%s LIMIT %L', query, remaining_limit);\n RAISE NOTICE '%', query;\n OPEN curs FOR EXECUTE query;\n LOOP\n FETCH curs INTO iter_record;\n EXIT WHEN NOT FOUND;\n IF exitwhenfull OR skipcovered THEN -- If we are not using exitwhenfull or skipcovered, we do not need to do expensive geometry operations\n clippedgeom := st_intersection(geom, iter_record.geometry);\n\n IF unionedgeom IS NULL THEN\n unionedgeom := clippedgeom;\n ELSE\n unionedgeom := st_union(unionedgeom, clippedgeom);\n END IF;\n\n unionedgeom_area := st_area(unionedgeom);\n\n IF skipcovered AND prev_area = unionedgeom_area THEN\n scancounter := scancounter + 1;\n CONTINUE;\n END IF;\n\n prev_area := unionedgeom_area;\n\n RAISE NOTICE '% % % %', unionedgeom_area/tilearea, counter, scancounter, ftime();\n END IF;\n RAISE NOTICE '% %', iter_record, content_hydrate(iter_record, fields);\n INSERT INTO pgstac_results (content) VALUES (content_hydrate(iter_record, fields));\n\n IF counter >= _limit\n OR scancounter > _scanlimit\n OR ftime() > _timelimit\n OR (exitwhenfull AND unionedgeom_area >= tilearea)\n THEN\n exit_flag := TRUE;\n EXIT;\n END IF;\n counter := counter + 1;\n scancounter := scancounter + 1;\n\n END LOOP;\n CLOSE curs;\n EXIT WHEN exit_flag;\n remaining_limit := _scanlimit - scancounter;\n END LOOP;\n\n SELECT jsonb_agg(content) INTO out_records FROM pgstac_results WHERE content IS NOT NULL;\n\n RETURN jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb)\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDROP FUNCTION IF EXISTS geojsonsearch;\nCREATE OR REPLACE FUNCTION geojsonsearch(\n IN geojson jsonb,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_geomfromgeojson(geojson),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS xyzsearch;\nCREATE OR REPLACE FUNCTION xyzsearch(\n IN _x int,\n IN _y int,\n IN _z int,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_transform(tileenvelope(_z, _x, _y), 4326),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE PROCEDURE analyze_items() AS $$\nDECLARE\n q text;\n timeout_ts timestamptz;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n SELECT format('ANALYZE (VERBOSE, SKIP_LOCKED) %I;', relname) INTO q\n FROM pg_stat_user_tables\n WHERE relname like '_item%' AND (n_mod_since_analyze>0 OR last_analyze IS NULL) LIMIT 1;\n IF NOT FOUND THEN\n EXIT;\n END IF;\n RAISE NOTICE '%', q;\n EXECUTE q;\n COMMIT;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE PROCEDURE validate_constraints() AS $$\nDECLARE\n q text;\nBEGIN\n FOR q IN\n SELECT\n FORMAT(\n 'ALTER TABLE %I.%I VALIDATE CONSTRAINT %I;',\n nsp.nspname,\n cls.relname,\n con.conname\n )\n\n FROM pg_constraint AS con\n JOIN pg_class AS cls\n ON con.conrelid = cls.oid\n JOIN pg_namespace AS nsp\n ON cls.relnamespace = nsp.oid\n WHERE convalidated = FALSE AND contype in ('c','f')\n AND nsp.nspname = 'pgstac'\n LOOP\n RAISE NOTICE '%', q;\n PERFORM run_or_queue(q);\n COMMIT;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION collection_extent(_collection text, runupdate boolean default false) RETURNS jsonb AS $$\nDECLARE\n geom_extent geometry;\n mind timestamptz;\n maxd timestamptz;\n extent jsonb;\nBEGIN\n IF runupdate THEN\n PERFORM update_partition_stats_q(partition)\n FROM partitions_view WHERE collection=_collection;\n END IF;\n SELECT\n min(lower(dtrange)),\n max(upper(edtrange)),\n st_extent(spatial)\n INTO\n mind,\n maxd,\n geom_extent\n FROM partitions_view\n WHERE collection=_collection;\n\n IF geom_extent IS NOT NULL AND mind IS NOT NULL AND maxd IS NOT NULL THEN\n extent := jsonb_build_object(\n 'extent', jsonb_build_object(\n 'spatial', jsonb_build_object(\n 'bbox', to_jsonb(array[array[st_xmin(geom_extent), st_ymin(geom_extent), st_xmax(geom_extent), st_ymax(geom_extent)]])\n ),\n 'temporal', jsonb_build_object(\n 'interval', to_jsonb(array[array[mind, maxd]])\n )\n )\n );\n RETURN extent;\n END IF;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('id', '{\"title\": \"Item ID\",\"description\": \"Item identifier\",\"$ref\": \"https://schemas.stacspec.org/v1.0.0/item-spec/json-schema/item.json#/definitions/core/allOf/2/properties/id\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('geometry', '{\"title\": \"Item Geometry\",\"description\": \"Item Geometry\",\"$ref\": \"https://geojson.org/schema/Feature.json\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('datetime','{\"description\": \"Datetime\",\"type\": \"string\",\"title\": \"Acquired\",\"format\": \"date-time\",\"pattern\": \"(\\\\+00:00|Z)$\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDELETE FROM queryables a USING queryables b\n WHERE a.name = b.name AND a.collection_ids IS NOT DISTINCT FROM b.collection_ids AND a.id > b.id;\n\n\nINSERT INTO pgstac_settings (name, value) VALUES\n ('context', 'off'),\n ('context_estimated_count', '100000'),\n ('context_estimated_cost', '100000'),\n ('context_stats_ttl', '1 day'),\n ('default_filter_lang', 'cql2-json'),\n ('additional_properties', 'true'),\n ('use_queue', 'false'),\n ('queue_timeout', '10 minutes'),\n ('update_collection_extent', 'false'),\n ('format_cache', 'false'),\n ('readonly', 'false')\nON CONFLICT DO NOTHING\n;\n\nALTER FUNCTION to_text COST 5000;\nALTER FUNCTION to_float COST 5000;\nALTER FUNCTION to_int COST 5000;\nALTER FUNCTION to_tstz COST 5000;\nALTER FUNCTION to_text_array COST 5000;\n\nALTER FUNCTION update_partition_stats SECURITY DEFINER;\nALTER FUNCTION partition_after_triggerfunc SECURITY DEFINER;\nALTER FUNCTION drop_table_constraints SECURITY DEFINER;\nALTER FUNCTION create_table_constraints SECURITY DEFINER;\nALTER FUNCTION check_partition SECURITY DEFINER;\nALTER FUNCTION repartition SECURITY DEFINER;\nALTER FUNCTION where_stats SECURITY DEFINER;\nALTER FUNCTION search_query SECURITY DEFINER;\nALTER FUNCTION format_item SECURITY DEFINER;\nALTER FUNCTION maintain_index SECURITY DEFINER;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nGRANT ALL ON SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac to pgstac_admin;\n\n-- pgstac_read role limited to using function apis\nGRANT EXECUTE ON FUNCTION search TO pgstac_read;\nGRANT EXECUTE ON FUNCTION search_query TO pgstac_read;\nGRANT EXECUTE ON FUNCTION item_by_id TO pgstac_read;\nGRANT EXECUTE ON FUNCTION get_item TO pgstac_read;\nGRANT SELECT ON ALL TABLES IN SCHEMA pgstac TO pgstac_read;\n\n\nGRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_ingest;\nGRANT USAGE ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_ingest;\n\nREVOKE ALL PRIVILEGES ON PROCEDURE run_queued_queries FROM public;\nGRANT ALL ON PROCEDURE run_queued_queries TO pgstac_admin;\n\nREVOKE ALL PRIVILEGES ON FUNCTION run_queued_queries_intransaction FROM public;\nGRANT ALL ON FUNCTION run_queued_queries_intransaction TO pgstac_admin;\n\nRESET ROLE;\n\nSET ROLE pgstac_ingest;\nSELECT update_partition_stats_q(partition) FROM partitions_view;\nSELECT set_version('0.8.2');\n" }, { "path": "src/pgstac/migrations/pgstac.0.8.3-0.8.4.sql", "content": "SET client_min_messages TO WARNING;\nSET SEARCH_PATH to pgstac, public;\nRESET ROLE;\nDO $$\nDECLARE\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='postgis') THEN\n CREATE EXTENSION IF NOT EXISTS postgis;\n END IF;\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='btree_gist') THEN\n CREATE EXTENSION IF NOT EXISTS btree_gist;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_admin;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_read;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_ingest;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n\nGRANT pgstac_admin TO current_user;\n\n-- Function to make sure pgstac_admin is the owner of items\nCREATE OR REPLACE FUNCTION pgstac_admin_owns() RETURNS VOID AS $$\nDECLARE\n f RECORD;\nBEGIN\n FOR f IN (\n SELECT\n concat(\n oid::regproc::text,\n '(',\n coalesce(pg_get_function_identity_arguments(oid),''),\n ')'\n ) AS name,\n CASE prokind WHEN 'f' THEN 'FUNCTION' WHEN 'p' THEN 'PROCEDURE' WHEN 'a' THEN 'AGGREGATE' END as typ\n FROM pg_proc\n WHERE\n pronamespace=to_regnamespace('pgstac')\n AND proowner != to_regrole('pgstac_admin')\n AND proname NOT LIKE 'pg_stat%'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n FOR f IN (\n SELECT\n oid::regclass::text as name,\n CASE relkind\n WHEN 'i' THEN 'INDEX'\n WHEN 'I' THEN 'INDEX'\n WHEN 'p' THEN 'TABLE'\n WHEN 'r' THEN 'TABLE'\n WHEN 'v' THEN 'VIEW'\n WHEN 'S' THEN 'SEQUENCE'\n ELSE NULL\n END as typ\n FROM pg_class\n WHERE relnamespace=to_regnamespace('pgstac') and relowner != to_regrole('pgstac_admin') AND relkind IN ('r','p','v','S') AND relname NOT LIKE 'pg_stat'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\nSELECT pgstac_admin_owns();\n\nCREATE SCHEMA IF NOT EXISTS pgstac AUTHORIZATION pgstac_admin;\n\nGRANT ALL ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_admin;\n\nALTER ROLE pgstac_admin SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_read SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_ingest SET SEARCH_PATH TO pgstac, public;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\n\nGRANT pgstac_read TO pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\n\nSET ROLE pgstac_admin;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\nRESET ROLE;\n\nSET ROLE pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\nRESET ROLE;\n\nSET SEARCH_PATH TO pgstac, public;\nSET ROLE pgstac_admin;\n\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS analyze_items;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS validate_constraints;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n-- Install these idempotently as migrations do not put them before trying to modify the collections table\n\n\nCREATE OR REPLACE FUNCTION collection_geom(content jsonb)\nRETURNS geometry AS $$\n WITH box AS (SELECT content->'extent'->'spatial'->'bbox'->0 as box)\n SELECT\n st_makeenvelope(\n (box->>0)::float,\n (box->>1)::float,\n (box->>2)::float,\n (box->>3)::float,\n 4326\n )\n FROM box;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION collection_datetime(content jsonb)\nRETURNS timestamptz AS $$\n SELECT\n CASE\n WHEN\n (content->'extent'->'temporal'->'interval'->0->>0) IS NULL\n THEN '-infinity'::timestamptz\n ELSE\n (content->'extent'->'temporal'->'interval'->0->>0)::timestamptz\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION collection_enddatetime(content jsonb)\nRETURNS timestamptz AS $$\n SELECT\n CASE\n WHEN\n (content->'extent'->'temporal'->'interval'->0->>1) IS NULL\n THEN 'infinity'::timestamptz\n ELSE\n (content->'extent'->'temporal'->'interval'->0->>1)::timestamptz\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n-- BEGIN migra calculated SQL\n-- END migra calculated SQL\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('id', '{\"title\": \"Item ID\",\"description\": \"Item identifier\",\"$ref\": \"https://schemas.stacspec.org/v1.0.0/item-spec/json-schema/item.json#/definitions/core/allOf/2/properties/id\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('geometry', '{\"title\": \"Item Geometry\",\"description\": \"Item Geometry\",\"$ref\": \"https://geojson.org/schema/Feature.json\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('datetime','{\"description\": \"Datetime\",\"type\": \"string\",\"title\": \"Acquired\",\"format\": \"date-time\",\"pattern\": \"(\\\\+00:00|Z)$\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDELETE FROM queryables a USING queryables b\n WHERE a.name = b.name AND a.collection_ids IS NOT DISTINCT FROM b.collection_ids AND a.id > b.id;\n\n\nINSERT INTO pgstac_settings (name, value) VALUES\n ('context', 'off'),\n ('context_estimated_count', '100000'),\n ('context_estimated_cost', '100000'),\n ('context_stats_ttl', '1 day'),\n ('default_filter_lang', 'cql2-json'),\n ('additional_properties', 'true'),\n ('use_queue', 'false'),\n ('queue_timeout', '10 minutes'),\n ('update_collection_extent', 'false'),\n ('format_cache', 'false'),\n ('readonly', 'false')\nON CONFLICT DO NOTHING\n;\n\nALTER FUNCTION to_text COST 5000;\nALTER FUNCTION to_float COST 5000;\nALTER FUNCTION to_int COST 5000;\nALTER FUNCTION to_tstz COST 5000;\nALTER FUNCTION to_text_array COST 5000;\n\nALTER FUNCTION update_partition_stats SECURITY DEFINER;\nALTER FUNCTION partition_after_triggerfunc SECURITY DEFINER;\nALTER FUNCTION drop_table_constraints SECURITY DEFINER;\nALTER FUNCTION create_table_constraints SECURITY DEFINER;\nALTER FUNCTION check_partition SECURITY DEFINER;\nALTER FUNCTION repartition SECURITY DEFINER;\nALTER FUNCTION where_stats SECURITY DEFINER;\nALTER FUNCTION search_query SECURITY DEFINER;\nALTER FUNCTION format_item SECURITY DEFINER;\nALTER FUNCTION maintain_index SECURITY DEFINER;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nGRANT ALL ON SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac to pgstac_admin;\n\n-- pgstac_read role limited to using function apis\nGRANT EXECUTE ON FUNCTION search TO pgstac_read;\nGRANT EXECUTE ON FUNCTION search_query TO pgstac_read;\nGRANT EXECUTE ON FUNCTION item_by_id TO pgstac_read;\nGRANT EXECUTE ON FUNCTION get_item TO pgstac_read;\nGRANT SELECT ON ALL TABLES IN SCHEMA pgstac TO pgstac_read;\n\n\nGRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_ingest;\nGRANT USAGE ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_ingest;\n\nREVOKE ALL PRIVILEGES ON PROCEDURE run_queued_queries FROM public;\nGRANT ALL ON PROCEDURE run_queued_queries TO pgstac_admin;\n\nREVOKE ALL PRIVILEGES ON FUNCTION run_queued_queries_intransaction FROM public;\nGRANT ALL ON FUNCTION run_queued_queries_intransaction TO pgstac_admin;\n\nRESET ROLE;\n\nSET ROLE pgstac_ingest;\nSELECT update_partition_stats_q(partition) FROM partitions_view;\nSELECT set_version('0.8.4');\n" }, { "path": "src/pgstac/migrations/pgstac.0.8.3.sql", "content": "RESET ROLE;\nDO $$\nDECLARE\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='postgis') THEN\n CREATE EXTENSION IF NOT EXISTS postgis;\n END IF;\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='btree_gist') THEN\n CREATE EXTENSION IF NOT EXISTS btree_gist;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_admin;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_read;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_ingest;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n\nGRANT pgstac_admin TO current_user;\n\n-- Function to make sure pgstac_admin is the owner of items\nCREATE OR REPLACE FUNCTION pgstac_admin_owns() RETURNS VOID AS $$\nDECLARE\n f RECORD;\nBEGIN\n FOR f IN (\n SELECT\n concat(\n oid::regproc::text,\n '(',\n coalesce(pg_get_function_identity_arguments(oid),''),\n ')'\n ) AS name,\n CASE prokind WHEN 'f' THEN 'FUNCTION' WHEN 'p' THEN 'PROCEDURE' WHEN 'a' THEN 'AGGREGATE' END as typ\n FROM pg_proc\n WHERE\n pronamespace=to_regnamespace('pgstac')\n AND proowner != to_regrole('pgstac_admin')\n AND proname NOT LIKE 'pg_stat%'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n FOR f IN (\n SELECT\n oid::regclass::text as name,\n CASE relkind\n WHEN 'i' THEN 'INDEX'\n WHEN 'I' THEN 'INDEX'\n WHEN 'p' THEN 'TABLE'\n WHEN 'r' THEN 'TABLE'\n WHEN 'v' THEN 'VIEW'\n WHEN 'S' THEN 'SEQUENCE'\n ELSE NULL\n END as typ\n FROM pg_class\n WHERE relnamespace=to_regnamespace('pgstac') and relowner != to_regrole('pgstac_admin') AND relkind IN ('r','p','v','S') AND relname NOT LIKE 'pg_stat'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\nSELECT pgstac_admin_owns();\n\nCREATE SCHEMA IF NOT EXISTS pgstac AUTHORIZATION pgstac_admin;\n\nGRANT ALL ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_admin;\n\nALTER ROLE pgstac_admin SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_read SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_ingest SET SEARCH_PATH TO pgstac, public;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\n\nGRANT pgstac_read TO pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\n\nSET ROLE pgstac_admin;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\nRESET ROLE;\n\nSET ROLE pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\nRESET ROLE;\n\nSET SEARCH_PATH TO pgstac, public;\nSET ROLE pgstac_admin;\n\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS analyze_items;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS validate_constraints;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n-- Install these idempotently as migrations do not put them before trying to modify the collections table\n\n\nCREATE OR REPLACE FUNCTION collection_geom(content jsonb)\nRETURNS geometry AS $$\n WITH box AS (SELECT content->'extent'->'spatial'->'bbox'->0 as box)\n SELECT\n st_makeenvelope(\n (box->>0)::float,\n (box->>1)::float,\n (box->>2)::float,\n (box->>3)::float,\n 4326\n )\n FROM box;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION collection_datetime(content jsonb)\nRETURNS timestamptz AS $$\n SELECT\n CASE\n WHEN\n (content->'extent'->'temporal'->'interval'->0->>0) IS NULL\n THEN '-infinity'::timestamptz\n ELSE\n (content->'extent'->'temporal'->'interval'->0->>0)::timestamptz\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION collection_enddatetime(content jsonb)\nRETURNS timestamptz AS $$\n SELECT\n CASE\n WHEN\n (content->'extent'->'temporal'->'interval'->0->>1) IS NULL\n THEN 'infinity'::timestamptz\n ELSE\n (content->'extent'->'temporal'->'interval'->0->>1)::timestamptz\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE TABLE IF NOT EXISTS migrations (\n version text PRIMARY KEY,\n datetime timestamptz DEFAULT clock_timestamp() NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION get_version() RETURNS text AS $$\n SELECT version FROM pgstac.migrations ORDER BY datetime DESC, version DESC LIMIT 1;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION set_version(text) RETURNS text AS $$\n INSERT INTO pgstac.migrations (version) VALUES ($1)\n ON CONFLICT DO NOTHING\n RETURNING version;\n$$ LANGUAGE SQL;\n\n\nCREATE TABLE IF NOT EXISTS pgstac_settings (\n name text PRIMARY KEY,\n value text NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION table_empty(text) RETURNS boolean AS $$\nDECLARE\n retval boolean;\nBEGIN\n EXECUTE format($q$\n SELECT NOT EXISTS (SELECT 1 FROM %I LIMIT 1)\n $q$,\n $1\n ) INTO retval;\n RETURN retval;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION get_setting(IN _setting text, IN conf jsonb DEFAULT NULL) RETURNS text AS $$\nSELECT COALESCE(\n nullif(conf->>_setting, ''),\n nullif(current_setting(concat('pgstac.',_setting), TRUE),''),\n nullif((SELECT value FROM pgstac.pgstac_settings WHERE name=_setting),'')\n);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_setting_bool(IN _setting text, IN conf jsonb DEFAULT NULL) RETURNS boolean AS $$\nSELECT COALESCE(\n nullif(conf->>_setting, ''),\n nullif(current_setting(concat('pgstac.',_setting), TRUE),''),\n nullif((SELECT value FROM pgstac.pgstac_settings WHERE name=_setting),''),\n 'FALSE'\n)::boolean;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION base_url(conf jsonb DEFAULT NULL) RETURNS text AS $$\n SELECT COALESCE(pgstac.get_setting('base_url', conf), '.');\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION additional_properties() RETURNS boolean AS $$\n SELECT pgstac.get_setting_bool('additional_properties');\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION readonly(conf jsonb DEFAULT NULL) RETURNS boolean AS $$\n SELECT pgstac.get_setting_bool('readonly', conf);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context(conf jsonb DEFAULT NULL) RETURNS text AS $$\n SELECT pgstac.get_setting('context', conf);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context_estimated_count(conf jsonb DEFAULT NULL) RETURNS int AS $$\n SELECT pgstac.get_setting('context_estimated_count', conf)::int;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_estimated_cost();\nCREATE OR REPLACE FUNCTION context_estimated_cost(conf jsonb DEFAULT NULL) RETURNS float AS $$\n SELECT pgstac.get_setting('context_estimated_cost', conf)::float;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_stats_ttl();\nCREATE OR REPLACE FUNCTION context_stats_ttl(conf jsonb DEFAULT NULL) RETURNS interval AS $$\n SELECT pgstac.get_setting('context_stats_ttl', conf)::interval;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION t2s(text) RETURNS text AS $$\n SELECT extract(epoch FROM $1::interval)::text || ' s';\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE STRICT;\n\nCREATE OR REPLACE FUNCTION age_ms(a timestamptz, b timestamptz DEFAULT clock_timestamp()) RETURNS float AS $$\n SELECT abs(extract(epoch from age(a,b)) * 1000);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION queue_timeout() RETURNS interval AS $$\n SELECT t2s(coalesce(\n get_setting('queue_timeout'),\n '1h'\n ))::interval;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION notice(VARIADIC text[]) RETURNS boolean AS $$\nDECLARE\ndebug boolean := current_setting('pgstac.debug', true);\nBEGIN\n IF debug THEN\n RAISE NOTICE 'NOTICE FROM FUNC: % >>>>> %', concat_ws(' | ', $1), clock_timestamp();\n RETURN TRUE;\n END IF;\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION empty_arr(ANYARRAY) RETURNS BOOLEAN AS $$\nSELECT CASE\n WHEN $1 IS NULL THEN TRUE\n WHEN cardinality($1)<1 THEN TRUE\nELSE FALSE\nEND;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_intersection(_a ANYARRAY, _b ANYARRAY) RETURNS ANYARRAY AS $$\n SELECT ARRAY ( SELECT unnest(_a) INTERSECT SELECT UNNEST(_b) );\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION array_map_ident(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_ident(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_map_literal(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_literal(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION array_reverse(anyarray) RETURNS anyarray AS $$\nSELECT ARRAY(\n SELECT $1[i]\n FROM generate_subscripts($1,1) AS s(i)\n ORDER BY i DESC\n);\n$$ LANGUAGE SQL STRICT IMMUTABLE;\n\nDROP TABLE IF EXISTS query_queue;\nCREATE TABLE query_queue (\n query text PRIMARY KEY,\n added timestamptz DEFAULT now()\n);\n\nDROP TABLE IF EXISTS query_queue_history;\nCREATE TABLE query_queue_history(\n query text,\n added timestamptz NOT NULL,\n finished timestamptz NOT NULL DEFAULT now(),\n error text\n);\n\nCREATE OR REPLACE PROCEDURE run_queued_queries() AS $$\nDECLARE\n qitem query_queue%ROWTYPE;\n timeout_ts timestamptz;\n error text;\n cnt int := 0;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n DELETE FROM query_queue WHERE query = (SELECT query FROM query_queue ORDER BY added DESC LIMIT 1 FOR UPDATE SKIP LOCKED) RETURNING * INTO qitem;\n IF NOT FOUND THEN\n EXIT;\n END IF;\n cnt := cnt + 1;\n BEGIN\n RAISE NOTICE 'RUNNING QUERY: %', qitem.query;\n EXECUTE qitem.query;\n EXCEPTION WHEN others THEN\n error := format('%s | %s', SQLERRM, SQLSTATE);\n END;\n INSERT INTO query_queue_history (query, added, finished, error)\n VALUES (qitem.query, qitem.added, clock_timestamp(), error);\n COMMIT;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION run_queued_queries_intransaction() RETURNS int AS $$\nDECLARE\n qitem query_queue%ROWTYPE;\n timeout_ts timestamptz;\n error text;\n cnt int := 0;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n DELETE FROM query_queue WHERE query = (SELECT query FROM query_queue ORDER BY added DESC LIMIT 1 FOR UPDATE SKIP LOCKED) RETURNING * INTO qitem;\n IF NOT FOUND THEN\n RETURN cnt;\n END IF;\n cnt := cnt + 1;\n BEGIN\n qitem.query := regexp_replace(qitem.query, 'CONCURRENTLY', '');\n RAISE NOTICE 'RUNNING QUERY: %', qitem.query;\n\n EXECUTE qitem.query;\n EXCEPTION WHEN others THEN\n error := format('%s | %s', SQLERRM, SQLSTATE);\n RAISE WARNING '%', error;\n END;\n INSERT INTO query_queue_history (query, added, finished, error)\n VALUES (qitem.query, qitem.added, clock_timestamp(), error);\n END LOOP;\n RETURN cnt;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION run_or_queue(query text) RETURNS VOID AS $$\nDECLARE\n use_queue text := COALESCE(get_setting('use_queue'), 'FALSE')::boolean;\nBEGIN\n IF get_setting_bool('debug') THEN\n RAISE NOTICE '%', query;\n END IF;\n IF use_queue THEN\n INSERT INTO query_queue (query) VALUES (query) ON CONFLICT DO NOTHING;\n ELSE\n EXECUTE query;\n END IF;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n\nDROP FUNCTION IF EXISTS check_pgstac_settings;\nCREATE OR REPLACE FUNCTION check_pgstac_settings(_sysmem text DEFAULT NULL) RETURNS VOID AS $$\nDECLARE\n settingval text;\n sysmem bigint := pg_size_bytes(_sysmem);\n effective_cache_size bigint := pg_size_bytes(current_setting('effective_cache_size', TRUE));\n shared_buffers bigint := pg_size_bytes(current_setting('shared_buffers', TRUE));\n work_mem bigint := pg_size_bytes(current_setting('work_mem', TRUE));\n max_connections int := current_setting('max_connections', TRUE);\n maintenance_work_mem bigint := pg_size_bytes(current_setting('maintenance_work_mem', TRUE));\n seq_page_cost float := current_setting('seq_page_cost', TRUE);\n random_page_cost float := current_setting('random_page_cost', TRUE);\n temp_buffers bigint := pg_size_bytes(current_setting('temp_buffers', TRUE));\n r record;\nBEGIN\n IF _sysmem IS NULL THEN\n RAISE NOTICE 'Call function with the size of your system memory `SELECT check_pgstac_settings(''4GB'')` to get pg system setting recommendations.';\n ELSE\n IF effective_cache_size < (sysmem * 0.5) THEN\n RAISE WARNING 'effective_cache_size of % is set low for a system with %. Recomended value between % and %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.5), pg_size_pretty(sysmem * 0.75);\n ELSIF effective_cache_size > (sysmem * 0.75) THEN\n RAISE WARNING 'effective_cache_size of % is set high for a system with %. Recomended value between % and %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.5), pg_size_pretty(sysmem * 0.75);\n ELSE\n RAISE NOTICE 'effective_cache_size of % is set appropriately for a system with %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem);\n END IF;\n\n IF shared_buffers < (sysmem * 0.2) THEN\n RAISE WARNING 'shared_buffers of % is set low for a system with %. Recomended value between % and %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.2), pg_size_pretty(sysmem * 0.3);\n ELSIF shared_buffers > (sysmem * 0.3) THEN\n RAISE WARNING 'shared_buffers of % is set high for a system with %. Recomended value between % and %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.2), pg_size_pretty(sysmem * 0.3);\n ELSE\n RAISE NOTICE 'shared_buffers of % is set appropriately for a system with %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem);\n END IF;\n shared_buffers = sysmem * 0.3;\n IF maintenance_work_mem < (sysmem * 0.2) THEN\n RAISE WARNING 'maintenance_work_mem of % is set low for shared_buffers of %. Recomended value between % and %', pg_size_pretty(maintenance_work_mem), pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers * 0.2), pg_size_pretty(shared_buffers * 0.3);\n ELSIF maintenance_work_mem > (shared_buffers * 0.3) THEN\n RAISE WARNING 'maintenance_work_mem of % is set high for shared_buffers of %. Recomended value between % and %', pg_size_pretty(maintenance_work_mem), pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers * 0.2), pg_size_pretty(shared_buffers * 0.3);\n ELSE\n RAISE NOTICE 'maintenance_work_mem of % is set appropriately for shared_buffers of %', pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers);\n END IF;\n\n IF work_mem * max_connections > shared_buffers THEN\n RAISE WARNING 'work_mem setting of % is set high for % max_connections please reduce work_mem to % or decrease max_connections to %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers/max_connections), floor(shared_buffers/work_mem);\n ELSIF work_mem * max_connections < (shared_buffers * 0.75) THEN\n RAISE WARNING 'work_mem setting of % is set low for % max_connections you may consider raising work_mem to % or increasing max_connections to %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers/max_connections), floor(shared_buffers/work_mem);\n ELSE\n RAISE NOTICE 'work_mem setting of % and max_connections of % are adequate for shared_buffers of %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers);\n END IF;\n\n IF random_page_cost / seq_page_cost != 1.1 THEN\n RAISE WARNING 'random_page_cost (%) /seq_page_cost (%) should be set to 1.1 for SSD. Change random_page_cost to %', random_page_cost, seq_page_cost, 1.1 * seq_page_cost;\n ELSE\n RAISE NOTICE 'random_page_cost and seq_page_cost set appropriately for SSD';\n END IF;\n\n IF temp_buffers < greatest(pg_size_bytes('128MB'),(maintenance_work_mem / 2)) THEN\n RAISE WARNING 'pgstac makes heavy use of temp tables, consider raising temp_buffers from % to %', pg_size_pretty(temp_buffers), greatest('128MB', pg_size_pretty((shared_buffers / 16)));\n END IF;\n END IF;\n\n RAISE NOTICE 'VALUES FOR PGSTAC VARIABLES';\n RAISE NOTICE 'These can be set either as GUC system variables or by setting in the pgstac_settings table.';\n\n FOR r IN SELECT name, get_setting(name) as setting, CASE WHEN current_setting(concat('pgstac.',name), TRUE) IS NOT NULL THEN concat('pgstac.',name, ' GUC') WHEN value IS NOT NULL THEN 'pgstac_settings table' ELSE 'Not Set' END as loc FROM pgstac_settings LOOP\n RAISE NOTICE '% is set to % from the %', r.name, r.setting, r.loc;\n END LOOP;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pg_cron';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider intalling pg_cron which can be used to automate tasks';\n ELSE\n RAISE NOTICE 'pg_cron % is installed', settingval;\n END IF;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pgstattuple';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider installing the pgstattuple extension which can be used to help maintain tables and indexes.';\n ELSE\n RAISE NOTICE 'pgstattuple % is installed', settingval;\n END IF;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pg_stat_statements';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider installing the pg_stat_statements extension which is very helpful for tracking the types of queries on the system';\n ELSE\n RAISE NOTICE 'pg_stat_statements % is installed', settingval;\n IF current_setting('pg_stat_statements.track_statements', TRUE) IS DISTINCT FROM 'all' THEN\n RAISE WARNING 'SET pg_stat_statements.track_statements TO ''all''; --In order to track statements within functions.';\n END IF;\n END IF;\n\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac, public SET CLIENT_MIN_MESSAGES TO NOTICE;\nCREATE OR REPLACE FUNCTION to_int(jsonb) RETURNS int AS $$\n SELECT floor(($1->>0)::float)::int;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION to_float(jsonb) RETURNS float AS $$\n SELECT ($1->>0)::float;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION to_tstz(jsonb) RETURNS timestamptz AS $$\n SELECT ($1->>0)::timestamptz;\n$$ LANGUAGE SQL IMMUTABLE STRICT SET TIME ZONE 'UTC' COST 5000 PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION to_text(jsonb) RETURNS text AS $$\n SELECT CASE WHEN jsonb_typeof($1) IN ('array','object') THEN $1::text ELSE $1->>0 END;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION to_text_array(jsonb) RETURNS text[] AS $$\n SELECT\n CASE jsonb_typeof($1)\n WHEN 'array' THEN ARRAY(SELECT jsonb_array_elements_text($1))\n ELSE ARRAY[$1->>0]\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION bbox_geom(_bbox jsonb) RETURNS geometry AS $$\nSELECT CASE jsonb_array_length(_bbox)\n WHEN 4 THEN\n ST_SetSRID(ST_MakeEnvelope(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float,\n (_bbox->>3)::float\n ),4326)\n WHEN 6 THEN\n ST_SetSRID(ST_3DMakeBox(\n ST_MakePoint(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float\n ),\n ST_MakePoint(\n (_bbox->>3)::float,\n (_bbox->>4)::float,\n (_bbox->>5)::float\n )\n ),4326)\n ELSE null END;\n;\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION geom_bbox(_geom geometry) RETURNS jsonb AS $$\n SELECT jsonb_build_array(\n st_xmin(_geom),\n st_ymin(_geom),\n st_xmax(_geom),\n st_ymax(_geom)\n );\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION flip_jsonb_array(j jsonb) RETURNS jsonb AS $$\n SELECT jsonb_agg(value) FROM (SELECT value FROM jsonb_array_elements(j) WITH ORDINALITY ORDER BY ordinality DESC) as t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION explode_dotpaths(j jsonb) RETURNS SETOF text[] AS $$\n SELECT string_to_array(p, '.') as e FROM jsonb_array_elements_text(j) p;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION explode_dotpaths_recurse(IN j jsonb) RETURNS SETOF text[] AS $$\n WITH RECURSIVE t AS (\n SELECT e FROM explode_dotpaths(j) e\n UNION ALL\n SELECT e[1:cardinality(e)-1]\n FROM t\n WHERE cardinality(e)>1\n ) SELECT e FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION jsonb_set_nested(j jsonb, path text[], val jsonb) RETURNS jsonb AS $$\nDECLARE\nBEGIN\n IF cardinality(path) > 1 THEN\n FOR i IN 1..(cardinality(path)-1) LOOP\n IF j #> path[:i] IS NULL THEN\n j := jsonb_set_lax(j, path[:i], '{}', TRUE);\n END IF;\n END LOOP;\n END IF;\n RETURN jsonb_set_lax(j, path, val, true);\n\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\n\n\nCREATE OR REPLACE FUNCTION jsonb_include(j jsonb, f jsonb) RETURNS jsonb AS $$\nDECLARE\n includes jsonb := f-> 'include';\n outj jsonb := '{}'::jsonb;\n path text[];\nBEGIN\n IF\n includes IS NULL\n OR jsonb_array_length(includes) = 0\n THEN\n RETURN j;\n ELSE\n includes := includes || '[\"id\",\"collection\"]'::jsonb;\n FOR path IN SELECT explode_dotpaths(includes) LOOP\n outj := jsonb_set_nested(outj, path, j #> path);\n END LOOP;\n END IF;\n RETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_exclude(j jsonb, f jsonb) RETURNS jsonb AS $$\nDECLARE\n excludes jsonb := f-> 'exclude';\n outj jsonb := j;\n path text[];\nBEGIN\n IF\n excludes IS NULL\n OR jsonb_array_length(excludes) = 0\n THEN\n RETURN j;\n ELSE\n FOR path IN SELECT explode_dotpaths(excludes) LOOP\n outj := outj #- path;\n END LOOP;\n END IF;\n RETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_fields(j jsonb, f jsonb DEFAULT '{\"fields\":[]}') RETURNS jsonb AS $$\n SELECT jsonb_exclude(jsonb_include(j, f), f);\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION merge_jsonb(_a jsonb, _b jsonb) RETURNS jsonb AS $$\n SELECT\n CASE\n WHEN _a = '\"𒍟※\"'::jsonb THEN NULL\n WHEN _a IS NULL OR jsonb_typeof(_a) = 'null' THEN _b\n WHEN jsonb_typeof(_a) = 'object' AND jsonb_typeof(_b) = 'object' THEN\n (\n SELECT\n jsonb_strip_nulls(\n jsonb_object_agg(\n key,\n merge_jsonb(a.value, b.value)\n )\n )\n FROM\n jsonb_each(coalesce(_a,'{}'::jsonb)) as a\n FULL JOIN\n jsonb_each(coalesce(_b,'{}'::jsonb)) as b\n USING (key)\n )\n WHEN\n jsonb_typeof(_a) = 'array'\n AND jsonb_typeof(_b) = 'array'\n AND jsonb_array_length(_a) = jsonb_array_length(_b)\n THEN\n (\n SELECT jsonb_agg(m) FROM\n ( SELECT\n merge_jsonb(\n jsonb_array_elements(_a),\n jsonb_array_elements(_b)\n ) as m\n ) as l\n )\n ELSE _a\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION strip_jsonb(_a jsonb, _b jsonb) RETURNS jsonb AS $$\n SELECT\n CASE\n\n WHEN (_a IS NULL OR jsonb_typeof(_a) = 'null') AND _b IS NOT NULL AND jsonb_typeof(_b) != 'null' THEN '\"𒍟※\"'::jsonb\n WHEN _b IS NULL OR jsonb_typeof(_a) = 'null' THEN _a\n WHEN _a = _b AND jsonb_typeof(_a) = 'object' THEN '{}'::jsonb\n WHEN _a = _b THEN NULL\n WHEN jsonb_typeof(_a) = 'object' AND jsonb_typeof(_b) = 'object' THEN\n (\n SELECT\n jsonb_strip_nulls(\n jsonb_object_agg(\n key,\n strip_jsonb(a.value, b.value)\n )\n )\n FROM\n jsonb_each(_a) as a\n FULL JOIN\n jsonb_each(_b) as b\n USING (key)\n )\n WHEN\n jsonb_typeof(_a) = 'array'\n AND jsonb_typeof(_b) = 'array'\n AND jsonb_array_length(_a) = jsonb_array_length(_b)\n THEN\n (\n SELECT jsonb_agg(m) FROM\n ( SELECT\n strip_jsonb(\n jsonb_array_elements(_a),\n jsonb_array_elements(_b)\n ) as m\n ) as l\n )\n ELSE _a\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION nullif_jsonbnullempty(j jsonb) RETURNS jsonb AS $$\n SELECT nullif(nullif(nullif(j,'null'::jsonb),'{}'::jsonb),'[]'::jsonb);\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION jsonb_array_unique(j jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(jsonb_agg(DISTINCT a)) v FROM jsonb_array_elements(j) a;\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_concat_ignorenull(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT coalesce(a,'[]'::jsonb) || coalesce(b,'[]'::jsonb);\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_least(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(least(nullif_jsonbnullempty(a), nullif_jsonbnullempty(b)));\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_greatest(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(greatest(a, b));\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION first_notnull_sfunc(anyelement, anyelement) RETURNS anyelement AS $$\n SELECT COALESCE($1,$2);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE AGGREGATE first_notnull(anyelement)(\n SFUNC = first_notnull_sfunc,\n STYPE = anyelement\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_array_unique_merge(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_concat_ignorenull,\n FINALFUNC = jsonb_array_unique\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_min(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_least\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_max(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_greatest\n);\n/* looks for a geometry in a stac item first from geometry and falling back to bbox */\nCREATE OR REPLACE FUNCTION stac_geom(value jsonb) RETURNS geometry AS $$\nSELECT\n CASE\n WHEN value ? 'intersects' THEN\n ST_GeomFromGeoJSON(value->>'intersects')\n WHEN value ? 'geometry' THEN\n ST_GeomFromGeoJSON(value->>'geometry')\n WHEN value ? 'bbox' THEN\n pgstac.bbox_geom(value->'bbox')\n ELSE NULL\n END as geometry\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION stac_daterange(\n value jsonb\n) RETURNS tstzrange AS $$\nDECLARE\n props jsonb := value;\n dt timestamptz;\n edt timestamptz;\nBEGIN\n IF props ? 'properties' THEN\n props := props->'properties';\n END IF;\n IF\n props ? 'start_datetime'\n AND props->>'start_datetime' IS NOT NULL\n AND props ? 'end_datetime'\n AND props->>'end_datetime' IS NOT NULL\n THEN\n dt := props->>'start_datetime';\n edt := props->>'end_datetime';\n IF dt > edt THEN\n RAISE EXCEPTION 'start_datetime must be < end_datetime';\n END IF;\n ELSE\n dt := props->>'datetime';\n edt := props->>'datetime';\n END IF;\n IF dt is NULL OR edt IS NULL THEN\n RAISE NOTICE 'DT: %, EDT: %', dt, edt;\n RAISE EXCEPTION 'Either datetime (%) or both start_datetime (%) and end_datetime (%) must be set.', props->>'datetime',props->>'start_datetime',props->>'end_datetime';\n END IF;\n RETURN tstzrange(dt, edt, '[]');\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_datetime(value jsonb) RETURNS timestamptz AS $$\n SELECT lower(stac_daterange(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_end_datetime(value jsonb) RETURNS timestamptz AS $$\n SELECT upper(stac_daterange(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE TABLE IF NOT EXISTS stac_extensions(\n url text PRIMARY KEY,\n content jsonb\n);\nCREATE OR REPLACE FUNCTION collection_base_item(content jsonb) RETURNS jsonb AS $$\n SELECT jsonb_build_object(\n 'type', 'Feature',\n 'stac_version', content->'stac_version',\n 'assets', content->'item_assets',\n 'collection', content->'id'\n );\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE TABLE IF NOT EXISTS collections (\n key bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,\n id text GENERATED ALWAYS AS (content->>'id') STORED UNIQUE NOT NULL,\n content JSONB NOT NULL,\n base_item jsonb GENERATED ALWAYS AS (pgstac.collection_base_item(content)) STORED,\n geometry geometry GENERATED ALWAYS AS (pgstac.collection_geom(content)) STORED,\n datetime timestamptz GENERATED ALWAYS AS (pgstac.collection_datetime(content)) STORED,\n end_datetime timestamptz GENERATED ALWAYS AS (pgstac.collection_enddatetime(content)) STORED,\n private jsonb,\n partition_trunc text CHECK (partition_trunc IN ('year', 'month'))\n);\n\nCREATE OR REPLACE FUNCTION collection_base_item(cid text) RETURNS jsonb AS $$\n SELECT pgstac.collection_base_item(content) FROM pgstac.collections WHERE id = cid LIMIT 1;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION create_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION update_collection(data jsonb) RETURNS VOID AS $$\nDECLARE\n out collections%ROWTYPE;\nBEGIN\n UPDATE collections SET content=data WHERE id = data->>'id' RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ON CONFLICT (id) DO\n UPDATE\n SET content=EXCLUDED.content\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION delete_collection(_id text) RETURNS VOID AS $$\nDECLARE\n out collections%ROWTYPE;\nBEGIN\n DELETE FROM collections WHERE id = _id RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION get_collection(id text) RETURNS jsonb AS $$\n SELECT content FROM collections\n WHERE id=$1\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION all_collections() RETURNS jsonb AS $$\n SELECT coalesce(jsonb_agg(content), '[]'::jsonb) FROM collections;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION collection_delete_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n collection_base_partition text := concat('_items_', OLD.key);\nBEGIN\n EXECUTE format($q$\n DELETE FROM partition_stats WHERE partition IN (\n SELECT partition FROM partition_sys_meta\n WHERE collection=%L\n );\n DROP TABLE IF EXISTS %I CASCADE;\n $q$,\n OLD.id,\n collection_base_partition\n );\n RETURN OLD;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER collection_delete_trigger BEFORE DELETE ON collections\nFOR EACH ROW EXECUTE FUNCTION collection_delete_trigger_func();\nCREATE OR REPLACE FUNCTION queryable_signature(n text, c text[]) RETURNS text AS $$\n SELECT concat(n, c);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TABLE queryables (\n id bigint GENERATED ALWAYS AS identity PRIMARY KEY,\n name text NOT NULL,\n collection_ids text[], -- used to determine what partitions to create indexes on\n definition jsonb,\n property_path text,\n property_wrapper text,\n property_index_type text\n);\nCREATE INDEX queryables_name_idx ON queryables (name);\nCREATE INDEX queryables_collection_idx ON queryables USING GIN (collection_ids);\nCREATE INDEX queryables_property_wrapper_idx ON queryables (property_wrapper);\n\nCREATE OR REPLACE FUNCTION pgstac.queryables_constraint_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n allcollections text[];\nBEGIN\n RAISE NOTICE 'Making sure that name/collection is unique for queryables %', NEW;\n IF NEW.collection_ids IS NOT NULL THEN\n IF EXISTS (\n SELECT 1\n FROM unnest(NEW.collection_ids) c\n LEFT JOIN\n collections\n ON (collections.id = c)\n WHERE collections.id IS NULL\n ) THEN\n RAISE foreign_key_violation USING MESSAGE = format(\n 'One or more collections in %s do not exist.', NEW.collection_ids\n );\n RETURN NULL;\n END IF;\n END IF;\n IF TG_OP = 'INSERT' THEN\n IF EXISTS (\n SELECT 1 FROM queryables q\n WHERE\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n )\n ) THEN\n RAISE unique_violation USING MESSAGE = format(\n 'There is already a queryable for %s for a collection in %s: %s',\n NEW.name,\n NEW.collection_ids,\n\t\t\t\t(SELECT json_agg(row_to_json(q)) FROM queryables q WHERE\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n ))\n );\n RETURN NULL;\n END IF;\n END IF;\n IF TG_OP = 'UPDATE' THEN\n IF EXISTS (\n SELECT 1 FROM queryables q\n WHERE\n q.id != NEW.id\n AND\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n )\n ) THEN\n RAISE unique_violation\n USING MESSAGE = format(\n 'There is already a queryable for %s for a collection in %s',\n NEW.name,\n NEW.collection_ids\n );\n RETURN NULL;\n END IF;\n END IF;\n\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER queryables_constraint_insert_trigger\nBEFORE INSERT ON queryables\nFOR EACH ROW EXECUTE PROCEDURE queryables_constraint_triggerfunc();\n\nCREATE TRIGGER queryables_constraint_update_trigger\nBEFORE UPDATE ON queryables\nFOR EACH ROW\nWHEN (NEW.name = OLD.name AND NEW.collection_ids IS DISTINCT FROM OLD.collection_ids)\nEXECUTE PROCEDURE queryables_constraint_triggerfunc();\n\n\nCREATE OR REPLACE FUNCTION array_to_path(arr text[]) RETURNS text AS $$\n SELECT string_agg(\n quote_literal(v),\n '->'\n ) FROM unnest(arr) v;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\n\n\nCREATE OR REPLACE FUNCTION queryable(\n IN dotpath text,\n OUT path text,\n OUT expression text,\n OUT wrapper text,\n OUT nulled_wrapper text\n) AS $$\nDECLARE\n q RECORD;\n path_elements text[];\nBEGIN\n IF dotpath IN ('id', 'geometry', 'datetime', 'end_datetime', 'collection') THEN\n path := dotpath;\n expression := dotpath;\n wrapper := NULL;\n RETURN;\n END IF;\n SELECT * INTO q FROM queryables\n WHERE\n name=dotpath\n OR name = 'properties.' || dotpath\n OR name = replace(dotpath, 'properties.', '')\n ;\n IF q.property_wrapper IS NULL THEN\n IF q.definition->>'type' = 'number' THEN\n wrapper := 'to_float';\n nulled_wrapper := wrapper;\n ELSIF q.definition->>'format' = 'date-time' THEN\n wrapper := 'to_tstz';\n nulled_wrapper := wrapper;\n ELSE\n nulled_wrapper := NULL;\n wrapper := 'to_text';\n END IF;\n ELSE\n wrapper := q.property_wrapper;\n nulled_wrapper := wrapper;\n END IF;\n IF q.property_path IS NOT NULL THEN\n path := q.property_path;\n ELSE\n path_elements := string_to_array(dotpath, '.');\n IF path_elements[1] IN ('links', 'assets', 'stac_version', 'stac_extensions') THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSIF path_elements[1] = 'properties' THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSE\n path := format($F$content->'properties'->%s$F$, array_to_path(path_elements));\n END IF;\n END IF;\n expression := format('%I(%s)', wrapper, path);\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\nCREATE OR REPLACE FUNCTION unnest_collection(collection_ids text[] DEFAULT NULL) RETURNS SETOF text AS $$\n DECLARE\n BEGIN\n IF collection_ids IS NULL THEN\n RETURN QUERY SELECT id FROM collections;\n END IF;\n RETURN QUERY SELECT unnest(collection_ids);\n END;\n$$ LANGUAGE PLPGSQL STABLE;\n\nCREATE OR REPLACE FUNCTION normalize_indexdef(def text) RETURNS text AS $$\nDECLARE\nBEGIN\n def := btrim(def, ' \\n\\t');\n\tdef := regexp_replace(def, '^CREATE (UNIQUE )?INDEX ([^ ]* )?ON (ONLY )?([^ ]* )?', '', 'i');\n RETURN def;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION indexdef(q queryables) RETURNS text AS $$\n DECLARE\n out text;\n BEGIN\n IF q.name = 'id' THEN\n out := 'CREATE UNIQUE INDEX ON %I USING btree (id)';\n ELSIF q.name = 'datetime' THEN\n out := 'CREATE INDEX ON %I USING btree (datetime DESC, end_datetime)';\n ELSIF q.name = 'geometry' THEN\n out := 'CREATE INDEX ON %I USING gist (geometry)';\n ELSE\n out := format($q$CREATE INDEX ON %%I USING %s (%s(((content -> 'properties'::text) -> %L::text)))$q$,\n lower(COALESCE(q.property_index_type, 'BTREE')),\n lower(COALESCE(q.property_wrapper, 'to_text')),\n q.name\n );\n END IF;\n RETURN btrim(out, ' \\n\\t');\n END;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nDROP VIEW IF EXISTS pgstac_indexes;\nCREATE VIEW pgstac_indexes AS\nSELECT\n i.schemaname,\n i.tablename,\n i.indexname,\n regexp_replace(btrim(replace(replace(indexdef, i.indexname, ''),'pgstac.',''),' \\t\\n'), '[ ]+', ' ', 'g') as idx,\n COALESCE(\n (regexp_match(indexdef, '\\(([a-zA-Z]+)\\)'))[1],\n (regexp_match(indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_-]+)''::text'))[1],\n CASE WHEN indexdef ~* '\\(datetime desc, end_datetime\\)' THEN 'datetime' ELSE NULL END\n ) AS field,\n pg_table_size(i.indexname::text) as index_size,\n pg_size_pretty(pg_table_size(i.indexname::text)) as index_size_pretty\nFROM\n pg_indexes i\nWHERE i.schemaname='pgstac' and i.tablename ~ '_items_' AND indexdef !~* ' only ';\n\nDROP VIEW IF EXISTS pgstac_index_stats;\nCREATE VIEW pgstac_indexes_stats AS\nSELECT\n i.schemaname,\n i.tablename,\n i.indexname,\n indexdef,\n COALESCE(\n (regexp_match(indexdef, '\\(([a-zA-Z]+)\\)'))[1],\n (regexp_match(indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_]+)''::text'))[1],\n CASE WHEN indexdef ~* '\\(datetime desc, end_datetime\\)' THEN 'datetime_end_datetime' ELSE NULL END\n ) AS field,\n pg_table_size(i.indexname::text) as index_size,\n pg_size_pretty(pg_table_size(i.indexname::text)) as index_size_pretty,\n n_distinct,\n most_common_vals::text::text[],\n most_common_freqs::text::text[],\n histogram_bounds::text::text[],\n correlation\nFROM\n pg_indexes i\n LEFT JOIN pg_stats s ON (s.tablename = i.indexname)\nWHERE i.schemaname='pgstac' and i.tablename ~ '_items_';\n\nCREATE OR REPLACE FUNCTION queryable_indexes(\n IN treeroot text DEFAULT 'items',\n IN changes boolean DEFAULT FALSE,\n OUT collection text,\n OUT partition text,\n OUT field text,\n OUT indexname text,\n OUT existing_idx text,\n OUT queryable_idx text\n) RETURNS SETOF RECORD AS $$\nWITH p AS (\n SELECT\n relid::text as partition,\n replace(replace(\n CASE\n WHEN parentrelid::regclass::text='items' THEN pg_get_expr(c.relpartbound, c.oid)\n ELSE pg_get_expr(parent.relpartbound, parent.oid)\n END,\n 'FOR VALUES IN (''',''), ''')',\n ''\n ) AS collection\n FROM pg_partition_tree(treeroot)\n JOIN pg_class c ON (relid::regclass = c.oid)\n JOIN pg_class parent ON (parentrelid::regclass = parent.oid AND isleaf)\n ), i AS (\n SELECT\n partition,\n indexname,\n regexp_replace(btrim(replace(replace(indexdef, indexname, ''),'pgstac.',''),' \\t\\n'), '[ ]+', ' ', 'g') as iidx,\n COALESCE(\n (regexp_match(indexdef, '\\(([a-zA-Z]+)\\)'))[1],\n (regexp_match(indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_-]+)''::text'))[1],\n CASE WHEN indexdef ~* '\\(datetime desc, end_datetime\\)' THEN 'datetime' ELSE NULL END\n ) AS field\n FROM\n pg_indexes\n JOIN p ON (tablename=partition)\n ), q AS (\n SELECT\n name AS field,\n collection,\n partition,\n format(indexdef(queryables), partition) as qidx\n FROM queryables, unnest_collection(queryables.collection_ids) collection\n JOIN p USING (collection)\n WHERE property_index_type IS NOT NULL OR name IN ('datetime','geometry','id')\n )\n SELECT\n collection,\n partition,\n field,\n indexname,\n iidx as existing_idx,\n qidx as queryable_idx\n FROM i FULL JOIN q USING (field, partition)\n WHERE CASE WHEN changes THEN lower(iidx) IS DISTINCT FROM lower(qidx) ELSE TRUE END;\n;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION maintain_index(\n indexname text,\n queryable_idx text,\n dropindexes boolean DEFAULT FALSE,\n rebuildindexes boolean DEFAULT FALSE,\n idxconcurrently boolean DEFAULT FALSE\n) RETURNS VOID AS $$\nDECLARE\nBEGIN\n IF indexname IS NOT NULL THEN\n IF dropindexes OR queryable_idx IS NOT NULL THEN\n EXECUTE format('DROP INDEX IF EXISTS %I;', indexname);\n ELSIF rebuildindexes THEN\n IF idxconcurrently THEN\n EXECUTE format('REINDEX INDEX CONCURRENTLY %I;', indexname);\n ELSE\n EXECUTE format('REINDEX INDEX CONCURRENTLY %I;', indexname);\n END IF;\n END IF;\n END IF;\n IF queryable_idx IS NOT NULL THEN\n IF idxconcurrently THEN\n EXECUTE replace(queryable_idx, 'INDEX', 'INDEX CONCURRENTLY');\n ELSE EXECUTE queryable_idx;\n END IF;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\n\nset check_function_bodies to off;\nCREATE OR REPLACE FUNCTION maintain_partition_queries(\n part text DEFAULT 'items',\n dropindexes boolean DEFAULT FALSE,\n rebuildindexes boolean DEFAULT FALSE,\n idxconcurrently boolean DEFAULT FALSE\n) RETURNS SETOF text AS $$\nDECLARE\n rec record;\n q text;\nBEGIN\n FOR rec IN (\n SELECT * FROM queryable_indexes(part,true)\n ) LOOP\n q := format(\n 'SELECT maintain_index(\n %L,%L,%L,%L,%L\n );',\n rec.indexname,\n rec.queryable_idx,\n dropindexes,\n rebuildindexes,\n idxconcurrently\n );\n RAISE NOTICE 'Q: %s', q;\n RETURN NEXT q;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION maintain_partitions(\n part text DEFAULT 'items',\n dropindexes boolean DEFAULT FALSE,\n rebuildindexes boolean DEFAULT FALSE\n) RETURNS VOID AS $$\n WITH t AS (\n SELECT run_or_queue(q) FROM maintain_partition_queries(part, dropindexes, rebuildindexes) q\n ) SELECT count(*) FROM t;\n$$ LANGUAGE SQL;\n\n\nCREATE OR REPLACE FUNCTION queryables_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\nBEGIN\n PERFORM maintain_partitions();\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER queryables_trigger AFTER INSERT OR UPDATE ON queryables\nFOR EACH STATEMENT EXECUTE PROCEDURE queryables_trigger_func();\n\n\nCREATE OR REPLACE FUNCTION get_queryables(_collection_ids text[] DEFAULT NULL) RETURNS jsonb AS $$\nDECLARE\nBEGIN\n -- Build up queryables if the input contains valid collection ids or is empty\n IF EXISTS (\n SELECT 1 FROM collections\n WHERE\n _collection_ids IS NULL\n OR cardinality(_collection_ids) = 0\n OR id = ANY(_collection_ids)\n )\n THEN\n RETURN (\n WITH base AS (\n SELECT\n unnest(collection_ids) as collection_id,\n name,\n coalesce(definition, '{\"type\":\"string\"}'::jsonb) as definition\n FROM queryables\n WHERE\n _collection_ids IS NULL OR\n _collection_ids = '{}'::text[] OR\n _collection_ids && collection_ids\n UNION ALL\n SELECT null, name, coalesce(definition, '{\"type\":\"string\"}'::jsonb) as definition\n FROM queryables WHERE collection_ids IS NULL OR collection_ids = '{}'::text[]\n ), g AS (\n SELECT\n name,\n first_notnull(definition) as definition,\n jsonb_array_unique_merge(definition->'enum') as enum,\n jsonb_min(definition->'minimum') as minimum,\n jsonb_min(definition->'maxiumn') as maximum\n FROM base\n GROUP BY 1\n )\n SELECT\n jsonb_build_object(\n '$schema', 'http://json-schema.org/draft-07/schema#',\n '$id', '',\n 'type', 'object',\n 'title', 'STAC Queryables.',\n 'properties', jsonb_object_agg(\n name,\n definition\n ||\n jsonb_strip_nulls(jsonb_build_object(\n 'enum', enum,\n 'minimum', minimum,\n 'maximum', maximum\n ))\n ),\n 'additionalProperties', pgstac.additional_properties()\n )\n FROM g\n );\n ELSE\n RETURN NULL;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\nCREATE OR REPLACE FUNCTION get_queryables(_collection text DEFAULT NULL) RETURNS jsonb AS $$\n SELECT\n CASE\n WHEN _collection IS NULL THEN get_queryables(NULL::text[])\n ELSE get_queryables(ARRAY[_collection])\n END\n ;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_queryables() RETURNS jsonb AS $$\n SELECT get_queryables(NULL::text[]);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION schema_qualify_refs(url text, j jsonb) returns jsonb as $$\n SELECT regexp_replace(j::text, '\"\\$ref\": \"#', concat('\"$ref\": \"', url, '#'), 'g')::jsonb;\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE VIEW stac_extension_queryables AS\nSELECT DISTINCT key as name, schema_qualify_refs(e.url, j.value) as definition FROM stac_extensions e, jsonb_each(e.content->'definitions'->'fields'->'properties') j;\n\n\nCREATE OR REPLACE FUNCTION missing_queryables(_collection text, _tablesample float DEFAULT 5, minrows float DEFAULT 10) RETURNS TABLE(collection text, name text, definition jsonb, property_wrapper text) AS $$\nDECLARE\n q text;\n _partition text;\n explain_json json;\n psize float;\n estrows float;\nBEGIN\n SELECT format('_items_%s', key) INTO _partition FROM collections WHERE id=_collection;\n\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM %I;', _partition)\n INTO explain_json;\n psize := explain_json->0->'Plan'->'Plan Rows';\n estrows := _tablesample * .01 * psize;\n IF estrows < minrows THEN\n _tablesample := least(100,greatest(_tablesample, (estrows / psize) / 100));\n RAISE NOTICE '%', (psize / estrows) / 100;\n END IF;\n RAISE NOTICE 'Using tablesample % to find missing queryables from % % that has ~% rows estrows: %', _tablesample, _collection, _partition, psize, estrows;\n\n q := format(\n $q$\n WITH q AS (\n SELECT * FROM queryables\n WHERE\n collection_ids IS NULL\n OR %L = ANY(collection_ids)\n ), t AS (\n SELECT\n content->'properties' AS properties\n FROM\n %I\n TABLESAMPLE SYSTEM(%L)\n ), p AS (\n SELECT DISTINCT ON (key)\n key,\n value,\n s.definition\n FROM t\n JOIN LATERAL jsonb_each(properties) ON TRUE\n LEFT JOIN q ON (q.name=key)\n LEFT JOIN stac_extension_queryables s ON (s.name=key)\n WHERE q.definition IS NULL\n )\n SELECT\n %L,\n key,\n COALESCE(definition, jsonb_build_object('type',jsonb_typeof(value))) as definition,\n CASE\n WHEN definition->>'type' = 'integer' THEN 'to_int'\n WHEN COALESCE(definition->>'type', jsonb_typeof(value)) = 'number' THEN 'to_float'\n WHEN COALESCE(definition->>'type', jsonb_typeof(value)) = 'array' THEN 'to_text_array'\n ELSE 'to_text'\n END\n FROM p;\n $q$,\n _collection,\n _partition,\n _tablesample,\n _collection\n );\n RETURN QUERY EXECUTE q;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION missing_queryables(_tablesample float DEFAULT 5) RETURNS TABLE(collection_ids text[], name text, definition jsonb, property_wrapper text) AS $$\n SELECT\n array_agg(collection),\n name,\n definition,\n property_wrapper\n FROM\n collections\n JOIN LATERAL\n missing_queryables(id, _tablesample) c\n ON TRUE\n GROUP BY\n 2,3,4\n ORDER BY 2,1\n ;\n$$ LANGUAGE SQL;\nCREATE OR REPLACE FUNCTION parse_dtrange(\n _indate jsonb,\n relative_base timestamptz DEFAULT date_trunc('hour', CURRENT_TIMESTAMP)\n) RETURNS tstzrange AS $$\nDECLARE\n timestrs text[];\n s timestamptz;\n e timestamptz;\nBEGIN\n timestrs :=\n CASE\n WHEN _indate ? 'timestamp' THEN\n ARRAY[_indate->>'timestamp']\n WHEN _indate ? 'interval' THEN\n to_text_array(_indate->'interval')\n WHEN jsonb_typeof(_indate) = 'array' THEN\n to_text_array(_indate)\n ELSE\n regexp_split_to_array(\n _indate->>0,\n '/'\n )\n END;\n RAISE NOTICE 'TIMESTRS %', timestrs;\n IF cardinality(timestrs) = 1 THEN\n IF timestrs[1] ILIKE 'P%' THEN\n RETURN tstzrange(relative_base - upper(timestrs[1])::interval, relative_base, '[)');\n END IF;\n s := timestrs[1]::timestamptz;\n RETURN tstzrange(s, s, '[]');\n END IF;\n\n IF cardinality(timestrs) != 2 THEN\n RAISE EXCEPTION 'Timestamp cannot have more than 2 values';\n END IF;\n\n IF timestrs[1] = '..' OR timestrs[1] = '' THEN\n s := '-infinity'::timestamptz;\n e := timestrs[2]::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] = '..' OR timestrs[2] = '' THEN\n s := timestrs[1]::timestamptz;\n e := 'infinity'::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[1] ILIKE 'P%' AND timestrs[2] NOT ILIKE 'P%' THEN\n e := timestrs[2]::timestamptz;\n s := e - upper(timestrs[1])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] ILIKE 'P%' AND timestrs[1] NOT ILIKE 'P%' THEN\n s := timestrs[1]::timestamptz;\n e := s + upper(timestrs[2])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n s := timestrs[1]::timestamptz;\n e := timestrs[2]::timestamptz;\n\n RETURN tstzrange(s,e,'[)');\n\n RETURN NULL;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT PARALLEL SAFE SET TIME ZONE 'UTC';\n\nCREATE OR REPLACE FUNCTION parse_dtrange(\n _indate text,\n relative_base timestamptz DEFAULT CURRENT_TIMESTAMP\n) RETURNS tstzrange AS $$\n SELECT parse_dtrange(to_jsonb(_indate), relative_base);\n$$ LANGUAGE SQL STABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION temporal_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\n ll text := 'datetime';\n lh text := 'end_datetime';\n rrange tstzrange;\n rl text;\n rh text;\n outq text;\nBEGIN\n rrange := parse_dtrange(args->1);\n RAISE NOTICE 'Constructing temporal query OP: %, ARGS: %, RRANGE: %', op, args, rrange;\n op := lower(op);\n rl := format('%L::timestamptz', lower(rrange));\n rh := format('%L::timestamptz', upper(rrange));\n outq := CASE op\n WHEN 't_before' THEN 'lh < rl'\n WHEN 't_after' THEN 'll > rh'\n WHEN 't_meets' THEN 'lh = rl'\n WHEN 't_metby' THEN 'll = rh'\n WHEN 't_overlaps' THEN 'll < rl AND rl < lh < rh'\n WHEN 't_overlappedby' THEN 'rl < ll < rh AND lh > rh'\n WHEN 't_starts' THEN 'll = rl AND lh < rh'\n WHEN 't_startedby' THEN 'll = rl AND lh > rh'\n WHEN 't_during' THEN 'll > rl AND lh < rh'\n WHEN 't_contains' THEN 'll < rl AND lh > rh'\n WHEN 't_finishes' THEN 'll > rl AND lh = rh'\n WHEN 't_finishedby' THEN 'll < rl AND lh = rh'\n WHEN 't_equals' THEN 'll = rl AND lh = rh'\n WHEN 't_disjoint' THEN 'NOT (ll <= rh AND lh >= rl)'\n WHEN 't_intersects' THEN 'll <= rh AND lh >= rl'\n WHEN 'anyinteracts' THEN 'll <= rh AND lh >= rl'\n END;\n outq := regexp_replace(outq, '\\mll\\M', ll);\n outq := regexp_replace(outq, '\\mlh\\M', lh);\n outq := regexp_replace(outq, '\\mrl\\M', rl);\n outq := regexp_replace(outq, '\\mrh\\M', rh);\n outq := format('(%s)', outq);\n RETURN outq;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\n\n\nCREATE OR REPLACE FUNCTION spatial_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\n geom text;\n j jsonb := args->1;\nBEGIN\n op := lower(op);\n RAISE NOTICE 'Constructing spatial query OP: %, ARGS: %', op, args;\n IF op NOT IN ('s_equals','s_disjoint','s_touches','s_within','s_overlaps','s_crosses','s_intersects','intersects','s_contains') THEN\n RAISE EXCEPTION 'Spatial Operator % Not Supported', op;\n END IF;\n op := regexp_replace(op, '^s_', 'st_');\n IF op = 'intersects' THEN\n op := 'st_intersects';\n END IF;\n -- Convert geometry to WKB string\n IF j ? 'type' AND j ? 'coordinates' THEN\n geom := st_geomfromgeojson(j)::text;\n ELSIF jsonb_typeof(j) = 'array' THEN\n geom := bbox_geom(j)::text;\n END IF;\n\n RETURN format('%s(geometry, %L::geometry)', op, geom);\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION query_to_cql2(q jsonb) RETURNS jsonb AS $$\n-- Translates anything passed in through the deprecated \"query\" into equivalent CQL2\nWITH t AS (\n SELECT key as property, value as ops\n FROM jsonb_each(q)\n), t2 AS (\n SELECT property, (jsonb_each(ops)).*\n FROM t WHERE jsonb_typeof(ops) = 'object'\n UNION ALL\n SELECT property, 'eq', ops\n FROM t WHERE jsonb_typeof(ops) != 'object'\n)\nSELECT\n jsonb_strip_nulls(jsonb_build_object(\n 'op', 'and',\n 'args', jsonb_agg(\n jsonb_build_object(\n 'op', key,\n 'args', jsonb_build_array(\n jsonb_build_object('property',property),\n value\n )\n )\n )\n )\n) as qcql FROM t2\n;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\nCREATE OR REPLACE FUNCTION cql1_to_cql2(j jsonb) RETURNS jsonb AS $$\nDECLARE\n args jsonb;\n ret jsonb;\nBEGIN\n RAISE NOTICE 'CQL1_TO_CQL2: %', j;\n IF j ? 'filter' THEN\n RETURN cql1_to_cql2(j->'filter');\n END IF;\n IF j ? 'property' THEN\n RETURN j;\n END IF;\n IF jsonb_typeof(j) = 'array' THEN\n SELECT jsonb_agg(cql1_to_cql2(el)) INTO args FROM jsonb_array_elements(j) el;\n RETURN args;\n END IF;\n IF jsonb_typeof(j) = 'number' THEN\n RETURN j;\n END IF;\n IF jsonb_typeof(j) = 'string' THEN\n RETURN j;\n END IF;\n\n IF jsonb_typeof(j) = 'object' THEN\n SELECT jsonb_build_object(\n 'op', key,\n 'args', cql1_to_cql2(value)\n ) INTO ret\n FROM jsonb_each(j)\n WHERE j IS NOT NULL;\n RETURN ret;\n END IF;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE STRICT;\n\nCREATE TABLE cql2_ops (\n op text PRIMARY KEY,\n template text,\n types text[]\n);\nINSERT INTO cql2_ops (op, template, types) VALUES\n ('eq', '%s = %s', NULL),\n ('neq', '%s != %s', NULL),\n ('ne', '%s != %s', NULL),\n ('!=', '%s != %s', NULL),\n ('<>', '%s != %s', NULL),\n ('lt', '%s < %s', NULL),\n ('lte', '%s <= %s', NULL),\n ('gt', '%s > %s', NULL),\n ('gte', '%s >= %s', NULL),\n ('le', '%s <= %s', NULL),\n ('ge', '%s >= %s', NULL),\n ('=', '%s = %s', NULL),\n ('<', '%s < %s', NULL),\n ('<=', '%s <= %s', NULL),\n ('>', '%s > %s', NULL),\n ('>=', '%s >= %s', NULL),\n ('like', '%s LIKE %s', NULL),\n ('ilike', '%s ILIKE %s', NULL),\n ('+', '%s + %s', NULL),\n ('-', '%s - %s', NULL),\n ('*', '%s * %s', NULL),\n ('/', '%s / %s', NULL),\n ('not', 'NOT (%s)', NULL),\n ('between', '%s BETWEEN %s AND %s', NULL),\n ('isnull', '%s IS NULL', NULL),\n ('upper', 'upper(%s)', NULL),\n ('lower', 'lower(%s)', NULL)\nON CONFLICT (op) DO UPDATE\n SET\n template = EXCLUDED.template\n;\n\n\nCREATE OR REPLACE FUNCTION cql2_query(j jsonb, wrapper text DEFAULT NULL) RETURNS text AS $$\n#variable_conflict use_variable\nDECLARE\n args jsonb := j->'args';\n arg jsonb;\n op text := lower(j->>'op');\n cql2op RECORD;\n literal text;\n _wrapper text;\n leftarg text;\n rightarg text;\n prop text;\n extra_props bool := pgstac.additional_properties();\nBEGIN\n IF j IS NULL OR (op IS NOT NULL AND args IS NULL) THEN\n RETURN NULL;\n END IF;\n RAISE NOTICE 'CQL2_QUERY: %', j;\n\n -- check if all properties are represented in the queryables\n IF NOT extra_props THEN\n FOR prop IN\n SELECT DISTINCT p->>0\n FROM jsonb_path_query(j, 'strict $.**.property') p\n WHERE p->>0 NOT IN ('id', 'datetime', 'end_datetime', 'collection')\n LOOP\n IF (queryable(prop)).nulled_wrapper IS NULL THEN\n RAISE EXCEPTION 'Term % is not found in queryables.', prop;\n END IF;\n END LOOP;\n END IF;\n\n IF j ? 'filter' THEN\n RETURN cql2_query(j->'filter');\n END IF;\n\n IF j ? 'upper' THEN\n RETURN cql2_query(jsonb_build_object('op', 'upper', 'args', j->'upper'));\n END IF;\n\n IF j ? 'lower' THEN\n RETURN cql2_query(jsonb_build_object('op', 'lower', 'args', j->'lower'));\n END IF;\n\n -- Temporal Query\n IF op ilike 't_%' or op = 'anyinteracts' THEN\n RETURN temporal_op_query(op, args);\n END IF;\n\n -- If property is a timestamp convert it to text to use with\n -- general operators\n IF j ? 'timestamp' THEN\n RETURN format('%L::timestamptz', to_tstz(j->'timestamp'));\n END IF;\n IF j ? 'interval' THEN\n RAISE EXCEPTION 'Please use temporal operators when using intervals.';\n RETURN NONE;\n END IF;\n\n -- Spatial Query\n IF op ilike 's_%' or op = 'intersects' THEN\n RETURN spatial_op_query(op, args);\n END IF;\n\n IF op IN ('a_equals','a_contains','a_contained_by','a_overlaps') THEN\n IF args->0 ? 'property' THEN\n leftarg := format('to_text_array(%s)', (queryable(args->0->>'property')).path);\n END IF;\n IF args->1 ? 'property' THEN\n rightarg := format('to_text_array(%s)', (queryable(args->1->>'property')).path);\n END IF;\n RETURN FORMAT(\n '%s %s %s',\n COALESCE(leftarg, quote_literal(to_text_array(args->0))),\n CASE op\n WHEN 'a_equals' THEN '='\n WHEN 'a_contains' THEN '@>'\n WHEN 'a_contained_by' THEN '<@'\n WHEN 'a_overlaps' THEN '&&'\n END,\n COALESCE(rightarg, quote_literal(to_text_array(args->1)))\n );\n END IF;\n\n IF op = 'in' THEN\n RAISE NOTICE 'IN : % % %', args, jsonb_build_array(args->0), args->1;\n args := jsonb_build_array(args->0) || (args->1);\n RAISE NOTICE 'IN2 : %', args;\n END IF;\n\n\n\n IF op = 'between' THEN\n args = jsonb_build_array(\n args->0,\n args->1->0,\n args->1->1\n );\n END IF;\n\n -- Make sure that args is an array and run cql2_query on\n -- each element of the array\n RAISE NOTICE 'ARGS PRE: %', args;\n IF j ? 'args' THEN\n IF jsonb_typeof(args) != 'array' THEN\n args := jsonb_build_array(args);\n END IF;\n\n IF jsonb_path_exists(args, '$[*] ? (@.property == \"id\" || @.property == \"datetime\" || @.property == \"end_datetime\" || @.property == \"collection\")') THEN\n wrapper := NULL;\n ELSE\n -- if any of the arguments are a property, try to get the property_wrapper\n FOR arg IN SELECT jsonb_path_query(args, '$[*] ? (@.property != null)') LOOP\n RAISE NOTICE 'Arg: %', arg;\n wrapper := (queryable(arg->>'property')).nulled_wrapper;\n RAISE NOTICE 'Property: %, Wrapper: %', arg, wrapper;\n IF wrapper IS NOT NULL THEN\n EXIT;\n END IF;\n END LOOP;\n\n -- if the property was not in queryables, see if any args were numbers\n IF\n wrapper IS NULL\n AND jsonb_path_exists(args, '$[*] ? (@.type()==\"number\")')\n THEN\n wrapper := 'to_float';\n END IF;\n wrapper := coalesce(wrapper, 'to_text');\n END IF;\n\n SELECT jsonb_agg(cql2_query(a, wrapper))\n INTO args\n FROM jsonb_array_elements(args) a;\n END IF;\n RAISE NOTICE 'ARGS: %', args;\n\n IF op IN ('and', 'or') THEN\n RETURN\n format(\n '(%s)',\n array_to_string(to_text_array(args), format(' %s ', upper(op)))\n );\n END IF;\n\n IF op = 'in' THEN\n RAISE NOTICE 'IN -- % %', args->0, to_text(args->0);\n RETURN format(\n '%s IN (%s)',\n to_text(args->0),\n array_to_string((to_text_array(args))[2:], ',')\n );\n END IF;\n\n -- Look up template from cql2_ops\n IF j ? 'op' THEN\n SELECT * INTO cql2op FROM cql2_ops WHERE cql2_ops.op ilike op;\n IF FOUND THEN\n -- If specific index set in queryables for a property cast other arguments to that type\n\n RETURN format(\n cql2op.template,\n VARIADIC (to_text_array(args))\n );\n ELSE\n RAISE EXCEPTION 'Operator % Not Supported.', op;\n END IF;\n END IF;\n\n\n IF wrapper IS NOT NULL THEN\n RAISE NOTICE 'Wrapping % with %', j, wrapper;\n IF j ? 'property' THEN\n RETURN format('%I(%s)', wrapper, (queryable(j->>'property')).path);\n ELSE\n RETURN format('%I(%L)', wrapper, j);\n END IF;\n ELSIF j ? 'property' THEN\n RETURN quote_ident(j->>'property');\n END IF;\n\n RETURN quote_literal(to_text(j));\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION paging_dtrange(\n j jsonb\n) RETURNS tstzrange AS $$\nDECLARE\n op text;\n filter jsonb := j->'filter';\n dtrange tstzrange := tstzrange('-infinity'::timestamptz,'infinity'::timestamptz);\n sdate timestamptz := '-infinity'::timestamptz;\n edate timestamptz := 'infinity'::timestamptz;\n jpitem jsonb;\nBEGIN\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n END IF;\n IF NOT (filter @? '$.**.op ? (@ == \"or\" || @ == \"not\")') THEN\n FOR jpitem IN SELECT j FROM jsonb_path_query(filter,'strict $.** ? (@.args[*].property == \"datetime\")'::jsonpath) j LOOP\n op := lower(jpitem->>'op');\n dtrange := parse_dtrange(jpitem->'args'->1);\n IF op IN ('<=', 'lt', 'lte', '<', 'le', 't_before') THEN\n sdate := greatest(sdate,'-infinity');\n edate := least(edate, upper(dtrange));\n ELSIF op IN ('>=', '>', 'gt', 'gte', 'ge', 't_after') THEN\n edate := least(edate, 'infinity');\n sdate := greatest(sdate, lower(dtrange));\n ELSIF op IN ('=', 'eq') THEN\n edate := least(edate, upper(dtrange));\n sdate := greatest(sdate, lower(dtrange));\n END IF;\n RAISE NOTICE '2 OP: %, ARGS: %, DTRANGE: %, SDATE: %, EDATE: %', op, jpitem->'args'->1, dtrange, sdate, edate;\n END LOOP;\n END IF;\n IF sdate > edate THEN\n RETURN 'empty'::tstzrange;\n END IF;\n RETURN tstzrange(sdate,edate, '[]');\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT SET TIME ZONE 'UTC';\n\nCREATE OR REPLACE FUNCTION paging_collections(\n IN j jsonb\n) RETURNS text[] AS $$\nDECLARE\n filter jsonb := j->'filter';\n jpitem jsonb;\n op text;\n args jsonb;\n arg jsonb;\n collections text[];\nBEGIN\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n END IF;\n IF NOT (filter @? '$.**.op ? (@ == \"or\" || @ == \"not\")') THEN\n FOR jpitem IN SELECT j FROM jsonb_path_query(filter,'strict $.** ? (@.args[*].property == \"collection\")'::jsonpath) j LOOP\n RAISE NOTICE 'JPITEM: %', jpitem;\n op := jpitem->>'op';\n args := jpitem->'args';\n IF op IN ('=', 'eq', 'in') THEN\n FOR arg IN SELECT a FROM jsonb_array_elements(args) a LOOP\n IF jsonb_typeof(arg) IN ('string', 'array') THEN\n RAISE NOTICE 'arg: %, collections: %', arg, collections;\n IF collections IS NULL OR collections = '{}'::text[] THEN\n collections := to_text_array(arg);\n ELSE\n collections := array_intersection(collections, to_text_array(arg));\n END IF;\n END IF;\n END LOOP;\n END IF;\n END LOOP;\n END IF;\n IF collections = '{}'::text[] THEN\n RETURN NULL;\n END IF;\n RETURN collections;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\nCREATE TABLE items (\n id text NOT NULL,\n geometry geometry NOT NULL,\n collection text NOT NULL,\n datetime timestamptz NOT NULL,\n end_datetime timestamptz NOT NULL,\n content JSONB NOT NULL,\n private jsonb\n)\nPARTITION BY LIST (collection)\n;\n\nCREATE INDEX \"datetime_idx\" ON items USING BTREE (datetime DESC, end_datetime ASC);\nCREATE INDEX \"geometry_idx\" ON items USING GIST (geometry);\n\nCREATE STATISTICS datetime_stats (dependencies) on datetime, end_datetime from items;\n\nALTER TABLE items ADD CONSTRAINT items_collections_fk FOREIGN KEY (collection) REFERENCES collections(id) ON DELETE CASCADE DEFERRABLE;\n\nCREATE OR REPLACE FUNCTION partition_after_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p text;\n t timestamptz := clock_timestamp();\nBEGIN\n RAISE NOTICE 'Updating partition stats %', t;\n FOR p IN SELECT DISTINCT partition\n FROM newdata n JOIN partition_sys_meta p\n ON (n.collection=p.collection AND n.datetime <@ p.partition_dtrange)\n LOOP\n PERFORM run_or_queue(format('SELECT update_partition_stats(%L, %L);', p, true));\n END LOOP;\n IF TG_OP IN ('DELETE','UPDATE') THEN\n DELETE FROM format_item_cache c USING newdata n WHERE c.collection = n.collection AND c.id = n.id;\n END IF;\n RAISE NOTICE 't: % %', t, clock_timestamp() - t;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE TRIGGER items_after_insert_trigger\nAFTER INSERT ON items\nREFERENCING NEW TABLE AS newdata\nFOR EACH STATEMENT\nEXECUTE FUNCTION partition_after_triggerfunc();\n\nCREATE TRIGGER items_after_update_trigger\nAFTER DELETE ON items\nREFERENCING OLD TABLE AS newdata\nFOR EACH STATEMENT\nEXECUTE FUNCTION partition_after_triggerfunc();\n\nCREATE TRIGGER items_after_delete_trigger\nAFTER UPDATE ON items\nREFERENCING NEW TABLE AS newdata\nFOR EACH STATEMENT\nEXECUTE FUNCTION partition_after_triggerfunc();\n\n\nCREATE OR REPLACE FUNCTION content_slim(_item jsonb) RETURNS jsonb AS $$\n SELECT strip_jsonb(_item - '{id,geometry,collection,type}'::text[], collection_base_item(_item->>'collection')) - '{id,geometry,collection,type}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_dehydrate(content jsonb) RETURNS items AS $$\n SELECT\n content->>'id' as id,\n stac_geom(content) as geometry,\n content->>'collection' as collection,\n stac_datetime(content) as datetime,\n stac_end_datetime(content) as end_datetime,\n content_slim(content) as content,\n null::jsonb as private\n ;\n$$ LANGUAGE SQL STABLE;\n\nCREATE OR REPLACE FUNCTION include_field(f text, fields jsonb DEFAULT '{}'::jsonb) RETURNS boolean AS $$\nDECLARE\n includes jsonb := fields->'include';\n excludes jsonb := fields->'exclude';\nBEGIN\n IF f IS NULL THEN\n RETURN NULL;\n END IF;\n\n\n IF\n jsonb_typeof(excludes) = 'array'\n AND jsonb_array_length(excludes)>0\n AND excludes ? f\n THEN\n RETURN FALSE;\n END IF;\n\n IF\n (\n jsonb_typeof(includes) = 'array'\n AND jsonb_array_length(includes) > 0\n AND includes ? f\n ) OR\n (\n includes IS NULL\n OR jsonb_typeof(includes) = 'null'\n OR jsonb_array_length(includes) = 0\n )\n THEN\n RETURN TRUE;\n END IF;\n\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nDROP FUNCTION IF EXISTS content_hydrate(jsonb, jsonb, jsonb);\nCREATE OR REPLACE FUNCTION content_hydrate(\n _item jsonb,\n _base_item jsonb,\n fields jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\n SELECT merge_jsonb(\n jsonb_fields(_item, fields),\n jsonb_fields(_base_item, fields)\n );\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION content_hydrate(_item items, _collection collections, fields jsonb DEFAULT '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\n content jsonb;\n base_item jsonb := _collection.base_item;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry, 20)::jsonb;\n END IF;\n output := content_hydrate(\n jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'collection', _item.collection,\n 'type', 'Feature'\n ) || _item.content,\n _collection.base_item,\n fields\n );\n\n RETURN output;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_nonhydrated(\n _item items,\n fields jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry, 20)::jsonb;\n END IF;\n output := jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'collection', _item.collection,\n 'type', 'Feature'\n ) || _item.content;\n RETURN output;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_hydrate(_item items, fields jsonb DEFAULT '{}'::jsonb) RETURNS jsonb AS $$\n SELECT content_hydrate(\n _item,\n (SELECT c FROM collections c WHERE id=_item.collection LIMIT 1),\n fields\n );\n$$ LANGUAGE SQL STABLE;\n\n\nCREATE UNLOGGED TABLE items_staging (\n content JSONB NOT NULL\n);\nCREATE UNLOGGED TABLE items_staging_ignore (\n content JSONB NOT NULL\n);\nCREATE UNLOGGED TABLE items_staging_upsert (\n content JSONB NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION items_staging_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p record;\n _partitions text[];\n part text;\n ts timestamptz := clock_timestamp();\n nrows int;\nBEGIN\n RAISE NOTICE 'Creating Partitions. %', clock_timestamp() - ts;\n\n FOR part IN WITH t AS (\n SELECT\n n.content->>'collection' as collection,\n stac_daterange(n.content->'properties') as dtr,\n partition_trunc\n FROM newdata n JOIN collections ON (n.content->>'collection'=collections.id)\n ), p AS (\n SELECT\n collection,\n COALESCE(date_trunc(partition_trunc::text, lower(dtr)),'-infinity') as d,\n tstzrange(min(lower(dtr)),max(lower(dtr)),'[]') as dtrange,\n tstzrange(min(upper(dtr)),max(upper(dtr)),'[]') as edtrange\n FROM t\n GROUP BY 1,2\n ) SELECT check_partition(collection, dtrange, edtrange) FROM p LOOP\n RAISE NOTICE 'Partition %', part;\n END LOOP;\n\n RAISE NOTICE 'Creating temp table with data to be added. %', clock_timestamp() - ts;\n DROP TABLE IF EXISTS tmpdata;\n CREATE TEMP TABLE tmpdata ON COMMIT DROP AS\n SELECT\n (content_dehydrate(content)).*\n FROM newdata;\n GET DIAGNOSTICS nrows = ROW_COUNT;\n RAISE NOTICE 'Added % rows to tmpdata. %', nrows, clock_timestamp() - ts;\n\n RAISE NOTICE 'Doing the insert. %', clock_timestamp() - ts;\n IF TG_TABLE_NAME = 'items_staging' THEN\n INSERT INTO items\n SELECT * FROM tmpdata;\n GET DIAGNOSTICS nrows = ROW_COUNT;\n RAISE NOTICE 'Inserted % rows to items. %', nrows, clock_timestamp() - ts;\n ELSIF TG_TABLE_NAME = 'items_staging_ignore' THEN\n INSERT INTO items\n SELECT * FROM tmpdata\n ON CONFLICT DO NOTHING;\n GET DIAGNOSTICS nrows = ROW_COUNT;\n RAISE NOTICE 'Inserted % rows to items. %', nrows, clock_timestamp() - ts;\n ELSIF TG_TABLE_NAME = 'items_staging_upsert' THEN\n DELETE FROM items i USING tmpdata s\n WHERE\n i.id = s.id\n AND i.collection = s.collection\n AND i IS DISTINCT FROM s\n ;\n GET DIAGNOSTICS nrows = ROW_COUNT;\n RAISE NOTICE 'Deleted % rows from items. %', nrows, clock_timestamp() - ts;\n INSERT INTO items AS t\n SELECT * FROM tmpdata\n ON CONFLICT DO NOTHING;\n GET DIAGNOSTICS nrows = ROW_COUNT;\n RAISE NOTICE 'Inserted % rows to items. %', nrows, clock_timestamp() - ts;\n END IF;\n\n RAISE NOTICE 'Deleting data from staging table. %', clock_timestamp() - ts;\n DELETE FROM items_staging;\n RAISE NOTICE 'Done. %', clock_timestamp() - ts;\n\n RETURN NULL;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER items_staging_insert_trigger AFTER INSERT ON items_staging REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\nCREATE TRIGGER items_staging_insert_ignore_trigger AFTER INSERT ON items_staging_ignore REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\nCREATE TRIGGER items_staging_insert_upsert_trigger AFTER INSERT ON items_staging_upsert REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\n\nCREATE OR REPLACE FUNCTION item_by_id(_id text, _collection text DEFAULT NULL) RETURNS items AS\n$$\nDECLARE\n i items%ROWTYPE;\nBEGIN\n SELECT * INTO i FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection) LIMIT 1;\n RETURN i;\nEND;\n$$ LANGUAGE PLPGSQL STABLE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION get_item(_id text, _collection text DEFAULT NULL) RETURNS jsonb AS $$\n SELECT content_hydrate(items) FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection);\n$$ LANGUAGE SQL STABLE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION delete_item(_id text, _collection text DEFAULT NULL) RETURNS VOID AS $$\nDECLARE\nout items%ROWTYPE;\nBEGIN\n DELETE FROM items WHERE id = _id AND (_collection IS NULL OR collection=_collection) RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n--/*\nCREATE OR REPLACE FUNCTION create_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION update_item(content jsonb) RETURNS VOID AS $$\nDECLARE\n old items %ROWTYPE;\n out items%ROWTYPE;\nBEGIN\n PERFORM delete_item(content->>'id', content->>'collection');\n PERFORM create_item(content);\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION create_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION collection_bbox(id text) RETURNS jsonb AS $$\n SELECT (replace(replace(replace(st_extent(geometry)::text,'BOX(','[['),')',']]'),' ',','))::jsonb\n FROM items WHERE collection=$1;\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION collection_temporal_extent(id text) RETURNS jsonb AS $$\n SELECT to_jsonb(array[array[min(datetime)::text, max(datetime)::text]])\n FROM items WHERE collection=$1;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection_extents() RETURNS VOID AS $$\nUPDATE collections SET\n content = content ||\n jsonb_build_object(\n 'extent', jsonb_build_object(\n 'spatial', jsonb_build_object(\n 'bbox', collection_bbox(collections.id)\n ),\n 'temporal', jsonb_build_object(\n 'interval', collection_temporal_extent(collections.id)\n )\n )\n )\n;\n$$ LANGUAGE SQL;\nCREATE TABLE partition_stats (\n partition text PRIMARY KEY,\n dtrange tstzrange,\n edtrange tstzrange,\n spatial geometry,\n last_updated timestamptz,\n keys text[]\n) WITH (FILLFACTOR=90);\n\nCREATE INDEX partitions_range_idx ON partition_stats USING GIST(dtrange);\n\n\nCREATE OR REPLACE FUNCTION constraint_tstzrange(expr text) RETURNS tstzrange AS $$\n WITH t AS (\n SELECT regexp_matches(\n expr,\n E'\\\\(''\\([0-9 :+-]*\\)''\\\\).*\\\\(''\\([0-9 :+-]*\\)''\\\\)'\n ) AS m\n ) SELECT tstzrange(m[1]::timestamptz, m[2]::timestamptz) FROM t\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE STRICT;\n\nCREATE OR REPLACE FUNCTION dt_constraint(coid oid, OUT dt tstzrange, OUT edt tstzrange) RETURNS RECORD AS $$\nDECLARE\n expr text := pg_get_constraintdef(coid);\n matches timestamptz[];\nBEGIN\n IF expr LIKE '%NULL%' THEN\n dt := tstzrange(null::timestamptz, null::timestamptz);\n edt := tstzrange(null::timestamptz, null::timestamptz);\n RETURN;\n END IF;\n WITH f AS (SELECT (regexp_matches(expr, E'([0-9]{4}-[0-1][0-9]-[0-3][0-9] [0-2][0-9]:[0-5][0-9]:[0-5][0-9]\\.?[0-9]*)', 'g'))[1] f)\n SELECT array_agg(f::timestamptz) INTO matches FROM f;\n IF cardinality(matches) = 4 THEN\n dt := tstzrange(matches[1], matches[2],'[]');\n edt := tstzrange(matches[3], matches[4], '[]');\n RETURN;\n ELSIF cardinality(matches) = 2 THEN\n edt := tstzrange(matches[1], matches[2],'[]');\n RETURN;\n END IF;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\nCREATE OR REPLACE VIEW partition_sys_meta AS\nSELECT\n relid::text as partition,\n replace(replace(CASE WHEN level = 1 THEN pg_get_expr(c.relpartbound, c.oid)\n ELSE pg_get_expr(parent.relpartbound, parent.oid)\n END, 'FOR VALUES IN (''',''), ''')','') AS collection,\n level,\n c.reltuples,\n c.relhastriggers,\n COALESCE(constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as partition_dtrange,\n COALESCE((dt_constraint(edt.oid)).dt, constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as constraint_dtrange,\n COALESCE((dt_constraint(edt.oid)).edt, tstzrange('-infinity', 'infinity','[]')) as constraint_edtrange\nFROM\n pg_partition_tree('items')\n JOIN pg_class c ON (relid::regclass = c.oid)\n JOIN pg_class parent ON (parentrelid::regclass = parent.oid AND isleaf)\n LEFT JOIN pg_constraint edt ON (conrelid=c.oid AND contype='c')\nWHERE isleaf\n;\n\nCREATE VIEW partitions_view AS\nSELECT\n relid::text as partition,\n replace(replace(CASE WHEN level = 1 THEN pg_get_expr(c.relpartbound, c.oid)\n ELSE pg_get_expr(parent.relpartbound, parent.oid)\n END, 'FOR VALUES IN (''',''), ''')','') AS collection,\n level,\n c.reltuples,\n c.relhastriggers,\n COALESCE(constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as partition_dtrange,\n COALESCE((dt_constraint(edt.oid)).dt, constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as constraint_dtrange,\n COALESCE((dt_constraint(edt.oid)).edt, tstzrange('-infinity', 'infinity','[]')) as constraint_edtrange,\n dtrange,\n edtrange,\n spatial,\n last_updated\nFROM\n pg_partition_tree('items')\n JOIN pg_class c ON (relid::regclass = c.oid)\n JOIN pg_class parent ON (parentrelid::regclass = parent.oid AND isleaf)\n LEFT JOIN pg_constraint edt ON (conrelid=c.oid AND contype='c')\n LEFT JOIN partition_stats ON (relid::text=partition)\nWHERE isleaf\n;\n\nCREATE MATERIALIZED VIEW partitions AS\nSELECT * FROM partitions_view;\nCREATE UNIQUE INDEX ON partitions (partition);\n\nCREATE MATERIALIZED VIEW partition_steps AS\nSELECT\n partition as name,\n date_trunc('month',lower(partition_dtrange)) as sdate,\n date_trunc('month', upper(partition_dtrange)) + '1 month'::interval as edate\n FROM partitions_view WHERE partition_dtrange IS NOT NULL AND partition_dtrange != 'empty'::tstzrange\n ORDER BY dtrange ASC\n;\n\n\nCREATE OR REPLACE FUNCTION update_partition_stats_q(_partition text, istrigger boolean default false) RETURNS VOID AS $$\nDECLARE\nBEGIN\n PERFORM run_or_queue(\n format('SELECT update_partition_stats(%L, %L);', _partition, istrigger)\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION update_partition_stats(_partition text, istrigger boolean default false) RETURNS VOID AS $$\nDECLARE\n dtrange tstzrange;\n edtrange tstzrange;\n cdtrange tstzrange;\n cedtrange tstzrange;\n extent geometry;\n collection text;\nBEGIN\n RAISE NOTICE 'Updating stats for %.', _partition;\n EXECUTE format(\n $q$\n SELECT\n tstzrange(min(datetime), max(datetime),'[]'),\n tstzrange(min(end_datetime), max(end_datetime), '[]')\n FROM %I\n $q$,\n _partition\n ) INTO dtrange, edtrange;\n extent := st_estimatedextent('pgstac', _partition, 'geometry');\n INSERT INTO partition_stats (partition, dtrange, edtrange, spatial, last_updated)\n SELECT _partition, dtrange, edtrange, extent, now()\n ON CONFLICT (partition) DO\n UPDATE SET\n dtrange=EXCLUDED.dtrange,\n edtrange=EXCLUDED.edtrange,\n spatial=EXCLUDED.spatial,\n last_updated=EXCLUDED.last_updated\n ;\n\n SELECT\n constraint_dtrange, constraint_edtrange, pv.collection\n INTO cdtrange, cedtrange, collection\n FROM partitions_view pv WHERE partition = _partition;\n REFRESH MATERIALIZED VIEW partitions;\n REFRESH MATERIALIZED VIEW partition_steps;\n\n\n RAISE NOTICE 'Checking if we need to modify constraints...';\n RAISE NOTICE 'cdtrange: % dtrange: % cedtrange: % edtrange: %',cdtrange, dtrange, cedtrange, edtrange;\n IF\n (cdtrange IS DISTINCT FROM dtrange OR edtrange IS DISTINCT FROM cedtrange)\n AND NOT istrigger\n THEN\n RAISE NOTICE 'Modifying Constraints';\n RAISE NOTICE 'Existing % %', cdtrange, cedtrange;\n RAISE NOTICE 'New % %', dtrange, edtrange;\n PERFORM drop_table_constraints(_partition);\n PERFORM create_table_constraints(_partition, dtrange, edtrange);\n REFRESH MATERIALIZED VIEW partitions;\n REFRESH MATERIALIZED VIEW partition_steps;\n END IF;\n RAISE NOTICE 'Checking if we need to update collection extents.';\n IF get_setting_bool('update_collection_extent') THEN\n RAISE NOTICE 'updating collection extent for %', collection;\n PERFORM run_or_queue(format($q$\n UPDATE collections\n SET content = jsonb_set_lax(\n content,\n '{extent}'::text[],\n collection_extent(%L, FALSE),\n true,\n 'use_json_null'\n ) WHERE id=%L\n ;\n $q$, collection, collection));\n ELSE\n RAISE NOTICE 'Not updating collection extent for %', collection;\n END IF;\n\nEND;\n$$ LANGUAGE PLPGSQL STRICT SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION partition_name( IN collection text, IN dt timestamptz, OUT partition_name text, OUT partition_range tstzrange) AS $$\nDECLARE\n c RECORD;\n parent_name text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n parent_name := format('_items_%s', c.key);\n\n\n IF c.partition_trunc = 'year' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYY'));\n ELSIF c.partition_trunc = 'month' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYYMM'));\n ELSE\n partition_name := parent_name;\n partition_range := tstzrange('-infinity'::timestamptz, 'infinity'::timestamptz, '[]');\n END IF;\n IF partition_range IS NULL THEN\n partition_range := tstzrange(\n date_trunc(c.partition_trunc::text, dt),\n date_trunc(c.partition_trunc::text, dt) + concat('1 ', c.partition_trunc)::interval\n );\n END IF;\n RETURN;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION drop_table_constraints(t text) RETURNS text AS $$\nDECLARE\n q text;\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM partitions_view WHERE partition=t) THEN\n RETURN NULL;\n END IF;\n FOR q IN SELECT FORMAT(\n $q$\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n $q$,\n t,\n conname\n ) FROM pg_constraint\n WHERE conrelid=t::regclass::oid AND contype='c'\n LOOP\n EXECUTE q;\n END LOOP;\n RETURN t;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE OR REPLACE FUNCTION create_table_constraints(t text, _dtrange tstzrange, _edtrange tstzrange) RETURNS text AS $$\nDECLARE\n q text;\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM partitions_view WHERE partition=t) THEN\n RETURN NULL;\n END IF;\n RAISE NOTICE 'Creating Table Constraints for % % %', t, _dtrange, _edtrange;\n IF _dtrange = 'empty' AND _edtrange = 'empty' THEN\n q :=format(\n $q$\n DO $block$\n BEGIN\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n ALTER TABLE %I\n ADD CONSTRAINT %I\n CHECK (((datetime IS NULL) AND (end_datetime IS NULL))) NOT VALID\n ;\n ALTER TABLE %I\n VALIDATE CONSTRAINT %I\n ;\n\n\n\n EXCEPTION WHEN others THEN\n RAISE WARNING '%%, Issue Altering Constraints. Please run update_partition_stats(%I)', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n $block$;\n $q$,\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t),\n t\n );\n ELSE\n q :=format(\n $q$\n DO $block$\n BEGIN\n\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n ALTER TABLE %I\n ADD CONSTRAINT %I\n CHECK (\n (datetime >= %L)\n AND (datetime <= %L)\n AND (end_datetime >= %L)\n AND (end_datetime <= %L)\n ) NOT VALID\n ;\n ALTER TABLE %I\n VALIDATE CONSTRAINT %I\n ;\n\n\n\n EXCEPTION WHEN others THEN\n RAISE WARNING '%%, Issue Altering Constraints. Please run update_partition_stats(%I)', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n $block$;\n $q$,\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t),\n lower(_dtrange),\n upper(_dtrange),\n lower(_edtrange),\n upper(_edtrange),\n t,\n format('%s_dt', t),\n t\n );\n END IF;\n PERFORM run_or_queue(q);\n RETURN t;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION check_partition(\n _collection text,\n _dtrange tstzrange,\n _edtrange tstzrange\n) RETURNS text AS $$\nDECLARE\n c RECORD;\n pm RECORD;\n _partition_name text;\n _partition_dtrange tstzrange;\n _constraint_dtrange tstzrange;\n _constraint_edtrange tstzrange;\n q text;\n deferrable_q text;\n err_context text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=_collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', _collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n\n IF c.partition_trunc IS NOT NULL THEN\n _partition_dtrange := tstzrange(\n date_trunc(c.partition_trunc, lower(_dtrange)),\n date_trunc(c.partition_trunc, lower(_dtrange)) + (concat('1 ', c.partition_trunc))::interval,\n '[)'\n );\n ELSE\n _partition_dtrange := '[-infinity, infinity]'::tstzrange;\n END IF;\n\n IF NOT _partition_dtrange @> _dtrange THEN\n RAISE EXCEPTION 'dtrange % is greater than the partition size % for collection %', _dtrange, c.partition_trunc, _collection;\n END IF;\n\n\n IF c.partition_trunc = 'year' THEN\n _partition_name := format('_items_%s_%s', c.key, to_char(lower(_partition_dtrange),'YYYY'));\n ELSIF c.partition_trunc = 'month' THEN\n _partition_name := format('_items_%s_%s', c.key, to_char(lower(_partition_dtrange),'YYYYMM'));\n ELSE\n _partition_name := format('_items_%s', c.key);\n END IF;\n\n SELECT * INTO pm FROM partition_sys_meta WHERE collection=_collection AND partition_dtrange @> _dtrange;\n IF FOUND THEN\n RAISE NOTICE '% % %', _edtrange, _dtrange, pm;\n _constraint_edtrange :=\n tstzrange(\n least(\n lower(_edtrange),\n nullif(lower(pm.constraint_edtrange), '-infinity')\n ),\n greatest(\n upper(_edtrange),\n nullif(upper(pm.constraint_edtrange), 'infinity')\n ),\n '[]'\n );\n _constraint_dtrange :=\n tstzrange(\n least(\n lower(_dtrange),\n nullif(lower(pm.constraint_dtrange), '-infinity')\n ),\n greatest(\n upper(_dtrange),\n nullif(upper(pm.constraint_dtrange), 'infinity')\n ),\n '[]'\n );\n\n IF pm.constraint_edtrange @> _edtrange AND pm.constraint_dtrange @> _dtrange THEN\n RETURN pm.partition;\n ELSE\n PERFORM drop_table_constraints(_partition_name);\n END IF;\n ELSE\n _constraint_edtrange := _edtrange;\n _constraint_dtrange := _dtrange;\n END IF;\n RAISE NOTICE 'EXISTING CONSTRAINTS % %, NEW % %', pm.constraint_dtrange, pm.constraint_edtrange, _constraint_dtrange, _constraint_edtrange;\n RAISE NOTICE 'Creating partition % %', _partition_name, _partition_dtrange;\n IF c.partition_trunc IS NULL THEN\n q := format(\n $q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n GRANT ALL ON %I to pgstac_ingest;\n $q$,\n _partition_name,\n _collection,\n concat(_partition_name,'_pk'),\n _partition_name,\n _partition_name\n );\n ELSE\n q := format(\n $q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L) PARTITION BY RANGE (datetime);\n CREATE TABLE IF NOT EXISTS %I partition OF %I FOR VALUES FROM (%L) TO (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n GRANT ALL ON %I TO pgstac_ingest;\n $q$,\n format('_items_%s', c.key),\n _collection,\n _partition_name,\n format('_items_%s', c.key),\n lower(_partition_dtrange),\n upper(_partition_dtrange),\n format('%s_pk', _partition_name),\n _partition_name,\n _partition_name\n );\n END IF;\n\n BEGIN\n EXECUTE q;\n EXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', _partition_name;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\n END;\n PERFORM maintain_partitions(_partition_name);\n PERFORM update_partition_stats_q(_partition_name, true);\n REFRESH MATERIALIZED VIEW partitions;\n REFRESH MATERIALIZED VIEW partition_steps;\n RETURN _partition_name;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION repartition(_collection text, _partition_trunc text, triggered boolean DEFAULT FALSE) RETURNS text AS $$\nDECLARE\n c RECORD;\n q text;\n from_trunc text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=_collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', _collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n IF triggered THEN\n RAISE NOTICE 'Converting % to % partitioning via Trigger', _collection, _partition_trunc;\n ELSE\n RAISE NOTICE 'Converting % from using % to % partitioning', _collection, c.partition_trunc, _partition_trunc;\n IF c.partition_trunc IS NOT DISTINCT FROM _partition_trunc THEN\n RAISE NOTICE 'Collection % already set to use partition by %', _collection, _partition_trunc;\n RETURN _collection;\n END IF;\n END IF;\n\n IF EXISTS (SELECT 1 FROM partitions_view WHERE collection=_collection LIMIT 1) THEN\n EXECUTE format(\n $q$\n CREATE TEMP TABLE changepartitionstaging ON COMMIT DROP AS SELECT * FROM %I;\n DROP TABLE IF EXISTS %I CASCADE;\n WITH p AS (\n SELECT\n collection,\n CASE\n WHEN %L IS NULL THEN '-infinity'::timestamptz\n ELSE date_trunc(%L, datetime)\n END as d,\n tstzrange(min(datetime),max(datetime),'[]') as dtrange,\n tstzrange(min(end_datetime),max(end_datetime),'[]') as edtrange\n FROM changepartitionstaging\n GROUP BY 1,2\n ) SELECT check_partition(collection, dtrange, edtrange) FROM p;\n INSERT INTO items SELECT * FROM changepartitionstaging;\n DROP TABLE changepartitionstaging;\n $q$,\n concat('_items_', c.key),\n concat('_items_', c.key),\n c.partition_trunc,\n c.partition_trunc\n );\n END IF;\n RETURN _collection;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE OR REPLACE FUNCTION collections_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n q text;\n partition_name text := format('_items_%s', NEW.key);\n partition_exists boolean := false;\n partition_empty boolean := true;\n err_context text;\n loadtemp boolean := FALSE;\nBEGIN\n RAISE NOTICE 'Collection Trigger. % %', NEW.id, NEW.key;\n IF TG_OP = 'UPDATE' AND NEW.partition_trunc IS DISTINCT FROM OLD.partition_trunc THEN\n PERFORM repartition(NEW.id, NEW.partition_trunc, TRUE);\n END IF;\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER collections_trigger AFTER\nINSERT\nOR\nUPDATE ON collections\nFOR EACH ROW EXECUTE FUNCTION collections_trigger_func();\n\nCREATE OR REPLACE FUNCTION chunker(\n IN _where text,\n OUT s timestamptz,\n OUT e timestamptz\n) RETURNS SETOF RECORD AS $$\nDECLARE\n explain jsonb;\nBEGIN\n IF _where IS NULL THEN\n _where := ' TRUE ';\n END IF;\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s;', _where)\n INTO explain;\n\n RETURN QUERY\n WITH t AS (\n SELECT j->>0 as p FROM\n jsonb_path_query(\n explain,\n 'strict $.**.\"Relation Name\" ? (@ != null)'\n ) j\n ),\n parts AS (\n SELECT sdate, edate FROM t JOIN partition_steps ON (t.p = name)\n ),\n times AS (\n SELECT sdate FROM parts\n UNION\n SELECT edate FROM parts\n ),\n uniq AS (\n SELECT DISTINCT sdate FROM times ORDER BY sdate\n ),\n last AS (\n SELECT sdate, lead(sdate, 1) over () as edate FROM uniq\n )\n SELECT sdate, edate FROM last WHERE edate IS NOT NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION partition_queries(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN partitions text[] DEFAULT NULL\n) RETURNS SETOF text AS $$\nDECLARE\n query text;\n sdate timestamptz;\n edate timestamptz;\nBEGIN\nIF _where IS NULL OR trim(_where) = '' THEN\n _where = ' TRUE ';\nEND IF;\nRAISE NOTICE 'Getting chunks for % %', _where, _orderby;\nIF _orderby ILIKE 'datetime d%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 DESC LOOP\n RETURN NEXT format($q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n $q$,\n sdate,\n edate,\n _where,\n _orderby\n );\n END LOOP;\nELSIF _orderby ILIKE 'datetime a%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 ASC LOOP\n RETURN NEXT format($q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n $q$,\n sdate,\n edate,\n _where,\n _orderby\n );\n END LOOP;\nELSE\n query := format($q$\n SELECT * FROM items\n WHERE %s\n ORDER BY %s\n $q$, _where, _orderby\n );\n\n RETURN NEXT query;\n RETURN;\nEND IF;\n\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION partition_query_view(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN _limit int DEFAULT 10\n) RETURNS text AS $$\n WITH p AS (\n SELECT * FROM partition_queries(_where, _orderby) p\n )\n SELECT\n CASE WHEN EXISTS (SELECT 1 FROM p) THEN\n (SELECT format($q$\n SELECT * FROM (\n %s\n ) total LIMIT %s\n $q$,\n string_agg(\n format($q$ SELECT * FROM ( %s ) AS sub $q$, p),\n '\n UNION ALL\n '\n ),\n _limit\n ))\n ELSE NULL\n END FROM p;\n$$ LANGUAGE SQL IMMUTABLE;\n\n\n\n\nCREATE OR REPLACE FUNCTION stac_search_to_where(j jsonb) RETURNS text AS $$\nDECLARE\n where_segments text[];\n _where text;\n dtrange tstzrange;\n collections text[];\n geom geometry;\n sdate timestamptz;\n edate timestamptz;\n filterlang text;\n filter jsonb := j->'filter';\nBEGIN\n IF j ? 'ids' THEN\n where_segments := where_segments || format('id = ANY (%L) ', to_text_array(j->'ids'));\n END IF;\n\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n where_segments := where_segments || format('collection = ANY (%L) ', collections);\n END IF;\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n\n where_segments := where_segments || format(' datetime <= %L::timestamptz AND end_datetime >= %L::timestamptz ',\n edate,\n sdate\n );\n END IF;\n\n geom := stac_geom(j);\n IF geom IS NOT NULL THEN\n where_segments := where_segments || format('st_intersects(geometry, %L)',geom);\n END IF;\n\n filterlang := COALESCE(\n j->>'filter-lang',\n get_setting('default_filter_lang', j->'conf')\n );\n IF NOT filter @? '$.**.op' THEN\n filterlang := 'cql-json';\n END IF;\n\n IF filterlang NOT IN ('cql-json','cql2-json') AND j ? 'filter' THEN\n RAISE EXCEPTION '% is not a supported filter-lang. Please use cql-json or cql2-json.', filterlang;\n END IF;\n\n IF j ? 'query' AND j ? 'filter' THEN\n RAISE EXCEPTION 'Can only use either query or filter at one time.';\n END IF;\n\n IF j ? 'query' THEN\n filter := query_to_cql2(j->'query');\n ELSIF filterlang = 'cql-json' THEN\n filter := cql1_to_cql2(filter);\n END IF;\n RAISE NOTICE 'FILTER: %', filter;\n where_segments := where_segments || cql2_query(filter);\n IF cardinality(where_segments) < 1 THEN\n RETURN ' TRUE ';\n END IF;\n\n _where := array_to_string(array_remove(where_segments, NULL), ' AND ');\n\n IF _where IS NULL OR BTRIM(_where) = '' THEN\n RETURN ' TRUE ';\n END IF;\n RETURN _where;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION parse_sort_dir(_dir text, reverse boolean default false) RETURNS text AS $$\n WITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n ) SELECT\n CASE\n WHEN NOT reverse THEN d\n WHEN d = 'ASC' THEN 'DESC'\n WHEN d = 'DESC' THEN 'ASC'\n END\n FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION sort_dir_to_op(_dir text, prev boolean default false) RETURNS text AS $$\n WITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n ) SELECT\n CASE\n WHEN d = 'ASC' AND prev THEN '<='\n WHEN d = 'DESC' AND prev THEN '>='\n WHEN d = 'ASC' THEN '>='\n WHEN d = 'DESC' THEN '<='\n END\n FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION sort_sqlorderby(\n _search jsonb DEFAULT NULL,\n reverse boolean DEFAULT FALSE\n) RETURNS text AS $$\n WITH sortby AS (\n SELECT coalesce(_search->'sortby','[{\"field\":\"datetime\", \"direction\":\"desc\"}]') as sort\n ), withid AS (\n SELECT CASE\n WHEN sort @? '$[*] ? (@.field == \"id\")' THEN sort\n ELSE sort || '[{\"field\":\"id\", \"direction\":\"desc\"}]'::jsonb\n END as sort\n FROM sortby\n ), withid_rows AS (\n SELECT jsonb_array_elements(sort) as value FROM withid\n ),sorts AS (\n SELECT\n coalesce(\n (queryable(value->>'field')).expression\n ) as key,\n parse_sort_dir(value->>'direction', reverse) as dir\n FROM withid_rows\n )\n SELECT array_to_string(\n array_agg(concat(key, ' ', dir)),\n ', '\n ) FROM sorts;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_sort_dir(sort_item jsonb) RETURNS text AS $$\n SELECT CASE WHEN sort_item->>'direction' ILIKE 'desc%' THEN 'DESC' ELSE 'ASC' END;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION get_token_val_str(\n _field text,\n _item items\n) RETURNS text AS $$\nDECLARE\n q text;\n literal text;\nBEGIN\n q := format($q$ SELECT quote_literal(%s) FROM (SELECT $1.*) as r;$q$, _field);\n EXECUTE q INTO literal USING _item;\n RETURN literal;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n\nCREATE OR REPLACE FUNCTION get_token_record(IN _token text, OUT prev BOOLEAN, OUT item items) RETURNS RECORD AS $$\nDECLARE\n _itemid text := _token;\n _collectionid text;\nBEGIN\n IF _token IS NULL THEN\n RETURN;\n END IF;\n RAISE NOTICE 'Looking for token: %', _token;\n prev := FALSE;\n IF _token ILIKE 'prev:%' THEN\n _itemid := replace(_token, 'prev:','');\n prev := TRUE;\n ELSIF _token ILIKE 'next:%' THEN\n _itemid := replace(_token, 'next:', '');\n END IF;\n SELECT id INTO _collectionid FROM collections WHERE _itemid LIKE concat(id,':%');\n IF FOUND THEN\n _itemid := replace(_itemid, concat(_collectionid,':'), '');\n SELECT * INTO item FROM items WHERE id=_itemid AND collection=_collectionid;\n ELSE\n SELECT * INTO item FROM items WHERE id=_itemid;\n END IF;\n IF item IS NULL THEN\n RAISE EXCEPTION 'Could not find item using token: % item: % collection: %', _token, _itemid, _collectionid;\n END IF;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\n\nCREATE OR REPLACE FUNCTION get_token_filter(\n _sortby jsonb DEFAULT '[{\"field\":\"datetime\",\"direction\":\"desc\"}]'::jsonb,\n token_item items DEFAULT NULL,\n prev boolean DEFAULT FALSE,\n inclusive boolean DEFAULT FALSE\n) RETURNS text AS $$\nDECLARE\n ltop text := '<';\n gtop text := '>';\n dir text;\n sort record;\n orfilter text := '';\n orfilters text[] := '{}'::text[];\n andfilters text[] := '{}'::text[];\n output text;\n token_where text;\nBEGIN\n IF _sortby IS NULL OR _sortby = '[]'::jsonb THEN\n _sortby := '[{\"field\":\"datetime\",\"direction\":\"desc\"}]'::jsonb;\n END IF;\n _sortby := _sortby || jsonb_build_object('field','id','direction',_sortby->0->>'direction');\n RAISE NOTICE 'Getting Token Filter. % %', _sortby, token_item;\n IF inclusive THEN\n orfilters := orfilters || format('( id=%L AND collection=%L )' , token_item.id, token_item.collection);\n END IF;\n\n FOR sort IN\n WITH s1 AS (\n SELECT\n _row,\n (queryable(value->>'field')).expression as _field,\n (value->>'field' = 'id') as _isid,\n get_sort_dir(value) as _dir\n FROM jsonb_array_elements(_sortby)\n WITH ORDINALITY AS t(value, _row)\n )\n SELECT\n _row,\n _field,\n _dir,\n get_token_val_str(_field, token_item) as _val\n FROM s1\n WHERE _row <= (SELECT min(_row) FROM s1 WHERE _isid)\n LOOP\n orfilter := NULL;\n RAISE NOTICE 'SORT: %', sort;\n IF sort._val IS NOT NULL AND ((prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC')) THEN\n orfilter := format($f$(\n (%s %s %s) OR (%s IS NULL)\n )$f$,\n sort._field,\n ltop,\n sort._val,\n sort._val\n );\n ELSIF sort._val IS NULL AND ((prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC')) THEN\n RAISE NOTICE '< but null';\n orfilter := format('%s IS NOT NULL', sort._field);\n ELSIF sort._val IS NULL THEN\n RAISE NOTICE '> but null';\n ELSE\n orfilter := format($f$(\n (%s %s %s) OR (%s IS NULL)\n )$f$,\n sort._field,\n gtop,\n sort._val,\n sort._field\n );\n END IF;\n RAISE NOTICE 'ORFILTER: %', orfilter;\n\n IF orfilter IS NOT NULL THEN\n IF sort._row = 1 THEN\n orfilters := orfilters || orfilter;\n ELSE\n orfilters := orfilters || format('(%s AND %s)', array_to_string(andfilters, ' AND '), orfilter);\n END IF;\n END IF;\n IF sort._val IS NOT NULL THEN\n andfilters := andfilters || format('%s = %s', sort._field, sort._val);\n ELSE\n andfilters := andfilters || format('%s IS NULL', sort._field);\n END IF;\n END LOOP;\n\n output := array_to_string(orfilters, ' OR ');\n\n token_where := concat('(',coalesce(output,'true'),')');\n IF trim(token_where) = '' THEN\n token_where := NULL;\n END IF;\n RAISE NOTICE 'TOKEN_WHERE: %',token_where;\n RETURN token_where;\n END;\n$$ LANGUAGE PLPGSQL SET transform_null_equals TO TRUE\n;\n\nCREATE OR REPLACE FUNCTION search_tohash(jsonb) RETURNS jsonb AS $$\n SELECT $1 - '{token,limit,context,includes,excludes}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION search_hash(jsonb, jsonb) RETURNS text AS $$\n SELECT md5(concat(search_tohash($1)::text,$2::text));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TABLE IF NOT EXISTS searches(\n hash text GENERATED ALWAYS AS (search_hash(search, metadata)) STORED PRIMARY KEY,\n search jsonb NOT NULL,\n _where text,\n orderby text,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n metadata jsonb DEFAULT '{}'::jsonb NOT NULL\n);\n\nCREATE TABLE IF NOT EXISTS search_wheres(\n id bigint generated always as identity primary key,\n _where text NOT NULL,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n statslastupdated timestamptz,\n estimated_count bigint,\n estimated_cost float,\n time_to_estimate float,\n total_count bigint,\n time_to_count float,\n partitions text[]\n);\n\nCREATE INDEX IF NOT EXISTS search_wheres_partitions ON search_wheres USING GIN (partitions);\nCREATE UNIQUE INDEX IF NOT EXISTS search_wheres_where ON search_wheres ((md5(_where)));\n\nCREATE OR REPLACE FUNCTION where_stats(inwhere text, updatestats boolean default false, conf jsonb default null) RETURNS search_wheres AS $$\nDECLARE\n t timestamptz;\n i interval;\n explain_json jsonb;\n partitions text[];\n sw search_wheres%ROWTYPE;\n inwhere_hash text := md5(inwhere);\n _context text := lower(context(conf));\n _stats_ttl interval := context_stats_ttl(conf);\n _estimated_cost float := context_estimated_cost(conf);\n _estimated_count int := context_estimated_count(conf);\n ro bool := pgstac.readonly(conf);\nBEGIN\n IF ro THEN\n updatestats := FALSE;\n END IF;\n\n IF _context = 'off' THEN\n sw._where := inwhere;\n return sw;\n END IF;\n\n SELECT * INTO sw FROM search_wheres WHERE md5(_where)=inwhere_hash FOR UPDATE;\n\n -- Update statistics if explicitly set, if statistics do not exist, or statistics ttl has expired\n IF NOT updatestats THEN\n RAISE NOTICE 'Checking if update is needed for: % .', inwhere;\n RAISE NOTICE 'Stats Last Updated: %', sw.statslastupdated;\n RAISE NOTICE 'TTL: %, Age: %', _stats_ttl, now() - sw.statslastupdated;\n RAISE NOTICE 'Context: %, Existing Total: %', _context, sw.total_count;\n IF\n (\n sw.statslastupdated IS NULL\n OR (now() - sw.statslastupdated) > _stats_ttl\n OR (context(conf) != 'off' AND sw.total_count IS NULL)\n ) AND NOT ro\n THEN\n updatestats := TRUE;\n END IF;\n END IF;\n\n sw._where := inwhere;\n sw.lastused := now();\n sw.usecount := coalesce(sw.usecount,0) + 1;\n\n IF NOT updatestats THEN\n UPDATE search_wheres SET\n lastused = sw.lastused,\n usecount = sw.usecount\n WHERE md5(_where) = inwhere_hash\n RETURNING * INTO sw\n ;\n RETURN sw;\n END IF;\n\n -- Use explain to get estimated count/cost and a list of the partitions that would be hit by the query\n t := clock_timestamp();\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s', inwhere)\n INTO explain_json;\n RAISE NOTICE 'Time for just the explain: %', clock_timestamp() - t;\n i := clock_timestamp() - t;\n\n sw.statslastupdated := now();\n sw.estimated_count := explain_json->0->'Plan'->'Plan Rows';\n sw.estimated_cost := explain_json->0->'Plan'->'Total Cost';\n sw.time_to_estimate := extract(epoch from i);\n\n RAISE NOTICE 'ESTIMATED_COUNT: % < %', sw.estimated_count, _estimated_count;\n RAISE NOTICE 'ESTIMATED_COST: % < %', sw.estimated_cost, _estimated_cost;\n\n -- Do a full count of rows if context is set to on or if auto is set and estimates are low enough\n IF\n _context = 'on'\n OR\n ( _context = 'auto' AND\n (\n sw.estimated_count < _estimated_count\n AND\n sw.estimated_cost < _estimated_cost\n )\n )\n THEN\n t := clock_timestamp();\n RAISE NOTICE 'Calculating actual count...';\n EXECUTE format(\n 'SELECT count(*) FROM items WHERE %s',\n inwhere\n ) INTO sw.total_count;\n i := clock_timestamp() - t;\n RAISE NOTICE 'Actual Count: % -- %', sw.total_count, i;\n sw.time_to_count := extract(epoch FROM i);\n ELSE\n sw.total_count := NULL;\n sw.time_to_count := NULL;\n END IF;\n\n IF NOT ro THEN\n INSERT INTO search_wheres\n (_where, lastused, usecount, statslastupdated, estimated_count, estimated_cost, time_to_estimate, partitions, total_count, time_to_count)\n SELECT sw._where, sw.lastused, sw.usecount, sw.statslastupdated, sw.estimated_count, sw.estimated_cost, sw.time_to_estimate, sw.partitions, sw.total_count, sw.time_to_count\n ON CONFLICT ((md5(_where)))\n DO UPDATE\n SET\n lastused = sw.lastused,\n usecount = sw.usecount,\n statslastupdated = sw.statslastupdated,\n estimated_count = sw.estimated_count,\n estimated_cost = sw.estimated_cost,\n time_to_estimate = sw.time_to_estimate,\n total_count = sw.total_count,\n time_to_count = sw.time_to_count\n ;\n END IF;\n RETURN sw;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION search_query(\n _search jsonb = '{}'::jsonb,\n updatestats boolean = false,\n _metadata jsonb = '{}'::jsonb\n) RETURNS searches AS $$\nDECLARE\n search searches%ROWTYPE;\n pexplain jsonb;\n t timestamptz;\n i interval;\n _hash text := search_hash(_search, _metadata);\n doupdate boolean := FALSE;\n insertfound boolean := FALSE;\n ro boolean := pgstac.readonly();\nBEGIN\n IF ro THEN\n updatestats := FALSE;\n END IF;\n\n SELECT * INTO search FROM searches\n WHERE hash=_hash;\n\n search.hash := _hash;\n\n -- Calculate the where clause if not already calculated\n IF search._where IS NULL THEN\n search._where := stac_search_to_where(_search);\n ELSE\n doupdate := TRUE;\n END IF;\n\n -- Calculate the order by clause if not already calculated\n IF search.orderby IS NULL THEN\n search.orderby := sort_sqlorderby(_search);\n ELSE\n doupdate := TRUE;\n END IF;\n\n PERFORM where_stats(search._where, updatestats, _search->'conf');\n\n IF NOT ro THEN\n IF NOT doupdate THEN\n INSERT INTO searches (search, _where, orderby, lastused, usecount, metadata)\n VALUES (_search, search._where, search.orderby, clock_timestamp(), 1, _metadata)\n ON CONFLICT (hash) DO NOTHING RETURNING * INTO search;\n IF FOUND THEN\n RETURN search;\n END IF;\n END IF;\n\n UPDATE searches\n SET\n lastused=clock_timestamp(),\n usecount=usecount+1\n WHERE hash=(\n SELECT hash FROM searches\n WHERE hash=_hash\n FOR UPDATE SKIP LOCKED\n );\n IF NOT FOUND THEN\n RAISE NOTICE 'Did not update stats for % due to lock. (This is generally OK)', _search;\n END IF;\n END IF;\n\n RETURN search;\n\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE OR REPLACE FUNCTION search_rows(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN partitions text[] DEFAULT NULL,\n IN _limit int DEFAULT 10\n) RETURNS SETOF items AS $$\nDECLARE\n base_query text;\n query text;\n sdate timestamptz;\n edate timestamptz;\n n int;\n records_left int := _limit;\n timer timestamptz := clock_timestamp();\n full_timer timestamptz := clock_timestamp();\nBEGIN\nIF _where IS NULL OR trim(_where) = '' THEN\n _where = ' TRUE ';\nEND IF;\nRAISE NOTICE 'Getting chunks for % %', _where, _orderby;\n\nbase_query := $q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n LIMIT %L\n$q$;\n\nIF _orderby ILIKE 'datetime d%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 DESC LOOP\n RAISE NOTICE 'Running Query for % to %. %', sdate, edate, age_ms(full_timer);\n query := format(\n base_query,\n sdate,\n edate,\n _where,\n _orderby,\n records_left\n );\n RAISE LOG 'QUERY: %', query;\n timer := clock_timestamp();\n RETURN QUERY EXECUTE query;\n\n GET DIAGNOSTICS n = ROW_COUNT;\n records_left := records_left - n;\n RAISE NOTICE 'Returned %/% Rows From % to %. % to go. Time: %ms', n, _limit, sdate, edate, records_left, age_ms(timer);\n timer := clock_timestamp();\n IF records_left <= 0 THEN\n RAISE NOTICE 'SEARCH_ROWS TOOK %ms', age_ms(full_timer);\n RETURN;\n END IF;\n END LOOP;\nELSIF _orderby ILIKE 'datetime a%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 ASC LOOP\n RAISE NOTICE 'Running Query for % to %. %', sdate, edate, age_ms(full_timer);\n query := format(\n base_query,\n sdate,\n edate,\n _where,\n _orderby,\n records_left\n );\n RAISE LOG 'QUERY: %', query;\n timer := clock_timestamp();\n RETURN QUERY EXECUTE query;\n\n GET DIAGNOSTICS n = ROW_COUNT;\n records_left := records_left - n;\n RAISE NOTICE 'Returned %/% Rows From % to %. % to go. Time: %ms', n, _limit, sdate, edate, records_left, age_ms(timer);\n timer := clock_timestamp();\n IF records_left <= 0 THEN\n RAISE NOTICE 'SEARCH_ROWS TOOK %ms', age_ms(full_timer);\n RETURN;\n END IF;\n END LOOP;\nELSE\n query := format($q$\n SELECT * FROM items\n WHERE %s\n ORDER BY %s\n LIMIT %L\n $q$, _where, _orderby, _limit\n );\n RAISE LOG 'QUERY: %', query;\n timer := clock_timestamp();\n RETURN QUERY EXECUTE query;\n RAISE NOTICE 'FULL QUERY TOOK %ms', age_ms(timer);\nEND IF;\nRAISE NOTICE 'SEARCH_ROWS TOOK %ms', age_ms(full_timer);\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE UNLOGGED TABLE format_item_cache(\n id text,\n collection text,\n fields text,\n hydrated bool,\n output jsonb,\n lastused timestamptz DEFAULT now(),\n usecount int DEFAULT 1,\n timetoformat float,\n PRIMARY KEY (collection, id, fields, hydrated)\n);\nCREATE INDEX ON format_item_cache (lastused);\n\nCREATE OR REPLACE FUNCTION format_item(_item items, _fields jsonb DEFAULT '{}', _hydrated bool DEFAULT TRUE) RETURNS jsonb AS $$\nDECLARE\n cache bool := get_setting_bool('format_cache');\n _output jsonb := null;\n t timestamptz := clock_timestamp();\nBEGIN\n IF cache THEN\n SELECT output INTO _output FROM format_item_cache\n WHERE id=_item.id AND collection=_item.collection AND fields=_fields::text AND hydrated=_hydrated;\n END IF;\n IF _output IS NULL THEN\n IF _hydrated THEN\n _output := content_hydrate(_item, _fields);\n ELSE\n _output := content_nonhydrated(_item, _fields);\n END IF;\n END IF;\n IF cache THEN\n INSERT INTO format_item_cache (id, collection, fields, hydrated, output, timetoformat)\n VALUES (_item.id, _item.collection, _fields::text, _hydrated, _output, age_ms(t))\n ON CONFLICT(collection, id, fields, hydrated) DO\n UPDATE\n SET lastused=now(), usecount = format_item_cache.usecount + 1\n ;\n END IF;\n RETURN _output;\n\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION search(_search jsonb = '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n searches searches%ROWTYPE;\n _where text;\n orderby text;\n search_where search_wheres%ROWTYPE;\n total_count bigint;\n token record;\n token_prev boolean;\n token_item items%ROWTYPE;\n token_where text;\n full_where text;\n init_ts timestamptz := clock_timestamp();\n timer timestamptz := clock_timestamp();\n hydrate bool := NOT (_search->'conf'->>'nohydrate' IS NOT NULL AND (_search->'conf'->>'nohydrate')::boolean = true);\n prev text;\n next text;\n context jsonb;\n collection jsonb;\n out_records jsonb;\n out_len int;\n _limit int := coalesce((_search->>'limit')::int, 10);\n _querylimit int;\n _fields jsonb := coalesce(_search->'fields', '{}'::jsonb);\n has_prev boolean := FALSE;\n has_next boolean := FALSE;\nBEGIN\n searches := search_query(_search);\n _where := searches._where;\n orderby := searches.orderby;\n search_where := where_stats(_where);\n total_count := coalesce(search_where.total_count, search_where.estimated_count);\n RAISE NOTICE 'SEARCH:TOKEN: %', _search->>'token';\n token := get_token_record(_search->>'token');\n RAISE NOTICE '***TOKEN: %', token;\n _querylimit := _limit + 1;\n IF token IS NOT NULL THEN\n token_prev := token.prev;\n token_item := token.item;\n token_where := get_token_filter(_search->'sortby', token_item, token_prev, FALSE);\n RAISE LOG 'TOKEN_WHERE: % (%ms from search start)', token_where, age_ms(timer);\n IF token_prev THEN -- if we are using a prev token, we know has_next is true\n RAISE LOG 'There is a previous token, so automatically setting has_next to true';\n has_next := TRUE;\n orderby := sort_sqlorderby(_search, TRUE);\n ELSE\n RAISE LOG 'There is a next token, so automatically setting has_prev to true';\n has_prev := TRUE;\n\n END IF;\n ELSE -- if there was no token, we know there is no prev\n RAISE LOG 'There is no token, so we know there is no prev. setting has_prev to false';\n has_prev := FALSE;\n END IF;\n\n full_where := concat_ws(' AND ', _where, token_where);\n RAISE NOTICE 'FULL WHERE CLAUSE: %', full_where;\n RAISE NOTICE 'Time to get counts and build query %', age_ms(timer);\n timer := clock_timestamp();\n\n IF hydrate THEN\n RAISE NOTICE 'Getting hydrated data.';\n ELSE\n RAISE NOTICE 'Getting non-hydrated data.';\n END IF;\n RAISE NOTICE 'CACHE SET TO %', get_setting_bool('format_cache');\n RAISE NOTICE 'Time to set hydration/formatting %', age_ms(timer);\n timer := clock_timestamp();\n SELECT jsonb_agg(format_item(i, _fields, hydrate)) INTO out_records\n FROM search_rows(\n full_where,\n orderby,\n search_where.partitions,\n _querylimit\n ) as i;\n\n RAISE NOTICE 'Time to fetch rows %', age_ms(timer);\n timer := clock_timestamp();\n\n\n IF token_prev THEN\n out_records := flip_jsonb_array(out_records);\n END IF;\n\n RAISE NOTICE 'Query returned % records.', jsonb_array_length(out_records);\n RAISE LOG 'TOKEN: % %', token_item.id, token_item.collection;\n RAISE LOG 'RECORD_1: % %', out_records->0->>'id', out_records->0->>'collection';\n RAISE LOG 'RECORD-1: % %', out_records->-1->>'id', out_records->-1->>'collection';\n\n -- REMOVE records that were from our token\n IF out_records->0->>'id' = token_item.id AND out_records->0->>'collection' = token_item.collection THEN\n out_records := out_records - 0;\n ELSIF out_records->-1->>'id' = token_item.id AND out_records->-1->>'collection' = token_item.collection THEN\n out_records := out_records - -1;\n END IF;\n\n out_len := jsonb_array_length(out_records);\n\n IF out_len = _limit + 1 THEN\n IF token_prev THEN\n has_prev := TRUE;\n out_records := out_records - 0;\n ELSE\n has_next := TRUE;\n out_records := out_records - -1;\n END IF;\n END IF;\n\n IF has_next THEN\n next := concat(out_records->-1->>'collection', ':', out_records->-1->>'id');\n RAISE NOTICE 'HAS NEXT | %', next;\n END IF;\n\n IF has_prev THEN\n prev := concat(out_records->0->>'collection', ':', out_records->0->>'id');\n RAISE NOTICE 'HAS PREV | %', prev;\n END IF;\n\n RAISE NOTICE 'Time to get prev/next %', age_ms(timer);\n timer := clock_timestamp();\n\n IF context(_search->'conf') != 'off' THEN\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'matched', total_count,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\n ELSE\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\n END IF;\n\n collection := jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb),\n 'next', next,\n 'prev', prev,\n 'context', context\n );\n\n IF get_setting_bool('timing', _search->'conf') THEN\n collection = collection || jsonb_build_object('timing', age_ms(init_ts));\n END IF;\n\n RAISE NOTICE 'Time to build final json %', age_ms(timer);\n timer := clock_timestamp();\n\n RAISE NOTICE 'Total Time: %', age_ms(current_timestamp);\n RAISE NOTICE 'RETURNING % records. NEXT: %. PREV: %', collection->'context'->>'returned', collection->>'next', collection->>'prev';\n RETURN collection;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION search_cursor(_search jsonb = '{}'::jsonb) RETURNS refcursor AS $$\nDECLARE\n curs refcursor;\n searches searches%ROWTYPE;\n _where text;\n _orderby text;\n q text;\n\nBEGIN\n searches := search_query(_search);\n _where := searches._where;\n _orderby := searches.orderby;\n\n OPEN curs FOR\n WITH p AS (\n SELECT * FROM partition_queries(_where, _orderby) p\n )\n SELECT\n CASE WHEN EXISTS (SELECT 1 FROM p) THEN\n (SELECT format($q$\n SELECT * FROM (\n %s\n ) total\n $q$,\n string_agg(\n format($q$ SELECT * FROM ( %s ) AS sub $q$, p),\n '\n UNION ALL\n '\n )\n ))\n ELSE NULL\n END FROM p;\n RETURN curs;\nEND;\n$$ LANGUAGE PLPGSQL;\nCREATE OR REPLACE VIEW collections_asitems AS\nSELECT\n id,\n geometry,\n 'collections' AS collection,\n datetime,\n end_datetime,\n jsonb_build_object(\n 'properties', content - '{links,assets,stac_version,stac_extensions}',\n 'links', content->'links',\n 'assets', content->'assets',\n 'stac_version', content->'stac_version',\n 'stac_extensions', content->'stac_extensions'\n ) AS content,\n content as collectionjson\nFROM collections;\n\n\nCREATE OR REPLACE FUNCTION collection_search_matched(\n IN _search jsonb DEFAULT '{}'::jsonb,\n OUT matched bigint\n) RETURNS bigint AS $$\nDECLARE\n _where text := stac_search_to_where(_search);\nBEGIN\n EXECUTE format(\n $query$\n SELECT\n count(*)\n FROM\n collections_asitems\n WHERE %s\n ;\n $query$,\n _where\n ) INTO matched;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION collection_search_rows(\n _search jsonb DEFAULT '{}'::jsonb\n) RETURNS SETOF jsonb AS $$\nDECLARE\n _where text := stac_search_to_where(_search);\n _limit int := coalesce((_search->>'limit')::int, 10);\n _fields jsonb := coalesce(_search->'fields', '{}'::jsonb);\n _orderby text;\n _offset int := COALESCE((_search->>'offset')::int, 0);\nBEGIN\n _orderby := sort_sqlorderby(\n jsonb_build_object(\n 'sortby',\n coalesce(\n _search->'sortby',\n '[{\"field\": \"id\", \"direction\": \"asc\"}]'::jsonb\n )\n )\n );\n RETURN QUERY EXECUTE format(\n $query$\n SELECT\n jsonb_fields(collectionjson, %L) as c\n FROM\n collections_asitems\n WHERE %s\n ORDER BY %s\n LIMIT %L\n OFFSET %L\n ;\n $query$,\n _fields,\n _where,\n _orderby,\n _limit,\n _offset\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION collection_search(\n _search jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\nDECLARE\n out_records jsonb;\n number_matched bigint := collection_search_matched(_search);\n number_returned bigint;\n _limit int := coalesce((_search->>'limit')::float::int, 10);\n _offset int := coalesce((_search->>'offset')::float::int, 0);\n links jsonb := '[]';\n ret jsonb;\n base_url text:= concat(rtrim(base_url(_search->'conf'),'/'), '/collections');\n prevoffset int;\n nextoffset int;\nBEGIN\n SELECT\n coalesce(jsonb_agg(c), '[]')\n INTO out_records\n FROM collection_search_rows(_search) c;\n\n number_returned := jsonb_array_length(out_records);\n\n IF _limit <= number_matched THEN --need to have paging links\n nextoffset := least(_offset + _limit, number_matched - 1);\n prevoffset := greatest(_offset - _limit, 0);\n IF _offset = 0 THEN -- no previous paging\n\n links := jsonb_build_array(\n jsonb_build_object(\n 'rel', 'next',\n 'type', 'application/json',\n 'method', 'GET' ,\n 'href', base_url,\n 'body', jsonb_build_object('offset', nextoffset),\n 'merge', TRUE\n )\n );\n ELSE\n links := jsonb_build_array(\n jsonb_build_object(\n 'rel', 'prev',\n 'type', 'application/json',\n 'method', 'GET' ,\n 'href', base_url,\n 'body', jsonb_build_object('offset', prevoffset),\n 'merge', TRUE\n ),\n jsonb_build_object(\n 'rel', 'next',\n 'type', 'application/json',\n 'method', 'GET' ,\n 'href', base_url,\n 'body', jsonb_build_object('offset', nextoffset),\n 'merge', TRUE\n )\n );\n END IF;\n END IF;\n\n ret := jsonb_build_object(\n 'collections', out_records,\n 'context', jsonb_build_object(\n 'limit', _limit,\n 'matched', number_matched,\n 'returned', number_returned\n ),\n 'links', links\n );\n RETURN ret;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION tileenvelope(zoom int, x int, y int) RETURNS geometry AS $$\nWITH t AS (\n SELECT\n 20037508.3427892 as merc_max,\n -20037508.3427892 as merc_min,\n (2 * 20037508.3427892) / (2 ^ zoom) as tile_size\n)\nSELECT st_makeenvelope(\n merc_min + (tile_size * x),\n merc_max - (tile_size * (y + 1)),\n merc_min + (tile_size * (x + 1)),\n merc_max - (tile_size * y),\n 3857\n) FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;DROP FUNCTION IF EXISTS mercgrid;\n\n\nCREATE OR REPLACE FUNCTION ftime() RETURNS interval as $$\nSELECT age(clock_timestamp(), transaction_timestamp());\n$$ LANGUAGE SQL;\nSET SEARCH_PATH to pgstac, public;\n\nDROP FUNCTION IF EXISTS geometrysearch;\nCREATE OR REPLACE FUNCTION geometrysearch(\n IN geom geometry,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE, -- Return as soon as the passed in geometry is full covered\n IN skipcovered boolean DEFAULT TRUE -- Skip any items that would show up completely under the previous items\n) RETURNS jsonb AS $$\nDECLARE\n search searches%ROWTYPE;\n curs refcursor;\n _where text;\n query text;\n iter_record items%ROWTYPE;\n out_records jsonb := '{}'::jsonb[];\n exit_flag boolean := FALSE;\n counter int := 1;\n scancounter int := 1;\n remaining_limit int := _scanlimit;\n tilearea float;\n unionedgeom geometry;\n clippedgeom geometry;\n unionedgeom_area float := 0;\n prev_area float := 0;\n excludes text[];\n includes text[];\n\nBEGIN\n DROP TABLE IF EXISTS pgstac_results;\n CREATE TEMP TABLE pgstac_results (content jsonb) ON COMMIT DROP;\n\n -- If the passed in geometry is not an area set exitwhenfull and skipcovered to false\n IF ST_GeometryType(geom) !~* 'polygon' THEN\n RAISE NOTICE 'GEOMETRY IS NOT AN AREA';\n skipcovered = FALSE;\n exitwhenfull = FALSE;\n END IF;\n\n -- If skipcovered is true then you will always want to exit when the passed in geometry is full\n IF skipcovered THEN\n exitwhenfull := TRUE;\n END IF;\n\n SELECT * INTO search FROM searches WHERE hash=queryhash;\n\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Search with Query Hash % Not Found', queryhash;\n END IF;\n\n tilearea := st_area(geom);\n _where := format('%s AND st_intersects(geometry, %L::geometry)', search._where, geom);\n\n\n FOR query IN SELECT * FROM partition_queries(_where, search.orderby) LOOP\n query := format('%s LIMIT %L', query, remaining_limit);\n RAISE NOTICE '%', query;\n OPEN curs FOR EXECUTE query;\n LOOP\n FETCH curs INTO iter_record;\n EXIT WHEN NOT FOUND;\n IF exitwhenfull OR skipcovered THEN -- If we are not using exitwhenfull or skipcovered, we do not need to do expensive geometry operations\n clippedgeom := st_intersection(geom, iter_record.geometry);\n\n IF unionedgeom IS NULL THEN\n unionedgeom := clippedgeom;\n ELSE\n unionedgeom := st_union(unionedgeom, clippedgeom);\n END IF;\n\n unionedgeom_area := st_area(unionedgeom);\n\n IF skipcovered AND prev_area = unionedgeom_area THEN\n scancounter := scancounter + 1;\n CONTINUE;\n END IF;\n\n prev_area := unionedgeom_area;\n\n RAISE NOTICE '% % % %', unionedgeom_area/tilearea, counter, scancounter, ftime();\n END IF;\n RAISE NOTICE '% %', iter_record, content_hydrate(iter_record, fields);\n INSERT INTO pgstac_results (content) VALUES (content_hydrate(iter_record, fields));\n\n IF counter >= _limit\n OR scancounter > _scanlimit\n OR ftime() > _timelimit\n OR (exitwhenfull AND unionedgeom_area >= tilearea)\n THEN\n exit_flag := TRUE;\n EXIT;\n END IF;\n counter := counter + 1;\n scancounter := scancounter + 1;\n\n END LOOP;\n CLOSE curs;\n EXIT WHEN exit_flag;\n remaining_limit := _scanlimit - scancounter;\n END LOOP;\n\n SELECT jsonb_agg(content) INTO out_records FROM pgstac_results WHERE content IS NOT NULL;\n\n RETURN jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb)\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDROP FUNCTION IF EXISTS geojsonsearch;\nCREATE OR REPLACE FUNCTION geojsonsearch(\n IN geojson jsonb,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_geomfromgeojson(geojson),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS xyzsearch;\nCREATE OR REPLACE FUNCTION xyzsearch(\n IN _x int,\n IN _y int,\n IN _z int,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_transform(tileenvelope(_z, _x, _y), 4326),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE PROCEDURE analyze_items() AS $$\nDECLARE\n q text;\n timeout_ts timestamptz;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n SELECT format('ANALYZE (VERBOSE, SKIP_LOCKED) %I;', relname) INTO q\n FROM pg_stat_user_tables\n WHERE relname like '_item%' AND (n_mod_since_analyze>0 OR last_analyze IS NULL) LIMIT 1;\n IF NOT FOUND THEN\n EXIT;\n END IF;\n RAISE NOTICE '%', q;\n EXECUTE q;\n COMMIT;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE PROCEDURE validate_constraints() AS $$\nDECLARE\n q text;\nBEGIN\n FOR q IN\n SELECT\n FORMAT(\n 'ALTER TABLE %I.%I VALIDATE CONSTRAINT %I;',\n nsp.nspname,\n cls.relname,\n con.conname\n )\n\n FROM pg_constraint AS con\n JOIN pg_class AS cls\n ON con.conrelid = cls.oid\n JOIN pg_namespace AS nsp\n ON cls.relnamespace = nsp.oid\n WHERE convalidated = FALSE AND contype in ('c','f')\n AND nsp.nspname = 'pgstac'\n LOOP\n RAISE NOTICE '%', q;\n PERFORM run_or_queue(q);\n COMMIT;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION collection_extent(_collection text, runupdate boolean default false) RETURNS jsonb AS $$\nDECLARE\n geom_extent geometry;\n mind timestamptz;\n maxd timestamptz;\n extent jsonb;\nBEGIN\n IF runupdate THEN\n PERFORM update_partition_stats_q(partition)\n FROM partitions_view WHERE collection=_collection;\n END IF;\n SELECT\n min(lower(dtrange)),\n max(upper(edtrange)),\n st_extent(spatial)\n INTO\n mind,\n maxd,\n geom_extent\n FROM partitions_view\n WHERE collection=_collection;\n\n IF geom_extent IS NOT NULL AND mind IS NOT NULL AND maxd IS NOT NULL THEN\n extent := jsonb_build_object(\n 'extent', jsonb_build_object(\n 'spatial', jsonb_build_object(\n 'bbox', to_jsonb(array[array[st_xmin(geom_extent), st_ymin(geom_extent), st_xmax(geom_extent), st_ymax(geom_extent)]])\n ),\n 'temporal', jsonb_build_object(\n 'interval', to_jsonb(array[array[mind, maxd]])\n )\n )\n );\n RETURN extent;\n END IF;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('id', '{\"title\": \"Item ID\",\"description\": \"Item identifier\",\"$ref\": \"https://schemas.stacspec.org/v1.0.0/item-spec/json-schema/item.json#/definitions/core/allOf/2/properties/id\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('geometry', '{\"title\": \"Item Geometry\",\"description\": \"Item Geometry\",\"$ref\": \"https://geojson.org/schema/Feature.json\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('datetime','{\"description\": \"Datetime\",\"type\": \"string\",\"title\": \"Acquired\",\"format\": \"date-time\",\"pattern\": \"(\\\\+00:00|Z)$\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDELETE FROM queryables a USING queryables b\n WHERE a.name = b.name AND a.collection_ids IS NOT DISTINCT FROM b.collection_ids AND a.id > b.id;\n\n\nINSERT INTO pgstac_settings (name, value) VALUES\n ('context', 'off'),\n ('context_estimated_count', '100000'),\n ('context_estimated_cost', '100000'),\n ('context_stats_ttl', '1 day'),\n ('default_filter_lang', 'cql2-json'),\n ('additional_properties', 'true'),\n ('use_queue', 'false'),\n ('queue_timeout', '10 minutes'),\n ('update_collection_extent', 'false'),\n ('format_cache', 'false'),\n ('readonly', 'false')\nON CONFLICT DO NOTHING\n;\n\nALTER FUNCTION to_text COST 5000;\nALTER FUNCTION to_float COST 5000;\nALTER FUNCTION to_int COST 5000;\nALTER FUNCTION to_tstz COST 5000;\nALTER FUNCTION to_text_array COST 5000;\n\nALTER FUNCTION update_partition_stats SECURITY DEFINER;\nALTER FUNCTION partition_after_triggerfunc SECURITY DEFINER;\nALTER FUNCTION drop_table_constraints SECURITY DEFINER;\nALTER FUNCTION create_table_constraints SECURITY DEFINER;\nALTER FUNCTION check_partition SECURITY DEFINER;\nALTER FUNCTION repartition SECURITY DEFINER;\nALTER FUNCTION where_stats SECURITY DEFINER;\nALTER FUNCTION search_query SECURITY DEFINER;\nALTER FUNCTION format_item SECURITY DEFINER;\nALTER FUNCTION maintain_index SECURITY DEFINER;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nGRANT ALL ON SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac to pgstac_admin;\n\n-- pgstac_read role limited to using function apis\nGRANT EXECUTE ON FUNCTION search TO pgstac_read;\nGRANT EXECUTE ON FUNCTION search_query TO pgstac_read;\nGRANT EXECUTE ON FUNCTION item_by_id TO pgstac_read;\nGRANT EXECUTE ON FUNCTION get_item TO pgstac_read;\nGRANT SELECT ON ALL TABLES IN SCHEMA pgstac TO pgstac_read;\n\n\nGRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_ingest;\nGRANT USAGE ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_ingest;\n\nREVOKE ALL PRIVILEGES ON PROCEDURE run_queued_queries FROM public;\nGRANT ALL ON PROCEDURE run_queued_queries TO pgstac_admin;\n\nREVOKE ALL PRIVILEGES ON FUNCTION run_queued_queries_intransaction FROM public;\nGRANT ALL ON FUNCTION run_queued_queries_intransaction TO pgstac_admin;\n\nRESET ROLE;\n\nSET ROLE pgstac_ingest;\nSELECT update_partition_stats_q(partition) FROM partitions_view;\nSELECT set_version('0.8.3');\n" }, { "path": "src/pgstac/migrations/pgstac.0.8.4-0.8.5.sql", "content": "SET client_min_messages TO WARNING;\nSET SEARCH_PATH to pgstac, public;\nRESET ROLE;\nDO $$\nDECLARE\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='postgis') THEN\n CREATE EXTENSION IF NOT EXISTS postgis;\n END IF;\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='btree_gist') THEN\n CREATE EXTENSION IF NOT EXISTS btree_gist;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_admin;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_read;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_ingest;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n\nGRANT pgstac_admin TO current_user;\n\n-- Function to make sure pgstac_admin is the owner of items\nCREATE OR REPLACE FUNCTION pgstac_admin_owns() RETURNS VOID AS $$\nDECLARE\n f RECORD;\nBEGIN\n FOR f IN (\n SELECT\n concat(\n oid::regproc::text,\n '(',\n coalesce(pg_get_function_identity_arguments(oid),''),\n ')'\n ) AS name,\n CASE prokind WHEN 'f' THEN 'FUNCTION' WHEN 'p' THEN 'PROCEDURE' WHEN 'a' THEN 'AGGREGATE' END as typ\n FROM pg_proc\n WHERE\n pronamespace=to_regnamespace('pgstac')\n AND proowner != to_regrole('pgstac_admin')\n AND proname NOT LIKE 'pg_stat%'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n FOR f IN (\n SELECT\n oid::regclass::text as name,\n CASE relkind\n WHEN 'i' THEN 'INDEX'\n WHEN 'I' THEN 'INDEX'\n WHEN 'p' THEN 'TABLE'\n WHEN 'r' THEN 'TABLE'\n WHEN 'v' THEN 'VIEW'\n WHEN 'S' THEN 'SEQUENCE'\n ELSE NULL\n END as typ\n FROM pg_class\n WHERE relnamespace=to_regnamespace('pgstac') and relowner != to_regrole('pgstac_admin') AND relkind IN ('r','p','v','S') AND relname NOT LIKE 'pg_stat'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\nSELECT pgstac_admin_owns();\n\nCREATE SCHEMA IF NOT EXISTS pgstac AUTHORIZATION pgstac_admin;\n\nGRANT ALL ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_admin;\n\nALTER ROLE pgstac_admin SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_read SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_ingest SET SEARCH_PATH TO pgstac, public;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\n\nGRANT pgstac_read TO pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\n\nSET ROLE pgstac_admin;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\nRESET ROLE;\n\nSET ROLE pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\nRESET ROLE;\n\nSET SEARCH_PATH TO pgstac, public;\nSET ROLE pgstac_admin;\n\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS analyze_items;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS validate_constraints;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n-- Install these idempotently as migrations do not put them before trying to modify the collections table\n\n\nCREATE OR REPLACE FUNCTION collection_geom(content jsonb)\nRETURNS geometry AS $$\n WITH box AS (SELECT content->'extent'->'spatial'->'bbox'->0 as box)\n SELECT\n st_makeenvelope(\n (box->>0)::float,\n (box->>1)::float,\n (box->>2)::float,\n (box->>3)::float,\n 4326\n )\n FROM box;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION collection_datetime(content jsonb)\nRETURNS timestamptz AS $$\n SELECT\n CASE\n WHEN\n (content->'extent'->'temporal'->'interval'->0->>0) IS NULL\n THEN '-infinity'::timestamptz\n ELSE\n (content->'extent'->'temporal'->'interval'->0->>0)::timestamptz\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION collection_enddatetime(content jsonb)\nRETURNS timestamptz AS $$\n SELECT\n CASE\n WHEN\n (content->'extent'->'temporal'->'interval'->0->>1) IS NULL\n THEN 'infinity'::timestamptz\n ELSE\n (content->'extent'->'temporal'->'interval'->0->>1)::timestamptz\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n-- BEGIN migra calculated SQL\nset check_function_bodies = off;\n\nCREATE OR REPLACE FUNCTION pgstac.cql2_query(j jsonb, wrapper text DEFAULT NULL::text)\n RETURNS text\n LANGUAGE plpgsql\n STABLE\nAS $function$\n#variable_conflict use_variable\nDECLARE\n args jsonb := j->'args';\n arg jsonb;\n op text := lower(j->>'op');\n cql2op RECORD;\n literal text;\n _wrapper text;\n leftarg text;\n rightarg text;\n prop text;\n extra_props bool := pgstac.additional_properties();\nBEGIN\n IF j IS NULL OR (op IS NOT NULL AND args IS NULL) THEN\n RETURN NULL;\n END IF;\n RAISE NOTICE 'CQL2_QUERY: %', j;\n\n -- check if all properties are represented in the queryables\n IF NOT extra_props THEN\n FOR prop IN\n SELECT DISTINCT p->>0\n FROM jsonb_path_query(j, 'strict $.**.property') p\n WHERE p->>0 NOT IN ('id', 'datetime', 'geometry', 'end_datetime', 'collection')\n LOOP\n IF (queryable(prop)).nulled_wrapper IS NULL THEN\n RAISE EXCEPTION 'Term % is not found in queryables.', prop;\n END IF;\n END LOOP;\n END IF;\n\n IF j ? 'filter' THEN\n RETURN cql2_query(j->'filter');\n END IF;\n\n IF j ? 'upper' THEN\n RETURN cql2_query(jsonb_build_object('op', 'upper', 'args', j->'upper'));\n END IF;\n\n IF j ? 'lower' THEN\n RETURN cql2_query(jsonb_build_object('op', 'lower', 'args', j->'lower'));\n END IF;\n\n -- Temporal Query\n IF op ilike 't_%' or op = 'anyinteracts' THEN\n RETURN temporal_op_query(op, args);\n END IF;\n\n -- If property is a timestamp convert it to text to use with\n -- general operators\n IF j ? 'timestamp' THEN\n RETURN format('%L::timestamptz', to_tstz(j->'timestamp'));\n END IF;\n IF j ? 'interval' THEN\n RAISE EXCEPTION 'Please use temporal operators when using intervals.';\n RETURN NONE;\n END IF;\n\n -- Spatial Query\n IF op ilike 's_%' or op = 'intersects' THEN\n RETURN spatial_op_query(op, args);\n END IF;\n\n IF op IN ('a_equals','a_contains','a_contained_by','a_overlaps') THEN\n IF args->0 ? 'property' THEN\n leftarg := format('to_text_array(%s)', (queryable(args->0->>'property')).path);\n END IF;\n IF args->1 ? 'property' THEN\n rightarg := format('to_text_array(%s)', (queryable(args->1->>'property')).path);\n END IF;\n RETURN FORMAT(\n '%s %s %s',\n COALESCE(leftarg, quote_literal(to_text_array(args->0))),\n CASE op\n WHEN 'a_equals' THEN '='\n WHEN 'a_contains' THEN '@>'\n WHEN 'a_contained_by' THEN '<@'\n WHEN 'a_overlaps' THEN '&&'\n END,\n COALESCE(rightarg, quote_literal(to_text_array(args->1)))\n );\n END IF;\n\n IF op = 'in' THEN\n RAISE NOTICE 'IN : % % %', args, jsonb_build_array(args->0), args->1;\n args := jsonb_build_array(args->0) || (args->1);\n RAISE NOTICE 'IN2 : %', args;\n END IF;\n\n\n\n IF op = 'between' THEN\n args = jsonb_build_array(\n args->0,\n args->1->0,\n args->1->1\n );\n END IF;\n\n -- Make sure that args is an array and run cql2_query on\n -- each element of the array\n RAISE NOTICE 'ARGS PRE: %', args;\n IF j ? 'args' THEN\n IF jsonb_typeof(args) != 'array' THEN\n args := jsonb_build_array(args);\n END IF;\n\n IF jsonb_path_exists(args, '$[*] ? (@.property == \"id\" || @.property == \"datetime\" || @.property == \"end_datetime\" || @.property == \"collection\")') THEN\n wrapper := NULL;\n ELSE\n -- if any of the arguments are a property, try to get the property_wrapper\n FOR arg IN SELECT jsonb_path_query(args, '$[*] ? (@.property != null)') LOOP\n RAISE NOTICE 'Arg: %', arg;\n wrapper := (queryable(arg->>'property')).nulled_wrapper;\n RAISE NOTICE 'Property: %, Wrapper: %', arg, wrapper;\n IF wrapper IS NOT NULL THEN\n EXIT;\n END IF;\n END LOOP;\n\n -- if the property was not in queryables, see if any args were numbers\n IF\n wrapper IS NULL\n AND jsonb_path_exists(args, '$[*] ? (@.type()==\"number\")')\n THEN\n wrapper := 'to_float';\n END IF;\n wrapper := coalesce(wrapper, 'to_text');\n END IF;\n\n SELECT jsonb_agg(cql2_query(a, wrapper))\n INTO args\n FROM jsonb_array_elements(args) a;\n END IF;\n RAISE NOTICE 'ARGS: %', args;\n\n IF op IN ('and', 'or') THEN\n RETURN\n format(\n '(%s)',\n array_to_string(to_text_array(args), format(' %s ', upper(op)))\n );\n END IF;\n\n IF op = 'in' THEN\n RAISE NOTICE 'IN -- % %', args->0, to_text(args->0);\n RETURN format(\n '%s IN (%s)',\n to_text(args->0),\n array_to_string((to_text_array(args))[2:], ',')\n );\n END IF;\n\n -- Look up template from cql2_ops\n IF j ? 'op' THEN\n SELECT * INTO cql2op FROM cql2_ops WHERE cql2_ops.op ilike op;\n IF FOUND THEN\n -- If specific index set in queryables for a property cast other arguments to that type\n\n RETURN format(\n cql2op.template,\n VARIADIC (to_text_array(args))\n );\n ELSE\n RAISE EXCEPTION 'Operator % Not Supported.', op;\n END IF;\n END IF;\n\n\n IF wrapper IS NOT NULL THEN\n RAISE NOTICE 'Wrapping % with %', j, wrapper;\n IF j ? 'property' THEN\n RETURN format('%I(%s)', wrapper, (queryable(j->>'property')).path);\n ELSE\n RETURN format('%I(%L)', wrapper, j);\n END IF;\n ELSIF j ? 'property' THEN\n RETURN quote_ident(j->>'property');\n END IF;\n\n RETURN quote_literal(to_text(j));\nEND;\n$function$\n;\n\n\n-- END migra calculated SQL\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('id', '{\"title\": \"Item ID\",\"description\": \"Item identifier\",\"$ref\": \"https://schemas.stacspec.org/v1.0.0/item-spec/json-schema/item.json#/definitions/core/allOf/2/properties/id\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('geometry', '{\"title\": \"Item Geometry\",\"description\": \"Item Geometry\",\"$ref\": \"https://geojson.org/schema/Feature.json\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('datetime','{\"description\": \"Datetime\",\"type\": \"string\",\"title\": \"Acquired\",\"format\": \"date-time\",\"pattern\": \"(\\\\+00:00|Z)$\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDELETE FROM queryables a USING queryables b\n WHERE a.name = b.name AND a.collection_ids IS NOT DISTINCT FROM b.collection_ids AND a.id > b.id;\n\n\nINSERT INTO pgstac_settings (name, value) VALUES\n ('context', 'off'),\n ('context_estimated_count', '100000'),\n ('context_estimated_cost', '100000'),\n ('context_stats_ttl', '1 day'),\n ('default_filter_lang', 'cql2-json'),\n ('additional_properties', 'true'),\n ('use_queue', 'false'),\n ('queue_timeout', '10 minutes'),\n ('update_collection_extent', 'false'),\n ('format_cache', 'false'),\n ('readonly', 'false')\nON CONFLICT DO NOTHING\n;\n\nALTER FUNCTION to_text COST 5000;\nALTER FUNCTION to_float COST 5000;\nALTER FUNCTION to_int COST 5000;\nALTER FUNCTION to_tstz COST 5000;\nALTER FUNCTION to_text_array COST 5000;\n\nALTER FUNCTION update_partition_stats SECURITY DEFINER;\nALTER FUNCTION partition_after_triggerfunc SECURITY DEFINER;\nALTER FUNCTION drop_table_constraints SECURITY DEFINER;\nALTER FUNCTION create_table_constraints SECURITY DEFINER;\nALTER FUNCTION check_partition SECURITY DEFINER;\nALTER FUNCTION repartition SECURITY DEFINER;\nALTER FUNCTION where_stats SECURITY DEFINER;\nALTER FUNCTION search_query SECURITY DEFINER;\nALTER FUNCTION format_item SECURITY DEFINER;\nALTER FUNCTION maintain_index SECURITY DEFINER;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nGRANT ALL ON SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac to pgstac_admin;\n\n-- pgstac_read role limited to using function apis\nGRANT EXECUTE ON FUNCTION search TO pgstac_read;\nGRANT EXECUTE ON FUNCTION search_query TO pgstac_read;\nGRANT EXECUTE ON FUNCTION item_by_id TO pgstac_read;\nGRANT EXECUTE ON FUNCTION get_item TO pgstac_read;\nGRANT SELECT ON ALL TABLES IN SCHEMA pgstac TO pgstac_read;\n\n\nGRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_ingest;\nGRANT USAGE ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_ingest;\n\nREVOKE ALL PRIVILEGES ON PROCEDURE run_queued_queries FROM public;\nGRANT ALL ON PROCEDURE run_queued_queries TO pgstac_admin;\n\nREVOKE ALL PRIVILEGES ON FUNCTION run_queued_queries_intransaction FROM public;\nGRANT ALL ON FUNCTION run_queued_queries_intransaction TO pgstac_admin;\n\nRESET ROLE;\n\nSET ROLE pgstac_ingest;\nSELECT update_partition_stats_q(partition) FROM partitions_view;\nSELECT set_version('0.8.5');\n" }, { "path": "src/pgstac/migrations/pgstac.0.8.4.sql", "content": "RESET ROLE;\nDO $$\nDECLARE\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='postgis') THEN\n CREATE EXTENSION IF NOT EXISTS postgis;\n END IF;\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='btree_gist') THEN\n CREATE EXTENSION IF NOT EXISTS btree_gist;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_admin;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_read;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_ingest;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n\nGRANT pgstac_admin TO current_user;\n\n-- Function to make sure pgstac_admin is the owner of items\nCREATE OR REPLACE FUNCTION pgstac_admin_owns() RETURNS VOID AS $$\nDECLARE\n f RECORD;\nBEGIN\n FOR f IN (\n SELECT\n concat(\n oid::regproc::text,\n '(',\n coalesce(pg_get_function_identity_arguments(oid),''),\n ')'\n ) AS name,\n CASE prokind WHEN 'f' THEN 'FUNCTION' WHEN 'p' THEN 'PROCEDURE' WHEN 'a' THEN 'AGGREGATE' END as typ\n FROM pg_proc\n WHERE\n pronamespace=to_regnamespace('pgstac')\n AND proowner != to_regrole('pgstac_admin')\n AND proname NOT LIKE 'pg_stat%'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n FOR f IN (\n SELECT\n oid::regclass::text as name,\n CASE relkind\n WHEN 'i' THEN 'INDEX'\n WHEN 'I' THEN 'INDEX'\n WHEN 'p' THEN 'TABLE'\n WHEN 'r' THEN 'TABLE'\n WHEN 'v' THEN 'VIEW'\n WHEN 'S' THEN 'SEQUENCE'\n ELSE NULL\n END as typ\n FROM pg_class\n WHERE relnamespace=to_regnamespace('pgstac') and relowner != to_regrole('pgstac_admin') AND relkind IN ('r','p','v','S') AND relname NOT LIKE 'pg_stat'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\nSELECT pgstac_admin_owns();\n\nCREATE SCHEMA IF NOT EXISTS pgstac AUTHORIZATION pgstac_admin;\n\nGRANT ALL ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_admin;\n\nALTER ROLE pgstac_admin SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_read SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_ingest SET SEARCH_PATH TO pgstac, public;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\n\nGRANT pgstac_read TO pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\n\nSET ROLE pgstac_admin;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\nRESET ROLE;\n\nSET ROLE pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\nRESET ROLE;\n\nSET SEARCH_PATH TO pgstac, public;\nSET ROLE pgstac_admin;\n\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS analyze_items;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS validate_constraints;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n-- Install these idempotently as migrations do not put them before trying to modify the collections table\n\n\nCREATE OR REPLACE FUNCTION collection_geom(content jsonb)\nRETURNS geometry AS $$\n WITH box AS (SELECT content->'extent'->'spatial'->'bbox'->0 as box)\n SELECT\n st_makeenvelope(\n (box->>0)::float,\n (box->>1)::float,\n (box->>2)::float,\n (box->>3)::float,\n 4326\n )\n FROM box;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION collection_datetime(content jsonb)\nRETURNS timestamptz AS $$\n SELECT\n CASE\n WHEN\n (content->'extent'->'temporal'->'interval'->0->>0) IS NULL\n THEN '-infinity'::timestamptz\n ELSE\n (content->'extent'->'temporal'->'interval'->0->>0)::timestamptz\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION collection_enddatetime(content jsonb)\nRETURNS timestamptz AS $$\n SELECT\n CASE\n WHEN\n (content->'extent'->'temporal'->'interval'->0->>1) IS NULL\n THEN 'infinity'::timestamptz\n ELSE\n (content->'extent'->'temporal'->'interval'->0->>1)::timestamptz\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE TABLE IF NOT EXISTS migrations (\n version text PRIMARY KEY,\n datetime timestamptz DEFAULT clock_timestamp() NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION get_version() RETURNS text AS $$\n SELECT version FROM pgstac.migrations ORDER BY datetime DESC, version DESC LIMIT 1;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION set_version(text) RETURNS text AS $$\n INSERT INTO pgstac.migrations (version) VALUES ($1)\n ON CONFLICT DO NOTHING\n RETURNING version;\n$$ LANGUAGE SQL;\n\n\nCREATE TABLE IF NOT EXISTS pgstac_settings (\n name text PRIMARY KEY,\n value text NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION table_empty(text) RETURNS boolean AS $$\nDECLARE\n retval boolean;\nBEGIN\n EXECUTE format($q$\n SELECT NOT EXISTS (SELECT 1 FROM %I LIMIT 1)\n $q$,\n $1\n ) INTO retval;\n RETURN retval;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION get_setting(IN _setting text, IN conf jsonb DEFAULT NULL) RETURNS text AS $$\nSELECT COALESCE(\n nullif(conf->>_setting, ''),\n nullif(current_setting(concat('pgstac.',_setting), TRUE),''),\n nullif((SELECT value FROM pgstac.pgstac_settings WHERE name=_setting),'')\n);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_setting_bool(IN _setting text, IN conf jsonb DEFAULT NULL) RETURNS boolean AS $$\nSELECT COALESCE(\n nullif(conf->>_setting, ''),\n nullif(current_setting(concat('pgstac.',_setting), TRUE),''),\n nullif((SELECT value FROM pgstac.pgstac_settings WHERE name=_setting),''),\n 'FALSE'\n)::boolean;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION base_url(conf jsonb DEFAULT NULL) RETURNS text AS $$\n SELECT COALESCE(pgstac.get_setting('base_url', conf), '.');\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION additional_properties() RETURNS boolean AS $$\n SELECT pgstac.get_setting_bool('additional_properties');\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION readonly(conf jsonb DEFAULT NULL) RETURNS boolean AS $$\n SELECT pgstac.get_setting_bool('readonly', conf);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context(conf jsonb DEFAULT NULL) RETURNS text AS $$\n SELECT pgstac.get_setting('context', conf);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context_estimated_count(conf jsonb DEFAULT NULL) RETURNS int AS $$\n SELECT pgstac.get_setting('context_estimated_count', conf)::int;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_estimated_cost();\nCREATE OR REPLACE FUNCTION context_estimated_cost(conf jsonb DEFAULT NULL) RETURNS float AS $$\n SELECT pgstac.get_setting('context_estimated_cost', conf)::float;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_stats_ttl();\nCREATE OR REPLACE FUNCTION context_stats_ttl(conf jsonb DEFAULT NULL) RETURNS interval AS $$\n SELECT pgstac.get_setting('context_stats_ttl', conf)::interval;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION t2s(text) RETURNS text AS $$\n SELECT extract(epoch FROM $1::interval)::text || ' s';\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE STRICT;\n\nCREATE OR REPLACE FUNCTION age_ms(a timestamptz, b timestamptz DEFAULT clock_timestamp()) RETURNS float AS $$\n SELECT abs(extract(epoch from age(a,b)) * 1000);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION queue_timeout() RETURNS interval AS $$\n SELECT t2s(coalesce(\n get_setting('queue_timeout'),\n '1h'\n ))::interval;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION notice(VARIADIC text[]) RETURNS boolean AS $$\nDECLARE\ndebug boolean := current_setting('pgstac.debug', true);\nBEGIN\n IF debug THEN\n RAISE NOTICE 'NOTICE FROM FUNC: % >>>>> %', concat_ws(' | ', $1), clock_timestamp();\n RETURN TRUE;\n END IF;\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION empty_arr(ANYARRAY) RETURNS BOOLEAN AS $$\nSELECT CASE\n WHEN $1 IS NULL THEN TRUE\n WHEN cardinality($1)<1 THEN TRUE\nELSE FALSE\nEND;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_intersection(_a ANYARRAY, _b ANYARRAY) RETURNS ANYARRAY AS $$\n SELECT ARRAY ( SELECT unnest(_a) INTERSECT SELECT UNNEST(_b) );\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION array_map_ident(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_ident(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_map_literal(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_literal(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION array_reverse(anyarray) RETURNS anyarray AS $$\nSELECT ARRAY(\n SELECT $1[i]\n FROM generate_subscripts($1,1) AS s(i)\n ORDER BY i DESC\n);\n$$ LANGUAGE SQL STRICT IMMUTABLE;\n\nDROP TABLE IF EXISTS query_queue;\nCREATE TABLE query_queue (\n query text PRIMARY KEY,\n added timestamptz DEFAULT now()\n);\n\nDROP TABLE IF EXISTS query_queue_history;\nCREATE TABLE query_queue_history(\n query text,\n added timestamptz NOT NULL,\n finished timestamptz NOT NULL DEFAULT now(),\n error text\n);\n\nCREATE OR REPLACE PROCEDURE run_queued_queries() AS $$\nDECLARE\n qitem query_queue%ROWTYPE;\n timeout_ts timestamptz;\n error text;\n cnt int := 0;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n DELETE FROM query_queue WHERE query = (SELECT query FROM query_queue ORDER BY added DESC LIMIT 1 FOR UPDATE SKIP LOCKED) RETURNING * INTO qitem;\n IF NOT FOUND THEN\n EXIT;\n END IF;\n cnt := cnt + 1;\n BEGIN\n RAISE NOTICE 'RUNNING QUERY: %', qitem.query;\n EXECUTE qitem.query;\n EXCEPTION WHEN others THEN\n error := format('%s | %s', SQLERRM, SQLSTATE);\n END;\n INSERT INTO query_queue_history (query, added, finished, error)\n VALUES (qitem.query, qitem.added, clock_timestamp(), error);\n COMMIT;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION run_queued_queries_intransaction() RETURNS int AS $$\nDECLARE\n qitem query_queue%ROWTYPE;\n timeout_ts timestamptz;\n error text;\n cnt int := 0;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n DELETE FROM query_queue WHERE query = (SELECT query FROM query_queue ORDER BY added DESC LIMIT 1 FOR UPDATE SKIP LOCKED) RETURNING * INTO qitem;\n IF NOT FOUND THEN\n RETURN cnt;\n END IF;\n cnt := cnt + 1;\n BEGIN\n qitem.query := regexp_replace(qitem.query, 'CONCURRENTLY', '');\n RAISE NOTICE 'RUNNING QUERY: %', qitem.query;\n\n EXECUTE qitem.query;\n EXCEPTION WHEN others THEN\n error := format('%s | %s', SQLERRM, SQLSTATE);\n RAISE WARNING '%', error;\n END;\n INSERT INTO query_queue_history (query, added, finished, error)\n VALUES (qitem.query, qitem.added, clock_timestamp(), error);\n END LOOP;\n RETURN cnt;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION run_or_queue(query text) RETURNS VOID AS $$\nDECLARE\n use_queue text := COALESCE(get_setting('use_queue'), 'FALSE')::boolean;\nBEGIN\n IF get_setting_bool('debug') THEN\n RAISE NOTICE '%', query;\n END IF;\n IF use_queue THEN\n INSERT INTO query_queue (query) VALUES (query) ON CONFLICT DO NOTHING;\n ELSE\n EXECUTE query;\n END IF;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n\nDROP FUNCTION IF EXISTS check_pgstac_settings;\nCREATE OR REPLACE FUNCTION check_pgstac_settings(_sysmem text DEFAULT NULL) RETURNS VOID AS $$\nDECLARE\n settingval text;\n sysmem bigint := pg_size_bytes(_sysmem);\n effective_cache_size bigint := pg_size_bytes(current_setting('effective_cache_size', TRUE));\n shared_buffers bigint := pg_size_bytes(current_setting('shared_buffers', TRUE));\n work_mem bigint := pg_size_bytes(current_setting('work_mem', TRUE));\n max_connections int := current_setting('max_connections', TRUE);\n maintenance_work_mem bigint := pg_size_bytes(current_setting('maintenance_work_mem', TRUE));\n seq_page_cost float := current_setting('seq_page_cost', TRUE);\n random_page_cost float := current_setting('random_page_cost', TRUE);\n temp_buffers bigint := pg_size_bytes(current_setting('temp_buffers', TRUE));\n r record;\nBEGIN\n IF _sysmem IS NULL THEN\n RAISE NOTICE 'Call function with the size of your system memory `SELECT check_pgstac_settings(''4GB'')` to get pg system setting recommendations.';\n ELSE\n IF effective_cache_size < (sysmem * 0.5) THEN\n RAISE WARNING 'effective_cache_size of % is set low for a system with %. Recomended value between % and %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.5), pg_size_pretty(sysmem * 0.75);\n ELSIF effective_cache_size > (sysmem * 0.75) THEN\n RAISE WARNING 'effective_cache_size of % is set high for a system with %. Recomended value between % and %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.5), pg_size_pretty(sysmem * 0.75);\n ELSE\n RAISE NOTICE 'effective_cache_size of % is set appropriately for a system with %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem);\n END IF;\n\n IF shared_buffers < (sysmem * 0.2) THEN\n RAISE WARNING 'shared_buffers of % is set low for a system with %. Recomended value between % and %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.2), pg_size_pretty(sysmem * 0.3);\n ELSIF shared_buffers > (sysmem * 0.3) THEN\n RAISE WARNING 'shared_buffers of % is set high for a system with %. Recomended value between % and %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.2), pg_size_pretty(sysmem * 0.3);\n ELSE\n RAISE NOTICE 'shared_buffers of % is set appropriately for a system with %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem);\n END IF;\n shared_buffers = sysmem * 0.3;\n IF maintenance_work_mem < (sysmem * 0.2) THEN\n RAISE WARNING 'maintenance_work_mem of % is set low for shared_buffers of %. Recomended value between % and %', pg_size_pretty(maintenance_work_mem), pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers * 0.2), pg_size_pretty(shared_buffers * 0.3);\n ELSIF maintenance_work_mem > (shared_buffers * 0.3) THEN\n RAISE WARNING 'maintenance_work_mem of % is set high for shared_buffers of %. Recomended value between % and %', pg_size_pretty(maintenance_work_mem), pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers * 0.2), pg_size_pretty(shared_buffers * 0.3);\n ELSE\n RAISE NOTICE 'maintenance_work_mem of % is set appropriately for shared_buffers of %', pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers);\n END IF;\n\n IF work_mem * max_connections > shared_buffers THEN\n RAISE WARNING 'work_mem setting of % is set high for % max_connections please reduce work_mem to % or decrease max_connections to %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers/max_connections), floor(shared_buffers/work_mem);\n ELSIF work_mem * max_connections < (shared_buffers * 0.75) THEN\n RAISE WARNING 'work_mem setting of % is set low for % max_connections you may consider raising work_mem to % or increasing max_connections to %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers/max_connections), floor(shared_buffers/work_mem);\n ELSE\n RAISE NOTICE 'work_mem setting of % and max_connections of % are adequate for shared_buffers of %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers);\n END IF;\n\n IF random_page_cost / seq_page_cost != 1.1 THEN\n RAISE WARNING 'random_page_cost (%) /seq_page_cost (%) should be set to 1.1 for SSD. Change random_page_cost to %', random_page_cost, seq_page_cost, 1.1 * seq_page_cost;\n ELSE\n RAISE NOTICE 'random_page_cost and seq_page_cost set appropriately for SSD';\n END IF;\n\n IF temp_buffers < greatest(pg_size_bytes('128MB'),(maintenance_work_mem / 2)) THEN\n RAISE WARNING 'pgstac makes heavy use of temp tables, consider raising temp_buffers from % to %', pg_size_pretty(temp_buffers), greatest('128MB', pg_size_pretty((shared_buffers / 16)));\n END IF;\n END IF;\n\n RAISE NOTICE 'VALUES FOR PGSTAC VARIABLES';\n RAISE NOTICE 'These can be set either as GUC system variables or by setting in the pgstac_settings table.';\n\n FOR r IN SELECT name, get_setting(name) as setting, CASE WHEN current_setting(concat('pgstac.',name), TRUE) IS NOT NULL THEN concat('pgstac.',name, ' GUC') WHEN value IS NOT NULL THEN 'pgstac_settings table' ELSE 'Not Set' END as loc FROM pgstac_settings LOOP\n RAISE NOTICE '% is set to % from the %', r.name, r.setting, r.loc;\n END LOOP;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pg_cron';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider intalling pg_cron which can be used to automate tasks';\n ELSE\n RAISE NOTICE 'pg_cron % is installed', settingval;\n END IF;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pgstattuple';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider installing the pgstattuple extension which can be used to help maintain tables and indexes.';\n ELSE\n RAISE NOTICE 'pgstattuple % is installed', settingval;\n END IF;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pg_stat_statements';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider installing the pg_stat_statements extension which is very helpful for tracking the types of queries on the system';\n ELSE\n RAISE NOTICE 'pg_stat_statements % is installed', settingval;\n IF current_setting('pg_stat_statements.track_statements', TRUE) IS DISTINCT FROM 'all' THEN\n RAISE WARNING 'SET pg_stat_statements.track_statements TO ''all''; --In order to track statements within functions.';\n END IF;\n END IF;\n\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac, public SET CLIENT_MIN_MESSAGES TO NOTICE;\nCREATE OR REPLACE FUNCTION to_int(jsonb) RETURNS int AS $$\n SELECT floor(($1->>0)::float)::int;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION to_float(jsonb) RETURNS float AS $$\n SELECT ($1->>0)::float;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION to_tstz(jsonb) RETURNS timestamptz AS $$\n SELECT ($1->>0)::timestamptz;\n$$ LANGUAGE SQL IMMUTABLE STRICT SET TIME ZONE 'UTC' COST 5000 PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION to_text(jsonb) RETURNS text AS $$\n SELECT CASE WHEN jsonb_typeof($1) IN ('array','object') THEN $1::text ELSE $1->>0 END;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION to_text_array(jsonb) RETURNS text[] AS $$\n SELECT\n CASE jsonb_typeof($1)\n WHEN 'array' THEN ARRAY(SELECT jsonb_array_elements_text($1))\n ELSE ARRAY[$1->>0]\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION bbox_geom(_bbox jsonb) RETURNS geometry AS $$\nSELECT CASE jsonb_array_length(_bbox)\n WHEN 4 THEN\n ST_SetSRID(ST_MakeEnvelope(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float,\n (_bbox->>3)::float\n ),4326)\n WHEN 6 THEN\n ST_SetSRID(ST_3DMakeBox(\n ST_MakePoint(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float\n ),\n ST_MakePoint(\n (_bbox->>3)::float,\n (_bbox->>4)::float,\n (_bbox->>5)::float\n )\n ),4326)\n ELSE null END;\n;\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION geom_bbox(_geom geometry) RETURNS jsonb AS $$\n SELECT jsonb_build_array(\n st_xmin(_geom),\n st_ymin(_geom),\n st_xmax(_geom),\n st_ymax(_geom)\n );\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION flip_jsonb_array(j jsonb) RETURNS jsonb AS $$\n SELECT jsonb_agg(value) FROM (SELECT value FROM jsonb_array_elements(j) WITH ORDINALITY ORDER BY ordinality DESC) as t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION explode_dotpaths(j jsonb) RETURNS SETOF text[] AS $$\n SELECT string_to_array(p, '.') as e FROM jsonb_array_elements_text(j) p;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION explode_dotpaths_recurse(IN j jsonb) RETURNS SETOF text[] AS $$\n WITH RECURSIVE t AS (\n SELECT e FROM explode_dotpaths(j) e\n UNION ALL\n SELECT e[1:cardinality(e)-1]\n FROM t\n WHERE cardinality(e)>1\n ) SELECT e FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION jsonb_set_nested(j jsonb, path text[], val jsonb) RETURNS jsonb AS $$\nDECLARE\nBEGIN\n IF cardinality(path) > 1 THEN\n FOR i IN 1..(cardinality(path)-1) LOOP\n IF j #> path[:i] IS NULL THEN\n j := jsonb_set_lax(j, path[:i], '{}', TRUE);\n END IF;\n END LOOP;\n END IF;\n RETURN jsonb_set_lax(j, path, val, true);\n\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\n\n\nCREATE OR REPLACE FUNCTION jsonb_include(j jsonb, f jsonb) RETURNS jsonb AS $$\nDECLARE\n includes jsonb := f-> 'include';\n outj jsonb := '{}'::jsonb;\n path text[];\nBEGIN\n IF\n includes IS NULL\n OR jsonb_array_length(includes) = 0\n THEN\n RETURN j;\n ELSE\n includes := includes || '[\"id\",\"collection\"]'::jsonb;\n FOR path IN SELECT explode_dotpaths(includes) LOOP\n outj := jsonb_set_nested(outj, path, j #> path);\n END LOOP;\n END IF;\n RETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_exclude(j jsonb, f jsonb) RETURNS jsonb AS $$\nDECLARE\n excludes jsonb := f-> 'exclude';\n outj jsonb := j;\n path text[];\nBEGIN\n IF\n excludes IS NULL\n OR jsonb_array_length(excludes) = 0\n THEN\n RETURN j;\n ELSE\n FOR path IN SELECT explode_dotpaths(excludes) LOOP\n outj := outj #- path;\n END LOOP;\n END IF;\n RETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_fields(j jsonb, f jsonb DEFAULT '{\"fields\":[]}') RETURNS jsonb AS $$\n SELECT jsonb_exclude(jsonb_include(j, f), f);\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION merge_jsonb(_a jsonb, _b jsonb) RETURNS jsonb AS $$\n SELECT\n CASE\n WHEN _a = '\"𒍟※\"'::jsonb THEN NULL\n WHEN _a IS NULL OR jsonb_typeof(_a) = 'null' THEN _b\n WHEN jsonb_typeof(_a) = 'object' AND jsonb_typeof(_b) = 'object' THEN\n (\n SELECT\n jsonb_strip_nulls(\n jsonb_object_agg(\n key,\n merge_jsonb(a.value, b.value)\n )\n )\n FROM\n jsonb_each(coalesce(_a,'{}'::jsonb)) as a\n FULL JOIN\n jsonb_each(coalesce(_b,'{}'::jsonb)) as b\n USING (key)\n )\n WHEN\n jsonb_typeof(_a) = 'array'\n AND jsonb_typeof(_b) = 'array'\n AND jsonb_array_length(_a) = jsonb_array_length(_b)\n THEN\n (\n SELECT jsonb_agg(m) FROM\n ( SELECT\n merge_jsonb(\n jsonb_array_elements(_a),\n jsonb_array_elements(_b)\n ) as m\n ) as l\n )\n ELSE _a\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION strip_jsonb(_a jsonb, _b jsonb) RETURNS jsonb AS $$\n SELECT\n CASE\n\n WHEN (_a IS NULL OR jsonb_typeof(_a) = 'null') AND _b IS NOT NULL AND jsonb_typeof(_b) != 'null' THEN '\"𒍟※\"'::jsonb\n WHEN _b IS NULL OR jsonb_typeof(_a) = 'null' THEN _a\n WHEN _a = _b AND jsonb_typeof(_a) = 'object' THEN '{}'::jsonb\n WHEN _a = _b THEN NULL\n WHEN jsonb_typeof(_a) = 'object' AND jsonb_typeof(_b) = 'object' THEN\n (\n SELECT\n jsonb_strip_nulls(\n jsonb_object_agg(\n key,\n strip_jsonb(a.value, b.value)\n )\n )\n FROM\n jsonb_each(_a) as a\n FULL JOIN\n jsonb_each(_b) as b\n USING (key)\n )\n WHEN\n jsonb_typeof(_a) = 'array'\n AND jsonb_typeof(_b) = 'array'\n AND jsonb_array_length(_a) = jsonb_array_length(_b)\n THEN\n (\n SELECT jsonb_agg(m) FROM\n ( SELECT\n strip_jsonb(\n jsonb_array_elements(_a),\n jsonb_array_elements(_b)\n ) as m\n ) as l\n )\n ELSE _a\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION nullif_jsonbnullempty(j jsonb) RETURNS jsonb AS $$\n SELECT nullif(nullif(nullif(j,'null'::jsonb),'{}'::jsonb),'[]'::jsonb);\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION jsonb_array_unique(j jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(jsonb_agg(DISTINCT a)) v FROM jsonb_array_elements(j) a;\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_concat_ignorenull(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT coalesce(a,'[]'::jsonb) || coalesce(b,'[]'::jsonb);\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_least(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(least(nullif_jsonbnullempty(a), nullif_jsonbnullempty(b)));\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_greatest(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(greatest(a, b));\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION first_notnull_sfunc(anyelement, anyelement) RETURNS anyelement AS $$\n SELECT COALESCE($1,$2);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE AGGREGATE first_notnull(anyelement)(\n SFUNC = first_notnull_sfunc,\n STYPE = anyelement\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_array_unique_merge(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_concat_ignorenull,\n FINALFUNC = jsonb_array_unique\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_min(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_least\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_max(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_greatest\n);\n/* looks for a geometry in a stac item first from geometry and falling back to bbox */\nCREATE OR REPLACE FUNCTION stac_geom(value jsonb) RETURNS geometry AS $$\nSELECT\n CASE\n WHEN value ? 'intersects' THEN\n ST_GeomFromGeoJSON(value->>'intersects')\n WHEN value ? 'geometry' THEN\n ST_GeomFromGeoJSON(value->>'geometry')\n WHEN value ? 'bbox' THEN\n pgstac.bbox_geom(value->'bbox')\n ELSE NULL\n END as geometry\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION stac_daterange(\n value jsonb\n) RETURNS tstzrange AS $$\nDECLARE\n props jsonb := value;\n dt timestamptz;\n edt timestamptz;\nBEGIN\n IF props ? 'properties' THEN\n props := props->'properties';\n END IF;\n IF\n props ? 'start_datetime'\n AND props->>'start_datetime' IS NOT NULL\n AND props ? 'end_datetime'\n AND props->>'end_datetime' IS NOT NULL\n THEN\n dt := props->>'start_datetime';\n edt := props->>'end_datetime';\n IF dt > edt THEN\n RAISE EXCEPTION 'start_datetime must be < end_datetime';\n END IF;\n ELSE\n dt := props->>'datetime';\n edt := props->>'datetime';\n END IF;\n IF dt is NULL OR edt IS NULL THEN\n RAISE NOTICE 'DT: %, EDT: %', dt, edt;\n RAISE EXCEPTION 'Either datetime (%) or both start_datetime (%) and end_datetime (%) must be set.', props->>'datetime',props->>'start_datetime',props->>'end_datetime';\n END IF;\n RETURN tstzrange(dt, edt, '[]');\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_datetime(value jsonb) RETURNS timestamptz AS $$\n SELECT lower(stac_daterange(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_end_datetime(value jsonb) RETURNS timestamptz AS $$\n SELECT upper(stac_daterange(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE TABLE IF NOT EXISTS stac_extensions(\n url text PRIMARY KEY,\n content jsonb\n);\nCREATE OR REPLACE FUNCTION collection_base_item(content jsonb) RETURNS jsonb AS $$\n SELECT jsonb_build_object(\n 'type', 'Feature',\n 'stac_version', content->'stac_version',\n 'assets', content->'item_assets',\n 'collection', content->'id'\n );\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE TABLE IF NOT EXISTS collections (\n key bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,\n id text GENERATED ALWAYS AS (content->>'id') STORED UNIQUE NOT NULL,\n content JSONB NOT NULL,\n base_item jsonb GENERATED ALWAYS AS (pgstac.collection_base_item(content)) STORED,\n geometry geometry GENERATED ALWAYS AS (pgstac.collection_geom(content)) STORED,\n datetime timestamptz GENERATED ALWAYS AS (pgstac.collection_datetime(content)) STORED,\n end_datetime timestamptz GENERATED ALWAYS AS (pgstac.collection_enddatetime(content)) STORED,\n private jsonb,\n partition_trunc text CHECK (partition_trunc IN ('year', 'month'))\n);\n\nCREATE OR REPLACE FUNCTION collection_base_item(cid text) RETURNS jsonb AS $$\n SELECT pgstac.collection_base_item(content) FROM pgstac.collections WHERE id = cid LIMIT 1;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION create_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION update_collection(data jsonb) RETURNS VOID AS $$\nDECLARE\n out collections%ROWTYPE;\nBEGIN\n UPDATE collections SET content=data WHERE id = data->>'id' RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ON CONFLICT (id) DO\n UPDATE\n SET content=EXCLUDED.content\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION delete_collection(_id text) RETURNS VOID AS $$\nDECLARE\n out collections%ROWTYPE;\nBEGIN\n DELETE FROM collections WHERE id = _id RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION get_collection(id text) RETURNS jsonb AS $$\n SELECT content FROM collections\n WHERE id=$1\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION all_collections() RETURNS jsonb AS $$\n SELECT coalesce(jsonb_agg(content), '[]'::jsonb) FROM collections;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION collection_delete_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n collection_base_partition text := concat('_items_', OLD.key);\nBEGIN\n EXECUTE format($q$\n DELETE FROM partition_stats WHERE partition IN (\n SELECT partition FROM partition_sys_meta\n WHERE collection=%L\n );\n DROP TABLE IF EXISTS %I CASCADE;\n $q$,\n OLD.id,\n collection_base_partition\n );\n RETURN OLD;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER collection_delete_trigger BEFORE DELETE ON collections\nFOR EACH ROW EXECUTE FUNCTION collection_delete_trigger_func();\nCREATE OR REPLACE FUNCTION queryable_signature(n text, c text[]) RETURNS text AS $$\n SELECT concat(n, c);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TABLE queryables (\n id bigint GENERATED ALWAYS AS identity PRIMARY KEY,\n name text NOT NULL,\n collection_ids text[], -- used to determine what partitions to create indexes on\n definition jsonb,\n property_path text,\n property_wrapper text,\n property_index_type text\n);\nCREATE INDEX queryables_name_idx ON queryables (name);\nCREATE INDEX queryables_collection_idx ON queryables USING GIN (collection_ids);\nCREATE INDEX queryables_property_wrapper_idx ON queryables (property_wrapper);\n\nCREATE OR REPLACE FUNCTION pgstac.queryables_constraint_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n allcollections text[];\nBEGIN\n RAISE NOTICE 'Making sure that name/collection is unique for queryables %', NEW;\n IF NEW.collection_ids IS NOT NULL THEN\n IF EXISTS (\n SELECT 1\n FROM unnest(NEW.collection_ids) c\n LEFT JOIN\n collections\n ON (collections.id = c)\n WHERE collections.id IS NULL\n ) THEN\n RAISE foreign_key_violation USING MESSAGE = format(\n 'One or more collections in %s do not exist.', NEW.collection_ids\n );\n RETURN NULL;\n END IF;\n END IF;\n IF TG_OP = 'INSERT' THEN\n IF EXISTS (\n SELECT 1 FROM queryables q\n WHERE\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n )\n ) THEN\n RAISE unique_violation USING MESSAGE = format(\n 'There is already a queryable for %s for a collection in %s: %s',\n NEW.name,\n NEW.collection_ids,\n\t\t\t\t(SELECT json_agg(row_to_json(q)) FROM queryables q WHERE\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n ))\n );\n RETURN NULL;\n END IF;\n END IF;\n IF TG_OP = 'UPDATE' THEN\n IF EXISTS (\n SELECT 1 FROM queryables q\n WHERE\n q.id != NEW.id\n AND\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n )\n ) THEN\n RAISE unique_violation\n USING MESSAGE = format(\n 'There is already a queryable for %s for a collection in %s',\n NEW.name,\n NEW.collection_ids\n );\n RETURN NULL;\n END IF;\n END IF;\n\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER queryables_constraint_insert_trigger\nBEFORE INSERT ON queryables\nFOR EACH ROW EXECUTE PROCEDURE queryables_constraint_triggerfunc();\n\nCREATE TRIGGER queryables_constraint_update_trigger\nBEFORE UPDATE ON queryables\nFOR EACH ROW\nWHEN (NEW.name = OLD.name AND NEW.collection_ids IS DISTINCT FROM OLD.collection_ids)\nEXECUTE PROCEDURE queryables_constraint_triggerfunc();\n\n\nCREATE OR REPLACE FUNCTION array_to_path(arr text[]) RETURNS text AS $$\n SELECT string_agg(\n quote_literal(v),\n '->'\n ) FROM unnest(arr) v;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\n\n\nCREATE OR REPLACE FUNCTION queryable(\n IN dotpath text,\n OUT path text,\n OUT expression text,\n OUT wrapper text,\n OUT nulled_wrapper text\n) AS $$\nDECLARE\n q RECORD;\n path_elements text[];\nBEGIN\n IF dotpath IN ('id', 'geometry', 'datetime', 'end_datetime', 'collection') THEN\n path := dotpath;\n expression := dotpath;\n wrapper := NULL;\n RETURN;\n END IF;\n SELECT * INTO q FROM queryables\n WHERE\n name=dotpath\n OR name = 'properties.' || dotpath\n OR name = replace(dotpath, 'properties.', '')\n ;\n IF q.property_wrapper IS NULL THEN\n IF q.definition->>'type' = 'number' THEN\n wrapper := 'to_float';\n nulled_wrapper := wrapper;\n ELSIF q.definition->>'format' = 'date-time' THEN\n wrapper := 'to_tstz';\n nulled_wrapper := wrapper;\n ELSE\n nulled_wrapper := NULL;\n wrapper := 'to_text';\n END IF;\n ELSE\n wrapper := q.property_wrapper;\n nulled_wrapper := wrapper;\n END IF;\n IF q.property_path IS NOT NULL THEN\n path := q.property_path;\n ELSE\n path_elements := string_to_array(dotpath, '.');\n IF path_elements[1] IN ('links', 'assets', 'stac_version', 'stac_extensions') THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSIF path_elements[1] = 'properties' THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSE\n path := format($F$content->'properties'->%s$F$, array_to_path(path_elements));\n END IF;\n END IF;\n expression := format('%I(%s)', wrapper, path);\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\nCREATE OR REPLACE FUNCTION unnest_collection(collection_ids text[] DEFAULT NULL) RETURNS SETOF text AS $$\n DECLARE\n BEGIN\n IF collection_ids IS NULL THEN\n RETURN QUERY SELECT id FROM collections;\n END IF;\n RETURN QUERY SELECT unnest(collection_ids);\n END;\n$$ LANGUAGE PLPGSQL STABLE;\n\nCREATE OR REPLACE FUNCTION normalize_indexdef(def text) RETURNS text AS $$\nDECLARE\nBEGIN\n def := btrim(def, ' \\n\\t');\n\tdef := regexp_replace(def, '^CREATE (UNIQUE )?INDEX ([^ ]* )?ON (ONLY )?([^ ]* )?', '', 'i');\n RETURN def;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION indexdef(q queryables) RETURNS text AS $$\n DECLARE\n out text;\n BEGIN\n IF q.name = 'id' THEN\n out := 'CREATE UNIQUE INDEX ON %I USING btree (id)';\n ELSIF q.name = 'datetime' THEN\n out := 'CREATE INDEX ON %I USING btree (datetime DESC, end_datetime)';\n ELSIF q.name = 'geometry' THEN\n out := 'CREATE INDEX ON %I USING gist (geometry)';\n ELSE\n out := format($q$CREATE INDEX ON %%I USING %s (%s(((content -> 'properties'::text) -> %L::text)))$q$,\n lower(COALESCE(q.property_index_type, 'BTREE')),\n lower(COALESCE(q.property_wrapper, 'to_text')),\n q.name\n );\n END IF;\n RETURN btrim(out, ' \\n\\t');\n END;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nDROP VIEW IF EXISTS pgstac_indexes;\nCREATE VIEW pgstac_indexes AS\nSELECT\n i.schemaname,\n i.tablename,\n i.indexname,\n regexp_replace(btrim(replace(replace(indexdef, i.indexname, ''),'pgstac.',''),' \\t\\n'), '[ ]+', ' ', 'g') as idx,\n COALESCE(\n (regexp_match(indexdef, '\\(([a-zA-Z]+)\\)'))[1],\n (regexp_match(indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_-]+)''::text'))[1],\n CASE WHEN indexdef ~* '\\(datetime desc, end_datetime\\)' THEN 'datetime' ELSE NULL END\n ) AS field,\n pg_table_size(i.indexname::text) as index_size,\n pg_size_pretty(pg_table_size(i.indexname::text)) as index_size_pretty\nFROM\n pg_indexes i\nWHERE i.schemaname='pgstac' and i.tablename ~ '_items_' AND indexdef !~* ' only ';\n\nDROP VIEW IF EXISTS pgstac_index_stats;\nCREATE VIEW pgstac_indexes_stats AS\nSELECT\n i.schemaname,\n i.tablename,\n i.indexname,\n indexdef,\n COALESCE(\n (regexp_match(indexdef, '\\(([a-zA-Z]+)\\)'))[1],\n (regexp_match(indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_]+)''::text'))[1],\n CASE WHEN indexdef ~* '\\(datetime desc, end_datetime\\)' THEN 'datetime_end_datetime' ELSE NULL END\n ) AS field,\n pg_table_size(i.indexname::text) as index_size,\n pg_size_pretty(pg_table_size(i.indexname::text)) as index_size_pretty,\n n_distinct,\n most_common_vals::text::text[],\n most_common_freqs::text::text[],\n histogram_bounds::text::text[],\n correlation\nFROM\n pg_indexes i\n LEFT JOIN pg_stats s ON (s.tablename = i.indexname)\nWHERE i.schemaname='pgstac' and i.tablename ~ '_items_';\n\nCREATE OR REPLACE FUNCTION queryable_indexes(\n IN treeroot text DEFAULT 'items',\n IN changes boolean DEFAULT FALSE,\n OUT collection text,\n OUT partition text,\n OUT field text,\n OUT indexname text,\n OUT existing_idx text,\n OUT queryable_idx text\n) RETURNS SETOF RECORD AS $$\nWITH p AS (\n SELECT\n relid::text as partition,\n replace(replace(\n CASE\n WHEN parentrelid::regclass::text='items' THEN pg_get_expr(c.relpartbound, c.oid)\n ELSE pg_get_expr(parent.relpartbound, parent.oid)\n END,\n 'FOR VALUES IN (''',''), ''')',\n ''\n ) AS collection\n FROM pg_partition_tree(treeroot)\n JOIN pg_class c ON (relid::regclass = c.oid)\n JOIN pg_class parent ON (parentrelid::regclass = parent.oid AND isleaf)\n ), i AS (\n SELECT\n partition,\n indexname,\n regexp_replace(btrim(replace(replace(indexdef, indexname, ''),'pgstac.',''),' \\t\\n'), '[ ]+', ' ', 'g') as iidx,\n COALESCE(\n (regexp_match(indexdef, '\\(([a-zA-Z]+)\\)'))[1],\n (regexp_match(indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_-]+)''::text'))[1],\n CASE WHEN indexdef ~* '\\(datetime desc, end_datetime\\)' THEN 'datetime' ELSE NULL END\n ) AS field\n FROM\n pg_indexes\n JOIN p ON (tablename=partition)\n ), q AS (\n SELECT\n name AS field,\n collection,\n partition,\n format(indexdef(queryables), partition) as qidx\n FROM queryables, unnest_collection(queryables.collection_ids) collection\n JOIN p USING (collection)\n WHERE property_index_type IS NOT NULL OR name IN ('datetime','geometry','id')\n )\n SELECT\n collection,\n partition,\n field,\n indexname,\n iidx as existing_idx,\n qidx as queryable_idx\n FROM i FULL JOIN q USING (field, partition)\n WHERE CASE WHEN changes THEN lower(iidx) IS DISTINCT FROM lower(qidx) ELSE TRUE END;\n;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION maintain_index(\n indexname text,\n queryable_idx text,\n dropindexes boolean DEFAULT FALSE,\n rebuildindexes boolean DEFAULT FALSE,\n idxconcurrently boolean DEFAULT FALSE\n) RETURNS VOID AS $$\nDECLARE\nBEGIN\n IF indexname IS NOT NULL THEN\n IF dropindexes OR queryable_idx IS NOT NULL THEN\n EXECUTE format('DROP INDEX IF EXISTS %I;', indexname);\n ELSIF rebuildindexes THEN\n IF idxconcurrently THEN\n EXECUTE format('REINDEX INDEX CONCURRENTLY %I;', indexname);\n ELSE\n EXECUTE format('REINDEX INDEX CONCURRENTLY %I;', indexname);\n END IF;\n END IF;\n END IF;\n IF queryable_idx IS NOT NULL THEN\n IF idxconcurrently THEN\n EXECUTE replace(queryable_idx, 'INDEX', 'INDEX CONCURRENTLY');\n ELSE EXECUTE queryable_idx;\n END IF;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\n\nset check_function_bodies to off;\nCREATE OR REPLACE FUNCTION maintain_partition_queries(\n part text DEFAULT 'items',\n dropindexes boolean DEFAULT FALSE,\n rebuildindexes boolean DEFAULT FALSE,\n idxconcurrently boolean DEFAULT FALSE\n) RETURNS SETOF text AS $$\nDECLARE\n rec record;\n q text;\nBEGIN\n FOR rec IN (\n SELECT * FROM queryable_indexes(part,true)\n ) LOOP\n q := format(\n 'SELECT maintain_index(\n %L,%L,%L,%L,%L\n );',\n rec.indexname,\n rec.queryable_idx,\n dropindexes,\n rebuildindexes,\n idxconcurrently\n );\n RAISE NOTICE 'Q: %s', q;\n RETURN NEXT q;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION maintain_partitions(\n part text DEFAULT 'items',\n dropindexes boolean DEFAULT FALSE,\n rebuildindexes boolean DEFAULT FALSE\n) RETURNS VOID AS $$\n WITH t AS (\n SELECT run_or_queue(q) FROM maintain_partition_queries(part, dropindexes, rebuildindexes) q\n ) SELECT count(*) FROM t;\n$$ LANGUAGE SQL;\n\n\nCREATE OR REPLACE FUNCTION queryables_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\nBEGIN\n PERFORM maintain_partitions();\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER queryables_trigger AFTER INSERT OR UPDATE ON queryables\nFOR EACH STATEMENT EXECUTE PROCEDURE queryables_trigger_func();\n\n\nCREATE OR REPLACE FUNCTION get_queryables(_collection_ids text[] DEFAULT NULL) RETURNS jsonb AS $$\nDECLARE\nBEGIN\n -- Build up queryables if the input contains valid collection ids or is empty\n IF EXISTS (\n SELECT 1 FROM collections\n WHERE\n _collection_ids IS NULL\n OR cardinality(_collection_ids) = 0\n OR id = ANY(_collection_ids)\n )\n THEN\n RETURN (\n WITH base AS (\n SELECT\n unnest(collection_ids) as collection_id,\n name,\n coalesce(definition, '{\"type\":\"string\"}'::jsonb) as definition\n FROM queryables\n WHERE\n _collection_ids IS NULL OR\n _collection_ids = '{}'::text[] OR\n _collection_ids && collection_ids\n UNION ALL\n SELECT null, name, coalesce(definition, '{\"type\":\"string\"}'::jsonb) as definition\n FROM queryables WHERE collection_ids IS NULL OR collection_ids = '{}'::text[]\n ), g AS (\n SELECT\n name,\n first_notnull(definition) as definition,\n jsonb_array_unique_merge(definition->'enum') as enum,\n jsonb_min(definition->'minimum') as minimum,\n jsonb_min(definition->'maxiumn') as maximum\n FROM base\n GROUP BY 1\n )\n SELECT\n jsonb_build_object(\n '$schema', 'http://json-schema.org/draft-07/schema#',\n '$id', '',\n 'type', 'object',\n 'title', 'STAC Queryables.',\n 'properties', jsonb_object_agg(\n name,\n definition\n ||\n jsonb_strip_nulls(jsonb_build_object(\n 'enum', enum,\n 'minimum', minimum,\n 'maximum', maximum\n ))\n ),\n 'additionalProperties', pgstac.additional_properties()\n )\n FROM g\n );\n ELSE\n RETURN NULL;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\nCREATE OR REPLACE FUNCTION get_queryables(_collection text DEFAULT NULL) RETURNS jsonb AS $$\n SELECT\n CASE\n WHEN _collection IS NULL THEN get_queryables(NULL::text[])\n ELSE get_queryables(ARRAY[_collection])\n END\n ;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_queryables() RETURNS jsonb AS $$\n SELECT get_queryables(NULL::text[]);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION schema_qualify_refs(url text, j jsonb) returns jsonb as $$\n SELECT regexp_replace(j::text, '\"\\$ref\": \"#', concat('\"$ref\": \"', url, '#'), 'g')::jsonb;\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE VIEW stac_extension_queryables AS\nSELECT DISTINCT key as name, schema_qualify_refs(e.url, j.value) as definition FROM stac_extensions e, jsonb_each(e.content->'definitions'->'fields'->'properties') j;\n\n\nCREATE OR REPLACE FUNCTION missing_queryables(_collection text, _tablesample float DEFAULT 5, minrows float DEFAULT 10) RETURNS TABLE(collection text, name text, definition jsonb, property_wrapper text) AS $$\nDECLARE\n q text;\n _partition text;\n explain_json json;\n psize float;\n estrows float;\nBEGIN\n SELECT format('_items_%s', key) INTO _partition FROM collections WHERE id=_collection;\n\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM %I;', _partition)\n INTO explain_json;\n psize := explain_json->0->'Plan'->'Plan Rows';\n estrows := _tablesample * .01 * psize;\n IF estrows < minrows THEN\n _tablesample := least(100,greatest(_tablesample, (estrows / psize) / 100));\n RAISE NOTICE '%', (psize / estrows) / 100;\n END IF;\n RAISE NOTICE 'Using tablesample % to find missing queryables from % % that has ~% rows estrows: %', _tablesample, _collection, _partition, psize, estrows;\n\n q := format(\n $q$\n WITH q AS (\n SELECT * FROM queryables\n WHERE\n collection_ids IS NULL\n OR %L = ANY(collection_ids)\n ), t AS (\n SELECT\n content->'properties' AS properties\n FROM\n %I\n TABLESAMPLE SYSTEM(%L)\n ), p AS (\n SELECT DISTINCT ON (key)\n key,\n value,\n s.definition\n FROM t\n JOIN LATERAL jsonb_each(properties) ON TRUE\n LEFT JOIN q ON (q.name=key)\n LEFT JOIN stac_extension_queryables s ON (s.name=key)\n WHERE q.definition IS NULL\n )\n SELECT\n %L,\n key,\n COALESCE(definition, jsonb_build_object('type',jsonb_typeof(value))) as definition,\n CASE\n WHEN definition->>'type' = 'integer' THEN 'to_int'\n WHEN COALESCE(definition->>'type', jsonb_typeof(value)) = 'number' THEN 'to_float'\n WHEN COALESCE(definition->>'type', jsonb_typeof(value)) = 'array' THEN 'to_text_array'\n ELSE 'to_text'\n END\n FROM p;\n $q$,\n _collection,\n _partition,\n _tablesample,\n _collection\n );\n RETURN QUERY EXECUTE q;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION missing_queryables(_tablesample float DEFAULT 5) RETURNS TABLE(collection_ids text[], name text, definition jsonb, property_wrapper text) AS $$\n SELECT\n array_agg(collection),\n name,\n definition,\n property_wrapper\n FROM\n collections\n JOIN LATERAL\n missing_queryables(id, _tablesample) c\n ON TRUE\n GROUP BY\n 2,3,4\n ORDER BY 2,1\n ;\n$$ LANGUAGE SQL;\nCREATE OR REPLACE FUNCTION parse_dtrange(\n _indate jsonb,\n relative_base timestamptz DEFAULT date_trunc('hour', CURRENT_TIMESTAMP)\n) RETURNS tstzrange AS $$\nDECLARE\n timestrs text[];\n s timestamptz;\n e timestamptz;\nBEGIN\n timestrs :=\n CASE\n WHEN _indate ? 'timestamp' THEN\n ARRAY[_indate->>'timestamp']\n WHEN _indate ? 'interval' THEN\n to_text_array(_indate->'interval')\n WHEN jsonb_typeof(_indate) = 'array' THEN\n to_text_array(_indate)\n ELSE\n regexp_split_to_array(\n _indate->>0,\n '/'\n )\n END;\n RAISE NOTICE 'TIMESTRS %', timestrs;\n IF cardinality(timestrs) = 1 THEN\n IF timestrs[1] ILIKE 'P%' THEN\n RETURN tstzrange(relative_base - upper(timestrs[1])::interval, relative_base, '[)');\n END IF;\n s := timestrs[1]::timestamptz;\n RETURN tstzrange(s, s, '[]');\n END IF;\n\n IF cardinality(timestrs) != 2 THEN\n RAISE EXCEPTION 'Timestamp cannot have more than 2 values';\n END IF;\n\n IF timestrs[1] = '..' OR timestrs[1] = '' THEN\n s := '-infinity'::timestamptz;\n e := timestrs[2]::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] = '..' OR timestrs[2] = '' THEN\n s := timestrs[1]::timestamptz;\n e := 'infinity'::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[1] ILIKE 'P%' AND timestrs[2] NOT ILIKE 'P%' THEN\n e := timestrs[2]::timestamptz;\n s := e - upper(timestrs[1])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] ILIKE 'P%' AND timestrs[1] NOT ILIKE 'P%' THEN\n s := timestrs[1]::timestamptz;\n e := s + upper(timestrs[2])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n s := timestrs[1]::timestamptz;\n e := timestrs[2]::timestamptz;\n\n RETURN tstzrange(s,e,'[)');\n\n RETURN NULL;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT PARALLEL SAFE SET TIME ZONE 'UTC';\n\nCREATE OR REPLACE FUNCTION parse_dtrange(\n _indate text,\n relative_base timestamptz DEFAULT CURRENT_TIMESTAMP\n) RETURNS tstzrange AS $$\n SELECT parse_dtrange(to_jsonb(_indate), relative_base);\n$$ LANGUAGE SQL STABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION temporal_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\n ll text := 'datetime';\n lh text := 'end_datetime';\n rrange tstzrange;\n rl text;\n rh text;\n outq text;\nBEGIN\n rrange := parse_dtrange(args->1);\n RAISE NOTICE 'Constructing temporal query OP: %, ARGS: %, RRANGE: %', op, args, rrange;\n op := lower(op);\n rl := format('%L::timestamptz', lower(rrange));\n rh := format('%L::timestamptz', upper(rrange));\n outq := CASE op\n WHEN 't_before' THEN 'lh < rl'\n WHEN 't_after' THEN 'll > rh'\n WHEN 't_meets' THEN 'lh = rl'\n WHEN 't_metby' THEN 'll = rh'\n WHEN 't_overlaps' THEN 'll < rl AND rl < lh < rh'\n WHEN 't_overlappedby' THEN 'rl < ll < rh AND lh > rh'\n WHEN 't_starts' THEN 'll = rl AND lh < rh'\n WHEN 't_startedby' THEN 'll = rl AND lh > rh'\n WHEN 't_during' THEN 'll > rl AND lh < rh'\n WHEN 't_contains' THEN 'll < rl AND lh > rh'\n WHEN 't_finishes' THEN 'll > rl AND lh = rh'\n WHEN 't_finishedby' THEN 'll < rl AND lh = rh'\n WHEN 't_equals' THEN 'll = rl AND lh = rh'\n WHEN 't_disjoint' THEN 'NOT (ll <= rh AND lh >= rl)'\n WHEN 't_intersects' THEN 'll <= rh AND lh >= rl'\n WHEN 'anyinteracts' THEN 'll <= rh AND lh >= rl'\n END;\n outq := regexp_replace(outq, '\\mll\\M', ll);\n outq := regexp_replace(outq, '\\mlh\\M', lh);\n outq := regexp_replace(outq, '\\mrl\\M', rl);\n outq := regexp_replace(outq, '\\mrh\\M', rh);\n outq := format('(%s)', outq);\n RETURN outq;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\n\n\nCREATE OR REPLACE FUNCTION spatial_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\n geom text;\n j jsonb := args->1;\nBEGIN\n op := lower(op);\n RAISE NOTICE 'Constructing spatial query OP: %, ARGS: %', op, args;\n IF op NOT IN ('s_equals','s_disjoint','s_touches','s_within','s_overlaps','s_crosses','s_intersects','intersects','s_contains') THEN\n RAISE EXCEPTION 'Spatial Operator % Not Supported', op;\n END IF;\n op := regexp_replace(op, '^s_', 'st_');\n IF op = 'intersects' THEN\n op := 'st_intersects';\n END IF;\n -- Convert geometry to WKB string\n IF j ? 'type' AND j ? 'coordinates' THEN\n geom := st_geomfromgeojson(j)::text;\n ELSIF jsonb_typeof(j) = 'array' THEN\n geom := bbox_geom(j)::text;\n END IF;\n\n RETURN format('%s(geometry, %L::geometry)', op, geom);\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION query_to_cql2(q jsonb) RETURNS jsonb AS $$\n-- Translates anything passed in through the deprecated \"query\" into equivalent CQL2\nWITH t AS (\n SELECT key as property, value as ops\n FROM jsonb_each(q)\n), t2 AS (\n SELECT property, (jsonb_each(ops)).*\n FROM t WHERE jsonb_typeof(ops) = 'object'\n UNION ALL\n SELECT property, 'eq', ops\n FROM t WHERE jsonb_typeof(ops) != 'object'\n)\nSELECT\n jsonb_strip_nulls(jsonb_build_object(\n 'op', 'and',\n 'args', jsonb_agg(\n jsonb_build_object(\n 'op', key,\n 'args', jsonb_build_array(\n jsonb_build_object('property',property),\n value\n )\n )\n )\n )\n) as qcql FROM t2\n;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\nCREATE OR REPLACE FUNCTION cql1_to_cql2(j jsonb) RETURNS jsonb AS $$\nDECLARE\n args jsonb;\n ret jsonb;\nBEGIN\n RAISE NOTICE 'CQL1_TO_CQL2: %', j;\n IF j ? 'filter' THEN\n RETURN cql1_to_cql2(j->'filter');\n END IF;\n IF j ? 'property' THEN\n RETURN j;\n END IF;\n IF jsonb_typeof(j) = 'array' THEN\n SELECT jsonb_agg(cql1_to_cql2(el)) INTO args FROM jsonb_array_elements(j) el;\n RETURN args;\n END IF;\n IF jsonb_typeof(j) = 'number' THEN\n RETURN j;\n END IF;\n IF jsonb_typeof(j) = 'string' THEN\n RETURN j;\n END IF;\n\n IF jsonb_typeof(j) = 'object' THEN\n SELECT jsonb_build_object(\n 'op', key,\n 'args', cql1_to_cql2(value)\n ) INTO ret\n FROM jsonb_each(j)\n WHERE j IS NOT NULL;\n RETURN ret;\n END IF;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE STRICT;\n\nCREATE TABLE cql2_ops (\n op text PRIMARY KEY,\n template text,\n types text[]\n);\nINSERT INTO cql2_ops (op, template, types) VALUES\n ('eq', '%s = %s', NULL),\n ('neq', '%s != %s', NULL),\n ('ne', '%s != %s', NULL),\n ('!=', '%s != %s', NULL),\n ('<>', '%s != %s', NULL),\n ('lt', '%s < %s', NULL),\n ('lte', '%s <= %s', NULL),\n ('gt', '%s > %s', NULL),\n ('gte', '%s >= %s', NULL),\n ('le', '%s <= %s', NULL),\n ('ge', '%s >= %s', NULL),\n ('=', '%s = %s', NULL),\n ('<', '%s < %s', NULL),\n ('<=', '%s <= %s', NULL),\n ('>', '%s > %s', NULL),\n ('>=', '%s >= %s', NULL),\n ('like', '%s LIKE %s', NULL),\n ('ilike', '%s ILIKE %s', NULL),\n ('+', '%s + %s', NULL),\n ('-', '%s - %s', NULL),\n ('*', '%s * %s', NULL),\n ('/', '%s / %s', NULL),\n ('not', 'NOT (%s)', NULL),\n ('between', '%s BETWEEN %s AND %s', NULL),\n ('isnull', '%s IS NULL', NULL),\n ('upper', 'upper(%s)', NULL),\n ('lower', 'lower(%s)', NULL)\nON CONFLICT (op) DO UPDATE\n SET\n template = EXCLUDED.template\n;\n\n\nCREATE OR REPLACE FUNCTION cql2_query(j jsonb, wrapper text DEFAULT NULL) RETURNS text AS $$\n#variable_conflict use_variable\nDECLARE\n args jsonb := j->'args';\n arg jsonb;\n op text := lower(j->>'op');\n cql2op RECORD;\n literal text;\n _wrapper text;\n leftarg text;\n rightarg text;\n prop text;\n extra_props bool := pgstac.additional_properties();\nBEGIN\n IF j IS NULL OR (op IS NOT NULL AND args IS NULL) THEN\n RETURN NULL;\n END IF;\n RAISE NOTICE 'CQL2_QUERY: %', j;\n\n -- check if all properties are represented in the queryables\n IF NOT extra_props THEN\n FOR prop IN\n SELECT DISTINCT p->>0\n FROM jsonb_path_query(j, 'strict $.**.property') p\n WHERE p->>0 NOT IN ('id', 'datetime', 'end_datetime', 'collection')\n LOOP\n IF (queryable(prop)).nulled_wrapper IS NULL THEN\n RAISE EXCEPTION 'Term % is not found in queryables.', prop;\n END IF;\n END LOOP;\n END IF;\n\n IF j ? 'filter' THEN\n RETURN cql2_query(j->'filter');\n END IF;\n\n IF j ? 'upper' THEN\n RETURN cql2_query(jsonb_build_object('op', 'upper', 'args', j->'upper'));\n END IF;\n\n IF j ? 'lower' THEN\n RETURN cql2_query(jsonb_build_object('op', 'lower', 'args', j->'lower'));\n END IF;\n\n -- Temporal Query\n IF op ilike 't_%' or op = 'anyinteracts' THEN\n RETURN temporal_op_query(op, args);\n END IF;\n\n -- If property is a timestamp convert it to text to use with\n -- general operators\n IF j ? 'timestamp' THEN\n RETURN format('%L::timestamptz', to_tstz(j->'timestamp'));\n END IF;\n IF j ? 'interval' THEN\n RAISE EXCEPTION 'Please use temporal operators when using intervals.';\n RETURN NONE;\n END IF;\n\n -- Spatial Query\n IF op ilike 's_%' or op = 'intersects' THEN\n RETURN spatial_op_query(op, args);\n END IF;\n\n IF op IN ('a_equals','a_contains','a_contained_by','a_overlaps') THEN\n IF args->0 ? 'property' THEN\n leftarg := format('to_text_array(%s)', (queryable(args->0->>'property')).path);\n END IF;\n IF args->1 ? 'property' THEN\n rightarg := format('to_text_array(%s)', (queryable(args->1->>'property')).path);\n END IF;\n RETURN FORMAT(\n '%s %s %s',\n COALESCE(leftarg, quote_literal(to_text_array(args->0))),\n CASE op\n WHEN 'a_equals' THEN '='\n WHEN 'a_contains' THEN '@>'\n WHEN 'a_contained_by' THEN '<@'\n WHEN 'a_overlaps' THEN '&&'\n END,\n COALESCE(rightarg, quote_literal(to_text_array(args->1)))\n );\n END IF;\n\n IF op = 'in' THEN\n RAISE NOTICE 'IN : % % %', args, jsonb_build_array(args->0), args->1;\n args := jsonb_build_array(args->0) || (args->1);\n RAISE NOTICE 'IN2 : %', args;\n END IF;\n\n\n\n IF op = 'between' THEN\n args = jsonb_build_array(\n args->0,\n args->1->0,\n args->1->1\n );\n END IF;\n\n -- Make sure that args is an array and run cql2_query on\n -- each element of the array\n RAISE NOTICE 'ARGS PRE: %', args;\n IF j ? 'args' THEN\n IF jsonb_typeof(args) != 'array' THEN\n args := jsonb_build_array(args);\n END IF;\n\n IF jsonb_path_exists(args, '$[*] ? (@.property == \"id\" || @.property == \"datetime\" || @.property == \"end_datetime\" || @.property == \"collection\")') THEN\n wrapper := NULL;\n ELSE\n -- if any of the arguments are a property, try to get the property_wrapper\n FOR arg IN SELECT jsonb_path_query(args, '$[*] ? (@.property != null)') LOOP\n RAISE NOTICE 'Arg: %', arg;\n wrapper := (queryable(arg->>'property')).nulled_wrapper;\n RAISE NOTICE 'Property: %, Wrapper: %', arg, wrapper;\n IF wrapper IS NOT NULL THEN\n EXIT;\n END IF;\n END LOOP;\n\n -- if the property was not in queryables, see if any args were numbers\n IF\n wrapper IS NULL\n AND jsonb_path_exists(args, '$[*] ? (@.type()==\"number\")')\n THEN\n wrapper := 'to_float';\n END IF;\n wrapper := coalesce(wrapper, 'to_text');\n END IF;\n\n SELECT jsonb_agg(cql2_query(a, wrapper))\n INTO args\n FROM jsonb_array_elements(args) a;\n END IF;\n RAISE NOTICE 'ARGS: %', args;\n\n IF op IN ('and', 'or') THEN\n RETURN\n format(\n '(%s)',\n array_to_string(to_text_array(args), format(' %s ', upper(op)))\n );\n END IF;\n\n IF op = 'in' THEN\n RAISE NOTICE 'IN -- % %', args->0, to_text(args->0);\n RETURN format(\n '%s IN (%s)',\n to_text(args->0),\n array_to_string((to_text_array(args))[2:], ',')\n );\n END IF;\n\n -- Look up template from cql2_ops\n IF j ? 'op' THEN\n SELECT * INTO cql2op FROM cql2_ops WHERE cql2_ops.op ilike op;\n IF FOUND THEN\n -- If specific index set in queryables for a property cast other arguments to that type\n\n RETURN format(\n cql2op.template,\n VARIADIC (to_text_array(args))\n );\n ELSE\n RAISE EXCEPTION 'Operator % Not Supported.', op;\n END IF;\n END IF;\n\n\n IF wrapper IS NOT NULL THEN\n RAISE NOTICE 'Wrapping % with %', j, wrapper;\n IF j ? 'property' THEN\n RETURN format('%I(%s)', wrapper, (queryable(j->>'property')).path);\n ELSE\n RETURN format('%I(%L)', wrapper, j);\n END IF;\n ELSIF j ? 'property' THEN\n RETURN quote_ident(j->>'property');\n END IF;\n\n RETURN quote_literal(to_text(j));\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION paging_dtrange(\n j jsonb\n) RETURNS tstzrange AS $$\nDECLARE\n op text;\n filter jsonb := j->'filter';\n dtrange tstzrange := tstzrange('-infinity'::timestamptz,'infinity'::timestamptz);\n sdate timestamptz := '-infinity'::timestamptz;\n edate timestamptz := 'infinity'::timestamptz;\n jpitem jsonb;\nBEGIN\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n END IF;\n IF NOT (filter @? '$.**.op ? (@ == \"or\" || @ == \"not\")') THEN\n FOR jpitem IN SELECT j FROM jsonb_path_query(filter,'strict $.** ? (@.args[*].property == \"datetime\")'::jsonpath) j LOOP\n op := lower(jpitem->>'op');\n dtrange := parse_dtrange(jpitem->'args'->1);\n IF op IN ('<=', 'lt', 'lte', '<', 'le', 't_before') THEN\n sdate := greatest(sdate,'-infinity');\n edate := least(edate, upper(dtrange));\n ELSIF op IN ('>=', '>', 'gt', 'gte', 'ge', 't_after') THEN\n edate := least(edate, 'infinity');\n sdate := greatest(sdate, lower(dtrange));\n ELSIF op IN ('=', 'eq') THEN\n edate := least(edate, upper(dtrange));\n sdate := greatest(sdate, lower(dtrange));\n END IF;\n RAISE NOTICE '2 OP: %, ARGS: %, DTRANGE: %, SDATE: %, EDATE: %', op, jpitem->'args'->1, dtrange, sdate, edate;\n END LOOP;\n END IF;\n IF sdate > edate THEN\n RETURN 'empty'::tstzrange;\n END IF;\n RETURN tstzrange(sdate,edate, '[]');\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT SET TIME ZONE 'UTC';\n\nCREATE OR REPLACE FUNCTION paging_collections(\n IN j jsonb\n) RETURNS text[] AS $$\nDECLARE\n filter jsonb := j->'filter';\n jpitem jsonb;\n op text;\n args jsonb;\n arg jsonb;\n collections text[];\nBEGIN\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n END IF;\n IF NOT (filter @? '$.**.op ? (@ == \"or\" || @ == \"not\")') THEN\n FOR jpitem IN SELECT j FROM jsonb_path_query(filter,'strict $.** ? (@.args[*].property == \"collection\")'::jsonpath) j LOOP\n RAISE NOTICE 'JPITEM: %', jpitem;\n op := jpitem->>'op';\n args := jpitem->'args';\n IF op IN ('=', 'eq', 'in') THEN\n FOR arg IN SELECT a FROM jsonb_array_elements(args) a LOOP\n IF jsonb_typeof(arg) IN ('string', 'array') THEN\n RAISE NOTICE 'arg: %, collections: %', arg, collections;\n IF collections IS NULL OR collections = '{}'::text[] THEN\n collections := to_text_array(arg);\n ELSE\n collections := array_intersection(collections, to_text_array(arg));\n END IF;\n END IF;\n END LOOP;\n END IF;\n END LOOP;\n END IF;\n IF collections = '{}'::text[] THEN\n RETURN NULL;\n END IF;\n RETURN collections;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\nCREATE TABLE items (\n id text NOT NULL,\n geometry geometry NOT NULL,\n collection text NOT NULL,\n datetime timestamptz NOT NULL,\n end_datetime timestamptz NOT NULL,\n content JSONB NOT NULL,\n private jsonb\n)\nPARTITION BY LIST (collection)\n;\n\nCREATE INDEX \"datetime_idx\" ON items USING BTREE (datetime DESC, end_datetime ASC);\nCREATE INDEX \"geometry_idx\" ON items USING GIST (geometry);\n\nCREATE STATISTICS datetime_stats (dependencies) on datetime, end_datetime from items;\n\nALTER TABLE items ADD CONSTRAINT items_collections_fk FOREIGN KEY (collection) REFERENCES collections(id) ON DELETE CASCADE DEFERRABLE;\n\nCREATE OR REPLACE FUNCTION partition_after_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p text;\n t timestamptz := clock_timestamp();\nBEGIN\n RAISE NOTICE 'Updating partition stats %', t;\n FOR p IN SELECT DISTINCT partition\n FROM newdata n JOIN partition_sys_meta p\n ON (n.collection=p.collection AND n.datetime <@ p.partition_dtrange)\n LOOP\n PERFORM run_or_queue(format('SELECT update_partition_stats(%L, %L);', p, true));\n END LOOP;\n IF TG_OP IN ('DELETE','UPDATE') THEN\n DELETE FROM format_item_cache c USING newdata n WHERE c.collection = n.collection AND c.id = n.id;\n END IF;\n RAISE NOTICE 't: % %', t, clock_timestamp() - t;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE TRIGGER items_after_insert_trigger\nAFTER INSERT ON items\nREFERENCING NEW TABLE AS newdata\nFOR EACH STATEMENT\nEXECUTE FUNCTION partition_after_triggerfunc();\n\nCREATE TRIGGER items_after_update_trigger\nAFTER DELETE ON items\nREFERENCING OLD TABLE AS newdata\nFOR EACH STATEMENT\nEXECUTE FUNCTION partition_after_triggerfunc();\n\nCREATE TRIGGER items_after_delete_trigger\nAFTER UPDATE ON items\nREFERENCING NEW TABLE AS newdata\nFOR EACH STATEMENT\nEXECUTE FUNCTION partition_after_triggerfunc();\n\n\nCREATE OR REPLACE FUNCTION content_slim(_item jsonb) RETURNS jsonb AS $$\n SELECT strip_jsonb(_item - '{id,geometry,collection,type}'::text[], collection_base_item(_item->>'collection')) - '{id,geometry,collection,type}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_dehydrate(content jsonb) RETURNS items AS $$\n SELECT\n content->>'id' as id,\n stac_geom(content) as geometry,\n content->>'collection' as collection,\n stac_datetime(content) as datetime,\n stac_end_datetime(content) as end_datetime,\n content_slim(content) as content,\n null::jsonb as private\n ;\n$$ LANGUAGE SQL STABLE;\n\nCREATE OR REPLACE FUNCTION include_field(f text, fields jsonb DEFAULT '{}'::jsonb) RETURNS boolean AS $$\nDECLARE\n includes jsonb := fields->'include';\n excludes jsonb := fields->'exclude';\nBEGIN\n IF f IS NULL THEN\n RETURN NULL;\n END IF;\n\n\n IF\n jsonb_typeof(excludes) = 'array'\n AND jsonb_array_length(excludes)>0\n AND excludes ? f\n THEN\n RETURN FALSE;\n END IF;\n\n IF\n (\n jsonb_typeof(includes) = 'array'\n AND jsonb_array_length(includes) > 0\n AND includes ? f\n ) OR\n (\n includes IS NULL\n OR jsonb_typeof(includes) = 'null'\n OR jsonb_array_length(includes) = 0\n )\n THEN\n RETURN TRUE;\n END IF;\n\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nDROP FUNCTION IF EXISTS content_hydrate(jsonb, jsonb, jsonb);\nCREATE OR REPLACE FUNCTION content_hydrate(\n _item jsonb,\n _base_item jsonb,\n fields jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\n SELECT merge_jsonb(\n jsonb_fields(_item, fields),\n jsonb_fields(_base_item, fields)\n );\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION content_hydrate(_item items, _collection collections, fields jsonb DEFAULT '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\n content jsonb;\n base_item jsonb := _collection.base_item;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry, 20)::jsonb;\n END IF;\n output := content_hydrate(\n jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'collection', _item.collection,\n 'type', 'Feature'\n ) || _item.content,\n _collection.base_item,\n fields\n );\n\n RETURN output;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_nonhydrated(\n _item items,\n fields jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry, 20)::jsonb;\n END IF;\n output := jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'collection', _item.collection,\n 'type', 'Feature'\n ) || _item.content;\n RETURN output;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_hydrate(_item items, fields jsonb DEFAULT '{}'::jsonb) RETURNS jsonb AS $$\n SELECT content_hydrate(\n _item,\n (SELECT c FROM collections c WHERE id=_item.collection LIMIT 1),\n fields\n );\n$$ LANGUAGE SQL STABLE;\n\n\nCREATE UNLOGGED TABLE items_staging (\n content JSONB NOT NULL\n);\nCREATE UNLOGGED TABLE items_staging_ignore (\n content JSONB NOT NULL\n);\nCREATE UNLOGGED TABLE items_staging_upsert (\n content JSONB NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION items_staging_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p record;\n _partitions text[];\n part text;\n ts timestamptz := clock_timestamp();\n nrows int;\nBEGIN\n RAISE NOTICE 'Creating Partitions. %', clock_timestamp() - ts;\n\n FOR part IN WITH t AS (\n SELECT\n n.content->>'collection' as collection,\n stac_daterange(n.content->'properties') as dtr,\n partition_trunc\n FROM newdata n JOIN collections ON (n.content->>'collection'=collections.id)\n ), p AS (\n SELECT\n collection,\n COALESCE(date_trunc(partition_trunc::text, lower(dtr)),'-infinity') as d,\n tstzrange(min(lower(dtr)),max(lower(dtr)),'[]') as dtrange,\n tstzrange(min(upper(dtr)),max(upper(dtr)),'[]') as edtrange\n FROM t\n GROUP BY 1,2\n ) SELECT check_partition(collection, dtrange, edtrange) FROM p LOOP\n RAISE NOTICE 'Partition %', part;\n END LOOP;\n\n RAISE NOTICE 'Creating temp table with data to be added. %', clock_timestamp() - ts;\n DROP TABLE IF EXISTS tmpdata;\n CREATE TEMP TABLE tmpdata ON COMMIT DROP AS\n SELECT\n (content_dehydrate(content)).*\n FROM newdata;\n GET DIAGNOSTICS nrows = ROW_COUNT;\n RAISE NOTICE 'Added % rows to tmpdata. %', nrows, clock_timestamp() - ts;\n\n RAISE NOTICE 'Doing the insert. %', clock_timestamp() - ts;\n IF TG_TABLE_NAME = 'items_staging' THEN\n INSERT INTO items\n SELECT * FROM tmpdata;\n GET DIAGNOSTICS nrows = ROW_COUNT;\n RAISE NOTICE 'Inserted % rows to items. %', nrows, clock_timestamp() - ts;\n ELSIF TG_TABLE_NAME = 'items_staging_ignore' THEN\n INSERT INTO items\n SELECT * FROM tmpdata\n ON CONFLICT DO NOTHING;\n GET DIAGNOSTICS nrows = ROW_COUNT;\n RAISE NOTICE 'Inserted % rows to items. %', nrows, clock_timestamp() - ts;\n ELSIF TG_TABLE_NAME = 'items_staging_upsert' THEN\n DELETE FROM items i USING tmpdata s\n WHERE\n i.id = s.id\n AND i.collection = s.collection\n AND i IS DISTINCT FROM s\n ;\n GET DIAGNOSTICS nrows = ROW_COUNT;\n RAISE NOTICE 'Deleted % rows from items. %', nrows, clock_timestamp() - ts;\n INSERT INTO items AS t\n SELECT * FROM tmpdata\n ON CONFLICT DO NOTHING;\n GET DIAGNOSTICS nrows = ROW_COUNT;\n RAISE NOTICE 'Inserted % rows to items. %', nrows, clock_timestamp() - ts;\n END IF;\n\n RAISE NOTICE 'Deleting data from staging table. %', clock_timestamp() - ts;\n DELETE FROM items_staging;\n RAISE NOTICE 'Done. %', clock_timestamp() - ts;\n\n RETURN NULL;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER items_staging_insert_trigger AFTER INSERT ON items_staging REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\nCREATE TRIGGER items_staging_insert_ignore_trigger AFTER INSERT ON items_staging_ignore REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\nCREATE TRIGGER items_staging_insert_upsert_trigger AFTER INSERT ON items_staging_upsert REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\n\nCREATE OR REPLACE FUNCTION item_by_id(_id text, _collection text DEFAULT NULL) RETURNS items AS\n$$\nDECLARE\n i items%ROWTYPE;\nBEGIN\n SELECT * INTO i FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection) LIMIT 1;\n RETURN i;\nEND;\n$$ LANGUAGE PLPGSQL STABLE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION get_item(_id text, _collection text DEFAULT NULL) RETURNS jsonb AS $$\n SELECT content_hydrate(items) FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection);\n$$ LANGUAGE SQL STABLE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION delete_item(_id text, _collection text DEFAULT NULL) RETURNS VOID AS $$\nDECLARE\nout items%ROWTYPE;\nBEGIN\n DELETE FROM items WHERE id = _id AND (_collection IS NULL OR collection=_collection) RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n--/*\nCREATE OR REPLACE FUNCTION create_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION update_item(content jsonb) RETURNS VOID AS $$\nDECLARE\n old items %ROWTYPE;\n out items%ROWTYPE;\nBEGIN\n PERFORM delete_item(content->>'id', content->>'collection');\n PERFORM create_item(content);\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION create_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION collection_bbox(id text) RETURNS jsonb AS $$\n SELECT (replace(replace(replace(st_extent(geometry)::text,'BOX(','[['),')',']]'),' ',','))::jsonb\n FROM items WHERE collection=$1;\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION collection_temporal_extent(id text) RETURNS jsonb AS $$\n SELECT to_jsonb(array[array[min(datetime)::text, max(datetime)::text]])\n FROM items WHERE collection=$1;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection_extents() RETURNS VOID AS $$\nUPDATE collections SET\n content = content ||\n jsonb_build_object(\n 'extent', jsonb_build_object(\n 'spatial', jsonb_build_object(\n 'bbox', collection_bbox(collections.id)\n ),\n 'temporal', jsonb_build_object(\n 'interval', collection_temporal_extent(collections.id)\n )\n )\n )\n;\n$$ LANGUAGE SQL;\nCREATE TABLE partition_stats (\n partition text PRIMARY KEY,\n dtrange tstzrange,\n edtrange tstzrange,\n spatial geometry,\n last_updated timestamptz,\n keys text[]\n) WITH (FILLFACTOR=90);\n\nCREATE INDEX partitions_range_idx ON partition_stats USING GIST(dtrange);\n\n\nCREATE OR REPLACE FUNCTION constraint_tstzrange(expr text) RETURNS tstzrange AS $$\n WITH t AS (\n SELECT regexp_matches(\n expr,\n E'\\\\(''\\([0-9 :+-]*\\)''\\\\).*\\\\(''\\([0-9 :+-]*\\)''\\\\)'\n ) AS m\n ) SELECT tstzrange(m[1]::timestamptz, m[2]::timestamptz) FROM t\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE STRICT;\n\nCREATE OR REPLACE FUNCTION dt_constraint(coid oid, OUT dt tstzrange, OUT edt tstzrange) RETURNS RECORD AS $$\nDECLARE\n expr text := pg_get_constraintdef(coid);\n matches timestamptz[];\nBEGIN\n IF expr LIKE '%NULL%' THEN\n dt := tstzrange(null::timestamptz, null::timestamptz);\n edt := tstzrange(null::timestamptz, null::timestamptz);\n RETURN;\n END IF;\n WITH f AS (SELECT (regexp_matches(expr, E'([0-9]{4}-[0-1][0-9]-[0-3][0-9] [0-2][0-9]:[0-5][0-9]:[0-5][0-9]\\.?[0-9]*)', 'g'))[1] f)\n SELECT array_agg(f::timestamptz) INTO matches FROM f;\n IF cardinality(matches) = 4 THEN\n dt := tstzrange(matches[1], matches[2],'[]');\n edt := tstzrange(matches[3], matches[4], '[]');\n RETURN;\n ELSIF cardinality(matches) = 2 THEN\n edt := tstzrange(matches[1], matches[2],'[]');\n RETURN;\n END IF;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\nCREATE OR REPLACE VIEW partition_sys_meta AS\nSELECT\n relid::text as partition,\n replace(replace(CASE WHEN level = 1 THEN pg_get_expr(c.relpartbound, c.oid)\n ELSE pg_get_expr(parent.relpartbound, parent.oid)\n END, 'FOR VALUES IN (''',''), ''')','') AS collection,\n level,\n c.reltuples,\n c.relhastriggers,\n COALESCE(constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as partition_dtrange,\n COALESCE((dt_constraint(edt.oid)).dt, constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as constraint_dtrange,\n COALESCE((dt_constraint(edt.oid)).edt, tstzrange('-infinity', 'infinity','[]')) as constraint_edtrange\nFROM\n pg_partition_tree('items')\n JOIN pg_class c ON (relid::regclass = c.oid)\n JOIN pg_class parent ON (parentrelid::regclass = parent.oid AND isleaf)\n LEFT JOIN pg_constraint edt ON (conrelid=c.oid AND contype='c')\nWHERE isleaf\n;\n\nCREATE VIEW partitions_view AS\nSELECT\n relid::text as partition,\n replace(replace(CASE WHEN level = 1 THEN pg_get_expr(c.relpartbound, c.oid)\n ELSE pg_get_expr(parent.relpartbound, parent.oid)\n END, 'FOR VALUES IN (''',''), ''')','') AS collection,\n level,\n c.reltuples,\n c.relhastriggers,\n COALESCE(constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as partition_dtrange,\n COALESCE((dt_constraint(edt.oid)).dt, constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as constraint_dtrange,\n COALESCE((dt_constraint(edt.oid)).edt, tstzrange('-infinity', 'infinity','[]')) as constraint_edtrange,\n dtrange,\n edtrange,\n spatial,\n last_updated\nFROM\n pg_partition_tree('items')\n JOIN pg_class c ON (relid::regclass = c.oid)\n JOIN pg_class parent ON (parentrelid::regclass = parent.oid AND isleaf)\n LEFT JOIN pg_constraint edt ON (conrelid=c.oid AND contype='c')\n LEFT JOIN partition_stats ON (relid::text=partition)\nWHERE isleaf\n;\n\nCREATE MATERIALIZED VIEW partitions AS\nSELECT * FROM partitions_view;\nCREATE UNIQUE INDEX ON partitions (partition);\n\nCREATE MATERIALIZED VIEW partition_steps AS\nSELECT\n partition as name,\n date_trunc('month',lower(partition_dtrange)) as sdate,\n date_trunc('month', upper(partition_dtrange)) + '1 month'::interval as edate\n FROM partitions_view WHERE partition_dtrange IS NOT NULL AND partition_dtrange != 'empty'::tstzrange\n ORDER BY dtrange ASC\n;\n\n\nCREATE OR REPLACE FUNCTION update_partition_stats_q(_partition text, istrigger boolean default false) RETURNS VOID AS $$\nDECLARE\nBEGIN\n PERFORM run_or_queue(\n format('SELECT update_partition_stats(%L, %L);', _partition, istrigger)\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION update_partition_stats(_partition text, istrigger boolean default false) RETURNS VOID AS $$\nDECLARE\n dtrange tstzrange;\n edtrange tstzrange;\n cdtrange tstzrange;\n cedtrange tstzrange;\n extent geometry;\n collection text;\nBEGIN\n RAISE NOTICE 'Updating stats for %.', _partition;\n EXECUTE format(\n $q$\n SELECT\n tstzrange(min(datetime), max(datetime),'[]'),\n tstzrange(min(end_datetime), max(end_datetime), '[]')\n FROM %I\n $q$,\n _partition\n ) INTO dtrange, edtrange;\n extent := st_estimatedextent('pgstac', _partition, 'geometry');\n INSERT INTO partition_stats (partition, dtrange, edtrange, spatial, last_updated)\n SELECT _partition, dtrange, edtrange, extent, now()\n ON CONFLICT (partition) DO\n UPDATE SET\n dtrange=EXCLUDED.dtrange,\n edtrange=EXCLUDED.edtrange,\n spatial=EXCLUDED.spatial,\n last_updated=EXCLUDED.last_updated\n ;\n\n SELECT\n constraint_dtrange, constraint_edtrange, pv.collection\n INTO cdtrange, cedtrange, collection\n FROM partitions_view pv WHERE partition = _partition;\n REFRESH MATERIALIZED VIEW partitions;\n REFRESH MATERIALIZED VIEW partition_steps;\n\n\n RAISE NOTICE 'Checking if we need to modify constraints...';\n RAISE NOTICE 'cdtrange: % dtrange: % cedtrange: % edtrange: %',cdtrange, dtrange, cedtrange, edtrange;\n IF\n (cdtrange IS DISTINCT FROM dtrange OR edtrange IS DISTINCT FROM cedtrange)\n AND NOT istrigger\n THEN\n RAISE NOTICE 'Modifying Constraints';\n RAISE NOTICE 'Existing % %', cdtrange, cedtrange;\n RAISE NOTICE 'New % %', dtrange, edtrange;\n PERFORM drop_table_constraints(_partition);\n PERFORM create_table_constraints(_partition, dtrange, edtrange);\n REFRESH MATERIALIZED VIEW partitions;\n REFRESH MATERIALIZED VIEW partition_steps;\n END IF;\n RAISE NOTICE 'Checking if we need to update collection extents.';\n IF get_setting_bool('update_collection_extent') THEN\n RAISE NOTICE 'updating collection extent for %', collection;\n PERFORM run_or_queue(format($q$\n UPDATE collections\n SET content = jsonb_set_lax(\n content,\n '{extent}'::text[],\n collection_extent(%L, FALSE),\n true,\n 'use_json_null'\n ) WHERE id=%L\n ;\n $q$, collection, collection));\n ELSE\n RAISE NOTICE 'Not updating collection extent for %', collection;\n END IF;\n\nEND;\n$$ LANGUAGE PLPGSQL STRICT SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION partition_name( IN collection text, IN dt timestamptz, OUT partition_name text, OUT partition_range tstzrange) AS $$\nDECLARE\n c RECORD;\n parent_name text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n parent_name := format('_items_%s', c.key);\n\n\n IF c.partition_trunc = 'year' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYY'));\n ELSIF c.partition_trunc = 'month' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYYMM'));\n ELSE\n partition_name := parent_name;\n partition_range := tstzrange('-infinity'::timestamptz, 'infinity'::timestamptz, '[]');\n END IF;\n IF partition_range IS NULL THEN\n partition_range := tstzrange(\n date_trunc(c.partition_trunc::text, dt),\n date_trunc(c.partition_trunc::text, dt) + concat('1 ', c.partition_trunc)::interval\n );\n END IF;\n RETURN;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION drop_table_constraints(t text) RETURNS text AS $$\nDECLARE\n q text;\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM partitions_view WHERE partition=t) THEN\n RETURN NULL;\n END IF;\n FOR q IN SELECT FORMAT(\n $q$\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n $q$,\n t,\n conname\n ) FROM pg_constraint\n WHERE conrelid=t::regclass::oid AND contype='c'\n LOOP\n EXECUTE q;\n END LOOP;\n RETURN t;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE OR REPLACE FUNCTION create_table_constraints(t text, _dtrange tstzrange, _edtrange tstzrange) RETURNS text AS $$\nDECLARE\n q text;\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM partitions_view WHERE partition=t) THEN\n RETURN NULL;\n END IF;\n RAISE NOTICE 'Creating Table Constraints for % % %', t, _dtrange, _edtrange;\n IF _dtrange = 'empty' AND _edtrange = 'empty' THEN\n q :=format(\n $q$\n DO $block$\n BEGIN\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n ALTER TABLE %I\n ADD CONSTRAINT %I\n CHECK (((datetime IS NULL) AND (end_datetime IS NULL))) NOT VALID\n ;\n ALTER TABLE %I\n VALIDATE CONSTRAINT %I\n ;\n\n\n\n EXCEPTION WHEN others THEN\n RAISE WARNING '%%, Issue Altering Constraints. Please run update_partition_stats(%I)', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n $block$;\n $q$,\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t),\n t\n );\n ELSE\n q :=format(\n $q$\n DO $block$\n BEGIN\n\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n ALTER TABLE %I\n ADD CONSTRAINT %I\n CHECK (\n (datetime >= %L)\n AND (datetime <= %L)\n AND (end_datetime >= %L)\n AND (end_datetime <= %L)\n ) NOT VALID\n ;\n ALTER TABLE %I\n VALIDATE CONSTRAINT %I\n ;\n\n\n\n EXCEPTION WHEN others THEN\n RAISE WARNING '%%, Issue Altering Constraints. Please run update_partition_stats(%I)', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n $block$;\n $q$,\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t),\n lower(_dtrange),\n upper(_dtrange),\n lower(_edtrange),\n upper(_edtrange),\n t,\n format('%s_dt', t),\n t\n );\n END IF;\n PERFORM run_or_queue(q);\n RETURN t;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION check_partition(\n _collection text,\n _dtrange tstzrange,\n _edtrange tstzrange\n) RETURNS text AS $$\nDECLARE\n c RECORD;\n pm RECORD;\n _partition_name text;\n _partition_dtrange tstzrange;\n _constraint_dtrange tstzrange;\n _constraint_edtrange tstzrange;\n q text;\n deferrable_q text;\n err_context text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=_collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', _collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n\n IF c.partition_trunc IS NOT NULL THEN\n _partition_dtrange := tstzrange(\n date_trunc(c.partition_trunc, lower(_dtrange)),\n date_trunc(c.partition_trunc, lower(_dtrange)) + (concat('1 ', c.partition_trunc))::interval,\n '[)'\n );\n ELSE\n _partition_dtrange := '[-infinity, infinity]'::tstzrange;\n END IF;\n\n IF NOT _partition_dtrange @> _dtrange THEN\n RAISE EXCEPTION 'dtrange % is greater than the partition size % for collection %', _dtrange, c.partition_trunc, _collection;\n END IF;\n\n\n IF c.partition_trunc = 'year' THEN\n _partition_name := format('_items_%s_%s', c.key, to_char(lower(_partition_dtrange),'YYYY'));\n ELSIF c.partition_trunc = 'month' THEN\n _partition_name := format('_items_%s_%s', c.key, to_char(lower(_partition_dtrange),'YYYYMM'));\n ELSE\n _partition_name := format('_items_%s', c.key);\n END IF;\n\n SELECT * INTO pm FROM partition_sys_meta WHERE collection=_collection AND partition_dtrange @> _dtrange;\n IF FOUND THEN\n RAISE NOTICE '% % %', _edtrange, _dtrange, pm;\n _constraint_edtrange :=\n tstzrange(\n least(\n lower(_edtrange),\n nullif(lower(pm.constraint_edtrange), '-infinity')\n ),\n greatest(\n upper(_edtrange),\n nullif(upper(pm.constraint_edtrange), 'infinity')\n ),\n '[]'\n );\n _constraint_dtrange :=\n tstzrange(\n least(\n lower(_dtrange),\n nullif(lower(pm.constraint_dtrange), '-infinity')\n ),\n greatest(\n upper(_dtrange),\n nullif(upper(pm.constraint_dtrange), 'infinity')\n ),\n '[]'\n );\n\n IF pm.constraint_edtrange @> _edtrange AND pm.constraint_dtrange @> _dtrange THEN\n RETURN pm.partition;\n ELSE\n PERFORM drop_table_constraints(_partition_name);\n END IF;\n ELSE\n _constraint_edtrange := _edtrange;\n _constraint_dtrange := _dtrange;\n END IF;\n RAISE NOTICE 'EXISTING CONSTRAINTS % %, NEW % %', pm.constraint_dtrange, pm.constraint_edtrange, _constraint_dtrange, _constraint_edtrange;\n RAISE NOTICE 'Creating partition % %', _partition_name, _partition_dtrange;\n IF c.partition_trunc IS NULL THEN\n q := format(\n $q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n GRANT ALL ON %I to pgstac_ingest;\n $q$,\n _partition_name,\n _collection,\n concat(_partition_name,'_pk'),\n _partition_name,\n _partition_name\n );\n ELSE\n q := format(\n $q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L) PARTITION BY RANGE (datetime);\n CREATE TABLE IF NOT EXISTS %I partition OF %I FOR VALUES FROM (%L) TO (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n GRANT ALL ON %I TO pgstac_ingest;\n $q$,\n format('_items_%s', c.key),\n _collection,\n _partition_name,\n format('_items_%s', c.key),\n lower(_partition_dtrange),\n upper(_partition_dtrange),\n format('%s_pk', _partition_name),\n _partition_name,\n _partition_name\n );\n END IF;\n\n BEGIN\n EXECUTE q;\n EXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', _partition_name;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\n END;\n PERFORM maintain_partitions(_partition_name);\n PERFORM update_partition_stats_q(_partition_name, true);\n REFRESH MATERIALIZED VIEW partitions;\n REFRESH MATERIALIZED VIEW partition_steps;\n RETURN _partition_name;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION repartition(_collection text, _partition_trunc text, triggered boolean DEFAULT FALSE) RETURNS text AS $$\nDECLARE\n c RECORD;\n q text;\n from_trunc text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=_collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', _collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n IF triggered THEN\n RAISE NOTICE 'Converting % to % partitioning via Trigger', _collection, _partition_trunc;\n ELSE\n RAISE NOTICE 'Converting % from using % to % partitioning', _collection, c.partition_trunc, _partition_trunc;\n IF c.partition_trunc IS NOT DISTINCT FROM _partition_trunc THEN\n RAISE NOTICE 'Collection % already set to use partition by %', _collection, _partition_trunc;\n RETURN _collection;\n END IF;\n END IF;\n\n IF EXISTS (SELECT 1 FROM partitions_view WHERE collection=_collection LIMIT 1) THEN\n EXECUTE format(\n $q$\n CREATE TEMP TABLE changepartitionstaging ON COMMIT DROP AS SELECT * FROM %I;\n DROP TABLE IF EXISTS %I CASCADE;\n WITH p AS (\n SELECT\n collection,\n CASE\n WHEN %L IS NULL THEN '-infinity'::timestamptz\n ELSE date_trunc(%L, datetime)\n END as d,\n tstzrange(min(datetime),max(datetime),'[]') as dtrange,\n tstzrange(min(end_datetime),max(end_datetime),'[]') as edtrange\n FROM changepartitionstaging\n GROUP BY 1,2\n ) SELECT check_partition(collection, dtrange, edtrange) FROM p;\n INSERT INTO items SELECT * FROM changepartitionstaging;\n DROP TABLE changepartitionstaging;\n $q$,\n concat('_items_', c.key),\n concat('_items_', c.key),\n c.partition_trunc,\n c.partition_trunc\n );\n END IF;\n RETURN _collection;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE OR REPLACE FUNCTION collections_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n q text;\n partition_name text := format('_items_%s', NEW.key);\n partition_exists boolean := false;\n partition_empty boolean := true;\n err_context text;\n loadtemp boolean := FALSE;\nBEGIN\n RAISE NOTICE 'Collection Trigger. % %', NEW.id, NEW.key;\n IF TG_OP = 'UPDATE' AND NEW.partition_trunc IS DISTINCT FROM OLD.partition_trunc THEN\n PERFORM repartition(NEW.id, NEW.partition_trunc, TRUE);\n END IF;\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER collections_trigger AFTER\nINSERT\nOR\nUPDATE ON collections\nFOR EACH ROW EXECUTE FUNCTION collections_trigger_func();\n\nCREATE OR REPLACE FUNCTION chunker(\n IN _where text,\n OUT s timestamptz,\n OUT e timestamptz\n) RETURNS SETOF RECORD AS $$\nDECLARE\n explain jsonb;\nBEGIN\n IF _where IS NULL THEN\n _where := ' TRUE ';\n END IF;\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s;', _where)\n INTO explain;\n\n RETURN QUERY\n WITH t AS (\n SELECT j->>0 as p FROM\n jsonb_path_query(\n explain,\n 'strict $.**.\"Relation Name\" ? (@ != null)'\n ) j\n ),\n parts AS (\n SELECT sdate, edate FROM t JOIN partition_steps ON (t.p = name)\n ),\n times AS (\n SELECT sdate FROM parts\n UNION\n SELECT edate FROM parts\n ),\n uniq AS (\n SELECT DISTINCT sdate FROM times ORDER BY sdate\n ),\n last AS (\n SELECT sdate, lead(sdate, 1) over () as edate FROM uniq\n )\n SELECT sdate, edate FROM last WHERE edate IS NOT NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION partition_queries(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN partitions text[] DEFAULT NULL\n) RETURNS SETOF text AS $$\nDECLARE\n query text;\n sdate timestamptz;\n edate timestamptz;\nBEGIN\nIF _where IS NULL OR trim(_where) = '' THEN\n _where = ' TRUE ';\nEND IF;\nRAISE NOTICE 'Getting chunks for % %', _where, _orderby;\nIF _orderby ILIKE 'datetime d%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 DESC LOOP\n RETURN NEXT format($q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n $q$,\n sdate,\n edate,\n _where,\n _orderby\n );\n END LOOP;\nELSIF _orderby ILIKE 'datetime a%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 ASC LOOP\n RETURN NEXT format($q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n $q$,\n sdate,\n edate,\n _where,\n _orderby\n );\n END LOOP;\nELSE\n query := format($q$\n SELECT * FROM items\n WHERE %s\n ORDER BY %s\n $q$, _where, _orderby\n );\n\n RETURN NEXT query;\n RETURN;\nEND IF;\n\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION partition_query_view(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN _limit int DEFAULT 10\n) RETURNS text AS $$\n WITH p AS (\n SELECT * FROM partition_queries(_where, _orderby) p\n )\n SELECT\n CASE WHEN EXISTS (SELECT 1 FROM p) THEN\n (SELECT format($q$\n SELECT * FROM (\n %s\n ) total LIMIT %s\n $q$,\n string_agg(\n format($q$ SELECT * FROM ( %s ) AS sub $q$, p),\n '\n UNION ALL\n '\n ),\n _limit\n ))\n ELSE NULL\n END FROM p;\n$$ LANGUAGE SQL IMMUTABLE;\n\n\n\n\nCREATE OR REPLACE FUNCTION stac_search_to_where(j jsonb) RETURNS text AS $$\nDECLARE\n where_segments text[];\n _where text;\n dtrange tstzrange;\n collections text[];\n geom geometry;\n sdate timestamptz;\n edate timestamptz;\n filterlang text;\n filter jsonb := j->'filter';\nBEGIN\n IF j ? 'ids' THEN\n where_segments := where_segments || format('id = ANY (%L) ', to_text_array(j->'ids'));\n END IF;\n\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n where_segments := where_segments || format('collection = ANY (%L) ', collections);\n END IF;\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n\n where_segments := where_segments || format(' datetime <= %L::timestamptz AND end_datetime >= %L::timestamptz ',\n edate,\n sdate\n );\n END IF;\n\n geom := stac_geom(j);\n IF geom IS NOT NULL THEN\n where_segments := where_segments || format('st_intersects(geometry, %L)',geom);\n END IF;\n\n filterlang := COALESCE(\n j->>'filter-lang',\n get_setting('default_filter_lang', j->'conf')\n );\n IF NOT filter @? '$.**.op' THEN\n filterlang := 'cql-json';\n END IF;\n\n IF filterlang NOT IN ('cql-json','cql2-json') AND j ? 'filter' THEN\n RAISE EXCEPTION '% is not a supported filter-lang. Please use cql-json or cql2-json.', filterlang;\n END IF;\n\n IF j ? 'query' AND j ? 'filter' THEN\n RAISE EXCEPTION 'Can only use either query or filter at one time.';\n END IF;\n\n IF j ? 'query' THEN\n filter := query_to_cql2(j->'query');\n ELSIF filterlang = 'cql-json' THEN\n filter := cql1_to_cql2(filter);\n END IF;\n RAISE NOTICE 'FILTER: %', filter;\n where_segments := where_segments || cql2_query(filter);\n IF cardinality(where_segments) < 1 THEN\n RETURN ' TRUE ';\n END IF;\n\n _where := array_to_string(array_remove(where_segments, NULL), ' AND ');\n\n IF _where IS NULL OR BTRIM(_where) = '' THEN\n RETURN ' TRUE ';\n END IF;\n RETURN _where;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION parse_sort_dir(_dir text, reverse boolean default false) RETURNS text AS $$\n WITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n ) SELECT\n CASE\n WHEN NOT reverse THEN d\n WHEN d = 'ASC' THEN 'DESC'\n WHEN d = 'DESC' THEN 'ASC'\n END\n FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION sort_dir_to_op(_dir text, prev boolean default false) RETURNS text AS $$\n WITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n ) SELECT\n CASE\n WHEN d = 'ASC' AND prev THEN '<='\n WHEN d = 'DESC' AND prev THEN '>='\n WHEN d = 'ASC' THEN '>='\n WHEN d = 'DESC' THEN '<='\n END\n FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION sort_sqlorderby(\n _search jsonb DEFAULT NULL,\n reverse boolean DEFAULT FALSE\n) RETURNS text AS $$\n WITH sortby AS (\n SELECT coalesce(_search->'sortby','[{\"field\":\"datetime\", \"direction\":\"desc\"}]') as sort\n ), withid AS (\n SELECT CASE\n WHEN sort @? '$[*] ? (@.field == \"id\")' THEN sort\n ELSE sort || '[{\"field\":\"id\", \"direction\":\"desc\"}]'::jsonb\n END as sort\n FROM sortby\n ), withid_rows AS (\n SELECT jsonb_array_elements(sort) as value FROM withid\n ),sorts AS (\n SELECT\n coalesce(\n (queryable(value->>'field')).expression\n ) as key,\n parse_sort_dir(value->>'direction', reverse) as dir\n FROM withid_rows\n )\n SELECT array_to_string(\n array_agg(concat(key, ' ', dir)),\n ', '\n ) FROM sorts;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_sort_dir(sort_item jsonb) RETURNS text AS $$\n SELECT CASE WHEN sort_item->>'direction' ILIKE 'desc%' THEN 'DESC' ELSE 'ASC' END;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION get_token_val_str(\n _field text,\n _item items\n) RETURNS text AS $$\nDECLARE\n q text;\n literal text;\nBEGIN\n q := format($q$ SELECT quote_literal(%s) FROM (SELECT $1.*) as r;$q$, _field);\n EXECUTE q INTO literal USING _item;\n RETURN literal;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n\nCREATE OR REPLACE FUNCTION get_token_record(IN _token text, OUT prev BOOLEAN, OUT item items) RETURNS RECORD AS $$\nDECLARE\n _itemid text := _token;\n _collectionid text;\nBEGIN\n IF _token IS NULL THEN\n RETURN;\n END IF;\n RAISE NOTICE 'Looking for token: %', _token;\n prev := FALSE;\n IF _token ILIKE 'prev:%' THEN\n _itemid := replace(_token, 'prev:','');\n prev := TRUE;\n ELSIF _token ILIKE 'next:%' THEN\n _itemid := replace(_token, 'next:', '');\n END IF;\n SELECT id INTO _collectionid FROM collections WHERE _itemid LIKE concat(id,':%');\n IF FOUND THEN\n _itemid := replace(_itemid, concat(_collectionid,':'), '');\n SELECT * INTO item FROM items WHERE id=_itemid AND collection=_collectionid;\n ELSE\n SELECT * INTO item FROM items WHERE id=_itemid;\n END IF;\n IF item IS NULL THEN\n RAISE EXCEPTION 'Could not find item using token: % item: % collection: %', _token, _itemid, _collectionid;\n END IF;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\n\nCREATE OR REPLACE FUNCTION get_token_filter(\n _sortby jsonb DEFAULT '[{\"field\":\"datetime\",\"direction\":\"desc\"}]'::jsonb,\n token_item items DEFAULT NULL,\n prev boolean DEFAULT FALSE,\n inclusive boolean DEFAULT FALSE\n) RETURNS text AS $$\nDECLARE\n ltop text := '<';\n gtop text := '>';\n dir text;\n sort record;\n orfilter text := '';\n orfilters text[] := '{}'::text[];\n andfilters text[] := '{}'::text[];\n output text;\n token_where text;\nBEGIN\n IF _sortby IS NULL OR _sortby = '[]'::jsonb THEN\n _sortby := '[{\"field\":\"datetime\",\"direction\":\"desc\"}]'::jsonb;\n END IF;\n _sortby := _sortby || jsonb_build_object('field','id','direction',_sortby->0->>'direction');\n RAISE NOTICE 'Getting Token Filter. % %', _sortby, token_item;\n IF inclusive THEN\n orfilters := orfilters || format('( id=%L AND collection=%L )' , token_item.id, token_item.collection);\n END IF;\n\n FOR sort IN\n WITH s1 AS (\n SELECT\n _row,\n (queryable(value->>'field')).expression as _field,\n (value->>'field' = 'id') as _isid,\n get_sort_dir(value) as _dir\n FROM jsonb_array_elements(_sortby)\n WITH ORDINALITY AS t(value, _row)\n )\n SELECT\n _row,\n _field,\n _dir,\n get_token_val_str(_field, token_item) as _val\n FROM s1\n WHERE _row <= (SELECT min(_row) FROM s1 WHERE _isid)\n LOOP\n orfilter := NULL;\n RAISE NOTICE 'SORT: %', sort;\n IF sort._val IS NOT NULL AND ((prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC')) THEN\n orfilter := format($f$(\n (%s %s %s) OR (%s IS NULL)\n )$f$,\n sort._field,\n ltop,\n sort._val,\n sort._val\n );\n ELSIF sort._val IS NULL AND ((prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC')) THEN\n RAISE NOTICE '< but null';\n orfilter := format('%s IS NOT NULL', sort._field);\n ELSIF sort._val IS NULL THEN\n RAISE NOTICE '> but null';\n ELSE\n orfilter := format($f$(\n (%s %s %s) OR (%s IS NULL)\n )$f$,\n sort._field,\n gtop,\n sort._val,\n sort._field\n );\n END IF;\n RAISE NOTICE 'ORFILTER: %', orfilter;\n\n IF orfilter IS NOT NULL THEN\n IF sort._row = 1 THEN\n orfilters := orfilters || orfilter;\n ELSE\n orfilters := orfilters || format('(%s AND %s)', array_to_string(andfilters, ' AND '), orfilter);\n END IF;\n END IF;\n IF sort._val IS NOT NULL THEN\n andfilters := andfilters || format('%s = %s', sort._field, sort._val);\n ELSE\n andfilters := andfilters || format('%s IS NULL', sort._field);\n END IF;\n END LOOP;\n\n output := array_to_string(orfilters, ' OR ');\n\n token_where := concat('(',coalesce(output,'true'),')');\n IF trim(token_where) = '' THEN\n token_where := NULL;\n END IF;\n RAISE NOTICE 'TOKEN_WHERE: %',token_where;\n RETURN token_where;\n END;\n$$ LANGUAGE PLPGSQL SET transform_null_equals TO TRUE\n;\n\nCREATE OR REPLACE FUNCTION search_tohash(jsonb) RETURNS jsonb AS $$\n SELECT $1 - '{token,limit,context,includes,excludes}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION search_hash(jsonb, jsonb) RETURNS text AS $$\n SELECT md5(concat(search_tohash($1)::text,$2::text));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TABLE IF NOT EXISTS searches(\n hash text GENERATED ALWAYS AS (search_hash(search, metadata)) STORED PRIMARY KEY,\n search jsonb NOT NULL,\n _where text,\n orderby text,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n metadata jsonb DEFAULT '{}'::jsonb NOT NULL\n);\n\nCREATE TABLE IF NOT EXISTS search_wheres(\n id bigint generated always as identity primary key,\n _where text NOT NULL,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n statslastupdated timestamptz,\n estimated_count bigint,\n estimated_cost float,\n time_to_estimate float,\n total_count bigint,\n time_to_count float,\n partitions text[]\n);\n\nCREATE INDEX IF NOT EXISTS search_wheres_partitions ON search_wheres USING GIN (partitions);\nCREATE UNIQUE INDEX IF NOT EXISTS search_wheres_where ON search_wheres ((md5(_where)));\n\nCREATE OR REPLACE FUNCTION where_stats(inwhere text, updatestats boolean default false, conf jsonb default null) RETURNS search_wheres AS $$\nDECLARE\n t timestamptz;\n i interval;\n explain_json jsonb;\n partitions text[];\n sw search_wheres%ROWTYPE;\n inwhere_hash text := md5(inwhere);\n _context text := lower(context(conf));\n _stats_ttl interval := context_stats_ttl(conf);\n _estimated_cost float := context_estimated_cost(conf);\n _estimated_count int := context_estimated_count(conf);\n ro bool := pgstac.readonly(conf);\nBEGIN\n IF ro THEN\n updatestats := FALSE;\n END IF;\n\n IF _context = 'off' THEN\n sw._where := inwhere;\n return sw;\n END IF;\n\n SELECT * INTO sw FROM search_wheres WHERE md5(_where)=inwhere_hash FOR UPDATE;\n\n -- Update statistics if explicitly set, if statistics do not exist, or statistics ttl has expired\n IF NOT updatestats THEN\n RAISE NOTICE 'Checking if update is needed for: % .', inwhere;\n RAISE NOTICE 'Stats Last Updated: %', sw.statslastupdated;\n RAISE NOTICE 'TTL: %, Age: %', _stats_ttl, now() - sw.statslastupdated;\n RAISE NOTICE 'Context: %, Existing Total: %', _context, sw.total_count;\n IF\n (\n sw.statslastupdated IS NULL\n OR (now() - sw.statslastupdated) > _stats_ttl\n OR (context(conf) != 'off' AND sw.total_count IS NULL)\n ) AND NOT ro\n THEN\n updatestats := TRUE;\n END IF;\n END IF;\n\n sw._where := inwhere;\n sw.lastused := now();\n sw.usecount := coalesce(sw.usecount,0) + 1;\n\n IF NOT updatestats THEN\n UPDATE search_wheres SET\n lastused = sw.lastused,\n usecount = sw.usecount\n WHERE md5(_where) = inwhere_hash\n RETURNING * INTO sw\n ;\n RETURN sw;\n END IF;\n\n -- Use explain to get estimated count/cost and a list of the partitions that would be hit by the query\n t := clock_timestamp();\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s', inwhere)\n INTO explain_json;\n RAISE NOTICE 'Time for just the explain: %', clock_timestamp() - t;\n i := clock_timestamp() - t;\n\n sw.statslastupdated := now();\n sw.estimated_count := explain_json->0->'Plan'->'Plan Rows';\n sw.estimated_cost := explain_json->0->'Plan'->'Total Cost';\n sw.time_to_estimate := extract(epoch from i);\n\n RAISE NOTICE 'ESTIMATED_COUNT: % < %', sw.estimated_count, _estimated_count;\n RAISE NOTICE 'ESTIMATED_COST: % < %', sw.estimated_cost, _estimated_cost;\n\n -- Do a full count of rows if context is set to on or if auto is set and estimates are low enough\n IF\n _context = 'on'\n OR\n ( _context = 'auto' AND\n (\n sw.estimated_count < _estimated_count\n AND\n sw.estimated_cost < _estimated_cost\n )\n )\n THEN\n t := clock_timestamp();\n RAISE NOTICE 'Calculating actual count...';\n EXECUTE format(\n 'SELECT count(*) FROM items WHERE %s',\n inwhere\n ) INTO sw.total_count;\n i := clock_timestamp() - t;\n RAISE NOTICE 'Actual Count: % -- %', sw.total_count, i;\n sw.time_to_count := extract(epoch FROM i);\n ELSE\n sw.total_count := NULL;\n sw.time_to_count := NULL;\n END IF;\n\n IF NOT ro THEN\n INSERT INTO search_wheres\n (_where, lastused, usecount, statslastupdated, estimated_count, estimated_cost, time_to_estimate, partitions, total_count, time_to_count)\n SELECT sw._where, sw.lastused, sw.usecount, sw.statslastupdated, sw.estimated_count, sw.estimated_cost, sw.time_to_estimate, sw.partitions, sw.total_count, sw.time_to_count\n ON CONFLICT ((md5(_where)))\n DO UPDATE\n SET\n lastused = sw.lastused,\n usecount = sw.usecount,\n statslastupdated = sw.statslastupdated,\n estimated_count = sw.estimated_count,\n estimated_cost = sw.estimated_cost,\n time_to_estimate = sw.time_to_estimate,\n total_count = sw.total_count,\n time_to_count = sw.time_to_count\n ;\n END IF;\n RETURN sw;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION search_query(\n _search jsonb = '{}'::jsonb,\n updatestats boolean = false,\n _metadata jsonb = '{}'::jsonb\n) RETURNS searches AS $$\nDECLARE\n search searches%ROWTYPE;\n pexplain jsonb;\n t timestamptz;\n i interval;\n _hash text := search_hash(_search, _metadata);\n doupdate boolean := FALSE;\n insertfound boolean := FALSE;\n ro boolean := pgstac.readonly();\nBEGIN\n IF ro THEN\n updatestats := FALSE;\n END IF;\n\n SELECT * INTO search FROM searches\n WHERE hash=_hash;\n\n search.hash := _hash;\n\n -- Calculate the where clause if not already calculated\n IF search._where IS NULL THEN\n search._where := stac_search_to_where(_search);\n ELSE\n doupdate := TRUE;\n END IF;\n\n -- Calculate the order by clause if not already calculated\n IF search.orderby IS NULL THEN\n search.orderby := sort_sqlorderby(_search);\n ELSE\n doupdate := TRUE;\n END IF;\n\n PERFORM where_stats(search._where, updatestats, _search->'conf');\n\n IF NOT ro THEN\n IF NOT doupdate THEN\n INSERT INTO searches (search, _where, orderby, lastused, usecount, metadata)\n VALUES (_search, search._where, search.orderby, clock_timestamp(), 1, _metadata)\n ON CONFLICT (hash) DO NOTHING RETURNING * INTO search;\n IF FOUND THEN\n RETURN search;\n END IF;\n END IF;\n\n UPDATE searches\n SET\n lastused=clock_timestamp(),\n usecount=usecount+1\n WHERE hash=(\n SELECT hash FROM searches\n WHERE hash=_hash\n FOR UPDATE SKIP LOCKED\n );\n IF NOT FOUND THEN\n RAISE NOTICE 'Did not update stats for % due to lock. (This is generally OK)', _search;\n END IF;\n END IF;\n\n RETURN search;\n\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE OR REPLACE FUNCTION search_rows(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN partitions text[] DEFAULT NULL,\n IN _limit int DEFAULT 10\n) RETURNS SETOF items AS $$\nDECLARE\n base_query text;\n query text;\n sdate timestamptz;\n edate timestamptz;\n n int;\n records_left int := _limit;\n timer timestamptz := clock_timestamp();\n full_timer timestamptz := clock_timestamp();\nBEGIN\nIF _where IS NULL OR trim(_where) = '' THEN\n _where = ' TRUE ';\nEND IF;\nRAISE NOTICE 'Getting chunks for % %', _where, _orderby;\n\nbase_query := $q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n LIMIT %L\n$q$;\n\nIF _orderby ILIKE 'datetime d%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 DESC LOOP\n RAISE NOTICE 'Running Query for % to %. %', sdate, edate, age_ms(full_timer);\n query := format(\n base_query,\n sdate,\n edate,\n _where,\n _orderby,\n records_left\n );\n RAISE LOG 'QUERY: %', query;\n timer := clock_timestamp();\n RETURN QUERY EXECUTE query;\n\n GET DIAGNOSTICS n = ROW_COUNT;\n records_left := records_left - n;\n RAISE NOTICE 'Returned %/% Rows From % to %. % to go. Time: %ms', n, _limit, sdate, edate, records_left, age_ms(timer);\n timer := clock_timestamp();\n IF records_left <= 0 THEN\n RAISE NOTICE 'SEARCH_ROWS TOOK %ms', age_ms(full_timer);\n RETURN;\n END IF;\n END LOOP;\nELSIF _orderby ILIKE 'datetime a%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 ASC LOOP\n RAISE NOTICE 'Running Query for % to %. %', sdate, edate, age_ms(full_timer);\n query := format(\n base_query,\n sdate,\n edate,\n _where,\n _orderby,\n records_left\n );\n RAISE LOG 'QUERY: %', query;\n timer := clock_timestamp();\n RETURN QUERY EXECUTE query;\n\n GET DIAGNOSTICS n = ROW_COUNT;\n records_left := records_left - n;\n RAISE NOTICE 'Returned %/% Rows From % to %. % to go. Time: %ms', n, _limit, sdate, edate, records_left, age_ms(timer);\n timer := clock_timestamp();\n IF records_left <= 0 THEN\n RAISE NOTICE 'SEARCH_ROWS TOOK %ms', age_ms(full_timer);\n RETURN;\n END IF;\n END LOOP;\nELSE\n query := format($q$\n SELECT * FROM items\n WHERE %s\n ORDER BY %s\n LIMIT %L\n $q$, _where, _orderby, _limit\n );\n RAISE LOG 'QUERY: %', query;\n timer := clock_timestamp();\n RETURN QUERY EXECUTE query;\n RAISE NOTICE 'FULL QUERY TOOK %ms', age_ms(timer);\nEND IF;\nRAISE NOTICE 'SEARCH_ROWS TOOK %ms', age_ms(full_timer);\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE UNLOGGED TABLE format_item_cache(\n id text,\n collection text,\n fields text,\n hydrated bool,\n output jsonb,\n lastused timestamptz DEFAULT now(),\n usecount int DEFAULT 1,\n timetoformat float,\n PRIMARY KEY (collection, id, fields, hydrated)\n);\nCREATE INDEX ON format_item_cache (lastused);\n\nCREATE OR REPLACE FUNCTION format_item(_item items, _fields jsonb DEFAULT '{}', _hydrated bool DEFAULT TRUE) RETURNS jsonb AS $$\nDECLARE\n cache bool := get_setting_bool('format_cache');\n _output jsonb := null;\n t timestamptz := clock_timestamp();\nBEGIN\n IF cache THEN\n SELECT output INTO _output FROM format_item_cache\n WHERE id=_item.id AND collection=_item.collection AND fields=_fields::text AND hydrated=_hydrated;\n END IF;\n IF _output IS NULL THEN\n IF _hydrated THEN\n _output := content_hydrate(_item, _fields);\n ELSE\n _output := content_nonhydrated(_item, _fields);\n END IF;\n END IF;\n IF cache THEN\n INSERT INTO format_item_cache (id, collection, fields, hydrated, output, timetoformat)\n VALUES (_item.id, _item.collection, _fields::text, _hydrated, _output, age_ms(t))\n ON CONFLICT(collection, id, fields, hydrated) DO\n UPDATE\n SET lastused=now(), usecount = format_item_cache.usecount + 1\n ;\n END IF;\n RETURN _output;\n\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION search(_search jsonb = '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n searches searches%ROWTYPE;\n _where text;\n orderby text;\n search_where search_wheres%ROWTYPE;\n total_count bigint;\n token record;\n token_prev boolean;\n token_item items%ROWTYPE;\n token_where text;\n full_where text;\n init_ts timestamptz := clock_timestamp();\n timer timestamptz := clock_timestamp();\n hydrate bool := NOT (_search->'conf'->>'nohydrate' IS NOT NULL AND (_search->'conf'->>'nohydrate')::boolean = true);\n prev text;\n next text;\n context jsonb;\n collection jsonb;\n out_records jsonb;\n out_len int;\n _limit int := coalesce((_search->>'limit')::int, 10);\n _querylimit int;\n _fields jsonb := coalesce(_search->'fields', '{}'::jsonb);\n has_prev boolean := FALSE;\n has_next boolean := FALSE;\nBEGIN\n searches := search_query(_search);\n _where := searches._where;\n orderby := searches.orderby;\n search_where := where_stats(_where);\n total_count := coalesce(search_where.total_count, search_where.estimated_count);\n RAISE NOTICE 'SEARCH:TOKEN: %', _search->>'token';\n token := get_token_record(_search->>'token');\n RAISE NOTICE '***TOKEN: %', token;\n _querylimit := _limit + 1;\n IF token IS NOT NULL THEN\n token_prev := token.prev;\n token_item := token.item;\n token_where := get_token_filter(_search->'sortby', token_item, token_prev, FALSE);\n RAISE LOG 'TOKEN_WHERE: % (%ms from search start)', token_where, age_ms(timer);\n IF token_prev THEN -- if we are using a prev token, we know has_next is true\n RAISE LOG 'There is a previous token, so automatically setting has_next to true';\n has_next := TRUE;\n orderby := sort_sqlorderby(_search, TRUE);\n ELSE\n RAISE LOG 'There is a next token, so automatically setting has_prev to true';\n has_prev := TRUE;\n\n END IF;\n ELSE -- if there was no token, we know there is no prev\n RAISE LOG 'There is no token, so we know there is no prev. setting has_prev to false';\n has_prev := FALSE;\n END IF;\n\n full_where := concat_ws(' AND ', _where, token_where);\n RAISE NOTICE 'FULL WHERE CLAUSE: %', full_where;\n RAISE NOTICE 'Time to get counts and build query %', age_ms(timer);\n timer := clock_timestamp();\n\n IF hydrate THEN\n RAISE NOTICE 'Getting hydrated data.';\n ELSE\n RAISE NOTICE 'Getting non-hydrated data.';\n END IF;\n RAISE NOTICE 'CACHE SET TO %', get_setting_bool('format_cache');\n RAISE NOTICE 'Time to set hydration/formatting %', age_ms(timer);\n timer := clock_timestamp();\n SELECT jsonb_agg(format_item(i, _fields, hydrate)) INTO out_records\n FROM search_rows(\n full_where,\n orderby,\n search_where.partitions,\n _querylimit\n ) as i;\n\n RAISE NOTICE 'Time to fetch rows %', age_ms(timer);\n timer := clock_timestamp();\n\n\n IF token_prev THEN\n out_records := flip_jsonb_array(out_records);\n END IF;\n\n RAISE NOTICE 'Query returned % records.', jsonb_array_length(out_records);\n RAISE LOG 'TOKEN: % %', token_item.id, token_item.collection;\n RAISE LOG 'RECORD_1: % %', out_records->0->>'id', out_records->0->>'collection';\n RAISE LOG 'RECORD-1: % %', out_records->-1->>'id', out_records->-1->>'collection';\n\n -- REMOVE records that were from our token\n IF out_records->0->>'id' = token_item.id AND out_records->0->>'collection' = token_item.collection THEN\n out_records := out_records - 0;\n ELSIF out_records->-1->>'id' = token_item.id AND out_records->-1->>'collection' = token_item.collection THEN\n out_records := out_records - -1;\n END IF;\n\n out_len := jsonb_array_length(out_records);\n\n IF out_len = _limit + 1 THEN\n IF token_prev THEN\n has_prev := TRUE;\n out_records := out_records - 0;\n ELSE\n has_next := TRUE;\n out_records := out_records - -1;\n END IF;\n END IF;\n\n IF has_next THEN\n next := concat(out_records->-1->>'collection', ':', out_records->-1->>'id');\n RAISE NOTICE 'HAS NEXT | %', next;\n END IF;\n\n IF has_prev THEN\n prev := concat(out_records->0->>'collection', ':', out_records->0->>'id');\n RAISE NOTICE 'HAS PREV | %', prev;\n END IF;\n\n RAISE NOTICE 'Time to get prev/next %', age_ms(timer);\n timer := clock_timestamp();\n\n IF context(_search->'conf') != 'off' THEN\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'matched', total_count,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\n ELSE\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\n END IF;\n\n collection := jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb),\n 'next', next,\n 'prev', prev,\n 'context', context\n );\n\n IF get_setting_bool('timing', _search->'conf') THEN\n collection = collection || jsonb_build_object('timing', age_ms(init_ts));\n END IF;\n\n RAISE NOTICE 'Time to build final json %', age_ms(timer);\n timer := clock_timestamp();\n\n RAISE NOTICE 'Total Time: %', age_ms(current_timestamp);\n RAISE NOTICE 'RETURNING % records. NEXT: %. PREV: %', collection->'context'->>'returned', collection->>'next', collection->>'prev';\n RETURN collection;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION search_cursor(_search jsonb = '{}'::jsonb) RETURNS refcursor AS $$\nDECLARE\n curs refcursor;\n searches searches%ROWTYPE;\n _where text;\n _orderby text;\n q text;\n\nBEGIN\n searches := search_query(_search);\n _where := searches._where;\n _orderby := searches.orderby;\n\n OPEN curs FOR\n WITH p AS (\n SELECT * FROM partition_queries(_where, _orderby) p\n )\n SELECT\n CASE WHEN EXISTS (SELECT 1 FROM p) THEN\n (SELECT format($q$\n SELECT * FROM (\n %s\n ) total\n $q$,\n string_agg(\n format($q$ SELECT * FROM ( %s ) AS sub $q$, p),\n '\n UNION ALL\n '\n )\n ))\n ELSE NULL\n END FROM p;\n RETURN curs;\nEND;\n$$ LANGUAGE PLPGSQL;\nCREATE OR REPLACE VIEW collections_asitems AS\nSELECT\n id,\n geometry,\n 'collections' AS collection,\n datetime,\n end_datetime,\n jsonb_build_object(\n 'properties', content - '{links,assets,stac_version,stac_extensions}',\n 'links', content->'links',\n 'assets', content->'assets',\n 'stac_version', content->'stac_version',\n 'stac_extensions', content->'stac_extensions'\n ) AS content,\n content as collectionjson\nFROM collections;\n\n\nCREATE OR REPLACE FUNCTION collection_search_matched(\n IN _search jsonb DEFAULT '{}'::jsonb,\n OUT matched bigint\n) RETURNS bigint AS $$\nDECLARE\n _where text := stac_search_to_where(_search);\nBEGIN\n EXECUTE format(\n $query$\n SELECT\n count(*)\n FROM\n collections_asitems\n WHERE %s\n ;\n $query$,\n _where\n ) INTO matched;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION collection_search_rows(\n _search jsonb DEFAULT '{}'::jsonb\n) RETURNS SETOF jsonb AS $$\nDECLARE\n _where text := stac_search_to_where(_search);\n _limit int := coalesce((_search->>'limit')::int, 10);\n _fields jsonb := coalesce(_search->'fields', '{}'::jsonb);\n _orderby text;\n _offset int := COALESCE((_search->>'offset')::int, 0);\nBEGIN\n _orderby := sort_sqlorderby(\n jsonb_build_object(\n 'sortby',\n coalesce(\n _search->'sortby',\n '[{\"field\": \"id\", \"direction\": \"asc\"}]'::jsonb\n )\n )\n );\n RETURN QUERY EXECUTE format(\n $query$\n SELECT\n jsonb_fields(collectionjson, %L) as c\n FROM\n collections_asitems\n WHERE %s\n ORDER BY %s\n LIMIT %L\n OFFSET %L\n ;\n $query$,\n _fields,\n _where,\n _orderby,\n _limit,\n _offset\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION collection_search(\n _search jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\nDECLARE\n out_records jsonb;\n number_matched bigint := collection_search_matched(_search);\n number_returned bigint;\n _limit int := coalesce((_search->>'limit')::float::int, 10);\n _offset int := coalesce((_search->>'offset')::float::int, 0);\n links jsonb := '[]';\n ret jsonb;\n base_url text:= concat(rtrim(base_url(_search->'conf'),'/'), '/collections');\n prevoffset int;\n nextoffset int;\nBEGIN\n SELECT\n coalesce(jsonb_agg(c), '[]')\n INTO out_records\n FROM collection_search_rows(_search) c;\n\n number_returned := jsonb_array_length(out_records);\n\n IF _limit <= number_matched THEN --need to have paging links\n nextoffset := least(_offset + _limit, number_matched - 1);\n prevoffset := greatest(_offset - _limit, 0);\n IF _offset = 0 THEN -- no previous paging\n\n links := jsonb_build_array(\n jsonb_build_object(\n 'rel', 'next',\n 'type', 'application/json',\n 'method', 'GET' ,\n 'href', base_url,\n 'body', jsonb_build_object('offset', nextoffset),\n 'merge', TRUE\n )\n );\n ELSE\n links := jsonb_build_array(\n jsonb_build_object(\n 'rel', 'prev',\n 'type', 'application/json',\n 'method', 'GET' ,\n 'href', base_url,\n 'body', jsonb_build_object('offset', prevoffset),\n 'merge', TRUE\n ),\n jsonb_build_object(\n 'rel', 'next',\n 'type', 'application/json',\n 'method', 'GET' ,\n 'href', base_url,\n 'body', jsonb_build_object('offset', nextoffset),\n 'merge', TRUE\n )\n );\n END IF;\n END IF;\n\n ret := jsonb_build_object(\n 'collections', out_records,\n 'context', jsonb_build_object(\n 'limit', _limit,\n 'matched', number_matched,\n 'returned', number_returned\n ),\n 'links', links\n );\n RETURN ret;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION tileenvelope(zoom int, x int, y int) RETURNS geometry AS $$\nWITH t AS (\n SELECT\n 20037508.3427892 as merc_max,\n -20037508.3427892 as merc_min,\n (2 * 20037508.3427892) / (2 ^ zoom) as tile_size\n)\nSELECT st_makeenvelope(\n merc_min + (tile_size * x),\n merc_max - (tile_size * (y + 1)),\n merc_min + (tile_size * (x + 1)),\n merc_max - (tile_size * y),\n 3857\n) FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;DROP FUNCTION IF EXISTS mercgrid;\n\n\nCREATE OR REPLACE FUNCTION ftime() RETURNS interval as $$\nSELECT age(clock_timestamp(), transaction_timestamp());\n$$ LANGUAGE SQL;\nSET SEARCH_PATH to pgstac, public;\n\nDROP FUNCTION IF EXISTS geometrysearch;\nCREATE OR REPLACE FUNCTION geometrysearch(\n IN geom geometry,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE, -- Return as soon as the passed in geometry is full covered\n IN skipcovered boolean DEFAULT TRUE -- Skip any items that would show up completely under the previous items\n) RETURNS jsonb AS $$\nDECLARE\n search searches%ROWTYPE;\n curs refcursor;\n _where text;\n query text;\n iter_record items%ROWTYPE;\n out_records jsonb := '{}'::jsonb[];\n exit_flag boolean := FALSE;\n counter int := 1;\n scancounter int := 1;\n remaining_limit int := _scanlimit;\n tilearea float;\n unionedgeom geometry;\n clippedgeom geometry;\n unionedgeom_area float := 0;\n prev_area float := 0;\n excludes text[];\n includes text[];\n\nBEGIN\n DROP TABLE IF EXISTS pgstac_results;\n CREATE TEMP TABLE pgstac_results (content jsonb) ON COMMIT DROP;\n\n -- If the passed in geometry is not an area set exitwhenfull and skipcovered to false\n IF ST_GeometryType(geom) !~* 'polygon' THEN\n RAISE NOTICE 'GEOMETRY IS NOT AN AREA';\n skipcovered = FALSE;\n exitwhenfull = FALSE;\n END IF;\n\n -- If skipcovered is true then you will always want to exit when the passed in geometry is full\n IF skipcovered THEN\n exitwhenfull := TRUE;\n END IF;\n\n SELECT * INTO search FROM searches WHERE hash=queryhash;\n\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Search with Query Hash % Not Found', queryhash;\n END IF;\n\n tilearea := st_area(geom);\n _where := format('%s AND st_intersects(geometry, %L::geometry)', search._where, geom);\n\n\n FOR query IN SELECT * FROM partition_queries(_where, search.orderby) LOOP\n query := format('%s LIMIT %L', query, remaining_limit);\n RAISE NOTICE '%', query;\n OPEN curs FOR EXECUTE query;\n LOOP\n FETCH curs INTO iter_record;\n EXIT WHEN NOT FOUND;\n IF exitwhenfull OR skipcovered THEN -- If we are not using exitwhenfull or skipcovered, we do not need to do expensive geometry operations\n clippedgeom := st_intersection(geom, iter_record.geometry);\n\n IF unionedgeom IS NULL THEN\n unionedgeom := clippedgeom;\n ELSE\n unionedgeom := st_union(unionedgeom, clippedgeom);\n END IF;\n\n unionedgeom_area := st_area(unionedgeom);\n\n IF skipcovered AND prev_area = unionedgeom_area THEN\n scancounter := scancounter + 1;\n CONTINUE;\n END IF;\n\n prev_area := unionedgeom_area;\n\n RAISE NOTICE '% % % %', unionedgeom_area/tilearea, counter, scancounter, ftime();\n END IF;\n RAISE NOTICE '% %', iter_record, content_hydrate(iter_record, fields);\n INSERT INTO pgstac_results (content) VALUES (content_hydrate(iter_record, fields));\n\n IF counter >= _limit\n OR scancounter > _scanlimit\n OR ftime() > _timelimit\n OR (exitwhenfull AND unionedgeom_area >= tilearea)\n THEN\n exit_flag := TRUE;\n EXIT;\n END IF;\n counter := counter + 1;\n scancounter := scancounter + 1;\n\n END LOOP;\n CLOSE curs;\n EXIT WHEN exit_flag;\n remaining_limit := _scanlimit - scancounter;\n END LOOP;\n\n SELECT jsonb_agg(content) INTO out_records FROM pgstac_results WHERE content IS NOT NULL;\n\n RETURN jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb)\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDROP FUNCTION IF EXISTS geojsonsearch;\nCREATE OR REPLACE FUNCTION geojsonsearch(\n IN geojson jsonb,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_geomfromgeojson(geojson),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS xyzsearch;\nCREATE OR REPLACE FUNCTION xyzsearch(\n IN _x int,\n IN _y int,\n IN _z int,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_transform(tileenvelope(_z, _x, _y), 4326),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE PROCEDURE analyze_items() AS $$\nDECLARE\n q text;\n timeout_ts timestamptz;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n SELECT format('ANALYZE (VERBOSE, SKIP_LOCKED) %I;', relname) INTO q\n FROM pg_stat_user_tables\n WHERE relname like '_item%' AND (n_mod_since_analyze>0 OR last_analyze IS NULL) LIMIT 1;\n IF NOT FOUND THEN\n EXIT;\n END IF;\n RAISE NOTICE '%', q;\n EXECUTE q;\n COMMIT;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE PROCEDURE validate_constraints() AS $$\nDECLARE\n q text;\nBEGIN\n FOR q IN\n SELECT\n FORMAT(\n 'ALTER TABLE %I.%I VALIDATE CONSTRAINT %I;',\n nsp.nspname,\n cls.relname,\n con.conname\n )\n\n FROM pg_constraint AS con\n JOIN pg_class AS cls\n ON con.conrelid = cls.oid\n JOIN pg_namespace AS nsp\n ON cls.relnamespace = nsp.oid\n WHERE convalidated = FALSE AND contype in ('c','f')\n AND nsp.nspname = 'pgstac'\n LOOP\n RAISE NOTICE '%', q;\n PERFORM run_or_queue(q);\n COMMIT;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION collection_extent(_collection text, runupdate boolean default false) RETURNS jsonb AS $$\nDECLARE\n geom_extent geometry;\n mind timestamptz;\n maxd timestamptz;\n extent jsonb;\nBEGIN\n IF runupdate THEN\n PERFORM update_partition_stats_q(partition)\n FROM partitions_view WHERE collection=_collection;\n END IF;\n SELECT\n min(lower(dtrange)),\n max(upper(edtrange)),\n st_extent(spatial)\n INTO\n mind,\n maxd,\n geom_extent\n FROM partitions_view\n WHERE collection=_collection;\n\n IF geom_extent IS NOT NULL AND mind IS NOT NULL AND maxd IS NOT NULL THEN\n extent := jsonb_build_object(\n 'extent', jsonb_build_object(\n 'spatial', jsonb_build_object(\n 'bbox', to_jsonb(array[array[st_xmin(geom_extent), st_ymin(geom_extent), st_xmax(geom_extent), st_ymax(geom_extent)]])\n ),\n 'temporal', jsonb_build_object(\n 'interval', to_jsonb(array[array[mind, maxd]])\n )\n )\n );\n RETURN extent;\n END IF;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('id', '{\"title\": \"Item ID\",\"description\": \"Item identifier\",\"$ref\": \"https://schemas.stacspec.org/v1.0.0/item-spec/json-schema/item.json#/definitions/core/allOf/2/properties/id\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('geometry', '{\"title\": \"Item Geometry\",\"description\": \"Item Geometry\",\"$ref\": \"https://geojson.org/schema/Feature.json\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('datetime','{\"description\": \"Datetime\",\"type\": \"string\",\"title\": \"Acquired\",\"format\": \"date-time\",\"pattern\": \"(\\\\+00:00|Z)$\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDELETE FROM queryables a USING queryables b\n WHERE a.name = b.name AND a.collection_ids IS NOT DISTINCT FROM b.collection_ids AND a.id > b.id;\n\n\nINSERT INTO pgstac_settings (name, value) VALUES\n ('context', 'off'),\n ('context_estimated_count', '100000'),\n ('context_estimated_cost', '100000'),\n ('context_stats_ttl', '1 day'),\n ('default_filter_lang', 'cql2-json'),\n ('additional_properties', 'true'),\n ('use_queue', 'false'),\n ('queue_timeout', '10 minutes'),\n ('update_collection_extent', 'false'),\n ('format_cache', 'false'),\n ('readonly', 'false')\nON CONFLICT DO NOTHING\n;\n\nALTER FUNCTION to_text COST 5000;\nALTER FUNCTION to_float COST 5000;\nALTER FUNCTION to_int COST 5000;\nALTER FUNCTION to_tstz COST 5000;\nALTER FUNCTION to_text_array COST 5000;\n\nALTER FUNCTION update_partition_stats SECURITY DEFINER;\nALTER FUNCTION partition_after_triggerfunc SECURITY DEFINER;\nALTER FUNCTION drop_table_constraints SECURITY DEFINER;\nALTER FUNCTION create_table_constraints SECURITY DEFINER;\nALTER FUNCTION check_partition SECURITY DEFINER;\nALTER FUNCTION repartition SECURITY DEFINER;\nALTER FUNCTION where_stats SECURITY DEFINER;\nALTER FUNCTION search_query SECURITY DEFINER;\nALTER FUNCTION format_item SECURITY DEFINER;\nALTER FUNCTION maintain_index SECURITY DEFINER;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nGRANT ALL ON SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac to pgstac_admin;\n\n-- pgstac_read role limited to using function apis\nGRANT EXECUTE ON FUNCTION search TO pgstac_read;\nGRANT EXECUTE ON FUNCTION search_query TO pgstac_read;\nGRANT EXECUTE ON FUNCTION item_by_id TO pgstac_read;\nGRANT EXECUTE ON FUNCTION get_item TO pgstac_read;\nGRANT SELECT ON ALL TABLES IN SCHEMA pgstac TO pgstac_read;\n\n\nGRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_ingest;\nGRANT USAGE ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_ingest;\n\nREVOKE ALL PRIVILEGES ON PROCEDURE run_queued_queries FROM public;\nGRANT ALL ON PROCEDURE run_queued_queries TO pgstac_admin;\n\nREVOKE ALL PRIVILEGES ON FUNCTION run_queued_queries_intransaction FROM public;\nGRANT ALL ON FUNCTION run_queued_queries_intransaction TO pgstac_admin;\n\nRESET ROLE;\n\nSET ROLE pgstac_ingest;\nSELECT update_partition_stats_q(partition) FROM partitions_view;\nSELECT set_version('0.8.4');\n" }, { "path": "src/pgstac/migrations/pgstac.0.8.5-0.9.0.sql", "content": "SET client_min_messages TO WARNING;\nSET SEARCH_PATH to pgstac, public;\nRESET ROLE;\nDO $$\nDECLARE\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='postgis') THEN\n CREATE EXTENSION IF NOT EXISTS postgis;\n END IF;\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='btree_gist') THEN\n CREATE EXTENSION IF NOT EXISTS btree_gist;\n END IF;\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='unaccent') THEN\n CREATE EXTENSION IF NOT EXISTS unaccent;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_admin;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_read;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_ingest;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n\nGRANT pgstac_admin TO current_user;\n\n-- Function to make sure pgstac_admin is the owner of items\nCREATE OR REPLACE FUNCTION pgstac_admin_owns() RETURNS VOID AS $$\nDECLARE\n f RECORD;\nBEGIN\n FOR f IN (\n SELECT\n concat(\n oid::regproc::text,\n '(',\n coalesce(pg_get_function_identity_arguments(oid),''),\n ')'\n ) AS name,\n CASE prokind WHEN 'f' THEN 'FUNCTION' WHEN 'p' THEN 'PROCEDURE' WHEN 'a' THEN 'AGGREGATE' END as typ\n FROM pg_proc\n WHERE\n pronamespace=to_regnamespace('pgstac')\n AND proowner != to_regrole('pgstac_admin')\n AND proname NOT LIKE 'pg_stat%'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n FOR f IN (\n SELECT\n oid::regclass::text as name,\n CASE relkind\n WHEN 'i' THEN 'INDEX'\n WHEN 'I' THEN 'INDEX'\n WHEN 'p' THEN 'TABLE'\n WHEN 'r' THEN 'TABLE'\n WHEN 'v' THEN 'VIEW'\n WHEN 'S' THEN 'SEQUENCE'\n ELSE NULL\n END as typ\n FROM pg_class\n WHERE relnamespace=to_regnamespace('pgstac') and relowner != to_regrole('pgstac_admin') AND relkind IN ('r','p','v','S') AND relname NOT LIKE 'pg_stat'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\nSELECT pgstac_admin_owns();\n\nCREATE SCHEMA IF NOT EXISTS pgstac AUTHORIZATION pgstac_admin;\n\nGRANT ALL ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_admin;\n\nALTER ROLE pgstac_admin SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_read SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_ingest SET SEARCH_PATH TO pgstac, public;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\n\nGRANT pgstac_read TO pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\n\nSET ROLE pgstac_admin;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\nRESET ROLE;\n\nSET ROLE pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\nRESET ROLE;\n\nSET SEARCH_PATH TO pgstac, public;\nSET ROLE pgstac_admin;\n\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS analyze_items;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS validate_constraints;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n-- Install these idempotently as migrations do not put them before trying to modify the collections table\n\n\nCREATE OR REPLACE FUNCTION collection_geom(content jsonb)\nRETURNS geometry AS $$\n WITH box AS (SELECT content->'extent'->'spatial'->'bbox'->0 as box)\n SELECT\n st_makeenvelope(\n (box->>0)::float,\n (box->>1)::float,\n (box->>2)::float,\n (box->>3)::float,\n 4326\n )\n FROM box;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION collection_datetime(content jsonb)\nRETURNS timestamptz AS $$\n SELECT\n CASE\n WHEN\n (content->'extent'->'temporal'->'interval'->0->>0) IS NULL\n THEN '-infinity'::timestamptz\n ELSE\n (content->'extent'->'temporal'->'interval'->0->>0)::timestamptz\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION collection_enddatetime(content jsonb)\nRETURNS timestamptz AS $$\n SELECT\n CASE\n WHEN\n (content->'extent'->'temporal'->'interval'->0->>1) IS NULL\n THEN 'infinity'::timestamptz\n ELSE\n (content->'extent'->'temporal'->'interval'->0->>1)::timestamptz\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n-- BEGIN migra calculated SQL\nset check_function_bodies = off;\n\nCREATE OR REPLACE FUNCTION pgstac.search_fromhash(_hash text)\n RETURNS searches\n LANGUAGE sql\n STRICT\nAS $function$\n SELECT * FROM search_query((SELECT search FROM searches WHERE hash=_hash LIMIT 1));\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.collection_search(_search jsonb DEFAULT '{}'::jsonb)\n RETURNS jsonb\n LANGUAGE plpgsql\n STABLE PARALLEL SAFE\nAS $function$\nDECLARE\n out_records jsonb;\n number_matched bigint := collection_search_matched(_search);\n number_returned bigint;\n _limit int := coalesce((_search->>'limit')::float::int, 10);\n _offset int := coalesce((_search->>'offset')::float::int, 0);\n links jsonb := '[]';\n ret jsonb;\n base_url text:= concat(rtrim(base_url(_search->'conf'),'/'), '/collections');\n prevoffset int;\n nextoffset int;\nBEGIN\n SELECT\n coalesce(jsonb_agg(c), '[]')\n INTO out_records\n FROM collection_search_rows(_search) c;\n\n number_returned := jsonb_array_length(out_records);\n\n IF _limit <= number_matched THEN --need to have paging links\n nextoffset := least(_offset + _limit, number_matched - 1);\n prevoffset := greatest(_offset - _limit, 0);\n IF _offset = 0 THEN -- no previous paging\n\n links := jsonb_build_array(\n jsonb_build_object(\n 'rel', 'next',\n 'type', 'application/json',\n 'method', 'GET' ,\n 'href', base_url,\n 'body', jsonb_build_object('offset', nextoffset),\n 'merge', TRUE\n )\n );\n ELSE\n links := jsonb_build_array(\n jsonb_build_object(\n 'rel', 'prev',\n 'type', 'application/json',\n 'method', 'GET' ,\n 'href', base_url,\n 'body', jsonb_build_object('offset', prevoffset),\n 'merge', TRUE\n ),\n jsonb_build_object(\n 'rel', 'next',\n 'type', 'application/json',\n 'method', 'GET' ,\n 'href', base_url,\n 'body', jsonb_build_object('offset', nextoffset),\n 'merge', TRUE\n )\n );\n END IF;\n END IF;\n\n ret := jsonb_build_object(\n 'collections', out_records,\n 'numberMatched', number_matched,\n 'numberReturned', number_returned,\n 'links', links\n );\n RETURN ret;\n\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.cql2_query(j jsonb, wrapper text DEFAULT NULL::text)\n RETURNS text\n LANGUAGE plpgsql\n STABLE\nAS $function$\n#variable_conflict use_variable\nDECLARE\n args jsonb := j->'args';\n arg jsonb;\n op text := lower(j->>'op');\n cql2op RECORD;\n literal text;\n _wrapper text;\n leftarg text;\n rightarg text;\n prop text;\n extra_props bool := pgstac.additional_properties();\nBEGIN\n IF j IS NULL OR (op IS NOT NULL AND args IS NULL) THEN\n RETURN NULL;\n END IF;\n RAISE NOTICE 'CQL2_QUERY: %', j;\n\n -- check if all properties are represented in the queryables\n IF NOT extra_props THEN\n FOR prop IN\n SELECT DISTINCT p->>0\n FROM jsonb_path_query(j, 'strict $.**.property') p\n WHERE p->>0 NOT IN ('id', 'datetime', 'geometry', 'end_datetime', 'collection')\n LOOP\n IF (queryable(prop)).nulled_wrapper IS NULL THEN\n RAISE EXCEPTION 'Term % is not found in queryables.', prop;\n END IF;\n END LOOP;\n END IF;\n\n IF j ? 'filter' THEN\n RETURN cql2_query(j->'filter');\n END IF;\n\n IF j ? 'upper' THEN\n RETURN cql2_query(jsonb_build_object('op', 'upper', 'args', j->'upper'));\n END IF;\n\n IF j ? 'lower' THEN\n RETURN cql2_query(jsonb_build_object('op', 'lower', 'args', j->'lower'));\n END IF;\n\n -- Temporal Query\n IF op ilike 't_%' or op = 'anyinteracts' THEN\n RETURN temporal_op_query(op, args);\n END IF;\n\n -- If property is a timestamp convert it to text to use with\n -- general operators\n IF j ? 'timestamp' THEN\n RETURN format('%L::timestamptz', to_tstz(j->'timestamp'));\n END IF;\n IF j ? 'interval' THEN\n RAISE EXCEPTION 'Please use temporal operators when using intervals.';\n RETURN NONE;\n END IF;\n\n -- Spatial Query\n IF op ilike 's_%' or op = 'intersects' THEN\n RETURN spatial_op_query(op, args);\n END IF;\n\n IF op IN ('a_equals','a_contains','a_contained_by','a_overlaps') THEN\n IF args->0 ? 'property' THEN\n leftarg := format('to_text_array(%s)', (queryable(args->0->>'property')).path);\n END IF;\n IF args->1 ? 'property' THEN\n rightarg := format('to_text_array(%s)', (queryable(args->1->>'property')).path);\n END IF;\n RETURN FORMAT(\n '%s %s %s',\n COALESCE(leftarg, quote_literal(to_text_array(args->0))),\n CASE op\n WHEN 'a_equals' THEN '='\n WHEN 'a_contains' THEN '@>'\n WHEN 'a_contained_by' THEN '<@'\n WHEN 'a_overlaps' THEN '&&'\n END,\n COALESCE(rightarg, quote_literal(to_text_array(args->1)))\n );\n END IF;\n\n IF op = 'in' THEN\n RAISE NOTICE 'IN : % % %', args, jsonb_build_array(args->0), args->1;\n args := jsonb_build_array(args->0) || (args->1);\n RAISE NOTICE 'IN2 : %', args;\n END IF;\n\n\n\n IF op = 'between' THEN\n args = jsonb_build_array(\n args->0,\n args->1,\n args->2\n );\n END IF;\n\n -- Make sure that args is an array and run cql2_query on\n -- each element of the array\n RAISE NOTICE 'ARGS PRE: %', args;\n IF j ? 'args' THEN\n IF jsonb_typeof(args) != 'array' THEN\n args := jsonb_build_array(args);\n END IF;\n\n IF jsonb_path_exists(args, '$[*] ? (@.property == \"id\" || @.property == \"datetime\" || @.property == \"end_datetime\" || @.property == \"collection\")') THEN\n wrapper := NULL;\n ELSE\n -- if any of the arguments are a property, try to get the property_wrapper\n FOR arg IN SELECT jsonb_path_query(args, '$[*] ? (@.property != null)') LOOP\n RAISE NOTICE 'Arg: %', arg;\n wrapper := (queryable(arg->>'property')).nulled_wrapper;\n RAISE NOTICE 'Property: %, Wrapper: %', arg, wrapper;\n IF wrapper IS NOT NULL THEN\n EXIT;\n END IF;\n END LOOP;\n\n -- if the property was not in queryables, see if any args were numbers\n IF\n wrapper IS NULL\n AND jsonb_path_exists(args, '$[*] ? (@.type()==\"number\")')\n THEN\n wrapper := 'to_float';\n END IF;\n wrapper := coalesce(wrapper, 'to_text');\n END IF;\n\n SELECT jsonb_agg(cql2_query(a, wrapper))\n INTO args\n FROM jsonb_array_elements(args) a;\n END IF;\n RAISE NOTICE 'ARGS: %', args;\n\n IF op IN ('and', 'or') THEN\n RETURN\n format(\n '(%s)',\n array_to_string(to_text_array(args), format(' %s ', upper(op)))\n );\n END IF;\n\n IF op = 'in' THEN\n RAISE NOTICE 'IN -- % %', args->0, to_text(args->0);\n RETURN format(\n '%s IN (%s)',\n to_text(args->0),\n array_to_string((to_text_array(args))[2:], ',')\n );\n END IF;\n\n -- Look up template from cql2_ops\n IF j ? 'op' THEN\n SELECT * INTO cql2op FROM cql2_ops WHERE cql2_ops.op ilike op;\n IF FOUND THEN\n -- If specific index set in queryables for a property cast other arguments to that type\n\n RETURN format(\n cql2op.template,\n VARIADIC (to_text_array(args))\n );\n ELSE\n RAISE EXCEPTION 'Operator % Not Supported.', op;\n END IF;\n END IF;\n\n\n IF wrapper IS NOT NULL THEN\n RAISE NOTICE 'Wrapping % with %', j, wrapper;\n IF j ? 'property' THEN\n RETURN format('%I(%s)', wrapper, (queryable(j->>'property')).path);\n ELSE\n RETURN format('%I(%L)', wrapper, j);\n END IF;\n ELSIF j ? 'property' THEN\n RETURN quote_ident(j->>'property');\n END IF;\n\n RETURN quote_literal(to_text(j));\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.geometrysearch(geom geometry, queryhash text, fields jsonb DEFAULT NULL::jsonb, _scanlimit integer DEFAULT 10000, _limit integer DEFAULT 100, _timelimit interval DEFAULT '00:00:05'::interval, exitwhenfull boolean DEFAULT true, skipcovered boolean DEFAULT true)\n RETURNS jsonb\n LANGUAGE plpgsql\nAS $function$\nDECLARE\n search searches%ROWTYPE;\n curs refcursor;\n _where text;\n query text;\n iter_record items%ROWTYPE;\n out_records jsonb := '{}'::jsonb[];\n exit_flag boolean := FALSE;\n counter int := 1;\n scancounter int := 1;\n remaining_limit int := _scanlimit;\n tilearea float;\n unionedgeom geometry;\n clippedgeom geometry;\n unionedgeom_area float := 0;\n prev_area float := 0;\n excludes text[];\n includes text[];\n\nBEGIN\n DROP TABLE IF EXISTS pgstac_results;\n CREATE TEMP TABLE pgstac_results (content jsonb) ON COMMIT DROP;\n\n -- If the passed in geometry is not an area set exitwhenfull and skipcovered to false\n IF ST_GeometryType(geom) !~* 'polygon' THEN\n RAISE NOTICE 'GEOMETRY IS NOT AN AREA';\n skipcovered = FALSE;\n exitwhenfull = FALSE;\n END IF;\n\n -- If skipcovered is true then you will always want to exit when the passed in geometry is full\n IF skipcovered THEN\n exitwhenfull := TRUE;\n END IF;\n\n search := search_fromhash(queryhash);\n\n IF search IS NULL THEN\n RAISE EXCEPTION 'Search with Query Hash % Not Found', queryhash;\n END IF;\n\n tilearea := st_area(geom);\n _where := format('%s AND st_intersects(geometry, %L::geometry)', search._where, geom);\n\n\n FOR query IN SELECT * FROM partition_queries(_where, search.orderby) LOOP\n query := format('%s LIMIT %L', query, remaining_limit);\n RAISE NOTICE '%', query;\n OPEN curs FOR EXECUTE query;\n LOOP\n FETCH curs INTO iter_record;\n EXIT WHEN NOT FOUND;\n IF exitwhenfull OR skipcovered THEN -- If we are not using exitwhenfull or skipcovered, we do not need to do expensive geometry operations\n clippedgeom := st_intersection(geom, iter_record.geometry);\n\n IF unionedgeom IS NULL THEN\n unionedgeom := clippedgeom;\n ELSE\n unionedgeom := st_union(unionedgeom, clippedgeom);\n END IF;\n\n unionedgeom_area := st_area(unionedgeom);\n\n IF skipcovered AND prev_area = unionedgeom_area THEN\n scancounter := scancounter + 1;\n CONTINUE;\n END IF;\n\n prev_area := unionedgeom_area;\n\n RAISE NOTICE '% % % %', unionedgeom_area/tilearea, counter, scancounter, ftime();\n END IF;\n RAISE NOTICE '% %', iter_record, content_hydrate(iter_record, fields);\n INSERT INTO pgstac_results (content) VALUES (content_hydrate(iter_record, fields));\n\n IF counter >= _limit\n OR scancounter > _scanlimit\n OR ftime() > _timelimit\n OR (exitwhenfull AND unionedgeom_area >= tilearea)\n THEN\n exit_flag := TRUE;\n EXIT;\n END IF;\n counter := counter + 1;\n scancounter := scancounter + 1;\n\n END LOOP;\n CLOSE curs;\n EXIT WHEN exit_flag;\n remaining_limit := _scanlimit - scancounter;\n END LOOP;\n\n SELECT jsonb_agg(content) INTO out_records FROM pgstac_results WHERE content IS NOT NULL;\n\n RETURN jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb)\n );\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.search(_search jsonb DEFAULT '{}'::jsonb)\n RETURNS jsonb\n LANGUAGE plpgsql\nAS $function$\nDECLARE\n searches searches%ROWTYPE;\n _where text;\n orderby text;\n search_where search_wheres%ROWTYPE;\n total_count bigint;\n token record;\n token_prev boolean;\n token_item items%ROWTYPE;\n token_where text;\n full_where text;\n init_ts timestamptz := clock_timestamp();\n timer timestamptz := clock_timestamp();\n hydrate bool := NOT (_search->'conf'->>'nohydrate' IS NOT NULL AND (_search->'conf'->>'nohydrate')::boolean = true);\n prev text;\n next text;\n context jsonb;\n collection jsonb;\n out_records jsonb;\n out_len int;\n _limit int := coalesce((_search->>'limit')::int, 10);\n _querylimit int;\n _fields jsonb := coalesce(_search->'fields', '{}'::jsonb);\n has_prev boolean := FALSE;\n has_next boolean := FALSE;\n links jsonb := '[]'::jsonb;\n base_url text:= concat(rtrim(base_url(_search->'conf'),'/'));\nBEGIN\n searches := search_query(_search);\n _where := searches._where;\n orderby := searches.orderby;\n search_where := where_stats(_where);\n total_count := coalesce(search_where.total_count, search_where.estimated_count);\n RAISE NOTICE 'SEARCH:TOKEN: %', _search->>'token';\n token := get_token_record(_search->>'token');\n RAISE NOTICE '***TOKEN: %', token;\n _querylimit := _limit + 1;\n IF token IS NOT NULL THEN\n token_prev := token.prev;\n token_item := token.item;\n token_where := get_token_filter(_search->'sortby', token_item, token_prev, FALSE);\n RAISE DEBUG 'TOKEN_WHERE: % (%ms from search start)', token_where, age_ms(timer);\n IF token_prev THEN -- if we are using a prev token, we know has_next is true\n RAISE DEBUG 'There is a previous token, so automatically setting has_next to true';\n has_next := TRUE;\n orderby := sort_sqlorderby(_search, TRUE);\n ELSE\n RAISE DEBUG 'There is a next token, so automatically setting has_prev to true';\n has_prev := TRUE;\n\n END IF;\n ELSE -- if there was no token, we know there is no prev\n RAISE DEBUG 'There is no token, so we know there is no prev. setting has_prev to false';\n has_prev := FALSE;\n END IF;\n\n full_where := concat_ws(' AND ', _where, token_where);\n RAISE NOTICE 'FULL WHERE CLAUSE: %', full_where;\n RAISE NOTICE 'Time to get counts and build query %', age_ms(timer);\n timer := clock_timestamp();\n\n IF hydrate THEN\n RAISE NOTICE 'Getting hydrated data.';\n ELSE\n RAISE NOTICE 'Getting non-hydrated data.';\n END IF;\n RAISE NOTICE 'CACHE SET TO %', get_setting_bool('format_cache');\n RAISE NOTICE 'Time to set hydration/formatting %', age_ms(timer);\n timer := clock_timestamp();\n SELECT jsonb_agg(format_item(i, _fields, hydrate)) INTO out_records\n FROM search_rows(\n full_where,\n orderby,\n search_where.partitions,\n _querylimit\n ) as i;\n\n RAISE NOTICE 'Time to fetch rows %', age_ms(timer);\n timer := clock_timestamp();\n\n\n IF token_prev THEN\n out_records := flip_jsonb_array(out_records);\n END IF;\n\n RAISE NOTICE 'Query returned % records.', jsonb_array_length(out_records);\n RAISE DEBUG 'TOKEN: % %', token_item.id, token_item.collection;\n RAISE DEBUG 'RECORD_1: % %', out_records->0->>'id', out_records->0->>'collection';\n RAISE DEBUG 'RECORD-1: % %', out_records->-1->>'id', out_records->-1->>'collection';\n\n -- REMOVE records that were from our token\n IF out_records->0->>'id' = token_item.id AND out_records->0->>'collection' = token_item.collection THEN\n out_records := out_records - 0;\n ELSIF out_records->-1->>'id' = token_item.id AND out_records->-1->>'collection' = token_item.collection THEN\n out_records := out_records - -1;\n END IF;\n\n out_len := jsonb_array_length(out_records);\n\n IF out_len = _limit + 1 THEN\n IF token_prev THEN\n has_prev := TRUE;\n out_records := out_records - 0;\n ELSE\n has_next := TRUE;\n out_records := out_records - -1;\n END IF;\n END IF;\n\n\n links := links || jsonb_build_object(\n 'rel', 'root',\n 'type', 'application/json',\n 'href', base_url\n ) || jsonb_build_object(\n 'rel', 'self',\n 'type', 'application/json',\n 'href', concat(base_url, '/search')\n );\n\n IF has_next THEN\n next := concat(out_records->-1->>'collection', ':', out_records->-1->>'id');\n RAISE NOTICE 'HAS NEXT | %', next;\n links := links || jsonb_build_object(\n 'rel', 'next',\n 'type', 'application/geo+json',\n 'method', 'GET',\n 'href', concat(base_url, '/search?token=next:', next)\n );\n END IF;\n\n IF has_prev THEN\n prev := concat(out_records->0->>'collection', ':', out_records->0->>'id');\n RAISE NOTICE 'HAS PREV | %', prev;\n links := links || jsonb_build_object(\n 'rel', 'prev',\n 'type', 'application/geo+json',\n 'method', 'GET',\n 'href', concat(base_url, '/search?token=prev:', prev)\n );\n END IF;\n\n RAISE NOTICE 'Time to get prev/next %', age_ms(timer);\n timer := clock_timestamp();\n\n\n collection := jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb),\n 'links', links\n );\n\n\n\n IF context(_search->'conf') != 'off' THEN\n collection := collection || jsonb_strip_nulls(jsonb_build_object(\n 'numberMatched', total_count,\n 'numberReturned', coalesce(jsonb_array_length(out_records), 0)\n ));\n ELSE\n collection := collection || jsonb_strip_nulls(jsonb_build_object(\n 'numberReturned', coalesce(jsonb_array_length(out_records), 0)\n ));\n END IF;\n\n IF get_setting_bool('timing', _search->'conf') THEN\n collection = collection || jsonb_build_object('timing', age_ms(init_ts));\n END IF;\n\n RAISE NOTICE 'Time to build final json %', age_ms(timer);\n timer := clock_timestamp();\n\n RAISE NOTICE 'Total Time: %', age_ms(current_timestamp);\n RAISE NOTICE 'RETURNING % records. NEXT: %. PREV: %', collection->>'numberReturned', collection->>'next', collection->>'prev';\n RETURN collection;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.search_query(_search jsonb DEFAULT '{}'::jsonb, updatestats boolean DEFAULT false, _metadata jsonb DEFAULT '{}'::jsonb)\n RETURNS searches\n LANGUAGE plpgsql\n SECURITY DEFINER\nAS $function$\nDECLARE\n search searches%ROWTYPE;\n cached_search searches%ROWTYPE;\n pexplain jsonb;\n t timestamptz;\n i interval;\n doupdate boolean := FALSE;\n insertfound boolean := FALSE;\n ro boolean := pgstac.readonly();\n found_search text;\nBEGIN\n RAISE NOTICE 'SEARCH: %', _search;\n -- Calculate hash, where clause, and order by statement\n search.search := _search;\n search.metadata := _metadata;\n search.hash := search_hash(_search, _metadata);\n search._where := stac_search_to_where(_search);\n search.orderby := sort_sqlorderby(_search);\n search.lastused := now();\n search.usecount := 1;\n\n -- If we are in read only mode, directly return search\n IF ro THEN\n RETURN search;\n END IF;\n\n RAISE NOTICE 'Updating Statistics for search: %s', search;\n -- Update statistics for times used and and when last used\n -- If the entry is locked, rather than waiting, skip updating the stats\n INSERT INTO searches (search, lastused, usecount, metadata)\n VALUES (search.search, now(), 1, search.metadata)\n ON CONFLICT DO NOTHING\n RETURNING * INTO cached_search\n ;\n\n IF NOT FOUND OR cached_search IS NULL THEN\n UPDATE searches SET\n lastused = now(),\n usecount = searches.usecount + 1\n WHERE hash = (\n SELECT hash FROM searches WHERE hash=search.hash FOR UPDATE SKIP LOCKED\n )\n RETURNING * INTO cached_search\n ;\n END IF;\n\n IF cached_search IS NOT NULL THEN\n cached_search._where = search._where;\n cached_search.orderby = search.orderby;\n RETURN cached_search;\n END IF;\n RETURN search;\n\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.search_rows(_where text DEFAULT 'TRUE'::text, _orderby text DEFAULT 'datetime DESC, id DESC'::text, partitions text[] DEFAULT NULL::text[], _limit integer DEFAULT 10)\n RETURNS SETOF items\n LANGUAGE plpgsql\n SET search_path TO 'pgstac', 'public'\nAS $function$\nDECLARE\n base_query text;\n query text;\n sdate timestamptz;\n edate timestamptz;\n n int;\n records_left int := _limit;\n timer timestamptz := clock_timestamp();\n full_timer timestamptz := clock_timestamp();\nBEGIN\nIF _where IS NULL OR trim(_where) = '' THEN\n _where = ' TRUE ';\nEND IF;\nRAISE NOTICE 'Getting chunks for % %', _where, _orderby;\n\nbase_query := $q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n LIMIT %L\n$q$;\n\nIF _orderby ILIKE 'datetime d%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 DESC LOOP\n RAISE NOTICE 'Running Query for % to %. %', sdate, edate, age_ms(full_timer);\n query := format(\n base_query,\n sdate,\n edate,\n _where,\n _orderby,\n records_left\n );\n RAISE DEBUG 'QUERY: %', query;\n timer := clock_timestamp();\n RETURN QUERY EXECUTE query;\n\n GET DIAGNOSTICS n = ROW_COUNT;\n records_left := records_left - n;\n RAISE NOTICE 'Returned %/% Rows From % to %. % to go. Time: %ms', n, _limit, sdate, edate, records_left, age_ms(timer);\n timer := clock_timestamp();\n IF records_left <= 0 THEN\n RAISE NOTICE 'SEARCH_ROWS TOOK %ms', age_ms(full_timer);\n RETURN;\n END IF;\n END LOOP;\nELSIF _orderby ILIKE 'datetime a%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 ASC LOOP\n RAISE NOTICE 'Running Query for % to %. %', sdate, edate, age_ms(full_timer);\n query := format(\n base_query,\n sdate,\n edate,\n _where,\n _orderby,\n records_left\n );\n RAISE DEBUG 'QUERY: %', query;\n timer := clock_timestamp();\n RETURN QUERY EXECUTE query;\n\n GET DIAGNOSTICS n = ROW_COUNT;\n records_left := records_left - n;\n RAISE NOTICE 'Returned %/% Rows From % to %. % to go. Time: %ms', n, _limit, sdate, edate, records_left, age_ms(timer);\n timer := clock_timestamp();\n IF records_left <= 0 THEN\n RAISE NOTICE 'SEARCH_ROWS TOOK %ms', age_ms(full_timer);\n RETURN;\n END IF;\n END LOOP;\nELSE\n query := format($q$\n SELECT * FROM items\n WHERE %s\n ORDER BY %s\n LIMIT %L\n $q$, _where, _orderby, _limit\n );\n RAISE DEBUG 'QUERY: %', query;\n timer := clock_timestamp();\n RETURN QUERY EXECUTE query;\n RAISE NOTICE 'FULL QUERY TOOK %ms', age_ms(timer);\nEND IF;\nRAISE NOTICE 'SEARCH_ROWS TOOK %ms', age_ms(full_timer);\nRETURN;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.where_stats(inwhere text, updatestats boolean DEFAULT false, conf jsonb DEFAULT NULL::jsonb)\n RETURNS search_wheres\n LANGUAGE plpgsql\n SECURITY DEFINER\nAS $function$\nDECLARE\n t timestamptz;\n i interval;\n explain_json jsonb;\n partitions text[];\n sw search_wheres%ROWTYPE;\n inwhere_hash text := md5(inwhere);\n _context text := lower(context(conf));\n _stats_ttl interval := context_stats_ttl(conf);\n _estimated_cost_threshold float := context_estimated_cost(conf);\n _estimated_count_threshold int := context_estimated_count(conf);\n ro bool := pgstac.readonly(conf);\nBEGIN\n -- If updatestats is true then set ttl to 0\n IF updatestats THEN\n RAISE DEBUG 'Updatestats set to TRUE, setting TTL to 0';\n _stats_ttl := '0'::interval;\n END IF;\n\n -- If we don't need to calculate context, just return\n IF _context = 'off' THEN\n sw._where = inwhere;\n RETURN sw;\n END IF;\n\n -- Get any stats that we have. If there is a lock where another process is\n -- updating the stats, wait so that we don't end up calculating a bunch of times.\n SELECT * INTO sw FROM search_wheres WHERE md5(_where)=inwhere_hash FOR UPDATE;\n\n -- If there is a cached row, figure out if we need to update\n IF\n sw IS NOT NULL\n AND sw.statslastupdated IS NOT NULL\n AND sw.total_count IS NOT NULL\n AND now() - sw.statslastupdated <= _stats_ttl\n THEN\n -- we have a cached row with data that is within our ttl\n RAISE DEBUG 'Stats present in table and lastupdated within ttl: %', sw;\n IF NOT ro THEN\n RAISE DEBUG 'Updating search_wheres only bumping lastused and usecount';\n UPDATE search_wheres SET\n lastused = now(),\n usecount = search_wheres.usecount + 1\n WHERE md5(_where) = inwhere_hash\n RETURNING * INTO sw;\n END IF;\n RAISE DEBUG 'Returning cached counts. %', sw;\n RETURN sw;\n END IF;\n\n -- Calculate estimated cost and rows\n -- Use explain to get estimated count/cost\n IF sw.estimated_count IS NULL OR sw.estimated_cost IS NULL THEN\n RAISE DEBUG 'Calculating estimated stats';\n t := clock_timestamp();\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s', inwhere)\n INTO explain_json;\n RAISE DEBUG 'Time for just the explain: %', clock_timestamp() - t;\n i := clock_timestamp() - t;\n\n sw.estimated_count := explain_json->0->'Plan'->'Plan Rows';\n sw.estimated_cost := explain_json->0->'Plan'->'Total Cost';\n sw.time_to_estimate := extract(epoch from i);\n END IF;\n\n RAISE DEBUG 'ESTIMATED_COUNT: %, THRESHOLD %', sw.estimated_count, _estimated_count_threshold;\n RAISE DEBUG 'ESTIMATED_COST: %, THRESHOLD %', sw.estimated_cost, _estimated_cost_threshold;\n\n -- If context is set to auto and the costs are within the threshold return the estimated costs\n IF\n _context = 'auto'\n AND sw.estimated_count >= _estimated_count_threshold\n AND sw.estimated_cost >= _estimated_cost_threshold\n THEN\n IF NOT ro THEN\n INSERT INTO search_wheres (\n _where,\n lastused,\n usecount,\n statslastupdated,\n estimated_count,\n estimated_cost,\n time_to_estimate,\n total_count,\n time_to_count\n ) VALUES (\n inwhere,\n now(),\n 1,\n now(),\n sw.estimated_count,\n sw.estimated_cost,\n sw.time_to_estimate,\n null,\n null\n ) ON CONFLICT ((md5(_where)))\n DO UPDATE SET\n lastused = EXCLUDED.lastused,\n usecount = search_wheres.usecount + 1,\n statslastupdated = EXCLUDED.statslastupdated,\n estimated_count = EXCLUDED.estimated_count,\n estimated_cost = EXCLUDED.estimated_cost,\n time_to_estimate = EXCLUDED.time_to_estimate,\n total_count = EXCLUDED.total_count,\n time_to_count = EXCLUDED.time_to_count\n RETURNING * INTO sw;\n END IF;\n RAISE DEBUG 'Estimates are within thresholds, returning estimates. %', sw;\n RETURN sw;\n END IF;\n\n -- Calculate Actual Count\n t := clock_timestamp();\n RAISE NOTICE 'Calculating actual count...';\n EXECUTE format(\n 'SELECT count(*) FROM items WHERE %s',\n inwhere\n ) INTO sw.total_count;\n i := clock_timestamp() - t;\n RAISE NOTICE 'Actual Count: % -- %', sw.total_count, i;\n sw.time_to_count := extract(epoch FROM i);\n\n IF NOT ro THEN\n INSERT INTO search_wheres (\n _where,\n lastused,\n usecount,\n statslastupdated,\n estimated_count,\n estimated_cost,\n time_to_estimate,\n total_count,\n time_to_count\n ) VALUES (\n inwhere,\n now(),\n 1,\n now(),\n sw.estimated_count,\n sw.estimated_cost,\n sw.time_to_estimate,\n sw.total_count,\n sw.time_to_count\n ) ON CONFLICT ((md5(_where)))\n DO UPDATE SET\n lastused = EXCLUDED.lastused,\n usecount = search_wheres.usecount + 1,\n statslastupdated = EXCLUDED.statslastupdated,\n estimated_count = EXCLUDED.estimated_count,\n estimated_cost = EXCLUDED.estimated_cost,\n time_to_estimate = EXCLUDED.time_to_estimate,\n total_count = EXCLUDED.total_count,\n time_to_count = EXCLUDED.time_to_count\n RETURNING * INTO sw;\n END IF;\n RAISE DEBUG 'Returning with actual count. %', sw;\n RETURN sw;\nEND;\n$function$\n;\n\n\n-- END migra calculated SQL\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('id', '{\"title\": \"Item ID\",\"description\": \"Item identifier\",\"$ref\": \"https://schemas.stacspec.org/v1.0.0/item-spec/json-schema/item.json#/definitions/core/allOf/2/properties/id\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('geometry', '{\"title\": \"Item Geometry\",\"description\": \"Item Geometry\",\"$ref\": \"https://geojson.org/schema/Feature.json\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('datetime','{\"description\": \"Datetime\",\"type\": \"string\",\"title\": \"Acquired\",\"format\": \"date-time\",\"pattern\": \"(\\\\+00:00|Z)$\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDELETE FROM queryables a USING queryables b\n WHERE a.name = b.name AND a.collection_ids IS NOT DISTINCT FROM b.collection_ids AND a.id > b.id;\n\n\nINSERT INTO pgstac_settings (name, value) VALUES\n ('context', 'off'),\n ('context_estimated_count', '100000'),\n ('context_estimated_cost', '100000'),\n ('context_stats_ttl', '1 day'),\n ('default_filter_lang', 'cql2-json'),\n ('additional_properties', 'true'),\n ('use_queue', 'false'),\n ('queue_timeout', '10 minutes'),\n ('update_collection_extent', 'false'),\n ('format_cache', 'false'),\n ('readonly', 'false')\nON CONFLICT DO NOTHING\n;\n\n\nINSERT INTO cql2_ops (op, template, types) VALUES\n ('eq', '%s = %s', NULL),\n ('neq', '%s != %s', NULL),\n ('ne', '%s != %s', NULL),\n ('!=', '%s != %s', NULL),\n ('<>', '%s != %s', NULL),\n ('lt', '%s < %s', NULL),\n ('lte', '%s <= %s', NULL),\n ('gt', '%s > %s', NULL),\n ('gte', '%s >= %s', NULL),\n ('le', '%s <= %s', NULL),\n ('ge', '%s >= %s', NULL),\n ('=', '%s = %s', NULL),\n ('<', '%s < %s', NULL),\n ('<=', '%s <= %s', NULL),\n ('>', '%s > %s', NULL),\n ('>=', '%s >= %s', NULL),\n ('like', '%s LIKE %s', NULL),\n ('ilike', '%s ILIKE %s', NULL),\n ('+', '%s + %s', NULL),\n ('-', '%s - %s', NULL),\n ('*', '%s * %s', NULL),\n ('/', '%s / %s', NULL),\n ('not', 'NOT (%s)', NULL),\n ('between', '%s BETWEEN %s AND %s', NULL),\n ('isnull', '%s IS NULL', NULL),\n ('upper', 'upper(%s)', NULL),\n ('lower', 'lower(%s)', NULL),\n ('casei', 'upper(%s)', NULL),\n ('accenti', 'unaccent(%s)', NULL)\nON CONFLICT (op) DO UPDATE\n SET\n template = EXCLUDED.template\n;\n\n\nALTER FUNCTION to_text COST 5000;\nALTER FUNCTION to_float COST 5000;\nALTER FUNCTION to_int COST 5000;\nALTER FUNCTION to_tstz COST 5000;\nALTER FUNCTION to_text_array COST 5000;\n\nALTER FUNCTION update_partition_stats SECURITY DEFINER;\nALTER FUNCTION partition_after_triggerfunc SECURITY DEFINER;\nALTER FUNCTION drop_table_constraints SECURITY DEFINER;\nALTER FUNCTION create_table_constraints SECURITY DEFINER;\nALTER FUNCTION check_partition SECURITY DEFINER;\nALTER FUNCTION repartition SECURITY DEFINER;\nALTER FUNCTION where_stats SECURITY DEFINER;\nALTER FUNCTION search_query SECURITY DEFINER;\nALTER FUNCTION format_item SECURITY DEFINER;\nALTER FUNCTION maintain_index SECURITY DEFINER;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nGRANT ALL ON SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac to pgstac_admin;\n\n-- pgstac_read role limited to using function apis\nGRANT EXECUTE ON FUNCTION search TO pgstac_read;\nGRANT EXECUTE ON FUNCTION search_query TO pgstac_read;\nGRANT EXECUTE ON FUNCTION item_by_id TO pgstac_read;\nGRANT EXECUTE ON FUNCTION get_item TO pgstac_read;\nGRANT SELECT ON ALL TABLES IN SCHEMA pgstac TO pgstac_read;\n\n\nGRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_ingest;\nGRANT USAGE ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_ingest;\n\nREVOKE ALL PRIVILEGES ON PROCEDURE run_queued_queries FROM public;\nGRANT ALL ON PROCEDURE run_queued_queries TO pgstac_admin;\n\nREVOKE ALL PRIVILEGES ON FUNCTION run_queued_queries_intransaction FROM public;\nGRANT ALL ON FUNCTION run_queued_queries_intransaction TO pgstac_admin;\n\nRESET ROLE;\n\nSET ROLE pgstac_ingest;\nSELECT update_partition_stats_q(partition) FROM partitions_view;\nSELECT set_version('0.9.0');\n" }, { "path": "src/pgstac/migrations/pgstac.0.8.5.sql", "content": "RESET ROLE;\nDO $$\nDECLARE\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='postgis') THEN\n CREATE EXTENSION IF NOT EXISTS postgis;\n END IF;\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='btree_gist') THEN\n CREATE EXTENSION IF NOT EXISTS btree_gist;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_admin;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_read;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_ingest;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n\nGRANT pgstac_admin TO current_user;\n\n-- Function to make sure pgstac_admin is the owner of items\nCREATE OR REPLACE FUNCTION pgstac_admin_owns() RETURNS VOID AS $$\nDECLARE\n f RECORD;\nBEGIN\n FOR f IN (\n SELECT\n concat(\n oid::regproc::text,\n '(',\n coalesce(pg_get_function_identity_arguments(oid),''),\n ')'\n ) AS name,\n CASE prokind WHEN 'f' THEN 'FUNCTION' WHEN 'p' THEN 'PROCEDURE' WHEN 'a' THEN 'AGGREGATE' END as typ\n FROM pg_proc\n WHERE\n pronamespace=to_regnamespace('pgstac')\n AND proowner != to_regrole('pgstac_admin')\n AND proname NOT LIKE 'pg_stat%'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n FOR f IN (\n SELECT\n oid::regclass::text as name,\n CASE relkind\n WHEN 'i' THEN 'INDEX'\n WHEN 'I' THEN 'INDEX'\n WHEN 'p' THEN 'TABLE'\n WHEN 'r' THEN 'TABLE'\n WHEN 'v' THEN 'VIEW'\n WHEN 'S' THEN 'SEQUENCE'\n ELSE NULL\n END as typ\n FROM pg_class\n WHERE relnamespace=to_regnamespace('pgstac') and relowner != to_regrole('pgstac_admin') AND relkind IN ('r','p','v','S') AND relname NOT LIKE 'pg_stat'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\nSELECT pgstac_admin_owns();\n\nCREATE SCHEMA IF NOT EXISTS pgstac AUTHORIZATION pgstac_admin;\n\nGRANT ALL ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_admin;\n\nALTER ROLE pgstac_admin SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_read SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_ingest SET SEARCH_PATH TO pgstac, public;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\n\nGRANT pgstac_read TO pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\n\nSET ROLE pgstac_admin;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\nRESET ROLE;\n\nSET ROLE pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\nRESET ROLE;\n\nSET SEARCH_PATH TO pgstac, public;\nSET ROLE pgstac_admin;\n\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS analyze_items;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS validate_constraints;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n-- Install these idempotently as migrations do not put them before trying to modify the collections table\n\n\nCREATE OR REPLACE FUNCTION collection_geom(content jsonb)\nRETURNS geometry AS $$\n WITH box AS (SELECT content->'extent'->'spatial'->'bbox'->0 as box)\n SELECT\n st_makeenvelope(\n (box->>0)::float,\n (box->>1)::float,\n (box->>2)::float,\n (box->>3)::float,\n 4326\n )\n FROM box;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION collection_datetime(content jsonb)\nRETURNS timestamptz AS $$\n SELECT\n CASE\n WHEN\n (content->'extent'->'temporal'->'interval'->0->>0) IS NULL\n THEN '-infinity'::timestamptz\n ELSE\n (content->'extent'->'temporal'->'interval'->0->>0)::timestamptz\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION collection_enddatetime(content jsonb)\nRETURNS timestamptz AS $$\n SELECT\n CASE\n WHEN\n (content->'extent'->'temporal'->'interval'->0->>1) IS NULL\n THEN 'infinity'::timestamptz\n ELSE\n (content->'extent'->'temporal'->'interval'->0->>1)::timestamptz\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE TABLE IF NOT EXISTS migrations (\n version text PRIMARY KEY,\n datetime timestamptz DEFAULT clock_timestamp() NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION get_version() RETURNS text AS $$\n SELECT version FROM pgstac.migrations ORDER BY datetime DESC, version DESC LIMIT 1;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION set_version(text) RETURNS text AS $$\n INSERT INTO pgstac.migrations (version) VALUES ($1)\n ON CONFLICT DO NOTHING\n RETURNING version;\n$$ LANGUAGE SQL;\n\n\nCREATE TABLE IF NOT EXISTS pgstac_settings (\n name text PRIMARY KEY,\n value text NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION table_empty(text) RETURNS boolean AS $$\nDECLARE\n retval boolean;\nBEGIN\n EXECUTE format($q$\n SELECT NOT EXISTS (SELECT 1 FROM %I LIMIT 1)\n $q$,\n $1\n ) INTO retval;\n RETURN retval;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION get_setting(IN _setting text, IN conf jsonb DEFAULT NULL) RETURNS text AS $$\nSELECT COALESCE(\n nullif(conf->>_setting, ''),\n nullif(current_setting(concat('pgstac.',_setting), TRUE),''),\n nullif((SELECT value FROM pgstac.pgstac_settings WHERE name=_setting),'')\n);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_setting_bool(IN _setting text, IN conf jsonb DEFAULT NULL) RETURNS boolean AS $$\nSELECT COALESCE(\n nullif(conf->>_setting, ''),\n nullif(current_setting(concat('pgstac.',_setting), TRUE),''),\n nullif((SELECT value FROM pgstac.pgstac_settings WHERE name=_setting),''),\n 'FALSE'\n)::boolean;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION base_url(conf jsonb DEFAULT NULL) RETURNS text AS $$\n SELECT COALESCE(pgstac.get_setting('base_url', conf), '.');\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION additional_properties() RETURNS boolean AS $$\n SELECT pgstac.get_setting_bool('additional_properties');\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION readonly(conf jsonb DEFAULT NULL) RETURNS boolean AS $$\n SELECT pgstac.get_setting_bool('readonly', conf);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context(conf jsonb DEFAULT NULL) RETURNS text AS $$\n SELECT pgstac.get_setting('context', conf);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context_estimated_count(conf jsonb DEFAULT NULL) RETURNS int AS $$\n SELECT pgstac.get_setting('context_estimated_count', conf)::int;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_estimated_cost();\nCREATE OR REPLACE FUNCTION context_estimated_cost(conf jsonb DEFAULT NULL) RETURNS float AS $$\n SELECT pgstac.get_setting('context_estimated_cost', conf)::float;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_stats_ttl();\nCREATE OR REPLACE FUNCTION context_stats_ttl(conf jsonb DEFAULT NULL) RETURNS interval AS $$\n SELECT pgstac.get_setting('context_stats_ttl', conf)::interval;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION t2s(text) RETURNS text AS $$\n SELECT extract(epoch FROM $1::interval)::text || ' s';\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE STRICT;\n\nCREATE OR REPLACE FUNCTION age_ms(a timestamptz, b timestamptz DEFAULT clock_timestamp()) RETURNS float AS $$\n SELECT abs(extract(epoch from age(a,b)) * 1000);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION queue_timeout() RETURNS interval AS $$\n SELECT t2s(coalesce(\n get_setting('queue_timeout'),\n '1h'\n ))::interval;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION notice(VARIADIC text[]) RETURNS boolean AS $$\nDECLARE\ndebug boolean := current_setting('pgstac.debug', true);\nBEGIN\n IF debug THEN\n RAISE NOTICE 'NOTICE FROM FUNC: % >>>>> %', concat_ws(' | ', $1), clock_timestamp();\n RETURN TRUE;\n END IF;\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION empty_arr(ANYARRAY) RETURNS BOOLEAN AS $$\nSELECT CASE\n WHEN $1 IS NULL THEN TRUE\n WHEN cardinality($1)<1 THEN TRUE\nELSE FALSE\nEND;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_intersection(_a ANYARRAY, _b ANYARRAY) RETURNS ANYARRAY AS $$\n SELECT ARRAY ( SELECT unnest(_a) INTERSECT SELECT UNNEST(_b) );\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION array_map_ident(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_ident(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_map_literal(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_literal(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION array_reverse(anyarray) RETURNS anyarray AS $$\nSELECT ARRAY(\n SELECT $1[i]\n FROM generate_subscripts($1,1) AS s(i)\n ORDER BY i DESC\n);\n$$ LANGUAGE SQL STRICT IMMUTABLE;\n\nDROP TABLE IF EXISTS query_queue;\nCREATE TABLE query_queue (\n query text PRIMARY KEY,\n added timestamptz DEFAULT now()\n);\n\nDROP TABLE IF EXISTS query_queue_history;\nCREATE TABLE query_queue_history(\n query text,\n added timestamptz NOT NULL,\n finished timestamptz NOT NULL DEFAULT now(),\n error text\n);\n\nCREATE OR REPLACE PROCEDURE run_queued_queries() AS $$\nDECLARE\n qitem query_queue%ROWTYPE;\n timeout_ts timestamptz;\n error text;\n cnt int := 0;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n DELETE FROM query_queue WHERE query = (SELECT query FROM query_queue ORDER BY added DESC LIMIT 1 FOR UPDATE SKIP LOCKED) RETURNING * INTO qitem;\n IF NOT FOUND THEN\n EXIT;\n END IF;\n cnt := cnt + 1;\n BEGIN\n RAISE NOTICE 'RUNNING QUERY: %', qitem.query;\n EXECUTE qitem.query;\n EXCEPTION WHEN others THEN\n error := format('%s | %s', SQLERRM, SQLSTATE);\n END;\n INSERT INTO query_queue_history (query, added, finished, error)\n VALUES (qitem.query, qitem.added, clock_timestamp(), error);\n COMMIT;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION run_queued_queries_intransaction() RETURNS int AS $$\nDECLARE\n qitem query_queue%ROWTYPE;\n timeout_ts timestamptz;\n error text;\n cnt int := 0;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n DELETE FROM query_queue WHERE query = (SELECT query FROM query_queue ORDER BY added DESC LIMIT 1 FOR UPDATE SKIP LOCKED) RETURNING * INTO qitem;\n IF NOT FOUND THEN\n RETURN cnt;\n END IF;\n cnt := cnt + 1;\n BEGIN\n qitem.query := regexp_replace(qitem.query, 'CONCURRENTLY', '');\n RAISE NOTICE 'RUNNING QUERY: %', qitem.query;\n\n EXECUTE qitem.query;\n EXCEPTION WHEN others THEN\n error := format('%s | %s', SQLERRM, SQLSTATE);\n RAISE WARNING '%', error;\n END;\n INSERT INTO query_queue_history (query, added, finished, error)\n VALUES (qitem.query, qitem.added, clock_timestamp(), error);\n END LOOP;\n RETURN cnt;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION run_or_queue(query text) RETURNS VOID AS $$\nDECLARE\n use_queue text := COALESCE(get_setting('use_queue'), 'FALSE')::boolean;\nBEGIN\n IF get_setting_bool('debug') THEN\n RAISE NOTICE '%', query;\n END IF;\n IF use_queue THEN\n INSERT INTO query_queue (query) VALUES (query) ON CONFLICT DO NOTHING;\n ELSE\n EXECUTE query;\n END IF;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n\nDROP FUNCTION IF EXISTS check_pgstac_settings;\nCREATE OR REPLACE FUNCTION check_pgstac_settings(_sysmem text DEFAULT NULL) RETURNS VOID AS $$\nDECLARE\n settingval text;\n sysmem bigint := pg_size_bytes(_sysmem);\n effective_cache_size bigint := pg_size_bytes(current_setting('effective_cache_size', TRUE));\n shared_buffers bigint := pg_size_bytes(current_setting('shared_buffers', TRUE));\n work_mem bigint := pg_size_bytes(current_setting('work_mem', TRUE));\n max_connections int := current_setting('max_connections', TRUE);\n maintenance_work_mem bigint := pg_size_bytes(current_setting('maintenance_work_mem', TRUE));\n seq_page_cost float := current_setting('seq_page_cost', TRUE);\n random_page_cost float := current_setting('random_page_cost', TRUE);\n temp_buffers bigint := pg_size_bytes(current_setting('temp_buffers', TRUE));\n r record;\nBEGIN\n IF _sysmem IS NULL THEN\n RAISE NOTICE 'Call function with the size of your system memory `SELECT check_pgstac_settings(''4GB'')` to get pg system setting recommendations.';\n ELSE\n IF effective_cache_size < (sysmem * 0.5) THEN\n RAISE WARNING 'effective_cache_size of % is set low for a system with %. Recomended value between % and %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.5), pg_size_pretty(sysmem * 0.75);\n ELSIF effective_cache_size > (sysmem * 0.75) THEN\n RAISE WARNING 'effective_cache_size of % is set high for a system with %. Recomended value between % and %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.5), pg_size_pretty(sysmem * 0.75);\n ELSE\n RAISE NOTICE 'effective_cache_size of % is set appropriately for a system with %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem);\n END IF;\n\n IF shared_buffers < (sysmem * 0.2) THEN\n RAISE WARNING 'shared_buffers of % is set low for a system with %. Recomended value between % and %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.2), pg_size_pretty(sysmem * 0.3);\n ELSIF shared_buffers > (sysmem * 0.3) THEN\n RAISE WARNING 'shared_buffers of % is set high for a system with %. Recomended value between % and %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.2), pg_size_pretty(sysmem * 0.3);\n ELSE\n RAISE NOTICE 'shared_buffers of % is set appropriately for a system with %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem);\n END IF;\n shared_buffers = sysmem * 0.3;\n IF maintenance_work_mem < (sysmem * 0.2) THEN\n RAISE WARNING 'maintenance_work_mem of % is set low for shared_buffers of %. Recomended value between % and %', pg_size_pretty(maintenance_work_mem), pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers * 0.2), pg_size_pretty(shared_buffers * 0.3);\n ELSIF maintenance_work_mem > (shared_buffers * 0.3) THEN\n RAISE WARNING 'maintenance_work_mem of % is set high for shared_buffers of %. Recomended value between % and %', pg_size_pretty(maintenance_work_mem), pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers * 0.2), pg_size_pretty(shared_buffers * 0.3);\n ELSE\n RAISE NOTICE 'maintenance_work_mem of % is set appropriately for shared_buffers of %', pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers);\n END IF;\n\n IF work_mem * max_connections > shared_buffers THEN\n RAISE WARNING 'work_mem setting of % is set high for % max_connections please reduce work_mem to % or decrease max_connections to %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers/max_connections), floor(shared_buffers/work_mem);\n ELSIF work_mem * max_connections < (shared_buffers * 0.75) THEN\n RAISE WARNING 'work_mem setting of % is set low for % max_connections you may consider raising work_mem to % or increasing max_connections to %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers/max_connections), floor(shared_buffers/work_mem);\n ELSE\n RAISE NOTICE 'work_mem setting of % and max_connections of % are adequate for shared_buffers of %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers);\n END IF;\n\n IF random_page_cost / seq_page_cost != 1.1 THEN\n RAISE WARNING 'random_page_cost (%) /seq_page_cost (%) should be set to 1.1 for SSD. Change random_page_cost to %', random_page_cost, seq_page_cost, 1.1 * seq_page_cost;\n ELSE\n RAISE NOTICE 'random_page_cost and seq_page_cost set appropriately for SSD';\n END IF;\n\n IF temp_buffers < greatest(pg_size_bytes('128MB'),(maintenance_work_mem / 2)) THEN\n RAISE WARNING 'pgstac makes heavy use of temp tables, consider raising temp_buffers from % to %', pg_size_pretty(temp_buffers), greatest('128MB', pg_size_pretty((shared_buffers / 16)));\n END IF;\n END IF;\n\n RAISE NOTICE 'VALUES FOR PGSTAC VARIABLES';\n RAISE NOTICE 'These can be set either as GUC system variables or by setting in the pgstac_settings table.';\n\n FOR r IN SELECT name, get_setting(name) as setting, CASE WHEN current_setting(concat('pgstac.',name), TRUE) IS NOT NULL THEN concat('pgstac.',name, ' GUC') WHEN value IS NOT NULL THEN 'pgstac_settings table' ELSE 'Not Set' END as loc FROM pgstac_settings LOOP\n RAISE NOTICE '% is set to % from the %', r.name, r.setting, r.loc;\n END LOOP;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pg_cron';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider intalling pg_cron which can be used to automate tasks';\n ELSE\n RAISE NOTICE 'pg_cron % is installed', settingval;\n END IF;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pgstattuple';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider installing the pgstattuple extension which can be used to help maintain tables and indexes.';\n ELSE\n RAISE NOTICE 'pgstattuple % is installed', settingval;\n END IF;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pg_stat_statements';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider installing the pg_stat_statements extension which is very helpful for tracking the types of queries on the system';\n ELSE\n RAISE NOTICE 'pg_stat_statements % is installed', settingval;\n IF current_setting('pg_stat_statements.track_statements', TRUE) IS DISTINCT FROM 'all' THEN\n RAISE WARNING 'SET pg_stat_statements.track_statements TO ''all''; --In order to track statements within functions.';\n END IF;\n END IF;\n\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac, public SET CLIENT_MIN_MESSAGES TO NOTICE;\nCREATE OR REPLACE FUNCTION to_int(jsonb) RETURNS int AS $$\n SELECT floor(($1->>0)::float)::int;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION to_float(jsonb) RETURNS float AS $$\n SELECT ($1->>0)::float;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION to_tstz(jsonb) RETURNS timestamptz AS $$\n SELECT ($1->>0)::timestamptz;\n$$ LANGUAGE SQL IMMUTABLE STRICT SET TIME ZONE 'UTC' COST 5000 PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION to_text(jsonb) RETURNS text AS $$\n SELECT CASE WHEN jsonb_typeof($1) IN ('array','object') THEN $1::text ELSE $1->>0 END;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION to_text_array(jsonb) RETURNS text[] AS $$\n SELECT\n CASE jsonb_typeof($1)\n WHEN 'array' THEN ARRAY(SELECT jsonb_array_elements_text($1))\n ELSE ARRAY[$1->>0]\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION bbox_geom(_bbox jsonb) RETURNS geometry AS $$\nSELECT CASE jsonb_array_length(_bbox)\n WHEN 4 THEN\n ST_SetSRID(ST_MakeEnvelope(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float,\n (_bbox->>3)::float\n ),4326)\n WHEN 6 THEN\n ST_SetSRID(ST_3DMakeBox(\n ST_MakePoint(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float\n ),\n ST_MakePoint(\n (_bbox->>3)::float,\n (_bbox->>4)::float,\n (_bbox->>5)::float\n )\n ),4326)\n ELSE null END;\n;\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION geom_bbox(_geom geometry) RETURNS jsonb AS $$\n SELECT jsonb_build_array(\n st_xmin(_geom),\n st_ymin(_geom),\n st_xmax(_geom),\n st_ymax(_geom)\n );\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION flip_jsonb_array(j jsonb) RETURNS jsonb AS $$\n SELECT jsonb_agg(value) FROM (SELECT value FROM jsonb_array_elements(j) WITH ORDINALITY ORDER BY ordinality DESC) as t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION explode_dotpaths(j jsonb) RETURNS SETOF text[] AS $$\n SELECT string_to_array(p, '.') as e FROM jsonb_array_elements_text(j) p;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION explode_dotpaths_recurse(IN j jsonb) RETURNS SETOF text[] AS $$\n WITH RECURSIVE t AS (\n SELECT e FROM explode_dotpaths(j) e\n UNION ALL\n SELECT e[1:cardinality(e)-1]\n FROM t\n WHERE cardinality(e)>1\n ) SELECT e FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION jsonb_set_nested(j jsonb, path text[], val jsonb) RETURNS jsonb AS $$\nDECLARE\nBEGIN\n IF cardinality(path) > 1 THEN\n FOR i IN 1..(cardinality(path)-1) LOOP\n IF j #> path[:i] IS NULL THEN\n j := jsonb_set_lax(j, path[:i], '{}', TRUE);\n END IF;\n END LOOP;\n END IF;\n RETURN jsonb_set_lax(j, path, val, true);\n\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\n\n\nCREATE OR REPLACE FUNCTION jsonb_include(j jsonb, f jsonb) RETURNS jsonb AS $$\nDECLARE\n includes jsonb := f-> 'include';\n outj jsonb := '{}'::jsonb;\n path text[];\nBEGIN\n IF\n includes IS NULL\n OR jsonb_array_length(includes) = 0\n THEN\n RETURN j;\n ELSE\n includes := includes || '[\"id\",\"collection\"]'::jsonb;\n FOR path IN SELECT explode_dotpaths(includes) LOOP\n outj := jsonb_set_nested(outj, path, j #> path);\n END LOOP;\n END IF;\n RETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_exclude(j jsonb, f jsonb) RETURNS jsonb AS $$\nDECLARE\n excludes jsonb := f-> 'exclude';\n outj jsonb := j;\n path text[];\nBEGIN\n IF\n excludes IS NULL\n OR jsonb_array_length(excludes) = 0\n THEN\n RETURN j;\n ELSE\n FOR path IN SELECT explode_dotpaths(excludes) LOOP\n outj := outj #- path;\n END LOOP;\n END IF;\n RETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_fields(j jsonb, f jsonb DEFAULT '{\"fields\":[]}') RETURNS jsonb AS $$\n SELECT jsonb_exclude(jsonb_include(j, f), f);\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION merge_jsonb(_a jsonb, _b jsonb) RETURNS jsonb AS $$\n SELECT\n CASE\n WHEN _a = '\"𒍟※\"'::jsonb THEN NULL\n WHEN _a IS NULL OR jsonb_typeof(_a) = 'null' THEN _b\n WHEN jsonb_typeof(_a) = 'object' AND jsonb_typeof(_b) = 'object' THEN\n (\n SELECT\n jsonb_strip_nulls(\n jsonb_object_agg(\n key,\n merge_jsonb(a.value, b.value)\n )\n )\n FROM\n jsonb_each(coalesce(_a,'{}'::jsonb)) as a\n FULL JOIN\n jsonb_each(coalesce(_b,'{}'::jsonb)) as b\n USING (key)\n )\n WHEN\n jsonb_typeof(_a) = 'array'\n AND jsonb_typeof(_b) = 'array'\n AND jsonb_array_length(_a) = jsonb_array_length(_b)\n THEN\n (\n SELECT jsonb_agg(m) FROM\n ( SELECT\n merge_jsonb(\n jsonb_array_elements(_a),\n jsonb_array_elements(_b)\n ) as m\n ) as l\n )\n ELSE _a\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION strip_jsonb(_a jsonb, _b jsonb) RETURNS jsonb AS $$\n SELECT\n CASE\n\n WHEN (_a IS NULL OR jsonb_typeof(_a) = 'null') AND _b IS NOT NULL AND jsonb_typeof(_b) != 'null' THEN '\"𒍟※\"'::jsonb\n WHEN _b IS NULL OR jsonb_typeof(_a) = 'null' THEN _a\n WHEN _a = _b AND jsonb_typeof(_a) = 'object' THEN '{}'::jsonb\n WHEN _a = _b THEN NULL\n WHEN jsonb_typeof(_a) = 'object' AND jsonb_typeof(_b) = 'object' THEN\n (\n SELECT\n jsonb_strip_nulls(\n jsonb_object_agg(\n key,\n strip_jsonb(a.value, b.value)\n )\n )\n FROM\n jsonb_each(_a) as a\n FULL JOIN\n jsonb_each(_b) as b\n USING (key)\n )\n WHEN\n jsonb_typeof(_a) = 'array'\n AND jsonb_typeof(_b) = 'array'\n AND jsonb_array_length(_a) = jsonb_array_length(_b)\n THEN\n (\n SELECT jsonb_agg(m) FROM\n ( SELECT\n strip_jsonb(\n jsonb_array_elements(_a),\n jsonb_array_elements(_b)\n ) as m\n ) as l\n )\n ELSE _a\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION nullif_jsonbnullempty(j jsonb) RETURNS jsonb AS $$\n SELECT nullif(nullif(nullif(j,'null'::jsonb),'{}'::jsonb),'[]'::jsonb);\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION jsonb_array_unique(j jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(jsonb_agg(DISTINCT a)) v FROM jsonb_array_elements(j) a;\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_concat_ignorenull(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT coalesce(a,'[]'::jsonb) || coalesce(b,'[]'::jsonb);\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_least(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(least(nullif_jsonbnullempty(a), nullif_jsonbnullempty(b)));\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_greatest(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(greatest(a, b));\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION first_notnull_sfunc(anyelement, anyelement) RETURNS anyelement AS $$\n SELECT COALESCE($1,$2);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE AGGREGATE first_notnull(anyelement)(\n SFUNC = first_notnull_sfunc,\n STYPE = anyelement\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_array_unique_merge(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_concat_ignorenull,\n FINALFUNC = jsonb_array_unique\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_min(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_least\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_max(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_greatest\n);\n/* looks for a geometry in a stac item first from geometry and falling back to bbox */\nCREATE OR REPLACE FUNCTION stac_geom(value jsonb) RETURNS geometry AS $$\nSELECT\n CASE\n WHEN value ? 'intersects' THEN\n ST_GeomFromGeoJSON(value->>'intersects')\n WHEN value ? 'geometry' THEN\n ST_GeomFromGeoJSON(value->>'geometry')\n WHEN value ? 'bbox' THEN\n pgstac.bbox_geom(value->'bbox')\n ELSE NULL\n END as geometry\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION stac_daterange(\n value jsonb\n) RETURNS tstzrange AS $$\nDECLARE\n props jsonb := value;\n dt timestamptz;\n edt timestamptz;\nBEGIN\n IF props ? 'properties' THEN\n props := props->'properties';\n END IF;\n IF\n props ? 'start_datetime'\n AND props->>'start_datetime' IS NOT NULL\n AND props ? 'end_datetime'\n AND props->>'end_datetime' IS NOT NULL\n THEN\n dt := props->>'start_datetime';\n edt := props->>'end_datetime';\n IF dt > edt THEN\n RAISE EXCEPTION 'start_datetime must be < end_datetime';\n END IF;\n ELSE\n dt := props->>'datetime';\n edt := props->>'datetime';\n END IF;\n IF dt is NULL OR edt IS NULL THEN\n RAISE NOTICE 'DT: %, EDT: %', dt, edt;\n RAISE EXCEPTION 'Either datetime (%) or both start_datetime (%) and end_datetime (%) must be set.', props->>'datetime',props->>'start_datetime',props->>'end_datetime';\n END IF;\n RETURN tstzrange(dt, edt, '[]');\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_datetime(value jsonb) RETURNS timestamptz AS $$\n SELECT lower(stac_daterange(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_end_datetime(value jsonb) RETURNS timestamptz AS $$\n SELECT upper(stac_daterange(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE TABLE IF NOT EXISTS stac_extensions(\n url text PRIMARY KEY,\n content jsonb\n);\nCREATE OR REPLACE FUNCTION collection_base_item(content jsonb) RETURNS jsonb AS $$\n SELECT jsonb_build_object(\n 'type', 'Feature',\n 'stac_version', content->'stac_version',\n 'assets', content->'item_assets',\n 'collection', content->'id'\n );\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE TABLE IF NOT EXISTS collections (\n key bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,\n id text GENERATED ALWAYS AS (content->>'id') STORED UNIQUE NOT NULL,\n content JSONB NOT NULL,\n base_item jsonb GENERATED ALWAYS AS (pgstac.collection_base_item(content)) STORED,\n geometry geometry GENERATED ALWAYS AS (pgstac.collection_geom(content)) STORED,\n datetime timestamptz GENERATED ALWAYS AS (pgstac.collection_datetime(content)) STORED,\n end_datetime timestamptz GENERATED ALWAYS AS (pgstac.collection_enddatetime(content)) STORED,\n private jsonb,\n partition_trunc text CHECK (partition_trunc IN ('year', 'month'))\n);\n\nCREATE OR REPLACE FUNCTION collection_base_item(cid text) RETURNS jsonb AS $$\n SELECT pgstac.collection_base_item(content) FROM pgstac.collections WHERE id = cid LIMIT 1;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION create_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION update_collection(data jsonb) RETURNS VOID AS $$\nDECLARE\n out collections%ROWTYPE;\nBEGIN\n UPDATE collections SET content=data WHERE id = data->>'id' RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ON CONFLICT (id) DO\n UPDATE\n SET content=EXCLUDED.content\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION delete_collection(_id text) RETURNS VOID AS $$\nDECLARE\n out collections%ROWTYPE;\nBEGIN\n DELETE FROM collections WHERE id = _id RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION get_collection(id text) RETURNS jsonb AS $$\n SELECT content FROM collections\n WHERE id=$1\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION all_collections() RETURNS jsonb AS $$\n SELECT coalesce(jsonb_agg(content), '[]'::jsonb) FROM collections;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION collection_delete_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n collection_base_partition text := concat('_items_', OLD.key);\nBEGIN\n EXECUTE format($q$\n DELETE FROM partition_stats WHERE partition IN (\n SELECT partition FROM partition_sys_meta\n WHERE collection=%L\n );\n DROP TABLE IF EXISTS %I CASCADE;\n $q$,\n OLD.id,\n collection_base_partition\n );\n RETURN OLD;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER collection_delete_trigger BEFORE DELETE ON collections\nFOR EACH ROW EXECUTE FUNCTION collection_delete_trigger_func();\nCREATE OR REPLACE FUNCTION queryable_signature(n text, c text[]) RETURNS text AS $$\n SELECT concat(n, c);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TABLE queryables (\n id bigint GENERATED ALWAYS AS identity PRIMARY KEY,\n name text NOT NULL,\n collection_ids text[], -- used to determine what partitions to create indexes on\n definition jsonb,\n property_path text,\n property_wrapper text,\n property_index_type text\n);\nCREATE INDEX queryables_name_idx ON queryables (name);\nCREATE INDEX queryables_collection_idx ON queryables USING GIN (collection_ids);\nCREATE INDEX queryables_property_wrapper_idx ON queryables (property_wrapper);\n\nCREATE OR REPLACE FUNCTION pgstac.queryables_constraint_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n allcollections text[];\nBEGIN\n RAISE NOTICE 'Making sure that name/collection is unique for queryables %', NEW;\n IF NEW.collection_ids IS NOT NULL THEN\n IF EXISTS (\n SELECT 1\n FROM unnest(NEW.collection_ids) c\n LEFT JOIN\n collections\n ON (collections.id = c)\n WHERE collections.id IS NULL\n ) THEN\n RAISE foreign_key_violation USING MESSAGE = format(\n 'One or more collections in %s do not exist.', NEW.collection_ids\n );\n RETURN NULL;\n END IF;\n END IF;\n IF TG_OP = 'INSERT' THEN\n IF EXISTS (\n SELECT 1 FROM queryables q\n WHERE\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n )\n ) THEN\n RAISE unique_violation USING MESSAGE = format(\n 'There is already a queryable for %s for a collection in %s: %s',\n NEW.name,\n NEW.collection_ids,\n\t\t\t\t(SELECT json_agg(row_to_json(q)) FROM queryables q WHERE\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n ))\n );\n RETURN NULL;\n END IF;\n END IF;\n IF TG_OP = 'UPDATE' THEN\n IF EXISTS (\n SELECT 1 FROM queryables q\n WHERE\n q.id != NEW.id\n AND\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n )\n ) THEN\n RAISE unique_violation\n USING MESSAGE = format(\n 'There is already a queryable for %s for a collection in %s',\n NEW.name,\n NEW.collection_ids\n );\n RETURN NULL;\n END IF;\n END IF;\n\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER queryables_constraint_insert_trigger\nBEFORE INSERT ON queryables\nFOR EACH ROW EXECUTE PROCEDURE queryables_constraint_triggerfunc();\n\nCREATE TRIGGER queryables_constraint_update_trigger\nBEFORE UPDATE ON queryables\nFOR EACH ROW\nWHEN (NEW.name = OLD.name AND NEW.collection_ids IS DISTINCT FROM OLD.collection_ids)\nEXECUTE PROCEDURE queryables_constraint_triggerfunc();\n\n\nCREATE OR REPLACE FUNCTION array_to_path(arr text[]) RETURNS text AS $$\n SELECT string_agg(\n quote_literal(v),\n '->'\n ) FROM unnest(arr) v;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\n\n\nCREATE OR REPLACE FUNCTION queryable(\n IN dotpath text,\n OUT path text,\n OUT expression text,\n OUT wrapper text,\n OUT nulled_wrapper text\n) AS $$\nDECLARE\n q RECORD;\n path_elements text[];\nBEGIN\n IF dotpath IN ('id', 'geometry', 'datetime', 'end_datetime', 'collection') THEN\n path := dotpath;\n expression := dotpath;\n wrapper := NULL;\n RETURN;\n END IF;\n SELECT * INTO q FROM queryables\n WHERE\n name=dotpath\n OR name = 'properties.' || dotpath\n OR name = replace(dotpath, 'properties.', '')\n ;\n IF q.property_wrapper IS NULL THEN\n IF q.definition->>'type' = 'number' THEN\n wrapper := 'to_float';\n nulled_wrapper := wrapper;\n ELSIF q.definition->>'format' = 'date-time' THEN\n wrapper := 'to_tstz';\n nulled_wrapper := wrapper;\n ELSE\n nulled_wrapper := NULL;\n wrapper := 'to_text';\n END IF;\n ELSE\n wrapper := q.property_wrapper;\n nulled_wrapper := wrapper;\n END IF;\n IF q.property_path IS NOT NULL THEN\n path := q.property_path;\n ELSE\n path_elements := string_to_array(dotpath, '.');\n IF path_elements[1] IN ('links', 'assets', 'stac_version', 'stac_extensions') THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSIF path_elements[1] = 'properties' THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSE\n path := format($F$content->'properties'->%s$F$, array_to_path(path_elements));\n END IF;\n END IF;\n expression := format('%I(%s)', wrapper, path);\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\nCREATE OR REPLACE FUNCTION unnest_collection(collection_ids text[] DEFAULT NULL) RETURNS SETOF text AS $$\n DECLARE\n BEGIN\n IF collection_ids IS NULL THEN\n RETURN QUERY SELECT id FROM collections;\n END IF;\n RETURN QUERY SELECT unnest(collection_ids);\n END;\n$$ LANGUAGE PLPGSQL STABLE;\n\nCREATE OR REPLACE FUNCTION normalize_indexdef(def text) RETURNS text AS $$\nDECLARE\nBEGIN\n def := btrim(def, ' \\n\\t');\n\tdef := regexp_replace(def, '^CREATE (UNIQUE )?INDEX ([^ ]* )?ON (ONLY )?([^ ]* )?', '', 'i');\n RETURN def;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION indexdef(q queryables) RETURNS text AS $$\n DECLARE\n out text;\n BEGIN\n IF q.name = 'id' THEN\n out := 'CREATE UNIQUE INDEX ON %I USING btree (id)';\n ELSIF q.name = 'datetime' THEN\n out := 'CREATE INDEX ON %I USING btree (datetime DESC, end_datetime)';\n ELSIF q.name = 'geometry' THEN\n out := 'CREATE INDEX ON %I USING gist (geometry)';\n ELSE\n out := format($q$CREATE INDEX ON %%I USING %s (%s(((content -> 'properties'::text) -> %L::text)))$q$,\n lower(COALESCE(q.property_index_type, 'BTREE')),\n lower(COALESCE(q.property_wrapper, 'to_text')),\n q.name\n );\n END IF;\n RETURN btrim(out, ' \\n\\t');\n END;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nDROP VIEW IF EXISTS pgstac_indexes;\nCREATE VIEW pgstac_indexes AS\nSELECT\n i.schemaname,\n i.tablename,\n i.indexname,\n regexp_replace(btrim(replace(replace(indexdef, i.indexname, ''),'pgstac.',''),' \\t\\n'), '[ ]+', ' ', 'g') as idx,\n COALESCE(\n (regexp_match(indexdef, '\\(([a-zA-Z]+)\\)'))[1],\n (regexp_match(indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_-]+)''::text'))[1],\n CASE WHEN indexdef ~* '\\(datetime desc, end_datetime\\)' THEN 'datetime' ELSE NULL END\n ) AS field,\n pg_table_size(i.indexname::text) as index_size,\n pg_size_pretty(pg_table_size(i.indexname::text)) as index_size_pretty\nFROM\n pg_indexes i\nWHERE i.schemaname='pgstac' and i.tablename ~ '_items_' AND indexdef !~* ' only ';\n\nDROP VIEW IF EXISTS pgstac_index_stats;\nCREATE VIEW pgstac_indexes_stats AS\nSELECT\n i.schemaname,\n i.tablename,\n i.indexname,\n indexdef,\n COALESCE(\n (regexp_match(indexdef, '\\(([a-zA-Z]+)\\)'))[1],\n (regexp_match(indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_]+)''::text'))[1],\n CASE WHEN indexdef ~* '\\(datetime desc, end_datetime\\)' THEN 'datetime_end_datetime' ELSE NULL END\n ) AS field,\n pg_table_size(i.indexname::text) as index_size,\n pg_size_pretty(pg_table_size(i.indexname::text)) as index_size_pretty,\n n_distinct,\n most_common_vals::text::text[],\n most_common_freqs::text::text[],\n histogram_bounds::text::text[],\n correlation\nFROM\n pg_indexes i\n LEFT JOIN pg_stats s ON (s.tablename = i.indexname)\nWHERE i.schemaname='pgstac' and i.tablename ~ '_items_';\n\nCREATE OR REPLACE FUNCTION queryable_indexes(\n IN treeroot text DEFAULT 'items',\n IN changes boolean DEFAULT FALSE,\n OUT collection text,\n OUT partition text,\n OUT field text,\n OUT indexname text,\n OUT existing_idx text,\n OUT queryable_idx text\n) RETURNS SETOF RECORD AS $$\nWITH p AS (\n SELECT\n relid::text as partition,\n replace(replace(\n CASE\n WHEN parentrelid::regclass::text='items' THEN pg_get_expr(c.relpartbound, c.oid)\n ELSE pg_get_expr(parent.relpartbound, parent.oid)\n END,\n 'FOR VALUES IN (''',''), ''')',\n ''\n ) AS collection\n FROM pg_partition_tree(treeroot)\n JOIN pg_class c ON (relid::regclass = c.oid)\n JOIN pg_class parent ON (parentrelid::regclass = parent.oid AND isleaf)\n ), i AS (\n SELECT\n partition,\n indexname,\n regexp_replace(btrim(replace(replace(indexdef, indexname, ''),'pgstac.',''),' \\t\\n'), '[ ]+', ' ', 'g') as iidx,\n COALESCE(\n (regexp_match(indexdef, '\\(([a-zA-Z]+)\\)'))[1],\n (regexp_match(indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_-]+)''::text'))[1],\n CASE WHEN indexdef ~* '\\(datetime desc, end_datetime\\)' THEN 'datetime' ELSE NULL END\n ) AS field\n FROM\n pg_indexes\n JOIN p ON (tablename=partition)\n ), q AS (\n SELECT\n name AS field,\n collection,\n partition,\n format(indexdef(queryables), partition) as qidx\n FROM queryables, unnest_collection(queryables.collection_ids) collection\n JOIN p USING (collection)\n WHERE property_index_type IS NOT NULL OR name IN ('datetime','geometry','id')\n )\n SELECT\n collection,\n partition,\n field,\n indexname,\n iidx as existing_idx,\n qidx as queryable_idx\n FROM i FULL JOIN q USING (field, partition)\n WHERE CASE WHEN changes THEN lower(iidx) IS DISTINCT FROM lower(qidx) ELSE TRUE END;\n;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION maintain_index(\n indexname text,\n queryable_idx text,\n dropindexes boolean DEFAULT FALSE,\n rebuildindexes boolean DEFAULT FALSE,\n idxconcurrently boolean DEFAULT FALSE\n) RETURNS VOID AS $$\nDECLARE\nBEGIN\n IF indexname IS NOT NULL THEN\n IF dropindexes OR queryable_idx IS NOT NULL THEN\n EXECUTE format('DROP INDEX IF EXISTS %I;', indexname);\n ELSIF rebuildindexes THEN\n IF idxconcurrently THEN\n EXECUTE format('REINDEX INDEX CONCURRENTLY %I;', indexname);\n ELSE\n EXECUTE format('REINDEX INDEX CONCURRENTLY %I;', indexname);\n END IF;\n END IF;\n END IF;\n IF queryable_idx IS NOT NULL THEN\n IF idxconcurrently THEN\n EXECUTE replace(queryable_idx, 'INDEX', 'INDEX CONCURRENTLY');\n ELSE EXECUTE queryable_idx;\n END IF;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\n\nset check_function_bodies to off;\nCREATE OR REPLACE FUNCTION maintain_partition_queries(\n part text DEFAULT 'items',\n dropindexes boolean DEFAULT FALSE,\n rebuildindexes boolean DEFAULT FALSE,\n idxconcurrently boolean DEFAULT FALSE\n) RETURNS SETOF text AS $$\nDECLARE\n rec record;\n q text;\nBEGIN\n FOR rec IN (\n SELECT * FROM queryable_indexes(part,true)\n ) LOOP\n q := format(\n 'SELECT maintain_index(\n %L,%L,%L,%L,%L\n );',\n rec.indexname,\n rec.queryable_idx,\n dropindexes,\n rebuildindexes,\n idxconcurrently\n );\n RAISE NOTICE 'Q: %s', q;\n RETURN NEXT q;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION maintain_partitions(\n part text DEFAULT 'items',\n dropindexes boolean DEFAULT FALSE,\n rebuildindexes boolean DEFAULT FALSE\n) RETURNS VOID AS $$\n WITH t AS (\n SELECT run_or_queue(q) FROM maintain_partition_queries(part, dropindexes, rebuildindexes) q\n ) SELECT count(*) FROM t;\n$$ LANGUAGE SQL;\n\n\nCREATE OR REPLACE FUNCTION queryables_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\nBEGIN\n PERFORM maintain_partitions();\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER queryables_trigger AFTER INSERT OR UPDATE ON queryables\nFOR EACH STATEMENT EXECUTE PROCEDURE queryables_trigger_func();\n\n\nCREATE OR REPLACE FUNCTION get_queryables(_collection_ids text[] DEFAULT NULL) RETURNS jsonb AS $$\nDECLARE\nBEGIN\n -- Build up queryables if the input contains valid collection ids or is empty\n IF EXISTS (\n SELECT 1 FROM collections\n WHERE\n _collection_ids IS NULL\n OR cardinality(_collection_ids) = 0\n OR id = ANY(_collection_ids)\n )\n THEN\n RETURN (\n WITH base AS (\n SELECT\n unnest(collection_ids) as collection_id,\n name,\n coalesce(definition, '{\"type\":\"string\"}'::jsonb) as definition\n FROM queryables\n WHERE\n _collection_ids IS NULL OR\n _collection_ids = '{}'::text[] OR\n _collection_ids && collection_ids\n UNION ALL\n SELECT null, name, coalesce(definition, '{\"type\":\"string\"}'::jsonb) as definition\n FROM queryables WHERE collection_ids IS NULL OR collection_ids = '{}'::text[]\n ), g AS (\n SELECT\n name,\n first_notnull(definition) as definition,\n jsonb_array_unique_merge(definition->'enum') as enum,\n jsonb_min(definition->'minimum') as minimum,\n jsonb_min(definition->'maxiumn') as maximum\n FROM base\n GROUP BY 1\n )\n SELECT\n jsonb_build_object(\n '$schema', 'http://json-schema.org/draft-07/schema#',\n '$id', '',\n 'type', 'object',\n 'title', 'STAC Queryables.',\n 'properties', jsonb_object_agg(\n name,\n definition\n ||\n jsonb_strip_nulls(jsonb_build_object(\n 'enum', enum,\n 'minimum', minimum,\n 'maximum', maximum\n ))\n ),\n 'additionalProperties', pgstac.additional_properties()\n )\n FROM g\n );\n ELSE\n RETURN NULL;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\nCREATE OR REPLACE FUNCTION get_queryables(_collection text DEFAULT NULL) RETURNS jsonb AS $$\n SELECT\n CASE\n WHEN _collection IS NULL THEN get_queryables(NULL::text[])\n ELSE get_queryables(ARRAY[_collection])\n END\n ;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_queryables() RETURNS jsonb AS $$\n SELECT get_queryables(NULL::text[]);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION schema_qualify_refs(url text, j jsonb) returns jsonb as $$\n SELECT regexp_replace(j::text, '\"\\$ref\": \"#', concat('\"$ref\": \"', url, '#'), 'g')::jsonb;\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE VIEW stac_extension_queryables AS\nSELECT DISTINCT key as name, schema_qualify_refs(e.url, j.value) as definition FROM stac_extensions e, jsonb_each(e.content->'definitions'->'fields'->'properties') j;\n\n\nCREATE OR REPLACE FUNCTION missing_queryables(_collection text, _tablesample float DEFAULT 5, minrows float DEFAULT 10) RETURNS TABLE(collection text, name text, definition jsonb, property_wrapper text) AS $$\nDECLARE\n q text;\n _partition text;\n explain_json json;\n psize float;\n estrows float;\nBEGIN\n SELECT format('_items_%s', key) INTO _partition FROM collections WHERE id=_collection;\n\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM %I;', _partition)\n INTO explain_json;\n psize := explain_json->0->'Plan'->'Plan Rows';\n estrows := _tablesample * .01 * psize;\n IF estrows < minrows THEN\n _tablesample := least(100,greatest(_tablesample, (estrows / psize) / 100));\n RAISE NOTICE '%', (psize / estrows) / 100;\n END IF;\n RAISE NOTICE 'Using tablesample % to find missing queryables from % % that has ~% rows estrows: %', _tablesample, _collection, _partition, psize, estrows;\n\n q := format(\n $q$\n WITH q AS (\n SELECT * FROM queryables\n WHERE\n collection_ids IS NULL\n OR %L = ANY(collection_ids)\n ), t AS (\n SELECT\n content->'properties' AS properties\n FROM\n %I\n TABLESAMPLE SYSTEM(%L)\n ), p AS (\n SELECT DISTINCT ON (key)\n key,\n value,\n s.definition\n FROM t\n JOIN LATERAL jsonb_each(properties) ON TRUE\n LEFT JOIN q ON (q.name=key)\n LEFT JOIN stac_extension_queryables s ON (s.name=key)\n WHERE q.definition IS NULL\n )\n SELECT\n %L,\n key,\n COALESCE(definition, jsonb_build_object('type',jsonb_typeof(value))) as definition,\n CASE\n WHEN definition->>'type' = 'integer' THEN 'to_int'\n WHEN COALESCE(definition->>'type', jsonb_typeof(value)) = 'number' THEN 'to_float'\n WHEN COALESCE(definition->>'type', jsonb_typeof(value)) = 'array' THEN 'to_text_array'\n ELSE 'to_text'\n END\n FROM p;\n $q$,\n _collection,\n _partition,\n _tablesample,\n _collection\n );\n RETURN QUERY EXECUTE q;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION missing_queryables(_tablesample float DEFAULT 5) RETURNS TABLE(collection_ids text[], name text, definition jsonb, property_wrapper text) AS $$\n SELECT\n array_agg(collection),\n name,\n definition,\n property_wrapper\n FROM\n collections\n JOIN LATERAL\n missing_queryables(id, _tablesample) c\n ON TRUE\n GROUP BY\n 2,3,4\n ORDER BY 2,1\n ;\n$$ LANGUAGE SQL;\nCREATE OR REPLACE FUNCTION parse_dtrange(\n _indate jsonb,\n relative_base timestamptz DEFAULT date_trunc('hour', CURRENT_TIMESTAMP)\n) RETURNS tstzrange AS $$\nDECLARE\n timestrs text[];\n s timestamptz;\n e timestamptz;\nBEGIN\n timestrs :=\n CASE\n WHEN _indate ? 'timestamp' THEN\n ARRAY[_indate->>'timestamp']\n WHEN _indate ? 'interval' THEN\n to_text_array(_indate->'interval')\n WHEN jsonb_typeof(_indate) = 'array' THEN\n to_text_array(_indate)\n ELSE\n regexp_split_to_array(\n _indate->>0,\n '/'\n )\n END;\n RAISE NOTICE 'TIMESTRS %', timestrs;\n IF cardinality(timestrs) = 1 THEN\n IF timestrs[1] ILIKE 'P%' THEN\n RETURN tstzrange(relative_base - upper(timestrs[1])::interval, relative_base, '[)');\n END IF;\n s := timestrs[1]::timestamptz;\n RETURN tstzrange(s, s, '[]');\n END IF;\n\n IF cardinality(timestrs) != 2 THEN\n RAISE EXCEPTION 'Timestamp cannot have more than 2 values';\n END IF;\n\n IF timestrs[1] = '..' OR timestrs[1] = '' THEN\n s := '-infinity'::timestamptz;\n e := timestrs[2]::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] = '..' OR timestrs[2] = '' THEN\n s := timestrs[1]::timestamptz;\n e := 'infinity'::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[1] ILIKE 'P%' AND timestrs[2] NOT ILIKE 'P%' THEN\n e := timestrs[2]::timestamptz;\n s := e - upper(timestrs[1])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] ILIKE 'P%' AND timestrs[1] NOT ILIKE 'P%' THEN\n s := timestrs[1]::timestamptz;\n e := s + upper(timestrs[2])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n s := timestrs[1]::timestamptz;\n e := timestrs[2]::timestamptz;\n\n RETURN tstzrange(s,e,'[)');\n\n RETURN NULL;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT PARALLEL SAFE SET TIME ZONE 'UTC';\n\nCREATE OR REPLACE FUNCTION parse_dtrange(\n _indate text,\n relative_base timestamptz DEFAULT CURRENT_TIMESTAMP\n) RETURNS tstzrange AS $$\n SELECT parse_dtrange(to_jsonb(_indate), relative_base);\n$$ LANGUAGE SQL STABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION temporal_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\n ll text := 'datetime';\n lh text := 'end_datetime';\n rrange tstzrange;\n rl text;\n rh text;\n outq text;\nBEGIN\n rrange := parse_dtrange(args->1);\n RAISE NOTICE 'Constructing temporal query OP: %, ARGS: %, RRANGE: %', op, args, rrange;\n op := lower(op);\n rl := format('%L::timestamptz', lower(rrange));\n rh := format('%L::timestamptz', upper(rrange));\n outq := CASE op\n WHEN 't_before' THEN 'lh < rl'\n WHEN 't_after' THEN 'll > rh'\n WHEN 't_meets' THEN 'lh = rl'\n WHEN 't_metby' THEN 'll = rh'\n WHEN 't_overlaps' THEN 'll < rl AND rl < lh < rh'\n WHEN 't_overlappedby' THEN 'rl < ll < rh AND lh > rh'\n WHEN 't_starts' THEN 'll = rl AND lh < rh'\n WHEN 't_startedby' THEN 'll = rl AND lh > rh'\n WHEN 't_during' THEN 'll > rl AND lh < rh'\n WHEN 't_contains' THEN 'll < rl AND lh > rh'\n WHEN 't_finishes' THEN 'll > rl AND lh = rh'\n WHEN 't_finishedby' THEN 'll < rl AND lh = rh'\n WHEN 't_equals' THEN 'll = rl AND lh = rh'\n WHEN 't_disjoint' THEN 'NOT (ll <= rh AND lh >= rl)'\n WHEN 't_intersects' THEN 'll <= rh AND lh >= rl'\n WHEN 'anyinteracts' THEN 'll <= rh AND lh >= rl'\n END;\n outq := regexp_replace(outq, '\\mll\\M', ll);\n outq := regexp_replace(outq, '\\mlh\\M', lh);\n outq := regexp_replace(outq, '\\mrl\\M', rl);\n outq := regexp_replace(outq, '\\mrh\\M', rh);\n outq := format('(%s)', outq);\n RETURN outq;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\n\n\nCREATE OR REPLACE FUNCTION spatial_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\n geom text;\n j jsonb := args->1;\nBEGIN\n op := lower(op);\n RAISE NOTICE 'Constructing spatial query OP: %, ARGS: %', op, args;\n IF op NOT IN ('s_equals','s_disjoint','s_touches','s_within','s_overlaps','s_crosses','s_intersects','intersects','s_contains') THEN\n RAISE EXCEPTION 'Spatial Operator % Not Supported', op;\n END IF;\n op := regexp_replace(op, '^s_', 'st_');\n IF op = 'intersects' THEN\n op := 'st_intersects';\n END IF;\n -- Convert geometry to WKB string\n IF j ? 'type' AND j ? 'coordinates' THEN\n geom := st_geomfromgeojson(j)::text;\n ELSIF jsonb_typeof(j) = 'array' THEN\n geom := bbox_geom(j)::text;\n END IF;\n\n RETURN format('%s(geometry, %L::geometry)', op, geom);\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION query_to_cql2(q jsonb) RETURNS jsonb AS $$\n-- Translates anything passed in through the deprecated \"query\" into equivalent CQL2\nWITH t AS (\n SELECT key as property, value as ops\n FROM jsonb_each(q)\n), t2 AS (\n SELECT property, (jsonb_each(ops)).*\n FROM t WHERE jsonb_typeof(ops) = 'object'\n UNION ALL\n SELECT property, 'eq', ops\n FROM t WHERE jsonb_typeof(ops) != 'object'\n)\nSELECT\n jsonb_strip_nulls(jsonb_build_object(\n 'op', 'and',\n 'args', jsonb_agg(\n jsonb_build_object(\n 'op', key,\n 'args', jsonb_build_array(\n jsonb_build_object('property',property),\n value\n )\n )\n )\n )\n) as qcql FROM t2\n;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\nCREATE OR REPLACE FUNCTION cql1_to_cql2(j jsonb) RETURNS jsonb AS $$\nDECLARE\n args jsonb;\n ret jsonb;\nBEGIN\n RAISE NOTICE 'CQL1_TO_CQL2: %', j;\n IF j ? 'filter' THEN\n RETURN cql1_to_cql2(j->'filter');\n END IF;\n IF j ? 'property' THEN\n RETURN j;\n END IF;\n IF jsonb_typeof(j) = 'array' THEN\n SELECT jsonb_agg(cql1_to_cql2(el)) INTO args FROM jsonb_array_elements(j) el;\n RETURN args;\n END IF;\n IF jsonb_typeof(j) = 'number' THEN\n RETURN j;\n END IF;\n IF jsonb_typeof(j) = 'string' THEN\n RETURN j;\n END IF;\n\n IF jsonb_typeof(j) = 'object' THEN\n SELECT jsonb_build_object(\n 'op', key,\n 'args', cql1_to_cql2(value)\n ) INTO ret\n FROM jsonb_each(j)\n WHERE j IS NOT NULL;\n RETURN ret;\n END IF;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE STRICT;\n\nCREATE TABLE cql2_ops (\n op text PRIMARY KEY,\n template text,\n types text[]\n);\nINSERT INTO cql2_ops (op, template, types) VALUES\n ('eq', '%s = %s', NULL),\n ('neq', '%s != %s', NULL),\n ('ne', '%s != %s', NULL),\n ('!=', '%s != %s', NULL),\n ('<>', '%s != %s', NULL),\n ('lt', '%s < %s', NULL),\n ('lte', '%s <= %s', NULL),\n ('gt', '%s > %s', NULL),\n ('gte', '%s >= %s', NULL),\n ('le', '%s <= %s', NULL),\n ('ge', '%s >= %s', NULL),\n ('=', '%s = %s', NULL),\n ('<', '%s < %s', NULL),\n ('<=', '%s <= %s', NULL),\n ('>', '%s > %s', NULL),\n ('>=', '%s >= %s', NULL),\n ('like', '%s LIKE %s', NULL),\n ('ilike', '%s ILIKE %s', NULL),\n ('+', '%s + %s', NULL),\n ('-', '%s - %s', NULL),\n ('*', '%s * %s', NULL),\n ('/', '%s / %s', NULL),\n ('not', 'NOT (%s)', NULL),\n ('between', '%s BETWEEN %s AND %s', NULL),\n ('isnull', '%s IS NULL', NULL),\n ('upper', 'upper(%s)', NULL),\n ('lower', 'lower(%s)', NULL)\nON CONFLICT (op) DO UPDATE\n SET\n template = EXCLUDED.template\n;\n\n\nCREATE OR REPLACE FUNCTION cql2_query(j jsonb, wrapper text DEFAULT NULL) RETURNS text AS $$\n#variable_conflict use_variable\nDECLARE\n args jsonb := j->'args';\n arg jsonb;\n op text := lower(j->>'op');\n cql2op RECORD;\n literal text;\n _wrapper text;\n leftarg text;\n rightarg text;\n prop text;\n extra_props bool := pgstac.additional_properties();\nBEGIN\n IF j IS NULL OR (op IS NOT NULL AND args IS NULL) THEN\n RETURN NULL;\n END IF;\n RAISE NOTICE 'CQL2_QUERY: %', j;\n\n -- check if all properties are represented in the queryables\n IF NOT extra_props THEN\n FOR prop IN\n SELECT DISTINCT p->>0\n FROM jsonb_path_query(j, 'strict $.**.property') p\n WHERE p->>0 NOT IN ('id', 'datetime', 'geometry', 'end_datetime', 'collection')\n LOOP\n IF (queryable(prop)).nulled_wrapper IS NULL THEN\n RAISE EXCEPTION 'Term % is not found in queryables.', prop;\n END IF;\n END LOOP;\n END IF;\n\n IF j ? 'filter' THEN\n RETURN cql2_query(j->'filter');\n END IF;\n\n IF j ? 'upper' THEN\n RETURN cql2_query(jsonb_build_object('op', 'upper', 'args', j->'upper'));\n END IF;\n\n IF j ? 'lower' THEN\n RETURN cql2_query(jsonb_build_object('op', 'lower', 'args', j->'lower'));\n END IF;\n\n -- Temporal Query\n IF op ilike 't_%' or op = 'anyinteracts' THEN\n RETURN temporal_op_query(op, args);\n END IF;\n\n -- If property is a timestamp convert it to text to use with\n -- general operators\n IF j ? 'timestamp' THEN\n RETURN format('%L::timestamptz', to_tstz(j->'timestamp'));\n END IF;\n IF j ? 'interval' THEN\n RAISE EXCEPTION 'Please use temporal operators when using intervals.';\n RETURN NONE;\n END IF;\n\n -- Spatial Query\n IF op ilike 's_%' or op = 'intersects' THEN\n RETURN spatial_op_query(op, args);\n END IF;\n\n IF op IN ('a_equals','a_contains','a_contained_by','a_overlaps') THEN\n IF args->0 ? 'property' THEN\n leftarg := format('to_text_array(%s)', (queryable(args->0->>'property')).path);\n END IF;\n IF args->1 ? 'property' THEN\n rightarg := format('to_text_array(%s)', (queryable(args->1->>'property')).path);\n END IF;\n RETURN FORMAT(\n '%s %s %s',\n COALESCE(leftarg, quote_literal(to_text_array(args->0))),\n CASE op\n WHEN 'a_equals' THEN '='\n WHEN 'a_contains' THEN '@>'\n WHEN 'a_contained_by' THEN '<@'\n WHEN 'a_overlaps' THEN '&&'\n END,\n COALESCE(rightarg, quote_literal(to_text_array(args->1)))\n );\n END IF;\n\n IF op = 'in' THEN\n RAISE NOTICE 'IN : % % %', args, jsonb_build_array(args->0), args->1;\n args := jsonb_build_array(args->0) || (args->1);\n RAISE NOTICE 'IN2 : %', args;\n END IF;\n\n\n\n IF op = 'between' THEN\n args = jsonb_build_array(\n args->0,\n args->1->0,\n args->1->1\n );\n END IF;\n\n -- Make sure that args is an array and run cql2_query on\n -- each element of the array\n RAISE NOTICE 'ARGS PRE: %', args;\n IF j ? 'args' THEN\n IF jsonb_typeof(args) != 'array' THEN\n args := jsonb_build_array(args);\n END IF;\n\n IF jsonb_path_exists(args, '$[*] ? (@.property == \"id\" || @.property == \"datetime\" || @.property == \"end_datetime\" || @.property == \"collection\")') THEN\n wrapper := NULL;\n ELSE\n -- if any of the arguments are a property, try to get the property_wrapper\n FOR arg IN SELECT jsonb_path_query(args, '$[*] ? (@.property != null)') LOOP\n RAISE NOTICE 'Arg: %', arg;\n wrapper := (queryable(arg->>'property')).nulled_wrapper;\n RAISE NOTICE 'Property: %, Wrapper: %', arg, wrapper;\n IF wrapper IS NOT NULL THEN\n EXIT;\n END IF;\n END LOOP;\n\n -- if the property was not in queryables, see if any args were numbers\n IF\n wrapper IS NULL\n AND jsonb_path_exists(args, '$[*] ? (@.type()==\"number\")')\n THEN\n wrapper := 'to_float';\n END IF;\n wrapper := coalesce(wrapper, 'to_text');\n END IF;\n\n SELECT jsonb_agg(cql2_query(a, wrapper))\n INTO args\n FROM jsonb_array_elements(args) a;\n END IF;\n RAISE NOTICE 'ARGS: %', args;\n\n IF op IN ('and', 'or') THEN\n RETURN\n format(\n '(%s)',\n array_to_string(to_text_array(args), format(' %s ', upper(op)))\n );\n END IF;\n\n IF op = 'in' THEN\n RAISE NOTICE 'IN -- % %', args->0, to_text(args->0);\n RETURN format(\n '%s IN (%s)',\n to_text(args->0),\n array_to_string((to_text_array(args))[2:], ',')\n );\n END IF;\n\n -- Look up template from cql2_ops\n IF j ? 'op' THEN\n SELECT * INTO cql2op FROM cql2_ops WHERE cql2_ops.op ilike op;\n IF FOUND THEN\n -- If specific index set in queryables for a property cast other arguments to that type\n\n RETURN format(\n cql2op.template,\n VARIADIC (to_text_array(args))\n );\n ELSE\n RAISE EXCEPTION 'Operator % Not Supported.', op;\n END IF;\n END IF;\n\n\n IF wrapper IS NOT NULL THEN\n RAISE NOTICE 'Wrapping % with %', j, wrapper;\n IF j ? 'property' THEN\n RETURN format('%I(%s)', wrapper, (queryable(j->>'property')).path);\n ELSE\n RETURN format('%I(%L)', wrapper, j);\n END IF;\n ELSIF j ? 'property' THEN\n RETURN quote_ident(j->>'property');\n END IF;\n\n RETURN quote_literal(to_text(j));\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION paging_dtrange(\n j jsonb\n) RETURNS tstzrange AS $$\nDECLARE\n op text;\n filter jsonb := j->'filter';\n dtrange tstzrange := tstzrange('-infinity'::timestamptz,'infinity'::timestamptz);\n sdate timestamptz := '-infinity'::timestamptz;\n edate timestamptz := 'infinity'::timestamptz;\n jpitem jsonb;\nBEGIN\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n END IF;\n IF NOT (filter @? '$.**.op ? (@ == \"or\" || @ == \"not\")') THEN\n FOR jpitem IN SELECT j FROM jsonb_path_query(filter,'strict $.** ? (@.args[*].property == \"datetime\")'::jsonpath) j LOOP\n op := lower(jpitem->>'op');\n dtrange := parse_dtrange(jpitem->'args'->1);\n IF op IN ('<=', 'lt', 'lte', '<', 'le', 't_before') THEN\n sdate := greatest(sdate,'-infinity');\n edate := least(edate, upper(dtrange));\n ELSIF op IN ('>=', '>', 'gt', 'gte', 'ge', 't_after') THEN\n edate := least(edate, 'infinity');\n sdate := greatest(sdate, lower(dtrange));\n ELSIF op IN ('=', 'eq') THEN\n edate := least(edate, upper(dtrange));\n sdate := greatest(sdate, lower(dtrange));\n END IF;\n RAISE NOTICE '2 OP: %, ARGS: %, DTRANGE: %, SDATE: %, EDATE: %', op, jpitem->'args'->1, dtrange, sdate, edate;\n END LOOP;\n END IF;\n IF sdate > edate THEN\n RETURN 'empty'::tstzrange;\n END IF;\n RETURN tstzrange(sdate,edate, '[]');\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT SET TIME ZONE 'UTC';\n\nCREATE OR REPLACE FUNCTION paging_collections(\n IN j jsonb\n) RETURNS text[] AS $$\nDECLARE\n filter jsonb := j->'filter';\n jpitem jsonb;\n op text;\n args jsonb;\n arg jsonb;\n collections text[];\nBEGIN\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n END IF;\n IF NOT (filter @? '$.**.op ? (@ == \"or\" || @ == \"not\")') THEN\n FOR jpitem IN SELECT j FROM jsonb_path_query(filter,'strict $.** ? (@.args[*].property == \"collection\")'::jsonpath) j LOOP\n RAISE NOTICE 'JPITEM: %', jpitem;\n op := jpitem->>'op';\n args := jpitem->'args';\n IF op IN ('=', 'eq', 'in') THEN\n FOR arg IN SELECT a FROM jsonb_array_elements(args) a LOOP\n IF jsonb_typeof(arg) IN ('string', 'array') THEN\n RAISE NOTICE 'arg: %, collections: %', arg, collections;\n IF collections IS NULL OR collections = '{}'::text[] THEN\n collections := to_text_array(arg);\n ELSE\n collections := array_intersection(collections, to_text_array(arg));\n END IF;\n END IF;\n END LOOP;\n END IF;\n END LOOP;\n END IF;\n IF collections = '{}'::text[] THEN\n RETURN NULL;\n END IF;\n RETURN collections;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\nCREATE TABLE items (\n id text NOT NULL,\n geometry geometry NOT NULL,\n collection text NOT NULL,\n datetime timestamptz NOT NULL,\n end_datetime timestamptz NOT NULL,\n content JSONB NOT NULL,\n private jsonb\n)\nPARTITION BY LIST (collection)\n;\n\nCREATE INDEX \"datetime_idx\" ON items USING BTREE (datetime DESC, end_datetime ASC);\nCREATE INDEX \"geometry_idx\" ON items USING GIST (geometry);\n\nCREATE STATISTICS datetime_stats (dependencies) on datetime, end_datetime from items;\n\nALTER TABLE items ADD CONSTRAINT items_collections_fk FOREIGN KEY (collection) REFERENCES collections(id) ON DELETE CASCADE DEFERRABLE;\n\nCREATE OR REPLACE FUNCTION partition_after_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p text;\n t timestamptz := clock_timestamp();\nBEGIN\n RAISE NOTICE 'Updating partition stats %', t;\n FOR p IN SELECT DISTINCT partition\n FROM newdata n JOIN partition_sys_meta p\n ON (n.collection=p.collection AND n.datetime <@ p.partition_dtrange)\n LOOP\n PERFORM run_or_queue(format('SELECT update_partition_stats(%L, %L);', p, true));\n END LOOP;\n IF TG_OP IN ('DELETE','UPDATE') THEN\n DELETE FROM format_item_cache c USING newdata n WHERE c.collection = n.collection AND c.id = n.id;\n END IF;\n RAISE NOTICE 't: % %', t, clock_timestamp() - t;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE TRIGGER items_after_insert_trigger\nAFTER INSERT ON items\nREFERENCING NEW TABLE AS newdata\nFOR EACH STATEMENT\nEXECUTE FUNCTION partition_after_triggerfunc();\n\nCREATE TRIGGER items_after_update_trigger\nAFTER DELETE ON items\nREFERENCING OLD TABLE AS newdata\nFOR EACH STATEMENT\nEXECUTE FUNCTION partition_after_triggerfunc();\n\nCREATE TRIGGER items_after_delete_trigger\nAFTER UPDATE ON items\nREFERENCING NEW TABLE AS newdata\nFOR EACH STATEMENT\nEXECUTE FUNCTION partition_after_triggerfunc();\n\n\nCREATE OR REPLACE FUNCTION content_slim(_item jsonb) RETURNS jsonb AS $$\n SELECT strip_jsonb(_item - '{id,geometry,collection,type}'::text[], collection_base_item(_item->>'collection')) - '{id,geometry,collection,type}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_dehydrate(content jsonb) RETURNS items AS $$\n SELECT\n content->>'id' as id,\n stac_geom(content) as geometry,\n content->>'collection' as collection,\n stac_datetime(content) as datetime,\n stac_end_datetime(content) as end_datetime,\n content_slim(content) as content,\n null::jsonb as private\n ;\n$$ LANGUAGE SQL STABLE;\n\nCREATE OR REPLACE FUNCTION include_field(f text, fields jsonb DEFAULT '{}'::jsonb) RETURNS boolean AS $$\nDECLARE\n includes jsonb := fields->'include';\n excludes jsonb := fields->'exclude';\nBEGIN\n IF f IS NULL THEN\n RETURN NULL;\n END IF;\n\n\n IF\n jsonb_typeof(excludes) = 'array'\n AND jsonb_array_length(excludes)>0\n AND excludes ? f\n THEN\n RETURN FALSE;\n END IF;\n\n IF\n (\n jsonb_typeof(includes) = 'array'\n AND jsonb_array_length(includes) > 0\n AND includes ? f\n ) OR\n (\n includes IS NULL\n OR jsonb_typeof(includes) = 'null'\n OR jsonb_array_length(includes) = 0\n )\n THEN\n RETURN TRUE;\n END IF;\n\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nDROP FUNCTION IF EXISTS content_hydrate(jsonb, jsonb, jsonb);\nCREATE OR REPLACE FUNCTION content_hydrate(\n _item jsonb,\n _base_item jsonb,\n fields jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\n SELECT merge_jsonb(\n jsonb_fields(_item, fields),\n jsonb_fields(_base_item, fields)\n );\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION content_hydrate(_item items, _collection collections, fields jsonb DEFAULT '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\n content jsonb;\n base_item jsonb := _collection.base_item;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry, 20)::jsonb;\n END IF;\n output := content_hydrate(\n jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'collection', _item.collection,\n 'type', 'Feature'\n ) || _item.content,\n _collection.base_item,\n fields\n );\n\n RETURN output;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_nonhydrated(\n _item items,\n fields jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry, 20)::jsonb;\n END IF;\n output := jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'collection', _item.collection,\n 'type', 'Feature'\n ) || _item.content;\n RETURN output;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_hydrate(_item items, fields jsonb DEFAULT '{}'::jsonb) RETURNS jsonb AS $$\n SELECT content_hydrate(\n _item,\n (SELECT c FROM collections c WHERE id=_item.collection LIMIT 1),\n fields\n );\n$$ LANGUAGE SQL STABLE;\n\n\nCREATE UNLOGGED TABLE items_staging (\n content JSONB NOT NULL\n);\nCREATE UNLOGGED TABLE items_staging_ignore (\n content JSONB NOT NULL\n);\nCREATE UNLOGGED TABLE items_staging_upsert (\n content JSONB NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION items_staging_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p record;\n _partitions text[];\n part text;\n ts timestamptz := clock_timestamp();\n nrows int;\nBEGIN\n RAISE NOTICE 'Creating Partitions. %', clock_timestamp() - ts;\n\n FOR part IN WITH t AS (\n SELECT\n n.content->>'collection' as collection,\n stac_daterange(n.content->'properties') as dtr,\n partition_trunc\n FROM newdata n JOIN collections ON (n.content->>'collection'=collections.id)\n ), p AS (\n SELECT\n collection,\n COALESCE(date_trunc(partition_trunc::text, lower(dtr)),'-infinity') as d,\n tstzrange(min(lower(dtr)),max(lower(dtr)),'[]') as dtrange,\n tstzrange(min(upper(dtr)),max(upper(dtr)),'[]') as edtrange\n FROM t\n GROUP BY 1,2\n ) SELECT check_partition(collection, dtrange, edtrange) FROM p LOOP\n RAISE NOTICE 'Partition %', part;\n END LOOP;\n\n RAISE NOTICE 'Creating temp table with data to be added. %', clock_timestamp() - ts;\n DROP TABLE IF EXISTS tmpdata;\n CREATE TEMP TABLE tmpdata ON COMMIT DROP AS\n SELECT\n (content_dehydrate(content)).*\n FROM newdata;\n GET DIAGNOSTICS nrows = ROW_COUNT;\n RAISE NOTICE 'Added % rows to tmpdata. %', nrows, clock_timestamp() - ts;\n\n RAISE NOTICE 'Doing the insert. %', clock_timestamp() - ts;\n IF TG_TABLE_NAME = 'items_staging' THEN\n INSERT INTO items\n SELECT * FROM tmpdata;\n GET DIAGNOSTICS nrows = ROW_COUNT;\n RAISE NOTICE 'Inserted % rows to items. %', nrows, clock_timestamp() - ts;\n ELSIF TG_TABLE_NAME = 'items_staging_ignore' THEN\n INSERT INTO items\n SELECT * FROM tmpdata\n ON CONFLICT DO NOTHING;\n GET DIAGNOSTICS nrows = ROW_COUNT;\n RAISE NOTICE 'Inserted % rows to items. %', nrows, clock_timestamp() - ts;\n ELSIF TG_TABLE_NAME = 'items_staging_upsert' THEN\n DELETE FROM items i USING tmpdata s\n WHERE\n i.id = s.id\n AND i.collection = s.collection\n AND i IS DISTINCT FROM s\n ;\n GET DIAGNOSTICS nrows = ROW_COUNT;\n RAISE NOTICE 'Deleted % rows from items. %', nrows, clock_timestamp() - ts;\n INSERT INTO items AS t\n SELECT * FROM tmpdata\n ON CONFLICT DO NOTHING;\n GET DIAGNOSTICS nrows = ROW_COUNT;\n RAISE NOTICE 'Inserted % rows to items. %', nrows, clock_timestamp() - ts;\n END IF;\n\n RAISE NOTICE 'Deleting data from staging table. %', clock_timestamp() - ts;\n DELETE FROM items_staging;\n RAISE NOTICE 'Done. %', clock_timestamp() - ts;\n\n RETURN NULL;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER items_staging_insert_trigger AFTER INSERT ON items_staging REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\nCREATE TRIGGER items_staging_insert_ignore_trigger AFTER INSERT ON items_staging_ignore REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\nCREATE TRIGGER items_staging_insert_upsert_trigger AFTER INSERT ON items_staging_upsert REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\n\nCREATE OR REPLACE FUNCTION item_by_id(_id text, _collection text DEFAULT NULL) RETURNS items AS\n$$\nDECLARE\n i items%ROWTYPE;\nBEGIN\n SELECT * INTO i FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection) LIMIT 1;\n RETURN i;\nEND;\n$$ LANGUAGE PLPGSQL STABLE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION get_item(_id text, _collection text DEFAULT NULL) RETURNS jsonb AS $$\n SELECT content_hydrate(items) FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection);\n$$ LANGUAGE SQL STABLE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION delete_item(_id text, _collection text DEFAULT NULL) RETURNS VOID AS $$\nDECLARE\nout items%ROWTYPE;\nBEGIN\n DELETE FROM items WHERE id = _id AND (_collection IS NULL OR collection=_collection) RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n--/*\nCREATE OR REPLACE FUNCTION create_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION update_item(content jsonb) RETURNS VOID AS $$\nDECLARE\n old items %ROWTYPE;\n out items%ROWTYPE;\nBEGIN\n PERFORM delete_item(content->>'id', content->>'collection');\n PERFORM create_item(content);\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION create_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION collection_bbox(id text) RETURNS jsonb AS $$\n SELECT (replace(replace(replace(st_extent(geometry)::text,'BOX(','[['),')',']]'),' ',','))::jsonb\n FROM items WHERE collection=$1;\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION collection_temporal_extent(id text) RETURNS jsonb AS $$\n SELECT to_jsonb(array[array[min(datetime)::text, max(datetime)::text]])\n FROM items WHERE collection=$1;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection_extents() RETURNS VOID AS $$\nUPDATE collections SET\n content = content ||\n jsonb_build_object(\n 'extent', jsonb_build_object(\n 'spatial', jsonb_build_object(\n 'bbox', collection_bbox(collections.id)\n ),\n 'temporal', jsonb_build_object(\n 'interval', collection_temporal_extent(collections.id)\n )\n )\n )\n;\n$$ LANGUAGE SQL;\nCREATE TABLE partition_stats (\n partition text PRIMARY KEY,\n dtrange tstzrange,\n edtrange tstzrange,\n spatial geometry,\n last_updated timestamptz,\n keys text[]\n) WITH (FILLFACTOR=90);\n\nCREATE INDEX partitions_range_idx ON partition_stats USING GIST(dtrange);\n\n\nCREATE OR REPLACE FUNCTION constraint_tstzrange(expr text) RETURNS tstzrange AS $$\n WITH t AS (\n SELECT regexp_matches(\n expr,\n E'\\\\(''\\([0-9 :+-]*\\)''\\\\).*\\\\(''\\([0-9 :+-]*\\)''\\\\)'\n ) AS m\n ) SELECT tstzrange(m[1]::timestamptz, m[2]::timestamptz) FROM t\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE STRICT;\n\nCREATE OR REPLACE FUNCTION dt_constraint(coid oid, OUT dt tstzrange, OUT edt tstzrange) RETURNS RECORD AS $$\nDECLARE\n expr text := pg_get_constraintdef(coid);\n matches timestamptz[];\nBEGIN\n IF expr LIKE '%NULL%' THEN\n dt := tstzrange(null::timestamptz, null::timestamptz);\n edt := tstzrange(null::timestamptz, null::timestamptz);\n RETURN;\n END IF;\n WITH f AS (SELECT (regexp_matches(expr, E'([0-9]{4}-[0-1][0-9]-[0-3][0-9] [0-2][0-9]:[0-5][0-9]:[0-5][0-9]\\.?[0-9]*)', 'g'))[1] f)\n SELECT array_agg(f::timestamptz) INTO matches FROM f;\n IF cardinality(matches) = 4 THEN\n dt := tstzrange(matches[1], matches[2],'[]');\n edt := tstzrange(matches[3], matches[4], '[]');\n RETURN;\n ELSIF cardinality(matches) = 2 THEN\n edt := tstzrange(matches[1], matches[2],'[]');\n RETURN;\n END IF;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\nCREATE OR REPLACE VIEW partition_sys_meta AS\nSELECT\n relid::text as partition,\n replace(replace(CASE WHEN level = 1 THEN pg_get_expr(c.relpartbound, c.oid)\n ELSE pg_get_expr(parent.relpartbound, parent.oid)\n END, 'FOR VALUES IN (''',''), ''')','') AS collection,\n level,\n c.reltuples,\n c.relhastriggers,\n COALESCE(constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as partition_dtrange,\n COALESCE((dt_constraint(edt.oid)).dt, constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as constraint_dtrange,\n COALESCE((dt_constraint(edt.oid)).edt, tstzrange('-infinity', 'infinity','[]')) as constraint_edtrange\nFROM\n pg_partition_tree('items')\n JOIN pg_class c ON (relid::regclass = c.oid)\n JOIN pg_class parent ON (parentrelid::regclass = parent.oid AND isleaf)\n LEFT JOIN pg_constraint edt ON (conrelid=c.oid AND contype='c')\nWHERE isleaf\n;\n\nCREATE VIEW partitions_view AS\nSELECT\n relid::text as partition,\n replace(replace(CASE WHEN level = 1 THEN pg_get_expr(c.relpartbound, c.oid)\n ELSE pg_get_expr(parent.relpartbound, parent.oid)\n END, 'FOR VALUES IN (''',''), ''')','') AS collection,\n level,\n c.reltuples,\n c.relhastriggers,\n COALESCE(constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as partition_dtrange,\n COALESCE((dt_constraint(edt.oid)).dt, constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as constraint_dtrange,\n COALESCE((dt_constraint(edt.oid)).edt, tstzrange('-infinity', 'infinity','[]')) as constraint_edtrange,\n dtrange,\n edtrange,\n spatial,\n last_updated\nFROM\n pg_partition_tree('items')\n JOIN pg_class c ON (relid::regclass = c.oid)\n JOIN pg_class parent ON (parentrelid::regclass = parent.oid AND isleaf)\n LEFT JOIN pg_constraint edt ON (conrelid=c.oid AND contype='c')\n LEFT JOIN partition_stats ON (relid::text=partition)\nWHERE isleaf\n;\n\nCREATE MATERIALIZED VIEW partitions AS\nSELECT * FROM partitions_view;\nCREATE UNIQUE INDEX ON partitions (partition);\n\nCREATE MATERIALIZED VIEW partition_steps AS\nSELECT\n partition as name,\n date_trunc('month',lower(partition_dtrange)) as sdate,\n date_trunc('month', upper(partition_dtrange)) + '1 month'::interval as edate\n FROM partitions_view WHERE partition_dtrange IS NOT NULL AND partition_dtrange != 'empty'::tstzrange\n ORDER BY dtrange ASC\n;\n\n\nCREATE OR REPLACE FUNCTION update_partition_stats_q(_partition text, istrigger boolean default false) RETURNS VOID AS $$\nDECLARE\nBEGIN\n PERFORM run_or_queue(\n format('SELECT update_partition_stats(%L, %L);', _partition, istrigger)\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION update_partition_stats(_partition text, istrigger boolean default false) RETURNS VOID AS $$\nDECLARE\n dtrange tstzrange;\n edtrange tstzrange;\n cdtrange tstzrange;\n cedtrange tstzrange;\n extent geometry;\n collection text;\nBEGIN\n RAISE NOTICE 'Updating stats for %.', _partition;\n EXECUTE format(\n $q$\n SELECT\n tstzrange(min(datetime), max(datetime),'[]'),\n tstzrange(min(end_datetime), max(end_datetime), '[]')\n FROM %I\n $q$,\n _partition\n ) INTO dtrange, edtrange;\n extent := st_estimatedextent('pgstac', _partition, 'geometry');\n INSERT INTO partition_stats (partition, dtrange, edtrange, spatial, last_updated)\n SELECT _partition, dtrange, edtrange, extent, now()\n ON CONFLICT (partition) DO\n UPDATE SET\n dtrange=EXCLUDED.dtrange,\n edtrange=EXCLUDED.edtrange,\n spatial=EXCLUDED.spatial,\n last_updated=EXCLUDED.last_updated\n ;\n\n SELECT\n constraint_dtrange, constraint_edtrange, pv.collection\n INTO cdtrange, cedtrange, collection\n FROM partitions_view pv WHERE partition = _partition;\n REFRESH MATERIALIZED VIEW partitions;\n REFRESH MATERIALIZED VIEW partition_steps;\n\n\n RAISE NOTICE 'Checking if we need to modify constraints...';\n RAISE NOTICE 'cdtrange: % dtrange: % cedtrange: % edtrange: %',cdtrange, dtrange, cedtrange, edtrange;\n IF\n (cdtrange IS DISTINCT FROM dtrange OR edtrange IS DISTINCT FROM cedtrange)\n AND NOT istrigger\n THEN\n RAISE NOTICE 'Modifying Constraints';\n RAISE NOTICE 'Existing % %', cdtrange, cedtrange;\n RAISE NOTICE 'New % %', dtrange, edtrange;\n PERFORM drop_table_constraints(_partition);\n PERFORM create_table_constraints(_partition, dtrange, edtrange);\n REFRESH MATERIALIZED VIEW partitions;\n REFRESH MATERIALIZED VIEW partition_steps;\n END IF;\n RAISE NOTICE 'Checking if we need to update collection extents.';\n IF get_setting_bool('update_collection_extent') THEN\n RAISE NOTICE 'updating collection extent for %', collection;\n PERFORM run_or_queue(format($q$\n UPDATE collections\n SET content = jsonb_set_lax(\n content,\n '{extent}'::text[],\n collection_extent(%L, FALSE),\n true,\n 'use_json_null'\n ) WHERE id=%L\n ;\n $q$, collection, collection));\n ELSE\n RAISE NOTICE 'Not updating collection extent for %', collection;\n END IF;\n\nEND;\n$$ LANGUAGE PLPGSQL STRICT SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION partition_name( IN collection text, IN dt timestamptz, OUT partition_name text, OUT partition_range tstzrange) AS $$\nDECLARE\n c RECORD;\n parent_name text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n parent_name := format('_items_%s', c.key);\n\n\n IF c.partition_trunc = 'year' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYY'));\n ELSIF c.partition_trunc = 'month' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYYMM'));\n ELSE\n partition_name := parent_name;\n partition_range := tstzrange('-infinity'::timestamptz, 'infinity'::timestamptz, '[]');\n END IF;\n IF partition_range IS NULL THEN\n partition_range := tstzrange(\n date_trunc(c.partition_trunc::text, dt),\n date_trunc(c.partition_trunc::text, dt) + concat('1 ', c.partition_trunc)::interval\n );\n END IF;\n RETURN;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION drop_table_constraints(t text) RETURNS text AS $$\nDECLARE\n q text;\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM partitions_view WHERE partition=t) THEN\n RETURN NULL;\n END IF;\n FOR q IN SELECT FORMAT(\n $q$\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n $q$,\n t,\n conname\n ) FROM pg_constraint\n WHERE conrelid=t::regclass::oid AND contype='c'\n LOOP\n EXECUTE q;\n END LOOP;\n RETURN t;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE OR REPLACE FUNCTION create_table_constraints(t text, _dtrange tstzrange, _edtrange tstzrange) RETURNS text AS $$\nDECLARE\n q text;\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM partitions_view WHERE partition=t) THEN\n RETURN NULL;\n END IF;\n RAISE NOTICE 'Creating Table Constraints for % % %', t, _dtrange, _edtrange;\n IF _dtrange = 'empty' AND _edtrange = 'empty' THEN\n q :=format(\n $q$\n DO $block$\n BEGIN\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n ALTER TABLE %I\n ADD CONSTRAINT %I\n CHECK (((datetime IS NULL) AND (end_datetime IS NULL))) NOT VALID\n ;\n ALTER TABLE %I\n VALIDATE CONSTRAINT %I\n ;\n\n\n\n EXCEPTION WHEN others THEN\n RAISE WARNING '%%, Issue Altering Constraints. Please run update_partition_stats(%I)', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n $block$;\n $q$,\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t),\n t\n );\n ELSE\n q :=format(\n $q$\n DO $block$\n BEGIN\n\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n ALTER TABLE %I\n ADD CONSTRAINT %I\n CHECK (\n (datetime >= %L)\n AND (datetime <= %L)\n AND (end_datetime >= %L)\n AND (end_datetime <= %L)\n ) NOT VALID\n ;\n ALTER TABLE %I\n VALIDATE CONSTRAINT %I\n ;\n\n\n\n EXCEPTION WHEN others THEN\n RAISE WARNING '%%, Issue Altering Constraints. Please run update_partition_stats(%I)', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n $block$;\n $q$,\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t),\n lower(_dtrange),\n upper(_dtrange),\n lower(_edtrange),\n upper(_edtrange),\n t,\n format('%s_dt', t),\n t\n );\n END IF;\n PERFORM run_or_queue(q);\n RETURN t;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION check_partition(\n _collection text,\n _dtrange tstzrange,\n _edtrange tstzrange\n) RETURNS text AS $$\nDECLARE\n c RECORD;\n pm RECORD;\n _partition_name text;\n _partition_dtrange tstzrange;\n _constraint_dtrange tstzrange;\n _constraint_edtrange tstzrange;\n q text;\n deferrable_q text;\n err_context text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=_collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', _collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n\n IF c.partition_trunc IS NOT NULL THEN\n _partition_dtrange := tstzrange(\n date_trunc(c.partition_trunc, lower(_dtrange)),\n date_trunc(c.partition_trunc, lower(_dtrange)) + (concat('1 ', c.partition_trunc))::interval,\n '[)'\n );\n ELSE\n _partition_dtrange := '[-infinity, infinity]'::tstzrange;\n END IF;\n\n IF NOT _partition_dtrange @> _dtrange THEN\n RAISE EXCEPTION 'dtrange % is greater than the partition size % for collection %', _dtrange, c.partition_trunc, _collection;\n END IF;\n\n\n IF c.partition_trunc = 'year' THEN\n _partition_name := format('_items_%s_%s', c.key, to_char(lower(_partition_dtrange),'YYYY'));\n ELSIF c.partition_trunc = 'month' THEN\n _partition_name := format('_items_%s_%s', c.key, to_char(lower(_partition_dtrange),'YYYYMM'));\n ELSE\n _partition_name := format('_items_%s', c.key);\n END IF;\n\n SELECT * INTO pm FROM partition_sys_meta WHERE collection=_collection AND partition_dtrange @> _dtrange;\n IF FOUND THEN\n RAISE NOTICE '% % %', _edtrange, _dtrange, pm;\n _constraint_edtrange :=\n tstzrange(\n least(\n lower(_edtrange),\n nullif(lower(pm.constraint_edtrange), '-infinity')\n ),\n greatest(\n upper(_edtrange),\n nullif(upper(pm.constraint_edtrange), 'infinity')\n ),\n '[]'\n );\n _constraint_dtrange :=\n tstzrange(\n least(\n lower(_dtrange),\n nullif(lower(pm.constraint_dtrange), '-infinity')\n ),\n greatest(\n upper(_dtrange),\n nullif(upper(pm.constraint_dtrange), 'infinity')\n ),\n '[]'\n );\n\n IF pm.constraint_edtrange @> _edtrange AND pm.constraint_dtrange @> _dtrange THEN\n RETURN pm.partition;\n ELSE\n PERFORM drop_table_constraints(_partition_name);\n END IF;\n ELSE\n _constraint_edtrange := _edtrange;\n _constraint_dtrange := _dtrange;\n END IF;\n RAISE NOTICE 'EXISTING CONSTRAINTS % %, NEW % %', pm.constraint_dtrange, pm.constraint_edtrange, _constraint_dtrange, _constraint_edtrange;\n RAISE NOTICE 'Creating partition % %', _partition_name, _partition_dtrange;\n IF c.partition_trunc IS NULL THEN\n q := format(\n $q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n GRANT ALL ON %I to pgstac_ingest;\n $q$,\n _partition_name,\n _collection,\n concat(_partition_name,'_pk'),\n _partition_name,\n _partition_name\n );\n ELSE\n q := format(\n $q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L) PARTITION BY RANGE (datetime);\n CREATE TABLE IF NOT EXISTS %I partition OF %I FOR VALUES FROM (%L) TO (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n GRANT ALL ON %I TO pgstac_ingest;\n $q$,\n format('_items_%s', c.key),\n _collection,\n _partition_name,\n format('_items_%s', c.key),\n lower(_partition_dtrange),\n upper(_partition_dtrange),\n format('%s_pk', _partition_name),\n _partition_name,\n _partition_name\n );\n END IF;\n\n BEGIN\n EXECUTE q;\n EXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', _partition_name;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\n END;\n PERFORM maintain_partitions(_partition_name);\n PERFORM update_partition_stats_q(_partition_name, true);\n REFRESH MATERIALIZED VIEW partitions;\n REFRESH MATERIALIZED VIEW partition_steps;\n RETURN _partition_name;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION repartition(_collection text, _partition_trunc text, triggered boolean DEFAULT FALSE) RETURNS text AS $$\nDECLARE\n c RECORD;\n q text;\n from_trunc text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=_collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', _collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n IF triggered THEN\n RAISE NOTICE 'Converting % to % partitioning via Trigger', _collection, _partition_trunc;\n ELSE\n RAISE NOTICE 'Converting % from using % to % partitioning', _collection, c.partition_trunc, _partition_trunc;\n IF c.partition_trunc IS NOT DISTINCT FROM _partition_trunc THEN\n RAISE NOTICE 'Collection % already set to use partition by %', _collection, _partition_trunc;\n RETURN _collection;\n END IF;\n END IF;\n\n IF EXISTS (SELECT 1 FROM partitions_view WHERE collection=_collection LIMIT 1) THEN\n EXECUTE format(\n $q$\n CREATE TEMP TABLE changepartitionstaging ON COMMIT DROP AS SELECT * FROM %I;\n DROP TABLE IF EXISTS %I CASCADE;\n WITH p AS (\n SELECT\n collection,\n CASE\n WHEN %L IS NULL THEN '-infinity'::timestamptz\n ELSE date_trunc(%L, datetime)\n END as d,\n tstzrange(min(datetime),max(datetime),'[]') as dtrange,\n tstzrange(min(end_datetime),max(end_datetime),'[]') as edtrange\n FROM changepartitionstaging\n GROUP BY 1,2\n ) SELECT check_partition(collection, dtrange, edtrange) FROM p;\n INSERT INTO items SELECT * FROM changepartitionstaging;\n DROP TABLE changepartitionstaging;\n $q$,\n concat('_items_', c.key),\n concat('_items_', c.key),\n c.partition_trunc,\n c.partition_trunc\n );\n END IF;\n RETURN _collection;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE OR REPLACE FUNCTION collections_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n q text;\n partition_name text := format('_items_%s', NEW.key);\n partition_exists boolean := false;\n partition_empty boolean := true;\n err_context text;\n loadtemp boolean := FALSE;\nBEGIN\n RAISE NOTICE 'Collection Trigger. % %', NEW.id, NEW.key;\n IF TG_OP = 'UPDATE' AND NEW.partition_trunc IS DISTINCT FROM OLD.partition_trunc THEN\n PERFORM repartition(NEW.id, NEW.partition_trunc, TRUE);\n END IF;\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER collections_trigger AFTER\nINSERT\nOR\nUPDATE ON collections\nFOR EACH ROW EXECUTE FUNCTION collections_trigger_func();\n\nCREATE OR REPLACE FUNCTION chunker(\n IN _where text,\n OUT s timestamptz,\n OUT e timestamptz\n) RETURNS SETOF RECORD AS $$\nDECLARE\n explain jsonb;\nBEGIN\n IF _where IS NULL THEN\n _where := ' TRUE ';\n END IF;\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s;', _where)\n INTO explain;\n\n RETURN QUERY\n WITH t AS (\n SELECT j->>0 as p FROM\n jsonb_path_query(\n explain,\n 'strict $.**.\"Relation Name\" ? (@ != null)'\n ) j\n ),\n parts AS (\n SELECT sdate, edate FROM t JOIN partition_steps ON (t.p = name)\n ),\n times AS (\n SELECT sdate FROM parts\n UNION\n SELECT edate FROM parts\n ),\n uniq AS (\n SELECT DISTINCT sdate FROM times ORDER BY sdate\n ),\n last AS (\n SELECT sdate, lead(sdate, 1) over () as edate FROM uniq\n )\n SELECT sdate, edate FROM last WHERE edate IS NOT NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION partition_queries(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN partitions text[] DEFAULT NULL\n) RETURNS SETOF text AS $$\nDECLARE\n query text;\n sdate timestamptz;\n edate timestamptz;\nBEGIN\nIF _where IS NULL OR trim(_where) = '' THEN\n _where = ' TRUE ';\nEND IF;\nRAISE NOTICE 'Getting chunks for % %', _where, _orderby;\nIF _orderby ILIKE 'datetime d%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 DESC LOOP\n RETURN NEXT format($q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n $q$,\n sdate,\n edate,\n _where,\n _orderby\n );\n END LOOP;\nELSIF _orderby ILIKE 'datetime a%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 ASC LOOP\n RETURN NEXT format($q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n $q$,\n sdate,\n edate,\n _where,\n _orderby\n );\n END LOOP;\nELSE\n query := format($q$\n SELECT * FROM items\n WHERE %s\n ORDER BY %s\n $q$, _where, _orderby\n );\n\n RETURN NEXT query;\n RETURN;\nEND IF;\n\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION partition_query_view(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN _limit int DEFAULT 10\n) RETURNS text AS $$\n WITH p AS (\n SELECT * FROM partition_queries(_where, _orderby) p\n )\n SELECT\n CASE WHEN EXISTS (SELECT 1 FROM p) THEN\n (SELECT format($q$\n SELECT * FROM (\n %s\n ) total LIMIT %s\n $q$,\n string_agg(\n format($q$ SELECT * FROM ( %s ) AS sub $q$, p),\n '\n UNION ALL\n '\n ),\n _limit\n ))\n ELSE NULL\n END FROM p;\n$$ LANGUAGE SQL IMMUTABLE;\n\n\n\n\nCREATE OR REPLACE FUNCTION stac_search_to_where(j jsonb) RETURNS text AS $$\nDECLARE\n where_segments text[];\n _where text;\n dtrange tstzrange;\n collections text[];\n geom geometry;\n sdate timestamptz;\n edate timestamptz;\n filterlang text;\n filter jsonb := j->'filter';\nBEGIN\n IF j ? 'ids' THEN\n where_segments := where_segments || format('id = ANY (%L) ', to_text_array(j->'ids'));\n END IF;\n\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n where_segments := where_segments || format('collection = ANY (%L) ', collections);\n END IF;\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n\n where_segments := where_segments || format(' datetime <= %L::timestamptz AND end_datetime >= %L::timestamptz ',\n edate,\n sdate\n );\n END IF;\n\n geom := stac_geom(j);\n IF geom IS NOT NULL THEN\n where_segments := where_segments || format('st_intersects(geometry, %L)',geom);\n END IF;\n\n filterlang := COALESCE(\n j->>'filter-lang',\n get_setting('default_filter_lang', j->'conf')\n );\n IF NOT filter @? '$.**.op' THEN\n filterlang := 'cql-json';\n END IF;\n\n IF filterlang NOT IN ('cql-json','cql2-json') AND j ? 'filter' THEN\n RAISE EXCEPTION '% is not a supported filter-lang. Please use cql-json or cql2-json.', filterlang;\n END IF;\n\n IF j ? 'query' AND j ? 'filter' THEN\n RAISE EXCEPTION 'Can only use either query or filter at one time.';\n END IF;\n\n IF j ? 'query' THEN\n filter := query_to_cql2(j->'query');\n ELSIF filterlang = 'cql-json' THEN\n filter := cql1_to_cql2(filter);\n END IF;\n RAISE NOTICE 'FILTER: %', filter;\n where_segments := where_segments || cql2_query(filter);\n IF cardinality(where_segments) < 1 THEN\n RETURN ' TRUE ';\n END IF;\n\n _where := array_to_string(array_remove(where_segments, NULL), ' AND ');\n\n IF _where IS NULL OR BTRIM(_where) = '' THEN\n RETURN ' TRUE ';\n END IF;\n RETURN _where;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION parse_sort_dir(_dir text, reverse boolean default false) RETURNS text AS $$\n WITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n ) SELECT\n CASE\n WHEN NOT reverse THEN d\n WHEN d = 'ASC' THEN 'DESC'\n WHEN d = 'DESC' THEN 'ASC'\n END\n FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION sort_dir_to_op(_dir text, prev boolean default false) RETURNS text AS $$\n WITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n ) SELECT\n CASE\n WHEN d = 'ASC' AND prev THEN '<='\n WHEN d = 'DESC' AND prev THEN '>='\n WHEN d = 'ASC' THEN '>='\n WHEN d = 'DESC' THEN '<='\n END\n FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION sort_sqlorderby(\n _search jsonb DEFAULT NULL,\n reverse boolean DEFAULT FALSE\n) RETURNS text AS $$\n WITH sortby AS (\n SELECT coalesce(_search->'sortby','[{\"field\":\"datetime\", \"direction\":\"desc\"}]') as sort\n ), withid AS (\n SELECT CASE\n WHEN sort @? '$[*] ? (@.field == \"id\")' THEN sort\n ELSE sort || '[{\"field\":\"id\", \"direction\":\"desc\"}]'::jsonb\n END as sort\n FROM sortby\n ), withid_rows AS (\n SELECT jsonb_array_elements(sort) as value FROM withid\n ),sorts AS (\n SELECT\n coalesce(\n (queryable(value->>'field')).expression\n ) as key,\n parse_sort_dir(value->>'direction', reverse) as dir\n FROM withid_rows\n )\n SELECT array_to_string(\n array_agg(concat(key, ' ', dir)),\n ', '\n ) FROM sorts;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_sort_dir(sort_item jsonb) RETURNS text AS $$\n SELECT CASE WHEN sort_item->>'direction' ILIKE 'desc%' THEN 'DESC' ELSE 'ASC' END;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION get_token_val_str(\n _field text,\n _item items\n) RETURNS text AS $$\nDECLARE\n q text;\n literal text;\nBEGIN\n q := format($q$ SELECT quote_literal(%s) FROM (SELECT $1.*) as r;$q$, _field);\n EXECUTE q INTO literal USING _item;\n RETURN literal;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n\nCREATE OR REPLACE FUNCTION get_token_record(IN _token text, OUT prev BOOLEAN, OUT item items) RETURNS RECORD AS $$\nDECLARE\n _itemid text := _token;\n _collectionid text;\nBEGIN\n IF _token IS NULL THEN\n RETURN;\n END IF;\n RAISE NOTICE 'Looking for token: %', _token;\n prev := FALSE;\n IF _token ILIKE 'prev:%' THEN\n _itemid := replace(_token, 'prev:','');\n prev := TRUE;\n ELSIF _token ILIKE 'next:%' THEN\n _itemid := replace(_token, 'next:', '');\n END IF;\n SELECT id INTO _collectionid FROM collections WHERE _itemid LIKE concat(id,':%');\n IF FOUND THEN\n _itemid := replace(_itemid, concat(_collectionid,':'), '');\n SELECT * INTO item FROM items WHERE id=_itemid AND collection=_collectionid;\n ELSE\n SELECT * INTO item FROM items WHERE id=_itemid;\n END IF;\n IF item IS NULL THEN\n RAISE EXCEPTION 'Could not find item using token: % item: % collection: %', _token, _itemid, _collectionid;\n END IF;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\n\nCREATE OR REPLACE FUNCTION get_token_filter(\n _sortby jsonb DEFAULT '[{\"field\":\"datetime\",\"direction\":\"desc\"}]'::jsonb,\n token_item items DEFAULT NULL,\n prev boolean DEFAULT FALSE,\n inclusive boolean DEFAULT FALSE\n) RETURNS text AS $$\nDECLARE\n ltop text := '<';\n gtop text := '>';\n dir text;\n sort record;\n orfilter text := '';\n orfilters text[] := '{}'::text[];\n andfilters text[] := '{}'::text[];\n output text;\n token_where text;\nBEGIN\n IF _sortby IS NULL OR _sortby = '[]'::jsonb THEN\n _sortby := '[{\"field\":\"datetime\",\"direction\":\"desc\"}]'::jsonb;\n END IF;\n _sortby := _sortby || jsonb_build_object('field','id','direction',_sortby->0->>'direction');\n RAISE NOTICE 'Getting Token Filter. % %', _sortby, token_item;\n IF inclusive THEN\n orfilters := orfilters || format('( id=%L AND collection=%L )' , token_item.id, token_item.collection);\n END IF;\n\n FOR sort IN\n WITH s1 AS (\n SELECT\n _row,\n (queryable(value->>'field')).expression as _field,\n (value->>'field' = 'id') as _isid,\n get_sort_dir(value) as _dir\n FROM jsonb_array_elements(_sortby)\n WITH ORDINALITY AS t(value, _row)\n )\n SELECT\n _row,\n _field,\n _dir,\n get_token_val_str(_field, token_item) as _val\n FROM s1\n WHERE _row <= (SELECT min(_row) FROM s1 WHERE _isid)\n LOOP\n orfilter := NULL;\n RAISE NOTICE 'SORT: %', sort;\n IF sort._val IS NOT NULL AND ((prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC')) THEN\n orfilter := format($f$(\n (%s %s %s) OR (%s IS NULL)\n )$f$,\n sort._field,\n ltop,\n sort._val,\n sort._val\n );\n ELSIF sort._val IS NULL AND ((prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC')) THEN\n RAISE NOTICE '< but null';\n orfilter := format('%s IS NOT NULL', sort._field);\n ELSIF sort._val IS NULL THEN\n RAISE NOTICE '> but null';\n ELSE\n orfilter := format($f$(\n (%s %s %s) OR (%s IS NULL)\n )$f$,\n sort._field,\n gtop,\n sort._val,\n sort._field\n );\n END IF;\n RAISE NOTICE 'ORFILTER: %', orfilter;\n\n IF orfilter IS NOT NULL THEN\n IF sort._row = 1 THEN\n orfilters := orfilters || orfilter;\n ELSE\n orfilters := orfilters || format('(%s AND %s)', array_to_string(andfilters, ' AND '), orfilter);\n END IF;\n END IF;\n IF sort._val IS NOT NULL THEN\n andfilters := andfilters || format('%s = %s', sort._field, sort._val);\n ELSE\n andfilters := andfilters || format('%s IS NULL', sort._field);\n END IF;\n END LOOP;\n\n output := array_to_string(orfilters, ' OR ');\n\n token_where := concat('(',coalesce(output,'true'),')');\n IF trim(token_where) = '' THEN\n token_where := NULL;\n END IF;\n RAISE NOTICE 'TOKEN_WHERE: %',token_where;\n RETURN token_where;\n END;\n$$ LANGUAGE PLPGSQL SET transform_null_equals TO TRUE\n;\n\nCREATE OR REPLACE FUNCTION search_tohash(jsonb) RETURNS jsonb AS $$\n SELECT $1 - '{token,limit,context,includes,excludes}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION search_hash(jsonb, jsonb) RETURNS text AS $$\n SELECT md5(concat(search_tohash($1)::text,$2::text));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TABLE IF NOT EXISTS searches(\n hash text GENERATED ALWAYS AS (search_hash(search, metadata)) STORED PRIMARY KEY,\n search jsonb NOT NULL,\n _where text,\n orderby text,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n metadata jsonb DEFAULT '{}'::jsonb NOT NULL\n);\n\nCREATE TABLE IF NOT EXISTS search_wheres(\n id bigint generated always as identity primary key,\n _where text NOT NULL,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n statslastupdated timestamptz,\n estimated_count bigint,\n estimated_cost float,\n time_to_estimate float,\n total_count bigint,\n time_to_count float,\n partitions text[]\n);\n\nCREATE INDEX IF NOT EXISTS search_wheres_partitions ON search_wheres USING GIN (partitions);\nCREATE UNIQUE INDEX IF NOT EXISTS search_wheres_where ON search_wheres ((md5(_where)));\n\nCREATE OR REPLACE FUNCTION where_stats(inwhere text, updatestats boolean default false, conf jsonb default null) RETURNS search_wheres AS $$\nDECLARE\n t timestamptz;\n i interval;\n explain_json jsonb;\n partitions text[];\n sw search_wheres%ROWTYPE;\n inwhere_hash text := md5(inwhere);\n _context text := lower(context(conf));\n _stats_ttl interval := context_stats_ttl(conf);\n _estimated_cost float := context_estimated_cost(conf);\n _estimated_count int := context_estimated_count(conf);\n ro bool := pgstac.readonly(conf);\nBEGIN\n IF ro THEN\n updatestats := FALSE;\n END IF;\n\n IF _context = 'off' THEN\n sw._where := inwhere;\n return sw;\n END IF;\n\n SELECT * INTO sw FROM search_wheres WHERE md5(_where)=inwhere_hash FOR UPDATE;\n\n -- Update statistics if explicitly set, if statistics do not exist, or statistics ttl has expired\n IF NOT updatestats THEN\n RAISE NOTICE 'Checking if update is needed for: % .', inwhere;\n RAISE NOTICE 'Stats Last Updated: %', sw.statslastupdated;\n RAISE NOTICE 'TTL: %, Age: %', _stats_ttl, now() - sw.statslastupdated;\n RAISE NOTICE 'Context: %, Existing Total: %', _context, sw.total_count;\n IF\n (\n sw.statslastupdated IS NULL\n OR (now() - sw.statslastupdated) > _stats_ttl\n OR (context(conf) != 'off' AND sw.total_count IS NULL)\n ) AND NOT ro\n THEN\n updatestats := TRUE;\n END IF;\n END IF;\n\n sw._where := inwhere;\n sw.lastused := now();\n sw.usecount := coalesce(sw.usecount,0) + 1;\n\n IF NOT updatestats THEN\n UPDATE search_wheres SET\n lastused = sw.lastused,\n usecount = sw.usecount\n WHERE md5(_where) = inwhere_hash\n RETURNING * INTO sw\n ;\n RETURN sw;\n END IF;\n\n -- Use explain to get estimated count/cost and a list of the partitions that would be hit by the query\n t := clock_timestamp();\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s', inwhere)\n INTO explain_json;\n RAISE NOTICE 'Time for just the explain: %', clock_timestamp() - t;\n i := clock_timestamp() - t;\n\n sw.statslastupdated := now();\n sw.estimated_count := explain_json->0->'Plan'->'Plan Rows';\n sw.estimated_cost := explain_json->0->'Plan'->'Total Cost';\n sw.time_to_estimate := extract(epoch from i);\n\n RAISE NOTICE 'ESTIMATED_COUNT: % < %', sw.estimated_count, _estimated_count;\n RAISE NOTICE 'ESTIMATED_COST: % < %', sw.estimated_cost, _estimated_cost;\n\n -- Do a full count of rows if context is set to on or if auto is set and estimates are low enough\n IF\n _context = 'on'\n OR\n ( _context = 'auto' AND\n (\n sw.estimated_count < _estimated_count\n AND\n sw.estimated_cost < _estimated_cost\n )\n )\n THEN\n t := clock_timestamp();\n RAISE NOTICE 'Calculating actual count...';\n EXECUTE format(\n 'SELECT count(*) FROM items WHERE %s',\n inwhere\n ) INTO sw.total_count;\n i := clock_timestamp() - t;\n RAISE NOTICE 'Actual Count: % -- %', sw.total_count, i;\n sw.time_to_count := extract(epoch FROM i);\n ELSE\n sw.total_count := NULL;\n sw.time_to_count := NULL;\n END IF;\n\n IF NOT ro THEN\n INSERT INTO search_wheres\n (_where, lastused, usecount, statslastupdated, estimated_count, estimated_cost, time_to_estimate, partitions, total_count, time_to_count)\n SELECT sw._where, sw.lastused, sw.usecount, sw.statslastupdated, sw.estimated_count, sw.estimated_cost, sw.time_to_estimate, sw.partitions, sw.total_count, sw.time_to_count\n ON CONFLICT ((md5(_where)))\n DO UPDATE\n SET\n lastused = sw.lastused,\n usecount = sw.usecount,\n statslastupdated = sw.statslastupdated,\n estimated_count = sw.estimated_count,\n estimated_cost = sw.estimated_cost,\n time_to_estimate = sw.time_to_estimate,\n total_count = sw.total_count,\n time_to_count = sw.time_to_count\n ;\n END IF;\n RETURN sw;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION search_query(\n _search jsonb = '{}'::jsonb,\n updatestats boolean = false,\n _metadata jsonb = '{}'::jsonb\n) RETURNS searches AS $$\nDECLARE\n search searches%ROWTYPE;\n pexplain jsonb;\n t timestamptz;\n i interval;\n _hash text := search_hash(_search, _metadata);\n doupdate boolean := FALSE;\n insertfound boolean := FALSE;\n ro boolean := pgstac.readonly();\nBEGIN\n IF ro THEN\n updatestats := FALSE;\n END IF;\n\n SELECT * INTO search FROM searches\n WHERE hash=_hash;\n\n search.hash := _hash;\n\n -- Calculate the where clause if not already calculated\n IF search._where IS NULL THEN\n search._where := stac_search_to_where(_search);\n ELSE\n doupdate := TRUE;\n END IF;\n\n -- Calculate the order by clause if not already calculated\n IF search.orderby IS NULL THEN\n search.orderby := sort_sqlorderby(_search);\n ELSE\n doupdate := TRUE;\n END IF;\n\n PERFORM where_stats(search._where, updatestats, _search->'conf');\n\n IF NOT ro THEN\n IF NOT doupdate THEN\n INSERT INTO searches (search, _where, orderby, lastused, usecount, metadata)\n VALUES (_search, search._where, search.orderby, clock_timestamp(), 1, _metadata)\n ON CONFLICT (hash) DO NOTHING RETURNING * INTO search;\n IF FOUND THEN\n RETURN search;\n END IF;\n END IF;\n\n UPDATE searches\n SET\n lastused=clock_timestamp(),\n usecount=usecount+1\n WHERE hash=(\n SELECT hash FROM searches\n WHERE hash=_hash\n FOR UPDATE SKIP LOCKED\n );\n IF NOT FOUND THEN\n RAISE NOTICE 'Did not update stats for % due to lock. (This is generally OK)', _search;\n END IF;\n END IF;\n\n RETURN search;\n\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE OR REPLACE FUNCTION search_rows(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN partitions text[] DEFAULT NULL,\n IN _limit int DEFAULT 10\n) RETURNS SETOF items AS $$\nDECLARE\n base_query text;\n query text;\n sdate timestamptz;\n edate timestamptz;\n n int;\n records_left int := _limit;\n timer timestamptz := clock_timestamp();\n full_timer timestamptz := clock_timestamp();\nBEGIN\nIF _where IS NULL OR trim(_where) = '' THEN\n _where = ' TRUE ';\nEND IF;\nRAISE NOTICE 'Getting chunks for % %', _where, _orderby;\n\nbase_query := $q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n LIMIT %L\n$q$;\n\nIF _orderby ILIKE 'datetime d%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 DESC LOOP\n RAISE NOTICE 'Running Query for % to %. %', sdate, edate, age_ms(full_timer);\n query := format(\n base_query,\n sdate,\n edate,\n _where,\n _orderby,\n records_left\n );\n RAISE LOG 'QUERY: %', query;\n timer := clock_timestamp();\n RETURN QUERY EXECUTE query;\n\n GET DIAGNOSTICS n = ROW_COUNT;\n records_left := records_left - n;\n RAISE NOTICE 'Returned %/% Rows From % to %. % to go. Time: %ms', n, _limit, sdate, edate, records_left, age_ms(timer);\n timer := clock_timestamp();\n IF records_left <= 0 THEN\n RAISE NOTICE 'SEARCH_ROWS TOOK %ms', age_ms(full_timer);\n RETURN;\n END IF;\n END LOOP;\nELSIF _orderby ILIKE 'datetime a%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 ASC LOOP\n RAISE NOTICE 'Running Query for % to %. %', sdate, edate, age_ms(full_timer);\n query := format(\n base_query,\n sdate,\n edate,\n _where,\n _orderby,\n records_left\n );\n RAISE LOG 'QUERY: %', query;\n timer := clock_timestamp();\n RETURN QUERY EXECUTE query;\n\n GET DIAGNOSTICS n = ROW_COUNT;\n records_left := records_left - n;\n RAISE NOTICE 'Returned %/% Rows From % to %. % to go. Time: %ms', n, _limit, sdate, edate, records_left, age_ms(timer);\n timer := clock_timestamp();\n IF records_left <= 0 THEN\n RAISE NOTICE 'SEARCH_ROWS TOOK %ms', age_ms(full_timer);\n RETURN;\n END IF;\n END LOOP;\nELSE\n query := format($q$\n SELECT * FROM items\n WHERE %s\n ORDER BY %s\n LIMIT %L\n $q$, _where, _orderby, _limit\n );\n RAISE LOG 'QUERY: %', query;\n timer := clock_timestamp();\n RETURN QUERY EXECUTE query;\n RAISE NOTICE 'FULL QUERY TOOK %ms', age_ms(timer);\nEND IF;\nRAISE NOTICE 'SEARCH_ROWS TOOK %ms', age_ms(full_timer);\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE UNLOGGED TABLE format_item_cache(\n id text,\n collection text,\n fields text,\n hydrated bool,\n output jsonb,\n lastused timestamptz DEFAULT now(),\n usecount int DEFAULT 1,\n timetoformat float,\n PRIMARY KEY (collection, id, fields, hydrated)\n);\nCREATE INDEX ON format_item_cache (lastused);\n\nCREATE OR REPLACE FUNCTION format_item(_item items, _fields jsonb DEFAULT '{}', _hydrated bool DEFAULT TRUE) RETURNS jsonb AS $$\nDECLARE\n cache bool := get_setting_bool('format_cache');\n _output jsonb := null;\n t timestamptz := clock_timestamp();\nBEGIN\n IF cache THEN\n SELECT output INTO _output FROM format_item_cache\n WHERE id=_item.id AND collection=_item.collection AND fields=_fields::text AND hydrated=_hydrated;\n END IF;\n IF _output IS NULL THEN\n IF _hydrated THEN\n _output := content_hydrate(_item, _fields);\n ELSE\n _output := content_nonhydrated(_item, _fields);\n END IF;\n END IF;\n IF cache THEN\n INSERT INTO format_item_cache (id, collection, fields, hydrated, output, timetoformat)\n VALUES (_item.id, _item.collection, _fields::text, _hydrated, _output, age_ms(t))\n ON CONFLICT(collection, id, fields, hydrated) DO\n UPDATE\n SET lastused=now(), usecount = format_item_cache.usecount + 1\n ;\n END IF;\n RETURN _output;\n\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION search(_search jsonb = '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n searches searches%ROWTYPE;\n _where text;\n orderby text;\n search_where search_wheres%ROWTYPE;\n total_count bigint;\n token record;\n token_prev boolean;\n token_item items%ROWTYPE;\n token_where text;\n full_where text;\n init_ts timestamptz := clock_timestamp();\n timer timestamptz := clock_timestamp();\n hydrate bool := NOT (_search->'conf'->>'nohydrate' IS NOT NULL AND (_search->'conf'->>'nohydrate')::boolean = true);\n prev text;\n next text;\n context jsonb;\n collection jsonb;\n out_records jsonb;\n out_len int;\n _limit int := coalesce((_search->>'limit')::int, 10);\n _querylimit int;\n _fields jsonb := coalesce(_search->'fields', '{}'::jsonb);\n has_prev boolean := FALSE;\n has_next boolean := FALSE;\nBEGIN\n searches := search_query(_search);\n _where := searches._where;\n orderby := searches.orderby;\n search_where := where_stats(_where);\n total_count := coalesce(search_where.total_count, search_where.estimated_count);\n RAISE NOTICE 'SEARCH:TOKEN: %', _search->>'token';\n token := get_token_record(_search->>'token');\n RAISE NOTICE '***TOKEN: %', token;\n _querylimit := _limit + 1;\n IF token IS NOT NULL THEN\n token_prev := token.prev;\n token_item := token.item;\n token_where := get_token_filter(_search->'sortby', token_item, token_prev, FALSE);\n RAISE LOG 'TOKEN_WHERE: % (%ms from search start)', token_where, age_ms(timer);\n IF token_prev THEN -- if we are using a prev token, we know has_next is true\n RAISE LOG 'There is a previous token, so automatically setting has_next to true';\n has_next := TRUE;\n orderby := sort_sqlorderby(_search, TRUE);\n ELSE\n RAISE LOG 'There is a next token, so automatically setting has_prev to true';\n has_prev := TRUE;\n\n END IF;\n ELSE -- if there was no token, we know there is no prev\n RAISE LOG 'There is no token, so we know there is no prev. setting has_prev to false';\n has_prev := FALSE;\n END IF;\n\n full_where := concat_ws(' AND ', _where, token_where);\n RAISE NOTICE 'FULL WHERE CLAUSE: %', full_where;\n RAISE NOTICE 'Time to get counts and build query %', age_ms(timer);\n timer := clock_timestamp();\n\n IF hydrate THEN\n RAISE NOTICE 'Getting hydrated data.';\n ELSE\n RAISE NOTICE 'Getting non-hydrated data.';\n END IF;\n RAISE NOTICE 'CACHE SET TO %', get_setting_bool('format_cache');\n RAISE NOTICE 'Time to set hydration/formatting %', age_ms(timer);\n timer := clock_timestamp();\n SELECT jsonb_agg(format_item(i, _fields, hydrate)) INTO out_records\n FROM search_rows(\n full_where,\n orderby,\n search_where.partitions,\n _querylimit\n ) as i;\n\n RAISE NOTICE 'Time to fetch rows %', age_ms(timer);\n timer := clock_timestamp();\n\n\n IF token_prev THEN\n out_records := flip_jsonb_array(out_records);\n END IF;\n\n RAISE NOTICE 'Query returned % records.', jsonb_array_length(out_records);\n RAISE LOG 'TOKEN: % %', token_item.id, token_item.collection;\n RAISE LOG 'RECORD_1: % %', out_records->0->>'id', out_records->0->>'collection';\n RAISE LOG 'RECORD-1: % %', out_records->-1->>'id', out_records->-1->>'collection';\n\n -- REMOVE records that were from our token\n IF out_records->0->>'id' = token_item.id AND out_records->0->>'collection' = token_item.collection THEN\n out_records := out_records - 0;\n ELSIF out_records->-1->>'id' = token_item.id AND out_records->-1->>'collection' = token_item.collection THEN\n out_records := out_records - -1;\n END IF;\n\n out_len := jsonb_array_length(out_records);\n\n IF out_len = _limit + 1 THEN\n IF token_prev THEN\n has_prev := TRUE;\n out_records := out_records - 0;\n ELSE\n has_next := TRUE;\n out_records := out_records - -1;\n END IF;\n END IF;\n\n IF has_next THEN\n next := concat(out_records->-1->>'collection', ':', out_records->-1->>'id');\n RAISE NOTICE 'HAS NEXT | %', next;\n END IF;\n\n IF has_prev THEN\n prev := concat(out_records->0->>'collection', ':', out_records->0->>'id');\n RAISE NOTICE 'HAS PREV | %', prev;\n END IF;\n\n RAISE NOTICE 'Time to get prev/next %', age_ms(timer);\n timer := clock_timestamp();\n\n IF context(_search->'conf') != 'off' THEN\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'matched', total_count,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\n ELSE\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\n END IF;\n\n collection := jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb),\n 'next', next,\n 'prev', prev,\n 'context', context\n );\n\n IF get_setting_bool('timing', _search->'conf') THEN\n collection = collection || jsonb_build_object('timing', age_ms(init_ts));\n END IF;\n\n RAISE NOTICE 'Time to build final json %', age_ms(timer);\n timer := clock_timestamp();\n\n RAISE NOTICE 'Total Time: %', age_ms(current_timestamp);\n RAISE NOTICE 'RETURNING % records. NEXT: %. PREV: %', collection->'context'->>'returned', collection->>'next', collection->>'prev';\n RETURN collection;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION search_cursor(_search jsonb = '{}'::jsonb) RETURNS refcursor AS $$\nDECLARE\n curs refcursor;\n searches searches%ROWTYPE;\n _where text;\n _orderby text;\n q text;\n\nBEGIN\n searches := search_query(_search);\n _where := searches._where;\n _orderby := searches.orderby;\n\n OPEN curs FOR\n WITH p AS (\n SELECT * FROM partition_queries(_where, _orderby) p\n )\n SELECT\n CASE WHEN EXISTS (SELECT 1 FROM p) THEN\n (SELECT format($q$\n SELECT * FROM (\n %s\n ) total\n $q$,\n string_agg(\n format($q$ SELECT * FROM ( %s ) AS sub $q$, p),\n '\n UNION ALL\n '\n )\n ))\n ELSE NULL\n END FROM p;\n RETURN curs;\nEND;\n$$ LANGUAGE PLPGSQL;\nCREATE OR REPLACE VIEW collections_asitems AS\nSELECT\n id,\n geometry,\n 'collections' AS collection,\n datetime,\n end_datetime,\n jsonb_build_object(\n 'properties', content - '{links,assets,stac_version,stac_extensions}',\n 'links', content->'links',\n 'assets', content->'assets',\n 'stac_version', content->'stac_version',\n 'stac_extensions', content->'stac_extensions'\n ) AS content,\n content as collectionjson\nFROM collections;\n\n\nCREATE OR REPLACE FUNCTION collection_search_matched(\n IN _search jsonb DEFAULT '{}'::jsonb,\n OUT matched bigint\n) RETURNS bigint AS $$\nDECLARE\n _where text := stac_search_to_where(_search);\nBEGIN\n EXECUTE format(\n $query$\n SELECT\n count(*)\n FROM\n collections_asitems\n WHERE %s\n ;\n $query$,\n _where\n ) INTO matched;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION collection_search_rows(\n _search jsonb DEFAULT '{}'::jsonb\n) RETURNS SETOF jsonb AS $$\nDECLARE\n _where text := stac_search_to_where(_search);\n _limit int := coalesce((_search->>'limit')::int, 10);\n _fields jsonb := coalesce(_search->'fields', '{}'::jsonb);\n _orderby text;\n _offset int := COALESCE((_search->>'offset')::int, 0);\nBEGIN\n _orderby := sort_sqlorderby(\n jsonb_build_object(\n 'sortby',\n coalesce(\n _search->'sortby',\n '[{\"field\": \"id\", \"direction\": \"asc\"}]'::jsonb\n )\n )\n );\n RETURN QUERY EXECUTE format(\n $query$\n SELECT\n jsonb_fields(collectionjson, %L) as c\n FROM\n collections_asitems\n WHERE %s\n ORDER BY %s\n LIMIT %L\n OFFSET %L\n ;\n $query$,\n _fields,\n _where,\n _orderby,\n _limit,\n _offset\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION collection_search(\n _search jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\nDECLARE\n out_records jsonb;\n number_matched bigint := collection_search_matched(_search);\n number_returned bigint;\n _limit int := coalesce((_search->>'limit')::float::int, 10);\n _offset int := coalesce((_search->>'offset')::float::int, 0);\n links jsonb := '[]';\n ret jsonb;\n base_url text:= concat(rtrim(base_url(_search->'conf'),'/'), '/collections');\n prevoffset int;\n nextoffset int;\nBEGIN\n SELECT\n coalesce(jsonb_agg(c), '[]')\n INTO out_records\n FROM collection_search_rows(_search) c;\n\n number_returned := jsonb_array_length(out_records);\n\n IF _limit <= number_matched THEN --need to have paging links\n nextoffset := least(_offset + _limit, number_matched - 1);\n prevoffset := greatest(_offset - _limit, 0);\n IF _offset = 0 THEN -- no previous paging\n\n links := jsonb_build_array(\n jsonb_build_object(\n 'rel', 'next',\n 'type', 'application/json',\n 'method', 'GET' ,\n 'href', base_url,\n 'body', jsonb_build_object('offset', nextoffset),\n 'merge', TRUE\n )\n );\n ELSE\n links := jsonb_build_array(\n jsonb_build_object(\n 'rel', 'prev',\n 'type', 'application/json',\n 'method', 'GET' ,\n 'href', base_url,\n 'body', jsonb_build_object('offset', prevoffset),\n 'merge', TRUE\n ),\n jsonb_build_object(\n 'rel', 'next',\n 'type', 'application/json',\n 'method', 'GET' ,\n 'href', base_url,\n 'body', jsonb_build_object('offset', nextoffset),\n 'merge', TRUE\n )\n );\n END IF;\n END IF;\n\n ret := jsonb_build_object(\n 'collections', out_records,\n 'context', jsonb_build_object(\n 'limit', _limit,\n 'matched', number_matched,\n 'returned', number_returned\n ),\n 'links', links\n );\n RETURN ret;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION tileenvelope(zoom int, x int, y int) RETURNS geometry AS $$\nWITH t AS (\n SELECT\n 20037508.3427892 as merc_max,\n -20037508.3427892 as merc_min,\n (2 * 20037508.3427892) / (2 ^ zoom) as tile_size\n)\nSELECT st_makeenvelope(\n merc_min + (tile_size * x),\n merc_max - (tile_size * (y + 1)),\n merc_min + (tile_size * (x + 1)),\n merc_max - (tile_size * y),\n 3857\n) FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;DROP FUNCTION IF EXISTS mercgrid;\n\n\nCREATE OR REPLACE FUNCTION ftime() RETURNS interval as $$\nSELECT age(clock_timestamp(), transaction_timestamp());\n$$ LANGUAGE SQL;\nSET SEARCH_PATH to pgstac, public;\n\nDROP FUNCTION IF EXISTS geometrysearch;\nCREATE OR REPLACE FUNCTION geometrysearch(\n IN geom geometry,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE, -- Return as soon as the passed in geometry is full covered\n IN skipcovered boolean DEFAULT TRUE -- Skip any items that would show up completely under the previous items\n) RETURNS jsonb AS $$\nDECLARE\n search searches%ROWTYPE;\n curs refcursor;\n _where text;\n query text;\n iter_record items%ROWTYPE;\n out_records jsonb := '{}'::jsonb[];\n exit_flag boolean := FALSE;\n counter int := 1;\n scancounter int := 1;\n remaining_limit int := _scanlimit;\n tilearea float;\n unionedgeom geometry;\n clippedgeom geometry;\n unionedgeom_area float := 0;\n prev_area float := 0;\n excludes text[];\n includes text[];\n\nBEGIN\n DROP TABLE IF EXISTS pgstac_results;\n CREATE TEMP TABLE pgstac_results (content jsonb) ON COMMIT DROP;\n\n -- If the passed in geometry is not an area set exitwhenfull and skipcovered to false\n IF ST_GeometryType(geom) !~* 'polygon' THEN\n RAISE NOTICE 'GEOMETRY IS NOT AN AREA';\n skipcovered = FALSE;\n exitwhenfull = FALSE;\n END IF;\n\n -- If skipcovered is true then you will always want to exit when the passed in geometry is full\n IF skipcovered THEN\n exitwhenfull := TRUE;\n END IF;\n\n SELECT * INTO search FROM searches WHERE hash=queryhash;\n\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Search with Query Hash % Not Found', queryhash;\n END IF;\n\n tilearea := st_area(geom);\n _where := format('%s AND st_intersects(geometry, %L::geometry)', search._where, geom);\n\n\n FOR query IN SELECT * FROM partition_queries(_where, search.orderby) LOOP\n query := format('%s LIMIT %L', query, remaining_limit);\n RAISE NOTICE '%', query;\n OPEN curs FOR EXECUTE query;\n LOOP\n FETCH curs INTO iter_record;\n EXIT WHEN NOT FOUND;\n IF exitwhenfull OR skipcovered THEN -- If we are not using exitwhenfull or skipcovered, we do not need to do expensive geometry operations\n clippedgeom := st_intersection(geom, iter_record.geometry);\n\n IF unionedgeom IS NULL THEN\n unionedgeom := clippedgeom;\n ELSE\n unionedgeom := st_union(unionedgeom, clippedgeom);\n END IF;\n\n unionedgeom_area := st_area(unionedgeom);\n\n IF skipcovered AND prev_area = unionedgeom_area THEN\n scancounter := scancounter + 1;\n CONTINUE;\n END IF;\n\n prev_area := unionedgeom_area;\n\n RAISE NOTICE '% % % %', unionedgeom_area/tilearea, counter, scancounter, ftime();\n END IF;\n RAISE NOTICE '% %', iter_record, content_hydrate(iter_record, fields);\n INSERT INTO pgstac_results (content) VALUES (content_hydrate(iter_record, fields));\n\n IF counter >= _limit\n OR scancounter > _scanlimit\n OR ftime() > _timelimit\n OR (exitwhenfull AND unionedgeom_area >= tilearea)\n THEN\n exit_flag := TRUE;\n EXIT;\n END IF;\n counter := counter + 1;\n scancounter := scancounter + 1;\n\n END LOOP;\n CLOSE curs;\n EXIT WHEN exit_flag;\n remaining_limit := _scanlimit - scancounter;\n END LOOP;\n\n SELECT jsonb_agg(content) INTO out_records FROM pgstac_results WHERE content IS NOT NULL;\n\n RETURN jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb)\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDROP FUNCTION IF EXISTS geojsonsearch;\nCREATE OR REPLACE FUNCTION geojsonsearch(\n IN geojson jsonb,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_geomfromgeojson(geojson),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS xyzsearch;\nCREATE OR REPLACE FUNCTION xyzsearch(\n IN _x int,\n IN _y int,\n IN _z int,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_transform(tileenvelope(_z, _x, _y), 4326),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE PROCEDURE analyze_items() AS $$\nDECLARE\n q text;\n timeout_ts timestamptz;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n SELECT format('ANALYZE (VERBOSE, SKIP_LOCKED) %I;', relname) INTO q\n FROM pg_stat_user_tables\n WHERE relname like '_item%' AND (n_mod_since_analyze>0 OR last_analyze IS NULL) LIMIT 1;\n IF NOT FOUND THEN\n EXIT;\n END IF;\n RAISE NOTICE '%', q;\n EXECUTE q;\n COMMIT;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE PROCEDURE validate_constraints() AS $$\nDECLARE\n q text;\nBEGIN\n FOR q IN\n SELECT\n FORMAT(\n 'ALTER TABLE %I.%I VALIDATE CONSTRAINT %I;',\n nsp.nspname,\n cls.relname,\n con.conname\n )\n\n FROM pg_constraint AS con\n JOIN pg_class AS cls\n ON con.conrelid = cls.oid\n JOIN pg_namespace AS nsp\n ON cls.relnamespace = nsp.oid\n WHERE convalidated = FALSE AND contype in ('c','f')\n AND nsp.nspname = 'pgstac'\n LOOP\n RAISE NOTICE '%', q;\n PERFORM run_or_queue(q);\n COMMIT;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION collection_extent(_collection text, runupdate boolean default false) RETURNS jsonb AS $$\nDECLARE\n geom_extent geometry;\n mind timestamptz;\n maxd timestamptz;\n extent jsonb;\nBEGIN\n IF runupdate THEN\n PERFORM update_partition_stats_q(partition)\n FROM partitions_view WHERE collection=_collection;\n END IF;\n SELECT\n min(lower(dtrange)),\n max(upper(edtrange)),\n st_extent(spatial)\n INTO\n mind,\n maxd,\n geom_extent\n FROM partitions_view\n WHERE collection=_collection;\n\n IF geom_extent IS NOT NULL AND mind IS NOT NULL AND maxd IS NOT NULL THEN\n extent := jsonb_build_object(\n 'extent', jsonb_build_object(\n 'spatial', jsonb_build_object(\n 'bbox', to_jsonb(array[array[st_xmin(geom_extent), st_ymin(geom_extent), st_xmax(geom_extent), st_ymax(geom_extent)]])\n ),\n 'temporal', jsonb_build_object(\n 'interval', to_jsonb(array[array[mind, maxd]])\n )\n )\n );\n RETURN extent;\n END IF;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('id', '{\"title\": \"Item ID\",\"description\": \"Item identifier\",\"$ref\": \"https://schemas.stacspec.org/v1.0.0/item-spec/json-schema/item.json#/definitions/core/allOf/2/properties/id\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('geometry', '{\"title\": \"Item Geometry\",\"description\": \"Item Geometry\",\"$ref\": \"https://geojson.org/schema/Feature.json\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('datetime','{\"description\": \"Datetime\",\"type\": \"string\",\"title\": \"Acquired\",\"format\": \"date-time\",\"pattern\": \"(\\\\+00:00|Z)$\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDELETE FROM queryables a USING queryables b\n WHERE a.name = b.name AND a.collection_ids IS NOT DISTINCT FROM b.collection_ids AND a.id > b.id;\n\n\nINSERT INTO pgstac_settings (name, value) VALUES\n ('context', 'off'),\n ('context_estimated_count', '100000'),\n ('context_estimated_cost', '100000'),\n ('context_stats_ttl', '1 day'),\n ('default_filter_lang', 'cql2-json'),\n ('additional_properties', 'true'),\n ('use_queue', 'false'),\n ('queue_timeout', '10 minutes'),\n ('update_collection_extent', 'false'),\n ('format_cache', 'false'),\n ('readonly', 'false')\nON CONFLICT DO NOTHING\n;\n\nALTER FUNCTION to_text COST 5000;\nALTER FUNCTION to_float COST 5000;\nALTER FUNCTION to_int COST 5000;\nALTER FUNCTION to_tstz COST 5000;\nALTER FUNCTION to_text_array COST 5000;\n\nALTER FUNCTION update_partition_stats SECURITY DEFINER;\nALTER FUNCTION partition_after_triggerfunc SECURITY DEFINER;\nALTER FUNCTION drop_table_constraints SECURITY DEFINER;\nALTER FUNCTION create_table_constraints SECURITY DEFINER;\nALTER FUNCTION check_partition SECURITY DEFINER;\nALTER FUNCTION repartition SECURITY DEFINER;\nALTER FUNCTION where_stats SECURITY DEFINER;\nALTER FUNCTION search_query SECURITY DEFINER;\nALTER FUNCTION format_item SECURITY DEFINER;\nALTER FUNCTION maintain_index SECURITY DEFINER;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nGRANT ALL ON SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac to pgstac_admin;\n\n-- pgstac_read role limited to using function apis\nGRANT EXECUTE ON FUNCTION search TO pgstac_read;\nGRANT EXECUTE ON FUNCTION search_query TO pgstac_read;\nGRANT EXECUTE ON FUNCTION item_by_id TO pgstac_read;\nGRANT EXECUTE ON FUNCTION get_item TO pgstac_read;\nGRANT SELECT ON ALL TABLES IN SCHEMA pgstac TO pgstac_read;\n\n\nGRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_ingest;\nGRANT USAGE ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_ingest;\n\nREVOKE ALL PRIVILEGES ON PROCEDURE run_queued_queries FROM public;\nGRANT ALL ON PROCEDURE run_queued_queries TO pgstac_admin;\n\nREVOKE ALL PRIVILEGES ON FUNCTION run_queued_queries_intransaction FROM public;\nGRANT ALL ON FUNCTION run_queued_queries_intransaction TO pgstac_admin;\n\nRESET ROLE;\n\nSET ROLE pgstac_ingest;\nSELECT update_partition_stats_q(partition) FROM partitions_view;\nSELECT set_version('0.8.5');\n" }, { "path": "src/pgstac/migrations/pgstac.0.8.6-0.9.0.sql", "content": "SET client_min_messages TO WARNING;\nSET SEARCH_PATH to pgstac, public;\nRESET ROLE;\nDO $$\nDECLARE\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='postgis') THEN\n CREATE EXTENSION IF NOT EXISTS postgis;\n END IF;\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='btree_gist') THEN\n CREATE EXTENSION IF NOT EXISTS btree_gist;\n END IF;\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='unaccent') THEN\n CREATE EXTENSION IF NOT EXISTS unaccent;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_admin;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_read;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_ingest;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n\nGRANT pgstac_admin TO current_user;\n\n-- Function to make sure pgstac_admin is the owner of items\nCREATE OR REPLACE FUNCTION pgstac_admin_owns() RETURNS VOID AS $$\nDECLARE\n f RECORD;\nBEGIN\n FOR f IN (\n SELECT\n concat(\n oid::regproc::text,\n '(',\n coalesce(pg_get_function_identity_arguments(oid),''),\n ')'\n ) AS name,\n CASE prokind WHEN 'f' THEN 'FUNCTION' WHEN 'p' THEN 'PROCEDURE' WHEN 'a' THEN 'AGGREGATE' END as typ\n FROM pg_proc\n WHERE\n pronamespace=to_regnamespace('pgstac')\n AND proowner != to_regrole('pgstac_admin')\n AND proname NOT LIKE 'pg_stat%'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n FOR f IN (\n SELECT\n oid::regclass::text as name,\n CASE relkind\n WHEN 'i' THEN 'INDEX'\n WHEN 'I' THEN 'INDEX'\n WHEN 'p' THEN 'TABLE'\n WHEN 'r' THEN 'TABLE'\n WHEN 'v' THEN 'VIEW'\n WHEN 'S' THEN 'SEQUENCE'\n ELSE NULL\n END as typ\n FROM pg_class\n WHERE relnamespace=to_regnamespace('pgstac') and relowner != to_regrole('pgstac_admin') AND relkind IN ('r','p','v','S') AND relname NOT LIKE 'pg_stat'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\nSELECT pgstac_admin_owns();\n\nCREATE SCHEMA IF NOT EXISTS pgstac AUTHORIZATION pgstac_admin;\n\nGRANT ALL ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_admin;\n\nALTER ROLE pgstac_admin SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_read SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_ingest SET SEARCH_PATH TO pgstac, public;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\n\nGRANT pgstac_read TO pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\n\nSET ROLE pgstac_admin;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\nRESET ROLE;\n\nSET ROLE pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\nRESET ROLE;\n\nSET SEARCH_PATH TO pgstac, public;\nSET ROLE pgstac_admin;\n\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS analyze_items;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS validate_constraints;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n-- Install these idempotently as migrations do not put them before trying to modify the collections table\n\n\nCREATE OR REPLACE FUNCTION collection_geom(content jsonb)\nRETURNS geometry AS $$\n WITH box AS (SELECT content->'extent'->'spatial'->'bbox'->0 as box)\n SELECT\n st_makeenvelope(\n (box->>0)::float,\n (box->>1)::float,\n (box->>2)::float,\n (box->>3)::float,\n 4326\n )\n FROM box;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION collection_datetime(content jsonb)\nRETURNS timestamptz AS $$\n SELECT\n CASE\n WHEN\n (content->'extent'->'temporal'->'interval'->0->>0) IS NULL\n THEN '-infinity'::timestamptz\n ELSE\n (content->'extent'->'temporal'->'interval'->0->>0)::timestamptz\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION collection_enddatetime(content jsonb)\nRETURNS timestamptz AS $$\n SELECT\n CASE\n WHEN\n (content->'extent'->'temporal'->'interval'->0->>1) IS NULL\n THEN 'infinity'::timestamptz\n ELSE\n (content->'extent'->'temporal'->'interval'->0->>1)::timestamptz\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n-- BEGIN migra calculated SQL\nset check_function_bodies = off;\n\nCREATE OR REPLACE FUNCTION pgstac.search_fromhash(_hash text)\n RETURNS searches\n LANGUAGE sql\n STRICT\nAS $function$\n SELECT * FROM search_query((SELECT search FROM searches WHERE hash=_hash LIMIT 1));\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.collection_search(_search jsonb DEFAULT '{}'::jsonb)\n RETURNS jsonb\n LANGUAGE plpgsql\n STABLE PARALLEL SAFE\nAS $function$\nDECLARE\n out_records jsonb;\n number_matched bigint := collection_search_matched(_search);\n number_returned bigint;\n _limit int := coalesce((_search->>'limit')::float::int, 10);\n _offset int := coalesce((_search->>'offset')::float::int, 0);\n links jsonb := '[]';\n ret jsonb;\n base_url text:= concat(rtrim(base_url(_search->'conf'),'/'), '/collections');\n prevoffset int;\n nextoffset int;\nBEGIN\n SELECT\n coalesce(jsonb_agg(c), '[]')\n INTO out_records\n FROM collection_search_rows(_search) c;\n\n number_returned := jsonb_array_length(out_records);\n\n IF _limit <= number_matched THEN --need to have paging links\n nextoffset := least(_offset + _limit, number_matched - 1);\n prevoffset := greatest(_offset - _limit, 0);\n IF _offset = 0 THEN -- no previous paging\n\n links := jsonb_build_array(\n jsonb_build_object(\n 'rel', 'next',\n 'type', 'application/json',\n 'method', 'GET' ,\n 'href', base_url,\n 'body', jsonb_build_object('offset', nextoffset),\n 'merge', TRUE\n )\n );\n ELSE\n links := jsonb_build_array(\n jsonb_build_object(\n 'rel', 'prev',\n 'type', 'application/json',\n 'method', 'GET' ,\n 'href', base_url,\n 'body', jsonb_build_object('offset', prevoffset),\n 'merge', TRUE\n ),\n jsonb_build_object(\n 'rel', 'next',\n 'type', 'application/json',\n 'method', 'GET' ,\n 'href', base_url,\n 'body', jsonb_build_object('offset', nextoffset),\n 'merge', TRUE\n )\n );\n END IF;\n END IF;\n\n ret := jsonb_build_object(\n 'collections', out_records,\n 'numberMatched', number_matched,\n 'numberReturned', number_returned,\n 'links', links\n );\n RETURN ret;\n\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.cql2_query(j jsonb, wrapper text DEFAULT NULL::text)\n RETURNS text\n LANGUAGE plpgsql\n STABLE\nAS $function$\n#variable_conflict use_variable\nDECLARE\n args jsonb := j->'args';\n arg jsonb;\n op text := lower(j->>'op');\n cql2op RECORD;\n literal text;\n _wrapper text;\n leftarg text;\n rightarg text;\n prop text;\n extra_props bool := pgstac.additional_properties();\nBEGIN\n IF j IS NULL OR (op IS NOT NULL AND args IS NULL) THEN\n RETURN NULL;\n END IF;\n RAISE NOTICE 'CQL2_QUERY: %', j;\n\n -- check if all properties are represented in the queryables\n IF NOT extra_props THEN\n FOR prop IN\n SELECT DISTINCT p->>0\n FROM jsonb_path_query(j, 'strict $.**.property') p\n WHERE p->>0 NOT IN ('id', 'datetime', 'geometry', 'end_datetime', 'collection')\n LOOP\n IF (queryable(prop)).nulled_wrapper IS NULL THEN\n RAISE EXCEPTION 'Term % is not found in queryables.', prop;\n END IF;\n END LOOP;\n END IF;\n\n IF j ? 'filter' THEN\n RETURN cql2_query(j->'filter');\n END IF;\n\n IF j ? 'upper' THEN\n RETURN cql2_query(jsonb_build_object('op', 'upper', 'args', j->'upper'));\n END IF;\n\n IF j ? 'lower' THEN\n RETURN cql2_query(jsonb_build_object('op', 'lower', 'args', j->'lower'));\n END IF;\n\n -- Temporal Query\n IF op ilike 't_%' or op = 'anyinteracts' THEN\n RETURN temporal_op_query(op, args);\n END IF;\n\n -- If property is a timestamp convert it to text to use with\n -- general operators\n IF j ? 'timestamp' THEN\n RETURN format('%L::timestamptz', to_tstz(j->'timestamp'));\n END IF;\n IF j ? 'interval' THEN\n RAISE EXCEPTION 'Please use temporal operators when using intervals.';\n RETURN NONE;\n END IF;\n\n -- Spatial Query\n IF op ilike 's_%' or op = 'intersects' THEN\n RETURN spatial_op_query(op, args);\n END IF;\n\n IF op IN ('a_equals','a_contains','a_contained_by','a_overlaps') THEN\n IF args->0 ? 'property' THEN\n leftarg := format('to_text_array(%s)', (queryable(args->0->>'property')).path);\n END IF;\n IF args->1 ? 'property' THEN\n rightarg := format('to_text_array(%s)', (queryable(args->1->>'property')).path);\n END IF;\n RETURN FORMAT(\n '%s %s %s',\n COALESCE(leftarg, quote_literal(to_text_array(args->0))),\n CASE op\n WHEN 'a_equals' THEN '='\n WHEN 'a_contains' THEN '@>'\n WHEN 'a_contained_by' THEN '<@'\n WHEN 'a_overlaps' THEN '&&'\n END,\n COALESCE(rightarg, quote_literal(to_text_array(args->1)))\n );\n END IF;\n\n IF op = 'in' THEN\n RAISE NOTICE 'IN : % % %', args, jsonb_build_array(args->0), args->1;\n args := jsonb_build_array(args->0) || (args->1);\n RAISE NOTICE 'IN2 : %', args;\n END IF;\n\n\n\n IF op = 'between' THEN\n args = jsonb_build_array(\n args->0,\n args->1,\n args->2\n );\n END IF;\n\n -- Make sure that args is an array and run cql2_query on\n -- each element of the array\n RAISE NOTICE 'ARGS PRE: %', args;\n IF j ? 'args' THEN\n IF jsonb_typeof(args) != 'array' THEN\n args := jsonb_build_array(args);\n END IF;\n\n IF jsonb_path_exists(args, '$[*] ? (@.property == \"id\" || @.property == \"datetime\" || @.property == \"end_datetime\" || @.property == \"collection\")') THEN\n wrapper := NULL;\n ELSE\n -- if any of the arguments are a property, try to get the property_wrapper\n FOR arg IN SELECT jsonb_path_query(args, '$[*] ? (@.property != null)') LOOP\n RAISE NOTICE 'Arg: %', arg;\n wrapper := (queryable(arg->>'property')).nulled_wrapper;\n RAISE NOTICE 'Property: %, Wrapper: %', arg, wrapper;\n IF wrapper IS NOT NULL THEN\n EXIT;\n END IF;\n END LOOP;\n\n -- if the property was not in queryables, see if any args were numbers\n IF\n wrapper IS NULL\n AND jsonb_path_exists(args, '$[*] ? (@.type()==\"number\")')\n THEN\n wrapper := 'to_float';\n END IF;\n wrapper := coalesce(wrapper, 'to_text');\n END IF;\n\n SELECT jsonb_agg(cql2_query(a, wrapper))\n INTO args\n FROM jsonb_array_elements(args) a;\n END IF;\n RAISE NOTICE 'ARGS: %', args;\n\n IF op IN ('and', 'or') THEN\n RETURN\n format(\n '(%s)',\n array_to_string(to_text_array(args), format(' %s ', upper(op)))\n );\n END IF;\n\n IF op = 'in' THEN\n RAISE NOTICE 'IN -- % %', args->0, to_text(args->0);\n RETURN format(\n '%s IN (%s)',\n to_text(args->0),\n array_to_string((to_text_array(args))[2:], ',')\n );\n END IF;\n\n -- Look up template from cql2_ops\n IF j ? 'op' THEN\n SELECT * INTO cql2op FROM cql2_ops WHERE cql2_ops.op ilike op;\n IF FOUND THEN\n -- If specific index set in queryables for a property cast other arguments to that type\n\n RETURN format(\n cql2op.template,\n VARIADIC (to_text_array(args))\n );\n ELSE\n RAISE EXCEPTION 'Operator % Not Supported.', op;\n END IF;\n END IF;\n\n\n IF wrapper IS NOT NULL THEN\n RAISE NOTICE 'Wrapping % with %', j, wrapper;\n IF j ? 'property' THEN\n RETURN format('%I(%s)', wrapper, (queryable(j->>'property')).path);\n ELSE\n RETURN format('%I(%L)', wrapper, j);\n END IF;\n ELSIF j ? 'property' THEN\n RETURN quote_ident(j->>'property');\n END IF;\n\n RETURN quote_literal(to_text(j));\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.geometrysearch(geom geometry, queryhash text, fields jsonb DEFAULT NULL::jsonb, _scanlimit integer DEFAULT 10000, _limit integer DEFAULT 100, _timelimit interval DEFAULT '00:00:05'::interval, exitwhenfull boolean DEFAULT true, skipcovered boolean DEFAULT true)\n RETURNS jsonb\n LANGUAGE plpgsql\nAS $function$\nDECLARE\n search searches%ROWTYPE;\n curs refcursor;\n _where text;\n query text;\n iter_record items%ROWTYPE;\n out_records jsonb := '{}'::jsonb[];\n exit_flag boolean := FALSE;\n counter int := 1;\n scancounter int := 1;\n remaining_limit int := _scanlimit;\n tilearea float;\n unionedgeom geometry;\n clippedgeom geometry;\n unionedgeom_area float := 0;\n prev_area float := 0;\n excludes text[];\n includes text[];\n\nBEGIN\n DROP TABLE IF EXISTS pgstac_results;\n CREATE TEMP TABLE pgstac_results (content jsonb) ON COMMIT DROP;\n\n -- If the passed in geometry is not an area set exitwhenfull and skipcovered to false\n IF ST_GeometryType(geom) !~* 'polygon' THEN\n RAISE NOTICE 'GEOMETRY IS NOT AN AREA';\n skipcovered = FALSE;\n exitwhenfull = FALSE;\n END IF;\n\n -- If skipcovered is true then you will always want to exit when the passed in geometry is full\n IF skipcovered THEN\n exitwhenfull := TRUE;\n END IF;\n\n search := search_fromhash(queryhash);\n\n IF search IS NULL THEN\n RAISE EXCEPTION 'Search with Query Hash % Not Found', queryhash;\n END IF;\n\n tilearea := st_area(geom);\n _where := format('%s AND st_intersects(geometry, %L::geometry)', search._where, geom);\n\n\n FOR query IN SELECT * FROM partition_queries(_where, search.orderby) LOOP\n query := format('%s LIMIT %L', query, remaining_limit);\n RAISE NOTICE '%', query;\n OPEN curs FOR EXECUTE query;\n LOOP\n FETCH curs INTO iter_record;\n EXIT WHEN NOT FOUND;\n IF exitwhenfull OR skipcovered THEN -- If we are not using exitwhenfull or skipcovered, we do not need to do expensive geometry operations\n clippedgeom := st_intersection(geom, iter_record.geometry);\n\n IF unionedgeom IS NULL THEN\n unionedgeom := clippedgeom;\n ELSE\n unionedgeom := st_union(unionedgeom, clippedgeom);\n END IF;\n\n unionedgeom_area := st_area(unionedgeom);\n\n IF skipcovered AND prev_area = unionedgeom_area THEN\n scancounter := scancounter + 1;\n CONTINUE;\n END IF;\n\n prev_area := unionedgeom_area;\n\n RAISE NOTICE '% % % %', unionedgeom_area/tilearea, counter, scancounter, ftime();\n END IF;\n RAISE NOTICE '% %', iter_record, content_hydrate(iter_record, fields);\n INSERT INTO pgstac_results (content) VALUES (content_hydrate(iter_record, fields));\n\n IF counter >= _limit\n OR scancounter > _scanlimit\n OR ftime() > _timelimit\n OR (exitwhenfull AND unionedgeom_area >= tilearea)\n THEN\n exit_flag := TRUE;\n EXIT;\n END IF;\n counter := counter + 1;\n scancounter := scancounter + 1;\n\n END LOOP;\n CLOSE curs;\n EXIT WHEN exit_flag;\n remaining_limit := _scanlimit - scancounter;\n END LOOP;\n\n SELECT jsonb_agg(content) INTO out_records FROM pgstac_results WHERE content IS NOT NULL;\n\n RETURN jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb)\n );\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.search(_search jsonb DEFAULT '{}'::jsonb)\n RETURNS jsonb\n LANGUAGE plpgsql\nAS $function$\nDECLARE\n searches searches%ROWTYPE;\n _where text;\n orderby text;\n search_where search_wheres%ROWTYPE;\n total_count bigint;\n token record;\n token_prev boolean;\n token_item items%ROWTYPE;\n token_where text;\n full_where text;\n init_ts timestamptz := clock_timestamp();\n timer timestamptz := clock_timestamp();\n hydrate bool := NOT (_search->'conf'->>'nohydrate' IS NOT NULL AND (_search->'conf'->>'nohydrate')::boolean = true);\n prev text;\n next text;\n context jsonb;\n collection jsonb;\n out_records jsonb;\n out_len int;\n _limit int := coalesce((_search->>'limit')::int, 10);\n _querylimit int;\n _fields jsonb := coalesce(_search->'fields', '{}'::jsonb);\n has_prev boolean := FALSE;\n has_next boolean := FALSE;\n links jsonb := '[]'::jsonb;\n base_url text:= concat(rtrim(base_url(_search->'conf'),'/'));\nBEGIN\n searches := search_query(_search);\n _where := searches._where;\n orderby := searches.orderby;\n search_where := where_stats(_where);\n total_count := coalesce(search_where.total_count, search_where.estimated_count);\n RAISE NOTICE 'SEARCH:TOKEN: %', _search->>'token';\n token := get_token_record(_search->>'token');\n RAISE NOTICE '***TOKEN: %', token;\n _querylimit := _limit + 1;\n IF token IS NOT NULL THEN\n token_prev := token.prev;\n token_item := token.item;\n token_where := get_token_filter(_search->'sortby', token_item, token_prev, FALSE);\n RAISE DEBUG 'TOKEN_WHERE: % (%ms from search start)', token_where, age_ms(timer);\n IF token_prev THEN -- if we are using a prev token, we know has_next is true\n RAISE DEBUG 'There is a previous token, so automatically setting has_next to true';\n has_next := TRUE;\n orderby := sort_sqlorderby(_search, TRUE);\n ELSE\n RAISE DEBUG 'There is a next token, so automatically setting has_prev to true';\n has_prev := TRUE;\n\n END IF;\n ELSE -- if there was no token, we know there is no prev\n RAISE DEBUG 'There is no token, so we know there is no prev. setting has_prev to false';\n has_prev := FALSE;\n END IF;\n\n full_where := concat_ws(' AND ', _where, token_where);\n RAISE NOTICE 'FULL WHERE CLAUSE: %', full_where;\n RAISE NOTICE 'Time to get counts and build query %', age_ms(timer);\n timer := clock_timestamp();\n\n IF hydrate THEN\n RAISE NOTICE 'Getting hydrated data.';\n ELSE\n RAISE NOTICE 'Getting non-hydrated data.';\n END IF;\n RAISE NOTICE 'CACHE SET TO %', get_setting_bool('format_cache');\n RAISE NOTICE 'Time to set hydration/formatting %', age_ms(timer);\n timer := clock_timestamp();\n SELECT jsonb_agg(format_item(i, _fields, hydrate)) INTO out_records\n FROM search_rows(\n full_where,\n orderby,\n search_where.partitions,\n _querylimit\n ) as i;\n\n RAISE NOTICE 'Time to fetch rows %', age_ms(timer);\n timer := clock_timestamp();\n\n\n IF token_prev THEN\n out_records := flip_jsonb_array(out_records);\n END IF;\n\n RAISE NOTICE 'Query returned % records.', jsonb_array_length(out_records);\n RAISE DEBUG 'TOKEN: % %', token_item.id, token_item.collection;\n RAISE DEBUG 'RECORD_1: % %', out_records->0->>'id', out_records->0->>'collection';\n RAISE DEBUG 'RECORD-1: % %', out_records->-1->>'id', out_records->-1->>'collection';\n\n -- REMOVE records that were from our token\n IF out_records->0->>'id' = token_item.id AND out_records->0->>'collection' = token_item.collection THEN\n out_records := out_records - 0;\n ELSIF out_records->-1->>'id' = token_item.id AND out_records->-1->>'collection' = token_item.collection THEN\n out_records := out_records - -1;\n END IF;\n\n out_len := jsonb_array_length(out_records);\n\n IF out_len = _limit + 1 THEN\n IF token_prev THEN\n has_prev := TRUE;\n out_records := out_records - 0;\n ELSE\n has_next := TRUE;\n out_records := out_records - -1;\n END IF;\n END IF;\n\n\n links := links || jsonb_build_object(\n 'rel', 'root',\n 'type', 'application/json',\n 'href', base_url\n ) || jsonb_build_object(\n 'rel', 'self',\n 'type', 'application/json',\n 'href', concat(base_url, '/search')\n );\n\n IF has_next THEN\n next := concat(out_records->-1->>'collection', ':', out_records->-1->>'id');\n RAISE NOTICE 'HAS NEXT | %', next;\n links := links || jsonb_build_object(\n 'rel', 'next',\n 'type', 'application/geo+json',\n 'method', 'GET',\n 'href', concat(base_url, '/search?token=next:', next)\n );\n END IF;\n\n IF has_prev THEN\n prev := concat(out_records->0->>'collection', ':', out_records->0->>'id');\n RAISE NOTICE 'HAS PREV | %', prev;\n links := links || jsonb_build_object(\n 'rel', 'prev',\n 'type', 'application/geo+json',\n 'method', 'GET',\n 'href', concat(base_url, '/search?token=prev:', prev)\n );\n END IF;\n\n RAISE NOTICE 'Time to get prev/next %', age_ms(timer);\n timer := clock_timestamp();\n\n\n collection := jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb),\n 'links', links\n );\n\n\n\n IF context(_search->'conf') != 'off' THEN\n collection := collection || jsonb_strip_nulls(jsonb_build_object(\n 'numberMatched', total_count,\n 'numberReturned', coalesce(jsonb_array_length(out_records), 0)\n ));\n ELSE\n collection := collection || jsonb_strip_nulls(jsonb_build_object(\n 'numberReturned', coalesce(jsonb_array_length(out_records), 0)\n ));\n END IF;\n\n IF get_setting_bool('timing', _search->'conf') THEN\n collection = collection || jsonb_build_object('timing', age_ms(init_ts));\n END IF;\n\n RAISE NOTICE 'Time to build final json %', age_ms(timer);\n timer := clock_timestamp();\n\n RAISE NOTICE 'Total Time: %', age_ms(current_timestamp);\n RAISE NOTICE 'RETURNING % records. NEXT: %. PREV: %', collection->>'numberReturned', collection->>'next', collection->>'prev';\n RETURN collection;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.search_query(_search jsonb DEFAULT '{}'::jsonb, updatestats boolean DEFAULT false, _metadata jsonb DEFAULT '{}'::jsonb)\n RETURNS searches\n LANGUAGE plpgsql\n SECURITY DEFINER\nAS $function$\nDECLARE\n search searches%ROWTYPE;\n cached_search searches%ROWTYPE;\n pexplain jsonb;\n t timestamptz;\n i interval;\n doupdate boolean := FALSE;\n insertfound boolean := FALSE;\n ro boolean := pgstac.readonly();\n found_search text;\nBEGIN\n RAISE NOTICE 'SEARCH: %', _search;\n -- Calculate hash, where clause, and order by statement\n search.search := _search;\n search.metadata := _metadata;\n search.hash := search_hash(_search, _metadata);\n search._where := stac_search_to_where(_search);\n search.orderby := sort_sqlorderby(_search);\n search.lastused := now();\n search.usecount := 1;\n\n -- If we are in read only mode, directly return search\n IF ro THEN\n RETURN search;\n END IF;\n\n RAISE NOTICE 'Updating Statistics for search: %s', search;\n -- Update statistics for times used and and when last used\n -- If the entry is locked, rather than waiting, skip updating the stats\n INSERT INTO searches (search, lastused, usecount, metadata)\n VALUES (search.search, now(), 1, search.metadata)\n ON CONFLICT DO NOTHING\n RETURNING * INTO cached_search\n ;\n\n IF NOT FOUND OR cached_search IS NULL THEN\n UPDATE searches SET\n lastused = now(),\n usecount = searches.usecount + 1\n WHERE hash = (\n SELECT hash FROM searches WHERE hash=search.hash FOR UPDATE SKIP LOCKED\n )\n RETURNING * INTO cached_search\n ;\n END IF;\n\n IF cached_search IS NOT NULL THEN\n cached_search._where = search._where;\n cached_search.orderby = search.orderby;\n RETURN cached_search;\n END IF;\n RETURN search;\n\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.search_rows(_where text DEFAULT 'TRUE'::text, _orderby text DEFAULT 'datetime DESC, id DESC'::text, partitions text[] DEFAULT NULL::text[], _limit integer DEFAULT 10)\n RETURNS SETOF items\n LANGUAGE plpgsql\n SET search_path TO 'pgstac', 'public'\nAS $function$\nDECLARE\n base_query text;\n query text;\n sdate timestamptz;\n edate timestamptz;\n n int;\n records_left int := _limit;\n timer timestamptz := clock_timestamp();\n full_timer timestamptz := clock_timestamp();\nBEGIN\nIF _where IS NULL OR trim(_where) = '' THEN\n _where = ' TRUE ';\nEND IF;\nRAISE NOTICE 'Getting chunks for % %', _where, _orderby;\n\nbase_query := $q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n LIMIT %L\n$q$;\n\nIF _orderby ILIKE 'datetime d%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 DESC LOOP\n RAISE NOTICE 'Running Query for % to %. %', sdate, edate, age_ms(full_timer);\n query := format(\n base_query,\n sdate,\n edate,\n _where,\n _orderby,\n records_left\n );\n RAISE DEBUG 'QUERY: %', query;\n timer := clock_timestamp();\n RETURN QUERY EXECUTE query;\n\n GET DIAGNOSTICS n = ROW_COUNT;\n records_left := records_left - n;\n RAISE NOTICE 'Returned %/% Rows From % to %. % to go. Time: %ms', n, _limit, sdate, edate, records_left, age_ms(timer);\n timer := clock_timestamp();\n IF records_left <= 0 THEN\n RAISE NOTICE 'SEARCH_ROWS TOOK %ms', age_ms(full_timer);\n RETURN;\n END IF;\n END LOOP;\nELSIF _orderby ILIKE 'datetime a%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 ASC LOOP\n RAISE NOTICE 'Running Query for % to %. %', sdate, edate, age_ms(full_timer);\n query := format(\n base_query,\n sdate,\n edate,\n _where,\n _orderby,\n records_left\n );\n RAISE DEBUG 'QUERY: %', query;\n timer := clock_timestamp();\n RETURN QUERY EXECUTE query;\n\n GET DIAGNOSTICS n = ROW_COUNT;\n records_left := records_left - n;\n RAISE NOTICE 'Returned %/% Rows From % to %. % to go. Time: %ms', n, _limit, sdate, edate, records_left, age_ms(timer);\n timer := clock_timestamp();\n IF records_left <= 0 THEN\n RAISE NOTICE 'SEARCH_ROWS TOOK %ms', age_ms(full_timer);\n RETURN;\n END IF;\n END LOOP;\nELSE\n query := format($q$\n SELECT * FROM items\n WHERE %s\n ORDER BY %s\n LIMIT %L\n $q$, _where, _orderby, _limit\n );\n RAISE DEBUG 'QUERY: %', query;\n timer := clock_timestamp();\n RETURN QUERY EXECUTE query;\n RAISE NOTICE 'FULL QUERY TOOK %ms', age_ms(timer);\nEND IF;\nRAISE NOTICE 'SEARCH_ROWS TOOK %ms', age_ms(full_timer);\nRETURN;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.where_stats(inwhere text, updatestats boolean DEFAULT false, conf jsonb DEFAULT NULL::jsonb)\n RETURNS search_wheres\n LANGUAGE plpgsql\n SECURITY DEFINER\nAS $function$\nDECLARE\n t timestamptz;\n i interval;\n explain_json jsonb;\n partitions text[];\n sw search_wheres%ROWTYPE;\n inwhere_hash text := md5(inwhere);\n _context text := lower(context(conf));\n _stats_ttl interval := context_stats_ttl(conf);\n _estimated_cost_threshold float := context_estimated_cost(conf);\n _estimated_count_threshold int := context_estimated_count(conf);\n ro bool := pgstac.readonly(conf);\nBEGIN\n -- If updatestats is true then set ttl to 0\n IF updatestats THEN\n RAISE DEBUG 'Updatestats set to TRUE, setting TTL to 0';\n _stats_ttl := '0'::interval;\n END IF;\n\n -- If we don't need to calculate context, just return\n IF _context = 'off' THEN\n sw._where = inwhere;\n RETURN sw;\n END IF;\n\n -- Get any stats that we have. If there is a lock where another process is\n -- updating the stats, wait so that we don't end up calculating a bunch of times.\n SELECT * INTO sw FROM search_wheres WHERE md5(_where)=inwhere_hash FOR UPDATE;\n\n -- If there is a cached row, figure out if we need to update\n IF\n sw IS NOT NULL\n AND sw.statslastupdated IS NOT NULL\n AND sw.total_count IS NOT NULL\n AND now() - sw.statslastupdated <= _stats_ttl\n THEN\n -- we have a cached row with data that is within our ttl\n RAISE DEBUG 'Stats present in table and lastupdated within ttl: %', sw;\n IF NOT ro THEN\n RAISE DEBUG 'Updating search_wheres only bumping lastused and usecount';\n UPDATE search_wheres SET\n lastused = now(),\n usecount = search_wheres.usecount + 1\n WHERE md5(_where) = inwhere_hash\n RETURNING * INTO sw;\n END IF;\n RAISE DEBUG 'Returning cached counts. %', sw;\n RETURN sw;\n END IF;\n\n -- Calculate estimated cost and rows\n -- Use explain to get estimated count/cost\n IF sw.estimated_count IS NULL OR sw.estimated_cost IS NULL THEN\n RAISE DEBUG 'Calculating estimated stats';\n t := clock_timestamp();\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s', inwhere)\n INTO explain_json;\n RAISE DEBUG 'Time for just the explain: %', clock_timestamp() - t;\n i := clock_timestamp() - t;\n\n sw.estimated_count := explain_json->0->'Plan'->'Plan Rows';\n sw.estimated_cost := explain_json->0->'Plan'->'Total Cost';\n sw.time_to_estimate := extract(epoch from i);\n END IF;\n\n RAISE DEBUG 'ESTIMATED_COUNT: %, THRESHOLD %', sw.estimated_count, _estimated_count_threshold;\n RAISE DEBUG 'ESTIMATED_COST: %, THRESHOLD %', sw.estimated_cost, _estimated_cost_threshold;\n\n -- If context is set to auto and the costs are within the threshold return the estimated costs\n IF\n _context = 'auto'\n AND sw.estimated_count >= _estimated_count_threshold\n AND sw.estimated_cost >= _estimated_cost_threshold\n THEN\n IF NOT ro THEN\n INSERT INTO search_wheres (\n _where,\n lastused,\n usecount,\n statslastupdated,\n estimated_count,\n estimated_cost,\n time_to_estimate,\n total_count,\n time_to_count\n ) VALUES (\n inwhere,\n now(),\n 1,\n now(),\n sw.estimated_count,\n sw.estimated_cost,\n sw.time_to_estimate,\n null,\n null\n ) ON CONFLICT ((md5(_where)))\n DO UPDATE SET\n lastused = EXCLUDED.lastused,\n usecount = search_wheres.usecount + 1,\n statslastupdated = EXCLUDED.statslastupdated,\n estimated_count = EXCLUDED.estimated_count,\n estimated_cost = EXCLUDED.estimated_cost,\n time_to_estimate = EXCLUDED.time_to_estimate,\n total_count = EXCLUDED.total_count,\n time_to_count = EXCLUDED.time_to_count\n RETURNING * INTO sw;\n END IF;\n RAISE DEBUG 'Estimates are within thresholds, returning estimates. %', sw;\n RETURN sw;\n END IF;\n\n -- Calculate Actual Count\n t := clock_timestamp();\n RAISE NOTICE 'Calculating actual count...';\n EXECUTE format(\n 'SELECT count(*) FROM items WHERE %s',\n inwhere\n ) INTO sw.total_count;\n i := clock_timestamp() - t;\n RAISE NOTICE 'Actual Count: % -- %', sw.total_count, i;\n sw.time_to_count := extract(epoch FROM i);\n\n IF NOT ro THEN\n INSERT INTO search_wheres (\n _where,\n lastused,\n usecount,\n statslastupdated,\n estimated_count,\n estimated_cost,\n time_to_estimate,\n total_count,\n time_to_count\n ) VALUES (\n inwhere,\n now(),\n 1,\n now(),\n sw.estimated_count,\n sw.estimated_cost,\n sw.time_to_estimate,\n sw.total_count,\n sw.time_to_count\n ) ON CONFLICT ((md5(_where)))\n DO UPDATE SET\n lastused = EXCLUDED.lastused,\n usecount = search_wheres.usecount + 1,\n statslastupdated = EXCLUDED.statslastupdated,\n estimated_count = EXCLUDED.estimated_count,\n estimated_cost = EXCLUDED.estimated_cost,\n time_to_estimate = EXCLUDED.time_to_estimate,\n total_count = EXCLUDED.total_count,\n time_to_count = EXCLUDED.time_to_count\n RETURNING * INTO sw;\n END IF;\n RAISE DEBUG 'Returning with actual count. %', sw;\n RETURN sw;\nEND;\n$function$\n;\n\n\n-- END migra calculated SQL\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('id', '{\"title\": \"Item ID\",\"description\": \"Item identifier\",\"$ref\": \"https://schemas.stacspec.org/v1.0.0/item-spec/json-schema/item.json#/definitions/core/allOf/2/properties/id\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('geometry', '{\"title\": \"Item Geometry\",\"description\": \"Item Geometry\",\"$ref\": \"https://geojson.org/schema/Feature.json\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('datetime','{\"description\": \"Datetime\",\"type\": \"string\",\"title\": \"Acquired\",\"format\": \"date-time\",\"pattern\": \"(\\\\+00:00|Z)$\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDELETE FROM queryables a USING queryables b\n WHERE a.name = b.name AND a.collection_ids IS NOT DISTINCT FROM b.collection_ids AND a.id > b.id;\n\n\nINSERT INTO pgstac_settings (name, value) VALUES\n ('context', 'off'),\n ('context_estimated_count', '100000'),\n ('context_estimated_cost', '100000'),\n ('context_stats_ttl', '1 day'),\n ('default_filter_lang', 'cql2-json'),\n ('additional_properties', 'true'),\n ('use_queue', 'false'),\n ('queue_timeout', '10 minutes'),\n ('update_collection_extent', 'false'),\n ('format_cache', 'false'),\n ('readonly', 'false')\nON CONFLICT DO NOTHING\n;\n\n\nINSERT INTO cql2_ops (op, template, types) VALUES\n ('eq', '%s = %s', NULL),\n ('neq', '%s != %s', NULL),\n ('ne', '%s != %s', NULL),\n ('!=', '%s != %s', NULL),\n ('<>', '%s != %s', NULL),\n ('lt', '%s < %s', NULL),\n ('lte', '%s <= %s', NULL),\n ('gt', '%s > %s', NULL),\n ('gte', '%s >= %s', NULL),\n ('le', '%s <= %s', NULL),\n ('ge', '%s >= %s', NULL),\n ('=', '%s = %s', NULL),\n ('<', '%s < %s', NULL),\n ('<=', '%s <= %s', NULL),\n ('>', '%s > %s', NULL),\n ('>=', '%s >= %s', NULL),\n ('like', '%s LIKE %s', NULL),\n ('ilike', '%s ILIKE %s', NULL),\n ('+', '%s + %s', NULL),\n ('-', '%s - %s', NULL),\n ('*', '%s * %s', NULL),\n ('/', '%s / %s', NULL),\n ('not', 'NOT (%s)', NULL),\n ('between', '%s BETWEEN %s AND %s', NULL),\n ('isnull', '%s IS NULL', NULL),\n ('upper', 'upper(%s)', NULL),\n ('lower', 'lower(%s)', NULL),\n ('casei', 'upper(%s)', NULL),\n ('accenti', 'unaccent(%s)', NULL)\nON CONFLICT (op) DO UPDATE\n SET\n template = EXCLUDED.template\n;\n\n\nALTER FUNCTION to_text COST 5000;\nALTER FUNCTION to_float COST 5000;\nALTER FUNCTION to_int COST 5000;\nALTER FUNCTION to_tstz COST 5000;\nALTER FUNCTION to_text_array COST 5000;\n\nALTER FUNCTION update_partition_stats SECURITY DEFINER;\nALTER FUNCTION partition_after_triggerfunc SECURITY DEFINER;\nALTER FUNCTION drop_table_constraints SECURITY DEFINER;\nALTER FUNCTION create_table_constraints SECURITY DEFINER;\nALTER FUNCTION check_partition SECURITY DEFINER;\nALTER FUNCTION repartition SECURITY DEFINER;\nALTER FUNCTION where_stats SECURITY DEFINER;\nALTER FUNCTION search_query SECURITY DEFINER;\nALTER FUNCTION format_item SECURITY DEFINER;\nALTER FUNCTION maintain_index SECURITY DEFINER;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nGRANT ALL ON SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac to pgstac_admin;\n\n-- pgstac_read role limited to using function apis\nGRANT EXECUTE ON FUNCTION search TO pgstac_read;\nGRANT EXECUTE ON FUNCTION search_query TO pgstac_read;\nGRANT EXECUTE ON FUNCTION item_by_id TO pgstac_read;\nGRANT EXECUTE ON FUNCTION get_item TO pgstac_read;\nGRANT SELECT ON ALL TABLES IN SCHEMA pgstac TO pgstac_read;\n\n\nGRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_ingest;\nGRANT USAGE ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_ingest;\n\nREVOKE ALL PRIVILEGES ON PROCEDURE run_queued_queries FROM public;\nGRANT ALL ON PROCEDURE run_queued_queries TO pgstac_admin;\n\nREVOKE ALL PRIVILEGES ON FUNCTION run_queued_queries_intransaction FROM public;\nGRANT ALL ON FUNCTION run_queued_queries_intransaction TO pgstac_admin;\n\nRESET ROLE;\n\nSET ROLE pgstac_ingest;\nSELECT update_partition_stats_q(partition) FROM partitions_view;\nSELECT set_version('0.9.0');\n" }, { "path": "src/pgstac/migrations/pgstac.0.8.6-0.9.10.sql", "content": "SET client_min_messages TO WARNING;\nSET SEARCH_PATH to pgstac, public;\nRESET ROLE;\nDO $$\nDECLARE\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='postgis') THEN\n CREATE EXTENSION IF NOT EXISTS postgis;\n END IF;\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='btree_gist') THEN\n CREATE EXTENSION IF NOT EXISTS btree_gist;\n END IF;\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='unaccent') THEN\n CREATE EXTENSION IF NOT EXISTS unaccent;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_admin;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_read;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_ingest;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n\nGRANT pgstac_admin TO current_user;\n\n-- Function to make sure pgstac_admin is the owner of items\nCREATE OR REPLACE FUNCTION pgstac_admin_owns() RETURNS VOID AS $$\nDECLARE\n f RECORD;\nBEGIN\n FOR f IN (\n SELECT\n concat(\n oid::regproc::text,\n '(',\n coalesce(pg_get_function_identity_arguments(oid),''),\n ')'\n ) AS name,\n CASE prokind WHEN 'f' THEN 'FUNCTION' WHEN 'p' THEN 'PROCEDURE' WHEN 'a' THEN 'AGGREGATE' END as typ\n FROM pg_proc\n WHERE\n pronamespace=to_regnamespace('pgstac')\n AND proowner != to_regrole('pgstac_admin')\n AND proname NOT LIKE 'pg_stat%'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n FOR f IN (\n SELECT\n oid::regclass::text as name,\n CASE relkind\n WHEN 'i' THEN 'INDEX'\n WHEN 'I' THEN 'INDEX'\n WHEN 'p' THEN 'TABLE'\n WHEN 'r' THEN 'TABLE'\n WHEN 'v' THEN 'VIEW'\n WHEN 'S' THEN 'SEQUENCE'\n ELSE NULL\n END as typ\n FROM pg_class\n WHERE relnamespace=to_regnamespace('pgstac') and relowner != to_regrole('pgstac_admin') AND relkind IN ('r','p','v','S') AND relname NOT LIKE 'pg_stat'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\nSELECT pgstac_admin_owns();\n\nCREATE SCHEMA IF NOT EXISTS pgstac AUTHORIZATION pgstac_admin;\n\nGRANT ALL ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_admin;\n\nALTER ROLE pgstac_admin SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_read SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_ingest SET SEARCH_PATH TO pgstac, public;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\n\nGRANT pgstac_read TO pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\n\nSET ROLE pgstac_admin;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\nRESET ROLE;\n\nSET ROLE pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\nRESET ROLE;\n\nSET SEARCH_PATH TO pgstac, public;\nSET ROLE pgstac_admin;\n\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS analyze_items;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS validate_constraints;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n-- Install these idempotently as migrations do not put them before trying to modify the collections table\n\n\nCREATE OR REPLACE FUNCTION collection_geom(content jsonb)\nRETURNS geometry AS $$\n WITH box AS (SELECT content->'extent'->'spatial'->'bbox'->0 as box)\n SELECT\n st_makeenvelope(\n (box->>0)::float,\n (box->>1)::float,\n (box->>2)::float,\n (box->>3)::float,\n 4326\n )\n FROM box;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION collection_datetime(content jsonb)\nRETURNS timestamptz AS $$\n SELECT\n CASE\n WHEN\n (content->'extent'->'temporal'->'interval'->0->>0) IS NULL\n THEN '-infinity'::timestamptz\n ELSE\n (content->'extent'->'temporal'->'interval'->0->>0)::timestamptz\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION collection_enddatetime(content jsonb)\nRETURNS timestamptz AS $$\n SELECT\n CASE\n WHEN\n (content->'extent'->'temporal'->'interval'->0->>1) IS NULL\n THEN 'infinity'::timestamptz\n ELSE\n (content->'extent'->'temporal'->'interval'->0->>1)::timestamptz\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n-- BEGIN migra calculated SQL\ndrop materialized view if exists \"pgstac\".\"partition_steps\";\n\ndrop view if exists \"pgstac\".\"partition_sys_meta\";\n\ndrop materialized view if exists \"pgstac\".\"partitions\";\n\ndrop view if exists \"pgstac\".\"partitions_view\";\n\ndrop function if exists \"pgstac\".\"dt_constraint\"(coid oid, OUT dt tstzrange, OUT edt tstzrange);\n\nset check_function_bodies = off;\n\nCREATE OR REPLACE FUNCTION pgstac.get_partition_name(relid regclass)\n RETURNS text\n LANGUAGE sql\n STABLE STRICT\nAS $function$\n SELECT (parse_ident(relid::text))[cardinality(parse_ident(relid::text))];\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.get_tstz_constraint(reloid oid, colname text)\n RETURNS tstzrange\n LANGUAGE plpgsql\n STABLE STRICT\nAS $function$\nDECLARE\n expr text := NULL;\n m text[];\n ts_lower timestamptz := NULL;\n ts_upper timestamptz := NULL;\n lower_inclusive text := '[';\n upper_inclusive text := ']';\n ts timestamptz;\nBEGIN\n SELECT INTO expr\n string_agg(def, ' AND ')\n FROM pg_constraint JOIN LATERAL pg_get_constraintdef(oid) AS def ON TRUE\n WHERE\n conrelid = reloid\n AND contype = 'c'\n AND def LIKE '%' || colname || '%'\n ;\n\n IF expr IS NULL THEN\n RETURN NULL;\n END IF;\n\n RAISE DEBUG 'Constraint expression for % on %: %', colname, reloid::regclass, expr;\n -- collect all constraints for the specified column\n FOR m IN SELECT regexp_matches(expr, '[ (]' || colname || $expr$\\s*([<>=]{1,2})\\s*'([0-9 :+\\-]+)'$expr$, 'g') LOOP\n ts := m[2]::timestamptz;\n IF m[1] IN ('>', '>=')\n THEN\n IF ts_lower IS NULL OR ts > ts_lower OR (ts = ts_lower AND m[1] = '>') THEN\n ts_lower := ts;\n lower_inclusive := CASE WHEN m[1] = '>' THEN '(' ELSE '[' END;\n END IF;\n ELSIF m[1] IN ('<', '<=')\n THEN\n IF ts_upper IS NULL OR ts < ts_upper OR (ts = ts_upper AND m[1] = '<') THEN\n ts_upper := ts;\n upper_inclusive := CASE WHEN m[1] = '<' THEN ')' ELSE ']' END;\n END IF;\n END IF;\n END LOOP;\n RAISE DEBUG 'Constraint % for %: % %', colname, reloid::regclass, ts_lower, ts_upper;\n RETURN tstzrange(ts_lower, ts_upper, lower_inclusive || upper_inclusive);\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.q_to_tsquery(jinput jsonb)\n RETURNS tsquery\n LANGUAGE plpgsql\nAS $function$\nDECLARE\n input text;\n processed_text text;\n temp_text text;\n quote_array text[];\n placeholder text := '@QUOTE@';\nBEGIN\n IF jsonb_typeof(jinput) = 'string' THEN\n input := jinput->>0;\n ELSIF jsonb_typeof(jinput) = 'array' THEN\n input := array_to_string(\n array(select jsonb_array_elements_text(jinput)),\n ' OR '\n );\n ELSE\n RAISE EXCEPTION 'Input must be a string or an array of strings.';\n END IF;\n -- Extract all quoted phrases and store in array\n quote_array := regexp_matches(input, '\"[^\"]*\"', 'g');\n\n -- Replace each quoted part with a unique placeholder if there are any quoted phrases\n IF array_length(quote_array, 1) IS NOT NULL THEN\n processed_text := input;\n FOR i IN array_lower(quote_array, 1) .. array_upper(quote_array, 1) LOOP\n processed_text := replace(processed_text, quote_array[i], placeholder || i || placeholder);\n END LOOP;\n ELSE\n processed_text := input;\n END IF;\n\n -- Replace non-quoted text using regular expressions\n\n -- , -> |\n processed_text := regexp_replace(processed_text, ',(?=(?:[^\"]*\"[^\"]*\")*[^\"]*$)', ' | ', 'g');\n\n -- and -> &\n processed_text := regexp_replace(processed_text, '\\s+AND\\s+', ' & ', 'gi');\n\n -- or -> |\n processed_text := regexp_replace(processed_text, '\\s+OR\\s+', ' | ', 'gi');\n\n -- + ->\n processed_text := regexp_replace(processed_text, '^\\s*\\+([a-zA-Z0-9_]+)', '\\1', 'g'); -- +term at start\n processed_text := regexp_replace(processed_text, '\\s*\\+([a-zA-Z0-9_]+)', ' & \\1', 'g'); -- +term elsewhere\n\n -- - -> !\n processed_text := regexp_replace(processed_text, '^\\s*\\-([a-zA-Z0-9_]+)', '! \\1', 'g'); -- -term at start\n processed_text := regexp_replace(processed_text, '\\s*\\-([a-zA-Z0-9_]+)', ' & ! \\1', 'g'); -- -term elsewhere\n\n -- terms separated with spaces are assumed to represent adjacent terms. loop through these\n -- occurrences and replace them with the adjacency operator (<->)\n LOOP\n temp_text := regexp_replace(processed_text, '([a-zA-Z0-9_]+)\\s+([a-zA-Z0-9_]+)(?!\\s*[&|<>])', '\\1 <-> \\2', 'g');\n IF temp_text = processed_text THEN\n EXIT; -- No more replacements were made\n END IF;\n processed_text := temp_text;\n END LOOP;\n\n\n -- Replace placeholders back with quoted phrases if there were any\n IF array_length(quote_array, 1) IS NOT NULL THEN\n FOR i IN array_lower(quote_array, 1) .. array_upper(quote_array, 1) LOOP\n processed_text := replace(processed_text, placeholder || i || placeholder, '''' || substring(quote_array[i] from 2 for length(quote_array[i]) - 2) || '''');\n END LOOP;\n END IF;\n\n -- Print processed_text to the console for debugging purposes\n RAISE NOTICE 'processed_text: %', processed_text;\n\n RETURN to_tsquery('english', processed_text);\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.search_fromhash(_hash text)\n RETURNS searches\n LANGUAGE sql\n STRICT\nAS $function$\n SELECT * FROM search_query((SELECT search FROM searches WHERE hash=_hash LIMIT 1));\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.chunker(_where text, OUT s timestamp with time zone, OUT e timestamp with time zone)\n RETURNS SETOF record\n LANGUAGE plpgsql\nAS $function$\nDECLARE\n explain jsonb;\nBEGIN\n IF _where IS NULL THEN\n _where := ' TRUE ';\n END IF;\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s;', _where)\n INTO explain;\n RAISE DEBUG 'EXPLAIN: %', explain;\n\n RETURN QUERY\n WITH t AS (\n SELECT j->>0 as p FROM\n jsonb_path_query(\n explain,\n 'strict $.**.\"Relation Name\" ? (@ != null)'\n ) j\n ),\n parts AS (\n SELECT sdate, edate FROM t JOIN partition_steps ON (t.p = name)\n ),\n times AS (\n SELECT sdate FROM parts\n UNION\n SELECT edate FROM parts\n ),\n uniq AS (\n SELECT DISTINCT sdate FROM times ORDER BY sdate\n ),\n last AS (\n SELECT sdate, lead(sdate, 1) over () as edate FROM uniq\n )\n SELECT sdate, edate FROM last WHERE edate IS NOT NULL;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.collection_extent(_collection text, runupdate boolean DEFAULT false)\n RETURNS jsonb\n LANGUAGE plpgsql\nAS $function$\nDECLARE\n geom_extent geometry;\n mind timestamptz;\n maxd timestamptz;\n extent jsonb;\nBEGIN\n IF runupdate THEN\n PERFORM update_partition_stats_q(partition)\n FROM partitions_view WHERE collection=_collection;\n END IF;\n SELECT\n min(lower(dtrange)),\n max(upper(edtrange)),\n st_extent(spatial)\n INTO\n mind,\n maxd,\n geom_extent\n FROM partitions_view\n WHERE collection=_collection;\n\n IF geom_extent IS NOT NULL AND mind IS NOT NULL AND maxd IS NOT NULL THEN\n extent := jsonb_build_object(\n 'spatial', jsonb_build_object(\n 'bbox', to_jsonb(array[array[st_xmin(geom_extent), st_ymin(geom_extent), st_xmax(geom_extent), st_ymax(geom_extent)]])\n ),\n 'temporal', jsonb_build_object(\n 'interval', to_jsonb(array[array[mind, maxd]])\n )\n );\n RETURN extent;\n END IF;\n RETURN NULL;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.collection_search(_search jsonb DEFAULT '{}'::jsonb)\n RETURNS jsonb\n LANGUAGE plpgsql\n STABLE PARALLEL SAFE\nAS $function$\nDECLARE\n out_records jsonb;\n number_matched bigint := collection_search_matched(_search);\n number_returned bigint;\n _limit int := coalesce((_search->>'limit')::float::int, 10);\n _offset int := coalesce((_search->>'offset')::float::int, 0);\n links jsonb := '[]';\n ret jsonb;\n base_url text:= concat(rtrim(base_url(_search->'conf'),'/'), '/collections');\n prevoffset int;\n nextoffset int;\nBEGIN\n SELECT\n coalesce(jsonb_agg(c), '[]')\n INTO out_records\n FROM collection_search_rows(_search) c;\n\n number_returned := jsonb_array_length(out_records);\n RAISE DEBUG 'nm: %, nr: %, l:%, o:%', number_matched, number_returned, _limit, _offset;\n\n\n\n IF _limit <= number_matched AND number_matched > 0 THEN --need to have paging links\n nextoffset := least(_offset + _limit, number_matched - 1);\n prevoffset := greatest(_offset - _limit, 0);\n\n IF _offset > 0 THEN\n links := links || jsonb_build_object(\n 'rel', 'prev',\n 'type', 'application/json',\n 'method', 'GET' ,\n 'href', base_url,\n 'body', jsonb_build_object('offset', prevoffset),\n 'merge', TRUE\n );\n END IF;\n\n IF (_offset + _limit < number_matched) THEN\n links := links || jsonb_build_object(\n 'rel', 'next',\n 'type', 'application/json',\n 'method', 'GET' ,\n 'href', base_url,\n 'body', jsonb_build_object('offset', nextoffset),\n 'merge', TRUE\n );\n END IF;\n\n END IF;\n\n ret := jsonb_build_object(\n 'collections', out_records,\n 'numberMatched', number_matched,\n 'numberReturned', number_returned,\n 'links', links\n );\n RETURN ret;\n\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.collection_temporal_extent(id text)\n RETURNS jsonb\n LANGUAGE sql\n IMMUTABLE PARALLEL SAFE\n SET search_path TO 'pgstac', 'public'\nAS $function$\n SELECT to_jsonb(array[array[min(datetime), max(datetime)]])\n FROM items WHERE collection=$1;\n;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.cql2_query(j jsonb, wrapper text DEFAULT NULL::text)\n RETURNS text\n LANGUAGE plpgsql\n STABLE\nAS $function$\n#variable_conflict use_variable\nDECLARE\n args jsonb := j->'args';\n arg jsonb;\n op text := lower(j->>'op');\n cql2op RECORD;\n literal text;\n _wrapper text;\n leftarg text;\n rightarg text;\n prop text;\n extra_props bool := pgstac.additional_properties();\nBEGIN\n IF j IS NULL OR (op IS NOT NULL AND args IS NULL) THEN\n RETURN NULL;\n END IF;\n RAISE NOTICE 'CQL2_QUERY: %', j;\n\n -- check if all properties are represented in the queryables\n IF NOT extra_props THEN\n FOR prop IN\n SELECT DISTINCT p->>0\n FROM jsonb_path_query(j, 'strict $.**.property') p\n WHERE p->>0 NOT IN ('id', 'datetime', 'geometry', 'end_datetime', 'collection')\n LOOP\n IF (queryable(prop)).nulled_wrapper IS NULL THEN\n RAISE EXCEPTION 'Term % is not found in queryables.', prop;\n END IF;\n END LOOP;\n END IF;\n\n IF j ? 'filter' THEN\n RETURN cql2_query(j->'filter');\n END IF;\n\n IF j ? 'upper' THEN\n RETURN cql2_query(jsonb_build_object('op', 'upper', 'args', j->'upper'));\n END IF;\n\n IF j ? 'lower' THEN\n RETURN cql2_query(jsonb_build_object('op', 'lower', 'args', j->'lower'));\n END IF;\n\n -- Temporal Query\n IF op ilike 't_%' or op = 'anyinteracts' THEN\n RETURN temporal_op_query(op, args);\n END IF;\n\n -- If property is a timestamp convert it to text to use with\n -- general operators\n IF j ? 'timestamp' THEN\n RETURN format('%L::timestamptz', to_tstz(j->'timestamp'));\n END IF;\n IF j ? 'interval' THEN\n RAISE EXCEPTION 'Please use temporal operators when using intervals.';\n RETURN NONE;\n END IF;\n\n -- Spatial Query\n IF op ilike 's_%' or op = 'intersects' THEN\n RETURN spatial_op_query(op, args);\n END IF;\n\n IF op IN ('a_equals','a_contains','a_contained_by','a_overlaps') THEN\n IF args->0 ? 'property' THEN\n leftarg := format('to_text_array(%s)', (queryable(args->0->>'property')).path);\n END IF;\n IF args->1 ? 'property' THEN\n rightarg := format('to_text_array(%s)', (queryable(args->1->>'property')).path);\n END IF;\n RETURN FORMAT(\n '%s %s %s',\n COALESCE(leftarg, quote_literal(to_text_array(args->0))),\n CASE op\n WHEN 'a_equals' THEN '='\n WHEN 'a_contains' THEN '@>'\n WHEN 'a_contained_by' THEN '<@'\n WHEN 'a_overlaps' THEN '&&'\n END,\n COALESCE(rightarg, quote_literal(to_text_array(args->1)))\n );\n END IF;\n\n IF op = 'in' THEN\n RAISE NOTICE 'IN : % % %', args, jsonb_build_array(args->0), args->1;\n args := jsonb_build_array(args->0) || (args->1);\n RAISE NOTICE 'IN2 : %', args;\n END IF;\n\n\n\n IF op = 'between' THEN\n args = jsonb_build_array(\n args->0,\n args->1,\n args->2\n );\n END IF;\n\n -- Make sure that args is an array and run cql2_query on\n -- each element of the array\n RAISE NOTICE 'ARGS PRE: %', args;\n IF j ? 'args' THEN\n IF jsonb_typeof(args) != 'array' THEN\n args := jsonb_build_array(args);\n END IF;\n\n IF jsonb_path_exists(args, '$[*] ? (@.property == \"id\" || @.property == \"datetime\" || @.property == \"end_datetime\" || @.property == \"collection\")') THEN\n wrapper := NULL;\n ELSE\n -- if any of the arguments are a property, try to get the property_wrapper\n FOR arg IN SELECT jsonb_path_query(args, '$[*] ? (@.property != null)') LOOP\n RAISE NOTICE 'Arg: %', arg;\n wrapper := (queryable(arg->>'property')).nulled_wrapper;\n RAISE NOTICE 'Property: %, Wrapper: %', arg, wrapper;\n IF wrapper IS NOT NULL THEN\n EXIT;\n END IF;\n END LOOP;\n\n -- if the property was not in queryables, see if any args were numbers\n IF\n wrapper IS NULL\n AND jsonb_path_exists(args, '$[*] ? (@.type()==\"number\")')\n THEN\n wrapper := 'to_float';\n END IF;\n wrapper := coalesce(wrapper, 'to_text');\n END IF;\n\n SELECT jsonb_agg(cql2_query(a, wrapper))\n INTO args\n FROM jsonb_array_elements(args) a;\n END IF;\n RAISE NOTICE 'ARGS: %', args;\n\n IF op IN ('and', 'or') THEN\n RETURN\n format(\n '(%s)',\n array_to_string(to_text_array(args), format(' %s ', upper(op)))\n );\n END IF;\n\n IF op = 'in' THEN\n RAISE NOTICE 'IN -- % %', args->0, to_text(args->0);\n RETURN format(\n '%s IN (%s)',\n to_text(args->0),\n array_to_string((to_text_array(args))[2:], ',')\n );\n END IF;\n\n -- Look up template from cql2_ops\n IF j ? 'op' THEN\n SELECT * INTO cql2op FROM cql2_ops WHERE cql2_ops.op ilike op;\n IF FOUND THEN\n -- If specific index set in queryables for a property cast other arguments to that type\n\n RETURN format(\n cql2op.template,\n VARIADIC (to_text_array(args))\n );\n ELSE\n RAISE EXCEPTION 'Operator % Not Supported.', op;\n END IF;\n END IF;\n\n\n IF wrapper IS NOT NULL THEN\n RAISE NOTICE 'Wrapping % with %', j, wrapper;\n IF j ? 'property' THEN\n RETURN format('%I(%s)', wrapper, (queryable(j->>'property')).path);\n ELSE\n RETURN format('%I(%L)', wrapper, j);\n END IF;\n ELSIF j ? 'property' THEN\n RETURN quote_ident(j->>'property');\n END IF;\n\n RETURN quote_literal(to_text(j));\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.geometrysearch(geom geometry, queryhash text, fields jsonb DEFAULT NULL::jsonb, _scanlimit integer DEFAULT 10000, _limit integer DEFAULT 100, _timelimit interval DEFAULT '00:00:05'::interval, exitwhenfull boolean DEFAULT true, skipcovered boolean DEFAULT true)\n RETURNS jsonb\n LANGUAGE plpgsql\nAS $function$\nDECLARE\n search searches%ROWTYPE;\n curs refcursor;\n _where text;\n query text;\n iter_record items%ROWTYPE;\n out_records jsonb := '{}'::jsonb[];\n exit_flag boolean := FALSE;\n counter int := 1;\n scancounter int := 1;\n remaining_limit int := _scanlimit;\n tilearea float;\n unionedgeom geometry;\n clippedgeom geometry;\n unionedgeom_area float := 0;\n prev_area float := 0;\n excludes text[];\n includes text[];\n\nBEGIN\n DROP TABLE IF EXISTS pgstac_results;\n CREATE TEMP TABLE pgstac_results (content jsonb) ON COMMIT DROP;\n\n -- If the passed in geometry is not an area set exitwhenfull and skipcovered to false\n IF ST_GeometryType(geom) !~* 'polygon' THEN\n RAISE NOTICE 'GEOMETRY IS NOT AN AREA';\n skipcovered = FALSE;\n exitwhenfull = FALSE;\n END IF;\n\n -- If skipcovered is true then you will always want to exit when the passed in geometry is full\n IF skipcovered THEN\n exitwhenfull := TRUE;\n END IF;\n\n search := search_fromhash(queryhash);\n\n IF search IS NULL THEN\n RAISE EXCEPTION 'Search with Query Hash % Not Found', queryhash;\n END IF;\n\n tilearea := st_area(geom);\n _where := format('%s AND st_intersects(geometry, %L::geometry)', search._where, geom);\n\n\n FOR query IN SELECT * FROM partition_queries(_where, search.orderby) LOOP\n query := format('%s LIMIT %L', query, remaining_limit);\n RAISE NOTICE '%', query;\n OPEN curs FOR EXECUTE query;\n LOOP\n FETCH curs INTO iter_record;\n EXIT WHEN NOT FOUND;\n IF exitwhenfull OR skipcovered THEN -- If we are not using exitwhenfull or skipcovered, we do not need to do expensive geometry operations\n clippedgeom := st_intersection(geom, iter_record.geometry);\n\n IF unionedgeom IS NULL THEN\n unionedgeom := clippedgeom;\n ELSE\n unionedgeom := st_union(unionedgeom, clippedgeom);\n END IF;\n\n unionedgeom_area := st_area(unionedgeom);\n\n IF skipcovered AND prev_area = unionedgeom_area THEN\n scancounter := scancounter + 1;\n CONTINUE;\n END IF;\n\n prev_area := unionedgeom_area;\n\n RAISE NOTICE '% % % %', unionedgeom_area/tilearea, counter, scancounter, ftime();\n END IF;\n RAISE NOTICE '% %', iter_record, content_hydrate(iter_record, fields);\n INSERT INTO pgstac_results (content) VALUES (content_hydrate(iter_record, fields));\n\n IF counter >= _limit\n OR scancounter > _scanlimit\n OR ftime() > _timelimit\n OR (exitwhenfull AND unionedgeom_area >= tilearea)\n THEN\n exit_flag := TRUE;\n EXIT;\n END IF;\n counter := counter + 1;\n scancounter := scancounter + 1;\n\n END LOOP;\n CLOSE curs;\n EXIT WHEN exit_flag;\n remaining_limit := _scanlimit - scancounter;\n END LOOP;\n\n SELECT jsonb_agg(content) INTO out_records FROM pgstac_results WHERE content IS NOT NULL;\n\n RETURN jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb)\n );\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.jsonb_include(j jsonb, f jsonb)\n RETURNS jsonb\n LANGUAGE plpgsql\n IMMUTABLE\nAS $function$\nDECLARE\n includes jsonb := f-> 'include';\n outj jsonb := '{}'::jsonb;\n path text[];\nBEGIN\n IF\n includes IS NULL\n OR jsonb_array_length(includes) = 0\n THEN\n RETURN j;\n ELSE\n includes := includes || (\n CASE WHEN j ? 'collection' THEN\n '[\"id\",\"collection\"]'\n ELSE\n '[\"id\"]'\n END)::jsonb;\n FOR path IN SELECT explode_dotpaths(includes) LOOP\n outj := jsonb_set_nested(outj, path, j #> path);\n END LOOP;\n END IF;\n RETURN outj;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.maintain_partition_queries(part text DEFAULT 'items'::text, dropindexes boolean DEFAULT false, rebuildindexes boolean DEFAULT false, idxconcurrently boolean DEFAULT false)\n RETURNS SETOF text\n LANGUAGE plpgsql\nAS $function$\nDECLARE\n rec record;\n q text;\nBEGIN\n FOR rec IN (\n SELECT * FROM queryable_indexes(part,true)\n ) LOOP\n q := format(\n 'SELECT maintain_index(\n %L,%L,%L,%L,%L\n );',\n rec.indexname,\n rec.queryable_idx,\n dropindexes,\n rebuildindexes,\n idxconcurrently\n );\n RAISE NOTICE 'Q: %', q;\n RETURN NEXT q;\n END LOOP;\n RETURN;\nEND;\n$function$\n;\n\ncreate or replace view \"pgstac\".\"partition_sys_meta\" as SELECT partition.partition,\n replace(replace(\n CASE\n WHEN (pg_partition_tree.level = 1) THEN partition_expr.partition_expr\n ELSE parent_partition_expr.parent_partition_expr\n END, 'FOR VALUES IN ('''::text, ''::text), ''')'::text, ''::text) AS collection,\n pg_partition_tree.level,\n c.reltuples,\n c.relhastriggers,\n partition_dtrange.partition_dtrange,\n COALESCE(get_tstz_constraint(c.oid, 'datetime'::text), partition_dtrange.partition_dtrange, inf_range.inf_range) AS constraint_dtrange,\n COALESCE(get_tstz_constraint(c.oid, 'end_datetime'::text), inf_range.inf_range) AS constraint_edtrange\n FROM ((((((((((pg_partition_tree('items'::regclass) pg_partition_tree(relid, parentrelid, isleaf, level)\n JOIN pg_class c ON (((pg_partition_tree.relid)::oid = c.oid)))\n JOIN pg_class parent ON ((((pg_partition_tree.parentrelid)::oid = parent.oid) AND pg_partition_tree.isleaf)))\n LEFT JOIN pg_constraint edt ON (((edt.conrelid = c.oid) AND (edt.contype = 'c'::\"char\"))))\n JOIN LATERAL get_partition_name(pg_partition_tree.relid) partition(partition) ON (true))\n JOIN LATERAL pg_get_expr(c.relpartbound, c.oid) partition_expr(partition_expr) ON (true))\n JOIN LATERAL pg_get_expr(parent.relpartbound, parent.oid) parent_partition_expr(parent_partition_expr) ON (true))\n JOIN LATERAL tstzrange('-infinity'::timestamp with time zone, 'infinity'::timestamp with time zone, '[]'::text) inf_range(inf_range) ON (true))\n JOIN LATERAL COALESCE(constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), inf_range.inf_range) partition_dtrange(partition_dtrange) ON (true))\n JOIN LATERAL get_tstz_constraint(c.oid, 'datetime'::text) datetime_constraint(datetime_constraint) ON (true))\n JOIN LATERAL get_tstz_constraint(c.oid, 'end_datetime'::text) end_datetime_constraint(end_datetime_constraint) ON (true))\n WHERE pg_partition_tree.isleaf;\n\n\ncreate or replace view \"pgstac\".\"partitions_view\" as SELECT (parse_ident((pg_partition_tree.relid)::text))[cardinality(parse_ident((pg_partition_tree.relid)::text))] AS partition,\n replace(replace(\n CASE\n WHEN (pg_partition_tree.level = 1) THEN partition_expr.partition_expr\n ELSE parent_partition_expr.parent_partition_expr\n END, 'FOR VALUES IN ('''::text, ''::text), ''')'::text, ''::text) AS collection,\n pg_partition_tree.level,\n c.reltuples,\n c.relhastriggers,\n partition_dtrange.partition_dtrange,\n COALESCE(get_tstz_constraint(c.oid, 'datetime'::text), partition_dtrange.partition_dtrange, inf_range.inf_range) AS constraint_dtrange,\n COALESCE(get_tstz_constraint(c.oid, 'end_datetime'::text), inf_range.inf_range) AS constraint_edtrange,\n partition_stats.dtrange,\n partition_stats.edtrange,\n partition_stats.spatial,\n partition_stats.last_updated\n FROM (((((((((((pg_partition_tree('items'::regclass) pg_partition_tree(relid, parentrelid, isleaf, level)\n JOIN pg_class c ON (((pg_partition_tree.relid)::oid = c.oid)))\n JOIN pg_class parent ON ((((pg_partition_tree.parentrelid)::oid = parent.oid) AND pg_partition_tree.isleaf)))\n LEFT JOIN pg_constraint edt ON (((edt.conrelid = c.oid) AND (edt.contype = 'c'::\"char\"))))\n JOIN LATERAL get_partition_name(pg_partition_tree.relid) partition(partition) ON (true))\n JOIN LATERAL pg_get_expr(c.relpartbound, c.oid) partition_expr(partition_expr) ON (true))\n JOIN LATERAL pg_get_expr(parent.relpartbound, parent.oid) parent_partition_expr(parent_partition_expr) ON (true))\n JOIN LATERAL tstzrange('-infinity'::timestamp with time zone, 'infinity'::timestamp with time zone, '[]'::text) inf_range(inf_range) ON (true))\n JOIN LATERAL COALESCE(constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), inf_range.inf_range) partition_dtrange(partition_dtrange) ON (true))\n JOIN LATERAL get_tstz_constraint(c.oid, 'datetime'::text) datetime_constraint(datetime_constraint) ON (true))\n JOIN LATERAL get_tstz_constraint(c.oid, 'end_datetime'::text) end_datetime_constraint(end_datetime_constraint) ON (true))\n LEFT JOIN partition_stats USING (partition))\n WHERE pg_partition_tree.isleaf;\n\n\nCREATE OR REPLACE FUNCTION pgstac.queryable(dotpath text, OUT path text, OUT expression text, OUT wrapper text, OUT nulled_wrapper text)\n RETURNS record\n LANGUAGE plpgsql\n STABLE STRICT\nAS $function$\nDECLARE\n q RECORD;\n path_elements text[];\nBEGIN\n dotpath := replace(dotpath, 'properties.', '');\n IF dotpath = 'start_datetime' THEN\n dotpath := 'datetime';\n END IF;\n IF dotpath IN ('id', 'geometry', 'datetime', 'end_datetime', 'collection') THEN\n path := dotpath;\n expression := dotpath;\n wrapper := NULL;\n RETURN;\n END IF;\n\n SELECT * INTO q FROM queryables\n WHERE\n name=dotpath\n OR name = 'properties.' || dotpath\n OR name = replace(dotpath, 'properties.', '')\n ;\n IF q.property_wrapper IS NULL THEN\n IF q.definition->>'type' = 'number' THEN\n wrapper := 'to_float';\n nulled_wrapper := wrapper;\n ELSIF q.definition->>'format' = 'date-time' THEN\n wrapper := 'to_tstz';\n nulled_wrapper := wrapper;\n ELSE\n nulled_wrapper := NULL;\n wrapper := 'to_text';\n END IF;\n ELSE\n wrapper := q.property_wrapper;\n nulled_wrapper := wrapper;\n END IF;\n IF q.property_path IS NOT NULL THEN\n path := q.property_path;\n ELSE\n path_elements := string_to_array(dotpath, '.');\n IF path_elements[1] IN ('links', 'assets', 'stac_version', 'stac_extensions') THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSIF path_elements[1] = 'properties' THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSE\n path := format($F$content->'properties'->%s$F$, array_to_path(path_elements));\n END IF;\n END IF;\n expression := format('%I(%s)', wrapper, path);\n RETURN;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.search(_search jsonb DEFAULT '{}'::jsonb)\n RETURNS jsonb\n LANGUAGE plpgsql\nAS $function$\nDECLARE\n searches searches%ROWTYPE;\n _where text;\n orderby text;\n search_where search_wheres%ROWTYPE;\n total_count bigint;\n token record;\n token_prev boolean;\n token_item items%ROWTYPE;\n token_where text;\n full_where text;\n init_ts timestamptz := clock_timestamp();\n timer timestamptz := clock_timestamp();\n hydrate bool := NOT (_search->'conf'->>'nohydrate' IS NOT NULL AND (_search->'conf'->>'nohydrate')::boolean = true);\n prev text;\n next text;\n context jsonb;\n collection jsonb;\n out_records jsonb;\n out_len int;\n _limit int := coalesce((_search->>'limit')::int, 10);\n _querylimit int;\n _fields jsonb := coalesce(_search->'fields', '{}'::jsonb);\n has_prev boolean := FALSE;\n has_next boolean := FALSE;\n links jsonb := '[]'::jsonb;\n base_url text:= concat(rtrim(base_url(_search->'conf'),'/'));\nBEGIN\n searches := search_query(_search);\n _where := searches._where;\n orderby := searches.orderby;\n search_where := where_stats(_where);\n total_count := coalesce(search_where.total_count, search_where.estimated_count);\n RAISE NOTICE 'SEARCH:TOKEN: %', _search->>'token';\n token := get_token_record(_search->>'token');\n RAISE NOTICE '***TOKEN: %', token;\n _querylimit := _limit + 1;\n IF token IS NOT NULL THEN\n token_prev := token.prev;\n token_item := token.item;\n token_where := get_token_filter(_search->'sortby', token_item, token_prev, FALSE);\n RAISE DEBUG 'TOKEN_WHERE: % (%ms from search start)', token_where, age_ms(timer);\n IF token_prev THEN -- if we are using a prev token, we know has_next is true\n RAISE DEBUG 'There is a previous token, so automatically setting has_next to true';\n has_next := TRUE;\n orderby := sort_sqlorderby(_search, TRUE);\n ELSE\n RAISE DEBUG 'There is a next token, so automatically setting has_prev to true';\n has_prev := TRUE;\n\n END IF;\n ELSE -- if there was no token, we know there is no prev\n RAISE DEBUG 'There is no token, so we know there is no prev. setting has_prev to false';\n has_prev := FALSE;\n END IF;\n\n full_where := concat_ws(' AND ', _where, token_where);\n RAISE NOTICE 'FULL WHERE CLAUSE: %', full_where;\n RAISE NOTICE 'Time to get counts and build query %', age_ms(timer);\n timer := clock_timestamp();\n\n IF hydrate THEN\n RAISE NOTICE 'Getting hydrated data.';\n ELSE\n RAISE NOTICE 'Getting non-hydrated data.';\n END IF;\n RAISE NOTICE 'CACHE SET TO %', get_setting_bool('format_cache');\n RAISE NOTICE 'Time to set hydration/formatting %', age_ms(timer);\n timer := clock_timestamp();\n SELECT jsonb_agg(format_item(i, _fields, hydrate)) INTO out_records\n FROM search_rows(\n full_where,\n orderby,\n search_where.partitions,\n _querylimit\n ) as i;\n\n RAISE NOTICE 'Time to fetch rows %', age_ms(timer);\n timer := clock_timestamp();\n\n\n IF token_prev THEN\n out_records := flip_jsonb_array(out_records);\n END IF;\n\n RAISE NOTICE 'Query returned % records.', jsonb_array_length(out_records);\n RAISE DEBUG 'TOKEN: % %', token_item.id, token_item.collection;\n RAISE DEBUG 'RECORD_1: % %', out_records->0->>'id', out_records->0->>'collection';\n RAISE DEBUG 'RECORD-1: % %', out_records->-1->>'id', out_records->-1->>'collection';\n\n -- REMOVE records that were from our token\n IF out_records->0->>'id' = token_item.id AND out_records->0->>'collection' = token_item.collection THEN\n out_records := out_records - 0;\n ELSIF out_records->-1->>'id' = token_item.id AND out_records->-1->>'collection' = token_item.collection THEN\n out_records := out_records - -1;\n END IF;\n\n out_len := jsonb_array_length(out_records);\n\n IF out_len = _limit + 1 THEN\n IF token_prev THEN\n has_prev := TRUE;\n out_records := out_records - 0;\n ELSE\n has_next := TRUE;\n out_records := out_records - -1;\n END IF;\n END IF;\n\n\n links := links || jsonb_build_object(\n 'rel', 'root',\n 'type', 'application/json',\n 'href', base_url\n ) || jsonb_build_object(\n 'rel', 'self',\n 'type', 'application/json',\n 'href', concat(base_url, '/search')\n );\n\n IF has_next THEN\n next := concat(out_records->-1->>'collection', ':', out_records->-1->>'id');\n RAISE NOTICE 'HAS NEXT | %', next;\n links := links || jsonb_build_object(\n 'rel', 'next',\n 'type', 'application/geo+json',\n 'method', 'GET',\n 'href', concat(base_url, '/search?token=next:', next)\n );\n END IF;\n\n IF has_prev THEN\n prev := concat(out_records->0->>'collection', ':', out_records->0->>'id');\n RAISE NOTICE 'HAS PREV | %', prev;\n links := links || jsonb_build_object(\n 'rel', 'prev',\n 'type', 'application/geo+json',\n 'method', 'GET',\n 'href', concat(base_url, '/search?token=prev:', prev)\n );\n END IF;\n\n RAISE NOTICE 'Time to get prev/next %', age_ms(timer);\n timer := clock_timestamp();\n\n\n collection := jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb),\n 'links', links\n );\n\n\n\n IF context(_search->'conf') != 'off' THEN\n collection := collection || jsonb_strip_nulls(jsonb_build_object(\n 'numberMatched', total_count,\n 'numberReturned', coalesce(jsonb_array_length(out_records), 0)\n ));\n ELSE\n collection := collection || jsonb_strip_nulls(jsonb_build_object(\n 'numberReturned', coalesce(jsonb_array_length(out_records), 0)\n ));\n END IF;\n\n IF get_setting_bool('timing', _search->'conf') THEN\n collection = collection || jsonb_build_object('timing', age_ms(init_ts));\n END IF;\n\n RAISE NOTICE 'Time to build final json %', age_ms(timer);\n timer := clock_timestamp();\n\n RAISE NOTICE 'Total Time: %', age_ms(current_timestamp);\n RAISE NOTICE 'RETURNING % records. NEXT: %. PREV: %', collection->>'numberReturned', collection->>'next', collection->>'prev';\n RETURN collection;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.search_query(_search jsonb DEFAULT '{}'::jsonb, updatestats boolean DEFAULT false, _metadata jsonb DEFAULT '{}'::jsonb)\n RETURNS searches\n LANGUAGE plpgsql\n SECURITY DEFINER\nAS $function$\nDECLARE\n search searches%ROWTYPE;\n cached_search searches%ROWTYPE;\n pexplain jsonb;\n t timestamptz;\n i interval;\n doupdate boolean := FALSE;\n insertfound boolean := FALSE;\n ro boolean := pgstac.readonly();\n found_search text;\nBEGIN\n RAISE NOTICE 'SEARCH: %', _search;\n -- Calculate hash, where clause, and order by statement\n search.search := _search;\n search.metadata := _metadata;\n search.hash := search_hash(_search, _metadata);\n search._where := stac_search_to_where(_search);\n search.orderby := sort_sqlorderby(_search);\n search.lastused := now();\n search.usecount := 1;\n\n -- If we are in read only mode, directly return search\n IF ro THEN\n RETURN search;\n END IF;\n\n RAISE NOTICE 'Updating Statistics for search: %s', search;\n -- Update statistics for times used and and when last used\n -- If the entry is locked, rather than waiting, skip updating the stats\n INSERT INTO searches (search, lastused, usecount, metadata)\n VALUES (search.search, now(), 1, search.metadata)\n ON CONFLICT DO NOTHING\n RETURNING * INTO cached_search\n ;\n\n IF NOT FOUND OR cached_search IS NULL THEN\n UPDATE searches SET\n lastused = now(),\n usecount = searches.usecount + 1\n WHERE hash = (\n SELECT hash FROM searches WHERE hash=search.hash FOR UPDATE SKIP LOCKED\n )\n RETURNING * INTO cached_search\n ;\n END IF;\n\n IF cached_search IS NOT NULL THEN\n cached_search._where = search._where;\n cached_search.orderby = search.orderby;\n RETURN cached_search;\n END IF;\n RETURN search;\n\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.search_rows(_where text DEFAULT 'TRUE'::text, _orderby text DEFAULT 'datetime DESC, id DESC'::text, partitions text[] DEFAULT NULL::text[], _limit integer DEFAULT 10)\n RETURNS SETOF items\n LANGUAGE plpgsql\n SET search_path TO 'pgstac', 'public'\nAS $function$\nDECLARE\n base_query text;\n query text;\n sdate timestamptz;\n edate timestamptz;\n n int;\n records_left int := _limit;\n timer timestamptz := clock_timestamp();\n full_timer timestamptz := clock_timestamp();\nBEGIN\nIF _where IS NULL OR trim(_where) = '' THEN\n _where = ' TRUE ';\nEND IF;\nRAISE NOTICE 'Getting chunks for % %', _where, _orderby;\n\nbase_query := $q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n LIMIT %L\n$q$;\n\nIF _orderby ILIKE 'datetime d%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 DESC LOOP\n RAISE NOTICE 'Running Query for % to %. %', sdate, edate, age_ms(full_timer);\n query := format(\n base_query,\n sdate,\n edate,\n _where,\n _orderby,\n records_left\n );\n RAISE DEBUG 'QUERY: %', query;\n timer := clock_timestamp();\n RETURN QUERY EXECUTE query;\n\n GET DIAGNOSTICS n = ROW_COUNT;\n records_left := records_left - n;\n RAISE NOTICE 'Returned %/% Rows From % to %. % to go. Time: %ms', n, _limit, sdate, edate, records_left, age_ms(timer);\n timer := clock_timestamp();\n IF records_left <= 0 THEN\n RAISE NOTICE 'SEARCH_ROWS TOOK %ms', age_ms(full_timer);\n RETURN;\n END IF;\n END LOOP;\nELSIF _orderby ILIKE 'datetime a%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 ASC LOOP\n RAISE NOTICE 'Running Query for % to %. %', sdate, edate, age_ms(full_timer);\n query := format(\n base_query,\n sdate,\n edate,\n _where,\n _orderby,\n records_left\n );\n RAISE DEBUG 'QUERY: %', query;\n timer := clock_timestamp();\n RETURN QUERY EXECUTE query;\n\n GET DIAGNOSTICS n = ROW_COUNT;\n records_left := records_left - n;\n RAISE NOTICE 'Returned %/% Rows From % to %. % to go. Time: %ms', n, _limit, sdate, edate, records_left, age_ms(timer);\n timer := clock_timestamp();\n IF records_left <= 0 THEN\n RAISE NOTICE 'SEARCH_ROWS TOOK %ms', age_ms(full_timer);\n RETURN;\n END IF;\n END LOOP;\nELSE\n query := format($q$\n SELECT * FROM items\n WHERE %s\n ORDER BY %s\n LIMIT %L\n $q$, _where, _orderby, _limit\n );\n RAISE DEBUG 'QUERY: %', query;\n timer := clock_timestamp();\n RETURN QUERY EXECUTE query;\n RAISE NOTICE 'FULL QUERY TOOK %ms', age_ms(timer);\nEND IF;\nRAISE NOTICE 'SEARCH_ROWS TOOK %ms', age_ms(full_timer);\nRETURN;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.stac_search_to_where(j jsonb)\n RETURNS text\n LANGUAGE plpgsql\n STABLE\nAS $function$\nDECLARE\n where_segments text[];\n _where text;\n dtrange tstzrange;\n collections text[];\n geom geometry;\n sdate timestamptz;\n edate timestamptz;\n filterlang text;\n filter jsonb := j->'filter';\n ft_query tsquery;\nBEGIN\n IF j ? 'ids' THEN\n where_segments := where_segments || format('id = ANY (%L) ', to_text_array(j->'ids'));\n END IF;\n\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n where_segments := where_segments || format('collection = ANY (%L) ', collections);\n END IF;\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n\n where_segments := where_segments || format(' datetime <= %L::timestamptz AND end_datetime >= %L::timestamptz ',\n edate,\n sdate\n );\n END IF;\n\n IF j ? 'q' THEN\n ft_query := q_to_tsquery(j->'q');\n where_segments := where_segments || format(\n $quote$\n (\n to_tsvector('english', content->'properties'->>'description') ||\n to_tsvector('english', coalesce(content->'properties'->>'title', '')) ||\n to_tsvector('english', coalesce(content->'properties'->>'keywords', ''))\n ) @@ %L\n $quote$,\n ft_query\n );\n END IF;\n\n geom := stac_geom(j);\n IF geom IS NOT NULL THEN\n where_segments := where_segments || format('st_intersects(geometry, %L)',geom);\n END IF;\n\n filterlang := COALESCE(\n j->>'filter-lang',\n get_setting('default_filter_lang', j->'conf')\n );\n IF NOT filter @? '$.**.op' THEN\n filterlang := 'cql-json';\n END IF;\n\n IF filterlang NOT IN ('cql-json','cql2-json') AND j ? 'filter' THEN\n RAISE EXCEPTION '% is not a supported filter-lang. Please use cql-json or cql2-json.', filterlang;\n END IF;\n\n IF j ? 'query' AND j ? 'filter' THEN\n RAISE EXCEPTION 'Can only use either query or filter at one time.';\n END IF;\n\n IF j ? 'query' THEN\n filter := query_to_cql2(j->'query');\n ELSIF filterlang = 'cql-json' THEN\n filter := cql1_to_cql2(filter);\n END IF;\n RAISE NOTICE 'FILTER: %', filter;\n where_segments := where_segments || cql2_query(filter);\n IF cardinality(where_segments) < 1 THEN\n RETURN ' TRUE ';\n END IF;\n\n _where := array_to_string(array_remove(where_segments, NULL), ' AND ');\n\n IF _where IS NULL OR BTRIM(_where) = '' THEN\n RETURN ' TRUE ';\n END IF;\n RETURN _where;\n\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.update_collection_extents()\n RETURNS void\n LANGUAGE sql\nAS $function$\nUPDATE collections\n SET content = jsonb_set_lax(\n content,\n '{extent}'::text[],\n collection_extent(id, FALSE),\n true,\n 'use_json_null'\n )\n;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.update_partition_stats(_partition text, istrigger boolean DEFAULT false)\n RETURNS void\n LANGUAGE plpgsql\n STRICT SECURITY DEFINER\nAS $function$\nDECLARE\n dtrange tstzrange;\n edtrange tstzrange;\n cdtrange tstzrange;\n cedtrange tstzrange;\n extent geometry;\n collection text;\nBEGIN\n RAISE NOTICE 'Updating stats for %.', _partition;\n EXECUTE format(\n $q$\n SELECT\n tstzrange(min(datetime), max(datetime),'[]'),\n tstzrange(min(end_datetime), max(end_datetime), '[]')\n FROM %I\n $q$,\n _partition\n ) INTO dtrange, edtrange;\n extent := st_estimatedextent('pgstac', _partition, 'geometry');\n RAISE DEBUG 'Estimated Extent: %', extent;\n INSERT INTO partition_stats (partition, dtrange, edtrange, spatial, last_updated)\n SELECT _partition, dtrange, edtrange, extent, now()\n ON CONFLICT (partition) DO\n UPDATE SET\n dtrange=EXCLUDED.dtrange,\n edtrange=EXCLUDED.edtrange,\n spatial=EXCLUDED.spatial,\n last_updated=EXCLUDED.last_updated\n ;\n\n SELECT\n constraint_dtrange, constraint_edtrange, pv.collection\n INTO cdtrange, cedtrange, collection\n FROM partitions_view pv WHERE partition = _partition;\n REFRESH MATERIALIZED VIEW partitions;\n REFRESH MATERIALIZED VIEW partition_steps;\n\n\n RAISE NOTICE 'Checking if we need to modify constraints...';\n RAISE NOTICE 'cdtrange: % dtrange: % cedtrange: % edtrange: %',cdtrange, dtrange, cedtrange, edtrange;\n IF\n (cdtrange IS DISTINCT FROM dtrange OR edtrange IS DISTINCT FROM cedtrange)\n AND NOT istrigger\n THEN\n RAISE NOTICE 'Modifying Constraints';\n RAISE NOTICE 'Existing % %', cdtrange, cedtrange;\n RAISE NOTICE 'New % %', dtrange, edtrange;\n PERFORM drop_table_constraints(_partition);\n PERFORM create_table_constraints(_partition, dtrange, edtrange);\n REFRESH MATERIALIZED VIEW partitions;\n REFRESH MATERIALIZED VIEW partition_steps;\n END IF;\n RAISE NOTICE 'Checking if we need to update collection extents.';\n IF get_setting_bool('update_collection_extent') THEN\n RAISE NOTICE 'updating collection extent for %', collection;\n PERFORM run_or_queue(format($q$\n UPDATE collections\n SET content = jsonb_set_lax(\n content,\n '{extent}'::text[],\n collection_extent(%L, FALSE),\n true,\n 'use_json_null'\n ) WHERE id=%L\n ;\n $q$, collection, collection));\n ELSE\n RAISE NOTICE 'Not updating collection extent for %', collection;\n END IF;\n\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.where_stats(inwhere text, updatestats boolean DEFAULT false, conf jsonb DEFAULT NULL::jsonb)\n RETURNS search_wheres\n LANGUAGE plpgsql\n SECURITY DEFINER\nAS $function$\nDECLARE\n t timestamptz;\n i interval;\n explain_json jsonb;\n partitions text[];\n sw search_wheres%ROWTYPE;\n inwhere_hash text := md5(inwhere);\n _context text := lower(context(conf));\n _stats_ttl interval := context_stats_ttl(conf);\n _estimated_cost_threshold float := context_estimated_cost(conf);\n _estimated_count_threshold int := context_estimated_count(conf);\n ro bool := pgstac.readonly(conf);\nBEGIN\n -- If updatestats is true then set ttl to 0\n IF updatestats THEN\n RAISE DEBUG 'Updatestats set to TRUE, setting TTL to 0';\n _stats_ttl := '0'::interval;\n END IF;\n\n -- If we don't need to calculate context, just return\n IF _context = 'off' THEN\n sw._where = inwhere;\n RETURN sw;\n END IF;\n\n -- Get any stats that we have.\n IF NOT ro THEN\n -- If there is a lock where another process is\n -- updating the stats, wait so that we don't end up calculating a bunch of times.\n SELECT * INTO sw FROM search_wheres WHERE md5(_where)=inwhere_hash FOR UPDATE;\n ELSE\n SELECT * INTO sw FROM search_wheres WHERE md5(_where)=inwhere_hash;\n END IF;\n\n -- If there is a cached row, figure out if we need to update\n IF\n sw IS NOT NULL\n AND sw.statslastupdated IS NOT NULL\n AND sw.total_count IS NOT NULL\n AND now() - sw.statslastupdated <= _stats_ttl\n THEN\n -- we have a cached row with data that is within our ttl\n RAISE DEBUG 'Stats present in table and lastupdated within ttl: %', sw;\n IF NOT ro THEN\n RAISE DEBUG 'Updating search_wheres only bumping lastused and usecount';\n UPDATE search_wheres SET\n lastused = now(),\n usecount = search_wheres.usecount + 1\n WHERE md5(_where) = inwhere_hash\n RETURNING * INTO sw;\n END IF;\n RAISE DEBUG 'Returning cached counts. %', sw;\n RETURN sw;\n END IF;\n\n -- Calculate estimated cost and rows\n -- Use explain to get estimated count/cost\n IF sw.estimated_count IS NULL OR sw.estimated_cost IS NULL THEN\n RAISE DEBUG 'Calculating estimated stats';\n t := clock_timestamp();\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s', inwhere)\n INTO explain_json;\n RAISE DEBUG 'Time for just the explain: %', clock_timestamp() - t;\n i := clock_timestamp() - t;\n\n sw.estimated_count := explain_json->0->'Plan'->'Plan Rows';\n sw.estimated_cost := explain_json->0->'Plan'->'Total Cost';\n sw.time_to_estimate := extract(epoch from i);\n END IF;\n\n RAISE DEBUG 'ESTIMATED_COUNT: %, THRESHOLD %', sw.estimated_count, _estimated_count_threshold;\n RAISE DEBUG 'ESTIMATED_COST: %, THRESHOLD %', sw.estimated_cost, _estimated_cost_threshold;\n\n -- If context is set to auto and the costs are within the threshold return the estimated costs\n IF\n _context = 'auto'\n AND sw.estimated_count >= _estimated_count_threshold\n AND sw.estimated_cost >= _estimated_cost_threshold\n THEN\n IF NOT ro THEN\n INSERT INTO search_wheres (\n _where,\n lastused,\n usecount,\n statslastupdated,\n estimated_count,\n estimated_cost,\n time_to_estimate,\n total_count,\n time_to_count\n ) VALUES (\n inwhere,\n now(),\n 1,\n now(),\n sw.estimated_count,\n sw.estimated_cost,\n sw.time_to_estimate,\n null,\n null\n ) ON CONFLICT ((md5(_where)))\n DO UPDATE SET\n lastused = EXCLUDED.lastused,\n usecount = search_wheres.usecount + 1,\n statslastupdated = EXCLUDED.statslastupdated,\n estimated_count = EXCLUDED.estimated_count,\n estimated_cost = EXCLUDED.estimated_cost,\n time_to_estimate = EXCLUDED.time_to_estimate,\n total_count = EXCLUDED.total_count,\n time_to_count = EXCLUDED.time_to_count\n RETURNING * INTO sw;\n END IF;\n RAISE DEBUG 'Estimates are within thresholds, returning estimates. %', sw;\n RETURN sw;\n END IF;\n\n -- Calculate Actual Count\n t := clock_timestamp();\n RAISE NOTICE 'Calculating actual count...';\n EXECUTE format(\n 'SELECT count(*) FROM items WHERE %s',\n inwhere\n ) INTO sw.total_count;\n i := clock_timestamp() - t;\n RAISE NOTICE 'Actual Count: % -- %', sw.total_count, i;\n sw.time_to_count := extract(epoch FROM i);\n\n IF NOT ro THEN\n INSERT INTO search_wheres (\n _where,\n lastused,\n usecount,\n statslastupdated,\n estimated_count,\n estimated_cost,\n time_to_estimate,\n total_count,\n time_to_count\n ) VALUES (\n inwhere,\n now(),\n 1,\n now(),\n sw.estimated_count,\n sw.estimated_cost,\n sw.time_to_estimate,\n sw.total_count,\n sw.time_to_count\n ) ON CONFLICT ((md5(_where)))\n DO UPDATE SET\n lastused = EXCLUDED.lastused,\n usecount = search_wheres.usecount + 1,\n statslastupdated = EXCLUDED.statslastupdated,\n estimated_count = EXCLUDED.estimated_count,\n estimated_cost = EXCLUDED.estimated_cost,\n time_to_estimate = EXCLUDED.time_to_estimate,\n total_count = EXCLUDED.total_count,\n time_to_count = EXCLUDED.time_to_count\n RETURNING * INTO sw;\n END IF;\n RAISE DEBUG 'Returning with actual count. %', sw;\n RETURN sw;\nEND;\n$function$\n;\n\ncreate materialized view \"pgstac\".\"partition_steps\" as SELECT partition AS name,\n date_trunc('month'::text, lower(partition_dtrange)) AS sdate,\n (date_trunc('month'::text, upper(partition_dtrange)) + '1 mon'::interval) AS edate\n FROM partitions_view\n WHERE ((partition_dtrange IS NOT NULL) AND (partition_dtrange <> 'empty'::tstzrange))\n ORDER BY dtrange;\n\n\ncreate materialized view \"pgstac\".\"partitions\" as SELECT partition,\n collection,\n level,\n reltuples,\n relhastriggers,\n partition_dtrange,\n constraint_dtrange,\n constraint_edtrange,\n dtrange,\n edtrange,\n spatial,\n last_updated\n FROM partitions_view;\n\n\n\n-- END migra calculated SQL\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('id', '{\"title\": \"Item ID\",\"description\": \"Item identifier\",\"$ref\": \"https://schemas.stacspec.org/v1.0.0/item-spec/json-schema/item.json#/definitions/core/allOf/2/properties/id\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('geometry', '{\"title\": \"Item Geometry\",\"description\": \"Item Geometry\",\"$ref\": \"https://geojson.org/schema/Feature.json\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('datetime','{\"description\": \"Datetime\",\"type\": \"string\",\"title\": \"Acquired\",\"format\": \"date-time\",\"pattern\": \"(\\\\+00:00|Z)$\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDELETE FROM queryables a USING queryables b\n WHERE a.name = b.name AND a.collection_ids IS NOT DISTINCT FROM b.collection_ids AND a.id > b.id;\n\n\nINSERT INTO pgstac_settings (name, value) VALUES\n ('context', 'off'),\n ('context_estimated_count', '100000'),\n ('context_estimated_cost', '100000'),\n ('context_stats_ttl', '1 day'),\n ('default_filter_lang', 'cql2-json'),\n ('additional_properties', 'true'),\n ('use_queue', 'false'),\n ('queue_timeout', '10 minutes'),\n ('update_collection_extent', 'false'),\n ('format_cache', 'false'),\n ('readonly', 'false')\nON CONFLICT DO NOTHING\n;\n\n\nINSERT INTO cql2_ops (op, template, types) VALUES\n ('eq', '%s = %s', NULL),\n ('neq', '%s != %s', NULL),\n ('ne', '%s != %s', NULL),\n ('!=', '%s != %s', NULL),\n ('<>', '%s != %s', NULL),\n ('lt', '%s < %s', NULL),\n ('lte', '%s <= %s', NULL),\n ('gt', '%s > %s', NULL),\n ('gte', '%s >= %s', NULL),\n ('le', '%s <= %s', NULL),\n ('ge', '%s >= %s', NULL),\n ('=', '%s = %s', NULL),\n ('<', '%s < %s', NULL),\n ('<=', '%s <= %s', NULL),\n ('>', '%s > %s', NULL),\n ('>=', '%s >= %s', NULL),\n ('like', '%s LIKE %s', NULL),\n ('ilike', '%s ILIKE %s', NULL),\n ('+', '%s + %s', NULL),\n ('-', '%s - %s', NULL),\n ('*', '%s * %s', NULL),\n ('/', '%s / %s', NULL),\n ('not', 'NOT (%s)', NULL),\n ('between', '%s BETWEEN %s AND %s', NULL),\n ('isnull', '%s IS NULL', NULL),\n ('upper', 'upper(%s)', NULL),\n ('lower', 'lower(%s)', NULL),\n ('casei', 'upper(%s)', NULL),\n ('accenti', 'unaccent(%s)', NULL)\nON CONFLICT (op) DO UPDATE\n SET\n template = EXCLUDED.template\n;\n\n\nALTER FUNCTION to_text COST 5000;\nALTER FUNCTION to_float COST 5000;\nALTER FUNCTION to_int COST 5000;\nALTER FUNCTION to_tstz COST 5000;\nALTER FUNCTION to_text_array COST 5000;\n\nALTER FUNCTION update_partition_stats SECURITY DEFINER;\nALTER FUNCTION partition_after_triggerfunc SECURITY DEFINER;\nALTER FUNCTION drop_table_constraints SECURITY DEFINER;\nALTER FUNCTION create_table_constraints SECURITY DEFINER;\nALTER FUNCTION check_partition SECURITY DEFINER;\nALTER FUNCTION repartition SECURITY DEFINER;\nALTER FUNCTION where_stats SECURITY DEFINER;\nALTER FUNCTION search_query SECURITY DEFINER;\nALTER FUNCTION format_item SECURITY DEFINER;\nALTER FUNCTION maintain_index SECURITY DEFINER;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nGRANT ALL ON SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac to pgstac_admin;\n\n-- pgstac_read role limited to using function apis\nGRANT EXECUTE ON FUNCTION search TO pgstac_read;\nGRANT EXECUTE ON FUNCTION search_query TO pgstac_read;\nGRANT EXECUTE ON FUNCTION item_by_id TO pgstac_read;\nGRANT EXECUTE ON FUNCTION get_item TO pgstac_read;\nGRANT SELECT ON ALL TABLES IN SCHEMA pgstac TO pgstac_read;\n\n\nGRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_ingest;\nGRANT USAGE ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_ingest;\n\nREVOKE ALL PRIVILEGES ON PROCEDURE run_queued_queries FROM public;\nGRANT ALL ON PROCEDURE run_queued_queries TO pgstac_admin;\n\nREVOKE ALL PRIVILEGES ON FUNCTION run_queued_queries_intransaction FROM public;\nGRANT ALL ON FUNCTION run_queued_queries_intransaction TO pgstac_admin;\n\nRESET ROLE;\n\nSET ROLE pgstac_ingest;\nSELECT update_partition_stats_q(partition) FROM partitions_view;\nSELECT set_version('0.9.10');\n" }, { "path": "src/pgstac/migrations/pgstac.0.8.6.sql", "content": "RESET ROLE;\nDO $$\nDECLARE\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='postgis') THEN\n CREATE EXTENSION IF NOT EXISTS postgis;\n END IF;\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='btree_gist') THEN\n CREATE EXTENSION IF NOT EXISTS btree_gist;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_admin;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_read;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_ingest;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n\nGRANT pgstac_admin TO current_user;\n\n-- Function to make sure pgstac_admin is the owner of items\nCREATE OR REPLACE FUNCTION pgstac_admin_owns() RETURNS VOID AS $$\nDECLARE\n f RECORD;\nBEGIN\n FOR f IN (\n SELECT\n concat(\n oid::regproc::text,\n '(',\n coalesce(pg_get_function_identity_arguments(oid),''),\n ')'\n ) AS name,\n CASE prokind WHEN 'f' THEN 'FUNCTION' WHEN 'p' THEN 'PROCEDURE' WHEN 'a' THEN 'AGGREGATE' END as typ\n FROM pg_proc\n WHERE\n pronamespace=to_regnamespace('pgstac')\n AND proowner != to_regrole('pgstac_admin')\n AND proname NOT LIKE 'pg_stat%'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n FOR f IN (\n SELECT\n oid::regclass::text as name,\n CASE relkind\n WHEN 'i' THEN 'INDEX'\n WHEN 'I' THEN 'INDEX'\n WHEN 'p' THEN 'TABLE'\n WHEN 'r' THEN 'TABLE'\n WHEN 'v' THEN 'VIEW'\n WHEN 'S' THEN 'SEQUENCE'\n ELSE NULL\n END as typ\n FROM pg_class\n WHERE relnamespace=to_regnamespace('pgstac') and relowner != to_regrole('pgstac_admin') AND relkind IN ('r','p','v','S') AND relname NOT LIKE 'pg_stat'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\nSELECT pgstac_admin_owns();\n\nCREATE SCHEMA IF NOT EXISTS pgstac AUTHORIZATION pgstac_admin;\n\nGRANT ALL ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_admin;\n\nALTER ROLE pgstac_admin SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_read SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_ingest SET SEARCH_PATH TO pgstac, public;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\n\nGRANT pgstac_read TO pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\n\nSET ROLE pgstac_admin;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\nRESET ROLE;\n\nSET ROLE pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\nRESET ROLE;\n\nSET SEARCH_PATH TO pgstac, public;\nSET ROLE pgstac_admin;\n\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS analyze_items;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS validate_constraints;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n-- Install these idempotently as migrations do not put them before trying to modify the collections table\n\n\nCREATE OR REPLACE FUNCTION collection_geom(content jsonb)\nRETURNS geometry AS $$\n WITH box AS (SELECT content->'extent'->'spatial'->'bbox'->0 as box)\n SELECT\n st_makeenvelope(\n (box->>0)::float,\n (box->>1)::float,\n (box->>2)::float,\n (box->>3)::float,\n 4326\n )\n FROM box;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION collection_datetime(content jsonb)\nRETURNS timestamptz AS $$\n SELECT\n CASE\n WHEN\n (content->'extent'->'temporal'->'interval'->0->>0) IS NULL\n THEN '-infinity'::timestamptz\n ELSE\n (content->'extent'->'temporal'->'interval'->0->>0)::timestamptz\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION collection_enddatetime(content jsonb)\nRETURNS timestamptz AS $$\n SELECT\n CASE\n WHEN\n (content->'extent'->'temporal'->'interval'->0->>1) IS NULL\n THEN 'infinity'::timestamptz\n ELSE\n (content->'extent'->'temporal'->'interval'->0->>1)::timestamptz\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE TABLE IF NOT EXISTS migrations (\n version text PRIMARY KEY,\n datetime timestamptz DEFAULT clock_timestamp() NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION get_version() RETURNS text AS $$\n SELECT version FROM pgstac.migrations ORDER BY datetime DESC, version DESC LIMIT 1;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION set_version(text) RETURNS text AS $$\n INSERT INTO pgstac.migrations (version) VALUES ($1)\n ON CONFLICT DO NOTHING\n RETURNING version;\n$$ LANGUAGE SQL;\n\n\nCREATE TABLE IF NOT EXISTS pgstac_settings (\n name text PRIMARY KEY,\n value text NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION table_empty(text) RETURNS boolean AS $$\nDECLARE\n retval boolean;\nBEGIN\n EXECUTE format($q$\n SELECT NOT EXISTS (SELECT 1 FROM %I LIMIT 1)\n $q$,\n $1\n ) INTO retval;\n RETURN retval;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION get_setting(IN _setting text, IN conf jsonb DEFAULT NULL) RETURNS text AS $$\nSELECT COALESCE(\n nullif(conf->>_setting, ''),\n nullif(current_setting(concat('pgstac.',_setting), TRUE),''),\n nullif((SELECT value FROM pgstac.pgstac_settings WHERE name=_setting),'')\n);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_setting_bool(IN _setting text, IN conf jsonb DEFAULT NULL) RETURNS boolean AS $$\nSELECT COALESCE(\n nullif(conf->>_setting, ''),\n nullif(current_setting(concat('pgstac.',_setting), TRUE),''),\n nullif((SELECT value FROM pgstac.pgstac_settings WHERE name=_setting),''),\n 'FALSE'\n)::boolean;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION base_url(conf jsonb DEFAULT NULL) RETURNS text AS $$\n SELECT COALESCE(pgstac.get_setting('base_url', conf), '.');\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION additional_properties() RETURNS boolean AS $$\n SELECT pgstac.get_setting_bool('additional_properties');\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION readonly(conf jsonb DEFAULT NULL) RETURNS boolean AS $$\n SELECT pgstac.get_setting_bool('readonly', conf);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context(conf jsonb DEFAULT NULL) RETURNS text AS $$\n SELECT pgstac.get_setting('context', conf);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context_estimated_count(conf jsonb DEFAULT NULL) RETURNS int AS $$\n SELECT pgstac.get_setting('context_estimated_count', conf)::int;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_estimated_cost();\nCREATE OR REPLACE FUNCTION context_estimated_cost(conf jsonb DEFAULT NULL) RETURNS float AS $$\n SELECT pgstac.get_setting('context_estimated_cost', conf)::float;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_stats_ttl();\nCREATE OR REPLACE FUNCTION context_stats_ttl(conf jsonb DEFAULT NULL) RETURNS interval AS $$\n SELECT pgstac.get_setting('context_stats_ttl', conf)::interval;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION t2s(text) RETURNS text AS $$\n SELECT extract(epoch FROM $1::interval)::text || ' s';\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE STRICT;\n\nCREATE OR REPLACE FUNCTION age_ms(a timestamptz, b timestamptz DEFAULT clock_timestamp()) RETURNS float AS $$\n SELECT abs(extract(epoch from age(a,b)) * 1000);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION queue_timeout() RETURNS interval AS $$\n SELECT t2s(coalesce(\n get_setting('queue_timeout'),\n '1h'\n ))::interval;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION notice(VARIADIC text[]) RETURNS boolean AS $$\nDECLARE\ndebug boolean := current_setting('pgstac.debug', true);\nBEGIN\n IF debug THEN\n RAISE NOTICE 'NOTICE FROM FUNC: % >>>>> %', concat_ws(' | ', $1), clock_timestamp();\n RETURN TRUE;\n END IF;\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION empty_arr(ANYARRAY) RETURNS BOOLEAN AS $$\nSELECT CASE\n WHEN $1 IS NULL THEN TRUE\n WHEN cardinality($1)<1 THEN TRUE\nELSE FALSE\nEND;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_intersection(_a ANYARRAY, _b ANYARRAY) RETURNS ANYARRAY AS $$\n SELECT ARRAY ( SELECT unnest(_a) INTERSECT SELECT UNNEST(_b) );\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION array_map_ident(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_ident(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_map_literal(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_literal(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION array_reverse(anyarray) RETURNS anyarray AS $$\nSELECT ARRAY(\n SELECT $1[i]\n FROM generate_subscripts($1,1) AS s(i)\n ORDER BY i DESC\n);\n$$ LANGUAGE SQL STRICT IMMUTABLE;\n\nDROP TABLE IF EXISTS query_queue;\nCREATE TABLE query_queue (\n query text PRIMARY KEY,\n added timestamptz DEFAULT now()\n);\n\nDROP TABLE IF EXISTS query_queue_history;\nCREATE TABLE query_queue_history(\n query text,\n added timestamptz NOT NULL,\n finished timestamptz NOT NULL DEFAULT now(),\n error text\n);\n\nCREATE OR REPLACE PROCEDURE run_queued_queries() AS $$\nDECLARE\n qitem query_queue%ROWTYPE;\n timeout_ts timestamptz;\n error text;\n cnt int := 0;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n DELETE FROM query_queue WHERE query = (SELECT query FROM query_queue ORDER BY added DESC LIMIT 1 FOR UPDATE SKIP LOCKED) RETURNING * INTO qitem;\n IF NOT FOUND THEN\n EXIT;\n END IF;\n cnt := cnt + 1;\n BEGIN\n RAISE NOTICE 'RUNNING QUERY: %', qitem.query;\n EXECUTE qitem.query;\n EXCEPTION WHEN others THEN\n error := format('%s | %s', SQLERRM, SQLSTATE);\n END;\n INSERT INTO query_queue_history (query, added, finished, error)\n VALUES (qitem.query, qitem.added, clock_timestamp(), error);\n COMMIT;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION run_queued_queries_intransaction() RETURNS int AS $$\nDECLARE\n qitem query_queue%ROWTYPE;\n timeout_ts timestamptz;\n error text;\n cnt int := 0;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n DELETE FROM query_queue WHERE query = (SELECT query FROM query_queue ORDER BY added DESC LIMIT 1 FOR UPDATE SKIP LOCKED) RETURNING * INTO qitem;\n IF NOT FOUND THEN\n RETURN cnt;\n END IF;\n cnt := cnt + 1;\n BEGIN\n qitem.query := regexp_replace(qitem.query, 'CONCURRENTLY', '');\n RAISE NOTICE 'RUNNING QUERY: %', qitem.query;\n\n EXECUTE qitem.query;\n EXCEPTION WHEN others THEN\n error := format('%s | %s', SQLERRM, SQLSTATE);\n RAISE WARNING '%', error;\n END;\n INSERT INTO query_queue_history (query, added, finished, error)\n VALUES (qitem.query, qitem.added, clock_timestamp(), error);\n END LOOP;\n RETURN cnt;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION run_or_queue(query text) RETURNS VOID AS $$\nDECLARE\n use_queue text := COALESCE(get_setting('use_queue'), 'FALSE')::boolean;\nBEGIN\n IF get_setting_bool('debug') THEN\n RAISE NOTICE '%', query;\n END IF;\n IF use_queue THEN\n INSERT INTO query_queue (query) VALUES (query) ON CONFLICT DO NOTHING;\n ELSE\n EXECUTE query;\n END IF;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n\nDROP FUNCTION IF EXISTS check_pgstac_settings;\nCREATE OR REPLACE FUNCTION check_pgstac_settings(_sysmem text DEFAULT NULL) RETURNS VOID AS $$\nDECLARE\n settingval text;\n sysmem bigint := pg_size_bytes(_sysmem);\n effective_cache_size bigint := pg_size_bytes(current_setting('effective_cache_size', TRUE));\n shared_buffers bigint := pg_size_bytes(current_setting('shared_buffers', TRUE));\n work_mem bigint := pg_size_bytes(current_setting('work_mem', TRUE));\n max_connections int := current_setting('max_connections', TRUE);\n maintenance_work_mem bigint := pg_size_bytes(current_setting('maintenance_work_mem', TRUE));\n seq_page_cost float := current_setting('seq_page_cost', TRUE);\n random_page_cost float := current_setting('random_page_cost', TRUE);\n temp_buffers bigint := pg_size_bytes(current_setting('temp_buffers', TRUE));\n r record;\nBEGIN\n IF _sysmem IS NULL THEN\n RAISE NOTICE 'Call function with the size of your system memory `SELECT check_pgstac_settings(''4GB'')` to get pg system setting recommendations.';\n ELSE\n IF effective_cache_size < (sysmem * 0.5) THEN\n RAISE WARNING 'effective_cache_size of % is set low for a system with %. Recomended value between % and %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.5), pg_size_pretty(sysmem * 0.75);\n ELSIF effective_cache_size > (sysmem * 0.75) THEN\n RAISE WARNING 'effective_cache_size of % is set high for a system with %. Recomended value between % and %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.5), pg_size_pretty(sysmem * 0.75);\n ELSE\n RAISE NOTICE 'effective_cache_size of % is set appropriately for a system with %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem);\n END IF;\n\n IF shared_buffers < (sysmem * 0.2) THEN\n RAISE WARNING 'shared_buffers of % is set low for a system with %. Recomended value between % and %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.2), pg_size_pretty(sysmem * 0.3);\n ELSIF shared_buffers > (sysmem * 0.3) THEN\n RAISE WARNING 'shared_buffers of % is set high for a system with %. Recomended value between % and %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.2), pg_size_pretty(sysmem * 0.3);\n ELSE\n RAISE NOTICE 'shared_buffers of % is set appropriately for a system with %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem);\n END IF;\n shared_buffers = sysmem * 0.3;\n IF maintenance_work_mem < (sysmem * 0.2) THEN\n RAISE WARNING 'maintenance_work_mem of % is set low for shared_buffers of %. Recomended value between % and %', pg_size_pretty(maintenance_work_mem), pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers * 0.2), pg_size_pretty(shared_buffers * 0.3);\n ELSIF maintenance_work_mem > (shared_buffers * 0.3) THEN\n RAISE WARNING 'maintenance_work_mem of % is set high for shared_buffers of %. Recomended value between % and %', pg_size_pretty(maintenance_work_mem), pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers * 0.2), pg_size_pretty(shared_buffers * 0.3);\n ELSE\n RAISE NOTICE 'maintenance_work_mem of % is set appropriately for shared_buffers of %', pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers);\n END IF;\n\n IF work_mem * max_connections > shared_buffers THEN\n RAISE WARNING 'work_mem setting of % is set high for % max_connections please reduce work_mem to % or decrease max_connections to %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers/max_connections), floor(shared_buffers/work_mem);\n ELSIF work_mem * max_connections < (shared_buffers * 0.75) THEN\n RAISE WARNING 'work_mem setting of % is set low for % max_connections you may consider raising work_mem to % or increasing max_connections to %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers/max_connections), floor(shared_buffers/work_mem);\n ELSE\n RAISE NOTICE 'work_mem setting of % and max_connections of % are adequate for shared_buffers of %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers);\n END IF;\n\n IF random_page_cost / seq_page_cost != 1.1 THEN\n RAISE WARNING 'random_page_cost (%) /seq_page_cost (%) should be set to 1.1 for SSD. Change random_page_cost to %', random_page_cost, seq_page_cost, 1.1 * seq_page_cost;\n ELSE\n RAISE NOTICE 'random_page_cost and seq_page_cost set appropriately for SSD';\n END IF;\n\n IF temp_buffers < greatest(pg_size_bytes('128MB'),(maintenance_work_mem / 2)) THEN\n RAISE WARNING 'pgstac makes heavy use of temp tables, consider raising temp_buffers from % to %', pg_size_pretty(temp_buffers), greatest('128MB', pg_size_pretty((shared_buffers / 16)));\n END IF;\n END IF;\n\n RAISE NOTICE 'VALUES FOR PGSTAC VARIABLES';\n RAISE NOTICE 'These can be set either as GUC system variables or by setting in the pgstac_settings table.';\n\n FOR r IN SELECT name, get_setting(name) as setting, CASE WHEN current_setting(concat('pgstac.',name), TRUE) IS NOT NULL THEN concat('pgstac.',name, ' GUC') WHEN value IS NOT NULL THEN 'pgstac_settings table' ELSE 'Not Set' END as loc FROM pgstac_settings LOOP\n RAISE NOTICE '% is set to % from the %', r.name, r.setting, r.loc;\n END LOOP;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pg_cron';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider intalling pg_cron which can be used to automate tasks';\n ELSE\n RAISE NOTICE 'pg_cron % is installed', settingval;\n END IF;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pgstattuple';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider installing the pgstattuple extension which can be used to help maintain tables and indexes.';\n ELSE\n RAISE NOTICE 'pgstattuple % is installed', settingval;\n END IF;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pg_stat_statements';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider installing the pg_stat_statements extension which is very helpful for tracking the types of queries on the system';\n ELSE\n RAISE NOTICE 'pg_stat_statements % is installed', settingval;\n IF current_setting('pg_stat_statements.track_statements', TRUE) IS DISTINCT FROM 'all' THEN\n RAISE WARNING 'SET pg_stat_statements.track_statements TO ''all''; --In order to track statements within functions.';\n END IF;\n END IF;\n\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac, public SET CLIENT_MIN_MESSAGES TO NOTICE;\nCREATE OR REPLACE FUNCTION to_int(jsonb) RETURNS int AS $$\n SELECT floor(($1->>0)::float)::int;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION to_float(jsonb) RETURNS float AS $$\n SELECT ($1->>0)::float;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION to_tstz(jsonb) RETURNS timestamptz AS $$\n SELECT ($1->>0)::timestamptz;\n$$ LANGUAGE SQL IMMUTABLE STRICT SET TIME ZONE 'UTC' COST 5000 PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION to_text(jsonb) RETURNS text AS $$\n SELECT CASE WHEN jsonb_typeof($1) IN ('array','object') THEN $1::text ELSE $1->>0 END;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION to_text_array(jsonb) RETURNS text[] AS $$\n SELECT\n CASE jsonb_typeof($1)\n WHEN 'array' THEN ARRAY(SELECT jsonb_array_elements_text($1))\n ELSE ARRAY[$1->>0]\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION bbox_geom(_bbox jsonb) RETURNS geometry AS $$\nSELECT CASE jsonb_array_length(_bbox)\n WHEN 4 THEN\n ST_SetSRID(ST_MakeEnvelope(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float,\n (_bbox->>3)::float\n ),4326)\n WHEN 6 THEN\n ST_SetSRID(ST_3DMakeBox(\n ST_MakePoint(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float\n ),\n ST_MakePoint(\n (_bbox->>3)::float,\n (_bbox->>4)::float,\n (_bbox->>5)::float\n )\n ),4326)\n ELSE null END;\n;\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION geom_bbox(_geom geometry) RETURNS jsonb AS $$\n SELECT jsonb_build_array(\n st_xmin(_geom),\n st_ymin(_geom),\n st_xmax(_geom),\n st_ymax(_geom)\n );\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION flip_jsonb_array(j jsonb) RETURNS jsonb AS $$\n SELECT jsonb_agg(value) FROM (SELECT value FROM jsonb_array_elements(j) WITH ORDINALITY ORDER BY ordinality DESC) as t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION explode_dotpaths(j jsonb) RETURNS SETOF text[] AS $$\n SELECT string_to_array(p, '.') as e FROM jsonb_array_elements_text(j) p;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION explode_dotpaths_recurse(IN j jsonb) RETURNS SETOF text[] AS $$\n WITH RECURSIVE t AS (\n SELECT e FROM explode_dotpaths(j) e\n UNION ALL\n SELECT e[1:cardinality(e)-1]\n FROM t\n WHERE cardinality(e)>1\n ) SELECT e FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION jsonb_set_nested(j jsonb, path text[], val jsonb) RETURNS jsonb AS $$\nDECLARE\nBEGIN\n IF cardinality(path) > 1 THEN\n FOR i IN 1..(cardinality(path)-1) LOOP\n IF j #> path[:i] IS NULL THEN\n j := jsonb_set_lax(j, path[:i], '{}', TRUE);\n END IF;\n END LOOP;\n END IF;\n RETURN jsonb_set_lax(j, path, val, true);\n\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\n\n\nCREATE OR REPLACE FUNCTION jsonb_include(j jsonb, f jsonb) RETURNS jsonb AS $$\nDECLARE\n includes jsonb := f-> 'include';\n outj jsonb := '{}'::jsonb;\n path text[];\nBEGIN\n IF\n includes IS NULL\n OR jsonb_array_length(includes) = 0\n THEN\n RETURN j;\n ELSE\n includes := includes || '[\"id\",\"collection\"]'::jsonb;\n FOR path IN SELECT explode_dotpaths(includes) LOOP\n outj := jsonb_set_nested(outj, path, j #> path);\n END LOOP;\n END IF;\n RETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_exclude(j jsonb, f jsonb) RETURNS jsonb AS $$\nDECLARE\n excludes jsonb := f-> 'exclude';\n outj jsonb := j;\n path text[];\nBEGIN\n IF\n excludes IS NULL\n OR jsonb_array_length(excludes) = 0\n THEN\n RETURN j;\n ELSE\n FOR path IN SELECT explode_dotpaths(excludes) LOOP\n outj := outj #- path;\n END LOOP;\n END IF;\n RETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_fields(j jsonb, f jsonb DEFAULT '{\"fields\":[]}') RETURNS jsonb AS $$\n SELECT jsonb_exclude(jsonb_include(j, f), f);\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION merge_jsonb(_a jsonb, _b jsonb) RETURNS jsonb AS $$\n SELECT\n CASE\n WHEN _a = '\"𒍟※\"'::jsonb THEN NULL\n WHEN _a IS NULL OR jsonb_typeof(_a) = 'null' THEN _b\n WHEN jsonb_typeof(_a) = 'object' AND jsonb_typeof(_b) = 'object' THEN\n (\n SELECT\n jsonb_strip_nulls(\n jsonb_object_agg(\n key,\n merge_jsonb(a.value, b.value)\n )\n )\n FROM\n jsonb_each(coalesce(_a,'{}'::jsonb)) as a\n FULL JOIN\n jsonb_each(coalesce(_b,'{}'::jsonb)) as b\n USING (key)\n )\n WHEN\n jsonb_typeof(_a) = 'array'\n AND jsonb_typeof(_b) = 'array'\n AND jsonb_array_length(_a) = jsonb_array_length(_b)\n THEN\n (\n SELECT jsonb_agg(m) FROM\n ( SELECT\n merge_jsonb(\n jsonb_array_elements(_a),\n jsonb_array_elements(_b)\n ) as m\n ) as l\n )\n ELSE _a\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION strip_jsonb(_a jsonb, _b jsonb) RETURNS jsonb AS $$\n SELECT\n CASE\n\n WHEN (_a IS NULL OR jsonb_typeof(_a) = 'null') AND _b IS NOT NULL AND jsonb_typeof(_b) != 'null' THEN '\"𒍟※\"'::jsonb\n WHEN _b IS NULL OR jsonb_typeof(_a) = 'null' THEN _a\n WHEN _a = _b AND jsonb_typeof(_a) = 'object' THEN '{}'::jsonb\n WHEN _a = _b THEN NULL\n WHEN jsonb_typeof(_a) = 'object' AND jsonb_typeof(_b) = 'object' THEN\n (\n SELECT\n jsonb_strip_nulls(\n jsonb_object_agg(\n key,\n strip_jsonb(a.value, b.value)\n )\n )\n FROM\n jsonb_each(_a) as a\n FULL JOIN\n jsonb_each(_b) as b\n USING (key)\n )\n WHEN\n jsonb_typeof(_a) = 'array'\n AND jsonb_typeof(_b) = 'array'\n AND jsonb_array_length(_a) = jsonb_array_length(_b)\n THEN\n (\n SELECT jsonb_agg(m) FROM\n ( SELECT\n strip_jsonb(\n jsonb_array_elements(_a),\n jsonb_array_elements(_b)\n ) as m\n ) as l\n )\n ELSE _a\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION nullif_jsonbnullempty(j jsonb) RETURNS jsonb AS $$\n SELECT nullif(nullif(nullif(j,'null'::jsonb),'{}'::jsonb),'[]'::jsonb);\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION jsonb_array_unique(j jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(jsonb_agg(DISTINCT a)) v FROM jsonb_array_elements(j) a;\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_concat_ignorenull(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT coalesce(a,'[]'::jsonb) || coalesce(b,'[]'::jsonb);\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_least(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(least(nullif_jsonbnullempty(a), nullif_jsonbnullempty(b)));\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_greatest(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(greatest(a, b));\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION first_notnull_sfunc(anyelement, anyelement) RETURNS anyelement AS $$\n SELECT COALESCE($1,$2);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE AGGREGATE first_notnull(anyelement)(\n SFUNC = first_notnull_sfunc,\n STYPE = anyelement\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_array_unique_merge(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_concat_ignorenull,\n FINALFUNC = jsonb_array_unique\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_min(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_least\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_max(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_greatest\n);\n/* looks for a geometry in a stac item first from geometry and falling back to bbox */\nCREATE OR REPLACE FUNCTION stac_geom(value jsonb) RETURNS geometry AS $$\nSELECT\n CASE\n WHEN value ? 'intersects' THEN\n ST_GeomFromGeoJSON(value->>'intersects')\n WHEN value ? 'geometry' THEN\n ST_GeomFromGeoJSON(value->>'geometry')\n WHEN value ? 'bbox' THEN\n pgstac.bbox_geom(value->'bbox')\n ELSE NULL\n END as geometry\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION stac_daterange(\n value jsonb\n) RETURNS tstzrange AS $$\nDECLARE\n props jsonb := value;\n dt timestamptz;\n edt timestamptz;\nBEGIN\n IF props ? 'properties' THEN\n props := props->'properties';\n END IF;\n IF\n props ? 'start_datetime'\n AND props->>'start_datetime' IS NOT NULL\n AND props ? 'end_datetime'\n AND props->>'end_datetime' IS NOT NULL\n THEN\n dt := props->>'start_datetime';\n edt := props->>'end_datetime';\n IF dt > edt THEN\n RAISE EXCEPTION 'start_datetime must be < end_datetime';\n END IF;\n ELSE\n dt := props->>'datetime';\n edt := props->>'datetime';\n END IF;\n IF dt is NULL OR edt IS NULL THEN\n RAISE NOTICE 'DT: %, EDT: %', dt, edt;\n RAISE EXCEPTION 'Either datetime (%) or both start_datetime (%) and end_datetime (%) must be set.', props->>'datetime',props->>'start_datetime',props->>'end_datetime';\n END IF;\n RETURN tstzrange(dt, edt, '[]');\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_datetime(value jsonb) RETURNS timestamptz AS $$\n SELECT lower(stac_daterange(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_end_datetime(value jsonb) RETURNS timestamptz AS $$\n SELECT upper(stac_daterange(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE TABLE IF NOT EXISTS stac_extensions(\n url text PRIMARY KEY,\n content jsonb\n);\nCREATE OR REPLACE FUNCTION collection_base_item(content jsonb) RETURNS jsonb AS $$\n SELECT jsonb_build_object(\n 'type', 'Feature',\n 'stac_version', content->'stac_version',\n 'assets', content->'item_assets',\n 'collection', content->'id'\n );\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE TABLE IF NOT EXISTS collections (\n key bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,\n id text GENERATED ALWAYS AS (content->>'id') STORED UNIQUE NOT NULL,\n content JSONB NOT NULL,\n base_item jsonb GENERATED ALWAYS AS (pgstac.collection_base_item(content)) STORED,\n geometry geometry GENERATED ALWAYS AS (pgstac.collection_geom(content)) STORED,\n datetime timestamptz GENERATED ALWAYS AS (pgstac.collection_datetime(content)) STORED,\n end_datetime timestamptz GENERATED ALWAYS AS (pgstac.collection_enddatetime(content)) STORED,\n private jsonb,\n partition_trunc text CHECK (partition_trunc IN ('year', 'month'))\n);\n\nCREATE OR REPLACE FUNCTION collection_base_item(cid text) RETURNS jsonb AS $$\n SELECT pgstac.collection_base_item(content) FROM pgstac.collections WHERE id = cid LIMIT 1;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION create_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION update_collection(data jsonb) RETURNS VOID AS $$\nDECLARE\n out collections%ROWTYPE;\nBEGIN\n UPDATE collections SET content=data WHERE id = data->>'id' RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ON CONFLICT (id) DO\n UPDATE\n SET content=EXCLUDED.content\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION delete_collection(_id text) RETURNS VOID AS $$\nDECLARE\n out collections%ROWTYPE;\nBEGIN\n DELETE FROM collections WHERE id = _id RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION get_collection(id text) RETURNS jsonb AS $$\n SELECT content FROM collections\n WHERE id=$1\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION all_collections() RETURNS jsonb AS $$\n SELECT coalesce(jsonb_agg(content), '[]'::jsonb) FROM collections;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION collection_delete_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n collection_base_partition text := concat('_items_', OLD.key);\nBEGIN\n EXECUTE format($q$\n DELETE FROM partition_stats WHERE partition IN (\n SELECT partition FROM partition_sys_meta\n WHERE collection=%L\n );\n DROP TABLE IF EXISTS %I CASCADE;\n $q$,\n OLD.id,\n collection_base_partition\n );\n RETURN OLD;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER collection_delete_trigger BEFORE DELETE ON collections\nFOR EACH ROW EXECUTE FUNCTION collection_delete_trigger_func();\nCREATE OR REPLACE FUNCTION queryable_signature(n text, c text[]) RETURNS text AS $$\n SELECT concat(n, c);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TABLE queryables (\n id bigint GENERATED ALWAYS AS identity PRIMARY KEY,\n name text NOT NULL,\n collection_ids text[], -- used to determine what partitions to create indexes on\n definition jsonb,\n property_path text,\n property_wrapper text,\n property_index_type text\n);\nCREATE INDEX queryables_name_idx ON queryables (name);\nCREATE INDEX queryables_collection_idx ON queryables USING GIN (collection_ids);\nCREATE INDEX queryables_property_wrapper_idx ON queryables (property_wrapper);\n\nCREATE OR REPLACE FUNCTION pgstac.queryables_constraint_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n allcollections text[];\nBEGIN\n RAISE NOTICE 'Making sure that name/collection is unique for queryables %', NEW;\n IF NEW.collection_ids IS NOT NULL THEN\n IF EXISTS (\n SELECT 1\n FROM unnest(NEW.collection_ids) c\n LEFT JOIN\n collections\n ON (collections.id = c)\n WHERE collections.id IS NULL\n ) THEN\n RAISE foreign_key_violation USING MESSAGE = format(\n 'One or more collections in %s do not exist.', NEW.collection_ids\n );\n RETURN NULL;\n END IF;\n END IF;\n IF TG_OP = 'INSERT' THEN\n IF EXISTS (\n SELECT 1 FROM queryables q\n WHERE\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n )\n ) THEN\n RAISE unique_violation USING MESSAGE = format(\n 'There is already a queryable for %s for a collection in %s: %s',\n NEW.name,\n NEW.collection_ids,\n\t\t\t\t(SELECT json_agg(row_to_json(q)) FROM queryables q WHERE\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n ))\n );\n RETURN NULL;\n END IF;\n END IF;\n IF TG_OP = 'UPDATE' THEN\n IF EXISTS (\n SELECT 1 FROM queryables q\n WHERE\n q.id != NEW.id\n AND\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n )\n ) THEN\n RAISE unique_violation\n USING MESSAGE = format(\n 'There is already a queryable for %s for a collection in %s',\n NEW.name,\n NEW.collection_ids\n );\n RETURN NULL;\n END IF;\n END IF;\n\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER queryables_constraint_insert_trigger\nBEFORE INSERT ON queryables\nFOR EACH ROW EXECUTE PROCEDURE queryables_constraint_triggerfunc();\n\nCREATE TRIGGER queryables_constraint_update_trigger\nBEFORE UPDATE ON queryables\nFOR EACH ROW\nWHEN (NEW.name = OLD.name AND NEW.collection_ids IS DISTINCT FROM OLD.collection_ids)\nEXECUTE PROCEDURE queryables_constraint_triggerfunc();\n\n\nCREATE OR REPLACE FUNCTION array_to_path(arr text[]) RETURNS text AS $$\n SELECT string_agg(\n quote_literal(v),\n '->'\n ) FROM unnest(arr) v;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\n\n\nCREATE OR REPLACE FUNCTION queryable(\n IN dotpath text,\n OUT path text,\n OUT expression text,\n OUT wrapper text,\n OUT nulled_wrapper text\n) AS $$\nDECLARE\n q RECORD;\n path_elements text[];\nBEGIN\n IF dotpath IN ('id', 'geometry', 'datetime', 'end_datetime', 'collection') THEN\n path := dotpath;\n expression := dotpath;\n wrapper := NULL;\n RETURN;\n END IF;\n SELECT * INTO q FROM queryables\n WHERE\n name=dotpath\n OR name = 'properties.' || dotpath\n OR name = replace(dotpath, 'properties.', '')\n ;\n IF q.property_wrapper IS NULL THEN\n IF q.definition->>'type' = 'number' THEN\n wrapper := 'to_float';\n nulled_wrapper := wrapper;\n ELSIF q.definition->>'format' = 'date-time' THEN\n wrapper := 'to_tstz';\n nulled_wrapper := wrapper;\n ELSE\n nulled_wrapper := NULL;\n wrapper := 'to_text';\n END IF;\n ELSE\n wrapper := q.property_wrapper;\n nulled_wrapper := wrapper;\n END IF;\n IF q.property_path IS NOT NULL THEN\n path := q.property_path;\n ELSE\n path_elements := string_to_array(dotpath, '.');\n IF path_elements[1] IN ('links', 'assets', 'stac_version', 'stac_extensions') THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSIF path_elements[1] = 'properties' THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSE\n path := format($F$content->'properties'->%s$F$, array_to_path(path_elements));\n END IF;\n END IF;\n expression := format('%I(%s)', wrapper, path);\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\nCREATE OR REPLACE FUNCTION unnest_collection(collection_ids text[] DEFAULT NULL) RETURNS SETOF text AS $$\n DECLARE\n BEGIN\n IF collection_ids IS NULL THEN\n RETURN QUERY SELECT id FROM collections;\n END IF;\n RETURN QUERY SELECT unnest(collection_ids);\n END;\n$$ LANGUAGE PLPGSQL STABLE;\n\nCREATE OR REPLACE FUNCTION normalize_indexdef(def text) RETURNS text AS $$\nDECLARE\nBEGIN\n def := btrim(def, ' \\n\\t');\n\tdef := regexp_replace(def, '^CREATE (UNIQUE )?INDEX ([^ ]* )?ON (ONLY )?([^ ]* )?', '', 'i');\n RETURN def;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION indexdef(q queryables) RETURNS text AS $$\n DECLARE\n out text;\n BEGIN\n IF q.name = 'id' THEN\n out := 'CREATE UNIQUE INDEX ON %I USING btree (id)';\n ELSIF q.name = 'datetime' THEN\n out := 'CREATE INDEX ON %I USING btree (datetime DESC, end_datetime)';\n ELSIF q.name = 'geometry' THEN\n out := 'CREATE INDEX ON %I USING gist (geometry)';\n ELSE\n out := format($q$CREATE INDEX ON %%I USING %s (%s(((content -> 'properties'::text) -> %L::text)))$q$,\n lower(COALESCE(q.property_index_type, 'BTREE')),\n lower(COALESCE(q.property_wrapper, 'to_text')),\n q.name\n );\n END IF;\n RETURN btrim(out, ' \\n\\t');\n END;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nDROP VIEW IF EXISTS pgstac_indexes;\nCREATE VIEW pgstac_indexes AS\nSELECT\n i.schemaname,\n i.tablename,\n i.indexname,\n regexp_replace(btrim(replace(replace(indexdef, i.indexname, ''),'pgstac.',''),' \\t\\n'), '[ ]+', ' ', 'g') as idx,\n COALESCE(\n (regexp_match(indexdef, '\\(([a-zA-Z]+)\\)'))[1],\n (regexp_match(indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_-]+)''::text'))[1],\n CASE WHEN indexdef ~* '\\(datetime desc, end_datetime\\)' THEN 'datetime' ELSE NULL END\n ) AS field,\n pg_table_size(i.indexname::text) as index_size,\n pg_size_pretty(pg_table_size(i.indexname::text)) as index_size_pretty\nFROM\n pg_indexes i\nWHERE i.schemaname='pgstac' and i.tablename ~ '_items_' AND indexdef !~* ' only ';\n\nDROP VIEW IF EXISTS pgstac_index_stats;\nCREATE VIEW pgstac_indexes_stats AS\nSELECT\n i.schemaname,\n i.tablename,\n i.indexname,\n indexdef,\n COALESCE(\n (regexp_match(indexdef, '\\(([a-zA-Z]+)\\)'))[1],\n (regexp_match(indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_]+)''::text'))[1],\n CASE WHEN indexdef ~* '\\(datetime desc, end_datetime\\)' THEN 'datetime_end_datetime' ELSE NULL END\n ) AS field,\n pg_table_size(i.indexname::text) as index_size,\n pg_size_pretty(pg_table_size(i.indexname::text)) as index_size_pretty,\n n_distinct,\n most_common_vals::text::text[],\n most_common_freqs::text::text[],\n histogram_bounds::text::text[],\n correlation\nFROM\n pg_indexes i\n LEFT JOIN pg_stats s ON (s.tablename = i.indexname)\nWHERE i.schemaname='pgstac' and i.tablename ~ '_items_';\n\nCREATE OR REPLACE FUNCTION queryable_indexes(\n IN treeroot text DEFAULT 'items',\n IN changes boolean DEFAULT FALSE,\n OUT collection text,\n OUT partition text,\n OUT field text,\n OUT indexname text,\n OUT existing_idx text,\n OUT queryable_idx text\n) RETURNS SETOF RECORD AS $$\nWITH p AS (\n SELECT\n relid::text as partition,\n replace(replace(\n CASE\n WHEN parentrelid::regclass::text='items' THEN pg_get_expr(c.relpartbound, c.oid)\n ELSE pg_get_expr(parent.relpartbound, parent.oid)\n END,\n 'FOR VALUES IN (''',''), ''')',\n ''\n ) AS collection\n FROM pg_partition_tree(treeroot)\n JOIN pg_class c ON (relid::regclass = c.oid)\n JOIN pg_class parent ON (parentrelid::regclass = parent.oid AND isleaf)\n ), i AS (\n SELECT\n partition,\n indexname,\n regexp_replace(btrim(replace(replace(indexdef, indexname, ''),'pgstac.',''),' \\t\\n'), '[ ]+', ' ', 'g') as iidx,\n COALESCE(\n (regexp_match(indexdef, '\\(([a-zA-Z]+)\\)'))[1],\n (regexp_match(indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_-]+)''::text'))[1],\n CASE WHEN indexdef ~* '\\(datetime desc, end_datetime\\)' THEN 'datetime' ELSE NULL END\n ) AS field\n FROM\n pg_indexes\n JOIN p ON (tablename=partition)\n ), q AS (\n SELECT\n name AS field,\n collection,\n partition,\n format(indexdef(queryables), partition) as qidx\n FROM queryables, unnest_collection(queryables.collection_ids) collection\n JOIN p USING (collection)\n WHERE property_index_type IS NOT NULL OR name IN ('datetime','geometry','id')\n )\n SELECT\n collection,\n partition,\n field,\n indexname,\n iidx as existing_idx,\n qidx as queryable_idx\n FROM i FULL JOIN q USING (field, partition)\n WHERE CASE WHEN changes THEN lower(iidx) IS DISTINCT FROM lower(qidx) ELSE TRUE END;\n;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION maintain_index(\n indexname text,\n queryable_idx text,\n dropindexes boolean DEFAULT FALSE,\n rebuildindexes boolean DEFAULT FALSE,\n idxconcurrently boolean DEFAULT FALSE\n) RETURNS VOID AS $$\nDECLARE\nBEGIN\n IF indexname IS NOT NULL THEN\n IF dropindexes OR queryable_idx IS NOT NULL THEN\n EXECUTE format('DROP INDEX IF EXISTS %I;', indexname);\n ELSIF rebuildindexes THEN\n IF idxconcurrently THEN\n EXECUTE format('REINDEX INDEX CONCURRENTLY %I;', indexname);\n ELSE\n EXECUTE format('REINDEX INDEX CONCURRENTLY %I;', indexname);\n END IF;\n END IF;\n END IF;\n IF queryable_idx IS NOT NULL THEN\n IF idxconcurrently THEN\n EXECUTE replace(queryable_idx, 'INDEX', 'INDEX CONCURRENTLY');\n ELSE EXECUTE queryable_idx;\n END IF;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\n\nset check_function_bodies to off;\nCREATE OR REPLACE FUNCTION maintain_partition_queries(\n part text DEFAULT 'items',\n dropindexes boolean DEFAULT FALSE,\n rebuildindexes boolean DEFAULT FALSE,\n idxconcurrently boolean DEFAULT FALSE\n) RETURNS SETOF text AS $$\nDECLARE\n rec record;\n q text;\nBEGIN\n FOR rec IN (\n SELECT * FROM queryable_indexes(part,true)\n ) LOOP\n q := format(\n 'SELECT maintain_index(\n %L,%L,%L,%L,%L\n );',\n rec.indexname,\n rec.queryable_idx,\n dropindexes,\n rebuildindexes,\n idxconcurrently\n );\n RAISE NOTICE 'Q: %s', q;\n RETURN NEXT q;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION maintain_partitions(\n part text DEFAULT 'items',\n dropindexes boolean DEFAULT FALSE,\n rebuildindexes boolean DEFAULT FALSE\n) RETURNS VOID AS $$\n WITH t AS (\n SELECT run_or_queue(q) FROM maintain_partition_queries(part, dropindexes, rebuildindexes) q\n ) SELECT count(*) FROM t;\n$$ LANGUAGE SQL;\n\n\nCREATE OR REPLACE FUNCTION queryables_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\nBEGIN\n PERFORM maintain_partitions();\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER queryables_trigger AFTER INSERT OR UPDATE ON queryables\nFOR EACH STATEMENT EXECUTE PROCEDURE queryables_trigger_func();\n\n\nCREATE OR REPLACE FUNCTION get_queryables(_collection_ids text[] DEFAULT NULL) RETURNS jsonb AS $$\nDECLARE\nBEGIN\n -- Build up queryables if the input contains valid collection ids or is empty\n IF EXISTS (\n SELECT 1 FROM collections\n WHERE\n _collection_ids IS NULL\n OR cardinality(_collection_ids) = 0\n OR id = ANY(_collection_ids)\n )\n THEN\n RETURN (\n WITH base AS (\n SELECT\n unnest(collection_ids) as collection_id,\n name,\n coalesce(definition, '{\"type\":\"string\"}'::jsonb) as definition\n FROM queryables\n WHERE\n _collection_ids IS NULL OR\n _collection_ids = '{}'::text[] OR\n _collection_ids && collection_ids\n UNION ALL\n SELECT null, name, coalesce(definition, '{\"type\":\"string\"}'::jsonb) as definition\n FROM queryables WHERE collection_ids IS NULL OR collection_ids = '{}'::text[]\n ), g AS (\n SELECT\n name,\n first_notnull(definition) as definition,\n jsonb_array_unique_merge(definition->'enum') as enum,\n jsonb_min(definition->'minimum') as minimum,\n jsonb_min(definition->'maxiumn') as maximum\n FROM base\n GROUP BY 1\n )\n SELECT\n jsonb_build_object(\n '$schema', 'http://json-schema.org/draft-07/schema#',\n '$id', '',\n 'type', 'object',\n 'title', 'STAC Queryables.',\n 'properties', jsonb_object_agg(\n name,\n definition\n ||\n jsonb_strip_nulls(jsonb_build_object(\n 'enum', enum,\n 'minimum', minimum,\n 'maximum', maximum\n ))\n ),\n 'additionalProperties', pgstac.additional_properties()\n )\n FROM g\n );\n ELSE\n RETURN NULL;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\nCREATE OR REPLACE FUNCTION get_queryables(_collection text DEFAULT NULL) RETURNS jsonb AS $$\n SELECT\n CASE\n WHEN _collection IS NULL THEN get_queryables(NULL::text[])\n ELSE get_queryables(ARRAY[_collection])\n END\n ;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_queryables() RETURNS jsonb AS $$\n SELECT get_queryables(NULL::text[]);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION schema_qualify_refs(url text, j jsonb) returns jsonb as $$\n SELECT regexp_replace(j::text, '\"\\$ref\": \"#', concat('\"$ref\": \"', url, '#'), 'g')::jsonb;\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE VIEW stac_extension_queryables AS\nSELECT DISTINCT key as name, schema_qualify_refs(e.url, j.value) as definition FROM stac_extensions e, jsonb_each(e.content->'definitions'->'fields'->'properties') j;\n\n\nCREATE OR REPLACE FUNCTION missing_queryables(_collection text, _tablesample float DEFAULT 5, minrows float DEFAULT 10) RETURNS TABLE(collection text, name text, definition jsonb, property_wrapper text) AS $$\nDECLARE\n q text;\n _partition text;\n explain_json json;\n psize float;\n estrows float;\nBEGIN\n SELECT format('_items_%s', key) INTO _partition FROM collections WHERE id=_collection;\n\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM %I;', _partition)\n INTO explain_json;\n psize := explain_json->0->'Plan'->'Plan Rows';\n estrows := _tablesample * .01 * psize;\n IF estrows < minrows THEN\n _tablesample := least(100,greatest(_tablesample, (estrows / psize) / 100));\n RAISE NOTICE '%', (psize / estrows) / 100;\n END IF;\n RAISE NOTICE 'Using tablesample % to find missing queryables from % % that has ~% rows estrows: %', _tablesample, _collection, _partition, psize, estrows;\n\n q := format(\n $q$\n WITH q AS (\n SELECT * FROM queryables\n WHERE\n collection_ids IS NULL\n OR %L = ANY(collection_ids)\n ), t AS (\n SELECT\n content->'properties' AS properties\n FROM\n %I\n TABLESAMPLE SYSTEM(%L)\n ), p AS (\n SELECT DISTINCT ON (key)\n key,\n value,\n s.definition\n FROM t\n JOIN LATERAL jsonb_each(properties) ON TRUE\n LEFT JOIN q ON (q.name=key)\n LEFT JOIN stac_extension_queryables s ON (s.name=key)\n WHERE q.definition IS NULL\n )\n SELECT\n %L,\n key,\n COALESCE(definition, jsonb_build_object('type',jsonb_typeof(value))) as definition,\n CASE\n WHEN definition->>'type' = 'integer' THEN 'to_int'\n WHEN COALESCE(definition->>'type', jsonb_typeof(value)) = 'number' THEN 'to_float'\n WHEN COALESCE(definition->>'type', jsonb_typeof(value)) = 'array' THEN 'to_text_array'\n ELSE 'to_text'\n END\n FROM p;\n $q$,\n _collection,\n _partition,\n _tablesample,\n _collection\n );\n RETURN QUERY EXECUTE q;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION missing_queryables(_tablesample float DEFAULT 5) RETURNS TABLE(collection_ids text[], name text, definition jsonb, property_wrapper text) AS $$\n SELECT\n array_agg(collection),\n name,\n definition,\n property_wrapper\n FROM\n collections\n JOIN LATERAL\n missing_queryables(id, _tablesample) c\n ON TRUE\n GROUP BY\n 2,3,4\n ORDER BY 2,1\n ;\n$$ LANGUAGE SQL;\nCREATE OR REPLACE FUNCTION parse_dtrange(\n _indate jsonb,\n relative_base timestamptz DEFAULT date_trunc('hour', CURRENT_TIMESTAMP)\n) RETURNS tstzrange AS $$\nDECLARE\n timestrs text[];\n s timestamptz;\n e timestamptz;\nBEGIN\n timestrs :=\n CASE\n WHEN _indate ? 'timestamp' THEN\n ARRAY[_indate->>'timestamp']\n WHEN _indate ? 'interval' THEN\n to_text_array(_indate->'interval')\n WHEN jsonb_typeof(_indate) = 'array' THEN\n to_text_array(_indate)\n ELSE\n regexp_split_to_array(\n _indate->>0,\n '/'\n )\n END;\n RAISE NOTICE 'TIMESTRS %', timestrs;\n IF cardinality(timestrs) = 1 THEN\n IF timestrs[1] ILIKE 'P%' THEN\n RETURN tstzrange(relative_base - upper(timestrs[1])::interval, relative_base, '[)');\n END IF;\n s := timestrs[1]::timestamptz;\n RETURN tstzrange(s, s, '[]');\n END IF;\n\n IF cardinality(timestrs) != 2 THEN\n RAISE EXCEPTION 'Timestamp cannot have more than 2 values';\n END IF;\n\n IF timestrs[1] = '..' OR timestrs[1] = '' THEN\n s := '-infinity'::timestamptz;\n e := timestrs[2]::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] = '..' OR timestrs[2] = '' THEN\n s := timestrs[1]::timestamptz;\n e := 'infinity'::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[1] ILIKE 'P%' AND timestrs[2] NOT ILIKE 'P%' THEN\n e := timestrs[2]::timestamptz;\n s := e - upper(timestrs[1])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] ILIKE 'P%' AND timestrs[1] NOT ILIKE 'P%' THEN\n s := timestrs[1]::timestamptz;\n e := s + upper(timestrs[2])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n s := timestrs[1]::timestamptz;\n e := timestrs[2]::timestamptz;\n\n RETURN tstzrange(s,e,'[)');\n\n RETURN NULL;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT PARALLEL SAFE SET TIME ZONE 'UTC';\n\nCREATE OR REPLACE FUNCTION parse_dtrange(\n _indate text,\n relative_base timestamptz DEFAULT CURRENT_TIMESTAMP\n) RETURNS tstzrange AS $$\n SELECT parse_dtrange(to_jsonb(_indate), relative_base);\n$$ LANGUAGE SQL STABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION temporal_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\n ll text := 'datetime';\n lh text := 'end_datetime';\n rrange tstzrange;\n rl text;\n rh text;\n outq text;\nBEGIN\n rrange := parse_dtrange(args->1);\n RAISE NOTICE 'Constructing temporal query OP: %, ARGS: %, RRANGE: %', op, args, rrange;\n op := lower(op);\n rl := format('%L::timestamptz', lower(rrange));\n rh := format('%L::timestamptz', upper(rrange));\n outq := CASE op\n WHEN 't_before' THEN 'lh < rl'\n WHEN 't_after' THEN 'll > rh'\n WHEN 't_meets' THEN 'lh = rl'\n WHEN 't_metby' THEN 'll = rh'\n WHEN 't_overlaps' THEN 'll < rl AND rl < lh < rh'\n WHEN 't_overlappedby' THEN 'rl < ll < rh AND lh > rh'\n WHEN 't_starts' THEN 'll = rl AND lh < rh'\n WHEN 't_startedby' THEN 'll = rl AND lh > rh'\n WHEN 't_during' THEN 'll > rl AND lh < rh'\n WHEN 't_contains' THEN 'll < rl AND lh > rh'\n WHEN 't_finishes' THEN 'll > rl AND lh = rh'\n WHEN 't_finishedby' THEN 'll < rl AND lh = rh'\n WHEN 't_equals' THEN 'll = rl AND lh = rh'\n WHEN 't_disjoint' THEN 'NOT (ll <= rh AND lh >= rl)'\n WHEN 't_intersects' THEN 'll <= rh AND lh >= rl'\n WHEN 'anyinteracts' THEN 'll <= rh AND lh >= rl'\n END;\n outq := regexp_replace(outq, '\\mll\\M', ll);\n outq := regexp_replace(outq, '\\mlh\\M', lh);\n outq := regexp_replace(outq, '\\mrl\\M', rl);\n outq := regexp_replace(outq, '\\mrh\\M', rh);\n outq := format('(%s)', outq);\n RETURN outq;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\n\n\nCREATE OR REPLACE FUNCTION spatial_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\n geom text;\n j jsonb := args->1;\nBEGIN\n op := lower(op);\n RAISE NOTICE 'Constructing spatial query OP: %, ARGS: %', op, args;\n IF op NOT IN ('s_equals','s_disjoint','s_touches','s_within','s_overlaps','s_crosses','s_intersects','intersects','s_contains') THEN\n RAISE EXCEPTION 'Spatial Operator % Not Supported', op;\n END IF;\n op := regexp_replace(op, '^s_', 'st_');\n IF op = 'intersects' THEN\n op := 'st_intersects';\n END IF;\n -- Convert geometry to WKB string\n IF j ? 'type' AND j ? 'coordinates' THEN\n geom := st_geomfromgeojson(j)::text;\n ELSIF jsonb_typeof(j) = 'array' THEN\n geom := bbox_geom(j)::text;\n END IF;\n\n RETURN format('%s(geometry, %L::geometry)', op, geom);\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION query_to_cql2(q jsonb) RETURNS jsonb AS $$\n-- Translates anything passed in through the deprecated \"query\" into equivalent CQL2\nWITH t AS (\n SELECT key as property, value as ops\n FROM jsonb_each(q)\n), t2 AS (\n SELECT property, (jsonb_each(ops)).*\n FROM t WHERE jsonb_typeof(ops) = 'object'\n UNION ALL\n SELECT property, 'eq', ops\n FROM t WHERE jsonb_typeof(ops) != 'object'\n)\nSELECT\n jsonb_strip_nulls(jsonb_build_object(\n 'op', 'and',\n 'args', jsonb_agg(\n jsonb_build_object(\n 'op', key,\n 'args', jsonb_build_array(\n jsonb_build_object('property',property),\n value\n )\n )\n )\n )\n) as qcql FROM t2\n;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\nCREATE OR REPLACE FUNCTION cql1_to_cql2(j jsonb) RETURNS jsonb AS $$\nDECLARE\n args jsonb;\n ret jsonb;\nBEGIN\n RAISE NOTICE 'CQL1_TO_CQL2: %', j;\n IF j ? 'filter' THEN\n RETURN cql1_to_cql2(j->'filter');\n END IF;\n IF j ? 'property' THEN\n RETURN j;\n END IF;\n IF jsonb_typeof(j) = 'array' THEN\n SELECT jsonb_agg(cql1_to_cql2(el)) INTO args FROM jsonb_array_elements(j) el;\n RETURN args;\n END IF;\n IF jsonb_typeof(j) = 'number' THEN\n RETURN j;\n END IF;\n IF jsonb_typeof(j) = 'string' THEN\n RETURN j;\n END IF;\n\n IF jsonb_typeof(j) = 'object' THEN\n SELECT jsonb_build_object(\n 'op', key,\n 'args', cql1_to_cql2(value)\n ) INTO ret\n FROM jsonb_each(j)\n WHERE j IS NOT NULL;\n RETURN ret;\n END IF;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE STRICT;\n\nCREATE TABLE cql2_ops (\n op text PRIMARY KEY,\n template text,\n types text[]\n);\nINSERT INTO cql2_ops (op, template, types) VALUES\n ('eq', '%s = %s', NULL),\n ('neq', '%s != %s', NULL),\n ('ne', '%s != %s', NULL),\n ('!=', '%s != %s', NULL),\n ('<>', '%s != %s', NULL),\n ('lt', '%s < %s', NULL),\n ('lte', '%s <= %s', NULL),\n ('gt', '%s > %s', NULL),\n ('gte', '%s >= %s', NULL),\n ('le', '%s <= %s', NULL),\n ('ge', '%s >= %s', NULL),\n ('=', '%s = %s', NULL),\n ('<', '%s < %s', NULL),\n ('<=', '%s <= %s', NULL),\n ('>', '%s > %s', NULL),\n ('>=', '%s >= %s', NULL),\n ('like', '%s LIKE %s', NULL),\n ('ilike', '%s ILIKE %s', NULL),\n ('+', '%s + %s', NULL),\n ('-', '%s - %s', NULL),\n ('*', '%s * %s', NULL),\n ('/', '%s / %s', NULL),\n ('not', 'NOT (%s)', NULL),\n ('between', '%s BETWEEN %s AND %s', NULL),\n ('isnull', '%s IS NULL', NULL),\n ('upper', 'upper(%s)', NULL),\n ('lower', 'lower(%s)', NULL)\nON CONFLICT (op) DO UPDATE\n SET\n template = EXCLUDED.template\n;\n\n\nCREATE OR REPLACE FUNCTION cql2_query(j jsonb, wrapper text DEFAULT NULL) RETURNS text AS $$\n#variable_conflict use_variable\nDECLARE\n args jsonb := j->'args';\n arg jsonb;\n op text := lower(j->>'op');\n cql2op RECORD;\n literal text;\n _wrapper text;\n leftarg text;\n rightarg text;\n prop text;\n extra_props bool := pgstac.additional_properties();\nBEGIN\n IF j IS NULL OR (op IS NOT NULL AND args IS NULL) THEN\n RETURN NULL;\n END IF;\n RAISE NOTICE 'CQL2_QUERY: %', j;\n\n -- check if all properties are represented in the queryables\n IF NOT extra_props THEN\n FOR prop IN\n SELECT DISTINCT p->>0\n FROM jsonb_path_query(j, 'strict $.**.property') p\n WHERE p->>0 NOT IN ('id', 'datetime', 'geometry', 'end_datetime', 'collection')\n LOOP\n IF (queryable(prop)).nulled_wrapper IS NULL THEN\n RAISE EXCEPTION 'Term % is not found in queryables.', prop;\n END IF;\n END LOOP;\n END IF;\n\n IF j ? 'filter' THEN\n RETURN cql2_query(j->'filter');\n END IF;\n\n IF j ? 'upper' THEN\n RETURN cql2_query(jsonb_build_object('op', 'upper', 'args', j->'upper'));\n END IF;\n\n IF j ? 'lower' THEN\n RETURN cql2_query(jsonb_build_object('op', 'lower', 'args', j->'lower'));\n END IF;\n\n -- Temporal Query\n IF op ilike 't_%' or op = 'anyinteracts' THEN\n RETURN temporal_op_query(op, args);\n END IF;\n\n -- If property is a timestamp convert it to text to use with\n -- general operators\n IF j ? 'timestamp' THEN\n RETURN format('%L::timestamptz', to_tstz(j->'timestamp'));\n END IF;\n IF j ? 'interval' THEN\n RAISE EXCEPTION 'Please use temporal operators when using intervals.';\n RETURN NONE;\n END IF;\n\n -- Spatial Query\n IF op ilike 's_%' or op = 'intersects' THEN\n RETURN spatial_op_query(op, args);\n END IF;\n\n IF op IN ('a_equals','a_contains','a_contained_by','a_overlaps') THEN\n IF args->0 ? 'property' THEN\n leftarg := format('to_text_array(%s)', (queryable(args->0->>'property')).path);\n END IF;\n IF args->1 ? 'property' THEN\n rightarg := format('to_text_array(%s)', (queryable(args->1->>'property')).path);\n END IF;\n RETURN FORMAT(\n '%s %s %s',\n COALESCE(leftarg, quote_literal(to_text_array(args->0))),\n CASE op\n WHEN 'a_equals' THEN '='\n WHEN 'a_contains' THEN '@>'\n WHEN 'a_contained_by' THEN '<@'\n WHEN 'a_overlaps' THEN '&&'\n END,\n COALESCE(rightarg, quote_literal(to_text_array(args->1)))\n );\n END IF;\n\n IF op = 'in' THEN\n RAISE NOTICE 'IN : % % %', args, jsonb_build_array(args->0), args->1;\n args := jsonb_build_array(args->0) || (args->1);\n RAISE NOTICE 'IN2 : %', args;\n END IF;\n\n\n\n IF op = 'between' THEN\n args = jsonb_build_array(\n args->0,\n args->1->0,\n args->1->1\n );\n END IF;\n\n -- Make sure that args is an array and run cql2_query on\n -- each element of the array\n RAISE NOTICE 'ARGS PRE: %', args;\n IF j ? 'args' THEN\n IF jsonb_typeof(args) != 'array' THEN\n args := jsonb_build_array(args);\n END IF;\n\n IF jsonb_path_exists(args, '$[*] ? (@.property == \"id\" || @.property == \"datetime\" || @.property == \"end_datetime\" || @.property == \"collection\")') THEN\n wrapper := NULL;\n ELSE\n -- if any of the arguments are a property, try to get the property_wrapper\n FOR arg IN SELECT jsonb_path_query(args, '$[*] ? (@.property != null)') LOOP\n RAISE NOTICE 'Arg: %', arg;\n wrapper := (queryable(arg->>'property')).nulled_wrapper;\n RAISE NOTICE 'Property: %, Wrapper: %', arg, wrapper;\n IF wrapper IS NOT NULL THEN\n EXIT;\n END IF;\n END LOOP;\n\n -- if the property was not in queryables, see if any args were numbers\n IF\n wrapper IS NULL\n AND jsonb_path_exists(args, '$[*] ? (@.type()==\"number\")')\n THEN\n wrapper := 'to_float';\n END IF;\n wrapper := coalesce(wrapper, 'to_text');\n END IF;\n\n SELECT jsonb_agg(cql2_query(a, wrapper))\n INTO args\n FROM jsonb_array_elements(args) a;\n END IF;\n RAISE NOTICE 'ARGS: %', args;\n\n IF op IN ('and', 'or') THEN\n RETURN\n format(\n '(%s)',\n array_to_string(to_text_array(args), format(' %s ', upper(op)))\n );\n END IF;\n\n IF op = 'in' THEN\n RAISE NOTICE 'IN -- % %', args->0, to_text(args->0);\n RETURN format(\n '%s IN (%s)',\n to_text(args->0),\n array_to_string((to_text_array(args))[2:], ',')\n );\n END IF;\n\n -- Look up template from cql2_ops\n IF j ? 'op' THEN\n SELECT * INTO cql2op FROM cql2_ops WHERE cql2_ops.op ilike op;\n IF FOUND THEN\n -- If specific index set in queryables for a property cast other arguments to that type\n\n RETURN format(\n cql2op.template,\n VARIADIC (to_text_array(args))\n );\n ELSE\n RAISE EXCEPTION 'Operator % Not Supported.', op;\n END IF;\n END IF;\n\n\n IF wrapper IS NOT NULL THEN\n RAISE NOTICE 'Wrapping % with %', j, wrapper;\n IF j ? 'property' THEN\n RETURN format('%I(%s)', wrapper, (queryable(j->>'property')).path);\n ELSE\n RETURN format('%I(%L)', wrapper, j);\n END IF;\n ELSIF j ? 'property' THEN\n RETURN quote_ident(j->>'property');\n END IF;\n\n RETURN quote_literal(to_text(j));\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION paging_dtrange(\n j jsonb\n) RETURNS tstzrange AS $$\nDECLARE\n op text;\n filter jsonb := j->'filter';\n dtrange tstzrange := tstzrange('-infinity'::timestamptz,'infinity'::timestamptz);\n sdate timestamptz := '-infinity'::timestamptz;\n edate timestamptz := 'infinity'::timestamptz;\n jpitem jsonb;\nBEGIN\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n END IF;\n IF NOT (filter @? '$.**.op ? (@ == \"or\" || @ == \"not\")') THEN\n FOR jpitem IN SELECT j FROM jsonb_path_query(filter,'strict $.** ? (@.args[*].property == \"datetime\")'::jsonpath) j LOOP\n op := lower(jpitem->>'op');\n dtrange := parse_dtrange(jpitem->'args'->1);\n IF op IN ('<=', 'lt', 'lte', '<', 'le', 't_before') THEN\n sdate := greatest(sdate,'-infinity');\n edate := least(edate, upper(dtrange));\n ELSIF op IN ('>=', '>', 'gt', 'gte', 'ge', 't_after') THEN\n edate := least(edate, 'infinity');\n sdate := greatest(sdate, lower(dtrange));\n ELSIF op IN ('=', 'eq') THEN\n edate := least(edate, upper(dtrange));\n sdate := greatest(sdate, lower(dtrange));\n END IF;\n RAISE NOTICE '2 OP: %, ARGS: %, DTRANGE: %, SDATE: %, EDATE: %', op, jpitem->'args'->1, dtrange, sdate, edate;\n END LOOP;\n END IF;\n IF sdate > edate THEN\n RETURN 'empty'::tstzrange;\n END IF;\n RETURN tstzrange(sdate,edate, '[]');\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT SET TIME ZONE 'UTC';\n\nCREATE OR REPLACE FUNCTION paging_collections(\n IN j jsonb\n) RETURNS text[] AS $$\nDECLARE\n filter jsonb := j->'filter';\n jpitem jsonb;\n op text;\n args jsonb;\n arg jsonb;\n collections text[];\nBEGIN\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n END IF;\n IF NOT (filter @? '$.**.op ? (@ == \"or\" || @ == \"not\")') THEN\n FOR jpitem IN SELECT j FROM jsonb_path_query(filter,'strict $.** ? (@.args[*].property == \"collection\")'::jsonpath) j LOOP\n RAISE NOTICE 'JPITEM: %', jpitem;\n op := jpitem->>'op';\n args := jpitem->'args';\n IF op IN ('=', 'eq', 'in') THEN\n FOR arg IN SELECT a FROM jsonb_array_elements(args) a LOOP\n IF jsonb_typeof(arg) IN ('string', 'array') THEN\n RAISE NOTICE 'arg: %, collections: %', arg, collections;\n IF collections IS NULL OR collections = '{}'::text[] THEN\n collections := to_text_array(arg);\n ELSE\n collections := array_intersection(collections, to_text_array(arg));\n END IF;\n END IF;\n END LOOP;\n END IF;\n END LOOP;\n END IF;\n IF collections = '{}'::text[] THEN\n RETURN NULL;\n END IF;\n RETURN collections;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\nCREATE TABLE items (\n id text NOT NULL,\n geometry geometry NOT NULL,\n collection text NOT NULL,\n datetime timestamptz NOT NULL,\n end_datetime timestamptz NOT NULL,\n content JSONB NOT NULL,\n private jsonb\n)\nPARTITION BY LIST (collection)\n;\n\nCREATE INDEX \"datetime_idx\" ON items USING BTREE (datetime DESC, end_datetime ASC);\nCREATE INDEX \"geometry_idx\" ON items USING GIST (geometry);\n\nCREATE STATISTICS datetime_stats (dependencies) on datetime, end_datetime from items;\n\nALTER TABLE items ADD CONSTRAINT items_collections_fk FOREIGN KEY (collection) REFERENCES collections(id) ON DELETE CASCADE DEFERRABLE;\n\nCREATE OR REPLACE FUNCTION partition_after_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p text;\n t timestamptz := clock_timestamp();\nBEGIN\n RAISE NOTICE 'Updating partition stats %', t;\n FOR p IN SELECT DISTINCT partition\n FROM newdata n JOIN partition_sys_meta p\n ON (n.collection=p.collection AND n.datetime <@ p.partition_dtrange)\n LOOP\n PERFORM run_or_queue(format('SELECT update_partition_stats(%L, %L);', p, true));\n END LOOP;\n IF TG_OP IN ('DELETE','UPDATE') THEN\n DELETE FROM format_item_cache c USING newdata n WHERE c.collection = n.collection AND c.id = n.id;\n END IF;\n RAISE NOTICE 't: % %', t, clock_timestamp() - t;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE TRIGGER items_after_insert_trigger\nAFTER INSERT ON items\nREFERENCING NEW TABLE AS newdata\nFOR EACH STATEMENT\nEXECUTE FUNCTION partition_after_triggerfunc();\n\nCREATE TRIGGER items_after_update_trigger\nAFTER DELETE ON items\nREFERENCING OLD TABLE AS newdata\nFOR EACH STATEMENT\nEXECUTE FUNCTION partition_after_triggerfunc();\n\nCREATE TRIGGER items_after_delete_trigger\nAFTER UPDATE ON items\nREFERENCING NEW TABLE AS newdata\nFOR EACH STATEMENT\nEXECUTE FUNCTION partition_after_triggerfunc();\n\n\nCREATE OR REPLACE FUNCTION content_slim(_item jsonb) RETURNS jsonb AS $$\n SELECT strip_jsonb(_item - '{id,geometry,collection,type}'::text[], collection_base_item(_item->>'collection')) - '{id,geometry,collection,type}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_dehydrate(content jsonb) RETURNS items AS $$\n SELECT\n content->>'id' as id,\n stac_geom(content) as geometry,\n content->>'collection' as collection,\n stac_datetime(content) as datetime,\n stac_end_datetime(content) as end_datetime,\n content_slim(content) as content,\n null::jsonb as private\n ;\n$$ LANGUAGE SQL STABLE;\n\nCREATE OR REPLACE FUNCTION include_field(f text, fields jsonb DEFAULT '{}'::jsonb) RETURNS boolean AS $$\nDECLARE\n includes jsonb := fields->'include';\n excludes jsonb := fields->'exclude';\nBEGIN\n IF f IS NULL THEN\n RETURN NULL;\n END IF;\n\n\n IF\n jsonb_typeof(excludes) = 'array'\n AND jsonb_array_length(excludes)>0\n AND excludes ? f\n THEN\n RETURN FALSE;\n END IF;\n\n IF\n (\n jsonb_typeof(includes) = 'array'\n AND jsonb_array_length(includes) > 0\n AND includes ? f\n ) OR\n (\n includes IS NULL\n OR jsonb_typeof(includes) = 'null'\n OR jsonb_array_length(includes) = 0\n )\n THEN\n RETURN TRUE;\n END IF;\n\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nDROP FUNCTION IF EXISTS content_hydrate(jsonb, jsonb, jsonb);\nCREATE OR REPLACE FUNCTION content_hydrate(\n _item jsonb,\n _base_item jsonb,\n fields jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\n SELECT merge_jsonb(\n jsonb_fields(_item, fields),\n jsonb_fields(_base_item, fields)\n );\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION content_hydrate(_item items, _collection collections, fields jsonb DEFAULT '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\n content jsonb;\n base_item jsonb := _collection.base_item;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry, 20)::jsonb;\n END IF;\n output := content_hydrate(\n jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'collection', _item.collection,\n 'type', 'Feature'\n ) || _item.content,\n _collection.base_item,\n fields\n );\n\n RETURN output;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_nonhydrated(\n _item items,\n fields jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry, 20)::jsonb;\n END IF;\n output := jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'collection', _item.collection,\n 'type', 'Feature'\n ) || _item.content;\n RETURN output;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_hydrate(_item items, fields jsonb DEFAULT '{}'::jsonb) RETURNS jsonb AS $$\n SELECT content_hydrate(\n _item,\n (SELECT c FROM collections c WHERE id=_item.collection LIMIT 1),\n fields\n );\n$$ LANGUAGE SQL STABLE;\n\n\nCREATE UNLOGGED TABLE items_staging (\n content JSONB NOT NULL\n);\nCREATE UNLOGGED TABLE items_staging_ignore (\n content JSONB NOT NULL\n);\nCREATE UNLOGGED TABLE items_staging_upsert (\n content JSONB NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION items_staging_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p record;\n _partitions text[];\n part text;\n ts timestamptz := clock_timestamp();\n nrows int;\nBEGIN\n RAISE NOTICE 'Creating Partitions. %', clock_timestamp() - ts;\n\n FOR part IN WITH t AS (\n SELECT\n n.content->>'collection' as collection,\n stac_daterange(n.content->'properties') as dtr,\n partition_trunc\n FROM newdata n JOIN collections ON (n.content->>'collection'=collections.id)\n ), p AS (\n SELECT\n collection,\n COALESCE(date_trunc(partition_trunc::text, lower(dtr)),'-infinity') as d,\n tstzrange(min(lower(dtr)),max(lower(dtr)),'[]') as dtrange,\n tstzrange(min(upper(dtr)),max(upper(dtr)),'[]') as edtrange\n FROM t\n GROUP BY 1,2\n ) SELECT check_partition(collection, dtrange, edtrange) FROM p LOOP\n RAISE NOTICE 'Partition %', part;\n END LOOP;\n\n RAISE NOTICE 'Creating temp table with data to be added. %', clock_timestamp() - ts;\n DROP TABLE IF EXISTS tmpdata;\n CREATE TEMP TABLE tmpdata ON COMMIT DROP AS\n SELECT\n (content_dehydrate(content)).*\n FROM newdata;\n GET DIAGNOSTICS nrows = ROW_COUNT;\n RAISE NOTICE 'Added % rows to tmpdata. %', nrows, clock_timestamp() - ts;\n\n RAISE NOTICE 'Doing the insert. %', clock_timestamp() - ts;\n IF TG_TABLE_NAME = 'items_staging' THEN\n INSERT INTO items\n SELECT * FROM tmpdata;\n GET DIAGNOSTICS nrows = ROW_COUNT;\n RAISE NOTICE 'Inserted % rows to items. %', nrows, clock_timestamp() - ts;\n ELSIF TG_TABLE_NAME = 'items_staging_ignore' THEN\n INSERT INTO items\n SELECT * FROM tmpdata\n ON CONFLICT DO NOTHING;\n GET DIAGNOSTICS nrows = ROW_COUNT;\n RAISE NOTICE 'Inserted % rows to items. %', nrows, clock_timestamp() - ts;\n ELSIF TG_TABLE_NAME = 'items_staging_upsert' THEN\n DELETE FROM items i USING tmpdata s\n WHERE\n i.id = s.id\n AND i.collection = s.collection\n AND i IS DISTINCT FROM s\n ;\n GET DIAGNOSTICS nrows = ROW_COUNT;\n RAISE NOTICE 'Deleted % rows from items. %', nrows, clock_timestamp() - ts;\n INSERT INTO items AS t\n SELECT * FROM tmpdata\n ON CONFLICT DO NOTHING;\n GET DIAGNOSTICS nrows = ROW_COUNT;\n RAISE NOTICE 'Inserted % rows to items. %', nrows, clock_timestamp() - ts;\n END IF;\n\n RAISE NOTICE 'Deleting data from staging table. %', clock_timestamp() - ts;\n DELETE FROM items_staging;\n RAISE NOTICE 'Done. %', clock_timestamp() - ts;\n\n RETURN NULL;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER items_staging_insert_trigger AFTER INSERT ON items_staging REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\nCREATE TRIGGER items_staging_insert_ignore_trigger AFTER INSERT ON items_staging_ignore REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\nCREATE TRIGGER items_staging_insert_upsert_trigger AFTER INSERT ON items_staging_upsert REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\n\nCREATE OR REPLACE FUNCTION item_by_id(_id text, _collection text DEFAULT NULL) RETURNS items AS\n$$\nDECLARE\n i items%ROWTYPE;\nBEGIN\n SELECT * INTO i FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection) LIMIT 1;\n RETURN i;\nEND;\n$$ LANGUAGE PLPGSQL STABLE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION get_item(_id text, _collection text DEFAULT NULL) RETURNS jsonb AS $$\n SELECT content_hydrate(items) FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection);\n$$ LANGUAGE SQL STABLE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION delete_item(_id text, _collection text DEFAULT NULL) RETURNS VOID AS $$\nDECLARE\nout items%ROWTYPE;\nBEGIN\n DELETE FROM items WHERE id = _id AND (_collection IS NULL OR collection=_collection) RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n--/*\nCREATE OR REPLACE FUNCTION create_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION update_item(content jsonb) RETURNS VOID AS $$\nDECLARE\n old items %ROWTYPE;\n out items%ROWTYPE;\nBEGIN\n PERFORM delete_item(content->>'id', content->>'collection');\n PERFORM create_item(content);\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION create_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION collection_bbox(id text) RETURNS jsonb AS $$\n SELECT (replace(replace(replace(st_extent(geometry)::text,'BOX(','[['),')',']]'),' ',','))::jsonb\n FROM items WHERE collection=$1;\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION collection_temporal_extent(id text) RETURNS jsonb AS $$\n SELECT to_jsonb(array[array[min(datetime)::text, max(datetime)::text]])\n FROM items WHERE collection=$1;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection_extents() RETURNS VOID AS $$\nUPDATE collections SET\n content = content ||\n jsonb_build_object(\n 'extent', jsonb_build_object(\n 'spatial', jsonb_build_object(\n 'bbox', collection_bbox(collections.id)\n ),\n 'temporal', jsonb_build_object(\n 'interval', collection_temporal_extent(collections.id)\n )\n )\n )\n;\n$$ LANGUAGE SQL;\nCREATE TABLE partition_stats (\n partition text PRIMARY KEY,\n dtrange tstzrange,\n edtrange tstzrange,\n spatial geometry,\n last_updated timestamptz,\n keys text[]\n) WITH (FILLFACTOR=90);\n\nCREATE INDEX partitions_range_idx ON partition_stats USING GIST(dtrange);\n\n\nCREATE OR REPLACE FUNCTION constraint_tstzrange(expr text) RETURNS tstzrange AS $$\n WITH t AS (\n SELECT regexp_matches(\n expr,\n E'\\\\(''\\([0-9 :+-]*\\)''\\\\).*\\\\(''\\([0-9 :+-]*\\)''\\\\)'\n ) AS m\n ) SELECT tstzrange(m[1]::timestamptz, m[2]::timestamptz) FROM t\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE STRICT;\n\nCREATE OR REPLACE FUNCTION dt_constraint(coid oid, OUT dt tstzrange, OUT edt tstzrange) RETURNS RECORD AS $$\nDECLARE\n expr text := pg_get_constraintdef(coid);\n matches timestamptz[];\nBEGIN\n IF expr LIKE '%NULL%' THEN\n dt := tstzrange(null::timestamptz, null::timestamptz);\n edt := tstzrange(null::timestamptz, null::timestamptz);\n RETURN;\n END IF;\n WITH f AS (SELECT (regexp_matches(expr, E'([0-9]{4}-[0-1][0-9]-[0-3][0-9] [0-2][0-9]:[0-5][0-9]:[0-5][0-9]\\.?[0-9]*)', 'g'))[1] f)\n SELECT array_agg(f::timestamptz) INTO matches FROM f;\n IF cardinality(matches) = 4 THEN\n dt := tstzrange(matches[1], matches[2],'[]');\n edt := tstzrange(matches[3], matches[4], '[]');\n RETURN;\n ELSIF cardinality(matches) = 2 THEN\n edt := tstzrange(matches[1], matches[2],'[]');\n RETURN;\n END IF;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\nCREATE OR REPLACE VIEW partition_sys_meta AS\nSELECT\n relid::text as partition,\n replace(replace(CASE WHEN level = 1 THEN pg_get_expr(c.relpartbound, c.oid)\n ELSE pg_get_expr(parent.relpartbound, parent.oid)\n END, 'FOR VALUES IN (''',''), ''')','') AS collection,\n level,\n c.reltuples,\n c.relhastriggers,\n COALESCE(constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as partition_dtrange,\n COALESCE((dt_constraint(edt.oid)).dt, constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as constraint_dtrange,\n COALESCE((dt_constraint(edt.oid)).edt, tstzrange('-infinity', 'infinity','[]')) as constraint_edtrange\nFROM\n pg_partition_tree('items')\n JOIN pg_class c ON (relid::regclass = c.oid)\n JOIN pg_class parent ON (parentrelid::regclass = parent.oid AND isleaf)\n LEFT JOIN pg_constraint edt ON (conrelid=c.oid AND contype='c')\nWHERE isleaf\n;\n\nCREATE VIEW partitions_view AS\nSELECT\n relid::text as partition,\n replace(replace(CASE WHEN level = 1 THEN pg_get_expr(c.relpartbound, c.oid)\n ELSE pg_get_expr(parent.relpartbound, parent.oid)\n END, 'FOR VALUES IN (''',''), ''')','') AS collection,\n level,\n c.reltuples,\n c.relhastriggers,\n COALESCE(constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as partition_dtrange,\n COALESCE((dt_constraint(edt.oid)).dt, constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as constraint_dtrange,\n COALESCE((dt_constraint(edt.oid)).edt, tstzrange('-infinity', 'infinity','[]')) as constraint_edtrange,\n dtrange,\n edtrange,\n spatial,\n last_updated\nFROM\n pg_partition_tree('items')\n JOIN pg_class c ON (relid::regclass = c.oid)\n JOIN pg_class parent ON (parentrelid::regclass = parent.oid AND isleaf)\n LEFT JOIN pg_constraint edt ON (conrelid=c.oid AND contype='c')\n LEFT JOIN partition_stats ON (relid::text=partition)\nWHERE isleaf\n;\n\nCREATE MATERIALIZED VIEW partitions AS\nSELECT * FROM partitions_view;\nCREATE UNIQUE INDEX ON partitions (partition);\n\nCREATE MATERIALIZED VIEW partition_steps AS\nSELECT\n partition as name,\n date_trunc('month',lower(partition_dtrange)) as sdate,\n date_trunc('month', upper(partition_dtrange)) + '1 month'::interval as edate\n FROM partitions_view WHERE partition_dtrange IS NOT NULL AND partition_dtrange != 'empty'::tstzrange\n ORDER BY dtrange ASC\n;\n\n\nCREATE OR REPLACE FUNCTION update_partition_stats_q(_partition text, istrigger boolean default false) RETURNS VOID AS $$\nDECLARE\nBEGIN\n PERFORM run_or_queue(\n format('SELECT update_partition_stats(%L, %L);', _partition, istrigger)\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION update_partition_stats(_partition text, istrigger boolean default false) RETURNS VOID AS $$\nDECLARE\n dtrange tstzrange;\n edtrange tstzrange;\n cdtrange tstzrange;\n cedtrange tstzrange;\n extent geometry;\n collection text;\nBEGIN\n RAISE NOTICE 'Updating stats for %.', _partition;\n EXECUTE format(\n $q$\n SELECT\n tstzrange(min(datetime), max(datetime),'[]'),\n tstzrange(min(end_datetime), max(end_datetime), '[]')\n FROM %I\n $q$,\n _partition\n ) INTO dtrange, edtrange;\n extent := st_estimatedextent('pgstac', _partition, 'geometry');\n INSERT INTO partition_stats (partition, dtrange, edtrange, spatial, last_updated)\n SELECT _partition, dtrange, edtrange, extent, now()\n ON CONFLICT (partition) DO\n UPDATE SET\n dtrange=EXCLUDED.dtrange,\n edtrange=EXCLUDED.edtrange,\n spatial=EXCLUDED.spatial,\n last_updated=EXCLUDED.last_updated\n ;\n\n SELECT\n constraint_dtrange, constraint_edtrange, pv.collection\n INTO cdtrange, cedtrange, collection\n FROM partitions_view pv WHERE partition = _partition;\n REFRESH MATERIALIZED VIEW partitions;\n REFRESH MATERIALIZED VIEW partition_steps;\n\n\n RAISE NOTICE 'Checking if we need to modify constraints...';\n RAISE NOTICE 'cdtrange: % dtrange: % cedtrange: % edtrange: %',cdtrange, dtrange, cedtrange, edtrange;\n IF\n (cdtrange IS DISTINCT FROM dtrange OR edtrange IS DISTINCT FROM cedtrange)\n AND NOT istrigger\n THEN\n RAISE NOTICE 'Modifying Constraints';\n RAISE NOTICE 'Existing % %', cdtrange, cedtrange;\n RAISE NOTICE 'New % %', dtrange, edtrange;\n PERFORM drop_table_constraints(_partition);\n PERFORM create_table_constraints(_partition, dtrange, edtrange);\n REFRESH MATERIALIZED VIEW partitions;\n REFRESH MATERIALIZED VIEW partition_steps;\n END IF;\n RAISE NOTICE 'Checking if we need to update collection extents.';\n IF get_setting_bool('update_collection_extent') THEN\n RAISE NOTICE 'updating collection extent for %', collection;\n PERFORM run_or_queue(format($q$\n UPDATE collections\n SET content = jsonb_set_lax(\n content,\n '{extent}'::text[],\n collection_extent(%L, FALSE),\n true,\n 'use_json_null'\n ) WHERE id=%L\n ;\n $q$, collection, collection));\n ELSE\n RAISE NOTICE 'Not updating collection extent for %', collection;\n END IF;\n\nEND;\n$$ LANGUAGE PLPGSQL STRICT SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION partition_name( IN collection text, IN dt timestamptz, OUT partition_name text, OUT partition_range tstzrange) AS $$\nDECLARE\n c RECORD;\n parent_name text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n parent_name := format('_items_%s', c.key);\n\n\n IF c.partition_trunc = 'year' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYY'));\n ELSIF c.partition_trunc = 'month' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYYMM'));\n ELSE\n partition_name := parent_name;\n partition_range := tstzrange('-infinity'::timestamptz, 'infinity'::timestamptz, '[]');\n END IF;\n IF partition_range IS NULL THEN\n partition_range := tstzrange(\n date_trunc(c.partition_trunc::text, dt),\n date_trunc(c.partition_trunc::text, dt) + concat('1 ', c.partition_trunc)::interval\n );\n END IF;\n RETURN;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION drop_table_constraints(t text) RETURNS text AS $$\nDECLARE\n q text;\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM partitions_view WHERE partition=t) THEN\n RETURN NULL;\n END IF;\n FOR q IN SELECT FORMAT(\n $q$\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n $q$,\n t,\n conname\n ) FROM pg_constraint\n WHERE conrelid=t::regclass::oid AND contype='c'\n LOOP\n EXECUTE q;\n END LOOP;\n RETURN t;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE OR REPLACE FUNCTION create_table_constraints(t text, _dtrange tstzrange, _edtrange tstzrange) RETURNS text AS $$\nDECLARE\n q text;\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM partitions_view WHERE partition=t) THEN\n RETURN NULL;\n END IF;\n RAISE NOTICE 'Creating Table Constraints for % % %', t, _dtrange, _edtrange;\n IF _dtrange = 'empty' AND _edtrange = 'empty' THEN\n q :=format(\n $q$\n DO $block$\n BEGIN\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n ALTER TABLE %I\n ADD CONSTRAINT %I\n CHECK (((datetime IS NULL) AND (end_datetime IS NULL))) NOT VALID\n ;\n ALTER TABLE %I\n VALIDATE CONSTRAINT %I\n ;\n\n\n\n EXCEPTION WHEN others THEN\n RAISE WARNING '%%, Issue Altering Constraints. Please run update_partition_stats(%I)', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n $block$;\n $q$,\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t),\n t\n );\n ELSE\n q :=format(\n $q$\n DO $block$\n BEGIN\n\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n ALTER TABLE %I\n ADD CONSTRAINT %I\n CHECK (\n (datetime >= %L)\n AND (datetime <= %L)\n AND (end_datetime >= %L)\n AND (end_datetime <= %L)\n ) NOT VALID\n ;\n ALTER TABLE %I\n VALIDATE CONSTRAINT %I\n ;\n\n\n\n EXCEPTION WHEN others THEN\n RAISE WARNING '%%, Issue Altering Constraints. Please run update_partition_stats(%I)', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n $block$;\n $q$,\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t),\n lower(_dtrange),\n upper(_dtrange),\n lower(_edtrange),\n upper(_edtrange),\n t,\n format('%s_dt', t),\n t\n );\n END IF;\n PERFORM run_or_queue(q);\n RETURN t;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION check_partition(\n _collection text,\n _dtrange tstzrange,\n _edtrange tstzrange\n) RETURNS text AS $$\nDECLARE\n c RECORD;\n pm RECORD;\n _partition_name text;\n _partition_dtrange tstzrange;\n _constraint_dtrange tstzrange;\n _constraint_edtrange tstzrange;\n q text;\n deferrable_q text;\n err_context text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=_collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', _collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n\n IF c.partition_trunc IS NOT NULL THEN\n _partition_dtrange := tstzrange(\n date_trunc(c.partition_trunc, lower(_dtrange)),\n date_trunc(c.partition_trunc, lower(_dtrange)) + (concat('1 ', c.partition_trunc))::interval,\n '[)'\n );\n ELSE\n _partition_dtrange := '[-infinity, infinity]'::tstzrange;\n END IF;\n\n IF NOT _partition_dtrange @> _dtrange THEN\n RAISE EXCEPTION 'dtrange % is greater than the partition size % for collection %', _dtrange, c.partition_trunc, _collection;\n END IF;\n\n\n IF c.partition_trunc = 'year' THEN\n _partition_name := format('_items_%s_%s', c.key, to_char(lower(_partition_dtrange),'YYYY'));\n ELSIF c.partition_trunc = 'month' THEN\n _partition_name := format('_items_%s_%s', c.key, to_char(lower(_partition_dtrange),'YYYYMM'));\n ELSE\n _partition_name := format('_items_%s', c.key);\n END IF;\n\n SELECT * INTO pm FROM partition_sys_meta WHERE collection=_collection AND partition_dtrange @> _dtrange;\n IF FOUND THEN\n RAISE NOTICE '% % %', _edtrange, _dtrange, pm;\n _constraint_edtrange :=\n tstzrange(\n least(\n lower(_edtrange),\n nullif(lower(pm.constraint_edtrange), '-infinity')\n ),\n greatest(\n upper(_edtrange),\n nullif(upper(pm.constraint_edtrange), 'infinity')\n ),\n '[]'\n );\n _constraint_dtrange :=\n tstzrange(\n least(\n lower(_dtrange),\n nullif(lower(pm.constraint_dtrange), '-infinity')\n ),\n greatest(\n upper(_dtrange),\n nullif(upper(pm.constraint_dtrange), 'infinity')\n ),\n '[]'\n );\n\n IF pm.constraint_edtrange @> _edtrange AND pm.constraint_dtrange @> _dtrange THEN\n RETURN pm.partition;\n ELSE\n PERFORM drop_table_constraints(_partition_name);\n END IF;\n ELSE\n _constraint_edtrange := _edtrange;\n _constraint_dtrange := _dtrange;\n END IF;\n RAISE NOTICE 'EXISTING CONSTRAINTS % %, NEW % %', pm.constraint_dtrange, pm.constraint_edtrange, _constraint_dtrange, _constraint_edtrange;\n RAISE NOTICE 'Creating partition % %', _partition_name, _partition_dtrange;\n IF c.partition_trunc IS NULL THEN\n q := format(\n $q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n GRANT ALL ON %I to pgstac_ingest;\n $q$,\n _partition_name,\n _collection,\n concat(_partition_name,'_pk'),\n _partition_name,\n _partition_name\n );\n ELSE\n q := format(\n $q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L) PARTITION BY RANGE (datetime);\n CREATE TABLE IF NOT EXISTS %I partition OF %I FOR VALUES FROM (%L) TO (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n GRANT ALL ON %I TO pgstac_ingest;\n $q$,\n format('_items_%s', c.key),\n _collection,\n _partition_name,\n format('_items_%s', c.key),\n lower(_partition_dtrange),\n upper(_partition_dtrange),\n format('%s_pk', _partition_name),\n _partition_name,\n _partition_name\n );\n END IF;\n\n BEGIN\n EXECUTE q;\n EXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', _partition_name;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\n END;\n PERFORM maintain_partitions(_partition_name);\n PERFORM update_partition_stats_q(_partition_name, true);\n REFRESH MATERIALIZED VIEW partitions;\n REFRESH MATERIALIZED VIEW partition_steps;\n RETURN _partition_name;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION repartition(_collection text, _partition_trunc text, triggered boolean DEFAULT FALSE) RETURNS text AS $$\nDECLARE\n c RECORD;\n q text;\n from_trunc text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=_collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', _collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n IF triggered THEN\n RAISE NOTICE 'Converting % to % partitioning via Trigger', _collection, _partition_trunc;\n ELSE\n RAISE NOTICE 'Converting % from using % to % partitioning', _collection, c.partition_trunc, _partition_trunc;\n IF c.partition_trunc IS NOT DISTINCT FROM _partition_trunc THEN\n RAISE NOTICE 'Collection % already set to use partition by %', _collection, _partition_trunc;\n RETURN _collection;\n END IF;\n END IF;\n\n IF EXISTS (SELECT 1 FROM partitions_view WHERE collection=_collection LIMIT 1) THEN\n EXECUTE format(\n $q$\n CREATE TEMP TABLE changepartitionstaging ON COMMIT DROP AS SELECT * FROM %I;\n DROP TABLE IF EXISTS %I CASCADE;\n WITH p AS (\n SELECT\n collection,\n CASE\n WHEN %L IS NULL THEN '-infinity'::timestamptz\n ELSE date_trunc(%L, datetime)\n END as d,\n tstzrange(min(datetime),max(datetime),'[]') as dtrange,\n tstzrange(min(end_datetime),max(end_datetime),'[]') as edtrange\n FROM changepartitionstaging\n GROUP BY 1,2\n ) SELECT check_partition(collection, dtrange, edtrange) FROM p;\n INSERT INTO items SELECT * FROM changepartitionstaging;\n DROP TABLE changepartitionstaging;\n $q$,\n concat('_items_', c.key),\n concat('_items_', c.key),\n c.partition_trunc,\n c.partition_trunc\n );\n END IF;\n RETURN _collection;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE OR REPLACE FUNCTION collections_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n q text;\n partition_name text := format('_items_%s', NEW.key);\n partition_exists boolean := false;\n partition_empty boolean := true;\n err_context text;\n loadtemp boolean := FALSE;\nBEGIN\n RAISE NOTICE 'Collection Trigger. % %', NEW.id, NEW.key;\n IF TG_OP = 'UPDATE' AND NEW.partition_trunc IS DISTINCT FROM OLD.partition_trunc THEN\n PERFORM repartition(NEW.id, NEW.partition_trunc, TRUE);\n END IF;\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER collections_trigger AFTER\nINSERT\nOR\nUPDATE ON collections\nFOR EACH ROW EXECUTE FUNCTION collections_trigger_func();\n\nCREATE OR REPLACE FUNCTION chunker(\n IN _where text,\n OUT s timestamptz,\n OUT e timestamptz\n) RETURNS SETOF RECORD AS $$\nDECLARE\n explain jsonb;\nBEGIN\n IF _where IS NULL THEN\n _where := ' TRUE ';\n END IF;\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s;', _where)\n INTO explain;\n\n RETURN QUERY\n WITH t AS (\n SELECT j->>0 as p FROM\n jsonb_path_query(\n explain,\n 'strict $.**.\"Relation Name\" ? (@ != null)'\n ) j\n ),\n parts AS (\n SELECT sdate, edate FROM t JOIN partition_steps ON (t.p = name)\n ),\n times AS (\n SELECT sdate FROM parts\n UNION\n SELECT edate FROM parts\n ),\n uniq AS (\n SELECT DISTINCT sdate FROM times ORDER BY sdate\n ),\n last AS (\n SELECT sdate, lead(sdate, 1) over () as edate FROM uniq\n )\n SELECT sdate, edate FROM last WHERE edate IS NOT NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION partition_queries(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN partitions text[] DEFAULT NULL\n) RETURNS SETOF text AS $$\nDECLARE\n query text;\n sdate timestamptz;\n edate timestamptz;\nBEGIN\nIF _where IS NULL OR trim(_where) = '' THEN\n _where = ' TRUE ';\nEND IF;\nRAISE NOTICE 'Getting chunks for % %', _where, _orderby;\nIF _orderby ILIKE 'datetime d%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 DESC LOOP\n RETURN NEXT format($q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n $q$,\n sdate,\n edate,\n _where,\n _orderby\n );\n END LOOP;\nELSIF _orderby ILIKE 'datetime a%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 ASC LOOP\n RETURN NEXT format($q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n $q$,\n sdate,\n edate,\n _where,\n _orderby\n );\n END LOOP;\nELSE\n query := format($q$\n SELECT * FROM items\n WHERE %s\n ORDER BY %s\n $q$, _where, _orderby\n );\n\n RETURN NEXT query;\n RETURN;\nEND IF;\n\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION partition_query_view(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN _limit int DEFAULT 10\n) RETURNS text AS $$\n WITH p AS (\n SELECT * FROM partition_queries(_where, _orderby) p\n )\n SELECT\n CASE WHEN EXISTS (SELECT 1 FROM p) THEN\n (SELECT format($q$\n SELECT * FROM (\n %s\n ) total LIMIT %s\n $q$,\n string_agg(\n format($q$ SELECT * FROM ( %s ) AS sub $q$, p),\n '\n UNION ALL\n '\n ),\n _limit\n ))\n ELSE NULL\n END FROM p;\n$$ LANGUAGE SQL IMMUTABLE;\n\n\n\n\nCREATE OR REPLACE FUNCTION stac_search_to_where(j jsonb) RETURNS text AS $$\nDECLARE\n where_segments text[];\n _where text;\n dtrange tstzrange;\n collections text[];\n geom geometry;\n sdate timestamptz;\n edate timestamptz;\n filterlang text;\n filter jsonb := j->'filter';\nBEGIN\n IF j ? 'ids' THEN\n where_segments := where_segments || format('id = ANY (%L) ', to_text_array(j->'ids'));\n END IF;\n\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n where_segments := where_segments || format('collection = ANY (%L) ', collections);\n END IF;\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n\n where_segments := where_segments || format(' datetime <= %L::timestamptz AND end_datetime >= %L::timestamptz ',\n edate,\n sdate\n );\n END IF;\n\n geom := stac_geom(j);\n IF geom IS NOT NULL THEN\n where_segments := where_segments || format('st_intersects(geometry, %L)',geom);\n END IF;\n\n filterlang := COALESCE(\n j->>'filter-lang',\n get_setting('default_filter_lang', j->'conf')\n );\n IF NOT filter @? '$.**.op' THEN\n filterlang := 'cql-json';\n END IF;\n\n IF filterlang NOT IN ('cql-json','cql2-json') AND j ? 'filter' THEN\n RAISE EXCEPTION '% is not a supported filter-lang. Please use cql-json or cql2-json.', filterlang;\n END IF;\n\n IF j ? 'query' AND j ? 'filter' THEN\n RAISE EXCEPTION 'Can only use either query or filter at one time.';\n END IF;\n\n IF j ? 'query' THEN\n filter := query_to_cql2(j->'query');\n ELSIF filterlang = 'cql-json' THEN\n filter := cql1_to_cql2(filter);\n END IF;\n RAISE NOTICE 'FILTER: %', filter;\n where_segments := where_segments || cql2_query(filter);\n IF cardinality(where_segments) < 1 THEN\n RETURN ' TRUE ';\n END IF;\n\n _where := array_to_string(array_remove(where_segments, NULL), ' AND ');\n\n IF _where IS NULL OR BTRIM(_where) = '' THEN\n RETURN ' TRUE ';\n END IF;\n RETURN _where;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION parse_sort_dir(_dir text, reverse boolean default false) RETURNS text AS $$\n WITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n ) SELECT\n CASE\n WHEN NOT reverse THEN d\n WHEN d = 'ASC' THEN 'DESC'\n WHEN d = 'DESC' THEN 'ASC'\n END\n FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION sort_dir_to_op(_dir text, prev boolean default false) RETURNS text AS $$\n WITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n ) SELECT\n CASE\n WHEN d = 'ASC' AND prev THEN '<='\n WHEN d = 'DESC' AND prev THEN '>='\n WHEN d = 'ASC' THEN '>='\n WHEN d = 'DESC' THEN '<='\n END\n FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION sort_sqlorderby(\n _search jsonb DEFAULT NULL,\n reverse boolean DEFAULT FALSE\n) RETURNS text AS $$\n WITH sortby AS (\n SELECT coalesce(_search->'sortby','[{\"field\":\"datetime\", \"direction\":\"desc\"}]') as sort\n ), withid AS (\n SELECT CASE\n WHEN sort @? '$[*] ? (@.field == \"id\")' THEN sort\n ELSE sort || '[{\"field\":\"id\", \"direction\":\"desc\"}]'::jsonb\n END as sort\n FROM sortby\n ), withid_rows AS (\n SELECT jsonb_array_elements(sort) as value FROM withid\n ),sorts AS (\n SELECT\n coalesce(\n (queryable(value->>'field')).expression\n ) as key,\n parse_sort_dir(value->>'direction', reverse) as dir\n FROM withid_rows\n )\n SELECT array_to_string(\n array_agg(concat(key, ' ', dir)),\n ', '\n ) FROM sorts;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_sort_dir(sort_item jsonb) RETURNS text AS $$\n SELECT CASE WHEN sort_item->>'direction' ILIKE 'desc%' THEN 'DESC' ELSE 'ASC' END;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION get_token_val_str(\n _field text,\n _item items\n) RETURNS text AS $$\nDECLARE\n q text;\n literal text;\nBEGIN\n q := format($q$ SELECT quote_literal(%s) FROM (SELECT $1.*) as r;$q$, _field);\n EXECUTE q INTO literal USING _item;\n RETURN literal;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n\nCREATE OR REPLACE FUNCTION get_token_record(IN _token text, OUT prev BOOLEAN, OUT item items) RETURNS RECORD AS $$\nDECLARE\n _itemid text := _token;\n _collectionid text;\nBEGIN\n IF _token IS NULL THEN\n RETURN;\n END IF;\n RAISE NOTICE 'Looking for token: %', _token;\n prev := FALSE;\n IF _token ILIKE 'prev:%' THEN\n _itemid := replace(_token, 'prev:','');\n prev := TRUE;\n ELSIF _token ILIKE 'next:%' THEN\n _itemid := replace(_token, 'next:', '');\n END IF;\n SELECT id INTO _collectionid FROM collections WHERE _itemid LIKE concat(id,':%');\n IF FOUND THEN\n _itemid := replace(_itemid, concat(_collectionid,':'), '');\n SELECT * INTO item FROM items WHERE id=_itemid AND collection=_collectionid;\n ELSE\n SELECT * INTO item FROM items WHERE id=_itemid;\n END IF;\n IF item IS NULL THEN\n RAISE EXCEPTION 'Could not find item using token: % item: % collection: %', _token, _itemid, _collectionid;\n END IF;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\n\nCREATE OR REPLACE FUNCTION get_token_filter(\n _sortby jsonb DEFAULT '[{\"field\":\"datetime\",\"direction\":\"desc\"}]'::jsonb,\n token_item items DEFAULT NULL,\n prev boolean DEFAULT FALSE,\n inclusive boolean DEFAULT FALSE\n) RETURNS text AS $$\nDECLARE\n ltop text := '<';\n gtop text := '>';\n dir text;\n sort record;\n orfilter text := '';\n orfilters text[] := '{}'::text[];\n andfilters text[] := '{}'::text[];\n output text;\n token_where text;\nBEGIN\n IF _sortby IS NULL OR _sortby = '[]'::jsonb THEN\n _sortby := '[{\"field\":\"datetime\",\"direction\":\"desc\"}]'::jsonb;\n END IF;\n _sortby := _sortby || jsonb_build_object('field','id','direction',_sortby->0->>'direction');\n RAISE NOTICE 'Getting Token Filter. % %', _sortby, token_item;\n IF inclusive THEN\n orfilters := orfilters || format('( id=%L AND collection=%L )' , token_item.id, token_item.collection);\n END IF;\n\n FOR sort IN\n WITH s1 AS (\n SELECT\n _row,\n (queryable(value->>'field')).expression as _field,\n (value->>'field' = 'id') as _isid,\n get_sort_dir(value) as _dir\n FROM jsonb_array_elements(_sortby)\n WITH ORDINALITY AS t(value, _row)\n )\n SELECT\n _row,\n _field,\n _dir,\n get_token_val_str(_field, token_item) as _val\n FROM s1\n WHERE _row <= (SELECT min(_row) FROM s1 WHERE _isid)\n LOOP\n orfilter := NULL;\n RAISE NOTICE 'SORT: %', sort;\n IF sort._val IS NOT NULL AND ((prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC')) THEN\n orfilter := format($f$(\n (%s %s %s) OR (%s IS NULL)\n )$f$,\n sort._field,\n ltop,\n sort._val,\n sort._val\n );\n ELSIF sort._val IS NULL AND ((prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC')) THEN\n RAISE NOTICE '< but null';\n orfilter := format('%s IS NOT NULL', sort._field);\n ELSIF sort._val IS NULL THEN\n RAISE NOTICE '> but null';\n ELSE\n orfilter := format($f$(\n (%s %s %s) OR (%s IS NULL)\n )$f$,\n sort._field,\n gtop,\n sort._val,\n sort._field\n );\n END IF;\n RAISE NOTICE 'ORFILTER: %', orfilter;\n\n IF orfilter IS NOT NULL THEN\n IF sort._row = 1 THEN\n orfilters := orfilters || orfilter;\n ELSE\n orfilters := orfilters || format('(%s AND %s)', array_to_string(andfilters, ' AND '), orfilter);\n END IF;\n END IF;\n IF sort._val IS NOT NULL THEN\n andfilters := andfilters || format('%s = %s', sort._field, sort._val);\n ELSE\n andfilters := andfilters || format('%s IS NULL', sort._field);\n END IF;\n END LOOP;\n\n output := array_to_string(orfilters, ' OR ');\n\n token_where := concat('(',coalesce(output,'true'),')');\n IF trim(token_where) = '' THEN\n token_where := NULL;\n END IF;\n RAISE NOTICE 'TOKEN_WHERE: %',token_where;\n RETURN token_where;\n END;\n$$ LANGUAGE PLPGSQL SET transform_null_equals TO TRUE\n;\n\nCREATE OR REPLACE FUNCTION search_tohash(jsonb) RETURNS jsonb AS $$\n SELECT $1 - '{token,limit,context,includes,excludes}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION search_hash(jsonb, jsonb) RETURNS text AS $$\n SELECT md5(concat(search_tohash($1)::text,$2::text));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TABLE IF NOT EXISTS searches(\n hash text GENERATED ALWAYS AS (search_hash(search, metadata)) STORED PRIMARY KEY,\n search jsonb NOT NULL,\n _where text,\n orderby text,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n metadata jsonb DEFAULT '{}'::jsonb NOT NULL\n);\n\nCREATE TABLE IF NOT EXISTS search_wheres(\n id bigint generated always as identity primary key,\n _where text NOT NULL,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n statslastupdated timestamptz,\n estimated_count bigint,\n estimated_cost float,\n time_to_estimate float,\n total_count bigint,\n time_to_count float,\n partitions text[]\n);\n\nCREATE INDEX IF NOT EXISTS search_wheres_partitions ON search_wheres USING GIN (partitions);\nCREATE UNIQUE INDEX IF NOT EXISTS search_wheres_where ON search_wheres ((md5(_where)));\n\nCREATE OR REPLACE FUNCTION where_stats(inwhere text, updatestats boolean default false, conf jsonb default null) RETURNS search_wheres AS $$\nDECLARE\n t timestamptz;\n i interval;\n explain_json jsonb;\n partitions text[];\n sw search_wheres%ROWTYPE;\n inwhere_hash text := md5(inwhere);\n _context text := lower(context(conf));\n _stats_ttl interval := context_stats_ttl(conf);\n _estimated_cost float := context_estimated_cost(conf);\n _estimated_count int := context_estimated_count(conf);\n ro bool := pgstac.readonly(conf);\nBEGIN\n IF ro THEN\n updatestats := FALSE;\n END IF;\n\n IF _context = 'off' THEN\n sw._where := inwhere;\n return sw;\n END IF;\n\n SELECT * INTO sw FROM search_wheres WHERE md5(_where)=inwhere_hash FOR UPDATE;\n\n -- Update statistics if explicitly set, if statistics do not exist, or statistics ttl has expired\n IF NOT updatestats THEN\n RAISE NOTICE 'Checking if update is needed for: % .', inwhere;\n RAISE NOTICE 'Stats Last Updated: %', sw.statslastupdated;\n RAISE NOTICE 'TTL: %, Age: %', _stats_ttl, now() - sw.statslastupdated;\n RAISE NOTICE 'Context: %, Existing Total: %', _context, sw.total_count;\n IF\n (\n sw.statslastupdated IS NULL\n OR (now() - sw.statslastupdated) > _stats_ttl\n OR (context(conf) != 'off' AND sw.total_count IS NULL)\n ) AND NOT ro\n THEN\n updatestats := TRUE;\n END IF;\n END IF;\n\n sw._where := inwhere;\n sw.lastused := now();\n sw.usecount := coalesce(sw.usecount,0) + 1;\n\n IF NOT updatestats THEN\n UPDATE search_wheres SET\n lastused = sw.lastused,\n usecount = sw.usecount\n WHERE md5(_where) = inwhere_hash\n RETURNING * INTO sw\n ;\n RETURN sw;\n END IF;\n\n -- Use explain to get estimated count/cost and a list of the partitions that would be hit by the query\n t := clock_timestamp();\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s', inwhere)\n INTO explain_json;\n RAISE NOTICE 'Time for just the explain: %', clock_timestamp() - t;\n i := clock_timestamp() - t;\n\n sw.statslastupdated := now();\n sw.estimated_count := explain_json->0->'Plan'->'Plan Rows';\n sw.estimated_cost := explain_json->0->'Plan'->'Total Cost';\n sw.time_to_estimate := extract(epoch from i);\n\n RAISE NOTICE 'ESTIMATED_COUNT: % < %', sw.estimated_count, _estimated_count;\n RAISE NOTICE 'ESTIMATED_COST: % < %', sw.estimated_cost, _estimated_cost;\n\n -- Do a full count of rows if context is set to on or if auto is set and estimates are low enough\n IF\n _context = 'on'\n OR\n ( _context = 'auto' AND\n (\n sw.estimated_count < _estimated_count\n AND\n sw.estimated_cost < _estimated_cost\n )\n )\n THEN\n t := clock_timestamp();\n RAISE NOTICE 'Calculating actual count...';\n EXECUTE format(\n 'SELECT count(*) FROM items WHERE %s',\n inwhere\n ) INTO sw.total_count;\n i := clock_timestamp() - t;\n RAISE NOTICE 'Actual Count: % -- %', sw.total_count, i;\n sw.time_to_count := extract(epoch FROM i);\n ELSE\n sw.total_count := NULL;\n sw.time_to_count := NULL;\n END IF;\n\n IF NOT ro THEN\n INSERT INTO search_wheres\n (_where, lastused, usecount, statslastupdated, estimated_count, estimated_cost, time_to_estimate, partitions, total_count, time_to_count)\n SELECT sw._where, sw.lastused, sw.usecount, sw.statslastupdated, sw.estimated_count, sw.estimated_cost, sw.time_to_estimate, sw.partitions, sw.total_count, sw.time_to_count\n ON CONFLICT ((md5(_where)))\n DO UPDATE\n SET\n lastused = sw.lastused,\n usecount = sw.usecount,\n statslastupdated = sw.statslastupdated,\n estimated_count = sw.estimated_count,\n estimated_cost = sw.estimated_cost,\n time_to_estimate = sw.time_to_estimate,\n total_count = sw.total_count,\n time_to_count = sw.time_to_count\n ;\n END IF;\n RETURN sw;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION search_query(\n _search jsonb = '{}'::jsonb,\n updatestats boolean = false,\n _metadata jsonb = '{}'::jsonb\n) RETURNS searches AS $$\nDECLARE\n search searches%ROWTYPE;\n pexplain jsonb;\n t timestamptz;\n i interval;\n _hash text := search_hash(_search, _metadata);\n doupdate boolean := FALSE;\n insertfound boolean := FALSE;\n ro boolean := pgstac.readonly();\nBEGIN\n IF ro THEN\n updatestats := FALSE;\n END IF;\n\n SELECT * INTO search FROM searches\n WHERE hash=_hash;\n\n search.hash := _hash;\n\n -- Calculate the where clause if not already calculated\n IF search._where IS NULL THEN\n search._where := stac_search_to_where(_search);\n ELSE\n doupdate := TRUE;\n END IF;\n\n -- Calculate the order by clause if not already calculated\n IF search.orderby IS NULL THEN\n search.orderby := sort_sqlorderby(_search);\n ELSE\n doupdate := TRUE;\n END IF;\n\n PERFORM where_stats(search._where, updatestats, _search->'conf');\n\n IF NOT ro THEN\n IF NOT doupdate THEN\n INSERT INTO searches (search, _where, orderby, lastused, usecount, metadata)\n VALUES (_search, search._where, search.orderby, clock_timestamp(), 1, _metadata)\n ON CONFLICT (hash) DO NOTHING RETURNING * INTO search;\n IF FOUND THEN\n RETURN search;\n END IF;\n END IF;\n\n UPDATE searches\n SET\n lastused=clock_timestamp(),\n usecount=usecount+1\n WHERE hash=(\n SELECT hash FROM searches\n WHERE hash=_hash\n FOR UPDATE SKIP LOCKED\n );\n IF NOT FOUND THEN\n RAISE NOTICE 'Did not update stats for % due to lock. (This is generally OK)', _search;\n END IF;\n END IF;\n\n RETURN search;\n\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE OR REPLACE FUNCTION search_rows(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN partitions text[] DEFAULT NULL,\n IN _limit int DEFAULT 10\n) RETURNS SETOF items AS $$\nDECLARE\n base_query text;\n query text;\n sdate timestamptz;\n edate timestamptz;\n n int;\n records_left int := _limit;\n timer timestamptz := clock_timestamp();\n full_timer timestamptz := clock_timestamp();\nBEGIN\nIF _where IS NULL OR trim(_where) = '' THEN\n _where = ' TRUE ';\nEND IF;\nRAISE NOTICE 'Getting chunks for % %', _where, _orderby;\n\nbase_query := $q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n LIMIT %L\n$q$;\n\nIF _orderby ILIKE 'datetime d%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 DESC LOOP\n RAISE NOTICE 'Running Query for % to %. %', sdate, edate, age_ms(full_timer);\n query := format(\n base_query,\n sdate,\n edate,\n _where,\n _orderby,\n records_left\n );\n RAISE LOG 'QUERY: %', query;\n timer := clock_timestamp();\n RETURN QUERY EXECUTE query;\n\n GET DIAGNOSTICS n = ROW_COUNT;\n records_left := records_left - n;\n RAISE NOTICE 'Returned %/% Rows From % to %. % to go. Time: %ms', n, _limit, sdate, edate, records_left, age_ms(timer);\n timer := clock_timestamp();\n IF records_left <= 0 THEN\n RAISE NOTICE 'SEARCH_ROWS TOOK %ms', age_ms(full_timer);\n RETURN;\n END IF;\n END LOOP;\nELSIF _orderby ILIKE 'datetime a%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 ASC LOOP\n RAISE NOTICE 'Running Query for % to %. %', sdate, edate, age_ms(full_timer);\n query := format(\n base_query,\n sdate,\n edate,\n _where,\n _orderby,\n records_left\n );\n RAISE LOG 'QUERY: %', query;\n timer := clock_timestamp();\n RETURN QUERY EXECUTE query;\n\n GET DIAGNOSTICS n = ROW_COUNT;\n records_left := records_left - n;\n RAISE NOTICE 'Returned %/% Rows From % to %. % to go. Time: %ms', n, _limit, sdate, edate, records_left, age_ms(timer);\n timer := clock_timestamp();\n IF records_left <= 0 THEN\n RAISE NOTICE 'SEARCH_ROWS TOOK %ms', age_ms(full_timer);\n RETURN;\n END IF;\n END LOOP;\nELSE\n query := format($q$\n SELECT * FROM items\n WHERE %s\n ORDER BY %s\n LIMIT %L\n $q$, _where, _orderby, _limit\n );\n RAISE LOG 'QUERY: %', query;\n timer := clock_timestamp();\n RETURN QUERY EXECUTE query;\n RAISE NOTICE 'FULL QUERY TOOK %ms', age_ms(timer);\nEND IF;\nRAISE NOTICE 'SEARCH_ROWS TOOK %ms', age_ms(full_timer);\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE UNLOGGED TABLE format_item_cache(\n id text,\n collection text,\n fields text,\n hydrated bool,\n output jsonb,\n lastused timestamptz DEFAULT now(),\n usecount int DEFAULT 1,\n timetoformat float,\n PRIMARY KEY (collection, id, fields, hydrated)\n);\nCREATE INDEX ON format_item_cache (lastused);\n\nCREATE OR REPLACE FUNCTION format_item(_item items, _fields jsonb DEFAULT '{}', _hydrated bool DEFAULT TRUE) RETURNS jsonb AS $$\nDECLARE\n cache bool := get_setting_bool('format_cache');\n _output jsonb := null;\n t timestamptz := clock_timestamp();\nBEGIN\n IF cache THEN\n SELECT output INTO _output FROM format_item_cache\n WHERE id=_item.id AND collection=_item.collection AND fields=_fields::text AND hydrated=_hydrated;\n END IF;\n IF _output IS NULL THEN\n IF _hydrated THEN\n _output := content_hydrate(_item, _fields);\n ELSE\n _output := content_nonhydrated(_item, _fields);\n END IF;\n END IF;\n IF cache THEN\n INSERT INTO format_item_cache (id, collection, fields, hydrated, output, timetoformat)\n VALUES (_item.id, _item.collection, _fields::text, _hydrated, _output, age_ms(t))\n ON CONFLICT(collection, id, fields, hydrated) DO\n UPDATE\n SET lastused=now(), usecount = format_item_cache.usecount + 1\n ;\n END IF;\n RETURN _output;\n\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION search(_search jsonb = '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n searches searches%ROWTYPE;\n _where text;\n orderby text;\n search_where search_wheres%ROWTYPE;\n total_count bigint;\n token record;\n token_prev boolean;\n token_item items%ROWTYPE;\n token_where text;\n full_where text;\n init_ts timestamptz := clock_timestamp();\n timer timestamptz := clock_timestamp();\n hydrate bool := NOT (_search->'conf'->>'nohydrate' IS NOT NULL AND (_search->'conf'->>'nohydrate')::boolean = true);\n prev text;\n next text;\n context jsonb;\n collection jsonb;\n out_records jsonb;\n out_len int;\n _limit int := coalesce((_search->>'limit')::int, 10);\n _querylimit int;\n _fields jsonb := coalesce(_search->'fields', '{}'::jsonb);\n has_prev boolean := FALSE;\n has_next boolean := FALSE;\nBEGIN\n searches := search_query(_search);\n _where := searches._where;\n orderby := searches.orderby;\n search_where := where_stats(_where);\n total_count := coalesce(search_where.total_count, search_where.estimated_count);\n RAISE NOTICE 'SEARCH:TOKEN: %', _search->>'token';\n token := get_token_record(_search->>'token');\n RAISE NOTICE '***TOKEN: %', token;\n _querylimit := _limit + 1;\n IF token IS NOT NULL THEN\n token_prev := token.prev;\n token_item := token.item;\n token_where := get_token_filter(_search->'sortby', token_item, token_prev, FALSE);\n RAISE LOG 'TOKEN_WHERE: % (%ms from search start)', token_where, age_ms(timer);\n IF token_prev THEN -- if we are using a prev token, we know has_next is true\n RAISE LOG 'There is a previous token, so automatically setting has_next to true';\n has_next := TRUE;\n orderby := sort_sqlorderby(_search, TRUE);\n ELSE\n RAISE LOG 'There is a next token, so automatically setting has_prev to true';\n has_prev := TRUE;\n\n END IF;\n ELSE -- if there was no token, we know there is no prev\n RAISE LOG 'There is no token, so we know there is no prev. setting has_prev to false';\n has_prev := FALSE;\n END IF;\n\n full_where := concat_ws(' AND ', _where, token_where);\n RAISE NOTICE 'FULL WHERE CLAUSE: %', full_where;\n RAISE NOTICE 'Time to get counts and build query %', age_ms(timer);\n timer := clock_timestamp();\n\n IF hydrate THEN\n RAISE NOTICE 'Getting hydrated data.';\n ELSE\n RAISE NOTICE 'Getting non-hydrated data.';\n END IF;\n RAISE NOTICE 'CACHE SET TO %', get_setting_bool('format_cache');\n RAISE NOTICE 'Time to set hydration/formatting %', age_ms(timer);\n timer := clock_timestamp();\n SELECT jsonb_agg(format_item(i, _fields, hydrate)) INTO out_records\n FROM search_rows(\n full_where,\n orderby,\n search_where.partitions,\n _querylimit\n ) as i;\n\n RAISE NOTICE 'Time to fetch rows %', age_ms(timer);\n timer := clock_timestamp();\n\n\n IF token_prev THEN\n out_records := flip_jsonb_array(out_records);\n END IF;\n\n RAISE NOTICE 'Query returned % records.', jsonb_array_length(out_records);\n RAISE LOG 'TOKEN: % %', token_item.id, token_item.collection;\n RAISE LOG 'RECORD_1: % %', out_records->0->>'id', out_records->0->>'collection';\n RAISE LOG 'RECORD-1: % %', out_records->-1->>'id', out_records->-1->>'collection';\n\n -- REMOVE records that were from our token\n IF out_records->0->>'id' = token_item.id AND out_records->0->>'collection' = token_item.collection THEN\n out_records := out_records - 0;\n ELSIF out_records->-1->>'id' = token_item.id AND out_records->-1->>'collection' = token_item.collection THEN\n out_records := out_records - -1;\n END IF;\n\n out_len := jsonb_array_length(out_records);\n\n IF out_len = _limit + 1 THEN\n IF token_prev THEN\n has_prev := TRUE;\n out_records := out_records - 0;\n ELSE\n has_next := TRUE;\n out_records := out_records - -1;\n END IF;\n END IF;\n\n IF has_next THEN\n next := concat(out_records->-1->>'collection', ':', out_records->-1->>'id');\n RAISE NOTICE 'HAS NEXT | %', next;\n END IF;\n\n IF has_prev THEN\n prev := concat(out_records->0->>'collection', ':', out_records->0->>'id');\n RAISE NOTICE 'HAS PREV | %', prev;\n END IF;\n\n RAISE NOTICE 'Time to get prev/next %', age_ms(timer);\n timer := clock_timestamp();\n\n IF context(_search->'conf') != 'off' THEN\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'matched', total_count,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\n ELSE\n context := jsonb_strip_nulls(jsonb_build_object(\n 'limit', _limit,\n 'returned', coalesce(jsonb_array_length(out_records), 0)\n ));\n END IF;\n\n collection := jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb),\n 'next', next,\n 'prev', prev,\n 'context', context\n );\n\n IF get_setting_bool('timing', _search->'conf') THEN\n collection = collection || jsonb_build_object('timing', age_ms(init_ts));\n END IF;\n\n RAISE NOTICE 'Time to build final json %', age_ms(timer);\n timer := clock_timestamp();\n\n RAISE NOTICE 'Total Time: %', age_ms(current_timestamp);\n RAISE NOTICE 'RETURNING % records. NEXT: %. PREV: %', collection->'context'->>'returned', collection->>'next', collection->>'prev';\n RETURN collection;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION search_cursor(_search jsonb = '{}'::jsonb) RETURNS refcursor AS $$\nDECLARE\n curs refcursor;\n searches searches%ROWTYPE;\n _where text;\n _orderby text;\n q text;\n\nBEGIN\n searches := search_query(_search);\n _where := searches._where;\n _orderby := searches.orderby;\n\n OPEN curs FOR\n WITH p AS (\n SELECT * FROM partition_queries(_where, _orderby) p\n )\n SELECT\n CASE WHEN EXISTS (SELECT 1 FROM p) THEN\n (SELECT format($q$\n SELECT * FROM (\n %s\n ) total\n $q$,\n string_agg(\n format($q$ SELECT * FROM ( %s ) AS sub $q$, p),\n '\n UNION ALL\n '\n )\n ))\n ELSE NULL\n END FROM p;\n RETURN curs;\nEND;\n$$ LANGUAGE PLPGSQL;\nCREATE OR REPLACE VIEW collections_asitems AS\nSELECT\n id,\n geometry,\n 'collections' AS collection,\n datetime,\n end_datetime,\n jsonb_build_object(\n 'properties', content - '{links,assets,stac_version,stac_extensions}',\n 'links', content->'links',\n 'assets', content->'assets',\n 'stac_version', content->'stac_version',\n 'stac_extensions', content->'stac_extensions'\n ) AS content,\n content as collectionjson\nFROM collections;\n\n\nCREATE OR REPLACE FUNCTION collection_search_matched(\n IN _search jsonb DEFAULT '{}'::jsonb,\n OUT matched bigint\n) RETURNS bigint AS $$\nDECLARE\n _where text := stac_search_to_where(_search);\nBEGIN\n EXECUTE format(\n $query$\n SELECT\n count(*)\n FROM\n collections_asitems\n WHERE %s\n ;\n $query$,\n _where\n ) INTO matched;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION collection_search_rows(\n _search jsonb DEFAULT '{}'::jsonb\n) RETURNS SETOF jsonb AS $$\nDECLARE\n _where text := stac_search_to_where(_search);\n _limit int := coalesce((_search->>'limit')::int, 10);\n _fields jsonb := coalesce(_search->'fields', '{}'::jsonb);\n _orderby text;\n _offset int := COALESCE((_search->>'offset')::int, 0);\nBEGIN\n _orderby := sort_sqlorderby(\n jsonb_build_object(\n 'sortby',\n coalesce(\n _search->'sortby',\n '[{\"field\": \"id\", \"direction\": \"asc\"}]'::jsonb\n )\n )\n );\n RETURN QUERY EXECUTE format(\n $query$\n SELECT\n jsonb_fields(collectionjson, %L) as c\n FROM\n collections_asitems\n WHERE %s\n ORDER BY %s\n LIMIT %L\n OFFSET %L\n ;\n $query$,\n _fields,\n _where,\n _orderby,\n _limit,\n _offset\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION collection_search(\n _search jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\nDECLARE\n out_records jsonb;\n number_matched bigint := collection_search_matched(_search);\n number_returned bigint;\n _limit int := coalesce((_search->>'limit')::float::int, 10);\n _offset int := coalesce((_search->>'offset')::float::int, 0);\n links jsonb := '[]';\n ret jsonb;\n base_url text:= concat(rtrim(base_url(_search->'conf'),'/'), '/collections');\n prevoffset int;\n nextoffset int;\nBEGIN\n SELECT\n coalesce(jsonb_agg(c), '[]')\n INTO out_records\n FROM collection_search_rows(_search) c;\n\n number_returned := jsonb_array_length(out_records);\n\n IF _limit <= number_matched THEN --need to have paging links\n nextoffset := least(_offset + _limit, number_matched - 1);\n prevoffset := greatest(_offset - _limit, 0);\n IF _offset = 0 THEN -- no previous paging\n\n links := jsonb_build_array(\n jsonb_build_object(\n 'rel', 'next',\n 'type', 'application/json',\n 'method', 'GET' ,\n 'href', base_url,\n 'body', jsonb_build_object('offset', nextoffset),\n 'merge', TRUE\n )\n );\n ELSE\n links := jsonb_build_array(\n jsonb_build_object(\n 'rel', 'prev',\n 'type', 'application/json',\n 'method', 'GET' ,\n 'href', base_url,\n 'body', jsonb_build_object('offset', prevoffset),\n 'merge', TRUE\n ),\n jsonb_build_object(\n 'rel', 'next',\n 'type', 'application/json',\n 'method', 'GET' ,\n 'href', base_url,\n 'body', jsonb_build_object('offset', nextoffset),\n 'merge', TRUE\n )\n );\n END IF;\n END IF;\n\n ret := jsonb_build_object(\n 'collections', out_records,\n 'context', jsonb_build_object(\n 'limit', _limit,\n 'matched', number_matched,\n 'returned', number_returned\n ),\n 'links', links\n );\n RETURN ret;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION tileenvelope(zoom int, x int, y int) RETURNS geometry AS $$\nWITH t AS (\n SELECT\n 20037508.3427892 as merc_max,\n -20037508.3427892 as merc_min,\n (2 * 20037508.3427892) / (2 ^ zoom) as tile_size\n)\nSELECT st_makeenvelope(\n merc_min + (tile_size * x),\n merc_max - (tile_size * (y + 1)),\n merc_min + (tile_size * (x + 1)),\n merc_max - (tile_size * y),\n 3857\n) FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;DROP FUNCTION IF EXISTS mercgrid;\n\n\nCREATE OR REPLACE FUNCTION ftime() RETURNS interval as $$\nSELECT age(clock_timestamp(), transaction_timestamp());\n$$ LANGUAGE SQL;\nSET SEARCH_PATH to pgstac, public;\n\nDROP FUNCTION IF EXISTS geometrysearch;\nCREATE OR REPLACE FUNCTION geometrysearch(\n IN geom geometry,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE, -- Return as soon as the passed in geometry is full covered\n IN skipcovered boolean DEFAULT TRUE -- Skip any items that would show up completely under the previous items\n) RETURNS jsonb AS $$\nDECLARE\n search searches%ROWTYPE;\n curs refcursor;\n _where text;\n query text;\n iter_record items%ROWTYPE;\n out_records jsonb := '{}'::jsonb[];\n exit_flag boolean := FALSE;\n counter int := 1;\n scancounter int := 1;\n remaining_limit int := _scanlimit;\n tilearea float;\n unionedgeom geometry;\n clippedgeom geometry;\n unionedgeom_area float := 0;\n prev_area float := 0;\n excludes text[];\n includes text[];\n\nBEGIN\n DROP TABLE IF EXISTS pgstac_results;\n CREATE TEMP TABLE pgstac_results (content jsonb) ON COMMIT DROP;\n\n -- If the passed in geometry is not an area set exitwhenfull and skipcovered to false\n IF ST_GeometryType(geom) !~* 'polygon' THEN\n RAISE NOTICE 'GEOMETRY IS NOT AN AREA';\n skipcovered = FALSE;\n exitwhenfull = FALSE;\n END IF;\n\n -- If skipcovered is true then you will always want to exit when the passed in geometry is full\n IF skipcovered THEN\n exitwhenfull := TRUE;\n END IF;\n\n SELECT * INTO search FROM searches WHERE hash=queryhash;\n\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Search with Query Hash % Not Found', queryhash;\n END IF;\n\n tilearea := st_area(geom);\n _where := format('%s AND st_intersects(geometry, %L::geometry)', search._where, geom);\n\n\n FOR query IN SELECT * FROM partition_queries(_where, search.orderby) LOOP\n query := format('%s LIMIT %L', query, remaining_limit);\n RAISE NOTICE '%', query;\n OPEN curs FOR EXECUTE query;\n LOOP\n FETCH curs INTO iter_record;\n EXIT WHEN NOT FOUND;\n IF exitwhenfull OR skipcovered THEN -- If we are not using exitwhenfull or skipcovered, we do not need to do expensive geometry operations\n clippedgeom := st_intersection(geom, iter_record.geometry);\n\n IF unionedgeom IS NULL THEN\n unionedgeom := clippedgeom;\n ELSE\n unionedgeom := st_union(unionedgeom, clippedgeom);\n END IF;\n\n unionedgeom_area := st_area(unionedgeom);\n\n IF skipcovered AND prev_area = unionedgeom_area THEN\n scancounter := scancounter + 1;\n CONTINUE;\n END IF;\n\n prev_area := unionedgeom_area;\n\n RAISE NOTICE '% % % %', unionedgeom_area/tilearea, counter, scancounter, ftime();\n END IF;\n RAISE NOTICE '% %', iter_record, content_hydrate(iter_record, fields);\n INSERT INTO pgstac_results (content) VALUES (content_hydrate(iter_record, fields));\n\n IF counter >= _limit\n OR scancounter > _scanlimit\n OR ftime() > _timelimit\n OR (exitwhenfull AND unionedgeom_area >= tilearea)\n THEN\n exit_flag := TRUE;\n EXIT;\n END IF;\n counter := counter + 1;\n scancounter := scancounter + 1;\n\n END LOOP;\n CLOSE curs;\n EXIT WHEN exit_flag;\n remaining_limit := _scanlimit - scancounter;\n END LOOP;\n\n SELECT jsonb_agg(content) INTO out_records FROM pgstac_results WHERE content IS NOT NULL;\n\n RETURN jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb)\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDROP FUNCTION IF EXISTS geojsonsearch;\nCREATE OR REPLACE FUNCTION geojsonsearch(\n IN geojson jsonb,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_geomfromgeojson(geojson),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS xyzsearch;\nCREATE OR REPLACE FUNCTION xyzsearch(\n IN _x int,\n IN _y int,\n IN _z int,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_transform(tileenvelope(_z, _x, _y), 4326),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE PROCEDURE analyze_items() AS $$\nDECLARE\n q text;\n timeout_ts timestamptz;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n SELECT format('ANALYZE (VERBOSE, SKIP_LOCKED) %I;', relname) INTO q\n FROM pg_stat_user_tables\n WHERE relname like '_item%' AND (n_mod_since_analyze>0 OR last_analyze IS NULL) LIMIT 1;\n IF NOT FOUND THEN\n EXIT;\n END IF;\n RAISE NOTICE '%', q;\n EXECUTE q;\n COMMIT;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE PROCEDURE validate_constraints() AS $$\nDECLARE\n q text;\nBEGIN\n FOR q IN\n SELECT\n FORMAT(\n 'ALTER TABLE %I.%I VALIDATE CONSTRAINT %I;',\n nsp.nspname,\n cls.relname,\n con.conname\n )\n\n FROM pg_constraint AS con\n JOIN pg_class AS cls\n ON con.conrelid = cls.oid\n JOIN pg_namespace AS nsp\n ON cls.relnamespace = nsp.oid\n WHERE convalidated = FALSE AND contype in ('c','f')\n AND nsp.nspname = 'pgstac'\n LOOP\n RAISE NOTICE '%', q;\n PERFORM run_or_queue(q);\n COMMIT;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION collection_extent(_collection text, runupdate boolean default false) RETURNS jsonb AS $$\nDECLARE\n geom_extent geometry;\n mind timestamptz;\n maxd timestamptz;\n extent jsonb;\nBEGIN\n IF runupdate THEN\n PERFORM update_partition_stats_q(partition)\n FROM partitions_view WHERE collection=_collection;\n END IF;\n SELECT\n min(lower(dtrange)),\n max(upper(edtrange)),\n st_extent(spatial)\n INTO\n mind,\n maxd,\n geom_extent\n FROM partitions_view\n WHERE collection=_collection;\n\n IF geom_extent IS NOT NULL AND mind IS NOT NULL AND maxd IS NOT NULL THEN\n extent := jsonb_build_object(\n 'extent', jsonb_build_object(\n 'spatial', jsonb_build_object(\n 'bbox', to_jsonb(array[array[st_xmin(geom_extent), st_ymin(geom_extent), st_xmax(geom_extent), st_ymax(geom_extent)]])\n ),\n 'temporal', jsonb_build_object(\n 'interval', to_jsonb(array[array[mind, maxd]])\n )\n )\n );\n RETURN extent;\n END IF;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('id', '{\"title\": \"Item ID\",\"description\": \"Item identifier\",\"$ref\": \"https://schemas.stacspec.org/v1.0.0/item-spec/json-schema/item.json#/definitions/core/allOf/2/properties/id\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('geometry', '{\"title\": \"Item Geometry\",\"description\": \"Item Geometry\",\"$ref\": \"https://geojson.org/schema/Feature.json\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('datetime','{\"description\": \"Datetime\",\"type\": \"string\",\"title\": \"Acquired\",\"format\": \"date-time\",\"pattern\": \"(\\\\+00:00|Z)$\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDELETE FROM queryables a USING queryables b\n WHERE a.name = b.name AND a.collection_ids IS NOT DISTINCT FROM b.collection_ids AND a.id > b.id;\n\n\nINSERT INTO pgstac_settings (name, value) VALUES\n ('context', 'off'),\n ('context_estimated_count', '100000'),\n ('context_estimated_cost', '100000'),\n ('context_stats_ttl', '1 day'),\n ('default_filter_lang', 'cql2-json'),\n ('additional_properties', 'true'),\n ('use_queue', 'false'),\n ('queue_timeout', '10 minutes'),\n ('update_collection_extent', 'false'),\n ('format_cache', 'false'),\n ('readonly', 'false')\nON CONFLICT DO NOTHING\n;\n\nALTER FUNCTION to_text COST 5000;\nALTER FUNCTION to_float COST 5000;\nALTER FUNCTION to_int COST 5000;\nALTER FUNCTION to_tstz COST 5000;\nALTER FUNCTION to_text_array COST 5000;\n\nALTER FUNCTION update_partition_stats SECURITY DEFINER;\nALTER FUNCTION partition_after_triggerfunc SECURITY DEFINER;\nALTER FUNCTION drop_table_constraints SECURITY DEFINER;\nALTER FUNCTION create_table_constraints SECURITY DEFINER;\nALTER FUNCTION check_partition SECURITY DEFINER;\nALTER FUNCTION repartition SECURITY DEFINER;\nALTER FUNCTION where_stats SECURITY DEFINER;\nALTER FUNCTION search_query SECURITY DEFINER;\nALTER FUNCTION format_item SECURITY DEFINER;\nALTER FUNCTION maintain_index SECURITY DEFINER;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nGRANT ALL ON SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac to pgstac_admin;\n\n-- pgstac_read role limited to using function apis\nGRANT EXECUTE ON FUNCTION search TO pgstac_read;\nGRANT EXECUTE ON FUNCTION search_query TO pgstac_read;\nGRANT EXECUTE ON FUNCTION item_by_id TO pgstac_read;\nGRANT EXECUTE ON FUNCTION get_item TO pgstac_read;\nGRANT SELECT ON ALL TABLES IN SCHEMA pgstac TO pgstac_read;\n\n\nGRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_ingest;\nGRANT USAGE ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_ingest;\n\nREVOKE ALL PRIVILEGES ON PROCEDURE run_queued_queries FROM public;\nGRANT ALL ON PROCEDURE run_queued_queries TO pgstac_admin;\n\nREVOKE ALL PRIVILEGES ON FUNCTION run_queued_queries_intransaction FROM public;\nGRANT ALL ON FUNCTION run_queued_queries_intransaction TO pgstac_admin;\n\nRESET ROLE;\n\nSET ROLE pgstac_ingest;\nSELECT update_partition_stats_q(partition) FROM partitions_view;\nSELECT set_version('0.8.6');\n" }, { "path": "src/pgstac/migrations/pgstac.0.9.0-0.9.1.sql", "content": "SET client_min_messages TO WARNING;\nSET SEARCH_PATH to pgstac, public;\nRESET ROLE;\nDO $$\nDECLARE\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='postgis') THEN\n CREATE EXTENSION IF NOT EXISTS postgis;\n END IF;\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='btree_gist') THEN\n CREATE EXTENSION IF NOT EXISTS btree_gist;\n END IF;\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='unaccent') THEN\n CREATE EXTENSION IF NOT EXISTS unaccent;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_admin;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_read;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_ingest;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n\nGRANT pgstac_admin TO current_user;\n\n-- Function to make sure pgstac_admin is the owner of items\nCREATE OR REPLACE FUNCTION pgstac_admin_owns() RETURNS VOID AS $$\nDECLARE\n f RECORD;\nBEGIN\n FOR f IN (\n SELECT\n concat(\n oid::regproc::text,\n '(',\n coalesce(pg_get_function_identity_arguments(oid),''),\n ')'\n ) AS name,\n CASE prokind WHEN 'f' THEN 'FUNCTION' WHEN 'p' THEN 'PROCEDURE' WHEN 'a' THEN 'AGGREGATE' END as typ\n FROM pg_proc\n WHERE\n pronamespace=to_regnamespace('pgstac')\n AND proowner != to_regrole('pgstac_admin')\n AND proname NOT LIKE 'pg_stat%'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n FOR f IN (\n SELECT\n oid::regclass::text as name,\n CASE relkind\n WHEN 'i' THEN 'INDEX'\n WHEN 'I' THEN 'INDEX'\n WHEN 'p' THEN 'TABLE'\n WHEN 'r' THEN 'TABLE'\n WHEN 'v' THEN 'VIEW'\n WHEN 'S' THEN 'SEQUENCE'\n ELSE NULL\n END as typ\n FROM pg_class\n WHERE relnamespace=to_regnamespace('pgstac') and relowner != to_regrole('pgstac_admin') AND relkind IN ('r','p','v','S') AND relname NOT LIKE 'pg_stat'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\nSELECT pgstac_admin_owns();\n\nCREATE SCHEMA IF NOT EXISTS pgstac AUTHORIZATION pgstac_admin;\n\nGRANT ALL ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_admin;\n\nALTER ROLE pgstac_admin SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_read SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_ingest SET SEARCH_PATH TO pgstac, public;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\n\nGRANT pgstac_read TO pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\n\nSET ROLE pgstac_admin;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\nRESET ROLE;\n\nSET ROLE pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\nRESET ROLE;\n\nSET SEARCH_PATH TO pgstac, public;\nSET ROLE pgstac_admin;\n\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS analyze_items;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS validate_constraints;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n-- Install these idempotently as migrations do not put them before trying to modify the collections table\n\n\nCREATE OR REPLACE FUNCTION collection_geom(content jsonb)\nRETURNS geometry AS $$\n WITH box AS (SELECT content->'extent'->'spatial'->'bbox'->0 as box)\n SELECT\n st_makeenvelope(\n (box->>0)::float,\n (box->>1)::float,\n (box->>2)::float,\n (box->>3)::float,\n 4326\n )\n FROM box;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION collection_datetime(content jsonb)\nRETURNS timestamptz AS $$\n SELECT\n CASE\n WHEN\n (content->'extent'->'temporal'->'interval'->0->>0) IS NULL\n THEN '-infinity'::timestamptz\n ELSE\n (content->'extent'->'temporal'->'interval'->0->>0)::timestamptz\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION collection_enddatetime(content jsonb)\nRETURNS timestamptz AS $$\n SELECT\n CASE\n WHEN\n (content->'extent'->'temporal'->'interval'->0->>1) IS NULL\n THEN 'infinity'::timestamptz\n ELSE\n (content->'extent'->'temporal'->'interval'->0->>1)::timestamptz\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n-- BEGIN migra calculated SQL\nset check_function_bodies = off;\n\nCREATE OR REPLACE FUNCTION pgstac.collection_extent(_collection text, runupdate boolean DEFAULT false)\n RETURNS jsonb\n LANGUAGE plpgsql\nAS $function$\nDECLARE\n geom_extent geometry;\n mind timestamptz;\n maxd timestamptz;\n extent jsonb;\nBEGIN\n IF runupdate THEN\n PERFORM update_partition_stats_q(partition)\n FROM partitions_view WHERE collection=_collection;\n END IF;\n SELECT\n min(lower(dtrange)),\n max(upper(edtrange)),\n st_extent(spatial)\n INTO\n mind,\n maxd,\n geom_extent\n FROM partitions_view\n WHERE collection=_collection;\n\n IF geom_extent IS NOT NULL AND mind IS NOT NULL AND maxd IS NOT NULL THEN\n extent := jsonb_build_object(\n 'spatial', jsonb_build_object(\n 'bbox', to_jsonb(array[array[st_xmin(geom_extent), st_ymin(geom_extent), st_xmax(geom_extent), st_ymax(geom_extent)]])\n ),\n 'temporal', jsonb_build_object(\n 'interval', to_jsonb(array[array[mind, maxd]])\n )\n );\n RETURN extent;\n END IF;\n RETURN NULL;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.collection_temporal_extent(id text)\n RETURNS jsonb\n LANGUAGE sql\n IMMUTABLE PARALLEL SAFE\n SET search_path TO 'pgstac', 'public'\nAS $function$\n SELECT to_jsonb(array[array[min(datetime), max(datetime)]])\n FROM items WHERE collection=$1;\n;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.update_collection_extents()\n RETURNS void\n LANGUAGE sql\nAS $function$\nUPDATE collections\n SET content = jsonb_set_lax(\n content,\n '{extent}'::text[],\n collection_extent(id, FALSE),\n true,\n 'use_json_null'\n )\n;\n$function$\n;\n\n\n-- END migra calculated SQL\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('id', '{\"title\": \"Item ID\",\"description\": \"Item identifier\",\"$ref\": \"https://schemas.stacspec.org/v1.0.0/item-spec/json-schema/item.json#/definitions/core/allOf/2/properties/id\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('geometry', '{\"title\": \"Item Geometry\",\"description\": \"Item Geometry\",\"$ref\": \"https://geojson.org/schema/Feature.json\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('datetime','{\"description\": \"Datetime\",\"type\": \"string\",\"title\": \"Acquired\",\"format\": \"date-time\",\"pattern\": \"(\\\\+00:00|Z)$\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDELETE FROM queryables a USING queryables b\n WHERE a.name = b.name AND a.collection_ids IS NOT DISTINCT FROM b.collection_ids AND a.id > b.id;\n\n\nINSERT INTO pgstac_settings (name, value) VALUES\n ('context', 'off'),\n ('context_estimated_count', '100000'),\n ('context_estimated_cost', '100000'),\n ('context_stats_ttl', '1 day'),\n ('default_filter_lang', 'cql2-json'),\n ('additional_properties', 'true'),\n ('use_queue', 'false'),\n ('queue_timeout', '10 minutes'),\n ('update_collection_extent', 'false'),\n ('format_cache', 'false'),\n ('readonly', 'false')\nON CONFLICT DO NOTHING\n;\n\n\nINSERT INTO cql2_ops (op, template, types) VALUES\n ('eq', '%s = %s', NULL),\n ('neq', '%s != %s', NULL),\n ('ne', '%s != %s', NULL),\n ('!=', '%s != %s', NULL),\n ('<>', '%s != %s', NULL),\n ('lt', '%s < %s', NULL),\n ('lte', '%s <= %s', NULL),\n ('gt', '%s > %s', NULL),\n ('gte', '%s >= %s', NULL),\n ('le', '%s <= %s', NULL),\n ('ge', '%s >= %s', NULL),\n ('=', '%s = %s', NULL),\n ('<', '%s < %s', NULL),\n ('<=', '%s <= %s', NULL),\n ('>', '%s > %s', NULL),\n ('>=', '%s >= %s', NULL),\n ('like', '%s LIKE %s', NULL),\n ('ilike', '%s ILIKE %s', NULL),\n ('+', '%s + %s', NULL),\n ('-', '%s - %s', NULL),\n ('*', '%s * %s', NULL),\n ('/', '%s / %s', NULL),\n ('not', 'NOT (%s)', NULL),\n ('between', '%s BETWEEN %s AND %s', NULL),\n ('isnull', '%s IS NULL', NULL),\n ('upper', 'upper(%s)', NULL),\n ('lower', 'lower(%s)', NULL),\n ('casei', 'upper(%s)', NULL),\n ('accenti', 'unaccent(%s)', NULL)\nON CONFLICT (op) DO UPDATE\n SET\n template = EXCLUDED.template\n;\n\n\nALTER FUNCTION to_text COST 5000;\nALTER FUNCTION to_float COST 5000;\nALTER FUNCTION to_int COST 5000;\nALTER FUNCTION to_tstz COST 5000;\nALTER FUNCTION to_text_array COST 5000;\n\nALTER FUNCTION update_partition_stats SECURITY DEFINER;\nALTER FUNCTION partition_after_triggerfunc SECURITY DEFINER;\nALTER FUNCTION drop_table_constraints SECURITY DEFINER;\nALTER FUNCTION create_table_constraints SECURITY DEFINER;\nALTER FUNCTION check_partition SECURITY DEFINER;\nALTER FUNCTION repartition SECURITY DEFINER;\nALTER FUNCTION where_stats SECURITY DEFINER;\nALTER FUNCTION search_query SECURITY DEFINER;\nALTER FUNCTION format_item SECURITY DEFINER;\nALTER FUNCTION maintain_index SECURITY DEFINER;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nGRANT ALL ON SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac to pgstac_admin;\n\n-- pgstac_read role limited to using function apis\nGRANT EXECUTE ON FUNCTION search TO pgstac_read;\nGRANT EXECUTE ON FUNCTION search_query TO pgstac_read;\nGRANT EXECUTE ON FUNCTION item_by_id TO pgstac_read;\nGRANT EXECUTE ON FUNCTION get_item TO pgstac_read;\nGRANT SELECT ON ALL TABLES IN SCHEMA pgstac TO pgstac_read;\n\n\nGRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_ingest;\nGRANT USAGE ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_ingest;\n\nREVOKE ALL PRIVILEGES ON PROCEDURE run_queued_queries FROM public;\nGRANT ALL ON PROCEDURE run_queued_queries TO pgstac_admin;\n\nREVOKE ALL PRIVILEGES ON FUNCTION run_queued_queries_intransaction FROM public;\nGRANT ALL ON FUNCTION run_queued_queries_intransaction TO pgstac_admin;\n\nRESET ROLE;\n\nSET ROLE pgstac_ingest;\nSELECT update_partition_stats_q(partition) FROM partitions_view;\nSELECT set_version('0.9.1');\n" }, { "path": "src/pgstac/migrations/pgstac.0.9.0.sql", "content": "RESET ROLE;\nDO $$\nDECLARE\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='postgis') THEN\n CREATE EXTENSION IF NOT EXISTS postgis;\n END IF;\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='btree_gist') THEN\n CREATE EXTENSION IF NOT EXISTS btree_gist;\n END IF;\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='unaccent') THEN\n CREATE EXTENSION IF NOT EXISTS unaccent;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_admin;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_read;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_ingest;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n\nGRANT pgstac_admin TO current_user;\n\n-- Function to make sure pgstac_admin is the owner of items\nCREATE OR REPLACE FUNCTION pgstac_admin_owns() RETURNS VOID AS $$\nDECLARE\n f RECORD;\nBEGIN\n FOR f IN (\n SELECT\n concat(\n oid::regproc::text,\n '(',\n coalesce(pg_get_function_identity_arguments(oid),''),\n ')'\n ) AS name,\n CASE prokind WHEN 'f' THEN 'FUNCTION' WHEN 'p' THEN 'PROCEDURE' WHEN 'a' THEN 'AGGREGATE' END as typ\n FROM pg_proc\n WHERE\n pronamespace=to_regnamespace('pgstac')\n AND proowner != to_regrole('pgstac_admin')\n AND proname NOT LIKE 'pg_stat%'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n FOR f IN (\n SELECT\n oid::regclass::text as name,\n CASE relkind\n WHEN 'i' THEN 'INDEX'\n WHEN 'I' THEN 'INDEX'\n WHEN 'p' THEN 'TABLE'\n WHEN 'r' THEN 'TABLE'\n WHEN 'v' THEN 'VIEW'\n WHEN 'S' THEN 'SEQUENCE'\n ELSE NULL\n END as typ\n FROM pg_class\n WHERE relnamespace=to_regnamespace('pgstac') and relowner != to_regrole('pgstac_admin') AND relkind IN ('r','p','v','S') AND relname NOT LIKE 'pg_stat'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\nSELECT pgstac_admin_owns();\n\nCREATE SCHEMA IF NOT EXISTS pgstac AUTHORIZATION pgstac_admin;\n\nGRANT ALL ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_admin;\n\nALTER ROLE pgstac_admin SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_read SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_ingest SET SEARCH_PATH TO pgstac, public;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\n\nGRANT pgstac_read TO pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\n\nSET ROLE pgstac_admin;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\nRESET ROLE;\n\nSET ROLE pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\nRESET ROLE;\n\nSET SEARCH_PATH TO pgstac, public;\nSET ROLE pgstac_admin;\n\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS analyze_items;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS validate_constraints;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n-- Install these idempotently as migrations do not put them before trying to modify the collections table\n\n\nCREATE OR REPLACE FUNCTION collection_geom(content jsonb)\nRETURNS geometry AS $$\n WITH box AS (SELECT content->'extent'->'spatial'->'bbox'->0 as box)\n SELECT\n st_makeenvelope(\n (box->>0)::float,\n (box->>1)::float,\n (box->>2)::float,\n (box->>3)::float,\n 4326\n )\n FROM box;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION collection_datetime(content jsonb)\nRETURNS timestamptz AS $$\n SELECT\n CASE\n WHEN\n (content->'extent'->'temporal'->'interval'->0->>0) IS NULL\n THEN '-infinity'::timestamptz\n ELSE\n (content->'extent'->'temporal'->'interval'->0->>0)::timestamptz\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION collection_enddatetime(content jsonb)\nRETURNS timestamptz AS $$\n SELECT\n CASE\n WHEN\n (content->'extent'->'temporal'->'interval'->0->>1) IS NULL\n THEN 'infinity'::timestamptz\n ELSE\n (content->'extent'->'temporal'->'interval'->0->>1)::timestamptz\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE TABLE IF NOT EXISTS migrations (\n version text PRIMARY KEY,\n datetime timestamptz DEFAULT clock_timestamp() NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION get_version() RETURNS text AS $$\n SELECT version FROM pgstac.migrations ORDER BY datetime DESC, version DESC LIMIT 1;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION set_version(text) RETURNS text AS $$\n INSERT INTO pgstac.migrations (version) VALUES ($1)\n ON CONFLICT DO NOTHING\n RETURNING version;\n$$ LANGUAGE SQL;\n\n\nCREATE TABLE IF NOT EXISTS pgstac_settings (\n name text PRIMARY KEY,\n value text NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION table_empty(text) RETURNS boolean AS $$\nDECLARE\n retval boolean;\nBEGIN\n EXECUTE format($q$\n SELECT NOT EXISTS (SELECT 1 FROM %I LIMIT 1)\n $q$,\n $1\n ) INTO retval;\n RETURN retval;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION get_setting(IN _setting text, IN conf jsonb DEFAULT NULL) RETURNS text AS $$\nSELECT COALESCE(\n nullif(conf->>_setting, ''),\n nullif(current_setting(concat('pgstac.',_setting), TRUE),''),\n nullif((SELECT value FROM pgstac.pgstac_settings WHERE name=_setting),'')\n);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_setting_bool(IN _setting text, IN conf jsonb DEFAULT NULL) RETURNS boolean AS $$\nSELECT COALESCE(\n nullif(conf->>_setting, ''),\n nullif(current_setting(concat('pgstac.',_setting), TRUE),''),\n nullif((SELECT value FROM pgstac.pgstac_settings WHERE name=_setting),''),\n 'FALSE'\n)::boolean;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION base_url(conf jsonb DEFAULT NULL) RETURNS text AS $$\n SELECT COALESCE(pgstac.get_setting('base_url', conf), '.');\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION additional_properties() RETURNS boolean AS $$\n SELECT pgstac.get_setting_bool('additional_properties');\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION readonly(conf jsonb DEFAULT NULL) RETURNS boolean AS $$\n SELECT pgstac.get_setting_bool('readonly', conf);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context(conf jsonb DEFAULT NULL) RETURNS text AS $$\n SELECT pgstac.get_setting('context', conf);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context_estimated_count(conf jsonb DEFAULT NULL) RETURNS int AS $$\n SELECT pgstac.get_setting('context_estimated_count', conf)::int;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_estimated_cost();\nCREATE OR REPLACE FUNCTION context_estimated_cost(conf jsonb DEFAULT NULL) RETURNS float AS $$\n SELECT pgstac.get_setting('context_estimated_cost', conf)::float;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_stats_ttl();\nCREATE OR REPLACE FUNCTION context_stats_ttl(conf jsonb DEFAULT NULL) RETURNS interval AS $$\n SELECT pgstac.get_setting('context_stats_ttl', conf)::interval;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION t2s(text) RETURNS text AS $$\n SELECT extract(epoch FROM $1::interval)::text || ' s';\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE STRICT;\n\nCREATE OR REPLACE FUNCTION age_ms(a timestamptz, b timestamptz DEFAULT clock_timestamp()) RETURNS float AS $$\n SELECT abs(extract(epoch from age(a,b)) * 1000);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION queue_timeout() RETURNS interval AS $$\n SELECT t2s(coalesce(\n get_setting('queue_timeout'),\n '1h'\n ))::interval;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION notice(VARIADIC text[]) RETURNS boolean AS $$\nDECLARE\ndebug boolean := current_setting('pgstac.debug', true);\nBEGIN\n IF debug THEN\n RAISE NOTICE 'NOTICE FROM FUNC: % >>>>> %', concat_ws(' | ', $1), clock_timestamp();\n RETURN TRUE;\n END IF;\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION empty_arr(ANYARRAY) RETURNS BOOLEAN AS $$\nSELECT CASE\n WHEN $1 IS NULL THEN TRUE\n WHEN cardinality($1)<1 THEN TRUE\nELSE FALSE\nEND;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_intersection(_a ANYARRAY, _b ANYARRAY) RETURNS ANYARRAY AS $$\n SELECT ARRAY ( SELECT unnest(_a) INTERSECT SELECT UNNEST(_b) );\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION array_map_ident(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_ident(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_map_literal(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_literal(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION array_reverse(anyarray) RETURNS anyarray AS $$\nSELECT ARRAY(\n SELECT $1[i]\n FROM generate_subscripts($1,1) AS s(i)\n ORDER BY i DESC\n);\n$$ LANGUAGE SQL STRICT IMMUTABLE;\n\nDROP TABLE IF EXISTS query_queue;\nCREATE TABLE query_queue (\n query text PRIMARY KEY,\n added timestamptz DEFAULT now()\n);\n\nDROP TABLE IF EXISTS query_queue_history;\nCREATE TABLE query_queue_history(\n query text,\n added timestamptz NOT NULL,\n finished timestamptz NOT NULL DEFAULT now(),\n error text\n);\n\nCREATE OR REPLACE PROCEDURE run_queued_queries() AS $$\nDECLARE\n qitem query_queue%ROWTYPE;\n timeout_ts timestamptz;\n error text;\n cnt int := 0;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n DELETE FROM query_queue WHERE query = (SELECT query FROM query_queue ORDER BY added DESC LIMIT 1 FOR UPDATE SKIP LOCKED) RETURNING * INTO qitem;\n IF NOT FOUND THEN\n EXIT;\n END IF;\n cnt := cnt + 1;\n BEGIN\n RAISE NOTICE 'RUNNING QUERY: %', qitem.query;\n EXECUTE qitem.query;\n EXCEPTION WHEN others THEN\n error := format('%s | %s', SQLERRM, SQLSTATE);\n END;\n INSERT INTO query_queue_history (query, added, finished, error)\n VALUES (qitem.query, qitem.added, clock_timestamp(), error);\n COMMIT;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION run_queued_queries_intransaction() RETURNS int AS $$\nDECLARE\n qitem query_queue%ROWTYPE;\n timeout_ts timestamptz;\n error text;\n cnt int := 0;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n DELETE FROM query_queue WHERE query = (SELECT query FROM query_queue ORDER BY added DESC LIMIT 1 FOR UPDATE SKIP LOCKED) RETURNING * INTO qitem;\n IF NOT FOUND THEN\n RETURN cnt;\n END IF;\n cnt := cnt + 1;\n BEGIN\n qitem.query := regexp_replace(qitem.query, 'CONCURRENTLY', '');\n RAISE NOTICE 'RUNNING QUERY: %', qitem.query;\n\n EXECUTE qitem.query;\n EXCEPTION WHEN others THEN\n error := format('%s | %s', SQLERRM, SQLSTATE);\n RAISE WARNING '%', error;\n END;\n INSERT INTO query_queue_history (query, added, finished, error)\n VALUES (qitem.query, qitem.added, clock_timestamp(), error);\n END LOOP;\n RETURN cnt;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION run_or_queue(query text) RETURNS VOID AS $$\nDECLARE\n use_queue text := COALESCE(get_setting('use_queue'), 'FALSE')::boolean;\nBEGIN\n IF get_setting_bool('debug') THEN\n RAISE NOTICE '%', query;\n END IF;\n IF use_queue THEN\n INSERT INTO query_queue (query) VALUES (query) ON CONFLICT DO NOTHING;\n ELSE\n EXECUTE query;\n END IF;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n\nDROP FUNCTION IF EXISTS check_pgstac_settings;\nCREATE OR REPLACE FUNCTION check_pgstac_settings(_sysmem text DEFAULT NULL) RETURNS VOID AS $$\nDECLARE\n settingval text;\n sysmem bigint := pg_size_bytes(_sysmem);\n effective_cache_size bigint := pg_size_bytes(current_setting('effective_cache_size', TRUE));\n shared_buffers bigint := pg_size_bytes(current_setting('shared_buffers', TRUE));\n work_mem bigint := pg_size_bytes(current_setting('work_mem', TRUE));\n max_connections int := current_setting('max_connections', TRUE);\n maintenance_work_mem bigint := pg_size_bytes(current_setting('maintenance_work_mem', TRUE));\n seq_page_cost float := current_setting('seq_page_cost', TRUE);\n random_page_cost float := current_setting('random_page_cost', TRUE);\n temp_buffers bigint := pg_size_bytes(current_setting('temp_buffers', TRUE));\n r record;\nBEGIN\n IF _sysmem IS NULL THEN\n RAISE NOTICE 'Call function with the size of your system memory `SELECT check_pgstac_settings(''4GB'')` to get pg system setting recommendations.';\n ELSE\n IF effective_cache_size < (sysmem * 0.5) THEN\n RAISE WARNING 'effective_cache_size of % is set low for a system with %. Recomended value between % and %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.5), pg_size_pretty(sysmem * 0.75);\n ELSIF effective_cache_size > (sysmem * 0.75) THEN\n RAISE WARNING 'effective_cache_size of % is set high for a system with %. Recomended value between % and %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.5), pg_size_pretty(sysmem * 0.75);\n ELSE\n RAISE NOTICE 'effective_cache_size of % is set appropriately for a system with %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem);\n END IF;\n\n IF shared_buffers < (sysmem * 0.2) THEN\n RAISE WARNING 'shared_buffers of % is set low for a system with %. Recomended value between % and %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.2), pg_size_pretty(sysmem * 0.3);\n ELSIF shared_buffers > (sysmem * 0.3) THEN\n RAISE WARNING 'shared_buffers of % is set high for a system with %. Recomended value between % and %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.2), pg_size_pretty(sysmem * 0.3);\n ELSE\n RAISE NOTICE 'shared_buffers of % is set appropriately for a system with %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem);\n END IF;\n shared_buffers = sysmem * 0.3;\n IF maintenance_work_mem < (sysmem * 0.2) THEN\n RAISE WARNING 'maintenance_work_mem of % is set low for shared_buffers of %. Recomended value between % and %', pg_size_pretty(maintenance_work_mem), pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers * 0.2), pg_size_pretty(shared_buffers * 0.3);\n ELSIF maintenance_work_mem > (shared_buffers * 0.3) THEN\n RAISE WARNING 'maintenance_work_mem of % is set high for shared_buffers of %. Recomended value between % and %', pg_size_pretty(maintenance_work_mem), pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers * 0.2), pg_size_pretty(shared_buffers * 0.3);\n ELSE\n RAISE NOTICE 'maintenance_work_mem of % is set appropriately for shared_buffers of %', pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers);\n END IF;\n\n IF work_mem * max_connections > shared_buffers THEN\n RAISE WARNING 'work_mem setting of % is set high for % max_connections please reduce work_mem to % or decrease max_connections to %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers/max_connections), floor(shared_buffers/work_mem);\n ELSIF work_mem * max_connections < (shared_buffers * 0.75) THEN\n RAISE WARNING 'work_mem setting of % is set low for % max_connections you may consider raising work_mem to % or increasing max_connections to %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers/max_connections), floor(shared_buffers/work_mem);\n ELSE\n RAISE NOTICE 'work_mem setting of % and max_connections of % are adequate for shared_buffers of %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers);\n END IF;\n\n IF random_page_cost / seq_page_cost != 1.1 THEN\n RAISE WARNING 'random_page_cost (%) /seq_page_cost (%) should be set to 1.1 for SSD. Change random_page_cost to %', random_page_cost, seq_page_cost, 1.1 * seq_page_cost;\n ELSE\n RAISE NOTICE 'random_page_cost and seq_page_cost set appropriately for SSD';\n END IF;\n\n IF temp_buffers < greatest(pg_size_bytes('128MB'),(maintenance_work_mem / 2)) THEN\n RAISE WARNING 'pgstac makes heavy use of temp tables, consider raising temp_buffers from % to %', pg_size_pretty(temp_buffers), greatest('128MB', pg_size_pretty((shared_buffers / 16)));\n END IF;\n END IF;\n\n RAISE NOTICE 'VALUES FOR PGSTAC VARIABLES';\n RAISE NOTICE 'These can be set either as GUC system variables or by setting in the pgstac_settings table.';\n\n FOR r IN SELECT name, get_setting(name) as setting, CASE WHEN current_setting(concat('pgstac.',name), TRUE) IS NOT NULL THEN concat('pgstac.',name, ' GUC') WHEN value IS NOT NULL THEN 'pgstac_settings table' ELSE 'Not Set' END as loc FROM pgstac_settings LOOP\n RAISE NOTICE '% is set to % from the %', r.name, r.setting, r.loc;\n END LOOP;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pg_cron';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider intalling pg_cron which can be used to automate tasks';\n ELSE\n RAISE NOTICE 'pg_cron % is installed', settingval;\n END IF;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pgstattuple';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider installing the pgstattuple extension which can be used to help maintain tables and indexes.';\n ELSE\n RAISE NOTICE 'pgstattuple % is installed', settingval;\n END IF;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pg_stat_statements';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider installing the pg_stat_statements extension which is very helpful for tracking the types of queries on the system';\n ELSE\n RAISE NOTICE 'pg_stat_statements % is installed', settingval;\n IF current_setting('pg_stat_statements.track_statements', TRUE) IS DISTINCT FROM 'all' THEN\n RAISE WARNING 'SET pg_stat_statements.track_statements TO ''all''; --In order to track statements within functions.';\n END IF;\n END IF;\n\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac, public SET CLIENT_MIN_MESSAGES TO NOTICE;\nCREATE OR REPLACE FUNCTION to_int(jsonb) RETURNS int AS $$\n SELECT floor(($1->>0)::float)::int;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION to_float(jsonb) RETURNS float AS $$\n SELECT ($1->>0)::float;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION to_tstz(jsonb) RETURNS timestamptz AS $$\n SELECT ($1->>0)::timestamptz;\n$$ LANGUAGE SQL IMMUTABLE STRICT SET TIME ZONE 'UTC' COST 5000 PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION to_text(jsonb) RETURNS text AS $$\n SELECT CASE WHEN jsonb_typeof($1) IN ('array','object') THEN $1::text ELSE $1->>0 END;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION to_text_array(jsonb) RETURNS text[] AS $$\n SELECT\n CASE jsonb_typeof($1)\n WHEN 'array' THEN ARRAY(SELECT jsonb_array_elements_text($1))\n ELSE ARRAY[$1->>0]\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION bbox_geom(_bbox jsonb) RETURNS geometry AS $$\nSELECT CASE jsonb_array_length(_bbox)\n WHEN 4 THEN\n ST_SetSRID(ST_MakeEnvelope(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float,\n (_bbox->>3)::float\n ),4326)\n WHEN 6 THEN\n ST_SetSRID(ST_3DMakeBox(\n ST_MakePoint(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float\n ),\n ST_MakePoint(\n (_bbox->>3)::float,\n (_bbox->>4)::float,\n (_bbox->>5)::float\n )\n ),4326)\n ELSE null END;\n;\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION geom_bbox(_geom geometry) RETURNS jsonb AS $$\n SELECT jsonb_build_array(\n st_xmin(_geom),\n st_ymin(_geom),\n st_xmax(_geom),\n st_ymax(_geom)\n );\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION flip_jsonb_array(j jsonb) RETURNS jsonb AS $$\n SELECT jsonb_agg(value) FROM (SELECT value FROM jsonb_array_elements(j) WITH ORDINALITY ORDER BY ordinality DESC) as t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION explode_dotpaths(j jsonb) RETURNS SETOF text[] AS $$\n SELECT string_to_array(p, '.') as e FROM jsonb_array_elements_text(j) p;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION explode_dotpaths_recurse(IN j jsonb) RETURNS SETOF text[] AS $$\n WITH RECURSIVE t AS (\n SELECT e FROM explode_dotpaths(j) e\n UNION ALL\n SELECT e[1:cardinality(e)-1]\n FROM t\n WHERE cardinality(e)>1\n ) SELECT e FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION jsonb_set_nested(j jsonb, path text[], val jsonb) RETURNS jsonb AS $$\nDECLARE\nBEGIN\n IF cardinality(path) > 1 THEN\n FOR i IN 1..(cardinality(path)-1) LOOP\n IF j #> path[:i] IS NULL THEN\n j := jsonb_set_lax(j, path[:i], '{}', TRUE);\n END IF;\n END LOOP;\n END IF;\n RETURN jsonb_set_lax(j, path, val, true);\n\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\n\n\nCREATE OR REPLACE FUNCTION jsonb_include(j jsonb, f jsonb) RETURNS jsonb AS $$\nDECLARE\n includes jsonb := f-> 'include';\n outj jsonb := '{}'::jsonb;\n path text[];\nBEGIN\n IF\n includes IS NULL\n OR jsonb_array_length(includes) = 0\n THEN\n RETURN j;\n ELSE\n includes := includes || '[\"id\",\"collection\"]'::jsonb;\n FOR path IN SELECT explode_dotpaths(includes) LOOP\n outj := jsonb_set_nested(outj, path, j #> path);\n END LOOP;\n END IF;\n RETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_exclude(j jsonb, f jsonb) RETURNS jsonb AS $$\nDECLARE\n excludes jsonb := f-> 'exclude';\n outj jsonb := j;\n path text[];\nBEGIN\n IF\n excludes IS NULL\n OR jsonb_array_length(excludes) = 0\n THEN\n RETURN j;\n ELSE\n FOR path IN SELECT explode_dotpaths(excludes) LOOP\n outj := outj #- path;\n END LOOP;\n END IF;\n RETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_fields(j jsonb, f jsonb DEFAULT '{\"fields\":[]}') RETURNS jsonb AS $$\n SELECT jsonb_exclude(jsonb_include(j, f), f);\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION merge_jsonb(_a jsonb, _b jsonb) RETURNS jsonb AS $$\n SELECT\n CASE\n WHEN _a = '\"𒍟※\"'::jsonb THEN NULL\n WHEN _a IS NULL OR jsonb_typeof(_a) = 'null' THEN _b\n WHEN jsonb_typeof(_a) = 'object' AND jsonb_typeof(_b) = 'object' THEN\n (\n SELECT\n jsonb_strip_nulls(\n jsonb_object_agg(\n key,\n merge_jsonb(a.value, b.value)\n )\n )\n FROM\n jsonb_each(coalesce(_a,'{}'::jsonb)) as a\n FULL JOIN\n jsonb_each(coalesce(_b,'{}'::jsonb)) as b\n USING (key)\n )\n WHEN\n jsonb_typeof(_a) = 'array'\n AND jsonb_typeof(_b) = 'array'\n AND jsonb_array_length(_a) = jsonb_array_length(_b)\n THEN\n (\n SELECT jsonb_agg(m) FROM\n ( SELECT\n merge_jsonb(\n jsonb_array_elements(_a),\n jsonb_array_elements(_b)\n ) as m\n ) as l\n )\n ELSE _a\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION strip_jsonb(_a jsonb, _b jsonb) RETURNS jsonb AS $$\n SELECT\n CASE\n\n WHEN (_a IS NULL OR jsonb_typeof(_a) = 'null') AND _b IS NOT NULL AND jsonb_typeof(_b) != 'null' THEN '\"𒍟※\"'::jsonb\n WHEN _b IS NULL OR jsonb_typeof(_a) = 'null' THEN _a\n WHEN _a = _b AND jsonb_typeof(_a) = 'object' THEN '{}'::jsonb\n WHEN _a = _b THEN NULL\n WHEN jsonb_typeof(_a) = 'object' AND jsonb_typeof(_b) = 'object' THEN\n (\n SELECT\n jsonb_strip_nulls(\n jsonb_object_agg(\n key,\n strip_jsonb(a.value, b.value)\n )\n )\n FROM\n jsonb_each(_a) as a\n FULL JOIN\n jsonb_each(_b) as b\n USING (key)\n )\n WHEN\n jsonb_typeof(_a) = 'array'\n AND jsonb_typeof(_b) = 'array'\n AND jsonb_array_length(_a) = jsonb_array_length(_b)\n THEN\n (\n SELECT jsonb_agg(m) FROM\n ( SELECT\n strip_jsonb(\n jsonb_array_elements(_a),\n jsonb_array_elements(_b)\n ) as m\n ) as l\n )\n ELSE _a\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION nullif_jsonbnullempty(j jsonb) RETURNS jsonb AS $$\n SELECT nullif(nullif(nullif(j,'null'::jsonb),'{}'::jsonb),'[]'::jsonb);\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION jsonb_array_unique(j jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(jsonb_agg(DISTINCT a)) v FROM jsonb_array_elements(j) a;\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_concat_ignorenull(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT coalesce(a,'[]'::jsonb) || coalesce(b,'[]'::jsonb);\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_least(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(least(nullif_jsonbnullempty(a), nullif_jsonbnullempty(b)));\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_greatest(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(greatest(a, b));\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION first_notnull_sfunc(anyelement, anyelement) RETURNS anyelement AS $$\n SELECT COALESCE($1,$2);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE AGGREGATE first_notnull(anyelement)(\n SFUNC = first_notnull_sfunc,\n STYPE = anyelement\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_array_unique_merge(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_concat_ignorenull,\n FINALFUNC = jsonb_array_unique\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_min(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_least\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_max(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_greatest\n);\n/* looks for a geometry in a stac item first from geometry and falling back to bbox */\nCREATE OR REPLACE FUNCTION stac_geom(value jsonb) RETURNS geometry AS $$\nSELECT\n CASE\n WHEN value ? 'intersects' THEN\n ST_GeomFromGeoJSON(value->>'intersects')\n WHEN value ? 'geometry' THEN\n ST_GeomFromGeoJSON(value->>'geometry')\n WHEN value ? 'bbox' THEN\n pgstac.bbox_geom(value->'bbox')\n ELSE NULL\n END as geometry\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION stac_daterange(\n value jsonb\n) RETURNS tstzrange AS $$\nDECLARE\n props jsonb := value;\n dt timestamptz;\n edt timestamptz;\nBEGIN\n IF props ? 'properties' THEN\n props := props->'properties';\n END IF;\n IF\n props ? 'start_datetime'\n AND props->>'start_datetime' IS NOT NULL\n AND props ? 'end_datetime'\n AND props->>'end_datetime' IS NOT NULL\n THEN\n dt := props->>'start_datetime';\n edt := props->>'end_datetime';\n IF dt > edt THEN\n RAISE EXCEPTION 'start_datetime must be < end_datetime';\n END IF;\n ELSE\n dt := props->>'datetime';\n edt := props->>'datetime';\n END IF;\n IF dt is NULL OR edt IS NULL THEN\n RAISE NOTICE 'DT: %, EDT: %', dt, edt;\n RAISE EXCEPTION 'Either datetime (%) or both start_datetime (%) and end_datetime (%) must be set.', props->>'datetime',props->>'start_datetime',props->>'end_datetime';\n END IF;\n RETURN tstzrange(dt, edt, '[]');\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_datetime(value jsonb) RETURNS timestamptz AS $$\n SELECT lower(stac_daterange(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_end_datetime(value jsonb) RETURNS timestamptz AS $$\n SELECT upper(stac_daterange(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE TABLE IF NOT EXISTS stac_extensions(\n url text PRIMARY KEY,\n content jsonb\n);\nCREATE OR REPLACE FUNCTION collection_base_item(content jsonb) RETURNS jsonb AS $$\n SELECT jsonb_build_object(\n 'type', 'Feature',\n 'stac_version', content->'stac_version',\n 'assets', content->'item_assets',\n 'collection', content->'id'\n );\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE TABLE IF NOT EXISTS collections (\n key bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,\n id text GENERATED ALWAYS AS (content->>'id') STORED UNIQUE NOT NULL,\n content JSONB NOT NULL,\n base_item jsonb GENERATED ALWAYS AS (pgstac.collection_base_item(content)) STORED,\n geometry geometry GENERATED ALWAYS AS (pgstac.collection_geom(content)) STORED,\n datetime timestamptz GENERATED ALWAYS AS (pgstac.collection_datetime(content)) STORED,\n end_datetime timestamptz GENERATED ALWAYS AS (pgstac.collection_enddatetime(content)) STORED,\n private jsonb,\n partition_trunc text CHECK (partition_trunc IN ('year', 'month'))\n);\n\nCREATE OR REPLACE FUNCTION collection_base_item(cid text) RETURNS jsonb AS $$\n SELECT pgstac.collection_base_item(content) FROM pgstac.collections WHERE id = cid LIMIT 1;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION create_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION update_collection(data jsonb) RETURNS VOID AS $$\nDECLARE\n out collections%ROWTYPE;\nBEGIN\n UPDATE collections SET content=data WHERE id = data->>'id' RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ON CONFLICT (id) DO\n UPDATE\n SET content=EXCLUDED.content\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION delete_collection(_id text) RETURNS VOID AS $$\nDECLARE\n out collections%ROWTYPE;\nBEGIN\n DELETE FROM collections WHERE id = _id RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION get_collection(id text) RETURNS jsonb AS $$\n SELECT content FROM collections\n WHERE id=$1\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION all_collections() RETURNS jsonb AS $$\n SELECT coalesce(jsonb_agg(content), '[]'::jsonb) FROM collections;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION collection_delete_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n collection_base_partition text := concat('_items_', OLD.key);\nBEGIN\n EXECUTE format($q$\n DELETE FROM partition_stats WHERE partition IN (\n SELECT partition FROM partition_sys_meta\n WHERE collection=%L\n );\n DROP TABLE IF EXISTS %I CASCADE;\n $q$,\n OLD.id,\n collection_base_partition\n );\n RETURN OLD;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER collection_delete_trigger BEFORE DELETE ON collections\nFOR EACH ROW EXECUTE FUNCTION collection_delete_trigger_func();\nCREATE OR REPLACE FUNCTION queryable_signature(n text, c text[]) RETURNS text AS $$\n SELECT concat(n, c);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TABLE queryables (\n id bigint GENERATED ALWAYS AS identity PRIMARY KEY,\n name text NOT NULL,\n collection_ids text[], -- used to determine what partitions to create indexes on\n definition jsonb,\n property_path text,\n property_wrapper text,\n property_index_type text\n);\nCREATE INDEX queryables_name_idx ON queryables (name);\nCREATE INDEX queryables_collection_idx ON queryables USING GIN (collection_ids);\nCREATE INDEX queryables_property_wrapper_idx ON queryables (property_wrapper);\n\nCREATE OR REPLACE FUNCTION pgstac.queryables_constraint_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n allcollections text[];\nBEGIN\n RAISE NOTICE 'Making sure that name/collection is unique for queryables %', NEW;\n IF NEW.collection_ids IS NOT NULL THEN\n IF EXISTS (\n SELECT 1\n FROM unnest(NEW.collection_ids) c\n LEFT JOIN\n collections\n ON (collections.id = c)\n WHERE collections.id IS NULL\n ) THEN\n RAISE foreign_key_violation USING MESSAGE = format(\n 'One or more collections in %s do not exist.', NEW.collection_ids\n );\n RETURN NULL;\n END IF;\n END IF;\n IF TG_OP = 'INSERT' THEN\n IF EXISTS (\n SELECT 1 FROM queryables q\n WHERE\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n )\n ) THEN\n RAISE unique_violation USING MESSAGE = format(\n 'There is already a queryable for %s for a collection in %s: %s',\n NEW.name,\n NEW.collection_ids,\n\t\t\t\t(SELECT json_agg(row_to_json(q)) FROM queryables q WHERE\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n ))\n );\n RETURN NULL;\n END IF;\n END IF;\n IF TG_OP = 'UPDATE' THEN\n IF EXISTS (\n SELECT 1 FROM queryables q\n WHERE\n q.id != NEW.id\n AND\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n )\n ) THEN\n RAISE unique_violation\n USING MESSAGE = format(\n 'There is already a queryable for %s for a collection in %s',\n NEW.name,\n NEW.collection_ids\n );\n RETURN NULL;\n END IF;\n END IF;\n\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER queryables_constraint_insert_trigger\nBEFORE INSERT ON queryables\nFOR EACH ROW EXECUTE PROCEDURE queryables_constraint_triggerfunc();\n\nCREATE TRIGGER queryables_constraint_update_trigger\nBEFORE UPDATE ON queryables\nFOR EACH ROW\nWHEN (NEW.name = OLD.name AND NEW.collection_ids IS DISTINCT FROM OLD.collection_ids)\nEXECUTE PROCEDURE queryables_constraint_triggerfunc();\n\n\nCREATE OR REPLACE FUNCTION array_to_path(arr text[]) RETURNS text AS $$\n SELECT string_agg(\n quote_literal(v),\n '->'\n ) FROM unnest(arr) v;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\n\n\nCREATE OR REPLACE FUNCTION queryable(\n IN dotpath text,\n OUT path text,\n OUT expression text,\n OUT wrapper text,\n OUT nulled_wrapper text\n) AS $$\nDECLARE\n q RECORD;\n path_elements text[];\nBEGIN\n IF dotpath IN ('id', 'geometry', 'datetime', 'end_datetime', 'collection') THEN\n path := dotpath;\n expression := dotpath;\n wrapper := NULL;\n RETURN;\n END IF;\n SELECT * INTO q FROM queryables\n WHERE\n name=dotpath\n OR name = 'properties.' || dotpath\n OR name = replace(dotpath, 'properties.', '')\n ;\n IF q.property_wrapper IS NULL THEN\n IF q.definition->>'type' = 'number' THEN\n wrapper := 'to_float';\n nulled_wrapper := wrapper;\n ELSIF q.definition->>'format' = 'date-time' THEN\n wrapper := 'to_tstz';\n nulled_wrapper := wrapper;\n ELSE\n nulled_wrapper := NULL;\n wrapper := 'to_text';\n END IF;\n ELSE\n wrapper := q.property_wrapper;\n nulled_wrapper := wrapper;\n END IF;\n IF q.property_path IS NOT NULL THEN\n path := q.property_path;\n ELSE\n path_elements := string_to_array(dotpath, '.');\n IF path_elements[1] IN ('links', 'assets', 'stac_version', 'stac_extensions') THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSIF path_elements[1] = 'properties' THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSE\n path := format($F$content->'properties'->%s$F$, array_to_path(path_elements));\n END IF;\n END IF;\n expression := format('%I(%s)', wrapper, path);\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\nCREATE OR REPLACE FUNCTION unnest_collection(collection_ids text[] DEFAULT NULL) RETURNS SETOF text AS $$\n DECLARE\n BEGIN\n IF collection_ids IS NULL THEN\n RETURN QUERY SELECT id FROM collections;\n END IF;\n RETURN QUERY SELECT unnest(collection_ids);\n END;\n$$ LANGUAGE PLPGSQL STABLE;\n\nCREATE OR REPLACE FUNCTION normalize_indexdef(def text) RETURNS text AS $$\nDECLARE\nBEGIN\n def := btrim(def, ' \\n\\t');\n\tdef := regexp_replace(def, '^CREATE (UNIQUE )?INDEX ([^ ]* )?ON (ONLY )?([^ ]* )?', '', 'i');\n RETURN def;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION indexdef(q queryables) RETURNS text AS $$\n DECLARE\n out text;\n BEGIN\n IF q.name = 'id' THEN\n out := 'CREATE UNIQUE INDEX ON %I USING btree (id)';\n ELSIF q.name = 'datetime' THEN\n out := 'CREATE INDEX ON %I USING btree (datetime DESC, end_datetime)';\n ELSIF q.name = 'geometry' THEN\n out := 'CREATE INDEX ON %I USING gist (geometry)';\n ELSE\n out := format($q$CREATE INDEX ON %%I USING %s (%s(((content -> 'properties'::text) -> %L::text)))$q$,\n lower(COALESCE(q.property_index_type, 'BTREE')),\n lower(COALESCE(q.property_wrapper, 'to_text')),\n q.name\n );\n END IF;\n RETURN btrim(out, ' \\n\\t');\n END;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nDROP VIEW IF EXISTS pgstac_indexes;\nCREATE VIEW pgstac_indexes AS\nSELECT\n i.schemaname,\n i.tablename,\n i.indexname,\n regexp_replace(btrim(replace(replace(indexdef, i.indexname, ''),'pgstac.',''),' \\t\\n'), '[ ]+', ' ', 'g') as idx,\n COALESCE(\n (regexp_match(indexdef, '\\(([a-zA-Z]+)\\)'))[1],\n (regexp_match(indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_-]+)''::text'))[1],\n CASE WHEN indexdef ~* '\\(datetime desc, end_datetime\\)' THEN 'datetime' ELSE NULL END\n ) AS field,\n pg_table_size(i.indexname::text) as index_size,\n pg_size_pretty(pg_table_size(i.indexname::text)) as index_size_pretty\nFROM\n pg_indexes i\nWHERE i.schemaname='pgstac' and i.tablename ~ '_items_' AND indexdef !~* ' only ';\n\nDROP VIEW IF EXISTS pgstac_index_stats;\nCREATE VIEW pgstac_indexes_stats AS\nSELECT\n i.schemaname,\n i.tablename,\n i.indexname,\n indexdef,\n COALESCE(\n (regexp_match(indexdef, '\\(([a-zA-Z]+)\\)'))[1],\n (regexp_match(indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_]+)''::text'))[1],\n CASE WHEN indexdef ~* '\\(datetime desc, end_datetime\\)' THEN 'datetime_end_datetime' ELSE NULL END\n ) AS field,\n pg_table_size(i.indexname::text) as index_size,\n pg_size_pretty(pg_table_size(i.indexname::text)) as index_size_pretty,\n n_distinct,\n most_common_vals::text::text[],\n most_common_freqs::text::text[],\n histogram_bounds::text::text[],\n correlation\nFROM\n pg_indexes i\n LEFT JOIN pg_stats s ON (s.tablename = i.indexname)\nWHERE i.schemaname='pgstac' and i.tablename ~ '_items_';\n\nCREATE OR REPLACE FUNCTION queryable_indexes(\n IN treeroot text DEFAULT 'items',\n IN changes boolean DEFAULT FALSE,\n OUT collection text,\n OUT partition text,\n OUT field text,\n OUT indexname text,\n OUT existing_idx text,\n OUT queryable_idx text\n) RETURNS SETOF RECORD AS $$\nWITH p AS (\n SELECT\n relid::text as partition,\n replace(replace(\n CASE\n WHEN parentrelid::regclass::text='items' THEN pg_get_expr(c.relpartbound, c.oid)\n ELSE pg_get_expr(parent.relpartbound, parent.oid)\n END,\n 'FOR VALUES IN (''',''), ''')',\n ''\n ) AS collection\n FROM pg_partition_tree(treeroot)\n JOIN pg_class c ON (relid::regclass = c.oid)\n JOIN pg_class parent ON (parentrelid::regclass = parent.oid AND isleaf)\n ), i AS (\n SELECT\n partition,\n indexname,\n regexp_replace(btrim(replace(replace(indexdef, indexname, ''),'pgstac.',''),' \\t\\n'), '[ ]+', ' ', 'g') as iidx,\n COALESCE(\n (regexp_match(indexdef, '\\(([a-zA-Z]+)\\)'))[1],\n (regexp_match(indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_-]+)''::text'))[1],\n CASE WHEN indexdef ~* '\\(datetime desc, end_datetime\\)' THEN 'datetime' ELSE NULL END\n ) AS field\n FROM\n pg_indexes\n JOIN p ON (tablename=partition)\n ), q AS (\n SELECT\n name AS field,\n collection,\n partition,\n format(indexdef(queryables), partition) as qidx\n FROM queryables, unnest_collection(queryables.collection_ids) collection\n JOIN p USING (collection)\n WHERE property_index_type IS NOT NULL OR name IN ('datetime','geometry','id')\n )\n SELECT\n collection,\n partition,\n field,\n indexname,\n iidx as existing_idx,\n qidx as queryable_idx\n FROM i FULL JOIN q USING (field, partition)\n WHERE CASE WHEN changes THEN lower(iidx) IS DISTINCT FROM lower(qidx) ELSE TRUE END;\n;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION maintain_index(\n indexname text,\n queryable_idx text,\n dropindexes boolean DEFAULT FALSE,\n rebuildindexes boolean DEFAULT FALSE,\n idxconcurrently boolean DEFAULT FALSE\n) RETURNS VOID AS $$\nDECLARE\nBEGIN\n IF indexname IS NOT NULL THEN\n IF dropindexes OR queryable_idx IS NOT NULL THEN\n EXECUTE format('DROP INDEX IF EXISTS %I;', indexname);\n ELSIF rebuildindexes THEN\n IF idxconcurrently THEN\n EXECUTE format('REINDEX INDEX CONCURRENTLY %I;', indexname);\n ELSE\n EXECUTE format('REINDEX INDEX CONCURRENTLY %I;', indexname);\n END IF;\n END IF;\n END IF;\n IF queryable_idx IS NOT NULL THEN\n IF idxconcurrently THEN\n EXECUTE replace(queryable_idx, 'INDEX', 'INDEX CONCURRENTLY');\n ELSE EXECUTE queryable_idx;\n END IF;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\n\nset check_function_bodies to off;\nCREATE OR REPLACE FUNCTION maintain_partition_queries(\n part text DEFAULT 'items',\n dropindexes boolean DEFAULT FALSE,\n rebuildindexes boolean DEFAULT FALSE,\n idxconcurrently boolean DEFAULT FALSE\n) RETURNS SETOF text AS $$\nDECLARE\n rec record;\n q text;\nBEGIN\n FOR rec IN (\n SELECT * FROM queryable_indexes(part,true)\n ) LOOP\n q := format(\n 'SELECT maintain_index(\n %L,%L,%L,%L,%L\n );',\n rec.indexname,\n rec.queryable_idx,\n dropindexes,\n rebuildindexes,\n idxconcurrently\n );\n RAISE NOTICE 'Q: %s', q;\n RETURN NEXT q;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION maintain_partitions(\n part text DEFAULT 'items',\n dropindexes boolean DEFAULT FALSE,\n rebuildindexes boolean DEFAULT FALSE\n) RETURNS VOID AS $$\n WITH t AS (\n SELECT run_or_queue(q) FROM maintain_partition_queries(part, dropindexes, rebuildindexes) q\n ) SELECT count(*) FROM t;\n$$ LANGUAGE SQL;\n\n\nCREATE OR REPLACE FUNCTION queryables_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\nBEGIN\n PERFORM maintain_partitions();\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER queryables_trigger AFTER INSERT OR UPDATE ON queryables\nFOR EACH STATEMENT EXECUTE PROCEDURE queryables_trigger_func();\n\n\nCREATE OR REPLACE FUNCTION get_queryables(_collection_ids text[] DEFAULT NULL) RETURNS jsonb AS $$\nDECLARE\nBEGIN\n -- Build up queryables if the input contains valid collection ids or is empty\n IF EXISTS (\n SELECT 1 FROM collections\n WHERE\n _collection_ids IS NULL\n OR cardinality(_collection_ids) = 0\n OR id = ANY(_collection_ids)\n )\n THEN\n RETURN (\n WITH base AS (\n SELECT\n unnest(collection_ids) as collection_id,\n name,\n coalesce(definition, '{\"type\":\"string\"}'::jsonb) as definition\n FROM queryables\n WHERE\n _collection_ids IS NULL OR\n _collection_ids = '{}'::text[] OR\n _collection_ids && collection_ids\n UNION ALL\n SELECT null, name, coalesce(definition, '{\"type\":\"string\"}'::jsonb) as definition\n FROM queryables WHERE collection_ids IS NULL OR collection_ids = '{}'::text[]\n ), g AS (\n SELECT\n name,\n first_notnull(definition) as definition,\n jsonb_array_unique_merge(definition->'enum') as enum,\n jsonb_min(definition->'minimum') as minimum,\n jsonb_min(definition->'maxiumn') as maximum\n FROM base\n GROUP BY 1\n )\n SELECT\n jsonb_build_object(\n '$schema', 'http://json-schema.org/draft-07/schema#',\n '$id', '',\n 'type', 'object',\n 'title', 'STAC Queryables.',\n 'properties', jsonb_object_agg(\n name,\n definition\n ||\n jsonb_strip_nulls(jsonb_build_object(\n 'enum', enum,\n 'minimum', minimum,\n 'maximum', maximum\n ))\n ),\n 'additionalProperties', pgstac.additional_properties()\n )\n FROM g\n );\n ELSE\n RETURN NULL;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\nCREATE OR REPLACE FUNCTION get_queryables(_collection text DEFAULT NULL) RETURNS jsonb AS $$\n SELECT\n CASE\n WHEN _collection IS NULL THEN get_queryables(NULL::text[])\n ELSE get_queryables(ARRAY[_collection])\n END\n ;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_queryables() RETURNS jsonb AS $$\n SELECT get_queryables(NULL::text[]);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION schema_qualify_refs(url text, j jsonb) returns jsonb as $$\n SELECT regexp_replace(j::text, '\"\\$ref\": \"#', concat('\"$ref\": \"', url, '#'), 'g')::jsonb;\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE VIEW stac_extension_queryables AS\nSELECT DISTINCT key as name, schema_qualify_refs(e.url, j.value) as definition FROM stac_extensions e, jsonb_each(e.content->'definitions'->'fields'->'properties') j;\n\n\nCREATE OR REPLACE FUNCTION missing_queryables(_collection text, _tablesample float DEFAULT 5, minrows float DEFAULT 10) RETURNS TABLE(collection text, name text, definition jsonb, property_wrapper text) AS $$\nDECLARE\n q text;\n _partition text;\n explain_json json;\n psize float;\n estrows float;\nBEGIN\n SELECT format('_items_%s', key) INTO _partition FROM collections WHERE id=_collection;\n\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM %I;', _partition)\n INTO explain_json;\n psize := explain_json->0->'Plan'->'Plan Rows';\n estrows := _tablesample * .01 * psize;\n IF estrows < minrows THEN\n _tablesample := least(100,greatest(_tablesample, (estrows / psize) / 100));\n RAISE NOTICE '%', (psize / estrows) / 100;\n END IF;\n RAISE NOTICE 'Using tablesample % to find missing queryables from % % that has ~% rows estrows: %', _tablesample, _collection, _partition, psize, estrows;\n\n q := format(\n $q$\n WITH q AS (\n SELECT * FROM queryables\n WHERE\n collection_ids IS NULL\n OR %L = ANY(collection_ids)\n ), t AS (\n SELECT\n content->'properties' AS properties\n FROM\n %I\n TABLESAMPLE SYSTEM(%L)\n ), p AS (\n SELECT DISTINCT ON (key)\n key,\n value,\n s.definition\n FROM t\n JOIN LATERAL jsonb_each(properties) ON TRUE\n LEFT JOIN q ON (q.name=key)\n LEFT JOIN stac_extension_queryables s ON (s.name=key)\n WHERE q.definition IS NULL\n )\n SELECT\n %L,\n key,\n COALESCE(definition, jsonb_build_object('type',jsonb_typeof(value))) as definition,\n CASE\n WHEN definition->>'type' = 'integer' THEN 'to_int'\n WHEN COALESCE(definition->>'type', jsonb_typeof(value)) = 'number' THEN 'to_float'\n WHEN COALESCE(definition->>'type', jsonb_typeof(value)) = 'array' THEN 'to_text_array'\n ELSE 'to_text'\n END\n FROM p;\n $q$,\n _collection,\n _partition,\n _tablesample,\n _collection\n );\n RETURN QUERY EXECUTE q;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION missing_queryables(_tablesample float DEFAULT 5) RETURNS TABLE(collection_ids text[], name text, definition jsonb, property_wrapper text) AS $$\n SELECT\n array_agg(collection),\n name,\n definition,\n property_wrapper\n FROM\n collections\n JOIN LATERAL\n missing_queryables(id, _tablesample) c\n ON TRUE\n GROUP BY\n 2,3,4\n ORDER BY 2,1\n ;\n$$ LANGUAGE SQL;\nCREATE OR REPLACE FUNCTION parse_dtrange(\n _indate jsonb,\n relative_base timestamptz DEFAULT date_trunc('hour', CURRENT_TIMESTAMP)\n) RETURNS tstzrange AS $$\nDECLARE\n timestrs text[];\n s timestamptz;\n e timestamptz;\nBEGIN\n timestrs :=\n CASE\n WHEN _indate ? 'timestamp' THEN\n ARRAY[_indate->>'timestamp']\n WHEN _indate ? 'interval' THEN\n to_text_array(_indate->'interval')\n WHEN jsonb_typeof(_indate) = 'array' THEN\n to_text_array(_indate)\n ELSE\n regexp_split_to_array(\n _indate->>0,\n '/'\n )\n END;\n RAISE NOTICE 'TIMESTRS %', timestrs;\n IF cardinality(timestrs) = 1 THEN\n IF timestrs[1] ILIKE 'P%' THEN\n RETURN tstzrange(relative_base - upper(timestrs[1])::interval, relative_base, '[)');\n END IF;\n s := timestrs[1]::timestamptz;\n RETURN tstzrange(s, s, '[]');\n END IF;\n\n IF cardinality(timestrs) != 2 THEN\n RAISE EXCEPTION 'Timestamp cannot have more than 2 values';\n END IF;\n\n IF timestrs[1] = '..' OR timestrs[1] = '' THEN\n s := '-infinity'::timestamptz;\n e := timestrs[2]::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] = '..' OR timestrs[2] = '' THEN\n s := timestrs[1]::timestamptz;\n e := 'infinity'::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[1] ILIKE 'P%' AND timestrs[2] NOT ILIKE 'P%' THEN\n e := timestrs[2]::timestamptz;\n s := e - upper(timestrs[1])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] ILIKE 'P%' AND timestrs[1] NOT ILIKE 'P%' THEN\n s := timestrs[1]::timestamptz;\n e := s + upper(timestrs[2])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n s := timestrs[1]::timestamptz;\n e := timestrs[2]::timestamptz;\n\n RETURN tstzrange(s,e,'[)');\n\n RETURN NULL;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT PARALLEL SAFE SET TIME ZONE 'UTC';\n\nCREATE OR REPLACE FUNCTION parse_dtrange(\n _indate text,\n relative_base timestamptz DEFAULT CURRENT_TIMESTAMP\n) RETURNS tstzrange AS $$\n SELECT parse_dtrange(to_jsonb(_indate), relative_base);\n$$ LANGUAGE SQL STABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION temporal_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\n ll text := 'datetime';\n lh text := 'end_datetime';\n rrange tstzrange;\n rl text;\n rh text;\n outq text;\nBEGIN\n rrange := parse_dtrange(args->1);\n RAISE NOTICE 'Constructing temporal query OP: %, ARGS: %, RRANGE: %', op, args, rrange;\n op := lower(op);\n rl := format('%L::timestamptz', lower(rrange));\n rh := format('%L::timestamptz', upper(rrange));\n outq := CASE op\n WHEN 't_before' THEN 'lh < rl'\n WHEN 't_after' THEN 'll > rh'\n WHEN 't_meets' THEN 'lh = rl'\n WHEN 't_metby' THEN 'll = rh'\n WHEN 't_overlaps' THEN 'll < rl AND rl < lh < rh'\n WHEN 't_overlappedby' THEN 'rl < ll < rh AND lh > rh'\n WHEN 't_starts' THEN 'll = rl AND lh < rh'\n WHEN 't_startedby' THEN 'll = rl AND lh > rh'\n WHEN 't_during' THEN 'll > rl AND lh < rh'\n WHEN 't_contains' THEN 'll < rl AND lh > rh'\n WHEN 't_finishes' THEN 'll > rl AND lh = rh'\n WHEN 't_finishedby' THEN 'll < rl AND lh = rh'\n WHEN 't_equals' THEN 'll = rl AND lh = rh'\n WHEN 't_disjoint' THEN 'NOT (ll <= rh AND lh >= rl)'\n WHEN 't_intersects' THEN 'll <= rh AND lh >= rl'\n WHEN 'anyinteracts' THEN 'll <= rh AND lh >= rl'\n END;\n outq := regexp_replace(outq, '\\mll\\M', ll);\n outq := regexp_replace(outq, '\\mlh\\M', lh);\n outq := regexp_replace(outq, '\\mrl\\M', rl);\n outq := regexp_replace(outq, '\\mrh\\M', rh);\n outq := format('(%s)', outq);\n RETURN outq;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\n\n\nCREATE OR REPLACE FUNCTION spatial_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\n geom text;\n j jsonb := args->1;\nBEGIN\n op := lower(op);\n RAISE NOTICE 'Constructing spatial query OP: %, ARGS: %', op, args;\n IF op NOT IN ('s_equals','s_disjoint','s_touches','s_within','s_overlaps','s_crosses','s_intersects','intersects','s_contains') THEN\n RAISE EXCEPTION 'Spatial Operator % Not Supported', op;\n END IF;\n op := regexp_replace(op, '^s_', 'st_');\n IF op = 'intersects' THEN\n op := 'st_intersects';\n END IF;\n -- Convert geometry to WKB string\n IF j ? 'type' AND j ? 'coordinates' THEN\n geom := st_geomfromgeojson(j)::text;\n ELSIF jsonb_typeof(j) = 'array' THEN\n geom := bbox_geom(j)::text;\n END IF;\n\n RETURN format('%s(geometry, %L::geometry)', op, geom);\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION query_to_cql2(q jsonb) RETURNS jsonb AS $$\n-- Translates anything passed in through the deprecated \"query\" into equivalent CQL2\nWITH t AS (\n SELECT key as property, value as ops\n FROM jsonb_each(q)\n), t2 AS (\n SELECT property, (jsonb_each(ops)).*\n FROM t WHERE jsonb_typeof(ops) = 'object'\n UNION ALL\n SELECT property, 'eq', ops\n FROM t WHERE jsonb_typeof(ops) != 'object'\n)\nSELECT\n jsonb_strip_nulls(jsonb_build_object(\n 'op', 'and',\n 'args', jsonb_agg(\n jsonb_build_object(\n 'op', key,\n 'args', jsonb_build_array(\n jsonb_build_object('property',property),\n value\n )\n )\n )\n )\n) as qcql FROM t2\n;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\nCREATE OR REPLACE FUNCTION cql1_to_cql2(j jsonb) RETURNS jsonb AS $$\nDECLARE\n args jsonb;\n ret jsonb;\nBEGIN\n RAISE NOTICE 'CQL1_TO_CQL2: %', j;\n IF j ? 'filter' THEN\n RETURN cql1_to_cql2(j->'filter');\n END IF;\n IF j ? 'property' THEN\n RETURN j;\n END IF;\n IF jsonb_typeof(j) = 'array' THEN\n SELECT jsonb_agg(cql1_to_cql2(el)) INTO args FROM jsonb_array_elements(j) el;\n RETURN args;\n END IF;\n IF jsonb_typeof(j) = 'number' THEN\n RETURN j;\n END IF;\n IF jsonb_typeof(j) = 'string' THEN\n RETURN j;\n END IF;\n\n IF jsonb_typeof(j) = 'object' THEN\n SELECT jsonb_build_object(\n 'op', key,\n 'args', cql1_to_cql2(value)\n ) INTO ret\n FROM jsonb_each(j)\n WHERE j IS NOT NULL;\n RETURN ret;\n END IF;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE STRICT;\n\nCREATE TABLE cql2_ops (\n op text PRIMARY KEY,\n template text,\n types text[]\n);\nINSERT INTO cql2_ops (op, template, types) VALUES\n ('eq', '%s = %s', NULL),\n ('neq', '%s != %s', NULL),\n ('ne', '%s != %s', NULL),\n ('!=', '%s != %s', NULL),\n ('<>', '%s != %s', NULL),\n ('lt', '%s < %s', NULL),\n ('lte', '%s <= %s', NULL),\n ('gt', '%s > %s', NULL),\n ('gte', '%s >= %s', NULL),\n ('le', '%s <= %s', NULL),\n ('ge', '%s >= %s', NULL),\n ('=', '%s = %s', NULL),\n ('<', '%s < %s', NULL),\n ('<=', '%s <= %s', NULL),\n ('>', '%s > %s', NULL),\n ('>=', '%s >= %s', NULL),\n ('like', '%s LIKE %s', NULL),\n ('ilike', '%s ILIKE %s', NULL),\n ('+', '%s + %s', NULL),\n ('-', '%s - %s', NULL),\n ('*', '%s * %s', NULL),\n ('/', '%s / %s', NULL),\n ('not', 'NOT (%s)', NULL),\n ('between', '%s BETWEEN %s AND %s', NULL),\n ('isnull', '%s IS NULL', NULL),\n ('upper', 'upper(%s)', NULL),\n ('lower', 'lower(%s)', NULL),\n ('casei', 'upper(%s)', NULL),\n ('accenti', 'unaccent(%s)', NULL)\nON CONFLICT (op) DO UPDATE\n SET\n template = EXCLUDED.template\n;\n\n\nCREATE OR REPLACE FUNCTION cql2_query(j jsonb, wrapper text DEFAULT NULL) RETURNS text AS $$\n#variable_conflict use_variable\nDECLARE\n args jsonb := j->'args';\n arg jsonb;\n op text := lower(j->>'op');\n cql2op RECORD;\n literal text;\n _wrapper text;\n leftarg text;\n rightarg text;\n prop text;\n extra_props bool := pgstac.additional_properties();\nBEGIN\n IF j IS NULL OR (op IS NOT NULL AND args IS NULL) THEN\n RETURN NULL;\n END IF;\n RAISE NOTICE 'CQL2_QUERY: %', j;\n\n -- check if all properties are represented in the queryables\n IF NOT extra_props THEN\n FOR prop IN\n SELECT DISTINCT p->>0\n FROM jsonb_path_query(j, 'strict $.**.property') p\n WHERE p->>0 NOT IN ('id', 'datetime', 'geometry', 'end_datetime', 'collection')\n LOOP\n IF (queryable(prop)).nulled_wrapper IS NULL THEN\n RAISE EXCEPTION 'Term % is not found in queryables.', prop;\n END IF;\n END LOOP;\n END IF;\n\n IF j ? 'filter' THEN\n RETURN cql2_query(j->'filter');\n END IF;\n\n IF j ? 'upper' THEN\n RETURN cql2_query(jsonb_build_object('op', 'upper', 'args', j->'upper'));\n END IF;\n\n IF j ? 'lower' THEN\n RETURN cql2_query(jsonb_build_object('op', 'lower', 'args', j->'lower'));\n END IF;\n\n -- Temporal Query\n IF op ilike 't_%' or op = 'anyinteracts' THEN\n RETURN temporal_op_query(op, args);\n END IF;\n\n -- If property is a timestamp convert it to text to use with\n -- general operators\n IF j ? 'timestamp' THEN\n RETURN format('%L::timestamptz', to_tstz(j->'timestamp'));\n END IF;\n IF j ? 'interval' THEN\n RAISE EXCEPTION 'Please use temporal operators when using intervals.';\n RETURN NONE;\n END IF;\n\n -- Spatial Query\n IF op ilike 's_%' or op = 'intersects' THEN\n RETURN spatial_op_query(op, args);\n END IF;\n\n IF op IN ('a_equals','a_contains','a_contained_by','a_overlaps') THEN\n IF args->0 ? 'property' THEN\n leftarg := format('to_text_array(%s)', (queryable(args->0->>'property')).path);\n END IF;\n IF args->1 ? 'property' THEN\n rightarg := format('to_text_array(%s)', (queryable(args->1->>'property')).path);\n END IF;\n RETURN FORMAT(\n '%s %s %s',\n COALESCE(leftarg, quote_literal(to_text_array(args->0))),\n CASE op\n WHEN 'a_equals' THEN '='\n WHEN 'a_contains' THEN '@>'\n WHEN 'a_contained_by' THEN '<@'\n WHEN 'a_overlaps' THEN '&&'\n END,\n COALESCE(rightarg, quote_literal(to_text_array(args->1)))\n );\n END IF;\n\n IF op = 'in' THEN\n RAISE NOTICE 'IN : % % %', args, jsonb_build_array(args->0), args->1;\n args := jsonb_build_array(args->0) || (args->1);\n RAISE NOTICE 'IN2 : %', args;\n END IF;\n\n\n\n IF op = 'between' THEN\n args = jsonb_build_array(\n args->0,\n args->1,\n args->2\n );\n END IF;\n\n -- Make sure that args is an array and run cql2_query on\n -- each element of the array\n RAISE NOTICE 'ARGS PRE: %', args;\n IF j ? 'args' THEN\n IF jsonb_typeof(args) != 'array' THEN\n args := jsonb_build_array(args);\n END IF;\n\n IF jsonb_path_exists(args, '$[*] ? (@.property == \"id\" || @.property == \"datetime\" || @.property == \"end_datetime\" || @.property == \"collection\")') THEN\n wrapper := NULL;\n ELSE\n -- if any of the arguments are a property, try to get the property_wrapper\n FOR arg IN SELECT jsonb_path_query(args, '$[*] ? (@.property != null)') LOOP\n RAISE NOTICE 'Arg: %', arg;\n wrapper := (queryable(arg->>'property')).nulled_wrapper;\n RAISE NOTICE 'Property: %, Wrapper: %', arg, wrapper;\n IF wrapper IS NOT NULL THEN\n EXIT;\n END IF;\n END LOOP;\n\n -- if the property was not in queryables, see if any args were numbers\n IF\n wrapper IS NULL\n AND jsonb_path_exists(args, '$[*] ? (@.type()==\"number\")')\n THEN\n wrapper := 'to_float';\n END IF;\n wrapper := coalesce(wrapper, 'to_text');\n END IF;\n\n SELECT jsonb_agg(cql2_query(a, wrapper))\n INTO args\n FROM jsonb_array_elements(args) a;\n END IF;\n RAISE NOTICE 'ARGS: %', args;\n\n IF op IN ('and', 'or') THEN\n RETURN\n format(\n '(%s)',\n array_to_string(to_text_array(args), format(' %s ', upper(op)))\n );\n END IF;\n\n IF op = 'in' THEN\n RAISE NOTICE 'IN -- % %', args->0, to_text(args->0);\n RETURN format(\n '%s IN (%s)',\n to_text(args->0),\n array_to_string((to_text_array(args))[2:], ',')\n );\n END IF;\n\n -- Look up template from cql2_ops\n IF j ? 'op' THEN\n SELECT * INTO cql2op FROM cql2_ops WHERE cql2_ops.op ilike op;\n IF FOUND THEN\n -- If specific index set in queryables for a property cast other arguments to that type\n\n RETURN format(\n cql2op.template,\n VARIADIC (to_text_array(args))\n );\n ELSE\n RAISE EXCEPTION 'Operator % Not Supported.', op;\n END IF;\n END IF;\n\n\n IF wrapper IS NOT NULL THEN\n RAISE NOTICE 'Wrapping % with %', j, wrapper;\n IF j ? 'property' THEN\n RETURN format('%I(%s)', wrapper, (queryable(j->>'property')).path);\n ELSE\n RETURN format('%I(%L)', wrapper, j);\n END IF;\n ELSIF j ? 'property' THEN\n RETURN quote_ident(j->>'property');\n END IF;\n\n RETURN quote_literal(to_text(j));\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION paging_dtrange(\n j jsonb\n) RETURNS tstzrange AS $$\nDECLARE\n op text;\n filter jsonb := j->'filter';\n dtrange tstzrange := tstzrange('-infinity'::timestamptz,'infinity'::timestamptz);\n sdate timestamptz := '-infinity'::timestamptz;\n edate timestamptz := 'infinity'::timestamptz;\n jpitem jsonb;\nBEGIN\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n END IF;\n IF NOT (filter @? '$.**.op ? (@ == \"or\" || @ == \"not\")') THEN\n FOR jpitem IN SELECT j FROM jsonb_path_query(filter,'strict $.** ? (@.args[*].property == \"datetime\")'::jsonpath) j LOOP\n op := lower(jpitem->>'op');\n dtrange := parse_dtrange(jpitem->'args'->1);\n IF op IN ('<=', 'lt', 'lte', '<', 'le', 't_before') THEN\n sdate := greatest(sdate,'-infinity');\n edate := least(edate, upper(dtrange));\n ELSIF op IN ('>=', '>', 'gt', 'gte', 'ge', 't_after') THEN\n edate := least(edate, 'infinity');\n sdate := greatest(sdate, lower(dtrange));\n ELSIF op IN ('=', 'eq') THEN\n edate := least(edate, upper(dtrange));\n sdate := greatest(sdate, lower(dtrange));\n END IF;\n RAISE NOTICE '2 OP: %, ARGS: %, DTRANGE: %, SDATE: %, EDATE: %', op, jpitem->'args'->1, dtrange, sdate, edate;\n END LOOP;\n END IF;\n IF sdate > edate THEN\n RETURN 'empty'::tstzrange;\n END IF;\n RETURN tstzrange(sdate,edate, '[]');\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT SET TIME ZONE 'UTC';\n\nCREATE OR REPLACE FUNCTION paging_collections(\n IN j jsonb\n) RETURNS text[] AS $$\nDECLARE\n filter jsonb := j->'filter';\n jpitem jsonb;\n op text;\n args jsonb;\n arg jsonb;\n collections text[];\nBEGIN\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n END IF;\n IF NOT (filter @? '$.**.op ? (@ == \"or\" || @ == \"not\")') THEN\n FOR jpitem IN SELECT j FROM jsonb_path_query(filter,'strict $.** ? (@.args[*].property == \"collection\")'::jsonpath) j LOOP\n RAISE NOTICE 'JPITEM: %', jpitem;\n op := jpitem->>'op';\n args := jpitem->'args';\n IF op IN ('=', 'eq', 'in') THEN\n FOR arg IN SELECT a FROM jsonb_array_elements(args) a LOOP\n IF jsonb_typeof(arg) IN ('string', 'array') THEN\n RAISE NOTICE 'arg: %, collections: %', arg, collections;\n IF collections IS NULL OR collections = '{}'::text[] THEN\n collections := to_text_array(arg);\n ELSE\n collections := array_intersection(collections, to_text_array(arg));\n END IF;\n END IF;\n END LOOP;\n END IF;\n END LOOP;\n END IF;\n IF collections = '{}'::text[] THEN\n RETURN NULL;\n END IF;\n RETURN collections;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\nCREATE TABLE items (\n id text NOT NULL,\n geometry geometry NOT NULL,\n collection text NOT NULL,\n datetime timestamptz NOT NULL,\n end_datetime timestamptz NOT NULL,\n content JSONB NOT NULL,\n private jsonb\n)\nPARTITION BY LIST (collection)\n;\n\nCREATE INDEX \"datetime_idx\" ON items USING BTREE (datetime DESC, end_datetime ASC);\nCREATE INDEX \"geometry_idx\" ON items USING GIST (geometry);\n\nCREATE STATISTICS datetime_stats (dependencies) on datetime, end_datetime from items;\n\nALTER TABLE items ADD CONSTRAINT items_collections_fk FOREIGN KEY (collection) REFERENCES collections(id) ON DELETE CASCADE DEFERRABLE;\n\nCREATE OR REPLACE FUNCTION partition_after_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p text;\n t timestamptz := clock_timestamp();\nBEGIN\n RAISE NOTICE 'Updating partition stats %', t;\n FOR p IN SELECT DISTINCT partition\n FROM newdata n JOIN partition_sys_meta p\n ON (n.collection=p.collection AND n.datetime <@ p.partition_dtrange)\n LOOP\n PERFORM run_or_queue(format('SELECT update_partition_stats(%L, %L);', p, true));\n END LOOP;\n IF TG_OP IN ('DELETE','UPDATE') THEN\n DELETE FROM format_item_cache c USING newdata n WHERE c.collection = n.collection AND c.id = n.id;\n END IF;\n RAISE NOTICE 't: % %', t, clock_timestamp() - t;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE TRIGGER items_after_insert_trigger\nAFTER INSERT ON items\nREFERENCING NEW TABLE AS newdata\nFOR EACH STATEMENT\nEXECUTE FUNCTION partition_after_triggerfunc();\n\nCREATE TRIGGER items_after_update_trigger\nAFTER DELETE ON items\nREFERENCING OLD TABLE AS newdata\nFOR EACH STATEMENT\nEXECUTE FUNCTION partition_after_triggerfunc();\n\nCREATE TRIGGER items_after_delete_trigger\nAFTER UPDATE ON items\nREFERENCING NEW TABLE AS newdata\nFOR EACH STATEMENT\nEXECUTE FUNCTION partition_after_triggerfunc();\n\n\nCREATE OR REPLACE FUNCTION content_slim(_item jsonb) RETURNS jsonb AS $$\n SELECT strip_jsonb(_item - '{id,geometry,collection,type}'::text[], collection_base_item(_item->>'collection')) - '{id,geometry,collection,type}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_dehydrate(content jsonb) RETURNS items AS $$\n SELECT\n content->>'id' as id,\n stac_geom(content) as geometry,\n content->>'collection' as collection,\n stac_datetime(content) as datetime,\n stac_end_datetime(content) as end_datetime,\n content_slim(content) as content,\n null::jsonb as private\n ;\n$$ LANGUAGE SQL STABLE;\n\nCREATE OR REPLACE FUNCTION include_field(f text, fields jsonb DEFAULT '{}'::jsonb) RETURNS boolean AS $$\nDECLARE\n includes jsonb := fields->'include';\n excludes jsonb := fields->'exclude';\nBEGIN\n IF f IS NULL THEN\n RETURN NULL;\n END IF;\n\n\n IF\n jsonb_typeof(excludes) = 'array'\n AND jsonb_array_length(excludes)>0\n AND excludes ? f\n THEN\n RETURN FALSE;\n END IF;\n\n IF\n (\n jsonb_typeof(includes) = 'array'\n AND jsonb_array_length(includes) > 0\n AND includes ? f\n ) OR\n (\n includes IS NULL\n OR jsonb_typeof(includes) = 'null'\n OR jsonb_array_length(includes) = 0\n )\n THEN\n RETURN TRUE;\n END IF;\n\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nDROP FUNCTION IF EXISTS content_hydrate(jsonb, jsonb, jsonb);\nCREATE OR REPLACE FUNCTION content_hydrate(\n _item jsonb,\n _base_item jsonb,\n fields jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\n SELECT merge_jsonb(\n jsonb_fields(_item, fields),\n jsonb_fields(_base_item, fields)\n );\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION content_hydrate(_item items, _collection collections, fields jsonb DEFAULT '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\n content jsonb;\n base_item jsonb := _collection.base_item;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry, 20)::jsonb;\n END IF;\n output := content_hydrate(\n jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'collection', _item.collection,\n 'type', 'Feature'\n ) || _item.content,\n _collection.base_item,\n fields\n );\n\n RETURN output;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_nonhydrated(\n _item items,\n fields jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry, 20)::jsonb;\n END IF;\n output := jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'collection', _item.collection,\n 'type', 'Feature'\n ) || _item.content;\n RETURN output;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_hydrate(_item items, fields jsonb DEFAULT '{}'::jsonb) RETURNS jsonb AS $$\n SELECT content_hydrate(\n _item,\n (SELECT c FROM collections c WHERE id=_item.collection LIMIT 1),\n fields\n );\n$$ LANGUAGE SQL STABLE;\n\n\nCREATE UNLOGGED TABLE items_staging (\n content JSONB NOT NULL\n);\nCREATE UNLOGGED TABLE items_staging_ignore (\n content JSONB NOT NULL\n);\nCREATE UNLOGGED TABLE items_staging_upsert (\n content JSONB NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION items_staging_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p record;\n _partitions text[];\n part text;\n ts timestamptz := clock_timestamp();\n nrows int;\nBEGIN\n RAISE NOTICE 'Creating Partitions. %', clock_timestamp() - ts;\n\n FOR part IN WITH t AS (\n SELECT\n n.content->>'collection' as collection,\n stac_daterange(n.content->'properties') as dtr,\n partition_trunc\n FROM newdata n JOIN collections ON (n.content->>'collection'=collections.id)\n ), p AS (\n SELECT\n collection,\n COALESCE(date_trunc(partition_trunc::text, lower(dtr)),'-infinity') as d,\n tstzrange(min(lower(dtr)),max(lower(dtr)),'[]') as dtrange,\n tstzrange(min(upper(dtr)),max(upper(dtr)),'[]') as edtrange\n FROM t\n GROUP BY 1,2\n ) SELECT check_partition(collection, dtrange, edtrange) FROM p LOOP\n RAISE NOTICE 'Partition %', part;\n END LOOP;\n\n RAISE NOTICE 'Creating temp table with data to be added. %', clock_timestamp() - ts;\n DROP TABLE IF EXISTS tmpdata;\n CREATE TEMP TABLE tmpdata ON COMMIT DROP AS\n SELECT\n (content_dehydrate(content)).*\n FROM newdata;\n GET DIAGNOSTICS nrows = ROW_COUNT;\n RAISE NOTICE 'Added % rows to tmpdata. %', nrows, clock_timestamp() - ts;\n\n RAISE NOTICE 'Doing the insert. %', clock_timestamp() - ts;\n IF TG_TABLE_NAME = 'items_staging' THEN\n INSERT INTO items\n SELECT * FROM tmpdata;\n GET DIAGNOSTICS nrows = ROW_COUNT;\n RAISE NOTICE 'Inserted % rows to items. %', nrows, clock_timestamp() - ts;\n ELSIF TG_TABLE_NAME = 'items_staging_ignore' THEN\n INSERT INTO items\n SELECT * FROM tmpdata\n ON CONFLICT DO NOTHING;\n GET DIAGNOSTICS nrows = ROW_COUNT;\n RAISE NOTICE 'Inserted % rows to items. %', nrows, clock_timestamp() - ts;\n ELSIF TG_TABLE_NAME = 'items_staging_upsert' THEN\n DELETE FROM items i USING tmpdata s\n WHERE\n i.id = s.id\n AND i.collection = s.collection\n AND i IS DISTINCT FROM s\n ;\n GET DIAGNOSTICS nrows = ROW_COUNT;\n RAISE NOTICE 'Deleted % rows from items. %', nrows, clock_timestamp() - ts;\n INSERT INTO items AS t\n SELECT * FROM tmpdata\n ON CONFLICT DO NOTHING;\n GET DIAGNOSTICS nrows = ROW_COUNT;\n RAISE NOTICE 'Inserted % rows to items. %', nrows, clock_timestamp() - ts;\n END IF;\n\n RAISE NOTICE 'Deleting data from staging table. %', clock_timestamp() - ts;\n DELETE FROM items_staging;\n RAISE NOTICE 'Done. %', clock_timestamp() - ts;\n\n RETURN NULL;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER items_staging_insert_trigger AFTER INSERT ON items_staging REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\nCREATE TRIGGER items_staging_insert_ignore_trigger AFTER INSERT ON items_staging_ignore REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\nCREATE TRIGGER items_staging_insert_upsert_trigger AFTER INSERT ON items_staging_upsert REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\n\nCREATE OR REPLACE FUNCTION item_by_id(_id text, _collection text DEFAULT NULL) RETURNS items AS\n$$\nDECLARE\n i items%ROWTYPE;\nBEGIN\n SELECT * INTO i FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection) LIMIT 1;\n RETURN i;\nEND;\n$$ LANGUAGE PLPGSQL STABLE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION get_item(_id text, _collection text DEFAULT NULL) RETURNS jsonb AS $$\n SELECT content_hydrate(items) FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection);\n$$ LANGUAGE SQL STABLE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION delete_item(_id text, _collection text DEFAULT NULL) RETURNS VOID AS $$\nDECLARE\nout items%ROWTYPE;\nBEGIN\n DELETE FROM items WHERE id = _id AND (_collection IS NULL OR collection=_collection) RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n--/*\nCREATE OR REPLACE FUNCTION create_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION update_item(content jsonb) RETURNS VOID AS $$\nDECLARE\n old items %ROWTYPE;\n out items%ROWTYPE;\nBEGIN\n PERFORM delete_item(content->>'id', content->>'collection');\n PERFORM create_item(content);\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION create_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION collection_bbox(id text) RETURNS jsonb AS $$\n SELECT (replace(replace(replace(st_extent(geometry)::text,'BOX(','[['),')',']]'),' ',','))::jsonb\n FROM items WHERE collection=$1;\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION collection_temporal_extent(id text) RETURNS jsonb AS $$\n SELECT to_jsonb(array[array[min(datetime)::text, max(datetime)::text]])\n FROM items WHERE collection=$1;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection_extents() RETURNS VOID AS $$\nUPDATE collections SET\n content = content ||\n jsonb_build_object(\n 'extent', jsonb_build_object(\n 'spatial', jsonb_build_object(\n 'bbox', collection_bbox(collections.id)\n ),\n 'temporal', jsonb_build_object(\n 'interval', collection_temporal_extent(collections.id)\n )\n )\n )\n;\n$$ LANGUAGE SQL;\nCREATE TABLE partition_stats (\n partition text PRIMARY KEY,\n dtrange tstzrange,\n edtrange tstzrange,\n spatial geometry,\n last_updated timestamptz,\n keys text[]\n) WITH (FILLFACTOR=90);\n\nCREATE INDEX partitions_range_idx ON partition_stats USING GIST(dtrange);\n\n\nCREATE OR REPLACE FUNCTION constraint_tstzrange(expr text) RETURNS tstzrange AS $$\n WITH t AS (\n SELECT regexp_matches(\n expr,\n E'\\\\(''\\([0-9 :+-]*\\)''\\\\).*\\\\(''\\([0-9 :+-]*\\)''\\\\)'\n ) AS m\n ) SELECT tstzrange(m[1]::timestamptz, m[2]::timestamptz) FROM t\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE STRICT;\n\nCREATE OR REPLACE FUNCTION dt_constraint(coid oid, OUT dt tstzrange, OUT edt tstzrange) RETURNS RECORD AS $$\nDECLARE\n expr text := pg_get_constraintdef(coid);\n matches timestamptz[];\nBEGIN\n IF expr LIKE '%NULL%' THEN\n dt := tstzrange(null::timestamptz, null::timestamptz);\n edt := tstzrange(null::timestamptz, null::timestamptz);\n RETURN;\n END IF;\n WITH f AS (SELECT (regexp_matches(expr, E'([0-9]{4}-[0-1][0-9]-[0-3][0-9] [0-2][0-9]:[0-5][0-9]:[0-5][0-9]\\.?[0-9]*)', 'g'))[1] f)\n SELECT array_agg(f::timestamptz) INTO matches FROM f;\n IF cardinality(matches) = 4 THEN\n dt := tstzrange(matches[1], matches[2],'[]');\n edt := tstzrange(matches[3], matches[4], '[]');\n RETURN;\n ELSIF cardinality(matches) = 2 THEN\n edt := tstzrange(matches[1], matches[2],'[]');\n RETURN;\n END IF;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\nCREATE OR REPLACE VIEW partition_sys_meta AS\nSELECT\n relid::text as partition,\n replace(replace(CASE WHEN level = 1 THEN pg_get_expr(c.relpartbound, c.oid)\n ELSE pg_get_expr(parent.relpartbound, parent.oid)\n END, 'FOR VALUES IN (''',''), ''')','') AS collection,\n level,\n c.reltuples,\n c.relhastriggers,\n COALESCE(constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as partition_dtrange,\n COALESCE((dt_constraint(edt.oid)).dt, constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as constraint_dtrange,\n COALESCE((dt_constraint(edt.oid)).edt, tstzrange('-infinity', 'infinity','[]')) as constraint_edtrange\nFROM\n pg_partition_tree('items')\n JOIN pg_class c ON (relid::regclass = c.oid)\n JOIN pg_class parent ON (parentrelid::regclass = parent.oid AND isleaf)\n LEFT JOIN pg_constraint edt ON (conrelid=c.oid AND contype='c')\nWHERE isleaf\n;\n\nCREATE VIEW partitions_view AS\nSELECT\n relid::text as partition,\n replace(replace(CASE WHEN level = 1 THEN pg_get_expr(c.relpartbound, c.oid)\n ELSE pg_get_expr(parent.relpartbound, parent.oid)\n END, 'FOR VALUES IN (''',''), ''')','') AS collection,\n level,\n c.reltuples,\n c.relhastriggers,\n COALESCE(constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as partition_dtrange,\n COALESCE((dt_constraint(edt.oid)).dt, constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as constraint_dtrange,\n COALESCE((dt_constraint(edt.oid)).edt, tstzrange('-infinity', 'infinity','[]')) as constraint_edtrange,\n dtrange,\n edtrange,\n spatial,\n last_updated\nFROM\n pg_partition_tree('items')\n JOIN pg_class c ON (relid::regclass = c.oid)\n JOIN pg_class parent ON (parentrelid::regclass = parent.oid AND isleaf)\n LEFT JOIN pg_constraint edt ON (conrelid=c.oid AND contype='c')\n LEFT JOIN partition_stats ON (relid::text=partition)\nWHERE isleaf\n;\n\nCREATE MATERIALIZED VIEW partitions AS\nSELECT * FROM partitions_view;\nCREATE UNIQUE INDEX ON partitions (partition);\n\nCREATE MATERIALIZED VIEW partition_steps AS\nSELECT\n partition as name,\n date_trunc('month',lower(partition_dtrange)) as sdate,\n date_trunc('month', upper(partition_dtrange)) + '1 month'::interval as edate\n FROM partitions_view WHERE partition_dtrange IS NOT NULL AND partition_dtrange != 'empty'::tstzrange\n ORDER BY dtrange ASC\n;\n\n\nCREATE OR REPLACE FUNCTION update_partition_stats_q(_partition text, istrigger boolean default false) RETURNS VOID AS $$\nDECLARE\nBEGIN\n PERFORM run_or_queue(\n format('SELECT update_partition_stats(%L, %L);', _partition, istrigger)\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION update_partition_stats(_partition text, istrigger boolean default false) RETURNS VOID AS $$\nDECLARE\n dtrange tstzrange;\n edtrange tstzrange;\n cdtrange tstzrange;\n cedtrange tstzrange;\n extent geometry;\n collection text;\nBEGIN\n RAISE NOTICE 'Updating stats for %.', _partition;\n EXECUTE format(\n $q$\n SELECT\n tstzrange(min(datetime), max(datetime),'[]'),\n tstzrange(min(end_datetime), max(end_datetime), '[]')\n FROM %I\n $q$,\n _partition\n ) INTO dtrange, edtrange;\n extent := st_estimatedextent('pgstac', _partition, 'geometry');\n INSERT INTO partition_stats (partition, dtrange, edtrange, spatial, last_updated)\n SELECT _partition, dtrange, edtrange, extent, now()\n ON CONFLICT (partition) DO\n UPDATE SET\n dtrange=EXCLUDED.dtrange,\n edtrange=EXCLUDED.edtrange,\n spatial=EXCLUDED.spatial,\n last_updated=EXCLUDED.last_updated\n ;\n\n SELECT\n constraint_dtrange, constraint_edtrange, pv.collection\n INTO cdtrange, cedtrange, collection\n FROM partitions_view pv WHERE partition = _partition;\n REFRESH MATERIALIZED VIEW partitions;\n REFRESH MATERIALIZED VIEW partition_steps;\n\n\n RAISE NOTICE 'Checking if we need to modify constraints...';\n RAISE NOTICE 'cdtrange: % dtrange: % cedtrange: % edtrange: %',cdtrange, dtrange, cedtrange, edtrange;\n IF\n (cdtrange IS DISTINCT FROM dtrange OR edtrange IS DISTINCT FROM cedtrange)\n AND NOT istrigger\n THEN\n RAISE NOTICE 'Modifying Constraints';\n RAISE NOTICE 'Existing % %', cdtrange, cedtrange;\n RAISE NOTICE 'New % %', dtrange, edtrange;\n PERFORM drop_table_constraints(_partition);\n PERFORM create_table_constraints(_partition, dtrange, edtrange);\n REFRESH MATERIALIZED VIEW partitions;\n REFRESH MATERIALIZED VIEW partition_steps;\n END IF;\n RAISE NOTICE 'Checking if we need to update collection extents.';\n IF get_setting_bool('update_collection_extent') THEN\n RAISE NOTICE 'updating collection extent for %', collection;\n PERFORM run_or_queue(format($q$\n UPDATE collections\n SET content = jsonb_set_lax(\n content,\n '{extent}'::text[],\n collection_extent(%L, FALSE),\n true,\n 'use_json_null'\n ) WHERE id=%L\n ;\n $q$, collection, collection));\n ELSE\n RAISE NOTICE 'Not updating collection extent for %', collection;\n END IF;\n\nEND;\n$$ LANGUAGE PLPGSQL STRICT SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION partition_name( IN collection text, IN dt timestamptz, OUT partition_name text, OUT partition_range tstzrange) AS $$\nDECLARE\n c RECORD;\n parent_name text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n parent_name := format('_items_%s', c.key);\n\n\n IF c.partition_trunc = 'year' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYY'));\n ELSIF c.partition_trunc = 'month' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYYMM'));\n ELSE\n partition_name := parent_name;\n partition_range := tstzrange('-infinity'::timestamptz, 'infinity'::timestamptz, '[]');\n END IF;\n IF partition_range IS NULL THEN\n partition_range := tstzrange(\n date_trunc(c.partition_trunc::text, dt),\n date_trunc(c.partition_trunc::text, dt) + concat('1 ', c.partition_trunc)::interval\n );\n END IF;\n RETURN;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION drop_table_constraints(t text) RETURNS text AS $$\nDECLARE\n q text;\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM partitions_view WHERE partition=t) THEN\n RETURN NULL;\n END IF;\n FOR q IN SELECT FORMAT(\n $q$\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n $q$,\n t,\n conname\n ) FROM pg_constraint\n WHERE conrelid=t::regclass::oid AND contype='c'\n LOOP\n EXECUTE q;\n END LOOP;\n RETURN t;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE OR REPLACE FUNCTION create_table_constraints(t text, _dtrange tstzrange, _edtrange tstzrange) RETURNS text AS $$\nDECLARE\n q text;\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM partitions_view WHERE partition=t) THEN\n RETURN NULL;\n END IF;\n RAISE NOTICE 'Creating Table Constraints for % % %', t, _dtrange, _edtrange;\n IF _dtrange = 'empty' AND _edtrange = 'empty' THEN\n q :=format(\n $q$\n DO $block$\n BEGIN\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n ALTER TABLE %I\n ADD CONSTRAINT %I\n CHECK (((datetime IS NULL) AND (end_datetime IS NULL))) NOT VALID\n ;\n ALTER TABLE %I\n VALIDATE CONSTRAINT %I\n ;\n\n\n\n EXCEPTION WHEN others THEN\n RAISE WARNING '%%, Issue Altering Constraints. Please run update_partition_stats(%I)', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n $block$;\n $q$,\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t),\n t\n );\n ELSE\n q :=format(\n $q$\n DO $block$\n BEGIN\n\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n ALTER TABLE %I\n ADD CONSTRAINT %I\n CHECK (\n (datetime >= %L)\n AND (datetime <= %L)\n AND (end_datetime >= %L)\n AND (end_datetime <= %L)\n ) NOT VALID\n ;\n ALTER TABLE %I\n VALIDATE CONSTRAINT %I\n ;\n\n\n\n EXCEPTION WHEN others THEN\n RAISE WARNING '%%, Issue Altering Constraints. Please run update_partition_stats(%I)', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n $block$;\n $q$,\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t),\n lower(_dtrange),\n upper(_dtrange),\n lower(_edtrange),\n upper(_edtrange),\n t,\n format('%s_dt', t),\n t\n );\n END IF;\n PERFORM run_or_queue(q);\n RETURN t;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION check_partition(\n _collection text,\n _dtrange tstzrange,\n _edtrange tstzrange\n) RETURNS text AS $$\nDECLARE\n c RECORD;\n pm RECORD;\n _partition_name text;\n _partition_dtrange tstzrange;\n _constraint_dtrange tstzrange;\n _constraint_edtrange tstzrange;\n q text;\n deferrable_q text;\n err_context text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=_collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', _collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n\n IF c.partition_trunc IS NOT NULL THEN\n _partition_dtrange := tstzrange(\n date_trunc(c.partition_trunc, lower(_dtrange)),\n date_trunc(c.partition_trunc, lower(_dtrange)) + (concat('1 ', c.partition_trunc))::interval,\n '[)'\n );\n ELSE\n _partition_dtrange := '[-infinity, infinity]'::tstzrange;\n END IF;\n\n IF NOT _partition_dtrange @> _dtrange THEN\n RAISE EXCEPTION 'dtrange % is greater than the partition size % for collection %', _dtrange, c.partition_trunc, _collection;\n END IF;\n\n\n IF c.partition_trunc = 'year' THEN\n _partition_name := format('_items_%s_%s', c.key, to_char(lower(_partition_dtrange),'YYYY'));\n ELSIF c.partition_trunc = 'month' THEN\n _partition_name := format('_items_%s_%s', c.key, to_char(lower(_partition_dtrange),'YYYYMM'));\n ELSE\n _partition_name := format('_items_%s', c.key);\n END IF;\n\n SELECT * INTO pm FROM partition_sys_meta WHERE collection=_collection AND partition_dtrange @> _dtrange;\n IF FOUND THEN\n RAISE NOTICE '% % %', _edtrange, _dtrange, pm;\n _constraint_edtrange :=\n tstzrange(\n least(\n lower(_edtrange),\n nullif(lower(pm.constraint_edtrange), '-infinity')\n ),\n greatest(\n upper(_edtrange),\n nullif(upper(pm.constraint_edtrange), 'infinity')\n ),\n '[]'\n );\n _constraint_dtrange :=\n tstzrange(\n least(\n lower(_dtrange),\n nullif(lower(pm.constraint_dtrange), '-infinity')\n ),\n greatest(\n upper(_dtrange),\n nullif(upper(pm.constraint_dtrange), 'infinity')\n ),\n '[]'\n );\n\n IF pm.constraint_edtrange @> _edtrange AND pm.constraint_dtrange @> _dtrange THEN\n RETURN pm.partition;\n ELSE\n PERFORM drop_table_constraints(_partition_name);\n END IF;\n ELSE\n _constraint_edtrange := _edtrange;\n _constraint_dtrange := _dtrange;\n END IF;\n RAISE NOTICE 'EXISTING CONSTRAINTS % %, NEW % %', pm.constraint_dtrange, pm.constraint_edtrange, _constraint_dtrange, _constraint_edtrange;\n RAISE NOTICE 'Creating partition % %', _partition_name, _partition_dtrange;\n IF c.partition_trunc IS NULL THEN\n q := format(\n $q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n GRANT ALL ON %I to pgstac_ingest;\n $q$,\n _partition_name,\n _collection,\n concat(_partition_name,'_pk'),\n _partition_name,\n _partition_name\n );\n ELSE\n q := format(\n $q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L) PARTITION BY RANGE (datetime);\n CREATE TABLE IF NOT EXISTS %I partition OF %I FOR VALUES FROM (%L) TO (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n GRANT ALL ON %I TO pgstac_ingest;\n $q$,\n format('_items_%s', c.key),\n _collection,\n _partition_name,\n format('_items_%s', c.key),\n lower(_partition_dtrange),\n upper(_partition_dtrange),\n format('%s_pk', _partition_name),\n _partition_name,\n _partition_name\n );\n END IF;\n\n BEGIN\n EXECUTE q;\n EXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', _partition_name;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\n END;\n PERFORM maintain_partitions(_partition_name);\n PERFORM update_partition_stats_q(_partition_name, true);\n REFRESH MATERIALIZED VIEW partitions;\n REFRESH MATERIALIZED VIEW partition_steps;\n RETURN _partition_name;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION repartition(_collection text, _partition_trunc text, triggered boolean DEFAULT FALSE) RETURNS text AS $$\nDECLARE\n c RECORD;\n q text;\n from_trunc text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=_collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', _collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n IF triggered THEN\n RAISE NOTICE 'Converting % to % partitioning via Trigger', _collection, _partition_trunc;\n ELSE\n RAISE NOTICE 'Converting % from using % to % partitioning', _collection, c.partition_trunc, _partition_trunc;\n IF c.partition_trunc IS NOT DISTINCT FROM _partition_trunc THEN\n RAISE NOTICE 'Collection % already set to use partition by %', _collection, _partition_trunc;\n RETURN _collection;\n END IF;\n END IF;\n\n IF EXISTS (SELECT 1 FROM partitions_view WHERE collection=_collection LIMIT 1) THEN\n EXECUTE format(\n $q$\n CREATE TEMP TABLE changepartitionstaging ON COMMIT DROP AS SELECT * FROM %I;\n DROP TABLE IF EXISTS %I CASCADE;\n WITH p AS (\n SELECT\n collection,\n CASE\n WHEN %L IS NULL THEN '-infinity'::timestamptz\n ELSE date_trunc(%L, datetime)\n END as d,\n tstzrange(min(datetime),max(datetime),'[]') as dtrange,\n tstzrange(min(end_datetime),max(end_datetime),'[]') as edtrange\n FROM changepartitionstaging\n GROUP BY 1,2\n ) SELECT check_partition(collection, dtrange, edtrange) FROM p;\n INSERT INTO items SELECT * FROM changepartitionstaging;\n DROP TABLE changepartitionstaging;\n $q$,\n concat('_items_', c.key),\n concat('_items_', c.key),\n c.partition_trunc,\n c.partition_trunc\n );\n END IF;\n RETURN _collection;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE OR REPLACE FUNCTION collections_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n q text;\n partition_name text := format('_items_%s', NEW.key);\n partition_exists boolean := false;\n partition_empty boolean := true;\n err_context text;\n loadtemp boolean := FALSE;\nBEGIN\n RAISE NOTICE 'Collection Trigger. % %', NEW.id, NEW.key;\n IF TG_OP = 'UPDATE' AND NEW.partition_trunc IS DISTINCT FROM OLD.partition_trunc THEN\n PERFORM repartition(NEW.id, NEW.partition_trunc, TRUE);\n END IF;\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER collections_trigger AFTER\nINSERT\nOR\nUPDATE ON collections\nFOR EACH ROW EXECUTE FUNCTION collections_trigger_func();\n\nCREATE OR REPLACE FUNCTION chunker(\n IN _where text,\n OUT s timestamptz,\n OUT e timestamptz\n) RETURNS SETOF RECORD AS $$\nDECLARE\n explain jsonb;\nBEGIN\n IF _where IS NULL THEN\n _where := ' TRUE ';\n END IF;\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s;', _where)\n INTO explain;\n\n RETURN QUERY\n WITH t AS (\n SELECT j->>0 as p FROM\n jsonb_path_query(\n explain,\n 'strict $.**.\"Relation Name\" ? (@ != null)'\n ) j\n ),\n parts AS (\n SELECT sdate, edate FROM t JOIN partition_steps ON (t.p = name)\n ),\n times AS (\n SELECT sdate FROM parts\n UNION\n SELECT edate FROM parts\n ),\n uniq AS (\n SELECT DISTINCT sdate FROM times ORDER BY sdate\n ),\n last AS (\n SELECT sdate, lead(sdate, 1) over () as edate FROM uniq\n )\n SELECT sdate, edate FROM last WHERE edate IS NOT NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION partition_queries(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN partitions text[] DEFAULT NULL\n) RETURNS SETOF text AS $$\nDECLARE\n query text;\n sdate timestamptz;\n edate timestamptz;\nBEGIN\nIF _where IS NULL OR trim(_where) = '' THEN\n _where = ' TRUE ';\nEND IF;\nRAISE NOTICE 'Getting chunks for % %', _where, _orderby;\nIF _orderby ILIKE 'datetime d%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 DESC LOOP\n RETURN NEXT format($q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n $q$,\n sdate,\n edate,\n _where,\n _orderby\n );\n END LOOP;\nELSIF _orderby ILIKE 'datetime a%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 ASC LOOP\n RETURN NEXT format($q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n $q$,\n sdate,\n edate,\n _where,\n _orderby\n );\n END LOOP;\nELSE\n query := format($q$\n SELECT * FROM items\n WHERE %s\n ORDER BY %s\n $q$, _where, _orderby\n );\n\n RETURN NEXT query;\n RETURN;\nEND IF;\n\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION partition_query_view(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN _limit int DEFAULT 10\n) RETURNS text AS $$\n WITH p AS (\n SELECT * FROM partition_queries(_where, _orderby) p\n )\n SELECT\n CASE WHEN EXISTS (SELECT 1 FROM p) THEN\n (SELECT format($q$\n SELECT * FROM (\n %s\n ) total LIMIT %s\n $q$,\n string_agg(\n format($q$ SELECT * FROM ( %s ) AS sub $q$, p),\n '\n UNION ALL\n '\n ),\n _limit\n ))\n ELSE NULL\n END FROM p;\n$$ LANGUAGE SQL IMMUTABLE;\n\n\n\n\nCREATE OR REPLACE FUNCTION stac_search_to_where(j jsonb) RETURNS text AS $$\nDECLARE\n where_segments text[];\n _where text;\n dtrange tstzrange;\n collections text[];\n geom geometry;\n sdate timestamptz;\n edate timestamptz;\n filterlang text;\n filter jsonb := j->'filter';\nBEGIN\n IF j ? 'ids' THEN\n where_segments := where_segments || format('id = ANY (%L) ', to_text_array(j->'ids'));\n END IF;\n\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n where_segments := where_segments || format('collection = ANY (%L) ', collections);\n END IF;\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n\n where_segments := where_segments || format(' datetime <= %L::timestamptz AND end_datetime >= %L::timestamptz ',\n edate,\n sdate\n );\n END IF;\n\n geom := stac_geom(j);\n IF geom IS NOT NULL THEN\n where_segments := where_segments || format('st_intersects(geometry, %L)',geom);\n END IF;\n\n filterlang := COALESCE(\n j->>'filter-lang',\n get_setting('default_filter_lang', j->'conf')\n );\n IF NOT filter @? '$.**.op' THEN\n filterlang := 'cql-json';\n END IF;\n\n IF filterlang NOT IN ('cql-json','cql2-json') AND j ? 'filter' THEN\n RAISE EXCEPTION '% is not a supported filter-lang. Please use cql-json or cql2-json.', filterlang;\n END IF;\n\n IF j ? 'query' AND j ? 'filter' THEN\n RAISE EXCEPTION 'Can only use either query or filter at one time.';\n END IF;\n\n IF j ? 'query' THEN\n filter := query_to_cql2(j->'query');\n ELSIF filterlang = 'cql-json' THEN\n filter := cql1_to_cql2(filter);\n END IF;\n RAISE NOTICE 'FILTER: %', filter;\n where_segments := where_segments || cql2_query(filter);\n IF cardinality(where_segments) < 1 THEN\n RETURN ' TRUE ';\n END IF;\n\n _where := array_to_string(array_remove(where_segments, NULL), ' AND ');\n\n IF _where IS NULL OR BTRIM(_where) = '' THEN\n RETURN ' TRUE ';\n END IF;\n RETURN _where;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION parse_sort_dir(_dir text, reverse boolean default false) RETURNS text AS $$\n WITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n ) SELECT\n CASE\n WHEN NOT reverse THEN d\n WHEN d = 'ASC' THEN 'DESC'\n WHEN d = 'DESC' THEN 'ASC'\n END\n FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION sort_dir_to_op(_dir text, prev boolean default false) RETURNS text AS $$\n WITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n ) SELECT\n CASE\n WHEN d = 'ASC' AND prev THEN '<='\n WHEN d = 'DESC' AND prev THEN '>='\n WHEN d = 'ASC' THEN '>='\n WHEN d = 'DESC' THEN '<='\n END\n FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION sort_sqlorderby(\n _search jsonb DEFAULT NULL,\n reverse boolean DEFAULT FALSE\n) RETURNS text AS $$\n WITH sortby AS (\n SELECT coalesce(_search->'sortby','[{\"field\":\"datetime\", \"direction\":\"desc\"}]') as sort\n ), withid AS (\n SELECT CASE\n WHEN sort @? '$[*] ? (@.field == \"id\")' THEN sort\n ELSE sort || '[{\"field\":\"id\", \"direction\":\"desc\"}]'::jsonb\n END as sort\n FROM sortby\n ), withid_rows AS (\n SELECT jsonb_array_elements(sort) as value FROM withid\n ),sorts AS (\n SELECT\n coalesce(\n (queryable(value->>'field')).expression\n ) as key,\n parse_sort_dir(value->>'direction', reverse) as dir\n FROM withid_rows\n )\n SELECT array_to_string(\n array_agg(concat(key, ' ', dir)),\n ', '\n ) FROM sorts;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_sort_dir(sort_item jsonb) RETURNS text AS $$\n SELECT CASE WHEN sort_item->>'direction' ILIKE 'desc%' THEN 'DESC' ELSE 'ASC' END;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION get_token_val_str(\n _field text,\n _item items\n) RETURNS text AS $$\nDECLARE\n q text;\n literal text;\nBEGIN\n q := format($q$ SELECT quote_literal(%s) FROM (SELECT $1.*) as r;$q$, _field);\n EXECUTE q INTO literal USING _item;\n RETURN literal;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n\nCREATE OR REPLACE FUNCTION get_token_record(IN _token text, OUT prev BOOLEAN, OUT item items) RETURNS RECORD AS $$\nDECLARE\n _itemid text := _token;\n _collectionid text;\nBEGIN\n IF _token IS NULL THEN\n RETURN;\n END IF;\n RAISE NOTICE 'Looking for token: %', _token;\n prev := FALSE;\n IF _token ILIKE 'prev:%' THEN\n _itemid := replace(_token, 'prev:','');\n prev := TRUE;\n ELSIF _token ILIKE 'next:%' THEN\n _itemid := replace(_token, 'next:', '');\n END IF;\n SELECT id INTO _collectionid FROM collections WHERE _itemid LIKE concat(id,':%');\n IF FOUND THEN\n _itemid := replace(_itemid, concat(_collectionid,':'), '');\n SELECT * INTO item FROM items WHERE id=_itemid AND collection=_collectionid;\n ELSE\n SELECT * INTO item FROM items WHERE id=_itemid;\n END IF;\n IF item IS NULL THEN\n RAISE EXCEPTION 'Could not find item using token: % item: % collection: %', _token, _itemid, _collectionid;\n END IF;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\n\nCREATE OR REPLACE FUNCTION get_token_filter(\n _sortby jsonb DEFAULT '[{\"field\":\"datetime\",\"direction\":\"desc\"}]'::jsonb,\n token_item items DEFAULT NULL,\n prev boolean DEFAULT FALSE,\n inclusive boolean DEFAULT FALSE\n) RETURNS text AS $$\nDECLARE\n ltop text := '<';\n gtop text := '>';\n dir text;\n sort record;\n orfilter text := '';\n orfilters text[] := '{}'::text[];\n andfilters text[] := '{}'::text[];\n output text;\n token_where text;\nBEGIN\n IF _sortby IS NULL OR _sortby = '[]'::jsonb THEN\n _sortby := '[{\"field\":\"datetime\",\"direction\":\"desc\"}]'::jsonb;\n END IF;\n _sortby := _sortby || jsonb_build_object('field','id','direction',_sortby->0->>'direction');\n RAISE NOTICE 'Getting Token Filter. % %', _sortby, token_item;\n IF inclusive THEN\n orfilters := orfilters || format('( id=%L AND collection=%L )' , token_item.id, token_item.collection);\n END IF;\n\n FOR sort IN\n WITH s1 AS (\n SELECT\n _row,\n (queryable(value->>'field')).expression as _field,\n (value->>'field' = 'id') as _isid,\n get_sort_dir(value) as _dir\n FROM jsonb_array_elements(_sortby)\n WITH ORDINALITY AS t(value, _row)\n )\n SELECT\n _row,\n _field,\n _dir,\n get_token_val_str(_field, token_item) as _val\n FROM s1\n WHERE _row <= (SELECT min(_row) FROM s1 WHERE _isid)\n LOOP\n orfilter := NULL;\n RAISE NOTICE 'SORT: %', sort;\n IF sort._val IS NOT NULL AND ((prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC')) THEN\n orfilter := format($f$(\n (%s %s %s) OR (%s IS NULL)\n )$f$,\n sort._field,\n ltop,\n sort._val,\n sort._val\n );\n ELSIF sort._val IS NULL AND ((prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC')) THEN\n RAISE NOTICE '< but null';\n orfilter := format('%s IS NOT NULL', sort._field);\n ELSIF sort._val IS NULL THEN\n RAISE NOTICE '> but null';\n ELSE\n orfilter := format($f$(\n (%s %s %s) OR (%s IS NULL)\n )$f$,\n sort._field,\n gtop,\n sort._val,\n sort._field\n );\n END IF;\n RAISE NOTICE 'ORFILTER: %', orfilter;\n\n IF orfilter IS NOT NULL THEN\n IF sort._row = 1 THEN\n orfilters := orfilters || orfilter;\n ELSE\n orfilters := orfilters || format('(%s AND %s)', array_to_string(andfilters, ' AND '), orfilter);\n END IF;\n END IF;\n IF sort._val IS NOT NULL THEN\n andfilters := andfilters || format('%s = %s', sort._field, sort._val);\n ELSE\n andfilters := andfilters || format('%s IS NULL', sort._field);\n END IF;\n END LOOP;\n\n output := array_to_string(orfilters, ' OR ');\n\n token_where := concat('(',coalesce(output,'true'),')');\n IF trim(token_where) = '' THEN\n token_where := NULL;\n END IF;\n RAISE NOTICE 'TOKEN_WHERE: %',token_where;\n RETURN token_where;\n END;\n$$ LANGUAGE PLPGSQL SET transform_null_equals TO TRUE\n;\n\nCREATE OR REPLACE FUNCTION search_tohash(jsonb) RETURNS jsonb AS $$\n SELECT $1 - '{token,limit,context,includes,excludes}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION search_hash(jsonb, jsonb) RETURNS text AS $$\n SELECT md5(concat(search_tohash($1)::text,$2::text));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TABLE IF NOT EXISTS searches(\n hash text GENERATED ALWAYS AS (search_hash(search, metadata)) STORED PRIMARY KEY,\n search jsonb NOT NULL,\n _where text,\n orderby text,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n metadata jsonb DEFAULT '{}'::jsonb NOT NULL\n);\n\nCREATE TABLE IF NOT EXISTS search_wheres(\n id bigint generated always as identity primary key,\n _where text NOT NULL,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n statslastupdated timestamptz,\n estimated_count bigint,\n estimated_cost float,\n time_to_estimate float,\n total_count bigint,\n time_to_count float,\n partitions text[]\n);\n\nCREATE INDEX IF NOT EXISTS search_wheres_partitions ON search_wheres USING GIN (partitions);\nCREATE UNIQUE INDEX IF NOT EXISTS search_wheres_where ON search_wheres ((md5(_where)));\n\nCREATE OR REPLACE FUNCTION where_stats(\n inwhere text,\n updatestats boolean default false,\n conf jsonb default null\n) RETURNS search_wheres AS $$\nDECLARE\n t timestamptz;\n i interval;\n explain_json jsonb;\n partitions text[];\n sw search_wheres%ROWTYPE;\n inwhere_hash text := md5(inwhere);\n _context text := lower(context(conf));\n _stats_ttl interval := context_stats_ttl(conf);\n _estimated_cost_threshold float := context_estimated_cost(conf);\n _estimated_count_threshold int := context_estimated_count(conf);\n ro bool := pgstac.readonly(conf);\nBEGIN\n -- If updatestats is true then set ttl to 0\n IF updatestats THEN\n RAISE DEBUG 'Updatestats set to TRUE, setting TTL to 0';\n _stats_ttl := '0'::interval;\n END IF;\n\n -- If we don't need to calculate context, just return\n IF _context = 'off' THEN\n sw._where = inwhere;\n RETURN sw;\n END IF;\n\n -- Get any stats that we have. If there is a lock where another process is\n -- updating the stats, wait so that we don't end up calculating a bunch of times.\n SELECT * INTO sw FROM search_wheres WHERE md5(_where)=inwhere_hash FOR UPDATE;\n\n -- If there is a cached row, figure out if we need to update\n IF\n sw IS NOT NULL\n AND sw.statslastupdated IS NOT NULL\n AND sw.total_count IS NOT NULL\n AND now() - sw.statslastupdated <= _stats_ttl\n THEN\n -- we have a cached row with data that is within our ttl\n RAISE DEBUG 'Stats present in table and lastupdated within ttl: %', sw;\n IF NOT ro THEN\n RAISE DEBUG 'Updating search_wheres only bumping lastused and usecount';\n UPDATE search_wheres SET\n lastused = now(),\n usecount = search_wheres.usecount + 1\n WHERE md5(_where) = inwhere_hash\n RETURNING * INTO sw;\n END IF;\n RAISE DEBUG 'Returning cached counts. %', sw;\n RETURN sw;\n END IF;\n\n -- Calculate estimated cost and rows\n -- Use explain to get estimated count/cost\n IF sw.estimated_count IS NULL OR sw.estimated_cost IS NULL THEN\n RAISE DEBUG 'Calculating estimated stats';\n t := clock_timestamp();\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s', inwhere)\n INTO explain_json;\n RAISE DEBUG 'Time for just the explain: %', clock_timestamp() - t;\n i := clock_timestamp() - t;\n\n sw.estimated_count := explain_json->0->'Plan'->'Plan Rows';\n sw.estimated_cost := explain_json->0->'Plan'->'Total Cost';\n sw.time_to_estimate := extract(epoch from i);\n END IF;\n\n RAISE DEBUG 'ESTIMATED_COUNT: %, THRESHOLD %', sw.estimated_count, _estimated_count_threshold;\n RAISE DEBUG 'ESTIMATED_COST: %, THRESHOLD %', sw.estimated_cost, _estimated_cost_threshold;\n\n -- If context is set to auto and the costs are within the threshold return the estimated costs\n IF\n _context = 'auto'\n AND sw.estimated_count >= _estimated_count_threshold\n AND sw.estimated_cost >= _estimated_cost_threshold\n THEN\n IF NOT ro THEN\n INSERT INTO search_wheres (\n _where,\n lastused,\n usecount,\n statslastupdated,\n estimated_count,\n estimated_cost,\n time_to_estimate,\n total_count,\n time_to_count\n ) VALUES (\n inwhere,\n now(),\n 1,\n now(),\n sw.estimated_count,\n sw.estimated_cost,\n sw.time_to_estimate,\n null,\n null\n ) ON CONFLICT ((md5(_where)))\n DO UPDATE SET\n lastused = EXCLUDED.lastused,\n usecount = search_wheres.usecount + 1,\n statslastupdated = EXCLUDED.statslastupdated,\n estimated_count = EXCLUDED.estimated_count,\n estimated_cost = EXCLUDED.estimated_cost,\n time_to_estimate = EXCLUDED.time_to_estimate,\n total_count = EXCLUDED.total_count,\n time_to_count = EXCLUDED.time_to_count\n RETURNING * INTO sw;\n END IF;\n RAISE DEBUG 'Estimates are within thresholds, returning estimates. %', sw;\n RETURN sw;\n END IF;\n\n -- Calculate Actual Count\n t := clock_timestamp();\n RAISE NOTICE 'Calculating actual count...';\n EXECUTE format(\n 'SELECT count(*) FROM items WHERE %s',\n inwhere\n ) INTO sw.total_count;\n i := clock_timestamp() - t;\n RAISE NOTICE 'Actual Count: % -- %', sw.total_count, i;\n sw.time_to_count := extract(epoch FROM i);\n\n IF NOT ro THEN\n INSERT INTO search_wheres (\n _where,\n lastused,\n usecount,\n statslastupdated,\n estimated_count,\n estimated_cost,\n time_to_estimate,\n total_count,\n time_to_count\n ) VALUES (\n inwhere,\n now(),\n 1,\n now(),\n sw.estimated_count,\n sw.estimated_cost,\n sw.time_to_estimate,\n sw.total_count,\n sw.time_to_count\n ) ON CONFLICT ((md5(_where)))\n DO UPDATE SET\n lastused = EXCLUDED.lastused,\n usecount = search_wheres.usecount + 1,\n statslastupdated = EXCLUDED.statslastupdated,\n estimated_count = EXCLUDED.estimated_count,\n estimated_cost = EXCLUDED.estimated_cost,\n time_to_estimate = EXCLUDED.time_to_estimate,\n total_count = EXCLUDED.total_count,\n time_to_count = EXCLUDED.time_to_count\n RETURNING * INTO sw;\n END IF;\n RAISE DEBUG 'Returning with actual count. %', sw;\n RETURN sw;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION search_query(\n _search jsonb = '{}'::jsonb,\n updatestats boolean = false,\n _metadata jsonb = '{}'::jsonb\n) RETURNS searches AS $$\nDECLARE\n search searches%ROWTYPE;\n cached_search searches%ROWTYPE;\n pexplain jsonb;\n t timestamptz;\n i interval;\n doupdate boolean := FALSE;\n insertfound boolean := FALSE;\n ro boolean := pgstac.readonly();\n found_search text;\nBEGIN\n RAISE NOTICE 'SEARCH: %', _search;\n -- Calculate hash, where clause, and order by statement\n search.search := _search;\n search.metadata := _metadata;\n search.hash := search_hash(_search, _metadata);\n search._where := stac_search_to_where(_search);\n search.orderby := sort_sqlorderby(_search);\n search.lastused := now();\n search.usecount := 1;\n\n -- If we are in read only mode, directly return search\n IF ro THEN\n RETURN search;\n END IF;\n\n RAISE NOTICE 'Updating Statistics for search: %s', search;\n -- Update statistics for times used and and when last used\n -- If the entry is locked, rather than waiting, skip updating the stats\n INSERT INTO searches (search, lastused, usecount, metadata)\n VALUES (search.search, now(), 1, search.metadata)\n ON CONFLICT DO NOTHING\n RETURNING * INTO cached_search\n ;\n\n IF NOT FOUND OR cached_search IS NULL THEN\n UPDATE searches SET\n lastused = now(),\n usecount = searches.usecount + 1\n WHERE hash = (\n SELECT hash FROM searches WHERE hash=search.hash FOR UPDATE SKIP LOCKED\n )\n RETURNING * INTO cached_search\n ;\n END IF;\n\n IF cached_search IS NOT NULL THEN\n cached_search._where = search._where;\n cached_search.orderby = search.orderby;\n RETURN cached_search;\n END IF;\n RETURN search;\n\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE OR REPLACE FUNCTION search_fromhash(\n _hash text\n) RETURNS searches AS $$\n SELECT * FROM search_query((SELECT search FROM searches WHERE hash=_hash LIMIT 1));\n$$ LANGUAGE SQL STRICT;\n\nCREATE OR REPLACE FUNCTION search_rows(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN partitions text[] DEFAULT NULL,\n IN _limit int DEFAULT 10\n) RETURNS SETOF items AS $$\nDECLARE\n base_query text;\n query text;\n sdate timestamptz;\n edate timestamptz;\n n int;\n records_left int := _limit;\n timer timestamptz := clock_timestamp();\n full_timer timestamptz := clock_timestamp();\nBEGIN\nIF _where IS NULL OR trim(_where) = '' THEN\n _where = ' TRUE ';\nEND IF;\nRAISE NOTICE 'Getting chunks for % %', _where, _orderby;\n\nbase_query := $q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n LIMIT %L\n$q$;\n\nIF _orderby ILIKE 'datetime d%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 DESC LOOP\n RAISE NOTICE 'Running Query for % to %. %', sdate, edate, age_ms(full_timer);\n query := format(\n base_query,\n sdate,\n edate,\n _where,\n _orderby,\n records_left\n );\n RAISE DEBUG 'QUERY: %', query;\n timer := clock_timestamp();\n RETURN QUERY EXECUTE query;\n\n GET DIAGNOSTICS n = ROW_COUNT;\n records_left := records_left - n;\n RAISE NOTICE 'Returned %/% Rows From % to %. % to go. Time: %ms', n, _limit, sdate, edate, records_left, age_ms(timer);\n timer := clock_timestamp();\n IF records_left <= 0 THEN\n RAISE NOTICE 'SEARCH_ROWS TOOK %ms', age_ms(full_timer);\n RETURN;\n END IF;\n END LOOP;\nELSIF _orderby ILIKE 'datetime a%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 ASC LOOP\n RAISE NOTICE 'Running Query for % to %. %', sdate, edate, age_ms(full_timer);\n query := format(\n base_query,\n sdate,\n edate,\n _where,\n _orderby,\n records_left\n );\n RAISE DEBUG 'QUERY: %', query;\n timer := clock_timestamp();\n RETURN QUERY EXECUTE query;\n\n GET DIAGNOSTICS n = ROW_COUNT;\n records_left := records_left - n;\n RAISE NOTICE 'Returned %/% Rows From % to %. % to go. Time: %ms', n, _limit, sdate, edate, records_left, age_ms(timer);\n timer := clock_timestamp();\n IF records_left <= 0 THEN\n RAISE NOTICE 'SEARCH_ROWS TOOK %ms', age_ms(full_timer);\n RETURN;\n END IF;\n END LOOP;\nELSE\n query := format($q$\n SELECT * FROM items\n WHERE %s\n ORDER BY %s\n LIMIT %L\n $q$, _where, _orderby, _limit\n );\n RAISE DEBUG 'QUERY: %', query;\n timer := clock_timestamp();\n RETURN QUERY EXECUTE query;\n RAISE NOTICE 'FULL QUERY TOOK %ms', age_ms(timer);\nEND IF;\nRAISE NOTICE 'SEARCH_ROWS TOOK %ms', age_ms(full_timer);\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE UNLOGGED TABLE format_item_cache(\n id text,\n collection text,\n fields text,\n hydrated bool,\n output jsonb,\n lastused timestamptz DEFAULT now(),\n usecount int DEFAULT 1,\n timetoformat float,\n PRIMARY KEY (collection, id, fields, hydrated)\n);\nCREATE INDEX ON format_item_cache (lastused);\n\nCREATE OR REPLACE FUNCTION format_item(_item items, _fields jsonb DEFAULT '{}', _hydrated bool DEFAULT TRUE) RETURNS jsonb AS $$\nDECLARE\n cache bool := get_setting_bool('format_cache');\n _output jsonb := null;\n t timestamptz := clock_timestamp();\nBEGIN\n IF cache THEN\n SELECT output INTO _output FROM format_item_cache\n WHERE id=_item.id AND collection=_item.collection AND fields=_fields::text AND hydrated=_hydrated;\n END IF;\n IF _output IS NULL THEN\n IF _hydrated THEN\n _output := content_hydrate(_item, _fields);\n ELSE\n _output := content_nonhydrated(_item, _fields);\n END IF;\n END IF;\n IF cache THEN\n INSERT INTO format_item_cache (id, collection, fields, hydrated, output, timetoformat)\n VALUES (_item.id, _item.collection, _fields::text, _hydrated, _output, age_ms(t))\n ON CONFLICT(collection, id, fields, hydrated) DO\n UPDATE\n SET lastused=now(), usecount = format_item_cache.usecount + 1\n ;\n END IF;\n RETURN _output;\n\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION search(_search jsonb = '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n searches searches%ROWTYPE;\n _where text;\n orderby text;\n search_where search_wheres%ROWTYPE;\n total_count bigint;\n token record;\n token_prev boolean;\n token_item items%ROWTYPE;\n token_where text;\n full_where text;\n init_ts timestamptz := clock_timestamp();\n timer timestamptz := clock_timestamp();\n hydrate bool := NOT (_search->'conf'->>'nohydrate' IS NOT NULL AND (_search->'conf'->>'nohydrate')::boolean = true);\n prev text;\n next text;\n context jsonb;\n collection jsonb;\n out_records jsonb;\n out_len int;\n _limit int := coalesce((_search->>'limit')::int, 10);\n _querylimit int;\n _fields jsonb := coalesce(_search->'fields', '{}'::jsonb);\n has_prev boolean := FALSE;\n has_next boolean := FALSE;\n links jsonb := '[]'::jsonb;\n base_url text:= concat(rtrim(base_url(_search->'conf'),'/'));\nBEGIN\n searches := search_query(_search);\n _where := searches._where;\n orderby := searches.orderby;\n search_where := where_stats(_where);\n total_count := coalesce(search_where.total_count, search_where.estimated_count);\n RAISE NOTICE 'SEARCH:TOKEN: %', _search->>'token';\n token := get_token_record(_search->>'token');\n RAISE NOTICE '***TOKEN: %', token;\n _querylimit := _limit + 1;\n IF token IS NOT NULL THEN\n token_prev := token.prev;\n token_item := token.item;\n token_where := get_token_filter(_search->'sortby', token_item, token_prev, FALSE);\n RAISE DEBUG 'TOKEN_WHERE: % (%ms from search start)', token_where, age_ms(timer);\n IF token_prev THEN -- if we are using a prev token, we know has_next is true\n RAISE DEBUG 'There is a previous token, so automatically setting has_next to true';\n has_next := TRUE;\n orderby := sort_sqlorderby(_search, TRUE);\n ELSE\n RAISE DEBUG 'There is a next token, so automatically setting has_prev to true';\n has_prev := TRUE;\n\n END IF;\n ELSE -- if there was no token, we know there is no prev\n RAISE DEBUG 'There is no token, so we know there is no prev. setting has_prev to false';\n has_prev := FALSE;\n END IF;\n\n full_where := concat_ws(' AND ', _where, token_where);\n RAISE NOTICE 'FULL WHERE CLAUSE: %', full_where;\n RAISE NOTICE 'Time to get counts and build query %', age_ms(timer);\n timer := clock_timestamp();\n\n IF hydrate THEN\n RAISE NOTICE 'Getting hydrated data.';\n ELSE\n RAISE NOTICE 'Getting non-hydrated data.';\n END IF;\n RAISE NOTICE 'CACHE SET TO %', get_setting_bool('format_cache');\n RAISE NOTICE 'Time to set hydration/formatting %', age_ms(timer);\n timer := clock_timestamp();\n SELECT jsonb_agg(format_item(i, _fields, hydrate)) INTO out_records\n FROM search_rows(\n full_where,\n orderby,\n search_where.partitions,\n _querylimit\n ) as i;\n\n RAISE NOTICE 'Time to fetch rows %', age_ms(timer);\n timer := clock_timestamp();\n\n\n IF token_prev THEN\n out_records := flip_jsonb_array(out_records);\n END IF;\n\n RAISE NOTICE 'Query returned % records.', jsonb_array_length(out_records);\n RAISE DEBUG 'TOKEN: % %', token_item.id, token_item.collection;\n RAISE DEBUG 'RECORD_1: % %', out_records->0->>'id', out_records->0->>'collection';\n RAISE DEBUG 'RECORD-1: % %', out_records->-1->>'id', out_records->-1->>'collection';\n\n -- REMOVE records that were from our token\n IF out_records->0->>'id' = token_item.id AND out_records->0->>'collection' = token_item.collection THEN\n out_records := out_records - 0;\n ELSIF out_records->-1->>'id' = token_item.id AND out_records->-1->>'collection' = token_item.collection THEN\n out_records := out_records - -1;\n END IF;\n\n out_len := jsonb_array_length(out_records);\n\n IF out_len = _limit + 1 THEN\n IF token_prev THEN\n has_prev := TRUE;\n out_records := out_records - 0;\n ELSE\n has_next := TRUE;\n out_records := out_records - -1;\n END IF;\n END IF;\n\n\n links := links || jsonb_build_object(\n 'rel', 'root',\n 'type', 'application/json',\n 'href', base_url\n ) || jsonb_build_object(\n 'rel', 'self',\n 'type', 'application/json',\n 'href', concat(base_url, '/search')\n );\n\n IF has_next THEN\n next := concat(out_records->-1->>'collection', ':', out_records->-1->>'id');\n RAISE NOTICE 'HAS NEXT | %', next;\n links := links || jsonb_build_object(\n 'rel', 'next',\n 'type', 'application/geo+json',\n 'method', 'GET',\n 'href', concat(base_url, '/search?token=next:', next)\n );\n END IF;\n\n IF has_prev THEN\n prev := concat(out_records->0->>'collection', ':', out_records->0->>'id');\n RAISE NOTICE 'HAS PREV | %', prev;\n links := links || jsonb_build_object(\n 'rel', 'prev',\n 'type', 'application/geo+json',\n 'method', 'GET',\n 'href', concat(base_url, '/search?token=prev:', prev)\n );\n END IF;\n\n RAISE NOTICE 'Time to get prev/next %', age_ms(timer);\n timer := clock_timestamp();\n\n\n collection := jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb),\n 'links', links\n );\n\n\n\n IF context(_search->'conf') != 'off' THEN\n collection := collection || jsonb_strip_nulls(jsonb_build_object(\n 'numberMatched', total_count,\n 'numberReturned', coalesce(jsonb_array_length(out_records), 0)\n ));\n ELSE\n collection := collection || jsonb_strip_nulls(jsonb_build_object(\n 'numberReturned', coalesce(jsonb_array_length(out_records), 0)\n ));\n END IF;\n\n IF get_setting_bool('timing', _search->'conf') THEN\n collection = collection || jsonb_build_object('timing', age_ms(init_ts));\n END IF;\n\n RAISE NOTICE 'Time to build final json %', age_ms(timer);\n timer := clock_timestamp();\n\n RAISE NOTICE 'Total Time: %', age_ms(current_timestamp);\n RAISE NOTICE 'RETURNING % records. NEXT: %. PREV: %', collection->>'numberReturned', collection->>'next', collection->>'prev';\n RETURN collection;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION search_cursor(_search jsonb = '{}'::jsonb) RETURNS refcursor AS $$\nDECLARE\n curs refcursor;\n searches searches%ROWTYPE;\n _where text;\n _orderby text;\n q text;\n\nBEGIN\n searches := search_query(_search);\n _where := searches._where;\n _orderby := searches.orderby;\n\n OPEN curs FOR\n WITH p AS (\n SELECT * FROM partition_queries(_where, _orderby) p\n )\n SELECT\n CASE WHEN EXISTS (SELECT 1 FROM p) THEN\n (SELECT format($q$\n SELECT * FROM (\n %s\n ) total\n $q$,\n string_agg(\n format($q$ SELECT * FROM ( %s ) AS sub $q$, p),\n '\n UNION ALL\n '\n )\n ))\n ELSE NULL\n END FROM p;\n RETURN curs;\nEND;\n$$ LANGUAGE PLPGSQL;\nCREATE OR REPLACE VIEW collections_asitems AS\nSELECT\n id,\n geometry,\n 'collections' AS collection,\n datetime,\n end_datetime,\n jsonb_build_object(\n 'properties', content - '{links,assets,stac_version,stac_extensions}',\n 'links', content->'links',\n 'assets', content->'assets',\n 'stac_version', content->'stac_version',\n 'stac_extensions', content->'stac_extensions'\n ) AS content,\n content as collectionjson\nFROM collections;\n\n\nCREATE OR REPLACE FUNCTION collection_search_matched(\n IN _search jsonb DEFAULT '{}'::jsonb,\n OUT matched bigint\n) RETURNS bigint AS $$\nDECLARE\n _where text := stac_search_to_where(_search);\nBEGIN\n EXECUTE format(\n $query$\n SELECT\n count(*)\n FROM\n collections_asitems\n WHERE %s\n ;\n $query$,\n _where\n ) INTO matched;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION collection_search_rows(\n _search jsonb DEFAULT '{}'::jsonb\n) RETURNS SETOF jsonb AS $$\nDECLARE\n _where text := stac_search_to_where(_search);\n _limit int := coalesce((_search->>'limit')::int, 10);\n _fields jsonb := coalesce(_search->'fields', '{}'::jsonb);\n _orderby text;\n _offset int := COALESCE((_search->>'offset')::int, 0);\nBEGIN\n _orderby := sort_sqlorderby(\n jsonb_build_object(\n 'sortby',\n coalesce(\n _search->'sortby',\n '[{\"field\": \"id\", \"direction\": \"asc\"}]'::jsonb\n )\n )\n );\n RETURN QUERY EXECUTE format(\n $query$\n SELECT\n jsonb_fields(collectionjson, %L) as c\n FROM\n collections_asitems\n WHERE %s\n ORDER BY %s\n LIMIT %L\n OFFSET %L\n ;\n $query$,\n _fields,\n _where,\n _orderby,\n _limit,\n _offset\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION collection_search(\n _search jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\nDECLARE\n out_records jsonb;\n number_matched bigint := collection_search_matched(_search);\n number_returned bigint;\n _limit int := coalesce((_search->>'limit')::float::int, 10);\n _offset int := coalesce((_search->>'offset')::float::int, 0);\n links jsonb := '[]';\n ret jsonb;\n base_url text:= concat(rtrim(base_url(_search->'conf'),'/'), '/collections');\n prevoffset int;\n nextoffset int;\nBEGIN\n SELECT\n coalesce(jsonb_agg(c), '[]')\n INTO out_records\n FROM collection_search_rows(_search) c;\n\n number_returned := jsonb_array_length(out_records);\n\n IF _limit <= number_matched THEN --need to have paging links\n nextoffset := least(_offset + _limit, number_matched - 1);\n prevoffset := greatest(_offset - _limit, 0);\n IF _offset = 0 THEN -- no previous paging\n\n links := jsonb_build_array(\n jsonb_build_object(\n 'rel', 'next',\n 'type', 'application/json',\n 'method', 'GET' ,\n 'href', base_url,\n 'body', jsonb_build_object('offset', nextoffset),\n 'merge', TRUE\n )\n );\n ELSE\n links := jsonb_build_array(\n jsonb_build_object(\n 'rel', 'prev',\n 'type', 'application/json',\n 'method', 'GET' ,\n 'href', base_url,\n 'body', jsonb_build_object('offset', prevoffset),\n 'merge', TRUE\n ),\n jsonb_build_object(\n 'rel', 'next',\n 'type', 'application/json',\n 'method', 'GET' ,\n 'href', base_url,\n 'body', jsonb_build_object('offset', nextoffset),\n 'merge', TRUE\n )\n );\n END IF;\n END IF;\n\n ret := jsonb_build_object(\n 'collections', out_records,\n 'numberMatched', number_matched,\n 'numberReturned', number_returned,\n 'links', links\n );\n RETURN ret;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION tileenvelope(zoom int, x int, y int) RETURNS geometry AS $$\nWITH t AS (\n SELECT\n 20037508.3427892 as merc_max,\n -20037508.3427892 as merc_min,\n (2 * 20037508.3427892) / (2 ^ zoom) as tile_size\n)\nSELECT st_makeenvelope(\n merc_min + (tile_size * x),\n merc_max - (tile_size * (y + 1)),\n merc_min + (tile_size * (x + 1)),\n merc_max - (tile_size * y),\n 3857\n) FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;DROP FUNCTION IF EXISTS mercgrid;\n\n\nCREATE OR REPLACE FUNCTION ftime() RETURNS interval as $$\nSELECT age(clock_timestamp(), transaction_timestamp());\n$$ LANGUAGE SQL;\nSET SEARCH_PATH to pgstac, public;\n\nDROP FUNCTION IF EXISTS geometrysearch;\nCREATE OR REPLACE FUNCTION geometrysearch(\n IN geom geometry,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE, -- Return as soon as the passed in geometry is full covered\n IN skipcovered boolean DEFAULT TRUE -- Skip any items that would show up completely under the previous items\n) RETURNS jsonb AS $$\nDECLARE\n search searches%ROWTYPE;\n curs refcursor;\n _where text;\n query text;\n iter_record items%ROWTYPE;\n out_records jsonb := '{}'::jsonb[];\n exit_flag boolean := FALSE;\n counter int := 1;\n scancounter int := 1;\n remaining_limit int := _scanlimit;\n tilearea float;\n unionedgeom geometry;\n clippedgeom geometry;\n unionedgeom_area float := 0;\n prev_area float := 0;\n excludes text[];\n includes text[];\n\nBEGIN\n DROP TABLE IF EXISTS pgstac_results;\n CREATE TEMP TABLE pgstac_results (content jsonb) ON COMMIT DROP;\n\n -- If the passed in geometry is not an area set exitwhenfull and skipcovered to false\n IF ST_GeometryType(geom) !~* 'polygon' THEN\n RAISE NOTICE 'GEOMETRY IS NOT AN AREA';\n skipcovered = FALSE;\n exitwhenfull = FALSE;\n END IF;\n\n -- If skipcovered is true then you will always want to exit when the passed in geometry is full\n IF skipcovered THEN\n exitwhenfull := TRUE;\n END IF;\n\n search := search_fromhash(queryhash);\n\n IF search IS NULL THEN\n RAISE EXCEPTION 'Search with Query Hash % Not Found', queryhash;\n END IF;\n\n tilearea := st_area(geom);\n _where := format('%s AND st_intersects(geometry, %L::geometry)', search._where, geom);\n\n\n FOR query IN SELECT * FROM partition_queries(_where, search.orderby) LOOP\n query := format('%s LIMIT %L', query, remaining_limit);\n RAISE NOTICE '%', query;\n OPEN curs FOR EXECUTE query;\n LOOP\n FETCH curs INTO iter_record;\n EXIT WHEN NOT FOUND;\n IF exitwhenfull OR skipcovered THEN -- If we are not using exitwhenfull or skipcovered, we do not need to do expensive geometry operations\n clippedgeom := st_intersection(geom, iter_record.geometry);\n\n IF unionedgeom IS NULL THEN\n unionedgeom := clippedgeom;\n ELSE\n unionedgeom := st_union(unionedgeom, clippedgeom);\n END IF;\n\n unionedgeom_area := st_area(unionedgeom);\n\n IF skipcovered AND prev_area = unionedgeom_area THEN\n scancounter := scancounter + 1;\n CONTINUE;\n END IF;\n\n prev_area := unionedgeom_area;\n\n RAISE NOTICE '% % % %', unionedgeom_area/tilearea, counter, scancounter, ftime();\n END IF;\n RAISE NOTICE '% %', iter_record, content_hydrate(iter_record, fields);\n INSERT INTO pgstac_results (content) VALUES (content_hydrate(iter_record, fields));\n\n IF counter >= _limit\n OR scancounter > _scanlimit\n OR ftime() > _timelimit\n OR (exitwhenfull AND unionedgeom_area >= tilearea)\n THEN\n exit_flag := TRUE;\n EXIT;\n END IF;\n counter := counter + 1;\n scancounter := scancounter + 1;\n\n END LOOP;\n CLOSE curs;\n EXIT WHEN exit_flag;\n remaining_limit := _scanlimit - scancounter;\n END LOOP;\n\n SELECT jsonb_agg(content) INTO out_records FROM pgstac_results WHERE content IS NOT NULL;\n\n RETURN jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb)\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDROP FUNCTION IF EXISTS geojsonsearch;\nCREATE OR REPLACE FUNCTION geojsonsearch(\n IN geojson jsonb,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_geomfromgeojson(geojson),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS xyzsearch;\nCREATE OR REPLACE FUNCTION xyzsearch(\n IN _x int,\n IN _y int,\n IN _z int,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_transform(tileenvelope(_z, _x, _y), 4326),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE PROCEDURE analyze_items() AS $$\nDECLARE\n q text;\n timeout_ts timestamptz;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n SELECT format('ANALYZE (VERBOSE, SKIP_LOCKED) %I;', relname) INTO q\n FROM pg_stat_user_tables\n WHERE relname like '_item%' AND (n_mod_since_analyze>0 OR last_analyze IS NULL) LIMIT 1;\n IF NOT FOUND THEN\n EXIT;\n END IF;\n RAISE NOTICE '%', q;\n EXECUTE q;\n COMMIT;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE PROCEDURE validate_constraints() AS $$\nDECLARE\n q text;\nBEGIN\n FOR q IN\n SELECT\n FORMAT(\n 'ALTER TABLE %I.%I VALIDATE CONSTRAINT %I;',\n nsp.nspname,\n cls.relname,\n con.conname\n )\n\n FROM pg_constraint AS con\n JOIN pg_class AS cls\n ON con.conrelid = cls.oid\n JOIN pg_namespace AS nsp\n ON cls.relnamespace = nsp.oid\n WHERE convalidated = FALSE AND contype in ('c','f')\n AND nsp.nspname = 'pgstac'\n LOOP\n RAISE NOTICE '%', q;\n PERFORM run_or_queue(q);\n COMMIT;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION collection_extent(_collection text, runupdate boolean default false) RETURNS jsonb AS $$\nDECLARE\n geom_extent geometry;\n mind timestamptz;\n maxd timestamptz;\n extent jsonb;\nBEGIN\n IF runupdate THEN\n PERFORM update_partition_stats_q(partition)\n FROM partitions_view WHERE collection=_collection;\n END IF;\n SELECT\n min(lower(dtrange)),\n max(upper(edtrange)),\n st_extent(spatial)\n INTO\n mind,\n maxd,\n geom_extent\n FROM partitions_view\n WHERE collection=_collection;\n\n IF geom_extent IS NOT NULL AND mind IS NOT NULL AND maxd IS NOT NULL THEN\n extent := jsonb_build_object(\n 'extent', jsonb_build_object(\n 'spatial', jsonb_build_object(\n 'bbox', to_jsonb(array[array[st_xmin(geom_extent), st_ymin(geom_extent), st_xmax(geom_extent), st_ymax(geom_extent)]])\n ),\n 'temporal', jsonb_build_object(\n 'interval', to_jsonb(array[array[mind, maxd]])\n )\n )\n );\n RETURN extent;\n END IF;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('id', '{\"title\": \"Item ID\",\"description\": \"Item identifier\",\"$ref\": \"https://schemas.stacspec.org/v1.0.0/item-spec/json-schema/item.json#/definitions/core/allOf/2/properties/id\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('geometry', '{\"title\": \"Item Geometry\",\"description\": \"Item Geometry\",\"$ref\": \"https://geojson.org/schema/Feature.json\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('datetime','{\"description\": \"Datetime\",\"type\": \"string\",\"title\": \"Acquired\",\"format\": \"date-time\",\"pattern\": \"(\\\\+00:00|Z)$\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDELETE FROM queryables a USING queryables b\n WHERE a.name = b.name AND a.collection_ids IS NOT DISTINCT FROM b.collection_ids AND a.id > b.id;\n\n\nINSERT INTO pgstac_settings (name, value) VALUES\n ('context', 'off'),\n ('context_estimated_count', '100000'),\n ('context_estimated_cost', '100000'),\n ('context_stats_ttl', '1 day'),\n ('default_filter_lang', 'cql2-json'),\n ('additional_properties', 'true'),\n ('use_queue', 'false'),\n ('queue_timeout', '10 minutes'),\n ('update_collection_extent', 'false'),\n ('format_cache', 'false'),\n ('readonly', 'false')\nON CONFLICT DO NOTHING\n;\n\n\nINSERT INTO cql2_ops (op, template, types) VALUES\n ('eq', '%s = %s', NULL),\n ('neq', '%s != %s', NULL),\n ('ne', '%s != %s', NULL),\n ('!=', '%s != %s', NULL),\n ('<>', '%s != %s', NULL),\n ('lt', '%s < %s', NULL),\n ('lte', '%s <= %s', NULL),\n ('gt', '%s > %s', NULL),\n ('gte', '%s >= %s', NULL),\n ('le', '%s <= %s', NULL),\n ('ge', '%s >= %s', NULL),\n ('=', '%s = %s', NULL),\n ('<', '%s < %s', NULL),\n ('<=', '%s <= %s', NULL),\n ('>', '%s > %s', NULL),\n ('>=', '%s >= %s', NULL),\n ('like', '%s LIKE %s', NULL),\n ('ilike', '%s ILIKE %s', NULL),\n ('+', '%s + %s', NULL),\n ('-', '%s - %s', NULL),\n ('*', '%s * %s', NULL),\n ('/', '%s / %s', NULL),\n ('not', 'NOT (%s)', NULL),\n ('between', '%s BETWEEN %s AND %s', NULL),\n ('isnull', '%s IS NULL', NULL),\n ('upper', 'upper(%s)', NULL),\n ('lower', 'lower(%s)', NULL),\n ('casei', 'upper(%s)', NULL),\n ('accenti', 'unaccent(%s)', NULL)\nON CONFLICT (op) DO UPDATE\n SET\n template = EXCLUDED.template\n;\n\n\nALTER FUNCTION to_text COST 5000;\nALTER FUNCTION to_float COST 5000;\nALTER FUNCTION to_int COST 5000;\nALTER FUNCTION to_tstz COST 5000;\nALTER FUNCTION to_text_array COST 5000;\n\nALTER FUNCTION update_partition_stats SECURITY DEFINER;\nALTER FUNCTION partition_after_triggerfunc SECURITY DEFINER;\nALTER FUNCTION drop_table_constraints SECURITY DEFINER;\nALTER FUNCTION create_table_constraints SECURITY DEFINER;\nALTER FUNCTION check_partition SECURITY DEFINER;\nALTER FUNCTION repartition SECURITY DEFINER;\nALTER FUNCTION where_stats SECURITY DEFINER;\nALTER FUNCTION search_query SECURITY DEFINER;\nALTER FUNCTION format_item SECURITY DEFINER;\nALTER FUNCTION maintain_index SECURITY DEFINER;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nGRANT ALL ON SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac to pgstac_admin;\n\n-- pgstac_read role limited to using function apis\nGRANT EXECUTE ON FUNCTION search TO pgstac_read;\nGRANT EXECUTE ON FUNCTION search_query TO pgstac_read;\nGRANT EXECUTE ON FUNCTION item_by_id TO pgstac_read;\nGRANT EXECUTE ON FUNCTION get_item TO pgstac_read;\nGRANT SELECT ON ALL TABLES IN SCHEMA pgstac TO pgstac_read;\n\n\nGRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_ingest;\nGRANT USAGE ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_ingest;\n\nREVOKE ALL PRIVILEGES ON PROCEDURE run_queued_queries FROM public;\nGRANT ALL ON PROCEDURE run_queued_queries TO pgstac_admin;\n\nREVOKE ALL PRIVILEGES ON FUNCTION run_queued_queries_intransaction FROM public;\nGRANT ALL ON FUNCTION run_queued_queries_intransaction TO pgstac_admin;\n\nRESET ROLE;\n\nSET ROLE pgstac_ingest;\nSELECT update_partition_stats_q(partition) FROM partitions_view;\nSELECT set_version('0.9.0');\n" }, { "path": "src/pgstac/migrations/pgstac.0.9.1-0.9.2.sql", "content": "SET client_min_messages TO WARNING;\nSET SEARCH_PATH to pgstac, public;\nRESET ROLE;\nDO $$\nDECLARE\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='postgis') THEN\n CREATE EXTENSION IF NOT EXISTS postgis;\n END IF;\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='btree_gist') THEN\n CREATE EXTENSION IF NOT EXISTS btree_gist;\n END IF;\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='unaccent') THEN\n CREATE EXTENSION IF NOT EXISTS unaccent;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_admin;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_read;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_ingest;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n\nGRANT pgstac_admin TO current_user;\n\n-- Function to make sure pgstac_admin is the owner of items\nCREATE OR REPLACE FUNCTION pgstac_admin_owns() RETURNS VOID AS $$\nDECLARE\n f RECORD;\nBEGIN\n FOR f IN (\n SELECT\n concat(\n oid::regproc::text,\n '(',\n coalesce(pg_get_function_identity_arguments(oid),''),\n ')'\n ) AS name,\n CASE prokind WHEN 'f' THEN 'FUNCTION' WHEN 'p' THEN 'PROCEDURE' WHEN 'a' THEN 'AGGREGATE' END as typ\n FROM pg_proc\n WHERE\n pronamespace=to_regnamespace('pgstac')\n AND proowner != to_regrole('pgstac_admin')\n AND proname NOT LIKE 'pg_stat%'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n FOR f IN (\n SELECT\n oid::regclass::text as name,\n CASE relkind\n WHEN 'i' THEN 'INDEX'\n WHEN 'I' THEN 'INDEX'\n WHEN 'p' THEN 'TABLE'\n WHEN 'r' THEN 'TABLE'\n WHEN 'v' THEN 'VIEW'\n WHEN 'S' THEN 'SEQUENCE'\n ELSE NULL\n END as typ\n FROM pg_class\n WHERE relnamespace=to_regnamespace('pgstac') and relowner != to_regrole('pgstac_admin') AND relkind IN ('r','p','v','S') AND relname NOT LIKE 'pg_stat'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\nSELECT pgstac_admin_owns();\n\nCREATE SCHEMA IF NOT EXISTS pgstac AUTHORIZATION pgstac_admin;\n\nGRANT ALL ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_admin;\n\nALTER ROLE pgstac_admin SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_read SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_ingest SET SEARCH_PATH TO pgstac, public;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\n\nGRANT pgstac_read TO pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\n\nSET ROLE pgstac_admin;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\nRESET ROLE;\n\nSET ROLE pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\nRESET ROLE;\n\nSET SEARCH_PATH TO pgstac, public;\nSET ROLE pgstac_admin;\n\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS analyze_items;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS validate_constraints;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n-- Install these idempotently as migrations do not put them before trying to modify the collections table\n\n\nCREATE OR REPLACE FUNCTION collection_geom(content jsonb)\nRETURNS geometry AS $$\n WITH box AS (SELECT content->'extent'->'spatial'->'bbox'->0 as box)\n SELECT\n st_makeenvelope(\n (box->>0)::float,\n (box->>1)::float,\n (box->>2)::float,\n (box->>3)::float,\n 4326\n )\n FROM box;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION collection_datetime(content jsonb)\nRETURNS timestamptz AS $$\n SELECT\n CASE\n WHEN\n (content->'extent'->'temporal'->'interval'->0->>0) IS NULL\n THEN '-infinity'::timestamptz\n ELSE\n (content->'extent'->'temporal'->'interval'->0->>0)::timestamptz\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION collection_enddatetime(content jsonb)\nRETURNS timestamptz AS $$\n SELECT\n CASE\n WHEN\n (content->'extent'->'temporal'->'interval'->0->>1) IS NULL\n THEN 'infinity'::timestamptz\n ELSE\n (content->'extent'->'temporal'->'interval'->0->>1)::timestamptz\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n-- BEGIN migra calculated SQL\nset check_function_bodies = off;\n\nCREATE OR REPLACE FUNCTION pgstac.q_to_tsquery(input text)\n RETURNS tsquery\n LANGUAGE plpgsql\nAS $function$\nDECLARE\n processed_text text;\n temp_text text;\n quote_array text[];\n placeholder text := '@QUOTE@';\nBEGIN\n -- Extract all quoted phrases and store in array\n quote_array := regexp_matches(input, '\"[^\"]*\"', 'g');\n\n -- Replace each quoted part with a unique placeholder if there are any quoted phrases\n IF array_length(quote_array, 1) IS NOT NULL THEN\n processed_text := input;\n FOR i IN array_lower(quote_array, 1) .. array_upper(quote_array, 1) LOOP\n processed_text := replace(processed_text, quote_array[i], placeholder || i || placeholder);\n END LOOP;\n ELSE\n processed_text := input;\n END IF;\n\n -- Replace non-quoted text using regular expressions\n\n -- , -> |\n processed_text := regexp_replace(processed_text, ',(?=(?:[^\"]*\"[^\"]*\")*[^\"]*$)', ' | ', 'g');\n\n -- and -> &\n processed_text := regexp_replace(processed_text, '\\s+AND\\s+', ' & ', 'gi');\n\n -- or -> |\n processed_text := regexp_replace(processed_text, '\\s+OR\\s+', ' | ', 'gi');\n\n -- +term -> & term\n processed_text := regexp_replace(processed_text, '\\+([a-zA-Z0-9_]+)', '& \\1', 'g');\n\n -- -term -> ! term\n processed_text := regexp_replace(processed_text, '\\-([a-zA-Z0-9_]+)', '& ! \\1', 'g');\n\n -- Replace placeholders back with quoted phrases if there were any\n IF array_length(quote_array, 1) IS NOT NULL THEN\n FOR i IN array_lower(quote_array, 1) .. array_upper(quote_array, 1) LOOP\n processed_text := replace(processed_text, placeholder || i || placeholder, '''' || substring(quote_array[i] from 2 for length(quote_array[i]) - 2) || '''');\n END LOOP;\n END IF;\n\n -- Print processed_text to the console for debugging purposes\n RAISE NOTICE 'processed_text: %', processed_text;\n\n RETURN to_tsquery(processed_text);\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.chunker(_where text, OUT s timestamp with time zone, OUT e timestamp with time zone)\n RETURNS SETOF record\n LANGUAGE plpgsql\nAS $function$\nDECLARE\n explain jsonb;\nBEGIN\n IF _where IS NULL THEN\n _where := ' TRUE ';\n END IF;\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s;', _where)\n INTO explain;\n RAISE DEBUG 'EXPLAIN: %', explain;\n\n RETURN QUERY\n WITH t AS (\n SELECT j->>0 as p FROM\n jsonb_path_query(\n explain,\n 'strict $.**.\"Relation Name\" ? (@ != null)'\n ) j\n ),\n parts AS (\n SELECT sdate, edate FROM t JOIN partition_steps ON (t.p = name)\n ),\n times AS (\n SELECT sdate FROM parts\n UNION\n SELECT edate FROM parts\n ),\n uniq AS (\n SELECT DISTINCT sdate FROM times ORDER BY sdate\n ),\n last AS (\n SELECT sdate, lead(sdate, 1) over () as edate FROM uniq\n )\n SELECT sdate, edate FROM last WHERE edate IS NOT NULL;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.collection_search(_search jsonb DEFAULT '{}'::jsonb)\n RETURNS jsonb\n LANGUAGE plpgsql\n STABLE PARALLEL SAFE\nAS $function$\nDECLARE\n out_records jsonb;\n number_matched bigint := collection_search_matched(_search);\n number_returned bigint;\n _limit int := coalesce((_search->>'limit')::float::int, 10);\n _offset int := coalesce((_search->>'offset')::float::int, 0);\n links jsonb := '[]';\n ret jsonb;\n base_url text:= concat(rtrim(base_url(_search->'conf'),'/'), '/collections');\n prevoffset int;\n nextoffset int;\nBEGIN\n SELECT\n coalesce(jsonb_agg(c), '[]')\n INTO out_records\n FROM collection_search_rows(_search) c;\n\n number_returned := jsonb_array_length(out_records);\n RAISE DEBUG 'nm: %, nr: %, l:%, o:%', number_matched, number_returned, _limit, _offset;\n\n\n\n IF _limit <= number_matched AND number_matched > 0 THEN --need to have paging links\n nextoffset := least(_offset + _limit, number_matched - 1);\n prevoffset := greatest(_offset - _limit, 0);\n\n IF _offset > 0 THEN\n links := links || jsonb_build_object(\n 'rel', 'prev',\n 'type', 'application/json',\n 'method', 'GET' ,\n 'href', base_url,\n 'body', jsonb_build_object('offset', prevoffset),\n 'merge', TRUE\n );\n END IF;\n\n IF (_offset + _limit < number_matched) THEN\n links := links || jsonb_build_object(\n 'rel', 'next',\n 'type', 'application/json',\n 'method', 'GET' ,\n 'href', base_url,\n 'body', jsonb_build_object('offset', nextoffset),\n 'merge', TRUE\n );\n END IF;\n\n END IF;\n\n ret := jsonb_build_object(\n 'collections', out_records,\n 'numberMatched', number_matched,\n 'numberReturned', number_returned,\n 'links', links\n );\n RETURN ret;\n\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.maintain_partition_queries(part text DEFAULT 'items'::text, dropindexes boolean DEFAULT false, rebuildindexes boolean DEFAULT false, idxconcurrently boolean DEFAULT false)\n RETURNS SETOF text\n LANGUAGE plpgsql\nAS $function$\nDECLARE\n rec record;\n q text;\nBEGIN\n FOR rec IN (\n SELECT * FROM queryable_indexes(part,true)\n ) LOOP\n q := format(\n 'SELECT maintain_index(\n %L,%L,%L,%L,%L\n );',\n rec.indexname,\n rec.queryable_idx,\n dropindexes,\n rebuildindexes,\n idxconcurrently\n );\n RAISE NOTICE 'Q: %', q;\n RETURN NEXT q;\n END LOOP;\n RETURN;\nEND;\n$function$\n;\n\ncreate or replace view \"pgstac\".\"partition_sys_meta\" as SELECT (parse_ident((pg_partition_tree.relid)::text))[cardinality(parse_ident((pg_partition_tree.relid)::text))] AS partition,\n replace(replace(\n CASE\n WHEN (pg_partition_tree.level = 1) THEN pg_get_expr(c.relpartbound, c.oid)\n ELSE pg_get_expr(parent.relpartbound, parent.oid)\n END, 'FOR VALUES IN ('''::text, ''::text), ''')'::text, ''::text) AS collection,\n pg_partition_tree.level,\n c.reltuples,\n c.relhastriggers,\n COALESCE(constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity'::timestamp with time zone, 'infinity'::timestamp with time zone, '[]'::text)) AS partition_dtrange,\n COALESCE((dt_constraint(edt.oid)).dt, constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity'::timestamp with time zone, 'infinity'::timestamp with time zone, '[]'::text)) AS constraint_dtrange,\n COALESCE((dt_constraint(edt.oid)).edt, tstzrange('-infinity'::timestamp with time zone, 'infinity'::timestamp with time zone, '[]'::text)) AS constraint_edtrange\n FROM (((pg_partition_tree('items'::regclass) pg_partition_tree(relid, parentrelid, isleaf, level)\n JOIN pg_class c ON (((pg_partition_tree.relid)::oid = c.oid)))\n JOIN pg_class parent ON ((((pg_partition_tree.parentrelid)::oid = parent.oid) AND pg_partition_tree.isleaf)))\n LEFT JOIN pg_constraint edt ON (((edt.conrelid = c.oid) AND (edt.contype = 'c'::\"char\"))))\n WHERE pg_partition_tree.isleaf;\n\n\ncreate or replace view \"pgstac\".\"partitions_view\" as SELECT (parse_ident((pg_partition_tree.relid)::text))[cardinality(parse_ident((pg_partition_tree.relid)::text))] AS partition,\n replace(replace(\n CASE\n WHEN (pg_partition_tree.level = 1) THEN pg_get_expr(c.relpartbound, c.oid)\n ELSE pg_get_expr(parent.relpartbound, parent.oid)\n END, 'FOR VALUES IN ('''::text, ''::text), ''')'::text, ''::text) AS collection,\n pg_partition_tree.level,\n c.reltuples,\n c.relhastriggers,\n COALESCE(constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity'::timestamp with time zone, 'infinity'::timestamp with time zone, '[]'::text)) AS partition_dtrange,\n COALESCE((dt_constraint(edt.oid)).dt, constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity'::timestamp with time zone, 'infinity'::timestamp with time zone, '[]'::text)) AS constraint_dtrange,\n COALESCE((dt_constraint(edt.oid)).edt, tstzrange('-infinity'::timestamp with time zone, 'infinity'::timestamp with time zone, '[]'::text)) AS constraint_edtrange,\n partition_stats.dtrange,\n partition_stats.edtrange,\n partition_stats.spatial,\n partition_stats.last_updated\n FROM ((((pg_partition_tree('items'::regclass) pg_partition_tree(relid, parentrelid, isleaf, level)\n JOIN pg_class c ON (((pg_partition_tree.relid)::oid = c.oid)))\n JOIN pg_class parent ON ((((pg_partition_tree.parentrelid)::oid = parent.oid) AND pg_partition_tree.isleaf)))\n LEFT JOIN pg_constraint edt ON (((edt.conrelid = c.oid) AND (edt.contype = 'c'::\"char\"))))\n LEFT JOIN partition_stats ON (((parse_ident((pg_partition_tree.relid)::text))[cardinality(parse_ident((pg_partition_tree.relid)::text))] = partition_stats.partition)))\n WHERE pg_partition_tree.isleaf;\n\n\nCREATE OR REPLACE FUNCTION pgstac.queryable(dotpath text, OUT path text, OUT expression text, OUT wrapper text, OUT nulled_wrapper text)\n RETURNS record\n LANGUAGE plpgsql\n STABLE STRICT\nAS $function$\nDECLARE\n q RECORD;\n path_elements text[];\nBEGIN\n dotpath := replace(dotpath, 'properties.', '');\n IF dotpath = 'start_datetime' THEN\n dotpath := 'datetime';\n END IF;\n IF dotpath IN ('id', 'geometry', 'datetime', 'end_datetime', 'collection') THEN\n path := dotpath;\n expression := dotpath;\n wrapper := NULL;\n RETURN;\n END IF;\n\n SELECT * INTO q FROM queryables\n WHERE\n name=dotpath\n OR name = 'properties.' || dotpath\n OR name = replace(dotpath, 'properties.', '')\n ;\n IF q.property_wrapper IS NULL THEN\n IF q.definition->>'type' = 'number' THEN\n wrapper := 'to_float';\n nulled_wrapper := wrapper;\n ELSIF q.definition->>'format' = 'date-time' THEN\n wrapper := 'to_tstz';\n nulled_wrapper := wrapper;\n ELSE\n nulled_wrapper := NULL;\n wrapper := 'to_text';\n END IF;\n ELSE\n wrapper := q.property_wrapper;\n nulled_wrapper := wrapper;\n END IF;\n IF q.property_path IS NOT NULL THEN\n path := q.property_path;\n ELSE\n path_elements := string_to_array(dotpath, '.');\n IF path_elements[1] IN ('links', 'assets', 'stac_version', 'stac_extensions') THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSIF path_elements[1] = 'properties' THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSE\n path := format($F$content->'properties'->%s$F$, array_to_path(path_elements));\n END IF;\n END IF;\n expression := format('%I(%s)', wrapper, path);\n RETURN;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.stac_search_to_where(j jsonb)\n RETURNS text\n LANGUAGE plpgsql\n STABLE\nAS $function$\nDECLARE\n where_segments text[];\n _where text;\n dtrange tstzrange;\n collections text[];\n geom geometry;\n sdate timestamptz;\n edate timestamptz;\n filterlang text;\n filter jsonb := j->'filter';\n ft_query tsquery;\nBEGIN\n IF j ? 'ids' THEN\n where_segments := where_segments || format('id = ANY (%L) ', to_text_array(j->'ids'));\n END IF;\n\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n where_segments := where_segments || format('collection = ANY (%L) ', collections);\n END IF;\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n\n where_segments := where_segments || format(' datetime <= %L::timestamptz AND end_datetime >= %L::timestamptz ',\n edate,\n sdate\n );\n END IF;\n\n IF j ? 'q' THEN\n ft_query := q_to_tsquery(j->>'q');\n where_segments := where_segments || format(\n $quote$\n (\n to_tsvector('english', content->'properties'->>'description') ||\n to_tsvector('english', content->'properties'->>'title') ||\n to_tsvector('english', content->'properties'->'keywords')\n ) @@ %L\n $quote$,\n ft_query\n );\n END IF;\n\n geom := stac_geom(j);\n IF geom IS NOT NULL THEN\n where_segments := where_segments || format('st_intersects(geometry, %L)',geom);\n END IF;\n\n filterlang := COALESCE(\n j->>'filter-lang',\n get_setting('default_filter_lang', j->'conf')\n );\n IF NOT filter @? '$.**.op' THEN\n filterlang := 'cql-json';\n END IF;\n\n IF filterlang NOT IN ('cql-json','cql2-json') AND j ? 'filter' THEN\n RAISE EXCEPTION '% is not a supported filter-lang. Please use cql-json or cql2-json.', filterlang;\n END IF;\n\n IF j ? 'query' AND j ? 'filter' THEN\n RAISE EXCEPTION 'Can only use either query or filter at one time.';\n END IF;\n\n IF j ? 'query' THEN\n filter := query_to_cql2(j->'query');\n ELSIF filterlang = 'cql-json' THEN\n filter := cql1_to_cql2(filter);\n END IF;\n RAISE NOTICE 'FILTER: %', filter;\n where_segments := where_segments || cql2_query(filter);\n IF cardinality(where_segments) < 1 THEN\n RETURN ' TRUE ';\n END IF;\n\n _where := array_to_string(array_remove(where_segments, NULL), ' AND ');\n\n IF _where IS NULL OR BTRIM(_where) = '' THEN\n RETURN ' TRUE ';\n END IF;\n RETURN _where;\n\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.update_partition_stats(_partition text, istrigger boolean DEFAULT false)\n RETURNS void\n LANGUAGE plpgsql\n STRICT SECURITY DEFINER\nAS $function$\nDECLARE\n dtrange tstzrange;\n edtrange tstzrange;\n cdtrange tstzrange;\n cedtrange tstzrange;\n extent geometry;\n collection text;\nBEGIN\n RAISE NOTICE 'Updating stats for %.', _partition;\n EXECUTE format(\n $q$\n SELECT\n tstzrange(min(datetime), max(datetime),'[]'),\n tstzrange(min(end_datetime), max(end_datetime), '[]')\n FROM %I\n $q$,\n _partition\n ) INTO dtrange, edtrange;\n extent := st_estimatedextent('pgstac', _partition, 'geometry');\n RAISE DEBUG 'Estimated Extent: %', extent;\n INSERT INTO partition_stats (partition, dtrange, edtrange, spatial, last_updated)\n SELECT _partition, dtrange, edtrange, extent, now()\n ON CONFLICT (partition) DO\n UPDATE SET\n dtrange=EXCLUDED.dtrange,\n edtrange=EXCLUDED.edtrange,\n spatial=EXCLUDED.spatial,\n last_updated=EXCLUDED.last_updated\n ;\n\n SELECT\n constraint_dtrange, constraint_edtrange, pv.collection\n INTO cdtrange, cedtrange, collection\n FROM partitions_view pv WHERE partition = _partition;\n REFRESH MATERIALIZED VIEW partitions;\n REFRESH MATERIALIZED VIEW partition_steps;\n\n\n RAISE NOTICE 'Checking if we need to modify constraints...';\n RAISE NOTICE 'cdtrange: % dtrange: % cedtrange: % edtrange: %',cdtrange, dtrange, cedtrange, edtrange;\n IF\n (cdtrange IS DISTINCT FROM dtrange OR edtrange IS DISTINCT FROM cedtrange)\n AND NOT istrigger\n THEN\n RAISE NOTICE 'Modifying Constraints';\n RAISE NOTICE 'Existing % %', cdtrange, cedtrange;\n RAISE NOTICE 'New % %', dtrange, edtrange;\n PERFORM drop_table_constraints(_partition);\n PERFORM create_table_constraints(_partition, dtrange, edtrange);\n REFRESH MATERIALIZED VIEW partitions;\n REFRESH MATERIALIZED VIEW partition_steps;\n END IF;\n RAISE NOTICE 'Checking if we need to update collection extents.';\n IF get_setting_bool('update_collection_extent') THEN\n RAISE NOTICE 'updating collection extent for %', collection;\n PERFORM run_or_queue(format($q$\n UPDATE collections\n SET content = jsonb_set_lax(\n content,\n '{extent}'::text[],\n collection_extent(%L, FALSE),\n true,\n 'use_json_null'\n ) WHERE id=%L\n ;\n $q$, collection, collection));\n ELSE\n RAISE NOTICE 'Not updating collection extent for %', collection;\n END IF;\n\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.where_stats(inwhere text, updatestats boolean DEFAULT false, conf jsonb DEFAULT NULL::jsonb)\n RETURNS search_wheres\n LANGUAGE plpgsql\n SECURITY DEFINER\nAS $function$\nDECLARE\n t timestamptz;\n i interval;\n explain_json jsonb;\n partitions text[];\n sw search_wheres%ROWTYPE;\n inwhere_hash text := md5(inwhere);\n _context text := lower(context(conf));\n _stats_ttl interval := context_stats_ttl(conf);\n _estimated_cost_threshold float := context_estimated_cost(conf);\n _estimated_count_threshold int := context_estimated_count(conf);\n ro bool := pgstac.readonly(conf);\nBEGIN\n -- If updatestats is true then set ttl to 0\n IF updatestats THEN\n RAISE DEBUG 'Updatestats set to TRUE, setting TTL to 0';\n _stats_ttl := '0'::interval;\n END IF;\n\n -- If we don't need to calculate context, just return\n IF _context = 'off' THEN\n sw._where = inwhere;\n RETURN sw;\n END IF;\n\n -- Get any stats that we have.\n IF NOT ro THEN\n -- If there is a lock where another process is\n -- updating the stats, wait so that we don't end up calculating a bunch of times.\n SELECT * INTO sw FROM search_wheres WHERE md5(_where)=inwhere_hash FOR UPDATE;\n ELSE\n SELECT * INTO sw FROM search_wheres WHERE md5(_where)=inwhere_hash;\n END IF;\n\n -- If there is a cached row, figure out if we need to update\n IF\n sw IS NOT NULL\n AND sw.statslastupdated IS NOT NULL\n AND sw.total_count IS NOT NULL\n AND now() - sw.statslastupdated <= _stats_ttl\n THEN\n -- we have a cached row with data that is within our ttl\n RAISE DEBUG 'Stats present in table and lastupdated within ttl: %', sw;\n IF NOT ro THEN\n RAISE DEBUG 'Updating search_wheres only bumping lastused and usecount';\n UPDATE search_wheres SET\n lastused = now(),\n usecount = search_wheres.usecount + 1\n WHERE md5(_where) = inwhere_hash\n RETURNING * INTO sw;\n END IF;\n RAISE DEBUG 'Returning cached counts. %', sw;\n RETURN sw;\n END IF;\n\n -- Calculate estimated cost and rows\n -- Use explain to get estimated count/cost\n IF sw.estimated_count IS NULL OR sw.estimated_cost IS NULL THEN\n RAISE DEBUG 'Calculating estimated stats';\n t := clock_timestamp();\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s', inwhere)\n INTO explain_json;\n RAISE DEBUG 'Time for just the explain: %', clock_timestamp() - t;\n i := clock_timestamp() - t;\n\n sw.estimated_count := explain_json->0->'Plan'->'Plan Rows';\n sw.estimated_cost := explain_json->0->'Plan'->'Total Cost';\n sw.time_to_estimate := extract(epoch from i);\n END IF;\n\n RAISE DEBUG 'ESTIMATED_COUNT: %, THRESHOLD %', sw.estimated_count, _estimated_count_threshold;\n RAISE DEBUG 'ESTIMATED_COST: %, THRESHOLD %', sw.estimated_cost, _estimated_cost_threshold;\n\n -- If context is set to auto and the costs are within the threshold return the estimated costs\n IF\n _context = 'auto'\n AND sw.estimated_count >= _estimated_count_threshold\n AND sw.estimated_cost >= _estimated_cost_threshold\n THEN\n IF NOT ro THEN\n INSERT INTO search_wheres (\n _where,\n lastused,\n usecount,\n statslastupdated,\n estimated_count,\n estimated_cost,\n time_to_estimate,\n total_count,\n time_to_count\n ) VALUES (\n inwhere,\n now(),\n 1,\n now(),\n sw.estimated_count,\n sw.estimated_cost,\n sw.time_to_estimate,\n null,\n null\n ) ON CONFLICT ((md5(_where)))\n DO UPDATE SET\n lastused = EXCLUDED.lastused,\n usecount = search_wheres.usecount + 1,\n statslastupdated = EXCLUDED.statslastupdated,\n estimated_count = EXCLUDED.estimated_count,\n estimated_cost = EXCLUDED.estimated_cost,\n time_to_estimate = EXCLUDED.time_to_estimate,\n total_count = EXCLUDED.total_count,\n time_to_count = EXCLUDED.time_to_count\n RETURNING * INTO sw;\n END IF;\n RAISE DEBUG 'Estimates are within thresholds, returning estimates. %', sw;\n RETURN sw;\n END IF;\n\n -- Calculate Actual Count\n t := clock_timestamp();\n RAISE NOTICE 'Calculating actual count...';\n EXECUTE format(\n 'SELECT count(*) FROM items WHERE %s',\n inwhere\n ) INTO sw.total_count;\n i := clock_timestamp() - t;\n RAISE NOTICE 'Actual Count: % -- %', sw.total_count, i;\n sw.time_to_count := extract(epoch FROM i);\n\n IF NOT ro THEN\n INSERT INTO search_wheres (\n _where,\n lastused,\n usecount,\n statslastupdated,\n estimated_count,\n estimated_cost,\n time_to_estimate,\n total_count,\n time_to_count\n ) VALUES (\n inwhere,\n now(),\n 1,\n now(),\n sw.estimated_count,\n sw.estimated_cost,\n sw.time_to_estimate,\n sw.total_count,\n sw.time_to_count\n ) ON CONFLICT ((md5(_where)))\n DO UPDATE SET\n lastused = EXCLUDED.lastused,\n usecount = search_wheres.usecount + 1,\n statslastupdated = EXCLUDED.statslastupdated,\n estimated_count = EXCLUDED.estimated_count,\n estimated_cost = EXCLUDED.estimated_cost,\n time_to_estimate = EXCLUDED.time_to_estimate,\n total_count = EXCLUDED.total_count,\n time_to_count = EXCLUDED.time_to_count\n RETURNING * INTO sw;\n END IF;\n RAISE DEBUG 'Returning with actual count. %', sw;\n RETURN sw;\nEND;\n$function$\n;\n\n\n-- END migra calculated SQL\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('id', '{\"title\": \"Item ID\",\"description\": \"Item identifier\",\"$ref\": \"https://schemas.stacspec.org/v1.0.0/item-spec/json-schema/item.json#/definitions/core/allOf/2/properties/id\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('geometry', '{\"title\": \"Item Geometry\",\"description\": \"Item Geometry\",\"$ref\": \"https://geojson.org/schema/Feature.json\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('datetime','{\"description\": \"Datetime\",\"type\": \"string\",\"title\": \"Acquired\",\"format\": \"date-time\",\"pattern\": \"(\\\\+00:00|Z)$\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDELETE FROM queryables a USING queryables b\n WHERE a.name = b.name AND a.collection_ids IS NOT DISTINCT FROM b.collection_ids AND a.id > b.id;\n\n\nINSERT INTO pgstac_settings (name, value) VALUES\n ('context', 'off'),\n ('context_estimated_count', '100000'),\n ('context_estimated_cost', '100000'),\n ('context_stats_ttl', '1 day'),\n ('default_filter_lang', 'cql2-json'),\n ('additional_properties', 'true'),\n ('use_queue', 'false'),\n ('queue_timeout', '10 minutes'),\n ('update_collection_extent', 'false'),\n ('format_cache', 'false'),\n ('readonly', 'false')\nON CONFLICT DO NOTHING\n;\n\n\nINSERT INTO cql2_ops (op, template, types) VALUES\n ('eq', '%s = %s', NULL),\n ('neq', '%s != %s', NULL),\n ('ne', '%s != %s', NULL),\n ('!=', '%s != %s', NULL),\n ('<>', '%s != %s', NULL),\n ('lt', '%s < %s', NULL),\n ('lte', '%s <= %s', NULL),\n ('gt', '%s > %s', NULL),\n ('gte', '%s >= %s', NULL),\n ('le', '%s <= %s', NULL),\n ('ge', '%s >= %s', NULL),\n ('=', '%s = %s', NULL),\n ('<', '%s < %s', NULL),\n ('<=', '%s <= %s', NULL),\n ('>', '%s > %s', NULL),\n ('>=', '%s >= %s', NULL),\n ('like', '%s LIKE %s', NULL),\n ('ilike', '%s ILIKE %s', NULL),\n ('+', '%s + %s', NULL),\n ('-', '%s - %s', NULL),\n ('*', '%s * %s', NULL),\n ('/', '%s / %s', NULL),\n ('not', 'NOT (%s)', NULL),\n ('between', '%s BETWEEN %s AND %s', NULL),\n ('isnull', '%s IS NULL', NULL),\n ('upper', 'upper(%s)', NULL),\n ('lower', 'lower(%s)', NULL),\n ('casei', 'upper(%s)', NULL),\n ('accenti', 'unaccent(%s)', NULL)\nON CONFLICT (op) DO UPDATE\n SET\n template = EXCLUDED.template\n;\n\n\nALTER FUNCTION to_text COST 5000;\nALTER FUNCTION to_float COST 5000;\nALTER FUNCTION to_int COST 5000;\nALTER FUNCTION to_tstz COST 5000;\nALTER FUNCTION to_text_array COST 5000;\n\nALTER FUNCTION update_partition_stats SECURITY DEFINER;\nALTER FUNCTION partition_after_triggerfunc SECURITY DEFINER;\nALTER FUNCTION drop_table_constraints SECURITY DEFINER;\nALTER FUNCTION create_table_constraints SECURITY DEFINER;\nALTER FUNCTION check_partition SECURITY DEFINER;\nALTER FUNCTION repartition SECURITY DEFINER;\nALTER FUNCTION where_stats SECURITY DEFINER;\nALTER FUNCTION search_query SECURITY DEFINER;\nALTER FUNCTION format_item SECURITY DEFINER;\nALTER FUNCTION maintain_index SECURITY DEFINER;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nGRANT ALL ON SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac to pgstac_admin;\n\n-- pgstac_read role limited to using function apis\nGRANT EXECUTE ON FUNCTION search TO pgstac_read;\nGRANT EXECUTE ON FUNCTION search_query TO pgstac_read;\nGRANT EXECUTE ON FUNCTION item_by_id TO pgstac_read;\nGRANT EXECUTE ON FUNCTION get_item TO pgstac_read;\nGRANT SELECT ON ALL TABLES IN SCHEMA pgstac TO pgstac_read;\n\n\nGRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_ingest;\nGRANT USAGE ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_ingest;\n\nREVOKE ALL PRIVILEGES ON PROCEDURE run_queued_queries FROM public;\nGRANT ALL ON PROCEDURE run_queued_queries TO pgstac_admin;\n\nREVOKE ALL PRIVILEGES ON FUNCTION run_queued_queries_intransaction FROM public;\nGRANT ALL ON FUNCTION run_queued_queries_intransaction TO pgstac_admin;\n\nRESET ROLE;\n\nSET ROLE pgstac_ingest;\nSELECT update_partition_stats_q(partition) FROM partitions_view;\nSELECT set_version('0.9.2');\n" }, { "path": "src/pgstac/migrations/pgstac.0.9.1.sql", "content": "RESET ROLE;\nDO $$\nDECLARE\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='postgis') THEN\n CREATE EXTENSION IF NOT EXISTS postgis;\n END IF;\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='btree_gist') THEN\n CREATE EXTENSION IF NOT EXISTS btree_gist;\n END IF;\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='unaccent') THEN\n CREATE EXTENSION IF NOT EXISTS unaccent;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_admin;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_read;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_ingest;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n\nGRANT pgstac_admin TO current_user;\n\n-- Function to make sure pgstac_admin is the owner of items\nCREATE OR REPLACE FUNCTION pgstac_admin_owns() RETURNS VOID AS $$\nDECLARE\n f RECORD;\nBEGIN\n FOR f IN (\n SELECT\n concat(\n oid::regproc::text,\n '(',\n coalesce(pg_get_function_identity_arguments(oid),''),\n ')'\n ) AS name,\n CASE prokind WHEN 'f' THEN 'FUNCTION' WHEN 'p' THEN 'PROCEDURE' WHEN 'a' THEN 'AGGREGATE' END as typ\n FROM pg_proc\n WHERE\n pronamespace=to_regnamespace('pgstac')\n AND proowner != to_regrole('pgstac_admin')\n AND proname NOT LIKE 'pg_stat%'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n FOR f IN (\n SELECT\n oid::regclass::text as name,\n CASE relkind\n WHEN 'i' THEN 'INDEX'\n WHEN 'I' THEN 'INDEX'\n WHEN 'p' THEN 'TABLE'\n WHEN 'r' THEN 'TABLE'\n WHEN 'v' THEN 'VIEW'\n WHEN 'S' THEN 'SEQUENCE'\n ELSE NULL\n END as typ\n FROM pg_class\n WHERE relnamespace=to_regnamespace('pgstac') and relowner != to_regrole('pgstac_admin') AND relkind IN ('r','p','v','S') AND relname NOT LIKE 'pg_stat'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\nSELECT pgstac_admin_owns();\n\nCREATE SCHEMA IF NOT EXISTS pgstac AUTHORIZATION pgstac_admin;\n\nGRANT ALL ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_admin;\n\nALTER ROLE pgstac_admin SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_read SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_ingest SET SEARCH_PATH TO pgstac, public;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\n\nGRANT pgstac_read TO pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\n\nSET ROLE pgstac_admin;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\nRESET ROLE;\n\nSET ROLE pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\nRESET ROLE;\n\nSET SEARCH_PATH TO pgstac, public;\nSET ROLE pgstac_admin;\n\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS analyze_items;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS validate_constraints;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n-- Install these idempotently as migrations do not put them before trying to modify the collections table\n\n\nCREATE OR REPLACE FUNCTION collection_geom(content jsonb)\nRETURNS geometry AS $$\n WITH box AS (SELECT content->'extent'->'spatial'->'bbox'->0 as box)\n SELECT\n st_makeenvelope(\n (box->>0)::float,\n (box->>1)::float,\n (box->>2)::float,\n (box->>3)::float,\n 4326\n )\n FROM box;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION collection_datetime(content jsonb)\nRETURNS timestamptz AS $$\n SELECT\n CASE\n WHEN\n (content->'extent'->'temporal'->'interval'->0->>0) IS NULL\n THEN '-infinity'::timestamptz\n ELSE\n (content->'extent'->'temporal'->'interval'->0->>0)::timestamptz\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION collection_enddatetime(content jsonb)\nRETURNS timestamptz AS $$\n SELECT\n CASE\n WHEN\n (content->'extent'->'temporal'->'interval'->0->>1) IS NULL\n THEN 'infinity'::timestamptz\n ELSE\n (content->'extent'->'temporal'->'interval'->0->>1)::timestamptz\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE TABLE IF NOT EXISTS migrations (\n version text PRIMARY KEY,\n datetime timestamptz DEFAULT clock_timestamp() NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION get_version() RETURNS text AS $$\n SELECT version FROM pgstac.migrations ORDER BY datetime DESC, version DESC LIMIT 1;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION set_version(text) RETURNS text AS $$\n INSERT INTO pgstac.migrations (version) VALUES ($1)\n ON CONFLICT DO NOTHING\n RETURNING version;\n$$ LANGUAGE SQL;\n\n\nCREATE TABLE IF NOT EXISTS pgstac_settings (\n name text PRIMARY KEY,\n value text NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION table_empty(text) RETURNS boolean AS $$\nDECLARE\n retval boolean;\nBEGIN\n EXECUTE format($q$\n SELECT NOT EXISTS (SELECT 1 FROM %I LIMIT 1)\n $q$,\n $1\n ) INTO retval;\n RETURN retval;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION get_setting(IN _setting text, IN conf jsonb DEFAULT NULL) RETURNS text AS $$\nSELECT COALESCE(\n nullif(conf->>_setting, ''),\n nullif(current_setting(concat('pgstac.',_setting), TRUE),''),\n nullif((SELECT value FROM pgstac.pgstac_settings WHERE name=_setting),'')\n);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_setting_bool(IN _setting text, IN conf jsonb DEFAULT NULL) RETURNS boolean AS $$\nSELECT COALESCE(\n nullif(conf->>_setting, ''),\n nullif(current_setting(concat('pgstac.',_setting), TRUE),''),\n nullif((SELECT value FROM pgstac.pgstac_settings WHERE name=_setting),''),\n 'FALSE'\n)::boolean;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION base_url(conf jsonb DEFAULT NULL) RETURNS text AS $$\n SELECT COALESCE(pgstac.get_setting('base_url', conf), '.');\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION additional_properties() RETURNS boolean AS $$\n SELECT pgstac.get_setting_bool('additional_properties');\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION readonly(conf jsonb DEFAULT NULL) RETURNS boolean AS $$\n SELECT pgstac.get_setting_bool('readonly', conf);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context(conf jsonb DEFAULT NULL) RETURNS text AS $$\n SELECT pgstac.get_setting('context', conf);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context_estimated_count(conf jsonb DEFAULT NULL) RETURNS int AS $$\n SELECT pgstac.get_setting('context_estimated_count', conf)::int;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_estimated_cost();\nCREATE OR REPLACE FUNCTION context_estimated_cost(conf jsonb DEFAULT NULL) RETURNS float AS $$\n SELECT pgstac.get_setting('context_estimated_cost', conf)::float;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_stats_ttl();\nCREATE OR REPLACE FUNCTION context_stats_ttl(conf jsonb DEFAULT NULL) RETURNS interval AS $$\n SELECT pgstac.get_setting('context_stats_ttl', conf)::interval;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION t2s(text) RETURNS text AS $$\n SELECT extract(epoch FROM $1::interval)::text || ' s';\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE STRICT;\n\nCREATE OR REPLACE FUNCTION age_ms(a timestamptz, b timestamptz DEFAULT clock_timestamp()) RETURNS float AS $$\n SELECT abs(extract(epoch from age(a,b)) * 1000);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION queue_timeout() RETURNS interval AS $$\n SELECT t2s(coalesce(\n get_setting('queue_timeout'),\n '1h'\n ))::interval;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION notice(VARIADIC text[]) RETURNS boolean AS $$\nDECLARE\ndebug boolean := current_setting('pgstac.debug', true);\nBEGIN\n IF debug THEN\n RAISE NOTICE 'NOTICE FROM FUNC: % >>>>> %', concat_ws(' | ', $1), clock_timestamp();\n RETURN TRUE;\n END IF;\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION empty_arr(ANYARRAY) RETURNS BOOLEAN AS $$\nSELECT CASE\n WHEN $1 IS NULL THEN TRUE\n WHEN cardinality($1)<1 THEN TRUE\nELSE FALSE\nEND;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_intersection(_a ANYARRAY, _b ANYARRAY) RETURNS ANYARRAY AS $$\n SELECT ARRAY ( SELECT unnest(_a) INTERSECT SELECT UNNEST(_b) );\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION array_map_ident(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_ident(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_map_literal(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_literal(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION array_reverse(anyarray) RETURNS anyarray AS $$\nSELECT ARRAY(\n SELECT $1[i]\n FROM generate_subscripts($1,1) AS s(i)\n ORDER BY i DESC\n);\n$$ LANGUAGE SQL STRICT IMMUTABLE;\n\nDROP TABLE IF EXISTS query_queue;\nCREATE TABLE query_queue (\n query text PRIMARY KEY,\n added timestamptz DEFAULT now()\n);\n\nDROP TABLE IF EXISTS query_queue_history;\nCREATE TABLE query_queue_history(\n query text,\n added timestamptz NOT NULL,\n finished timestamptz NOT NULL DEFAULT now(),\n error text\n);\n\nCREATE OR REPLACE PROCEDURE run_queued_queries() AS $$\nDECLARE\n qitem query_queue%ROWTYPE;\n timeout_ts timestamptz;\n error text;\n cnt int := 0;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n DELETE FROM query_queue WHERE query = (SELECT query FROM query_queue ORDER BY added DESC LIMIT 1 FOR UPDATE SKIP LOCKED) RETURNING * INTO qitem;\n IF NOT FOUND THEN\n EXIT;\n END IF;\n cnt := cnt + 1;\n BEGIN\n RAISE NOTICE 'RUNNING QUERY: %', qitem.query;\n EXECUTE qitem.query;\n EXCEPTION WHEN others THEN\n error := format('%s | %s', SQLERRM, SQLSTATE);\n END;\n INSERT INTO query_queue_history (query, added, finished, error)\n VALUES (qitem.query, qitem.added, clock_timestamp(), error);\n COMMIT;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION run_queued_queries_intransaction() RETURNS int AS $$\nDECLARE\n qitem query_queue%ROWTYPE;\n timeout_ts timestamptz;\n error text;\n cnt int := 0;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n DELETE FROM query_queue WHERE query = (SELECT query FROM query_queue ORDER BY added DESC LIMIT 1 FOR UPDATE SKIP LOCKED) RETURNING * INTO qitem;\n IF NOT FOUND THEN\n RETURN cnt;\n END IF;\n cnt := cnt + 1;\n BEGIN\n qitem.query := regexp_replace(qitem.query, 'CONCURRENTLY', '');\n RAISE NOTICE 'RUNNING QUERY: %', qitem.query;\n\n EXECUTE qitem.query;\n EXCEPTION WHEN others THEN\n error := format('%s | %s', SQLERRM, SQLSTATE);\n RAISE WARNING '%', error;\n END;\n INSERT INTO query_queue_history (query, added, finished, error)\n VALUES (qitem.query, qitem.added, clock_timestamp(), error);\n END LOOP;\n RETURN cnt;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION run_or_queue(query text) RETURNS VOID AS $$\nDECLARE\n use_queue text := COALESCE(get_setting('use_queue'), 'FALSE')::boolean;\nBEGIN\n IF get_setting_bool('debug') THEN\n RAISE NOTICE '%', query;\n END IF;\n IF use_queue THEN\n INSERT INTO query_queue (query) VALUES (query) ON CONFLICT DO NOTHING;\n ELSE\n EXECUTE query;\n END IF;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n\nDROP FUNCTION IF EXISTS check_pgstac_settings;\nCREATE OR REPLACE FUNCTION check_pgstac_settings(_sysmem text DEFAULT NULL) RETURNS VOID AS $$\nDECLARE\n settingval text;\n sysmem bigint := pg_size_bytes(_sysmem);\n effective_cache_size bigint := pg_size_bytes(current_setting('effective_cache_size', TRUE));\n shared_buffers bigint := pg_size_bytes(current_setting('shared_buffers', TRUE));\n work_mem bigint := pg_size_bytes(current_setting('work_mem', TRUE));\n max_connections int := current_setting('max_connections', TRUE);\n maintenance_work_mem bigint := pg_size_bytes(current_setting('maintenance_work_mem', TRUE));\n seq_page_cost float := current_setting('seq_page_cost', TRUE);\n random_page_cost float := current_setting('random_page_cost', TRUE);\n temp_buffers bigint := pg_size_bytes(current_setting('temp_buffers', TRUE));\n r record;\nBEGIN\n IF _sysmem IS NULL THEN\n RAISE NOTICE 'Call function with the size of your system memory `SELECT check_pgstac_settings(''4GB'')` to get pg system setting recommendations.';\n ELSE\n IF effective_cache_size < (sysmem * 0.5) THEN\n RAISE WARNING 'effective_cache_size of % is set low for a system with %. Recomended value between % and %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.5), pg_size_pretty(sysmem * 0.75);\n ELSIF effective_cache_size > (sysmem * 0.75) THEN\n RAISE WARNING 'effective_cache_size of % is set high for a system with %. Recomended value between % and %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.5), pg_size_pretty(sysmem * 0.75);\n ELSE\n RAISE NOTICE 'effective_cache_size of % is set appropriately for a system with %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem);\n END IF;\n\n IF shared_buffers < (sysmem * 0.2) THEN\n RAISE WARNING 'shared_buffers of % is set low for a system with %. Recomended value between % and %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.2), pg_size_pretty(sysmem * 0.3);\n ELSIF shared_buffers > (sysmem * 0.3) THEN\n RAISE WARNING 'shared_buffers of % is set high for a system with %. Recomended value between % and %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.2), pg_size_pretty(sysmem * 0.3);\n ELSE\n RAISE NOTICE 'shared_buffers of % is set appropriately for a system with %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem);\n END IF;\n shared_buffers = sysmem * 0.3;\n IF maintenance_work_mem < (sysmem * 0.2) THEN\n RAISE WARNING 'maintenance_work_mem of % is set low for shared_buffers of %. Recomended value between % and %', pg_size_pretty(maintenance_work_mem), pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers * 0.2), pg_size_pretty(shared_buffers * 0.3);\n ELSIF maintenance_work_mem > (shared_buffers * 0.3) THEN\n RAISE WARNING 'maintenance_work_mem of % is set high for shared_buffers of %. Recomended value between % and %', pg_size_pretty(maintenance_work_mem), pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers * 0.2), pg_size_pretty(shared_buffers * 0.3);\n ELSE\n RAISE NOTICE 'maintenance_work_mem of % is set appropriately for shared_buffers of %', pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers);\n END IF;\n\n IF work_mem * max_connections > shared_buffers THEN\n RAISE WARNING 'work_mem setting of % is set high for % max_connections please reduce work_mem to % or decrease max_connections to %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers/max_connections), floor(shared_buffers/work_mem);\n ELSIF work_mem * max_connections < (shared_buffers * 0.75) THEN\n RAISE WARNING 'work_mem setting of % is set low for % max_connections you may consider raising work_mem to % or increasing max_connections to %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers/max_connections), floor(shared_buffers/work_mem);\n ELSE\n RAISE NOTICE 'work_mem setting of % and max_connections of % are adequate for shared_buffers of %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers);\n END IF;\n\n IF random_page_cost / seq_page_cost != 1.1 THEN\n RAISE WARNING 'random_page_cost (%) /seq_page_cost (%) should be set to 1.1 for SSD. Change random_page_cost to %', random_page_cost, seq_page_cost, 1.1 * seq_page_cost;\n ELSE\n RAISE NOTICE 'random_page_cost and seq_page_cost set appropriately for SSD';\n END IF;\n\n IF temp_buffers < greatest(pg_size_bytes('128MB'),(maintenance_work_mem / 2)) THEN\n RAISE WARNING 'pgstac makes heavy use of temp tables, consider raising temp_buffers from % to %', pg_size_pretty(temp_buffers), greatest('128MB', pg_size_pretty((shared_buffers / 16)));\n END IF;\n END IF;\n\n RAISE NOTICE 'VALUES FOR PGSTAC VARIABLES';\n RAISE NOTICE 'These can be set either as GUC system variables or by setting in the pgstac_settings table.';\n\n FOR r IN SELECT name, get_setting(name) as setting, CASE WHEN current_setting(concat('pgstac.',name), TRUE) IS NOT NULL THEN concat('pgstac.',name, ' GUC') WHEN value IS NOT NULL THEN 'pgstac_settings table' ELSE 'Not Set' END as loc FROM pgstac_settings LOOP\n RAISE NOTICE '% is set to % from the %', r.name, r.setting, r.loc;\n END LOOP;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pg_cron';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider intalling pg_cron which can be used to automate tasks';\n ELSE\n RAISE NOTICE 'pg_cron % is installed', settingval;\n END IF;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pgstattuple';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider installing the pgstattuple extension which can be used to help maintain tables and indexes.';\n ELSE\n RAISE NOTICE 'pgstattuple % is installed', settingval;\n END IF;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pg_stat_statements';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider installing the pg_stat_statements extension which is very helpful for tracking the types of queries on the system';\n ELSE\n RAISE NOTICE 'pg_stat_statements % is installed', settingval;\n IF current_setting('pg_stat_statements.track_statements', TRUE) IS DISTINCT FROM 'all' THEN\n RAISE WARNING 'SET pg_stat_statements.track_statements TO ''all''; --In order to track statements within functions.';\n END IF;\n END IF;\n\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac, public SET CLIENT_MIN_MESSAGES TO NOTICE;\nCREATE OR REPLACE FUNCTION to_int(jsonb) RETURNS int AS $$\n SELECT floor(($1->>0)::float)::int;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION to_float(jsonb) RETURNS float AS $$\n SELECT ($1->>0)::float;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION to_tstz(jsonb) RETURNS timestamptz AS $$\n SELECT ($1->>0)::timestamptz;\n$$ LANGUAGE SQL IMMUTABLE STRICT SET TIME ZONE 'UTC' COST 5000 PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION to_text(jsonb) RETURNS text AS $$\n SELECT CASE WHEN jsonb_typeof($1) IN ('array','object') THEN $1::text ELSE $1->>0 END;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION to_text_array(jsonb) RETURNS text[] AS $$\n SELECT\n CASE jsonb_typeof($1)\n WHEN 'array' THEN ARRAY(SELECT jsonb_array_elements_text($1))\n ELSE ARRAY[$1->>0]\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION bbox_geom(_bbox jsonb) RETURNS geometry AS $$\nSELECT CASE jsonb_array_length(_bbox)\n WHEN 4 THEN\n ST_SetSRID(ST_MakeEnvelope(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float,\n (_bbox->>3)::float\n ),4326)\n WHEN 6 THEN\n ST_SetSRID(ST_3DMakeBox(\n ST_MakePoint(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float\n ),\n ST_MakePoint(\n (_bbox->>3)::float,\n (_bbox->>4)::float,\n (_bbox->>5)::float\n )\n ),4326)\n ELSE null END;\n;\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION geom_bbox(_geom geometry) RETURNS jsonb AS $$\n SELECT jsonb_build_array(\n st_xmin(_geom),\n st_ymin(_geom),\n st_xmax(_geom),\n st_ymax(_geom)\n );\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION flip_jsonb_array(j jsonb) RETURNS jsonb AS $$\n SELECT jsonb_agg(value) FROM (SELECT value FROM jsonb_array_elements(j) WITH ORDINALITY ORDER BY ordinality DESC) as t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION explode_dotpaths(j jsonb) RETURNS SETOF text[] AS $$\n SELECT string_to_array(p, '.') as e FROM jsonb_array_elements_text(j) p;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION explode_dotpaths_recurse(IN j jsonb) RETURNS SETOF text[] AS $$\n WITH RECURSIVE t AS (\n SELECT e FROM explode_dotpaths(j) e\n UNION ALL\n SELECT e[1:cardinality(e)-1]\n FROM t\n WHERE cardinality(e)>1\n ) SELECT e FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION jsonb_set_nested(j jsonb, path text[], val jsonb) RETURNS jsonb AS $$\nDECLARE\nBEGIN\n IF cardinality(path) > 1 THEN\n FOR i IN 1..(cardinality(path)-1) LOOP\n IF j #> path[:i] IS NULL THEN\n j := jsonb_set_lax(j, path[:i], '{}', TRUE);\n END IF;\n END LOOP;\n END IF;\n RETURN jsonb_set_lax(j, path, val, true);\n\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\n\n\nCREATE OR REPLACE FUNCTION jsonb_include(j jsonb, f jsonb) RETURNS jsonb AS $$\nDECLARE\n includes jsonb := f-> 'include';\n outj jsonb := '{}'::jsonb;\n path text[];\nBEGIN\n IF\n includes IS NULL\n OR jsonb_array_length(includes) = 0\n THEN\n RETURN j;\n ELSE\n includes := includes || '[\"id\",\"collection\"]'::jsonb;\n FOR path IN SELECT explode_dotpaths(includes) LOOP\n outj := jsonb_set_nested(outj, path, j #> path);\n END LOOP;\n END IF;\n RETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_exclude(j jsonb, f jsonb) RETURNS jsonb AS $$\nDECLARE\n excludes jsonb := f-> 'exclude';\n outj jsonb := j;\n path text[];\nBEGIN\n IF\n excludes IS NULL\n OR jsonb_array_length(excludes) = 0\n THEN\n RETURN j;\n ELSE\n FOR path IN SELECT explode_dotpaths(excludes) LOOP\n outj := outj #- path;\n END LOOP;\n END IF;\n RETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_fields(j jsonb, f jsonb DEFAULT '{\"fields\":[]}') RETURNS jsonb AS $$\n SELECT jsonb_exclude(jsonb_include(j, f), f);\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION merge_jsonb(_a jsonb, _b jsonb) RETURNS jsonb AS $$\n SELECT\n CASE\n WHEN _a = '\"𒍟※\"'::jsonb THEN NULL\n WHEN _a IS NULL OR jsonb_typeof(_a) = 'null' THEN _b\n WHEN jsonb_typeof(_a) = 'object' AND jsonb_typeof(_b) = 'object' THEN\n (\n SELECT\n jsonb_strip_nulls(\n jsonb_object_agg(\n key,\n merge_jsonb(a.value, b.value)\n )\n )\n FROM\n jsonb_each(coalesce(_a,'{}'::jsonb)) as a\n FULL JOIN\n jsonb_each(coalesce(_b,'{}'::jsonb)) as b\n USING (key)\n )\n WHEN\n jsonb_typeof(_a) = 'array'\n AND jsonb_typeof(_b) = 'array'\n AND jsonb_array_length(_a) = jsonb_array_length(_b)\n THEN\n (\n SELECT jsonb_agg(m) FROM\n ( SELECT\n merge_jsonb(\n jsonb_array_elements(_a),\n jsonb_array_elements(_b)\n ) as m\n ) as l\n )\n ELSE _a\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION strip_jsonb(_a jsonb, _b jsonb) RETURNS jsonb AS $$\n SELECT\n CASE\n\n WHEN (_a IS NULL OR jsonb_typeof(_a) = 'null') AND _b IS NOT NULL AND jsonb_typeof(_b) != 'null' THEN '\"𒍟※\"'::jsonb\n WHEN _b IS NULL OR jsonb_typeof(_a) = 'null' THEN _a\n WHEN _a = _b AND jsonb_typeof(_a) = 'object' THEN '{}'::jsonb\n WHEN _a = _b THEN NULL\n WHEN jsonb_typeof(_a) = 'object' AND jsonb_typeof(_b) = 'object' THEN\n (\n SELECT\n jsonb_strip_nulls(\n jsonb_object_agg(\n key,\n strip_jsonb(a.value, b.value)\n )\n )\n FROM\n jsonb_each(_a) as a\n FULL JOIN\n jsonb_each(_b) as b\n USING (key)\n )\n WHEN\n jsonb_typeof(_a) = 'array'\n AND jsonb_typeof(_b) = 'array'\n AND jsonb_array_length(_a) = jsonb_array_length(_b)\n THEN\n (\n SELECT jsonb_agg(m) FROM\n ( SELECT\n strip_jsonb(\n jsonb_array_elements(_a),\n jsonb_array_elements(_b)\n ) as m\n ) as l\n )\n ELSE _a\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION nullif_jsonbnullempty(j jsonb) RETURNS jsonb AS $$\n SELECT nullif(nullif(nullif(j,'null'::jsonb),'{}'::jsonb),'[]'::jsonb);\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION jsonb_array_unique(j jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(jsonb_agg(DISTINCT a)) v FROM jsonb_array_elements(j) a;\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_concat_ignorenull(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT coalesce(a,'[]'::jsonb) || coalesce(b,'[]'::jsonb);\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_least(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(least(nullif_jsonbnullempty(a), nullif_jsonbnullempty(b)));\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_greatest(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(greatest(a, b));\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION first_notnull_sfunc(anyelement, anyelement) RETURNS anyelement AS $$\n SELECT COALESCE($1,$2);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE AGGREGATE first_notnull(anyelement)(\n SFUNC = first_notnull_sfunc,\n STYPE = anyelement\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_array_unique_merge(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_concat_ignorenull,\n FINALFUNC = jsonb_array_unique\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_min(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_least\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_max(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_greatest\n);\n/* looks for a geometry in a stac item first from geometry and falling back to bbox */\nCREATE OR REPLACE FUNCTION stac_geom(value jsonb) RETURNS geometry AS $$\nSELECT\n CASE\n WHEN value ? 'intersects' THEN\n ST_GeomFromGeoJSON(value->>'intersects')\n WHEN value ? 'geometry' THEN\n ST_GeomFromGeoJSON(value->>'geometry')\n WHEN value ? 'bbox' THEN\n pgstac.bbox_geom(value->'bbox')\n ELSE NULL\n END as geometry\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION stac_daterange(\n value jsonb\n) RETURNS tstzrange AS $$\nDECLARE\n props jsonb := value;\n dt timestamptz;\n edt timestamptz;\nBEGIN\n IF props ? 'properties' THEN\n props := props->'properties';\n END IF;\n IF\n props ? 'start_datetime'\n AND props->>'start_datetime' IS NOT NULL\n AND props ? 'end_datetime'\n AND props->>'end_datetime' IS NOT NULL\n THEN\n dt := props->>'start_datetime';\n edt := props->>'end_datetime';\n IF dt > edt THEN\n RAISE EXCEPTION 'start_datetime must be < end_datetime';\n END IF;\n ELSE\n dt := props->>'datetime';\n edt := props->>'datetime';\n END IF;\n IF dt is NULL OR edt IS NULL THEN\n RAISE NOTICE 'DT: %, EDT: %', dt, edt;\n RAISE EXCEPTION 'Either datetime (%) or both start_datetime (%) and end_datetime (%) must be set.', props->>'datetime',props->>'start_datetime',props->>'end_datetime';\n END IF;\n RETURN tstzrange(dt, edt, '[]');\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_datetime(value jsonb) RETURNS timestamptz AS $$\n SELECT lower(stac_daterange(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_end_datetime(value jsonb) RETURNS timestamptz AS $$\n SELECT upper(stac_daterange(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE TABLE IF NOT EXISTS stac_extensions(\n url text PRIMARY KEY,\n content jsonb\n);\nCREATE OR REPLACE FUNCTION collection_base_item(content jsonb) RETURNS jsonb AS $$\n SELECT jsonb_build_object(\n 'type', 'Feature',\n 'stac_version', content->'stac_version',\n 'assets', content->'item_assets',\n 'collection', content->'id'\n );\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE TABLE IF NOT EXISTS collections (\n key bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,\n id text GENERATED ALWAYS AS (content->>'id') STORED UNIQUE NOT NULL,\n content JSONB NOT NULL,\n base_item jsonb GENERATED ALWAYS AS (pgstac.collection_base_item(content)) STORED,\n geometry geometry GENERATED ALWAYS AS (pgstac.collection_geom(content)) STORED,\n datetime timestamptz GENERATED ALWAYS AS (pgstac.collection_datetime(content)) STORED,\n end_datetime timestamptz GENERATED ALWAYS AS (pgstac.collection_enddatetime(content)) STORED,\n private jsonb,\n partition_trunc text CHECK (partition_trunc IN ('year', 'month'))\n);\n\nCREATE OR REPLACE FUNCTION collection_base_item(cid text) RETURNS jsonb AS $$\n SELECT pgstac.collection_base_item(content) FROM pgstac.collections WHERE id = cid LIMIT 1;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION create_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION update_collection(data jsonb) RETURNS VOID AS $$\nDECLARE\n out collections%ROWTYPE;\nBEGIN\n UPDATE collections SET content=data WHERE id = data->>'id' RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ON CONFLICT (id) DO\n UPDATE\n SET content=EXCLUDED.content\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION delete_collection(_id text) RETURNS VOID AS $$\nDECLARE\n out collections%ROWTYPE;\nBEGIN\n DELETE FROM collections WHERE id = _id RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION get_collection(id text) RETURNS jsonb AS $$\n SELECT content FROM collections\n WHERE id=$1\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION all_collections() RETURNS jsonb AS $$\n SELECT coalesce(jsonb_agg(content), '[]'::jsonb) FROM collections;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION collection_delete_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n collection_base_partition text := concat('_items_', OLD.key);\nBEGIN\n EXECUTE format($q$\n DELETE FROM partition_stats WHERE partition IN (\n SELECT partition FROM partition_sys_meta\n WHERE collection=%L\n );\n DROP TABLE IF EXISTS %I CASCADE;\n $q$,\n OLD.id,\n collection_base_partition\n );\n RETURN OLD;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER collection_delete_trigger BEFORE DELETE ON collections\nFOR EACH ROW EXECUTE FUNCTION collection_delete_trigger_func();\nCREATE OR REPLACE FUNCTION queryable_signature(n text, c text[]) RETURNS text AS $$\n SELECT concat(n, c);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TABLE queryables (\n id bigint GENERATED ALWAYS AS identity PRIMARY KEY,\n name text NOT NULL,\n collection_ids text[], -- used to determine what partitions to create indexes on\n definition jsonb,\n property_path text,\n property_wrapper text,\n property_index_type text\n);\nCREATE INDEX queryables_name_idx ON queryables (name);\nCREATE INDEX queryables_collection_idx ON queryables USING GIN (collection_ids);\nCREATE INDEX queryables_property_wrapper_idx ON queryables (property_wrapper);\n\nCREATE OR REPLACE FUNCTION pgstac.queryables_constraint_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n allcollections text[];\nBEGIN\n RAISE NOTICE 'Making sure that name/collection is unique for queryables %', NEW;\n IF NEW.collection_ids IS NOT NULL THEN\n IF EXISTS (\n SELECT 1\n FROM unnest(NEW.collection_ids) c\n LEFT JOIN\n collections\n ON (collections.id = c)\n WHERE collections.id IS NULL\n ) THEN\n RAISE foreign_key_violation USING MESSAGE = format(\n 'One or more collections in %s do not exist.', NEW.collection_ids\n );\n RETURN NULL;\n END IF;\n END IF;\n IF TG_OP = 'INSERT' THEN\n IF EXISTS (\n SELECT 1 FROM queryables q\n WHERE\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n )\n ) THEN\n RAISE unique_violation USING MESSAGE = format(\n 'There is already a queryable for %s for a collection in %s: %s',\n NEW.name,\n NEW.collection_ids,\n\t\t\t\t(SELECT json_agg(row_to_json(q)) FROM queryables q WHERE\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n ))\n );\n RETURN NULL;\n END IF;\n END IF;\n IF TG_OP = 'UPDATE' THEN\n IF EXISTS (\n SELECT 1 FROM queryables q\n WHERE\n q.id != NEW.id\n AND\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n )\n ) THEN\n RAISE unique_violation\n USING MESSAGE = format(\n 'There is already a queryable for %s for a collection in %s',\n NEW.name,\n NEW.collection_ids\n );\n RETURN NULL;\n END IF;\n END IF;\n\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER queryables_constraint_insert_trigger\nBEFORE INSERT ON queryables\nFOR EACH ROW EXECUTE PROCEDURE queryables_constraint_triggerfunc();\n\nCREATE TRIGGER queryables_constraint_update_trigger\nBEFORE UPDATE ON queryables\nFOR EACH ROW\nWHEN (NEW.name = OLD.name AND NEW.collection_ids IS DISTINCT FROM OLD.collection_ids)\nEXECUTE PROCEDURE queryables_constraint_triggerfunc();\n\n\nCREATE OR REPLACE FUNCTION array_to_path(arr text[]) RETURNS text AS $$\n SELECT string_agg(\n quote_literal(v),\n '->'\n ) FROM unnest(arr) v;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\n\n\nCREATE OR REPLACE FUNCTION queryable(\n IN dotpath text,\n OUT path text,\n OUT expression text,\n OUT wrapper text,\n OUT nulled_wrapper text\n) AS $$\nDECLARE\n q RECORD;\n path_elements text[];\nBEGIN\n IF dotpath IN ('id', 'geometry', 'datetime', 'end_datetime', 'collection') THEN\n path := dotpath;\n expression := dotpath;\n wrapper := NULL;\n RETURN;\n END IF;\n SELECT * INTO q FROM queryables\n WHERE\n name=dotpath\n OR name = 'properties.' || dotpath\n OR name = replace(dotpath, 'properties.', '')\n ;\n IF q.property_wrapper IS NULL THEN\n IF q.definition->>'type' = 'number' THEN\n wrapper := 'to_float';\n nulled_wrapper := wrapper;\n ELSIF q.definition->>'format' = 'date-time' THEN\n wrapper := 'to_tstz';\n nulled_wrapper := wrapper;\n ELSE\n nulled_wrapper := NULL;\n wrapper := 'to_text';\n END IF;\n ELSE\n wrapper := q.property_wrapper;\n nulled_wrapper := wrapper;\n END IF;\n IF q.property_path IS NOT NULL THEN\n path := q.property_path;\n ELSE\n path_elements := string_to_array(dotpath, '.');\n IF path_elements[1] IN ('links', 'assets', 'stac_version', 'stac_extensions') THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSIF path_elements[1] = 'properties' THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSE\n path := format($F$content->'properties'->%s$F$, array_to_path(path_elements));\n END IF;\n END IF;\n expression := format('%I(%s)', wrapper, path);\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\nCREATE OR REPLACE FUNCTION unnest_collection(collection_ids text[] DEFAULT NULL) RETURNS SETOF text AS $$\n DECLARE\n BEGIN\n IF collection_ids IS NULL THEN\n RETURN QUERY SELECT id FROM collections;\n END IF;\n RETURN QUERY SELECT unnest(collection_ids);\n END;\n$$ LANGUAGE PLPGSQL STABLE;\n\nCREATE OR REPLACE FUNCTION normalize_indexdef(def text) RETURNS text AS $$\nDECLARE\nBEGIN\n def := btrim(def, ' \\n\\t');\n\tdef := regexp_replace(def, '^CREATE (UNIQUE )?INDEX ([^ ]* )?ON (ONLY )?([^ ]* )?', '', 'i');\n RETURN def;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION indexdef(q queryables) RETURNS text AS $$\n DECLARE\n out text;\n BEGIN\n IF q.name = 'id' THEN\n out := 'CREATE UNIQUE INDEX ON %I USING btree (id)';\n ELSIF q.name = 'datetime' THEN\n out := 'CREATE INDEX ON %I USING btree (datetime DESC, end_datetime)';\n ELSIF q.name = 'geometry' THEN\n out := 'CREATE INDEX ON %I USING gist (geometry)';\n ELSE\n out := format($q$CREATE INDEX ON %%I USING %s (%s(((content -> 'properties'::text) -> %L::text)))$q$,\n lower(COALESCE(q.property_index_type, 'BTREE')),\n lower(COALESCE(q.property_wrapper, 'to_text')),\n q.name\n );\n END IF;\n RETURN btrim(out, ' \\n\\t');\n END;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nDROP VIEW IF EXISTS pgstac_indexes;\nCREATE VIEW pgstac_indexes AS\nSELECT\n i.schemaname,\n i.tablename,\n i.indexname,\n regexp_replace(btrim(replace(replace(indexdef, i.indexname, ''),'pgstac.',''),' \\t\\n'), '[ ]+', ' ', 'g') as idx,\n COALESCE(\n (regexp_match(indexdef, '\\(([a-zA-Z]+)\\)'))[1],\n (regexp_match(indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_-]+)''::text'))[1],\n CASE WHEN indexdef ~* '\\(datetime desc, end_datetime\\)' THEN 'datetime' ELSE NULL END\n ) AS field,\n pg_table_size(i.indexname::text) as index_size,\n pg_size_pretty(pg_table_size(i.indexname::text)) as index_size_pretty\nFROM\n pg_indexes i\nWHERE i.schemaname='pgstac' and i.tablename ~ '_items_' AND indexdef !~* ' only ';\n\nDROP VIEW IF EXISTS pgstac_index_stats;\nCREATE VIEW pgstac_indexes_stats AS\nSELECT\n i.schemaname,\n i.tablename,\n i.indexname,\n indexdef,\n COALESCE(\n (regexp_match(indexdef, '\\(([a-zA-Z]+)\\)'))[1],\n (regexp_match(indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_]+)''::text'))[1],\n CASE WHEN indexdef ~* '\\(datetime desc, end_datetime\\)' THEN 'datetime_end_datetime' ELSE NULL END\n ) AS field,\n pg_table_size(i.indexname::text) as index_size,\n pg_size_pretty(pg_table_size(i.indexname::text)) as index_size_pretty,\n n_distinct,\n most_common_vals::text::text[],\n most_common_freqs::text::text[],\n histogram_bounds::text::text[],\n correlation\nFROM\n pg_indexes i\n LEFT JOIN pg_stats s ON (s.tablename = i.indexname)\nWHERE i.schemaname='pgstac' and i.tablename ~ '_items_';\n\nCREATE OR REPLACE FUNCTION queryable_indexes(\n IN treeroot text DEFAULT 'items',\n IN changes boolean DEFAULT FALSE,\n OUT collection text,\n OUT partition text,\n OUT field text,\n OUT indexname text,\n OUT existing_idx text,\n OUT queryable_idx text\n) RETURNS SETOF RECORD AS $$\nWITH p AS (\n SELECT\n relid::text as partition,\n replace(replace(\n CASE\n WHEN parentrelid::regclass::text='items' THEN pg_get_expr(c.relpartbound, c.oid)\n ELSE pg_get_expr(parent.relpartbound, parent.oid)\n END,\n 'FOR VALUES IN (''',''), ''')',\n ''\n ) AS collection\n FROM pg_partition_tree(treeroot)\n JOIN pg_class c ON (relid::regclass = c.oid)\n JOIN pg_class parent ON (parentrelid::regclass = parent.oid AND isleaf)\n ), i AS (\n SELECT\n partition,\n indexname,\n regexp_replace(btrim(replace(replace(indexdef, indexname, ''),'pgstac.',''),' \\t\\n'), '[ ]+', ' ', 'g') as iidx,\n COALESCE(\n (regexp_match(indexdef, '\\(([a-zA-Z]+)\\)'))[1],\n (regexp_match(indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_-]+)''::text'))[1],\n CASE WHEN indexdef ~* '\\(datetime desc, end_datetime\\)' THEN 'datetime' ELSE NULL END\n ) AS field\n FROM\n pg_indexes\n JOIN p ON (tablename=partition)\n ), q AS (\n SELECT\n name AS field,\n collection,\n partition,\n format(indexdef(queryables), partition) as qidx\n FROM queryables, unnest_collection(queryables.collection_ids) collection\n JOIN p USING (collection)\n WHERE property_index_type IS NOT NULL OR name IN ('datetime','geometry','id')\n )\n SELECT\n collection,\n partition,\n field,\n indexname,\n iidx as existing_idx,\n qidx as queryable_idx\n FROM i FULL JOIN q USING (field, partition)\n WHERE CASE WHEN changes THEN lower(iidx) IS DISTINCT FROM lower(qidx) ELSE TRUE END;\n;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION maintain_index(\n indexname text,\n queryable_idx text,\n dropindexes boolean DEFAULT FALSE,\n rebuildindexes boolean DEFAULT FALSE,\n idxconcurrently boolean DEFAULT FALSE\n) RETURNS VOID AS $$\nDECLARE\nBEGIN\n IF indexname IS NOT NULL THEN\n IF dropindexes OR queryable_idx IS NOT NULL THEN\n EXECUTE format('DROP INDEX IF EXISTS %I;', indexname);\n ELSIF rebuildindexes THEN\n IF idxconcurrently THEN\n EXECUTE format('REINDEX INDEX CONCURRENTLY %I;', indexname);\n ELSE\n EXECUTE format('REINDEX INDEX CONCURRENTLY %I;', indexname);\n END IF;\n END IF;\n END IF;\n IF queryable_idx IS NOT NULL THEN\n IF idxconcurrently THEN\n EXECUTE replace(queryable_idx, 'INDEX', 'INDEX CONCURRENTLY');\n ELSE EXECUTE queryable_idx;\n END IF;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\n\nset check_function_bodies to off;\nCREATE OR REPLACE FUNCTION maintain_partition_queries(\n part text DEFAULT 'items',\n dropindexes boolean DEFAULT FALSE,\n rebuildindexes boolean DEFAULT FALSE,\n idxconcurrently boolean DEFAULT FALSE\n) RETURNS SETOF text AS $$\nDECLARE\n rec record;\n q text;\nBEGIN\n FOR rec IN (\n SELECT * FROM queryable_indexes(part,true)\n ) LOOP\n q := format(\n 'SELECT maintain_index(\n %L,%L,%L,%L,%L\n );',\n rec.indexname,\n rec.queryable_idx,\n dropindexes,\n rebuildindexes,\n idxconcurrently\n );\n RAISE NOTICE 'Q: %s', q;\n RETURN NEXT q;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION maintain_partitions(\n part text DEFAULT 'items',\n dropindexes boolean DEFAULT FALSE,\n rebuildindexes boolean DEFAULT FALSE\n) RETURNS VOID AS $$\n WITH t AS (\n SELECT run_or_queue(q) FROM maintain_partition_queries(part, dropindexes, rebuildindexes) q\n ) SELECT count(*) FROM t;\n$$ LANGUAGE SQL;\n\n\nCREATE OR REPLACE FUNCTION queryables_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\nBEGIN\n PERFORM maintain_partitions();\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER queryables_trigger AFTER INSERT OR UPDATE ON queryables\nFOR EACH STATEMENT EXECUTE PROCEDURE queryables_trigger_func();\n\n\nCREATE OR REPLACE FUNCTION get_queryables(_collection_ids text[] DEFAULT NULL) RETURNS jsonb AS $$\nDECLARE\nBEGIN\n -- Build up queryables if the input contains valid collection ids or is empty\n IF EXISTS (\n SELECT 1 FROM collections\n WHERE\n _collection_ids IS NULL\n OR cardinality(_collection_ids) = 0\n OR id = ANY(_collection_ids)\n )\n THEN\n RETURN (\n WITH base AS (\n SELECT\n unnest(collection_ids) as collection_id,\n name,\n coalesce(definition, '{\"type\":\"string\"}'::jsonb) as definition\n FROM queryables\n WHERE\n _collection_ids IS NULL OR\n _collection_ids = '{}'::text[] OR\n _collection_ids && collection_ids\n UNION ALL\n SELECT null, name, coalesce(definition, '{\"type\":\"string\"}'::jsonb) as definition\n FROM queryables WHERE collection_ids IS NULL OR collection_ids = '{}'::text[]\n ), g AS (\n SELECT\n name,\n first_notnull(definition) as definition,\n jsonb_array_unique_merge(definition->'enum') as enum,\n jsonb_min(definition->'minimum') as minimum,\n jsonb_min(definition->'maxiumn') as maximum\n FROM base\n GROUP BY 1\n )\n SELECT\n jsonb_build_object(\n '$schema', 'http://json-schema.org/draft-07/schema#',\n '$id', '',\n 'type', 'object',\n 'title', 'STAC Queryables.',\n 'properties', jsonb_object_agg(\n name,\n definition\n ||\n jsonb_strip_nulls(jsonb_build_object(\n 'enum', enum,\n 'minimum', minimum,\n 'maximum', maximum\n ))\n ),\n 'additionalProperties', pgstac.additional_properties()\n )\n FROM g\n );\n ELSE\n RETURN NULL;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\nCREATE OR REPLACE FUNCTION get_queryables(_collection text DEFAULT NULL) RETURNS jsonb AS $$\n SELECT\n CASE\n WHEN _collection IS NULL THEN get_queryables(NULL::text[])\n ELSE get_queryables(ARRAY[_collection])\n END\n ;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_queryables() RETURNS jsonb AS $$\n SELECT get_queryables(NULL::text[]);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION schema_qualify_refs(url text, j jsonb) returns jsonb as $$\n SELECT regexp_replace(j::text, '\"\\$ref\": \"#', concat('\"$ref\": \"', url, '#'), 'g')::jsonb;\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE VIEW stac_extension_queryables AS\nSELECT DISTINCT key as name, schema_qualify_refs(e.url, j.value) as definition FROM stac_extensions e, jsonb_each(e.content->'definitions'->'fields'->'properties') j;\n\n\nCREATE OR REPLACE FUNCTION missing_queryables(_collection text, _tablesample float DEFAULT 5, minrows float DEFAULT 10) RETURNS TABLE(collection text, name text, definition jsonb, property_wrapper text) AS $$\nDECLARE\n q text;\n _partition text;\n explain_json json;\n psize float;\n estrows float;\nBEGIN\n SELECT format('_items_%s', key) INTO _partition FROM collections WHERE id=_collection;\n\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM %I;', _partition)\n INTO explain_json;\n psize := explain_json->0->'Plan'->'Plan Rows';\n estrows := _tablesample * .01 * psize;\n IF estrows < minrows THEN\n _tablesample := least(100,greatest(_tablesample, (estrows / psize) / 100));\n RAISE NOTICE '%', (psize / estrows) / 100;\n END IF;\n RAISE NOTICE 'Using tablesample % to find missing queryables from % % that has ~% rows estrows: %', _tablesample, _collection, _partition, psize, estrows;\n\n q := format(\n $q$\n WITH q AS (\n SELECT * FROM queryables\n WHERE\n collection_ids IS NULL\n OR %L = ANY(collection_ids)\n ), t AS (\n SELECT\n content->'properties' AS properties\n FROM\n %I\n TABLESAMPLE SYSTEM(%L)\n ), p AS (\n SELECT DISTINCT ON (key)\n key,\n value,\n s.definition\n FROM t\n JOIN LATERAL jsonb_each(properties) ON TRUE\n LEFT JOIN q ON (q.name=key)\n LEFT JOIN stac_extension_queryables s ON (s.name=key)\n WHERE q.definition IS NULL\n )\n SELECT\n %L,\n key,\n COALESCE(definition, jsonb_build_object('type',jsonb_typeof(value))) as definition,\n CASE\n WHEN definition->>'type' = 'integer' THEN 'to_int'\n WHEN COALESCE(definition->>'type', jsonb_typeof(value)) = 'number' THEN 'to_float'\n WHEN COALESCE(definition->>'type', jsonb_typeof(value)) = 'array' THEN 'to_text_array'\n ELSE 'to_text'\n END\n FROM p;\n $q$,\n _collection,\n _partition,\n _tablesample,\n _collection\n );\n RETURN QUERY EXECUTE q;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION missing_queryables(_tablesample float DEFAULT 5) RETURNS TABLE(collection_ids text[], name text, definition jsonb, property_wrapper text) AS $$\n SELECT\n array_agg(collection),\n name,\n definition,\n property_wrapper\n FROM\n collections\n JOIN LATERAL\n missing_queryables(id, _tablesample) c\n ON TRUE\n GROUP BY\n 2,3,4\n ORDER BY 2,1\n ;\n$$ LANGUAGE SQL;\nCREATE OR REPLACE FUNCTION parse_dtrange(\n _indate jsonb,\n relative_base timestamptz DEFAULT date_trunc('hour', CURRENT_TIMESTAMP)\n) RETURNS tstzrange AS $$\nDECLARE\n timestrs text[];\n s timestamptz;\n e timestamptz;\nBEGIN\n timestrs :=\n CASE\n WHEN _indate ? 'timestamp' THEN\n ARRAY[_indate->>'timestamp']\n WHEN _indate ? 'interval' THEN\n to_text_array(_indate->'interval')\n WHEN jsonb_typeof(_indate) = 'array' THEN\n to_text_array(_indate)\n ELSE\n regexp_split_to_array(\n _indate->>0,\n '/'\n )\n END;\n RAISE NOTICE 'TIMESTRS %', timestrs;\n IF cardinality(timestrs) = 1 THEN\n IF timestrs[1] ILIKE 'P%' THEN\n RETURN tstzrange(relative_base - upper(timestrs[1])::interval, relative_base, '[)');\n END IF;\n s := timestrs[1]::timestamptz;\n RETURN tstzrange(s, s, '[]');\n END IF;\n\n IF cardinality(timestrs) != 2 THEN\n RAISE EXCEPTION 'Timestamp cannot have more than 2 values';\n END IF;\n\n IF timestrs[1] = '..' OR timestrs[1] = '' THEN\n s := '-infinity'::timestamptz;\n e := timestrs[2]::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] = '..' OR timestrs[2] = '' THEN\n s := timestrs[1]::timestamptz;\n e := 'infinity'::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[1] ILIKE 'P%' AND timestrs[2] NOT ILIKE 'P%' THEN\n e := timestrs[2]::timestamptz;\n s := e - upper(timestrs[1])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] ILIKE 'P%' AND timestrs[1] NOT ILIKE 'P%' THEN\n s := timestrs[1]::timestamptz;\n e := s + upper(timestrs[2])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n s := timestrs[1]::timestamptz;\n e := timestrs[2]::timestamptz;\n\n RETURN tstzrange(s,e,'[)');\n\n RETURN NULL;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT PARALLEL SAFE SET TIME ZONE 'UTC';\n\nCREATE OR REPLACE FUNCTION parse_dtrange(\n _indate text,\n relative_base timestamptz DEFAULT CURRENT_TIMESTAMP\n) RETURNS tstzrange AS $$\n SELECT parse_dtrange(to_jsonb(_indate), relative_base);\n$$ LANGUAGE SQL STABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION temporal_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\n ll text := 'datetime';\n lh text := 'end_datetime';\n rrange tstzrange;\n rl text;\n rh text;\n outq text;\nBEGIN\n rrange := parse_dtrange(args->1);\n RAISE NOTICE 'Constructing temporal query OP: %, ARGS: %, RRANGE: %', op, args, rrange;\n op := lower(op);\n rl := format('%L::timestamptz', lower(rrange));\n rh := format('%L::timestamptz', upper(rrange));\n outq := CASE op\n WHEN 't_before' THEN 'lh < rl'\n WHEN 't_after' THEN 'll > rh'\n WHEN 't_meets' THEN 'lh = rl'\n WHEN 't_metby' THEN 'll = rh'\n WHEN 't_overlaps' THEN 'll < rl AND rl < lh < rh'\n WHEN 't_overlappedby' THEN 'rl < ll < rh AND lh > rh'\n WHEN 't_starts' THEN 'll = rl AND lh < rh'\n WHEN 't_startedby' THEN 'll = rl AND lh > rh'\n WHEN 't_during' THEN 'll > rl AND lh < rh'\n WHEN 't_contains' THEN 'll < rl AND lh > rh'\n WHEN 't_finishes' THEN 'll > rl AND lh = rh'\n WHEN 't_finishedby' THEN 'll < rl AND lh = rh'\n WHEN 't_equals' THEN 'll = rl AND lh = rh'\n WHEN 't_disjoint' THEN 'NOT (ll <= rh AND lh >= rl)'\n WHEN 't_intersects' THEN 'll <= rh AND lh >= rl'\n WHEN 'anyinteracts' THEN 'll <= rh AND lh >= rl'\n END;\n outq := regexp_replace(outq, '\\mll\\M', ll);\n outq := regexp_replace(outq, '\\mlh\\M', lh);\n outq := regexp_replace(outq, '\\mrl\\M', rl);\n outq := regexp_replace(outq, '\\mrh\\M', rh);\n outq := format('(%s)', outq);\n RETURN outq;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\n\n\nCREATE OR REPLACE FUNCTION spatial_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\n geom text;\n j jsonb := args->1;\nBEGIN\n op := lower(op);\n RAISE NOTICE 'Constructing spatial query OP: %, ARGS: %', op, args;\n IF op NOT IN ('s_equals','s_disjoint','s_touches','s_within','s_overlaps','s_crosses','s_intersects','intersects','s_contains') THEN\n RAISE EXCEPTION 'Spatial Operator % Not Supported', op;\n END IF;\n op := regexp_replace(op, '^s_', 'st_');\n IF op = 'intersects' THEN\n op := 'st_intersects';\n END IF;\n -- Convert geometry to WKB string\n IF j ? 'type' AND j ? 'coordinates' THEN\n geom := st_geomfromgeojson(j)::text;\n ELSIF jsonb_typeof(j) = 'array' THEN\n geom := bbox_geom(j)::text;\n END IF;\n\n RETURN format('%s(geometry, %L::geometry)', op, geom);\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION query_to_cql2(q jsonb) RETURNS jsonb AS $$\n-- Translates anything passed in through the deprecated \"query\" into equivalent CQL2\nWITH t AS (\n SELECT key as property, value as ops\n FROM jsonb_each(q)\n), t2 AS (\n SELECT property, (jsonb_each(ops)).*\n FROM t WHERE jsonb_typeof(ops) = 'object'\n UNION ALL\n SELECT property, 'eq', ops\n FROM t WHERE jsonb_typeof(ops) != 'object'\n)\nSELECT\n jsonb_strip_nulls(jsonb_build_object(\n 'op', 'and',\n 'args', jsonb_agg(\n jsonb_build_object(\n 'op', key,\n 'args', jsonb_build_array(\n jsonb_build_object('property',property),\n value\n )\n )\n )\n )\n) as qcql FROM t2\n;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\nCREATE OR REPLACE FUNCTION cql1_to_cql2(j jsonb) RETURNS jsonb AS $$\nDECLARE\n args jsonb;\n ret jsonb;\nBEGIN\n RAISE NOTICE 'CQL1_TO_CQL2: %', j;\n IF j ? 'filter' THEN\n RETURN cql1_to_cql2(j->'filter');\n END IF;\n IF j ? 'property' THEN\n RETURN j;\n END IF;\n IF jsonb_typeof(j) = 'array' THEN\n SELECT jsonb_agg(cql1_to_cql2(el)) INTO args FROM jsonb_array_elements(j) el;\n RETURN args;\n END IF;\n IF jsonb_typeof(j) = 'number' THEN\n RETURN j;\n END IF;\n IF jsonb_typeof(j) = 'string' THEN\n RETURN j;\n END IF;\n\n IF jsonb_typeof(j) = 'object' THEN\n SELECT jsonb_build_object(\n 'op', key,\n 'args', cql1_to_cql2(value)\n ) INTO ret\n FROM jsonb_each(j)\n WHERE j IS NOT NULL;\n RETURN ret;\n END IF;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE STRICT;\n\nCREATE TABLE cql2_ops (\n op text PRIMARY KEY,\n template text,\n types text[]\n);\nINSERT INTO cql2_ops (op, template, types) VALUES\n ('eq', '%s = %s', NULL),\n ('neq', '%s != %s', NULL),\n ('ne', '%s != %s', NULL),\n ('!=', '%s != %s', NULL),\n ('<>', '%s != %s', NULL),\n ('lt', '%s < %s', NULL),\n ('lte', '%s <= %s', NULL),\n ('gt', '%s > %s', NULL),\n ('gte', '%s >= %s', NULL),\n ('le', '%s <= %s', NULL),\n ('ge', '%s >= %s', NULL),\n ('=', '%s = %s', NULL),\n ('<', '%s < %s', NULL),\n ('<=', '%s <= %s', NULL),\n ('>', '%s > %s', NULL),\n ('>=', '%s >= %s', NULL),\n ('like', '%s LIKE %s', NULL),\n ('ilike', '%s ILIKE %s', NULL),\n ('+', '%s + %s', NULL),\n ('-', '%s - %s', NULL),\n ('*', '%s * %s', NULL),\n ('/', '%s / %s', NULL),\n ('not', 'NOT (%s)', NULL),\n ('between', '%s BETWEEN %s AND %s', NULL),\n ('isnull', '%s IS NULL', NULL),\n ('upper', 'upper(%s)', NULL),\n ('lower', 'lower(%s)', NULL),\n ('casei', 'upper(%s)', NULL),\n ('accenti', 'unaccent(%s)', NULL)\nON CONFLICT (op) DO UPDATE\n SET\n template = EXCLUDED.template\n;\n\n\nCREATE OR REPLACE FUNCTION cql2_query(j jsonb, wrapper text DEFAULT NULL) RETURNS text AS $$\n#variable_conflict use_variable\nDECLARE\n args jsonb := j->'args';\n arg jsonb;\n op text := lower(j->>'op');\n cql2op RECORD;\n literal text;\n _wrapper text;\n leftarg text;\n rightarg text;\n prop text;\n extra_props bool := pgstac.additional_properties();\nBEGIN\n IF j IS NULL OR (op IS NOT NULL AND args IS NULL) THEN\n RETURN NULL;\n END IF;\n RAISE NOTICE 'CQL2_QUERY: %', j;\n\n -- check if all properties are represented in the queryables\n IF NOT extra_props THEN\n FOR prop IN\n SELECT DISTINCT p->>0\n FROM jsonb_path_query(j, 'strict $.**.property') p\n WHERE p->>0 NOT IN ('id', 'datetime', 'geometry', 'end_datetime', 'collection')\n LOOP\n IF (queryable(prop)).nulled_wrapper IS NULL THEN\n RAISE EXCEPTION 'Term % is not found in queryables.', prop;\n END IF;\n END LOOP;\n END IF;\n\n IF j ? 'filter' THEN\n RETURN cql2_query(j->'filter');\n END IF;\n\n IF j ? 'upper' THEN\n RETURN cql2_query(jsonb_build_object('op', 'upper', 'args', j->'upper'));\n END IF;\n\n IF j ? 'lower' THEN\n RETURN cql2_query(jsonb_build_object('op', 'lower', 'args', j->'lower'));\n END IF;\n\n -- Temporal Query\n IF op ilike 't_%' or op = 'anyinteracts' THEN\n RETURN temporal_op_query(op, args);\n END IF;\n\n -- If property is a timestamp convert it to text to use with\n -- general operators\n IF j ? 'timestamp' THEN\n RETURN format('%L::timestamptz', to_tstz(j->'timestamp'));\n END IF;\n IF j ? 'interval' THEN\n RAISE EXCEPTION 'Please use temporal operators when using intervals.';\n RETURN NONE;\n END IF;\n\n -- Spatial Query\n IF op ilike 's_%' or op = 'intersects' THEN\n RETURN spatial_op_query(op, args);\n END IF;\n\n IF op IN ('a_equals','a_contains','a_contained_by','a_overlaps') THEN\n IF args->0 ? 'property' THEN\n leftarg := format('to_text_array(%s)', (queryable(args->0->>'property')).path);\n END IF;\n IF args->1 ? 'property' THEN\n rightarg := format('to_text_array(%s)', (queryable(args->1->>'property')).path);\n END IF;\n RETURN FORMAT(\n '%s %s %s',\n COALESCE(leftarg, quote_literal(to_text_array(args->0))),\n CASE op\n WHEN 'a_equals' THEN '='\n WHEN 'a_contains' THEN '@>'\n WHEN 'a_contained_by' THEN '<@'\n WHEN 'a_overlaps' THEN '&&'\n END,\n COALESCE(rightarg, quote_literal(to_text_array(args->1)))\n );\n END IF;\n\n IF op = 'in' THEN\n RAISE NOTICE 'IN : % % %', args, jsonb_build_array(args->0), args->1;\n args := jsonb_build_array(args->0) || (args->1);\n RAISE NOTICE 'IN2 : %', args;\n END IF;\n\n\n\n IF op = 'between' THEN\n args = jsonb_build_array(\n args->0,\n args->1,\n args->2\n );\n END IF;\n\n -- Make sure that args is an array and run cql2_query on\n -- each element of the array\n RAISE NOTICE 'ARGS PRE: %', args;\n IF j ? 'args' THEN\n IF jsonb_typeof(args) != 'array' THEN\n args := jsonb_build_array(args);\n END IF;\n\n IF jsonb_path_exists(args, '$[*] ? (@.property == \"id\" || @.property == \"datetime\" || @.property == \"end_datetime\" || @.property == \"collection\")') THEN\n wrapper := NULL;\n ELSE\n -- if any of the arguments are a property, try to get the property_wrapper\n FOR arg IN SELECT jsonb_path_query(args, '$[*] ? (@.property != null)') LOOP\n RAISE NOTICE 'Arg: %', arg;\n wrapper := (queryable(arg->>'property')).nulled_wrapper;\n RAISE NOTICE 'Property: %, Wrapper: %', arg, wrapper;\n IF wrapper IS NOT NULL THEN\n EXIT;\n END IF;\n END LOOP;\n\n -- if the property was not in queryables, see if any args were numbers\n IF\n wrapper IS NULL\n AND jsonb_path_exists(args, '$[*] ? (@.type()==\"number\")')\n THEN\n wrapper := 'to_float';\n END IF;\n wrapper := coalesce(wrapper, 'to_text');\n END IF;\n\n SELECT jsonb_agg(cql2_query(a, wrapper))\n INTO args\n FROM jsonb_array_elements(args) a;\n END IF;\n RAISE NOTICE 'ARGS: %', args;\n\n IF op IN ('and', 'or') THEN\n RETURN\n format(\n '(%s)',\n array_to_string(to_text_array(args), format(' %s ', upper(op)))\n );\n END IF;\n\n IF op = 'in' THEN\n RAISE NOTICE 'IN -- % %', args->0, to_text(args->0);\n RETURN format(\n '%s IN (%s)',\n to_text(args->0),\n array_to_string((to_text_array(args))[2:], ',')\n );\n END IF;\n\n -- Look up template from cql2_ops\n IF j ? 'op' THEN\n SELECT * INTO cql2op FROM cql2_ops WHERE cql2_ops.op ilike op;\n IF FOUND THEN\n -- If specific index set in queryables for a property cast other arguments to that type\n\n RETURN format(\n cql2op.template,\n VARIADIC (to_text_array(args))\n );\n ELSE\n RAISE EXCEPTION 'Operator % Not Supported.', op;\n END IF;\n END IF;\n\n\n IF wrapper IS NOT NULL THEN\n RAISE NOTICE 'Wrapping % with %', j, wrapper;\n IF j ? 'property' THEN\n RETURN format('%I(%s)', wrapper, (queryable(j->>'property')).path);\n ELSE\n RETURN format('%I(%L)', wrapper, j);\n END IF;\n ELSIF j ? 'property' THEN\n RETURN quote_ident(j->>'property');\n END IF;\n\n RETURN quote_literal(to_text(j));\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION paging_dtrange(\n j jsonb\n) RETURNS tstzrange AS $$\nDECLARE\n op text;\n filter jsonb := j->'filter';\n dtrange tstzrange := tstzrange('-infinity'::timestamptz,'infinity'::timestamptz);\n sdate timestamptz := '-infinity'::timestamptz;\n edate timestamptz := 'infinity'::timestamptz;\n jpitem jsonb;\nBEGIN\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n END IF;\n IF NOT (filter @? '$.**.op ? (@ == \"or\" || @ == \"not\")') THEN\n FOR jpitem IN SELECT j FROM jsonb_path_query(filter,'strict $.** ? (@.args[*].property == \"datetime\")'::jsonpath) j LOOP\n op := lower(jpitem->>'op');\n dtrange := parse_dtrange(jpitem->'args'->1);\n IF op IN ('<=', 'lt', 'lte', '<', 'le', 't_before') THEN\n sdate := greatest(sdate,'-infinity');\n edate := least(edate, upper(dtrange));\n ELSIF op IN ('>=', '>', 'gt', 'gte', 'ge', 't_after') THEN\n edate := least(edate, 'infinity');\n sdate := greatest(sdate, lower(dtrange));\n ELSIF op IN ('=', 'eq') THEN\n edate := least(edate, upper(dtrange));\n sdate := greatest(sdate, lower(dtrange));\n END IF;\n RAISE NOTICE '2 OP: %, ARGS: %, DTRANGE: %, SDATE: %, EDATE: %', op, jpitem->'args'->1, dtrange, sdate, edate;\n END LOOP;\n END IF;\n IF sdate > edate THEN\n RETURN 'empty'::tstzrange;\n END IF;\n RETURN tstzrange(sdate,edate, '[]');\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT SET TIME ZONE 'UTC';\n\nCREATE OR REPLACE FUNCTION paging_collections(\n IN j jsonb\n) RETURNS text[] AS $$\nDECLARE\n filter jsonb := j->'filter';\n jpitem jsonb;\n op text;\n args jsonb;\n arg jsonb;\n collections text[];\nBEGIN\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n END IF;\n IF NOT (filter @? '$.**.op ? (@ == \"or\" || @ == \"not\")') THEN\n FOR jpitem IN SELECT j FROM jsonb_path_query(filter,'strict $.** ? (@.args[*].property == \"collection\")'::jsonpath) j LOOP\n RAISE NOTICE 'JPITEM: %', jpitem;\n op := jpitem->>'op';\n args := jpitem->'args';\n IF op IN ('=', 'eq', 'in') THEN\n FOR arg IN SELECT a FROM jsonb_array_elements(args) a LOOP\n IF jsonb_typeof(arg) IN ('string', 'array') THEN\n RAISE NOTICE 'arg: %, collections: %', arg, collections;\n IF collections IS NULL OR collections = '{}'::text[] THEN\n collections := to_text_array(arg);\n ELSE\n collections := array_intersection(collections, to_text_array(arg));\n END IF;\n END IF;\n END LOOP;\n END IF;\n END LOOP;\n END IF;\n IF collections = '{}'::text[] THEN\n RETURN NULL;\n END IF;\n RETURN collections;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\nCREATE TABLE items (\n id text NOT NULL,\n geometry geometry NOT NULL,\n collection text NOT NULL,\n datetime timestamptz NOT NULL,\n end_datetime timestamptz NOT NULL,\n content JSONB NOT NULL,\n private jsonb\n)\nPARTITION BY LIST (collection)\n;\n\nCREATE INDEX \"datetime_idx\" ON items USING BTREE (datetime DESC, end_datetime ASC);\nCREATE INDEX \"geometry_idx\" ON items USING GIST (geometry);\n\nCREATE STATISTICS datetime_stats (dependencies) on datetime, end_datetime from items;\n\nALTER TABLE items ADD CONSTRAINT items_collections_fk FOREIGN KEY (collection) REFERENCES collections(id) ON DELETE CASCADE DEFERRABLE;\n\nCREATE OR REPLACE FUNCTION partition_after_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p text;\n t timestamptz := clock_timestamp();\nBEGIN\n RAISE NOTICE 'Updating partition stats %', t;\n FOR p IN SELECT DISTINCT partition\n FROM newdata n JOIN partition_sys_meta p\n ON (n.collection=p.collection AND n.datetime <@ p.partition_dtrange)\n LOOP\n PERFORM run_or_queue(format('SELECT update_partition_stats(%L, %L);', p, true));\n END LOOP;\n IF TG_OP IN ('DELETE','UPDATE') THEN\n DELETE FROM format_item_cache c USING newdata n WHERE c.collection = n.collection AND c.id = n.id;\n END IF;\n RAISE NOTICE 't: % %', t, clock_timestamp() - t;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE TRIGGER items_after_insert_trigger\nAFTER INSERT ON items\nREFERENCING NEW TABLE AS newdata\nFOR EACH STATEMENT\nEXECUTE FUNCTION partition_after_triggerfunc();\n\nCREATE TRIGGER items_after_update_trigger\nAFTER DELETE ON items\nREFERENCING OLD TABLE AS newdata\nFOR EACH STATEMENT\nEXECUTE FUNCTION partition_after_triggerfunc();\n\nCREATE TRIGGER items_after_delete_trigger\nAFTER UPDATE ON items\nREFERENCING NEW TABLE AS newdata\nFOR EACH STATEMENT\nEXECUTE FUNCTION partition_after_triggerfunc();\n\n\nCREATE OR REPLACE FUNCTION content_slim(_item jsonb) RETURNS jsonb AS $$\n SELECT strip_jsonb(_item - '{id,geometry,collection,type}'::text[], collection_base_item(_item->>'collection')) - '{id,geometry,collection,type}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_dehydrate(content jsonb) RETURNS items AS $$\n SELECT\n content->>'id' as id,\n stac_geom(content) as geometry,\n content->>'collection' as collection,\n stac_datetime(content) as datetime,\n stac_end_datetime(content) as end_datetime,\n content_slim(content) as content,\n null::jsonb as private\n ;\n$$ LANGUAGE SQL STABLE;\n\nCREATE OR REPLACE FUNCTION include_field(f text, fields jsonb DEFAULT '{}'::jsonb) RETURNS boolean AS $$\nDECLARE\n includes jsonb := fields->'include';\n excludes jsonb := fields->'exclude';\nBEGIN\n IF f IS NULL THEN\n RETURN NULL;\n END IF;\n\n\n IF\n jsonb_typeof(excludes) = 'array'\n AND jsonb_array_length(excludes)>0\n AND excludes ? f\n THEN\n RETURN FALSE;\n END IF;\n\n IF\n (\n jsonb_typeof(includes) = 'array'\n AND jsonb_array_length(includes) > 0\n AND includes ? f\n ) OR\n (\n includes IS NULL\n OR jsonb_typeof(includes) = 'null'\n OR jsonb_array_length(includes) = 0\n )\n THEN\n RETURN TRUE;\n END IF;\n\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nDROP FUNCTION IF EXISTS content_hydrate(jsonb, jsonb, jsonb);\nCREATE OR REPLACE FUNCTION content_hydrate(\n _item jsonb,\n _base_item jsonb,\n fields jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\n SELECT merge_jsonb(\n jsonb_fields(_item, fields),\n jsonb_fields(_base_item, fields)\n );\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION content_hydrate(_item items, _collection collections, fields jsonb DEFAULT '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\n content jsonb;\n base_item jsonb := _collection.base_item;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry, 20)::jsonb;\n END IF;\n output := content_hydrate(\n jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'collection', _item.collection,\n 'type', 'Feature'\n ) || _item.content,\n _collection.base_item,\n fields\n );\n\n RETURN output;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_nonhydrated(\n _item items,\n fields jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry, 20)::jsonb;\n END IF;\n output := jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'collection', _item.collection,\n 'type', 'Feature'\n ) || _item.content;\n RETURN output;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_hydrate(_item items, fields jsonb DEFAULT '{}'::jsonb) RETURNS jsonb AS $$\n SELECT content_hydrate(\n _item,\n (SELECT c FROM collections c WHERE id=_item.collection LIMIT 1),\n fields\n );\n$$ LANGUAGE SQL STABLE;\n\n\nCREATE UNLOGGED TABLE items_staging (\n content JSONB NOT NULL\n);\nCREATE UNLOGGED TABLE items_staging_ignore (\n content JSONB NOT NULL\n);\nCREATE UNLOGGED TABLE items_staging_upsert (\n content JSONB NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION items_staging_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p record;\n _partitions text[];\n part text;\n ts timestamptz := clock_timestamp();\n nrows int;\nBEGIN\n RAISE NOTICE 'Creating Partitions. %', clock_timestamp() - ts;\n\n FOR part IN WITH t AS (\n SELECT\n n.content->>'collection' as collection,\n stac_daterange(n.content->'properties') as dtr,\n partition_trunc\n FROM newdata n JOIN collections ON (n.content->>'collection'=collections.id)\n ), p AS (\n SELECT\n collection,\n COALESCE(date_trunc(partition_trunc::text, lower(dtr)),'-infinity') as d,\n tstzrange(min(lower(dtr)),max(lower(dtr)),'[]') as dtrange,\n tstzrange(min(upper(dtr)),max(upper(dtr)),'[]') as edtrange\n FROM t\n GROUP BY 1,2\n ) SELECT check_partition(collection, dtrange, edtrange) FROM p LOOP\n RAISE NOTICE 'Partition %', part;\n END LOOP;\n\n RAISE NOTICE 'Creating temp table with data to be added. %', clock_timestamp() - ts;\n DROP TABLE IF EXISTS tmpdata;\n CREATE TEMP TABLE tmpdata ON COMMIT DROP AS\n SELECT\n (content_dehydrate(content)).*\n FROM newdata;\n GET DIAGNOSTICS nrows = ROW_COUNT;\n RAISE NOTICE 'Added % rows to tmpdata. %', nrows, clock_timestamp() - ts;\n\n RAISE NOTICE 'Doing the insert. %', clock_timestamp() - ts;\n IF TG_TABLE_NAME = 'items_staging' THEN\n INSERT INTO items\n SELECT * FROM tmpdata;\n GET DIAGNOSTICS nrows = ROW_COUNT;\n RAISE NOTICE 'Inserted % rows to items. %', nrows, clock_timestamp() - ts;\n ELSIF TG_TABLE_NAME = 'items_staging_ignore' THEN\n INSERT INTO items\n SELECT * FROM tmpdata\n ON CONFLICT DO NOTHING;\n GET DIAGNOSTICS nrows = ROW_COUNT;\n RAISE NOTICE 'Inserted % rows to items. %', nrows, clock_timestamp() - ts;\n ELSIF TG_TABLE_NAME = 'items_staging_upsert' THEN\n DELETE FROM items i USING tmpdata s\n WHERE\n i.id = s.id\n AND i.collection = s.collection\n AND i IS DISTINCT FROM s\n ;\n GET DIAGNOSTICS nrows = ROW_COUNT;\n RAISE NOTICE 'Deleted % rows from items. %', nrows, clock_timestamp() - ts;\n INSERT INTO items AS t\n SELECT * FROM tmpdata\n ON CONFLICT DO NOTHING;\n GET DIAGNOSTICS nrows = ROW_COUNT;\n RAISE NOTICE 'Inserted % rows to items. %', nrows, clock_timestamp() - ts;\n END IF;\n\n RAISE NOTICE 'Deleting data from staging table. %', clock_timestamp() - ts;\n DELETE FROM items_staging;\n RAISE NOTICE 'Done. %', clock_timestamp() - ts;\n\n RETURN NULL;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER items_staging_insert_trigger AFTER INSERT ON items_staging REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\nCREATE TRIGGER items_staging_insert_ignore_trigger AFTER INSERT ON items_staging_ignore REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\nCREATE TRIGGER items_staging_insert_upsert_trigger AFTER INSERT ON items_staging_upsert REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\n\nCREATE OR REPLACE FUNCTION item_by_id(_id text, _collection text DEFAULT NULL) RETURNS items AS\n$$\nDECLARE\n i items%ROWTYPE;\nBEGIN\n SELECT * INTO i FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection) LIMIT 1;\n RETURN i;\nEND;\n$$ LANGUAGE PLPGSQL STABLE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION get_item(_id text, _collection text DEFAULT NULL) RETURNS jsonb AS $$\n SELECT content_hydrate(items) FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection);\n$$ LANGUAGE SQL STABLE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION delete_item(_id text, _collection text DEFAULT NULL) RETURNS VOID AS $$\nDECLARE\nout items%ROWTYPE;\nBEGIN\n DELETE FROM items WHERE id = _id AND (_collection IS NULL OR collection=_collection) RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n--/*\nCREATE OR REPLACE FUNCTION create_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION update_item(content jsonb) RETURNS VOID AS $$\nDECLARE\n old items %ROWTYPE;\n out items%ROWTYPE;\nBEGIN\n PERFORM delete_item(content->>'id', content->>'collection');\n PERFORM create_item(content);\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION create_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION collection_bbox(id text) RETURNS jsonb AS $$\n SELECT (replace(replace(replace(st_extent(geometry)::text,'BOX(','[['),')',']]'),' ',','))::jsonb\n FROM items WHERE collection=$1;\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION collection_temporal_extent(id text) RETURNS jsonb AS $$\n SELECT to_jsonb(array[array[min(datetime), max(datetime)]])\n FROM items WHERE collection=$1;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection_extents() RETURNS VOID AS $$\nUPDATE collections\n SET content = jsonb_set_lax(\n content,\n '{extent}'::text[],\n collection_extent(id, FALSE),\n true,\n 'use_json_null'\n )\n;\n$$ LANGUAGE SQL;\nCREATE TABLE partition_stats (\n partition text PRIMARY KEY,\n dtrange tstzrange,\n edtrange tstzrange,\n spatial geometry,\n last_updated timestamptz,\n keys text[]\n) WITH (FILLFACTOR=90);\n\nCREATE INDEX partitions_range_idx ON partition_stats USING GIST(dtrange);\n\n\nCREATE OR REPLACE FUNCTION constraint_tstzrange(expr text) RETURNS tstzrange AS $$\n WITH t AS (\n SELECT regexp_matches(\n expr,\n E'\\\\(''\\([0-9 :+-]*\\)''\\\\).*\\\\(''\\([0-9 :+-]*\\)''\\\\)'\n ) AS m\n ) SELECT tstzrange(m[1]::timestamptz, m[2]::timestamptz) FROM t\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE STRICT;\n\nCREATE OR REPLACE FUNCTION dt_constraint(coid oid, OUT dt tstzrange, OUT edt tstzrange) RETURNS RECORD AS $$\nDECLARE\n expr text := pg_get_constraintdef(coid);\n matches timestamptz[];\nBEGIN\n IF expr LIKE '%NULL%' THEN\n dt := tstzrange(null::timestamptz, null::timestamptz);\n edt := tstzrange(null::timestamptz, null::timestamptz);\n RETURN;\n END IF;\n WITH f AS (SELECT (regexp_matches(expr, E'([0-9]{4}-[0-1][0-9]-[0-3][0-9] [0-2][0-9]:[0-5][0-9]:[0-5][0-9]\\.?[0-9]*)', 'g'))[1] f)\n SELECT array_agg(f::timestamptz) INTO matches FROM f;\n IF cardinality(matches) = 4 THEN\n dt := tstzrange(matches[1], matches[2],'[]');\n edt := tstzrange(matches[3], matches[4], '[]');\n RETURN;\n ELSIF cardinality(matches) = 2 THEN\n edt := tstzrange(matches[1], matches[2],'[]');\n RETURN;\n END IF;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\nCREATE OR REPLACE VIEW partition_sys_meta AS\nSELECT\n relid::text as partition,\n replace(replace(CASE WHEN level = 1 THEN pg_get_expr(c.relpartbound, c.oid)\n ELSE pg_get_expr(parent.relpartbound, parent.oid)\n END, 'FOR VALUES IN (''',''), ''')','') AS collection,\n level,\n c.reltuples,\n c.relhastriggers,\n COALESCE(constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as partition_dtrange,\n COALESCE((dt_constraint(edt.oid)).dt, constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as constraint_dtrange,\n COALESCE((dt_constraint(edt.oid)).edt, tstzrange('-infinity', 'infinity','[]')) as constraint_edtrange\nFROM\n pg_partition_tree('items')\n JOIN pg_class c ON (relid::regclass = c.oid)\n JOIN pg_class parent ON (parentrelid::regclass = parent.oid AND isleaf)\n LEFT JOIN pg_constraint edt ON (conrelid=c.oid AND contype='c')\nWHERE isleaf\n;\n\nCREATE VIEW partitions_view AS\nSELECT\n relid::text as partition,\n replace(replace(CASE WHEN level = 1 THEN pg_get_expr(c.relpartbound, c.oid)\n ELSE pg_get_expr(parent.relpartbound, parent.oid)\n END, 'FOR VALUES IN (''',''), ''')','') AS collection,\n level,\n c.reltuples,\n c.relhastriggers,\n COALESCE(constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as partition_dtrange,\n COALESCE((dt_constraint(edt.oid)).dt, constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as constraint_dtrange,\n COALESCE((dt_constraint(edt.oid)).edt, tstzrange('-infinity', 'infinity','[]')) as constraint_edtrange,\n dtrange,\n edtrange,\n spatial,\n last_updated\nFROM\n pg_partition_tree('items')\n JOIN pg_class c ON (relid::regclass = c.oid)\n JOIN pg_class parent ON (parentrelid::regclass = parent.oid AND isleaf)\n LEFT JOIN pg_constraint edt ON (conrelid=c.oid AND contype='c')\n LEFT JOIN partition_stats ON (relid::text=partition)\nWHERE isleaf\n;\n\nCREATE MATERIALIZED VIEW partitions AS\nSELECT * FROM partitions_view;\nCREATE UNIQUE INDEX ON partitions (partition);\n\nCREATE MATERIALIZED VIEW partition_steps AS\nSELECT\n partition as name,\n date_trunc('month',lower(partition_dtrange)) as sdate,\n date_trunc('month', upper(partition_dtrange)) + '1 month'::interval as edate\n FROM partitions_view WHERE partition_dtrange IS NOT NULL AND partition_dtrange != 'empty'::tstzrange\n ORDER BY dtrange ASC\n;\n\n\nCREATE OR REPLACE FUNCTION update_partition_stats_q(_partition text, istrigger boolean default false) RETURNS VOID AS $$\nDECLARE\nBEGIN\n PERFORM run_or_queue(\n format('SELECT update_partition_stats(%L, %L);', _partition, istrigger)\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION update_partition_stats(_partition text, istrigger boolean default false) RETURNS VOID AS $$\nDECLARE\n dtrange tstzrange;\n edtrange tstzrange;\n cdtrange tstzrange;\n cedtrange tstzrange;\n extent geometry;\n collection text;\nBEGIN\n RAISE NOTICE 'Updating stats for %.', _partition;\n EXECUTE format(\n $q$\n SELECT\n tstzrange(min(datetime), max(datetime),'[]'),\n tstzrange(min(end_datetime), max(end_datetime), '[]')\n FROM %I\n $q$,\n _partition\n ) INTO dtrange, edtrange;\n extent := st_estimatedextent('pgstac', _partition, 'geometry');\n INSERT INTO partition_stats (partition, dtrange, edtrange, spatial, last_updated)\n SELECT _partition, dtrange, edtrange, extent, now()\n ON CONFLICT (partition) DO\n UPDATE SET\n dtrange=EXCLUDED.dtrange,\n edtrange=EXCLUDED.edtrange,\n spatial=EXCLUDED.spatial,\n last_updated=EXCLUDED.last_updated\n ;\n\n SELECT\n constraint_dtrange, constraint_edtrange, pv.collection\n INTO cdtrange, cedtrange, collection\n FROM partitions_view pv WHERE partition = _partition;\n REFRESH MATERIALIZED VIEW partitions;\n REFRESH MATERIALIZED VIEW partition_steps;\n\n\n RAISE NOTICE 'Checking if we need to modify constraints...';\n RAISE NOTICE 'cdtrange: % dtrange: % cedtrange: % edtrange: %',cdtrange, dtrange, cedtrange, edtrange;\n IF\n (cdtrange IS DISTINCT FROM dtrange OR edtrange IS DISTINCT FROM cedtrange)\n AND NOT istrigger\n THEN\n RAISE NOTICE 'Modifying Constraints';\n RAISE NOTICE 'Existing % %', cdtrange, cedtrange;\n RAISE NOTICE 'New % %', dtrange, edtrange;\n PERFORM drop_table_constraints(_partition);\n PERFORM create_table_constraints(_partition, dtrange, edtrange);\n REFRESH MATERIALIZED VIEW partitions;\n REFRESH MATERIALIZED VIEW partition_steps;\n END IF;\n RAISE NOTICE 'Checking if we need to update collection extents.';\n IF get_setting_bool('update_collection_extent') THEN\n RAISE NOTICE 'updating collection extent for %', collection;\n PERFORM run_or_queue(format($q$\n UPDATE collections\n SET content = jsonb_set_lax(\n content,\n '{extent}'::text[],\n collection_extent(%L, FALSE),\n true,\n 'use_json_null'\n ) WHERE id=%L\n ;\n $q$, collection, collection));\n ELSE\n RAISE NOTICE 'Not updating collection extent for %', collection;\n END IF;\n\nEND;\n$$ LANGUAGE PLPGSQL STRICT SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION partition_name( IN collection text, IN dt timestamptz, OUT partition_name text, OUT partition_range tstzrange) AS $$\nDECLARE\n c RECORD;\n parent_name text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n parent_name := format('_items_%s', c.key);\n\n\n IF c.partition_trunc = 'year' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYY'));\n ELSIF c.partition_trunc = 'month' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYYMM'));\n ELSE\n partition_name := parent_name;\n partition_range := tstzrange('-infinity'::timestamptz, 'infinity'::timestamptz, '[]');\n END IF;\n IF partition_range IS NULL THEN\n partition_range := tstzrange(\n date_trunc(c.partition_trunc::text, dt),\n date_trunc(c.partition_trunc::text, dt) + concat('1 ', c.partition_trunc)::interval\n );\n END IF;\n RETURN;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION drop_table_constraints(t text) RETURNS text AS $$\nDECLARE\n q text;\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM partitions_view WHERE partition=t) THEN\n RETURN NULL;\n END IF;\n FOR q IN SELECT FORMAT(\n $q$\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n $q$,\n t,\n conname\n ) FROM pg_constraint\n WHERE conrelid=t::regclass::oid AND contype='c'\n LOOP\n EXECUTE q;\n END LOOP;\n RETURN t;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE OR REPLACE FUNCTION create_table_constraints(t text, _dtrange tstzrange, _edtrange tstzrange) RETURNS text AS $$\nDECLARE\n q text;\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM partitions_view WHERE partition=t) THEN\n RETURN NULL;\n END IF;\n RAISE NOTICE 'Creating Table Constraints for % % %', t, _dtrange, _edtrange;\n IF _dtrange = 'empty' AND _edtrange = 'empty' THEN\n q :=format(\n $q$\n DO $block$\n BEGIN\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n ALTER TABLE %I\n ADD CONSTRAINT %I\n CHECK (((datetime IS NULL) AND (end_datetime IS NULL))) NOT VALID\n ;\n ALTER TABLE %I\n VALIDATE CONSTRAINT %I\n ;\n\n\n\n EXCEPTION WHEN others THEN\n RAISE WARNING '%%, Issue Altering Constraints. Please run update_partition_stats(%I)', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n $block$;\n $q$,\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t),\n t\n );\n ELSE\n q :=format(\n $q$\n DO $block$\n BEGIN\n\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n ALTER TABLE %I\n ADD CONSTRAINT %I\n CHECK (\n (datetime >= %L)\n AND (datetime <= %L)\n AND (end_datetime >= %L)\n AND (end_datetime <= %L)\n ) NOT VALID\n ;\n ALTER TABLE %I\n VALIDATE CONSTRAINT %I\n ;\n\n\n\n EXCEPTION WHEN others THEN\n RAISE WARNING '%%, Issue Altering Constraints. Please run update_partition_stats(%I)', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n $block$;\n $q$,\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t),\n lower(_dtrange),\n upper(_dtrange),\n lower(_edtrange),\n upper(_edtrange),\n t,\n format('%s_dt', t),\n t\n );\n END IF;\n PERFORM run_or_queue(q);\n RETURN t;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION check_partition(\n _collection text,\n _dtrange tstzrange,\n _edtrange tstzrange\n) RETURNS text AS $$\nDECLARE\n c RECORD;\n pm RECORD;\n _partition_name text;\n _partition_dtrange tstzrange;\n _constraint_dtrange tstzrange;\n _constraint_edtrange tstzrange;\n q text;\n deferrable_q text;\n err_context text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=_collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', _collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n\n IF c.partition_trunc IS NOT NULL THEN\n _partition_dtrange := tstzrange(\n date_trunc(c.partition_trunc, lower(_dtrange)),\n date_trunc(c.partition_trunc, lower(_dtrange)) + (concat('1 ', c.partition_trunc))::interval,\n '[)'\n );\n ELSE\n _partition_dtrange := '[-infinity, infinity]'::tstzrange;\n END IF;\n\n IF NOT _partition_dtrange @> _dtrange THEN\n RAISE EXCEPTION 'dtrange % is greater than the partition size % for collection %', _dtrange, c.partition_trunc, _collection;\n END IF;\n\n\n IF c.partition_trunc = 'year' THEN\n _partition_name := format('_items_%s_%s', c.key, to_char(lower(_partition_dtrange),'YYYY'));\n ELSIF c.partition_trunc = 'month' THEN\n _partition_name := format('_items_%s_%s', c.key, to_char(lower(_partition_dtrange),'YYYYMM'));\n ELSE\n _partition_name := format('_items_%s', c.key);\n END IF;\n\n SELECT * INTO pm FROM partition_sys_meta WHERE collection=_collection AND partition_dtrange @> _dtrange;\n IF FOUND THEN\n RAISE NOTICE '% % %', _edtrange, _dtrange, pm;\n _constraint_edtrange :=\n tstzrange(\n least(\n lower(_edtrange),\n nullif(lower(pm.constraint_edtrange), '-infinity')\n ),\n greatest(\n upper(_edtrange),\n nullif(upper(pm.constraint_edtrange), 'infinity')\n ),\n '[]'\n );\n _constraint_dtrange :=\n tstzrange(\n least(\n lower(_dtrange),\n nullif(lower(pm.constraint_dtrange), '-infinity')\n ),\n greatest(\n upper(_dtrange),\n nullif(upper(pm.constraint_dtrange), 'infinity')\n ),\n '[]'\n );\n\n IF pm.constraint_edtrange @> _edtrange AND pm.constraint_dtrange @> _dtrange THEN\n RETURN pm.partition;\n ELSE\n PERFORM drop_table_constraints(_partition_name);\n END IF;\n ELSE\n _constraint_edtrange := _edtrange;\n _constraint_dtrange := _dtrange;\n END IF;\n RAISE NOTICE 'EXISTING CONSTRAINTS % %, NEW % %', pm.constraint_dtrange, pm.constraint_edtrange, _constraint_dtrange, _constraint_edtrange;\n RAISE NOTICE 'Creating partition % %', _partition_name, _partition_dtrange;\n IF c.partition_trunc IS NULL THEN\n q := format(\n $q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n GRANT ALL ON %I to pgstac_ingest;\n $q$,\n _partition_name,\n _collection,\n concat(_partition_name,'_pk'),\n _partition_name,\n _partition_name\n );\n ELSE\n q := format(\n $q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L) PARTITION BY RANGE (datetime);\n CREATE TABLE IF NOT EXISTS %I partition OF %I FOR VALUES FROM (%L) TO (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n GRANT ALL ON %I TO pgstac_ingest;\n $q$,\n format('_items_%s', c.key),\n _collection,\n _partition_name,\n format('_items_%s', c.key),\n lower(_partition_dtrange),\n upper(_partition_dtrange),\n format('%s_pk', _partition_name),\n _partition_name,\n _partition_name\n );\n END IF;\n\n BEGIN\n EXECUTE q;\n EXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', _partition_name;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\n END;\n PERFORM maintain_partitions(_partition_name);\n PERFORM update_partition_stats_q(_partition_name, true);\n REFRESH MATERIALIZED VIEW partitions;\n REFRESH MATERIALIZED VIEW partition_steps;\n RETURN _partition_name;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION repartition(_collection text, _partition_trunc text, triggered boolean DEFAULT FALSE) RETURNS text AS $$\nDECLARE\n c RECORD;\n q text;\n from_trunc text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=_collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', _collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n IF triggered THEN\n RAISE NOTICE 'Converting % to % partitioning via Trigger', _collection, _partition_trunc;\n ELSE\n RAISE NOTICE 'Converting % from using % to % partitioning', _collection, c.partition_trunc, _partition_trunc;\n IF c.partition_trunc IS NOT DISTINCT FROM _partition_trunc THEN\n RAISE NOTICE 'Collection % already set to use partition by %', _collection, _partition_trunc;\n RETURN _collection;\n END IF;\n END IF;\n\n IF EXISTS (SELECT 1 FROM partitions_view WHERE collection=_collection LIMIT 1) THEN\n EXECUTE format(\n $q$\n CREATE TEMP TABLE changepartitionstaging ON COMMIT DROP AS SELECT * FROM %I;\n DROP TABLE IF EXISTS %I CASCADE;\n WITH p AS (\n SELECT\n collection,\n CASE\n WHEN %L IS NULL THEN '-infinity'::timestamptz\n ELSE date_trunc(%L, datetime)\n END as d,\n tstzrange(min(datetime),max(datetime),'[]') as dtrange,\n tstzrange(min(end_datetime),max(end_datetime),'[]') as edtrange\n FROM changepartitionstaging\n GROUP BY 1,2\n ) SELECT check_partition(collection, dtrange, edtrange) FROM p;\n INSERT INTO items SELECT * FROM changepartitionstaging;\n DROP TABLE changepartitionstaging;\n $q$,\n concat('_items_', c.key),\n concat('_items_', c.key),\n c.partition_trunc,\n c.partition_trunc\n );\n END IF;\n RETURN _collection;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE OR REPLACE FUNCTION collections_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n q text;\n partition_name text := format('_items_%s', NEW.key);\n partition_exists boolean := false;\n partition_empty boolean := true;\n err_context text;\n loadtemp boolean := FALSE;\nBEGIN\n RAISE NOTICE 'Collection Trigger. % %', NEW.id, NEW.key;\n IF TG_OP = 'UPDATE' AND NEW.partition_trunc IS DISTINCT FROM OLD.partition_trunc THEN\n PERFORM repartition(NEW.id, NEW.partition_trunc, TRUE);\n END IF;\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER collections_trigger AFTER\nINSERT\nOR\nUPDATE ON collections\nFOR EACH ROW EXECUTE FUNCTION collections_trigger_func();\n\nCREATE OR REPLACE FUNCTION chunker(\n IN _where text,\n OUT s timestamptz,\n OUT e timestamptz\n) RETURNS SETOF RECORD AS $$\nDECLARE\n explain jsonb;\nBEGIN\n IF _where IS NULL THEN\n _where := ' TRUE ';\n END IF;\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s;', _where)\n INTO explain;\n\n RETURN QUERY\n WITH t AS (\n SELECT j->>0 as p FROM\n jsonb_path_query(\n explain,\n 'strict $.**.\"Relation Name\" ? (@ != null)'\n ) j\n ),\n parts AS (\n SELECT sdate, edate FROM t JOIN partition_steps ON (t.p = name)\n ),\n times AS (\n SELECT sdate FROM parts\n UNION\n SELECT edate FROM parts\n ),\n uniq AS (\n SELECT DISTINCT sdate FROM times ORDER BY sdate\n ),\n last AS (\n SELECT sdate, lead(sdate, 1) over () as edate FROM uniq\n )\n SELECT sdate, edate FROM last WHERE edate IS NOT NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION partition_queries(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN partitions text[] DEFAULT NULL\n) RETURNS SETOF text AS $$\nDECLARE\n query text;\n sdate timestamptz;\n edate timestamptz;\nBEGIN\nIF _where IS NULL OR trim(_where) = '' THEN\n _where = ' TRUE ';\nEND IF;\nRAISE NOTICE 'Getting chunks for % %', _where, _orderby;\nIF _orderby ILIKE 'datetime d%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 DESC LOOP\n RETURN NEXT format($q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n $q$,\n sdate,\n edate,\n _where,\n _orderby\n );\n END LOOP;\nELSIF _orderby ILIKE 'datetime a%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 ASC LOOP\n RETURN NEXT format($q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n $q$,\n sdate,\n edate,\n _where,\n _orderby\n );\n END LOOP;\nELSE\n query := format($q$\n SELECT * FROM items\n WHERE %s\n ORDER BY %s\n $q$, _where, _orderby\n );\n\n RETURN NEXT query;\n RETURN;\nEND IF;\n\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION partition_query_view(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN _limit int DEFAULT 10\n) RETURNS text AS $$\n WITH p AS (\n SELECT * FROM partition_queries(_where, _orderby) p\n )\n SELECT\n CASE WHEN EXISTS (SELECT 1 FROM p) THEN\n (SELECT format($q$\n SELECT * FROM (\n %s\n ) total LIMIT %s\n $q$,\n string_agg(\n format($q$ SELECT * FROM ( %s ) AS sub $q$, p),\n '\n UNION ALL\n '\n ),\n _limit\n ))\n ELSE NULL\n END FROM p;\n$$ LANGUAGE SQL IMMUTABLE;\n\n\n\n\nCREATE OR REPLACE FUNCTION stac_search_to_where(j jsonb) RETURNS text AS $$\nDECLARE\n where_segments text[];\n _where text;\n dtrange tstzrange;\n collections text[];\n geom geometry;\n sdate timestamptz;\n edate timestamptz;\n filterlang text;\n filter jsonb := j->'filter';\nBEGIN\n IF j ? 'ids' THEN\n where_segments := where_segments || format('id = ANY (%L) ', to_text_array(j->'ids'));\n END IF;\n\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n where_segments := where_segments || format('collection = ANY (%L) ', collections);\n END IF;\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n\n where_segments := where_segments || format(' datetime <= %L::timestamptz AND end_datetime >= %L::timestamptz ',\n edate,\n sdate\n );\n END IF;\n\n geom := stac_geom(j);\n IF geom IS NOT NULL THEN\n where_segments := where_segments || format('st_intersects(geometry, %L)',geom);\n END IF;\n\n filterlang := COALESCE(\n j->>'filter-lang',\n get_setting('default_filter_lang', j->'conf')\n );\n IF NOT filter @? '$.**.op' THEN\n filterlang := 'cql-json';\n END IF;\n\n IF filterlang NOT IN ('cql-json','cql2-json') AND j ? 'filter' THEN\n RAISE EXCEPTION '% is not a supported filter-lang. Please use cql-json or cql2-json.', filterlang;\n END IF;\n\n IF j ? 'query' AND j ? 'filter' THEN\n RAISE EXCEPTION 'Can only use either query or filter at one time.';\n END IF;\n\n IF j ? 'query' THEN\n filter := query_to_cql2(j->'query');\n ELSIF filterlang = 'cql-json' THEN\n filter := cql1_to_cql2(filter);\n END IF;\n RAISE NOTICE 'FILTER: %', filter;\n where_segments := where_segments || cql2_query(filter);\n IF cardinality(where_segments) < 1 THEN\n RETURN ' TRUE ';\n END IF;\n\n _where := array_to_string(array_remove(where_segments, NULL), ' AND ');\n\n IF _where IS NULL OR BTRIM(_where) = '' THEN\n RETURN ' TRUE ';\n END IF;\n RETURN _where;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION parse_sort_dir(_dir text, reverse boolean default false) RETURNS text AS $$\n WITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n ) SELECT\n CASE\n WHEN NOT reverse THEN d\n WHEN d = 'ASC' THEN 'DESC'\n WHEN d = 'DESC' THEN 'ASC'\n END\n FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION sort_dir_to_op(_dir text, prev boolean default false) RETURNS text AS $$\n WITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n ) SELECT\n CASE\n WHEN d = 'ASC' AND prev THEN '<='\n WHEN d = 'DESC' AND prev THEN '>='\n WHEN d = 'ASC' THEN '>='\n WHEN d = 'DESC' THEN '<='\n END\n FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION sort_sqlorderby(\n _search jsonb DEFAULT NULL,\n reverse boolean DEFAULT FALSE\n) RETURNS text AS $$\n WITH sortby AS (\n SELECT coalesce(_search->'sortby','[{\"field\":\"datetime\", \"direction\":\"desc\"}]') as sort\n ), withid AS (\n SELECT CASE\n WHEN sort @? '$[*] ? (@.field == \"id\")' THEN sort\n ELSE sort || '[{\"field\":\"id\", \"direction\":\"desc\"}]'::jsonb\n END as sort\n FROM sortby\n ), withid_rows AS (\n SELECT jsonb_array_elements(sort) as value FROM withid\n ),sorts AS (\n SELECT\n coalesce(\n (queryable(value->>'field')).expression\n ) as key,\n parse_sort_dir(value->>'direction', reverse) as dir\n FROM withid_rows\n )\n SELECT array_to_string(\n array_agg(concat(key, ' ', dir)),\n ', '\n ) FROM sorts;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_sort_dir(sort_item jsonb) RETURNS text AS $$\n SELECT CASE WHEN sort_item->>'direction' ILIKE 'desc%' THEN 'DESC' ELSE 'ASC' END;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION get_token_val_str(\n _field text,\n _item items\n) RETURNS text AS $$\nDECLARE\n q text;\n literal text;\nBEGIN\n q := format($q$ SELECT quote_literal(%s) FROM (SELECT $1.*) as r;$q$, _field);\n EXECUTE q INTO literal USING _item;\n RETURN literal;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n\nCREATE OR REPLACE FUNCTION get_token_record(IN _token text, OUT prev BOOLEAN, OUT item items) RETURNS RECORD AS $$\nDECLARE\n _itemid text := _token;\n _collectionid text;\nBEGIN\n IF _token IS NULL THEN\n RETURN;\n END IF;\n RAISE NOTICE 'Looking for token: %', _token;\n prev := FALSE;\n IF _token ILIKE 'prev:%' THEN\n _itemid := replace(_token, 'prev:','');\n prev := TRUE;\n ELSIF _token ILIKE 'next:%' THEN\n _itemid := replace(_token, 'next:', '');\n END IF;\n SELECT id INTO _collectionid FROM collections WHERE _itemid LIKE concat(id,':%');\n IF FOUND THEN\n _itemid := replace(_itemid, concat(_collectionid,':'), '');\n SELECT * INTO item FROM items WHERE id=_itemid AND collection=_collectionid;\n ELSE\n SELECT * INTO item FROM items WHERE id=_itemid;\n END IF;\n IF item IS NULL THEN\n RAISE EXCEPTION 'Could not find item using token: % item: % collection: %', _token, _itemid, _collectionid;\n END IF;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\n\nCREATE OR REPLACE FUNCTION get_token_filter(\n _sortby jsonb DEFAULT '[{\"field\":\"datetime\",\"direction\":\"desc\"}]'::jsonb,\n token_item items DEFAULT NULL,\n prev boolean DEFAULT FALSE,\n inclusive boolean DEFAULT FALSE\n) RETURNS text AS $$\nDECLARE\n ltop text := '<';\n gtop text := '>';\n dir text;\n sort record;\n orfilter text := '';\n orfilters text[] := '{}'::text[];\n andfilters text[] := '{}'::text[];\n output text;\n token_where text;\nBEGIN\n IF _sortby IS NULL OR _sortby = '[]'::jsonb THEN\n _sortby := '[{\"field\":\"datetime\",\"direction\":\"desc\"}]'::jsonb;\n END IF;\n _sortby := _sortby || jsonb_build_object('field','id','direction',_sortby->0->>'direction');\n RAISE NOTICE 'Getting Token Filter. % %', _sortby, token_item;\n IF inclusive THEN\n orfilters := orfilters || format('( id=%L AND collection=%L )' , token_item.id, token_item.collection);\n END IF;\n\n FOR sort IN\n WITH s1 AS (\n SELECT\n _row,\n (queryable(value->>'field')).expression as _field,\n (value->>'field' = 'id') as _isid,\n get_sort_dir(value) as _dir\n FROM jsonb_array_elements(_sortby)\n WITH ORDINALITY AS t(value, _row)\n )\n SELECT\n _row,\n _field,\n _dir,\n get_token_val_str(_field, token_item) as _val\n FROM s1\n WHERE _row <= (SELECT min(_row) FROM s1 WHERE _isid)\n LOOP\n orfilter := NULL;\n RAISE NOTICE 'SORT: %', sort;\n IF sort._val IS NOT NULL AND ((prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC')) THEN\n orfilter := format($f$(\n (%s %s %s) OR (%s IS NULL)\n )$f$,\n sort._field,\n ltop,\n sort._val,\n sort._val\n );\n ELSIF sort._val IS NULL AND ((prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC')) THEN\n RAISE NOTICE '< but null';\n orfilter := format('%s IS NOT NULL', sort._field);\n ELSIF sort._val IS NULL THEN\n RAISE NOTICE '> but null';\n ELSE\n orfilter := format($f$(\n (%s %s %s) OR (%s IS NULL)\n )$f$,\n sort._field,\n gtop,\n sort._val,\n sort._field\n );\n END IF;\n RAISE NOTICE 'ORFILTER: %', orfilter;\n\n IF orfilter IS NOT NULL THEN\n IF sort._row = 1 THEN\n orfilters := orfilters || orfilter;\n ELSE\n orfilters := orfilters || format('(%s AND %s)', array_to_string(andfilters, ' AND '), orfilter);\n END IF;\n END IF;\n IF sort._val IS NOT NULL THEN\n andfilters := andfilters || format('%s = %s', sort._field, sort._val);\n ELSE\n andfilters := andfilters || format('%s IS NULL', sort._field);\n END IF;\n END LOOP;\n\n output := array_to_string(orfilters, ' OR ');\n\n token_where := concat('(',coalesce(output,'true'),')');\n IF trim(token_where) = '' THEN\n token_where := NULL;\n END IF;\n RAISE NOTICE 'TOKEN_WHERE: %',token_where;\n RETURN token_where;\n END;\n$$ LANGUAGE PLPGSQL SET transform_null_equals TO TRUE\n;\n\nCREATE OR REPLACE FUNCTION search_tohash(jsonb) RETURNS jsonb AS $$\n SELECT $1 - '{token,limit,context,includes,excludes}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION search_hash(jsonb, jsonb) RETURNS text AS $$\n SELECT md5(concat(search_tohash($1)::text,$2::text));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TABLE IF NOT EXISTS searches(\n hash text GENERATED ALWAYS AS (search_hash(search, metadata)) STORED PRIMARY KEY,\n search jsonb NOT NULL,\n _where text,\n orderby text,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n metadata jsonb DEFAULT '{}'::jsonb NOT NULL\n);\n\nCREATE TABLE IF NOT EXISTS search_wheres(\n id bigint generated always as identity primary key,\n _where text NOT NULL,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n statslastupdated timestamptz,\n estimated_count bigint,\n estimated_cost float,\n time_to_estimate float,\n total_count bigint,\n time_to_count float,\n partitions text[]\n);\n\nCREATE INDEX IF NOT EXISTS search_wheres_partitions ON search_wheres USING GIN (partitions);\nCREATE UNIQUE INDEX IF NOT EXISTS search_wheres_where ON search_wheres ((md5(_where)));\n\nCREATE OR REPLACE FUNCTION where_stats(\n inwhere text,\n updatestats boolean default false,\n conf jsonb default null\n) RETURNS search_wheres AS $$\nDECLARE\n t timestamptz;\n i interval;\n explain_json jsonb;\n partitions text[];\n sw search_wheres%ROWTYPE;\n inwhere_hash text := md5(inwhere);\n _context text := lower(context(conf));\n _stats_ttl interval := context_stats_ttl(conf);\n _estimated_cost_threshold float := context_estimated_cost(conf);\n _estimated_count_threshold int := context_estimated_count(conf);\n ro bool := pgstac.readonly(conf);\nBEGIN\n -- If updatestats is true then set ttl to 0\n IF updatestats THEN\n RAISE DEBUG 'Updatestats set to TRUE, setting TTL to 0';\n _stats_ttl := '0'::interval;\n END IF;\n\n -- If we don't need to calculate context, just return\n IF _context = 'off' THEN\n sw._where = inwhere;\n RETURN sw;\n END IF;\n\n -- Get any stats that we have. If there is a lock where another process is\n -- updating the stats, wait so that we don't end up calculating a bunch of times.\n SELECT * INTO sw FROM search_wheres WHERE md5(_where)=inwhere_hash FOR UPDATE;\n\n -- If there is a cached row, figure out if we need to update\n IF\n sw IS NOT NULL\n AND sw.statslastupdated IS NOT NULL\n AND sw.total_count IS NOT NULL\n AND now() - sw.statslastupdated <= _stats_ttl\n THEN\n -- we have a cached row with data that is within our ttl\n RAISE DEBUG 'Stats present in table and lastupdated within ttl: %', sw;\n IF NOT ro THEN\n RAISE DEBUG 'Updating search_wheres only bumping lastused and usecount';\n UPDATE search_wheres SET\n lastused = now(),\n usecount = search_wheres.usecount + 1\n WHERE md5(_where) = inwhere_hash\n RETURNING * INTO sw;\n END IF;\n RAISE DEBUG 'Returning cached counts. %', sw;\n RETURN sw;\n END IF;\n\n -- Calculate estimated cost and rows\n -- Use explain to get estimated count/cost\n IF sw.estimated_count IS NULL OR sw.estimated_cost IS NULL THEN\n RAISE DEBUG 'Calculating estimated stats';\n t := clock_timestamp();\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s', inwhere)\n INTO explain_json;\n RAISE DEBUG 'Time for just the explain: %', clock_timestamp() - t;\n i := clock_timestamp() - t;\n\n sw.estimated_count := explain_json->0->'Plan'->'Plan Rows';\n sw.estimated_cost := explain_json->0->'Plan'->'Total Cost';\n sw.time_to_estimate := extract(epoch from i);\n END IF;\n\n RAISE DEBUG 'ESTIMATED_COUNT: %, THRESHOLD %', sw.estimated_count, _estimated_count_threshold;\n RAISE DEBUG 'ESTIMATED_COST: %, THRESHOLD %', sw.estimated_cost, _estimated_cost_threshold;\n\n -- If context is set to auto and the costs are within the threshold return the estimated costs\n IF\n _context = 'auto'\n AND sw.estimated_count >= _estimated_count_threshold\n AND sw.estimated_cost >= _estimated_cost_threshold\n THEN\n IF NOT ro THEN\n INSERT INTO search_wheres (\n _where,\n lastused,\n usecount,\n statslastupdated,\n estimated_count,\n estimated_cost,\n time_to_estimate,\n total_count,\n time_to_count\n ) VALUES (\n inwhere,\n now(),\n 1,\n now(),\n sw.estimated_count,\n sw.estimated_cost,\n sw.time_to_estimate,\n null,\n null\n ) ON CONFLICT ((md5(_where)))\n DO UPDATE SET\n lastused = EXCLUDED.lastused,\n usecount = search_wheres.usecount + 1,\n statslastupdated = EXCLUDED.statslastupdated,\n estimated_count = EXCLUDED.estimated_count,\n estimated_cost = EXCLUDED.estimated_cost,\n time_to_estimate = EXCLUDED.time_to_estimate,\n total_count = EXCLUDED.total_count,\n time_to_count = EXCLUDED.time_to_count\n RETURNING * INTO sw;\n END IF;\n RAISE DEBUG 'Estimates are within thresholds, returning estimates. %', sw;\n RETURN sw;\n END IF;\n\n -- Calculate Actual Count\n t := clock_timestamp();\n RAISE NOTICE 'Calculating actual count...';\n EXECUTE format(\n 'SELECT count(*) FROM items WHERE %s',\n inwhere\n ) INTO sw.total_count;\n i := clock_timestamp() - t;\n RAISE NOTICE 'Actual Count: % -- %', sw.total_count, i;\n sw.time_to_count := extract(epoch FROM i);\n\n IF NOT ro THEN\n INSERT INTO search_wheres (\n _where,\n lastused,\n usecount,\n statslastupdated,\n estimated_count,\n estimated_cost,\n time_to_estimate,\n total_count,\n time_to_count\n ) VALUES (\n inwhere,\n now(),\n 1,\n now(),\n sw.estimated_count,\n sw.estimated_cost,\n sw.time_to_estimate,\n sw.total_count,\n sw.time_to_count\n ) ON CONFLICT ((md5(_where)))\n DO UPDATE SET\n lastused = EXCLUDED.lastused,\n usecount = search_wheres.usecount + 1,\n statslastupdated = EXCLUDED.statslastupdated,\n estimated_count = EXCLUDED.estimated_count,\n estimated_cost = EXCLUDED.estimated_cost,\n time_to_estimate = EXCLUDED.time_to_estimate,\n total_count = EXCLUDED.total_count,\n time_to_count = EXCLUDED.time_to_count\n RETURNING * INTO sw;\n END IF;\n RAISE DEBUG 'Returning with actual count. %', sw;\n RETURN sw;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION search_query(\n _search jsonb = '{}'::jsonb,\n updatestats boolean = false,\n _metadata jsonb = '{}'::jsonb\n) RETURNS searches AS $$\nDECLARE\n search searches%ROWTYPE;\n cached_search searches%ROWTYPE;\n pexplain jsonb;\n t timestamptz;\n i interval;\n doupdate boolean := FALSE;\n insertfound boolean := FALSE;\n ro boolean := pgstac.readonly();\n found_search text;\nBEGIN\n RAISE NOTICE 'SEARCH: %', _search;\n -- Calculate hash, where clause, and order by statement\n search.search := _search;\n search.metadata := _metadata;\n search.hash := search_hash(_search, _metadata);\n search._where := stac_search_to_where(_search);\n search.orderby := sort_sqlorderby(_search);\n search.lastused := now();\n search.usecount := 1;\n\n -- If we are in read only mode, directly return search\n IF ro THEN\n RETURN search;\n END IF;\n\n RAISE NOTICE 'Updating Statistics for search: %s', search;\n -- Update statistics for times used and and when last used\n -- If the entry is locked, rather than waiting, skip updating the stats\n INSERT INTO searches (search, lastused, usecount, metadata)\n VALUES (search.search, now(), 1, search.metadata)\n ON CONFLICT DO NOTHING\n RETURNING * INTO cached_search\n ;\n\n IF NOT FOUND OR cached_search IS NULL THEN\n UPDATE searches SET\n lastused = now(),\n usecount = searches.usecount + 1\n WHERE hash = (\n SELECT hash FROM searches WHERE hash=search.hash FOR UPDATE SKIP LOCKED\n )\n RETURNING * INTO cached_search\n ;\n END IF;\n\n IF cached_search IS NOT NULL THEN\n cached_search._where = search._where;\n cached_search.orderby = search.orderby;\n RETURN cached_search;\n END IF;\n RETURN search;\n\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE OR REPLACE FUNCTION search_fromhash(\n _hash text\n) RETURNS searches AS $$\n SELECT * FROM search_query((SELECT search FROM searches WHERE hash=_hash LIMIT 1));\n$$ LANGUAGE SQL STRICT;\n\nCREATE OR REPLACE FUNCTION search_rows(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN partitions text[] DEFAULT NULL,\n IN _limit int DEFAULT 10\n) RETURNS SETOF items AS $$\nDECLARE\n base_query text;\n query text;\n sdate timestamptz;\n edate timestamptz;\n n int;\n records_left int := _limit;\n timer timestamptz := clock_timestamp();\n full_timer timestamptz := clock_timestamp();\nBEGIN\nIF _where IS NULL OR trim(_where) = '' THEN\n _where = ' TRUE ';\nEND IF;\nRAISE NOTICE 'Getting chunks for % %', _where, _orderby;\n\nbase_query := $q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n LIMIT %L\n$q$;\n\nIF _orderby ILIKE 'datetime d%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 DESC LOOP\n RAISE NOTICE 'Running Query for % to %. %', sdate, edate, age_ms(full_timer);\n query := format(\n base_query,\n sdate,\n edate,\n _where,\n _orderby,\n records_left\n );\n RAISE DEBUG 'QUERY: %', query;\n timer := clock_timestamp();\n RETURN QUERY EXECUTE query;\n\n GET DIAGNOSTICS n = ROW_COUNT;\n records_left := records_left - n;\n RAISE NOTICE 'Returned %/% Rows From % to %. % to go. Time: %ms', n, _limit, sdate, edate, records_left, age_ms(timer);\n timer := clock_timestamp();\n IF records_left <= 0 THEN\n RAISE NOTICE 'SEARCH_ROWS TOOK %ms', age_ms(full_timer);\n RETURN;\n END IF;\n END LOOP;\nELSIF _orderby ILIKE 'datetime a%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 ASC LOOP\n RAISE NOTICE 'Running Query for % to %. %', sdate, edate, age_ms(full_timer);\n query := format(\n base_query,\n sdate,\n edate,\n _where,\n _orderby,\n records_left\n );\n RAISE DEBUG 'QUERY: %', query;\n timer := clock_timestamp();\n RETURN QUERY EXECUTE query;\n\n GET DIAGNOSTICS n = ROW_COUNT;\n records_left := records_left - n;\n RAISE NOTICE 'Returned %/% Rows From % to %. % to go. Time: %ms', n, _limit, sdate, edate, records_left, age_ms(timer);\n timer := clock_timestamp();\n IF records_left <= 0 THEN\n RAISE NOTICE 'SEARCH_ROWS TOOK %ms', age_ms(full_timer);\n RETURN;\n END IF;\n END LOOP;\nELSE\n query := format($q$\n SELECT * FROM items\n WHERE %s\n ORDER BY %s\n LIMIT %L\n $q$, _where, _orderby, _limit\n );\n RAISE DEBUG 'QUERY: %', query;\n timer := clock_timestamp();\n RETURN QUERY EXECUTE query;\n RAISE NOTICE 'FULL QUERY TOOK %ms', age_ms(timer);\nEND IF;\nRAISE NOTICE 'SEARCH_ROWS TOOK %ms', age_ms(full_timer);\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE UNLOGGED TABLE format_item_cache(\n id text,\n collection text,\n fields text,\n hydrated bool,\n output jsonb,\n lastused timestamptz DEFAULT now(),\n usecount int DEFAULT 1,\n timetoformat float,\n PRIMARY KEY (collection, id, fields, hydrated)\n);\nCREATE INDEX ON format_item_cache (lastused);\n\nCREATE OR REPLACE FUNCTION format_item(_item items, _fields jsonb DEFAULT '{}', _hydrated bool DEFAULT TRUE) RETURNS jsonb AS $$\nDECLARE\n cache bool := get_setting_bool('format_cache');\n _output jsonb := null;\n t timestamptz := clock_timestamp();\nBEGIN\n IF cache THEN\n SELECT output INTO _output FROM format_item_cache\n WHERE id=_item.id AND collection=_item.collection AND fields=_fields::text AND hydrated=_hydrated;\n END IF;\n IF _output IS NULL THEN\n IF _hydrated THEN\n _output := content_hydrate(_item, _fields);\n ELSE\n _output := content_nonhydrated(_item, _fields);\n END IF;\n END IF;\n IF cache THEN\n INSERT INTO format_item_cache (id, collection, fields, hydrated, output, timetoformat)\n VALUES (_item.id, _item.collection, _fields::text, _hydrated, _output, age_ms(t))\n ON CONFLICT(collection, id, fields, hydrated) DO\n UPDATE\n SET lastused=now(), usecount = format_item_cache.usecount + 1\n ;\n END IF;\n RETURN _output;\n\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION search(_search jsonb = '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n searches searches%ROWTYPE;\n _where text;\n orderby text;\n search_where search_wheres%ROWTYPE;\n total_count bigint;\n token record;\n token_prev boolean;\n token_item items%ROWTYPE;\n token_where text;\n full_where text;\n init_ts timestamptz := clock_timestamp();\n timer timestamptz := clock_timestamp();\n hydrate bool := NOT (_search->'conf'->>'nohydrate' IS NOT NULL AND (_search->'conf'->>'nohydrate')::boolean = true);\n prev text;\n next text;\n context jsonb;\n collection jsonb;\n out_records jsonb;\n out_len int;\n _limit int := coalesce((_search->>'limit')::int, 10);\n _querylimit int;\n _fields jsonb := coalesce(_search->'fields', '{}'::jsonb);\n has_prev boolean := FALSE;\n has_next boolean := FALSE;\n links jsonb := '[]'::jsonb;\n base_url text:= concat(rtrim(base_url(_search->'conf'),'/'));\nBEGIN\n searches := search_query(_search);\n _where := searches._where;\n orderby := searches.orderby;\n search_where := where_stats(_where);\n total_count := coalesce(search_where.total_count, search_where.estimated_count);\n RAISE NOTICE 'SEARCH:TOKEN: %', _search->>'token';\n token := get_token_record(_search->>'token');\n RAISE NOTICE '***TOKEN: %', token;\n _querylimit := _limit + 1;\n IF token IS NOT NULL THEN\n token_prev := token.prev;\n token_item := token.item;\n token_where := get_token_filter(_search->'sortby', token_item, token_prev, FALSE);\n RAISE DEBUG 'TOKEN_WHERE: % (%ms from search start)', token_where, age_ms(timer);\n IF token_prev THEN -- if we are using a prev token, we know has_next is true\n RAISE DEBUG 'There is a previous token, so automatically setting has_next to true';\n has_next := TRUE;\n orderby := sort_sqlorderby(_search, TRUE);\n ELSE\n RAISE DEBUG 'There is a next token, so automatically setting has_prev to true';\n has_prev := TRUE;\n\n END IF;\n ELSE -- if there was no token, we know there is no prev\n RAISE DEBUG 'There is no token, so we know there is no prev. setting has_prev to false';\n has_prev := FALSE;\n END IF;\n\n full_where := concat_ws(' AND ', _where, token_where);\n RAISE NOTICE 'FULL WHERE CLAUSE: %', full_where;\n RAISE NOTICE 'Time to get counts and build query %', age_ms(timer);\n timer := clock_timestamp();\n\n IF hydrate THEN\n RAISE NOTICE 'Getting hydrated data.';\n ELSE\n RAISE NOTICE 'Getting non-hydrated data.';\n END IF;\n RAISE NOTICE 'CACHE SET TO %', get_setting_bool('format_cache');\n RAISE NOTICE 'Time to set hydration/formatting %', age_ms(timer);\n timer := clock_timestamp();\n SELECT jsonb_agg(format_item(i, _fields, hydrate)) INTO out_records\n FROM search_rows(\n full_where,\n orderby,\n search_where.partitions,\n _querylimit\n ) as i;\n\n RAISE NOTICE 'Time to fetch rows %', age_ms(timer);\n timer := clock_timestamp();\n\n\n IF token_prev THEN\n out_records := flip_jsonb_array(out_records);\n END IF;\n\n RAISE NOTICE 'Query returned % records.', jsonb_array_length(out_records);\n RAISE DEBUG 'TOKEN: % %', token_item.id, token_item.collection;\n RAISE DEBUG 'RECORD_1: % %', out_records->0->>'id', out_records->0->>'collection';\n RAISE DEBUG 'RECORD-1: % %', out_records->-1->>'id', out_records->-1->>'collection';\n\n -- REMOVE records that were from our token\n IF out_records->0->>'id' = token_item.id AND out_records->0->>'collection' = token_item.collection THEN\n out_records := out_records - 0;\n ELSIF out_records->-1->>'id' = token_item.id AND out_records->-1->>'collection' = token_item.collection THEN\n out_records := out_records - -1;\n END IF;\n\n out_len := jsonb_array_length(out_records);\n\n IF out_len = _limit + 1 THEN\n IF token_prev THEN\n has_prev := TRUE;\n out_records := out_records - 0;\n ELSE\n has_next := TRUE;\n out_records := out_records - -1;\n END IF;\n END IF;\n\n\n links := links || jsonb_build_object(\n 'rel', 'root',\n 'type', 'application/json',\n 'href', base_url\n ) || jsonb_build_object(\n 'rel', 'self',\n 'type', 'application/json',\n 'href', concat(base_url, '/search')\n );\n\n IF has_next THEN\n next := concat(out_records->-1->>'collection', ':', out_records->-1->>'id');\n RAISE NOTICE 'HAS NEXT | %', next;\n links := links || jsonb_build_object(\n 'rel', 'next',\n 'type', 'application/geo+json',\n 'method', 'GET',\n 'href', concat(base_url, '/search?token=next:', next)\n );\n END IF;\n\n IF has_prev THEN\n prev := concat(out_records->0->>'collection', ':', out_records->0->>'id');\n RAISE NOTICE 'HAS PREV | %', prev;\n links := links || jsonb_build_object(\n 'rel', 'prev',\n 'type', 'application/geo+json',\n 'method', 'GET',\n 'href', concat(base_url, '/search?token=prev:', prev)\n );\n END IF;\n\n RAISE NOTICE 'Time to get prev/next %', age_ms(timer);\n timer := clock_timestamp();\n\n\n collection := jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb),\n 'links', links\n );\n\n\n\n IF context(_search->'conf') != 'off' THEN\n collection := collection || jsonb_strip_nulls(jsonb_build_object(\n 'numberMatched', total_count,\n 'numberReturned', coalesce(jsonb_array_length(out_records), 0)\n ));\n ELSE\n collection := collection || jsonb_strip_nulls(jsonb_build_object(\n 'numberReturned', coalesce(jsonb_array_length(out_records), 0)\n ));\n END IF;\n\n IF get_setting_bool('timing', _search->'conf') THEN\n collection = collection || jsonb_build_object('timing', age_ms(init_ts));\n END IF;\n\n RAISE NOTICE 'Time to build final json %', age_ms(timer);\n timer := clock_timestamp();\n\n RAISE NOTICE 'Total Time: %', age_ms(current_timestamp);\n RAISE NOTICE 'RETURNING % records. NEXT: %. PREV: %', collection->>'numberReturned', collection->>'next', collection->>'prev';\n RETURN collection;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION search_cursor(_search jsonb = '{}'::jsonb) RETURNS refcursor AS $$\nDECLARE\n curs refcursor;\n searches searches%ROWTYPE;\n _where text;\n _orderby text;\n q text;\n\nBEGIN\n searches := search_query(_search);\n _where := searches._where;\n _orderby := searches.orderby;\n\n OPEN curs FOR\n WITH p AS (\n SELECT * FROM partition_queries(_where, _orderby) p\n )\n SELECT\n CASE WHEN EXISTS (SELECT 1 FROM p) THEN\n (SELECT format($q$\n SELECT * FROM (\n %s\n ) total\n $q$,\n string_agg(\n format($q$ SELECT * FROM ( %s ) AS sub $q$, p),\n '\n UNION ALL\n '\n )\n ))\n ELSE NULL\n END FROM p;\n RETURN curs;\nEND;\n$$ LANGUAGE PLPGSQL;\nCREATE OR REPLACE VIEW collections_asitems AS\nSELECT\n id,\n geometry,\n 'collections' AS collection,\n datetime,\n end_datetime,\n jsonb_build_object(\n 'properties', content - '{links,assets,stac_version,stac_extensions}',\n 'links', content->'links',\n 'assets', content->'assets',\n 'stac_version', content->'stac_version',\n 'stac_extensions', content->'stac_extensions'\n ) AS content,\n content as collectionjson\nFROM collections;\n\n\nCREATE OR REPLACE FUNCTION collection_search_matched(\n IN _search jsonb DEFAULT '{}'::jsonb,\n OUT matched bigint\n) RETURNS bigint AS $$\nDECLARE\n _where text := stac_search_to_where(_search);\nBEGIN\n EXECUTE format(\n $query$\n SELECT\n count(*)\n FROM\n collections_asitems\n WHERE %s\n ;\n $query$,\n _where\n ) INTO matched;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION collection_search_rows(\n _search jsonb DEFAULT '{}'::jsonb\n) RETURNS SETOF jsonb AS $$\nDECLARE\n _where text := stac_search_to_where(_search);\n _limit int := coalesce((_search->>'limit')::int, 10);\n _fields jsonb := coalesce(_search->'fields', '{}'::jsonb);\n _orderby text;\n _offset int := COALESCE((_search->>'offset')::int, 0);\nBEGIN\n _orderby := sort_sqlorderby(\n jsonb_build_object(\n 'sortby',\n coalesce(\n _search->'sortby',\n '[{\"field\": \"id\", \"direction\": \"asc\"}]'::jsonb\n )\n )\n );\n RETURN QUERY EXECUTE format(\n $query$\n SELECT\n jsonb_fields(collectionjson, %L) as c\n FROM\n collections_asitems\n WHERE %s\n ORDER BY %s\n LIMIT %L\n OFFSET %L\n ;\n $query$,\n _fields,\n _where,\n _orderby,\n _limit,\n _offset\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION collection_search(\n _search jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\nDECLARE\n out_records jsonb;\n number_matched bigint := collection_search_matched(_search);\n number_returned bigint;\n _limit int := coalesce((_search->>'limit')::float::int, 10);\n _offset int := coalesce((_search->>'offset')::float::int, 0);\n links jsonb := '[]';\n ret jsonb;\n base_url text:= concat(rtrim(base_url(_search->'conf'),'/'), '/collections');\n prevoffset int;\n nextoffset int;\nBEGIN\n SELECT\n coalesce(jsonb_agg(c), '[]')\n INTO out_records\n FROM collection_search_rows(_search) c;\n\n number_returned := jsonb_array_length(out_records);\n\n IF _limit <= number_matched THEN --need to have paging links\n nextoffset := least(_offset + _limit, number_matched - 1);\n prevoffset := greatest(_offset - _limit, 0);\n IF _offset = 0 THEN -- no previous paging\n\n links := jsonb_build_array(\n jsonb_build_object(\n 'rel', 'next',\n 'type', 'application/json',\n 'method', 'GET' ,\n 'href', base_url,\n 'body', jsonb_build_object('offset', nextoffset),\n 'merge', TRUE\n )\n );\n ELSE\n links := jsonb_build_array(\n jsonb_build_object(\n 'rel', 'prev',\n 'type', 'application/json',\n 'method', 'GET' ,\n 'href', base_url,\n 'body', jsonb_build_object('offset', prevoffset),\n 'merge', TRUE\n ),\n jsonb_build_object(\n 'rel', 'next',\n 'type', 'application/json',\n 'method', 'GET' ,\n 'href', base_url,\n 'body', jsonb_build_object('offset', nextoffset),\n 'merge', TRUE\n )\n );\n END IF;\n END IF;\n\n ret := jsonb_build_object(\n 'collections', out_records,\n 'numberMatched', number_matched,\n 'numberReturned', number_returned,\n 'links', links\n );\n RETURN ret;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION tileenvelope(zoom int, x int, y int) RETURNS geometry AS $$\nWITH t AS (\n SELECT\n 20037508.3427892 as merc_max,\n -20037508.3427892 as merc_min,\n (2 * 20037508.3427892) / (2 ^ zoom) as tile_size\n)\nSELECT st_makeenvelope(\n merc_min + (tile_size * x),\n merc_max - (tile_size * (y + 1)),\n merc_min + (tile_size * (x + 1)),\n merc_max - (tile_size * y),\n 3857\n) FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;DROP FUNCTION IF EXISTS mercgrid;\n\n\nCREATE OR REPLACE FUNCTION ftime() RETURNS interval as $$\nSELECT age(clock_timestamp(), transaction_timestamp());\n$$ LANGUAGE SQL;\nSET SEARCH_PATH to pgstac, public;\n\nDROP FUNCTION IF EXISTS geometrysearch;\nCREATE OR REPLACE FUNCTION geometrysearch(\n IN geom geometry,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE, -- Return as soon as the passed in geometry is full covered\n IN skipcovered boolean DEFAULT TRUE -- Skip any items that would show up completely under the previous items\n) RETURNS jsonb AS $$\nDECLARE\n search searches%ROWTYPE;\n curs refcursor;\n _where text;\n query text;\n iter_record items%ROWTYPE;\n out_records jsonb := '{}'::jsonb[];\n exit_flag boolean := FALSE;\n counter int := 1;\n scancounter int := 1;\n remaining_limit int := _scanlimit;\n tilearea float;\n unionedgeom geometry;\n clippedgeom geometry;\n unionedgeom_area float := 0;\n prev_area float := 0;\n excludes text[];\n includes text[];\n\nBEGIN\n DROP TABLE IF EXISTS pgstac_results;\n CREATE TEMP TABLE pgstac_results (content jsonb) ON COMMIT DROP;\n\n -- If the passed in geometry is not an area set exitwhenfull and skipcovered to false\n IF ST_GeometryType(geom) !~* 'polygon' THEN\n RAISE NOTICE 'GEOMETRY IS NOT AN AREA';\n skipcovered = FALSE;\n exitwhenfull = FALSE;\n END IF;\n\n -- If skipcovered is true then you will always want to exit when the passed in geometry is full\n IF skipcovered THEN\n exitwhenfull := TRUE;\n END IF;\n\n search := search_fromhash(queryhash);\n\n IF search IS NULL THEN\n RAISE EXCEPTION 'Search with Query Hash % Not Found', queryhash;\n END IF;\n\n tilearea := st_area(geom);\n _where := format('%s AND st_intersects(geometry, %L::geometry)', search._where, geom);\n\n\n FOR query IN SELECT * FROM partition_queries(_where, search.orderby) LOOP\n query := format('%s LIMIT %L', query, remaining_limit);\n RAISE NOTICE '%', query;\n OPEN curs FOR EXECUTE query;\n LOOP\n FETCH curs INTO iter_record;\n EXIT WHEN NOT FOUND;\n IF exitwhenfull OR skipcovered THEN -- If we are not using exitwhenfull or skipcovered, we do not need to do expensive geometry operations\n clippedgeom := st_intersection(geom, iter_record.geometry);\n\n IF unionedgeom IS NULL THEN\n unionedgeom := clippedgeom;\n ELSE\n unionedgeom := st_union(unionedgeom, clippedgeom);\n END IF;\n\n unionedgeom_area := st_area(unionedgeom);\n\n IF skipcovered AND prev_area = unionedgeom_area THEN\n scancounter := scancounter + 1;\n CONTINUE;\n END IF;\n\n prev_area := unionedgeom_area;\n\n RAISE NOTICE '% % % %', unionedgeom_area/tilearea, counter, scancounter, ftime();\n END IF;\n RAISE NOTICE '% %', iter_record, content_hydrate(iter_record, fields);\n INSERT INTO pgstac_results (content) VALUES (content_hydrate(iter_record, fields));\n\n IF counter >= _limit\n OR scancounter > _scanlimit\n OR ftime() > _timelimit\n OR (exitwhenfull AND unionedgeom_area >= tilearea)\n THEN\n exit_flag := TRUE;\n EXIT;\n END IF;\n counter := counter + 1;\n scancounter := scancounter + 1;\n\n END LOOP;\n CLOSE curs;\n EXIT WHEN exit_flag;\n remaining_limit := _scanlimit - scancounter;\n END LOOP;\n\n SELECT jsonb_agg(content) INTO out_records FROM pgstac_results WHERE content IS NOT NULL;\n\n RETURN jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb)\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDROP FUNCTION IF EXISTS geojsonsearch;\nCREATE OR REPLACE FUNCTION geojsonsearch(\n IN geojson jsonb,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_geomfromgeojson(geojson),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS xyzsearch;\nCREATE OR REPLACE FUNCTION xyzsearch(\n IN _x int,\n IN _y int,\n IN _z int,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_transform(tileenvelope(_z, _x, _y), 4326),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE PROCEDURE analyze_items() AS $$\nDECLARE\n q text;\n timeout_ts timestamptz;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n SELECT format('ANALYZE (VERBOSE, SKIP_LOCKED) %I;', relname) INTO q\n FROM pg_stat_user_tables\n WHERE relname like '_item%' AND (n_mod_since_analyze>0 OR last_analyze IS NULL) LIMIT 1;\n IF NOT FOUND THEN\n EXIT;\n END IF;\n RAISE NOTICE '%', q;\n EXECUTE q;\n COMMIT;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE PROCEDURE validate_constraints() AS $$\nDECLARE\n q text;\nBEGIN\n FOR q IN\n SELECT\n FORMAT(\n 'ALTER TABLE %I.%I VALIDATE CONSTRAINT %I;',\n nsp.nspname,\n cls.relname,\n con.conname\n )\n\n FROM pg_constraint AS con\n JOIN pg_class AS cls\n ON con.conrelid = cls.oid\n JOIN pg_namespace AS nsp\n ON cls.relnamespace = nsp.oid\n WHERE convalidated = FALSE AND contype in ('c','f')\n AND nsp.nspname = 'pgstac'\n LOOP\n RAISE NOTICE '%', q;\n PERFORM run_or_queue(q);\n COMMIT;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION collection_extent(_collection text, runupdate boolean default false) RETURNS jsonb AS $$\nDECLARE\n geom_extent geometry;\n mind timestamptz;\n maxd timestamptz;\n extent jsonb;\nBEGIN\n IF runupdate THEN\n PERFORM update_partition_stats_q(partition)\n FROM partitions_view WHERE collection=_collection;\n END IF;\n SELECT\n min(lower(dtrange)),\n max(upper(edtrange)),\n st_extent(spatial)\n INTO\n mind,\n maxd,\n geom_extent\n FROM partitions_view\n WHERE collection=_collection;\n\n IF geom_extent IS NOT NULL AND mind IS NOT NULL AND maxd IS NOT NULL THEN\n extent := jsonb_build_object(\n 'spatial', jsonb_build_object(\n 'bbox', to_jsonb(array[array[st_xmin(geom_extent), st_ymin(geom_extent), st_xmax(geom_extent), st_ymax(geom_extent)]])\n ),\n 'temporal', jsonb_build_object(\n 'interval', to_jsonb(array[array[mind, maxd]])\n )\n );\n RETURN extent;\n END IF;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('id', '{\"title\": \"Item ID\",\"description\": \"Item identifier\",\"$ref\": \"https://schemas.stacspec.org/v1.0.0/item-spec/json-schema/item.json#/definitions/core/allOf/2/properties/id\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('geometry', '{\"title\": \"Item Geometry\",\"description\": \"Item Geometry\",\"$ref\": \"https://geojson.org/schema/Feature.json\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('datetime','{\"description\": \"Datetime\",\"type\": \"string\",\"title\": \"Acquired\",\"format\": \"date-time\",\"pattern\": \"(\\\\+00:00|Z)$\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDELETE FROM queryables a USING queryables b\n WHERE a.name = b.name AND a.collection_ids IS NOT DISTINCT FROM b.collection_ids AND a.id > b.id;\n\n\nINSERT INTO pgstac_settings (name, value) VALUES\n ('context', 'off'),\n ('context_estimated_count', '100000'),\n ('context_estimated_cost', '100000'),\n ('context_stats_ttl', '1 day'),\n ('default_filter_lang', 'cql2-json'),\n ('additional_properties', 'true'),\n ('use_queue', 'false'),\n ('queue_timeout', '10 minutes'),\n ('update_collection_extent', 'false'),\n ('format_cache', 'false'),\n ('readonly', 'false')\nON CONFLICT DO NOTHING\n;\n\n\nINSERT INTO cql2_ops (op, template, types) VALUES\n ('eq', '%s = %s', NULL),\n ('neq', '%s != %s', NULL),\n ('ne', '%s != %s', NULL),\n ('!=', '%s != %s', NULL),\n ('<>', '%s != %s', NULL),\n ('lt', '%s < %s', NULL),\n ('lte', '%s <= %s', NULL),\n ('gt', '%s > %s', NULL),\n ('gte', '%s >= %s', NULL),\n ('le', '%s <= %s', NULL),\n ('ge', '%s >= %s', NULL),\n ('=', '%s = %s', NULL),\n ('<', '%s < %s', NULL),\n ('<=', '%s <= %s', NULL),\n ('>', '%s > %s', NULL),\n ('>=', '%s >= %s', NULL),\n ('like', '%s LIKE %s', NULL),\n ('ilike', '%s ILIKE %s', NULL),\n ('+', '%s + %s', NULL),\n ('-', '%s - %s', NULL),\n ('*', '%s * %s', NULL),\n ('/', '%s / %s', NULL),\n ('not', 'NOT (%s)', NULL),\n ('between', '%s BETWEEN %s AND %s', NULL),\n ('isnull', '%s IS NULL', NULL),\n ('upper', 'upper(%s)', NULL),\n ('lower', 'lower(%s)', NULL),\n ('casei', 'upper(%s)', NULL),\n ('accenti', 'unaccent(%s)', NULL)\nON CONFLICT (op) DO UPDATE\n SET\n template = EXCLUDED.template\n;\n\n\nALTER FUNCTION to_text COST 5000;\nALTER FUNCTION to_float COST 5000;\nALTER FUNCTION to_int COST 5000;\nALTER FUNCTION to_tstz COST 5000;\nALTER FUNCTION to_text_array COST 5000;\n\nALTER FUNCTION update_partition_stats SECURITY DEFINER;\nALTER FUNCTION partition_after_triggerfunc SECURITY DEFINER;\nALTER FUNCTION drop_table_constraints SECURITY DEFINER;\nALTER FUNCTION create_table_constraints SECURITY DEFINER;\nALTER FUNCTION check_partition SECURITY DEFINER;\nALTER FUNCTION repartition SECURITY DEFINER;\nALTER FUNCTION where_stats SECURITY DEFINER;\nALTER FUNCTION search_query SECURITY DEFINER;\nALTER FUNCTION format_item SECURITY DEFINER;\nALTER FUNCTION maintain_index SECURITY DEFINER;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nGRANT ALL ON SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac to pgstac_admin;\n\n-- pgstac_read role limited to using function apis\nGRANT EXECUTE ON FUNCTION search TO pgstac_read;\nGRANT EXECUTE ON FUNCTION search_query TO pgstac_read;\nGRANT EXECUTE ON FUNCTION item_by_id TO pgstac_read;\nGRANT EXECUTE ON FUNCTION get_item TO pgstac_read;\nGRANT SELECT ON ALL TABLES IN SCHEMA pgstac TO pgstac_read;\n\n\nGRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_ingest;\nGRANT USAGE ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_ingest;\n\nREVOKE ALL PRIVILEGES ON PROCEDURE run_queued_queries FROM public;\nGRANT ALL ON PROCEDURE run_queued_queries TO pgstac_admin;\n\nREVOKE ALL PRIVILEGES ON FUNCTION run_queued_queries_intransaction FROM public;\nGRANT ALL ON FUNCTION run_queued_queries_intransaction TO pgstac_admin;\n\nRESET ROLE;\n\nSET ROLE pgstac_ingest;\nSELECT update_partition_stats_q(partition) FROM partitions_view;\nSELECT set_version('0.9.1');\n" }, { "path": "src/pgstac/migrations/pgstac.0.9.10-0.9.11.sql", "content": "SET client_min_messages TO WARNING;\nSET SEARCH_PATH to pgstac, public;\nRESET ROLE;\nDO $$\nDECLARE\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='postgis') THEN\n CREATE EXTENSION IF NOT EXISTS postgis;\n END IF;\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='btree_gist') THEN\n CREATE EXTENSION IF NOT EXISTS btree_gist;\n END IF;\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='unaccent') THEN\n CREATE EXTENSION IF NOT EXISTS unaccent;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_admin;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_read;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_ingest;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n\nGRANT pgstac_admin TO current_user;\n\n-- Function to make sure pgstac_admin is the owner of items\nCREATE OR REPLACE FUNCTION pgstac_admin_owns() RETURNS VOID AS $$\nDECLARE\n f RECORD;\nBEGIN\n FOR f IN (\n SELECT\n concat(\n oid::regproc::text,\n '(',\n coalesce(pg_get_function_identity_arguments(oid),''),\n ')'\n ) AS name,\n CASE prokind WHEN 'f' THEN 'FUNCTION' WHEN 'p' THEN 'PROCEDURE' WHEN 'a' THEN 'AGGREGATE' END as typ\n FROM pg_proc\n WHERE\n pronamespace=to_regnamespace('pgstac')\n AND proowner != to_regrole('pgstac_admin')\n AND proname NOT LIKE 'pg_stat%'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n FOR f IN (\n SELECT\n oid::regclass::text as name,\n CASE relkind\n WHEN 'i' THEN 'INDEX'\n WHEN 'I' THEN 'INDEX'\n WHEN 'p' THEN 'TABLE'\n WHEN 'r' THEN 'TABLE'\n WHEN 'v' THEN 'VIEW'\n WHEN 'S' THEN 'SEQUENCE'\n ELSE NULL\n END as typ\n FROM pg_class\n WHERE relnamespace=to_regnamespace('pgstac') and relowner != to_regrole('pgstac_admin') AND relkind IN ('r','p','v','S') AND relname NOT LIKE 'pg_stat'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\nSELECT pgstac_admin_owns();\n\nCREATE SCHEMA IF NOT EXISTS pgstac AUTHORIZATION pgstac_admin;\n\nGRANT ALL ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_admin;\n\nALTER ROLE pgstac_admin SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_read SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_ingest SET SEARCH_PATH TO pgstac, public;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\n\nGRANT pgstac_read TO pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\n\nSET ROLE pgstac_admin;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\nRESET ROLE;\n\nSET ROLE pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\nRESET ROLE;\n\nSET SEARCH_PATH TO pgstac, public;\nSET ROLE pgstac_admin;\n\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS analyze_items;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS validate_constraints;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n-- Install these idempotently as migrations do not put them before trying to modify the collections table\n\n\nCREATE OR REPLACE FUNCTION collection_geom(content jsonb)\nRETURNS geometry AS $$\n WITH box AS (SELECT content->'extent'->'spatial'->'bbox'->0 as box)\n SELECT\n st_makeenvelope(\n (box->>0)::float,\n (box->>1)::float,\n (box->>2)::float,\n (box->>3)::float,\n 4326\n )\n FROM box;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION collection_datetime(content jsonb)\nRETURNS timestamptz AS $$\n SELECT\n CASE\n WHEN\n (content->'extent'->'temporal'->'interval'->0->>0) IS NULL\n THEN '-infinity'::timestamptz\n ELSE\n (content->'extent'->'temporal'->'interval'->0->>0)::timestamptz\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION collection_enddatetime(content jsonb)\nRETURNS timestamptz AS $$\n SELECT\n CASE\n WHEN\n (content->'extent'->'temporal'->'interval'->0->>1) IS NULL\n THEN 'infinity'::timestamptz\n ELSE\n (content->'extent'->'temporal'->'interval'->0->>1)::timestamptz\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n-- BEGIN migra calculated SQL\ndrop function if exists \"pgstac\".\"search_tohash\"(jsonb);\n\nset check_function_bodies = off;\n\nCREATE OR REPLACE FUNCTION pgstac.get_tstz_constraint(reloid oid, colname text)\n RETURNS tstzrange\n LANGUAGE plpgsql\n STABLE STRICT\nAS $function$\nDECLARE\n expr text := NULL;\n m text[];\n ts_lower timestamptz := NULL;\n ts_upper timestamptz := NULL;\n lower_inclusive text := '[';\n upper_inclusive text := ']';\n ts timestamptz;\nBEGIN\n SELECT INTO expr\n string_agg(def, ' AND ')\n FROM pg_constraint JOIN LATERAL pg_get_constraintdef(oid) AS def ON TRUE\n WHERE\n conrelid = reloid\n AND contype = 'c'\n AND def LIKE '%' || colname || '%'\n ;\n\n IF expr IS NULL THEN\n RETURN NULL;\n END IF;\n\n RAISE DEBUG 'Constraint expression for % on %: %', colname, reloid::regclass, expr;\n -- collect all constraints for the specified column\n FOR m IN SELECT regexp_matches(expr, '[ (]' || colname || $expr$\\s*([<>=]{1,2})\\s*'([0-9 :.+\\-]+)'$expr$, 'g') LOOP\n ts := m[2]::timestamptz;\n IF m[1] IN ('>', '>=')\n THEN\n IF ts_lower IS NULL OR ts > ts_lower OR (ts = ts_lower AND m[1] = '>') THEN\n ts_lower := ts;\n lower_inclusive := CASE WHEN m[1] = '>' THEN '(' ELSE '[' END;\n END IF;\n ELSIF m[1] IN ('<', '<=')\n THEN\n IF ts_upper IS NULL OR ts < ts_upper OR (ts = ts_upper AND m[1] = '<') THEN\n ts_upper := ts;\n upper_inclusive := CASE WHEN m[1] = '<' THEN ')' ELSE ']' END;\n END IF;\n END IF;\n END LOOP;\n RAISE DEBUG 'Constraint % for %: % %', colname, reloid::regclass, ts_lower, ts_upper;\n RETURN tstzrange(ts_lower, ts_upper, lower_inclusive || upper_inclusive);\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.search_hash(jsonb, jsonb)\n RETURNS text\n LANGUAGE sql\n IMMUTABLE PARALLEL SAFE\nAS $function$\n SELECT md5(concat(($1 - '{token,limit,context,includes,excludes}'::text[])::text,$2::text));\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.update_partition_stats(_partition text, istrigger boolean DEFAULT false)\n RETURNS void\n LANGUAGE plpgsql\n STRICT SECURITY DEFINER\nAS $function$\nDECLARE\n dtrange tstzrange;\n edtrange tstzrange;\n cdtrange tstzrange;\n cedtrange tstzrange;\n extent geometry;\n collection text;\nBEGIN\n RAISE NOTICE 'Updating stats for %.', _partition;\n EXECUTE format(\n $q$\n SELECT\n tstzrange(min(datetime), max(datetime),'[]'),\n tstzrange(min(end_datetime), max(end_datetime), '[]')\n FROM %I\n $q$,\n _partition\n ) INTO dtrange, edtrange;\n EXECUTE format('ANALYZE %I;', _partition);\n extent := st_estimatedextent('pgstac', _partition, 'geometry');\n RAISE DEBUG 'Estimated Extent: %', extent;\n INSERT INTO partition_stats (partition, dtrange, edtrange, spatial, last_updated)\n SELECT _partition, dtrange, edtrange, extent, now()\n ON CONFLICT (partition) DO\n UPDATE SET\n dtrange=EXCLUDED.dtrange,\n edtrange=EXCLUDED.edtrange,\n spatial=EXCLUDED.spatial,\n last_updated=EXCLUDED.last_updated\n ;\n\n SELECT\n constraint_dtrange, constraint_edtrange, pv.collection\n INTO cdtrange, cedtrange, collection\n FROM partitions_view pv WHERE partition = _partition;\n\n RAISE NOTICE 'Checking if we need to modify constraints...';\n RAISE NOTICE 'cdtrange: % dtrange: % cedtrange: % edtrange: %',cdtrange, dtrange, cedtrange, edtrange;\n IF\n (cdtrange IS DISTINCT FROM dtrange OR edtrange IS DISTINCT FROM cedtrange)\n AND NOT istrigger\n THEN\n RAISE NOTICE 'Modifying Constraints';\n RAISE NOTICE 'Existing % %', cdtrange, cedtrange;\n RAISE NOTICE 'New % %', dtrange, edtrange;\n PERFORM drop_table_constraints(_partition);\n PERFORM create_table_constraints(_partition, dtrange, edtrange);\n END IF;\n REFRESH MATERIALIZED VIEW partitions;\n REFRESH MATERIALIZED VIEW partition_steps;\n RAISE NOTICE 'Checking if we need to update collection extents.';\n IF get_setting_bool('update_collection_extent') THEN\n RAISE NOTICE 'updating collection extent for %', collection;\n PERFORM run_or_queue(format($q$\n UPDATE collections\n SET content = jsonb_set_lax(\n content,\n '{extent}'::text[],\n collection_extent(%L, FALSE),\n true,\n 'use_json_null'\n ) WHERE id=%L\n ;\n $q$, collection, collection));\n ELSE\n RAISE NOTICE 'Not updating collection extent for %', collection;\n END IF;\n\nEND;\n$function$\n;\n\n\n-- END migra calculated SQL\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('id', '{\"title\": \"Item ID\",\"description\": \"Item identifier\",\"$ref\": \"https://schemas.stacspec.org/v1.0.0/item-spec/json-schema/item.json#/definitions/core/allOf/2/properties/id\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('geometry', '{\"title\": \"Item Geometry\",\"description\": \"Item Geometry\",\"$ref\": \"https://geojson.org/schema/Feature.json\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('datetime','{\"description\": \"Datetime\",\"type\": \"string\",\"title\": \"Acquired\",\"format\": \"date-time\",\"pattern\": \"(\\\\+00:00|Z)$\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDELETE FROM queryables a USING queryables b\n WHERE a.name = b.name AND a.collection_ids IS NOT DISTINCT FROM b.collection_ids AND a.id > b.id;\n\n\nINSERT INTO pgstac_settings (name, value) VALUES\n ('context', 'off'),\n ('context_estimated_count', '100000'),\n ('context_estimated_cost', '100000'),\n ('context_stats_ttl', '1 day'),\n ('default_filter_lang', 'cql2-json'),\n ('additional_properties', 'true'),\n ('use_queue', 'false'),\n ('queue_timeout', '10 minutes'),\n ('update_collection_extent', 'false'),\n ('format_cache', 'false'),\n ('readonly', 'false')\nON CONFLICT DO NOTHING\n;\n\n\nINSERT INTO cql2_ops (op, template, types) VALUES\n ('eq', '%s = %s', NULL),\n ('neq', '%s != %s', NULL),\n ('ne', '%s != %s', NULL),\n ('!=', '%s != %s', NULL),\n ('<>', '%s != %s', NULL),\n ('lt', '%s < %s', NULL),\n ('lte', '%s <= %s', NULL),\n ('gt', '%s > %s', NULL),\n ('gte', '%s >= %s', NULL),\n ('le', '%s <= %s', NULL),\n ('ge', '%s >= %s', NULL),\n ('=', '%s = %s', NULL),\n ('<', '%s < %s', NULL),\n ('<=', '%s <= %s', NULL),\n ('>', '%s > %s', NULL),\n ('>=', '%s >= %s', NULL),\n ('like', '%s LIKE %s', NULL),\n ('ilike', '%s ILIKE %s', NULL),\n ('+', '%s + %s', NULL),\n ('-', '%s - %s', NULL),\n ('*', '%s * %s', NULL),\n ('/', '%s / %s', NULL),\n ('not', 'NOT (%s)', NULL),\n ('between', '%s BETWEEN %s AND %s', NULL),\n ('isnull', '%s IS NULL', NULL),\n ('upper', 'upper(%s)', NULL),\n ('lower', 'lower(%s)', NULL),\n ('casei', 'upper(%s)', NULL),\n ('accenti', 'unaccent(%s)', NULL)\nON CONFLICT (op) DO UPDATE\n SET\n template = EXCLUDED.template\n;\n\n\nALTER FUNCTION to_text COST 5000;\nALTER FUNCTION to_float COST 5000;\nALTER FUNCTION to_int COST 5000;\nALTER FUNCTION to_tstz COST 5000;\nALTER FUNCTION to_text_array COST 5000;\n\nALTER FUNCTION update_partition_stats SECURITY DEFINER;\nALTER FUNCTION partition_after_triggerfunc SECURITY DEFINER;\nALTER FUNCTION drop_table_constraints SECURITY DEFINER;\nALTER FUNCTION create_table_constraints SECURITY DEFINER;\nALTER FUNCTION check_partition SECURITY DEFINER;\nALTER FUNCTION repartition SECURITY DEFINER;\nALTER FUNCTION where_stats SECURITY DEFINER;\nALTER FUNCTION search_query SECURITY DEFINER;\nALTER FUNCTION format_item SECURITY DEFINER;\nALTER FUNCTION maintain_index SECURITY DEFINER;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nGRANT ALL ON SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac to pgstac_admin;\n\n-- pgstac_read role limited to using function apis\nGRANT EXECUTE ON FUNCTION search TO pgstac_read;\nGRANT EXECUTE ON FUNCTION search_query TO pgstac_read;\nGRANT EXECUTE ON FUNCTION item_by_id TO pgstac_read;\nGRANT EXECUTE ON FUNCTION get_item TO pgstac_read;\nGRANT SELECT ON ALL TABLES IN SCHEMA pgstac TO pgstac_read;\n\n\nGRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_ingest;\nGRANT USAGE ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_ingest;\n\nREVOKE ALL PRIVILEGES ON PROCEDURE run_queued_queries FROM public;\nGRANT ALL ON PROCEDURE run_queued_queries TO pgstac_admin;\n\nREVOKE ALL PRIVILEGES ON FUNCTION run_queued_queries_intransaction FROM public;\nGRANT ALL ON FUNCTION run_queued_queries_intransaction TO pgstac_admin;\n\nRESET ROLE;\n\nSET ROLE pgstac_ingest;\nSELECT update_partition_stats_q(partition) FROM partitions_view;\nSELECT set_version('0.9.11');\n" }, { "path": "src/pgstac/migrations/pgstac.0.9.10.sql", "content": "RESET ROLE;\nDO $$\nDECLARE\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='postgis') THEN\n CREATE EXTENSION IF NOT EXISTS postgis;\n END IF;\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='btree_gist') THEN\n CREATE EXTENSION IF NOT EXISTS btree_gist;\n END IF;\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='unaccent') THEN\n CREATE EXTENSION IF NOT EXISTS unaccent;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_admin;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_read;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_ingest;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n\nGRANT pgstac_admin TO current_user;\n\n-- Function to make sure pgstac_admin is the owner of items\nCREATE OR REPLACE FUNCTION pgstac_admin_owns() RETURNS VOID AS $$\nDECLARE\n f RECORD;\nBEGIN\n FOR f IN (\n SELECT\n concat(\n oid::regproc::text,\n '(',\n coalesce(pg_get_function_identity_arguments(oid),''),\n ')'\n ) AS name,\n CASE prokind WHEN 'f' THEN 'FUNCTION' WHEN 'p' THEN 'PROCEDURE' WHEN 'a' THEN 'AGGREGATE' END as typ\n FROM pg_proc\n WHERE\n pronamespace=to_regnamespace('pgstac')\n AND proowner != to_regrole('pgstac_admin')\n AND proname NOT LIKE 'pg_stat%'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n FOR f IN (\n SELECT\n oid::regclass::text as name,\n CASE relkind\n WHEN 'i' THEN 'INDEX'\n WHEN 'I' THEN 'INDEX'\n WHEN 'p' THEN 'TABLE'\n WHEN 'r' THEN 'TABLE'\n WHEN 'v' THEN 'VIEW'\n WHEN 'S' THEN 'SEQUENCE'\n ELSE NULL\n END as typ\n FROM pg_class\n WHERE relnamespace=to_regnamespace('pgstac') and relowner != to_regrole('pgstac_admin') AND relkind IN ('r','p','v','S') AND relname NOT LIKE 'pg_stat'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\nSELECT pgstac_admin_owns();\n\nCREATE SCHEMA IF NOT EXISTS pgstac AUTHORIZATION pgstac_admin;\n\nGRANT ALL ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_admin;\n\nALTER ROLE pgstac_admin SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_read SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_ingest SET SEARCH_PATH TO pgstac, public;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\n\nGRANT pgstac_read TO pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\n\nSET ROLE pgstac_admin;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\nRESET ROLE;\n\nSET ROLE pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\nRESET ROLE;\n\nSET SEARCH_PATH TO pgstac, public;\nSET ROLE pgstac_admin;\n\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS analyze_items;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS validate_constraints;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n-- Install these idempotently as migrations do not put them before trying to modify the collections table\n\n\nCREATE OR REPLACE FUNCTION collection_geom(content jsonb)\nRETURNS geometry AS $$\n WITH box AS (SELECT content->'extent'->'spatial'->'bbox'->0 as box)\n SELECT\n st_makeenvelope(\n (box->>0)::float,\n (box->>1)::float,\n (box->>2)::float,\n (box->>3)::float,\n 4326\n )\n FROM box;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION collection_datetime(content jsonb)\nRETURNS timestamptz AS $$\n SELECT\n CASE\n WHEN\n (content->'extent'->'temporal'->'interval'->0->>0) IS NULL\n THEN '-infinity'::timestamptz\n ELSE\n (content->'extent'->'temporal'->'interval'->0->>0)::timestamptz\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION collection_enddatetime(content jsonb)\nRETURNS timestamptz AS $$\n SELECT\n CASE\n WHEN\n (content->'extent'->'temporal'->'interval'->0->>1) IS NULL\n THEN 'infinity'::timestamptz\n ELSE\n (content->'extent'->'temporal'->'interval'->0->>1)::timestamptz\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE TABLE IF NOT EXISTS migrations (\n version text PRIMARY KEY,\n datetime timestamptz DEFAULT clock_timestamp() NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION get_version() RETURNS text AS $$\n SELECT version FROM pgstac.migrations ORDER BY datetime DESC, version DESC LIMIT 1;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION set_version(text) RETURNS text AS $$\n INSERT INTO pgstac.migrations (version) VALUES ($1)\n ON CONFLICT DO NOTHING\n RETURNING version;\n$$ LANGUAGE SQL;\n\n\nCREATE TABLE IF NOT EXISTS pgstac_settings (\n name text PRIMARY KEY,\n value text NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION table_empty(text) RETURNS boolean AS $$\nDECLARE\n retval boolean;\nBEGIN\n EXECUTE format($q$\n SELECT NOT EXISTS (SELECT 1 FROM %I LIMIT 1)\n $q$,\n $1\n ) INTO retval;\n RETURN retval;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION get_setting(IN _setting text, IN conf jsonb DEFAULT NULL) RETURNS text AS $$\nSELECT COALESCE(\n nullif(conf->>_setting, ''),\n nullif(current_setting(concat('pgstac.',_setting), TRUE),''),\n nullif((SELECT value FROM pgstac.pgstac_settings WHERE name=_setting),'')\n);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_setting_bool(IN _setting text, IN conf jsonb DEFAULT NULL) RETURNS boolean AS $$\nSELECT COALESCE(\n nullif(conf->>_setting, ''),\n nullif(current_setting(concat('pgstac.',_setting), TRUE),''),\n nullif((SELECT value FROM pgstac.pgstac_settings WHERE name=_setting),''),\n 'FALSE'\n)::boolean;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION base_url(conf jsonb DEFAULT NULL) RETURNS text AS $$\n SELECT COALESCE(pgstac.get_setting('base_url', conf), '.');\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION additional_properties() RETURNS boolean AS $$\n SELECT pgstac.get_setting_bool('additional_properties');\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION readonly(conf jsonb DEFAULT NULL) RETURNS boolean AS $$\n SELECT pgstac.get_setting_bool('readonly', conf);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context(conf jsonb DEFAULT NULL) RETURNS text AS $$\n SELECT pgstac.get_setting('context', conf);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context_estimated_count(conf jsonb DEFAULT NULL) RETURNS int AS $$\n SELECT pgstac.get_setting('context_estimated_count', conf)::int;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_estimated_cost();\nCREATE OR REPLACE FUNCTION context_estimated_cost(conf jsonb DEFAULT NULL) RETURNS float AS $$\n SELECT pgstac.get_setting('context_estimated_cost', conf)::float;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_stats_ttl();\nCREATE OR REPLACE FUNCTION context_stats_ttl(conf jsonb DEFAULT NULL) RETURNS interval AS $$\n SELECT pgstac.get_setting('context_stats_ttl', conf)::interval;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION t2s(text) RETURNS text AS $$\n SELECT extract(epoch FROM $1::interval)::text || ' s';\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE STRICT;\n\nCREATE OR REPLACE FUNCTION age_ms(a timestamptz, b timestamptz DEFAULT clock_timestamp()) RETURNS float AS $$\n SELECT abs(extract(epoch from age(a,b)) * 1000);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION queue_timeout() RETURNS interval AS $$\n SELECT t2s(coalesce(\n get_setting('queue_timeout'),\n '1h'\n ))::interval;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION notice(VARIADIC text[]) RETURNS boolean AS $$\nDECLARE\ndebug boolean := current_setting('pgstac.debug', true);\nBEGIN\n IF debug THEN\n RAISE NOTICE 'NOTICE FROM FUNC: % >>>>> %', concat_ws(' | ', $1), clock_timestamp();\n RETURN TRUE;\n END IF;\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION empty_arr(ANYARRAY) RETURNS BOOLEAN AS $$\nSELECT CASE\n WHEN $1 IS NULL THEN TRUE\n WHEN cardinality($1)<1 THEN TRUE\nELSE FALSE\nEND;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_intersection(_a ANYARRAY, _b ANYARRAY) RETURNS ANYARRAY AS $$\n SELECT ARRAY ( SELECT unnest(_a) INTERSECT SELECT UNNEST(_b) );\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION array_map_ident(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_ident(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_map_literal(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_literal(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION array_reverse(anyarray) RETURNS anyarray AS $$\nSELECT ARRAY(\n SELECT $1[i]\n FROM generate_subscripts($1,1) AS s(i)\n ORDER BY i DESC\n);\n$$ LANGUAGE SQL STRICT IMMUTABLE;\n\nDROP TABLE IF EXISTS query_queue;\nCREATE TABLE query_queue (\n query text PRIMARY KEY,\n added timestamptz DEFAULT now()\n);\n\nDROP TABLE IF EXISTS query_queue_history;\nCREATE TABLE query_queue_history(\n query text,\n added timestamptz NOT NULL,\n finished timestamptz NOT NULL DEFAULT now(),\n error text\n);\n\nCREATE OR REPLACE PROCEDURE run_queued_queries() AS $$\nDECLARE\n qitem query_queue%ROWTYPE;\n timeout_ts timestamptz;\n error text;\n cnt int := 0;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n DELETE FROM query_queue WHERE query = (SELECT query FROM query_queue ORDER BY added DESC LIMIT 1 FOR UPDATE SKIP LOCKED) RETURNING * INTO qitem;\n IF NOT FOUND THEN\n EXIT;\n END IF;\n cnt := cnt + 1;\n BEGIN\n RAISE NOTICE 'RUNNING QUERY: %', qitem.query;\n EXECUTE qitem.query;\n EXCEPTION WHEN others THEN\n error := format('%s | %s', SQLERRM, SQLSTATE);\n END;\n INSERT INTO query_queue_history (query, added, finished, error)\n VALUES (qitem.query, qitem.added, clock_timestamp(), error);\n COMMIT;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION run_queued_queries_intransaction() RETURNS int AS $$\nDECLARE\n qitem query_queue%ROWTYPE;\n timeout_ts timestamptz;\n error text;\n cnt int := 0;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n DELETE FROM query_queue WHERE query = (SELECT query FROM query_queue ORDER BY added DESC LIMIT 1 FOR UPDATE SKIP LOCKED) RETURNING * INTO qitem;\n IF NOT FOUND THEN\n RETURN cnt;\n END IF;\n cnt := cnt + 1;\n BEGIN\n qitem.query := regexp_replace(qitem.query, 'CONCURRENTLY', '');\n RAISE NOTICE 'RUNNING QUERY: %', qitem.query;\n\n EXECUTE qitem.query;\n EXCEPTION WHEN others THEN\n error := format('%s | %s', SQLERRM, SQLSTATE);\n RAISE WARNING '%', error;\n END;\n INSERT INTO query_queue_history (query, added, finished, error)\n VALUES (qitem.query, qitem.added, clock_timestamp(), error);\n END LOOP;\n RETURN cnt;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION run_or_queue(query text) RETURNS VOID AS $$\nDECLARE\n use_queue text := COALESCE(get_setting('use_queue'), 'FALSE')::boolean;\nBEGIN\n IF get_setting_bool('debug') THEN\n RAISE NOTICE '%', query;\n END IF;\n IF use_queue THEN\n INSERT INTO query_queue (query) VALUES (query) ON CONFLICT DO NOTHING;\n ELSE\n EXECUTE query;\n END IF;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n\nDROP FUNCTION IF EXISTS check_pgstac_settings;\nCREATE OR REPLACE FUNCTION check_pgstac_settings(_sysmem text DEFAULT NULL) RETURNS VOID AS $$\nDECLARE\n settingval text;\n sysmem bigint := pg_size_bytes(_sysmem);\n effective_cache_size bigint := pg_size_bytes(current_setting('effective_cache_size', TRUE));\n shared_buffers bigint := pg_size_bytes(current_setting('shared_buffers', TRUE));\n work_mem bigint := pg_size_bytes(current_setting('work_mem', TRUE));\n max_connections int := current_setting('max_connections', TRUE);\n maintenance_work_mem bigint := pg_size_bytes(current_setting('maintenance_work_mem', TRUE));\n seq_page_cost float := current_setting('seq_page_cost', TRUE);\n random_page_cost float := current_setting('random_page_cost', TRUE);\n temp_buffers bigint := pg_size_bytes(current_setting('temp_buffers', TRUE));\n r record;\nBEGIN\n IF _sysmem IS NULL THEN\n RAISE NOTICE 'Call function with the size of your system memory `SELECT check_pgstac_settings(''4GB'')` to get pg system setting recommendations.';\n ELSE\n IF effective_cache_size < (sysmem * 0.5) THEN\n RAISE WARNING 'effective_cache_size of % is set low for a system with %. Recomended value between % and %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.5), pg_size_pretty(sysmem * 0.75);\n ELSIF effective_cache_size > (sysmem * 0.75) THEN\n RAISE WARNING 'effective_cache_size of % is set high for a system with %. Recomended value between % and %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.5), pg_size_pretty(sysmem * 0.75);\n ELSE\n RAISE NOTICE 'effective_cache_size of % is set appropriately for a system with %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem);\n END IF;\n\n IF shared_buffers < (sysmem * 0.2) THEN\n RAISE WARNING 'shared_buffers of % is set low for a system with %. Recomended value between % and %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.2), pg_size_pretty(sysmem * 0.3);\n ELSIF shared_buffers > (sysmem * 0.3) THEN\n RAISE WARNING 'shared_buffers of % is set high for a system with %. Recomended value between % and %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.2), pg_size_pretty(sysmem * 0.3);\n ELSE\n RAISE NOTICE 'shared_buffers of % is set appropriately for a system with %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem);\n END IF;\n shared_buffers = sysmem * 0.3;\n IF maintenance_work_mem < (sysmem * 0.2) THEN\n RAISE WARNING 'maintenance_work_mem of % is set low for shared_buffers of %. Recomended value between % and %', pg_size_pretty(maintenance_work_mem), pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers * 0.2), pg_size_pretty(shared_buffers * 0.3);\n ELSIF maintenance_work_mem > (shared_buffers * 0.3) THEN\n RAISE WARNING 'maintenance_work_mem of % is set high for shared_buffers of %. Recomended value between % and %', pg_size_pretty(maintenance_work_mem), pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers * 0.2), pg_size_pretty(shared_buffers * 0.3);\n ELSE\n RAISE NOTICE 'maintenance_work_mem of % is set appropriately for shared_buffers of %', pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers);\n END IF;\n\n IF work_mem * max_connections > shared_buffers THEN\n RAISE WARNING 'work_mem setting of % is set high for % max_connections please reduce work_mem to % or decrease max_connections to %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers/max_connections), floor(shared_buffers/work_mem);\n ELSIF work_mem * max_connections < (shared_buffers * 0.75) THEN\n RAISE WARNING 'work_mem setting of % is set low for % max_connections you may consider raising work_mem to % or increasing max_connections to %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers/max_connections), floor(shared_buffers/work_mem);\n ELSE\n RAISE NOTICE 'work_mem setting of % and max_connections of % are adequate for shared_buffers of %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers);\n END IF;\n\n IF random_page_cost / seq_page_cost != 1.1 THEN\n RAISE WARNING 'random_page_cost (%) /seq_page_cost (%) should be set to 1.1 for SSD. Change random_page_cost to %', random_page_cost, seq_page_cost, 1.1 * seq_page_cost;\n ELSE\n RAISE NOTICE 'random_page_cost and seq_page_cost set appropriately for SSD';\n END IF;\n\n IF temp_buffers < greatest(pg_size_bytes('128MB'),(maintenance_work_mem / 2)) THEN\n RAISE WARNING 'pgstac makes heavy use of temp tables, consider raising temp_buffers from % to %', pg_size_pretty(temp_buffers), greatest('128MB', pg_size_pretty((shared_buffers / 16)));\n END IF;\n END IF;\n\n RAISE NOTICE 'VALUES FOR PGSTAC VARIABLES';\n RAISE NOTICE 'These can be set either as GUC system variables or by setting in the pgstac_settings table.';\n\n FOR r IN SELECT name, get_setting(name) as setting, CASE WHEN current_setting(concat('pgstac.',name), TRUE) IS NOT NULL THEN concat('pgstac.',name, ' GUC') WHEN value IS NOT NULL THEN 'pgstac_settings table' ELSE 'Not Set' END as loc FROM pgstac_settings LOOP\n RAISE NOTICE '% is set to % from the %', r.name, r.setting, r.loc;\n END LOOP;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pg_cron';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider intalling pg_cron which can be used to automate tasks';\n ELSE\n RAISE NOTICE 'pg_cron % is installed', settingval;\n END IF;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pgstattuple';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider installing the pgstattuple extension which can be used to help maintain tables and indexes.';\n ELSE\n RAISE NOTICE 'pgstattuple % is installed', settingval;\n END IF;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pg_stat_statements';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider installing the pg_stat_statements extension which is very helpful for tracking the types of queries on the system';\n ELSE\n RAISE NOTICE 'pg_stat_statements % is installed', settingval;\n IF current_setting('pg_stat_statements.track_statements', TRUE) IS DISTINCT FROM 'all' THEN\n RAISE WARNING 'SET pg_stat_statements.track_statements TO ''all''; --In order to track statements within functions.';\n END IF;\n END IF;\n\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac, public SET CLIENT_MIN_MESSAGES TO NOTICE;\nCREATE OR REPLACE FUNCTION to_int(jsonb) RETURNS int AS $$\n SELECT floor(($1->>0)::float)::int;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION to_float(jsonb) RETURNS float AS $$\n SELECT ($1->>0)::float;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION to_tstz(jsonb) RETURNS timestamptz AS $$\n SELECT ($1->>0)::timestamptz;\n$$ LANGUAGE SQL IMMUTABLE STRICT SET TIME ZONE 'UTC' COST 5000 PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION to_text(jsonb) RETURNS text AS $$\n SELECT CASE WHEN jsonb_typeof($1) IN ('array','object') THEN $1::text ELSE $1->>0 END;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION to_text_array(jsonb) RETURNS text[] AS $$\n SELECT\n CASE jsonb_typeof($1)\n WHEN 'array' THEN ARRAY(SELECT jsonb_array_elements_text($1))\n ELSE ARRAY[$1->>0]\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION bbox_geom(_bbox jsonb) RETURNS geometry AS $$\nSELECT CASE jsonb_array_length(_bbox)\n WHEN 4 THEN\n ST_SetSRID(ST_MakeEnvelope(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float,\n (_bbox->>3)::float\n ),4326)\n WHEN 6 THEN\n ST_SetSRID(ST_3DMakeBox(\n ST_MakePoint(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float\n ),\n ST_MakePoint(\n (_bbox->>3)::float,\n (_bbox->>4)::float,\n (_bbox->>5)::float\n )\n ),4326)\n ELSE null END;\n;\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION geom_bbox(_geom geometry) RETURNS jsonb AS $$\n SELECT jsonb_build_array(\n st_xmin(_geom),\n st_ymin(_geom),\n st_xmax(_geom),\n st_ymax(_geom)\n );\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION flip_jsonb_array(j jsonb) RETURNS jsonb AS $$\n SELECT jsonb_agg(value) FROM (SELECT value FROM jsonb_array_elements(j) WITH ORDINALITY ORDER BY ordinality DESC) as t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION explode_dotpaths(j jsonb) RETURNS SETOF text[] AS $$\n SELECT string_to_array(p, '.') as e FROM jsonb_array_elements_text(j) p;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION explode_dotpaths_recurse(IN j jsonb) RETURNS SETOF text[] AS $$\n WITH RECURSIVE t AS (\n SELECT e FROM explode_dotpaths(j) e\n UNION ALL\n SELECT e[1:cardinality(e)-1]\n FROM t\n WHERE cardinality(e)>1\n ) SELECT e FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION jsonb_set_nested(j jsonb, path text[], val jsonb) RETURNS jsonb AS $$\nDECLARE\nBEGIN\n IF cardinality(path) > 1 THEN\n FOR i IN 1..(cardinality(path)-1) LOOP\n IF j #> path[:i] IS NULL THEN\n j := jsonb_set_lax(j, path[:i], '{}', TRUE);\n END IF;\n END LOOP;\n END IF;\n RETURN jsonb_set_lax(j, path, val, true);\n\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\n\n\nCREATE OR REPLACE FUNCTION jsonb_include(j jsonb, f jsonb) RETURNS jsonb AS $$\nDECLARE\n includes jsonb := f-> 'include';\n outj jsonb := '{}'::jsonb;\n path text[];\nBEGIN\n IF\n includes IS NULL\n OR jsonb_array_length(includes) = 0\n THEN\n RETURN j;\n ELSE\n includes := includes || (\n CASE WHEN j ? 'collection' THEN\n '[\"id\",\"collection\"]'\n ELSE\n '[\"id\"]'\n END)::jsonb;\n FOR path IN SELECT explode_dotpaths(includes) LOOP\n outj := jsonb_set_nested(outj, path, j #> path);\n END LOOP;\n END IF;\n RETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_exclude(j jsonb, f jsonb) RETURNS jsonb AS $$\nDECLARE\n excludes jsonb := f-> 'exclude';\n outj jsonb := j;\n path text[];\nBEGIN\n IF\n excludes IS NULL\n OR jsonb_array_length(excludes) = 0\n THEN\n RETURN j;\n ELSE\n FOR path IN SELECT explode_dotpaths(excludes) LOOP\n outj := outj #- path;\n END LOOP;\n END IF;\n RETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_fields(j jsonb, f jsonb DEFAULT '{\"fields\":[]}') RETURNS jsonb AS $$\n SELECT jsonb_exclude(jsonb_include(j, f), f);\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION merge_jsonb(_a jsonb, _b jsonb) RETURNS jsonb AS $$\n SELECT\n CASE\n WHEN _a = '\"𒍟※\"'::jsonb THEN NULL\n WHEN _a IS NULL OR jsonb_typeof(_a) = 'null' THEN _b\n WHEN jsonb_typeof(_a) = 'object' AND jsonb_typeof(_b) = 'object' THEN\n (\n SELECT\n jsonb_strip_nulls(\n jsonb_object_agg(\n key,\n merge_jsonb(a.value, b.value)\n )\n )\n FROM\n jsonb_each(coalesce(_a,'{}'::jsonb)) as a\n FULL JOIN\n jsonb_each(coalesce(_b,'{}'::jsonb)) as b\n USING (key)\n )\n WHEN\n jsonb_typeof(_a) = 'array'\n AND jsonb_typeof(_b) = 'array'\n AND jsonb_array_length(_a) = jsonb_array_length(_b)\n THEN\n (\n SELECT jsonb_agg(m) FROM\n ( SELECT\n merge_jsonb(\n jsonb_array_elements(_a),\n jsonb_array_elements(_b)\n ) as m\n ) as l\n )\n ELSE _a\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION strip_jsonb(_a jsonb, _b jsonb) RETURNS jsonb AS $$\n SELECT\n CASE\n\n WHEN (_a IS NULL OR jsonb_typeof(_a) = 'null') AND _b IS NOT NULL AND jsonb_typeof(_b) != 'null' THEN '\"𒍟※\"'::jsonb\n WHEN _b IS NULL OR jsonb_typeof(_a) = 'null' THEN _a\n WHEN _a = _b AND jsonb_typeof(_a) = 'object' THEN '{}'::jsonb\n WHEN _a = _b THEN NULL\n WHEN jsonb_typeof(_a) = 'object' AND jsonb_typeof(_b) = 'object' THEN\n (\n SELECT\n jsonb_strip_nulls(\n jsonb_object_agg(\n key,\n strip_jsonb(a.value, b.value)\n )\n )\n FROM\n jsonb_each(_a) as a\n FULL JOIN\n jsonb_each(_b) as b\n USING (key)\n )\n WHEN\n jsonb_typeof(_a) = 'array'\n AND jsonb_typeof(_b) = 'array'\n AND jsonb_array_length(_a) = jsonb_array_length(_b)\n THEN\n (\n SELECT jsonb_agg(m) FROM\n ( SELECT\n strip_jsonb(\n jsonb_array_elements(_a),\n jsonb_array_elements(_b)\n ) as m\n ) as l\n )\n ELSE _a\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION nullif_jsonbnullempty(j jsonb) RETURNS jsonb AS $$\n SELECT nullif(nullif(nullif(j,'null'::jsonb),'{}'::jsonb),'[]'::jsonb);\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION jsonb_array_unique(j jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(jsonb_agg(DISTINCT a)) v FROM jsonb_array_elements(j) a;\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_concat_ignorenull(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT coalesce(a,'[]'::jsonb) || coalesce(b,'[]'::jsonb);\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_least(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(least(nullif_jsonbnullempty(a), nullif_jsonbnullempty(b)));\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_greatest(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(greatest(a, b));\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION first_notnull_sfunc(anyelement, anyelement) RETURNS anyelement AS $$\n SELECT COALESCE($1,$2);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE AGGREGATE first_notnull(anyelement)(\n SFUNC = first_notnull_sfunc,\n STYPE = anyelement\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_array_unique_merge(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_concat_ignorenull,\n FINALFUNC = jsonb_array_unique\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_min(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_least\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_max(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_greatest\n);\n/* looks for a geometry in a stac item first from geometry and falling back to bbox */\nCREATE OR REPLACE FUNCTION stac_geom(value jsonb) RETURNS geometry AS $$\nSELECT\n CASE\n WHEN value ? 'intersects' THEN\n ST_GeomFromGeoJSON(value->>'intersects')\n WHEN value ? 'geometry' THEN\n ST_GeomFromGeoJSON(value->>'geometry')\n WHEN value ? 'bbox' THEN\n pgstac.bbox_geom(value->'bbox')\n ELSE NULL\n END as geometry\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION stac_daterange(\n value jsonb\n) RETURNS tstzrange AS $$\nDECLARE\n props jsonb := value;\n dt timestamptz;\n edt timestamptz;\nBEGIN\n IF props ? 'properties' THEN\n props := props->'properties';\n END IF;\n IF\n props ? 'start_datetime'\n AND props->>'start_datetime' IS NOT NULL\n AND props ? 'end_datetime'\n AND props->>'end_datetime' IS NOT NULL\n THEN\n dt := props->>'start_datetime';\n edt := props->>'end_datetime';\n IF dt > edt THEN\n RAISE EXCEPTION 'start_datetime must be < end_datetime';\n END IF;\n ELSE\n dt := props->>'datetime';\n edt := props->>'datetime';\n END IF;\n IF dt is NULL OR edt IS NULL THEN\n RAISE NOTICE 'DT: %, EDT: %', dt, edt;\n RAISE EXCEPTION 'Either datetime (%) or both start_datetime (%) and end_datetime (%) must be set.', props->>'datetime',props->>'start_datetime',props->>'end_datetime';\n END IF;\n RETURN tstzrange(dt, edt, '[]');\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_datetime(value jsonb) RETURNS timestamptz AS $$\n SELECT lower(stac_daterange(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_end_datetime(value jsonb) RETURNS timestamptz AS $$\n SELECT upper(stac_daterange(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE TABLE IF NOT EXISTS stac_extensions(\n url text PRIMARY KEY,\n content jsonb\n);\nCREATE OR REPLACE FUNCTION collection_base_item(content jsonb) RETURNS jsonb AS $$\n SELECT jsonb_build_object(\n 'type', 'Feature',\n 'stac_version', content->'stac_version',\n 'assets', content->'item_assets',\n 'collection', content->'id'\n );\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE TABLE IF NOT EXISTS collections (\n key bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,\n id text GENERATED ALWAYS AS (content->>'id') STORED UNIQUE NOT NULL,\n content JSONB NOT NULL,\n base_item jsonb GENERATED ALWAYS AS (pgstac.collection_base_item(content)) STORED,\n geometry geometry GENERATED ALWAYS AS (pgstac.collection_geom(content)) STORED,\n datetime timestamptz GENERATED ALWAYS AS (pgstac.collection_datetime(content)) STORED,\n end_datetime timestamptz GENERATED ALWAYS AS (pgstac.collection_enddatetime(content)) STORED,\n private jsonb,\n partition_trunc text CHECK (partition_trunc IN ('year', 'month'))\n);\n\nCREATE OR REPLACE FUNCTION collection_base_item(cid text) RETURNS jsonb AS $$\n SELECT pgstac.collection_base_item(content) FROM pgstac.collections WHERE id = cid LIMIT 1;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION create_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION update_collection(data jsonb) RETURNS VOID AS $$\nDECLARE\n out collections%ROWTYPE;\nBEGIN\n UPDATE collections SET content=data WHERE id = data->>'id' RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ON CONFLICT (id) DO\n UPDATE\n SET content=EXCLUDED.content\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION delete_collection(_id text) RETURNS VOID AS $$\nDECLARE\n out collections%ROWTYPE;\nBEGIN\n DELETE FROM collections WHERE id = _id RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION get_collection(id text) RETURNS jsonb AS $$\n SELECT content FROM collections\n WHERE id=$1\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION all_collections() RETURNS jsonb AS $$\n SELECT coalesce(jsonb_agg(content), '[]'::jsonb) FROM collections;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION collection_delete_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n collection_base_partition text := concat('_items_', OLD.key);\nBEGIN\n EXECUTE format($q$\n DELETE FROM partition_stats WHERE partition IN (\n SELECT partition FROM partition_sys_meta\n WHERE collection=%L\n );\n DROP TABLE IF EXISTS %I CASCADE;\n $q$,\n OLD.id,\n collection_base_partition\n );\n RETURN OLD;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER collection_delete_trigger BEFORE DELETE ON collections\nFOR EACH ROW EXECUTE FUNCTION collection_delete_trigger_func();\nCREATE OR REPLACE FUNCTION queryable_signature(n text, c text[]) RETURNS text AS $$\n SELECT concat(n, c);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TABLE queryables (\n id bigint GENERATED ALWAYS AS identity PRIMARY KEY,\n name text NOT NULL,\n collection_ids text[], -- used to determine what partitions to create indexes on\n definition jsonb,\n property_path text,\n property_wrapper text,\n property_index_type text\n);\nCREATE INDEX queryables_name_idx ON queryables (name);\nCREATE INDEX queryables_collection_idx ON queryables USING GIN (collection_ids);\nCREATE INDEX queryables_property_wrapper_idx ON queryables (property_wrapper);\n\nCREATE OR REPLACE FUNCTION pgstac.queryables_constraint_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n allcollections text[];\nBEGIN\n RAISE NOTICE 'Making sure that name/collection is unique for queryables %', NEW;\n IF NEW.collection_ids IS NOT NULL THEN\n IF EXISTS (\n SELECT 1\n FROM unnest(NEW.collection_ids) c\n LEFT JOIN\n collections\n ON (collections.id = c)\n WHERE collections.id IS NULL\n ) THEN\n RAISE foreign_key_violation USING MESSAGE = format(\n 'One or more collections in %s do not exist.', NEW.collection_ids\n );\n RETURN NULL;\n END IF;\n END IF;\n IF TG_OP = 'INSERT' THEN\n IF EXISTS (\n SELECT 1 FROM queryables q\n WHERE\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n )\n ) THEN\n RAISE unique_violation USING MESSAGE = format(\n 'There is already a queryable for %s for a collection in %s: %s',\n NEW.name,\n NEW.collection_ids,\n\t\t\t\t(SELECT json_agg(row_to_json(q)) FROM queryables q WHERE\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n ))\n );\n RETURN NULL;\n END IF;\n END IF;\n IF TG_OP = 'UPDATE' THEN\n IF EXISTS (\n SELECT 1 FROM queryables q\n WHERE\n q.id != NEW.id\n AND\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n )\n ) THEN\n RAISE unique_violation\n USING MESSAGE = format(\n 'There is already a queryable for %s for a collection in %s',\n NEW.name,\n NEW.collection_ids\n );\n RETURN NULL;\n END IF;\n END IF;\n\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER queryables_constraint_insert_trigger\nBEFORE INSERT ON queryables\nFOR EACH ROW EXECUTE PROCEDURE queryables_constraint_triggerfunc();\n\nCREATE TRIGGER queryables_constraint_update_trigger\nBEFORE UPDATE ON queryables\nFOR EACH ROW\nWHEN (NEW.name = OLD.name AND NEW.collection_ids IS DISTINCT FROM OLD.collection_ids)\nEXECUTE PROCEDURE queryables_constraint_triggerfunc();\n\n\nCREATE OR REPLACE FUNCTION array_to_path(arr text[]) RETURNS text AS $$\n SELECT string_agg(\n quote_literal(v),\n '->'\n ) FROM unnest(arr) v;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\n\n\nCREATE OR REPLACE FUNCTION queryable(\n IN dotpath text,\n OUT path text,\n OUT expression text,\n OUT wrapper text,\n OUT nulled_wrapper text\n) AS $$\nDECLARE\n q RECORD;\n path_elements text[];\nBEGIN\n dotpath := replace(dotpath, 'properties.', '');\n IF dotpath = 'start_datetime' THEN\n dotpath := 'datetime';\n END IF;\n IF dotpath IN ('id', 'geometry', 'datetime', 'end_datetime', 'collection') THEN\n path := dotpath;\n expression := dotpath;\n wrapper := NULL;\n RETURN;\n END IF;\n\n SELECT * INTO q FROM queryables\n WHERE\n name=dotpath\n OR name = 'properties.' || dotpath\n OR name = replace(dotpath, 'properties.', '')\n ;\n IF q.property_wrapper IS NULL THEN\n IF q.definition->>'type' = 'number' THEN\n wrapper := 'to_float';\n nulled_wrapper := wrapper;\n ELSIF q.definition->>'format' = 'date-time' THEN\n wrapper := 'to_tstz';\n nulled_wrapper := wrapper;\n ELSE\n nulled_wrapper := NULL;\n wrapper := 'to_text';\n END IF;\n ELSE\n wrapper := q.property_wrapper;\n nulled_wrapper := wrapper;\n END IF;\n IF q.property_path IS NOT NULL THEN\n path := q.property_path;\n ELSE\n path_elements := string_to_array(dotpath, '.');\n IF path_elements[1] IN ('links', 'assets', 'stac_version', 'stac_extensions') THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSIF path_elements[1] = 'properties' THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSE\n path := format($F$content->'properties'->%s$F$, array_to_path(path_elements));\n END IF;\n END IF;\n expression := format('%I(%s)', wrapper, path);\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\nCREATE OR REPLACE FUNCTION unnest_collection(collection_ids text[] DEFAULT NULL) RETURNS SETOF text AS $$\n DECLARE\n BEGIN\n IF collection_ids IS NULL THEN\n RETURN QUERY SELECT id FROM collections;\n END IF;\n RETURN QUERY SELECT unnest(collection_ids);\n END;\n$$ LANGUAGE PLPGSQL STABLE;\n\nCREATE OR REPLACE FUNCTION normalize_indexdef(def text) RETURNS text AS $$\nDECLARE\nBEGIN\n def := btrim(def, ' \\n\\t');\n\tdef := regexp_replace(def, '^CREATE (UNIQUE )?INDEX ([^ ]* )?ON (ONLY )?([^ ]* )?', '', 'i');\n RETURN def;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION indexdef(q queryables) RETURNS text AS $$\n DECLARE\n out text;\n BEGIN\n IF q.name = 'id' THEN\n out := 'CREATE UNIQUE INDEX ON %I USING btree (id)';\n ELSIF q.name = 'datetime' THEN\n out := 'CREATE INDEX ON %I USING btree (datetime DESC, end_datetime)';\n ELSIF q.name = 'geometry' THEN\n out := 'CREATE INDEX ON %I USING gist (geometry)';\n ELSE\n out := format($q$CREATE INDEX ON %%I USING %s (%s(((content -> 'properties'::text) -> %L::text)))$q$,\n lower(COALESCE(q.property_index_type, 'BTREE')),\n lower(COALESCE(q.property_wrapper, 'to_text')),\n q.name\n );\n END IF;\n RETURN btrim(out, ' \\n\\t');\n END;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nDROP VIEW IF EXISTS pgstac_indexes;\nCREATE VIEW pgstac_indexes AS\nSELECT\n i.schemaname,\n i.tablename,\n i.indexname,\n regexp_replace(btrim(replace(replace(indexdef, i.indexname, ''),'pgstac.',''),' \\t\\n'), '[ ]+', ' ', 'g') as idx,\n COALESCE(\n (regexp_match(indexdef, '\\(([a-zA-Z]+)\\)'))[1],\n (regexp_match(indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_-]+)''::text'))[1],\n CASE WHEN indexdef ~* '\\(datetime desc, end_datetime\\)' THEN 'datetime' ELSE NULL END\n ) AS field,\n pg_table_size(i.indexname::text) as index_size,\n pg_size_pretty(pg_table_size(i.indexname::text)) as index_size_pretty\nFROM\n pg_indexes i\nWHERE i.schemaname='pgstac' and i.tablename ~ '_items_' AND indexdef !~* ' only ';\n\nDROP VIEW IF EXISTS pgstac_index_stats;\nCREATE VIEW pgstac_indexes_stats AS\nSELECT\n i.schemaname,\n i.tablename,\n i.indexname,\n indexdef,\n COALESCE(\n (regexp_match(indexdef, '\\(([a-zA-Z]+)\\)'))[1],\n (regexp_match(indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_]+)''::text'))[1],\n CASE WHEN indexdef ~* '\\(datetime desc, end_datetime\\)' THEN 'datetime_end_datetime' ELSE NULL END\n ) AS field,\n pg_table_size(i.indexname::text) as index_size,\n pg_size_pretty(pg_table_size(i.indexname::text)) as index_size_pretty,\n n_distinct,\n most_common_vals::text::text[],\n most_common_freqs::text::text[],\n histogram_bounds::text::text[],\n correlation\nFROM\n pg_indexes i\n LEFT JOIN pg_stats s ON (s.tablename = i.indexname)\nWHERE i.schemaname='pgstac' and i.tablename ~ '_items_';\n\nCREATE OR REPLACE FUNCTION queryable_indexes(\n IN treeroot text DEFAULT 'items',\n IN changes boolean DEFAULT FALSE,\n OUT collection text,\n OUT partition text,\n OUT field text,\n OUT indexname text,\n OUT existing_idx text,\n OUT queryable_idx text\n) RETURNS SETOF RECORD AS $$\nWITH p AS (\n SELECT\n relid::text as partition,\n replace(replace(\n CASE\n WHEN parentrelid::regclass::text='items' THEN pg_get_expr(c.relpartbound, c.oid)\n ELSE pg_get_expr(parent.relpartbound, parent.oid)\n END,\n 'FOR VALUES IN (''',''), ''')',\n ''\n ) AS collection\n FROM pg_partition_tree(treeroot)\n JOIN pg_class c ON (relid::regclass = c.oid)\n JOIN pg_class parent ON (parentrelid::regclass = parent.oid AND isleaf)\n ), i AS (\n SELECT\n partition,\n indexname,\n regexp_replace(btrim(replace(replace(indexdef, indexname, ''),'pgstac.',''),' \\t\\n'), '[ ]+', ' ', 'g') as iidx,\n COALESCE(\n (regexp_match(indexdef, '\\(([a-zA-Z]+)\\)'))[1],\n (regexp_match(indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_-]+)''::text'))[1],\n CASE WHEN indexdef ~* '\\(datetime desc, end_datetime\\)' THEN 'datetime' ELSE NULL END\n ) AS field\n FROM\n pg_indexes\n JOIN p ON (tablename=partition)\n ), q AS (\n SELECT\n name AS field,\n collection,\n partition,\n format(indexdef(queryables), partition) as qidx\n FROM queryables, unnest_collection(queryables.collection_ids) collection\n JOIN p USING (collection)\n WHERE property_index_type IS NOT NULL OR name IN ('datetime','geometry','id')\n )\n SELECT\n collection,\n partition,\n field,\n indexname,\n iidx as existing_idx,\n qidx as queryable_idx\n FROM i FULL JOIN q USING (field, partition)\n WHERE CASE WHEN changes THEN lower(iidx) IS DISTINCT FROM lower(qidx) ELSE TRUE END;\n;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION maintain_index(\n indexname text,\n queryable_idx text,\n dropindexes boolean DEFAULT FALSE,\n rebuildindexes boolean DEFAULT FALSE,\n idxconcurrently boolean DEFAULT FALSE\n) RETURNS VOID AS $$\nDECLARE\nBEGIN\n IF indexname IS NOT NULL THEN\n IF dropindexes OR queryable_idx IS NOT NULL THEN\n EXECUTE format('DROP INDEX IF EXISTS %I;', indexname);\n ELSIF rebuildindexes THEN\n IF idxconcurrently THEN\n EXECUTE format('REINDEX INDEX CONCURRENTLY %I;', indexname);\n ELSE\n EXECUTE format('REINDEX INDEX CONCURRENTLY %I;', indexname);\n END IF;\n END IF;\n END IF;\n IF queryable_idx IS NOT NULL THEN\n IF idxconcurrently THEN\n EXECUTE replace(queryable_idx, 'INDEX', 'INDEX CONCURRENTLY');\n ELSE EXECUTE queryable_idx;\n END IF;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\n\nset check_function_bodies to off;\nCREATE OR REPLACE FUNCTION maintain_partition_queries(\n part text DEFAULT 'items',\n dropindexes boolean DEFAULT FALSE,\n rebuildindexes boolean DEFAULT FALSE,\n idxconcurrently boolean DEFAULT FALSE\n) RETURNS SETOF text AS $$\nDECLARE\n rec record;\n q text;\nBEGIN\n FOR rec IN (\n SELECT * FROM queryable_indexes(part,true)\n ) LOOP\n q := format(\n 'SELECT maintain_index(\n %L,%L,%L,%L,%L\n );',\n rec.indexname,\n rec.queryable_idx,\n dropindexes,\n rebuildindexes,\n idxconcurrently\n );\n RAISE NOTICE 'Q: %', q;\n RETURN NEXT q;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION maintain_partitions(\n part text DEFAULT 'items',\n dropindexes boolean DEFAULT FALSE,\n rebuildindexes boolean DEFAULT FALSE\n) RETURNS VOID AS $$\n WITH t AS (\n SELECT run_or_queue(q) FROM maintain_partition_queries(part, dropindexes, rebuildindexes) q\n ) SELECT count(*) FROM t;\n$$ LANGUAGE SQL;\n\n\nCREATE OR REPLACE FUNCTION queryables_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\nBEGIN\n PERFORM maintain_partitions();\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER queryables_trigger AFTER INSERT OR UPDATE ON queryables\nFOR EACH STATEMENT EXECUTE PROCEDURE queryables_trigger_func();\n\n\nCREATE OR REPLACE FUNCTION get_queryables(_collection_ids text[] DEFAULT NULL) RETURNS jsonb AS $$\nDECLARE\nBEGIN\n -- Build up queryables if the input contains valid collection ids or is empty\n IF EXISTS (\n SELECT 1 FROM collections\n WHERE\n _collection_ids IS NULL\n OR cardinality(_collection_ids) = 0\n OR id = ANY(_collection_ids)\n )\n THEN\n RETURN (\n WITH base AS (\n SELECT\n unnest(collection_ids) as collection_id,\n name,\n coalesce(definition, '{\"type\":\"string\"}'::jsonb) as definition\n FROM queryables\n WHERE\n _collection_ids IS NULL OR\n _collection_ids = '{}'::text[] OR\n _collection_ids && collection_ids\n UNION ALL\n SELECT null, name, coalesce(definition, '{\"type\":\"string\"}'::jsonb) as definition\n FROM queryables WHERE collection_ids IS NULL OR collection_ids = '{}'::text[]\n ), g AS (\n SELECT\n name,\n first_notnull(definition) as definition,\n jsonb_array_unique_merge(definition->'enum') as enum,\n jsonb_min(definition->'minimum') as minimum,\n jsonb_min(definition->'maxiumn') as maximum\n FROM base\n GROUP BY 1\n )\n SELECT\n jsonb_build_object(\n '$schema', 'http://json-schema.org/draft-07/schema#',\n '$id', '',\n 'type', 'object',\n 'title', 'STAC Queryables.',\n 'properties', jsonb_object_agg(\n name,\n definition\n ||\n jsonb_strip_nulls(jsonb_build_object(\n 'enum', enum,\n 'minimum', minimum,\n 'maximum', maximum\n ))\n ),\n 'additionalProperties', pgstac.additional_properties()\n )\n FROM g\n );\n ELSE\n RETURN NULL;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\nCREATE OR REPLACE FUNCTION get_queryables(_collection text DEFAULT NULL) RETURNS jsonb AS $$\n SELECT\n CASE\n WHEN _collection IS NULL THEN get_queryables(NULL::text[])\n ELSE get_queryables(ARRAY[_collection])\n END\n ;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_queryables() RETURNS jsonb AS $$\n SELECT get_queryables(NULL::text[]);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION schema_qualify_refs(url text, j jsonb) returns jsonb as $$\n SELECT regexp_replace(j::text, '\"\\$ref\": \"#', concat('\"$ref\": \"', url, '#'), 'g')::jsonb;\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE VIEW stac_extension_queryables AS\nSELECT DISTINCT key as name, schema_qualify_refs(e.url, j.value) as definition FROM stac_extensions e, jsonb_each(e.content->'definitions'->'fields'->'properties') j;\n\n\nCREATE OR REPLACE FUNCTION missing_queryables(_collection text, _tablesample float DEFAULT 5, minrows float DEFAULT 10) RETURNS TABLE(collection text, name text, definition jsonb, property_wrapper text) AS $$\nDECLARE\n q text;\n _partition text;\n explain_json json;\n psize float;\n estrows float;\nBEGIN\n SELECT format('_items_%s', key) INTO _partition FROM collections WHERE id=_collection;\n\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM %I;', _partition)\n INTO explain_json;\n psize := explain_json->0->'Plan'->'Plan Rows';\n estrows := _tablesample * .01 * psize;\n IF estrows < minrows THEN\n _tablesample := least(100,greatest(_tablesample, (estrows / psize) / 100));\n RAISE NOTICE '%', (psize / estrows) / 100;\n END IF;\n RAISE NOTICE 'Using tablesample % to find missing queryables from % % that has ~% rows estrows: %', _tablesample, _collection, _partition, psize, estrows;\n\n q := format(\n $q$\n WITH q AS (\n SELECT * FROM queryables\n WHERE\n collection_ids IS NULL\n OR %L = ANY(collection_ids)\n ), t AS (\n SELECT\n content->'properties' AS properties\n FROM\n %I\n TABLESAMPLE SYSTEM(%L)\n ), p AS (\n SELECT DISTINCT ON (key)\n key,\n value,\n s.definition\n FROM t\n JOIN LATERAL jsonb_each(properties) ON TRUE\n LEFT JOIN q ON (q.name=key)\n LEFT JOIN stac_extension_queryables s ON (s.name=key)\n WHERE q.definition IS NULL\n )\n SELECT\n %L,\n key,\n COALESCE(definition, jsonb_build_object('type',jsonb_typeof(value))) as definition,\n CASE\n WHEN definition->>'type' = 'integer' THEN 'to_int'\n WHEN COALESCE(definition->>'type', jsonb_typeof(value)) = 'number' THEN 'to_float'\n WHEN COALESCE(definition->>'type', jsonb_typeof(value)) = 'array' THEN 'to_text_array'\n ELSE 'to_text'\n END\n FROM p;\n $q$,\n _collection,\n _partition,\n _tablesample,\n _collection\n );\n RETURN QUERY EXECUTE q;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION missing_queryables(_tablesample float DEFAULT 5) RETURNS TABLE(collection_ids text[], name text, definition jsonb, property_wrapper text) AS $$\n SELECT\n array_agg(collection),\n name,\n definition,\n property_wrapper\n FROM\n collections\n JOIN LATERAL\n missing_queryables(id, _tablesample) c\n ON TRUE\n GROUP BY\n 2,3,4\n ORDER BY 2,1\n ;\n$$ LANGUAGE SQL;\nCREATE OR REPLACE FUNCTION parse_dtrange(\n _indate jsonb,\n relative_base timestamptz DEFAULT date_trunc('hour', CURRENT_TIMESTAMP)\n) RETURNS tstzrange AS $$\nDECLARE\n timestrs text[];\n s timestamptz;\n e timestamptz;\nBEGIN\n timestrs :=\n CASE\n WHEN _indate ? 'timestamp' THEN\n ARRAY[_indate->>'timestamp']\n WHEN _indate ? 'interval' THEN\n to_text_array(_indate->'interval')\n WHEN jsonb_typeof(_indate) = 'array' THEN\n to_text_array(_indate)\n ELSE\n regexp_split_to_array(\n _indate->>0,\n '/'\n )\n END;\n RAISE NOTICE 'TIMESTRS %', timestrs;\n IF cardinality(timestrs) = 1 THEN\n IF timestrs[1] ILIKE 'P%' THEN\n RETURN tstzrange(relative_base - upper(timestrs[1])::interval, relative_base, '[)');\n END IF;\n s := timestrs[1]::timestamptz;\n RETURN tstzrange(s, s, '[]');\n END IF;\n\n IF cardinality(timestrs) != 2 THEN\n RAISE EXCEPTION 'Timestamp cannot have more than 2 values';\n END IF;\n\n IF timestrs[1] = '..' OR timestrs[1] = '' THEN\n s := '-infinity'::timestamptz;\n e := timestrs[2]::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] = '..' OR timestrs[2] = '' THEN\n s := timestrs[1]::timestamptz;\n e := 'infinity'::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[1] ILIKE 'P%' AND timestrs[2] NOT ILIKE 'P%' THEN\n e := timestrs[2]::timestamptz;\n s := e - upper(timestrs[1])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] ILIKE 'P%' AND timestrs[1] NOT ILIKE 'P%' THEN\n s := timestrs[1]::timestamptz;\n e := s + upper(timestrs[2])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n s := timestrs[1]::timestamptz;\n e := timestrs[2]::timestamptz;\n\n RETURN tstzrange(s,e,'[)');\n\n RETURN NULL;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT PARALLEL SAFE SET TIME ZONE 'UTC';\n\nCREATE OR REPLACE FUNCTION parse_dtrange(\n _indate text,\n relative_base timestamptz DEFAULT CURRENT_TIMESTAMP\n) RETURNS tstzrange AS $$\n SELECT parse_dtrange(to_jsonb(_indate), relative_base);\n$$ LANGUAGE SQL STABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION temporal_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\n ll text := 'datetime';\n lh text := 'end_datetime';\n rrange tstzrange;\n rl text;\n rh text;\n outq text;\nBEGIN\n rrange := parse_dtrange(args->1);\n RAISE NOTICE 'Constructing temporal query OP: %, ARGS: %, RRANGE: %', op, args, rrange;\n op := lower(op);\n rl := format('%L::timestamptz', lower(rrange));\n rh := format('%L::timestamptz', upper(rrange));\n outq := CASE op\n WHEN 't_before' THEN 'lh < rl'\n WHEN 't_after' THEN 'll > rh'\n WHEN 't_meets' THEN 'lh = rl'\n WHEN 't_metby' THEN 'll = rh'\n WHEN 't_overlaps' THEN 'll < rl AND rl < lh < rh'\n WHEN 't_overlappedby' THEN 'rl < ll < rh AND lh > rh'\n WHEN 't_starts' THEN 'll = rl AND lh < rh'\n WHEN 't_startedby' THEN 'll = rl AND lh > rh'\n WHEN 't_during' THEN 'll > rl AND lh < rh'\n WHEN 't_contains' THEN 'll < rl AND lh > rh'\n WHEN 't_finishes' THEN 'll > rl AND lh = rh'\n WHEN 't_finishedby' THEN 'll < rl AND lh = rh'\n WHEN 't_equals' THEN 'll = rl AND lh = rh'\n WHEN 't_disjoint' THEN 'NOT (ll <= rh AND lh >= rl)'\n WHEN 't_intersects' THEN 'll <= rh AND lh >= rl'\n WHEN 'anyinteracts' THEN 'll <= rh AND lh >= rl'\n END;\n outq := regexp_replace(outq, '\\mll\\M', ll);\n outq := regexp_replace(outq, '\\mlh\\M', lh);\n outq := regexp_replace(outq, '\\mrl\\M', rl);\n outq := regexp_replace(outq, '\\mrh\\M', rh);\n outq := format('(%s)', outq);\n RETURN outq;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\n\n\nCREATE OR REPLACE FUNCTION spatial_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\n geom text;\n j jsonb := args->1;\nBEGIN\n op := lower(op);\n RAISE NOTICE 'Constructing spatial query OP: %, ARGS: %', op, args;\n IF op NOT IN ('s_equals','s_disjoint','s_touches','s_within','s_overlaps','s_crosses','s_intersects','intersects','s_contains') THEN\n RAISE EXCEPTION 'Spatial Operator % Not Supported', op;\n END IF;\n op := regexp_replace(op, '^s_', 'st_');\n IF op = 'intersects' THEN\n op := 'st_intersects';\n END IF;\n -- Convert geometry to WKB string\n IF j ? 'type' AND j ? 'coordinates' THEN\n geom := st_geomfromgeojson(j)::text;\n ELSIF jsonb_typeof(j) = 'array' THEN\n geom := bbox_geom(j)::text;\n END IF;\n\n RETURN format('%s(geometry, %L::geometry)', op, geom);\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION query_to_cql2(q jsonb) RETURNS jsonb AS $$\n-- Translates anything passed in through the deprecated \"query\" into equivalent CQL2\nWITH t AS (\n SELECT key as property, value as ops\n FROM jsonb_each(q)\n), t2 AS (\n SELECT property, (jsonb_each(ops)).*\n FROM t WHERE jsonb_typeof(ops) = 'object'\n UNION ALL\n SELECT property, 'eq', ops\n FROM t WHERE jsonb_typeof(ops) != 'object'\n)\nSELECT\n jsonb_strip_nulls(jsonb_build_object(\n 'op', 'and',\n 'args', jsonb_agg(\n jsonb_build_object(\n 'op', key,\n 'args', jsonb_build_array(\n jsonb_build_object('property',property),\n value\n )\n )\n )\n )\n) as qcql FROM t2\n;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\nCREATE OR REPLACE FUNCTION cql1_to_cql2(j jsonb) RETURNS jsonb AS $$\nDECLARE\n args jsonb;\n ret jsonb;\nBEGIN\n RAISE NOTICE 'CQL1_TO_CQL2: %', j;\n IF j ? 'filter' THEN\n RETURN cql1_to_cql2(j->'filter');\n END IF;\n IF j ? 'property' THEN\n RETURN j;\n END IF;\n IF jsonb_typeof(j) = 'array' THEN\n SELECT jsonb_agg(cql1_to_cql2(el)) INTO args FROM jsonb_array_elements(j) el;\n RETURN args;\n END IF;\n IF jsonb_typeof(j) = 'number' THEN\n RETURN j;\n END IF;\n IF jsonb_typeof(j) = 'string' THEN\n RETURN j;\n END IF;\n\n IF jsonb_typeof(j) = 'object' THEN\n SELECT jsonb_build_object(\n 'op', key,\n 'args', cql1_to_cql2(value)\n ) INTO ret\n FROM jsonb_each(j)\n WHERE j IS NOT NULL;\n RETURN ret;\n END IF;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE STRICT;\n\nCREATE TABLE cql2_ops (\n op text PRIMARY KEY,\n template text,\n types text[]\n);\nINSERT INTO cql2_ops (op, template, types) VALUES\n ('eq', '%s = %s', NULL),\n ('neq', '%s != %s', NULL),\n ('ne', '%s != %s', NULL),\n ('!=', '%s != %s', NULL),\n ('<>', '%s != %s', NULL),\n ('lt', '%s < %s', NULL),\n ('lte', '%s <= %s', NULL),\n ('gt', '%s > %s', NULL),\n ('gte', '%s >= %s', NULL),\n ('le', '%s <= %s', NULL),\n ('ge', '%s >= %s', NULL),\n ('=', '%s = %s', NULL),\n ('<', '%s < %s', NULL),\n ('<=', '%s <= %s', NULL),\n ('>', '%s > %s', NULL),\n ('>=', '%s >= %s', NULL),\n ('like', '%s LIKE %s', NULL),\n ('ilike', '%s ILIKE %s', NULL),\n ('+', '%s + %s', NULL),\n ('-', '%s - %s', NULL),\n ('*', '%s * %s', NULL),\n ('/', '%s / %s', NULL),\n ('not', 'NOT (%s)', NULL),\n ('between', '%s BETWEEN %s AND %s', NULL),\n ('isnull', '%s IS NULL', NULL),\n ('upper', 'upper(%s)', NULL),\n ('lower', 'lower(%s)', NULL),\n ('casei', 'upper(%s)', NULL),\n ('accenti', 'unaccent(%s)', NULL)\nON CONFLICT (op) DO UPDATE\n SET\n template = EXCLUDED.template\n;\n\n\nCREATE OR REPLACE FUNCTION cql2_query(j jsonb, wrapper text DEFAULT NULL) RETURNS text AS $$\n#variable_conflict use_variable\nDECLARE\n args jsonb := j->'args';\n arg jsonb;\n op text := lower(j->>'op');\n cql2op RECORD;\n literal text;\n _wrapper text;\n leftarg text;\n rightarg text;\n prop text;\n extra_props bool := pgstac.additional_properties();\nBEGIN\n IF j IS NULL OR (op IS NOT NULL AND args IS NULL) THEN\n RETURN NULL;\n END IF;\n RAISE NOTICE 'CQL2_QUERY: %', j;\n\n -- check if all properties are represented in the queryables\n IF NOT extra_props THEN\n FOR prop IN\n SELECT DISTINCT p->>0\n FROM jsonb_path_query(j, 'strict $.**.property') p\n WHERE p->>0 NOT IN ('id', 'datetime', 'geometry', 'end_datetime', 'collection')\n LOOP\n IF (queryable(prop)).nulled_wrapper IS NULL THEN\n RAISE EXCEPTION 'Term % is not found in queryables.', prop;\n END IF;\n END LOOP;\n END IF;\n\n IF j ? 'filter' THEN\n RETURN cql2_query(j->'filter');\n END IF;\n\n IF j ? 'upper' THEN\n RETURN cql2_query(jsonb_build_object('op', 'upper', 'args', j->'upper'));\n END IF;\n\n IF j ? 'lower' THEN\n RETURN cql2_query(jsonb_build_object('op', 'lower', 'args', j->'lower'));\n END IF;\n\n -- Temporal Query\n IF op ilike 't_%' or op = 'anyinteracts' THEN\n RETURN temporal_op_query(op, args);\n END IF;\n\n -- If property is a timestamp convert it to text to use with\n -- general operators\n IF j ? 'timestamp' THEN\n RETURN format('%L::timestamptz', to_tstz(j->'timestamp'));\n END IF;\n IF j ? 'interval' THEN\n RAISE EXCEPTION 'Please use temporal operators when using intervals.';\n RETURN NONE;\n END IF;\n\n -- Spatial Query\n IF op ilike 's_%' or op = 'intersects' THEN\n RETURN spatial_op_query(op, args);\n END IF;\n\n IF op IN ('a_equals','a_contains','a_contained_by','a_overlaps') THEN\n IF args->0 ? 'property' THEN\n leftarg := format('to_text_array(%s)', (queryable(args->0->>'property')).path);\n END IF;\n IF args->1 ? 'property' THEN\n rightarg := format('to_text_array(%s)', (queryable(args->1->>'property')).path);\n END IF;\n RETURN FORMAT(\n '%s %s %s',\n COALESCE(leftarg, quote_literal(to_text_array(args->0))),\n CASE op\n WHEN 'a_equals' THEN '='\n WHEN 'a_contains' THEN '@>'\n WHEN 'a_contained_by' THEN '<@'\n WHEN 'a_overlaps' THEN '&&'\n END,\n COALESCE(rightarg, quote_literal(to_text_array(args->1)))\n );\n END IF;\n\n IF op = 'in' THEN\n RAISE NOTICE 'IN : % % %', args, jsonb_build_array(args->0), args->1;\n args := jsonb_build_array(args->0) || (args->1);\n RAISE NOTICE 'IN2 : %', args;\n END IF;\n\n\n\n IF op = 'between' THEN\n args = jsonb_build_array(\n args->0,\n args->1,\n args->2\n );\n END IF;\n\n -- Make sure that args is an array and run cql2_query on\n -- each element of the array\n RAISE NOTICE 'ARGS PRE: %', args;\n IF j ? 'args' THEN\n IF jsonb_typeof(args) != 'array' THEN\n args := jsonb_build_array(args);\n END IF;\n\n IF jsonb_path_exists(args, '$[*] ? (@.property == \"id\" || @.property == \"datetime\" || @.property == \"end_datetime\" || @.property == \"collection\")') THEN\n wrapper := NULL;\n ELSE\n -- if any of the arguments are a property, try to get the property_wrapper\n FOR arg IN SELECT jsonb_path_query(args, '$[*] ? (@.property != null)') LOOP\n RAISE NOTICE 'Arg: %', arg;\n wrapper := (queryable(arg->>'property')).nulled_wrapper;\n RAISE NOTICE 'Property: %, Wrapper: %', arg, wrapper;\n IF wrapper IS NOT NULL THEN\n EXIT;\n END IF;\n END LOOP;\n\n -- if the property was not in queryables, see if any args were numbers\n IF\n wrapper IS NULL\n AND jsonb_path_exists(args, '$[*] ? (@.type()==\"number\")')\n THEN\n wrapper := 'to_float';\n END IF;\n wrapper := coalesce(wrapper, 'to_text');\n END IF;\n\n SELECT jsonb_agg(cql2_query(a, wrapper))\n INTO args\n FROM jsonb_array_elements(args) a;\n END IF;\n RAISE NOTICE 'ARGS: %', args;\n\n IF op IN ('and', 'or') THEN\n RETURN\n format(\n '(%s)',\n array_to_string(to_text_array(args), format(' %s ', upper(op)))\n );\n END IF;\n\n IF op = 'in' THEN\n RAISE NOTICE 'IN -- % %', args->0, to_text(args->0);\n RETURN format(\n '%s IN (%s)',\n to_text(args->0),\n array_to_string((to_text_array(args))[2:], ',')\n );\n END IF;\n\n -- Look up template from cql2_ops\n IF j ? 'op' THEN\n SELECT * INTO cql2op FROM cql2_ops WHERE cql2_ops.op ilike op;\n IF FOUND THEN\n -- If specific index set in queryables for a property cast other arguments to that type\n\n RETURN format(\n cql2op.template,\n VARIADIC (to_text_array(args))\n );\n ELSE\n RAISE EXCEPTION 'Operator % Not Supported.', op;\n END IF;\n END IF;\n\n\n IF wrapper IS NOT NULL THEN\n RAISE NOTICE 'Wrapping % with %', j, wrapper;\n IF j ? 'property' THEN\n RETURN format('%I(%s)', wrapper, (queryable(j->>'property')).path);\n ELSE\n RETURN format('%I(%L)', wrapper, j);\n END IF;\n ELSIF j ? 'property' THEN\n RETURN quote_ident(j->>'property');\n END IF;\n\n RETURN quote_literal(to_text(j));\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION paging_dtrange(\n j jsonb\n) RETURNS tstzrange AS $$\nDECLARE\n op text;\n filter jsonb := j->'filter';\n dtrange tstzrange := tstzrange('-infinity'::timestamptz,'infinity'::timestamptz);\n sdate timestamptz := '-infinity'::timestamptz;\n edate timestamptz := 'infinity'::timestamptz;\n jpitem jsonb;\nBEGIN\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n END IF;\n IF NOT (filter @? '$.**.op ? (@ == \"or\" || @ == \"not\")') THEN\n FOR jpitem IN SELECT j FROM jsonb_path_query(filter,'strict $.** ? (@.args[*].property == \"datetime\")'::jsonpath) j LOOP\n op := lower(jpitem->>'op');\n dtrange := parse_dtrange(jpitem->'args'->1);\n IF op IN ('<=', 'lt', 'lte', '<', 'le', 't_before') THEN\n sdate := greatest(sdate,'-infinity');\n edate := least(edate, upper(dtrange));\n ELSIF op IN ('>=', '>', 'gt', 'gte', 'ge', 't_after') THEN\n edate := least(edate, 'infinity');\n sdate := greatest(sdate, lower(dtrange));\n ELSIF op IN ('=', 'eq') THEN\n edate := least(edate, upper(dtrange));\n sdate := greatest(sdate, lower(dtrange));\n END IF;\n RAISE NOTICE '2 OP: %, ARGS: %, DTRANGE: %, SDATE: %, EDATE: %', op, jpitem->'args'->1, dtrange, sdate, edate;\n END LOOP;\n END IF;\n IF sdate > edate THEN\n RETURN 'empty'::tstzrange;\n END IF;\n RETURN tstzrange(sdate,edate, '[]');\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT SET TIME ZONE 'UTC';\n\nCREATE OR REPLACE FUNCTION paging_collections(\n IN j jsonb\n) RETURNS text[] AS $$\nDECLARE\n filter jsonb := j->'filter';\n jpitem jsonb;\n op text;\n args jsonb;\n arg jsonb;\n collections text[];\nBEGIN\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n END IF;\n IF NOT (filter @? '$.**.op ? (@ == \"or\" || @ == \"not\")') THEN\n FOR jpitem IN SELECT j FROM jsonb_path_query(filter,'strict $.** ? (@.args[*].property == \"collection\")'::jsonpath) j LOOP\n RAISE NOTICE 'JPITEM: %', jpitem;\n op := jpitem->>'op';\n args := jpitem->'args';\n IF op IN ('=', 'eq', 'in') THEN\n FOR arg IN SELECT a FROM jsonb_array_elements(args) a LOOP\n IF jsonb_typeof(arg) IN ('string', 'array') THEN\n RAISE NOTICE 'arg: %, collections: %', arg, collections;\n IF collections IS NULL OR collections = '{}'::text[] THEN\n collections := to_text_array(arg);\n ELSE\n collections := array_intersection(collections, to_text_array(arg));\n END IF;\n END IF;\n END LOOP;\n END IF;\n END LOOP;\n END IF;\n IF collections = '{}'::text[] THEN\n RETURN NULL;\n END IF;\n RETURN collections;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\nCREATE TABLE items (\n id text NOT NULL,\n geometry geometry NOT NULL,\n collection text NOT NULL,\n datetime timestamptz NOT NULL,\n end_datetime timestamptz NOT NULL,\n content JSONB NOT NULL,\n private jsonb\n)\nPARTITION BY LIST (collection)\n;\n\nCREATE INDEX \"datetime_idx\" ON items USING BTREE (datetime DESC, end_datetime ASC);\nCREATE INDEX \"geometry_idx\" ON items USING GIST (geometry);\n\nCREATE STATISTICS datetime_stats (dependencies) on datetime, end_datetime from items;\n\nALTER TABLE items ADD CONSTRAINT items_collections_fk FOREIGN KEY (collection) REFERENCES collections(id) ON DELETE CASCADE DEFERRABLE;\n\nCREATE OR REPLACE FUNCTION partition_after_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p text;\n t timestamptz := clock_timestamp();\nBEGIN\n RAISE NOTICE 'Updating partition stats %', t;\n FOR p IN SELECT DISTINCT partition\n FROM newdata n JOIN partition_sys_meta p\n ON (n.collection=p.collection AND n.datetime <@ p.partition_dtrange)\n LOOP\n PERFORM run_or_queue(format('SELECT update_partition_stats(%L, %L);', p, true));\n END LOOP;\n IF TG_OP IN ('DELETE','UPDATE') THEN\n DELETE FROM format_item_cache c USING newdata n WHERE c.collection = n.collection AND c.id = n.id;\n END IF;\n RAISE NOTICE 't: % %', t, clock_timestamp() - t;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE TRIGGER items_after_insert_trigger\nAFTER INSERT ON items\nREFERENCING NEW TABLE AS newdata\nFOR EACH STATEMENT\nEXECUTE FUNCTION partition_after_triggerfunc();\n\nCREATE TRIGGER items_after_update_trigger\nAFTER DELETE ON items\nREFERENCING OLD TABLE AS newdata\nFOR EACH STATEMENT\nEXECUTE FUNCTION partition_after_triggerfunc();\n\nCREATE TRIGGER items_after_delete_trigger\nAFTER UPDATE ON items\nREFERENCING NEW TABLE AS newdata\nFOR EACH STATEMENT\nEXECUTE FUNCTION partition_after_triggerfunc();\n\n\nCREATE OR REPLACE FUNCTION content_slim(_item jsonb) RETURNS jsonb AS $$\n SELECT strip_jsonb(_item - '{id,geometry,collection,type}'::text[], collection_base_item(_item->>'collection')) - '{id,geometry,collection,type}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_dehydrate(content jsonb) RETURNS items AS $$\n SELECT\n content->>'id' as id,\n stac_geom(content) as geometry,\n content->>'collection' as collection,\n stac_datetime(content) as datetime,\n stac_end_datetime(content) as end_datetime,\n content_slim(content) as content,\n null::jsonb as private\n ;\n$$ LANGUAGE SQL STABLE;\n\nCREATE OR REPLACE FUNCTION include_field(f text, fields jsonb DEFAULT '{}'::jsonb) RETURNS boolean AS $$\nDECLARE\n includes jsonb := fields->'include';\n excludes jsonb := fields->'exclude';\nBEGIN\n IF f IS NULL THEN\n RETURN NULL;\n END IF;\n\n\n IF\n jsonb_typeof(excludes) = 'array'\n AND jsonb_array_length(excludes)>0\n AND excludes ? f\n THEN\n RETURN FALSE;\n END IF;\n\n IF\n (\n jsonb_typeof(includes) = 'array'\n AND jsonb_array_length(includes) > 0\n AND includes ? f\n ) OR\n (\n includes IS NULL\n OR jsonb_typeof(includes) = 'null'\n OR jsonb_array_length(includes) = 0\n )\n THEN\n RETURN TRUE;\n END IF;\n\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nDROP FUNCTION IF EXISTS content_hydrate(jsonb, jsonb, jsonb);\nCREATE OR REPLACE FUNCTION content_hydrate(\n _item jsonb,\n _base_item jsonb,\n fields jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\n SELECT merge_jsonb(\n jsonb_fields(_item, fields),\n jsonb_fields(_base_item, fields)\n );\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION content_hydrate(_item items, _collection collections, fields jsonb DEFAULT '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\n content jsonb;\n base_item jsonb := _collection.base_item;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry, 20)::jsonb;\n END IF;\n output := content_hydrate(\n jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'collection', _item.collection,\n 'type', 'Feature'\n ) || _item.content,\n _collection.base_item,\n fields\n );\n\n RETURN output;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_nonhydrated(\n _item items,\n fields jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry, 20)::jsonb;\n END IF;\n output := jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'collection', _item.collection,\n 'type', 'Feature'\n ) || _item.content;\n RETURN output;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_hydrate(_item items, fields jsonb DEFAULT '{}'::jsonb) RETURNS jsonb AS $$\n SELECT content_hydrate(\n _item,\n (SELECT c FROM collections c WHERE id=_item.collection LIMIT 1),\n fields\n );\n$$ LANGUAGE SQL STABLE;\n\n\nCREATE UNLOGGED TABLE items_staging (\n content JSONB NOT NULL\n);\nCREATE UNLOGGED TABLE items_staging_ignore (\n content JSONB NOT NULL\n);\nCREATE UNLOGGED TABLE items_staging_upsert (\n content JSONB NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION items_staging_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p record;\n _partitions text[];\n part text;\n ts timestamptz := clock_timestamp();\n nrows int;\nBEGIN\n RAISE NOTICE 'Creating Partitions. %', clock_timestamp() - ts;\n\n FOR part IN WITH t AS (\n SELECT\n n.content->>'collection' as collection,\n stac_daterange(n.content->'properties') as dtr,\n partition_trunc\n FROM newdata n JOIN collections ON (n.content->>'collection'=collections.id)\n ), p AS (\n SELECT\n collection,\n COALESCE(date_trunc(partition_trunc::text, lower(dtr)),'-infinity') as d,\n tstzrange(min(lower(dtr)),max(lower(dtr)),'[]') as dtrange,\n tstzrange(min(upper(dtr)),max(upper(dtr)),'[]') as edtrange\n FROM t\n GROUP BY 1,2\n ) SELECT check_partition(collection, dtrange, edtrange) FROM p LOOP\n RAISE NOTICE 'Partition %', part;\n END LOOP;\n\n RAISE NOTICE 'Creating temp table with data to be added. %', clock_timestamp() - ts;\n DROP TABLE IF EXISTS tmpdata;\n CREATE TEMP TABLE tmpdata ON COMMIT DROP AS\n SELECT\n (content_dehydrate(content)).*\n FROM newdata;\n GET DIAGNOSTICS nrows = ROW_COUNT;\n RAISE NOTICE 'Added % rows to tmpdata. %', nrows, clock_timestamp() - ts;\n\n RAISE NOTICE 'Doing the insert. %', clock_timestamp() - ts;\n IF TG_TABLE_NAME = 'items_staging' THEN\n INSERT INTO items\n SELECT * FROM tmpdata;\n GET DIAGNOSTICS nrows = ROW_COUNT;\n RAISE NOTICE 'Inserted % rows to items. %', nrows, clock_timestamp() - ts;\n ELSIF TG_TABLE_NAME = 'items_staging_ignore' THEN\n INSERT INTO items\n SELECT * FROM tmpdata\n ON CONFLICT DO NOTHING;\n GET DIAGNOSTICS nrows = ROW_COUNT;\n RAISE NOTICE 'Inserted % rows to items. %', nrows, clock_timestamp() - ts;\n ELSIF TG_TABLE_NAME = 'items_staging_upsert' THEN\n DELETE FROM items i USING tmpdata s\n WHERE\n i.id = s.id\n AND i.collection = s.collection\n AND i IS DISTINCT FROM s\n ;\n GET DIAGNOSTICS nrows = ROW_COUNT;\n RAISE NOTICE 'Deleted % rows from items. %', nrows, clock_timestamp() - ts;\n INSERT INTO items AS t\n SELECT * FROM tmpdata\n ON CONFLICT DO NOTHING;\n GET DIAGNOSTICS nrows = ROW_COUNT;\n RAISE NOTICE 'Inserted % rows to items. %', nrows, clock_timestamp() - ts;\n END IF;\n\n RAISE NOTICE 'Deleting data from staging table. %', clock_timestamp() - ts;\n DELETE FROM items_staging;\n RAISE NOTICE 'Done. %', clock_timestamp() - ts;\n\n RETURN NULL;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER items_staging_insert_trigger AFTER INSERT ON items_staging REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\nCREATE TRIGGER items_staging_insert_ignore_trigger AFTER INSERT ON items_staging_ignore REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\nCREATE TRIGGER items_staging_insert_upsert_trigger AFTER INSERT ON items_staging_upsert REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\n\nCREATE OR REPLACE FUNCTION item_by_id(_id text, _collection text DEFAULT NULL) RETURNS items AS\n$$\nDECLARE\n i items%ROWTYPE;\nBEGIN\n SELECT * INTO i FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection) LIMIT 1;\n RETURN i;\nEND;\n$$ LANGUAGE PLPGSQL STABLE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION get_item(_id text, _collection text DEFAULT NULL) RETURNS jsonb AS $$\n SELECT content_hydrate(items) FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection);\n$$ LANGUAGE SQL STABLE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION delete_item(_id text, _collection text DEFAULT NULL) RETURNS VOID AS $$\nDECLARE\nout items%ROWTYPE;\nBEGIN\n DELETE FROM items WHERE id = _id AND (_collection IS NULL OR collection=_collection) RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n--/*\nCREATE OR REPLACE FUNCTION create_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION update_item(content jsonb) RETURNS VOID AS $$\nDECLARE\n old items %ROWTYPE;\n out items%ROWTYPE;\nBEGIN\n PERFORM delete_item(content->>'id', content->>'collection');\n PERFORM create_item(content);\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION create_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION collection_bbox(id text) RETURNS jsonb AS $$\n SELECT (replace(replace(replace(st_extent(geometry)::text,'BOX(','[['),')',']]'),' ',','))::jsonb\n FROM items WHERE collection=$1;\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION collection_temporal_extent(id text) RETURNS jsonb AS $$\n SELECT to_jsonb(array[array[min(datetime), max(datetime)]])\n FROM items WHERE collection=$1;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection_extents() RETURNS VOID AS $$\nUPDATE collections\n SET content = jsonb_set_lax(\n content,\n '{extent}'::text[],\n collection_extent(id, FALSE),\n true,\n 'use_json_null'\n )\n;\n$$ LANGUAGE SQL;\nCREATE TABLE partition_stats (\n partition text PRIMARY KEY,\n dtrange tstzrange,\n edtrange tstzrange,\n spatial geometry,\n last_updated timestamptz,\n keys text[]\n) WITH (FILLFACTOR=90);\n\nCREATE INDEX partitions_range_idx ON partition_stats USING GIST(dtrange);\n\n\nCREATE OR REPLACE FUNCTION constraint_tstzrange(expr text) RETURNS tstzrange AS $$\n WITH t AS (\n SELECT regexp_matches(\n expr,\n E'\\\\(''\\([0-9 :+-]*\\)''\\\\).*\\\\(''\\([0-9 :+-]*\\)''\\\\)'\n ) AS m\n ) SELECT tstzrange(m[1]::timestamptz, m[2]::timestamptz) FROM t\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE STRICT;\n\nCREATE OR REPLACE FUNCTION get_tstz_constraint(reloid oid, colname text) RETURNS tstzrange AS $$\nDECLARE\n expr text := NULL;\n m text[];\n ts_lower timestamptz := NULL;\n ts_upper timestamptz := NULL;\n lower_inclusive text := '[';\n upper_inclusive text := ']';\n ts timestamptz;\nBEGIN\n SELECT INTO expr\n string_agg(def, ' AND ')\n FROM pg_constraint JOIN LATERAL pg_get_constraintdef(oid) AS def ON TRUE\n WHERE\n conrelid = reloid\n AND contype = 'c'\n AND def LIKE '%' || colname || '%'\n ;\n\n IF expr IS NULL THEN\n RETURN NULL;\n END IF;\n\n RAISE DEBUG 'Constraint expression for % on %: %', colname, reloid::regclass, expr;\n -- collect all constraints for the specified column\n FOR m IN SELECT regexp_matches(expr, '[ (]' || colname || $expr$\\s*([<>=]{1,2})\\s*'([0-9 :+\\-]+)'$expr$, 'g') LOOP\n ts := m[2]::timestamptz;\n IF m[1] IN ('>', '>=')\n THEN\n IF ts_lower IS NULL OR ts > ts_lower OR (ts = ts_lower AND m[1] = '>') THEN\n ts_lower := ts;\n lower_inclusive := CASE WHEN m[1] = '>' THEN '(' ELSE '[' END;\n END IF;\n ELSIF m[1] IN ('<', '<=')\n THEN\n IF ts_upper IS NULL OR ts < ts_upper OR (ts = ts_upper AND m[1] = '<') THEN\n ts_upper := ts;\n upper_inclusive := CASE WHEN m[1] = '<' THEN ')' ELSE ']' END;\n END IF;\n END IF;\n END LOOP;\n RAISE DEBUG 'Constraint % for %: % %', colname, reloid::regclass, ts_lower, ts_upper;\n RETURN tstzrange(ts_lower, ts_upper, lower_inclusive || upper_inclusive);\nEND;\n$$ LANGUAGE plpgsql STRICT STABLE;\n\nCREATE OR REPLACE FUNCTION get_partition_name(relid regclass) RETURNS text AS $$\n SELECT (parse_ident(relid::text))[cardinality(parse_ident(relid::text))];\n$$ LANGUAGE SQL STABLE STRICT;\n\nCREATE OR REPLACE VIEW partition_sys_meta AS\nSELECT\n partition,\n replace(\n replace(\n CASE WHEN level = 1 THEN partition_expr ELSE parent_partition_expr END,\n 'FOR VALUES IN (''',\n ''\n ),\n ''')',\n ''\n ) AS collection,\n level,\n c.reltuples,\n c.relhastriggers,\n partition_dtrange,\n COALESCE(\n get_tstz_constraint(c.oid, 'datetime'),\n partition_dtrange,\n inf_range\n ) as constraint_dtrange,\n COALESCE(\n get_tstz_constraint(c.oid, 'end_datetime'),\n inf_range\n ) as constraint_edtrange\nFROM\n pg_partition_tree('items')\n JOIN pg_class c ON (relid::regclass = c.oid)\n JOIN pg_class parent ON (parentrelid::regclass = parent.oid AND isleaf)\n LEFT JOIN pg_constraint edt ON (conrelid=c.oid AND contype='c')\n JOIN LATERAL get_partition_name(relid) AS partition ON TRUE\n JOIN LATERAL pg_get_expr(c.relpartbound, c.oid) as partition_expr ON TRUE\n JOIN LATERAL pg_get_expr(parent.relpartbound, parent.oid) as parent_partition_expr ON TRUE\n JOIN LATERAL tstzrange('-infinity', 'infinity','[]') as inf_range ON TRUE\n JOIN LATERAL COALESCE(constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), inf_range) as partition_dtrange ON TRUE\n JOIN LATERAL get_tstz_constraint(c.oid, 'datetime') as datetime_constraint ON TRUE\n JOIN LATERAL get_tstz_constraint(c.oid, 'end_datetime') as end_datetime_constraint ON TRUE\nWHERE isleaf\n;\n\nCREATE OR REPLACE VIEW partitions_view AS\nSELECT\n (parse_ident(relid::text))[cardinality(parse_ident(relid::text))] as partition,\n replace(\n replace(\n CASE WHEN level = 1 THEN partition_expr ELSE parent_partition_expr END,\n 'FOR VALUES IN (''',\n ''\n ),\n ''')',\n ''\n ) AS collection,\n level,\n c.reltuples,\n c.relhastriggers,\n partition_dtrange,\n COALESCE(\n get_tstz_constraint(c.oid, 'datetime'),\n partition_dtrange,\n inf_range\n ) as constraint_dtrange,\n COALESCE(\n get_tstz_constraint(c.oid, 'end_datetime'),\n inf_range\n ) as constraint_edtrange,\n dtrange,\n edtrange,\n spatial,\n last_updated\nFROM\n pg_partition_tree('items')\n JOIN pg_class c ON (relid::regclass = c.oid)\n JOIN pg_class parent ON (parentrelid::regclass = parent.oid AND isleaf)\n LEFT JOIN pg_constraint edt ON (conrelid=c.oid AND contype='c')\n JOIN LATERAL get_partition_name(relid) AS partition ON TRUE\n JOIN LATERAL pg_get_expr(c.relpartbound, c.oid) as partition_expr ON TRUE\n JOIN LATERAL pg_get_expr(parent.relpartbound, parent.oid) as parent_partition_expr ON TRUE\n JOIN LATERAL tstzrange('-infinity', 'infinity','[]') as inf_range ON TRUE\n JOIN LATERAL COALESCE(constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), inf_range) as partition_dtrange ON TRUE\n JOIN LATERAL get_tstz_constraint(c.oid, 'datetime') as datetime_constraint ON TRUE\n JOIN LATERAL get_tstz_constraint(c.oid, 'end_datetime') as end_datetime_constraint ON TRUE\n LEFT JOIN pgstac.partition_stats USING (partition)\nWHERE isleaf\n;\n\nCREATE MATERIALIZED VIEW partitions AS\nSELECT * FROM partitions_view;\nCREATE UNIQUE INDEX ON partitions (partition);\n\nCREATE MATERIALIZED VIEW partition_steps AS\nSELECT\n partition as name,\n date_trunc('month',lower(partition_dtrange)) as sdate,\n date_trunc('month', upper(partition_dtrange)) + '1 month'::interval as edate\n FROM partitions_view WHERE partition_dtrange IS NOT NULL AND partition_dtrange != 'empty'::tstzrange\n ORDER BY dtrange ASC\n;\n\n\nCREATE OR REPLACE FUNCTION update_partition_stats_q(_partition text, istrigger boolean default false) RETURNS VOID AS $$\nDECLARE\nBEGIN\n PERFORM run_or_queue(\n format('SELECT update_partition_stats(%L, %L);', _partition, istrigger)\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION update_partition_stats(_partition text, istrigger boolean default false) RETURNS VOID AS $$\nDECLARE\n dtrange tstzrange;\n edtrange tstzrange;\n cdtrange tstzrange;\n cedtrange tstzrange;\n extent geometry;\n collection text;\nBEGIN\n RAISE NOTICE 'Updating stats for %.', _partition;\n EXECUTE format(\n $q$\n SELECT\n tstzrange(min(datetime), max(datetime),'[]'),\n tstzrange(min(end_datetime), max(end_datetime), '[]')\n FROM %I\n $q$,\n _partition\n ) INTO dtrange, edtrange;\n extent := st_estimatedextent('pgstac', _partition, 'geometry');\n RAISE DEBUG 'Estimated Extent: %', extent;\n INSERT INTO partition_stats (partition, dtrange, edtrange, spatial, last_updated)\n SELECT _partition, dtrange, edtrange, extent, now()\n ON CONFLICT (partition) DO\n UPDATE SET\n dtrange=EXCLUDED.dtrange,\n edtrange=EXCLUDED.edtrange,\n spatial=EXCLUDED.spatial,\n last_updated=EXCLUDED.last_updated\n ;\n\n SELECT\n constraint_dtrange, constraint_edtrange, pv.collection\n INTO cdtrange, cedtrange, collection\n FROM partitions_view pv WHERE partition = _partition;\n REFRESH MATERIALIZED VIEW partitions;\n REFRESH MATERIALIZED VIEW partition_steps;\n\n\n RAISE NOTICE 'Checking if we need to modify constraints...';\n RAISE NOTICE 'cdtrange: % dtrange: % cedtrange: % edtrange: %',cdtrange, dtrange, cedtrange, edtrange;\n IF\n (cdtrange IS DISTINCT FROM dtrange OR edtrange IS DISTINCT FROM cedtrange)\n AND NOT istrigger\n THEN\n RAISE NOTICE 'Modifying Constraints';\n RAISE NOTICE 'Existing % %', cdtrange, cedtrange;\n RAISE NOTICE 'New % %', dtrange, edtrange;\n PERFORM drop_table_constraints(_partition);\n PERFORM create_table_constraints(_partition, dtrange, edtrange);\n REFRESH MATERIALIZED VIEW partitions;\n REFRESH MATERIALIZED VIEW partition_steps;\n END IF;\n RAISE NOTICE 'Checking if we need to update collection extents.';\n IF get_setting_bool('update_collection_extent') THEN\n RAISE NOTICE 'updating collection extent for %', collection;\n PERFORM run_or_queue(format($q$\n UPDATE collections\n SET content = jsonb_set_lax(\n content,\n '{extent}'::text[],\n collection_extent(%L, FALSE),\n true,\n 'use_json_null'\n ) WHERE id=%L\n ;\n $q$, collection, collection));\n ELSE\n RAISE NOTICE 'Not updating collection extent for %', collection;\n END IF;\n\nEND;\n$$ LANGUAGE PLPGSQL STRICT SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION partition_name( IN collection text, IN dt timestamptz, OUT partition_name text, OUT partition_range tstzrange) AS $$\nDECLARE\n c RECORD;\n parent_name text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n parent_name := format('_items_%s', c.key);\n\n\n IF c.partition_trunc = 'year' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYY'));\n ELSIF c.partition_trunc = 'month' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYYMM'));\n ELSE\n partition_name := parent_name;\n partition_range := tstzrange('-infinity'::timestamptz, 'infinity'::timestamptz, '[]');\n END IF;\n IF partition_range IS NULL THEN\n partition_range := tstzrange(\n date_trunc(c.partition_trunc::text, dt),\n date_trunc(c.partition_trunc::text, dt) + concat('1 ', c.partition_trunc)::interval\n );\n END IF;\n RETURN;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION drop_table_constraints(t text) RETURNS text AS $$\nDECLARE\n q text;\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM partitions_view WHERE partition=t) THEN\n RETURN NULL;\n END IF;\n FOR q IN SELECT FORMAT(\n $q$\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n $q$,\n t,\n conname\n ) FROM pg_constraint\n WHERE conrelid=t::regclass::oid AND contype='c'\n LOOP\n EXECUTE q;\n END LOOP;\n RETURN t;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE OR REPLACE FUNCTION create_table_constraints(t text, _dtrange tstzrange, _edtrange tstzrange) RETURNS text AS $$\nDECLARE\n q text;\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM partitions_view WHERE partition=t) THEN\n RETURN NULL;\n END IF;\n RAISE NOTICE 'Creating Table Constraints for % % %', t, _dtrange, _edtrange;\n IF _dtrange = 'empty' AND _edtrange = 'empty' THEN\n q :=format(\n $q$\n DO $block$\n BEGIN\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n ALTER TABLE %I\n ADD CONSTRAINT %I\n CHECK (((datetime IS NULL) AND (end_datetime IS NULL))) NOT VALID\n ;\n ALTER TABLE %I\n VALIDATE CONSTRAINT %I\n ;\n\n\n\n EXCEPTION WHEN others THEN\n RAISE WARNING '%%, Issue Altering Constraints. Please run update_partition_stats(%I)', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n $block$;\n $q$,\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t),\n t\n );\n ELSE\n q :=format(\n $q$\n DO $block$\n BEGIN\n\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n ALTER TABLE %I\n ADD CONSTRAINT %I\n CHECK (\n (datetime >= %L)\n AND (datetime <= %L)\n AND (end_datetime >= %L)\n AND (end_datetime <= %L)\n ) NOT VALID\n ;\n ALTER TABLE %I\n VALIDATE CONSTRAINT %I\n ;\n\n\n\n EXCEPTION WHEN others THEN\n RAISE WARNING '%%, Issue Altering Constraints. Please run update_partition_stats(%I)', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n $block$;\n $q$,\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t),\n lower(_dtrange),\n upper(_dtrange),\n lower(_edtrange),\n upper(_edtrange),\n t,\n format('%s_dt', t),\n t\n );\n END IF;\n PERFORM run_or_queue(q);\n RETURN t;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION check_partition(\n _collection text,\n _dtrange tstzrange,\n _edtrange tstzrange\n) RETURNS text AS $$\nDECLARE\n c RECORD;\n pm RECORD;\n _partition_name text;\n _partition_dtrange tstzrange;\n _constraint_dtrange tstzrange;\n _constraint_edtrange tstzrange;\n q text;\n deferrable_q text;\n err_context text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=_collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', _collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n\n IF c.partition_trunc IS NOT NULL THEN\n _partition_dtrange := tstzrange(\n date_trunc(c.partition_trunc, lower(_dtrange)),\n date_trunc(c.partition_trunc, lower(_dtrange)) + (concat('1 ', c.partition_trunc))::interval,\n '[)'\n );\n ELSE\n _partition_dtrange := '[-infinity, infinity]'::tstzrange;\n END IF;\n\n IF NOT _partition_dtrange @> _dtrange THEN\n RAISE EXCEPTION 'dtrange % is greater than the partition size % for collection %', _dtrange, c.partition_trunc, _collection;\n END IF;\n\n\n IF c.partition_trunc = 'year' THEN\n _partition_name := format('_items_%s_%s', c.key, to_char(lower(_partition_dtrange),'YYYY'));\n ELSIF c.partition_trunc = 'month' THEN\n _partition_name := format('_items_%s_%s', c.key, to_char(lower(_partition_dtrange),'YYYYMM'));\n ELSE\n _partition_name := format('_items_%s', c.key);\n END IF;\n\n SELECT * INTO pm FROM partition_sys_meta WHERE collection=_collection AND partition_dtrange @> _dtrange;\n IF FOUND THEN\n RAISE NOTICE '% % %', _edtrange, _dtrange, pm;\n _constraint_edtrange :=\n tstzrange(\n least(\n lower(_edtrange),\n nullif(lower(pm.constraint_edtrange), '-infinity')\n ),\n greatest(\n upper(_edtrange),\n nullif(upper(pm.constraint_edtrange), 'infinity')\n ),\n '[]'\n );\n _constraint_dtrange :=\n tstzrange(\n least(\n lower(_dtrange),\n nullif(lower(pm.constraint_dtrange), '-infinity')\n ),\n greatest(\n upper(_dtrange),\n nullif(upper(pm.constraint_dtrange), 'infinity')\n ),\n '[]'\n );\n\n IF pm.constraint_edtrange @> _edtrange AND pm.constraint_dtrange @> _dtrange THEN\n RETURN pm.partition;\n ELSE\n PERFORM drop_table_constraints(_partition_name);\n END IF;\n ELSE\n _constraint_edtrange := _edtrange;\n _constraint_dtrange := _dtrange;\n END IF;\n RAISE NOTICE 'EXISTING CONSTRAINTS % %, NEW % %', pm.constraint_dtrange, pm.constraint_edtrange, _constraint_dtrange, _constraint_edtrange;\n RAISE NOTICE 'Creating partition % %', _partition_name, _partition_dtrange;\n IF c.partition_trunc IS NULL THEN\n q := format(\n $q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n GRANT ALL ON %I to pgstac_ingest;\n $q$,\n _partition_name,\n _collection,\n concat(_partition_name,'_pk'),\n _partition_name,\n _partition_name\n );\n ELSE\n q := format(\n $q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L) PARTITION BY RANGE (datetime);\n CREATE TABLE IF NOT EXISTS %I partition OF %I FOR VALUES FROM (%L) TO (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n GRANT ALL ON %I TO pgstac_ingest;\n $q$,\n format('_items_%s', c.key),\n _collection,\n _partition_name,\n format('_items_%s', c.key),\n lower(_partition_dtrange),\n upper(_partition_dtrange),\n format('%s_pk', _partition_name),\n _partition_name,\n _partition_name\n );\n END IF;\n\n BEGIN\n EXECUTE q;\n EXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', _partition_name;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\n END;\n PERFORM maintain_partitions(_partition_name);\n PERFORM update_partition_stats_q(_partition_name, true);\n REFRESH MATERIALIZED VIEW partitions;\n REFRESH MATERIALIZED VIEW partition_steps;\n RETURN _partition_name;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION repartition(_collection text, _partition_trunc text, triggered boolean DEFAULT FALSE) RETURNS text AS $$\nDECLARE\n c RECORD;\n q text;\n from_trunc text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=_collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', _collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n IF triggered THEN\n RAISE NOTICE 'Converting % to % partitioning via Trigger', _collection, _partition_trunc;\n ELSE\n RAISE NOTICE 'Converting % from using % to % partitioning', _collection, c.partition_trunc, _partition_trunc;\n IF c.partition_trunc IS NOT DISTINCT FROM _partition_trunc THEN\n RAISE NOTICE 'Collection % already set to use partition by %', _collection, _partition_trunc;\n RETURN _collection;\n END IF;\n END IF;\n\n IF EXISTS (SELECT 1 FROM partitions_view WHERE collection=_collection LIMIT 1) THEN\n EXECUTE format(\n $q$\n CREATE TEMP TABLE changepartitionstaging ON COMMIT DROP AS SELECT * FROM %I;\n DROP TABLE IF EXISTS %I CASCADE;\n WITH p AS (\n SELECT\n collection,\n CASE\n WHEN %L IS NULL THEN '-infinity'::timestamptz\n ELSE date_trunc(%L, datetime)\n END as d,\n tstzrange(min(datetime),max(datetime),'[]') as dtrange,\n tstzrange(min(end_datetime),max(end_datetime),'[]') as edtrange\n FROM changepartitionstaging\n GROUP BY 1,2\n ) SELECT check_partition(collection, dtrange, edtrange) FROM p;\n INSERT INTO items SELECT * FROM changepartitionstaging;\n DROP TABLE changepartitionstaging;\n $q$,\n concat('_items_', c.key),\n concat('_items_', c.key),\n c.partition_trunc,\n c.partition_trunc\n );\n END IF;\n RETURN _collection;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE OR REPLACE FUNCTION collections_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n q text;\n partition_name text := format('_items_%s', NEW.key);\n partition_exists boolean := false;\n partition_empty boolean := true;\n err_context text;\n loadtemp boolean := FALSE;\nBEGIN\n RAISE NOTICE 'Collection Trigger. % %', NEW.id, NEW.key;\n IF TG_OP = 'UPDATE' AND NEW.partition_trunc IS DISTINCT FROM OLD.partition_trunc THEN\n PERFORM repartition(NEW.id, NEW.partition_trunc, TRUE);\n END IF;\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER collections_trigger AFTER\nINSERT\nOR\nUPDATE ON collections\nFOR EACH ROW EXECUTE FUNCTION collections_trigger_func();\n\nCREATE OR REPLACE FUNCTION chunker(\n IN _where text,\n OUT s timestamptz,\n OUT e timestamptz\n) RETURNS SETOF RECORD AS $$\nDECLARE\n explain jsonb;\nBEGIN\n IF _where IS NULL THEN\n _where := ' TRUE ';\n END IF;\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s;', _where)\n INTO explain;\n RAISE DEBUG 'EXPLAIN: %', explain;\n\n RETURN QUERY\n WITH t AS (\n SELECT j->>0 as p FROM\n jsonb_path_query(\n explain,\n 'strict $.**.\"Relation Name\" ? (@ != null)'\n ) j\n ),\n parts AS (\n SELECT sdate, edate FROM t JOIN partition_steps ON (t.p = name)\n ),\n times AS (\n SELECT sdate FROM parts\n UNION\n SELECT edate FROM parts\n ),\n uniq AS (\n SELECT DISTINCT sdate FROM times ORDER BY sdate\n ),\n last AS (\n SELECT sdate, lead(sdate, 1) over () as edate FROM uniq\n )\n SELECT sdate, edate FROM last WHERE edate IS NOT NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION partition_queries(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN partitions text[] DEFAULT NULL\n) RETURNS SETOF text AS $$\nDECLARE\n query text;\n sdate timestamptz;\n edate timestamptz;\nBEGIN\nIF _where IS NULL OR trim(_where) = '' THEN\n _where = ' TRUE ';\nEND IF;\nRAISE NOTICE 'Getting chunks for % %', _where, _orderby;\nIF _orderby ILIKE 'datetime d%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 DESC LOOP\n RETURN NEXT format($q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n $q$,\n sdate,\n edate,\n _where,\n _orderby\n );\n END LOOP;\nELSIF _orderby ILIKE 'datetime a%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 ASC LOOP\n RETURN NEXT format($q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n $q$,\n sdate,\n edate,\n _where,\n _orderby\n );\n END LOOP;\nELSE\n query := format($q$\n SELECT * FROM items\n WHERE %s\n ORDER BY %s\n $q$, _where, _orderby\n );\n\n RETURN NEXT query;\n RETURN;\nEND IF;\n\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION partition_query_view(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN _limit int DEFAULT 10\n) RETURNS text AS $$\n WITH p AS (\n SELECT * FROM partition_queries(_where, _orderby) p\n )\n SELECT\n CASE WHEN EXISTS (SELECT 1 FROM p) THEN\n (SELECT format($q$\n SELECT * FROM (\n %s\n ) total LIMIT %s\n $q$,\n string_agg(\n format($q$ SELECT * FROM ( %s ) AS sub $q$, p),\n '\n UNION ALL\n '\n ),\n _limit\n ))\n ELSE NULL\n END FROM p;\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION q_to_tsquery (jinput jsonb)\n RETURNS tsquery\n AS $$\nDECLARE\n input text;\n processed_text text;\n temp_text text;\n quote_array text[];\n placeholder text := '@QUOTE@';\nBEGIN\n IF jsonb_typeof(jinput) = 'string' THEN\n input := jinput->>0;\n ELSIF jsonb_typeof(jinput) = 'array' THEN\n input := array_to_string(\n array(select jsonb_array_elements_text(jinput)),\n ' OR '\n );\n ELSE\n RAISE EXCEPTION 'Input must be a string or an array of strings.';\n END IF;\n -- Extract all quoted phrases and store in array\n quote_array := regexp_matches(input, '\"[^\"]*\"', 'g');\n\n -- Replace each quoted part with a unique placeholder if there are any quoted phrases\n IF array_length(quote_array, 1) IS NOT NULL THEN\n processed_text := input;\n FOR i IN array_lower(quote_array, 1) .. array_upper(quote_array, 1) LOOP\n processed_text := replace(processed_text, quote_array[i], placeholder || i || placeholder);\n END LOOP;\n ELSE\n processed_text := input;\n END IF;\n\n -- Replace non-quoted text using regular expressions\n\n -- , -> |\n processed_text := regexp_replace(processed_text, ',(?=(?:[^\"]*\"[^\"]*\")*[^\"]*$)', ' | ', 'g');\n\n -- and -> &\n processed_text := regexp_replace(processed_text, '\\s+AND\\s+', ' & ', 'gi');\n\n -- or -> |\n processed_text := regexp_replace(processed_text, '\\s+OR\\s+', ' | ', 'gi');\n\n -- + ->\n processed_text := regexp_replace(processed_text, '^\\s*\\+([a-zA-Z0-9_]+)', '\\1', 'g'); -- +term at start\n processed_text := regexp_replace(processed_text, '\\s*\\+([a-zA-Z0-9_]+)', ' & \\1', 'g'); -- +term elsewhere\n\n -- - -> !\n processed_text := regexp_replace(processed_text, '^\\s*\\-([a-zA-Z0-9_]+)', '! \\1', 'g'); -- -term at start\n processed_text := regexp_replace(processed_text, '\\s*\\-([a-zA-Z0-9_]+)', ' & ! \\1', 'g'); -- -term elsewhere\n\n -- terms separated with spaces are assumed to represent adjacent terms. loop through these\n -- occurrences and replace them with the adjacency operator (<->)\n LOOP\n temp_text := regexp_replace(processed_text, '([a-zA-Z0-9_]+)\\s+([a-zA-Z0-9_]+)(?!\\s*[&|<>])', '\\1 <-> \\2', 'g');\n IF temp_text = processed_text THEN\n EXIT; -- No more replacements were made\n END IF;\n processed_text := temp_text;\n END LOOP;\n\n\n -- Replace placeholders back with quoted phrases if there were any\n IF array_length(quote_array, 1) IS NOT NULL THEN\n FOR i IN array_lower(quote_array, 1) .. array_upper(quote_array, 1) LOOP\n processed_text := replace(processed_text, placeholder || i || placeholder, '''' || substring(quote_array[i] from 2 for length(quote_array[i]) - 2) || '''');\n END LOOP;\n END IF;\n\n -- Print processed_text to the console for debugging purposes\n RAISE NOTICE 'processed_text: %', processed_text;\n\n RETURN to_tsquery('english', processed_text);\nEND;\n$$\nLANGUAGE plpgsql;\n\n\nCREATE OR REPLACE FUNCTION stac_search_to_where(j jsonb) RETURNS text AS $$\nDECLARE\n where_segments text[];\n _where text;\n dtrange tstzrange;\n collections text[];\n geom geometry;\n sdate timestamptz;\n edate timestamptz;\n filterlang text;\n filter jsonb := j->'filter';\n ft_query tsquery;\nBEGIN\n IF j ? 'ids' THEN\n where_segments := where_segments || format('id = ANY (%L) ', to_text_array(j->'ids'));\n END IF;\n\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n where_segments := where_segments || format('collection = ANY (%L) ', collections);\n END IF;\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n\n where_segments := where_segments || format(' datetime <= %L::timestamptz AND end_datetime >= %L::timestamptz ',\n edate,\n sdate\n );\n END IF;\n\n IF j ? 'q' THEN\n ft_query := q_to_tsquery(j->'q');\n where_segments := where_segments || format(\n $quote$\n (\n to_tsvector('english', content->'properties'->>'description') ||\n to_tsvector('english', coalesce(content->'properties'->>'title', '')) ||\n to_tsvector('english', coalesce(content->'properties'->>'keywords', ''))\n ) @@ %L\n $quote$,\n ft_query\n );\n END IF;\n\n geom := stac_geom(j);\n IF geom IS NOT NULL THEN\n where_segments := where_segments || format('st_intersects(geometry, %L)',geom);\n END IF;\n\n filterlang := COALESCE(\n j->>'filter-lang',\n get_setting('default_filter_lang', j->'conf')\n );\n IF NOT filter @? '$.**.op' THEN\n filterlang := 'cql-json';\n END IF;\n\n IF filterlang NOT IN ('cql-json','cql2-json') AND j ? 'filter' THEN\n RAISE EXCEPTION '% is not a supported filter-lang. Please use cql-json or cql2-json.', filterlang;\n END IF;\n\n IF j ? 'query' AND j ? 'filter' THEN\n RAISE EXCEPTION 'Can only use either query or filter at one time.';\n END IF;\n\n IF j ? 'query' THEN\n filter := query_to_cql2(j->'query');\n ELSIF filterlang = 'cql-json' THEN\n filter := cql1_to_cql2(filter);\n END IF;\n RAISE NOTICE 'FILTER: %', filter;\n where_segments := where_segments || cql2_query(filter);\n IF cardinality(where_segments) < 1 THEN\n RETURN ' TRUE ';\n END IF;\n\n _where := array_to_string(array_remove(where_segments, NULL), ' AND ');\n\n IF _where IS NULL OR BTRIM(_where) = '' THEN\n RETURN ' TRUE ';\n END IF;\n RETURN _where;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION parse_sort_dir(_dir text, reverse boolean default false) RETURNS text AS $$\n WITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n ) SELECT\n CASE\n WHEN NOT reverse THEN d\n WHEN d = 'ASC' THEN 'DESC'\n WHEN d = 'DESC' THEN 'ASC'\n END\n FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION sort_dir_to_op(_dir text, prev boolean default false) RETURNS text AS $$\n WITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n ) SELECT\n CASE\n WHEN d = 'ASC' AND prev THEN '<='\n WHEN d = 'DESC' AND prev THEN '>='\n WHEN d = 'ASC' THEN '>='\n WHEN d = 'DESC' THEN '<='\n END\n FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION sort_sqlorderby(\n _search jsonb DEFAULT NULL,\n reverse boolean DEFAULT FALSE\n) RETURNS text AS $$\n WITH sortby AS (\n SELECT coalesce(_search->'sortby','[{\"field\":\"datetime\", \"direction\":\"desc\"}]') as sort\n ), withid AS (\n SELECT CASE\n WHEN sort @? '$[*] ? (@.field == \"id\")' THEN sort\n ELSE sort || '[{\"field\":\"id\", \"direction\":\"desc\"}]'::jsonb\n END as sort\n FROM sortby\n ), withid_rows AS (\n SELECT jsonb_array_elements(sort) as value FROM withid\n ),sorts AS (\n SELECT\n coalesce(\n (queryable(value->>'field')).expression\n ) as key,\n parse_sort_dir(value->>'direction', reverse) as dir\n FROM withid_rows\n )\n SELECT array_to_string(\n array_agg(concat(key, ' ', dir)),\n ', '\n ) FROM sorts;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_sort_dir(sort_item jsonb) RETURNS text AS $$\n SELECT CASE WHEN sort_item->>'direction' ILIKE 'desc%' THEN 'DESC' ELSE 'ASC' END;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION get_token_val_str(\n _field text,\n _item items\n) RETURNS text AS $$\nDECLARE\n q text;\n literal text;\nBEGIN\n q := format($q$ SELECT quote_literal(%s) FROM (SELECT $1.*) as r;$q$, _field);\n EXECUTE q INTO literal USING _item;\n RETURN literal;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n\nCREATE OR REPLACE FUNCTION get_token_record(IN _token text, OUT prev BOOLEAN, OUT item items) RETURNS RECORD AS $$\nDECLARE\n _itemid text := _token;\n _collectionid text;\nBEGIN\n IF _token IS NULL THEN\n RETURN;\n END IF;\n RAISE NOTICE 'Looking for token: %', _token;\n prev := FALSE;\n IF _token ILIKE 'prev:%' THEN\n _itemid := replace(_token, 'prev:','');\n prev := TRUE;\n ELSIF _token ILIKE 'next:%' THEN\n _itemid := replace(_token, 'next:', '');\n END IF;\n SELECT id INTO _collectionid FROM collections WHERE _itemid LIKE concat(id,':%');\n IF FOUND THEN\n _itemid := replace(_itemid, concat(_collectionid,':'), '');\n SELECT * INTO item FROM items WHERE id=_itemid AND collection=_collectionid;\n ELSE\n SELECT * INTO item FROM items WHERE id=_itemid;\n END IF;\n IF item IS NULL THEN\n RAISE EXCEPTION 'Could not find item using token: % item: % collection: %', _token, _itemid, _collectionid;\n END IF;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\n\nCREATE OR REPLACE FUNCTION get_token_filter(\n _sortby jsonb DEFAULT '[{\"field\":\"datetime\",\"direction\":\"desc\"}]'::jsonb,\n token_item items DEFAULT NULL,\n prev boolean DEFAULT FALSE,\n inclusive boolean DEFAULT FALSE\n) RETURNS text AS $$\nDECLARE\n ltop text := '<';\n gtop text := '>';\n dir text;\n sort record;\n orfilter text := '';\n orfilters text[] := '{}'::text[];\n andfilters text[] := '{}'::text[];\n output text;\n token_where text;\nBEGIN\n IF _sortby IS NULL OR _sortby = '[]'::jsonb THEN\n _sortby := '[{\"field\":\"datetime\",\"direction\":\"desc\"}]'::jsonb;\n END IF;\n _sortby := _sortby || jsonb_build_object('field','id','direction',_sortby->0->>'direction');\n RAISE NOTICE 'Getting Token Filter. % %', _sortby, token_item;\n IF inclusive THEN\n orfilters := orfilters || format('( id=%L AND collection=%L )' , token_item.id, token_item.collection);\n END IF;\n\n FOR sort IN\n WITH s1 AS (\n SELECT\n _row,\n (queryable(value->>'field')).expression as _field,\n (value->>'field' = 'id') as _isid,\n get_sort_dir(value) as _dir\n FROM jsonb_array_elements(_sortby)\n WITH ORDINALITY AS t(value, _row)\n )\n SELECT\n _row,\n _field,\n _dir,\n get_token_val_str(_field, token_item) as _val\n FROM s1\n WHERE _row <= (SELECT min(_row) FROM s1 WHERE _isid)\n LOOP\n orfilter := NULL;\n RAISE NOTICE 'SORT: %', sort;\n IF sort._val IS NOT NULL AND ((prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC')) THEN\n orfilter := format($f$(\n (%s %s %s) OR (%s IS NULL)\n )$f$,\n sort._field,\n ltop,\n sort._val,\n sort._val\n );\n ELSIF sort._val IS NULL AND ((prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC')) THEN\n RAISE NOTICE '< but null';\n orfilter := format('%s IS NOT NULL', sort._field);\n ELSIF sort._val IS NULL THEN\n RAISE NOTICE '> but null';\n ELSE\n orfilter := format($f$(\n (%s %s %s) OR (%s IS NULL)\n )$f$,\n sort._field,\n gtop,\n sort._val,\n sort._field\n );\n END IF;\n RAISE NOTICE 'ORFILTER: %', orfilter;\n\n IF orfilter IS NOT NULL THEN\n IF sort._row = 1 THEN\n orfilters := orfilters || orfilter;\n ELSE\n orfilters := orfilters || format('(%s AND %s)', array_to_string(andfilters, ' AND '), orfilter);\n END IF;\n END IF;\n IF sort._val IS NOT NULL THEN\n andfilters := andfilters || format('%s = %s', sort._field, sort._val);\n ELSE\n andfilters := andfilters || format('%s IS NULL', sort._field);\n END IF;\n END LOOP;\n\n output := array_to_string(orfilters, ' OR ');\n\n token_where := concat('(',coalesce(output,'true'),')');\n IF trim(token_where) = '' THEN\n token_where := NULL;\n END IF;\n RAISE NOTICE 'TOKEN_WHERE: %',token_where;\n RETURN token_where;\n END;\n$$ LANGUAGE PLPGSQL SET transform_null_equals TO TRUE\n;\n\nCREATE OR REPLACE FUNCTION search_tohash(jsonb) RETURNS jsonb AS $$\n SELECT $1 - '{token,limit,context,includes,excludes}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION search_hash(jsonb, jsonb) RETURNS text AS $$\n SELECT md5(concat(search_tohash($1)::text,$2::text));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TABLE IF NOT EXISTS searches(\n hash text GENERATED ALWAYS AS (search_hash(search, metadata)) STORED PRIMARY KEY,\n search jsonb NOT NULL,\n _where text,\n orderby text,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n metadata jsonb DEFAULT '{}'::jsonb NOT NULL\n);\n\nCREATE TABLE IF NOT EXISTS search_wheres(\n id bigint generated always as identity primary key,\n _where text NOT NULL,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n statslastupdated timestamptz,\n estimated_count bigint,\n estimated_cost float,\n time_to_estimate float,\n total_count bigint,\n time_to_count float,\n partitions text[]\n);\n\nCREATE INDEX IF NOT EXISTS search_wheres_partitions ON search_wheres USING GIN (partitions);\nCREATE UNIQUE INDEX IF NOT EXISTS search_wheres_where ON search_wheres ((md5(_where)));\n\nCREATE OR REPLACE FUNCTION where_stats(\n inwhere text,\n updatestats boolean default false,\n conf jsonb default null\n) RETURNS search_wheres AS $$\nDECLARE\n t timestamptz;\n i interval;\n explain_json jsonb;\n partitions text[];\n sw search_wheres%ROWTYPE;\n inwhere_hash text := md5(inwhere);\n _context text := lower(context(conf));\n _stats_ttl interval := context_stats_ttl(conf);\n _estimated_cost_threshold float := context_estimated_cost(conf);\n _estimated_count_threshold int := context_estimated_count(conf);\n ro bool := pgstac.readonly(conf);\nBEGIN\n -- If updatestats is true then set ttl to 0\n IF updatestats THEN\n RAISE DEBUG 'Updatestats set to TRUE, setting TTL to 0';\n _stats_ttl := '0'::interval;\n END IF;\n\n -- If we don't need to calculate context, just return\n IF _context = 'off' THEN\n sw._where = inwhere;\n RETURN sw;\n END IF;\n\n -- Get any stats that we have.\n IF NOT ro THEN\n -- If there is a lock where another process is\n -- updating the stats, wait so that we don't end up calculating a bunch of times.\n SELECT * INTO sw FROM search_wheres WHERE md5(_where)=inwhere_hash FOR UPDATE;\n ELSE\n SELECT * INTO sw FROM search_wheres WHERE md5(_where)=inwhere_hash;\n END IF;\n\n -- If there is a cached row, figure out if we need to update\n IF\n sw IS NOT NULL\n AND sw.statslastupdated IS NOT NULL\n AND sw.total_count IS NOT NULL\n AND now() - sw.statslastupdated <= _stats_ttl\n THEN\n -- we have a cached row with data that is within our ttl\n RAISE DEBUG 'Stats present in table and lastupdated within ttl: %', sw;\n IF NOT ro THEN\n RAISE DEBUG 'Updating search_wheres only bumping lastused and usecount';\n UPDATE search_wheres SET\n lastused = now(),\n usecount = search_wheres.usecount + 1\n WHERE md5(_where) = inwhere_hash\n RETURNING * INTO sw;\n END IF;\n RAISE DEBUG 'Returning cached counts. %', sw;\n RETURN sw;\n END IF;\n\n -- Calculate estimated cost and rows\n -- Use explain to get estimated count/cost\n IF sw.estimated_count IS NULL OR sw.estimated_cost IS NULL THEN\n RAISE DEBUG 'Calculating estimated stats';\n t := clock_timestamp();\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s', inwhere)\n INTO explain_json;\n RAISE DEBUG 'Time for just the explain: %', clock_timestamp() - t;\n i := clock_timestamp() - t;\n\n sw.estimated_count := explain_json->0->'Plan'->'Plan Rows';\n sw.estimated_cost := explain_json->0->'Plan'->'Total Cost';\n sw.time_to_estimate := extract(epoch from i);\n END IF;\n\n RAISE DEBUG 'ESTIMATED_COUNT: %, THRESHOLD %', sw.estimated_count, _estimated_count_threshold;\n RAISE DEBUG 'ESTIMATED_COST: %, THRESHOLD %', sw.estimated_cost, _estimated_cost_threshold;\n\n -- If context is set to auto and the costs are within the threshold return the estimated costs\n IF\n _context = 'auto'\n AND sw.estimated_count >= _estimated_count_threshold\n AND sw.estimated_cost >= _estimated_cost_threshold\n THEN\n IF NOT ro THEN\n INSERT INTO search_wheres (\n _where,\n lastused,\n usecount,\n statslastupdated,\n estimated_count,\n estimated_cost,\n time_to_estimate,\n total_count,\n time_to_count\n ) VALUES (\n inwhere,\n now(),\n 1,\n now(),\n sw.estimated_count,\n sw.estimated_cost,\n sw.time_to_estimate,\n null,\n null\n ) ON CONFLICT ((md5(_where)))\n DO UPDATE SET\n lastused = EXCLUDED.lastused,\n usecount = search_wheres.usecount + 1,\n statslastupdated = EXCLUDED.statslastupdated,\n estimated_count = EXCLUDED.estimated_count,\n estimated_cost = EXCLUDED.estimated_cost,\n time_to_estimate = EXCLUDED.time_to_estimate,\n total_count = EXCLUDED.total_count,\n time_to_count = EXCLUDED.time_to_count\n RETURNING * INTO sw;\n END IF;\n RAISE DEBUG 'Estimates are within thresholds, returning estimates. %', sw;\n RETURN sw;\n END IF;\n\n -- Calculate Actual Count\n t := clock_timestamp();\n RAISE NOTICE 'Calculating actual count...';\n EXECUTE format(\n 'SELECT count(*) FROM items WHERE %s',\n inwhere\n ) INTO sw.total_count;\n i := clock_timestamp() - t;\n RAISE NOTICE 'Actual Count: % -- %', sw.total_count, i;\n sw.time_to_count := extract(epoch FROM i);\n\n IF NOT ro THEN\n INSERT INTO search_wheres (\n _where,\n lastused,\n usecount,\n statslastupdated,\n estimated_count,\n estimated_cost,\n time_to_estimate,\n total_count,\n time_to_count\n ) VALUES (\n inwhere,\n now(),\n 1,\n now(),\n sw.estimated_count,\n sw.estimated_cost,\n sw.time_to_estimate,\n sw.total_count,\n sw.time_to_count\n ) ON CONFLICT ((md5(_where)))\n DO UPDATE SET\n lastused = EXCLUDED.lastused,\n usecount = search_wheres.usecount + 1,\n statslastupdated = EXCLUDED.statslastupdated,\n estimated_count = EXCLUDED.estimated_count,\n estimated_cost = EXCLUDED.estimated_cost,\n time_to_estimate = EXCLUDED.time_to_estimate,\n total_count = EXCLUDED.total_count,\n time_to_count = EXCLUDED.time_to_count\n RETURNING * INTO sw;\n END IF;\n RAISE DEBUG 'Returning with actual count. %', sw;\n RETURN sw;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION search_query(\n _search jsonb = '{}'::jsonb,\n updatestats boolean = false,\n _metadata jsonb = '{}'::jsonb\n) RETURNS searches AS $$\nDECLARE\n search searches%ROWTYPE;\n cached_search searches%ROWTYPE;\n pexplain jsonb;\n t timestamptz;\n i interval;\n doupdate boolean := FALSE;\n insertfound boolean := FALSE;\n ro boolean := pgstac.readonly();\n found_search text;\nBEGIN\n RAISE NOTICE 'SEARCH: %', _search;\n -- Calculate hash, where clause, and order by statement\n search.search := _search;\n search.metadata := _metadata;\n search.hash := search_hash(_search, _metadata);\n search._where := stac_search_to_where(_search);\n search.orderby := sort_sqlorderby(_search);\n search.lastused := now();\n search.usecount := 1;\n\n -- If we are in read only mode, directly return search\n IF ro THEN\n RETURN search;\n END IF;\n\n RAISE NOTICE 'Updating Statistics for search: %s', search;\n -- Update statistics for times used and and when last used\n -- If the entry is locked, rather than waiting, skip updating the stats\n INSERT INTO searches (search, lastused, usecount, metadata)\n VALUES (search.search, now(), 1, search.metadata)\n ON CONFLICT DO NOTHING\n RETURNING * INTO cached_search\n ;\n\n IF NOT FOUND OR cached_search IS NULL THEN\n UPDATE searches SET\n lastused = now(),\n usecount = searches.usecount + 1\n WHERE hash = (\n SELECT hash FROM searches WHERE hash=search.hash FOR UPDATE SKIP LOCKED\n )\n RETURNING * INTO cached_search\n ;\n END IF;\n\n IF cached_search IS NOT NULL THEN\n cached_search._where = search._where;\n cached_search.orderby = search.orderby;\n RETURN cached_search;\n END IF;\n RETURN search;\n\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE OR REPLACE FUNCTION search_fromhash(\n _hash text\n) RETURNS searches AS $$\n SELECT * FROM search_query((SELECT search FROM searches WHERE hash=_hash LIMIT 1));\n$$ LANGUAGE SQL STRICT;\n\nCREATE OR REPLACE FUNCTION search_rows(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN partitions text[] DEFAULT NULL,\n IN _limit int DEFAULT 10\n) RETURNS SETOF items AS $$\nDECLARE\n base_query text;\n query text;\n sdate timestamptz;\n edate timestamptz;\n n int;\n records_left int := _limit;\n timer timestamptz := clock_timestamp();\n full_timer timestamptz := clock_timestamp();\nBEGIN\nIF _where IS NULL OR trim(_where) = '' THEN\n _where = ' TRUE ';\nEND IF;\nRAISE NOTICE 'Getting chunks for % %', _where, _orderby;\n\nbase_query := $q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n LIMIT %L\n$q$;\n\nIF _orderby ILIKE 'datetime d%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 DESC LOOP\n RAISE NOTICE 'Running Query for % to %. %', sdate, edate, age_ms(full_timer);\n query := format(\n base_query,\n sdate,\n edate,\n _where,\n _orderby,\n records_left\n );\n RAISE DEBUG 'QUERY: %', query;\n timer := clock_timestamp();\n RETURN QUERY EXECUTE query;\n\n GET DIAGNOSTICS n = ROW_COUNT;\n records_left := records_left - n;\n RAISE NOTICE 'Returned %/% Rows From % to %. % to go. Time: %ms', n, _limit, sdate, edate, records_left, age_ms(timer);\n timer := clock_timestamp();\n IF records_left <= 0 THEN\n RAISE NOTICE 'SEARCH_ROWS TOOK %ms', age_ms(full_timer);\n RETURN;\n END IF;\n END LOOP;\nELSIF _orderby ILIKE 'datetime a%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 ASC LOOP\n RAISE NOTICE 'Running Query for % to %. %', sdate, edate, age_ms(full_timer);\n query := format(\n base_query,\n sdate,\n edate,\n _where,\n _orderby,\n records_left\n );\n RAISE DEBUG 'QUERY: %', query;\n timer := clock_timestamp();\n RETURN QUERY EXECUTE query;\n\n GET DIAGNOSTICS n = ROW_COUNT;\n records_left := records_left - n;\n RAISE NOTICE 'Returned %/% Rows From % to %. % to go. Time: %ms', n, _limit, sdate, edate, records_left, age_ms(timer);\n timer := clock_timestamp();\n IF records_left <= 0 THEN\n RAISE NOTICE 'SEARCH_ROWS TOOK %ms', age_ms(full_timer);\n RETURN;\n END IF;\n END LOOP;\nELSE\n query := format($q$\n SELECT * FROM items\n WHERE %s\n ORDER BY %s\n LIMIT %L\n $q$, _where, _orderby, _limit\n );\n RAISE DEBUG 'QUERY: %', query;\n timer := clock_timestamp();\n RETURN QUERY EXECUTE query;\n RAISE NOTICE 'FULL QUERY TOOK %ms', age_ms(timer);\nEND IF;\nRAISE NOTICE 'SEARCH_ROWS TOOK %ms', age_ms(full_timer);\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE UNLOGGED TABLE format_item_cache(\n id text,\n collection text,\n fields text,\n hydrated bool,\n output jsonb,\n lastused timestamptz DEFAULT now(),\n usecount int DEFAULT 1,\n timetoformat float,\n PRIMARY KEY (collection, id, fields, hydrated)\n);\nCREATE INDEX ON format_item_cache (lastused);\n\nCREATE OR REPLACE FUNCTION format_item(_item items, _fields jsonb DEFAULT '{}', _hydrated bool DEFAULT TRUE) RETURNS jsonb AS $$\nDECLARE\n cache bool := get_setting_bool('format_cache');\n _output jsonb := null;\n t timestamptz := clock_timestamp();\nBEGIN\n IF cache THEN\n SELECT output INTO _output FROM format_item_cache\n WHERE id=_item.id AND collection=_item.collection AND fields=_fields::text AND hydrated=_hydrated;\n END IF;\n IF _output IS NULL THEN\n IF _hydrated THEN\n _output := content_hydrate(_item, _fields);\n ELSE\n _output := content_nonhydrated(_item, _fields);\n END IF;\n END IF;\n IF cache THEN\n INSERT INTO format_item_cache (id, collection, fields, hydrated, output, timetoformat)\n VALUES (_item.id, _item.collection, _fields::text, _hydrated, _output, age_ms(t))\n ON CONFLICT(collection, id, fields, hydrated) DO\n UPDATE\n SET lastused=now(), usecount = format_item_cache.usecount + 1\n ;\n END IF;\n RETURN _output;\n\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION search(_search jsonb = '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n searches searches%ROWTYPE;\n _where text;\n orderby text;\n search_where search_wheres%ROWTYPE;\n total_count bigint;\n token record;\n token_prev boolean;\n token_item items%ROWTYPE;\n token_where text;\n full_where text;\n init_ts timestamptz := clock_timestamp();\n timer timestamptz := clock_timestamp();\n hydrate bool := NOT (_search->'conf'->>'nohydrate' IS NOT NULL AND (_search->'conf'->>'nohydrate')::boolean = true);\n prev text;\n next text;\n context jsonb;\n collection jsonb;\n out_records jsonb;\n out_len int;\n _limit int := coalesce((_search->>'limit')::int, 10);\n _querylimit int;\n _fields jsonb := coalesce(_search->'fields', '{}'::jsonb);\n has_prev boolean := FALSE;\n has_next boolean := FALSE;\n links jsonb := '[]'::jsonb;\n base_url text:= concat(rtrim(base_url(_search->'conf'),'/'));\nBEGIN\n searches := search_query(_search);\n _where := searches._where;\n orderby := searches.orderby;\n search_where := where_stats(_where);\n total_count := coalesce(search_where.total_count, search_where.estimated_count);\n RAISE NOTICE 'SEARCH:TOKEN: %', _search->>'token';\n token := get_token_record(_search->>'token');\n RAISE NOTICE '***TOKEN: %', token;\n _querylimit := _limit + 1;\n IF token IS NOT NULL THEN\n token_prev := token.prev;\n token_item := token.item;\n token_where := get_token_filter(_search->'sortby', token_item, token_prev, FALSE);\n RAISE DEBUG 'TOKEN_WHERE: % (%ms from search start)', token_where, age_ms(timer);\n IF token_prev THEN -- if we are using a prev token, we know has_next is true\n RAISE DEBUG 'There is a previous token, so automatically setting has_next to true';\n has_next := TRUE;\n orderby := sort_sqlorderby(_search, TRUE);\n ELSE\n RAISE DEBUG 'There is a next token, so automatically setting has_prev to true';\n has_prev := TRUE;\n\n END IF;\n ELSE -- if there was no token, we know there is no prev\n RAISE DEBUG 'There is no token, so we know there is no prev. setting has_prev to false';\n has_prev := FALSE;\n END IF;\n\n full_where := concat_ws(' AND ', _where, token_where);\n RAISE NOTICE 'FULL WHERE CLAUSE: %', full_where;\n RAISE NOTICE 'Time to get counts and build query %', age_ms(timer);\n timer := clock_timestamp();\n\n IF hydrate THEN\n RAISE NOTICE 'Getting hydrated data.';\n ELSE\n RAISE NOTICE 'Getting non-hydrated data.';\n END IF;\n RAISE NOTICE 'CACHE SET TO %', get_setting_bool('format_cache');\n RAISE NOTICE 'Time to set hydration/formatting %', age_ms(timer);\n timer := clock_timestamp();\n SELECT jsonb_agg(format_item(i, _fields, hydrate)) INTO out_records\n FROM search_rows(\n full_where,\n orderby,\n search_where.partitions,\n _querylimit\n ) as i;\n\n RAISE NOTICE 'Time to fetch rows %', age_ms(timer);\n timer := clock_timestamp();\n\n\n IF token_prev THEN\n out_records := flip_jsonb_array(out_records);\n END IF;\n\n RAISE NOTICE 'Query returned % records.', jsonb_array_length(out_records);\n RAISE DEBUG 'TOKEN: % %', token_item.id, token_item.collection;\n RAISE DEBUG 'RECORD_1: % %', out_records->0->>'id', out_records->0->>'collection';\n RAISE DEBUG 'RECORD-1: % %', out_records->-1->>'id', out_records->-1->>'collection';\n\n -- REMOVE records that were from our token\n IF out_records->0->>'id' = token_item.id AND out_records->0->>'collection' = token_item.collection THEN\n out_records := out_records - 0;\n ELSIF out_records->-1->>'id' = token_item.id AND out_records->-1->>'collection' = token_item.collection THEN\n out_records := out_records - -1;\n END IF;\n\n out_len := jsonb_array_length(out_records);\n\n IF out_len = _limit + 1 THEN\n IF token_prev THEN\n has_prev := TRUE;\n out_records := out_records - 0;\n ELSE\n has_next := TRUE;\n out_records := out_records - -1;\n END IF;\n END IF;\n\n\n links := links || jsonb_build_object(\n 'rel', 'root',\n 'type', 'application/json',\n 'href', base_url\n ) || jsonb_build_object(\n 'rel', 'self',\n 'type', 'application/json',\n 'href', concat(base_url, '/search')\n );\n\n IF has_next THEN\n next := concat(out_records->-1->>'collection', ':', out_records->-1->>'id');\n RAISE NOTICE 'HAS NEXT | %', next;\n links := links || jsonb_build_object(\n 'rel', 'next',\n 'type', 'application/geo+json',\n 'method', 'GET',\n 'href', concat(base_url, '/search?token=next:', next)\n );\n END IF;\n\n IF has_prev THEN\n prev := concat(out_records->0->>'collection', ':', out_records->0->>'id');\n RAISE NOTICE 'HAS PREV | %', prev;\n links := links || jsonb_build_object(\n 'rel', 'prev',\n 'type', 'application/geo+json',\n 'method', 'GET',\n 'href', concat(base_url, '/search?token=prev:', prev)\n );\n END IF;\n\n RAISE NOTICE 'Time to get prev/next %', age_ms(timer);\n timer := clock_timestamp();\n\n\n collection := jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb),\n 'links', links\n );\n\n\n\n IF context(_search->'conf') != 'off' THEN\n collection := collection || jsonb_strip_nulls(jsonb_build_object(\n 'numberMatched', total_count,\n 'numberReturned', coalesce(jsonb_array_length(out_records), 0)\n ));\n ELSE\n collection := collection || jsonb_strip_nulls(jsonb_build_object(\n 'numberReturned', coalesce(jsonb_array_length(out_records), 0)\n ));\n END IF;\n\n IF get_setting_bool('timing', _search->'conf') THEN\n collection = collection || jsonb_build_object('timing', age_ms(init_ts));\n END IF;\n\n RAISE NOTICE 'Time to build final json %', age_ms(timer);\n timer := clock_timestamp();\n\n RAISE NOTICE 'Total Time: %', age_ms(current_timestamp);\n RAISE NOTICE 'RETURNING % records. NEXT: %. PREV: %', collection->>'numberReturned', collection->>'next', collection->>'prev';\n RETURN collection;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION search_cursor(_search jsonb = '{}'::jsonb) RETURNS refcursor AS $$\nDECLARE\n curs refcursor;\n searches searches%ROWTYPE;\n _where text;\n _orderby text;\n q text;\n\nBEGIN\n searches := search_query(_search);\n _where := searches._where;\n _orderby := searches.orderby;\n\n OPEN curs FOR\n WITH p AS (\n SELECT * FROM partition_queries(_where, _orderby) p\n )\n SELECT\n CASE WHEN EXISTS (SELECT 1 FROM p) THEN\n (SELECT format($q$\n SELECT * FROM (\n %s\n ) total\n $q$,\n string_agg(\n format($q$ SELECT * FROM ( %s ) AS sub $q$, p),\n '\n UNION ALL\n '\n )\n ))\n ELSE NULL\n END FROM p;\n RETURN curs;\nEND;\n$$ LANGUAGE PLPGSQL;\nCREATE OR REPLACE VIEW collections_asitems AS\nSELECT\n id,\n geometry,\n 'collections' AS collection,\n datetime,\n end_datetime,\n jsonb_build_object(\n 'properties', content - '{links,assets,stac_version,stac_extensions}',\n 'links', content->'links',\n 'assets', content->'assets',\n 'stac_version', content->'stac_version',\n 'stac_extensions', content->'stac_extensions'\n ) AS content,\n content as collectionjson\nFROM collections;\n\n\nCREATE OR REPLACE FUNCTION collection_search_matched(\n IN _search jsonb DEFAULT '{}'::jsonb,\n OUT matched bigint\n) RETURNS bigint AS $$\nDECLARE\n _where text := stac_search_to_where(_search);\nBEGIN\n EXECUTE format(\n $query$\n SELECT\n count(*)\n FROM\n collections_asitems\n WHERE %s\n ;\n $query$,\n _where\n ) INTO matched;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION collection_search_rows(\n _search jsonb DEFAULT '{}'::jsonb\n) RETURNS SETOF jsonb AS $$\nDECLARE\n _where text := stac_search_to_where(_search);\n _limit int := coalesce((_search->>'limit')::int, 10);\n _fields jsonb := coalesce(_search->'fields', '{}'::jsonb);\n _orderby text;\n _offset int := COALESCE((_search->>'offset')::int, 0);\nBEGIN\n _orderby := sort_sqlorderby(\n jsonb_build_object(\n 'sortby',\n coalesce(\n _search->'sortby',\n '[{\"field\": \"id\", \"direction\": \"asc\"}]'::jsonb\n )\n )\n );\n RETURN QUERY EXECUTE format(\n $query$\n SELECT\n jsonb_fields(collectionjson, %L) as c\n FROM\n collections_asitems\n WHERE %s\n ORDER BY %s\n LIMIT %L\n OFFSET %L\n ;\n $query$,\n _fields,\n _where,\n _orderby,\n _limit,\n _offset\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION collection_search(\n _search jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\nDECLARE\n out_records jsonb;\n number_matched bigint := collection_search_matched(_search);\n number_returned bigint;\n _limit int := coalesce((_search->>'limit')::float::int, 10);\n _offset int := coalesce((_search->>'offset')::float::int, 0);\n links jsonb := '[]';\n ret jsonb;\n base_url text:= concat(rtrim(base_url(_search->'conf'),'/'), '/collections');\n prevoffset int;\n nextoffset int;\nBEGIN\n SELECT\n coalesce(jsonb_agg(c), '[]')\n INTO out_records\n FROM collection_search_rows(_search) c;\n\n number_returned := jsonb_array_length(out_records);\n RAISE DEBUG 'nm: %, nr: %, l:%, o:%', number_matched, number_returned, _limit, _offset;\n\n\n\n IF _limit <= number_matched AND number_matched > 0 THEN --need to have paging links\n nextoffset := least(_offset + _limit, number_matched - 1);\n prevoffset := greatest(_offset - _limit, 0);\n\n IF _offset > 0 THEN\n links := links || jsonb_build_object(\n 'rel', 'prev',\n 'type', 'application/json',\n 'method', 'GET' ,\n 'href', base_url,\n 'body', jsonb_build_object('offset', prevoffset),\n 'merge', TRUE\n );\n END IF;\n\n IF (_offset + _limit < number_matched) THEN\n links := links || jsonb_build_object(\n 'rel', 'next',\n 'type', 'application/json',\n 'method', 'GET' ,\n 'href', base_url,\n 'body', jsonb_build_object('offset', nextoffset),\n 'merge', TRUE\n );\n END IF;\n\n END IF;\n\n ret := jsonb_build_object(\n 'collections', out_records,\n 'numberMatched', number_matched,\n 'numberReturned', number_returned,\n 'links', links\n );\n RETURN ret;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION tileenvelope(zoom int, x int, y int) RETURNS geometry AS $$\nWITH t AS (\n SELECT\n 20037508.3427892 as merc_max,\n -20037508.3427892 as merc_min,\n (2 * 20037508.3427892) / (2 ^ zoom) as tile_size\n)\nSELECT st_makeenvelope(\n merc_min + (tile_size * x),\n merc_max - (tile_size * (y + 1)),\n merc_min + (tile_size * (x + 1)),\n merc_max - (tile_size * y),\n 3857\n) FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;DROP FUNCTION IF EXISTS mercgrid;\n\n\nCREATE OR REPLACE FUNCTION ftime() RETURNS interval as $$\nSELECT age(clock_timestamp(), transaction_timestamp());\n$$ LANGUAGE SQL;\nSET SEARCH_PATH to pgstac, public;\n\nDROP FUNCTION IF EXISTS geometrysearch;\nCREATE OR REPLACE FUNCTION geometrysearch(\n IN geom geometry,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE, -- Return as soon as the passed in geometry is full covered\n IN skipcovered boolean DEFAULT TRUE -- Skip any items that would show up completely under the previous items\n) RETURNS jsonb AS $$\nDECLARE\n search searches%ROWTYPE;\n curs refcursor;\n _where text;\n query text;\n iter_record items%ROWTYPE;\n out_records jsonb := '{}'::jsonb[];\n exit_flag boolean := FALSE;\n counter int := 1;\n scancounter int := 1;\n remaining_limit int := _scanlimit;\n tilearea float;\n unionedgeom geometry;\n clippedgeom geometry;\n unionedgeom_area float := 0;\n prev_area float := 0;\n excludes text[];\n includes text[];\n\nBEGIN\n DROP TABLE IF EXISTS pgstac_results;\n CREATE TEMP TABLE pgstac_results (content jsonb) ON COMMIT DROP;\n\n -- If the passed in geometry is not an area set exitwhenfull and skipcovered to false\n IF ST_GeometryType(geom) !~* 'polygon' THEN\n RAISE NOTICE 'GEOMETRY IS NOT AN AREA';\n skipcovered = FALSE;\n exitwhenfull = FALSE;\n END IF;\n\n -- If skipcovered is true then you will always want to exit when the passed in geometry is full\n IF skipcovered THEN\n exitwhenfull := TRUE;\n END IF;\n\n search := search_fromhash(queryhash);\n\n IF search IS NULL THEN\n RAISE EXCEPTION 'Search with Query Hash % Not Found', queryhash;\n END IF;\n\n tilearea := st_area(geom);\n _where := format('%s AND st_intersects(geometry, %L::geometry)', search._where, geom);\n\n\n FOR query IN SELECT * FROM partition_queries(_where, search.orderby) LOOP\n query := format('%s LIMIT %L', query, remaining_limit);\n RAISE NOTICE '%', query;\n OPEN curs FOR EXECUTE query;\n LOOP\n FETCH curs INTO iter_record;\n EXIT WHEN NOT FOUND;\n IF exitwhenfull OR skipcovered THEN -- If we are not using exitwhenfull or skipcovered, we do not need to do expensive geometry operations\n clippedgeom := st_intersection(geom, iter_record.geometry);\n\n IF unionedgeom IS NULL THEN\n unionedgeom := clippedgeom;\n ELSE\n unionedgeom := st_union(unionedgeom, clippedgeom);\n END IF;\n\n unionedgeom_area := st_area(unionedgeom);\n\n IF skipcovered AND prev_area = unionedgeom_area THEN\n scancounter := scancounter + 1;\n CONTINUE;\n END IF;\n\n prev_area := unionedgeom_area;\n\n RAISE NOTICE '% % % %', unionedgeom_area/tilearea, counter, scancounter, ftime();\n END IF;\n RAISE NOTICE '% %', iter_record, content_hydrate(iter_record, fields);\n INSERT INTO pgstac_results (content) VALUES (content_hydrate(iter_record, fields));\n\n IF counter >= _limit\n OR scancounter > _scanlimit\n OR ftime() > _timelimit\n OR (exitwhenfull AND unionedgeom_area >= tilearea)\n THEN\n exit_flag := TRUE;\n EXIT;\n END IF;\n counter := counter + 1;\n scancounter := scancounter + 1;\n\n END LOOP;\n CLOSE curs;\n EXIT WHEN exit_flag;\n remaining_limit := _scanlimit - scancounter;\n END LOOP;\n\n SELECT jsonb_agg(content) INTO out_records FROM pgstac_results WHERE content IS NOT NULL;\n\n RETURN jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb)\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDROP FUNCTION IF EXISTS geojsonsearch;\nCREATE OR REPLACE FUNCTION geojsonsearch(\n IN geojson jsonb,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_geomfromgeojson(geojson),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS xyzsearch;\nCREATE OR REPLACE FUNCTION xyzsearch(\n IN _x int,\n IN _y int,\n IN _z int,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_transform(tileenvelope(_z, _x, _y), 4326),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE PROCEDURE analyze_items() AS $$\nDECLARE\n q text;\n timeout_ts timestamptz;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n SELECT format('ANALYZE (VERBOSE, SKIP_LOCKED) %I;', relname) INTO q\n FROM pg_stat_user_tables\n WHERE relname like '_item%' AND (n_mod_since_analyze>0 OR last_analyze IS NULL) LIMIT 1;\n IF NOT FOUND THEN\n EXIT;\n END IF;\n RAISE NOTICE '%', q;\n EXECUTE q;\n COMMIT;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE PROCEDURE validate_constraints() AS $$\nDECLARE\n q text;\nBEGIN\n FOR q IN\n SELECT\n FORMAT(\n 'ALTER TABLE %I.%I VALIDATE CONSTRAINT %I;',\n nsp.nspname,\n cls.relname,\n con.conname\n )\n\n FROM pg_constraint AS con\n JOIN pg_class AS cls\n ON con.conrelid = cls.oid\n JOIN pg_namespace AS nsp\n ON cls.relnamespace = nsp.oid\n WHERE convalidated = FALSE AND contype in ('c','f')\n AND nsp.nspname = 'pgstac'\n LOOP\n RAISE NOTICE '%', q;\n PERFORM run_or_queue(q);\n COMMIT;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION collection_extent(_collection text, runupdate boolean default false) RETURNS jsonb AS $$\nDECLARE\n geom_extent geometry;\n mind timestamptz;\n maxd timestamptz;\n extent jsonb;\nBEGIN\n IF runupdate THEN\n PERFORM update_partition_stats_q(partition)\n FROM partitions_view WHERE collection=_collection;\n END IF;\n SELECT\n min(lower(dtrange)),\n max(upper(edtrange)),\n st_extent(spatial)\n INTO\n mind,\n maxd,\n geom_extent\n FROM partitions_view\n WHERE collection=_collection;\n\n IF geom_extent IS NOT NULL AND mind IS NOT NULL AND maxd IS NOT NULL THEN\n extent := jsonb_build_object(\n 'spatial', jsonb_build_object(\n 'bbox', to_jsonb(array[array[st_xmin(geom_extent), st_ymin(geom_extent), st_xmax(geom_extent), st_ymax(geom_extent)]])\n ),\n 'temporal', jsonb_build_object(\n 'interval', to_jsonb(array[array[mind, maxd]])\n )\n );\n RETURN extent;\n END IF;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('id', '{\"title\": \"Item ID\",\"description\": \"Item identifier\",\"$ref\": \"https://schemas.stacspec.org/v1.0.0/item-spec/json-schema/item.json#/definitions/core/allOf/2/properties/id\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('geometry', '{\"title\": \"Item Geometry\",\"description\": \"Item Geometry\",\"$ref\": \"https://geojson.org/schema/Feature.json\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('datetime','{\"description\": \"Datetime\",\"type\": \"string\",\"title\": \"Acquired\",\"format\": \"date-time\",\"pattern\": \"(\\\\+00:00|Z)$\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDELETE FROM queryables a USING queryables b\n WHERE a.name = b.name AND a.collection_ids IS NOT DISTINCT FROM b.collection_ids AND a.id > b.id;\n\n\nINSERT INTO pgstac_settings (name, value) VALUES\n ('context', 'off'),\n ('context_estimated_count', '100000'),\n ('context_estimated_cost', '100000'),\n ('context_stats_ttl', '1 day'),\n ('default_filter_lang', 'cql2-json'),\n ('additional_properties', 'true'),\n ('use_queue', 'false'),\n ('queue_timeout', '10 minutes'),\n ('update_collection_extent', 'false'),\n ('format_cache', 'false'),\n ('readonly', 'false')\nON CONFLICT DO NOTHING\n;\n\n\nINSERT INTO cql2_ops (op, template, types) VALUES\n ('eq', '%s = %s', NULL),\n ('neq', '%s != %s', NULL),\n ('ne', '%s != %s', NULL),\n ('!=', '%s != %s', NULL),\n ('<>', '%s != %s', NULL),\n ('lt', '%s < %s', NULL),\n ('lte', '%s <= %s', NULL),\n ('gt', '%s > %s', NULL),\n ('gte', '%s >= %s', NULL),\n ('le', '%s <= %s', NULL),\n ('ge', '%s >= %s', NULL),\n ('=', '%s = %s', NULL),\n ('<', '%s < %s', NULL),\n ('<=', '%s <= %s', NULL),\n ('>', '%s > %s', NULL),\n ('>=', '%s >= %s', NULL),\n ('like', '%s LIKE %s', NULL),\n ('ilike', '%s ILIKE %s', NULL),\n ('+', '%s + %s', NULL),\n ('-', '%s - %s', NULL),\n ('*', '%s * %s', NULL),\n ('/', '%s / %s', NULL),\n ('not', 'NOT (%s)', NULL),\n ('between', '%s BETWEEN %s AND %s', NULL),\n ('isnull', '%s IS NULL', NULL),\n ('upper', 'upper(%s)', NULL),\n ('lower', 'lower(%s)', NULL),\n ('casei', 'upper(%s)', NULL),\n ('accenti', 'unaccent(%s)', NULL)\nON CONFLICT (op) DO UPDATE\n SET\n template = EXCLUDED.template\n;\n\n\nALTER FUNCTION to_text COST 5000;\nALTER FUNCTION to_float COST 5000;\nALTER FUNCTION to_int COST 5000;\nALTER FUNCTION to_tstz COST 5000;\nALTER FUNCTION to_text_array COST 5000;\n\nALTER FUNCTION update_partition_stats SECURITY DEFINER;\nALTER FUNCTION partition_after_triggerfunc SECURITY DEFINER;\nALTER FUNCTION drop_table_constraints SECURITY DEFINER;\nALTER FUNCTION create_table_constraints SECURITY DEFINER;\nALTER FUNCTION check_partition SECURITY DEFINER;\nALTER FUNCTION repartition SECURITY DEFINER;\nALTER FUNCTION where_stats SECURITY DEFINER;\nALTER FUNCTION search_query SECURITY DEFINER;\nALTER FUNCTION format_item SECURITY DEFINER;\nALTER FUNCTION maintain_index SECURITY DEFINER;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nGRANT ALL ON SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac to pgstac_admin;\n\n-- pgstac_read role limited to using function apis\nGRANT EXECUTE ON FUNCTION search TO pgstac_read;\nGRANT EXECUTE ON FUNCTION search_query TO pgstac_read;\nGRANT EXECUTE ON FUNCTION item_by_id TO pgstac_read;\nGRANT EXECUTE ON FUNCTION get_item TO pgstac_read;\nGRANT SELECT ON ALL TABLES IN SCHEMA pgstac TO pgstac_read;\n\n\nGRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_ingest;\nGRANT USAGE ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_ingest;\n\nREVOKE ALL PRIVILEGES ON PROCEDURE run_queued_queries FROM public;\nGRANT ALL ON PROCEDURE run_queued_queries TO pgstac_admin;\n\nREVOKE ALL PRIVILEGES ON FUNCTION run_queued_queries_intransaction FROM public;\nGRANT ALL ON FUNCTION run_queued_queries_intransaction TO pgstac_admin;\n\nRESET ROLE;\n\nSET ROLE pgstac_ingest;\nSELECT update_partition_stats_q(partition) FROM partitions_view;\nSELECT set_version('0.9.10');\n" }, { "path": "src/pgstac/migrations/pgstac.0.9.11-unreleased.sql", "content": "SET client_min_messages TO WARNING;\nSET SEARCH_PATH to pgstac, public;\nRESET ROLE;\nDO $$\nDECLARE\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='postgis') THEN\n CREATE EXTENSION IF NOT EXISTS postgis;\n END IF;\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='btree_gist') THEN\n CREATE EXTENSION IF NOT EXISTS btree_gist;\n END IF;\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='unaccent') THEN\n CREATE EXTENSION IF NOT EXISTS unaccent;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_admin;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_read;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_ingest;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n\nGRANT pgstac_admin TO current_user;\n\n-- Function to make sure pgstac_admin is the owner of items\nCREATE OR REPLACE FUNCTION pgstac_admin_owns() RETURNS VOID AS $$\nDECLARE\n f RECORD;\nBEGIN\n FOR f IN (\n SELECT\n concat(\n oid::regproc::text,\n '(',\n coalesce(pg_get_function_identity_arguments(oid),''),\n ')'\n ) AS name,\n CASE prokind WHEN 'f' THEN 'FUNCTION' WHEN 'p' THEN 'PROCEDURE' WHEN 'a' THEN 'AGGREGATE' END as typ\n FROM pg_proc\n WHERE\n pronamespace=to_regnamespace('pgstac')\n AND proowner != to_regrole('pgstac_admin')\n AND proname NOT LIKE 'pg_stat%'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n FOR f IN (\n SELECT\n oid::regclass::text as name,\n CASE relkind\n WHEN 'i' THEN 'INDEX'\n WHEN 'I' THEN 'INDEX'\n WHEN 'p' THEN 'TABLE'\n WHEN 'r' THEN 'TABLE'\n WHEN 'v' THEN 'VIEW'\n WHEN 'S' THEN 'SEQUENCE'\n ELSE NULL\n END as typ\n FROM pg_class\n WHERE relnamespace=to_regnamespace('pgstac') and relowner != to_regrole('pgstac_admin') AND relkind IN ('r','p','v','S') AND relname NOT LIKE 'pg_stat'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\nSELECT pgstac_admin_owns();\n\nCREATE SCHEMA IF NOT EXISTS pgstac AUTHORIZATION pgstac_admin;\n\nGRANT ALL ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_admin;\n\nALTER ROLE pgstac_admin SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_read SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_ingest SET SEARCH_PATH TO pgstac, public;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\n\nGRANT pgstac_read TO pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\n\nSET ROLE pgstac_admin;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\nRESET ROLE;\n\nSET ROLE pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\nRESET ROLE;\n\nSET SEARCH_PATH TO pgstac, public;\nSET ROLE pgstac_admin;\n\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS analyze_items;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS validate_constraints;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n-- Install these idempotently as migrations do not put them before trying to modify the collections table\n\n\nCREATE OR REPLACE FUNCTION collection_geom(content jsonb)\nRETURNS geometry AS $$\n WITH box AS (SELECT content->'extent'->'spatial'->'bbox'->0 as box)\n SELECT\n st_makeenvelope(\n (box->>0)::float,\n (box->>1)::float,\n (box->>2)::float,\n (box->>3)::float,\n 4326\n )\n FROM box;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION collection_datetime(content jsonb)\nRETURNS timestamptz AS $$\n SELECT\n CASE\n WHEN\n (content->'extent'->'temporal'->'interval'->0->>0) IS NULL\n THEN '-infinity'::timestamptz\n ELSE\n (content->'extent'->'temporal'->'interval'->0->>0)::timestamptz\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION collection_enddatetime(content jsonb)\nRETURNS timestamptz AS $$\n SELECT\n CASE\n WHEN\n (content->'extent'->'temporal'->'interval'->0->>1) IS NULL\n THEN 'infinity'::timestamptz\n ELSE\n (content->'extent'->'temporal'->'interval'->0->>1)::timestamptz\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n-- BEGIN migra calculated SQL\n-- END migra calculated SQL\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('id', '{\"title\": \"Item ID\",\"description\": \"Item identifier\",\"$ref\": \"https://schemas.stacspec.org/v1.0.0/item-spec/json-schema/item.json#/definitions/core/allOf/2/properties/id\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('geometry', '{\"title\": \"Item Geometry\",\"description\": \"Item Geometry\",\"$ref\": \"https://geojson.org/schema/Feature.json\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('datetime','{\"description\": \"Datetime\",\"type\": \"string\",\"title\": \"Acquired\",\"format\": \"date-time\",\"pattern\": \"(\\\\+00:00|Z)$\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDELETE FROM queryables a USING queryables b\n WHERE a.name = b.name AND a.collection_ids IS NOT DISTINCT FROM b.collection_ids AND a.id > b.id;\n\n\nINSERT INTO pgstac_settings (name, value) VALUES\n ('context', 'off'),\n ('context_estimated_count', '100000'),\n ('context_estimated_cost', '100000'),\n ('context_stats_ttl', '1 day'),\n ('default_filter_lang', 'cql2-json'),\n ('additional_properties', 'true'),\n ('use_queue', 'false'),\n ('queue_timeout', '10 minutes'),\n ('update_collection_extent', 'false'),\n ('format_cache', 'false'),\n ('readonly', 'false')\nON CONFLICT DO NOTHING\n;\n\n\nINSERT INTO cql2_ops (op, template, types) VALUES\n ('eq', '%s = %s', NULL),\n ('neq', '%s != %s', NULL),\n ('ne', '%s != %s', NULL),\n ('!=', '%s != %s', NULL),\n ('<>', '%s != %s', NULL),\n ('lt', '%s < %s', NULL),\n ('lte', '%s <= %s', NULL),\n ('gt', '%s > %s', NULL),\n ('gte', '%s >= %s', NULL),\n ('le', '%s <= %s', NULL),\n ('ge', '%s >= %s', NULL),\n ('=', '%s = %s', NULL),\n ('<', '%s < %s', NULL),\n ('<=', '%s <= %s', NULL),\n ('>', '%s > %s', NULL),\n ('>=', '%s >= %s', NULL),\n ('like', '%s LIKE %s', NULL),\n ('ilike', '%s ILIKE %s', NULL),\n ('+', '%s + %s', NULL),\n ('-', '%s - %s', NULL),\n ('*', '%s * %s', NULL),\n ('/', '%s / %s', NULL),\n ('not', 'NOT (%s)', NULL),\n ('between', '%s BETWEEN %s AND %s', NULL),\n ('isnull', '%s IS NULL', NULL),\n ('upper', 'upper(%s)', NULL),\n ('lower', 'lower(%s)', NULL),\n ('casei', 'upper(%s)', NULL),\n ('accenti', 'unaccent(%s)', NULL)\nON CONFLICT (op) DO UPDATE\n SET\n template = EXCLUDED.template\n;\n\n\nALTER FUNCTION to_text COST 5000;\nALTER FUNCTION to_float COST 5000;\nALTER FUNCTION to_int COST 5000;\nALTER FUNCTION to_tstz COST 5000;\nALTER FUNCTION to_text_array COST 5000;\n\nALTER FUNCTION update_partition_stats SECURITY DEFINER;\nALTER FUNCTION partition_after_triggerfunc SECURITY DEFINER;\nALTER FUNCTION drop_table_constraints SECURITY DEFINER;\nALTER FUNCTION create_table_constraints SECURITY DEFINER;\nALTER FUNCTION check_partition SECURITY DEFINER;\nALTER FUNCTION repartition SECURITY DEFINER;\nALTER FUNCTION where_stats SECURITY DEFINER;\nALTER FUNCTION search_query SECURITY DEFINER;\nALTER FUNCTION format_item SECURITY DEFINER;\nALTER FUNCTION maintain_index SECURITY DEFINER;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nGRANT ALL ON SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac to pgstac_admin;\n\n-- pgstac_read role limited to using function apis\nGRANT EXECUTE ON FUNCTION search TO pgstac_read;\nGRANT EXECUTE ON FUNCTION search_query TO pgstac_read;\nGRANT EXECUTE ON FUNCTION item_by_id TO pgstac_read;\nGRANT EXECUTE ON FUNCTION get_item TO pgstac_read;\nGRANT SELECT ON ALL TABLES IN SCHEMA pgstac TO pgstac_read;\n\n\nGRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_ingest;\nGRANT USAGE ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_ingest;\n\nREVOKE ALL PRIVILEGES ON PROCEDURE run_queued_queries FROM public;\nGRANT ALL ON PROCEDURE run_queued_queries TO pgstac_admin;\n\nREVOKE ALL PRIVILEGES ON FUNCTION run_queued_queries_intransaction FROM public;\nGRANT ALL ON FUNCTION run_queued_queries_intransaction TO pgstac_admin;\n\nRESET ROLE;\n\nSET ROLE pgstac_ingest;\nSELECT update_partition_stats_q(partition) FROM partitions_view;\nSELECT set_version('unreleased');\n" }, { "path": "src/pgstac/migrations/pgstac.0.9.11.sql", "content": "RESET ROLE;\nDO $$\nDECLARE\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='postgis') THEN\n CREATE EXTENSION IF NOT EXISTS postgis;\n END IF;\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='btree_gist') THEN\n CREATE EXTENSION IF NOT EXISTS btree_gist;\n END IF;\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='unaccent') THEN\n CREATE EXTENSION IF NOT EXISTS unaccent;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_admin;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_read;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_ingest;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n\nGRANT pgstac_admin TO current_user;\n\n-- Function to make sure pgstac_admin is the owner of items\nCREATE OR REPLACE FUNCTION pgstac_admin_owns() RETURNS VOID AS $$\nDECLARE\n f RECORD;\nBEGIN\n FOR f IN (\n SELECT\n concat(\n oid::regproc::text,\n '(',\n coalesce(pg_get_function_identity_arguments(oid),''),\n ')'\n ) AS name,\n CASE prokind WHEN 'f' THEN 'FUNCTION' WHEN 'p' THEN 'PROCEDURE' WHEN 'a' THEN 'AGGREGATE' END as typ\n FROM pg_proc\n WHERE\n pronamespace=to_regnamespace('pgstac')\n AND proowner != to_regrole('pgstac_admin')\n AND proname NOT LIKE 'pg_stat%'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n FOR f IN (\n SELECT\n oid::regclass::text as name,\n CASE relkind\n WHEN 'i' THEN 'INDEX'\n WHEN 'I' THEN 'INDEX'\n WHEN 'p' THEN 'TABLE'\n WHEN 'r' THEN 'TABLE'\n WHEN 'v' THEN 'VIEW'\n WHEN 'S' THEN 'SEQUENCE'\n ELSE NULL\n END as typ\n FROM pg_class\n WHERE relnamespace=to_regnamespace('pgstac') and relowner != to_regrole('pgstac_admin') AND relkind IN ('r','p','v','S') AND relname NOT LIKE 'pg_stat'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\nSELECT pgstac_admin_owns();\n\nCREATE SCHEMA IF NOT EXISTS pgstac AUTHORIZATION pgstac_admin;\n\nGRANT ALL ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_admin;\n\nALTER ROLE pgstac_admin SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_read SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_ingest SET SEARCH_PATH TO pgstac, public;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\n\nGRANT pgstac_read TO pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\n\nSET ROLE pgstac_admin;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\nRESET ROLE;\n\nSET ROLE pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\nRESET ROLE;\n\nSET SEARCH_PATH TO pgstac, public;\nSET ROLE pgstac_admin;\n\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS analyze_items;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS validate_constraints;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n-- Install these idempotently as migrations do not put them before trying to modify the collections table\n\n\nCREATE OR REPLACE FUNCTION collection_geom(content jsonb)\nRETURNS geometry AS $$\n WITH box AS (SELECT content->'extent'->'spatial'->'bbox'->0 as box)\n SELECT\n st_makeenvelope(\n (box->>0)::float,\n (box->>1)::float,\n (box->>2)::float,\n (box->>3)::float,\n 4326\n )\n FROM box;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION collection_datetime(content jsonb)\nRETURNS timestamptz AS $$\n SELECT\n CASE\n WHEN\n (content->'extent'->'temporal'->'interval'->0->>0) IS NULL\n THEN '-infinity'::timestamptz\n ELSE\n (content->'extent'->'temporal'->'interval'->0->>0)::timestamptz\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION collection_enddatetime(content jsonb)\nRETURNS timestamptz AS $$\n SELECT\n CASE\n WHEN\n (content->'extent'->'temporal'->'interval'->0->>1) IS NULL\n THEN 'infinity'::timestamptz\n ELSE\n (content->'extent'->'temporal'->'interval'->0->>1)::timestamptz\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE TABLE IF NOT EXISTS migrations (\n version text PRIMARY KEY,\n datetime timestamptz DEFAULT clock_timestamp() NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION get_version() RETURNS text AS $$\n SELECT version FROM pgstac.migrations ORDER BY datetime DESC, version DESC LIMIT 1;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION set_version(text) RETURNS text AS $$\n INSERT INTO pgstac.migrations (version) VALUES ($1)\n ON CONFLICT DO NOTHING\n RETURNING version;\n$$ LANGUAGE SQL;\n\n\nCREATE TABLE IF NOT EXISTS pgstac_settings (\n name text PRIMARY KEY,\n value text NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION table_empty(text) RETURNS boolean AS $$\nDECLARE\n retval boolean;\nBEGIN\n EXECUTE format($q$\n SELECT NOT EXISTS (SELECT 1 FROM %I LIMIT 1)\n $q$,\n $1\n ) INTO retval;\n RETURN retval;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION get_setting(IN _setting text, IN conf jsonb DEFAULT NULL) RETURNS text AS $$\nSELECT COALESCE(\n nullif(conf->>_setting, ''),\n nullif(current_setting(concat('pgstac.',_setting), TRUE),''),\n nullif((SELECT value FROM pgstac.pgstac_settings WHERE name=_setting),'')\n);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_setting_bool(IN _setting text, IN conf jsonb DEFAULT NULL) RETURNS boolean AS $$\nSELECT COALESCE(\n nullif(conf->>_setting, ''),\n nullif(current_setting(concat('pgstac.',_setting), TRUE),''),\n nullif((SELECT value FROM pgstac.pgstac_settings WHERE name=_setting),''),\n 'FALSE'\n)::boolean;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION base_url(conf jsonb DEFAULT NULL) RETURNS text AS $$\n SELECT COALESCE(pgstac.get_setting('base_url', conf), '.');\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION additional_properties() RETURNS boolean AS $$\n SELECT pgstac.get_setting_bool('additional_properties');\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION readonly(conf jsonb DEFAULT NULL) RETURNS boolean AS $$\n SELECT pgstac.get_setting_bool('readonly', conf);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context(conf jsonb DEFAULT NULL) RETURNS text AS $$\n SELECT pgstac.get_setting('context', conf);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context_estimated_count(conf jsonb DEFAULT NULL) RETURNS int AS $$\n SELECT pgstac.get_setting('context_estimated_count', conf)::int;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_estimated_cost();\nCREATE OR REPLACE FUNCTION context_estimated_cost(conf jsonb DEFAULT NULL) RETURNS float AS $$\n SELECT pgstac.get_setting('context_estimated_cost', conf)::float;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_stats_ttl();\nCREATE OR REPLACE FUNCTION context_stats_ttl(conf jsonb DEFAULT NULL) RETURNS interval AS $$\n SELECT pgstac.get_setting('context_stats_ttl', conf)::interval;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION t2s(text) RETURNS text AS $$\n SELECT extract(epoch FROM $1::interval)::text || ' s';\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE STRICT;\n\nCREATE OR REPLACE FUNCTION age_ms(a timestamptz, b timestamptz DEFAULT clock_timestamp()) RETURNS float AS $$\n SELECT abs(extract(epoch from age(a,b)) * 1000);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION queue_timeout() RETURNS interval AS $$\n SELECT t2s(coalesce(\n get_setting('queue_timeout'),\n '1h'\n ))::interval;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION notice(VARIADIC text[]) RETURNS boolean AS $$\nDECLARE\ndebug boolean := current_setting('pgstac.debug', true);\nBEGIN\n IF debug THEN\n RAISE NOTICE 'NOTICE FROM FUNC: % >>>>> %', concat_ws(' | ', $1), clock_timestamp();\n RETURN TRUE;\n END IF;\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION empty_arr(ANYARRAY) RETURNS BOOLEAN AS $$\nSELECT CASE\n WHEN $1 IS NULL THEN TRUE\n WHEN cardinality($1)<1 THEN TRUE\nELSE FALSE\nEND;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_intersection(_a ANYARRAY, _b ANYARRAY) RETURNS ANYARRAY AS $$\n SELECT ARRAY ( SELECT unnest(_a) INTERSECT SELECT UNNEST(_b) );\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION array_map_ident(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_ident(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_map_literal(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_literal(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION array_reverse(anyarray) RETURNS anyarray AS $$\nSELECT ARRAY(\n SELECT $1[i]\n FROM generate_subscripts($1,1) AS s(i)\n ORDER BY i DESC\n);\n$$ LANGUAGE SQL STRICT IMMUTABLE;\n\nDROP TABLE IF EXISTS query_queue;\nCREATE TABLE query_queue (\n query text PRIMARY KEY,\n added timestamptz DEFAULT now()\n);\n\nDROP TABLE IF EXISTS query_queue_history;\nCREATE TABLE query_queue_history(\n query text,\n added timestamptz NOT NULL,\n finished timestamptz NOT NULL DEFAULT now(),\n error text\n);\n\nCREATE OR REPLACE PROCEDURE run_queued_queries() AS $$\nDECLARE\n qitem query_queue%ROWTYPE;\n timeout_ts timestamptz;\n error text;\n cnt int := 0;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n DELETE FROM query_queue WHERE query = (SELECT query FROM query_queue ORDER BY added DESC LIMIT 1 FOR UPDATE SKIP LOCKED) RETURNING * INTO qitem;\n IF NOT FOUND THEN\n EXIT;\n END IF;\n cnt := cnt + 1;\n BEGIN\n RAISE NOTICE 'RUNNING QUERY: %', qitem.query;\n EXECUTE qitem.query;\n EXCEPTION WHEN others THEN\n error := format('%s | %s', SQLERRM, SQLSTATE);\n END;\n INSERT INTO query_queue_history (query, added, finished, error)\n VALUES (qitem.query, qitem.added, clock_timestamp(), error);\n COMMIT;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION run_queued_queries_intransaction() RETURNS int AS $$\nDECLARE\n qitem query_queue%ROWTYPE;\n timeout_ts timestamptz;\n error text;\n cnt int := 0;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n DELETE FROM query_queue WHERE query = (SELECT query FROM query_queue ORDER BY added DESC LIMIT 1 FOR UPDATE SKIP LOCKED) RETURNING * INTO qitem;\n IF NOT FOUND THEN\n RETURN cnt;\n END IF;\n cnt := cnt + 1;\n BEGIN\n qitem.query := regexp_replace(qitem.query, 'CONCURRENTLY', '');\n RAISE NOTICE 'RUNNING QUERY: %', qitem.query;\n\n EXECUTE qitem.query;\n EXCEPTION WHEN others THEN\n error := format('%s | %s', SQLERRM, SQLSTATE);\n RAISE WARNING '%', error;\n END;\n INSERT INTO query_queue_history (query, added, finished, error)\n VALUES (qitem.query, qitem.added, clock_timestamp(), error);\n END LOOP;\n RETURN cnt;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION run_or_queue(query text) RETURNS VOID AS $$\nDECLARE\n use_queue text := COALESCE(get_setting('use_queue'), 'FALSE')::boolean;\nBEGIN\n IF get_setting_bool('debug') THEN\n RAISE NOTICE '%', query;\n END IF;\n IF use_queue THEN\n INSERT INTO query_queue (query) VALUES (query) ON CONFLICT DO NOTHING;\n ELSE\n EXECUTE query;\n END IF;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n\nDROP FUNCTION IF EXISTS check_pgstac_settings;\nCREATE OR REPLACE FUNCTION check_pgstac_settings(_sysmem text DEFAULT NULL) RETURNS VOID AS $$\nDECLARE\n settingval text;\n sysmem bigint := pg_size_bytes(_sysmem);\n effective_cache_size bigint := pg_size_bytes(current_setting('effective_cache_size', TRUE));\n shared_buffers bigint := pg_size_bytes(current_setting('shared_buffers', TRUE));\n work_mem bigint := pg_size_bytes(current_setting('work_mem', TRUE));\n max_connections int := current_setting('max_connections', TRUE);\n maintenance_work_mem bigint := pg_size_bytes(current_setting('maintenance_work_mem', TRUE));\n seq_page_cost float := current_setting('seq_page_cost', TRUE);\n random_page_cost float := current_setting('random_page_cost', TRUE);\n temp_buffers bigint := pg_size_bytes(current_setting('temp_buffers', TRUE));\n r record;\nBEGIN\n IF _sysmem IS NULL THEN\n RAISE NOTICE 'Call function with the size of your system memory `SELECT check_pgstac_settings(''4GB'')` to get pg system setting recommendations.';\n ELSE\n IF effective_cache_size < (sysmem * 0.5) THEN\n RAISE WARNING 'effective_cache_size of % is set low for a system with %. Recomended value between % and %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.5), pg_size_pretty(sysmem * 0.75);\n ELSIF effective_cache_size > (sysmem * 0.75) THEN\n RAISE WARNING 'effective_cache_size of % is set high for a system with %. Recomended value between % and %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.5), pg_size_pretty(sysmem * 0.75);\n ELSE\n RAISE NOTICE 'effective_cache_size of % is set appropriately for a system with %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem);\n END IF;\n\n IF shared_buffers < (sysmem * 0.2) THEN\n RAISE WARNING 'shared_buffers of % is set low for a system with %. Recomended value between % and %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.2), pg_size_pretty(sysmem * 0.3);\n ELSIF shared_buffers > (sysmem * 0.3) THEN\n RAISE WARNING 'shared_buffers of % is set high for a system with %. Recomended value between % and %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.2), pg_size_pretty(sysmem * 0.3);\n ELSE\n RAISE NOTICE 'shared_buffers of % is set appropriately for a system with %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem);\n END IF;\n shared_buffers = sysmem * 0.3;\n IF maintenance_work_mem < (sysmem * 0.2) THEN\n RAISE WARNING 'maintenance_work_mem of % is set low for shared_buffers of %. Recomended value between % and %', pg_size_pretty(maintenance_work_mem), pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers * 0.2), pg_size_pretty(shared_buffers * 0.3);\n ELSIF maintenance_work_mem > (shared_buffers * 0.3) THEN\n RAISE WARNING 'maintenance_work_mem of % is set high for shared_buffers of %. Recomended value between % and %', pg_size_pretty(maintenance_work_mem), pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers * 0.2), pg_size_pretty(shared_buffers * 0.3);\n ELSE\n RAISE NOTICE 'maintenance_work_mem of % is set appropriately for shared_buffers of %', pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers);\n END IF;\n\n IF work_mem * max_connections > shared_buffers THEN\n RAISE WARNING 'work_mem setting of % is set high for % max_connections please reduce work_mem to % or decrease max_connections to %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers/max_connections), floor(shared_buffers/work_mem);\n ELSIF work_mem * max_connections < (shared_buffers * 0.75) THEN\n RAISE WARNING 'work_mem setting of % is set low for % max_connections you may consider raising work_mem to % or increasing max_connections to %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers/max_connections), floor(shared_buffers/work_mem);\n ELSE\n RAISE NOTICE 'work_mem setting of % and max_connections of % are adequate for shared_buffers of %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers);\n END IF;\n\n IF random_page_cost / seq_page_cost != 1.1 THEN\n RAISE WARNING 'random_page_cost (%) /seq_page_cost (%) should be set to 1.1 for SSD. Change random_page_cost to %', random_page_cost, seq_page_cost, 1.1 * seq_page_cost;\n ELSE\n RAISE NOTICE 'random_page_cost and seq_page_cost set appropriately for SSD';\n END IF;\n\n IF temp_buffers < greatest(pg_size_bytes('128MB'),(maintenance_work_mem / 2)) THEN\n RAISE WARNING 'pgstac makes heavy use of temp tables, consider raising temp_buffers from % to %', pg_size_pretty(temp_buffers), greatest('128MB', pg_size_pretty((shared_buffers / 16)));\n END IF;\n END IF;\n\n RAISE NOTICE 'VALUES FOR PGSTAC VARIABLES';\n RAISE NOTICE 'These can be set either as GUC system variables or by setting in the pgstac_settings table.';\n\n FOR r IN SELECT name, get_setting(name) as setting, CASE WHEN current_setting(concat('pgstac.',name), TRUE) IS NOT NULL THEN concat('pgstac.',name, ' GUC') WHEN value IS NOT NULL THEN 'pgstac_settings table' ELSE 'Not Set' END as loc FROM pgstac_settings LOOP\n RAISE NOTICE '% is set to % from the %', r.name, r.setting, r.loc;\n END LOOP;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pg_cron';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider intalling pg_cron which can be used to automate tasks';\n ELSE\n RAISE NOTICE 'pg_cron % is installed', settingval;\n END IF;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pgstattuple';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider installing the pgstattuple extension which can be used to help maintain tables and indexes.';\n ELSE\n RAISE NOTICE 'pgstattuple % is installed', settingval;\n END IF;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pg_stat_statements';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider installing the pg_stat_statements extension which is very helpful for tracking the types of queries on the system';\n ELSE\n RAISE NOTICE 'pg_stat_statements % is installed', settingval;\n IF current_setting('pg_stat_statements.track_statements', TRUE) IS DISTINCT FROM 'all' THEN\n RAISE WARNING 'SET pg_stat_statements.track_statements TO ''all''; --In order to track statements within functions.';\n END IF;\n END IF;\n\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac, public SET CLIENT_MIN_MESSAGES TO NOTICE;\nCREATE OR REPLACE FUNCTION to_int(jsonb) RETURNS int AS $$\n SELECT floor(($1->>0)::float)::int;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION to_float(jsonb) RETURNS float AS $$\n SELECT ($1->>0)::float;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION to_tstz(jsonb) RETURNS timestamptz AS $$\n SELECT ($1->>0)::timestamptz;\n$$ LANGUAGE SQL IMMUTABLE STRICT SET TIME ZONE 'UTC' COST 5000 PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION to_text(jsonb) RETURNS text AS $$\n SELECT CASE WHEN jsonb_typeof($1) IN ('array','object') THEN $1::text ELSE $1->>0 END;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION to_text_array(jsonb) RETURNS text[] AS $$\n SELECT\n CASE jsonb_typeof($1)\n WHEN 'array' THEN ARRAY(SELECT jsonb_array_elements_text($1))\n ELSE ARRAY[$1->>0]\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION bbox_geom(_bbox jsonb) RETURNS geometry AS $$\nSELECT CASE jsonb_array_length(_bbox)\n WHEN 4 THEN\n ST_SetSRID(ST_MakeEnvelope(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float,\n (_bbox->>3)::float\n ),4326)\n WHEN 6 THEN\n ST_SetSRID(ST_3DMakeBox(\n ST_MakePoint(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float\n ),\n ST_MakePoint(\n (_bbox->>3)::float,\n (_bbox->>4)::float,\n (_bbox->>5)::float\n )\n ),4326)\n ELSE null END;\n;\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION geom_bbox(_geom geometry) RETURNS jsonb AS $$\n SELECT jsonb_build_array(\n st_xmin(_geom),\n st_ymin(_geom),\n st_xmax(_geom),\n st_ymax(_geom)\n );\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION flip_jsonb_array(j jsonb) RETURNS jsonb AS $$\n SELECT jsonb_agg(value) FROM (SELECT value FROM jsonb_array_elements(j) WITH ORDINALITY ORDER BY ordinality DESC) as t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION explode_dotpaths(j jsonb) RETURNS SETOF text[] AS $$\n SELECT string_to_array(p, '.') as e FROM jsonb_array_elements_text(j) p;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION explode_dotpaths_recurse(IN j jsonb) RETURNS SETOF text[] AS $$\n WITH RECURSIVE t AS (\n SELECT e FROM explode_dotpaths(j) e\n UNION ALL\n SELECT e[1:cardinality(e)-1]\n FROM t\n WHERE cardinality(e)>1\n ) SELECT e FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION jsonb_set_nested(j jsonb, path text[], val jsonb) RETURNS jsonb AS $$\nDECLARE\nBEGIN\n IF cardinality(path) > 1 THEN\n FOR i IN 1..(cardinality(path)-1) LOOP\n IF j #> path[:i] IS NULL THEN\n j := jsonb_set_lax(j, path[:i], '{}', TRUE);\n END IF;\n END LOOP;\n END IF;\n RETURN jsonb_set_lax(j, path, val, true);\n\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\n\n\nCREATE OR REPLACE FUNCTION jsonb_include(j jsonb, f jsonb) RETURNS jsonb AS $$\nDECLARE\n includes jsonb := f-> 'include';\n outj jsonb := '{}'::jsonb;\n path text[];\nBEGIN\n IF\n includes IS NULL\n OR jsonb_array_length(includes) = 0\n THEN\n RETURN j;\n ELSE\n includes := includes || (\n CASE WHEN j ? 'collection' THEN\n '[\"id\",\"collection\"]'\n ELSE\n '[\"id\"]'\n END)::jsonb;\n FOR path IN SELECT explode_dotpaths(includes) LOOP\n outj := jsonb_set_nested(outj, path, j #> path);\n END LOOP;\n END IF;\n RETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_exclude(j jsonb, f jsonb) RETURNS jsonb AS $$\nDECLARE\n excludes jsonb := f-> 'exclude';\n outj jsonb := j;\n path text[];\nBEGIN\n IF\n excludes IS NULL\n OR jsonb_array_length(excludes) = 0\n THEN\n RETURN j;\n ELSE\n FOR path IN SELECT explode_dotpaths(excludes) LOOP\n outj := outj #- path;\n END LOOP;\n END IF;\n RETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_fields(j jsonb, f jsonb DEFAULT '{\"fields\":[]}') RETURNS jsonb AS $$\n SELECT jsonb_exclude(jsonb_include(j, f), f);\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION merge_jsonb(_a jsonb, _b jsonb) RETURNS jsonb AS $$\n SELECT\n CASE\n WHEN _a = '\"𒍟※\"'::jsonb THEN NULL\n WHEN _a IS NULL OR jsonb_typeof(_a) = 'null' THEN _b\n WHEN jsonb_typeof(_a) = 'object' AND jsonb_typeof(_b) = 'object' THEN\n (\n SELECT\n jsonb_strip_nulls(\n jsonb_object_agg(\n key,\n merge_jsonb(a.value, b.value)\n )\n )\n FROM\n jsonb_each(coalesce(_a,'{}'::jsonb)) as a\n FULL JOIN\n jsonb_each(coalesce(_b,'{}'::jsonb)) as b\n USING (key)\n )\n WHEN\n jsonb_typeof(_a) = 'array'\n AND jsonb_typeof(_b) = 'array'\n AND jsonb_array_length(_a) = jsonb_array_length(_b)\n THEN\n (\n SELECT jsonb_agg(m) FROM\n ( SELECT\n merge_jsonb(\n jsonb_array_elements(_a),\n jsonb_array_elements(_b)\n ) as m\n ) as l\n )\n ELSE _a\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION strip_jsonb(_a jsonb, _b jsonb) RETURNS jsonb AS $$\n SELECT\n CASE\n\n WHEN (_a IS NULL OR jsonb_typeof(_a) = 'null') AND _b IS NOT NULL AND jsonb_typeof(_b) != 'null' THEN '\"𒍟※\"'::jsonb\n WHEN _b IS NULL OR jsonb_typeof(_a) = 'null' THEN _a\n WHEN _a = _b AND jsonb_typeof(_a) = 'object' THEN '{}'::jsonb\n WHEN _a = _b THEN NULL\n WHEN jsonb_typeof(_a) = 'object' AND jsonb_typeof(_b) = 'object' THEN\n (\n SELECT\n jsonb_strip_nulls(\n jsonb_object_agg(\n key,\n strip_jsonb(a.value, b.value)\n )\n )\n FROM\n jsonb_each(_a) as a\n FULL JOIN\n jsonb_each(_b) as b\n USING (key)\n )\n WHEN\n jsonb_typeof(_a) = 'array'\n AND jsonb_typeof(_b) = 'array'\n AND jsonb_array_length(_a) = jsonb_array_length(_b)\n THEN\n (\n SELECT jsonb_agg(m) FROM\n ( SELECT\n strip_jsonb(\n jsonb_array_elements(_a),\n jsonb_array_elements(_b)\n ) as m\n ) as l\n )\n ELSE _a\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION nullif_jsonbnullempty(j jsonb) RETURNS jsonb AS $$\n SELECT nullif(nullif(nullif(j,'null'::jsonb),'{}'::jsonb),'[]'::jsonb);\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION jsonb_array_unique(j jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(jsonb_agg(DISTINCT a)) v FROM jsonb_array_elements(j) a;\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_concat_ignorenull(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT coalesce(a,'[]'::jsonb) || coalesce(b,'[]'::jsonb);\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_least(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(least(nullif_jsonbnullempty(a), nullif_jsonbnullempty(b)));\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_greatest(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(greatest(a, b));\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION first_notnull_sfunc(anyelement, anyelement) RETURNS anyelement AS $$\n SELECT COALESCE($1,$2);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE AGGREGATE first_notnull(anyelement)(\n SFUNC = first_notnull_sfunc,\n STYPE = anyelement\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_array_unique_merge(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_concat_ignorenull,\n FINALFUNC = jsonb_array_unique\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_min(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_least\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_max(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_greatest\n);\n/* looks for a geometry in a stac item first from geometry and falling back to bbox */\nCREATE OR REPLACE FUNCTION stac_geom(value jsonb) RETURNS geometry AS $$\nSELECT\n CASE\n WHEN value ? 'intersects' THEN\n ST_GeomFromGeoJSON(value->>'intersects')\n WHEN value ? 'geometry' THEN\n ST_GeomFromGeoJSON(value->>'geometry')\n WHEN value ? 'bbox' THEN\n pgstac.bbox_geom(value->'bbox')\n ELSE NULL\n END as geometry\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION stac_daterange(\n value jsonb\n) RETURNS tstzrange AS $$\nDECLARE\n props jsonb := value;\n dt timestamptz;\n edt timestamptz;\nBEGIN\n IF props ? 'properties' THEN\n props := props->'properties';\n END IF;\n IF\n props ? 'start_datetime'\n AND props->>'start_datetime' IS NOT NULL\n AND props ? 'end_datetime'\n AND props->>'end_datetime' IS NOT NULL\n THEN\n dt := props->>'start_datetime';\n edt := props->>'end_datetime';\n IF dt > edt THEN\n RAISE EXCEPTION 'start_datetime must be < end_datetime';\n END IF;\n ELSE\n dt := props->>'datetime';\n edt := props->>'datetime';\n END IF;\n IF dt is NULL OR edt IS NULL THEN\n RAISE NOTICE 'DT: %, EDT: %', dt, edt;\n RAISE EXCEPTION 'Either datetime (%) or both start_datetime (%) and end_datetime (%) must be set.', props->>'datetime',props->>'start_datetime',props->>'end_datetime';\n END IF;\n RETURN tstzrange(dt, edt, '[]');\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_datetime(value jsonb) RETURNS timestamptz AS $$\n SELECT lower(stac_daterange(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_end_datetime(value jsonb) RETURNS timestamptz AS $$\n SELECT upper(stac_daterange(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE TABLE IF NOT EXISTS stac_extensions(\n url text PRIMARY KEY,\n content jsonb\n);\nCREATE OR REPLACE FUNCTION collection_base_item(content jsonb) RETURNS jsonb AS $$\n SELECT jsonb_build_object(\n 'type', 'Feature',\n 'stac_version', content->'stac_version',\n 'assets', content->'item_assets',\n 'collection', content->'id'\n );\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE TABLE IF NOT EXISTS collections (\n key bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,\n id text GENERATED ALWAYS AS (content->>'id') STORED UNIQUE NOT NULL,\n content JSONB NOT NULL,\n base_item jsonb GENERATED ALWAYS AS (pgstac.collection_base_item(content)) STORED,\n geometry geometry GENERATED ALWAYS AS (pgstac.collection_geom(content)) STORED,\n datetime timestamptz GENERATED ALWAYS AS (pgstac.collection_datetime(content)) STORED,\n end_datetime timestamptz GENERATED ALWAYS AS (pgstac.collection_enddatetime(content)) STORED,\n private jsonb,\n partition_trunc text CHECK (partition_trunc IN ('year', 'month'))\n);\n\nCREATE OR REPLACE FUNCTION collection_base_item(cid text) RETURNS jsonb AS $$\n SELECT pgstac.collection_base_item(content) FROM pgstac.collections WHERE id = cid LIMIT 1;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION create_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION update_collection(data jsonb) RETURNS VOID AS $$\nDECLARE\n out collections%ROWTYPE;\nBEGIN\n UPDATE collections SET content=data WHERE id = data->>'id' RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ON CONFLICT (id) DO\n UPDATE\n SET content=EXCLUDED.content\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION delete_collection(_id text) RETURNS VOID AS $$\nDECLARE\n out collections%ROWTYPE;\nBEGIN\n DELETE FROM collections WHERE id = _id RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION get_collection(id text) RETURNS jsonb AS $$\n SELECT content FROM collections\n WHERE id=$1\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION all_collections() RETURNS jsonb AS $$\n SELECT coalesce(jsonb_agg(content), '[]'::jsonb) FROM collections;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION collection_delete_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n collection_base_partition text := concat('_items_', OLD.key);\nBEGIN\n EXECUTE format($q$\n DELETE FROM partition_stats WHERE partition IN (\n SELECT partition FROM partition_sys_meta\n WHERE collection=%L\n );\n DROP TABLE IF EXISTS %I CASCADE;\n $q$,\n OLD.id,\n collection_base_partition\n );\n RETURN OLD;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER collection_delete_trigger BEFORE DELETE ON collections\nFOR EACH ROW EXECUTE FUNCTION collection_delete_trigger_func();\nCREATE OR REPLACE FUNCTION queryable_signature(n text, c text[]) RETURNS text AS $$\n SELECT concat(n, c);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TABLE queryables (\n id bigint GENERATED ALWAYS AS identity PRIMARY KEY,\n name text NOT NULL,\n collection_ids text[], -- used to determine what partitions to create indexes on\n definition jsonb,\n property_path text,\n property_wrapper text,\n property_index_type text\n);\nCREATE INDEX queryables_name_idx ON queryables (name);\nCREATE INDEX queryables_collection_idx ON queryables USING GIN (collection_ids);\nCREATE INDEX queryables_property_wrapper_idx ON queryables (property_wrapper);\n\nCREATE OR REPLACE FUNCTION pgstac.queryables_constraint_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n allcollections text[];\nBEGIN\n RAISE NOTICE 'Making sure that name/collection is unique for queryables %', NEW;\n IF NEW.collection_ids IS NOT NULL THEN\n IF EXISTS (\n SELECT 1\n FROM unnest(NEW.collection_ids) c\n LEFT JOIN\n collections\n ON (collections.id = c)\n WHERE collections.id IS NULL\n ) THEN\n RAISE foreign_key_violation USING MESSAGE = format(\n 'One or more collections in %s do not exist.', NEW.collection_ids\n );\n RETURN NULL;\n END IF;\n END IF;\n IF TG_OP = 'INSERT' THEN\n IF EXISTS (\n SELECT 1 FROM queryables q\n WHERE\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n )\n ) THEN\n RAISE unique_violation USING MESSAGE = format(\n 'There is already a queryable for %s for a collection in %s: %s',\n NEW.name,\n NEW.collection_ids,\n\t\t\t\t(SELECT json_agg(row_to_json(q)) FROM queryables q WHERE\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n ))\n );\n RETURN NULL;\n END IF;\n END IF;\n IF TG_OP = 'UPDATE' THEN\n IF EXISTS (\n SELECT 1 FROM queryables q\n WHERE\n q.id != NEW.id\n AND\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n )\n ) THEN\n RAISE unique_violation\n USING MESSAGE = format(\n 'There is already a queryable for %s for a collection in %s',\n NEW.name,\n NEW.collection_ids\n );\n RETURN NULL;\n END IF;\n END IF;\n\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER queryables_constraint_insert_trigger\nBEFORE INSERT ON queryables\nFOR EACH ROW EXECUTE PROCEDURE queryables_constraint_triggerfunc();\n\nCREATE TRIGGER queryables_constraint_update_trigger\nBEFORE UPDATE ON queryables\nFOR EACH ROW\nWHEN (NEW.name = OLD.name AND NEW.collection_ids IS DISTINCT FROM OLD.collection_ids)\nEXECUTE PROCEDURE queryables_constraint_triggerfunc();\n\n\nCREATE OR REPLACE FUNCTION array_to_path(arr text[]) RETURNS text AS $$\n SELECT string_agg(\n quote_literal(v),\n '->'\n ) FROM unnest(arr) v;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\n\n\nCREATE OR REPLACE FUNCTION queryable(\n IN dotpath text,\n OUT path text,\n OUT expression text,\n OUT wrapper text,\n OUT nulled_wrapper text\n) AS $$\nDECLARE\n q RECORD;\n path_elements text[];\nBEGIN\n dotpath := replace(dotpath, 'properties.', '');\n IF dotpath = 'start_datetime' THEN\n dotpath := 'datetime';\n END IF;\n IF dotpath IN ('id', 'geometry', 'datetime', 'end_datetime', 'collection') THEN\n path := dotpath;\n expression := dotpath;\n wrapper := NULL;\n RETURN;\n END IF;\n\n SELECT * INTO q FROM queryables\n WHERE\n name=dotpath\n OR name = 'properties.' || dotpath\n OR name = replace(dotpath, 'properties.', '')\n ;\n IF q.property_wrapper IS NULL THEN\n IF q.definition->>'type' = 'number' THEN\n wrapper := 'to_float';\n nulled_wrapper := wrapper;\n ELSIF q.definition->>'format' = 'date-time' THEN\n wrapper := 'to_tstz';\n nulled_wrapper := wrapper;\n ELSE\n nulled_wrapper := NULL;\n wrapper := 'to_text';\n END IF;\n ELSE\n wrapper := q.property_wrapper;\n nulled_wrapper := wrapper;\n END IF;\n IF q.property_path IS NOT NULL THEN\n path := q.property_path;\n ELSE\n path_elements := string_to_array(dotpath, '.');\n IF path_elements[1] IN ('links', 'assets', 'stac_version', 'stac_extensions') THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSIF path_elements[1] = 'properties' THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSE\n path := format($F$content->'properties'->%s$F$, array_to_path(path_elements));\n END IF;\n END IF;\n expression := format('%I(%s)', wrapper, path);\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\nCREATE OR REPLACE FUNCTION unnest_collection(collection_ids text[] DEFAULT NULL) RETURNS SETOF text AS $$\n DECLARE\n BEGIN\n IF collection_ids IS NULL THEN\n RETURN QUERY SELECT id FROM collections;\n END IF;\n RETURN QUERY SELECT unnest(collection_ids);\n END;\n$$ LANGUAGE PLPGSQL STABLE;\n\nCREATE OR REPLACE FUNCTION normalize_indexdef(def text) RETURNS text AS $$\nDECLARE\nBEGIN\n def := btrim(def, ' \\n\\t');\n\tdef := regexp_replace(def, '^CREATE (UNIQUE )?INDEX ([^ ]* )?ON (ONLY )?([^ ]* )?', '', 'i');\n RETURN def;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION indexdef(q queryables) RETURNS text AS $$\n DECLARE\n out text;\n BEGIN\n IF q.name = 'id' THEN\n out := 'CREATE UNIQUE INDEX ON %I USING btree (id)';\n ELSIF q.name = 'datetime' THEN\n out := 'CREATE INDEX ON %I USING btree (datetime DESC, end_datetime)';\n ELSIF q.name = 'geometry' THEN\n out := 'CREATE INDEX ON %I USING gist (geometry)';\n ELSE\n out := format($q$CREATE INDEX ON %%I USING %s (%s(((content -> 'properties'::text) -> %L::text)))$q$,\n lower(COALESCE(q.property_index_type, 'BTREE')),\n lower(COALESCE(q.property_wrapper, 'to_text')),\n q.name\n );\n END IF;\n RETURN btrim(out, ' \\n\\t');\n END;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nDROP VIEW IF EXISTS pgstac_indexes;\nCREATE VIEW pgstac_indexes AS\nSELECT\n i.schemaname,\n i.tablename,\n i.indexname,\n regexp_replace(btrim(replace(replace(indexdef, i.indexname, ''),'pgstac.',''),' \\t\\n'), '[ ]+', ' ', 'g') as idx,\n COALESCE(\n (regexp_match(indexdef, '\\(([a-zA-Z]+)\\)'))[1],\n (regexp_match(indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_-]+)''::text'))[1],\n CASE WHEN indexdef ~* '\\(datetime desc, end_datetime\\)' THEN 'datetime' ELSE NULL END\n ) AS field,\n pg_table_size(i.indexname::text) as index_size,\n pg_size_pretty(pg_table_size(i.indexname::text)) as index_size_pretty\nFROM\n pg_indexes i\nWHERE i.schemaname='pgstac' and i.tablename ~ '_items_' AND indexdef !~* ' only ';\n\nDROP VIEW IF EXISTS pgstac_index_stats;\nCREATE VIEW pgstac_indexes_stats AS\nSELECT\n i.schemaname,\n i.tablename,\n i.indexname,\n indexdef,\n COALESCE(\n (regexp_match(indexdef, '\\(([a-zA-Z]+)\\)'))[1],\n (regexp_match(indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_]+)''::text'))[1],\n CASE WHEN indexdef ~* '\\(datetime desc, end_datetime\\)' THEN 'datetime_end_datetime' ELSE NULL END\n ) AS field,\n pg_table_size(i.indexname::text) as index_size,\n pg_size_pretty(pg_table_size(i.indexname::text)) as index_size_pretty,\n n_distinct,\n most_common_vals::text::text[],\n most_common_freqs::text::text[],\n histogram_bounds::text::text[],\n correlation\nFROM\n pg_indexes i\n LEFT JOIN pg_stats s ON (s.tablename = i.indexname)\nWHERE i.schemaname='pgstac' and i.tablename ~ '_items_';\n\nCREATE OR REPLACE FUNCTION queryable_indexes(\n IN treeroot text DEFAULT 'items',\n IN changes boolean DEFAULT FALSE,\n OUT collection text,\n OUT partition text,\n OUT field text,\n OUT indexname text,\n OUT existing_idx text,\n OUT queryable_idx text\n) RETURNS SETOF RECORD AS $$\nWITH p AS (\n SELECT\n relid::text as partition,\n replace(replace(\n CASE\n WHEN parentrelid::regclass::text='items' THEN pg_get_expr(c.relpartbound, c.oid)\n ELSE pg_get_expr(parent.relpartbound, parent.oid)\n END,\n 'FOR VALUES IN (''',''), ''')',\n ''\n ) AS collection\n FROM pg_partition_tree(treeroot)\n JOIN pg_class c ON (relid::regclass = c.oid)\n JOIN pg_class parent ON (parentrelid::regclass = parent.oid AND isleaf)\n ), i AS (\n SELECT\n partition,\n indexname,\n regexp_replace(btrim(replace(replace(indexdef, indexname, ''),'pgstac.',''),' \\t\\n'), '[ ]+', ' ', 'g') as iidx,\n COALESCE(\n (regexp_match(indexdef, '\\(([a-zA-Z]+)\\)'))[1],\n (regexp_match(indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_-]+)''::text'))[1],\n CASE WHEN indexdef ~* '\\(datetime desc, end_datetime\\)' THEN 'datetime' ELSE NULL END\n ) AS field\n FROM\n pg_indexes\n JOIN p ON (tablename=partition)\n ), q AS (\n SELECT\n name AS field,\n collection,\n partition,\n format(indexdef(queryables), partition) as qidx\n FROM queryables, unnest_collection(queryables.collection_ids) collection\n JOIN p USING (collection)\n WHERE property_index_type IS NOT NULL OR name IN ('datetime','geometry','id')\n )\n SELECT\n collection,\n partition,\n field,\n indexname,\n iidx as existing_idx,\n qidx as queryable_idx\n FROM i FULL JOIN q USING (field, partition)\n WHERE CASE WHEN changes THEN lower(iidx) IS DISTINCT FROM lower(qidx) ELSE TRUE END;\n;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION maintain_index(\n indexname text,\n queryable_idx text,\n dropindexes boolean DEFAULT FALSE,\n rebuildindexes boolean DEFAULT FALSE,\n idxconcurrently boolean DEFAULT FALSE\n) RETURNS VOID AS $$\nDECLARE\nBEGIN\n IF indexname IS NOT NULL THEN\n IF dropindexes OR queryable_idx IS NOT NULL THEN\n EXECUTE format('DROP INDEX IF EXISTS %I;', indexname);\n ELSIF rebuildindexes THEN\n IF idxconcurrently THEN\n EXECUTE format('REINDEX INDEX CONCURRENTLY %I;', indexname);\n ELSE\n EXECUTE format('REINDEX INDEX CONCURRENTLY %I;', indexname);\n END IF;\n END IF;\n END IF;\n IF queryable_idx IS NOT NULL THEN\n IF idxconcurrently THEN\n EXECUTE replace(queryable_idx, 'INDEX', 'INDEX CONCURRENTLY');\n ELSE EXECUTE queryable_idx;\n END IF;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\n\nset check_function_bodies to off;\nCREATE OR REPLACE FUNCTION maintain_partition_queries(\n part text DEFAULT 'items',\n dropindexes boolean DEFAULT FALSE,\n rebuildindexes boolean DEFAULT FALSE,\n idxconcurrently boolean DEFAULT FALSE\n) RETURNS SETOF text AS $$\nDECLARE\n rec record;\n q text;\nBEGIN\n FOR rec IN (\n SELECT * FROM queryable_indexes(part,true)\n ) LOOP\n q := format(\n 'SELECT maintain_index(\n %L,%L,%L,%L,%L\n );',\n rec.indexname,\n rec.queryable_idx,\n dropindexes,\n rebuildindexes,\n idxconcurrently\n );\n RAISE NOTICE 'Q: %', q;\n RETURN NEXT q;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION maintain_partitions(\n part text DEFAULT 'items',\n dropindexes boolean DEFAULT FALSE,\n rebuildindexes boolean DEFAULT FALSE\n) RETURNS VOID AS $$\n WITH t AS (\n SELECT run_or_queue(q) FROM maintain_partition_queries(part, dropindexes, rebuildindexes) q\n ) SELECT count(*) FROM t;\n$$ LANGUAGE SQL;\n\n\nCREATE OR REPLACE FUNCTION queryables_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\nBEGIN\n PERFORM maintain_partitions();\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER queryables_trigger AFTER INSERT OR UPDATE ON queryables\nFOR EACH STATEMENT EXECUTE PROCEDURE queryables_trigger_func();\n\n\nCREATE OR REPLACE FUNCTION get_queryables(_collection_ids text[] DEFAULT NULL) RETURNS jsonb AS $$\nDECLARE\nBEGIN\n -- Build up queryables if the input contains valid collection ids or is empty\n IF EXISTS (\n SELECT 1 FROM collections\n WHERE\n _collection_ids IS NULL\n OR cardinality(_collection_ids) = 0\n OR id = ANY(_collection_ids)\n )\n THEN\n RETURN (\n WITH base AS (\n SELECT\n unnest(collection_ids) as collection_id,\n name,\n coalesce(definition, '{\"type\":\"string\"}'::jsonb) as definition\n FROM queryables\n WHERE\n _collection_ids IS NULL OR\n _collection_ids = '{}'::text[] OR\n _collection_ids && collection_ids\n UNION ALL\n SELECT null, name, coalesce(definition, '{\"type\":\"string\"}'::jsonb) as definition\n FROM queryables WHERE collection_ids IS NULL OR collection_ids = '{}'::text[]\n ), g AS (\n SELECT\n name,\n first_notnull(definition) as definition,\n jsonb_array_unique_merge(definition->'enum') as enum,\n jsonb_min(definition->'minimum') as minimum,\n jsonb_min(definition->'maxiumn') as maximum\n FROM base\n GROUP BY 1\n )\n SELECT\n jsonb_build_object(\n '$schema', 'http://json-schema.org/draft-07/schema#',\n '$id', '',\n 'type', 'object',\n 'title', 'STAC Queryables.',\n 'properties', jsonb_object_agg(\n name,\n definition\n ||\n jsonb_strip_nulls(jsonb_build_object(\n 'enum', enum,\n 'minimum', minimum,\n 'maximum', maximum\n ))\n ),\n 'additionalProperties', pgstac.additional_properties()\n )\n FROM g\n );\n ELSE\n RETURN NULL;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\nCREATE OR REPLACE FUNCTION get_queryables(_collection text DEFAULT NULL) RETURNS jsonb AS $$\n SELECT\n CASE\n WHEN _collection IS NULL THEN get_queryables(NULL::text[])\n ELSE get_queryables(ARRAY[_collection])\n END\n ;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_queryables() RETURNS jsonb AS $$\n SELECT get_queryables(NULL::text[]);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION schema_qualify_refs(url text, j jsonb) returns jsonb as $$\n SELECT regexp_replace(j::text, '\"\\$ref\": \"#', concat('\"$ref\": \"', url, '#'), 'g')::jsonb;\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE VIEW stac_extension_queryables AS\nSELECT DISTINCT key as name, schema_qualify_refs(e.url, j.value) as definition FROM stac_extensions e, jsonb_each(e.content->'definitions'->'fields'->'properties') j;\n\n\nCREATE OR REPLACE FUNCTION missing_queryables(_collection text, _tablesample float DEFAULT 5, minrows float DEFAULT 10) RETURNS TABLE(collection text, name text, definition jsonb, property_wrapper text) AS $$\nDECLARE\n q text;\n _partition text;\n explain_json json;\n psize float;\n estrows float;\nBEGIN\n SELECT format('_items_%s', key) INTO _partition FROM collections WHERE id=_collection;\n\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM %I;', _partition)\n INTO explain_json;\n psize := explain_json->0->'Plan'->'Plan Rows';\n estrows := _tablesample * .01 * psize;\n IF estrows < minrows THEN\n _tablesample := least(100,greatest(_tablesample, (estrows / psize) / 100));\n RAISE NOTICE '%', (psize / estrows) / 100;\n END IF;\n RAISE NOTICE 'Using tablesample % to find missing queryables from % % that has ~% rows estrows: %', _tablesample, _collection, _partition, psize, estrows;\n\n q := format(\n $q$\n WITH q AS (\n SELECT * FROM queryables\n WHERE\n collection_ids IS NULL\n OR %L = ANY(collection_ids)\n ), t AS (\n SELECT\n content->'properties' AS properties\n FROM\n %I\n TABLESAMPLE SYSTEM(%L)\n ), p AS (\n SELECT DISTINCT ON (key)\n key,\n value,\n s.definition\n FROM t\n JOIN LATERAL jsonb_each(properties) ON TRUE\n LEFT JOIN q ON (q.name=key)\n LEFT JOIN stac_extension_queryables s ON (s.name=key)\n WHERE q.definition IS NULL\n )\n SELECT\n %L,\n key,\n COALESCE(definition, jsonb_build_object('type',jsonb_typeof(value))) as definition,\n CASE\n WHEN definition->>'type' = 'integer' THEN 'to_int'\n WHEN COALESCE(definition->>'type', jsonb_typeof(value)) = 'number' THEN 'to_float'\n WHEN COALESCE(definition->>'type', jsonb_typeof(value)) = 'array' THEN 'to_text_array'\n ELSE 'to_text'\n END\n FROM p;\n $q$,\n _collection,\n _partition,\n _tablesample,\n _collection\n );\n RETURN QUERY EXECUTE q;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION missing_queryables(_tablesample float DEFAULT 5) RETURNS TABLE(collection_ids text[], name text, definition jsonb, property_wrapper text) AS $$\n SELECT\n array_agg(collection),\n name,\n definition,\n property_wrapper\n FROM\n collections\n JOIN LATERAL\n missing_queryables(id, _tablesample) c\n ON TRUE\n GROUP BY\n 2,3,4\n ORDER BY 2,1\n ;\n$$ LANGUAGE SQL;\nCREATE OR REPLACE FUNCTION parse_dtrange(\n _indate jsonb,\n relative_base timestamptz DEFAULT date_trunc('hour', CURRENT_TIMESTAMP)\n) RETURNS tstzrange AS $$\nDECLARE\n timestrs text[];\n s timestamptz;\n e timestamptz;\nBEGIN\n timestrs :=\n CASE\n WHEN _indate ? 'timestamp' THEN\n ARRAY[_indate->>'timestamp']\n WHEN _indate ? 'interval' THEN\n to_text_array(_indate->'interval')\n WHEN jsonb_typeof(_indate) = 'array' THEN\n to_text_array(_indate)\n ELSE\n regexp_split_to_array(\n _indate->>0,\n '/'\n )\n END;\n RAISE NOTICE 'TIMESTRS %', timestrs;\n IF cardinality(timestrs) = 1 THEN\n IF timestrs[1] ILIKE 'P%' THEN\n RETURN tstzrange(relative_base - upper(timestrs[1])::interval, relative_base, '[)');\n END IF;\n s := timestrs[1]::timestamptz;\n RETURN tstzrange(s, s, '[]');\n END IF;\n\n IF cardinality(timestrs) != 2 THEN\n RAISE EXCEPTION 'Timestamp cannot have more than 2 values';\n END IF;\n\n IF timestrs[1] = '..' OR timestrs[1] = '' THEN\n s := '-infinity'::timestamptz;\n e := timestrs[2]::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] = '..' OR timestrs[2] = '' THEN\n s := timestrs[1]::timestamptz;\n e := 'infinity'::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[1] ILIKE 'P%' AND timestrs[2] NOT ILIKE 'P%' THEN\n e := timestrs[2]::timestamptz;\n s := e - upper(timestrs[1])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] ILIKE 'P%' AND timestrs[1] NOT ILIKE 'P%' THEN\n s := timestrs[1]::timestamptz;\n e := s + upper(timestrs[2])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n s := timestrs[1]::timestamptz;\n e := timestrs[2]::timestamptz;\n\n RETURN tstzrange(s,e,'[)');\n\n RETURN NULL;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT PARALLEL SAFE SET TIME ZONE 'UTC';\n\nCREATE OR REPLACE FUNCTION parse_dtrange(\n _indate text,\n relative_base timestamptz DEFAULT CURRENT_TIMESTAMP\n) RETURNS tstzrange AS $$\n SELECT parse_dtrange(to_jsonb(_indate), relative_base);\n$$ LANGUAGE SQL STABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION temporal_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\n ll text := 'datetime';\n lh text := 'end_datetime';\n rrange tstzrange;\n rl text;\n rh text;\n outq text;\nBEGIN\n rrange := parse_dtrange(args->1);\n RAISE NOTICE 'Constructing temporal query OP: %, ARGS: %, RRANGE: %', op, args, rrange;\n op := lower(op);\n rl := format('%L::timestamptz', lower(rrange));\n rh := format('%L::timestamptz', upper(rrange));\n outq := CASE op\n WHEN 't_before' THEN 'lh < rl'\n WHEN 't_after' THEN 'll > rh'\n WHEN 't_meets' THEN 'lh = rl'\n WHEN 't_metby' THEN 'll = rh'\n WHEN 't_overlaps' THEN 'll < rl AND rl < lh < rh'\n WHEN 't_overlappedby' THEN 'rl < ll < rh AND lh > rh'\n WHEN 't_starts' THEN 'll = rl AND lh < rh'\n WHEN 't_startedby' THEN 'll = rl AND lh > rh'\n WHEN 't_during' THEN 'll > rl AND lh < rh'\n WHEN 't_contains' THEN 'll < rl AND lh > rh'\n WHEN 't_finishes' THEN 'll > rl AND lh = rh'\n WHEN 't_finishedby' THEN 'll < rl AND lh = rh'\n WHEN 't_equals' THEN 'll = rl AND lh = rh'\n WHEN 't_disjoint' THEN 'NOT (ll <= rh AND lh >= rl)'\n WHEN 't_intersects' THEN 'll <= rh AND lh >= rl'\n WHEN 'anyinteracts' THEN 'll <= rh AND lh >= rl'\n END;\n outq := regexp_replace(outq, '\\mll\\M', ll);\n outq := regexp_replace(outq, '\\mlh\\M', lh);\n outq := regexp_replace(outq, '\\mrl\\M', rl);\n outq := regexp_replace(outq, '\\mrh\\M', rh);\n outq := format('(%s)', outq);\n RETURN outq;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\n\n\nCREATE OR REPLACE FUNCTION spatial_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\n geom text;\n j jsonb := args->1;\nBEGIN\n op := lower(op);\n RAISE NOTICE 'Constructing spatial query OP: %, ARGS: %', op, args;\n IF op NOT IN ('s_equals','s_disjoint','s_touches','s_within','s_overlaps','s_crosses','s_intersects','intersects','s_contains') THEN\n RAISE EXCEPTION 'Spatial Operator % Not Supported', op;\n END IF;\n op := regexp_replace(op, '^s_', 'st_');\n IF op = 'intersects' THEN\n op := 'st_intersects';\n END IF;\n -- Convert geometry to WKB string\n IF j ? 'type' AND j ? 'coordinates' THEN\n geom := st_geomfromgeojson(j)::text;\n ELSIF jsonb_typeof(j) = 'array' THEN\n geom := bbox_geom(j)::text;\n END IF;\n\n RETURN format('%s(geometry, %L::geometry)', op, geom);\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION query_to_cql2(q jsonb) RETURNS jsonb AS $$\n-- Translates anything passed in through the deprecated \"query\" into equivalent CQL2\nWITH t AS (\n SELECT key as property, value as ops\n FROM jsonb_each(q)\n), t2 AS (\n SELECT property, (jsonb_each(ops)).*\n FROM t WHERE jsonb_typeof(ops) = 'object'\n UNION ALL\n SELECT property, 'eq', ops\n FROM t WHERE jsonb_typeof(ops) != 'object'\n)\nSELECT\n jsonb_strip_nulls(jsonb_build_object(\n 'op', 'and',\n 'args', jsonb_agg(\n jsonb_build_object(\n 'op', key,\n 'args', jsonb_build_array(\n jsonb_build_object('property',property),\n value\n )\n )\n )\n )\n) as qcql FROM t2\n;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\nCREATE OR REPLACE FUNCTION cql1_to_cql2(j jsonb) RETURNS jsonb AS $$\nDECLARE\n args jsonb;\n ret jsonb;\nBEGIN\n RAISE NOTICE 'CQL1_TO_CQL2: %', j;\n IF j ? 'filter' THEN\n RETURN cql1_to_cql2(j->'filter');\n END IF;\n IF j ? 'property' THEN\n RETURN j;\n END IF;\n IF jsonb_typeof(j) = 'array' THEN\n SELECT jsonb_agg(cql1_to_cql2(el)) INTO args FROM jsonb_array_elements(j) el;\n RETURN args;\n END IF;\n IF jsonb_typeof(j) = 'number' THEN\n RETURN j;\n END IF;\n IF jsonb_typeof(j) = 'string' THEN\n RETURN j;\n END IF;\n\n IF jsonb_typeof(j) = 'object' THEN\n SELECT jsonb_build_object(\n 'op', key,\n 'args', cql1_to_cql2(value)\n ) INTO ret\n FROM jsonb_each(j)\n WHERE j IS NOT NULL;\n RETURN ret;\n END IF;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE STRICT;\n\nCREATE TABLE cql2_ops (\n op text PRIMARY KEY,\n template text,\n types text[]\n);\nINSERT INTO cql2_ops (op, template, types) VALUES\n ('eq', '%s = %s', NULL),\n ('neq', '%s != %s', NULL),\n ('ne', '%s != %s', NULL),\n ('!=', '%s != %s', NULL),\n ('<>', '%s != %s', NULL),\n ('lt', '%s < %s', NULL),\n ('lte', '%s <= %s', NULL),\n ('gt', '%s > %s', NULL),\n ('gte', '%s >= %s', NULL),\n ('le', '%s <= %s', NULL),\n ('ge', '%s >= %s', NULL),\n ('=', '%s = %s', NULL),\n ('<', '%s < %s', NULL),\n ('<=', '%s <= %s', NULL),\n ('>', '%s > %s', NULL),\n ('>=', '%s >= %s', NULL),\n ('like', '%s LIKE %s', NULL),\n ('ilike', '%s ILIKE %s', NULL),\n ('+', '%s + %s', NULL),\n ('-', '%s - %s', NULL),\n ('*', '%s * %s', NULL),\n ('/', '%s / %s', NULL),\n ('not', 'NOT (%s)', NULL),\n ('between', '%s BETWEEN %s AND %s', NULL),\n ('isnull', '%s IS NULL', NULL),\n ('upper', 'upper(%s)', NULL),\n ('lower', 'lower(%s)', NULL),\n ('casei', 'upper(%s)', NULL),\n ('accenti', 'unaccent(%s)', NULL)\nON CONFLICT (op) DO UPDATE\n SET\n template = EXCLUDED.template\n;\n\n\nCREATE OR REPLACE FUNCTION cql2_query(j jsonb, wrapper text DEFAULT NULL) RETURNS text AS $$\n#variable_conflict use_variable\nDECLARE\n args jsonb := j->'args';\n arg jsonb;\n op text := lower(j->>'op');\n cql2op RECORD;\n literal text;\n _wrapper text;\n leftarg text;\n rightarg text;\n prop text;\n extra_props bool := pgstac.additional_properties();\nBEGIN\n IF j IS NULL OR (op IS NOT NULL AND args IS NULL) THEN\n RETURN NULL;\n END IF;\n RAISE NOTICE 'CQL2_QUERY: %', j;\n\n -- check if all properties are represented in the queryables\n IF NOT extra_props THEN\n FOR prop IN\n SELECT DISTINCT p->>0\n FROM jsonb_path_query(j, 'strict $.**.property') p\n WHERE p->>0 NOT IN ('id', 'datetime', 'geometry', 'end_datetime', 'collection')\n LOOP\n IF (queryable(prop)).nulled_wrapper IS NULL THEN\n RAISE EXCEPTION 'Term % is not found in queryables.', prop;\n END IF;\n END LOOP;\n END IF;\n\n IF j ? 'filter' THEN\n RETURN cql2_query(j->'filter');\n END IF;\n\n IF j ? 'upper' THEN\n RETURN cql2_query(jsonb_build_object('op', 'upper', 'args', j->'upper'));\n END IF;\n\n IF j ? 'lower' THEN\n RETURN cql2_query(jsonb_build_object('op', 'lower', 'args', j->'lower'));\n END IF;\n\n -- Temporal Query\n IF op ilike 't_%' or op = 'anyinteracts' THEN\n RETURN temporal_op_query(op, args);\n END IF;\n\n -- If property is a timestamp convert it to text to use with\n -- general operators\n IF j ? 'timestamp' THEN\n RETURN format('%L::timestamptz', to_tstz(j->'timestamp'));\n END IF;\n IF j ? 'interval' THEN\n RAISE EXCEPTION 'Please use temporal operators when using intervals.';\n RETURN NONE;\n END IF;\n\n -- Spatial Query\n IF op ilike 's_%' or op = 'intersects' THEN\n RETURN spatial_op_query(op, args);\n END IF;\n\n IF op IN ('a_equals','a_contains','a_contained_by','a_overlaps') THEN\n IF args->0 ? 'property' THEN\n leftarg := format('to_text_array(%s)', (queryable(args->0->>'property')).path);\n END IF;\n IF args->1 ? 'property' THEN\n rightarg := format('to_text_array(%s)', (queryable(args->1->>'property')).path);\n END IF;\n RETURN FORMAT(\n '%s %s %s',\n COALESCE(leftarg, quote_literal(to_text_array(args->0))),\n CASE op\n WHEN 'a_equals' THEN '='\n WHEN 'a_contains' THEN '@>'\n WHEN 'a_contained_by' THEN '<@'\n WHEN 'a_overlaps' THEN '&&'\n END,\n COALESCE(rightarg, quote_literal(to_text_array(args->1)))\n );\n END IF;\n\n IF op = 'in' THEN\n RAISE NOTICE 'IN : % % %', args, jsonb_build_array(args->0), args->1;\n args := jsonb_build_array(args->0) || (args->1);\n RAISE NOTICE 'IN2 : %', args;\n END IF;\n\n\n\n IF op = 'between' THEN\n args = jsonb_build_array(\n args->0,\n args->1,\n args->2\n );\n END IF;\n\n -- Make sure that args is an array and run cql2_query on\n -- each element of the array\n RAISE NOTICE 'ARGS PRE: %', args;\n IF j ? 'args' THEN\n IF jsonb_typeof(args) != 'array' THEN\n args := jsonb_build_array(args);\n END IF;\n\n IF jsonb_path_exists(args, '$[*] ? (@.property == \"id\" || @.property == \"datetime\" || @.property == \"end_datetime\" || @.property == \"collection\")') THEN\n wrapper := NULL;\n ELSE\n -- if any of the arguments are a property, try to get the property_wrapper\n FOR arg IN SELECT jsonb_path_query(args, '$[*] ? (@.property != null)') LOOP\n RAISE NOTICE 'Arg: %', arg;\n wrapper := (queryable(arg->>'property')).nulled_wrapper;\n RAISE NOTICE 'Property: %, Wrapper: %', arg, wrapper;\n IF wrapper IS NOT NULL THEN\n EXIT;\n END IF;\n END LOOP;\n\n -- if the property was not in queryables, see if any args were numbers\n IF\n wrapper IS NULL\n AND jsonb_path_exists(args, '$[*] ? (@.type()==\"number\")')\n THEN\n wrapper := 'to_float';\n END IF;\n wrapper := coalesce(wrapper, 'to_text');\n END IF;\n\n SELECT jsonb_agg(cql2_query(a, wrapper))\n INTO args\n FROM jsonb_array_elements(args) a;\n END IF;\n RAISE NOTICE 'ARGS: %', args;\n\n IF op IN ('and', 'or') THEN\n RETURN\n format(\n '(%s)',\n array_to_string(to_text_array(args), format(' %s ', upper(op)))\n );\n END IF;\n\n IF op = 'in' THEN\n RAISE NOTICE 'IN -- % %', args->0, to_text(args->0);\n RETURN format(\n '%s IN (%s)',\n to_text(args->0),\n array_to_string((to_text_array(args))[2:], ',')\n );\n END IF;\n\n -- Look up template from cql2_ops\n IF j ? 'op' THEN\n SELECT * INTO cql2op FROM cql2_ops WHERE cql2_ops.op ilike op;\n IF FOUND THEN\n -- If specific index set in queryables for a property cast other arguments to that type\n\n RETURN format(\n cql2op.template,\n VARIADIC (to_text_array(args))\n );\n ELSE\n RAISE EXCEPTION 'Operator % Not Supported.', op;\n END IF;\n END IF;\n\n\n IF wrapper IS NOT NULL THEN\n RAISE NOTICE 'Wrapping % with %', j, wrapper;\n IF j ? 'property' THEN\n RETURN format('%I(%s)', wrapper, (queryable(j->>'property')).path);\n ELSE\n RETURN format('%I(%L)', wrapper, j);\n END IF;\n ELSIF j ? 'property' THEN\n RETURN quote_ident(j->>'property');\n END IF;\n\n RETURN quote_literal(to_text(j));\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION paging_dtrange(\n j jsonb\n) RETURNS tstzrange AS $$\nDECLARE\n op text;\n filter jsonb := j->'filter';\n dtrange tstzrange := tstzrange('-infinity'::timestamptz,'infinity'::timestamptz);\n sdate timestamptz := '-infinity'::timestamptz;\n edate timestamptz := 'infinity'::timestamptz;\n jpitem jsonb;\nBEGIN\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n END IF;\n IF NOT (filter @? '$.**.op ? (@ == \"or\" || @ == \"not\")') THEN\n FOR jpitem IN SELECT j FROM jsonb_path_query(filter,'strict $.** ? (@.args[*].property == \"datetime\")'::jsonpath) j LOOP\n op := lower(jpitem->>'op');\n dtrange := parse_dtrange(jpitem->'args'->1);\n IF op IN ('<=', 'lt', 'lte', '<', 'le', 't_before') THEN\n sdate := greatest(sdate,'-infinity');\n edate := least(edate, upper(dtrange));\n ELSIF op IN ('>=', '>', 'gt', 'gte', 'ge', 't_after') THEN\n edate := least(edate, 'infinity');\n sdate := greatest(sdate, lower(dtrange));\n ELSIF op IN ('=', 'eq') THEN\n edate := least(edate, upper(dtrange));\n sdate := greatest(sdate, lower(dtrange));\n END IF;\n RAISE NOTICE '2 OP: %, ARGS: %, DTRANGE: %, SDATE: %, EDATE: %', op, jpitem->'args'->1, dtrange, sdate, edate;\n END LOOP;\n END IF;\n IF sdate > edate THEN\n RETURN 'empty'::tstzrange;\n END IF;\n RETURN tstzrange(sdate,edate, '[]');\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT SET TIME ZONE 'UTC';\n\nCREATE OR REPLACE FUNCTION paging_collections(\n IN j jsonb\n) RETURNS text[] AS $$\nDECLARE\n filter jsonb := j->'filter';\n jpitem jsonb;\n op text;\n args jsonb;\n arg jsonb;\n collections text[];\nBEGIN\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n END IF;\n IF NOT (filter @? '$.**.op ? (@ == \"or\" || @ == \"not\")') THEN\n FOR jpitem IN SELECT j FROM jsonb_path_query(filter,'strict $.** ? (@.args[*].property == \"collection\")'::jsonpath) j LOOP\n RAISE NOTICE 'JPITEM: %', jpitem;\n op := jpitem->>'op';\n args := jpitem->'args';\n IF op IN ('=', 'eq', 'in') THEN\n FOR arg IN SELECT a FROM jsonb_array_elements(args) a LOOP\n IF jsonb_typeof(arg) IN ('string', 'array') THEN\n RAISE NOTICE 'arg: %, collections: %', arg, collections;\n IF collections IS NULL OR collections = '{}'::text[] THEN\n collections := to_text_array(arg);\n ELSE\n collections := array_intersection(collections, to_text_array(arg));\n END IF;\n END IF;\n END LOOP;\n END IF;\n END LOOP;\n END IF;\n IF collections = '{}'::text[] THEN\n RETURN NULL;\n END IF;\n RETURN collections;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\nCREATE TABLE items (\n id text NOT NULL,\n geometry geometry NOT NULL,\n collection text NOT NULL,\n datetime timestamptz NOT NULL,\n end_datetime timestamptz NOT NULL,\n content JSONB NOT NULL,\n private jsonb\n)\nPARTITION BY LIST (collection)\n;\n\nCREATE INDEX \"datetime_idx\" ON items USING BTREE (datetime DESC, end_datetime ASC);\nCREATE INDEX \"geometry_idx\" ON items USING GIST (geometry);\n\nCREATE STATISTICS datetime_stats (dependencies) on datetime, end_datetime from items;\n\nALTER TABLE items ADD CONSTRAINT items_collections_fk FOREIGN KEY (collection) REFERENCES collections(id) ON DELETE CASCADE DEFERRABLE;\n\nCREATE OR REPLACE FUNCTION partition_after_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p text;\n t timestamptz := clock_timestamp();\nBEGIN\n RAISE NOTICE 'Updating partition stats %', t;\n FOR p IN SELECT DISTINCT partition\n FROM newdata n JOIN partition_sys_meta p\n ON (n.collection=p.collection AND n.datetime <@ p.partition_dtrange)\n LOOP\n PERFORM run_or_queue(format('SELECT update_partition_stats(%L, %L);', p, true));\n END LOOP;\n IF TG_OP IN ('DELETE','UPDATE') THEN\n DELETE FROM format_item_cache c USING newdata n WHERE c.collection = n.collection AND c.id = n.id;\n END IF;\n RAISE NOTICE 't: % %', t, clock_timestamp() - t;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE TRIGGER items_after_insert_trigger\nAFTER INSERT ON items\nREFERENCING NEW TABLE AS newdata\nFOR EACH STATEMENT\nEXECUTE FUNCTION partition_after_triggerfunc();\n\nCREATE TRIGGER items_after_update_trigger\nAFTER DELETE ON items\nREFERENCING OLD TABLE AS newdata\nFOR EACH STATEMENT\nEXECUTE FUNCTION partition_after_triggerfunc();\n\nCREATE TRIGGER items_after_delete_trigger\nAFTER UPDATE ON items\nREFERENCING NEW TABLE AS newdata\nFOR EACH STATEMENT\nEXECUTE FUNCTION partition_after_triggerfunc();\n\n\nCREATE OR REPLACE FUNCTION content_slim(_item jsonb) RETURNS jsonb AS $$\n SELECT strip_jsonb(_item - '{id,geometry,collection,type}'::text[], collection_base_item(_item->>'collection')) - '{id,geometry,collection,type}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_dehydrate(content jsonb) RETURNS items AS $$\n SELECT\n content->>'id' as id,\n stac_geom(content) as geometry,\n content->>'collection' as collection,\n stac_datetime(content) as datetime,\n stac_end_datetime(content) as end_datetime,\n content_slim(content) as content,\n null::jsonb as private\n ;\n$$ LANGUAGE SQL STABLE;\n\nCREATE OR REPLACE FUNCTION include_field(f text, fields jsonb DEFAULT '{}'::jsonb) RETURNS boolean AS $$\nDECLARE\n includes jsonb := fields->'include';\n excludes jsonb := fields->'exclude';\nBEGIN\n IF f IS NULL THEN\n RETURN NULL;\n END IF;\n\n\n IF\n jsonb_typeof(excludes) = 'array'\n AND jsonb_array_length(excludes)>0\n AND excludes ? f\n THEN\n RETURN FALSE;\n END IF;\n\n IF\n (\n jsonb_typeof(includes) = 'array'\n AND jsonb_array_length(includes) > 0\n AND includes ? f\n ) OR\n (\n includes IS NULL\n OR jsonb_typeof(includes) = 'null'\n OR jsonb_array_length(includes) = 0\n )\n THEN\n RETURN TRUE;\n END IF;\n\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nDROP FUNCTION IF EXISTS content_hydrate(jsonb, jsonb, jsonb);\nCREATE OR REPLACE FUNCTION content_hydrate(\n _item jsonb,\n _base_item jsonb,\n fields jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\n SELECT merge_jsonb(\n jsonb_fields(_item, fields),\n jsonb_fields(_base_item, fields)\n );\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION content_hydrate(_item items, _collection collections, fields jsonb DEFAULT '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\n content jsonb;\n base_item jsonb := _collection.base_item;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry, 20)::jsonb;\n END IF;\n output := content_hydrate(\n jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'collection', _item.collection,\n 'type', 'Feature'\n ) || _item.content,\n _collection.base_item,\n fields\n );\n\n RETURN output;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_nonhydrated(\n _item items,\n fields jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry, 20)::jsonb;\n END IF;\n output := jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'collection', _item.collection,\n 'type', 'Feature'\n ) || _item.content;\n RETURN output;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_hydrate(_item items, fields jsonb DEFAULT '{}'::jsonb) RETURNS jsonb AS $$\n SELECT content_hydrate(\n _item,\n (SELECT c FROM collections c WHERE id=_item.collection LIMIT 1),\n fields\n );\n$$ LANGUAGE SQL STABLE;\n\n\nCREATE UNLOGGED TABLE items_staging (\n content JSONB NOT NULL\n);\nCREATE UNLOGGED TABLE items_staging_ignore (\n content JSONB NOT NULL\n);\nCREATE UNLOGGED TABLE items_staging_upsert (\n content JSONB NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION items_staging_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p record;\n _partitions text[];\n part text;\n ts timestamptz := clock_timestamp();\n nrows int;\nBEGIN\n RAISE NOTICE 'Creating Partitions. %', clock_timestamp() - ts;\n\n FOR part IN WITH t AS (\n SELECT\n n.content->>'collection' as collection,\n stac_daterange(n.content->'properties') as dtr,\n partition_trunc\n FROM newdata n JOIN collections ON (n.content->>'collection'=collections.id)\n ), p AS (\n SELECT\n collection,\n COALESCE(date_trunc(partition_trunc::text, lower(dtr)),'-infinity') as d,\n tstzrange(min(lower(dtr)),max(lower(dtr)),'[]') as dtrange,\n tstzrange(min(upper(dtr)),max(upper(dtr)),'[]') as edtrange\n FROM t\n GROUP BY 1,2\n ) SELECT check_partition(collection, dtrange, edtrange) FROM p LOOP\n RAISE NOTICE 'Partition %', part;\n END LOOP;\n\n RAISE NOTICE 'Creating temp table with data to be added. %', clock_timestamp() - ts;\n DROP TABLE IF EXISTS tmpdata;\n CREATE TEMP TABLE tmpdata ON COMMIT DROP AS\n SELECT\n (content_dehydrate(content)).*\n FROM newdata;\n GET DIAGNOSTICS nrows = ROW_COUNT;\n RAISE NOTICE 'Added % rows to tmpdata. %', nrows, clock_timestamp() - ts;\n\n RAISE NOTICE 'Doing the insert. %', clock_timestamp() - ts;\n IF TG_TABLE_NAME = 'items_staging' THEN\n INSERT INTO items\n SELECT * FROM tmpdata;\n GET DIAGNOSTICS nrows = ROW_COUNT;\n RAISE NOTICE 'Inserted % rows to items. %', nrows, clock_timestamp() - ts;\n ELSIF TG_TABLE_NAME = 'items_staging_ignore' THEN\n INSERT INTO items\n SELECT * FROM tmpdata\n ON CONFLICT DO NOTHING;\n GET DIAGNOSTICS nrows = ROW_COUNT;\n RAISE NOTICE 'Inserted % rows to items. %', nrows, clock_timestamp() - ts;\n ELSIF TG_TABLE_NAME = 'items_staging_upsert' THEN\n DELETE FROM items i USING tmpdata s\n WHERE\n i.id = s.id\n AND i.collection = s.collection\n AND i IS DISTINCT FROM s\n ;\n GET DIAGNOSTICS nrows = ROW_COUNT;\n RAISE NOTICE 'Deleted % rows from items. %', nrows, clock_timestamp() - ts;\n INSERT INTO items AS t\n SELECT * FROM tmpdata\n ON CONFLICT DO NOTHING;\n GET DIAGNOSTICS nrows = ROW_COUNT;\n RAISE NOTICE 'Inserted % rows to items. %', nrows, clock_timestamp() - ts;\n END IF;\n\n RAISE NOTICE 'Deleting data from staging table. %', clock_timestamp() - ts;\n DELETE FROM items_staging;\n RAISE NOTICE 'Done. %', clock_timestamp() - ts;\n\n RETURN NULL;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER items_staging_insert_trigger AFTER INSERT ON items_staging REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\nCREATE TRIGGER items_staging_insert_ignore_trigger AFTER INSERT ON items_staging_ignore REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\nCREATE TRIGGER items_staging_insert_upsert_trigger AFTER INSERT ON items_staging_upsert REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\n\nCREATE OR REPLACE FUNCTION item_by_id(_id text, _collection text DEFAULT NULL) RETURNS items AS\n$$\nDECLARE\n i items%ROWTYPE;\nBEGIN\n SELECT * INTO i FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection) LIMIT 1;\n RETURN i;\nEND;\n$$ LANGUAGE PLPGSQL STABLE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION get_item(_id text, _collection text DEFAULT NULL) RETURNS jsonb AS $$\n SELECT content_hydrate(items) FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection);\n$$ LANGUAGE SQL STABLE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION delete_item(_id text, _collection text DEFAULT NULL) RETURNS VOID AS $$\nDECLARE\nout items%ROWTYPE;\nBEGIN\n DELETE FROM items WHERE id = _id AND (_collection IS NULL OR collection=_collection) RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n--/*\nCREATE OR REPLACE FUNCTION create_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION update_item(content jsonb) RETURNS VOID AS $$\nDECLARE\n old items %ROWTYPE;\n out items%ROWTYPE;\nBEGIN\n PERFORM delete_item(content->>'id', content->>'collection');\n PERFORM create_item(content);\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION create_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION collection_bbox(id text) RETURNS jsonb AS $$\n SELECT (replace(replace(replace(st_extent(geometry)::text,'BOX(','[['),')',']]'),' ',','))::jsonb\n FROM items WHERE collection=$1;\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION collection_temporal_extent(id text) RETURNS jsonb AS $$\n SELECT to_jsonb(array[array[min(datetime), max(datetime)]])\n FROM items WHERE collection=$1;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection_extents() RETURNS VOID AS $$\nUPDATE collections\n SET content = jsonb_set_lax(\n content,\n '{extent}'::text[],\n collection_extent(id, FALSE),\n true,\n 'use_json_null'\n )\n;\n$$ LANGUAGE SQL;\nCREATE TABLE partition_stats (\n partition text PRIMARY KEY,\n dtrange tstzrange,\n edtrange tstzrange,\n spatial geometry,\n last_updated timestamptz,\n keys text[]\n) WITH (FILLFACTOR=90);\n\nCREATE INDEX partitions_range_idx ON partition_stats USING GIST(dtrange);\n\n\nCREATE OR REPLACE FUNCTION constraint_tstzrange(expr text) RETURNS tstzrange AS $$\n WITH t AS (\n SELECT regexp_matches(\n expr,\n E'\\\\(''\\([0-9 :+-]*\\)''\\\\).*\\\\(''\\([0-9 :+-]*\\)''\\\\)'\n ) AS m\n ) SELECT tstzrange(m[1]::timestamptz, m[2]::timestamptz) FROM t\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE STRICT;\n\nCREATE OR REPLACE FUNCTION get_tstz_constraint(reloid oid, colname text) RETURNS tstzrange AS $$\nDECLARE\n expr text := NULL;\n m text[];\n ts_lower timestamptz := NULL;\n ts_upper timestamptz := NULL;\n lower_inclusive text := '[';\n upper_inclusive text := ']';\n ts timestamptz;\nBEGIN\n SELECT INTO expr\n string_agg(def, ' AND ')\n FROM pg_constraint JOIN LATERAL pg_get_constraintdef(oid) AS def ON TRUE\n WHERE\n conrelid = reloid\n AND contype = 'c'\n AND def LIKE '%' || colname || '%'\n ;\n\n IF expr IS NULL THEN\n RETURN NULL;\n END IF;\n\n RAISE DEBUG 'Constraint expression for % on %: %', colname, reloid::regclass, expr;\n -- collect all constraints for the specified column\n FOR m IN SELECT regexp_matches(expr, '[ (]' || colname || $expr$\\s*([<>=]{1,2})\\s*'([0-9 :.+\\-]+)'$expr$, 'g') LOOP\n ts := m[2]::timestamptz;\n IF m[1] IN ('>', '>=')\n THEN\n IF ts_lower IS NULL OR ts > ts_lower OR (ts = ts_lower AND m[1] = '>') THEN\n ts_lower := ts;\n lower_inclusive := CASE WHEN m[1] = '>' THEN '(' ELSE '[' END;\n END IF;\n ELSIF m[1] IN ('<', '<=')\n THEN\n IF ts_upper IS NULL OR ts < ts_upper OR (ts = ts_upper AND m[1] = '<') THEN\n ts_upper := ts;\n upper_inclusive := CASE WHEN m[1] = '<' THEN ')' ELSE ']' END;\n END IF;\n END IF;\n END LOOP;\n RAISE DEBUG 'Constraint % for %: % %', colname, reloid::regclass, ts_lower, ts_upper;\n RETURN tstzrange(ts_lower, ts_upper, lower_inclusive || upper_inclusive);\nEND;\n$$ LANGUAGE plpgsql STRICT STABLE;\n\nCREATE OR REPLACE FUNCTION get_partition_name(relid regclass) RETURNS text AS $$\n SELECT (parse_ident(relid::text))[cardinality(parse_ident(relid::text))];\n$$ LANGUAGE SQL STABLE STRICT;\n\nCREATE OR REPLACE VIEW partition_sys_meta AS\nSELECT\n partition,\n replace(\n replace(\n CASE WHEN level = 1 THEN partition_expr ELSE parent_partition_expr END,\n 'FOR VALUES IN (''',\n ''\n ),\n ''')',\n ''\n ) AS collection,\n level,\n c.reltuples,\n c.relhastriggers,\n partition_dtrange,\n COALESCE(\n get_tstz_constraint(c.oid, 'datetime'),\n partition_dtrange,\n inf_range\n ) as constraint_dtrange,\n COALESCE(\n get_tstz_constraint(c.oid, 'end_datetime'),\n inf_range\n ) as constraint_edtrange\nFROM\n pg_partition_tree('items')\n JOIN pg_class c ON (relid::regclass = c.oid)\n JOIN pg_class parent ON (parentrelid::regclass = parent.oid AND isleaf)\n LEFT JOIN pg_constraint edt ON (conrelid=c.oid AND contype='c')\n JOIN LATERAL get_partition_name(relid) AS partition ON TRUE\n JOIN LATERAL pg_get_expr(c.relpartbound, c.oid) as partition_expr ON TRUE\n JOIN LATERAL pg_get_expr(parent.relpartbound, parent.oid) as parent_partition_expr ON TRUE\n JOIN LATERAL tstzrange('-infinity', 'infinity','[]') as inf_range ON TRUE\n JOIN LATERAL COALESCE(constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), inf_range) as partition_dtrange ON TRUE\n JOIN LATERAL get_tstz_constraint(c.oid, 'datetime') as datetime_constraint ON TRUE\n JOIN LATERAL get_tstz_constraint(c.oid, 'end_datetime') as end_datetime_constraint ON TRUE\nWHERE isleaf\n;\n\nCREATE OR REPLACE VIEW partitions_view AS\nSELECT\n (parse_ident(relid::text))[cardinality(parse_ident(relid::text))] as partition,\n replace(\n replace(\n CASE WHEN level = 1 THEN partition_expr ELSE parent_partition_expr END,\n 'FOR VALUES IN (''',\n ''\n ),\n ''')',\n ''\n ) AS collection,\n level,\n c.reltuples,\n c.relhastriggers,\n partition_dtrange,\n COALESCE(\n get_tstz_constraint(c.oid, 'datetime'),\n partition_dtrange,\n inf_range\n ) as constraint_dtrange,\n COALESCE(\n get_tstz_constraint(c.oid, 'end_datetime'),\n inf_range\n ) as constraint_edtrange,\n dtrange,\n edtrange,\n spatial,\n last_updated\nFROM\n pg_partition_tree('items')\n JOIN pg_class c ON (relid::regclass = c.oid)\n JOIN pg_class parent ON (parentrelid::regclass = parent.oid AND isleaf)\n LEFT JOIN pg_constraint edt ON (conrelid=c.oid AND contype='c')\n JOIN LATERAL get_partition_name(relid) AS partition ON TRUE\n JOIN LATERAL pg_get_expr(c.relpartbound, c.oid) as partition_expr ON TRUE\n JOIN LATERAL pg_get_expr(parent.relpartbound, parent.oid) as parent_partition_expr ON TRUE\n JOIN LATERAL tstzrange('-infinity', 'infinity','[]') as inf_range ON TRUE\n JOIN LATERAL COALESCE(constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), inf_range) as partition_dtrange ON TRUE\n JOIN LATERAL get_tstz_constraint(c.oid, 'datetime') as datetime_constraint ON TRUE\n JOIN LATERAL get_tstz_constraint(c.oid, 'end_datetime') as end_datetime_constraint ON TRUE\n LEFT JOIN pgstac.partition_stats USING (partition)\nWHERE isleaf\n;\n\nCREATE MATERIALIZED VIEW partitions AS\nSELECT * FROM partitions_view;\nCREATE UNIQUE INDEX ON partitions (partition);\n\nCREATE MATERIALIZED VIEW partition_steps AS\nSELECT\n partition as name,\n date_trunc('month',lower(partition_dtrange)) as sdate,\n date_trunc('month', upper(partition_dtrange)) + '1 month'::interval as edate\n FROM partitions_view WHERE partition_dtrange IS NOT NULL AND partition_dtrange != 'empty'::tstzrange\n ORDER BY dtrange ASC\n;\n\n\nCREATE OR REPLACE FUNCTION update_partition_stats_q(_partition text, istrigger boolean default false) RETURNS VOID AS $$\nDECLARE\nBEGIN\n PERFORM run_or_queue(\n format('SELECT update_partition_stats(%L, %L);', _partition, istrigger)\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION update_partition_stats(_partition text, istrigger boolean default false) RETURNS VOID AS $$\nDECLARE\n dtrange tstzrange;\n edtrange tstzrange;\n cdtrange tstzrange;\n cedtrange tstzrange;\n extent geometry;\n collection text;\nBEGIN\n RAISE NOTICE 'Updating stats for %.', _partition;\n EXECUTE format(\n $q$\n SELECT\n tstzrange(min(datetime), max(datetime),'[]'),\n tstzrange(min(end_datetime), max(end_datetime), '[]')\n FROM %I\n $q$,\n _partition\n ) INTO dtrange, edtrange;\n EXECUTE format('ANALYZE %I;', _partition);\n extent := st_estimatedextent('pgstac', _partition, 'geometry');\n RAISE DEBUG 'Estimated Extent: %', extent;\n INSERT INTO partition_stats (partition, dtrange, edtrange, spatial, last_updated)\n SELECT _partition, dtrange, edtrange, extent, now()\n ON CONFLICT (partition) DO\n UPDATE SET\n dtrange=EXCLUDED.dtrange,\n edtrange=EXCLUDED.edtrange,\n spatial=EXCLUDED.spatial,\n last_updated=EXCLUDED.last_updated\n ;\n\n SELECT\n constraint_dtrange, constraint_edtrange, pv.collection\n INTO cdtrange, cedtrange, collection\n FROM partitions_view pv WHERE partition = _partition;\n\n RAISE NOTICE 'Checking if we need to modify constraints...';\n RAISE NOTICE 'cdtrange: % dtrange: % cedtrange: % edtrange: %',cdtrange, dtrange, cedtrange, edtrange;\n IF\n (cdtrange IS DISTINCT FROM dtrange OR edtrange IS DISTINCT FROM cedtrange)\n AND NOT istrigger\n THEN\n RAISE NOTICE 'Modifying Constraints';\n RAISE NOTICE 'Existing % %', cdtrange, cedtrange;\n RAISE NOTICE 'New % %', dtrange, edtrange;\n PERFORM drop_table_constraints(_partition);\n PERFORM create_table_constraints(_partition, dtrange, edtrange);\n END IF;\n REFRESH MATERIALIZED VIEW partitions;\n REFRESH MATERIALIZED VIEW partition_steps;\n RAISE NOTICE 'Checking if we need to update collection extents.';\n IF get_setting_bool('update_collection_extent') THEN\n RAISE NOTICE 'updating collection extent for %', collection;\n PERFORM run_or_queue(format($q$\n UPDATE collections\n SET content = jsonb_set_lax(\n content,\n '{extent}'::text[],\n collection_extent(%L, FALSE),\n true,\n 'use_json_null'\n ) WHERE id=%L\n ;\n $q$, collection, collection));\n ELSE\n RAISE NOTICE 'Not updating collection extent for %', collection;\n END IF;\n\nEND;\n$$ LANGUAGE PLPGSQL STRICT SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION partition_name( IN collection text, IN dt timestamptz, OUT partition_name text, OUT partition_range tstzrange) AS $$\nDECLARE\n c RECORD;\n parent_name text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n parent_name := format('_items_%s', c.key);\n\n\n IF c.partition_trunc = 'year' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYY'));\n ELSIF c.partition_trunc = 'month' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYYMM'));\n ELSE\n partition_name := parent_name;\n partition_range := tstzrange('-infinity'::timestamptz, 'infinity'::timestamptz, '[]');\n END IF;\n IF partition_range IS NULL THEN\n partition_range := tstzrange(\n date_trunc(c.partition_trunc::text, dt),\n date_trunc(c.partition_trunc::text, dt) + concat('1 ', c.partition_trunc)::interval\n );\n END IF;\n RETURN;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION drop_table_constraints(t text) RETURNS text AS $$\nDECLARE\n q text;\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM partitions_view WHERE partition=t) THEN\n RETURN NULL;\n END IF;\n FOR q IN SELECT FORMAT(\n $q$\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n $q$,\n t,\n conname\n ) FROM pg_constraint\n WHERE conrelid=t::regclass::oid AND contype='c'\n LOOP\n EXECUTE q;\n END LOOP;\n RETURN t;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE OR REPLACE FUNCTION create_table_constraints(t text, _dtrange tstzrange, _edtrange tstzrange) RETURNS text AS $$\nDECLARE\n q text;\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM partitions_view WHERE partition=t) THEN\n RETURN NULL;\n END IF;\n RAISE NOTICE 'Creating Table Constraints for % % %', t, _dtrange, _edtrange;\n IF _dtrange = 'empty' AND _edtrange = 'empty' THEN\n q :=format(\n $q$\n DO $block$\n BEGIN\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n ALTER TABLE %I\n ADD CONSTRAINT %I\n CHECK (((datetime IS NULL) AND (end_datetime IS NULL))) NOT VALID\n ;\n ALTER TABLE %I\n VALIDATE CONSTRAINT %I\n ;\n\n\n\n EXCEPTION WHEN others THEN\n RAISE WARNING '%%, Issue Altering Constraints. Please run update_partition_stats(%I)', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n $block$;\n $q$,\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t),\n t\n );\n ELSE\n q :=format(\n $q$\n DO $block$\n BEGIN\n\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n ALTER TABLE %I\n ADD CONSTRAINT %I\n CHECK (\n (datetime >= %L)\n AND (datetime <= %L)\n AND (end_datetime >= %L)\n AND (end_datetime <= %L)\n ) NOT VALID\n ;\n ALTER TABLE %I\n VALIDATE CONSTRAINT %I\n ;\n\n\n\n EXCEPTION WHEN others THEN\n RAISE WARNING '%%, Issue Altering Constraints. Please run update_partition_stats(%I)', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n $block$;\n $q$,\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t),\n lower(_dtrange),\n upper(_dtrange),\n lower(_edtrange),\n upper(_edtrange),\n t,\n format('%s_dt', t),\n t\n );\n END IF;\n PERFORM run_or_queue(q);\n RETURN t;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION check_partition(\n _collection text,\n _dtrange tstzrange,\n _edtrange tstzrange\n) RETURNS text AS $$\nDECLARE\n c RECORD;\n pm RECORD;\n _partition_name text;\n _partition_dtrange tstzrange;\n _constraint_dtrange tstzrange;\n _constraint_edtrange tstzrange;\n q text;\n deferrable_q text;\n err_context text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=_collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', _collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n\n IF c.partition_trunc IS NOT NULL THEN\n _partition_dtrange := tstzrange(\n date_trunc(c.partition_trunc, lower(_dtrange)),\n date_trunc(c.partition_trunc, lower(_dtrange)) + (concat('1 ', c.partition_trunc))::interval,\n '[)'\n );\n ELSE\n _partition_dtrange := '[-infinity, infinity]'::tstzrange;\n END IF;\n\n IF NOT _partition_dtrange @> _dtrange THEN\n RAISE EXCEPTION 'dtrange % is greater than the partition size % for collection %', _dtrange, c.partition_trunc, _collection;\n END IF;\n\n\n IF c.partition_trunc = 'year' THEN\n _partition_name := format('_items_%s_%s', c.key, to_char(lower(_partition_dtrange),'YYYY'));\n ELSIF c.partition_trunc = 'month' THEN\n _partition_name := format('_items_%s_%s', c.key, to_char(lower(_partition_dtrange),'YYYYMM'));\n ELSE\n _partition_name := format('_items_%s', c.key);\n END IF;\n\n SELECT * INTO pm FROM partition_sys_meta WHERE collection=_collection AND partition_dtrange @> _dtrange;\n IF FOUND THEN\n RAISE NOTICE '% % %', _edtrange, _dtrange, pm;\n _constraint_edtrange :=\n tstzrange(\n least(\n lower(_edtrange),\n nullif(lower(pm.constraint_edtrange), '-infinity')\n ),\n greatest(\n upper(_edtrange),\n nullif(upper(pm.constraint_edtrange), 'infinity')\n ),\n '[]'\n );\n _constraint_dtrange :=\n tstzrange(\n least(\n lower(_dtrange),\n nullif(lower(pm.constraint_dtrange), '-infinity')\n ),\n greatest(\n upper(_dtrange),\n nullif(upper(pm.constraint_dtrange), 'infinity')\n ),\n '[]'\n );\n\n IF pm.constraint_edtrange @> _edtrange AND pm.constraint_dtrange @> _dtrange THEN\n RETURN pm.partition;\n ELSE\n PERFORM drop_table_constraints(_partition_name);\n END IF;\n ELSE\n _constraint_edtrange := _edtrange;\n _constraint_dtrange := _dtrange;\n END IF;\n RAISE NOTICE 'EXISTING CONSTRAINTS % %, NEW % %', pm.constraint_dtrange, pm.constraint_edtrange, _constraint_dtrange, _constraint_edtrange;\n RAISE NOTICE 'Creating partition % %', _partition_name, _partition_dtrange;\n IF c.partition_trunc IS NULL THEN\n q := format(\n $q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n GRANT ALL ON %I to pgstac_ingest;\n $q$,\n _partition_name,\n _collection,\n concat(_partition_name,'_pk'),\n _partition_name,\n _partition_name\n );\n ELSE\n q := format(\n $q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L) PARTITION BY RANGE (datetime);\n CREATE TABLE IF NOT EXISTS %I partition OF %I FOR VALUES FROM (%L) TO (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n GRANT ALL ON %I TO pgstac_ingest;\n $q$,\n format('_items_%s', c.key),\n _collection,\n _partition_name,\n format('_items_%s', c.key),\n lower(_partition_dtrange),\n upper(_partition_dtrange),\n format('%s_pk', _partition_name),\n _partition_name,\n _partition_name\n );\n END IF;\n\n BEGIN\n EXECUTE q;\n EXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', _partition_name;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\n END;\n PERFORM maintain_partitions(_partition_name);\n PERFORM update_partition_stats_q(_partition_name, true);\n REFRESH MATERIALIZED VIEW partitions;\n REFRESH MATERIALIZED VIEW partition_steps;\n RETURN _partition_name;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION repartition(_collection text, _partition_trunc text, triggered boolean DEFAULT FALSE) RETURNS text AS $$\nDECLARE\n c RECORD;\n q text;\n from_trunc text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=_collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', _collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n IF triggered THEN\n RAISE NOTICE 'Converting % to % partitioning via Trigger', _collection, _partition_trunc;\n ELSE\n RAISE NOTICE 'Converting % from using % to % partitioning', _collection, c.partition_trunc, _partition_trunc;\n IF c.partition_trunc IS NOT DISTINCT FROM _partition_trunc THEN\n RAISE NOTICE 'Collection % already set to use partition by %', _collection, _partition_trunc;\n RETURN _collection;\n END IF;\n END IF;\n\n IF EXISTS (SELECT 1 FROM partitions_view WHERE collection=_collection LIMIT 1) THEN\n EXECUTE format(\n $q$\n CREATE TEMP TABLE changepartitionstaging ON COMMIT DROP AS SELECT * FROM %I;\n DROP TABLE IF EXISTS %I CASCADE;\n WITH p AS (\n SELECT\n collection,\n CASE\n WHEN %L IS NULL THEN '-infinity'::timestamptz\n ELSE date_trunc(%L, datetime)\n END as d,\n tstzrange(min(datetime),max(datetime),'[]') as dtrange,\n tstzrange(min(end_datetime),max(end_datetime),'[]') as edtrange\n FROM changepartitionstaging\n GROUP BY 1,2\n ) SELECT check_partition(collection, dtrange, edtrange) FROM p;\n INSERT INTO items SELECT * FROM changepartitionstaging;\n DROP TABLE changepartitionstaging;\n $q$,\n concat('_items_', c.key),\n concat('_items_', c.key),\n c.partition_trunc,\n c.partition_trunc\n );\n END IF;\n RETURN _collection;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE OR REPLACE FUNCTION collections_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n q text;\n partition_name text := format('_items_%s', NEW.key);\n partition_exists boolean := false;\n partition_empty boolean := true;\n err_context text;\n loadtemp boolean := FALSE;\nBEGIN\n RAISE NOTICE 'Collection Trigger. % %', NEW.id, NEW.key;\n IF TG_OP = 'UPDATE' AND NEW.partition_trunc IS DISTINCT FROM OLD.partition_trunc THEN\n PERFORM repartition(NEW.id, NEW.partition_trunc, TRUE);\n END IF;\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER collections_trigger AFTER\nINSERT\nOR\nUPDATE ON collections\nFOR EACH ROW EXECUTE FUNCTION collections_trigger_func();\n\nCREATE OR REPLACE FUNCTION chunker(\n IN _where text,\n OUT s timestamptz,\n OUT e timestamptz\n) RETURNS SETOF RECORD AS $$\nDECLARE\n explain jsonb;\nBEGIN\n IF _where IS NULL THEN\n _where := ' TRUE ';\n END IF;\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s;', _where)\n INTO explain;\n RAISE DEBUG 'EXPLAIN: %', explain;\n\n RETURN QUERY\n WITH t AS (\n SELECT j->>0 as p FROM\n jsonb_path_query(\n explain,\n 'strict $.**.\"Relation Name\" ? (@ != null)'\n ) j\n ),\n parts AS (\n SELECT sdate, edate FROM t JOIN partition_steps ON (t.p = name)\n ),\n times AS (\n SELECT sdate FROM parts\n UNION\n SELECT edate FROM parts\n ),\n uniq AS (\n SELECT DISTINCT sdate FROM times ORDER BY sdate\n ),\n last AS (\n SELECT sdate, lead(sdate, 1) over () as edate FROM uniq\n )\n SELECT sdate, edate FROM last WHERE edate IS NOT NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION partition_queries(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN partitions text[] DEFAULT NULL\n) RETURNS SETOF text AS $$\nDECLARE\n query text;\n sdate timestamptz;\n edate timestamptz;\nBEGIN\nIF _where IS NULL OR trim(_where) = '' THEN\n _where = ' TRUE ';\nEND IF;\nRAISE NOTICE 'Getting chunks for % %', _where, _orderby;\nIF _orderby ILIKE 'datetime d%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 DESC LOOP\n RETURN NEXT format($q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n $q$,\n sdate,\n edate,\n _where,\n _orderby\n );\n END LOOP;\nELSIF _orderby ILIKE 'datetime a%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 ASC LOOP\n RETURN NEXT format($q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n $q$,\n sdate,\n edate,\n _where,\n _orderby\n );\n END LOOP;\nELSE\n query := format($q$\n SELECT * FROM items\n WHERE %s\n ORDER BY %s\n $q$, _where, _orderby\n );\n\n RETURN NEXT query;\n RETURN;\nEND IF;\n\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION partition_query_view(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN _limit int DEFAULT 10\n) RETURNS text AS $$\n WITH p AS (\n SELECT * FROM partition_queries(_where, _orderby) p\n )\n SELECT\n CASE WHEN EXISTS (SELECT 1 FROM p) THEN\n (SELECT format($q$\n SELECT * FROM (\n %s\n ) total LIMIT %s\n $q$,\n string_agg(\n format($q$ SELECT * FROM ( %s ) AS sub $q$, p),\n '\n UNION ALL\n '\n ),\n _limit\n ))\n ELSE NULL\n END FROM p;\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION q_to_tsquery (jinput jsonb)\n RETURNS tsquery\n AS $$\nDECLARE\n input text;\n processed_text text;\n temp_text text;\n quote_array text[];\n placeholder text := '@QUOTE@';\nBEGIN\n IF jsonb_typeof(jinput) = 'string' THEN\n input := jinput->>0;\n ELSIF jsonb_typeof(jinput) = 'array' THEN\n input := array_to_string(\n array(select jsonb_array_elements_text(jinput)),\n ' OR '\n );\n ELSE\n RAISE EXCEPTION 'Input must be a string or an array of strings.';\n END IF;\n -- Extract all quoted phrases and store in array\n quote_array := regexp_matches(input, '\"[^\"]*\"', 'g');\n\n -- Replace each quoted part with a unique placeholder if there are any quoted phrases\n IF array_length(quote_array, 1) IS NOT NULL THEN\n processed_text := input;\n FOR i IN array_lower(quote_array, 1) .. array_upper(quote_array, 1) LOOP\n processed_text := replace(processed_text, quote_array[i], placeholder || i || placeholder);\n END LOOP;\n ELSE\n processed_text := input;\n END IF;\n\n -- Replace non-quoted text using regular expressions\n\n -- , -> |\n processed_text := regexp_replace(processed_text, ',(?=(?:[^\"]*\"[^\"]*\")*[^\"]*$)', ' | ', 'g');\n\n -- and -> &\n processed_text := regexp_replace(processed_text, '\\s+AND\\s+', ' & ', 'gi');\n\n -- or -> |\n processed_text := regexp_replace(processed_text, '\\s+OR\\s+', ' | ', 'gi');\n\n -- + ->\n processed_text := regexp_replace(processed_text, '^\\s*\\+([a-zA-Z0-9_]+)', '\\1', 'g'); -- +term at start\n processed_text := regexp_replace(processed_text, '\\s*\\+([a-zA-Z0-9_]+)', ' & \\1', 'g'); -- +term elsewhere\n\n -- - -> !\n processed_text := regexp_replace(processed_text, '^\\s*\\-([a-zA-Z0-9_]+)', '! \\1', 'g'); -- -term at start\n processed_text := regexp_replace(processed_text, '\\s*\\-([a-zA-Z0-9_]+)', ' & ! \\1', 'g'); -- -term elsewhere\n\n -- terms separated with spaces are assumed to represent adjacent terms. loop through these\n -- occurrences and replace them with the adjacency operator (<->)\n LOOP\n temp_text := regexp_replace(processed_text, '([a-zA-Z0-9_]+)\\s+([a-zA-Z0-9_]+)(?!\\s*[&|<>])', '\\1 <-> \\2', 'g');\n IF temp_text = processed_text THEN\n EXIT; -- No more replacements were made\n END IF;\n processed_text := temp_text;\n END LOOP;\n\n\n -- Replace placeholders back with quoted phrases if there were any\n IF array_length(quote_array, 1) IS NOT NULL THEN\n FOR i IN array_lower(quote_array, 1) .. array_upper(quote_array, 1) LOOP\n processed_text := replace(processed_text, placeholder || i || placeholder, '''' || substring(quote_array[i] from 2 for length(quote_array[i]) - 2) || '''');\n END LOOP;\n END IF;\n\n -- Print processed_text to the console for debugging purposes\n RAISE NOTICE 'processed_text: %', processed_text;\n\n RETURN to_tsquery('english', processed_text);\nEND;\n$$\nLANGUAGE plpgsql;\n\n\nCREATE OR REPLACE FUNCTION stac_search_to_where(j jsonb) RETURNS text AS $$\nDECLARE\n where_segments text[];\n _where text;\n dtrange tstzrange;\n collections text[];\n geom geometry;\n sdate timestamptz;\n edate timestamptz;\n filterlang text;\n filter jsonb := j->'filter';\n ft_query tsquery;\nBEGIN\n IF j ? 'ids' THEN\n where_segments := where_segments || format('id = ANY (%L) ', to_text_array(j->'ids'));\n END IF;\n\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n where_segments := where_segments || format('collection = ANY (%L) ', collections);\n END IF;\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n\n where_segments := where_segments || format(' datetime <= %L::timestamptz AND end_datetime >= %L::timestamptz ',\n edate,\n sdate\n );\n END IF;\n\n IF j ? 'q' THEN\n ft_query := q_to_tsquery(j->'q');\n where_segments := where_segments || format(\n $quote$\n (\n to_tsvector('english', content->'properties'->>'description') ||\n to_tsvector('english', coalesce(content->'properties'->>'title', '')) ||\n to_tsvector('english', coalesce(content->'properties'->>'keywords', ''))\n ) @@ %L\n $quote$,\n ft_query\n );\n END IF;\n\n geom := stac_geom(j);\n IF geom IS NOT NULL THEN\n where_segments := where_segments || format('st_intersects(geometry, %L)',geom);\n END IF;\n\n filterlang := COALESCE(\n j->>'filter-lang',\n get_setting('default_filter_lang', j->'conf')\n );\n IF NOT filter @? '$.**.op' THEN\n filterlang := 'cql-json';\n END IF;\n\n IF filterlang NOT IN ('cql-json','cql2-json') AND j ? 'filter' THEN\n RAISE EXCEPTION '% is not a supported filter-lang. Please use cql-json or cql2-json.', filterlang;\n END IF;\n\n IF j ? 'query' AND j ? 'filter' THEN\n RAISE EXCEPTION 'Can only use either query or filter at one time.';\n END IF;\n\n IF j ? 'query' THEN\n filter := query_to_cql2(j->'query');\n ELSIF filterlang = 'cql-json' THEN\n filter := cql1_to_cql2(filter);\n END IF;\n RAISE NOTICE 'FILTER: %', filter;\n where_segments := where_segments || cql2_query(filter);\n IF cardinality(where_segments) < 1 THEN\n RETURN ' TRUE ';\n END IF;\n\n _where := array_to_string(array_remove(where_segments, NULL), ' AND ');\n\n IF _where IS NULL OR BTRIM(_where) = '' THEN\n RETURN ' TRUE ';\n END IF;\n RETURN _where;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION parse_sort_dir(_dir text, reverse boolean default false) RETURNS text AS $$\n WITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n ) SELECT\n CASE\n WHEN NOT reverse THEN d\n WHEN d = 'ASC' THEN 'DESC'\n WHEN d = 'DESC' THEN 'ASC'\n END\n FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION sort_dir_to_op(_dir text, prev boolean default false) RETURNS text AS $$\n WITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n ) SELECT\n CASE\n WHEN d = 'ASC' AND prev THEN '<='\n WHEN d = 'DESC' AND prev THEN '>='\n WHEN d = 'ASC' THEN '>='\n WHEN d = 'DESC' THEN '<='\n END\n FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION sort_sqlorderby(\n _search jsonb DEFAULT NULL,\n reverse boolean DEFAULT FALSE\n) RETURNS text AS $$\n WITH sortby AS (\n SELECT coalesce(_search->'sortby','[{\"field\":\"datetime\", \"direction\":\"desc\"}]') as sort\n ), withid AS (\n SELECT CASE\n WHEN sort @? '$[*] ? (@.field == \"id\")' THEN sort\n ELSE sort || '[{\"field\":\"id\", \"direction\":\"desc\"}]'::jsonb\n END as sort\n FROM sortby\n ), withid_rows AS (\n SELECT jsonb_array_elements(sort) as value FROM withid\n ),sorts AS (\n SELECT\n coalesce(\n (queryable(value->>'field')).expression\n ) as key,\n parse_sort_dir(value->>'direction', reverse) as dir\n FROM withid_rows\n )\n SELECT array_to_string(\n array_agg(concat(key, ' ', dir)),\n ', '\n ) FROM sorts;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_sort_dir(sort_item jsonb) RETURNS text AS $$\n SELECT CASE WHEN sort_item->>'direction' ILIKE 'desc%' THEN 'DESC' ELSE 'ASC' END;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION get_token_val_str(\n _field text,\n _item items\n) RETURNS text AS $$\nDECLARE\n q text;\n literal text;\nBEGIN\n q := format($q$ SELECT quote_literal(%s) FROM (SELECT $1.*) as r;$q$, _field);\n EXECUTE q INTO literal USING _item;\n RETURN literal;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n\nCREATE OR REPLACE FUNCTION get_token_record(IN _token text, OUT prev BOOLEAN, OUT item items) RETURNS RECORD AS $$\nDECLARE\n _itemid text := _token;\n _collectionid text;\nBEGIN\n IF _token IS NULL THEN\n RETURN;\n END IF;\n RAISE NOTICE 'Looking for token: %', _token;\n prev := FALSE;\n IF _token ILIKE 'prev:%' THEN\n _itemid := replace(_token, 'prev:','');\n prev := TRUE;\n ELSIF _token ILIKE 'next:%' THEN\n _itemid := replace(_token, 'next:', '');\n END IF;\n SELECT id INTO _collectionid FROM collections WHERE _itemid LIKE concat(id,':%');\n IF FOUND THEN\n _itemid := replace(_itemid, concat(_collectionid,':'), '');\n SELECT * INTO item FROM items WHERE id=_itemid AND collection=_collectionid;\n ELSE\n SELECT * INTO item FROM items WHERE id=_itemid;\n END IF;\n IF item IS NULL THEN\n RAISE EXCEPTION 'Could not find item using token: % item: % collection: %', _token, _itemid, _collectionid;\n END IF;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\n\nCREATE OR REPLACE FUNCTION get_token_filter(\n _sortby jsonb DEFAULT '[{\"field\":\"datetime\",\"direction\":\"desc\"}]'::jsonb,\n token_item items DEFAULT NULL,\n prev boolean DEFAULT FALSE,\n inclusive boolean DEFAULT FALSE\n) RETURNS text AS $$\nDECLARE\n ltop text := '<';\n gtop text := '>';\n dir text;\n sort record;\n orfilter text := '';\n orfilters text[] := '{}'::text[];\n andfilters text[] := '{}'::text[];\n output text;\n token_where text;\nBEGIN\n IF _sortby IS NULL OR _sortby = '[]'::jsonb THEN\n _sortby := '[{\"field\":\"datetime\",\"direction\":\"desc\"}]'::jsonb;\n END IF;\n _sortby := _sortby || jsonb_build_object('field','id','direction',_sortby->0->>'direction');\n RAISE NOTICE 'Getting Token Filter. % %', _sortby, token_item;\n IF inclusive THEN\n orfilters := orfilters || format('( id=%L AND collection=%L )' , token_item.id, token_item.collection);\n END IF;\n\n FOR sort IN\n WITH s1 AS (\n SELECT\n _row,\n (queryable(value->>'field')).expression as _field,\n (value->>'field' = 'id') as _isid,\n get_sort_dir(value) as _dir\n FROM jsonb_array_elements(_sortby)\n WITH ORDINALITY AS t(value, _row)\n )\n SELECT\n _row,\n _field,\n _dir,\n get_token_val_str(_field, token_item) as _val\n FROM s1\n WHERE _row <= (SELECT min(_row) FROM s1 WHERE _isid)\n LOOP\n orfilter := NULL;\n RAISE NOTICE 'SORT: %', sort;\n IF sort._val IS NOT NULL AND ((prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC')) THEN\n orfilter := format($f$(\n (%s %s %s) OR (%s IS NULL)\n )$f$,\n sort._field,\n ltop,\n sort._val,\n sort._val\n );\n ELSIF sort._val IS NULL AND ((prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC')) THEN\n RAISE NOTICE '< but null';\n orfilter := format('%s IS NOT NULL', sort._field);\n ELSIF sort._val IS NULL THEN\n RAISE NOTICE '> but null';\n ELSE\n orfilter := format($f$(\n (%s %s %s) OR (%s IS NULL)\n )$f$,\n sort._field,\n gtop,\n sort._val,\n sort._field\n );\n END IF;\n RAISE NOTICE 'ORFILTER: %', orfilter;\n\n IF orfilter IS NOT NULL THEN\n IF sort._row = 1 THEN\n orfilters := orfilters || orfilter;\n ELSE\n orfilters := orfilters || format('(%s AND %s)', array_to_string(andfilters, ' AND '), orfilter);\n END IF;\n END IF;\n IF sort._val IS NOT NULL THEN\n andfilters := andfilters || format('%s = %s', sort._field, sort._val);\n ELSE\n andfilters := andfilters || format('%s IS NULL', sort._field);\n END IF;\n END LOOP;\n\n output := array_to_string(orfilters, ' OR ');\n\n token_where := concat('(',coalesce(output,'true'),')');\n IF trim(token_where) = '' THEN\n token_where := NULL;\n END IF;\n RAISE NOTICE 'TOKEN_WHERE: %',token_where;\n RETURN token_where;\n END;\n$$ LANGUAGE PLPGSQL SET transform_null_equals TO TRUE\n;\n\nCREATE OR REPLACE FUNCTION search_hash(jsonb, jsonb) RETURNS text AS $$\n SELECT md5(concat(($1 - '{token,limit,context,includes,excludes}'::text[])::text,$2::text));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\nDROP FUNCTION IF EXISTS search_tohash(jsonb);\n\nCREATE TABLE IF NOT EXISTS searches(\n hash text GENERATED ALWAYS AS (search_hash(search, metadata)) STORED PRIMARY KEY,\n search jsonb NOT NULL,\n _where text,\n orderby text,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n metadata jsonb DEFAULT '{}'::jsonb NOT NULL\n);\n\nCREATE TABLE IF NOT EXISTS search_wheres(\n id bigint generated always as identity primary key,\n _where text NOT NULL,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n statslastupdated timestamptz,\n estimated_count bigint,\n estimated_cost float,\n time_to_estimate float,\n total_count bigint,\n time_to_count float,\n partitions text[]\n);\n\nCREATE INDEX IF NOT EXISTS search_wheres_partitions ON search_wheres USING GIN (partitions);\nCREATE UNIQUE INDEX IF NOT EXISTS search_wheres_where ON search_wheres ((md5(_where)));\n\nCREATE OR REPLACE FUNCTION where_stats(\n inwhere text,\n updatestats boolean default false,\n conf jsonb default null\n) RETURNS search_wheres AS $$\nDECLARE\n t timestamptz;\n i interval;\n explain_json jsonb;\n partitions text[];\n sw search_wheres%ROWTYPE;\n inwhere_hash text := md5(inwhere);\n _context text := lower(context(conf));\n _stats_ttl interval := context_stats_ttl(conf);\n _estimated_cost_threshold float := context_estimated_cost(conf);\n _estimated_count_threshold int := context_estimated_count(conf);\n ro bool := pgstac.readonly(conf);\nBEGIN\n -- If updatestats is true then set ttl to 0\n IF updatestats THEN\n RAISE DEBUG 'Updatestats set to TRUE, setting TTL to 0';\n _stats_ttl := '0'::interval;\n END IF;\n\n -- If we don't need to calculate context, just return\n IF _context = 'off' THEN\n sw._where = inwhere;\n RETURN sw;\n END IF;\n\n -- Get any stats that we have.\n IF NOT ro THEN\n -- If there is a lock where another process is\n -- updating the stats, wait so that we don't end up calculating a bunch of times.\n SELECT * INTO sw FROM search_wheres WHERE md5(_where)=inwhere_hash FOR UPDATE;\n ELSE\n SELECT * INTO sw FROM search_wheres WHERE md5(_where)=inwhere_hash;\n END IF;\n\n -- If there is a cached row, figure out if we need to update\n IF\n sw IS NOT NULL\n AND sw.statslastupdated IS NOT NULL\n AND sw.total_count IS NOT NULL\n AND now() - sw.statslastupdated <= _stats_ttl\n THEN\n -- we have a cached row with data that is within our ttl\n RAISE DEBUG 'Stats present in table and lastupdated within ttl: %', sw;\n IF NOT ro THEN\n RAISE DEBUG 'Updating search_wheres only bumping lastused and usecount';\n UPDATE search_wheres SET\n lastused = now(),\n usecount = search_wheres.usecount + 1\n WHERE md5(_where) = inwhere_hash\n RETURNING * INTO sw;\n END IF;\n RAISE DEBUG 'Returning cached counts. %', sw;\n RETURN sw;\n END IF;\n\n -- Calculate estimated cost and rows\n -- Use explain to get estimated count/cost\n IF sw.estimated_count IS NULL OR sw.estimated_cost IS NULL THEN\n RAISE DEBUG 'Calculating estimated stats';\n t := clock_timestamp();\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s', inwhere)\n INTO explain_json;\n RAISE DEBUG 'Time for just the explain: %', clock_timestamp() - t;\n i := clock_timestamp() - t;\n\n sw.estimated_count := explain_json->0->'Plan'->'Plan Rows';\n sw.estimated_cost := explain_json->0->'Plan'->'Total Cost';\n sw.time_to_estimate := extract(epoch from i);\n END IF;\n\n RAISE DEBUG 'ESTIMATED_COUNT: %, THRESHOLD %', sw.estimated_count, _estimated_count_threshold;\n RAISE DEBUG 'ESTIMATED_COST: %, THRESHOLD %', sw.estimated_cost, _estimated_cost_threshold;\n\n -- If context is set to auto and the costs are within the threshold return the estimated costs\n IF\n _context = 'auto'\n AND sw.estimated_count >= _estimated_count_threshold\n AND sw.estimated_cost >= _estimated_cost_threshold\n THEN\n IF NOT ro THEN\n INSERT INTO search_wheres (\n _where,\n lastused,\n usecount,\n statslastupdated,\n estimated_count,\n estimated_cost,\n time_to_estimate,\n total_count,\n time_to_count\n ) VALUES (\n inwhere,\n now(),\n 1,\n now(),\n sw.estimated_count,\n sw.estimated_cost,\n sw.time_to_estimate,\n null,\n null\n ) ON CONFLICT ((md5(_where)))\n DO UPDATE SET\n lastused = EXCLUDED.lastused,\n usecount = search_wheres.usecount + 1,\n statslastupdated = EXCLUDED.statslastupdated,\n estimated_count = EXCLUDED.estimated_count,\n estimated_cost = EXCLUDED.estimated_cost,\n time_to_estimate = EXCLUDED.time_to_estimate,\n total_count = EXCLUDED.total_count,\n time_to_count = EXCLUDED.time_to_count\n RETURNING * INTO sw;\n END IF;\n RAISE DEBUG 'Estimates are within thresholds, returning estimates. %', sw;\n RETURN sw;\n END IF;\n\n -- Calculate Actual Count\n t := clock_timestamp();\n RAISE NOTICE 'Calculating actual count...';\n EXECUTE format(\n 'SELECT count(*) FROM items WHERE %s',\n inwhere\n ) INTO sw.total_count;\n i := clock_timestamp() - t;\n RAISE NOTICE 'Actual Count: % -- %', sw.total_count, i;\n sw.time_to_count := extract(epoch FROM i);\n\n IF NOT ro THEN\n INSERT INTO search_wheres (\n _where,\n lastused,\n usecount,\n statslastupdated,\n estimated_count,\n estimated_cost,\n time_to_estimate,\n total_count,\n time_to_count\n ) VALUES (\n inwhere,\n now(),\n 1,\n now(),\n sw.estimated_count,\n sw.estimated_cost,\n sw.time_to_estimate,\n sw.total_count,\n sw.time_to_count\n ) ON CONFLICT ((md5(_where)))\n DO UPDATE SET\n lastused = EXCLUDED.lastused,\n usecount = search_wheres.usecount + 1,\n statslastupdated = EXCLUDED.statslastupdated,\n estimated_count = EXCLUDED.estimated_count,\n estimated_cost = EXCLUDED.estimated_cost,\n time_to_estimate = EXCLUDED.time_to_estimate,\n total_count = EXCLUDED.total_count,\n time_to_count = EXCLUDED.time_to_count\n RETURNING * INTO sw;\n END IF;\n RAISE DEBUG 'Returning with actual count. %', sw;\n RETURN sw;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION search_query(\n _search jsonb = '{}'::jsonb,\n updatestats boolean = false,\n _metadata jsonb = '{}'::jsonb\n) RETURNS searches AS $$\nDECLARE\n search searches%ROWTYPE;\n cached_search searches%ROWTYPE;\n pexplain jsonb;\n t timestamptz;\n i interval;\n doupdate boolean := FALSE;\n insertfound boolean := FALSE;\n ro boolean := pgstac.readonly();\n found_search text;\nBEGIN\n RAISE NOTICE 'SEARCH: %', _search;\n -- Calculate hash, where clause, and order by statement\n search.search := _search;\n search.metadata := _metadata;\n search.hash := search_hash(_search, _metadata);\n search._where := stac_search_to_where(_search);\n search.orderby := sort_sqlorderby(_search);\n search.lastused := now();\n search.usecount := 1;\n\n -- If we are in read only mode, directly return search\n IF ro THEN\n RETURN search;\n END IF;\n\n RAISE NOTICE 'Updating Statistics for search: %s', search;\n -- Update statistics for times used and and when last used\n -- If the entry is locked, rather than waiting, skip updating the stats\n INSERT INTO searches (search, lastused, usecount, metadata)\n VALUES (search.search, now(), 1, search.metadata)\n ON CONFLICT DO NOTHING\n RETURNING * INTO cached_search\n ;\n\n IF NOT FOUND OR cached_search IS NULL THEN\n UPDATE searches SET\n lastused = now(),\n usecount = searches.usecount + 1\n WHERE hash = (\n SELECT hash FROM searches WHERE hash=search.hash FOR UPDATE SKIP LOCKED\n )\n RETURNING * INTO cached_search\n ;\n END IF;\n\n IF cached_search IS NOT NULL THEN\n cached_search._where = search._where;\n cached_search.orderby = search.orderby;\n RETURN cached_search;\n END IF;\n RETURN search;\n\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE OR REPLACE FUNCTION search_fromhash(\n _hash text\n) RETURNS searches AS $$\n SELECT * FROM search_query((SELECT search FROM searches WHERE hash=_hash LIMIT 1));\n$$ LANGUAGE SQL STRICT;\n\nCREATE OR REPLACE FUNCTION search_rows(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN partitions text[] DEFAULT NULL,\n IN _limit int DEFAULT 10\n) RETURNS SETOF items AS $$\nDECLARE\n base_query text;\n query text;\n sdate timestamptz;\n edate timestamptz;\n n int;\n records_left int := _limit;\n timer timestamptz := clock_timestamp();\n full_timer timestamptz := clock_timestamp();\nBEGIN\nIF _where IS NULL OR trim(_where) = '' THEN\n _where = ' TRUE ';\nEND IF;\nRAISE NOTICE 'Getting chunks for % %', _where, _orderby;\n\nbase_query := $q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n LIMIT %L\n$q$;\n\nIF _orderby ILIKE 'datetime d%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 DESC LOOP\n RAISE NOTICE 'Running Query for % to %. %', sdate, edate, age_ms(full_timer);\n query := format(\n base_query,\n sdate,\n edate,\n _where,\n _orderby,\n records_left\n );\n RAISE DEBUG 'QUERY: %', query;\n timer := clock_timestamp();\n RETURN QUERY EXECUTE query;\n\n GET DIAGNOSTICS n = ROW_COUNT;\n records_left := records_left - n;\n RAISE NOTICE 'Returned %/% Rows From % to %. % to go. Time: %ms', n, _limit, sdate, edate, records_left, age_ms(timer);\n timer := clock_timestamp();\n IF records_left <= 0 THEN\n RAISE NOTICE 'SEARCH_ROWS TOOK %ms', age_ms(full_timer);\n RETURN;\n END IF;\n END LOOP;\nELSIF _orderby ILIKE 'datetime a%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 ASC LOOP\n RAISE NOTICE 'Running Query for % to %. %', sdate, edate, age_ms(full_timer);\n query := format(\n base_query,\n sdate,\n edate,\n _where,\n _orderby,\n records_left\n );\n RAISE DEBUG 'QUERY: %', query;\n timer := clock_timestamp();\n RETURN QUERY EXECUTE query;\n\n GET DIAGNOSTICS n = ROW_COUNT;\n records_left := records_left - n;\n RAISE NOTICE 'Returned %/% Rows From % to %. % to go. Time: %ms', n, _limit, sdate, edate, records_left, age_ms(timer);\n timer := clock_timestamp();\n IF records_left <= 0 THEN\n RAISE NOTICE 'SEARCH_ROWS TOOK %ms', age_ms(full_timer);\n RETURN;\n END IF;\n END LOOP;\nELSE\n query := format($q$\n SELECT * FROM items\n WHERE %s\n ORDER BY %s\n LIMIT %L\n $q$, _where, _orderby, _limit\n );\n RAISE DEBUG 'QUERY: %', query;\n timer := clock_timestamp();\n RETURN QUERY EXECUTE query;\n RAISE NOTICE 'FULL QUERY TOOK %ms', age_ms(timer);\nEND IF;\nRAISE NOTICE 'SEARCH_ROWS TOOK %ms', age_ms(full_timer);\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE UNLOGGED TABLE format_item_cache(\n id text,\n collection text,\n fields text,\n hydrated bool,\n output jsonb,\n lastused timestamptz DEFAULT now(),\n usecount int DEFAULT 1,\n timetoformat float,\n PRIMARY KEY (collection, id, fields, hydrated)\n);\nCREATE INDEX ON format_item_cache (lastused);\n\nCREATE OR REPLACE FUNCTION format_item(_item items, _fields jsonb DEFAULT '{}', _hydrated bool DEFAULT TRUE) RETURNS jsonb AS $$\nDECLARE\n cache bool := get_setting_bool('format_cache');\n _output jsonb := null;\n t timestamptz := clock_timestamp();\nBEGIN\n IF cache THEN\n SELECT output INTO _output FROM format_item_cache\n WHERE id=_item.id AND collection=_item.collection AND fields=_fields::text AND hydrated=_hydrated;\n END IF;\n IF _output IS NULL THEN\n IF _hydrated THEN\n _output := content_hydrate(_item, _fields);\n ELSE\n _output := content_nonhydrated(_item, _fields);\n END IF;\n END IF;\n IF cache THEN\n INSERT INTO format_item_cache (id, collection, fields, hydrated, output, timetoformat)\n VALUES (_item.id, _item.collection, _fields::text, _hydrated, _output, age_ms(t))\n ON CONFLICT(collection, id, fields, hydrated) DO\n UPDATE\n SET lastused=now(), usecount = format_item_cache.usecount + 1\n ;\n END IF;\n RETURN _output;\n\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION search(_search jsonb = '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n searches searches%ROWTYPE;\n _where text;\n orderby text;\n search_where search_wheres%ROWTYPE;\n total_count bigint;\n token record;\n token_prev boolean;\n token_item items%ROWTYPE;\n token_where text;\n full_where text;\n init_ts timestamptz := clock_timestamp();\n timer timestamptz := clock_timestamp();\n hydrate bool := NOT (_search->'conf'->>'nohydrate' IS NOT NULL AND (_search->'conf'->>'nohydrate')::boolean = true);\n prev text;\n next text;\n context jsonb;\n collection jsonb;\n out_records jsonb;\n out_len int;\n _limit int := coalesce((_search->>'limit')::int, 10);\n _querylimit int;\n _fields jsonb := coalesce(_search->'fields', '{}'::jsonb);\n has_prev boolean := FALSE;\n has_next boolean := FALSE;\n links jsonb := '[]'::jsonb;\n base_url text:= concat(rtrim(base_url(_search->'conf'),'/'));\nBEGIN\n searches := search_query(_search);\n _where := searches._where;\n orderby := searches.orderby;\n search_where := where_stats(_where);\n total_count := coalesce(search_where.total_count, search_where.estimated_count);\n RAISE NOTICE 'SEARCH:TOKEN: %', _search->>'token';\n token := get_token_record(_search->>'token');\n RAISE NOTICE '***TOKEN: %', token;\n _querylimit := _limit + 1;\n IF token IS NOT NULL THEN\n token_prev := token.prev;\n token_item := token.item;\n token_where := get_token_filter(_search->'sortby', token_item, token_prev, FALSE);\n RAISE DEBUG 'TOKEN_WHERE: % (%ms from search start)', token_where, age_ms(timer);\n IF token_prev THEN -- if we are using a prev token, we know has_next is true\n RAISE DEBUG 'There is a previous token, so automatically setting has_next to true';\n has_next := TRUE;\n orderby := sort_sqlorderby(_search, TRUE);\n ELSE\n RAISE DEBUG 'There is a next token, so automatically setting has_prev to true';\n has_prev := TRUE;\n\n END IF;\n ELSE -- if there was no token, we know there is no prev\n RAISE DEBUG 'There is no token, so we know there is no prev. setting has_prev to false';\n has_prev := FALSE;\n END IF;\n\n full_where := concat_ws(' AND ', _where, token_where);\n RAISE NOTICE 'FULL WHERE CLAUSE: %', full_where;\n RAISE NOTICE 'Time to get counts and build query %', age_ms(timer);\n timer := clock_timestamp();\n\n IF hydrate THEN\n RAISE NOTICE 'Getting hydrated data.';\n ELSE\n RAISE NOTICE 'Getting non-hydrated data.';\n END IF;\n RAISE NOTICE 'CACHE SET TO %', get_setting_bool('format_cache');\n RAISE NOTICE 'Time to set hydration/formatting %', age_ms(timer);\n timer := clock_timestamp();\n SELECT jsonb_agg(format_item(i, _fields, hydrate)) INTO out_records\n FROM search_rows(\n full_where,\n orderby,\n search_where.partitions,\n _querylimit\n ) as i;\n\n RAISE NOTICE 'Time to fetch rows %', age_ms(timer);\n timer := clock_timestamp();\n\n\n IF token_prev THEN\n out_records := flip_jsonb_array(out_records);\n END IF;\n\n RAISE NOTICE 'Query returned % records.', jsonb_array_length(out_records);\n RAISE DEBUG 'TOKEN: % %', token_item.id, token_item.collection;\n RAISE DEBUG 'RECORD_1: % %', out_records->0->>'id', out_records->0->>'collection';\n RAISE DEBUG 'RECORD-1: % %', out_records->-1->>'id', out_records->-1->>'collection';\n\n -- REMOVE records that were from our token\n IF out_records->0->>'id' = token_item.id AND out_records->0->>'collection' = token_item.collection THEN\n out_records := out_records - 0;\n ELSIF out_records->-1->>'id' = token_item.id AND out_records->-1->>'collection' = token_item.collection THEN\n out_records := out_records - -1;\n END IF;\n\n out_len := jsonb_array_length(out_records);\n\n IF out_len = _limit + 1 THEN\n IF token_prev THEN\n has_prev := TRUE;\n out_records := out_records - 0;\n ELSE\n has_next := TRUE;\n out_records := out_records - -1;\n END IF;\n END IF;\n\n\n links := links || jsonb_build_object(\n 'rel', 'root',\n 'type', 'application/json',\n 'href', base_url\n ) || jsonb_build_object(\n 'rel', 'self',\n 'type', 'application/json',\n 'href', concat(base_url, '/search')\n );\n\n IF has_next THEN\n next := concat(out_records->-1->>'collection', ':', out_records->-1->>'id');\n RAISE NOTICE 'HAS NEXT | %', next;\n links := links || jsonb_build_object(\n 'rel', 'next',\n 'type', 'application/geo+json',\n 'method', 'GET',\n 'href', concat(base_url, '/search?token=next:', next)\n );\n END IF;\n\n IF has_prev THEN\n prev := concat(out_records->0->>'collection', ':', out_records->0->>'id');\n RAISE NOTICE 'HAS PREV | %', prev;\n links := links || jsonb_build_object(\n 'rel', 'prev',\n 'type', 'application/geo+json',\n 'method', 'GET',\n 'href', concat(base_url, '/search?token=prev:', prev)\n );\n END IF;\n\n RAISE NOTICE 'Time to get prev/next %', age_ms(timer);\n timer := clock_timestamp();\n\n\n collection := jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb),\n 'links', links\n );\n\n\n\n IF context(_search->'conf') != 'off' THEN\n collection := collection || jsonb_strip_nulls(jsonb_build_object(\n 'numberMatched', total_count,\n 'numberReturned', coalesce(jsonb_array_length(out_records), 0)\n ));\n ELSE\n collection := collection || jsonb_strip_nulls(jsonb_build_object(\n 'numberReturned', coalesce(jsonb_array_length(out_records), 0)\n ));\n END IF;\n\n IF get_setting_bool('timing', _search->'conf') THEN\n collection = collection || jsonb_build_object('timing', age_ms(init_ts));\n END IF;\n\n RAISE NOTICE 'Time to build final json %', age_ms(timer);\n timer := clock_timestamp();\n\n RAISE NOTICE 'Total Time: %', age_ms(current_timestamp);\n RAISE NOTICE 'RETURNING % records. NEXT: %. PREV: %', collection->>'numberReturned', collection->>'next', collection->>'prev';\n RETURN collection;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION search_cursor(_search jsonb = '{}'::jsonb) RETURNS refcursor AS $$\nDECLARE\n curs refcursor;\n searches searches%ROWTYPE;\n _where text;\n _orderby text;\n q text;\n\nBEGIN\n searches := search_query(_search);\n _where := searches._where;\n _orderby := searches.orderby;\n\n OPEN curs FOR\n WITH p AS (\n SELECT * FROM partition_queries(_where, _orderby) p\n )\n SELECT\n CASE WHEN EXISTS (SELECT 1 FROM p) THEN\n (SELECT format($q$\n SELECT * FROM (\n %s\n ) total\n $q$,\n string_agg(\n format($q$ SELECT * FROM ( %s ) AS sub $q$, p),\n '\n UNION ALL\n '\n )\n ))\n ELSE NULL\n END FROM p;\n RETURN curs;\nEND;\n$$ LANGUAGE PLPGSQL;\nCREATE OR REPLACE VIEW collections_asitems AS\nSELECT\n id,\n geometry,\n 'collections' AS collection,\n datetime,\n end_datetime,\n jsonb_build_object(\n 'properties', content - '{links,assets,stac_version,stac_extensions}',\n 'links', content->'links',\n 'assets', content->'assets',\n 'stac_version', content->'stac_version',\n 'stac_extensions', content->'stac_extensions'\n ) AS content,\n content as collectionjson\nFROM collections;\n\n\nCREATE OR REPLACE FUNCTION collection_search_matched(\n IN _search jsonb DEFAULT '{}'::jsonb,\n OUT matched bigint\n) RETURNS bigint AS $$\nDECLARE\n _where text := stac_search_to_where(_search);\nBEGIN\n EXECUTE format(\n $query$\n SELECT\n count(*)\n FROM\n collections_asitems\n WHERE %s\n ;\n $query$,\n _where\n ) INTO matched;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION collection_search_rows(\n _search jsonb DEFAULT '{}'::jsonb\n) RETURNS SETOF jsonb AS $$\nDECLARE\n _where text := stac_search_to_where(_search);\n _limit int := coalesce((_search->>'limit')::int, 10);\n _fields jsonb := coalesce(_search->'fields', '{}'::jsonb);\n _orderby text;\n _offset int := COALESCE((_search->>'offset')::int, 0);\nBEGIN\n _orderby := sort_sqlorderby(\n jsonb_build_object(\n 'sortby',\n coalesce(\n _search->'sortby',\n '[{\"field\": \"id\", \"direction\": \"asc\"}]'::jsonb\n )\n )\n );\n RETURN QUERY EXECUTE format(\n $query$\n SELECT\n jsonb_fields(collectionjson, %L) as c\n FROM\n collections_asitems\n WHERE %s\n ORDER BY %s\n LIMIT %L\n OFFSET %L\n ;\n $query$,\n _fields,\n _where,\n _orderby,\n _limit,\n _offset\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION collection_search(\n _search jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\nDECLARE\n out_records jsonb;\n number_matched bigint := collection_search_matched(_search);\n number_returned bigint;\n _limit int := coalesce((_search->>'limit')::float::int, 10);\n _offset int := coalesce((_search->>'offset')::float::int, 0);\n links jsonb := '[]';\n ret jsonb;\n base_url text:= concat(rtrim(base_url(_search->'conf'),'/'), '/collections');\n prevoffset int;\n nextoffset int;\nBEGIN\n SELECT\n coalesce(jsonb_agg(c), '[]')\n INTO out_records\n FROM collection_search_rows(_search) c;\n\n number_returned := jsonb_array_length(out_records);\n RAISE DEBUG 'nm: %, nr: %, l:%, o:%', number_matched, number_returned, _limit, _offset;\n\n\n\n IF _limit <= number_matched AND number_matched > 0 THEN --need to have paging links\n nextoffset := least(_offset + _limit, number_matched - 1);\n prevoffset := greatest(_offset - _limit, 0);\n\n IF _offset > 0 THEN\n links := links || jsonb_build_object(\n 'rel', 'prev',\n 'type', 'application/json',\n 'method', 'GET' ,\n 'href', base_url,\n 'body', jsonb_build_object('offset', prevoffset),\n 'merge', TRUE\n );\n END IF;\n\n IF (_offset + _limit < number_matched) THEN\n links := links || jsonb_build_object(\n 'rel', 'next',\n 'type', 'application/json',\n 'method', 'GET' ,\n 'href', base_url,\n 'body', jsonb_build_object('offset', nextoffset),\n 'merge', TRUE\n );\n END IF;\n\n END IF;\n\n ret := jsonb_build_object(\n 'collections', out_records,\n 'numberMatched', number_matched,\n 'numberReturned', number_returned,\n 'links', links\n );\n RETURN ret;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION tileenvelope(zoom int, x int, y int) RETURNS geometry AS $$\nWITH t AS (\n SELECT\n 20037508.3427892 as merc_max,\n -20037508.3427892 as merc_min,\n (2 * 20037508.3427892) / (2 ^ zoom) as tile_size\n)\nSELECT st_makeenvelope(\n merc_min + (tile_size * x),\n merc_max - (tile_size * (y + 1)),\n merc_min + (tile_size * (x + 1)),\n merc_max - (tile_size * y),\n 3857\n) FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;DROP FUNCTION IF EXISTS mercgrid;\n\n\nCREATE OR REPLACE FUNCTION ftime() RETURNS interval as $$\nSELECT age(clock_timestamp(), transaction_timestamp());\n$$ LANGUAGE SQL;\nSET SEARCH_PATH to pgstac, public;\n\nDROP FUNCTION IF EXISTS geometrysearch;\nCREATE OR REPLACE FUNCTION geometrysearch(\n IN geom geometry,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE, -- Return as soon as the passed in geometry is full covered\n IN skipcovered boolean DEFAULT TRUE -- Skip any items that would show up completely under the previous items\n) RETURNS jsonb AS $$\nDECLARE\n search searches%ROWTYPE;\n curs refcursor;\n _where text;\n query text;\n iter_record items%ROWTYPE;\n out_records jsonb := '{}'::jsonb[];\n exit_flag boolean := FALSE;\n counter int := 1;\n scancounter int := 1;\n remaining_limit int := _scanlimit;\n tilearea float;\n unionedgeom geometry;\n clippedgeom geometry;\n unionedgeom_area float := 0;\n prev_area float := 0;\n excludes text[];\n includes text[];\n\nBEGIN\n DROP TABLE IF EXISTS pgstac_results;\n CREATE TEMP TABLE pgstac_results (content jsonb) ON COMMIT DROP;\n\n -- If the passed in geometry is not an area set exitwhenfull and skipcovered to false\n IF ST_GeometryType(geom) !~* 'polygon' THEN\n RAISE NOTICE 'GEOMETRY IS NOT AN AREA';\n skipcovered = FALSE;\n exitwhenfull = FALSE;\n END IF;\n\n -- If skipcovered is true then you will always want to exit when the passed in geometry is full\n IF skipcovered THEN\n exitwhenfull := TRUE;\n END IF;\n\n search := search_fromhash(queryhash);\n\n IF search IS NULL THEN\n RAISE EXCEPTION 'Search with Query Hash % Not Found', queryhash;\n END IF;\n\n tilearea := st_area(geom);\n _where := format('%s AND st_intersects(geometry, %L::geometry)', search._where, geom);\n\n\n FOR query IN SELECT * FROM partition_queries(_where, search.orderby) LOOP\n query := format('%s LIMIT %L', query, remaining_limit);\n RAISE NOTICE '%', query;\n OPEN curs FOR EXECUTE query;\n LOOP\n FETCH curs INTO iter_record;\n EXIT WHEN NOT FOUND;\n IF exitwhenfull OR skipcovered THEN -- If we are not using exitwhenfull or skipcovered, we do not need to do expensive geometry operations\n clippedgeom := st_intersection(geom, iter_record.geometry);\n\n IF unionedgeom IS NULL THEN\n unionedgeom := clippedgeom;\n ELSE\n unionedgeom := st_union(unionedgeom, clippedgeom);\n END IF;\n\n unionedgeom_area := st_area(unionedgeom);\n\n IF skipcovered AND prev_area = unionedgeom_area THEN\n scancounter := scancounter + 1;\n CONTINUE;\n END IF;\n\n prev_area := unionedgeom_area;\n\n RAISE NOTICE '% % % %', unionedgeom_area/tilearea, counter, scancounter, ftime();\n END IF;\n RAISE NOTICE '% %', iter_record, content_hydrate(iter_record, fields);\n INSERT INTO pgstac_results (content) VALUES (content_hydrate(iter_record, fields));\n\n IF counter >= _limit\n OR scancounter > _scanlimit\n OR ftime() > _timelimit\n OR (exitwhenfull AND unionedgeom_area >= tilearea)\n THEN\n exit_flag := TRUE;\n EXIT;\n END IF;\n counter := counter + 1;\n scancounter := scancounter + 1;\n\n END LOOP;\n CLOSE curs;\n EXIT WHEN exit_flag;\n remaining_limit := _scanlimit - scancounter;\n END LOOP;\n\n SELECT jsonb_agg(content) INTO out_records FROM pgstac_results WHERE content IS NOT NULL;\n\n RETURN jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb)\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDROP FUNCTION IF EXISTS geojsonsearch;\nCREATE OR REPLACE FUNCTION geojsonsearch(\n IN geojson jsonb,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_geomfromgeojson(geojson),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS xyzsearch;\nCREATE OR REPLACE FUNCTION xyzsearch(\n IN _x int,\n IN _y int,\n IN _z int,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_transform(tileenvelope(_z, _x, _y), 4326),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE PROCEDURE analyze_items() AS $$\nDECLARE\n q text;\n timeout_ts timestamptz;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n SELECT format('ANALYZE (VERBOSE, SKIP_LOCKED) %I;', relname) INTO q\n FROM pg_stat_user_tables\n WHERE relname like '_item%' AND (n_mod_since_analyze>0 OR last_analyze IS NULL) LIMIT 1;\n IF NOT FOUND THEN\n EXIT;\n END IF;\n RAISE NOTICE '%', q;\n EXECUTE q;\n COMMIT;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE PROCEDURE validate_constraints() AS $$\nDECLARE\n q text;\nBEGIN\n FOR q IN\n SELECT\n FORMAT(\n 'ALTER TABLE %I.%I VALIDATE CONSTRAINT %I;',\n nsp.nspname,\n cls.relname,\n con.conname\n )\n\n FROM pg_constraint AS con\n JOIN pg_class AS cls\n ON con.conrelid = cls.oid\n JOIN pg_namespace AS nsp\n ON cls.relnamespace = nsp.oid\n WHERE convalidated = FALSE AND contype in ('c','f')\n AND nsp.nspname = 'pgstac'\n LOOP\n RAISE NOTICE '%', q;\n PERFORM run_or_queue(q);\n COMMIT;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION collection_extent(_collection text, runupdate boolean default false) RETURNS jsonb AS $$\nDECLARE\n geom_extent geometry;\n mind timestamptz;\n maxd timestamptz;\n extent jsonb;\nBEGIN\n IF runupdate THEN\n PERFORM update_partition_stats_q(partition)\n FROM partitions_view WHERE collection=_collection;\n END IF;\n SELECT\n min(lower(dtrange)),\n max(upper(edtrange)),\n st_extent(spatial)\n INTO\n mind,\n maxd,\n geom_extent\n FROM partitions_view\n WHERE collection=_collection;\n\n IF geom_extent IS NOT NULL AND mind IS NOT NULL AND maxd IS NOT NULL THEN\n extent := jsonb_build_object(\n 'spatial', jsonb_build_object(\n 'bbox', to_jsonb(array[array[st_xmin(geom_extent), st_ymin(geom_extent), st_xmax(geom_extent), st_ymax(geom_extent)]])\n ),\n 'temporal', jsonb_build_object(\n 'interval', to_jsonb(array[array[mind, maxd]])\n )\n );\n RETURN extent;\n END IF;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('id', '{\"title\": \"Item ID\",\"description\": \"Item identifier\",\"$ref\": \"https://schemas.stacspec.org/v1.0.0/item-spec/json-schema/item.json#/definitions/core/allOf/2/properties/id\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('geometry', '{\"title\": \"Item Geometry\",\"description\": \"Item Geometry\",\"$ref\": \"https://geojson.org/schema/Feature.json\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('datetime','{\"description\": \"Datetime\",\"type\": \"string\",\"title\": \"Acquired\",\"format\": \"date-time\",\"pattern\": \"(\\\\+00:00|Z)$\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDELETE FROM queryables a USING queryables b\n WHERE a.name = b.name AND a.collection_ids IS NOT DISTINCT FROM b.collection_ids AND a.id > b.id;\n\n\nINSERT INTO pgstac_settings (name, value) VALUES\n ('context', 'off'),\n ('context_estimated_count', '100000'),\n ('context_estimated_cost', '100000'),\n ('context_stats_ttl', '1 day'),\n ('default_filter_lang', 'cql2-json'),\n ('additional_properties', 'true'),\n ('use_queue', 'false'),\n ('queue_timeout', '10 minutes'),\n ('update_collection_extent', 'false'),\n ('format_cache', 'false'),\n ('readonly', 'false')\nON CONFLICT DO NOTHING\n;\n\n\nINSERT INTO cql2_ops (op, template, types) VALUES\n ('eq', '%s = %s', NULL),\n ('neq', '%s != %s', NULL),\n ('ne', '%s != %s', NULL),\n ('!=', '%s != %s', NULL),\n ('<>', '%s != %s', NULL),\n ('lt', '%s < %s', NULL),\n ('lte', '%s <= %s', NULL),\n ('gt', '%s > %s', NULL),\n ('gte', '%s >= %s', NULL),\n ('le', '%s <= %s', NULL),\n ('ge', '%s >= %s', NULL),\n ('=', '%s = %s', NULL),\n ('<', '%s < %s', NULL),\n ('<=', '%s <= %s', NULL),\n ('>', '%s > %s', NULL),\n ('>=', '%s >= %s', NULL),\n ('like', '%s LIKE %s', NULL),\n ('ilike', '%s ILIKE %s', NULL),\n ('+', '%s + %s', NULL),\n ('-', '%s - %s', NULL),\n ('*', '%s * %s', NULL),\n ('/', '%s / %s', NULL),\n ('not', 'NOT (%s)', NULL),\n ('between', '%s BETWEEN %s AND %s', NULL),\n ('isnull', '%s IS NULL', NULL),\n ('upper', 'upper(%s)', NULL),\n ('lower', 'lower(%s)', NULL),\n ('casei', 'upper(%s)', NULL),\n ('accenti', 'unaccent(%s)', NULL)\nON CONFLICT (op) DO UPDATE\n SET\n template = EXCLUDED.template\n;\n\n\nALTER FUNCTION to_text COST 5000;\nALTER FUNCTION to_float COST 5000;\nALTER FUNCTION to_int COST 5000;\nALTER FUNCTION to_tstz COST 5000;\nALTER FUNCTION to_text_array COST 5000;\n\nALTER FUNCTION update_partition_stats SECURITY DEFINER;\nALTER FUNCTION partition_after_triggerfunc SECURITY DEFINER;\nALTER FUNCTION drop_table_constraints SECURITY DEFINER;\nALTER FUNCTION create_table_constraints SECURITY DEFINER;\nALTER FUNCTION check_partition SECURITY DEFINER;\nALTER FUNCTION repartition SECURITY DEFINER;\nALTER FUNCTION where_stats SECURITY DEFINER;\nALTER FUNCTION search_query SECURITY DEFINER;\nALTER FUNCTION format_item SECURITY DEFINER;\nALTER FUNCTION maintain_index SECURITY DEFINER;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nGRANT ALL ON SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac to pgstac_admin;\n\n-- pgstac_read role limited to using function apis\nGRANT EXECUTE ON FUNCTION search TO pgstac_read;\nGRANT EXECUTE ON FUNCTION search_query TO pgstac_read;\nGRANT EXECUTE ON FUNCTION item_by_id TO pgstac_read;\nGRANT EXECUTE ON FUNCTION get_item TO pgstac_read;\nGRANT SELECT ON ALL TABLES IN SCHEMA pgstac TO pgstac_read;\n\n\nGRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_ingest;\nGRANT USAGE ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_ingest;\n\nREVOKE ALL PRIVILEGES ON PROCEDURE run_queued_queries FROM public;\nGRANT ALL ON PROCEDURE run_queued_queries TO pgstac_admin;\n\nREVOKE ALL PRIVILEGES ON FUNCTION run_queued_queries_intransaction FROM public;\nGRANT ALL ON FUNCTION run_queued_queries_intransaction TO pgstac_admin;\n\nRESET ROLE;\n\nSET ROLE pgstac_ingest;\nSELECT update_partition_stats_q(partition) FROM partitions_view;\nSELECT set_version('0.9.11');\n" }, { "path": "src/pgstac/migrations/pgstac.0.9.2-0.9.3.sql", "content": "SET client_min_messages TO WARNING;\nSET SEARCH_PATH to pgstac, public;\nRESET ROLE;\nDO $$\nDECLARE\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='postgis') THEN\n CREATE EXTENSION IF NOT EXISTS postgis;\n END IF;\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='btree_gist') THEN\n CREATE EXTENSION IF NOT EXISTS btree_gist;\n END IF;\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='unaccent') THEN\n CREATE EXTENSION IF NOT EXISTS unaccent;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_admin;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_read;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_ingest;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n\nGRANT pgstac_admin TO current_user;\n\n-- Function to make sure pgstac_admin is the owner of items\nCREATE OR REPLACE FUNCTION pgstac_admin_owns() RETURNS VOID AS $$\nDECLARE\n f RECORD;\nBEGIN\n FOR f IN (\n SELECT\n concat(\n oid::regproc::text,\n '(',\n coalesce(pg_get_function_identity_arguments(oid),''),\n ')'\n ) AS name,\n CASE prokind WHEN 'f' THEN 'FUNCTION' WHEN 'p' THEN 'PROCEDURE' WHEN 'a' THEN 'AGGREGATE' END as typ\n FROM pg_proc\n WHERE\n pronamespace=to_regnamespace('pgstac')\n AND proowner != to_regrole('pgstac_admin')\n AND proname NOT LIKE 'pg_stat%'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n FOR f IN (\n SELECT\n oid::regclass::text as name,\n CASE relkind\n WHEN 'i' THEN 'INDEX'\n WHEN 'I' THEN 'INDEX'\n WHEN 'p' THEN 'TABLE'\n WHEN 'r' THEN 'TABLE'\n WHEN 'v' THEN 'VIEW'\n WHEN 'S' THEN 'SEQUENCE'\n ELSE NULL\n END as typ\n FROM pg_class\n WHERE relnamespace=to_regnamespace('pgstac') and relowner != to_regrole('pgstac_admin') AND relkind IN ('r','p','v','S') AND relname NOT LIKE 'pg_stat'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\nSELECT pgstac_admin_owns();\n\nCREATE SCHEMA IF NOT EXISTS pgstac AUTHORIZATION pgstac_admin;\n\nGRANT ALL ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_admin;\n\nALTER ROLE pgstac_admin SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_read SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_ingest SET SEARCH_PATH TO pgstac, public;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\n\nGRANT pgstac_read TO pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\n\nSET ROLE pgstac_admin;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\nRESET ROLE;\n\nSET ROLE pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\nRESET ROLE;\n\nSET SEARCH_PATH TO pgstac, public;\nSET ROLE pgstac_admin;\n\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS analyze_items;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS validate_constraints;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n-- Install these idempotently as migrations do not put them before trying to modify the collections table\n\n\nCREATE OR REPLACE FUNCTION collection_geom(content jsonb)\nRETURNS geometry AS $$\n WITH box AS (SELECT content->'extent'->'spatial'->'bbox'->0 as box)\n SELECT\n st_makeenvelope(\n (box->>0)::float,\n (box->>1)::float,\n (box->>2)::float,\n (box->>3)::float,\n 4326\n )\n FROM box;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION collection_datetime(content jsonb)\nRETURNS timestamptz AS $$\n SELECT\n CASE\n WHEN\n (content->'extent'->'temporal'->'interval'->0->>0) IS NULL\n THEN '-infinity'::timestamptz\n ELSE\n (content->'extent'->'temporal'->'interval'->0->>0)::timestamptz\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION collection_enddatetime(content jsonb)\nRETURNS timestamptz AS $$\n SELECT\n CASE\n WHEN\n (content->'extent'->'temporal'->'interval'->0->>1) IS NULL\n THEN 'infinity'::timestamptz\n ELSE\n (content->'extent'->'temporal'->'interval'->0->>1)::timestamptz\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n-- BEGIN migra calculated SQL\nset check_function_bodies = off;\n\nCREATE OR REPLACE FUNCTION pgstac.stac_search_to_where(j jsonb)\n RETURNS text\n LANGUAGE plpgsql\n STABLE\nAS $function$\nDECLARE\n where_segments text[];\n _where text;\n dtrange tstzrange;\n collections text[];\n geom geometry;\n sdate timestamptz;\n edate timestamptz;\n filterlang text;\n filter jsonb := j->'filter';\n ft_query tsquery;\nBEGIN\n IF j ? 'ids' THEN\n where_segments := where_segments || format('id = ANY (%L) ', to_text_array(j->'ids'));\n END IF;\n\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n where_segments := where_segments || format('collection = ANY (%L) ', collections);\n END IF;\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n\n where_segments := where_segments || format(' datetime <= %L::timestamptz AND end_datetime >= %L::timestamptz ',\n edate,\n sdate\n );\n END IF;\n\n IF j ? 'q' THEN\n ft_query := q_to_tsquery(j->>'q');\n where_segments := where_segments || format(\n $quote$\n (\n to_tsvector('english', content->'properties'->>'description') ||\n to_tsvector('english', coalesce(content->'properties'->>'title', '')) ||\n to_tsvector('english', coalesce(content->'properties'->>'keywords', ''))\n ) @@ %L\n $quote$,\n ft_query\n );\n END IF;\n\n geom := stac_geom(j);\n IF geom IS NOT NULL THEN\n where_segments := where_segments || format('st_intersects(geometry, %L)',geom);\n END IF;\n\n filterlang := COALESCE(\n j->>'filter-lang',\n get_setting('default_filter_lang', j->'conf')\n );\n IF NOT filter @? '$.**.op' THEN\n filterlang := 'cql-json';\n END IF;\n\n IF filterlang NOT IN ('cql-json','cql2-json') AND j ? 'filter' THEN\n RAISE EXCEPTION '% is not a supported filter-lang. Please use cql-json or cql2-json.', filterlang;\n END IF;\n\n IF j ? 'query' AND j ? 'filter' THEN\n RAISE EXCEPTION 'Can only use either query or filter at one time.';\n END IF;\n\n IF j ? 'query' THEN\n filter := query_to_cql2(j->'query');\n ELSIF filterlang = 'cql-json' THEN\n filter := cql1_to_cql2(filter);\n END IF;\n RAISE NOTICE 'FILTER: %', filter;\n where_segments := where_segments || cql2_query(filter);\n IF cardinality(where_segments) < 1 THEN\n RETURN ' TRUE ';\n END IF;\n\n _where := array_to_string(array_remove(where_segments, NULL), ' AND ');\n\n IF _where IS NULL OR BTRIM(_where) = '' THEN\n RETURN ' TRUE ';\n END IF;\n RETURN _where;\n\nEND;\n$function$\n;\n\n\n-- END migra calculated SQL\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('id', '{\"title\": \"Item ID\",\"description\": \"Item identifier\",\"$ref\": \"https://schemas.stacspec.org/v1.0.0/item-spec/json-schema/item.json#/definitions/core/allOf/2/properties/id\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('geometry', '{\"title\": \"Item Geometry\",\"description\": \"Item Geometry\",\"$ref\": \"https://geojson.org/schema/Feature.json\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('datetime','{\"description\": \"Datetime\",\"type\": \"string\",\"title\": \"Acquired\",\"format\": \"date-time\",\"pattern\": \"(\\\\+00:00|Z)$\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDELETE FROM queryables a USING queryables b\n WHERE a.name = b.name AND a.collection_ids IS NOT DISTINCT FROM b.collection_ids AND a.id > b.id;\n\n\nINSERT INTO pgstac_settings (name, value) VALUES\n ('context', 'off'),\n ('context_estimated_count', '100000'),\n ('context_estimated_cost', '100000'),\n ('context_stats_ttl', '1 day'),\n ('default_filter_lang', 'cql2-json'),\n ('additional_properties', 'true'),\n ('use_queue', 'false'),\n ('queue_timeout', '10 minutes'),\n ('update_collection_extent', 'false'),\n ('format_cache', 'false'),\n ('readonly', 'false')\nON CONFLICT DO NOTHING\n;\n\n\nINSERT INTO cql2_ops (op, template, types) VALUES\n ('eq', '%s = %s', NULL),\n ('neq', '%s != %s', NULL),\n ('ne', '%s != %s', NULL),\n ('!=', '%s != %s', NULL),\n ('<>', '%s != %s', NULL),\n ('lt', '%s < %s', NULL),\n ('lte', '%s <= %s', NULL),\n ('gt', '%s > %s', NULL),\n ('gte', '%s >= %s', NULL),\n ('le', '%s <= %s', NULL),\n ('ge', '%s >= %s', NULL),\n ('=', '%s = %s', NULL),\n ('<', '%s < %s', NULL),\n ('<=', '%s <= %s', NULL),\n ('>', '%s > %s', NULL),\n ('>=', '%s >= %s', NULL),\n ('like', '%s LIKE %s', NULL),\n ('ilike', '%s ILIKE %s', NULL),\n ('+', '%s + %s', NULL),\n ('-', '%s - %s', NULL),\n ('*', '%s * %s', NULL),\n ('/', '%s / %s', NULL),\n ('not', 'NOT (%s)', NULL),\n ('between', '%s BETWEEN %s AND %s', NULL),\n ('isnull', '%s IS NULL', NULL),\n ('upper', 'upper(%s)', NULL),\n ('lower', 'lower(%s)', NULL),\n ('casei', 'upper(%s)', NULL),\n ('accenti', 'unaccent(%s)', NULL)\nON CONFLICT (op) DO UPDATE\n SET\n template = EXCLUDED.template\n;\n\n\nALTER FUNCTION to_text COST 5000;\nALTER FUNCTION to_float COST 5000;\nALTER FUNCTION to_int COST 5000;\nALTER FUNCTION to_tstz COST 5000;\nALTER FUNCTION to_text_array COST 5000;\n\nALTER FUNCTION update_partition_stats SECURITY DEFINER;\nALTER FUNCTION partition_after_triggerfunc SECURITY DEFINER;\nALTER FUNCTION drop_table_constraints SECURITY DEFINER;\nALTER FUNCTION create_table_constraints SECURITY DEFINER;\nALTER FUNCTION check_partition SECURITY DEFINER;\nALTER FUNCTION repartition SECURITY DEFINER;\nALTER FUNCTION where_stats SECURITY DEFINER;\nALTER FUNCTION search_query SECURITY DEFINER;\nALTER FUNCTION format_item SECURITY DEFINER;\nALTER FUNCTION maintain_index SECURITY DEFINER;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nGRANT ALL ON SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac to pgstac_admin;\n\n-- pgstac_read role limited to using function apis\nGRANT EXECUTE ON FUNCTION search TO pgstac_read;\nGRANT EXECUTE ON FUNCTION search_query TO pgstac_read;\nGRANT EXECUTE ON FUNCTION item_by_id TO pgstac_read;\nGRANT EXECUTE ON FUNCTION get_item TO pgstac_read;\nGRANT SELECT ON ALL TABLES IN SCHEMA pgstac TO pgstac_read;\n\n\nGRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_ingest;\nGRANT USAGE ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_ingest;\n\nREVOKE ALL PRIVILEGES ON PROCEDURE run_queued_queries FROM public;\nGRANT ALL ON PROCEDURE run_queued_queries TO pgstac_admin;\n\nREVOKE ALL PRIVILEGES ON FUNCTION run_queued_queries_intransaction FROM public;\nGRANT ALL ON FUNCTION run_queued_queries_intransaction TO pgstac_admin;\n\nRESET ROLE;\n\nSET ROLE pgstac_ingest;\nSELECT update_partition_stats_q(partition) FROM partitions_view;\nSELECT set_version('0.9.3');\n" }, { "path": "src/pgstac/migrations/pgstac.0.9.2.sql", "content": "RESET ROLE;\nDO $$\nDECLARE\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='postgis') THEN\n CREATE EXTENSION IF NOT EXISTS postgis;\n END IF;\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='btree_gist') THEN\n CREATE EXTENSION IF NOT EXISTS btree_gist;\n END IF;\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='unaccent') THEN\n CREATE EXTENSION IF NOT EXISTS unaccent;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_admin;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_read;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_ingest;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n\nGRANT pgstac_admin TO current_user;\n\n-- Function to make sure pgstac_admin is the owner of items\nCREATE OR REPLACE FUNCTION pgstac_admin_owns() RETURNS VOID AS $$\nDECLARE\n f RECORD;\nBEGIN\n FOR f IN (\n SELECT\n concat(\n oid::regproc::text,\n '(',\n coalesce(pg_get_function_identity_arguments(oid),''),\n ')'\n ) AS name,\n CASE prokind WHEN 'f' THEN 'FUNCTION' WHEN 'p' THEN 'PROCEDURE' WHEN 'a' THEN 'AGGREGATE' END as typ\n FROM pg_proc\n WHERE\n pronamespace=to_regnamespace('pgstac')\n AND proowner != to_regrole('pgstac_admin')\n AND proname NOT LIKE 'pg_stat%'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n FOR f IN (\n SELECT\n oid::regclass::text as name,\n CASE relkind\n WHEN 'i' THEN 'INDEX'\n WHEN 'I' THEN 'INDEX'\n WHEN 'p' THEN 'TABLE'\n WHEN 'r' THEN 'TABLE'\n WHEN 'v' THEN 'VIEW'\n WHEN 'S' THEN 'SEQUENCE'\n ELSE NULL\n END as typ\n FROM pg_class\n WHERE relnamespace=to_regnamespace('pgstac') and relowner != to_regrole('pgstac_admin') AND relkind IN ('r','p','v','S') AND relname NOT LIKE 'pg_stat'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\nSELECT pgstac_admin_owns();\n\nCREATE SCHEMA IF NOT EXISTS pgstac AUTHORIZATION pgstac_admin;\n\nGRANT ALL ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_admin;\n\nALTER ROLE pgstac_admin SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_read SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_ingest SET SEARCH_PATH TO pgstac, public;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\n\nGRANT pgstac_read TO pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\n\nSET ROLE pgstac_admin;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\nRESET ROLE;\n\nSET ROLE pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\nRESET ROLE;\n\nSET SEARCH_PATH TO pgstac, public;\nSET ROLE pgstac_admin;\n\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS analyze_items;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS validate_constraints;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n-- Install these idempotently as migrations do not put them before trying to modify the collections table\n\n\nCREATE OR REPLACE FUNCTION collection_geom(content jsonb)\nRETURNS geometry AS $$\n WITH box AS (SELECT content->'extent'->'spatial'->'bbox'->0 as box)\n SELECT\n st_makeenvelope(\n (box->>0)::float,\n (box->>1)::float,\n (box->>2)::float,\n (box->>3)::float,\n 4326\n )\n FROM box;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION collection_datetime(content jsonb)\nRETURNS timestamptz AS $$\n SELECT\n CASE\n WHEN\n (content->'extent'->'temporal'->'interval'->0->>0) IS NULL\n THEN '-infinity'::timestamptz\n ELSE\n (content->'extent'->'temporal'->'interval'->0->>0)::timestamptz\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION collection_enddatetime(content jsonb)\nRETURNS timestamptz AS $$\n SELECT\n CASE\n WHEN\n (content->'extent'->'temporal'->'interval'->0->>1) IS NULL\n THEN 'infinity'::timestamptz\n ELSE\n (content->'extent'->'temporal'->'interval'->0->>1)::timestamptz\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE TABLE IF NOT EXISTS migrations (\n version text PRIMARY KEY,\n datetime timestamptz DEFAULT clock_timestamp() NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION get_version() RETURNS text AS $$\n SELECT version FROM pgstac.migrations ORDER BY datetime DESC, version DESC LIMIT 1;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION set_version(text) RETURNS text AS $$\n INSERT INTO pgstac.migrations (version) VALUES ($1)\n ON CONFLICT DO NOTHING\n RETURNING version;\n$$ LANGUAGE SQL;\n\n\nCREATE TABLE IF NOT EXISTS pgstac_settings (\n name text PRIMARY KEY,\n value text NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION table_empty(text) RETURNS boolean AS $$\nDECLARE\n retval boolean;\nBEGIN\n EXECUTE format($q$\n SELECT NOT EXISTS (SELECT 1 FROM %I LIMIT 1)\n $q$,\n $1\n ) INTO retval;\n RETURN retval;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION get_setting(IN _setting text, IN conf jsonb DEFAULT NULL) RETURNS text AS $$\nSELECT COALESCE(\n nullif(conf->>_setting, ''),\n nullif(current_setting(concat('pgstac.',_setting), TRUE),''),\n nullif((SELECT value FROM pgstac.pgstac_settings WHERE name=_setting),'')\n);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_setting_bool(IN _setting text, IN conf jsonb DEFAULT NULL) RETURNS boolean AS $$\nSELECT COALESCE(\n nullif(conf->>_setting, ''),\n nullif(current_setting(concat('pgstac.',_setting), TRUE),''),\n nullif((SELECT value FROM pgstac.pgstac_settings WHERE name=_setting),''),\n 'FALSE'\n)::boolean;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION base_url(conf jsonb DEFAULT NULL) RETURNS text AS $$\n SELECT COALESCE(pgstac.get_setting('base_url', conf), '.');\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION additional_properties() RETURNS boolean AS $$\n SELECT pgstac.get_setting_bool('additional_properties');\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION readonly(conf jsonb DEFAULT NULL) RETURNS boolean AS $$\n SELECT pgstac.get_setting_bool('readonly', conf);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context(conf jsonb DEFAULT NULL) RETURNS text AS $$\n SELECT pgstac.get_setting('context', conf);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context_estimated_count(conf jsonb DEFAULT NULL) RETURNS int AS $$\n SELECT pgstac.get_setting('context_estimated_count', conf)::int;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_estimated_cost();\nCREATE OR REPLACE FUNCTION context_estimated_cost(conf jsonb DEFAULT NULL) RETURNS float AS $$\n SELECT pgstac.get_setting('context_estimated_cost', conf)::float;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_stats_ttl();\nCREATE OR REPLACE FUNCTION context_stats_ttl(conf jsonb DEFAULT NULL) RETURNS interval AS $$\n SELECT pgstac.get_setting('context_stats_ttl', conf)::interval;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION t2s(text) RETURNS text AS $$\n SELECT extract(epoch FROM $1::interval)::text || ' s';\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE STRICT;\n\nCREATE OR REPLACE FUNCTION age_ms(a timestamptz, b timestamptz DEFAULT clock_timestamp()) RETURNS float AS $$\n SELECT abs(extract(epoch from age(a,b)) * 1000);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION queue_timeout() RETURNS interval AS $$\n SELECT t2s(coalesce(\n get_setting('queue_timeout'),\n '1h'\n ))::interval;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION notice(VARIADIC text[]) RETURNS boolean AS $$\nDECLARE\ndebug boolean := current_setting('pgstac.debug', true);\nBEGIN\n IF debug THEN\n RAISE NOTICE 'NOTICE FROM FUNC: % >>>>> %', concat_ws(' | ', $1), clock_timestamp();\n RETURN TRUE;\n END IF;\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION empty_arr(ANYARRAY) RETURNS BOOLEAN AS $$\nSELECT CASE\n WHEN $1 IS NULL THEN TRUE\n WHEN cardinality($1)<1 THEN TRUE\nELSE FALSE\nEND;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_intersection(_a ANYARRAY, _b ANYARRAY) RETURNS ANYARRAY AS $$\n SELECT ARRAY ( SELECT unnest(_a) INTERSECT SELECT UNNEST(_b) );\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION array_map_ident(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_ident(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_map_literal(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_literal(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION array_reverse(anyarray) RETURNS anyarray AS $$\nSELECT ARRAY(\n SELECT $1[i]\n FROM generate_subscripts($1,1) AS s(i)\n ORDER BY i DESC\n);\n$$ LANGUAGE SQL STRICT IMMUTABLE;\n\nDROP TABLE IF EXISTS query_queue;\nCREATE TABLE query_queue (\n query text PRIMARY KEY,\n added timestamptz DEFAULT now()\n);\n\nDROP TABLE IF EXISTS query_queue_history;\nCREATE TABLE query_queue_history(\n query text,\n added timestamptz NOT NULL,\n finished timestamptz NOT NULL DEFAULT now(),\n error text\n);\n\nCREATE OR REPLACE PROCEDURE run_queued_queries() AS $$\nDECLARE\n qitem query_queue%ROWTYPE;\n timeout_ts timestamptz;\n error text;\n cnt int := 0;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n DELETE FROM query_queue WHERE query = (SELECT query FROM query_queue ORDER BY added DESC LIMIT 1 FOR UPDATE SKIP LOCKED) RETURNING * INTO qitem;\n IF NOT FOUND THEN\n EXIT;\n END IF;\n cnt := cnt + 1;\n BEGIN\n RAISE NOTICE 'RUNNING QUERY: %', qitem.query;\n EXECUTE qitem.query;\n EXCEPTION WHEN others THEN\n error := format('%s | %s', SQLERRM, SQLSTATE);\n END;\n INSERT INTO query_queue_history (query, added, finished, error)\n VALUES (qitem.query, qitem.added, clock_timestamp(), error);\n COMMIT;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION run_queued_queries_intransaction() RETURNS int AS $$\nDECLARE\n qitem query_queue%ROWTYPE;\n timeout_ts timestamptz;\n error text;\n cnt int := 0;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n DELETE FROM query_queue WHERE query = (SELECT query FROM query_queue ORDER BY added DESC LIMIT 1 FOR UPDATE SKIP LOCKED) RETURNING * INTO qitem;\n IF NOT FOUND THEN\n RETURN cnt;\n END IF;\n cnt := cnt + 1;\n BEGIN\n qitem.query := regexp_replace(qitem.query, 'CONCURRENTLY', '');\n RAISE NOTICE 'RUNNING QUERY: %', qitem.query;\n\n EXECUTE qitem.query;\n EXCEPTION WHEN others THEN\n error := format('%s | %s', SQLERRM, SQLSTATE);\n RAISE WARNING '%', error;\n END;\n INSERT INTO query_queue_history (query, added, finished, error)\n VALUES (qitem.query, qitem.added, clock_timestamp(), error);\n END LOOP;\n RETURN cnt;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION run_or_queue(query text) RETURNS VOID AS $$\nDECLARE\n use_queue text := COALESCE(get_setting('use_queue'), 'FALSE')::boolean;\nBEGIN\n IF get_setting_bool('debug') THEN\n RAISE NOTICE '%', query;\n END IF;\n IF use_queue THEN\n INSERT INTO query_queue (query) VALUES (query) ON CONFLICT DO NOTHING;\n ELSE\n EXECUTE query;\n END IF;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n\nDROP FUNCTION IF EXISTS check_pgstac_settings;\nCREATE OR REPLACE FUNCTION check_pgstac_settings(_sysmem text DEFAULT NULL) RETURNS VOID AS $$\nDECLARE\n settingval text;\n sysmem bigint := pg_size_bytes(_sysmem);\n effective_cache_size bigint := pg_size_bytes(current_setting('effective_cache_size', TRUE));\n shared_buffers bigint := pg_size_bytes(current_setting('shared_buffers', TRUE));\n work_mem bigint := pg_size_bytes(current_setting('work_mem', TRUE));\n max_connections int := current_setting('max_connections', TRUE);\n maintenance_work_mem bigint := pg_size_bytes(current_setting('maintenance_work_mem', TRUE));\n seq_page_cost float := current_setting('seq_page_cost', TRUE);\n random_page_cost float := current_setting('random_page_cost', TRUE);\n temp_buffers bigint := pg_size_bytes(current_setting('temp_buffers', TRUE));\n r record;\nBEGIN\n IF _sysmem IS NULL THEN\n RAISE NOTICE 'Call function with the size of your system memory `SELECT check_pgstac_settings(''4GB'')` to get pg system setting recommendations.';\n ELSE\n IF effective_cache_size < (sysmem * 0.5) THEN\n RAISE WARNING 'effective_cache_size of % is set low for a system with %. Recomended value between % and %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.5), pg_size_pretty(sysmem * 0.75);\n ELSIF effective_cache_size > (sysmem * 0.75) THEN\n RAISE WARNING 'effective_cache_size of % is set high for a system with %. Recomended value between % and %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.5), pg_size_pretty(sysmem * 0.75);\n ELSE\n RAISE NOTICE 'effective_cache_size of % is set appropriately for a system with %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem);\n END IF;\n\n IF shared_buffers < (sysmem * 0.2) THEN\n RAISE WARNING 'shared_buffers of % is set low for a system with %. Recomended value between % and %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.2), pg_size_pretty(sysmem * 0.3);\n ELSIF shared_buffers > (sysmem * 0.3) THEN\n RAISE WARNING 'shared_buffers of % is set high for a system with %. Recomended value between % and %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.2), pg_size_pretty(sysmem * 0.3);\n ELSE\n RAISE NOTICE 'shared_buffers of % is set appropriately for a system with %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem);\n END IF;\n shared_buffers = sysmem * 0.3;\n IF maintenance_work_mem < (sysmem * 0.2) THEN\n RAISE WARNING 'maintenance_work_mem of % is set low for shared_buffers of %. Recomended value between % and %', pg_size_pretty(maintenance_work_mem), pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers * 0.2), pg_size_pretty(shared_buffers * 0.3);\n ELSIF maintenance_work_mem > (shared_buffers * 0.3) THEN\n RAISE WARNING 'maintenance_work_mem of % is set high for shared_buffers of %. Recomended value between % and %', pg_size_pretty(maintenance_work_mem), pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers * 0.2), pg_size_pretty(shared_buffers * 0.3);\n ELSE\n RAISE NOTICE 'maintenance_work_mem of % is set appropriately for shared_buffers of %', pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers);\n END IF;\n\n IF work_mem * max_connections > shared_buffers THEN\n RAISE WARNING 'work_mem setting of % is set high for % max_connections please reduce work_mem to % or decrease max_connections to %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers/max_connections), floor(shared_buffers/work_mem);\n ELSIF work_mem * max_connections < (shared_buffers * 0.75) THEN\n RAISE WARNING 'work_mem setting of % is set low for % max_connections you may consider raising work_mem to % or increasing max_connections to %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers/max_connections), floor(shared_buffers/work_mem);\n ELSE\n RAISE NOTICE 'work_mem setting of % and max_connections of % are adequate for shared_buffers of %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers);\n END IF;\n\n IF random_page_cost / seq_page_cost != 1.1 THEN\n RAISE WARNING 'random_page_cost (%) /seq_page_cost (%) should be set to 1.1 for SSD. Change random_page_cost to %', random_page_cost, seq_page_cost, 1.1 * seq_page_cost;\n ELSE\n RAISE NOTICE 'random_page_cost and seq_page_cost set appropriately for SSD';\n END IF;\n\n IF temp_buffers < greatest(pg_size_bytes('128MB'),(maintenance_work_mem / 2)) THEN\n RAISE WARNING 'pgstac makes heavy use of temp tables, consider raising temp_buffers from % to %', pg_size_pretty(temp_buffers), greatest('128MB', pg_size_pretty((shared_buffers / 16)));\n END IF;\n END IF;\n\n RAISE NOTICE 'VALUES FOR PGSTAC VARIABLES';\n RAISE NOTICE 'These can be set either as GUC system variables or by setting in the pgstac_settings table.';\n\n FOR r IN SELECT name, get_setting(name) as setting, CASE WHEN current_setting(concat('pgstac.',name), TRUE) IS NOT NULL THEN concat('pgstac.',name, ' GUC') WHEN value IS NOT NULL THEN 'pgstac_settings table' ELSE 'Not Set' END as loc FROM pgstac_settings LOOP\n RAISE NOTICE '% is set to % from the %', r.name, r.setting, r.loc;\n END LOOP;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pg_cron';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider intalling pg_cron which can be used to automate tasks';\n ELSE\n RAISE NOTICE 'pg_cron % is installed', settingval;\n END IF;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pgstattuple';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider installing the pgstattuple extension which can be used to help maintain tables and indexes.';\n ELSE\n RAISE NOTICE 'pgstattuple % is installed', settingval;\n END IF;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pg_stat_statements';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider installing the pg_stat_statements extension which is very helpful for tracking the types of queries on the system';\n ELSE\n RAISE NOTICE 'pg_stat_statements % is installed', settingval;\n IF current_setting('pg_stat_statements.track_statements', TRUE) IS DISTINCT FROM 'all' THEN\n RAISE WARNING 'SET pg_stat_statements.track_statements TO ''all''; --In order to track statements within functions.';\n END IF;\n END IF;\n\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac, public SET CLIENT_MIN_MESSAGES TO NOTICE;\nCREATE OR REPLACE FUNCTION to_int(jsonb) RETURNS int AS $$\n SELECT floor(($1->>0)::float)::int;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION to_float(jsonb) RETURNS float AS $$\n SELECT ($1->>0)::float;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION to_tstz(jsonb) RETURNS timestamptz AS $$\n SELECT ($1->>0)::timestamptz;\n$$ LANGUAGE SQL IMMUTABLE STRICT SET TIME ZONE 'UTC' COST 5000 PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION to_text(jsonb) RETURNS text AS $$\n SELECT CASE WHEN jsonb_typeof($1) IN ('array','object') THEN $1::text ELSE $1->>0 END;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION to_text_array(jsonb) RETURNS text[] AS $$\n SELECT\n CASE jsonb_typeof($1)\n WHEN 'array' THEN ARRAY(SELECT jsonb_array_elements_text($1))\n ELSE ARRAY[$1->>0]\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION bbox_geom(_bbox jsonb) RETURNS geometry AS $$\nSELECT CASE jsonb_array_length(_bbox)\n WHEN 4 THEN\n ST_SetSRID(ST_MakeEnvelope(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float,\n (_bbox->>3)::float\n ),4326)\n WHEN 6 THEN\n ST_SetSRID(ST_3DMakeBox(\n ST_MakePoint(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float\n ),\n ST_MakePoint(\n (_bbox->>3)::float,\n (_bbox->>4)::float,\n (_bbox->>5)::float\n )\n ),4326)\n ELSE null END;\n;\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION geom_bbox(_geom geometry) RETURNS jsonb AS $$\n SELECT jsonb_build_array(\n st_xmin(_geom),\n st_ymin(_geom),\n st_xmax(_geom),\n st_ymax(_geom)\n );\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION flip_jsonb_array(j jsonb) RETURNS jsonb AS $$\n SELECT jsonb_agg(value) FROM (SELECT value FROM jsonb_array_elements(j) WITH ORDINALITY ORDER BY ordinality DESC) as t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION explode_dotpaths(j jsonb) RETURNS SETOF text[] AS $$\n SELECT string_to_array(p, '.') as e FROM jsonb_array_elements_text(j) p;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION explode_dotpaths_recurse(IN j jsonb) RETURNS SETOF text[] AS $$\n WITH RECURSIVE t AS (\n SELECT e FROM explode_dotpaths(j) e\n UNION ALL\n SELECT e[1:cardinality(e)-1]\n FROM t\n WHERE cardinality(e)>1\n ) SELECT e FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION jsonb_set_nested(j jsonb, path text[], val jsonb) RETURNS jsonb AS $$\nDECLARE\nBEGIN\n IF cardinality(path) > 1 THEN\n FOR i IN 1..(cardinality(path)-1) LOOP\n IF j #> path[:i] IS NULL THEN\n j := jsonb_set_lax(j, path[:i], '{}', TRUE);\n END IF;\n END LOOP;\n END IF;\n RETURN jsonb_set_lax(j, path, val, true);\n\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\n\n\nCREATE OR REPLACE FUNCTION jsonb_include(j jsonb, f jsonb) RETURNS jsonb AS $$\nDECLARE\n includes jsonb := f-> 'include';\n outj jsonb := '{}'::jsonb;\n path text[];\nBEGIN\n IF\n includes IS NULL\n OR jsonb_array_length(includes) = 0\n THEN\n RETURN j;\n ELSE\n includes := includes || '[\"id\",\"collection\"]'::jsonb;\n FOR path IN SELECT explode_dotpaths(includes) LOOP\n outj := jsonb_set_nested(outj, path, j #> path);\n END LOOP;\n END IF;\n RETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_exclude(j jsonb, f jsonb) RETURNS jsonb AS $$\nDECLARE\n excludes jsonb := f-> 'exclude';\n outj jsonb := j;\n path text[];\nBEGIN\n IF\n excludes IS NULL\n OR jsonb_array_length(excludes) = 0\n THEN\n RETURN j;\n ELSE\n FOR path IN SELECT explode_dotpaths(excludes) LOOP\n outj := outj #- path;\n END LOOP;\n END IF;\n RETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_fields(j jsonb, f jsonb DEFAULT '{\"fields\":[]}') RETURNS jsonb AS $$\n SELECT jsonb_exclude(jsonb_include(j, f), f);\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION merge_jsonb(_a jsonb, _b jsonb) RETURNS jsonb AS $$\n SELECT\n CASE\n WHEN _a = '\"𒍟※\"'::jsonb THEN NULL\n WHEN _a IS NULL OR jsonb_typeof(_a) = 'null' THEN _b\n WHEN jsonb_typeof(_a) = 'object' AND jsonb_typeof(_b) = 'object' THEN\n (\n SELECT\n jsonb_strip_nulls(\n jsonb_object_agg(\n key,\n merge_jsonb(a.value, b.value)\n )\n )\n FROM\n jsonb_each(coalesce(_a,'{}'::jsonb)) as a\n FULL JOIN\n jsonb_each(coalesce(_b,'{}'::jsonb)) as b\n USING (key)\n )\n WHEN\n jsonb_typeof(_a) = 'array'\n AND jsonb_typeof(_b) = 'array'\n AND jsonb_array_length(_a) = jsonb_array_length(_b)\n THEN\n (\n SELECT jsonb_agg(m) FROM\n ( SELECT\n merge_jsonb(\n jsonb_array_elements(_a),\n jsonb_array_elements(_b)\n ) as m\n ) as l\n )\n ELSE _a\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION strip_jsonb(_a jsonb, _b jsonb) RETURNS jsonb AS $$\n SELECT\n CASE\n\n WHEN (_a IS NULL OR jsonb_typeof(_a) = 'null') AND _b IS NOT NULL AND jsonb_typeof(_b) != 'null' THEN '\"𒍟※\"'::jsonb\n WHEN _b IS NULL OR jsonb_typeof(_a) = 'null' THEN _a\n WHEN _a = _b AND jsonb_typeof(_a) = 'object' THEN '{}'::jsonb\n WHEN _a = _b THEN NULL\n WHEN jsonb_typeof(_a) = 'object' AND jsonb_typeof(_b) = 'object' THEN\n (\n SELECT\n jsonb_strip_nulls(\n jsonb_object_agg(\n key,\n strip_jsonb(a.value, b.value)\n )\n )\n FROM\n jsonb_each(_a) as a\n FULL JOIN\n jsonb_each(_b) as b\n USING (key)\n )\n WHEN\n jsonb_typeof(_a) = 'array'\n AND jsonb_typeof(_b) = 'array'\n AND jsonb_array_length(_a) = jsonb_array_length(_b)\n THEN\n (\n SELECT jsonb_agg(m) FROM\n ( SELECT\n strip_jsonb(\n jsonb_array_elements(_a),\n jsonb_array_elements(_b)\n ) as m\n ) as l\n )\n ELSE _a\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION nullif_jsonbnullempty(j jsonb) RETURNS jsonb AS $$\n SELECT nullif(nullif(nullif(j,'null'::jsonb),'{}'::jsonb),'[]'::jsonb);\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION jsonb_array_unique(j jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(jsonb_agg(DISTINCT a)) v FROM jsonb_array_elements(j) a;\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_concat_ignorenull(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT coalesce(a,'[]'::jsonb) || coalesce(b,'[]'::jsonb);\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_least(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(least(nullif_jsonbnullempty(a), nullif_jsonbnullempty(b)));\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_greatest(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(greatest(a, b));\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION first_notnull_sfunc(anyelement, anyelement) RETURNS anyelement AS $$\n SELECT COALESCE($1,$2);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE AGGREGATE first_notnull(anyelement)(\n SFUNC = first_notnull_sfunc,\n STYPE = anyelement\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_array_unique_merge(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_concat_ignorenull,\n FINALFUNC = jsonb_array_unique\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_min(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_least\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_max(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_greatest\n);\n/* looks for a geometry in a stac item first from geometry and falling back to bbox */\nCREATE OR REPLACE FUNCTION stac_geom(value jsonb) RETURNS geometry AS $$\nSELECT\n CASE\n WHEN value ? 'intersects' THEN\n ST_GeomFromGeoJSON(value->>'intersects')\n WHEN value ? 'geometry' THEN\n ST_GeomFromGeoJSON(value->>'geometry')\n WHEN value ? 'bbox' THEN\n pgstac.bbox_geom(value->'bbox')\n ELSE NULL\n END as geometry\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION stac_daterange(\n value jsonb\n) RETURNS tstzrange AS $$\nDECLARE\n props jsonb := value;\n dt timestamptz;\n edt timestamptz;\nBEGIN\n IF props ? 'properties' THEN\n props := props->'properties';\n END IF;\n IF\n props ? 'start_datetime'\n AND props->>'start_datetime' IS NOT NULL\n AND props ? 'end_datetime'\n AND props->>'end_datetime' IS NOT NULL\n THEN\n dt := props->>'start_datetime';\n edt := props->>'end_datetime';\n IF dt > edt THEN\n RAISE EXCEPTION 'start_datetime must be < end_datetime';\n END IF;\n ELSE\n dt := props->>'datetime';\n edt := props->>'datetime';\n END IF;\n IF dt is NULL OR edt IS NULL THEN\n RAISE NOTICE 'DT: %, EDT: %', dt, edt;\n RAISE EXCEPTION 'Either datetime (%) or both start_datetime (%) and end_datetime (%) must be set.', props->>'datetime',props->>'start_datetime',props->>'end_datetime';\n END IF;\n RETURN tstzrange(dt, edt, '[]');\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_datetime(value jsonb) RETURNS timestamptz AS $$\n SELECT lower(stac_daterange(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_end_datetime(value jsonb) RETURNS timestamptz AS $$\n SELECT upper(stac_daterange(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE TABLE IF NOT EXISTS stac_extensions(\n url text PRIMARY KEY,\n content jsonb\n);\nCREATE OR REPLACE FUNCTION collection_base_item(content jsonb) RETURNS jsonb AS $$\n SELECT jsonb_build_object(\n 'type', 'Feature',\n 'stac_version', content->'stac_version',\n 'assets', content->'item_assets',\n 'collection', content->'id'\n );\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE TABLE IF NOT EXISTS collections (\n key bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,\n id text GENERATED ALWAYS AS (content->>'id') STORED UNIQUE NOT NULL,\n content JSONB NOT NULL,\n base_item jsonb GENERATED ALWAYS AS (pgstac.collection_base_item(content)) STORED,\n geometry geometry GENERATED ALWAYS AS (pgstac.collection_geom(content)) STORED,\n datetime timestamptz GENERATED ALWAYS AS (pgstac.collection_datetime(content)) STORED,\n end_datetime timestamptz GENERATED ALWAYS AS (pgstac.collection_enddatetime(content)) STORED,\n private jsonb,\n partition_trunc text CHECK (partition_trunc IN ('year', 'month'))\n);\n\nCREATE OR REPLACE FUNCTION collection_base_item(cid text) RETURNS jsonb AS $$\n SELECT pgstac.collection_base_item(content) FROM pgstac.collections WHERE id = cid LIMIT 1;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION create_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION update_collection(data jsonb) RETURNS VOID AS $$\nDECLARE\n out collections%ROWTYPE;\nBEGIN\n UPDATE collections SET content=data WHERE id = data->>'id' RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ON CONFLICT (id) DO\n UPDATE\n SET content=EXCLUDED.content\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION delete_collection(_id text) RETURNS VOID AS $$\nDECLARE\n out collections%ROWTYPE;\nBEGIN\n DELETE FROM collections WHERE id = _id RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION get_collection(id text) RETURNS jsonb AS $$\n SELECT content FROM collections\n WHERE id=$1\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION all_collections() RETURNS jsonb AS $$\n SELECT coalesce(jsonb_agg(content), '[]'::jsonb) FROM collections;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION collection_delete_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n collection_base_partition text := concat('_items_', OLD.key);\nBEGIN\n EXECUTE format($q$\n DELETE FROM partition_stats WHERE partition IN (\n SELECT partition FROM partition_sys_meta\n WHERE collection=%L\n );\n DROP TABLE IF EXISTS %I CASCADE;\n $q$,\n OLD.id,\n collection_base_partition\n );\n RETURN OLD;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER collection_delete_trigger BEFORE DELETE ON collections\nFOR EACH ROW EXECUTE FUNCTION collection_delete_trigger_func();\nCREATE OR REPLACE FUNCTION queryable_signature(n text, c text[]) RETURNS text AS $$\n SELECT concat(n, c);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TABLE queryables (\n id bigint GENERATED ALWAYS AS identity PRIMARY KEY,\n name text NOT NULL,\n collection_ids text[], -- used to determine what partitions to create indexes on\n definition jsonb,\n property_path text,\n property_wrapper text,\n property_index_type text\n);\nCREATE INDEX queryables_name_idx ON queryables (name);\nCREATE INDEX queryables_collection_idx ON queryables USING GIN (collection_ids);\nCREATE INDEX queryables_property_wrapper_idx ON queryables (property_wrapper);\n\nCREATE OR REPLACE FUNCTION pgstac.queryables_constraint_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n allcollections text[];\nBEGIN\n RAISE NOTICE 'Making sure that name/collection is unique for queryables %', NEW;\n IF NEW.collection_ids IS NOT NULL THEN\n IF EXISTS (\n SELECT 1\n FROM unnest(NEW.collection_ids) c\n LEFT JOIN\n collections\n ON (collections.id = c)\n WHERE collections.id IS NULL\n ) THEN\n RAISE foreign_key_violation USING MESSAGE = format(\n 'One or more collections in %s do not exist.', NEW.collection_ids\n );\n RETURN NULL;\n END IF;\n END IF;\n IF TG_OP = 'INSERT' THEN\n IF EXISTS (\n SELECT 1 FROM queryables q\n WHERE\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n )\n ) THEN\n RAISE unique_violation USING MESSAGE = format(\n 'There is already a queryable for %s for a collection in %s: %s',\n NEW.name,\n NEW.collection_ids,\n\t\t\t\t(SELECT json_agg(row_to_json(q)) FROM queryables q WHERE\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n ))\n );\n RETURN NULL;\n END IF;\n END IF;\n IF TG_OP = 'UPDATE' THEN\n IF EXISTS (\n SELECT 1 FROM queryables q\n WHERE\n q.id != NEW.id\n AND\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n )\n ) THEN\n RAISE unique_violation\n USING MESSAGE = format(\n 'There is already a queryable for %s for a collection in %s',\n NEW.name,\n NEW.collection_ids\n );\n RETURN NULL;\n END IF;\n END IF;\n\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER queryables_constraint_insert_trigger\nBEFORE INSERT ON queryables\nFOR EACH ROW EXECUTE PROCEDURE queryables_constraint_triggerfunc();\n\nCREATE TRIGGER queryables_constraint_update_trigger\nBEFORE UPDATE ON queryables\nFOR EACH ROW\nWHEN (NEW.name = OLD.name AND NEW.collection_ids IS DISTINCT FROM OLD.collection_ids)\nEXECUTE PROCEDURE queryables_constraint_triggerfunc();\n\n\nCREATE OR REPLACE FUNCTION array_to_path(arr text[]) RETURNS text AS $$\n SELECT string_agg(\n quote_literal(v),\n '->'\n ) FROM unnest(arr) v;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\n\n\nCREATE OR REPLACE FUNCTION queryable(\n IN dotpath text,\n OUT path text,\n OUT expression text,\n OUT wrapper text,\n OUT nulled_wrapper text\n) AS $$\nDECLARE\n q RECORD;\n path_elements text[];\nBEGIN\n dotpath := replace(dotpath, 'properties.', '');\n IF dotpath = 'start_datetime' THEN\n dotpath := 'datetime';\n END IF;\n IF dotpath IN ('id', 'geometry', 'datetime', 'end_datetime', 'collection') THEN\n path := dotpath;\n expression := dotpath;\n wrapper := NULL;\n RETURN;\n END IF;\n\n SELECT * INTO q FROM queryables\n WHERE\n name=dotpath\n OR name = 'properties.' || dotpath\n OR name = replace(dotpath, 'properties.', '')\n ;\n IF q.property_wrapper IS NULL THEN\n IF q.definition->>'type' = 'number' THEN\n wrapper := 'to_float';\n nulled_wrapper := wrapper;\n ELSIF q.definition->>'format' = 'date-time' THEN\n wrapper := 'to_tstz';\n nulled_wrapper := wrapper;\n ELSE\n nulled_wrapper := NULL;\n wrapper := 'to_text';\n END IF;\n ELSE\n wrapper := q.property_wrapper;\n nulled_wrapper := wrapper;\n END IF;\n IF q.property_path IS NOT NULL THEN\n path := q.property_path;\n ELSE\n path_elements := string_to_array(dotpath, '.');\n IF path_elements[1] IN ('links', 'assets', 'stac_version', 'stac_extensions') THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSIF path_elements[1] = 'properties' THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSE\n path := format($F$content->'properties'->%s$F$, array_to_path(path_elements));\n END IF;\n END IF;\n expression := format('%I(%s)', wrapper, path);\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\nCREATE OR REPLACE FUNCTION unnest_collection(collection_ids text[] DEFAULT NULL) RETURNS SETOF text AS $$\n DECLARE\n BEGIN\n IF collection_ids IS NULL THEN\n RETURN QUERY SELECT id FROM collections;\n END IF;\n RETURN QUERY SELECT unnest(collection_ids);\n END;\n$$ LANGUAGE PLPGSQL STABLE;\n\nCREATE OR REPLACE FUNCTION normalize_indexdef(def text) RETURNS text AS $$\nDECLARE\nBEGIN\n def := btrim(def, ' \\n\\t');\n\tdef := regexp_replace(def, '^CREATE (UNIQUE )?INDEX ([^ ]* )?ON (ONLY )?([^ ]* )?', '', 'i');\n RETURN def;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION indexdef(q queryables) RETURNS text AS $$\n DECLARE\n out text;\n BEGIN\n IF q.name = 'id' THEN\n out := 'CREATE UNIQUE INDEX ON %I USING btree (id)';\n ELSIF q.name = 'datetime' THEN\n out := 'CREATE INDEX ON %I USING btree (datetime DESC, end_datetime)';\n ELSIF q.name = 'geometry' THEN\n out := 'CREATE INDEX ON %I USING gist (geometry)';\n ELSE\n out := format($q$CREATE INDEX ON %%I USING %s (%s(((content -> 'properties'::text) -> %L::text)))$q$,\n lower(COALESCE(q.property_index_type, 'BTREE')),\n lower(COALESCE(q.property_wrapper, 'to_text')),\n q.name\n );\n END IF;\n RETURN btrim(out, ' \\n\\t');\n END;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nDROP VIEW IF EXISTS pgstac_indexes;\nCREATE VIEW pgstac_indexes AS\nSELECT\n i.schemaname,\n i.tablename,\n i.indexname,\n regexp_replace(btrim(replace(replace(indexdef, i.indexname, ''),'pgstac.',''),' \\t\\n'), '[ ]+', ' ', 'g') as idx,\n COALESCE(\n (regexp_match(indexdef, '\\(([a-zA-Z]+)\\)'))[1],\n (regexp_match(indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_-]+)''::text'))[1],\n CASE WHEN indexdef ~* '\\(datetime desc, end_datetime\\)' THEN 'datetime' ELSE NULL END\n ) AS field,\n pg_table_size(i.indexname::text) as index_size,\n pg_size_pretty(pg_table_size(i.indexname::text)) as index_size_pretty\nFROM\n pg_indexes i\nWHERE i.schemaname='pgstac' and i.tablename ~ '_items_' AND indexdef !~* ' only ';\n\nDROP VIEW IF EXISTS pgstac_index_stats;\nCREATE VIEW pgstac_indexes_stats AS\nSELECT\n i.schemaname,\n i.tablename,\n i.indexname,\n indexdef,\n COALESCE(\n (regexp_match(indexdef, '\\(([a-zA-Z]+)\\)'))[1],\n (regexp_match(indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_]+)''::text'))[1],\n CASE WHEN indexdef ~* '\\(datetime desc, end_datetime\\)' THEN 'datetime_end_datetime' ELSE NULL END\n ) AS field,\n pg_table_size(i.indexname::text) as index_size,\n pg_size_pretty(pg_table_size(i.indexname::text)) as index_size_pretty,\n n_distinct,\n most_common_vals::text::text[],\n most_common_freqs::text::text[],\n histogram_bounds::text::text[],\n correlation\nFROM\n pg_indexes i\n LEFT JOIN pg_stats s ON (s.tablename = i.indexname)\nWHERE i.schemaname='pgstac' and i.tablename ~ '_items_';\n\nCREATE OR REPLACE FUNCTION queryable_indexes(\n IN treeroot text DEFAULT 'items',\n IN changes boolean DEFAULT FALSE,\n OUT collection text,\n OUT partition text,\n OUT field text,\n OUT indexname text,\n OUT existing_idx text,\n OUT queryable_idx text\n) RETURNS SETOF RECORD AS $$\nWITH p AS (\n SELECT\n relid::text as partition,\n replace(replace(\n CASE\n WHEN parentrelid::regclass::text='items' THEN pg_get_expr(c.relpartbound, c.oid)\n ELSE pg_get_expr(parent.relpartbound, parent.oid)\n END,\n 'FOR VALUES IN (''',''), ''')',\n ''\n ) AS collection\n FROM pg_partition_tree(treeroot)\n JOIN pg_class c ON (relid::regclass = c.oid)\n JOIN pg_class parent ON (parentrelid::regclass = parent.oid AND isleaf)\n ), i AS (\n SELECT\n partition,\n indexname,\n regexp_replace(btrim(replace(replace(indexdef, indexname, ''),'pgstac.',''),' \\t\\n'), '[ ]+', ' ', 'g') as iidx,\n COALESCE(\n (regexp_match(indexdef, '\\(([a-zA-Z]+)\\)'))[1],\n (regexp_match(indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_-]+)''::text'))[1],\n CASE WHEN indexdef ~* '\\(datetime desc, end_datetime\\)' THEN 'datetime' ELSE NULL END\n ) AS field\n FROM\n pg_indexes\n JOIN p ON (tablename=partition)\n ), q AS (\n SELECT\n name AS field,\n collection,\n partition,\n format(indexdef(queryables), partition) as qidx\n FROM queryables, unnest_collection(queryables.collection_ids) collection\n JOIN p USING (collection)\n WHERE property_index_type IS NOT NULL OR name IN ('datetime','geometry','id')\n )\n SELECT\n collection,\n partition,\n field,\n indexname,\n iidx as existing_idx,\n qidx as queryable_idx\n FROM i FULL JOIN q USING (field, partition)\n WHERE CASE WHEN changes THEN lower(iidx) IS DISTINCT FROM lower(qidx) ELSE TRUE END;\n;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION maintain_index(\n indexname text,\n queryable_idx text,\n dropindexes boolean DEFAULT FALSE,\n rebuildindexes boolean DEFAULT FALSE,\n idxconcurrently boolean DEFAULT FALSE\n) RETURNS VOID AS $$\nDECLARE\nBEGIN\n IF indexname IS NOT NULL THEN\n IF dropindexes OR queryable_idx IS NOT NULL THEN\n EXECUTE format('DROP INDEX IF EXISTS %I;', indexname);\n ELSIF rebuildindexes THEN\n IF idxconcurrently THEN\n EXECUTE format('REINDEX INDEX CONCURRENTLY %I;', indexname);\n ELSE\n EXECUTE format('REINDEX INDEX CONCURRENTLY %I;', indexname);\n END IF;\n END IF;\n END IF;\n IF queryable_idx IS NOT NULL THEN\n IF idxconcurrently THEN\n EXECUTE replace(queryable_idx, 'INDEX', 'INDEX CONCURRENTLY');\n ELSE EXECUTE queryable_idx;\n END IF;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\n\nset check_function_bodies to off;\nCREATE OR REPLACE FUNCTION maintain_partition_queries(\n part text DEFAULT 'items',\n dropindexes boolean DEFAULT FALSE,\n rebuildindexes boolean DEFAULT FALSE,\n idxconcurrently boolean DEFAULT FALSE\n) RETURNS SETOF text AS $$\nDECLARE\n rec record;\n q text;\nBEGIN\n FOR rec IN (\n SELECT * FROM queryable_indexes(part,true)\n ) LOOP\n q := format(\n 'SELECT maintain_index(\n %L,%L,%L,%L,%L\n );',\n rec.indexname,\n rec.queryable_idx,\n dropindexes,\n rebuildindexes,\n idxconcurrently\n );\n RAISE NOTICE 'Q: %', q;\n RETURN NEXT q;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION maintain_partitions(\n part text DEFAULT 'items',\n dropindexes boolean DEFAULT FALSE,\n rebuildindexes boolean DEFAULT FALSE\n) RETURNS VOID AS $$\n WITH t AS (\n SELECT run_or_queue(q) FROM maintain_partition_queries(part, dropindexes, rebuildindexes) q\n ) SELECT count(*) FROM t;\n$$ LANGUAGE SQL;\n\n\nCREATE OR REPLACE FUNCTION queryables_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\nBEGIN\n PERFORM maintain_partitions();\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER queryables_trigger AFTER INSERT OR UPDATE ON queryables\nFOR EACH STATEMENT EXECUTE PROCEDURE queryables_trigger_func();\n\n\nCREATE OR REPLACE FUNCTION get_queryables(_collection_ids text[] DEFAULT NULL) RETURNS jsonb AS $$\nDECLARE\nBEGIN\n -- Build up queryables if the input contains valid collection ids or is empty\n IF EXISTS (\n SELECT 1 FROM collections\n WHERE\n _collection_ids IS NULL\n OR cardinality(_collection_ids) = 0\n OR id = ANY(_collection_ids)\n )\n THEN\n RETURN (\n WITH base AS (\n SELECT\n unnest(collection_ids) as collection_id,\n name,\n coalesce(definition, '{\"type\":\"string\"}'::jsonb) as definition\n FROM queryables\n WHERE\n _collection_ids IS NULL OR\n _collection_ids = '{}'::text[] OR\n _collection_ids && collection_ids\n UNION ALL\n SELECT null, name, coalesce(definition, '{\"type\":\"string\"}'::jsonb) as definition\n FROM queryables WHERE collection_ids IS NULL OR collection_ids = '{}'::text[]\n ), g AS (\n SELECT\n name,\n first_notnull(definition) as definition,\n jsonb_array_unique_merge(definition->'enum') as enum,\n jsonb_min(definition->'minimum') as minimum,\n jsonb_min(definition->'maxiumn') as maximum\n FROM base\n GROUP BY 1\n )\n SELECT\n jsonb_build_object(\n '$schema', 'http://json-schema.org/draft-07/schema#',\n '$id', '',\n 'type', 'object',\n 'title', 'STAC Queryables.',\n 'properties', jsonb_object_agg(\n name,\n definition\n ||\n jsonb_strip_nulls(jsonb_build_object(\n 'enum', enum,\n 'minimum', minimum,\n 'maximum', maximum\n ))\n ),\n 'additionalProperties', pgstac.additional_properties()\n )\n FROM g\n );\n ELSE\n RETURN NULL;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\nCREATE OR REPLACE FUNCTION get_queryables(_collection text DEFAULT NULL) RETURNS jsonb AS $$\n SELECT\n CASE\n WHEN _collection IS NULL THEN get_queryables(NULL::text[])\n ELSE get_queryables(ARRAY[_collection])\n END\n ;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_queryables() RETURNS jsonb AS $$\n SELECT get_queryables(NULL::text[]);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION schema_qualify_refs(url text, j jsonb) returns jsonb as $$\n SELECT regexp_replace(j::text, '\"\\$ref\": \"#', concat('\"$ref\": \"', url, '#'), 'g')::jsonb;\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE VIEW stac_extension_queryables AS\nSELECT DISTINCT key as name, schema_qualify_refs(e.url, j.value) as definition FROM stac_extensions e, jsonb_each(e.content->'definitions'->'fields'->'properties') j;\n\n\nCREATE OR REPLACE FUNCTION missing_queryables(_collection text, _tablesample float DEFAULT 5, minrows float DEFAULT 10) RETURNS TABLE(collection text, name text, definition jsonb, property_wrapper text) AS $$\nDECLARE\n q text;\n _partition text;\n explain_json json;\n psize float;\n estrows float;\nBEGIN\n SELECT format('_items_%s', key) INTO _partition FROM collections WHERE id=_collection;\n\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM %I;', _partition)\n INTO explain_json;\n psize := explain_json->0->'Plan'->'Plan Rows';\n estrows := _tablesample * .01 * psize;\n IF estrows < minrows THEN\n _tablesample := least(100,greatest(_tablesample, (estrows / psize) / 100));\n RAISE NOTICE '%', (psize / estrows) / 100;\n END IF;\n RAISE NOTICE 'Using tablesample % to find missing queryables from % % that has ~% rows estrows: %', _tablesample, _collection, _partition, psize, estrows;\n\n q := format(\n $q$\n WITH q AS (\n SELECT * FROM queryables\n WHERE\n collection_ids IS NULL\n OR %L = ANY(collection_ids)\n ), t AS (\n SELECT\n content->'properties' AS properties\n FROM\n %I\n TABLESAMPLE SYSTEM(%L)\n ), p AS (\n SELECT DISTINCT ON (key)\n key,\n value,\n s.definition\n FROM t\n JOIN LATERAL jsonb_each(properties) ON TRUE\n LEFT JOIN q ON (q.name=key)\n LEFT JOIN stac_extension_queryables s ON (s.name=key)\n WHERE q.definition IS NULL\n )\n SELECT\n %L,\n key,\n COALESCE(definition, jsonb_build_object('type',jsonb_typeof(value))) as definition,\n CASE\n WHEN definition->>'type' = 'integer' THEN 'to_int'\n WHEN COALESCE(definition->>'type', jsonb_typeof(value)) = 'number' THEN 'to_float'\n WHEN COALESCE(definition->>'type', jsonb_typeof(value)) = 'array' THEN 'to_text_array'\n ELSE 'to_text'\n END\n FROM p;\n $q$,\n _collection,\n _partition,\n _tablesample,\n _collection\n );\n RETURN QUERY EXECUTE q;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION missing_queryables(_tablesample float DEFAULT 5) RETURNS TABLE(collection_ids text[], name text, definition jsonb, property_wrapper text) AS $$\n SELECT\n array_agg(collection),\n name,\n definition,\n property_wrapper\n FROM\n collections\n JOIN LATERAL\n missing_queryables(id, _tablesample) c\n ON TRUE\n GROUP BY\n 2,3,4\n ORDER BY 2,1\n ;\n$$ LANGUAGE SQL;\nCREATE OR REPLACE FUNCTION parse_dtrange(\n _indate jsonb,\n relative_base timestamptz DEFAULT date_trunc('hour', CURRENT_TIMESTAMP)\n) RETURNS tstzrange AS $$\nDECLARE\n timestrs text[];\n s timestamptz;\n e timestamptz;\nBEGIN\n timestrs :=\n CASE\n WHEN _indate ? 'timestamp' THEN\n ARRAY[_indate->>'timestamp']\n WHEN _indate ? 'interval' THEN\n to_text_array(_indate->'interval')\n WHEN jsonb_typeof(_indate) = 'array' THEN\n to_text_array(_indate)\n ELSE\n regexp_split_to_array(\n _indate->>0,\n '/'\n )\n END;\n RAISE NOTICE 'TIMESTRS %', timestrs;\n IF cardinality(timestrs) = 1 THEN\n IF timestrs[1] ILIKE 'P%' THEN\n RETURN tstzrange(relative_base - upper(timestrs[1])::interval, relative_base, '[)');\n END IF;\n s := timestrs[1]::timestamptz;\n RETURN tstzrange(s, s, '[]');\n END IF;\n\n IF cardinality(timestrs) != 2 THEN\n RAISE EXCEPTION 'Timestamp cannot have more than 2 values';\n END IF;\n\n IF timestrs[1] = '..' OR timestrs[1] = '' THEN\n s := '-infinity'::timestamptz;\n e := timestrs[2]::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] = '..' OR timestrs[2] = '' THEN\n s := timestrs[1]::timestamptz;\n e := 'infinity'::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[1] ILIKE 'P%' AND timestrs[2] NOT ILIKE 'P%' THEN\n e := timestrs[2]::timestamptz;\n s := e - upper(timestrs[1])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] ILIKE 'P%' AND timestrs[1] NOT ILIKE 'P%' THEN\n s := timestrs[1]::timestamptz;\n e := s + upper(timestrs[2])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n s := timestrs[1]::timestamptz;\n e := timestrs[2]::timestamptz;\n\n RETURN tstzrange(s,e,'[)');\n\n RETURN NULL;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT PARALLEL SAFE SET TIME ZONE 'UTC';\n\nCREATE OR REPLACE FUNCTION parse_dtrange(\n _indate text,\n relative_base timestamptz DEFAULT CURRENT_TIMESTAMP\n) RETURNS tstzrange AS $$\n SELECT parse_dtrange(to_jsonb(_indate), relative_base);\n$$ LANGUAGE SQL STABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION temporal_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\n ll text := 'datetime';\n lh text := 'end_datetime';\n rrange tstzrange;\n rl text;\n rh text;\n outq text;\nBEGIN\n rrange := parse_dtrange(args->1);\n RAISE NOTICE 'Constructing temporal query OP: %, ARGS: %, RRANGE: %', op, args, rrange;\n op := lower(op);\n rl := format('%L::timestamptz', lower(rrange));\n rh := format('%L::timestamptz', upper(rrange));\n outq := CASE op\n WHEN 't_before' THEN 'lh < rl'\n WHEN 't_after' THEN 'll > rh'\n WHEN 't_meets' THEN 'lh = rl'\n WHEN 't_metby' THEN 'll = rh'\n WHEN 't_overlaps' THEN 'll < rl AND rl < lh < rh'\n WHEN 't_overlappedby' THEN 'rl < ll < rh AND lh > rh'\n WHEN 't_starts' THEN 'll = rl AND lh < rh'\n WHEN 't_startedby' THEN 'll = rl AND lh > rh'\n WHEN 't_during' THEN 'll > rl AND lh < rh'\n WHEN 't_contains' THEN 'll < rl AND lh > rh'\n WHEN 't_finishes' THEN 'll > rl AND lh = rh'\n WHEN 't_finishedby' THEN 'll < rl AND lh = rh'\n WHEN 't_equals' THEN 'll = rl AND lh = rh'\n WHEN 't_disjoint' THEN 'NOT (ll <= rh AND lh >= rl)'\n WHEN 't_intersects' THEN 'll <= rh AND lh >= rl'\n WHEN 'anyinteracts' THEN 'll <= rh AND lh >= rl'\n END;\n outq := regexp_replace(outq, '\\mll\\M', ll);\n outq := regexp_replace(outq, '\\mlh\\M', lh);\n outq := regexp_replace(outq, '\\mrl\\M', rl);\n outq := regexp_replace(outq, '\\mrh\\M', rh);\n outq := format('(%s)', outq);\n RETURN outq;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\n\n\nCREATE OR REPLACE FUNCTION spatial_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\n geom text;\n j jsonb := args->1;\nBEGIN\n op := lower(op);\n RAISE NOTICE 'Constructing spatial query OP: %, ARGS: %', op, args;\n IF op NOT IN ('s_equals','s_disjoint','s_touches','s_within','s_overlaps','s_crosses','s_intersects','intersects','s_contains') THEN\n RAISE EXCEPTION 'Spatial Operator % Not Supported', op;\n END IF;\n op := regexp_replace(op, '^s_', 'st_');\n IF op = 'intersects' THEN\n op := 'st_intersects';\n END IF;\n -- Convert geometry to WKB string\n IF j ? 'type' AND j ? 'coordinates' THEN\n geom := st_geomfromgeojson(j)::text;\n ELSIF jsonb_typeof(j) = 'array' THEN\n geom := bbox_geom(j)::text;\n END IF;\n\n RETURN format('%s(geometry, %L::geometry)', op, geom);\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION query_to_cql2(q jsonb) RETURNS jsonb AS $$\n-- Translates anything passed in through the deprecated \"query\" into equivalent CQL2\nWITH t AS (\n SELECT key as property, value as ops\n FROM jsonb_each(q)\n), t2 AS (\n SELECT property, (jsonb_each(ops)).*\n FROM t WHERE jsonb_typeof(ops) = 'object'\n UNION ALL\n SELECT property, 'eq', ops\n FROM t WHERE jsonb_typeof(ops) != 'object'\n)\nSELECT\n jsonb_strip_nulls(jsonb_build_object(\n 'op', 'and',\n 'args', jsonb_agg(\n jsonb_build_object(\n 'op', key,\n 'args', jsonb_build_array(\n jsonb_build_object('property',property),\n value\n )\n )\n )\n )\n) as qcql FROM t2\n;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\nCREATE OR REPLACE FUNCTION cql1_to_cql2(j jsonb) RETURNS jsonb AS $$\nDECLARE\n args jsonb;\n ret jsonb;\nBEGIN\n RAISE NOTICE 'CQL1_TO_CQL2: %', j;\n IF j ? 'filter' THEN\n RETURN cql1_to_cql2(j->'filter');\n END IF;\n IF j ? 'property' THEN\n RETURN j;\n END IF;\n IF jsonb_typeof(j) = 'array' THEN\n SELECT jsonb_agg(cql1_to_cql2(el)) INTO args FROM jsonb_array_elements(j) el;\n RETURN args;\n END IF;\n IF jsonb_typeof(j) = 'number' THEN\n RETURN j;\n END IF;\n IF jsonb_typeof(j) = 'string' THEN\n RETURN j;\n END IF;\n\n IF jsonb_typeof(j) = 'object' THEN\n SELECT jsonb_build_object(\n 'op', key,\n 'args', cql1_to_cql2(value)\n ) INTO ret\n FROM jsonb_each(j)\n WHERE j IS NOT NULL;\n RETURN ret;\n END IF;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE STRICT;\n\nCREATE TABLE cql2_ops (\n op text PRIMARY KEY,\n template text,\n types text[]\n);\nINSERT INTO cql2_ops (op, template, types) VALUES\n ('eq', '%s = %s', NULL),\n ('neq', '%s != %s', NULL),\n ('ne', '%s != %s', NULL),\n ('!=', '%s != %s', NULL),\n ('<>', '%s != %s', NULL),\n ('lt', '%s < %s', NULL),\n ('lte', '%s <= %s', NULL),\n ('gt', '%s > %s', NULL),\n ('gte', '%s >= %s', NULL),\n ('le', '%s <= %s', NULL),\n ('ge', '%s >= %s', NULL),\n ('=', '%s = %s', NULL),\n ('<', '%s < %s', NULL),\n ('<=', '%s <= %s', NULL),\n ('>', '%s > %s', NULL),\n ('>=', '%s >= %s', NULL),\n ('like', '%s LIKE %s', NULL),\n ('ilike', '%s ILIKE %s', NULL),\n ('+', '%s + %s', NULL),\n ('-', '%s - %s', NULL),\n ('*', '%s * %s', NULL),\n ('/', '%s / %s', NULL),\n ('not', 'NOT (%s)', NULL),\n ('between', '%s BETWEEN %s AND %s', NULL),\n ('isnull', '%s IS NULL', NULL),\n ('upper', 'upper(%s)', NULL),\n ('lower', 'lower(%s)', NULL),\n ('casei', 'upper(%s)', NULL),\n ('accenti', 'unaccent(%s)', NULL)\nON CONFLICT (op) DO UPDATE\n SET\n template = EXCLUDED.template\n;\n\n\nCREATE OR REPLACE FUNCTION cql2_query(j jsonb, wrapper text DEFAULT NULL) RETURNS text AS $$\n#variable_conflict use_variable\nDECLARE\n args jsonb := j->'args';\n arg jsonb;\n op text := lower(j->>'op');\n cql2op RECORD;\n literal text;\n _wrapper text;\n leftarg text;\n rightarg text;\n prop text;\n extra_props bool := pgstac.additional_properties();\nBEGIN\n IF j IS NULL OR (op IS NOT NULL AND args IS NULL) THEN\n RETURN NULL;\n END IF;\n RAISE NOTICE 'CQL2_QUERY: %', j;\n\n -- check if all properties are represented in the queryables\n IF NOT extra_props THEN\n FOR prop IN\n SELECT DISTINCT p->>0\n FROM jsonb_path_query(j, 'strict $.**.property') p\n WHERE p->>0 NOT IN ('id', 'datetime', 'geometry', 'end_datetime', 'collection')\n LOOP\n IF (queryable(prop)).nulled_wrapper IS NULL THEN\n RAISE EXCEPTION 'Term % is not found in queryables.', prop;\n END IF;\n END LOOP;\n END IF;\n\n IF j ? 'filter' THEN\n RETURN cql2_query(j->'filter');\n END IF;\n\n IF j ? 'upper' THEN\n RETURN cql2_query(jsonb_build_object('op', 'upper', 'args', j->'upper'));\n END IF;\n\n IF j ? 'lower' THEN\n RETURN cql2_query(jsonb_build_object('op', 'lower', 'args', j->'lower'));\n END IF;\n\n -- Temporal Query\n IF op ilike 't_%' or op = 'anyinteracts' THEN\n RETURN temporal_op_query(op, args);\n END IF;\n\n -- If property is a timestamp convert it to text to use with\n -- general operators\n IF j ? 'timestamp' THEN\n RETURN format('%L::timestamptz', to_tstz(j->'timestamp'));\n END IF;\n IF j ? 'interval' THEN\n RAISE EXCEPTION 'Please use temporal operators when using intervals.';\n RETURN NONE;\n END IF;\n\n -- Spatial Query\n IF op ilike 's_%' or op = 'intersects' THEN\n RETURN spatial_op_query(op, args);\n END IF;\n\n IF op IN ('a_equals','a_contains','a_contained_by','a_overlaps') THEN\n IF args->0 ? 'property' THEN\n leftarg := format('to_text_array(%s)', (queryable(args->0->>'property')).path);\n END IF;\n IF args->1 ? 'property' THEN\n rightarg := format('to_text_array(%s)', (queryable(args->1->>'property')).path);\n END IF;\n RETURN FORMAT(\n '%s %s %s',\n COALESCE(leftarg, quote_literal(to_text_array(args->0))),\n CASE op\n WHEN 'a_equals' THEN '='\n WHEN 'a_contains' THEN '@>'\n WHEN 'a_contained_by' THEN '<@'\n WHEN 'a_overlaps' THEN '&&'\n END,\n COALESCE(rightarg, quote_literal(to_text_array(args->1)))\n );\n END IF;\n\n IF op = 'in' THEN\n RAISE NOTICE 'IN : % % %', args, jsonb_build_array(args->0), args->1;\n args := jsonb_build_array(args->0) || (args->1);\n RAISE NOTICE 'IN2 : %', args;\n END IF;\n\n\n\n IF op = 'between' THEN\n args = jsonb_build_array(\n args->0,\n args->1,\n args->2\n );\n END IF;\n\n -- Make sure that args is an array and run cql2_query on\n -- each element of the array\n RAISE NOTICE 'ARGS PRE: %', args;\n IF j ? 'args' THEN\n IF jsonb_typeof(args) != 'array' THEN\n args := jsonb_build_array(args);\n END IF;\n\n IF jsonb_path_exists(args, '$[*] ? (@.property == \"id\" || @.property == \"datetime\" || @.property == \"end_datetime\" || @.property == \"collection\")') THEN\n wrapper := NULL;\n ELSE\n -- if any of the arguments are a property, try to get the property_wrapper\n FOR arg IN SELECT jsonb_path_query(args, '$[*] ? (@.property != null)') LOOP\n RAISE NOTICE 'Arg: %', arg;\n wrapper := (queryable(arg->>'property')).nulled_wrapper;\n RAISE NOTICE 'Property: %, Wrapper: %', arg, wrapper;\n IF wrapper IS NOT NULL THEN\n EXIT;\n END IF;\n END LOOP;\n\n -- if the property was not in queryables, see if any args were numbers\n IF\n wrapper IS NULL\n AND jsonb_path_exists(args, '$[*] ? (@.type()==\"number\")')\n THEN\n wrapper := 'to_float';\n END IF;\n wrapper := coalesce(wrapper, 'to_text');\n END IF;\n\n SELECT jsonb_agg(cql2_query(a, wrapper))\n INTO args\n FROM jsonb_array_elements(args) a;\n END IF;\n RAISE NOTICE 'ARGS: %', args;\n\n IF op IN ('and', 'or') THEN\n RETURN\n format(\n '(%s)',\n array_to_string(to_text_array(args), format(' %s ', upper(op)))\n );\n END IF;\n\n IF op = 'in' THEN\n RAISE NOTICE 'IN -- % %', args->0, to_text(args->0);\n RETURN format(\n '%s IN (%s)',\n to_text(args->0),\n array_to_string((to_text_array(args))[2:], ',')\n );\n END IF;\n\n -- Look up template from cql2_ops\n IF j ? 'op' THEN\n SELECT * INTO cql2op FROM cql2_ops WHERE cql2_ops.op ilike op;\n IF FOUND THEN\n -- If specific index set in queryables for a property cast other arguments to that type\n\n RETURN format(\n cql2op.template,\n VARIADIC (to_text_array(args))\n );\n ELSE\n RAISE EXCEPTION 'Operator % Not Supported.', op;\n END IF;\n END IF;\n\n\n IF wrapper IS NOT NULL THEN\n RAISE NOTICE 'Wrapping % with %', j, wrapper;\n IF j ? 'property' THEN\n RETURN format('%I(%s)', wrapper, (queryable(j->>'property')).path);\n ELSE\n RETURN format('%I(%L)', wrapper, j);\n END IF;\n ELSIF j ? 'property' THEN\n RETURN quote_ident(j->>'property');\n END IF;\n\n RETURN quote_literal(to_text(j));\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION paging_dtrange(\n j jsonb\n) RETURNS tstzrange AS $$\nDECLARE\n op text;\n filter jsonb := j->'filter';\n dtrange tstzrange := tstzrange('-infinity'::timestamptz,'infinity'::timestamptz);\n sdate timestamptz := '-infinity'::timestamptz;\n edate timestamptz := 'infinity'::timestamptz;\n jpitem jsonb;\nBEGIN\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n END IF;\n IF NOT (filter @? '$.**.op ? (@ == \"or\" || @ == \"not\")') THEN\n FOR jpitem IN SELECT j FROM jsonb_path_query(filter,'strict $.** ? (@.args[*].property == \"datetime\")'::jsonpath) j LOOP\n op := lower(jpitem->>'op');\n dtrange := parse_dtrange(jpitem->'args'->1);\n IF op IN ('<=', 'lt', 'lte', '<', 'le', 't_before') THEN\n sdate := greatest(sdate,'-infinity');\n edate := least(edate, upper(dtrange));\n ELSIF op IN ('>=', '>', 'gt', 'gte', 'ge', 't_after') THEN\n edate := least(edate, 'infinity');\n sdate := greatest(sdate, lower(dtrange));\n ELSIF op IN ('=', 'eq') THEN\n edate := least(edate, upper(dtrange));\n sdate := greatest(sdate, lower(dtrange));\n END IF;\n RAISE NOTICE '2 OP: %, ARGS: %, DTRANGE: %, SDATE: %, EDATE: %', op, jpitem->'args'->1, dtrange, sdate, edate;\n END LOOP;\n END IF;\n IF sdate > edate THEN\n RETURN 'empty'::tstzrange;\n END IF;\n RETURN tstzrange(sdate,edate, '[]');\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT SET TIME ZONE 'UTC';\n\nCREATE OR REPLACE FUNCTION paging_collections(\n IN j jsonb\n) RETURNS text[] AS $$\nDECLARE\n filter jsonb := j->'filter';\n jpitem jsonb;\n op text;\n args jsonb;\n arg jsonb;\n collections text[];\nBEGIN\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n END IF;\n IF NOT (filter @? '$.**.op ? (@ == \"or\" || @ == \"not\")') THEN\n FOR jpitem IN SELECT j FROM jsonb_path_query(filter,'strict $.** ? (@.args[*].property == \"collection\")'::jsonpath) j LOOP\n RAISE NOTICE 'JPITEM: %', jpitem;\n op := jpitem->>'op';\n args := jpitem->'args';\n IF op IN ('=', 'eq', 'in') THEN\n FOR arg IN SELECT a FROM jsonb_array_elements(args) a LOOP\n IF jsonb_typeof(arg) IN ('string', 'array') THEN\n RAISE NOTICE 'arg: %, collections: %', arg, collections;\n IF collections IS NULL OR collections = '{}'::text[] THEN\n collections := to_text_array(arg);\n ELSE\n collections := array_intersection(collections, to_text_array(arg));\n END IF;\n END IF;\n END LOOP;\n END IF;\n END LOOP;\n END IF;\n IF collections = '{}'::text[] THEN\n RETURN NULL;\n END IF;\n RETURN collections;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\nCREATE TABLE items (\n id text NOT NULL,\n geometry geometry NOT NULL,\n collection text NOT NULL,\n datetime timestamptz NOT NULL,\n end_datetime timestamptz NOT NULL,\n content JSONB NOT NULL,\n private jsonb\n)\nPARTITION BY LIST (collection)\n;\n\nCREATE INDEX \"datetime_idx\" ON items USING BTREE (datetime DESC, end_datetime ASC);\nCREATE INDEX \"geometry_idx\" ON items USING GIST (geometry);\n\nCREATE STATISTICS datetime_stats (dependencies) on datetime, end_datetime from items;\n\nALTER TABLE items ADD CONSTRAINT items_collections_fk FOREIGN KEY (collection) REFERENCES collections(id) ON DELETE CASCADE DEFERRABLE;\n\nCREATE OR REPLACE FUNCTION partition_after_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p text;\n t timestamptz := clock_timestamp();\nBEGIN\n RAISE NOTICE 'Updating partition stats %', t;\n FOR p IN SELECT DISTINCT partition\n FROM newdata n JOIN partition_sys_meta p\n ON (n.collection=p.collection AND n.datetime <@ p.partition_dtrange)\n LOOP\n PERFORM run_or_queue(format('SELECT update_partition_stats(%L, %L);', p, true));\n END LOOP;\n IF TG_OP IN ('DELETE','UPDATE') THEN\n DELETE FROM format_item_cache c USING newdata n WHERE c.collection = n.collection AND c.id = n.id;\n END IF;\n RAISE NOTICE 't: % %', t, clock_timestamp() - t;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE TRIGGER items_after_insert_trigger\nAFTER INSERT ON items\nREFERENCING NEW TABLE AS newdata\nFOR EACH STATEMENT\nEXECUTE FUNCTION partition_after_triggerfunc();\n\nCREATE TRIGGER items_after_update_trigger\nAFTER DELETE ON items\nREFERENCING OLD TABLE AS newdata\nFOR EACH STATEMENT\nEXECUTE FUNCTION partition_after_triggerfunc();\n\nCREATE TRIGGER items_after_delete_trigger\nAFTER UPDATE ON items\nREFERENCING NEW TABLE AS newdata\nFOR EACH STATEMENT\nEXECUTE FUNCTION partition_after_triggerfunc();\n\n\nCREATE OR REPLACE FUNCTION content_slim(_item jsonb) RETURNS jsonb AS $$\n SELECT strip_jsonb(_item - '{id,geometry,collection,type}'::text[], collection_base_item(_item->>'collection')) - '{id,geometry,collection,type}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_dehydrate(content jsonb) RETURNS items AS $$\n SELECT\n content->>'id' as id,\n stac_geom(content) as geometry,\n content->>'collection' as collection,\n stac_datetime(content) as datetime,\n stac_end_datetime(content) as end_datetime,\n content_slim(content) as content,\n null::jsonb as private\n ;\n$$ LANGUAGE SQL STABLE;\n\nCREATE OR REPLACE FUNCTION include_field(f text, fields jsonb DEFAULT '{}'::jsonb) RETURNS boolean AS $$\nDECLARE\n includes jsonb := fields->'include';\n excludes jsonb := fields->'exclude';\nBEGIN\n IF f IS NULL THEN\n RETURN NULL;\n END IF;\n\n\n IF\n jsonb_typeof(excludes) = 'array'\n AND jsonb_array_length(excludes)>0\n AND excludes ? f\n THEN\n RETURN FALSE;\n END IF;\n\n IF\n (\n jsonb_typeof(includes) = 'array'\n AND jsonb_array_length(includes) > 0\n AND includes ? f\n ) OR\n (\n includes IS NULL\n OR jsonb_typeof(includes) = 'null'\n OR jsonb_array_length(includes) = 0\n )\n THEN\n RETURN TRUE;\n END IF;\n\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nDROP FUNCTION IF EXISTS content_hydrate(jsonb, jsonb, jsonb);\nCREATE OR REPLACE FUNCTION content_hydrate(\n _item jsonb,\n _base_item jsonb,\n fields jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\n SELECT merge_jsonb(\n jsonb_fields(_item, fields),\n jsonb_fields(_base_item, fields)\n );\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION content_hydrate(_item items, _collection collections, fields jsonb DEFAULT '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\n content jsonb;\n base_item jsonb := _collection.base_item;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry, 20)::jsonb;\n END IF;\n output := content_hydrate(\n jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'collection', _item.collection,\n 'type', 'Feature'\n ) || _item.content,\n _collection.base_item,\n fields\n );\n\n RETURN output;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_nonhydrated(\n _item items,\n fields jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry, 20)::jsonb;\n END IF;\n output := jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'collection', _item.collection,\n 'type', 'Feature'\n ) || _item.content;\n RETURN output;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_hydrate(_item items, fields jsonb DEFAULT '{}'::jsonb) RETURNS jsonb AS $$\n SELECT content_hydrate(\n _item,\n (SELECT c FROM collections c WHERE id=_item.collection LIMIT 1),\n fields\n );\n$$ LANGUAGE SQL STABLE;\n\n\nCREATE UNLOGGED TABLE items_staging (\n content JSONB NOT NULL\n);\nCREATE UNLOGGED TABLE items_staging_ignore (\n content JSONB NOT NULL\n);\nCREATE UNLOGGED TABLE items_staging_upsert (\n content JSONB NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION items_staging_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p record;\n _partitions text[];\n part text;\n ts timestamptz := clock_timestamp();\n nrows int;\nBEGIN\n RAISE NOTICE 'Creating Partitions. %', clock_timestamp() - ts;\n\n FOR part IN WITH t AS (\n SELECT\n n.content->>'collection' as collection,\n stac_daterange(n.content->'properties') as dtr,\n partition_trunc\n FROM newdata n JOIN collections ON (n.content->>'collection'=collections.id)\n ), p AS (\n SELECT\n collection,\n COALESCE(date_trunc(partition_trunc::text, lower(dtr)),'-infinity') as d,\n tstzrange(min(lower(dtr)),max(lower(dtr)),'[]') as dtrange,\n tstzrange(min(upper(dtr)),max(upper(dtr)),'[]') as edtrange\n FROM t\n GROUP BY 1,2\n ) SELECT check_partition(collection, dtrange, edtrange) FROM p LOOP\n RAISE NOTICE 'Partition %', part;\n END LOOP;\n\n RAISE NOTICE 'Creating temp table with data to be added. %', clock_timestamp() - ts;\n DROP TABLE IF EXISTS tmpdata;\n CREATE TEMP TABLE tmpdata ON COMMIT DROP AS\n SELECT\n (content_dehydrate(content)).*\n FROM newdata;\n GET DIAGNOSTICS nrows = ROW_COUNT;\n RAISE NOTICE 'Added % rows to tmpdata. %', nrows, clock_timestamp() - ts;\n\n RAISE NOTICE 'Doing the insert. %', clock_timestamp() - ts;\n IF TG_TABLE_NAME = 'items_staging' THEN\n INSERT INTO items\n SELECT * FROM tmpdata;\n GET DIAGNOSTICS nrows = ROW_COUNT;\n RAISE NOTICE 'Inserted % rows to items. %', nrows, clock_timestamp() - ts;\n ELSIF TG_TABLE_NAME = 'items_staging_ignore' THEN\n INSERT INTO items\n SELECT * FROM tmpdata\n ON CONFLICT DO NOTHING;\n GET DIAGNOSTICS nrows = ROW_COUNT;\n RAISE NOTICE 'Inserted % rows to items. %', nrows, clock_timestamp() - ts;\n ELSIF TG_TABLE_NAME = 'items_staging_upsert' THEN\n DELETE FROM items i USING tmpdata s\n WHERE\n i.id = s.id\n AND i.collection = s.collection\n AND i IS DISTINCT FROM s\n ;\n GET DIAGNOSTICS nrows = ROW_COUNT;\n RAISE NOTICE 'Deleted % rows from items. %', nrows, clock_timestamp() - ts;\n INSERT INTO items AS t\n SELECT * FROM tmpdata\n ON CONFLICT DO NOTHING;\n GET DIAGNOSTICS nrows = ROW_COUNT;\n RAISE NOTICE 'Inserted % rows to items. %', nrows, clock_timestamp() - ts;\n END IF;\n\n RAISE NOTICE 'Deleting data from staging table. %', clock_timestamp() - ts;\n DELETE FROM items_staging;\n RAISE NOTICE 'Done. %', clock_timestamp() - ts;\n\n RETURN NULL;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER items_staging_insert_trigger AFTER INSERT ON items_staging REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\nCREATE TRIGGER items_staging_insert_ignore_trigger AFTER INSERT ON items_staging_ignore REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\nCREATE TRIGGER items_staging_insert_upsert_trigger AFTER INSERT ON items_staging_upsert REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\n\nCREATE OR REPLACE FUNCTION item_by_id(_id text, _collection text DEFAULT NULL) RETURNS items AS\n$$\nDECLARE\n i items%ROWTYPE;\nBEGIN\n SELECT * INTO i FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection) LIMIT 1;\n RETURN i;\nEND;\n$$ LANGUAGE PLPGSQL STABLE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION get_item(_id text, _collection text DEFAULT NULL) RETURNS jsonb AS $$\n SELECT content_hydrate(items) FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection);\n$$ LANGUAGE SQL STABLE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION delete_item(_id text, _collection text DEFAULT NULL) RETURNS VOID AS $$\nDECLARE\nout items%ROWTYPE;\nBEGIN\n DELETE FROM items WHERE id = _id AND (_collection IS NULL OR collection=_collection) RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n--/*\nCREATE OR REPLACE FUNCTION create_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION update_item(content jsonb) RETURNS VOID AS $$\nDECLARE\n old items %ROWTYPE;\n out items%ROWTYPE;\nBEGIN\n PERFORM delete_item(content->>'id', content->>'collection');\n PERFORM create_item(content);\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION create_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION collection_bbox(id text) RETURNS jsonb AS $$\n SELECT (replace(replace(replace(st_extent(geometry)::text,'BOX(','[['),')',']]'),' ',','))::jsonb\n FROM items WHERE collection=$1;\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION collection_temporal_extent(id text) RETURNS jsonb AS $$\n SELECT to_jsonb(array[array[min(datetime), max(datetime)]])\n FROM items WHERE collection=$1;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection_extents() RETURNS VOID AS $$\nUPDATE collections\n SET content = jsonb_set_lax(\n content,\n '{extent}'::text[],\n collection_extent(id, FALSE),\n true,\n 'use_json_null'\n )\n;\n$$ LANGUAGE SQL;\nCREATE TABLE partition_stats (\n partition text PRIMARY KEY,\n dtrange tstzrange,\n edtrange tstzrange,\n spatial geometry,\n last_updated timestamptz,\n keys text[]\n) WITH (FILLFACTOR=90);\n\nCREATE INDEX partitions_range_idx ON partition_stats USING GIST(dtrange);\n\n\nCREATE OR REPLACE FUNCTION constraint_tstzrange(expr text) RETURNS tstzrange AS $$\n WITH t AS (\n SELECT regexp_matches(\n expr,\n E'\\\\(''\\([0-9 :+-]*\\)''\\\\).*\\\\(''\\([0-9 :+-]*\\)''\\\\)'\n ) AS m\n ) SELECT tstzrange(m[1]::timestamptz, m[2]::timestamptz) FROM t\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE STRICT;\n\nCREATE OR REPLACE FUNCTION dt_constraint(coid oid, OUT dt tstzrange, OUT edt tstzrange) RETURNS RECORD AS $$\nDECLARE\n expr text := pg_get_constraintdef(coid);\n matches timestamptz[];\nBEGIN\n IF expr LIKE '%NULL%' THEN\n dt := tstzrange(null::timestamptz, null::timestamptz);\n edt := tstzrange(null::timestamptz, null::timestamptz);\n RETURN;\n END IF;\n WITH f AS (SELECT (regexp_matches(expr, E'([0-9]{4}-[0-1][0-9]-[0-3][0-9] [0-2][0-9]:[0-5][0-9]:[0-5][0-9]\\.?[0-9]*)', 'g'))[1] f)\n SELECT array_agg(f::timestamptz) INTO matches FROM f;\n IF cardinality(matches) = 4 THEN\n dt := tstzrange(matches[1], matches[2],'[]');\n edt := tstzrange(matches[3], matches[4], '[]');\n RETURN;\n ELSIF cardinality(matches) = 2 THEN\n edt := tstzrange(matches[1], matches[2],'[]');\n RETURN;\n END IF;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\nCREATE OR REPLACE VIEW partition_sys_meta AS\nSELECT\n (parse_ident(relid::text))[cardinality(parse_ident(relid::text))] as partition,\n replace(replace(CASE WHEN level = 1 THEN pg_get_expr(c.relpartbound, c.oid)\n ELSE pg_get_expr(parent.relpartbound, parent.oid)\n END, 'FOR VALUES IN (''',''), ''')','') AS collection,\n level,\n c.reltuples,\n c.relhastriggers,\n COALESCE(constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as partition_dtrange,\n COALESCE((dt_constraint(edt.oid)).dt, constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as constraint_dtrange,\n COALESCE((dt_constraint(edt.oid)).edt, tstzrange('-infinity', 'infinity','[]')) as constraint_edtrange\nFROM\n pg_partition_tree('items')\n JOIN pg_class c ON (relid::regclass = c.oid)\n JOIN pg_class parent ON (parentrelid::regclass = parent.oid AND isleaf)\n LEFT JOIN pg_constraint edt ON (conrelid=c.oid AND contype='c')\nWHERE isleaf\n;\n\nCREATE OR REPLACE VIEW partitions_view AS\nSELECT\n (parse_ident(relid::text))[cardinality(parse_ident(relid::text))] as partition,\n replace(replace(CASE WHEN level = 1 THEN pg_get_expr(c.relpartbound, c.oid)\n ELSE pg_get_expr(parent.relpartbound, parent.oid)\n END, 'FOR VALUES IN (''',''), ''')','') AS collection,\n level,\n c.reltuples,\n c.relhastriggers,\n COALESCE(pgstac.constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as partition_dtrange,\n COALESCE((pgstac.dt_constraint(edt.oid)).dt, pgstac.constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as constraint_dtrange,\n COALESCE((pgstac.dt_constraint(edt.oid)).edt, tstzrange('-infinity', 'infinity','[]')) as constraint_edtrange,\n dtrange,\n edtrange,\n spatial,\n last_updated\nFROM\n pg_partition_tree('pgstac.items')\n JOIN pg_class c ON (relid::regclass = c.oid)\n JOIN pg_class parent ON (parentrelid::regclass = parent.oid AND isleaf)\n LEFT JOIN pg_constraint edt ON (conrelid=c.oid AND contype='c')\n LEFT JOIN pgstac.partition_stats ON ((parse_ident(relid::text))[cardinality(parse_ident(relid::text))]=partition)\nWHERE isleaf\n;\n\nCREATE MATERIALIZED VIEW partitions AS\nSELECT * FROM partitions_view;\nCREATE UNIQUE INDEX ON partitions (partition);\n\nCREATE MATERIALIZED VIEW partition_steps AS\nSELECT\n partition as name,\n date_trunc('month',lower(partition_dtrange)) as sdate,\n date_trunc('month', upper(partition_dtrange)) + '1 month'::interval as edate\n FROM partitions_view WHERE partition_dtrange IS NOT NULL AND partition_dtrange != 'empty'::tstzrange\n ORDER BY dtrange ASC\n;\n\n\nCREATE OR REPLACE FUNCTION update_partition_stats_q(_partition text, istrigger boolean default false) RETURNS VOID AS $$\nDECLARE\nBEGIN\n PERFORM run_or_queue(\n format('SELECT update_partition_stats(%L, %L);', _partition, istrigger)\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION update_partition_stats(_partition text, istrigger boolean default false) RETURNS VOID AS $$\nDECLARE\n dtrange tstzrange;\n edtrange tstzrange;\n cdtrange tstzrange;\n cedtrange tstzrange;\n extent geometry;\n collection text;\nBEGIN\n RAISE NOTICE 'Updating stats for %.', _partition;\n EXECUTE format(\n $q$\n SELECT\n tstzrange(min(datetime), max(datetime),'[]'),\n tstzrange(min(end_datetime), max(end_datetime), '[]')\n FROM %I\n $q$,\n _partition\n ) INTO dtrange, edtrange;\n extent := st_estimatedextent('pgstac', _partition, 'geometry');\n RAISE DEBUG 'Estimated Extent: %', extent;\n INSERT INTO partition_stats (partition, dtrange, edtrange, spatial, last_updated)\n SELECT _partition, dtrange, edtrange, extent, now()\n ON CONFLICT (partition) DO\n UPDATE SET\n dtrange=EXCLUDED.dtrange,\n edtrange=EXCLUDED.edtrange,\n spatial=EXCLUDED.spatial,\n last_updated=EXCLUDED.last_updated\n ;\n\n SELECT\n constraint_dtrange, constraint_edtrange, pv.collection\n INTO cdtrange, cedtrange, collection\n FROM partitions_view pv WHERE partition = _partition;\n REFRESH MATERIALIZED VIEW partitions;\n REFRESH MATERIALIZED VIEW partition_steps;\n\n\n RAISE NOTICE 'Checking if we need to modify constraints...';\n RAISE NOTICE 'cdtrange: % dtrange: % cedtrange: % edtrange: %',cdtrange, dtrange, cedtrange, edtrange;\n IF\n (cdtrange IS DISTINCT FROM dtrange OR edtrange IS DISTINCT FROM cedtrange)\n AND NOT istrigger\n THEN\n RAISE NOTICE 'Modifying Constraints';\n RAISE NOTICE 'Existing % %', cdtrange, cedtrange;\n RAISE NOTICE 'New % %', dtrange, edtrange;\n PERFORM drop_table_constraints(_partition);\n PERFORM create_table_constraints(_partition, dtrange, edtrange);\n REFRESH MATERIALIZED VIEW partitions;\n REFRESH MATERIALIZED VIEW partition_steps;\n END IF;\n RAISE NOTICE 'Checking if we need to update collection extents.';\n IF get_setting_bool('update_collection_extent') THEN\n RAISE NOTICE 'updating collection extent for %', collection;\n PERFORM run_or_queue(format($q$\n UPDATE collections\n SET content = jsonb_set_lax(\n content,\n '{extent}'::text[],\n collection_extent(%L, FALSE),\n true,\n 'use_json_null'\n ) WHERE id=%L\n ;\n $q$, collection, collection));\n ELSE\n RAISE NOTICE 'Not updating collection extent for %', collection;\n END IF;\n\nEND;\n$$ LANGUAGE PLPGSQL STRICT SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION partition_name( IN collection text, IN dt timestamptz, OUT partition_name text, OUT partition_range tstzrange) AS $$\nDECLARE\n c RECORD;\n parent_name text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n parent_name := format('_items_%s', c.key);\n\n\n IF c.partition_trunc = 'year' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYY'));\n ELSIF c.partition_trunc = 'month' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYYMM'));\n ELSE\n partition_name := parent_name;\n partition_range := tstzrange('-infinity'::timestamptz, 'infinity'::timestamptz, '[]');\n END IF;\n IF partition_range IS NULL THEN\n partition_range := tstzrange(\n date_trunc(c.partition_trunc::text, dt),\n date_trunc(c.partition_trunc::text, dt) + concat('1 ', c.partition_trunc)::interval\n );\n END IF;\n RETURN;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION drop_table_constraints(t text) RETURNS text AS $$\nDECLARE\n q text;\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM partitions_view WHERE partition=t) THEN\n RETURN NULL;\n END IF;\n FOR q IN SELECT FORMAT(\n $q$\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n $q$,\n t,\n conname\n ) FROM pg_constraint\n WHERE conrelid=t::regclass::oid AND contype='c'\n LOOP\n EXECUTE q;\n END LOOP;\n RETURN t;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE OR REPLACE FUNCTION create_table_constraints(t text, _dtrange tstzrange, _edtrange tstzrange) RETURNS text AS $$\nDECLARE\n q text;\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM partitions_view WHERE partition=t) THEN\n RETURN NULL;\n END IF;\n RAISE NOTICE 'Creating Table Constraints for % % %', t, _dtrange, _edtrange;\n IF _dtrange = 'empty' AND _edtrange = 'empty' THEN\n q :=format(\n $q$\n DO $block$\n BEGIN\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n ALTER TABLE %I\n ADD CONSTRAINT %I\n CHECK (((datetime IS NULL) AND (end_datetime IS NULL))) NOT VALID\n ;\n ALTER TABLE %I\n VALIDATE CONSTRAINT %I\n ;\n\n\n\n EXCEPTION WHEN others THEN\n RAISE WARNING '%%, Issue Altering Constraints. Please run update_partition_stats(%I)', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n $block$;\n $q$,\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t),\n t\n );\n ELSE\n q :=format(\n $q$\n DO $block$\n BEGIN\n\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n ALTER TABLE %I\n ADD CONSTRAINT %I\n CHECK (\n (datetime >= %L)\n AND (datetime <= %L)\n AND (end_datetime >= %L)\n AND (end_datetime <= %L)\n ) NOT VALID\n ;\n ALTER TABLE %I\n VALIDATE CONSTRAINT %I\n ;\n\n\n\n EXCEPTION WHEN others THEN\n RAISE WARNING '%%, Issue Altering Constraints. Please run update_partition_stats(%I)', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n $block$;\n $q$,\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t),\n lower(_dtrange),\n upper(_dtrange),\n lower(_edtrange),\n upper(_edtrange),\n t,\n format('%s_dt', t),\n t\n );\n END IF;\n PERFORM run_or_queue(q);\n RETURN t;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION check_partition(\n _collection text,\n _dtrange tstzrange,\n _edtrange tstzrange\n) RETURNS text AS $$\nDECLARE\n c RECORD;\n pm RECORD;\n _partition_name text;\n _partition_dtrange tstzrange;\n _constraint_dtrange tstzrange;\n _constraint_edtrange tstzrange;\n q text;\n deferrable_q text;\n err_context text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=_collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', _collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n\n IF c.partition_trunc IS NOT NULL THEN\n _partition_dtrange := tstzrange(\n date_trunc(c.partition_trunc, lower(_dtrange)),\n date_trunc(c.partition_trunc, lower(_dtrange)) + (concat('1 ', c.partition_trunc))::interval,\n '[)'\n );\n ELSE\n _partition_dtrange := '[-infinity, infinity]'::tstzrange;\n END IF;\n\n IF NOT _partition_dtrange @> _dtrange THEN\n RAISE EXCEPTION 'dtrange % is greater than the partition size % for collection %', _dtrange, c.partition_trunc, _collection;\n END IF;\n\n\n IF c.partition_trunc = 'year' THEN\n _partition_name := format('_items_%s_%s', c.key, to_char(lower(_partition_dtrange),'YYYY'));\n ELSIF c.partition_trunc = 'month' THEN\n _partition_name := format('_items_%s_%s', c.key, to_char(lower(_partition_dtrange),'YYYYMM'));\n ELSE\n _partition_name := format('_items_%s', c.key);\n END IF;\n\n SELECT * INTO pm FROM partition_sys_meta WHERE collection=_collection AND partition_dtrange @> _dtrange;\n IF FOUND THEN\n RAISE NOTICE '% % %', _edtrange, _dtrange, pm;\n _constraint_edtrange :=\n tstzrange(\n least(\n lower(_edtrange),\n nullif(lower(pm.constraint_edtrange), '-infinity')\n ),\n greatest(\n upper(_edtrange),\n nullif(upper(pm.constraint_edtrange), 'infinity')\n ),\n '[]'\n );\n _constraint_dtrange :=\n tstzrange(\n least(\n lower(_dtrange),\n nullif(lower(pm.constraint_dtrange), '-infinity')\n ),\n greatest(\n upper(_dtrange),\n nullif(upper(pm.constraint_dtrange), 'infinity')\n ),\n '[]'\n );\n\n IF pm.constraint_edtrange @> _edtrange AND pm.constraint_dtrange @> _dtrange THEN\n RETURN pm.partition;\n ELSE\n PERFORM drop_table_constraints(_partition_name);\n END IF;\n ELSE\n _constraint_edtrange := _edtrange;\n _constraint_dtrange := _dtrange;\n END IF;\n RAISE NOTICE 'EXISTING CONSTRAINTS % %, NEW % %', pm.constraint_dtrange, pm.constraint_edtrange, _constraint_dtrange, _constraint_edtrange;\n RAISE NOTICE 'Creating partition % %', _partition_name, _partition_dtrange;\n IF c.partition_trunc IS NULL THEN\n q := format(\n $q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n GRANT ALL ON %I to pgstac_ingest;\n $q$,\n _partition_name,\n _collection,\n concat(_partition_name,'_pk'),\n _partition_name,\n _partition_name\n );\n ELSE\n q := format(\n $q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L) PARTITION BY RANGE (datetime);\n CREATE TABLE IF NOT EXISTS %I partition OF %I FOR VALUES FROM (%L) TO (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n GRANT ALL ON %I TO pgstac_ingest;\n $q$,\n format('_items_%s', c.key),\n _collection,\n _partition_name,\n format('_items_%s', c.key),\n lower(_partition_dtrange),\n upper(_partition_dtrange),\n format('%s_pk', _partition_name),\n _partition_name,\n _partition_name\n );\n END IF;\n\n BEGIN\n EXECUTE q;\n EXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', _partition_name;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\n END;\n PERFORM maintain_partitions(_partition_name);\n PERFORM update_partition_stats_q(_partition_name, true);\n REFRESH MATERIALIZED VIEW partitions;\n REFRESH MATERIALIZED VIEW partition_steps;\n RETURN _partition_name;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION repartition(_collection text, _partition_trunc text, triggered boolean DEFAULT FALSE) RETURNS text AS $$\nDECLARE\n c RECORD;\n q text;\n from_trunc text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=_collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', _collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n IF triggered THEN\n RAISE NOTICE 'Converting % to % partitioning via Trigger', _collection, _partition_trunc;\n ELSE\n RAISE NOTICE 'Converting % from using % to % partitioning', _collection, c.partition_trunc, _partition_trunc;\n IF c.partition_trunc IS NOT DISTINCT FROM _partition_trunc THEN\n RAISE NOTICE 'Collection % already set to use partition by %', _collection, _partition_trunc;\n RETURN _collection;\n END IF;\n END IF;\n\n IF EXISTS (SELECT 1 FROM partitions_view WHERE collection=_collection LIMIT 1) THEN\n EXECUTE format(\n $q$\n CREATE TEMP TABLE changepartitionstaging ON COMMIT DROP AS SELECT * FROM %I;\n DROP TABLE IF EXISTS %I CASCADE;\n WITH p AS (\n SELECT\n collection,\n CASE\n WHEN %L IS NULL THEN '-infinity'::timestamptz\n ELSE date_trunc(%L, datetime)\n END as d,\n tstzrange(min(datetime),max(datetime),'[]') as dtrange,\n tstzrange(min(end_datetime),max(end_datetime),'[]') as edtrange\n FROM changepartitionstaging\n GROUP BY 1,2\n ) SELECT check_partition(collection, dtrange, edtrange) FROM p;\n INSERT INTO items SELECT * FROM changepartitionstaging;\n DROP TABLE changepartitionstaging;\n $q$,\n concat('_items_', c.key),\n concat('_items_', c.key),\n c.partition_trunc,\n c.partition_trunc\n );\n END IF;\n RETURN _collection;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE OR REPLACE FUNCTION collections_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n q text;\n partition_name text := format('_items_%s', NEW.key);\n partition_exists boolean := false;\n partition_empty boolean := true;\n err_context text;\n loadtemp boolean := FALSE;\nBEGIN\n RAISE NOTICE 'Collection Trigger. % %', NEW.id, NEW.key;\n IF TG_OP = 'UPDATE' AND NEW.partition_trunc IS DISTINCT FROM OLD.partition_trunc THEN\n PERFORM repartition(NEW.id, NEW.partition_trunc, TRUE);\n END IF;\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER collections_trigger AFTER\nINSERT\nOR\nUPDATE ON collections\nFOR EACH ROW EXECUTE FUNCTION collections_trigger_func();\n\nCREATE OR REPLACE FUNCTION chunker(\n IN _where text,\n OUT s timestamptz,\n OUT e timestamptz\n) RETURNS SETOF RECORD AS $$\nDECLARE\n explain jsonb;\nBEGIN\n IF _where IS NULL THEN\n _where := ' TRUE ';\n END IF;\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s;', _where)\n INTO explain;\n RAISE DEBUG 'EXPLAIN: %', explain;\n\n RETURN QUERY\n WITH t AS (\n SELECT j->>0 as p FROM\n jsonb_path_query(\n explain,\n 'strict $.**.\"Relation Name\" ? (@ != null)'\n ) j\n ),\n parts AS (\n SELECT sdate, edate FROM t JOIN partition_steps ON (t.p = name)\n ),\n times AS (\n SELECT sdate FROM parts\n UNION\n SELECT edate FROM parts\n ),\n uniq AS (\n SELECT DISTINCT sdate FROM times ORDER BY sdate\n ),\n last AS (\n SELECT sdate, lead(sdate, 1) over () as edate FROM uniq\n )\n SELECT sdate, edate FROM last WHERE edate IS NOT NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION partition_queries(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN partitions text[] DEFAULT NULL\n) RETURNS SETOF text AS $$\nDECLARE\n query text;\n sdate timestamptz;\n edate timestamptz;\nBEGIN\nIF _where IS NULL OR trim(_where) = '' THEN\n _where = ' TRUE ';\nEND IF;\nRAISE NOTICE 'Getting chunks for % %', _where, _orderby;\nIF _orderby ILIKE 'datetime d%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 DESC LOOP\n RETURN NEXT format($q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n $q$,\n sdate,\n edate,\n _where,\n _orderby\n );\n END LOOP;\nELSIF _orderby ILIKE 'datetime a%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 ASC LOOP\n RETURN NEXT format($q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n $q$,\n sdate,\n edate,\n _where,\n _orderby\n );\n END LOOP;\nELSE\n query := format($q$\n SELECT * FROM items\n WHERE %s\n ORDER BY %s\n $q$, _where, _orderby\n );\n\n RETURN NEXT query;\n RETURN;\nEND IF;\n\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION partition_query_view(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN _limit int DEFAULT 10\n) RETURNS text AS $$\n WITH p AS (\n SELECT * FROM partition_queries(_where, _orderby) p\n )\n SELECT\n CASE WHEN EXISTS (SELECT 1 FROM p) THEN\n (SELECT format($q$\n SELECT * FROM (\n %s\n ) total LIMIT %s\n $q$,\n string_agg(\n format($q$ SELECT * FROM ( %s ) AS sub $q$, p),\n '\n UNION ALL\n '\n ),\n _limit\n ))\n ELSE NULL\n END FROM p;\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION q_to_tsquery (input text)\n RETURNS tsquery\n AS $$\nDECLARE\n processed_text text;\n temp_text text;\n quote_array text[];\n placeholder text := '@QUOTE@';\nBEGIN\n -- Extract all quoted phrases and store in array\n quote_array := regexp_matches(input, '\"[^\"]*\"', 'g');\n\n -- Replace each quoted part with a unique placeholder if there are any quoted phrases\n IF array_length(quote_array, 1) IS NOT NULL THEN\n processed_text := input;\n FOR i IN array_lower(quote_array, 1) .. array_upper(quote_array, 1) LOOP\n processed_text := replace(processed_text, quote_array[i], placeholder || i || placeholder);\n END LOOP;\n ELSE\n processed_text := input;\n END IF;\n\n -- Replace non-quoted text using regular expressions\n\n -- , -> |\n processed_text := regexp_replace(processed_text, ',(?=(?:[^\"]*\"[^\"]*\")*[^\"]*$)', ' | ', 'g');\n\n -- and -> &\n processed_text := regexp_replace(processed_text, '\\s+AND\\s+', ' & ', 'gi');\n\n -- or -> |\n processed_text := regexp_replace(processed_text, '\\s+OR\\s+', ' | ', 'gi');\n\n -- +term -> & term\n processed_text := regexp_replace(processed_text, '\\+([a-zA-Z0-9_]+)', '& \\1', 'g');\n\n -- -term -> ! term\n processed_text := regexp_replace(processed_text, '\\-([a-zA-Z0-9_]+)', '& ! \\1', 'g');\n\n -- Replace placeholders back with quoted phrases if there were any\n IF array_length(quote_array, 1) IS NOT NULL THEN\n FOR i IN array_lower(quote_array, 1) .. array_upper(quote_array, 1) LOOP\n processed_text := replace(processed_text, placeholder || i || placeholder, '''' || substring(quote_array[i] from 2 for length(quote_array[i]) - 2) || '''');\n END LOOP;\n END IF;\n\n -- Print processed_text to the console for debugging purposes\n RAISE NOTICE 'processed_text: %', processed_text;\n\n RETURN to_tsquery(processed_text);\nEND;\n$$\nLANGUAGE plpgsql;\n\n\nCREATE OR REPLACE FUNCTION stac_search_to_where(j jsonb) RETURNS text AS $$\nDECLARE\n where_segments text[];\n _where text;\n dtrange tstzrange;\n collections text[];\n geom geometry;\n sdate timestamptz;\n edate timestamptz;\n filterlang text;\n filter jsonb := j->'filter';\n ft_query tsquery;\nBEGIN\n IF j ? 'ids' THEN\n where_segments := where_segments || format('id = ANY (%L) ', to_text_array(j->'ids'));\n END IF;\n\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n where_segments := where_segments || format('collection = ANY (%L) ', collections);\n END IF;\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n\n where_segments := where_segments || format(' datetime <= %L::timestamptz AND end_datetime >= %L::timestamptz ',\n edate,\n sdate\n );\n END IF;\n\n IF j ? 'q' THEN\n ft_query := q_to_tsquery(j->>'q');\n where_segments := where_segments || format(\n $quote$\n (\n to_tsvector('english', content->'properties'->>'description') ||\n to_tsvector('english', content->'properties'->>'title') ||\n to_tsvector('english', content->'properties'->'keywords')\n ) @@ %L\n $quote$,\n ft_query\n );\n END IF;\n\n geom := stac_geom(j);\n IF geom IS NOT NULL THEN\n where_segments := where_segments || format('st_intersects(geometry, %L)',geom);\n END IF;\n\n filterlang := COALESCE(\n j->>'filter-lang',\n get_setting('default_filter_lang', j->'conf')\n );\n IF NOT filter @? '$.**.op' THEN\n filterlang := 'cql-json';\n END IF;\n\n IF filterlang NOT IN ('cql-json','cql2-json') AND j ? 'filter' THEN\n RAISE EXCEPTION '% is not a supported filter-lang. Please use cql-json or cql2-json.', filterlang;\n END IF;\n\n IF j ? 'query' AND j ? 'filter' THEN\n RAISE EXCEPTION 'Can only use either query or filter at one time.';\n END IF;\n\n IF j ? 'query' THEN\n filter := query_to_cql2(j->'query');\n ELSIF filterlang = 'cql-json' THEN\n filter := cql1_to_cql2(filter);\n END IF;\n RAISE NOTICE 'FILTER: %', filter;\n where_segments := where_segments || cql2_query(filter);\n IF cardinality(where_segments) < 1 THEN\n RETURN ' TRUE ';\n END IF;\n\n _where := array_to_string(array_remove(where_segments, NULL), ' AND ');\n\n IF _where IS NULL OR BTRIM(_where) = '' THEN\n RETURN ' TRUE ';\n END IF;\n RETURN _where;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION parse_sort_dir(_dir text, reverse boolean default false) RETURNS text AS $$\n WITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n ) SELECT\n CASE\n WHEN NOT reverse THEN d\n WHEN d = 'ASC' THEN 'DESC'\n WHEN d = 'DESC' THEN 'ASC'\n END\n FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION sort_dir_to_op(_dir text, prev boolean default false) RETURNS text AS $$\n WITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n ) SELECT\n CASE\n WHEN d = 'ASC' AND prev THEN '<='\n WHEN d = 'DESC' AND prev THEN '>='\n WHEN d = 'ASC' THEN '>='\n WHEN d = 'DESC' THEN '<='\n END\n FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION sort_sqlorderby(\n _search jsonb DEFAULT NULL,\n reverse boolean DEFAULT FALSE\n) RETURNS text AS $$\n WITH sortby AS (\n SELECT coalesce(_search->'sortby','[{\"field\":\"datetime\", \"direction\":\"desc\"}]') as sort\n ), withid AS (\n SELECT CASE\n WHEN sort @? '$[*] ? (@.field == \"id\")' THEN sort\n ELSE sort || '[{\"field\":\"id\", \"direction\":\"desc\"}]'::jsonb\n END as sort\n FROM sortby\n ), withid_rows AS (\n SELECT jsonb_array_elements(sort) as value FROM withid\n ),sorts AS (\n SELECT\n coalesce(\n (queryable(value->>'field')).expression\n ) as key,\n parse_sort_dir(value->>'direction', reverse) as dir\n FROM withid_rows\n )\n SELECT array_to_string(\n array_agg(concat(key, ' ', dir)),\n ', '\n ) FROM sorts;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_sort_dir(sort_item jsonb) RETURNS text AS $$\n SELECT CASE WHEN sort_item->>'direction' ILIKE 'desc%' THEN 'DESC' ELSE 'ASC' END;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION get_token_val_str(\n _field text,\n _item items\n) RETURNS text AS $$\nDECLARE\n q text;\n literal text;\nBEGIN\n q := format($q$ SELECT quote_literal(%s) FROM (SELECT $1.*) as r;$q$, _field);\n EXECUTE q INTO literal USING _item;\n RETURN literal;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n\nCREATE OR REPLACE FUNCTION get_token_record(IN _token text, OUT prev BOOLEAN, OUT item items) RETURNS RECORD AS $$\nDECLARE\n _itemid text := _token;\n _collectionid text;\nBEGIN\n IF _token IS NULL THEN\n RETURN;\n END IF;\n RAISE NOTICE 'Looking for token: %', _token;\n prev := FALSE;\n IF _token ILIKE 'prev:%' THEN\n _itemid := replace(_token, 'prev:','');\n prev := TRUE;\n ELSIF _token ILIKE 'next:%' THEN\n _itemid := replace(_token, 'next:', '');\n END IF;\n SELECT id INTO _collectionid FROM collections WHERE _itemid LIKE concat(id,':%');\n IF FOUND THEN\n _itemid := replace(_itemid, concat(_collectionid,':'), '');\n SELECT * INTO item FROM items WHERE id=_itemid AND collection=_collectionid;\n ELSE\n SELECT * INTO item FROM items WHERE id=_itemid;\n END IF;\n IF item IS NULL THEN\n RAISE EXCEPTION 'Could not find item using token: % item: % collection: %', _token, _itemid, _collectionid;\n END IF;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\n\nCREATE OR REPLACE FUNCTION get_token_filter(\n _sortby jsonb DEFAULT '[{\"field\":\"datetime\",\"direction\":\"desc\"}]'::jsonb,\n token_item items DEFAULT NULL,\n prev boolean DEFAULT FALSE,\n inclusive boolean DEFAULT FALSE\n) RETURNS text AS $$\nDECLARE\n ltop text := '<';\n gtop text := '>';\n dir text;\n sort record;\n orfilter text := '';\n orfilters text[] := '{}'::text[];\n andfilters text[] := '{}'::text[];\n output text;\n token_where text;\nBEGIN\n IF _sortby IS NULL OR _sortby = '[]'::jsonb THEN\n _sortby := '[{\"field\":\"datetime\",\"direction\":\"desc\"}]'::jsonb;\n END IF;\n _sortby := _sortby || jsonb_build_object('field','id','direction',_sortby->0->>'direction');\n RAISE NOTICE 'Getting Token Filter. % %', _sortby, token_item;\n IF inclusive THEN\n orfilters := orfilters || format('( id=%L AND collection=%L )' , token_item.id, token_item.collection);\n END IF;\n\n FOR sort IN\n WITH s1 AS (\n SELECT\n _row,\n (queryable(value->>'field')).expression as _field,\n (value->>'field' = 'id') as _isid,\n get_sort_dir(value) as _dir\n FROM jsonb_array_elements(_sortby)\n WITH ORDINALITY AS t(value, _row)\n )\n SELECT\n _row,\n _field,\n _dir,\n get_token_val_str(_field, token_item) as _val\n FROM s1\n WHERE _row <= (SELECT min(_row) FROM s1 WHERE _isid)\n LOOP\n orfilter := NULL;\n RAISE NOTICE 'SORT: %', sort;\n IF sort._val IS NOT NULL AND ((prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC')) THEN\n orfilter := format($f$(\n (%s %s %s) OR (%s IS NULL)\n )$f$,\n sort._field,\n ltop,\n sort._val,\n sort._val\n );\n ELSIF sort._val IS NULL AND ((prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC')) THEN\n RAISE NOTICE '< but null';\n orfilter := format('%s IS NOT NULL', sort._field);\n ELSIF sort._val IS NULL THEN\n RAISE NOTICE '> but null';\n ELSE\n orfilter := format($f$(\n (%s %s %s) OR (%s IS NULL)\n )$f$,\n sort._field,\n gtop,\n sort._val,\n sort._field\n );\n END IF;\n RAISE NOTICE 'ORFILTER: %', orfilter;\n\n IF orfilter IS NOT NULL THEN\n IF sort._row = 1 THEN\n orfilters := orfilters || orfilter;\n ELSE\n orfilters := orfilters || format('(%s AND %s)', array_to_string(andfilters, ' AND '), orfilter);\n END IF;\n END IF;\n IF sort._val IS NOT NULL THEN\n andfilters := andfilters || format('%s = %s', sort._field, sort._val);\n ELSE\n andfilters := andfilters || format('%s IS NULL', sort._field);\n END IF;\n END LOOP;\n\n output := array_to_string(orfilters, ' OR ');\n\n token_where := concat('(',coalesce(output,'true'),')');\n IF trim(token_where) = '' THEN\n token_where := NULL;\n END IF;\n RAISE NOTICE 'TOKEN_WHERE: %',token_where;\n RETURN token_where;\n END;\n$$ LANGUAGE PLPGSQL SET transform_null_equals TO TRUE\n;\n\nCREATE OR REPLACE FUNCTION search_tohash(jsonb) RETURNS jsonb AS $$\n SELECT $1 - '{token,limit,context,includes,excludes}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION search_hash(jsonb, jsonb) RETURNS text AS $$\n SELECT md5(concat(search_tohash($1)::text,$2::text));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TABLE IF NOT EXISTS searches(\n hash text GENERATED ALWAYS AS (search_hash(search, metadata)) STORED PRIMARY KEY,\n search jsonb NOT NULL,\n _where text,\n orderby text,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n metadata jsonb DEFAULT '{}'::jsonb NOT NULL\n);\n\nCREATE TABLE IF NOT EXISTS search_wheres(\n id bigint generated always as identity primary key,\n _where text NOT NULL,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n statslastupdated timestamptz,\n estimated_count bigint,\n estimated_cost float,\n time_to_estimate float,\n total_count bigint,\n time_to_count float,\n partitions text[]\n);\n\nCREATE INDEX IF NOT EXISTS search_wheres_partitions ON search_wheres USING GIN (partitions);\nCREATE UNIQUE INDEX IF NOT EXISTS search_wheres_where ON search_wheres ((md5(_where)));\n\nCREATE OR REPLACE FUNCTION where_stats(\n inwhere text,\n updatestats boolean default false,\n conf jsonb default null\n) RETURNS search_wheres AS $$\nDECLARE\n t timestamptz;\n i interval;\n explain_json jsonb;\n partitions text[];\n sw search_wheres%ROWTYPE;\n inwhere_hash text := md5(inwhere);\n _context text := lower(context(conf));\n _stats_ttl interval := context_stats_ttl(conf);\n _estimated_cost_threshold float := context_estimated_cost(conf);\n _estimated_count_threshold int := context_estimated_count(conf);\n ro bool := pgstac.readonly(conf);\nBEGIN\n -- If updatestats is true then set ttl to 0\n IF updatestats THEN\n RAISE DEBUG 'Updatestats set to TRUE, setting TTL to 0';\n _stats_ttl := '0'::interval;\n END IF;\n\n -- If we don't need to calculate context, just return\n IF _context = 'off' THEN\n sw._where = inwhere;\n RETURN sw;\n END IF;\n\n -- Get any stats that we have.\n IF NOT ro THEN\n -- If there is a lock where another process is\n -- updating the stats, wait so that we don't end up calculating a bunch of times.\n SELECT * INTO sw FROM search_wheres WHERE md5(_where)=inwhere_hash FOR UPDATE;\n ELSE\n SELECT * INTO sw FROM search_wheres WHERE md5(_where)=inwhere_hash;\n END IF;\n\n -- If there is a cached row, figure out if we need to update\n IF\n sw IS NOT NULL\n AND sw.statslastupdated IS NOT NULL\n AND sw.total_count IS NOT NULL\n AND now() - sw.statslastupdated <= _stats_ttl\n THEN\n -- we have a cached row with data that is within our ttl\n RAISE DEBUG 'Stats present in table and lastupdated within ttl: %', sw;\n IF NOT ro THEN\n RAISE DEBUG 'Updating search_wheres only bumping lastused and usecount';\n UPDATE search_wheres SET\n lastused = now(),\n usecount = search_wheres.usecount + 1\n WHERE md5(_where) = inwhere_hash\n RETURNING * INTO sw;\n END IF;\n RAISE DEBUG 'Returning cached counts. %', sw;\n RETURN sw;\n END IF;\n\n -- Calculate estimated cost and rows\n -- Use explain to get estimated count/cost\n IF sw.estimated_count IS NULL OR sw.estimated_cost IS NULL THEN\n RAISE DEBUG 'Calculating estimated stats';\n t := clock_timestamp();\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s', inwhere)\n INTO explain_json;\n RAISE DEBUG 'Time for just the explain: %', clock_timestamp() - t;\n i := clock_timestamp() - t;\n\n sw.estimated_count := explain_json->0->'Plan'->'Plan Rows';\n sw.estimated_cost := explain_json->0->'Plan'->'Total Cost';\n sw.time_to_estimate := extract(epoch from i);\n END IF;\n\n RAISE DEBUG 'ESTIMATED_COUNT: %, THRESHOLD %', sw.estimated_count, _estimated_count_threshold;\n RAISE DEBUG 'ESTIMATED_COST: %, THRESHOLD %', sw.estimated_cost, _estimated_cost_threshold;\n\n -- If context is set to auto and the costs are within the threshold return the estimated costs\n IF\n _context = 'auto'\n AND sw.estimated_count >= _estimated_count_threshold\n AND sw.estimated_cost >= _estimated_cost_threshold\n THEN\n IF NOT ro THEN\n INSERT INTO search_wheres (\n _where,\n lastused,\n usecount,\n statslastupdated,\n estimated_count,\n estimated_cost,\n time_to_estimate,\n total_count,\n time_to_count\n ) VALUES (\n inwhere,\n now(),\n 1,\n now(),\n sw.estimated_count,\n sw.estimated_cost,\n sw.time_to_estimate,\n null,\n null\n ) ON CONFLICT ((md5(_where)))\n DO UPDATE SET\n lastused = EXCLUDED.lastused,\n usecount = search_wheres.usecount + 1,\n statslastupdated = EXCLUDED.statslastupdated,\n estimated_count = EXCLUDED.estimated_count,\n estimated_cost = EXCLUDED.estimated_cost,\n time_to_estimate = EXCLUDED.time_to_estimate,\n total_count = EXCLUDED.total_count,\n time_to_count = EXCLUDED.time_to_count\n RETURNING * INTO sw;\n END IF;\n RAISE DEBUG 'Estimates are within thresholds, returning estimates. %', sw;\n RETURN sw;\n END IF;\n\n -- Calculate Actual Count\n t := clock_timestamp();\n RAISE NOTICE 'Calculating actual count...';\n EXECUTE format(\n 'SELECT count(*) FROM items WHERE %s',\n inwhere\n ) INTO sw.total_count;\n i := clock_timestamp() - t;\n RAISE NOTICE 'Actual Count: % -- %', sw.total_count, i;\n sw.time_to_count := extract(epoch FROM i);\n\n IF NOT ro THEN\n INSERT INTO search_wheres (\n _where,\n lastused,\n usecount,\n statslastupdated,\n estimated_count,\n estimated_cost,\n time_to_estimate,\n total_count,\n time_to_count\n ) VALUES (\n inwhere,\n now(),\n 1,\n now(),\n sw.estimated_count,\n sw.estimated_cost,\n sw.time_to_estimate,\n sw.total_count,\n sw.time_to_count\n ) ON CONFLICT ((md5(_where)))\n DO UPDATE SET\n lastused = EXCLUDED.lastused,\n usecount = search_wheres.usecount + 1,\n statslastupdated = EXCLUDED.statslastupdated,\n estimated_count = EXCLUDED.estimated_count,\n estimated_cost = EXCLUDED.estimated_cost,\n time_to_estimate = EXCLUDED.time_to_estimate,\n total_count = EXCLUDED.total_count,\n time_to_count = EXCLUDED.time_to_count\n RETURNING * INTO sw;\n END IF;\n RAISE DEBUG 'Returning with actual count. %', sw;\n RETURN sw;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION search_query(\n _search jsonb = '{}'::jsonb,\n updatestats boolean = false,\n _metadata jsonb = '{}'::jsonb\n) RETURNS searches AS $$\nDECLARE\n search searches%ROWTYPE;\n cached_search searches%ROWTYPE;\n pexplain jsonb;\n t timestamptz;\n i interval;\n doupdate boolean := FALSE;\n insertfound boolean := FALSE;\n ro boolean := pgstac.readonly();\n found_search text;\nBEGIN\n RAISE NOTICE 'SEARCH: %', _search;\n -- Calculate hash, where clause, and order by statement\n search.search := _search;\n search.metadata := _metadata;\n search.hash := search_hash(_search, _metadata);\n search._where := stac_search_to_where(_search);\n search.orderby := sort_sqlorderby(_search);\n search.lastused := now();\n search.usecount := 1;\n\n -- If we are in read only mode, directly return search\n IF ro THEN\n RETURN search;\n END IF;\n\n RAISE NOTICE 'Updating Statistics for search: %s', search;\n -- Update statistics for times used and and when last used\n -- If the entry is locked, rather than waiting, skip updating the stats\n INSERT INTO searches (search, lastused, usecount, metadata)\n VALUES (search.search, now(), 1, search.metadata)\n ON CONFLICT DO NOTHING\n RETURNING * INTO cached_search\n ;\n\n IF NOT FOUND OR cached_search IS NULL THEN\n UPDATE searches SET\n lastused = now(),\n usecount = searches.usecount + 1\n WHERE hash = (\n SELECT hash FROM searches WHERE hash=search.hash FOR UPDATE SKIP LOCKED\n )\n RETURNING * INTO cached_search\n ;\n END IF;\n\n IF cached_search IS NOT NULL THEN\n cached_search._where = search._where;\n cached_search.orderby = search.orderby;\n RETURN cached_search;\n END IF;\n RETURN search;\n\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE OR REPLACE FUNCTION search_fromhash(\n _hash text\n) RETURNS searches AS $$\n SELECT * FROM search_query((SELECT search FROM searches WHERE hash=_hash LIMIT 1));\n$$ LANGUAGE SQL STRICT;\n\nCREATE OR REPLACE FUNCTION search_rows(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN partitions text[] DEFAULT NULL,\n IN _limit int DEFAULT 10\n) RETURNS SETOF items AS $$\nDECLARE\n base_query text;\n query text;\n sdate timestamptz;\n edate timestamptz;\n n int;\n records_left int := _limit;\n timer timestamptz := clock_timestamp();\n full_timer timestamptz := clock_timestamp();\nBEGIN\nIF _where IS NULL OR trim(_where) = '' THEN\n _where = ' TRUE ';\nEND IF;\nRAISE NOTICE 'Getting chunks for % %', _where, _orderby;\n\nbase_query := $q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n LIMIT %L\n$q$;\n\nIF _orderby ILIKE 'datetime d%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 DESC LOOP\n RAISE NOTICE 'Running Query for % to %. %', sdate, edate, age_ms(full_timer);\n query := format(\n base_query,\n sdate,\n edate,\n _where,\n _orderby,\n records_left\n );\n RAISE DEBUG 'QUERY: %', query;\n timer := clock_timestamp();\n RETURN QUERY EXECUTE query;\n\n GET DIAGNOSTICS n = ROW_COUNT;\n records_left := records_left - n;\n RAISE NOTICE 'Returned %/% Rows From % to %. % to go. Time: %ms', n, _limit, sdate, edate, records_left, age_ms(timer);\n timer := clock_timestamp();\n IF records_left <= 0 THEN\n RAISE NOTICE 'SEARCH_ROWS TOOK %ms', age_ms(full_timer);\n RETURN;\n END IF;\n END LOOP;\nELSIF _orderby ILIKE 'datetime a%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 ASC LOOP\n RAISE NOTICE 'Running Query for % to %. %', sdate, edate, age_ms(full_timer);\n query := format(\n base_query,\n sdate,\n edate,\n _where,\n _orderby,\n records_left\n );\n RAISE DEBUG 'QUERY: %', query;\n timer := clock_timestamp();\n RETURN QUERY EXECUTE query;\n\n GET DIAGNOSTICS n = ROW_COUNT;\n records_left := records_left - n;\n RAISE NOTICE 'Returned %/% Rows From % to %. % to go. Time: %ms', n, _limit, sdate, edate, records_left, age_ms(timer);\n timer := clock_timestamp();\n IF records_left <= 0 THEN\n RAISE NOTICE 'SEARCH_ROWS TOOK %ms', age_ms(full_timer);\n RETURN;\n END IF;\n END LOOP;\nELSE\n query := format($q$\n SELECT * FROM items\n WHERE %s\n ORDER BY %s\n LIMIT %L\n $q$, _where, _orderby, _limit\n );\n RAISE DEBUG 'QUERY: %', query;\n timer := clock_timestamp();\n RETURN QUERY EXECUTE query;\n RAISE NOTICE 'FULL QUERY TOOK %ms', age_ms(timer);\nEND IF;\nRAISE NOTICE 'SEARCH_ROWS TOOK %ms', age_ms(full_timer);\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE UNLOGGED TABLE format_item_cache(\n id text,\n collection text,\n fields text,\n hydrated bool,\n output jsonb,\n lastused timestamptz DEFAULT now(),\n usecount int DEFAULT 1,\n timetoformat float,\n PRIMARY KEY (collection, id, fields, hydrated)\n);\nCREATE INDEX ON format_item_cache (lastused);\n\nCREATE OR REPLACE FUNCTION format_item(_item items, _fields jsonb DEFAULT '{}', _hydrated bool DEFAULT TRUE) RETURNS jsonb AS $$\nDECLARE\n cache bool := get_setting_bool('format_cache');\n _output jsonb := null;\n t timestamptz := clock_timestamp();\nBEGIN\n IF cache THEN\n SELECT output INTO _output FROM format_item_cache\n WHERE id=_item.id AND collection=_item.collection AND fields=_fields::text AND hydrated=_hydrated;\n END IF;\n IF _output IS NULL THEN\n IF _hydrated THEN\n _output := content_hydrate(_item, _fields);\n ELSE\n _output := content_nonhydrated(_item, _fields);\n END IF;\n END IF;\n IF cache THEN\n INSERT INTO format_item_cache (id, collection, fields, hydrated, output, timetoformat)\n VALUES (_item.id, _item.collection, _fields::text, _hydrated, _output, age_ms(t))\n ON CONFLICT(collection, id, fields, hydrated) DO\n UPDATE\n SET lastused=now(), usecount = format_item_cache.usecount + 1\n ;\n END IF;\n RETURN _output;\n\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION search(_search jsonb = '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n searches searches%ROWTYPE;\n _where text;\n orderby text;\n search_where search_wheres%ROWTYPE;\n total_count bigint;\n token record;\n token_prev boolean;\n token_item items%ROWTYPE;\n token_where text;\n full_where text;\n init_ts timestamptz := clock_timestamp();\n timer timestamptz := clock_timestamp();\n hydrate bool := NOT (_search->'conf'->>'nohydrate' IS NOT NULL AND (_search->'conf'->>'nohydrate')::boolean = true);\n prev text;\n next text;\n context jsonb;\n collection jsonb;\n out_records jsonb;\n out_len int;\n _limit int := coalesce((_search->>'limit')::int, 10);\n _querylimit int;\n _fields jsonb := coalesce(_search->'fields', '{}'::jsonb);\n has_prev boolean := FALSE;\n has_next boolean := FALSE;\n links jsonb := '[]'::jsonb;\n base_url text:= concat(rtrim(base_url(_search->'conf'),'/'));\nBEGIN\n searches := search_query(_search);\n _where := searches._where;\n orderby := searches.orderby;\n search_where := where_stats(_where);\n total_count := coalesce(search_where.total_count, search_where.estimated_count);\n RAISE NOTICE 'SEARCH:TOKEN: %', _search->>'token';\n token := get_token_record(_search->>'token');\n RAISE NOTICE '***TOKEN: %', token;\n _querylimit := _limit + 1;\n IF token IS NOT NULL THEN\n token_prev := token.prev;\n token_item := token.item;\n token_where := get_token_filter(_search->'sortby', token_item, token_prev, FALSE);\n RAISE DEBUG 'TOKEN_WHERE: % (%ms from search start)', token_where, age_ms(timer);\n IF token_prev THEN -- if we are using a prev token, we know has_next is true\n RAISE DEBUG 'There is a previous token, so automatically setting has_next to true';\n has_next := TRUE;\n orderby := sort_sqlorderby(_search, TRUE);\n ELSE\n RAISE DEBUG 'There is a next token, so automatically setting has_prev to true';\n has_prev := TRUE;\n\n END IF;\n ELSE -- if there was no token, we know there is no prev\n RAISE DEBUG 'There is no token, so we know there is no prev. setting has_prev to false';\n has_prev := FALSE;\n END IF;\n\n full_where := concat_ws(' AND ', _where, token_where);\n RAISE NOTICE 'FULL WHERE CLAUSE: %', full_where;\n RAISE NOTICE 'Time to get counts and build query %', age_ms(timer);\n timer := clock_timestamp();\n\n IF hydrate THEN\n RAISE NOTICE 'Getting hydrated data.';\n ELSE\n RAISE NOTICE 'Getting non-hydrated data.';\n END IF;\n RAISE NOTICE 'CACHE SET TO %', get_setting_bool('format_cache');\n RAISE NOTICE 'Time to set hydration/formatting %', age_ms(timer);\n timer := clock_timestamp();\n SELECT jsonb_agg(format_item(i, _fields, hydrate)) INTO out_records\n FROM search_rows(\n full_where,\n orderby,\n search_where.partitions,\n _querylimit\n ) as i;\n\n RAISE NOTICE 'Time to fetch rows %', age_ms(timer);\n timer := clock_timestamp();\n\n\n IF token_prev THEN\n out_records := flip_jsonb_array(out_records);\n END IF;\n\n RAISE NOTICE 'Query returned % records.', jsonb_array_length(out_records);\n RAISE DEBUG 'TOKEN: % %', token_item.id, token_item.collection;\n RAISE DEBUG 'RECORD_1: % %', out_records->0->>'id', out_records->0->>'collection';\n RAISE DEBUG 'RECORD-1: % %', out_records->-1->>'id', out_records->-1->>'collection';\n\n -- REMOVE records that were from our token\n IF out_records->0->>'id' = token_item.id AND out_records->0->>'collection' = token_item.collection THEN\n out_records := out_records - 0;\n ELSIF out_records->-1->>'id' = token_item.id AND out_records->-1->>'collection' = token_item.collection THEN\n out_records := out_records - -1;\n END IF;\n\n out_len := jsonb_array_length(out_records);\n\n IF out_len = _limit + 1 THEN\n IF token_prev THEN\n has_prev := TRUE;\n out_records := out_records - 0;\n ELSE\n has_next := TRUE;\n out_records := out_records - -1;\n END IF;\n END IF;\n\n\n links := links || jsonb_build_object(\n 'rel', 'root',\n 'type', 'application/json',\n 'href', base_url\n ) || jsonb_build_object(\n 'rel', 'self',\n 'type', 'application/json',\n 'href', concat(base_url, '/search')\n );\n\n IF has_next THEN\n next := concat(out_records->-1->>'collection', ':', out_records->-1->>'id');\n RAISE NOTICE 'HAS NEXT | %', next;\n links := links || jsonb_build_object(\n 'rel', 'next',\n 'type', 'application/geo+json',\n 'method', 'GET',\n 'href', concat(base_url, '/search?token=next:', next)\n );\n END IF;\n\n IF has_prev THEN\n prev := concat(out_records->0->>'collection', ':', out_records->0->>'id');\n RAISE NOTICE 'HAS PREV | %', prev;\n links := links || jsonb_build_object(\n 'rel', 'prev',\n 'type', 'application/geo+json',\n 'method', 'GET',\n 'href', concat(base_url, '/search?token=prev:', prev)\n );\n END IF;\n\n RAISE NOTICE 'Time to get prev/next %', age_ms(timer);\n timer := clock_timestamp();\n\n\n collection := jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb),\n 'links', links\n );\n\n\n\n IF context(_search->'conf') != 'off' THEN\n collection := collection || jsonb_strip_nulls(jsonb_build_object(\n 'numberMatched', total_count,\n 'numberReturned', coalesce(jsonb_array_length(out_records), 0)\n ));\n ELSE\n collection := collection || jsonb_strip_nulls(jsonb_build_object(\n 'numberReturned', coalesce(jsonb_array_length(out_records), 0)\n ));\n END IF;\n\n IF get_setting_bool('timing', _search->'conf') THEN\n collection = collection || jsonb_build_object('timing', age_ms(init_ts));\n END IF;\n\n RAISE NOTICE 'Time to build final json %', age_ms(timer);\n timer := clock_timestamp();\n\n RAISE NOTICE 'Total Time: %', age_ms(current_timestamp);\n RAISE NOTICE 'RETURNING % records. NEXT: %. PREV: %', collection->>'numberReturned', collection->>'next', collection->>'prev';\n RETURN collection;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION search_cursor(_search jsonb = '{}'::jsonb) RETURNS refcursor AS $$\nDECLARE\n curs refcursor;\n searches searches%ROWTYPE;\n _where text;\n _orderby text;\n q text;\n\nBEGIN\n searches := search_query(_search);\n _where := searches._where;\n _orderby := searches.orderby;\n\n OPEN curs FOR\n WITH p AS (\n SELECT * FROM partition_queries(_where, _orderby) p\n )\n SELECT\n CASE WHEN EXISTS (SELECT 1 FROM p) THEN\n (SELECT format($q$\n SELECT * FROM (\n %s\n ) total\n $q$,\n string_agg(\n format($q$ SELECT * FROM ( %s ) AS sub $q$, p),\n '\n UNION ALL\n '\n )\n ))\n ELSE NULL\n END FROM p;\n RETURN curs;\nEND;\n$$ LANGUAGE PLPGSQL;\nCREATE OR REPLACE VIEW collections_asitems AS\nSELECT\n id,\n geometry,\n 'collections' AS collection,\n datetime,\n end_datetime,\n jsonb_build_object(\n 'properties', content - '{links,assets,stac_version,stac_extensions}',\n 'links', content->'links',\n 'assets', content->'assets',\n 'stac_version', content->'stac_version',\n 'stac_extensions', content->'stac_extensions'\n ) AS content,\n content as collectionjson\nFROM collections;\n\n\nCREATE OR REPLACE FUNCTION collection_search_matched(\n IN _search jsonb DEFAULT '{}'::jsonb,\n OUT matched bigint\n) RETURNS bigint AS $$\nDECLARE\n _where text := stac_search_to_where(_search);\nBEGIN\n EXECUTE format(\n $query$\n SELECT\n count(*)\n FROM\n collections_asitems\n WHERE %s\n ;\n $query$,\n _where\n ) INTO matched;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION collection_search_rows(\n _search jsonb DEFAULT '{}'::jsonb\n) RETURNS SETOF jsonb AS $$\nDECLARE\n _where text := stac_search_to_where(_search);\n _limit int := coalesce((_search->>'limit')::int, 10);\n _fields jsonb := coalesce(_search->'fields', '{}'::jsonb);\n _orderby text;\n _offset int := COALESCE((_search->>'offset')::int, 0);\nBEGIN\n _orderby := sort_sqlorderby(\n jsonb_build_object(\n 'sortby',\n coalesce(\n _search->'sortby',\n '[{\"field\": \"id\", \"direction\": \"asc\"}]'::jsonb\n )\n )\n );\n RETURN QUERY EXECUTE format(\n $query$\n SELECT\n jsonb_fields(collectionjson, %L) as c\n FROM\n collections_asitems\n WHERE %s\n ORDER BY %s\n LIMIT %L\n OFFSET %L\n ;\n $query$,\n _fields,\n _where,\n _orderby,\n _limit,\n _offset\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION collection_search(\n _search jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\nDECLARE\n out_records jsonb;\n number_matched bigint := collection_search_matched(_search);\n number_returned bigint;\n _limit int := coalesce((_search->>'limit')::float::int, 10);\n _offset int := coalesce((_search->>'offset')::float::int, 0);\n links jsonb := '[]';\n ret jsonb;\n base_url text:= concat(rtrim(base_url(_search->'conf'),'/'), '/collections');\n prevoffset int;\n nextoffset int;\nBEGIN\n SELECT\n coalesce(jsonb_agg(c), '[]')\n INTO out_records\n FROM collection_search_rows(_search) c;\n\n number_returned := jsonb_array_length(out_records);\n RAISE DEBUG 'nm: %, nr: %, l:%, o:%', number_matched, number_returned, _limit, _offset;\n\n\n\n IF _limit <= number_matched AND number_matched > 0 THEN --need to have paging links\n nextoffset := least(_offset + _limit, number_matched - 1);\n prevoffset := greatest(_offset - _limit, 0);\n\n IF _offset > 0 THEN\n links := links || jsonb_build_object(\n 'rel', 'prev',\n 'type', 'application/json',\n 'method', 'GET' ,\n 'href', base_url,\n 'body', jsonb_build_object('offset', prevoffset),\n 'merge', TRUE\n );\n END IF;\n\n IF (_offset + _limit < number_matched) THEN\n links := links || jsonb_build_object(\n 'rel', 'next',\n 'type', 'application/json',\n 'method', 'GET' ,\n 'href', base_url,\n 'body', jsonb_build_object('offset', nextoffset),\n 'merge', TRUE\n );\n END IF;\n\n END IF;\n\n ret := jsonb_build_object(\n 'collections', out_records,\n 'numberMatched', number_matched,\n 'numberReturned', number_returned,\n 'links', links\n );\n RETURN ret;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION tileenvelope(zoom int, x int, y int) RETURNS geometry AS $$\nWITH t AS (\n SELECT\n 20037508.3427892 as merc_max,\n -20037508.3427892 as merc_min,\n (2 * 20037508.3427892) / (2 ^ zoom) as tile_size\n)\nSELECT st_makeenvelope(\n merc_min + (tile_size * x),\n merc_max - (tile_size * (y + 1)),\n merc_min + (tile_size * (x + 1)),\n merc_max - (tile_size * y),\n 3857\n) FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;DROP FUNCTION IF EXISTS mercgrid;\n\n\nCREATE OR REPLACE FUNCTION ftime() RETURNS interval as $$\nSELECT age(clock_timestamp(), transaction_timestamp());\n$$ LANGUAGE SQL;\nSET SEARCH_PATH to pgstac, public;\n\nDROP FUNCTION IF EXISTS geometrysearch;\nCREATE OR REPLACE FUNCTION geometrysearch(\n IN geom geometry,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE, -- Return as soon as the passed in geometry is full covered\n IN skipcovered boolean DEFAULT TRUE -- Skip any items that would show up completely under the previous items\n) RETURNS jsonb AS $$\nDECLARE\n search searches%ROWTYPE;\n curs refcursor;\n _where text;\n query text;\n iter_record items%ROWTYPE;\n out_records jsonb := '{}'::jsonb[];\n exit_flag boolean := FALSE;\n counter int := 1;\n scancounter int := 1;\n remaining_limit int := _scanlimit;\n tilearea float;\n unionedgeom geometry;\n clippedgeom geometry;\n unionedgeom_area float := 0;\n prev_area float := 0;\n excludes text[];\n includes text[];\n\nBEGIN\n DROP TABLE IF EXISTS pgstac_results;\n CREATE TEMP TABLE pgstac_results (content jsonb) ON COMMIT DROP;\n\n -- If the passed in geometry is not an area set exitwhenfull and skipcovered to false\n IF ST_GeometryType(geom) !~* 'polygon' THEN\n RAISE NOTICE 'GEOMETRY IS NOT AN AREA';\n skipcovered = FALSE;\n exitwhenfull = FALSE;\n END IF;\n\n -- If skipcovered is true then you will always want to exit when the passed in geometry is full\n IF skipcovered THEN\n exitwhenfull := TRUE;\n END IF;\n\n search := search_fromhash(queryhash);\n\n IF search IS NULL THEN\n RAISE EXCEPTION 'Search with Query Hash % Not Found', queryhash;\n END IF;\n\n tilearea := st_area(geom);\n _where := format('%s AND st_intersects(geometry, %L::geometry)', search._where, geom);\n\n\n FOR query IN SELECT * FROM partition_queries(_where, search.orderby) LOOP\n query := format('%s LIMIT %L', query, remaining_limit);\n RAISE NOTICE '%', query;\n OPEN curs FOR EXECUTE query;\n LOOP\n FETCH curs INTO iter_record;\n EXIT WHEN NOT FOUND;\n IF exitwhenfull OR skipcovered THEN -- If we are not using exitwhenfull or skipcovered, we do not need to do expensive geometry operations\n clippedgeom := st_intersection(geom, iter_record.geometry);\n\n IF unionedgeom IS NULL THEN\n unionedgeom := clippedgeom;\n ELSE\n unionedgeom := st_union(unionedgeom, clippedgeom);\n END IF;\n\n unionedgeom_area := st_area(unionedgeom);\n\n IF skipcovered AND prev_area = unionedgeom_area THEN\n scancounter := scancounter + 1;\n CONTINUE;\n END IF;\n\n prev_area := unionedgeom_area;\n\n RAISE NOTICE '% % % %', unionedgeom_area/tilearea, counter, scancounter, ftime();\n END IF;\n RAISE NOTICE '% %', iter_record, content_hydrate(iter_record, fields);\n INSERT INTO pgstac_results (content) VALUES (content_hydrate(iter_record, fields));\n\n IF counter >= _limit\n OR scancounter > _scanlimit\n OR ftime() > _timelimit\n OR (exitwhenfull AND unionedgeom_area >= tilearea)\n THEN\n exit_flag := TRUE;\n EXIT;\n END IF;\n counter := counter + 1;\n scancounter := scancounter + 1;\n\n END LOOP;\n CLOSE curs;\n EXIT WHEN exit_flag;\n remaining_limit := _scanlimit - scancounter;\n END LOOP;\n\n SELECT jsonb_agg(content) INTO out_records FROM pgstac_results WHERE content IS NOT NULL;\n\n RETURN jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb)\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDROP FUNCTION IF EXISTS geojsonsearch;\nCREATE OR REPLACE FUNCTION geojsonsearch(\n IN geojson jsonb,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_geomfromgeojson(geojson),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS xyzsearch;\nCREATE OR REPLACE FUNCTION xyzsearch(\n IN _x int,\n IN _y int,\n IN _z int,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_transform(tileenvelope(_z, _x, _y), 4326),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE PROCEDURE analyze_items() AS $$\nDECLARE\n q text;\n timeout_ts timestamptz;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n SELECT format('ANALYZE (VERBOSE, SKIP_LOCKED) %I;', relname) INTO q\n FROM pg_stat_user_tables\n WHERE relname like '_item%' AND (n_mod_since_analyze>0 OR last_analyze IS NULL) LIMIT 1;\n IF NOT FOUND THEN\n EXIT;\n END IF;\n RAISE NOTICE '%', q;\n EXECUTE q;\n COMMIT;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE PROCEDURE validate_constraints() AS $$\nDECLARE\n q text;\nBEGIN\n FOR q IN\n SELECT\n FORMAT(\n 'ALTER TABLE %I.%I VALIDATE CONSTRAINT %I;',\n nsp.nspname,\n cls.relname,\n con.conname\n )\n\n FROM pg_constraint AS con\n JOIN pg_class AS cls\n ON con.conrelid = cls.oid\n JOIN pg_namespace AS nsp\n ON cls.relnamespace = nsp.oid\n WHERE convalidated = FALSE AND contype in ('c','f')\n AND nsp.nspname = 'pgstac'\n LOOP\n RAISE NOTICE '%', q;\n PERFORM run_or_queue(q);\n COMMIT;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION collection_extent(_collection text, runupdate boolean default false) RETURNS jsonb AS $$\nDECLARE\n geom_extent geometry;\n mind timestamptz;\n maxd timestamptz;\n extent jsonb;\nBEGIN\n IF runupdate THEN\n PERFORM update_partition_stats_q(partition)\n FROM partitions_view WHERE collection=_collection;\n END IF;\n SELECT\n min(lower(dtrange)),\n max(upper(edtrange)),\n st_extent(spatial)\n INTO\n mind,\n maxd,\n geom_extent\n FROM partitions_view\n WHERE collection=_collection;\n\n IF geom_extent IS NOT NULL AND mind IS NOT NULL AND maxd IS NOT NULL THEN\n extent := jsonb_build_object(\n 'spatial', jsonb_build_object(\n 'bbox', to_jsonb(array[array[st_xmin(geom_extent), st_ymin(geom_extent), st_xmax(geom_extent), st_ymax(geom_extent)]])\n ),\n 'temporal', jsonb_build_object(\n 'interval', to_jsonb(array[array[mind, maxd]])\n )\n );\n RETURN extent;\n END IF;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('id', '{\"title\": \"Item ID\",\"description\": \"Item identifier\",\"$ref\": \"https://schemas.stacspec.org/v1.0.0/item-spec/json-schema/item.json#/definitions/core/allOf/2/properties/id\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('geometry', '{\"title\": \"Item Geometry\",\"description\": \"Item Geometry\",\"$ref\": \"https://geojson.org/schema/Feature.json\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('datetime','{\"description\": \"Datetime\",\"type\": \"string\",\"title\": \"Acquired\",\"format\": \"date-time\",\"pattern\": \"(\\\\+00:00|Z)$\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDELETE FROM queryables a USING queryables b\n WHERE a.name = b.name AND a.collection_ids IS NOT DISTINCT FROM b.collection_ids AND a.id > b.id;\n\n\nINSERT INTO pgstac_settings (name, value) VALUES\n ('context', 'off'),\n ('context_estimated_count', '100000'),\n ('context_estimated_cost', '100000'),\n ('context_stats_ttl', '1 day'),\n ('default_filter_lang', 'cql2-json'),\n ('additional_properties', 'true'),\n ('use_queue', 'false'),\n ('queue_timeout', '10 minutes'),\n ('update_collection_extent', 'false'),\n ('format_cache', 'false'),\n ('readonly', 'false')\nON CONFLICT DO NOTHING\n;\n\n\nINSERT INTO cql2_ops (op, template, types) VALUES\n ('eq', '%s = %s', NULL),\n ('neq', '%s != %s', NULL),\n ('ne', '%s != %s', NULL),\n ('!=', '%s != %s', NULL),\n ('<>', '%s != %s', NULL),\n ('lt', '%s < %s', NULL),\n ('lte', '%s <= %s', NULL),\n ('gt', '%s > %s', NULL),\n ('gte', '%s >= %s', NULL),\n ('le', '%s <= %s', NULL),\n ('ge', '%s >= %s', NULL),\n ('=', '%s = %s', NULL),\n ('<', '%s < %s', NULL),\n ('<=', '%s <= %s', NULL),\n ('>', '%s > %s', NULL),\n ('>=', '%s >= %s', NULL),\n ('like', '%s LIKE %s', NULL),\n ('ilike', '%s ILIKE %s', NULL),\n ('+', '%s + %s', NULL),\n ('-', '%s - %s', NULL),\n ('*', '%s * %s', NULL),\n ('/', '%s / %s', NULL),\n ('not', 'NOT (%s)', NULL),\n ('between', '%s BETWEEN %s AND %s', NULL),\n ('isnull', '%s IS NULL', NULL),\n ('upper', 'upper(%s)', NULL),\n ('lower', 'lower(%s)', NULL),\n ('casei', 'upper(%s)', NULL),\n ('accenti', 'unaccent(%s)', NULL)\nON CONFLICT (op) DO UPDATE\n SET\n template = EXCLUDED.template\n;\n\n\nALTER FUNCTION to_text COST 5000;\nALTER FUNCTION to_float COST 5000;\nALTER FUNCTION to_int COST 5000;\nALTER FUNCTION to_tstz COST 5000;\nALTER FUNCTION to_text_array COST 5000;\n\nALTER FUNCTION update_partition_stats SECURITY DEFINER;\nALTER FUNCTION partition_after_triggerfunc SECURITY DEFINER;\nALTER FUNCTION drop_table_constraints SECURITY DEFINER;\nALTER FUNCTION create_table_constraints SECURITY DEFINER;\nALTER FUNCTION check_partition SECURITY DEFINER;\nALTER FUNCTION repartition SECURITY DEFINER;\nALTER FUNCTION where_stats SECURITY DEFINER;\nALTER FUNCTION search_query SECURITY DEFINER;\nALTER FUNCTION format_item SECURITY DEFINER;\nALTER FUNCTION maintain_index SECURITY DEFINER;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nGRANT ALL ON SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac to pgstac_admin;\n\n-- pgstac_read role limited to using function apis\nGRANT EXECUTE ON FUNCTION search TO pgstac_read;\nGRANT EXECUTE ON FUNCTION search_query TO pgstac_read;\nGRANT EXECUTE ON FUNCTION item_by_id TO pgstac_read;\nGRANT EXECUTE ON FUNCTION get_item TO pgstac_read;\nGRANT SELECT ON ALL TABLES IN SCHEMA pgstac TO pgstac_read;\n\n\nGRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_ingest;\nGRANT USAGE ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_ingest;\n\nREVOKE ALL PRIVILEGES ON PROCEDURE run_queued_queries FROM public;\nGRANT ALL ON PROCEDURE run_queued_queries TO pgstac_admin;\n\nREVOKE ALL PRIVILEGES ON FUNCTION run_queued_queries_intransaction FROM public;\nGRANT ALL ON FUNCTION run_queued_queries_intransaction TO pgstac_admin;\n\nRESET ROLE;\n\nSET ROLE pgstac_ingest;\nSELECT update_partition_stats_q(partition) FROM partitions_view;\nSELECT set_version('0.9.2');\n" }, { "path": "src/pgstac/migrations/pgstac.0.9.3-0.9.4.sql", "content": "SET client_min_messages TO WARNING;\nSET SEARCH_PATH to pgstac, public;\nRESET ROLE;\nDO $$\nDECLARE\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='postgis') THEN\n CREATE EXTENSION IF NOT EXISTS postgis;\n END IF;\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='btree_gist') THEN\n CREATE EXTENSION IF NOT EXISTS btree_gist;\n END IF;\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='unaccent') THEN\n CREATE EXTENSION IF NOT EXISTS unaccent;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_admin;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_read;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_ingest;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n\nGRANT pgstac_admin TO current_user;\n\n-- Function to make sure pgstac_admin is the owner of items\nCREATE OR REPLACE FUNCTION pgstac_admin_owns() RETURNS VOID AS $$\nDECLARE\n f RECORD;\nBEGIN\n FOR f IN (\n SELECT\n concat(\n oid::regproc::text,\n '(',\n coalesce(pg_get_function_identity_arguments(oid),''),\n ')'\n ) AS name,\n CASE prokind WHEN 'f' THEN 'FUNCTION' WHEN 'p' THEN 'PROCEDURE' WHEN 'a' THEN 'AGGREGATE' END as typ\n FROM pg_proc\n WHERE\n pronamespace=to_regnamespace('pgstac')\n AND proowner != to_regrole('pgstac_admin')\n AND proname NOT LIKE 'pg_stat%'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n FOR f IN (\n SELECT\n oid::regclass::text as name,\n CASE relkind\n WHEN 'i' THEN 'INDEX'\n WHEN 'I' THEN 'INDEX'\n WHEN 'p' THEN 'TABLE'\n WHEN 'r' THEN 'TABLE'\n WHEN 'v' THEN 'VIEW'\n WHEN 'S' THEN 'SEQUENCE'\n ELSE NULL\n END as typ\n FROM pg_class\n WHERE relnamespace=to_regnamespace('pgstac') and relowner != to_regrole('pgstac_admin') AND relkind IN ('r','p','v','S') AND relname NOT LIKE 'pg_stat'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\nSELECT pgstac_admin_owns();\n\nCREATE SCHEMA IF NOT EXISTS pgstac AUTHORIZATION pgstac_admin;\n\nGRANT ALL ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_admin;\n\nALTER ROLE pgstac_admin SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_read SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_ingest SET SEARCH_PATH TO pgstac, public;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\n\nGRANT pgstac_read TO pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\n\nSET ROLE pgstac_admin;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\nRESET ROLE;\n\nSET ROLE pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\nRESET ROLE;\n\nSET SEARCH_PATH TO pgstac, public;\nSET ROLE pgstac_admin;\n\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS analyze_items;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS validate_constraints;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n-- Install these idempotently as migrations do not put them before trying to modify the collections table\n\n\nCREATE OR REPLACE FUNCTION collection_geom(content jsonb)\nRETURNS geometry AS $$\n WITH box AS (SELECT content->'extent'->'spatial'->'bbox'->0 as box)\n SELECT\n st_makeenvelope(\n (box->>0)::float,\n (box->>1)::float,\n (box->>2)::float,\n (box->>3)::float,\n 4326\n )\n FROM box;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION collection_datetime(content jsonb)\nRETURNS timestamptz AS $$\n SELECT\n CASE\n WHEN\n (content->'extent'->'temporal'->'interval'->0->>0) IS NULL\n THEN '-infinity'::timestamptz\n ELSE\n (content->'extent'->'temporal'->'interval'->0->>0)::timestamptz\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION collection_enddatetime(content jsonb)\nRETURNS timestamptz AS $$\n SELECT\n CASE\n WHEN\n (content->'extent'->'temporal'->'interval'->0->>1) IS NULL\n THEN 'infinity'::timestamptz\n ELSE\n (content->'extent'->'temporal'->'interval'->0->>1)::timestamptz\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n-- BEGIN migra calculated SQL\n-- END migra calculated SQL\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('id', '{\"title\": \"Item ID\",\"description\": \"Item identifier\",\"$ref\": \"https://schemas.stacspec.org/v1.0.0/item-spec/json-schema/item.json#/definitions/core/allOf/2/properties/id\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('geometry', '{\"title\": \"Item Geometry\",\"description\": \"Item Geometry\",\"$ref\": \"https://geojson.org/schema/Feature.json\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('datetime','{\"description\": \"Datetime\",\"type\": \"string\",\"title\": \"Acquired\",\"format\": \"date-time\",\"pattern\": \"(\\\\+00:00|Z)$\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDELETE FROM queryables a USING queryables b\n WHERE a.name = b.name AND a.collection_ids IS NOT DISTINCT FROM b.collection_ids AND a.id > b.id;\n\n\nINSERT INTO pgstac_settings (name, value) VALUES\n ('context', 'off'),\n ('context_estimated_count', '100000'),\n ('context_estimated_cost', '100000'),\n ('context_stats_ttl', '1 day'),\n ('default_filter_lang', 'cql2-json'),\n ('additional_properties', 'true'),\n ('use_queue', 'false'),\n ('queue_timeout', '10 minutes'),\n ('update_collection_extent', 'false'),\n ('format_cache', 'false'),\n ('readonly', 'false')\nON CONFLICT DO NOTHING\n;\n\n\nINSERT INTO cql2_ops (op, template, types) VALUES\n ('eq', '%s = %s', NULL),\n ('neq', '%s != %s', NULL),\n ('ne', '%s != %s', NULL),\n ('!=', '%s != %s', NULL),\n ('<>', '%s != %s', NULL),\n ('lt', '%s < %s', NULL),\n ('lte', '%s <= %s', NULL),\n ('gt', '%s > %s', NULL),\n ('gte', '%s >= %s', NULL),\n ('le', '%s <= %s', NULL),\n ('ge', '%s >= %s', NULL),\n ('=', '%s = %s', NULL),\n ('<', '%s < %s', NULL),\n ('<=', '%s <= %s', NULL),\n ('>', '%s > %s', NULL),\n ('>=', '%s >= %s', NULL),\n ('like', '%s LIKE %s', NULL),\n ('ilike', '%s ILIKE %s', NULL),\n ('+', '%s + %s', NULL),\n ('-', '%s - %s', NULL),\n ('*', '%s * %s', NULL),\n ('/', '%s / %s', NULL),\n ('not', 'NOT (%s)', NULL),\n ('between', '%s BETWEEN %s AND %s', NULL),\n ('isnull', '%s IS NULL', NULL),\n ('upper', 'upper(%s)', NULL),\n ('lower', 'lower(%s)', NULL),\n ('casei', 'upper(%s)', NULL),\n ('accenti', 'unaccent(%s)', NULL)\nON CONFLICT (op) DO UPDATE\n SET\n template = EXCLUDED.template\n;\n\n\nALTER FUNCTION to_text COST 5000;\nALTER FUNCTION to_float COST 5000;\nALTER FUNCTION to_int COST 5000;\nALTER FUNCTION to_tstz COST 5000;\nALTER FUNCTION to_text_array COST 5000;\n\nALTER FUNCTION update_partition_stats SECURITY DEFINER;\nALTER FUNCTION partition_after_triggerfunc SECURITY DEFINER;\nALTER FUNCTION drop_table_constraints SECURITY DEFINER;\nALTER FUNCTION create_table_constraints SECURITY DEFINER;\nALTER FUNCTION check_partition SECURITY DEFINER;\nALTER FUNCTION repartition SECURITY DEFINER;\nALTER FUNCTION where_stats SECURITY DEFINER;\nALTER FUNCTION search_query SECURITY DEFINER;\nALTER FUNCTION format_item SECURITY DEFINER;\nALTER FUNCTION maintain_index SECURITY DEFINER;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nGRANT ALL ON SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac to pgstac_admin;\n\n-- pgstac_read role limited to using function apis\nGRANT EXECUTE ON FUNCTION search TO pgstac_read;\nGRANT EXECUTE ON FUNCTION search_query TO pgstac_read;\nGRANT EXECUTE ON FUNCTION item_by_id TO pgstac_read;\nGRANT EXECUTE ON FUNCTION get_item TO pgstac_read;\nGRANT SELECT ON ALL TABLES IN SCHEMA pgstac TO pgstac_read;\n\n\nGRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_ingest;\nGRANT USAGE ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_ingest;\n\nREVOKE ALL PRIVILEGES ON PROCEDURE run_queued_queries FROM public;\nGRANT ALL ON PROCEDURE run_queued_queries TO pgstac_admin;\n\nREVOKE ALL PRIVILEGES ON FUNCTION run_queued_queries_intransaction FROM public;\nGRANT ALL ON FUNCTION run_queued_queries_intransaction TO pgstac_admin;\n\nRESET ROLE;\n\nSET ROLE pgstac_ingest;\nSELECT update_partition_stats_q(partition) FROM partitions_view;\nSELECT set_version('0.9.4');\n" }, { "path": "src/pgstac/migrations/pgstac.0.9.3.sql", "content": "RESET ROLE;\nDO $$\nDECLARE\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='postgis') THEN\n CREATE EXTENSION IF NOT EXISTS postgis;\n END IF;\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='btree_gist') THEN\n CREATE EXTENSION IF NOT EXISTS btree_gist;\n END IF;\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='unaccent') THEN\n CREATE EXTENSION IF NOT EXISTS unaccent;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_admin;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_read;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_ingest;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n\nGRANT pgstac_admin TO current_user;\n\n-- Function to make sure pgstac_admin is the owner of items\nCREATE OR REPLACE FUNCTION pgstac_admin_owns() RETURNS VOID AS $$\nDECLARE\n f RECORD;\nBEGIN\n FOR f IN (\n SELECT\n concat(\n oid::regproc::text,\n '(',\n coalesce(pg_get_function_identity_arguments(oid),''),\n ')'\n ) AS name,\n CASE prokind WHEN 'f' THEN 'FUNCTION' WHEN 'p' THEN 'PROCEDURE' WHEN 'a' THEN 'AGGREGATE' END as typ\n FROM pg_proc\n WHERE\n pronamespace=to_regnamespace('pgstac')\n AND proowner != to_regrole('pgstac_admin')\n AND proname NOT LIKE 'pg_stat%'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n FOR f IN (\n SELECT\n oid::regclass::text as name,\n CASE relkind\n WHEN 'i' THEN 'INDEX'\n WHEN 'I' THEN 'INDEX'\n WHEN 'p' THEN 'TABLE'\n WHEN 'r' THEN 'TABLE'\n WHEN 'v' THEN 'VIEW'\n WHEN 'S' THEN 'SEQUENCE'\n ELSE NULL\n END as typ\n FROM pg_class\n WHERE relnamespace=to_regnamespace('pgstac') and relowner != to_regrole('pgstac_admin') AND relkind IN ('r','p','v','S') AND relname NOT LIKE 'pg_stat'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\nSELECT pgstac_admin_owns();\n\nCREATE SCHEMA IF NOT EXISTS pgstac AUTHORIZATION pgstac_admin;\n\nGRANT ALL ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_admin;\n\nALTER ROLE pgstac_admin SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_read SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_ingest SET SEARCH_PATH TO pgstac, public;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\n\nGRANT pgstac_read TO pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\n\nSET ROLE pgstac_admin;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\nRESET ROLE;\n\nSET ROLE pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\nRESET ROLE;\n\nSET SEARCH_PATH TO pgstac, public;\nSET ROLE pgstac_admin;\n\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS analyze_items;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS validate_constraints;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n-- Install these idempotently as migrations do not put them before trying to modify the collections table\n\n\nCREATE OR REPLACE FUNCTION collection_geom(content jsonb)\nRETURNS geometry AS $$\n WITH box AS (SELECT content->'extent'->'spatial'->'bbox'->0 as box)\n SELECT\n st_makeenvelope(\n (box->>0)::float,\n (box->>1)::float,\n (box->>2)::float,\n (box->>3)::float,\n 4326\n )\n FROM box;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION collection_datetime(content jsonb)\nRETURNS timestamptz AS $$\n SELECT\n CASE\n WHEN\n (content->'extent'->'temporal'->'interval'->0->>0) IS NULL\n THEN '-infinity'::timestamptz\n ELSE\n (content->'extent'->'temporal'->'interval'->0->>0)::timestamptz\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION collection_enddatetime(content jsonb)\nRETURNS timestamptz AS $$\n SELECT\n CASE\n WHEN\n (content->'extent'->'temporal'->'interval'->0->>1) IS NULL\n THEN 'infinity'::timestamptz\n ELSE\n (content->'extent'->'temporal'->'interval'->0->>1)::timestamptz\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE TABLE IF NOT EXISTS migrations (\n version text PRIMARY KEY,\n datetime timestamptz DEFAULT clock_timestamp() NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION get_version() RETURNS text AS $$\n SELECT version FROM pgstac.migrations ORDER BY datetime DESC, version DESC LIMIT 1;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION set_version(text) RETURNS text AS $$\n INSERT INTO pgstac.migrations (version) VALUES ($1)\n ON CONFLICT DO NOTHING\n RETURNING version;\n$$ LANGUAGE SQL;\n\n\nCREATE TABLE IF NOT EXISTS pgstac_settings (\n name text PRIMARY KEY,\n value text NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION table_empty(text) RETURNS boolean AS $$\nDECLARE\n retval boolean;\nBEGIN\n EXECUTE format($q$\n SELECT NOT EXISTS (SELECT 1 FROM %I LIMIT 1)\n $q$,\n $1\n ) INTO retval;\n RETURN retval;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION get_setting(IN _setting text, IN conf jsonb DEFAULT NULL) RETURNS text AS $$\nSELECT COALESCE(\n nullif(conf->>_setting, ''),\n nullif(current_setting(concat('pgstac.',_setting), TRUE),''),\n nullif((SELECT value FROM pgstac.pgstac_settings WHERE name=_setting),'')\n);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_setting_bool(IN _setting text, IN conf jsonb DEFAULT NULL) RETURNS boolean AS $$\nSELECT COALESCE(\n nullif(conf->>_setting, ''),\n nullif(current_setting(concat('pgstac.',_setting), TRUE),''),\n nullif((SELECT value FROM pgstac.pgstac_settings WHERE name=_setting),''),\n 'FALSE'\n)::boolean;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION base_url(conf jsonb DEFAULT NULL) RETURNS text AS $$\n SELECT COALESCE(pgstac.get_setting('base_url', conf), '.');\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION additional_properties() RETURNS boolean AS $$\n SELECT pgstac.get_setting_bool('additional_properties');\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION readonly(conf jsonb DEFAULT NULL) RETURNS boolean AS $$\n SELECT pgstac.get_setting_bool('readonly', conf);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context(conf jsonb DEFAULT NULL) RETURNS text AS $$\n SELECT pgstac.get_setting('context', conf);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context_estimated_count(conf jsonb DEFAULT NULL) RETURNS int AS $$\n SELECT pgstac.get_setting('context_estimated_count', conf)::int;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_estimated_cost();\nCREATE OR REPLACE FUNCTION context_estimated_cost(conf jsonb DEFAULT NULL) RETURNS float AS $$\n SELECT pgstac.get_setting('context_estimated_cost', conf)::float;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_stats_ttl();\nCREATE OR REPLACE FUNCTION context_stats_ttl(conf jsonb DEFAULT NULL) RETURNS interval AS $$\n SELECT pgstac.get_setting('context_stats_ttl', conf)::interval;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION t2s(text) RETURNS text AS $$\n SELECT extract(epoch FROM $1::interval)::text || ' s';\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE STRICT;\n\nCREATE OR REPLACE FUNCTION age_ms(a timestamptz, b timestamptz DEFAULT clock_timestamp()) RETURNS float AS $$\n SELECT abs(extract(epoch from age(a,b)) * 1000);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION queue_timeout() RETURNS interval AS $$\n SELECT t2s(coalesce(\n get_setting('queue_timeout'),\n '1h'\n ))::interval;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION notice(VARIADIC text[]) RETURNS boolean AS $$\nDECLARE\ndebug boolean := current_setting('pgstac.debug', true);\nBEGIN\n IF debug THEN\n RAISE NOTICE 'NOTICE FROM FUNC: % >>>>> %', concat_ws(' | ', $1), clock_timestamp();\n RETURN TRUE;\n END IF;\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION empty_arr(ANYARRAY) RETURNS BOOLEAN AS $$\nSELECT CASE\n WHEN $1 IS NULL THEN TRUE\n WHEN cardinality($1)<1 THEN TRUE\nELSE FALSE\nEND;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_intersection(_a ANYARRAY, _b ANYARRAY) RETURNS ANYARRAY AS $$\n SELECT ARRAY ( SELECT unnest(_a) INTERSECT SELECT UNNEST(_b) );\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION array_map_ident(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_ident(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_map_literal(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_literal(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION array_reverse(anyarray) RETURNS anyarray AS $$\nSELECT ARRAY(\n SELECT $1[i]\n FROM generate_subscripts($1,1) AS s(i)\n ORDER BY i DESC\n);\n$$ LANGUAGE SQL STRICT IMMUTABLE;\n\nDROP TABLE IF EXISTS query_queue;\nCREATE TABLE query_queue (\n query text PRIMARY KEY,\n added timestamptz DEFAULT now()\n);\n\nDROP TABLE IF EXISTS query_queue_history;\nCREATE TABLE query_queue_history(\n query text,\n added timestamptz NOT NULL,\n finished timestamptz NOT NULL DEFAULT now(),\n error text\n);\n\nCREATE OR REPLACE PROCEDURE run_queued_queries() AS $$\nDECLARE\n qitem query_queue%ROWTYPE;\n timeout_ts timestamptz;\n error text;\n cnt int := 0;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n DELETE FROM query_queue WHERE query = (SELECT query FROM query_queue ORDER BY added DESC LIMIT 1 FOR UPDATE SKIP LOCKED) RETURNING * INTO qitem;\n IF NOT FOUND THEN\n EXIT;\n END IF;\n cnt := cnt + 1;\n BEGIN\n RAISE NOTICE 'RUNNING QUERY: %', qitem.query;\n EXECUTE qitem.query;\n EXCEPTION WHEN others THEN\n error := format('%s | %s', SQLERRM, SQLSTATE);\n END;\n INSERT INTO query_queue_history (query, added, finished, error)\n VALUES (qitem.query, qitem.added, clock_timestamp(), error);\n COMMIT;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION run_queued_queries_intransaction() RETURNS int AS $$\nDECLARE\n qitem query_queue%ROWTYPE;\n timeout_ts timestamptz;\n error text;\n cnt int := 0;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n DELETE FROM query_queue WHERE query = (SELECT query FROM query_queue ORDER BY added DESC LIMIT 1 FOR UPDATE SKIP LOCKED) RETURNING * INTO qitem;\n IF NOT FOUND THEN\n RETURN cnt;\n END IF;\n cnt := cnt + 1;\n BEGIN\n qitem.query := regexp_replace(qitem.query, 'CONCURRENTLY', '');\n RAISE NOTICE 'RUNNING QUERY: %', qitem.query;\n\n EXECUTE qitem.query;\n EXCEPTION WHEN others THEN\n error := format('%s | %s', SQLERRM, SQLSTATE);\n RAISE WARNING '%', error;\n END;\n INSERT INTO query_queue_history (query, added, finished, error)\n VALUES (qitem.query, qitem.added, clock_timestamp(), error);\n END LOOP;\n RETURN cnt;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION run_or_queue(query text) RETURNS VOID AS $$\nDECLARE\n use_queue text := COALESCE(get_setting('use_queue'), 'FALSE')::boolean;\nBEGIN\n IF get_setting_bool('debug') THEN\n RAISE NOTICE '%', query;\n END IF;\n IF use_queue THEN\n INSERT INTO query_queue (query) VALUES (query) ON CONFLICT DO NOTHING;\n ELSE\n EXECUTE query;\n END IF;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n\nDROP FUNCTION IF EXISTS check_pgstac_settings;\nCREATE OR REPLACE FUNCTION check_pgstac_settings(_sysmem text DEFAULT NULL) RETURNS VOID AS $$\nDECLARE\n settingval text;\n sysmem bigint := pg_size_bytes(_sysmem);\n effective_cache_size bigint := pg_size_bytes(current_setting('effective_cache_size', TRUE));\n shared_buffers bigint := pg_size_bytes(current_setting('shared_buffers', TRUE));\n work_mem bigint := pg_size_bytes(current_setting('work_mem', TRUE));\n max_connections int := current_setting('max_connections', TRUE);\n maintenance_work_mem bigint := pg_size_bytes(current_setting('maintenance_work_mem', TRUE));\n seq_page_cost float := current_setting('seq_page_cost', TRUE);\n random_page_cost float := current_setting('random_page_cost', TRUE);\n temp_buffers bigint := pg_size_bytes(current_setting('temp_buffers', TRUE));\n r record;\nBEGIN\n IF _sysmem IS NULL THEN\n RAISE NOTICE 'Call function with the size of your system memory `SELECT check_pgstac_settings(''4GB'')` to get pg system setting recommendations.';\n ELSE\n IF effective_cache_size < (sysmem * 0.5) THEN\n RAISE WARNING 'effective_cache_size of % is set low for a system with %. Recomended value between % and %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.5), pg_size_pretty(sysmem * 0.75);\n ELSIF effective_cache_size > (sysmem * 0.75) THEN\n RAISE WARNING 'effective_cache_size of % is set high for a system with %. Recomended value between % and %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.5), pg_size_pretty(sysmem * 0.75);\n ELSE\n RAISE NOTICE 'effective_cache_size of % is set appropriately for a system with %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem);\n END IF;\n\n IF shared_buffers < (sysmem * 0.2) THEN\n RAISE WARNING 'shared_buffers of % is set low for a system with %. Recomended value between % and %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.2), pg_size_pretty(sysmem * 0.3);\n ELSIF shared_buffers > (sysmem * 0.3) THEN\n RAISE WARNING 'shared_buffers of % is set high for a system with %. Recomended value between % and %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.2), pg_size_pretty(sysmem * 0.3);\n ELSE\n RAISE NOTICE 'shared_buffers of % is set appropriately for a system with %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem);\n END IF;\n shared_buffers = sysmem * 0.3;\n IF maintenance_work_mem < (sysmem * 0.2) THEN\n RAISE WARNING 'maintenance_work_mem of % is set low for shared_buffers of %. Recomended value between % and %', pg_size_pretty(maintenance_work_mem), pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers * 0.2), pg_size_pretty(shared_buffers * 0.3);\n ELSIF maintenance_work_mem > (shared_buffers * 0.3) THEN\n RAISE WARNING 'maintenance_work_mem of % is set high for shared_buffers of %. Recomended value between % and %', pg_size_pretty(maintenance_work_mem), pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers * 0.2), pg_size_pretty(shared_buffers * 0.3);\n ELSE\n RAISE NOTICE 'maintenance_work_mem of % is set appropriately for shared_buffers of %', pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers);\n END IF;\n\n IF work_mem * max_connections > shared_buffers THEN\n RAISE WARNING 'work_mem setting of % is set high for % max_connections please reduce work_mem to % or decrease max_connections to %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers/max_connections), floor(shared_buffers/work_mem);\n ELSIF work_mem * max_connections < (shared_buffers * 0.75) THEN\n RAISE WARNING 'work_mem setting of % is set low for % max_connections you may consider raising work_mem to % or increasing max_connections to %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers/max_connections), floor(shared_buffers/work_mem);\n ELSE\n RAISE NOTICE 'work_mem setting of % and max_connections of % are adequate for shared_buffers of %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers);\n END IF;\n\n IF random_page_cost / seq_page_cost != 1.1 THEN\n RAISE WARNING 'random_page_cost (%) /seq_page_cost (%) should be set to 1.1 for SSD. Change random_page_cost to %', random_page_cost, seq_page_cost, 1.1 * seq_page_cost;\n ELSE\n RAISE NOTICE 'random_page_cost and seq_page_cost set appropriately for SSD';\n END IF;\n\n IF temp_buffers < greatest(pg_size_bytes('128MB'),(maintenance_work_mem / 2)) THEN\n RAISE WARNING 'pgstac makes heavy use of temp tables, consider raising temp_buffers from % to %', pg_size_pretty(temp_buffers), greatest('128MB', pg_size_pretty((shared_buffers / 16)));\n END IF;\n END IF;\n\n RAISE NOTICE 'VALUES FOR PGSTAC VARIABLES';\n RAISE NOTICE 'These can be set either as GUC system variables or by setting in the pgstac_settings table.';\n\n FOR r IN SELECT name, get_setting(name) as setting, CASE WHEN current_setting(concat('pgstac.',name), TRUE) IS NOT NULL THEN concat('pgstac.',name, ' GUC') WHEN value IS NOT NULL THEN 'pgstac_settings table' ELSE 'Not Set' END as loc FROM pgstac_settings LOOP\n RAISE NOTICE '% is set to % from the %', r.name, r.setting, r.loc;\n END LOOP;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pg_cron';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider intalling pg_cron which can be used to automate tasks';\n ELSE\n RAISE NOTICE 'pg_cron % is installed', settingval;\n END IF;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pgstattuple';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider installing the pgstattuple extension which can be used to help maintain tables and indexes.';\n ELSE\n RAISE NOTICE 'pgstattuple % is installed', settingval;\n END IF;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pg_stat_statements';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider installing the pg_stat_statements extension which is very helpful for tracking the types of queries on the system';\n ELSE\n RAISE NOTICE 'pg_stat_statements % is installed', settingval;\n IF current_setting('pg_stat_statements.track_statements', TRUE) IS DISTINCT FROM 'all' THEN\n RAISE WARNING 'SET pg_stat_statements.track_statements TO ''all''; --In order to track statements within functions.';\n END IF;\n END IF;\n\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac, public SET CLIENT_MIN_MESSAGES TO NOTICE;\nCREATE OR REPLACE FUNCTION to_int(jsonb) RETURNS int AS $$\n SELECT floor(($1->>0)::float)::int;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION to_float(jsonb) RETURNS float AS $$\n SELECT ($1->>0)::float;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION to_tstz(jsonb) RETURNS timestamptz AS $$\n SELECT ($1->>0)::timestamptz;\n$$ LANGUAGE SQL IMMUTABLE STRICT SET TIME ZONE 'UTC' COST 5000 PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION to_text(jsonb) RETURNS text AS $$\n SELECT CASE WHEN jsonb_typeof($1) IN ('array','object') THEN $1::text ELSE $1->>0 END;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION to_text_array(jsonb) RETURNS text[] AS $$\n SELECT\n CASE jsonb_typeof($1)\n WHEN 'array' THEN ARRAY(SELECT jsonb_array_elements_text($1))\n ELSE ARRAY[$1->>0]\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION bbox_geom(_bbox jsonb) RETURNS geometry AS $$\nSELECT CASE jsonb_array_length(_bbox)\n WHEN 4 THEN\n ST_SetSRID(ST_MakeEnvelope(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float,\n (_bbox->>3)::float\n ),4326)\n WHEN 6 THEN\n ST_SetSRID(ST_3DMakeBox(\n ST_MakePoint(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float\n ),\n ST_MakePoint(\n (_bbox->>3)::float,\n (_bbox->>4)::float,\n (_bbox->>5)::float\n )\n ),4326)\n ELSE null END;\n;\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION geom_bbox(_geom geometry) RETURNS jsonb AS $$\n SELECT jsonb_build_array(\n st_xmin(_geom),\n st_ymin(_geom),\n st_xmax(_geom),\n st_ymax(_geom)\n );\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION flip_jsonb_array(j jsonb) RETURNS jsonb AS $$\n SELECT jsonb_agg(value) FROM (SELECT value FROM jsonb_array_elements(j) WITH ORDINALITY ORDER BY ordinality DESC) as t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION explode_dotpaths(j jsonb) RETURNS SETOF text[] AS $$\n SELECT string_to_array(p, '.') as e FROM jsonb_array_elements_text(j) p;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION explode_dotpaths_recurse(IN j jsonb) RETURNS SETOF text[] AS $$\n WITH RECURSIVE t AS (\n SELECT e FROM explode_dotpaths(j) e\n UNION ALL\n SELECT e[1:cardinality(e)-1]\n FROM t\n WHERE cardinality(e)>1\n ) SELECT e FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION jsonb_set_nested(j jsonb, path text[], val jsonb) RETURNS jsonb AS $$\nDECLARE\nBEGIN\n IF cardinality(path) > 1 THEN\n FOR i IN 1..(cardinality(path)-1) LOOP\n IF j #> path[:i] IS NULL THEN\n j := jsonb_set_lax(j, path[:i], '{}', TRUE);\n END IF;\n END LOOP;\n END IF;\n RETURN jsonb_set_lax(j, path, val, true);\n\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\n\n\nCREATE OR REPLACE FUNCTION jsonb_include(j jsonb, f jsonb) RETURNS jsonb AS $$\nDECLARE\n includes jsonb := f-> 'include';\n outj jsonb := '{}'::jsonb;\n path text[];\nBEGIN\n IF\n includes IS NULL\n OR jsonb_array_length(includes) = 0\n THEN\n RETURN j;\n ELSE\n includes := includes || '[\"id\",\"collection\"]'::jsonb;\n FOR path IN SELECT explode_dotpaths(includes) LOOP\n outj := jsonb_set_nested(outj, path, j #> path);\n END LOOP;\n END IF;\n RETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_exclude(j jsonb, f jsonb) RETURNS jsonb AS $$\nDECLARE\n excludes jsonb := f-> 'exclude';\n outj jsonb := j;\n path text[];\nBEGIN\n IF\n excludes IS NULL\n OR jsonb_array_length(excludes) = 0\n THEN\n RETURN j;\n ELSE\n FOR path IN SELECT explode_dotpaths(excludes) LOOP\n outj := outj #- path;\n END LOOP;\n END IF;\n RETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_fields(j jsonb, f jsonb DEFAULT '{\"fields\":[]}') RETURNS jsonb AS $$\n SELECT jsonb_exclude(jsonb_include(j, f), f);\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION merge_jsonb(_a jsonb, _b jsonb) RETURNS jsonb AS $$\n SELECT\n CASE\n WHEN _a = '\"𒍟※\"'::jsonb THEN NULL\n WHEN _a IS NULL OR jsonb_typeof(_a) = 'null' THEN _b\n WHEN jsonb_typeof(_a) = 'object' AND jsonb_typeof(_b) = 'object' THEN\n (\n SELECT\n jsonb_strip_nulls(\n jsonb_object_agg(\n key,\n merge_jsonb(a.value, b.value)\n )\n )\n FROM\n jsonb_each(coalesce(_a,'{}'::jsonb)) as a\n FULL JOIN\n jsonb_each(coalesce(_b,'{}'::jsonb)) as b\n USING (key)\n )\n WHEN\n jsonb_typeof(_a) = 'array'\n AND jsonb_typeof(_b) = 'array'\n AND jsonb_array_length(_a) = jsonb_array_length(_b)\n THEN\n (\n SELECT jsonb_agg(m) FROM\n ( SELECT\n merge_jsonb(\n jsonb_array_elements(_a),\n jsonb_array_elements(_b)\n ) as m\n ) as l\n )\n ELSE _a\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION strip_jsonb(_a jsonb, _b jsonb) RETURNS jsonb AS $$\n SELECT\n CASE\n\n WHEN (_a IS NULL OR jsonb_typeof(_a) = 'null') AND _b IS NOT NULL AND jsonb_typeof(_b) != 'null' THEN '\"𒍟※\"'::jsonb\n WHEN _b IS NULL OR jsonb_typeof(_a) = 'null' THEN _a\n WHEN _a = _b AND jsonb_typeof(_a) = 'object' THEN '{}'::jsonb\n WHEN _a = _b THEN NULL\n WHEN jsonb_typeof(_a) = 'object' AND jsonb_typeof(_b) = 'object' THEN\n (\n SELECT\n jsonb_strip_nulls(\n jsonb_object_agg(\n key,\n strip_jsonb(a.value, b.value)\n )\n )\n FROM\n jsonb_each(_a) as a\n FULL JOIN\n jsonb_each(_b) as b\n USING (key)\n )\n WHEN\n jsonb_typeof(_a) = 'array'\n AND jsonb_typeof(_b) = 'array'\n AND jsonb_array_length(_a) = jsonb_array_length(_b)\n THEN\n (\n SELECT jsonb_agg(m) FROM\n ( SELECT\n strip_jsonb(\n jsonb_array_elements(_a),\n jsonb_array_elements(_b)\n ) as m\n ) as l\n )\n ELSE _a\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION nullif_jsonbnullempty(j jsonb) RETURNS jsonb AS $$\n SELECT nullif(nullif(nullif(j,'null'::jsonb),'{}'::jsonb),'[]'::jsonb);\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION jsonb_array_unique(j jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(jsonb_agg(DISTINCT a)) v FROM jsonb_array_elements(j) a;\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_concat_ignorenull(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT coalesce(a,'[]'::jsonb) || coalesce(b,'[]'::jsonb);\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_least(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(least(nullif_jsonbnullempty(a), nullif_jsonbnullempty(b)));\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_greatest(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(greatest(a, b));\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION first_notnull_sfunc(anyelement, anyelement) RETURNS anyelement AS $$\n SELECT COALESCE($1,$2);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE AGGREGATE first_notnull(anyelement)(\n SFUNC = first_notnull_sfunc,\n STYPE = anyelement\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_array_unique_merge(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_concat_ignorenull,\n FINALFUNC = jsonb_array_unique\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_min(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_least\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_max(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_greatest\n);\n/* looks for a geometry in a stac item first from geometry and falling back to bbox */\nCREATE OR REPLACE FUNCTION stac_geom(value jsonb) RETURNS geometry AS $$\nSELECT\n CASE\n WHEN value ? 'intersects' THEN\n ST_GeomFromGeoJSON(value->>'intersects')\n WHEN value ? 'geometry' THEN\n ST_GeomFromGeoJSON(value->>'geometry')\n WHEN value ? 'bbox' THEN\n pgstac.bbox_geom(value->'bbox')\n ELSE NULL\n END as geometry\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION stac_daterange(\n value jsonb\n) RETURNS tstzrange AS $$\nDECLARE\n props jsonb := value;\n dt timestamptz;\n edt timestamptz;\nBEGIN\n IF props ? 'properties' THEN\n props := props->'properties';\n END IF;\n IF\n props ? 'start_datetime'\n AND props->>'start_datetime' IS NOT NULL\n AND props ? 'end_datetime'\n AND props->>'end_datetime' IS NOT NULL\n THEN\n dt := props->>'start_datetime';\n edt := props->>'end_datetime';\n IF dt > edt THEN\n RAISE EXCEPTION 'start_datetime must be < end_datetime';\n END IF;\n ELSE\n dt := props->>'datetime';\n edt := props->>'datetime';\n END IF;\n IF dt is NULL OR edt IS NULL THEN\n RAISE NOTICE 'DT: %, EDT: %', dt, edt;\n RAISE EXCEPTION 'Either datetime (%) or both start_datetime (%) and end_datetime (%) must be set.', props->>'datetime',props->>'start_datetime',props->>'end_datetime';\n END IF;\n RETURN tstzrange(dt, edt, '[]');\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_datetime(value jsonb) RETURNS timestamptz AS $$\n SELECT lower(stac_daterange(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_end_datetime(value jsonb) RETURNS timestamptz AS $$\n SELECT upper(stac_daterange(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE TABLE IF NOT EXISTS stac_extensions(\n url text PRIMARY KEY,\n content jsonb\n);\nCREATE OR REPLACE FUNCTION collection_base_item(content jsonb) RETURNS jsonb AS $$\n SELECT jsonb_build_object(\n 'type', 'Feature',\n 'stac_version', content->'stac_version',\n 'assets', content->'item_assets',\n 'collection', content->'id'\n );\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE TABLE IF NOT EXISTS collections (\n key bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,\n id text GENERATED ALWAYS AS (content->>'id') STORED UNIQUE NOT NULL,\n content JSONB NOT NULL,\n base_item jsonb GENERATED ALWAYS AS (pgstac.collection_base_item(content)) STORED,\n geometry geometry GENERATED ALWAYS AS (pgstac.collection_geom(content)) STORED,\n datetime timestamptz GENERATED ALWAYS AS (pgstac.collection_datetime(content)) STORED,\n end_datetime timestamptz GENERATED ALWAYS AS (pgstac.collection_enddatetime(content)) STORED,\n private jsonb,\n partition_trunc text CHECK (partition_trunc IN ('year', 'month'))\n);\n\nCREATE OR REPLACE FUNCTION collection_base_item(cid text) RETURNS jsonb AS $$\n SELECT pgstac.collection_base_item(content) FROM pgstac.collections WHERE id = cid LIMIT 1;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION create_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION update_collection(data jsonb) RETURNS VOID AS $$\nDECLARE\n out collections%ROWTYPE;\nBEGIN\n UPDATE collections SET content=data WHERE id = data->>'id' RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ON CONFLICT (id) DO\n UPDATE\n SET content=EXCLUDED.content\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION delete_collection(_id text) RETURNS VOID AS $$\nDECLARE\n out collections%ROWTYPE;\nBEGIN\n DELETE FROM collections WHERE id = _id RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION get_collection(id text) RETURNS jsonb AS $$\n SELECT content FROM collections\n WHERE id=$1\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION all_collections() RETURNS jsonb AS $$\n SELECT coalesce(jsonb_agg(content), '[]'::jsonb) FROM collections;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION collection_delete_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n collection_base_partition text := concat('_items_', OLD.key);\nBEGIN\n EXECUTE format($q$\n DELETE FROM partition_stats WHERE partition IN (\n SELECT partition FROM partition_sys_meta\n WHERE collection=%L\n );\n DROP TABLE IF EXISTS %I CASCADE;\n $q$,\n OLD.id,\n collection_base_partition\n );\n RETURN OLD;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER collection_delete_trigger BEFORE DELETE ON collections\nFOR EACH ROW EXECUTE FUNCTION collection_delete_trigger_func();\nCREATE OR REPLACE FUNCTION queryable_signature(n text, c text[]) RETURNS text AS $$\n SELECT concat(n, c);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TABLE queryables (\n id bigint GENERATED ALWAYS AS identity PRIMARY KEY,\n name text NOT NULL,\n collection_ids text[], -- used to determine what partitions to create indexes on\n definition jsonb,\n property_path text,\n property_wrapper text,\n property_index_type text\n);\nCREATE INDEX queryables_name_idx ON queryables (name);\nCREATE INDEX queryables_collection_idx ON queryables USING GIN (collection_ids);\nCREATE INDEX queryables_property_wrapper_idx ON queryables (property_wrapper);\n\nCREATE OR REPLACE FUNCTION pgstac.queryables_constraint_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n allcollections text[];\nBEGIN\n RAISE NOTICE 'Making sure that name/collection is unique for queryables %', NEW;\n IF NEW.collection_ids IS NOT NULL THEN\n IF EXISTS (\n SELECT 1\n FROM unnest(NEW.collection_ids) c\n LEFT JOIN\n collections\n ON (collections.id = c)\n WHERE collections.id IS NULL\n ) THEN\n RAISE foreign_key_violation USING MESSAGE = format(\n 'One or more collections in %s do not exist.', NEW.collection_ids\n );\n RETURN NULL;\n END IF;\n END IF;\n IF TG_OP = 'INSERT' THEN\n IF EXISTS (\n SELECT 1 FROM queryables q\n WHERE\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n )\n ) THEN\n RAISE unique_violation USING MESSAGE = format(\n 'There is already a queryable for %s for a collection in %s: %s',\n NEW.name,\n NEW.collection_ids,\n\t\t\t\t(SELECT json_agg(row_to_json(q)) FROM queryables q WHERE\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n ))\n );\n RETURN NULL;\n END IF;\n END IF;\n IF TG_OP = 'UPDATE' THEN\n IF EXISTS (\n SELECT 1 FROM queryables q\n WHERE\n q.id != NEW.id\n AND\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n )\n ) THEN\n RAISE unique_violation\n USING MESSAGE = format(\n 'There is already a queryable for %s for a collection in %s',\n NEW.name,\n NEW.collection_ids\n );\n RETURN NULL;\n END IF;\n END IF;\n\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER queryables_constraint_insert_trigger\nBEFORE INSERT ON queryables\nFOR EACH ROW EXECUTE PROCEDURE queryables_constraint_triggerfunc();\n\nCREATE TRIGGER queryables_constraint_update_trigger\nBEFORE UPDATE ON queryables\nFOR EACH ROW\nWHEN (NEW.name = OLD.name AND NEW.collection_ids IS DISTINCT FROM OLD.collection_ids)\nEXECUTE PROCEDURE queryables_constraint_triggerfunc();\n\n\nCREATE OR REPLACE FUNCTION array_to_path(arr text[]) RETURNS text AS $$\n SELECT string_agg(\n quote_literal(v),\n '->'\n ) FROM unnest(arr) v;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\n\n\nCREATE OR REPLACE FUNCTION queryable(\n IN dotpath text,\n OUT path text,\n OUT expression text,\n OUT wrapper text,\n OUT nulled_wrapper text\n) AS $$\nDECLARE\n q RECORD;\n path_elements text[];\nBEGIN\n dotpath := replace(dotpath, 'properties.', '');\n IF dotpath = 'start_datetime' THEN\n dotpath := 'datetime';\n END IF;\n IF dotpath IN ('id', 'geometry', 'datetime', 'end_datetime', 'collection') THEN\n path := dotpath;\n expression := dotpath;\n wrapper := NULL;\n RETURN;\n END IF;\n\n SELECT * INTO q FROM queryables\n WHERE\n name=dotpath\n OR name = 'properties.' || dotpath\n OR name = replace(dotpath, 'properties.', '')\n ;\n IF q.property_wrapper IS NULL THEN\n IF q.definition->>'type' = 'number' THEN\n wrapper := 'to_float';\n nulled_wrapper := wrapper;\n ELSIF q.definition->>'format' = 'date-time' THEN\n wrapper := 'to_tstz';\n nulled_wrapper := wrapper;\n ELSE\n nulled_wrapper := NULL;\n wrapper := 'to_text';\n END IF;\n ELSE\n wrapper := q.property_wrapper;\n nulled_wrapper := wrapper;\n END IF;\n IF q.property_path IS NOT NULL THEN\n path := q.property_path;\n ELSE\n path_elements := string_to_array(dotpath, '.');\n IF path_elements[1] IN ('links', 'assets', 'stac_version', 'stac_extensions') THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSIF path_elements[1] = 'properties' THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSE\n path := format($F$content->'properties'->%s$F$, array_to_path(path_elements));\n END IF;\n END IF;\n expression := format('%I(%s)', wrapper, path);\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\nCREATE OR REPLACE FUNCTION unnest_collection(collection_ids text[] DEFAULT NULL) RETURNS SETOF text AS $$\n DECLARE\n BEGIN\n IF collection_ids IS NULL THEN\n RETURN QUERY SELECT id FROM collections;\n END IF;\n RETURN QUERY SELECT unnest(collection_ids);\n END;\n$$ LANGUAGE PLPGSQL STABLE;\n\nCREATE OR REPLACE FUNCTION normalize_indexdef(def text) RETURNS text AS $$\nDECLARE\nBEGIN\n def := btrim(def, ' \\n\\t');\n\tdef := regexp_replace(def, '^CREATE (UNIQUE )?INDEX ([^ ]* )?ON (ONLY )?([^ ]* )?', '', 'i');\n RETURN def;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION indexdef(q queryables) RETURNS text AS $$\n DECLARE\n out text;\n BEGIN\n IF q.name = 'id' THEN\n out := 'CREATE UNIQUE INDEX ON %I USING btree (id)';\n ELSIF q.name = 'datetime' THEN\n out := 'CREATE INDEX ON %I USING btree (datetime DESC, end_datetime)';\n ELSIF q.name = 'geometry' THEN\n out := 'CREATE INDEX ON %I USING gist (geometry)';\n ELSE\n out := format($q$CREATE INDEX ON %%I USING %s (%s(((content -> 'properties'::text) -> %L::text)))$q$,\n lower(COALESCE(q.property_index_type, 'BTREE')),\n lower(COALESCE(q.property_wrapper, 'to_text')),\n q.name\n );\n END IF;\n RETURN btrim(out, ' \\n\\t');\n END;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nDROP VIEW IF EXISTS pgstac_indexes;\nCREATE VIEW pgstac_indexes AS\nSELECT\n i.schemaname,\n i.tablename,\n i.indexname,\n regexp_replace(btrim(replace(replace(indexdef, i.indexname, ''),'pgstac.',''),' \\t\\n'), '[ ]+', ' ', 'g') as idx,\n COALESCE(\n (regexp_match(indexdef, '\\(([a-zA-Z]+)\\)'))[1],\n (regexp_match(indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_-]+)''::text'))[1],\n CASE WHEN indexdef ~* '\\(datetime desc, end_datetime\\)' THEN 'datetime' ELSE NULL END\n ) AS field,\n pg_table_size(i.indexname::text) as index_size,\n pg_size_pretty(pg_table_size(i.indexname::text)) as index_size_pretty\nFROM\n pg_indexes i\nWHERE i.schemaname='pgstac' and i.tablename ~ '_items_' AND indexdef !~* ' only ';\n\nDROP VIEW IF EXISTS pgstac_index_stats;\nCREATE VIEW pgstac_indexes_stats AS\nSELECT\n i.schemaname,\n i.tablename,\n i.indexname,\n indexdef,\n COALESCE(\n (regexp_match(indexdef, '\\(([a-zA-Z]+)\\)'))[1],\n (regexp_match(indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_]+)''::text'))[1],\n CASE WHEN indexdef ~* '\\(datetime desc, end_datetime\\)' THEN 'datetime_end_datetime' ELSE NULL END\n ) AS field,\n pg_table_size(i.indexname::text) as index_size,\n pg_size_pretty(pg_table_size(i.indexname::text)) as index_size_pretty,\n n_distinct,\n most_common_vals::text::text[],\n most_common_freqs::text::text[],\n histogram_bounds::text::text[],\n correlation\nFROM\n pg_indexes i\n LEFT JOIN pg_stats s ON (s.tablename = i.indexname)\nWHERE i.schemaname='pgstac' and i.tablename ~ '_items_';\n\nCREATE OR REPLACE FUNCTION queryable_indexes(\n IN treeroot text DEFAULT 'items',\n IN changes boolean DEFAULT FALSE,\n OUT collection text,\n OUT partition text,\n OUT field text,\n OUT indexname text,\n OUT existing_idx text,\n OUT queryable_idx text\n) RETURNS SETOF RECORD AS $$\nWITH p AS (\n SELECT\n relid::text as partition,\n replace(replace(\n CASE\n WHEN parentrelid::regclass::text='items' THEN pg_get_expr(c.relpartbound, c.oid)\n ELSE pg_get_expr(parent.relpartbound, parent.oid)\n END,\n 'FOR VALUES IN (''',''), ''')',\n ''\n ) AS collection\n FROM pg_partition_tree(treeroot)\n JOIN pg_class c ON (relid::regclass = c.oid)\n JOIN pg_class parent ON (parentrelid::regclass = parent.oid AND isleaf)\n ), i AS (\n SELECT\n partition,\n indexname,\n regexp_replace(btrim(replace(replace(indexdef, indexname, ''),'pgstac.',''),' \\t\\n'), '[ ]+', ' ', 'g') as iidx,\n COALESCE(\n (regexp_match(indexdef, '\\(([a-zA-Z]+)\\)'))[1],\n (regexp_match(indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_-]+)''::text'))[1],\n CASE WHEN indexdef ~* '\\(datetime desc, end_datetime\\)' THEN 'datetime' ELSE NULL END\n ) AS field\n FROM\n pg_indexes\n JOIN p ON (tablename=partition)\n ), q AS (\n SELECT\n name AS field,\n collection,\n partition,\n format(indexdef(queryables), partition) as qidx\n FROM queryables, unnest_collection(queryables.collection_ids) collection\n JOIN p USING (collection)\n WHERE property_index_type IS NOT NULL OR name IN ('datetime','geometry','id')\n )\n SELECT\n collection,\n partition,\n field,\n indexname,\n iidx as existing_idx,\n qidx as queryable_idx\n FROM i FULL JOIN q USING (field, partition)\n WHERE CASE WHEN changes THEN lower(iidx) IS DISTINCT FROM lower(qidx) ELSE TRUE END;\n;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION maintain_index(\n indexname text,\n queryable_idx text,\n dropindexes boolean DEFAULT FALSE,\n rebuildindexes boolean DEFAULT FALSE,\n idxconcurrently boolean DEFAULT FALSE\n) RETURNS VOID AS $$\nDECLARE\nBEGIN\n IF indexname IS NOT NULL THEN\n IF dropindexes OR queryable_idx IS NOT NULL THEN\n EXECUTE format('DROP INDEX IF EXISTS %I;', indexname);\n ELSIF rebuildindexes THEN\n IF idxconcurrently THEN\n EXECUTE format('REINDEX INDEX CONCURRENTLY %I;', indexname);\n ELSE\n EXECUTE format('REINDEX INDEX CONCURRENTLY %I;', indexname);\n END IF;\n END IF;\n END IF;\n IF queryable_idx IS NOT NULL THEN\n IF idxconcurrently THEN\n EXECUTE replace(queryable_idx, 'INDEX', 'INDEX CONCURRENTLY');\n ELSE EXECUTE queryable_idx;\n END IF;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\n\nset check_function_bodies to off;\nCREATE OR REPLACE FUNCTION maintain_partition_queries(\n part text DEFAULT 'items',\n dropindexes boolean DEFAULT FALSE,\n rebuildindexes boolean DEFAULT FALSE,\n idxconcurrently boolean DEFAULT FALSE\n) RETURNS SETOF text AS $$\nDECLARE\n rec record;\n q text;\nBEGIN\n FOR rec IN (\n SELECT * FROM queryable_indexes(part,true)\n ) LOOP\n q := format(\n 'SELECT maintain_index(\n %L,%L,%L,%L,%L\n );',\n rec.indexname,\n rec.queryable_idx,\n dropindexes,\n rebuildindexes,\n idxconcurrently\n );\n RAISE NOTICE 'Q: %', q;\n RETURN NEXT q;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION maintain_partitions(\n part text DEFAULT 'items',\n dropindexes boolean DEFAULT FALSE,\n rebuildindexes boolean DEFAULT FALSE\n) RETURNS VOID AS $$\n WITH t AS (\n SELECT run_or_queue(q) FROM maintain_partition_queries(part, dropindexes, rebuildindexes) q\n ) SELECT count(*) FROM t;\n$$ LANGUAGE SQL;\n\n\nCREATE OR REPLACE FUNCTION queryables_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\nBEGIN\n PERFORM maintain_partitions();\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER queryables_trigger AFTER INSERT OR UPDATE ON queryables\nFOR EACH STATEMENT EXECUTE PROCEDURE queryables_trigger_func();\n\n\nCREATE OR REPLACE FUNCTION get_queryables(_collection_ids text[] DEFAULT NULL) RETURNS jsonb AS $$\nDECLARE\nBEGIN\n -- Build up queryables if the input contains valid collection ids or is empty\n IF EXISTS (\n SELECT 1 FROM collections\n WHERE\n _collection_ids IS NULL\n OR cardinality(_collection_ids) = 0\n OR id = ANY(_collection_ids)\n )\n THEN\n RETURN (\n WITH base AS (\n SELECT\n unnest(collection_ids) as collection_id,\n name,\n coalesce(definition, '{\"type\":\"string\"}'::jsonb) as definition\n FROM queryables\n WHERE\n _collection_ids IS NULL OR\n _collection_ids = '{}'::text[] OR\n _collection_ids && collection_ids\n UNION ALL\n SELECT null, name, coalesce(definition, '{\"type\":\"string\"}'::jsonb) as definition\n FROM queryables WHERE collection_ids IS NULL OR collection_ids = '{}'::text[]\n ), g AS (\n SELECT\n name,\n first_notnull(definition) as definition,\n jsonb_array_unique_merge(definition->'enum') as enum,\n jsonb_min(definition->'minimum') as minimum,\n jsonb_min(definition->'maxiumn') as maximum\n FROM base\n GROUP BY 1\n )\n SELECT\n jsonb_build_object(\n '$schema', 'http://json-schema.org/draft-07/schema#',\n '$id', '',\n 'type', 'object',\n 'title', 'STAC Queryables.',\n 'properties', jsonb_object_agg(\n name,\n definition\n ||\n jsonb_strip_nulls(jsonb_build_object(\n 'enum', enum,\n 'minimum', minimum,\n 'maximum', maximum\n ))\n ),\n 'additionalProperties', pgstac.additional_properties()\n )\n FROM g\n );\n ELSE\n RETURN NULL;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\nCREATE OR REPLACE FUNCTION get_queryables(_collection text DEFAULT NULL) RETURNS jsonb AS $$\n SELECT\n CASE\n WHEN _collection IS NULL THEN get_queryables(NULL::text[])\n ELSE get_queryables(ARRAY[_collection])\n END\n ;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_queryables() RETURNS jsonb AS $$\n SELECT get_queryables(NULL::text[]);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION schema_qualify_refs(url text, j jsonb) returns jsonb as $$\n SELECT regexp_replace(j::text, '\"\\$ref\": \"#', concat('\"$ref\": \"', url, '#'), 'g')::jsonb;\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE VIEW stac_extension_queryables AS\nSELECT DISTINCT key as name, schema_qualify_refs(e.url, j.value) as definition FROM stac_extensions e, jsonb_each(e.content->'definitions'->'fields'->'properties') j;\n\n\nCREATE OR REPLACE FUNCTION missing_queryables(_collection text, _tablesample float DEFAULT 5, minrows float DEFAULT 10) RETURNS TABLE(collection text, name text, definition jsonb, property_wrapper text) AS $$\nDECLARE\n q text;\n _partition text;\n explain_json json;\n psize float;\n estrows float;\nBEGIN\n SELECT format('_items_%s', key) INTO _partition FROM collections WHERE id=_collection;\n\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM %I;', _partition)\n INTO explain_json;\n psize := explain_json->0->'Plan'->'Plan Rows';\n estrows := _tablesample * .01 * psize;\n IF estrows < minrows THEN\n _tablesample := least(100,greatest(_tablesample, (estrows / psize) / 100));\n RAISE NOTICE '%', (psize / estrows) / 100;\n END IF;\n RAISE NOTICE 'Using tablesample % to find missing queryables from % % that has ~% rows estrows: %', _tablesample, _collection, _partition, psize, estrows;\n\n q := format(\n $q$\n WITH q AS (\n SELECT * FROM queryables\n WHERE\n collection_ids IS NULL\n OR %L = ANY(collection_ids)\n ), t AS (\n SELECT\n content->'properties' AS properties\n FROM\n %I\n TABLESAMPLE SYSTEM(%L)\n ), p AS (\n SELECT DISTINCT ON (key)\n key,\n value,\n s.definition\n FROM t\n JOIN LATERAL jsonb_each(properties) ON TRUE\n LEFT JOIN q ON (q.name=key)\n LEFT JOIN stac_extension_queryables s ON (s.name=key)\n WHERE q.definition IS NULL\n )\n SELECT\n %L,\n key,\n COALESCE(definition, jsonb_build_object('type',jsonb_typeof(value))) as definition,\n CASE\n WHEN definition->>'type' = 'integer' THEN 'to_int'\n WHEN COALESCE(definition->>'type', jsonb_typeof(value)) = 'number' THEN 'to_float'\n WHEN COALESCE(definition->>'type', jsonb_typeof(value)) = 'array' THEN 'to_text_array'\n ELSE 'to_text'\n END\n FROM p;\n $q$,\n _collection,\n _partition,\n _tablesample,\n _collection\n );\n RETURN QUERY EXECUTE q;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION missing_queryables(_tablesample float DEFAULT 5) RETURNS TABLE(collection_ids text[], name text, definition jsonb, property_wrapper text) AS $$\n SELECT\n array_agg(collection),\n name,\n definition,\n property_wrapper\n FROM\n collections\n JOIN LATERAL\n missing_queryables(id, _tablesample) c\n ON TRUE\n GROUP BY\n 2,3,4\n ORDER BY 2,1\n ;\n$$ LANGUAGE SQL;\nCREATE OR REPLACE FUNCTION parse_dtrange(\n _indate jsonb,\n relative_base timestamptz DEFAULT date_trunc('hour', CURRENT_TIMESTAMP)\n) RETURNS tstzrange AS $$\nDECLARE\n timestrs text[];\n s timestamptz;\n e timestamptz;\nBEGIN\n timestrs :=\n CASE\n WHEN _indate ? 'timestamp' THEN\n ARRAY[_indate->>'timestamp']\n WHEN _indate ? 'interval' THEN\n to_text_array(_indate->'interval')\n WHEN jsonb_typeof(_indate) = 'array' THEN\n to_text_array(_indate)\n ELSE\n regexp_split_to_array(\n _indate->>0,\n '/'\n )\n END;\n RAISE NOTICE 'TIMESTRS %', timestrs;\n IF cardinality(timestrs) = 1 THEN\n IF timestrs[1] ILIKE 'P%' THEN\n RETURN tstzrange(relative_base - upper(timestrs[1])::interval, relative_base, '[)');\n END IF;\n s := timestrs[1]::timestamptz;\n RETURN tstzrange(s, s, '[]');\n END IF;\n\n IF cardinality(timestrs) != 2 THEN\n RAISE EXCEPTION 'Timestamp cannot have more than 2 values';\n END IF;\n\n IF timestrs[1] = '..' OR timestrs[1] = '' THEN\n s := '-infinity'::timestamptz;\n e := timestrs[2]::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] = '..' OR timestrs[2] = '' THEN\n s := timestrs[1]::timestamptz;\n e := 'infinity'::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[1] ILIKE 'P%' AND timestrs[2] NOT ILIKE 'P%' THEN\n e := timestrs[2]::timestamptz;\n s := e - upper(timestrs[1])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] ILIKE 'P%' AND timestrs[1] NOT ILIKE 'P%' THEN\n s := timestrs[1]::timestamptz;\n e := s + upper(timestrs[2])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n s := timestrs[1]::timestamptz;\n e := timestrs[2]::timestamptz;\n\n RETURN tstzrange(s,e,'[)');\n\n RETURN NULL;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT PARALLEL SAFE SET TIME ZONE 'UTC';\n\nCREATE OR REPLACE FUNCTION parse_dtrange(\n _indate text,\n relative_base timestamptz DEFAULT CURRENT_TIMESTAMP\n) RETURNS tstzrange AS $$\n SELECT parse_dtrange(to_jsonb(_indate), relative_base);\n$$ LANGUAGE SQL STABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION temporal_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\n ll text := 'datetime';\n lh text := 'end_datetime';\n rrange tstzrange;\n rl text;\n rh text;\n outq text;\nBEGIN\n rrange := parse_dtrange(args->1);\n RAISE NOTICE 'Constructing temporal query OP: %, ARGS: %, RRANGE: %', op, args, rrange;\n op := lower(op);\n rl := format('%L::timestamptz', lower(rrange));\n rh := format('%L::timestamptz', upper(rrange));\n outq := CASE op\n WHEN 't_before' THEN 'lh < rl'\n WHEN 't_after' THEN 'll > rh'\n WHEN 't_meets' THEN 'lh = rl'\n WHEN 't_metby' THEN 'll = rh'\n WHEN 't_overlaps' THEN 'll < rl AND rl < lh < rh'\n WHEN 't_overlappedby' THEN 'rl < ll < rh AND lh > rh'\n WHEN 't_starts' THEN 'll = rl AND lh < rh'\n WHEN 't_startedby' THEN 'll = rl AND lh > rh'\n WHEN 't_during' THEN 'll > rl AND lh < rh'\n WHEN 't_contains' THEN 'll < rl AND lh > rh'\n WHEN 't_finishes' THEN 'll > rl AND lh = rh'\n WHEN 't_finishedby' THEN 'll < rl AND lh = rh'\n WHEN 't_equals' THEN 'll = rl AND lh = rh'\n WHEN 't_disjoint' THEN 'NOT (ll <= rh AND lh >= rl)'\n WHEN 't_intersects' THEN 'll <= rh AND lh >= rl'\n WHEN 'anyinteracts' THEN 'll <= rh AND lh >= rl'\n END;\n outq := regexp_replace(outq, '\\mll\\M', ll);\n outq := regexp_replace(outq, '\\mlh\\M', lh);\n outq := regexp_replace(outq, '\\mrl\\M', rl);\n outq := regexp_replace(outq, '\\mrh\\M', rh);\n outq := format('(%s)', outq);\n RETURN outq;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\n\n\nCREATE OR REPLACE FUNCTION spatial_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\n geom text;\n j jsonb := args->1;\nBEGIN\n op := lower(op);\n RAISE NOTICE 'Constructing spatial query OP: %, ARGS: %', op, args;\n IF op NOT IN ('s_equals','s_disjoint','s_touches','s_within','s_overlaps','s_crosses','s_intersects','intersects','s_contains') THEN\n RAISE EXCEPTION 'Spatial Operator % Not Supported', op;\n END IF;\n op := regexp_replace(op, '^s_', 'st_');\n IF op = 'intersects' THEN\n op := 'st_intersects';\n END IF;\n -- Convert geometry to WKB string\n IF j ? 'type' AND j ? 'coordinates' THEN\n geom := st_geomfromgeojson(j)::text;\n ELSIF jsonb_typeof(j) = 'array' THEN\n geom := bbox_geom(j)::text;\n END IF;\n\n RETURN format('%s(geometry, %L::geometry)', op, geom);\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION query_to_cql2(q jsonb) RETURNS jsonb AS $$\n-- Translates anything passed in through the deprecated \"query\" into equivalent CQL2\nWITH t AS (\n SELECT key as property, value as ops\n FROM jsonb_each(q)\n), t2 AS (\n SELECT property, (jsonb_each(ops)).*\n FROM t WHERE jsonb_typeof(ops) = 'object'\n UNION ALL\n SELECT property, 'eq', ops\n FROM t WHERE jsonb_typeof(ops) != 'object'\n)\nSELECT\n jsonb_strip_nulls(jsonb_build_object(\n 'op', 'and',\n 'args', jsonb_agg(\n jsonb_build_object(\n 'op', key,\n 'args', jsonb_build_array(\n jsonb_build_object('property',property),\n value\n )\n )\n )\n )\n) as qcql FROM t2\n;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\nCREATE OR REPLACE FUNCTION cql1_to_cql2(j jsonb) RETURNS jsonb AS $$\nDECLARE\n args jsonb;\n ret jsonb;\nBEGIN\n RAISE NOTICE 'CQL1_TO_CQL2: %', j;\n IF j ? 'filter' THEN\n RETURN cql1_to_cql2(j->'filter');\n END IF;\n IF j ? 'property' THEN\n RETURN j;\n END IF;\n IF jsonb_typeof(j) = 'array' THEN\n SELECT jsonb_agg(cql1_to_cql2(el)) INTO args FROM jsonb_array_elements(j) el;\n RETURN args;\n END IF;\n IF jsonb_typeof(j) = 'number' THEN\n RETURN j;\n END IF;\n IF jsonb_typeof(j) = 'string' THEN\n RETURN j;\n END IF;\n\n IF jsonb_typeof(j) = 'object' THEN\n SELECT jsonb_build_object(\n 'op', key,\n 'args', cql1_to_cql2(value)\n ) INTO ret\n FROM jsonb_each(j)\n WHERE j IS NOT NULL;\n RETURN ret;\n END IF;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE STRICT;\n\nCREATE TABLE cql2_ops (\n op text PRIMARY KEY,\n template text,\n types text[]\n);\nINSERT INTO cql2_ops (op, template, types) VALUES\n ('eq', '%s = %s', NULL),\n ('neq', '%s != %s', NULL),\n ('ne', '%s != %s', NULL),\n ('!=', '%s != %s', NULL),\n ('<>', '%s != %s', NULL),\n ('lt', '%s < %s', NULL),\n ('lte', '%s <= %s', NULL),\n ('gt', '%s > %s', NULL),\n ('gte', '%s >= %s', NULL),\n ('le', '%s <= %s', NULL),\n ('ge', '%s >= %s', NULL),\n ('=', '%s = %s', NULL),\n ('<', '%s < %s', NULL),\n ('<=', '%s <= %s', NULL),\n ('>', '%s > %s', NULL),\n ('>=', '%s >= %s', NULL),\n ('like', '%s LIKE %s', NULL),\n ('ilike', '%s ILIKE %s', NULL),\n ('+', '%s + %s', NULL),\n ('-', '%s - %s', NULL),\n ('*', '%s * %s', NULL),\n ('/', '%s / %s', NULL),\n ('not', 'NOT (%s)', NULL),\n ('between', '%s BETWEEN %s AND %s', NULL),\n ('isnull', '%s IS NULL', NULL),\n ('upper', 'upper(%s)', NULL),\n ('lower', 'lower(%s)', NULL),\n ('casei', 'upper(%s)', NULL),\n ('accenti', 'unaccent(%s)', NULL)\nON CONFLICT (op) DO UPDATE\n SET\n template = EXCLUDED.template\n;\n\n\nCREATE OR REPLACE FUNCTION cql2_query(j jsonb, wrapper text DEFAULT NULL) RETURNS text AS $$\n#variable_conflict use_variable\nDECLARE\n args jsonb := j->'args';\n arg jsonb;\n op text := lower(j->>'op');\n cql2op RECORD;\n literal text;\n _wrapper text;\n leftarg text;\n rightarg text;\n prop text;\n extra_props bool := pgstac.additional_properties();\nBEGIN\n IF j IS NULL OR (op IS NOT NULL AND args IS NULL) THEN\n RETURN NULL;\n END IF;\n RAISE NOTICE 'CQL2_QUERY: %', j;\n\n -- check if all properties are represented in the queryables\n IF NOT extra_props THEN\n FOR prop IN\n SELECT DISTINCT p->>0\n FROM jsonb_path_query(j, 'strict $.**.property') p\n WHERE p->>0 NOT IN ('id', 'datetime', 'geometry', 'end_datetime', 'collection')\n LOOP\n IF (queryable(prop)).nulled_wrapper IS NULL THEN\n RAISE EXCEPTION 'Term % is not found in queryables.', prop;\n END IF;\n END LOOP;\n END IF;\n\n IF j ? 'filter' THEN\n RETURN cql2_query(j->'filter');\n END IF;\n\n IF j ? 'upper' THEN\n RETURN cql2_query(jsonb_build_object('op', 'upper', 'args', j->'upper'));\n END IF;\n\n IF j ? 'lower' THEN\n RETURN cql2_query(jsonb_build_object('op', 'lower', 'args', j->'lower'));\n END IF;\n\n -- Temporal Query\n IF op ilike 't_%' or op = 'anyinteracts' THEN\n RETURN temporal_op_query(op, args);\n END IF;\n\n -- If property is a timestamp convert it to text to use with\n -- general operators\n IF j ? 'timestamp' THEN\n RETURN format('%L::timestamptz', to_tstz(j->'timestamp'));\n END IF;\n IF j ? 'interval' THEN\n RAISE EXCEPTION 'Please use temporal operators when using intervals.';\n RETURN NONE;\n END IF;\n\n -- Spatial Query\n IF op ilike 's_%' or op = 'intersects' THEN\n RETURN spatial_op_query(op, args);\n END IF;\n\n IF op IN ('a_equals','a_contains','a_contained_by','a_overlaps') THEN\n IF args->0 ? 'property' THEN\n leftarg := format('to_text_array(%s)', (queryable(args->0->>'property')).path);\n END IF;\n IF args->1 ? 'property' THEN\n rightarg := format('to_text_array(%s)', (queryable(args->1->>'property')).path);\n END IF;\n RETURN FORMAT(\n '%s %s %s',\n COALESCE(leftarg, quote_literal(to_text_array(args->0))),\n CASE op\n WHEN 'a_equals' THEN '='\n WHEN 'a_contains' THEN '@>'\n WHEN 'a_contained_by' THEN '<@'\n WHEN 'a_overlaps' THEN '&&'\n END,\n COALESCE(rightarg, quote_literal(to_text_array(args->1)))\n );\n END IF;\n\n IF op = 'in' THEN\n RAISE NOTICE 'IN : % % %', args, jsonb_build_array(args->0), args->1;\n args := jsonb_build_array(args->0) || (args->1);\n RAISE NOTICE 'IN2 : %', args;\n END IF;\n\n\n\n IF op = 'between' THEN\n args = jsonb_build_array(\n args->0,\n args->1,\n args->2\n );\n END IF;\n\n -- Make sure that args is an array and run cql2_query on\n -- each element of the array\n RAISE NOTICE 'ARGS PRE: %', args;\n IF j ? 'args' THEN\n IF jsonb_typeof(args) != 'array' THEN\n args := jsonb_build_array(args);\n END IF;\n\n IF jsonb_path_exists(args, '$[*] ? (@.property == \"id\" || @.property == \"datetime\" || @.property == \"end_datetime\" || @.property == \"collection\")') THEN\n wrapper := NULL;\n ELSE\n -- if any of the arguments are a property, try to get the property_wrapper\n FOR arg IN SELECT jsonb_path_query(args, '$[*] ? (@.property != null)') LOOP\n RAISE NOTICE 'Arg: %', arg;\n wrapper := (queryable(arg->>'property')).nulled_wrapper;\n RAISE NOTICE 'Property: %, Wrapper: %', arg, wrapper;\n IF wrapper IS NOT NULL THEN\n EXIT;\n END IF;\n END LOOP;\n\n -- if the property was not in queryables, see if any args were numbers\n IF\n wrapper IS NULL\n AND jsonb_path_exists(args, '$[*] ? (@.type()==\"number\")')\n THEN\n wrapper := 'to_float';\n END IF;\n wrapper := coalesce(wrapper, 'to_text');\n END IF;\n\n SELECT jsonb_agg(cql2_query(a, wrapper))\n INTO args\n FROM jsonb_array_elements(args) a;\n END IF;\n RAISE NOTICE 'ARGS: %', args;\n\n IF op IN ('and', 'or') THEN\n RETURN\n format(\n '(%s)',\n array_to_string(to_text_array(args), format(' %s ', upper(op)))\n );\n END IF;\n\n IF op = 'in' THEN\n RAISE NOTICE 'IN -- % %', args->0, to_text(args->0);\n RETURN format(\n '%s IN (%s)',\n to_text(args->0),\n array_to_string((to_text_array(args))[2:], ',')\n );\n END IF;\n\n -- Look up template from cql2_ops\n IF j ? 'op' THEN\n SELECT * INTO cql2op FROM cql2_ops WHERE cql2_ops.op ilike op;\n IF FOUND THEN\n -- If specific index set in queryables for a property cast other arguments to that type\n\n RETURN format(\n cql2op.template,\n VARIADIC (to_text_array(args))\n );\n ELSE\n RAISE EXCEPTION 'Operator % Not Supported.', op;\n END IF;\n END IF;\n\n\n IF wrapper IS NOT NULL THEN\n RAISE NOTICE 'Wrapping % with %', j, wrapper;\n IF j ? 'property' THEN\n RETURN format('%I(%s)', wrapper, (queryable(j->>'property')).path);\n ELSE\n RETURN format('%I(%L)', wrapper, j);\n END IF;\n ELSIF j ? 'property' THEN\n RETURN quote_ident(j->>'property');\n END IF;\n\n RETURN quote_literal(to_text(j));\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION paging_dtrange(\n j jsonb\n) RETURNS tstzrange AS $$\nDECLARE\n op text;\n filter jsonb := j->'filter';\n dtrange tstzrange := tstzrange('-infinity'::timestamptz,'infinity'::timestamptz);\n sdate timestamptz := '-infinity'::timestamptz;\n edate timestamptz := 'infinity'::timestamptz;\n jpitem jsonb;\nBEGIN\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n END IF;\n IF NOT (filter @? '$.**.op ? (@ == \"or\" || @ == \"not\")') THEN\n FOR jpitem IN SELECT j FROM jsonb_path_query(filter,'strict $.** ? (@.args[*].property == \"datetime\")'::jsonpath) j LOOP\n op := lower(jpitem->>'op');\n dtrange := parse_dtrange(jpitem->'args'->1);\n IF op IN ('<=', 'lt', 'lte', '<', 'le', 't_before') THEN\n sdate := greatest(sdate,'-infinity');\n edate := least(edate, upper(dtrange));\n ELSIF op IN ('>=', '>', 'gt', 'gte', 'ge', 't_after') THEN\n edate := least(edate, 'infinity');\n sdate := greatest(sdate, lower(dtrange));\n ELSIF op IN ('=', 'eq') THEN\n edate := least(edate, upper(dtrange));\n sdate := greatest(sdate, lower(dtrange));\n END IF;\n RAISE NOTICE '2 OP: %, ARGS: %, DTRANGE: %, SDATE: %, EDATE: %', op, jpitem->'args'->1, dtrange, sdate, edate;\n END LOOP;\n END IF;\n IF sdate > edate THEN\n RETURN 'empty'::tstzrange;\n END IF;\n RETURN tstzrange(sdate,edate, '[]');\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT SET TIME ZONE 'UTC';\n\nCREATE OR REPLACE FUNCTION paging_collections(\n IN j jsonb\n) RETURNS text[] AS $$\nDECLARE\n filter jsonb := j->'filter';\n jpitem jsonb;\n op text;\n args jsonb;\n arg jsonb;\n collections text[];\nBEGIN\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n END IF;\n IF NOT (filter @? '$.**.op ? (@ == \"or\" || @ == \"not\")') THEN\n FOR jpitem IN SELECT j FROM jsonb_path_query(filter,'strict $.** ? (@.args[*].property == \"collection\")'::jsonpath) j LOOP\n RAISE NOTICE 'JPITEM: %', jpitem;\n op := jpitem->>'op';\n args := jpitem->'args';\n IF op IN ('=', 'eq', 'in') THEN\n FOR arg IN SELECT a FROM jsonb_array_elements(args) a LOOP\n IF jsonb_typeof(arg) IN ('string', 'array') THEN\n RAISE NOTICE 'arg: %, collections: %', arg, collections;\n IF collections IS NULL OR collections = '{}'::text[] THEN\n collections := to_text_array(arg);\n ELSE\n collections := array_intersection(collections, to_text_array(arg));\n END IF;\n END IF;\n END LOOP;\n END IF;\n END LOOP;\n END IF;\n IF collections = '{}'::text[] THEN\n RETURN NULL;\n END IF;\n RETURN collections;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\nCREATE TABLE items (\n id text NOT NULL,\n geometry geometry NOT NULL,\n collection text NOT NULL,\n datetime timestamptz NOT NULL,\n end_datetime timestamptz NOT NULL,\n content JSONB NOT NULL,\n private jsonb\n)\nPARTITION BY LIST (collection)\n;\n\nCREATE INDEX \"datetime_idx\" ON items USING BTREE (datetime DESC, end_datetime ASC);\nCREATE INDEX \"geometry_idx\" ON items USING GIST (geometry);\n\nCREATE STATISTICS datetime_stats (dependencies) on datetime, end_datetime from items;\n\nALTER TABLE items ADD CONSTRAINT items_collections_fk FOREIGN KEY (collection) REFERENCES collections(id) ON DELETE CASCADE DEFERRABLE;\n\nCREATE OR REPLACE FUNCTION partition_after_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p text;\n t timestamptz := clock_timestamp();\nBEGIN\n RAISE NOTICE 'Updating partition stats %', t;\n FOR p IN SELECT DISTINCT partition\n FROM newdata n JOIN partition_sys_meta p\n ON (n.collection=p.collection AND n.datetime <@ p.partition_dtrange)\n LOOP\n PERFORM run_or_queue(format('SELECT update_partition_stats(%L, %L);', p, true));\n END LOOP;\n IF TG_OP IN ('DELETE','UPDATE') THEN\n DELETE FROM format_item_cache c USING newdata n WHERE c.collection = n.collection AND c.id = n.id;\n END IF;\n RAISE NOTICE 't: % %', t, clock_timestamp() - t;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE TRIGGER items_after_insert_trigger\nAFTER INSERT ON items\nREFERENCING NEW TABLE AS newdata\nFOR EACH STATEMENT\nEXECUTE FUNCTION partition_after_triggerfunc();\n\nCREATE TRIGGER items_after_update_trigger\nAFTER DELETE ON items\nREFERENCING OLD TABLE AS newdata\nFOR EACH STATEMENT\nEXECUTE FUNCTION partition_after_triggerfunc();\n\nCREATE TRIGGER items_after_delete_trigger\nAFTER UPDATE ON items\nREFERENCING NEW TABLE AS newdata\nFOR EACH STATEMENT\nEXECUTE FUNCTION partition_after_triggerfunc();\n\n\nCREATE OR REPLACE FUNCTION content_slim(_item jsonb) RETURNS jsonb AS $$\n SELECT strip_jsonb(_item - '{id,geometry,collection,type}'::text[], collection_base_item(_item->>'collection')) - '{id,geometry,collection,type}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_dehydrate(content jsonb) RETURNS items AS $$\n SELECT\n content->>'id' as id,\n stac_geom(content) as geometry,\n content->>'collection' as collection,\n stac_datetime(content) as datetime,\n stac_end_datetime(content) as end_datetime,\n content_slim(content) as content,\n null::jsonb as private\n ;\n$$ LANGUAGE SQL STABLE;\n\nCREATE OR REPLACE FUNCTION include_field(f text, fields jsonb DEFAULT '{}'::jsonb) RETURNS boolean AS $$\nDECLARE\n includes jsonb := fields->'include';\n excludes jsonb := fields->'exclude';\nBEGIN\n IF f IS NULL THEN\n RETURN NULL;\n END IF;\n\n\n IF\n jsonb_typeof(excludes) = 'array'\n AND jsonb_array_length(excludes)>0\n AND excludes ? f\n THEN\n RETURN FALSE;\n END IF;\n\n IF\n (\n jsonb_typeof(includes) = 'array'\n AND jsonb_array_length(includes) > 0\n AND includes ? f\n ) OR\n (\n includes IS NULL\n OR jsonb_typeof(includes) = 'null'\n OR jsonb_array_length(includes) = 0\n )\n THEN\n RETURN TRUE;\n END IF;\n\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nDROP FUNCTION IF EXISTS content_hydrate(jsonb, jsonb, jsonb);\nCREATE OR REPLACE FUNCTION content_hydrate(\n _item jsonb,\n _base_item jsonb,\n fields jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\n SELECT merge_jsonb(\n jsonb_fields(_item, fields),\n jsonb_fields(_base_item, fields)\n );\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION content_hydrate(_item items, _collection collections, fields jsonb DEFAULT '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\n content jsonb;\n base_item jsonb := _collection.base_item;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry, 20)::jsonb;\n END IF;\n output := content_hydrate(\n jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'collection', _item.collection,\n 'type', 'Feature'\n ) || _item.content,\n _collection.base_item,\n fields\n );\n\n RETURN output;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_nonhydrated(\n _item items,\n fields jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry, 20)::jsonb;\n END IF;\n output := jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'collection', _item.collection,\n 'type', 'Feature'\n ) || _item.content;\n RETURN output;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_hydrate(_item items, fields jsonb DEFAULT '{}'::jsonb) RETURNS jsonb AS $$\n SELECT content_hydrate(\n _item,\n (SELECT c FROM collections c WHERE id=_item.collection LIMIT 1),\n fields\n );\n$$ LANGUAGE SQL STABLE;\n\n\nCREATE UNLOGGED TABLE items_staging (\n content JSONB NOT NULL\n);\nCREATE UNLOGGED TABLE items_staging_ignore (\n content JSONB NOT NULL\n);\nCREATE UNLOGGED TABLE items_staging_upsert (\n content JSONB NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION items_staging_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p record;\n _partitions text[];\n part text;\n ts timestamptz := clock_timestamp();\n nrows int;\nBEGIN\n RAISE NOTICE 'Creating Partitions. %', clock_timestamp() - ts;\n\n FOR part IN WITH t AS (\n SELECT\n n.content->>'collection' as collection,\n stac_daterange(n.content->'properties') as dtr,\n partition_trunc\n FROM newdata n JOIN collections ON (n.content->>'collection'=collections.id)\n ), p AS (\n SELECT\n collection,\n COALESCE(date_trunc(partition_trunc::text, lower(dtr)),'-infinity') as d,\n tstzrange(min(lower(dtr)),max(lower(dtr)),'[]') as dtrange,\n tstzrange(min(upper(dtr)),max(upper(dtr)),'[]') as edtrange\n FROM t\n GROUP BY 1,2\n ) SELECT check_partition(collection, dtrange, edtrange) FROM p LOOP\n RAISE NOTICE 'Partition %', part;\n END LOOP;\n\n RAISE NOTICE 'Creating temp table with data to be added. %', clock_timestamp() - ts;\n DROP TABLE IF EXISTS tmpdata;\n CREATE TEMP TABLE tmpdata ON COMMIT DROP AS\n SELECT\n (content_dehydrate(content)).*\n FROM newdata;\n GET DIAGNOSTICS nrows = ROW_COUNT;\n RAISE NOTICE 'Added % rows to tmpdata. %', nrows, clock_timestamp() - ts;\n\n RAISE NOTICE 'Doing the insert. %', clock_timestamp() - ts;\n IF TG_TABLE_NAME = 'items_staging' THEN\n INSERT INTO items\n SELECT * FROM tmpdata;\n GET DIAGNOSTICS nrows = ROW_COUNT;\n RAISE NOTICE 'Inserted % rows to items. %', nrows, clock_timestamp() - ts;\n ELSIF TG_TABLE_NAME = 'items_staging_ignore' THEN\n INSERT INTO items\n SELECT * FROM tmpdata\n ON CONFLICT DO NOTHING;\n GET DIAGNOSTICS nrows = ROW_COUNT;\n RAISE NOTICE 'Inserted % rows to items. %', nrows, clock_timestamp() - ts;\n ELSIF TG_TABLE_NAME = 'items_staging_upsert' THEN\n DELETE FROM items i USING tmpdata s\n WHERE\n i.id = s.id\n AND i.collection = s.collection\n AND i IS DISTINCT FROM s\n ;\n GET DIAGNOSTICS nrows = ROW_COUNT;\n RAISE NOTICE 'Deleted % rows from items. %', nrows, clock_timestamp() - ts;\n INSERT INTO items AS t\n SELECT * FROM tmpdata\n ON CONFLICT DO NOTHING;\n GET DIAGNOSTICS nrows = ROW_COUNT;\n RAISE NOTICE 'Inserted % rows to items. %', nrows, clock_timestamp() - ts;\n END IF;\n\n RAISE NOTICE 'Deleting data from staging table. %', clock_timestamp() - ts;\n DELETE FROM items_staging;\n RAISE NOTICE 'Done. %', clock_timestamp() - ts;\n\n RETURN NULL;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER items_staging_insert_trigger AFTER INSERT ON items_staging REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\nCREATE TRIGGER items_staging_insert_ignore_trigger AFTER INSERT ON items_staging_ignore REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\nCREATE TRIGGER items_staging_insert_upsert_trigger AFTER INSERT ON items_staging_upsert REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\n\nCREATE OR REPLACE FUNCTION item_by_id(_id text, _collection text DEFAULT NULL) RETURNS items AS\n$$\nDECLARE\n i items%ROWTYPE;\nBEGIN\n SELECT * INTO i FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection) LIMIT 1;\n RETURN i;\nEND;\n$$ LANGUAGE PLPGSQL STABLE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION get_item(_id text, _collection text DEFAULT NULL) RETURNS jsonb AS $$\n SELECT content_hydrate(items) FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection);\n$$ LANGUAGE SQL STABLE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION delete_item(_id text, _collection text DEFAULT NULL) RETURNS VOID AS $$\nDECLARE\nout items%ROWTYPE;\nBEGIN\n DELETE FROM items WHERE id = _id AND (_collection IS NULL OR collection=_collection) RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n--/*\nCREATE OR REPLACE FUNCTION create_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION update_item(content jsonb) RETURNS VOID AS $$\nDECLARE\n old items %ROWTYPE;\n out items%ROWTYPE;\nBEGIN\n PERFORM delete_item(content->>'id', content->>'collection');\n PERFORM create_item(content);\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION create_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION collection_bbox(id text) RETURNS jsonb AS $$\n SELECT (replace(replace(replace(st_extent(geometry)::text,'BOX(','[['),')',']]'),' ',','))::jsonb\n FROM items WHERE collection=$1;\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION collection_temporal_extent(id text) RETURNS jsonb AS $$\n SELECT to_jsonb(array[array[min(datetime), max(datetime)]])\n FROM items WHERE collection=$1;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection_extents() RETURNS VOID AS $$\nUPDATE collections\n SET content = jsonb_set_lax(\n content,\n '{extent}'::text[],\n collection_extent(id, FALSE),\n true,\n 'use_json_null'\n )\n;\n$$ LANGUAGE SQL;\nCREATE TABLE partition_stats (\n partition text PRIMARY KEY,\n dtrange tstzrange,\n edtrange tstzrange,\n spatial geometry,\n last_updated timestamptz,\n keys text[]\n) WITH (FILLFACTOR=90);\n\nCREATE INDEX partitions_range_idx ON partition_stats USING GIST(dtrange);\n\n\nCREATE OR REPLACE FUNCTION constraint_tstzrange(expr text) RETURNS tstzrange AS $$\n WITH t AS (\n SELECT regexp_matches(\n expr,\n E'\\\\(''\\([0-9 :+-]*\\)''\\\\).*\\\\(''\\([0-9 :+-]*\\)''\\\\)'\n ) AS m\n ) SELECT tstzrange(m[1]::timestamptz, m[2]::timestamptz) FROM t\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE STRICT;\n\nCREATE OR REPLACE FUNCTION dt_constraint(coid oid, OUT dt tstzrange, OUT edt tstzrange) RETURNS RECORD AS $$\nDECLARE\n expr text := pg_get_constraintdef(coid);\n matches timestamptz[];\nBEGIN\n IF expr LIKE '%NULL%' THEN\n dt := tstzrange(null::timestamptz, null::timestamptz);\n edt := tstzrange(null::timestamptz, null::timestamptz);\n RETURN;\n END IF;\n WITH f AS (SELECT (regexp_matches(expr, E'([0-9]{4}-[0-1][0-9]-[0-3][0-9] [0-2][0-9]:[0-5][0-9]:[0-5][0-9]\\.?[0-9]*)', 'g'))[1] f)\n SELECT array_agg(f::timestamptz) INTO matches FROM f;\n IF cardinality(matches) = 4 THEN\n dt := tstzrange(matches[1], matches[2],'[]');\n edt := tstzrange(matches[3], matches[4], '[]');\n RETURN;\n ELSIF cardinality(matches) = 2 THEN\n edt := tstzrange(matches[1], matches[2],'[]');\n RETURN;\n END IF;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\nCREATE OR REPLACE VIEW partition_sys_meta AS\nSELECT\n (parse_ident(relid::text))[cardinality(parse_ident(relid::text))] as partition,\n replace(replace(CASE WHEN level = 1 THEN pg_get_expr(c.relpartbound, c.oid)\n ELSE pg_get_expr(parent.relpartbound, parent.oid)\n END, 'FOR VALUES IN (''',''), ''')','') AS collection,\n level,\n c.reltuples,\n c.relhastriggers,\n COALESCE(constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as partition_dtrange,\n COALESCE((dt_constraint(edt.oid)).dt, constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as constraint_dtrange,\n COALESCE((dt_constraint(edt.oid)).edt, tstzrange('-infinity', 'infinity','[]')) as constraint_edtrange\nFROM\n pg_partition_tree('items')\n JOIN pg_class c ON (relid::regclass = c.oid)\n JOIN pg_class parent ON (parentrelid::regclass = parent.oid AND isleaf)\n LEFT JOIN pg_constraint edt ON (conrelid=c.oid AND contype='c')\nWHERE isleaf\n;\n\nCREATE OR REPLACE VIEW partitions_view AS\nSELECT\n (parse_ident(relid::text))[cardinality(parse_ident(relid::text))] as partition,\n replace(replace(CASE WHEN level = 1 THEN pg_get_expr(c.relpartbound, c.oid)\n ELSE pg_get_expr(parent.relpartbound, parent.oid)\n END, 'FOR VALUES IN (''',''), ''')','') AS collection,\n level,\n c.reltuples,\n c.relhastriggers,\n COALESCE(pgstac.constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as partition_dtrange,\n COALESCE((pgstac.dt_constraint(edt.oid)).dt, pgstac.constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as constraint_dtrange,\n COALESCE((pgstac.dt_constraint(edt.oid)).edt, tstzrange('-infinity', 'infinity','[]')) as constraint_edtrange,\n dtrange,\n edtrange,\n spatial,\n last_updated\nFROM\n pg_partition_tree('pgstac.items')\n JOIN pg_class c ON (relid::regclass = c.oid)\n JOIN pg_class parent ON (parentrelid::regclass = parent.oid AND isleaf)\n LEFT JOIN pg_constraint edt ON (conrelid=c.oid AND contype='c')\n LEFT JOIN pgstac.partition_stats ON ((parse_ident(relid::text))[cardinality(parse_ident(relid::text))]=partition)\nWHERE isleaf\n;\n\nCREATE MATERIALIZED VIEW partitions AS\nSELECT * FROM partitions_view;\nCREATE UNIQUE INDEX ON partitions (partition);\n\nCREATE MATERIALIZED VIEW partition_steps AS\nSELECT\n partition as name,\n date_trunc('month',lower(partition_dtrange)) as sdate,\n date_trunc('month', upper(partition_dtrange)) + '1 month'::interval as edate\n FROM partitions_view WHERE partition_dtrange IS NOT NULL AND partition_dtrange != 'empty'::tstzrange\n ORDER BY dtrange ASC\n;\n\n\nCREATE OR REPLACE FUNCTION update_partition_stats_q(_partition text, istrigger boolean default false) RETURNS VOID AS $$\nDECLARE\nBEGIN\n PERFORM run_or_queue(\n format('SELECT update_partition_stats(%L, %L);', _partition, istrigger)\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION update_partition_stats(_partition text, istrigger boolean default false) RETURNS VOID AS $$\nDECLARE\n dtrange tstzrange;\n edtrange tstzrange;\n cdtrange tstzrange;\n cedtrange tstzrange;\n extent geometry;\n collection text;\nBEGIN\n RAISE NOTICE 'Updating stats for %.', _partition;\n EXECUTE format(\n $q$\n SELECT\n tstzrange(min(datetime), max(datetime),'[]'),\n tstzrange(min(end_datetime), max(end_datetime), '[]')\n FROM %I\n $q$,\n _partition\n ) INTO dtrange, edtrange;\n extent := st_estimatedextent('pgstac', _partition, 'geometry');\n RAISE DEBUG 'Estimated Extent: %', extent;\n INSERT INTO partition_stats (partition, dtrange, edtrange, spatial, last_updated)\n SELECT _partition, dtrange, edtrange, extent, now()\n ON CONFLICT (partition) DO\n UPDATE SET\n dtrange=EXCLUDED.dtrange,\n edtrange=EXCLUDED.edtrange,\n spatial=EXCLUDED.spatial,\n last_updated=EXCLUDED.last_updated\n ;\n\n SELECT\n constraint_dtrange, constraint_edtrange, pv.collection\n INTO cdtrange, cedtrange, collection\n FROM partitions_view pv WHERE partition = _partition;\n REFRESH MATERIALIZED VIEW partitions;\n REFRESH MATERIALIZED VIEW partition_steps;\n\n\n RAISE NOTICE 'Checking if we need to modify constraints...';\n RAISE NOTICE 'cdtrange: % dtrange: % cedtrange: % edtrange: %',cdtrange, dtrange, cedtrange, edtrange;\n IF\n (cdtrange IS DISTINCT FROM dtrange OR edtrange IS DISTINCT FROM cedtrange)\n AND NOT istrigger\n THEN\n RAISE NOTICE 'Modifying Constraints';\n RAISE NOTICE 'Existing % %', cdtrange, cedtrange;\n RAISE NOTICE 'New % %', dtrange, edtrange;\n PERFORM drop_table_constraints(_partition);\n PERFORM create_table_constraints(_partition, dtrange, edtrange);\n REFRESH MATERIALIZED VIEW partitions;\n REFRESH MATERIALIZED VIEW partition_steps;\n END IF;\n RAISE NOTICE 'Checking if we need to update collection extents.';\n IF get_setting_bool('update_collection_extent') THEN\n RAISE NOTICE 'updating collection extent for %', collection;\n PERFORM run_or_queue(format($q$\n UPDATE collections\n SET content = jsonb_set_lax(\n content,\n '{extent}'::text[],\n collection_extent(%L, FALSE),\n true,\n 'use_json_null'\n ) WHERE id=%L\n ;\n $q$, collection, collection));\n ELSE\n RAISE NOTICE 'Not updating collection extent for %', collection;\n END IF;\n\nEND;\n$$ LANGUAGE PLPGSQL STRICT SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION partition_name( IN collection text, IN dt timestamptz, OUT partition_name text, OUT partition_range tstzrange) AS $$\nDECLARE\n c RECORD;\n parent_name text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n parent_name := format('_items_%s', c.key);\n\n\n IF c.partition_trunc = 'year' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYY'));\n ELSIF c.partition_trunc = 'month' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYYMM'));\n ELSE\n partition_name := parent_name;\n partition_range := tstzrange('-infinity'::timestamptz, 'infinity'::timestamptz, '[]');\n END IF;\n IF partition_range IS NULL THEN\n partition_range := tstzrange(\n date_trunc(c.partition_trunc::text, dt),\n date_trunc(c.partition_trunc::text, dt) + concat('1 ', c.partition_trunc)::interval\n );\n END IF;\n RETURN;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION drop_table_constraints(t text) RETURNS text AS $$\nDECLARE\n q text;\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM partitions_view WHERE partition=t) THEN\n RETURN NULL;\n END IF;\n FOR q IN SELECT FORMAT(\n $q$\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n $q$,\n t,\n conname\n ) FROM pg_constraint\n WHERE conrelid=t::regclass::oid AND contype='c'\n LOOP\n EXECUTE q;\n END LOOP;\n RETURN t;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE OR REPLACE FUNCTION create_table_constraints(t text, _dtrange tstzrange, _edtrange tstzrange) RETURNS text AS $$\nDECLARE\n q text;\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM partitions_view WHERE partition=t) THEN\n RETURN NULL;\n END IF;\n RAISE NOTICE 'Creating Table Constraints for % % %', t, _dtrange, _edtrange;\n IF _dtrange = 'empty' AND _edtrange = 'empty' THEN\n q :=format(\n $q$\n DO $block$\n BEGIN\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n ALTER TABLE %I\n ADD CONSTRAINT %I\n CHECK (((datetime IS NULL) AND (end_datetime IS NULL))) NOT VALID\n ;\n ALTER TABLE %I\n VALIDATE CONSTRAINT %I\n ;\n\n\n\n EXCEPTION WHEN others THEN\n RAISE WARNING '%%, Issue Altering Constraints. Please run update_partition_stats(%I)', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n $block$;\n $q$,\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t),\n t\n );\n ELSE\n q :=format(\n $q$\n DO $block$\n BEGIN\n\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n ALTER TABLE %I\n ADD CONSTRAINT %I\n CHECK (\n (datetime >= %L)\n AND (datetime <= %L)\n AND (end_datetime >= %L)\n AND (end_datetime <= %L)\n ) NOT VALID\n ;\n ALTER TABLE %I\n VALIDATE CONSTRAINT %I\n ;\n\n\n\n EXCEPTION WHEN others THEN\n RAISE WARNING '%%, Issue Altering Constraints. Please run update_partition_stats(%I)', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n $block$;\n $q$,\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t),\n lower(_dtrange),\n upper(_dtrange),\n lower(_edtrange),\n upper(_edtrange),\n t,\n format('%s_dt', t),\n t\n );\n END IF;\n PERFORM run_or_queue(q);\n RETURN t;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION check_partition(\n _collection text,\n _dtrange tstzrange,\n _edtrange tstzrange\n) RETURNS text AS $$\nDECLARE\n c RECORD;\n pm RECORD;\n _partition_name text;\n _partition_dtrange tstzrange;\n _constraint_dtrange tstzrange;\n _constraint_edtrange tstzrange;\n q text;\n deferrable_q text;\n err_context text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=_collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', _collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n\n IF c.partition_trunc IS NOT NULL THEN\n _partition_dtrange := tstzrange(\n date_trunc(c.partition_trunc, lower(_dtrange)),\n date_trunc(c.partition_trunc, lower(_dtrange)) + (concat('1 ', c.partition_trunc))::interval,\n '[)'\n );\n ELSE\n _partition_dtrange := '[-infinity, infinity]'::tstzrange;\n END IF;\n\n IF NOT _partition_dtrange @> _dtrange THEN\n RAISE EXCEPTION 'dtrange % is greater than the partition size % for collection %', _dtrange, c.partition_trunc, _collection;\n END IF;\n\n\n IF c.partition_trunc = 'year' THEN\n _partition_name := format('_items_%s_%s', c.key, to_char(lower(_partition_dtrange),'YYYY'));\n ELSIF c.partition_trunc = 'month' THEN\n _partition_name := format('_items_%s_%s', c.key, to_char(lower(_partition_dtrange),'YYYYMM'));\n ELSE\n _partition_name := format('_items_%s', c.key);\n END IF;\n\n SELECT * INTO pm FROM partition_sys_meta WHERE collection=_collection AND partition_dtrange @> _dtrange;\n IF FOUND THEN\n RAISE NOTICE '% % %', _edtrange, _dtrange, pm;\n _constraint_edtrange :=\n tstzrange(\n least(\n lower(_edtrange),\n nullif(lower(pm.constraint_edtrange), '-infinity')\n ),\n greatest(\n upper(_edtrange),\n nullif(upper(pm.constraint_edtrange), 'infinity')\n ),\n '[]'\n );\n _constraint_dtrange :=\n tstzrange(\n least(\n lower(_dtrange),\n nullif(lower(pm.constraint_dtrange), '-infinity')\n ),\n greatest(\n upper(_dtrange),\n nullif(upper(pm.constraint_dtrange), 'infinity')\n ),\n '[]'\n );\n\n IF pm.constraint_edtrange @> _edtrange AND pm.constraint_dtrange @> _dtrange THEN\n RETURN pm.partition;\n ELSE\n PERFORM drop_table_constraints(_partition_name);\n END IF;\n ELSE\n _constraint_edtrange := _edtrange;\n _constraint_dtrange := _dtrange;\n END IF;\n RAISE NOTICE 'EXISTING CONSTRAINTS % %, NEW % %', pm.constraint_dtrange, pm.constraint_edtrange, _constraint_dtrange, _constraint_edtrange;\n RAISE NOTICE 'Creating partition % %', _partition_name, _partition_dtrange;\n IF c.partition_trunc IS NULL THEN\n q := format(\n $q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n GRANT ALL ON %I to pgstac_ingest;\n $q$,\n _partition_name,\n _collection,\n concat(_partition_name,'_pk'),\n _partition_name,\n _partition_name\n );\n ELSE\n q := format(\n $q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L) PARTITION BY RANGE (datetime);\n CREATE TABLE IF NOT EXISTS %I partition OF %I FOR VALUES FROM (%L) TO (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n GRANT ALL ON %I TO pgstac_ingest;\n $q$,\n format('_items_%s', c.key),\n _collection,\n _partition_name,\n format('_items_%s', c.key),\n lower(_partition_dtrange),\n upper(_partition_dtrange),\n format('%s_pk', _partition_name),\n _partition_name,\n _partition_name\n );\n END IF;\n\n BEGIN\n EXECUTE q;\n EXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', _partition_name;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\n END;\n PERFORM maintain_partitions(_partition_name);\n PERFORM update_partition_stats_q(_partition_name, true);\n REFRESH MATERIALIZED VIEW partitions;\n REFRESH MATERIALIZED VIEW partition_steps;\n RETURN _partition_name;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION repartition(_collection text, _partition_trunc text, triggered boolean DEFAULT FALSE) RETURNS text AS $$\nDECLARE\n c RECORD;\n q text;\n from_trunc text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=_collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', _collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n IF triggered THEN\n RAISE NOTICE 'Converting % to % partitioning via Trigger', _collection, _partition_trunc;\n ELSE\n RAISE NOTICE 'Converting % from using % to % partitioning', _collection, c.partition_trunc, _partition_trunc;\n IF c.partition_trunc IS NOT DISTINCT FROM _partition_trunc THEN\n RAISE NOTICE 'Collection % already set to use partition by %', _collection, _partition_trunc;\n RETURN _collection;\n END IF;\n END IF;\n\n IF EXISTS (SELECT 1 FROM partitions_view WHERE collection=_collection LIMIT 1) THEN\n EXECUTE format(\n $q$\n CREATE TEMP TABLE changepartitionstaging ON COMMIT DROP AS SELECT * FROM %I;\n DROP TABLE IF EXISTS %I CASCADE;\n WITH p AS (\n SELECT\n collection,\n CASE\n WHEN %L IS NULL THEN '-infinity'::timestamptz\n ELSE date_trunc(%L, datetime)\n END as d,\n tstzrange(min(datetime),max(datetime),'[]') as dtrange,\n tstzrange(min(end_datetime),max(end_datetime),'[]') as edtrange\n FROM changepartitionstaging\n GROUP BY 1,2\n ) SELECT check_partition(collection, dtrange, edtrange) FROM p;\n INSERT INTO items SELECT * FROM changepartitionstaging;\n DROP TABLE changepartitionstaging;\n $q$,\n concat('_items_', c.key),\n concat('_items_', c.key),\n c.partition_trunc,\n c.partition_trunc\n );\n END IF;\n RETURN _collection;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE OR REPLACE FUNCTION collections_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n q text;\n partition_name text := format('_items_%s', NEW.key);\n partition_exists boolean := false;\n partition_empty boolean := true;\n err_context text;\n loadtemp boolean := FALSE;\nBEGIN\n RAISE NOTICE 'Collection Trigger. % %', NEW.id, NEW.key;\n IF TG_OP = 'UPDATE' AND NEW.partition_trunc IS DISTINCT FROM OLD.partition_trunc THEN\n PERFORM repartition(NEW.id, NEW.partition_trunc, TRUE);\n END IF;\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER collections_trigger AFTER\nINSERT\nOR\nUPDATE ON collections\nFOR EACH ROW EXECUTE FUNCTION collections_trigger_func();\n\nCREATE OR REPLACE FUNCTION chunker(\n IN _where text,\n OUT s timestamptz,\n OUT e timestamptz\n) RETURNS SETOF RECORD AS $$\nDECLARE\n explain jsonb;\nBEGIN\n IF _where IS NULL THEN\n _where := ' TRUE ';\n END IF;\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s;', _where)\n INTO explain;\n RAISE DEBUG 'EXPLAIN: %', explain;\n\n RETURN QUERY\n WITH t AS (\n SELECT j->>0 as p FROM\n jsonb_path_query(\n explain,\n 'strict $.**.\"Relation Name\" ? (@ != null)'\n ) j\n ),\n parts AS (\n SELECT sdate, edate FROM t JOIN partition_steps ON (t.p = name)\n ),\n times AS (\n SELECT sdate FROM parts\n UNION\n SELECT edate FROM parts\n ),\n uniq AS (\n SELECT DISTINCT sdate FROM times ORDER BY sdate\n ),\n last AS (\n SELECT sdate, lead(sdate, 1) over () as edate FROM uniq\n )\n SELECT sdate, edate FROM last WHERE edate IS NOT NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION partition_queries(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN partitions text[] DEFAULT NULL\n) RETURNS SETOF text AS $$\nDECLARE\n query text;\n sdate timestamptz;\n edate timestamptz;\nBEGIN\nIF _where IS NULL OR trim(_where) = '' THEN\n _where = ' TRUE ';\nEND IF;\nRAISE NOTICE 'Getting chunks for % %', _where, _orderby;\nIF _orderby ILIKE 'datetime d%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 DESC LOOP\n RETURN NEXT format($q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n $q$,\n sdate,\n edate,\n _where,\n _orderby\n );\n END LOOP;\nELSIF _orderby ILIKE 'datetime a%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 ASC LOOP\n RETURN NEXT format($q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n $q$,\n sdate,\n edate,\n _where,\n _orderby\n );\n END LOOP;\nELSE\n query := format($q$\n SELECT * FROM items\n WHERE %s\n ORDER BY %s\n $q$, _where, _orderby\n );\n\n RETURN NEXT query;\n RETURN;\nEND IF;\n\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION partition_query_view(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN _limit int DEFAULT 10\n) RETURNS text AS $$\n WITH p AS (\n SELECT * FROM partition_queries(_where, _orderby) p\n )\n SELECT\n CASE WHEN EXISTS (SELECT 1 FROM p) THEN\n (SELECT format($q$\n SELECT * FROM (\n %s\n ) total LIMIT %s\n $q$,\n string_agg(\n format($q$ SELECT * FROM ( %s ) AS sub $q$, p),\n '\n UNION ALL\n '\n ),\n _limit\n ))\n ELSE NULL\n END FROM p;\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION q_to_tsquery (input text)\n RETURNS tsquery\n AS $$\nDECLARE\n processed_text text;\n temp_text text;\n quote_array text[];\n placeholder text := '@QUOTE@';\nBEGIN\n -- Extract all quoted phrases and store in array\n quote_array := regexp_matches(input, '\"[^\"]*\"', 'g');\n\n -- Replace each quoted part with a unique placeholder if there are any quoted phrases\n IF array_length(quote_array, 1) IS NOT NULL THEN\n processed_text := input;\n FOR i IN array_lower(quote_array, 1) .. array_upper(quote_array, 1) LOOP\n processed_text := replace(processed_text, quote_array[i], placeholder || i || placeholder);\n END LOOP;\n ELSE\n processed_text := input;\n END IF;\n\n -- Replace non-quoted text using regular expressions\n\n -- , -> |\n processed_text := regexp_replace(processed_text, ',(?=(?:[^\"]*\"[^\"]*\")*[^\"]*$)', ' | ', 'g');\n\n -- and -> &\n processed_text := regexp_replace(processed_text, '\\s+AND\\s+', ' & ', 'gi');\n\n -- or -> |\n processed_text := regexp_replace(processed_text, '\\s+OR\\s+', ' | ', 'gi');\n\n -- +term -> & term\n processed_text := regexp_replace(processed_text, '\\+([a-zA-Z0-9_]+)', '& \\1', 'g');\n\n -- -term -> ! term\n processed_text := regexp_replace(processed_text, '\\-([a-zA-Z0-9_]+)', '& ! \\1', 'g');\n\n -- Replace placeholders back with quoted phrases if there were any\n IF array_length(quote_array, 1) IS NOT NULL THEN\n FOR i IN array_lower(quote_array, 1) .. array_upper(quote_array, 1) LOOP\n processed_text := replace(processed_text, placeholder || i || placeholder, '''' || substring(quote_array[i] from 2 for length(quote_array[i]) - 2) || '''');\n END LOOP;\n END IF;\n\n -- Print processed_text to the console for debugging purposes\n RAISE NOTICE 'processed_text: %', processed_text;\n\n RETURN to_tsquery(processed_text);\nEND;\n$$\nLANGUAGE plpgsql;\n\n\nCREATE OR REPLACE FUNCTION stac_search_to_where(j jsonb) RETURNS text AS $$\nDECLARE\n where_segments text[];\n _where text;\n dtrange tstzrange;\n collections text[];\n geom geometry;\n sdate timestamptz;\n edate timestamptz;\n filterlang text;\n filter jsonb := j->'filter';\n ft_query tsquery;\nBEGIN\n IF j ? 'ids' THEN\n where_segments := where_segments || format('id = ANY (%L) ', to_text_array(j->'ids'));\n END IF;\n\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n where_segments := where_segments || format('collection = ANY (%L) ', collections);\n END IF;\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n\n where_segments := where_segments || format(' datetime <= %L::timestamptz AND end_datetime >= %L::timestamptz ',\n edate,\n sdate\n );\n END IF;\n\n IF j ? 'q' THEN\n ft_query := q_to_tsquery(j->>'q');\n where_segments := where_segments || format(\n $quote$\n (\n to_tsvector('english', content->'properties'->>'description') ||\n to_tsvector('english', coalesce(content->'properties'->>'title', '')) ||\n to_tsvector('english', coalesce(content->'properties'->>'keywords', ''))\n ) @@ %L\n $quote$,\n ft_query\n );\n END IF;\n\n geom := stac_geom(j);\n IF geom IS NOT NULL THEN\n where_segments := where_segments || format('st_intersects(geometry, %L)',geom);\n END IF;\n\n filterlang := COALESCE(\n j->>'filter-lang',\n get_setting('default_filter_lang', j->'conf')\n );\n IF NOT filter @? '$.**.op' THEN\n filterlang := 'cql-json';\n END IF;\n\n IF filterlang NOT IN ('cql-json','cql2-json') AND j ? 'filter' THEN\n RAISE EXCEPTION '% is not a supported filter-lang. Please use cql-json or cql2-json.', filterlang;\n END IF;\n\n IF j ? 'query' AND j ? 'filter' THEN\n RAISE EXCEPTION 'Can only use either query or filter at one time.';\n END IF;\n\n IF j ? 'query' THEN\n filter := query_to_cql2(j->'query');\n ELSIF filterlang = 'cql-json' THEN\n filter := cql1_to_cql2(filter);\n END IF;\n RAISE NOTICE 'FILTER: %', filter;\n where_segments := where_segments || cql2_query(filter);\n IF cardinality(where_segments) < 1 THEN\n RETURN ' TRUE ';\n END IF;\n\n _where := array_to_string(array_remove(where_segments, NULL), ' AND ');\n\n IF _where IS NULL OR BTRIM(_where) = '' THEN\n RETURN ' TRUE ';\n END IF;\n RETURN _where;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION parse_sort_dir(_dir text, reverse boolean default false) RETURNS text AS $$\n WITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n ) SELECT\n CASE\n WHEN NOT reverse THEN d\n WHEN d = 'ASC' THEN 'DESC'\n WHEN d = 'DESC' THEN 'ASC'\n END\n FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION sort_dir_to_op(_dir text, prev boolean default false) RETURNS text AS $$\n WITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n ) SELECT\n CASE\n WHEN d = 'ASC' AND prev THEN '<='\n WHEN d = 'DESC' AND prev THEN '>='\n WHEN d = 'ASC' THEN '>='\n WHEN d = 'DESC' THEN '<='\n END\n FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION sort_sqlorderby(\n _search jsonb DEFAULT NULL,\n reverse boolean DEFAULT FALSE\n) RETURNS text AS $$\n WITH sortby AS (\n SELECT coalesce(_search->'sortby','[{\"field\":\"datetime\", \"direction\":\"desc\"}]') as sort\n ), withid AS (\n SELECT CASE\n WHEN sort @? '$[*] ? (@.field == \"id\")' THEN sort\n ELSE sort || '[{\"field\":\"id\", \"direction\":\"desc\"}]'::jsonb\n END as sort\n FROM sortby\n ), withid_rows AS (\n SELECT jsonb_array_elements(sort) as value FROM withid\n ),sorts AS (\n SELECT\n coalesce(\n (queryable(value->>'field')).expression\n ) as key,\n parse_sort_dir(value->>'direction', reverse) as dir\n FROM withid_rows\n )\n SELECT array_to_string(\n array_agg(concat(key, ' ', dir)),\n ', '\n ) FROM sorts;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_sort_dir(sort_item jsonb) RETURNS text AS $$\n SELECT CASE WHEN sort_item->>'direction' ILIKE 'desc%' THEN 'DESC' ELSE 'ASC' END;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION get_token_val_str(\n _field text,\n _item items\n) RETURNS text AS $$\nDECLARE\n q text;\n literal text;\nBEGIN\n q := format($q$ SELECT quote_literal(%s) FROM (SELECT $1.*) as r;$q$, _field);\n EXECUTE q INTO literal USING _item;\n RETURN literal;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n\nCREATE OR REPLACE FUNCTION get_token_record(IN _token text, OUT prev BOOLEAN, OUT item items) RETURNS RECORD AS $$\nDECLARE\n _itemid text := _token;\n _collectionid text;\nBEGIN\n IF _token IS NULL THEN\n RETURN;\n END IF;\n RAISE NOTICE 'Looking for token: %', _token;\n prev := FALSE;\n IF _token ILIKE 'prev:%' THEN\n _itemid := replace(_token, 'prev:','');\n prev := TRUE;\n ELSIF _token ILIKE 'next:%' THEN\n _itemid := replace(_token, 'next:', '');\n END IF;\n SELECT id INTO _collectionid FROM collections WHERE _itemid LIKE concat(id,':%');\n IF FOUND THEN\n _itemid := replace(_itemid, concat(_collectionid,':'), '');\n SELECT * INTO item FROM items WHERE id=_itemid AND collection=_collectionid;\n ELSE\n SELECT * INTO item FROM items WHERE id=_itemid;\n END IF;\n IF item IS NULL THEN\n RAISE EXCEPTION 'Could not find item using token: % item: % collection: %', _token, _itemid, _collectionid;\n END IF;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\n\nCREATE OR REPLACE FUNCTION get_token_filter(\n _sortby jsonb DEFAULT '[{\"field\":\"datetime\",\"direction\":\"desc\"}]'::jsonb,\n token_item items DEFAULT NULL,\n prev boolean DEFAULT FALSE,\n inclusive boolean DEFAULT FALSE\n) RETURNS text AS $$\nDECLARE\n ltop text := '<';\n gtop text := '>';\n dir text;\n sort record;\n orfilter text := '';\n orfilters text[] := '{}'::text[];\n andfilters text[] := '{}'::text[];\n output text;\n token_where text;\nBEGIN\n IF _sortby IS NULL OR _sortby = '[]'::jsonb THEN\n _sortby := '[{\"field\":\"datetime\",\"direction\":\"desc\"}]'::jsonb;\n END IF;\n _sortby := _sortby || jsonb_build_object('field','id','direction',_sortby->0->>'direction');\n RAISE NOTICE 'Getting Token Filter. % %', _sortby, token_item;\n IF inclusive THEN\n orfilters := orfilters || format('( id=%L AND collection=%L )' , token_item.id, token_item.collection);\n END IF;\n\n FOR sort IN\n WITH s1 AS (\n SELECT\n _row,\n (queryable(value->>'field')).expression as _field,\n (value->>'field' = 'id') as _isid,\n get_sort_dir(value) as _dir\n FROM jsonb_array_elements(_sortby)\n WITH ORDINALITY AS t(value, _row)\n )\n SELECT\n _row,\n _field,\n _dir,\n get_token_val_str(_field, token_item) as _val\n FROM s1\n WHERE _row <= (SELECT min(_row) FROM s1 WHERE _isid)\n LOOP\n orfilter := NULL;\n RAISE NOTICE 'SORT: %', sort;\n IF sort._val IS NOT NULL AND ((prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC')) THEN\n orfilter := format($f$(\n (%s %s %s) OR (%s IS NULL)\n )$f$,\n sort._field,\n ltop,\n sort._val,\n sort._val\n );\n ELSIF sort._val IS NULL AND ((prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC')) THEN\n RAISE NOTICE '< but null';\n orfilter := format('%s IS NOT NULL', sort._field);\n ELSIF sort._val IS NULL THEN\n RAISE NOTICE '> but null';\n ELSE\n orfilter := format($f$(\n (%s %s %s) OR (%s IS NULL)\n )$f$,\n sort._field,\n gtop,\n sort._val,\n sort._field\n );\n END IF;\n RAISE NOTICE 'ORFILTER: %', orfilter;\n\n IF orfilter IS NOT NULL THEN\n IF sort._row = 1 THEN\n orfilters := orfilters || orfilter;\n ELSE\n orfilters := orfilters || format('(%s AND %s)', array_to_string(andfilters, ' AND '), orfilter);\n END IF;\n END IF;\n IF sort._val IS NOT NULL THEN\n andfilters := andfilters || format('%s = %s', sort._field, sort._val);\n ELSE\n andfilters := andfilters || format('%s IS NULL', sort._field);\n END IF;\n END LOOP;\n\n output := array_to_string(orfilters, ' OR ');\n\n token_where := concat('(',coalesce(output,'true'),')');\n IF trim(token_where) = '' THEN\n token_where := NULL;\n END IF;\n RAISE NOTICE 'TOKEN_WHERE: %',token_where;\n RETURN token_where;\n END;\n$$ LANGUAGE PLPGSQL SET transform_null_equals TO TRUE\n;\n\nCREATE OR REPLACE FUNCTION search_tohash(jsonb) RETURNS jsonb AS $$\n SELECT $1 - '{token,limit,context,includes,excludes}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION search_hash(jsonb, jsonb) RETURNS text AS $$\n SELECT md5(concat(search_tohash($1)::text,$2::text));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TABLE IF NOT EXISTS searches(\n hash text GENERATED ALWAYS AS (search_hash(search, metadata)) STORED PRIMARY KEY,\n search jsonb NOT NULL,\n _where text,\n orderby text,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n metadata jsonb DEFAULT '{}'::jsonb NOT NULL\n);\n\nCREATE TABLE IF NOT EXISTS search_wheres(\n id bigint generated always as identity primary key,\n _where text NOT NULL,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n statslastupdated timestamptz,\n estimated_count bigint,\n estimated_cost float,\n time_to_estimate float,\n total_count bigint,\n time_to_count float,\n partitions text[]\n);\n\nCREATE INDEX IF NOT EXISTS search_wheres_partitions ON search_wheres USING GIN (partitions);\nCREATE UNIQUE INDEX IF NOT EXISTS search_wheres_where ON search_wheres ((md5(_where)));\n\nCREATE OR REPLACE FUNCTION where_stats(\n inwhere text,\n updatestats boolean default false,\n conf jsonb default null\n) RETURNS search_wheres AS $$\nDECLARE\n t timestamptz;\n i interval;\n explain_json jsonb;\n partitions text[];\n sw search_wheres%ROWTYPE;\n inwhere_hash text := md5(inwhere);\n _context text := lower(context(conf));\n _stats_ttl interval := context_stats_ttl(conf);\n _estimated_cost_threshold float := context_estimated_cost(conf);\n _estimated_count_threshold int := context_estimated_count(conf);\n ro bool := pgstac.readonly(conf);\nBEGIN\n -- If updatestats is true then set ttl to 0\n IF updatestats THEN\n RAISE DEBUG 'Updatestats set to TRUE, setting TTL to 0';\n _stats_ttl := '0'::interval;\n END IF;\n\n -- If we don't need to calculate context, just return\n IF _context = 'off' THEN\n sw._where = inwhere;\n RETURN sw;\n END IF;\n\n -- Get any stats that we have.\n IF NOT ro THEN\n -- If there is a lock where another process is\n -- updating the stats, wait so that we don't end up calculating a bunch of times.\n SELECT * INTO sw FROM search_wheres WHERE md5(_where)=inwhere_hash FOR UPDATE;\n ELSE\n SELECT * INTO sw FROM search_wheres WHERE md5(_where)=inwhere_hash;\n END IF;\n\n -- If there is a cached row, figure out if we need to update\n IF\n sw IS NOT NULL\n AND sw.statslastupdated IS NOT NULL\n AND sw.total_count IS NOT NULL\n AND now() - sw.statslastupdated <= _stats_ttl\n THEN\n -- we have a cached row with data that is within our ttl\n RAISE DEBUG 'Stats present in table and lastupdated within ttl: %', sw;\n IF NOT ro THEN\n RAISE DEBUG 'Updating search_wheres only bumping lastused and usecount';\n UPDATE search_wheres SET\n lastused = now(),\n usecount = search_wheres.usecount + 1\n WHERE md5(_where) = inwhere_hash\n RETURNING * INTO sw;\n END IF;\n RAISE DEBUG 'Returning cached counts. %', sw;\n RETURN sw;\n END IF;\n\n -- Calculate estimated cost and rows\n -- Use explain to get estimated count/cost\n IF sw.estimated_count IS NULL OR sw.estimated_cost IS NULL THEN\n RAISE DEBUG 'Calculating estimated stats';\n t := clock_timestamp();\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s', inwhere)\n INTO explain_json;\n RAISE DEBUG 'Time for just the explain: %', clock_timestamp() - t;\n i := clock_timestamp() - t;\n\n sw.estimated_count := explain_json->0->'Plan'->'Plan Rows';\n sw.estimated_cost := explain_json->0->'Plan'->'Total Cost';\n sw.time_to_estimate := extract(epoch from i);\n END IF;\n\n RAISE DEBUG 'ESTIMATED_COUNT: %, THRESHOLD %', sw.estimated_count, _estimated_count_threshold;\n RAISE DEBUG 'ESTIMATED_COST: %, THRESHOLD %', sw.estimated_cost, _estimated_cost_threshold;\n\n -- If context is set to auto and the costs are within the threshold return the estimated costs\n IF\n _context = 'auto'\n AND sw.estimated_count >= _estimated_count_threshold\n AND sw.estimated_cost >= _estimated_cost_threshold\n THEN\n IF NOT ro THEN\n INSERT INTO search_wheres (\n _where,\n lastused,\n usecount,\n statslastupdated,\n estimated_count,\n estimated_cost,\n time_to_estimate,\n total_count,\n time_to_count\n ) VALUES (\n inwhere,\n now(),\n 1,\n now(),\n sw.estimated_count,\n sw.estimated_cost,\n sw.time_to_estimate,\n null,\n null\n ) ON CONFLICT ((md5(_where)))\n DO UPDATE SET\n lastused = EXCLUDED.lastused,\n usecount = search_wheres.usecount + 1,\n statslastupdated = EXCLUDED.statslastupdated,\n estimated_count = EXCLUDED.estimated_count,\n estimated_cost = EXCLUDED.estimated_cost,\n time_to_estimate = EXCLUDED.time_to_estimate,\n total_count = EXCLUDED.total_count,\n time_to_count = EXCLUDED.time_to_count\n RETURNING * INTO sw;\n END IF;\n RAISE DEBUG 'Estimates are within thresholds, returning estimates. %', sw;\n RETURN sw;\n END IF;\n\n -- Calculate Actual Count\n t := clock_timestamp();\n RAISE NOTICE 'Calculating actual count...';\n EXECUTE format(\n 'SELECT count(*) FROM items WHERE %s',\n inwhere\n ) INTO sw.total_count;\n i := clock_timestamp() - t;\n RAISE NOTICE 'Actual Count: % -- %', sw.total_count, i;\n sw.time_to_count := extract(epoch FROM i);\n\n IF NOT ro THEN\n INSERT INTO search_wheres (\n _where,\n lastused,\n usecount,\n statslastupdated,\n estimated_count,\n estimated_cost,\n time_to_estimate,\n total_count,\n time_to_count\n ) VALUES (\n inwhere,\n now(),\n 1,\n now(),\n sw.estimated_count,\n sw.estimated_cost,\n sw.time_to_estimate,\n sw.total_count,\n sw.time_to_count\n ) ON CONFLICT ((md5(_where)))\n DO UPDATE SET\n lastused = EXCLUDED.lastused,\n usecount = search_wheres.usecount + 1,\n statslastupdated = EXCLUDED.statslastupdated,\n estimated_count = EXCLUDED.estimated_count,\n estimated_cost = EXCLUDED.estimated_cost,\n time_to_estimate = EXCLUDED.time_to_estimate,\n total_count = EXCLUDED.total_count,\n time_to_count = EXCLUDED.time_to_count\n RETURNING * INTO sw;\n END IF;\n RAISE DEBUG 'Returning with actual count. %', sw;\n RETURN sw;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION search_query(\n _search jsonb = '{}'::jsonb,\n updatestats boolean = false,\n _metadata jsonb = '{}'::jsonb\n) RETURNS searches AS $$\nDECLARE\n search searches%ROWTYPE;\n cached_search searches%ROWTYPE;\n pexplain jsonb;\n t timestamptz;\n i interval;\n doupdate boolean := FALSE;\n insertfound boolean := FALSE;\n ro boolean := pgstac.readonly();\n found_search text;\nBEGIN\n RAISE NOTICE 'SEARCH: %', _search;\n -- Calculate hash, where clause, and order by statement\n search.search := _search;\n search.metadata := _metadata;\n search.hash := search_hash(_search, _metadata);\n search._where := stac_search_to_where(_search);\n search.orderby := sort_sqlorderby(_search);\n search.lastused := now();\n search.usecount := 1;\n\n -- If we are in read only mode, directly return search\n IF ro THEN\n RETURN search;\n END IF;\n\n RAISE NOTICE 'Updating Statistics for search: %s', search;\n -- Update statistics for times used and and when last used\n -- If the entry is locked, rather than waiting, skip updating the stats\n INSERT INTO searches (search, lastused, usecount, metadata)\n VALUES (search.search, now(), 1, search.metadata)\n ON CONFLICT DO NOTHING\n RETURNING * INTO cached_search\n ;\n\n IF NOT FOUND OR cached_search IS NULL THEN\n UPDATE searches SET\n lastused = now(),\n usecount = searches.usecount + 1\n WHERE hash = (\n SELECT hash FROM searches WHERE hash=search.hash FOR UPDATE SKIP LOCKED\n )\n RETURNING * INTO cached_search\n ;\n END IF;\n\n IF cached_search IS NOT NULL THEN\n cached_search._where = search._where;\n cached_search.orderby = search.orderby;\n RETURN cached_search;\n END IF;\n RETURN search;\n\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE OR REPLACE FUNCTION search_fromhash(\n _hash text\n) RETURNS searches AS $$\n SELECT * FROM search_query((SELECT search FROM searches WHERE hash=_hash LIMIT 1));\n$$ LANGUAGE SQL STRICT;\n\nCREATE OR REPLACE FUNCTION search_rows(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN partitions text[] DEFAULT NULL,\n IN _limit int DEFAULT 10\n) RETURNS SETOF items AS $$\nDECLARE\n base_query text;\n query text;\n sdate timestamptz;\n edate timestamptz;\n n int;\n records_left int := _limit;\n timer timestamptz := clock_timestamp();\n full_timer timestamptz := clock_timestamp();\nBEGIN\nIF _where IS NULL OR trim(_where) = '' THEN\n _where = ' TRUE ';\nEND IF;\nRAISE NOTICE 'Getting chunks for % %', _where, _orderby;\n\nbase_query := $q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n LIMIT %L\n$q$;\n\nIF _orderby ILIKE 'datetime d%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 DESC LOOP\n RAISE NOTICE 'Running Query for % to %. %', sdate, edate, age_ms(full_timer);\n query := format(\n base_query,\n sdate,\n edate,\n _where,\n _orderby,\n records_left\n );\n RAISE DEBUG 'QUERY: %', query;\n timer := clock_timestamp();\n RETURN QUERY EXECUTE query;\n\n GET DIAGNOSTICS n = ROW_COUNT;\n records_left := records_left - n;\n RAISE NOTICE 'Returned %/% Rows From % to %. % to go. Time: %ms', n, _limit, sdate, edate, records_left, age_ms(timer);\n timer := clock_timestamp();\n IF records_left <= 0 THEN\n RAISE NOTICE 'SEARCH_ROWS TOOK %ms', age_ms(full_timer);\n RETURN;\n END IF;\n END LOOP;\nELSIF _orderby ILIKE 'datetime a%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 ASC LOOP\n RAISE NOTICE 'Running Query for % to %. %', sdate, edate, age_ms(full_timer);\n query := format(\n base_query,\n sdate,\n edate,\n _where,\n _orderby,\n records_left\n );\n RAISE DEBUG 'QUERY: %', query;\n timer := clock_timestamp();\n RETURN QUERY EXECUTE query;\n\n GET DIAGNOSTICS n = ROW_COUNT;\n records_left := records_left - n;\n RAISE NOTICE 'Returned %/% Rows From % to %. % to go. Time: %ms', n, _limit, sdate, edate, records_left, age_ms(timer);\n timer := clock_timestamp();\n IF records_left <= 0 THEN\n RAISE NOTICE 'SEARCH_ROWS TOOK %ms', age_ms(full_timer);\n RETURN;\n END IF;\n END LOOP;\nELSE\n query := format($q$\n SELECT * FROM items\n WHERE %s\n ORDER BY %s\n LIMIT %L\n $q$, _where, _orderby, _limit\n );\n RAISE DEBUG 'QUERY: %', query;\n timer := clock_timestamp();\n RETURN QUERY EXECUTE query;\n RAISE NOTICE 'FULL QUERY TOOK %ms', age_ms(timer);\nEND IF;\nRAISE NOTICE 'SEARCH_ROWS TOOK %ms', age_ms(full_timer);\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE UNLOGGED TABLE format_item_cache(\n id text,\n collection text,\n fields text,\n hydrated bool,\n output jsonb,\n lastused timestamptz DEFAULT now(),\n usecount int DEFAULT 1,\n timetoformat float,\n PRIMARY KEY (collection, id, fields, hydrated)\n);\nCREATE INDEX ON format_item_cache (lastused);\n\nCREATE OR REPLACE FUNCTION format_item(_item items, _fields jsonb DEFAULT '{}', _hydrated bool DEFAULT TRUE) RETURNS jsonb AS $$\nDECLARE\n cache bool := get_setting_bool('format_cache');\n _output jsonb := null;\n t timestamptz := clock_timestamp();\nBEGIN\n IF cache THEN\n SELECT output INTO _output FROM format_item_cache\n WHERE id=_item.id AND collection=_item.collection AND fields=_fields::text AND hydrated=_hydrated;\n END IF;\n IF _output IS NULL THEN\n IF _hydrated THEN\n _output := content_hydrate(_item, _fields);\n ELSE\n _output := content_nonhydrated(_item, _fields);\n END IF;\n END IF;\n IF cache THEN\n INSERT INTO format_item_cache (id, collection, fields, hydrated, output, timetoformat)\n VALUES (_item.id, _item.collection, _fields::text, _hydrated, _output, age_ms(t))\n ON CONFLICT(collection, id, fields, hydrated) DO\n UPDATE\n SET lastused=now(), usecount = format_item_cache.usecount + 1\n ;\n END IF;\n RETURN _output;\n\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION search(_search jsonb = '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n searches searches%ROWTYPE;\n _where text;\n orderby text;\n search_where search_wheres%ROWTYPE;\n total_count bigint;\n token record;\n token_prev boolean;\n token_item items%ROWTYPE;\n token_where text;\n full_where text;\n init_ts timestamptz := clock_timestamp();\n timer timestamptz := clock_timestamp();\n hydrate bool := NOT (_search->'conf'->>'nohydrate' IS NOT NULL AND (_search->'conf'->>'nohydrate')::boolean = true);\n prev text;\n next text;\n context jsonb;\n collection jsonb;\n out_records jsonb;\n out_len int;\n _limit int := coalesce((_search->>'limit')::int, 10);\n _querylimit int;\n _fields jsonb := coalesce(_search->'fields', '{}'::jsonb);\n has_prev boolean := FALSE;\n has_next boolean := FALSE;\n links jsonb := '[]'::jsonb;\n base_url text:= concat(rtrim(base_url(_search->'conf'),'/'));\nBEGIN\n searches := search_query(_search);\n _where := searches._where;\n orderby := searches.orderby;\n search_where := where_stats(_where);\n total_count := coalesce(search_where.total_count, search_where.estimated_count);\n RAISE NOTICE 'SEARCH:TOKEN: %', _search->>'token';\n token := get_token_record(_search->>'token');\n RAISE NOTICE '***TOKEN: %', token;\n _querylimit := _limit + 1;\n IF token IS NOT NULL THEN\n token_prev := token.prev;\n token_item := token.item;\n token_where := get_token_filter(_search->'sortby', token_item, token_prev, FALSE);\n RAISE DEBUG 'TOKEN_WHERE: % (%ms from search start)', token_where, age_ms(timer);\n IF token_prev THEN -- if we are using a prev token, we know has_next is true\n RAISE DEBUG 'There is a previous token, so automatically setting has_next to true';\n has_next := TRUE;\n orderby := sort_sqlorderby(_search, TRUE);\n ELSE\n RAISE DEBUG 'There is a next token, so automatically setting has_prev to true';\n has_prev := TRUE;\n\n END IF;\n ELSE -- if there was no token, we know there is no prev\n RAISE DEBUG 'There is no token, so we know there is no prev. setting has_prev to false';\n has_prev := FALSE;\n END IF;\n\n full_where := concat_ws(' AND ', _where, token_where);\n RAISE NOTICE 'FULL WHERE CLAUSE: %', full_where;\n RAISE NOTICE 'Time to get counts and build query %', age_ms(timer);\n timer := clock_timestamp();\n\n IF hydrate THEN\n RAISE NOTICE 'Getting hydrated data.';\n ELSE\n RAISE NOTICE 'Getting non-hydrated data.';\n END IF;\n RAISE NOTICE 'CACHE SET TO %', get_setting_bool('format_cache');\n RAISE NOTICE 'Time to set hydration/formatting %', age_ms(timer);\n timer := clock_timestamp();\n SELECT jsonb_agg(format_item(i, _fields, hydrate)) INTO out_records\n FROM search_rows(\n full_where,\n orderby,\n search_where.partitions,\n _querylimit\n ) as i;\n\n RAISE NOTICE 'Time to fetch rows %', age_ms(timer);\n timer := clock_timestamp();\n\n\n IF token_prev THEN\n out_records := flip_jsonb_array(out_records);\n END IF;\n\n RAISE NOTICE 'Query returned % records.', jsonb_array_length(out_records);\n RAISE DEBUG 'TOKEN: % %', token_item.id, token_item.collection;\n RAISE DEBUG 'RECORD_1: % %', out_records->0->>'id', out_records->0->>'collection';\n RAISE DEBUG 'RECORD-1: % %', out_records->-1->>'id', out_records->-1->>'collection';\n\n -- REMOVE records that were from our token\n IF out_records->0->>'id' = token_item.id AND out_records->0->>'collection' = token_item.collection THEN\n out_records := out_records - 0;\n ELSIF out_records->-1->>'id' = token_item.id AND out_records->-1->>'collection' = token_item.collection THEN\n out_records := out_records - -1;\n END IF;\n\n out_len := jsonb_array_length(out_records);\n\n IF out_len = _limit + 1 THEN\n IF token_prev THEN\n has_prev := TRUE;\n out_records := out_records - 0;\n ELSE\n has_next := TRUE;\n out_records := out_records - -1;\n END IF;\n END IF;\n\n\n links := links || jsonb_build_object(\n 'rel', 'root',\n 'type', 'application/json',\n 'href', base_url\n ) || jsonb_build_object(\n 'rel', 'self',\n 'type', 'application/json',\n 'href', concat(base_url, '/search')\n );\n\n IF has_next THEN\n next := concat(out_records->-1->>'collection', ':', out_records->-1->>'id');\n RAISE NOTICE 'HAS NEXT | %', next;\n links := links || jsonb_build_object(\n 'rel', 'next',\n 'type', 'application/geo+json',\n 'method', 'GET',\n 'href', concat(base_url, '/search?token=next:', next)\n );\n END IF;\n\n IF has_prev THEN\n prev := concat(out_records->0->>'collection', ':', out_records->0->>'id');\n RAISE NOTICE 'HAS PREV | %', prev;\n links := links || jsonb_build_object(\n 'rel', 'prev',\n 'type', 'application/geo+json',\n 'method', 'GET',\n 'href', concat(base_url, '/search?token=prev:', prev)\n );\n END IF;\n\n RAISE NOTICE 'Time to get prev/next %', age_ms(timer);\n timer := clock_timestamp();\n\n\n collection := jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb),\n 'links', links\n );\n\n\n\n IF context(_search->'conf') != 'off' THEN\n collection := collection || jsonb_strip_nulls(jsonb_build_object(\n 'numberMatched', total_count,\n 'numberReturned', coalesce(jsonb_array_length(out_records), 0)\n ));\n ELSE\n collection := collection || jsonb_strip_nulls(jsonb_build_object(\n 'numberReturned', coalesce(jsonb_array_length(out_records), 0)\n ));\n END IF;\n\n IF get_setting_bool('timing', _search->'conf') THEN\n collection = collection || jsonb_build_object('timing', age_ms(init_ts));\n END IF;\n\n RAISE NOTICE 'Time to build final json %', age_ms(timer);\n timer := clock_timestamp();\n\n RAISE NOTICE 'Total Time: %', age_ms(current_timestamp);\n RAISE NOTICE 'RETURNING % records. NEXT: %. PREV: %', collection->>'numberReturned', collection->>'next', collection->>'prev';\n RETURN collection;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION search_cursor(_search jsonb = '{}'::jsonb) RETURNS refcursor AS $$\nDECLARE\n curs refcursor;\n searches searches%ROWTYPE;\n _where text;\n _orderby text;\n q text;\n\nBEGIN\n searches := search_query(_search);\n _where := searches._where;\n _orderby := searches.orderby;\n\n OPEN curs FOR\n WITH p AS (\n SELECT * FROM partition_queries(_where, _orderby) p\n )\n SELECT\n CASE WHEN EXISTS (SELECT 1 FROM p) THEN\n (SELECT format($q$\n SELECT * FROM (\n %s\n ) total\n $q$,\n string_agg(\n format($q$ SELECT * FROM ( %s ) AS sub $q$, p),\n '\n UNION ALL\n '\n )\n ))\n ELSE NULL\n END FROM p;\n RETURN curs;\nEND;\n$$ LANGUAGE PLPGSQL;\nCREATE OR REPLACE VIEW collections_asitems AS\nSELECT\n id,\n geometry,\n 'collections' AS collection,\n datetime,\n end_datetime,\n jsonb_build_object(\n 'properties', content - '{links,assets,stac_version,stac_extensions}',\n 'links', content->'links',\n 'assets', content->'assets',\n 'stac_version', content->'stac_version',\n 'stac_extensions', content->'stac_extensions'\n ) AS content,\n content as collectionjson\nFROM collections;\n\n\nCREATE OR REPLACE FUNCTION collection_search_matched(\n IN _search jsonb DEFAULT '{}'::jsonb,\n OUT matched bigint\n) RETURNS bigint AS $$\nDECLARE\n _where text := stac_search_to_where(_search);\nBEGIN\n EXECUTE format(\n $query$\n SELECT\n count(*)\n FROM\n collections_asitems\n WHERE %s\n ;\n $query$,\n _where\n ) INTO matched;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION collection_search_rows(\n _search jsonb DEFAULT '{}'::jsonb\n) RETURNS SETOF jsonb AS $$\nDECLARE\n _where text := stac_search_to_where(_search);\n _limit int := coalesce((_search->>'limit')::int, 10);\n _fields jsonb := coalesce(_search->'fields', '{}'::jsonb);\n _orderby text;\n _offset int := COALESCE((_search->>'offset')::int, 0);\nBEGIN\n _orderby := sort_sqlorderby(\n jsonb_build_object(\n 'sortby',\n coalesce(\n _search->'sortby',\n '[{\"field\": \"id\", \"direction\": \"asc\"}]'::jsonb\n )\n )\n );\n RETURN QUERY EXECUTE format(\n $query$\n SELECT\n jsonb_fields(collectionjson, %L) as c\n FROM\n collections_asitems\n WHERE %s\n ORDER BY %s\n LIMIT %L\n OFFSET %L\n ;\n $query$,\n _fields,\n _where,\n _orderby,\n _limit,\n _offset\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION collection_search(\n _search jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\nDECLARE\n out_records jsonb;\n number_matched bigint := collection_search_matched(_search);\n number_returned bigint;\n _limit int := coalesce((_search->>'limit')::float::int, 10);\n _offset int := coalesce((_search->>'offset')::float::int, 0);\n links jsonb := '[]';\n ret jsonb;\n base_url text:= concat(rtrim(base_url(_search->'conf'),'/'), '/collections');\n prevoffset int;\n nextoffset int;\nBEGIN\n SELECT\n coalesce(jsonb_agg(c), '[]')\n INTO out_records\n FROM collection_search_rows(_search) c;\n\n number_returned := jsonb_array_length(out_records);\n RAISE DEBUG 'nm: %, nr: %, l:%, o:%', number_matched, number_returned, _limit, _offset;\n\n\n\n IF _limit <= number_matched AND number_matched > 0 THEN --need to have paging links\n nextoffset := least(_offset + _limit, number_matched - 1);\n prevoffset := greatest(_offset - _limit, 0);\n\n IF _offset > 0 THEN\n links := links || jsonb_build_object(\n 'rel', 'prev',\n 'type', 'application/json',\n 'method', 'GET' ,\n 'href', base_url,\n 'body', jsonb_build_object('offset', prevoffset),\n 'merge', TRUE\n );\n END IF;\n\n IF (_offset + _limit < number_matched) THEN\n links := links || jsonb_build_object(\n 'rel', 'next',\n 'type', 'application/json',\n 'method', 'GET' ,\n 'href', base_url,\n 'body', jsonb_build_object('offset', nextoffset),\n 'merge', TRUE\n );\n END IF;\n\n END IF;\n\n ret := jsonb_build_object(\n 'collections', out_records,\n 'numberMatched', number_matched,\n 'numberReturned', number_returned,\n 'links', links\n );\n RETURN ret;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION tileenvelope(zoom int, x int, y int) RETURNS geometry AS $$\nWITH t AS (\n SELECT\n 20037508.3427892 as merc_max,\n -20037508.3427892 as merc_min,\n (2 * 20037508.3427892) / (2 ^ zoom) as tile_size\n)\nSELECT st_makeenvelope(\n merc_min + (tile_size * x),\n merc_max - (tile_size * (y + 1)),\n merc_min + (tile_size * (x + 1)),\n merc_max - (tile_size * y),\n 3857\n) FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;DROP FUNCTION IF EXISTS mercgrid;\n\n\nCREATE OR REPLACE FUNCTION ftime() RETURNS interval as $$\nSELECT age(clock_timestamp(), transaction_timestamp());\n$$ LANGUAGE SQL;\nSET SEARCH_PATH to pgstac, public;\n\nDROP FUNCTION IF EXISTS geometrysearch;\nCREATE OR REPLACE FUNCTION geometrysearch(\n IN geom geometry,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE, -- Return as soon as the passed in geometry is full covered\n IN skipcovered boolean DEFAULT TRUE -- Skip any items that would show up completely under the previous items\n) RETURNS jsonb AS $$\nDECLARE\n search searches%ROWTYPE;\n curs refcursor;\n _where text;\n query text;\n iter_record items%ROWTYPE;\n out_records jsonb := '{}'::jsonb[];\n exit_flag boolean := FALSE;\n counter int := 1;\n scancounter int := 1;\n remaining_limit int := _scanlimit;\n tilearea float;\n unionedgeom geometry;\n clippedgeom geometry;\n unionedgeom_area float := 0;\n prev_area float := 0;\n excludes text[];\n includes text[];\n\nBEGIN\n DROP TABLE IF EXISTS pgstac_results;\n CREATE TEMP TABLE pgstac_results (content jsonb) ON COMMIT DROP;\n\n -- If the passed in geometry is not an area set exitwhenfull and skipcovered to false\n IF ST_GeometryType(geom) !~* 'polygon' THEN\n RAISE NOTICE 'GEOMETRY IS NOT AN AREA';\n skipcovered = FALSE;\n exitwhenfull = FALSE;\n END IF;\n\n -- If skipcovered is true then you will always want to exit when the passed in geometry is full\n IF skipcovered THEN\n exitwhenfull := TRUE;\n END IF;\n\n search := search_fromhash(queryhash);\n\n IF search IS NULL THEN\n RAISE EXCEPTION 'Search with Query Hash % Not Found', queryhash;\n END IF;\n\n tilearea := st_area(geom);\n _where := format('%s AND st_intersects(geometry, %L::geometry)', search._where, geom);\n\n\n FOR query IN SELECT * FROM partition_queries(_where, search.orderby) LOOP\n query := format('%s LIMIT %L', query, remaining_limit);\n RAISE NOTICE '%', query;\n OPEN curs FOR EXECUTE query;\n LOOP\n FETCH curs INTO iter_record;\n EXIT WHEN NOT FOUND;\n IF exitwhenfull OR skipcovered THEN -- If we are not using exitwhenfull or skipcovered, we do not need to do expensive geometry operations\n clippedgeom := st_intersection(geom, iter_record.geometry);\n\n IF unionedgeom IS NULL THEN\n unionedgeom := clippedgeom;\n ELSE\n unionedgeom := st_union(unionedgeom, clippedgeom);\n END IF;\n\n unionedgeom_area := st_area(unionedgeom);\n\n IF skipcovered AND prev_area = unionedgeom_area THEN\n scancounter := scancounter + 1;\n CONTINUE;\n END IF;\n\n prev_area := unionedgeom_area;\n\n RAISE NOTICE '% % % %', unionedgeom_area/tilearea, counter, scancounter, ftime();\n END IF;\n RAISE NOTICE '% %', iter_record, content_hydrate(iter_record, fields);\n INSERT INTO pgstac_results (content) VALUES (content_hydrate(iter_record, fields));\n\n IF counter >= _limit\n OR scancounter > _scanlimit\n OR ftime() > _timelimit\n OR (exitwhenfull AND unionedgeom_area >= tilearea)\n THEN\n exit_flag := TRUE;\n EXIT;\n END IF;\n counter := counter + 1;\n scancounter := scancounter + 1;\n\n END LOOP;\n CLOSE curs;\n EXIT WHEN exit_flag;\n remaining_limit := _scanlimit - scancounter;\n END LOOP;\n\n SELECT jsonb_agg(content) INTO out_records FROM pgstac_results WHERE content IS NOT NULL;\n\n RETURN jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb)\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDROP FUNCTION IF EXISTS geojsonsearch;\nCREATE OR REPLACE FUNCTION geojsonsearch(\n IN geojson jsonb,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_geomfromgeojson(geojson),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS xyzsearch;\nCREATE OR REPLACE FUNCTION xyzsearch(\n IN _x int,\n IN _y int,\n IN _z int,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_transform(tileenvelope(_z, _x, _y), 4326),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE PROCEDURE analyze_items() AS $$\nDECLARE\n q text;\n timeout_ts timestamptz;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n SELECT format('ANALYZE (VERBOSE, SKIP_LOCKED) %I;', relname) INTO q\n FROM pg_stat_user_tables\n WHERE relname like '_item%' AND (n_mod_since_analyze>0 OR last_analyze IS NULL) LIMIT 1;\n IF NOT FOUND THEN\n EXIT;\n END IF;\n RAISE NOTICE '%', q;\n EXECUTE q;\n COMMIT;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE PROCEDURE validate_constraints() AS $$\nDECLARE\n q text;\nBEGIN\n FOR q IN\n SELECT\n FORMAT(\n 'ALTER TABLE %I.%I VALIDATE CONSTRAINT %I;',\n nsp.nspname,\n cls.relname,\n con.conname\n )\n\n FROM pg_constraint AS con\n JOIN pg_class AS cls\n ON con.conrelid = cls.oid\n JOIN pg_namespace AS nsp\n ON cls.relnamespace = nsp.oid\n WHERE convalidated = FALSE AND contype in ('c','f')\n AND nsp.nspname = 'pgstac'\n LOOP\n RAISE NOTICE '%', q;\n PERFORM run_or_queue(q);\n COMMIT;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION collection_extent(_collection text, runupdate boolean default false) RETURNS jsonb AS $$\nDECLARE\n geom_extent geometry;\n mind timestamptz;\n maxd timestamptz;\n extent jsonb;\nBEGIN\n IF runupdate THEN\n PERFORM update_partition_stats_q(partition)\n FROM partitions_view WHERE collection=_collection;\n END IF;\n SELECT\n min(lower(dtrange)),\n max(upper(edtrange)),\n st_extent(spatial)\n INTO\n mind,\n maxd,\n geom_extent\n FROM partitions_view\n WHERE collection=_collection;\n\n IF geom_extent IS NOT NULL AND mind IS NOT NULL AND maxd IS NOT NULL THEN\n extent := jsonb_build_object(\n 'spatial', jsonb_build_object(\n 'bbox', to_jsonb(array[array[st_xmin(geom_extent), st_ymin(geom_extent), st_xmax(geom_extent), st_ymax(geom_extent)]])\n ),\n 'temporal', jsonb_build_object(\n 'interval', to_jsonb(array[array[mind, maxd]])\n )\n );\n RETURN extent;\n END IF;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('id', '{\"title\": \"Item ID\",\"description\": \"Item identifier\",\"$ref\": \"https://schemas.stacspec.org/v1.0.0/item-spec/json-schema/item.json#/definitions/core/allOf/2/properties/id\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('geometry', '{\"title\": \"Item Geometry\",\"description\": \"Item Geometry\",\"$ref\": \"https://geojson.org/schema/Feature.json\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('datetime','{\"description\": \"Datetime\",\"type\": \"string\",\"title\": \"Acquired\",\"format\": \"date-time\",\"pattern\": \"(\\\\+00:00|Z)$\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDELETE FROM queryables a USING queryables b\n WHERE a.name = b.name AND a.collection_ids IS NOT DISTINCT FROM b.collection_ids AND a.id > b.id;\n\n\nINSERT INTO pgstac_settings (name, value) VALUES\n ('context', 'off'),\n ('context_estimated_count', '100000'),\n ('context_estimated_cost', '100000'),\n ('context_stats_ttl', '1 day'),\n ('default_filter_lang', 'cql2-json'),\n ('additional_properties', 'true'),\n ('use_queue', 'false'),\n ('queue_timeout', '10 minutes'),\n ('update_collection_extent', 'false'),\n ('format_cache', 'false'),\n ('readonly', 'false')\nON CONFLICT DO NOTHING\n;\n\n\nINSERT INTO cql2_ops (op, template, types) VALUES\n ('eq', '%s = %s', NULL),\n ('neq', '%s != %s', NULL),\n ('ne', '%s != %s', NULL),\n ('!=', '%s != %s', NULL),\n ('<>', '%s != %s', NULL),\n ('lt', '%s < %s', NULL),\n ('lte', '%s <= %s', NULL),\n ('gt', '%s > %s', NULL),\n ('gte', '%s >= %s', NULL),\n ('le', '%s <= %s', NULL),\n ('ge', '%s >= %s', NULL),\n ('=', '%s = %s', NULL),\n ('<', '%s < %s', NULL),\n ('<=', '%s <= %s', NULL),\n ('>', '%s > %s', NULL),\n ('>=', '%s >= %s', NULL),\n ('like', '%s LIKE %s', NULL),\n ('ilike', '%s ILIKE %s', NULL),\n ('+', '%s + %s', NULL),\n ('-', '%s - %s', NULL),\n ('*', '%s * %s', NULL),\n ('/', '%s / %s', NULL),\n ('not', 'NOT (%s)', NULL),\n ('between', '%s BETWEEN %s AND %s', NULL),\n ('isnull', '%s IS NULL', NULL),\n ('upper', 'upper(%s)', NULL),\n ('lower', 'lower(%s)', NULL),\n ('casei', 'upper(%s)', NULL),\n ('accenti', 'unaccent(%s)', NULL)\nON CONFLICT (op) DO UPDATE\n SET\n template = EXCLUDED.template\n;\n\n\nALTER FUNCTION to_text COST 5000;\nALTER FUNCTION to_float COST 5000;\nALTER FUNCTION to_int COST 5000;\nALTER FUNCTION to_tstz COST 5000;\nALTER FUNCTION to_text_array COST 5000;\n\nALTER FUNCTION update_partition_stats SECURITY DEFINER;\nALTER FUNCTION partition_after_triggerfunc SECURITY DEFINER;\nALTER FUNCTION drop_table_constraints SECURITY DEFINER;\nALTER FUNCTION create_table_constraints SECURITY DEFINER;\nALTER FUNCTION check_partition SECURITY DEFINER;\nALTER FUNCTION repartition SECURITY DEFINER;\nALTER FUNCTION where_stats SECURITY DEFINER;\nALTER FUNCTION search_query SECURITY DEFINER;\nALTER FUNCTION format_item SECURITY DEFINER;\nALTER FUNCTION maintain_index SECURITY DEFINER;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nGRANT ALL ON SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac to pgstac_admin;\n\n-- pgstac_read role limited to using function apis\nGRANT EXECUTE ON FUNCTION search TO pgstac_read;\nGRANT EXECUTE ON FUNCTION search_query TO pgstac_read;\nGRANT EXECUTE ON FUNCTION item_by_id TO pgstac_read;\nGRANT EXECUTE ON FUNCTION get_item TO pgstac_read;\nGRANT SELECT ON ALL TABLES IN SCHEMA pgstac TO pgstac_read;\n\n\nGRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_ingest;\nGRANT USAGE ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_ingest;\n\nREVOKE ALL PRIVILEGES ON PROCEDURE run_queued_queries FROM public;\nGRANT ALL ON PROCEDURE run_queued_queries TO pgstac_admin;\n\nREVOKE ALL PRIVILEGES ON FUNCTION run_queued_queries_intransaction FROM public;\nGRANT ALL ON FUNCTION run_queued_queries_intransaction TO pgstac_admin;\n\nRESET ROLE;\n\nSET ROLE pgstac_ingest;\nSELECT update_partition_stats_q(partition) FROM partitions_view;\nSELECT set_version('0.9.3');\n" }, { "path": "src/pgstac/migrations/pgstac.0.9.4-0.9.5.sql", "content": "SET client_min_messages TO WARNING;\nSET SEARCH_PATH to pgstac, public;\nRESET ROLE;\nDO $$\nDECLARE\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='postgis') THEN\n CREATE EXTENSION IF NOT EXISTS postgis;\n END IF;\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='btree_gist') THEN\n CREATE EXTENSION IF NOT EXISTS btree_gist;\n END IF;\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='unaccent') THEN\n CREATE EXTENSION IF NOT EXISTS unaccent;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_admin;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_read;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_ingest;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n\nGRANT pgstac_admin TO current_user;\n\n-- Function to make sure pgstac_admin is the owner of items\nCREATE OR REPLACE FUNCTION pgstac_admin_owns() RETURNS VOID AS $$\nDECLARE\n f RECORD;\nBEGIN\n FOR f IN (\n SELECT\n concat(\n oid::regproc::text,\n '(',\n coalesce(pg_get_function_identity_arguments(oid),''),\n ')'\n ) AS name,\n CASE prokind WHEN 'f' THEN 'FUNCTION' WHEN 'p' THEN 'PROCEDURE' WHEN 'a' THEN 'AGGREGATE' END as typ\n FROM pg_proc\n WHERE\n pronamespace=to_regnamespace('pgstac')\n AND proowner != to_regrole('pgstac_admin')\n AND proname NOT LIKE 'pg_stat%'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n FOR f IN (\n SELECT\n oid::regclass::text as name,\n CASE relkind\n WHEN 'i' THEN 'INDEX'\n WHEN 'I' THEN 'INDEX'\n WHEN 'p' THEN 'TABLE'\n WHEN 'r' THEN 'TABLE'\n WHEN 'v' THEN 'VIEW'\n WHEN 'S' THEN 'SEQUENCE'\n ELSE NULL\n END as typ\n FROM pg_class\n WHERE relnamespace=to_regnamespace('pgstac') and relowner != to_regrole('pgstac_admin') AND relkind IN ('r','p','v','S') AND relname NOT LIKE 'pg_stat'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\nSELECT pgstac_admin_owns();\n\nCREATE SCHEMA IF NOT EXISTS pgstac AUTHORIZATION pgstac_admin;\n\nGRANT ALL ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_admin;\n\nALTER ROLE pgstac_admin SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_read SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_ingest SET SEARCH_PATH TO pgstac, public;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\n\nGRANT pgstac_read TO pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\n\nSET ROLE pgstac_admin;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\nRESET ROLE;\n\nSET ROLE pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\nRESET ROLE;\n\nSET SEARCH_PATH TO pgstac, public;\nSET ROLE pgstac_admin;\n\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS analyze_items;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS validate_constraints;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n-- Install these idempotently as migrations do not put them before trying to modify the collections table\n\n\nCREATE OR REPLACE FUNCTION collection_geom(content jsonb)\nRETURNS geometry AS $$\n WITH box AS (SELECT content->'extent'->'spatial'->'bbox'->0 as box)\n SELECT\n st_makeenvelope(\n (box->>0)::float,\n (box->>1)::float,\n (box->>2)::float,\n (box->>3)::float,\n 4326\n )\n FROM box;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION collection_datetime(content jsonb)\nRETURNS timestamptz AS $$\n SELECT\n CASE\n WHEN\n (content->'extent'->'temporal'->'interval'->0->>0) IS NULL\n THEN '-infinity'::timestamptz\n ELSE\n (content->'extent'->'temporal'->'interval'->0->>0)::timestamptz\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION collection_enddatetime(content jsonb)\nRETURNS timestamptz AS $$\n SELECT\n CASE\n WHEN\n (content->'extent'->'temporal'->'interval'->0->>1) IS NULL\n THEN 'infinity'::timestamptz\n ELSE\n (content->'extent'->'temporal'->'interval'->0->>1)::timestamptz\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n-- BEGIN migra calculated SQL\n-- END migra calculated SQL\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('id', '{\"title\": \"Item ID\",\"description\": \"Item identifier\",\"$ref\": \"https://schemas.stacspec.org/v1.0.0/item-spec/json-schema/item.json#/definitions/core/allOf/2/properties/id\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('geometry', '{\"title\": \"Item Geometry\",\"description\": \"Item Geometry\",\"$ref\": \"https://geojson.org/schema/Feature.json\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('datetime','{\"description\": \"Datetime\",\"type\": \"string\",\"title\": \"Acquired\",\"format\": \"date-time\",\"pattern\": \"(\\\\+00:00|Z)$\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDELETE FROM queryables a USING queryables b\n WHERE a.name = b.name AND a.collection_ids IS NOT DISTINCT FROM b.collection_ids AND a.id > b.id;\n\n\nINSERT INTO pgstac_settings (name, value) VALUES\n ('context', 'off'),\n ('context_estimated_count', '100000'),\n ('context_estimated_cost', '100000'),\n ('context_stats_ttl', '1 day'),\n ('default_filter_lang', 'cql2-json'),\n ('additional_properties', 'true'),\n ('use_queue', 'false'),\n ('queue_timeout', '10 minutes'),\n ('update_collection_extent', 'false'),\n ('format_cache', 'false'),\n ('readonly', 'false')\nON CONFLICT DO NOTHING\n;\n\n\nINSERT INTO cql2_ops (op, template, types) VALUES\n ('eq', '%s = %s', NULL),\n ('neq', '%s != %s', NULL),\n ('ne', '%s != %s', NULL),\n ('!=', '%s != %s', NULL),\n ('<>', '%s != %s', NULL),\n ('lt', '%s < %s', NULL),\n ('lte', '%s <= %s', NULL),\n ('gt', '%s > %s', NULL),\n ('gte', '%s >= %s', NULL),\n ('le', '%s <= %s', NULL),\n ('ge', '%s >= %s', NULL),\n ('=', '%s = %s', NULL),\n ('<', '%s < %s', NULL),\n ('<=', '%s <= %s', NULL),\n ('>', '%s > %s', NULL),\n ('>=', '%s >= %s', NULL),\n ('like', '%s LIKE %s', NULL),\n ('ilike', '%s ILIKE %s', NULL),\n ('+', '%s + %s', NULL),\n ('-', '%s - %s', NULL),\n ('*', '%s * %s', NULL),\n ('/', '%s / %s', NULL),\n ('not', 'NOT (%s)', NULL),\n ('between', '%s BETWEEN %s AND %s', NULL),\n ('isnull', '%s IS NULL', NULL),\n ('upper', 'upper(%s)', NULL),\n ('lower', 'lower(%s)', NULL),\n ('casei', 'upper(%s)', NULL),\n ('accenti', 'unaccent(%s)', NULL)\nON CONFLICT (op) DO UPDATE\n SET\n template = EXCLUDED.template\n;\n\n\nALTER FUNCTION to_text COST 5000;\nALTER FUNCTION to_float COST 5000;\nALTER FUNCTION to_int COST 5000;\nALTER FUNCTION to_tstz COST 5000;\nALTER FUNCTION to_text_array COST 5000;\n\nALTER FUNCTION update_partition_stats SECURITY DEFINER;\nALTER FUNCTION partition_after_triggerfunc SECURITY DEFINER;\nALTER FUNCTION drop_table_constraints SECURITY DEFINER;\nALTER FUNCTION create_table_constraints SECURITY DEFINER;\nALTER FUNCTION check_partition SECURITY DEFINER;\nALTER FUNCTION repartition SECURITY DEFINER;\nALTER FUNCTION where_stats SECURITY DEFINER;\nALTER FUNCTION search_query SECURITY DEFINER;\nALTER FUNCTION format_item SECURITY DEFINER;\nALTER FUNCTION maintain_index SECURITY DEFINER;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nGRANT ALL ON SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac to pgstac_admin;\n\n-- pgstac_read role limited to using function apis\nGRANT EXECUTE ON FUNCTION search TO pgstac_read;\nGRANT EXECUTE ON FUNCTION search_query TO pgstac_read;\nGRANT EXECUTE ON FUNCTION item_by_id TO pgstac_read;\nGRANT EXECUTE ON FUNCTION get_item TO pgstac_read;\nGRANT SELECT ON ALL TABLES IN SCHEMA pgstac TO pgstac_read;\n\n\nGRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_ingest;\nGRANT USAGE ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_ingest;\n\nREVOKE ALL PRIVILEGES ON PROCEDURE run_queued_queries FROM public;\nGRANT ALL ON PROCEDURE run_queued_queries TO pgstac_admin;\n\nREVOKE ALL PRIVILEGES ON FUNCTION run_queued_queries_intransaction FROM public;\nGRANT ALL ON FUNCTION run_queued_queries_intransaction TO pgstac_admin;\n\nRESET ROLE;\n\nSET ROLE pgstac_ingest;\nSELECT update_partition_stats_q(partition) FROM partitions_view;\nSELECT set_version('0.9.5');\n" }, { "path": "src/pgstac/migrations/pgstac.0.9.4.sql", "content": "RESET ROLE;\nDO $$\nDECLARE\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='postgis') THEN\n CREATE EXTENSION IF NOT EXISTS postgis;\n END IF;\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='btree_gist') THEN\n CREATE EXTENSION IF NOT EXISTS btree_gist;\n END IF;\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='unaccent') THEN\n CREATE EXTENSION IF NOT EXISTS unaccent;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_admin;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_read;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_ingest;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n\nGRANT pgstac_admin TO current_user;\n\n-- Function to make sure pgstac_admin is the owner of items\nCREATE OR REPLACE FUNCTION pgstac_admin_owns() RETURNS VOID AS $$\nDECLARE\n f RECORD;\nBEGIN\n FOR f IN (\n SELECT\n concat(\n oid::regproc::text,\n '(',\n coalesce(pg_get_function_identity_arguments(oid),''),\n ')'\n ) AS name,\n CASE prokind WHEN 'f' THEN 'FUNCTION' WHEN 'p' THEN 'PROCEDURE' WHEN 'a' THEN 'AGGREGATE' END as typ\n FROM pg_proc\n WHERE\n pronamespace=to_regnamespace('pgstac')\n AND proowner != to_regrole('pgstac_admin')\n AND proname NOT LIKE 'pg_stat%'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n FOR f IN (\n SELECT\n oid::regclass::text as name,\n CASE relkind\n WHEN 'i' THEN 'INDEX'\n WHEN 'I' THEN 'INDEX'\n WHEN 'p' THEN 'TABLE'\n WHEN 'r' THEN 'TABLE'\n WHEN 'v' THEN 'VIEW'\n WHEN 'S' THEN 'SEQUENCE'\n ELSE NULL\n END as typ\n FROM pg_class\n WHERE relnamespace=to_regnamespace('pgstac') and relowner != to_regrole('pgstac_admin') AND relkind IN ('r','p','v','S') AND relname NOT LIKE 'pg_stat'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\nSELECT pgstac_admin_owns();\n\nCREATE SCHEMA IF NOT EXISTS pgstac AUTHORIZATION pgstac_admin;\n\nGRANT ALL ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_admin;\n\nALTER ROLE pgstac_admin SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_read SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_ingest SET SEARCH_PATH TO pgstac, public;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\n\nGRANT pgstac_read TO pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\n\nSET ROLE pgstac_admin;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\nRESET ROLE;\n\nSET ROLE pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\nRESET ROLE;\n\nSET SEARCH_PATH TO pgstac, public;\nSET ROLE pgstac_admin;\n\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS analyze_items;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS validate_constraints;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n-- Install these idempotently as migrations do not put them before trying to modify the collections table\n\n\nCREATE OR REPLACE FUNCTION collection_geom(content jsonb)\nRETURNS geometry AS $$\n WITH box AS (SELECT content->'extent'->'spatial'->'bbox'->0 as box)\n SELECT\n st_makeenvelope(\n (box->>0)::float,\n (box->>1)::float,\n (box->>2)::float,\n (box->>3)::float,\n 4326\n )\n FROM box;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION collection_datetime(content jsonb)\nRETURNS timestamptz AS $$\n SELECT\n CASE\n WHEN\n (content->'extent'->'temporal'->'interval'->0->>0) IS NULL\n THEN '-infinity'::timestamptz\n ELSE\n (content->'extent'->'temporal'->'interval'->0->>0)::timestamptz\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION collection_enddatetime(content jsonb)\nRETURNS timestamptz AS $$\n SELECT\n CASE\n WHEN\n (content->'extent'->'temporal'->'interval'->0->>1) IS NULL\n THEN 'infinity'::timestamptz\n ELSE\n (content->'extent'->'temporal'->'interval'->0->>1)::timestamptz\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE TABLE IF NOT EXISTS migrations (\n version text PRIMARY KEY,\n datetime timestamptz DEFAULT clock_timestamp() NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION get_version() RETURNS text AS $$\n SELECT version FROM pgstac.migrations ORDER BY datetime DESC, version DESC LIMIT 1;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION set_version(text) RETURNS text AS $$\n INSERT INTO pgstac.migrations (version) VALUES ($1)\n ON CONFLICT DO NOTHING\n RETURNING version;\n$$ LANGUAGE SQL;\n\n\nCREATE TABLE IF NOT EXISTS pgstac_settings (\n name text PRIMARY KEY,\n value text NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION table_empty(text) RETURNS boolean AS $$\nDECLARE\n retval boolean;\nBEGIN\n EXECUTE format($q$\n SELECT NOT EXISTS (SELECT 1 FROM %I LIMIT 1)\n $q$,\n $1\n ) INTO retval;\n RETURN retval;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION get_setting(IN _setting text, IN conf jsonb DEFAULT NULL) RETURNS text AS $$\nSELECT COALESCE(\n nullif(conf->>_setting, ''),\n nullif(current_setting(concat('pgstac.',_setting), TRUE),''),\n nullif((SELECT value FROM pgstac.pgstac_settings WHERE name=_setting),'')\n);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_setting_bool(IN _setting text, IN conf jsonb DEFAULT NULL) RETURNS boolean AS $$\nSELECT COALESCE(\n nullif(conf->>_setting, ''),\n nullif(current_setting(concat('pgstac.',_setting), TRUE),''),\n nullif((SELECT value FROM pgstac.pgstac_settings WHERE name=_setting),''),\n 'FALSE'\n)::boolean;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION base_url(conf jsonb DEFAULT NULL) RETURNS text AS $$\n SELECT COALESCE(pgstac.get_setting('base_url', conf), '.');\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION additional_properties() RETURNS boolean AS $$\n SELECT pgstac.get_setting_bool('additional_properties');\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION readonly(conf jsonb DEFAULT NULL) RETURNS boolean AS $$\n SELECT pgstac.get_setting_bool('readonly', conf);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context(conf jsonb DEFAULT NULL) RETURNS text AS $$\n SELECT pgstac.get_setting('context', conf);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context_estimated_count(conf jsonb DEFAULT NULL) RETURNS int AS $$\n SELECT pgstac.get_setting('context_estimated_count', conf)::int;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_estimated_cost();\nCREATE OR REPLACE FUNCTION context_estimated_cost(conf jsonb DEFAULT NULL) RETURNS float AS $$\n SELECT pgstac.get_setting('context_estimated_cost', conf)::float;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_stats_ttl();\nCREATE OR REPLACE FUNCTION context_stats_ttl(conf jsonb DEFAULT NULL) RETURNS interval AS $$\n SELECT pgstac.get_setting('context_stats_ttl', conf)::interval;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION t2s(text) RETURNS text AS $$\n SELECT extract(epoch FROM $1::interval)::text || ' s';\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE STRICT;\n\nCREATE OR REPLACE FUNCTION age_ms(a timestamptz, b timestamptz DEFAULT clock_timestamp()) RETURNS float AS $$\n SELECT abs(extract(epoch from age(a,b)) * 1000);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION queue_timeout() RETURNS interval AS $$\n SELECT t2s(coalesce(\n get_setting('queue_timeout'),\n '1h'\n ))::interval;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION notice(VARIADIC text[]) RETURNS boolean AS $$\nDECLARE\ndebug boolean := current_setting('pgstac.debug', true);\nBEGIN\n IF debug THEN\n RAISE NOTICE 'NOTICE FROM FUNC: % >>>>> %', concat_ws(' | ', $1), clock_timestamp();\n RETURN TRUE;\n END IF;\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION empty_arr(ANYARRAY) RETURNS BOOLEAN AS $$\nSELECT CASE\n WHEN $1 IS NULL THEN TRUE\n WHEN cardinality($1)<1 THEN TRUE\nELSE FALSE\nEND;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_intersection(_a ANYARRAY, _b ANYARRAY) RETURNS ANYARRAY AS $$\n SELECT ARRAY ( SELECT unnest(_a) INTERSECT SELECT UNNEST(_b) );\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION array_map_ident(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_ident(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_map_literal(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_literal(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION array_reverse(anyarray) RETURNS anyarray AS $$\nSELECT ARRAY(\n SELECT $1[i]\n FROM generate_subscripts($1,1) AS s(i)\n ORDER BY i DESC\n);\n$$ LANGUAGE SQL STRICT IMMUTABLE;\n\nDROP TABLE IF EXISTS query_queue;\nCREATE TABLE query_queue (\n query text PRIMARY KEY,\n added timestamptz DEFAULT now()\n);\n\nDROP TABLE IF EXISTS query_queue_history;\nCREATE TABLE query_queue_history(\n query text,\n added timestamptz NOT NULL,\n finished timestamptz NOT NULL DEFAULT now(),\n error text\n);\n\nCREATE OR REPLACE PROCEDURE run_queued_queries() AS $$\nDECLARE\n qitem query_queue%ROWTYPE;\n timeout_ts timestamptz;\n error text;\n cnt int := 0;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n DELETE FROM query_queue WHERE query = (SELECT query FROM query_queue ORDER BY added DESC LIMIT 1 FOR UPDATE SKIP LOCKED) RETURNING * INTO qitem;\n IF NOT FOUND THEN\n EXIT;\n END IF;\n cnt := cnt + 1;\n BEGIN\n RAISE NOTICE 'RUNNING QUERY: %', qitem.query;\n EXECUTE qitem.query;\n EXCEPTION WHEN others THEN\n error := format('%s | %s', SQLERRM, SQLSTATE);\n END;\n INSERT INTO query_queue_history (query, added, finished, error)\n VALUES (qitem.query, qitem.added, clock_timestamp(), error);\n COMMIT;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION run_queued_queries_intransaction() RETURNS int AS $$\nDECLARE\n qitem query_queue%ROWTYPE;\n timeout_ts timestamptz;\n error text;\n cnt int := 0;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n DELETE FROM query_queue WHERE query = (SELECT query FROM query_queue ORDER BY added DESC LIMIT 1 FOR UPDATE SKIP LOCKED) RETURNING * INTO qitem;\n IF NOT FOUND THEN\n RETURN cnt;\n END IF;\n cnt := cnt + 1;\n BEGIN\n qitem.query := regexp_replace(qitem.query, 'CONCURRENTLY', '');\n RAISE NOTICE 'RUNNING QUERY: %', qitem.query;\n\n EXECUTE qitem.query;\n EXCEPTION WHEN others THEN\n error := format('%s | %s', SQLERRM, SQLSTATE);\n RAISE WARNING '%', error;\n END;\n INSERT INTO query_queue_history (query, added, finished, error)\n VALUES (qitem.query, qitem.added, clock_timestamp(), error);\n END LOOP;\n RETURN cnt;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION run_or_queue(query text) RETURNS VOID AS $$\nDECLARE\n use_queue text := COALESCE(get_setting('use_queue'), 'FALSE')::boolean;\nBEGIN\n IF get_setting_bool('debug') THEN\n RAISE NOTICE '%', query;\n END IF;\n IF use_queue THEN\n INSERT INTO query_queue (query) VALUES (query) ON CONFLICT DO NOTHING;\n ELSE\n EXECUTE query;\n END IF;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n\nDROP FUNCTION IF EXISTS check_pgstac_settings;\nCREATE OR REPLACE FUNCTION check_pgstac_settings(_sysmem text DEFAULT NULL) RETURNS VOID AS $$\nDECLARE\n settingval text;\n sysmem bigint := pg_size_bytes(_sysmem);\n effective_cache_size bigint := pg_size_bytes(current_setting('effective_cache_size', TRUE));\n shared_buffers bigint := pg_size_bytes(current_setting('shared_buffers', TRUE));\n work_mem bigint := pg_size_bytes(current_setting('work_mem', TRUE));\n max_connections int := current_setting('max_connections', TRUE);\n maintenance_work_mem bigint := pg_size_bytes(current_setting('maintenance_work_mem', TRUE));\n seq_page_cost float := current_setting('seq_page_cost', TRUE);\n random_page_cost float := current_setting('random_page_cost', TRUE);\n temp_buffers bigint := pg_size_bytes(current_setting('temp_buffers', TRUE));\n r record;\nBEGIN\n IF _sysmem IS NULL THEN\n RAISE NOTICE 'Call function with the size of your system memory `SELECT check_pgstac_settings(''4GB'')` to get pg system setting recommendations.';\n ELSE\n IF effective_cache_size < (sysmem * 0.5) THEN\n RAISE WARNING 'effective_cache_size of % is set low for a system with %. Recomended value between % and %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.5), pg_size_pretty(sysmem * 0.75);\n ELSIF effective_cache_size > (sysmem * 0.75) THEN\n RAISE WARNING 'effective_cache_size of % is set high for a system with %. Recomended value between % and %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.5), pg_size_pretty(sysmem * 0.75);\n ELSE\n RAISE NOTICE 'effective_cache_size of % is set appropriately for a system with %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem);\n END IF;\n\n IF shared_buffers < (sysmem * 0.2) THEN\n RAISE WARNING 'shared_buffers of % is set low for a system with %. Recomended value between % and %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.2), pg_size_pretty(sysmem * 0.3);\n ELSIF shared_buffers > (sysmem * 0.3) THEN\n RAISE WARNING 'shared_buffers of % is set high for a system with %. Recomended value between % and %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.2), pg_size_pretty(sysmem * 0.3);\n ELSE\n RAISE NOTICE 'shared_buffers of % is set appropriately for a system with %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem);\n END IF;\n shared_buffers = sysmem * 0.3;\n IF maintenance_work_mem < (sysmem * 0.2) THEN\n RAISE WARNING 'maintenance_work_mem of % is set low for shared_buffers of %. Recomended value between % and %', pg_size_pretty(maintenance_work_mem), pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers * 0.2), pg_size_pretty(shared_buffers * 0.3);\n ELSIF maintenance_work_mem > (shared_buffers * 0.3) THEN\n RAISE WARNING 'maintenance_work_mem of % is set high for shared_buffers of %. Recomended value between % and %', pg_size_pretty(maintenance_work_mem), pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers * 0.2), pg_size_pretty(shared_buffers * 0.3);\n ELSE\n RAISE NOTICE 'maintenance_work_mem of % is set appropriately for shared_buffers of %', pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers);\n END IF;\n\n IF work_mem * max_connections > shared_buffers THEN\n RAISE WARNING 'work_mem setting of % is set high for % max_connections please reduce work_mem to % or decrease max_connections to %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers/max_connections), floor(shared_buffers/work_mem);\n ELSIF work_mem * max_connections < (shared_buffers * 0.75) THEN\n RAISE WARNING 'work_mem setting of % is set low for % max_connections you may consider raising work_mem to % or increasing max_connections to %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers/max_connections), floor(shared_buffers/work_mem);\n ELSE\n RAISE NOTICE 'work_mem setting of % and max_connections of % are adequate for shared_buffers of %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers);\n END IF;\n\n IF random_page_cost / seq_page_cost != 1.1 THEN\n RAISE WARNING 'random_page_cost (%) /seq_page_cost (%) should be set to 1.1 for SSD. Change random_page_cost to %', random_page_cost, seq_page_cost, 1.1 * seq_page_cost;\n ELSE\n RAISE NOTICE 'random_page_cost and seq_page_cost set appropriately for SSD';\n END IF;\n\n IF temp_buffers < greatest(pg_size_bytes('128MB'),(maintenance_work_mem / 2)) THEN\n RAISE WARNING 'pgstac makes heavy use of temp tables, consider raising temp_buffers from % to %', pg_size_pretty(temp_buffers), greatest('128MB', pg_size_pretty((shared_buffers / 16)));\n END IF;\n END IF;\n\n RAISE NOTICE 'VALUES FOR PGSTAC VARIABLES';\n RAISE NOTICE 'These can be set either as GUC system variables or by setting in the pgstac_settings table.';\n\n FOR r IN SELECT name, get_setting(name) as setting, CASE WHEN current_setting(concat('pgstac.',name), TRUE) IS NOT NULL THEN concat('pgstac.',name, ' GUC') WHEN value IS NOT NULL THEN 'pgstac_settings table' ELSE 'Not Set' END as loc FROM pgstac_settings LOOP\n RAISE NOTICE '% is set to % from the %', r.name, r.setting, r.loc;\n END LOOP;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pg_cron';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider intalling pg_cron which can be used to automate tasks';\n ELSE\n RAISE NOTICE 'pg_cron % is installed', settingval;\n END IF;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pgstattuple';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider installing the pgstattuple extension which can be used to help maintain tables and indexes.';\n ELSE\n RAISE NOTICE 'pgstattuple % is installed', settingval;\n END IF;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pg_stat_statements';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider installing the pg_stat_statements extension which is very helpful for tracking the types of queries on the system';\n ELSE\n RAISE NOTICE 'pg_stat_statements % is installed', settingval;\n IF current_setting('pg_stat_statements.track_statements', TRUE) IS DISTINCT FROM 'all' THEN\n RAISE WARNING 'SET pg_stat_statements.track_statements TO ''all''; --In order to track statements within functions.';\n END IF;\n END IF;\n\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac, public SET CLIENT_MIN_MESSAGES TO NOTICE;\nCREATE OR REPLACE FUNCTION to_int(jsonb) RETURNS int AS $$\n SELECT floor(($1->>0)::float)::int;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION to_float(jsonb) RETURNS float AS $$\n SELECT ($1->>0)::float;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION to_tstz(jsonb) RETURNS timestamptz AS $$\n SELECT ($1->>0)::timestamptz;\n$$ LANGUAGE SQL IMMUTABLE STRICT SET TIME ZONE 'UTC' COST 5000 PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION to_text(jsonb) RETURNS text AS $$\n SELECT CASE WHEN jsonb_typeof($1) IN ('array','object') THEN $1::text ELSE $1->>0 END;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION to_text_array(jsonb) RETURNS text[] AS $$\n SELECT\n CASE jsonb_typeof($1)\n WHEN 'array' THEN ARRAY(SELECT jsonb_array_elements_text($1))\n ELSE ARRAY[$1->>0]\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION bbox_geom(_bbox jsonb) RETURNS geometry AS $$\nSELECT CASE jsonb_array_length(_bbox)\n WHEN 4 THEN\n ST_SetSRID(ST_MakeEnvelope(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float,\n (_bbox->>3)::float\n ),4326)\n WHEN 6 THEN\n ST_SetSRID(ST_3DMakeBox(\n ST_MakePoint(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float\n ),\n ST_MakePoint(\n (_bbox->>3)::float,\n (_bbox->>4)::float,\n (_bbox->>5)::float\n )\n ),4326)\n ELSE null END;\n;\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION geom_bbox(_geom geometry) RETURNS jsonb AS $$\n SELECT jsonb_build_array(\n st_xmin(_geom),\n st_ymin(_geom),\n st_xmax(_geom),\n st_ymax(_geom)\n );\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION flip_jsonb_array(j jsonb) RETURNS jsonb AS $$\n SELECT jsonb_agg(value) FROM (SELECT value FROM jsonb_array_elements(j) WITH ORDINALITY ORDER BY ordinality DESC) as t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION explode_dotpaths(j jsonb) RETURNS SETOF text[] AS $$\n SELECT string_to_array(p, '.') as e FROM jsonb_array_elements_text(j) p;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION explode_dotpaths_recurse(IN j jsonb) RETURNS SETOF text[] AS $$\n WITH RECURSIVE t AS (\n SELECT e FROM explode_dotpaths(j) e\n UNION ALL\n SELECT e[1:cardinality(e)-1]\n FROM t\n WHERE cardinality(e)>1\n ) SELECT e FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION jsonb_set_nested(j jsonb, path text[], val jsonb) RETURNS jsonb AS $$\nDECLARE\nBEGIN\n IF cardinality(path) > 1 THEN\n FOR i IN 1..(cardinality(path)-1) LOOP\n IF j #> path[:i] IS NULL THEN\n j := jsonb_set_lax(j, path[:i], '{}', TRUE);\n END IF;\n END LOOP;\n END IF;\n RETURN jsonb_set_lax(j, path, val, true);\n\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\n\n\nCREATE OR REPLACE FUNCTION jsonb_include(j jsonb, f jsonb) RETURNS jsonb AS $$\nDECLARE\n includes jsonb := f-> 'include';\n outj jsonb := '{}'::jsonb;\n path text[];\nBEGIN\n IF\n includes IS NULL\n OR jsonb_array_length(includes) = 0\n THEN\n RETURN j;\n ELSE\n includes := includes || '[\"id\",\"collection\"]'::jsonb;\n FOR path IN SELECT explode_dotpaths(includes) LOOP\n outj := jsonb_set_nested(outj, path, j #> path);\n END LOOP;\n END IF;\n RETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_exclude(j jsonb, f jsonb) RETURNS jsonb AS $$\nDECLARE\n excludes jsonb := f-> 'exclude';\n outj jsonb := j;\n path text[];\nBEGIN\n IF\n excludes IS NULL\n OR jsonb_array_length(excludes) = 0\n THEN\n RETURN j;\n ELSE\n FOR path IN SELECT explode_dotpaths(excludes) LOOP\n outj := outj #- path;\n END LOOP;\n END IF;\n RETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_fields(j jsonb, f jsonb DEFAULT '{\"fields\":[]}') RETURNS jsonb AS $$\n SELECT jsonb_exclude(jsonb_include(j, f), f);\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION merge_jsonb(_a jsonb, _b jsonb) RETURNS jsonb AS $$\n SELECT\n CASE\n WHEN _a = '\"𒍟※\"'::jsonb THEN NULL\n WHEN _a IS NULL OR jsonb_typeof(_a) = 'null' THEN _b\n WHEN jsonb_typeof(_a) = 'object' AND jsonb_typeof(_b) = 'object' THEN\n (\n SELECT\n jsonb_strip_nulls(\n jsonb_object_agg(\n key,\n merge_jsonb(a.value, b.value)\n )\n )\n FROM\n jsonb_each(coalesce(_a,'{}'::jsonb)) as a\n FULL JOIN\n jsonb_each(coalesce(_b,'{}'::jsonb)) as b\n USING (key)\n )\n WHEN\n jsonb_typeof(_a) = 'array'\n AND jsonb_typeof(_b) = 'array'\n AND jsonb_array_length(_a) = jsonb_array_length(_b)\n THEN\n (\n SELECT jsonb_agg(m) FROM\n ( SELECT\n merge_jsonb(\n jsonb_array_elements(_a),\n jsonb_array_elements(_b)\n ) as m\n ) as l\n )\n ELSE _a\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION strip_jsonb(_a jsonb, _b jsonb) RETURNS jsonb AS $$\n SELECT\n CASE\n\n WHEN (_a IS NULL OR jsonb_typeof(_a) = 'null') AND _b IS NOT NULL AND jsonb_typeof(_b) != 'null' THEN '\"𒍟※\"'::jsonb\n WHEN _b IS NULL OR jsonb_typeof(_a) = 'null' THEN _a\n WHEN _a = _b AND jsonb_typeof(_a) = 'object' THEN '{}'::jsonb\n WHEN _a = _b THEN NULL\n WHEN jsonb_typeof(_a) = 'object' AND jsonb_typeof(_b) = 'object' THEN\n (\n SELECT\n jsonb_strip_nulls(\n jsonb_object_agg(\n key,\n strip_jsonb(a.value, b.value)\n )\n )\n FROM\n jsonb_each(_a) as a\n FULL JOIN\n jsonb_each(_b) as b\n USING (key)\n )\n WHEN\n jsonb_typeof(_a) = 'array'\n AND jsonb_typeof(_b) = 'array'\n AND jsonb_array_length(_a) = jsonb_array_length(_b)\n THEN\n (\n SELECT jsonb_agg(m) FROM\n ( SELECT\n strip_jsonb(\n jsonb_array_elements(_a),\n jsonb_array_elements(_b)\n ) as m\n ) as l\n )\n ELSE _a\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION nullif_jsonbnullempty(j jsonb) RETURNS jsonb AS $$\n SELECT nullif(nullif(nullif(j,'null'::jsonb),'{}'::jsonb),'[]'::jsonb);\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION jsonb_array_unique(j jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(jsonb_agg(DISTINCT a)) v FROM jsonb_array_elements(j) a;\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_concat_ignorenull(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT coalesce(a,'[]'::jsonb) || coalesce(b,'[]'::jsonb);\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_least(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(least(nullif_jsonbnullempty(a), nullif_jsonbnullempty(b)));\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_greatest(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(greatest(a, b));\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION first_notnull_sfunc(anyelement, anyelement) RETURNS anyelement AS $$\n SELECT COALESCE($1,$2);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE AGGREGATE first_notnull(anyelement)(\n SFUNC = first_notnull_sfunc,\n STYPE = anyelement\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_array_unique_merge(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_concat_ignorenull,\n FINALFUNC = jsonb_array_unique\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_min(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_least\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_max(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_greatest\n);\n/* looks for a geometry in a stac item first from geometry and falling back to bbox */\nCREATE OR REPLACE FUNCTION stac_geom(value jsonb) RETURNS geometry AS $$\nSELECT\n CASE\n WHEN value ? 'intersects' THEN\n ST_GeomFromGeoJSON(value->>'intersects')\n WHEN value ? 'geometry' THEN\n ST_GeomFromGeoJSON(value->>'geometry')\n WHEN value ? 'bbox' THEN\n pgstac.bbox_geom(value->'bbox')\n ELSE NULL\n END as geometry\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION stac_daterange(\n value jsonb\n) RETURNS tstzrange AS $$\nDECLARE\n props jsonb := value;\n dt timestamptz;\n edt timestamptz;\nBEGIN\n IF props ? 'properties' THEN\n props := props->'properties';\n END IF;\n IF\n props ? 'start_datetime'\n AND props->>'start_datetime' IS NOT NULL\n AND props ? 'end_datetime'\n AND props->>'end_datetime' IS NOT NULL\n THEN\n dt := props->>'start_datetime';\n edt := props->>'end_datetime';\n IF dt > edt THEN\n RAISE EXCEPTION 'start_datetime must be < end_datetime';\n END IF;\n ELSE\n dt := props->>'datetime';\n edt := props->>'datetime';\n END IF;\n IF dt is NULL OR edt IS NULL THEN\n RAISE NOTICE 'DT: %, EDT: %', dt, edt;\n RAISE EXCEPTION 'Either datetime (%) or both start_datetime (%) and end_datetime (%) must be set.', props->>'datetime',props->>'start_datetime',props->>'end_datetime';\n END IF;\n RETURN tstzrange(dt, edt, '[]');\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_datetime(value jsonb) RETURNS timestamptz AS $$\n SELECT lower(stac_daterange(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_end_datetime(value jsonb) RETURNS timestamptz AS $$\n SELECT upper(stac_daterange(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE TABLE IF NOT EXISTS stac_extensions(\n url text PRIMARY KEY,\n content jsonb\n);\nCREATE OR REPLACE FUNCTION collection_base_item(content jsonb) RETURNS jsonb AS $$\n SELECT jsonb_build_object(\n 'type', 'Feature',\n 'stac_version', content->'stac_version',\n 'assets', content->'item_assets',\n 'collection', content->'id'\n );\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE TABLE IF NOT EXISTS collections (\n key bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,\n id text GENERATED ALWAYS AS (content->>'id') STORED UNIQUE NOT NULL,\n content JSONB NOT NULL,\n base_item jsonb GENERATED ALWAYS AS (pgstac.collection_base_item(content)) STORED,\n geometry geometry GENERATED ALWAYS AS (pgstac.collection_geom(content)) STORED,\n datetime timestamptz GENERATED ALWAYS AS (pgstac.collection_datetime(content)) STORED,\n end_datetime timestamptz GENERATED ALWAYS AS (pgstac.collection_enddatetime(content)) STORED,\n private jsonb,\n partition_trunc text CHECK (partition_trunc IN ('year', 'month'))\n);\n\nCREATE OR REPLACE FUNCTION collection_base_item(cid text) RETURNS jsonb AS $$\n SELECT pgstac.collection_base_item(content) FROM pgstac.collections WHERE id = cid LIMIT 1;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION create_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION update_collection(data jsonb) RETURNS VOID AS $$\nDECLARE\n out collections%ROWTYPE;\nBEGIN\n UPDATE collections SET content=data WHERE id = data->>'id' RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ON CONFLICT (id) DO\n UPDATE\n SET content=EXCLUDED.content\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION delete_collection(_id text) RETURNS VOID AS $$\nDECLARE\n out collections%ROWTYPE;\nBEGIN\n DELETE FROM collections WHERE id = _id RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION get_collection(id text) RETURNS jsonb AS $$\n SELECT content FROM collections\n WHERE id=$1\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION all_collections() RETURNS jsonb AS $$\n SELECT coalesce(jsonb_agg(content), '[]'::jsonb) FROM collections;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION collection_delete_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n collection_base_partition text := concat('_items_', OLD.key);\nBEGIN\n EXECUTE format($q$\n DELETE FROM partition_stats WHERE partition IN (\n SELECT partition FROM partition_sys_meta\n WHERE collection=%L\n );\n DROP TABLE IF EXISTS %I CASCADE;\n $q$,\n OLD.id,\n collection_base_partition\n );\n RETURN OLD;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER collection_delete_trigger BEFORE DELETE ON collections\nFOR EACH ROW EXECUTE FUNCTION collection_delete_trigger_func();\nCREATE OR REPLACE FUNCTION queryable_signature(n text, c text[]) RETURNS text AS $$\n SELECT concat(n, c);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TABLE queryables (\n id bigint GENERATED ALWAYS AS identity PRIMARY KEY,\n name text NOT NULL,\n collection_ids text[], -- used to determine what partitions to create indexes on\n definition jsonb,\n property_path text,\n property_wrapper text,\n property_index_type text\n);\nCREATE INDEX queryables_name_idx ON queryables (name);\nCREATE INDEX queryables_collection_idx ON queryables USING GIN (collection_ids);\nCREATE INDEX queryables_property_wrapper_idx ON queryables (property_wrapper);\n\nCREATE OR REPLACE FUNCTION pgstac.queryables_constraint_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n allcollections text[];\nBEGIN\n RAISE NOTICE 'Making sure that name/collection is unique for queryables %', NEW;\n IF NEW.collection_ids IS NOT NULL THEN\n IF EXISTS (\n SELECT 1\n FROM unnest(NEW.collection_ids) c\n LEFT JOIN\n collections\n ON (collections.id = c)\n WHERE collections.id IS NULL\n ) THEN\n RAISE foreign_key_violation USING MESSAGE = format(\n 'One or more collections in %s do not exist.', NEW.collection_ids\n );\n RETURN NULL;\n END IF;\n END IF;\n IF TG_OP = 'INSERT' THEN\n IF EXISTS (\n SELECT 1 FROM queryables q\n WHERE\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n )\n ) THEN\n RAISE unique_violation USING MESSAGE = format(\n 'There is already a queryable for %s for a collection in %s: %s',\n NEW.name,\n NEW.collection_ids,\n\t\t\t\t(SELECT json_agg(row_to_json(q)) FROM queryables q WHERE\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n ))\n );\n RETURN NULL;\n END IF;\n END IF;\n IF TG_OP = 'UPDATE' THEN\n IF EXISTS (\n SELECT 1 FROM queryables q\n WHERE\n q.id != NEW.id\n AND\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n )\n ) THEN\n RAISE unique_violation\n USING MESSAGE = format(\n 'There is already a queryable for %s for a collection in %s',\n NEW.name,\n NEW.collection_ids\n );\n RETURN NULL;\n END IF;\n END IF;\n\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER queryables_constraint_insert_trigger\nBEFORE INSERT ON queryables\nFOR EACH ROW EXECUTE PROCEDURE queryables_constraint_triggerfunc();\n\nCREATE TRIGGER queryables_constraint_update_trigger\nBEFORE UPDATE ON queryables\nFOR EACH ROW\nWHEN (NEW.name = OLD.name AND NEW.collection_ids IS DISTINCT FROM OLD.collection_ids)\nEXECUTE PROCEDURE queryables_constraint_triggerfunc();\n\n\nCREATE OR REPLACE FUNCTION array_to_path(arr text[]) RETURNS text AS $$\n SELECT string_agg(\n quote_literal(v),\n '->'\n ) FROM unnest(arr) v;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\n\n\nCREATE OR REPLACE FUNCTION queryable(\n IN dotpath text,\n OUT path text,\n OUT expression text,\n OUT wrapper text,\n OUT nulled_wrapper text\n) AS $$\nDECLARE\n q RECORD;\n path_elements text[];\nBEGIN\n dotpath := replace(dotpath, 'properties.', '');\n IF dotpath = 'start_datetime' THEN\n dotpath := 'datetime';\n END IF;\n IF dotpath IN ('id', 'geometry', 'datetime', 'end_datetime', 'collection') THEN\n path := dotpath;\n expression := dotpath;\n wrapper := NULL;\n RETURN;\n END IF;\n\n SELECT * INTO q FROM queryables\n WHERE\n name=dotpath\n OR name = 'properties.' || dotpath\n OR name = replace(dotpath, 'properties.', '')\n ;\n IF q.property_wrapper IS NULL THEN\n IF q.definition->>'type' = 'number' THEN\n wrapper := 'to_float';\n nulled_wrapper := wrapper;\n ELSIF q.definition->>'format' = 'date-time' THEN\n wrapper := 'to_tstz';\n nulled_wrapper := wrapper;\n ELSE\n nulled_wrapper := NULL;\n wrapper := 'to_text';\n END IF;\n ELSE\n wrapper := q.property_wrapper;\n nulled_wrapper := wrapper;\n END IF;\n IF q.property_path IS NOT NULL THEN\n path := q.property_path;\n ELSE\n path_elements := string_to_array(dotpath, '.');\n IF path_elements[1] IN ('links', 'assets', 'stac_version', 'stac_extensions') THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSIF path_elements[1] = 'properties' THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSE\n path := format($F$content->'properties'->%s$F$, array_to_path(path_elements));\n END IF;\n END IF;\n expression := format('%I(%s)', wrapper, path);\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\nCREATE OR REPLACE FUNCTION unnest_collection(collection_ids text[] DEFAULT NULL) RETURNS SETOF text AS $$\n DECLARE\n BEGIN\n IF collection_ids IS NULL THEN\n RETURN QUERY SELECT id FROM collections;\n END IF;\n RETURN QUERY SELECT unnest(collection_ids);\n END;\n$$ LANGUAGE PLPGSQL STABLE;\n\nCREATE OR REPLACE FUNCTION normalize_indexdef(def text) RETURNS text AS $$\nDECLARE\nBEGIN\n def := btrim(def, ' \\n\\t');\n\tdef := regexp_replace(def, '^CREATE (UNIQUE )?INDEX ([^ ]* )?ON (ONLY )?([^ ]* )?', '', 'i');\n RETURN def;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION indexdef(q queryables) RETURNS text AS $$\n DECLARE\n out text;\n BEGIN\n IF q.name = 'id' THEN\n out := 'CREATE UNIQUE INDEX ON %I USING btree (id)';\n ELSIF q.name = 'datetime' THEN\n out := 'CREATE INDEX ON %I USING btree (datetime DESC, end_datetime)';\n ELSIF q.name = 'geometry' THEN\n out := 'CREATE INDEX ON %I USING gist (geometry)';\n ELSE\n out := format($q$CREATE INDEX ON %%I USING %s (%s(((content -> 'properties'::text) -> %L::text)))$q$,\n lower(COALESCE(q.property_index_type, 'BTREE')),\n lower(COALESCE(q.property_wrapper, 'to_text')),\n q.name\n );\n END IF;\n RETURN btrim(out, ' \\n\\t');\n END;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nDROP VIEW IF EXISTS pgstac_indexes;\nCREATE VIEW pgstac_indexes AS\nSELECT\n i.schemaname,\n i.tablename,\n i.indexname,\n regexp_replace(btrim(replace(replace(indexdef, i.indexname, ''),'pgstac.',''),' \\t\\n'), '[ ]+', ' ', 'g') as idx,\n COALESCE(\n (regexp_match(indexdef, '\\(([a-zA-Z]+)\\)'))[1],\n (regexp_match(indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_-]+)''::text'))[1],\n CASE WHEN indexdef ~* '\\(datetime desc, end_datetime\\)' THEN 'datetime' ELSE NULL END\n ) AS field,\n pg_table_size(i.indexname::text) as index_size,\n pg_size_pretty(pg_table_size(i.indexname::text)) as index_size_pretty\nFROM\n pg_indexes i\nWHERE i.schemaname='pgstac' and i.tablename ~ '_items_' AND indexdef !~* ' only ';\n\nDROP VIEW IF EXISTS pgstac_index_stats;\nCREATE VIEW pgstac_indexes_stats AS\nSELECT\n i.schemaname,\n i.tablename,\n i.indexname,\n indexdef,\n COALESCE(\n (regexp_match(indexdef, '\\(([a-zA-Z]+)\\)'))[1],\n (regexp_match(indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_]+)''::text'))[1],\n CASE WHEN indexdef ~* '\\(datetime desc, end_datetime\\)' THEN 'datetime_end_datetime' ELSE NULL END\n ) AS field,\n pg_table_size(i.indexname::text) as index_size,\n pg_size_pretty(pg_table_size(i.indexname::text)) as index_size_pretty,\n n_distinct,\n most_common_vals::text::text[],\n most_common_freqs::text::text[],\n histogram_bounds::text::text[],\n correlation\nFROM\n pg_indexes i\n LEFT JOIN pg_stats s ON (s.tablename = i.indexname)\nWHERE i.schemaname='pgstac' and i.tablename ~ '_items_';\n\nCREATE OR REPLACE FUNCTION queryable_indexes(\n IN treeroot text DEFAULT 'items',\n IN changes boolean DEFAULT FALSE,\n OUT collection text,\n OUT partition text,\n OUT field text,\n OUT indexname text,\n OUT existing_idx text,\n OUT queryable_idx text\n) RETURNS SETOF RECORD AS $$\nWITH p AS (\n SELECT\n relid::text as partition,\n replace(replace(\n CASE\n WHEN parentrelid::regclass::text='items' THEN pg_get_expr(c.relpartbound, c.oid)\n ELSE pg_get_expr(parent.relpartbound, parent.oid)\n END,\n 'FOR VALUES IN (''',''), ''')',\n ''\n ) AS collection\n FROM pg_partition_tree(treeroot)\n JOIN pg_class c ON (relid::regclass = c.oid)\n JOIN pg_class parent ON (parentrelid::regclass = parent.oid AND isleaf)\n ), i AS (\n SELECT\n partition,\n indexname,\n regexp_replace(btrim(replace(replace(indexdef, indexname, ''),'pgstac.',''),' \\t\\n'), '[ ]+', ' ', 'g') as iidx,\n COALESCE(\n (regexp_match(indexdef, '\\(([a-zA-Z]+)\\)'))[1],\n (regexp_match(indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_-]+)''::text'))[1],\n CASE WHEN indexdef ~* '\\(datetime desc, end_datetime\\)' THEN 'datetime' ELSE NULL END\n ) AS field\n FROM\n pg_indexes\n JOIN p ON (tablename=partition)\n ), q AS (\n SELECT\n name AS field,\n collection,\n partition,\n format(indexdef(queryables), partition) as qidx\n FROM queryables, unnest_collection(queryables.collection_ids) collection\n JOIN p USING (collection)\n WHERE property_index_type IS NOT NULL OR name IN ('datetime','geometry','id')\n )\n SELECT\n collection,\n partition,\n field,\n indexname,\n iidx as existing_idx,\n qidx as queryable_idx\n FROM i FULL JOIN q USING (field, partition)\n WHERE CASE WHEN changes THEN lower(iidx) IS DISTINCT FROM lower(qidx) ELSE TRUE END;\n;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION maintain_index(\n indexname text,\n queryable_idx text,\n dropindexes boolean DEFAULT FALSE,\n rebuildindexes boolean DEFAULT FALSE,\n idxconcurrently boolean DEFAULT FALSE\n) RETURNS VOID AS $$\nDECLARE\nBEGIN\n IF indexname IS NOT NULL THEN\n IF dropindexes OR queryable_idx IS NOT NULL THEN\n EXECUTE format('DROP INDEX IF EXISTS %I;', indexname);\n ELSIF rebuildindexes THEN\n IF idxconcurrently THEN\n EXECUTE format('REINDEX INDEX CONCURRENTLY %I;', indexname);\n ELSE\n EXECUTE format('REINDEX INDEX CONCURRENTLY %I;', indexname);\n END IF;\n END IF;\n END IF;\n IF queryable_idx IS NOT NULL THEN\n IF idxconcurrently THEN\n EXECUTE replace(queryable_idx, 'INDEX', 'INDEX CONCURRENTLY');\n ELSE EXECUTE queryable_idx;\n END IF;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\n\nset check_function_bodies to off;\nCREATE OR REPLACE FUNCTION maintain_partition_queries(\n part text DEFAULT 'items',\n dropindexes boolean DEFAULT FALSE,\n rebuildindexes boolean DEFAULT FALSE,\n idxconcurrently boolean DEFAULT FALSE\n) RETURNS SETOF text AS $$\nDECLARE\n rec record;\n q text;\nBEGIN\n FOR rec IN (\n SELECT * FROM queryable_indexes(part,true)\n ) LOOP\n q := format(\n 'SELECT maintain_index(\n %L,%L,%L,%L,%L\n );',\n rec.indexname,\n rec.queryable_idx,\n dropindexes,\n rebuildindexes,\n idxconcurrently\n );\n RAISE NOTICE 'Q: %', q;\n RETURN NEXT q;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION maintain_partitions(\n part text DEFAULT 'items',\n dropindexes boolean DEFAULT FALSE,\n rebuildindexes boolean DEFAULT FALSE\n) RETURNS VOID AS $$\n WITH t AS (\n SELECT run_or_queue(q) FROM maintain_partition_queries(part, dropindexes, rebuildindexes) q\n ) SELECT count(*) FROM t;\n$$ LANGUAGE SQL;\n\n\nCREATE OR REPLACE FUNCTION queryables_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\nBEGIN\n PERFORM maintain_partitions();\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER queryables_trigger AFTER INSERT OR UPDATE ON queryables\nFOR EACH STATEMENT EXECUTE PROCEDURE queryables_trigger_func();\n\n\nCREATE OR REPLACE FUNCTION get_queryables(_collection_ids text[] DEFAULT NULL) RETURNS jsonb AS $$\nDECLARE\nBEGIN\n -- Build up queryables if the input contains valid collection ids or is empty\n IF EXISTS (\n SELECT 1 FROM collections\n WHERE\n _collection_ids IS NULL\n OR cardinality(_collection_ids) = 0\n OR id = ANY(_collection_ids)\n )\n THEN\n RETURN (\n WITH base AS (\n SELECT\n unnest(collection_ids) as collection_id,\n name,\n coalesce(definition, '{\"type\":\"string\"}'::jsonb) as definition\n FROM queryables\n WHERE\n _collection_ids IS NULL OR\n _collection_ids = '{}'::text[] OR\n _collection_ids && collection_ids\n UNION ALL\n SELECT null, name, coalesce(definition, '{\"type\":\"string\"}'::jsonb) as definition\n FROM queryables WHERE collection_ids IS NULL OR collection_ids = '{}'::text[]\n ), g AS (\n SELECT\n name,\n first_notnull(definition) as definition,\n jsonb_array_unique_merge(definition->'enum') as enum,\n jsonb_min(definition->'minimum') as minimum,\n jsonb_min(definition->'maxiumn') as maximum\n FROM base\n GROUP BY 1\n )\n SELECT\n jsonb_build_object(\n '$schema', 'http://json-schema.org/draft-07/schema#',\n '$id', '',\n 'type', 'object',\n 'title', 'STAC Queryables.',\n 'properties', jsonb_object_agg(\n name,\n definition\n ||\n jsonb_strip_nulls(jsonb_build_object(\n 'enum', enum,\n 'minimum', minimum,\n 'maximum', maximum\n ))\n ),\n 'additionalProperties', pgstac.additional_properties()\n )\n FROM g\n );\n ELSE\n RETURN NULL;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\nCREATE OR REPLACE FUNCTION get_queryables(_collection text DEFAULT NULL) RETURNS jsonb AS $$\n SELECT\n CASE\n WHEN _collection IS NULL THEN get_queryables(NULL::text[])\n ELSE get_queryables(ARRAY[_collection])\n END\n ;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_queryables() RETURNS jsonb AS $$\n SELECT get_queryables(NULL::text[]);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION schema_qualify_refs(url text, j jsonb) returns jsonb as $$\n SELECT regexp_replace(j::text, '\"\\$ref\": \"#', concat('\"$ref\": \"', url, '#'), 'g')::jsonb;\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE VIEW stac_extension_queryables AS\nSELECT DISTINCT key as name, schema_qualify_refs(e.url, j.value) as definition FROM stac_extensions e, jsonb_each(e.content->'definitions'->'fields'->'properties') j;\n\n\nCREATE OR REPLACE FUNCTION missing_queryables(_collection text, _tablesample float DEFAULT 5, minrows float DEFAULT 10) RETURNS TABLE(collection text, name text, definition jsonb, property_wrapper text) AS $$\nDECLARE\n q text;\n _partition text;\n explain_json json;\n psize float;\n estrows float;\nBEGIN\n SELECT format('_items_%s', key) INTO _partition FROM collections WHERE id=_collection;\n\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM %I;', _partition)\n INTO explain_json;\n psize := explain_json->0->'Plan'->'Plan Rows';\n estrows := _tablesample * .01 * psize;\n IF estrows < minrows THEN\n _tablesample := least(100,greatest(_tablesample, (estrows / psize) / 100));\n RAISE NOTICE '%', (psize / estrows) / 100;\n END IF;\n RAISE NOTICE 'Using tablesample % to find missing queryables from % % that has ~% rows estrows: %', _tablesample, _collection, _partition, psize, estrows;\n\n q := format(\n $q$\n WITH q AS (\n SELECT * FROM queryables\n WHERE\n collection_ids IS NULL\n OR %L = ANY(collection_ids)\n ), t AS (\n SELECT\n content->'properties' AS properties\n FROM\n %I\n TABLESAMPLE SYSTEM(%L)\n ), p AS (\n SELECT DISTINCT ON (key)\n key,\n value,\n s.definition\n FROM t\n JOIN LATERAL jsonb_each(properties) ON TRUE\n LEFT JOIN q ON (q.name=key)\n LEFT JOIN stac_extension_queryables s ON (s.name=key)\n WHERE q.definition IS NULL\n )\n SELECT\n %L,\n key,\n COALESCE(definition, jsonb_build_object('type',jsonb_typeof(value))) as definition,\n CASE\n WHEN definition->>'type' = 'integer' THEN 'to_int'\n WHEN COALESCE(definition->>'type', jsonb_typeof(value)) = 'number' THEN 'to_float'\n WHEN COALESCE(definition->>'type', jsonb_typeof(value)) = 'array' THEN 'to_text_array'\n ELSE 'to_text'\n END\n FROM p;\n $q$,\n _collection,\n _partition,\n _tablesample,\n _collection\n );\n RETURN QUERY EXECUTE q;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION missing_queryables(_tablesample float DEFAULT 5) RETURNS TABLE(collection_ids text[], name text, definition jsonb, property_wrapper text) AS $$\n SELECT\n array_agg(collection),\n name,\n definition,\n property_wrapper\n FROM\n collections\n JOIN LATERAL\n missing_queryables(id, _tablesample) c\n ON TRUE\n GROUP BY\n 2,3,4\n ORDER BY 2,1\n ;\n$$ LANGUAGE SQL;\nCREATE OR REPLACE FUNCTION parse_dtrange(\n _indate jsonb,\n relative_base timestamptz DEFAULT date_trunc('hour', CURRENT_TIMESTAMP)\n) RETURNS tstzrange AS $$\nDECLARE\n timestrs text[];\n s timestamptz;\n e timestamptz;\nBEGIN\n timestrs :=\n CASE\n WHEN _indate ? 'timestamp' THEN\n ARRAY[_indate->>'timestamp']\n WHEN _indate ? 'interval' THEN\n to_text_array(_indate->'interval')\n WHEN jsonb_typeof(_indate) = 'array' THEN\n to_text_array(_indate)\n ELSE\n regexp_split_to_array(\n _indate->>0,\n '/'\n )\n END;\n RAISE NOTICE 'TIMESTRS %', timestrs;\n IF cardinality(timestrs) = 1 THEN\n IF timestrs[1] ILIKE 'P%' THEN\n RETURN tstzrange(relative_base - upper(timestrs[1])::interval, relative_base, '[)');\n END IF;\n s := timestrs[1]::timestamptz;\n RETURN tstzrange(s, s, '[]');\n END IF;\n\n IF cardinality(timestrs) != 2 THEN\n RAISE EXCEPTION 'Timestamp cannot have more than 2 values';\n END IF;\n\n IF timestrs[1] = '..' OR timestrs[1] = '' THEN\n s := '-infinity'::timestamptz;\n e := timestrs[2]::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] = '..' OR timestrs[2] = '' THEN\n s := timestrs[1]::timestamptz;\n e := 'infinity'::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[1] ILIKE 'P%' AND timestrs[2] NOT ILIKE 'P%' THEN\n e := timestrs[2]::timestamptz;\n s := e - upper(timestrs[1])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] ILIKE 'P%' AND timestrs[1] NOT ILIKE 'P%' THEN\n s := timestrs[1]::timestamptz;\n e := s + upper(timestrs[2])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n s := timestrs[1]::timestamptz;\n e := timestrs[2]::timestamptz;\n\n RETURN tstzrange(s,e,'[)');\n\n RETURN NULL;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT PARALLEL SAFE SET TIME ZONE 'UTC';\n\nCREATE OR REPLACE FUNCTION parse_dtrange(\n _indate text,\n relative_base timestamptz DEFAULT CURRENT_TIMESTAMP\n) RETURNS tstzrange AS $$\n SELECT parse_dtrange(to_jsonb(_indate), relative_base);\n$$ LANGUAGE SQL STABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION temporal_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\n ll text := 'datetime';\n lh text := 'end_datetime';\n rrange tstzrange;\n rl text;\n rh text;\n outq text;\nBEGIN\n rrange := parse_dtrange(args->1);\n RAISE NOTICE 'Constructing temporal query OP: %, ARGS: %, RRANGE: %', op, args, rrange;\n op := lower(op);\n rl := format('%L::timestamptz', lower(rrange));\n rh := format('%L::timestamptz', upper(rrange));\n outq := CASE op\n WHEN 't_before' THEN 'lh < rl'\n WHEN 't_after' THEN 'll > rh'\n WHEN 't_meets' THEN 'lh = rl'\n WHEN 't_metby' THEN 'll = rh'\n WHEN 't_overlaps' THEN 'll < rl AND rl < lh < rh'\n WHEN 't_overlappedby' THEN 'rl < ll < rh AND lh > rh'\n WHEN 't_starts' THEN 'll = rl AND lh < rh'\n WHEN 't_startedby' THEN 'll = rl AND lh > rh'\n WHEN 't_during' THEN 'll > rl AND lh < rh'\n WHEN 't_contains' THEN 'll < rl AND lh > rh'\n WHEN 't_finishes' THEN 'll > rl AND lh = rh'\n WHEN 't_finishedby' THEN 'll < rl AND lh = rh'\n WHEN 't_equals' THEN 'll = rl AND lh = rh'\n WHEN 't_disjoint' THEN 'NOT (ll <= rh AND lh >= rl)'\n WHEN 't_intersects' THEN 'll <= rh AND lh >= rl'\n WHEN 'anyinteracts' THEN 'll <= rh AND lh >= rl'\n END;\n outq := regexp_replace(outq, '\\mll\\M', ll);\n outq := regexp_replace(outq, '\\mlh\\M', lh);\n outq := regexp_replace(outq, '\\mrl\\M', rl);\n outq := regexp_replace(outq, '\\mrh\\M', rh);\n outq := format('(%s)', outq);\n RETURN outq;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\n\n\nCREATE OR REPLACE FUNCTION spatial_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\n geom text;\n j jsonb := args->1;\nBEGIN\n op := lower(op);\n RAISE NOTICE 'Constructing spatial query OP: %, ARGS: %', op, args;\n IF op NOT IN ('s_equals','s_disjoint','s_touches','s_within','s_overlaps','s_crosses','s_intersects','intersects','s_contains') THEN\n RAISE EXCEPTION 'Spatial Operator % Not Supported', op;\n END IF;\n op := regexp_replace(op, '^s_', 'st_');\n IF op = 'intersects' THEN\n op := 'st_intersects';\n END IF;\n -- Convert geometry to WKB string\n IF j ? 'type' AND j ? 'coordinates' THEN\n geom := st_geomfromgeojson(j)::text;\n ELSIF jsonb_typeof(j) = 'array' THEN\n geom := bbox_geom(j)::text;\n END IF;\n\n RETURN format('%s(geometry, %L::geometry)', op, geom);\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION query_to_cql2(q jsonb) RETURNS jsonb AS $$\n-- Translates anything passed in through the deprecated \"query\" into equivalent CQL2\nWITH t AS (\n SELECT key as property, value as ops\n FROM jsonb_each(q)\n), t2 AS (\n SELECT property, (jsonb_each(ops)).*\n FROM t WHERE jsonb_typeof(ops) = 'object'\n UNION ALL\n SELECT property, 'eq', ops\n FROM t WHERE jsonb_typeof(ops) != 'object'\n)\nSELECT\n jsonb_strip_nulls(jsonb_build_object(\n 'op', 'and',\n 'args', jsonb_agg(\n jsonb_build_object(\n 'op', key,\n 'args', jsonb_build_array(\n jsonb_build_object('property',property),\n value\n )\n )\n )\n )\n) as qcql FROM t2\n;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\nCREATE OR REPLACE FUNCTION cql1_to_cql2(j jsonb) RETURNS jsonb AS $$\nDECLARE\n args jsonb;\n ret jsonb;\nBEGIN\n RAISE NOTICE 'CQL1_TO_CQL2: %', j;\n IF j ? 'filter' THEN\n RETURN cql1_to_cql2(j->'filter');\n END IF;\n IF j ? 'property' THEN\n RETURN j;\n END IF;\n IF jsonb_typeof(j) = 'array' THEN\n SELECT jsonb_agg(cql1_to_cql2(el)) INTO args FROM jsonb_array_elements(j) el;\n RETURN args;\n END IF;\n IF jsonb_typeof(j) = 'number' THEN\n RETURN j;\n END IF;\n IF jsonb_typeof(j) = 'string' THEN\n RETURN j;\n END IF;\n\n IF jsonb_typeof(j) = 'object' THEN\n SELECT jsonb_build_object(\n 'op', key,\n 'args', cql1_to_cql2(value)\n ) INTO ret\n FROM jsonb_each(j)\n WHERE j IS NOT NULL;\n RETURN ret;\n END IF;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE STRICT;\n\nCREATE TABLE cql2_ops (\n op text PRIMARY KEY,\n template text,\n types text[]\n);\nINSERT INTO cql2_ops (op, template, types) VALUES\n ('eq', '%s = %s', NULL),\n ('neq', '%s != %s', NULL),\n ('ne', '%s != %s', NULL),\n ('!=', '%s != %s', NULL),\n ('<>', '%s != %s', NULL),\n ('lt', '%s < %s', NULL),\n ('lte', '%s <= %s', NULL),\n ('gt', '%s > %s', NULL),\n ('gte', '%s >= %s', NULL),\n ('le', '%s <= %s', NULL),\n ('ge', '%s >= %s', NULL),\n ('=', '%s = %s', NULL),\n ('<', '%s < %s', NULL),\n ('<=', '%s <= %s', NULL),\n ('>', '%s > %s', NULL),\n ('>=', '%s >= %s', NULL),\n ('like', '%s LIKE %s', NULL),\n ('ilike', '%s ILIKE %s', NULL),\n ('+', '%s + %s', NULL),\n ('-', '%s - %s', NULL),\n ('*', '%s * %s', NULL),\n ('/', '%s / %s', NULL),\n ('not', 'NOT (%s)', NULL),\n ('between', '%s BETWEEN %s AND %s', NULL),\n ('isnull', '%s IS NULL', NULL),\n ('upper', 'upper(%s)', NULL),\n ('lower', 'lower(%s)', NULL),\n ('casei', 'upper(%s)', NULL),\n ('accenti', 'unaccent(%s)', NULL)\nON CONFLICT (op) DO UPDATE\n SET\n template = EXCLUDED.template\n;\n\n\nCREATE OR REPLACE FUNCTION cql2_query(j jsonb, wrapper text DEFAULT NULL) RETURNS text AS $$\n#variable_conflict use_variable\nDECLARE\n args jsonb := j->'args';\n arg jsonb;\n op text := lower(j->>'op');\n cql2op RECORD;\n literal text;\n _wrapper text;\n leftarg text;\n rightarg text;\n prop text;\n extra_props bool := pgstac.additional_properties();\nBEGIN\n IF j IS NULL OR (op IS NOT NULL AND args IS NULL) THEN\n RETURN NULL;\n END IF;\n RAISE NOTICE 'CQL2_QUERY: %', j;\n\n -- check if all properties are represented in the queryables\n IF NOT extra_props THEN\n FOR prop IN\n SELECT DISTINCT p->>0\n FROM jsonb_path_query(j, 'strict $.**.property') p\n WHERE p->>0 NOT IN ('id', 'datetime', 'geometry', 'end_datetime', 'collection')\n LOOP\n IF (queryable(prop)).nulled_wrapper IS NULL THEN\n RAISE EXCEPTION 'Term % is not found in queryables.', prop;\n END IF;\n END LOOP;\n END IF;\n\n IF j ? 'filter' THEN\n RETURN cql2_query(j->'filter');\n END IF;\n\n IF j ? 'upper' THEN\n RETURN cql2_query(jsonb_build_object('op', 'upper', 'args', j->'upper'));\n END IF;\n\n IF j ? 'lower' THEN\n RETURN cql2_query(jsonb_build_object('op', 'lower', 'args', j->'lower'));\n END IF;\n\n -- Temporal Query\n IF op ilike 't_%' or op = 'anyinteracts' THEN\n RETURN temporal_op_query(op, args);\n END IF;\n\n -- If property is a timestamp convert it to text to use with\n -- general operators\n IF j ? 'timestamp' THEN\n RETURN format('%L::timestamptz', to_tstz(j->'timestamp'));\n END IF;\n IF j ? 'interval' THEN\n RAISE EXCEPTION 'Please use temporal operators when using intervals.';\n RETURN NONE;\n END IF;\n\n -- Spatial Query\n IF op ilike 's_%' or op = 'intersects' THEN\n RETURN spatial_op_query(op, args);\n END IF;\n\n IF op IN ('a_equals','a_contains','a_contained_by','a_overlaps') THEN\n IF args->0 ? 'property' THEN\n leftarg := format('to_text_array(%s)', (queryable(args->0->>'property')).path);\n END IF;\n IF args->1 ? 'property' THEN\n rightarg := format('to_text_array(%s)', (queryable(args->1->>'property')).path);\n END IF;\n RETURN FORMAT(\n '%s %s %s',\n COALESCE(leftarg, quote_literal(to_text_array(args->0))),\n CASE op\n WHEN 'a_equals' THEN '='\n WHEN 'a_contains' THEN '@>'\n WHEN 'a_contained_by' THEN '<@'\n WHEN 'a_overlaps' THEN '&&'\n END,\n COALESCE(rightarg, quote_literal(to_text_array(args->1)))\n );\n END IF;\n\n IF op = 'in' THEN\n RAISE NOTICE 'IN : % % %', args, jsonb_build_array(args->0), args->1;\n args := jsonb_build_array(args->0) || (args->1);\n RAISE NOTICE 'IN2 : %', args;\n END IF;\n\n\n\n IF op = 'between' THEN\n args = jsonb_build_array(\n args->0,\n args->1,\n args->2\n );\n END IF;\n\n -- Make sure that args is an array and run cql2_query on\n -- each element of the array\n RAISE NOTICE 'ARGS PRE: %', args;\n IF j ? 'args' THEN\n IF jsonb_typeof(args) != 'array' THEN\n args := jsonb_build_array(args);\n END IF;\n\n IF jsonb_path_exists(args, '$[*] ? (@.property == \"id\" || @.property == \"datetime\" || @.property == \"end_datetime\" || @.property == \"collection\")') THEN\n wrapper := NULL;\n ELSE\n -- if any of the arguments are a property, try to get the property_wrapper\n FOR arg IN SELECT jsonb_path_query(args, '$[*] ? (@.property != null)') LOOP\n RAISE NOTICE 'Arg: %', arg;\n wrapper := (queryable(arg->>'property')).nulled_wrapper;\n RAISE NOTICE 'Property: %, Wrapper: %', arg, wrapper;\n IF wrapper IS NOT NULL THEN\n EXIT;\n END IF;\n END LOOP;\n\n -- if the property was not in queryables, see if any args were numbers\n IF\n wrapper IS NULL\n AND jsonb_path_exists(args, '$[*] ? (@.type()==\"number\")')\n THEN\n wrapper := 'to_float';\n END IF;\n wrapper := coalesce(wrapper, 'to_text');\n END IF;\n\n SELECT jsonb_agg(cql2_query(a, wrapper))\n INTO args\n FROM jsonb_array_elements(args) a;\n END IF;\n RAISE NOTICE 'ARGS: %', args;\n\n IF op IN ('and', 'or') THEN\n RETURN\n format(\n '(%s)',\n array_to_string(to_text_array(args), format(' %s ', upper(op)))\n );\n END IF;\n\n IF op = 'in' THEN\n RAISE NOTICE 'IN -- % %', args->0, to_text(args->0);\n RETURN format(\n '%s IN (%s)',\n to_text(args->0),\n array_to_string((to_text_array(args))[2:], ',')\n );\n END IF;\n\n -- Look up template from cql2_ops\n IF j ? 'op' THEN\n SELECT * INTO cql2op FROM cql2_ops WHERE cql2_ops.op ilike op;\n IF FOUND THEN\n -- If specific index set in queryables for a property cast other arguments to that type\n\n RETURN format(\n cql2op.template,\n VARIADIC (to_text_array(args))\n );\n ELSE\n RAISE EXCEPTION 'Operator % Not Supported.', op;\n END IF;\n END IF;\n\n\n IF wrapper IS NOT NULL THEN\n RAISE NOTICE 'Wrapping % with %', j, wrapper;\n IF j ? 'property' THEN\n RETURN format('%I(%s)', wrapper, (queryable(j->>'property')).path);\n ELSE\n RETURN format('%I(%L)', wrapper, j);\n END IF;\n ELSIF j ? 'property' THEN\n RETURN quote_ident(j->>'property');\n END IF;\n\n RETURN quote_literal(to_text(j));\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION paging_dtrange(\n j jsonb\n) RETURNS tstzrange AS $$\nDECLARE\n op text;\n filter jsonb := j->'filter';\n dtrange tstzrange := tstzrange('-infinity'::timestamptz,'infinity'::timestamptz);\n sdate timestamptz := '-infinity'::timestamptz;\n edate timestamptz := 'infinity'::timestamptz;\n jpitem jsonb;\nBEGIN\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n END IF;\n IF NOT (filter @? '$.**.op ? (@ == \"or\" || @ == \"not\")') THEN\n FOR jpitem IN SELECT j FROM jsonb_path_query(filter,'strict $.** ? (@.args[*].property == \"datetime\")'::jsonpath) j LOOP\n op := lower(jpitem->>'op');\n dtrange := parse_dtrange(jpitem->'args'->1);\n IF op IN ('<=', 'lt', 'lte', '<', 'le', 't_before') THEN\n sdate := greatest(sdate,'-infinity');\n edate := least(edate, upper(dtrange));\n ELSIF op IN ('>=', '>', 'gt', 'gte', 'ge', 't_after') THEN\n edate := least(edate, 'infinity');\n sdate := greatest(sdate, lower(dtrange));\n ELSIF op IN ('=', 'eq') THEN\n edate := least(edate, upper(dtrange));\n sdate := greatest(sdate, lower(dtrange));\n END IF;\n RAISE NOTICE '2 OP: %, ARGS: %, DTRANGE: %, SDATE: %, EDATE: %', op, jpitem->'args'->1, dtrange, sdate, edate;\n END LOOP;\n END IF;\n IF sdate > edate THEN\n RETURN 'empty'::tstzrange;\n END IF;\n RETURN tstzrange(sdate,edate, '[]');\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT SET TIME ZONE 'UTC';\n\nCREATE OR REPLACE FUNCTION paging_collections(\n IN j jsonb\n) RETURNS text[] AS $$\nDECLARE\n filter jsonb := j->'filter';\n jpitem jsonb;\n op text;\n args jsonb;\n arg jsonb;\n collections text[];\nBEGIN\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n END IF;\n IF NOT (filter @? '$.**.op ? (@ == \"or\" || @ == \"not\")') THEN\n FOR jpitem IN SELECT j FROM jsonb_path_query(filter,'strict $.** ? (@.args[*].property == \"collection\")'::jsonpath) j LOOP\n RAISE NOTICE 'JPITEM: %', jpitem;\n op := jpitem->>'op';\n args := jpitem->'args';\n IF op IN ('=', 'eq', 'in') THEN\n FOR arg IN SELECT a FROM jsonb_array_elements(args) a LOOP\n IF jsonb_typeof(arg) IN ('string', 'array') THEN\n RAISE NOTICE 'arg: %, collections: %', arg, collections;\n IF collections IS NULL OR collections = '{}'::text[] THEN\n collections := to_text_array(arg);\n ELSE\n collections := array_intersection(collections, to_text_array(arg));\n END IF;\n END IF;\n END LOOP;\n END IF;\n END LOOP;\n END IF;\n IF collections = '{}'::text[] THEN\n RETURN NULL;\n END IF;\n RETURN collections;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\nCREATE TABLE items (\n id text NOT NULL,\n geometry geometry NOT NULL,\n collection text NOT NULL,\n datetime timestamptz NOT NULL,\n end_datetime timestamptz NOT NULL,\n content JSONB NOT NULL,\n private jsonb\n)\nPARTITION BY LIST (collection)\n;\n\nCREATE INDEX \"datetime_idx\" ON items USING BTREE (datetime DESC, end_datetime ASC);\nCREATE INDEX \"geometry_idx\" ON items USING GIST (geometry);\n\nCREATE STATISTICS datetime_stats (dependencies) on datetime, end_datetime from items;\n\nALTER TABLE items ADD CONSTRAINT items_collections_fk FOREIGN KEY (collection) REFERENCES collections(id) ON DELETE CASCADE DEFERRABLE;\n\nCREATE OR REPLACE FUNCTION partition_after_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p text;\n t timestamptz := clock_timestamp();\nBEGIN\n RAISE NOTICE 'Updating partition stats %', t;\n FOR p IN SELECT DISTINCT partition\n FROM newdata n JOIN partition_sys_meta p\n ON (n.collection=p.collection AND n.datetime <@ p.partition_dtrange)\n LOOP\n PERFORM run_or_queue(format('SELECT update_partition_stats(%L, %L);', p, true));\n END LOOP;\n IF TG_OP IN ('DELETE','UPDATE') THEN\n DELETE FROM format_item_cache c USING newdata n WHERE c.collection = n.collection AND c.id = n.id;\n END IF;\n RAISE NOTICE 't: % %', t, clock_timestamp() - t;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE TRIGGER items_after_insert_trigger\nAFTER INSERT ON items\nREFERENCING NEW TABLE AS newdata\nFOR EACH STATEMENT\nEXECUTE FUNCTION partition_after_triggerfunc();\n\nCREATE TRIGGER items_after_update_trigger\nAFTER DELETE ON items\nREFERENCING OLD TABLE AS newdata\nFOR EACH STATEMENT\nEXECUTE FUNCTION partition_after_triggerfunc();\n\nCREATE TRIGGER items_after_delete_trigger\nAFTER UPDATE ON items\nREFERENCING NEW TABLE AS newdata\nFOR EACH STATEMENT\nEXECUTE FUNCTION partition_after_triggerfunc();\n\n\nCREATE OR REPLACE FUNCTION content_slim(_item jsonb) RETURNS jsonb AS $$\n SELECT strip_jsonb(_item - '{id,geometry,collection,type}'::text[], collection_base_item(_item->>'collection')) - '{id,geometry,collection,type}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_dehydrate(content jsonb) RETURNS items AS $$\n SELECT\n content->>'id' as id,\n stac_geom(content) as geometry,\n content->>'collection' as collection,\n stac_datetime(content) as datetime,\n stac_end_datetime(content) as end_datetime,\n content_slim(content) as content,\n null::jsonb as private\n ;\n$$ LANGUAGE SQL STABLE;\n\nCREATE OR REPLACE FUNCTION include_field(f text, fields jsonb DEFAULT '{}'::jsonb) RETURNS boolean AS $$\nDECLARE\n includes jsonb := fields->'include';\n excludes jsonb := fields->'exclude';\nBEGIN\n IF f IS NULL THEN\n RETURN NULL;\n END IF;\n\n\n IF\n jsonb_typeof(excludes) = 'array'\n AND jsonb_array_length(excludes)>0\n AND excludes ? f\n THEN\n RETURN FALSE;\n END IF;\n\n IF\n (\n jsonb_typeof(includes) = 'array'\n AND jsonb_array_length(includes) > 0\n AND includes ? f\n ) OR\n (\n includes IS NULL\n OR jsonb_typeof(includes) = 'null'\n OR jsonb_array_length(includes) = 0\n )\n THEN\n RETURN TRUE;\n END IF;\n\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nDROP FUNCTION IF EXISTS content_hydrate(jsonb, jsonb, jsonb);\nCREATE OR REPLACE FUNCTION content_hydrate(\n _item jsonb,\n _base_item jsonb,\n fields jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\n SELECT merge_jsonb(\n jsonb_fields(_item, fields),\n jsonb_fields(_base_item, fields)\n );\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION content_hydrate(_item items, _collection collections, fields jsonb DEFAULT '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\n content jsonb;\n base_item jsonb := _collection.base_item;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry, 20)::jsonb;\n END IF;\n output := content_hydrate(\n jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'collection', _item.collection,\n 'type', 'Feature'\n ) || _item.content,\n _collection.base_item,\n fields\n );\n\n RETURN output;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_nonhydrated(\n _item items,\n fields jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry, 20)::jsonb;\n END IF;\n output := jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'collection', _item.collection,\n 'type', 'Feature'\n ) || _item.content;\n RETURN output;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_hydrate(_item items, fields jsonb DEFAULT '{}'::jsonb) RETURNS jsonb AS $$\n SELECT content_hydrate(\n _item,\n (SELECT c FROM collections c WHERE id=_item.collection LIMIT 1),\n fields\n );\n$$ LANGUAGE SQL STABLE;\n\n\nCREATE UNLOGGED TABLE items_staging (\n content JSONB NOT NULL\n);\nCREATE UNLOGGED TABLE items_staging_ignore (\n content JSONB NOT NULL\n);\nCREATE UNLOGGED TABLE items_staging_upsert (\n content JSONB NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION items_staging_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p record;\n _partitions text[];\n part text;\n ts timestamptz := clock_timestamp();\n nrows int;\nBEGIN\n RAISE NOTICE 'Creating Partitions. %', clock_timestamp() - ts;\n\n FOR part IN WITH t AS (\n SELECT\n n.content->>'collection' as collection,\n stac_daterange(n.content->'properties') as dtr,\n partition_trunc\n FROM newdata n JOIN collections ON (n.content->>'collection'=collections.id)\n ), p AS (\n SELECT\n collection,\n COALESCE(date_trunc(partition_trunc::text, lower(dtr)),'-infinity') as d,\n tstzrange(min(lower(dtr)),max(lower(dtr)),'[]') as dtrange,\n tstzrange(min(upper(dtr)),max(upper(dtr)),'[]') as edtrange\n FROM t\n GROUP BY 1,2\n ) SELECT check_partition(collection, dtrange, edtrange) FROM p LOOP\n RAISE NOTICE 'Partition %', part;\n END LOOP;\n\n RAISE NOTICE 'Creating temp table with data to be added. %', clock_timestamp() - ts;\n DROP TABLE IF EXISTS tmpdata;\n CREATE TEMP TABLE tmpdata ON COMMIT DROP AS\n SELECT\n (content_dehydrate(content)).*\n FROM newdata;\n GET DIAGNOSTICS nrows = ROW_COUNT;\n RAISE NOTICE 'Added % rows to tmpdata. %', nrows, clock_timestamp() - ts;\n\n RAISE NOTICE 'Doing the insert. %', clock_timestamp() - ts;\n IF TG_TABLE_NAME = 'items_staging' THEN\n INSERT INTO items\n SELECT * FROM tmpdata;\n GET DIAGNOSTICS nrows = ROW_COUNT;\n RAISE NOTICE 'Inserted % rows to items. %', nrows, clock_timestamp() - ts;\n ELSIF TG_TABLE_NAME = 'items_staging_ignore' THEN\n INSERT INTO items\n SELECT * FROM tmpdata\n ON CONFLICT DO NOTHING;\n GET DIAGNOSTICS nrows = ROW_COUNT;\n RAISE NOTICE 'Inserted % rows to items. %', nrows, clock_timestamp() - ts;\n ELSIF TG_TABLE_NAME = 'items_staging_upsert' THEN\n DELETE FROM items i USING tmpdata s\n WHERE\n i.id = s.id\n AND i.collection = s.collection\n AND i IS DISTINCT FROM s\n ;\n GET DIAGNOSTICS nrows = ROW_COUNT;\n RAISE NOTICE 'Deleted % rows from items. %', nrows, clock_timestamp() - ts;\n INSERT INTO items AS t\n SELECT * FROM tmpdata\n ON CONFLICT DO NOTHING;\n GET DIAGNOSTICS nrows = ROW_COUNT;\n RAISE NOTICE 'Inserted % rows to items. %', nrows, clock_timestamp() - ts;\n END IF;\n\n RAISE NOTICE 'Deleting data from staging table. %', clock_timestamp() - ts;\n DELETE FROM items_staging;\n RAISE NOTICE 'Done. %', clock_timestamp() - ts;\n\n RETURN NULL;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER items_staging_insert_trigger AFTER INSERT ON items_staging REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\nCREATE TRIGGER items_staging_insert_ignore_trigger AFTER INSERT ON items_staging_ignore REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\nCREATE TRIGGER items_staging_insert_upsert_trigger AFTER INSERT ON items_staging_upsert REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\n\nCREATE OR REPLACE FUNCTION item_by_id(_id text, _collection text DEFAULT NULL) RETURNS items AS\n$$\nDECLARE\n i items%ROWTYPE;\nBEGIN\n SELECT * INTO i FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection) LIMIT 1;\n RETURN i;\nEND;\n$$ LANGUAGE PLPGSQL STABLE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION get_item(_id text, _collection text DEFAULT NULL) RETURNS jsonb AS $$\n SELECT content_hydrate(items) FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection);\n$$ LANGUAGE SQL STABLE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION delete_item(_id text, _collection text DEFAULT NULL) RETURNS VOID AS $$\nDECLARE\nout items%ROWTYPE;\nBEGIN\n DELETE FROM items WHERE id = _id AND (_collection IS NULL OR collection=_collection) RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n--/*\nCREATE OR REPLACE FUNCTION create_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION update_item(content jsonb) RETURNS VOID AS $$\nDECLARE\n old items %ROWTYPE;\n out items%ROWTYPE;\nBEGIN\n PERFORM delete_item(content->>'id', content->>'collection');\n PERFORM create_item(content);\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION create_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION collection_bbox(id text) RETURNS jsonb AS $$\n SELECT (replace(replace(replace(st_extent(geometry)::text,'BOX(','[['),')',']]'),' ',','))::jsonb\n FROM items WHERE collection=$1;\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION collection_temporal_extent(id text) RETURNS jsonb AS $$\n SELECT to_jsonb(array[array[min(datetime), max(datetime)]])\n FROM items WHERE collection=$1;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection_extents() RETURNS VOID AS $$\nUPDATE collections\n SET content = jsonb_set_lax(\n content,\n '{extent}'::text[],\n collection_extent(id, FALSE),\n true,\n 'use_json_null'\n )\n;\n$$ LANGUAGE SQL;\nCREATE TABLE partition_stats (\n partition text PRIMARY KEY,\n dtrange tstzrange,\n edtrange tstzrange,\n spatial geometry,\n last_updated timestamptz,\n keys text[]\n) WITH (FILLFACTOR=90);\n\nCREATE INDEX partitions_range_idx ON partition_stats USING GIST(dtrange);\n\n\nCREATE OR REPLACE FUNCTION constraint_tstzrange(expr text) RETURNS tstzrange AS $$\n WITH t AS (\n SELECT regexp_matches(\n expr,\n E'\\\\(''\\([0-9 :+-]*\\)''\\\\).*\\\\(''\\([0-9 :+-]*\\)''\\\\)'\n ) AS m\n ) SELECT tstzrange(m[1]::timestamptz, m[2]::timestamptz) FROM t\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE STRICT;\n\nCREATE OR REPLACE FUNCTION dt_constraint(coid oid, OUT dt tstzrange, OUT edt tstzrange) RETURNS RECORD AS $$\nDECLARE\n expr text := pg_get_constraintdef(coid);\n matches timestamptz[];\nBEGIN\n IF expr LIKE '%NULL%' THEN\n dt := tstzrange(null::timestamptz, null::timestamptz);\n edt := tstzrange(null::timestamptz, null::timestamptz);\n RETURN;\n END IF;\n WITH f AS (SELECT (regexp_matches(expr, E'([0-9]{4}-[0-1][0-9]-[0-3][0-9] [0-2][0-9]:[0-5][0-9]:[0-5][0-9]\\.?[0-9]*)', 'g'))[1] f)\n SELECT array_agg(f::timestamptz) INTO matches FROM f;\n IF cardinality(matches) = 4 THEN\n dt := tstzrange(matches[1], matches[2],'[]');\n edt := tstzrange(matches[3], matches[4], '[]');\n RETURN;\n ELSIF cardinality(matches) = 2 THEN\n edt := tstzrange(matches[1], matches[2],'[]');\n RETURN;\n END IF;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\nCREATE OR REPLACE VIEW partition_sys_meta AS\nSELECT\n (parse_ident(relid::text))[cardinality(parse_ident(relid::text))] as partition,\n replace(replace(CASE WHEN level = 1 THEN pg_get_expr(c.relpartbound, c.oid)\n ELSE pg_get_expr(parent.relpartbound, parent.oid)\n END, 'FOR VALUES IN (''',''), ''')','') AS collection,\n level,\n c.reltuples,\n c.relhastriggers,\n COALESCE(constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as partition_dtrange,\n COALESCE((dt_constraint(edt.oid)).dt, constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as constraint_dtrange,\n COALESCE((dt_constraint(edt.oid)).edt, tstzrange('-infinity', 'infinity','[]')) as constraint_edtrange\nFROM\n pg_partition_tree('items')\n JOIN pg_class c ON (relid::regclass = c.oid)\n JOIN pg_class parent ON (parentrelid::regclass = parent.oid AND isleaf)\n LEFT JOIN pg_constraint edt ON (conrelid=c.oid AND contype='c')\nWHERE isleaf\n;\n\nCREATE OR REPLACE VIEW partitions_view AS\nSELECT\n (parse_ident(relid::text))[cardinality(parse_ident(relid::text))] as partition,\n replace(replace(CASE WHEN level = 1 THEN pg_get_expr(c.relpartbound, c.oid)\n ELSE pg_get_expr(parent.relpartbound, parent.oid)\n END, 'FOR VALUES IN (''',''), ''')','') AS collection,\n level,\n c.reltuples,\n c.relhastriggers,\n COALESCE(pgstac.constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as partition_dtrange,\n COALESCE((pgstac.dt_constraint(edt.oid)).dt, pgstac.constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as constraint_dtrange,\n COALESCE((pgstac.dt_constraint(edt.oid)).edt, tstzrange('-infinity', 'infinity','[]')) as constraint_edtrange,\n dtrange,\n edtrange,\n spatial,\n last_updated\nFROM\n pg_partition_tree('pgstac.items')\n JOIN pg_class c ON (relid::regclass = c.oid)\n JOIN pg_class parent ON (parentrelid::regclass = parent.oid AND isleaf)\n LEFT JOIN pg_constraint edt ON (conrelid=c.oid AND contype='c')\n LEFT JOIN pgstac.partition_stats ON ((parse_ident(relid::text))[cardinality(parse_ident(relid::text))]=partition)\nWHERE isleaf\n;\n\nCREATE MATERIALIZED VIEW partitions AS\nSELECT * FROM partitions_view;\nCREATE UNIQUE INDEX ON partitions (partition);\n\nCREATE MATERIALIZED VIEW partition_steps AS\nSELECT\n partition as name,\n date_trunc('month',lower(partition_dtrange)) as sdate,\n date_trunc('month', upper(partition_dtrange)) + '1 month'::interval as edate\n FROM partitions_view WHERE partition_dtrange IS NOT NULL AND partition_dtrange != 'empty'::tstzrange\n ORDER BY dtrange ASC\n;\n\n\nCREATE OR REPLACE FUNCTION update_partition_stats_q(_partition text, istrigger boolean default false) RETURNS VOID AS $$\nDECLARE\nBEGIN\n PERFORM run_or_queue(\n format('SELECT update_partition_stats(%L, %L);', _partition, istrigger)\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION update_partition_stats(_partition text, istrigger boolean default false) RETURNS VOID AS $$\nDECLARE\n dtrange tstzrange;\n edtrange tstzrange;\n cdtrange tstzrange;\n cedtrange tstzrange;\n extent geometry;\n collection text;\nBEGIN\n RAISE NOTICE 'Updating stats for %.', _partition;\n EXECUTE format(\n $q$\n SELECT\n tstzrange(min(datetime), max(datetime),'[]'),\n tstzrange(min(end_datetime), max(end_datetime), '[]')\n FROM %I\n $q$,\n _partition\n ) INTO dtrange, edtrange;\n extent := st_estimatedextent('pgstac', _partition, 'geometry');\n RAISE DEBUG 'Estimated Extent: %', extent;\n INSERT INTO partition_stats (partition, dtrange, edtrange, spatial, last_updated)\n SELECT _partition, dtrange, edtrange, extent, now()\n ON CONFLICT (partition) DO\n UPDATE SET\n dtrange=EXCLUDED.dtrange,\n edtrange=EXCLUDED.edtrange,\n spatial=EXCLUDED.spatial,\n last_updated=EXCLUDED.last_updated\n ;\n\n SELECT\n constraint_dtrange, constraint_edtrange, pv.collection\n INTO cdtrange, cedtrange, collection\n FROM partitions_view pv WHERE partition = _partition;\n REFRESH MATERIALIZED VIEW partitions;\n REFRESH MATERIALIZED VIEW partition_steps;\n\n\n RAISE NOTICE 'Checking if we need to modify constraints...';\n RAISE NOTICE 'cdtrange: % dtrange: % cedtrange: % edtrange: %',cdtrange, dtrange, cedtrange, edtrange;\n IF\n (cdtrange IS DISTINCT FROM dtrange OR edtrange IS DISTINCT FROM cedtrange)\n AND NOT istrigger\n THEN\n RAISE NOTICE 'Modifying Constraints';\n RAISE NOTICE 'Existing % %', cdtrange, cedtrange;\n RAISE NOTICE 'New % %', dtrange, edtrange;\n PERFORM drop_table_constraints(_partition);\n PERFORM create_table_constraints(_partition, dtrange, edtrange);\n REFRESH MATERIALIZED VIEW partitions;\n REFRESH MATERIALIZED VIEW partition_steps;\n END IF;\n RAISE NOTICE 'Checking if we need to update collection extents.';\n IF get_setting_bool('update_collection_extent') THEN\n RAISE NOTICE 'updating collection extent for %', collection;\n PERFORM run_or_queue(format($q$\n UPDATE collections\n SET content = jsonb_set_lax(\n content,\n '{extent}'::text[],\n collection_extent(%L, FALSE),\n true,\n 'use_json_null'\n ) WHERE id=%L\n ;\n $q$, collection, collection));\n ELSE\n RAISE NOTICE 'Not updating collection extent for %', collection;\n END IF;\n\nEND;\n$$ LANGUAGE PLPGSQL STRICT SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION partition_name( IN collection text, IN dt timestamptz, OUT partition_name text, OUT partition_range tstzrange) AS $$\nDECLARE\n c RECORD;\n parent_name text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n parent_name := format('_items_%s', c.key);\n\n\n IF c.partition_trunc = 'year' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYY'));\n ELSIF c.partition_trunc = 'month' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYYMM'));\n ELSE\n partition_name := parent_name;\n partition_range := tstzrange('-infinity'::timestamptz, 'infinity'::timestamptz, '[]');\n END IF;\n IF partition_range IS NULL THEN\n partition_range := tstzrange(\n date_trunc(c.partition_trunc::text, dt),\n date_trunc(c.partition_trunc::text, dt) + concat('1 ', c.partition_trunc)::interval\n );\n END IF;\n RETURN;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION drop_table_constraints(t text) RETURNS text AS $$\nDECLARE\n q text;\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM partitions_view WHERE partition=t) THEN\n RETURN NULL;\n END IF;\n FOR q IN SELECT FORMAT(\n $q$\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n $q$,\n t,\n conname\n ) FROM pg_constraint\n WHERE conrelid=t::regclass::oid AND contype='c'\n LOOP\n EXECUTE q;\n END LOOP;\n RETURN t;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE OR REPLACE FUNCTION create_table_constraints(t text, _dtrange tstzrange, _edtrange tstzrange) RETURNS text AS $$\nDECLARE\n q text;\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM partitions_view WHERE partition=t) THEN\n RETURN NULL;\n END IF;\n RAISE NOTICE 'Creating Table Constraints for % % %', t, _dtrange, _edtrange;\n IF _dtrange = 'empty' AND _edtrange = 'empty' THEN\n q :=format(\n $q$\n DO $block$\n BEGIN\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n ALTER TABLE %I\n ADD CONSTRAINT %I\n CHECK (((datetime IS NULL) AND (end_datetime IS NULL))) NOT VALID\n ;\n ALTER TABLE %I\n VALIDATE CONSTRAINT %I\n ;\n\n\n\n EXCEPTION WHEN others THEN\n RAISE WARNING '%%, Issue Altering Constraints. Please run update_partition_stats(%I)', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n $block$;\n $q$,\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t),\n t\n );\n ELSE\n q :=format(\n $q$\n DO $block$\n BEGIN\n\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n ALTER TABLE %I\n ADD CONSTRAINT %I\n CHECK (\n (datetime >= %L)\n AND (datetime <= %L)\n AND (end_datetime >= %L)\n AND (end_datetime <= %L)\n ) NOT VALID\n ;\n ALTER TABLE %I\n VALIDATE CONSTRAINT %I\n ;\n\n\n\n EXCEPTION WHEN others THEN\n RAISE WARNING '%%, Issue Altering Constraints. Please run update_partition_stats(%I)', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n $block$;\n $q$,\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t),\n lower(_dtrange),\n upper(_dtrange),\n lower(_edtrange),\n upper(_edtrange),\n t,\n format('%s_dt', t),\n t\n );\n END IF;\n PERFORM run_or_queue(q);\n RETURN t;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION check_partition(\n _collection text,\n _dtrange tstzrange,\n _edtrange tstzrange\n) RETURNS text AS $$\nDECLARE\n c RECORD;\n pm RECORD;\n _partition_name text;\n _partition_dtrange tstzrange;\n _constraint_dtrange tstzrange;\n _constraint_edtrange tstzrange;\n q text;\n deferrable_q text;\n err_context text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=_collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', _collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n\n IF c.partition_trunc IS NOT NULL THEN\n _partition_dtrange := tstzrange(\n date_trunc(c.partition_trunc, lower(_dtrange)),\n date_trunc(c.partition_trunc, lower(_dtrange)) + (concat('1 ', c.partition_trunc))::interval,\n '[)'\n );\n ELSE\n _partition_dtrange := '[-infinity, infinity]'::tstzrange;\n END IF;\n\n IF NOT _partition_dtrange @> _dtrange THEN\n RAISE EXCEPTION 'dtrange % is greater than the partition size % for collection %', _dtrange, c.partition_trunc, _collection;\n END IF;\n\n\n IF c.partition_trunc = 'year' THEN\n _partition_name := format('_items_%s_%s', c.key, to_char(lower(_partition_dtrange),'YYYY'));\n ELSIF c.partition_trunc = 'month' THEN\n _partition_name := format('_items_%s_%s', c.key, to_char(lower(_partition_dtrange),'YYYYMM'));\n ELSE\n _partition_name := format('_items_%s', c.key);\n END IF;\n\n SELECT * INTO pm FROM partition_sys_meta WHERE collection=_collection AND partition_dtrange @> _dtrange;\n IF FOUND THEN\n RAISE NOTICE '% % %', _edtrange, _dtrange, pm;\n _constraint_edtrange :=\n tstzrange(\n least(\n lower(_edtrange),\n nullif(lower(pm.constraint_edtrange), '-infinity')\n ),\n greatest(\n upper(_edtrange),\n nullif(upper(pm.constraint_edtrange), 'infinity')\n ),\n '[]'\n );\n _constraint_dtrange :=\n tstzrange(\n least(\n lower(_dtrange),\n nullif(lower(pm.constraint_dtrange), '-infinity')\n ),\n greatest(\n upper(_dtrange),\n nullif(upper(pm.constraint_dtrange), 'infinity')\n ),\n '[]'\n );\n\n IF pm.constraint_edtrange @> _edtrange AND pm.constraint_dtrange @> _dtrange THEN\n RETURN pm.partition;\n ELSE\n PERFORM drop_table_constraints(_partition_name);\n END IF;\n ELSE\n _constraint_edtrange := _edtrange;\n _constraint_dtrange := _dtrange;\n END IF;\n RAISE NOTICE 'EXISTING CONSTRAINTS % %, NEW % %', pm.constraint_dtrange, pm.constraint_edtrange, _constraint_dtrange, _constraint_edtrange;\n RAISE NOTICE 'Creating partition % %', _partition_name, _partition_dtrange;\n IF c.partition_trunc IS NULL THEN\n q := format(\n $q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n GRANT ALL ON %I to pgstac_ingest;\n $q$,\n _partition_name,\n _collection,\n concat(_partition_name,'_pk'),\n _partition_name,\n _partition_name\n );\n ELSE\n q := format(\n $q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L) PARTITION BY RANGE (datetime);\n CREATE TABLE IF NOT EXISTS %I partition OF %I FOR VALUES FROM (%L) TO (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n GRANT ALL ON %I TO pgstac_ingest;\n $q$,\n format('_items_%s', c.key),\n _collection,\n _partition_name,\n format('_items_%s', c.key),\n lower(_partition_dtrange),\n upper(_partition_dtrange),\n format('%s_pk', _partition_name),\n _partition_name,\n _partition_name\n );\n END IF;\n\n BEGIN\n EXECUTE q;\n EXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', _partition_name;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\n END;\n PERFORM maintain_partitions(_partition_name);\n PERFORM update_partition_stats_q(_partition_name, true);\n REFRESH MATERIALIZED VIEW partitions;\n REFRESH MATERIALIZED VIEW partition_steps;\n RETURN _partition_name;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION repartition(_collection text, _partition_trunc text, triggered boolean DEFAULT FALSE) RETURNS text AS $$\nDECLARE\n c RECORD;\n q text;\n from_trunc text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=_collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', _collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n IF triggered THEN\n RAISE NOTICE 'Converting % to % partitioning via Trigger', _collection, _partition_trunc;\n ELSE\n RAISE NOTICE 'Converting % from using % to % partitioning', _collection, c.partition_trunc, _partition_trunc;\n IF c.partition_trunc IS NOT DISTINCT FROM _partition_trunc THEN\n RAISE NOTICE 'Collection % already set to use partition by %', _collection, _partition_trunc;\n RETURN _collection;\n END IF;\n END IF;\n\n IF EXISTS (SELECT 1 FROM partitions_view WHERE collection=_collection LIMIT 1) THEN\n EXECUTE format(\n $q$\n CREATE TEMP TABLE changepartitionstaging ON COMMIT DROP AS SELECT * FROM %I;\n DROP TABLE IF EXISTS %I CASCADE;\n WITH p AS (\n SELECT\n collection,\n CASE\n WHEN %L IS NULL THEN '-infinity'::timestamptz\n ELSE date_trunc(%L, datetime)\n END as d,\n tstzrange(min(datetime),max(datetime),'[]') as dtrange,\n tstzrange(min(end_datetime),max(end_datetime),'[]') as edtrange\n FROM changepartitionstaging\n GROUP BY 1,2\n ) SELECT check_partition(collection, dtrange, edtrange) FROM p;\n INSERT INTO items SELECT * FROM changepartitionstaging;\n DROP TABLE changepartitionstaging;\n $q$,\n concat('_items_', c.key),\n concat('_items_', c.key),\n c.partition_trunc,\n c.partition_trunc\n );\n END IF;\n RETURN _collection;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE OR REPLACE FUNCTION collections_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n q text;\n partition_name text := format('_items_%s', NEW.key);\n partition_exists boolean := false;\n partition_empty boolean := true;\n err_context text;\n loadtemp boolean := FALSE;\nBEGIN\n RAISE NOTICE 'Collection Trigger. % %', NEW.id, NEW.key;\n IF TG_OP = 'UPDATE' AND NEW.partition_trunc IS DISTINCT FROM OLD.partition_trunc THEN\n PERFORM repartition(NEW.id, NEW.partition_trunc, TRUE);\n END IF;\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER collections_trigger AFTER\nINSERT\nOR\nUPDATE ON collections\nFOR EACH ROW EXECUTE FUNCTION collections_trigger_func();\n\nCREATE OR REPLACE FUNCTION chunker(\n IN _where text,\n OUT s timestamptz,\n OUT e timestamptz\n) RETURNS SETOF RECORD AS $$\nDECLARE\n explain jsonb;\nBEGIN\n IF _where IS NULL THEN\n _where := ' TRUE ';\n END IF;\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s;', _where)\n INTO explain;\n RAISE DEBUG 'EXPLAIN: %', explain;\n\n RETURN QUERY\n WITH t AS (\n SELECT j->>0 as p FROM\n jsonb_path_query(\n explain,\n 'strict $.**.\"Relation Name\" ? (@ != null)'\n ) j\n ),\n parts AS (\n SELECT sdate, edate FROM t JOIN partition_steps ON (t.p = name)\n ),\n times AS (\n SELECT sdate FROM parts\n UNION\n SELECT edate FROM parts\n ),\n uniq AS (\n SELECT DISTINCT sdate FROM times ORDER BY sdate\n ),\n last AS (\n SELECT sdate, lead(sdate, 1) over () as edate FROM uniq\n )\n SELECT sdate, edate FROM last WHERE edate IS NOT NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION partition_queries(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN partitions text[] DEFAULT NULL\n) RETURNS SETOF text AS $$\nDECLARE\n query text;\n sdate timestamptz;\n edate timestamptz;\nBEGIN\nIF _where IS NULL OR trim(_where) = '' THEN\n _where = ' TRUE ';\nEND IF;\nRAISE NOTICE 'Getting chunks for % %', _where, _orderby;\nIF _orderby ILIKE 'datetime d%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 DESC LOOP\n RETURN NEXT format($q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n $q$,\n sdate,\n edate,\n _where,\n _orderby\n );\n END LOOP;\nELSIF _orderby ILIKE 'datetime a%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 ASC LOOP\n RETURN NEXT format($q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n $q$,\n sdate,\n edate,\n _where,\n _orderby\n );\n END LOOP;\nELSE\n query := format($q$\n SELECT * FROM items\n WHERE %s\n ORDER BY %s\n $q$, _where, _orderby\n );\n\n RETURN NEXT query;\n RETURN;\nEND IF;\n\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION partition_query_view(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN _limit int DEFAULT 10\n) RETURNS text AS $$\n WITH p AS (\n SELECT * FROM partition_queries(_where, _orderby) p\n )\n SELECT\n CASE WHEN EXISTS (SELECT 1 FROM p) THEN\n (SELECT format($q$\n SELECT * FROM (\n %s\n ) total LIMIT %s\n $q$,\n string_agg(\n format($q$ SELECT * FROM ( %s ) AS sub $q$, p),\n '\n UNION ALL\n '\n ),\n _limit\n ))\n ELSE NULL\n END FROM p;\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION q_to_tsquery (input text)\n RETURNS tsquery\n AS $$\nDECLARE\n processed_text text;\n temp_text text;\n quote_array text[];\n placeholder text := '@QUOTE@';\nBEGIN\n -- Extract all quoted phrases and store in array\n quote_array := regexp_matches(input, '\"[^\"]*\"', 'g');\n\n -- Replace each quoted part with a unique placeholder if there are any quoted phrases\n IF array_length(quote_array, 1) IS NOT NULL THEN\n processed_text := input;\n FOR i IN array_lower(quote_array, 1) .. array_upper(quote_array, 1) LOOP\n processed_text := replace(processed_text, quote_array[i], placeholder || i || placeholder);\n END LOOP;\n ELSE\n processed_text := input;\n END IF;\n\n -- Replace non-quoted text using regular expressions\n\n -- , -> |\n processed_text := regexp_replace(processed_text, ',(?=(?:[^\"]*\"[^\"]*\")*[^\"]*$)', ' | ', 'g');\n\n -- and -> &\n processed_text := regexp_replace(processed_text, '\\s+AND\\s+', ' & ', 'gi');\n\n -- or -> |\n processed_text := regexp_replace(processed_text, '\\s+OR\\s+', ' | ', 'gi');\n\n -- +term -> & term\n processed_text := regexp_replace(processed_text, '\\+([a-zA-Z0-9_]+)', '& \\1', 'g');\n\n -- -term -> ! term\n processed_text := regexp_replace(processed_text, '\\-([a-zA-Z0-9_]+)', '& ! \\1', 'g');\n\n -- Replace placeholders back with quoted phrases if there were any\n IF array_length(quote_array, 1) IS NOT NULL THEN\n FOR i IN array_lower(quote_array, 1) .. array_upper(quote_array, 1) LOOP\n processed_text := replace(processed_text, placeholder || i || placeholder, '''' || substring(quote_array[i] from 2 for length(quote_array[i]) - 2) || '''');\n END LOOP;\n END IF;\n\n -- Print processed_text to the console for debugging purposes\n RAISE NOTICE 'processed_text: %', processed_text;\n\n RETURN to_tsquery(processed_text);\nEND;\n$$\nLANGUAGE plpgsql;\n\n\nCREATE OR REPLACE FUNCTION stac_search_to_where(j jsonb) RETURNS text AS $$\nDECLARE\n where_segments text[];\n _where text;\n dtrange tstzrange;\n collections text[];\n geom geometry;\n sdate timestamptz;\n edate timestamptz;\n filterlang text;\n filter jsonb := j->'filter';\n ft_query tsquery;\nBEGIN\n IF j ? 'ids' THEN\n where_segments := where_segments || format('id = ANY (%L) ', to_text_array(j->'ids'));\n END IF;\n\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n where_segments := where_segments || format('collection = ANY (%L) ', collections);\n END IF;\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n\n where_segments := where_segments || format(' datetime <= %L::timestamptz AND end_datetime >= %L::timestamptz ',\n edate,\n sdate\n );\n END IF;\n\n IF j ? 'q' THEN\n ft_query := q_to_tsquery(j->>'q');\n where_segments := where_segments || format(\n $quote$\n (\n to_tsvector('english', content->'properties'->>'description') ||\n to_tsvector('english', coalesce(content->'properties'->>'title', '')) ||\n to_tsvector('english', coalesce(content->'properties'->>'keywords', ''))\n ) @@ %L\n $quote$,\n ft_query\n );\n END IF;\n\n geom := stac_geom(j);\n IF geom IS NOT NULL THEN\n where_segments := where_segments || format('st_intersects(geometry, %L)',geom);\n END IF;\n\n filterlang := COALESCE(\n j->>'filter-lang',\n get_setting('default_filter_lang', j->'conf')\n );\n IF NOT filter @? '$.**.op' THEN\n filterlang := 'cql-json';\n END IF;\n\n IF filterlang NOT IN ('cql-json','cql2-json') AND j ? 'filter' THEN\n RAISE EXCEPTION '% is not a supported filter-lang. Please use cql-json or cql2-json.', filterlang;\n END IF;\n\n IF j ? 'query' AND j ? 'filter' THEN\n RAISE EXCEPTION 'Can only use either query or filter at one time.';\n END IF;\n\n IF j ? 'query' THEN\n filter := query_to_cql2(j->'query');\n ELSIF filterlang = 'cql-json' THEN\n filter := cql1_to_cql2(filter);\n END IF;\n RAISE NOTICE 'FILTER: %', filter;\n where_segments := where_segments || cql2_query(filter);\n IF cardinality(where_segments) < 1 THEN\n RETURN ' TRUE ';\n END IF;\n\n _where := array_to_string(array_remove(where_segments, NULL), ' AND ');\n\n IF _where IS NULL OR BTRIM(_where) = '' THEN\n RETURN ' TRUE ';\n END IF;\n RETURN _where;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION parse_sort_dir(_dir text, reverse boolean default false) RETURNS text AS $$\n WITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n ) SELECT\n CASE\n WHEN NOT reverse THEN d\n WHEN d = 'ASC' THEN 'DESC'\n WHEN d = 'DESC' THEN 'ASC'\n END\n FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION sort_dir_to_op(_dir text, prev boolean default false) RETURNS text AS $$\n WITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n ) SELECT\n CASE\n WHEN d = 'ASC' AND prev THEN '<='\n WHEN d = 'DESC' AND prev THEN '>='\n WHEN d = 'ASC' THEN '>='\n WHEN d = 'DESC' THEN '<='\n END\n FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION sort_sqlorderby(\n _search jsonb DEFAULT NULL,\n reverse boolean DEFAULT FALSE\n) RETURNS text AS $$\n WITH sortby AS (\n SELECT coalesce(_search->'sortby','[{\"field\":\"datetime\", \"direction\":\"desc\"}]') as sort\n ), withid AS (\n SELECT CASE\n WHEN sort @? '$[*] ? (@.field == \"id\")' THEN sort\n ELSE sort || '[{\"field\":\"id\", \"direction\":\"desc\"}]'::jsonb\n END as sort\n FROM sortby\n ), withid_rows AS (\n SELECT jsonb_array_elements(sort) as value FROM withid\n ),sorts AS (\n SELECT\n coalesce(\n (queryable(value->>'field')).expression\n ) as key,\n parse_sort_dir(value->>'direction', reverse) as dir\n FROM withid_rows\n )\n SELECT array_to_string(\n array_agg(concat(key, ' ', dir)),\n ', '\n ) FROM sorts;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_sort_dir(sort_item jsonb) RETURNS text AS $$\n SELECT CASE WHEN sort_item->>'direction' ILIKE 'desc%' THEN 'DESC' ELSE 'ASC' END;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION get_token_val_str(\n _field text,\n _item items\n) RETURNS text AS $$\nDECLARE\n q text;\n literal text;\nBEGIN\n q := format($q$ SELECT quote_literal(%s) FROM (SELECT $1.*) as r;$q$, _field);\n EXECUTE q INTO literal USING _item;\n RETURN literal;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n\nCREATE OR REPLACE FUNCTION get_token_record(IN _token text, OUT prev BOOLEAN, OUT item items) RETURNS RECORD AS $$\nDECLARE\n _itemid text := _token;\n _collectionid text;\nBEGIN\n IF _token IS NULL THEN\n RETURN;\n END IF;\n RAISE NOTICE 'Looking for token: %', _token;\n prev := FALSE;\n IF _token ILIKE 'prev:%' THEN\n _itemid := replace(_token, 'prev:','');\n prev := TRUE;\n ELSIF _token ILIKE 'next:%' THEN\n _itemid := replace(_token, 'next:', '');\n END IF;\n SELECT id INTO _collectionid FROM collections WHERE _itemid LIKE concat(id,':%');\n IF FOUND THEN\n _itemid := replace(_itemid, concat(_collectionid,':'), '');\n SELECT * INTO item FROM items WHERE id=_itemid AND collection=_collectionid;\n ELSE\n SELECT * INTO item FROM items WHERE id=_itemid;\n END IF;\n IF item IS NULL THEN\n RAISE EXCEPTION 'Could not find item using token: % item: % collection: %', _token, _itemid, _collectionid;\n END IF;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\n\nCREATE OR REPLACE FUNCTION get_token_filter(\n _sortby jsonb DEFAULT '[{\"field\":\"datetime\",\"direction\":\"desc\"}]'::jsonb,\n token_item items DEFAULT NULL,\n prev boolean DEFAULT FALSE,\n inclusive boolean DEFAULT FALSE\n) RETURNS text AS $$\nDECLARE\n ltop text := '<';\n gtop text := '>';\n dir text;\n sort record;\n orfilter text := '';\n orfilters text[] := '{}'::text[];\n andfilters text[] := '{}'::text[];\n output text;\n token_where text;\nBEGIN\n IF _sortby IS NULL OR _sortby = '[]'::jsonb THEN\n _sortby := '[{\"field\":\"datetime\",\"direction\":\"desc\"}]'::jsonb;\n END IF;\n _sortby := _sortby || jsonb_build_object('field','id','direction',_sortby->0->>'direction');\n RAISE NOTICE 'Getting Token Filter. % %', _sortby, token_item;\n IF inclusive THEN\n orfilters := orfilters || format('( id=%L AND collection=%L )' , token_item.id, token_item.collection);\n END IF;\n\n FOR sort IN\n WITH s1 AS (\n SELECT\n _row,\n (queryable(value->>'field')).expression as _field,\n (value->>'field' = 'id') as _isid,\n get_sort_dir(value) as _dir\n FROM jsonb_array_elements(_sortby)\n WITH ORDINALITY AS t(value, _row)\n )\n SELECT\n _row,\n _field,\n _dir,\n get_token_val_str(_field, token_item) as _val\n FROM s1\n WHERE _row <= (SELECT min(_row) FROM s1 WHERE _isid)\n LOOP\n orfilter := NULL;\n RAISE NOTICE 'SORT: %', sort;\n IF sort._val IS NOT NULL AND ((prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC')) THEN\n orfilter := format($f$(\n (%s %s %s) OR (%s IS NULL)\n )$f$,\n sort._field,\n ltop,\n sort._val,\n sort._val\n );\n ELSIF sort._val IS NULL AND ((prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC')) THEN\n RAISE NOTICE '< but null';\n orfilter := format('%s IS NOT NULL', sort._field);\n ELSIF sort._val IS NULL THEN\n RAISE NOTICE '> but null';\n ELSE\n orfilter := format($f$(\n (%s %s %s) OR (%s IS NULL)\n )$f$,\n sort._field,\n gtop,\n sort._val,\n sort._field\n );\n END IF;\n RAISE NOTICE 'ORFILTER: %', orfilter;\n\n IF orfilter IS NOT NULL THEN\n IF sort._row = 1 THEN\n orfilters := orfilters || orfilter;\n ELSE\n orfilters := orfilters || format('(%s AND %s)', array_to_string(andfilters, ' AND '), orfilter);\n END IF;\n END IF;\n IF sort._val IS NOT NULL THEN\n andfilters := andfilters || format('%s = %s', sort._field, sort._val);\n ELSE\n andfilters := andfilters || format('%s IS NULL', sort._field);\n END IF;\n END LOOP;\n\n output := array_to_string(orfilters, ' OR ');\n\n token_where := concat('(',coalesce(output,'true'),')');\n IF trim(token_where) = '' THEN\n token_where := NULL;\n END IF;\n RAISE NOTICE 'TOKEN_WHERE: %',token_where;\n RETURN token_where;\n END;\n$$ LANGUAGE PLPGSQL SET transform_null_equals TO TRUE\n;\n\nCREATE OR REPLACE FUNCTION search_tohash(jsonb) RETURNS jsonb AS $$\n SELECT $1 - '{token,limit,context,includes,excludes}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION search_hash(jsonb, jsonb) RETURNS text AS $$\n SELECT md5(concat(search_tohash($1)::text,$2::text));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TABLE IF NOT EXISTS searches(\n hash text GENERATED ALWAYS AS (search_hash(search, metadata)) STORED PRIMARY KEY,\n search jsonb NOT NULL,\n _where text,\n orderby text,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n metadata jsonb DEFAULT '{}'::jsonb NOT NULL\n);\n\nCREATE TABLE IF NOT EXISTS search_wheres(\n id bigint generated always as identity primary key,\n _where text NOT NULL,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n statslastupdated timestamptz,\n estimated_count bigint,\n estimated_cost float,\n time_to_estimate float,\n total_count bigint,\n time_to_count float,\n partitions text[]\n);\n\nCREATE INDEX IF NOT EXISTS search_wheres_partitions ON search_wheres USING GIN (partitions);\nCREATE UNIQUE INDEX IF NOT EXISTS search_wheres_where ON search_wheres ((md5(_where)));\n\nCREATE OR REPLACE FUNCTION where_stats(\n inwhere text,\n updatestats boolean default false,\n conf jsonb default null\n) RETURNS search_wheres AS $$\nDECLARE\n t timestamptz;\n i interval;\n explain_json jsonb;\n partitions text[];\n sw search_wheres%ROWTYPE;\n inwhere_hash text := md5(inwhere);\n _context text := lower(context(conf));\n _stats_ttl interval := context_stats_ttl(conf);\n _estimated_cost_threshold float := context_estimated_cost(conf);\n _estimated_count_threshold int := context_estimated_count(conf);\n ro bool := pgstac.readonly(conf);\nBEGIN\n -- If updatestats is true then set ttl to 0\n IF updatestats THEN\n RAISE DEBUG 'Updatestats set to TRUE, setting TTL to 0';\n _stats_ttl := '0'::interval;\n END IF;\n\n -- If we don't need to calculate context, just return\n IF _context = 'off' THEN\n sw._where = inwhere;\n RETURN sw;\n END IF;\n\n -- Get any stats that we have.\n IF NOT ro THEN\n -- If there is a lock where another process is\n -- updating the stats, wait so that we don't end up calculating a bunch of times.\n SELECT * INTO sw FROM search_wheres WHERE md5(_where)=inwhere_hash FOR UPDATE;\n ELSE\n SELECT * INTO sw FROM search_wheres WHERE md5(_where)=inwhere_hash;\n END IF;\n\n -- If there is a cached row, figure out if we need to update\n IF\n sw IS NOT NULL\n AND sw.statslastupdated IS NOT NULL\n AND sw.total_count IS NOT NULL\n AND now() - sw.statslastupdated <= _stats_ttl\n THEN\n -- we have a cached row with data that is within our ttl\n RAISE DEBUG 'Stats present in table and lastupdated within ttl: %', sw;\n IF NOT ro THEN\n RAISE DEBUG 'Updating search_wheres only bumping lastused and usecount';\n UPDATE search_wheres SET\n lastused = now(),\n usecount = search_wheres.usecount + 1\n WHERE md5(_where) = inwhere_hash\n RETURNING * INTO sw;\n END IF;\n RAISE DEBUG 'Returning cached counts. %', sw;\n RETURN sw;\n END IF;\n\n -- Calculate estimated cost and rows\n -- Use explain to get estimated count/cost\n IF sw.estimated_count IS NULL OR sw.estimated_cost IS NULL THEN\n RAISE DEBUG 'Calculating estimated stats';\n t := clock_timestamp();\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s', inwhere)\n INTO explain_json;\n RAISE DEBUG 'Time for just the explain: %', clock_timestamp() - t;\n i := clock_timestamp() - t;\n\n sw.estimated_count := explain_json->0->'Plan'->'Plan Rows';\n sw.estimated_cost := explain_json->0->'Plan'->'Total Cost';\n sw.time_to_estimate := extract(epoch from i);\n END IF;\n\n RAISE DEBUG 'ESTIMATED_COUNT: %, THRESHOLD %', sw.estimated_count, _estimated_count_threshold;\n RAISE DEBUG 'ESTIMATED_COST: %, THRESHOLD %', sw.estimated_cost, _estimated_cost_threshold;\n\n -- If context is set to auto and the costs are within the threshold return the estimated costs\n IF\n _context = 'auto'\n AND sw.estimated_count >= _estimated_count_threshold\n AND sw.estimated_cost >= _estimated_cost_threshold\n THEN\n IF NOT ro THEN\n INSERT INTO search_wheres (\n _where,\n lastused,\n usecount,\n statslastupdated,\n estimated_count,\n estimated_cost,\n time_to_estimate,\n total_count,\n time_to_count\n ) VALUES (\n inwhere,\n now(),\n 1,\n now(),\n sw.estimated_count,\n sw.estimated_cost,\n sw.time_to_estimate,\n null,\n null\n ) ON CONFLICT ((md5(_where)))\n DO UPDATE SET\n lastused = EXCLUDED.lastused,\n usecount = search_wheres.usecount + 1,\n statslastupdated = EXCLUDED.statslastupdated,\n estimated_count = EXCLUDED.estimated_count,\n estimated_cost = EXCLUDED.estimated_cost,\n time_to_estimate = EXCLUDED.time_to_estimate,\n total_count = EXCLUDED.total_count,\n time_to_count = EXCLUDED.time_to_count\n RETURNING * INTO sw;\n END IF;\n RAISE DEBUG 'Estimates are within thresholds, returning estimates. %', sw;\n RETURN sw;\n END IF;\n\n -- Calculate Actual Count\n t := clock_timestamp();\n RAISE NOTICE 'Calculating actual count...';\n EXECUTE format(\n 'SELECT count(*) FROM items WHERE %s',\n inwhere\n ) INTO sw.total_count;\n i := clock_timestamp() - t;\n RAISE NOTICE 'Actual Count: % -- %', sw.total_count, i;\n sw.time_to_count := extract(epoch FROM i);\n\n IF NOT ro THEN\n INSERT INTO search_wheres (\n _where,\n lastused,\n usecount,\n statslastupdated,\n estimated_count,\n estimated_cost,\n time_to_estimate,\n total_count,\n time_to_count\n ) VALUES (\n inwhere,\n now(),\n 1,\n now(),\n sw.estimated_count,\n sw.estimated_cost,\n sw.time_to_estimate,\n sw.total_count,\n sw.time_to_count\n ) ON CONFLICT ((md5(_where)))\n DO UPDATE SET\n lastused = EXCLUDED.lastused,\n usecount = search_wheres.usecount + 1,\n statslastupdated = EXCLUDED.statslastupdated,\n estimated_count = EXCLUDED.estimated_count,\n estimated_cost = EXCLUDED.estimated_cost,\n time_to_estimate = EXCLUDED.time_to_estimate,\n total_count = EXCLUDED.total_count,\n time_to_count = EXCLUDED.time_to_count\n RETURNING * INTO sw;\n END IF;\n RAISE DEBUG 'Returning with actual count. %', sw;\n RETURN sw;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION search_query(\n _search jsonb = '{}'::jsonb,\n updatestats boolean = false,\n _metadata jsonb = '{}'::jsonb\n) RETURNS searches AS $$\nDECLARE\n search searches%ROWTYPE;\n cached_search searches%ROWTYPE;\n pexplain jsonb;\n t timestamptz;\n i interval;\n doupdate boolean := FALSE;\n insertfound boolean := FALSE;\n ro boolean := pgstac.readonly();\n found_search text;\nBEGIN\n RAISE NOTICE 'SEARCH: %', _search;\n -- Calculate hash, where clause, and order by statement\n search.search := _search;\n search.metadata := _metadata;\n search.hash := search_hash(_search, _metadata);\n search._where := stac_search_to_where(_search);\n search.orderby := sort_sqlorderby(_search);\n search.lastused := now();\n search.usecount := 1;\n\n -- If we are in read only mode, directly return search\n IF ro THEN\n RETURN search;\n END IF;\n\n RAISE NOTICE 'Updating Statistics for search: %s', search;\n -- Update statistics for times used and and when last used\n -- If the entry is locked, rather than waiting, skip updating the stats\n INSERT INTO searches (search, lastused, usecount, metadata)\n VALUES (search.search, now(), 1, search.metadata)\n ON CONFLICT DO NOTHING\n RETURNING * INTO cached_search\n ;\n\n IF NOT FOUND OR cached_search IS NULL THEN\n UPDATE searches SET\n lastused = now(),\n usecount = searches.usecount + 1\n WHERE hash = (\n SELECT hash FROM searches WHERE hash=search.hash FOR UPDATE SKIP LOCKED\n )\n RETURNING * INTO cached_search\n ;\n END IF;\n\n IF cached_search IS NOT NULL THEN\n cached_search._where = search._where;\n cached_search.orderby = search.orderby;\n RETURN cached_search;\n END IF;\n RETURN search;\n\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE OR REPLACE FUNCTION search_fromhash(\n _hash text\n) RETURNS searches AS $$\n SELECT * FROM search_query((SELECT search FROM searches WHERE hash=_hash LIMIT 1));\n$$ LANGUAGE SQL STRICT;\n\nCREATE OR REPLACE FUNCTION search_rows(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN partitions text[] DEFAULT NULL,\n IN _limit int DEFAULT 10\n) RETURNS SETOF items AS $$\nDECLARE\n base_query text;\n query text;\n sdate timestamptz;\n edate timestamptz;\n n int;\n records_left int := _limit;\n timer timestamptz := clock_timestamp();\n full_timer timestamptz := clock_timestamp();\nBEGIN\nIF _where IS NULL OR trim(_where) = '' THEN\n _where = ' TRUE ';\nEND IF;\nRAISE NOTICE 'Getting chunks for % %', _where, _orderby;\n\nbase_query := $q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n LIMIT %L\n$q$;\n\nIF _orderby ILIKE 'datetime d%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 DESC LOOP\n RAISE NOTICE 'Running Query for % to %. %', sdate, edate, age_ms(full_timer);\n query := format(\n base_query,\n sdate,\n edate,\n _where,\n _orderby,\n records_left\n );\n RAISE DEBUG 'QUERY: %', query;\n timer := clock_timestamp();\n RETURN QUERY EXECUTE query;\n\n GET DIAGNOSTICS n = ROW_COUNT;\n records_left := records_left - n;\n RAISE NOTICE 'Returned %/% Rows From % to %. % to go. Time: %ms', n, _limit, sdate, edate, records_left, age_ms(timer);\n timer := clock_timestamp();\n IF records_left <= 0 THEN\n RAISE NOTICE 'SEARCH_ROWS TOOK %ms', age_ms(full_timer);\n RETURN;\n END IF;\n END LOOP;\nELSIF _orderby ILIKE 'datetime a%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 ASC LOOP\n RAISE NOTICE 'Running Query for % to %. %', sdate, edate, age_ms(full_timer);\n query := format(\n base_query,\n sdate,\n edate,\n _where,\n _orderby,\n records_left\n );\n RAISE DEBUG 'QUERY: %', query;\n timer := clock_timestamp();\n RETURN QUERY EXECUTE query;\n\n GET DIAGNOSTICS n = ROW_COUNT;\n records_left := records_left - n;\n RAISE NOTICE 'Returned %/% Rows From % to %. % to go. Time: %ms', n, _limit, sdate, edate, records_left, age_ms(timer);\n timer := clock_timestamp();\n IF records_left <= 0 THEN\n RAISE NOTICE 'SEARCH_ROWS TOOK %ms', age_ms(full_timer);\n RETURN;\n END IF;\n END LOOP;\nELSE\n query := format($q$\n SELECT * FROM items\n WHERE %s\n ORDER BY %s\n LIMIT %L\n $q$, _where, _orderby, _limit\n );\n RAISE DEBUG 'QUERY: %', query;\n timer := clock_timestamp();\n RETURN QUERY EXECUTE query;\n RAISE NOTICE 'FULL QUERY TOOK %ms', age_ms(timer);\nEND IF;\nRAISE NOTICE 'SEARCH_ROWS TOOK %ms', age_ms(full_timer);\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE UNLOGGED TABLE format_item_cache(\n id text,\n collection text,\n fields text,\n hydrated bool,\n output jsonb,\n lastused timestamptz DEFAULT now(),\n usecount int DEFAULT 1,\n timetoformat float,\n PRIMARY KEY (collection, id, fields, hydrated)\n);\nCREATE INDEX ON format_item_cache (lastused);\n\nCREATE OR REPLACE FUNCTION format_item(_item items, _fields jsonb DEFAULT '{}', _hydrated bool DEFAULT TRUE) RETURNS jsonb AS $$\nDECLARE\n cache bool := get_setting_bool('format_cache');\n _output jsonb := null;\n t timestamptz := clock_timestamp();\nBEGIN\n IF cache THEN\n SELECT output INTO _output FROM format_item_cache\n WHERE id=_item.id AND collection=_item.collection AND fields=_fields::text AND hydrated=_hydrated;\n END IF;\n IF _output IS NULL THEN\n IF _hydrated THEN\n _output := content_hydrate(_item, _fields);\n ELSE\n _output := content_nonhydrated(_item, _fields);\n END IF;\n END IF;\n IF cache THEN\n INSERT INTO format_item_cache (id, collection, fields, hydrated, output, timetoformat)\n VALUES (_item.id, _item.collection, _fields::text, _hydrated, _output, age_ms(t))\n ON CONFLICT(collection, id, fields, hydrated) DO\n UPDATE\n SET lastused=now(), usecount = format_item_cache.usecount + 1\n ;\n END IF;\n RETURN _output;\n\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION search(_search jsonb = '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n searches searches%ROWTYPE;\n _where text;\n orderby text;\n search_where search_wheres%ROWTYPE;\n total_count bigint;\n token record;\n token_prev boolean;\n token_item items%ROWTYPE;\n token_where text;\n full_where text;\n init_ts timestamptz := clock_timestamp();\n timer timestamptz := clock_timestamp();\n hydrate bool := NOT (_search->'conf'->>'nohydrate' IS NOT NULL AND (_search->'conf'->>'nohydrate')::boolean = true);\n prev text;\n next text;\n context jsonb;\n collection jsonb;\n out_records jsonb;\n out_len int;\n _limit int := coalesce((_search->>'limit')::int, 10);\n _querylimit int;\n _fields jsonb := coalesce(_search->'fields', '{}'::jsonb);\n has_prev boolean := FALSE;\n has_next boolean := FALSE;\n links jsonb := '[]'::jsonb;\n base_url text:= concat(rtrim(base_url(_search->'conf'),'/'));\nBEGIN\n searches := search_query(_search);\n _where := searches._where;\n orderby := searches.orderby;\n search_where := where_stats(_where);\n total_count := coalesce(search_where.total_count, search_where.estimated_count);\n RAISE NOTICE 'SEARCH:TOKEN: %', _search->>'token';\n token := get_token_record(_search->>'token');\n RAISE NOTICE '***TOKEN: %', token;\n _querylimit := _limit + 1;\n IF token IS NOT NULL THEN\n token_prev := token.prev;\n token_item := token.item;\n token_where := get_token_filter(_search->'sortby', token_item, token_prev, FALSE);\n RAISE DEBUG 'TOKEN_WHERE: % (%ms from search start)', token_where, age_ms(timer);\n IF token_prev THEN -- if we are using a prev token, we know has_next is true\n RAISE DEBUG 'There is a previous token, so automatically setting has_next to true';\n has_next := TRUE;\n orderby := sort_sqlorderby(_search, TRUE);\n ELSE\n RAISE DEBUG 'There is a next token, so automatically setting has_prev to true';\n has_prev := TRUE;\n\n END IF;\n ELSE -- if there was no token, we know there is no prev\n RAISE DEBUG 'There is no token, so we know there is no prev. setting has_prev to false';\n has_prev := FALSE;\n END IF;\n\n full_where := concat_ws(' AND ', _where, token_where);\n RAISE NOTICE 'FULL WHERE CLAUSE: %', full_where;\n RAISE NOTICE 'Time to get counts and build query %', age_ms(timer);\n timer := clock_timestamp();\n\n IF hydrate THEN\n RAISE NOTICE 'Getting hydrated data.';\n ELSE\n RAISE NOTICE 'Getting non-hydrated data.';\n END IF;\n RAISE NOTICE 'CACHE SET TO %', get_setting_bool('format_cache');\n RAISE NOTICE 'Time to set hydration/formatting %', age_ms(timer);\n timer := clock_timestamp();\n SELECT jsonb_agg(format_item(i, _fields, hydrate)) INTO out_records\n FROM search_rows(\n full_where,\n orderby,\n search_where.partitions,\n _querylimit\n ) as i;\n\n RAISE NOTICE 'Time to fetch rows %', age_ms(timer);\n timer := clock_timestamp();\n\n\n IF token_prev THEN\n out_records := flip_jsonb_array(out_records);\n END IF;\n\n RAISE NOTICE 'Query returned % records.', jsonb_array_length(out_records);\n RAISE DEBUG 'TOKEN: % %', token_item.id, token_item.collection;\n RAISE DEBUG 'RECORD_1: % %', out_records->0->>'id', out_records->0->>'collection';\n RAISE DEBUG 'RECORD-1: % %', out_records->-1->>'id', out_records->-1->>'collection';\n\n -- REMOVE records that were from our token\n IF out_records->0->>'id' = token_item.id AND out_records->0->>'collection' = token_item.collection THEN\n out_records := out_records - 0;\n ELSIF out_records->-1->>'id' = token_item.id AND out_records->-1->>'collection' = token_item.collection THEN\n out_records := out_records - -1;\n END IF;\n\n out_len := jsonb_array_length(out_records);\n\n IF out_len = _limit + 1 THEN\n IF token_prev THEN\n has_prev := TRUE;\n out_records := out_records - 0;\n ELSE\n has_next := TRUE;\n out_records := out_records - -1;\n END IF;\n END IF;\n\n\n links := links || jsonb_build_object(\n 'rel', 'root',\n 'type', 'application/json',\n 'href', base_url\n ) || jsonb_build_object(\n 'rel', 'self',\n 'type', 'application/json',\n 'href', concat(base_url, '/search')\n );\n\n IF has_next THEN\n next := concat(out_records->-1->>'collection', ':', out_records->-1->>'id');\n RAISE NOTICE 'HAS NEXT | %', next;\n links := links || jsonb_build_object(\n 'rel', 'next',\n 'type', 'application/geo+json',\n 'method', 'GET',\n 'href', concat(base_url, '/search?token=next:', next)\n );\n END IF;\n\n IF has_prev THEN\n prev := concat(out_records->0->>'collection', ':', out_records->0->>'id');\n RAISE NOTICE 'HAS PREV | %', prev;\n links := links || jsonb_build_object(\n 'rel', 'prev',\n 'type', 'application/geo+json',\n 'method', 'GET',\n 'href', concat(base_url, '/search?token=prev:', prev)\n );\n END IF;\n\n RAISE NOTICE 'Time to get prev/next %', age_ms(timer);\n timer := clock_timestamp();\n\n\n collection := jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb),\n 'links', links\n );\n\n\n\n IF context(_search->'conf') != 'off' THEN\n collection := collection || jsonb_strip_nulls(jsonb_build_object(\n 'numberMatched', total_count,\n 'numberReturned', coalesce(jsonb_array_length(out_records), 0)\n ));\n ELSE\n collection := collection || jsonb_strip_nulls(jsonb_build_object(\n 'numberReturned', coalesce(jsonb_array_length(out_records), 0)\n ));\n END IF;\n\n IF get_setting_bool('timing', _search->'conf') THEN\n collection = collection || jsonb_build_object('timing', age_ms(init_ts));\n END IF;\n\n RAISE NOTICE 'Time to build final json %', age_ms(timer);\n timer := clock_timestamp();\n\n RAISE NOTICE 'Total Time: %', age_ms(current_timestamp);\n RAISE NOTICE 'RETURNING % records. NEXT: %. PREV: %', collection->>'numberReturned', collection->>'next', collection->>'prev';\n RETURN collection;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION search_cursor(_search jsonb = '{}'::jsonb) RETURNS refcursor AS $$\nDECLARE\n curs refcursor;\n searches searches%ROWTYPE;\n _where text;\n _orderby text;\n q text;\n\nBEGIN\n searches := search_query(_search);\n _where := searches._where;\n _orderby := searches.orderby;\n\n OPEN curs FOR\n WITH p AS (\n SELECT * FROM partition_queries(_where, _orderby) p\n )\n SELECT\n CASE WHEN EXISTS (SELECT 1 FROM p) THEN\n (SELECT format($q$\n SELECT * FROM (\n %s\n ) total\n $q$,\n string_agg(\n format($q$ SELECT * FROM ( %s ) AS sub $q$, p),\n '\n UNION ALL\n '\n )\n ))\n ELSE NULL\n END FROM p;\n RETURN curs;\nEND;\n$$ LANGUAGE PLPGSQL;\nCREATE OR REPLACE VIEW collections_asitems AS\nSELECT\n id,\n geometry,\n 'collections' AS collection,\n datetime,\n end_datetime,\n jsonb_build_object(\n 'properties', content - '{links,assets,stac_version,stac_extensions}',\n 'links', content->'links',\n 'assets', content->'assets',\n 'stac_version', content->'stac_version',\n 'stac_extensions', content->'stac_extensions'\n ) AS content,\n content as collectionjson\nFROM collections;\n\n\nCREATE OR REPLACE FUNCTION collection_search_matched(\n IN _search jsonb DEFAULT '{}'::jsonb,\n OUT matched bigint\n) RETURNS bigint AS $$\nDECLARE\n _where text := stac_search_to_where(_search);\nBEGIN\n EXECUTE format(\n $query$\n SELECT\n count(*)\n FROM\n collections_asitems\n WHERE %s\n ;\n $query$,\n _where\n ) INTO matched;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION collection_search_rows(\n _search jsonb DEFAULT '{}'::jsonb\n) RETURNS SETOF jsonb AS $$\nDECLARE\n _where text := stac_search_to_where(_search);\n _limit int := coalesce((_search->>'limit')::int, 10);\n _fields jsonb := coalesce(_search->'fields', '{}'::jsonb);\n _orderby text;\n _offset int := COALESCE((_search->>'offset')::int, 0);\nBEGIN\n _orderby := sort_sqlorderby(\n jsonb_build_object(\n 'sortby',\n coalesce(\n _search->'sortby',\n '[{\"field\": \"id\", \"direction\": \"asc\"}]'::jsonb\n )\n )\n );\n RETURN QUERY EXECUTE format(\n $query$\n SELECT\n jsonb_fields(collectionjson, %L) as c\n FROM\n collections_asitems\n WHERE %s\n ORDER BY %s\n LIMIT %L\n OFFSET %L\n ;\n $query$,\n _fields,\n _where,\n _orderby,\n _limit,\n _offset\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION collection_search(\n _search jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\nDECLARE\n out_records jsonb;\n number_matched bigint := collection_search_matched(_search);\n number_returned bigint;\n _limit int := coalesce((_search->>'limit')::float::int, 10);\n _offset int := coalesce((_search->>'offset')::float::int, 0);\n links jsonb := '[]';\n ret jsonb;\n base_url text:= concat(rtrim(base_url(_search->'conf'),'/'), '/collections');\n prevoffset int;\n nextoffset int;\nBEGIN\n SELECT\n coalesce(jsonb_agg(c), '[]')\n INTO out_records\n FROM collection_search_rows(_search) c;\n\n number_returned := jsonb_array_length(out_records);\n RAISE DEBUG 'nm: %, nr: %, l:%, o:%', number_matched, number_returned, _limit, _offset;\n\n\n\n IF _limit <= number_matched AND number_matched > 0 THEN --need to have paging links\n nextoffset := least(_offset + _limit, number_matched - 1);\n prevoffset := greatest(_offset - _limit, 0);\n\n IF _offset > 0 THEN\n links := links || jsonb_build_object(\n 'rel', 'prev',\n 'type', 'application/json',\n 'method', 'GET' ,\n 'href', base_url,\n 'body', jsonb_build_object('offset', prevoffset),\n 'merge', TRUE\n );\n END IF;\n\n IF (_offset + _limit < number_matched) THEN\n links := links || jsonb_build_object(\n 'rel', 'next',\n 'type', 'application/json',\n 'method', 'GET' ,\n 'href', base_url,\n 'body', jsonb_build_object('offset', nextoffset),\n 'merge', TRUE\n );\n END IF;\n\n END IF;\n\n ret := jsonb_build_object(\n 'collections', out_records,\n 'numberMatched', number_matched,\n 'numberReturned', number_returned,\n 'links', links\n );\n RETURN ret;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION tileenvelope(zoom int, x int, y int) RETURNS geometry AS $$\nWITH t AS (\n SELECT\n 20037508.3427892 as merc_max,\n -20037508.3427892 as merc_min,\n (2 * 20037508.3427892) / (2 ^ zoom) as tile_size\n)\nSELECT st_makeenvelope(\n merc_min + (tile_size * x),\n merc_max - (tile_size * (y + 1)),\n merc_min + (tile_size * (x + 1)),\n merc_max - (tile_size * y),\n 3857\n) FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;DROP FUNCTION IF EXISTS mercgrid;\n\n\nCREATE OR REPLACE FUNCTION ftime() RETURNS interval as $$\nSELECT age(clock_timestamp(), transaction_timestamp());\n$$ LANGUAGE SQL;\nSET SEARCH_PATH to pgstac, public;\n\nDROP FUNCTION IF EXISTS geometrysearch;\nCREATE OR REPLACE FUNCTION geometrysearch(\n IN geom geometry,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE, -- Return as soon as the passed in geometry is full covered\n IN skipcovered boolean DEFAULT TRUE -- Skip any items that would show up completely under the previous items\n) RETURNS jsonb AS $$\nDECLARE\n search searches%ROWTYPE;\n curs refcursor;\n _where text;\n query text;\n iter_record items%ROWTYPE;\n out_records jsonb := '{}'::jsonb[];\n exit_flag boolean := FALSE;\n counter int := 1;\n scancounter int := 1;\n remaining_limit int := _scanlimit;\n tilearea float;\n unionedgeom geometry;\n clippedgeom geometry;\n unionedgeom_area float := 0;\n prev_area float := 0;\n excludes text[];\n includes text[];\n\nBEGIN\n DROP TABLE IF EXISTS pgstac_results;\n CREATE TEMP TABLE pgstac_results (content jsonb) ON COMMIT DROP;\n\n -- If the passed in geometry is not an area set exitwhenfull and skipcovered to false\n IF ST_GeometryType(geom) !~* 'polygon' THEN\n RAISE NOTICE 'GEOMETRY IS NOT AN AREA';\n skipcovered = FALSE;\n exitwhenfull = FALSE;\n END IF;\n\n -- If skipcovered is true then you will always want to exit when the passed in geometry is full\n IF skipcovered THEN\n exitwhenfull := TRUE;\n END IF;\n\n search := search_fromhash(queryhash);\n\n IF search IS NULL THEN\n RAISE EXCEPTION 'Search with Query Hash % Not Found', queryhash;\n END IF;\n\n tilearea := st_area(geom);\n _where := format('%s AND st_intersects(geometry, %L::geometry)', search._where, geom);\n\n\n FOR query IN SELECT * FROM partition_queries(_where, search.orderby) LOOP\n query := format('%s LIMIT %L', query, remaining_limit);\n RAISE NOTICE '%', query;\n OPEN curs FOR EXECUTE query;\n LOOP\n FETCH curs INTO iter_record;\n EXIT WHEN NOT FOUND;\n IF exitwhenfull OR skipcovered THEN -- If we are not using exitwhenfull or skipcovered, we do not need to do expensive geometry operations\n clippedgeom := st_intersection(geom, iter_record.geometry);\n\n IF unionedgeom IS NULL THEN\n unionedgeom := clippedgeom;\n ELSE\n unionedgeom := st_union(unionedgeom, clippedgeom);\n END IF;\n\n unionedgeom_area := st_area(unionedgeom);\n\n IF skipcovered AND prev_area = unionedgeom_area THEN\n scancounter := scancounter + 1;\n CONTINUE;\n END IF;\n\n prev_area := unionedgeom_area;\n\n RAISE NOTICE '% % % %', unionedgeom_area/tilearea, counter, scancounter, ftime();\n END IF;\n RAISE NOTICE '% %', iter_record, content_hydrate(iter_record, fields);\n INSERT INTO pgstac_results (content) VALUES (content_hydrate(iter_record, fields));\n\n IF counter >= _limit\n OR scancounter > _scanlimit\n OR ftime() > _timelimit\n OR (exitwhenfull AND unionedgeom_area >= tilearea)\n THEN\n exit_flag := TRUE;\n EXIT;\n END IF;\n counter := counter + 1;\n scancounter := scancounter + 1;\n\n END LOOP;\n CLOSE curs;\n EXIT WHEN exit_flag;\n remaining_limit := _scanlimit - scancounter;\n END LOOP;\n\n SELECT jsonb_agg(content) INTO out_records FROM pgstac_results WHERE content IS NOT NULL;\n\n RETURN jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb)\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDROP FUNCTION IF EXISTS geojsonsearch;\nCREATE OR REPLACE FUNCTION geojsonsearch(\n IN geojson jsonb,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_geomfromgeojson(geojson),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS xyzsearch;\nCREATE OR REPLACE FUNCTION xyzsearch(\n IN _x int,\n IN _y int,\n IN _z int,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_transform(tileenvelope(_z, _x, _y), 4326),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE PROCEDURE analyze_items() AS $$\nDECLARE\n q text;\n timeout_ts timestamptz;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n SELECT format('ANALYZE (VERBOSE, SKIP_LOCKED) %I;', relname) INTO q\n FROM pg_stat_user_tables\n WHERE relname like '_item%' AND (n_mod_since_analyze>0 OR last_analyze IS NULL) LIMIT 1;\n IF NOT FOUND THEN\n EXIT;\n END IF;\n RAISE NOTICE '%', q;\n EXECUTE q;\n COMMIT;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE PROCEDURE validate_constraints() AS $$\nDECLARE\n q text;\nBEGIN\n FOR q IN\n SELECT\n FORMAT(\n 'ALTER TABLE %I.%I VALIDATE CONSTRAINT %I;',\n nsp.nspname,\n cls.relname,\n con.conname\n )\n\n FROM pg_constraint AS con\n JOIN pg_class AS cls\n ON con.conrelid = cls.oid\n JOIN pg_namespace AS nsp\n ON cls.relnamespace = nsp.oid\n WHERE convalidated = FALSE AND contype in ('c','f')\n AND nsp.nspname = 'pgstac'\n LOOP\n RAISE NOTICE '%', q;\n PERFORM run_or_queue(q);\n COMMIT;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION collection_extent(_collection text, runupdate boolean default false) RETURNS jsonb AS $$\nDECLARE\n geom_extent geometry;\n mind timestamptz;\n maxd timestamptz;\n extent jsonb;\nBEGIN\n IF runupdate THEN\n PERFORM update_partition_stats_q(partition)\n FROM partitions_view WHERE collection=_collection;\n END IF;\n SELECT\n min(lower(dtrange)),\n max(upper(edtrange)),\n st_extent(spatial)\n INTO\n mind,\n maxd,\n geom_extent\n FROM partitions_view\n WHERE collection=_collection;\n\n IF geom_extent IS NOT NULL AND mind IS NOT NULL AND maxd IS NOT NULL THEN\n extent := jsonb_build_object(\n 'spatial', jsonb_build_object(\n 'bbox', to_jsonb(array[array[st_xmin(geom_extent), st_ymin(geom_extent), st_xmax(geom_extent), st_ymax(geom_extent)]])\n ),\n 'temporal', jsonb_build_object(\n 'interval', to_jsonb(array[array[mind, maxd]])\n )\n );\n RETURN extent;\n END IF;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('id', '{\"title\": \"Item ID\",\"description\": \"Item identifier\",\"$ref\": \"https://schemas.stacspec.org/v1.0.0/item-spec/json-schema/item.json#/definitions/core/allOf/2/properties/id\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('geometry', '{\"title\": \"Item Geometry\",\"description\": \"Item Geometry\",\"$ref\": \"https://geojson.org/schema/Feature.json\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('datetime','{\"description\": \"Datetime\",\"type\": \"string\",\"title\": \"Acquired\",\"format\": \"date-time\",\"pattern\": \"(\\\\+00:00|Z)$\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDELETE FROM queryables a USING queryables b\n WHERE a.name = b.name AND a.collection_ids IS NOT DISTINCT FROM b.collection_ids AND a.id > b.id;\n\n\nINSERT INTO pgstac_settings (name, value) VALUES\n ('context', 'off'),\n ('context_estimated_count', '100000'),\n ('context_estimated_cost', '100000'),\n ('context_stats_ttl', '1 day'),\n ('default_filter_lang', 'cql2-json'),\n ('additional_properties', 'true'),\n ('use_queue', 'false'),\n ('queue_timeout', '10 minutes'),\n ('update_collection_extent', 'false'),\n ('format_cache', 'false'),\n ('readonly', 'false')\nON CONFLICT DO NOTHING\n;\n\n\nINSERT INTO cql2_ops (op, template, types) VALUES\n ('eq', '%s = %s', NULL),\n ('neq', '%s != %s', NULL),\n ('ne', '%s != %s', NULL),\n ('!=', '%s != %s', NULL),\n ('<>', '%s != %s', NULL),\n ('lt', '%s < %s', NULL),\n ('lte', '%s <= %s', NULL),\n ('gt', '%s > %s', NULL),\n ('gte', '%s >= %s', NULL),\n ('le', '%s <= %s', NULL),\n ('ge', '%s >= %s', NULL),\n ('=', '%s = %s', NULL),\n ('<', '%s < %s', NULL),\n ('<=', '%s <= %s', NULL),\n ('>', '%s > %s', NULL),\n ('>=', '%s >= %s', NULL),\n ('like', '%s LIKE %s', NULL),\n ('ilike', '%s ILIKE %s', NULL),\n ('+', '%s + %s', NULL),\n ('-', '%s - %s', NULL),\n ('*', '%s * %s', NULL),\n ('/', '%s / %s', NULL),\n ('not', 'NOT (%s)', NULL),\n ('between', '%s BETWEEN %s AND %s', NULL),\n ('isnull', '%s IS NULL', NULL),\n ('upper', 'upper(%s)', NULL),\n ('lower', 'lower(%s)', NULL),\n ('casei', 'upper(%s)', NULL),\n ('accenti', 'unaccent(%s)', NULL)\nON CONFLICT (op) DO UPDATE\n SET\n template = EXCLUDED.template\n;\n\n\nALTER FUNCTION to_text COST 5000;\nALTER FUNCTION to_float COST 5000;\nALTER FUNCTION to_int COST 5000;\nALTER FUNCTION to_tstz COST 5000;\nALTER FUNCTION to_text_array COST 5000;\n\nALTER FUNCTION update_partition_stats SECURITY DEFINER;\nALTER FUNCTION partition_after_triggerfunc SECURITY DEFINER;\nALTER FUNCTION drop_table_constraints SECURITY DEFINER;\nALTER FUNCTION create_table_constraints SECURITY DEFINER;\nALTER FUNCTION check_partition SECURITY DEFINER;\nALTER FUNCTION repartition SECURITY DEFINER;\nALTER FUNCTION where_stats SECURITY DEFINER;\nALTER FUNCTION search_query SECURITY DEFINER;\nALTER FUNCTION format_item SECURITY DEFINER;\nALTER FUNCTION maintain_index SECURITY DEFINER;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nGRANT ALL ON SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac to pgstac_admin;\n\n-- pgstac_read role limited to using function apis\nGRANT EXECUTE ON FUNCTION search TO pgstac_read;\nGRANT EXECUTE ON FUNCTION search_query TO pgstac_read;\nGRANT EXECUTE ON FUNCTION item_by_id TO pgstac_read;\nGRANT EXECUTE ON FUNCTION get_item TO pgstac_read;\nGRANT SELECT ON ALL TABLES IN SCHEMA pgstac TO pgstac_read;\n\n\nGRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_ingest;\nGRANT USAGE ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_ingest;\n\nREVOKE ALL PRIVILEGES ON PROCEDURE run_queued_queries FROM public;\nGRANT ALL ON PROCEDURE run_queued_queries TO pgstac_admin;\n\nREVOKE ALL PRIVILEGES ON FUNCTION run_queued_queries_intransaction FROM public;\nGRANT ALL ON FUNCTION run_queued_queries_intransaction TO pgstac_admin;\n\nRESET ROLE;\n\nSET ROLE pgstac_ingest;\nSELECT update_partition_stats_q(partition) FROM partitions_view;\nSELECT set_version('0.9.4');\n" }, { "path": "src/pgstac/migrations/pgstac.0.9.5-0.9.6.sql", "content": "SET client_min_messages TO WARNING;\nSET SEARCH_PATH to pgstac, public;\nRESET ROLE;\nDO $$\nDECLARE\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='postgis') THEN\n CREATE EXTENSION IF NOT EXISTS postgis;\n END IF;\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='btree_gist') THEN\n CREATE EXTENSION IF NOT EXISTS btree_gist;\n END IF;\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='unaccent') THEN\n CREATE EXTENSION IF NOT EXISTS unaccent;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_admin;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_read;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_ingest;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n\nGRANT pgstac_admin TO current_user;\n\n-- Function to make sure pgstac_admin is the owner of items\nCREATE OR REPLACE FUNCTION pgstac_admin_owns() RETURNS VOID AS $$\nDECLARE\n f RECORD;\nBEGIN\n FOR f IN (\n SELECT\n concat(\n oid::regproc::text,\n '(',\n coalesce(pg_get_function_identity_arguments(oid),''),\n ')'\n ) AS name,\n CASE prokind WHEN 'f' THEN 'FUNCTION' WHEN 'p' THEN 'PROCEDURE' WHEN 'a' THEN 'AGGREGATE' END as typ\n FROM pg_proc\n WHERE\n pronamespace=to_regnamespace('pgstac')\n AND proowner != to_regrole('pgstac_admin')\n AND proname NOT LIKE 'pg_stat%'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n FOR f IN (\n SELECT\n oid::regclass::text as name,\n CASE relkind\n WHEN 'i' THEN 'INDEX'\n WHEN 'I' THEN 'INDEX'\n WHEN 'p' THEN 'TABLE'\n WHEN 'r' THEN 'TABLE'\n WHEN 'v' THEN 'VIEW'\n WHEN 'S' THEN 'SEQUENCE'\n ELSE NULL\n END as typ\n FROM pg_class\n WHERE relnamespace=to_regnamespace('pgstac') and relowner != to_regrole('pgstac_admin') AND relkind IN ('r','p','v','S') AND relname NOT LIKE 'pg_stat'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\nSELECT pgstac_admin_owns();\n\nCREATE SCHEMA IF NOT EXISTS pgstac AUTHORIZATION pgstac_admin;\n\nGRANT ALL ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_admin;\n\nALTER ROLE pgstac_admin SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_read SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_ingest SET SEARCH_PATH TO pgstac, public;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\n\nGRANT pgstac_read TO pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\n\nSET ROLE pgstac_admin;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\nRESET ROLE;\n\nSET ROLE pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\nRESET ROLE;\n\nSET SEARCH_PATH TO pgstac, public;\nSET ROLE pgstac_admin;\n\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS analyze_items;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS validate_constraints;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n-- Install these idempotently as migrations do not put them before trying to modify the collections table\n\n\nCREATE OR REPLACE FUNCTION collection_geom(content jsonb)\nRETURNS geometry AS $$\n WITH box AS (SELECT content->'extent'->'spatial'->'bbox'->0 as box)\n SELECT\n st_makeenvelope(\n (box->>0)::float,\n (box->>1)::float,\n (box->>2)::float,\n (box->>3)::float,\n 4326\n )\n FROM box;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION collection_datetime(content jsonb)\nRETURNS timestamptz AS $$\n SELECT\n CASE\n WHEN\n (content->'extent'->'temporal'->'interval'->0->>0) IS NULL\n THEN '-infinity'::timestamptz\n ELSE\n (content->'extent'->'temporal'->'interval'->0->>0)::timestamptz\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION collection_enddatetime(content jsonb)\nRETURNS timestamptz AS $$\n SELECT\n CASE\n WHEN\n (content->'extent'->'temporal'->'interval'->0->>1) IS NULL\n THEN 'infinity'::timestamptz\n ELSE\n (content->'extent'->'temporal'->'interval'->0->>1)::timestamptz\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n-- BEGIN migra calculated SQL\n-- END migra calculated SQL\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('id', '{\"title\": \"Item ID\",\"description\": \"Item identifier\",\"$ref\": \"https://schemas.stacspec.org/v1.0.0/item-spec/json-schema/item.json#/definitions/core/allOf/2/properties/id\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('geometry', '{\"title\": \"Item Geometry\",\"description\": \"Item Geometry\",\"$ref\": \"https://geojson.org/schema/Feature.json\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('datetime','{\"description\": \"Datetime\",\"type\": \"string\",\"title\": \"Acquired\",\"format\": \"date-time\",\"pattern\": \"(\\\\+00:00|Z)$\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDELETE FROM queryables a USING queryables b\n WHERE a.name = b.name AND a.collection_ids IS NOT DISTINCT FROM b.collection_ids AND a.id > b.id;\n\n\nINSERT INTO pgstac_settings (name, value) VALUES\n ('context', 'off'),\n ('context_estimated_count', '100000'),\n ('context_estimated_cost', '100000'),\n ('context_stats_ttl', '1 day'),\n ('default_filter_lang', 'cql2-json'),\n ('additional_properties', 'true'),\n ('use_queue', 'false'),\n ('queue_timeout', '10 minutes'),\n ('update_collection_extent', 'false'),\n ('format_cache', 'false'),\n ('readonly', 'false')\nON CONFLICT DO NOTHING\n;\n\n\nINSERT INTO cql2_ops (op, template, types) VALUES\n ('eq', '%s = %s', NULL),\n ('neq', '%s != %s', NULL),\n ('ne', '%s != %s', NULL),\n ('!=', '%s != %s', NULL),\n ('<>', '%s != %s', NULL),\n ('lt', '%s < %s', NULL),\n ('lte', '%s <= %s', NULL),\n ('gt', '%s > %s', NULL),\n ('gte', '%s >= %s', NULL),\n ('le', '%s <= %s', NULL),\n ('ge', '%s >= %s', NULL),\n ('=', '%s = %s', NULL),\n ('<', '%s < %s', NULL),\n ('<=', '%s <= %s', NULL),\n ('>', '%s > %s', NULL),\n ('>=', '%s >= %s', NULL),\n ('like', '%s LIKE %s', NULL),\n ('ilike', '%s ILIKE %s', NULL),\n ('+', '%s + %s', NULL),\n ('-', '%s - %s', NULL),\n ('*', '%s * %s', NULL),\n ('/', '%s / %s', NULL),\n ('not', 'NOT (%s)', NULL),\n ('between', '%s BETWEEN %s AND %s', NULL),\n ('isnull', '%s IS NULL', NULL),\n ('upper', 'upper(%s)', NULL),\n ('lower', 'lower(%s)', NULL),\n ('casei', 'upper(%s)', NULL),\n ('accenti', 'unaccent(%s)', NULL)\nON CONFLICT (op) DO UPDATE\n SET\n template = EXCLUDED.template\n;\n\n\nALTER FUNCTION to_text COST 5000;\nALTER FUNCTION to_float COST 5000;\nALTER FUNCTION to_int COST 5000;\nALTER FUNCTION to_tstz COST 5000;\nALTER FUNCTION to_text_array COST 5000;\n\nALTER FUNCTION update_partition_stats SECURITY DEFINER;\nALTER FUNCTION partition_after_triggerfunc SECURITY DEFINER;\nALTER FUNCTION drop_table_constraints SECURITY DEFINER;\nALTER FUNCTION create_table_constraints SECURITY DEFINER;\nALTER FUNCTION check_partition SECURITY DEFINER;\nALTER FUNCTION repartition SECURITY DEFINER;\nALTER FUNCTION where_stats SECURITY DEFINER;\nALTER FUNCTION search_query SECURITY DEFINER;\nALTER FUNCTION format_item SECURITY DEFINER;\nALTER FUNCTION maintain_index SECURITY DEFINER;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nGRANT ALL ON SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac to pgstac_admin;\n\n-- pgstac_read role limited to using function apis\nGRANT EXECUTE ON FUNCTION search TO pgstac_read;\nGRANT EXECUTE ON FUNCTION search_query TO pgstac_read;\nGRANT EXECUTE ON FUNCTION item_by_id TO pgstac_read;\nGRANT EXECUTE ON FUNCTION get_item TO pgstac_read;\nGRANT SELECT ON ALL TABLES IN SCHEMA pgstac TO pgstac_read;\n\n\nGRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_ingest;\nGRANT USAGE ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_ingest;\n\nREVOKE ALL PRIVILEGES ON PROCEDURE run_queued_queries FROM public;\nGRANT ALL ON PROCEDURE run_queued_queries TO pgstac_admin;\n\nREVOKE ALL PRIVILEGES ON FUNCTION run_queued_queries_intransaction FROM public;\nGRANT ALL ON FUNCTION run_queued_queries_intransaction TO pgstac_admin;\n\nRESET ROLE;\n\nSET ROLE pgstac_ingest;\nSELECT update_partition_stats_q(partition) FROM partitions_view;\nSELECT set_version('0.9.6');\n" }, { "path": "src/pgstac/migrations/pgstac.0.9.5.sql", "content": "RESET ROLE;\nDO $$\nDECLARE\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='postgis') THEN\n CREATE EXTENSION IF NOT EXISTS postgis;\n END IF;\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='btree_gist') THEN\n CREATE EXTENSION IF NOT EXISTS btree_gist;\n END IF;\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='unaccent') THEN\n CREATE EXTENSION IF NOT EXISTS unaccent;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_admin;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_read;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_ingest;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n\nGRANT pgstac_admin TO current_user;\n\n-- Function to make sure pgstac_admin is the owner of items\nCREATE OR REPLACE FUNCTION pgstac_admin_owns() RETURNS VOID AS $$\nDECLARE\n f RECORD;\nBEGIN\n FOR f IN (\n SELECT\n concat(\n oid::regproc::text,\n '(',\n coalesce(pg_get_function_identity_arguments(oid),''),\n ')'\n ) AS name,\n CASE prokind WHEN 'f' THEN 'FUNCTION' WHEN 'p' THEN 'PROCEDURE' WHEN 'a' THEN 'AGGREGATE' END as typ\n FROM pg_proc\n WHERE\n pronamespace=to_regnamespace('pgstac')\n AND proowner != to_regrole('pgstac_admin')\n AND proname NOT LIKE 'pg_stat%'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n FOR f IN (\n SELECT\n oid::regclass::text as name,\n CASE relkind\n WHEN 'i' THEN 'INDEX'\n WHEN 'I' THEN 'INDEX'\n WHEN 'p' THEN 'TABLE'\n WHEN 'r' THEN 'TABLE'\n WHEN 'v' THEN 'VIEW'\n WHEN 'S' THEN 'SEQUENCE'\n ELSE NULL\n END as typ\n FROM pg_class\n WHERE relnamespace=to_regnamespace('pgstac') and relowner != to_regrole('pgstac_admin') AND relkind IN ('r','p','v','S') AND relname NOT LIKE 'pg_stat'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\nSELECT pgstac_admin_owns();\n\nCREATE SCHEMA IF NOT EXISTS pgstac AUTHORIZATION pgstac_admin;\n\nGRANT ALL ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_admin;\n\nALTER ROLE pgstac_admin SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_read SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_ingest SET SEARCH_PATH TO pgstac, public;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\n\nGRANT pgstac_read TO pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\n\nSET ROLE pgstac_admin;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\nRESET ROLE;\n\nSET ROLE pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\nRESET ROLE;\n\nSET SEARCH_PATH TO pgstac, public;\nSET ROLE pgstac_admin;\n\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS analyze_items;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS validate_constraints;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n-- Install these idempotently as migrations do not put them before trying to modify the collections table\n\n\nCREATE OR REPLACE FUNCTION collection_geom(content jsonb)\nRETURNS geometry AS $$\n WITH box AS (SELECT content->'extent'->'spatial'->'bbox'->0 as box)\n SELECT\n st_makeenvelope(\n (box->>0)::float,\n (box->>1)::float,\n (box->>2)::float,\n (box->>3)::float,\n 4326\n )\n FROM box;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION collection_datetime(content jsonb)\nRETURNS timestamptz AS $$\n SELECT\n CASE\n WHEN\n (content->'extent'->'temporal'->'interval'->0->>0) IS NULL\n THEN '-infinity'::timestamptz\n ELSE\n (content->'extent'->'temporal'->'interval'->0->>0)::timestamptz\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION collection_enddatetime(content jsonb)\nRETURNS timestamptz AS $$\n SELECT\n CASE\n WHEN\n (content->'extent'->'temporal'->'interval'->0->>1) IS NULL\n THEN 'infinity'::timestamptz\n ELSE\n (content->'extent'->'temporal'->'interval'->0->>1)::timestamptz\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE TABLE IF NOT EXISTS migrations (\n version text PRIMARY KEY,\n datetime timestamptz DEFAULT clock_timestamp() NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION get_version() RETURNS text AS $$\n SELECT version FROM pgstac.migrations ORDER BY datetime DESC, version DESC LIMIT 1;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION set_version(text) RETURNS text AS $$\n INSERT INTO pgstac.migrations (version) VALUES ($1)\n ON CONFLICT DO NOTHING\n RETURNING version;\n$$ LANGUAGE SQL;\n\n\nCREATE TABLE IF NOT EXISTS pgstac_settings (\n name text PRIMARY KEY,\n value text NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION table_empty(text) RETURNS boolean AS $$\nDECLARE\n retval boolean;\nBEGIN\n EXECUTE format($q$\n SELECT NOT EXISTS (SELECT 1 FROM %I LIMIT 1)\n $q$,\n $1\n ) INTO retval;\n RETURN retval;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION get_setting(IN _setting text, IN conf jsonb DEFAULT NULL) RETURNS text AS $$\nSELECT COALESCE(\n nullif(conf->>_setting, ''),\n nullif(current_setting(concat('pgstac.',_setting), TRUE),''),\n nullif((SELECT value FROM pgstac.pgstac_settings WHERE name=_setting),'')\n);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_setting_bool(IN _setting text, IN conf jsonb DEFAULT NULL) RETURNS boolean AS $$\nSELECT COALESCE(\n nullif(conf->>_setting, ''),\n nullif(current_setting(concat('pgstac.',_setting), TRUE),''),\n nullif((SELECT value FROM pgstac.pgstac_settings WHERE name=_setting),''),\n 'FALSE'\n)::boolean;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION base_url(conf jsonb DEFAULT NULL) RETURNS text AS $$\n SELECT COALESCE(pgstac.get_setting('base_url', conf), '.');\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION additional_properties() RETURNS boolean AS $$\n SELECT pgstac.get_setting_bool('additional_properties');\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION readonly(conf jsonb DEFAULT NULL) RETURNS boolean AS $$\n SELECT pgstac.get_setting_bool('readonly', conf);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context(conf jsonb DEFAULT NULL) RETURNS text AS $$\n SELECT pgstac.get_setting('context', conf);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context_estimated_count(conf jsonb DEFAULT NULL) RETURNS int AS $$\n SELECT pgstac.get_setting('context_estimated_count', conf)::int;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_estimated_cost();\nCREATE OR REPLACE FUNCTION context_estimated_cost(conf jsonb DEFAULT NULL) RETURNS float AS $$\n SELECT pgstac.get_setting('context_estimated_cost', conf)::float;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_stats_ttl();\nCREATE OR REPLACE FUNCTION context_stats_ttl(conf jsonb DEFAULT NULL) RETURNS interval AS $$\n SELECT pgstac.get_setting('context_stats_ttl', conf)::interval;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION t2s(text) RETURNS text AS $$\n SELECT extract(epoch FROM $1::interval)::text || ' s';\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE STRICT;\n\nCREATE OR REPLACE FUNCTION age_ms(a timestamptz, b timestamptz DEFAULT clock_timestamp()) RETURNS float AS $$\n SELECT abs(extract(epoch from age(a,b)) * 1000);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION queue_timeout() RETURNS interval AS $$\n SELECT t2s(coalesce(\n get_setting('queue_timeout'),\n '1h'\n ))::interval;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION notice(VARIADIC text[]) RETURNS boolean AS $$\nDECLARE\ndebug boolean := current_setting('pgstac.debug', true);\nBEGIN\n IF debug THEN\n RAISE NOTICE 'NOTICE FROM FUNC: % >>>>> %', concat_ws(' | ', $1), clock_timestamp();\n RETURN TRUE;\n END IF;\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION empty_arr(ANYARRAY) RETURNS BOOLEAN AS $$\nSELECT CASE\n WHEN $1 IS NULL THEN TRUE\n WHEN cardinality($1)<1 THEN TRUE\nELSE FALSE\nEND;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_intersection(_a ANYARRAY, _b ANYARRAY) RETURNS ANYARRAY AS $$\n SELECT ARRAY ( SELECT unnest(_a) INTERSECT SELECT UNNEST(_b) );\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION array_map_ident(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_ident(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_map_literal(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_literal(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION array_reverse(anyarray) RETURNS anyarray AS $$\nSELECT ARRAY(\n SELECT $1[i]\n FROM generate_subscripts($1,1) AS s(i)\n ORDER BY i DESC\n);\n$$ LANGUAGE SQL STRICT IMMUTABLE;\n\nDROP TABLE IF EXISTS query_queue;\nCREATE TABLE query_queue (\n query text PRIMARY KEY,\n added timestamptz DEFAULT now()\n);\n\nDROP TABLE IF EXISTS query_queue_history;\nCREATE TABLE query_queue_history(\n query text,\n added timestamptz NOT NULL,\n finished timestamptz NOT NULL DEFAULT now(),\n error text\n);\n\nCREATE OR REPLACE PROCEDURE run_queued_queries() AS $$\nDECLARE\n qitem query_queue%ROWTYPE;\n timeout_ts timestamptz;\n error text;\n cnt int := 0;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n DELETE FROM query_queue WHERE query = (SELECT query FROM query_queue ORDER BY added DESC LIMIT 1 FOR UPDATE SKIP LOCKED) RETURNING * INTO qitem;\n IF NOT FOUND THEN\n EXIT;\n END IF;\n cnt := cnt + 1;\n BEGIN\n RAISE NOTICE 'RUNNING QUERY: %', qitem.query;\n EXECUTE qitem.query;\n EXCEPTION WHEN others THEN\n error := format('%s | %s', SQLERRM, SQLSTATE);\n END;\n INSERT INTO query_queue_history (query, added, finished, error)\n VALUES (qitem.query, qitem.added, clock_timestamp(), error);\n COMMIT;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION run_queued_queries_intransaction() RETURNS int AS $$\nDECLARE\n qitem query_queue%ROWTYPE;\n timeout_ts timestamptz;\n error text;\n cnt int := 0;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n DELETE FROM query_queue WHERE query = (SELECT query FROM query_queue ORDER BY added DESC LIMIT 1 FOR UPDATE SKIP LOCKED) RETURNING * INTO qitem;\n IF NOT FOUND THEN\n RETURN cnt;\n END IF;\n cnt := cnt + 1;\n BEGIN\n qitem.query := regexp_replace(qitem.query, 'CONCURRENTLY', '');\n RAISE NOTICE 'RUNNING QUERY: %', qitem.query;\n\n EXECUTE qitem.query;\n EXCEPTION WHEN others THEN\n error := format('%s | %s', SQLERRM, SQLSTATE);\n RAISE WARNING '%', error;\n END;\n INSERT INTO query_queue_history (query, added, finished, error)\n VALUES (qitem.query, qitem.added, clock_timestamp(), error);\n END LOOP;\n RETURN cnt;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION run_or_queue(query text) RETURNS VOID AS $$\nDECLARE\n use_queue text := COALESCE(get_setting('use_queue'), 'FALSE')::boolean;\nBEGIN\n IF get_setting_bool('debug') THEN\n RAISE NOTICE '%', query;\n END IF;\n IF use_queue THEN\n INSERT INTO query_queue (query) VALUES (query) ON CONFLICT DO NOTHING;\n ELSE\n EXECUTE query;\n END IF;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n\nDROP FUNCTION IF EXISTS check_pgstac_settings;\nCREATE OR REPLACE FUNCTION check_pgstac_settings(_sysmem text DEFAULT NULL) RETURNS VOID AS $$\nDECLARE\n settingval text;\n sysmem bigint := pg_size_bytes(_sysmem);\n effective_cache_size bigint := pg_size_bytes(current_setting('effective_cache_size', TRUE));\n shared_buffers bigint := pg_size_bytes(current_setting('shared_buffers', TRUE));\n work_mem bigint := pg_size_bytes(current_setting('work_mem', TRUE));\n max_connections int := current_setting('max_connections', TRUE);\n maintenance_work_mem bigint := pg_size_bytes(current_setting('maintenance_work_mem', TRUE));\n seq_page_cost float := current_setting('seq_page_cost', TRUE);\n random_page_cost float := current_setting('random_page_cost', TRUE);\n temp_buffers bigint := pg_size_bytes(current_setting('temp_buffers', TRUE));\n r record;\nBEGIN\n IF _sysmem IS NULL THEN\n RAISE NOTICE 'Call function with the size of your system memory `SELECT check_pgstac_settings(''4GB'')` to get pg system setting recommendations.';\n ELSE\n IF effective_cache_size < (sysmem * 0.5) THEN\n RAISE WARNING 'effective_cache_size of % is set low for a system with %. Recomended value between % and %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.5), pg_size_pretty(sysmem * 0.75);\n ELSIF effective_cache_size > (sysmem * 0.75) THEN\n RAISE WARNING 'effective_cache_size of % is set high for a system with %. Recomended value between % and %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.5), pg_size_pretty(sysmem * 0.75);\n ELSE\n RAISE NOTICE 'effective_cache_size of % is set appropriately for a system with %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem);\n END IF;\n\n IF shared_buffers < (sysmem * 0.2) THEN\n RAISE WARNING 'shared_buffers of % is set low for a system with %. Recomended value between % and %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.2), pg_size_pretty(sysmem * 0.3);\n ELSIF shared_buffers > (sysmem * 0.3) THEN\n RAISE WARNING 'shared_buffers of % is set high for a system with %. Recomended value between % and %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.2), pg_size_pretty(sysmem * 0.3);\n ELSE\n RAISE NOTICE 'shared_buffers of % is set appropriately for a system with %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem);\n END IF;\n shared_buffers = sysmem * 0.3;\n IF maintenance_work_mem < (sysmem * 0.2) THEN\n RAISE WARNING 'maintenance_work_mem of % is set low for shared_buffers of %. Recomended value between % and %', pg_size_pretty(maintenance_work_mem), pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers * 0.2), pg_size_pretty(shared_buffers * 0.3);\n ELSIF maintenance_work_mem > (shared_buffers * 0.3) THEN\n RAISE WARNING 'maintenance_work_mem of % is set high for shared_buffers of %. Recomended value between % and %', pg_size_pretty(maintenance_work_mem), pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers * 0.2), pg_size_pretty(shared_buffers * 0.3);\n ELSE\n RAISE NOTICE 'maintenance_work_mem of % is set appropriately for shared_buffers of %', pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers);\n END IF;\n\n IF work_mem * max_connections > shared_buffers THEN\n RAISE WARNING 'work_mem setting of % is set high for % max_connections please reduce work_mem to % or decrease max_connections to %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers/max_connections), floor(shared_buffers/work_mem);\n ELSIF work_mem * max_connections < (shared_buffers * 0.75) THEN\n RAISE WARNING 'work_mem setting of % is set low for % max_connections you may consider raising work_mem to % or increasing max_connections to %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers/max_connections), floor(shared_buffers/work_mem);\n ELSE\n RAISE NOTICE 'work_mem setting of % and max_connections of % are adequate for shared_buffers of %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers);\n END IF;\n\n IF random_page_cost / seq_page_cost != 1.1 THEN\n RAISE WARNING 'random_page_cost (%) /seq_page_cost (%) should be set to 1.1 for SSD. Change random_page_cost to %', random_page_cost, seq_page_cost, 1.1 * seq_page_cost;\n ELSE\n RAISE NOTICE 'random_page_cost and seq_page_cost set appropriately for SSD';\n END IF;\n\n IF temp_buffers < greatest(pg_size_bytes('128MB'),(maintenance_work_mem / 2)) THEN\n RAISE WARNING 'pgstac makes heavy use of temp tables, consider raising temp_buffers from % to %', pg_size_pretty(temp_buffers), greatest('128MB', pg_size_pretty((shared_buffers / 16)));\n END IF;\n END IF;\n\n RAISE NOTICE 'VALUES FOR PGSTAC VARIABLES';\n RAISE NOTICE 'These can be set either as GUC system variables or by setting in the pgstac_settings table.';\n\n FOR r IN SELECT name, get_setting(name) as setting, CASE WHEN current_setting(concat('pgstac.',name), TRUE) IS NOT NULL THEN concat('pgstac.',name, ' GUC') WHEN value IS NOT NULL THEN 'pgstac_settings table' ELSE 'Not Set' END as loc FROM pgstac_settings LOOP\n RAISE NOTICE '% is set to % from the %', r.name, r.setting, r.loc;\n END LOOP;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pg_cron';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider intalling pg_cron which can be used to automate tasks';\n ELSE\n RAISE NOTICE 'pg_cron % is installed', settingval;\n END IF;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pgstattuple';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider installing the pgstattuple extension which can be used to help maintain tables and indexes.';\n ELSE\n RAISE NOTICE 'pgstattuple % is installed', settingval;\n END IF;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pg_stat_statements';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider installing the pg_stat_statements extension which is very helpful for tracking the types of queries on the system';\n ELSE\n RAISE NOTICE 'pg_stat_statements % is installed', settingval;\n IF current_setting('pg_stat_statements.track_statements', TRUE) IS DISTINCT FROM 'all' THEN\n RAISE WARNING 'SET pg_stat_statements.track_statements TO ''all''; --In order to track statements within functions.';\n END IF;\n END IF;\n\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac, public SET CLIENT_MIN_MESSAGES TO NOTICE;\nCREATE OR REPLACE FUNCTION to_int(jsonb) RETURNS int AS $$\n SELECT floor(($1->>0)::float)::int;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION to_float(jsonb) RETURNS float AS $$\n SELECT ($1->>0)::float;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION to_tstz(jsonb) RETURNS timestamptz AS $$\n SELECT ($1->>0)::timestamptz;\n$$ LANGUAGE SQL IMMUTABLE STRICT SET TIME ZONE 'UTC' COST 5000 PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION to_text(jsonb) RETURNS text AS $$\n SELECT CASE WHEN jsonb_typeof($1) IN ('array','object') THEN $1::text ELSE $1->>0 END;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION to_text_array(jsonb) RETURNS text[] AS $$\n SELECT\n CASE jsonb_typeof($1)\n WHEN 'array' THEN ARRAY(SELECT jsonb_array_elements_text($1))\n ELSE ARRAY[$1->>0]\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION bbox_geom(_bbox jsonb) RETURNS geometry AS $$\nSELECT CASE jsonb_array_length(_bbox)\n WHEN 4 THEN\n ST_SetSRID(ST_MakeEnvelope(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float,\n (_bbox->>3)::float\n ),4326)\n WHEN 6 THEN\n ST_SetSRID(ST_3DMakeBox(\n ST_MakePoint(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float\n ),\n ST_MakePoint(\n (_bbox->>3)::float,\n (_bbox->>4)::float,\n (_bbox->>5)::float\n )\n ),4326)\n ELSE null END;\n;\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION geom_bbox(_geom geometry) RETURNS jsonb AS $$\n SELECT jsonb_build_array(\n st_xmin(_geom),\n st_ymin(_geom),\n st_xmax(_geom),\n st_ymax(_geom)\n );\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION flip_jsonb_array(j jsonb) RETURNS jsonb AS $$\n SELECT jsonb_agg(value) FROM (SELECT value FROM jsonb_array_elements(j) WITH ORDINALITY ORDER BY ordinality DESC) as t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION explode_dotpaths(j jsonb) RETURNS SETOF text[] AS $$\n SELECT string_to_array(p, '.') as e FROM jsonb_array_elements_text(j) p;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION explode_dotpaths_recurse(IN j jsonb) RETURNS SETOF text[] AS $$\n WITH RECURSIVE t AS (\n SELECT e FROM explode_dotpaths(j) e\n UNION ALL\n SELECT e[1:cardinality(e)-1]\n FROM t\n WHERE cardinality(e)>1\n ) SELECT e FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION jsonb_set_nested(j jsonb, path text[], val jsonb) RETURNS jsonb AS $$\nDECLARE\nBEGIN\n IF cardinality(path) > 1 THEN\n FOR i IN 1..(cardinality(path)-1) LOOP\n IF j #> path[:i] IS NULL THEN\n j := jsonb_set_lax(j, path[:i], '{}', TRUE);\n END IF;\n END LOOP;\n END IF;\n RETURN jsonb_set_lax(j, path, val, true);\n\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\n\n\nCREATE OR REPLACE FUNCTION jsonb_include(j jsonb, f jsonb) RETURNS jsonb AS $$\nDECLARE\n includes jsonb := f-> 'include';\n outj jsonb := '{}'::jsonb;\n path text[];\nBEGIN\n IF\n includes IS NULL\n OR jsonb_array_length(includes) = 0\n THEN\n RETURN j;\n ELSE\n includes := includes || '[\"id\",\"collection\"]'::jsonb;\n FOR path IN SELECT explode_dotpaths(includes) LOOP\n outj := jsonb_set_nested(outj, path, j #> path);\n END LOOP;\n END IF;\n RETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_exclude(j jsonb, f jsonb) RETURNS jsonb AS $$\nDECLARE\n excludes jsonb := f-> 'exclude';\n outj jsonb := j;\n path text[];\nBEGIN\n IF\n excludes IS NULL\n OR jsonb_array_length(excludes) = 0\n THEN\n RETURN j;\n ELSE\n FOR path IN SELECT explode_dotpaths(excludes) LOOP\n outj := outj #- path;\n END LOOP;\n END IF;\n RETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_fields(j jsonb, f jsonb DEFAULT '{\"fields\":[]}') RETURNS jsonb AS $$\n SELECT jsonb_exclude(jsonb_include(j, f), f);\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION merge_jsonb(_a jsonb, _b jsonb) RETURNS jsonb AS $$\n SELECT\n CASE\n WHEN _a = '\"𒍟※\"'::jsonb THEN NULL\n WHEN _a IS NULL OR jsonb_typeof(_a) = 'null' THEN _b\n WHEN jsonb_typeof(_a) = 'object' AND jsonb_typeof(_b) = 'object' THEN\n (\n SELECT\n jsonb_strip_nulls(\n jsonb_object_agg(\n key,\n merge_jsonb(a.value, b.value)\n )\n )\n FROM\n jsonb_each(coalesce(_a,'{}'::jsonb)) as a\n FULL JOIN\n jsonb_each(coalesce(_b,'{}'::jsonb)) as b\n USING (key)\n )\n WHEN\n jsonb_typeof(_a) = 'array'\n AND jsonb_typeof(_b) = 'array'\n AND jsonb_array_length(_a) = jsonb_array_length(_b)\n THEN\n (\n SELECT jsonb_agg(m) FROM\n ( SELECT\n merge_jsonb(\n jsonb_array_elements(_a),\n jsonb_array_elements(_b)\n ) as m\n ) as l\n )\n ELSE _a\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION strip_jsonb(_a jsonb, _b jsonb) RETURNS jsonb AS $$\n SELECT\n CASE\n\n WHEN (_a IS NULL OR jsonb_typeof(_a) = 'null') AND _b IS NOT NULL AND jsonb_typeof(_b) != 'null' THEN '\"𒍟※\"'::jsonb\n WHEN _b IS NULL OR jsonb_typeof(_a) = 'null' THEN _a\n WHEN _a = _b AND jsonb_typeof(_a) = 'object' THEN '{}'::jsonb\n WHEN _a = _b THEN NULL\n WHEN jsonb_typeof(_a) = 'object' AND jsonb_typeof(_b) = 'object' THEN\n (\n SELECT\n jsonb_strip_nulls(\n jsonb_object_agg(\n key,\n strip_jsonb(a.value, b.value)\n )\n )\n FROM\n jsonb_each(_a) as a\n FULL JOIN\n jsonb_each(_b) as b\n USING (key)\n )\n WHEN\n jsonb_typeof(_a) = 'array'\n AND jsonb_typeof(_b) = 'array'\n AND jsonb_array_length(_a) = jsonb_array_length(_b)\n THEN\n (\n SELECT jsonb_agg(m) FROM\n ( SELECT\n strip_jsonb(\n jsonb_array_elements(_a),\n jsonb_array_elements(_b)\n ) as m\n ) as l\n )\n ELSE _a\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION nullif_jsonbnullempty(j jsonb) RETURNS jsonb AS $$\n SELECT nullif(nullif(nullif(j,'null'::jsonb),'{}'::jsonb),'[]'::jsonb);\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION jsonb_array_unique(j jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(jsonb_agg(DISTINCT a)) v FROM jsonb_array_elements(j) a;\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_concat_ignorenull(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT coalesce(a,'[]'::jsonb) || coalesce(b,'[]'::jsonb);\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_least(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(least(nullif_jsonbnullempty(a), nullif_jsonbnullempty(b)));\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_greatest(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(greatest(a, b));\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION first_notnull_sfunc(anyelement, anyelement) RETURNS anyelement AS $$\n SELECT COALESCE($1,$2);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE AGGREGATE first_notnull(anyelement)(\n SFUNC = first_notnull_sfunc,\n STYPE = anyelement\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_array_unique_merge(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_concat_ignorenull,\n FINALFUNC = jsonb_array_unique\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_min(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_least\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_max(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_greatest\n);\n/* looks for a geometry in a stac item first from geometry and falling back to bbox */\nCREATE OR REPLACE FUNCTION stac_geom(value jsonb) RETURNS geometry AS $$\nSELECT\n CASE\n WHEN value ? 'intersects' THEN\n ST_GeomFromGeoJSON(value->>'intersects')\n WHEN value ? 'geometry' THEN\n ST_GeomFromGeoJSON(value->>'geometry')\n WHEN value ? 'bbox' THEN\n pgstac.bbox_geom(value->'bbox')\n ELSE NULL\n END as geometry\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION stac_daterange(\n value jsonb\n) RETURNS tstzrange AS $$\nDECLARE\n props jsonb := value;\n dt timestamptz;\n edt timestamptz;\nBEGIN\n IF props ? 'properties' THEN\n props := props->'properties';\n END IF;\n IF\n props ? 'start_datetime'\n AND props->>'start_datetime' IS NOT NULL\n AND props ? 'end_datetime'\n AND props->>'end_datetime' IS NOT NULL\n THEN\n dt := props->>'start_datetime';\n edt := props->>'end_datetime';\n IF dt > edt THEN\n RAISE EXCEPTION 'start_datetime must be < end_datetime';\n END IF;\n ELSE\n dt := props->>'datetime';\n edt := props->>'datetime';\n END IF;\n IF dt is NULL OR edt IS NULL THEN\n RAISE NOTICE 'DT: %, EDT: %', dt, edt;\n RAISE EXCEPTION 'Either datetime (%) or both start_datetime (%) and end_datetime (%) must be set.', props->>'datetime',props->>'start_datetime',props->>'end_datetime';\n END IF;\n RETURN tstzrange(dt, edt, '[]');\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_datetime(value jsonb) RETURNS timestamptz AS $$\n SELECT lower(stac_daterange(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_end_datetime(value jsonb) RETURNS timestamptz AS $$\n SELECT upper(stac_daterange(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE TABLE IF NOT EXISTS stac_extensions(\n url text PRIMARY KEY,\n content jsonb\n);\nCREATE OR REPLACE FUNCTION collection_base_item(content jsonb) RETURNS jsonb AS $$\n SELECT jsonb_build_object(\n 'type', 'Feature',\n 'stac_version', content->'stac_version',\n 'assets', content->'item_assets',\n 'collection', content->'id'\n );\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE TABLE IF NOT EXISTS collections (\n key bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,\n id text GENERATED ALWAYS AS (content->>'id') STORED UNIQUE NOT NULL,\n content JSONB NOT NULL,\n base_item jsonb GENERATED ALWAYS AS (pgstac.collection_base_item(content)) STORED,\n geometry geometry GENERATED ALWAYS AS (pgstac.collection_geom(content)) STORED,\n datetime timestamptz GENERATED ALWAYS AS (pgstac.collection_datetime(content)) STORED,\n end_datetime timestamptz GENERATED ALWAYS AS (pgstac.collection_enddatetime(content)) STORED,\n private jsonb,\n partition_trunc text CHECK (partition_trunc IN ('year', 'month'))\n);\n\nCREATE OR REPLACE FUNCTION collection_base_item(cid text) RETURNS jsonb AS $$\n SELECT pgstac.collection_base_item(content) FROM pgstac.collections WHERE id = cid LIMIT 1;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION create_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION update_collection(data jsonb) RETURNS VOID AS $$\nDECLARE\n out collections%ROWTYPE;\nBEGIN\n UPDATE collections SET content=data WHERE id = data->>'id' RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ON CONFLICT (id) DO\n UPDATE\n SET content=EXCLUDED.content\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION delete_collection(_id text) RETURNS VOID AS $$\nDECLARE\n out collections%ROWTYPE;\nBEGIN\n DELETE FROM collections WHERE id = _id RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION get_collection(id text) RETURNS jsonb AS $$\n SELECT content FROM collections\n WHERE id=$1\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION all_collections() RETURNS jsonb AS $$\n SELECT coalesce(jsonb_agg(content), '[]'::jsonb) FROM collections;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION collection_delete_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n collection_base_partition text := concat('_items_', OLD.key);\nBEGIN\n EXECUTE format($q$\n DELETE FROM partition_stats WHERE partition IN (\n SELECT partition FROM partition_sys_meta\n WHERE collection=%L\n );\n DROP TABLE IF EXISTS %I CASCADE;\n $q$,\n OLD.id,\n collection_base_partition\n );\n RETURN OLD;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER collection_delete_trigger BEFORE DELETE ON collections\nFOR EACH ROW EXECUTE FUNCTION collection_delete_trigger_func();\nCREATE OR REPLACE FUNCTION queryable_signature(n text, c text[]) RETURNS text AS $$\n SELECT concat(n, c);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TABLE queryables (\n id bigint GENERATED ALWAYS AS identity PRIMARY KEY,\n name text NOT NULL,\n collection_ids text[], -- used to determine what partitions to create indexes on\n definition jsonb,\n property_path text,\n property_wrapper text,\n property_index_type text\n);\nCREATE INDEX queryables_name_idx ON queryables (name);\nCREATE INDEX queryables_collection_idx ON queryables USING GIN (collection_ids);\nCREATE INDEX queryables_property_wrapper_idx ON queryables (property_wrapper);\n\nCREATE OR REPLACE FUNCTION pgstac.queryables_constraint_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n allcollections text[];\nBEGIN\n RAISE NOTICE 'Making sure that name/collection is unique for queryables %', NEW;\n IF NEW.collection_ids IS NOT NULL THEN\n IF EXISTS (\n SELECT 1\n FROM unnest(NEW.collection_ids) c\n LEFT JOIN\n collections\n ON (collections.id = c)\n WHERE collections.id IS NULL\n ) THEN\n RAISE foreign_key_violation USING MESSAGE = format(\n 'One or more collections in %s do not exist.', NEW.collection_ids\n );\n RETURN NULL;\n END IF;\n END IF;\n IF TG_OP = 'INSERT' THEN\n IF EXISTS (\n SELECT 1 FROM queryables q\n WHERE\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n )\n ) THEN\n RAISE unique_violation USING MESSAGE = format(\n 'There is already a queryable for %s for a collection in %s: %s',\n NEW.name,\n NEW.collection_ids,\n\t\t\t\t(SELECT json_agg(row_to_json(q)) FROM queryables q WHERE\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n ))\n );\n RETURN NULL;\n END IF;\n END IF;\n IF TG_OP = 'UPDATE' THEN\n IF EXISTS (\n SELECT 1 FROM queryables q\n WHERE\n q.id != NEW.id\n AND\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n )\n ) THEN\n RAISE unique_violation\n USING MESSAGE = format(\n 'There is already a queryable for %s for a collection in %s',\n NEW.name,\n NEW.collection_ids\n );\n RETURN NULL;\n END IF;\n END IF;\n\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER queryables_constraint_insert_trigger\nBEFORE INSERT ON queryables\nFOR EACH ROW EXECUTE PROCEDURE queryables_constraint_triggerfunc();\n\nCREATE TRIGGER queryables_constraint_update_trigger\nBEFORE UPDATE ON queryables\nFOR EACH ROW\nWHEN (NEW.name = OLD.name AND NEW.collection_ids IS DISTINCT FROM OLD.collection_ids)\nEXECUTE PROCEDURE queryables_constraint_triggerfunc();\n\n\nCREATE OR REPLACE FUNCTION array_to_path(arr text[]) RETURNS text AS $$\n SELECT string_agg(\n quote_literal(v),\n '->'\n ) FROM unnest(arr) v;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\n\n\nCREATE OR REPLACE FUNCTION queryable(\n IN dotpath text,\n OUT path text,\n OUT expression text,\n OUT wrapper text,\n OUT nulled_wrapper text\n) AS $$\nDECLARE\n q RECORD;\n path_elements text[];\nBEGIN\n dotpath := replace(dotpath, 'properties.', '');\n IF dotpath = 'start_datetime' THEN\n dotpath := 'datetime';\n END IF;\n IF dotpath IN ('id', 'geometry', 'datetime', 'end_datetime', 'collection') THEN\n path := dotpath;\n expression := dotpath;\n wrapper := NULL;\n RETURN;\n END IF;\n\n SELECT * INTO q FROM queryables\n WHERE\n name=dotpath\n OR name = 'properties.' || dotpath\n OR name = replace(dotpath, 'properties.', '')\n ;\n IF q.property_wrapper IS NULL THEN\n IF q.definition->>'type' = 'number' THEN\n wrapper := 'to_float';\n nulled_wrapper := wrapper;\n ELSIF q.definition->>'format' = 'date-time' THEN\n wrapper := 'to_tstz';\n nulled_wrapper := wrapper;\n ELSE\n nulled_wrapper := NULL;\n wrapper := 'to_text';\n END IF;\n ELSE\n wrapper := q.property_wrapper;\n nulled_wrapper := wrapper;\n END IF;\n IF q.property_path IS NOT NULL THEN\n path := q.property_path;\n ELSE\n path_elements := string_to_array(dotpath, '.');\n IF path_elements[1] IN ('links', 'assets', 'stac_version', 'stac_extensions') THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSIF path_elements[1] = 'properties' THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSE\n path := format($F$content->'properties'->%s$F$, array_to_path(path_elements));\n END IF;\n END IF;\n expression := format('%I(%s)', wrapper, path);\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\nCREATE OR REPLACE FUNCTION unnest_collection(collection_ids text[] DEFAULT NULL) RETURNS SETOF text AS $$\n DECLARE\n BEGIN\n IF collection_ids IS NULL THEN\n RETURN QUERY SELECT id FROM collections;\n END IF;\n RETURN QUERY SELECT unnest(collection_ids);\n END;\n$$ LANGUAGE PLPGSQL STABLE;\n\nCREATE OR REPLACE FUNCTION normalize_indexdef(def text) RETURNS text AS $$\nDECLARE\nBEGIN\n def := btrim(def, ' \\n\\t');\n\tdef := regexp_replace(def, '^CREATE (UNIQUE )?INDEX ([^ ]* )?ON (ONLY )?([^ ]* )?', '', 'i');\n RETURN def;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION indexdef(q queryables) RETURNS text AS $$\n DECLARE\n out text;\n BEGIN\n IF q.name = 'id' THEN\n out := 'CREATE UNIQUE INDEX ON %I USING btree (id)';\n ELSIF q.name = 'datetime' THEN\n out := 'CREATE INDEX ON %I USING btree (datetime DESC, end_datetime)';\n ELSIF q.name = 'geometry' THEN\n out := 'CREATE INDEX ON %I USING gist (geometry)';\n ELSE\n out := format($q$CREATE INDEX ON %%I USING %s (%s(((content -> 'properties'::text) -> %L::text)))$q$,\n lower(COALESCE(q.property_index_type, 'BTREE')),\n lower(COALESCE(q.property_wrapper, 'to_text')),\n q.name\n );\n END IF;\n RETURN btrim(out, ' \\n\\t');\n END;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nDROP VIEW IF EXISTS pgstac_indexes;\nCREATE VIEW pgstac_indexes AS\nSELECT\n i.schemaname,\n i.tablename,\n i.indexname,\n regexp_replace(btrim(replace(replace(indexdef, i.indexname, ''),'pgstac.',''),' \\t\\n'), '[ ]+', ' ', 'g') as idx,\n COALESCE(\n (regexp_match(indexdef, '\\(([a-zA-Z]+)\\)'))[1],\n (regexp_match(indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_-]+)''::text'))[1],\n CASE WHEN indexdef ~* '\\(datetime desc, end_datetime\\)' THEN 'datetime' ELSE NULL END\n ) AS field,\n pg_table_size(i.indexname::text) as index_size,\n pg_size_pretty(pg_table_size(i.indexname::text)) as index_size_pretty\nFROM\n pg_indexes i\nWHERE i.schemaname='pgstac' and i.tablename ~ '_items_' AND indexdef !~* ' only ';\n\nDROP VIEW IF EXISTS pgstac_index_stats;\nCREATE VIEW pgstac_indexes_stats AS\nSELECT\n i.schemaname,\n i.tablename,\n i.indexname,\n indexdef,\n COALESCE(\n (regexp_match(indexdef, '\\(([a-zA-Z]+)\\)'))[1],\n (regexp_match(indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_]+)''::text'))[1],\n CASE WHEN indexdef ~* '\\(datetime desc, end_datetime\\)' THEN 'datetime_end_datetime' ELSE NULL END\n ) AS field,\n pg_table_size(i.indexname::text) as index_size,\n pg_size_pretty(pg_table_size(i.indexname::text)) as index_size_pretty,\n n_distinct,\n most_common_vals::text::text[],\n most_common_freqs::text::text[],\n histogram_bounds::text::text[],\n correlation\nFROM\n pg_indexes i\n LEFT JOIN pg_stats s ON (s.tablename = i.indexname)\nWHERE i.schemaname='pgstac' and i.tablename ~ '_items_';\n\nCREATE OR REPLACE FUNCTION queryable_indexes(\n IN treeroot text DEFAULT 'items',\n IN changes boolean DEFAULT FALSE,\n OUT collection text,\n OUT partition text,\n OUT field text,\n OUT indexname text,\n OUT existing_idx text,\n OUT queryable_idx text\n) RETURNS SETOF RECORD AS $$\nWITH p AS (\n SELECT\n relid::text as partition,\n replace(replace(\n CASE\n WHEN parentrelid::regclass::text='items' THEN pg_get_expr(c.relpartbound, c.oid)\n ELSE pg_get_expr(parent.relpartbound, parent.oid)\n END,\n 'FOR VALUES IN (''',''), ''')',\n ''\n ) AS collection\n FROM pg_partition_tree(treeroot)\n JOIN pg_class c ON (relid::regclass = c.oid)\n JOIN pg_class parent ON (parentrelid::regclass = parent.oid AND isleaf)\n ), i AS (\n SELECT\n partition,\n indexname,\n regexp_replace(btrim(replace(replace(indexdef, indexname, ''),'pgstac.',''),' \\t\\n'), '[ ]+', ' ', 'g') as iidx,\n COALESCE(\n (regexp_match(indexdef, '\\(([a-zA-Z]+)\\)'))[1],\n (regexp_match(indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_-]+)''::text'))[1],\n CASE WHEN indexdef ~* '\\(datetime desc, end_datetime\\)' THEN 'datetime' ELSE NULL END\n ) AS field\n FROM\n pg_indexes\n JOIN p ON (tablename=partition)\n ), q AS (\n SELECT\n name AS field,\n collection,\n partition,\n format(indexdef(queryables), partition) as qidx\n FROM queryables, unnest_collection(queryables.collection_ids) collection\n JOIN p USING (collection)\n WHERE property_index_type IS NOT NULL OR name IN ('datetime','geometry','id')\n )\n SELECT\n collection,\n partition,\n field,\n indexname,\n iidx as existing_idx,\n qidx as queryable_idx\n FROM i FULL JOIN q USING (field, partition)\n WHERE CASE WHEN changes THEN lower(iidx) IS DISTINCT FROM lower(qidx) ELSE TRUE END;\n;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION maintain_index(\n indexname text,\n queryable_idx text,\n dropindexes boolean DEFAULT FALSE,\n rebuildindexes boolean DEFAULT FALSE,\n idxconcurrently boolean DEFAULT FALSE\n) RETURNS VOID AS $$\nDECLARE\nBEGIN\n IF indexname IS NOT NULL THEN\n IF dropindexes OR queryable_idx IS NOT NULL THEN\n EXECUTE format('DROP INDEX IF EXISTS %I;', indexname);\n ELSIF rebuildindexes THEN\n IF idxconcurrently THEN\n EXECUTE format('REINDEX INDEX CONCURRENTLY %I;', indexname);\n ELSE\n EXECUTE format('REINDEX INDEX CONCURRENTLY %I;', indexname);\n END IF;\n END IF;\n END IF;\n IF queryable_idx IS NOT NULL THEN\n IF idxconcurrently THEN\n EXECUTE replace(queryable_idx, 'INDEX', 'INDEX CONCURRENTLY');\n ELSE EXECUTE queryable_idx;\n END IF;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\n\nset check_function_bodies to off;\nCREATE OR REPLACE FUNCTION maintain_partition_queries(\n part text DEFAULT 'items',\n dropindexes boolean DEFAULT FALSE,\n rebuildindexes boolean DEFAULT FALSE,\n idxconcurrently boolean DEFAULT FALSE\n) RETURNS SETOF text AS $$\nDECLARE\n rec record;\n q text;\nBEGIN\n FOR rec IN (\n SELECT * FROM queryable_indexes(part,true)\n ) LOOP\n q := format(\n 'SELECT maintain_index(\n %L,%L,%L,%L,%L\n );',\n rec.indexname,\n rec.queryable_idx,\n dropindexes,\n rebuildindexes,\n idxconcurrently\n );\n RAISE NOTICE 'Q: %', q;\n RETURN NEXT q;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION maintain_partitions(\n part text DEFAULT 'items',\n dropindexes boolean DEFAULT FALSE,\n rebuildindexes boolean DEFAULT FALSE\n) RETURNS VOID AS $$\n WITH t AS (\n SELECT run_or_queue(q) FROM maintain_partition_queries(part, dropindexes, rebuildindexes) q\n ) SELECT count(*) FROM t;\n$$ LANGUAGE SQL;\n\n\nCREATE OR REPLACE FUNCTION queryables_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\nBEGIN\n PERFORM maintain_partitions();\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER queryables_trigger AFTER INSERT OR UPDATE ON queryables\nFOR EACH STATEMENT EXECUTE PROCEDURE queryables_trigger_func();\n\n\nCREATE OR REPLACE FUNCTION get_queryables(_collection_ids text[] DEFAULT NULL) RETURNS jsonb AS $$\nDECLARE\nBEGIN\n -- Build up queryables if the input contains valid collection ids or is empty\n IF EXISTS (\n SELECT 1 FROM collections\n WHERE\n _collection_ids IS NULL\n OR cardinality(_collection_ids) = 0\n OR id = ANY(_collection_ids)\n )\n THEN\n RETURN (\n WITH base AS (\n SELECT\n unnest(collection_ids) as collection_id,\n name,\n coalesce(definition, '{\"type\":\"string\"}'::jsonb) as definition\n FROM queryables\n WHERE\n _collection_ids IS NULL OR\n _collection_ids = '{}'::text[] OR\n _collection_ids && collection_ids\n UNION ALL\n SELECT null, name, coalesce(definition, '{\"type\":\"string\"}'::jsonb) as definition\n FROM queryables WHERE collection_ids IS NULL OR collection_ids = '{}'::text[]\n ), g AS (\n SELECT\n name,\n first_notnull(definition) as definition,\n jsonb_array_unique_merge(definition->'enum') as enum,\n jsonb_min(definition->'minimum') as minimum,\n jsonb_min(definition->'maxiumn') as maximum\n FROM base\n GROUP BY 1\n )\n SELECT\n jsonb_build_object(\n '$schema', 'http://json-schema.org/draft-07/schema#',\n '$id', '',\n 'type', 'object',\n 'title', 'STAC Queryables.',\n 'properties', jsonb_object_agg(\n name,\n definition\n ||\n jsonb_strip_nulls(jsonb_build_object(\n 'enum', enum,\n 'minimum', minimum,\n 'maximum', maximum\n ))\n ),\n 'additionalProperties', pgstac.additional_properties()\n )\n FROM g\n );\n ELSE\n RETURN NULL;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\nCREATE OR REPLACE FUNCTION get_queryables(_collection text DEFAULT NULL) RETURNS jsonb AS $$\n SELECT\n CASE\n WHEN _collection IS NULL THEN get_queryables(NULL::text[])\n ELSE get_queryables(ARRAY[_collection])\n END\n ;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_queryables() RETURNS jsonb AS $$\n SELECT get_queryables(NULL::text[]);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION schema_qualify_refs(url text, j jsonb) returns jsonb as $$\n SELECT regexp_replace(j::text, '\"\\$ref\": \"#', concat('\"$ref\": \"', url, '#'), 'g')::jsonb;\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE VIEW stac_extension_queryables AS\nSELECT DISTINCT key as name, schema_qualify_refs(e.url, j.value) as definition FROM stac_extensions e, jsonb_each(e.content->'definitions'->'fields'->'properties') j;\n\n\nCREATE OR REPLACE FUNCTION missing_queryables(_collection text, _tablesample float DEFAULT 5, minrows float DEFAULT 10) RETURNS TABLE(collection text, name text, definition jsonb, property_wrapper text) AS $$\nDECLARE\n q text;\n _partition text;\n explain_json json;\n psize float;\n estrows float;\nBEGIN\n SELECT format('_items_%s', key) INTO _partition FROM collections WHERE id=_collection;\n\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM %I;', _partition)\n INTO explain_json;\n psize := explain_json->0->'Plan'->'Plan Rows';\n estrows := _tablesample * .01 * psize;\n IF estrows < minrows THEN\n _tablesample := least(100,greatest(_tablesample, (estrows / psize) / 100));\n RAISE NOTICE '%', (psize / estrows) / 100;\n END IF;\n RAISE NOTICE 'Using tablesample % to find missing queryables from % % that has ~% rows estrows: %', _tablesample, _collection, _partition, psize, estrows;\n\n q := format(\n $q$\n WITH q AS (\n SELECT * FROM queryables\n WHERE\n collection_ids IS NULL\n OR %L = ANY(collection_ids)\n ), t AS (\n SELECT\n content->'properties' AS properties\n FROM\n %I\n TABLESAMPLE SYSTEM(%L)\n ), p AS (\n SELECT DISTINCT ON (key)\n key,\n value,\n s.definition\n FROM t\n JOIN LATERAL jsonb_each(properties) ON TRUE\n LEFT JOIN q ON (q.name=key)\n LEFT JOIN stac_extension_queryables s ON (s.name=key)\n WHERE q.definition IS NULL\n )\n SELECT\n %L,\n key,\n COALESCE(definition, jsonb_build_object('type',jsonb_typeof(value))) as definition,\n CASE\n WHEN definition->>'type' = 'integer' THEN 'to_int'\n WHEN COALESCE(definition->>'type', jsonb_typeof(value)) = 'number' THEN 'to_float'\n WHEN COALESCE(definition->>'type', jsonb_typeof(value)) = 'array' THEN 'to_text_array'\n ELSE 'to_text'\n END\n FROM p;\n $q$,\n _collection,\n _partition,\n _tablesample,\n _collection\n );\n RETURN QUERY EXECUTE q;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION missing_queryables(_tablesample float DEFAULT 5) RETURNS TABLE(collection_ids text[], name text, definition jsonb, property_wrapper text) AS $$\n SELECT\n array_agg(collection),\n name,\n definition,\n property_wrapper\n FROM\n collections\n JOIN LATERAL\n missing_queryables(id, _tablesample) c\n ON TRUE\n GROUP BY\n 2,3,4\n ORDER BY 2,1\n ;\n$$ LANGUAGE SQL;\nCREATE OR REPLACE FUNCTION parse_dtrange(\n _indate jsonb,\n relative_base timestamptz DEFAULT date_trunc('hour', CURRENT_TIMESTAMP)\n) RETURNS tstzrange AS $$\nDECLARE\n timestrs text[];\n s timestamptz;\n e timestamptz;\nBEGIN\n timestrs :=\n CASE\n WHEN _indate ? 'timestamp' THEN\n ARRAY[_indate->>'timestamp']\n WHEN _indate ? 'interval' THEN\n to_text_array(_indate->'interval')\n WHEN jsonb_typeof(_indate) = 'array' THEN\n to_text_array(_indate)\n ELSE\n regexp_split_to_array(\n _indate->>0,\n '/'\n )\n END;\n RAISE NOTICE 'TIMESTRS %', timestrs;\n IF cardinality(timestrs) = 1 THEN\n IF timestrs[1] ILIKE 'P%' THEN\n RETURN tstzrange(relative_base - upper(timestrs[1])::interval, relative_base, '[)');\n END IF;\n s := timestrs[1]::timestamptz;\n RETURN tstzrange(s, s, '[]');\n END IF;\n\n IF cardinality(timestrs) != 2 THEN\n RAISE EXCEPTION 'Timestamp cannot have more than 2 values';\n END IF;\n\n IF timestrs[1] = '..' OR timestrs[1] = '' THEN\n s := '-infinity'::timestamptz;\n e := timestrs[2]::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] = '..' OR timestrs[2] = '' THEN\n s := timestrs[1]::timestamptz;\n e := 'infinity'::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[1] ILIKE 'P%' AND timestrs[2] NOT ILIKE 'P%' THEN\n e := timestrs[2]::timestamptz;\n s := e - upper(timestrs[1])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] ILIKE 'P%' AND timestrs[1] NOT ILIKE 'P%' THEN\n s := timestrs[1]::timestamptz;\n e := s + upper(timestrs[2])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n s := timestrs[1]::timestamptz;\n e := timestrs[2]::timestamptz;\n\n RETURN tstzrange(s,e,'[)');\n\n RETURN NULL;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT PARALLEL SAFE SET TIME ZONE 'UTC';\n\nCREATE OR REPLACE FUNCTION parse_dtrange(\n _indate text,\n relative_base timestamptz DEFAULT CURRENT_TIMESTAMP\n) RETURNS tstzrange AS $$\n SELECT parse_dtrange(to_jsonb(_indate), relative_base);\n$$ LANGUAGE SQL STABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION temporal_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\n ll text := 'datetime';\n lh text := 'end_datetime';\n rrange tstzrange;\n rl text;\n rh text;\n outq text;\nBEGIN\n rrange := parse_dtrange(args->1);\n RAISE NOTICE 'Constructing temporal query OP: %, ARGS: %, RRANGE: %', op, args, rrange;\n op := lower(op);\n rl := format('%L::timestamptz', lower(rrange));\n rh := format('%L::timestamptz', upper(rrange));\n outq := CASE op\n WHEN 't_before' THEN 'lh < rl'\n WHEN 't_after' THEN 'll > rh'\n WHEN 't_meets' THEN 'lh = rl'\n WHEN 't_metby' THEN 'll = rh'\n WHEN 't_overlaps' THEN 'll < rl AND rl < lh < rh'\n WHEN 't_overlappedby' THEN 'rl < ll < rh AND lh > rh'\n WHEN 't_starts' THEN 'll = rl AND lh < rh'\n WHEN 't_startedby' THEN 'll = rl AND lh > rh'\n WHEN 't_during' THEN 'll > rl AND lh < rh'\n WHEN 't_contains' THEN 'll < rl AND lh > rh'\n WHEN 't_finishes' THEN 'll > rl AND lh = rh'\n WHEN 't_finishedby' THEN 'll < rl AND lh = rh'\n WHEN 't_equals' THEN 'll = rl AND lh = rh'\n WHEN 't_disjoint' THEN 'NOT (ll <= rh AND lh >= rl)'\n WHEN 't_intersects' THEN 'll <= rh AND lh >= rl'\n WHEN 'anyinteracts' THEN 'll <= rh AND lh >= rl'\n END;\n outq := regexp_replace(outq, '\\mll\\M', ll);\n outq := regexp_replace(outq, '\\mlh\\M', lh);\n outq := regexp_replace(outq, '\\mrl\\M', rl);\n outq := regexp_replace(outq, '\\mrh\\M', rh);\n outq := format('(%s)', outq);\n RETURN outq;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\n\n\nCREATE OR REPLACE FUNCTION spatial_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\n geom text;\n j jsonb := args->1;\nBEGIN\n op := lower(op);\n RAISE NOTICE 'Constructing spatial query OP: %, ARGS: %', op, args;\n IF op NOT IN ('s_equals','s_disjoint','s_touches','s_within','s_overlaps','s_crosses','s_intersects','intersects','s_contains') THEN\n RAISE EXCEPTION 'Spatial Operator % Not Supported', op;\n END IF;\n op := regexp_replace(op, '^s_', 'st_');\n IF op = 'intersects' THEN\n op := 'st_intersects';\n END IF;\n -- Convert geometry to WKB string\n IF j ? 'type' AND j ? 'coordinates' THEN\n geom := st_geomfromgeojson(j)::text;\n ELSIF jsonb_typeof(j) = 'array' THEN\n geom := bbox_geom(j)::text;\n END IF;\n\n RETURN format('%s(geometry, %L::geometry)', op, geom);\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION query_to_cql2(q jsonb) RETURNS jsonb AS $$\n-- Translates anything passed in through the deprecated \"query\" into equivalent CQL2\nWITH t AS (\n SELECT key as property, value as ops\n FROM jsonb_each(q)\n), t2 AS (\n SELECT property, (jsonb_each(ops)).*\n FROM t WHERE jsonb_typeof(ops) = 'object'\n UNION ALL\n SELECT property, 'eq', ops\n FROM t WHERE jsonb_typeof(ops) != 'object'\n)\nSELECT\n jsonb_strip_nulls(jsonb_build_object(\n 'op', 'and',\n 'args', jsonb_agg(\n jsonb_build_object(\n 'op', key,\n 'args', jsonb_build_array(\n jsonb_build_object('property',property),\n value\n )\n )\n )\n )\n) as qcql FROM t2\n;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\nCREATE OR REPLACE FUNCTION cql1_to_cql2(j jsonb) RETURNS jsonb AS $$\nDECLARE\n args jsonb;\n ret jsonb;\nBEGIN\n RAISE NOTICE 'CQL1_TO_CQL2: %', j;\n IF j ? 'filter' THEN\n RETURN cql1_to_cql2(j->'filter');\n END IF;\n IF j ? 'property' THEN\n RETURN j;\n END IF;\n IF jsonb_typeof(j) = 'array' THEN\n SELECT jsonb_agg(cql1_to_cql2(el)) INTO args FROM jsonb_array_elements(j) el;\n RETURN args;\n END IF;\n IF jsonb_typeof(j) = 'number' THEN\n RETURN j;\n END IF;\n IF jsonb_typeof(j) = 'string' THEN\n RETURN j;\n END IF;\n\n IF jsonb_typeof(j) = 'object' THEN\n SELECT jsonb_build_object(\n 'op', key,\n 'args', cql1_to_cql2(value)\n ) INTO ret\n FROM jsonb_each(j)\n WHERE j IS NOT NULL;\n RETURN ret;\n END IF;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE STRICT;\n\nCREATE TABLE cql2_ops (\n op text PRIMARY KEY,\n template text,\n types text[]\n);\nINSERT INTO cql2_ops (op, template, types) VALUES\n ('eq', '%s = %s', NULL),\n ('neq', '%s != %s', NULL),\n ('ne', '%s != %s', NULL),\n ('!=', '%s != %s', NULL),\n ('<>', '%s != %s', NULL),\n ('lt', '%s < %s', NULL),\n ('lte', '%s <= %s', NULL),\n ('gt', '%s > %s', NULL),\n ('gte', '%s >= %s', NULL),\n ('le', '%s <= %s', NULL),\n ('ge', '%s >= %s', NULL),\n ('=', '%s = %s', NULL),\n ('<', '%s < %s', NULL),\n ('<=', '%s <= %s', NULL),\n ('>', '%s > %s', NULL),\n ('>=', '%s >= %s', NULL),\n ('like', '%s LIKE %s', NULL),\n ('ilike', '%s ILIKE %s', NULL),\n ('+', '%s + %s', NULL),\n ('-', '%s - %s', NULL),\n ('*', '%s * %s', NULL),\n ('/', '%s / %s', NULL),\n ('not', 'NOT (%s)', NULL),\n ('between', '%s BETWEEN %s AND %s', NULL),\n ('isnull', '%s IS NULL', NULL),\n ('upper', 'upper(%s)', NULL),\n ('lower', 'lower(%s)', NULL),\n ('casei', 'upper(%s)', NULL),\n ('accenti', 'unaccent(%s)', NULL)\nON CONFLICT (op) DO UPDATE\n SET\n template = EXCLUDED.template\n;\n\n\nCREATE OR REPLACE FUNCTION cql2_query(j jsonb, wrapper text DEFAULT NULL) RETURNS text AS $$\n#variable_conflict use_variable\nDECLARE\n args jsonb := j->'args';\n arg jsonb;\n op text := lower(j->>'op');\n cql2op RECORD;\n literal text;\n _wrapper text;\n leftarg text;\n rightarg text;\n prop text;\n extra_props bool := pgstac.additional_properties();\nBEGIN\n IF j IS NULL OR (op IS NOT NULL AND args IS NULL) THEN\n RETURN NULL;\n END IF;\n RAISE NOTICE 'CQL2_QUERY: %', j;\n\n -- check if all properties are represented in the queryables\n IF NOT extra_props THEN\n FOR prop IN\n SELECT DISTINCT p->>0\n FROM jsonb_path_query(j, 'strict $.**.property') p\n WHERE p->>0 NOT IN ('id', 'datetime', 'geometry', 'end_datetime', 'collection')\n LOOP\n IF (queryable(prop)).nulled_wrapper IS NULL THEN\n RAISE EXCEPTION 'Term % is not found in queryables.', prop;\n END IF;\n END LOOP;\n END IF;\n\n IF j ? 'filter' THEN\n RETURN cql2_query(j->'filter');\n END IF;\n\n IF j ? 'upper' THEN\n RETURN cql2_query(jsonb_build_object('op', 'upper', 'args', j->'upper'));\n END IF;\n\n IF j ? 'lower' THEN\n RETURN cql2_query(jsonb_build_object('op', 'lower', 'args', j->'lower'));\n END IF;\n\n -- Temporal Query\n IF op ilike 't_%' or op = 'anyinteracts' THEN\n RETURN temporal_op_query(op, args);\n END IF;\n\n -- If property is a timestamp convert it to text to use with\n -- general operators\n IF j ? 'timestamp' THEN\n RETURN format('%L::timestamptz', to_tstz(j->'timestamp'));\n END IF;\n IF j ? 'interval' THEN\n RAISE EXCEPTION 'Please use temporal operators when using intervals.';\n RETURN NONE;\n END IF;\n\n -- Spatial Query\n IF op ilike 's_%' or op = 'intersects' THEN\n RETURN spatial_op_query(op, args);\n END IF;\n\n IF op IN ('a_equals','a_contains','a_contained_by','a_overlaps') THEN\n IF args->0 ? 'property' THEN\n leftarg := format('to_text_array(%s)', (queryable(args->0->>'property')).path);\n END IF;\n IF args->1 ? 'property' THEN\n rightarg := format('to_text_array(%s)', (queryable(args->1->>'property')).path);\n END IF;\n RETURN FORMAT(\n '%s %s %s',\n COALESCE(leftarg, quote_literal(to_text_array(args->0))),\n CASE op\n WHEN 'a_equals' THEN '='\n WHEN 'a_contains' THEN '@>'\n WHEN 'a_contained_by' THEN '<@'\n WHEN 'a_overlaps' THEN '&&'\n END,\n COALESCE(rightarg, quote_literal(to_text_array(args->1)))\n );\n END IF;\n\n IF op = 'in' THEN\n RAISE NOTICE 'IN : % % %', args, jsonb_build_array(args->0), args->1;\n args := jsonb_build_array(args->0) || (args->1);\n RAISE NOTICE 'IN2 : %', args;\n END IF;\n\n\n\n IF op = 'between' THEN\n args = jsonb_build_array(\n args->0,\n args->1,\n args->2\n );\n END IF;\n\n -- Make sure that args is an array and run cql2_query on\n -- each element of the array\n RAISE NOTICE 'ARGS PRE: %', args;\n IF j ? 'args' THEN\n IF jsonb_typeof(args) != 'array' THEN\n args := jsonb_build_array(args);\n END IF;\n\n IF jsonb_path_exists(args, '$[*] ? (@.property == \"id\" || @.property == \"datetime\" || @.property == \"end_datetime\" || @.property == \"collection\")') THEN\n wrapper := NULL;\n ELSE\n -- if any of the arguments are a property, try to get the property_wrapper\n FOR arg IN SELECT jsonb_path_query(args, '$[*] ? (@.property != null)') LOOP\n RAISE NOTICE 'Arg: %', arg;\n wrapper := (queryable(arg->>'property')).nulled_wrapper;\n RAISE NOTICE 'Property: %, Wrapper: %', arg, wrapper;\n IF wrapper IS NOT NULL THEN\n EXIT;\n END IF;\n END LOOP;\n\n -- if the property was not in queryables, see if any args were numbers\n IF\n wrapper IS NULL\n AND jsonb_path_exists(args, '$[*] ? (@.type()==\"number\")')\n THEN\n wrapper := 'to_float';\n END IF;\n wrapper := coalesce(wrapper, 'to_text');\n END IF;\n\n SELECT jsonb_agg(cql2_query(a, wrapper))\n INTO args\n FROM jsonb_array_elements(args) a;\n END IF;\n RAISE NOTICE 'ARGS: %', args;\n\n IF op IN ('and', 'or') THEN\n RETURN\n format(\n '(%s)',\n array_to_string(to_text_array(args), format(' %s ', upper(op)))\n );\n END IF;\n\n IF op = 'in' THEN\n RAISE NOTICE 'IN -- % %', args->0, to_text(args->0);\n RETURN format(\n '%s IN (%s)',\n to_text(args->0),\n array_to_string((to_text_array(args))[2:], ',')\n );\n END IF;\n\n -- Look up template from cql2_ops\n IF j ? 'op' THEN\n SELECT * INTO cql2op FROM cql2_ops WHERE cql2_ops.op ilike op;\n IF FOUND THEN\n -- If specific index set in queryables for a property cast other arguments to that type\n\n RETURN format(\n cql2op.template,\n VARIADIC (to_text_array(args))\n );\n ELSE\n RAISE EXCEPTION 'Operator % Not Supported.', op;\n END IF;\n END IF;\n\n\n IF wrapper IS NOT NULL THEN\n RAISE NOTICE 'Wrapping % with %', j, wrapper;\n IF j ? 'property' THEN\n RETURN format('%I(%s)', wrapper, (queryable(j->>'property')).path);\n ELSE\n RETURN format('%I(%L)', wrapper, j);\n END IF;\n ELSIF j ? 'property' THEN\n RETURN quote_ident(j->>'property');\n END IF;\n\n RETURN quote_literal(to_text(j));\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION paging_dtrange(\n j jsonb\n) RETURNS tstzrange AS $$\nDECLARE\n op text;\n filter jsonb := j->'filter';\n dtrange tstzrange := tstzrange('-infinity'::timestamptz,'infinity'::timestamptz);\n sdate timestamptz := '-infinity'::timestamptz;\n edate timestamptz := 'infinity'::timestamptz;\n jpitem jsonb;\nBEGIN\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n END IF;\n IF NOT (filter @? '$.**.op ? (@ == \"or\" || @ == \"not\")') THEN\n FOR jpitem IN SELECT j FROM jsonb_path_query(filter,'strict $.** ? (@.args[*].property == \"datetime\")'::jsonpath) j LOOP\n op := lower(jpitem->>'op');\n dtrange := parse_dtrange(jpitem->'args'->1);\n IF op IN ('<=', 'lt', 'lte', '<', 'le', 't_before') THEN\n sdate := greatest(sdate,'-infinity');\n edate := least(edate, upper(dtrange));\n ELSIF op IN ('>=', '>', 'gt', 'gte', 'ge', 't_after') THEN\n edate := least(edate, 'infinity');\n sdate := greatest(sdate, lower(dtrange));\n ELSIF op IN ('=', 'eq') THEN\n edate := least(edate, upper(dtrange));\n sdate := greatest(sdate, lower(dtrange));\n END IF;\n RAISE NOTICE '2 OP: %, ARGS: %, DTRANGE: %, SDATE: %, EDATE: %', op, jpitem->'args'->1, dtrange, sdate, edate;\n END LOOP;\n END IF;\n IF sdate > edate THEN\n RETURN 'empty'::tstzrange;\n END IF;\n RETURN tstzrange(sdate,edate, '[]');\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT SET TIME ZONE 'UTC';\n\nCREATE OR REPLACE FUNCTION paging_collections(\n IN j jsonb\n) RETURNS text[] AS $$\nDECLARE\n filter jsonb := j->'filter';\n jpitem jsonb;\n op text;\n args jsonb;\n arg jsonb;\n collections text[];\nBEGIN\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n END IF;\n IF NOT (filter @? '$.**.op ? (@ == \"or\" || @ == \"not\")') THEN\n FOR jpitem IN SELECT j FROM jsonb_path_query(filter,'strict $.** ? (@.args[*].property == \"collection\")'::jsonpath) j LOOP\n RAISE NOTICE 'JPITEM: %', jpitem;\n op := jpitem->>'op';\n args := jpitem->'args';\n IF op IN ('=', 'eq', 'in') THEN\n FOR arg IN SELECT a FROM jsonb_array_elements(args) a LOOP\n IF jsonb_typeof(arg) IN ('string', 'array') THEN\n RAISE NOTICE 'arg: %, collections: %', arg, collections;\n IF collections IS NULL OR collections = '{}'::text[] THEN\n collections := to_text_array(arg);\n ELSE\n collections := array_intersection(collections, to_text_array(arg));\n END IF;\n END IF;\n END LOOP;\n END IF;\n END LOOP;\n END IF;\n IF collections = '{}'::text[] THEN\n RETURN NULL;\n END IF;\n RETURN collections;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\nCREATE TABLE items (\n id text NOT NULL,\n geometry geometry NOT NULL,\n collection text NOT NULL,\n datetime timestamptz NOT NULL,\n end_datetime timestamptz NOT NULL,\n content JSONB NOT NULL,\n private jsonb\n)\nPARTITION BY LIST (collection)\n;\n\nCREATE INDEX \"datetime_idx\" ON items USING BTREE (datetime DESC, end_datetime ASC);\nCREATE INDEX \"geometry_idx\" ON items USING GIST (geometry);\n\nCREATE STATISTICS datetime_stats (dependencies) on datetime, end_datetime from items;\n\nALTER TABLE items ADD CONSTRAINT items_collections_fk FOREIGN KEY (collection) REFERENCES collections(id) ON DELETE CASCADE DEFERRABLE;\n\nCREATE OR REPLACE FUNCTION partition_after_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p text;\n t timestamptz := clock_timestamp();\nBEGIN\n RAISE NOTICE 'Updating partition stats %', t;\n FOR p IN SELECT DISTINCT partition\n FROM newdata n JOIN partition_sys_meta p\n ON (n.collection=p.collection AND n.datetime <@ p.partition_dtrange)\n LOOP\n PERFORM run_or_queue(format('SELECT update_partition_stats(%L, %L);', p, true));\n END LOOP;\n IF TG_OP IN ('DELETE','UPDATE') THEN\n DELETE FROM format_item_cache c USING newdata n WHERE c.collection = n.collection AND c.id = n.id;\n END IF;\n RAISE NOTICE 't: % %', t, clock_timestamp() - t;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE TRIGGER items_after_insert_trigger\nAFTER INSERT ON items\nREFERENCING NEW TABLE AS newdata\nFOR EACH STATEMENT\nEXECUTE FUNCTION partition_after_triggerfunc();\n\nCREATE TRIGGER items_after_update_trigger\nAFTER DELETE ON items\nREFERENCING OLD TABLE AS newdata\nFOR EACH STATEMENT\nEXECUTE FUNCTION partition_after_triggerfunc();\n\nCREATE TRIGGER items_after_delete_trigger\nAFTER UPDATE ON items\nREFERENCING NEW TABLE AS newdata\nFOR EACH STATEMENT\nEXECUTE FUNCTION partition_after_triggerfunc();\n\n\nCREATE OR REPLACE FUNCTION content_slim(_item jsonb) RETURNS jsonb AS $$\n SELECT strip_jsonb(_item - '{id,geometry,collection,type}'::text[], collection_base_item(_item->>'collection')) - '{id,geometry,collection,type}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_dehydrate(content jsonb) RETURNS items AS $$\n SELECT\n content->>'id' as id,\n stac_geom(content) as geometry,\n content->>'collection' as collection,\n stac_datetime(content) as datetime,\n stac_end_datetime(content) as end_datetime,\n content_slim(content) as content,\n null::jsonb as private\n ;\n$$ LANGUAGE SQL STABLE;\n\nCREATE OR REPLACE FUNCTION include_field(f text, fields jsonb DEFAULT '{}'::jsonb) RETURNS boolean AS $$\nDECLARE\n includes jsonb := fields->'include';\n excludes jsonb := fields->'exclude';\nBEGIN\n IF f IS NULL THEN\n RETURN NULL;\n END IF;\n\n\n IF\n jsonb_typeof(excludes) = 'array'\n AND jsonb_array_length(excludes)>0\n AND excludes ? f\n THEN\n RETURN FALSE;\n END IF;\n\n IF\n (\n jsonb_typeof(includes) = 'array'\n AND jsonb_array_length(includes) > 0\n AND includes ? f\n ) OR\n (\n includes IS NULL\n OR jsonb_typeof(includes) = 'null'\n OR jsonb_array_length(includes) = 0\n )\n THEN\n RETURN TRUE;\n END IF;\n\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nDROP FUNCTION IF EXISTS content_hydrate(jsonb, jsonb, jsonb);\nCREATE OR REPLACE FUNCTION content_hydrate(\n _item jsonb,\n _base_item jsonb,\n fields jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\n SELECT merge_jsonb(\n jsonb_fields(_item, fields),\n jsonb_fields(_base_item, fields)\n );\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION content_hydrate(_item items, _collection collections, fields jsonb DEFAULT '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\n content jsonb;\n base_item jsonb := _collection.base_item;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry, 20)::jsonb;\n END IF;\n output := content_hydrate(\n jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'collection', _item.collection,\n 'type', 'Feature'\n ) || _item.content,\n _collection.base_item,\n fields\n );\n\n RETURN output;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_nonhydrated(\n _item items,\n fields jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry, 20)::jsonb;\n END IF;\n output := jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'collection', _item.collection,\n 'type', 'Feature'\n ) || _item.content;\n RETURN output;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_hydrate(_item items, fields jsonb DEFAULT '{}'::jsonb) RETURNS jsonb AS $$\n SELECT content_hydrate(\n _item,\n (SELECT c FROM collections c WHERE id=_item.collection LIMIT 1),\n fields\n );\n$$ LANGUAGE SQL STABLE;\n\n\nCREATE UNLOGGED TABLE items_staging (\n content JSONB NOT NULL\n);\nCREATE UNLOGGED TABLE items_staging_ignore (\n content JSONB NOT NULL\n);\nCREATE UNLOGGED TABLE items_staging_upsert (\n content JSONB NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION items_staging_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p record;\n _partitions text[];\n part text;\n ts timestamptz := clock_timestamp();\n nrows int;\nBEGIN\n RAISE NOTICE 'Creating Partitions. %', clock_timestamp() - ts;\n\n FOR part IN WITH t AS (\n SELECT\n n.content->>'collection' as collection,\n stac_daterange(n.content->'properties') as dtr,\n partition_trunc\n FROM newdata n JOIN collections ON (n.content->>'collection'=collections.id)\n ), p AS (\n SELECT\n collection,\n COALESCE(date_trunc(partition_trunc::text, lower(dtr)),'-infinity') as d,\n tstzrange(min(lower(dtr)),max(lower(dtr)),'[]') as dtrange,\n tstzrange(min(upper(dtr)),max(upper(dtr)),'[]') as edtrange\n FROM t\n GROUP BY 1,2\n ) SELECT check_partition(collection, dtrange, edtrange) FROM p LOOP\n RAISE NOTICE 'Partition %', part;\n END LOOP;\n\n RAISE NOTICE 'Creating temp table with data to be added. %', clock_timestamp() - ts;\n DROP TABLE IF EXISTS tmpdata;\n CREATE TEMP TABLE tmpdata ON COMMIT DROP AS\n SELECT\n (content_dehydrate(content)).*\n FROM newdata;\n GET DIAGNOSTICS nrows = ROW_COUNT;\n RAISE NOTICE 'Added % rows to tmpdata. %', nrows, clock_timestamp() - ts;\n\n RAISE NOTICE 'Doing the insert. %', clock_timestamp() - ts;\n IF TG_TABLE_NAME = 'items_staging' THEN\n INSERT INTO items\n SELECT * FROM tmpdata;\n GET DIAGNOSTICS nrows = ROW_COUNT;\n RAISE NOTICE 'Inserted % rows to items. %', nrows, clock_timestamp() - ts;\n ELSIF TG_TABLE_NAME = 'items_staging_ignore' THEN\n INSERT INTO items\n SELECT * FROM tmpdata\n ON CONFLICT DO NOTHING;\n GET DIAGNOSTICS nrows = ROW_COUNT;\n RAISE NOTICE 'Inserted % rows to items. %', nrows, clock_timestamp() - ts;\n ELSIF TG_TABLE_NAME = 'items_staging_upsert' THEN\n DELETE FROM items i USING tmpdata s\n WHERE\n i.id = s.id\n AND i.collection = s.collection\n AND i IS DISTINCT FROM s\n ;\n GET DIAGNOSTICS nrows = ROW_COUNT;\n RAISE NOTICE 'Deleted % rows from items. %', nrows, clock_timestamp() - ts;\n INSERT INTO items AS t\n SELECT * FROM tmpdata\n ON CONFLICT DO NOTHING;\n GET DIAGNOSTICS nrows = ROW_COUNT;\n RAISE NOTICE 'Inserted % rows to items. %', nrows, clock_timestamp() - ts;\n END IF;\n\n RAISE NOTICE 'Deleting data from staging table. %', clock_timestamp() - ts;\n DELETE FROM items_staging;\n RAISE NOTICE 'Done. %', clock_timestamp() - ts;\n\n RETURN NULL;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER items_staging_insert_trigger AFTER INSERT ON items_staging REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\nCREATE TRIGGER items_staging_insert_ignore_trigger AFTER INSERT ON items_staging_ignore REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\nCREATE TRIGGER items_staging_insert_upsert_trigger AFTER INSERT ON items_staging_upsert REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\n\nCREATE OR REPLACE FUNCTION item_by_id(_id text, _collection text DEFAULT NULL) RETURNS items AS\n$$\nDECLARE\n i items%ROWTYPE;\nBEGIN\n SELECT * INTO i FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection) LIMIT 1;\n RETURN i;\nEND;\n$$ LANGUAGE PLPGSQL STABLE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION get_item(_id text, _collection text DEFAULT NULL) RETURNS jsonb AS $$\n SELECT content_hydrate(items) FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection);\n$$ LANGUAGE SQL STABLE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION delete_item(_id text, _collection text DEFAULT NULL) RETURNS VOID AS $$\nDECLARE\nout items%ROWTYPE;\nBEGIN\n DELETE FROM items WHERE id = _id AND (_collection IS NULL OR collection=_collection) RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n--/*\nCREATE OR REPLACE FUNCTION create_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION update_item(content jsonb) RETURNS VOID AS $$\nDECLARE\n old items %ROWTYPE;\n out items%ROWTYPE;\nBEGIN\n PERFORM delete_item(content->>'id', content->>'collection');\n PERFORM create_item(content);\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION create_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION collection_bbox(id text) RETURNS jsonb AS $$\n SELECT (replace(replace(replace(st_extent(geometry)::text,'BOX(','[['),')',']]'),' ',','))::jsonb\n FROM items WHERE collection=$1;\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION collection_temporal_extent(id text) RETURNS jsonb AS $$\n SELECT to_jsonb(array[array[min(datetime), max(datetime)]])\n FROM items WHERE collection=$1;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection_extents() RETURNS VOID AS $$\nUPDATE collections\n SET content = jsonb_set_lax(\n content,\n '{extent}'::text[],\n collection_extent(id, FALSE),\n true,\n 'use_json_null'\n )\n;\n$$ LANGUAGE SQL;\nCREATE TABLE partition_stats (\n partition text PRIMARY KEY,\n dtrange tstzrange,\n edtrange tstzrange,\n spatial geometry,\n last_updated timestamptz,\n keys text[]\n) WITH (FILLFACTOR=90);\n\nCREATE INDEX partitions_range_idx ON partition_stats USING GIST(dtrange);\n\n\nCREATE OR REPLACE FUNCTION constraint_tstzrange(expr text) RETURNS tstzrange AS $$\n WITH t AS (\n SELECT regexp_matches(\n expr,\n E'\\\\(''\\([0-9 :+-]*\\)''\\\\).*\\\\(''\\([0-9 :+-]*\\)''\\\\)'\n ) AS m\n ) SELECT tstzrange(m[1]::timestamptz, m[2]::timestamptz) FROM t\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE STRICT;\n\nCREATE OR REPLACE FUNCTION dt_constraint(coid oid, OUT dt tstzrange, OUT edt tstzrange) RETURNS RECORD AS $$\nDECLARE\n expr text := pg_get_constraintdef(coid);\n matches timestamptz[];\nBEGIN\n IF expr LIKE '%NULL%' THEN\n dt := tstzrange(null::timestamptz, null::timestamptz);\n edt := tstzrange(null::timestamptz, null::timestamptz);\n RETURN;\n END IF;\n WITH f AS (SELECT (regexp_matches(expr, E'([0-9]{4}-[0-1][0-9]-[0-3][0-9] [0-2][0-9]:[0-5][0-9]:[0-5][0-9]\\.?[0-9]*)', 'g'))[1] f)\n SELECT array_agg(f::timestamptz) INTO matches FROM f;\n IF cardinality(matches) = 4 THEN\n dt := tstzrange(matches[1], matches[2],'[]');\n edt := tstzrange(matches[3], matches[4], '[]');\n RETURN;\n ELSIF cardinality(matches) = 2 THEN\n edt := tstzrange(matches[1], matches[2],'[]');\n RETURN;\n END IF;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\nCREATE OR REPLACE VIEW partition_sys_meta AS\nSELECT\n (parse_ident(relid::text))[cardinality(parse_ident(relid::text))] as partition,\n replace(replace(CASE WHEN level = 1 THEN pg_get_expr(c.relpartbound, c.oid)\n ELSE pg_get_expr(parent.relpartbound, parent.oid)\n END, 'FOR VALUES IN (''',''), ''')','') AS collection,\n level,\n c.reltuples,\n c.relhastriggers,\n COALESCE(constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as partition_dtrange,\n COALESCE((dt_constraint(edt.oid)).dt, constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as constraint_dtrange,\n COALESCE((dt_constraint(edt.oid)).edt, tstzrange('-infinity', 'infinity','[]')) as constraint_edtrange\nFROM\n pg_partition_tree('items')\n JOIN pg_class c ON (relid::regclass = c.oid)\n JOIN pg_class parent ON (parentrelid::regclass = parent.oid AND isleaf)\n LEFT JOIN pg_constraint edt ON (conrelid=c.oid AND contype='c')\nWHERE isleaf\n;\n\nCREATE OR REPLACE VIEW partitions_view AS\nSELECT\n (parse_ident(relid::text))[cardinality(parse_ident(relid::text))] as partition,\n replace(replace(CASE WHEN level = 1 THEN pg_get_expr(c.relpartbound, c.oid)\n ELSE pg_get_expr(parent.relpartbound, parent.oid)\n END, 'FOR VALUES IN (''',''), ''')','') AS collection,\n level,\n c.reltuples,\n c.relhastriggers,\n COALESCE(pgstac.constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as partition_dtrange,\n COALESCE((pgstac.dt_constraint(edt.oid)).dt, pgstac.constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as constraint_dtrange,\n COALESCE((pgstac.dt_constraint(edt.oid)).edt, tstzrange('-infinity', 'infinity','[]')) as constraint_edtrange,\n dtrange,\n edtrange,\n spatial,\n last_updated\nFROM\n pg_partition_tree('pgstac.items')\n JOIN pg_class c ON (relid::regclass = c.oid)\n JOIN pg_class parent ON (parentrelid::regclass = parent.oid AND isleaf)\n LEFT JOIN pg_constraint edt ON (conrelid=c.oid AND contype='c')\n LEFT JOIN pgstac.partition_stats ON ((parse_ident(relid::text))[cardinality(parse_ident(relid::text))]=partition)\nWHERE isleaf\n;\n\nCREATE MATERIALIZED VIEW partitions AS\nSELECT * FROM partitions_view;\nCREATE UNIQUE INDEX ON partitions (partition);\n\nCREATE MATERIALIZED VIEW partition_steps AS\nSELECT\n partition as name,\n date_trunc('month',lower(partition_dtrange)) as sdate,\n date_trunc('month', upper(partition_dtrange)) + '1 month'::interval as edate\n FROM partitions_view WHERE partition_dtrange IS NOT NULL AND partition_dtrange != 'empty'::tstzrange\n ORDER BY dtrange ASC\n;\n\n\nCREATE OR REPLACE FUNCTION update_partition_stats_q(_partition text, istrigger boolean default false) RETURNS VOID AS $$\nDECLARE\nBEGIN\n PERFORM run_or_queue(\n format('SELECT update_partition_stats(%L, %L);', _partition, istrigger)\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION update_partition_stats(_partition text, istrigger boolean default false) RETURNS VOID AS $$\nDECLARE\n dtrange tstzrange;\n edtrange tstzrange;\n cdtrange tstzrange;\n cedtrange tstzrange;\n extent geometry;\n collection text;\nBEGIN\n RAISE NOTICE 'Updating stats for %.', _partition;\n EXECUTE format(\n $q$\n SELECT\n tstzrange(min(datetime), max(datetime),'[]'),\n tstzrange(min(end_datetime), max(end_datetime), '[]')\n FROM %I\n $q$,\n _partition\n ) INTO dtrange, edtrange;\n extent := st_estimatedextent('pgstac', _partition, 'geometry');\n RAISE DEBUG 'Estimated Extent: %', extent;\n INSERT INTO partition_stats (partition, dtrange, edtrange, spatial, last_updated)\n SELECT _partition, dtrange, edtrange, extent, now()\n ON CONFLICT (partition) DO\n UPDATE SET\n dtrange=EXCLUDED.dtrange,\n edtrange=EXCLUDED.edtrange,\n spatial=EXCLUDED.spatial,\n last_updated=EXCLUDED.last_updated\n ;\n\n SELECT\n constraint_dtrange, constraint_edtrange, pv.collection\n INTO cdtrange, cedtrange, collection\n FROM partitions_view pv WHERE partition = _partition;\n REFRESH MATERIALIZED VIEW partitions;\n REFRESH MATERIALIZED VIEW partition_steps;\n\n\n RAISE NOTICE 'Checking if we need to modify constraints...';\n RAISE NOTICE 'cdtrange: % dtrange: % cedtrange: % edtrange: %',cdtrange, dtrange, cedtrange, edtrange;\n IF\n (cdtrange IS DISTINCT FROM dtrange OR edtrange IS DISTINCT FROM cedtrange)\n AND NOT istrigger\n THEN\n RAISE NOTICE 'Modifying Constraints';\n RAISE NOTICE 'Existing % %', cdtrange, cedtrange;\n RAISE NOTICE 'New % %', dtrange, edtrange;\n PERFORM drop_table_constraints(_partition);\n PERFORM create_table_constraints(_partition, dtrange, edtrange);\n REFRESH MATERIALIZED VIEW partitions;\n REFRESH MATERIALIZED VIEW partition_steps;\n END IF;\n RAISE NOTICE 'Checking if we need to update collection extents.';\n IF get_setting_bool('update_collection_extent') THEN\n RAISE NOTICE 'updating collection extent for %', collection;\n PERFORM run_or_queue(format($q$\n UPDATE collections\n SET content = jsonb_set_lax(\n content,\n '{extent}'::text[],\n collection_extent(%L, FALSE),\n true,\n 'use_json_null'\n ) WHERE id=%L\n ;\n $q$, collection, collection));\n ELSE\n RAISE NOTICE 'Not updating collection extent for %', collection;\n END IF;\n\nEND;\n$$ LANGUAGE PLPGSQL STRICT SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION partition_name( IN collection text, IN dt timestamptz, OUT partition_name text, OUT partition_range tstzrange) AS $$\nDECLARE\n c RECORD;\n parent_name text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n parent_name := format('_items_%s', c.key);\n\n\n IF c.partition_trunc = 'year' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYY'));\n ELSIF c.partition_trunc = 'month' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYYMM'));\n ELSE\n partition_name := parent_name;\n partition_range := tstzrange('-infinity'::timestamptz, 'infinity'::timestamptz, '[]');\n END IF;\n IF partition_range IS NULL THEN\n partition_range := tstzrange(\n date_trunc(c.partition_trunc::text, dt),\n date_trunc(c.partition_trunc::text, dt) + concat('1 ', c.partition_trunc)::interval\n );\n END IF;\n RETURN;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION drop_table_constraints(t text) RETURNS text AS $$\nDECLARE\n q text;\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM partitions_view WHERE partition=t) THEN\n RETURN NULL;\n END IF;\n FOR q IN SELECT FORMAT(\n $q$\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n $q$,\n t,\n conname\n ) FROM pg_constraint\n WHERE conrelid=t::regclass::oid AND contype='c'\n LOOP\n EXECUTE q;\n END LOOP;\n RETURN t;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE OR REPLACE FUNCTION create_table_constraints(t text, _dtrange tstzrange, _edtrange tstzrange) RETURNS text AS $$\nDECLARE\n q text;\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM partitions_view WHERE partition=t) THEN\n RETURN NULL;\n END IF;\n RAISE NOTICE 'Creating Table Constraints for % % %', t, _dtrange, _edtrange;\n IF _dtrange = 'empty' AND _edtrange = 'empty' THEN\n q :=format(\n $q$\n DO $block$\n BEGIN\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n ALTER TABLE %I\n ADD CONSTRAINT %I\n CHECK (((datetime IS NULL) AND (end_datetime IS NULL))) NOT VALID\n ;\n ALTER TABLE %I\n VALIDATE CONSTRAINT %I\n ;\n\n\n\n EXCEPTION WHEN others THEN\n RAISE WARNING '%%, Issue Altering Constraints. Please run update_partition_stats(%I)', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n $block$;\n $q$,\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t),\n t\n );\n ELSE\n q :=format(\n $q$\n DO $block$\n BEGIN\n\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n ALTER TABLE %I\n ADD CONSTRAINT %I\n CHECK (\n (datetime >= %L)\n AND (datetime <= %L)\n AND (end_datetime >= %L)\n AND (end_datetime <= %L)\n ) NOT VALID\n ;\n ALTER TABLE %I\n VALIDATE CONSTRAINT %I\n ;\n\n\n\n EXCEPTION WHEN others THEN\n RAISE WARNING '%%, Issue Altering Constraints. Please run update_partition_stats(%I)', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n $block$;\n $q$,\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t),\n lower(_dtrange),\n upper(_dtrange),\n lower(_edtrange),\n upper(_edtrange),\n t,\n format('%s_dt', t),\n t\n );\n END IF;\n PERFORM run_or_queue(q);\n RETURN t;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION check_partition(\n _collection text,\n _dtrange tstzrange,\n _edtrange tstzrange\n) RETURNS text AS $$\nDECLARE\n c RECORD;\n pm RECORD;\n _partition_name text;\n _partition_dtrange tstzrange;\n _constraint_dtrange tstzrange;\n _constraint_edtrange tstzrange;\n q text;\n deferrable_q text;\n err_context text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=_collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', _collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n\n IF c.partition_trunc IS NOT NULL THEN\n _partition_dtrange := tstzrange(\n date_trunc(c.partition_trunc, lower(_dtrange)),\n date_trunc(c.partition_trunc, lower(_dtrange)) + (concat('1 ', c.partition_trunc))::interval,\n '[)'\n );\n ELSE\n _partition_dtrange := '[-infinity, infinity]'::tstzrange;\n END IF;\n\n IF NOT _partition_dtrange @> _dtrange THEN\n RAISE EXCEPTION 'dtrange % is greater than the partition size % for collection %', _dtrange, c.partition_trunc, _collection;\n END IF;\n\n\n IF c.partition_trunc = 'year' THEN\n _partition_name := format('_items_%s_%s', c.key, to_char(lower(_partition_dtrange),'YYYY'));\n ELSIF c.partition_trunc = 'month' THEN\n _partition_name := format('_items_%s_%s', c.key, to_char(lower(_partition_dtrange),'YYYYMM'));\n ELSE\n _partition_name := format('_items_%s', c.key);\n END IF;\n\n SELECT * INTO pm FROM partition_sys_meta WHERE collection=_collection AND partition_dtrange @> _dtrange;\n IF FOUND THEN\n RAISE NOTICE '% % %', _edtrange, _dtrange, pm;\n _constraint_edtrange :=\n tstzrange(\n least(\n lower(_edtrange),\n nullif(lower(pm.constraint_edtrange), '-infinity')\n ),\n greatest(\n upper(_edtrange),\n nullif(upper(pm.constraint_edtrange), 'infinity')\n ),\n '[]'\n );\n _constraint_dtrange :=\n tstzrange(\n least(\n lower(_dtrange),\n nullif(lower(pm.constraint_dtrange), '-infinity')\n ),\n greatest(\n upper(_dtrange),\n nullif(upper(pm.constraint_dtrange), 'infinity')\n ),\n '[]'\n );\n\n IF pm.constraint_edtrange @> _edtrange AND pm.constraint_dtrange @> _dtrange THEN\n RETURN pm.partition;\n ELSE\n PERFORM drop_table_constraints(_partition_name);\n END IF;\n ELSE\n _constraint_edtrange := _edtrange;\n _constraint_dtrange := _dtrange;\n END IF;\n RAISE NOTICE 'EXISTING CONSTRAINTS % %, NEW % %', pm.constraint_dtrange, pm.constraint_edtrange, _constraint_dtrange, _constraint_edtrange;\n RAISE NOTICE 'Creating partition % %', _partition_name, _partition_dtrange;\n IF c.partition_trunc IS NULL THEN\n q := format(\n $q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n GRANT ALL ON %I to pgstac_ingest;\n $q$,\n _partition_name,\n _collection,\n concat(_partition_name,'_pk'),\n _partition_name,\n _partition_name\n );\n ELSE\n q := format(\n $q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L) PARTITION BY RANGE (datetime);\n CREATE TABLE IF NOT EXISTS %I partition OF %I FOR VALUES FROM (%L) TO (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n GRANT ALL ON %I TO pgstac_ingest;\n $q$,\n format('_items_%s', c.key),\n _collection,\n _partition_name,\n format('_items_%s', c.key),\n lower(_partition_dtrange),\n upper(_partition_dtrange),\n format('%s_pk', _partition_name),\n _partition_name,\n _partition_name\n );\n END IF;\n\n BEGIN\n EXECUTE q;\n EXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', _partition_name;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\n END;\n PERFORM maintain_partitions(_partition_name);\n PERFORM update_partition_stats_q(_partition_name, true);\n REFRESH MATERIALIZED VIEW partitions;\n REFRESH MATERIALIZED VIEW partition_steps;\n RETURN _partition_name;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION repartition(_collection text, _partition_trunc text, triggered boolean DEFAULT FALSE) RETURNS text AS $$\nDECLARE\n c RECORD;\n q text;\n from_trunc text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=_collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', _collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n IF triggered THEN\n RAISE NOTICE 'Converting % to % partitioning via Trigger', _collection, _partition_trunc;\n ELSE\n RAISE NOTICE 'Converting % from using % to % partitioning', _collection, c.partition_trunc, _partition_trunc;\n IF c.partition_trunc IS NOT DISTINCT FROM _partition_trunc THEN\n RAISE NOTICE 'Collection % already set to use partition by %', _collection, _partition_trunc;\n RETURN _collection;\n END IF;\n END IF;\n\n IF EXISTS (SELECT 1 FROM partitions_view WHERE collection=_collection LIMIT 1) THEN\n EXECUTE format(\n $q$\n CREATE TEMP TABLE changepartitionstaging ON COMMIT DROP AS SELECT * FROM %I;\n DROP TABLE IF EXISTS %I CASCADE;\n WITH p AS (\n SELECT\n collection,\n CASE\n WHEN %L IS NULL THEN '-infinity'::timestamptz\n ELSE date_trunc(%L, datetime)\n END as d,\n tstzrange(min(datetime),max(datetime),'[]') as dtrange,\n tstzrange(min(end_datetime),max(end_datetime),'[]') as edtrange\n FROM changepartitionstaging\n GROUP BY 1,2\n ) SELECT check_partition(collection, dtrange, edtrange) FROM p;\n INSERT INTO items SELECT * FROM changepartitionstaging;\n DROP TABLE changepartitionstaging;\n $q$,\n concat('_items_', c.key),\n concat('_items_', c.key),\n c.partition_trunc,\n c.partition_trunc\n );\n END IF;\n RETURN _collection;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE OR REPLACE FUNCTION collections_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n q text;\n partition_name text := format('_items_%s', NEW.key);\n partition_exists boolean := false;\n partition_empty boolean := true;\n err_context text;\n loadtemp boolean := FALSE;\nBEGIN\n RAISE NOTICE 'Collection Trigger. % %', NEW.id, NEW.key;\n IF TG_OP = 'UPDATE' AND NEW.partition_trunc IS DISTINCT FROM OLD.partition_trunc THEN\n PERFORM repartition(NEW.id, NEW.partition_trunc, TRUE);\n END IF;\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER collections_trigger AFTER\nINSERT\nOR\nUPDATE ON collections\nFOR EACH ROW EXECUTE FUNCTION collections_trigger_func();\n\nCREATE OR REPLACE FUNCTION chunker(\n IN _where text,\n OUT s timestamptz,\n OUT e timestamptz\n) RETURNS SETOF RECORD AS $$\nDECLARE\n explain jsonb;\nBEGIN\n IF _where IS NULL THEN\n _where := ' TRUE ';\n END IF;\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s;', _where)\n INTO explain;\n RAISE DEBUG 'EXPLAIN: %', explain;\n\n RETURN QUERY\n WITH t AS (\n SELECT j->>0 as p FROM\n jsonb_path_query(\n explain,\n 'strict $.**.\"Relation Name\" ? (@ != null)'\n ) j\n ),\n parts AS (\n SELECT sdate, edate FROM t JOIN partition_steps ON (t.p = name)\n ),\n times AS (\n SELECT sdate FROM parts\n UNION\n SELECT edate FROM parts\n ),\n uniq AS (\n SELECT DISTINCT sdate FROM times ORDER BY sdate\n ),\n last AS (\n SELECT sdate, lead(sdate, 1) over () as edate FROM uniq\n )\n SELECT sdate, edate FROM last WHERE edate IS NOT NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION partition_queries(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN partitions text[] DEFAULT NULL\n) RETURNS SETOF text AS $$\nDECLARE\n query text;\n sdate timestamptz;\n edate timestamptz;\nBEGIN\nIF _where IS NULL OR trim(_where) = '' THEN\n _where = ' TRUE ';\nEND IF;\nRAISE NOTICE 'Getting chunks for % %', _where, _orderby;\nIF _orderby ILIKE 'datetime d%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 DESC LOOP\n RETURN NEXT format($q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n $q$,\n sdate,\n edate,\n _where,\n _orderby\n );\n END LOOP;\nELSIF _orderby ILIKE 'datetime a%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 ASC LOOP\n RETURN NEXT format($q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n $q$,\n sdate,\n edate,\n _where,\n _orderby\n );\n END LOOP;\nELSE\n query := format($q$\n SELECT * FROM items\n WHERE %s\n ORDER BY %s\n $q$, _where, _orderby\n );\n\n RETURN NEXT query;\n RETURN;\nEND IF;\n\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION partition_query_view(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN _limit int DEFAULT 10\n) RETURNS text AS $$\n WITH p AS (\n SELECT * FROM partition_queries(_where, _orderby) p\n )\n SELECT\n CASE WHEN EXISTS (SELECT 1 FROM p) THEN\n (SELECT format($q$\n SELECT * FROM (\n %s\n ) total LIMIT %s\n $q$,\n string_agg(\n format($q$ SELECT * FROM ( %s ) AS sub $q$, p),\n '\n UNION ALL\n '\n ),\n _limit\n ))\n ELSE NULL\n END FROM p;\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION q_to_tsquery (input text)\n RETURNS tsquery\n AS $$\nDECLARE\n processed_text text;\n temp_text text;\n quote_array text[];\n placeholder text := '@QUOTE@';\nBEGIN\n -- Extract all quoted phrases and store in array\n quote_array := regexp_matches(input, '\"[^\"]*\"', 'g');\n\n -- Replace each quoted part with a unique placeholder if there are any quoted phrases\n IF array_length(quote_array, 1) IS NOT NULL THEN\n processed_text := input;\n FOR i IN array_lower(quote_array, 1) .. array_upper(quote_array, 1) LOOP\n processed_text := replace(processed_text, quote_array[i], placeholder || i || placeholder);\n END LOOP;\n ELSE\n processed_text := input;\n END IF;\n\n -- Replace non-quoted text using regular expressions\n\n -- , -> |\n processed_text := regexp_replace(processed_text, ',(?=(?:[^\"]*\"[^\"]*\")*[^\"]*$)', ' | ', 'g');\n\n -- and -> &\n processed_text := regexp_replace(processed_text, '\\s+AND\\s+', ' & ', 'gi');\n\n -- or -> |\n processed_text := regexp_replace(processed_text, '\\s+OR\\s+', ' | ', 'gi');\n\n -- +term -> & term\n processed_text := regexp_replace(processed_text, '\\+([a-zA-Z0-9_]+)', '& \\1', 'g');\n\n -- -term -> ! term\n processed_text := regexp_replace(processed_text, '\\-([a-zA-Z0-9_]+)', '& ! \\1', 'g');\n\n -- Replace placeholders back with quoted phrases if there were any\n IF array_length(quote_array, 1) IS NOT NULL THEN\n FOR i IN array_lower(quote_array, 1) .. array_upper(quote_array, 1) LOOP\n processed_text := replace(processed_text, placeholder || i || placeholder, '''' || substring(quote_array[i] from 2 for length(quote_array[i]) - 2) || '''');\n END LOOP;\n END IF;\n\n -- Print processed_text to the console for debugging purposes\n RAISE NOTICE 'processed_text: %', processed_text;\n\n RETURN to_tsquery(processed_text);\nEND;\n$$\nLANGUAGE plpgsql;\n\n\nCREATE OR REPLACE FUNCTION stac_search_to_where(j jsonb) RETURNS text AS $$\nDECLARE\n where_segments text[];\n _where text;\n dtrange tstzrange;\n collections text[];\n geom geometry;\n sdate timestamptz;\n edate timestamptz;\n filterlang text;\n filter jsonb := j->'filter';\n ft_query tsquery;\nBEGIN\n IF j ? 'ids' THEN\n where_segments := where_segments || format('id = ANY (%L) ', to_text_array(j->'ids'));\n END IF;\n\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n where_segments := where_segments || format('collection = ANY (%L) ', collections);\n END IF;\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n\n where_segments := where_segments || format(' datetime <= %L::timestamptz AND end_datetime >= %L::timestamptz ',\n edate,\n sdate\n );\n END IF;\n\n IF j ? 'q' THEN\n ft_query := q_to_tsquery(j->>'q');\n where_segments := where_segments || format(\n $quote$\n (\n to_tsvector('english', content->'properties'->>'description') ||\n to_tsvector('english', coalesce(content->'properties'->>'title', '')) ||\n to_tsvector('english', coalesce(content->'properties'->>'keywords', ''))\n ) @@ %L\n $quote$,\n ft_query\n );\n END IF;\n\n geom := stac_geom(j);\n IF geom IS NOT NULL THEN\n where_segments := where_segments || format('st_intersects(geometry, %L)',geom);\n END IF;\n\n filterlang := COALESCE(\n j->>'filter-lang',\n get_setting('default_filter_lang', j->'conf')\n );\n IF NOT filter @? '$.**.op' THEN\n filterlang := 'cql-json';\n END IF;\n\n IF filterlang NOT IN ('cql-json','cql2-json') AND j ? 'filter' THEN\n RAISE EXCEPTION '% is not a supported filter-lang. Please use cql-json or cql2-json.', filterlang;\n END IF;\n\n IF j ? 'query' AND j ? 'filter' THEN\n RAISE EXCEPTION 'Can only use either query or filter at one time.';\n END IF;\n\n IF j ? 'query' THEN\n filter := query_to_cql2(j->'query');\n ELSIF filterlang = 'cql-json' THEN\n filter := cql1_to_cql2(filter);\n END IF;\n RAISE NOTICE 'FILTER: %', filter;\n where_segments := where_segments || cql2_query(filter);\n IF cardinality(where_segments) < 1 THEN\n RETURN ' TRUE ';\n END IF;\n\n _where := array_to_string(array_remove(where_segments, NULL), ' AND ');\n\n IF _where IS NULL OR BTRIM(_where) = '' THEN\n RETURN ' TRUE ';\n END IF;\n RETURN _where;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION parse_sort_dir(_dir text, reverse boolean default false) RETURNS text AS $$\n WITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n ) SELECT\n CASE\n WHEN NOT reverse THEN d\n WHEN d = 'ASC' THEN 'DESC'\n WHEN d = 'DESC' THEN 'ASC'\n END\n FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION sort_dir_to_op(_dir text, prev boolean default false) RETURNS text AS $$\n WITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n ) SELECT\n CASE\n WHEN d = 'ASC' AND prev THEN '<='\n WHEN d = 'DESC' AND prev THEN '>='\n WHEN d = 'ASC' THEN '>='\n WHEN d = 'DESC' THEN '<='\n END\n FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION sort_sqlorderby(\n _search jsonb DEFAULT NULL,\n reverse boolean DEFAULT FALSE\n) RETURNS text AS $$\n WITH sortby AS (\n SELECT coalesce(_search->'sortby','[{\"field\":\"datetime\", \"direction\":\"desc\"}]') as sort\n ), withid AS (\n SELECT CASE\n WHEN sort @? '$[*] ? (@.field == \"id\")' THEN sort\n ELSE sort || '[{\"field\":\"id\", \"direction\":\"desc\"}]'::jsonb\n END as sort\n FROM sortby\n ), withid_rows AS (\n SELECT jsonb_array_elements(sort) as value FROM withid\n ),sorts AS (\n SELECT\n coalesce(\n (queryable(value->>'field')).expression\n ) as key,\n parse_sort_dir(value->>'direction', reverse) as dir\n FROM withid_rows\n )\n SELECT array_to_string(\n array_agg(concat(key, ' ', dir)),\n ', '\n ) FROM sorts;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_sort_dir(sort_item jsonb) RETURNS text AS $$\n SELECT CASE WHEN sort_item->>'direction' ILIKE 'desc%' THEN 'DESC' ELSE 'ASC' END;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION get_token_val_str(\n _field text,\n _item items\n) RETURNS text AS $$\nDECLARE\n q text;\n literal text;\nBEGIN\n q := format($q$ SELECT quote_literal(%s) FROM (SELECT $1.*) as r;$q$, _field);\n EXECUTE q INTO literal USING _item;\n RETURN literal;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n\nCREATE OR REPLACE FUNCTION get_token_record(IN _token text, OUT prev BOOLEAN, OUT item items) RETURNS RECORD AS $$\nDECLARE\n _itemid text := _token;\n _collectionid text;\nBEGIN\n IF _token IS NULL THEN\n RETURN;\n END IF;\n RAISE NOTICE 'Looking for token: %', _token;\n prev := FALSE;\n IF _token ILIKE 'prev:%' THEN\n _itemid := replace(_token, 'prev:','');\n prev := TRUE;\n ELSIF _token ILIKE 'next:%' THEN\n _itemid := replace(_token, 'next:', '');\n END IF;\n SELECT id INTO _collectionid FROM collections WHERE _itemid LIKE concat(id,':%');\n IF FOUND THEN\n _itemid := replace(_itemid, concat(_collectionid,':'), '');\n SELECT * INTO item FROM items WHERE id=_itemid AND collection=_collectionid;\n ELSE\n SELECT * INTO item FROM items WHERE id=_itemid;\n END IF;\n IF item IS NULL THEN\n RAISE EXCEPTION 'Could not find item using token: % item: % collection: %', _token, _itemid, _collectionid;\n END IF;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\n\nCREATE OR REPLACE FUNCTION get_token_filter(\n _sortby jsonb DEFAULT '[{\"field\":\"datetime\",\"direction\":\"desc\"}]'::jsonb,\n token_item items DEFAULT NULL,\n prev boolean DEFAULT FALSE,\n inclusive boolean DEFAULT FALSE\n) RETURNS text AS $$\nDECLARE\n ltop text := '<';\n gtop text := '>';\n dir text;\n sort record;\n orfilter text := '';\n orfilters text[] := '{}'::text[];\n andfilters text[] := '{}'::text[];\n output text;\n token_where text;\nBEGIN\n IF _sortby IS NULL OR _sortby = '[]'::jsonb THEN\n _sortby := '[{\"field\":\"datetime\",\"direction\":\"desc\"}]'::jsonb;\n END IF;\n _sortby := _sortby || jsonb_build_object('field','id','direction',_sortby->0->>'direction');\n RAISE NOTICE 'Getting Token Filter. % %', _sortby, token_item;\n IF inclusive THEN\n orfilters := orfilters || format('( id=%L AND collection=%L )' , token_item.id, token_item.collection);\n END IF;\n\n FOR sort IN\n WITH s1 AS (\n SELECT\n _row,\n (queryable(value->>'field')).expression as _field,\n (value->>'field' = 'id') as _isid,\n get_sort_dir(value) as _dir\n FROM jsonb_array_elements(_sortby)\n WITH ORDINALITY AS t(value, _row)\n )\n SELECT\n _row,\n _field,\n _dir,\n get_token_val_str(_field, token_item) as _val\n FROM s1\n WHERE _row <= (SELECT min(_row) FROM s1 WHERE _isid)\n LOOP\n orfilter := NULL;\n RAISE NOTICE 'SORT: %', sort;\n IF sort._val IS NOT NULL AND ((prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC')) THEN\n orfilter := format($f$(\n (%s %s %s) OR (%s IS NULL)\n )$f$,\n sort._field,\n ltop,\n sort._val,\n sort._val\n );\n ELSIF sort._val IS NULL AND ((prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC')) THEN\n RAISE NOTICE '< but null';\n orfilter := format('%s IS NOT NULL', sort._field);\n ELSIF sort._val IS NULL THEN\n RAISE NOTICE '> but null';\n ELSE\n orfilter := format($f$(\n (%s %s %s) OR (%s IS NULL)\n )$f$,\n sort._field,\n gtop,\n sort._val,\n sort._field\n );\n END IF;\n RAISE NOTICE 'ORFILTER: %', orfilter;\n\n IF orfilter IS NOT NULL THEN\n IF sort._row = 1 THEN\n orfilters := orfilters || orfilter;\n ELSE\n orfilters := orfilters || format('(%s AND %s)', array_to_string(andfilters, ' AND '), orfilter);\n END IF;\n END IF;\n IF sort._val IS NOT NULL THEN\n andfilters := andfilters || format('%s = %s', sort._field, sort._val);\n ELSE\n andfilters := andfilters || format('%s IS NULL', sort._field);\n END IF;\n END LOOP;\n\n output := array_to_string(orfilters, ' OR ');\n\n token_where := concat('(',coalesce(output,'true'),')');\n IF trim(token_where) = '' THEN\n token_where := NULL;\n END IF;\n RAISE NOTICE 'TOKEN_WHERE: %',token_where;\n RETURN token_where;\n END;\n$$ LANGUAGE PLPGSQL SET transform_null_equals TO TRUE\n;\n\nCREATE OR REPLACE FUNCTION search_tohash(jsonb) RETURNS jsonb AS $$\n SELECT $1 - '{token,limit,context,includes,excludes}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION search_hash(jsonb, jsonb) RETURNS text AS $$\n SELECT md5(concat(search_tohash($1)::text,$2::text));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TABLE IF NOT EXISTS searches(\n hash text GENERATED ALWAYS AS (search_hash(search, metadata)) STORED PRIMARY KEY,\n search jsonb NOT NULL,\n _where text,\n orderby text,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n metadata jsonb DEFAULT '{}'::jsonb NOT NULL\n);\n\nCREATE TABLE IF NOT EXISTS search_wheres(\n id bigint generated always as identity primary key,\n _where text NOT NULL,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n statslastupdated timestamptz,\n estimated_count bigint,\n estimated_cost float,\n time_to_estimate float,\n total_count bigint,\n time_to_count float,\n partitions text[]\n);\n\nCREATE INDEX IF NOT EXISTS search_wheres_partitions ON search_wheres USING GIN (partitions);\nCREATE UNIQUE INDEX IF NOT EXISTS search_wheres_where ON search_wheres ((md5(_where)));\n\nCREATE OR REPLACE FUNCTION where_stats(\n inwhere text,\n updatestats boolean default false,\n conf jsonb default null\n) RETURNS search_wheres AS $$\nDECLARE\n t timestamptz;\n i interval;\n explain_json jsonb;\n partitions text[];\n sw search_wheres%ROWTYPE;\n inwhere_hash text := md5(inwhere);\n _context text := lower(context(conf));\n _stats_ttl interval := context_stats_ttl(conf);\n _estimated_cost_threshold float := context_estimated_cost(conf);\n _estimated_count_threshold int := context_estimated_count(conf);\n ro bool := pgstac.readonly(conf);\nBEGIN\n -- If updatestats is true then set ttl to 0\n IF updatestats THEN\n RAISE DEBUG 'Updatestats set to TRUE, setting TTL to 0';\n _stats_ttl := '0'::interval;\n END IF;\n\n -- If we don't need to calculate context, just return\n IF _context = 'off' THEN\n sw._where = inwhere;\n RETURN sw;\n END IF;\n\n -- Get any stats that we have.\n IF NOT ro THEN\n -- If there is a lock where another process is\n -- updating the stats, wait so that we don't end up calculating a bunch of times.\n SELECT * INTO sw FROM search_wheres WHERE md5(_where)=inwhere_hash FOR UPDATE;\n ELSE\n SELECT * INTO sw FROM search_wheres WHERE md5(_where)=inwhere_hash;\n END IF;\n\n -- If there is a cached row, figure out if we need to update\n IF\n sw IS NOT NULL\n AND sw.statslastupdated IS NOT NULL\n AND sw.total_count IS NOT NULL\n AND now() - sw.statslastupdated <= _stats_ttl\n THEN\n -- we have a cached row with data that is within our ttl\n RAISE DEBUG 'Stats present in table and lastupdated within ttl: %', sw;\n IF NOT ro THEN\n RAISE DEBUG 'Updating search_wheres only bumping lastused and usecount';\n UPDATE search_wheres SET\n lastused = now(),\n usecount = search_wheres.usecount + 1\n WHERE md5(_where) = inwhere_hash\n RETURNING * INTO sw;\n END IF;\n RAISE DEBUG 'Returning cached counts. %', sw;\n RETURN sw;\n END IF;\n\n -- Calculate estimated cost and rows\n -- Use explain to get estimated count/cost\n IF sw.estimated_count IS NULL OR sw.estimated_cost IS NULL THEN\n RAISE DEBUG 'Calculating estimated stats';\n t := clock_timestamp();\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s', inwhere)\n INTO explain_json;\n RAISE DEBUG 'Time for just the explain: %', clock_timestamp() - t;\n i := clock_timestamp() - t;\n\n sw.estimated_count := explain_json->0->'Plan'->'Plan Rows';\n sw.estimated_cost := explain_json->0->'Plan'->'Total Cost';\n sw.time_to_estimate := extract(epoch from i);\n END IF;\n\n RAISE DEBUG 'ESTIMATED_COUNT: %, THRESHOLD %', sw.estimated_count, _estimated_count_threshold;\n RAISE DEBUG 'ESTIMATED_COST: %, THRESHOLD %', sw.estimated_cost, _estimated_cost_threshold;\n\n -- If context is set to auto and the costs are within the threshold return the estimated costs\n IF\n _context = 'auto'\n AND sw.estimated_count >= _estimated_count_threshold\n AND sw.estimated_cost >= _estimated_cost_threshold\n THEN\n IF NOT ro THEN\n INSERT INTO search_wheres (\n _where,\n lastused,\n usecount,\n statslastupdated,\n estimated_count,\n estimated_cost,\n time_to_estimate,\n total_count,\n time_to_count\n ) VALUES (\n inwhere,\n now(),\n 1,\n now(),\n sw.estimated_count,\n sw.estimated_cost,\n sw.time_to_estimate,\n null,\n null\n ) ON CONFLICT ((md5(_where)))\n DO UPDATE SET\n lastused = EXCLUDED.lastused,\n usecount = search_wheres.usecount + 1,\n statslastupdated = EXCLUDED.statslastupdated,\n estimated_count = EXCLUDED.estimated_count,\n estimated_cost = EXCLUDED.estimated_cost,\n time_to_estimate = EXCLUDED.time_to_estimate,\n total_count = EXCLUDED.total_count,\n time_to_count = EXCLUDED.time_to_count\n RETURNING * INTO sw;\n END IF;\n RAISE DEBUG 'Estimates are within thresholds, returning estimates. %', sw;\n RETURN sw;\n END IF;\n\n -- Calculate Actual Count\n t := clock_timestamp();\n RAISE NOTICE 'Calculating actual count...';\n EXECUTE format(\n 'SELECT count(*) FROM items WHERE %s',\n inwhere\n ) INTO sw.total_count;\n i := clock_timestamp() - t;\n RAISE NOTICE 'Actual Count: % -- %', sw.total_count, i;\n sw.time_to_count := extract(epoch FROM i);\n\n IF NOT ro THEN\n INSERT INTO search_wheres (\n _where,\n lastused,\n usecount,\n statslastupdated,\n estimated_count,\n estimated_cost,\n time_to_estimate,\n total_count,\n time_to_count\n ) VALUES (\n inwhere,\n now(),\n 1,\n now(),\n sw.estimated_count,\n sw.estimated_cost,\n sw.time_to_estimate,\n sw.total_count,\n sw.time_to_count\n ) ON CONFLICT ((md5(_where)))\n DO UPDATE SET\n lastused = EXCLUDED.lastused,\n usecount = search_wheres.usecount + 1,\n statslastupdated = EXCLUDED.statslastupdated,\n estimated_count = EXCLUDED.estimated_count,\n estimated_cost = EXCLUDED.estimated_cost,\n time_to_estimate = EXCLUDED.time_to_estimate,\n total_count = EXCLUDED.total_count,\n time_to_count = EXCLUDED.time_to_count\n RETURNING * INTO sw;\n END IF;\n RAISE DEBUG 'Returning with actual count. %', sw;\n RETURN sw;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION search_query(\n _search jsonb = '{}'::jsonb,\n updatestats boolean = false,\n _metadata jsonb = '{}'::jsonb\n) RETURNS searches AS $$\nDECLARE\n search searches%ROWTYPE;\n cached_search searches%ROWTYPE;\n pexplain jsonb;\n t timestamptz;\n i interval;\n doupdate boolean := FALSE;\n insertfound boolean := FALSE;\n ro boolean := pgstac.readonly();\n found_search text;\nBEGIN\n RAISE NOTICE 'SEARCH: %', _search;\n -- Calculate hash, where clause, and order by statement\n search.search := _search;\n search.metadata := _metadata;\n search.hash := search_hash(_search, _metadata);\n search._where := stac_search_to_where(_search);\n search.orderby := sort_sqlorderby(_search);\n search.lastused := now();\n search.usecount := 1;\n\n -- If we are in read only mode, directly return search\n IF ro THEN\n RETURN search;\n END IF;\n\n RAISE NOTICE 'Updating Statistics for search: %s', search;\n -- Update statistics for times used and and when last used\n -- If the entry is locked, rather than waiting, skip updating the stats\n INSERT INTO searches (search, lastused, usecount, metadata)\n VALUES (search.search, now(), 1, search.metadata)\n ON CONFLICT DO NOTHING\n RETURNING * INTO cached_search\n ;\n\n IF NOT FOUND OR cached_search IS NULL THEN\n UPDATE searches SET\n lastused = now(),\n usecount = searches.usecount + 1\n WHERE hash = (\n SELECT hash FROM searches WHERE hash=search.hash FOR UPDATE SKIP LOCKED\n )\n RETURNING * INTO cached_search\n ;\n END IF;\n\n IF cached_search IS NOT NULL THEN\n cached_search._where = search._where;\n cached_search.orderby = search.orderby;\n RETURN cached_search;\n END IF;\n RETURN search;\n\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE OR REPLACE FUNCTION search_fromhash(\n _hash text\n) RETURNS searches AS $$\n SELECT * FROM search_query((SELECT search FROM searches WHERE hash=_hash LIMIT 1));\n$$ LANGUAGE SQL STRICT;\n\nCREATE OR REPLACE FUNCTION search_rows(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN partitions text[] DEFAULT NULL,\n IN _limit int DEFAULT 10\n) RETURNS SETOF items AS $$\nDECLARE\n base_query text;\n query text;\n sdate timestamptz;\n edate timestamptz;\n n int;\n records_left int := _limit;\n timer timestamptz := clock_timestamp();\n full_timer timestamptz := clock_timestamp();\nBEGIN\nIF _where IS NULL OR trim(_where) = '' THEN\n _where = ' TRUE ';\nEND IF;\nRAISE NOTICE 'Getting chunks for % %', _where, _orderby;\n\nbase_query := $q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n LIMIT %L\n$q$;\n\nIF _orderby ILIKE 'datetime d%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 DESC LOOP\n RAISE NOTICE 'Running Query for % to %. %', sdate, edate, age_ms(full_timer);\n query := format(\n base_query,\n sdate,\n edate,\n _where,\n _orderby,\n records_left\n );\n RAISE DEBUG 'QUERY: %', query;\n timer := clock_timestamp();\n RETURN QUERY EXECUTE query;\n\n GET DIAGNOSTICS n = ROW_COUNT;\n records_left := records_left - n;\n RAISE NOTICE 'Returned %/% Rows From % to %. % to go. Time: %ms', n, _limit, sdate, edate, records_left, age_ms(timer);\n timer := clock_timestamp();\n IF records_left <= 0 THEN\n RAISE NOTICE 'SEARCH_ROWS TOOK %ms', age_ms(full_timer);\n RETURN;\n END IF;\n END LOOP;\nELSIF _orderby ILIKE 'datetime a%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 ASC LOOP\n RAISE NOTICE 'Running Query for % to %. %', sdate, edate, age_ms(full_timer);\n query := format(\n base_query,\n sdate,\n edate,\n _where,\n _orderby,\n records_left\n );\n RAISE DEBUG 'QUERY: %', query;\n timer := clock_timestamp();\n RETURN QUERY EXECUTE query;\n\n GET DIAGNOSTICS n = ROW_COUNT;\n records_left := records_left - n;\n RAISE NOTICE 'Returned %/% Rows From % to %. % to go. Time: %ms', n, _limit, sdate, edate, records_left, age_ms(timer);\n timer := clock_timestamp();\n IF records_left <= 0 THEN\n RAISE NOTICE 'SEARCH_ROWS TOOK %ms', age_ms(full_timer);\n RETURN;\n END IF;\n END LOOP;\nELSE\n query := format($q$\n SELECT * FROM items\n WHERE %s\n ORDER BY %s\n LIMIT %L\n $q$, _where, _orderby, _limit\n );\n RAISE DEBUG 'QUERY: %', query;\n timer := clock_timestamp();\n RETURN QUERY EXECUTE query;\n RAISE NOTICE 'FULL QUERY TOOK %ms', age_ms(timer);\nEND IF;\nRAISE NOTICE 'SEARCH_ROWS TOOK %ms', age_ms(full_timer);\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE UNLOGGED TABLE format_item_cache(\n id text,\n collection text,\n fields text,\n hydrated bool,\n output jsonb,\n lastused timestamptz DEFAULT now(),\n usecount int DEFAULT 1,\n timetoformat float,\n PRIMARY KEY (collection, id, fields, hydrated)\n);\nCREATE INDEX ON format_item_cache (lastused);\n\nCREATE OR REPLACE FUNCTION format_item(_item items, _fields jsonb DEFAULT '{}', _hydrated bool DEFAULT TRUE) RETURNS jsonb AS $$\nDECLARE\n cache bool := get_setting_bool('format_cache');\n _output jsonb := null;\n t timestamptz := clock_timestamp();\nBEGIN\n IF cache THEN\n SELECT output INTO _output FROM format_item_cache\n WHERE id=_item.id AND collection=_item.collection AND fields=_fields::text AND hydrated=_hydrated;\n END IF;\n IF _output IS NULL THEN\n IF _hydrated THEN\n _output := content_hydrate(_item, _fields);\n ELSE\n _output := content_nonhydrated(_item, _fields);\n END IF;\n END IF;\n IF cache THEN\n INSERT INTO format_item_cache (id, collection, fields, hydrated, output, timetoformat)\n VALUES (_item.id, _item.collection, _fields::text, _hydrated, _output, age_ms(t))\n ON CONFLICT(collection, id, fields, hydrated) DO\n UPDATE\n SET lastused=now(), usecount = format_item_cache.usecount + 1\n ;\n END IF;\n RETURN _output;\n\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION search(_search jsonb = '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n searches searches%ROWTYPE;\n _where text;\n orderby text;\n search_where search_wheres%ROWTYPE;\n total_count bigint;\n token record;\n token_prev boolean;\n token_item items%ROWTYPE;\n token_where text;\n full_where text;\n init_ts timestamptz := clock_timestamp();\n timer timestamptz := clock_timestamp();\n hydrate bool := NOT (_search->'conf'->>'nohydrate' IS NOT NULL AND (_search->'conf'->>'nohydrate')::boolean = true);\n prev text;\n next text;\n context jsonb;\n collection jsonb;\n out_records jsonb;\n out_len int;\n _limit int := coalesce((_search->>'limit')::int, 10);\n _querylimit int;\n _fields jsonb := coalesce(_search->'fields', '{}'::jsonb);\n has_prev boolean := FALSE;\n has_next boolean := FALSE;\n links jsonb := '[]'::jsonb;\n base_url text:= concat(rtrim(base_url(_search->'conf'),'/'));\nBEGIN\n searches := search_query(_search);\n _where := searches._where;\n orderby := searches.orderby;\n search_where := where_stats(_where);\n total_count := coalesce(search_where.total_count, search_where.estimated_count);\n RAISE NOTICE 'SEARCH:TOKEN: %', _search->>'token';\n token := get_token_record(_search->>'token');\n RAISE NOTICE '***TOKEN: %', token;\n _querylimit := _limit + 1;\n IF token IS NOT NULL THEN\n token_prev := token.prev;\n token_item := token.item;\n token_where := get_token_filter(_search->'sortby', token_item, token_prev, FALSE);\n RAISE DEBUG 'TOKEN_WHERE: % (%ms from search start)', token_where, age_ms(timer);\n IF token_prev THEN -- if we are using a prev token, we know has_next is true\n RAISE DEBUG 'There is a previous token, so automatically setting has_next to true';\n has_next := TRUE;\n orderby := sort_sqlorderby(_search, TRUE);\n ELSE\n RAISE DEBUG 'There is a next token, so automatically setting has_prev to true';\n has_prev := TRUE;\n\n END IF;\n ELSE -- if there was no token, we know there is no prev\n RAISE DEBUG 'There is no token, so we know there is no prev. setting has_prev to false';\n has_prev := FALSE;\n END IF;\n\n full_where := concat_ws(' AND ', _where, token_where);\n RAISE NOTICE 'FULL WHERE CLAUSE: %', full_where;\n RAISE NOTICE 'Time to get counts and build query %', age_ms(timer);\n timer := clock_timestamp();\n\n IF hydrate THEN\n RAISE NOTICE 'Getting hydrated data.';\n ELSE\n RAISE NOTICE 'Getting non-hydrated data.';\n END IF;\n RAISE NOTICE 'CACHE SET TO %', get_setting_bool('format_cache');\n RAISE NOTICE 'Time to set hydration/formatting %', age_ms(timer);\n timer := clock_timestamp();\n SELECT jsonb_agg(format_item(i, _fields, hydrate)) INTO out_records\n FROM search_rows(\n full_where,\n orderby,\n search_where.partitions,\n _querylimit\n ) as i;\n\n RAISE NOTICE 'Time to fetch rows %', age_ms(timer);\n timer := clock_timestamp();\n\n\n IF token_prev THEN\n out_records := flip_jsonb_array(out_records);\n END IF;\n\n RAISE NOTICE 'Query returned % records.', jsonb_array_length(out_records);\n RAISE DEBUG 'TOKEN: % %', token_item.id, token_item.collection;\n RAISE DEBUG 'RECORD_1: % %', out_records->0->>'id', out_records->0->>'collection';\n RAISE DEBUG 'RECORD-1: % %', out_records->-1->>'id', out_records->-1->>'collection';\n\n -- REMOVE records that were from our token\n IF out_records->0->>'id' = token_item.id AND out_records->0->>'collection' = token_item.collection THEN\n out_records := out_records - 0;\n ELSIF out_records->-1->>'id' = token_item.id AND out_records->-1->>'collection' = token_item.collection THEN\n out_records := out_records - -1;\n END IF;\n\n out_len := jsonb_array_length(out_records);\n\n IF out_len = _limit + 1 THEN\n IF token_prev THEN\n has_prev := TRUE;\n out_records := out_records - 0;\n ELSE\n has_next := TRUE;\n out_records := out_records - -1;\n END IF;\n END IF;\n\n\n links := links || jsonb_build_object(\n 'rel', 'root',\n 'type', 'application/json',\n 'href', base_url\n ) || jsonb_build_object(\n 'rel', 'self',\n 'type', 'application/json',\n 'href', concat(base_url, '/search')\n );\n\n IF has_next THEN\n next := concat(out_records->-1->>'collection', ':', out_records->-1->>'id');\n RAISE NOTICE 'HAS NEXT | %', next;\n links := links || jsonb_build_object(\n 'rel', 'next',\n 'type', 'application/geo+json',\n 'method', 'GET',\n 'href', concat(base_url, '/search?token=next:', next)\n );\n END IF;\n\n IF has_prev THEN\n prev := concat(out_records->0->>'collection', ':', out_records->0->>'id');\n RAISE NOTICE 'HAS PREV | %', prev;\n links := links || jsonb_build_object(\n 'rel', 'prev',\n 'type', 'application/geo+json',\n 'method', 'GET',\n 'href', concat(base_url, '/search?token=prev:', prev)\n );\n END IF;\n\n RAISE NOTICE 'Time to get prev/next %', age_ms(timer);\n timer := clock_timestamp();\n\n\n collection := jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb),\n 'links', links\n );\n\n\n\n IF context(_search->'conf') != 'off' THEN\n collection := collection || jsonb_strip_nulls(jsonb_build_object(\n 'numberMatched', total_count,\n 'numberReturned', coalesce(jsonb_array_length(out_records), 0)\n ));\n ELSE\n collection := collection || jsonb_strip_nulls(jsonb_build_object(\n 'numberReturned', coalesce(jsonb_array_length(out_records), 0)\n ));\n END IF;\n\n IF get_setting_bool('timing', _search->'conf') THEN\n collection = collection || jsonb_build_object('timing', age_ms(init_ts));\n END IF;\n\n RAISE NOTICE 'Time to build final json %', age_ms(timer);\n timer := clock_timestamp();\n\n RAISE NOTICE 'Total Time: %', age_ms(current_timestamp);\n RAISE NOTICE 'RETURNING % records. NEXT: %. PREV: %', collection->>'numberReturned', collection->>'next', collection->>'prev';\n RETURN collection;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION search_cursor(_search jsonb = '{}'::jsonb) RETURNS refcursor AS $$\nDECLARE\n curs refcursor;\n searches searches%ROWTYPE;\n _where text;\n _orderby text;\n q text;\n\nBEGIN\n searches := search_query(_search);\n _where := searches._where;\n _orderby := searches.orderby;\n\n OPEN curs FOR\n WITH p AS (\n SELECT * FROM partition_queries(_where, _orderby) p\n )\n SELECT\n CASE WHEN EXISTS (SELECT 1 FROM p) THEN\n (SELECT format($q$\n SELECT * FROM (\n %s\n ) total\n $q$,\n string_agg(\n format($q$ SELECT * FROM ( %s ) AS sub $q$, p),\n '\n UNION ALL\n '\n )\n ))\n ELSE NULL\n END FROM p;\n RETURN curs;\nEND;\n$$ LANGUAGE PLPGSQL;\nCREATE OR REPLACE VIEW collections_asitems AS\nSELECT\n id,\n geometry,\n 'collections' AS collection,\n datetime,\n end_datetime,\n jsonb_build_object(\n 'properties', content - '{links,assets,stac_version,stac_extensions}',\n 'links', content->'links',\n 'assets', content->'assets',\n 'stac_version', content->'stac_version',\n 'stac_extensions', content->'stac_extensions'\n ) AS content,\n content as collectionjson\nFROM collections;\n\n\nCREATE OR REPLACE FUNCTION collection_search_matched(\n IN _search jsonb DEFAULT '{}'::jsonb,\n OUT matched bigint\n) RETURNS bigint AS $$\nDECLARE\n _where text := stac_search_to_where(_search);\nBEGIN\n EXECUTE format(\n $query$\n SELECT\n count(*)\n FROM\n collections_asitems\n WHERE %s\n ;\n $query$,\n _where\n ) INTO matched;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION collection_search_rows(\n _search jsonb DEFAULT '{}'::jsonb\n) RETURNS SETOF jsonb AS $$\nDECLARE\n _where text := stac_search_to_where(_search);\n _limit int := coalesce((_search->>'limit')::int, 10);\n _fields jsonb := coalesce(_search->'fields', '{}'::jsonb);\n _orderby text;\n _offset int := COALESCE((_search->>'offset')::int, 0);\nBEGIN\n _orderby := sort_sqlorderby(\n jsonb_build_object(\n 'sortby',\n coalesce(\n _search->'sortby',\n '[{\"field\": \"id\", \"direction\": \"asc\"}]'::jsonb\n )\n )\n );\n RETURN QUERY EXECUTE format(\n $query$\n SELECT\n jsonb_fields(collectionjson, %L) as c\n FROM\n collections_asitems\n WHERE %s\n ORDER BY %s\n LIMIT %L\n OFFSET %L\n ;\n $query$,\n _fields,\n _where,\n _orderby,\n _limit,\n _offset\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION collection_search(\n _search jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\nDECLARE\n out_records jsonb;\n number_matched bigint := collection_search_matched(_search);\n number_returned bigint;\n _limit int := coalesce((_search->>'limit')::float::int, 10);\n _offset int := coalesce((_search->>'offset')::float::int, 0);\n links jsonb := '[]';\n ret jsonb;\n base_url text:= concat(rtrim(base_url(_search->'conf'),'/'), '/collections');\n prevoffset int;\n nextoffset int;\nBEGIN\n SELECT\n coalesce(jsonb_agg(c), '[]')\n INTO out_records\n FROM collection_search_rows(_search) c;\n\n number_returned := jsonb_array_length(out_records);\n RAISE DEBUG 'nm: %, nr: %, l:%, o:%', number_matched, number_returned, _limit, _offset;\n\n\n\n IF _limit <= number_matched AND number_matched > 0 THEN --need to have paging links\n nextoffset := least(_offset + _limit, number_matched - 1);\n prevoffset := greatest(_offset - _limit, 0);\n\n IF _offset > 0 THEN\n links := links || jsonb_build_object(\n 'rel', 'prev',\n 'type', 'application/json',\n 'method', 'GET' ,\n 'href', base_url,\n 'body', jsonb_build_object('offset', prevoffset),\n 'merge', TRUE\n );\n END IF;\n\n IF (_offset + _limit < number_matched) THEN\n links := links || jsonb_build_object(\n 'rel', 'next',\n 'type', 'application/json',\n 'method', 'GET' ,\n 'href', base_url,\n 'body', jsonb_build_object('offset', nextoffset),\n 'merge', TRUE\n );\n END IF;\n\n END IF;\n\n ret := jsonb_build_object(\n 'collections', out_records,\n 'numberMatched', number_matched,\n 'numberReturned', number_returned,\n 'links', links\n );\n RETURN ret;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION tileenvelope(zoom int, x int, y int) RETURNS geometry AS $$\nWITH t AS (\n SELECT\n 20037508.3427892 as merc_max,\n -20037508.3427892 as merc_min,\n (2 * 20037508.3427892) / (2 ^ zoom) as tile_size\n)\nSELECT st_makeenvelope(\n merc_min + (tile_size * x),\n merc_max - (tile_size * (y + 1)),\n merc_min + (tile_size * (x + 1)),\n merc_max - (tile_size * y),\n 3857\n) FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;DROP FUNCTION IF EXISTS mercgrid;\n\n\nCREATE OR REPLACE FUNCTION ftime() RETURNS interval as $$\nSELECT age(clock_timestamp(), transaction_timestamp());\n$$ LANGUAGE SQL;\nSET SEARCH_PATH to pgstac, public;\n\nDROP FUNCTION IF EXISTS geometrysearch;\nCREATE OR REPLACE FUNCTION geometrysearch(\n IN geom geometry,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE, -- Return as soon as the passed in geometry is full covered\n IN skipcovered boolean DEFAULT TRUE -- Skip any items that would show up completely under the previous items\n) RETURNS jsonb AS $$\nDECLARE\n search searches%ROWTYPE;\n curs refcursor;\n _where text;\n query text;\n iter_record items%ROWTYPE;\n out_records jsonb := '{}'::jsonb[];\n exit_flag boolean := FALSE;\n counter int := 1;\n scancounter int := 1;\n remaining_limit int := _scanlimit;\n tilearea float;\n unionedgeom geometry;\n clippedgeom geometry;\n unionedgeom_area float := 0;\n prev_area float := 0;\n excludes text[];\n includes text[];\n\nBEGIN\n DROP TABLE IF EXISTS pgstac_results;\n CREATE TEMP TABLE pgstac_results (content jsonb) ON COMMIT DROP;\n\n -- If the passed in geometry is not an area set exitwhenfull and skipcovered to false\n IF ST_GeometryType(geom) !~* 'polygon' THEN\n RAISE NOTICE 'GEOMETRY IS NOT AN AREA';\n skipcovered = FALSE;\n exitwhenfull = FALSE;\n END IF;\n\n -- If skipcovered is true then you will always want to exit when the passed in geometry is full\n IF skipcovered THEN\n exitwhenfull := TRUE;\n END IF;\n\n search := search_fromhash(queryhash);\n\n IF search IS NULL THEN\n RAISE EXCEPTION 'Search with Query Hash % Not Found', queryhash;\n END IF;\n\n tilearea := st_area(geom);\n _where := format('%s AND st_intersects(geometry, %L::geometry)', search._where, geom);\n\n\n FOR query IN SELECT * FROM partition_queries(_where, search.orderby) LOOP\n query := format('%s LIMIT %L', query, remaining_limit);\n RAISE NOTICE '%', query;\n OPEN curs FOR EXECUTE query;\n LOOP\n FETCH curs INTO iter_record;\n EXIT WHEN NOT FOUND;\n IF exitwhenfull OR skipcovered THEN -- If we are not using exitwhenfull or skipcovered, we do not need to do expensive geometry operations\n clippedgeom := st_intersection(geom, iter_record.geometry);\n\n IF unionedgeom IS NULL THEN\n unionedgeom := clippedgeom;\n ELSE\n unionedgeom := st_union(unionedgeom, clippedgeom);\n END IF;\n\n unionedgeom_area := st_area(unionedgeom);\n\n IF skipcovered AND prev_area = unionedgeom_area THEN\n scancounter := scancounter + 1;\n CONTINUE;\n END IF;\n\n prev_area := unionedgeom_area;\n\n RAISE NOTICE '% % % %', unionedgeom_area/tilearea, counter, scancounter, ftime();\n END IF;\n RAISE NOTICE '% %', iter_record, content_hydrate(iter_record, fields);\n INSERT INTO pgstac_results (content) VALUES (content_hydrate(iter_record, fields));\n\n IF counter >= _limit\n OR scancounter > _scanlimit\n OR ftime() > _timelimit\n OR (exitwhenfull AND unionedgeom_area >= tilearea)\n THEN\n exit_flag := TRUE;\n EXIT;\n END IF;\n counter := counter + 1;\n scancounter := scancounter + 1;\n\n END LOOP;\n CLOSE curs;\n EXIT WHEN exit_flag;\n remaining_limit := _scanlimit - scancounter;\n END LOOP;\n\n SELECT jsonb_agg(content) INTO out_records FROM pgstac_results WHERE content IS NOT NULL;\n\n RETURN jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb)\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDROP FUNCTION IF EXISTS geojsonsearch;\nCREATE OR REPLACE FUNCTION geojsonsearch(\n IN geojson jsonb,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_geomfromgeojson(geojson),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS xyzsearch;\nCREATE OR REPLACE FUNCTION xyzsearch(\n IN _x int,\n IN _y int,\n IN _z int,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_transform(tileenvelope(_z, _x, _y), 4326),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE PROCEDURE analyze_items() AS $$\nDECLARE\n q text;\n timeout_ts timestamptz;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n SELECT format('ANALYZE (VERBOSE, SKIP_LOCKED) %I;', relname) INTO q\n FROM pg_stat_user_tables\n WHERE relname like '_item%' AND (n_mod_since_analyze>0 OR last_analyze IS NULL) LIMIT 1;\n IF NOT FOUND THEN\n EXIT;\n END IF;\n RAISE NOTICE '%', q;\n EXECUTE q;\n COMMIT;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE PROCEDURE validate_constraints() AS $$\nDECLARE\n q text;\nBEGIN\n FOR q IN\n SELECT\n FORMAT(\n 'ALTER TABLE %I.%I VALIDATE CONSTRAINT %I;',\n nsp.nspname,\n cls.relname,\n con.conname\n )\n\n FROM pg_constraint AS con\n JOIN pg_class AS cls\n ON con.conrelid = cls.oid\n JOIN pg_namespace AS nsp\n ON cls.relnamespace = nsp.oid\n WHERE convalidated = FALSE AND contype in ('c','f')\n AND nsp.nspname = 'pgstac'\n LOOP\n RAISE NOTICE '%', q;\n PERFORM run_or_queue(q);\n COMMIT;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION collection_extent(_collection text, runupdate boolean default false) RETURNS jsonb AS $$\nDECLARE\n geom_extent geometry;\n mind timestamptz;\n maxd timestamptz;\n extent jsonb;\nBEGIN\n IF runupdate THEN\n PERFORM update_partition_stats_q(partition)\n FROM partitions_view WHERE collection=_collection;\n END IF;\n SELECT\n min(lower(dtrange)),\n max(upper(edtrange)),\n st_extent(spatial)\n INTO\n mind,\n maxd,\n geom_extent\n FROM partitions_view\n WHERE collection=_collection;\n\n IF geom_extent IS NOT NULL AND mind IS NOT NULL AND maxd IS NOT NULL THEN\n extent := jsonb_build_object(\n 'spatial', jsonb_build_object(\n 'bbox', to_jsonb(array[array[st_xmin(geom_extent), st_ymin(geom_extent), st_xmax(geom_extent), st_ymax(geom_extent)]])\n ),\n 'temporal', jsonb_build_object(\n 'interval', to_jsonb(array[array[mind, maxd]])\n )\n );\n RETURN extent;\n END IF;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('id', '{\"title\": \"Item ID\",\"description\": \"Item identifier\",\"$ref\": \"https://schemas.stacspec.org/v1.0.0/item-spec/json-schema/item.json#/definitions/core/allOf/2/properties/id\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('geometry', '{\"title\": \"Item Geometry\",\"description\": \"Item Geometry\",\"$ref\": \"https://geojson.org/schema/Feature.json\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('datetime','{\"description\": \"Datetime\",\"type\": \"string\",\"title\": \"Acquired\",\"format\": \"date-time\",\"pattern\": \"(\\\\+00:00|Z)$\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDELETE FROM queryables a USING queryables b\n WHERE a.name = b.name AND a.collection_ids IS NOT DISTINCT FROM b.collection_ids AND a.id > b.id;\n\n\nINSERT INTO pgstac_settings (name, value) VALUES\n ('context', 'off'),\n ('context_estimated_count', '100000'),\n ('context_estimated_cost', '100000'),\n ('context_stats_ttl', '1 day'),\n ('default_filter_lang', 'cql2-json'),\n ('additional_properties', 'true'),\n ('use_queue', 'false'),\n ('queue_timeout', '10 minutes'),\n ('update_collection_extent', 'false'),\n ('format_cache', 'false'),\n ('readonly', 'false')\nON CONFLICT DO NOTHING\n;\n\n\nINSERT INTO cql2_ops (op, template, types) VALUES\n ('eq', '%s = %s', NULL),\n ('neq', '%s != %s', NULL),\n ('ne', '%s != %s', NULL),\n ('!=', '%s != %s', NULL),\n ('<>', '%s != %s', NULL),\n ('lt', '%s < %s', NULL),\n ('lte', '%s <= %s', NULL),\n ('gt', '%s > %s', NULL),\n ('gte', '%s >= %s', NULL),\n ('le', '%s <= %s', NULL),\n ('ge', '%s >= %s', NULL),\n ('=', '%s = %s', NULL),\n ('<', '%s < %s', NULL),\n ('<=', '%s <= %s', NULL),\n ('>', '%s > %s', NULL),\n ('>=', '%s >= %s', NULL),\n ('like', '%s LIKE %s', NULL),\n ('ilike', '%s ILIKE %s', NULL),\n ('+', '%s + %s', NULL),\n ('-', '%s - %s', NULL),\n ('*', '%s * %s', NULL),\n ('/', '%s / %s', NULL),\n ('not', 'NOT (%s)', NULL),\n ('between', '%s BETWEEN %s AND %s', NULL),\n ('isnull', '%s IS NULL', NULL),\n ('upper', 'upper(%s)', NULL),\n ('lower', 'lower(%s)', NULL),\n ('casei', 'upper(%s)', NULL),\n ('accenti', 'unaccent(%s)', NULL)\nON CONFLICT (op) DO UPDATE\n SET\n template = EXCLUDED.template\n;\n\n\nALTER FUNCTION to_text COST 5000;\nALTER FUNCTION to_float COST 5000;\nALTER FUNCTION to_int COST 5000;\nALTER FUNCTION to_tstz COST 5000;\nALTER FUNCTION to_text_array COST 5000;\n\nALTER FUNCTION update_partition_stats SECURITY DEFINER;\nALTER FUNCTION partition_after_triggerfunc SECURITY DEFINER;\nALTER FUNCTION drop_table_constraints SECURITY DEFINER;\nALTER FUNCTION create_table_constraints SECURITY DEFINER;\nALTER FUNCTION check_partition SECURITY DEFINER;\nALTER FUNCTION repartition SECURITY DEFINER;\nALTER FUNCTION where_stats SECURITY DEFINER;\nALTER FUNCTION search_query SECURITY DEFINER;\nALTER FUNCTION format_item SECURITY DEFINER;\nALTER FUNCTION maintain_index SECURITY DEFINER;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nGRANT ALL ON SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac to pgstac_admin;\n\n-- pgstac_read role limited to using function apis\nGRANT EXECUTE ON FUNCTION search TO pgstac_read;\nGRANT EXECUTE ON FUNCTION search_query TO pgstac_read;\nGRANT EXECUTE ON FUNCTION item_by_id TO pgstac_read;\nGRANT EXECUTE ON FUNCTION get_item TO pgstac_read;\nGRANT SELECT ON ALL TABLES IN SCHEMA pgstac TO pgstac_read;\n\n\nGRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_ingest;\nGRANT USAGE ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_ingest;\n\nREVOKE ALL PRIVILEGES ON PROCEDURE run_queued_queries FROM public;\nGRANT ALL ON PROCEDURE run_queued_queries TO pgstac_admin;\n\nREVOKE ALL PRIVILEGES ON FUNCTION run_queued_queries_intransaction FROM public;\nGRANT ALL ON FUNCTION run_queued_queries_intransaction TO pgstac_admin;\n\nRESET ROLE;\n\nSET ROLE pgstac_ingest;\nSELECT update_partition_stats_q(partition) FROM partitions_view;\nSELECT set_version('0.9.5');\n" }, { "path": "src/pgstac/migrations/pgstac.0.9.6-0.9.7.sql", "content": "SET client_min_messages TO WARNING;\nSET SEARCH_PATH to pgstac, public;\nRESET ROLE;\nDO $$\nDECLARE\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='postgis') THEN\n CREATE EXTENSION IF NOT EXISTS postgis;\n END IF;\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='btree_gist') THEN\n CREATE EXTENSION IF NOT EXISTS btree_gist;\n END IF;\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='unaccent') THEN\n CREATE EXTENSION IF NOT EXISTS unaccent;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_admin;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_read;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_ingest;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n\nGRANT pgstac_admin TO current_user;\n\n-- Function to make sure pgstac_admin is the owner of items\nCREATE OR REPLACE FUNCTION pgstac_admin_owns() RETURNS VOID AS $$\nDECLARE\n f RECORD;\nBEGIN\n FOR f IN (\n SELECT\n concat(\n oid::regproc::text,\n '(',\n coalesce(pg_get_function_identity_arguments(oid),''),\n ')'\n ) AS name,\n CASE prokind WHEN 'f' THEN 'FUNCTION' WHEN 'p' THEN 'PROCEDURE' WHEN 'a' THEN 'AGGREGATE' END as typ\n FROM pg_proc\n WHERE\n pronamespace=to_regnamespace('pgstac')\n AND proowner != to_regrole('pgstac_admin')\n AND proname NOT LIKE 'pg_stat%'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n FOR f IN (\n SELECT\n oid::regclass::text as name,\n CASE relkind\n WHEN 'i' THEN 'INDEX'\n WHEN 'I' THEN 'INDEX'\n WHEN 'p' THEN 'TABLE'\n WHEN 'r' THEN 'TABLE'\n WHEN 'v' THEN 'VIEW'\n WHEN 'S' THEN 'SEQUENCE'\n ELSE NULL\n END as typ\n FROM pg_class\n WHERE relnamespace=to_regnamespace('pgstac') and relowner != to_regrole('pgstac_admin') AND relkind IN ('r','p','v','S') AND relname NOT LIKE 'pg_stat'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\nSELECT pgstac_admin_owns();\n\nCREATE SCHEMA IF NOT EXISTS pgstac AUTHORIZATION pgstac_admin;\n\nGRANT ALL ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_admin;\n\nALTER ROLE pgstac_admin SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_read SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_ingest SET SEARCH_PATH TO pgstac, public;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\n\nGRANT pgstac_read TO pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\n\nSET ROLE pgstac_admin;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\nRESET ROLE;\n\nSET ROLE pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\nRESET ROLE;\n\nSET SEARCH_PATH TO pgstac, public;\nSET ROLE pgstac_admin;\n\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS analyze_items;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS validate_constraints;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n-- Install these idempotently as migrations do not put them before trying to modify the collections table\n\n\nCREATE OR REPLACE FUNCTION collection_geom(content jsonb)\nRETURNS geometry AS $$\n WITH box AS (SELECT content->'extent'->'spatial'->'bbox'->0 as box)\n SELECT\n st_makeenvelope(\n (box->>0)::float,\n (box->>1)::float,\n (box->>2)::float,\n (box->>3)::float,\n 4326\n )\n FROM box;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION collection_datetime(content jsonb)\nRETURNS timestamptz AS $$\n SELECT\n CASE\n WHEN\n (content->'extent'->'temporal'->'interval'->0->>0) IS NULL\n THEN '-infinity'::timestamptz\n ELSE\n (content->'extent'->'temporal'->'interval'->0->>0)::timestamptz\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION collection_enddatetime(content jsonb)\nRETURNS timestamptz AS $$\n SELECT\n CASE\n WHEN\n (content->'extent'->'temporal'->'interval'->0->>1) IS NULL\n THEN 'infinity'::timestamptz\n ELSE\n (content->'extent'->'temporal'->'interval'->0->>1)::timestamptz\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n-- BEGIN migra calculated SQL\nset check_function_bodies = off;\n\nCREATE OR REPLACE FUNCTION pgstac.q_to_tsquery(input text)\n RETURNS tsquery\n LANGUAGE plpgsql\nAS $function$\nDECLARE\n processed_text text;\n temp_text text;\n quote_array text[];\n placeholder text := '@QUOTE@';\nBEGIN\n -- Extract all quoted phrases and store in array\n quote_array := regexp_matches(input, '\"[^\"]*\"', 'g');\n\n -- Replace each quoted part with a unique placeholder if there are any quoted phrases\n IF array_length(quote_array, 1) IS NOT NULL THEN\n processed_text := input;\n FOR i IN array_lower(quote_array, 1) .. array_upper(quote_array, 1) LOOP\n processed_text := replace(processed_text, quote_array[i], placeholder || i || placeholder);\n END LOOP;\n ELSE\n processed_text := input;\n END IF;\n\n -- Replace non-quoted text using regular expressions\n\n -- , -> |\n processed_text := regexp_replace(processed_text, ',(?=(?:[^\"]*\"[^\"]*\")*[^\"]*$)', ' | ', 'g');\n\n -- and -> &\n processed_text := regexp_replace(processed_text, '\\s+AND\\s+', ' & ', 'gi');\n\n -- or -> |\n processed_text := regexp_replace(processed_text, '\\s+OR\\s+', ' | ', 'gi');\n\n -- + ->\n processed_text := regexp_replace(processed_text, '^\\s*\\+([a-zA-Z0-9_]+)', '\\1', 'g'); -- +term at start\n processed_text := regexp_replace(processed_text, '\\s*\\+([a-zA-Z0-9_]+)', ' & \\1', 'g'); -- +term elsewhere\n\n -- - -> !\n processed_text := regexp_replace(processed_text, '^\\s*\\-([a-zA-Z0-9_]+)', '! \\1', 'g'); -- -term at start\n processed_text := regexp_replace(processed_text, '\\s*\\-([a-zA-Z0-9_]+)', ' & ! \\1', 'g'); -- -term elsewhere\n -- Replace placeholders back with quoted phrases if there were any\n IF array_length(quote_array, 1) IS NOT NULL THEN\n FOR i IN array_lower(quote_array, 1) .. array_upper(quote_array, 1) LOOP\n processed_text := replace(processed_text, placeholder || i || placeholder, '''' || substring(quote_array[i] from 2 for length(quote_array[i]) - 2) || '''');\n END LOOP;\n END IF;\n\n -- Print processed_text to the console for debugging purposes\n RAISE NOTICE 'processed_text: %', processed_text;\n\n RETURN to_tsquery('english', processed_text);\nEND;\n$function$\n;\n\n\n-- END migra calculated SQL\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('id', '{\"title\": \"Item ID\",\"description\": \"Item identifier\",\"$ref\": \"https://schemas.stacspec.org/v1.0.0/item-spec/json-schema/item.json#/definitions/core/allOf/2/properties/id\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('geometry', '{\"title\": \"Item Geometry\",\"description\": \"Item Geometry\",\"$ref\": \"https://geojson.org/schema/Feature.json\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('datetime','{\"description\": \"Datetime\",\"type\": \"string\",\"title\": \"Acquired\",\"format\": \"date-time\",\"pattern\": \"(\\\\+00:00|Z)$\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDELETE FROM queryables a USING queryables b\n WHERE a.name = b.name AND a.collection_ids IS NOT DISTINCT FROM b.collection_ids AND a.id > b.id;\n\n\nINSERT INTO pgstac_settings (name, value) VALUES\n ('context', 'off'),\n ('context_estimated_count', '100000'),\n ('context_estimated_cost', '100000'),\n ('context_stats_ttl', '1 day'),\n ('default_filter_lang', 'cql2-json'),\n ('additional_properties', 'true'),\n ('use_queue', 'false'),\n ('queue_timeout', '10 minutes'),\n ('update_collection_extent', 'false'),\n ('format_cache', 'false'),\n ('readonly', 'false')\nON CONFLICT DO NOTHING\n;\n\n\nINSERT INTO cql2_ops (op, template, types) VALUES\n ('eq', '%s = %s', NULL),\n ('neq', '%s != %s', NULL),\n ('ne', '%s != %s', NULL),\n ('!=', '%s != %s', NULL),\n ('<>', '%s != %s', NULL),\n ('lt', '%s < %s', NULL),\n ('lte', '%s <= %s', NULL),\n ('gt', '%s > %s', NULL),\n ('gte', '%s >= %s', NULL),\n ('le', '%s <= %s', NULL),\n ('ge', '%s >= %s', NULL),\n ('=', '%s = %s', NULL),\n ('<', '%s < %s', NULL),\n ('<=', '%s <= %s', NULL),\n ('>', '%s > %s', NULL),\n ('>=', '%s >= %s', NULL),\n ('like', '%s LIKE %s', NULL),\n ('ilike', '%s ILIKE %s', NULL),\n ('+', '%s + %s', NULL),\n ('-', '%s - %s', NULL),\n ('*', '%s * %s', NULL),\n ('/', '%s / %s', NULL),\n ('not', 'NOT (%s)', NULL),\n ('between', '%s BETWEEN %s AND %s', NULL),\n ('isnull', '%s IS NULL', NULL),\n ('upper', 'upper(%s)', NULL),\n ('lower', 'lower(%s)', NULL),\n ('casei', 'upper(%s)', NULL),\n ('accenti', 'unaccent(%s)', NULL)\nON CONFLICT (op) DO UPDATE\n SET\n template = EXCLUDED.template\n;\n\n\nALTER FUNCTION to_text COST 5000;\nALTER FUNCTION to_float COST 5000;\nALTER FUNCTION to_int COST 5000;\nALTER FUNCTION to_tstz COST 5000;\nALTER FUNCTION to_text_array COST 5000;\n\nALTER FUNCTION update_partition_stats SECURITY DEFINER;\nALTER FUNCTION partition_after_triggerfunc SECURITY DEFINER;\nALTER FUNCTION drop_table_constraints SECURITY DEFINER;\nALTER FUNCTION create_table_constraints SECURITY DEFINER;\nALTER FUNCTION check_partition SECURITY DEFINER;\nALTER FUNCTION repartition SECURITY DEFINER;\nALTER FUNCTION where_stats SECURITY DEFINER;\nALTER FUNCTION search_query SECURITY DEFINER;\nALTER FUNCTION format_item SECURITY DEFINER;\nALTER FUNCTION maintain_index SECURITY DEFINER;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nGRANT ALL ON SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac to pgstac_admin;\n\n-- pgstac_read role limited to using function apis\nGRANT EXECUTE ON FUNCTION search TO pgstac_read;\nGRANT EXECUTE ON FUNCTION search_query TO pgstac_read;\nGRANT EXECUTE ON FUNCTION item_by_id TO pgstac_read;\nGRANT EXECUTE ON FUNCTION get_item TO pgstac_read;\nGRANT SELECT ON ALL TABLES IN SCHEMA pgstac TO pgstac_read;\n\n\nGRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_ingest;\nGRANT USAGE ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_ingest;\n\nREVOKE ALL PRIVILEGES ON PROCEDURE run_queued_queries FROM public;\nGRANT ALL ON PROCEDURE run_queued_queries TO pgstac_admin;\n\nREVOKE ALL PRIVILEGES ON FUNCTION run_queued_queries_intransaction FROM public;\nGRANT ALL ON FUNCTION run_queued_queries_intransaction TO pgstac_admin;\n\nRESET ROLE;\n\nSET ROLE pgstac_ingest;\nSELECT update_partition_stats_q(partition) FROM partitions_view;\nSELECT set_version('0.9.7');\n" }, { "path": "src/pgstac/migrations/pgstac.0.9.6.sql", "content": "RESET ROLE;\nDO $$\nDECLARE\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='postgis') THEN\n CREATE EXTENSION IF NOT EXISTS postgis;\n END IF;\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='btree_gist') THEN\n CREATE EXTENSION IF NOT EXISTS btree_gist;\n END IF;\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='unaccent') THEN\n CREATE EXTENSION IF NOT EXISTS unaccent;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_admin;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_read;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_ingest;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n\nGRANT pgstac_admin TO current_user;\n\n-- Function to make sure pgstac_admin is the owner of items\nCREATE OR REPLACE FUNCTION pgstac_admin_owns() RETURNS VOID AS $$\nDECLARE\n f RECORD;\nBEGIN\n FOR f IN (\n SELECT\n concat(\n oid::regproc::text,\n '(',\n coalesce(pg_get_function_identity_arguments(oid),''),\n ')'\n ) AS name,\n CASE prokind WHEN 'f' THEN 'FUNCTION' WHEN 'p' THEN 'PROCEDURE' WHEN 'a' THEN 'AGGREGATE' END as typ\n FROM pg_proc\n WHERE\n pronamespace=to_regnamespace('pgstac')\n AND proowner != to_regrole('pgstac_admin')\n AND proname NOT LIKE 'pg_stat%'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n FOR f IN (\n SELECT\n oid::regclass::text as name,\n CASE relkind\n WHEN 'i' THEN 'INDEX'\n WHEN 'I' THEN 'INDEX'\n WHEN 'p' THEN 'TABLE'\n WHEN 'r' THEN 'TABLE'\n WHEN 'v' THEN 'VIEW'\n WHEN 'S' THEN 'SEQUENCE'\n ELSE NULL\n END as typ\n FROM pg_class\n WHERE relnamespace=to_regnamespace('pgstac') and relowner != to_regrole('pgstac_admin') AND relkind IN ('r','p','v','S') AND relname NOT LIKE 'pg_stat'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\nSELECT pgstac_admin_owns();\n\nCREATE SCHEMA IF NOT EXISTS pgstac AUTHORIZATION pgstac_admin;\n\nGRANT ALL ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_admin;\n\nALTER ROLE pgstac_admin SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_read SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_ingest SET SEARCH_PATH TO pgstac, public;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\n\nGRANT pgstac_read TO pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\n\nSET ROLE pgstac_admin;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\nRESET ROLE;\n\nSET ROLE pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\nRESET ROLE;\n\nSET SEARCH_PATH TO pgstac, public;\nSET ROLE pgstac_admin;\n\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS analyze_items;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS validate_constraints;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n-- Install these idempotently as migrations do not put them before trying to modify the collections table\n\n\nCREATE OR REPLACE FUNCTION collection_geom(content jsonb)\nRETURNS geometry AS $$\n WITH box AS (SELECT content->'extent'->'spatial'->'bbox'->0 as box)\n SELECT\n st_makeenvelope(\n (box->>0)::float,\n (box->>1)::float,\n (box->>2)::float,\n (box->>3)::float,\n 4326\n )\n FROM box;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION collection_datetime(content jsonb)\nRETURNS timestamptz AS $$\n SELECT\n CASE\n WHEN\n (content->'extent'->'temporal'->'interval'->0->>0) IS NULL\n THEN '-infinity'::timestamptz\n ELSE\n (content->'extent'->'temporal'->'interval'->0->>0)::timestamptz\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION collection_enddatetime(content jsonb)\nRETURNS timestamptz AS $$\n SELECT\n CASE\n WHEN\n (content->'extent'->'temporal'->'interval'->0->>1) IS NULL\n THEN 'infinity'::timestamptz\n ELSE\n (content->'extent'->'temporal'->'interval'->0->>1)::timestamptz\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE TABLE IF NOT EXISTS migrations (\n version text PRIMARY KEY,\n datetime timestamptz DEFAULT clock_timestamp() NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION get_version() RETURNS text AS $$\n SELECT version FROM pgstac.migrations ORDER BY datetime DESC, version DESC LIMIT 1;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION set_version(text) RETURNS text AS $$\n INSERT INTO pgstac.migrations (version) VALUES ($1)\n ON CONFLICT DO NOTHING\n RETURNING version;\n$$ LANGUAGE SQL;\n\n\nCREATE TABLE IF NOT EXISTS pgstac_settings (\n name text PRIMARY KEY,\n value text NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION table_empty(text) RETURNS boolean AS $$\nDECLARE\n retval boolean;\nBEGIN\n EXECUTE format($q$\n SELECT NOT EXISTS (SELECT 1 FROM %I LIMIT 1)\n $q$,\n $1\n ) INTO retval;\n RETURN retval;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION get_setting(IN _setting text, IN conf jsonb DEFAULT NULL) RETURNS text AS $$\nSELECT COALESCE(\n nullif(conf->>_setting, ''),\n nullif(current_setting(concat('pgstac.',_setting), TRUE),''),\n nullif((SELECT value FROM pgstac.pgstac_settings WHERE name=_setting),'')\n);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_setting_bool(IN _setting text, IN conf jsonb DEFAULT NULL) RETURNS boolean AS $$\nSELECT COALESCE(\n nullif(conf->>_setting, ''),\n nullif(current_setting(concat('pgstac.',_setting), TRUE),''),\n nullif((SELECT value FROM pgstac.pgstac_settings WHERE name=_setting),''),\n 'FALSE'\n)::boolean;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION base_url(conf jsonb DEFAULT NULL) RETURNS text AS $$\n SELECT COALESCE(pgstac.get_setting('base_url', conf), '.');\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION additional_properties() RETURNS boolean AS $$\n SELECT pgstac.get_setting_bool('additional_properties');\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION readonly(conf jsonb DEFAULT NULL) RETURNS boolean AS $$\n SELECT pgstac.get_setting_bool('readonly', conf);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context(conf jsonb DEFAULT NULL) RETURNS text AS $$\n SELECT pgstac.get_setting('context', conf);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context_estimated_count(conf jsonb DEFAULT NULL) RETURNS int AS $$\n SELECT pgstac.get_setting('context_estimated_count', conf)::int;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_estimated_cost();\nCREATE OR REPLACE FUNCTION context_estimated_cost(conf jsonb DEFAULT NULL) RETURNS float AS $$\n SELECT pgstac.get_setting('context_estimated_cost', conf)::float;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_stats_ttl();\nCREATE OR REPLACE FUNCTION context_stats_ttl(conf jsonb DEFAULT NULL) RETURNS interval AS $$\n SELECT pgstac.get_setting('context_stats_ttl', conf)::interval;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION t2s(text) RETURNS text AS $$\n SELECT extract(epoch FROM $1::interval)::text || ' s';\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE STRICT;\n\nCREATE OR REPLACE FUNCTION age_ms(a timestamptz, b timestamptz DEFAULT clock_timestamp()) RETURNS float AS $$\n SELECT abs(extract(epoch from age(a,b)) * 1000);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION queue_timeout() RETURNS interval AS $$\n SELECT t2s(coalesce(\n get_setting('queue_timeout'),\n '1h'\n ))::interval;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION notice(VARIADIC text[]) RETURNS boolean AS $$\nDECLARE\ndebug boolean := current_setting('pgstac.debug', true);\nBEGIN\n IF debug THEN\n RAISE NOTICE 'NOTICE FROM FUNC: % >>>>> %', concat_ws(' | ', $1), clock_timestamp();\n RETURN TRUE;\n END IF;\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION empty_arr(ANYARRAY) RETURNS BOOLEAN AS $$\nSELECT CASE\n WHEN $1 IS NULL THEN TRUE\n WHEN cardinality($1)<1 THEN TRUE\nELSE FALSE\nEND;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_intersection(_a ANYARRAY, _b ANYARRAY) RETURNS ANYARRAY AS $$\n SELECT ARRAY ( SELECT unnest(_a) INTERSECT SELECT UNNEST(_b) );\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION array_map_ident(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_ident(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_map_literal(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_literal(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION array_reverse(anyarray) RETURNS anyarray AS $$\nSELECT ARRAY(\n SELECT $1[i]\n FROM generate_subscripts($1,1) AS s(i)\n ORDER BY i DESC\n);\n$$ LANGUAGE SQL STRICT IMMUTABLE;\n\nDROP TABLE IF EXISTS query_queue;\nCREATE TABLE query_queue (\n query text PRIMARY KEY,\n added timestamptz DEFAULT now()\n);\n\nDROP TABLE IF EXISTS query_queue_history;\nCREATE TABLE query_queue_history(\n query text,\n added timestamptz NOT NULL,\n finished timestamptz NOT NULL DEFAULT now(),\n error text\n);\n\nCREATE OR REPLACE PROCEDURE run_queued_queries() AS $$\nDECLARE\n qitem query_queue%ROWTYPE;\n timeout_ts timestamptz;\n error text;\n cnt int := 0;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n DELETE FROM query_queue WHERE query = (SELECT query FROM query_queue ORDER BY added DESC LIMIT 1 FOR UPDATE SKIP LOCKED) RETURNING * INTO qitem;\n IF NOT FOUND THEN\n EXIT;\n END IF;\n cnt := cnt + 1;\n BEGIN\n RAISE NOTICE 'RUNNING QUERY: %', qitem.query;\n EXECUTE qitem.query;\n EXCEPTION WHEN others THEN\n error := format('%s | %s', SQLERRM, SQLSTATE);\n END;\n INSERT INTO query_queue_history (query, added, finished, error)\n VALUES (qitem.query, qitem.added, clock_timestamp(), error);\n COMMIT;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION run_queued_queries_intransaction() RETURNS int AS $$\nDECLARE\n qitem query_queue%ROWTYPE;\n timeout_ts timestamptz;\n error text;\n cnt int := 0;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n DELETE FROM query_queue WHERE query = (SELECT query FROM query_queue ORDER BY added DESC LIMIT 1 FOR UPDATE SKIP LOCKED) RETURNING * INTO qitem;\n IF NOT FOUND THEN\n RETURN cnt;\n END IF;\n cnt := cnt + 1;\n BEGIN\n qitem.query := regexp_replace(qitem.query, 'CONCURRENTLY', '');\n RAISE NOTICE 'RUNNING QUERY: %', qitem.query;\n\n EXECUTE qitem.query;\n EXCEPTION WHEN others THEN\n error := format('%s | %s', SQLERRM, SQLSTATE);\n RAISE WARNING '%', error;\n END;\n INSERT INTO query_queue_history (query, added, finished, error)\n VALUES (qitem.query, qitem.added, clock_timestamp(), error);\n END LOOP;\n RETURN cnt;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION run_or_queue(query text) RETURNS VOID AS $$\nDECLARE\n use_queue text := COALESCE(get_setting('use_queue'), 'FALSE')::boolean;\nBEGIN\n IF get_setting_bool('debug') THEN\n RAISE NOTICE '%', query;\n END IF;\n IF use_queue THEN\n INSERT INTO query_queue (query) VALUES (query) ON CONFLICT DO NOTHING;\n ELSE\n EXECUTE query;\n END IF;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n\nDROP FUNCTION IF EXISTS check_pgstac_settings;\nCREATE OR REPLACE FUNCTION check_pgstac_settings(_sysmem text DEFAULT NULL) RETURNS VOID AS $$\nDECLARE\n settingval text;\n sysmem bigint := pg_size_bytes(_sysmem);\n effective_cache_size bigint := pg_size_bytes(current_setting('effective_cache_size', TRUE));\n shared_buffers bigint := pg_size_bytes(current_setting('shared_buffers', TRUE));\n work_mem bigint := pg_size_bytes(current_setting('work_mem', TRUE));\n max_connections int := current_setting('max_connections', TRUE);\n maintenance_work_mem bigint := pg_size_bytes(current_setting('maintenance_work_mem', TRUE));\n seq_page_cost float := current_setting('seq_page_cost', TRUE);\n random_page_cost float := current_setting('random_page_cost', TRUE);\n temp_buffers bigint := pg_size_bytes(current_setting('temp_buffers', TRUE));\n r record;\nBEGIN\n IF _sysmem IS NULL THEN\n RAISE NOTICE 'Call function with the size of your system memory `SELECT check_pgstac_settings(''4GB'')` to get pg system setting recommendations.';\n ELSE\n IF effective_cache_size < (sysmem * 0.5) THEN\n RAISE WARNING 'effective_cache_size of % is set low for a system with %. Recomended value between % and %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.5), pg_size_pretty(sysmem * 0.75);\n ELSIF effective_cache_size > (sysmem * 0.75) THEN\n RAISE WARNING 'effective_cache_size of % is set high for a system with %. Recomended value between % and %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.5), pg_size_pretty(sysmem * 0.75);\n ELSE\n RAISE NOTICE 'effective_cache_size of % is set appropriately for a system with %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem);\n END IF;\n\n IF shared_buffers < (sysmem * 0.2) THEN\n RAISE WARNING 'shared_buffers of % is set low for a system with %. Recomended value between % and %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.2), pg_size_pretty(sysmem * 0.3);\n ELSIF shared_buffers > (sysmem * 0.3) THEN\n RAISE WARNING 'shared_buffers of % is set high for a system with %. Recomended value between % and %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.2), pg_size_pretty(sysmem * 0.3);\n ELSE\n RAISE NOTICE 'shared_buffers of % is set appropriately for a system with %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem);\n END IF;\n shared_buffers = sysmem * 0.3;\n IF maintenance_work_mem < (sysmem * 0.2) THEN\n RAISE WARNING 'maintenance_work_mem of % is set low for shared_buffers of %. Recomended value between % and %', pg_size_pretty(maintenance_work_mem), pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers * 0.2), pg_size_pretty(shared_buffers * 0.3);\n ELSIF maintenance_work_mem > (shared_buffers * 0.3) THEN\n RAISE WARNING 'maintenance_work_mem of % is set high for shared_buffers of %. Recomended value between % and %', pg_size_pretty(maintenance_work_mem), pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers * 0.2), pg_size_pretty(shared_buffers * 0.3);\n ELSE\n RAISE NOTICE 'maintenance_work_mem of % is set appropriately for shared_buffers of %', pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers);\n END IF;\n\n IF work_mem * max_connections > shared_buffers THEN\n RAISE WARNING 'work_mem setting of % is set high for % max_connections please reduce work_mem to % or decrease max_connections to %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers/max_connections), floor(shared_buffers/work_mem);\n ELSIF work_mem * max_connections < (shared_buffers * 0.75) THEN\n RAISE WARNING 'work_mem setting of % is set low for % max_connections you may consider raising work_mem to % or increasing max_connections to %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers/max_connections), floor(shared_buffers/work_mem);\n ELSE\n RAISE NOTICE 'work_mem setting of % and max_connections of % are adequate for shared_buffers of %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers);\n END IF;\n\n IF random_page_cost / seq_page_cost != 1.1 THEN\n RAISE WARNING 'random_page_cost (%) /seq_page_cost (%) should be set to 1.1 for SSD. Change random_page_cost to %', random_page_cost, seq_page_cost, 1.1 * seq_page_cost;\n ELSE\n RAISE NOTICE 'random_page_cost and seq_page_cost set appropriately for SSD';\n END IF;\n\n IF temp_buffers < greatest(pg_size_bytes('128MB'),(maintenance_work_mem / 2)) THEN\n RAISE WARNING 'pgstac makes heavy use of temp tables, consider raising temp_buffers from % to %', pg_size_pretty(temp_buffers), greatest('128MB', pg_size_pretty((shared_buffers / 16)));\n END IF;\n END IF;\n\n RAISE NOTICE 'VALUES FOR PGSTAC VARIABLES';\n RAISE NOTICE 'These can be set either as GUC system variables or by setting in the pgstac_settings table.';\n\n FOR r IN SELECT name, get_setting(name) as setting, CASE WHEN current_setting(concat('pgstac.',name), TRUE) IS NOT NULL THEN concat('pgstac.',name, ' GUC') WHEN value IS NOT NULL THEN 'pgstac_settings table' ELSE 'Not Set' END as loc FROM pgstac_settings LOOP\n RAISE NOTICE '% is set to % from the %', r.name, r.setting, r.loc;\n END LOOP;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pg_cron';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider intalling pg_cron which can be used to automate tasks';\n ELSE\n RAISE NOTICE 'pg_cron % is installed', settingval;\n END IF;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pgstattuple';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider installing the pgstattuple extension which can be used to help maintain tables and indexes.';\n ELSE\n RAISE NOTICE 'pgstattuple % is installed', settingval;\n END IF;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pg_stat_statements';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider installing the pg_stat_statements extension which is very helpful for tracking the types of queries on the system';\n ELSE\n RAISE NOTICE 'pg_stat_statements % is installed', settingval;\n IF current_setting('pg_stat_statements.track_statements', TRUE) IS DISTINCT FROM 'all' THEN\n RAISE WARNING 'SET pg_stat_statements.track_statements TO ''all''; --In order to track statements within functions.';\n END IF;\n END IF;\n\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac, public SET CLIENT_MIN_MESSAGES TO NOTICE;\nCREATE OR REPLACE FUNCTION to_int(jsonb) RETURNS int AS $$\n SELECT floor(($1->>0)::float)::int;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION to_float(jsonb) RETURNS float AS $$\n SELECT ($1->>0)::float;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION to_tstz(jsonb) RETURNS timestamptz AS $$\n SELECT ($1->>0)::timestamptz;\n$$ LANGUAGE SQL IMMUTABLE STRICT SET TIME ZONE 'UTC' COST 5000 PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION to_text(jsonb) RETURNS text AS $$\n SELECT CASE WHEN jsonb_typeof($1) IN ('array','object') THEN $1::text ELSE $1->>0 END;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION to_text_array(jsonb) RETURNS text[] AS $$\n SELECT\n CASE jsonb_typeof($1)\n WHEN 'array' THEN ARRAY(SELECT jsonb_array_elements_text($1))\n ELSE ARRAY[$1->>0]\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION bbox_geom(_bbox jsonb) RETURNS geometry AS $$\nSELECT CASE jsonb_array_length(_bbox)\n WHEN 4 THEN\n ST_SetSRID(ST_MakeEnvelope(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float,\n (_bbox->>3)::float\n ),4326)\n WHEN 6 THEN\n ST_SetSRID(ST_3DMakeBox(\n ST_MakePoint(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float\n ),\n ST_MakePoint(\n (_bbox->>3)::float,\n (_bbox->>4)::float,\n (_bbox->>5)::float\n )\n ),4326)\n ELSE null END;\n;\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION geom_bbox(_geom geometry) RETURNS jsonb AS $$\n SELECT jsonb_build_array(\n st_xmin(_geom),\n st_ymin(_geom),\n st_xmax(_geom),\n st_ymax(_geom)\n );\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION flip_jsonb_array(j jsonb) RETURNS jsonb AS $$\n SELECT jsonb_agg(value) FROM (SELECT value FROM jsonb_array_elements(j) WITH ORDINALITY ORDER BY ordinality DESC) as t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION explode_dotpaths(j jsonb) RETURNS SETOF text[] AS $$\n SELECT string_to_array(p, '.') as e FROM jsonb_array_elements_text(j) p;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION explode_dotpaths_recurse(IN j jsonb) RETURNS SETOF text[] AS $$\n WITH RECURSIVE t AS (\n SELECT e FROM explode_dotpaths(j) e\n UNION ALL\n SELECT e[1:cardinality(e)-1]\n FROM t\n WHERE cardinality(e)>1\n ) SELECT e FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION jsonb_set_nested(j jsonb, path text[], val jsonb) RETURNS jsonb AS $$\nDECLARE\nBEGIN\n IF cardinality(path) > 1 THEN\n FOR i IN 1..(cardinality(path)-1) LOOP\n IF j #> path[:i] IS NULL THEN\n j := jsonb_set_lax(j, path[:i], '{}', TRUE);\n END IF;\n END LOOP;\n END IF;\n RETURN jsonb_set_lax(j, path, val, true);\n\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\n\n\nCREATE OR REPLACE FUNCTION jsonb_include(j jsonb, f jsonb) RETURNS jsonb AS $$\nDECLARE\n includes jsonb := f-> 'include';\n outj jsonb := '{}'::jsonb;\n path text[];\nBEGIN\n IF\n includes IS NULL\n OR jsonb_array_length(includes) = 0\n THEN\n RETURN j;\n ELSE\n includes := includes || '[\"id\",\"collection\"]'::jsonb;\n FOR path IN SELECT explode_dotpaths(includes) LOOP\n outj := jsonb_set_nested(outj, path, j #> path);\n END LOOP;\n END IF;\n RETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_exclude(j jsonb, f jsonb) RETURNS jsonb AS $$\nDECLARE\n excludes jsonb := f-> 'exclude';\n outj jsonb := j;\n path text[];\nBEGIN\n IF\n excludes IS NULL\n OR jsonb_array_length(excludes) = 0\n THEN\n RETURN j;\n ELSE\n FOR path IN SELECT explode_dotpaths(excludes) LOOP\n outj := outj #- path;\n END LOOP;\n END IF;\n RETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_fields(j jsonb, f jsonb DEFAULT '{\"fields\":[]}') RETURNS jsonb AS $$\n SELECT jsonb_exclude(jsonb_include(j, f), f);\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION merge_jsonb(_a jsonb, _b jsonb) RETURNS jsonb AS $$\n SELECT\n CASE\n WHEN _a = '\"𒍟※\"'::jsonb THEN NULL\n WHEN _a IS NULL OR jsonb_typeof(_a) = 'null' THEN _b\n WHEN jsonb_typeof(_a) = 'object' AND jsonb_typeof(_b) = 'object' THEN\n (\n SELECT\n jsonb_strip_nulls(\n jsonb_object_agg(\n key,\n merge_jsonb(a.value, b.value)\n )\n )\n FROM\n jsonb_each(coalesce(_a,'{}'::jsonb)) as a\n FULL JOIN\n jsonb_each(coalesce(_b,'{}'::jsonb)) as b\n USING (key)\n )\n WHEN\n jsonb_typeof(_a) = 'array'\n AND jsonb_typeof(_b) = 'array'\n AND jsonb_array_length(_a) = jsonb_array_length(_b)\n THEN\n (\n SELECT jsonb_agg(m) FROM\n ( SELECT\n merge_jsonb(\n jsonb_array_elements(_a),\n jsonb_array_elements(_b)\n ) as m\n ) as l\n )\n ELSE _a\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION strip_jsonb(_a jsonb, _b jsonb) RETURNS jsonb AS $$\n SELECT\n CASE\n\n WHEN (_a IS NULL OR jsonb_typeof(_a) = 'null') AND _b IS NOT NULL AND jsonb_typeof(_b) != 'null' THEN '\"𒍟※\"'::jsonb\n WHEN _b IS NULL OR jsonb_typeof(_a) = 'null' THEN _a\n WHEN _a = _b AND jsonb_typeof(_a) = 'object' THEN '{}'::jsonb\n WHEN _a = _b THEN NULL\n WHEN jsonb_typeof(_a) = 'object' AND jsonb_typeof(_b) = 'object' THEN\n (\n SELECT\n jsonb_strip_nulls(\n jsonb_object_agg(\n key,\n strip_jsonb(a.value, b.value)\n )\n )\n FROM\n jsonb_each(_a) as a\n FULL JOIN\n jsonb_each(_b) as b\n USING (key)\n )\n WHEN\n jsonb_typeof(_a) = 'array'\n AND jsonb_typeof(_b) = 'array'\n AND jsonb_array_length(_a) = jsonb_array_length(_b)\n THEN\n (\n SELECT jsonb_agg(m) FROM\n ( SELECT\n strip_jsonb(\n jsonb_array_elements(_a),\n jsonb_array_elements(_b)\n ) as m\n ) as l\n )\n ELSE _a\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION nullif_jsonbnullempty(j jsonb) RETURNS jsonb AS $$\n SELECT nullif(nullif(nullif(j,'null'::jsonb),'{}'::jsonb),'[]'::jsonb);\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION jsonb_array_unique(j jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(jsonb_agg(DISTINCT a)) v FROM jsonb_array_elements(j) a;\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_concat_ignorenull(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT coalesce(a,'[]'::jsonb) || coalesce(b,'[]'::jsonb);\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_least(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(least(nullif_jsonbnullempty(a), nullif_jsonbnullempty(b)));\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_greatest(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(greatest(a, b));\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION first_notnull_sfunc(anyelement, anyelement) RETURNS anyelement AS $$\n SELECT COALESCE($1,$2);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE AGGREGATE first_notnull(anyelement)(\n SFUNC = first_notnull_sfunc,\n STYPE = anyelement\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_array_unique_merge(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_concat_ignorenull,\n FINALFUNC = jsonb_array_unique\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_min(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_least\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_max(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_greatest\n);\n/* looks for a geometry in a stac item first from geometry and falling back to bbox */\nCREATE OR REPLACE FUNCTION stac_geom(value jsonb) RETURNS geometry AS $$\nSELECT\n CASE\n WHEN value ? 'intersects' THEN\n ST_GeomFromGeoJSON(value->>'intersects')\n WHEN value ? 'geometry' THEN\n ST_GeomFromGeoJSON(value->>'geometry')\n WHEN value ? 'bbox' THEN\n pgstac.bbox_geom(value->'bbox')\n ELSE NULL\n END as geometry\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION stac_daterange(\n value jsonb\n) RETURNS tstzrange AS $$\nDECLARE\n props jsonb := value;\n dt timestamptz;\n edt timestamptz;\nBEGIN\n IF props ? 'properties' THEN\n props := props->'properties';\n END IF;\n IF\n props ? 'start_datetime'\n AND props->>'start_datetime' IS NOT NULL\n AND props ? 'end_datetime'\n AND props->>'end_datetime' IS NOT NULL\n THEN\n dt := props->>'start_datetime';\n edt := props->>'end_datetime';\n IF dt > edt THEN\n RAISE EXCEPTION 'start_datetime must be < end_datetime';\n END IF;\n ELSE\n dt := props->>'datetime';\n edt := props->>'datetime';\n END IF;\n IF dt is NULL OR edt IS NULL THEN\n RAISE NOTICE 'DT: %, EDT: %', dt, edt;\n RAISE EXCEPTION 'Either datetime (%) or both start_datetime (%) and end_datetime (%) must be set.', props->>'datetime',props->>'start_datetime',props->>'end_datetime';\n END IF;\n RETURN tstzrange(dt, edt, '[]');\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_datetime(value jsonb) RETURNS timestamptz AS $$\n SELECT lower(stac_daterange(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_end_datetime(value jsonb) RETURNS timestamptz AS $$\n SELECT upper(stac_daterange(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE TABLE IF NOT EXISTS stac_extensions(\n url text PRIMARY KEY,\n content jsonb\n);\nCREATE OR REPLACE FUNCTION collection_base_item(content jsonb) RETURNS jsonb AS $$\n SELECT jsonb_build_object(\n 'type', 'Feature',\n 'stac_version', content->'stac_version',\n 'assets', content->'item_assets',\n 'collection', content->'id'\n );\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE TABLE IF NOT EXISTS collections (\n key bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,\n id text GENERATED ALWAYS AS (content->>'id') STORED UNIQUE NOT NULL,\n content JSONB NOT NULL,\n base_item jsonb GENERATED ALWAYS AS (pgstac.collection_base_item(content)) STORED,\n geometry geometry GENERATED ALWAYS AS (pgstac.collection_geom(content)) STORED,\n datetime timestamptz GENERATED ALWAYS AS (pgstac.collection_datetime(content)) STORED,\n end_datetime timestamptz GENERATED ALWAYS AS (pgstac.collection_enddatetime(content)) STORED,\n private jsonb,\n partition_trunc text CHECK (partition_trunc IN ('year', 'month'))\n);\n\nCREATE OR REPLACE FUNCTION collection_base_item(cid text) RETURNS jsonb AS $$\n SELECT pgstac.collection_base_item(content) FROM pgstac.collections WHERE id = cid LIMIT 1;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION create_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION update_collection(data jsonb) RETURNS VOID AS $$\nDECLARE\n out collections%ROWTYPE;\nBEGIN\n UPDATE collections SET content=data WHERE id = data->>'id' RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ON CONFLICT (id) DO\n UPDATE\n SET content=EXCLUDED.content\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION delete_collection(_id text) RETURNS VOID AS $$\nDECLARE\n out collections%ROWTYPE;\nBEGIN\n DELETE FROM collections WHERE id = _id RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION get_collection(id text) RETURNS jsonb AS $$\n SELECT content FROM collections\n WHERE id=$1\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION all_collections() RETURNS jsonb AS $$\n SELECT coalesce(jsonb_agg(content), '[]'::jsonb) FROM collections;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION collection_delete_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n collection_base_partition text := concat('_items_', OLD.key);\nBEGIN\n EXECUTE format($q$\n DELETE FROM partition_stats WHERE partition IN (\n SELECT partition FROM partition_sys_meta\n WHERE collection=%L\n );\n DROP TABLE IF EXISTS %I CASCADE;\n $q$,\n OLD.id,\n collection_base_partition\n );\n RETURN OLD;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER collection_delete_trigger BEFORE DELETE ON collections\nFOR EACH ROW EXECUTE FUNCTION collection_delete_trigger_func();\nCREATE OR REPLACE FUNCTION queryable_signature(n text, c text[]) RETURNS text AS $$\n SELECT concat(n, c);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TABLE queryables (\n id bigint GENERATED ALWAYS AS identity PRIMARY KEY,\n name text NOT NULL,\n collection_ids text[], -- used to determine what partitions to create indexes on\n definition jsonb,\n property_path text,\n property_wrapper text,\n property_index_type text\n);\nCREATE INDEX queryables_name_idx ON queryables (name);\nCREATE INDEX queryables_collection_idx ON queryables USING GIN (collection_ids);\nCREATE INDEX queryables_property_wrapper_idx ON queryables (property_wrapper);\n\nCREATE OR REPLACE FUNCTION pgstac.queryables_constraint_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n allcollections text[];\nBEGIN\n RAISE NOTICE 'Making sure that name/collection is unique for queryables %', NEW;\n IF NEW.collection_ids IS NOT NULL THEN\n IF EXISTS (\n SELECT 1\n FROM unnest(NEW.collection_ids) c\n LEFT JOIN\n collections\n ON (collections.id = c)\n WHERE collections.id IS NULL\n ) THEN\n RAISE foreign_key_violation USING MESSAGE = format(\n 'One or more collections in %s do not exist.', NEW.collection_ids\n );\n RETURN NULL;\n END IF;\n END IF;\n IF TG_OP = 'INSERT' THEN\n IF EXISTS (\n SELECT 1 FROM queryables q\n WHERE\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n )\n ) THEN\n RAISE unique_violation USING MESSAGE = format(\n 'There is already a queryable for %s for a collection in %s: %s',\n NEW.name,\n NEW.collection_ids,\n\t\t\t\t(SELECT json_agg(row_to_json(q)) FROM queryables q WHERE\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n ))\n );\n RETURN NULL;\n END IF;\n END IF;\n IF TG_OP = 'UPDATE' THEN\n IF EXISTS (\n SELECT 1 FROM queryables q\n WHERE\n q.id != NEW.id\n AND\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n )\n ) THEN\n RAISE unique_violation\n USING MESSAGE = format(\n 'There is already a queryable for %s for a collection in %s',\n NEW.name,\n NEW.collection_ids\n );\n RETURN NULL;\n END IF;\n END IF;\n\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER queryables_constraint_insert_trigger\nBEFORE INSERT ON queryables\nFOR EACH ROW EXECUTE PROCEDURE queryables_constraint_triggerfunc();\n\nCREATE TRIGGER queryables_constraint_update_trigger\nBEFORE UPDATE ON queryables\nFOR EACH ROW\nWHEN (NEW.name = OLD.name AND NEW.collection_ids IS DISTINCT FROM OLD.collection_ids)\nEXECUTE PROCEDURE queryables_constraint_triggerfunc();\n\n\nCREATE OR REPLACE FUNCTION array_to_path(arr text[]) RETURNS text AS $$\n SELECT string_agg(\n quote_literal(v),\n '->'\n ) FROM unnest(arr) v;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\n\n\nCREATE OR REPLACE FUNCTION queryable(\n IN dotpath text,\n OUT path text,\n OUT expression text,\n OUT wrapper text,\n OUT nulled_wrapper text\n) AS $$\nDECLARE\n q RECORD;\n path_elements text[];\nBEGIN\n dotpath := replace(dotpath, 'properties.', '');\n IF dotpath = 'start_datetime' THEN\n dotpath := 'datetime';\n END IF;\n IF dotpath IN ('id', 'geometry', 'datetime', 'end_datetime', 'collection') THEN\n path := dotpath;\n expression := dotpath;\n wrapper := NULL;\n RETURN;\n END IF;\n\n SELECT * INTO q FROM queryables\n WHERE\n name=dotpath\n OR name = 'properties.' || dotpath\n OR name = replace(dotpath, 'properties.', '')\n ;\n IF q.property_wrapper IS NULL THEN\n IF q.definition->>'type' = 'number' THEN\n wrapper := 'to_float';\n nulled_wrapper := wrapper;\n ELSIF q.definition->>'format' = 'date-time' THEN\n wrapper := 'to_tstz';\n nulled_wrapper := wrapper;\n ELSE\n nulled_wrapper := NULL;\n wrapper := 'to_text';\n END IF;\n ELSE\n wrapper := q.property_wrapper;\n nulled_wrapper := wrapper;\n END IF;\n IF q.property_path IS NOT NULL THEN\n path := q.property_path;\n ELSE\n path_elements := string_to_array(dotpath, '.');\n IF path_elements[1] IN ('links', 'assets', 'stac_version', 'stac_extensions') THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSIF path_elements[1] = 'properties' THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSE\n path := format($F$content->'properties'->%s$F$, array_to_path(path_elements));\n END IF;\n END IF;\n expression := format('%I(%s)', wrapper, path);\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\nCREATE OR REPLACE FUNCTION unnest_collection(collection_ids text[] DEFAULT NULL) RETURNS SETOF text AS $$\n DECLARE\n BEGIN\n IF collection_ids IS NULL THEN\n RETURN QUERY SELECT id FROM collections;\n END IF;\n RETURN QUERY SELECT unnest(collection_ids);\n END;\n$$ LANGUAGE PLPGSQL STABLE;\n\nCREATE OR REPLACE FUNCTION normalize_indexdef(def text) RETURNS text AS $$\nDECLARE\nBEGIN\n def := btrim(def, ' \\n\\t');\n\tdef := regexp_replace(def, '^CREATE (UNIQUE )?INDEX ([^ ]* )?ON (ONLY )?([^ ]* )?', '', 'i');\n RETURN def;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION indexdef(q queryables) RETURNS text AS $$\n DECLARE\n out text;\n BEGIN\n IF q.name = 'id' THEN\n out := 'CREATE UNIQUE INDEX ON %I USING btree (id)';\n ELSIF q.name = 'datetime' THEN\n out := 'CREATE INDEX ON %I USING btree (datetime DESC, end_datetime)';\n ELSIF q.name = 'geometry' THEN\n out := 'CREATE INDEX ON %I USING gist (geometry)';\n ELSE\n out := format($q$CREATE INDEX ON %%I USING %s (%s(((content -> 'properties'::text) -> %L::text)))$q$,\n lower(COALESCE(q.property_index_type, 'BTREE')),\n lower(COALESCE(q.property_wrapper, 'to_text')),\n q.name\n );\n END IF;\n RETURN btrim(out, ' \\n\\t');\n END;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nDROP VIEW IF EXISTS pgstac_indexes;\nCREATE VIEW pgstac_indexes AS\nSELECT\n i.schemaname,\n i.tablename,\n i.indexname,\n regexp_replace(btrim(replace(replace(indexdef, i.indexname, ''),'pgstac.',''),' \\t\\n'), '[ ]+', ' ', 'g') as idx,\n COALESCE(\n (regexp_match(indexdef, '\\(([a-zA-Z]+)\\)'))[1],\n (regexp_match(indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_-]+)''::text'))[1],\n CASE WHEN indexdef ~* '\\(datetime desc, end_datetime\\)' THEN 'datetime' ELSE NULL END\n ) AS field,\n pg_table_size(i.indexname::text) as index_size,\n pg_size_pretty(pg_table_size(i.indexname::text)) as index_size_pretty\nFROM\n pg_indexes i\nWHERE i.schemaname='pgstac' and i.tablename ~ '_items_' AND indexdef !~* ' only ';\n\nDROP VIEW IF EXISTS pgstac_index_stats;\nCREATE VIEW pgstac_indexes_stats AS\nSELECT\n i.schemaname,\n i.tablename,\n i.indexname,\n indexdef,\n COALESCE(\n (regexp_match(indexdef, '\\(([a-zA-Z]+)\\)'))[1],\n (regexp_match(indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_]+)''::text'))[1],\n CASE WHEN indexdef ~* '\\(datetime desc, end_datetime\\)' THEN 'datetime_end_datetime' ELSE NULL END\n ) AS field,\n pg_table_size(i.indexname::text) as index_size,\n pg_size_pretty(pg_table_size(i.indexname::text)) as index_size_pretty,\n n_distinct,\n most_common_vals::text::text[],\n most_common_freqs::text::text[],\n histogram_bounds::text::text[],\n correlation\nFROM\n pg_indexes i\n LEFT JOIN pg_stats s ON (s.tablename = i.indexname)\nWHERE i.schemaname='pgstac' and i.tablename ~ '_items_';\n\nCREATE OR REPLACE FUNCTION queryable_indexes(\n IN treeroot text DEFAULT 'items',\n IN changes boolean DEFAULT FALSE,\n OUT collection text,\n OUT partition text,\n OUT field text,\n OUT indexname text,\n OUT existing_idx text,\n OUT queryable_idx text\n) RETURNS SETOF RECORD AS $$\nWITH p AS (\n SELECT\n relid::text as partition,\n replace(replace(\n CASE\n WHEN parentrelid::regclass::text='items' THEN pg_get_expr(c.relpartbound, c.oid)\n ELSE pg_get_expr(parent.relpartbound, parent.oid)\n END,\n 'FOR VALUES IN (''',''), ''')',\n ''\n ) AS collection\n FROM pg_partition_tree(treeroot)\n JOIN pg_class c ON (relid::regclass = c.oid)\n JOIN pg_class parent ON (parentrelid::regclass = parent.oid AND isleaf)\n ), i AS (\n SELECT\n partition,\n indexname,\n regexp_replace(btrim(replace(replace(indexdef, indexname, ''),'pgstac.',''),' \\t\\n'), '[ ]+', ' ', 'g') as iidx,\n COALESCE(\n (regexp_match(indexdef, '\\(([a-zA-Z]+)\\)'))[1],\n (regexp_match(indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_-]+)''::text'))[1],\n CASE WHEN indexdef ~* '\\(datetime desc, end_datetime\\)' THEN 'datetime' ELSE NULL END\n ) AS field\n FROM\n pg_indexes\n JOIN p ON (tablename=partition)\n ), q AS (\n SELECT\n name AS field,\n collection,\n partition,\n format(indexdef(queryables), partition) as qidx\n FROM queryables, unnest_collection(queryables.collection_ids) collection\n JOIN p USING (collection)\n WHERE property_index_type IS NOT NULL OR name IN ('datetime','geometry','id')\n )\n SELECT\n collection,\n partition,\n field,\n indexname,\n iidx as existing_idx,\n qidx as queryable_idx\n FROM i FULL JOIN q USING (field, partition)\n WHERE CASE WHEN changes THEN lower(iidx) IS DISTINCT FROM lower(qidx) ELSE TRUE END;\n;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION maintain_index(\n indexname text,\n queryable_idx text,\n dropindexes boolean DEFAULT FALSE,\n rebuildindexes boolean DEFAULT FALSE,\n idxconcurrently boolean DEFAULT FALSE\n) RETURNS VOID AS $$\nDECLARE\nBEGIN\n IF indexname IS NOT NULL THEN\n IF dropindexes OR queryable_idx IS NOT NULL THEN\n EXECUTE format('DROP INDEX IF EXISTS %I;', indexname);\n ELSIF rebuildindexes THEN\n IF idxconcurrently THEN\n EXECUTE format('REINDEX INDEX CONCURRENTLY %I;', indexname);\n ELSE\n EXECUTE format('REINDEX INDEX CONCURRENTLY %I;', indexname);\n END IF;\n END IF;\n END IF;\n IF queryable_idx IS NOT NULL THEN\n IF idxconcurrently THEN\n EXECUTE replace(queryable_idx, 'INDEX', 'INDEX CONCURRENTLY');\n ELSE EXECUTE queryable_idx;\n END IF;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\n\nset check_function_bodies to off;\nCREATE OR REPLACE FUNCTION maintain_partition_queries(\n part text DEFAULT 'items',\n dropindexes boolean DEFAULT FALSE,\n rebuildindexes boolean DEFAULT FALSE,\n idxconcurrently boolean DEFAULT FALSE\n) RETURNS SETOF text AS $$\nDECLARE\n rec record;\n q text;\nBEGIN\n FOR rec IN (\n SELECT * FROM queryable_indexes(part,true)\n ) LOOP\n q := format(\n 'SELECT maintain_index(\n %L,%L,%L,%L,%L\n );',\n rec.indexname,\n rec.queryable_idx,\n dropindexes,\n rebuildindexes,\n idxconcurrently\n );\n RAISE NOTICE 'Q: %', q;\n RETURN NEXT q;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION maintain_partitions(\n part text DEFAULT 'items',\n dropindexes boolean DEFAULT FALSE,\n rebuildindexes boolean DEFAULT FALSE\n) RETURNS VOID AS $$\n WITH t AS (\n SELECT run_or_queue(q) FROM maintain_partition_queries(part, dropindexes, rebuildindexes) q\n ) SELECT count(*) FROM t;\n$$ LANGUAGE SQL;\n\n\nCREATE OR REPLACE FUNCTION queryables_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\nBEGIN\n PERFORM maintain_partitions();\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER queryables_trigger AFTER INSERT OR UPDATE ON queryables\nFOR EACH STATEMENT EXECUTE PROCEDURE queryables_trigger_func();\n\n\nCREATE OR REPLACE FUNCTION get_queryables(_collection_ids text[] DEFAULT NULL) RETURNS jsonb AS $$\nDECLARE\nBEGIN\n -- Build up queryables if the input contains valid collection ids or is empty\n IF EXISTS (\n SELECT 1 FROM collections\n WHERE\n _collection_ids IS NULL\n OR cardinality(_collection_ids) = 0\n OR id = ANY(_collection_ids)\n )\n THEN\n RETURN (\n WITH base AS (\n SELECT\n unnest(collection_ids) as collection_id,\n name,\n coalesce(definition, '{\"type\":\"string\"}'::jsonb) as definition\n FROM queryables\n WHERE\n _collection_ids IS NULL OR\n _collection_ids = '{}'::text[] OR\n _collection_ids && collection_ids\n UNION ALL\n SELECT null, name, coalesce(definition, '{\"type\":\"string\"}'::jsonb) as definition\n FROM queryables WHERE collection_ids IS NULL OR collection_ids = '{}'::text[]\n ), g AS (\n SELECT\n name,\n first_notnull(definition) as definition,\n jsonb_array_unique_merge(definition->'enum') as enum,\n jsonb_min(definition->'minimum') as minimum,\n jsonb_min(definition->'maxiumn') as maximum\n FROM base\n GROUP BY 1\n )\n SELECT\n jsonb_build_object(\n '$schema', 'http://json-schema.org/draft-07/schema#',\n '$id', '',\n 'type', 'object',\n 'title', 'STAC Queryables.',\n 'properties', jsonb_object_agg(\n name,\n definition\n ||\n jsonb_strip_nulls(jsonb_build_object(\n 'enum', enum,\n 'minimum', minimum,\n 'maximum', maximum\n ))\n ),\n 'additionalProperties', pgstac.additional_properties()\n )\n FROM g\n );\n ELSE\n RETURN NULL;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\nCREATE OR REPLACE FUNCTION get_queryables(_collection text DEFAULT NULL) RETURNS jsonb AS $$\n SELECT\n CASE\n WHEN _collection IS NULL THEN get_queryables(NULL::text[])\n ELSE get_queryables(ARRAY[_collection])\n END\n ;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_queryables() RETURNS jsonb AS $$\n SELECT get_queryables(NULL::text[]);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION schema_qualify_refs(url text, j jsonb) returns jsonb as $$\n SELECT regexp_replace(j::text, '\"\\$ref\": \"#', concat('\"$ref\": \"', url, '#'), 'g')::jsonb;\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE VIEW stac_extension_queryables AS\nSELECT DISTINCT key as name, schema_qualify_refs(e.url, j.value) as definition FROM stac_extensions e, jsonb_each(e.content->'definitions'->'fields'->'properties') j;\n\n\nCREATE OR REPLACE FUNCTION missing_queryables(_collection text, _tablesample float DEFAULT 5, minrows float DEFAULT 10) RETURNS TABLE(collection text, name text, definition jsonb, property_wrapper text) AS $$\nDECLARE\n q text;\n _partition text;\n explain_json json;\n psize float;\n estrows float;\nBEGIN\n SELECT format('_items_%s', key) INTO _partition FROM collections WHERE id=_collection;\n\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM %I;', _partition)\n INTO explain_json;\n psize := explain_json->0->'Plan'->'Plan Rows';\n estrows := _tablesample * .01 * psize;\n IF estrows < minrows THEN\n _tablesample := least(100,greatest(_tablesample, (estrows / psize) / 100));\n RAISE NOTICE '%', (psize / estrows) / 100;\n END IF;\n RAISE NOTICE 'Using tablesample % to find missing queryables from % % that has ~% rows estrows: %', _tablesample, _collection, _partition, psize, estrows;\n\n q := format(\n $q$\n WITH q AS (\n SELECT * FROM queryables\n WHERE\n collection_ids IS NULL\n OR %L = ANY(collection_ids)\n ), t AS (\n SELECT\n content->'properties' AS properties\n FROM\n %I\n TABLESAMPLE SYSTEM(%L)\n ), p AS (\n SELECT DISTINCT ON (key)\n key,\n value,\n s.definition\n FROM t\n JOIN LATERAL jsonb_each(properties) ON TRUE\n LEFT JOIN q ON (q.name=key)\n LEFT JOIN stac_extension_queryables s ON (s.name=key)\n WHERE q.definition IS NULL\n )\n SELECT\n %L,\n key,\n COALESCE(definition, jsonb_build_object('type',jsonb_typeof(value))) as definition,\n CASE\n WHEN definition->>'type' = 'integer' THEN 'to_int'\n WHEN COALESCE(definition->>'type', jsonb_typeof(value)) = 'number' THEN 'to_float'\n WHEN COALESCE(definition->>'type', jsonb_typeof(value)) = 'array' THEN 'to_text_array'\n ELSE 'to_text'\n END\n FROM p;\n $q$,\n _collection,\n _partition,\n _tablesample,\n _collection\n );\n RETURN QUERY EXECUTE q;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION missing_queryables(_tablesample float DEFAULT 5) RETURNS TABLE(collection_ids text[], name text, definition jsonb, property_wrapper text) AS $$\n SELECT\n array_agg(collection),\n name,\n definition,\n property_wrapper\n FROM\n collections\n JOIN LATERAL\n missing_queryables(id, _tablesample) c\n ON TRUE\n GROUP BY\n 2,3,4\n ORDER BY 2,1\n ;\n$$ LANGUAGE SQL;\nCREATE OR REPLACE FUNCTION parse_dtrange(\n _indate jsonb,\n relative_base timestamptz DEFAULT date_trunc('hour', CURRENT_TIMESTAMP)\n) RETURNS tstzrange AS $$\nDECLARE\n timestrs text[];\n s timestamptz;\n e timestamptz;\nBEGIN\n timestrs :=\n CASE\n WHEN _indate ? 'timestamp' THEN\n ARRAY[_indate->>'timestamp']\n WHEN _indate ? 'interval' THEN\n to_text_array(_indate->'interval')\n WHEN jsonb_typeof(_indate) = 'array' THEN\n to_text_array(_indate)\n ELSE\n regexp_split_to_array(\n _indate->>0,\n '/'\n )\n END;\n RAISE NOTICE 'TIMESTRS %', timestrs;\n IF cardinality(timestrs) = 1 THEN\n IF timestrs[1] ILIKE 'P%' THEN\n RETURN tstzrange(relative_base - upper(timestrs[1])::interval, relative_base, '[)');\n END IF;\n s := timestrs[1]::timestamptz;\n RETURN tstzrange(s, s, '[]');\n END IF;\n\n IF cardinality(timestrs) != 2 THEN\n RAISE EXCEPTION 'Timestamp cannot have more than 2 values';\n END IF;\n\n IF timestrs[1] = '..' OR timestrs[1] = '' THEN\n s := '-infinity'::timestamptz;\n e := timestrs[2]::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] = '..' OR timestrs[2] = '' THEN\n s := timestrs[1]::timestamptz;\n e := 'infinity'::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[1] ILIKE 'P%' AND timestrs[2] NOT ILIKE 'P%' THEN\n e := timestrs[2]::timestamptz;\n s := e - upper(timestrs[1])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] ILIKE 'P%' AND timestrs[1] NOT ILIKE 'P%' THEN\n s := timestrs[1]::timestamptz;\n e := s + upper(timestrs[2])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n s := timestrs[1]::timestamptz;\n e := timestrs[2]::timestamptz;\n\n RETURN tstzrange(s,e,'[)');\n\n RETURN NULL;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT PARALLEL SAFE SET TIME ZONE 'UTC';\n\nCREATE OR REPLACE FUNCTION parse_dtrange(\n _indate text,\n relative_base timestamptz DEFAULT CURRENT_TIMESTAMP\n) RETURNS tstzrange AS $$\n SELECT parse_dtrange(to_jsonb(_indate), relative_base);\n$$ LANGUAGE SQL STABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION temporal_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\n ll text := 'datetime';\n lh text := 'end_datetime';\n rrange tstzrange;\n rl text;\n rh text;\n outq text;\nBEGIN\n rrange := parse_dtrange(args->1);\n RAISE NOTICE 'Constructing temporal query OP: %, ARGS: %, RRANGE: %', op, args, rrange;\n op := lower(op);\n rl := format('%L::timestamptz', lower(rrange));\n rh := format('%L::timestamptz', upper(rrange));\n outq := CASE op\n WHEN 't_before' THEN 'lh < rl'\n WHEN 't_after' THEN 'll > rh'\n WHEN 't_meets' THEN 'lh = rl'\n WHEN 't_metby' THEN 'll = rh'\n WHEN 't_overlaps' THEN 'll < rl AND rl < lh < rh'\n WHEN 't_overlappedby' THEN 'rl < ll < rh AND lh > rh'\n WHEN 't_starts' THEN 'll = rl AND lh < rh'\n WHEN 't_startedby' THEN 'll = rl AND lh > rh'\n WHEN 't_during' THEN 'll > rl AND lh < rh'\n WHEN 't_contains' THEN 'll < rl AND lh > rh'\n WHEN 't_finishes' THEN 'll > rl AND lh = rh'\n WHEN 't_finishedby' THEN 'll < rl AND lh = rh'\n WHEN 't_equals' THEN 'll = rl AND lh = rh'\n WHEN 't_disjoint' THEN 'NOT (ll <= rh AND lh >= rl)'\n WHEN 't_intersects' THEN 'll <= rh AND lh >= rl'\n WHEN 'anyinteracts' THEN 'll <= rh AND lh >= rl'\n END;\n outq := regexp_replace(outq, '\\mll\\M', ll);\n outq := regexp_replace(outq, '\\mlh\\M', lh);\n outq := regexp_replace(outq, '\\mrl\\M', rl);\n outq := regexp_replace(outq, '\\mrh\\M', rh);\n outq := format('(%s)', outq);\n RETURN outq;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\n\n\nCREATE OR REPLACE FUNCTION spatial_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\n geom text;\n j jsonb := args->1;\nBEGIN\n op := lower(op);\n RAISE NOTICE 'Constructing spatial query OP: %, ARGS: %', op, args;\n IF op NOT IN ('s_equals','s_disjoint','s_touches','s_within','s_overlaps','s_crosses','s_intersects','intersects','s_contains') THEN\n RAISE EXCEPTION 'Spatial Operator % Not Supported', op;\n END IF;\n op := regexp_replace(op, '^s_', 'st_');\n IF op = 'intersects' THEN\n op := 'st_intersects';\n END IF;\n -- Convert geometry to WKB string\n IF j ? 'type' AND j ? 'coordinates' THEN\n geom := st_geomfromgeojson(j)::text;\n ELSIF jsonb_typeof(j) = 'array' THEN\n geom := bbox_geom(j)::text;\n END IF;\n\n RETURN format('%s(geometry, %L::geometry)', op, geom);\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION query_to_cql2(q jsonb) RETURNS jsonb AS $$\n-- Translates anything passed in through the deprecated \"query\" into equivalent CQL2\nWITH t AS (\n SELECT key as property, value as ops\n FROM jsonb_each(q)\n), t2 AS (\n SELECT property, (jsonb_each(ops)).*\n FROM t WHERE jsonb_typeof(ops) = 'object'\n UNION ALL\n SELECT property, 'eq', ops\n FROM t WHERE jsonb_typeof(ops) != 'object'\n)\nSELECT\n jsonb_strip_nulls(jsonb_build_object(\n 'op', 'and',\n 'args', jsonb_agg(\n jsonb_build_object(\n 'op', key,\n 'args', jsonb_build_array(\n jsonb_build_object('property',property),\n value\n )\n )\n )\n )\n) as qcql FROM t2\n;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\nCREATE OR REPLACE FUNCTION cql1_to_cql2(j jsonb) RETURNS jsonb AS $$\nDECLARE\n args jsonb;\n ret jsonb;\nBEGIN\n RAISE NOTICE 'CQL1_TO_CQL2: %', j;\n IF j ? 'filter' THEN\n RETURN cql1_to_cql2(j->'filter');\n END IF;\n IF j ? 'property' THEN\n RETURN j;\n END IF;\n IF jsonb_typeof(j) = 'array' THEN\n SELECT jsonb_agg(cql1_to_cql2(el)) INTO args FROM jsonb_array_elements(j) el;\n RETURN args;\n END IF;\n IF jsonb_typeof(j) = 'number' THEN\n RETURN j;\n END IF;\n IF jsonb_typeof(j) = 'string' THEN\n RETURN j;\n END IF;\n\n IF jsonb_typeof(j) = 'object' THEN\n SELECT jsonb_build_object(\n 'op', key,\n 'args', cql1_to_cql2(value)\n ) INTO ret\n FROM jsonb_each(j)\n WHERE j IS NOT NULL;\n RETURN ret;\n END IF;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE STRICT;\n\nCREATE TABLE cql2_ops (\n op text PRIMARY KEY,\n template text,\n types text[]\n);\nINSERT INTO cql2_ops (op, template, types) VALUES\n ('eq', '%s = %s', NULL),\n ('neq', '%s != %s', NULL),\n ('ne', '%s != %s', NULL),\n ('!=', '%s != %s', NULL),\n ('<>', '%s != %s', NULL),\n ('lt', '%s < %s', NULL),\n ('lte', '%s <= %s', NULL),\n ('gt', '%s > %s', NULL),\n ('gte', '%s >= %s', NULL),\n ('le', '%s <= %s', NULL),\n ('ge', '%s >= %s', NULL),\n ('=', '%s = %s', NULL),\n ('<', '%s < %s', NULL),\n ('<=', '%s <= %s', NULL),\n ('>', '%s > %s', NULL),\n ('>=', '%s >= %s', NULL),\n ('like', '%s LIKE %s', NULL),\n ('ilike', '%s ILIKE %s', NULL),\n ('+', '%s + %s', NULL),\n ('-', '%s - %s', NULL),\n ('*', '%s * %s', NULL),\n ('/', '%s / %s', NULL),\n ('not', 'NOT (%s)', NULL),\n ('between', '%s BETWEEN %s AND %s', NULL),\n ('isnull', '%s IS NULL', NULL),\n ('upper', 'upper(%s)', NULL),\n ('lower', 'lower(%s)', NULL),\n ('casei', 'upper(%s)', NULL),\n ('accenti', 'unaccent(%s)', NULL)\nON CONFLICT (op) DO UPDATE\n SET\n template = EXCLUDED.template\n;\n\n\nCREATE OR REPLACE FUNCTION cql2_query(j jsonb, wrapper text DEFAULT NULL) RETURNS text AS $$\n#variable_conflict use_variable\nDECLARE\n args jsonb := j->'args';\n arg jsonb;\n op text := lower(j->>'op');\n cql2op RECORD;\n literal text;\n _wrapper text;\n leftarg text;\n rightarg text;\n prop text;\n extra_props bool := pgstac.additional_properties();\nBEGIN\n IF j IS NULL OR (op IS NOT NULL AND args IS NULL) THEN\n RETURN NULL;\n END IF;\n RAISE NOTICE 'CQL2_QUERY: %', j;\n\n -- check if all properties are represented in the queryables\n IF NOT extra_props THEN\n FOR prop IN\n SELECT DISTINCT p->>0\n FROM jsonb_path_query(j, 'strict $.**.property') p\n WHERE p->>0 NOT IN ('id', 'datetime', 'geometry', 'end_datetime', 'collection')\n LOOP\n IF (queryable(prop)).nulled_wrapper IS NULL THEN\n RAISE EXCEPTION 'Term % is not found in queryables.', prop;\n END IF;\n END LOOP;\n END IF;\n\n IF j ? 'filter' THEN\n RETURN cql2_query(j->'filter');\n END IF;\n\n IF j ? 'upper' THEN\n RETURN cql2_query(jsonb_build_object('op', 'upper', 'args', j->'upper'));\n END IF;\n\n IF j ? 'lower' THEN\n RETURN cql2_query(jsonb_build_object('op', 'lower', 'args', j->'lower'));\n END IF;\n\n -- Temporal Query\n IF op ilike 't_%' or op = 'anyinteracts' THEN\n RETURN temporal_op_query(op, args);\n END IF;\n\n -- If property is a timestamp convert it to text to use with\n -- general operators\n IF j ? 'timestamp' THEN\n RETURN format('%L::timestamptz', to_tstz(j->'timestamp'));\n END IF;\n IF j ? 'interval' THEN\n RAISE EXCEPTION 'Please use temporal operators when using intervals.';\n RETURN NONE;\n END IF;\n\n -- Spatial Query\n IF op ilike 's_%' or op = 'intersects' THEN\n RETURN spatial_op_query(op, args);\n END IF;\n\n IF op IN ('a_equals','a_contains','a_contained_by','a_overlaps') THEN\n IF args->0 ? 'property' THEN\n leftarg := format('to_text_array(%s)', (queryable(args->0->>'property')).path);\n END IF;\n IF args->1 ? 'property' THEN\n rightarg := format('to_text_array(%s)', (queryable(args->1->>'property')).path);\n END IF;\n RETURN FORMAT(\n '%s %s %s',\n COALESCE(leftarg, quote_literal(to_text_array(args->0))),\n CASE op\n WHEN 'a_equals' THEN '='\n WHEN 'a_contains' THEN '@>'\n WHEN 'a_contained_by' THEN '<@'\n WHEN 'a_overlaps' THEN '&&'\n END,\n COALESCE(rightarg, quote_literal(to_text_array(args->1)))\n );\n END IF;\n\n IF op = 'in' THEN\n RAISE NOTICE 'IN : % % %', args, jsonb_build_array(args->0), args->1;\n args := jsonb_build_array(args->0) || (args->1);\n RAISE NOTICE 'IN2 : %', args;\n END IF;\n\n\n\n IF op = 'between' THEN\n args = jsonb_build_array(\n args->0,\n args->1,\n args->2\n );\n END IF;\n\n -- Make sure that args is an array and run cql2_query on\n -- each element of the array\n RAISE NOTICE 'ARGS PRE: %', args;\n IF j ? 'args' THEN\n IF jsonb_typeof(args) != 'array' THEN\n args := jsonb_build_array(args);\n END IF;\n\n IF jsonb_path_exists(args, '$[*] ? (@.property == \"id\" || @.property == \"datetime\" || @.property == \"end_datetime\" || @.property == \"collection\")') THEN\n wrapper := NULL;\n ELSE\n -- if any of the arguments are a property, try to get the property_wrapper\n FOR arg IN SELECT jsonb_path_query(args, '$[*] ? (@.property != null)') LOOP\n RAISE NOTICE 'Arg: %', arg;\n wrapper := (queryable(arg->>'property')).nulled_wrapper;\n RAISE NOTICE 'Property: %, Wrapper: %', arg, wrapper;\n IF wrapper IS NOT NULL THEN\n EXIT;\n END IF;\n END LOOP;\n\n -- if the property was not in queryables, see if any args were numbers\n IF\n wrapper IS NULL\n AND jsonb_path_exists(args, '$[*] ? (@.type()==\"number\")')\n THEN\n wrapper := 'to_float';\n END IF;\n wrapper := coalesce(wrapper, 'to_text');\n END IF;\n\n SELECT jsonb_agg(cql2_query(a, wrapper))\n INTO args\n FROM jsonb_array_elements(args) a;\n END IF;\n RAISE NOTICE 'ARGS: %', args;\n\n IF op IN ('and', 'or') THEN\n RETURN\n format(\n '(%s)',\n array_to_string(to_text_array(args), format(' %s ', upper(op)))\n );\n END IF;\n\n IF op = 'in' THEN\n RAISE NOTICE 'IN -- % %', args->0, to_text(args->0);\n RETURN format(\n '%s IN (%s)',\n to_text(args->0),\n array_to_string((to_text_array(args))[2:], ',')\n );\n END IF;\n\n -- Look up template from cql2_ops\n IF j ? 'op' THEN\n SELECT * INTO cql2op FROM cql2_ops WHERE cql2_ops.op ilike op;\n IF FOUND THEN\n -- If specific index set in queryables for a property cast other arguments to that type\n\n RETURN format(\n cql2op.template,\n VARIADIC (to_text_array(args))\n );\n ELSE\n RAISE EXCEPTION 'Operator % Not Supported.', op;\n END IF;\n END IF;\n\n\n IF wrapper IS NOT NULL THEN\n RAISE NOTICE 'Wrapping % with %', j, wrapper;\n IF j ? 'property' THEN\n RETURN format('%I(%s)', wrapper, (queryable(j->>'property')).path);\n ELSE\n RETURN format('%I(%L)', wrapper, j);\n END IF;\n ELSIF j ? 'property' THEN\n RETURN quote_ident(j->>'property');\n END IF;\n\n RETURN quote_literal(to_text(j));\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION paging_dtrange(\n j jsonb\n) RETURNS tstzrange AS $$\nDECLARE\n op text;\n filter jsonb := j->'filter';\n dtrange tstzrange := tstzrange('-infinity'::timestamptz,'infinity'::timestamptz);\n sdate timestamptz := '-infinity'::timestamptz;\n edate timestamptz := 'infinity'::timestamptz;\n jpitem jsonb;\nBEGIN\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n END IF;\n IF NOT (filter @? '$.**.op ? (@ == \"or\" || @ == \"not\")') THEN\n FOR jpitem IN SELECT j FROM jsonb_path_query(filter,'strict $.** ? (@.args[*].property == \"datetime\")'::jsonpath) j LOOP\n op := lower(jpitem->>'op');\n dtrange := parse_dtrange(jpitem->'args'->1);\n IF op IN ('<=', 'lt', 'lte', '<', 'le', 't_before') THEN\n sdate := greatest(sdate,'-infinity');\n edate := least(edate, upper(dtrange));\n ELSIF op IN ('>=', '>', 'gt', 'gte', 'ge', 't_after') THEN\n edate := least(edate, 'infinity');\n sdate := greatest(sdate, lower(dtrange));\n ELSIF op IN ('=', 'eq') THEN\n edate := least(edate, upper(dtrange));\n sdate := greatest(sdate, lower(dtrange));\n END IF;\n RAISE NOTICE '2 OP: %, ARGS: %, DTRANGE: %, SDATE: %, EDATE: %', op, jpitem->'args'->1, dtrange, sdate, edate;\n END LOOP;\n END IF;\n IF sdate > edate THEN\n RETURN 'empty'::tstzrange;\n END IF;\n RETURN tstzrange(sdate,edate, '[]');\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT SET TIME ZONE 'UTC';\n\nCREATE OR REPLACE FUNCTION paging_collections(\n IN j jsonb\n) RETURNS text[] AS $$\nDECLARE\n filter jsonb := j->'filter';\n jpitem jsonb;\n op text;\n args jsonb;\n arg jsonb;\n collections text[];\nBEGIN\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n END IF;\n IF NOT (filter @? '$.**.op ? (@ == \"or\" || @ == \"not\")') THEN\n FOR jpitem IN SELECT j FROM jsonb_path_query(filter,'strict $.** ? (@.args[*].property == \"collection\")'::jsonpath) j LOOP\n RAISE NOTICE 'JPITEM: %', jpitem;\n op := jpitem->>'op';\n args := jpitem->'args';\n IF op IN ('=', 'eq', 'in') THEN\n FOR arg IN SELECT a FROM jsonb_array_elements(args) a LOOP\n IF jsonb_typeof(arg) IN ('string', 'array') THEN\n RAISE NOTICE 'arg: %, collections: %', arg, collections;\n IF collections IS NULL OR collections = '{}'::text[] THEN\n collections := to_text_array(arg);\n ELSE\n collections := array_intersection(collections, to_text_array(arg));\n END IF;\n END IF;\n END LOOP;\n END IF;\n END LOOP;\n END IF;\n IF collections = '{}'::text[] THEN\n RETURN NULL;\n END IF;\n RETURN collections;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\nCREATE TABLE items (\n id text NOT NULL,\n geometry geometry NOT NULL,\n collection text NOT NULL,\n datetime timestamptz NOT NULL,\n end_datetime timestamptz NOT NULL,\n content JSONB NOT NULL,\n private jsonb\n)\nPARTITION BY LIST (collection)\n;\n\nCREATE INDEX \"datetime_idx\" ON items USING BTREE (datetime DESC, end_datetime ASC);\nCREATE INDEX \"geometry_idx\" ON items USING GIST (geometry);\n\nCREATE STATISTICS datetime_stats (dependencies) on datetime, end_datetime from items;\n\nALTER TABLE items ADD CONSTRAINT items_collections_fk FOREIGN KEY (collection) REFERENCES collections(id) ON DELETE CASCADE DEFERRABLE;\n\nCREATE OR REPLACE FUNCTION partition_after_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p text;\n t timestamptz := clock_timestamp();\nBEGIN\n RAISE NOTICE 'Updating partition stats %', t;\n FOR p IN SELECT DISTINCT partition\n FROM newdata n JOIN partition_sys_meta p\n ON (n.collection=p.collection AND n.datetime <@ p.partition_dtrange)\n LOOP\n PERFORM run_or_queue(format('SELECT update_partition_stats(%L, %L);', p, true));\n END LOOP;\n IF TG_OP IN ('DELETE','UPDATE') THEN\n DELETE FROM format_item_cache c USING newdata n WHERE c.collection = n.collection AND c.id = n.id;\n END IF;\n RAISE NOTICE 't: % %', t, clock_timestamp() - t;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE TRIGGER items_after_insert_trigger\nAFTER INSERT ON items\nREFERENCING NEW TABLE AS newdata\nFOR EACH STATEMENT\nEXECUTE FUNCTION partition_after_triggerfunc();\n\nCREATE TRIGGER items_after_update_trigger\nAFTER DELETE ON items\nREFERENCING OLD TABLE AS newdata\nFOR EACH STATEMENT\nEXECUTE FUNCTION partition_after_triggerfunc();\n\nCREATE TRIGGER items_after_delete_trigger\nAFTER UPDATE ON items\nREFERENCING NEW TABLE AS newdata\nFOR EACH STATEMENT\nEXECUTE FUNCTION partition_after_triggerfunc();\n\n\nCREATE OR REPLACE FUNCTION content_slim(_item jsonb) RETURNS jsonb AS $$\n SELECT strip_jsonb(_item - '{id,geometry,collection,type}'::text[], collection_base_item(_item->>'collection')) - '{id,geometry,collection,type}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_dehydrate(content jsonb) RETURNS items AS $$\n SELECT\n content->>'id' as id,\n stac_geom(content) as geometry,\n content->>'collection' as collection,\n stac_datetime(content) as datetime,\n stac_end_datetime(content) as end_datetime,\n content_slim(content) as content,\n null::jsonb as private\n ;\n$$ LANGUAGE SQL STABLE;\n\nCREATE OR REPLACE FUNCTION include_field(f text, fields jsonb DEFAULT '{}'::jsonb) RETURNS boolean AS $$\nDECLARE\n includes jsonb := fields->'include';\n excludes jsonb := fields->'exclude';\nBEGIN\n IF f IS NULL THEN\n RETURN NULL;\n END IF;\n\n\n IF\n jsonb_typeof(excludes) = 'array'\n AND jsonb_array_length(excludes)>0\n AND excludes ? f\n THEN\n RETURN FALSE;\n END IF;\n\n IF\n (\n jsonb_typeof(includes) = 'array'\n AND jsonb_array_length(includes) > 0\n AND includes ? f\n ) OR\n (\n includes IS NULL\n OR jsonb_typeof(includes) = 'null'\n OR jsonb_array_length(includes) = 0\n )\n THEN\n RETURN TRUE;\n END IF;\n\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nDROP FUNCTION IF EXISTS content_hydrate(jsonb, jsonb, jsonb);\nCREATE OR REPLACE FUNCTION content_hydrate(\n _item jsonb,\n _base_item jsonb,\n fields jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\n SELECT merge_jsonb(\n jsonb_fields(_item, fields),\n jsonb_fields(_base_item, fields)\n );\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION content_hydrate(_item items, _collection collections, fields jsonb DEFAULT '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\n content jsonb;\n base_item jsonb := _collection.base_item;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry, 20)::jsonb;\n END IF;\n output := content_hydrate(\n jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'collection', _item.collection,\n 'type', 'Feature'\n ) || _item.content,\n _collection.base_item,\n fields\n );\n\n RETURN output;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_nonhydrated(\n _item items,\n fields jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry, 20)::jsonb;\n END IF;\n output := jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'collection', _item.collection,\n 'type', 'Feature'\n ) || _item.content;\n RETURN output;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_hydrate(_item items, fields jsonb DEFAULT '{}'::jsonb) RETURNS jsonb AS $$\n SELECT content_hydrate(\n _item,\n (SELECT c FROM collections c WHERE id=_item.collection LIMIT 1),\n fields\n );\n$$ LANGUAGE SQL STABLE;\n\n\nCREATE UNLOGGED TABLE items_staging (\n content JSONB NOT NULL\n);\nCREATE UNLOGGED TABLE items_staging_ignore (\n content JSONB NOT NULL\n);\nCREATE UNLOGGED TABLE items_staging_upsert (\n content JSONB NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION items_staging_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p record;\n _partitions text[];\n part text;\n ts timestamptz := clock_timestamp();\n nrows int;\nBEGIN\n RAISE NOTICE 'Creating Partitions. %', clock_timestamp() - ts;\n\n FOR part IN WITH t AS (\n SELECT\n n.content->>'collection' as collection,\n stac_daterange(n.content->'properties') as dtr,\n partition_trunc\n FROM newdata n JOIN collections ON (n.content->>'collection'=collections.id)\n ), p AS (\n SELECT\n collection,\n COALESCE(date_trunc(partition_trunc::text, lower(dtr)),'-infinity') as d,\n tstzrange(min(lower(dtr)),max(lower(dtr)),'[]') as dtrange,\n tstzrange(min(upper(dtr)),max(upper(dtr)),'[]') as edtrange\n FROM t\n GROUP BY 1,2\n ) SELECT check_partition(collection, dtrange, edtrange) FROM p LOOP\n RAISE NOTICE 'Partition %', part;\n END LOOP;\n\n RAISE NOTICE 'Creating temp table with data to be added. %', clock_timestamp() - ts;\n DROP TABLE IF EXISTS tmpdata;\n CREATE TEMP TABLE tmpdata ON COMMIT DROP AS\n SELECT\n (content_dehydrate(content)).*\n FROM newdata;\n GET DIAGNOSTICS nrows = ROW_COUNT;\n RAISE NOTICE 'Added % rows to tmpdata. %', nrows, clock_timestamp() - ts;\n\n RAISE NOTICE 'Doing the insert. %', clock_timestamp() - ts;\n IF TG_TABLE_NAME = 'items_staging' THEN\n INSERT INTO items\n SELECT * FROM tmpdata;\n GET DIAGNOSTICS nrows = ROW_COUNT;\n RAISE NOTICE 'Inserted % rows to items. %', nrows, clock_timestamp() - ts;\n ELSIF TG_TABLE_NAME = 'items_staging_ignore' THEN\n INSERT INTO items\n SELECT * FROM tmpdata\n ON CONFLICT DO NOTHING;\n GET DIAGNOSTICS nrows = ROW_COUNT;\n RAISE NOTICE 'Inserted % rows to items. %', nrows, clock_timestamp() - ts;\n ELSIF TG_TABLE_NAME = 'items_staging_upsert' THEN\n DELETE FROM items i USING tmpdata s\n WHERE\n i.id = s.id\n AND i.collection = s.collection\n AND i IS DISTINCT FROM s\n ;\n GET DIAGNOSTICS nrows = ROW_COUNT;\n RAISE NOTICE 'Deleted % rows from items. %', nrows, clock_timestamp() - ts;\n INSERT INTO items AS t\n SELECT * FROM tmpdata\n ON CONFLICT DO NOTHING;\n GET DIAGNOSTICS nrows = ROW_COUNT;\n RAISE NOTICE 'Inserted % rows to items. %', nrows, clock_timestamp() - ts;\n END IF;\n\n RAISE NOTICE 'Deleting data from staging table. %', clock_timestamp() - ts;\n DELETE FROM items_staging;\n RAISE NOTICE 'Done. %', clock_timestamp() - ts;\n\n RETURN NULL;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER items_staging_insert_trigger AFTER INSERT ON items_staging REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\nCREATE TRIGGER items_staging_insert_ignore_trigger AFTER INSERT ON items_staging_ignore REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\nCREATE TRIGGER items_staging_insert_upsert_trigger AFTER INSERT ON items_staging_upsert REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\n\nCREATE OR REPLACE FUNCTION item_by_id(_id text, _collection text DEFAULT NULL) RETURNS items AS\n$$\nDECLARE\n i items%ROWTYPE;\nBEGIN\n SELECT * INTO i FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection) LIMIT 1;\n RETURN i;\nEND;\n$$ LANGUAGE PLPGSQL STABLE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION get_item(_id text, _collection text DEFAULT NULL) RETURNS jsonb AS $$\n SELECT content_hydrate(items) FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection);\n$$ LANGUAGE SQL STABLE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION delete_item(_id text, _collection text DEFAULT NULL) RETURNS VOID AS $$\nDECLARE\nout items%ROWTYPE;\nBEGIN\n DELETE FROM items WHERE id = _id AND (_collection IS NULL OR collection=_collection) RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n--/*\nCREATE OR REPLACE FUNCTION create_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION update_item(content jsonb) RETURNS VOID AS $$\nDECLARE\n old items %ROWTYPE;\n out items%ROWTYPE;\nBEGIN\n PERFORM delete_item(content->>'id', content->>'collection');\n PERFORM create_item(content);\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION create_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION collection_bbox(id text) RETURNS jsonb AS $$\n SELECT (replace(replace(replace(st_extent(geometry)::text,'BOX(','[['),')',']]'),' ',','))::jsonb\n FROM items WHERE collection=$1;\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION collection_temporal_extent(id text) RETURNS jsonb AS $$\n SELECT to_jsonb(array[array[min(datetime), max(datetime)]])\n FROM items WHERE collection=$1;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection_extents() RETURNS VOID AS $$\nUPDATE collections\n SET content = jsonb_set_lax(\n content,\n '{extent}'::text[],\n collection_extent(id, FALSE),\n true,\n 'use_json_null'\n )\n;\n$$ LANGUAGE SQL;\nCREATE TABLE partition_stats (\n partition text PRIMARY KEY,\n dtrange tstzrange,\n edtrange tstzrange,\n spatial geometry,\n last_updated timestamptz,\n keys text[]\n) WITH (FILLFACTOR=90);\n\nCREATE INDEX partitions_range_idx ON partition_stats USING GIST(dtrange);\n\n\nCREATE OR REPLACE FUNCTION constraint_tstzrange(expr text) RETURNS tstzrange AS $$\n WITH t AS (\n SELECT regexp_matches(\n expr,\n E'\\\\(''\\([0-9 :+-]*\\)''\\\\).*\\\\(''\\([0-9 :+-]*\\)''\\\\)'\n ) AS m\n ) SELECT tstzrange(m[1]::timestamptz, m[2]::timestamptz) FROM t\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE STRICT;\n\nCREATE OR REPLACE FUNCTION dt_constraint(coid oid, OUT dt tstzrange, OUT edt tstzrange) RETURNS RECORD AS $$\nDECLARE\n expr text := pg_get_constraintdef(coid);\n matches timestamptz[];\nBEGIN\n IF expr LIKE '%NULL%' THEN\n dt := tstzrange(null::timestamptz, null::timestamptz);\n edt := tstzrange(null::timestamptz, null::timestamptz);\n RETURN;\n END IF;\n WITH f AS (SELECT (regexp_matches(expr, E'([0-9]{4}-[0-1][0-9]-[0-3][0-9] [0-2][0-9]:[0-5][0-9]:[0-5][0-9]\\.?[0-9]*)', 'g'))[1] f)\n SELECT array_agg(f::timestamptz) INTO matches FROM f;\n IF cardinality(matches) = 4 THEN\n dt := tstzrange(matches[1], matches[2],'[]');\n edt := tstzrange(matches[3], matches[4], '[]');\n RETURN;\n ELSIF cardinality(matches) = 2 THEN\n edt := tstzrange(matches[1], matches[2],'[]');\n RETURN;\n END IF;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\nCREATE OR REPLACE VIEW partition_sys_meta AS\nSELECT\n (parse_ident(relid::text))[cardinality(parse_ident(relid::text))] as partition,\n replace(replace(CASE WHEN level = 1 THEN pg_get_expr(c.relpartbound, c.oid)\n ELSE pg_get_expr(parent.relpartbound, parent.oid)\n END, 'FOR VALUES IN (''',''), ''')','') AS collection,\n level,\n c.reltuples,\n c.relhastriggers,\n COALESCE(constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as partition_dtrange,\n COALESCE((dt_constraint(edt.oid)).dt, constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as constraint_dtrange,\n COALESCE((dt_constraint(edt.oid)).edt, tstzrange('-infinity', 'infinity','[]')) as constraint_edtrange\nFROM\n pg_partition_tree('items')\n JOIN pg_class c ON (relid::regclass = c.oid)\n JOIN pg_class parent ON (parentrelid::regclass = parent.oid AND isleaf)\n LEFT JOIN pg_constraint edt ON (conrelid=c.oid AND contype='c')\nWHERE isleaf\n;\n\nCREATE OR REPLACE VIEW partitions_view AS\nSELECT\n (parse_ident(relid::text))[cardinality(parse_ident(relid::text))] as partition,\n replace(replace(CASE WHEN level = 1 THEN pg_get_expr(c.relpartbound, c.oid)\n ELSE pg_get_expr(parent.relpartbound, parent.oid)\n END, 'FOR VALUES IN (''',''), ''')','') AS collection,\n level,\n c.reltuples,\n c.relhastriggers,\n COALESCE(pgstac.constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as partition_dtrange,\n COALESCE((pgstac.dt_constraint(edt.oid)).dt, pgstac.constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as constraint_dtrange,\n COALESCE((pgstac.dt_constraint(edt.oid)).edt, tstzrange('-infinity', 'infinity','[]')) as constraint_edtrange,\n dtrange,\n edtrange,\n spatial,\n last_updated\nFROM\n pg_partition_tree('pgstac.items')\n JOIN pg_class c ON (relid::regclass = c.oid)\n JOIN pg_class parent ON (parentrelid::regclass = parent.oid AND isleaf)\n LEFT JOIN pg_constraint edt ON (conrelid=c.oid AND contype='c')\n LEFT JOIN pgstac.partition_stats ON ((parse_ident(relid::text))[cardinality(parse_ident(relid::text))]=partition)\nWHERE isleaf\n;\n\nCREATE MATERIALIZED VIEW partitions AS\nSELECT * FROM partitions_view;\nCREATE UNIQUE INDEX ON partitions (partition);\n\nCREATE MATERIALIZED VIEW partition_steps AS\nSELECT\n partition as name,\n date_trunc('month',lower(partition_dtrange)) as sdate,\n date_trunc('month', upper(partition_dtrange)) + '1 month'::interval as edate\n FROM partitions_view WHERE partition_dtrange IS NOT NULL AND partition_dtrange != 'empty'::tstzrange\n ORDER BY dtrange ASC\n;\n\n\nCREATE OR REPLACE FUNCTION update_partition_stats_q(_partition text, istrigger boolean default false) RETURNS VOID AS $$\nDECLARE\nBEGIN\n PERFORM run_or_queue(\n format('SELECT update_partition_stats(%L, %L);', _partition, istrigger)\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION update_partition_stats(_partition text, istrigger boolean default false) RETURNS VOID AS $$\nDECLARE\n dtrange tstzrange;\n edtrange tstzrange;\n cdtrange tstzrange;\n cedtrange tstzrange;\n extent geometry;\n collection text;\nBEGIN\n RAISE NOTICE 'Updating stats for %.', _partition;\n EXECUTE format(\n $q$\n SELECT\n tstzrange(min(datetime), max(datetime),'[]'),\n tstzrange(min(end_datetime), max(end_datetime), '[]')\n FROM %I\n $q$,\n _partition\n ) INTO dtrange, edtrange;\n extent := st_estimatedextent('pgstac', _partition, 'geometry');\n RAISE DEBUG 'Estimated Extent: %', extent;\n INSERT INTO partition_stats (partition, dtrange, edtrange, spatial, last_updated)\n SELECT _partition, dtrange, edtrange, extent, now()\n ON CONFLICT (partition) DO\n UPDATE SET\n dtrange=EXCLUDED.dtrange,\n edtrange=EXCLUDED.edtrange,\n spatial=EXCLUDED.spatial,\n last_updated=EXCLUDED.last_updated\n ;\n\n SELECT\n constraint_dtrange, constraint_edtrange, pv.collection\n INTO cdtrange, cedtrange, collection\n FROM partitions_view pv WHERE partition = _partition;\n REFRESH MATERIALIZED VIEW partitions;\n REFRESH MATERIALIZED VIEW partition_steps;\n\n\n RAISE NOTICE 'Checking if we need to modify constraints...';\n RAISE NOTICE 'cdtrange: % dtrange: % cedtrange: % edtrange: %',cdtrange, dtrange, cedtrange, edtrange;\n IF\n (cdtrange IS DISTINCT FROM dtrange OR edtrange IS DISTINCT FROM cedtrange)\n AND NOT istrigger\n THEN\n RAISE NOTICE 'Modifying Constraints';\n RAISE NOTICE 'Existing % %', cdtrange, cedtrange;\n RAISE NOTICE 'New % %', dtrange, edtrange;\n PERFORM drop_table_constraints(_partition);\n PERFORM create_table_constraints(_partition, dtrange, edtrange);\n REFRESH MATERIALIZED VIEW partitions;\n REFRESH MATERIALIZED VIEW partition_steps;\n END IF;\n RAISE NOTICE 'Checking if we need to update collection extents.';\n IF get_setting_bool('update_collection_extent') THEN\n RAISE NOTICE 'updating collection extent for %', collection;\n PERFORM run_or_queue(format($q$\n UPDATE collections\n SET content = jsonb_set_lax(\n content,\n '{extent}'::text[],\n collection_extent(%L, FALSE),\n true,\n 'use_json_null'\n ) WHERE id=%L\n ;\n $q$, collection, collection));\n ELSE\n RAISE NOTICE 'Not updating collection extent for %', collection;\n END IF;\n\nEND;\n$$ LANGUAGE PLPGSQL STRICT SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION partition_name( IN collection text, IN dt timestamptz, OUT partition_name text, OUT partition_range tstzrange) AS $$\nDECLARE\n c RECORD;\n parent_name text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n parent_name := format('_items_%s', c.key);\n\n\n IF c.partition_trunc = 'year' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYY'));\n ELSIF c.partition_trunc = 'month' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYYMM'));\n ELSE\n partition_name := parent_name;\n partition_range := tstzrange('-infinity'::timestamptz, 'infinity'::timestamptz, '[]');\n END IF;\n IF partition_range IS NULL THEN\n partition_range := tstzrange(\n date_trunc(c.partition_trunc::text, dt),\n date_trunc(c.partition_trunc::text, dt) + concat('1 ', c.partition_trunc)::interval\n );\n END IF;\n RETURN;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION drop_table_constraints(t text) RETURNS text AS $$\nDECLARE\n q text;\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM partitions_view WHERE partition=t) THEN\n RETURN NULL;\n END IF;\n FOR q IN SELECT FORMAT(\n $q$\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n $q$,\n t,\n conname\n ) FROM pg_constraint\n WHERE conrelid=t::regclass::oid AND contype='c'\n LOOP\n EXECUTE q;\n END LOOP;\n RETURN t;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE OR REPLACE FUNCTION create_table_constraints(t text, _dtrange tstzrange, _edtrange tstzrange) RETURNS text AS $$\nDECLARE\n q text;\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM partitions_view WHERE partition=t) THEN\n RETURN NULL;\n END IF;\n RAISE NOTICE 'Creating Table Constraints for % % %', t, _dtrange, _edtrange;\n IF _dtrange = 'empty' AND _edtrange = 'empty' THEN\n q :=format(\n $q$\n DO $block$\n BEGIN\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n ALTER TABLE %I\n ADD CONSTRAINT %I\n CHECK (((datetime IS NULL) AND (end_datetime IS NULL))) NOT VALID\n ;\n ALTER TABLE %I\n VALIDATE CONSTRAINT %I\n ;\n\n\n\n EXCEPTION WHEN others THEN\n RAISE WARNING '%%, Issue Altering Constraints. Please run update_partition_stats(%I)', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n $block$;\n $q$,\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t),\n t\n );\n ELSE\n q :=format(\n $q$\n DO $block$\n BEGIN\n\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n ALTER TABLE %I\n ADD CONSTRAINT %I\n CHECK (\n (datetime >= %L)\n AND (datetime <= %L)\n AND (end_datetime >= %L)\n AND (end_datetime <= %L)\n ) NOT VALID\n ;\n ALTER TABLE %I\n VALIDATE CONSTRAINT %I\n ;\n\n\n\n EXCEPTION WHEN others THEN\n RAISE WARNING '%%, Issue Altering Constraints. Please run update_partition_stats(%I)', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n $block$;\n $q$,\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t),\n lower(_dtrange),\n upper(_dtrange),\n lower(_edtrange),\n upper(_edtrange),\n t,\n format('%s_dt', t),\n t\n );\n END IF;\n PERFORM run_or_queue(q);\n RETURN t;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION check_partition(\n _collection text,\n _dtrange tstzrange,\n _edtrange tstzrange\n) RETURNS text AS $$\nDECLARE\n c RECORD;\n pm RECORD;\n _partition_name text;\n _partition_dtrange tstzrange;\n _constraint_dtrange tstzrange;\n _constraint_edtrange tstzrange;\n q text;\n deferrable_q text;\n err_context text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=_collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', _collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n\n IF c.partition_trunc IS NOT NULL THEN\n _partition_dtrange := tstzrange(\n date_trunc(c.partition_trunc, lower(_dtrange)),\n date_trunc(c.partition_trunc, lower(_dtrange)) + (concat('1 ', c.partition_trunc))::interval,\n '[)'\n );\n ELSE\n _partition_dtrange := '[-infinity, infinity]'::tstzrange;\n END IF;\n\n IF NOT _partition_dtrange @> _dtrange THEN\n RAISE EXCEPTION 'dtrange % is greater than the partition size % for collection %', _dtrange, c.partition_trunc, _collection;\n END IF;\n\n\n IF c.partition_trunc = 'year' THEN\n _partition_name := format('_items_%s_%s', c.key, to_char(lower(_partition_dtrange),'YYYY'));\n ELSIF c.partition_trunc = 'month' THEN\n _partition_name := format('_items_%s_%s', c.key, to_char(lower(_partition_dtrange),'YYYYMM'));\n ELSE\n _partition_name := format('_items_%s', c.key);\n END IF;\n\n SELECT * INTO pm FROM partition_sys_meta WHERE collection=_collection AND partition_dtrange @> _dtrange;\n IF FOUND THEN\n RAISE NOTICE '% % %', _edtrange, _dtrange, pm;\n _constraint_edtrange :=\n tstzrange(\n least(\n lower(_edtrange),\n nullif(lower(pm.constraint_edtrange), '-infinity')\n ),\n greatest(\n upper(_edtrange),\n nullif(upper(pm.constraint_edtrange), 'infinity')\n ),\n '[]'\n );\n _constraint_dtrange :=\n tstzrange(\n least(\n lower(_dtrange),\n nullif(lower(pm.constraint_dtrange), '-infinity')\n ),\n greatest(\n upper(_dtrange),\n nullif(upper(pm.constraint_dtrange), 'infinity')\n ),\n '[]'\n );\n\n IF pm.constraint_edtrange @> _edtrange AND pm.constraint_dtrange @> _dtrange THEN\n RETURN pm.partition;\n ELSE\n PERFORM drop_table_constraints(_partition_name);\n END IF;\n ELSE\n _constraint_edtrange := _edtrange;\n _constraint_dtrange := _dtrange;\n END IF;\n RAISE NOTICE 'EXISTING CONSTRAINTS % %, NEW % %', pm.constraint_dtrange, pm.constraint_edtrange, _constraint_dtrange, _constraint_edtrange;\n RAISE NOTICE 'Creating partition % %', _partition_name, _partition_dtrange;\n IF c.partition_trunc IS NULL THEN\n q := format(\n $q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n GRANT ALL ON %I to pgstac_ingest;\n $q$,\n _partition_name,\n _collection,\n concat(_partition_name,'_pk'),\n _partition_name,\n _partition_name\n );\n ELSE\n q := format(\n $q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L) PARTITION BY RANGE (datetime);\n CREATE TABLE IF NOT EXISTS %I partition OF %I FOR VALUES FROM (%L) TO (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n GRANT ALL ON %I TO pgstac_ingest;\n $q$,\n format('_items_%s', c.key),\n _collection,\n _partition_name,\n format('_items_%s', c.key),\n lower(_partition_dtrange),\n upper(_partition_dtrange),\n format('%s_pk', _partition_name),\n _partition_name,\n _partition_name\n );\n END IF;\n\n BEGIN\n EXECUTE q;\n EXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', _partition_name;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\n END;\n PERFORM maintain_partitions(_partition_name);\n PERFORM update_partition_stats_q(_partition_name, true);\n REFRESH MATERIALIZED VIEW partitions;\n REFRESH MATERIALIZED VIEW partition_steps;\n RETURN _partition_name;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION repartition(_collection text, _partition_trunc text, triggered boolean DEFAULT FALSE) RETURNS text AS $$\nDECLARE\n c RECORD;\n q text;\n from_trunc text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=_collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', _collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n IF triggered THEN\n RAISE NOTICE 'Converting % to % partitioning via Trigger', _collection, _partition_trunc;\n ELSE\n RAISE NOTICE 'Converting % from using % to % partitioning', _collection, c.partition_trunc, _partition_trunc;\n IF c.partition_trunc IS NOT DISTINCT FROM _partition_trunc THEN\n RAISE NOTICE 'Collection % already set to use partition by %', _collection, _partition_trunc;\n RETURN _collection;\n END IF;\n END IF;\n\n IF EXISTS (SELECT 1 FROM partitions_view WHERE collection=_collection LIMIT 1) THEN\n EXECUTE format(\n $q$\n CREATE TEMP TABLE changepartitionstaging ON COMMIT DROP AS SELECT * FROM %I;\n DROP TABLE IF EXISTS %I CASCADE;\n WITH p AS (\n SELECT\n collection,\n CASE\n WHEN %L IS NULL THEN '-infinity'::timestamptz\n ELSE date_trunc(%L, datetime)\n END as d,\n tstzrange(min(datetime),max(datetime),'[]') as dtrange,\n tstzrange(min(end_datetime),max(end_datetime),'[]') as edtrange\n FROM changepartitionstaging\n GROUP BY 1,2\n ) SELECT check_partition(collection, dtrange, edtrange) FROM p;\n INSERT INTO items SELECT * FROM changepartitionstaging;\n DROP TABLE changepartitionstaging;\n $q$,\n concat('_items_', c.key),\n concat('_items_', c.key),\n c.partition_trunc,\n c.partition_trunc\n );\n END IF;\n RETURN _collection;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE OR REPLACE FUNCTION collections_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n q text;\n partition_name text := format('_items_%s', NEW.key);\n partition_exists boolean := false;\n partition_empty boolean := true;\n err_context text;\n loadtemp boolean := FALSE;\nBEGIN\n RAISE NOTICE 'Collection Trigger. % %', NEW.id, NEW.key;\n IF TG_OP = 'UPDATE' AND NEW.partition_trunc IS DISTINCT FROM OLD.partition_trunc THEN\n PERFORM repartition(NEW.id, NEW.partition_trunc, TRUE);\n END IF;\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER collections_trigger AFTER\nINSERT\nOR\nUPDATE ON collections\nFOR EACH ROW EXECUTE FUNCTION collections_trigger_func();\n\nCREATE OR REPLACE FUNCTION chunker(\n IN _where text,\n OUT s timestamptz,\n OUT e timestamptz\n) RETURNS SETOF RECORD AS $$\nDECLARE\n explain jsonb;\nBEGIN\n IF _where IS NULL THEN\n _where := ' TRUE ';\n END IF;\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s;', _where)\n INTO explain;\n RAISE DEBUG 'EXPLAIN: %', explain;\n\n RETURN QUERY\n WITH t AS (\n SELECT j->>0 as p FROM\n jsonb_path_query(\n explain,\n 'strict $.**.\"Relation Name\" ? (@ != null)'\n ) j\n ),\n parts AS (\n SELECT sdate, edate FROM t JOIN partition_steps ON (t.p = name)\n ),\n times AS (\n SELECT sdate FROM parts\n UNION\n SELECT edate FROM parts\n ),\n uniq AS (\n SELECT DISTINCT sdate FROM times ORDER BY sdate\n ),\n last AS (\n SELECT sdate, lead(sdate, 1) over () as edate FROM uniq\n )\n SELECT sdate, edate FROM last WHERE edate IS NOT NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION partition_queries(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN partitions text[] DEFAULT NULL\n) RETURNS SETOF text AS $$\nDECLARE\n query text;\n sdate timestamptz;\n edate timestamptz;\nBEGIN\nIF _where IS NULL OR trim(_where) = '' THEN\n _where = ' TRUE ';\nEND IF;\nRAISE NOTICE 'Getting chunks for % %', _where, _orderby;\nIF _orderby ILIKE 'datetime d%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 DESC LOOP\n RETURN NEXT format($q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n $q$,\n sdate,\n edate,\n _where,\n _orderby\n );\n END LOOP;\nELSIF _orderby ILIKE 'datetime a%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 ASC LOOP\n RETURN NEXT format($q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n $q$,\n sdate,\n edate,\n _where,\n _orderby\n );\n END LOOP;\nELSE\n query := format($q$\n SELECT * FROM items\n WHERE %s\n ORDER BY %s\n $q$, _where, _orderby\n );\n\n RETURN NEXT query;\n RETURN;\nEND IF;\n\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION partition_query_view(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN _limit int DEFAULT 10\n) RETURNS text AS $$\n WITH p AS (\n SELECT * FROM partition_queries(_where, _orderby) p\n )\n SELECT\n CASE WHEN EXISTS (SELECT 1 FROM p) THEN\n (SELECT format($q$\n SELECT * FROM (\n %s\n ) total LIMIT %s\n $q$,\n string_agg(\n format($q$ SELECT * FROM ( %s ) AS sub $q$, p),\n '\n UNION ALL\n '\n ),\n _limit\n ))\n ELSE NULL\n END FROM p;\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION q_to_tsquery (input text)\n RETURNS tsquery\n AS $$\nDECLARE\n processed_text text;\n temp_text text;\n quote_array text[];\n placeholder text := '@QUOTE@';\nBEGIN\n -- Extract all quoted phrases and store in array\n quote_array := regexp_matches(input, '\"[^\"]*\"', 'g');\n\n -- Replace each quoted part with a unique placeholder if there are any quoted phrases\n IF array_length(quote_array, 1) IS NOT NULL THEN\n processed_text := input;\n FOR i IN array_lower(quote_array, 1) .. array_upper(quote_array, 1) LOOP\n processed_text := replace(processed_text, quote_array[i], placeholder || i || placeholder);\n END LOOP;\n ELSE\n processed_text := input;\n END IF;\n\n -- Replace non-quoted text using regular expressions\n\n -- , -> |\n processed_text := regexp_replace(processed_text, ',(?=(?:[^\"]*\"[^\"]*\")*[^\"]*$)', ' | ', 'g');\n\n -- and -> &\n processed_text := regexp_replace(processed_text, '\\s+AND\\s+', ' & ', 'gi');\n\n -- or -> |\n processed_text := regexp_replace(processed_text, '\\s+OR\\s+', ' | ', 'gi');\n\n -- +term -> & term\n processed_text := regexp_replace(processed_text, '\\+([a-zA-Z0-9_]+)', '& \\1', 'g');\n\n -- -term -> ! term\n processed_text := regexp_replace(processed_text, '\\-([a-zA-Z0-9_]+)', '& ! \\1', 'g');\n\n -- Replace placeholders back with quoted phrases if there were any\n IF array_length(quote_array, 1) IS NOT NULL THEN\n FOR i IN array_lower(quote_array, 1) .. array_upper(quote_array, 1) LOOP\n processed_text := replace(processed_text, placeholder || i || placeholder, '''' || substring(quote_array[i] from 2 for length(quote_array[i]) - 2) || '''');\n END LOOP;\n END IF;\n\n -- Print processed_text to the console for debugging purposes\n RAISE NOTICE 'processed_text: %', processed_text;\n\n RETURN to_tsquery(processed_text);\nEND;\n$$\nLANGUAGE plpgsql;\n\n\nCREATE OR REPLACE FUNCTION stac_search_to_where(j jsonb) RETURNS text AS $$\nDECLARE\n where_segments text[];\n _where text;\n dtrange tstzrange;\n collections text[];\n geom geometry;\n sdate timestamptz;\n edate timestamptz;\n filterlang text;\n filter jsonb := j->'filter';\n ft_query tsquery;\nBEGIN\n IF j ? 'ids' THEN\n where_segments := where_segments || format('id = ANY (%L) ', to_text_array(j->'ids'));\n END IF;\n\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n where_segments := where_segments || format('collection = ANY (%L) ', collections);\n END IF;\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n\n where_segments := where_segments || format(' datetime <= %L::timestamptz AND end_datetime >= %L::timestamptz ',\n edate,\n sdate\n );\n END IF;\n\n IF j ? 'q' THEN\n ft_query := q_to_tsquery(j->>'q');\n where_segments := where_segments || format(\n $quote$\n (\n to_tsvector('english', content->'properties'->>'description') ||\n to_tsvector('english', coalesce(content->'properties'->>'title', '')) ||\n to_tsvector('english', coalesce(content->'properties'->>'keywords', ''))\n ) @@ %L\n $quote$,\n ft_query\n );\n END IF;\n\n geom := stac_geom(j);\n IF geom IS NOT NULL THEN\n where_segments := where_segments || format('st_intersects(geometry, %L)',geom);\n END IF;\n\n filterlang := COALESCE(\n j->>'filter-lang',\n get_setting('default_filter_lang', j->'conf')\n );\n IF NOT filter @? '$.**.op' THEN\n filterlang := 'cql-json';\n END IF;\n\n IF filterlang NOT IN ('cql-json','cql2-json') AND j ? 'filter' THEN\n RAISE EXCEPTION '% is not a supported filter-lang. Please use cql-json or cql2-json.', filterlang;\n END IF;\n\n IF j ? 'query' AND j ? 'filter' THEN\n RAISE EXCEPTION 'Can only use either query or filter at one time.';\n END IF;\n\n IF j ? 'query' THEN\n filter := query_to_cql2(j->'query');\n ELSIF filterlang = 'cql-json' THEN\n filter := cql1_to_cql2(filter);\n END IF;\n RAISE NOTICE 'FILTER: %', filter;\n where_segments := where_segments || cql2_query(filter);\n IF cardinality(where_segments) < 1 THEN\n RETURN ' TRUE ';\n END IF;\n\n _where := array_to_string(array_remove(where_segments, NULL), ' AND ');\n\n IF _where IS NULL OR BTRIM(_where) = '' THEN\n RETURN ' TRUE ';\n END IF;\n RETURN _where;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION parse_sort_dir(_dir text, reverse boolean default false) RETURNS text AS $$\n WITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n ) SELECT\n CASE\n WHEN NOT reverse THEN d\n WHEN d = 'ASC' THEN 'DESC'\n WHEN d = 'DESC' THEN 'ASC'\n END\n FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION sort_dir_to_op(_dir text, prev boolean default false) RETURNS text AS $$\n WITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n ) SELECT\n CASE\n WHEN d = 'ASC' AND prev THEN '<='\n WHEN d = 'DESC' AND prev THEN '>='\n WHEN d = 'ASC' THEN '>='\n WHEN d = 'DESC' THEN '<='\n END\n FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION sort_sqlorderby(\n _search jsonb DEFAULT NULL,\n reverse boolean DEFAULT FALSE\n) RETURNS text AS $$\n WITH sortby AS (\n SELECT coalesce(_search->'sortby','[{\"field\":\"datetime\", \"direction\":\"desc\"}]') as sort\n ), withid AS (\n SELECT CASE\n WHEN sort @? '$[*] ? (@.field == \"id\")' THEN sort\n ELSE sort || '[{\"field\":\"id\", \"direction\":\"desc\"}]'::jsonb\n END as sort\n FROM sortby\n ), withid_rows AS (\n SELECT jsonb_array_elements(sort) as value FROM withid\n ),sorts AS (\n SELECT\n coalesce(\n (queryable(value->>'field')).expression\n ) as key,\n parse_sort_dir(value->>'direction', reverse) as dir\n FROM withid_rows\n )\n SELECT array_to_string(\n array_agg(concat(key, ' ', dir)),\n ', '\n ) FROM sorts;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_sort_dir(sort_item jsonb) RETURNS text AS $$\n SELECT CASE WHEN sort_item->>'direction' ILIKE 'desc%' THEN 'DESC' ELSE 'ASC' END;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION get_token_val_str(\n _field text,\n _item items\n) RETURNS text AS $$\nDECLARE\n q text;\n literal text;\nBEGIN\n q := format($q$ SELECT quote_literal(%s) FROM (SELECT $1.*) as r;$q$, _field);\n EXECUTE q INTO literal USING _item;\n RETURN literal;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n\nCREATE OR REPLACE FUNCTION get_token_record(IN _token text, OUT prev BOOLEAN, OUT item items) RETURNS RECORD AS $$\nDECLARE\n _itemid text := _token;\n _collectionid text;\nBEGIN\n IF _token IS NULL THEN\n RETURN;\n END IF;\n RAISE NOTICE 'Looking for token: %', _token;\n prev := FALSE;\n IF _token ILIKE 'prev:%' THEN\n _itemid := replace(_token, 'prev:','');\n prev := TRUE;\n ELSIF _token ILIKE 'next:%' THEN\n _itemid := replace(_token, 'next:', '');\n END IF;\n SELECT id INTO _collectionid FROM collections WHERE _itemid LIKE concat(id,':%');\n IF FOUND THEN\n _itemid := replace(_itemid, concat(_collectionid,':'), '');\n SELECT * INTO item FROM items WHERE id=_itemid AND collection=_collectionid;\n ELSE\n SELECT * INTO item FROM items WHERE id=_itemid;\n END IF;\n IF item IS NULL THEN\n RAISE EXCEPTION 'Could not find item using token: % item: % collection: %', _token, _itemid, _collectionid;\n END IF;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\n\nCREATE OR REPLACE FUNCTION get_token_filter(\n _sortby jsonb DEFAULT '[{\"field\":\"datetime\",\"direction\":\"desc\"}]'::jsonb,\n token_item items DEFAULT NULL,\n prev boolean DEFAULT FALSE,\n inclusive boolean DEFAULT FALSE\n) RETURNS text AS $$\nDECLARE\n ltop text := '<';\n gtop text := '>';\n dir text;\n sort record;\n orfilter text := '';\n orfilters text[] := '{}'::text[];\n andfilters text[] := '{}'::text[];\n output text;\n token_where text;\nBEGIN\n IF _sortby IS NULL OR _sortby = '[]'::jsonb THEN\n _sortby := '[{\"field\":\"datetime\",\"direction\":\"desc\"}]'::jsonb;\n END IF;\n _sortby := _sortby || jsonb_build_object('field','id','direction',_sortby->0->>'direction');\n RAISE NOTICE 'Getting Token Filter. % %', _sortby, token_item;\n IF inclusive THEN\n orfilters := orfilters || format('( id=%L AND collection=%L )' , token_item.id, token_item.collection);\n END IF;\n\n FOR sort IN\n WITH s1 AS (\n SELECT\n _row,\n (queryable(value->>'field')).expression as _field,\n (value->>'field' = 'id') as _isid,\n get_sort_dir(value) as _dir\n FROM jsonb_array_elements(_sortby)\n WITH ORDINALITY AS t(value, _row)\n )\n SELECT\n _row,\n _field,\n _dir,\n get_token_val_str(_field, token_item) as _val\n FROM s1\n WHERE _row <= (SELECT min(_row) FROM s1 WHERE _isid)\n LOOP\n orfilter := NULL;\n RAISE NOTICE 'SORT: %', sort;\n IF sort._val IS NOT NULL AND ((prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC')) THEN\n orfilter := format($f$(\n (%s %s %s) OR (%s IS NULL)\n )$f$,\n sort._field,\n ltop,\n sort._val,\n sort._val\n );\n ELSIF sort._val IS NULL AND ((prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC')) THEN\n RAISE NOTICE '< but null';\n orfilter := format('%s IS NOT NULL', sort._field);\n ELSIF sort._val IS NULL THEN\n RAISE NOTICE '> but null';\n ELSE\n orfilter := format($f$(\n (%s %s %s) OR (%s IS NULL)\n )$f$,\n sort._field,\n gtop,\n sort._val,\n sort._field\n );\n END IF;\n RAISE NOTICE 'ORFILTER: %', orfilter;\n\n IF orfilter IS NOT NULL THEN\n IF sort._row = 1 THEN\n orfilters := orfilters || orfilter;\n ELSE\n orfilters := orfilters || format('(%s AND %s)', array_to_string(andfilters, ' AND '), orfilter);\n END IF;\n END IF;\n IF sort._val IS NOT NULL THEN\n andfilters := andfilters || format('%s = %s', sort._field, sort._val);\n ELSE\n andfilters := andfilters || format('%s IS NULL', sort._field);\n END IF;\n END LOOP;\n\n output := array_to_string(orfilters, ' OR ');\n\n token_where := concat('(',coalesce(output,'true'),')');\n IF trim(token_where) = '' THEN\n token_where := NULL;\n END IF;\n RAISE NOTICE 'TOKEN_WHERE: %',token_where;\n RETURN token_where;\n END;\n$$ LANGUAGE PLPGSQL SET transform_null_equals TO TRUE\n;\n\nCREATE OR REPLACE FUNCTION search_tohash(jsonb) RETURNS jsonb AS $$\n SELECT $1 - '{token,limit,context,includes,excludes}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION search_hash(jsonb, jsonb) RETURNS text AS $$\n SELECT md5(concat(search_tohash($1)::text,$2::text));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TABLE IF NOT EXISTS searches(\n hash text GENERATED ALWAYS AS (search_hash(search, metadata)) STORED PRIMARY KEY,\n search jsonb NOT NULL,\n _where text,\n orderby text,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n metadata jsonb DEFAULT '{}'::jsonb NOT NULL\n);\n\nCREATE TABLE IF NOT EXISTS search_wheres(\n id bigint generated always as identity primary key,\n _where text NOT NULL,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n statslastupdated timestamptz,\n estimated_count bigint,\n estimated_cost float,\n time_to_estimate float,\n total_count bigint,\n time_to_count float,\n partitions text[]\n);\n\nCREATE INDEX IF NOT EXISTS search_wheres_partitions ON search_wheres USING GIN (partitions);\nCREATE UNIQUE INDEX IF NOT EXISTS search_wheres_where ON search_wheres ((md5(_where)));\n\nCREATE OR REPLACE FUNCTION where_stats(\n inwhere text,\n updatestats boolean default false,\n conf jsonb default null\n) RETURNS search_wheres AS $$\nDECLARE\n t timestamptz;\n i interval;\n explain_json jsonb;\n partitions text[];\n sw search_wheres%ROWTYPE;\n inwhere_hash text := md5(inwhere);\n _context text := lower(context(conf));\n _stats_ttl interval := context_stats_ttl(conf);\n _estimated_cost_threshold float := context_estimated_cost(conf);\n _estimated_count_threshold int := context_estimated_count(conf);\n ro bool := pgstac.readonly(conf);\nBEGIN\n -- If updatestats is true then set ttl to 0\n IF updatestats THEN\n RAISE DEBUG 'Updatestats set to TRUE, setting TTL to 0';\n _stats_ttl := '0'::interval;\n END IF;\n\n -- If we don't need to calculate context, just return\n IF _context = 'off' THEN\n sw._where = inwhere;\n RETURN sw;\n END IF;\n\n -- Get any stats that we have.\n IF NOT ro THEN\n -- If there is a lock where another process is\n -- updating the stats, wait so that we don't end up calculating a bunch of times.\n SELECT * INTO sw FROM search_wheres WHERE md5(_where)=inwhere_hash FOR UPDATE;\n ELSE\n SELECT * INTO sw FROM search_wheres WHERE md5(_where)=inwhere_hash;\n END IF;\n\n -- If there is a cached row, figure out if we need to update\n IF\n sw IS NOT NULL\n AND sw.statslastupdated IS NOT NULL\n AND sw.total_count IS NOT NULL\n AND now() - sw.statslastupdated <= _stats_ttl\n THEN\n -- we have a cached row with data that is within our ttl\n RAISE DEBUG 'Stats present in table and lastupdated within ttl: %', sw;\n IF NOT ro THEN\n RAISE DEBUG 'Updating search_wheres only bumping lastused and usecount';\n UPDATE search_wheres SET\n lastused = now(),\n usecount = search_wheres.usecount + 1\n WHERE md5(_where) = inwhere_hash\n RETURNING * INTO sw;\n END IF;\n RAISE DEBUG 'Returning cached counts. %', sw;\n RETURN sw;\n END IF;\n\n -- Calculate estimated cost and rows\n -- Use explain to get estimated count/cost\n IF sw.estimated_count IS NULL OR sw.estimated_cost IS NULL THEN\n RAISE DEBUG 'Calculating estimated stats';\n t := clock_timestamp();\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s', inwhere)\n INTO explain_json;\n RAISE DEBUG 'Time for just the explain: %', clock_timestamp() - t;\n i := clock_timestamp() - t;\n\n sw.estimated_count := explain_json->0->'Plan'->'Plan Rows';\n sw.estimated_cost := explain_json->0->'Plan'->'Total Cost';\n sw.time_to_estimate := extract(epoch from i);\n END IF;\n\n RAISE DEBUG 'ESTIMATED_COUNT: %, THRESHOLD %', sw.estimated_count, _estimated_count_threshold;\n RAISE DEBUG 'ESTIMATED_COST: %, THRESHOLD %', sw.estimated_cost, _estimated_cost_threshold;\n\n -- If context is set to auto and the costs are within the threshold return the estimated costs\n IF\n _context = 'auto'\n AND sw.estimated_count >= _estimated_count_threshold\n AND sw.estimated_cost >= _estimated_cost_threshold\n THEN\n IF NOT ro THEN\n INSERT INTO search_wheres (\n _where,\n lastused,\n usecount,\n statslastupdated,\n estimated_count,\n estimated_cost,\n time_to_estimate,\n total_count,\n time_to_count\n ) VALUES (\n inwhere,\n now(),\n 1,\n now(),\n sw.estimated_count,\n sw.estimated_cost,\n sw.time_to_estimate,\n null,\n null\n ) ON CONFLICT ((md5(_where)))\n DO UPDATE SET\n lastused = EXCLUDED.lastused,\n usecount = search_wheres.usecount + 1,\n statslastupdated = EXCLUDED.statslastupdated,\n estimated_count = EXCLUDED.estimated_count,\n estimated_cost = EXCLUDED.estimated_cost,\n time_to_estimate = EXCLUDED.time_to_estimate,\n total_count = EXCLUDED.total_count,\n time_to_count = EXCLUDED.time_to_count\n RETURNING * INTO sw;\n END IF;\n RAISE DEBUG 'Estimates are within thresholds, returning estimates. %', sw;\n RETURN sw;\n END IF;\n\n -- Calculate Actual Count\n t := clock_timestamp();\n RAISE NOTICE 'Calculating actual count...';\n EXECUTE format(\n 'SELECT count(*) FROM items WHERE %s',\n inwhere\n ) INTO sw.total_count;\n i := clock_timestamp() - t;\n RAISE NOTICE 'Actual Count: % -- %', sw.total_count, i;\n sw.time_to_count := extract(epoch FROM i);\n\n IF NOT ro THEN\n INSERT INTO search_wheres (\n _where,\n lastused,\n usecount,\n statslastupdated,\n estimated_count,\n estimated_cost,\n time_to_estimate,\n total_count,\n time_to_count\n ) VALUES (\n inwhere,\n now(),\n 1,\n now(),\n sw.estimated_count,\n sw.estimated_cost,\n sw.time_to_estimate,\n sw.total_count,\n sw.time_to_count\n ) ON CONFLICT ((md5(_where)))\n DO UPDATE SET\n lastused = EXCLUDED.lastused,\n usecount = search_wheres.usecount + 1,\n statslastupdated = EXCLUDED.statslastupdated,\n estimated_count = EXCLUDED.estimated_count,\n estimated_cost = EXCLUDED.estimated_cost,\n time_to_estimate = EXCLUDED.time_to_estimate,\n total_count = EXCLUDED.total_count,\n time_to_count = EXCLUDED.time_to_count\n RETURNING * INTO sw;\n END IF;\n RAISE DEBUG 'Returning with actual count. %', sw;\n RETURN sw;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION search_query(\n _search jsonb = '{}'::jsonb,\n updatestats boolean = false,\n _metadata jsonb = '{}'::jsonb\n) RETURNS searches AS $$\nDECLARE\n search searches%ROWTYPE;\n cached_search searches%ROWTYPE;\n pexplain jsonb;\n t timestamptz;\n i interval;\n doupdate boolean := FALSE;\n insertfound boolean := FALSE;\n ro boolean := pgstac.readonly();\n found_search text;\nBEGIN\n RAISE NOTICE 'SEARCH: %', _search;\n -- Calculate hash, where clause, and order by statement\n search.search := _search;\n search.metadata := _metadata;\n search.hash := search_hash(_search, _metadata);\n search._where := stac_search_to_where(_search);\n search.orderby := sort_sqlorderby(_search);\n search.lastused := now();\n search.usecount := 1;\n\n -- If we are in read only mode, directly return search\n IF ro THEN\n RETURN search;\n END IF;\n\n RAISE NOTICE 'Updating Statistics for search: %s', search;\n -- Update statistics for times used and and when last used\n -- If the entry is locked, rather than waiting, skip updating the stats\n INSERT INTO searches (search, lastused, usecount, metadata)\n VALUES (search.search, now(), 1, search.metadata)\n ON CONFLICT DO NOTHING\n RETURNING * INTO cached_search\n ;\n\n IF NOT FOUND OR cached_search IS NULL THEN\n UPDATE searches SET\n lastused = now(),\n usecount = searches.usecount + 1\n WHERE hash = (\n SELECT hash FROM searches WHERE hash=search.hash FOR UPDATE SKIP LOCKED\n )\n RETURNING * INTO cached_search\n ;\n END IF;\n\n IF cached_search IS NOT NULL THEN\n cached_search._where = search._where;\n cached_search.orderby = search.orderby;\n RETURN cached_search;\n END IF;\n RETURN search;\n\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE OR REPLACE FUNCTION search_fromhash(\n _hash text\n) RETURNS searches AS $$\n SELECT * FROM search_query((SELECT search FROM searches WHERE hash=_hash LIMIT 1));\n$$ LANGUAGE SQL STRICT;\n\nCREATE OR REPLACE FUNCTION search_rows(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN partitions text[] DEFAULT NULL,\n IN _limit int DEFAULT 10\n) RETURNS SETOF items AS $$\nDECLARE\n base_query text;\n query text;\n sdate timestamptz;\n edate timestamptz;\n n int;\n records_left int := _limit;\n timer timestamptz := clock_timestamp();\n full_timer timestamptz := clock_timestamp();\nBEGIN\nIF _where IS NULL OR trim(_where) = '' THEN\n _where = ' TRUE ';\nEND IF;\nRAISE NOTICE 'Getting chunks for % %', _where, _orderby;\n\nbase_query := $q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n LIMIT %L\n$q$;\n\nIF _orderby ILIKE 'datetime d%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 DESC LOOP\n RAISE NOTICE 'Running Query for % to %. %', sdate, edate, age_ms(full_timer);\n query := format(\n base_query,\n sdate,\n edate,\n _where,\n _orderby,\n records_left\n );\n RAISE DEBUG 'QUERY: %', query;\n timer := clock_timestamp();\n RETURN QUERY EXECUTE query;\n\n GET DIAGNOSTICS n = ROW_COUNT;\n records_left := records_left - n;\n RAISE NOTICE 'Returned %/% Rows From % to %. % to go. Time: %ms', n, _limit, sdate, edate, records_left, age_ms(timer);\n timer := clock_timestamp();\n IF records_left <= 0 THEN\n RAISE NOTICE 'SEARCH_ROWS TOOK %ms', age_ms(full_timer);\n RETURN;\n END IF;\n END LOOP;\nELSIF _orderby ILIKE 'datetime a%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 ASC LOOP\n RAISE NOTICE 'Running Query for % to %. %', sdate, edate, age_ms(full_timer);\n query := format(\n base_query,\n sdate,\n edate,\n _where,\n _orderby,\n records_left\n );\n RAISE DEBUG 'QUERY: %', query;\n timer := clock_timestamp();\n RETURN QUERY EXECUTE query;\n\n GET DIAGNOSTICS n = ROW_COUNT;\n records_left := records_left - n;\n RAISE NOTICE 'Returned %/% Rows From % to %. % to go. Time: %ms', n, _limit, sdate, edate, records_left, age_ms(timer);\n timer := clock_timestamp();\n IF records_left <= 0 THEN\n RAISE NOTICE 'SEARCH_ROWS TOOK %ms', age_ms(full_timer);\n RETURN;\n END IF;\n END LOOP;\nELSE\n query := format($q$\n SELECT * FROM items\n WHERE %s\n ORDER BY %s\n LIMIT %L\n $q$, _where, _orderby, _limit\n );\n RAISE DEBUG 'QUERY: %', query;\n timer := clock_timestamp();\n RETURN QUERY EXECUTE query;\n RAISE NOTICE 'FULL QUERY TOOK %ms', age_ms(timer);\nEND IF;\nRAISE NOTICE 'SEARCH_ROWS TOOK %ms', age_ms(full_timer);\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE UNLOGGED TABLE format_item_cache(\n id text,\n collection text,\n fields text,\n hydrated bool,\n output jsonb,\n lastused timestamptz DEFAULT now(),\n usecount int DEFAULT 1,\n timetoformat float,\n PRIMARY KEY (collection, id, fields, hydrated)\n);\nCREATE INDEX ON format_item_cache (lastused);\n\nCREATE OR REPLACE FUNCTION format_item(_item items, _fields jsonb DEFAULT '{}', _hydrated bool DEFAULT TRUE) RETURNS jsonb AS $$\nDECLARE\n cache bool := get_setting_bool('format_cache');\n _output jsonb := null;\n t timestamptz := clock_timestamp();\nBEGIN\n IF cache THEN\n SELECT output INTO _output FROM format_item_cache\n WHERE id=_item.id AND collection=_item.collection AND fields=_fields::text AND hydrated=_hydrated;\n END IF;\n IF _output IS NULL THEN\n IF _hydrated THEN\n _output := content_hydrate(_item, _fields);\n ELSE\n _output := content_nonhydrated(_item, _fields);\n END IF;\n END IF;\n IF cache THEN\n INSERT INTO format_item_cache (id, collection, fields, hydrated, output, timetoformat)\n VALUES (_item.id, _item.collection, _fields::text, _hydrated, _output, age_ms(t))\n ON CONFLICT(collection, id, fields, hydrated) DO\n UPDATE\n SET lastused=now(), usecount = format_item_cache.usecount + 1\n ;\n END IF;\n RETURN _output;\n\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION search(_search jsonb = '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n searches searches%ROWTYPE;\n _where text;\n orderby text;\n search_where search_wheres%ROWTYPE;\n total_count bigint;\n token record;\n token_prev boolean;\n token_item items%ROWTYPE;\n token_where text;\n full_where text;\n init_ts timestamptz := clock_timestamp();\n timer timestamptz := clock_timestamp();\n hydrate bool := NOT (_search->'conf'->>'nohydrate' IS NOT NULL AND (_search->'conf'->>'nohydrate')::boolean = true);\n prev text;\n next text;\n context jsonb;\n collection jsonb;\n out_records jsonb;\n out_len int;\n _limit int := coalesce((_search->>'limit')::int, 10);\n _querylimit int;\n _fields jsonb := coalesce(_search->'fields', '{}'::jsonb);\n has_prev boolean := FALSE;\n has_next boolean := FALSE;\n links jsonb := '[]'::jsonb;\n base_url text:= concat(rtrim(base_url(_search->'conf'),'/'));\nBEGIN\n searches := search_query(_search);\n _where := searches._where;\n orderby := searches.orderby;\n search_where := where_stats(_where);\n total_count := coalesce(search_where.total_count, search_where.estimated_count);\n RAISE NOTICE 'SEARCH:TOKEN: %', _search->>'token';\n token := get_token_record(_search->>'token');\n RAISE NOTICE '***TOKEN: %', token;\n _querylimit := _limit + 1;\n IF token IS NOT NULL THEN\n token_prev := token.prev;\n token_item := token.item;\n token_where := get_token_filter(_search->'sortby', token_item, token_prev, FALSE);\n RAISE DEBUG 'TOKEN_WHERE: % (%ms from search start)', token_where, age_ms(timer);\n IF token_prev THEN -- if we are using a prev token, we know has_next is true\n RAISE DEBUG 'There is a previous token, so automatically setting has_next to true';\n has_next := TRUE;\n orderby := sort_sqlorderby(_search, TRUE);\n ELSE\n RAISE DEBUG 'There is a next token, so automatically setting has_prev to true';\n has_prev := TRUE;\n\n END IF;\n ELSE -- if there was no token, we know there is no prev\n RAISE DEBUG 'There is no token, so we know there is no prev. setting has_prev to false';\n has_prev := FALSE;\n END IF;\n\n full_where := concat_ws(' AND ', _where, token_where);\n RAISE NOTICE 'FULL WHERE CLAUSE: %', full_where;\n RAISE NOTICE 'Time to get counts and build query %', age_ms(timer);\n timer := clock_timestamp();\n\n IF hydrate THEN\n RAISE NOTICE 'Getting hydrated data.';\n ELSE\n RAISE NOTICE 'Getting non-hydrated data.';\n END IF;\n RAISE NOTICE 'CACHE SET TO %', get_setting_bool('format_cache');\n RAISE NOTICE 'Time to set hydration/formatting %', age_ms(timer);\n timer := clock_timestamp();\n SELECT jsonb_agg(format_item(i, _fields, hydrate)) INTO out_records\n FROM search_rows(\n full_where,\n orderby,\n search_where.partitions,\n _querylimit\n ) as i;\n\n RAISE NOTICE 'Time to fetch rows %', age_ms(timer);\n timer := clock_timestamp();\n\n\n IF token_prev THEN\n out_records := flip_jsonb_array(out_records);\n END IF;\n\n RAISE NOTICE 'Query returned % records.', jsonb_array_length(out_records);\n RAISE DEBUG 'TOKEN: % %', token_item.id, token_item.collection;\n RAISE DEBUG 'RECORD_1: % %', out_records->0->>'id', out_records->0->>'collection';\n RAISE DEBUG 'RECORD-1: % %', out_records->-1->>'id', out_records->-1->>'collection';\n\n -- REMOVE records that were from our token\n IF out_records->0->>'id' = token_item.id AND out_records->0->>'collection' = token_item.collection THEN\n out_records := out_records - 0;\n ELSIF out_records->-1->>'id' = token_item.id AND out_records->-1->>'collection' = token_item.collection THEN\n out_records := out_records - -1;\n END IF;\n\n out_len := jsonb_array_length(out_records);\n\n IF out_len = _limit + 1 THEN\n IF token_prev THEN\n has_prev := TRUE;\n out_records := out_records - 0;\n ELSE\n has_next := TRUE;\n out_records := out_records - -1;\n END IF;\n END IF;\n\n\n links := links || jsonb_build_object(\n 'rel', 'root',\n 'type', 'application/json',\n 'href', base_url\n ) || jsonb_build_object(\n 'rel', 'self',\n 'type', 'application/json',\n 'href', concat(base_url, '/search')\n );\n\n IF has_next THEN\n next := concat(out_records->-1->>'collection', ':', out_records->-1->>'id');\n RAISE NOTICE 'HAS NEXT | %', next;\n links := links || jsonb_build_object(\n 'rel', 'next',\n 'type', 'application/geo+json',\n 'method', 'GET',\n 'href', concat(base_url, '/search?token=next:', next)\n );\n END IF;\n\n IF has_prev THEN\n prev := concat(out_records->0->>'collection', ':', out_records->0->>'id');\n RAISE NOTICE 'HAS PREV | %', prev;\n links := links || jsonb_build_object(\n 'rel', 'prev',\n 'type', 'application/geo+json',\n 'method', 'GET',\n 'href', concat(base_url, '/search?token=prev:', prev)\n );\n END IF;\n\n RAISE NOTICE 'Time to get prev/next %', age_ms(timer);\n timer := clock_timestamp();\n\n\n collection := jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb),\n 'links', links\n );\n\n\n\n IF context(_search->'conf') != 'off' THEN\n collection := collection || jsonb_strip_nulls(jsonb_build_object(\n 'numberMatched', total_count,\n 'numberReturned', coalesce(jsonb_array_length(out_records), 0)\n ));\n ELSE\n collection := collection || jsonb_strip_nulls(jsonb_build_object(\n 'numberReturned', coalesce(jsonb_array_length(out_records), 0)\n ));\n END IF;\n\n IF get_setting_bool('timing', _search->'conf') THEN\n collection = collection || jsonb_build_object('timing', age_ms(init_ts));\n END IF;\n\n RAISE NOTICE 'Time to build final json %', age_ms(timer);\n timer := clock_timestamp();\n\n RAISE NOTICE 'Total Time: %', age_ms(current_timestamp);\n RAISE NOTICE 'RETURNING % records. NEXT: %. PREV: %', collection->>'numberReturned', collection->>'next', collection->>'prev';\n RETURN collection;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION search_cursor(_search jsonb = '{}'::jsonb) RETURNS refcursor AS $$\nDECLARE\n curs refcursor;\n searches searches%ROWTYPE;\n _where text;\n _orderby text;\n q text;\n\nBEGIN\n searches := search_query(_search);\n _where := searches._where;\n _orderby := searches.orderby;\n\n OPEN curs FOR\n WITH p AS (\n SELECT * FROM partition_queries(_where, _orderby) p\n )\n SELECT\n CASE WHEN EXISTS (SELECT 1 FROM p) THEN\n (SELECT format($q$\n SELECT * FROM (\n %s\n ) total\n $q$,\n string_agg(\n format($q$ SELECT * FROM ( %s ) AS sub $q$, p),\n '\n UNION ALL\n '\n )\n ))\n ELSE NULL\n END FROM p;\n RETURN curs;\nEND;\n$$ LANGUAGE PLPGSQL;\nCREATE OR REPLACE VIEW collections_asitems AS\nSELECT\n id,\n geometry,\n 'collections' AS collection,\n datetime,\n end_datetime,\n jsonb_build_object(\n 'properties', content - '{links,assets,stac_version,stac_extensions}',\n 'links', content->'links',\n 'assets', content->'assets',\n 'stac_version', content->'stac_version',\n 'stac_extensions', content->'stac_extensions'\n ) AS content,\n content as collectionjson\nFROM collections;\n\n\nCREATE OR REPLACE FUNCTION collection_search_matched(\n IN _search jsonb DEFAULT '{}'::jsonb,\n OUT matched bigint\n) RETURNS bigint AS $$\nDECLARE\n _where text := stac_search_to_where(_search);\nBEGIN\n EXECUTE format(\n $query$\n SELECT\n count(*)\n FROM\n collections_asitems\n WHERE %s\n ;\n $query$,\n _where\n ) INTO matched;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION collection_search_rows(\n _search jsonb DEFAULT '{}'::jsonb\n) RETURNS SETOF jsonb AS $$\nDECLARE\n _where text := stac_search_to_where(_search);\n _limit int := coalesce((_search->>'limit')::int, 10);\n _fields jsonb := coalesce(_search->'fields', '{}'::jsonb);\n _orderby text;\n _offset int := COALESCE((_search->>'offset')::int, 0);\nBEGIN\n _orderby := sort_sqlorderby(\n jsonb_build_object(\n 'sortby',\n coalesce(\n _search->'sortby',\n '[{\"field\": \"id\", \"direction\": \"asc\"}]'::jsonb\n )\n )\n );\n RETURN QUERY EXECUTE format(\n $query$\n SELECT\n jsonb_fields(collectionjson, %L) as c\n FROM\n collections_asitems\n WHERE %s\n ORDER BY %s\n LIMIT %L\n OFFSET %L\n ;\n $query$,\n _fields,\n _where,\n _orderby,\n _limit,\n _offset\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION collection_search(\n _search jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\nDECLARE\n out_records jsonb;\n number_matched bigint := collection_search_matched(_search);\n number_returned bigint;\n _limit int := coalesce((_search->>'limit')::float::int, 10);\n _offset int := coalesce((_search->>'offset')::float::int, 0);\n links jsonb := '[]';\n ret jsonb;\n base_url text:= concat(rtrim(base_url(_search->'conf'),'/'), '/collections');\n prevoffset int;\n nextoffset int;\nBEGIN\n SELECT\n coalesce(jsonb_agg(c), '[]')\n INTO out_records\n FROM collection_search_rows(_search) c;\n\n number_returned := jsonb_array_length(out_records);\n RAISE DEBUG 'nm: %, nr: %, l:%, o:%', number_matched, number_returned, _limit, _offset;\n\n\n\n IF _limit <= number_matched AND number_matched > 0 THEN --need to have paging links\n nextoffset := least(_offset + _limit, number_matched - 1);\n prevoffset := greatest(_offset - _limit, 0);\n\n IF _offset > 0 THEN\n links := links || jsonb_build_object(\n 'rel', 'prev',\n 'type', 'application/json',\n 'method', 'GET' ,\n 'href', base_url,\n 'body', jsonb_build_object('offset', prevoffset),\n 'merge', TRUE\n );\n END IF;\n\n IF (_offset + _limit < number_matched) THEN\n links := links || jsonb_build_object(\n 'rel', 'next',\n 'type', 'application/json',\n 'method', 'GET' ,\n 'href', base_url,\n 'body', jsonb_build_object('offset', nextoffset),\n 'merge', TRUE\n );\n END IF;\n\n END IF;\n\n ret := jsonb_build_object(\n 'collections', out_records,\n 'numberMatched', number_matched,\n 'numberReturned', number_returned,\n 'links', links\n );\n RETURN ret;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION tileenvelope(zoom int, x int, y int) RETURNS geometry AS $$\nWITH t AS (\n SELECT\n 20037508.3427892 as merc_max,\n -20037508.3427892 as merc_min,\n (2 * 20037508.3427892) / (2 ^ zoom) as tile_size\n)\nSELECT st_makeenvelope(\n merc_min + (tile_size * x),\n merc_max - (tile_size * (y + 1)),\n merc_min + (tile_size * (x + 1)),\n merc_max - (tile_size * y),\n 3857\n) FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;DROP FUNCTION IF EXISTS mercgrid;\n\n\nCREATE OR REPLACE FUNCTION ftime() RETURNS interval as $$\nSELECT age(clock_timestamp(), transaction_timestamp());\n$$ LANGUAGE SQL;\nSET SEARCH_PATH to pgstac, public;\n\nDROP FUNCTION IF EXISTS geometrysearch;\nCREATE OR REPLACE FUNCTION geometrysearch(\n IN geom geometry,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE, -- Return as soon as the passed in geometry is full covered\n IN skipcovered boolean DEFAULT TRUE -- Skip any items that would show up completely under the previous items\n) RETURNS jsonb AS $$\nDECLARE\n search searches%ROWTYPE;\n curs refcursor;\n _where text;\n query text;\n iter_record items%ROWTYPE;\n out_records jsonb := '{}'::jsonb[];\n exit_flag boolean := FALSE;\n counter int := 1;\n scancounter int := 1;\n remaining_limit int := _scanlimit;\n tilearea float;\n unionedgeom geometry;\n clippedgeom geometry;\n unionedgeom_area float := 0;\n prev_area float := 0;\n excludes text[];\n includes text[];\n\nBEGIN\n DROP TABLE IF EXISTS pgstac_results;\n CREATE TEMP TABLE pgstac_results (content jsonb) ON COMMIT DROP;\n\n -- If the passed in geometry is not an area set exitwhenfull and skipcovered to false\n IF ST_GeometryType(geom) !~* 'polygon' THEN\n RAISE NOTICE 'GEOMETRY IS NOT AN AREA';\n skipcovered = FALSE;\n exitwhenfull = FALSE;\n END IF;\n\n -- If skipcovered is true then you will always want to exit when the passed in geometry is full\n IF skipcovered THEN\n exitwhenfull := TRUE;\n END IF;\n\n search := search_fromhash(queryhash);\n\n IF search IS NULL THEN\n RAISE EXCEPTION 'Search with Query Hash % Not Found', queryhash;\n END IF;\n\n tilearea := st_area(geom);\n _where := format('%s AND st_intersects(geometry, %L::geometry)', search._where, geom);\n\n\n FOR query IN SELECT * FROM partition_queries(_where, search.orderby) LOOP\n query := format('%s LIMIT %L', query, remaining_limit);\n RAISE NOTICE '%', query;\n OPEN curs FOR EXECUTE query;\n LOOP\n FETCH curs INTO iter_record;\n EXIT WHEN NOT FOUND;\n IF exitwhenfull OR skipcovered THEN -- If we are not using exitwhenfull or skipcovered, we do not need to do expensive geometry operations\n clippedgeom := st_intersection(geom, iter_record.geometry);\n\n IF unionedgeom IS NULL THEN\n unionedgeom := clippedgeom;\n ELSE\n unionedgeom := st_union(unionedgeom, clippedgeom);\n END IF;\n\n unionedgeom_area := st_area(unionedgeom);\n\n IF skipcovered AND prev_area = unionedgeom_area THEN\n scancounter := scancounter + 1;\n CONTINUE;\n END IF;\n\n prev_area := unionedgeom_area;\n\n RAISE NOTICE '% % % %', unionedgeom_area/tilearea, counter, scancounter, ftime();\n END IF;\n RAISE NOTICE '% %', iter_record, content_hydrate(iter_record, fields);\n INSERT INTO pgstac_results (content) VALUES (content_hydrate(iter_record, fields));\n\n IF counter >= _limit\n OR scancounter > _scanlimit\n OR ftime() > _timelimit\n OR (exitwhenfull AND unionedgeom_area >= tilearea)\n THEN\n exit_flag := TRUE;\n EXIT;\n END IF;\n counter := counter + 1;\n scancounter := scancounter + 1;\n\n END LOOP;\n CLOSE curs;\n EXIT WHEN exit_flag;\n remaining_limit := _scanlimit - scancounter;\n END LOOP;\n\n SELECT jsonb_agg(content) INTO out_records FROM pgstac_results WHERE content IS NOT NULL;\n\n RETURN jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb)\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDROP FUNCTION IF EXISTS geojsonsearch;\nCREATE OR REPLACE FUNCTION geojsonsearch(\n IN geojson jsonb,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_geomfromgeojson(geojson),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS xyzsearch;\nCREATE OR REPLACE FUNCTION xyzsearch(\n IN _x int,\n IN _y int,\n IN _z int,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_transform(tileenvelope(_z, _x, _y), 4326),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE PROCEDURE analyze_items() AS $$\nDECLARE\n q text;\n timeout_ts timestamptz;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n SELECT format('ANALYZE (VERBOSE, SKIP_LOCKED) %I;', relname) INTO q\n FROM pg_stat_user_tables\n WHERE relname like '_item%' AND (n_mod_since_analyze>0 OR last_analyze IS NULL) LIMIT 1;\n IF NOT FOUND THEN\n EXIT;\n END IF;\n RAISE NOTICE '%', q;\n EXECUTE q;\n COMMIT;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE PROCEDURE validate_constraints() AS $$\nDECLARE\n q text;\nBEGIN\n FOR q IN\n SELECT\n FORMAT(\n 'ALTER TABLE %I.%I VALIDATE CONSTRAINT %I;',\n nsp.nspname,\n cls.relname,\n con.conname\n )\n\n FROM pg_constraint AS con\n JOIN pg_class AS cls\n ON con.conrelid = cls.oid\n JOIN pg_namespace AS nsp\n ON cls.relnamespace = nsp.oid\n WHERE convalidated = FALSE AND contype in ('c','f')\n AND nsp.nspname = 'pgstac'\n LOOP\n RAISE NOTICE '%', q;\n PERFORM run_or_queue(q);\n COMMIT;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION collection_extent(_collection text, runupdate boolean default false) RETURNS jsonb AS $$\nDECLARE\n geom_extent geometry;\n mind timestamptz;\n maxd timestamptz;\n extent jsonb;\nBEGIN\n IF runupdate THEN\n PERFORM update_partition_stats_q(partition)\n FROM partitions_view WHERE collection=_collection;\n END IF;\n SELECT\n min(lower(dtrange)),\n max(upper(edtrange)),\n st_extent(spatial)\n INTO\n mind,\n maxd,\n geom_extent\n FROM partitions_view\n WHERE collection=_collection;\n\n IF geom_extent IS NOT NULL AND mind IS NOT NULL AND maxd IS NOT NULL THEN\n extent := jsonb_build_object(\n 'spatial', jsonb_build_object(\n 'bbox', to_jsonb(array[array[st_xmin(geom_extent), st_ymin(geom_extent), st_xmax(geom_extent), st_ymax(geom_extent)]])\n ),\n 'temporal', jsonb_build_object(\n 'interval', to_jsonb(array[array[mind, maxd]])\n )\n );\n RETURN extent;\n END IF;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('id', '{\"title\": \"Item ID\",\"description\": \"Item identifier\",\"$ref\": \"https://schemas.stacspec.org/v1.0.0/item-spec/json-schema/item.json#/definitions/core/allOf/2/properties/id\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('geometry', '{\"title\": \"Item Geometry\",\"description\": \"Item Geometry\",\"$ref\": \"https://geojson.org/schema/Feature.json\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('datetime','{\"description\": \"Datetime\",\"type\": \"string\",\"title\": \"Acquired\",\"format\": \"date-time\",\"pattern\": \"(\\\\+00:00|Z)$\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDELETE FROM queryables a USING queryables b\n WHERE a.name = b.name AND a.collection_ids IS NOT DISTINCT FROM b.collection_ids AND a.id > b.id;\n\n\nINSERT INTO pgstac_settings (name, value) VALUES\n ('context', 'off'),\n ('context_estimated_count', '100000'),\n ('context_estimated_cost', '100000'),\n ('context_stats_ttl', '1 day'),\n ('default_filter_lang', 'cql2-json'),\n ('additional_properties', 'true'),\n ('use_queue', 'false'),\n ('queue_timeout', '10 minutes'),\n ('update_collection_extent', 'false'),\n ('format_cache', 'false'),\n ('readonly', 'false')\nON CONFLICT DO NOTHING\n;\n\n\nINSERT INTO cql2_ops (op, template, types) VALUES\n ('eq', '%s = %s', NULL),\n ('neq', '%s != %s', NULL),\n ('ne', '%s != %s', NULL),\n ('!=', '%s != %s', NULL),\n ('<>', '%s != %s', NULL),\n ('lt', '%s < %s', NULL),\n ('lte', '%s <= %s', NULL),\n ('gt', '%s > %s', NULL),\n ('gte', '%s >= %s', NULL),\n ('le', '%s <= %s', NULL),\n ('ge', '%s >= %s', NULL),\n ('=', '%s = %s', NULL),\n ('<', '%s < %s', NULL),\n ('<=', '%s <= %s', NULL),\n ('>', '%s > %s', NULL),\n ('>=', '%s >= %s', NULL),\n ('like', '%s LIKE %s', NULL),\n ('ilike', '%s ILIKE %s', NULL),\n ('+', '%s + %s', NULL),\n ('-', '%s - %s', NULL),\n ('*', '%s * %s', NULL),\n ('/', '%s / %s', NULL),\n ('not', 'NOT (%s)', NULL),\n ('between', '%s BETWEEN %s AND %s', NULL),\n ('isnull', '%s IS NULL', NULL),\n ('upper', 'upper(%s)', NULL),\n ('lower', 'lower(%s)', NULL),\n ('casei', 'upper(%s)', NULL),\n ('accenti', 'unaccent(%s)', NULL)\nON CONFLICT (op) DO UPDATE\n SET\n template = EXCLUDED.template\n;\n\n\nALTER FUNCTION to_text COST 5000;\nALTER FUNCTION to_float COST 5000;\nALTER FUNCTION to_int COST 5000;\nALTER FUNCTION to_tstz COST 5000;\nALTER FUNCTION to_text_array COST 5000;\n\nALTER FUNCTION update_partition_stats SECURITY DEFINER;\nALTER FUNCTION partition_after_triggerfunc SECURITY DEFINER;\nALTER FUNCTION drop_table_constraints SECURITY DEFINER;\nALTER FUNCTION create_table_constraints SECURITY DEFINER;\nALTER FUNCTION check_partition SECURITY DEFINER;\nALTER FUNCTION repartition SECURITY DEFINER;\nALTER FUNCTION where_stats SECURITY DEFINER;\nALTER FUNCTION search_query SECURITY DEFINER;\nALTER FUNCTION format_item SECURITY DEFINER;\nALTER FUNCTION maintain_index SECURITY DEFINER;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nGRANT ALL ON SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac to pgstac_admin;\n\n-- pgstac_read role limited to using function apis\nGRANT EXECUTE ON FUNCTION search TO pgstac_read;\nGRANT EXECUTE ON FUNCTION search_query TO pgstac_read;\nGRANT EXECUTE ON FUNCTION item_by_id TO pgstac_read;\nGRANT EXECUTE ON FUNCTION get_item TO pgstac_read;\nGRANT SELECT ON ALL TABLES IN SCHEMA pgstac TO pgstac_read;\n\n\nGRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_ingest;\nGRANT USAGE ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_ingest;\n\nREVOKE ALL PRIVILEGES ON PROCEDURE run_queued_queries FROM public;\nGRANT ALL ON PROCEDURE run_queued_queries TO pgstac_admin;\n\nREVOKE ALL PRIVILEGES ON FUNCTION run_queued_queries_intransaction FROM public;\nGRANT ALL ON FUNCTION run_queued_queries_intransaction TO pgstac_admin;\n\nRESET ROLE;\n\nSET ROLE pgstac_ingest;\nSELECT update_partition_stats_q(partition) FROM partitions_view;\nSELECT set_version('0.9.6');\n" }, { "path": "src/pgstac/migrations/pgstac.0.9.7-0.9.8.sql", "content": "SET client_min_messages TO WARNING;\nSET SEARCH_PATH to pgstac, public;\nRESET ROLE;\nDO $$\nDECLARE\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='postgis') THEN\n CREATE EXTENSION IF NOT EXISTS postgis;\n END IF;\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='btree_gist') THEN\n CREATE EXTENSION IF NOT EXISTS btree_gist;\n END IF;\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='unaccent') THEN\n CREATE EXTENSION IF NOT EXISTS unaccent;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_admin;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_read;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_ingest;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n\nGRANT pgstac_admin TO current_user;\n\n-- Function to make sure pgstac_admin is the owner of items\nCREATE OR REPLACE FUNCTION pgstac_admin_owns() RETURNS VOID AS $$\nDECLARE\n f RECORD;\nBEGIN\n FOR f IN (\n SELECT\n concat(\n oid::regproc::text,\n '(',\n coalesce(pg_get_function_identity_arguments(oid),''),\n ')'\n ) AS name,\n CASE prokind WHEN 'f' THEN 'FUNCTION' WHEN 'p' THEN 'PROCEDURE' WHEN 'a' THEN 'AGGREGATE' END as typ\n FROM pg_proc\n WHERE\n pronamespace=to_regnamespace('pgstac')\n AND proowner != to_regrole('pgstac_admin')\n AND proname NOT LIKE 'pg_stat%'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n FOR f IN (\n SELECT\n oid::regclass::text as name,\n CASE relkind\n WHEN 'i' THEN 'INDEX'\n WHEN 'I' THEN 'INDEX'\n WHEN 'p' THEN 'TABLE'\n WHEN 'r' THEN 'TABLE'\n WHEN 'v' THEN 'VIEW'\n WHEN 'S' THEN 'SEQUENCE'\n ELSE NULL\n END as typ\n FROM pg_class\n WHERE relnamespace=to_regnamespace('pgstac') and relowner != to_regrole('pgstac_admin') AND relkind IN ('r','p','v','S') AND relname NOT LIKE 'pg_stat'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\nSELECT pgstac_admin_owns();\n\nCREATE SCHEMA IF NOT EXISTS pgstac AUTHORIZATION pgstac_admin;\n\nGRANT ALL ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_admin;\n\nALTER ROLE pgstac_admin SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_read SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_ingest SET SEARCH_PATH TO pgstac, public;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\n\nGRANT pgstac_read TO pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\n\nSET ROLE pgstac_admin;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\nRESET ROLE;\n\nSET ROLE pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\nRESET ROLE;\n\nSET SEARCH_PATH TO pgstac, public;\nSET ROLE pgstac_admin;\n\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS analyze_items;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS validate_constraints;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n-- Install these idempotently as migrations do not put them before trying to modify the collections table\n\n\nCREATE OR REPLACE FUNCTION collection_geom(content jsonb)\nRETURNS geometry AS $$\n WITH box AS (SELECT content->'extent'->'spatial'->'bbox'->0 as box)\n SELECT\n st_makeenvelope(\n (box->>0)::float,\n (box->>1)::float,\n (box->>2)::float,\n (box->>3)::float,\n 4326\n )\n FROM box;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION collection_datetime(content jsonb)\nRETURNS timestamptz AS $$\n SELECT\n CASE\n WHEN\n (content->'extent'->'temporal'->'interval'->0->>0) IS NULL\n THEN '-infinity'::timestamptz\n ELSE\n (content->'extent'->'temporal'->'interval'->0->>0)::timestamptz\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION collection_enddatetime(content jsonb)\nRETURNS timestamptz AS $$\n SELECT\n CASE\n WHEN\n (content->'extent'->'temporal'->'interval'->0->>1) IS NULL\n THEN 'infinity'::timestamptz\n ELSE\n (content->'extent'->'temporal'->'interval'->0->>1)::timestamptz\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n-- BEGIN migra calculated SQL\ndrop function if exists \"pgstac\".\"q_to_tsquery\"(input text);\n\nset check_function_bodies = off;\n\nDROP FUNCTION IF EXISTS pgstac.q_to_tsquery(text);\nCREATE OR REPLACE FUNCTION pgstac.q_to_tsquery(jinput jsonb)\n RETURNS tsquery\n LANGUAGE plpgsql\nAS $function$\nDECLARE\n input text;\n processed_text text;\n temp_text text;\n quote_array text[];\n placeholder text := '@QUOTE@';\nBEGIN\n IF jsonb_typeof(jinput) = 'string' THEN\n input := jinput->>0;\n ELSIF jsonb_typeof(jinput) = 'array' THEN\n input := array_to_string(\n array(select jsonb_array_elements_text(jinput)),\n ' OR '\n );\n ELSE\n RAISE EXCEPTION 'Input must be a string or an array of strings.';\n END IF;\n -- Extract all quoted phrases and store in array\n quote_array := regexp_matches(input, '\"[^\"]*\"', 'g');\n\n -- Replace each quoted part with a unique placeholder if there are any quoted phrases\n IF array_length(quote_array, 1) IS NOT NULL THEN\n processed_text := input;\n FOR i IN array_lower(quote_array, 1) .. array_upper(quote_array, 1) LOOP\n processed_text := replace(processed_text, quote_array[i], placeholder || i || placeholder);\n END LOOP;\n ELSE\n processed_text := input;\n END IF;\n\n -- Replace non-quoted text using regular expressions\n\n -- , -> |\n processed_text := regexp_replace(processed_text, ',(?=(?:[^\"]*\"[^\"]*\")*[^\"]*$)', ' | ', 'g');\n\n -- and -> &\n processed_text := regexp_replace(processed_text, '\\s+AND\\s+', ' & ', 'gi');\n\n -- or -> |\n processed_text := regexp_replace(processed_text, '\\s+OR\\s+', ' | ', 'gi');\n\n -- + ->\n processed_text := regexp_replace(processed_text, '^\\s*\\+([a-zA-Z0-9_]+)', '\\1', 'g'); -- +term at start\n processed_text := regexp_replace(processed_text, '\\s*\\+([a-zA-Z0-9_]+)', ' & \\1', 'g'); -- +term elsewhere\n\n -- - -> !\n processed_text := regexp_replace(processed_text, '^\\s*\\-([a-zA-Z0-9_]+)', '! \\1', 'g'); -- -term at start\n processed_text := regexp_replace(processed_text, '\\s*\\-([a-zA-Z0-9_]+)', ' & ! \\1', 'g'); -- -term elsewhere\n -- Replace placeholders back with quoted phrases if there were any\n IF array_length(quote_array, 1) IS NOT NULL THEN\n FOR i IN array_lower(quote_array, 1) .. array_upper(quote_array, 1) LOOP\n processed_text := replace(processed_text, placeholder || i || placeholder, '''' || substring(quote_array[i] from 2 for length(quote_array[i]) - 2) || '''');\n END LOOP;\n END IF;\n\n -- Print processed_text to the console for debugging purposes\n RAISE NOTICE 'processed_text: %', processed_text;\n\n RETURN to_tsquery('english', processed_text);\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.stac_search_to_where(j jsonb)\n RETURNS text\n LANGUAGE plpgsql\n STABLE\nAS $function$\nDECLARE\n where_segments text[];\n _where text;\n dtrange tstzrange;\n collections text[];\n geom geometry;\n sdate timestamptz;\n edate timestamptz;\n filterlang text;\n filter jsonb := j->'filter';\n ft_query tsquery;\nBEGIN\n IF j ? 'ids' THEN\n where_segments := where_segments || format('id = ANY (%L) ', to_text_array(j->'ids'));\n END IF;\n\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n where_segments := where_segments || format('collection = ANY (%L) ', collections);\n END IF;\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n\n where_segments := where_segments || format(' datetime <= %L::timestamptz AND end_datetime >= %L::timestamptz ',\n edate,\n sdate\n );\n END IF;\n\n IF j ? 'q' THEN\n ft_query := q_to_tsquery(j->'q');\n where_segments := where_segments || format(\n $quote$\n (\n to_tsvector('english', content->'properties'->>'description') ||\n to_tsvector('english', coalesce(content->'properties'->>'title', '')) ||\n to_tsvector('english', coalesce(content->'properties'->>'keywords', ''))\n ) @@ %L\n $quote$,\n ft_query\n );\n END IF;\n\n geom := stac_geom(j);\n IF geom IS NOT NULL THEN\n where_segments := where_segments || format('st_intersects(geometry, %L)',geom);\n END IF;\n\n filterlang := COALESCE(\n j->>'filter-lang',\n get_setting('default_filter_lang', j->'conf')\n );\n IF NOT filter @? '$.**.op' THEN\n filterlang := 'cql-json';\n END IF;\n\n IF filterlang NOT IN ('cql-json','cql2-json') AND j ? 'filter' THEN\n RAISE EXCEPTION '% is not a supported filter-lang. Please use cql-json or cql2-json.', filterlang;\n END IF;\n\n IF j ? 'query' AND j ? 'filter' THEN\n RAISE EXCEPTION 'Can only use either query or filter at one time.';\n END IF;\n\n IF j ? 'query' THEN\n filter := query_to_cql2(j->'query');\n ELSIF filterlang = 'cql-json' THEN\n filter := cql1_to_cql2(filter);\n END IF;\n RAISE NOTICE 'FILTER: %', filter;\n where_segments := where_segments || cql2_query(filter);\n IF cardinality(where_segments) < 1 THEN\n RETURN ' TRUE ';\n END IF;\n\n _where := array_to_string(array_remove(where_segments, NULL), ' AND ');\n\n IF _where IS NULL OR BTRIM(_where) = '' THEN\n RETURN ' TRUE ';\n END IF;\n RETURN _where;\n\nEND;\n$function$\n;\n\n\n-- END migra calculated SQL\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('id', '{\"title\": \"Item ID\",\"description\": \"Item identifier\",\"$ref\": \"https://schemas.stacspec.org/v1.0.0/item-spec/json-schema/item.json#/definitions/core/allOf/2/properties/id\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('geometry', '{\"title\": \"Item Geometry\",\"description\": \"Item Geometry\",\"$ref\": \"https://geojson.org/schema/Feature.json\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('datetime','{\"description\": \"Datetime\",\"type\": \"string\",\"title\": \"Acquired\",\"format\": \"date-time\",\"pattern\": \"(\\\\+00:00|Z)$\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDELETE FROM queryables a USING queryables b\n WHERE a.name = b.name AND a.collection_ids IS NOT DISTINCT FROM b.collection_ids AND a.id > b.id;\n\n\nINSERT INTO pgstac_settings (name, value) VALUES\n ('context', 'off'),\n ('context_estimated_count', '100000'),\n ('context_estimated_cost', '100000'),\n ('context_stats_ttl', '1 day'),\n ('default_filter_lang', 'cql2-json'),\n ('additional_properties', 'true'),\n ('use_queue', 'false'),\n ('queue_timeout', '10 minutes'),\n ('update_collection_extent', 'false'),\n ('format_cache', 'false'),\n ('readonly', 'false')\nON CONFLICT DO NOTHING\n;\n\n\nINSERT INTO cql2_ops (op, template, types) VALUES\n ('eq', '%s = %s', NULL),\n ('neq', '%s != %s', NULL),\n ('ne', '%s != %s', NULL),\n ('!=', '%s != %s', NULL),\n ('<>', '%s != %s', NULL),\n ('lt', '%s < %s', NULL),\n ('lte', '%s <= %s', NULL),\n ('gt', '%s > %s', NULL),\n ('gte', '%s >= %s', NULL),\n ('le', '%s <= %s', NULL),\n ('ge', '%s >= %s', NULL),\n ('=', '%s = %s', NULL),\n ('<', '%s < %s', NULL),\n ('<=', '%s <= %s', NULL),\n ('>', '%s > %s', NULL),\n ('>=', '%s >= %s', NULL),\n ('like', '%s LIKE %s', NULL),\n ('ilike', '%s ILIKE %s', NULL),\n ('+', '%s + %s', NULL),\n ('-', '%s - %s', NULL),\n ('*', '%s * %s', NULL),\n ('/', '%s / %s', NULL),\n ('not', 'NOT (%s)', NULL),\n ('between', '%s BETWEEN %s AND %s', NULL),\n ('isnull', '%s IS NULL', NULL),\n ('upper', 'upper(%s)', NULL),\n ('lower', 'lower(%s)', NULL),\n ('casei', 'upper(%s)', NULL),\n ('accenti', 'unaccent(%s)', NULL)\nON CONFLICT (op) DO UPDATE\n SET\n template = EXCLUDED.template\n;\n\n\nALTER FUNCTION to_text COST 5000;\nALTER FUNCTION to_float COST 5000;\nALTER FUNCTION to_int COST 5000;\nALTER FUNCTION to_tstz COST 5000;\nALTER FUNCTION to_text_array COST 5000;\n\nALTER FUNCTION update_partition_stats SECURITY DEFINER;\nALTER FUNCTION partition_after_triggerfunc SECURITY DEFINER;\nALTER FUNCTION drop_table_constraints SECURITY DEFINER;\nALTER FUNCTION create_table_constraints SECURITY DEFINER;\nALTER FUNCTION check_partition SECURITY DEFINER;\nALTER FUNCTION repartition SECURITY DEFINER;\nALTER FUNCTION where_stats SECURITY DEFINER;\nALTER FUNCTION search_query SECURITY DEFINER;\nALTER FUNCTION format_item SECURITY DEFINER;\nALTER FUNCTION maintain_index SECURITY DEFINER;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nGRANT ALL ON SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac to pgstac_admin;\n\n-- pgstac_read role limited to using function apis\nGRANT EXECUTE ON FUNCTION search TO pgstac_read;\nGRANT EXECUTE ON FUNCTION search_query TO pgstac_read;\nGRANT EXECUTE ON FUNCTION item_by_id TO pgstac_read;\nGRANT EXECUTE ON FUNCTION get_item TO pgstac_read;\nGRANT SELECT ON ALL TABLES IN SCHEMA pgstac TO pgstac_read;\n\n\nGRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_ingest;\nGRANT USAGE ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_ingest;\n\nREVOKE ALL PRIVILEGES ON PROCEDURE run_queued_queries FROM public;\nGRANT ALL ON PROCEDURE run_queued_queries TO pgstac_admin;\n\nREVOKE ALL PRIVILEGES ON FUNCTION run_queued_queries_intransaction FROM public;\nGRANT ALL ON FUNCTION run_queued_queries_intransaction TO pgstac_admin;\n\nRESET ROLE;\n\nSET ROLE pgstac_ingest;\nSELECT update_partition_stats_q(partition) FROM partitions_view;\nSELECT set_version('0.9.8');\n" }, { "path": "src/pgstac/migrations/pgstac.0.9.7.sql", "content": "RESET ROLE;\nDO $$\nDECLARE\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='postgis') THEN\n CREATE EXTENSION IF NOT EXISTS postgis;\n END IF;\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='btree_gist') THEN\n CREATE EXTENSION IF NOT EXISTS btree_gist;\n END IF;\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='unaccent') THEN\n CREATE EXTENSION IF NOT EXISTS unaccent;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_admin;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_read;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_ingest;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n\nGRANT pgstac_admin TO current_user;\n\n-- Function to make sure pgstac_admin is the owner of items\nCREATE OR REPLACE FUNCTION pgstac_admin_owns() RETURNS VOID AS $$\nDECLARE\n f RECORD;\nBEGIN\n FOR f IN (\n SELECT\n concat(\n oid::regproc::text,\n '(',\n coalesce(pg_get_function_identity_arguments(oid),''),\n ')'\n ) AS name,\n CASE prokind WHEN 'f' THEN 'FUNCTION' WHEN 'p' THEN 'PROCEDURE' WHEN 'a' THEN 'AGGREGATE' END as typ\n FROM pg_proc\n WHERE\n pronamespace=to_regnamespace('pgstac')\n AND proowner != to_regrole('pgstac_admin')\n AND proname NOT LIKE 'pg_stat%'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n FOR f IN (\n SELECT\n oid::regclass::text as name,\n CASE relkind\n WHEN 'i' THEN 'INDEX'\n WHEN 'I' THEN 'INDEX'\n WHEN 'p' THEN 'TABLE'\n WHEN 'r' THEN 'TABLE'\n WHEN 'v' THEN 'VIEW'\n WHEN 'S' THEN 'SEQUENCE'\n ELSE NULL\n END as typ\n FROM pg_class\n WHERE relnamespace=to_regnamespace('pgstac') and relowner != to_regrole('pgstac_admin') AND relkind IN ('r','p','v','S') AND relname NOT LIKE 'pg_stat'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\nSELECT pgstac_admin_owns();\n\nCREATE SCHEMA IF NOT EXISTS pgstac AUTHORIZATION pgstac_admin;\n\nGRANT ALL ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_admin;\n\nALTER ROLE pgstac_admin SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_read SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_ingest SET SEARCH_PATH TO pgstac, public;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\n\nGRANT pgstac_read TO pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\n\nSET ROLE pgstac_admin;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\nRESET ROLE;\n\nSET ROLE pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\nRESET ROLE;\n\nSET SEARCH_PATH TO pgstac, public;\nSET ROLE pgstac_admin;\n\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS analyze_items;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS validate_constraints;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n-- Install these idempotently as migrations do not put them before trying to modify the collections table\n\n\nCREATE OR REPLACE FUNCTION collection_geom(content jsonb)\nRETURNS geometry AS $$\n WITH box AS (SELECT content->'extent'->'spatial'->'bbox'->0 as box)\n SELECT\n st_makeenvelope(\n (box->>0)::float,\n (box->>1)::float,\n (box->>2)::float,\n (box->>3)::float,\n 4326\n )\n FROM box;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION collection_datetime(content jsonb)\nRETURNS timestamptz AS $$\n SELECT\n CASE\n WHEN\n (content->'extent'->'temporal'->'interval'->0->>0) IS NULL\n THEN '-infinity'::timestamptz\n ELSE\n (content->'extent'->'temporal'->'interval'->0->>0)::timestamptz\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION collection_enddatetime(content jsonb)\nRETURNS timestamptz AS $$\n SELECT\n CASE\n WHEN\n (content->'extent'->'temporal'->'interval'->0->>1) IS NULL\n THEN 'infinity'::timestamptz\n ELSE\n (content->'extent'->'temporal'->'interval'->0->>1)::timestamptz\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE TABLE IF NOT EXISTS migrations (\n version text PRIMARY KEY,\n datetime timestamptz DEFAULT clock_timestamp() NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION get_version() RETURNS text AS $$\n SELECT version FROM pgstac.migrations ORDER BY datetime DESC, version DESC LIMIT 1;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION set_version(text) RETURNS text AS $$\n INSERT INTO pgstac.migrations (version) VALUES ($1)\n ON CONFLICT DO NOTHING\n RETURNING version;\n$$ LANGUAGE SQL;\n\n\nCREATE TABLE IF NOT EXISTS pgstac_settings (\n name text PRIMARY KEY,\n value text NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION table_empty(text) RETURNS boolean AS $$\nDECLARE\n retval boolean;\nBEGIN\n EXECUTE format($q$\n SELECT NOT EXISTS (SELECT 1 FROM %I LIMIT 1)\n $q$,\n $1\n ) INTO retval;\n RETURN retval;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION get_setting(IN _setting text, IN conf jsonb DEFAULT NULL) RETURNS text AS $$\nSELECT COALESCE(\n nullif(conf->>_setting, ''),\n nullif(current_setting(concat('pgstac.',_setting), TRUE),''),\n nullif((SELECT value FROM pgstac.pgstac_settings WHERE name=_setting),'')\n);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_setting_bool(IN _setting text, IN conf jsonb DEFAULT NULL) RETURNS boolean AS $$\nSELECT COALESCE(\n nullif(conf->>_setting, ''),\n nullif(current_setting(concat('pgstac.',_setting), TRUE),''),\n nullif((SELECT value FROM pgstac.pgstac_settings WHERE name=_setting),''),\n 'FALSE'\n)::boolean;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION base_url(conf jsonb DEFAULT NULL) RETURNS text AS $$\n SELECT COALESCE(pgstac.get_setting('base_url', conf), '.');\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION additional_properties() RETURNS boolean AS $$\n SELECT pgstac.get_setting_bool('additional_properties');\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION readonly(conf jsonb DEFAULT NULL) RETURNS boolean AS $$\n SELECT pgstac.get_setting_bool('readonly', conf);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context(conf jsonb DEFAULT NULL) RETURNS text AS $$\n SELECT pgstac.get_setting('context', conf);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context_estimated_count(conf jsonb DEFAULT NULL) RETURNS int AS $$\n SELECT pgstac.get_setting('context_estimated_count', conf)::int;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_estimated_cost();\nCREATE OR REPLACE FUNCTION context_estimated_cost(conf jsonb DEFAULT NULL) RETURNS float AS $$\n SELECT pgstac.get_setting('context_estimated_cost', conf)::float;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_stats_ttl();\nCREATE OR REPLACE FUNCTION context_stats_ttl(conf jsonb DEFAULT NULL) RETURNS interval AS $$\n SELECT pgstac.get_setting('context_stats_ttl', conf)::interval;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION t2s(text) RETURNS text AS $$\n SELECT extract(epoch FROM $1::interval)::text || ' s';\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE STRICT;\n\nCREATE OR REPLACE FUNCTION age_ms(a timestamptz, b timestamptz DEFAULT clock_timestamp()) RETURNS float AS $$\n SELECT abs(extract(epoch from age(a,b)) * 1000);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION queue_timeout() RETURNS interval AS $$\n SELECT t2s(coalesce(\n get_setting('queue_timeout'),\n '1h'\n ))::interval;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION notice(VARIADIC text[]) RETURNS boolean AS $$\nDECLARE\ndebug boolean := current_setting('pgstac.debug', true);\nBEGIN\n IF debug THEN\n RAISE NOTICE 'NOTICE FROM FUNC: % >>>>> %', concat_ws(' | ', $1), clock_timestamp();\n RETURN TRUE;\n END IF;\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION empty_arr(ANYARRAY) RETURNS BOOLEAN AS $$\nSELECT CASE\n WHEN $1 IS NULL THEN TRUE\n WHEN cardinality($1)<1 THEN TRUE\nELSE FALSE\nEND;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_intersection(_a ANYARRAY, _b ANYARRAY) RETURNS ANYARRAY AS $$\n SELECT ARRAY ( SELECT unnest(_a) INTERSECT SELECT UNNEST(_b) );\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION array_map_ident(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_ident(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_map_literal(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_literal(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION array_reverse(anyarray) RETURNS anyarray AS $$\nSELECT ARRAY(\n SELECT $1[i]\n FROM generate_subscripts($1,1) AS s(i)\n ORDER BY i DESC\n);\n$$ LANGUAGE SQL STRICT IMMUTABLE;\n\nDROP TABLE IF EXISTS query_queue;\nCREATE TABLE query_queue (\n query text PRIMARY KEY,\n added timestamptz DEFAULT now()\n);\n\nDROP TABLE IF EXISTS query_queue_history;\nCREATE TABLE query_queue_history(\n query text,\n added timestamptz NOT NULL,\n finished timestamptz NOT NULL DEFAULT now(),\n error text\n);\n\nCREATE OR REPLACE PROCEDURE run_queued_queries() AS $$\nDECLARE\n qitem query_queue%ROWTYPE;\n timeout_ts timestamptz;\n error text;\n cnt int := 0;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n DELETE FROM query_queue WHERE query = (SELECT query FROM query_queue ORDER BY added DESC LIMIT 1 FOR UPDATE SKIP LOCKED) RETURNING * INTO qitem;\n IF NOT FOUND THEN\n EXIT;\n END IF;\n cnt := cnt + 1;\n BEGIN\n RAISE NOTICE 'RUNNING QUERY: %', qitem.query;\n EXECUTE qitem.query;\n EXCEPTION WHEN others THEN\n error := format('%s | %s', SQLERRM, SQLSTATE);\n END;\n INSERT INTO query_queue_history (query, added, finished, error)\n VALUES (qitem.query, qitem.added, clock_timestamp(), error);\n COMMIT;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION run_queued_queries_intransaction() RETURNS int AS $$\nDECLARE\n qitem query_queue%ROWTYPE;\n timeout_ts timestamptz;\n error text;\n cnt int := 0;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n DELETE FROM query_queue WHERE query = (SELECT query FROM query_queue ORDER BY added DESC LIMIT 1 FOR UPDATE SKIP LOCKED) RETURNING * INTO qitem;\n IF NOT FOUND THEN\n RETURN cnt;\n END IF;\n cnt := cnt + 1;\n BEGIN\n qitem.query := regexp_replace(qitem.query, 'CONCURRENTLY', '');\n RAISE NOTICE 'RUNNING QUERY: %', qitem.query;\n\n EXECUTE qitem.query;\n EXCEPTION WHEN others THEN\n error := format('%s | %s', SQLERRM, SQLSTATE);\n RAISE WARNING '%', error;\n END;\n INSERT INTO query_queue_history (query, added, finished, error)\n VALUES (qitem.query, qitem.added, clock_timestamp(), error);\n END LOOP;\n RETURN cnt;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION run_or_queue(query text) RETURNS VOID AS $$\nDECLARE\n use_queue text := COALESCE(get_setting('use_queue'), 'FALSE')::boolean;\nBEGIN\n IF get_setting_bool('debug') THEN\n RAISE NOTICE '%', query;\n END IF;\n IF use_queue THEN\n INSERT INTO query_queue (query) VALUES (query) ON CONFLICT DO NOTHING;\n ELSE\n EXECUTE query;\n END IF;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n\nDROP FUNCTION IF EXISTS check_pgstac_settings;\nCREATE OR REPLACE FUNCTION check_pgstac_settings(_sysmem text DEFAULT NULL) RETURNS VOID AS $$\nDECLARE\n settingval text;\n sysmem bigint := pg_size_bytes(_sysmem);\n effective_cache_size bigint := pg_size_bytes(current_setting('effective_cache_size', TRUE));\n shared_buffers bigint := pg_size_bytes(current_setting('shared_buffers', TRUE));\n work_mem bigint := pg_size_bytes(current_setting('work_mem', TRUE));\n max_connections int := current_setting('max_connections', TRUE);\n maintenance_work_mem bigint := pg_size_bytes(current_setting('maintenance_work_mem', TRUE));\n seq_page_cost float := current_setting('seq_page_cost', TRUE);\n random_page_cost float := current_setting('random_page_cost', TRUE);\n temp_buffers bigint := pg_size_bytes(current_setting('temp_buffers', TRUE));\n r record;\nBEGIN\n IF _sysmem IS NULL THEN\n RAISE NOTICE 'Call function with the size of your system memory `SELECT check_pgstac_settings(''4GB'')` to get pg system setting recommendations.';\n ELSE\n IF effective_cache_size < (sysmem * 0.5) THEN\n RAISE WARNING 'effective_cache_size of % is set low for a system with %. Recomended value between % and %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.5), pg_size_pretty(sysmem * 0.75);\n ELSIF effective_cache_size > (sysmem * 0.75) THEN\n RAISE WARNING 'effective_cache_size of % is set high for a system with %. Recomended value between % and %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.5), pg_size_pretty(sysmem * 0.75);\n ELSE\n RAISE NOTICE 'effective_cache_size of % is set appropriately for a system with %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem);\n END IF;\n\n IF shared_buffers < (sysmem * 0.2) THEN\n RAISE WARNING 'shared_buffers of % is set low for a system with %. Recomended value between % and %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.2), pg_size_pretty(sysmem * 0.3);\n ELSIF shared_buffers > (sysmem * 0.3) THEN\n RAISE WARNING 'shared_buffers of % is set high for a system with %. Recomended value between % and %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.2), pg_size_pretty(sysmem * 0.3);\n ELSE\n RAISE NOTICE 'shared_buffers of % is set appropriately for a system with %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem);\n END IF;\n shared_buffers = sysmem * 0.3;\n IF maintenance_work_mem < (sysmem * 0.2) THEN\n RAISE WARNING 'maintenance_work_mem of % is set low for shared_buffers of %. Recomended value between % and %', pg_size_pretty(maintenance_work_mem), pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers * 0.2), pg_size_pretty(shared_buffers * 0.3);\n ELSIF maintenance_work_mem > (shared_buffers * 0.3) THEN\n RAISE WARNING 'maintenance_work_mem of % is set high for shared_buffers of %. Recomended value between % and %', pg_size_pretty(maintenance_work_mem), pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers * 0.2), pg_size_pretty(shared_buffers * 0.3);\n ELSE\n RAISE NOTICE 'maintenance_work_mem of % is set appropriately for shared_buffers of %', pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers);\n END IF;\n\n IF work_mem * max_connections > shared_buffers THEN\n RAISE WARNING 'work_mem setting of % is set high for % max_connections please reduce work_mem to % or decrease max_connections to %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers/max_connections), floor(shared_buffers/work_mem);\n ELSIF work_mem * max_connections < (shared_buffers * 0.75) THEN\n RAISE WARNING 'work_mem setting of % is set low for % max_connections you may consider raising work_mem to % or increasing max_connections to %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers/max_connections), floor(shared_buffers/work_mem);\n ELSE\n RAISE NOTICE 'work_mem setting of % and max_connections of % are adequate for shared_buffers of %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers);\n END IF;\n\n IF random_page_cost / seq_page_cost != 1.1 THEN\n RAISE WARNING 'random_page_cost (%) /seq_page_cost (%) should be set to 1.1 for SSD. Change random_page_cost to %', random_page_cost, seq_page_cost, 1.1 * seq_page_cost;\n ELSE\n RAISE NOTICE 'random_page_cost and seq_page_cost set appropriately for SSD';\n END IF;\n\n IF temp_buffers < greatest(pg_size_bytes('128MB'),(maintenance_work_mem / 2)) THEN\n RAISE WARNING 'pgstac makes heavy use of temp tables, consider raising temp_buffers from % to %', pg_size_pretty(temp_buffers), greatest('128MB', pg_size_pretty((shared_buffers / 16)));\n END IF;\n END IF;\n\n RAISE NOTICE 'VALUES FOR PGSTAC VARIABLES';\n RAISE NOTICE 'These can be set either as GUC system variables or by setting in the pgstac_settings table.';\n\n FOR r IN SELECT name, get_setting(name) as setting, CASE WHEN current_setting(concat('pgstac.',name), TRUE) IS NOT NULL THEN concat('pgstac.',name, ' GUC') WHEN value IS NOT NULL THEN 'pgstac_settings table' ELSE 'Not Set' END as loc FROM pgstac_settings LOOP\n RAISE NOTICE '% is set to % from the %', r.name, r.setting, r.loc;\n END LOOP;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pg_cron';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider intalling pg_cron which can be used to automate tasks';\n ELSE\n RAISE NOTICE 'pg_cron % is installed', settingval;\n END IF;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pgstattuple';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider installing the pgstattuple extension which can be used to help maintain tables and indexes.';\n ELSE\n RAISE NOTICE 'pgstattuple % is installed', settingval;\n END IF;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pg_stat_statements';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider installing the pg_stat_statements extension which is very helpful for tracking the types of queries on the system';\n ELSE\n RAISE NOTICE 'pg_stat_statements % is installed', settingval;\n IF current_setting('pg_stat_statements.track_statements', TRUE) IS DISTINCT FROM 'all' THEN\n RAISE WARNING 'SET pg_stat_statements.track_statements TO ''all''; --In order to track statements within functions.';\n END IF;\n END IF;\n\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac, public SET CLIENT_MIN_MESSAGES TO NOTICE;\nCREATE OR REPLACE FUNCTION to_int(jsonb) RETURNS int AS $$\n SELECT floor(($1->>0)::float)::int;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION to_float(jsonb) RETURNS float AS $$\n SELECT ($1->>0)::float;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION to_tstz(jsonb) RETURNS timestamptz AS $$\n SELECT ($1->>0)::timestamptz;\n$$ LANGUAGE SQL IMMUTABLE STRICT SET TIME ZONE 'UTC' COST 5000 PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION to_text(jsonb) RETURNS text AS $$\n SELECT CASE WHEN jsonb_typeof($1) IN ('array','object') THEN $1::text ELSE $1->>0 END;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION to_text_array(jsonb) RETURNS text[] AS $$\n SELECT\n CASE jsonb_typeof($1)\n WHEN 'array' THEN ARRAY(SELECT jsonb_array_elements_text($1))\n ELSE ARRAY[$1->>0]\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION bbox_geom(_bbox jsonb) RETURNS geometry AS $$\nSELECT CASE jsonb_array_length(_bbox)\n WHEN 4 THEN\n ST_SetSRID(ST_MakeEnvelope(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float,\n (_bbox->>3)::float\n ),4326)\n WHEN 6 THEN\n ST_SetSRID(ST_3DMakeBox(\n ST_MakePoint(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float\n ),\n ST_MakePoint(\n (_bbox->>3)::float,\n (_bbox->>4)::float,\n (_bbox->>5)::float\n )\n ),4326)\n ELSE null END;\n;\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION geom_bbox(_geom geometry) RETURNS jsonb AS $$\n SELECT jsonb_build_array(\n st_xmin(_geom),\n st_ymin(_geom),\n st_xmax(_geom),\n st_ymax(_geom)\n );\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION flip_jsonb_array(j jsonb) RETURNS jsonb AS $$\n SELECT jsonb_agg(value) FROM (SELECT value FROM jsonb_array_elements(j) WITH ORDINALITY ORDER BY ordinality DESC) as t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION explode_dotpaths(j jsonb) RETURNS SETOF text[] AS $$\n SELECT string_to_array(p, '.') as e FROM jsonb_array_elements_text(j) p;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION explode_dotpaths_recurse(IN j jsonb) RETURNS SETOF text[] AS $$\n WITH RECURSIVE t AS (\n SELECT e FROM explode_dotpaths(j) e\n UNION ALL\n SELECT e[1:cardinality(e)-1]\n FROM t\n WHERE cardinality(e)>1\n ) SELECT e FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION jsonb_set_nested(j jsonb, path text[], val jsonb) RETURNS jsonb AS $$\nDECLARE\nBEGIN\n IF cardinality(path) > 1 THEN\n FOR i IN 1..(cardinality(path)-1) LOOP\n IF j #> path[:i] IS NULL THEN\n j := jsonb_set_lax(j, path[:i], '{}', TRUE);\n END IF;\n END LOOP;\n END IF;\n RETURN jsonb_set_lax(j, path, val, true);\n\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\n\n\nCREATE OR REPLACE FUNCTION jsonb_include(j jsonb, f jsonb) RETURNS jsonb AS $$\nDECLARE\n includes jsonb := f-> 'include';\n outj jsonb := '{}'::jsonb;\n path text[];\nBEGIN\n IF\n includes IS NULL\n OR jsonb_array_length(includes) = 0\n THEN\n RETURN j;\n ELSE\n includes := includes || '[\"id\",\"collection\"]'::jsonb;\n FOR path IN SELECT explode_dotpaths(includes) LOOP\n outj := jsonb_set_nested(outj, path, j #> path);\n END LOOP;\n END IF;\n RETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_exclude(j jsonb, f jsonb) RETURNS jsonb AS $$\nDECLARE\n excludes jsonb := f-> 'exclude';\n outj jsonb := j;\n path text[];\nBEGIN\n IF\n excludes IS NULL\n OR jsonb_array_length(excludes) = 0\n THEN\n RETURN j;\n ELSE\n FOR path IN SELECT explode_dotpaths(excludes) LOOP\n outj := outj #- path;\n END LOOP;\n END IF;\n RETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_fields(j jsonb, f jsonb DEFAULT '{\"fields\":[]}') RETURNS jsonb AS $$\n SELECT jsonb_exclude(jsonb_include(j, f), f);\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION merge_jsonb(_a jsonb, _b jsonb) RETURNS jsonb AS $$\n SELECT\n CASE\n WHEN _a = '\"𒍟※\"'::jsonb THEN NULL\n WHEN _a IS NULL OR jsonb_typeof(_a) = 'null' THEN _b\n WHEN jsonb_typeof(_a) = 'object' AND jsonb_typeof(_b) = 'object' THEN\n (\n SELECT\n jsonb_strip_nulls(\n jsonb_object_agg(\n key,\n merge_jsonb(a.value, b.value)\n )\n )\n FROM\n jsonb_each(coalesce(_a,'{}'::jsonb)) as a\n FULL JOIN\n jsonb_each(coalesce(_b,'{}'::jsonb)) as b\n USING (key)\n )\n WHEN\n jsonb_typeof(_a) = 'array'\n AND jsonb_typeof(_b) = 'array'\n AND jsonb_array_length(_a) = jsonb_array_length(_b)\n THEN\n (\n SELECT jsonb_agg(m) FROM\n ( SELECT\n merge_jsonb(\n jsonb_array_elements(_a),\n jsonb_array_elements(_b)\n ) as m\n ) as l\n )\n ELSE _a\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION strip_jsonb(_a jsonb, _b jsonb) RETURNS jsonb AS $$\n SELECT\n CASE\n\n WHEN (_a IS NULL OR jsonb_typeof(_a) = 'null') AND _b IS NOT NULL AND jsonb_typeof(_b) != 'null' THEN '\"𒍟※\"'::jsonb\n WHEN _b IS NULL OR jsonb_typeof(_a) = 'null' THEN _a\n WHEN _a = _b AND jsonb_typeof(_a) = 'object' THEN '{}'::jsonb\n WHEN _a = _b THEN NULL\n WHEN jsonb_typeof(_a) = 'object' AND jsonb_typeof(_b) = 'object' THEN\n (\n SELECT\n jsonb_strip_nulls(\n jsonb_object_agg(\n key,\n strip_jsonb(a.value, b.value)\n )\n )\n FROM\n jsonb_each(_a) as a\n FULL JOIN\n jsonb_each(_b) as b\n USING (key)\n )\n WHEN\n jsonb_typeof(_a) = 'array'\n AND jsonb_typeof(_b) = 'array'\n AND jsonb_array_length(_a) = jsonb_array_length(_b)\n THEN\n (\n SELECT jsonb_agg(m) FROM\n ( SELECT\n strip_jsonb(\n jsonb_array_elements(_a),\n jsonb_array_elements(_b)\n ) as m\n ) as l\n )\n ELSE _a\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION nullif_jsonbnullempty(j jsonb) RETURNS jsonb AS $$\n SELECT nullif(nullif(nullif(j,'null'::jsonb),'{}'::jsonb),'[]'::jsonb);\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION jsonb_array_unique(j jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(jsonb_agg(DISTINCT a)) v FROM jsonb_array_elements(j) a;\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_concat_ignorenull(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT coalesce(a,'[]'::jsonb) || coalesce(b,'[]'::jsonb);\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_least(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(least(nullif_jsonbnullempty(a), nullif_jsonbnullempty(b)));\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_greatest(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(greatest(a, b));\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION first_notnull_sfunc(anyelement, anyelement) RETURNS anyelement AS $$\n SELECT COALESCE($1,$2);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE AGGREGATE first_notnull(anyelement)(\n SFUNC = first_notnull_sfunc,\n STYPE = anyelement\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_array_unique_merge(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_concat_ignorenull,\n FINALFUNC = jsonb_array_unique\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_min(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_least\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_max(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_greatest\n);\n/* looks for a geometry in a stac item first from geometry and falling back to bbox */\nCREATE OR REPLACE FUNCTION stac_geom(value jsonb) RETURNS geometry AS $$\nSELECT\n CASE\n WHEN value ? 'intersects' THEN\n ST_GeomFromGeoJSON(value->>'intersects')\n WHEN value ? 'geometry' THEN\n ST_GeomFromGeoJSON(value->>'geometry')\n WHEN value ? 'bbox' THEN\n pgstac.bbox_geom(value->'bbox')\n ELSE NULL\n END as geometry\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION stac_daterange(\n value jsonb\n) RETURNS tstzrange AS $$\nDECLARE\n props jsonb := value;\n dt timestamptz;\n edt timestamptz;\nBEGIN\n IF props ? 'properties' THEN\n props := props->'properties';\n END IF;\n IF\n props ? 'start_datetime'\n AND props->>'start_datetime' IS NOT NULL\n AND props ? 'end_datetime'\n AND props->>'end_datetime' IS NOT NULL\n THEN\n dt := props->>'start_datetime';\n edt := props->>'end_datetime';\n IF dt > edt THEN\n RAISE EXCEPTION 'start_datetime must be < end_datetime';\n END IF;\n ELSE\n dt := props->>'datetime';\n edt := props->>'datetime';\n END IF;\n IF dt is NULL OR edt IS NULL THEN\n RAISE NOTICE 'DT: %, EDT: %', dt, edt;\n RAISE EXCEPTION 'Either datetime (%) or both start_datetime (%) and end_datetime (%) must be set.', props->>'datetime',props->>'start_datetime',props->>'end_datetime';\n END IF;\n RETURN tstzrange(dt, edt, '[]');\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_datetime(value jsonb) RETURNS timestamptz AS $$\n SELECT lower(stac_daterange(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_end_datetime(value jsonb) RETURNS timestamptz AS $$\n SELECT upper(stac_daterange(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE TABLE IF NOT EXISTS stac_extensions(\n url text PRIMARY KEY,\n content jsonb\n);\nCREATE OR REPLACE FUNCTION collection_base_item(content jsonb) RETURNS jsonb AS $$\n SELECT jsonb_build_object(\n 'type', 'Feature',\n 'stac_version', content->'stac_version',\n 'assets', content->'item_assets',\n 'collection', content->'id'\n );\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE TABLE IF NOT EXISTS collections (\n key bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,\n id text GENERATED ALWAYS AS (content->>'id') STORED UNIQUE NOT NULL,\n content JSONB NOT NULL,\n base_item jsonb GENERATED ALWAYS AS (pgstac.collection_base_item(content)) STORED,\n geometry geometry GENERATED ALWAYS AS (pgstac.collection_geom(content)) STORED,\n datetime timestamptz GENERATED ALWAYS AS (pgstac.collection_datetime(content)) STORED,\n end_datetime timestamptz GENERATED ALWAYS AS (pgstac.collection_enddatetime(content)) STORED,\n private jsonb,\n partition_trunc text CHECK (partition_trunc IN ('year', 'month'))\n);\n\nCREATE OR REPLACE FUNCTION collection_base_item(cid text) RETURNS jsonb AS $$\n SELECT pgstac.collection_base_item(content) FROM pgstac.collections WHERE id = cid LIMIT 1;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION create_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION update_collection(data jsonb) RETURNS VOID AS $$\nDECLARE\n out collections%ROWTYPE;\nBEGIN\n UPDATE collections SET content=data WHERE id = data->>'id' RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ON CONFLICT (id) DO\n UPDATE\n SET content=EXCLUDED.content\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION delete_collection(_id text) RETURNS VOID AS $$\nDECLARE\n out collections%ROWTYPE;\nBEGIN\n DELETE FROM collections WHERE id = _id RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION get_collection(id text) RETURNS jsonb AS $$\n SELECT content FROM collections\n WHERE id=$1\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION all_collections() RETURNS jsonb AS $$\n SELECT coalesce(jsonb_agg(content), '[]'::jsonb) FROM collections;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION collection_delete_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n collection_base_partition text := concat('_items_', OLD.key);\nBEGIN\n EXECUTE format($q$\n DELETE FROM partition_stats WHERE partition IN (\n SELECT partition FROM partition_sys_meta\n WHERE collection=%L\n );\n DROP TABLE IF EXISTS %I CASCADE;\n $q$,\n OLD.id,\n collection_base_partition\n );\n RETURN OLD;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER collection_delete_trigger BEFORE DELETE ON collections\nFOR EACH ROW EXECUTE FUNCTION collection_delete_trigger_func();\nCREATE OR REPLACE FUNCTION queryable_signature(n text, c text[]) RETURNS text AS $$\n SELECT concat(n, c);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TABLE queryables (\n id bigint GENERATED ALWAYS AS identity PRIMARY KEY,\n name text NOT NULL,\n collection_ids text[], -- used to determine what partitions to create indexes on\n definition jsonb,\n property_path text,\n property_wrapper text,\n property_index_type text\n);\nCREATE INDEX queryables_name_idx ON queryables (name);\nCREATE INDEX queryables_collection_idx ON queryables USING GIN (collection_ids);\nCREATE INDEX queryables_property_wrapper_idx ON queryables (property_wrapper);\n\nCREATE OR REPLACE FUNCTION pgstac.queryables_constraint_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n allcollections text[];\nBEGIN\n RAISE NOTICE 'Making sure that name/collection is unique for queryables %', NEW;\n IF NEW.collection_ids IS NOT NULL THEN\n IF EXISTS (\n SELECT 1\n FROM unnest(NEW.collection_ids) c\n LEFT JOIN\n collections\n ON (collections.id = c)\n WHERE collections.id IS NULL\n ) THEN\n RAISE foreign_key_violation USING MESSAGE = format(\n 'One or more collections in %s do not exist.', NEW.collection_ids\n );\n RETURN NULL;\n END IF;\n END IF;\n IF TG_OP = 'INSERT' THEN\n IF EXISTS (\n SELECT 1 FROM queryables q\n WHERE\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n )\n ) THEN\n RAISE unique_violation USING MESSAGE = format(\n 'There is already a queryable for %s for a collection in %s: %s',\n NEW.name,\n NEW.collection_ids,\n\t\t\t\t(SELECT json_agg(row_to_json(q)) FROM queryables q WHERE\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n ))\n );\n RETURN NULL;\n END IF;\n END IF;\n IF TG_OP = 'UPDATE' THEN\n IF EXISTS (\n SELECT 1 FROM queryables q\n WHERE\n q.id != NEW.id\n AND\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n )\n ) THEN\n RAISE unique_violation\n USING MESSAGE = format(\n 'There is already a queryable for %s for a collection in %s',\n NEW.name,\n NEW.collection_ids\n );\n RETURN NULL;\n END IF;\n END IF;\n\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER queryables_constraint_insert_trigger\nBEFORE INSERT ON queryables\nFOR EACH ROW EXECUTE PROCEDURE queryables_constraint_triggerfunc();\n\nCREATE TRIGGER queryables_constraint_update_trigger\nBEFORE UPDATE ON queryables\nFOR EACH ROW\nWHEN (NEW.name = OLD.name AND NEW.collection_ids IS DISTINCT FROM OLD.collection_ids)\nEXECUTE PROCEDURE queryables_constraint_triggerfunc();\n\n\nCREATE OR REPLACE FUNCTION array_to_path(arr text[]) RETURNS text AS $$\n SELECT string_agg(\n quote_literal(v),\n '->'\n ) FROM unnest(arr) v;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\n\n\nCREATE OR REPLACE FUNCTION queryable(\n IN dotpath text,\n OUT path text,\n OUT expression text,\n OUT wrapper text,\n OUT nulled_wrapper text\n) AS $$\nDECLARE\n q RECORD;\n path_elements text[];\nBEGIN\n dotpath := replace(dotpath, 'properties.', '');\n IF dotpath = 'start_datetime' THEN\n dotpath := 'datetime';\n END IF;\n IF dotpath IN ('id', 'geometry', 'datetime', 'end_datetime', 'collection') THEN\n path := dotpath;\n expression := dotpath;\n wrapper := NULL;\n RETURN;\n END IF;\n\n SELECT * INTO q FROM queryables\n WHERE\n name=dotpath\n OR name = 'properties.' || dotpath\n OR name = replace(dotpath, 'properties.', '')\n ;\n IF q.property_wrapper IS NULL THEN\n IF q.definition->>'type' = 'number' THEN\n wrapper := 'to_float';\n nulled_wrapper := wrapper;\n ELSIF q.definition->>'format' = 'date-time' THEN\n wrapper := 'to_tstz';\n nulled_wrapper := wrapper;\n ELSE\n nulled_wrapper := NULL;\n wrapper := 'to_text';\n END IF;\n ELSE\n wrapper := q.property_wrapper;\n nulled_wrapper := wrapper;\n END IF;\n IF q.property_path IS NOT NULL THEN\n path := q.property_path;\n ELSE\n path_elements := string_to_array(dotpath, '.');\n IF path_elements[1] IN ('links', 'assets', 'stac_version', 'stac_extensions') THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSIF path_elements[1] = 'properties' THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSE\n path := format($F$content->'properties'->%s$F$, array_to_path(path_elements));\n END IF;\n END IF;\n expression := format('%I(%s)', wrapper, path);\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\nCREATE OR REPLACE FUNCTION unnest_collection(collection_ids text[] DEFAULT NULL) RETURNS SETOF text AS $$\n DECLARE\n BEGIN\n IF collection_ids IS NULL THEN\n RETURN QUERY SELECT id FROM collections;\n END IF;\n RETURN QUERY SELECT unnest(collection_ids);\n END;\n$$ LANGUAGE PLPGSQL STABLE;\n\nCREATE OR REPLACE FUNCTION normalize_indexdef(def text) RETURNS text AS $$\nDECLARE\nBEGIN\n def := btrim(def, ' \\n\\t');\n\tdef := regexp_replace(def, '^CREATE (UNIQUE )?INDEX ([^ ]* )?ON (ONLY )?([^ ]* )?', '', 'i');\n RETURN def;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION indexdef(q queryables) RETURNS text AS $$\n DECLARE\n out text;\n BEGIN\n IF q.name = 'id' THEN\n out := 'CREATE UNIQUE INDEX ON %I USING btree (id)';\n ELSIF q.name = 'datetime' THEN\n out := 'CREATE INDEX ON %I USING btree (datetime DESC, end_datetime)';\n ELSIF q.name = 'geometry' THEN\n out := 'CREATE INDEX ON %I USING gist (geometry)';\n ELSE\n out := format($q$CREATE INDEX ON %%I USING %s (%s(((content -> 'properties'::text) -> %L::text)))$q$,\n lower(COALESCE(q.property_index_type, 'BTREE')),\n lower(COALESCE(q.property_wrapper, 'to_text')),\n q.name\n );\n END IF;\n RETURN btrim(out, ' \\n\\t');\n END;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nDROP VIEW IF EXISTS pgstac_indexes;\nCREATE VIEW pgstac_indexes AS\nSELECT\n i.schemaname,\n i.tablename,\n i.indexname,\n regexp_replace(btrim(replace(replace(indexdef, i.indexname, ''),'pgstac.',''),' \\t\\n'), '[ ]+', ' ', 'g') as idx,\n COALESCE(\n (regexp_match(indexdef, '\\(([a-zA-Z]+)\\)'))[1],\n (regexp_match(indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_-]+)''::text'))[1],\n CASE WHEN indexdef ~* '\\(datetime desc, end_datetime\\)' THEN 'datetime' ELSE NULL END\n ) AS field,\n pg_table_size(i.indexname::text) as index_size,\n pg_size_pretty(pg_table_size(i.indexname::text)) as index_size_pretty\nFROM\n pg_indexes i\nWHERE i.schemaname='pgstac' and i.tablename ~ '_items_' AND indexdef !~* ' only ';\n\nDROP VIEW IF EXISTS pgstac_index_stats;\nCREATE VIEW pgstac_indexes_stats AS\nSELECT\n i.schemaname,\n i.tablename,\n i.indexname,\n indexdef,\n COALESCE(\n (regexp_match(indexdef, '\\(([a-zA-Z]+)\\)'))[1],\n (regexp_match(indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_]+)''::text'))[1],\n CASE WHEN indexdef ~* '\\(datetime desc, end_datetime\\)' THEN 'datetime_end_datetime' ELSE NULL END\n ) AS field,\n pg_table_size(i.indexname::text) as index_size,\n pg_size_pretty(pg_table_size(i.indexname::text)) as index_size_pretty,\n n_distinct,\n most_common_vals::text::text[],\n most_common_freqs::text::text[],\n histogram_bounds::text::text[],\n correlation\nFROM\n pg_indexes i\n LEFT JOIN pg_stats s ON (s.tablename = i.indexname)\nWHERE i.schemaname='pgstac' and i.tablename ~ '_items_';\n\nCREATE OR REPLACE FUNCTION queryable_indexes(\n IN treeroot text DEFAULT 'items',\n IN changes boolean DEFAULT FALSE,\n OUT collection text,\n OUT partition text,\n OUT field text,\n OUT indexname text,\n OUT existing_idx text,\n OUT queryable_idx text\n) RETURNS SETOF RECORD AS $$\nWITH p AS (\n SELECT\n relid::text as partition,\n replace(replace(\n CASE\n WHEN parentrelid::regclass::text='items' THEN pg_get_expr(c.relpartbound, c.oid)\n ELSE pg_get_expr(parent.relpartbound, parent.oid)\n END,\n 'FOR VALUES IN (''',''), ''')',\n ''\n ) AS collection\n FROM pg_partition_tree(treeroot)\n JOIN pg_class c ON (relid::regclass = c.oid)\n JOIN pg_class parent ON (parentrelid::regclass = parent.oid AND isleaf)\n ), i AS (\n SELECT\n partition,\n indexname,\n regexp_replace(btrim(replace(replace(indexdef, indexname, ''),'pgstac.',''),' \\t\\n'), '[ ]+', ' ', 'g') as iidx,\n COALESCE(\n (regexp_match(indexdef, '\\(([a-zA-Z]+)\\)'))[1],\n (regexp_match(indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_-]+)''::text'))[1],\n CASE WHEN indexdef ~* '\\(datetime desc, end_datetime\\)' THEN 'datetime' ELSE NULL END\n ) AS field\n FROM\n pg_indexes\n JOIN p ON (tablename=partition)\n ), q AS (\n SELECT\n name AS field,\n collection,\n partition,\n format(indexdef(queryables), partition) as qidx\n FROM queryables, unnest_collection(queryables.collection_ids) collection\n JOIN p USING (collection)\n WHERE property_index_type IS NOT NULL OR name IN ('datetime','geometry','id')\n )\n SELECT\n collection,\n partition,\n field,\n indexname,\n iidx as existing_idx,\n qidx as queryable_idx\n FROM i FULL JOIN q USING (field, partition)\n WHERE CASE WHEN changes THEN lower(iidx) IS DISTINCT FROM lower(qidx) ELSE TRUE END;\n;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION maintain_index(\n indexname text,\n queryable_idx text,\n dropindexes boolean DEFAULT FALSE,\n rebuildindexes boolean DEFAULT FALSE,\n idxconcurrently boolean DEFAULT FALSE\n) RETURNS VOID AS $$\nDECLARE\nBEGIN\n IF indexname IS NOT NULL THEN\n IF dropindexes OR queryable_idx IS NOT NULL THEN\n EXECUTE format('DROP INDEX IF EXISTS %I;', indexname);\n ELSIF rebuildindexes THEN\n IF idxconcurrently THEN\n EXECUTE format('REINDEX INDEX CONCURRENTLY %I;', indexname);\n ELSE\n EXECUTE format('REINDEX INDEX CONCURRENTLY %I;', indexname);\n END IF;\n END IF;\n END IF;\n IF queryable_idx IS NOT NULL THEN\n IF idxconcurrently THEN\n EXECUTE replace(queryable_idx, 'INDEX', 'INDEX CONCURRENTLY');\n ELSE EXECUTE queryable_idx;\n END IF;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\n\nset check_function_bodies to off;\nCREATE OR REPLACE FUNCTION maintain_partition_queries(\n part text DEFAULT 'items',\n dropindexes boolean DEFAULT FALSE,\n rebuildindexes boolean DEFAULT FALSE,\n idxconcurrently boolean DEFAULT FALSE\n) RETURNS SETOF text AS $$\nDECLARE\n rec record;\n q text;\nBEGIN\n FOR rec IN (\n SELECT * FROM queryable_indexes(part,true)\n ) LOOP\n q := format(\n 'SELECT maintain_index(\n %L,%L,%L,%L,%L\n );',\n rec.indexname,\n rec.queryable_idx,\n dropindexes,\n rebuildindexes,\n idxconcurrently\n );\n RAISE NOTICE 'Q: %', q;\n RETURN NEXT q;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION maintain_partitions(\n part text DEFAULT 'items',\n dropindexes boolean DEFAULT FALSE,\n rebuildindexes boolean DEFAULT FALSE\n) RETURNS VOID AS $$\n WITH t AS (\n SELECT run_or_queue(q) FROM maintain_partition_queries(part, dropindexes, rebuildindexes) q\n ) SELECT count(*) FROM t;\n$$ LANGUAGE SQL;\n\n\nCREATE OR REPLACE FUNCTION queryables_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\nBEGIN\n PERFORM maintain_partitions();\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER queryables_trigger AFTER INSERT OR UPDATE ON queryables\nFOR EACH STATEMENT EXECUTE PROCEDURE queryables_trigger_func();\n\n\nCREATE OR REPLACE FUNCTION get_queryables(_collection_ids text[] DEFAULT NULL) RETURNS jsonb AS $$\nDECLARE\nBEGIN\n -- Build up queryables if the input contains valid collection ids or is empty\n IF EXISTS (\n SELECT 1 FROM collections\n WHERE\n _collection_ids IS NULL\n OR cardinality(_collection_ids) = 0\n OR id = ANY(_collection_ids)\n )\n THEN\n RETURN (\n WITH base AS (\n SELECT\n unnest(collection_ids) as collection_id,\n name,\n coalesce(definition, '{\"type\":\"string\"}'::jsonb) as definition\n FROM queryables\n WHERE\n _collection_ids IS NULL OR\n _collection_ids = '{}'::text[] OR\n _collection_ids && collection_ids\n UNION ALL\n SELECT null, name, coalesce(definition, '{\"type\":\"string\"}'::jsonb) as definition\n FROM queryables WHERE collection_ids IS NULL OR collection_ids = '{}'::text[]\n ), g AS (\n SELECT\n name,\n first_notnull(definition) as definition,\n jsonb_array_unique_merge(definition->'enum') as enum,\n jsonb_min(definition->'minimum') as minimum,\n jsonb_min(definition->'maxiumn') as maximum\n FROM base\n GROUP BY 1\n )\n SELECT\n jsonb_build_object(\n '$schema', 'http://json-schema.org/draft-07/schema#',\n '$id', '',\n 'type', 'object',\n 'title', 'STAC Queryables.',\n 'properties', jsonb_object_agg(\n name,\n definition\n ||\n jsonb_strip_nulls(jsonb_build_object(\n 'enum', enum,\n 'minimum', minimum,\n 'maximum', maximum\n ))\n ),\n 'additionalProperties', pgstac.additional_properties()\n )\n FROM g\n );\n ELSE\n RETURN NULL;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\nCREATE OR REPLACE FUNCTION get_queryables(_collection text DEFAULT NULL) RETURNS jsonb AS $$\n SELECT\n CASE\n WHEN _collection IS NULL THEN get_queryables(NULL::text[])\n ELSE get_queryables(ARRAY[_collection])\n END\n ;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_queryables() RETURNS jsonb AS $$\n SELECT get_queryables(NULL::text[]);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION schema_qualify_refs(url text, j jsonb) returns jsonb as $$\n SELECT regexp_replace(j::text, '\"\\$ref\": \"#', concat('\"$ref\": \"', url, '#'), 'g')::jsonb;\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE VIEW stac_extension_queryables AS\nSELECT DISTINCT key as name, schema_qualify_refs(e.url, j.value) as definition FROM stac_extensions e, jsonb_each(e.content->'definitions'->'fields'->'properties') j;\n\n\nCREATE OR REPLACE FUNCTION missing_queryables(_collection text, _tablesample float DEFAULT 5, minrows float DEFAULT 10) RETURNS TABLE(collection text, name text, definition jsonb, property_wrapper text) AS $$\nDECLARE\n q text;\n _partition text;\n explain_json json;\n psize float;\n estrows float;\nBEGIN\n SELECT format('_items_%s', key) INTO _partition FROM collections WHERE id=_collection;\n\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM %I;', _partition)\n INTO explain_json;\n psize := explain_json->0->'Plan'->'Plan Rows';\n estrows := _tablesample * .01 * psize;\n IF estrows < minrows THEN\n _tablesample := least(100,greatest(_tablesample, (estrows / psize) / 100));\n RAISE NOTICE '%', (psize / estrows) / 100;\n END IF;\n RAISE NOTICE 'Using tablesample % to find missing queryables from % % that has ~% rows estrows: %', _tablesample, _collection, _partition, psize, estrows;\n\n q := format(\n $q$\n WITH q AS (\n SELECT * FROM queryables\n WHERE\n collection_ids IS NULL\n OR %L = ANY(collection_ids)\n ), t AS (\n SELECT\n content->'properties' AS properties\n FROM\n %I\n TABLESAMPLE SYSTEM(%L)\n ), p AS (\n SELECT DISTINCT ON (key)\n key,\n value,\n s.definition\n FROM t\n JOIN LATERAL jsonb_each(properties) ON TRUE\n LEFT JOIN q ON (q.name=key)\n LEFT JOIN stac_extension_queryables s ON (s.name=key)\n WHERE q.definition IS NULL\n )\n SELECT\n %L,\n key,\n COALESCE(definition, jsonb_build_object('type',jsonb_typeof(value))) as definition,\n CASE\n WHEN definition->>'type' = 'integer' THEN 'to_int'\n WHEN COALESCE(definition->>'type', jsonb_typeof(value)) = 'number' THEN 'to_float'\n WHEN COALESCE(definition->>'type', jsonb_typeof(value)) = 'array' THEN 'to_text_array'\n ELSE 'to_text'\n END\n FROM p;\n $q$,\n _collection,\n _partition,\n _tablesample,\n _collection\n );\n RETURN QUERY EXECUTE q;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION missing_queryables(_tablesample float DEFAULT 5) RETURNS TABLE(collection_ids text[], name text, definition jsonb, property_wrapper text) AS $$\n SELECT\n array_agg(collection),\n name,\n definition,\n property_wrapper\n FROM\n collections\n JOIN LATERAL\n missing_queryables(id, _tablesample) c\n ON TRUE\n GROUP BY\n 2,3,4\n ORDER BY 2,1\n ;\n$$ LANGUAGE SQL;\nCREATE OR REPLACE FUNCTION parse_dtrange(\n _indate jsonb,\n relative_base timestamptz DEFAULT date_trunc('hour', CURRENT_TIMESTAMP)\n) RETURNS tstzrange AS $$\nDECLARE\n timestrs text[];\n s timestamptz;\n e timestamptz;\nBEGIN\n timestrs :=\n CASE\n WHEN _indate ? 'timestamp' THEN\n ARRAY[_indate->>'timestamp']\n WHEN _indate ? 'interval' THEN\n to_text_array(_indate->'interval')\n WHEN jsonb_typeof(_indate) = 'array' THEN\n to_text_array(_indate)\n ELSE\n regexp_split_to_array(\n _indate->>0,\n '/'\n )\n END;\n RAISE NOTICE 'TIMESTRS %', timestrs;\n IF cardinality(timestrs) = 1 THEN\n IF timestrs[1] ILIKE 'P%' THEN\n RETURN tstzrange(relative_base - upper(timestrs[1])::interval, relative_base, '[)');\n END IF;\n s := timestrs[1]::timestamptz;\n RETURN tstzrange(s, s, '[]');\n END IF;\n\n IF cardinality(timestrs) != 2 THEN\n RAISE EXCEPTION 'Timestamp cannot have more than 2 values';\n END IF;\n\n IF timestrs[1] = '..' OR timestrs[1] = '' THEN\n s := '-infinity'::timestamptz;\n e := timestrs[2]::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] = '..' OR timestrs[2] = '' THEN\n s := timestrs[1]::timestamptz;\n e := 'infinity'::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[1] ILIKE 'P%' AND timestrs[2] NOT ILIKE 'P%' THEN\n e := timestrs[2]::timestamptz;\n s := e - upper(timestrs[1])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] ILIKE 'P%' AND timestrs[1] NOT ILIKE 'P%' THEN\n s := timestrs[1]::timestamptz;\n e := s + upper(timestrs[2])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n s := timestrs[1]::timestamptz;\n e := timestrs[2]::timestamptz;\n\n RETURN tstzrange(s,e,'[)');\n\n RETURN NULL;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT PARALLEL SAFE SET TIME ZONE 'UTC';\n\nCREATE OR REPLACE FUNCTION parse_dtrange(\n _indate text,\n relative_base timestamptz DEFAULT CURRENT_TIMESTAMP\n) RETURNS tstzrange AS $$\n SELECT parse_dtrange(to_jsonb(_indate), relative_base);\n$$ LANGUAGE SQL STABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION temporal_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\n ll text := 'datetime';\n lh text := 'end_datetime';\n rrange tstzrange;\n rl text;\n rh text;\n outq text;\nBEGIN\n rrange := parse_dtrange(args->1);\n RAISE NOTICE 'Constructing temporal query OP: %, ARGS: %, RRANGE: %', op, args, rrange;\n op := lower(op);\n rl := format('%L::timestamptz', lower(rrange));\n rh := format('%L::timestamptz', upper(rrange));\n outq := CASE op\n WHEN 't_before' THEN 'lh < rl'\n WHEN 't_after' THEN 'll > rh'\n WHEN 't_meets' THEN 'lh = rl'\n WHEN 't_metby' THEN 'll = rh'\n WHEN 't_overlaps' THEN 'll < rl AND rl < lh < rh'\n WHEN 't_overlappedby' THEN 'rl < ll < rh AND lh > rh'\n WHEN 't_starts' THEN 'll = rl AND lh < rh'\n WHEN 't_startedby' THEN 'll = rl AND lh > rh'\n WHEN 't_during' THEN 'll > rl AND lh < rh'\n WHEN 't_contains' THEN 'll < rl AND lh > rh'\n WHEN 't_finishes' THEN 'll > rl AND lh = rh'\n WHEN 't_finishedby' THEN 'll < rl AND lh = rh'\n WHEN 't_equals' THEN 'll = rl AND lh = rh'\n WHEN 't_disjoint' THEN 'NOT (ll <= rh AND lh >= rl)'\n WHEN 't_intersects' THEN 'll <= rh AND lh >= rl'\n WHEN 'anyinteracts' THEN 'll <= rh AND lh >= rl'\n END;\n outq := regexp_replace(outq, '\\mll\\M', ll);\n outq := regexp_replace(outq, '\\mlh\\M', lh);\n outq := regexp_replace(outq, '\\mrl\\M', rl);\n outq := regexp_replace(outq, '\\mrh\\M', rh);\n outq := format('(%s)', outq);\n RETURN outq;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\n\n\nCREATE OR REPLACE FUNCTION spatial_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\n geom text;\n j jsonb := args->1;\nBEGIN\n op := lower(op);\n RAISE NOTICE 'Constructing spatial query OP: %, ARGS: %', op, args;\n IF op NOT IN ('s_equals','s_disjoint','s_touches','s_within','s_overlaps','s_crosses','s_intersects','intersects','s_contains') THEN\n RAISE EXCEPTION 'Spatial Operator % Not Supported', op;\n END IF;\n op := regexp_replace(op, '^s_', 'st_');\n IF op = 'intersects' THEN\n op := 'st_intersects';\n END IF;\n -- Convert geometry to WKB string\n IF j ? 'type' AND j ? 'coordinates' THEN\n geom := st_geomfromgeojson(j)::text;\n ELSIF jsonb_typeof(j) = 'array' THEN\n geom := bbox_geom(j)::text;\n END IF;\n\n RETURN format('%s(geometry, %L::geometry)', op, geom);\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION query_to_cql2(q jsonb) RETURNS jsonb AS $$\n-- Translates anything passed in through the deprecated \"query\" into equivalent CQL2\nWITH t AS (\n SELECT key as property, value as ops\n FROM jsonb_each(q)\n), t2 AS (\n SELECT property, (jsonb_each(ops)).*\n FROM t WHERE jsonb_typeof(ops) = 'object'\n UNION ALL\n SELECT property, 'eq', ops\n FROM t WHERE jsonb_typeof(ops) != 'object'\n)\nSELECT\n jsonb_strip_nulls(jsonb_build_object(\n 'op', 'and',\n 'args', jsonb_agg(\n jsonb_build_object(\n 'op', key,\n 'args', jsonb_build_array(\n jsonb_build_object('property',property),\n value\n )\n )\n )\n )\n) as qcql FROM t2\n;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\nCREATE OR REPLACE FUNCTION cql1_to_cql2(j jsonb) RETURNS jsonb AS $$\nDECLARE\n args jsonb;\n ret jsonb;\nBEGIN\n RAISE NOTICE 'CQL1_TO_CQL2: %', j;\n IF j ? 'filter' THEN\n RETURN cql1_to_cql2(j->'filter');\n END IF;\n IF j ? 'property' THEN\n RETURN j;\n END IF;\n IF jsonb_typeof(j) = 'array' THEN\n SELECT jsonb_agg(cql1_to_cql2(el)) INTO args FROM jsonb_array_elements(j) el;\n RETURN args;\n END IF;\n IF jsonb_typeof(j) = 'number' THEN\n RETURN j;\n END IF;\n IF jsonb_typeof(j) = 'string' THEN\n RETURN j;\n END IF;\n\n IF jsonb_typeof(j) = 'object' THEN\n SELECT jsonb_build_object(\n 'op', key,\n 'args', cql1_to_cql2(value)\n ) INTO ret\n FROM jsonb_each(j)\n WHERE j IS NOT NULL;\n RETURN ret;\n END IF;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE STRICT;\n\nCREATE TABLE cql2_ops (\n op text PRIMARY KEY,\n template text,\n types text[]\n);\nINSERT INTO cql2_ops (op, template, types) VALUES\n ('eq', '%s = %s', NULL),\n ('neq', '%s != %s', NULL),\n ('ne', '%s != %s', NULL),\n ('!=', '%s != %s', NULL),\n ('<>', '%s != %s', NULL),\n ('lt', '%s < %s', NULL),\n ('lte', '%s <= %s', NULL),\n ('gt', '%s > %s', NULL),\n ('gte', '%s >= %s', NULL),\n ('le', '%s <= %s', NULL),\n ('ge', '%s >= %s', NULL),\n ('=', '%s = %s', NULL),\n ('<', '%s < %s', NULL),\n ('<=', '%s <= %s', NULL),\n ('>', '%s > %s', NULL),\n ('>=', '%s >= %s', NULL),\n ('like', '%s LIKE %s', NULL),\n ('ilike', '%s ILIKE %s', NULL),\n ('+', '%s + %s', NULL),\n ('-', '%s - %s', NULL),\n ('*', '%s * %s', NULL),\n ('/', '%s / %s', NULL),\n ('not', 'NOT (%s)', NULL),\n ('between', '%s BETWEEN %s AND %s', NULL),\n ('isnull', '%s IS NULL', NULL),\n ('upper', 'upper(%s)', NULL),\n ('lower', 'lower(%s)', NULL),\n ('casei', 'upper(%s)', NULL),\n ('accenti', 'unaccent(%s)', NULL)\nON CONFLICT (op) DO UPDATE\n SET\n template = EXCLUDED.template\n;\n\n\nCREATE OR REPLACE FUNCTION cql2_query(j jsonb, wrapper text DEFAULT NULL) RETURNS text AS $$\n#variable_conflict use_variable\nDECLARE\n args jsonb := j->'args';\n arg jsonb;\n op text := lower(j->>'op');\n cql2op RECORD;\n literal text;\n _wrapper text;\n leftarg text;\n rightarg text;\n prop text;\n extra_props bool := pgstac.additional_properties();\nBEGIN\n IF j IS NULL OR (op IS NOT NULL AND args IS NULL) THEN\n RETURN NULL;\n END IF;\n RAISE NOTICE 'CQL2_QUERY: %', j;\n\n -- check if all properties are represented in the queryables\n IF NOT extra_props THEN\n FOR prop IN\n SELECT DISTINCT p->>0\n FROM jsonb_path_query(j, 'strict $.**.property') p\n WHERE p->>0 NOT IN ('id', 'datetime', 'geometry', 'end_datetime', 'collection')\n LOOP\n IF (queryable(prop)).nulled_wrapper IS NULL THEN\n RAISE EXCEPTION 'Term % is not found in queryables.', prop;\n END IF;\n END LOOP;\n END IF;\n\n IF j ? 'filter' THEN\n RETURN cql2_query(j->'filter');\n END IF;\n\n IF j ? 'upper' THEN\n RETURN cql2_query(jsonb_build_object('op', 'upper', 'args', j->'upper'));\n END IF;\n\n IF j ? 'lower' THEN\n RETURN cql2_query(jsonb_build_object('op', 'lower', 'args', j->'lower'));\n END IF;\n\n -- Temporal Query\n IF op ilike 't_%' or op = 'anyinteracts' THEN\n RETURN temporal_op_query(op, args);\n END IF;\n\n -- If property is a timestamp convert it to text to use with\n -- general operators\n IF j ? 'timestamp' THEN\n RETURN format('%L::timestamptz', to_tstz(j->'timestamp'));\n END IF;\n IF j ? 'interval' THEN\n RAISE EXCEPTION 'Please use temporal operators when using intervals.';\n RETURN NONE;\n END IF;\n\n -- Spatial Query\n IF op ilike 's_%' or op = 'intersects' THEN\n RETURN spatial_op_query(op, args);\n END IF;\n\n IF op IN ('a_equals','a_contains','a_contained_by','a_overlaps') THEN\n IF args->0 ? 'property' THEN\n leftarg := format('to_text_array(%s)', (queryable(args->0->>'property')).path);\n END IF;\n IF args->1 ? 'property' THEN\n rightarg := format('to_text_array(%s)', (queryable(args->1->>'property')).path);\n END IF;\n RETURN FORMAT(\n '%s %s %s',\n COALESCE(leftarg, quote_literal(to_text_array(args->0))),\n CASE op\n WHEN 'a_equals' THEN '='\n WHEN 'a_contains' THEN '@>'\n WHEN 'a_contained_by' THEN '<@'\n WHEN 'a_overlaps' THEN '&&'\n END,\n COALESCE(rightarg, quote_literal(to_text_array(args->1)))\n );\n END IF;\n\n IF op = 'in' THEN\n RAISE NOTICE 'IN : % % %', args, jsonb_build_array(args->0), args->1;\n args := jsonb_build_array(args->0) || (args->1);\n RAISE NOTICE 'IN2 : %', args;\n END IF;\n\n\n\n IF op = 'between' THEN\n args = jsonb_build_array(\n args->0,\n args->1,\n args->2\n );\n END IF;\n\n -- Make sure that args is an array and run cql2_query on\n -- each element of the array\n RAISE NOTICE 'ARGS PRE: %', args;\n IF j ? 'args' THEN\n IF jsonb_typeof(args) != 'array' THEN\n args := jsonb_build_array(args);\n END IF;\n\n IF jsonb_path_exists(args, '$[*] ? (@.property == \"id\" || @.property == \"datetime\" || @.property == \"end_datetime\" || @.property == \"collection\")') THEN\n wrapper := NULL;\n ELSE\n -- if any of the arguments are a property, try to get the property_wrapper\n FOR arg IN SELECT jsonb_path_query(args, '$[*] ? (@.property != null)') LOOP\n RAISE NOTICE 'Arg: %', arg;\n wrapper := (queryable(arg->>'property')).nulled_wrapper;\n RAISE NOTICE 'Property: %, Wrapper: %', arg, wrapper;\n IF wrapper IS NOT NULL THEN\n EXIT;\n END IF;\n END LOOP;\n\n -- if the property was not in queryables, see if any args were numbers\n IF\n wrapper IS NULL\n AND jsonb_path_exists(args, '$[*] ? (@.type()==\"number\")')\n THEN\n wrapper := 'to_float';\n END IF;\n wrapper := coalesce(wrapper, 'to_text');\n END IF;\n\n SELECT jsonb_agg(cql2_query(a, wrapper))\n INTO args\n FROM jsonb_array_elements(args) a;\n END IF;\n RAISE NOTICE 'ARGS: %', args;\n\n IF op IN ('and', 'or') THEN\n RETURN\n format(\n '(%s)',\n array_to_string(to_text_array(args), format(' %s ', upper(op)))\n );\n END IF;\n\n IF op = 'in' THEN\n RAISE NOTICE 'IN -- % %', args->0, to_text(args->0);\n RETURN format(\n '%s IN (%s)',\n to_text(args->0),\n array_to_string((to_text_array(args))[2:], ',')\n );\n END IF;\n\n -- Look up template from cql2_ops\n IF j ? 'op' THEN\n SELECT * INTO cql2op FROM cql2_ops WHERE cql2_ops.op ilike op;\n IF FOUND THEN\n -- If specific index set in queryables for a property cast other arguments to that type\n\n RETURN format(\n cql2op.template,\n VARIADIC (to_text_array(args))\n );\n ELSE\n RAISE EXCEPTION 'Operator % Not Supported.', op;\n END IF;\n END IF;\n\n\n IF wrapper IS NOT NULL THEN\n RAISE NOTICE 'Wrapping % with %', j, wrapper;\n IF j ? 'property' THEN\n RETURN format('%I(%s)', wrapper, (queryable(j->>'property')).path);\n ELSE\n RETURN format('%I(%L)', wrapper, j);\n END IF;\n ELSIF j ? 'property' THEN\n RETURN quote_ident(j->>'property');\n END IF;\n\n RETURN quote_literal(to_text(j));\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION paging_dtrange(\n j jsonb\n) RETURNS tstzrange AS $$\nDECLARE\n op text;\n filter jsonb := j->'filter';\n dtrange tstzrange := tstzrange('-infinity'::timestamptz,'infinity'::timestamptz);\n sdate timestamptz := '-infinity'::timestamptz;\n edate timestamptz := 'infinity'::timestamptz;\n jpitem jsonb;\nBEGIN\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n END IF;\n IF NOT (filter @? '$.**.op ? (@ == \"or\" || @ == \"not\")') THEN\n FOR jpitem IN SELECT j FROM jsonb_path_query(filter,'strict $.** ? (@.args[*].property == \"datetime\")'::jsonpath) j LOOP\n op := lower(jpitem->>'op');\n dtrange := parse_dtrange(jpitem->'args'->1);\n IF op IN ('<=', 'lt', 'lte', '<', 'le', 't_before') THEN\n sdate := greatest(sdate,'-infinity');\n edate := least(edate, upper(dtrange));\n ELSIF op IN ('>=', '>', 'gt', 'gte', 'ge', 't_after') THEN\n edate := least(edate, 'infinity');\n sdate := greatest(sdate, lower(dtrange));\n ELSIF op IN ('=', 'eq') THEN\n edate := least(edate, upper(dtrange));\n sdate := greatest(sdate, lower(dtrange));\n END IF;\n RAISE NOTICE '2 OP: %, ARGS: %, DTRANGE: %, SDATE: %, EDATE: %', op, jpitem->'args'->1, dtrange, sdate, edate;\n END LOOP;\n END IF;\n IF sdate > edate THEN\n RETURN 'empty'::tstzrange;\n END IF;\n RETURN tstzrange(sdate,edate, '[]');\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT SET TIME ZONE 'UTC';\n\nCREATE OR REPLACE FUNCTION paging_collections(\n IN j jsonb\n) RETURNS text[] AS $$\nDECLARE\n filter jsonb := j->'filter';\n jpitem jsonb;\n op text;\n args jsonb;\n arg jsonb;\n collections text[];\nBEGIN\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n END IF;\n IF NOT (filter @? '$.**.op ? (@ == \"or\" || @ == \"not\")') THEN\n FOR jpitem IN SELECT j FROM jsonb_path_query(filter,'strict $.** ? (@.args[*].property == \"collection\")'::jsonpath) j LOOP\n RAISE NOTICE 'JPITEM: %', jpitem;\n op := jpitem->>'op';\n args := jpitem->'args';\n IF op IN ('=', 'eq', 'in') THEN\n FOR arg IN SELECT a FROM jsonb_array_elements(args) a LOOP\n IF jsonb_typeof(arg) IN ('string', 'array') THEN\n RAISE NOTICE 'arg: %, collections: %', arg, collections;\n IF collections IS NULL OR collections = '{}'::text[] THEN\n collections := to_text_array(arg);\n ELSE\n collections := array_intersection(collections, to_text_array(arg));\n END IF;\n END IF;\n END LOOP;\n END IF;\n END LOOP;\n END IF;\n IF collections = '{}'::text[] THEN\n RETURN NULL;\n END IF;\n RETURN collections;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\nCREATE TABLE items (\n id text NOT NULL,\n geometry geometry NOT NULL,\n collection text NOT NULL,\n datetime timestamptz NOT NULL,\n end_datetime timestamptz NOT NULL,\n content JSONB NOT NULL,\n private jsonb\n)\nPARTITION BY LIST (collection)\n;\n\nCREATE INDEX \"datetime_idx\" ON items USING BTREE (datetime DESC, end_datetime ASC);\nCREATE INDEX \"geometry_idx\" ON items USING GIST (geometry);\n\nCREATE STATISTICS datetime_stats (dependencies) on datetime, end_datetime from items;\n\nALTER TABLE items ADD CONSTRAINT items_collections_fk FOREIGN KEY (collection) REFERENCES collections(id) ON DELETE CASCADE DEFERRABLE;\n\nCREATE OR REPLACE FUNCTION partition_after_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p text;\n t timestamptz := clock_timestamp();\nBEGIN\n RAISE NOTICE 'Updating partition stats %', t;\n FOR p IN SELECT DISTINCT partition\n FROM newdata n JOIN partition_sys_meta p\n ON (n.collection=p.collection AND n.datetime <@ p.partition_dtrange)\n LOOP\n PERFORM run_or_queue(format('SELECT update_partition_stats(%L, %L);', p, true));\n END LOOP;\n IF TG_OP IN ('DELETE','UPDATE') THEN\n DELETE FROM format_item_cache c USING newdata n WHERE c.collection = n.collection AND c.id = n.id;\n END IF;\n RAISE NOTICE 't: % %', t, clock_timestamp() - t;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE TRIGGER items_after_insert_trigger\nAFTER INSERT ON items\nREFERENCING NEW TABLE AS newdata\nFOR EACH STATEMENT\nEXECUTE FUNCTION partition_after_triggerfunc();\n\nCREATE TRIGGER items_after_update_trigger\nAFTER DELETE ON items\nREFERENCING OLD TABLE AS newdata\nFOR EACH STATEMENT\nEXECUTE FUNCTION partition_after_triggerfunc();\n\nCREATE TRIGGER items_after_delete_trigger\nAFTER UPDATE ON items\nREFERENCING NEW TABLE AS newdata\nFOR EACH STATEMENT\nEXECUTE FUNCTION partition_after_triggerfunc();\n\n\nCREATE OR REPLACE FUNCTION content_slim(_item jsonb) RETURNS jsonb AS $$\n SELECT strip_jsonb(_item - '{id,geometry,collection,type}'::text[], collection_base_item(_item->>'collection')) - '{id,geometry,collection,type}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_dehydrate(content jsonb) RETURNS items AS $$\n SELECT\n content->>'id' as id,\n stac_geom(content) as geometry,\n content->>'collection' as collection,\n stac_datetime(content) as datetime,\n stac_end_datetime(content) as end_datetime,\n content_slim(content) as content,\n null::jsonb as private\n ;\n$$ LANGUAGE SQL STABLE;\n\nCREATE OR REPLACE FUNCTION include_field(f text, fields jsonb DEFAULT '{}'::jsonb) RETURNS boolean AS $$\nDECLARE\n includes jsonb := fields->'include';\n excludes jsonb := fields->'exclude';\nBEGIN\n IF f IS NULL THEN\n RETURN NULL;\n END IF;\n\n\n IF\n jsonb_typeof(excludes) = 'array'\n AND jsonb_array_length(excludes)>0\n AND excludes ? f\n THEN\n RETURN FALSE;\n END IF;\n\n IF\n (\n jsonb_typeof(includes) = 'array'\n AND jsonb_array_length(includes) > 0\n AND includes ? f\n ) OR\n (\n includes IS NULL\n OR jsonb_typeof(includes) = 'null'\n OR jsonb_array_length(includes) = 0\n )\n THEN\n RETURN TRUE;\n END IF;\n\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nDROP FUNCTION IF EXISTS content_hydrate(jsonb, jsonb, jsonb);\nCREATE OR REPLACE FUNCTION content_hydrate(\n _item jsonb,\n _base_item jsonb,\n fields jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\n SELECT merge_jsonb(\n jsonb_fields(_item, fields),\n jsonb_fields(_base_item, fields)\n );\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION content_hydrate(_item items, _collection collections, fields jsonb DEFAULT '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\n content jsonb;\n base_item jsonb := _collection.base_item;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry, 20)::jsonb;\n END IF;\n output := content_hydrate(\n jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'collection', _item.collection,\n 'type', 'Feature'\n ) || _item.content,\n _collection.base_item,\n fields\n );\n\n RETURN output;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_nonhydrated(\n _item items,\n fields jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry, 20)::jsonb;\n END IF;\n output := jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'collection', _item.collection,\n 'type', 'Feature'\n ) || _item.content;\n RETURN output;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_hydrate(_item items, fields jsonb DEFAULT '{}'::jsonb) RETURNS jsonb AS $$\n SELECT content_hydrate(\n _item,\n (SELECT c FROM collections c WHERE id=_item.collection LIMIT 1),\n fields\n );\n$$ LANGUAGE SQL STABLE;\n\n\nCREATE UNLOGGED TABLE items_staging (\n content JSONB NOT NULL\n);\nCREATE UNLOGGED TABLE items_staging_ignore (\n content JSONB NOT NULL\n);\nCREATE UNLOGGED TABLE items_staging_upsert (\n content JSONB NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION items_staging_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p record;\n _partitions text[];\n part text;\n ts timestamptz := clock_timestamp();\n nrows int;\nBEGIN\n RAISE NOTICE 'Creating Partitions. %', clock_timestamp() - ts;\n\n FOR part IN WITH t AS (\n SELECT\n n.content->>'collection' as collection,\n stac_daterange(n.content->'properties') as dtr,\n partition_trunc\n FROM newdata n JOIN collections ON (n.content->>'collection'=collections.id)\n ), p AS (\n SELECT\n collection,\n COALESCE(date_trunc(partition_trunc::text, lower(dtr)),'-infinity') as d,\n tstzrange(min(lower(dtr)),max(lower(dtr)),'[]') as dtrange,\n tstzrange(min(upper(dtr)),max(upper(dtr)),'[]') as edtrange\n FROM t\n GROUP BY 1,2\n ) SELECT check_partition(collection, dtrange, edtrange) FROM p LOOP\n RAISE NOTICE 'Partition %', part;\n END LOOP;\n\n RAISE NOTICE 'Creating temp table with data to be added. %', clock_timestamp() - ts;\n DROP TABLE IF EXISTS tmpdata;\n CREATE TEMP TABLE tmpdata ON COMMIT DROP AS\n SELECT\n (content_dehydrate(content)).*\n FROM newdata;\n GET DIAGNOSTICS nrows = ROW_COUNT;\n RAISE NOTICE 'Added % rows to tmpdata. %', nrows, clock_timestamp() - ts;\n\n RAISE NOTICE 'Doing the insert. %', clock_timestamp() - ts;\n IF TG_TABLE_NAME = 'items_staging' THEN\n INSERT INTO items\n SELECT * FROM tmpdata;\n GET DIAGNOSTICS nrows = ROW_COUNT;\n RAISE NOTICE 'Inserted % rows to items. %', nrows, clock_timestamp() - ts;\n ELSIF TG_TABLE_NAME = 'items_staging_ignore' THEN\n INSERT INTO items\n SELECT * FROM tmpdata\n ON CONFLICT DO NOTHING;\n GET DIAGNOSTICS nrows = ROW_COUNT;\n RAISE NOTICE 'Inserted % rows to items. %', nrows, clock_timestamp() - ts;\n ELSIF TG_TABLE_NAME = 'items_staging_upsert' THEN\n DELETE FROM items i USING tmpdata s\n WHERE\n i.id = s.id\n AND i.collection = s.collection\n AND i IS DISTINCT FROM s\n ;\n GET DIAGNOSTICS nrows = ROW_COUNT;\n RAISE NOTICE 'Deleted % rows from items. %', nrows, clock_timestamp() - ts;\n INSERT INTO items AS t\n SELECT * FROM tmpdata\n ON CONFLICT DO NOTHING;\n GET DIAGNOSTICS nrows = ROW_COUNT;\n RAISE NOTICE 'Inserted % rows to items. %', nrows, clock_timestamp() - ts;\n END IF;\n\n RAISE NOTICE 'Deleting data from staging table. %', clock_timestamp() - ts;\n DELETE FROM items_staging;\n RAISE NOTICE 'Done. %', clock_timestamp() - ts;\n\n RETURN NULL;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER items_staging_insert_trigger AFTER INSERT ON items_staging REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\nCREATE TRIGGER items_staging_insert_ignore_trigger AFTER INSERT ON items_staging_ignore REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\nCREATE TRIGGER items_staging_insert_upsert_trigger AFTER INSERT ON items_staging_upsert REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\n\nCREATE OR REPLACE FUNCTION item_by_id(_id text, _collection text DEFAULT NULL) RETURNS items AS\n$$\nDECLARE\n i items%ROWTYPE;\nBEGIN\n SELECT * INTO i FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection) LIMIT 1;\n RETURN i;\nEND;\n$$ LANGUAGE PLPGSQL STABLE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION get_item(_id text, _collection text DEFAULT NULL) RETURNS jsonb AS $$\n SELECT content_hydrate(items) FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection);\n$$ LANGUAGE SQL STABLE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION delete_item(_id text, _collection text DEFAULT NULL) RETURNS VOID AS $$\nDECLARE\nout items%ROWTYPE;\nBEGIN\n DELETE FROM items WHERE id = _id AND (_collection IS NULL OR collection=_collection) RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n--/*\nCREATE OR REPLACE FUNCTION create_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION update_item(content jsonb) RETURNS VOID AS $$\nDECLARE\n old items %ROWTYPE;\n out items%ROWTYPE;\nBEGIN\n PERFORM delete_item(content->>'id', content->>'collection');\n PERFORM create_item(content);\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION create_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION collection_bbox(id text) RETURNS jsonb AS $$\n SELECT (replace(replace(replace(st_extent(geometry)::text,'BOX(','[['),')',']]'),' ',','))::jsonb\n FROM items WHERE collection=$1;\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION collection_temporal_extent(id text) RETURNS jsonb AS $$\n SELECT to_jsonb(array[array[min(datetime), max(datetime)]])\n FROM items WHERE collection=$1;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection_extents() RETURNS VOID AS $$\nUPDATE collections\n SET content = jsonb_set_lax(\n content,\n '{extent}'::text[],\n collection_extent(id, FALSE),\n true,\n 'use_json_null'\n )\n;\n$$ LANGUAGE SQL;\nCREATE TABLE partition_stats (\n partition text PRIMARY KEY,\n dtrange tstzrange,\n edtrange tstzrange,\n spatial geometry,\n last_updated timestamptz,\n keys text[]\n) WITH (FILLFACTOR=90);\n\nCREATE INDEX partitions_range_idx ON partition_stats USING GIST(dtrange);\n\n\nCREATE OR REPLACE FUNCTION constraint_tstzrange(expr text) RETURNS tstzrange AS $$\n WITH t AS (\n SELECT regexp_matches(\n expr,\n E'\\\\(''\\([0-9 :+-]*\\)''\\\\).*\\\\(''\\([0-9 :+-]*\\)''\\\\)'\n ) AS m\n ) SELECT tstzrange(m[1]::timestamptz, m[2]::timestamptz) FROM t\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE STRICT;\n\nCREATE OR REPLACE FUNCTION dt_constraint(coid oid, OUT dt tstzrange, OUT edt tstzrange) RETURNS RECORD AS $$\nDECLARE\n expr text := pg_get_constraintdef(coid);\n matches timestamptz[];\nBEGIN\n IF expr LIKE '%NULL%' THEN\n dt := tstzrange(null::timestamptz, null::timestamptz);\n edt := tstzrange(null::timestamptz, null::timestamptz);\n RETURN;\n END IF;\n WITH f AS (SELECT (regexp_matches(expr, E'([0-9]{4}-[0-1][0-9]-[0-3][0-9] [0-2][0-9]:[0-5][0-9]:[0-5][0-9]\\.?[0-9]*)', 'g'))[1] f)\n SELECT array_agg(f::timestamptz) INTO matches FROM f;\n IF cardinality(matches) = 4 THEN\n dt := tstzrange(matches[1], matches[2],'[]');\n edt := tstzrange(matches[3], matches[4], '[]');\n RETURN;\n ELSIF cardinality(matches) = 2 THEN\n edt := tstzrange(matches[1], matches[2],'[]');\n RETURN;\n END IF;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\nCREATE OR REPLACE VIEW partition_sys_meta AS\nSELECT\n (parse_ident(relid::text))[cardinality(parse_ident(relid::text))] as partition,\n replace(replace(CASE WHEN level = 1 THEN pg_get_expr(c.relpartbound, c.oid)\n ELSE pg_get_expr(parent.relpartbound, parent.oid)\n END, 'FOR VALUES IN (''',''), ''')','') AS collection,\n level,\n c.reltuples,\n c.relhastriggers,\n COALESCE(constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as partition_dtrange,\n COALESCE((dt_constraint(edt.oid)).dt, constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as constraint_dtrange,\n COALESCE((dt_constraint(edt.oid)).edt, tstzrange('-infinity', 'infinity','[]')) as constraint_edtrange\nFROM\n pg_partition_tree('items')\n JOIN pg_class c ON (relid::regclass = c.oid)\n JOIN pg_class parent ON (parentrelid::regclass = parent.oid AND isleaf)\n LEFT JOIN pg_constraint edt ON (conrelid=c.oid AND contype='c')\nWHERE isleaf\n;\n\nCREATE OR REPLACE VIEW partitions_view AS\nSELECT\n (parse_ident(relid::text))[cardinality(parse_ident(relid::text))] as partition,\n replace(replace(CASE WHEN level = 1 THEN pg_get_expr(c.relpartbound, c.oid)\n ELSE pg_get_expr(parent.relpartbound, parent.oid)\n END, 'FOR VALUES IN (''',''), ''')','') AS collection,\n level,\n c.reltuples,\n c.relhastriggers,\n COALESCE(pgstac.constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as partition_dtrange,\n COALESCE((pgstac.dt_constraint(edt.oid)).dt, pgstac.constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as constraint_dtrange,\n COALESCE((pgstac.dt_constraint(edt.oid)).edt, tstzrange('-infinity', 'infinity','[]')) as constraint_edtrange,\n dtrange,\n edtrange,\n spatial,\n last_updated\nFROM\n pg_partition_tree('pgstac.items')\n JOIN pg_class c ON (relid::regclass = c.oid)\n JOIN pg_class parent ON (parentrelid::regclass = parent.oid AND isleaf)\n LEFT JOIN pg_constraint edt ON (conrelid=c.oid AND contype='c')\n LEFT JOIN pgstac.partition_stats ON ((parse_ident(relid::text))[cardinality(parse_ident(relid::text))]=partition)\nWHERE isleaf\n;\n\nCREATE MATERIALIZED VIEW partitions AS\nSELECT * FROM partitions_view;\nCREATE UNIQUE INDEX ON partitions (partition);\n\nCREATE MATERIALIZED VIEW partition_steps AS\nSELECT\n partition as name,\n date_trunc('month',lower(partition_dtrange)) as sdate,\n date_trunc('month', upper(partition_dtrange)) + '1 month'::interval as edate\n FROM partitions_view WHERE partition_dtrange IS NOT NULL AND partition_dtrange != 'empty'::tstzrange\n ORDER BY dtrange ASC\n;\n\n\nCREATE OR REPLACE FUNCTION update_partition_stats_q(_partition text, istrigger boolean default false) RETURNS VOID AS $$\nDECLARE\nBEGIN\n PERFORM run_or_queue(\n format('SELECT update_partition_stats(%L, %L);', _partition, istrigger)\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION update_partition_stats(_partition text, istrigger boolean default false) RETURNS VOID AS $$\nDECLARE\n dtrange tstzrange;\n edtrange tstzrange;\n cdtrange tstzrange;\n cedtrange tstzrange;\n extent geometry;\n collection text;\nBEGIN\n RAISE NOTICE 'Updating stats for %.', _partition;\n EXECUTE format(\n $q$\n SELECT\n tstzrange(min(datetime), max(datetime),'[]'),\n tstzrange(min(end_datetime), max(end_datetime), '[]')\n FROM %I\n $q$,\n _partition\n ) INTO dtrange, edtrange;\n extent := st_estimatedextent('pgstac', _partition, 'geometry');\n RAISE DEBUG 'Estimated Extent: %', extent;\n INSERT INTO partition_stats (partition, dtrange, edtrange, spatial, last_updated)\n SELECT _partition, dtrange, edtrange, extent, now()\n ON CONFLICT (partition) DO\n UPDATE SET\n dtrange=EXCLUDED.dtrange,\n edtrange=EXCLUDED.edtrange,\n spatial=EXCLUDED.spatial,\n last_updated=EXCLUDED.last_updated\n ;\n\n SELECT\n constraint_dtrange, constraint_edtrange, pv.collection\n INTO cdtrange, cedtrange, collection\n FROM partitions_view pv WHERE partition = _partition;\n REFRESH MATERIALIZED VIEW partitions;\n REFRESH MATERIALIZED VIEW partition_steps;\n\n\n RAISE NOTICE 'Checking if we need to modify constraints...';\n RAISE NOTICE 'cdtrange: % dtrange: % cedtrange: % edtrange: %',cdtrange, dtrange, cedtrange, edtrange;\n IF\n (cdtrange IS DISTINCT FROM dtrange OR edtrange IS DISTINCT FROM cedtrange)\n AND NOT istrigger\n THEN\n RAISE NOTICE 'Modifying Constraints';\n RAISE NOTICE 'Existing % %', cdtrange, cedtrange;\n RAISE NOTICE 'New % %', dtrange, edtrange;\n PERFORM drop_table_constraints(_partition);\n PERFORM create_table_constraints(_partition, dtrange, edtrange);\n REFRESH MATERIALIZED VIEW partitions;\n REFRESH MATERIALIZED VIEW partition_steps;\n END IF;\n RAISE NOTICE 'Checking if we need to update collection extents.';\n IF get_setting_bool('update_collection_extent') THEN\n RAISE NOTICE 'updating collection extent for %', collection;\n PERFORM run_or_queue(format($q$\n UPDATE collections\n SET content = jsonb_set_lax(\n content,\n '{extent}'::text[],\n collection_extent(%L, FALSE),\n true,\n 'use_json_null'\n ) WHERE id=%L\n ;\n $q$, collection, collection));\n ELSE\n RAISE NOTICE 'Not updating collection extent for %', collection;\n END IF;\n\nEND;\n$$ LANGUAGE PLPGSQL STRICT SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION partition_name( IN collection text, IN dt timestamptz, OUT partition_name text, OUT partition_range tstzrange) AS $$\nDECLARE\n c RECORD;\n parent_name text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n parent_name := format('_items_%s', c.key);\n\n\n IF c.partition_trunc = 'year' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYY'));\n ELSIF c.partition_trunc = 'month' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYYMM'));\n ELSE\n partition_name := parent_name;\n partition_range := tstzrange('-infinity'::timestamptz, 'infinity'::timestamptz, '[]');\n END IF;\n IF partition_range IS NULL THEN\n partition_range := tstzrange(\n date_trunc(c.partition_trunc::text, dt),\n date_trunc(c.partition_trunc::text, dt) + concat('1 ', c.partition_trunc)::interval\n );\n END IF;\n RETURN;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION drop_table_constraints(t text) RETURNS text AS $$\nDECLARE\n q text;\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM partitions_view WHERE partition=t) THEN\n RETURN NULL;\n END IF;\n FOR q IN SELECT FORMAT(\n $q$\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n $q$,\n t,\n conname\n ) FROM pg_constraint\n WHERE conrelid=t::regclass::oid AND contype='c'\n LOOP\n EXECUTE q;\n END LOOP;\n RETURN t;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE OR REPLACE FUNCTION create_table_constraints(t text, _dtrange tstzrange, _edtrange tstzrange) RETURNS text AS $$\nDECLARE\n q text;\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM partitions_view WHERE partition=t) THEN\n RETURN NULL;\n END IF;\n RAISE NOTICE 'Creating Table Constraints for % % %', t, _dtrange, _edtrange;\n IF _dtrange = 'empty' AND _edtrange = 'empty' THEN\n q :=format(\n $q$\n DO $block$\n BEGIN\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n ALTER TABLE %I\n ADD CONSTRAINT %I\n CHECK (((datetime IS NULL) AND (end_datetime IS NULL))) NOT VALID\n ;\n ALTER TABLE %I\n VALIDATE CONSTRAINT %I\n ;\n\n\n\n EXCEPTION WHEN others THEN\n RAISE WARNING '%%, Issue Altering Constraints. Please run update_partition_stats(%I)', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n $block$;\n $q$,\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t),\n t\n );\n ELSE\n q :=format(\n $q$\n DO $block$\n BEGIN\n\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n ALTER TABLE %I\n ADD CONSTRAINT %I\n CHECK (\n (datetime >= %L)\n AND (datetime <= %L)\n AND (end_datetime >= %L)\n AND (end_datetime <= %L)\n ) NOT VALID\n ;\n ALTER TABLE %I\n VALIDATE CONSTRAINT %I\n ;\n\n\n\n EXCEPTION WHEN others THEN\n RAISE WARNING '%%, Issue Altering Constraints. Please run update_partition_stats(%I)', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n $block$;\n $q$,\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t),\n lower(_dtrange),\n upper(_dtrange),\n lower(_edtrange),\n upper(_edtrange),\n t,\n format('%s_dt', t),\n t\n );\n END IF;\n PERFORM run_or_queue(q);\n RETURN t;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION check_partition(\n _collection text,\n _dtrange tstzrange,\n _edtrange tstzrange\n) RETURNS text AS $$\nDECLARE\n c RECORD;\n pm RECORD;\n _partition_name text;\n _partition_dtrange tstzrange;\n _constraint_dtrange tstzrange;\n _constraint_edtrange tstzrange;\n q text;\n deferrable_q text;\n err_context text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=_collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', _collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n\n IF c.partition_trunc IS NOT NULL THEN\n _partition_dtrange := tstzrange(\n date_trunc(c.partition_trunc, lower(_dtrange)),\n date_trunc(c.partition_trunc, lower(_dtrange)) + (concat('1 ', c.partition_trunc))::interval,\n '[)'\n );\n ELSE\n _partition_dtrange := '[-infinity, infinity]'::tstzrange;\n END IF;\n\n IF NOT _partition_dtrange @> _dtrange THEN\n RAISE EXCEPTION 'dtrange % is greater than the partition size % for collection %', _dtrange, c.partition_trunc, _collection;\n END IF;\n\n\n IF c.partition_trunc = 'year' THEN\n _partition_name := format('_items_%s_%s', c.key, to_char(lower(_partition_dtrange),'YYYY'));\n ELSIF c.partition_trunc = 'month' THEN\n _partition_name := format('_items_%s_%s', c.key, to_char(lower(_partition_dtrange),'YYYYMM'));\n ELSE\n _partition_name := format('_items_%s', c.key);\n END IF;\n\n SELECT * INTO pm FROM partition_sys_meta WHERE collection=_collection AND partition_dtrange @> _dtrange;\n IF FOUND THEN\n RAISE NOTICE '% % %', _edtrange, _dtrange, pm;\n _constraint_edtrange :=\n tstzrange(\n least(\n lower(_edtrange),\n nullif(lower(pm.constraint_edtrange), '-infinity')\n ),\n greatest(\n upper(_edtrange),\n nullif(upper(pm.constraint_edtrange), 'infinity')\n ),\n '[]'\n );\n _constraint_dtrange :=\n tstzrange(\n least(\n lower(_dtrange),\n nullif(lower(pm.constraint_dtrange), '-infinity')\n ),\n greatest(\n upper(_dtrange),\n nullif(upper(pm.constraint_dtrange), 'infinity')\n ),\n '[]'\n );\n\n IF pm.constraint_edtrange @> _edtrange AND pm.constraint_dtrange @> _dtrange THEN\n RETURN pm.partition;\n ELSE\n PERFORM drop_table_constraints(_partition_name);\n END IF;\n ELSE\n _constraint_edtrange := _edtrange;\n _constraint_dtrange := _dtrange;\n END IF;\n RAISE NOTICE 'EXISTING CONSTRAINTS % %, NEW % %', pm.constraint_dtrange, pm.constraint_edtrange, _constraint_dtrange, _constraint_edtrange;\n RAISE NOTICE 'Creating partition % %', _partition_name, _partition_dtrange;\n IF c.partition_trunc IS NULL THEN\n q := format(\n $q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n GRANT ALL ON %I to pgstac_ingest;\n $q$,\n _partition_name,\n _collection,\n concat(_partition_name,'_pk'),\n _partition_name,\n _partition_name\n );\n ELSE\n q := format(\n $q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L) PARTITION BY RANGE (datetime);\n CREATE TABLE IF NOT EXISTS %I partition OF %I FOR VALUES FROM (%L) TO (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n GRANT ALL ON %I TO pgstac_ingest;\n $q$,\n format('_items_%s', c.key),\n _collection,\n _partition_name,\n format('_items_%s', c.key),\n lower(_partition_dtrange),\n upper(_partition_dtrange),\n format('%s_pk', _partition_name),\n _partition_name,\n _partition_name\n );\n END IF;\n\n BEGIN\n EXECUTE q;\n EXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', _partition_name;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\n END;\n PERFORM maintain_partitions(_partition_name);\n PERFORM update_partition_stats_q(_partition_name, true);\n REFRESH MATERIALIZED VIEW partitions;\n REFRESH MATERIALIZED VIEW partition_steps;\n RETURN _partition_name;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION repartition(_collection text, _partition_trunc text, triggered boolean DEFAULT FALSE) RETURNS text AS $$\nDECLARE\n c RECORD;\n q text;\n from_trunc text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=_collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', _collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n IF triggered THEN\n RAISE NOTICE 'Converting % to % partitioning via Trigger', _collection, _partition_trunc;\n ELSE\n RAISE NOTICE 'Converting % from using % to % partitioning', _collection, c.partition_trunc, _partition_trunc;\n IF c.partition_trunc IS NOT DISTINCT FROM _partition_trunc THEN\n RAISE NOTICE 'Collection % already set to use partition by %', _collection, _partition_trunc;\n RETURN _collection;\n END IF;\n END IF;\n\n IF EXISTS (SELECT 1 FROM partitions_view WHERE collection=_collection LIMIT 1) THEN\n EXECUTE format(\n $q$\n CREATE TEMP TABLE changepartitionstaging ON COMMIT DROP AS SELECT * FROM %I;\n DROP TABLE IF EXISTS %I CASCADE;\n WITH p AS (\n SELECT\n collection,\n CASE\n WHEN %L IS NULL THEN '-infinity'::timestamptz\n ELSE date_trunc(%L, datetime)\n END as d,\n tstzrange(min(datetime),max(datetime),'[]') as dtrange,\n tstzrange(min(end_datetime),max(end_datetime),'[]') as edtrange\n FROM changepartitionstaging\n GROUP BY 1,2\n ) SELECT check_partition(collection, dtrange, edtrange) FROM p;\n INSERT INTO items SELECT * FROM changepartitionstaging;\n DROP TABLE changepartitionstaging;\n $q$,\n concat('_items_', c.key),\n concat('_items_', c.key),\n c.partition_trunc,\n c.partition_trunc\n );\n END IF;\n RETURN _collection;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE OR REPLACE FUNCTION collections_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n q text;\n partition_name text := format('_items_%s', NEW.key);\n partition_exists boolean := false;\n partition_empty boolean := true;\n err_context text;\n loadtemp boolean := FALSE;\nBEGIN\n RAISE NOTICE 'Collection Trigger. % %', NEW.id, NEW.key;\n IF TG_OP = 'UPDATE' AND NEW.partition_trunc IS DISTINCT FROM OLD.partition_trunc THEN\n PERFORM repartition(NEW.id, NEW.partition_trunc, TRUE);\n END IF;\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER collections_trigger AFTER\nINSERT\nOR\nUPDATE ON collections\nFOR EACH ROW EXECUTE FUNCTION collections_trigger_func();\n\nCREATE OR REPLACE FUNCTION chunker(\n IN _where text,\n OUT s timestamptz,\n OUT e timestamptz\n) RETURNS SETOF RECORD AS $$\nDECLARE\n explain jsonb;\nBEGIN\n IF _where IS NULL THEN\n _where := ' TRUE ';\n END IF;\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s;', _where)\n INTO explain;\n RAISE DEBUG 'EXPLAIN: %', explain;\n\n RETURN QUERY\n WITH t AS (\n SELECT j->>0 as p FROM\n jsonb_path_query(\n explain,\n 'strict $.**.\"Relation Name\" ? (@ != null)'\n ) j\n ),\n parts AS (\n SELECT sdate, edate FROM t JOIN partition_steps ON (t.p = name)\n ),\n times AS (\n SELECT sdate FROM parts\n UNION\n SELECT edate FROM parts\n ),\n uniq AS (\n SELECT DISTINCT sdate FROM times ORDER BY sdate\n ),\n last AS (\n SELECT sdate, lead(sdate, 1) over () as edate FROM uniq\n )\n SELECT sdate, edate FROM last WHERE edate IS NOT NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION partition_queries(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN partitions text[] DEFAULT NULL\n) RETURNS SETOF text AS $$\nDECLARE\n query text;\n sdate timestamptz;\n edate timestamptz;\nBEGIN\nIF _where IS NULL OR trim(_where) = '' THEN\n _where = ' TRUE ';\nEND IF;\nRAISE NOTICE 'Getting chunks for % %', _where, _orderby;\nIF _orderby ILIKE 'datetime d%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 DESC LOOP\n RETURN NEXT format($q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n $q$,\n sdate,\n edate,\n _where,\n _orderby\n );\n END LOOP;\nELSIF _orderby ILIKE 'datetime a%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 ASC LOOP\n RETURN NEXT format($q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n $q$,\n sdate,\n edate,\n _where,\n _orderby\n );\n END LOOP;\nELSE\n query := format($q$\n SELECT * FROM items\n WHERE %s\n ORDER BY %s\n $q$, _where, _orderby\n );\n\n RETURN NEXT query;\n RETURN;\nEND IF;\n\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION partition_query_view(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN _limit int DEFAULT 10\n) RETURNS text AS $$\n WITH p AS (\n SELECT * FROM partition_queries(_where, _orderby) p\n )\n SELECT\n CASE WHEN EXISTS (SELECT 1 FROM p) THEN\n (SELECT format($q$\n SELECT * FROM (\n %s\n ) total LIMIT %s\n $q$,\n string_agg(\n format($q$ SELECT * FROM ( %s ) AS sub $q$, p),\n '\n UNION ALL\n '\n ),\n _limit\n ))\n ELSE NULL\n END FROM p;\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION q_to_tsquery (input text)\n RETURNS tsquery\n AS $$\nDECLARE\n processed_text text;\n temp_text text;\n quote_array text[];\n placeholder text := '@QUOTE@';\nBEGIN\n -- Extract all quoted phrases and store in array\n quote_array := regexp_matches(input, '\"[^\"]*\"', 'g');\n\n -- Replace each quoted part with a unique placeholder if there are any quoted phrases\n IF array_length(quote_array, 1) IS NOT NULL THEN\n processed_text := input;\n FOR i IN array_lower(quote_array, 1) .. array_upper(quote_array, 1) LOOP\n processed_text := replace(processed_text, quote_array[i], placeholder || i || placeholder);\n END LOOP;\n ELSE\n processed_text := input;\n END IF;\n\n -- Replace non-quoted text using regular expressions\n\n -- , -> |\n processed_text := regexp_replace(processed_text, ',(?=(?:[^\"]*\"[^\"]*\")*[^\"]*$)', ' | ', 'g');\n\n -- and -> &\n processed_text := regexp_replace(processed_text, '\\s+AND\\s+', ' & ', 'gi');\n\n -- or -> |\n processed_text := regexp_replace(processed_text, '\\s+OR\\s+', ' | ', 'gi');\n\n -- + ->\n processed_text := regexp_replace(processed_text, '^\\s*\\+([a-zA-Z0-9_]+)', '\\1', 'g'); -- +term at start\n processed_text := regexp_replace(processed_text, '\\s*\\+([a-zA-Z0-9_]+)', ' & \\1', 'g'); -- +term elsewhere\n\n -- - -> !\n processed_text := regexp_replace(processed_text, '^\\s*\\-([a-zA-Z0-9_]+)', '! \\1', 'g'); -- -term at start\n processed_text := regexp_replace(processed_text, '\\s*\\-([a-zA-Z0-9_]+)', ' & ! \\1', 'g'); -- -term elsewhere\n -- Replace placeholders back with quoted phrases if there were any\n IF array_length(quote_array, 1) IS NOT NULL THEN\n FOR i IN array_lower(quote_array, 1) .. array_upper(quote_array, 1) LOOP\n processed_text := replace(processed_text, placeholder || i || placeholder, '''' || substring(quote_array[i] from 2 for length(quote_array[i]) - 2) || '''');\n END LOOP;\n END IF;\n\n -- Print processed_text to the console for debugging purposes\n RAISE NOTICE 'processed_text: %', processed_text;\n\n RETURN to_tsquery('english', processed_text);\nEND;\n$$\nLANGUAGE plpgsql;\n\n\nCREATE OR REPLACE FUNCTION stac_search_to_where(j jsonb) RETURNS text AS $$\nDECLARE\n where_segments text[];\n _where text;\n dtrange tstzrange;\n collections text[];\n geom geometry;\n sdate timestamptz;\n edate timestamptz;\n filterlang text;\n filter jsonb := j->'filter';\n ft_query tsquery;\nBEGIN\n IF j ? 'ids' THEN\n where_segments := where_segments || format('id = ANY (%L) ', to_text_array(j->'ids'));\n END IF;\n\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n where_segments := where_segments || format('collection = ANY (%L) ', collections);\n END IF;\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n\n where_segments := where_segments || format(' datetime <= %L::timestamptz AND end_datetime >= %L::timestamptz ',\n edate,\n sdate\n );\n END IF;\n\n IF j ? 'q' THEN\n ft_query := q_to_tsquery(j->>'q');\n where_segments := where_segments || format(\n $quote$\n (\n to_tsvector('english', content->'properties'->>'description') ||\n to_tsvector('english', coalesce(content->'properties'->>'title', '')) ||\n to_tsvector('english', coalesce(content->'properties'->>'keywords', ''))\n ) @@ %L\n $quote$,\n ft_query\n );\n END IF;\n\n geom := stac_geom(j);\n IF geom IS NOT NULL THEN\n where_segments := where_segments || format('st_intersects(geometry, %L)',geom);\n END IF;\n\n filterlang := COALESCE(\n j->>'filter-lang',\n get_setting('default_filter_lang', j->'conf')\n );\n IF NOT filter @? '$.**.op' THEN\n filterlang := 'cql-json';\n END IF;\n\n IF filterlang NOT IN ('cql-json','cql2-json') AND j ? 'filter' THEN\n RAISE EXCEPTION '% is not a supported filter-lang. Please use cql-json or cql2-json.', filterlang;\n END IF;\n\n IF j ? 'query' AND j ? 'filter' THEN\n RAISE EXCEPTION 'Can only use either query or filter at one time.';\n END IF;\n\n IF j ? 'query' THEN\n filter := query_to_cql2(j->'query');\n ELSIF filterlang = 'cql-json' THEN\n filter := cql1_to_cql2(filter);\n END IF;\n RAISE NOTICE 'FILTER: %', filter;\n where_segments := where_segments || cql2_query(filter);\n IF cardinality(where_segments) < 1 THEN\n RETURN ' TRUE ';\n END IF;\n\n _where := array_to_string(array_remove(where_segments, NULL), ' AND ');\n\n IF _where IS NULL OR BTRIM(_where) = '' THEN\n RETURN ' TRUE ';\n END IF;\n RETURN _where;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION parse_sort_dir(_dir text, reverse boolean default false) RETURNS text AS $$\n WITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n ) SELECT\n CASE\n WHEN NOT reverse THEN d\n WHEN d = 'ASC' THEN 'DESC'\n WHEN d = 'DESC' THEN 'ASC'\n END\n FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION sort_dir_to_op(_dir text, prev boolean default false) RETURNS text AS $$\n WITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n ) SELECT\n CASE\n WHEN d = 'ASC' AND prev THEN '<='\n WHEN d = 'DESC' AND prev THEN '>='\n WHEN d = 'ASC' THEN '>='\n WHEN d = 'DESC' THEN '<='\n END\n FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION sort_sqlorderby(\n _search jsonb DEFAULT NULL,\n reverse boolean DEFAULT FALSE\n) RETURNS text AS $$\n WITH sortby AS (\n SELECT coalesce(_search->'sortby','[{\"field\":\"datetime\", \"direction\":\"desc\"}]') as sort\n ), withid AS (\n SELECT CASE\n WHEN sort @? '$[*] ? (@.field == \"id\")' THEN sort\n ELSE sort || '[{\"field\":\"id\", \"direction\":\"desc\"}]'::jsonb\n END as sort\n FROM sortby\n ), withid_rows AS (\n SELECT jsonb_array_elements(sort) as value FROM withid\n ),sorts AS (\n SELECT\n coalesce(\n (queryable(value->>'field')).expression\n ) as key,\n parse_sort_dir(value->>'direction', reverse) as dir\n FROM withid_rows\n )\n SELECT array_to_string(\n array_agg(concat(key, ' ', dir)),\n ', '\n ) FROM sorts;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_sort_dir(sort_item jsonb) RETURNS text AS $$\n SELECT CASE WHEN sort_item->>'direction' ILIKE 'desc%' THEN 'DESC' ELSE 'ASC' END;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION get_token_val_str(\n _field text,\n _item items\n) RETURNS text AS $$\nDECLARE\n q text;\n literal text;\nBEGIN\n q := format($q$ SELECT quote_literal(%s) FROM (SELECT $1.*) as r;$q$, _field);\n EXECUTE q INTO literal USING _item;\n RETURN literal;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n\nCREATE OR REPLACE FUNCTION get_token_record(IN _token text, OUT prev BOOLEAN, OUT item items) RETURNS RECORD AS $$\nDECLARE\n _itemid text := _token;\n _collectionid text;\nBEGIN\n IF _token IS NULL THEN\n RETURN;\n END IF;\n RAISE NOTICE 'Looking for token: %', _token;\n prev := FALSE;\n IF _token ILIKE 'prev:%' THEN\n _itemid := replace(_token, 'prev:','');\n prev := TRUE;\n ELSIF _token ILIKE 'next:%' THEN\n _itemid := replace(_token, 'next:', '');\n END IF;\n SELECT id INTO _collectionid FROM collections WHERE _itemid LIKE concat(id,':%');\n IF FOUND THEN\n _itemid := replace(_itemid, concat(_collectionid,':'), '');\n SELECT * INTO item FROM items WHERE id=_itemid AND collection=_collectionid;\n ELSE\n SELECT * INTO item FROM items WHERE id=_itemid;\n END IF;\n IF item IS NULL THEN\n RAISE EXCEPTION 'Could not find item using token: % item: % collection: %', _token, _itemid, _collectionid;\n END IF;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\n\nCREATE OR REPLACE FUNCTION get_token_filter(\n _sortby jsonb DEFAULT '[{\"field\":\"datetime\",\"direction\":\"desc\"}]'::jsonb,\n token_item items DEFAULT NULL,\n prev boolean DEFAULT FALSE,\n inclusive boolean DEFAULT FALSE\n) RETURNS text AS $$\nDECLARE\n ltop text := '<';\n gtop text := '>';\n dir text;\n sort record;\n orfilter text := '';\n orfilters text[] := '{}'::text[];\n andfilters text[] := '{}'::text[];\n output text;\n token_where text;\nBEGIN\n IF _sortby IS NULL OR _sortby = '[]'::jsonb THEN\n _sortby := '[{\"field\":\"datetime\",\"direction\":\"desc\"}]'::jsonb;\n END IF;\n _sortby := _sortby || jsonb_build_object('field','id','direction',_sortby->0->>'direction');\n RAISE NOTICE 'Getting Token Filter. % %', _sortby, token_item;\n IF inclusive THEN\n orfilters := orfilters || format('( id=%L AND collection=%L )' , token_item.id, token_item.collection);\n END IF;\n\n FOR sort IN\n WITH s1 AS (\n SELECT\n _row,\n (queryable(value->>'field')).expression as _field,\n (value->>'field' = 'id') as _isid,\n get_sort_dir(value) as _dir\n FROM jsonb_array_elements(_sortby)\n WITH ORDINALITY AS t(value, _row)\n )\n SELECT\n _row,\n _field,\n _dir,\n get_token_val_str(_field, token_item) as _val\n FROM s1\n WHERE _row <= (SELECT min(_row) FROM s1 WHERE _isid)\n LOOP\n orfilter := NULL;\n RAISE NOTICE 'SORT: %', sort;\n IF sort._val IS NOT NULL AND ((prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC')) THEN\n orfilter := format($f$(\n (%s %s %s) OR (%s IS NULL)\n )$f$,\n sort._field,\n ltop,\n sort._val,\n sort._val\n );\n ELSIF sort._val IS NULL AND ((prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC')) THEN\n RAISE NOTICE '< but null';\n orfilter := format('%s IS NOT NULL', sort._field);\n ELSIF sort._val IS NULL THEN\n RAISE NOTICE '> but null';\n ELSE\n orfilter := format($f$(\n (%s %s %s) OR (%s IS NULL)\n )$f$,\n sort._field,\n gtop,\n sort._val,\n sort._field\n );\n END IF;\n RAISE NOTICE 'ORFILTER: %', orfilter;\n\n IF orfilter IS NOT NULL THEN\n IF sort._row = 1 THEN\n orfilters := orfilters || orfilter;\n ELSE\n orfilters := orfilters || format('(%s AND %s)', array_to_string(andfilters, ' AND '), orfilter);\n END IF;\n END IF;\n IF sort._val IS NOT NULL THEN\n andfilters := andfilters || format('%s = %s', sort._field, sort._val);\n ELSE\n andfilters := andfilters || format('%s IS NULL', sort._field);\n END IF;\n END LOOP;\n\n output := array_to_string(orfilters, ' OR ');\n\n token_where := concat('(',coalesce(output,'true'),')');\n IF trim(token_where) = '' THEN\n token_where := NULL;\n END IF;\n RAISE NOTICE 'TOKEN_WHERE: %',token_where;\n RETURN token_where;\n END;\n$$ LANGUAGE PLPGSQL SET transform_null_equals TO TRUE\n;\n\nCREATE OR REPLACE FUNCTION search_tohash(jsonb) RETURNS jsonb AS $$\n SELECT $1 - '{token,limit,context,includes,excludes}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION search_hash(jsonb, jsonb) RETURNS text AS $$\n SELECT md5(concat(search_tohash($1)::text,$2::text));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TABLE IF NOT EXISTS searches(\n hash text GENERATED ALWAYS AS (search_hash(search, metadata)) STORED PRIMARY KEY,\n search jsonb NOT NULL,\n _where text,\n orderby text,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n metadata jsonb DEFAULT '{}'::jsonb NOT NULL\n);\n\nCREATE TABLE IF NOT EXISTS search_wheres(\n id bigint generated always as identity primary key,\n _where text NOT NULL,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n statslastupdated timestamptz,\n estimated_count bigint,\n estimated_cost float,\n time_to_estimate float,\n total_count bigint,\n time_to_count float,\n partitions text[]\n);\n\nCREATE INDEX IF NOT EXISTS search_wheres_partitions ON search_wheres USING GIN (partitions);\nCREATE UNIQUE INDEX IF NOT EXISTS search_wheres_where ON search_wheres ((md5(_where)));\n\nCREATE OR REPLACE FUNCTION where_stats(\n inwhere text,\n updatestats boolean default false,\n conf jsonb default null\n) RETURNS search_wheres AS $$\nDECLARE\n t timestamptz;\n i interval;\n explain_json jsonb;\n partitions text[];\n sw search_wheres%ROWTYPE;\n inwhere_hash text := md5(inwhere);\n _context text := lower(context(conf));\n _stats_ttl interval := context_stats_ttl(conf);\n _estimated_cost_threshold float := context_estimated_cost(conf);\n _estimated_count_threshold int := context_estimated_count(conf);\n ro bool := pgstac.readonly(conf);\nBEGIN\n -- If updatestats is true then set ttl to 0\n IF updatestats THEN\n RAISE DEBUG 'Updatestats set to TRUE, setting TTL to 0';\n _stats_ttl := '0'::interval;\n END IF;\n\n -- If we don't need to calculate context, just return\n IF _context = 'off' THEN\n sw._where = inwhere;\n RETURN sw;\n END IF;\n\n -- Get any stats that we have.\n IF NOT ro THEN\n -- If there is a lock where another process is\n -- updating the stats, wait so that we don't end up calculating a bunch of times.\n SELECT * INTO sw FROM search_wheres WHERE md5(_where)=inwhere_hash FOR UPDATE;\n ELSE\n SELECT * INTO sw FROM search_wheres WHERE md5(_where)=inwhere_hash;\n END IF;\n\n -- If there is a cached row, figure out if we need to update\n IF\n sw IS NOT NULL\n AND sw.statslastupdated IS NOT NULL\n AND sw.total_count IS NOT NULL\n AND now() - sw.statslastupdated <= _stats_ttl\n THEN\n -- we have a cached row with data that is within our ttl\n RAISE DEBUG 'Stats present in table and lastupdated within ttl: %', sw;\n IF NOT ro THEN\n RAISE DEBUG 'Updating search_wheres only bumping lastused and usecount';\n UPDATE search_wheres SET\n lastused = now(),\n usecount = search_wheres.usecount + 1\n WHERE md5(_where) = inwhere_hash\n RETURNING * INTO sw;\n END IF;\n RAISE DEBUG 'Returning cached counts. %', sw;\n RETURN sw;\n END IF;\n\n -- Calculate estimated cost and rows\n -- Use explain to get estimated count/cost\n IF sw.estimated_count IS NULL OR sw.estimated_cost IS NULL THEN\n RAISE DEBUG 'Calculating estimated stats';\n t := clock_timestamp();\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s', inwhere)\n INTO explain_json;\n RAISE DEBUG 'Time for just the explain: %', clock_timestamp() - t;\n i := clock_timestamp() - t;\n\n sw.estimated_count := explain_json->0->'Plan'->'Plan Rows';\n sw.estimated_cost := explain_json->0->'Plan'->'Total Cost';\n sw.time_to_estimate := extract(epoch from i);\n END IF;\n\n RAISE DEBUG 'ESTIMATED_COUNT: %, THRESHOLD %', sw.estimated_count, _estimated_count_threshold;\n RAISE DEBUG 'ESTIMATED_COST: %, THRESHOLD %', sw.estimated_cost, _estimated_cost_threshold;\n\n -- If context is set to auto and the costs are within the threshold return the estimated costs\n IF\n _context = 'auto'\n AND sw.estimated_count >= _estimated_count_threshold\n AND sw.estimated_cost >= _estimated_cost_threshold\n THEN\n IF NOT ro THEN\n INSERT INTO search_wheres (\n _where,\n lastused,\n usecount,\n statslastupdated,\n estimated_count,\n estimated_cost,\n time_to_estimate,\n total_count,\n time_to_count\n ) VALUES (\n inwhere,\n now(),\n 1,\n now(),\n sw.estimated_count,\n sw.estimated_cost,\n sw.time_to_estimate,\n null,\n null\n ) ON CONFLICT ((md5(_where)))\n DO UPDATE SET\n lastused = EXCLUDED.lastused,\n usecount = search_wheres.usecount + 1,\n statslastupdated = EXCLUDED.statslastupdated,\n estimated_count = EXCLUDED.estimated_count,\n estimated_cost = EXCLUDED.estimated_cost,\n time_to_estimate = EXCLUDED.time_to_estimate,\n total_count = EXCLUDED.total_count,\n time_to_count = EXCLUDED.time_to_count\n RETURNING * INTO sw;\n END IF;\n RAISE DEBUG 'Estimates are within thresholds, returning estimates. %', sw;\n RETURN sw;\n END IF;\n\n -- Calculate Actual Count\n t := clock_timestamp();\n RAISE NOTICE 'Calculating actual count...';\n EXECUTE format(\n 'SELECT count(*) FROM items WHERE %s',\n inwhere\n ) INTO sw.total_count;\n i := clock_timestamp() - t;\n RAISE NOTICE 'Actual Count: % -- %', sw.total_count, i;\n sw.time_to_count := extract(epoch FROM i);\n\n IF NOT ro THEN\n INSERT INTO search_wheres (\n _where,\n lastused,\n usecount,\n statslastupdated,\n estimated_count,\n estimated_cost,\n time_to_estimate,\n total_count,\n time_to_count\n ) VALUES (\n inwhere,\n now(),\n 1,\n now(),\n sw.estimated_count,\n sw.estimated_cost,\n sw.time_to_estimate,\n sw.total_count,\n sw.time_to_count\n ) ON CONFLICT ((md5(_where)))\n DO UPDATE SET\n lastused = EXCLUDED.lastused,\n usecount = search_wheres.usecount + 1,\n statslastupdated = EXCLUDED.statslastupdated,\n estimated_count = EXCLUDED.estimated_count,\n estimated_cost = EXCLUDED.estimated_cost,\n time_to_estimate = EXCLUDED.time_to_estimate,\n total_count = EXCLUDED.total_count,\n time_to_count = EXCLUDED.time_to_count\n RETURNING * INTO sw;\n END IF;\n RAISE DEBUG 'Returning with actual count. %', sw;\n RETURN sw;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION search_query(\n _search jsonb = '{}'::jsonb,\n updatestats boolean = false,\n _metadata jsonb = '{}'::jsonb\n) RETURNS searches AS $$\nDECLARE\n search searches%ROWTYPE;\n cached_search searches%ROWTYPE;\n pexplain jsonb;\n t timestamptz;\n i interval;\n doupdate boolean := FALSE;\n insertfound boolean := FALSE;\n ro boolean := pgstac.readonly();\n found_search text;\nBEGIN\n RAISE NOTICE 'SEARCH: %', _search;\n -- Calculate hash, where clause, and order by statement\n search.search := _search;\n search.metadata := _metadata;\n search.hash := search_hash(_search, _metadata);\n search._where := stac_search_to_where(_search);\n search.orderby := sort_sqlorderby(_search);\n search.lastused := now();\n search.usecount := 1;\n\n -- If we are in read only mode, directly return search\n IF ro THEN\n RETURN search;\n END IF;\n\n RAISE NOTICE 'Updating Statistics for search: %s', search;\n -- Update statistics for times used and and when last used\n -- If the entry is locked, rather than waiting, skip updating the stats\n INSERT INTO searches (search, lastused, usecount, metadata)\n VALUES (search.search, now(), 1, search.metadata)\n ON CONFLICT DO NOTHING\n RETURNING * INTO cached_search\n ;\n\n IF NOT FOUND OR cached_search IS NULL THEN\n UPDATE searches SET\n lastused = now(),\n usecount = searches.usecount + 1\n WHERE hash = (\n SELECT hash FROM searches WHERE hash=search.hash FOR UPDATE SKIP LOCKED\n )\n RETURNING * INTO cached_search\n ;\n END IF;\n\n IF cached_search IS NOT NULL THEN\n cached_search._where = search._where;\n cached_search.orderby = search.orderby;\n RETURN cached_search;\n END IF;\n RETURN search;\n\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE OR REPLACE FUNCTION search_fromhash(\n _hash text\n) RETURNS searches AS $$\n SELECT * FROM search_query((SELECT search FROM searches WHERE hash=_hash LIMIT 1));\n$$ LANGUAGE SQL STRICT;\n\nCREATE OR REPLACE FUNCTION search_rows(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN partitions text[] DEFAULT NULL,\n IN _limit int DEFAULT 10\n) RETURNS SETOF items AS $$\nDECLARE\n base_query text;\n query text;\n sdate timestamptz;\n edate timestamptz;\n n int;\n records_left int := _limit;\n timer timestamptz := clock_timestamp();\n full_timer timestamptz := clock_timestamp();\nBEGIN\nIF _where IS NULL OR trim(_where) = '' THEN\n _where = ' TRUE ';\nEND IF;\nRAISE NOTICE 'Getting chunks for % %', _where, _orderby;\n\nbase_query := $q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n LIMIT %L\n$q$;\n\nIF _orderby ILIKE 'datetime d%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 DESC LOOP\n RAISE NOTICE 'Running Query for % to %. %', sdate, edate, age_ms(full_timer);\n query := format(\n base_query,\n sdate,\n edate,\n _where,\n _orderby,\n records_left\n );\n RAISE DEBUG 'QUERY: %', query;\n timer := clock_timestamp();\n RETURN QUERY EXECUTE query;\n\n GET DIAGNOSTICS n = ROW_COUNT;\n records_left := records_left - n;\n RAISE NOTICE 'Returned %/% Rows From % to %. % to go. Time: %ms', n, _limit, sdate, edate, records_left, age_ms(timer);\n timer := clock_timestamp();\n IF records_left <= 0 THEN\n RAISE NOTICE 'SEARCH_ROWS TOOK %ms', age_ms(full_timer);\n RETURN;\n END IF;\n END LOOP;\nELSIF _orderby ILIKE 'datetime a%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 ASC LOOP\n RAISE NOTICE 'Running Query for % to %. %', sdate, edate, age_ms(full_timer);\n query := format(\n base_query,\n sdate,\n edate,\n _where,\n _orderby,\n records_left\n );\n RAISE DEBUG 'QUERY: %', query;\n timer := clock_timestamp();\n RETURN QUERY EXECUTE query;\n\n GET DIAGNOSTICS n = ROW_COUNT;\n records_left := records_left - n;\n RAISE NOTICE 'Returned %/% Rows From % to %. % to go. Time: %ms', n, _limit, sdate, edate, records_left, age_ms(timer);\n timer := clock_timestamp();\n IF records_left <= 0 THEN\n RAISE NOTICE 'SEARCH_ROWS TOOK %ms', age_ms(full_timer);\n RETURN;\n END IF;\n END LOOP;\nELSE\n query := format($q$\n SELECT * FROM items\n WHERE %s\n ORDER BY %s\n LIMIT %L\n $q$, _where, _orderby, _limit\n );\n RAISE DEBUG 'QUERY: %', query;\n timer := clock_timestamp();\n RETURN QUERY EXECUTE query;\n RAISE NOTICE 'FULL QUERY TOOK %ms', age_ms(timer);\nEND IF;\nRAISE NOTICE 'SEARCH_ROWS TOOK %ms', age_ms(full_timer);\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE UNLOGGED TABLE format_item_cache(\n id text,\n collection text,\n fields text,\n hydrated bool,\n output jsonb,\n lastused timestamptz DEFAULT now(),\n usecount int DEFAULT 1,\n timetoformat float,\n PRIMARY KEY (collection, id, fields, hydrated)\n);\nCREATE INDEX ON format_item_cache (lastused);\n\nCREATE OR REPLACE FUNCTION format_item(_item items, _fields jsonb DEFAULT '{}', _hydrated bool DEFAULT TRUE) RETURNS jsonb AS $$\nDECLARE\n cache bool := get_setting_bool('format_cache');\n _output jsonb := null;\n t timestamptz := clock_timestamp();\nBEGIN\n IF cache THEN\n SELECT output INTO _output FROM format_item_cache\n WHERE id=_item.id AND collection=_item.collection AND fields=_fields::text AND hydrated=_hydrated;\n END IF;\n IF _output IS NULL THEN\n IF _hydrated THEN\n _output := content_hydrate(_item, _fields);\n ELSE\n _output := content_nonhydrated(_item, _fields);\n END IF;\n END IF;\n IF cache THEN\n INSERT INTO format_item_cache (id, collection, fields, hydrated, output, timetoformat)\n VALUES (_item.id, _item.collection, _fields::text, _hydrated, _output, age_ms(t))\n ON CONFLICT(collection, id, fields, hydrated) DO\n UPDATE\n SET lastused=now(), usecount = format_item_cache.usecount + 1\n ;\n END IF;\n RETURN _output;\n\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION search(_search jsonb = '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n searches searches%ROWTYPE;\n _where text;\n orderby text;\n search_where search_wheres%ROWTYPE;\n total_count bigint;\n token record;\n token_prev boolean;\n token_item items%ROWTYPE;\n token_where text;\n full_where text;\n init_ts timestamptz := clock_timestamp();\n timer timestamptz := clock_timestamp();\n hydrate bool := NOT (_search->'conf'->>'nohydrate' IS NOT NULL AND (_search->'conf'->>'nohydrate')::boolean = true);\n prev text;\n next text;\n context jsonb;\n collection jsonb;\n out_records jsonb;\n out_len int;\n _limit int := coalesce((_search->>'limit')::int, 10);\n _querylimit int;\n _fields jsonb := coalesce(_search->'fields', '{}'::jsonb);\n has_prev boolean := FALSE;\n has_next boolean := FALSE;\n links jsonb := '[]'::jsonb;\n base_url text:= concat(rtrim(base_url(_search->'conf'),'/'));\nBEGIN\n searches := search_query(_search);\n _where := searches._where;\n orderby := searches.orderby;\n search_where := where_stats(_where);\n total_count := coalesce(search_where.total_count, search_where.estimated_count);\n RAISE NOTICE 'SEARCH:TOKEN: %', _search->>'token';\n token := get_token_record(_search->>'token');\n RAISE NOTICE '***TOKEN: %', token;\n _querylimit := _limit + 1;\n IF token IS NOT NULL THEN\n token_prev := token.prev;\n token_item := token.item;\n token_where := get_token_filter(_search->'sortby', token_item, token_prev, FALSE);\n RAISE DEBUG 'TOKEN_WHERE: % (%ms from search start)', token_where, age_ms(timer);\n IF token_prev THEN -- if we are using a prev token, we know has_next is true\n RAISE DEBUG 'There is a previous token, so automatically setting has_next to true';\n has_next := TRUE;\n orderby := sort_sqlorderby(_search, TRUE);\n ELSE\n RAISE DEBUG 'There is a next token, so automatically setting has_prev to true';\n has_prev := TRUE;\n\n END IF;\n ELSE -- if there was no token, we know there is no prev\n RAISE DEBUG 'There is no token, so we know there is no prev. setting has_prev to false';\n has_prev := FALSE;\n END IF;\n\n full_where := concat_ws(' AND ', _where, token_where);\n RAISE NOTICE 'FULL WHERE CLAUSE: %', full_where;\n RAISE NOTICE 'Time to get counts and build query %', age_ms(timer);\n timer := clock_timestamp();\n\n IF hydrate THEN\n RAISE NOTICE 'Getting hydrated data.';\n ELSE\n RAISE NOTICE 'Getting non-hydrated data.';\n END IF;\n RAISE NOTICE 'CACHE SET TO %', get_setting_bool('format_cache');\n RAISE NOTICE 'Time to set hydration/formatting %', age_ms(timer);\n timer := clock_timestamp();\n SELECT jsonb_agg(format_item(i, _fields, hydrate)) INTO out_records\n FROM search_rows(\n full_where,\n orderby,\n search_where.partitions,\n _querylimit\n ) as i;\n\n RAISE NOTICE 'Time to fetch rows %', age_ms(timer);\n timer := clock_timestamp();\n\n\n IF token_prev THEN\n out_records := flip_jsonb_array(out_records);\n END IF;\n\n RAISE NOTICE 'Query returned % records.', jsonb_array_length(out_records);\n RAISE DEBUG 'TOKEN: % %', token_item.id, token_item.collection;\n RAISE DEBUG 'RECORD_1: % %', out_records->0->>'id', out_records->0->>'collection';\n RAISE DEBUG 'RECORD-1: % %', out_records->-1->>'id', out_records->-1->>'collection';\n\n -- REMOVE records that were from our token\n IF out_records->0->>'id' = token_item.id AND out_records->0->>'collection' = token_item.collection THEN\n out_records := out_records - 0;\n ELSIF out_records->-1->>'id' = token_item.id AND out_records->-1->>'collection' = token_item.collection THEN\n out_records := out_records - -1;\n END IF;\n\n out_len := jsonb_array_length(out_records);\n\n IF out_len = _limit + 1 THEN\n IF token_prev THEN\n has_prev := TRUE;\n out_records := out_records - 0;\n ELSE\n has_next := TRUE;\n out_records := out_records - -1;\n END IF;\n END IF;\n\n\n links := links || jsonb_build_object(\n 'rel', 'root',\n 'type', 'application/json',\n 'href', base_url\n ) || jsonb_build_object(\n 'rel', 'self',\n 'type', 'application/json',\n 'href', concat(base_url, '/search')\n );\n\n IF has_next THEN\n next := concat(out_records->-1->>'collection', ':', out_records->-1->>'id');\n RAISE NOTICE 'HAS NEXT | %', next;\n links := links || jsonb_build_object(\n 'rel', 'next',\n 'type', 'application/geo+json',\n 'method', 'GET',\n 'href', concat(base_url, '/search?token=next:', next)\n );\n END IF;\n\n IF has_prev THEN\n prev := concat(out_records->0->>'collection', ':', out_records->0->>'id');\n RAISE NOTICE 'HAS PREV | %', prev;\n links := links || jsonb_build_object(\n 'rel', 'prev',\n 'type', 'application/geo+json',\n 'method', 'GET',\n 'href', concat(base_url, '/search?token=prev:', prev)\n );\n END IF;\n\n RAISE NOTICE 'Time to get prev/next %', age_ms(timer);\n timer := clock_timestamp();\n\n\n collection := jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb),\n 'links', links\n );\n\n\n\n IF context(_search->'conf') != 'off' THEN\n collection := collection || jsonb_strip_nulls(jsonb_build_object(\n 'numberMatched', total_count,\n 'numberReturned', coalesce(jsonb_array_length(out_records), 0)\n ));\n ELSE\n collection := collection || jsonb_strip_nulls(jsonb_build_object(\n 'numberReturned', coalesce(jsonb_array_length(out_records), 0)\n ));\n END IF;\n\n IF get_setting_bool('timing', _search->'conf') THEN\n collection = collection || jsonb_build_object('timing', age_ms(init_ts));\n END IF;\n\n RAISE NOTICE 'Time to build final json %', age_ms(timer);\n timer := clock_timestamp();\n\n RAISE NOTICE 'Total Time: %', age_ms(current_timestamp);\n RAISE NOTICE 'RETURNING % records. NEXT: %. PREV: %', collection->>'numberReturned', collection->>'next', collection->>'prev';\n RETURN collection;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION search_cursor(_search jsonb = '{}'::jsonb) RETURNS refcursor AS $$\nDECLARE\n curs refcursor;\n searches searches%ROWTYPE;\n _where text;\n _orderby text;\n q text;\n\nBEGIN\n searches := search_query(_search);\n _where := searches._where;\n _orderby := searches.orderby;\n\n OPEN curs FOR\n WITH p AS (\n SELECT * FROM partition_queries(_where, _orderby) p\n )\n SELECT\n CASE WHEN EXISTS (SELECT 1 FROM p) THEN\n (SELECT format($q$\n SELECT * FROM (\n %s\n ) total\n $q$,\n string_agg(\n format($q$ SELECT * FROM ( %s ) AS sub $q$, p),\n '\n UNION ALL\n '\n )\n ))\n ELSE NULL\n END FROM p;\n RETURN curs;\nEND;\n$$ LANGUAGE PLPGSQL;\nCREATE OR REPLACE VIEW collections_asitems AS\nSELECT\n id,\n geometry,\n 'collections' AS collection,\n datetime,\n end_datetime,\n jsonb_build_object(\n 'properties', content - '{links,assets,stac_version,stac_extensions}',\n 'links', content->'links',\n 'assets', content->'assets',\n 'stac_version', content->'stac_version',\n 'stac_extensions', content->'stac_extensions'\n ) AS content,\n content as collectionjson\nFROM collections;\n\n\nCREATE OR REPLACE FUNCTION collection_search_matched(\n IN _search jsonb DEFAULT '{}'::jsonb,\n OUT matched bigint\n) RETURNS bigint AS $$\nDECLARE\n _where text := stac_search_to_where(_search);\nBEGIN\n EXECUTE format(\n $query$\n SELECT\n count(*)\n FROM\n collections_asitems\n WHERE %s\n ;\n $query$,\n _where\n ) INTO matched;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION collection_search_rows(\n _search jsonb DEFAULT '{}'::jsonb\n) RETURNS SETOF jsonb AS $$\nDECLARE\n _where text := stac_search_to_where(_search);\n _limit int := coalesce((_search->>'limit')::int, 10);\n _fields jsonb := coalesce(_search->'fields', '{}'::jsonb);\n _orderby text;\n _offset int := COALESCE((_search->>'offset')::int, 0);\nBEGIN\n _orderby := sort_sqlorderby(\n jsonb_build_object(\n 'sortby',\n coalesce(\n _search->'sortby',\n '[{\"field\": \"id\", \"direction\": \"asc\"}]'::jsonb\n )\n )\n );\n RETURN QUERY EXECUTE format(\n $query$\n SELECT\n jsonb_fields(collectionjson, %L) as c\n FROM\n collections_asitems\n WHERE %s\n ORDER BY %s\n LIMIT %L\n OFFSET %L\n ;\n $query$,\n _fields,\n _where,\n _orderby,\n _limit,\n _offset\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION collection_search(\n _search jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\nDECLARE\n out_records jsonb;\n number_matched bigint := collection_search_matched(_search);\n number_returned bigint;\n _limit int := coalesce((_search->>'limit')::float::int, 10);\n _offset int := coalesce((_search->>'offset')::float::int, 0);\n links jsonb := '[]';\n ret jsonb;\n base_url text:= concat(rtrim(base_url(_search->'conf'),'/'), '/collections');\n prevoffset int;\n nextoffset int;\nBEGIN\n SELECT\n coalesce(jsonb_agg(c), '[]')\n INTO out_records\n FROM collection_search_rows(_search) c;\n\n number_returned := jsonb_array_length(out_records);\n RAISE DEBUG 'nm: %, nr: %, l:%, o:%', number_matched, number_returned, _limit, _offset;\n\n\n\n IF _limit <= number_matched AND number_matched > 0 THEN --need to have paging links\n nextoffset := least(_offset + _limit, number_matched - 1);\n prevoffset := greatest(_offset - _limit, 0);\n\n IF _offset > 0 THEN\n links := links || jsonb_build_object(\n 'rel', 'prev',\n 'type', 'application/json',\n 'method', 'GET' ,\n 'href', base_url,\n 'body', jsonb_build_object('offset', prevoffset),\n 'merge', TRUE\n );\n END IF;\n\n IF (_offset + _limit < number_matched) THEN\n links := links || jsonb_build_object(\n 'rel', 'next',\n 'type', 'application/json',\n 'method', 'GET' ,\n 'href', base_url,\n 'body', jsonb_build_object('offset', nextoffset),\n 'merge', TRUE\n );\n END IF;\n\n END IF;\n\n ret := jsonb_build_object(\n 'collections', out_records,\n 'numberMatched', number_matched,\n 'numberReturned', number_returned,\n 'links', links\n );\n RETURN ret;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION tileenvelope(zoom int, x int, y int) RETURNS geometry AS $$\nWITH t AS (\n SELECT\n 20037508.3427892 as merc_max,\n -20037508.3427892 as merc_min,\n (2 * 20037508.3427892) / (2 ^ zoom) as tile_size\n)\nSELECT st_makeenvelope(\n merc_min + (tile_size * x),\n merc_max - (tile_size * (y + 1)),\n merc_min + (tile_size * (x + 1)),\n merc_max - (tile_size * y),\n 3857\n) FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;DROP FUNCTION IF EXISTS mercgrid;\n\n\nCREATE OR REPLACE FUNCTION ftime() RETURNS interval as $$\nSELECT age(clock_timestamp(), transaction_timestamp());\n$$ LANGUAGE SQL;\nSET SEARCH_PATH to pgstac, public;\n\nDROP FUNCTION IF EXISTS geometrysearch;\nCREATE OR REPLACE FUNCTION geometrysearch(\n IN geom geometry,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE, -- Return as soon as the passed in geometry is full covered\n IN skipcovered boolean DEFAULT TRUE -- Skip any items that would show up completely under the previous items\n) RETURNS jsonb AS $$\nDECLARE\n search searches%ROWTYPE;\n curs refcursor;\n _where text;\n query text;\n iter_record items%ROWTYPE;\n out_records jsonb := '{}'::jsonb[];\n exit_flag boolean := FALSE;\n counter int := 1;\n scancounter int := 1;\n remaining_limit int := _scanlimit;\n tilearea float;\n unionedgeom geometry;\n clippedgeom geometry;\n unionedgeom_area float := 0;\n prev_area float := 0;\n excludes text[];\n includes text[];\n\nBEGIN\n DROP TABLE IF EXISTS pgstac_results;\n CREATE TEMP TABLE pgstac_results (content jsonb) ON COMMIT DROP;\n\n -- If the passed in geometry is not an area set exitwhenfull and skipcovered to false\n IF ST_GeometryType(geom) !~* 'polygon' THEN\n RAISE NOTICE 'GEOMETRY IS NOT AN AREA';\n skipcovered = FALSE;\n exitwhenfull = FALSE;\n END IF;\n\n -- If skipcovered is true then you will always want to exit when the passed in geometry is full\n IF skipcovered THEN\n exitwhenfull := TRUE;\n END IF;\n\n search := search_fromhash(queryhash);\n\n IF search IS NULL THEN\n RAISE EXCEPTION 'Search with Query Hash % Not Found', queryhash;\n END IF;\n\n tilearea := st_area(geom);\n _where := format('%s AND st_intersects(geometry, %L::geometry)', search._where, geom);\n\n\n FOR query IN SELECT * FROM partition_queries(_where, search.orderby) LOOP\n query := format('%s LIMIT %L', query, remaining_limit);\n RAISE NOTICE '%', query;\n OPEN curs FOR EXECUTE query;\n LOOP\n FETCH curs INTO iter_record;\n EXIT WHEN NOT FOUND;\n IF exitwhenfull OR skipcovered THEN -- If we are not using exitwhenfull or skipcovered, we do not need to do expensive geometry operations\n clippedgeom := st_intersection(geom, iter_record.geometry);\n\n IF unionedgeom IS NULL THEN\n unionedgeom := clippedgeom;\n ELSE\n unionedgeom := st_union(unionedgeom, clippedgeom);\n END IF;\n\n unionedgeom_area := st_area(unionedgeom);\n\n IF skipcovered AND prev_area = unionedgeom_area THEN\n scancounter := scancounter + 1;\n CONTINUE;\n END IF;\n\n prev_area := unionedgeom_area;\n\n RAISE NOTICE '% % % %', unionedgeom_area/tilearea, counter, scancounter, ftime();\n END IF;\n RAISE NOTICE '% %', iter_record, content_hydrate(iter_record, fields);\n INSERT INTO pgstac_results (content) VALUES (content_hydrate(iter_record, fields));\n\n IF counter >= _limit\n OR scancounter > _scanlimit\n OR ftime() > _timelimit\n OR (exitwhenfull AND unionedgeom_area >= tilearea)\n THEN\n exit_flag := TRUE;\n EXIT;\n END IF;\n counter := counter + 1;\n scancounter := scancounter + 1;\n\n END LOOP;\n CLOSE curs;\n EXIT WHEN exit_flag;\n remaining_limit := _scanlimit - scancounter;\n END LOOP;\n\n SELECT jsonb_agg(content) INTO out_records FROM pgstac_results WHERE content IS NOT NULL;\n\n RETURN jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb)\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDROP FUNCTION IF EXISTS geojsonsearch;\nCREATE OR REPLACE FUNCTION geojsonsearch(\n IN geojson jsonb,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_geomfromgeojson(geojson),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS xyzsearch;\nCREATE OR REPLACE FUNCTION xyzsearch(\n IN _x int,\n IN _y int,\n IN _z int,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_transform(tileenvelope(_z, _x, _y), 4326),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE PROCEDURE analyze_items() AS $$\nDECLARE\n q text;\n timeout_ts timestamptz;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n SELECT format('ANALYZE (VERBOSE, SKIP_LOCKED) %I;', relname) INTO q\n FROM pg_stat_user_tables\n WHERE relname like '_item%' AND (n_mod_since_analyze>0 OR last_analyze IS NULL) LIMIT 1;\n IF NOT FOUND THEN\n EXIT;\n END IF;\n RAISE NOTICE '%', q;\n EXECUTE q;\n COMMIT;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE PROCEDURE validate_constraints() AS $$\nDECLARE\n q text;\nBEGIN\n FOR q IN\n SELECT\n FORMAT(\n 'ALTER TABLE %I.%I VALIDATE CONSTRAINT %I;',\n nsp.nspname,\n cls.relname,\n con.conname\n )\n\n FROM pg_constraint AS con\n JOIN pg_class AS cls\n ON con.conrelid = cls.oid\n JOIN pg_namespace AS nsp\n ON cls.relnamespace = nsp.oid\n WHERE convalidated = FALSE AND contype in ('c','f')\n AND nsp.nspname = 'pgstac'\n LOOP\n RAISE NOTICE '%', q;\n PERFORM run_or_queue(q);\n COMMIT;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION collection_extent(_collection text, runupdate boolean default false) RETURNS jsonb AS $$\nDECLARE\n geom_extent geometry;\n mind timestamptz;\n maxd timestamptz;\n extent jsonb;\nBEGIN\n IF runupdate THEN\n PERFORM update_partition_stats_q(partition)\n FROM partitions_view WHERE collection=_collection;\n END IF;\n SELECT\n min(lower(dtrange)),\n max(upper(edtrange)),\n st_extent(spatial)\n INTO\n mind,\n maxd,\n geom_extent\n FROM partitions_view\n WHERE collection=_collection;\n\n IF geom_extent IS NOT NULL AND mind IS NOT NULL AND maxd IS NOT NULL THEN\n extent := jsonb_build_object(\n 'spatial', jsonb_build_object(\n 'bbox', to_jsonb(array[array[st_xmin(geom_extent), st_ymin(geom_extent), st_xmax(geom_extent), st_ymax(geom_extent)]])\n ),\n 'temporal', jsonb_build_object(\n 'interval', to_jsonb(array[array[mind, maxd]])\n )\n );\n RETURN extent;\n END IF;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('id', '{\"title\": \"Item ID\",\"description\": \"Item identifier\",\"$ref\": \"https://schemas.stacspec.org/v1.0.0/item-spec/json-schema/item.json#/definitions/core/allOf/2/properties/id\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('geometry', '{\"title\": \"Item Geometry\",\"description\": \"Item Geometry\",\"$ref\": \"https://geojson.org/schema/Feature.json\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('datetime','{\"description\": \"Datetime\",\"type\": \"string\",\"title\": \"Acquired\",\"format\": \"date-time\",\"pattern\": \"(\\\\+00:00|Z)$\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDELETE FROM queryables a USING queryables b\n WHERE a.name = b.name AND a.collection_ids IS NOT DISTINCT FROM b.collection_ids AND a.id > b.id;\n\n\nINSERT INTO pgstac_settings (name, value) VALUES\n ('context', 'off'),\n ('context_estimated_count', '100000'),\n ('context_estimated_cost', '100000'),\n ('context_stats_ttl', '1 day'),\n ('default_filter_lang', 'cql2-json'),\n ('additional_properties', 'true'),\n ('use_queue', 'false'),\n ('queue_timeout', '10 minutes'),\n ('update_collection_extent', 'false'),\n ('format_cache', 'false'),\n ('readonly', 'false')\nON CONFLICT DO NOTHING\n;\n\n\nINSERT INTO cql2_ops (op, template, types) VALUES\n ('eq', '%s = %s', NULL),\n ('neq', '%s != %s', NULL),\n ('ne', '%s != %s', NULL),\n ('!=', '%s != %s', NULL),\n ('<>', '%s != %s', NULL),\n ('lt', '%s < %s', NULL),\n ('lte', '%s <= %s', NULL),\n ('gt', '%s > %s', NULL),\n ('gte', '%s >= %s', NULL),\n ('le', '%s <= %s', NULL),\n ('ge', '%s >= %s', NULL),\n ('=', '%s = %s', NULL),\n ('<', '%s < %s', NULL),\n ('<=', '%s <= %s', NULL),\n ('>', '%s > %s', NULL),\n ('>=', '%s >= %s', NULL),\n ('like', '%s LIKE %s', NULL),\n ('ilike', '%s ILIKE %s', NULL),\n ('+', '%s + %s', NULL),\n ('-', '%s - %s', NULL),\n ('*', '%s * %s', NULL),\n ('/', '%s / %s', NULL),\n ('not', 'NOT (%s)', NULL),\n ('between', '%s BETWEEN %s AND %s', NULL),\n ('isnull', '%s IS NULL', NULL),\n ('upper', 'upper(%s)', NULL),\n ('lower', 'lower(%s)', NULL),\n ('casei', 'upper(%s)', NULL),\n ('accenti', 'unaccent(%s)', NULL)\nON CONFLICT (op) DO UPDATE\n SET\n template = EXCLUDED.template\n;\n\n\nALTER FUNCTION to_text COST 5000;\nALTER FUNCTION to_float COST 5000;\nALTER FUNCTION to_int COST 5000;\nALTER FUNCTION to_tstz COST 5000;\nALTER FUNCTION to_text_array COST 5000;\n\nALTER FUNCTION update_partition_stats SECURITY DEFINER;\nALTER FUNCTION partition_after_triggerfunc SECURITY DEFINER;\nALTER FUNCTION drop_table_constraints SECURITY DEFINER;\nALTER FUNCTION create_table_constraints SECURITY DEFINER;\nALTER FUNCTION check_partition SECURITY DEFINER;\nALTER FUNCTION repartition SECURITY DEFINER;\nALTER FUNCTION where_stats SECURITY DEFINER;\nALTER FUNCTION search_query SECURITY DEFINER;\nALTER FUNCTION format_item SECURITY DEFINER;\nALTER FUNCTION maintain_index SECURITY DEFINER;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nGRANT ALL ON SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac to pgstac_admin;\n\n-- pgstac_read role limited to using function apis\nGRANT EXECUTE ON FUNCTION search TO pgstac_read;\nGRANT EXECUTE ON FUNCTION search_query TO pgstac_read;\nGRANT EXECUTE ON FUNCTION item_by_id TO pgstac_read;\nGRANT EXECUTE ON FUNCTION get_item TO pgstac_read;\nGRANT SELECT ON ALL TABLES IN SCHEMA pgstac TO pgstac_read;\n\n\nGRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_ingest;\nGRANT USAGE ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_ingest;\n\nREVOKE ALL PRIVILEGES ON PROCEDURE run_queued_queries FROM public;\nGRANT ALL ON PROCEDURE run_queued_queries TO pgstac_admin;\n\nREVOKE ALL PRIVILEGES ON FUNCTION run_queued_queries_intransaction FROM public;\nGRANT ALL ON FUNCTION run_queued_queries_intransaction TO pgstac_admin;\n\nRESET ROLE;\n\nSET ROLE pgstac_ingest;\nSELECT update_partition_stats_q(partition) FROM partitions_view;\nSELECT set_version('0.9.7');\n" }, { "path": "src/pgstac/migrations/pgstac.0.9.8-0.9.9.sql", "content": "SET client_min_messages TO WARNING;\nSET SEARCH_PATH to pgstac, public;\nRESET ROLE;\nDO $$\nDECLARE\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='postgis') THEN\n CREATE EXTENSION IF NOT EXISTS postgis;\n END IF;\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='btree_gist') THEN\n CREATE EXTENSION IF NOT EXISTS btree_gist;\n END IF;\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='unaccent') THEN\n CREATE EXTENSION IF NOT EXISTS unaccent;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_admin;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_read;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_ingest;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n\nGRANT pgstac_admin TO current_user;\n\n-- Function to make sure pgstac_admin is the owner of items\nCREATE OR REPLACE FUNCTION pgstac_admin_owns() RETURNS VOID AS $$\nDECLARE\n f RECORD;\nBEGIN\n FOR f IN (\n SELECT\n concat(\n oid::regproc::text,\n '(',\n coalesce(pg_get_function_identity_arguments(oid),''),\n ')'\n ) AS name,\n CASE prokind WHEN 'f' THEN 'FUNCTION' WHEN 'p' THEN 'PROCEDURE' WHEN 'a' THEN 'AGGREGATE' END as typ\n FROM pg_proc\n WHERE\n pronamespace=to_regnamespace('pgstac')\n AND proowner != to_regrole('pgstac_admin')\n AND proname NOT LIKE 'pg_stat%'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n FOR f IN (\n SELECT\n oid::regclass::text as name,\n CASE relkind\n WHEN 'i' THEN 'INDEX'\n WHEN 'I' THEN 'INDEX'\n WHEN 'p' THEN 'TABLE'\n WHEN 'r' THEN 'TABLE'\n WHEN 'v' THEN 'VIEW'\n WHEN 'S' THEN 'SEQUENCE'\n ELSE NULL\n END as typ\n FROM pg_class\n WHERE relnamespace=to_regnamespace('pgstac') and relowner != to_regrole('pgstac_admin') AND relkind IN ('r','p','v','S') AND relname NOT LIKE 'pg_stat'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\nSELECT pgstac_admin_owns();\n\nCREATE SCHEMA IF NOT EXISTS pgstac AUTHORIZATION pgstac_admin;\n\nGRANT ALL ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_admin;\n\nALTER ROLE pgstac_admin SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_read SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_ingest SET SEARCH_PATH TO pgstac, public;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\n\nGRANT pgstac_read TO pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\n\nSET ROLE pgstac_admin;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\nRESET ROLE;\n\nSET ROLE pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\nRESET ROLE;\n\nSET SEARCH_PATH TO pgstac, public;\nSET ROLE pgstac_admin;\n\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS analyze_items;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS validate_constraints;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n-- Install these idempotently as migrations do not put them before trying to modify the collections table\n\n\nCREATE OR REPLACE FUNCTION collection_geom(content jsonb)\nRETURNS geometry AS $$\n WITH box AS (SELECT content->'extent'->'spatial'->'bbox'->0 as box)\n SELECT\n st_makeenvelope(\n (box->>0)::float,\n (box->>1)::float,\n (box->>2)::float,\n (box->>3)::float,\n 4326\n )\n FROM box;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION collection_datetime(content jsonb)\nRETURNS timestamptz AS $$\n SELECT\n CASE\n WHEN\n (content->'extent'->'temporal'->'interval'->0->>0) IS NULL\n THEN '-infinity'::timestamptz\n ELSE\n (content->'extent'->'temporal'->'interval'->0->>0)::timestamptz\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION collection_enddatetime(content jsonb)\nRETURNS timestamptz AS $$\n SELECT\n CASE\n WHEN\n (content->'extent'->'temporal'->'interval'->0->>1) IS NULL\n THEN 'infinity'::timestamptz\n ELSE\n (content->'extent'->'temporal'->'interval'->0->>1)::timestamptz\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n-- BEGIN migra calculated SQL\nset check_function_bodies = off;\n\nCREATE OR REPLACE FUNCTION pgstac.jsonb_include(j jsonb, f jsonb)\n RETURNS jsonb\n LANGUAGE plpgsql\n IMMUTABLE\nAS $function$\nDECLARE\n includes jsonb := f-> 'include';\n outj jsonb := '{}'::jsonb;\n path text[];\nBEGIN\n IF\n includes IS NULL\n OR jsonb_array_length(includes) = 0\n THEN\n RETURN j;\n ELSE\n includes := includes || (\n CASE WHEN j ? 'collection' THEN\n '[\"id\",\"collection\"]'\n ELSE\n '[\"id\"]'\n END)::jsonb;\n FOR path IN SELECT explode_dotpaths(includes) LOOP\n outj := jsonb_set_nested(outj, path, j #> path);\n END LOOP;\n END IF;\n RETURN outj;\nEND;\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.q_to_tsquery(jinput jsonb)\n RETURNS tsquery\n LANGUAGE plpgsql\nAS $function$\nDECLARE\n input text;\n processed_text text;\n temp_text text;\n quote_array text[];\n placeholder text := '@QUOTE@';\nBEGIN\n IF jsonb_typeof(jinput) = 'string' THEN\n input := jinput->>0;\n ELSIF jsonb_typeof(jinput) = 'array' THEN\n input := array_to_string(\n array(select jsonb_array_elements_text(jinput)),\n ' OR '\n );\n ELSE\n RAISE EXCEPTION 'Input must be a string or an array of strings.';\n END IF;\n -- Extract all quoted phrases and store in array\n quote_array := regexp_matches(input, '\"[^\"]*\"', 'g');\n\n -- Replace each quoted part with a unique placeholder if there are any quoted phrases\n IF array_length(quote_array, 1) IS NOT NULL THEN\n processed_text := input;\n FOR i IN array_lower(quote_array, 1) .. array_upper(quote_array, 1) LOOP\n processed_text := replace(processed_text, quote_array[i], placeholder || i || placeholder);\n END LOOP;\n ELSE\n processed_text := input;\n END IF;\n\n -- Replace non-quoted text using regular expressions\n\n -- , -> |\n processed_text := regexp_replace(processed_text, ',(?=(?:[^\"]*\"[^\"]*\")*[^\"]*$)', ' | ', 'g');\n\n -- and -> &\n processed_text := regexp_replace(processed_text, '\\s+AND\\s+', ' & ', 'gi');\n\n -- or -> |\n processed_text := regexp_replace(processed_text, '\\s+OR\\s+', ' | ', 'gi');\n\n -- + ->\n processed_text := regexp_replace(processed_text, '^\\s*\\+([a-zA-Z0-9_]+)', '\\1', 'g'); -- +term at start\n processed_text := regexp_replace(processed_text, '\\s*\\+([a-zA-Z0-9_]+)', ' & \\1', 'g'); -- +term elsewhere\n\n -- - -> !\n processed_text := regexp_replace(processed_text, '^\\s*\\-([a-zA-Z0-9_]+)', '! \\1', 'g'); -- -term at start\n processed_text := regexp_replace(processed_text, '\\s*\\-([a-zA-Z0-9_]+)', ' & ! \\1', 'g'); -- -term elsewhere\n\n -- terms separated with spaces are assumed to represent adjacent terms. loop through these\n -- occurrences and replace them with the adjacency operator (<->)\n LOOP\n temp_text := regexp_replace(processed_text, '([a-zA-Z0-9_]+)\\s+([a-zA-Z0-9_]+)(?!\\s*[&|<>])', '\\1 <-> \\2', 'g');\n IF temp_text = processed_text THEN\n EXIT; -- No more replacements were made\n END IF;\n processed_text := temp_text;\n END LOOP;\n\n\n -- Replace placeholders back with quoted phrases if there were any\n IF array_length(quote_array, 1) IS NOT NULL THEN\n FOR i IN array_lower(quote_array, 1) .. array_upper(quote_array, 1) LOOP\n processed_text := replace(processed_text, placeholder || i || placeholder, '''' || substring(quote_array[i] from 2 for length(quote_array[i]) - 2) || '''');\n END LOOP;\n END IF;\n\n -- Print processed_text to the console for debugging purposes\n RAISE NOTICE 'processed_text: %', processed_text;\n\n RETURN to_tsquery('english', processed_text);\nEND;\n$function$\n;\n\n\n-- END migra calculated SQL\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('id', '{\"title\": \"Item ID\",\"description\": \"Item identifier\",\"$ref\": \"https://schemas.stacspec.org/v1.0.0/item-spec/json-schema/item.json#/definitions/core/allOf/2/properties/id\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('geometry', '{\"title\": \"Item Geometry\",\"description\": \"Item Geometry\",\"$ref\": \"https://geojson.org/schema/Feature.json\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('datetime','{\"description\": \"Datetime\",\"type\": \"string\",\"title\": \"Acquired\",\"format\": \"date-time\",\"pattern\": \"(\\\\+00:00|Z)$\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDELETE FROM queryables a USING queryables b\n WHERE a.name = b.name AND a.collection_ids IS NOT DISTINCT FROM b.collection_ids AND a.id > b.id;\n\n\nINSERT INTO pgstac_settings (name, value) VALUES\n ('context', 'off'),\n ('context_estimated_count', '100000'),\n ('context_estimated_cost', '100000'),\n ('context_stats_ttl', '1 day'),\n ('default_filter_lang', 'cql2-json'),\n ('additional_properties', 'true'),\n ('use_queue', 'false'),\n ('queue_timeout', '10 minutes'),\n ('update_collection_extent', 'false'),\n ('format_cache', 'false'),\n ('readonly', 'false')\nON CONFLICT DO NOTHING\n;\n\n\nINSERT INTO cql2_ops (op, template, types) VALUES\n ('eq', '%s = %s', NULL),\n ('neq', '%s != %s', NULL),\n ('ne', '%s != %s', NULL),\n ('!=', '%s != %s', NULL),\n ('<>', '%s != %s', NULL),\n ('lt', '%s < %s', NULL),\n ('lte', '%s <= %s', NULL),\n ('gt', '%s > %s', NULL),\n ('gte', '%s >= %s', NULL),\n ('le', '%s <= %s', NULL),\n ('ge', '%s >= %s', NULL),\n ('=', '%s = %s', NULL),\n ('<', '%s < %s', NULL),\n ('<=', '%s <= %s', NULL),\n ('>', '%s > %s', NULL),\n ('>=', '%s >= %s', NULL),\n ('like', '%s LIKE %s', NULL),\n ('ilike', '%s ILIKE %s', NULL),\n ('+', '%s + %s', NULL),\n ('-', '%s - %s', NULL),\n ('*', '%s * %s', NULL),\n ('/', '%s / %s', NULL),\n ('not', 'NOT (%s)', NULL),\n ('between', '%s BETWEEN %s AND %s', NULL),\n ('isnull', '%s IS NULL', NULL),\n ('upper', 'upper(%s)', NULL),\n ('lower', 'lower(%s)', NULL),\n ('casei', 'upper(%s)', NULL),\n ('accenti', 'unaccent(%s)', NULL)\nON CONFLICT (op) DO UPDATE\n SET\n template = EXCLUDED.template\n;\n\n\nALTER FUNCTION to_text COST 5000;\nALTER FUNCTION to_float COST 5000;\nALTER FUNCTION to_int COST 5000;\nALTER FUNCTION to_tstz COST 5000;\nALTER FUNCTION to_text_array COST 5000;\n\nALTER FUNCTION update_partition_stats SECURITY DEFINER;\nALTER FUNCTION partition_after_triggerfunc SECURITY DEFINER;\nALTER FUNCTION drop_table_constraints SECURITY DEFINER;\nALTER FUNCTION create_table_constraints SECURITY DEFINER;\nALTER FUNCTION check_partition SECURITY DEFINER;\nALTER FUNCTION repartition SECURITY DEFINER;\nALTER FUNCTION where_stats SECURITY DEFINER;\nALTER FUNCTION search_query SECURITY DEFINER;\nALTER FUNCTION format_item SECURITY DEFINER;\nALTER FUNCTION maintain_index SECURITY DEFINER;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nGRANT ALL ON SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac to pgstac_admin;\n\n-- pgstac_read role limited to using function apis\nGRANT EXECUTE ON FUNCTION search TO pgstac_read;\nGRANT EXECUTE ON FUNCTION search_query TO pgstac_read;\nGRANT EXECUTE ON FUNCTION item_by_id TO pgstac_read;\nGRANT EXECUTE ON FUNCTION get_item TO pgstac_read;\nGRANT SELECT ON ALL TABLES IN SCHEMA pgstac TO pgstac_read;\n\n\nGRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_ingest;\nGRANT USAGE ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_ingest;\n\nREVOKE ALL PRIVILEGES ON PROCEDURE run_queued_queries FROM public;\nGRANT ALL ON PROCEDURE run_queued_queries TO pgstac_admin;\n\nREVOKE ALL PRIVILEGES ON FUNCTION run_queued_queries_intransaction FROM public;\nGRANT ALL ON FUNCTION run_queued_queries_intransaction TO pgstac_admin;\n\nRESET ROLE;\n\nSET ROLE pgstac_ingest;\nSELECT update_partition_stats_q(partition) FROM partitions_view;\nSELECT set_version('0.9.9');\n" }, { "path": "src/pgstac/migrations/pgstac.0.9.8.sql", "content": "RESET ROLE;\nDO $$\nDECLARE\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='postgis') THEN\n CREATE EXTENSION IF NOT EXISTS postgis;\n END IF;\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='btree_gist') THEN\n CREATE EXTENSION IF NOT EXISTS btree_gist;\n END IF;\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='unaccent') THEN\n CREATE EXTENSION IF NOT EXISTS unaccent;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_admin;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_read;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_ingest;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n\nGRANT pgstac_admin TO current_user;\n\n-- Function to make sure pgstac_admin is the owner of items\nCREATE OR REPLACE FUNCTION pgstac_admin_owns() RETURNS VOID AS $$\nDECLARE\n f RECORD;\nBEGIN\n FOR f IN (\n SELECT\n concat(\n oid::regproc::text,\n '(',\n coalesce(pg_get_function_identity_arguments(oid),''),\n ')'\n ) AS name,\n CASE prokind WHEN 'f' THEN 'FUNCTION' WHEN 'p' THEN 'PROCEDURE' WHEN 'a' THEN 'AGGREGATE' END as typ\n FROM pg_proc\n WHERE\n pronamespace=to_regnamespace('pgstac')\n AND proowner != to_regrole('pgstac_admin')\n AND proname NOT LIKE 'pg_stat%'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n FOR f IN (\n SELECT\n oid::regclass::text as name,\n CASE relkind\n WHEN 'i' THEN 'INDEX'\n WHEN 'I' THEN 'INDEX'\n WHEN 'p' THEN 'TABLE'\n WHEN 'r' THEN 'TABLE'\n WHEN 'v' THEN 'VIEW'\n WHEN 'S' THEN 'SEQUENCE'\n ELSE NULL\n END as typ\n FROM pg_class\n WHERE relnamespace=to_regnamespace('pgstac') and relowner != to_regrole('pgstac_admin') AND relkind IN ('r','p','v','S') AND relname NOT LIKE 'pg_stat'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\nSELECT pgstac_admin_owns();\n\nCREATE SCHEMA IF NOT EXISTS pgstac AUTHORIZATION pgstac_admin;\n\nGRANT ALL ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_admin;\n\nALTER ROLE pgstac_admin SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_read SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_ingest SET SEARCH_PATH TO pgstac, public;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\n\nGRANT pgstac_read TO pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\n\nSET ROLE pgstac_admin;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\nRESET ROLE;\n\nSET ROLE pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\nRESET ROLE;\n\nSET SEARCH_PATH TO pgstac, public;\nSET ROLE pgstac_admin;\n\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS analyze_items;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS validate_constraints;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n-- Install these idempotently as migrations do not put them before trying to modify the collections table\n\n\nCREATE OR REPLACE FUNCTION collection_geom(content jsonb)\nRETURNS geometry AS $$\n WITH box AS (SELECT content->'extent'->'spatial'->'bbox'->0 as box)\n SELECT\n st_makeenvelope(\n (box->>0)::float,\n (box->>1)::float,\n (box->>2)::float,\n (box->>3)::float,\n 4326\n )\n FROM box;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION collection_datetime(content jsonb)\nRETURNS timestamptz AS $$\n SELECT\n CASE\n WHEN\n (content->'extent'->'temporal'->'interval'->0->>0) IS NULL\n THEN '-infinity'::timestamptz\n ELSE\n (content->'extent'->'temporal'->'interval'->0->>0)::timestamptz\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION collection_enddatetime(content jsonb)\nRETURNS timestamptz AS $$\n SELECT\n CASE\n WHEN\n (content->'extent'->'temporal'->'interval'->0->>1) IS NULL\n THEN 'infinity'::timestamptz\n ELSE\n (content->'extent'->'temporal'->'interval'->0->>1)::timestamptz\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE TABLE IF NOT EXISTS migrations (\n version text PRIMARY KEY,\n datetime timestamptz DEFAULT clock_timestamp() NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION get_version() RETURNS text AS $$\n SELECT version FROM pgstac.migrations ORDER BY datetime DESC, version DESC LIMIT 1;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION set_version(text) RETURNS text AS $$\n INSERT INTO pgstac.migrations (version) VALUES ($1)\n ON CONFLICT DO NOTHING\n RETURNING version;\n$$ LANGUAGE SQL;\n\n\nCREATE TABLE IF NOT EXISTS pgstac_settings (\n name text PRIMARY KEY,\n value text NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION table_empty(text) RETURNS boolean AS $$\nDECLARE\n retval boolean;\nBEGIN\n EXECUTE format($q$\n SELECT NOT EXISTS (SELECT 1 FROM %I LIMIT 1)\n $q$,\n $1\n ) INTO retval;\n RETURN retval;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION get_setting(IN _setting text, IN conf jsonb DEFAULT NULL) RETURNS text AS $$\nSELECT COALESCE(\n nullif(conf->>_setting, ''),\n nullif(current_setting(concat('pgstac.',_setting), TRUE),''),\n nullif((SELECT value FROM pgstac.pgstac_settings WHERE name=_setting),'')\n);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_setting_bool(IN _setting text, IN conf jsonb DEFAULT NULL) RETURNS boolean AS $$\nSELECT COALESCE(\n nullif(conf->>_setting, ''),\n nullif(current_setting(concat('pgstac.',_setting), TRUE),''),\n nullif((SELECT value FROM pgstac.pgstac_settings WHERE name=_setting),''),\n 'FALSE'\n)::boolean;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION base_url(conf jsonb DEFAULT NULL) RETURNS text AS $$\n SELECT COALESCE(pgstac.get_setting('base_url', conf), '.');\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION additional_properties() RETURNS boolean AS $$\n SELECT pgstac.get_setting_bool('additional_properties');\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION readonly(conf jsonb DEFAULT NULL) RETURNS boolean AS $$\n SELECT pgstac.get_setting_bool('readonly', conf);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context(conf jsonb DEFAULT NULL) RETURNS text AS $$\n SELECT pgstac.get_setting('context', conf);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context_estimated_count(conf jsonb DEFAULT NULL) RETURNS int AS $$\n SELECT pgstac.get_setting('context_estimated_count', conf)::int;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_estimated_cost();\nCREATE OR REPLACE FUNCTION context_estimated_cost(conf jsonb DEFAULT NULL) RETURNS float AS $$\n SELECT pgstac.get_setting('context_estimated_cost', conf)::float;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_stats_ttl();\nCREATE OR REPLACE FUNCTION context_stats_ttl(conf jsonb DEFAULT NULL) RETURNS interval AS $$\n SELECT pgstac.get_setting('context_stats_ttl', conf)::interval;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION t2s(text) RETURNS text AS $$\n SELECT extract(epoch FROM $1::interval)::text || ' s';\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE STRICT;\n\nCREATE OR REPLACE FUNCTION age_ms(a timestamptz, b timestamptz DEFAULT clock_timestamp()) RETURNS float AS $$\n SELECT abs(extract(epoch from age(a,b)) * 1000);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION queue_timeout() RETURNS interval AS $$\n SELECT t2s(coalesce(\n get_setting('queue_timeout'),\n '1h'\n ))::interval;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION notice(VARIADIC text[]) RETURNS boolean AS $$\nDECLARE\ndebug boolean := current_setting('pgstac.debug', true);\nBEGIN\n IF debug THEN\n RAISE NOTICE 'NOTICE FROM FUNC: % >>>>> %', concat_ws(' | ', $1), clock_timestamp();\n RETURN TRUE;\n END IF;\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION empty_arr(ANYARRAY) RETURNS BOOLEAN AS $$\nSELECT CASE\n WHEN $1 IS NULL THEN TRUE\n WHEN cardinality($1)<1 THEN TRUE\nELSE FALSE\nEND;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_intersection(_a ANYARRAY, _b ANYARRAY) RETURNS ANYARRAY AS $$\n SELECT ARRAY ( SELECT unnest(_a) INTERSECT SELECT UNNEST(_b) );\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION array_map_ident(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_ident(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_map_literal(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_literal(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION array_reverse(anyarray) RETURNS anyarray AS $$\nSELECT ARRAY(\n SELECT $1[i]\n FROM generate_subscripts($1,1) AS s(i)\n ORDER BY i DESC\n);\n$$ LANGUAGE SQL STRICT IMMUTABLE;\n\nDROP TABLE IF EXISTS query_queue;\nCREATE TABLE query_queue (\n query text PRIMARY KEY,\n added timestamptz DEFAULT now()\n);\n\nDROP TABLE IF EXISTS query_queue_history;\nCREATE TABLE query_queue_history(\n query text,\n added timestamptz NOT NULL,\n finished timestamptz NOT NULL DEFAULT now(),\n error text\n);\n\nCREATE OR REPLACE PROCEDURE run_queued_queries() AS $$\nDECLARE\n qitem query_queue%ROWTYPE;\n timeout_ts timestamptz;\n error text;\n cnt int := 0;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n DELETE FROM query_queue WHERE query = (SELECT query FROM query_queue ORDER BY added DESC LIMIT 1 FOR UPDATE SKIP LOCKED) RETURNING * INTO qitem;\n IF NOT FOUND THEN\n EXIT;\n END IF;\n cnt := cnt + 1;\n BEGIN\n RAISE NOTICE 'RUNNING QUERY: %', qitem.query;\n EXECUTE qitem.query;\n EXCEPTION WHEN others THEN\n error := format('%s | %s', SQLERRM, SQLSTATE);\n END;\n INSERT INTO query_queue_history (query, added, finished, error)\n VALUES (qitem.query, qitem.added, clock_timestamp(), error);\n COMMIT;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION run_queued_queries_intransaction() RETURNS int AS $$\nDECLARE\n qitem query_queue%ROWTYPE;\n timeout_ts timestamptz;\n error text;\n cnt int := 0;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n DELETE FROM query_queue WHERE query = (SELECT query FROM query_queue ORDER BY added DESC LIMIT 1 FOR UPDATE SKIP LOCKED) RETURNING * INTO qitem;\n IF NOT FOUND THEN\n RETURN cnt;\n END IF;\n cnt := cnt + 1;\n BEGIN\n qitem.query := regexp_replace(qitem.query, 'CONCURRENTLY', '');\n RAISE NOTICE 'RUNNING QUERY: %', qitem.query;\n\n EXECUTE qitem.query;\n EXCEPTION WHEN others THEN\n error := format('%s | %s', SQLERRM, SQLSTATE);\n RAISE WARNING '%', error;\n END;\n INSERT INTO query_queue_history (query, added, finished, error)\n VALUES (qitem.query, qitem.added, clock_timestamp(), error);\n END LOOP;\n RETURN cnt;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION run_or_queue(query text) RETURNS VOID AS $$\nDECLARE\n use_queue text := COALESCE(get_setting('use_queue'), 'FALSE')::boolean;\nBEGIN\n IF get_setting_bool('debug') THEN\n RAISE NOTICE '%', query;\n END IF;\n IF use_queue THEN\n INSERT INTO query_queue (query) VALUES (query) ON CONFLICT DO NOTHING;\n ELSE\n EXECUTE query;\n END IF;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n\nDROP FUNCTION IF EXISTS check_pgstac_settings;\nCREATE OR REPLACE FUNCTION check_pgstac_settings(_sysmem text DEFAULT NULL) RETURNS VOID AS $$\nDECLARE\n settingval text;\n sysmem bigint := pg_size_bytes(_sysmem);\n effective_cache_size bigint := pg_size_bytes(current_setting('effective_cache_size', TRUE));\n shared_buffers bigint := pg_size_bytes(current_setting('shared_buffers', TRUE));\n work_mem bigint := pg_size_bytes(current_setting('work_mem', TRUE));\n max_connections int := current_setting('max_connections', TRUE);\n maintenance_work_mem bigint := pg_size_bytes(current_setting('maintenance_work_mem', TRUE));\n seq_page_cost float := current_setting('seq_page_cost', TRUE);\n random_page_cost float := current_setting('random_page_cost', TRUE);\n temp_buffers bigint := pg_size_bytes(current_setting('temp_buffers', TRUE));\n r record;\nBEGIN\n IF _sysmem IS NULL THEN\n RAISE NOTICE 'Call function with the size of your system memory `SELECT check_pgstac_settings(''4GB'')` to get pg system setting recommendations.';\n ELSE\n IF effective_cache_size < (sysmem * 0.5) THEN\n RAISE WARNING 'effective_cache_size of % is set low for a system with %. Recomended value between % and %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.5), pg_size_pretty(sysmem * 0.75);\n ELSIF effective_cache_size > (sysmem * 0.75) THEN\n RAISE WARNING 'effective_cache_size of % is set high for a system with %. Recomended value between % and %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.5), pg_size_pretty(sysmem * 0.75);\n ELSE\n RAISE NOTICE 'effective_cache_size of % is set appropriately for a system with %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem);\n END IF;\n\n IF shared_buffers < (sysmem * 0.2) THEN\n RAISE WARNING 'shared_buffers of % is set low for a system with %. Recomended value between % and %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.2), pg_size_pretty(sysmem * 0.3);\n ELSIF shared_buffers > (sysmem * 0.3) THEN\n RAISE WARNING 'shared_buffers of % is set high for a system with %. Recomended value between % and %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.2), pg_size_pretty(sysmem * 0.3);\n ELSE\n RAISE NOTICE 'shared_buffers of % is set appropriately for a system with %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem);\n END IF;\n shared_buffers = sysmem * 0.3;\n IF maintenance_work_mem < (sysmem * 0.2) THEN\n RAISE WARNING 'maintenance_work_mem of % is set low for shared_buffers of %. Recomended value between % and %', pg_size_pretty(maintenance_work_mem), pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers * 0.2), pg_size_pretty(shared_buffers * 0.3);\n ELSIF maintenance_work_mem > (shared_buffers * 0.3) THEN\n RAISE WARNING 'maintenance_work_mem of % is set high for shared_buffers of %. Recomended value between % and %', pg_size_pretty(maintenance_work_mem), pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers * 0.2), pg_size_pretty(shared_buffers * 0.3);\n ELSE\n RAISE NOTICE 'maintenance_work_mem of % is set appropriately for shared_buffers of %', pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers);\n END IF;\n\n IF work_mem * max_connections > shared_buffers THEN\n RAISE WARNING 'work_mem setting of % is set high for % max_connections please reduce work_mem to % or decrease max_connections to %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers/max_connections), floor(shared_buffers/work_mem);\n ELSIF work_mem * max_connections < (shared_buffers * 0.75) THEN\n RAISE WARNING 'work_mem setting of % is set low for % max_connections you may consider raising work_mem to % or increasing max_connections to %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers/max_connections), floor(shared_buffers/work_mem);\n ELSE\n RAISE NOTICE 'work_mem setting of % and max_connections of % are adequate for shared_buffers of %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers);\n END IF;\n\n IF random_page_cost / seq_page_cost != 1.1 THEN\n RAISE WARNING 'random_page_cost (%) /seq_page_cost (%) should be set to 1.1 for SSD. Change random_page_cost to %', random_page_cost, seq_page_cost, 1.1 * seq_page_cost;\n ELSE\n RAISE NOTICE 'random_page_cost and seq_page_cost set appropriately for SSD';\n END IF;\n\n IF temp_buffers < greatest(pg_size_bytes('128MB'),(maintenance_work_mem / 2)) THEN\n RAISE WARNING 'pgstac makes heavy use of temp tables, consider raising temp_buffers from % to %', pg_size_pretty(temp_buffers), greatest('128MB', pg_size_pretty((shared_buffers / 16)));\n END IF;\n END IF;\n\n RAISE NOTICE 'VALUES FOR PGSTAC VARIABLES';\n RAISE NOTICE 'These can be set either as GUC system variables or by setting in the pgstac_settings table.';\n\n FOR r IN SELECT name, get_setting(name) as setting, CASE WHEN current_setting(concat('pgstac.',name), TRUE) IS NOT NULL THEN concat('pgstac.',name, ' GUC') WHEN value IS NOT NULL THEN 'pgstac_settings table' ELSE 'Not Set' END as loc FROM pgstac_settings LOOP\n RAISE NOTICE '% is set to % from the %', r.name, r.setting, r.loc;\n END LOOP;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pg_cron';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider intalling pg_cron which can be used to automate tasks';\n ELSE\n RAISE NOTICE 'pg_cron % is installed', settingval;\n END IF;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pgstattuple';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider installing the pgstattuple extension which can be used to help maintain tables and indexes.';\n ELSE\n RAISE NOTICE 'pgstattuple % is installed', settingval;\n END IF;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pg_stat_statements';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider installing the pg_stat_statements extension which is very helpful for tracking the types of queries on the system';\n ELSE\n RAISE NOTICE 'pg_stat_statements % is installed', settingval;\n IF current_setting('pg_stat_statements.track_statements', TRUE) IS DISTINCT FROM 'all' THEN\n RAISE WARNING 'SET pg_stat_statements.track_statements TO ''all''; --In order to track statements within functions.';\n END IF;\n END IF;\n\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac, public SET CLIENT_MIN_MESSAGES TO NOTICE;\nCREATE OR REPLACE FUNCTION to_int(jsonb) RETURNS int AS $$\n SELECT floor(($1->>0)::float)::int;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION to_float(jsonb) RETURNS float AS $$\n SELECT ($1->>0)::float;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION to_tstz(jsonb) RETURNS timestamptz AS $$\n SELECT ($1->>0)::timestamptz;\n$$ LANGUAGE SQL IMMUTABLE STRICT SET TIME ZONE 'UTC' COST 5000 PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION to_text(jsonb) RETURNS text AS $$\n SELECT CASE WHEN jsonb_typeof($1) IN ('array','object') THEN $1::text ELSE $1->>0 END;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION to_text_array(jsonb) RETURNS text[] AS $$\n SELECT\n CASE jsonb_typeof($1)\n WHEN 'array' THEN ARRAY(SELECT jsonb_array_elements_text($1))\n ELSE ARRAY[$1->>0]\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION bbox_geom(_bbox jsonb) RETURNS geometry AS $$\nSELECT CASE jsonb_array_length(_bbox)\n WHEN 4 THEN\n ST_SetSRID(ST_MakeEnvelope(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float,\n (_bbox->>3)::float\n ),4326)\n WHEN 6 THEN\n ST_SetSRID(ST_3DMakeBox(\n ST_MakePoint(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float\n ),\n ST_MakePoint(\n (_bbox->>3)::float,\n (_bbox->>4)::float,\n (_bbox->>5)::float\n )\n ),4326)\n ELSE null END;\n;\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION geom_bbox(_geom geometry) RETURNS jsonb AS $$\n SELECT jsonb_build_array(\n st_xmin(_geom),\n st_ymin(_geom),\n st_xmax(_geom),\n st_ymax(_geom)\n );\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION flip_jsonb_array(j jsonb) RETURNS jsonb AS $$\n SELECT jsonb_agg(value) FROM (SELECT value FROM jsonb_array_elements(j) WITH ORDINALITY ORDER BY ordinality DESC) as t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION explode_dotpaths(j jsonb) RETURNS SETOF text[] AS $$\n SELECT string_to_array(p, '.') as e FROM jsonb_array_elements_text(j) p;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION explode_dotpaths_recurse(IN j jsonb) RETURNS SETOF text[] AS $$\n WITH RECURSIVE t AS (\n SELECT e FROM explode_dotpaths(j) e\n UNION ALL\n SELECT e[1:cardinality(e)-1]\n FROM t\n WHERE cardinality(e)>1\n ) SELECT e FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION jsonb_set_nested(j jsonb, path text[], val jsonb) RETURNS jsonb AS $$\nDECLARE\nBEGIN\n IF cardinality(path) > 1 THEN\n FOR i IN 1..(cardinality(path)-1) LOOP\n IF j #> path[:i] IS NULL THEN\n j := jsonb_set_lax(j, path[:i], '{}', TRUE);\n END IF;\n END LOOP;\n END IF;\n RETURN jsonb_set_lax(j, path, val, true);\n\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\n\n\nCREATE OR REPLACE FUNCTION jsonb_include(j jsonb, f jsonb) RETURNS jsonb AS $$\nDECLARE\n includes jsonb := f-> 'include';\n outj jsonb := '{}'::jsonb;\n path text[];\nBEGIN\n IF\n includes IS NULL\n OR jsonb_array_length(includes) = 0\n THEN\n RETURN j;\n ELSE\n includes := includes || '[\"id\",\"collection\"]'::jsonb;\n FOR path IN SELECT explode_dotpaths(includes) LOOP\n outj := jsonb_set_nested(outj, path, j #> path);\n END LOOP;\n END IF;\n RETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_exclude(j jsonb, f jsonb) RETURNS jsonb AS $$\nDECLARE\n excludes jsonb := f-> 'exclude';\n outj jsonb := j;\n path text[];\nBEGIN\n IF\n excludes IS NULL\n OR jsonb_array_length(excludes) = 0\n THEN\n RETURN j;\n ELSE\n FOR path IN SELECT explode_dotpaths(excludes) LOOP\n outj := outj #- path;\n END LOOP;\n END IF;\n RETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_fields(j jsonb, f jsonb DEFAULT '{\"fields\":[]}') RETURNS jsonb AS $$\n SELECT jsonb_exclude(jsonb_include(j, f), f);\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION merge_jsonb(_a jsonb, _b jsonb) RETURNS jsonb AS $$\n SELECT\n CASE\n WHEN _a = '\"𒍟※\"'::jsonb THEN NULL\n WHEN _a IS NULL OR jsonb_typeof(_a) = 'null' THEN _b\n WHEN jsonb_typeof(_a) = 'object' AND jsonb_typeof(_b) = 'object' THEN\n (\n SELECT\n jsonb_strip_nulls(\n jsonb_object_agg(\n key,\n merge_jsonb(a.value, b.value)\n )\n )\n FROM\n jsonb_each(coalesce(_a,'{}'::jsonb)) as a\n FULL JOIN\n jsonb_each(coalesce(_b,'{}'::jsonb)) as b\n USING (key)\n )\n WHEN\n jsonb_typeof(_a) = 'array'\n AND jsonb_typeof(_b) = 'array'\n AND jsonb_array_length(_a) = jsonb_array_length(_b)\n THEN\n (\n SELECT jsonb_agg(m) FROM\n ( SELECT\n merge_jsonb(\n jsonb_array_elements(_a),\n jsonb_array_elements(_b)\n ) as m\n ) as l\n )\n ELSE _a\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION strip_jsonb(_a jsonb, _b jsonb) RETURNS jsonb AS $$\n SELECT\n CASE\n\n WHEN (_a IS NULL OR jsonb_typeof(_a) = 'null') AND _b IS NOT NULL AND jsonb_typeof(_b) != 'null' THEN '\"𒍟※\"'::jsonb\n WHEN _b IS NULL OR jsonb_typeof(_a) = 'null' THEN _a\n WHEN _a = _b AND jsonb_typeof(_a) = 'object' THEN '{}'::jsonb\n WHEN _a = _b THEN NULL\n WHEN jsonb_typeof(_a) = 'object' AND jsonb_typeof(_b) = 'object' THEN\n (\n SELECT\n jsonb_strip_nulls(\n jsonb_object_agg(\n key,\n strip_jsonb(a.value, b.value)\n )\n )\n FROM\n jsonb_each(_a) as a\n FULL JOIN\n jsonb_each(_b) as b\n USING (key)\n )\n WHEN\n jsonb_typeof(_a) = 'array'\n AND jsonb_typeof(_b) = 'array'\n AND jsonb_array_length(_a) = jsonb_array_length(_b)\n THEN\n (\n SELECT jsonb_agg(m) FROM\n ( SELECT\n strip_jsonb(\n jsonb_array_elements(_a),\n jsonb_array_elements(_b)\n ) as m\n ) as l\n )\n ELSE _a\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION nullif_jsonbnullempty(j jsonb) RETURNS jsonb AS $$\n SELECT nullif(nullif(nullif(j,'null'::jsonb),'{}'::jsonb),'[]'::jsonb);\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION jsonb_array_unique(j jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(jsonb_agg(DISTINCT a)) v FROM jsonb_array_elements(j) a;\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_concat_ignorenull(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT coalesce(a,'[]'::jsonb) || coalesce(b,'[]'::jsonb);\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_least(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(least(nullif_jsonbnullempty(a), nullif_jsonbnullempty(b)));\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_greatest(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(greatest(a, b));\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION first_notnull_sfunc(anyelement, anyelement) RETURNS anyelement AS $$\n SELECT COALESCE($1,$2);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE AGGREGATE first_notnull(anyelement)(\n SFUNC = first_notnull_sfunc,\n STYPE = anyelement\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_array_unique_merge(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_concat_ignorenull,\n FINALFUNC = jsonb_array_unique\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_min(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_least\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_max(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_greatest\n);\n/* looks for a geometry in a stac item first from geometry and falling back to bbox */\nCREATE OR REPLACE FUNCTION stac_geom(value jsonb) RETURNS geometry AS $$\nSELECT\n CASE\n WHEN value ? 'intersects' THEN\n ST_GeomFromGeoJSON(value->>'intersects')\n WHEN value ? 'geometry' THEN\n ST_GeomFromGeoJSON(value->>'geometry')\n WHEN value ? 'bbox' THEN\n pgstac.bbox_geom(value->'bbox')\n ELSE NULL\n END as geometry\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION stac_daterange(\n value jsonb\n) RETURNS tstzrange AS $$\nDECLARE\n props jsonb := value;\n dt timestamptz;\n edt timestamptz;\nBEGIN\n IF props ? 'properties' THEN\n props := props->'properties';\n END IF;\n IF\n props ? 'start_datetime'\n AND props->>'start_datetime' IS NOT NULL\n AND props ? 'end_datetime'\n AND props->>'end_datetime' IS NOT NULL\n THEN\n dt := props->>'start_datetime';\n edt := props->>'end_datetime';\n IF dt > edt THEN\n RAISE EXCEPTION 'start_datetime must be < end_datetime';\n END IF;\n ELSE\n dt := props->>'datetime';\n edt := props->>'datetime';\n END IF;\n IF dt is NULL OR edt IS NULL THEN\n RAISE NOTICE 'DT: %, EDT: %', dt, edt;\n RAISE EXCEPTION 'Either datetime (%) or both start_datetime (%) and end_datetime (%) must be set.', props->>'datetime',props->>'start_datetime',props->>'end_datetime';\n END IF;\n RETURN tstzrange(dt, edt, '[]');\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_datetime(value jsonb) RETURNS timestamptz AS $$\n SELECT lower(stac_daterange(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_end_datetime(value jsonb) RETURNS timestamptz AS $$\n SELECT upper(stac_daterange(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE TABLE IF NOT EXISTS stac_extensions(\n url text PRIMARY KEY,\n content jsonb\n);\nCREATE OR REPLACE FUNCTION collection_base_item(content jsonb) RETURNS jsonb AS $$\n SELECT jsonb_build_object(\n 'type', 'Feature',\n 'stac_version', content->'stac_version',\n 'assets', content->'item_assets',\n 'collection', content->'id'\n );\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE TABLE IF NOT EXISTS collections (\n key bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,\n id text GENERATED ALWAYS AS (content->>'id') STORED UNIQUE NOT NULL,\n content JSONB NOT NULL,\n base_item jsonb GENERATED ALWAYS AS (pgstac.collection_base_item(content)) STORED,\n geometry geometry GENERATED ALWAYS AS (pgstac.collection_geom(content)) STORED,\n datetime timestamptz GENERATED ALWAYS AS (pgstac.collection_datetime(content)) STORED,\n end_datetime timestamptz GENERATED ALWAYS AS (pgstac.collection_enddatetime(content)) STORED,\n private jsonb,\n partition_trunc text CHECK (partition_trunc IN ('year', 'month'))\n);\n\nCREATE OR REPLACE FUNCTION collection_base_item(cid text) RETURNS jsonb AS $$\n SELECT pgstac.collection_base_item(content) FROM pgstac.collections WHERE id = cid LIMIT 1;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION create_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION update_collection(data jsonb) RETURNS VOID AS $$\nDECLARE\n out collections%ROWTYPE;\nBEGIN\n UPDATE collections SET content=data WHERE id = data->>'id' RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ON CONFLICT (id) DO\n UPDATE\n SET content=EXCLUDED.content\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION delete_collection(_id text) RETURNS VOID AS $$\nDECLARE\n out collections%ROWTYPE;\nBEGIN\n DELETE FROM collections WHERE id = _id RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION get_collection(id text) RETURNS jsonb AS $$\n SELECT content FROM collections\n WHERE id=$1\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION all_collections() RETURNS jsonb AS $$\n SELECT coalesce(jsonb_agg(content), '[]'::jsonb) FROM collections;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION collection_delete_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n collection_base_partition text := concat('_items_', OLD.key);\nBEGIN\n EXECUTE format($q$\n DELETE FROM partition_stats WHERE partition IN (\n SELECT partition FROM partition_sys_meta\n WHERE collection=%L\n );\n DROP TABLE IF EXISTS %I CASCADE;\n $q$,\n OLD.id,\n collection_base_partition\n );\n RETURN OLD;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER collection_delete_trigger BEFORE DELETE ON collections\nFOR EACH ROW EXECUTE FUNCTION collection_delete_trigger_func();\nCREATE OR REPLACE FUNCTION queryable_signature(n text, c text[]) RETURNS text AS $$\n SELECT concat(n, c);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TABLE queryables (\n id bigint GENERATED ALWAYS AS identity PRIMARY KEY,\n name text NOT NULL,\n collection_ids text[], -- used to determine what partitions to create indexes on\n definition jsonb,\n property_path text,\n property_wrapper text,\n property_index_type text\n);\nCREATE INDEX queryables_name_idx ON queryables (name);\nCREATE INDEX queryables_collection_idx ON queryables USING GIN (collection_ids);\nCREATE INDEX queryables_property_wrapper_idx ON queryables (property_wrapper);\n\nCREATE OR REPLACE FUNCTION pgstac.queryables_constraint_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n allcollections text[];\nBEGIN\n RAISE NOTICE 'Making sure that name/collection is unique for queryables %', NEW;\n IF NEW.collection_ids IS NOT NULL THEN\n IF EXISTS (\n SELECT 1\n FROM unnest(NEW.collection_ids) c\n LEFT JOIN\n collections\n ON (collections.id = c)\n WHERE collections.id IS NULL\n ) THEN\n RAISE foreign_key_violation USING MESSAGE = format(\n 'One or more collections in %s do not exist.', NEW.collection_ids\n );\n RETURN NULL;\n END IF;\n END IF;\n IF TG_OP = 'INSERT' THEN\n IF EXISTS (\n SELECT 1 FROM queryables q\n WHERE\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n )\n ) THEN\n RAISE unique_violation USING MESSAGE = format(\n 'There is already a queryable for %s for a collection in %s: %s',\n NEW.name,\n NEW.collection_ids,\n\t\t\t\t(SELECT json_agg(row_to_json(q)) FROM queryables q WHERE\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n ))\n );\n RETURN NULL;\n END IF;\n END IF;\n IF TG_OP = 'UPDATE' THEN\n IF EXISTS (\n SELECT 1 FROM queryables q\n WHERE\n q.id != NEW.id\n AND\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n )\n ) THEN\n RAISE unique_violation\n USING MESSAGE = format(\n 'There is already a queryable for %s for a collection in %s',\n NEW.name,\n NEW.collection_ids\n );\n RETURN NULL;\n END IF;\n END IF;\n\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER queryables_constraint_insert_trigger\nBEFORE INSERT ON queryables\nFOR EACH ROW EXECUTE PROCEDURE queryables_constraint_triggerfunc();\n\nCREATE TRIGGER queryables_constraint_update_trigger\nBEFORE UPDATE ON queryables\nFOR EACH ROW\nWHEN (NEW.name = OLD.name AND NEW.collection_ids IS DISTINCT FROM OLD.collection_ids)\nEXECUTE PROCEDURE queryables_constraint_triggerfunc();\n\n\nCREATE OR REPLACE FUNCTION array_to_path(arr text[]) RETURNS text AS $$\n SELECT string_agg(\n quote_literal(v),\n '->'\n ) FROM unnest(arr) v;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\n\n\nCREATE OR REPLACE FUNCTION queryable(\n IN dotpath text,\n OUT path text,\n OUT expression text,\n OUT wrapper text,\n OUT nulled_wrapper text\n) AS $$\nDECLARE\n q RECORD;\n path_elements text[];\nBEGIN\n dotpath := replace(dotpath, 'properties.', '');\n IF dotpath = 'start_datetime' THEN\n dotpath := 'datetime';\n END IF;\n IF dotpath IN ('id', 'geometry', 'datetime', 'end_datetime', 'collection') THEN\n path := dotpath;\n expression := dotpath;\n wrapper := NULL;\n RETURN;\n END IF;\n\n SELECT * INTO q FROM queryables\n WHERE\n name=dotpath\n OR name = 'properties.' || dotpath\n OR name = replace(dotpath, 'properties.', '')\n ;\n IF q.property_wrapper IS NULL THEN\n IF q.definition->>'type' = 'number' THEN\n wrapper := 'to_float';\n nulled_wrapper := wrapper;\n ELSIF q.definition->>'format' = 'date-time' THEN\n wrapper := 'to_tstz';\n nulled_wrapper := wrapper;\n ELSE\n nulled_wrapper := NULL;\n wrapper := 'to_text';\n END IF;\n ELSE\n wrapper := q.property_wrapper;\n nulled_wrapper := wrapper;\n END IF;\n IF q.property_path IS NOT NULL THEN\n path := q.property_path;\n ELSE\n path_elements := string_to_array(dotpath, '.');\n IF path_elements[1] IN ('links', 'assets', 'stac_version', 'stac_extensions') THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSIF path_elements[1] = 'properties' THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSE\n path := format($F$content->'properties'->%s$F$, array_to_path(path_elements));\n END IF;\n END IF;\n expression := format('%I(%s)', wrapper, path);\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\nCREATE OR REPLACE FUNCTION unnest_collection(collection_ids text[] DEFAULT NULL) RETURNS SETOF text AS $$\n DECLARE\n BEGIN\n IF collection_ids IS NULL THEN\n RETURN QUERY SELECT id FROM collections;\n END IF;\n RETURN QUERY SELECT unnest(collection_ids);\n END;\n$$ LANGUAGE PLPGSQL STABLE;\n\nCREATE OR REPLACE FUNCTION normalize_indexdef(def text) RETURNS text AS $$\nDECLARE\nBEGIN\n def := btrim(def, ' \\n\\t');\n\tdef := regexp_replace(def, '^CREATE (UNIQUE )?INDEX ([^ ]* )?ON (ONLY )?([^ ]* )?', '', 'i');\n RETURN def;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION indexdef(q queryables) RETURNS text AS $$\n DECLARE\n out text;\n BEGIN\n IF q.name = 'id' THEN\n out := 'CREATE UNIQUE INDEX ON %I USING btree (id)';\n ELSIF q.name = 'datetime' THEN\n out := 'CREATE INDEX ON %I USING btree (datetime DESC, end_datetime)';\n ELSIF q.name = 'geometry' THEN\n out := 'CREATE INDEX ON %I USING gist (geometry)';\n ELSE\n out := format($q$CREATE INDEX ON %%I USING %s (%s(((content -> 'properties'::text) -> %L::text)))$q$,\n lower(COALESCE(q.property_index_type, 'BTREE')),\n lower(COALESCE(q.property_wrapper, 'to_text')),\n q.name\n );\n END IF;\n RETURN btrim(out, ' \\n\\t');\n END;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nDROP VIEW IF EXISTS pgstac_indexes;\nCREATE VIEW pgstac_indexes AS\nSELECT\n i.schemaname,\n i.tablename,\n i.indexname,\n regexp_replace(btrim(replace(replace(indexdef, i.indexname, ''),'pgstac.',''),' \\t\\n'), '[ ]+', ' ', 'g') as idx,\n COALESCE(\n (regexp_match(indexdef, '\\(([a-zA-Z]+)\\)'))[1],\n (regexp_match(indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_-]+)''::text'))[1],\n CASE WHEN indexdef ~* '\\(datetime desc, end_datetime\\)' THEN 'datetime' ELSE NULL END\n ) AS field,\n pg_table_size(i.indexname::text) as index_size,\n pg_size_pretty(pg_table_size(i.indexname::text)) as index_size_pretty\nFROM\n pg_indexes i\nWHERE i.schemaname='pgstac' and i.tablename ~ '_items_' AND indexdef !~* ' only ';\n\nDROP VIEW IF EXISTS pgstac_index_stats;\nCREATE VIEW pgstac_indexes_stats AS\nSELECT\n i.schemaname,\n i.tablename,\n i.indexname,\n indexdef,\n COALESCE(\n (regexp_match(indexdef, '\\(([a-zA-Z]+)\\)'))[1],\n (regexp_match(indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_]+)''::text'))[1],\n CASE WHEN indexdef ~* '\\(datetime desc, end_datetime\\)' THEN 'datetime_end_datetime' ELSE NULL END\n ) AS field,\n pg_table_size(i.indexname::text) as index_size,\n pg_size_pretty(pg_table_size(i.indexname::text)) as index_size_pretty,\n n_distinct,\n most_common_vals::text::text[],\n most_common_freqs::text::text[],\n histogram_bounds::text::text[],\n correlation\nFROM\n pg_indexes i\n LEFT JOIN pg_stats s ON (s.tablename = i.indexname)\nWHERE i.schemaname='pgstac' and i.tablename ~ '_items_';\n\nCREATE OR REPLACE FUNCTION queryable_indexes(\n IN treeroot text DEFAULT 'items',\n IN changes boolean DEFAULT FALSE,\n OUT collection text,\n OUT partition text,\n OUT field text,\n OUT indexname text,\n OUT existing_idx text,\n OUT queryable_idx text\n) RETURNS SETOF RECORD AS $$\nWITH p AS (\n SELECT\n relid::text as partition,\n replace(replace(\n CASE\n WHEN parentrelid::regclass::text='items' THEN pg_get_expr(c.relpartbound, c.oid)\n ELSE pg_get_expr(parent.relpartbound, parent.oid)\n END,\n 'FOR VALUES IN (''',''), ''')',\n ''\n ) AS collection\n FROM pg_partition_tree(treeroot)\n JOIN pg_class c ON (relid::regclass = c.oid)\n JOIN pg_class parent ON (parentrelid::regclass = parent.oid AND isleaf)\n ), i AS (\n SELECT\n partition,\n indexname,\n regexp_replace(btrim(replace(replace(indexdef, indexname, ''),'pgstac.',''),' \\t\\n'), '[ ]+', ' ', 'g') as iidx,\n COALESCE(\n (regexp_match(indexdef, '\\(([a-zA-Z]+)\\)'))[1],\n (regexp_match(indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_-]+)''::text'))[1],\n CASE WHEN indexdef ~* '\\(datetime desc, end_datetime\\)' THEN 'datetime' ELSE NULL END\n ) AS field\n FROM\n pg_indexes\n JOIN p ON (tablename=partition)\n ), q AS (\n SELECT\n name AS field,\n collection,\n partition,\n format(indexdef(queryables), partition) as qidx\n FROM queryables, unnest_collection(queryables.collection_ids) collection\n JOIN p USING (collection)\n WHERE property_index_type IS NOT NULL OR name IN ('datetime','geometry','id')\n )\n SELECT\n collection,\n partition,\n field,\n indexname,\n iidx as existing_idx,\n qidx as queryable_idx\n FROM i FULL JOIN q USING (field, partition)\n WHERE CASE WHEN changes THEN lower(iidx) IS DISTINCT FROM lower(qidx) ELSE TRUE END;\n;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION maintain_index(\n indexname text,\n queryable_idx text,\n dropindexes boolean DEFAULT FALSE,\n rebuildindexes boolean DEFAULT FALSE,\n idxconcurrently boolean DEFAULT FALSE\n) RETURNS VOID AS $$\nDECLARE\nBEGIN\n IF indexname IS NOT NULL THEN\n IF dropindexes OR queryable_idx IS NOT NULL THEN\n EXECUTE format('DROP INDEX IF EXISTS %I;', indexname);\n ELSIF rebuildindexes THEN\n IF idxconcurrently THEN\n EXECUTE format('REINDEX INDEX CONCURRENTLY %I;', indexname);\n ELSE\n EXECUTE format('REINDEX INDEX CONCURRENTLY %I;', indexname);\n END IF;\n END IF;\n END IF;\n IF queryable_idx IS NOT NULL THEN\n IF idxconcurrently THEN\n EXECUTE replace(queryable_idx, 'INDEX', 'INDEX CONCURRENTLY');\n ELSE EXECUTE queryable_idx;\n END IF;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\n\nset check_function_bodies to off;\nCREATE OR REPLACE FUNCTION maintain_partition_queries(\n part text DEFAULT 'items',\n dropindexes boolean DEFAULT FALSE,\n rebuildindexes boolean DEFAULT FALSE,\n idxconcurrently boolean DEFAULT FALSE\n) RETURNS SETOF text AS $$\nDECLARE\n rec record;\n q text;\nBEGIN\n FOR rec IN (\n SELECT * FROM queryable_indexes(part,true)\n ) LOOP\n q := format(\n 'SELECT maintain_index(\n %L,%L,%L,%L,%L\n );',\n rec.indexname,\n rec.queryable_idx,\n dropindexes,\n rebuildindexes,\n idxconcurrently\n );\n RAISE NOTICE 'Q: %', q;\n RETURN NEXT q;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION maintain_partitions(\n part text DEFAULT 'items',\n dropindexes boolean DEFAULT FALSE,\n rebuildindexes boolean DEFAULT FALSE\n) RETURNS VOID AS $$\n WITH t AS (\n SELECT run_or_queue(q) FROM maintain_partition_queries(part, dropindexes, rebuildindexes) q\n ) SELECT count(*) FROM t;\n$$ LANGUAGE SQL;\n\n\nCREATE OR REPLACE FUNCTION queryables_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\nBEGIN\n PERFORM maintain_partitions();\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER queryables_trigger AFTER INSERT OR UPDATE ON queryables\nFOR EACH STATEMENT EXECUTE PROCEDURE queryables_trigger_func();\n\n\nCREATE OR REPLACE FUNCTION get_queryables(_collection_ids text[] DEFAULT NULL) RETURNS jsonb AS $$\nDECLARE\nBEGIN\n -- Build up queryables if the input contains valid collection ids or is empty\n IF EXISTS (\n SELECT 1 FROM collections\n WHERE\n _collection_ids IS NULL\n OR cardinality(_collection_ids) = 0\n OR id = ANY(_collection_ids)\n )\n THEN\n RETURN (\n WITH base AS (\n SELECT\n unnest(collection_ids) as collection_id,\n name,\n coalesce(definition, '{\"type\":\"string\"}'::jsonb) as definition\n FROM queryables\n WHERE\n _collection_ids IS NULL OR\n _collection_ids = '{}'::text[] OR\n _collection_ids && collection_ids\n UNION ALL\n SELECT null, name, coalesce(definition, '{\"type\":\"string\"}'::jsonb) as definition\n FROM queryables WHERE collection_ids IS NULL OR collection_ids = '{}'::text[]\n ), g AS (\n SELECT\n name,\n first_notnull(definition) as definition,\n jsonb_array_unique_merge(definition->'enum') as enum,\n jsonb_min(definition->'minimum') as minimum,\n jsonb_min(definition->'maxiumn') as maximum\n FROM base\n GROUP BY 1\n )\n SELECT\n jsonb_build_object(\n '$schema', 'http://json-schema.org/draft-07/schema#',\n '$id', '',\n 'type', 'object',\n 'title', 'STAC Queryables.',\n 'properties', jsonb_object_agg(\n name,\n definition\n ||\n jsonb_strip_nulls(jsonb_build_object(\n 'enum', enum,\n 'minimum', minimum,\n 'maximum', maximum\n ))\n ),\n 'additionalProperties', pgstac.additional_properties()\n )\n FROM g\n );\n ELSE\n RETURN NULL;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\nCREATE OR REPLACE FUNCTION get_queryables(_collection text DEFAULT NULL) RETURNS jsonb AS $$\n SELECT\n CASE\n WHEN _collection IS NULL THEN get_queryables(NULL::text[])\n ELSE get_queryables(ARRAY[_collection])\n END\n ;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_queryables() RETURNS jsonb AS $$\n SELECT get_queryables(NULL::text[]);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION schema_qualify_refs(url text, j jsonb) returns jsonb as $$\n SELECT regexp_replace(j::text, '\"\\$ref\": \"#', concat('\"$ref\": \"', url, '#'), 'g')::jsonb;\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE VIEW stac_extension_queryables AS\nSELECT DISTINCT key as name, schema_qualify_refs(e.url, j.value) as definition FROM stac_extensions e, jsonb_each(e.content->'definitions'->'fields'->'properties') j;\n\n\nCREATE OR REPLACE FUNCTION missing_queryables(_collection text, _tablesample float DEFAULT 5, minrows float DEFAULT 10) RETURNS TABLE(collection text, name text, definition jsonb, property_wrapper text) AS $$\nDECLARE\n q text;\n _partition text;\n explain_json json;\n psize float;\n estrows float;\nBEGIN\n SELECT format('_items_%s', key) INTO _partition FROM collections WHERE id=_collection;\n\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM %I;', _partition)\n INTO explain_json;\n psize := explain_json->0->'Plan'->'Plan Rows';\n estrows := _tablesample * .01 * psize;\n IF estrows < minrows THEN\n _tablesample := least(100,greatest(_tablesample, (estrows / psize) / 100));\n RAISE NOTICE '%', (psize / estrows) / 100;\n END IF;\n RAISE NOTICE 'Using tablesample % to find missing queryables from % % that has ~% rows estrows: %', _tablesample, _collection, _partition, psize, estrows;\n\n q := format(\n $q$\n WITH q AS (\n SELECT * FROM queryables\n WHERE\n collection_ids IS NULL\n OR %L = ANY(collection_ids)\n ), t AS (\n SELECT\n content->'properties' AS properties\n FROM\n %I\n TABLESAMPLE SYSTEM(%L)\n ), p AS (\n SELECT DISTINCT ON (key)\n key,\n value,\n s.definition\n FROM t\n JOIN LATERAL jsonb_each(properties) ON TRUE\n LEFT JOIN q ON (q.name=key)\n LEFT JOIN stac_extension_queryables s ON (s.name=key)\n WHERE q.definition IS NULL\n )\n SELECT\n %L,\n key,\n COALESCE(definition, jsonb_build_object('type',jsonb_typeof(value))) as definition,\n CASE\n WHEN definition->>'type' = 'integer' THEN 'to_int'\n WHEN COALESCE(definition->>'type', jsonb_typeof(value)) = 'number' THEN 'to_float'\n WHEN COALESCE(definition->>'type', jsonb_typeof(value)) = 'array' THEN 'to_text_array'\n ELSE 'to_text'\n END\n FROM p;\n $q$,\n _collection,\n _partition,\n _tablesample,\n _collection\n );\n RETURN QUERY EXECUTE q;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION missing_queryables(_tablesample float DEFAULT 5) RETURNS TABLE(collection_ids text[], name text, definition jsonb, property_wrapper text) AS $$\n SELECT\n array_agg(collection),\n name,\n definition,\n property_wrapper\n FROM\n collections\n JOIN LATERAL\n missing_queryables(id, _tablesample) c\n ON TRUE\n GROUP BY\n 2,3,4\n ORDER BY 2,1\n ;\n$$ LANGUAGE SQL;\nCREATE OR REPLACE FUNCTION parse_dtrange(\n _indate jsonb,\n relative_base timestamptz DEFAULT date_trunc('hour', CURRENT_TIMESTAMP)\n) RETURNS tstzrange AS $$\nDECLARE\n timestrs text[];\n s timestamptz;\n e timestamptz;\nBEGIN\n timestrs :=\n CASE\n WHEN _indate ? 'timestamp' THEN\n ARRAY[_indate->>'timestamp']\n WHEN _indate ? 'interval' THEN\n to_text_array(_indate->'interval')\n WHEN jsonb_typeof(_indate) = 'array' THEN\n to_text_array(_indate)\n ELSE\n regexp_split_to_array(\n _indate->>0,\n '/'\n )\n END;\n RAISE NOTICE 'TIMESTRS %', timestrs;\n IF cardinality(timestrs) = 1 THEN\n IF timestrs[1] ILIKE 'P%' THEN\n RETURN tstzrange(relative_base - upper(timestrs[1])::interval, relative_base, '[)');\n END IF;\n s := timestrs[1]::timestamptz;\n RETURN tstzrange(s, s, '[]');\n END IF;\n\n IF cardinality(timestrs) != 2 THEN\n RAISE EXCEPTION 'Timestamp cannot have more than 2 values';\n END IF;\n\n IF timestrs[1] = '..' OR timestrs[1] = '' THEN\n s := '-infinity'::timestamptz;\n e := timestrs[2]::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] = '..' OR timestrs[2] = '' THEN\n s := timestrs[1]::timestamptz;\n e := 'infinity'::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[1] ILIKE 'P%' AND timestrs[2] NOT ILIKE 'P%' THEN\n e := timestrs[2]::timestamptz;\n s := e - upper(timestrs[1])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] ILIKE 'P%' AND timestrs[1] NOT ILIKE 'P%' THEN\n s := timestrs[1]::timestamptz;\n e := s + upper(timestrs[2])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n s := timestrs[1]::timestamptz;\n e := timestrs[2]::timestamptz;\n\n RETURN tstzrange(s,e,'[)');\n\n RETURN NULL;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT PARALLEL SAFE SET TIME ZONE 'UTC';\n\nCREATE OR REPLACE FUNCTION parse_dtrange(\n _indate text,\n relative_base timestamptz DEFAULT CURRENT_TIMESTAMP\n) RETURNS tstzrange AS $$\n SELECT parse_dtrange(to_jsonb(_indate), relative_base);\n$$ LANGUAGE SQL STABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION temporal_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\n ll text := 'datetime';\n lh text := 'end_datetime';\n rrange tstzrange;\n rl text;\n rh text;\n outq text;\nBEGIN\n rrange := parse_dtrange(args->1);\n RAISE NOTICE 'Constructing temporal query OP: %, ARGS: %, RRANGE: %', op, args, rrange;\n op := lower(op);\n rl := format('%L::timestamptz', lower(rrange));\n rh := format('%L::timestamptz', upper(rrange));\n outq := CASE op\n WHEN 't_before' THEN 'lh < rl'\n WHEN 't_after' THEN 'll > rh'\n WHEN 't_meets' THEN 'lh = rl'\n WHEN 't_metby' THEN 'll = rh'\n WHEN 't_overlaps' THEN 'll < rl AND rl < lh < rh'\n WHEN 't_overlappedby' THEN 'rl < ll < rh AND lh > rh'\n WHEN 't_starts' THEN 'll = rl AND lh < rh'\n WHEN 't_startedby' THEN 'll = rl AND lh > rh'\n WHEN 't_during' THEN 'll > rl AND lh < rh'\n WHEN 't_contains' THEN 'll < rl AND lh > rh'\n WHEN 't_finishes' THEN 'll > rl AND lh = rh'\n WHEN 't_finishedby' THEN 'll < rl AND lh = rh'\n WHEN 't_equals' THEN 'll = rl AND lh = rh'\n WHEN 't_disjoint' THEN 'NOT (ll <= rh AND lh >= rl)'\n WHEN 't_intersects' THEN 'll <= rh AND lh >= rl'\n WHEN 'anyinteracts' THEN 'll <= rh AND lh >= rl'\n END;\n outq := regexp_replace(outq, '\\mll\\M', ll);\n outq := regexp_replace(outq, '\\mlh\\M', lh);\n outq := regexp_replace(outq, '\\mrl\\M', rl);\n outq := regexp_replace(outq, '\\mrh\\M', rh);\n outq := format('(%s)', outq);\n RETURN outq;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\n\n\nCREATE OR REPLACE FUNCTION spatial_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\n geom text;\n j jsonb := args->1;\nBEGIN\n op := lower(op);\n RAISE NOTICE 'Constructing spatial query OP: %, ARGS: %', op, args;\n IF op NOT IN ('s_equals','s_disjoint','s_touches','s_within','s_overlaps','s_crosses','s_intersects','intersects','s_contains') THEN\n RAISE EXCEPTION 'Spatial Operator % Not Supported', op;\n END IF;\n op := regexp_replace(op, '^s_', 'st_');\n IF op = 'intersects' THEN\n op := 'st_intersects';\n END IF;\n -- Convert geometry to WKB string\n IF j ? 'type' AND j ? 'coordinates' THEN\n geom := st_geomfromgeojson(j)::text;\n ELSIF jsonb_typeof(j) = 'array' THEN\n geom := bbox_geom(j)::text;\n END IF;\n\n RETURN format('%s(geometry, %L::geometry)', op, geom);\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION query_to_cql2(q jsonb) RETURNS jsonb AS $$\n-- Translates anything passed in through the deprecated \"query\" into equivalent CQL2\nWITH t AS (\n SELECT key as property, value as ops\n FROM jsonb_each(q)\n), t2 AS (\n SELECT property, (jsonb_each(ops)).*\n FROM t WHERE jsonb_typeof(ops) = 'object'\n UNION ALL\n SELECT property, 'eq', ops\n FROM t WHERE jsonb_typeof(ops) != 'object'\n)\nSELECT\n jsonb_strip_nulls(jsonb_build_object(\n 'op', 'and',\n 'args', jsonb_agg(\n jsonb_build_object(\n 'op', key,\n 'args', jsonb_build_array(\n jsonb_build_object('property',property),\n value\n )\n )\n )\n )\n) as qcql FROM t2\n;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\nCREATE OR REPLACE FUNCTION cql1_to_cql2(j jsonb) RETURNS jsonb AS $$\nDECLARE\n args jsonb;\n ret jsonb;\nBEGIN\n RAISE NOTICE 'CQL1_TO_CQL2: %', j;\n IF j ? 'filter' THEN\n RETURN cql1_to_cql2(j->'filter');\n END IF;\n IF j ? 'property' THEN\n RETURN j;\n END IF;\n IF jsonb_typeof(j) = 'array' THEN\n SELECT jsonb_agg(cql1_to_cql2(el)) INTO args FROM jsonb_array_elements(j) el;\n RETURN args;\n END IF;\n IF jsonb_typeof(j) = 'number' THEN\n RETURN j;\n END IF;\n IF jsonb_typeof(j) = 'string' THEN\n RETURN j;\n END IF;\n\n IF jsonb_typeof(j) = 'object' THEN\n SELECT jsonb_build_object(\n 'op', key,\n 'args', cql1_to_cql2(value)\n ) INTO ret\n FROM jsonb_each(j)\n WHERE j IS NOT NULL;\n RETURN ret;\n END IF;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE STRICT;\n\nCREATE TABLE cql2_ops (\n op text PRIMARY KEY,\n template text,\n types text[]\n);\nINSERT INTO cql2_ops (op, template, types) VALUES\n ('eq', '%s = %s', NULL),\n ('neq', '%s != %s', NULL),\n ('ne', '%s != %s', NULL),\n ('!=', '%s != %s', NULL),\n ('<>', '%s != %s', NULL),\n ('lt', '%s < %s', NULL),\n ('lte', '%s <= %s', NULL),\n ('gt', '%s > %s', NULL),\n ('gte', '%s >= %s', NULL),\n ('le', '%s <= %s', NULL),\n ('ge', '%s >= %s', NULL),\n ('=', '%s = %s', NULL),\n ('<', '%s < %s', NULL),\n ('<=', '%s <= %s', NULL),\n ('>', '%s > %s', NULL),\n ('>=', '%s >= %s', NULL),\n ('like', '%s LIKE %s', NULL),\n ('ilike', '%s ILIKE %s', NULL),\n ('+', '%s + %s', NULL),\n ('-', '%s - %s', NULL),\n ('*', '%s * %s', NULL),\n ('/', '%s / %s', NULL),\n ('not', 'NOT (%s)', NULL),\n ('between', '%s BETWEEN %s AND %s', NULL),\n ('isnull', '%s IS NULL', NULL),\n ('upper', 'upper(%s)', NULL),\n ('lower', 'lower(%s)', NULL),\n ('casei', 'upper(%s)', NULL),\n ('accenti', 'unaccent(%s)', NULL)\nON CONFLICT (op) DO UPDATE\n SET\n template = EXCLUDED.template\n;\n\n\nCREATE OR REPLACE FUNCTION cql2_query(j jsonb, wrapper text DEFAULT NULL) RETURNS text AS $$\n#variable_conflict use_variable\nDECLARE\n args jsonb := j->'args';\n arg jsonb;\n op text := lower(j->>'op');\n cql2op RECORD;\n literal text;\n _wrapper text;\n leftarg text;\n rightarg text;\n prop text;\n extra_props bool := pgstac.additional_properties();\nBEGIN\n IF j IS NULL OR (op IS NOT NULL AND args IS NULL) THEN\n RETURN NULL;\n END IF;\n RAISE NOTICE 'CQL2_QUERY: %', j;\n\n -- check if all properties are represented in the queryables\n IF NOT extra_props THEN\n FOR prop IN\n SELECT DISTINCT p->>0\n FROM jsonb_path_query(j, 'strict $.**.property') p\n WHERE p->>0 NOT IN ('id', 'datetime', 'geometry', 'end_datetime', 'collection')\n LOOP\n IF (queryable(prop)).nulled_wrapper IS NULL THEN\n RAISE EXCEPTION 'Term % is not found in queryables.', prop;\n END IF;\n END LOOP;\n END IF;\n\n IF j ? 'filter' THEN\n RETURN cql2_query(j->'filter');\n END IF;\n\n IF j ? 'upper' THEN\n RETURN cql2_query(jsonb_build_object('op', 'upper', 'args', j->'upper'));\n END IF;\n\n IF j ? 'lower' THEN\n RETURN cql2_query(jsonb_build_object('op', 'lower', 'args', j->'lower'));\n END IF;\n\n -- Temporal Query\n IF op ilike 't_%' or op = 'anyinteracts' THEN\n RETURN temporal_op_query(op, args);\n END IF;\n\n -- If property is a timestamp convert it to text to use with\n -- general operators\n IF j ? 'timestamp' THEN\n RETURN format('%L::timestamptz', to_tstz(j->'timestamp'));\n END IF;\n IF j ? 'interval' THEN\n RAISE EXCEPTION 'Please use temporal operators when using intervals.';\n RETURN NONE;\n END IF;\n\n -- Spatial Query\n IF op ilike 's_%' or op = 'intersects' THEN\n RETURN spatial_op_query(op, args);\n END IF;\n\n IF op IN ('a_equals','a_contains','a_contained_by','a_overlaps') THEN\n IF args->0 ? 'property' THEN\n leftarg := format('to_text_array(%s)', (queryable(args->0->>'property')).path);\n END IF;\n IF args->1 ? 'property' THEN\n rightarg := format('to_text_array(%s)', (queryable(args->1->>'property')).path);\n END IF;\n RETURN FORMAT(\n '%s %s %s',\n COALESCE(leftarg, quote_literal(to_text_array(args->0))),\n CASE op\n WHEN 'a_equals' THEN '='\n WHEN 'a_contains' THEN '@>'\n WHEN 'a_contained_by' THEN '<@'\n WHEN 'a_overlaps' THEN '&&'\n END,\n COALESCE(rightarg, quote_literal(to_text_array(args->1)))\n );\n END IF;\n\n IF op = 'in' THEN\n RAISE NOTICE 'IN : % % %', args, jsonb_build_array(args->0), args->1;\n args := jsonb_build_array(args->0) || (args->1);\n RAISE NOTICE 'IN2 : %', args;\n END IF;\n\n\n\n IF op = 'between' THEN\n args = jsonb_build_array(\n args->0,\n args->1,\n args->2\n );\n END IF;\n\n -- Make sure that args is an array and run cql2_query on\n -- each element of the array\n RAISE NOTICE 'ARGS PRE: %', args;\n IF j ? 'args' THEN\n IF jsonb_typeof(args) != 'array' THEN\n args := jsonb_build_array(args);\n END IF;\n\n IF jsonb_path_exists(args, '$[*] ? (@.property == \"id\" || @.property == \"datetime\" || @.property == \"end_datetime\" || @.property == \"collection\")') THEN\n wrapper := NULL;\n ELSE\n -- if any of the arguments are a property, try to get the property_wrapper\n FOR arg IN SELECT jsonb_path_query(args, '$[*] ? (@.property != null)') LOOP\n RAISE NOTICE 'Arg: %', arg;\n wrapper := (queryable(arg->>'property')).nulled_wrapper;\n RAISE NOTICE 'Property: %, Wrapper: %', arg, wrapper;\n IF wrapper IS NOT NULL THEN\n EXIT;\n END IF;\n END LOOP;\n\n -- if the property was not in queryables, see if any args were numbers\n IF\n wrapper IS NULL\n AND jsonb_path_exists(args, '$[*] ? (@.type()==\"number\")')\n THEN\n wrapper := 'to_float';\n END IF;\n wrapper := coalesce(wrapper, 'to_text');\n END IF;\n\n SELECT jsonb_agg(cql2_query(a, wrapper))\n INTO args\n FROM jsonb_array_elements(args) a;\n END IF;\n RAISE NOTICE 'ARGS: %', args;\n\n IF op IN ('and', 'or') THEN\n RETURN\n format(\n '(%s)',\n array_to_string(to_text_array(args), format(' %s ', upper(op)))\n );\n END IF;\n\n IF op = 'in' THEN\n RAISE NOTICE 'IN -- % %', args->0, to_text(args->0);\n RETURN format(\n '%s IN (%s)',\n to_text(args->0),\n array_to_string((to_text_array(args))[2:], ',')\n );\n END IF;\n\n -- Look up template from cql2_ops\n IF j ? 'op' THEN\n SELECT * INTO cql2op FROM cql2_ops WHERE cql2_ops.op ilike op;\n IF FOUND THEN\n -- If specific index set in queryables for a property cast other arguments to that type\n\n RETURN format(\n cql2op.template,\n VARIADIC (to_text_array(args))\n );\n ELSE\n RAISE EXCEPTION 'Operator % Not Supported.', op;\n END IF;\n END IF;\n\n\n IF wrapper IS NOT NULL THEN\n RAISE NOTICE 'Wrapping % with %', j, wrapper;\n IF j ? 'property' THEN\n RETURN format('%I(%s)', wrapper, (queryable(j->>'property')).path);\n ELSE\n RETURN format('%I(%L)', wrapper, j);\n END IF;\n ELSIF j ? 'property' THEN\n RETURN quote_ident(j->>'property');\n END IF;\n\n RETURN quote_literal(to_text(j));\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION paging_dtrange(\n j jsonb\n) RETURNS tstzrange AS $$\nDECLARE\n op text;\n filter jsonb := j->'filter';\n dtrange tstzrange := tstzrange('-infinity'::timestamptz,'infinity'::timestamptz);\n sdate timestamptz := '-infinity'::timestamptz;\n edate timestamptz := 'infinity'::timestamptz;\n jpitem jsonb;\nBEGIN\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n END IF;\n IF NOT (filter @? '$.**.op ? (@ == \"or\" || @ == \"not\")') THEN\n FOR jpitem IN SELECT j FROM jsonb_path_query(filter,'strict $.** ? (@.args[*].property == \"datetime\")'::jsonpath) j LOOP\n op := lower(jpitem->>'op');\n dtrange := parse_dtrange(jpitem->'args'->1);\n IF op IN ('<=', 'lt', 'lte', '<', 'le', 't_before') THEN\n sdate := greatest(sdate,'-infinity');\n edate := least(edate, upper(dtrange));\n ELSIF op IN ('>=', '>', 'gt', 'gte', 'ge', 't_after') THEN\n edate := least(edate, 'infinity');\n sdate := greatest(sdate, lower(dtrange));\n ELSIF op IN ('=', 'eq') THEN\n edate := least(edate, upper(dtrange));\n sdate := greatest(sdate, lower(dtrange));\n END IF;\n RAISE NOTICE '2 OP: %, ARGS: %, DTRANGE: %, SDATE: %, EDATE: %', op, jpitem->'args'->1, dtrange, sdate, edate;\n END LOOP;\n END IF;\n IF sdate > edate THEN\n RETURN 'empty'::tstzrange;\n END IF;\n RETURN tstzrange(sdate,edate, '[]');\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT SET TIME ZONE 'UTC';\n\nCREATE OR REPLACE FUNCTION paging_collections(\n IN j jsonb\n) RETURNS text[] AS $$\nDECLARE\n filter jsonb := j->'filter';\n jpitem jsonb;\n op text;\n args jsonb;\n arg jsonb;\n collections text[];\nBEGIN\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n END IF;\n IF NOT (filter @? '$.**.op ? (@ == \"or\" || @ == \"not\")') THEN\n FOR jpitem IN SELECT j FROM jsonb_path_query(filter,'strict $.** ? (@.args[*].property == \"collection\")'::jsonpath) j LOOP\n RAISE NOTICE 'JPITEM: %', jpitem;\n op := jpitem->>'op';\n args := jpitem->'args';\n IF op IN ('=', 'eq', 'in') THEN\n FOR arg IN SELECT a FROM jsonb_array_elements(args) a LOOP\n IF jsonb_typeof(arg) IN ('string', 'array') THEN\n RAISE NOTICE 'arg: %, collections: %', arg, collections;\n IF collections IS NULL OR collections = '{}'::text[] THEN\n collections := to_text_array(arg);\n ELSE\n collections := array_intersection(collections, to_text_array(arg));\n END IF;\n END IF;\n END LOOP;\n END IF;\n END LOOP;\n END IF;\n IF collections = '{}'::text[] THEN\n RETURN NULL;\n END IF;\n RETURN collections;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\nCREATE TABLE items (\n id text NOT NULL,\n geometry geometry NOT NULL,\n collection text NOT NULL,\n datetime timestamptz NOT NULL,\n end_datetime timestamptz NOT NULL,\n content JSONB NOT NULL,\n private jsonb\n)\nPARTITION BY LIST (collection)\n;\n\nCREATE INDEX \"datetime_idx\" ON items USING BTREE (datetime DESC, end_datetime ASC);\nCREATE INDEX \"geometry_idx\" ON items USING GIST (geometry);\n\nCREATE STATISTICS datetime_stats (dependencies) on datetime, end_datetime from items;\n\nALTER TABLE items ADD CONSTRAINT items_collections_fk FOREIGN KEY (collection) REFERENCES collections(id) ON DELETE CASCADE DEFERRABLE;\n\nCREATE OR REPLACE FUNCTION partition_after_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p text;\n t timestamptz := clock_timestamp();\nBEGIN\n RAISE NOTICE 'Updating partition stats %', t;\n FOR p IN SELECT DISTINCT partition\n FROM newdata n JOIN partition_sys_meta p\n ON (n.collection=p.collection AND n.datetime <@ p.partition_dtrange)\n LOOP\n PERFORM run_or_queue(format('SELECT update_partition_stats(%L, %L);', p, true));\n END LOOP;\n IF TG_OP IN ('DELETE','UPDATE') THEN\n DELETE FROM format_item_cache c USING newdata n WHERE c.collection = n.collection AND c.id = n.id;\n END IF;\n RAISE NOTICE 't: % %', t, clock_timestamp() - t;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE TRIGGER items_after_insert_trigger\nAFTER INSERT ON items\nREFERENCING NEW TABLE AS newdata\nFOR EACH STATEMENT\nEXECUTE FUNCTION partition_after_triggerfunc();\n\nCREATE TRIGGER items_after_update_trigger\nAFTER DELETE ON items\nREFERENCING OLD TABLE AS newdata\nFOR EACH STATEMENT\nEXECUTE FUNCTION partition_after_triggerfunc();\n\nCREATE TRIGGER items_after_delete_trigger\nAFTER UPDATE ON items\nREFERENCING NEW TABLE AS newdata\nFOR EACH STATEMENT\nEXECUTE FUNCTION partition_after_triggerfunc();\n\n\nCREATE OR REPLACE FUNCTION content_slim(_item jsonb) RETURNS jsonb AS $$\n SELECT strip_jsonb(_item - '{id,geometry,collection,type}'::text[], collection_base_item(_item->>'collection')) - '{id,geometry,collection,type}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_dehydrate(content jsonb) RETURNS items AS $$\n SELECT\n content->>'id' as id,\n stac_geom(content) as geometry,\n content->>'collection' as collection,\n stac_datetime(content) as datetime,\n stac_end_datetime(content) as end_datetime,\n content_slim(content) as content,\n null::jsonb as private\n ;\n$$ LANGUAGE SQL STABLE;\n\nCREATE OR REPLACE FUNCTION include_field(f text, fields jsonb DEFAULT '{}'::jsonb) RETURNS boolean AS $$\nDECLARE\n includes jsonb := fields->'include';\n excludes jsonb := fields->'exclude';\nBEGIN\n IF f IS NULL THEN\n RETURN NULL;\n END IF;\n\n\n IF\n jsonb_typeof(excludes) = 'array'\n AND jsonb_array_length(excludes)>0\n AND excludes ? f\n THEN\n RETURN FALSE;\n END IF;\n\n IF\n (\n jsonb_typeof(includes) = 'array'\n AND jsonb_array_length(includes) > 0\n AND includes ? f\n ) OR\n (\n includes IS NULL\n OR jsonb_typeof(includes) = 'null'\n OR jsonb_array_length(includes) = 0\n )\n THEN\n RETURN TRUE;\n END IF;\n\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nDROP FUNCTION IF EXISTS content_hydrate(jsonb, jsonb, jsonb);\nCREATE OR REPLACE FUNCTION content_hydrate(\n _item jsonb,\n _base_item jsonb,\n fields jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\n SELECT merge_jsonb(\n jsonb_fields(_item, fields),\n jsonb_fields(_base_item, fields)\n );\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION content_hydrate(_item items, _collection collections, fields jsonb DEFAULT '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\n content jsonb;\n base_item jsonb := _collection.base_item;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry, 20)::jsonb;\n END IF;\n output := content_hydrate(\n jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'collection', _item.collection,\n 'type', 'Feature'\n ) || _item.content,\n _collection.base_item,\n fields\n );\n\n RETURN output;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_nonhydrated(\n _item items,\n fields jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry, 20)::jsonb;\n END IF;\n output := jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'collection', _item.collection,\n 'type', 'Feature'\n ) || _item.content;\n RETURN output;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_hydrate(_item items, fields jsonb DEFAULT '{}'::jsonb) RETURNS jsonb AS $$\n SELECT content_hydrate(\n _item,\n (SELECT c FROM collections c WHERE id=_item.collection LIMIT 1),\n fields\n );\n$$ LANGUAGE SQL STABLE;\n\n\nCREATE UNLOGGED TABLE items_staging (\n content JSONB NOT NULL\n);\nCREATE UNLOGGED TABLE items_staging_ignore (\n content JSONB NOT NULL\n);\nCREATE UNLOGGED TABLE items_staging_upsert (\n content JSONB NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION items_staging_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p record;\n _partitions text[];\n part text;\n ts timestamptz := clock_timestamp();\n nrows int;\nBEGIN\n RAISE NOTICE 'Creating Partitions. %', clock_timestamp() - ts;\n\n FOR part IN WITH t AS (\n SELECT\n n.content->>'collection' as collection,\n stac_daterange(n.content->'properties') as dtr,\n partition_trunc\n FROM newdata n JOIN collections ON (n.content->>'collection'=collections.id)\n ), p AS (\n SELECT\n collection,\n COALESCE(date_trunc(partition_trunc::text, lower(dtr)),'-infinity') as d,\n tstzrange(min(lower(dtr)),max(lower(dtr)),'[]') as dtrange,\n tstzrange(min(upper(dtr)),max(upper(dtr)),'[]') as edtrange\n FROM t\n GROUP BY 1,2\n ) SELECT check_partition(collection, dtrange, edtrange) FROM p LOOP\n RAISE NOTICE 'Partition %', part;\n END LOOP;\n\n RAISE NOTICE 'Creating temp table with data to be added. %', clock_timestamp() - ts;\n DROP TABLE IF EXISTS tmpdata;\n CREATE TEMP TABLE tmpdata ON COMMIT DROP AS\n SELECT\n (content_dehydrate(content)).*\n FROM newdata;\n GET DIAGNOSTICS nrows = ROW_COUNT;\n RAISE NOTICE 'Added % rows to tmpdata. %', nrows, clock_timestamp() - ts;\n\n RAISE NOTICE 'Doing the insert. %', clock_timestamp() - ts;\n IF TG_TABLE_NAME = 'items_staging' THEN\n INSERT INTO items\n SELECT * FROM tmpdata;\n GET DIAGNOSTICS nrows = ROW_COUNT;\n RAISE NOTICE 'Inserted % rows to items. %', nrows, clock_timestamp() - ts;\n ELSIF TG_TABLE_NAME = 'items_staging_ignore' THEN\n INSERT INTO items\n SELECT * FROM tmpdata\n ON CONFLICT DO NOTHING;\n GET DIAGNOSTICS nrows = ROW_COUNT;\n RAISE NOTICE 'Inserted % rows to items. %', nrows, clock_timestamp() - ts;\n ELSIF TG_TABLE_NAME = 'items_staging_upsert' THEN\n DELETE FROM items i USING tmpdata s\n WHERE\n i.id = s.id\n AND i.collection = s.collection\n AND i IS DISTINCT FROM s\n ;\n GET DIAGNOSTICS nrows = ROW_COUNT;\n RAISE NOTICE 'Deleted % rows from items. %', nrows, clock_timestamp() - ts;\n INSERT INTO items AS t\n SELECT * FROM tmpdata\n ON CONFLICT DO NOTHING;\n GET DIAGNOSTICS nrows = ROW_COUNT;\n RAISE NOTICE 'Inserted % rows to items. %', nrows, clock_timestamp() - ts;\n END IF;\n\n RAISE NOTICE 'Deleting data from staging table. %', clock_timestamp() - ts;\n DELETE FROM items_staging;\n RAISE NOTICE 'Done. %', clock_timestamp() - ts;\n\n RETURN NULL;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER items_staging_insert_trigger AFTER INSERT ON items_staging REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\nCREATE TRIGGER items_staging_insert_ignore_trigger AFTER INSERT ON items_staging_ignore REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\nCREATE TRIGGER items_staging_insert_upsert_trigger AFTER INSERT ON items_staging_upsert REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\n\nCREATE OR REPLACE FUNCTION item_by_id(_id text, _collection text DEFAULT NULL) RETURNS items AS\n$$\nDECLARE\n i items%ROWTYPE;\nBEGIN\n SELECT * INTO i FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection) LIMIT 1;\n RETURN i;\nEND;\n$$ LANGUAGE PLPGSQL STABLE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION get_item(_id text, _collection text DEFAULT NULL) RETURNS jsonb AS $$\n SELECT content_hydrate(items) FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection);\n$$ LANGUAGE SQL STABLE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION delete_item(_id text, _collection text DEFAULT NULL) RETURNS VOID AS $$\nDECLARE\nout items%ROWTYPE;\nBEGIN\n DELETE FROM items WHERE id = _id AND (_collection IS NULL OR collection=_collection) RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n--/*\nCREATE OR REPLACE FUNCTION create_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION update_item(content jsonb) RETURNS VOID AS $$\nDECLARE\n old items %ROWTYPE;\n out items%ROWTYPE;\nBEGIN\n PERFORM delete_item(content->>'id', content->>'collection');\n PERFORM create_item(content);\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION create_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION collection_bbox(id text) RETURNS jsonb AS $$\n SELECT (replace(replace(replace(st_extent(geometry)::text,'BOX(','[['),')',']]'),' ',','))::jsonb\n FROM items WHERE collection=$1;\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION collection_temporal_extent(id text) RETURNS jsonb AS $$\n SELECT to_jsonb(array[array[min(datetime), max(datetime)]])\n FROM items WHERE collection=$1;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection_extents() RETURNS VOID AS $$\nUPDATE collections\n SET content = jsonb_set_lax(\n content,\n '{extent}'::text[],\n collection_extent(id, FALSE),\n true,\n 'use_json_null'\n )\n;\n$$ LANGUAGE SQL;\nCREATE TABLE partition_stats (\n partition text PRIMARY KEY,\n dtrange tstzrange,\n edtrange tstzrange,\n spatial geometry,\n last_updated timestamptz,\n keys text[]\n) WITH (FILLFACTOR=90);\n\nCREATE INDEX partitions_range_idx ON partition_stats USING GIST(dtrange);\n\n\nCREATE OR REPLACE FUNCTION constraint_tstzrange(expr text) RETURNS tstzrange AS $$\n WITH t AS (\n SELECT regexp_matches(\n expr,\n E'\\\\(''\\([0-9 :+-]*\\)''\\\\).*\\\\(''\\([0-9 :+-]*\\)''\\\\)'\n ) AS m\n ) SELECT tstzrange(m[1]::timestamptz, m[2]::timestamptz) FROM t\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE STRICT;\n\nCREATE OR REPLACE FUNCTION dt_constraint(coid oid, OUT dt tstzrange, OUT edt tstzrange) RETURNS RECORD AS $$\nDECLARE\n expr text := pg_get_constraintdef(coid);\n matches timestamptz[];\nBEGIN\n IF expr LIKE '%NULL%' THEN\n dt := tstzrange(null::timestamptz, null::timestamptz);\n edt := tstzrange(null::timestamptz, null::timestamptz);\n RETURN;\n END IF;\n WITH f AS (SELECT (regexp_matches(expr, E'([0-9]{4}-[0-1][0-9]-[0-3][0-9] [0-2][0-9]:[0-5][0-9]:[0-5][0-9]\\.?[0-9]*)', 'g'))[1] f)\n SELECT array_agg(f::timestamptz) INTO matches FROM f;\n IF cardinality(matches) = 4 THEN\n dt := tstzrange(matches[1], matches[2],'[]');\n edt := tstzrange(matches[3], matches[4], '[]');\n RETURN;\n ELSIF cardinality(matches) = 2 THEN\n edt := tstzrange(matches[1], matches[2],'[]');\n RETURN;\n END IF;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\nCREATE OR REPLACE VIEW partition_sys_meta AS\nSELECT\n (parse_ident(relid::text))[cardinality(parse_ident(relid::text))] as partition,\n replace(replace(CASE WHEN level = 1 THEN pg_get_expr(c.relpartbound, c.oid)\n ELSE pg_get_expr(parent.relpartbound, parent.oid)\n END, 'FOR VALUES IN (''',''), ''')','') AS collection,\n level,\n c.reltuples,\n c.relhastriggers,\n COALESCE(constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as partition_dtrange,\n COALESCE((dt_constraint(edt.oid)).dt, constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as constraint_dtrange,\n COALESCE((dt_constraint(edt.oid)).edt, tstzrange('-infinity', 'infinity','[]')) as constraint_edtrange\nFROM\n pg_partition_tree('items')\n JOIN pg_class c ON (relid::regclass = c.oid)\n JOIN pg_class parent ON (parentrelid::regclass = parent.oid AND isleaf)\n LEFT JOIN pg_constraint edt ON (conrelid=c.oid AND contype='c')\nWHERE isleaf\n;\n\nCREATE OR REPLACE VIEW partitions_view AS\nSELECT\n (parse_ident(relid::text))[cardinality(parse_ident(relid::text))] as partition,\n replace(replace(CASE WHEN level = 1 THEN pg_get_expr(c.relpartbound, c.oid)\n ELSE pg_get_expr(parent.relpartbound, parent.oid)\n END, 'FOR VALUES IN (''',''), ''')','') AS collection,\n level,\n c.reltuples,\n c.relhastriggers,\n COALESCE(pgstac.constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as partition_dtrange,\n COALESCE((pgstac.dt_constraint(edt.oid)).dt, pgstac.constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as constraint_dtrange,\n COALESCE((pgstac.dt_constraint(edt.oid)).edt, tstzrange('-infinity', 'infinity','[]')) as constraint_edtrange,\n dtrange,\n edtrange,\n spatial,\n last_updated\nFROM\n pg_partition_tree('pgstac.items')\n JOIN pg_class c ON (relid::regclass = c.oid)\n JOIN pg_class parent ON (parentrelid::regclass = parent.oid AND isleaf)\n LEFT JOIN pg_constraint edt ON (conrelid=c.oid AND contype='c')\n LEFT JOIN pgstac.partition_stats ON ((parse_ident(relid::text))[cardinality(parse_ident(relid::text))]=partition)\nWHERE isleaf\n;\n\nCREATE MATERIALIZED VIEW partitions AS\nSELECT * FROM partitions_view;\nCREATE UNIQUE INDEX ON partitions (partition);\n\nCREATE MATERIALIZED VIEW partition_steps AS\nSELECT\n partition as name,\n date_trunc('month',lower(partition_dtrange)) as sdate,\n date_trunc('month', upper(partition_dtrange)) + '1 month'::interval as edate\n FROM partitions_view WHERE partition_dtrange IS NOT NULL AND partition_dtrange != 'empty'::tstzrange\n ORDER BY dtrange ASC\n;\n\n\nCREATE OR REPLACE FUNCTION update_partition_stats_q(_partition text, istrigger boolean default false) RETURNS VOID AS $$\nDECLARE\nBEGIN\n PERFORM run_or_queue(\n format('SELECT update_partition_stats(%L, %L);', _partition, istrigger)\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION update_partition_stats(_partition text, istrigger boolean default false) RETURNS VOID AS $$\nDECLARE\n dtrange tstzrange;\n edtrange tstzrange;\n cdtrange tstzrange;\n cedtrange tstzrange;\n extent geometry;\n collection text;\nBEGIN\n RAISE NOTICE 'Updating stats for %.', _partition;\n EXECUTE format(\n $q$\n SELECT\n tstzrange(min(datetime), max(datetime),'[]'),\n tstzrange(min(end_datetime), max(end_datetime), '[]')\n FROM %I\n $q$,\n _partition\n ) INTO dtrange, edtrange;\n extent := st_estimatedextent('pgstac', _partition, 'geometry');\n RAISE DEBUG 'Estimated Extent: %', extent;\n INSERT INTO partition_stats (partition, dtrange, edtrange, spatial, last_updated)\n SELECT _partition, dtrange, edtrange, extent, now()\n ON CONFLICT (partition) DO\n UPDATE SET\n dtrange=EXCLUDED.dtrange,\n edtrange=EXCLUDED.edtrange,\n spatial=EXCLUDED.spatial,\n last_updated=EXCLUDED.last_updated\n ;\n\n SELECT\n constraint_dtrange, constraint_edtrange, pv.collection\n INTO cdtrange, cedtrange, collection\n FROM partitions_view pv WHERE partition = _partition;\n REFRESH MATERIALIZED VIEW partitions;\n REFRESH MATERIALIZED VIEW partition_steps;\n\n\n RAISE NOTICE 'Checking if we need to modify constraints...';\n RAISE NOTICE 'cdtrange: % dtrange: % cedtrange: % edtrange: %',cdtrange, dtrange, cedtrange, edtrange;\n IF\n (cdtrange IS DISTINCT FROM dtrange OR edtrange IS DISTINCT FROM cedtrange)\n AND NOT istrigger\n THEN\n RAISE NOTICE 'Modifying Constraints';\n RAISE NOTICE 'Existing % %', cdtrange, cedtrange;\n RAISE NOTICE 'New % %', dtrange, edtrange;\n PERFORM drop_table_constraints(_partition);\n PERFORM create_table_constraints(_partition, dtrange, edtrange);\n REFRESH MATERIALIZED VIEW partitions;\n REFRESH MATERIALIZED VIEW partition_steps;\n END IF;\n RAISE NOTICE 'Checking if we need to update collection extents.';\n IF get_setting_bool('update_collection_extent') THEN\n RAISE NOTICE 'updating collection extent for %', collection;\n PERFORM run_or_queue(format($q$\n UPDATE collections\n SET content = jsonb_set_lax(\n content,\n '{extent}'::text[],\n collection_extent(%L, FALSE),\n true,\n 'use_json_null'\n ) WHERE id=%L\n ;\n $q$, collection, collection));\n ELSE\n RAISE NOTICE 'Not updating collection extent for %', collection;\n END IF;\n\nEND;\n$$ LANGUAGE PLPGSQL STRICT SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION partition_name( IN collection text, IN dt timestamptz, OUT partition_name text, OUT partition_range tstzrange) AS $$\nDECLARE\n c RECORD;\n parent_name text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n parent_name := format('_items_%s', c.key);\n\n\n IF c.partition_trunc = 'year' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYY'));\n ELSIF c.partition_trunc = 'month' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYYMM'));\n ELSE\n partition_name := parent_name;\n partition_range := tstzrange('-infinity'::timestamptz, 'infinity'::timestamptz, '[]');\n END IF;\n IF partition_range IS NULL THEN\n partition_range := tstzrange(\n date_trunc(c.partition_trunc::text, dt),\n date_trunc(c.partition_trunc::text, dt) + concat('1 ', c.partition_trunc)::interval\n );\n END IF;\n RETURN;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION drop_table_constraints(t text) RETURNS text AS $$\nDECLARE\n q text;\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM partitions_view WHERE partition=t) THEN\n RETURN NULL;\n END IF;\n FOR q IN SELECT FORMAT(\n $q$\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n $q$,\n t,\n conname\n ) FROM pg_constraint\n WHERE conrelid=t::regclass::oid AND contype='c'\n LOOP\n EXECUTE q;\n END LOOP;\n RETURN t;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE OR REPLACE FUNCTION create_table_constraints(t text, _dtrange tstzrange, _edtrange tstzrange) RETURNS text AS $$\nDECLARE\n q text;\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM partitions_view WHERE partition=t) THEN\n RETURN NULL;\n END IF;\n RAISE NOTICE 'Creating Table Constraints for % % %', t, _dtrange, _edtrange;\n IF _dtrange = 'empty' AND _edtrange = 'empty' THEN\n q :=format(\n $q$\n DO $block$\n BEGIN\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n ALTER TABLE %I\n ADD CONSTRAINT %I\n CHECK (((datetime IS NULL) AND (end_datetime IS NULL))) NOT VALID\n ;\n ALTER TABLE %I\n VALIDATE CONSTRAINT %I\n ;\n\n\n\n EXCEPTION WHEN others THEN\n RAISE WARNING '%%, Issue Altering Constraints. Please run update_partition_stats(%I)', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n $block$;\n $q$,\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t),\n t\n );\n ELSE\n q :=format(\n $q$\n DO $block$\n BEGIN\n\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n ALTER TABLE %I\n ADD CONSTRAINT %I\n CHECK (\n (datetime >= %L)\n AND (datetime <= %L)\n AND (end_datetime >= %L)\n AND (end_datetime <= %L)\n ) NOT VALID\n ;\n ALTER TABLE %I\n VALIDATE CONSTRAINT %I\n ;\n\n\n\n EXCEPTION WHEN others THEN\n RAISE WARNING '%%, Issue Altering Constraints. Please run update_partition_stats(%I)', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n $block$;\n $q$,\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t),\n lower(_dtrange),\n upper(_dtrange),\n lower(_edtrange),\n upper(_edtrange),\n t,\n format('%s_dt', t),\n t\n );\n END IF;\n PERFORM run_or_queue(q);\n RETURN t;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION check_partition(\n _collection text,\n _dtrange tstzrange,\n _edtrange tstzrange\n) RETURNS text AS $$\nDECLARE\n c RECORD;\n pm RECORD;\n _partition_name text;\n _partition_dtrange tstzrange;\n _constraint_dtrange tstzrange;\n _constraint_edtrange tstzrange;\n q text;\n deferrable_q text;\n err_context text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=_collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', _collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n\n IF c.partition_trunc IS NOT NULL THEN\n _partition_dtrange := tstzrange(\n date_trunc(c.partition_trunc, lower(_dtrange)),\n date_trunc(c.partition_trunc, lower(_dtrange)) + (concat('1 ', c.partition_trunc))::interval,\n '[)'\n );\n ELSE\n _partition_dtrange := '[-infinity, infinity]'::tstzrange;\n END IF;\n\n IF NOT _partition_dtrange @> _dtrange THEN\n RAISE EXCEPTION 'dtrange % is greater than the partition size % for collection %', _dtrange, c.partition_trunc, _collection;\n END IF;\n\n\n IF c.partition_trunc = 'year' THEN\n _partition_name := format('_items_%s_%s', c.key, to_char(lower(_partition_dtrange),'YYYY'));\n ELSIF c.partition_trunc = 'month' THEN\n _partition_name := format('_items_%s_%s', c.key, to_char(lower(_partition_dtrange),'YYYYMM'));\n ELSE\n _partition_name := format('_items_%s', c.key);\n END IF;\n\n SELECT * INTO pm FROM partition_sys_meta WHERE collection=_collection AND partition_dtrange @> _dtrange;\n IF FOUND THEN\n RAISE NOTICE '% % %', _edtrange, _dtrange, pm;\n _constraint_edtrange :=\n tstzrange(\n least(\n lower(_edtrange),\n nullif(lower(pm.constraint_edtrange), '-infinity')\n ),\n greatest(\n upper(_edtrange),\n nullif(upper(pm.constraint_edtrange), 'infinity')\n ),\n '[]'\n );\n _constraint_dtrange :=\n tstzrange(\n least(\n lower(_dtrange),\n nullif(lower(pm.constraint_dtrange), '-infinity')\n ),\n greatest(\n upper(_dtrange),\n nullif(upper(pm.constraint_dtrange), 'infinity')\n ),\n '[]'\n );\n\n IF pm.constraint_edtrange @> _edtrange AND pm.constraint_dtrange @> _dtrange THEN\n RETURN pm.partition;\n ELSE\n PERFORM drop_table_constraints(_partition_name);\n END IF;\n ELSE\n _constraint_edtrange := _edtrange;\n _constraint_dtrange := _dtrange;\n END IF;\n RAISE NOTICE 'EXISTING CONSTRAINTS % %, NEW % %', pm.constraint_dtrange, pm.constraint_edtrange, _constraint_dtrange, _constraint_edtrange;\n RAISE NOTICE 'Creating partition % %', _partition_name, _partition_dtrange;\n IF c.partition_trunc IS NULL THEN\n q := format(\n $q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n GRANT ALL ON %I to pgstac_ingest;\n $q$,\n _partition_name,\n _collection,\n concat(_partition_name,'_pk'),\n _partition_name,\n _partition_name\n );\n ELSE\n q := format(\n $q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L) PARTITION BY RANGE (datetime);\n CREATE TABLE IF NOT EXISTS %I partition OF %I FOR VALUES FROM (%L) TO (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n GRANT ALL ON %I TO pgstac_ingest;\n $q$,\n format('_items_%s', c.key),\n _collection,\n _partition_name,\n format('_items_%s', c.key),\n lower(_partition_dtrange),\n upper(_partition_dtrange),\n format('%s_pk', _partition_name),\n _partition_name,\n _partition_name\n );\n END IF;\n\n BEGIN\n EXECUTE q;\n EXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', _partition_name;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\n END;\n PERFORM maintain_partitions(_partition_name);\n PERFORM update_partition_stats_q(_partition_name, true);\n REFRESH MATERIALIZED VIEW partitions;\n REFRESH MATERIALIZED VIEW partition_steps;\n RETURN _partition_name;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION repartition(_collection text, _partition_trunc text, triggered boolean DEFAULT FALSE) RETURNS text AS $$\nDECLARE\n c RECORD;\n q text;\n from_trunc text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=_collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', _collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n IF triggered THEN\n RAISE NOTICE 'Converting % to % partitioning via Trigger', _collection, _partition_trunc;\n ELSE\n RAISE NOTICE 'Converting % from using % to % partitioning', _collection, c.partition_trunc, _partition_trunc;\n IF c.partition_trunc IS NOT DISTINCT FROM _partition_trunc THEN\n RAISE NOTICE 'Collection % already set to use partition by %', _collection, _partition_trunc;\n RETURN _collection;\n END IF;\n END IF;\n\n IF EXISTS (SELECT 1 FROM partitions_view WHERE collection=_collection LIMIT 1) THEN\n EXECUTE format(\n $q$\n CREATE TEMP TABLE changepartitionstaging ON COMMIT DROP AS SELECT * FROM %I;\n DROP TABLE IF EXISTS %I CASCADE;\n WITH p AS (\n SELECT\n collection,\n CASE\n WHEN %L IS NULL THEN '-infinity'::timestamptz\n ELSE date_trunc(%L, datetime)\n END as d,\n tstzrange(min(datetime),max(datetime),'[]') as dtrange,\n tstzrange(min(end_datetime),max(end_datetime),'[]') as edtrange\n FROM changepartitionstaging\n GROUP BY 1,2\n ) SELECT check_partition(collection, dtrange, edtrange) FROM p;\n INSERT INTO items SELECT * FROM changepartitionstaging;\n DROP TABLE changepartitionstaging;\n $q$,\n concat('_items_', c.key),\n concat('_items_', c.key),\n c.partition_trunc,\n c.partition_trunc\n );\n END IF;\n RETURN _collection;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE OR REPLACE FUNCTION collections_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n q text;\n partition_name text := format('_items_%s', NEW.key);\n partition_exists boolean := false;\n partition_empty boolean := true;\n err_context text;\n loadtemp boolean := FALSE;\nBEGIN\n RAISE NOTICE 'Collection Trigger. % %', NEW.id, NEW.key;\n IF TG_OP = 'UPDATE' AND NEW.partition_trunc IS DISTINCT FROM OLD.partition_trunc THEN\n PERFORM repartition(NEW.id, NEW.partition_trunc, TRUE);\n END IF;\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER collections_trigger AFTER\nINSERT\nOR\nUPDATE ON collections\nFOR EACH ROW EXECUTE FUNCTION collections_trigger_func();\n\nCREATE OR REPLACE FUNCTION chunker(\n IN _where text,\n OUT s timestamptz,\n OUT e timestamptz\n) RETURNS SETOF RECORD AS $$\nDECLARE\n explain jsonb;\nBEGIN\n IF _where IS NULL THEN\n _where := ' TRUE ';\n END IF;\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s;', _where)\n INTO explain;\n RAISE DEBUG 'EXPLAIN: %', explain;\n\n RETURN QUERY\n WITH t AS (\n SELECT j->>0 as p FROM\n jsonb_path_query(\n explain,\n 'strict $.**.\"Relation Name\" ? (@ != null)'\n ) j\n ),\n parts AS (\n SELECT sdate, edate FROM t JOIN partition_steps ON (t.p = name)\n ),\n times AS (\n SELECT sdate FROM parts\n UNION\n SELECT edate FROM parts\n ),\n uniq AS (\n SELECT DISTINCT sdate FROM times ORDER BY sdate\n ),\n last AS (\n SELECT sdate, lead(sdate, 1) over () as edate FROM uniq\n )\n SELECT sdate, edate FROM last WHERE edate IS NOT NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION partition_queries(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN partitions text[] DEFAULT NULL\n) RETURNS SETOF text AS $$\nDECLARE\n query text;\n sdate timestamptz;\n edate timestamptz;\nBEGIN\nIF _where IS NULL OR trim(_where) = '' THEN\n _where = ' TRUE ';\nEND IF;\nRAISE NOTICE 'Getting chunks for % %', _where, _orderby;\nIF _orderby ILIKE 'datetime d%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 DESC LOOP\n RETURN NEXT format($q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n $q$,\n sdate,\n edate,\n _where,\n _orderby\n );\n END LOOP;\nELSIF _orderby ILIKE 'datetime a%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 ASC LOOP\n RETURN NEXT format($q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n $q$,\n sdate,\n edate,\n _where,\n _orderby\n );\n END LOOP;\nELSE\n query := format($q$\n SELECT * FROM items\n WHERE %s\n ORDER BY %s\n $q$, _where, _orderby\n );\n\n RETURN NEXT query;\n RETURN;\nEND IF;\n\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION partition_query_view(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN _limit int DEFAULT 10\n) RETURNS text AS $$\n WITH p AS (\n SELECT * FROM partition_queries(_where, _orderby) p\n )\n SELECT\n CASE WHEN EXISTS (SELECT 1 FROM p) THEN\n (SELECT format($q$\n SELECT * FROM (\n %s\n ) total LIMIT %s\n $q$,\n string_agg(\n format($q$ SELECT * FROM ( %s ) AS sub $q$, p),\n '\n UNION ALL\n '\n ),\n _limit\n ))\n ELSE NULL\n END FROM p;\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION q_to_tsquery (jinput jsonb)\n RETURNS tsquery\n AS $$\nDECLARE\n input text;\n processed_text text;\n temp_text text;\n quote_array text[];\n placeholder text := '@QUOTE@';\nBEGIN\n IF jsonb_typeof(jinput) = 'string' THEN\n input := jinput->>0;\n ELSIF jsonb_typeof(jinput) = 'array' THEN\n input := array_to_string(\n array(select jsonb_array_elements_text(jinput)),\n ' OR '\n );\n ELSE\n RAISE EXCEPTION 'Input must be a string or an array of strings.';\n END IF;\n -- Extract all quoted phrases and store in array\n quote_array := regexp_matches(input, '\"[^\"]*\"', 'g');\n\n -- Replace each quoted part with a unique placeholder if there are any quoted phrases\n IF array_length(quote_array, 1) IS NOT NULL THEN\n processed_text := input;\n FOR i IN array_lower(quote_array, 1) .. array_upper(quote_array, 1) LOOP\n processed_text := replace(processed_text, quote_array[i], placeholder || i || placeholder);\n END LOOP;\n ELSE\n processed_text := input;\n END IF;\n\n -- Replace non-quoted text using regular expressions\n\n -- , -> |\n processed_text := regexp_replace(processed_text, ',(?=(?:[^\"]*\"[^\"]*\")*[^\"]*$)', ' | ', 'g');\n\n -- and -> &\n processed_text := regexp_replace(processed_text, '\\s+AND\\s+', ' & ', 'gi');\n\n -- or -> |\n processed_text := regexp_replace(processed_text, '\\s+OR\\s+', ' | ', 'gi');\n\n -- + ->\n processed_text := regexp_replace(processed_text, '^\\s*\\+([a-zA-Z0-9_]+)', '\\1', 'g'); -- +term at start\n processed_text := regexp_replace(processed_text, '\\s*\\+([a-zA-Z0-9_]+)', ' & \\1', 'g'); -- +term elsewhere\n\n -- - -> !\n processed_text := regexp_replace(processed_text, '^\\s*\\-([a-zA-Z0-9_]+)', '! \\1', 'g'); -- -term at start\n processed_text := regexp_replace(processed_text, '\\s*\\-([a-zA-Z0-9_]+)', ' & ! \\1', 'g'); -- -term elsewhere\n -- Replace placeholders back with quoted phrases if there were any\n IF array_length(quote_array, 1) IS NOT NULL THEN\n FOR i IN array_lower(quote_array, 1) .. array_upper(quote_array, 1) LOOP\n processed_text := replace(processed_text, placeholder || i || placeholder, '''' || substring(quote_array[i] from 2 for length(quote_array[i]) - 2) || '''');\n END LOOP;\n END IF;\n\n -- Print processed_text to the console for debugging purposes\n RAISE NOTICE 'processed_text: %', processed_text;\n\n RETURN to_tsquery('english', processed_text);\nEND;\n$$\nLANGUAGE plpgsql;\n\n\nCREATE OR REPLACE FUNCTION stac_search_to_where(j jsonb) RETURNS text AS $$\nDECLARE\n where_segments text[];\n _where text;\n dtrange tstzrange;\n collections text[];\n geom geometry;\n sdate timestamptz;\n edate timestamptz;\n filterlang text;\n filter jsonb := j->'filter';\n ft_query tsquery;\nBEGIN\n IF j ? 'ids' THEN\n where_segments := where_segments || format('id = ANY (%L) ', to_text_array(j->'ids'));\n END IF;\n\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n where_segments := where_segments || format('collection = ANY (%L) ', collections);\n END IF;\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n\n where_segments := where_segments || format(' datetime <= %L::timestamptz AND end_datetime >= %L::timestamptz ',\n edate,\n sdate\n );\n END IF;\n\n IF j ? 'q' THEN\n ft_query := q_to_tsquery(j->'q');\n where_segments := where_segments || format(\n $quote$\n (\n to_tsvector('english', content->'properties'->>'description') ||\n to_tsvector('english', coalesce(content->'properties'->>'title', '')) ||\n to_tsvector('english', coalesce(content->'properties'->>'keywords', ''))\n ) @@ %L\n $quote$,\n ft_query\n );\n END IF;\n\n geom := stac_geom(j);\n IF geom IS NOT NULL THEN\n where_segments := where_segments || format('st_intersects(geometry, %L)',geom);\n END IF;\n\n filterlang := COALESCE(\n j->>'filter-lang',\n get_setting('default_filter_lang', j->'conf')\n );\n IF NOT filter @? '$.**.op' THEN\n filterlang := 'cql-json';\n END IF;\n\n IF filterlang NOT IN ('cql-json','cql2-json') AND j ? 'filter' THEN\n RAISE EXCEPTION '% is not a supported filter-lang. Please use cql-json or cql2-json.', filterlang;\n END IF;\n\n IF j ? 'query' AND j ? 'filter' THEN\n RAISE EXCEPTION 'Can only use either query or filter at one time.';\n END IF;\n\n IF j ? 'query' THEN\n filter := query_to_cql2(j->'query');\n ELSIF filterlang = 'cql-json' THEN\n filter := cql1_to_cql2(filter);\n END IF;\n RAISE NOTICE 'FILTER: %', filter;\n where_segments := where_segments || cql2_query(filter);\n IF cardinality(where_segments) < 1 THEN\n RETURN ' TRUE ';\n END IF;\n\n _where := array_to_string(array_remove(where_segments, NULL), ' AND ');\n\n IF _where IS NULL OR BTRIM(_where) = '' THEN\n RETURN ' TRUE ';\n END IF;\n RETURN _where;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION parse_sort_dir(_dir text, reverse boolean default false) RETURNS text AS $$\n WITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n ) SELECT\n CASE\n WHEN NOT reverse THEN d\n WHEN d = 'ASC' THEN 'DESC'\n WHEN d = 'DESC' THEN 'ASC'\n END\n FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION sort_dir_to_op(_dir text, prev boolean default false) RETURNS text AS $$\n WITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n ) SELECT\n CASE\n WHEN d = 'ASC' AND prev THEN '<='\n WHEN d = 'DESC' AND prev THEN '>='\n WHEN d = 'ASC' THEN '>='\n WHEN d = 'DESC' THEN '<='\n END\n FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION sort_sqlorderby(\n _search jsonb DEFAULT NULL,\n reverse boolean DEFAULT FALSE\n) RETURNS text AS $$\n WITH sortby AS (\n SELECT coalesce(_search->'sortby','[{\"field\":\"datetime\", \"direction\":\"desc\"}]') as sort\n ), withid AS (\n SELECT CASE\n WHEN sort @? '$[*] ? (@.field == \"id\")' THEN sort\n ELSE sort || '[{\"field\":\"id\", \"direction\":\"desc\"}]'::jsonb\n END as sort\n FROM sortby\n ), withid_rows AS (\n SELECT jsonb_array_elements(sort) as value FROM withid\n ),sorts AS (\n SELECT\n coalesce(\n (queryable(value->>'field')).expression\n ) as key,\n parse_sort_dir(value->>'direction', reverse) as dir\n FROM withid_rows\n )\n SELECT array_to_string(\n array_agg(concat(key, ' ', dir)),\n ', '\n ) FROM sorts;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_sort_dir(sort_item jsonb) RETURNS text AS $$\n SELECT CASE WHEN sort_item->>'direction' ILIKE 'desc%' THEN 'DESC' ELSE 'ASC' END;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION get_token_val_str(\n _field text,\n _item items\n) RETURNS text AS $$\nDECLARE\n q text;\n literal text;\nBEGIN\n q := format($q$ SELECT quote_literal(%s) FROM (SELECT $1.*) as r;$q$, _field);\n EXECUTE q INTO literal USING _item;\n RETURN literal;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n\nCREATE OR REPLACE FUNCTION get_token_record(IN _token text, OUT prev BOOLEAN, OUT item items) RETURNS RECORD AS $$\nDECLARE\n _itemid text := _token;\n _collectionid text;\nBEGIN\n IF _token IS NULL THEN\n RETURN;\n END IF;\n RAISE NOTICE 'Looking for token: %', _token;\n prev := FALSE;\n IF _token ILIKE 'prev:%' THEN\n _itemid := replace(_token, 'prev:','');\n prev := TRUE;\n ELSIF _token ILIKE 'next:%' THEN\n _itemid := replace(_token, 'next:', '');\n END IF;\n SELECT id INTO _collectionid FROM collections WHERE _itemid LIKE concat(id,':%');\n IF FOUND THEN\n _itemid := replace(_itemid, concat(_collectionid,':'), '');\n SELECT * INTO item FROM items WHERE id=_itemid AND collection=_collectionid;\n ELSE\n SELECT * INTO item FROM items WHERE id=_itemid;\n END IF;\n IF item IS NULL THEN\n RAISE EXCEPTION 'Could not find item using token: % item: % collection: %', _token, _itemid, _collectionid;\n END IF;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\n\nCREATE OR REPLACE FUNCTION get_token_filter(\n _sortby jsonb DEFAULT '[{\"field\":\"datetime\",\"direction\":\"desc\"}]'::jsonb,\n token_item items DEFAULT NULL,\n prev boolean DEFAULT FALSE,\n inclusive boolean DEFAULT FALSE\n) RETURNS text AS $$\nDECLARE\n ltop text := '<';\n gtop text := '>';\n dir text;\n sort record;\n orfilter text := '';\n orfilters text[] := '{}'::text[];\n andfilters text[] := '{}'::text[];\n output text;\n token_where text;\nBEGIN\n IF _sortby IS NULL OR _sortby = '[]'::jsonb THEN\n _sortby := '[{\"field\":\"datetime\",\"direction\":\"desc\"}]'::jsonb;\n END IF;\n _sortby := _sortby || jsonb_build_object('field','id','direction',_sortby->0->>'direction');\n RAISE NOTICE 'Getting Token Filter. % %', _sortby, token_item;\n IF inclusive THEN\n orfilters := orfilters || format('( id=%L AND collection=%L )' , token_item.id, token_item.collection);\n END IF;\n\n FOR sort IN\n WITH s1 AS (\n SELECT\n _row,\n (queryable(value->>'field')).expression as _field,\n (value->>'field' = 'id') as _isid,\n get_sort_dir(value) as _dir\n FROM jsonb_array_elements(_sortby)\n WITH ORDINALITY AS t(value, _row)\n )\n SELECT\n _row,\n _field,\n _dir,\n get_token_val_str(_field, token_item) as _val\n FROM s1\n WHERE _row <= (SELECT min(_row) FROM s1 WHERE _isid)\n LOOP\n orfilter := NULL;\n RAISE NOTICE 'SORT: %', sort;\n IF sort._val IS NOT NULL AND ((prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC')) THEN\n orfilter := format($f$(\n (%s %s %s) OR (%s IS NULL)\n )$f$,\n sort._field,\n ltop,\n sort._val,\n sort._val\n );\n ELSIF sort._val IS NULL AND ((prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC')) THEN\n RAISE NOTICE '< but null';\n orfilter := format('%s IS NOT NULL', sort._field);\n ELSIF sort._val IS NULL THEN\n RAISE NOTICE '> but null';\n ELSE\n orfilter := format($f$(\n (%s %s %s) OR (%s IS NULL)\n )$f$,\n sort._field,\n gtop,\n sort._val,\n sort._field\n );\n END IF;\n RAISE NOTICE 'ORFILTER: %', orfilter;\n\n IF orfilter IS NOT NULL THEN\n IF sort._row = 1 THEN\n orfilters := orfilters || orfilter;\n ELSE\n orfilters := orfilters || format('(%s AND %s)', array_to_string(andfilters, ' AND '), orfilter);\n END IF;\n END IF;\n IF sort._val IS NOT NULL THEN\n andfilters := andfilters || format('%s = %s', sort._field, sort._val);\n ELSE\n andfilters := andfilters || format('%s IS NULL', sort._field);\n END IF;\n END LOOP;\n\n output := array_to_string(orfilters, ' OR ');\n\n token_where := concat('(',coalesce(output,'true'),')');\n IF trim(token_where) = '' THEN\n token_where := NULL;\n END IF;\n RAISE NOTICE 'TOKEN_WHERE: %',token_where;\n RETURN token_where;\n END;\n$$ LANGUAGE PLPGSQL SET transform_null_equals TO TRUE\n;\n\nCREATE OR REPLACE FUNCTION search_tohash(jsonb) RETURNS jsonb AS $$\n SELECT $1 - '{token,limit,context,includes,excludes}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION search_hash(jsonb, jsonb) RETURNS text AS $$\n SELECT md5(concat(search_tohash($1)::text,$2::text));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TABLE IF NOT EXISTS searches(\n hash text GENERATED ALWAYS AS (search_hash(search, metadata)) STORED PRIMARY KEY,\n search jsonb NOT NULL,\n _where text,\n orderby text,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n metadata jsonb DEFAULT '{}'::jsonb NOT NULL\n);\n\nCREATE TABLE IF NOT EXISTS search_wheres(\n id bigint generated always as identity primary key,\n _where text NOT NULL,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n statslastupdated timestamptz,\n estimated_count bigint,\n estimated_cost float,\n time_to_estimate float,\n total_count bigint,\n time_to_count float,\n partitions text[]\n);\n\nCREATE INDEX IF NOT EXISTS search_wheres_partitions ON search_wheres USING GIN (partitions);\nCREATE UNIQUE INDEX IF NOT EXISTS search_wheres_where ON search_wheres ((md5(_where)));\n\nCREATE OR REPLACE FUNCTION where_stats(\n inwhere text,\n updatestats boolean default false,\n conf jsonb default null\n) RETURNS search_wheres AS $$\nDECLARE\n t timestamptz;\n i interval;\n explain_json jsonb;\n partitions text[];\n sw search_wheres%ROWTYPE;\n inwhere_hash text := md5(inwhere);\n _context text := lower(context(conf));\n _stats_ttl interval := context_stats_ttl(conf);\n _estimated_cost_threshold float := context_estimated_cost(conf);\n _estimated_count_threshold int := context_estimated_count(conf);\n ro bool := pgstac.readonly(conf);\nBEGIN\n -- If updatestats is true then set ttl to 0\n IF updatestats THEN\n RAISE DEBUG 'Updatestats set to TRUE, setting TTL to 0';\n _stats_ttl := '0'::interval;\n END IF;\n\n -- If we don't need to calculate context, just return\n IF _context = 'off' THEN\n sw._where = inwhere;\n RETURN sw;\n END IF;\n\n -- Get any stats that we have.\n IF NOT ro THEN\n -- If there is a lock where another process is\n -- updating the stats, wait so that we don't end up calculating a bunch of times.\n SELECT * INTO sw FROM search_wheres WHERE md5(_where)=inwhere_hash FOR UPDATE;\n ELSE\n SELECT * INTO sw FROM search_wheres WHERE md5(_where)=inwhere_hash;\n END IF;\n\n -- If there is a cached row, figure out if we need to update\n IF\n sw IS NOT NULL\n AND sw.statslastupdated IS NOT NULL\n AND sw.total_count IS NOT NULL\n AND now() - sw.statslastupdated <= _stats_ttl\n THEN\n -- we have a cached row with data that is within our ttl\n RAISE DEBUG 'Stats present in table and lastupdated within ttl: %', sw;\n IF NOT ro THEN\n RAISE DEBUG 'Updating search_wheres only bumping lastused and usecount';\n UPDATE search_wheres SET\n lastused = now(),\n usecount = search_wheres.usecount + 1\n WHERE md5(_where) = inwhere_hash\n RETURNING * INTO sw;\n END IF;\n RAISE DEBUG 'Returning cached counts. %', sw;\n RETURN sw;\n END IF;\n\n -- Calculate estimated cost and rows\n -- Use explain to get estimated count/cost\n IF sw.estimated_count IS NULL OR sw.estimated_cost IS NULL THEN\n RAISE DEBUG 'Calculating estimated stats';\n t := clock_timestamp();\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s', inwhere)\n INTO explain_json;\n RAISE DEBUG 'Time for just the explain: %', clock_timestamp() - t;\n i := clock_timestamp() - t;\n\n sw.estimated_count := explain_json->0->'Plan'->'Plan Rows';\n sw.estimated_cost := explain_json->0->'Plan'->'Total Cost';\n sw.time_to_estimate := extract(epoch from i);\n END IF;\n\n RAISE DEBUG 'ESTIMATED_COUNT: %, THRESHOLD %', sw.estimated_count, _estimated_count_threshold;\n RAISE DEBUG 'ESTIMATED_COST: %, THRESHOLD %', sw.estimated_cost, _estimated_cost_threshold;\n\n -- If context is set to auto and the costs are within the threshold return the estimated costs\n IF\n _context = 'auto'\n AND sw.estimated_count >= _estimated_count_threshold\n AND sw.estimated_cost >= _estimated_cost_threshold\n THEN\n IF NOT ro THEN\n INSERT INTO search_wheres (\n _where,\n lastused,\n usecount,\n statslastupdated,\n estimated_count,\n estimated_cost,\n time_to_estimate,\n total_count,\n time_to_count\n ) VALUES (\n inwhere,\n now(),\n 1,\n now(),\n sw.estimated_count,\n sw.estimated_cost,\n sw.time_to_estimate,\n null,\n null\n ) ON CONFLICT ((md5(_where)))\n DO UPDATE SET\n lastused = EXCLUDED.lastused,\n usecount = search_wheres.usecount + 1,\n statslastupdated = EXCLUDED.statslastupdated,\n estimated_count = EXCLUDED.estimated_count,\n estimated_cost = EXCLUDED.estimated_cost,\n time_to_estimate = EXCLUDED.time_to_estimate,\n total_count = EXCLUDED.total_count,\n time_to_count = EXCLUDED.time_to_count\n RETURNING * INTO sw;\n END IF;\n RAISE DEBUG 'Estimates are within thresholds, returning estimates. %', sw;\n RETURN sw;\n END IF;\n\n -- Calculate Actual Count\n t := clock_timestamp();\n RAISE NOTICE 'Calculating actual count...';\n EXECUTE format(\n 'SELECT count(*) FROM items WHERE %s',\n inwhere\n ) INTO sw.total_count;\n i := clock_timestamp() - t;\n RAISE NOTICE 'Actual Count: % -- %', sw.total_count, i;\n sw.time_to_count := extract(epoch FROM i);\n\n IF NOT ro THEN\n INSERT INTO search_wheres (\n _where,\n lastused,\n usecount,\n statslastupdated,\n estimated_count,\n estimated_cost,\n time_to_estimate,\n total_count,\n time_to_count\n ) VALUES (\n inwhere,\n now(),\n 1,\n now(),\n sw.estimated_count,\n sw.estimated_cost,\n sw.time_to_estimate,\n sw.total_count,\n sw.time_to_count\n ) ON CONFLICT ((md5(_where)))\n DO UPDATE SET\n lastused = EXCLUDED.lastused,\n usecount = search_wheres.usecount + 1,\n statslastupdated = EXCLUDED.statslastupdated,\n estimated_count = EXCLUDED.estimated_count,\n estimated_cost = EXCLUDED.estimated_cost,\n time_to_estimate = EXCLUDED.time_to_estimate,\n total_count = EXCLUDED.total_count,\n time_to_count = EXCLUDED.time_to_count\n RETURNING * INTO sw;\n END IF;\n RAISE DEBUG 'Returning with actual count. %', sw;\n RETURN sw;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION search_query(\n _search jsonb = '{}'::jsonb,\n updatestats boolean = false,\n _metadata jsonb = '{}'::jsonb\n) RETURNS searches AS $$\nDECLARE\n search searches%ROWTYPE;\n cached_search searches%ROWTYPE;\n pexplain jsonb;\n t timestamptz;\n i interval;\n doupdate boolean := FALSE;\n insertfound boolean := FALSE;\n ro boolean := pgstac.readonly();\n found_search text;\nBEGIN\n RAISE NOTICE 'SEARCH: %', _search;\n -- Calculate hash, where clause, and order by statement\n search.search := _search;\n search.metadata := _metadata;\n search.hash := search_hash(_search, _metadata);\n search._where := stac_search_to_where(_search);\n search.orderby := sort_sqlorderby(_search);\n search.lastused := now();\n search.usecount := 1;\n\n -- If we are in read only mode, directly return search\n IF ro THEN\n RETURN search;\n END IF;\n\n RAISE NOTICE 'Updating Statistics for search: %s', search;\n -- Update statistics for times used and and when last used\n -- If the entry is locked, rather than waiting, skip updating the stats\n INSERT INTO searches (search, lastused, usecount, metadata)\n VALUES (search.search, now(), 1, search.metadata)\n ON CONFLICT DO NOTHING\n RETURNING * INTO cached_search\n ;\n\n IF NOT FOUND OR cached_search IS NULL THEN\n UPDATE searches SET\n lastused = now(),\n usecount = searches.usecount + 1\n WHERE hash = (\n SELECT hash FROM searches WHERE hash=search.hash FOR UPDATE SKIP LOCKED\n )\n RETURNING * INTO cached_search\n ;\n END IF;\n\n IF cached_search IS NOT NULL THEN\n cached_search._where = search._where;\n cached_search.orderby = search.orderby;\n RETURN cached_search;\n END IF;\n RETURN search;\n\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE OR REPLACE FUNCTION search_fromhash(\n _hash text\n) RETURNS searches AS $$\n SELECT * FROM search_query((SELECT search FROM searches WHERE hash=_hash LIMIT 1));\n$$ LANGUAGE SQL STRICT;\n\nCREATE OR REPLACE FUNCTION search_rows(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN partitions text[] DEFAULT NULL,\n IN _limit int DEFAULT 10\n) RETURNS SETOF items AS $$\nDECLARE\n base_query text;\n query text;\n sdate timestamptz;\n edate timestamptz;\n n int;\n records_left int := _limit;\n timer timestamptz := clock_timestamp();\n full_timer timestamptz := clock_timestamp();\nBEGIN\nIF _where IS NULL OR trim(_where) = '' THEN\n _where = ' TRUE ';\nEND IF;\nRAISE NOTICE 'Getting chunks for % %', _where, _orderby;\n\nbase_query := $q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n LIMIT %L\n$q$;\n\nIF _orderby ILIKE 'datetime d%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 DESC LOOP\n RAISE NOTICE 'Running Query for % to %. %', sdate, edate, age_ms(full_timer);\n query := format(\n base_query,\n sdate,\n edate,\n _where,\n _orderby,\n records_left\n );\n RAISE DEBUG 'QUERY: %', query;\n timer := clock_timestamp();\n RETURN QUERY EXECUTE query;\n\n GET DIAGNOSTICS n = ROW_COUNT;\n records_left := records_left - n;\n RAISE NOTICE 'Returned %/% Rows From % to %. % to go. Time: %ms', n, _limit, sdate, edate, records_left, age_ms(timer);\n timer := clock_timestamp();\n IF records_left <= 0 THEN\n RAISE NOTICE 'SEARCH_ROWS TOOK %ms', age_ms(full_timer);\n RETURN;\n END IF;\n END LOOP;\nELSIF _orderby ILIKE 'datetime a%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 ASC LOOP\n RAISE NOTICE 'Running Query for % to %. %', sdate, edate, age_ms(full_timer);\n query := format(\n base_query,\n sdate,\n edate,\n _where,\n _orderby,\n records_left\n );\n RAISE DEBUG 'QUERY: %', query;\n timer := clock_timestamp();\n RETURN QUERY EXECUTE query;\n\n GET DIAGNOSTICS n = ROW_COUNT;\n records_left := records_left - n;\n RAISE NOTICE 'Returned %/% Rows From % to %. % to go. Time: %ms', n, _limit, sdate, edate, records_left, age_ms(timer);\n timer := clock_timestamp();\n IF records_left <= 0 THEN\n RAISE NOTICE 'SEARCH_ROWS TOOK %ms', age_ms(full_timer);\n RETURN;\n END IF;\n END LOOP;\nELSE\n query := format($q$\n SELECT * FROM items\n WHERE %s\n ORDER BY %s\n LIMIT %L\n $q$, _where, _orderby, _limit\n );\n RAISE DEBUG 'QUERY: %', query;\n timer := clock_timestamp();\n RETURN QUERY EXECUTE query;\n RAISE NOTICE 'FULL QUERY TOOK %ms', age_ms(timer);\nEND IF;\nRAISE NOTICE 'SEARCH_ROWS TOOK %ms', age_ms(full_timer);\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE UNLOGGED TABLE format_item_cache(\n id text,\n collection text,\n fields text,\n hydrated bool,\n output jsonb,\n lastused timestamptz DEFAULT now(),\n usecount int DEFAULT 1,\n timetoformat float,\n PRIMARY KEY (collection, id, fields, hydrated)\n);\nCREATE INDEX ON format_item_cache (lastused);\n\nCREATE OR REPLACE FUNCTION format_item(_item items, _fields jsonb DEFAULT '{}', _hydrated bool DEFAULT TRUE) RETURNS jsonb AS $$\nDECLARE\n cache bool := get_setting_bool('format_cache');\n _output jsonb := null;\n t timestamptz := clock_timestamp();\nBEGIN\n IF cache THEN\n SELECT output INTO _output FROM format_item_cache\n WHERE id=_item.id AND collection=_item.collection AND fields=_fields::text AND hydrated=_hydrated;\n END IF;\n IF _output IS NULL THEN\n IF _hydrated THEN\n _output := content_hydrate(_item, _fields);\n ELSE\n _output := content_nonhydrated(_item, _fields);\n END IF;\n END IF;\n IF cache THEN\n INSERT INTO format_item_cache (id, collection, fields, hydrated, output, timetoformat)\n VALUES (_item.id, _item.collection, _fields::text, _hydrated, _output, age_ms(t))\n ON CONFLICT(collection, id, fields, hydrated) DO\n UPDATE\n SET lastused=now(), usecount = format_item_cache.usecount + 1\n ;\n END IF;\n RETURN _output;\n\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION search(_search jsonb = '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n searches searches%ROWTYPE;\n _where text;\n orderby text;\n search_where search_wheres%ROWTYPE;\n total_count bigint;\n token record;\n token_prev boolean;\n token_item items%ROWTYPE;\n token_where text;\n full_where text;\n init_ts timestamptz := clock_timestamp();\n timer timestamptz := clock_timestamp();\n hydrate bool := NOT (_search->'conf'->>'nohydrate' IS NOT NULL AND (_search->'conf'->>'nohydrate')::boolean = true);\n prev text;\n next text;\n context jsonb;\n collection jsonb;\n out_records jsonb;\n out_len int;\n _limit int := coalesce((_search->>'limit')::int, 10);\n _querylimit int;\n _fields jsonb := coalesce(_search->'fields', '{}'::jsonb);\n has_prev boolean := FALSE;\n has_next boolean := FALSE;\n links jsonb := '[]'::jsonb;\n base_url text:= concat(rtrim(base_url(_search->'conf'),'/'));\nBEGIN\n searches := search_query(_search);\n _where := searches._where;\n orderby := searches.orderby;\n search_where := where_stats(_where);\n total_count := coalesce(search_where.total_count, search_where.estimated_count);\n RAISE NOTICE 'SEARCH:TOKEN: %', _search->>'token';\n token := get_token_record(_search->>'token');\n RAISE NOTICE '***TOKEN: %', token;\n _querylimit := _limit + 1;\n IF token IS NOT NULL THEN\n token_prev := token.prev;\n token_item := token.item;\n token_where := get_token_filter(_search->'sortby', token_item, token_prev, FALSE);\n RAISE DEBUG 'TOKEN_WHERE: % (%ms from search start)', token_where, age_ms(timer);\n IF token_prev THEN -- if we are using a prev token, we know has_next is true\n RAISE DEBUG 'There is a previous token, so automatically setting has_next to true';\n has_next := TRUE;\n orderby := sort_sqlorderby(_search, TRUE);\n ELSE\n RAISE DEBUG 'There is a next token, so automatically setting has_prev to true';\n has_prev := TRUE;\n\n END IF;\n ELSE -- if there was no token, we know there is no prev\n RAISE DEBUG 'There is no token, so we know there is no prev. setting has_prev to false';\n has_prev := FALSE;\n END IF;\n\n full_where := concat_ws(' AND ', _where, token_where);\n RAISE NOTICE 'FULL WHERE CLAUSE: %', full_where;\n RAISE NOTICE 'Time to get counts and build query %', age_ms(timer);\n timer := clock_timestamp();\n\n IF hydrate THEN\n RAISE NOTICE 'Getting hydrated data.';\n ELSE\n RAISE NOTICE 'Getting non-hydrated data.';\n END IF;\n RAISE NOTICE 'CACHE SET TO %', get_setting_bool('format_cache');\n RAISE NOTICE 'Time to set hydration/formatting %', age_ms(timer);\n timer := clock_timestamp();\n SELECT jsonb_agg(format_item(i, _fields, hydrate)) INTO out_records\n FROM search_rows(\n full_where,\n orderby,\n search_where.partitions,\n _querylimit\n ) as i;\n\n RAISE NOTICE 'Time to fetch rows %', age_ms(timer);\n timer := clock_timestamp();\n\n\n IF token_prev THEN\n out_records := flip_jsonb_array(out_records);\n END IF;\n\n RAISE NOTICE 'Query returned % records.', jsonb_array_length(out_records);\n RAISE DEBUG 'TOKEN: % %', token_item.id, token_item.collection;\n RAISE DEBUG 'RECORD_1: % %', out_records->0->>'id', out_records->0->>'collection';\n RAISE DEBUG 'RECORD-1: % %', out_records->-1->>'id', out_records->-1->>'collection';\n\n -- REMOVE records that were from our token\n IF out_records->0->>'id' = token_item.id AND out_records->0->>'collection' = token_item.collection THEN\n out_records := out_records - 0;\n ELSIF out_records->-1->>'id' = token_item.id AND out_records->-1->>'collection' = token_item.collection THEN\n out_records := out_records - -1;\n END IF;\n\n out_len := jsonb_array_length(out_records);\n\n IF out_len = _limit + 1 THEN\n IF token_prev THEN\n has_prev := TRUE;\n out_records := out_records - 0;\n ELSE\n has_next := TRUE;\n out_records := out_records - -1;\n END IF;\n END IF;\n\n\n links := links || jsonb_build_object(\n 'rel', 'root',\n 'type', 'application/json',\n 'href', base_url\n ) || jsonb_build_object(\n 'rel', 'self',\n 'type', 'application/json',\n 'href', concat(base_url, '/search')\n );\n\n IF has_next THEN\n next := concat(out_records->-1->>'collection', ':', out_records->-1->>'id');\n RAISE NOTICE 'HAS NEXT | %', next;\n links := links || jsonb_build_object(\n 'rel', 'next',\n 'type', 'application/geo+json',\n 'method', 'GET',\n 'href', concat(base_url, '/search?token=next:', next)\n );\n END IF;\n\n IF has_prev THEN\n prev := concat(out_records->0->>'collection', ':', out_records->0->>'id');\n RAISE NOTICE 'HAS PREV | %', prev;\n links := links || jsonb_build_object(\n 'rel', 'prev',\n 'type', 'application/geo+json',\n 'method', 'GET',\n 'href', concat(base_url, '/search?token=prev:', prev)\n );\n END IF;\n\n RAISE NOTICE 'Time to get prev/next %', age_ms(timer);\n timer := clock_timestamp();\n\n\n collection := jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb),\n 'links', links\n );\n\n\n\n IF context(_search->'conf') != 'off' THEN\n collection := collection || jsonb_strip_nulls(jsonb_build_object(\n 'numberMatched', total_count,\n 'numberReturned', coalesce(jsonb_array_length(out_records), 0)\n ));\n ELSE\n collection := collection || jsonb_strip_nulls(jsonb_build_object(\n 'numberReturned', coalesce(jsonb_array_length(out_records), 0)\n ));\n END IF;\n\n IF get_setting_bool('timing', _search->'conf') THEN\n collection = collection || jsonb_build_object('timing', age_ms(init_ts));\n END IF;\n\n RAISE NOTICE 'Time to build final json %', age_ms(timer);\n timer := clock_timestamp();\n\n RAISE NOTICE 'Total Time: %', age_ms(current_timestamp);\n RAISE NOTICE 'RETURNING % records. NEXT: %. PREV: %', collection->>'numberReturned', collection->>'next', collection->>'prev';\n RETURN collection;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION search_cursor(_search jsonb = '{}'::jsonb) RETURNS refcursor AS $$\nDECLARE\n curs refcursor;\n searches searches%ROWTYPE;\n _where text;\n _orderby text;\n q text;\n\nBEGIN\n searches := search_query(_search);\n _where := searches._where;\n _orderby := searches.orderby;\n\n OPEN curs FOR\n WITH p AS (\n SELECT * FROM partition_queries(_where, _orderby) p\n )\n SELECT\n CASE WHEN EXISTS (SELECT 1 FROM p) THEN\n (SELECT format($q$\n SELECT * FROM (\n %s\n ) total\n $q$,\n string_agg(\n format($q$ SELECT * FROM ( %s ) AS sub $q$, p),\n '\n UNION ALL\n '\n )\n ))\n ELSE NULL\n END FROM p;\n RETURN curs;\nEND;\n$$ LANGUAGE PLPGSQL;\nCREATE OR REPLACE VIEW collections_asitems AS\nSELECT\n id,\n geometry,\n 'collections' AS collection,\n datetime,\n end_datetime,\n jsonb_build_object(\n 'properties', content - '{links,assets,stac_version,stac_extensions}',\n 'links', content->'links',\n 'assets', content->'assets',\n 'stac_version', content->'stac_version',\n 'stac_extensions', content->'stac_extensions'\n ) AS content,\n content as collectionjson\nFROM collections;\n\n\nCREATE OR REPLACE FUNCTION collection_search_matched(\n IN _search jsonb DEFAULT '{}'::jsonb,\n OUT matched bigint\n) RETURNS bigint AS $$\nDECLARE\n _where text := stac_search_to_where(_search);\nBEGIN\n EXECUTE format(\n $query$\n SELECT\n count(*)\n FROM\n collections_asitems\n WHERE %s\n ;\n $query$,\n _where\n ) INTO matched;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION collection_search_rows(\n _search jsonb DEFAULT '{}'::jsonb\n) RETURNS SETOF jsonb AS $$\nDECLARE\n _where text := stac_search_to_where(_search);\n _limit int := coalesce((_search->>'limit')::int, 10);\n _fields jsonb := coalesce(_search->'fields', '{}'::jsonb);\n _orderby text;\n _offset int := COALESCE((_search->>'offset')::int, 0);\nBEGIN\n _orderby := sort_sqlorderby(\n jsonb_build_object(\n 'sortby',\n coalesce(\n _search->'sortby',\n '[{\"field\": \"id\", \"direction\": \"asc\"}]'::jsonb\n )\n )\n );\n RETURN QUERY EXECUTE format(\n $query$\n SELECT\n jsonb_fields(collectionjson, %L) as c\n FROM\n collections_asitems\n WHERE %s\n ORDER BY %s\n LIMIT %L\n OFFSET %L\n ;\n $query$,\n _fields,\n _where,\n _orderby,\n _limit,\n _offset\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION collection_search(\n _search jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\nDECLARE\n out_records jsonb;\n number_matched bigint := collection_search_matched(_search);\n number_returned bigint;\n _limit int := coalesce((_search->>'limit')::float::int, 10);\n _offset int := coalesce((_search->>'offset')::float::int, 0);\n links jsonb := '[]';\n ret jsonb;\n base_url text:= concat(rtrim(base_url(_search->'conf'),'/'), '/collections');\n prevoffset int;\n nextoffset int;\nBEGIN\n SELECT\n coalesce(jsonb_agg(c), '[]')\n INTO out_records\n FROM collection_search_rows(_search) c;\n\n number_returned := jsonb_array_length(out_records);\n RAISE DEBUG 'nm: %, nr: %, l:%, o:%', number_matched, number_returned, _limit, _offset;\n\n\n\n IF _limit <= number_matched AND number_matched > 0 THEN --need to have paging links\n nextoffset := least(_offset + _limit, number_matched - 1);\n prevoffset := greatest(_offset - _limit, 0);\n\n IF _offset > 0 THEN\n links := links || jsonb_build_object(\n 'rel', 'prev',\n 'type', 'application/json',\n 'method', 'GET' ,\n 'href', base_url,\n 'body', jsonb_build_object('offset', prevoffset),\n 'merge', TRUE\n );\n END IF;\n\n IF (_offset + _limit < number_matched) THEN\n links := links || jsonb_build_object(\n 'rel', 'next',\n 'type', 'application/json',\n 'method', 'GET' ,\n 'href', base_url,\n 'body', jsonb_build_object('offset', nextoffset),\n 'merge', TRUE\n );\n END IF;\n\n END IF;\n\n ret := jsonb_build_object(\n 'collections', out_records,\n 'numberMatched', number_matched,\n 'numberReturned', number_returned,\n 'links', links\n );\n RETURN ret;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION tileenvelope(zoom int, x int, y int) RETURNS geometry AS $$\nWITH t AS (\n SELECT\n 20037508.3427892 as merc_max,\n -20037508.3427892 as merc_min,\n (2 * 20037508.3427892) / (2 ^ zoom) as tile_size\n)\nSELECT st_makeenvelope(\n merc_min + (tile_size * x),\n merc_max - (tile_size * (y + 1)),\n merc_min + (tile_size * (x + 1)),\n merc_max - (tile_size * y),\n 3857\n) FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;DROP FUNCTION IF EXISTS mercgrid;\n\n\nCREATE OR REPLACE FUNCTION ftime() RETURNS interval as $$\nSELECT age(clock_timestamp(), transaction_timestamp());\n$$ LANGUAGE SQL;\nSET SEARCH_PATH to pgstac, public;\n\nDROP FUNCTION IF EXISTS geometrysearch;\nCREATE OR REPLACE FUNCTION geometrysearch(\n IN geom geometry,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE, -- Return as soon as the passed in geometry is full covered\n IN skipcovered boolean DEFAULT TRUE -- Skip any items that would show up completely under the previous items\n) RETURNS jsonb AS $$\nDECLARE\n search searches%ROWTYPE;\n curs refcursor;\n _where text;\n query text;\n iter_record items%ROWTYPE;\n out_records jsonb := '{}'::jsonb[];\n exit_flag boolean := FALSE;\n counter int := 1;\n scancounter int := 1;\n remaining_limit int := _scanlimit;\n tilearea float;\n unionedgeom geometry;\n clippedgeom geometry;\n unionedgeom_area float := 0;\n prev_area float := 0;\n excludes text[];\n includes text[];\n\nBEGIN\n DROP TABLE IF EXISTS pgstac_results;\n CREATE TEMP TABLE pgstac_results (content jsonb) ON COMMIT DROP;\n\n -- If the passed in geometry is not an area set exitwhenfull and skipcovered to false\n IF ST_GeometryType(geom) !~* 'polygon' THEN\n RAISE NOTICE 'GEOMETRY IS NOT AN AREA';\n skipcovered = FALSE;\n exitwhenfull = FALSE;\n END IF;\n\n -- If skipcovered is true then you will always want to exit when the passed in geometry is full\n IF skipcovered THEN\n exitwhenfull := TRUE;\n END IF;\n\n search := search_fromhash(queryhash);\n\n IF search IS NULL THEN\n RAISE EXCEPTION 'Search with Query Hash % Not Found', queryhash;\n END IF;\n\n tilearea := st_area(geom);\n _where := format('%s AND st_intersects(geometry, %L::geometry)', search._where, geom);\n\n\n FOR query IN SELECT * FROM partition_queries(_where, search.orderby) LOOP\n query := format('%s LIMIT %L', query, remaining_limit);\n RAISE NOTICE '%', query;\n OPEN curs FOR EXECUTE query;\n LOOP\n FETCH curs INTO iter_record;\n EXIT WHEN NOT FOUND;\n IF exitwhenfull OR skipcovered THEN -- If we are not using exitwhenfull or skipcovered, we do not need to do expensive geometry operations\n clippedgeom := st_intersection(geom, iter_record.geometry);\n\n IF unionedgeom IS NULL THEN\n unionedgeom := clippedgeom;\n ELSE\n unionedgeom := st_union(unionedgeom, clippedgeom);\n END IF;\n\n unionedgeom_area := st_area(unionedgeom);\n\n IF skipcovered AND prev_area = unionedgeom_area THEN\n scancounter := scancounter + 1;\n CONTINUE;\n END IF;\n\n prev_area := unionedgeom_area;\n\n RAISE NOTICE '% % % %', unionedgeom_area/tilearea, counter, scancounter, ftime();\n END IF;\n RAISE NOTICE '% %', iter_record, content_hydrate(iter_record, fields);\n INSERT INTO pgstac_results (content) VALUES (content_hydrate(iter_record, fields));\n\n IF counter >= _limit\n OR scancounter > _scanlimit\n OR ftime() > _timelimit\n OR (exitwhenfull AND unionedgeom_area >= tilearea)\n THEN\n exit_flag := TRUE;\n EXIT;\n END IF;\n counter := counter + 1;\n scancounter := scancounter + 1;\n\n END LOOP;\n CLOSE curs;\n EXIT WHEN exit_flag;\n remaining_limit := _scanlimit - scancounter;\n END LOOP;\n\n SELECT jsonb_agg(content) INTO out_records FROM pgstac_results WHERE content IS NOT NULL;\n\n RETURN jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb)\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDROP FUNCTION IF EXISTS geojsonsearch;\nCREATE OR REPLACE FUNCTION geojsonsearch(\n IN geojson jsonb,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_geomfromgeojson(geojson),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS xyzsearch;\nCREATE OR REPLACE FUNCTION xyzsearch(\n IN _x int,\n IN _y int,\n IN _z int,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_transform(tileenvelope(_z, _x, _y), 4326),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE PROCEDURE analyze_items() AS $$\nDECLARE\n q text;\n timeout_ts timestamptz;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n SELECT format('ANALYZE (VERBOSE, SKIP_LOCKED) %I;', relname) INTO q\n FROM pg_stat_user_tables\n WHERE relname like '_item%' AND (n_mod_since_analyze>0 OR last_analyze IS NULL) LIMIT 1;\n IF NOT FOUND THEN\n EXIT;\n END IF;\n RAISE NOTICE '%', q;\n EXECUTE q;\n COMMIT;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE PROCEDURE validate_constraints() AS $$\nDECLARE\n q text;\nBEGIN\n FOR q IN\n SELECT\n FORMAT(\n 'ALTER TABLE %I.%I VALIDATE CONSTRAINT %I;',\n nsp.nspname,\n cls.relname,\n con.conname\n )\n\n FROM pg_constraint AS con\n JOIN pg_class AS cls\n ON con.conrelid = cls.oid\n JOIN pg_namespace AS nsp\n ON cls.relnamespace = nsp.oid\n WHERE convalidated = FALSE AND contype in ('c','f')\n AND nsp.nspname = 'pgstac'\n LOOP\n RAISE NOTICE '%', q;\n PERFORM run_or_queue(q);\n COMMIT;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION collection_extent(_collection text, runupdate boolean default false) RETURNS jsonb AS $$\nDECLARE\n geom_extent geometry;\n mind timestamptz;\n maxd timestamptz;\n extent jsonb;\nBEGIN\n IF runupdate THEN\n PERFORM update_partition_stats_q(partition)\n FROM partitions_view WHERE collection=_collection;\n END IF;\n SELECT\n min(lower(dtrange)),\n max(upper(edtrange)),\n st_extent(spatial)\n INTO\n mind,\n maxd,\n geom_extent\n FROM partitions_view\n WHERE collection=_collection;\n\n IF geom_extent IS NOT NULL AND mind IS NOT NULL AND maxd IS NOT NULL THEN\n extent := jsonb_build_object(\n 'spatial', jsonb_build_object(\n 'bbox', to_jsonb(array[array[st_xmin(geom_extent), st_ymin(geom_extent), st_xmax(geom_extent), st_ymax(geom_extent)]])\n ),\n 'temporal', jsonb_build_object(\n 'interval', to_jsonb(array[array[mind, maxd]])\n )\n );\n RETURN extent;\n END IF;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('id', '{\"title\": \"Item ID\",\"description\": \"Item identifier\",\"$ref\": \"https://schemas.stacspec.org/v1.0.0/item-spec/json-schema/item.json#/definitions/core/allOf/2/properties/id\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('geometry', '{\"title\": \"Item Geometry\",\"description\": \"Item Geometry\",\"$ref\": \"https://geojson.org/schema/Feature.json\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('datetime','{\"description\": \"Datetime\",\"type\": \"string\",\"title\": \"Acquired\",\"format\": \"date-time\",\"pattern\": \"(\\\\+00:00|Z)$\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDELETE FROM queryables a USING queryables b\n WHERE a.name = b.name AND a.collection_ids IS NOT DISTINCT FROM b.collection_ids AND a.id > b.id;\n\n\nINSERT INTO pgstac_settings (name, value) VALUES\n ('context', 'off'),\n ('context_estimated_count', '100000'),\n ('context_estimated_cost', '100000'),\n ('context_stats_ttl', '1 day'),\n ('default_filter_lang', 'cql2-json'),\n ('additional_properties', 'true'),\n ('use_queue', 'false'),\n ('queue_timeout', '10 minutes'),\n ('update_collection_extent', 'false'),\n ('format_cache', 'false'),\n ('readonly', 'false')\nON CONFLICT DO NOTHING\n;\n\n\nINSERT INTO cql2_ops (op, template, types) VALUES\n ('eq', '%s = %s', NULL),\n ('neq', '%s != %s', NULL),\n ('ne', '%s != %s', NULL),\n ('!=', '%s != %s', NULL),\n ('<>', '%s != %s', NULL),\n ('lt', '%s < %s', NULL),\n ('lte', '%s <= %s', NULL),\n ('gt', '%s > %s', NULL),\n ('gte', '%s >= %s', NULL),\n ('le', '%s <= %s', NULL),\n ('ge', '%s >= %s', NULL),\n ('=', '%s = %s', NULL),\n ('<', '%s < %s', NULL),\n ('<=', '%s <= %s', NULL),\n ('>', '%s > %s', NULL),\n ('>=', '%s >= %s', NULL),\n ('like', '%s LIKE %s', NULL),\n ('ilike', '%s ILIKE %s', NULL),\n ('+', '%s + %s', NULL),\n ('-', '%s - %s', NULL),\n ('*', '%s * %s', NULL),\n ('/', '%s / %s', NULL),\n ('not', 'NOT (%s)', NULL),\n ('between', '%s BETWEEN %s AND %s', NULL),\n ('isnull', '%s IS NULL', NULL),\n ('upper', 'upper(%s)', NULL),\n ('lower', 'lower(%s)', NULL),\n ('casei', 'upper(%s)', NULL),\n ('accenti', 'unaccent(%s)', NULL)\nON CONFLICT (op) DO UPDATE\n SET\n template = EXCLUDED.template\n;\n\n\nALTER FUNCTION to_text COST 5000;\nALTER FUNCTION to_float COST 5000;\nALTER FUNCTION to_int COST 5000;\nALTER FUNCTION to_tstz COST 5000;\nALTER FUNCTION to_text_array COST 5000;\n\nALTER FUNCTION update_partition_stats SECURITY DEFINER;\nALTER FUNCTION partition_after_triggerfunc SECURITY DEFINER;\nALTER FUNCTION drop_table_constraints SECURITY DEFINER;\nALTER FUNCTION create_table_constraints SECURITY DEFINER;\nALTER FUNCTION check_partition SECURITY DEFINER;\nALTER FUNCTION repartition SECURITY DEFINER;\nALTER FUNCTION where_stats SECURITY DEFINER;\nALTER FUNCTION search_query SECURITY DEFINER;\nALTER FUNCTION format_item SECURITY DEFINER;\nALTER FUNCTION maintain_index SECURITY DEFINER;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nGRANT ALL ON SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac to pgstac_admin;\n\n-- pgstac_read role limited to using function apis\nGRANT EXECUTE ON FUNCTION search TO pgstac_read;\nGRANT EXECUTE ON FUNCTION search_query TO pgstac_read;\nGRANT EXECUTE ON FUNCTION item_by_id TO pgstac_read;\nGRANT EXECUTE ON FUNCTION get_item TO pgstac_read;\nGRANT SELECT ON ALL TABLES IN SCHEMA pgstac TO pgstac_read;\n\n\nGRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_ingest;\nGRANT USAGE ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_ingest;\n\nREVOKE ALL PRIVILEGES ON PROCEDURE run_queued_queries FROM public;\nGRANT ALL ON PROCEDURE run_queued_queries TO pgstac_admin;\n\nREVOKE ALL PRIVILEGES ON FUNCTION run_queued_queries_intransaction FROM public;\nGRANT ALL ON FUNCTION run_queued_queries_intransaction TO pgstac_admin;\n\nRESET ROLE;\n\nSET ROLE pgstac_ingest;\nSELECT update_partition_stats_q(partition) FROM partitions_view;\nSELECT set_version('0.9.8');\n" }, { "path": "src/pgstac/migrations/pgstac.0.9.9-0.9.10.sql", "content": "SET client_min_messages TO WARNING;\nSET SEARCH_PATH to pgstac, public;\nRESET ROLE;\nDO $$\nDECLARE\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='postgis') THEN\n CREATE EXTENSION IF NOT EXISTS postgis;\n END IF;\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='btree_gist') THEN\n CREATE EXTENSION IF NOT EXISTS btree_gist;\n END IF;\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='unaccent') THEN\n CREATE EXTENSION IF NOT EXISTS unaccent;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_admin;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_read;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_ingest;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n\nGRANT pgstac_admin TO current_user;\n\n-- Function to make sure pgstac_admin is the owner of items\nCREATE OR REPLACE FUNCTION pgstac_admin_owns() RETURNS VOID AS $$\nDECLARE\n f RECORD;\nBEGIN\n FOR f IN (\n SELECT\n concat(\n oid::regproc::text,\n '(',\n coalesce(pg_get_function_identity_arguments(oid),''),\n ')'\n ) AS name,\n CASE prokind WHEN 'f' THEN 'FUNCTION' WHEN 'p' THEN 'PROCEDURE' WHEN 'a' THEN 'AGGREGATE' END as typ\n FROM pg_proc\n WHERE\n pronamespace=to_regnamespace('pgstac')\n AND proowner != to_regrole('pgstac_admin')\n AND proname NOT LIKE 'pg_stat%'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n FOR f IN (\n SELECT\n oid::regclass::text as name,\n CASE relkind\n WHEN 'i' THEN 'INDEX'\n WHEN 'I' THEN 'INDEX'\n WHEN 'p' THEN 'TABLE'\n WHEN 'r' THEN 'TABLE'\n WHEN 'v' THEN 'VIEW'\n WHEN 'S' THEN 'SEQUENCE'\n ELSE NULL\n END as typ\n FROM pg_class\n WHERE relnamespace=to_regnamespace('pgstac') and relowner != to_regrole('pgstac_admin') AND relkind IN ('r','p','v','S') AND relname NOT LIKE 'pg_stat'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\nSELECT pgstac_admin_owns();\n\nCREATE SCHEMA IF NOT EXISTS pgstac AUTHORIZATION pgstac_admin;\n\nGRANT ALL ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_admin;\n\nALTER ROLE pgstac_admin SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_read SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_ingest SET SEARCH_PATH TO pgstac, public;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\n\nGRANT pgstac_read TO pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\n\nSET ROLE pgstac_admin;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\nRESET ROLE;\n\nSET ROLE pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\nRESET ROLE;\n\nSET SEARCH_PATH TO pgstac, public;\nSET ROLE pgstac_admin;\n\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS analyze_items;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS validate_constraints;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n-- Install these idempotently as migrations do not put them before trying to modify the collections table\n\n\nCREATE OR REPLACE FUNCTION collection_geom(content jsonb)\nRETURNS geometry AS $$\n WITH box AS (SELECT content->'extent'->'spatial'->'bbox'->0 as box)\n SELECT\n st_makeenvelope(\n (box->>0)::float,\n (box->>1)::float,\n (box->>2)::float,\n (box->>3)::float,\n 4326\n )\n FROM box;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION collection_datetime(content jsonb)\nRETURNS timestamptz AS $$\n SELECT\n CASE\n WHEN\n (content->'extent'->'temporal'->'interval'->0->>0) IS NULL\n THEN '-infinity'::timestamptz\n ELSE\n (content->'extent'->'temporal'->'interval'->0->>0)::timestamptz\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION collection_enddatetime(content jsonb)\nRETURNS timestamptz AS $$\n SELECT\n CASE\n WHEN\n (content->'extent'->'temporal'->'interval'->0->>1) IS NULL\n THEN 'infinity'::timestamptz\n ELSE\n (content->'extent'->'temporal'->'interval'->0->>1)::timestamptz\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n-- BEGIN migra calculated SQL\ndrop materialized view if exists \"pgstac\".\"partition_steps\";\n\ndrop view if exists \"pgstac\".\"partition_sys_meta\";\n\ndrop materialized view if exists \"pgstac\".\"partitions\";\n\ndrop view if exists \"pgstac\".\"partitions_view\";\n\ndrop function if exists \"pgstac\".\"dt_constraint\"(coid oid, OUT dt tstzrange, OUT edt tstzrange);\n\nset check_function_bodies = off;\n\nCREATE OR REPLACE FUNCTION pgstac.get_partition_name(relid regclass)\n RETURNS text\n LANGUAGE sql\n STABLE STRICT\nAS $function$\n SELECT (parse_ident(relid::text))[cardinality(parse_ident(relid::text))];\n$function$\n;\n\nCREATE OR REPLACE FUNCTION pgstac.get_tstz_constraint(reloid oid, colname text)\n RETURNS tstzrange\n LANGUAGE plpgsql\n STABLE STRICT\nAS $function$\nDECLARE\n expr text := NULL;\n m text[];\n ts_lower timestamptz := NULL;\n ts_upper timestamptz := NULL;\n lower_inclusive text := '[';\n upper_inclusive text := ']';\n ts timestamptz;\nBEGIN\n SELECT INTO expr\n string_agg(def, ' AND ')\n FROM pg_constraint JOIN LATERAL pg_get_constraintdef(oid) AS def ON TRUE\n WHERE\n conrelid = reloid\n AND contype = 'c'\n AND def LIKE '%' || colname || '%'\n ;\n\n IF expr IS NULL THEN\n RETURN NULL;\n END IF;\n\n RAISE DEBUG 'Constraint expression for % on %: %', colname, reloid::regclass, expr;\n -- collect all constraints for the specified column\n FOR m IN SELECT regexp_matches(expr, '[ (]' || colname || $expr$\\s*([<>=]{1,2})\\s*'([0-9 :+\\-]+)'$expr$, 'g') LOOP\n ts := m[2]::timestamptz;\n IF m[1] IN ('>', '>=')\n THEN\n IF ts_lower IS NULL OR ts > ts_lower OR (ts = ts_lower AND m[1] = '>') THEN\n ts_lower := ts;\n lower_inclusive := CASE WHEN m[1] = '>' THEN '(' ELSE '[' END;\n END IF;\n ELSIF m[1] IN ('<', '<=')\n THEN\n IF ts_upper IS NULL OR ts < ts_upper OR (ts = ts_upper AND m[1] = '<') THEN\n ts_upper := ts;\n upper_inclusive := CASE WHEN m[1] = '<' THEN ')' ELSE ']' END;\n END IF;\n END IF;\n END LOOP;\n RAISE DEBUG 'Constraint % for %: % %', colname, reloid::regclass, ts_lower, ts_upper;\n RETURN tstzrange(ts_lower, ts_upper, lower_inclusive || upper_inclusive);\nEND;\n$function$\n;\n\ncreate or replace view \"pgstac\".\"partition_sys_meta\" as SELECT partition.partition,\n replace(replace(\n CASE\n WHEN (pg_partition_tree.level = 1) THEN partition_expr.partition_expr\n ELSE parent_partition_expr.parent_partition_expr\n END, 'FOR VALUES IN ('''::text, ''::text), ''')'::text, ''::text) AS collection,\n pg_partition_tree.level,\n c.reltuples,\n c.relhastriggers,\n partition_dtrange.partition_dtrange,\n COALESCE(get_tstz_constraint(c.oid, 'datetime'::text), partition_dtrange.partition_dtrange, inf_range.inf_range) AS constraint_dtrange,\n COALESCE(get_tstz_constraint(c.oid, 'end_datetime'::text), inf_range.inf_range) AS constraint_edtrange\n FROM ((((((((((pg_partition_tree('items'::regclass) pg_partition_tree(relid, parentrelid, isleaf, level)\n JOIN pg_class c ON (((pg_partition_tree.relid)::oid = c.oid)))\n JOIN pg_class parent ON ((((pg_partition_tree.parentrelid)::oid = parent.oid) AND pg_partition_tree.isleaf)))\n LEFT JOIN pg_constraint edt ON (((edt.conrelid = c.oid) AND (edt.contype = 'c'::\"char\"))))\n JOIN LATERAL get_partition_name(pg_partition_tree.relid) partition(partition) ON (true))\n JOIN LATERAL pg_get_expr(c.relpartbound, c.oid) partition_expr(partition_expr) ON (true))\n JOIN LATERAL pg_get_expr(parent.relpartbound, parent.oid) parent_partition_expr(parent_partition_expr) ON (true))\n JOIN LATERAL tstzrange('-infinity'::timestamp with time zone, 'infinity'::timestamp with time zone, '[]'::text) inf_range(inf_range) ON (true))\n JOIN LATERAL COALESCE(constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), inf_range.inf_range) partition_dtrange(partition_dtrange) ON (true))\n JOIN LATERAL get_tstz_constraint(c.oid, 'datetime'::text) datetime_constraint(datetime_constraint) ON (true))\n JOIN LATERAL get_tstz_constraint(c.oid, 'end_datetime'::text) end_datetime_constraint(end_datetime_constraint) ON (true))\n WHERE pg_partition_tree.isleaf;\n\n\ncreate or replace view \"pgstac\".\"partitions_view\" as SELECT (parse_ident((pg_partition_tree.relid)::text))[cardinality(parse_ident((pg_partition_tree.relid)::text))] AS partition,\n replace(replace(\n CASE\n WHEN (pg_partition_tree.level = 1) THEN partition_expr.partition_expr\n ELSE parent_partition_expr.parent_partition_expr\n END, 'FOR VALUES IN ('''::text, ''::text), ''')'::text, ''::text) AS collection,\n pg_partition_tree.level,\n c.reltuples,\n c.relhastriggers,\n partition_dtrange.partition_dtrange,\n COALESCE(get_tstz_constraint(c.oid, 'datetime'::text), partition_dtrange.partition_dtrange, inf_range.inf_range) AS constraint_dtrange,\n COALESCE(get_tstz_constraint(c.oid, 'end_datetime'::text), inf_range.inf_range) AS constraint_edtrange,\n partition_stats.dtrange,\n partition_stats.edtrange,\n partition_stats.spatial,\n partition_stats.last_updated\n FROM (((((((((((pg_partition_tree('items'::regclass) pg_partition_tree(relid, parentrelid, isleaf, level)\n JOIN pg_class c ON (((pg_partition_tree.relid)::oid = c.oid)))\n JOIN pg_class parent ON ((((pg_partition_tree.parentrelid)::oid = parent.oid) AND pg_partition_tree.isleaf)))\n LEFT JOIN pg_constraint edt ON (((edt.conrelid = c.oid) AND (edt.contype = 'c'::\"char\"))))\n JOIN LATERAL get_partition_name(pg_partition_tree.relid) partition(partition) ON (true))\n JOIN LATERAL pg_get_expr(c.relpartbound, c.oid) partition_expr(partition_expr) ON (true))\n JOIN LATERAL pg_get_expr(parent.relpartbound, parent.oid) parent_partition_expr(parent_partition_expr) ON (true))\n JOIN LATERAL tstzrange('-infinity'::timestamp with time zone, 'infinity'::timestamp with time zone, '[]'::text) inf_range(inf_range) ON (true))\n JOIN LATERAL COALESCE(constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), inf_range.inf_range) partition_dtrange(partition_dtrange) ON (true))\n JOIN LATERAL get_tstz_constraint(c.oid, 'datetime'::text) datetime_constraint(datetime_constraint) ON (true))\n JOIN LATERAL get_tstz_constraint(c.oid, 'end_datetime'::text) end_datetime_constraint(end_datetime_constraint) ON (true))\n LEFT JOIN partition_stats USING (partition))\n WHERE pg_partition_tree.isleaf;\n\n\ncreate materialized view \"pgstac\".\"partition_steps\" as SELECT partition AS name,\n date_trunc('month'::text, lower(partition_dtrange)) AS sdate,\n (date_trunc('month'::text, upper(partition_dtrange)) + '1 mon'::interval) AS edate\n FROM partitions_view\n WHERE ((partition_dtrange IS NOT NULL) AND (partition_dtrange <> 'empty'::tstzrange))\n ORDER BY dtrange;\n\n\ncreate materialized view \"pgstac\".\"partitions\" as SELECT partition,\n collection,\n level,\n reltuples,\n relhastriggers,\n partition_dtrange,\n constraint_dtrange,\n constraint_edtrange,\n dtrange,\n edtrange,\n spatial,\n last_updated\n FROM partitions_view;\n\n\n\n-- END migra calculated SQL\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('id', '{\"title\": \"Item ID\",\"description\": \"Item identifier\",\"$ref\": \"https://schemas.stacspec.org/v1.0.0/item-spec/json-schema/item.json#/definitions/core/allOf/2/properties/id\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('geometry', '{\"title\": \"Item Geometry\",\"description\": \"Item Geometry\",\"$ref\": \"https://geojson.org/schema/Feature.json\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('datetime','{\"description\": \"Datetime\",\"type\": \"string\",\"title\": \"Acquired\",\"format\": \"date-time\",\"pattern\": \"(\\\\+00:00|Z)$\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDELETE FROM queryables a USING queryables b\n WHERE a.name = b.name AND a.collection_ids IS NOT DISTINCT FROM b.collection_ids AND a.id > b.id;\n\n\nINSERT INTO pgstac_settings (name, value) VALUES\n ('context', 'off'),\n ('context_estimated_count', '100000'),\n ('context_estimated_cost', '100000'),\n ('context_stats_ttl', '1 day'),\n ('default_filter_lang', 'cql2-json'),\n ('additional_properties', 'true'),\n ('use_queue', 'false'),\n ('queue_timeout', '10 minutes'),\n ('update_collection_extent', 'false'),\n ('format_cache', 'false'),\n ('readonly', 'false')\nON CONFLICT DO NOTHING\n;\n\n\nINSERT INTO cql2_ops (op, template, types) VALUES\n ('eq', '%s = %s', NULL),\n ('neq', '%s != %s', NULL),\n ('ne', '%s != %s', NULL),\n ('!=', '%s != %s', NULL),\n ('<>', '%s != %s', NULL),\n ('lt', '%s < %s', NULL),\n ('lte', '%s <= %s', NULL),\n ('gt', '%s > %s', NULL),\n ('gte', '%s >= %s', NULL),\n ('le', '%s <= %s', NULL),\n ('ge', '%s >= %s', NULL),\n ('=', '%s = %s', NULL),\n ('<', '%s < %s', NULL),\n ('<=', '%s <= %s', NULL),\n ('>', '%s > %s', NULL),\n ('>=', '%s >= %s', NULL),\n ('like', '%s LIKE %s', NULL),\n ('ilike', '%s ILIKE %s', NULL),\n ('+', '%s + %s', NULL),\n ('-', '%s - %s', NULL),\n ('*', '%s * %s', NULL),\n ('/', '%s / %s', NULL),\n ('not', 'NOT (%s)', NULL),\n ('between', '%s BETWEEN %s AND %s', NULL),\n ('isnull', '%s IS NULL', NULL),\n ('upper', 'upper(%s)', NULL),\n ('lower', 'lower(%s)', NULL),\n ('casei', 'upper(%s)', NULL),\n ('accenti', 'unaccent(%s)', NULL)\nON CONFLICT (op) DO UPDATE\n SET\n template = EXCLUDED.template\n;\n\n\nALTER FUNCTION to_text COST 5000;\nALTER FUNCTION to_float COST 5000;\nALTER FUNCTION to_int COST 5000;\nALTER FUNCTION to_tstz COST 5000;\nALTER FUNCTION to_text_array COST 5000;\n\nALTER FUNCTION update_partition_stats SECURITY DEFINER;\nALTER FUNCTION partition_after_triggerfunc SECURITY DEFINER;\nALTER FUNCTION drop_table_constraints SECURITY DEFINER;\nALTER FUNCTION create_table_constraints SECURITY DEFINER;\nALTER FUNCTION check_partition SECURITY DEFINER;\nALTER FUNCTION repartition SECURITY DEFINER;\nALTER FUNCTION where_stats SECURITY DEFINER;\nALTER FUNCTION search_query SECURITY DEFINER;\nALTER FUNCTION format_item SECURITY DEFINER;\nALTER FUNCTION maintain_index SECURITY DEFINER;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nGRANT ALL ON SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac to pgstac_admin;\n\n-- pgstac_read role limited to using function apis\nGRANT EXECUTE ON FUNCTION search TO pgstac_read;\nGRANT EXECUTE ON FUNCTION search_query TO pgstac_read;\nGRANT EXECUTE ON FUNCTION item_by_id TO pgstac_read;\nGRANT EXECUTE ON FUNCTION get_item TO pgstac_read;\nGRANT SELECT ON ALL TABLES IN SCHEMA pgstac TO pgstac_read;\n\n\nGRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_ingest;\nGRANT USAGE ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_ingest;\n\nREVOKE ALL PRIVILEGES ON PROCEDURE run_queued_queries FROM public;\nGRANT ALL ON PROCEDURE run_queued_queries TO pgstac_admin;\n\nREVOKE ALL PRIVILEGES ON FUNCTION run_queued_queries_intransaction FROM public;\nGRANT ALL ON FUNCTION run_queued_queries_intransaction TO pgstac_admin;\n\nRESET ROLE;\n\nSET ROLE pgstac_ingest;\nSELECT update_partition_stats_q(partition) FROM partitions_view;\nSELECT set_version('0.9.10');\n" }, { "path": "src/pgstac/migrations/pgstac.0.9.9.sql", "content": "RESET ROLE;\nDO $$\nDECLARE\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='postgis') THEN\n CREATE EXTENSION IF NOT EXISTS postgis;\n END IF;\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='btree_gist') THEN\n CREATE EXTENSION IF NOT EXISTS btree_gist;\n END IF;\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='unaccent') THEN\n CREATE EXTENSION IF NOT EXISTS unaccent;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_admin;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_read;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_ingest;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n\nGRANT pgstac_admin TO current_user;\n\n-- Function to make sure pgstac_admin is the owner of items\nCREATE OR REPLACE FUNCTION pgstac_admin_owns() RETURNS VOID AS $$\nDECLARE\n f RECORD;\nBEGIN\n FOR f IN (\n SELECT\n concat(\n oid::regproc::text,\n '(',\n coalesce(pg_get_function_identity_arguments(oid),''),\n ')'\n ) AS name,\n CASE prokind WHEN 'f' THEN 'FUNCTION' WHEN 'p' THEN 'PROCEDURE' WHEN 'a' THEN 'AGGREGATE' END as typ\n FROM pg_proc\n WHERE\n pronamespace=to_regnamespace('pgstac')\n AND proowner != to_regrole('pgstac_admin')\n AND proname NOT LIKE 'pg_stat%'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n FOR f IN (\n SELECT\n oid::regclass::text as name,\n CASE relkind\n WHEN 'i' THEN 'INDEX'\n WHEN 'I' THEN 'INDEX'\n WHEN 'p' THEN 'TABLE'\n WHEN 'r' THEN 'TABLE'\n WHEN 'v' THEN 'VIEW'\n WHEN 'S' THEN 'SEQUENCE'\n ELSE NULL\n END as typ\n FROM pg_class\n WHERE relnamespace=to_regnamespace('pgstac') and relowner != to_regrole('pgstac_admin') AND relkind IN ('r','p','v','S') AND relname NOT LIKE 'pg_stat'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\nSELECT pgstac_admin_owns();\n\nCREATE SCHEMA IF NOT EXISTS pgstac AUTHORIZATION pgstac_admin;\n\nGRANT ALL ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_admin;\n\nALTER ROLE pgstac_admin SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_read SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_ingest SET SEARCH_PATH TO pgstac, public;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\n\nGRANT pgstac_read TO pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\n\nSET ROLE pgstac_admin;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\nRESET ROLE;\n\nSET ROLE pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\nRESET ROLE;\n\nSET SEARCH_PATH TO pgstac, public;\nSET ROLE pgstac_admin;\n\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS analyze_items;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS validate_constraints;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n-- Install these idempotently as migrations do not put them before trying to modify the collections table\n\n\nCREATE OR REPLACE FUNCTION collection_geom(content jsonb)\nRETURNS geometry AS $$\n WITH box AS (SELECT content->'extent'->'spatial'->'bbox'->0 as box)\n SELECT\n st_makeenvelope(\n (box->>0)::float,\n (box->>1)::float,\n (box->>2)::float,\n (box->>3)::float,\n 4326\n )\n FROM box;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION collection_datetime(content jsonb)\nRETURNS timestamptz AS $$\n SELECT\n CASE\n WHEN\n (content->'extent'->'temporal'->'interval'->0->>0) IS NULL\n THEN '-infinity'::timestamptz\n ELSE\n (content->'extent'->'temporal'->'interval'->0->>0)::timestamptz\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION collection_enddatetime(content jsonb)\nRETURNS timestamptz AS $$\n SELECT\n CASE\n WHEN\n (content->'extent'->'temporal'->'interval'->0->>1) IS NULL\n THEN 'infinity'::timestamptz\n ELSE\n (content->'extent'->'temporal'->'interval'->0->>1)::timestamptz\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE TABLE IF NOT EXISTS migrations (\n version text PRIMARY KEY,\n datetime timestamptz DEFAULT clock_timestamp() NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION get_version() RETURNS text AS $$\n SELECT version FROM pgstac.migrations ORDER BY datetime DESC, version DESC LIMIT 1;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION set_version(text) RETURNS text AS $$\n INSERT INTO pgstac.migrations (version) VALUES ($1)\n ON CONFLICT DO NOTHING\n RETURNING version;\n$$ LANGUAGE SQL;\n\n\nCREATE TABLE IF NOT EXISTS pgstac_settings (\n name text PRIMARY KEY,\n value text NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION table_empty(text) RETURNS boolean AS $$\nDECLARE\n retval boolean;\nBEGIN\n EXECUTE format($q$\n SELECT NOT EXISTS (SELECT 1 FROM %I LIMIT 1)\n $q$,\n $1\n ) INTO retval;\n RETURN retval;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION get_setting(IN _setting text, IN conf jsonb DEFAULT NULL) RETURNS text AS $$\nSELECT COALESCE(\n nullif(conf->>_setting, ''),\n nullif(current_setting(concat('pgstac.',_setting), TRUE),''),\n nullif((SELECT value FROM pgstac.pgstac_settings WHERE name=_setting),'')\n);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_setting_bool(IN _setting text, IN conf jsonb DEFAULT NULL) RETURNS boolean AS $$\nSELECT COALESCE(\n nullif(conf->>_setting, ''),\n nullif(current_setting(concat('pgstac.',_setting), TRUE),''),\n nullif((SELECT value FROM pgstac.pgstac_settings WHERE name=_setting),''),\n 'FALSE'\n)::boolean;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION base_url(conf jsonb DEFAULT NULL) RETURNS text AS $$\n SELECT COALESCE(pgstac.get_setting('base_url', conf), '.');\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION additional_properties() RETURNS boolean AS $$\n SELECT pgstac.get_setting_bool('additional_properties');\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION readonly(conf jsonb DEFAULT NULL) RETURNS boolean AS $$\n SELECT pgstac.get_setting_bool('readonly', conf);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context(conf jsonb DEFAULT NULL) RETURNS text AS $$\n SELECT pgstac.get_setting('context', conf);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context_estimated_count(conf jsonb DEFAULT NULL) RETURNS int AS $$\n SELECT pgstac.get_setting('context_estimated_count', conf)::int;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_estimated_cost();\nCREATE OR REPLACE FUNCTION context_estimated_cost(conf jsonb DEFAULT NULL) RETURNS float AS $$\n SELECT pgstac.get_setting('context_estimated_cost', conf)::float;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_stats_ttl();\nCREATE OR REPLACE FUNCTION context_stats_ttl(conf jsonb DEFAULT NULL) RETURNS interval AS $$\n SELECT pgstac.get_setting('context_stats_ttl', conf)::interval;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION t2s(text) RETURNS text AS $$\n SELECT extract(epoch FROM $1::interval)::text || ' s';\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE STRICT;\n\nCREATE OR REPLACE FUNCTION age_ms(a timestamptz, b timestamptz DEFAULT clock_timestamp()) RETURNS float AS $$\n SELECT abs(extract(epoch from age(a,b)) * 1000);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION queue_timeout() RETURNS interval AS $$\n SELECT t2s(coalesce(\n get_setting('queue_timeout'),\n '1h'\n ))::interval;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION notice(VARIADIC text[]) RETURNS boolean AS $$\nDECLARE\ndebug boolean := current_setting('pgstac.debug', true);\nBEGIN\n IF debug THEN\n RAISE NOTICE 'NOTICE FROM FUNC: % >>>>> %', concat_ws(' | ', $1), clock_timestamp();\n RETURN TRUE;\n END IF;\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION empty_arr(ANYARRAY) RETURNS BOOLEAN AS $$\nSELECT CASE\n WHEN $1 IS NULL THEN TRUE\n WHEN cardinality($1)<1 THEN TRUE\nELSE FALSE\nEND;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_intersection(_a ANYARRAY, _b ANYARRAY) RETURNS ANYARRAY AS $$\n SELECT ARRAY ( SELECT unnest(_a) INTERSECT SELECT UNNEST(_b) );\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION array_map_ident(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_ident(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_map_literal(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_literal(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION array_reverse(anyarray) RETURNS anyarray AS $$\nSELECT ARRAY(\n SELECT $1[i]\n FROM generate_subscripts($1,1) AS s(i)\n ORDER BY i DESC\n);\n$$ LANGUAGE SQL STRICT IMMUTABLE;\n\nDROP TABLE IF EXISTS query_queue;\nCREATE TABLE query_queue (\n query text PRIMARY KEY,\n added timestamptz DEFAULT now()\n);\n\nDROP TABLE IF EXISTS query_queue_history;\nCREATE TABLE query_queue_history(\n query text,\n added timestamptz NOT NULL,\n finished timestamptz NOT NULL DEFAULT now(),\n error text\n);\n\nCREATE OR REPLACE PROCEDURE run_queued_queries() AS $$\nDECLARE\n qitem query_queue%ROWTYPE;\n timeout_ts timestamptz;\n error text;\n cnt int := 0;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n DELETE FROM query_queue WHERE query = (SELECT query FROM query_queue ORDER BY added DESC LIMIT 1 FOR UPDATE SKIP LOCKED) RETURNING * INTO qitem;\n IF NOT FOUND THEN\n EXIT;\n END IF;\n cnt := cnt + 1;\n BEGIN\n RAISE NOTICE 'RUNNING QUERY: %', qitem.query;\n EXECUTE qitem.query;\n EXCEPTION WHEN others THEN\n error := format('%s | %s', SQLERRM, SQLSTATE);\n END;\n INSERT INTO query_queue_history (query, added, finished, error)\n VALUES (qitem.query, qitem.added, clock_timestamp(), error);\n COMMIT;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION run_queued_queries_intransaction() RETURNS int AS $$\nDECLARE\n qitem query_queue%ROWTYPE;\n timeout_ts timestamptz;\n error text;\n cnt int := 0;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n DELETE FROM query_queue WHERE query = (SELECT query FROM query_queue ORDER BY added DESC LIMIT 1 FOR UPDATE SKIP LOCKED) RETURNING * INTO qitem;\n IF NOT FOUND THEN\n RETURN cnt;\n END IF;\n cnt := cnt + 1;\n BEGIN\n qitem.query := regexp_replace(qitem.query, 'CONCURRENTLY', '');\n RAISE NOTICE 'RUNNING QUERY: %', qitem.query;\n\n EXECUTE qitem.query;\n EXCEPTION WHEN others THEN\n error := format('%s | %s', SQLERRM, SQLSTATE);\n RAISE WARNING '%', error;\n END;\n INSERT INTO query_queue_history (query, added, finished, error)\n VALUES (qitem.query, qitem.added, clock_timestamp(), error);\n END LOOP;\n RETURN cnt;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION run_or_queue(query text) RETURNS VOID AS $$\nDECLARE\n use_queue text := COALESCE(get_setting('use_queue'), 'FALSE')::boolean;\nBEGIN\n IF get_setting_bool('debug') THEN\n RAISE NOTICE '%', query;\n END IF;\n IF use_queue THEN\n INSERT INTO query_queue (query) VALUES (query) ON CONFLICT DO NOTHING;\n ELSE\n EXECUTE query;\n END IF;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n\nDROP FUNCTION IF EXISTS check_pgstac_settings;\nCREATE OR REPLACE FUNCTION check_pgstac_settings(_sysmem text DEFAULT NULL) RETURNS VOID AS $$\nDECLARE\n settingval text;\n sysmem bigint := pg_size_bytes(_sysmem);\n effective_cache_size bigint := pg_size_bytes(current_setting('effective_cache_size', TRUE));\n shared_buffers bigint := pg_size_bytes(current_setting('shared_buffers', TRUE));\n work_mem bigint := pg_size_bytes(current_setting('work_mem', TRUE));\n max_connections int := current_setting('max_connections', TRUE);\n maintenance_work_mem bigint := pg_size_bytes(current_setting('maintenance_work_mem', TRUE));\n seq_page_cost float := current_setting('seq_page_cost', TRUE);\n random_page_cost float := current_setting('random_page_cost', TRUE);\n temp_buffers bigint := pg_size_bytes(current_setting('temp_buffers', TRUE));\n r record;\nBEGIN\n IF _sysmem IS NULL THEN\n RAISE NOTICE 'Call function with the size of your system memory `SELECT check_pgstac_settings(''4GB'')` to get pg system setting recommendations.';\n ELSE\n IF effective_cache_size < (sysmem * 0.5) THEN\n RAISE WARNING 'effective_cache_size of % is set low for a system with %. Recomended value between % and %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.5), pg_size_pretty(sysmem * 0.75);\n ELSIF effective_cache_size > (sysmem * 0.75) THEN\n RAISE WARNING 'effective_cache_size of % is set high for a system with %. Recomended value between % and %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.5), pg_size_pretty(sysmem * 0.75);\n ELSE\n RAISE NOTICE 'effective_cache_size of % is set appropriately for a system with %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem);\n END IF;\n\n IF shared_buffers < (sysmem * 0.2) THEN\n RAISE WARNING 'shared_buffers of % is set low for a system with %. Recomended value between % and %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.2), pg_size_pretty(sysmem * 0.3);\n ELSIF shared_buffers > (sysmem * 0.3) THEN\n RAISE WARNING 'shared_buffers of % is set high for a system with %. Recomended value between % and %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.2), pg_size_pretty(sysmem * 0.3);\n ELSE\n RAISE NOTICE 'shared_buffers of % is set appropriately for a system with %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem);\n END IF;\n shared_buffers = sysmem * 0.3;\n IF maintenance_work_mem < (sysmem * 0.2) THEN\n RAISE WARNING 'maintenance_work_mem of % is set low for shared_buffers of %. Recomended value between % and %', pg_size_pretty(maintenance_work_mem), pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers * 0.2), pg_size_pretty(shared_buffers * 0.3);\n ELSIF maintenance_work_mem > (shared_buffers * 0.3) THEN\n RAISE WARNING 'maintenance_work_mem of % is set high for shared_buffers of %. Recomended value between % and %', pg_size_pretty(maintenance_work_mem), pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers * 0.2), pg_size_pretty(shared_buffers * 0.3);\n ELSE\n RAISE NOTICE 'maintenance_work_mem of % is set appropriately for shared_buffers of %', pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers);\n END IF;\n\n IF work_mem * max_connections > shared_buffers THEN\n RAISE WARNING 'work_mem setting of % is set high for % max_connections please reduce work_mem to % or decrease max_connections to %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers/max_connections), floor(shared_buffers/work_mem);\n ELSIF work_mem * max_connections < (shared_buffers * 0.75) THEN\n RAISE WARNING 'work_mem setting of % is set low for % max_connections you may consider raising work_mem to % or increasing max_connections to %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers/max_connections), floor(shared_buffers/work_mem);\n ELSE\n RAISE NOTICE 'work_mem setting of % and max_connections of % are adequate for shared_buffers of %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers);\n END IF;\n\n IF random_page_cost / seq_page_cost != 1.1 THEN\n RAISE WARNING 'random_page_cost (%) /seq_page_cost (%) should be set to 1.1 for SSD. Change random_page_cost to %', random_page_cost, seq_page_cost, 1.1 * seq_page_cost;\n ELSE\n RAISE NOTICE 'random_page_cost and seq_page_cost set appropriately for SSD';\n END IF;\n\n IF temp_buffers < greatest(pg_size_bytes('128MB'),(maintenance_work_mem / 2)) THEN\n RAISE WARNING 'pgstac makes heavy use of temp tables, consider raising temp_buffers from % to %', pg_size_pretty(temp_buffers), greatest('128MB', pg_size_pretty((shared_buffers / 16)));\n END IF;\n END IF;\n\n RAISE NOTICE 'VALUES FOR PGSTAC VARIABLES';\n RAISE NOTICE 'These can be set either as GUC system variables or by setting in the pgstac_settings table.';\n\n FOR r IN SELECT name, get_setting(name) as setting, CASE WHEN current_setting(concat('pgstac.',name), TRUE) IS NOT NULL THEN concat('pgstac.',name, ' GUC') WHEN value IS NOT NULL THEN 'pgstac_settings table' ELSE 'Not Set' END as loc FROM pgstac_settings LOOP\n RAISE NOTICE '% is set to % from the %', r.name, r.setting, r.loc;\n END LOOP;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pg_cron';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider intalling pg_cron which can be used to automate tasks';\n ELSE\n RAISE NOTICE 'pg_cron % is installed', settingval;\n END IF;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pgstattuple';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider installing the pgstattuple extension which can be used to help maintain tables and indexes.';\n ELSE\n RAISE NOTICE 'pgstattuple % is installed', settingval;\n END IF;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pg_stat_statements';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider installing the pg_stat_statements extension which is very helpful for tracking the types of queries on the system';\n ELSE\n RAISE NOTICE 'pg_stat_statements % is installed', settingval;\n IF current_setting('pg_stat_statements.track_statements', TRUE) IS DISTINCT FROM 'all' THEN\n RAISE WARNING 'SET pg_stat_statements.track_statements TO ''all''; --In order to track statements within functions.';\n END IF;\n END IF;\n\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac, public SET CLIENT_MIN_MESSAGES TO NOTICE;\nCREATE OR REPLACE FUNCTION to_int(jsonb) RETURNS int AS $$\n SELECT floor(($1->>0)::float)::int;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION to_float(jsonb) RETURNS float AS $$\n SELECT ($1->>0)::float;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION to_tstz(jsonb) RETURNS timestamptz AS $$\n SELECT ($1->>0)::timestamptz;\n$$ LANGUAGE SQL IMMUTABLE STRICT SET TIME ZONE 'UTC' COST 5000 PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION to_text(jsonb) RETURNS text AS $$\n SELECT CASE WHEN jsonb_typeof($1) IN ('array','object') THEN $1::text ELSE $1->>0 END;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION to_text_array(jsonb) RETURNS text[] AS $$\n SELECT\n CASE jsonb_typeof($1)\n WHEN 'array' THEN ARRAY(SELECT jsonb_array_elements_text($1))\n ELSE ARRAY[$1->>0]\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION bbox_geom(_bbox jsonb) RETURNS geometry AS $$\nSELECT CASE jsonb_array_length(_bbox)\n WHEN 4 THEN\n ST_SetSRID(ST_MakeEnvelope(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float,\n (_bbox->>3)::float\n ),4326)\n WHEN 6 THEN\n ST_SetSRID(ST_3DMakeBox(\n ST_MakePoint(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float\n ),\n ST_MakePoint(\n (_bbox->>3)::float,\n (_bbox->>4)::float,\n (_bbox->>5)::float\n )\n ),4326)\n ELSE null END;\n;\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION geom_bbox(_geom geometry) RETURNS jsonb AS $$\n SELECT jsonb_build_array(\n st_xmin(_geom),\n st_ymin(_geom),\n st_xmax(_geom),\n st_ymax(_geom)\n );\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION flip_jsonb_array(j jsonb) RETURNS jsonb AS $$\n SELECT jsonb_agg(value) FROM (SELECT value FROM jsonb_array_elements(j) WITH ORDINALITY ORDER BY ordinality DESC) as t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION explode_dotpaths(j jsonb) RETURNS SETOF text[] AS $$\n SELECT string_to_array(p, '.') as e FROM jsonb_array_elements_text(j) p;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION explode_dotpaths_recurse(IN j jsonb) RETURNS SETOF text[] AS $$\n WITH RECURSIVE t AS (\n SELECT e FROM explode_dotpaths(j) e\n UNION ALL\n SELECT e[1:cardinality(e)-1]\n FROM t\n WHERE cardinality(e)>1\n ) SELECT e FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION jsonb_set_nested(j jsonb, path text[], val jsonb) RETURNS jsonb AS $$\nDECLARE\nBEGIN\n IF cardinality(path) > 1 THEN\n FOR i IN 1..(cardinality(path)-1) LOOP\n IF j #> path[:i] IS NULL THEN\n j := jsonb_set_lax(j, path[:i], '{}', TRUE);\n END IF;\n END LOOP;\n END IF;\n RETURN jsonb_set_lax(j, path, val, true);\n\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\n\n\nCREATE OR REPLACE FUNCTION jsonb_include(j jsonb, f jsonb) RETURNS jsonb AS $$\nDECLARE\n includes jsonb := f-> 'include';\n outj jsonb := '{}'::jsonb;\n path text[];\nBEGIN\n IF\n includes IS NULL\n OR jsonb_array_length(includes) = 0\n THEN\n RETURN j;\n ELSE\n includes := includes || (\n CASE WHEN j ? 'collection' THEN\n '[\"id\",\"collection\"]'\n ELSE\n '[\"id\"]'\n END)::jsonb;\n FOR path IN SELECT explode_dotpaths(includes) LOOP\n outj := jsonb_set_nested(outj, path, j #> path);\n END LOOP;\n END IF;\n RETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_exclude(j jsonb, f jsonb) RETURNS jsonb AS $$\nDECLARE\n excludes jsonb := f-> 'exclude';\n outj jsonb := j;\n path text[];\nBEGIN\n IF\n excludes IS NULL\n OR jsonb_array_length(excludes) = 0\n THEN\n RETURN j;\n ELSE\n FOR path IN SELECT explode_dotpaths(excludes) LOOP\n outj := outj #- path;\n END LOOP;\n END IF;\n RETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_fields(j jsonb, f jsonb DEFAULT '{\"fields\":[]}') RETURNS jsonb AS $$\n SELECT jsonb_exclude(jsonb_include(j, f), f);\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION merge_jsonb(_a jsonb, _b jsonb) RETURNS jsonb AS $$\n SELECT\n CASE\n WHEN _a = '\"𒍟※\"'::jsonb THEN NULL\n WHEN _a IS NULL OR jsonb_typeof(_a) = 'null' THEN _b\n WHEN jsonb_typeof(_a) = 'object' AND jsonb_typeof(_b) = 'object' THEN\n (\n SELECT\n jsonb_strip_nulls(\n jsonb_object_agg(\n key,\n merge_jsonb(a.value, b.value)\n )\n )\n FROM\n jsonb_each(coalesce(_a,'{}'::jsonb)) as a\n FULL JOIN\n jsonb_each(coalesce(_b,'{}'::jsonb)) as b\n USING (key)\n )\n WHEN\n jsonb_typeof(_a) = 'array'\n AND jsonb_typeof(_b) = 'array'\n AND jsonb_array_length(_a) = jsonb_array_length(_b)\n THEN\n (\n SELECT jsonb_agg(m) FROM\n ( SELECT\n merge_jsonb(\n jsonb_array_elements(_a),\n jsonb_array_elements(_b)\n ) as m\n ) as l\n )\n ELSE _a\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION strip_jsonb(_a jsonb, _b jsonb) RETURNS jsonb AS $$\n SELECT\n CASE\n\n WHEN (_a IS NULL OR jsonb_typeof(_a) = 'null') AND _b IS NOT NULL AND jsonb_typeof(_b) != 'null' THEN '\"𒍟※\"'::jsonb\n WHEN _b IS NULL OR jsonb_typeof(_a) = 'null' THEN _a\n WHEN _a = _b AND jsonb_typeof(_a) = 'object' THEN '{}'::jsonb\n WHEN _a = _b THEN NULL\n WHEN jsonb_typeof(_a) = 'object' AND jsonb_typeof(_b) = 'object' THEN\n (\n SELECT\n jsonb_strip_nulls(\n jsonb_object_agg(\n key,\n strip_jsonb(a.value, b.value)\n )\n )\n FROM\n jsonb_each(_a) as a\n FULL JOIN\n jsonb_each(_b) as b\n USING (key)\n )\n WHEN\n jsonb_typeof(_a) = 'array'\n AND jsonb_typeof(_b) = 'array'\n AND jsonb_array_length(_a) = jsonb_array_length(_b)\n THEN\n (\n SELECT jsonb_agg(m) FROM\n ( SELECT\n strip_jsonb(\n jsonb_array_elements(_a),\n jsonb_array_elements(_b)\n ) as m\n ) as l\n )\n ELSE _a\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION nullif_jsonbnullempty(j jsonb) RETURNS jsonb AS $$\n SELECT nullif(nullif(nullif(j,'null'::jsonb),'{}'::jsonb),'[]'::jsonb);\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION jsonb_array_unique(j jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(jsonb_agg(DISTINCT a)) v FROM jsonb_array_elements(j) a;\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_concat_ignorenull(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT coalesce(a,'[]'::jsonb) || coalesce(b,'[]'::jsonb);\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_least(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(least(nullif_jsonbnullempty(a), nullif_jsonbnullempty(b)));\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_greatest(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(greatest(a, b));\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION first_notnull_sfunc(anyelement, anyelement) RETURNS anyelement AS $$\n SELECT COALESCE($1,$2);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE AGGREGATE first_notnull(anyelement)(\n SFUNC = first_notnull_sfunc,\n STYPE = anyelement\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_array_unique_merge(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_concat_ignorenull,\n FINALFUNC = jsonb_array_unique\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_min(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_least\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_max(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_greatest\n);\n/* looks for a geometry in a stac item first from geometry and falling back to bbox */\nCREATE OR REPLACE FUNCTION stac_geom(value jsonb) RETURNS geometry AS $$\nSELECT\n CASE\n WHEN value ? 'intersects' THEN\n ST_GeomFromGeoJSON(value->>'intersects')\n WHEN value ? 'geometry' THEN\n ST_GeomFromGeoJSON(value->>'geometry')\n WHEN value ? 'bbox' THEN\n pgstac.bbox_geom(value->'bbox')\n ELSE NULL\n END as geometry\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION stac_daterange(\n value jsonb\n) RETURNS tstzrange AS $$\nDECLARE\n props jsonb := value;\n dt timestamptz;\n edt timestamptz;\nBEGIN\n IF props ? 'properties' THEN\n props := props->'properties';\n END IF;\n IF\n props ? 'start_datetime'\n AND props->>'start_datetime' IS NOT NULL\n AND props ? 'end_datetime'\n AND props->>'end_datetime' IS NOT NULL\n THEN\n dt := props->>'start_datetime';\n edt := props->>'end_datetime';\n IF dt > edt THEN\n RAISE EXCEPTION 'start_datetime must be < end_datetime';\n END IF;\n ELSE\n dt := props->>'datetime';\n edt := props->>'datetime';\n END IF;\n IF dt is NULL OR edt IS NULL THEN\n RAISE NOTICE 'DT: %, EDT: %', dt, edt;\n RAISE EXCEPTION 'Either datetime (%) or both start_datetime (%) and end_datetime (%) must be set.', props->>'datetime',props->>'start_datetime',props->>'end_datetime';\n END IF;\n RETURN tstzrange(dt, edt, '[]');\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_datetime(value jsonb) RETURNS timestamptz AS $$\n SELECT lower(stac_daterange(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_end_datetime(value jsonb) RETURNS timestamptz AS $$\n SELECT upper(stac_daterange(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE TABLE IF NOT EXISTS stac_extensions(\n url text PRIMARY KEY,\n content jsonb\n);\nCREATE OR REPLACE FUNCTION collection_base_item(content jsonb) RETURNS jsonb AS $$\n SELECT jsonb_build_object(\n 'type', 'Feature',\n 'stac_version', content->'stac_version',\n 'assets', content->'item_assets',\n 'collection', content->'id'\n );\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE TABLE IF NOT EXISTS collections (\n key bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,\n id text GENERATED ALWAYS AS (content->>'id') STORED UNIQUE NOT NULL,\n content JSONB NOT NULL,\n base_item jsonb GENERATED ALWAYS AS (pgstac.collection_base_item(content)) STORED,\n geometry geometry GENERATED ALWAYS AS (pgstac.collection_geom(content)) STORED,\n datetime timestamptz GENERATED ALWAYS AS (pgstac.collection_datetime(content)) STORED,\n end_datetime timestamptz GENERATED ALWAYS AS (pgstac.collection_enddatetime(content)) STORED,\n private jsonb,\n partition_trunc text CHECK (partition_trunc IN ('year', 'month'))\n);\n\nCREATE OR REPLACE FUNCTION collection_base_item(cid text) RETURNS jsonb AS $$\n SELECT pgstac.collection_base_item(content) FROM pgstac.collections WHERE id = cid LIMIT 1;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION create_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION update_collection(data jsonb) RETURNS VOID AS $$\nDECLARE\n out collections%ROWTYPE;\nBEGIN\n UPDATE collections SET content=data WHERE id = data->>'id' RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ON CONFLICT (id) DO\n UPDATE\n SET content=EXCLUDED.content\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION delete_collection(_id text) RETURNS VOID AS $$\nDECLARE\n out collections%ROWTYPE;\nBEGIN\n DELETE FROM collections WHERE id = _id RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION get_collection(id text) RETURNS jsonb AS $$\n SELECT content FROM collections\n WHERE id=$1\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION all_collections() RETURNS jsonb AS $$\n SELECT coalesce(jsonb_agg(content), '[]'::jsonb) FROM collections;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION collection_delete_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n collection_base_partition text := concat('_items_', OLD.key);\nBEGIN\n EXECUTE format($q$\n DELETE FROM partition_stats WHERE partition IN (\n SELECT partition FROM partition_sys_meta\n WHERE collection=%L\n );\n DROP TABLE IF EXISTS %I CASCADE;\n $q$,\n OLD.id,\n collection_base_partition\n );\n RETURN OLD;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER collection_delete_trigger BEFORE DELETE ON collections\nFOR EACH ROW EXECUTE FUNCTION collection_delete_trigger_func();\nCREATE OR REPLACE FUNCTION queryable_signature(n text, c text[]) RETURNS text AS $$\n SELECT concat(n, c);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TABLE queryables (\n id bigint GENERATED ALWAYS AS identity PRIMARY KEY,\n name text NOT NULL,\n collection_ids text[], -- used to determine what partitions to create indexes on\n definition jsonb,\n property_path text,\n property_wrapper text,\n property_index_type text\n);\nCREATE INDEX queryables_name_idx ON queryables (name);\nCREATE INDEX queryables_collection_idx ON queryables USING GIN (collection_ids);\nCREATE INDEX queryables_property_wrapper_idx ON queryables (property_wrapper);\n\nCREATE OR REPLACE FUNCTION pgstac.queryables_constraint_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n allcollections text[];\nBEGIN\n RAISE NOTICE 'Making sure that name/collection is unique for queryables %', NEW;\n IF NEW.collection_ids IS NOT NULL THEN\n IF EXISTS (\n SELECT 1\n FROM unnest(NEW.collection_ids) c\n LEFT JOIN\n collections\n ON (collections.id = c)\n WHERE collections.id IS NULL\n ) THEN\n RAISE foreign_key_violation USING MESSAGE = format(\n 'One or more collections in %s do not exist.', NEW.collection_ids\n );\n RETURN NULL;\n END IF;\n END IF;\n IF TG_OP = 'INSERT' THEN\n IF EXISTS (\n SELECT 1 FROM queryables q\n WHERE\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n )\n ) THEN\n RAISE unique_violation USING MESSAGE = format(\n 'There is already a queryable for %s for a collection in %s: %s',\n NEW.name,\n NEW.collection_ids,\n\t\t\t\t(SELECT json_agg(row_to_json(q)) FROM queryables q WHERE\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n ))\n );\n RETURN NULL;\n END IF;\n END IF;\n IF TG_OP = 'UPDATE' THEN\n IF EXISTS (\n SELECT 1 FROM queryables q\n WHERE\n q.id != NEW.id\n AND\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n )\n ) THEN\n RAISE unique_violation\n USING MESSAGE = format(\n 'There is already a queryable for %s for a collection in %s',\n NEW.name,\n NEW.collection_ids\n );\n RETURN NULL;\n END IF;\n END IF;\n\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER queryables_constraint_insert_trigger\nBEFORE INSERT ON queryables\nFOR EACH ROW EXECUTE PROCEDURE queryables_constraint_triggerfunc();\n\nCREATE TRIGGER queryables_constraint_update_trigger\nBEFORE UPDATE ON queryables\nFOR EACH ROW\nWHEN (NEW.name = OLD.name AND NEW.collection_ids IS DISTINCT FROM OLD.collection_ids)\nEXECUTE PROCEDURE queryables_constraint_triggerfunc();\n\n\nCREATE OR REPLACE FUNCTION array_to_path(arr text[]) RETURNS text AS $$\n SELECT string_agg(\n quote_literal(v),\n '->'\n ) FROM unnest(arr) v;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\n\n\nCREATE OR REPLACE FUNCTION queryable(\n IN dotpath text,\n OUT path text,\n OUT expression text,\n OUT wrapper text,\n OUT nulled_wrapper text\n) AS $$\nDECLARE\n q RECORD;\n path_elements text[];\nBEGIN\n dotpath := replace(dotpath, 'properties.', '');\n IF dotpath = 'start_datetime' THEN\n dotpath := 'datetime';\n END IF;\n IF dotpath IN ('id', 'geometry', 'datetime', 'end_datetime', 'collection') THEN\n path := dotpath;\n expression := dotpath;\n wrapper := NULL;\n RETURN;\n END IF;\n\n SELECT * INTO q FROM queryables\n WHERE\n name=dotpath\n OR name = 'properties.' || dotpath\n OR name = replace(dotpath, 'properties.', '')\n ;\n IF q.property_wrapper IS NULL THEN\n IF q.definition->>'type' = 'number' THEN\n wrapper := 'to_float';\n nulled_wrapper := wrapper;\n ELSIF q.definition->>'format' = 'date-time' THEN\n wrapper := 'to_tstz';\n nulled_wrapper := wrapper;\n ELSE\n nulled_wrapper := NULL;\n wrapper := 'to_text';\n END IF;\n ELSE\n wrapper := q.property_wrapper;\n nulled_wrapper := wrapper;\n END IF;\n IF q.property_path IS NOT NULL THEN\n path := q.property_path;\n ELSE\n path_elements := string_to_array(dotpath, '.');\n IF path_elements[1] IN ('links', 'assets', 'stac_version', 'stac_extensions') THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSIF path_elements[1] = 'properties' THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSE\n path := format($F$content->'properties'->%s$F$, array_to_path(path_elements));\n END IF;\n END IF;\n expression := format('%I(%s)', wrapper, path);\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\nCREATE OR REPLACE FUNCTION unnest_collection(collection_ids text[] DEFAULT NULL) RETURNS SETOF text AS $$\n DECLARE\n BEGIN\n IF collection_ids IS NULL THEN\n RETURN QUERY SELECT id FROM collections;\n END IF;\n RETURN QUERY SELECT unnest(collection_ids);\n END;\n$$ LANGUAGE PLPGSQL STABLE;\n\nCREATE OR REPLACE FUNCTION normalize_indexdef(def text) RETURNS text AS $$\nDECLARE\nBEGIN\n def := btrim(def, ' \\n\\t');\n\tdef := regexp_replace(def, '^CREATE (UNIQUE )?INDEX ([^ ]* )?ON (ONLY )?([^ ]* )?', '', 'i');\n RETURN def;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION indexdef(q queryables) RETURNS text AS $$\n DECLARE\n out text;\n BEGIN\n IF q.name = 'id' THEN\n out := 'CREATE UNIQUE INDEX ON %I USING btree (id)';\n ELSIF q.name = 'datetime' THEN\n out := 'CREATE INDEX ON %I USING btree (datetime DESC, end_datetime)';\n ELSIF q.name = 'geometry' THEN\n out := 'CREATE INDEX ON %I USING gist (geometry)';\n ELSE\n out := format($q$CREATE INDEX ON %%I USING %s (%s(((content -> 'properties'::text) -> %L::text)))$q$,\n lower(COALESCE(q.property_index_type, 'BTREE')),\n lower(COALESCE(q.property_wrapper, 'to_text')),\n q.name\n );\n END IF;\n RETURN btrim(out, ' \\n\\t');\n END;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nDROP VIEW IF EXISTS pgstac_indexes;\nCREATE VIEW pgstac_indexes AS\nSELECT\n i.schemaname,\n i.tablename,\n i.indexname,\n regexp_replace(btrim(replace(replace(indexdef, i.indexname, ''),'pgstac.',''),' \\t\\n'), '[ ]+', ' ', 'g') as idx,\n COALESCE(\n (regexp_match(indexdef, '\\(([a-zA-Z]+)\\)'))[1],\n (regexp_match(indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_-]+)''::text'))[1],\n CASE WHEN indexdef ~* '\\(datetime desc, end_datetime\\)' THEN 'datetime' ELSE NULL END\n ) AS field,\n pg_table_size(i.indexname::text) as index_size,\n pg_size_pretty(pg_table_size(i.indexname::text)) as index_size_pretty\nFROM\n pg_indexes i\nWHERE i.schemaname='pgstac' and i.tablename ~ '_items_' AND indexdef !~* ' only ';\n\nDROP VIEW IF EXISTS pgstac_index_stats;\nCREATE VIEW pgstac_indexes_stats AS\nSELECT\n i.schemaname,\n i.tablename,\n i.indexname,\n indexdef,\n COALESCE(\n (regexp_match(indexdef, '\\(([a-zA-Z]+)\\)'))[1],\n (regexp_match(indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_]+)''::text'))[1],\n CASE WHEN indexdef ~* '\\(datetime desc, end_datetime\\)' THEN 'datetime_end_datetime' ELSE NULL END\n ) AS field,\n pg_table_size(i.indexname::text) as index_size,\n pg_size_pretty(pg_table_size(i.indexname::text)) as index_size_pretty,\n n_distinct,\n most_common_vals::text::text[],\n most_common_freqs::text::text[],\n histogram_bounds::text::text[],\n correlation\nFROM\n pg_indexes i\n LEFT JOIN pg_stats s ON (s.tablename = i.indexname)\nWHERE i.schemaname='pgstac' and i.tablename ~ '_items_';\n\nCREATE OR REPLACE FUNCTION queryable_indexes(\n IN treeroot text DEFAULT 'items',\n IN changes boolean DEFAULT FALSE,\n OUT collection text,\n OUT partition text,\n OUT field text,\n OUT indexname text,\n OUT existing_idx text,\n OUT queryable_idx text\n) RETURNS SETOF RECORD AS $$\nWITH p AS (\n SELECT\n relid::text as partition,\n replace(replace(\n CASE\n WHEN parentrelid::regclass::text='items' THEN pg_get_expr(c.relpartbound, c.oid)\n ELSE pg_get_expr(parent.relpartbound, parent.oid)\n END,\n 'FOR VALUES IN (''',''), ''')',\n ''\n ) AS collection\n FROM pg_partition_tree(treeroot)\n JOIN pg_class c ON (relid::regclass = c.oid)\n JOIN pg_class parent ON (parentrelid::regclass = parent.oid AND isleaf)\n ), i AS (\n SELECT\n partition,\n indexname,\n regexp_replace(btrim(replace(replace(indexdef, indexname, ''),'pgstac.',''),' \\t\\n'), '[ ]+', ' ', 'g') as iidx,\n COALESCE(\n (regexp_match(indexdef, '\\(([a-zA-Z]+)\\)'))[1],\n (regexp_match(indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_-]+)''::text'))[1],\n CASE WHEN indexdef ~* '\\(datetime desc, end_datetime\\)' THEN 'datetime' ELSE NULL END\n ) AS field\n FROM\n pg_indexes\n JOIN p ON (tablename=partition)\n ), q AS (\n SELECT\n name AS field,\n collection,\n partition,\n format(indexdef(queryables), partition) as qidx\n FROM queryables, unnest_collection(queryables.collection_ids) collection\n JOIN p USING (collection)\n WHERE property_index_type IS NOT NULL OR name IN ('datetime','geometry','id')\n )\n SELECT\n collection,\n partition,\n field,\n indexname,\n iidx as existing_idx,\n qidx as queryable_idx\n FROM i FULL JOIN q USING (field, partition)\n WHERE CASE WHEN changes THEN lower(iidx) IS DISTINCT FROM lower(qidx) ELSE TRUE END;\n;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION maintain_index(\n indexname text,\n queryable_idx text,\n dropindexes boolean DEFAULT FALSE,\n rebuildindexes boolean DEFAULT FALSE,\n idxconcurrently boolean DEFAULT FALSE\n) RETURNS VOID AS $$\nDECLARE\nBEGIN\n IF indexname IS NOT NULL THEN\n IF dropindexes OR queryable_idx IS NOT NULL THEN\n EXECUTE format('DROP INDEX IF EXISTS %I;', indexname);\n ELSIF rebuildindexes THEN\n IF idxconcurrently THEN\n EXECUTE format('REINDEX INDEX CONCURRENTLY %I;', indexname);\n ELSE\n EXECUTE format('REINDEX INDEX CONCURRENTLY %I;', indexname);\n END IF;\n END IF;\n END IF;\n IF queryable_idx IS NOT NULL THEN\n IF idxconcurrently THEN\n EXECUTE replace(queryable_idx, 'INDEX', 'INDEX CONCURRENTLY');\n ELSE EXECUTE queryable_idx;\n END IF;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\n\nset check_function_bodies to off;\nCREATE OR REPLACE FUNCTION maintain_partition_queries(\n part text DEFAULT 'items',\n dropindexes boolean DEFAULT FALSE,\n rebuildindexes boolean DEFAULT FALSE,\n idxconcurrently boolean DEFAULT FALSE\n) RETURNS SETOF text AS $$\nDECLARE\n rec record;\n q text;\nBEGIN\n FOR rec IN (\n SELECT * FROM queryable_indexes(part,true)\n ) LOOP\n q := format(\n 'SELECT maintain_index(\n %L,%L,%L,%L,%L\n );',\n rec.indexname,\n rec.queryable_idx,\n dropindexes,\n rebuildindexes,\n idxconcurrently\n );\n RAISE NOTICE 'Q: %', q;\n RETURN NEXT q;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION maintain_partitions(\n part text DEFAULT 'items',\n dropindexes boolean DEFAULT FALSE,\n rebuildindexes boolean DEFAULT FALSE\n) RETURNS VOID AS $$\n WITH t AS (\n SELECT run_or_queue(q) FROM maintain_partition_queries(part, dropindexes, rebuildindexes) q\n ) SELECT count(*) FROM t;\n$$ LANGUAGE SQL;\n\n\nCREATE OR REPLACE FUNCTION queryables_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\nBEGIN\n PERFORM maintain_partitions();\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER queryables_trigger AFTER INSERT OR UPDATE ON queryables\nFOR EACH STATEMENT EXECUTE PROCEDURE queryables_trigger_func();\n\n\nCREATE OR REPLACE FUNCTION get_queryables(_collection_ids text[] DEFAULT NULL) RETURNS jsonb AS $$\nDECLARE\nBEGIN\n -- Build up queryables if the input contains valid collection ids or is empty\n IF EXISTS (\n SELECT 1 FROM collections\n WHERE\n _collection_ids IS NULL\n OR cardinality(_collection_ids) = 0\n OR id = ANY(_collection_ids)\n )\n THEN\n RETURN (\n WITH base AS (\n SELECT\n unnest(collection_ids) as collection_id,\n name,\n coalesce(definition, '{\"type\":\"string\"}'::jsonb) as definition\n FROM queryables\n WHERE\n _collection_ids IS NULL OR\n _collection_ids = '{}'::text[] OR\n _collection_ids && collection_ids\n UNION ALL\n SELECT null, name, coalesce(definition, '{\"type\":\"string\"}'::jsonb) as definition\n FROM queryables WHERE collection_ids IS NULL OR collection_ids = '{}'::text[]\n ), g AS (\n SELECT\n name,\n first_notnull(definition) as definition,\n jsonb_array_unique_merge(definition->'enum') as enum,\n jsonb_min(definition->'minimum') as minimum,\n jsonb_min(definition->'maxiumn') as maximum\n FROM base\n GROUP BY 1\n )\n SELECT\n jsonb_build_object(\n '$schema', 'http://json-schema.org/draft-07/schema#',\n '$id', '',\n 'type', 'object',\n 'title', 'STAC Queryables.',\n 'properties', jsonb_object_agg(\n name,\n definition\n ||\n jsonb_strip_nulls(jsonb_build_object(\n 'enum', enum,\n 'minimum', minimum,\n 'maximum', maximum\n ))\n ),\n 'additionalProperties', pgstac.additional_properties()\n )\n FROM g\n );\n ELSE\n RETURN NULL;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\nCREATE OR REPLACE FUNCTION get_queryables(_collection text DEFAULT NULL) RETURNS jsonb AS $$\n SELECT\n CASE\n WHEN _collection IS NULL THEN get_queryables(NULL::text[])\n ELSE get_queryables(ARRAY[_collection])\n END\n ;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_queryables() RETURNS jsonb AS $$\n SELECT get_queryables(NULL::text[]);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION schema_qualify_refs(url text, j jsonb) returns jsonb as $$\n SELECT regexp_replace(j::text, '\"\\$ref\": \"#', concat('\"$ref\": \"', url, '#'), 'g')::jsonb;\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE VIEW stac_extension_queryables AS\nSELECT DISTINCT key as name, schema_qualify_refs(e.url, j.value) as definition FROM stac_extensions e, jsonb_each(e.content->'definitions'->'fields'->'properties') j;\n\n\nCREATE OR REPLACE FUNCTION missing_queryables(_collection text, _tablesample float DEFAULT 5, minrows float DEFAULT 10) RETURNS TABLE(collection text, name text, definition jsonb, property_wrapper text) AS $$\nDECLARE\n q text;\n _partition text;\n explain_json json;\n psize float;\n estrows float;\nBEGIN\n SELECT format('_items_%s', key) INTO _partition FROM collections WHERE id=_collection;\n\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM %I;', _partition)\n INTO explain_json;\n psize := explain_json->0->'Plan'->'Plan Rows';\n estrows := _tablesample * .01 * psize;\n IF estrows < minrows THEN\n _tablesample := least(100,greatest(_tablesample, (estrows / psize) / 100));\n RAISE NOTICE '%', (psize / estrows) / 100;\n END IF;\n RAISE NOTICE 'Using tablesample % to find missing queryables from % % that has ~% rows estrows: %', _tablesample, _collection, _partition, psize, estrows;\n\n q := format(\n $q$\n WITH q AS (\n SELECT * FROM queryables\n WHERE\n collection_ids IS NULL\n OR %L = ANY(collection_ids)\n ), t AS (\n SELECT\n content->'properties' AS properties\n FROM\n %I\n TABLESAMPLE SYSTEM(%L)\n ), p AS (\n SELECT DISTINCT ON (key)\n key,\n value,\n s.definition\n FROM t\n JOIN LATERAL jsonb_each(properties) ON TRUE\n LEFT JOIN q ON (q.name=key)\n LEFT JOIN stac_extension_queryables s ON (s.name=key)\n WHERE q.definition IS NULL\n )\n SELECT\n %L,\n key,\n COALESCE(definition, jsonb_build_object('type',jsonb_typeof(value))) as definition,\n CASE\n WHEN definition->>'type' = 'integer' THEN 'to_int'\n WHEN COALESCE(definition->>'type', jsonb_typeof(value)) = 'number' THEN 'to_float'\n WHEN COALESCE(definition->>'type', jsonb_typeof(value)) = 'array' THEN 'to_text_array'\n ELSE 'to_text'\n END\n FROM p;\n $q$,\n _collection,\n _partition,\n _tablesample,\n _collection\n );\n RETURN QUERY EXECUTE q;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION missing_queryables(_tablesample float DEFAULT 5) RETURNS TABLE(collection_ids text[], name text, definition jsonb, property_wrapper text) AS $$\n SELECT\n array_agg(collection),\n name,\n definition,\n property_wrapper\n FROM\n collections\n JOIN LATERAL\n missing_queryables(id, _tablesample) c\n ON TRUE\n GROUP BY\n 2,3,4\n ORDER BY 2,1\n ;\n$$ LANGUAGE SQL;\nCREATE OR REPLACE FUNCTION parse_dtrange(\n _indate jsonb,\n relative_base timestamptz DEFAULT date_trunc('hour', CURRENT_TIMESTAMP)\n) RETURNS tstzrange AS $$\nDECLARE\n timestrs text[];\n s timestamptz;\n e timestamptz;\nBEGIN\n timestrs :=\n CASE\n WHEN _indate ? 'timestamp' THEN\n ARRAY[_indate->>'timestamp']\n WHEN _indate ? 'interval' THEN\n to_text_array(_indate->'interval')\n WHEN jsonb_typeof(_indate) = 'array' THEN\n to_text_array(_indate)\n ELSE\n regexp_split_to_array(\n _indate->>0,\n '/'\n )\n END;\n RAISE NOTICE 'TIMESTRS %', timestrs;\n IF cardinality(timestrs) = 1 THEN\n IF timestrs[1] ILIKE 'P%' THEN\n RETURN tstzrange(relative_base - upper(timestrs[1])::interval, relative_base, '[)');\n END IF;\n s := timestrs[1]::timestamptz;\n RETURN tstzrange(s, s, '[]');\n END IF;\n\n IF cardinality(timestrs) != 2 THEN\n RAISE EXCEPTION 'Timestamp cannot have more than 2 values';\n END IF;\n\n IF timestrs[1] = '..' OR timestrs[1] = '' THEN\n s := '-infinity'::timestamptz;\n e := timestrs[2]::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] = '..' OR timestrs[2] = '' THEN\n s := timestrs[1]::timestamptz;\n e := 'infinity'::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[1] ILIKE 'P%' AND timestrs[2] NOT ILIKE 'P%' THEN\n e := timestrs[2]::timestamptz;\n s := e - upper(timestrs[1])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] ILIKE 'P%' AND timestrs[1] NOT ILIKE 'P%' THEN\n s := timestrs[1]::timestamptz;\n e := s + upper(timestrs[2])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n s := timestrs[1]::timestamptz;\n e := timestrs[2]::timestamptz;\n\n RETURN tstzrange(s,e,'[)');\n\n RETURN NULL;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT PARALLEL SAFE SET TIME ZONE 'UTC';\n\nCREATE OR REPLACE FUNCTION parse_dtrange(\n _indate text,\n relative_base timestamptz DEFAULT CURRENT_TIMESTAMP\n) RETURNS tstzrange AS $$\n SELECT parse_dtrange(to_jsonb(_indate), relative_base);\n$$ LANGUAGE SQL STABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION temporal_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\n ll text := 'datetime';\n lh text := 'end_datetime';\n rrange tstzrange;\n rl text;\n rh text;\n outq text;\nBEGIN\n rrange := parse_dtrange(args->1);\n RAISE NOTICE 'Constructing temporal query OP: %, ARGS: %, RRANGE: %', op, args, rrange;\n op := lower(op);\n rl := format('%L::timestamptz', lower(rrange));\n rh := format('%L::timestamptz', upper(rrange));\n outq := CASE op\n WHEN 't_before' THEN 'lh < rl'\n WHEN 't_after' THEN 'll > rh'\n WHEN 't_meets' THEN 'lh = rl'\n WHEN 't_metby' THEN 'll = rh'\n WHEN 't_overlaps' THEN 'll < rl AND rl < lh < rh'\n WHEN 't_overlappedby' THEN 'rl < ll < rh AND lh > rh'\n WHEN 't_starts' THEN 'll = rl AND lh < rh'\n WHEN 't_startedby' THEN 'll = rl AND lh > rh'\n WHEN 't_during' THEN 'll > rl AND lh < rh'\n WHEN 't_contains' THEN 'll < rl AND lh > rh'\n WHEN 't_finishes' THEN 'll > rl AND lh = rh'\n WHEN 't_finishedby' THEN 'll < rl AND lh = rh'\n WHEN 't_equals' THEN 'll = rl AND lh = rh'\n WHEN 't_disjoint' THEN 'NOT (ll <= rh AND lh >= rl)'\n WHEN 't_intersects' THEN 'll <= rh AND lh >= rl'\n WHEN 'anyinteracts' THEN 'll <= rh AND lh >= rl'\n END;\n outq := regexp_replace(outq, '\\mll\\M', ll);\n outq := regexp_replace(outq, '\\mlh\\M', lh);\n outq := regexp_replace(outq, '\\mrl\\M', rl);\n outq := regexp_replace(outq, '\\mrh\\M', rh);\n outq := format('(%s)', outq);\n RETURN outq;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\n\n\nCREATE OR REPLACE FUNCTION spatial_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\n geom text;\n j jsonb := args->1;\nBEGIN\n op := lower(op);\n RAISE NOTICE 'Constructing spatial query OP: %, ARGS: %', op, args;\n IF op NOT IN ('s_equals','s_disjoint','s_touches','s_within','s_overlaps','s_crosses','s_intersects','intersects','s_contains') THEN\n RAISE EXCEPTION 'Spatial Operator % Not Supported', op;\n END IF;\n op := regexp_replace(op, '^s_', 'st_');\n IF op = 'intersects' THEN\n op := 'st_intersects';\n END IF;\n -- Convert geometry to WKB string\n IF j ? 'type' AND j ? 'coordinates' THEN\n geom := st_geomfromgeojson(j)::text;\n ELSIF jsonb_typeof(j) = 'array' THEN\n geom := bbox_geom(j)::text;\n END IF;\n\n RETURN format('%s(geometry, %L::geometry)', op, geom);\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION query_to_cql2(q jsonb) RETURNS jsonb AS $$\n-- Translates anything passed in through the deprecated \"query\" into equivalent CQL2\nWITH t AS (\n SELECT key as property, value as ops\n FROM jsonb_each(q)\n), t2 AS (\n SELECT property, (jsonb_each(ops)).*\n FROM t WHERE jsonb_typeof(ops) = 'object'\n UNION ALL\n SELECT property, 'eq', ops\n FROM t WHERE jsonb_typeof(ops) != 'object'\n)\nSELECT\n jsonb_strip_nulls(jsonb_build_object(\n 'op', 'and',\n 'args', jsonb_agg(\n jsonb_build_object(\n 'op', key,\n 'args', jsonb_build_array(\n jsonb_build_object('property',property),\n value\n )\n )\n )\n )\n) as qcql FROM t2\n;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\nCREATE OR REPLACE FUNCTION cql1_to_cql2(j jsonb) RETURNS jsonb AS $$\nDECLARE\n args jsonb;\n ret jsonb;\nBEGIN\n RAISE NOTICE 'CQL1_TO_CQL2: %', j;\n IF j ? 'filter' THEN\n RETURN cql1_to_cql2(j->'filter');\n END IF;\n IF j ? 'property' THEN\n RETURN j;\n END IF;\n IF jsonb_typeof(j) = 'array' THEN\n SELECT jsonb_agg(cql1_to_cql2(el)) INTO args FROM jsonb_array_elements(j) el;\n RETURN args;\n END IF;\n IF jsonb_typeof(j) = 'number' THEN\n RETURN j;\n END IF;\n IF jsonb_typeof(j) = 'string' THEN\n RETURN j;\n END IF;\n\n IF jsonb_typeof(j) = 'object' THEN\n SELECT jsonb_build_object(\n 'op', key,\n 'args', cql1_to_cql2(value)\n ) INTO ret\n FROM jsonb_each(j)\n WHERE j IS NOT NULL;\n RETURN ret;\n END IF;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE STRICT;\n\nCREATE TABLE cql2_ops (\n op text PRIMARY KEY,\n template text,\n types text[]\n);\nINSERT INTO cql2_ops (op, template, types) VALUES\n ('eq', '%s = %s', NULL),\n ('neq', '%s != %s', NULL),\n ('ne', '%s != %s', NULL),\n ('!=', '%s != %s', NULL),\n ('<>', '%s != %s', NULL),\n ('lt', '%s < %s', NULL),\n ('lte', '%s <= %s', NULL),\n ('gt', '%s > %s', NULL),\n ('gte', '%s >= %s', NULL),\n ('le', '%s <= %s', NULL),\n ('ge', '%s >= %s', NULL),\n ('=', '%s = %s', NULL),\n ('<', '%s < %s', NULL),\n ('<=', '%s <= %s', NULL),\n ('>', '%s > %s', NULL),\n ('>=', '%s >= %s', NULL),\n ('like', '%s LIKE %s', NULL),\n ('ilike', '%s ILIKE %s', NULL),\n ('+', '%s + %s', NULL),\n ('-', '%s - %s', NULL),\n ('*', '%s * %s', NULL),\n ('/', '%s / %s', NULL),\n ('not', 'NOT (%s)', NULL),\n ('between', '%s BETWEEN %s AND %s', NULL),\n ('isnull', '%s IS NULL', NULL),\n ('upper', 'upper(%s)', NULL),\n ('lower', 'lower(%s)', NULL),\n ('casei', 'upper(%s)', NULL),\n ('accenti', 'unaccent(%s)', NULL)\nON CONFLICT (op) DO UPDATE\n SET\n template = EXCLUDED.template\n;\n\n\nCREATE OR REPLACE FUNCTION cql2_query(j jsonb, wrapper text DEFAULT NULL) RETURNS text AS $$\n#variable_conflict use_variable\nDECLARE\n args jsonb := j->'args';\n arg jsonb;\n op text := lower(j->>'op');\n cql2op RECORD;\n literal text;\n _wrapper text;\n leftarg text;\n rightarg text;\n prop text;\n extra_props bool := pgstac.additional_properties();\nBEGIN\n IF j IS NULL OR (op IS NOT NULL AND args IS NULL) THEN\n RETURN NULL;\n END IF;\n RAISE NOTICE 'CQL2_QUERY: %', j;\n\n -- check if all properties are represented in the queryables\n IF NOT extra_props THEN\n FOR prop IN\n SELECT DISTINCT p->>0\n FROM jsonb_path_query(j, 'strict $.**.property') p\n WHERE p->>0 NOT IN ('id', 'datetime', 'geometry', 'end_datetime', 'collection')\n LOOP\n IF (queryable(prop)).nulled_wrapper IS NULL THEN\n RAISE EXCEPTION 'Term % is not found in queryables.', prop;\n END IF;\n END LOOP;\n END IF;\n\n IF j ? 'filter' THEN\n RETURN cql2_query(j->'filter');\n END IF;\n\n IF j ? 'upper' THEN\n RETURN cql2_query(jsonb_build_object('op', 'upper', 'args', j->'upper'));\n END IF;\n\n IF j ? 'lower' THEN\n RETURN cql2_query(jsonb_build_object('op', 'lower', 'args', j->'lower'));\n END IF;\n\n -- Temporal Query\n IF op ilike 't_%' or op = 'anyinteracts' THEN\n RETURN temporal_op_query(op, args);\n END IF;\n\n -- If property is a timestamp convert it to text to use with\n -- general operators\n IF j ? 'timestamp' THEN\n RETURN format('%L::timestamptz', to_tstz(j->'timestamp'));\n END IF;\n IF j ? 'interval' THEN\n RAISE EXCEPTION 'Please use temporal operators when using intervals.';\n RETURN NONE;\n END IF;\n\n -- Spatial Query\n IF op ilike 's_%' or op = 'intersects' THEN\n RETURN spatial_op_query(op, args);\n END IF;\n\n IF op IN ('a_equals','a_contains','a_contained_by','a_overlaps') THEN\n IF args->0 ? 'property' THEN\n leftarg := format('to_text_array(%s)', (queryable(args->0->>'property')).path);\n END IF;\n IF args->1 ? 'property' THEN\n rightarg := format('to_text_array(%s)', (queryable(args->1->>'property')).path);\n END IF;\n RETURN FORMAT(\n '%s %s %s',\n COALESCE(leftarg, quote_literal(to_text_array(args->0))),\n CASE op\n WHEN 'a_equals' THEN '='\n WHEN 'a_contains' THEN '@>'\n WHEN 'a_contained_by' THEN '<@'\n WHEN 'a_overlaps' THEN '&&'\n END,\n COALESCE(rightarg, quote_literal(to_text_array(args->1)))\n );\n END IF;\n\n IF op = 'in' THEN\n RAISE NOTICE 'IN : % % %', args, jsonb_build_array(args->0), args->1;\n args := jsonb_build_array(args->0) || (args->1);\n RAISE NOTICE 'IN2 : %', args;\n END IF;\n\n\n\n IF op = 'between' THEN\n args = jsonb_build_array(\n args->0,\n args->1,\n args->2\n );\n END IF;\n\n -- Make sure that args is an array and run cql2_query on\n -- each element of the array\n RAISE NOTICE 'ARGS PRE: %', args;\n IF j ? 'args' THEN\n IF jsonb_typeof(args) != 'array' THEN\n args := jsonb_build_array(args);\n END IF;\n\n IF jsonb_path_exists(args, '$[*] ? (@.property == \"id\" || @.property == \"datetime\" || @.property == \"end_datetime\" || @.property == \"collection\")') THEN\n wrapper := NULL;\n ELSE\n -- if any of the arguments are a property, try to get the property_wrapper\n FOR arg IN SELECT jsonb_path_query(args, '$[*] ? (@.property != null)') LOOP\n RAISE NOTICE 'Arg: %', arg;\n wrapper := (queryable(arg->>'property')).nulled_wrapper;\n RAISE NOTICE 'Property: %, Wrapper: %', arg, wrapper;\n IF wrapper IS NOT NULL THEN\n EXIT;\n END IF;\n END LOOP;\n\n -- if the property was not in queryables, see if any args were numbers\n IF\n wrapper IS NULL\n AND jsonb_path_exists(args, '$[*] ? (@.type()==\"number\")')\n THEN\n wrapper := 'to_float';\n END IF;\n wrapper := coalesce(wrapper, 'to_text');\n END IF;\n\n SELECT jsonb_agg(cql2_query(a, wrapper))\n INTO args\n FROM jsonb_array_elements(args) a;\n END IF;\n RAISE NOTICE 'ARGS: %', args;\n\n IF op IN ('and', 'or') THEN\n RETURN\n format(\n '(%s)',\n array_to_string(to_text_array(args), format(' %s ', upper(op)))\n );\n END IF;\n\n IF op = 'in' THEN\n RAISE NOTICE 'IN -- % %', args->0, to_text(args->0);\n RETURN format(\n '%s IN (%s)',\n to_text(args->0),\n array_to_string((to_text_array(args))[2:], ',')\n );\n END IF;\n\n -- Look up template from cql2_ops\n IF j ? 'op' THEN\n SELECT * INTO cql2op FROM cql2_ops WHERE cql2_ops.op ilike op;\n IF FOUND THEN\n -- If specific index set in queryables for a property cast other arguments to that type\n\n RETURN format(\n cql2op.template,\n VARIADIC (to_text_array(args))\n );\n ELSE\n RAISE EXCEPTION 'Operator % Not Supported.', op;\n END IF;\n END IF;\n\n\n IF wrapper IS NOT NULL THEN\n RAISE NOTICE 'Wrapping % with %', j, wrapper;\n IF j ? 'property' THEN\n RETURN format('%I(%s)', wrapper, (queryable(j->>'property')).path);\n ELSE\n RETURN format('%I(%L)', wrapper, j);\n END IF;\n ELSIF j ? 'property' THEN\n RETURN quote_ident(j->>'property');\n END IF;\n\n RETURN quote_literal(to_text(j));\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION paging_dtrange(\n j jsonb\n) RETURNS tstzrange AS $$\nDECLARE\n op text;\n filter jsonb := j->'filter';\n dtrange tstzrange := tstzrange('-infinity'::timestamptz,'infinity'::timestamptz);\n sdate timestamptz := '-infinity'::timestamptz;\n edate timestamptz := 'infinity'::timestamptz;\n jpitem jsonb;\nBEGIN\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n END IF;\n IF NOT (filter @? '$.**.op ? (@ == \"or\" || @ == \"not\")') THEN\n FOR jpitem IN SELECT j FROM jsonb_path_query(filter,'strict $.** ? (@.args[*].property == \"datetime\")'::jsonpath) j LOOP\n op := lower(jpitem->>'op');\n dtrange := parse_dtrange(jpitem->'args'->1);\n IF op IN ('<=', 'lt', 'lte', '<', 'le', 't_before') THEN\n sdate := greatest(sdate,'-infinity');\n edate := least(edate, upper(dtrange));\n ELSIF op IN ('>=', '>', 'gt', 'gte', 'ge', 't_after') THEN\n edate := least(edate, 'infinity');\n sdate := greatest(sdate, lower(dtrange));\n ELSIF op IN ('=', 'eq') THEN\n edate := least(edate, upper(dtrange));\n sdate := greatest(sdate, lower(dtrange));\n END IF;\n RAISE NOTICE '2 OP: %, ARGS: %, DTRANGE: %, SDATE: %, EDATE: %', op, jpitem->'args'->1, dtrange, sdate, edate;\n END LOOP;\n END IF;\n IF sdate > edate THEN\n RETURN 'empty'::tstzrange;\n END IF;\n RETURN tstzrange(sdate,edate, '[]');\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT SET TIME ZONE 'UTC';\n\nCREATE OR REPLACE FUNCTION paging_collections(\n IN j jsonb\n) RETURNS text[] AS $$\nDECLARE\n filter jsonb := j->'filter';\n jpitem jsonb;\n op text;\n args jsonb;\n arg jsonb;\n collections text[];\nBEGIN\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n END IF;\n IF NOT (filter @? '$.**.op ? (@ == \"or\" || @ == \"not\")') THEN\n FOR jpitem IN SELECT j FROM jsonb_path_query(filter,'strict $.** ? (@.args[*].property == \"collection\")'::jsonpath) j LOOP\n RAISE NOTICE 'JPITEM: %', jpitem;\n op := jpitem->>'op';\n args := jpitem->'args';\n IF op IN ('=', 'eq', 'in') THEN\n FOR arg IN SELECT a FROM jsonb_array_elements(args) a LOOP\n IF jsonb_typeof(arg) IN ('string', 'array') THEN\n RAISE NOTICE 'arg: %, collections: %', arg, collections;\n IF collections IS NULL OR collections = '{}'::text[] THEN\n collections := to_text_array(arg);\n ELSE\n collections := array_intersection(collections, to_text_array(arg));\n END IF;\n END IF;\n END LOOP;\n END IF;\n END LOOP;\n END IF;\n IF collections = '{}'::text[] THEN\n RETURN NULL;\n END IF;\n RETURN collections;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\nCREATE TABLE items (\n id text NOT NULL,\n geometry geometry NOT NULL,\n collection text NOT NULL,\n datetime timestamptz NOT NULL,\n end_datetime timestamptz NOT NULL,\n content JSONB NOT NULL,\n private jsonb\n)\nPARTITION BY LIST (collection)\n;\n\nCREATE INDEX \"datetime_idx\" ON items USING BTREE (datetime DESC, end_datetime ASC);\nCREATE INDEX \"geometry_idx\" ON items USING GIST (geometry);\n\nCREATE STATISTICS datetime_stats (dependencies) on datetime, end_datetime from items;\n\nALTER TABLE items ADD CONSTRAINT items_collections_fk FOREIGN KEY (collection) REFERENCES collections(id) ON DELETE CASCADE DEFERRABLE;\n\nCREATE OR REPLACE FUNCTION partition_after_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p text;\n t timestamptz := clock_timestamp();\nBEGIN\n RAISE NOTICE 'Updating partition stats %', t;\n FOR p IN SELECT DISTINCT partition\n FROM newdata n JOIN partition_sys_meta p\n ON (n.collection=p.collection AND n.datetime <@ p.partition_dtrange)\n LOOP\n PERFORM run_or_queue(format('SELECT update_partition_stats(%L, %L);', p, true));\n END LOOP;\n IF TG_OP IN ('DELETE','UPDATE') THEN\n DELETE FROM format_item_cache c USING newdata n WHERE c.collection = n.collection AND c.id = n.id;\n END IF;\n RAISE NOTICE 't: % %', t, clock_timestamp() - t;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE TRIGGER items_after_insert_trigger\nAFTER INSERT ON items\nREFERENCING NEW TABLE AS newdata\nFOR EACH STATEMENT\nEXECUTE FUNCTION partition_after_triggerfunc();\n\nCREATE TRIGGER items_after_update_trigger\nAFTER DELETE ON items\nREFERENCING OLD TABLE AS newdata\nFOR EACH STATEMENT\nEXECUTE FUNCTION partition_after_triggerfunc();\n\nCREATE TRIGGER items_after_delete_trigger\nAFTER UPDATE ON items\nREFERENCING NEW TABLE AS newdata\nFOR EACH STATEMENT\nEXECUTE FUNCTION partition_after_triggerfunc();\n\n\nCREATE OR REPLACE FUNCTION content_slim(_item jsonb) RETURNS jsonb AS $$\n SELECT strip_jsonb(_item - '{id,geometry,collection,type}'::text[], collection_base_item(_item->>'collection')) - '{id,geometry,collection,type}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_dehydrate(content jsonb) RETURNS items AS $$\n SELECT\n content->>'id' as id,\n stac_geom(content) as geometry,\n content->>'collection' as collection,\n stac_datetime(content) as datetime,\n stac_end_datetime(content) as end_datetime,\n content_slim(content) as content,\n null::jsonb as private\n ;\n$$ LANGUAGE SQL STABLE;\n\nCREATE OR REPLACE FUNCTION include_field(f text, fields jsonb DEFAULT '{}'::jsonb) RETURNS boolean AS $$\nDECLARE\n includes jsonb := fields->'include';\n excludes jsonb := fields->'exclude';\nBEGIN\n IF f IS NULL THEN\n RETURN NULL;\n END IF;\n\n\n IF\n jsonb_typeof(excludes) = 'array'\n AND jsonb_array_length(excludes)>0\n AND excludes ? f\n THEN\n RETURN FALSE;\n END IF;\n\n IF\n (\n jsonb_typeof(includes) = 'array'\n AND jsonb_array_length(includes) > 0\n AND includes ? f\n ) OR\n (\n includes IS NULL\n OR jsonb_typeof(includes) = 'null'\n OR jsonb_array_length(includes) = 0\n )\n THEN\n RETURN TRUE;\n END IF;\n\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nDROP FUNCTION IF EXISTS content_hydrate(jsonb, jsonb, jsonb);\nCREATE OR REPLACE FUNCTION content_hydrate(\n _item jsonb,\n _base_item jsonb,\n fields jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\n SELECT merge_jsonb(\n jsonb_fields(_item, fields),\n jsonb_fields(_base_item, fields)\n );\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION content_hydrate(_item items, _collection collections, fields jsonb DEFAULT '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\n content jsonb;\n base_item jsonb := _collection.base_item;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry, 20)::jsonb;\n END IF;\n output := content_hydrate(\n jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'collection', _item.collection,\n 'type', 'Feature'\n ) || _item.content,\n _collection.base_item,\n fields\n );\n\n RETURN output;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_nonhydrated(\n _item items,\n fields jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry, 20)::jsonb;\n END IF;\n output := jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'collection', _item.collection,\n 'type', 'Feature'\n ) || _item.content;\n RETURN output;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_hydrate(_item items, fields jsonb DEFAULT '{}'::jsonb) RETURNS jsonb AS $$\n SELECT content_hydrate(\n _item,\n (SELECT c FROM collections c WHERE id=_item.collection LIMIT 1),\n fields\n );\n$$ LANGUAGE SQL STABLE;\n\n\nCREATE UNLOGGED TABLE items_staging (\n content JSONB NOT NULL\n);\nCREATE UNLOGGED TABLE items_staging_ignore (\n content JSONB NOT NULL\n);\nCREATE UNLOGGED TABLE items_staging_upsert (\n content JSONB NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION items_staging_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p record;\n _partitions text[];\n part text;\n ts timestamptz := clock_timestamp();\n nrows int;\nBEGIN\n RAISE NOTICE 'Creating Partitions. %', clock_timestamp() - ts;\n\n FOR part IN WITH t AS (\n SELECT\n n.content->>'collection' as collection,\n stac_daterange(n.content->'properties') as dtr,\n partition_trunc\n FROM newdata n JOIN collections ON (n.content->>'collection'=collections.id)\n ), p AS (\n SELECT\n collection,\n COALESCE(date_trunc(partition_trunc::text, lower(dtr)),'-infinity') as d,\n tstzrange(min(lower(dtr)),max(lower(dtr)),'[]') as dtrange,\n tstzrange(min(upper(dtr)),max(upper(dtr)),'[]') as edtrange\n FROM t\n GROUP BY 1,2\n ) SELECT check_partition(collection, dtrange, edtrange) FROM p LOOP\n RAISE NOTICE 'Partition %', part;\n END LOOP;\n\n RAISE NOTICE 'Creating temp table with data to be added. %', clock_timestamp() - ts;\n DROP TABLE IF EXISTS tmpdata;\n CREATE TEMP TABLE tmpdata ON COMMIT DROP AS\n SELECT\n (content_dehydrate(content)).*\n FROM newdata;\n GET DIAGNOSTICS nrows = ROW_COUNT;\n RAISE NOTICE 'Added % rows to tmpdata. %', nrows, clock_timestamp() - ts;\n\n RAISE NOTICE 'Doing the insert. %', clock_timestamp() - ts;\n IF TG_TABLE_NAME = 'items_staging' THEN\n INSERT INTO items\n SELECT * FROM tmpdata;\n GET DIAGNOSTICS nrows = ROW_COUNT;\n RAISE NOTICE 'Inserted % rows to items. %', nrows, clock_timestamp() - ts;\n ELSIF TG_TABLE_NAME = 'items_staging_ignore' THEN\n INSERT INTO items\n SELECT * FROM tmpdata\n ON CONFLICT DO NOTHING;\n GET DIAGNOSTICS nrows = ROW_COUNT;\n RAISE NOTICE 'Inserted % rows to items. %', nrows, clock_timestamp() - ts;\n ELSIF TG_TABLE_NAME = 'items_staging_upsert' THEN\n DELETE FROM items i USING tmpdata s\n WHERE\n i.id = s.id\n AND i.collection = s.collection\n AND i IS DISTINCT FROM s\n ;\n GET DIAGNOSTICS nrows = ROW_COUNT;\n RAISE NOTICE 'Deleted % rows from items. %', nrows, clock_timestamp() - ts;\n INSERT INTO items AS t\n SELECT * FROM tmpdata\n ON CONFLICT DO NOTHING;\n GET DIAGNOSTICS nrows = ROW_COUNT;\n RAISE NOTICE 'Inserted % rows to items. %', nrows, clock_timestamp() - ts;\n END IF;\n\n RAISE NOTICE 'Deleting data from staging table. %', clock_timestamp() - ts;\n DELETE FROM items_staging;\n RAISE NOTICE 'Done. %', clock_timestamp() - ts;\n\n RETURN NULL;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER items_staging_insert_trigger AFTER INSERT ON items_staging REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\nCREATE TRIGGER items_staging_insert_ignore_trigger AFTER INSERT ON items_staging_ignore REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\nCREATE TRIGGER items_staging_insert_upsert_trigger AFTER INSERT ON items_staging_upsert REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\n\nCREATE OR REPLACE FUNCTION item_by_id(_id text, _collection text DEFAULT NULL) RETURNS items AS\n$$\nDECLARE\n i items%ROWTYPE;\nBEGIN\n SELECT * INTO i FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection) LIMIT 1;\n RETURN i;\nEND;\n$$ LANGUAGE PLPGSQL STABLE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION get_item(_id text, _collection text DEFAULT NULL) RETURNS jsonb AS $$\n SELECT content_hydrate(items) FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection);\n$$ LANGUAGE SQL STABLE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION delete_item(_id text, _collection text DEFAULT NULL) RETURNS VOID AS $$\nDECLARE\nout items%ROWTYPE;\nBEGIN\n DELETE FROM items WHERE id = _id AND (_collection IS NULL OR collection=_collection) RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n--/*\nCREATE OR REPLACE FUNCTION create_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION update_item(content jsonb) RETURNS VOID AS $$\nDECLARE\n old items %ROWTYPE;\n out items%ROWTYPE;\nBEGIN\n PERFORM delete_item(content->>'id', content->>'collection');\n PERFORM create_item(content);\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION create_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION collection_bbox(id text) RETURNS jsonb AS $$\n SELECT (replace(replace(replace(st_extent(geometry)::text,'BOX(','[['),')',']]'),' ',','))::jsonb\n FROM items WHERE collection=$1;\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION collection_temporal_extent(id text) RETURNS jsonb AS $$\n SELECT to_jsonb(array[array[min(datetime), max(datetime)]])\n FROM items WHERE collection=$1;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection_extents() RETURNS VOID AS $$\nUPDATE collections\n SET content = jsonb_set_lax(\n content,\n '{extent}'::text[],\n collection_extent(id, FALSE),\n true,\n 'use_json_null'\n )\n;\n$$ LANGUAGE SQL;\nCREATE TABLE partition_stats (\n partition text PRIMARY KEY,\n dtrange tstzrange,\n edtrange tstzrange,\n spatial geometry,\n last_updated timestamptz,\n keys text[]\n) WITH (FILLFACTOR=90);\n\nCREATE INDEX partitions_range_idx ON partition_stats USING GIST(dtrange);\n\n\nCREATE OR REPLACE FUNCTION constraint_tstzrange(expr text) RETURNS tstzrange AS $$\n WITH t AS (\n SELECT regexp_matches(\n expr,\n E'\\\\(''\\([0-9 :+-]*\\)''\\\\).*\\\\(''\\([0-9 :+-]*\\)''\\\\)'\n ) AS m\n ) SELECT tstzrange(m[1]::timestamptz, m[2]::timestamptz) FROM t\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE STRICT;\n\nCREATE OR REPLACE FUNCTION dt_constraint(coid oid, OUT dt tstzrange, OUT edt tstzrange) RETURNS RECORD AS $$\nDECLARE\n expr text := pg_get_constraintdef(coid);\n matches timestamptz[];\nBEGIN\n IF expr LIKE '%NULL%' THEN\n dt := tstzrange(null::timestamptz, null::timestamptz);\n edt := tstzrange(null::timestamptz, null::timestamptz);\n RETURN;\n END IF;\n WITH f AS (SELECT (regexp_matches(expr, E'([0-9]{4}-[0-1][0-9]-[0-3][0-9] [0-2][0-9]:[0-5][0-9]:[0-5][0-9]\\.?[0-9]*)', 'g'))[1] f)\n SELECT array_agg(f::timestamptz) INTO matches FROM f;\n IF cardinality(matches) = 4 THEN\n dt := tstzrange(matches[1], matches[2],'[]');\n edt := tstzrange(matches[3], matches[4], '[]');\n RETURN;\n ELSIF cardinality(matches) = 2 THEN\n edt := tstzrange(matches[1], matches[2],'[]');\n RETURN;\n END IF;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\nCREATE OR REPLACE VIEW partition_sys_meta AS\nSELECT\n (parse_ident(relid::text))[cardinality(parse_ident(relid::text))] as partition,\n replace(replace(CASE WHEN level = 1 THEN pg_get_expr(c.relpartbound, c.oid)\n ELSE pg_get_expr(parent.relpartbound, parent.oid)\n END, 'FOR VALUES IN (''',''), ''')','') AS collection,\n level,\n c.reltuples,\n c.relhastriggers,\n COALESCE(constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as partition_dtrange,\n COALESCE((dt_constraint(edt.oid)).dt, constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as constraint_dtrange,\n COALESCE((dt_constraint(edt.oid)).edt, tstzrange('-infinity', 'infinity','[]')) as constraint_edtrange\nFROM\n pg_partition_tree('items')\n JOIN pg_class c ON (relid::regclass = c.oid)\n JOIN pg_class parent ON (parentrelid::regclass = parent.oid AND isleaf)\n LEFT JOIN pg_constraint edt ON (conrelid=c.oid AND contype='c')\nWHERE isleaf\n;\n\nCREATE OR REPLACE VIEW partitions_view AS\nSELECT\n (parse_ident(relid::text))[cardinality(parse_ident(relid::text))] as partition,\n replace(replace(CASE WHEN level = 1 THEN pg_get_expr(c.relpartbound, c.oid)\n ELSE pg_get_expr(parent.relpartbound, parent.oid)\n END, 'FOR VALUES IN (''',''), ''')','') AS collection,\n level,\n c.reltuples,\n c.relhastriggers,\n COALESCE(pgstac.constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as partition_dtrange,\n COALESCE((pgstac.dt_constraint(edt.oid)).dt, pgstac.constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), tstzrange('-infinity', 'infinity','[]')) as constraint_dtrange,\n COALESCE((pgstac.dt_constraint(edt.oid)).edt, tstzrange('-infinity', 'infinity','[]')) as constraint_edtrange,\n dtrange,\n edtrange,\n spatial,\n last_updated\nFROM\n pg_partition_tree('pgstac.items')\n JOIN pg_class c ON (relid::regclass = c.oid)\n JOIN pg_class parent ON (parentrelid::regclass = parent.oid AND isleaf)\n LEFT JOIN pg_constraint edt ON (conrelid=c.oid AND contype='c')\n LEFT JOIN pgstac.partition_stats ON ((parse_ident(relid::text))[cardinality(parse_ident(relid::text))]=partition)\nWHERE isleaf\n;\n\nCREATE MATERIALIZED VIEW partitions AS\nSELECT * FROM partitions_view;\nCREATE UNIQUE INDEX ON partitions (partition);\n\nCREATE MATERIALIZED VIEW partition_steps AS\nSELECT\n partition as name,\n date_trunc('month',lower(partition_dtrange)) as sdate,\n date_trunc('month', upper(partition_dtrange)) + '1 month'::interval as edate\n FROM partitions_view WHERE partition_dtrange IS NOT NULL AND partition_dtrange != 'empty'::tstzrange\n ORDER BY dtrange ASC\n;\n\n\nCREATE OR REPLACE FUNCTION update_partition_stats_q(_partition text, istrigger boolean default false) RETURNS VOID AS $$\nDECLARE\nBEGIN\n PERFORM run_or_queue(\n format('SELECT update_partition_stats(%L, %L);', _partition, istrigger)\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION update_partition_stats(_partition text, istrigger boolean default false) RETURNS VOID AS $$\nDECLARE\n dtrange tstzrange;\n edtrange tstzrange;\n cdtrange tstzrange;\n cedtrange tstzrange;\n extent geometry;\n collection text;\nBEGIN\n RAISE NOTICE 'Updating stats for %.', _partition;\n EXECUTE format(\n $q$\n SELECT\n tstzrange(min(datetime), max(datetime),'[]'),\n tstzrange(min(end_datetime), max(end_datetime), '[]')\n FROM %I\n $q$,\n _partition\n ) INTO dtrange, edtrange;\n extent := st_estimatedextent('pgstac', _partition, 'geometry');\n RAISE DEBUG 'Estimated Extent: %', extent;\n INSERT INTO partition_stats (partition, dtrange, edtrange, spatial, last_updated)\n SELECT _partition, dtrange, edtrange, extent, now()\n ON CONFLICT (partition) DO\n UPDATE SET\n dtrange=EXCLUDED.dtrange,\n edtrange=EXCLUDED.edtrange,\n spatial=EXCLUDED.spatial,\n last_updated=EXCLUDED.last_updated\n ;\n\n SELECT\n constraint_dtrange, constraint_edtrange, pv.collection\n INTO cdtrange, cedtrange, collection\n FROM partitions_view pv WHERE partition = _partition;\n REFRESH MATERIALIZED VIEW partitions;\n REFRESH MATERIALIZED VIEW partition_steps;\n\n\n RAISE NOTICE 'Checking if we need to modify constraints...';\n RAISE NOTICE 'cdtrange: % dtrange: % cedtrange: % edtrange: %',cdtrange, dtrange, cedtrange, edtrange;\n IF\n (cdtrange IS DISTINCT FROM dtrange OR edtrange IS DISTINCT FROM cedtrange)\n AND NOT istrigger\n THEN\n RAISE NOTICE 'Modifying Constraints';\n RAISE NOTICE 'Existing % %', cdtrange, cedtrange;\n RAISE NOTICE 'New % %', dtrange, edtrange;\n PERFORM drop_table_constraints(_partition);\n PERFORM create_table_constraints(_partition, dtrange, edtrange);\n REFRESH MATERIALIZED VIEW partitions;\n REFRESH MATERIALIZED VIEW partition_steps;\n END IF;\n RAISE NOTICE 'Checking if we need to update collection extents.';\n IF get_setting_bool('update_collection_extent') THEN\n RAISE NOTICE 'updating collection extent for %', collection;\n PERFORM run_or_queue(format($q$\n UPDATE collections\n SET content = jsonb_set_lax(\n content,\n '{extent}'::text[],\n collection_extent(%L, FALSE),\n true,\n 'use_json_null'\n ) WHERE id=%L\n ;\n $q$, collection, collection));\n ELSE\n RAISE NOTICE 'Not updating collection extent for %', collection;\n END IF;\n\nEND;\n$$ LANGUAGE PLPGSQL STRICT SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION partition_name( IN collection text, IN dt timestamptz, OUT partition_name text, OUT partition_range tstzrange) AS $$\nDECLARE\n c RECORD;\n parent_name text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n parent_name := format('_items_%s', c.key);\n\n\n IF c.partition_trunc = 'year' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYY'));\n ELSIF c.partition_trunc = 'month' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYYMM'));\n ELSE\n partition_name := parent_name;\n partition_range := tstzrange('-infinity'::timestamptz, 'infinity'::timestamptz, '[]');\n END IF;\n IF partition_range IS NULL THEN\n partition_range := tstzrange(\n date_trunc(c.partition_trunc::text, dt),\n date_trunc(c.partition_trunc::text, dt) + concat('1 ', c.partition_trunc)::interval\n );\n END IF;\n RETURN;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION drop_table_constraints(t text) RETURNS text AS $$\nDECLARE\n q text;\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM partitions_view WHERE partition=t) THEN\n RETURN NULL;\n END IF;\n FOR q IN SELECT FORMAT(\n $q$\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n $q$,\n t,\n conname\n ) FROM pg_constraint\n WHERE conrelid=t::regclass::oid AND contype='c'\n LOOP\n EXECUTE q;\n END LOOP;\n RETURN t;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE OR REPLACE FUNCTION create_table_constraints(t text, _dtrange tstzrange, _edtrange tstzrange) RETURNS text AS $$\nDECLARE\n q text;\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM partitions_view WHERE partition=t) THEN\n RETURN NULL;\n END IF;\n RAISE NOTICE 'Creating Table Constraints for % % %', t, _dtrange, _edtrange;\n IF _dtrange = 'empty' AND _edtrange = 'empty' THEN\n q :=format(\n $q$\n DO $block$\n BEGIN\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n ALTER TABLE %I\n ADD CONSTRAINT %I\n CHECK (((datetime IS NULL) AND (end_datetime IS NULL))) NOT VALID\n ;\n ALTER TABLE %I\n VALIDATE CONSTRAINT %I\n ;\n\n\n\n EXCEPTION WHEN others THEN\n RAISE WARNING '%%, Issue Altering Constraints. Please run update_partition_stats(%I)', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n $block$;\n $q$,\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t),\n t\n );\n ELSE\n q :=format(\n $q$\n DO $block$\n BEGIN\n\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n ALTER TABLE %I\n ADD CONSTRAINT %I\n CHECK (\n (datetime >= %L)\n AND (datetime <= %L)\n AND (end_datetime >= %L)\n AND (end_datetime <= %L)\n ) NOT VALID\n ;\n ALTER TABLE %I\n VALIDATE CONSTRAINT %I\n ;\n\n\n\n EXCEPTION WHEN others THEN\n RAISE WARNING '%%, Issue Altering Constraints. Please run update_partition_stats(%I)', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n $block$;\n $q$,\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t),\n lower(_dtrange),\n upper(_dtrange),\n lower(_edtrange),\n upper(_edtrange),\n t,\n format('%s_dt', t),\n t\n );\n END IF;\n PERFORM run_or_queue(q);\n RETURN t;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION check_partition(\n _collection text,\n _dtrange tstzrange,\n _edtrange tstzrange\n) RETURNS text AS $$\nDECLARE\n c RECORD;\n pm RECORD;\n _partition_name text;\n _partition_dtrange tstzrange;\n _constraint_dtrange tstzrange;\n _constraint_edtrange tstzrange;\n q text;\n deferrable_q text;\n err_context text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=_collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', _collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n\n IF c.partition_trunc IS NOT NULL THEN\n _partition_dtrange := tstzrange(\n date_trunc(c.partition_trunc, lower(_dtrange)),\n date_trunc(c.partition_trunc, lower(_dtrange)) + (concat('1 ', c.partition_trunc))::interval,\n '[)'\n );\n ELSE\n _partition_dtrange := '[-infinity, infinity]'::tstzrange;\n END IF;\n\n IF NOT _partition_dtrange @> _dtrange THEN\n RAISE EXCEPTION 'dtrange % is greater than the partition size % for collection %', _dtrange, c.partition_trunc, _collection;\n END IF;\n\n\n IF c.partition_trunc = 'year' THEN\n _partition_name := format('_items_%s_%s', c.key, to_char(lower(_partition_dtrange),'YYYY'));\n ELSIF c.partition_trunc = 'month' THEN\n _partition_name := format('_items_%s_%s', c.key, to_char(lower(_partition_dtrange),'YYYYMM'));\n ELSE\n _partition_name := format('_items_%s', c.key);\n END IF;\n\n SELECT * INTO pm FROM partition_sys_meta WHERE collection=_collection AND partition_dtrange @> _dtrange;\n IF FOUND THEN\n RAISE NOTICE '% % %', _edtrange, _dtrange, pm;\n _constraint_edtrange :=\n tstzrange(\n least(\n lower(_edtrange),\n nullif(lower(pm.constraint_edtrange), '-infinity')\n ),\n greatest(\n upper(_edtrange),\n nullif(upper(pm.constraint_edtrange), 'infinity')\n ),\n '[]'\n );\n _constraint_dtrange :=\n tstzrange(\n least(\n lower(_dtrange),\n nullif(lower(pm.constraint_dtrange), '-infinity')\n ),\n greatest(\n upper(_dtrange),\n nullif(upper(pm.constraint_dtrange), 'infinity')\n ),\n '[]'\n );\n\n IF pm.constraint_edtrange @> _edtrange AND pm.constraint_dtrange @> _dtrange THEN\n RETURN pm.partition;\n ELSE\n PERFORM drop_table_constraints(_partition_name);\n END IF;\n ELSE\n _constraint_edtrange := _edtrange;\n _constraint_dtrange := _dtrange;\n END IF;\n RAISE NOTICE 'EXISTING CONSTRAINTS % %, NEW % %', pm.constraint_dtrange, pm.constraint_edtrange, _constraint_dtrange, _constraint_edtrange;\n RAISE NOTICE 'Creating partition % %', _partition_name, _partition_dtrange;\n IF c.partition_trunc IS NULL THEN\n q := format(\n $q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n GRANT ALL ON %I to pgstac_ingest;\n $q$,\n _partition_name,\n _collection,\n concat(_partition_name,'_pk'),\n _partition_name,\n _partition_name\n );\n ELSE\n q := format(\n $q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L) PARTITION BY RANGE (datetime);\n CREATE TABLE IF NOT EXISTS %I partition OF %I FOR VALUES FROM (%L) TO (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n GRANT ALL ON %I TO pgstac_ingest;\n $q$,\n format('_items_%s', c.key),\n _collection,\n _partition_name,\n format('_items_%s', c.key),\n lower(_partition_dtrange),\n upper(_partition_dtrange),\n format('%s_pk', _partition_name),\n _partition_name,\n _partition_name\n );\n END IF;\n\n BEGIN\n EXECUTE q;\n EXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', _partition_name;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\n END;\n PERFORM maintain_partitions(_partition_name);\n PERFORM update_partition_stats_q(_partition_name, true);\n REFRESH MATERIALIZED VIEW partitions;\n REFRESH MATERIALIZED VIEW partition_steps;\n RETURN _partition_name;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION repartition(_collection text, _partition_trunc text, triggered boolean DEFAULT FALSE) RETURNS text AS $$\nDECLARE\n c RECORD;\n q text;\n from_trunc text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=_collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', _collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n IF triggered THEN\n RAISE NOTICE 'Converting % to % partitioning via Trigger', _collection, _partition_trunc;\n ELSE\n RAISE NOTICE 'Converting % from using % to % partitioning', _collection, c.partition_trunc, _partition_trunc;\n IF c.partition_trunc IS NOT DISTINCT FROM _partition_trunc THEN\n RAISE NOTICE 'Collection % already set to use partition by %', _collection, _partition_trunc;\n RETURN _collection;\n END IF;\n END IF;\n\n IF EXISTS (SELECT 1 FROM partitions_view WHERE collection=_collection LIMIT 1) THEN\n EXECUTE format(\n $q$\n CREATE TEMP TABLE changepartitionstaging ON COMMIT DROP AS SELECT * FROM %I;\n DROP TABLE IF EXISTS %I CASCADE;\n WITH p AS (\n SELECT\n collection,\n CASE\n WHEN %L IS NULL THEN '-infinity'::timestamptz\n ELSE date_trunc(%L, datetime)\n END as d,\n tstzrange(min(datetime),max(datetime),'[]') as dtrange,\n tstzrange(min(end_datetime),max(end_datetime),'[]') as edtrange\n FROM changepartitionstaging\n GROUP BY 1,2\n ) SELECT check_partition(collection, dtrange, edtrange) FROM p;\n INSERT INTO items SELECT * FROM changepartitionstaging;\n DROP TABLE changepartitionstaging;\n $q$,\n concat('_items_', c.key),\n concat('_items_', c.key),\n c.partition_trunc,\n c.partition_trunc\n );\n END IF;\n RETURN _collection;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE OR REPLACE FUNCTION collections_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n q text;\n partition_name text := format('_items_%s', NEW.key);\n partition_exists boolean := false;\n partition_empty boolean := true;\n err_context text;\n loadtemp boolean := FALSE;\nBEGIN\n RAISE NOTICE 'Collection Trigger. % %', NEW.id, NEW.key;\n IF TG_OP = 'UPDATE' AND NEW.partition_trunc IS DISTINCT FROM OLD.partition_trunc THEN\n PERFORM repartition(NEW.id, NEW.partition_trunc, TRUE);\n END IF;\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER collections_trigger AFTER\nINSERT\nOR\nUPDATE ON collections\nFOR EACH ROW EXECUTE FUNCTION collections_trigger_func();\n\nCREATE OR REPLACE FUNCTION chunker(\n IN _where text,\n OUT s timestamptz,\n OUT e timestamptz\n) RETURNS SETOF RECORD AS $$\nDECLARE\n explain jsonb;\nBEGIN\n IF _where IS NULL THEN\n _where := ' TRUE ';\n END IF;\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s;', _where)\n INTO explain;\n RAISE DEBUG 'EXPLAIN: %', explain;\n\n RETURN QUERY\n WITH t AS (\n SELECT j->>0 as p FROM\n jsonb_path_query(\n explain,\n 'strict $.**.\"Relation Name\" ? (@ != null)'\n ) j\n ),\n parts AS (\n SELECT sdate, edate FROM t JOIN partition_steps ON (t.p = name)\n ),\n times AS (\n SELECT sdate FROM parts\n UNION\n SELECT edate FROM parts\n ),\n uniq AS (\n SELECT DISTINCT sdate FROM times ORDER BY sdate\n ),\n last AS (\n SELECT sdate, lead(sdate, 1) over () as edate FROM uniq\n )\n SELECT sdate, edate FROM last WHERE edate IS NOT NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION partition_queries(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN partitions text[] DEFAULT NULL\n) RETURNS SETOF text AS $$\nDECLARE\n query text;\n sdate timestamptz;\n edate timestamptz;\nBEGIN\nIF _where IS NULL OR trim(_where) = '' THEN\n _where = ' TRUE ';\nEND IF;\nRAISE NOTICE 'Getting chunks for % %', _where, _orderby;\nIF _orderby ILIKE 'datetime d%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 DESC LOOP\n RETURN NEXT format($q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n $q$,\n sdate,\n edate,\n _where,\n _orderby\n );\n END LOOP;\nELSIF _orderby ILIKE 'datetime a%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 ASC LOOP\n RETURN NEXT format($q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n $q$,\n sdate,\n edate,\n _where,\n _orderby\n );\n END LOOP;\nELSE\n query := format($q$\n SELECT * FROM items\n WHERE %s\n ORDER BY %s\n $q$, _where, _orderby\n );\n\n RETURN NEXT query;\n RETURN;\nEND IF;\n\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION partition_query_view(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN _limit int DEFAULT 10\n) RETURNS text AS $$\n WITH p AS (\n SELECT * FROM partition_queries(_where, _orderby) p\n )\n SELECT\n CASE WHEN EXISTS (SELECT 1 FROM p) THEN\n (SELECT format($q$\n SELECT * FROM (\n %s\n ) total LIMIT %s\n $q$,\n string_agg(\n format($q$ SELECT * FROM ( %s ) AS sub $q$, p),\n '\n UNION ALL\n '\n ),\n _limit\n ))\n ELSE NULL\n END FROM p;\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION q_to_tsquery (jinput jsonb)\n RETURNS tsquery\n AS $$\nDECLARE\n input text;\n processed_text text;\n temp_text text;\n quote_array text[];\n placeholder text := '@QUOTE@';\nBEGIN\n IF jsonb_typeof(jinput) = 'string' THEN\n input := jinput->>0;\n ELSIF jsonb_typeof(jinput) = 'array' THEN\n input := array_to_string(\n array(select jsonb_array_elements_text(jinput)),\n ' OR '\n );\n ELSE\n RAISE EXCEPTION 'Input must be a string or an array of strings.';\n END IF;\n -- Extract all quoted phrases and store in array\n quote_array := regexp_matches(input, '\"[^\"]*\"', 'g');\n\n -- Replace each quoted part with a unique placeholder if there are any quoted phrases\n IF array_length(quote_array, 1) IS NOT NULL THEN\n processed_text := input;\n FOR i IN array_lower(quote_array, 1) .. array_upper(quote_array, 1) LOOP\n processed_text := replace(processed_text, quote_array[i], placeholder || i || placeholder);\n END LOOP;\n ELSE\n processed_text := input;\n END IF;\n\n -- Replace non-quoted text using regular expressions\n\n -- , -> |\n processed_text := regexp_replace(processed_text, ',(?=(?:[^\"]*\"[^\"]*\")*[^\"]*$)', ' | ', 'g');\n\n -- and -> &\n processed_text := regexp_replace(processed_text, '\\s+AND\\s+', ' & ', 'gi');\n\n -- or -> |\n processed_text := regexp_replace(processed_text, '\\s+OR\\s+', ' | ', 'gi');\n\n -- + ->\n processed_text := regexp_replace(processed_text, '^\\s*\\+([a-zA-Z0-9_]+)', '\\1', 'g'); -- +term at start\n processed_text := regexp_replace(processed_text, '\\s*\\+([a-zA-Z0-9_]+)', ' & \\1', 'g'); -- +term elsewhere\n\n -- - -> !\n processed_text := regexp_replace(processed_text, '^\\s*\\-([a-zA-Z0-9_]+)', '! \\1', 'g'); -- -term at start\n processed_text := regexp_replace(processed_text, '\\s*\\-([a-zA-Z0-9_]+)', ' & ! \\1', 'g'); -- -term elsewhere\n\n -- terms separated with spaces are assumed to represent adjacent terms. loop through these\n -- occurrences and replace them with the adjacency operator (<->)\n LOOP\n temp_text := regexp_replace(processed_text, '([a-zA-Z0-9_]+)\\s+([a-zA-Z0-9_]+)(?!\\s*[&|<>])', '\\1 <-> \\2', 'g');\n IF temp_text = processed_text THEN\n EXIT; -- No more replacements were made\n END IF;\n processed_text := temp_text;\n END LOOP;\n\n\n -- Replace placeholders back with quoted phrases if there were any\n IF array_length(quote_array, 1) IS NOT NULL THEN\n FOR i IN array_lower(quote_array, 1) .. array_upper(quote_array, 1) LOOP\n processed_text := replace(processed_text, placeholder || i || placeholder, '''' || substring(quote_array[i] from 2 for length(quote_array[i]) - 2) || '''');\n END LOOP;\n END IF;\n\n -- Print processed_text to the console for debugging purposes\n RAISE NOTICE 'processed_text: %', processed_text;\n\n RETURN to_tsquery('english', processed_text);\nEND;\n$$\nLANGUAGE plpgsql;\n\n\nCREATE OR REPLACE FUNCTION stac_search_to_where(j jsonb) RETURNS text AS $$\nDECLARE\n where_segments text[];\n _where text;\n dtrange tstzrange;\n collections text[];\n geom geometry;\n sdate timestamptz;\n edate timestamptz;\n filterlang text;\n filter jsonb := j->'filter';\n ft_query tsquery;\nBEGIN\n IF j ? 'ids' THEN\n where_segments := where_segments || format('id = ANY (%L) ', to_text_array(j->'ids'));\n END IF;\n\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n where_segments := where_segments || format('collection = ANY (%L) ', collections);\n END IF;\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n\n where_segments := where_segments || format(' datetime <= %L::timestamptz AND end_datetime >= %L::timestamptz ',\n edate,\n sdate\n );\n END IF;\n\n IF j ? 'q' THEN\n ft_query := q_to_tsquery(j->'q');\n where_segments := where_segments || format(\n $quote$\n (\n to_tsvector('english', content->'properties'->>'description') ||\n to_tsvector('english', coalesce(content->'properties'->>'title', '')) ||\n to_tsvector('english', coalesce(content->'properties'->>'keywords', ''))\n ) @@ %L\n $quote$,\n ft_query\n );\n END IF;\n\n geom := stac_geom(j);\n IF geom IS NOT NULL THEN\n where_segments := where_segments || format('st_intersects(geometry, %L)',geom);\n END IF;\n\n filterlang := COALESCE(\n j->>'filter-lang',\n get_setting('default_filter_lang', j->'conf')\n );\n IF NOT filter @? '$.**.op' THEN\n filterlang := 'cql-json';\n END IF;\n\n IF filterlang NOT IN ('cql-json','cql2-json') AND j ? 'filter' THEN\n RAISE EXCEPTION '% is not a supported filter-lang. Please use cql-json or cql2-json.', filterlang;\n END IF;\n\n IF j ? 'query' AND j ? 'filter' THEN\n RAISE EXCEPTION 'Can only use either query or filter at one time.';\n END IF;\n\n IF j ? 'query' THEN\n filter := query_to_cql2(j->'query');\n ELSIF filterlang = 'cql-json' THEN\n filter := cql1_to_cql2(filter);\n END IF;\n RAISE NOTICE 'FILTER: %', filter;\n where_segments := where_segments || cql2_query(filter);\n IF cardinality(where_segments) < 1 THEN\n RETURN ' TRUE ';\n END IF;\n\n _where := array_to_string(array_remove(where_segments, NULL), ' AND ');\n\n IF _where IS NULL OR BTRIM(_where) = '' THEN\n RETURN ' TRUE ';\n END IF;\n RETURN _where;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION parse_sort_dir(_dir text, reverse boolean default false) RETURNS text AS $$\n WITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n ) SELECT\n CASE\n WHEN NOT reverse THEN d\n WHEN d = 'ASC' THEN 'DESC'\n WHEN d = 'DESC' THEN 'ASC'\n END\n FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION sort_dir_to_op(_dir text, prev boolean default false) RETURNS text AS $$\n WITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n ) SELECT\n CASE\n WHEN d = 'ASC' AND prev THEN '<='\n WHEN d = 'DESC' AND prev THEN '>='\n WHEN d = 'ASC' THEN '>='\n WHEN d = 'DESC' THEN '<='\n END\n FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION sort_sqlorderby(\n _search jsonb DEFAULT NULL,\n reverse boolean DEFAULT FALSE\n) RETURNS text AS $$\n WITH sortby AS (\n SELECT coalesce(_search->'sortby','[{\"field\":\"datetime\", \"direction\":\"desc\"}]') as sort\n ), withid AS (\n SELECT CASE\n WHEN sort @? '$[*] ? (@.field == \"id\")' THEN sort\n ELSE sort || '[{\"field\":\"id\", \"direction\":\"desc\"}]'::jsonb\n END as sort\n FROM sortby\n ), withid_rows AS (\n SELECT jsonb_array_elements(sort) as value FROM withid\n ),sorts AS (\n SELECT\n coalesce(\n (queryable(value->>'field')).expression\n ) as key,\n parse_sort_dir(value->>'direction', reverse) as dir\n FROM withid_rows\n )\n SELECT array_to_string(\n array_agg(concat(key, ' ', dir)),\n ', '\n ) FROM sorts;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_sort_dir(sort_item jsonb) RETURNS text AS $$\n SELECT CASE WHEN sort_item->>'direction' ILIKE 'desc%' THEN 'DESC' ELSE 'ASC' END;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION get_token_val_str(\n _field text,\n _item items\n) RETURNS text AS $$\nDECLARE\n q text;\n literal text;\nBEGIN\n q := format($q$ SELECT quote_literal(%s) FROM (SELECT $1.*) as r;$q$, _field);\n EXECUTE q INTO literal USING _item;\n RETURN literal;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n\nCREATE OR REPLACE FUNCTION get_token_record(IN _token text, OUT prev BOOLEAN, OUT item items) RETURNS RECORD AS $$\nDECLARE\n _itemid text := _token;\n _collectionid text;\nBEGIN\n IF _token IS NULL THEN\n RETURN;\n END IF;\n RAISE NOTICE 'Looking for token: %', _token;\n prev := FALSE;\n IF _token ILIKE 'prev:%' THEN\n _itemid := replace(_token, 'prev:','');\n prev := TRUE;\n ELSIF _token ILIKE 'next:%' THEN\n _itemid := replace(_token, 'next:', '');\n END IF;\n SELECT id INTO _collectionid FROM collections WHERE _itemid LIKE concat(id,':%');\n IF FOUND THEN\n _itemid := replace(_itemid, concat(_collectionid,':'), '');\n SELECT * INTO item FROM items WHERE id=_itemid AND collection=_collectionid;\n ELSE\n SELECT * INTO item FROM items WHERE id=_itemid;\n END IF;\n IF item IS NULL THEN\n RAISE EXCEPTION 'Could not find item using token: % item: % collection: %', _token, _itemid, _collectionid;\n END IF;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\n\nCREATE OR REPLACE FUNCTION get_token_filter(\n _sortby jsonb DEFAULT '[{\"field\":\"datetime\",\"direction\":\"desc\"}]'::jsonb,\n token_item items DEFAULT NULL,\n prev boolean DEFAULT FALSE,\n inclusive boolean DEFAULT FALSE\n) RETURNS text AS $$\nDECLARE\n ltop text := '<';\n gtop text := '>';\n dir text;\n sort record;\n orfilter text := '';\n orfilters text[] := '{}'::text[];\n andfilters text[] := '{}'::text[];\n output text;\n token_where text;\nBEGIN\n IF _sortby IS NULL OR _sortby = '[]'::jsonb THEN\n _sortby := '[{\"field\":\"datetime\",\"direction\":\"desc\"}]'::jsonb;\n END IF;\n _sortby := _sortby || jsonb_build_object('field','id','direction',_sortby->0->>'direction');\n RAISE NOTICE 'Getting Token Filter. % %', _sortby, token_item;\n IF inclusive THEN\n orfilters := orfilters || format('( id=%L AND collection=%L )' , token_item.id, token_item.collection);\n END IF;\n\n FOR sort IN\n WITH s1 AS (\n SELECT\n _row,\n (queryable(value->>'field')).expression as _field,\n (value->>'field' = 'id') as _isid,\n get_sort_dir(value) as _dir\n FROM jsonb_array_elements(_sortby)\n WITH ORDINALITY AS t(value, _row)\n )\n SELECT\n _row,\n _field,\n _dir,\n get_token_val_str(_field, token_item) as _val\n FROM s1\n WHERE _row <= (SELECT min(_row) FROM s1 WHERE _isid)\n LOOP\n orfilter := NULL;\n RAISE NOTICE 'SORT: %', sort;\n IF sort._val IS NOT NULL AND ((prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC')) THEN\n orfilter := format($f$(\n (%s %s %s) OR (%s IS NULL)\n )$f$,\n sort._field,\n ltop,\n sort._val,\n sort._val\n );\n ELSIF sort._val IS NULL AND ((prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC')) THEN\n RAISE NOTICE '< but null';\n orfilter := format('%s IS NOT NULL', sort._field);\n ELSIF sort._val IS NULL THEN\n RAISE NOTICE '> but null';\n ELSE\n orfilter := format($f$(\n (%s %s %s) OR (%s IS NULL)\n )$f$,\n sort._field,\n gtop,\n sort._val,\n sort._field\n );\n END IF;\n RAISE NOTICE 'ORFILTER: %', orfilter;\n\n IF orfilter IS NOT NULL THEN\n IF sort._row = 1 THEN\n orfilters := orfilters || orfilter;\n ELSE\n orfilters := orfilters || format('(%s AND %s)', array_to_string(andfilters, ' AND '), orfilter);\n END IF;\n END IF;\n IF sort._val IS NOT NULL THEN\n andfilters := andfilters || format('%s = %s', sort._field, sort._val);\n ELSE\n andfilters := andfilters || format('%s IS NULL', sort._field);\n END IF;\n END LOOP;\n\n output := array_to_string(orfilters, ' OR ');\n\n token_where := concat('(',coalesce(output,'true'),')');\n IF trim(token_where) = '' THEN\n token_where := NULL;\n END IF;\n RAISE NOTICE 'TOKEN_WHERE: %',token_where;\n RETURN token_where;\n END;\n$$ LANGUAGE PLPGSQL SET transform_null_equals TO TRUE\n;\n\nCREATE OR REPLACE FUNCTION search_tohash(jsonb) RETURNS jsonb AS $$\n SELECT $1 - '{token,limit,context,includes,excludes}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION search_hash(jsonb, jsonb) RETURNS text AS $$\n SELECT md5(concat(search_tohash($1)::text,$2::text));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TABLE IF NOT EXISTS searches(\n hash text GENERATED ALWAYS AS (search_hash(search, metadata)) STORED PRIMARY KEY,\n search jsonb NOT NULL,\n _where text,\n orderby text,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n metadata jsonb DEFAULT '{}'::jsonb NOT NULL\n);\n\nCREATE TABLE IF NOT EXISTS search_wheres(\n id bigint generated always as identity primary key,\n _where text NOT NULL,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n statslastupdated timestamptz,\n estimated_count bigint,\n estimated_cost float,\n time_to_estimate float,\n total_count bigint,\n time_to_count float,\n partitions text[]\n);\n\nCREATE INDEX IF NOT EXISTS search_wheres_partitions ON search_wheres USING GIN (partitions);\nCREATE UNIQUE INDEX IF NOT EXISTS search_wheres_where ON search_wheres ((md5(_where)));\n\nCREATE OR REPLACE FUNCTION where_stats(\n inwhere text,\n updatestats boolean default false,\n conf jsonb default null\n) RETURNS search_wheres AS $$\nDECLARE\n t timestamptz;\n i interval;\n explain_json jsonb;\n partitions text[];\n sw search_wheres%ROWTYPE;\n inwhere_hash text := md5(inwhere);\n _context text := lower(context(conf));\n _stats_ttl interval := context_stats_ttl(conf);\n _estimated_cost_threshold float := context_estimated_cost(conf);\n _estimated_count_threshold int := context_estimated_count(conf);\n ro bool := pgstac.readonly(conf);\nBEGIN\n -- If updatestats is true then set ttl to 0\n IF updatestats THEN\n RAISE DEBUG 'Updatestats set to TRUE, setting TTL to 0';\n _stats_ttl := '0'::interval;\n END IF;\n\n -- If we don't need to calculate context, just return\n IF _context = 'off' THEN\n sw._where = inwhere;\n RETURN sw;\n END IF;\n\n -- Get any stats that we have.\n IF NOT ro THEN\n -- If there is a lock where another process is\n -- updating the stats, wait so that we don't end up calculating a bunch of times.\n SELECT * INTO sw FROM search_wheres WHERE md5(_where)=inwhere_hash FOR UPDATE;\n ELSE\n SELECT * INTO sw FROM search_wheres WHERE md5(_where)=inwhere_hash;\n END IF;\n\n -- If there is a cached row, figure out if we need to update\n IF\n sw IS NOT NULL\n AND sw.statslastupdated IS NOT NULL\n AND sw.total_count IS NOT NULL\n AND now() - sw.statslastupdated <= _stats_ttl\n THEN\n -- we have a cached row with data that is within our ttl\n RAISE DEBUG 'Stats present in table and lastupdated within ttl: %', sw;\n IF NOT ro THEN\n RAISE DEBUG 'Updating search_wheres only bumping lastused and usecount';\n UPDATE search_wheres SET\n lastused = now(),\n usecount = search_wheres.usecount + 1\n WHERE md5(_where) = inwhere_hash\n RETURNING * INTO sw;\n END IF;\n RAISE DEBUG 'Returning cached counts. %', sw;\n RETURN sw;\n END IF;\n\n -- Calculate estimated cost and rows\n -- Use explain to get estimated count/cost\n IF sw.estimated_count IS NULL OR sw.estimated_cost IS NULL THEN\n RAISE DEBUG 'Calculating estimated stats';\n t := clock_timestamp();\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s', inwhere)\n INTO explain_json;\n RAISE DEBUG 'Time for just the explain: %', clock_timestamp() - t;\n i := clock_timestamp() - t;\n\n sw.estimated_count := explain_json->0->'Plan'->'Plan Rows';\n sw.estimated_cost := explain_json->0->'Plan'->'Total Cost';\n sw.time_to_estimate := extract(epoch from i);\n END IF;\n\n RAISE DEBUG 'ESTIMATED_COUNT: %, THRESHOLD %', sw.estimated_count, _estimated_count_threshold;\n RAISE DEBUG 'ESTIMATED_COST: %, THRESHOLD %', sw.estimated_cost, _estimated_cost_threshold;\n\n -- If context is set to auto and the costs are within the threshold return the estimated costs\n IF\n _context = 'auto'\n AND sw.estimated_count >= _estimated_count_threshold\n AND sw.estimated_cost >= _estimated_cost_threshold\n THEN\n IF NOT ro THEN\n INSERT INTO search_wheres (\n _where,\n lastused,\n usecount,\n statslastupdated,\n estimated_count,\n estimated_cost,\n time_to_estimate,\n total_count,\n time_to_count\n ) VALUES (\n inwhere,\n now(),\n 1,\n now(),\n sw.estimated_count,\n sw.estimated_cost,\n sw.time_to_estimate,\n null,\n null\n ) ON CONFLICT ((md5(_where)))\n DO UPDATE SET\n lastused = EXCLUDED.lastused,\n usecount = search_wheres.usecount + 1,\n statslastupdated = EXCLUDED.statslastupdated,\n estimated_count = EXCLUDED.estimated_count,\n estimated_cost = EXCLUDED.estimated_cost,\n time_to_estimate = EXCLUDED.time_to_estimate,\n total_count = EXCLUDED.total_count,\n time_to_count = EXCLUDED.time_to_count\n RETURNING * INTO sw;\n END IF;\n RAISE DEBUG 'Estimates are within thresholds, returning estimates. %', sw;\n RETURN sw;\n END IF;\n\n -- Calculate Actual Count\n t := clock_timestamp();\n RAISE NOTICE 'Calculating actual count...';\n EXECUTE format(\n 'SELECT count(*) FROM items WHERE %s',\n inwhere\n ) INTO sw.total_count;\n i := clock_timestamp() - t;\n RAISE NOTICE 'Actual Count: % -- %', sw.total_count, i;\n sw.time_to_count := extract(epoch FROM i);\n\n IF NOT ro THEN\n INSERT INTO search_wheres (\n _where,\n lastused,\n usecount,\n statslastupdated,\n estimated_count,\n estimated_cost,\n time_to_estimate,\n total_count,\n time_to_count\n ) VALUES (\n inwhere,\n now(),\n 1,\n now(),\n sw.estimated_count,\n sw.estimated_cost,\n sw.time_to_estimate,\n sw.total_count,\n sw.time_to_count\n ) ON CONFLICT ((md5(_where)))\n DO UPDATE SET\n lastused = EXCLUDED.lastused,\n usecount = search_wheres.usecount + 1,\n statslastupdated = EXCLUDED.statslastupdated,\n estimated_count = EXCLUDED.estimated_count,\n estimated_cost = EXCLUDED.estimated_cost,\n time_to_estimate = EXCLUDED.time_to_estimate,\n total_count = EXCLUDED.total_count,\n time_to_count = EXCLUDED.time_to_count\n RETURNING * INTO sw;\n END IF;\n RAISE DEBUG 'Returning with actual count. %', sw;\n RETURN sw;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION search_query(\n _search jsonb = '{}'::jsonb,\n updatestats boolean = false,\n _metadata jsonb = '{}'::jsonb\n) RETURNS searches AS $$\nDECLARE\n search searches%ROWTYPE;\n cached_search searches%ROWTYPE;\n pexplain jsonb;\n t timestamptz;\n i interval;\n doupdate boolean := FALSE;\n insertfound boolean := FALSE;\n ro boolean := pgstac.readonly();\n found_search text;\nBEGIN\n RAISE NOTICE 'SEARCH: %', _search;\n -- Calculate hash, where clause, and order by statement\n search.search := _search;\n search.metadata := _metadata;\n search.hash := search_hash(_search, _metadata);\n search._where := stac_search_to_where(_search);\n search.orderby := sort_sqlorderby(_search);\n search.lastused := now();\n search.usecount := 1;\n\n -- If we are in read only mode, directly return search\n IF ro THEN\n RETURN search;\n END IF;\n\n RAISE NOTICE 'Updating Statistics for search: %s', search;\n -- Update statistics for times used and and when last used\n -- If the entry is locked, rather than waiting, skip updating the stats\n INSERT INTO searches (search, lastused, usecount, metadata)\n VALUES (search.search, now(), 1, search.metadata)\n ON CONFLICT DO NOTHING\n RETURNING * INTO cached_search\n ;\n\n IF NOT FOUND OR cached_search IS NULL THEN\n UPDATE searches SET\n lastused = now(),\n usecount = searches.usecount + 1\n WHERE hash = (\n SELECT hash FROM searches WHERE hash=search.hash FOR UPDATE SKIP LOCKED\n )\n RETURNING * INTO cached_search\n ;\n END IF;\n\n IF cached_search IS NOT NULL THEN\n cached_search._where = search._where;\n cached_search.orderby = search.orderby;\n RETURN cached_search;\n END IF;\n RETURN search;\n\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE OR REPLACE FUNCTION search_fromhash(\n _hash text\n) RETURNS searches AS $$\n SELECT * FROM search_query((SELECT search FROM searches WHERE hash=_hash LIMIT 1));\n$$ LANGUAGE SQL STRICT;\n\nCREATE OR REPLACE FUNCTION search_rows(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN partitions text[] DEFAULT NULL,\n IN _limit int DEFAULT 10\n) RETURNS SETOF items AS $$\nDECLARE\n base_query text;\n query text;\n sdate timestamptz;\n edate timestamptz;\n n int;\n records_left int := _limit;\n timer timestamptz := clock_timestamp();\n full_timer timestamptz := clock_timestamp();\nBEGIN\nIF _where IS NULL OR trim(_where) = '' THEN\n _where = ' TRUE ';\nEND IF;\nRAISE NOTICE 'Getting chunks for % %', _where, _orderby;\n\nbase_query := $q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n LIMIT %L\n$q$;\n\nIF _orderby ILIKE 'datetime d%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 DESC LOOP\n RAISE NOTICE 'Running Query for % to %. %', sdate, edate, age_ms(full_timer);\n query := format(\n base_query,\n sdate,\n edate,\n _where,\n _orderby,\n records_left\n );\n RAISE DEBUG 'QUERY: %', query;\n timer := clock_timestamp();\n RETURN QUERY EXECUTE query;\n\n GET DIAGNOSTICS n = ROW_COUNT;\n records_left := records_left - n;\n RAISE NOTICE 'Returned %/% Rows From % to %. % to go. Time: %ms', n, _limit, sdate, edate, records_left, age_ms(timer);\n timer := clock_timestamp();\n IF records_left <= 0 THEN\n RAISE NOTICE 'SEARCH_ROWS TOOK %ms', age_ms(full_timer);\n RETURN;\n END IF;\n END LOOP;\nELSIF _orderby ILIKE 'datetime a%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 ASC LOOP\n RAISE NOTICE 'Running Query for % to %. %', sdate, edate, age_ms(full_timer);\n query := format(\n base_query,\n sdate,\n edate,\n _where,\n _orderby,\n records_left\n );\n RAISE DEBUG 'QUERY: %', query;\n timer := clock_timestamp();\n RETURN QUERY EXECUTE query;\n\n GET DIAGNOSTICS n = ROW_COUNT;\n records_left := records_left - n;\n RAISE NOTICE 'Returned %/% Rows From % to %. % to go. Time: %ms', n, _limit, sdate, edate, records_left, age_ms(timer);\n timer := clock_timestamp();\n IF records_left <= 0 THEN\n RAISE NOTICE 'SEARCH_ROWS TOOK %ms', age_ms(full_timer);\n RETURN;\n END IF;\n END LOOP;\nELSE\n query := format($q$\n SELECT * FROM items\n WHERE %s\n ORDER BY %s\n LIMIT %L\n $q$, _where, _orderby, _limit\n );\n RAISE DEBUG 'QUERY: %', query;\n timer := clock_timestamp();\n RETURN QUERY EXECUTE query;\n RAISE NOTICE 'FULL QUERY TOOK %ms', age_ms(timer);\nEND IF;\nRAISE NOTICE 'SEARCH_ROWS TOOK %ms', age_ms(full_timer);\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE UNLOGGED TABLE format_item_cache(\n id text,\n collection text,\n fields text,\n hydrated bool,\n output jsonb,\n lastused timestamptz DEFAULT now(),\n usecount int DEFAULT 1,\n timetoformat float,\n PRIMARY KEY (collection, id, fields, hydrated)\n);\nCREATE INDEX ON format_item_cache (lastused);\n\nCREATE OR REPLACE FUNCTION format_item(_item items, _fields jsonb DEFAULT '{}', _hydrated bool DEFAULT TRUE) RETURNS jsonb AS $$\nDECLARE\n cache bool := get_setting_bool('format_cache');\n _output jsonb := null;\n t timestamptz := clock_timestamp();\nBEGIN\n IF cache THEN\n SELECT output INTO _output FROM format_item_cache\n WHERE id=_item.id AND collection=_item.collection AND fields=_fields::text AND hydrated=_hydrated;\n END IF;\n IF _output IS NULL THEN\n IF _hydrated THEN\n _output := content_hydrate(_item, _fields);\n ELSE\n _output := content_nonhydrated(_item, _fields);\n END IF;\n END IF;\n IF cache THEN\n INSERT INTO format_item_cache (id, collection, fields, hydrated, output, timetoformat)\n VALUES (_item.id, _item.collection, _fields::text, _hydrated, _output, age_ms(t))\n ON CONFLICT(collection, id, fields, hydrated) DO\n UPDATE\n SET lastused=now(), usecount = format_item_cache.usecount + 1\n ;\n END IF;\n RETURN _output;\n\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION search(_search jsonb = '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n searches searches%ROWTYPE;\n _where text;\n orderby text;\n search_where search_wheres%ROWTYPE;\n total_count bigint;\n token record;\n token_prev boolean;\n token_item items%ROWTYPE;\n token_where text;\n full_where text;\n init_ts timestamptz := clock_timestamp();\n timer timestamptz := clock_timestamp();\n hydrate bool := NOT (_search->'conf'->>'nohydrate' IS NOT NULL AND (_search->'conf'->>'nohydrate')::boolean = true);\n prev text;\n next text;\n context jsonb;\n collection jsonb;\n out_records jsonb;\n out_len int;\n _limit int := coalesce((_search->>'limit')::int, 10);\n _querylimit int;\n _fields jsonb := coalesce(_search->'fields', '{}'::jsonb);\n has_prev boolean := FALSE;\n has_next boolean := FALSE;\n links jsonb := '[]'::jsonb;\n base_url text:= concat(rtrim(base_url(_search->'conf'),'/'));\nBEGIN\n searches := search_query(_search);\n _where := searches._where;\n orderby := searches.orderby;\n search_where := where_stats(_where);\n total_count := coalesce(search_where.total_count, search_where.estimated_count);\n RAISE NOTICE 'SEARCH:TOKEN: %', _search->>'token';\n token := get_token_record(_search->>'token');\n RAISE NOTICE '***TOKEN: %', token;\n _querylimit := _limit + 1;\n IF token IS NOT NULL THEN\n token_prev := token.prev;\n token_item := token.item;\n token_where := get_token_filter(_search->'sortby', token_item, token_prev, FALSE);\n RAISE DEBUG 'TOKEN_WHERE: % (%ms from search start)', token_where, age_ms(timer);\n IF token_prev THEN -- if we are using a prev token, we know has_next is true\n RAISE DEBUG 'There is a previous token, so automatically setting has_next to true';\n has_next := TRUE;\n orderby := sort_sqlorderby(_search, TRUE);\n ELSE\n RAISE DEBUG 'There is a next token, so automatically setting has_prev to true';\n has_prev := TRUE;\n\n END IF;\n ELSE -- if there was no token, we know there is no prev\n RAISE DEBUG 'There is no token, so we know there is no prev. setting has_prev to false';\n has_prev := FALSE;\n END IF;\n\n full_where := concat_ws(' AND ', _where, token_where);\n RAISE NOTICE 'FULL WHERE CLAUSE: %', full_where;\n RAISE NOTICE 'Time to get counts and build query %', age_ms(timer);\n timer := clock_timestamp();\n\n IF hydrate THEN\n RAISE NOTICE 'Getting hydrated data.';\n ELSE\n RAISE NOTICE 'Getting non-hydrated data.';\n END IF;\n RAISE NOTICE 'CACHE SET TO %', get_setting_bool('format_cache');\n RAISE NOTICE 'Time to set hydration/formatting %', age_ms(timer);\n timer := clock_timestamp();\n SELECT jsonb_agg(format_item(i, _fields, hydrate)) INTO out_records\n FROM search_rows(\n full_where,\n orderby,\n search_where.partitions,\n _querylimit\n ) as i;\n\n RAISE NOTICE 'Time to fetch rows %', age_ms(timer);\n timer := clock_timestamp();\n\n\n IF token_prev THEN\n out_records := flip_jsonb_array(out_records);\n END IF;\n\n RAISE NOTICE 'Query returned % records.', jsonb_array_length(out_records);\n RAISE DEBUG 'TOKEN: % %', token_item.id, token_item.collection;\n RAISE DEBUG 'RECORD_1: % %', out_records->0->>'id', out_records->0->>'collection';\n RAISE DEBUG 'RECORD-1: % %', out_records->-1->>'id', out_records->-1->>'collection';\n\n -- REMOVE records that were from our token\n IF out_records->0->>'id' = token_item.id AND out_records->0->>'collection' = token_item.collection THEN\n out_records := out_records - 0;\n ELSIF out_records->-1->>'id' = token_item.id AND out_records->-1->>'collection' = token_item.collection THEN\n out_records := out_records - -1;\n END IF;\n\n out_len := jsonb_array_length(out_records);\n\n IF out_len = _limit + 1 THEN\n IF token_prev THEN\n has_prev := TRUE;\n out_records := out_records - 0;\n ELSE\n has_next := TRUE;\n out_records := out_records - -1;\n END IF;\n END IF;\n\n\n links := links || jsonb_build_object(\n 'rel', 'root',\n 'type', 'application/json',\n 'href', base_url\n ) || jsonb_build_object(\n 'rel', 'self',\n 'type', 'application/json',\n 'href', concat(base_url, '/search')\n );\n\n IF has_next THEN\n next := concat(out_records->-1->>'collection', ':', out_records->-1->>'id');\n RAISE NOTICE 'HAS NEXT | %', next;\n links := links || jsonb_build_object(\n 'rel', 'next',\n 'type', 'application/geo+json',\n 'method', 'GET',\n 'href', concat(base_url, '/search?token=next:', next)\n );\n END IF;\n\n IF has_prev THEN\n prev := concat(out_records->0->>'collection', ':', out_records->0->>'id');\n RAISE NOTICE 'HAS PREV | %', prev;\n links := links || jsonb_build_object(\n 'rel', 'prev',\n 'type', 'application/geo+json',\n 'method', 'GET',\n 'href', concat(base_url, '/search?token=prev:', prev)\n );\n END IF;\n\n RAISE NOTICE 'Time to get prev/next %', age_ms(timer);\n timer := clock_timestamp();\n\n\n collection := jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb),\n 'links', links\n );\n\n\n\n IF context(_search->'conf') != 'off' THEN\n collection := collection || jsonb_strip_nulls(jsonb_build_object(\n 'numberMatched', total_count,\n 'numberReturned', coalesce(jsonb_array_length(out_records), 0)\n ));\n ELSE\n collection := collection || jsonb_strip_nulls(jsonb_build_object(\n 'numberReturned', coalesce(jsonb_array_length(out_records), 0)\n ));\n END IF;\n\n IF get_setting_bool('timing', _search->'conf') THEN\n collection = collection || jsonb_build_object('timing', age_ms(init_ts));\n END IF;\n\n RAISE NOTICE 'Time to build final json %', age_ms(timer);\n timer := clock_timestamp();\n\n RAISE NOTICE 'Total Time: %', age_ms(current_timestamp);\n RAISE NOTICE 'RETURNING % records. NEXT: %. PREV: %', collection->>'numberReturned', collection->>'next', collection->>'prev';\n RETURN collection;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION search_cursor(_search jsonb = '{}'::jsonb) RETURNS refcursor AS $$\nDECLARE\n curs refcursor;\n searches searches%ROWTYPE;\n _where text;\n _orderby text;\n q text;\n\nBEGIN\n searches := search_query(_search);\n _where := searches._where;\n _orderby := searches.orderby;\n\n OPEN curs FOR\n WITH p AS (\n SELECT * FROM partition_queries(_where, _orderby) p\n )\n SELECT\n CASE WHEN EXISTS (SELECT 1 FROM p) THEN\n (SELECT format($q$\n SELECT * FROM (\n %s\n ) total\n $q$,\n string_agg(\n format($q$ SELECT * FROM ( %s ) AS sub $q$, p),\n '\n UNION ALL\n '\n )\n ))\n ELSE NULL\n END FROM p;\n RETURN curs;\nEND;\n$$ LANGUAGE PLPGSQL;\nCREATE OR REPLACE VIEW collections_asitems AS\nSELECT\n id,\n geometry,\n 'collections' AS collection,\n datetime,\n end_datetime,\n jsonb_build_object(\n 'properties', content - '{links,assets,stac_version,stac_extensions}',\n 'links', content->'links',\n 'assets', content->'assets',\n 'stac_version', content->'stac_version',\n 'stac_extensions', content->'stac_extensions'\n ) AS content,\n content as collectionjson\nFROM collections;\n\n\nCREATE OR REPLACE FUNCTION collection_search_matched(\n IN _search jsonb DEFAULT '{}'::jsonb,\n OUT matched bigint\n) RETURNS bigint AS $$\nDECLARE\n _where text := stac_search_to_where(_search);\nBEGIN\n EXECUTE format(\n $query$\n SELECT\n count(*)\n FROM\n collections_asitems\n WHERE %s\n ;\n $query$,\n _where\n ) INTO matched;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION collection_search_rows(\n _search jsonb DEFAULT '{}'::jsonb\n) RETURNS SETOF jsonb AS $$\nDECLARE\n _where text := stac_search_to_where(_search);\n _limit int := coalesce((_search->>'limit')::int, 10);\n _fields jsonb := coalesce(_search->'fields', '{}'::jsonb);\n _orderby text;\n _offset int := COALESCE((_search->>'offset')::int, 0);\nBEGIN\n _orderby := sort_sqlorderby(\n jsonb_build_object(\n 'sortby',\n coalesce(\n _search->'sortby',\n '[{\"field\": \"id\", \"direction\": \"asc\"}]'::jsonb\n )\n )\n );\n RETURN QUERY EXECUTE format(\n $query$\n SELECT\n jsonb_fields(collectionjson, %L) as c\n FROM\n collections_asitems\n WHERE %s\n ORDER BY %s\n LIMIT %L\n OFFSET %L\n ;\n $query$,\n _fields,\n _where,\n _orderby,\n _limit,\n _offset\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION collection_search(\n _search jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\nDECLARE\n out_records jsonb;\n number_matched bigint := collection_search_matched(_search);\n number_returned bigint;\n _limit int := coalesce((_search->>'limit')::float::int, 10);\n _offset int := coalesce((_search->>'offset')::float::int, 0);\n links jsonb := '[]';\n ret jsonb;\n base_url text:= concat(rtrim(base_url(_search->'conf'),'/'), '/collections');\n prevoffset int;\n nextoffset int;\nBEGIN\n SELECT\n coalesce(jsonb_agg(c), '[]')\n INTO out_records\n FROM collection_search_rows(_search) c;\n\n number_returned := jsonb_array_length(out_records);\n RAISE DEBUG 'nm: %, nr: %, l:%, o:%', number_matched, number_returned, _limit, _offset;\n\n\n\n IF _limit <= number_matched AND number_matched > 0 THEN --need to have paging links\n nextoffset := least(_offset + _limit, number_matched - 1);\n prevoffset := greatest(_offset - _limit, 0);\n\n IF _offset > 0 THEN\n links := links || jsonb_build_object(\n 'rel', 'prev',\n 'type', 'application/json',\n 'method', 'GET' ,\n 'href', base_url,\n 'body', jsonb_build_object('offset', prevoffset),\n 'merge', TRUE\n );\n END IF;\n\n IF (_offset + _limit < number_matched) THEN\n links := links || jsonb_build_object(\n 'rel', 'next',\n 'type', 'application/json',\n 'method', 'GET' ,\n 'href', base_url,\n 'body', jsonb_build_object('offset', nextoffset),\n 'merge', TRUE\n );\n END IF;\n\n END IF;\n\n ret := jsonb_build_object(\n 'collections', out_records,\n 'numberMatched', number_matched,\n 'numberReturned', number_returned,\n 'links', links\n );\n RETURN ret;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION tileenvelope(zoom int, x int, y int) RETURNS geometry AS $$\nWITH t AS (\n SELECT\n 20037508.3427892 as merc_max,\n -20037508.3427892 as merc_min,\n (2 * 20037508.3427892) / (2 ^ zoom) as tile_size\n)\nSELECT st_makeenvelope(\n merc_min + (tile_size * x),\n merc_max - (tile_size * (y + 1)),\n merc_min + (tile_size * (x + 1)),\n merc_max - (tile_size * y),\n 3857\n) FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;DROP FUNCTION IF EXISTS mercgrid;\n\n\nCREATE OR REPLACE FUNCTION ftime() RETURNS interval as $$\nSELECT age(clock_timestamp(), transaction_timestamp());\n$$ LANGUAGE SQL;\nSET SEARCH_PATH to pgstac, public;\n\nDROP FUNCTION IF EXISTS geometrysearch;\nCREATE OR REPLACE FUNCTION geometrysearch(\n IN geom geometry,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE, -- Return as soon as the passed in geometry is full covered\n IN skipcovered boolean DEFAULT TRUE -- Skip any items that would show up completely under the previous items\n) RETURNS jsonb AS $$\nDECLARE\n search searches%ROWTYPE;\n curs refcursor;\n _where text;\n query text;\n iter_record items%ROWTYPE;\n out_records jsonb := '{}'::jsonb[];\n exit_flag boolean := FALSE;\n counter int := 1;\n scancounter int := 1;\n remaining_limit int := _scanlimit;\n tilearea float;\n unionedgeom geometry;\n clippedgeom geometry;\n unionedgeom_area float := 0;\n prev_area float := 0;\n excludes text[];\n includes text[];\n\nBEGIN\n DROP TABLE IF EXISTS pgstac_results;\n CREATE TEMP TABLE pgstac_results (content jsonb) ON COMMIT DROP;\n\n -- If the passed in geometry is not an area set exitwhenfull and skipcovered to false\n IF ST_GeometryType(geom) !~* 'polygon' THEN\n RAISE NOTICE 'GEOMETRY IS NOT AN AREA';\n skipcovered = FALSE;\n exitwhenfull = FALSE;\n END IF;\n\n -- If skipcovered is true then you will always want to exit when the passed in geometry is full\n IF skipcovered THEN\n exitwhenfull := TRUE;\n END IF;\n\n search := search_fromhash(queryhash);\n\n IF search IS NULL THEN\n RAISE EXCEPTION 'Search with Query Hash % Not Found', queryhash;\n END IF;\n\n tilearea := st_area(geom);\n _where := format('%s AND st_intersects(geometry, %L::geometry)', search._where, geom);\n\n\n FOR query IN SELECT * FROM partition_queries(_where, search.orderby) LOOP\n query := format('%s LIMIT %L', query, remaining_limit);\n RAISE NOTICE '%', query;\n OPEN curs FOR EXECUTE query;\n LOOP\n FETCH curs INTO iter_record;\n EXIT WHEN NOT FOUND;\n IF exitwhenfull OR skipcovered THEN -- If we are not using exitwhenfull or skipcovered, we do not need to do expensive geometry operations\n clippedgeom := st_intersection(geom, iter_record.geometry);\n\n IF unionedgeom IS NULL THEN\n unionedgeom := clippedgeom;\n ELSE\n unionedgeom := st_union(unionedgeom, clippedgeom);\n END IF;\n\n unionedgeom_area := st_area(unionedgeom);\n\n IF skipcovered AND prev_area = unionedgeom_area THEN\n scancounter := scancounter + 1;\n CONTINUE;\n END IF;\n\n prev_area := unionedgeom_area;\n\n RAISE NOTICE '% % % %', unionedgeom_area/tilearea, counter, scancounter, ftime();\n END IF;\n RAISE NOTICE '% %', iter_record, content_hydrate(iter_record, fields);\n INSERT INTO pgstac_results (content) VALUES (content_hydrate(iter_record, fields));\n\n IF counter >= _limit\n OR scancounter > _scanlimit\n OR ftime() > _timelimit\n OR (exitwhenfull AND unionedgeom_area >= tilearea)\n THEN\n exit_flag := TRUE;\n EXIT;\n END IF;\n counter := counter + 1;\n scancounter := scancounter + 1;\n\n END LOOP;\n CLOSE curs;\n EXIT WHEN exit_flag;\n remaining_limit := _scanlimit - scancounter;\n END LOOP;\n\n SELECT jsonb_agg(content) INTO out_records FROM pgstac_results WHERE content IS NOT NULL;\n\n RETURN jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb)\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDROP FUNCTION IF EXISTS geojsonsearch;\nCREATE OR REPLACE FUNCTION geojsonsearch(\n IN geojson jsonb,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_geomfromgeojson(geojson),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS xyzsearch;\nCREATE OR REPLACE FUNCTION xyzsearch(\n IN _x int,\n IN _y int,\n IN _z int,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_transform(tileenvelope(_z, _x, _y), 4326),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE PROCEDURE analyze_items() AS $$\nDECLARE\n q text;\n timeout_ts timestamptz;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n SELECT format('ANALYZE (VERBOSE, SKIP_LOCKED) %I;', relname) INTO q\n FROM pg_stat_user_tables\n WHERE relname like '_item%' AND (n_mod_since_analyze>0 OR last_analyze IS NULL) LIMIT 1;\n IF NOT FOUND THEN\n EXIT;\n END IF;\n RAISE NOTICE '%', q;\n EXECUTE q;\n COMMIT;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE PROCEDURE validate_constraints() AS $$\nDECLARE\n q text;\nBEGIN\n FOR q IN\n SELECT\n FORMAT(\n 'ALTER TABLE %I.%I VALIDATE CONSTRAINT %I;',\n nsp.nspname,\n cls.relname,\n con.conname\n )\n\n FROM pg_constraint AS con\n JOIN pg_class AS cls\n ON con.conrelid = cls.oid\n JOIN pg_namespace AS nsp\n ON cls.relnamespace = nsp.oid\n WHERE convalidated = FALSE AND contype in ('c','f')\n AND nsp.nspname = 'pgstac'\n LOOP\n RAISE NOTICE '%', q;\n PERFORM run_or_queue(q);\n COMMIT;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION collection_extent(_collection text, runupdate boolean default false) RETURNS jsonb AS $$\nDECLARE\n geom_extent geometry;\n mind timestamptz;\n maxd timestamptz;\n extent jsonb;\nBEGIN\n IF runupdate THEN\n PERFORM update_partition_stats_q(partition)\n FROM partitions_view WHERE collection=_collection;\n END IF;\n SELECT\n min(lower(dtrange)),\n max(upper(edtrange)),\n st_extent(spatial)\n INTO\n mind,\n maxd,\n geom_extent\n FROM partitions_view\n WHERE collection=_collection;\n\n IF geom_extent IS NOT NULL AND mind IS NOT NULL AND maxd IS NOT NULL THEN\n extent := jsonb_build_object(\n 'spatial', jsonb_build_object(\n 'bbox', to_jsonb(array[array[st_xmin(geom_extent), st_ymin(geom_extent), st_xmax(geom_extent), st_ymax(geom_extent)]])\n ),\n 'temporal', jsonb_build_object(\n 'interval', to_jsonb(array[array[mind, maxd]])\n )\n );\n RETURN extent;\n END IF;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('id', '{\"title\": \"Item ID\",\"description\": \"Item identifier\",\"$ref\": \"https://schemas.stacspec.org/v1.0.0/item-spec/json-schema/item.json#/definitions/core/allOf/2/properties/id\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('geometry', '{\"title\": \"Item Geometry\",\"description\": \"Item Geometry\",\"$ref\": \"https://geojson.org/schema/Feature.json\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('datetime','{\"description\": \"Datetime\",\"type\": \"string\",\"title\": \"Acquired\",\"format\": \"date-time\",\"pattern\": \"(\\\\+00:00|Z)$\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDELETE FROM queryables a USING queryables b\n WHERE a.name = b.name AND a.collection_ids IS NOT DISTINCT FROM b.collection_ids AND a.id > b.id;\n\n\nINSERT INTO pgstac_settings (name, value) VALUES\n ('context', 'off'),\n ('context_estimated_count', '100000'),\n ('context_estimated_cost', '100000'),\n ('context_stats_ttl', '1 day'),\n ('default_filter_lang', 'cql2-json'),\n ('additional_properties', 'true'),\n ('use_queue', 'false'),\n ('queue_timeout', '10 minutes'),\n ('update_collection_extent', 'false'),\n ('format_cache', 'false'),\n ('readonly', 'false')\nON CONFLICT DO NOTHING\n;\n\n\nINSERT INTO cql2_ops (op, template, types) VALUES\n ('eq', '%s = %s', NULL),\n ('neq', '%s != %s', NULL),\n ('ne', '%s != %s', NULL),\n ('!=', '%s != %s', NULL),\n ('<>', '%s != %s', NULL),\n ('lt', '%s < %s', NULL),\n ('lte', '%s <= %s', NULL),\n ('gt', '%s > %s', NULL),\n ('gte', '%s >= %s', NULL),\n ('le', '%s <= %s', NULL),\n ('ge', '%s >= %s', NULL),\n ('=', '%s = %s', NULL),\n ('<', '%s < %s', NULL),\n ('<=', '%s <= %s', NULL),\n ('>', '%s > %s', NULL),\n ('>=', '%s >= %s', NULL),\n ('like', '%s LIKE %s', NULL),\n ('ilike', '%s ILIKE %s', NULL),\n ('+', '%s + %s', NULL),\n ('-', '%s - %s', NULL),\n ('*', '%s * %s', NULL),\n ('/', '%s / %s', NULL),\n ('not', 'NOT (%s)', NULL),\n ('between', '%s BETWEEN %s AND %s', NULL),\n ('isnull', '%s IS NULL', NULL),\n ('upper', 'upper(%s)', NULL),\n ('lower', 'lower(%s)', NULL),\n ('casei', 'upper(%s)', NULL),\n ('accenti', 'unaccent(%s)', NULL)\nON CONFLICT (op) DO UPDATE\n SET\n template = EXCLUDED.template\n;\n\n\nALTER FUNCTION to_text COST 5000;\nALTER FUNCTION to_float COST 5000;\nALTER FUNCTION to_int COST 5000;\nALTER FUNCTION to_tstz COST 5000;\nALTER FUNCTION to_text_array COST 5000;\n\nALTER FUNCTION update_partition_stats SECURITY DEFINER;\nALTER FUNCTION partition_after_triggerfunc SECURITY DEFINER;\nALTER FUNCTION drop_table_constraints SECURITY DEFINER;\nALTER FUNCTION create_table_constraints SECURITY DEFINER;\nALTER FUNCTION check_partition SECURITY DEFINER;\nALTER FUNCTION repartition SECURITY DEFINER;\nALTER FUNCTION where_stats SECURITY DEFINER;\nALTER FUNCTION search_query SECURITY DEFINER;\nALTER FUNCTION format_item SECURITY DEFINER;\nALTER FUNCTION maintain_index SECURITY DEFINER;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nGRANT ALL ON SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac to pgstac_admin;\n\n-- pgstac_read role limited to using function apis\nGRANT EXECUTE ON FUNCTION search TO pgstac_read;\nGRANT EXECUTE ON FUNCTION search_query TO pgstac_read;\nGRANT EXECUTE ON FUNCTION item_by_id TO pgstac_read;\nGRANT EXECUTE ON FUNCTION get_item TO pgstac_read;\nGRANT SELECT ON ALL TABLES IN SCHEMA pgstac TO pgstac_read;\n\n\nGRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_ingest;\nGRANT USAGE ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_ingest;\n\nREVOKE ALL PRIVILEGES ON PROCEDURE run_queued_queries FROM public;\nGRANT ALL ON PROCEDURE run_queued_queries TO pgstac_admin;\n\nREVOKE ALL PRIVILEGES ON FUNCTION run_queued_queries_intransaction FROM public;\nGRANT ALL ON FUNCTION run_queued_queries_intransaction TO pgstac_admin;\n\nRESET ROLE;\n\nSET ROLE pgstac_ingest;\nSELECT update_partition_stats_q(partition) FROM partitions_view;\nSELECT set_version('0.9.9');\n" }, { "path": "src/pgstac/migrations/pgstac.unreleased.sql", "content": "RESET ROLE;\nDO $$\nDECLARE\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='postgis') THEN\n CREATE EXTENSION IF NOT EXISTS postgis;\n END IF;\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='btree_gist') THEN\n CREATE EXTENSION IF NOT EXISTS btree_gist;\n END IF;\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='unaccent') THEN\n CREATE EXTENSION IF NOT EXISTS unaccent;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_admin;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_read;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_ingest;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n\nGRANT pgstac_admin TO current_user;\n\n-- Function to make sure pgstac_admin is the owner of items\nCREATE OR REPLACE FUNCTION pgstac_admin_owns() RETURNS VOID AS $$\nDECLARE\n f RECORD;\nBEGIN\n FOR f IN (\n SELECT\n concat(\n oid::regproc::text,\n '(',\n coalesce(pg_get_function_identity_arguments(oid),''),\n ')'\n ) AS name,\n CASE prokind WHEN 'f' THEN 'FUNCTION' WHEN 'p' THEN 'PROCEDURE' WHEN 'a' THEN 'AGGREGATE' END as typ\n FROM pg_proc\n WHERE\n pronamespace=to_regnamespace('pgstac')\n AND proowner != to_regrole('pgstac_admin')\n AND proname NOT LIKE 'pg_stat%'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n FOR f IN (\n SELECT\n oid::regclass::text as name,\n CASE relkind\n WHEN 'i' THEN 'INDEX'\n WHEN 'I' THEN 'INDEX'\n WHEN 'p' THEN 'TABLE'\n WHEN 'r' THEN 'TABLE'\n WHEN 'v' THEN 'VIEW'\n WHEN 'S' THEN 'SEQUENCE'\n ELSE NULL\n END as typ\n FROM pg_class\n WHERE relnamespace=to_regnamespace('pgstac') and relowner != to_regrole('pgstac_admin') AND relkind IN ('r','p','v','S') AND relname NOT LIKE 'pg_stat'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\nSELECT pgstac_admin_owns();\n\nCREATE SCHEMA IF NOT EXISTS pgstac AUTHORIZATION pgstac_admin;\n\nGRANT ALL ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_admin;\n\nALTER ROLE pgstac_admin SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_read SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_ingest SET SEARCH_PATH TO pgstac, public;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\n\nGRANT pgstac_read TO pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\n\nSET ROLE pgstac_admin;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\nRESET ROLE;\n\nSET ROLE pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\nRESET ROLE;\n\nSET SEARCH_PATH TO pgstac, public;\nSET ROLE pgstac_admin;\n\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS analyze_items;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS validate_constraints;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n-- Install these idempotently as migrations do not put them before trying to modify the collections table\n\n\nCREATE OR REPLACE FUNCTION collection_geom(content jsonb)\nRETURNS geometry AS $$\n WITH box AS (SELECT content->'extent'->'spatial'->'bbox'->0 as box)\n SELECT\n st_makeenvelope(\n (box->>0)::float,\n (box->>1)::float,\n (box->>2)::float,\n (box->>3)::float,\n 4326\n )\n FROM box;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION collection_datetime(content jsonb)\nRETURNS timestamptz AS $$\n SELECT\n CASE\n WHEN\n (content->'extent'->'temporal'->'interval'->0->>0) IS NULL\n THEN '-infinity'::timestamptz\n ELSE\n (content->'extent'->'temporal'->'interval'->0->>0)::timestamptz\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION collection_enddatetime(content jsonb)\nRETURNS timestamptz AS $$\n SELECT\n CASE\n WHEN\n (content->'extent'->'temporal'->'interval'->0->>1) IS NULL\n THEN 'infinity'::timestamptz\n ELSE\n (content->'extent'->'temporal'->'interval'->0->>1)::timestamptz\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE TABLE IF NOT EXISTS migrations (\n version text PRIMARY KEY,\n datetime timestamptz DEFAULT clock_timestamp() NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION get_version() RETURNS text AS $$\n SELECT version FROM pgstac.migrations ORDER BY datetime DESC, version DESC LIMIT 1;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION set_version(text) RETURNS text AS $$\n INSERT INTO pgstac.migrations (version) VALUES ($1)\n ON CONFLICT DO NOTHING\n RETURNING version;\n$$ LANGUAGE SQL;\n\n\nCREATE TABLE IF NOT EXISTS pgstac_settings (\n name text PRIMARY KEY,\n value text NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION table_empty(text) RETURNS boolean AS $$\nDECLARE\n retval boolean;\nBEGIN\n EXECUTE format($q$\n SELECT NOT EXISTS (SELECT 1 FROM %I LIMIT 1)\n $q$,\n $1\n ) INTO retval;\n RETURN retval;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION get_setting(IN _setting text, IN conf jsonb DEFAULT NULL) RETURNS text AS $$\nSELECT COALESCE(\n nullif(conf->>_setting, ''),\n nullif(current_setting(concat('pgstac.',_setting), TRUE),''),\n nullif((SELECT value FROM pgstac.pgstac_settings WHERE name=_setting),'')\n);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_setting_bool(IN _setting text, IN conf jsonb DEFAULT NULL) RETURNS boolean AS $$\nSELECT COALESCE(\n nullif(conf->>_setting, ''),\n nullif(current_setting(concat('pgstac.',_setting), TRUE),''),\n nullif((SELECT value FROM pgstac.pgstac_settings WHERE name=_setting),''),\n 'FALSE'\n)::boolean;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION base_url(conf jsonb DEFAULT NULL) RETURNS text AS $$\n SELECT COALESCE(pgstac.get_setting('base_url', conf), '.');\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION additional_properties() RETURNS boolean AS $$\n SELECT pgstac.get_setting_bool('additional_properties');\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION readonly(conf jsonb DEFAULT NULL) RETURNS boolean AS $$\n SELECT pgstac.get_setting_bool('readonly', conf);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context(conf jsonb DEFAULT NULL) RETURNS text AS $$\n SELECT pgstac.get_setting('context', conf);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context_estimated_count(conf jsonb DEFAULT NULL) RETURNS int AS $$\n SELECT pgstac.get_setting('context_estimated_count', conf)::int;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_estimated_cost();\nCREATE OR REPLACE FUNCTION context_estimated_cost(conf jsonb DEFAULT NULL) RETURNS float AS $$\n SELECT pgstac.get_setting('context_estimated_cost', conf)::float;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_stats_ttl();\nCREATE OR REPLACE FUNCTION context_stats_ttl(conf jsonb DEFAULT NULL) RETURNS interval AS $$\n SELECT pgstac.get_setting('context_stats_ttl', conf)::interval;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION t2s(text) RETURNS text AS $$\n SELECT extract(epoch FROM $1::interval)::text || ' s';\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE STRICT;\n\nCREATE OR REPLACE FUNCTION age_ms(a timestamptz, b timestamptz DEFAULT clock_timestamp()) RETURNS float AS $$\n SELECT abs(extract(epoch from age(a,b)) * 1000);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION queue_timeout() RETURNS interval AS $$\n SELECT t2s(coalesce(\n get_setting('queue_timeout'),\n '1h'\n ))::interval;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION notice(VARIADIC text[]) RETURNS boolean AS $$\nDECLARE\ndebug boolean := current_setting('pgstac.debug', true);\nBEGIN\n IF debug THEN\n RAISE NOTICE 'NOTICE FROM FUNC: % >>>>> %', concat_ws(' | ', $1), clock_timestamp();\n RETURN TRUE;\n END IF;\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION empty_arr(ANYARRAY) RETURNS BOOLEAN AS $$\nSELECT CASE\n WHEN $1 IS NULL THEN TRUE\n WHEN cardinality($1)<1 THEN TRUE\nELSE FALSE\nEND;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_intersection(_a ANYARRAY, _b ANYARRAY) RETURNS ANYARRAY AS $$\n SELECT ARRAY ( SELECT unnest(_a) INTERSECT SELECT UNNEST(_b) );\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION array_map_ident(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_ident(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_map_literal(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_literal(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION array_reverse(anyarray) RETURNS anyarray AS $$\nSELECT ARRAY(\n SELECT $1[i]\n FROM generate_subscripts($1,1) AS s(i)\n ORDER BY i DESC\n);\n$$ LANGUAGE SQL STRICT IMMUTABLE;\n\nDROP TABLE IF EXISTS query_queue;\nCREATE TABLE query_queue (\n query text PRIMARY KEY,\n added timestamptz DEFAULT now()\n);\n\nDROP TABLE IF EXISTS query_queue_history;\nCREATE TABLE query_queue_history(\n query text,\n added timestamptz NOT NULL,\n finished timestamptz NOT NULL DEFAULT now(),\n error text\n);\n\nCREATE OR REPLACE PROCEDURE run_queued_queries() AS $$\nDECLARE\n qitem query_queue%ROWTYPE;\n timeout_ts timestamptz;\n error text;\n cnt int := 0;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n DELETE FROM query_queue WHERE query = (SELECT query FROM query_queue ORDER BY added DESC LIMIT 1 FOR UPDATE SKIP LOCKED) RETURNING * INTO qitem;\n IF NOT FOUND THEN\n EXIT;\n END IF;\n cnt := cnt + 1;\n BEGIN\n RAISE NOTICE 'RUNNING QUERY: %', qitem.query;\n EXECUTE qitem.query;\n EXCEPTION WHEN others THEN\n error := format('%s | %s', SQLERRM, SQLSTATE);\n END;\n INSERT INTO query_queue_history (query, added, finished, error)\n VALUES (qitem.query, qitem.added, clock_timestamp(), error);\n COMMIT;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION run_queued_queries_intransaction() RETURNS int AS $$\nDECLARE\n qitem query_queue%ROWTYPE;\n timeout_ts timestamptz;\n error text;\n cnt int := 0;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n DELETE FROM query_queue WHERE query = (SELECT query FROM query_queue ORDER BY added DESC LIMIT 1 FOR UPDATE SKIP LOCKED) RETURNING * INTO qitem;\n IF NOT FOUND THEN\n RETURN cnt;\n END IF;\n cnt := cnt + 1;\n BEGIN\n qitem.query := regexp_replace(qitem.query, 'CONCURRENTLY', '');\n RAISE NOTICE 'RUNNING QUERY: %', qitem.query;\n\n EXECUTE qitem.query;\n EXCEPTION WHEN others THEN\n error := format('%s | %s', SQLERRM, SQLSTATE);\n RAISE WARNING '%', error;\n END;\n INSERT INTO query_queue_history (query, added, finished, error)\n VALUES (qitem.query, qitem.added, clock_timestamp(), error);\n END LOOP;\n RETURN cnt;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION run_or_queue(query text) RETURNS VOID AS $$\nDECLARE\n use_queue text := COALESCE(get_setting('use_queue'), 'FALSE')::boolean;\nBEGIN\n IF get_setting_bool('debug') THEN\n RAISE NOTICE '%', query;\n END IF;\n IF use_queue THEN\n INSERT INTO query_queue (query) VALUES (query) ON CONFLICT DO NOTHING;\n ELSE\n EXECUTE query;\n END IF;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n\nDROP FUNCTION IF EXISTS check_pgstac_settings;\nCREATE OR REPLACE FUNCTION check_pgstac_settings(_sysmem text DEFAULT NULL) RETURNS VOID AS $$\nDECLARE\n settingval text;\n sysmem bigint := pg_size_bytes(_sysmem);\n effective_cache_size bigint := pg_size_bytes(current_setting('effective_cache_size', TRUE));\n shared_buffers bigint := pg_size_bytes(current_setting('shared_buffers', TRUE));\n work_mem bigint := pg_size_bytes(current_setting('work_mem', TRUE));\n max_connections int := current_setting('max_connections', TRUE);\n maintenance_work_mem bigint := pg_size_bytes(current_setting('maintenance_work_mem', TRUE));\n seq_page_cost float := current_setting('seq_page_cost', TRUE);\n random_page_cost float := current_setting('random_page_cost', TRUE);\n temp_buffers bigint := pg_size_bytes(current_setting('temp_buffers', TRUE));\n r record;\nBEGIN\n IF _sysmem IS NULL THEN\n RAISE NOTICE 'Call function with the size of your system memory `SELECT check_pgstac_settings(''4GB'')` to get pg system setting recommendations.';\n ELSE\n IF effective_cache_size < (sysmem * 0.5) THEN\n RAISE WARNING 'effective_cache_size of % is set low for a system with %. Recomended value between % and %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.5), pg_size_pretty(sysmem * 0.75);\n ELSIF effective_cache_size > (sysmem * 0.75) THEN\n RAISE WARNING 'effective_cache_size of % is set high for a system with %. Recomended value between % and %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.5), pg_size_pretty(sysmem * 0.75);\n ELSE\n RAISE NOTICE 'effective_cache_size of % is set appropriately for a system with %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem);\n END IF;\n\n IF shared_buffers < (sysmem * 0.2) THEN\n RAISE WARNING 'shared_buffers of % is set low for a system with %. Recomended value between % and %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.2), pg_size_pretty(sysmem * 0.3);\n ELSIF shared_buffers > (sysmem * 0.3) THEN\n RAISE WARNING 'shared_buffers of % is set high for a system with %. Recomended value between % and %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.2), pg_size_pretty(sysmem * 0.3);\n ELSE\n RAISE NOTICE 'shared_buffers of % is set appropriately for a system with %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem);\n END IF;\n shared_buffers = sysmem * 0.3;\n IF maintenance_work_mem < (sysmem * 0.2) THEN\n RAISE WARNING 'maintenance_work_mem of % is set low for shared_buffers of %. Recomended value between % and %', pg_size_pretty(maintenance_work_mem), pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers * 0.2), pg_size_pretty(shared_buffers * 0.3);\n ELSIF maintenance_work_mem > (shared_buffers * 0.3) THEN\n RAISE WARNING 'maintenance_work_mem of % is set high for shared_buffers of %. Recomended value between % and %', pg_size_pretty(maintenance_work_mem), pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers * 0.2), pg_size_pretty(shared_buffers * 0.3);\n ELSE\n RAISE NOTICE 'maintenance_work_mem of % is set appropriately for shared_buffers of %', pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers);\n END IF;\n\n IF work_mem * max_connections > shared_buffers THEN\n RAISE WARNING 'work_mem setting of % is set high for % max_connections please reduce work_mem to % or decrease max_connections to %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers/max_connections), floor(shared_buffers/work_mem);\n ELSIF work_mem * max_connections < (shared_buffers * 0.75) THEN\n RAISE WARNING 'work_mem setting of % is set low for % max_connections you may consider raising work_mem to % or increasing max_connections to %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers/max_connections), floor(shared_buffers/work_mem);\n ELSE\n RAISE NOTICE 'work_mem setting of % and max_connections of % are adequate for shared_buffers of %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers);\n END IF;\n\n IF random_page_cost / seq_page_cost != 1.1 THEN\n RAISE WARNING 'random_page_cost (%) /seq_page_cost (%) should be set to 1.1 for SSD. Change random_page_cost to %', random_page_cost, seq_page_cost, 1.1 * seq_page_cost;\n ELSE\n RAISE NOTICE 'random_page_cost and seq_page_cost set appropriately for SSD';\n END IF;\n\n IF temp_buffers < greatest(pg_size_bytes('128MB'),(maintenance_work_mem / 2)) THEN\n RAISE WARNING 'pgstac makes heavy use of temp tables, consider raising temp_buffers from % to %', pg_size_pretty(temp_buffers), greatest('128MB', pg_size_pretty((shared_buffers / 16)));\n END IF;\n END IF;\n\n RAISE NOTICE 'VALUES FOR PGSTAC VARIABLES';\n RAISE NOTICE 'These can be set either as GUC system variables or by setting in the pgstac_settings table.';\n\n FOR r IN SELECT name, get_setting(name) as setting, CASE WHEN current_setting(concat('pgstac.',name), TRUE) IS NOT NULL THEN concat('pgstac.',name, ' GUC') WHEN value IS NOT NULL THEN 'pgstac_settings table' ELSE 'Not Set' END as loc FROM pgstac_settings LOOP\n RAISE NOTICE '% is set to % from the %', r.name, r.setting, r.loc;\n END LOOP;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pg_cron';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider intalling pg_cron which can be used to automate tasks';\n ELSE\n RAISE NOTICE 'pg_cron % is installed', settingval;\n END IF;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pgstattuple';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider installing the pgstattuple extension which can be used to help maintain tables and indexes.';\n ELSE\n RAISE NOTICE 'pgstattuple % is installed', settingval;\n END IF;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pg_stat_statements';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider installing the pg_stat_statements extension which is very helpful for tracking the types of queries on the system';\n ELSE\n RAISE NOTICE 'pg_stat_statements % is installed', settingval;\n IF current_setting('pg_stat_statements.track_statements', TRUE) IS DISTINCT FROM 'all' THEN\n RAISE WARNING 'SET pg_stat_statements.track_statements TO ''all''; --In order to track statements within functions.';\n END IF;\n END IF;\n\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac, public SET CLIENT_MIN_MESSAGES TO NOTICE;\nCREATE OR REPLACE FUNCTION to_int(jsonb) RETURNS int AS $$\n SELECT floor(($1->>0)::float)::int;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION to_float(jsonb) RETURNS float AS $$\n SELECT ($1->>0)::float;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION to_tstz(jsonb) RETURNS timestamptz AS $$\n SELECT ($1->>0)::timestamptz;\n$$ LANGUAGE SQL IMMUTABLE STRICT SET TIME ZONE 'UTC' COST 5000 PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION to_text(jsonb) RETURNS text AS $$\n SELECT CASE WHEN jsonb_typeof($1) IN ('array','object') THEN $1::text ELSE $1->>0 END;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION to_text_array(jsonb) RETURNS text[] AS $$\n SELECT\n CASE jsonb_typeof($1)\n WHEN 'array' THEN ARRAY(SELECT jsonb_array_elements_text($1))\n ELSE ARRAY[$1->>0]\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION bbox_geom(_bbox jsonb) RETURNS geometry AS $$\nSELECT CASE jsonb_array_length(_bbox)\n WHEN 4 THEN\n ST_SetSRID(ST_MakeEnvelope(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float,\n (_bbox->>3)::float\n ),4326)\n WHEN 6 THEN\n ST_SetSRID(ST_3DMakeBox(\n ST_MakePoint(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float\n ),\n ST_MakePoint(\n (_bbox->>3)::float,\n (_bbox->>4)::float,\n (_bbox->>5)::float\n )\n ),4326)\n ELSE null END;\n;\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION geom_bbox(_geom geometry) RETURNS jsonb AS $$\n SELECT jsonb_build_array(\n st_xmin(_geom),\n st_ymin(_geom),\n st_xmax(_geom),\n st_ymax(_geom)\n );\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION flip_jsonb_array(j jsonb) RETURNS jsonb AS $$\n SELECT jsonb_agg(value) FROM (SELECT value FROM jsonb_array_elements(j) WITH ORDINALITY ORDER BY ordinality DESC) as t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION explode_dotpaths(j jsonb) RETURNS SETOF text[] AS $$\n SELECT string_to_array(p, '.') as e FROM jsonb_array_elements_text(j) p;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION explode_dotpaths_recurse(IN j jsonb) RETURNS SETOF text[] AS $$\n WITH RECURSIVE t AS (\n SELECT e FROM explode_dotpaths(j) e\n UNION ALL\n SELECT e[1:cardinality(e)-1]\n FROM t\n WHERE cardinality(e)>1\n ) SELECT e FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION jsonb_set_nested(j jsonb, path text[], val jsonb) RETURNS jsonb AS $$\nDECLARE\nBEGIN\n IF cardinality(path) > 1 THEN\n FOR i IN 1..(cardinality(path)-1) LOOP\n IF j #> path[:i] IS NULL THEN\n j := jsonb_set_lax(j, path[:i], '{}', TRUE);\n END IF;\n END LOOP;\n END IF;\n RETURN jsonb_set_lax(j, path, val, true);\n\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\n\n\nCREATE OR REPLACE FUNCTION jsonb_include(j jsonb, f jsonb) RETURNS jsonb AS $$\nDECLARE\n includes jsonb := f-> 'include';\n outj jsonb := '{}'::jsonb;\n path text[];\nBEGIN\n IF\n includes IS NULL\n OR jsonb_array_length(includes) = 0\n THEN\n RETURN j;\n ELSE\n includes := includes || (\n CASE WHEN j ? 'collection' THEN\n '[\"id\",\"collection\"]'\n ELSE\n '[\"id\"]'\n END)::jsonb;\n FOR path IN SELECT explode_dotpaths(includes) LOOP\n outj := jsonb_set_nested(outj, path, j #> path);\n END LOOP;\n END IF;\n RETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_exclude(j jsonb, f jsonb) RETURNS jsonb AS $$\nDECLARE\n excludes jsonb := f-> 'exclude';\n outj jsonb := j;\n path text[];\nBEGIN\n IF\n excludes IS NULL\n OR jsonb_array_length(excludes) = 0\n THEN\n RETURN j;\n ELSE\n FOR path IN SELECT explode_dotpaths(excludes) LOOP\n outj := outj #- path;\n END LOOP;\n END IF;\n RETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_fields(j jsonb, f jsonb DEFAULT '{\"fields\":[]}') RETURNS jsonb AS $$\n SELECT jsonb_exclude(jsonb_include(j, f), f);\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION merge_jsonb(_a jsonb, _b jsonb) RETURNS jsonb AS $$\n SELECT\n CASE\n WHEN _a = '\"𒍟※\"'::jsonb THEN NULL\n WHEN _a IS NULL OR jsonb_typeof(_a) = 'null' THEN _b\n WHEN jsonb_typeof(_a) = 'object' AND jsonb_typeof(_b) = 'object' THEN\n (\n SELECT\n jsonb_strip_nulls(\n jsonb_object_agg(\n key,\n merge_jsonb(a.value, b.value)\n )\n )\n FROM\n jsonb_each(coalesce(_a,'{}'::jsonb)) as a\n FULL JOIN\n jsonb_each(coalesce(_b,'{}'::jsonb)) as b\n USING (key)\n )\n WHEN\n jsonb_typeof(_a) = 'array'\n AND jsonb_typeof(_b) = 'array'\n AND jsonb_array_length(_a) = jsonb_array_length(_b)\n THEN\n (\n SELECT jsonb_agg(m) FROM\n ( SELECT\n merge_jsonb(\n jsonb_array_elements(_a),\n jsonb_array_elements(_b)\n ) as m\n ) as l\n )\n ELSE _a\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION strip_jsonb(_a jsonb, _b jsonb) RETURNS jsonb AS $$\n SELECT\n CASE\n\n WHEN (_a IS NULL OR jsonb_typeof(_a) = 'null') AND _b IS NOT NULL AND jsonb_typeof(_b) != 'null' THEN '\"𒍟※\"'::jsonb\n WHEN _b IS NULL OR jsonb_typeof(_a) = 'null' THEN _a\n WHEN _a = _b AND jsonb_typeof(_a) = 'object' THEN '{}'::jsonb\n WHEN _a = _b THEN NULL\n WHEN jsonb_typeof(_a) = 'object' AND jsonb_typeof(_b) = 'object' THEN\n (\n SELECT\n jsonb_strip_nulls(\n jsonb_object_agg(\n key,\n strip_jsonb(a.value, b.value)\n )\n )\n FROM\n jsonb_each(_a) as a\n FULL JOIN\n jsonb_each(_b) as b\n USING (key)\n )\n WHEN\n jsonb_typeof(_a) = 'array'\n AND jsonb_typeof(_b) = 'array'\n AND jsonb_array_length(_a) = jsonb_array_length(_b)\n THEN\n (\n SELECT jsonb_agg(m) FROM\n ( SELECT\n strip_jsonb(\n jsonb_array_elements(_a),\n jsonb_array_elements(_b)\n ) as m\n ) as l\n )\n ELSE _a\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION nullif_jsonbnullempty(j jsonb) RETURNS jsonb AS $$\n SELECT nullif(nullif(nullif(j,'null'::jsonb),'{}'::jsonb),'[]'::jsonb);\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION jsonb_array_unique(j jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(jsonb_agg(DISTINCT a)) v FROM jsonb_array_elements(j) a;\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_concat_ignorenull(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT coalesce(a,'[]'::jsonb) || coalesce(b,'[]'::jsonb);\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_least(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(least(nullif_jsonbnullempty(a), nullif_jsonbnullempty(b)));\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_greatest(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(greatest(a, b));\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION first_notnull_sfunc(anyelement, anyelement) RETURNS anyelement AS $$\n SELECT COALESCE($1,$2);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE AGGREGATE first_notnull(anyelement)(\n SFUNC = first_notnull_sfunc,\n STYPE = anyelement\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_array_unique_merge(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_concat_ignorenull,\n FINALFUNC = jsonb_array_unique\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_min(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_least\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_max(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_greatest\n);\n/* looks for a geometry in a stac item first from geometry and falling back to bbox */\nCREATE OR REPLACE FUNCTION stac_geom(value jsonb) RETURNS geometry AS $$\nSELECT\n CASE\n WHEN value ? 'intersects' THEN\n ST_GeomFromGeoJSON(value->>'intersects')\n WHEN value ? 'geometry' THEN\n ST_GeomFromGeoJSON(value->>'geometry')\n WHEN value ? 'bbox' THEN\n pgstac.bbox_geom(value->'bbox')\n ELSE NULL\n END as geometry\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION stac_daterange(\n value jsonb\n) RETURNS tstzrange AS $$\nDECLARE\n props jsonb := value;\n dt timestamptz;\n edt timestamptz;\nBEGIN\n IF props ? 'properties' THEN\n props := props->'properties';\n END IF;\n IF\n props ? 'start_datetime'\n AND props->>'start_datetime' IS NOT NULL\n AND props ? 'end_datetime'\n AND props->>'end_datetime' IS NOT NULL\n THEN\n dt := props->>'start_datetime';\n edt := props->>'end_datetime';\n IF dt > edt THEN\n RAISE EXCEPTION 'start_datetime must be < end_datetime';\n END IF;\n ELSE\n dt := props->>'datetime';\n edt := props->>'datetime';\n END IF;\n IF dt is NULL OR edt IS NULL THEN\n RAISE NOTICE 'DT: %, EDT: %', dt, edt;\n RAISE EXCEPTION 'Either datetime (%) or both start_datetime (%) and end_datetime (%) must be set.', props->>'datetime',props->>'start_datetime',props->>'end_datetime';\n END IF;\n RETURN tstzrange(dt, edt, '[]');\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_datetime(value jsonb) RETURNS timestamptz AS $$\n SELECT lower(stac_daterange(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_end_datetime(value jsonb) RETURNS timestamptz AS $$\n SELECT upper(stac_daterange(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE TABLE IF NOT EXISTS stac_extensions(\n url text PRIMARY KEY,\n content jsonb\n);\nCREATE OR REPLACE FUNCTION collection_base_item(content jsonb) RETURNS jsonb AS $$\n SELECT jsonb_build_object(\n 'type', 'Feature',\n 'stac_version', content->'stac_version',\n 'assets', content->'item_assets',\n 'collection', content->'id'\n );\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE TABLE IF NOT EXISTS collections (\n key bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,\n id text GENERATED ALWAYS AS (content->>'id') STORED UNIQUE NOT NULL,\n content JSONB NOT NULL,\n base_item jsonb GENERATED ALWAYS AS (pgstac.collection_base_item(content)) STORED,\n geometry geometry GENERATED ALWAYS AS (pgstac.collection_geom(content)) STORED,\n datetime timestamptz GENERATED ALWAYS AS (pgstac.collection_datetime(content)) STORED,\n end_datetime timestamptz GENERATED ALWAYS AS (pgstac.collection_enddatetime(content)) STORED,\n private jsonb,\n partition_trunc text CHECK (partition_trunc IN ('year', 'month'))\n);\n\nCREATE OR REPLACE FUNCTION collection_base_item(cid text) RETURNS jsonb AS $$\n SELECT pgstac.collection_base_item(content) FROM pgstac.collections WHERE id = cid LIMIT 1;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION create_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION update_collection(data jsonb) RETURNS VOID AS $$\nDECLARE\n out collections%ROWTYPE;\nBEGIN\n UPDATE collections SET content=data WHERE id = data->>'id' RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ON CONFLICT (id) DO\n UPDATE\n SET content=EXCLUDED.content\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION delete_collection(_id text) RETURNS VOID AS $$\nDECLARE\n out collections%ROWTYPE;\nBEGIN\n DELETE FROM collections WHERE id = _id RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION get_collection(id text) RETURNS jsonb AS $$\n SELECT content FROM collections\n WHERE id=$1\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION all_collections() RETURNS jsonb AS $$\n SELECT coalesce(jsonb_agg(content), '[]'::jsonb) FROM collections;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION collection_delete_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n collection_base_partition text := concat('_items_', OLD.key);\nBEGIN\n EXECUTE format($q$\n DELETE FROM partition_stats WHERE partition IN (\n SELECT partition FROM partition_sys_meta\n WHERE collection=%L\n );\n DROP TABLE IF EXISTS %I CASCADE;\n $q$,\n OLD.id,\n collection_base_partition\n );\n RETURN OLD;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER collection_delete_trigger BEFORE DELETE ON collections\nFOR EACH ROW EXECUTE FUNCTION collection_delete_trigger_func();\nCREATE OR REPLACE FUNCTION queryable_signature(n text, c text[]) RETURNS text AS $$\n SELECT concat(n, c);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TABLE queryables (\n id bigint GENERATED ALWAYS AS identity PRIMARY KEY,\n name text NOT NULL,\n collection_ids text[], -- used to determine what partitions to create indexes on\n definition jsonb,\n property_path text,\n property_wrapper text,\n property_index_type text\n);\nCREATE INDEX queryables_name_idx ON queryables (name);\nCREATE INDEX queryables_collection_idx ON queryables USING GIN (collection_ids);\nCREATE INDEX queryables_property_wrapper_idx ON queryables (property_wrapper);\n\nCREATE OR REPLACE FUNCTION pgstac.queryables_constraint_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n allcollections text[];\nBEGIN\n RAISE NOTICE 'Making sure that name/collection is unique for queryables %', NEW;\n IF NEW.collection_ids IS NOT NULL THEN\n IF EXISTS (\n SELECT 1\n FROM unnest(NEW.collection_ids) c\n LEFT JOIN\n collections\n ON (collections.id = c)\n WHERE collections.id IS NULL\n ) THEN\n RAISE foreign_key_violation USING MESSAGE = format(\n 'One or more collections in %s do not exist.', NEW.collection_ids\n );\n RETURN NULL;\n END IF;\n END IF;\n IF TG_OP = 'INSERT' THEN\n IF EXISTS (\n SELECT 1 FROM queryables q\n WHERE\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n )\n ) THEN\n RAISE unique_violation USING MESSAGE = format(\n 'There is already a queryable for %s for a collection in %s: %s',\n NEW.name,\n NEW.collection_ids,\n\t\t\t\t(SELECT json_agg(row_to_json(q)) FROM queryables q WHERE\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n ))\n );\n RETURN NULL;\n END IF;\n END IF;\n IF TG_OP = 'UPDATE' THEN\n IF EXISTS (\n SELECT 1 FROM queryables q\n WHERE\n q.id != NEW.id\n AND\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n )\n ) THEN\n RAISE unique_violation\n USING MESSAGE = format(\n 'There is already a queryable for %s for a collection in %s',\n NEW.name,\n NEW.collection_ids\n );\n RETURN NULL;\n END IF;\n END IF;\n\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER queryables_constraint_insert_trigger\nBEFORE INSERT ON queryables\nFOR EACH ROW EXECUTE PROCEDURE queryables_constraint_triggerfunc();\n\nCREATE TRIGGER queryables_constraint_update_trigger\nBEFORE UPDATE ON queryables\nFOR EACH ROW\nWHEN (NEW.name = OLD.name AND NEW.collection_ids IS DISTINCT FROM OLD.collection_ids)\nEXECUTE PROCEDURE queryables_constraint_triggerfunc();\n\n\nCREATE OR REPLACE FUNCTION array_to_path(arr text[]) RETURNS text AS $$\n SELECT string_agg(\n quote_literal(v),\n '->'\n ) FROM unnest(arr) v;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\n\n\nCREATE OR REPLACE FUNCTION queryable(\n IN dotpath text,\n OUT path text,\n OUT expression text,\n OUT wrapper text,\n OUT nulled_wrapper text\n) AS $$\nDECLARE\n q RECORD;\n path_elements text[];\nBEGIN\n dotpath := replace(dotpath, 'properties.', '');\n IF dotpath = 'start_datetime' THEN\n dotpath := 'datetime';\n END IF;\n IF dotpath IN ('id', 'geometry', 'datetime', 'end_datetime', 'collection') THEN\n path := dotpath;\n expression := dotpath;\n wrapper := NULL;\n RETURN;\n END IF;\n\n SELECT * INTO q FROM queryables\n WHERE\n name=dotpath\n OR name = 'properties.' || dotpath\n OR name = replace(dotpath, 'properties.', '')\n ;\n IF q.property_wrapper IS NULL THEN\n IF q.definition->>'type' = 'number' THEN\n wrapper := 'to_float';\n nulled_wrapper := wrapper;\n ELSIF q.definition->>'format' = 'date-time' THEN\n wrapper := 'to_tstz';\n nulled_wrapper := wrapper;\n ELSE\n nulled_wrapper := NULL;\n wrapper := 'to_text';\n END IF;\n ELSE\n wrapper := q.property_wrapper;\n nulled_wrapper := wrapper;\n END IF;\n IF q.property_path IS NOT NULL THEN\n path := q.property_path;\n ELSE\n path_elements := string_to_array(dotpath, '.');\n IF path_elements[1] IN ('links', 'assets', 'stac_version', 'stac_extensions') THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSIF path_elements[1] = 'properties' THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSE\n path := format($F$content->'properties'->%s$F$, array_to_path(path_elements));\n END IF;\n END IF;\n expression := format('%I(%s)', wrapper, path);\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\nCREATE OR REPLACE FUNCTION unnest_collection(collection_ids text[] DEFAULT NULL) RETURNS SETOF text AS $$\n DECLARE\n BEGIN\n IF collection_ids IS NULL THEN\n RETURN QUERY SELECT id FROM collections;\n END IF;\n RETURN QUERY SELECT unnest(collection_ids);\n END;\n$$ LANGUAGE PLPGSQL STABLE;\n\nCREATE OR REPLACE FUNCTION normalize_indexdef(def text) RETURNS text AS $$\nDECLARE\nBEGIN\n def := btrim(def, ' \\n\\t');\n\tdef := regexp_replace(def, '^CREATE (UNIQUE )?INDEX ([^ ]* )?ON (ONLY )?([^ ]* )?', '', 'i');\n RETURN def;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION indexdef(q queryables) RETURNS text AS $$\n DECLARE\n out text;\n BEGIN\n IF q.name = 'id' THEN\n out := 'CREATE UNIQUE INDEX ON %I USING btree (id)';\n ELSIF q.name = 'datetime' THEN\n out := 'CREATE INDEX ON %I USING btree (datetime DESC, end_datetime)';\n ELSIF q.name = 'geometry' THEN\n out := 'CREATE INDEX ON %I USING gist (geometry)';\n ELSE\n out := format($q$CREATE INDEX ON %%I USING %s (%s(((content -> 'properties'::text) -> %L::text)))$q$,\n lower(COALESCE(q.property_index_type, 'BTREE')),\n lower(COALESCE(q.property_wrapper, 'to_text')),\n q.name\n );\n END IF;\n RETURN btrim(out, ' \\n\\t');\n END;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nDROP VIEW IF EXISTS pgstac_indexes;\nCREATE VIEW pgstac_indexes AS\nSELECT\n i.schemaname,\n i.tablename,\n i.indexname,\n regexp_replace(btrim(replace(replace(indexdef, i.indexname, ''),'pgstac.',''),' \\t\\n'), '[ ]+', ' ', 'g') as idx,\n COALESCE(\n (regexp_match(indexdef, '\\(([a-zA-Z]+)\\)'))[1],\n (regexp_match(indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_-]+)''::text'))[1],\n CASE WHEN indexdef ~* '\\(datetime desc, end_datetime\\)' THEN 'datetime' ELSE NULL END\n ) AS field,\n pg_table_size(i.indexname::text) as index_size,\n pg_size_pretty(pg_table_size(i.indexname::text)) as index_size_pretty\nFROM\n pg_indexes i\nWHERE i.schemaname='pgstac' and i.tablename ~ '_items_' AND indexdef !~* ' only ';\n\nDROP VIEW IF EXISTS pgstac_index_stats;\nCREATE VIEW pgstac_indexes_stats AS\nSELECT\n i.schemaname,\n i.tablename,\n i.indexname,\n indexdef,\n COALESCE(\n (regexp_match(indexdef, '\\(([a-zA-Z]+)\\)'))[1],\n (regexp_match(indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_]+)''::text'))[1],\n CASE WHEN indexdef ~* '\\(datetime desc, end_datetime\\)' THEN 'datetime_end_datetime' ELSE NULL END\n ) AS field,\n pg_table_size(i.indexname::text) as index_size,\n pg_size_pretty(pg_table_size(i.indexname::text)) as index_size_pretty,\n n_distinct,\n most_common_vals::text::text[],\n most_common_freqs::text::text[],\n histogram_bounds::text::text[],\n correlation\nFROM\n pg_indexes i\n LEFT JOIN pg_stats s ON (s.tablename = i.indexname)\nWHERE i.schemaname='pgstac' and i.tablename ~ '_items_';\n\nCREATE OR REPLACE FUNCTION queryable_indexes(\n IN treeroot text DEFAULT 'items',\n IN changes boolean DEFAULT FALSE,\n OUT collection text,\n OUT partition text,\n OUT field text,\n OUT indexname text,\n OUT existing_idx text,\n OUT queryable_idx text\n) RETURNS SETOF RECORD AS $$\nWITH p AS (\n SELECT\n relid::text as partition,\n replace(replace(\n CASE\n WHEN parentrelid::regclass::text='items' THEN pg_get_expr(c.relpartbound, c.oid)\n ELSE pg_get_expr(parent.relpartbound, parent.oid)\n END,\n 'FOR VALUES IN (''',''), ''')',\n ''\n ) AS collection\n FROM pg_partition_tree(treeroot)\n JOIN pg_class c ON (relid::regclass = c.oid)\n JOIN pg_class parent ON (parentrelid::regclass = parent.oid AND isleaf)\n ), i AS (\n SELECT\n partition,\n indexname,\n regexp_replace(btrim(replace(replace(indexdef, indexname, ''),'pgstac.',''),' \\t\\n'), '[ ]+', ' ', 'g') as iidx,\n COALESCE(\n (regexp_match(indexdef, '\\(([a-zA-Z]+)\\)'))[1],\n (regexp_match(indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_-]+)''::text'))[1],\n CASE WHEN indexdef ~* '\\(datetime desc, end_datetime\\)' THEN 'datetime' ELSE NULL END\n ) AS field\n FROM\n pg_indexes\n JOIN p ON (tablename=partition)\n ), q AS (\n SELECT\n name AS field,\n collection,\n partition,\n format(indexdef(queryables), partition) as qidx\n FROM queryables, unnest_collection(queryables.collection_ids) collection\n JOIN p USING (collection)\n WHERE property_index_type IS NOT NULL OR name IN ('datetime','geometry','id')\n )\n SELECT\n collection,\n partition,\n field,\n indexname,\n iidx as existing_idx,\n qidx as queryable_idx\n FROM i FULL JOIN q USING (field, partition)\n WHERE CASE WHEN changes THEN lower(iidx) IS DISTINCT FROM lower(qidx) ELSE TRUE END;\n;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION maintain_index(\n indexname text,\n queryable_idx text,\n dropindexes boolean DEFAULT FALSE,\n rebuildindexes boolean DEFAULT FALSE,\n idxconcurrently boolean DEFAULT FALSE\n) RETURNS VOID AS $$\nDECLARE\nBEGIN\n IF indexname IS NOT NULL THEN\n IF dropindexes OR queryable_idx IS NOT NULL THEN\n EXECUTE format('DROP INDEX IF EXISTS %I;', indexname);\n ELSIF rebuildindexes THEN\n IF idxconcurrently THEN\n EXECUTE format('REINDEX INDEX CONCURRENTLY %I;', indexname);\n ELSE\n EXECUTE format('REINDEX INDEX CONCURRENTLY %I;', indexname);\n END IF;\n END IF;\n END IF;\n IF queryable_idx IS NOT NULL THEN\n IF idxconcurrently THEN\n EXECUTE replace(queryable_idx, 'INDEX', 'INDEX CONCURRENTLY');\n ELSE EXECUTE queryable_idx;\n END IF;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\n\nset check_function_bodies to off;\nCREATE OR REPLACE FUNCTION maintain_partition_queries(\n part text DEFAULT 'items',\n dropindexes boolean DEFAULT FALSE,\n rebuildindexes boolean DEFAULT FALSE,\n idxconcurrently boolean DEFAULT FALSE\n) RETURNS SETOF text AS $$\nDECLARE\n rec record;\n q text;\nBEGIN\n FOR rec IN (\n SELECT * FROM queryable_indexes(part,true)\n ) LOOP\n q := format(\n 'SELECT maintain_index(\n %L,%L,%L,%L,%L\n );',\n rec.indexname,\n rec.queryable_idx,\n dropindexes,\n rebuildindexes,\n idxconcurrently\n );\n RAISE NOTICE 'Q: %', q;\n RETURN NEXT q;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION maintain_partitions(\n part text DEFAULT 'items',\n dropindexes boolean DEFAULT FALSE,\n rebuildindexes boolean DEFAULT FALSE\n) RETURNS VOID AS $$\n WITH t AS (\n SELECT run_or_queue(q) FROM maintain_partition_queries(part, dropindexes, rebuildindexes) q\n ) SELECT count(*) FROM t;\n$$ LANGUAGE SQL;\n\n\nCREATE OR REPLACE FUNCTION queryables_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\nBEGIN\n PERFORM maintain_partitions();\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER queryables_trigger AFTER INSERT OR UPDATE ON queryables\nFOR EACH STATEMENT EXECUTE PROCEDURE queryables_trigger_func();\n\n\nCREATE OR REPLACE FUNCTION get_queryables(_collection_ids text[] DEFAULT NULL) RETURNS jsonb AS $$\nDECLARE\nBEGIN\n -- Build up queryables if the input contains valid collection ids or is empty\n IF EXISTS (\n SELECT 1 FROM collections\n WHERE\n _collection_ids IS NULL\n OR cardinality(_collection_ids) = 0\n OR id = ANY(_collection_ids)\n )\n THEN\n RETURN (\n WITH base AS (\n SELECT\n unnest(collection_ids) as collection_id,\n name,\n coalesce(definition, '{\"type\":\"string\"}'::jsonb) as definition\n FROM queryables\n WHERE\n _collection_ids IS NULL OR\n _collection_ids = '{}'::text[] OR\n _collection_ids && collection_ids\n UNION ALL\n SELECT null, name, coalesce(definition, '{\"type\":\"string\"}'::jsonb) as definition\n FROM queryables WHERE collection_ids IS NULL OR collection_ids = '{}'::text[]\n ), g AS (\n SELECT\n name,\n first_notnull(definition) as definition,\n jsonb_array_unique_merge(definition->'enum') as enum,\n jsonb_min(definition->'minimum') as minimum,\n jsonb_min(definition->'maxiumn') as maximum\n FROM base\n GROUP BY 1\n )\n SELECT\n jsonb_build_object(\n '$schema', 'http://json-schema.org/draft-07/schema#',\n '$id', '',\n 'type', 'object',\n 'title', 'STAC Queryables.',\n 'properties', jsonb_object_agg(\n name,\n definition\n ||\n jsonb_strip_nulls(jsonb_build_object(\n 'enum', enum,\n 'minimum', minimum,\n 'maximum', maximum\n ))\n ),\n 'additionalProperties', pgstac.additional_properties()\n )\n FROM g\n );\n ELSE\n RETURN NULL;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\nCREATE OR REPLACE FUNCTION get_queryables(_collection text DEFAULT NULL) RETURNS jsonb AS $$\n SELECT\n CASE\n WHEN _collection IS NULL THEN get_queryables(NULL::text[])\n ELSE get_queryables(ARRAY[_collection])\n END\n ;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_queryables() RETURNS jsonb AS $$\n SELECT get_queryables(NULL::text[]);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION schema_qualify_refs(url text, j jsonb) returns jsonb as $$\n SELECT regexp_replace(j::text, '\"\\$ref\": \"#', concat('\"$ref\": \"', url, '#'), 'g')::jsonb;\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE VIEW stac_extension_queryables AS\nSELECT DISTINCT key as name, schema_qualify_refs(e.url, j.value) as definition FROM stac_extensions e, jsonb_each(e.content->'definitions'->'fields'->'properties') j;\n\n\nCREATE OR REPLACE FUNCTION missing_queryables(_collection text, _tablesample float DEFAULT 5, minrows float DEFAULT 10) RETURNS TABLE(collection text, name text, definition jsonb, property_wrapper text) AS $$\nDECLARE\n q text;\n _partition text;\n explain_json json;\n psize float;\n estrows float;\nBEGIN\n SELECT format('_items_%s', key) INTO _partition FROM collections WHERE id=_collection;\n\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM %I;', _partition)\n INTO explain_json;\n psize := explain_json->0->'Plan'->'Plan Rows';\n estrows := _tablesample * .01 * psize;\n IF estrows < minrows THEN\n _tablesample := least(100,greatest(_tablesample, (estrows / psize) / 100));\n RAISE NOTICE '%', (psize / estrows) / 100;\n END IF;\n RAISE NOTICE 'Using tablesample % to find missing queryables from % % that has ~% rows estrows: %', _tablesample, _collection, _partition, psize, estrows;\n\n q := format(\n $q$\n WITH q AS (\n SELECT * FROM queryables\n WHERE\n collection_ids IS NULL\n OR %L = ANY(collection_ids)\n ), t AS (\n SELECT\n content->'properties' AS properties\n FROM\n %I\n TABLESAMPLE SYSTEM(%L)\n ), p AS (\n SELECT DISTINCT ON (key)\n key,\n value,\n s.definition\n FROM t\n JOIN LATERAL jsonb_each(properties) ON TRUE\n LEFT JOIN q ON (q.name=key)\n LEFT JOIN stac_extension_queryables s ON (s.name=key)\n WHERE q.definition IS NULL\n )\n SELECT\n %L,\n key,\n COALESCE(definition, jsonb_build_object('type',jsonb_typeof(value))) as definition,\n CASE\n WHEN definition->>'type' = 'integer' THEN 'to_int'\n WHEN COALESCE(definition->>'type', jsonb_typeof(value)) = 'number' THEN 'to_float'\n WHEN COALESCE(definition->>'type', jsonb_typeof(value)) = 'array' THEN 'to_text_array'\n ELSE 'to_text'\n END\n FROM p;\n $q$,\n _collection,\n _partition,\n _tablesample,\n _collection\n );\n RETURN QUERY EXECUTE q;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION missing_queryables(_tablesample float DEFAULT 5) RETURNS TABLE(collection_ids text[], name text, definition jsonb, property_wrapper text) AS $$\n SELECT\n array_agg(collection),\n name,\n definition,\n property_wrapper\n FROM\n collections\n JOIN LATERAL\n missing_queryables(id, _tablesample) c\n ON TRUE\n GROUP BY\n 2,3,4\n ORDER BY 2,1\n ;\n$$ LANGUAGE SQL;\nCREATE OR REPLACE FUNCTION parse_dtrange(\n _indate jsonb,\n relative_base timestamptz DEFAULT date_trunc('hour', CURRENT_TIMESTAMP)\n) RETURNS tstzrange AS $$\nDECLARE\n timestrs text[];\n s timestamptz;\n e timestamptz;\nBEGIN\n timestrs :=\n CASE\n WHEN _indate ? 'timestamp' THEN\n ARRAY[_indate->>'timestamp']\n WHEN _indate ? 'interval' THEN\n to_text_array(_indate->'interval')\n WHEN jsonb_typeof(_indate) = 'array' THEN\n to_text_array(_indate)\n ELSE\n regexp_split_to_array(\n _indate->>0,\n '/'\n )\n END;\n RAISE NOTICE 'TIMESTRS %', timestrs;\n IF cardinality(timestrs) = 1 THEN\n IF timestrs[1] ILIKE 'P%' THEN\n RETURN tstzrange(relative_base - upper(timestrs[1])::interval, relative_base, '[)');\n END IF;\n s := timestrs[1]::timestamptz;\n RETURN tstzrange(s, s, '[]');\n END IF;\n\n IF cardinality(timestrs) != 2 THEN\n RAISE EXCEPTION 'Timestamp cannot have more than 2 values';\n END IF;\n\n IF timestrs[1] = '..' OR timestrs[1] = '' THEN\n s := '-infinity'::timestamptz;\n e := timestrs[2]::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] = '..' OR timestrs[2] = '' THEN\n s := timestrs[1]::timestamptz;\n e := 'infinity'::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[1] ILIKE 'P%' AND timestrs[2] NOT ILIKE 'P%' THEN\n e := timestrs[2]::timestamptz;\n s := e - upper(timestrs[1])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] ILIKE 'P%' AND timestrs[1] NOT ILIKE 'P%' THEN\n s := timestrs[1]::timestamptz;\n e := s + upper(timestrs[2])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n s := timestrs[1]::timestamptz;\n e := timestrs[2]::timestamptz;\n\n RETURN tstzrange(s,e,'[)');\n\n RETURN NULL;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT PARALLEL SAFE SET TIME ZONE 'UTC';\n\nCREATE OR REPLACE FUNCTION parse_dtrange(\n _indate text,\n relative_base timestamptz DEFAULT CURRENT_TIMESTAMP\n) RETURNS tstzrange AS $$\n SELECT parse_dtrange(to_jsonb(_indate), relative_base);\n$$ LANGUAGE SQL STABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION temporal_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\n ll text := 'datetime';\n lh text := 'end_datetime';\n rrange tstzrange;\n rl text;\n rh text;\n outq text;\nBEGIN\n rrange := parse_dtrange(args->1);\n RAISE NOTICE 'Constructing temporal query OP: %, ARGS: %, RRANGE: %', op, args, rrange;\n op := lower(op);\n rl := format('%L::timestamptz', lower(rrange));\n rh := format('%L::timestamptz', upper(rrange));\n outq := CASE op\n WHEN 't_before' THEN 'lh < rl'\n WHEN 't_after' THEN 'll > rh'\n WHEN 't_meets' THEN 'lh = rl'\n WHEN 't_metby' THEN 'll = rh'\n WHEN 't_overlaps' THEN 'll < rl AND rl < lh < rh'\n WHEN 't_overlappedby' THEN 'rl < ll < rh AND lh > rh'\n WHEN 't_starts' THEN 'll = rl AND lh < rh'\n WHEN 't_startedby' THEN 'll = rl AND lh > rh'\n WHEN 't_during' THEN 'll > rl AND lh < rh'\n WHEN 't_contains' THEN 'll < rl AND lh > rh'\n WHEN 't_finishes' THEN 'll > rl AND lh = rh'\n WHEN 't_finishedby' THEN 'll < rl AND lh = rh'\n WHEN 't_equals' THEN 'll = rl AND lh = rh'\n WHEN 't_disjoint' THEN 'NOT (ll <= rh AND lh >= rl)'\n WHEN 't_intersects' THEN 'll <= rh AND lh >= rl'\n WHEN 'anyinteracts' THEN 'll <= rh AND lh >= rl'\n END;\n outq := regexp_replace(outq, '\\mll\\M', ll);\n outq := regexp_replace(outq, '\\mlh\\M', lh);\n outq := regexp_replace(outq, '\\mrl\\M', rl);\n outq := regexp_replace(outq, '\\mrh\\M', rh);\n outq := format('(%s)', outq);\n RETURN outq;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\n\n\nCREATE OR REPLACE FUNCTION spatial_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\n geom text;\n j jsonb := args->1;\nBEGIN\n op := lower(op);\n RAISE NOTICE 'Constructing spatial query OP: %, ARGS: %', op, args;\n IF op NOT IN ('s_equals','s_disjoint','s_touches','s_within','s_overlaps','s_crosses','s_intersects','intersects','s_contains') THEN\n RAISE EXCEPTION 'Spatial Operator % Not Supported', op;\n END IF;\n op := regexp_replace(op, '^s_', 'st_');\n IF op = 'intersects' THEN\n op := 'st_intersects';\n END IF;\n -- Convert geometry to WKB string\n IF j ? 'type' AND j ? 'coordinates' THEN\n geom := st_geomfromgeojson(j)::text;\n ELSIF jsonb_typeof(j) = 'array' THEN\n geom := bbox_geom(j)::text;\n END IF;\n\n RETURN format('%s(geometry, %L::geometry)', op, geom);\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION query_to_cql2(q jsonb) RETURNS jsonb AS $$\n-- Translates anything passed in through the deprecated \"query\" into equivalent CQL2\nWITH t AS (\n SELECT key as property, value as ops\n FROM jsonb_each(q)\n), t2 AS (\n SELECT property, (jsonb_each(ops)).*\n FROM t WHERE jsonb_typeof(ops) = 'object'\n UNION ALL\n SELECT property, 'eq', ops\n FROM t WHERE jsonb_typeof(ops) != 'object'\n)\nSELECT\n jsonb_strip_nulls(jsonb_build_object(\n 'op', 'and',\n 'args', jsonb_agg(\n jsonb_build_object(\n 'op', key,\n 'args', jsonb_build_array(\n jsonb_build_object('property',property),\n value\n )\n )\n )\n )\n) as qcql FROM t2\n;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\nCREATE OR REPLACE FUNCTION cql1_to_cql2(j jsonb) RETURNS jsonb AS $$\nDECLARE\n args jsonb;\n ret jsonb;\nBEGIN\n RAISE NOTICE 'CQL1_TO_CQL2: %', j;\n IF j ? 'filter' THEN\n RETURN cql1_to_cql2(j->'filter');\n END IF;\n IF j ? 'property' THEN\n RETURN j;\n END IF;\n IF jsonb_typeof(j) = 'array' THEN\n SELECT jsonb_agg(cql1_to_cql2(el)) INTO args FROM jsonb_array_elements(j) el;\n RETURN args;\n END IF;\n IF jsonb_typeof(j) = 'number' THEN\n RETURN j;\n END IF;\n IF jsonb_typeof(j) = 'string' THEN\n RETURN j;\n END IF;\n\n IF jsonb_typeof(j) = 'object' THEN\n SELECT jsonb_build_object(\n 'op', key,\n 'args', cql1_to_cql2(value)\n ) INTO ret\n FROM jsonb_each(j)\n WHERE j IS NOT NULL;\n RETURN ret;\n END IF;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE STRICT;\n\nCREATE TABLE cql2_ops (\n op text PRIMARY KEY,\n template text,\n types text[]\n);\nINSERT INTO cql2_ops (op, template, types) VALUES\n ('eq', '%s = %s', NULL),\n ('neq', '%s != %s', NULL),\n ('ne', '%s != %s', NULL),\n ('!=', '%s != %s', NULL),\n ('<>', '%s != %s', NULL),\n ('lt', '%s < %s', NULL),\n ('lte', '%s <= %s', NULL),\n ('gt', '%s > %s', NULL),\n ('gte', '%s >= %s', NULL),\n ('le', '%s <= %s', NULL),\n ('ge', '%s >= %s', NULL),\n ('=', '%s = %s', NULL),\n ('<', '%s < %s', NULL),\n ('<=', '%s <= %s', NULL),\n ('>', '%s > %s', NULL),\n ('>=', '%s >= %s', NULL),\n ('like', '%s LIKE %s', NULL),\n ('ilike', '%s ILIKE %s', NULL),\n ('+', '%s + %s', NULL),\n ('-', '%s - %s', NULL),\n ('*', '%s * %s', NULL),\n ('/', '%s / %s', NULL),\n ('not', 'NOT (%s)', NULL),\n ('between', '%s BETWEEN %s AND %s', NULL),\n ('isnull', '%s IS NULL', NULL),\n ('upper', 'upper(%s)', NULL),\n ('lower', 'lower(%s)', NULL),\n ('casei', 'upper(%s)', NULL),\n ('accenti', 'unaccent(%s)', NULL)\nON CONFLICT (op) DO UPDATE\n SET\n template = EXCLUDED.template\n;\n\n\nCREATE OR REPLACE FUNCTION cql2_query(j jsonb, wrapper text DEFAULT NULL) RETURNS text AS $$\n#variable_conflict use_variable\nDECLARE\n args jsonb := j->'args';\n arg jsonb;\n op text := lower(j->>'op');\n cql2op RECORD;\n literal text;\n _wrapper text;\n leftarg text;\n rightarg text;\n prop text;\n extra_props bool := pgstac.additional_properties();\nBEGIN\n IF j IS NULL OR (op IS NOT NULL AND args IS NULL) THEN\n RETURN NULL;\n END IF;\n RAISE NOTICE 'CQL2_QUERY: %', j;\n\n -- check if all properties are represented in the queryables\n IF NOT extra_props THEN\n FOR prop IN\n SELECT DISTINCT p->>0\n FROM jsonb_path_query(j, 'strict $.**.property') p\n WHERE p->>0 NOT IN ('id', 'datetime', 'geometry', 'end_datetime', 'collection')\n LOOP\n IF (queryable(prop)).nulled_wrapper IS NULL THEN\n RAISE EXCEPTION 'Term % is not found in queryables.', prop;\n END IF;\n END LOOP;\n END IF;\n\n IF j ? 'filter' THEN\n RETURN cql2_query(j->'filter');\n END IF;\n\n IF j ? 'upper' THEN\n RETURN cql2_query(jsonb_build_object('op', 'upper', 'args', j->'upper'));\n END IF;\n\n IF j ? 'lower' THEN\n RETURN cql2_query(jsonb_build_object('op', 'lower', 'args', j->'lower'));\n END IF;\n\n -- Temporal Query\n IF op ilike 't_%' or op = 'anyinteracts' THEN\n RETURN temporal_op_query(op, args);\n END IF;\n\n -- If property is a timestamp convert it to text to use with\n -- general operators\n IF j ? 'timestamp' THEN\n RETURN format('%L::timestamptz', to_tstz(j->'timestamp'));\n END IF;\n IF j ? 'interval' THEN\n RAISE EXCEPTION 'Please use temporal operators when using intervals.';\n RETURN NONE;\n END IF;\n\n -- Spatial Query\n IF op ilike 's_%' or op = 'intersects' THEN\n RETURN spatial_op_query(op, args);\n END IF;\n\n IF op IN ('a_equals','a_contains','a_contained_by','a_overlaps') THEN\n IF args->0 ? 'property' THEN\n leftarg := format('to_text_array(%s)', (queryable(args->0->>'property')).path);\n END IF;\n IF args->1 ? 'property' THEN\n rightarg := format('to_text_array(%s)', (queryable(args->1->>'property')).path);\n END IF;\n RETURN FORMAT(\n '%s %s %s',\n COALESCE(leftarg, quote_literal(to_text_array(args->0))),\n CASE op\n WHEN 'a_equals' THEN '='\n WHEN 'a_contains' THEN '@>'\n WHEN 'a_contained_by' THEN '<@'\n WHEN 'a_overlaps' THEN '&&'\n END,\n COALESCE(rightarg, quote_literal(to_text_array(args->1)))\n );\n END IF;\n\n IF op = 'in' THEN\n RAISE NOTICE 'IN : % % %', args, jsonb_build_array(args->0), args->1;\n args := jsonb_build_array(args->0) || (args->1);\n RAISE NOTICE 'IN2 : %', args;\n END IF;\n\n\n\n IF op = 'between' THEN\n args = jsonb_build_array(\n args->0,\n args->1,\n args->2\n );\n END IF;\n\n -- Make sure that args is an array and run cql2_query on\n -- each element of the array\n RAISE NOTICE 'ARGS PRE: %', args;\n IF j ? 'args' THEN\n IF jsonb_typeof(args) != 'array' THEN\n args := jsonb_build_array(args);\n END IF;\n\n IF jsonb_path_exists(args, '$[*] ? (@.property == \"id\" || @.property == \"datetime\" || @.property == \"end_datetime\" || @.property == \"collection\")') THEN\n wrapper := NULL;\n ELSE\n -- if any of the arguments are a property, try to get the property_wrapper\n FOR arg IN SELECT jsonb_path_query(args, '$[*] ? (@.property != null)') LOOP\n RAISE NOTICE 'Arg: %', arg;\n wrapper := (queryable(arg->>'property')).nulled_wrapper;\n RAISE NOTICE 'Property: %, Wrapper: %', arg, wrapper;\n IF wrapper IS NOT NULL THEN\n EXIT;\n END IF;\n END LOOP;\n\n -- if the property was not in queryables, see if any args were numbers\n IF\n wrapper IS NULL\n AND jsonb_path_exists(args, '$[*] ? (@.type()==\"number\")')\n THEN\n wrapper := 'to_float';\n END IF;\n wrapper := coalesce(wrapper, 'to_text');\n END IF;\n\n SELECT jsonb_agg(cql2_query(a, wrapper))\n INTO args\n FROM jsonb_array_elements(args) a;\n END IF;\n RAISE NOTICE 'ARGS: %', args;\n\n IF op IN ('and', 'or') THEN\n RETURN\n format(\n '(%s)',\n array_to_string(to_text_array(args), format(' %s ', upper(op)))\n );\n END IF;\n\n IF op = 'in' THEN\n RAISE NOTICE 'IN -- % %', args->0, to_text(args->0);\n RETURN format(\n '%s IN (%s)',\n to_text(args->0),\n array_to_string((to_text_array(args))[2:], ',')\n );\n END IF;\n\n -- Look up template from cql2_ops\n IF j ? 'op' THEN\n SELECT * INTO cql2op FROM cql2_ops WHERE cql2_ops.op ilike op;\n IF FOUND THEN\n -- If specific index set in queryables for a property cast other arguments to that type\n\n RETURN format(\n cql2op.template,\n VARIADIC (to_text_array(args))\n );\n ELSE\n RAISE EXCEPTION 'Operator % Not Supported.', op;\n END IF;\n END IF;\n\n\n IF wrapper IS NOT NULL THEN\n RAISE NOTICE 'Wrapping % with %', j, wrapper;\n IF j ? 'property' THEN\n RETURN format('%I(%s)', wrapper, (queryable(j->>'property')).path);\n ELSE\n RETURN format('%I(%L)', wrapper, j);\n END IF;\n ELSIF j ? 'property' THEN\n RETURN quote_ident(j->>'property');\n END IF;\n\n RETURN quote_literal(to_text(j));\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION paging_dtrange(\n j jsonb\n) RETURNS tstzrange AS $$\nDECLARE\n op text;\n filter jsonb := j->'filter';\n dtrange tstzrange := tstzrange('-infinity'::timestamptz,'infinity'::timestamptz);\n sdate timestamptz := '-infinity'::timestamptz;\n edate timestamptz := 'infinity'::timestamptz;\n jpitem jsonb;\nBEGIN\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n END IF;\n IF NOT (filter @? '$.**.op ? (@ == \"or\" || @ == \"not\")') THEN\n FOR jpitem IN SELECT j FROM jsonb_path_query(filter,'strict $.** ? (@.args[*].property == \"datetime\")'::jsonpath) j LOOP\n op := lower(jpitem->>'op');\n dtrange := parse_dtrange(jpitem->'args'->1);\n IF op IN ('<=', 'lt', 'lte', '<', 'le', 't_before') THEN\n sdate := greatest(sdate,'-infinity');\n edate := least(edate, upper(dtrange));\n ELSIF op IN ('>=', '>', 'gt', 'gte', 'ge', 't_after') THEN\n edate := least(edate, 'infinity');\n sdate := greatest(sdate, lower(dtrange));\n ELSIF op IN ('=', 'eq') THEN\n edate := least(edate, upper(dtrange));\n sdate := greatest(sdate, lower(dtrange));\n END IF;\n RAISE NOTICE '2 OP: %, ARGS: %, DTRANGE: %, SDATE: %, EDATE: %', op, jpitem->'args'->1, dtrange, sdate, edate;\n END LOOP;\n END IF;\n IF sdate > edate THEN\n RETURN 'empty'::tstzrange;\n END IF;\n RETURN tstzrange(sdate,edate, '[]');\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT SET TIME ZONE 'UTC';\n\nCREATE OR REPLACE FUNCTION paging_collections(\n IN j jsonb\n) RETURNS text[] AS $$\nDECLARE\n filter jsonb := j->'filter';\n jpitem jsonb;\n op text;\n args jsonb;\n arg jsonb;\n collections text[];\nBEGIN\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n END IF;\n IF NOT (filter @? '$.**.op ? (@ == \"or\" || @ == \"not\")') THEN\n FOR jpitem IN SELECT j FROM jsonb_path_query(filter,'strict $.** ? (@.args[*].property == \"collection\")'::jsonpath) j LOOP\n RAISE NOTICE 'JPITEM: %', jpitem;\n op := jpitem->>'op';\n args := jpitem->'args';\n IF op IN ('=', 'eq', 'in') THEN\n FOR arg IN SELECT a FROM jsonb_array_elements(args) a LOOP\n IF jsonb_typeof(arg) IN ('string', 'array') THEN\n RAISE NOTICE 'arg: %, collections: %', arg, collections;\n IF collections IS NULL OR collections = '{}'::text[] THEN\n collections := to_text_array(arg);\n ELSE\n collections := array_intersection(collections, to_text_array(arg));\n END IF;\n END IF;\n END LOOP;\n END IF;\n END LOOP;\n END IF;\n IF collections = '{}'::text[] THEN\n RETURN NULL;\n END IF;\n RETURN collections;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\nCREATE TABLE items (\n id text NOT NULL,\n geometry geometry NOT NULL,\n collection text NOT NULL,\n datetime timestamptz NOT NULL,\n end_datetime timestamptz NOT NULL,\n content JSONB NOT NULL,\n private jsonb\n)\nPARTITION BY LIST (collection)\n;\n\nCREATE INDEX \"datetime_idx\" ON items USING BTREE (datetime DESC, end_datetime ASC);\nCREATE INDEX \"geometry_idx\" ON items USING GIST (geometry);\n\nCREATE STATISTICS datetime_stats (dependencies) on datetime, end_datetime from items;\n\nALTER TABLE items ADD CONSTRAINT items_collections_fk FOREIGN KEY (collection) REFERENCES collections(id) ON DELETE CASCADE DEFERRABLE;\n\nCREATE OR REPLACE FUNCTION partition_after_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p text;\n t timestamptz := clock_timestamp();\nBEGIN\n RAISE NOTICE 'Updating partition stats %', t;\n FOR p IN SELECT DISTINCT partition\n FROM newdata n JOIN partition_sys_meta p\n ON (n.collection=p.collection AND n.datetime <@ p.partition_dtrange)\n LOOP\n PERFORM run_or_queue(format('SELECT update_partition_stats(%L, %L);', p, true));\n END LOOP;\n IF TG_OP IN ('DELETE','UPDATE') THEN\n DELETE FROM format_item_cache c USING newdata n WHERE c.collection = n.collection AND c.id = n.id;\n END IF;\n RAISE NOTICE 't: % %', t, clock_timestamp() - t;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE TRIGGER items_after_insert_trigger\nAFTER INSERT ON items\nREFERENCING NEW TABLE AS newdata\nFOR EACH STATEMENT\nEXECUTE FUNCTION partition_after_triggerfunc();\n\nCREATE TRIGGER items_after_update_trigger\nAFTER DELETE ON items\nREFERENCING OLD TABLE AS newdata\nFOR EACH STATEMENT\nEXECUTE FUNCTION partition_after_triggerfunc();\n\nCREATE TRIGGER items_after_delete_trigger\nAFTER UPDATE ON items\nREFERENCING NEW TABLE AS newdata\nFOR EACH STATEMENT\nEXECUTE FUNCTION partition_after_triggerfunc();\n\n\nCREATE OR REPLACE FUNCTION content_slim(_item jsonb) RETURNS jsonb AS $$\n SELECT strip_jsonb(_item - '{id,geometry,collection,type}'::text[], collection_base_item(_item->>'collection')) - '{id,geometry,collection,type}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_dehydrate(content jsonb) RETURNS items AS $$\n SELECT\n content->>'id' as id,\n stac_geom(content) as geometry,\n content->>'collection' as collection,\n stac_datetime(content) as datetime,\n stac_end_datetime(content) as end_datetime,\n content_slim(content) as content,\n null::jsonb as private\n ;\n$$ LANGUAGE SQL STABLE;\n\nCREATE OR REPLACE FUNCTION include_field(f text, fields jsonb DEFAULT '{}'::jsonb) RETURNS boolean AS $$\nDECLARE\n includes jsonb := fields->'include';\n excludes jsonb := fields->'exclude';\nBEGIN\n IF f IS NULL THEN\n RETURN NULL;\n END IF;\n\n\n IF\n jsonb_typeof(excludes) = 'array'\n AND jsonb_array_length(excludes)>0\n AND excludes ? f\n THEN\n RETURN FALSE;\n END IF;\n\n IF\n (\n jsonb_typeof(includes) = 'array'\n AND jsonb_array_length(includes) > 0\n AND includes ? f\n ) OR\n (\n includes IS NULL\n OR jsonb_typeof(includes) = 'null'\n OR jsonb_array_length(includes) = 0\n )\n THEN\n RETURN TRUE;\n END IF;\n\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nDROP FUNCTION IF EXISTS content_hydrate(jsonb, jsonb, jsonb);\nCREATE OR REPLACE FUNCTION content_hydrate(\n _item jsonb,\n _base_item jsonb,\n fields jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\n SELECT merge_jsonb(\n jsonb_fields(_item, fields),\n jsonb_fields(_base_item, fields)\n );\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION content_hydrate(_item items, _collection collections, fields jsonb DEFAULT '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\n content jsonb;\n base_item jsonb := _collection.base_item;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry, 20)::jsonb;\n END IF;\n output := content_hydrate(\n jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'collection', _item.collection,\n 'type', 'Feature'\n ) || _item.content,\n _collection.base_item,\n fields\n );\n\n RETURN output;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_nonhydrated(\n _item items,\n fields jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry, 20)::jsonb;\n END IF;\n output := jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'collection', _item.collection,\n 'type', 'Feature'\n ) || _item.content;\n RETURN output;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_hydrate(_item items, fields jsonb DEFAULT '{}'::jsonb) RETURNS jsonb AS $$\n SELECT content_hydrate(\n _item,\n (SELECT c FROM collections c WHERE id=_item.collection LIMIT 1),\n fields\n );\n$$ LANGUAGE SQL STABLE;\n\n\nCREATE UNLOGGED TABLE items_staging (\n content JSONB NOT NULL\n);\nCREATE UNLOGGED TABLE items_staging_ignore (\n content JSONB NOT NULL\n);\nCREATE UNLOGGED TABLE items_staging_upsert (\n content JSONB NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION items_staging_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p record;\n _partitions text[];\n part text;\n ts timestamptz := clock_timestamp();\n nrows int;\nBEGIN\n RAISE NOTICE 'Creating Partitions. %', clock_timestamp() - ts;\n\n FOR part IN WITH t AS (\n SELECT\n n.content->>'collection' as collection,\n stac_daterange(n.content->'properties') as dtr,\n partition_trunc\n FROM newdata n JOIN collections ON (n.content->>'collection'=collections.id)\n ), p AS (\n SELECT\n collection,\n COALESCE(date_trunc(partition_trunc::text, lower(dtr)),'-infinity') as d,\n tstzrange(min(lower(dtr)),max(lower(dtr)),'[]') as dtrange,\n tstzrange(min(upper(dtr)),max(upper(dtr)),'[]') as edtrange\n FROM t\n GROUP BY 1,2\n ) SELECT check_partition(collection, dtrange, edtrange) FROM p LOOP\n RAISE NOTICE 'Partition %', part;\n END LOOP;\n\n RAISE NOTICE 'Creating temp table with data to be added. %', clock_timestamp() - ts;\n DROP TABLE IF EXISTS tmpdata;\n CREATE TEMP TABLE tmpdata ON COMMIT DROP AS\n SELECT\n (content_dehydrate(content)).*\n FROM newdata;\n GET DIAGNOSTICS nrows = ROW_COUNT;\n RAISE NOTICE 'Added % rows to tmpdata. %', nrows, clock_timestamp() - ts;\n\n RAISE NOTICE 'Doing the insert. %', clock_timestamp() - ts;\n IF TG_TABLE_NAME = 'items_staging' THEN\n INSERT INTO items\n SELECT * FROM tmpdata;\n GET DIAGNOSTICS nrows = ROW_COUNT;\n RAISE NOTICE 'Inserted % rows to items. %', nrows, clock_timestamp() - ts;\n ELSIF TG_TABLE_NAME = 'items_staging_ignore' THEN\n INSERT INTO items\n SELECT * FROM tmpdata\n ON CONFLICT DO NOTHING;\n GET DIAGNOSTICS nrows = ROW_COUNT;\n RAISE NOTICE 'Inserted % rows to items. %', nrows, clock_timestamp() - ts;\n ELSIF TG_TABLE_NAME = 'items_staging_upsert' THEN\n DELETE FROM items i USING tmpdata s\n WHERE\n i.id = s.id\n AND i.collection = s.collection\n AND i IS DISTINCT FROM s\n ;\n GET DIAGNOSTICS nrows = ROW_COUNT;\n RAISE NOTICE 'Deleted % rows from items. %', nrows, clock_timestamp() - ts;\n INSERT INTO items AS t\n SELECT * FROM tmpdata\n ON CONFLICT DO NOTHING;\n GET DIAGNOSTICS nrows = ROW_COUNT;\n RAISE NOTICE 'Inserted % rows to items. %', nrows, clock_timestamp() - ts;\n END IF;\n\n RAISE NOTICE 'Deleting data from staging table. %', clock_timestamp() - ts;\n DELETE FROM items_staging;\n RAISE NOTICE 'Done. %', clock_timestamp() - ts;\n\n RETURN NULL;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER items_staging_insert_trigger AFTER INSERT ON items_staging REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\nCREATE TRIGGER items_staging_insert_ignore_trigger AFTER INSERT ON items_staging_ignore REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\nCREATE TRIGGER items_staging_insert_upsert_trigger AFTER INSERT ON items_staging_upsert REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\n\nCREATE OR REPLACE FUNCTION item_by_id(_id text, _collection text DEFAULT NULL) RETURNS items AS\n$$\nDECLARE\n i items%ROWTYPE;\nBEGIN\n SELECT * INTO i FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection) LIMIT 1;\n RETURN i;\nEND;\n$$ LANGUAGE PLPGSQL STABLE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION get_item(_id text, _collection text DEFAULT NULL) RETURNS jsonb AS $$\n SELECT content_hydrate(items) FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection);\n$$ LANGUAGE SQL STABLE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION delete_item(_id text, _collection text DEFAULT NULL) RETURNS VOID AS $$\nDECLARE\nout items%ROWTYPE;\nBEGIN\n DELETE FROM items WHERE id = _id AND (_collection IS NULL OR collection=_collection) RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n--/*\nCREATE OR REPLACE FUNCTION create_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION update_item(content jsonb) RETURNS VOID AS $$\nDECLARE\n old items %ROWTYPE;\n out items%ROWTYPE;\nBEGIN\n PERFORM delete_item(content->>'id', content->>'collection');\n PERFORM create_item(content);\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION create_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION collection_bbox(id text) RETURNS jsonb AS $$\n SELECT (replace(replace(replace(st_extent(geometry)::text,'BOX(','[['),')',']]'),' ',','))::jsonb\n FROM items WHERE collection=$1;\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION collection_temporal_extent(id text) RETURNS jsonb AS $$\n SELECT to_jsonb(array[array[min(datetime), max(datetime)]])\n FROM items WHERE collection=$1;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection_extents() RETURNS VOID AS $$\nUPDATE collections\n SET content = jsonb_set_lax(\n content,\n '{extent}'::text[],\n collection_extent(id, FALSE),\n true,\n 'use_json_null'\n )\n;\n$$ LANGUAGE SQL;\nCREATE TABLE partition_stats (\n partition text PRIMARY KEY,\n dtrange tstzrange,\n edtrange tstzrange,\n spatial geometry,\n last_updated timestamptz,\n keys text[]\n) WITH (FILLFACTOR=90);\n\nCREATE INDEX partitions_range_idx ON partition_stats USING GIST(dtrange);\n\n\nCREATE OR REPLACE FUNCTION constraint_tstzrange(expr text) RETURNS tstzrange AS $$\n WITH t AS (\n SELECT regexp_matches(\n expr,\n E'\\\\(''\\([0-9 :+-]*\\)''\\\\).*\\\\(''\\([0-9 :+-]*\\)''\\\\)'\n ) AS m\n ) SELECT tstzrange(m[1]::timestamptz, m[2]::timestamptz) FROM t\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE STRICT;\n\nCREATE OR REPLACE FUNCTION get_tstz_constraint(reloid oid, colname text) RETURNS tstzrange AS $$\nDECLARE\n expr text := NULL;\n m text[];\n ts_lower timestamptz := NULL;\n ts_upper timestamptz := NULL;\n lower_inclusive text := '[';\n upper_inclusive text := ']';\n ts timestamptz;\nBEGIN\n SELECT INTO expr\n string_agg(def, ' AND ')\n FROM pg_constraint JOIN LATERAL pg_get_constraintdef(oid) AS def ON TRUE\n WHERE\n conrelid = reloid\n AND contype = 'c'\n AND def LIKE '%' || colname || '%'\n ;\n\n IF expr IS NULL THEN\n RETURN NULL;\n END IF;\n\n RAISE DEBUG 'Constraint expression for % on %: %', colname, reloid::regclass, expr;\n -- collect all constraints for the specified column\n FOR m IN SELECT regexp_matches(expr, '[ (]' || colname || $expr$\\s*([<>=]{1,2})\\s*'([0-9 :.+\\-]+)'$expr$, 'g') LOOP\n ts := m[2]::timestamptz;\n IF m[1] IN ('>', '>=')\n THEN\n IF ts_lower IS NULL OR ts > ts_lower OR (ts = ts_lower AND m[1] = '>') THEN\n ts_lower := ts;\n lower_inclusive := CASE WHEN m[1] = '>' THEN '(' ELSE '[' END;\n END IF;\n ELSIF m[1] IN ('<', '<=')\n THEN\n IF ts_upper IS NULL OR ts < ts_upper OR (ts = ts_upper AND m[1] = '<') THEN\n ts_upper := ts;\n upper_inclusive := CASE WHEN m[1] = '<' THEN ')' ELSE ']' END;\n END IF;\n END IF;\n END LOOP;\n RAISE DEBUG 'Constraint % for %: % %', colname, reloid::regclass, ts_lower, ts_upper;\n RETURN tstzrange(ts_lower, ts_upper, lower_inclusive || upper_inclusive);\nEND;\n$$ LANGUAGE plpgsql STRICT STABLE;\n\nCREATE OR REPLACE FUNCTION get_partition_name(relid regclass) RETURNS text AS $$\n SELECT (parse_ident(relid::text))[cardinality(parse_ident(relid::text))];\n$$ LANGUAGE SQL STABLE STRICT;\n\nCREATE OR REPLACE VIEW partition_sys_meta AS\nSELECT\n partition,\n replace(\n replace(\n CASE WHEN level = 1 THEN partition_expr ELSE parent_partition_expr END,\n 'FOR VALUES IN (''',\n ''\n ),\n ''')',\n ''\n ) AS collection,\n level,\n c.reltuples,\n c.relhastriggers,\n partition_dtrange,\n COALESCE(\n get_tstz_constraint(c.oid, 'datetime'),\n partition_dtrange,\n inf_range\n ) as constraint_dtrange,\n COALESCE(\n get_tstz_constraint(c.oid, 'end_datetime'),\n inf_range\n ) as constraint_edtrange\nFROM\n pg_partition_tree('items')\n JOIN pg_class c ON (relid::regclass = c.oid)\n JOIN pg_class parent ON (parentrelid::regclass = parent.oid AND isleaf)\n LEFT JOIN pg_constraint edt ON (conrelid=c.oid AND contype='c')\n JOIN LATERAL get_partition_name(relid) AS partition ON TRUE\n JOIN LATERAL pg_get_expr(c.relpartbound, c.oid) as partition_expr ON TRUE\n JOIN LATERAL pg_get_expr(parent.relpartbound, parent.oid) as parent_partition_expr ON TRUE\n JOIN LATERAL tstzrange('-infinity', 'infinity','[]') as inf_range ON TRUE\n JOIN LATERAL COALESCE(constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), inf_range) as partition_dtrange ON TRUE\n JOIN LATERAL get_tstz_constraint(c.oid, 'datetime') as datetime_constraint ON TRUE\n JOIN LATERAL get_tstz_constraint(c.oid, 'end_datetime') as end_datetime_constraint ON TRUE\nWHERE isleaf\n;\n\nCREATE OR REPLACE VIEW partitions_view AS\nSELECT\n (parse_ident(relid::text))[cardinality(parse_ident(relid::text))] as partition,\n replace(\n replace(\n CASE WHEN level = 1 THEN partition_expr ELSE parent_partition_expr END,\n 'FOR VALUES IN (''',\n ''\n ),\n ''')',\n ''\n ) AS collection,\n level,\n c.reltuples,\n c.relhastriggers,\n partition_dtrange,\n COALESCE(\n get_tstz_constraint(c.oid, 'datetime'),\n partition_dtrange,\n inf_range\n ) as constraint_dtrange,\n COALESCE(\n get_tstz_constraint(c.oid, 'end_datetime'),\n inf_range\n ) as constraint_edtrange,\n dtrange,\n edtrange,\n spatial,\n last_updated\nFROM\n pg_partition_tree('items')\n JOIN pg_class c ON (relid::regclass = c.oid)\n JOIN pg_class parent ON (parentrelid::regclass = parent.oid AND isleaf)\n LEFT JOIN pg_constraint edt ON (conrelid=c.oid AND contype='c')\n JOIN LATERAL get_partition_name(relid) AS partition ON TRUE\n JOIN LATERAL pg_get_expr(c.relpartbound, c.oid) as partition_expr ON TRUE\n JOIN LATERAL pg_get_expr(parent.relpartbound, parent.oid) as parent_partition_expr ON TRUE\n JOIN LATERAL tstzrange('-infinity', 'infinity','[]') as inf_range ON TRUE\n JOIN LATERAL COALESCE(constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), inf_range) as partition_dtrange ON TRUE\n JOIN LATERAL get_tstz_constraint(c.oid, 'datetime') as datetime_constraint ON TRUE\n JOIN LATERAL get_tstz_constraint(c.oid, 'end_datetime') as end_datetime_constraint ON TRUE\n LEFT JOIN pgstac.partition_stats USING (partition)\nWHERE isleaf\n;\n\nCREATE MATERIALIZED VIEW partitions AS\nSELECT * FROM partitions_view;\nCREATE UNIQUE INDEX ON partitions (partition);\n\nCREATE MATERIALIZED VIEW partition_steps AS\nSELECT\n partition as name,\n date_trunc('month',lower(partition_dtrange)) as sdate,\n date_trunc('month', upper(partition_dtrange)) + '1 month'::interval as edate\n FROM partitions_view WHERE partition_dtrange IS NOT NULL AND partition_dtrange != 'empty'::tstzrange\n ORDER BY dtrange ASC\n;\n\n\nCREATE OR REPLACE FUNCTION update_partition_stats_q(_partition text, istrigger boolean default false) RETURNS VOID AS $$\nDECLARE\nBEGIN\n PERFORM run_or_queue(\n format('SELECT update_partition_stats(%L, %L);', _partition, istrigger)\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION update_partition_stats(_partition text, istrigger boolean default false) RETURNS VOID AS $$\nDECLARE\n dtrange tstzrange;\n edtrange tstzrange;\n cdtrange tstzrange;\n cedtrange tstzrange;\n extent geometry;\n collection text;\nBEGIN\n RAISE NOTICE 'Updating stats for %.', _partition;\n EXECUTE format(\n $q$\n SELECT\n tstzrange(min(datetime), max(datetime),'[]'),\n tstzrange(min(end_datetime), max(end_datetime), '[]')\n FROM %I\n $q$,\n _partition\n ) INTO dtrange, edtrange;\n EXECUTE format('ANALYZE %I;', _partition);\n extent := st_estimatedextent('pgstac', _partition, 'geometry');\n RAISE DEBUG 'Estimated Extent: %', extent;\n INSERT INTO partition_stats (partition, dtrange, edtrange, spatial, last_updated)\n SELECT _partition, dtrange, edtrange, extent, now()\n ON CONFLICT (partition) DO\n UPDATE SET\n dtrange=EXCLUDED.dtrange,\n edtrange=EXCLUDED.edtrange,\n spatial=EXCLUDED.spatial,\n last_updated=EXCLUDED.last_updated\n ;\n\n SELECT\n constraint_dtrange, constraint_edtrange, pv.collection\n INTO cdtrange, cedtrange, collection\n FROM partitions_view pv WHERE partition = _partition;\n\n RAISE NOTICE 'Checking if we need to modify constraints...';\n RAISE NOTICE 'cdtrange: % dtrange: % cedtrange: % edtrange: %',cdtrange, dtrange, cedtrange, edtrange;\n IF\n (cdtrange IS DISTINCT FROM dtrange OR edtrange IS DISTINCT FROM cedtrange)\n AND NOT istrigger\n THEN\n RAISE NOTICE 'Modifying Constraints';\n RAISE NOTICE 'Existing % %', cdtrange, cedtrange;\n RAISE NOTICE 'New % %', dtrange, edtrange;\n PERFORM drop_table_constraints(_partition);\n PERFORM create_table_constraints(_partition, dtrange, edtrange);\n END IF;\n REFRESH MATERIALIZED VIEW partitions;\n REFRESH MATERIALIZED VIEW partition_steps;\n RAISE NOTICE 'Checking if we need to update collection extents.';\n IF get_setting_bool('update_collection_extent') THEN\n RAISE NOTICE 'updating collection extent for %', collection;\n PERFORM run_or_queue(format($q$\n UPDATE collections\n SET content = jsonb_set_lax(\n content,\n '{extent}'::text[],\n collection_extent(%L, FALSE),\n true,\n 'use_json_null'\n ) WHERE id=%L\n ;\n $q$, collection, collection));\n ELSE\n RAISE NOTICE 'Not updating collection extent for %', collection;\n END IF;\n\nEND;\n$$ LANGUAGE PLPGSQL STRICT SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION partition_name( IN collection text, IN dt timestamptz, OUT partition_name text, OUT partition_range tstzrange) AS $$\nDECLARE\n c RECORD;\n parent_name text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n parent_name := format('_items_%s', c.key);\n\n\n IF c.partition_trunc = 'year' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYY'));\n ELSIF c.partition_trunc = 'month' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYYMM'));\n ELSE\n partition_name := parent_name;\n partition_range := tstzrange('-infinity'::timestamptz, 'infinity'::timestamptz, '[]');\n END IF;\n IF partition_range IS NULL THEN\n partition_range := tstzrange(\n date_trunc(c.partition_trunc::text, dt),\n date_trunc(c.partition_trunc::text, dt) + concat('1 ', c.partition_trunc)::interval\n );\n END IF;\n RETURN;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION drop_table_constraints(t text) RETURNS text AS $$\nDECLARE\n q text;\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM partitions_view WHERE partition=t) THEN\n RETURN NULL;\n END IF;\n FOR q IN SELECT FORMAT(\n $q$\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n $q$,\n t,\n conname\n ) FROM pg_constraint\n WHERE conrelid=t::regclass::oid AND contype='c'\n LOOP\n EXECUTE q;\n END LOOP;\n RETURN t;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE OR REPLACE FUNCTION create_table_constraints(t text, _dtrange tstzrange, _edtrange tstzrange) RETURNS text AS $$\nDECLARE\n q text;\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM partitions_view WHERE partition=t) THEN\n RETURN NULL;\n END IF;\n RAISE NOTICE 'Creating Table Constraints for % % %', t, _dtrange, _edtrange;\n IF _dtrange = 'empty' AND _edtrange = 'empty' THEN\n q :=format(\n $q$\n DO $block$\n BEGIN\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n ALTER TABLE %I\n ADD CONSTRAINT %I\n CHECK (((datetime IS NULL) AND (end_datetime IS NULL))) NOT VALID\n ;\n ALTER TABLE %I\n VALIDATE CONSTRAINT %I\n ;\n\n\n\n EXCEPTION WHEN others THEN\n RAISE WARNING '%%, Issue Altering Constraints. Please run update_partition_stats(%I)', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n $block$;\n $q$,\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t),\n t\n );\n ELSE\n q :=format(\n $q$\n DO $block$\n BEGIN\n\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n ALTER TABLE %I\n ADD CONSTRAINT %I\n CHECK (\n (datetime >= %L)\n AND (datetime <= %L)\n AND (end_datetime >= %L)\n AND (end_datetime <= %L)\n ) NOT VALID\n ;\n ALTER TABLE %I\n VALIDATE CONSTRAINT %I\n ;\n\n\n\n EXCEPTION WHEN others THEN\n RAISE WARNING '%%, Issue Altering Constraints. Please run update_partition_stats(%I)', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n $block$;\n $q$,\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t),\n lower(_dtrange),\n upper(_dtrange),\n lower(_edtrange),\n upper(_edtrange),\n t,\n format('%s_dt', t),\n t\n );\n END IF;\n PERFORM run_or_queue(q);\n RETURN t;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION check_partition(\n _collection text,\n _dtrange tstzrange,\n _edtrange tstzrange\n) RETURNS text AS $$\nDECLARE\n c RECORD;\n pm RECORD;\n _partition_name text;\n _partition_dtrange tstzrange;\n _constraint_dtrange tstzrange;\n _constraint_edtrange tstzrange;\n q text;\n deferrable_q text;\n err_context text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=_collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', _collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n\n IF c.partition_trunc IS NOT NULL THEN\n _partition_dtrange := tstzrange(\n date_trunc(c.partition_trunc, lower(_dtrange)),\n date_trunc(c.partition_trunc, lower(_dtrange)) + (concat('1 ', c.partition_trunc))::interval,\n '[)'\n );\n ELSE\n _partition_dtrange := '[-infinity, infinity]'::tstzrange;\n END IF;\n\n IF NOT _partition_dtrange @> _dtrange THEN\n RAISE EXCEPTION 'dtrange % is greater than the partition size % for collection %', _dtrange, c.partition_trunc, _collection;\n END IF;\n\n\n IF c.partition_trunc = 'year' THEN\n _partition_name := format('_items_%s_%s', c.key, to_char(lower(_partition_dtrange),'YYYY'));\n ELSIF c.partition_trunc = 'month' THEN\n _partition_name := format('_items_%s_%s', c.key, to_char(lower(_partition_dtrange),'YYYYMM'));\n ELSE\n _partition_name := format('_items_%s', c.key);\n END IF;\n\n SELECT * INTO pm FROM partition_sys_meta WHERE collection=_collection AND partition_dtrange @> _dtrange;\n IF FOUND THEN\n RAISE NOTICE '% % %', _edtrange, _dtrange, pm;\n _constraint_edtrange :=\n tstzrange(\n least(\n lower(_edtrange),\n nullif(lower(pm.constraint_edtrange), '-infinity')\n ),\n greatest(\n upper(_edtrange),\n nullif(upper(pm.constraint_edtrange), 'infinity')\n ),\n '[]'\n );\n _constraint_dtrange :=\n tstzrange(\n least(\n lower(_dtrange),\n nullif(lower(pm.constraint_dtrange), '-infinity')\n ),\n greatest(\n upper(_dtrange),\n nullif(upper(pm.constraint_dtrange), 'infinity')\n ),\n '[]'\n );\n\n IF pm.constraint_edtrange @> _edtrange AND pm.constraint_dtrange @> _dtrange THEN\n RETURN pm.partition;\n ELSE\n PERFORM drop_table_constraints(_partition_name);\n END IF;\n ELSE\n _constraint_edtrange := _edtrange;\n _constraint_dtrange := _dtrange;\n END IF;\n RAISE NOTICE 'EXISTING CONSTRAINTS % %, NEW % %', pm.constraint_dtrange, pm.constraint_edtrange, _constraint_dtrange, _constraint_edtrange;\n RAISE NOTICE 'Creating partition % %', _partition_name, _partition_dtrange;\n IF c.partition_trunc IS NULL THEN\n q := format(\n $q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n GRANT ALL ON %I to pgstac_ingest;\n $q$,\n _partition_name,\n _collection,\n concat(_partition_name,'_pk'),\n _partition_name,\n _partition_name\n );\n ELSE\n q := format(\n $q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L) PARTITION BY RANGE (datetime);\n CREATE TABLE IF NOT EXISTS %I partition OF %I FOR VALUES FROM (%L) TO (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n GRANT ALL ON %I TO pgstac_ingest;\n $q$,\n format('_items_%s', c.key),\n _collection,\n _partition_name,\n format('_items_%s', c.key),\n lower(_partition_dtrange),\n upper(_partition_dtrange),\n format('%s_pk', _partition_name),\n _partition_name,\n _partition_name\n );\n END IF;\n\n BEGIN\n EXECUTE q;\n EXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', _partition_name;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\n END;\n PERFORM maintain_partitions(_partition_name);\n PERFORM update_partition_stats_q(_partition_name, true);\n REFRESH MATERIALIZED VIEW partitions;\n REFRESH MATERIALIZED VIEW partition_steps;\n RETURN _partition_name;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION repartition(_collection text, _partition_trunc text, triggered boolean DEFAULT FALSE) RETURNS text AS $$\nDECLARE\n c RECORD;\n q text;\n from_trunc text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=_collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', _collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n IF triggered THEN\n RAISE NOTICE 'Converting % to % partitioning via Trigger', _collection, _partition_trunc;\n ELSE\n RAISE NOTICE 'Converting % from using % to % partitioning', _collection, c.partition_trunc, _partition_trunc;\n IF c.partition_trunc IS NOT DISTINCT FROM _partition_trunc THEN\n RAISE NOTICE 'Collection % already set to use partition by %', _collection, _partition_trunc;\n RETURN _collection;\n END IF;\n END IF;\n\n IF EXISTS (SELECT 1 FROM partitions_view WHERE collection=_collection LIMIT 1) THEN\n EXECUTE format(\n $q$\n CREATE TEMP TABLE changepartitionstaging ON COMMIT DROP AS SELECT * FROM %I;\n DROP TABLE IF EXISTS %I CASCADE;\n WITH p AS (\n SELECT\n collection,\n CASE\n WHEN %L IS NULL THEN '-infinity'::timestamptz\n ELSE date_trunc(%L, datetime)\n END as d,\n tstzrange(min(datetime),max(datetime),'[]') as dtrange,\n tstzrange(min(end_datetime),max(end_datetime),'[]') as edtrange\n FROM changepartitionstaging\n GROUP BY 1,2\n ) SELECT check_partition(collection, dtrange, edtrange) FROM p;\n INSERT INTO items SELECT * FROM changepartitionstaging;\n DROP TABLE changepartitionstaging;\n $q$,\n concat('_items_', c.key),\n concat('_items_', c.key),\n c.partition_trunc,\n c.partition_trunc\n );\n END IF;\n RETURN _collection;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE OR REPLACE FUNCTION collections_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n q text;\n partition_name text := format('_items_%s', NEW.key);\n partition_exists boolean := false;\n partition_empty boolean := true;\n err_context text;\n loadtemp boolean := FALSE;\nBEGIN\n RAISE NOTICE 'Collection Trigger. % %', NEW.id, NEW.key;\n IF TG_OP = 'UPDATE' AND NEW.partition_trunc IS DISTINCT FROM OLD.partition_trunc THEN\n PERFORM repartition(NEW.id, NEW.partition_trunc, TRUE);\n END IF;\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER collections_trigger AFTER\nINSERT\nOR\nUPDATE ON collections\nFOR EACH ROW EXECUTE FUNCTION collections_trigger_func();\n\nCREATE OR REPLACE FUNCTION chunker(\n IN _where text,\n OUT s timestamptz,\n OUT e timestamptz\n) RETURNS SETOF RECORD AS $$\nDECLARE\n explain jsonb;\nBEGIN\n IF _where IS NULL THEN\n _where := ' TRUE ';\n END IF;\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s;', _where)\n INTO explain;\n RAISE DEBUG 'EXPLAIN: %', explain;\n\n RETURN QUERY\n WITH t AS (\n SELECT j->>0 as p FROM\n jsonb_path_query(\n explain,\n 'strict $.**.\"Relation Name\" ? (@ != null)'\n ) j\n ),\n parts AS (\n SELECT sdate, edate FROM t JOIN partition_steps ON (t.p = name)\n ),\n times AS (\n SELECT sdate FROM parts\n UNION\n SELECT edate FROM parts\n ),\n uniq AS (\n SELECT DISTINCT sdate FROM times ORDER BY sdate\n ),\n last AS (\n SELECT sdate, lead(sdate, 1) over () as edate FROM uniq\n )\n SELECT sdate, edate FROM last WHERE edate IS NOT NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION partition_queries(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN partitions text[] DEFAULT NULL\n) RETURNS SETOF text AS $$\nDECLARE\n query text;\n sdate timestamptz;\n edate timestamptz;\nBEGIN\nIF _where IS NULL OR trim(_where) = '' THEN\n _where = ' TRUE ';\nEND IF;\nRAISE NOTICE 'Getting chunks for % %', _where, _orderby;\nIF _orderby ILIKE 'datetime d%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 DESC LOOP\n RETURN NEXT format($q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n $q$,\n sdate,\n edate,\n _where,\n _orderby\n );\n END LOOP;\nELSIF _orderby ILIKE 'datetime a%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 ASC LOOP\n RETURN NEXT format($q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n $q$,\n sdate,\n edate,\n _where,\n _orderby\n );\n END LOOP;\nELSE\n query := format($q$\n SELECT * FROM items\n WHERE %s\n ORDER BY %s\n $q$, _where, _orderby\n );\n\n RETURN NEXT query;\n RETURN;\nEND IF;\n\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION partition_query_view(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN _limit int DEFAULT 10\n) RETURNS text AS $$\n WITH p AS (\n SELECT * FROM partition_queries(_where, _orderby) p\n )\n SELECT\n CASE WHEN EXISTS (SELECT 1 FROM p) THEN\n (SELECT format($q$\n SELECT * FROM (\n %s\n ) total LIMIT %s\n $q$,\n string_agg(\n format($q$ SELECT * FROM ( %s ) AS sub $q$, p),\n '\n UNION ALL\n '\n ),\n _limit\n ))\n ELSE NULL\n END FROM p;\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION q_to_tsquery (jinput jsonb)\n RETURNS tsquery\n AS $$\nDECLARE\n input text;\n processed_text text;\n temp_text text;\n quote_array text[];\n placeholder text := '@QUOTE@';\nBEGIN\n IF jsonb_typeof(jinput) = 'string' THEN\n input := jinput->>0;\n ELSIF jsonb_typeof(jinput) = 'array' THEN\n input := array_to_string(\n array(select jsonb_array_elements_text(jinput)),\n ' OR '\n );\n ELSE\n RAISE EXCEPTION 'Input must be a string or an array of strings.';\n END IF;\n -- Extract all quoted phrases and store in array\n quote_array := regexp_matches(input, '\"[^\"]*\"', 'g');\n\n -- Replace each quoted part with a unique placeholder if there are any quoted phrases\n IF array_length(quote_array, 1) IS NOT NULL THEN\n processed_text := input;\n FOR i IN array_lower(quote_array, 1) .. array_upper(quote_array, 1) LOOP\n processed_text := replace(processed_text, quote_array[i], placeholder || i || placeholder);\n END LOOP;\n ELSE\n processed_text := input;\n END IF;\n\n -- Replace non-quoted text using regular expressions\n\n -- , -> |\n processed_text := regexp_replace(processed_text, ',(?=(?:[^\"]*\"[^\"]*\")*[^\"]*$)', ' | ', 'g');\n\n -- and -> &\n processed_text := regexp_replace(processed_text, '\\s+AND\\s+', ' & ', 'gi');\n\n -- or -> |\n processed_text := regexp_replace(processed_text, '\\s+OR\\s+', ' | ', 'gi');\n\n -- + ->\n processed_text := regexp_replace(processed_text, '^\\s*\\+([a-zA-Z0-9_]+)', '\\1', 'g'); -- +term at start\n processed_text := regexp_replace(processed_text, '\\s*\\+([a-zA-Z0-9_]+)', ' & \\1', 'g'); -- +term elsewhere\n\n -- - -> !\n processed_text := regexp_replace(processed_text, '^\\s*\\-([a-zA-Z0-9_]+)', '! \\1', 'g'); -- -term at start\n processed_text := regexp_replace(processed_text, '\\s*\\-([a-zA-Z0-9_]+)', ' & ! \\1', 'g'); -- -term elsewhere\n\n -- terms separated with spaces are assumed to represent adjacent terms. loop through these\n -- occurrences and replace them with the adjacency operator (<->)\n LOOP\n temp_text := regexp_replace(processed_text, '([a-zA-Z0-9_]+)\\s+([a-zA-Z0-9_]+)(?!\\s*[&|<>])', '\\1 <-> \\2', 'g');\n IF temp_text = processed_text THEN\n EXIT; -- No more replacements were made\n END IF;\n processed_text := temp_text;\n END LOOP;\n\n\n -- Replace placeholders back with quoted phrases if there were any\n IF array_length(quote_array, 1) IS NOT NULL THEN\n FOR i IN array_lower(quote_array, 1) .. array_upper(quote_array, 1) LOOP\n processed_text := replace(processed_text, placeholder || i || placeholder, '''' || substring(quote_array[i] from 2 for length(quote_array[i]) - 2) || '''');\n END LOOP;\n END IF;\n\n -- Print processed_text to the console for debugging purposes\n RAISE NOTICE 'processed_text: %', processed_text;\n\n RETURN to_tsquery('english', processed_text);\nEND;\n$$\nLANGUAGE plpgsql;\n\n\nCREATE OR REPLACE FUNCTION stac_search_to_where(j jsonb) RETURNS text AS $$\nDECLARE\n where_segments text[];\n _where text;\n dtrange tstzrange;\n collections text[];\n geom geometry;\n sdate timestamptz;\n edate timestamptz;\n filterlang text;\n filter jsonb := j->'filter';\n ft_query tsquery;\nBEGIN\n IF j ? 'ids' THEN\n where_segments := where_segments || format('id = ANY (%L) ', to_text_array(j->'ids'));\n END IF;\n\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n where_segments := where_segments || format('collection = ANY (%L) ', collections);\n END IF;\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n\n where_segments := where_segments || format(' datetime <= %L::timestamptz AND end_datetime >= %L::timestamptz ',\n edate,\n sdate\n );\n END IF;\n\n IF j ? 'q' THEN\n ft_query := q_to_tsquery(j->'q');\n where_segments := where_segments || format(\n $quote$\n (\n to_tsvector('english', content->'properties'->>'description') ||\n to_tsvector('english', coalesce(content->'properties'->>'title', '')) ||\n to_tsvector('english', coalesce(content->'properties'->>'keywords', ''))\n ) @@ %L\n $quote$,\n ft_query\n );\n END IF;\n\n geom := stac_geom(j);\n IF geom IS NOT NULL THEN\n where_segments := where_segments || format('st_intersects(geometry, %L)',geom);\n END IF;\n\n filterlang := COALESCE(\n j->>'filter-lang',\n get_setting('default_filter_lang', j->'conf')\n );\n IF NOT filter @? '$.**.op' THEN\n filterlang := 'cql-json';\n END IF;\n\n IF filterlang NOT IN ('cql-json','cql2-json') AND j ? 'filter' THEN\n RAISE EXCEPTION '% is not a supported filter-lang. Please use cql-json or cql2-json.', filterlang;\n END IF;\n\n IF j ? 'query' AND j ? 'filter' THEN\n RAISE EXCEPTION 'Can only use either query or filter at one time.';\n END IF;\n\n IF j ? 'query' THEN\n filter := query_to_cql2(j->'query');\n ELSIF filterlang = 'cql-json' THEN\n filter := cql1_to_cql2(filter);\n END IF;\n RAISE NOTICE 'FILTER: %', filter;\n where_segments := where_segments || cql2_query(filter);\n IF cardinality(where_segments) < 1 THEN\n RETURN ' TRUE ';\n END IF;\n\n _where := array_to_string(array_remove(where_segments, NULL), ' AND ');\n\n IF _where IS NULL OR BTRIM(_where) = '' THEN\n RETURN ' TRUE ';\n END IF;\n RETURN _where;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION parse_sort_dir(_dir text, reverse boolean default false) RETURNS text AS $$\n WITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n ) SELECT\n CASE\n WHEN NOT reverse THEN d\n WHEN d = 'ASC' THEN 'DESC'\n WHEN d = 'DESC' THEN 'ASC'\n END\n FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION sort_dir_to_op(_dir text, prev boolean default false) RETURNS text AS $$\n WITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n ) SELECT\n CASE\n WHEN d = 'ASC' AND prev THEN '<='\n WHEN d = 'DESC' AND prev THEN '>='\n WHEN d = 'ASC' THEN '>='\n WHEN d = 'DESC' THEN '<='\n END\n FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION sort_sqlorderby(\n _search jsonb DEFAULT NULL,\n reverse boolean DEFAULT FALSE\n) RETURNS text AS $$\n WITH sortby AS (\n SELECT coalesce(_search->'sortby','[{\"field\":\"datetime\", \"direction\":\"desc\"}]') as sort\n ), withid AS (\n SELECT CASE\n WHEN sort @? '$[*] ? (@.field == \"id\")' THEN sort\n ELSE sort || '[{\"field\":\"id\", \"direction\":\"desc\"}]'::jsonb\n END as sort\n FROM sortby\n ), withid_rows AS (\n SELECT jsonb_array_elements(sort) as value FROM withid\n ),sorts AS (\n SELECT\n coalesce(\n (queryable(value->>'field')).expression\n ) as key,\n parse_sort_dir(value->>'direction', reverse) as dir\n FROM withid_rows\n )\n SELECT array_to_string(\n array_agg(concat(key, ' ', dir)),\n ', '\n ) FROM sorts;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_sort_dir(sort_item jsonb) RETURNS text AS $$\n SELECT CASE WHEN sort_item->>'direction' ILIKE 'desc%' THEN 'DESC' ELSE 'ASC' END;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION get_token_val_str(\n _field text,\n _item items\n) RETURNS text AS $$\nDECLARE\n q text;\n literal text;\nBEGIN\n q := format($q$ SELECT quote_literal(%s) FROM (SELECT $1.*) as r;$q$, _field);\n EXECUTE q INTO literal USING _item;\n RETURN literal;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n\nCREATE OR REPLACE FUNCTION get_token_record(IN _token text, OUT prev BOOLEAN, OUT item items) RETURNS RECORD AS $$\nDECLARE\n _itemid text := _token;\n _collectionid text;\nBEGIN\n IF _token IS NULL THEN\n RETURN;\n END IF;\n RAISE NOTICE 'Looking for token: %', _token;\n prev := FALSE;\n IF _token ILIKE 'prev:%' THEN\n _itemid := replace(_token, 'prev:','');\n prev := TRUE;\n ELSIF _token ILIKE 'next:%' THEN\n _itemid := replace(_token, 'next:', '');\n END IF;\n SELECT id INTO _collectionid FROM collections WHERE _itemid LIKE concat(id,':%');\n IF FOUND THEN\n _itemid := replace(_itemid, concat(_collectionid,':'), '');\n SELECT * INTO item FROM items WHERE id=_itemid AND collection=_collectionid;\n ELSE\n SELECT * INTO item FROM items WHERE id=_itemid;\n END IF;\n IF item IS NULL THEN\n RAISE EXCEPTION 'Could not find item using token: % item: % collection: %', _token, _itemid, _collectionid;\n END IF;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\n\nCREATE OR REPLACE FUNCTION get_token_filter(\n _sortby jsonb DEFAULT '[{\"field\":\"datetime\",\"direction\":\"desc\"}]'::jsonb,\n token_item items DEFAULT NULL,\n prev boolean DEFAULT FALSE,\n inclusive boolean DEFAULT FALSE\n) RETURNS text AS $$\nDECLARE\n ltop text := '<';\n gtop text := '>';\n dir text;\n sort record;\n orfilter text := '';\n orfilters text[] := '{}'::text[];\n andfilters text[] := '{}'::text[];\n output text;\n token_where text;\nBEGIN\n IF _sortby IS NULL OR _sortby = '[]'::jsonb THEN\n _sortby := '[{\"field\":\"datetime\",\"direction\":\"desc\"}]'::jsonb;\n END IF;\n _sortby := _sortby || jsonb_build_object('field','id','direction',_sortby->0->>'direction');\n RAISE NOTICE 'Getting Token Filter. % %', _sortby, token_item;\n IF inclusive THEN\n orfilters := orfilters || format('( id=%L AND collection=%L )' , token_item.id, token_item.collection);\n END IF;\n\n FOR sort IN\n WITH s1 AS (\n SELECT\n _row,\n (queryable(value->>'field')).expression as _field,\n (value->>'field' = 'id') as _isid,\n get_sort_dir(value) as _dir\n FROM jsonb_array_elements(_sortby)\n WITH ORDINALITY AS t(value, _row)\n )\n SELECT\n _row,\n _field,\n _dir,\n get_token_val_str(_field, token_item) as _val\n FROM s1\n WHERE _row <= (SELECT min(_row) FROM s1 WHERE _isid)\n LOOP\n orfilter := NULL;\n RAISE NOTICE 'SORT: %', sort;\n IF sort._val IS NOT NULL AND ((prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC')) THEN\n orfilter := format($f$(\n (%s %s %s) OR (%s IS NULL)\n )$f$,\n sort._field,\n ltop,\n sort._val,\n sort._val\n );\n ELSIF sort._val IS NULL AND ((prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC')) THEN\n RAISE NOTICE '< but null';\n orfilter := format('%s IS NOT NULL', sort._field);\n ELSIF sort._val IS NULL THEN\n RAISE NOTICE '> but null';\n ELSE\n orfilter := format($f$(\n (%s %s %s) OR (%s IS NULL)\n )$f$,\n sort._field,\n gtop,\n sort._val,\n sort._field\n );\n END IF;\n RAISE NOTICE 'ORFILTER: %', orfilter;\n\n IF orfilter IS NOT NULL THEN\n IF sort._row = 1 THEN\n orfilters := orfilters || orfilter;\n ELSE\n orfilters := orfilters || format('(%s AND %s)', array_to_string(andfilters, ' AND '), orfilter);\n END IF;\n END IF;\n IF sort._val IS NOT NULL THEN\n andfilters := andfilters || format('%s = %s', sort._field, sort._val);\n ELSE\n andfilters := andfilters || format('%s IS NULL', sort._field);\n END IF;\n END LOOP;\n\n output := array_to_string(orfilters, ' OR ');\n\n token_where := concat('(',coalesce(output,'true'),')');\n IF trim(token_where) = '' THEN\n token_where := NULL;\n END IF;\n RAISE NOTICE 'TOKEN_WHERE: %',token_where;\n RETURN token_where;\n END;\n$$ LANGUAGE PLPGSQL SET transform_null_equals TO TRUE\n;\n\nCREATE OR REPLACE FUNCTION search_hash(jsonb, jsonb) RETURNS text AS $$\n SELECT md5(concat(($1 - '{token,limit,context,includes,excludes}'::text[])::text,$2::text));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\nDROP FUNCTION IF EXISTS search_tohash(jsonb);\n\nCREATE TABLE IF NOT EXISTS searches(\n hash text GENERATED ALWAYS AS (search_hash(search, metadata)) STORED PRIMARY KEY,\n search jsonb NOT NULL,\n _where text,\n orderby text,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n metadata jsonb DEFAULT '{}'::jsonb NOT NULL\n);\n\nCREATE TABLE IF NOT EXISTS search_wheres(\n id bigint generated always as identity primary key,\n _where text NOT NULL,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n statslastupdated timestamptz,\n estimated_count bigint,\n estimated_cost float,\n time_to_estimate float,\n total_count bigint,\n time_to_count float,\n partitions text[]\n);\n\nCREATE INDEX IF NOT EXISTS search_wheres_partitions ON search_wheres USING GIN (partitions);\nCREATE UNIQUE INDEX IF NOT EXISTS search_wheres_where ON search_wheres ((md5(_where)));\n\nCREATE OR REPLACE FUNCTION where_stats(\n inwhere text,\n updatestats boolean default false,\n conf jsonb default null\n) RETURNS search_wheres AS $$\nDECLARE\n t timestamptz;\n i interval;\n explain_json jsonb;\n partitions text[];\n sw search_wheres%ROWTYPE;\n inwhere_hash text := md5(inwhere);\n _context text := lower(context(conf));\n _stats_ttl interval := context_stats_ttl(conf);\n _estimated_cost_threshold float := context_estimated_cost(conf);\n _estimated_count_threshold int := context_estimated_count(conf);\n ro bool := pgstac.readonly(conf);\nBEGIN\n -- If updatestats is true then set ttl to 0\n IF updatestats THEN\n RAISE DEBUG 'Updatestats set to TRUE, setting TTL to 0';\n _stats_ttl := '0'::interval;\n END IF;\n\n -- If we don't need to calculate context, just return\n IF _context = 'off' THEN\n sw._where = inwhere;\n RETURN sw;\n END IF;\n\n -- Get any stats that we have.\n IF NOT ro THEN\n -- If there is a lock where another process is\n -- updating the stats, wait so that we don't end up calculating a bunch of times.\n SELECT * INTO sw FROM search_wheres WHERE md5(_where)=inwhere_hash FOR UPDATE;\n ELSE\n SELECT * INTO sw FROM search_wheres WHERE md5(_where)=inwhere_hash;\n END IF;\n\n -- If there is a cached row, figure out if we need to update\n IF\n sw IS NOT NULL\n AND sw.statslastupdated IS NOT NULL\n AND sw.total_count IS NOT NULL\n AND now() - sw.statslastupdated <= _stats_ttl\n THEN\n -- we have a cached row with data that is within our ttl\n RAISE DEBUG 'Stats present in table and lastupdated within ttl: %', sw;\n IF NOT ro THEN\n RAISE DEBUG 'Updating search_wheres only bumping lastused and usecount';\n UPDATE search_wheres SET\n lastused = now(),\n usecount = search_wheres.usecount + 1\n WHERE md5(_where) = inwhere_hash\n RETURNING * INTO sw;\n END IF;\n RAISE DEBUG 'Returning cached counts. %', sw;\n RETURN sw;\n END IF;\n\n -- Calculate estimated cost and rows\n -- Use explain to get estimated count/cost\n IF sw.estimated_count IS NULL OR sw.estimated_cost IS NULL THEN\n RAISE DEBUG 'Calculating estimated stats';\n t := clock_timestamp();\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s', inwhere)\n INTO explain_json;\n RAISE DEBUG 'Time for just the explain: %', clock_timestamp() - t;\n i := clock_timestamp() - t;\n\n sw.estimated_count := explain_json->0->'Plan'->'Plan Rows';\n sw.estimated_cost := explain_json->0->'Plan'->'Total Cost';\n sw.time_to_estimate := extract(epoch from i);\n END IF;\n\n RAISE DEBUG 'ESTIMATED_COUNT: %, THRESHOLD %', sw.estimated_count, _estimated_count_threshold;\n RAISE DEBUG 'ESTIMATED_COST: %, THRESHOLD %', sw.estimated_cost, _estimated_cost_threshold;\n\n -- If context is set to auto and the costs are within the threshold return the estimated costs\n IF\n _context = 'auto'\n AND sw.estimated_count >= _estimated_count_threshold\n AND sw.estimated_cost >= _estimated_cost_threshold\n THEN\n IF NOT ro THEN\n INSERT INTO search_wheres (\n _where,\n lastused,\n usecount,\n statslastupdated,\n estimated_count,\n estimated_cost,\n time_to_estimate,\n total_count,\n time_to_count\n ) VALUES (\n inwhere,\n now(),\n 1,\n now(),\n sw.estimated_count,\n sw.estimated_cost,\n sw.time_to_estimate,\n null,\n null\n ) ON CONFLICT ((md5(_where)))\n DO UPDATE SET\n lastused = EXCLUDED.lastused,\n usecount = search_wheres.usecount + 1,\n statslastupdated = EXCLUDED.statslastupdated,\n estimated_count = EXCLUDED.estimated_count,\n estimated_cost = EXCLUDED.estimated_cost,\n time_to_estimate = EXCLUDED.time_to_estimate,\n total_count = EXCLUDED.total_count,\n time_to_count = EXCLUDED.time_to_count\n RETURNING * INTO sw;\n END IF;\n RAISE DEBUG 'Estimates are within thresholds, returning estimates. %', sw;\n RETURN sw;\n END IF;\n\n -- Calculate Actual Count\n t := clock_timestamp();\n RAISE NOTICE 'Calculating actual count...';\n EXECUTE format(\n 'SELECT count(*) FROM items WHERE %s',\n inwhere\n ) INTO sw.total_count;\n i := clock_timestamp() - t;\n RAISE NOTICE 'Actual Count: % -- %', sw.total_count, i;\n sw.time_to_count := extract(epoch FROM i);\n\n IF NOT ro THEN\n INSERT INTO search_wheres (\n _where,\n lastused,\n usecount,\n statslastupdated,\n estimated_count,\n estimated_cost,\n time_to_estimate,\n total_count,\n time_to_count\n ) VALUES (\n inwhere,\n now(),\n 1,\n now(),\n sw.estimated_count,\n sw.estimated_cost,\n sw.time_to_estimate,\n sw.total_count,\n sw.time_to_count\n ) ON CONFLICT ((md5(_where)))\n DO UPDATE SET\n lastused = EXCLUDED.lastused,\n usecount = search_wheres.usecount + 1,\n statslastupdated = EXCLUDED.statslastupdated,\n estimated_count = EXCLUDED.estimated_count,\n estimated_cost = EXCLUDED.estimated_cost,\n time_to_estimate = EXCLUDED.time_to_estimate,\n total_count = EXCLUDED.total_count,\n time_to_count = EXCLUDED.time_to_count\n RETURNING * INTO sw;\n END IF;\n RAISE DEBUG 'Returning with actual count. %', sw;\n RETURN sw;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION search_query(\n _search jsonb = '{}'::jsonb,\n updatestats boolean = false,\n _metadata jsonb = '{}'::jsonb\n) RETURNS searches AS $$\nDECLARE\n search searches%ROWTYPE;\n cached_search searches%ROWTYPE;\n pexplain jsonb;\n t timestamptz;\n i interval;\n doupdate boolean := FALSE;\n insertfound boolean := FALSE;\n ro boolean := pgstac.readonly();\n found_search text;\nBEGIN\n RAISE NOTICE 'SEARCH: %', _search;\n -- Calculate hash, where clause, and order by statement\n search.search := _search;\n search.metadata := _metadata;\n search.hash := search_hash(_search, _metadata);\n search._where := stac_search_to_where(_search);\n search.orderby := sort_sqlorderby(_search);\n search.lastused := now();\n search.usecount := 1;\n\n -- If we are in read only mode, directly return search\n IF ro THEN\n RETURN search;\n END IF;\n\n RAISE NOTICE 'Updating Statistics for search: %s', search;\n -- Update statistics for times used and and when last used\n -- If the entry is locked, rather than waiting, skip updating the stats\n INSERT INTO searches (search, lastused, usecount, metadata)\n VALUES (search.search, now(), 1, search.metadata)\n ON CONFLICT DO NOTHING\n RETURNING * INTO cached_search\n ;\n\n IF NOT FOUND OR cached_search IS NULL THEN\n UPDATE searches SET\n lastused = now(),\n usecount = searches.usecount + 1\n WHERE hash = (\n SELECT hash FROM searches WHERE hash=search.hash FOR UPDATE SKIP LOCKED\n )\n RETURNING * INTO cached_search\n ;\n END IF;\n\n IF cached_search IS NOT NULL THEN\n cached_search._where = search._where;\n cached_search.orderby = search.orderby;\n RETURN cached_search;\n END IF;\n RETURN search;\n\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE OR REPLACE FUNCTION search_fromhash(\n _hash text\n) RETURNS searches AS $$\n SELECT * FROM search_query((SELECT search FROM searches WHERE hash=_hash LIMIT 1));\n$$ LANGUAGE SQL STRICT;\n\nCREATE OR REPLACE FUNCTION search_rows(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN partitions text[] DEFAULT NULL,\n IN _limit int DEFAULT 10\n) RETURNS SETOF items AS $$\nDECLARE\n base_query text;\n query text;\n sdate timestamptz;\n edate timestamptz;\n n int;\n records_left int := _limit;\n timer timestamptz := clock_timestamp();\n full_timer timestamptz := clock_timestamp();\nBEGIN\nIF _where IS NULL OR trim(_where) = '' THEN\n _where = ' TRUE ';\nEND IF;\nRAISE NOTICE 'Getting chunks for % %', _where, _orderby;\n\nbase_query := $q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n LIMIT %L\n$q$;\n\nIF _orderby ILIKE 'datetime d%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 DESC LOOP\n RAISE NOTICE 'Running Query for % to %. %', sdate, edate, age_ms(full_timer);\n query := format(\n base_query,\n sdate,\n edate,\n _where,\n _orderby,\n records_left\n );\n RAISE DEBUG 'QUERY: %', query;\n timer := clock_timestamp();\n RETURN QUERY EXECUTE query;\n\n GET DIAGNOSTICS n = ROW_COUNT;\n records_left := records_left - n;\n RAISE NOTICE 'Returned %/% Rows From % to %. % to go. Time: %ms', n, _limit, sdate, edate, records_left, age_ms(timer);\n timer := clock_timestamp();\n IF records_left <= 0 THEN\n RAISE NOTICE 'SEARCH_ROWS TOOK %ms', age_ms(full_timer);\n RETURN;\n END IF;\n END LOOP;\nELSIF _orderby ILIKE 'datetime a%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 ASC LOOP\n RAISE NOTICE 'Running Query for % to %. %', sdate, edate, age_ms(full_timer);\n query := format(\n base_query,\n sdate,\n edate,\n _where,\n _orderby,\n records_left\n );\n RAISE DEBUG 'QUERY: %', query;\n timer := clock_timestamp();\n RETURN QUERY EXECUTE query;\n\n GET DIAGNOSTICS n = ROW_COUNT;\n records_left := records_left - n;\n RAISE NOTICE 'Returned %/% Rows From % to %. % to go. Time: %ms', n, _limit, sdate, edate, records_left, age_ms(timer);\n timer := clock_timestamp();\n IF records_left <= 0 THEN\n RAISE NOTICE 'SEARCH_ROWS TOOK %ms', age_ms(full_timer);\n RETURN;\n END IF;\n END LOOP;\nELSE\n query := format($q$\n SELECT * FROM items\n WHERE %s\n ORDER BY %s\n LIMIT %L\n $q$, _where, _orderby, _limit\n );\n RAISE DEBUG 'QUERY: %', query;\n timer := clock_timestamp();\n RETURN QUERY EXECUTE query;\n RAISE NOTICE 'FULL QUERY TOOK %ms', age_ms(timer);\nEND IF;\nRAISE NOTICE 'SEARCH_ROWS TOOK %ms', age_ms(full_timer);\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE UNLOGGED TABLE format_item_cache(\n id text,\n collection text,\n fields text,\n hydrated bool,\n output jsonb,\n lastused timestamptz DEFAULT now(),\n usecount int DEFAULT 1,\n timetoformat float,\n PRIMARY KEY (collection, id, fields, hydrated)\n);\nCREATE INDEX ON format_item_cache (lastused);\n\nCREATE OR REPLACE FUNCTION format_item(_item items, _fields jsonb DEFAULT '{}', _hydrated bool DEFAULT TRUE) RETURNS jsonb AS $$\nDECLARE\n cache bool := get_setting_bool('format_cache');\n _output jsonb := null;\n t timestamptz := clock_timestamp();\nBEGIN\n IF cache THEN\n SELECT output INTO _output FROM format_item_cache\n WHERE id=_item.id AND collection=_item.collection AND fields=_fields::text AND hydrated=_hydrated;\n END IF;\n IF _output IS NULL THEN\n IF _hydrated THEN\n _output := content_hydrate(_item, _fields);\n ELSE\n _output := content_nonhydrated(_item, _fields);\n END IF;\n END IF;\n IF cache THEN\n INSERT INTO format_item_cache (id, collection, fields, hydrated, output, timetoformat)\n VALUES (_item.id, _item.collection, _fields::text, _hydrated, _output, age_ms(t))\n ON CONFLICT(collection, id, fields, hydrated) DO\n UPDATE\n SET lastused=now(), usecount = format_item_cache.usecount + 1\n ;\n END IF;\n RETURN _output;\n\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION search(_search jsonb = '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n searches searches%ROWTYPE;\n _where text;\n orderby text;\n search_where search_wheres%ROWTYPE;\n total_count bigint;\n token record;\n token_prev boolean;\n token_item items%ROWTYPE;\n token_where text;\n full_where text;\n init_ts timestamptz := clock_timestamp();\n timer timestamptz := clock_timestamp();\n hydrate bool := NOT (_search->'conf'->>'nohydrate' IS NOT NULL AND (_search->'conf'->>'nohydrate')::boolean = true);\n prev text;\n next text;\n context jsonb;\n collection jsonb;\n out_records jsonb;\n out_len int;\n _limit int := coalesce((_search->>'limit')::int, 10);\n _querylimit int;\n _fields jsonb := coalesce(_search->'fields', '{}'::jsonb);\n has_prev boolean := FALSE;\n has_next boolean := FALSE;\n links jsonb := '[]'::jsonb;\n base_url text:= concat(rtrim(base_url(_search->'conf'),'/'));\nBEGIN\n searches := search_query(_search);\n _where := searches._where;\n orderby := searches.orderby;\n search_where := where_stats(_where);\n total_count := coalesce(search_where.total_count, search_where.estimated_count);\n RAISE NOTICE 'SEARCH:TOKEN: %', _search->>'token';\n token := get_token_record(_search->>'token');\n RAISE NOTICE '***TOKEN: %', token;\n _querylimit := _limit + 1;\n IF token IS NOT NULL THEN\n token_prev := token.prev;\n token_item := token.item;\n token_where := get_token_filter(_search->'sortby', token_item, token_prev, FALSE);\n RAISE DEBUG 'TOKEN_WHERE: % (%ms from search start)', token_where, age_ms(timer);\n IF token_prev THEN -- if we are using a prev token, we know has_next is true\n RAISE DEBUG 'There is a previous token, so automatically setting has_next to true';\n has_next := TRUE;\n orderby := sort_sqlorderby(_search, TRUE);\n ELSE\n RAISE DEBUG 'There is a next token, so automatically setting has_prev to true';\n has_prev := TRUE;\n\n END IF;\n ELSE -- if there was no token, we know there is no prev\n RAISE DEBUG 'There is no token, so we know there is no prev. setting has_prev to false';\n has_prev := FALSE;\n END IF;\n\n full_where := concat_ws(' AND ', _where, token_where);\n RAISE NOTICE 'FULL WHERE CLAUSE: %', full_where;\n RAISE NOTICE 'Time to get counts and build query %', age_ms(timer);\n timer := clock_timestamp();\n\n IF hydrate THEN\n RAISE NOTICE 'Getting hydrated data.';\n ELSE\n RAISE NOTICE 'Getting non-hydrated data.';\n END IF;\n RAISE NOTICE 'CACHE SET TO %', get_setting_bool('format_cache');\n RAISE NOTICE 'Time to set hydration/formatting %', age_ms(timer);\n timer := clock_timestamp();\n SELECT jsonb_agg(format_item(i, _fields, hydrate)) INTO out_records\n FROM search_rows(\n full_where,\n orderby,\n search_where.partitions,\n _querylimit\n ) as i;\n\n RAISE NOTICE 'Time to fetch rows %', age_ms(timer);\n timer := clock_timestamp();\n\n\n IF token_prev THEN\n out_records := flip_jsonb_array(out_records);\n END IF;\n\n RAISE NOTICE 'Query returned % records.', jsonb_array_length(out_records);\n RAISE DEBUG 'TOKEN: % %', token_item.id, token_item.collection;\n RAISE DEBUG 'RECORD_1: % %', out_records->0->>'id', out_records->0->>'collection';\n RAISE DEBUG 'RECORD-1: % %', out_records->-1->>'id', out_records->-1->>'collection';\n\n -- REMOVE records that were from our token\n IF out_records->0->>'id' = token_item.id AND out_records->0->>'collection' = token_item.collection THEN\n out_records := out_records - 0;\n ELSIF out_records->-1->>'id' = token_item.id AND out_records->-1->>'collection' = token_item.collection THEN\n out_records := out_records - -1;\n END IF;\n\n out_len := jsonb_array_length(out_records);\n\n IF out_len = _limit + 1 THEN\n IF token_prev THEN\n has_prev := TRUE;\n out_records := out_records - 0;\n ELSE\n has_next := TRUE;\n out_records := out_records - -1;\n END IF;\n END IF;\n\n\n links := links || jsonb_build_object(\n 'rel', 'root',\n 'type', 'application/json',\n 'href', base_url\n ) || jsonb_build_object(\n 'rel', 'self',\n 'type', 'application/json',\n 'href', concat(base_url, '/search')\n );\n\n IF has_next THEN\n next := concat(out_records->-1->>'collection', ':', out_records->-1->>'id');\n RAISE NOTICE 'HAS NEXT | %', next;\n links := links || jsonb_build_object(\n 'rel', 'next',\n 'type', 'application/geo+json',\n 'method', 'GET',\n 'href', concat(base_url, '/search?token=next:', next)\n );\n END IF;\n\n IF has_prev THEN\n prev := concat(out_records->0->>'collection', ':', out_records->0->>'id');\n RAISE NOTICE 'HAS PREV | %', prev;\n links := links || jsonb_build_object(\n 'rel', 'prev',\n 'type', 'application/geo+json',\n 'method', 'GET',\n 'href', concat(base_url, '/search?token=prev:', prev)\n );\n END IF;\n\n RAISE NOTICE 'Time to get prev/next %', age_ms(timer);\n timer := clock_timestamp();\n\n\n collection := jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb),\n 'links', links\n );\n\n\n\n IF context(_search->'conf') != 'off' THEN\n collection := collection || jsonb_strip_nulls(jsonb_build_object(\n 'numberMatched', total_count,\n 'numberReturned', coalesce(jsonb_array_length(out_records), 0)\n ));\n ELSE\n collection := collection || jsonb_strip_nulls(jsonb_build_object(\n 'numberReturned', coalesce(jsonb_array_length(out_records), 0)\n ));\n END IF;\n\n IF get_setting_bool('timing', _search->'conf') THEN\n collection = collection || jsonb_build_object('timing', age_ms(init_ts));\n END IF;\n\n RAISE NOTICE 'Time to build final json %', age_ms(timer);\n timer := clock_timestamp();\n\n RAISE NOTICE 'Total Time: %', age_ms(current_timestamp);\n RAISE NOTICE 'RETURNING % records. NEXT: %. PREV: %', collection->>'numberReturned', collection->>'next', collection->>'prev';\n RETURN collection;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION search_cursor(_search jsonb = '{}'::jsonb) RETURNS refcursor AS $$\nDECLARE\n curs refcursor;\n searches searches%ROWTYPE;\n _where text;\n _orderby text;\n q text;\n\nBEGIN\n searches := search_query(_search);\n _where := searches._where;\n _orderby := searches.orderby;\n\n OPEN curs FOR\n WITH p AS (\n SELECT * FROM partition_queries(_where, _orderby) p\n )\n SELECT\n CASE WHEN EXISTS (SELECT 1 FROM p) THEN\n (SELECT format($q$\n SELECT * FROM (\n %s\n ) total\n $q$,\n string_agg(\n format($q$ SELECT * FROM ( %s ) AS sub $q$, p),\n '\n UNION ALL\n '\n )\n ))\n ELSE NULL\n END FROM p;\n RETURN curs;\nEND;\n$$ LANGUAGE PLPGSQL;\nCREATE OR REPLACE VIEW collections_asitems AS\nSELECT\n id,\n geometry,\n 'collections' AS collection,\n datetime,\n end_datetime,\n jsonb_build_object(\n 'properties', content - '{links,assets,stac_version,stac_extensions}',\n 'links', content->'links',\n 'assets', content->'assets',\n 'stac_version', content->'stac_version',\n 'stac_extensions', content->'stac_extensions'\n ) AS content,\n content as collectionjson\nFROM collections;\n\n\nCREATE OR REPLACE FUNCTION collection_search_matched(\n IN _search jsonb DEFAULT '{}'::jsonb,\n OUT matched bigint\n) RETURNS bigint AS $$\nDECLARE\n _where text := stac_search_to_where(_search);\nBEGIN\n EXECUTE format(\n $query$\n SELECT\n count(*)\n FROM\n collections_asitems\n WHERE %s\n ;\n $query$,\n _where\n ) INTO matched;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION collection_search_rows(\n _search jsonb DEFAULT '{}'::jsonb\n) RETURNS SETOF jsonb AS $$\nDECLARE\n _where text := stac_search_to_where(_search);\n _limit int := coalesce((_search->>'limit')::int, 10);\n _fields jsonb := coalesce(_search->'fields', '{}'::jsonb);\n _orderby text;\n _offset int := COALESCE((_search->>'offset')::int, 0);\nBEGIN\n _orderby := sort_sqlorderby(\n jsonb_build_object(\n 'sortby',\n coalesce(\n _search->'sortby',\n '[{\"field\": \"id\", \"direction\": \"asc\"}]'::jsonb\n )\n )\n );\n RETURN QUERY EXECUTE format(\n $query$\n SELECT\n jsonb_fields(collectionjson, %L) as c\n FROM\n collections_asitems\n WHERE %s\n ORDER BY %s\n LIMIT %L\n OFFSET %L\n ;\n $query$,\n _fields,\n _where,\n _orderby,\n _limit,\n _offset\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION collection_search(\n _search jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\nDECLARE\n out_records jsonb;\n number_matched bigint := collection_search_matched(_search);\n number_returned bigint;\n _limit int := coalesce((_search->>'limit')::float::int, 10);\n _offset int := coalesce((_search->>'offset')::float::int, 0);\n links jsonb := '[]';\n ret jsonb;\n base_url text:= concat(rtrim(base_url(_search->'conf'),'/'), '/collections');\n prevoffset int;\n nextoffset int;\nBEGIN\n SELECT\n coalesce(jsonb_agg(c), '[]')\n INTO out_records\n FROM collection_search_rows(_search) c;\n\n number_returned := jsonb_array_length(out_records);\n RAISE DEBUG 'nm: %, nr: %, l:%, o:%', number_matched, number_returned, _limit, _offset;\n\n\n\n IF _limit <= number_matched AND number_matched > 0 THEN --need to have paging links\n nextoffset := least(_offset + _limit, number_matched - 1);\n prevoffset := greatest(_offset - _limit, 0);\n\n IF _offset > 0 THEN\n links := links || jsonb_build_object(\n 'rel', 'prev',\n 'type', 'application/json',\n 'method', 'GET' ,\n 'href', base_url,\n 'body', jsonb_build_object('offset', prevoffset),\n 'merge', TRUE\n );\n END IF;\n\n IF (_offset + _limit < number_matched) THEN\n links := links || jsonb_build_object(\n 'rel', 'next',\n 'type', 'application/json',\n 'method', 'GET' ,\n 'href', base_url,\n 'body', jsonb_build_object('offset', nextoffset),\n 'merge', TRUE\n );\n END IF;\n\n END IF;\n\n ret := jsonb_build_object(\n 'collections', out_records,\n 'numberMatched', number_matched,\n 'numberReturned', number_returned,\n 'links', links\n );\n RETURN ret;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION tileenvelope(zoom int, x int, y int) RETURNS geometry AS $$\nWITH t AS (\n SELECT\n 20037508.3427892 as merc_max,\n -20037508.3427892 as merc_min,\n (2 * 20037508.3427892) / (2 ^ zoom) as tile_size\n)\nSELECT st_makeenvelope(\n merc_min + (tile_size * x),\n merc_max - (tile_size * (y + 1)),\n merc_min + (tile_size * (x + 1)),\n merc_max - (tile_size * y),\n 3857\n) FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;DROP FUNCTION IF EXISTS mercgrid;\n\n\nCREATE OR REPLACE FUNCTION ftime() RETURNS interval as $$\nSELECT age(clock_timestamp(), transaction_timestamp());\n$$ LANGUAGE SQL;\nSET SEARCH_PATH to pgstac, public;\n\nDROP FUNCTION IF EXISTS geometrysearch;\nCREATE OR REPLACE FUNCTION geometrysearch(\n IN geom geometry,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE, -- Return as soon as the passed in geometry is full covered\n IN skipcovered boolean DEFAULT TRUE -- Skip any items that would show up completely under the previous items\n) RETURNS jsonb AS $$\nDECLARE\n search searches%ROWTYPE;\n curs refcursor;\n _where text;\n query text;\n iter_record items%ROWTYPE;\n out_records jsonb := '{}'::jsonb[];\n exit_flag boolean := FALSE;\n counter int := 1;\n scancounter int := 1;\n remaining_limit int := _scanlimit;\n tilearea float;\n unionedgeom geometry;\n clippedgeom geometry;\n unionedgeom_area float := 0;\n prev_area float := 0;\n excludes text[];\n includes text[];\n\nBEGIN\n DROP TABLE IF EXISTS pgstac_results;\n CREATE TEMP TABLE pgstac_results (content jsonb) ON COMMIT DROP;\n\n -- If the passed in geometry is not an area set exitwhenfull and skipcovered to false\n IF ST_GeometryType(geom) !~* 'polygon' THEN\n RAISE NOTICE 'GEOMETRY IS NOT AN AREA';\n skipcovered = FALSE;\n exitwhenfull = FALSE;\n END IF;\n\n -- If skipcovered is true then you will always want to exit when the passed in geometry is full\n IF skipcovered THEN\n exitwhenfull := TRUE;\n END IF;\n\n search := search_fromhash(queryhash);\n\n IF search IS NULL THEN\n RAISE EXCEPTION 'Search with Query Hash % Not Found', queryhash;\n END IF;\n\n tilearea := st_area(geom);\n _where := format('%s AND st_intersects(geometry, %L::geometry)', search._where, geom);\n\n\n FOR query IN SELECT * FROM partition_queries(_where, search.orderby) LOOP\n query := format('%s LIMIT %L', query, remaining_limit);\n RAISE NOTICE '%', query;\n OPEN curs FOR EXECUTE query;\n LOOP\n FETCH curs INTO iter_record;\n EXIT WHEN NOT FOUND;\n IF exitwhenfull OR skipcovered THEN -- If we are not using exitwhenfull or skipcovered, we do not need to do expensive geometry operations\n clippedgeom := st_intersection(geom, iter_record.geometry);\n\n IF unionedgeom IS NULL THEN\n unionedgeom := clippedgeom;\n ELSE\n unionedgeom := st_union(unionedgeom, clippedgeom);\n END IF;\n\n unionedgeom_area := st_area(unionedgeom);\n\n IF skipcovered AND prev_area = unionedgeom_area THEN\n scancounter := scancounter + 1;\n CONTINUE;\n END IF;\n\n prev_area := unionedgeom_area;\n\n RAISE NOTICE '% % % %', unionedgeom_area/tilearea, counter, scancounter, ftime();\n END IF;\n RAISE NOTICE '% %', iter_record, content_hydrate(iter_record, fields);\n INSERT INTO pgstac_results (content) VALUES (content_hydrate(iter_record, fields));\n\n IF counter >= _limit\n OR scancounter > _scanlimit\n OR ftime() > _timelimit\n OR (exitwhenfull AND unionedgeom_area >= tilearea)\n THEN\n exit_flag := TRUE;\n EXIT;\n END IF;\n counter := counter + 1;\n scancounter := scancounter + 1;\n\n END LOOP;\n CLOSE curs;\n EXIT WHEN exit_flag;\n remaining_limit := _scanlimit - scancounter;\n END LOOP;\n\n SELECT jsonb_agg(content) INTO out_records FROM pgstac_results WHERE content IS NOT NULL;\n\n RETURN jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb)\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDROP FUNCTION IF EXISTS geojsonsearch;\nCREATE OR REPLACE FUNCTION geojsonsearch(\n IN geojson jsonb,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_geomfromgeojson(geojson),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS xyzsearch;\nCREATE OR REPLACE FUNCTION xyzsearch(\n IN _x int,\n IN _y int,\n IN _z int,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_transform(tileenvelope(_z, _x, _y), 4326),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE PROCEDURE analyze_items() AS $$\nDECLARE\n q text;\n timeout_ts timestamptz;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n SELECT format('ANALYZE (VERBOSE, SKIP_LOCKED) %I;', relname) INTO q\n FROM pg_stat_user_tables\n WHERE relname like '_item%' AND (n_mod_since_analyze>0 OR last_analyze IS NULL) LIMIT 1;\n IF NOT FOUND THEN\n EXIT;\n END IF;\n RAISE NOTICE '%', q;\n EXECUTE q;\n COMMIT;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE PROCEDURE validate_constraints() AS $$\nDECLARE\n q text;\nBEGIN\n FOR q IN\n SELECT\n FORMAT(\n 'ALTER TABLE %I.%I VALIDATE CONSTRAINT %I;',\n nsp.nspname,\n cls.relname,\n con.conname\n )\n\n FROM pg_constraint AS con\n JOIN pg_class AS cls\n ON con.conrelid = cls.oid\n JOIN pg_namespace AS nsp\n ON cls.relnamespace = nsp.oid\n WHERE convalidated = FALSE AND contype in ('c','f')\n AND nsp.nspname = 'pgstac'\n LOOP\n RAISE NOTICE '%', q;\n PERFORM run_or_queue(q);\n COMMIT;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION collection_extent(_collection text, runupdate boolean default false) RETURNS jsonb AS $$\nDECLARE\n geom_extent geometry;\n mind timestamptz;\n maxd timestamptz;\n extent jsonb;\nBEGIN\n IF runupdate THEN\n PERFORM update_partition_stats_q(partition)\n FROM partitions_view WHERE collection=_collection;\n END IF;\n SELECT\n min(lower(dtrange)),\n max(upper(edtrange)),\n st_extent(spatial)\n INTO\n mind,\n maxd,\n geom_extent\n FROM partitions_view\n WHERE collection=_collection;\n\n IF geom_extent IS NOT NULL AND mind IS NOT NULL AND maxd IS NOT NULL THEN\n extent := jsonb_build_object(\n 'spatial', jsonb_build_object(\n 'bbox', to_jsonb(array[array[st_xmin(geom_extent), st_ymin(geom_extent), st_xmax(geom_extent), st_ymax(geom_extent)]])\n ),\n 'temporal', jsonb_build_object(\n 'interval', to_jsonb(array[array[mind, maxd]])\n )\n );\n RETURN extent;\n END IF;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('id', '{\"title\": \"Item ID\",\"description\": \"Item identifier\",\"$ref\": \"https://schemas.stacspec.org/v1.0.0/item-spec/json-schema/item.json#/definitions/core/allOf/2/properties/id\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('geometry', '{\"title\": \"Item Geometry\",\"description\": \"Item Geometry\",\"$ref\": \"https://geojson.org/schema/Feature.json\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('datetime','{\"description\": \"Datetime\",\"type\": \"string\",\"title\": \"Acquired\",\"format\": \"date-time\",\"pattern\": \"(\\\\+00:00|Z)$\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDELETE FROM queryables a USING queryables b\n WHERE a.name = b.name AND a.collection_ids IS NOT DISTINCT FROM b.collection_ids AND a.id > b.id;\n\n\nINSERT INTO pgstac_settings (name, value) VALUES\n ('context', 'off'),\n ('context_estimated_count', '100000'),\n ('context_estimated_cost', '100000'),\n ('context_stats_ttl', '1 day'),\n ('default_filter_lang', 'cql2-json'),\n ('additional_properties', 'true'),\n ('use_queue', 'false'),\n ('queue_timeout', '10 minutes'),\n ('update_collection_extent', 'false'),\n ('format_cache', 'false'),\n ('readonly', 'false')\nON CONFLICT DO NOTHING\n;\n\n\nINSERT INTO cql2_ops (op, template, types) VALUES\n ('eq', '%s = %s', NULL),\n ('neq', '%s != %s', NULL),\n ('ne', '%s != %s', NULL),\n ('!=', '%s != %s', NULL),\n ('<>', '%s != %s', NULL),\n ('lt', '%s < %s', NULL),\n ('lte', '%s <= %s', NULL),\n ('gt', '%s > %s', NULL),\n ('gte', '%s >= %s', NULL),\n ('le', '%s <= %s', NULL),\n ('ge', '%s >= %s', NULL),\n ('=', '%s = %s', NULL),\n ('<', '%s < %s', NULL),\n ('<=', '%s <= %s', NULL),\n ('>', '%s > %s', NULL),\n ('>=', '%s >= %s', NULL),\n ('like', '%s LIKE %s', NULL),\n ('ilike', '%s ILIKE %s', NULL),\n ('+', '%s + %s', NULL),\n ('-', '%s - %s', NULL),\n ('*', '%s * %s', NULL),\n ('/', '%s / %s', NULL),\n ('not', 'NOT (%s)', NULL),\n ('between', '%s BETWEEN %s AND %s', NULL),\n ('isnull', '%s IS NULL', NULL),\n ('upper', 'upper(%s)', NULL),\n ('lower', 'lower(%s)', NULL),\n ('casei', 'upper(%s)', NULL),\n ('accenti', 'unaccent(%s)', NULL)\nON CONFLICT (op) DO UPDATE\n SET\n template = EXCLUDED.template\n;\n\n\nALTER FUNCTION to_text COST 5000;\nALTER FUNCTION to_float COST 5000;\nALTER FUNCTION to_int COST 5000;\nALTER FUNCTION to_tstz COST 5000;\nALTER FUNCTION to_text_array COST 5000;\n\nALTER FUNCTION update_partition_stats SECURITY DEFINER;\nALTER FUNCTION partition_after_triggerfunc SECURITY DEFINER;\nALTER FUNCTION drop_table_constraints SECURITY DEFINER;\nALTER FUNCTION create_table_constraints SECURITY DEFINER;\nALTER FUNCTION check_partition SECURITY DEFINER;\nALTER FUNCTION repartition SECURITY DEFINER;\nALTER FUNCTION where_stats SECURITY DEFINER;\nALTER FUNCTION search_query SECURITY DEFINER;\nALTER FUNCTION format_item SECURITY DEFINER;\nALTER FUNCTION maintain_index SECURITY DEFINER;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nGRANT ALL ON SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac to pgstac_admin;\n\n-- pgstac_read role limited to using function apis\nGRANT EXECUTE ON FUNCTION search TO pgstac_read;\nGRANT EXECUTE ON FUNCTION search_query TO pgstac_read;\nGRANT EXECUTE ON FUNCTION item_by_id TO pgstac_read;\nGRANT EXECUTE ON FUNCTION get_item TO pgstac_read;\nGRANT SELECT ON ALL TABLES IN SCHEMA pgstac TO pgstac_read;\n\n\nGRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_ingest;\nGRANT USAGE ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_ingest;\n\nREVOKE ALL PRIVILEGES ON PROCEDURE run_queued_queries FROM public;\nGRANT ALL ON PROCEDURE run_queued_queries TO pgstac_admin;\n\nREVOKE ALL PRIVILEGES ON FUNCTION run_queued_queries_intransaction FROM public;\nGRANT ALL ON FUNCTION run_queued_queries_intransaction TO pgstac_admin;\n\nRESET ROLE;\n\nSET ROLE pgstac_ingest;\nSELECT update_partition_stats_q(partition) FROM partitions_view;\nSELECT set_version('unreleased');\n" }, { "path": "src/pgstac/pgstac.sql", "content": "RESET ROLE;\nDO $$\nDECLARE\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='postgis') THEN\n CREATE EXTENSION IF NOT EXISTS postgis;\n END IF;\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='btree_gist') THEN\n CREATE EXTENSION IF NOT EXISTS btree_gist;\n END IF;\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='unaccent') THEN\n CREATE EXTENSION IF NOT EXISTS unaccent;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_admin;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_read;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_ingest;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n\nGRANT pgstac_admin TO current_user;\n\n-- Function to make sure pgstac_admin is the owner of items\nCREATE OR REPLACE FUNCTION pgstac_admin_owns() RETURNS VOID AS $$\nDECLARE\n f RECORD;\nBEGIN\n FOR f IN (\n SELECT\n concat(\n oid::regproc::text,\n '(',\n coalesce(pg_get_function_identity_arguments(oid),''),\n ')'\n ) AS name,\n CASE prokind WHEN 'f' THEN 'FUNCTION' WHEN 'p' THEN 'PROCEDURE' WHEN 'a' THEN 'AGGREGATE' END as typ\n FROM pg_proc\n WHERE\n pronamespace=to_regnamespace('pgstac')\n AND proowner != to_regrole('pgstac_admin')\n AND proname NOT LIKE 'pg_stat%'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n FOR f IN (\n SELECT\n oid::regclass::text as name,\n CASE relkind\n WHEN 'i' THEN 'INDEX'\n WHEN 'I' THEN 'INDEX'\n WHEN 'p' THEN 'TABLE'\n WHEN 'r' THEN 'TABLE'\n WHEN 'v' THEN 'VIEW'\n WHEN 'S' THEN 'SEQUENCE'\n ELSE NULL\n END as typ\n FROM pg_class\n WHERE relnamespace=to_regnamespace('pgstac') and relowner != to_regrole('pgstac_admin') AND relkind IN ('r','p','v','S') AND relname NOT LIKE 'pg_stat'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\nSELECT pgstac_admin_owns();\n\nCREATE SCHEMA IF NOT EXISTS pgstac AUTHORIZATION pgstac_admin;\n\nGRANT ALL ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_admin;\n\nALTER ROLE pgstac_admin SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_read SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_ingest SET SEARCH_PATH TO pgstac, public;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\n\nGRANT pgstac_read TO pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\n\nSET ROLE pgstac_admin;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\nRESET ROLE;\n\nSET ROLE pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\nRESET ROLE;\n\nSET SEARCH_PATH TO pgstac, public;\nSET ROLE pgstac_admin;\n\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS analyze_items;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS validate_constraints;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n-- Install these idempotently as migrations do not put them before trying to modify the collections table\n\n\nCREATE OR REPLACE FUNCTION collection_geom(content jsonb)\nRETURNS geometry AS $$\n WITH box AS (SELECT content->'extent'->'spatial'->'bbox'->0 as box)\n SELECT\n st_makeenvelope(\n (box->>0)::float,\n (box->>1)::float,\n (box->>2)::float,\n (box->>3)::float,\n 4326\n )\n FROM box;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION collection_datetime(content jsonb)\nRETURNS timestamptz AS $$\n SELECT\n CASE\n WHEN\n (content->'extent'->'temporal'->'interval'->0->>0) IS NULL\n THEN '-infinity'::timestamptz\n ELSE\n (content->'extent'->'temporal'->'interval'->0->>0)::timestamptz\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION collection_enddatetime(content jsonb)\nRETURNS timestamptz AS $$\n SELECT\n CASE\n WHEN\n (content->'extent'->'temporal'->'interval'->0->>1) IS NULL\n THEN 'infinity'::timestamptz\n ELSE\n (content->'extent'->'temporal'->'interval'->0->>1)::timestamptz\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE TABLE IF NOT EXISTS migrations (\n version text PRIMARY KEY,\n datetime timestamptz DEFAULT clock_timestamp() NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION get_version() RETURNS text AS $$\n SELECT version FROM pgstac.migrations ORDER BY datetime DESC, version DESC LIMIT 1;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION set_version(text) RETURNS text AS $$\n INSERT INTO pgstac.migrations (version) VALUES ($1)\n ON CONFLICT DO NOTHING\n RETURNING version;\n$$ LANGUAGE SQL;\n\n\nCREATE TABLE IF NOT EXISTS pgstac_settings (\n name text PRIMARY KEY,\n value text NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION table_empty(text) RETURNS boolean AS $$\nDECLARE\n retval boolean;\nBEGIN\n EXECUTE format($q$\n SELECT NOT EXISTS (SELECT 1 FROM %I LIMIT 1)\n $q$,\n $1\n ) INTO retval;\n RETURN retval;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION get_setting(IN _setting text, IN conf jsonb DEFAULT NULL) RETURNS text AS $$\nSELECT COALESCE(\n nullif(conf->>_setting, ''),\n nullif(current_setting(concat('pgstac.',_setting), TRUE),''),\n nullif((SELECT value FROM pgstac.pgstac_settings WHERE name=_setting),'')\n);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_setting_bool(IN _setting text, IN conf jsonb DEFAULT NULL) RETURNS boolean AS $$\nSELECT COALESCE(\n nullif(conf->>_setting, ''),\n nullif(current_setting(concat('pgstac.',_setting), TRUE),''),\n nullif((SELECT value FROM pgstac.pgstac_settings WHERE name=_setting),''),\n 'FALSE'\n)::boolean;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION base_url(conf jsonb DEFAULT NULL) RETURNS text AS $$\n SELECT COALESCE(pgstac.get_setting('base_url', conf), '.');\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION additional_properties() RETURNS boolean AS $$\n SELECT pgstac.get_setting_bool('additional_properties');\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION readonly(conf jsonb DEFAULT NULL) RETURNS boolean AS $$\n SELECT pgstac.get_setting_bool('readonly', conf);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context(conf jsonb DEFAULT NULL) RETURNS text AS $$\n SELECT pgstac.get_setting('context', conf);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context_estimated_count(conf jsonb DEFAULT NULL) RETURNS int AS $$\n SELECT pgstac.get_setting('context_estimated_count', conf)::int;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_estimated_cost();\nCREATE OR REPLACE FUNCTION context_estimated_cost(conf jsonb DEFAULT NULL) RETURNS float AS $$\n SELECT pgstac.get_setting('context_estimated_cost', conf)::float;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_stats_ttl();\nCREATE OR REPLACE FUNCTION context_stats_ttl(conf jsonb DEFAULT NULL) RETURNS interval AS $$\n SELECT pgstac.get_setting('context_stats_ttl', conf)::interval;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION t2s(text) RETURNS text AS $$\n SELECT extract(epoch FROM $1::interval)::text || ' s';\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE STRICT;\n\nCREATE OR REPLACE FUNCTION age_ms(a timestamptz, b timestamptz DEFAULT clock_timestamp()) RETURNS float AS $$\n SELECT abs(extract(epoch from age(a,b)) * 1000);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION queue_timeout() RETURNS interval AS $$\n SELECT t2s(coalesce(\n get_setting('queue_timeout'),\n '1h'\n ))::interval;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION notice(VARIADIC text[]) RETURNS boolean AS $$\nDECLARE\ndebug boolean := current_setting('pgstac.debug', true);\nBEGIN\n IF debug THEN\n RAISE NOTICE 'NOTICE FROM FUNC: % >>>>> %', concat_ws(' | ', $1), clock_timestamp();\n RETURN TRUE;\n END IF;\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION empty_arr(ANYARRAY) RETURNS BOOLEAN AS $$\nSELECT CASE\n WHEN $1 IS NULL THEN TRUE\n WHEN cardinality($1)<1 THEN TRUE\nELSE FALSE\nEND;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_intersection(_a ANYARRAY, _b ANYARRAY) RETURNS ANYARRAY AS $$\n SELECT ARRAY ( SELECT unnest(_a) INTERSECT SELECT UNNEST(_b) );\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION array_map_ident(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_ident(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_map_literal(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_literal(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION array_reverse(anyarray) RETURNS anyarray AS $$\nSELECT ARRAY(\n SELECT $1[i]\n FROM generate_subscripts($1,1) AS s(i)\n ORDER BY i DESC\n);\n$$ LANGUAGE SQL STRICT IMMUTABLE;\n\nDROP TABLE IF EXISTS query_queue;\nCREATE TABLE query_queue (\n query text PRIMARY KEY,\n added timestamptz DEFAULT now()\n);\n\nDROP TABLE IF EXISTS query_queue_history;\nCREATE TABLE query_queue_history(\n query text,\n added timestamptz NOT NULL,\n finished timestamptz NOT NULL DEFAULT now(),\n error text\n);\n\nCREATE OR REPLACE PROCEDURE run_queued_queries() AS $$\nDECLARE\n qitem query_queue%ROWTYPE;\n timeout_ts timestamptz;\n error text;\n cnt int := 0;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n DELETE FROM query_queue WHERE query = (SELECT query FROM query_queue ORDER BY added DESC LIMIT 1 FOR UPDATE SKIP LOCKED) RETURNING * INTO qitem;\n IF NOT FOUND THEN\n EXIT;\n END IF;\n cnt := cnt + 1;\n BEGIN\n RAISE NOTICE 'RUNNING QUERY: %', qitem.query;\n EXECUTE qitem.query;\n EXCEPTION WHEN others THEN\n error := format('%s | %s', SQLERRM, SQLSTATE);\n END;\n INSERT INTO query_queue_history (query, added, finished, error)\n VALUES (qitem.query, qitem.added, clock_timestamp(), error);\n COMMIT;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION run_queued_queries_intransaction() RETURNS int AS $$\nDECLARE\n qitem query_queue%ROWTYPE;\n timeout_ts timestamptz;\n error text;\n cnt int := 0;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n DELETE FROM query_queue WHERE query = (SELECT query FROM query_queue ORDER BY added DESC LIMIT 1 FOR UPDATE SKIP LOCKED) RETURNING * INTO qitem;\n IF NOT FOUND THEN\n RETURN cnt;\n END IF;\n cnt := cnt + 1;\n BEGIN\n qitem.query := regexp_replace(qitem.query, 'CONCURRENTLY', '');\n RAISE NOTICE 'RUNNING QUERY: %', qitem.query;\n\n EXECUTE qitem.query;\n EXCEPTION WHEN others THEN\n error := format('%s | %s', SQLERRM, SQLSTATE);\n RAISE WARNING '%', error;\n END;\n INSERT INTO query_queue_history (query, added, finished, error)\n VALUES (qitem.query, qitem.added, clock_timestamp(), error);\n END LOOP;\n RETURN cnt;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION run_or_queue(query text) RETURNS VOID AS $$\nDECLARE\n use_queue text := COALESCE(get_setting('use_queue'), 'FALSE')::boolean;\nBEGIN\n IF get_setting_bool('debug') THEN\n RAISE NOTICE '%', query;\n END IF;\n IF use_queue THEN\n INSERT INTO query_queue (query) VALUES (query) ON CONFLICT DO NOTHING;\n ELSE\n EXECUTE query;\n END IF;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n\nDROP FUNCTION IF EXISTS check_pgstac_settings;\nCREATE OR REPLACE FUNCTION check_pgstac_settings(_sysmem text DEFAULT NULL) RETURNS VOID AS $$\nDECLARE\n settingval text;\n sysmem bigint := pg_size_bytes(_sysmem);\n effective_cache_size bigint := pg_size_bytes(current_setting('effective_cache_size', TRUE));\n shared_buffers bigint := pg_size_bytes(current_setting('shared_buffers', TRUE));\n work_mem bigint := pg_size_bytes(current_setting('work_mem', TRUE));\n max_connections int := current_setting('max_connections', TRUE);\n maintenance_work_mem bigint := pg_size_bytes(current_setting('maintenance_work_mem', TRUE));\n seq_page_cost float := current_setting('seq_page_cost', TRUE);\n random_page_cost float := current_setting('random_page_cost', TRUE);\n temp_buffers bigint := pg_size_bytes(current_setting('temp_buffers', TRUE));\n r record;\nBEGIN\n IF _sysmem IS NULL THEN\n RAISE NOTICE 'Call function with the size of your system memory `SELECT check_pgstac_settings(''4GB'')` to get pg system setting recommendations.';\n ELSE\n IF effective_cache_size < (sysmem * 0.5) THEN\n RAISE WARNING 'effective_cache_size of % is set low for a system with %. Recomended value between % and %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.5), pg_size_pretty(sysmem * 0.75);\n ELSIF effective_cache_size > (sysmem * 0.75) THEN\n RAISE WARNING 'effective_cache_size of % is set high for a system with %. Recomended value between % and %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.5), pg_size_pretty(sysmem * 0.75);\n ELSE\n RAISE NOTICE 'effective_cache_size of % is set appropriately for a system with %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem);\n END IF;\n\n IF shared_buffers < (sysmem * 0.2) THEN\n RAISE WARNING 'shared_buffers of % is set low for a system with %. Recomended value between % and %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.2), pg_size_pretty(sysmem * 0.3);\n ELSIF shared_buffers > (sysmem * 0.3) THEN\n RAISE WARNING 'shared_buffers of % is set high for a system with %. Recomended value between % and %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.2), pg_size_pretty(sysmem * 0.3);\n ELSE\n RAISE NOTICE 'shared_buffers of % is set appropriately for a system with %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem);\n END IF;\n shared_buffers = sysmem * 0.3;\n IF maintenance_work_mem < (sysmem * 0.2) THEN\n RAISE WARNING 'maintenance_work_mem of % is set low for shared_buffers of %. Recomended value between % and %', pg_size_pretty(maintenance_work_mem), pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers * 0.2), pg_size_pretty(shared_buffers * 0.3);\n ELSIF maintenance_work_mem > (shared_buffers * 0.3) THEN\n RAISE WARNING 'maintenance_work_mem of % is set high for shared_buffers of %. Recomended value between % and %', pg_size_pretty(maintenance_work_mem), pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers * 0.2), pg_size_pretty(shared_buffers * 0.3);\n ELSE\n RAISE NOTICE 'maintenance_work_mem of % is set appropriately for shared_buffers of %', pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers);\n END IF;\n\n IF work_mem * max_connections > shared_buffers THEN\n RAISE WARNING 'work_mem setting of % is set high for % max_connections please reduce work_mem to % or decrease max_connections to %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers/max_connections), floor(shared_buffers/work_mem);\n ELSIF work_mem * max_connections < (shared_buffers * 0.75) THEN\n RAISE WARNING 'work_mem setting of % is set low for % max_connections you may consider raising work_mem to % or increasing max_connections to %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers/max_connections), floor(shared_buffers/work_mem);\n ELSE\n RAISE NOTICE 'work_mem setting of % and max_connections of % are adequate for shared_buffers of %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers);\n END IF;\n\n IF random_page_cost / seq_page_cost != 1.1 THEN\n RAISE WARNING 'random_page_cost (%) /seq_page_cost (%) should be set to 1.1 for SSD. Change random_page_cost to %', random_page_cost, seq_page_cost, 1.1 * seq_page_cost;\n ELSE\n RAISE NOTICE 'random_page_cost and seq_page_cost set appropriately for SSD';\n END IF;\n\n IF temp_buffers < greatest(pg_size_bytes('128MB'),(maintenance_work_mem / 2)) THEN\n RAISE WARNING 'pgstac makes heavy use of temp tables, consider raising temp_buffers from % to %', pg_size_pretty(temp_buffers), greatest('128MB', pg_size_pretty((shared_buffers / 16)));\n END IF;\n END IF;\n\n RAISE NOTICE 'VALUES FOR PGSTAC VARIABLES';\n RAISE NOTICE 'These can be set either as GUC system variables or by setting in the pgstac_settings table.';\n\n FOR r IN SELECT name, get_setting(name) as setting, CASE WHEN current_setting(concat('pgstac.',name), TRUE) IS NOT NULL THEN concat('pgstac.',name, ' GUC') WHEN value IS NOT NULL THEN 'pgstac_settings table' ELSE 'Not Set' END as loc FROM pgstac_settings LOOP\n RAISE NOTICE '% is set to % from the %', r.name, r.setting, r.loc;\n END LOOP;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pg_cron';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider intalling pg_cron which can be used to automate tasks';\n ELSE\n RAISE NOTICE 'pg_cron % is installed', settingval;\n END IF;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pgstattuple';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider installing the pgstattuple extension which can be used to help maintain tables and indexes.';\n ELSE\n RAISE NOTICE 'pgstattuple % is installed', settingval;\n END IF;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pg_stat_statements';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider installing the pg_stat_statements extension which is very helpful for tracking the types of queries on the system';\n ELSE\n RAISE NOTICE 'pg_stat_statements % is installed', settingval;\n IF current_setting('pg_stat_statements.track_statements', TRUE) IS DISTINCT FROM 'all' THEN\n RAISE WARNING 'SET pg_stat_statements.track_statements TO ''all''; --In order to track statements within functions.';\n END IF;\n END IF;\n\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac, public SET CLIENT_MIN_MESSAGES TO NOTICE;\nCREATE OR REPLACE FUNCTION to_int(jsonb) RETURNS int AS $$\n SELECT floor(($1->>0)::float)::int;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION to_float(jsonb) RETURNS float AS $$\n SELECT ($1->>0)::float;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION to_tstz(jsonb) RETURNS timestamptz AS $$\n SELECT ($1->>0)::timestamptz;\n$$ LANGUAGE SQL IMMUTABLE STRICT SET TIME ZONE 'UTC' COST 5000 PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION to_text(jsonb) RETURNS text AS $$\n SELECT CASE WHEN jsonb_typeof($1) IN ('array','object') THEN $1::text ELSE $1->>0 END;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION to_text_array(jsonb) RETURNS text[] AS $$\n SELECT\n CASE jsonb_typeof($1)\n WHEN 'array' THEN ARRAY(SELECT jsonb_array_elements_text($1))\n ELSE ARRAY[$1->>0]\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION bbox_geom(_bbox jsonb) RETURNS geometry AS $$\nSELECT CASE jsonb_array_length(_bbox)\n WHEN 4 THEN\n ST_SetSRID(ST_MakeEnvelope(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float,\n (_bbox->>3)::float\n ),4326)\n WHEN 6 THEN\n ST_SetSRID(ST_3DMakeBox(\n ST_MakePoint(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float\n ),\n ST_MakePoint(\n (_bbox->>3)::float,\n (_bbox->>4)::float,\n (_bbox->>5)::float\n )\n ),4326)\n ELSE null END;\n;\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION geom_bbox(_geom geometry) RETURNS jsonb AS $$\n SELECT jsonb_build_array(\n st_xmin(_geom),\n st_ymin(_geom),\n st_xmax(_geom),\n st_ymax(_geom)\n );\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION flip_jsonb_array(j jsonb) RETURNS jsonb AS $$\n SELECT jsonb_agg(value) FROM (SELECT value FROM jsonb_array_elements(j) WITH ORDINALITY ORDER BY ordinality DESC) as t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION explode_dotpaths(j jsonb) RETURNS SETOF text[] AS $$\n SELECT string_to_array(p, '.') as e FROM jsonb_array_elements_text(j) p;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION explode_dotpaths_recurse(IN j jsonb) RETURNS SETOF text[] AS $$\n WITH RECURSIVE t AS (\n SELECT e FROM explode_dotpaths(j) e\n UNION ALL\n SELECT e[1:cardinality(e)-1]\n FROM t\n WHERE cardinality(e)>1\n ) SELECT e FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION jsonb_set_nested(j jsonb, path text[], val jsonb) RETURNS jsonb AS $$\nDECLARE\nBEGIN\n IF cardinality(path) > 1 THEN\n FOR i IN 1..(cardinality(path)-1) LOOP\n IF j #> path[:i] IS NULL THEN\n j := jsonb_set_lax(j, path[:i], '{}', TRUE);\n END IF;\n END LOOP;\n END IF;\n RETURN jsonb_set_lax(j, path, val, true);\n\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\n\n\nCREATE OR REPLACE FUNCTION jsonb_include(j jsonb, f jsonb) RETURNS jsonb AS $$\nDECLARE\n includes jsonb := f-> 'include';\n outj jsonb := '{}'::jsonb;\n path text[];\nBEGIN\n IF\n includes IS NULL\n OR jsonb_array_length(includes) = 0\n THEN\n RETURN j;\n ELSE\n includes := includes || (\n CASE WHEN j ? 'collection' THEN\n '[\"id\",\"collection\"]'\n ELSE\n '[\"id\"]'\n END)::jsonb;\n FOR path IN SELECT explode_dotpaths(includes) LOOP\n outj := jsonb_set_nested(outj, path, j #> path);\n END LOOP;\n END IF;\n RETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_exclude(j jsonb, f jsonb) RETURNS jsonb AS $$\nDECLARE\n excludes jsonb := f-> 'exclude';\n outj jsonb := j;\n path text[];\nBEGIN\n IF\n excludes IS NULL\n OR jsonb_array_length(excludes) = 0\n THEN\n RETURN j;\n ELSE\n FOR path IN SELECT explode_dotpaths(excludes) LOOP\n outj := outj #- path;\n END LOOP;\n END IF;\n RETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_fields(j jsonb, f jsonb DEFAULT '{\"fields\":[]}') RETURNS jsonb AS $$\n SELECT jsonb_exclude(jsonb_include(j, f), f);\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION merge_jsonb(_a jsonb, _b jsonb) RETURNS jsonb AS $$\n SELECT\n CASE\n WHEN _a = '\"𒍟※\"'::jsonb THEN NULL\n WHEN _a IS NULL OR jsonb_typeof(_a) = 'null' THEN _b\n WHEN jsonb_typeof(_a) = 'object' AND jsonb_typeof(_b) = 'object' THEN\n (\n SELECT\n jsonb_strip_nulls(\n jsonb_object_agg(\n key,\n merge_jsonb(a.value, b.value)\n )\n )\n FROM\n jsonb_each(coalesce(_a,'{}'::jsonb)) as a\n FULL JOIN\n jsonb_each(coalesce(_b,'{}'::jsonb)) as b\n USING (key)\n )\n WHEN\n jsonb_typeof(_a) = 'array'\n AND jsonb_typeof(_b) = 'array'\n AND jsonb_array_length(_a) = jsonb_array_length(_b)\n THEN\n (\n SELECT jsonb_agg(m) FROM\n ( SELECT\n merge_jsonb(\n jsonb_array_elements(_a),\n jsonb_array_elements(_b)\n ) as m\n ) as l\n )\n ELSE _a\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION strip_jsonb(_a jsonb, _b jsonb) RETURNS jsonb AS $$\n SELECT\n CASE\n\n WHEN (_a IS NULL OR jsonb_typeof(_a) = 'null') AND _b IS NOT NULL AND jsonb_typeof(_b) != 'null' THEN '\"𒍟※\"'::jsonb\n WHEN _b IS NULL OR jsonb_typeof(_a) = 'null' THEN _a\n WHEN _a = _b AND jsonb_typeof(_a) = 'object' THEN '{}'::jsonb\n WHEN _a = _b THEN NULL\n WHEN jsonb_typeof(_a) = 'object' AND jsonb_typeof(_b) = 'object' THEN\n (\n SELECT\n jsonb_strip_nulls(\n jsonb_object_agg(\n key,\n strip_jsonb(a.value, b.value)\n )\n )\n FROM\n jsonb_each(_a) as a\n FULL JOIN\n jsonb_each(_b) as b\n USING (key)\n )\n WHEN\n jsonb_typeof(_a) = 'array'\n AND jsonb_typeof(_b) = 'array'\n AND jsonb_array_length(_a) = jsonb_array_length(_b)\n THEN\n (\n SELECT jsonb_agg(m) FROM\n ( SELECT\n strip_jsonb(\n jsonb_array_elements(_a),\n jsonb_array_elements(_b)\n ) as m\n ) as l\n )\n ELSE _a\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION nullif_jsonbnullempty(j jsonb) RETURNS jsonb AS $$\n SELECT nullif(nullif(nullif(j,'null'::jsonb),'{}'::jsonb),'[]'::jsonb);\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION jsonb_array_unique(j jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(jsonb_agg(DISTINCT a)) v FROM jsonb_array_elements(j) a;\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_concat_ignorenull(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT coalesce(a,'[]'::jsonb) || coalesce(b,'[]'::jsonb);\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_least(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(least(nullif_jsonbnullempty(a), nullif_jsonbnullempty(b)));\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_greatest(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(greatest(a, b));\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION first_notnull_sfunc(anyelement, anyelement) RETURNS anyelement AS $$\n SELECT COALESCE($1,$2);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE AGGREGATE first_notnull(anyelement)(\n SFUNC = first_notnull_sfunc,\n STYPE = anyelement\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_array_unique_merge(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_concat_ignorenull,\n FINALFUNC = jsonb_array_unique\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_min(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_least\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_max(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_greatest\n);\n/* looks for a geometry in a stac item first from geometry and falling back to bbox */\nCREATE OR REPLACE FUNCTION stac_geom(value jsonb) RETURNS geometry AS $$\nSELECT\n CASE\n WHEN value ? 'intersects' THEN\n ST_GeomFromGeoJSON(value->>'intersects')\n WHEN value ? 'geometry' THEN\n ST_GeomFromGeoJSON(value->>'geometry')\n WHEN value ? 'bbox' THEN\n pgstac.bbox_geom(value->'bbox')\n ELSE NULL\n END as geometry\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION stac_daterange(\n value jsonb\n) RETURNS tstzrange AS $$\nDECLARE\n props jsonb := value;\n dt timestamptz;\n edt timestamptz;\nBEGIN\n IF props ? 'properties' THEN\n props := props->'properties';\n END IF;\n IF\n props ? 'start_datetime'\n AND props->>'start_datetime' IS NOT NULL\n AND props ? 'end_datetime'\n AND props->>'end_datetime' IS NOT NULL\n THEN\n dt := props->>'start_datetime';\n edt := props->>'end_datetime';\n IF dt > edt THEN\n RAISE EXCEPTION 'start_datetime must be < end_datetime';\n END IF;\n ELSE\n dt := props->>'datetime';\n edt := props->>'datetime';\n END IF;\n IF dt is NULL OR edt IS NULL THEN\n RAISE NOTICE 'DT: %, EDT: %', dt, edt;\n RAISE EXCEPTION 'Either datetime (%) or both start_datetime (%) and end_datetime (%) must be set.', props->>'datetime',props->>'start_datetime',props->>'end_datetime';\n END IF;\n RETURN tstzrange(dt, edt, '[]');\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_datetime(value jsonb) RETURNS timestamptz AS $$\n SELECT lower(stac_daterange(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_end_datetime(value jsonb) RETURNS timestamptz AS $$\n SELECT upper(stac_daterange(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE TABLE IF NOT EXISTS stac_extensions(\n url text PRIMARY KEY,\n content jsonb\n);\nCREATE OR REPLACE FUNCTION collection_base_item(content jsonb) RETURNS jsonb AS $$\n SELECT jsonb_build_object(\n 'type', 'Feature',\n 'stac_version', content->'stac_version',\n 'assets', content->'item_assets',\n 'collection', content->'id'\n );\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE TABLE IF NOT EXISTS collections (\n key bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,\n id text GENERATED ALWAYS AS (content->>'id') STORED UNIQUE NOT NULL,\n content JSONB NOT NULL,\n base_item jsonb GENERATED ALWAYS AS (pgstac.collection_base_item(content)) STORED,\n geometry geometry GENERATED ALWAYS AS (pgstac.collection_geom(content)) STORED,\n datetime timestamptz GENERATED ALWAYS AS (pgstac.collection_datetime(content)) STORED,\n end_datetime timestamptz GENERATED ALWAYS AS (pgstac.collection_enddatetime(content)) STORED,\n private jsonb,\n partition_trunc text CHECK (partition_trunc IN ('year', 'month'))\n);\n\nCREATE OR REPLACE FUNCTION collection_base_item(cid text) RETURNS jsonb AS $$\n SELECT pgstac.collection_base_item(content) FROM pgstac.collections WHERE id = cid LIMIT 1;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION create_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION update_collection(data jsonb) RETURNS VOID AS $$\nDECLARE\n out collections%ROWTYPE;\nBEGIN\n UPDATE collections SET content=data WHERE id = data->>'id' RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ON CONFLICT (id) DO\n UPDATE\n SET content=EXCLUDED.content\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION delete_collection(_id text) RETURNS VOID AS $$\nDECLARE\n out collections%ROWTYPE;\nBEGIN\n DELETE FROM collections WHERE id = _id RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION get_collection(id text) RETURNS jsonb AS $$\n SELECT content FROM collections\n WHERE id=$1\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION all_collections() RETURNS jsonb AS $$\n SELECT coalesce(jsonb_agg(content), '[]'::jsonb) FROM collections;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION collection_delete_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n collection_base_partition text := concat('_items_', OLD.key);\nBEGIN\n EXECUTE format($q$\n DELETE FROM partition_stats WHERE partition IN (\n SELECT partition FROM partition_sys_meta\n WHERE collection=%L\n );\n DROP TABLE IF EXISTS %I CASCADE;\n $q$,\n OLD.id,\n collection_base_partition\n );\n RETURN OLD;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER collection_delete_trigger BEFORE DELETE ON collections\nFOR EACH ROW EXECUTE FUNCTION collection_delete_trigger_func();\nCREATE OR REPLACE FUNCTION queryable_signature(n text, c text[]) RETURNS text AS $$\n SELECT concat(n, c);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TABLE queryables (\n id bigint GENERATED ALWAYS AS identity PRIMARY KEY,\n name text NOT NULL,\n collection_ids text[], -- used to determine what partitions to create indexes on\n definition jsonb,\n property_path text,\n property_wrapper text,\n property_index_type text\n);\nCREATE INDEX queryables_name_idx ON queryables (name);\nCREATE INDEX queryables_collection_idx ON queryables USING GIN (collection_ids);\nCREATE INDEX queryables_property_wrapper_idx ON queryables (property_wrapper);\n\nCREATE OR REPLACE FUNCTION pgstac.queryables_constraint_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n allcollections text[];\nBEGIN\n RAISE NOTICE 'Making sure that name/collection is unique for queryables %', NEW;\n IF NEW.collection_ids IS NOT NULL THEN\n IF EXISTS (\n SELECT 1\n FROM unnest(NEW.collection_ids) c\n LEFT JOIN\n collections\n ON (collections.id = c)\n WHERE collections.id IS NULL\n ) THEN\n RAISE foreign_key_violation USING MESSAGE = format(\n 'One or more collections in %s do not exist.', NEW.collection_ids\n );\n RETURN NULL;\n END IF;\n END IF;\n IF TG_OP = 'INSERT' THEN\n IF EXISTS (\n SELECT 1 FROM queryables q\n WHERE\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n )\n ) THEN\n RAISE unique_violation USING MESSAGE = format(\n 'There is already a queryable for %s for a collection in %s: %s',\n NEW.name,\n NEW.collection_ids,\n\t\t\t\t(SELECT json_agg(row_to_json(q)) FROM queryables q WHERE\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n ))\n );\n RETURN NULL;\n END IF;\n END IF;\n IF TG_OP = 'UPDATE' THEN\n IF EXISTS (\n SELECT 1 FROM queryables q\n WHERE\n q.id != NEW.id\n AND\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n )\n ) THEN\n RAISE unique_violation\n USING MESSAGE = format(\n 'There is already a queryable for %s for a collection in %s',\n NEW.name,\n NEW.collection_ids\n );\n RETURN NULL;\n END IF;\n END IF;\n\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER queryables_constraint_insert_trigger\nBEFORE INSERT ON queryables\nFOR EACH ROW EXECUTE PROCEDURE queryables_constraint_triggerfunc();\n\nCREATE TRIGGER queryables_constraint_update_trigger\nBEFORE UPDATE ON queryables\nFOR EACH ROW\nWHEN (NEW.name = OLD.name AND NEW.collection_ids IS DISTINCT FROM OLD.collection_ids)\nEXECUTE PROCEDURE queryables_constraint_triggerfunc();\n\n\nCREATE OR REPLACE FUNCTION array_to_path(arr text[]) RETURNS text AS $$\n SELECT string_agg(\n quote_literal(v),\n '->'\n ) FROM unnest(arr) v;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\n\n\nCREATE OR REPLACE FUNCTION queryable(\n IN dotpath text,\n OUT path text,\n OUT expression text,\n OUT wrapper text,\n OUT nulled_wrapper text\n) AS $$\nDECLARE\n q RECORD;\n path_elements text[];\nBEGIN\n dotpath := replace(dotpath, 'properties.', '');\n IF dotpath = 'start_datetime' THEN\n dotpath := 'datetime';\n END IF;\n IF dotpath IN ('id', 'geometry', 'datetime', 'end_datetime', 'collection') THEN\n path := dotpath;\n expression := dotpath;\n wrapper := NULL;\n RETURN;\n END IF;\n\n SELECT * INTO q FROM queryables\n WHERE\n name=dotpath\n OR name = 'properties.' || dotpath\n OR name = replace(dotpath, 'properties.', '')\n ;\n IF q.property_wrapper IS NULL THEN\n IF q.definition->>'type' = 'number' THEN\n wrapper := 'to_float';\n nulled_wrapper := wrapper;\n ELSIF q.definition->>'format' = 'date-time' THEN\n wrapper := 'to_tstz';\n nulled_wrapper := wrapper;\n ELSE\n nulled_wrapper := NULL;\n wrapper := 'to_text';\n END IF;\n ELSE\n wrapper := q.property_wrapper;\n nulled_wrapper := wrapper;\n END IF;\n IF q.property_path IS NOT NULL THEN\n path := q.property_path;\n ELSE\n path_elements := string_to_array(dotpath, '.');\n IF path_elements[1] IN ('links', 'assets', 'stac_version', 'stac_extensions') THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSIF path_elements[1] = 'properties' THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSE\n path := format($F$content->'properties'->%s$F$, array_to_path(path_elements));\n END IF;\n END IF;\n expression := format('%I(%s)', wrapper, path);\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\nCREATE OR REPLACE FUNCTION unnest_collection(collection_ids text[] DEFAULT NULL) RETURNS SETOF text AS $$\n DECLARE\n BEGIN\n IF collection_ids IS NULL THEN\n RETURN QUERY SELECT id FROM collections;\n END IF;\n RETURN QUERY SELECT unnest(collection_ids);\n END;\n$$ LANGUAGE PLPGSQL STABLE;\n\nCREATE OR REPLACE FUNCTION normalize_indexdef(def text) RETURNS text AS $$\nDECLARE\nBEGIN\n def := btrim(def, ' \\n\\t');\n\tdef := regexp_replace(def, '^CREATE (UNIQUE )?INDEX ([^ ]* )?ON (ONLY )?([^ ]* )?', '', 'i');\n RETURN def;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION indexdef(q queryables) RETURNS text AS $$\n DECLARE\n out text;\n BEGIN\n IF q.name = 'id' THEN\n out := 'CREATE UNIQUE INDEX ON %I USING btree (id)';\n ELSIF q.name = 'datetime' THEN\n out := 'CREATE INDEX ON %I USING btree (datetime DESC, end_datetime)';\n ELSIF q.name = 'geometry' THEN\n out := 'CREATE INDEX ON %I USING gist (geometry)';\n ELSE\n out := format($q$CREATE INDEX ON %%I USING %s (%s(((content -> 'properties'::text) -> %L::text)))$q$,\n lower(COALESCE(q.property_index_type, 'BTREE')),\n lower(COALESCE(q.property_wrapper, 'to_text')),\n q.name\n );\n END IF;\n RETURN btrim(out, ' \\n\\t');\n END;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nDROP VIEW IF EXISTS pgstac_indexes;\nCREATE VIEW pgstac_indexes AS\nSELECT\n i.schemaname,\n i.tablename,\n i.indexname,\n regexp_replace(btrim(replace(replace(indexdef, i.indexname, ''),'pgstac.',''),' \\t\\n'), '[ ]+', ' ', 'g') as idx,\n COALESCE(\n (regexp_match(indexdef, '\\(([a-zA-Z]+)\\)'))[1],\n (regexp_match(indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_-]+)''::text'))[1],\n CASE WHEN indexdef ~* '\\(datetime desc, end_datetime\\)' THEN 'datetime' ELSE NULL END\n ) AS field,\n pg_table_size(i.indexname::text) as index_size,\n pg_size_pretty(pg_table_size(i.indexname::text)) as index_size_pretty\nFROM\n pg_indexes i\nWHERE i.schemaname='pgstac' and i.tablename ~ '_items_' AND indexdef !~* ' only ';\n\nDROP VIEW IF EXISTS pgstac_index_stats;\nCREATE VIEW pgstac_indexes_stats AS\nSELECT\n i.schemaname,\n i.tablename,\n i.indexname,\n indexdef,\n COALESCE(\n (regexp_match(indexdef, '\\(([a-zA-Z]+)\\)'))[1],\n (regexp_match(indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_]+)''::text'))[1],\n CASE WHEN indexdef ~* '\\(datetime desc, end_datetime\\)' THEN 'datetime_end_datetime' ELSE NULL END\n ) AS field,\n pg_table_size(i.indexname::text) as index_size,\n pg_size_pretty(pg_table_size(i.indexname::text)) as index_size_pretty,\n n_distinct,\n most_common_vals::text::text[],\n most_common_freqs::text::text[],\n histogram_bounds::text::text[],\n correlation\nFROM\n pg_indexes i\n LEFT JOIN pg_stats s ON (s.tablename = i.indexname)\nWHERE i.schemaname='pgstac' and i.tablename ~ '_items_';\n\nCREATE OR REPLACE FUNCTION queryable_indexes(\n IN treeroot text DEFAULT 'items',\n IN changes boolean DEFAULT FALSE,\n OUT collection text,\n OUT partition text,\n OUT field text,\n OUT indexname text,\n OUT existing_idx text,\n OUT queryable_idx text\n) RETURNS SETOF RECORD AS $$\nWITH p AS (\n SELECT\n relid::text as partition,\n replace(replace(\n CASE\n WHEN parentrelid::regclass::text='items' THEN pg_get_expr(c.relpartbound, c.oid)\n ELSE pg_get_expr(parent.relpartbound, parent.oid)\n END,\n 'FOR VALUES IN (''',''), ''')',\n ''\n ) AS collection\n FROM pg_partition_tree(treeroot)\n JOIN pg_class c ON (relid::regclass = c.oid)\n JOIN pg_class parent ON (parentrelid::regclass = parent.oid AND isleaf)\n ), i AS (\n SELECT\n partition,\n indexname,\n regexp_replace(btrim(replace(replace(indexdef, indexname, ''),'pgstac.',''),' \\t\\n'), '[ ]+', ' ', 'g') as iidx,\n COALESCE(\n (regexp_match(indexdef, '\\(([a-zA-Z]+)\\)'))[1],\n (regexp_match(indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_-]+)''::text'))[1],\n CASE WHEN indexdef ~* '\\(datetime desc, end_datetime\\)' THEN 'datetime' ELSE NULL END\n ) AS field\n FROM\n pg_indexes\n JOIN p ON (tablename=partition)\n ), q AS (\n SELECT\n name AS field,\n collection,\n partition,\n format(indexdef(queryables), partition) as qidx\n FROM queryables, unnest_collection(queryables.collection_ids) collection\n JOIN p USING (collection)\n WHERE property_index_type IS NOT NULL OR name IN ('datetime','geometry','id')\n )\n SELECT\n collection,\n partition,\n field,\n indexname,\n iidx as existing_idx,\n qidx as queryable_idx\n FROM i FULL JOIN q USING (field, partition)\n WHERE CASE WHEN changes THEN lower(iidx) IS DISTINCT FROM lower(qidx) ELSE TRUE END;\n;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION maintain_index(\n indexname text,\n queryable_idx text,\n dropindexes boolean DEFAULT FALSE,\n rebuildindexes boolean DEFAULT FALSE,\n idxconcurrently boolean DEFAULT FALSE\n) RETURNS VOID AS $$\nDECLARE\nBEGIN\n IF indexname IS NOT NULL THEN\n IF dropindexes OR queryable_idx IS NOT NULL THEN\n EXECUTE format('DROP INDEX IF EXISTS %I;', indexname);\n ELSIF rebuildindexes THEN\n IF idxconcurrently THEN\n EXECUTE format('REINDEX INDEX CONCURRENTLY %I;', indexname);\n ELSE\n EXECUTE format('REINDEX INDEX CONCURRENTLY %I;', indexname);\n END IF;\n END IF;\n END IF;\n IF queryable_idx IS NOT NULL THEN\n IF idxconcurrently THEN\n EXECUTE replace(queryable_idx, 'INDEX', 'INDEX CONCURRENTLY');\n ELSE EXECUTE queryable_idx;\n END IF;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\n\nset check_function_bodies to off;\nCREATE OR REPLACE FUNCTION maintain_partition_queries(\n part text DEFAULT 'items',\n dropindexes boolean DEFAULT FALSE,\n rebuildindexes boolean DEFAULT FALSE,\n idxconcurrently boolean DEFAULT FALSE\n) RETURNS SETOF text AS $$\nDECLARE\n rec record;\n q text;\nBEGIN\n FOR rec IN (\n SELECT * FROM queryable_indexes(part,true)\n ) LOOP\n q := format(\n 'SELECT maintain_index(\n %L,%L,%L,%L,%L\n );',\n rec.indexname,\n rec.queryable_idx,\n dropindexes,\n rebuildindexes,\n idxconcurrently\n );\n RAISE NOTICE 'Q: %', q;\n RETURN NEXT q;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION maintain_partitions(\n part text DEFAULT 'items',\n dropindexes boolean DEFAULT FALSE,\n rebuildindexes boolean DEFAULT FALSE\n) RETURNS VOID AS $$\n WITH t AS (\n SELECT run_or_queue(q) FROM maintain_partition_queries(part, dropindexes, rebuildindexes) q\n ) SELECT count(*) FROM t;\n$$ LANGUAGE SQL;\n\n\nCREATE OR REPLACE FUNCTION queryables_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\nBEGIN\n PERFORM maintain_partitions();\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER queryables_trigger AFTER INSERT OR UPDATE ON queryables\nFOR EACH STATEMENT EXECUTE PROCEDURE queryables_trigger_func();\n\n\nCREATE OR REPLACE FUNCTION get_queryables(_collection_ids text[] DEFAULT NULL) RETURNS jsonb AS $$\nDECLARE\nBEGIN\n -- Build up queryables if the input contains valid collection ids or is empty\n IF EXISTS (\n SELECT 1 FROM collections\n WHERE\n _collection_ids IS NULL\n OR cardinality(_collection_ids) = 0\n OR id = ANY(_collection_ids)\n )\n THEN\n RETURN (\n WITH base AS (\n SELECT\n unnest(collection_ids) as collection_id,\n name,\n coalesce(definition, '{\"type\":\"string\"}'::jsonb) as definition\n FROM queryables\n WHERE\n _collection_ids IS NULL OR\n _collection_ids = '{}'::text[] OR\n _collection_ids && collection_ids\n UNION ALL\n SELECT null, name, coalesce(definition, '{\"type\":\"string\"}'::jsonb) as definition\n FROM queryables WHERE collection_ids IS NULL OR collection_ids = '{}'::text[]\n ), g AS (\n SELECT\n name,\n first_notnull(definition) as definition,\n jsonb_array_unique_merge(definition->'enum') as enum,\n jsonb_min(definition->'minimum') as minimum,\n jsonb_min(definition->'maxiumn') as maximum\n FROM base\n GROUP BY 1\n )\n SELECT\n jsonb_build_object(\n '$schema', 'http://json-schema.org/draft-07/schema#',\n '$id', '',\n 'type', 'object',\n 'title', 'STAC Queryables.',\n 'properties', jsonb_object_agg(\n name,\n definition\n ||\n jsonb_strip_nulls(jsonb_build_object(\n 'enum', enum,\n 'minimum', minimum,\n 'maximum', maximum\n ))\n ),\n 'additionalProperties', pgstac.additional_properties()\n )\n FROM g\n );\n ELSE\n RETURN NULL;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\nCREATE OR REPLACE FUNCTION get_queryables(_collection text DEFAULT NULL) RETURNS jsonb AS $$\n SELECT\n CASE\n WHEN _collection IS NULL THEN get_queryables(NULL::text[])\n ELSE get_queryables(ARRAY[_collection])\n END\n ;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_queryables() RETURNS jsonb AS $$\n SELECT get_queryables(NULL::text[]);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION schema_qualify_refs(url text, j jsonb) returns jsonb as $$\n SELECT regexp_replace(j::text, '\"\\$ref\": \"#', concat('\"$ref\": \"', url, '#'), 'g')::jsonb;\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE VIEW stac_extension_queryables AS\nSELECT DISTINCT key as name, schema_qualify_refs(e.url, j.value) as definition FROM stac_extensions e, jsonb_each(e.content->'definitions'->'fields'->'properties') j;\n\n\nCREATE OR REPLACE FUNCTION missing_queryables(_collection text, _tablesample float DEFAULT 5, minrows float DEFAULT 10) RETURNS TABLE(collection text, name text, definition jsonb, property_wrapper text) AS $$\nDECLARE\n q text;\n _partition text;\n explain_json json;\n psize float;\n estrows float;\nBEGIN\n SELECT format('_items_%s', key) INTO _partition FROM collections WHERE id=_collection;\n\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM %I;', _partition)\n INTO explain_json;\n psize := explain_json->0->'Plan'->'Plan Rows';\n estrows := _tablesample * .01 * psize;\n IF estrows < minrows THEN\n _tablesample := least(100,greatest(_tablesample, (estrows / psize) / 100));\n RAISE NOTICE '%', (psize / estrows) / 100;\n END IF;\n RAISE NOTICE 'Using tablesample % to find missing queryables from % % that has ~% rows estrows: %', _tablesample, _collection, _partition, psize, estrows;\n\n q := format(\n $q$\n WITH q AS (\n SELECT * FROM queryables\n WHERE\n collection_ids IS NULL\n OR %L = ANY(collection_ids)\n ), t AS (\n SELECT\n content->'properties' AS properties\n FROM\n %I\n TABLESAMPLE SYSTEM(%L)\n ), p AS (\n SELECT DISTINCT ON (key)\n key,\n value,\n s.definition\n FROM t\n JOIN LATERAL jsonb_each(properties) ON TRUE\n LEFT JOIN q ON (q.name=key)\n LEFT JOIN stac_extension_queryables s ON (s.name=key)\n WHERE q.definition IS NULL\n )\n SELECT\n %L,\n key,\n COALESCE(definition, jsonb_build_object('type',jsonb_typeof(value))) as definition,\n CASE\n WHEN definition->>'type' = 'integer' THEN 'to_int'\n WHEN COALESCE(definition->>'type', jsonb_typeof(value)) = 'number' THEN 'to_float'\n WHEN COALESCE(definition->>'type', jsonb_typeof(value)) = 'array' THEN 'to_text_array'\n ELSE 'to_text'\n END\n FROM p;\n $q$,\n _collection,\n _partition,\n _tablesample,\n _collection\n );\n RETURN QUERY EXECUTE q;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION missing_queryables(_tablesample float DEFAULT 5) RETURNS TABLE(collection_ids text[], name text, definition jsonb, property_wrapper text) AS $$\n SELECT\n array_agg(collection),\n name,\n definition,\n property_wrapper\n FROM\n collections\n JOIN LATERAL\n missing_queryables(id, _tablesample) c\n ON TRUE\n GROUP BY\n 2,3,4\n ORDER BY 2,1\n ;\n$$ LANGUAGE SQL;\nCREATE OR REPLACE FUNCTION parse_dtrange(\n _indate jsonb,\n relative_base timestamptz DEFAULT date_trunc('hour', CURRENT_TIMESTAMP)\n) RETURNS tstzrange AS $$\nDECLARE\n timestrs text[];\n s timestamptz;\n e timestamptz;\nBEGIN\n timestrs :=\n CASE\n WHEN _indate ? 'timestamp' THEN\n ARRAY[_indate->>'timestamp']\n WHEN _indate ? 'interval' THEN\n to_text_array(_indate->'interval')\n WHEN jsonb_typeof(_indate) = 'array' THEN\n to_text_array(_indate)\n ELSE\n regexp_split_to_array(\n _indate->>0,\n '/'\n )\n END;\n RAISE NOTICE 'TIMESTRS %', timestrs;\n IF cardinality(timestrs) = 1 THEN\n IF timestrs[1] ILIKE 'P%' THEN\n RETURN tstzrange(relative_base - upper(timestrs[1])::interval, relative_base, '[)');\n END IF;\n s := timestrs[1]::timestamptz;\n RETURN tstzrange(s, s, '[]');\n END IF;\n\n IF cardinality(timestrs) != 2 THEN\n RAISE EXCEPTION 'Timestamp cannot have more than 2 values';\n END IF;\n\n IF timestrs[1] = '..' OR timestrs[1] = '' THEN\n s := '-infinity'::timestamptz;\n e := timestrs[2]::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] = '..' OR timestrs[2] = '' THEN\n s := timestrs[1]::timestamptz;\n e := 'infinity'::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[1] ILIKE 'P%' AND timestrs[2] NOT ILIKE 'P%' THEN\n e := timestrs[2]::timestamptz;\n s := e - upper(timestrs[1])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] ILIKE 'P%' AND timestrs[1] NOT ILIKE 'P%' THEN\n s := timestrs[1]::timestamptz;\n e := s + upper(timestrs[2])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n s := timestrs[1]::timestamptz;\n e := timestrs[2]::timestamptz;\n\n RETURN tstzrange(s,e,'[)');\n\n RETURN NULL;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT PARALLEL SAFE SET TIME ZONE 'UTC';\n\nCREATE OR REPLACE FUNCTION parse_dtrange(\n _indate text,\n relative_base timestamptz DEFAULT CURRENT_TIMESTAMP\n) RETURNS tstzrange AS $$\n SELECT parse_dtrange(to_jsonb(_indate), relative_base);\n$$ LANGUAGE SQL STABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION temporal_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\n ll text := 'datetime';\n lh text := 'end_datetime';\n rrange tstzrange;\n rl text;\n rh text;\n outq text;\nBEGIN\n rrange := parse_dtrange(args->1);\n RAISE NOTICE 'Constructing temporal query OP: %, ARGS: %, RRANGE: %', op, args, rrange;\n op := lower(op);\n rl := format('%L::timestamptz', lower(rrange));\n rh := format('%L::timestamptz', upper(rrange));\n outq := CASE op\n WHEN 't_before' THEN 'lh < rl'\n WHEN 't_after' THEN 'll > rh'\n WHEN 't_meets' THEN 'lh = rl'\n WHEN 't_metby' THEN 'll = rh'\n WHEN 't_overlaps' THEN 'll < rl AND rl < lh < rh'\n WHEN 't_overlappedby' THEN 'rl < ll < rh AND lh > rh'\n WHEN 't_starts' THEN 'll = rl AND lh < rh'\n WHEN 't_startedby' THEN 'll = rl AND lh > rh'\n WHEN 't_during' THEN 'll > rl AND lh < rh'\n WHEN 't_contains' THEN 'll < rl AND lh > rh'\n WHEN 't_finishes' THEN 'll > rl AND lh = rh'\n WHEN 't_finishedby' THEN 'll < rl AND lh = rh'\n WHEN 't_equals' THEN 'll = rl AND lh = rh'\n WHEN 't_disjoint' THEN 'NOT (ll <= rh AND lh >= rl)'\n WHEN 't_intersects' THEN 'll <= rh AND lh >= rl'\n WHEN 'anyinteracts' THEN 'll <= rh AND lh >= rl'\n END;\n outq := regexp_replace(outq, '\\mll\\M', ll);\n outq := regexp_replace(outq, '\\mlh\\M', lh);\n outq := regexp_replace(outq, '\\mrl\\M', rl);\n outq := regexp_replace(outq, '\\mrh\\M', rh);\n outq := format('(%s)', outq);\n RETURN outq;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\n\n\nCREATE OR REPLACE FUNCTION spatial_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\n geom text;\n j jsonb := args->1;\nBEGIN\n op := lower(op);\n RAISE NOTICE 'Constructing spatial query OP: %, ARGS: %', op, args;\n IF op NOT IN ('s_equals','s_disjoint','s_touches','s_within','s_overlaps','s_crosses','s_intersects','intersects','s_contains') THEN\n RAISE EXCEPTION 'Spatial Operator % Not Supported', op;\n END IF;\n op := regexp_replace(op, '^s_', 'st_');\n IF op = 'intersects' THEN\n op := 'st_intersects';\n END IF;\n -- Convert geometry to WKB string\n IF j ? 'type' AND j ? 'coordinates' THEN\n geom := st_geomfromgeojson(j)::text;\n ELSIF jsonb_typeof(j) = 'array' THEN\n geom := bbox_geom(j)::text;\n END IF;\n\n RETURN format('%s(geometry, %L::geometry)', op, geom);\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION query_to_cql2(q jsonb) RETURNS jsonb AS $$\n-- Translates anything passed in through the deprecated \"query\" into equivalent CQL2\nWITH t AS (\n SELECT key as property, value as ops\n FROM jsonb_each(q)\n), t2 AS (\n SELECT property, (jsonb_each(ops)).*\n FROM t WHERE jsonb_typeof(ops) = 'object'\n UNION ALL\n SELECT property, 'eq', ops\n FROM t WHERE jsonb_typeof(ops) != 'object'\n)\nSELECT\n jsonb_strip_nulls(jsonb_build_object(\n 'op', 'and',\n 'args', jsonb_agg(\n jsonb_build_object(\n 'op', key,\n 'args', jsonb_build_array(\n jsonb_build_object('property',property),\n value\n )\n )\n )\n )\n) as qcql FROM t2\n;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\nCREATE OR REPLACE FUNCTION cql1_to_cql2(j jsonb) RETURNS jsonb AS $$\nDECLARE\n args jsonb;\n ret jsonb;\nBEGIN\n RAISE NOTICE 'CQL1_TO_CQL2: %', j;\n IF j ? 'filter' THEN\n RETURN cql1_to_cql2(j->'filter');\n END IF;\n IF j ? 'property' THEN\n RETURN j;\n END IF;\n IF jsonb_typeof(j) = 'array' THEN\n SELECT jsonb_agg(cql1_to_cql2(el)) INTO args FROM jsonb_array_elements(j) el;\n RETURN args;\n END IF;\n IF jsonb_typeof(j) = 'number' THEN\n RETURN j;\n END IF;\n IF jsonb_typeof(j) = 'string' THEN\n RETURN j;\n END IF;\n\n IF jsonb_typeof(j) = 'object' THEN\n SELECT jsonb_build_object(\n 'op', key,\n 'args', cql1_to_cql2(value)\n ) INTO ret\n FROM jsonb_each(j)\n WHERE j IS NOT NULL;\n RETURN ret;\n END IF;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE STRICT;\n\nCREATE TABLE cql2_ops (\n op text PRIMARY KEY,\n template text,\n types text[]\n);\nINSERT INTO cql2_ops (op, template, types) VALUES\n ('eq', '%s = %s', NULL),\n ('neq', '%s != %s', NULL),\n ('ne', '%s != %s', NULL),\n ('!=', '%s != %s', NULL),\n ('<>', '%s != %s', NULL),\n ('lt', '%s < %s', NULL),\n ('lte', '%s <= %s', NULL),\n ('gt', '%s > %s', NULL),\n ('gte', '%s >= %s', NULL),\n ('le', '%s <= %s', NULL),\n ('ge', '%s >= %s', NULL),\n ('=', '%s = %s', NULL),\n ('<', '%s < %s', NULL),\n ('<=', '%s <= %s', NULL),\n ('>', '%s > %s', NULL),\n ('>=', '%s >= %s', NULL),\n ('like', '%s LIKE %s', NULL),\n ('ilike', '%s ILIKE %s', NULL),\n ('+', '%s + %s', NULL),\n ('-', '%s - %s', NULL),\n ('*', '%s * %s', NULL),\n ('/', '%s / %s', NULL),\n ('not', 'NOT (%s)', NULL),\n ('between', '%s BETWEEN %s AND %s', NULL),\n ('isnull', '%s IS NULL', NULL),\n ('upper', 'upper(%s)', NULL),\n ('lower', 'lower(%s)', NULL),\n ('casei', 'upper(%s)', NULL),\n ('accenti', 'unaccent(%s)', NULL)\nON CONFLICT (op) DO UPDATE\n SET\n template = EXCLUDED.template\n;\n\n\nCREATE OR REPLACE FUNCTION cql2_query(j jsonb, wrapper text DEFAULT NULL) RETURNS text AS $$\n#variable_conflict use_variable\nDECLARE\n args jsonb := j->'args';\n arg jsonb;\n op text := lower(j->>'op');\n cql2op RECORD;\n literal text;\n _wrapper text;\n leftarg text;\n rightarg text;\n prop text;\n extra_props bool := pgstac.additional_properties();\nBEGIN\n IF j IS NULL OR (op IS NOT NULL AND args IS NULL) THEN\n RETURN NULL;\n END IF;\n RAISE NOTICE 'CQL2_QUERY: %', j;\n\n -- check if all properties are represented in the queryables\n IF NOT extra_props THEN\n FOR prop IN\n SELECT DISTINCT p->>0\n FROM jsonb_path_query(j, 'strict $.**.property') p\n WHERE p->>0 NOT IN ('id', 'datetime', 'geometry', 'end_datetime', 'collection')\n LOOP\n IF (queryable(prop)).nulled_wrapper IS NULL THEN\n RAISE EXCEPTION 'Term % is not found in queryables.', prop;\n END IF;\n END LOOP;\n END IF;\n\n IF j ? 'filter' THEN\n RETURN cql2_query(j->'filter');\n END IF;\n\n IF j ? 'upper' THEN\n RETURN cql2_query(jsonb_build_object('op', 'upper', 'args', j->'upper'));\n END IF;\n\n IF j ? 'lower' THEN\n RETURN cql2_query(jsonb_build_object('op', 'lower', 'args', j->'lower'));\n END IF;\n\n -- Temporal Query\n IF op ilike 't_%' or op = 'anyinteracts' THEN\n RETURN temporal_op_query(op, args);\n END IF;\n\n -- If property is a timestamp convert it to text to use with\n -- general operators\n IF j ? 'timestamp' THEN\n RETURN format('%L::timestamptz', to_tstz(j->'timestamp'));\n END IF;\n IF j ? 'interval' THEN\n RAISE EXCEPTION 'Please use temporal operators when using intervals.';\n RETURN NONE;\n END IF;\n\n -- Spatial Query\n IF op ilike 's_%' or op = 'intersects' THEN\n RETURN spatial_op_query(op, args);\n END IF;\n\n IF op IN ('a_equals','a_contains','a_contained_by','a_overlaps') THEN\n IF args->0 ? 'property' THEN\n leftarg := format('to_text_array(%s)', (queryable(args->0->>'property')).path);\n END IF;\n IF args->1 ? 'property' THEN\n rightarg := format('to_text_array(%s)', (queryable(args->1->>'property')).path);\n END IF;\n RETURN FORMAT(\n '%s %s %s',\n COALESCE(leftarg, quote_literal(to_text_array(args->0))),\n CASE op\n WHEN 'a_equals' THEN '='\n WHEN 'a_contains' THEN '@>'\n WHEN 'a_contained_by' THEN '<@'\n WHEN 'a_overlaps' THEN '&&'\n END,\n COALESCE(rightarg, quote_literal(to_text_array(args->1)))\n );\n END IF;\n\n IF op = 'in' THEN\n RAISE NOTICE 'IN : % % %', args, jsonb_build_array(args->0), args->1;\n args := jsonb_build_array(args->0) || (args->1);\n RAISE NOTICE 'IN2 : %', args;\n END IF;\n\n\n\n IF op = 'between' THEN\n args = jsonb_build_array(\n args->0,\n args->1,\n args->2\n );\n END IF;\n\n -- Make sure that args is an array and run cql2_query on\n -- each element of the array\n RAISE NOTICE 'ARGS PRE: %', args;\n IF j ? 'args' THEN\n IF jsonb_typeof(args) != 'array' THEN\n args := jsonb_build_array(args);\n END IF;\n\n IF jsonb_path_exists(args, '$[*] ? (@.property == \"id\" || @.property == \"datetime\" || @.property == \"end_datetime\" || @.property == \"collection\")') THEN\n wrapper := NULL;\n ELSE\n -- if any of the arguments are a property, try to get the property_wrapper\n FOR arg IN SELECT jsonb_path_query(args, '$[*] ? (@.property != null)') LOOP\n RAISE NOTICE 'Arg: %', arg;\n wrapper := (queryable(arg->>'property')).nulled_wrapper;\n RAISE NOTICE 'Property: %, Wrapper: %', arg, wrapper;\n IF wrapper IS NOT NULL THEN\n EXIT;\n END IF;\n END LOOP;\n\n -- if the property was not in queryables, see if any args were numbers\n IF\n wrapper IS NULL\n AND jsonb_path_exists(args, '$[*] ? (@.type()==\"number\")')\n THEN\n wrapper := 'to_float';\n END IF;\n wrapper := coalesce(wrapper, 'to_text');\n END IF;\n\n SELECT jsonb_agg(cql2_query(a, wrapper))\n INTO args\n FROM jsonb_array_elements(args) a;\n END IF;\n RAISE NOTICE 'ARGS: %', args;\n\n IF op IN ('and', 'or') THEN\n RETURN\n format(\n '(%s)',\n array_to_string(to_text_array(args), format(' %s ', upper(op)))\n );\n END IF;\n\n IF op = 'in' THEN\n RAISE NOTICE 'IN -- % %', args->0, to_text(args->0);\n RETURN format(\n '%s IN (%s)',\n to_text(args->0),\n array_to_string((to_text_array(args))[2:], ',')\n );\n END IF;\n\n -- Look up template from cql2_ops\n IF j ? 'op' THEN\n SELECT * INTO cql2op FROM cql2_ops WHERE cql2_ops.op ilike op;\n IF FOUND THEN\n -- If specific index set in queryables for a property cast other arguments to that type\n\n RETURN format(\n cql2op.template,\n VARIADIC (to_text_array(args))\n );\n ELSE\n RAISE EXCEPTION 'Operator % Not Supported.', op;\n END IF;\n END IF;\n\n\n IF wrapper IS NOT NULL THEN\n RAISE NOTICE 'Wrapping % with %', j, wrapper;\n IF j ? 'property' THEN\n RETURN format('%I(%s)', wrapper, (queryable(j->>'property')).path);\n ELSE\n RETURN format('%I(%L)', wrapper, j);\n END IF;\n ELSIF j ? 'property' THEN\n RETURN quote_ident(j->>'property');\n END IF;\n\n RETURN quote_literal(to_text(j));\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION paging_dtrange(\n j jsonb\n) RETURNS tstzrange AS $$\nDECLARE\n op text;\n filter jsonb := j->'filter';\n dtrange tstzrange := tstzrange('-infinity'::timestamptz,'infinity'::timestamptz);\n sdate timestamptz := '-infinity'::timestamptz;\n edate timestamptz := 'infinity'::timestamptz;\n jpitem jsonb;\nBEGIN\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n END IF;\n IF NOT (filter @? '$.**.op ? (@ == \"or\" || @ == \"not\")') THEN\n FOR jpitem IN SELECT j FROM jsonb_path_query(filter,'strict $.** ? (@.args[*].property == \"datetime\")'::jsonpath) j LOOP\n op := lower(jpitem->>'op');\n dtrange := parse_dtrange(jpitem->'args'->1);\n IF op IN ('<=', 'lt', 'lte', '<', 'le', 't_before') THEN\n sdate := greatest(sdate,'-infinity');\n edate := least(edate, upper(dtrange));\n ELSIF op IN ('>=', '>', 'gt', 'gte', 'ge', 't_after') THEN\n edate := least(edate, 'infinity');\n sdate := greatest(sdate, lower(dtrange));\n ELSIF op IN ('=', 'eq') THEN\n edate := least(edate, upper(dtrange));\n sdate := greatest(sdate, lower(dtrange));\n END IF;\n RAISE NOTICE '2 OP: %, ARGS: %, DTRANGE: %, SDATE: %, EDATE: %', op, jpitem->'args'->1, dtrange, sdate, edate;\n END LOOP;\n END IF;\n IF sdate > edate THEN\n RETURN 'empty'::tstzrange;\n END IF;\n RETURN tstzrange(sdate,edate, '[]');\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT SET TIME ZONE 'UTC';\n\nCREATE OR REPLACE FUNCTION paging_collections(\n IN j jsonb\n) RETURNS text[] AS $$\nDECLARE\n filter jsonb := j->'filter';\n jpitem jsonb;\n op text;\n args jsonb;\n arg jsonb;\n collections text[];\nBEGIN\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n END IF;\n IF NOT (filter @? '$.**.op ? (@ == \"or\" || @ == \"not\")') THEN\n FOR jpitem IN SELECT j FROM jsonb_path_query(filter,'strict $.** ? (@.args[*].property == \"collection\")'::jsonpath) j LOOP\n RAISE NOTICE 'JPITEM: %', jpitem;\n op := jpitem->>'op';\n args := jpitem->'args';\n IF op IN ('=', 'eq', 'in') THEN\n FOR arg IN SELECT a FROM jsonb_array_elements(args) a LOOP\n IF jsonb_typeof(arg) IN ('string', 'array') THEN\n RAISE NOTICE 'arg: %, collections: %', arg, collections;\n IF collections IS NULL OR collections = '{}'::text[] THEN\n collections := to_text_array(arg);\n ELSE\n collections := array_intersection(collections, to_text_array(arg));\n END IF;\n END IF;\n END LOOP;\n END IF;\n END LOOP;\n END IF;\n IF collections = '{}'::text[] THEN\n RETURN NULL;\n END IF;\n RETURN collections;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\nCREATE TABLE items (\n id text NOT NULL,\n geometry geometry NOT NULL,\n collection text NOT NULL,\n datetime timestamptz NOT NULL,\n end_datetime timestamptz NOT NULL,\n content JSONB NOT NULL,\n private jsonb\n)\nPARTITION BY LIST (collection)\n;\n\nCREATE INDEX \"datetime_idx\" ON items USING BTREE (datetime DESC, end_datetime ASC);\nCREATE INDEX \"geometry_idx\" ON items USING GIST (geometry);\n\nCREATE STATISTICS datetime_stats (dependencies) on datetime, end_datetime from items;\n\nALTER TABLE items ADD CONSTRAINT items_collections_fk FOREIGN KEY (collection) REFERENCES collections(id) ON DELETE CASCADE DEFERRABLE;\n\nCREATE OR REPLACE FUNCTION partition_after_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p text;\n t timestamptz := clock_timestamp();\nBEGIN\n RAISE NOTICE 'Updating partition stats %', t;\n FOR p IN SELECT DISTINCT partition\n FROM newdata n JOIN partition_sys_meta p\n ON (n.collection=p.collection AND n.datetime <@ p.partition_dtrange)\n LOOP\n PERFORM run_or_queue(format('SELECT update_partition_stats(%L, %L);', p, true));\n END LOOP;\n IF TG_OP IN ('DELETE','UPDATE') THEN\n DELETE FROM format_item_cache c USING newdata n WHERE c.collection = n.collection AND c.id = n.id;\n END IF;\n RAISE NOTICE 't: % %', t, clock_timestamp() - t;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE TRIGGER items_after_insert_trigger\nAFTER INSERT ON items\nREFERENCING NEW TABLE AS newdata\nFOR EACH STATEMENT\nEXECUTE FUNCTION partition_after_triggerfunc();\n\nCREATE TRIGGER items_after_update_trigger\nAFTER DELETE ON items\nREFERENCING OLD TABLE AS newdata\nFOR EACH STATEMENT\nEXECUTE FUNCTION partition_after_triggerfunc();\n\nCREATE TRIGGER items_after_delete_trigger\nAFTER UPDATE ON items\nREFERENCING NEW TABLE AS newdata\nFOR EACH STATEMENT\nEXECUTE FUNCTION partition_after_triggerfunc();\n\n\nCREATE OR REPLACE FUNCTION content_slim(_item jsonb) RETURNS jsonb AS $$\n SELECT strip_jsonb(_item - '{id,geometry,collection,type}'::text[], collection_base_item(_item->>'collection')) - '{id,geometry,collection,type}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_dehydrate(content jsonb) RETURNS items AS $$\n SELECT\n content->>'id' as id,\n stac_geom(content) as geometry,\n content->>'collection' as collection,\n stac_datetime(content) as datetime,\n stac_end_datetime(content) as end_datetime,\n content_slim(content) as content,\n null::jsonb as private\n ;\n$$ LANGUAGE SQL STABLE;\n\nCREATE OR REPLACE FUNCTION include_field(f text, fields jsonb DEFAULT '{}'::jsonb) RETURNS boolean AS $$\nDECLARE\n includes jsonb := fields->'include';\n excludes jsonb := fields->'exclude';\nBEGIN\n IF f IS NULL THEN\n RETURN NULL;\n END IF;\n\n\n IF\n jsonb_typeof(excludes) = 'array'\n AND jsonb_array_length(excludes)>0\n AND excludes ? f\n THEN\n RETURN FALSE;\n END IF;\n\n IF\n (\n jsonb_typeof(includes) = 'array'\n AND jsonb_array_length(includes) > 0\n AND includes ? f\n ) OR\n (\n includes IS NULL\n OR jsonb_typeof(includes) = 'null'\n OR jsonb_array_length(includes) = 0\n )\n THEN\n RETURN TRUE;\n END IF;\n\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nDROP FUNCTION IF EXISTS content_hydrate(jsonb, jsonb, jsonb);\nCREATE OR REPLACE FUNCTION content_hydrate(\n _item jsonb,\n _base_item jsonb,\n fields jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\n SELECT merge_jsonb(\n jsonb_fields(_item, fields),\n jsonb_fields(_base_item, fields)\n );\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION content_hydrate(_item items, _collection collections, fields jsonb DEFAULT '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\n content jsonb;\n base_item jsonb := _collection.base_item;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry, 20)::jsonb;\n END IF;\n output := content_hydrate(\n jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'collection', _item.collection,\n 'type', 'Feature'\n ) || _item.content,\n _collection.base_item,\n fields\n );\n\n RETURN output;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_nonhydrated(\n _item items,\n fields jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry, 20)::jsonb;\n END IF;\n output := jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'collection', _item.collection,\n 'type', 'Feature'\n ) || _item.content;\n RETURN output;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_hydrate(_item items, fields jsonb DEFAULT '{}'::jsonb) RETURNS jsonb AS $$\n SELECT content_hydrate(\n _item,\n (SELECT c FROM collections c WHERE id=_item.collection LIMIT 1),\n fields\n );\n$$ LANGUAGE SQL STABLE;\n\n\nCREATE UNLOGGED TABLE items_staging (\n content JSONB NOT NULL\n);\nCREATE UNLOGGED TABLE items_staging_ignore (\n content JSONB NOT NULL\n);\nCREATE UNLOGGED TABLE items_staging_upsert (\n content JSONB NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION items_staging_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p record;\n _partitions text[];\n part text;\n ts timestamptz := clock_timestamp();\n nrows int;\nBEGIN\n RAISE NOTICE 'Creating Partitions. %', clock_timestamp() - ts;\n\n FOR part IN WITH t AS (\n SELECT\n n.content->>'collection' as collection,\n stac_daterange(n.content->'properties') as dtr,\n partition_trunc\n FROM newdata n JOIN collections ON (n.content->>'collection'=collections.id)\n ), p AS (\n SELECT\n collection,\n COALESCE(date_trunc(partition_trunc::text, lower(dtr)),'-infinity') as d,\n tstzrange(min(lower(dtr)),max(lower(dtr)),'[]') as dtrange,\n tstzrange(min(upper(dtr)),max(upper(dtr)),'[]') as edtrange\n FROM t\n GROUP BY 1,2\n ) SELECT check_partition(collection, dtrange, edtrange) FROM p LOOP\n RAISE NOTICE 'Partition %', part;\n END LOOP;\n\n RAISE NOTICE 'Creating temp table with data to be added. %', clock_timestamp() - ts;\n DROP TABLE IF EXISTS tmpdata;\n CREATE TEMP TABLE tmpdata ON COMMIT DROP AS\n SELECT\n (content_dehydrate(content)).*\n FROM newdata;\n GET DIAGNOSTICS nrows = ROW_COUNT;\n RAISE NOTICE 'Added % rows to tmpdata. %', nrows, clock_timestamp() - ts;\n\n RAISE NOTICE 'Doing the insert. %', clock_timestamp() - ts;\n IF TG_TABLE_NAME = 'items_staging' THEN\n INSERT INTO items\n SELECT * FROM tmpdata;\n GET DIAGNOSTICS nrows = ROW_COUNT;\n RAISE NOTICE 'Inserted % rows to items. %', nrows, clock_timestamp() - ts;\n ELSIF TG_TABLE_NAME = 'items_staging_ignore' THEN\n INSERT INTO items\n SELECT * FROM tmpdata\n ON CONFLICT DO NOTHING;\n GET DIAGNOSTICS nrows = ROW_COUNT;\n RAISE NOTICE 'Inserted % rows to items. %', nrows, clock_timestamp() - ts;\n ELSIF TG_TABLE_NAME = 'items_staging_upsert' THEN\n DELETE FROM items i USING tmpdata s\n WHERE\n i.id = s.id\n AND i.collection = s.collection\n AND i IS DISTINCT FROM s\n ;\n GET DIAGNOSTICS nrows = ROW_COUNT;\n RAISE NOTICE 'Deleted % rows from items. %', nrows, clock_timestamp() - ts;\n INSERT INTO items AS t\n SELECT * FROM tmpdata\n ON CONFLICT DO NOTHING;\n GET DIAGNOSTICS nrows = ROW_COUNT;\n RAISE NOTICE 'Inserted % rows to items. %', nrows, clock_timestamp() - ts;\n END IF;\n\n RAISE NOTICE 'Deleting data from staging table. %', clock_timestamp() - ts;\n DELETE FROM items_staging;\n RAISE NOTICE 'Done. %', clock_timestamp() - ts;\n\n RETURN NULL;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER items_staging_insert_trigger AFTER INSERT ON items_staging REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\nCREATE TRIGGER items_staging_insert_ignore_trigger AFTER INSERT ON items_staging_ignore REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\nCREATE TRIGGER items_staging_insert_upsert_trigger AFTER INSERT ON items_staging_upsert REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\n\nCREATE OR REPLACE FUNCTION item_by_id(_id text, _collection text DEFAULT NULL) RETURNS items AS\n$$\nDECLARE\n i items%ROWTYPE;\nBEGIN\n SELECT * INTO i FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection) LIMIT 1;\n RETURN i;\nEND;\n$$ LANGUAGE PLPGSQL STABLE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION get_item(_id text, _collection text DEFAULT NULL) RETURNS jsonb AS $$\n SELECT content_hydrate(items) FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection);\n$$ LANGUAGE SQL STABLE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION delete_item(_id text, _collection text DEFAULT NULL) RETURNS VOID AS $$\nDECLARE\nout items%ROWTYPE;\nBEGIN\n DELETE FROM items WHERE id = _id AND (_collection IS NULL OR collection=_collection) RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n--/*\nCREATE OR REPLACE FUNCTION create_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION update_item(content jsonb) RETURNS VOID AS $$\nDECLARE\n old items %ROWTYPE;\n out items%ROWTYPE;\nBEGIN\n PERFORM delete_item(content->>'id', content->>'collection');\n PERFORM create_item(content);\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION create_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION collection_bbox(id text) RETURNS jsonb AS $$\n SELECT (replace(replace(replace(st_extent(geometry)::text,'BOX(','[['),')',']]'),' ',','))::jsonb\n FROM items WHERE collection=$1;\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION collection_temporal_extent(id text) RETURNS jsonb AS $$\n SELECT to_jsonb(array[array[min(datetime), max(datetime)]])\n FROM items WHERE collection=$1;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection_extents() RETURNS VOID AS $$\nUPDATE collections\n SET content = jsonb_set_lax(\n content,\n '{extent}'::text[],\n collection_extent(id, FALSE),\n true,\n 'use_json_null'\n )\n;\n$$ LANGUAGE SQL;\nCREATE TABLE partition_stats (\n partition text PRIMARY KEY,\n dtrange tstzrange,\n edtrange tstzrange,\n spatial geometry,\n last_updated timestamptz,\n keys text[]\n) WITH (FILLFACTOR=90);\n\nCREATE INDEX partitions_range_idx ON partition_stats USING GIST(dtrange);\n\n\nCREATE OR REPLACE FUNCTION constraint_tstzrange(expr text) RETURNS tstzrange AS $$\n WITH t AS (\n SELECT regexp_matches(\n expr,\n E'\\\\(''\\([0-9 :+-]*\\)''\\\\).*\\\\(''\\([0-9 :+-]*\\)''\\\\)'\n ) AS m\n ) SELECT tstzrange(m[1]::timestamptz, m[2]::timestamptz) FROM t\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE STRICT;\n\nCREATE OR REPLACE FUNCTION get_tstz_constraint(reloid oid, colname text) RETURNS tstzrange AS $$\nDECLARE\n expr text := NULL;\n m text[];\n ts_lower timestamptz := NULL;\n ts_upper timestamptz := NULL;\n lower_inclusive text := '[';\n upper_inclusive text := ']';\n ts timestamptz;\nBEGIN\n SELECT INTO expr\n string_agg(def, ' AND ')\n FROM pg_constraint JOIN LATERAL pg_get_constraintdef(oid) AS def ON TRUE\n WHERE\n conrelid = reloid\n AND contype = 'c'\n AND def LIKE '%' || colname || '%'\n ;\n\n IF expr IS NULL THEN\n RETURN NULL;\n END IF;\n\n RAISE DEBUG 'Constraint expression for % on %: %', colname, reloid::regclass, expr;\n -- collect all constraints for the specified column\n FOR m IN SELECT regexp_matches(expr, '[ (]' || colname || $expr$\\s*([<>=]{1,2})\\s*'([0-9 :.+\\-]+)'$expr$, 'g') LOOP\n ts := m[2]::timestamptz;\n IF m[1] IN ('>', '>=')\n THEN\n IF ts_lower IS NULL OR ts > ts_lower OR (ts = ts_lower AND m[1] = '>') THEN\n ts_lower := ts;\n lower_inclusive := CASE WHEN m[1] = '>' THEN '(' ELSE '[' END;\n END IF;\n ELSIF m[1] IN ('<', '<=')\n THEN\n IF ts_upper IS NULL OR ts < ts_upper OR (ts = ts_upper AND m[1] = '<') THEN\n ts_upper := ts;\n upper_inclusive := CASE WHEN m[1] = '<' THEN ')' ELSE ']' END;\n END IF;\n END IF;\n END LOOP;\n RAISE DEBUG 'Constraint % for %: % %', colname, reloid::regclass, ts_lower, ts_upper;\n RETURN tstzrange(ts_lower, ts_upper, lower_inclusive || upper_inclusive);\nEND;\n$$ LANGUAGE plpgsql STRICT STABLE;\n\nCREATE OR REPLACE FUNCTION get_partition_name(relid regclass) RETURNS text AS $$\n SELECT (parse_ident(relid::text))[cardinality(parse_ident(relid::text))];\n$$ LANGUAGE SQL STABLE STRICT;\n\nCREATE OR REPLACE VIEW partition_sys_meta AS\nSELECT\n partition,\n replace(\n replace(\n CASE WHEN level = 1 THEN partition_expr ELSE parent_partition_expr END,\n 'FOR VALUES IN (''',\n ''\n ),\n ''')',\n ''\n ) AS collection,\n level,\n c.reltuples,\n c.relhastriggers,\n partition_dtrange,\n COALESCE(\n get_tstz_constraint(c.oid, 'datetime'),\n partition_dtrange,\n inf_range\n ) as constraint_dtrange,\n COALESCE(\n get_tstz_constraint(c.oid, 'end_datetime'),\n inf_range\n ) as constraint_edtrange\nFROM\n pg_partition_tree('items')\n JOIN pg_class c ON (relid::regclass = c.oid)\n JOIN pg_class parent ON (parentrelid::regclass = parent.oid AND isleaf)\n LEFT JOIN pg_constraint edt ON (conrelid=c.oid AND contype='c')\n JOIN LATERAL get_partition_name(relid) AS partition ON TRUE\n JOIN LATERAL pg_get_expr(c.relpartbound, c.oid) as partition_expr ON TRUE\n JOIN LATERAL pg_get_expr(parent.relpartbound, parent.oid) as parent_partition_expr ON TRUE\n JOIN LATERAL tstzrange('-infinity', 'infinity','[]') as inf_range ON TRUE\n JOIN LATERAL COALESCE(constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), inf_range) as partition_dtrange ON TRUE\n JOIN LATERAL get_tstz_constraint(c.oid, 'datetime') as datetime_constraint ON TRUE\n JOIN LATERAL get_tstz_constraint(c.oid, 'end_datetime') as end_datetime_constraint ON TRUE\nWHERE isleaf\n;\n\nCREATE OR REPLACE VIEW partitions_view AS\nSELECT\n (parse_ident(relid::text))[cardinality(parse_ident(relid::text))] as partition,\n replace(\n replace(\n CASE WHEN level = 1 THEN partition_expr ELSE parent_partition_expr END,\n 'FOR VALUES IN (''',\n ''\n ),\n ''')',\n ''\n ) AS collection,\n level,\n c.reltuples,\n c.relhastriggers,\n partition_dtrange,\n COALESCE(\n get_tstz_constraint(c.oid, 'datetime'),\n partition_dtrange,\n inf_range\n ) as constraint_dtrange,\n COALESCE(\n get_tstz_constraint(c.oid, 'end_datetime'),\n inf_range\n ) as constraint_edtrange,\n dtrange,\n edtrange,\n spatial,\n last_updated\nFROM\n pg_partition_tree('items')\n JOIN pg_class c ON (relid::regclass = c.oid)\n JOIN pg_class parent ON (parentrelid::regclass = parent.oid AND isleaf)\n LEFT JOIN pg_constraint edt ON (conrelid=c.oid AND contype='c')\n JOIN LATERAL get_partition_name(relid) AS partition ON TRUE\n JOIN LATERAL pg_get_expr(c.relpartbound, c.oid) as partition_expr ON TRUE\n JOIN LATERAL pg_get_expr(parent.relpartbound, parent.oid) as parent_partition_expr ON TRUE\n JOIN LATERAL tstzrange('-infinity', 'infinity','[]') as inf_range ON TRUE\n JOIN LATERAL COALESCE(constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), inf_range) as partition_dtrange ON TRUE\n JOIN LATERAL get_tstz_constraint(c.oid, 'datetime') as datetime_constraint ON TRUE\n JOIN LATERAL get_tstz_constraint(c.oid, 'end_datetime') as end_datetime_constraint ON TRUE\n LEFT JOIN pgstac.partition_stats USING (partition)\nWHERE isleaf\n;\n\nCREATE MATERIALIZED VIEW partitions AS\nSELECT * FROM partitions_view;\nCREATE UNIQUE INDEX ON partitions (partition);\n\nCREATE MATERIALIZED VIEW partition_steps AS\nSELECT\n partition as name,\n date_trunc('month',lower(partition_dtrange)) as sdate,\n date_trunc('month', upper(partition_dtrange)) + '1 month'::interval as edate\n FROM partitions_view WHERE partition_dtrange IS NOT NULL AND partition_dtrange != 'empty'::tstzrange\n ORDER BY dtrange ASC\n;\n\n\nCREATE OR REPLACE FUNCTION update_partition_stats_q(_partition text, istrigger boolean default false) RETURNS VOID AS $$\nDECLARE\nBEGIN\n PERFORM run_or_queue(\n format('SELECT update_partition_stats(%L, %L);', _partition, istrigger)\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION update_partition_stats(_partition text, istrigger boolean default false) RETURNS VOID AS $$\nDECLARE\n dtrange tstzrange;\n edtrange tstzrange;\n cdtrange tstzrange;\n cedtrange tstzrange;\n extent geometry;\n collection text;\nBEGIN\n RAISE NOTICE 'Updating stats for %.', _partition;\n EXECUTE format(\n $q$\n SELECT\n tstzrange(min(datetime), max(datetime),'[]'),\n tstzrange(min(end_datetime), max(end_datetime), '[]')\n FROM %I\n $q$,\n _partition\n ) INTO dtrange, edtrange;\n EXECUTE format('ANALYZE %I;', _partition);\n extent := st_estimatedextent('pgstac', _partition, 'geometry');\n RAISE DEBUG 'Estimated Extent: %', extent;\n INSERT INTO partition_stats (partition, dtrange, edtrange, spatial, last_updated)\n SELECT _partition, dtrange, edtrange, extent, now()\n ON CONFLICT (partition) DO\n UPDATE SET\n dtrange=EXCLUDED.dtrange,\n edtrange=EXCLUDED.edtrange,\n spatial=EXCLUDED.spatial,\n last_updated=EXCLUDED.last_updated\n ;\n\n SELECT\n constraint_dtrange, constraint_edtrange, pv.collection\n INTO cdtrange, cedtrange, collection\n FROM partitions_view pv WHERE partition = _partition;\n\n RAISE NOTICE 'Checking if we need to modify constraints...';\n RAISE NOTICE 'cdtrange: % dtrange: % cedtrange: % edtrange: %',cdtrange, dtrange, cedtrange, edtrange;\n IF\n (cdtrange IS DISTINCT FROM dtrange OR edtrange IS DISTINCT FROM cedtrange)\n AND NOT istrigger\n THEN\n RAISE NOTICE 'Modifying Constraints';\n RAISE NOTICE 'Existing % %', cdtrange, cedtrange;\n RAISE NOTICE 'New % %', dtrange, edtrange;\n PERFORM drop_table_constraints(_partition);\n PERFORM create_table_constraints(_partition, dtrange, edtrange);\n END IF;\n REFRESH MATERIALIZED VIEW partitions;\n REFRESH MATERIALIZED VIEW partition_steps;\n RAISE NOTICE 'Checking if we need to update collection extents.';\n IF get_setting_bool('update_collection_extent') THEN\n RAISE NOTICE 'updating collection extent for %', collection;\n PERFORM run_or_queue(format($q$\n UPDATE collections\n SET content = jsonb_set_lax(\n content,\n '{extent}'::text[],\n collection_extent(%L, FALSE),\n true,\n 'use_json_null'\n ) WHERE id=%L\n ;\n $q$, collection, collection));\n ELSE\n RAISE NOTICE 'Not updating collection extent for %', collection;\n END IF;\n\nEND;\n$$ LANGUAGE PLPGSQL STRICT SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION partition_name( IN collection text, IN dt timestamptz, OUT partition_name text, OUT partition_range tstzrange) AS $$\nDECLARE\n c RECORD;\n parent_name text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n parent_name := format('_items_%s', c.key);\n\n\n IF c.partition_trunc = 'year' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYY'));\n ELSIF c.partition_trunc = 'month' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYYMM'));\n ELSE\n partition_name := parent_name;\n partition_range := tstzrange('-infinity'::timestamptz, 'infinity'::timestamptz, '[]');\n END IF;\n IF partition_range IS NULL THEN\n partition_range := tstzrange(\n date_trunc(c.partition_trunc::text, dt),\n date_trunc(c.partition_trunc::text, dt) + concat('1 ', c.partition_trunc)::interval\n );\n END IF;\n RETURN;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION drop_table_constraints(t text) RETURNS text AS $$\nDECLARE\n q text;\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM partitions_view WHERE partition=t) THEN\n RETURN NULL;\n END IF;\n FOR q IN SELECT FORMAT(\n $q$\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n $q$,\n t,\n conname\n ) FROM pg_constraint\n WHERE conrelid=t::regclass::oid AND contype='c'\n LOOP\n EXECUTE q;\n END LOOP;\n RETURN t;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE OR REPLACE FUNCTION create_table_constraints(t text, _dtrange tstzrange, _edtrange tstzrange) RETURNS text AS $$\nDECLARE\n q text;\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM partitions_view WHERE partition=t) THEN\n RETURN NULL;\n END IF;\n RAISE NOTICE 'Creating Table Constraints for % % %', t, _dtrange, _edtrange;\n IF _dtrange = 'empty' AND _edtrange = 'empty' THEN\n q :=format(\n $q$\n DO $block$\n BEGIN\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n ALTER TABLE %I\n ADD CONSTRAINT %I\n CHECK (((datetime IS NULL) AND (end_datetime IS NULL))) NOT VALID\n ;\n ALTER TABLE %I\n VALIDATE CONSTRAINT %I\n ;\n\n\n\n EXCEPTION WHEN others THEN\n RAISE WARNING '%%, Issue Altering Constraints. Please run update_partition_stats(%I)', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n $block$;\n $q$,\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t),\n t\n );\n ELSE\n q :=format(\n $q$\n DO $block$\n BEGIN\n\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n ALTER TABLE %I\n ADD CONSTRAINT %I\n CHECK (\n (datetime >= %L)\n AND (datetime <= %L)\n AND (end_datetime >= %L)\n AND (end_datetime <= %L)\n ) NOT VALID\n ;\n ALTER TABLE %I\n VALIDATE CONSTRAINT %I\n ;\n\n\n\n EXCEPTION WHEN others THEN\n RAISE WARNING '%%, Issue Altering Constraints. Please run update_partition_stats(%I)', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n $block$;\n $q$,\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t),\n lower(_dtrange),\n upper(_dtrange),\n lower(_edtrange),\n upper(_edtrange),\n t,\n format('%s_dt', t),\n t\n );\n END IF;\n PERFORM run_or_queue(q);\n RETURN t;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION check_partition(\n _collection text,\n _dtrange tstzrange,\n _edtrange tstzrange\n) RETURNS text AS $$\nDECLARE\n c RECORD;\n pm RECORD;\n _partition_name text;\n _partition_dtrange tstzrange;\n _constraint_dtrange tstzrange;\n _constraint_edtrange tstzrange;\n q text;\n deferrable_q text;\n err_context text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=_collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', _collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n\n IF c.partition_trunc IS NOT NULL THEN\n _partition_dtrange := tstzrange(\n date_trunc(c.partition_trunc, lower(_dtrange)),\n date_trunc(c.partition_trunc, lower(_dtrange)) + (concat('1 ', c.partition_trunc))::interval,\n '[)'\n );\n ELSE\n _partition_dtrange := '[-infinity, infinity]'::tstzrange;\n END IF;\n\n IF NOT _partition_dtrange @> _dtrange THEN\n RAISE EXCEPTION 'dtrange % is greater than the partition size % for collection %', _dtrange, c.partition_trunc, _collection;\n END IF;\n\n\n IF c.partition_trunc = 'year' THEN\n _partition_name := format('_items_%s_%s', c.key, to_char(lower(_partition_dtrange),'YYYY'));\n ELSIF c.partition_trunc = 'month' THEN\n _partition_name := format('_items_%s_%s', c.key, to_char(lower(_partition_dtrange),'YYYYMM'));\n ELSE\n _partition_name := format('_items_%s', c.key);\n END IF;\n\n SELECT * INTO pm FROM partition_sys_meta WHERE collection=_collection AND partition_dtrange @> _dtrange;\n IF FOUND THEN\n RAISE NOTICE '% % %', _edtrange, _dtrange, pm;\n _constraint_edtrange :=\n tstzrange(\n least(\n lower(_edtrange),\n nullif(lower(pm.constraint_edtrange), '-infinity')\n ),\n greatest(\n upper(_edtrange),\n nullif(upper(pm.constraint_edtrange), 'infinity')\n ),\n '[]'\n );\n _constraint_dtrange :=\n tstzrange(\n least(\n lower(_dtrange),\n nullif(lower(pm.constraint_dtrange), '-infinity')\n ),\n greatest(\n upper(_dtrange),\n nullif(upper(pm.constraint_dtrange), 'infinity')\n ),\n '[]'\n );\n\n IF pm.constraint_edtrange @> _edtrange AND pm.constraint_dtrange @> _dtrange THEN\n RETURN pm.partition;\n ELSE\n PERFORM drop_table_constraints(_partition_name);\n END IF;\n ELSE\n _constraint_edtrange := _edtrange;\n _constraint_dtrange := _dtrange;\n END IF;\n RAISE NOTICE 'EXISTING CONSTRAINTS % %, NEW % %', pm.constraint_dtrange, pm.constraint_edtrange, _constraint_dtrange, _constraint_edtrange;\n RAISE NOTICE 'Creating partition % %', _partition_name, _partition_dtrange;\n IF c.partition_trunc IS NULL THEN\n q := format(\n $q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n GRANT ALL ON %I to pgstac_ingest;\n $q$,\n _partition_name,\n _collection,\n concat(_partition_name,'_pk'),\n _partition_name,\n _partition_name\n );\n ELSE\n q := format(\n $q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L) PARTITION BY RANGE (datetime);\n CREATE TABLE IF NOT EXISTS %I partition OF %I FOR VALUES FROM (%L) TO (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n GRANT ALL ON %I TO pgstac_ingest;\n $q$,\n format('_items_%s', c.key),\n _collection,\n _partition_name,\n format('_items_%s', c.key),\n lower(_partition_dtrange),\n upper(_partition_dtrange),\n format('%s_pk', _partition_name),\n _partition_name,\n _partition_name\n );\n END IF;\n\n BEGIN\n EXECUTE q;\n EXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', _partition_name;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\n END;\n PERFORM maintain_partitions(_partition_name);\n PERFORM update_partition_stats_q(_partition_name, true);\n REFRESH MATERIALIZED VIEW partitions;\n REFRESH MATERIALIZED VIEW partition_steps;\n RETURN _partition_name;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION repartition(_collection text, _partition_trunc text, triggered boolean DEFAULT FALSE) RETURNS text AS $$\nDECLARE\n c RECORD;\n q text;\n from_trunc text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=_collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', _collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n IF triggered THEN\n RAISE NOTICE 'Converting % to % partitioning via Trigger', _collection, _partition_trunc;\n ELSE\n RAISE NOTICE 'Converting % from using % to % partitioning', _collection, c.partition_trunc, _partition_trunc;\n IF c.partition_trunc IS NOT DISTINCT FROM _partition_trunc THEN\n RAISE NOTICE 'Collection % already set to use partition by %', _collection, _partition_trunc;\n RETURN _collection;\n END IF;\n END IF;\n\n IF EXISTS (SELECT 1 FROM partitions_view WHERE collection=_collection LIMIT 1) THEN\n EXECUTE format(\n $q$\n CREATE TEMP TABLE changepartitionstaging ON COMMIT DROP AS SELECT * FROM %I;\n DROP TABLE IF EXISTS %I CASCADE;\n WITH p AS (\n SELECT\n collection,\n CASE\n WHEN %L IS NULL THEN '-infinity'::timestamptz\n ELSE date_trunc(%L, datetime)\n END as d,\n tstzrange(min(datetime),max(datetime),'[]') as dtrange,\n tstzrange(min(end_datetime),max(end_datetime),'[]') as edtrange\n FROM changepartitionstaging\n GROUP BY 1,2\n ) SELECT check_partition(collection, dtrange, edtrange) FROM p;\n INSERT INTO items SELECT * FROM changepartitionstaging;\n DROP TABLE changepartitionstaging;\n $q$,\n concat('_items_', c.key),\n concat('_items_', c.key),\n c.partition_trunc,\n c.partition_trunc\n );\n END IF;\n RETURN _collection;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE OR REPLACE FUNCTION collections_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n q text;\n partition_name text := format('_items_%s', NEW.key);\n partition_exists boolean := false;\n partition_empty boolean := true;\n err_context text;\n loadtemp boolean := FALSE;\nBEGIN\n RAISE NOTICE 'Collection Trigger. % %', NEW.id, NEW.key;\n IF TG_OP = 'UPDATE' AND NEW.partition_trunc IS DISTINCT FROM OLD.partition_trunc THEN\n PERFORM repartition(NEW.id, NEW.partition_trunc, TRUE);\n END IF;\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER collections_trigger AFTER\nINSERT\nOR\nUPDATE ON collections\nFOR EACH ROW EXECUTE FUNCTION collections_trigger_func();\n\nCREATE OR REPLACE FUNCTION chunker(\n IN _where text,\n OUT s timestamptz,\n OUT e timestamptz\n) RETURNS SETOF RECORD AS $$\nDECLARE\n explain jsonb;\nBEGIN\n IF _where IS NULL THEN\n _where := ' TRUE ';\n END IF;\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s;', _where)\n INTO explain;\n RAISE DEBUG 'EXPLAIN: %', explain;\n\n RETURN QUERY\n WITH t AS (\n SELECT j->>0 as p FROM\n jsonb_path_query(\n explain,\n 'strict $.**.\"Relation Name\" ? (@ != null)'\n ) j\n ),\n parts AS (\n SELECT sdate, edate FROM t JOIN partition_steps ON (t.p = name)\n ),\n times AS (\n SELECT sdate FROM parts\n UNION\n SELECT edate FROM parts\n ),\n uniq AS (\n SELECT DISTINCT sdate FROM times ORDER BY sdate\n ),\n last AS (\n SELECT sdate, lead(sdate, 1) over () as edate FROM uniq\n )\n SELECT sdate, edate FROM last WHERE edate IS NOT NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION partition_queries(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN partitions text[] DEFAULT NULL\n) RETURNS SETOF text AS $$\nDECLARE\n query text;\n sdate timestamptz;\n edate timestamptz;\nBEGIN\nIF _where IS NULL OR trim(_where) = '' THEN\n _where = ' TRUE ';\nEND IF;\nRAISE NOTICE 'Getting chunks for % %', _where, _orderby;\nIF _orderby ILIKE 'datetime d%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 DESC LOOP\n RETURN NEXT format($q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n $q$,\n sdate,\n edate,\n _where,\n _orderby\n );\n END LOOP;\nELSIF _orderby ILIKE 'datetime a%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 ASC LOOP\n RETURN NEXT format($q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n $q$,\n sdate,\n edate,\n _where,\n _orderby\n );\n END LOOP;\nELSE\n query := format($q$\n SELECT * FROM items\n WHERE %s\n ORDER BY %s\n $q$, _where, _orderby\n );\n\n RETURN NEXT query;\n RETURN;\nEND IF;\n\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION partition_query_view(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN _limit int DEFAULT 10\n) RETURNS text AS $$\n WITH p AS (\n SELECT * FROM partition_queries(_where, _orderby) p\n )\n SELECT\n CASE WHEN EXISTS (SELECT 1 FROM p) THEN\n (SELECT format($q$\n SELECT * FROM (\n %s\n ) total LIMIT %s\n $q$,\n string_agg(\n format($q$ SELECT * FROM ( %s ) AS sub $q$, p),\n '\n UNION ALL\n '\n ),\n _limit\n ))\n ELSE NULL\n END FROM p;\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION q_to_tsquery (jinput jsonb)\n RETURNS tsquery\n AS $$\nDECLARE\n input text;\n processed_text text;\n temp_text text;\n quote_array text[];\n placeholder text := '@QUOTE@';\nBEGIN\n IF jsonb_typeof(jinput) = 'string' THEN\n input := jinput->>0;\n ELSIF jsonb_typeof(jinput) = 'array' THEN\n input := array_to_string(\n array(select jsonb_array_elements_text(jinput)),\n ' OR '\n );\n ELSE\n RAISE EXCEPTION 'Input must be a string or an array of strings.';\n END IF;\n -- Extract all quoted phrases and store in array\n quote_array := regexp_matches(input, '\"[^\"]*\"', 'g');\n\n -- Replace each quoted part with a unique placeholder if there are any quoted phrases\n IF array_length(quote_array, 1) IS NOT NULL THEN\n processed_text := input;\n FOR i IN array_lower(quote_array, 1) .. array_upper(quote_array, 1) LOOP\n processed_text := replace(processed_text, quote_array[i], placeholder || i || placeholder);\n END LOOP;\n ELSE\n processed_text := input;\n END IF;\n\n -- Replace non-quoted text using regular expressions\n\n -- , -> |\n processed_text := regexp_replace(processed_text, ',(?=(?:[^\"]*\"[^\"]*\")*[^\"]*$)', ' | ', 'g');\n\n -- and -> &\n processed_text := regexp_replace(processed_text, '\\s+AND\\s+', ' & ', 'gi');\n\n -- or -> |\n processed_text := regexp_replace(processed_text, '\\s+OR\\s+', ' | ', 'gi');\n\n -- + ->\n processed_text := regexp_replace(processed_text, '^\\s*\\+([a-zA-Z0-9_]+)', '\\1', 'g'); -- +term at start\n processed_text := regexp_replace(processed_text, '\\s*\\+([a-zA-Z0-9_]+)', ' & \\1', 'g'); -- +term elsewhere\n\n -- - -> !\n processed_text := regexp_replace(processed_text, '^\\s*\\-([a-zA-Z0-9_]+)', '! \\1', 'g'); -- -term at start\n processed_text := regexp_replace(processed_text, '\\s*\\-([a-zA-Z0-9_]+)', ' & ! \\1', 'g'); -- -term elsewhere\n\n -- terms separated with spaces are assumed to represent adjacent terms. loop through these\n -- occurrences and replace them with the adjacency operator (<->)\n LOOP\n temp_text := regexp_replace(processed_text, '([a-zA-Z0-9_]+)\\s+([a-zA-Z0-9_]+)(?!\\s*[&|<>])', '\\1 <-> \\2', 'g');\n IF temp_text = processed_text THEN\n EXIT; -- No more replacements were made\n END IF;\n processed_text := temp_text;\n END LOOP;\n\n\n -- Replace placeholders back with quoted phrases if there were any\n IF array_length(quote_array, 1) IS NOT NULL THEN\n FOR i IN array_lower(quote_array, 1) .. array_upper(quote_array, 1) LOOP\n processed_text := replace(processed_text, placeholder || i || placeholder, '''' || substring(quote_array[i] from 2 for length(quote_array[i]) - 2) || '''');\n END LOOP;\n END IF;\n\n -- Print processed_text to the console for debugging purposes\n RAISE NOTICE 'processed_text: %', processed_text;\n\n RETURN to_tsquery('english', processed_text);\nEND;\n$$\nLANGUAGE plpgsql;\n\n\nCREATE OR REPLACE FUNCTION stac_search_to_where(j jsonb) RETURNS text AS $$\nDECLARE\n where_segments text[];\n _where text;\n dtrange tstzrange;\n collections text[];\n geom geometry;\n sdate timestamptz;\n edate timestamptz;\n filterlang text;\n filter jsonb := j->'filter';\n ft_query tsquery;\nBEGIN\n IF j ? 'ids' THEN\n where_segments := where_segments || format('id = ANY (%L) ', to_text_array(j->'ids'));\n END IF;\n\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n where_segments := where_segments || format('collection = ANY (%L) ', collections);\n END IF;\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n\n where_segments := where_segments || format(' datetime <= %L::timestamptz AND end_datetime >= %L::timestamptz ',\n edate,\n sdate\n );\n END IF;\n\n IF j ? 'q' THEN\n ft_query := q_to_tsquery(j->'q');\n where_segments := where_segments || format(\n $quote$\n (\n to_tsvector('english', content->'properties'->>'description') ||\n to_tsvector('english', coalesce(content->'properties'->>'title', '')) ||\n to_tsvector('english', coalesce(content->'properties'->>'keywords', ''))\n ) @@ %L\n $quote$,\n ft_query\n );\n END IF;\n\n geom := stac_geom(j);\n IF geom IS NOT NULL THEN\n where_segments := where_segments || format('st_intersects(geometry, %L)',geom);\n END IF;\n\n filterlang := COALESCE(\n j->>'filter-lang',\n get_setting('default_filter_lang', j->'conf')\n );\n IF NOT filter @? '$.**.op' THEN\n filterlang := 'cql-json';\n END IF;\n\n IF filterlang NOT IN ('cql-json','cql2-json') AND j ? 'filter' THEN\n RAISE EXCEPTION '% is not a supported filter-lang. Please use cql-json or cql2-json.', filterlang;\n END IF;\n\n IF j ? 'query' AND j ? 'filter' THEN\n RAISE EXCEPTION 'Can only use either query or filter at one time.';\n END IF;\n\n IF j ? 'query' THEN\n filter := query_to_cql2(j->'query');\n ELSIF filterlang = 'cql-json' THEN\n filter := cql1_to_cql2(filter);\n END IF;\n RAISE NOTICE 'FILTER: %', filter;\n where_segments := where_segments || cql2_query(filter);\n IF cardinality(where_segments) < 1 THEN\n RETURN ' TRUE ';\n END IF;\n\n _where := array_to_string(array_remove(where_segments, NULL), ' AND ');\n\n IF _where IS NULL OR BTRIM(_where) = '' THEN\n RETURN ' TRUE ';\n END IF;\n RETURN _where;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION parse_sort_dir(_dir text, reverse boolean default false) RETURNS text AS $$\n WITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n ) SELECT\n CASE\n WHEN NOT reverse THEN d\n WHEN d = 'ASC' THEN 'DESC'\n WHEN d = 'DESC' THEN 'ASC'\n END\n FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION sort_dir_to_op(_dir text, prev boolean default false) RETURNS text AS $$\n WITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n ) SELECT\n CASE\n WHEN d = 'ASC' AND prev THEN '<='\n WHEN d = 'DESC' AND prev THEN '>='\n WHEN d = 'ASC' THEN '>='\n WHEN d = 'DESC' THEN '<='\n END\n FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION sort_sqlorderby(\n _search jsonb DEFAULT NULL,\n reverse boolean DEFAULT FALSE\n) RETURNS text AS $$\n WITH sortby AS (\n SELECT coalesce(_search->'sortby','[{\"field\":\"datetime\", \"direction\":\"desc\"}]') as sort\n ), withid AS (\n SELECT CASE\n WHEN sort @? '$[*] ? (@.field == \"id\")' THEN sort\n ELSE sort || '[{\"field\":\"id\", \"direction\":\"desc\"}]'::jsonb\n END as sort\n FROM sortby\n ), withid_rows AS (\n SELECT jsonb_array_elements(sort) as value FROM withid\n ),sorts AS (\n SELECT\n coalesce(\n (queryable(value->>'field')).expression\n ) as key,\n parse_sort_dir(value->>'direction', reverse) as dir\n FROM withid_rows\n )\n SELECT array_to_string(\n array_agg(concat(key, ' ', dir)),\n ', '\n ) FROM sorts;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_sort_dir(sort_item jsonb) RETURNS text AS $$\n SELECT CASE WHEN sort_item->>'direction' ILIKE 'desc%' THEN 'DESC' ELSE 'ASC' END;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION get_token_val_str(\n _field text,\n _item items\n) RETURNS text AS $$\nDECLARE\n q text;\n literal text;\nBEGIN\n q := format($q$ SELECT quote_literal(%s) FROM (SELECT $1.*) as r;$q$, _field);\n EXECUTE q INTO literal USING _item;\n RETURN literal;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n\nCREATE OR REPLACE FUNCTION get_token_record(IN _token text, OUT prev BOOLEAN, OUT item items) RETURNS RECORD AS $$\nDECLARE\n _itemid text := _token;\n _collectionid text;\nBEGIN\n IF _token IS NULL THEN\n RETURN;\n END IF;\n RAISE NOTICE 'Looking for token: %', _token;\n prev := FALSE;\n IF _token ILIKE 'prev:%' THEN\n _itemid := replace(_token, 'prev:','');\n prev := TRUE;\n ELSIF _token ILIKE 'next:%' THEN\n _itemid := replace(_token, 'next:', '');\n END IF;\n SELECT id INTO _collectionid FROM collections WHERE _itemid LIKE concat(id,':%');\n IF FOUND THEN\n _itemid := replace(_itemid, concat(_collectionid,':'), '');\n SELECT * INTO item FROM items WHERE id=_itemid AND collection=_collectionid;\n ELSE\n SELECT * INTO item FROM items WHERE id=_itemid;\n END IF;\n IF item IS NULL THEN\n RAISE EXCEPTION 'Could not find item using token: % item: % collection: %', _token, _itemid, _collectionid;\n END IF;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\n\nCREATE OR REPLACE FUNCTION get_token_filter(\n _sortby jsonb DEFAULT '[{\"field\":\"datetime\",\"direction\":\"desc\"}]'::jsonb,\n token_item items DEFAULT NULL,\n prev boolean DEFAULT FALSE,\n inclusive boolean DEFAULT FALSE\n) RETURNS text AS $$\nDECLARE\n ltop text := '<';\n gtop text := '>';\n dir text;\n sort record;\n orfilter text := '';\n orfilters text[] := '{}'::text[];\n andfilters text[] := '{}'::text[];\n output text;\n token_where text;\nBEGIN\n IF _sortby IS NULL OR _sortby = '[]'::jsonb THEN\n _sortby := '[{\"field\":\"datetime\",\"direction\":\"desc\"}]'::jsonb;\n END IF;\n _sortby := _sortby || jsonb_build_object('field','id','direction',_sortby->0->>'direction');\n RAISE NOTICE 'Getting Token Filter. % %', _sortby, token_item;\n IF inclusive THEN\n orfilters := orfilters || format('( id=%L AND collection=%L )' , token_item.id, token_item.collection);\n END IF;\n\n FOR sort IN\n WITH s1 AS (\n SELECT\n _row,\n (queryable(value->>'field')).expression as _field,\n (value->>'field' = 'id') as _isid,\n get_sort_dir(value) as _dir\n FROM jsonb_array_elements(_sortby)\n WITH ORDINALITY AS t(value, _row)\n )\n SELECT\n _row,\n _field,\n _dir,\n get_token_val_str(_field, token_item) as _val\n FROM s1\n WHERE _row <= (SELECT min(_row) FROM s1 WHERE _isid)\n LOOP\n orfilter := NULL;\n RAISE NOTICE 'SORT: %', sort;\n IF sort._val IS NOT NULL AND ((prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC')) THEN\n orfilter := format($f$(\n (%s %s %s) OR (%s IS NULL)\n )$f$,\n sort._field,\n ltop,\n sort._val,\n sort._val\n );\n ELSIF sort._val IS NULL AND ((prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC')) THEN\n RAISE NOTICE '< but null';\n orfilter := format('%s IS NOT NULL', sort._field);\n ELSIF sort._val IS NULL THEN\n RAISE NOTICE '> but null';\n ELSE\n orfilter := format($f$(\n (%s %s %s) OR (%s IS NULL)\n )$f$,\n sort._field,\n gtop,\n sort._val,\n sort._field\n );\n END IF;\n RAISE NOTICE 'ORFILTER: %', orfilter;\n\n IF orfilter IS NOT NULL THEN\n IF sort._row = 1 THEN\n orfilters := orfilters || orfilter;\n ELSE\n orfilters := orfilters || format('(%s AND %s)', array_to_string(andfilters, ' AND '), orfilter);\n END IF;\n END IF;\n IF sort._val IS NOT NULL THEN\n andfilters := andfilters || format('%s = %s', sort._field, sort._val);\n ELSE\n andfilters := andfilters || format('%s IS NULL', sort._field);\n END IF;\n END LOOP;\n\n output := array_to_string(orfilters, ' OR ');\n\n token_where := concat('(',coalesce(output,'true'),')');\n IF trim(token_where) = '' THEN\n token_where := NULL;\n END IF;\n RAISE NOTICE 'TOKEN_WHERE: %',token_where;\n RETURN token_where;\n END;\n$$ LANGUAGE PLPGSQL SET transform_null_equals TO TRUE\n;\n\nCREATE OR REPLACE FUNCTION search_hash(jsonb, jsonb) RETURNS text AS $$\n SELECT md5(concat(($1 - '{token,limit,context,includes,excludes}'::text[])::text,$2::text));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\nDROP FUNCTION IF EXISTS search_tohash(jsonb);\n\nCREATE TABLE IF NOT EXISTS searches(\n hash text GENERATED ALWAYS AS (search_hash(search, metadata)) STORED PRIMARY KEY,\n search jsonb NOT NULL,\n _where text,\n orderby text,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n metadata jsonb DEFAULT '{}'::jsonb NOT NULL\n);\n\nCREATE TABLE IF NOT EXISTS search_wheres(\n id bigint generated always as identity primary key,\n _where text NOT NULL,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n statslastupdated timestamptz,\n estimated_count bigint,\n estimated_cost float,\n time_to_estimate float,\n total_count bigint,\n time_to_count float,\n partitions text[]\n);\n\nCREATE INDEX IF NOT EXISTS search_wheres_partitions ON search_wheres USING GIN (partitions);\nCREATE UNIQUE INDEX IF NOT EXISTS search_wheres_where ON search_wheres ((md5(_where)));\n\nCREATE OR REPLACE FUNCTION where_stats(\n inwhere text,\n updatestats boolean default false,\n conf jsonb default null\n) RETURNS search_wheres AS $$\nDECLARE\n t timestamptz;\n i interval;\n explain_json jsonb;\n partitions text[];\n sw search_wheres%ROWTYPE;\n inwhere_hash text := md5(inwhere);\n _context text := lower(context(conf));\n _stats_ttl interval := context_stats_ttl(conf);\n _estimated_cost_threshold float := context_estimated_cost(conf);\n _estimated_count_threshold int := context_estimated_count(conf);\n ro bool := pgstac.readonly(conf);\nBEGIN\n -- If updatestats is true then set ttl to 0\n IF updatestats THEN\n RAISE DEBUG 'Updatestats set to TRUE, setting TTL to 0';\n _stats_ttl := '0'::interval;\n END IF;\n\n -- If we don't need to calculate context, just return\n IF _context = 'off' THEN\n sw._where = inwhere;\n RETURN sw;\n END IF;\n\n -- Get any stats that we have.\n IF NOT ro THEN\n -- If there is a lock where another process is\n -- updating the stats, wait so that we don't end up calculating a bunch of times.\n SELECT * INTO sw FROM search_wheres WHERE md5(_where)=inwhere_hash FOR UPDATE;\n ELSE\n SELECT * INTO sw FROM search_wheres WHERE md5(_where)=inwhere_hash;\n END IF;\n\n -- If there is a cached row, figure out if we need to update\n IF\n sw IS NOT NULL\n AND sw.statslastupdated IS NOT NULL\n AND sw.total_count IS NOT NULL\n AND now() - sw.statslastupdated <= _stats_ttl\n THEN\n -- we have a cached row with data that is within our ttl\n RAISE DEBUG 'Stats present in table and lastupdated within ttl: %', sw;\n IF NOT ro THEN\n RAISE DEBUG 'Updating search_wheres only bumping lastused and usecount';\n UPDATE search_wheres SET\n lastused = now(),\n usecount = search_wheres.usecount + 1\n WHERE md5(_where) = inwhere_hash\n RETURNING * INTO sw;\n END IF;\n RAISE DEBUG 'Returning cached counts. %', sw;\n RETURN sw;\n END IF;\n\n -- Calculate estimated cost and rows\n -- Use explain to get estimated count/cost\n IF sw.estimated_count IS NULL OR sw.estimated_cost IS NULL THEN\n RAISE DEBUG 'Calculating estimated stats';\n t := clock_timestamp();\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s', inwhere)\n INTO explain_json;\n RAISE DEBUG 'Time for just the explain: %', clock_timestamp() - t;\n i := clock_timestamp() - t;\n\n sw.estimated_count := explain_json->0->'Plan'->'Plan Rows';\n sw.estimated_cost := explain_json->0->'Plan'->'Total Cost';\n sw.time_to_estimate := extract(epoch from i);\n END IF;\n\n RAISE DEBUG 'ESTIMATED_COUNT: %, THRESHOLD %', sw.estimated_count, _estimated_count_threshold;\n RAISE DEBUG 'ESTIMATED_COST: %, THRESHOLD %', sw.estimated_cost, _estimated_cost_threshold;\n\n -- If context is set to auto and the costs are within the threshold return the estimated costs\n IF\n _context = 'auto'\n AND sw.estimated_count >= _estimated_count_threshold\n AND sw.estimated_cost >= _estimated_cost_threshold\n THEN\n IF NOT ro THEN\n INSERT INTO search_wheres (\n _where,\n lastused,\n usecount,\n statslastupdated,\n estimated_count,\n estimated_cost,\n time_to_estimate,\n total_count,\n time_to_count\n ) VALUES (\n inwhere,\n now(),\n 1,\n now(),\n sw.estimated_count,\n sw.estimated_cost,\n sw.time_to_estimate,\n null,\n null\n ) ON CONFLICT ((md5(_where)))\n DO UPDATE SET\n lastused = EXCLUDED.lastused,\n usecount = search_wheres.usecount + 1,\n statslastupdated = EXCLUDED.statslastupdated,\n estimated_count = EXCLUDED.estimated_count,\n estimated_cost = EXCLUDED.estimated_cost,\n time_to_estimate = EXCLUDED.time_to_estimate,\n total_count = EXCLUDED.total_count,\n time_to_count = EXCLUDED.time_to_count\n RETURNING * INTO sw;\n END IF;\n RAISE DEBUG 'Estimates are within thresholds, returning estimates. %', sw;\n RETURN sw;\n END IF;\n\n -- Calculate Actual Count\n t := clock_timestamp();\n RAISE NOTICE 'Calculating actual count...';\n EXECUTE format(\n 'SELECT count(*) FROM items WHERE %s',\n inwhere\n ) INTO sw.total_count;\n i := clock_timestamp() - t;\n RAISE NOTICE 'Actual Count: % -- %', sw.total_count, i;\n sw.time_to_count := extract(epoch FROM i);\n\n IF NOT ro THEN\n INSERT INTO search_wheres (\n _where,\n lastused,\n usecount,\n statslastupdated,\n estimated_count,\n estimated_cost,\n time_to_estimate,\n total_count,\n time_to_count\n ) VALUES (\n inwhere,\n now(),\n 1,\n now(),\n sw.estimated_count,\n sw.estimated_cost,\n sw.time_to_estimate,\n sw.total_count,\n sw.time_to_count\n ) ON CONFLICT ((md5(_where)))\n DO UPDATE SET\n lastused = EXCLUDED.lastused,\n usecount = search_wheres.usecount + 1,\n statslastupdated = EXCLUDED.statslastupdated,\n estimated_count = EXCLUDED.estimated_count,\n estimated_cost = EXCLUDED.estimated_cost,\n time_to_estimate = EXCLUDED.time_to_estimate,\n total_count = EXCLUDED.total_count,\n time_to_count = EXCLUDED.time_to_count\n RETURNING * INTO sw;\n END IF;\n RAISE DEBUG 'Returning with actual count. %', sw;\n RETURN sw;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION search_query(\n _search jsonb = '{}'::jsonb,\n updatestats boolean = false,\n _metadata jsonb = '{}'::jsonb\n) RETURNS searches AS $$\nDECLARE\n search searches%ROWTYPE;\n cached_search searches%ROWTYPE;\n pexplain jsonb;\n t timestamptz;\n i interval;\n doupdate boolean := FALSE;\n insertfound boolean := FALSE;\n ro boolean := pgstac.readonly();\n found_search text;\nBEGIN\n RAISE NOTICE 'SEARCH: %', _search;\n -- Calculate hash, where clause, and order by statement\n search.search := _search;\n search.metadata := _metadata;\n search.hash := search_hash(_search, _metadata);\n search._where := stac_search_to_where(_search);\n search.orderby := sort_sqlorderby(_search);\n search.lastused := now();\n search.usecount := 1;\n\n -- If we are in read only mode, directly return search\n IF ro THEN\n RETURN search;\n END IF;\n\n RAISE NOTICE 'Updating Statistics for search: %s', search;\n -- Update statistics for times used and and when last used\n -- If the entry is locked, rather than waiting, skip updating the stats\n INSERT INTO searches (search, lastused, usecount, metadata)\n VALUES (search.search, now(), 1, search.metadata)\n ON CONFLICT DO NOTHING\n RETURNING * INTO cached_search\n ;\n\n IF NOT FOUND OR cached_search IS NULL THEN\n UPDATE searches SET\n lastused = now(),\n usecount = searches.usecount + 1\n WHERE hash = (\n SELECT hash FROM searches WHERE hash=search.hash FOR UPDATE SKIP LOCKED\n )\n RETURNING * INTO cached_search\n ;\n END IF;\n\n IF cached_search IS NOT NULL THEN\n cached_search._where = search._where;\n cached_search.orderby = search.orderby;\n RETURN cached_search;\n END IF;\n RETURN search;\n\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE OR REPLACE FUNCTION search_fromhash(\n _hash text\n) RETURNS searches AS $$\n SELECT * FROM search_query((SELECT search FROM searches WHERE hash=_hash LIMIT 1));\n$$ LANGUAGE SQL STRICT;\n\nCREATE OR REPLACE FUNCTION search_rows(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN partitions text[] DEFAULT NULL,\n IN _limit int DEFAULT 10\n) RETURNS SETOF items AS $$\nDECLARE\n base_query text;\n query text;\n sdate timestamptz;\n edate timestamptz;\n n int;\n records_left int := _limit;\n timer timestamptz := clock_timestamp();\n full_timer timestamptz := clock_timestamp();\nBEGIN\nIF _where IS NULL OR trim(_where) = '' THEN\n _where = ' TRUE ';\nEND IF;\nRAISE NOTICE 'Getting chunks for % %', _where, _orderby;\n\nbase_query := $q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n LIMIT %L\n$q$;\n\nIF _orderby ILIKE 'datetime d%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 DESC LOOP\n RAISE NOTICE 'Running Query for % to %. %', sdate, edate, age_ms(full_timer);\n query := format(\n base_query,\n sdate,\n edate,\n _where,\n _orderby,\n records_left\n );\n RAISE DEBUG 'QUERY: %', query;\n timer := clock_timestamp();\n RETURN QUERY EXECUTE query;\n\n GET DIAGNOSTICS n = ROW_COUNT;\n records_left := records_left - n;\n RAISE NOTICE 'Returned %/% Rows From % to %. % to go. Time: %ms', n, _limit, sdate, edate, records_left, age_ms(timer);\n timer := clock_timestamp();\n IF records_left <= 0 THEN\n RAISE NOTICE 'SEARCH_ROWS TOOK %ms', age_ms(full_timer);\n RETURN;\n END IF;\n END LOOP;\nELSIF _orderby ILIKE 'datetime a%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 ASC LOOP\n RAISE NOTICE 'Running Query for % to %. %', sdate, edate, age_ms(full_timer);\n query := format(\n base_query,\n sdate,\n edate,\n _where,\n _orderby,\n records_left\n );\n RAISE DEBUG 'QUERY: %', query;\n timer := clock_timestamp();\n RETURN QUERY EXECUTE query;\n\n GET DIAGNOSTICS n = ROW_COUNT;\n records_left := records_left - n;\n RAISE NOTICE 'Returned %/% Rows From % to %. % to go. Time: %ms', n, _limit, sdate, edate, records_left, age_ms(timer);\n timer := clock_timestamp();\n IF records_left <= 0 THEN\n RAISE NOTICE 'SEARCH_ROWS TOOK %ms', age_ms(full_timer);\n RETURN;\n END IF;\n END LOOP;\nELSE\n query := format($q$\n SELECT * FROM items\n WHERE %s\n ORDER BY %s\n LIMIT %L\n $q$, _where, _orderby, _limit\n );\n RAISE DEBUG 'QUERY: %', query;\n timer := clock_timestamp();\n RETURN QUERY EXECUTE query;\n RAISE NOTICE 'FULL QUERY TOOK %ms', age_ms(timer);\nEND IF;\nRAISE NOTICE 'SEARCH_ROWS TOOK %ms', age_ms(full_timer);\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE UNLOGGED TABLE format_item_cache(\n id text,\n collection text,\n fields text,\n hydrated bool,\n output jsonb,\n lastused timestamptz DEFAULT now(),\n usecount int DEFAULT 1,\n timetoformat float,\n PRIMARY KEY (collection, id, fields, hydrated)\n);\nCREATE INDEX ON format_item_cache (lastused);\n\nCREATE OR REPLACE FUNCTION format_item(_item items, _fields jsonb DEFAULT '{}', _hydrated bool DEFAULT TRUE) RETURNS jsonb AS $$\nDECLARE\n cache bool := get_setting_bool('format_cache');\n _output jsonb := null;\n t timestamptz := clock_timestamp();\nBEGIN\n IF cache THEN\n SELECT output INTO _output FROM format_item_cache\n WHERE id=_item.id AND collection=_item.collection AND fields=_fields::text AND hydrated=_hydrated;\n END IF;\n IF _output IS NULL THEN\n IF _hydrated THEN\n _output := content_hydrate(_item, _fields);\n ELSE\n _output := content_nonhydrated(_item, _fields);\n END IF;\n END IF;\n IF cache THEN\n INSERT INTO format_item_cache (id, collection, fields, hydrated, output, timetoformat)\n VALUES (_item.id, _item.collection, _fields::text, _hydrated, _output, age_ms(t))\n ON CONFLICT(collection, id, fields, hydrated) DO\n UPDATE\n SET lastused=now(), usecount = format_item_cache.usecount + 1\n ;\n END IF;\n RETURN _output;\n\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION search(_search jsonb = '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n searches searches%ROWTYPE;\n _where text;\n orderby text;\n search_where search_wheres%ROWTYPE;\n total_count bigint;\n token record;\n token_prev boolean;\n token_item items%ROWTYPE;\n token_where text;\n full_where text;\n init_ts timestamptz := clock_timestamp();\n timer timestamptz := clock_timestamp();\n hydrate bool := NOT (_search->'conf'->>'nohydrate' IS NOT NULL AND (_search->'conf'->>'nohydrate')::boolean = true);\n prev text;\n next text;\n context jsonb;\n collection jsonb;\n out_records jsonb;\n out_len int;\n _limit int := coalesce((_search->>'limit')::int, 10);\n _querylimit int;\n _fields jsonb := coalesce(_search->'fields', '{}'::jsonb);\n has_prev boolean := FALSE;\n has_next boolean := FALSE;\n links jsonb := '[]'::jsonb;\n base_url text:= concat(rtrim(base_url(_search->'conf'),'/'));\nBEGIN\n searches := search_query(_search);\n _where := searches._where;\n orderby := searches.orderby;\n search_where := where_stats(_where);\n total_count := coalesce(search_where.total_count, search_where.estimated_count);\n RAISE NOTICE 'SEARCH:TOKEN: %', _search->>'token';\n token := get_token_record(_search->>'token');\n RAISE NOTICE '***TOKEN: %', token;\n _querylimit := _limit + 1;\n IF token IS NOT NULL THEN\n token_prev := token.prev;\n token_item := token.item;\n token_where := get_token_filter(_search->'sortby', token_item, token_prev, FALSE);\n RAISE DEBUG 'TOKEN_WHERE: % (%ms from search start)', token_where, age_ms(timer);\n IF token_prev THEN -- if we are using a prev token, we know has_next is true\n RAISE DEBUG 'There is a previous token, so automatically setting has_next to true';\n has_next := TRUE;\n orderby := sort_sqlorderby(_search, TRUE);\n ELSE\n RAISE DEBUG 'There is a next token, so automatically setting has_prev to true';\n has_prev := TRUE;\n\n END IF;\n ELSE -- if there was no token, we know there is no prev\n RAISE DEBUG 'There is no token, so we know there is no prev. setting has_prev to false';\n has_prev := FALSE;\n END IF;\n\n full_where := concat_ws(' AND ', _where, token_where);\n RAISE NOTICE 'FULL WHERE CLAUSE: %', full_where;\n RAISE NOTICE 'Time to get counts and build query %', age_ms(timer);\n timer := clock_timestamp();\n\n IF hydrate THEN\n RAISE NOTICE 'Getting hydrated data.';\n ELSE\n RAISE NOTICE 'Getting non-hydrated data.';\n END IF;\n RAISE NOTICE 'CACHE SET TO %', get_setting_bool('format_cache');\n RAISE NOTICE 'Time to set hydration/formatting %', age_ms(timer);\n timer := clock_timestamp();\n SELECT jsonb_agg(format_item(i, _fields, hydrate)) INTO out_records\n FROM search_rows(\n full_where,\n orderby,\n search_where.partitions,\n _querylimit\n ) as i;\n\n RAISE NOTICE 'Time to fetch rows %', age_ms(timer);\n timer := clock_timestamp();\n\n\n IF token_prev THEN\n out_records := flip_jsonb_array(out_records);\n END IF;\n\n RAISE NOTICE 'Query returned % records.', jsonb_array_length(out_records);\n RAISE DEBUG 'TOKEN: % %', token_item.id, token_item.collection;\n RAISE DEBUG 'RECORD_1: % %', out_records->0->>'id', out_records->0->>'collection';\n RAISE DEBUG 'RECORD-1: % %', out_records->-1->>'id', out_records->-1->>'collection';\n\n -- REMOVE records that were from our token\n IF out_records->0->>'id' = token_item.id AND out_records->0->>'collection' = token_item.collection THEN\n out_records := out_records - 0;\n ELSIF out_records->-1->>'id' = token_item.id AND out_records->-1->>'collection' = token_item.collection THEN\n out_records := out_records - -1;\n END IF;\n\n out_len := jsonb_array_length(out_records);\n\n IF out_len = _limit + 1 THEN\n IF token_prev THEN\n has_prev := TRUE;\n out_records := out_records - 0;\n ELSE\n has_next := TRUE;\n out_records := out_records - -1;\n END IF;\n END IF;\n\n\n links := links || jsonb_build_object(\n 'rel', 'root',\n 'type', 'application/json',\n 'href', base_url\n ) || jsonb_build_object(\n 'rel', 'self',\n 'type', 'application/json',\n 'href', concat(base_url, '/search')\n );\n\n IF has_next THEN\n next := concat(out_records->-1->>'collection', ':', out_records->-1->>'id');\n RAISE NOTICE 'HAS NEXT | %', next;\n links := links || jsonb_build_object(\n 'rel', 'next',\n 'type', 'application/geo+json',\n 'method', 'GET',\n 'href', concat(base_url, '/search?token=next:', next)\n );\n END IF;\n\n IF has_prev THEN\n prev := concat(out_records->0->>'collection', ':', out_records->0->>'id');\n RAISE NOTICE 'HAS PREV | %', prev;\n links := links || jsonb_build_object(\n 'rel', 'prev',\n 'type', 'application/geo+json',\n 'method', 'GET',\n 'href', concat(base_url, '/search?token=prev:', prev)\n );\n END IF;\n\n RAISE NOTICE 'Time to get prev/next %', age_ms(timer);\n timer := clock_timestamp();\n\n\n collection := jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb),\n 'links', links\n );\n\n\n\n IF context(_search->'conf') != 'off' THEN\n collection := collection || jsonb_strip_nulls(jsonb_build_object(\n 'numberMatched', total_count,\n 'numberReturned', coalesce(jsonb_array_length(out_records), 0)\n ));\n ELSE\n collection := collection || jsonb_strip_nulls(jsonb_build_object(\n 'numberReturned', coalesce(jsonb_array_length(out_records), 0)\n ));\n END IF;\n\n IF get_setting_bool('timing', _search->'conf') THEN\n collection = collection || jsonb_build_object('timing', age_ms(init_ts));\n END IF;\n\n RAISE NOTICE 'Time to build final json %', age_ms(timer);\n timer := clock_timestamp();\n\n RAISE NOTICE 'Total Time: %', age_ms(current_timestamp);\n RAISE NOTICE 'RETURNING % records. NEXT: %. PREV: %', collection->>'numberReturned', collection->>'next', collection->>'prev';\n RETURN collection;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION search_cursor(_search jsonb = '{}'::jsonb) RETURNS refcursor AS $$\nDECLARE\n curs refcursor;\n searches searches%ROWTYPE;\n _where text;\n _orderby text;\n q text;\n\nBEGIN\n searches := search_query(_search);\n _where := searches._where;\n _orderby := searches.orderby;\n\n OPEN curs FOR\n WITH p AS (\n SELECT * FROM partition_queries(_where, _orderby) p\n )\n SELECT\n CASE WHEN EXISTS (SELECT 1 FROM p) THEN\n (SELECT format($q$\n SELECT * FROM (\n %s\n ) total\n $q$,\n string_agg(\n format($q$ SELECT * FROM ( %s ) AS sub $q$, p),\n '\n UNION ALL\n '\n )\n ))\n ELSE NULL\n END FROM p;\n RETURN curs;\nEND;\n$$ LANGUAGE PLPGSQL;\nCREATE OR REPLACE VIEW collections_asitems AS\nSELECT\n id,\n geometry,\n 'collections' AS collection,\n datetime,\n end_datetime,\n jsonb_build_object(\n 'properties', content - '{links,assets,stac_version,stac_extensions}',\n 'links', content->'links',\n 'assets', content->'assets',\n 'stac_version', content->'stac_version',\n 'stac_extensions', content->'stac_extensions'\n ) AS content,\n content as collectionjson\nFROM collections;\n\n\nCREATE OR REPLACE FUNCTION collection_search_matched(\n IN _search jsonb DEFAULT '{}'::jsonb,\n OUT matched bigint\n) RETURNS bigint AS $$\nDECLARE\n _where text := stac_search_to_where(_search);\nBEGIN\n EXECUTE format(\n $query$\n SELECT\n count(*)\n FROM\n collections_asitems\n WHERE %s\n ;\n $query$,\n _where\n ) INTO matched;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION collection_search_rows(\n _search jsonb DEFAULT '{}'::jsonb\n) RETURNS SETOF jsonb AS $$\nDECLARE\n _where text := stac_search_to_where(_search);\n _limit int := coalesce((_search->>'limit')::int, 10);\n _fields jsonb := coalesce(_search->'fields', '{}'::jsonb);\n _orderby text;\n _offset int := COALESCE((_search->>'offset')::int, 0);\nBEGIN\n _orderby := sort_sqlorderby(\n jsonb_build_object(\n 'sortby',\n coalesce(\n _search->'sortby',\n '[{\"field\": \"id\", \"direction\": \"asc\"}]'::jsonb\n )\n )\n );\n RETURN QUERY EXECUTE format(\n $query$\n SELECT\n jsonb_fields(collectionjson, %L) as c\n FROM\n collections_asitems\n WHERE %s\n ORDER BY %s\n LIMIT %L\n OFFSET %L\n ;\n $query$,\n _fields,\n _where,\n _orderby,\n _limit,\n _offset\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION collection_search(\n _search jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\nDECLARE\n out_records jsonb;\n number_matched bigint := collection_search_matched(_search);\n number_returned bigint;\n _limit int := coalesce((_search->>'limit')::float::int, 10);\n _offset int := coalesce((_search->>'offset')::float::int, 0);\n links jsonb := '[]';\n ret jsonb;\n base_url text:= concat(rtrim(base_url(_search->'conf'),'/'), '/collections');\n prevoffset int;\n nextoffset int;\nBEGIN\n SELECT\n coalesce(jsonb_agg(c), '[]')\n INTO out_records\n FROM collection_search_rows(_search) c;\n\n number_returned := jsonb_array_length(out_records);\n RAISE DEBUG 'nm: %, nr: %, l:%, o:%', number_matched, number_returned, _limit, _offset;\n\n\n\n IF _limit <= number_matched AND number_matched > 0 THEN --need to have paging links\n nextoffset := least(_offset + _limit, number_matched - 1);\n prevoffset := greatest(_offset - _limit, 0);\n\n IF _offset > 0 THEN\n links := links || jsonb_build_object(\n 'rel', 'prev',\n 'type', 'application/json',\n 'method', 'GET' ,\n 'href', base_url,\n 'body', jsonb_build_object('offset', prevoffset),\n 'merge', TRUE\n );\n END IF;\n\n IF (_offset + _limit < number_matched) THEN\n links := links || jsonb_build_object(\n 'rel', 'next',\n 'type', 'application/json',\n 'method', 'GET' ,\n 'href', base_url,\n 'body', jsonb_build_object('offset', nextoffset),\n 'merge', TRUE\n );\n END IF;\n\n END IF;\n\n ret := jsonb_build_object(\n 'collections', out_records,\n 'numberMatched', number_matched,\n 'numberReturned', number_returned,\n 'links', links\n );\n RETURN ret;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\nSET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION tileenvelope(zoom int, x int, y int) RETURNS geometry AS $$\nWITH t AS (\n SELECT\n 20037508.3427892 as merc_max,\n -20037508.3427892 as merc_min,\n (2 * 20037508.3427892) / (2 ^ zoom) as tile_size\n)\nSELECT st_makeenvelope(\n merc_min + (tile_size * x),\n merc_max - (tile_size * (y + 1)),\n merc_min + (tile_size * (x + 1)),\n merc_max - (tile_size * y),\n 3857\n) FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;DROP FUNCTION IF EXISTS mercgrid;\n\n\nCREATE OR REPLACE FUNCTION ftime() RETURNS interval as $$\nSELECT age(clock_timestamp(), transaction_timestamp());\n$$ LANGUAGE SQL;\nSET SEARCH_PATH to pgstac, public;\n\nDROP FUNCTION IF EXISTS geometrysearch;\nCREATE OR REPLACE FUNCTION geometrysearch(\n IN geom geometry,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE, -- Return as soon as the passed in geometry is full covered\n IN skipcovered boolean DEFAULT TRUE -- Skip any items that would show up completely under the previous items\n) RETURNS jsonb AS $$\nDECLARE\n search searches%ROWTYPE;\n curs refcursor;\n _where text;\n query text;\n iter_record items%ROWTYPE;\n out_records jsonb := '{}'::jsonb[];\n exit_flag boolean := FALSE;\n counter int := 1;\n scancounter int := 1;\n remaining_limit int := _scanlimit;\n tilearea float;\n unionedgeom geometry;\n clippedgeom geometry;\n unionedgeom_area float := 0;\n prev_area float := 0;\n excludes text[];\n includes text[];\n\nBEGIN\n DROP TABLE IF EXISTS pgstac_results;\n CREATE TEMP TABLE pgstac_results (content jsonb) ON COMMIT DROP;\n\n -- If the passed in geometry is not an area set exitwhenfull and skipcovered to false\n IF ST_GeometryType(geom) !~* 'polygon' THEN\n RAISE NOTICE 'GEOMETRY IS NOT AN AREA';\n skipcovered = FALSE;\n exitwhenfull = FALSE;\n END IF;\n\n -- If skipcovered is true then you will always want to exit when the passed in geometry is full\n IF skipcovered THEN\n exitwhenfull := TRUE;\n END IF;\n\n search := search_fromhash(queryhash);\n\n IF search IS NULL THEN\n RAISE EXCEPTION 'Search with Query Hash % Not Found', queryhash;\n END IF;\n\n tilearea := st_area(geom);\n _where := format('%s AND st_intersects(geometry, %L::geometry)', search._where, geom);\n\n\n FOR query IN SELECT * FROM partition_queries(_where, search.orderby) LOOP\n query := format('%s LIMIT %L', query, remaining_limit);\n RAISE NOTICE '%', query;\n OPEN curs FOR EXECUTE query;\n LOOP\n FETCH curs INTO iter_record;\n EXIT WHEN NOT FOUND;\n IF exitwhenfull OR skipcovered THEN -- If we are not using exitwhenfull or skipcovered, we do not need to do expensive geometry operations\n clippedgeom := st_intersection(geom, iter_record.geometry);\n\n IF unionedgeom IS NULL THEN\n unionedgeom := clippedgeom;\n ELSE\n unionedgeom := st_union(unionedgeom, clippedgeom);\n END IF;\n\n unionedgeom_area := st_area(unionedgeom);\n\n IF skipcovered AND prev_area = unionedgeom_area THEN\n scancounter := scancounter + 1;\n CONTINUE;\n END IF;\n\n prev_area := unionedgeom_area;\n\n RAISE NOTICE '% % % %', unionedgeom_area/tilearea, counter, scancounter, ftime();\n END IF;\n RAISE NOTICE '% %', iter_record, content_hydrate(iter_record, fields);\n INSERT INTO pgstac_results (content) VALUES (content_hydrate(iter_record, fields));\n\n IF counter >= _limit\n OR scancounter > _scanlimit\n OR ftime() > _timelimit\n OR (exitwhenfull AND unionedgeom_area >= tilearea)\n THEN\n exit_flag := TRUE;\n EXIT;\n END IF;\n counter := counter + 1;\n scancounter := scancounter + 1;\n\n END LOOP;\n CLOSE curs;\n EXIT WHEN exit_flag;\n remaining_limit := _scanlimit - scancounter;\n END LOOP;\n\n SELECT jsonb_agg(content) INTO out_records FROM pgstac_results WHERE content IS NOT NULL;\n\n RETURN jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb)\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDROP FUNCTION IF EXISTS geojsonsearch;\nCREATE OR REPLACE FUNCTION geojsonsearch(\n IN geojson jsonb,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_geomfromgeojson(geojson),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS xyzsearch;\nCREATE OR REPLACE FUNCTION xyzsearch(\n IN _x int,\n IN _y int,\n IN _z int,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_transform(tileenvelope(_z, _x, _y), 4326),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE PROCEDURE analyze_items() AS $$\nDECLARE\n q text;\n timeout_ts timestamptz;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n SELECT format('ANALYZE (VERBOSE, SKIP_LOCKED) %I;', relname) INTO q\n FROM pg_stat_user_tables\n WHERE relname like '_item%' AND (n_mod_since_analyze>0 OR last_analyze IS NULL) LIMIT 1;\n IF NOT FOUND THEN\n EXIT;\n END IF;\n RAISE NOTICE '%', q;\n EXECUTE q;\n COMMIT;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE PROCEDURE validate_constraints() AS $$\nDECLARE\n q text;\nBEGIN\n FOR q IN\n SELECT\n FORMAT(\n 'ALTER TABLE %I.%I VALIDATE CONSTRAINT %I;',\n nsp.nspname,\n cls.relname,\n con.conname\n )\n\n FROM pg_constraint AS con\n JOIN pg_class AS cls\n ON con.conrelid = cls.oid\n JOIN pg_namespace AS nsp\n ON cls.relnamespace = nsp.oid\n WHERE convalidated = FALSE AND contype in ('c','f')\n AND nsp.nspname = 'pgstac'\n LOOP\n RAISE NOTICE '%', q;\n PERFORM run_or_queue(q);\n COMMIT;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION collection_extent(_collection text, runupdate boolean default false) RETURNS jsonb AS $$\nDECLARE\n geom_extent geometry;\n mind timestamptz;\n maxd timestamptz;\n extent jsonb;\nBEGIN\n IF runupdate THEN\n PERFORM update_partition_stats_q(partition)\n FROM partitions_view WHERE collection=_collection;\n END IF;\n SELECT\n min(lower(dtrange)),\n max(upper(edtrange)),\n st_extent(spatial)\n INTO\n mind,\n maxd,\n geom_extent\n FROM partitions_view\n WHERE collection=_collection;\n\n IF geom_extent IS NOT NULL AND mind IS NOT NULL AND maxd IS NOT NULL THEN\n extent := jsonb_build_object(\n 'spatial', jsonb_build_object(\n 'bbox', to_jsonb(array[array[st_xmin(geom_extent), st_ymin(geom_extent), st_xmax(geom_extent), st_ymax(geom_extent)]])\n ),\n 'temporal', jsonb_build_object(\n 'interval', to_jsonb(array[array[mind, maxd]])\n )\n );\n RETURN extent;\n END IF;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('id', '{\"title\": \"Item ID\",\"description\": \"Item identifier\",\"$ref\": \"https://schemas.stacspec.org/v1.0.0/item-spec/json-schema/item.json#/definitions/core/allOf/2/properties/id\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('geometry', '{\"title\": \"Item Geometry\",\"description\": \"Item Geometry\",\"$ref\": \"https://geojson.org/schema/Feature.json\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('datetime','{\"description\": \"Datetime\",\"type\": \"string\",\"title\": \"Acquired\",\"format\": \"date-time\",\"pattern\": \"(\\\\+00:00|Z)$\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDELETE FROM queryables a USING queryables b\n WHERE a.name = b.name AND a.collection_ids IS NOT DISTINCT FROM b.collection_ids AND a.id > b.id;\n\n\nINSERT INTO pgstac_settings (name, value) VALUES\n ('context', 'off'),\n ('context_estimated_count', '100000'),\n ('context_estimated_cost', '100000'),\n ('context_stats_ttl', '1 day'),\n ('default_filter_lang', 'cql2-json'),\n ('additional_properties', 'true'),\n ('use_queue', 'false'),\n ('queue_timeout', '10 minutes'),\n ('update_collection_extent', 'false'),\n ('format_cache', 'false'),\n ('readonly', 'false')\nON CONFLICT DO NOTHING\n;\n\n\nINSERT INTO cql2_ops (op, template, types) VALUES\n ('eq', '%s = %s', NULL),\n ('neq', '%s != %s', NULL),\n ('ne', '%s != %s', NULL),\n ('!=', '%s != %s', NULL),\n ('<>', '%s != %s', NULL),\n ('lt', '%s < %s', NULL),\n ('lte', '%s <= %s', NULL),\n ('gt', '%s > %s', NULL),\n ('gte', '%s >= %s', NULL),\n ('le', '%s <= %s', NULL),\n ('ge', '%s >= %s', NULL),\n ('=', '%s = %s', NULL),\n ('<', '%s < %s', NULL),\n ('<=', '%s <= %s', NULL),\n ('>', '%s > %s', NULL),\n ('>=', '%s >= %s', NULL),\n ('like', '%s LIKE %s', NULL),\n ('ilike', '%s ILIKE %s', NULL),\n ('+', '%s + %s', NULL),\n ('-', '%s - %s', NULL),\n ('*', '%s * %s', NULL),\n ('/', '%s / %s', NULL),\n ('not', 'NOT (%s)', NULL),\n ('between', '%s BETWEEN %s AND %s', NULL),\n ('isnull', '%s IS NULL', NULL),\n ('upper', 'upper(%s)', NULL),\n ('lower', 'lower(%s)', NULL),\n ('casei', 'upper(%s)', NULL),\n ('accenti', 'unaccent(%s)', NULL)\nON CONFLICT (op) DO UPDATE\n SET\n template = EXCLUDED.template\n;\n\n\nALTER FUNCTION to_text COST 5000;\nALTER FUNCTION to_float COST 5000;\nALTER FUNCTION to_int COST 5000;\nALTER FUNCTION to_tstz COST 5000;\nALTER FUNCTION to_text_array COST 5000;\n\nALTER FUNCTION update_partition_stats SECURITY DEFINER;\nALTER FUNCTION partition_after_triggerfunc SECURITY DEFINER;\nALTER FUNCTION drop_table_constraints SECURITY DEFINER;\nALTER FUNCTION create_table_constraints SECURITY DEFINER;\nALTER FUNCTION check_partition SECURITY DEFINER;\nALTER FUNCTION repartition SECURITY DEFINER;\nALTER FUNCTION where_stats SECURITY DEFINER;\nALTER FUNCTION search_query SECURITY DEFINER;\nALTER FUNCTION format_item SECURITY DEFINER;\nALTER FUNCTION maintain_index SECURITY DEFINER;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nGRANT ALL ON SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac to pgstac_admin;\n\n-- pgstac_read role limited to using function apis\nGRANT EXECUTE ON FUNCTION search TO pgstac_read;\nGRANT EXECUTE ON FUNCTION search_query TO pgstac_read;\nGRANT EXECUTE ON FUNCTION item_by_id TO pgstac_read;\nGRANT EXECUTE ON FUNCTION get_item TO pgstac_read;\nGRANT SELECT ON ALL TABLES IN SCHEMA pgstac TO pgstac_read;\n\n\nGRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_ingest;\nGRANT USAGE ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_ingest;\n\nREVOKE ALL PRIVILEGES ON PROCEDURE run_queued_queries FROM public;\nGRANT ALL ON PROCEDURE run_queued_queries TO pgstac_admin;\n\nREVOKE ALL PRIVILEGES ON FUNCTION run_queued_queries_intransaction FROM public;\nGRANT ALL ON FUNCTION run_queued_queries_intransaction TO pgstac_admin;\n\nRESET ROLE;\n\nSET ROLE pgstac_ingest;\nSELECT update_partition_stats_q(partition) FROM partitions_view;\nSELECT set_version('unreleased');\n" }, { "path": "src/pgstac/sql/000_idempotent_pre.sql", "content": "RESET ROLE;\nDO $$\nDECLARE\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='postgis') THEN\n CREATE EXTENSION IF NOT EXISTS postgis;\n END IF;\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='btree_gist') THEN\n CREATE EXTENSION IF NOT EXISTS btree_gist;\n END IF;\n IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname='unaccent') THEN\n CREATE EXTENSION IF NOT EXISTS unaccent;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_admin;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_read;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n CREATE ROLE pgstac_ingest;\n EXCEPTION WHEN duplicate_object THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n\nGRANT pgstac_admin TO current_user;\n\n-- Function to make sure pgstac_admin is the owner of items\nCREATE OR REPLACE FUNCTION pgstac_admin_owns() RETURNS VOID AS $$\nDECLARE\n f RECORD;\nBEGIN\n FOR f IN (\n SELECT\n concat(\n oid::regproc::text,\n '(',\n coalesce(pg_get_function_identity_arguments(oid),''),\n ')'\n ) AS name,\n CASE prokind WHEN 'f' THEN 'FUNCTION' WHEN 'p' THEN 'PROCEDURE' WHEN 'a' THEN 'AGGREGATE' END as typ\n FROM pg_proc\n WHERE\n pronamespace=to_regnamespace('pgstac')\n AND proowner != to_regrole('pgstac_admin')\n AND proname NOT LIKE 'pg_stat%'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n FOR f IN (\n SELECT\n oid::regclass::text as name,\n CASE relkind\n WHEN 'i' THEN 'INDEX'\n WHEN 'I' THEN 'INDEX'\n WHEN 'p' THEN 'TABLE'\n WHEN 'r' THEN 'TABLE'\n WHEN 'v' THEN 'VIEW'\n WHEN 'S' THEN 'SEQUENCE'\n ELSE NULL\n END as typ\n FROM pg_class\n WHERE relnamespace=to_regnamespace('pgstac') and relowner != to_regrole('pgstac_admin') AND relkind IN ('r','p','v','S') AND relname NOT LIKE 'pg_stat'\n )\n LOOP\n BEGIN\n EXECUTE format('ALTER %s %s OWNER TO pgstac_admin;', f.typ, f.name);\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\nSELECT pgstac_admin_owns();\n\nCREATE SCHEMA IF NOT EXISTS pgstac AUTHORIZATION pgstac_admin;\n\nGRANT ALL ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_admin;\nGRANT ALL ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_admin;\n\nALTER ROLE pgstac_admin SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_read SET SEARCH_PATH TO pgstac, public;\nALTER ROLE pgstac_ingest SET SEARCH_PATH TO pgstac, public;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\n\nGRANT pgstac_read TO pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\n\nSET ROLE pgstac_admin;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_admin IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\nRESET ROLE;\n\nSET ROLE pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT SELECT ON TABLES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT USAGE ON TYPES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON SEQUENCES TO pgstac_read;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON TABLES TO pgstac_ingest;\nALTER DEFAULT PRIVILEGES FOR ROLE pgstac_ingest IN SCHEMA pgstac GRANT ALL ON FUNCTIONS TO pgstac_ingest;\nRESET ROLE;\n\nSET SEARCH_PATH TO pgstac, public;\nSET ROLE pgstac_admin;\n\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS analyze_items;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\nDO $$\n BEGIN\n DROP FUNCTION IF EXISTS validate_constraints;\n EXCEPTION WHEN others THEN\n RAISE NOTICE '%, skipping', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\n-- Install these idempotently as migrations do not put them before trying to modify the collections table\n\n\nCREATE OR REPLACE FUNCTION collection_geom(content jsonb)\nRETURNS geometry AS $$\n WITH box AS (SELECT content->'extent'->'spatial'->'bbox'->0 as box)\n SELECT\n st_makeenvelope(\n (box->>0)::float,\n (box->>1)::float,\n (box->>2)::float,\n (box->>3)::float,\n 4326\n )\n FROM box;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION collection_datetime(content jsonb)\nRETURNS timestamptz AS $$\n SELECT\n CASE\n WHEN\n (content->'extent'->'temporal'->'interval'->0->>0) IS NULL\n THEN '-infinity'::timestamptz\n ELSE\n (content->'extent'->'temporal'->'interval'->0->>0)::timestamptz\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION collection_enddatetime(content jsonb)\nRETURNS timestamptz AS $$\n SELECT\n CASE\n WHEN\n (content->'extent'->'temporal'->'interval'->0->>1) IS NULL\n THEN 'infinity'::timestamptz\n ELSE\n (content->'extent'->'temporal'->'interval'->0->>1)::timestamptz\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n" }, { "path": "src/pgstac/sql/001_core.sql", "content": "\nCREATE TABLE IF NOT EXISTS migrations (\n version text PRIMARY KEY,\n datetime timestamptz DEFAULT clock_timestamp() NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION get_version() RETURNS text AS $$\n SELECT version FROM pgstac.migrations ORDER BY datetime DESC, version DESC LIMIT 1;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION set_version(text) RETURNS text AS $$\n INSERT INTO pgstac.migrations (version) VALUES ($1)\n ON CONFLICT DO NOTHING\n RETURNING version;\n$$ LANGUAGE SQL;\n\n\nCREATE TABLE IF NOT EXISTS pgstac_settings (\n name text PRIMARY KEY,\n value text NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION table_empty(text) RETURNS boolean AS $$\nDECLARE\n retval boolean;\nBEGIN\n EXECUTE format($q$\n SELECT NOT EXISTS (SELECT 1 FROM %I LIMIT 1)\n $q$,\n $1\n ) INTO retval;\n RETURN retval;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION get_setting(IN _setting text, IN conf jsonb DEFAULT NULL) RETURNS text AS $$\nSELECT COALESCE(\n nullif(conf->>_setting, ''),\n nullif(current_setting(concat('pgstac.',_setting), TRUE),''),\n nullif((SELECT value FROM pgstac.pgstac_settings WHERE name=_setting),'')\n);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_setting_bool(IN _setting text, IN conf jsonb DEFAULT NULL) RETURNS boolean AS $$\nSELECT COALESCE(\n nullif(conf->>_setting, ''),\n nullif(current_setting(concat('pgstac.',_setting), TRUE),''),\n nullif((SELECT value FROM pgstac.pgstac_settings WHERE name=_setting),''),\n 'FALSE'\n)::boolean;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION base_url(conf jsonb DEFAULT NULL) RETURNS text AS $$\n SELECT COALESCE(pgstac.get_setting('base_url', conf), '.');\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION additional_properties() RETURNS boolean AS $$\n SELECT pgstac.get_setting_bool('additional_properties');\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION readonly(conf jsonb DEFAULT NULL) RETURNS boolean AS $$\n SELECT pgstac.get_setting_bool('readonly', conf);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context(conf jsonb DEFAULT NULL) RETURNS text AS $$\n SELECT pgstac.get_setting('context', conf);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION context_estimated_count(conf jsonb DEFAULT NULL) RETURNS int AS $$\n SELECT pgstac.get_setting('context_estimated_count', conf)::int;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_estimated_cost();\nCREATE OR REPLACE FUNCTION context_estimated_cost(conf jsonb DEFAULT NULL) RETURNS float AS $$\n SELECT pgstac.get_setting('context_estimated_cost', conf)::float;\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS context_stats_ttl();\nCREATE OR REPLACE FUNCTION context_stats_ttl(conf jsonb DEFAULT NULL) RETURNS interval AS $$\n SELECT pgstac.get_setting('context_stats_ttl', conf)::interval;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION t2s(text) RETURNS text AS $$\n SELECT extract(epoch FROM $1::interval)::text || ' s';\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE STRICT;\n\nCREATE OR REPLACE FUNCTION age_ms(a timestamptz, b timestamptz DEFAULT clock_timestamp()) RETURNS float AS $$\n SELECT abs(extract(epoch from age(a,b)) * 1000);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION queue_timeout() RETURNS interval AS $$\n SELECT t2s(coalesce(\n get_setting('queue_timeout'),\n '1h'\n ))::interval;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION notice(VARIADIC text[]) RETURNS boolean AS $$\nDECLARE\ndebug boolean := current_setting('pgstac.debug', true);\nBEGIN\n IF debug THEN\n RAISE NOTICE 'NOTICE FROM FUNC: % >>>>> %', concat_ws(' | ', $1), clock_timestamp();\n RETURN TRUE;\n END IF;\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION empty_arr(ANYARRAY) RETURNS BOOLEAN AS $$\nSELECT CASE\n WHEN $1 IS NULL THEN TRUE\n WHEN cardinality($1)<1 THEN TRUE\nELSE FALSE\nEND;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_intersection(_a ANYARRAY, _b ANYARRAY) RETURNS ANYARRAY AS $$\n SELECT ARRAY ( SELECT unnest(_a) INTERSECT SELECT UNNEST(_b) );\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION array_map_ident(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_ident(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION array_map_literal(_a text[])\n RETURNS text[] AS $$\n SELECT array_agg(quote_literal(v)) FROM unnest(_a) v;\n$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION array_reverse(anyarray) RETURNS anyarray AS $$\nSELECT ARRAY(\n SELECT $1[i]\n FROM generate_subscripts($1,1) AS s(i)\n ORDER BY i DESC\n);\n$$ LANGUAGE SQL STRICT IMMUTABLE;\n\nDROP TABLE IF EXISTS query_queue;\nCREATE TABLE query_queue (\n query text PRIMARY KEY,\n added timestamptz DEFAULT now()\n);\n\nDROP TABLE IF EXISTS query_queue_history;\nCREATE TABLE query_queue_history(\n query text,\n added timestamptz NOT NULL,\n finished timestamptz NOT NULL DEFAULT now(),\n error text\n);\n\nCREATE OR REPLACE PROCEDURE run_queued_queries() AS $$\nDECLARE\n qitem query_queue%ROWTYPE;\n timeout_ts timestamptz;\n error text;\n cnt int := 0;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n DELETE FROM query_queue WHERE query = (SELECT query FROM query_queue ORDER BY added DESC LIMIT 1 FOR UPDATE SKIP LOCKED) RETURNING * INTO qitem;\n IF NOT FOUND THEN\n EXIT;\n END IF;\n cnt := cnt + 1;\n BEGIN\n RAISE NOTICE 'RUNNING QUERY: %', qitem.query;\n EXECUTE qitem.query;\n EXCEPTION WHEN others THEN\n error := format('%s | %s', SQLERRM, SQLSTATE);\n END;\n INSERT INTO query_queue_history (query, added, finished, error)\n VALUES (qitem.query, qitem.added, clock_timestamp(), error);\n COMMIT;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION run_queued_queries_intransaction() RETURNS int AS $$\nDECLARE\n qitem query_queue%ROWTYPE;\n timeout_ts timestamptz;\n error text;\n cnt int := 0;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n DELETE FROM query_queue WHERE query = (SELECT query FROM query_queue ORDER BY added DESC LIMIT 1 FOR UPDATE SKIP LOCKED) RETURNING * INTO qitem;\n IF NOT FOUND THEN\n RETURN cnt;\n END IF;\n cnt := cnt + 1;\n BEGIN\n qitem.query := regexp_replace(qitem.query, 'CONCURRENTLY', '');\n RAISE NOTICE 'RUNNING QUERY: %', qitem.query;\n\n EXECUTE qitem.query;\n EXCEPTION WHEN others THEN\n error := format('%s | %s', SQLERRM, SQLSTATE);\n RAISE WARNING '%', error;\n END;\n INSERT INTO query_queue_history (query, added, finished, error)\n VALUES (qitem.query, qitem.added, clock_timestamp(), error);\n END LOOP;\n RETURN cnt;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION run_or_queue(query text) RETURNS VOID AS $$\nDECLARE\n use_queue text := COALESCE(get_setting('use_queue'), 'FALSE')::boolean;\nBEGIN\n IF get_setting_bool('debug') THEN\n RAISE NOTICE '%', query;\n END IF;\n IF use_queue THEN\n INSERT INTO query_queue (query) VALUES (query) ON CONFLICT DO NOTHING;\n ELSE\n EXECUTE query;\n END IF;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n\nDROP FUNCTION IF EXISTS check_pgstac_settings;\nCREATE OR REPLACE FUNCTION check_pgstac_settings(_sysmem text DEFAULT NULL) RETURNS VOID AS $$\nDECLARE\n settingval text;\n sysmem bigint := pg_size_bytes(_sysmem);\n effective_cache_size bigint := pg_size_bytes(current_setting('effective_cache_size', TRUE));\n shared_buffers bigint := pg_size_bytes(current_setting('shared_buffers', TRUE));\n work_mem bigint := pg_size_bytes(current_setting('work_mem', TRUE));\n max_connections int := current_setting('max_connections', TRUE);\n maintenance_work_mem bigint := pg_size_bytes(current_setting('maintenance_work_mem', TRUE));\n seq_page_cost float := current_setting('seq_page_cost', TRUE);\n random_page_cost float := current_setting('random_page_cost', TRUE);\n temp_buffers bigint := pg_size_bytes(current_setting('temp_buffers', TRUE));\n r record;\nBEGIN\n IF _sysmem IS NULL THEN\n RAISE NOTICE 'Call function with the size of your system memory `SELECT check_pgstac_settings(''4GB'')` to get pg system setting recommendations.';\n ELSE\n IF effective_cache_size < (sysmem * 0.5) THEN\n RAISE WARNING 'effective_cache_size of % is set low for a system with %. Recomended value between % and %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.5), pg_size_pretty(sysmem * 0.75);\n ELSIF effective_cache_size > (sysmem * 0.75) THEN\n RAISE WARNING 'effective_cache_size of % is set high for a system with %. Recomended value between % and %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.5), pg_size_pretty(sysmem * 0.75);\n ELSE\n RAISE NOTICE 'effective_cache_size of % is set appropriately for a system with %', pg_size_pretty(effective_cache_size), pg_size_pretty(sysmem);\n END IF;\n\n IF shared_buffers < (sysmem * 0.2) THEN\n RAISE WARNING 'shared_buffers of % is set low for a system with %. Recomended value between % and %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.2), pg_size_pretty(sysmem * 0.3);\n ELSIF shared_buffers > (sysmem * 0.3) THEN\n RAISE WARNING 'shared_buffers of % is set high for a system with %. Recomended value between % and %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem), pg_size_pretty(sysmem * 0.2), pg_size_pretty(sysmem * 0.3);\n ELSE\n RAISE NOTICE 'shared_buffers of % is set appropriately for a system with %', pg_size_pretty(shared_buffers), pg_size_pretty(sysmem);\n END IF;\n shared_buffers = sysmem * 0.3;\n IF maintenance_work_mem < (sysmem * 0.2) THEN\n RAISE WARNING 'maintenance_work_mem of % is set low for shared_buffers of %. Recomended value between % and %', pg_size_pretty(maintenance_work_mem), pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers * 0.2), pg_size_pretty(shared_buffers * 0.3);\n ELSIF maintenance_work_mem > (shared_buffers * 0.3) THEN\n RAISE WARNING 'maintenance_work_mem of % is set high for shared_buffers of %. Recomended value between % and %', pg_size_pretty(maintenance_work_mem), pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers * 0.2), pg_size_pretty(shared_buffers * 0.3);\n ELSE\n RAISE NOTICE 'maintenance_work_mem of % is set appropriately for shared_buffers of %', pg_size_pretty(shared_buffers), pg_size_pretty(shared_buffers);\n END IF;\n\n IF work_mem * max_connections > shared_buffers THEN\n RAISE WARNING 'work_mem setting of % is set high for % max_connections please reduce work_mem to % or decrease max_connections to %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers/max_connections), floor(shared_buffers/work_mem);\n ELSIF work_mem * max_connections < (shared_buffers * 0.75) THEN\n RAISE WARNING 'work_mem setting of % is set low for % max_connections you may consider raising work_mem to % or increasing max_connections to %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers/max_connections), floor(shared_buffers/work_mem);\n ELSE\n RAISE NOTICE 'work_mem setting of % and max_connections of % are adequate for shared_buffers of %', pg_size_pretty(work_mem), max_connections, pg_size_pretty(shared_buffers);\n END IF;\n\n IF random_page_cost / seq_page_cost != 1.1 THEN\n RAISE WARNING 'random_page_cost (%) /seq_page_cost (%) should be set to 1.1 for SSD. Change random_page_cost to %', random_page_cost, seq_page_cost, 1.1 * seq_page_cost;\n ELSE\n RAISE NOTICE 'random_page_cost and seq_page_cost set appropriately for SSD';\n END IF;\n\n IF temp_buffers < greatest(pg_size_bytes('128MB'),(maintenance_work_mem / 2)) THEN\n RAISE WARNING 'pgstac makes heavy use of temp tables, consider raising temp_buffers from % to %', pg_size_pretty(temp_buffers), greatest('128MB', pg_size_pretty((shared_buffers / 16)));\n END IF;\n END IF;\n\n RAISE NOTICE 'VALUES FOR PGSTAC VARIABLES';\n RAISE NOTICE 'These can be set either as GUC system variables or by setting in the pgstac_settings table.';\n\n FOR r IN SELECT name, get_setting(name) as setting, CASE WHEN current_setting(concat('pgstac.',name), TRUE) IS NOT NULL THEN concat('pgstac.',name, ' GUC') WHEN value IS NOT NULL THEN 'pgstac_settings table' ELSE 'Not Set' END as loc FROM pgstac_settings LOOP\n RAISE NOTICE '% is set to % from the %', r.name, r.setting, r.loc;\n END LOOP;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pg_cron';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider intalling pg_cron which can be used to automate tasks';\n ELSE\n RAISE NOTICE 'pg_cron % is installed', settingval;\n END IF;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pgstattuple';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider installing the pgstattuple extension which can be used to help maintain tables and indexes.';\n ELSE\n RAISE NOTICE 'pgstattuple % is installed', settingval;\n END IF;\n\n SELECT installed_version INTO settingval from pg_available_extensions WHERE name = 'pg_stat_statements';\n IF NOT FOUND OR settingval IS NULL THEN\n RAISE NOTICE 'Consider installing the pg_stat_statements extension which is very helpful for tracking the types of queries on the system';\n ELSE\n RAISE NOTICE 'pg_stat_statements % is installed', settingval;\n IF current_setting('pg_stat_statements.track_statements', TRUE) IS DISTINCT FROM 'all' THEN\n RAISE WARNING 'SET pg_stat_statements.track_statements TO ''all''; --In order to track statements within functions.';\n END IF;\n END IF;\n\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac, public SET CLIENT_MIN_MESSAGES TO NOTICE;\n" }, { "path": "src/pgstac/sql/001a_jsonutils.sql", "content": "CREATE OR REPLACE FUNCTION to_int(jsonb) RETURNS int AS $$\n SELECT floor(($1->>0)::float)::int;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION to_float(jsonb) RETURNS float AS $$\n SELECT ($1->>0)::float;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION to_tstz(jsonb) RETURNS timestamptz AS $$\n SELECT ($1->>0)::timestamptz;\n$$ LANGUAGE SQL IMMUTABLE STRICT SET TIME ZONE 'UTC' COST 5000 PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION to_text(jsonb) RETURNS text AS $$\n SELECT CASE WHEN jsonb_typeof($1) IN ('array','object') THEN $1::text ELSE $1->>0 END;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION to_text_array(jsonb) RETURNS text[] AS $$\n SELECT\n CASE jsonb_typeof($1)\n WHEN 'array' THEN ARRAY(SELECT jsonb_array_elements_text($1))\n ELSE ARRAY[$1->>0]\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE STRICT COST 5000 PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION bbox_geom(_bbox jsonb) RETURNS geometry AS $$\nSELECT CASE jsonb_array_length(_bbox)\n WHEN 4 THEN\n ST_SetSRID(ST_MakeEnvelope(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float,\n (_bbox->>3)::float\n ),4326)\n WHEN 6 THEN\n ST_SetSRID(ST_3DMakeBox(\n ST_MakePoint(\n (_bbox->>0)::float,\n (_bbox->>1)::float,\n (_bbox->>2)::float\n ),\n ST_MakePoint(\n (_bbox->>3)::float,\n (_bbox->>4)::float,\n (_bbox->>5)::float\n )\n ),4326)\n ELSE null END;\n;\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION geom_bbox(_geom geometry) RETURNS jsonb AS $$\n SELECT jsonb_build_array(\n st_xmin(_geom),\n st_ymin(_geom),\n st_xmax(_geom),\n st_ymax(_geom)\n );\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION flip_jsonb_array(j jsonb) RETURNS jsonb AS $$\n SELECT jsonb_agg(value) FROM (SELECT value FROM jsonb_array_elements(j) WITH ORDINALITY ORDER BY ordinality DESC) as t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION explode_dotpaths(j jsonb) RETURNS SETOF text[] AS $$\n SELECT string_to_array(p, '.') as e FROM jsonb_array_elements_text(j) p;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION explode_dotpaths_recurse(IN j jsonb) RETURNS SETOF text[] AS $$\n WITH RECURSIVE t AS (\n SELECT e FROM explode_dotpaths(j) e\n UNION ALL\n SELECT e[1:cardinality(e)-1]\n FROM t\n WHERE cardinality(e)>1\n ) SELECT e FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION jsonb_set_nested(j jsonb, path text[], val jsonb) RETURNS jsonb AS $$\nDECLARE\nBEGIN\n IF cardinality(path) > 1 THEN\n FOR i IN 1..(cardinality(path)-1) LOOP\n IF j #> path[:i] IS NULL THEN\n j := jsonb_set_lax(j, path[:i], '{}', TRUE);\n END IF;\n END LOOP;\n END IF;\n RETURN jsonb_set_lax(j, path, val, true);\n\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\n\n\nCREATE OR REPLACE FUNCTION jsonb_include(j jsonb, f jsonb) RETURNS jsonb AS $$\nDECLARE\n includes jsonb := f-> 'include';\n outj jsonb := '{}'::jsonb;\n path text[];\nBEGIN\n IF\n includes IS NULL\n OR jsonb_array_length(includes) = 0\n THEN\n RETURN j;\n ELSE\n includes := includes || (\n CASE WHEN j ? 'collection' THEN\n '[\"id\",\"collection\"]'\n ELSE\n '[\"id\"]'\n END)::jsonb;\n FOR path IN SELECT explode_dotpaths(includes) LOOP\n outj := jsonb_set_nested(outj, path, j #> path);\n END LOOP;\n END IF;\n RETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_exclude(j jsonb, f jsonb) RETURNS jsonb AS $$\nDECLARE\n excludes jsonb := f-> 'exclude';\n outj jsonb := j;\n path text[];\nBEGIN\n IF\n excludes IS NULL\n OR jsonb_array_length(excludes) = 0\n THEN\n RETURN j;\n ELSE\n FOR path IN SELECT explode_dotpaths(excludes) LOOP\n outj := outj #- path;\n END LOOP;\n END IF;\n RETURN outj;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_fields(j jsonb, f jsonb DEFAULT '{\"fields\":[]}') RETURNS jsonb AS $$\n SELECT jsonb_exclude(jsonb_include(j, f), f);\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION merge_jsonb(_a jsonb, _b jsonb) RETURNS jsonb AS $$\n SELECT\n CASE\n WHEN _a = '\"𒍟※\"'::jsonb THEN NULL\n WHEN _a IS NULL OR jsonb_typeof(_a) = 'null' THEN _b\n WHEN jsonb_typeof(_a) = 'object' AND jsonb_typeof(_b) = 'object' THEN\n (\n SELECT\n jsonb_strip_nulls(\n jsonb_object_agg(\n key,\n merge_jsonb(a.value, b.value)\n )\n )\n FROM\n jsonb_each(coalesce(_a,'{}'::jsonb)) as a\n FULL JOIN\n jsonb_each(coalesce(_b,'{}'::jsonb)) as b\n USING (key)\n )\n WHEN\n jsonb_typeof(_a) = 'array'\n AND jsonb_typeof(_b) = 'array'\n AND jsonb_array_length(_a) = jsonb_array_length(_b)\n THEN\n (\n SELECT jsonb_agg(m) FROM\n ( SELECT\n merge_jsonb(\n jsonb_array_elements(_a),\n jsonb_array_elements(_b)\n ) as m\n ) as l\n )\n ELSE _a\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION strip_jsonb(_a jsonb, _b jsonb) RETURNS jsonb AS $$\n SELECT\n CASE\n\n WHEN (_a IS NULL OR jsonb_typeof(_a) = 'null') AND _b IS NOT NULL AND jsonb_typeof(_b) != 'null' THEN '\"𒍟※\"'::jsonb\n WHEN _b IS NULL OR jsonb_typeof(_a) = 'null' THEN _a\n WHEN _a = _b AND jsonb_typeof(_a) = 'object' THEN '{}'::jsonb\n WHEN _a = _b THEN NULL\n WHEN jsonb_typeof(_a) = 'object' AND jsonb_typeof(_b) = 'object' THEN\n (\n SELECT\n jsonb_strip_nulls(\n jsonb_object_agg(\n key,\n strip_jsonb(a.value, b.value)\n )\n )\n FROM\n jsonb_each(_a) as a\n FULL JOIN\n jsonb_each(_b) as b\n USING (key)\n )\n WHEN\n jsonb_typeof(_a) = 'array'\n AND jsonb_typeof(_b) = 'array'\n AND jsonb_array_length(_a) = jsonb_array_length(_b)\n THEN\n (\n SELECT jsonb_agg(m) FROM\n ( SELECT\n strip_jsonb(\n jsonb_array_elements(_a),\n jsonb_array_elements(_b)\n ) as m\n ) as l\n )\n ELSE _a\n END\n ;\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION nullif_jsonbnullempty(j jsonb) RETURNS jsonb AS $$\n SELECT nullif(nullif(nullif(j,'null'::jsonb),'{}'::jsonb),'[]'::jsonb);\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION jsonb_array_unique(j jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(jsonb_agg(DISTINCT a)) v FROM jsonb_array_elements(j) a;\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_concat_ignorenull(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT coalesce(a,'[]'::jsonb) || coalesce(b,'[]'::jsonb);\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_least(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(least(nullif_jsonbnullempty(a), nullif_jsonbnullempty(b)));\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION jsonb_greatest(a jsonb, b jsonb) RETURNS jsonb AS $$\n SELECT nullif_jsonbnullempty(greatest(a, b));\n$$ LANGUAGE SQL IMMUTABLE;\n\nCREATE OR REPLACE FUNCTION first_notnull_sfunc(anyelement, anyelement) RETURNS anyelement AS $$\n SELECT COALESCE($1,$2);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE AGGREGATE first_notnull(anyelement)(\n SFUNC = first_notnull_sfunc,\n STYPE = anyelement\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_array_unique_merge(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_concat_ignorenull,\n FINALFUNC = jsonb_array_unique\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_min(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_least\n);\n\nCREATE OR REPLACE AGGREGATE jsonb_max(jsonb) (\n STYPE = jsonb,\n SFUNC = jsonb_greatest\n);\n" }, { "path": "src/pgstac/sql/001s_stacutils.sql", "content": "/* looks for a geometry in a stac item first from geometry and falling back to bbox */\nCREATE OR REPLACE FUNCTION stac_geom(value jsonb) RETURNS geometry AS $$\nSELECT\n CASE\n WHEN value ? 'intersects' THEN\n ST_GeomFromGeoJSON(value->>'intersects')\n WHEN value ? 'geometry' THEN\n ST_GeomFromGeoJSON(value->>'geometry')\n WHEN value ? 'bbox' THEN\n pgstac.bbox_geom(value->'bbox')\n ELSE NULL\n END as geometry\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION stac_daterange(\n value jsonb\n) RETURNS tstzrange AS $$\nDECLARE\n props jsonb := value;\n dt timestamptz;\n edt timestamptz;\nBEGIN\n IF props ? 'properties' THEN\n props := props->'properties';\n END IF;\n IF\n props ? 'start_datetime'\n AND props->>'start_datetime' IS NOT NULL\n AND props ? 'end_datetime'\n AND props->>'end_datetime' IS NOT NULL\n THEN\n dt := props->>'start_datetime';\n edt := props->>'end_datetime';\n IF dt > edt THEN\n RAISE EXCEPTION 'start_datetime must be < end_datetime';\n END IF;\n ELSE\n dt := props->>'datetime';\n edt := props->>'datetime';\n END IF;\n IF dt is NULL OR edt IS NULL THEN\n RAISE NOTICE 'DT: %, EDT: %', dt, edt;\n RAISE EXCEPTION 'Either datetime (%) or both start_datetime (%) and end_datetime (%) must be set.', props->>'datetime',props->>'start_datetime',props->>'end_datetime';\n END IF;\n RETURN tstzrange(dt, edt, '[]');\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_datetime(value jsonb) RETURNS timestamptz AS $$\n SELECT lower(stac_daterange(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE OR REPLACE FUNCTION stac_end_datetime(value jsonb) RETURNS timestamptz AS $$\n SELECT upper(stac_daterange(value));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET TIMEZONE='UTC';\n\nCREATE TABLE IF NOT EXISTS stac_extensions(\n url text PRIMARY KEY,\n content jsonb\n);\n" }, { "path": "src/pgstac/sql/002_collections.sql", "content": "CREATE OR REPLACE FUNCTION collection_base_item(content jsonb) RETURNS jsonb AS $$\n SELECT jsonb_build_object(\n 'type', 'Feature',\n 'stac_version', content->'stac_version',\n 'assets', content->'item_assets',\n 'collection', content->'id'\n );\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE TABLE IF NOT EXISTS collections (\n key bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,\n id text GENERATED ALWAYS AS (content->>'id') STORED UNIQUE NOT NULL,\n content JSONB NOT NULL,\n base_item jsonb GENERATED ALWAYS AS (pgstac.collection_base_item(content)) STORED,\n geometry geometry GENERATED ALWAYS AS (pgstac.collection_geom(content)) STORED,\n datetime timestamptz GENERATED ALWAYS AS (pgstac.collection_datetime(content)) STORED,\n end_datetime timestamptz GENERATED ALWAYS AS (pgstac.collection_enddatetime(content)) STORED,\n private jsonb,\n partition_trunc text CHECK (partition_trunc IN ('year', 'month'))\n);\n\nCREATE OR REPLACE FUNCTION collection_base_item(cid text) RETURNS jsonb AS $$\n SELECT pgstac.collection_base_item(content) FROM pgstac.collections WHERE id = cid LIMIT 1;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION create_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION update_collection(data jsonb) RETURNS VOID AS $$\nDECLARE\n out collections%ROWTYPE;\nBEGIN\n UPDATE collections SET content=data WHERE id = data->>'id' RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_collection(data jsonb) RETURNS VOID AS $$\n INSERT INTO collections (content)\n VALUES (data)\n ON CONFLICT (id) DO\n UPDATE\n SET content=EXCLUDED.content\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION delete_collection(_id text) RETURNS VOID AS $$\nDECLARE\n out collections%ROWTYPE;\nBEGIN\n DELETE FROM collections WHERE id = _id RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION get_collection(id text) RETURNS jsonb AS $$\n SELECT content FROM collections\n WHERE id=$1\n ;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION all_collections() RETURNS jsonb AS $$\n SELECT coalesce(jsonb_agg(content), '[]'::jsonb) FROM collections;\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION collection_delete_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n collection_base_partition text := concat('_items_', OLD.key);\nBEGIN\n EXECUTE format($q$\n DELETE FROM partition_stats WHERE partition IN (\n SELECT partition FROM partition_sys_meta\n WHERE collection=%L\n );\n DROP TABLE IF EXISTS %I CASCADE;\n $q$,\n OLD.id,\n collection_base_partition\n );\n RETURN OLD;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER collection_delete_trigger BEFORE DELETE ON collections\nFOR EACH ROW EXECUTE FUNCTION collection_delete_trigger_func();\n" }, { "path": "src/pgstac/sql/002a_queryables.sql", "content": "CREATE OR REPLACE FUNCTION queryable_signature(n text, c text[]) RETURNS text AS $$\n SELECT concat(n, c);\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE TABLE queryables (\n id bigint GENERATED ALWAYS AS identity PRIMARY KEY,\n name text NOT NULL,\n collection_ids text[], -- used to determine what partitions to create indexes on\n definition jsonb,\n property_path text,\n property_wrapper text,\n property_index_type text\n);\nCREATE INDEX queryables_name_idx ON queryables (name);\nCREATE INDEX queryables_collection_idx ON queryables USING GIN (collection_ids);\nCREATE INDEX queryables_property_wrapper_idx ON queryables (property_wrapper);\n\nCREATE OR REPLACE FUNCTION pgstac.queryables_constraint_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n allcollections text[];\nBEGIN\n RAISE NOTICE 'Making sure that name/collection is unique for queryables %', NEW;\n IF NEW.collection_ids IS NOT NULL THEN\n IF EXISTS (\n SELECT 1\n FROM unnest(NEW.collection_ids) c\n LEFT JOIN\n collections\n ON (collections.id = c)\n WHERE collections.id IS NULL\n ) THEN\n RAISE foreign_key_violation USING MESSAGE = format(\n 'One or more collections in %s do not exist.', NEW.collection_ids\n );\n RETURN NULL;\n END IF;\n END IF;\n IF TG_OP = 'INSERT' THEN\n IF EXISTS (\n SELECT 1 FROM queryables q\n WHERE\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n )\n ) THEN\n RAISE unique_violation USING MESSAGE = format(\n 'There is already a queryable for %s for a collection in %s: %s',\n NEW.name,\n NEW.collection_ids,\n\t\t\t\t(SELECT json_agg(row_to_json(q)) FROM queryables q WHERE\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n ))\n );\n RETURN NULL;\n END IF;\n END IF;\n IF TG_OP = 'UPDATE' THEN\n IF EXISTS (\n SELECT 1 FROM queryables q\n WHERE\n q.id != NEW.id\n AND\n q.name = NEW.name\n AND (\n q.collection_ids && NEW.collection_ids\n OR\n q.collection_ids IS NULL\n OR\n NEW.collection_ids IS NULL\n )\n ) THEN\n RAISE unique_violation\n USING MESSAGE = format(\n 'There is already a queryable for %s for a collection in %s',\n NEW.name,\n NEW.collection_ids\n );\n RETURN NULL;\n END IF;\n END IF;\n\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER queryables_constraint_insert_trigger\nBEFORE INSERT ON queryables\nFOR EACH ROW EXECUTE PROCEDURE queryables_constraint_triggerfunc();\n\nCREATE TRIGGER queryables_constraint_update_trigger\nBEFORE UPDATE ON queryables\nFOR EACH ROW\nWHEN (NEW.name = OLD.name AND NEW.collection_ids IS DISTINCT FROM OLD.collection_ids)\nEXECUTE PROCEDURE queryables_constraint_triggerfunc();\n\n\nCREATE OR REPLACE FUNCTION array_to_path(arr text[]) RETURNS text AS $$\n SELECT string_agg(\n quote_literal(v),\n '->'\n ) FROM unnest(arr) v;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\n\n\nCREATE OR REPLACE FUNCTION queryable(\n IN dotpath text,\n OUT path text,\n OUT expression text,\n OUT wrapper text,\n OUT nulled_wrapper text\n) AS $$\nDECLARE\n q RECORD;\n path_elements text[];\nBEGIN\n dotpath := replace(dotpath, 'properties.', '');\n IF dotpath = 'start_datetime' THEN\n dotpath := 'datetime';\n END IF;\n IF dotpath IN ('id', 'geometry', 'datetime', 'end_datetime', 'collection') THEN\n path := dotpath;\n expression := dotpath;\n wrapper := NULL;\n RETURN;\n END IF;\n\n SELECT * INTO q FROM queryables\n WHERE\n name=dotpath\n OR name = 'properties.' || dotpath\n OR name = replace(dotpath, 'properties.', '')\n ;\n IF q.property_wrapper IS NULL THEN\n IF q.definition->>'type' = 'number' THEN\n wrapper := 'to_float';\n nulled_wrapper := wrapper;\n ELSIF q.definition->>'format' = 'date-time' THEN\n wrapper := 'to_tstz';\n nulled_wrapper := wrapper;\n ELSE\n nulled_wrapper := NULL;\n wrapper := 'to_text';\n END IF;\n ELSE\n wrapper := q.property_wrapper;\n nulled_wrapper := wrapper;\n END IF;\n IF q.property_path IS NOT NULL THEN\n path := q.property_path;\n ELSE\n path_elements := string_to_array(dotpath, '.');\n IF path_elements[1] IN ('links', 'assets', 'stac_version', 'stac_extensions') THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSIF path_elements[1] = 'properties' THEN\n path := format('content->%s', array_to_path(path_elements));\n ELSE\n path := format($F$content->'properties'->%s$F$, array_to_path(path_elements));\n END IF;\n END IF;\n expression := format('%I(%s)', wrapper, path);\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\nCREATE OR REPLACE FUNCTION unnest_collection(collection_ids text[] DEFAULT NULL) RETURNS SETOF text AS $$\n DECLARE\n BEGIN\n IF collection_ids IS NULL THEN\n RETURN QUERY SELECT id FROM collections;\n END IF;\n RETURN QUERY SELECT unnest(collection_ids);\n END;\n$$ LANGUAGE PLPGSQL STABLE;\n\nCREATE OR REPLACE FUNCTION normalize_indexdef(def text) RETURNS text AS $$\nDECLARE\nBEGIN\n def := btrim(def, ' \\n\\t');\n\tdef := regexp_replace(def, '^CREATE (UNIQUE )?INDEX ([^ ]* )?ON (ONLY )?([^ ]* )?', '', 'i');\n RETURN def;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION indexdef(q queryables) RETURNS text AS $$\n DECLARE\n out text;\n BEGIN\n IF q.name = 'id' THEN\n out := 'CREATE UNIQUE INDEX ON %I USING btree (id)';\n ELSIF q.name = 'datetime' THEN\n out := 'CREATE INDEX ON %I USING btree (datetime DESC, end_datetime)';\n ELSIF q.name = 'geometry' THEN\n out := 'CREATE INDEX ON %I USING gist (geometry)';\n ELSE\n out := format($q$CREATE INDEX ON %%I USING %s (%s(((content -> 'properties'::text) -> %L::text)))$q$,\n lower(COALESCE(q.property_index_type, 'BTREE')),\n lower(COALESCE(q.property_wrapper, 'to_text')),\n q.name\n );\n END IF;\n RETURN btrim(out, ' \\n\\t');\n END;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nDROP VIEW IF EXISTS pgstac_indexes;\nCREATE VIEW pgstac_indexes AS\nSELECT\n i.schemaname,\n i.tablename,\n i.indexname,\n regexp_replace(btrim(replace(replace(indexdef, i.indexname, ''),'pgstac.',''),' \\t\\n'), '[ ]+', ' ', 'g') as idx,\n COALESCE(\n (regexp_match(indexdef, '\\(([a-zA-Z]+)\\)'))[1],\n (regexp_match(indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_-]+)''::text'))[1],\n CASE WHEN indexdef ~* '\\(datetime desc, end_datetime\\)' THEN 'datetime' ELSE NULL END\n ) AS field,\n pg_table_size(i.indexname::text) as index_size,\n pg_size_pretty(pg_table_size(i.indexname::text)) as index_size_pretty\nFROM\n pg_indexes i\nWHERE i.schemaname='pgstac' and i.tablename ~ '_items_' AND indexdef !~* ' only ';\n\nDROP VIEW IF EXISTS pgstac_index_stats;\nCREATE VIEW pgstac_indexes_stats AS\nSELECT\n i.schemaname,\n i.tablename,\n i.indexname,\n indexdef,\n COALESCE(\n (regexp_match(indexdef, '\\(([a-zA-Z]+)\\)'))[1],\n (regexp_match(indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_]+)''::text'))[1],\n CASE WHEN indexdef ~* '\\(datetime desc, end_datetime\\)' THEN 'datetime_end_datetime' ELSE NULL END\n ) AS field,\n pg_table_size(i.indexname::text) as index_size,\n pg_size_pretty(pg_table_size(i.indexname::text)) as index_size_pretty,\n n_distinct,\n most_common_vals::text::text[],\n most_common_freqs::text::text[],\n histogram_bounds::text::text[],\n correlation\nFROM\n pg_indexes i\n LEFT JOIN pg_stats s ON (s.tablename = i.indexname)\nWHERE i.schemaname='pgstac' and i.tablename ~ '_items_';\n\nCREATE OR REPLACE FUNCTION queryable_indexes(\n IN treeroot text DEFAULT 'items',\n IN changes boolean DEFAULT FALSE,\n OUT collection text,\n OUT partition text,\n OUT field text,\n OUT indexname text,\n OUT existing_idx text,\n OUT queryable_idx text\n) RETURNS SETOF RECORD AS $$\nWITH p AS (\n SELECT\n relid::text as partition,\n replace(replace(\n CASE\n WHEN parentrelid::regclass::text='items' THEN pg_get_expr(c.relpartbound, c.oid)\n ELSE pg_get_expr(parent.relpartbound, parent.oid)\n END,\n 'FOR VALUES IN (''',''), ''')',\n ''\n ) AS collection\n FROM pg_partition_tree(treeroot)\n JOIN pg_class c ON (relid::regclass = c.oid)\n JOIN pg_class parent ON (parentrelid::regclass = parent.oid AND isleaf)\n ), i AS (\n SELECT\n partition,\n indexname,\n regexp_replace(btrim(replace(replace(indexdef, indexname, ''),'pgstac.',''),' \\t\\n'), '[ ]+', ' ', 'g') as iidx,\n COALESCE(\n (regexp_match(indexdef, '\\(([a-zA-Z]+)\\)'))[1],\n (regexp_match(indexdef, '\\(content -> ''properties''::text\\) -> ''([a-zA-Z0-9\\:\\_-]+)''::text'))[1],\n CASE WHEN indexdef ~* '\\(datetime desc, end_datetime\\)' THEN 'datetime' ELSE NULL END\n ) AS field\n FROM\n pg_indexes\n JOIN p ON (tablename=partition)\n ), q AS (\n SELECT\n name AS field,\n collection,\n partition,\n format(indexdef(queryables), partition) as qidx\n FROM queryables, unnest_collection(queryables.collection_ids) collection\n JOIN p USING (collection)\n WHERE property_index_type IS NOT NULL OR name IN ('datetime','geometry','id')\n )\n SELECT\n collection,\n partition,\n field,\n indexname,\n iidx as existing_idx,\n qidx as queryable_idx\n FROM i FULL JOIN q USING (field, partition)\n WHERE CASE WHEN changes THEN lower(iidx) IS DISTINCT FROM lower(qidx) ELSE TRUE END;\n;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION maintain_index(\n indexname text,\n queryable_idx text,\n dropindexes boolean DEFAULT FALSE,\n rebuildindexes boolean DEFAULT FALSE,\n idxconcurrently boolean DEFAULT FALSE\n) RETURNS VOID AS $$\nDECLARE\nBEGIN\n IF indexname IS NOT NULL THEN\n IF dropindexes OR queryable_idx IS NOT NULL THEN\n EXECUTE format('DROP INDEX IF EXISTS %I;', indexname);\n ELSIF rebuildindexes THEN\n IF idxconcurrently THEN\n EXECUTE format('REINDEX INDEX CONCURRENTLY %I;', indexname);\n ELSE\n EXECUTE format('REINDEX INDEX CONCURRENTLY %I;', indexname);\n END IF;\n END IF;\n END IF;\n IF queryable_idx IS NOT NULL THEN\n IF idxconcurrently THEN\n EXECUTE replace(queryable_idx, 'INDEX', 'INDEX CONCURRENTLY');\n ELSE EXECUTE queryable_idx;\n END IF;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\n\nset check_function_bodies to off;\nCREATE OR REPLACE FUNCTION maintain_partition_queries(\n part text DEFAULT 'items',\n dropindexes boolean DEFAULT FALSE,\n rebuildindexes boolean DEFAULT FALSE,\n idxconcurrently boolean DEFAULT FALSE\n) RETURNS SETOF text AS $$\nDECLARE\n rec record;\n q text;\nBEGIN\n FOR rec IN (\n SELECT * FROM queryable_indexes(part,true)\n ) LOOP\n q := format(\n 'SELECT maintain_index(\n %L,%L,%L,%L,%L\n );',\n rec.indexname,\n rec.queryable_idx,\n dropindexes,\n rebuildindexes,\n idxconcurrently\n );\n RAISE NOTICE 'Q: %', q;\n RETURN NEXT q;\n END LOOP;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION maintain_partitions(\n part text DEFAULT 'items',\n dropindexes boolean DEFAULT FALSE,\n rebuildindexes boolean DEFAULT FALSE\n) RETURNS VOID AS $$\n WITH t AS (\n SELECT run_or_queue(q) FROM maintain_partition_queries(part, dropindexes, rebuildindexes) q\n ) SELECT count(*) FROM t;\n$$ LANGUAGE SQL;\n\n\nCREATE OR REPLACE FUNCTION queryables_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\nBEGIN\n PERFORM maintain_partitions();\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE TRIGGER queryables_trigger AFTER INSERT OR UPDATE ON queryables\nFOR EACH STATEMENT EXECUTE PROCEDURE queryables_trigger_func();\n\n\nCREATE OR REPLACE FUNCTION get_queryables(_collection_ids text[] DEFAULT NULL) RETURNS jsonb AS $$\nDECLARE\nBEGIN\n -- Build up queryables if the input contains valid collection ids or is empty\n IF EXISTS (\n SELECT 1 FROM collections\n WHERE\n _collection_ids IS NULL\n OR cardinality(_collection_ids) = 0\n OR id = ANY(_collection_ids)\n )\n THEN\n RETURN (\n WITH base AS (\n SELECT\n unnest(collection_ids) as collection_id,\n name,\n coalesce(definition, '{\"type\":\"string\"}'::jsonb) as definition\n FROM queryables\n WHERE\n _collection_ids IS NULL OR\n _collection_ids = '{}'::text[] OR\n _collection_ids && collection_ids\n UNION ALL\n SELECT null, name, coalesce(definition, '{\"type\":\"string\"}'::jsonb) as definition\n FROM queryables WHERE collection_ids IS NULL OR collection_ids = '{}'::text[]\n ), g AS (\n SELECT\n name,\n first_notnull(definition) as definition,\n jsonb_array_unique_merge(definition->'enum') as enum,\n jsonb_min(definition->'minimum') as minimum,\n jsonb_min(definition->'maxiumn') as maximum\n FROM base\n GROUP BY 1\n )\n SELECT\n jsonb_build_object(\n '$schema', 'http://json-schema.org/draft-07/schema#',\n '$id', '',\n 'type', 'object',\n 'title', 'STAC Queryables.',\n 'properties', jsonb_object_agg(\n name,\n definition\n ||\n jsonb_strip_nulls(jsonb_build_object(\n 'enum', enum,\n 'minimum', minimum,\n 'maximum', maximum\n ))\n ),\n 'additionalProperties', pgstac.additional_properties()\n )\n FROM g\n );\n ELSE\n RETURN NULL;\n END IF;\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\nCREATE OR REPLACE FUNCTION get_queryables(_collection text DEFAULT NULL) RETURNS jsonb AS $$\n SELECT\n CASE\n WHEN _collection IS NULL THEN get_queryables(NULL::text[])\n ELSE get_queryables(ARRAY[_collection])\n END\n ;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_queryables() RETURNS jsonb AS $$\n SELECT get_queryables(NULL::text[]);\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION schema_qualify_refs(url text, j jsonb) returns jsonb as $$\n SELECT regexp_replace(j::text, '\"\\$ref\": \"#', concat('\"$ref\": \"', url, '#'), 'g')::jsonb;\n$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE VIEW stac_extension_queryables AS\nSELECT DISTINCT key as name, schema_qualify_refs(e.url, j.value) as definition FROM stac_extensions e, jsonb_each(e.content->'definitions'->'fields'->'properties') j;\n\n\nCREATE OR REPLACE FUNCTION missing_queryables(_collection text, _tablesample float DEFAULT 5, minrows float DEFAULT 10) RETURNS TABLE(collection text, name text, definition jsonb, property_wrapper text) AS $$\nDECLARE\n q text;\n _partition text;\n explain_json json;\n psize float;\n estrows float;\nBEGIN\n SELECT format('_items_%s', key) INTO _partition FROM collections WHERE id=_collection;\n\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM %I;', _partition)\n INTO explain_json;\n psize := explain_json->0->'Plan'->'Plan Rows';\n estrows := _tablesample * .01 * psize;\n IF estrows < minrows THEN\n _tablesample := least(100,greatest(_tablesample, (estrows / psize) / 100));\n RAISE NOTICE '%', (psize / estrows) / 100;\n END IF;\n RAISE NOTICE 'Using tablesample % to find missing queryables from % % that has ~% rows estrows: %', _tablesample, _collection, _partition, psize, estrows;\n\n q := format(\n $q$\n WITH q AS (\n SELECT * FROM queryables\n WHERE\n collection_ids IS NULL\n OR %L = ANY(collection_ids)\n ), t AS (\n SELECT\n content->'properties' AS properties\n FROM\n %I\n TABLESAMPLE SYSTEM(%L)\n ), p AS (\n SELECT DISTINCT ON (key)\n key,\n value,\n s.definition\n FROM t\n JOIN LATERAL jsonb_each(properties) ON TRUE\n LEFT JOIN q ON (q.name=key)\n LEFT JOIN stac_extension_queryables s ON (s.name=key)\n WHERE q.definition IS NULL\n )\n SELECT\n %L,\n key,\n COALESCE(definition, jsonb_build_object('type',jsonb_typeof(value))) as definition,\n CASE\n WHEN definition->>'type' = 'integer' THEN 'to_int'\n WHEN COALESCE(definition->>'type', jsonb_typeof(value)) = 'number' THEN 'to_float'\n WHEN COALESCE(definition->>'type', jsonb_typeof(value)) = 'array' THEN 'to_text_array'\n ELSE 'to_text'\n END\n FROM p;\n $q$,\n _collection,\n _partition,\n _tablesample,\n _collection\n );\n RETURN QUERY EXECUTE q;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION missing_queryables(_tablesample float DEFAULT 5) RETURNS TABLE(collection_ids text[], name text, definition jsonb, property_wrapper text) AS $$\n SELECT\n array_agg(collection),\n name,\n definition,\n property_wrapper\n FROM\n collections\n JOIN LATERAL\n missing_queryables(id, _tablesample) c\n ON TRUE\n GROUP BY\n 2,3,4\n ORDER BY 2,1\n ;\n$$ LANGUAGE SQL;\n" }, { "path": "src/pgstac/sql/002b_cql.sql", "content": "CREATE OR REPLACE FUNCTION parse_dtrange(\n _indate jsonb,\n relative_base timestamptz DEFAULT date_trunc('hour', CURRENT_TIMESTAMP)\n) RETURNS tstzrange AS $$\nDECLARE\n timestrs text[];\n s timestamptz;\n e timestamptz;\nBEGIN\n timestrs :=\n CASE\n WHEN _indate ? 'timestamp' THEN\n ARRAY[_indate->>'timestamp']\n WHEN _indate ? 'interval' THEN\n to_text_array(_indate->'interval')\n WHEN jsonb_typeof(_indate) = 'array' THEN\n to_text_array(_indate)\n ELSE\n regexp_split_to_array(\n _indate->>0,\n '/'\n )\n END;\n RAISE NOTICE 'TIMESTRS %', timestrs;\n IF cardinality(timestrs) = 1 THEN\n IF timestrs[1] ILIKE 'P%' THEN\n RETURN tstzrange(relative_base - upper(timestrs[1])::interval, relative_base, '[)');\n END IF;\n s := timestrs[1]::timestamptz;\n RETURN tstzrange(s, s, '[]');\n END IF;\n\n IF cardinality(timestrs) != 2 THEN\n RAISE EXCEPTION 'Timestamp cannot have more than 2 values';\n END IF;\n\n IF timestrs[1] = '..' OR timestrs[1] = '' THEN\n s := '-infinity'::timestamptz;\n e := timestrs[2]::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] = '..' OR timestrs[2] = '' THEN\n s := timestrs[1]::timestamptz;\n e := 'infinity'::timestamptz;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[1] ILIKE 'P%' AND timestrs[2] NOT ILIKE 'P%' THEN\n e := timestrs[2]::timestamptz;\n s := e - upper(timestrs[1])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n IF timestrs[2] ILIKE 'P%' AND timestrs[1] NOT ILIKE 'P%' THEN\n s := timestrs[1]::timestamptz;\n e := s + upper(timestrs[2])::interval;\n RETURN tstzrange(s,e,'[)');\n END IF;\n\n s := timestrs[1]::timestamptz;\n e := timestrs[2]::timestamptz;\n\n RETURN tstzrange(s,e,'[)');\n\n RETURN NULL;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT PARALLEL SAFE SET TIME ZONE 'UTC';\n\nCREATE OR REPLACE FUNCTION parse_dtrange(\n _indate text,\n relative_base timestamptz DEFAULT CURRENT_TIMESTAMP\n) RETURNS tstzrange AS $$\n SELECT parse_dtrange(to_jsonb(_indate), relative_base);\n$$ LANGUAGE SQL STABLE STRICT PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION temporal_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\n ll text := 'datetime';\n lh text := 'end_datetime';\n rrange tstzrange;\n rl text;\n rh text;\n outq text;\nBEGIN\n rrange := parse_dtrange(args->1);\n RAISE NOTICE 'Constructing temporal query OP: %, ARGS: %, RRANGE: %', op, args, rrange;\n op := lower(op);\n rl := format('%L::timestamptz', lower(rrange));\n rh := format('%L::timestamptz', upper(rrange));\n outq := CASE op\n WHEN 't_before' THEN 'lh < rl'\n WHEN 't_after' THEN 'll > rh'\n WHEN 't_meets' THEN 'lh = rl'\n WHEN 't_metby' THEN 'll = rh'\n WHEN 't_overlaps' THEN 'll < rl AND rl < lh < rh'\n WHEN 't_overlappedby' THEN 'rl < ll < rh AND lh > rh'\n WHEN 't_starts' THEN 'll = rl AND lh < rh'\n WHEN 't_startedby' THEN 'll = rl AND lh > rh'\n WHEN 't_during' THEN 'll > rl AND lh < rh'\n WHEN 't_contains' THEN 'll < rl AND lh > rh'\n WHEN 't_finishes' THEN 'll > rl AND lh = rh'\n WHEN 't_finishedby' THEN 'll < rl AND lh = rh'\n WHEN 't_equals' THEN 'll = rl AND lh = rh'\n WHEN 't_disjoint' THEN 'NOT (ll <= rh AND lh >= rl)'\n WHEN 't_intersects' THEN 'll <= rh AND lh >= rl'\n WHEN 'anyinteracts' THEN 'll <= rh AND lh >= rl'\n END;\n outq := regexp_replace(outq, '\\mll\\M', ll);\n outq := regexp_replace(outq, '\\mlh\\M', lh);\n outq := regexp_replace(outq, '\\mrl\\M', rl);\n outq := regexp_replace(outq, '\\mrh\\M', rh);\n outq := format('(%s)', outq);\n RETURN outq;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\n\n\nCREATE OR REPLACE FUNCTION spatial_op_query(op text, args jsonb) RETURNS text AS $$\nDECLARE\n geom text;\n j jsonb := args->1;\nBEGIN\n op := lower(op);\n RAISE NOTICE 'Constructing spatial query OP: %, ARGS: %', op, args;\n IF op NOT IN ('s_equals','s_disjoint','s_touches','s_within','s_overlaps','s_crosses','s_intersects','intersects','s_contains') THEN\n RAISE EXCEPTION 'Spatial Operator % Not Supported', op;\n END IF;\n op := regexp_replace(op, '^s_', 'st_');\n IF op = 'intersects' THEN\n op := 'st_intersects';\n END IF;\n -- Convert geometry to WKB string\n IF j ? 'type' AND j ? 'coordinates' THEN\n geom := st_geomfromgeojson(j)::text;\n ELSIF jsonb_typeof(j) = 'array' THEN\n geom := bbox_geom(j)::text;\n END IF;\n\n RETURN format('%s(geometry, %L::geometry)', op, geom);\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION query_to_cql2(q jsonb) RETURNS jsonb AS $$\n-- Translates anything passed in through the deprecated \"query\" into equivalent CQL2\nWITH t AS (\n SELECT key as property, value as ops\n FROM jsonb_each(q)\n), t2 AS (\n SELECT property, (jsonb_each(ops)).*\n FROM t WHERE jsonb_typeof(ops) = 'object'\n UNION ALL\n SELECT property, 'eq', ops\n FROM t WHERE jsonb_typeof(ops) != 'object'\n)\nSELECT\n jsonb_strip_nulls(jsonb_build_object(\n 'op', 'and',\n 'args', jsonb_agg(\n jsonb_build_object(\n 'op', key,\n 'args', jsonb_build_array(\n jsonb_build_object('property',property),\n value\n )\n )\n )\n )\n) as qcql FROM t2\n;\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\n\nCREATE OR REPLACE FUNCTION cql1_to_cql2(j jsonb) RETURNS jsonb AS $$\nDECLARE\n args jsonb;\n ret jsonb;\nBEGIN\n RAISE NOTICE 'CQL1_TO_CQL2: %', j;\n IF j ? 'filter' THEN\n RETURN cql1_to_cql2(j->'filter');\n END IF;\n IF j ? 'property' THEN\n RETURN j;\n END IF;\n IF jsonb_typeof(j) = 'array' THEN\n SELECT jsonb_agg(cql1_to_cql2(el)) INTO args FROM jsonb_array_elements(j) el;\n RETURN args;\n END IF;\n IF jsonb_typeof(j) = 'number' THEN\n RETURN j;\n END IF;\n IF jsonb_typeof(j) = 'string' THEN\n RETURN j;\n END IF;\n\n IF jsonb_typeof(j) = 'object' THEN\n SELECT jsonb_build_object(\n 'op', key,\n 'args', cql1_to_cql2(value)\n ) INTO ret\n FROM jsonb_each(j)\n WHERE j IS NOT NULL;\n RETURN ret;\n END IF;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE STRICT;\n\nCREATE TABLE cql2_ops (\n op text PRIMARY KEY,\n template text,\n types text[]\n);\nINSERT INTO cql2_ops (op, template, types) VALUES\n ('eq', '%s = %s', NULL),\n ('neq', '%s != %s', NULL),\n ('ne', '%s != %s', NULL),\n ('!=', '%s != %s', NULL),\n ('<>', '%s != %s', NULL),\n ('lt', '%s < %s', NULL),\n ('lte', '%s <= %s', NULL),\n ('gt', '%s > %s', NULL),\n ('gte', '%s >= %s', NULL),\n ('le', '%s <= %s', NULL),\n ('ge', '%s >= %s', NULL),\n ('=', '%s = %s', NULL),\n ('<', '%s < %s', NULL),\n ('<=', '%s <= %s', NULL),\n ('>', '%s > %s', NULL),\n ('>=', '%s >= %s', NULL),\n ('like', '%s LIKE %s', NULL),\n ('ilike', '%s ILIKE %s', NULL),\n ('+', '%s + %s', NULL),\n ('-', '%s - %s', NULL),\n ('*', '%s * %s', NULL),\n ('/', '%s / %s', NULL),\n ('not', 'NOT (%s)', NULL),\n ('between', '%s BETWEEN %s AND %s', NULL),\n ('isnull', '%s IS NULL', NULL),\n ('upper', 'upper(%s)', NULL),\n ('lower', 'lower(%s)', NULL),\n ('casei', 'upper(%s)', NULL),\n ('accenti', 'unaccent(%s)', NULL)\nON CONFLICT (op) DO UPDATE\n SET\n template = EXCLUDED.template\n;\n\n\nCREATE OR REPLACE FUNCTION cql2_query(j jsonb, wrapper text DEFAULT NULL) RETURNS text AS $$\n#variable_conflict use_variable\nDECLARE\n args jsonb := j->'args';\n arg jsonb;\n op text := lower(j->>'op');\n cql2op RECORD;\n literal text;\n _wrapper text;\n leftarg text;\n rightarg text;\n prop text;\n extra_props bool := pgstac.additional_properties();\nBEGIN\n IF j IS NULL OR (op IS NOT NULL AND args IS NULL) THEN\n RETURN NULL;\n END IF;\n RAISE NOTICE 'CQL2_QUERY: %', j;\n\n -- check if all properties are represented in the queryables\n IF NOT extra_props THEN\n FOR prop IN\n SELECT DISTINCT p->>0\n FROM jsonb_path_query(j, 'strict $.**.property') p\n WHERE p->>0 NOT IN ('id', 'datetime', 'geometry', 'end_datetime', 'collection')\n LOOP\n IF (queryable(prop)).nulled_wrapper IS NULL THEN\n RAISE EXCEPTION 'Term % is not found in queryables.', prop;\n END IF;\n END LOOP;\n END IF;\n\n IF j ? 'filter' THEN\n RETURN cql2_query(j->'filter');\n END IF;\n\n IF j ? 'upper' THEN\n RETURN cql2_query(jsonb_build_object('op', 'upper', 'args', j->'upper'));\n END IF;\n\n IF j ? 'lower' THEN\n RETURN cql2_query(jsonb_build_object('op', 'lower', 'args', j->'lower'));\n END IF;\n\n -- Temporal Query\n IF op ilike 't_%' or op = 'anyinteracts' THEN\n RETURN temporal_op_query(op, args);\n END IF;\n\n -- If property is a timestamp convert it to text to use with\n -- general operators\n IF j ? 'timestamp' THEN\n RETURN format('%L::timestamptz', to_tstz(j->'timestamp'));\n END IF;\n IF j ? 'interval' THEN\n RAISE EXCEPTION 'Please use temporal operators when using intervals.';\n RETURN NONE;\n END IF;\n\n -- Spatial Query\n IF op ilike 's_%' or op = 'intersects' THEN\n RETURN spatial_op_query(op, args);\n END IF;\n\n IF op IN ('a_equals','a_contains','a_contained_by','a_overlaps') THEN\n IF args->0 ? 'property' THEN\n leftarg := format('to_text_array(%s)', (queryable(args->0->>'property')).path);\n END IF;\n IF args->1 ? 'property' THEN\n rightarg := format('to_text_array(%s)', (queryable(args->1->>'property')).path);\n END IF;\n RETURN FORMAT(\n '%s %s %s',\n COALESCE(leftarg, quote_literal(to_text_array(args->0))),\n CASE op\n WHEN 'a_equals' THEN '='\n WHEN 'a_contains' THEN '@>'\n WHEN 'a_contained_by' THEN '<@'\n WHEN 'a_overlaps' THEN '&&'\n END,\n COALESCE(rightarg, quote_literal(to_text_array(args->1)))\n );\n END IF;\n\n IF op = 'in' THEN\n RAISE NOTICE 'IN : % % %', args, jsonb_build_array(args->0), args->1;\n args := jsonb_build_array(args->0) || (args->1);\n RAISE NOTICE 'IN2 : %', args;\n END IF;\n\n\n\n IF op = 'between' THEN\n args = jsonb_build_array(\n args->0,\n args->1,\n args->2\n );\n END IF;\n\n -- Make sure that args is an array and run cql2_query on\n -- each element of the array\n RAISE NOTICE 'ARGS PRE: %', args;\n IF j ? 'args' THEN\n IF jsonb_typeof(args) != 'array' THEN\n args := jsonb_build_array(args);\n END IF;\n\n IF jsonb_path_exists(args, '$[*] ? (@.property == \"id\" || @.property == \"datetime\" || @.property == \"end_datetime\" || @.property == \"collection\")') THEN\n wrapper := NULL;\n ELSE\n -- if any of the arguments are a property, try to get the property_wrapper\n FOR arg IN SELECT jsonb_path_query(args, '$[*] ? (@.property != null)') LOOP\n RAISE NOTICE 'Arg: %', arg;\n wrapper := (queryable(arg->>'property')).nulled_wrapper;\n RAISE NOTICE 'Property: %, Wrapper: %', arg, wrapper;\n IF wrapper IS NOT NULL THEN\n EXIT;\n END IF;\n END LOOP;\n\n -- if the property was not in queryables, see if any args were numbers\n IF\n wrapper IS NULL\n AND jsonb_path_exists(args, '$[*] ? (@.type()==\"number\")')\n THEN\n wrapper := 'to_float';\n END IF;\n wrapper := coalesce(wrapper, 'to_text');\n END IF;\n\n SELECT jsonb_agg(cql2_query(a, wrapper))\n INTO args\n FROM jsonb_array_elements(args) a;\n END IF;\n RAISE NOTICE 'ARGS: %', args;\n\n IF op IN ('and', 'or') THEN\n RETURN\n format(\n '(%s)',\n array_to_string(to_text_array(args), format(' %s ', upper(op)))\n );\n END IF;\n\n IF op = 'in' THEN\n RAISE NOTICE 'IN -- % %', args->0, to_text(args->0);\n RETURN format(\n '%s IN (%s)',\n to_text(args->0),\n array_to_string((to_text_array(args))[2:], ',')\n );\n END IF;\n\n -- Look up template from cql2_ops\n IF j ? 'op' THEN\n SELECT * INTO cql2op FROM cql2_ops WHERE cql2_ops.op ilike op;\n IF FOUND THEN\n -- If specific index set in queryables for a property cast other arguments to that type\n\n RETURN format(\n cql2op.template,\n VARIADIC (to_text_array(args))\n );\n ELSE\n RAISE EXCEPTION 'Operator % Not Supported.', op;\n END IF;\n END IF;\n\n\n IF wrapper IS NOT NULL THEN\n RAISE NOTICE 'Wrapping % with %', j, wrapper;\n IF j ? 'property' THEN\n RETURN format('%I(%s)', wrapper, (queryable(j->>'property')).path);\n ELSE\n RETURN format('%I(%L)', wrapper, j);\n END IF;\n ELSIF j ? 'property' THEN\n RETURN quote_ident(j->>'property');\n END IF;\n\n RETURN quote_literal(to_text(j));\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION paging_dtrange(\n j jsonb\n) RETURNS tstzrange AS $$\nDECLARE\n op text;\n filter jsonb := j->'filter';\n dtrange tstzrange := tstzrange('-infinity'::timestamptz,'infinity'::timestamptz);\n sdate timestamptz := '-infinity'::timestamptz;\n edate timestamptz := 'infinity'::timestamptz;\n jpitem jsonb;\nBEGIN\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n END IF;\n IF NOT (filter @? '$.**.op ? (@ == \"or\" || @ == \"not\")') THEN\n FOR jpitem IN SELECT j FROM jsonb_path_query(filter,'strict $.** ? (@.args[*].property == \"datetime\")'::jsonpath) j LOOP\n op := lower(jpitem->>'op');\n dtrange := parse_dtrange(jpitem->'args'->1);\n IF op IN ('<=', 'lt', 'lte', '<', 'le', 't_before') THEN\n sdate := greatest(sdate,'-infinity');\n edate := least(edate, upper(dtrange));\n ELSIF op IN ('>=', '>', 'gt', 'gte', 'ge', 't_after') THEN\n edate := least(edate, 'infinity');\n sdate := greatest(sdate, lower(dtrange));\n ELSIF op IN ('=', 'eq') THEN\n edate := least(edate, upper(dtrange));\n sdate := greatest(sdate, lower(dtrange));\n END IF;\n RAISE NOTICE '2 OP: %, ARGS: %, DTRANGE: %, SDATE: %, EDATE: %', op, jpitem->'args'->1, dtrange, sdate, edate;\n END LOOP;\n END IF;\n IF sdate > edate THEN\n RETURN 'empty'::tstzrange;\n END IF;\n RETURN tstzrange(sdate,edate, '[]');\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT SET TIME ZONE 'UTC';\n\nCREATE OR REPLACE FUNCTION paging_collections(\n IN j jsonb\n) RETURNS text[] AS $$\nDECLARE\n filter jsonb := j->'filter';\n jpitem jsonb;\n op text;\n args jsonb;\n arg jsonb;\n collections text[];\nBEGIN\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n END IF;\n IF NOT (filter @? '$.**.op ? (@ == \"or\" || @ == \"not\")') THEN\n FOR jpitem IN SELECT j FROM jsonb_path_query(filter,'strict $.** ? (@.args[*].property == \"collection\")'::jsonpath) j LOOP\n RAISE NOTICE 'JPITEM: %', jpitem;\n op := jpitem->>'op';\n args := jpitem->'args';\n IF op IN ('=', 'eq', 'in') THEN\n FOR arg IN SELECT a FROM jsonb_array_elements(args) a LOOP\n IF jsonb_typeof(arg) IN ('string', 'array') THEN\n RAISE NOTICE 'arg: %, collections: %', arg, collections;\n IF collections IS NULL OR collections = '{}'::text[] THEN\n collections := to_text_array(arg);\n ELSE\n collections := array_intersection(collections, to_text_array(arg));\n END IF;\n END IF;\n END LOOP;\n END IF;\n END LOOP;\n END IF;\n IF collections = '{}'::text[] THEN\n RETURN NULL;\n END IF;\n RETURN collections;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n" }, { "path": "src/pgstac/sql/003a_items.sql", "content": "CREATE TABLE items (\n id text NOT NULL,\n geometry geometry NOT NULL,\n collection text NOT NULL,\n datetime timestamptz NOT NULL,\n end_datetime timestamptz NOT NULL,\n content JSONB NOT NULL,\n private jsonb\n)\nPARTITION BY LIST (collection)\n;\n\nCREATE INDEX \"datetime_idx\" ON items USING BTREE (datetime DESC, end_datetime ASC);\nCREATE INDEX \"geometry_idx\" ON items USING GIST (geometry);\n\nCREATE STATISTICS datetime_stats (dependencies) on datetime, end_datetime from items;\n\nALTER TABLE items ADD CONSTRAINT items_collections_fk FOREIGN KEY (collection) REFERENCES collections(id) ON DELETE CASCADE DEFERRABLE;\n\nCREATE OR REPLACE FUNCTION partition_after_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p text;\n t timestamptz := clock_timestamp();\nBEGIN\n RAISE NOTICE 'Updating partition stats %', t;\n FOR p IN SELECT DISTINCT partition\n FROM newdata n JOIN partition_sys_meta p\n ON (n.collection=p.collection AND n.datetime <@ p.partition_dtrange)\n LOOP\n PERFORM run_or_queue(format('SELECT update_partition_stats(%L, %L);', p, true));\n END LOOP;\n IF TG_OP IN ('DELETE','UPDATE') THEN\n DELETE FROM format_item_cache c USING newdata n WHERE c.collection = n.collection AND c.id = n.id;\n END IF;\n RAISE NOTICE 't: % %', t, clock_timestamp() - t;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE TRIGGER items_after_insert_trigger\nAFTER INSERT ON items\nREFERENCING NEW TABLE AS newdata\nFOR EACH STATEMENT\nEXECUTE FUNCTION partition_after_triggerfunc();\n\nCREATE TRIGGER items_after_update_trigger\nAFTER DELETE ON items\nREFERENCING OLD TABLE AS newdata\nFOR EACH STATEMENT\nEXECUTE FUNCTION partition_after_triggerfunc();\n\nCREATE TRIGGER items_after_delete_trigger\nAFTER UPDATE ON items\nREFERENCING NEW TABLE AS newdata\nFOR EACH STATEMENT\nEXECUTE FUNCTION partition_after_triggerfunc();\n\n\nCREATE OR REPLACE FUNCTION content_slim(_item jsonb) RETURNS jsonb AS $$\n SELECT strip_jsonb(_item - '{id,geometry,collection,type}'::text[], collection_base_item(_item->>'collection')) - '{id,geometry,collection,type}'::text[];\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_dehydrate(content jsonb) RETURNS items AS $$\n SELECT\n content->>'id' as id,\n stac_geom(content) as geometry,\n content->>'collection' as collection,\n stac_datetime(content) as datetime,\n stac_end_datetime(content) as end_datetime,\n content_slim(content) as content,\n null::jsonb as private\n ;\n$$ LANGUAGE SQL STABLE;\n\nCREATE OR REPLACE FUNCTION include_field(f text, fields jsonb DEFAULT '{}'::jsonb) RETURNS boolean AS $$\nDECLARE\n includes jsonb := fields->'include';\n excludes jsonb := fields->'exclude';\nBEGIN\n IF f IS NULL THEN\n RETURN NULL;\n END IF;\n\n\n IF\n jsonb_typeof(excludes) = 'array'\n AND jsonb_array_length(excludes)>0\n AND excludes ? f\n THEN\n RETURN FALSE;\n END IF;\n\n IF\n (\n jsonb_typeof(includes) = 'array'\n AND jsonb_array_length(includes) > 0\n AND includes ? f\n ) OR\n (\n includes IS NULL\n OR jsonb_typeof(includes) = 'null'\n OR jsonb_array_length(includes) = 0\n )\n THEN\n RETURN TRUE;\n END IF;\n\n RETURN FALSE;\nEND;\n$$ LANGUAGE PLPGSQL IMMUTABLE;\n\nDROP FUNCTION IF EXISTS content_hydrate(jsonb, jsonb, jsonb);\nCREATE OR REPLACE FUNCTION content_hydrate(\n _item jsonb,\n _base_item jsonb,\n fields jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\n SELECT merge_jsonb(\n jsonb_fields(_item, fields),\n jsonb_fields(_base_item, fields)\n );\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\n\nCREATE OR REPLACE FUNCTION content_hydrate(_item items, _collection collections, fields jsonb DEFAULT '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\n content jsonb;\n base_item jsonb := _collection.base_item;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry, 20)::jsonb;\n END IF;\n output := content_hydrate(\n jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'collection', _item.collection,\n 'type', 'Feature'\n ) || _item.content,\n _collection.base_item,\n fields\n );\n\n RETURN output;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_nonhydrated(\n _item items,\n fields jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\nDECLARE\n geom jsonb;\n bbox jsonb;\n output jsonb;\nBEGIN\n IF include_field('geometry', fields) THEN\n geom := ST_ASGeoJson(_item.geometry, 20)::jsonb;\n END IF;\n output := jsonb_build_object(\n 'id', _item.id,\n 'geometry', geom,\n 'collection', _item.collection,\n 'type', 'Feature'\n ) || _item.content;\n RETURN output;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION content_hydrate(_item items, fields jsonb DEFAULT '{}'::jsonb) RETURNS jsonb AS $$\n SELECT content_hydrate(\n _item,\n (SELECT c FROM collections c WHERE id=_item.collection LIMIT 1),\n fields\n );\n$$ LANGUAGE SQL STABLE;\n\n\nCREATE UNLOGGED TABLE items_staging (\n content JSONB NOT NULL\n);\nCREATE UNLOGGED TABLE items_staging_ignore (\n content JSONB NOT NULL\n);\nCREATE UNLOGGED TABLE items_staging_upsert (\n content JSONB NOT NULL\n);\n\nCREATE OR REPLACE FUNCTION items_staging_triggerfunc() RETURNS TRIGGER AS $$\nDECLARE\n p record;\n _partitions text[];\n part text;\n ts timestamptz := clock_timestamp();\n nrows int;\nBEGIN\n RAISE NOTICE 'Creating Partitions. %', clock_timestamp() - ts;\n\n FOR part IN WITH t AS (\n SELECT\n n.content->>'collection' as collection,\n stac_daterange(n.content->'properties') as dtr,\n partition_trunc\n FROM newdata n JOIN collections ON (n.content->>'collection'=collections.id)\n ), p AS (\n SELECT\n collection,\n COALESCE(date_trunc(partition_trunc::text, lower(dtr)),'-infinity') as d,\n tstzrange(min(lower(dtr)),max(lower(dtr)),'[]') as dtrange,\n tstzrange(min(upper(dtr)),max(upper(dtr)),'[]') as edtrange\n FROM t\n GROUP BY 1,2\n ) SELECT check_partition(collection, dtrange, edtrange) FROM p LOOP\n RAISE NOTICE 'Partition %', part;\n END LOOP;\n\n RAISE NOTICE 'Creating temp table with data to be added. %', clock_timestamp() - ts;\n DROP TABLE IF EXISTS tmpdata;\n CREATE TEMP TABLE tmpdata ON COMMIT DROP AS\n SELECT\n (content_dehydrate(content)).*\n FROM newdata;\n GET DIAGNOSTICS nrows = ROW_COUNT;\n RAISE NOTICE 'Added % rows to tmpdata. %', nrows, clock_timestamp() - ts;\n\n RAISE NOTICE 'Doing the insert. %', clock_timestamp() - ts;\n IF TG_TABLE_NAME = 'items_staging' THEN\n INSERT INTO items\n SELECT * FROM tmpdata;\n GET DIAGNOSTICS nrows = ROW_COUNT;\n RAISE NOTICE 'Inserted % rows to items. %', nrows, clock_timestamp() - ts;\n ELSIF TG_TABLE_NAME = 'items_staging_ignore' THEN\n INSERT INTO items\n SELECT * FROM tmpdata\n ON CONFLICT DO NOTHING;\n GET DIAGNOSTICS nrows = ROW_COUNT;\n RAISE NOTICE 'Inserted % rows to items. %', nrows, clock_timestamp() - ts;\n ELSIF TG_TABLE_NAME = 'items_staging_upsert' THEN\n DELETE FROM items i USING tmpdata s\n WHERE\n i.id = s.id\n AND i.collection = s.collection\n AND i IS DISTINCT FROM s\n ;\n GET DIAGNOSTICS nrows = ROW_COUNT;\n RAISE NOTICE 'Deleted % rows from items. %', nrows, clock_timestamp() - ts;\n INSERT INTO items AS t\n SELECT * FROM tmpdata\n ON CONFLICT DO NOTHING;\n GET DIAGNOSTICS nrows = ROW_COUNT;\n RAISE NOTICE 'Inserted % rows to items. %', nrows, clock_timestamp() - ts;\n END IF;\n\n RAISE NOTICE 'Deleting data from staging table. %', clock_timestamp() - ts;\n DELETE FROM items_staging;\n RAISE NOTICE 'Done. %', clock_timestamp() - ts;\n\n RETURN NULL;\n\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER items_staging_insert_trigger AFTER INSERT ON items_staging REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\nCREATE TRIGGER items_staging_insert_ignore_trigger AFTER INSERT ON items_staging_ignore REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\nCREATE TRIGGER items_staging_insert_upsert_trigger AFTER INSERT ON items_staging_upsert REFERENCING NEW TABLE AS newdata\n FOR EACH STATEMENT EXECUTE PROCEDURE items_staging_triggerfunc();\n\n\nCREATE OR REPLACE FUNCTION item_by_id(_id text, _collection text DEFAULT NULL) RETURNS items AS\n$$\nDECLARE\n i items%ROWTYPE;\nBEGIN\n SELECT * INTO i FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection) LIMIT 1;\n RETURN i;\nEND;\n$$ LANGUAGE PLPGSQL STABLE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION get_item(_id text, _collection text DEFAULT NULL) RETURNS jsonb AS $$\n SELECT content_hydrate(items) FROM items WHERE id=_id AND (_collection IS NULL OR collection=_collection);\n$$ LANGUAGE SQL STABLE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION delete_item(_id text, _collection text DEFAULT NULL) RETURNS VOID AS $$\nDECLARE\nout items%ROWTYPE;\nBEGIN\n DELETE FROM items WHERE id = _id AND (_collection IS NULL OR collection=_collection) RETURNING * INTO STRICT out;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n--/*\nCREATE OR REPLACE FUNCTION create_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION update_item(content jsonb) RETURNS VOID AS $$\nDECLARE\n old items %ROWTYPE;\n out items%ROWTYPE;\nBEGIN\n PERFORM delete_item(content->>'id', content->>'collection');\n PERFORM create_item(content);\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_item(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content) VALUES (data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION create_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION upsert_items(data jsonb) RETURNS VOID AS $$\n INSERT INTO items_staging_upsert (content)\n SELECT * FROM jsonb_array_elements(data);\n$$ LANGUAGE SQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE OR REPLACE FUNCTION collection_bbox(id text) RETURNS jsonb AS $$\n SELECT (replace(replace(replace(st_extent(geometry)::text,'BOX(','[['),')',']]'),' ',','))::jsonb\n FROM items WHERE collection=$1;\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION collection_temporal_extent(id text) RETURNS jsonb AS $$\n SELECT to_jsonb(array[array[min(datetime), max(datetime)]])\n FROM items WHERE collection=$1;\n;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION update_collection_extents() RETURNS VOID AS $$\nUPDATE collections\n SET content = jsonb_set_lax(\n content,\n '{extent}'::text[],\n collection_extent(id, FALSE),\n true,\n 'use_json_null'\n )\n;\n$$ LANGUAGE SQL;\n" }, { "path": "src/pgstac/sql/003b_partitions.sql", "content": "CREATE TABLE partition_stats (\n partition text PRIMARY KEY,\n dtrange tstzrange,\n edtrange tstzrange,\n spatial geometry,\n last_updated timestamptz,\n keys text[]\n) WITH (FILLFACTOR=90);\n\nCREATE INDEX partitions_range_idx ON partition_stats USING GIST(dtrange);\n\n\nCREATE OR REPLACE FUNCTION constraint_tstzrange(expr text) RETURNS tstzrange AS $$\n WITH t AS (\n SELECT regexp_matches(\n expr,\n E'\\\\(''\\([0-9 :+-]*\\)''\\\\).*\\\\(''\\([0-9 :+-]*\\)''\\\\)'\n ) AS m\n ) SELECT tstzrange(m[1]::timestamptz, m[2]::timestamptz) FROM t\n ;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE STRICT;\n\nCREATE OR REPLACE FUNCTION get_tstz_constraint(reloid oid, colname text) RETURNS tstzrange AS $$\nDECLARE\n expr text := NULL;\n m text[];\n ts_lower timestamptz := NULL;\n ts_upper timestamptz := NULL;\n lower_inclusive text := '[';\n upper_inclusive text := ']';\n ts timestamptz;\nBEGIN\n SELECT INTO expr\n string_agg(def, ' AND ')\n FROM pg_constraint JOIN LATERAL pg_get_constraintdef(oid) AS def ON TRUE\n WHERE\n conrelid = reloid\n AND contype = 'c'\n AND def LIKE '%' || colname || '%'\n ;\n\n IF expr IS NULL THEN\n RETURN NULL;\n END IF;\n\n RAISE DEBUG 'Constraint expression for % on %: %', colname, reloid::regclass, expr;\n -- collect all constraints for the specified column\n FOR m IN SELECT regexp_matches(expr, '[ (]' || colname || $expr$\\s*([<>=]{1,2})\\s*'([0-9 :.+\\-]+)'$expr$, 'g') LOOP\n ts := m[2]::timestamptz;\n IF m[1] IN ('>', '>=')\n THEN\n IF ts_lower IS NULL OR ts > ts_lower OR (ts = ts_lower AND m[1] = '>') THEN\n ts_lower := ts;\n lower_inclusive := CASE WHEN m[1] = '>' THEN '(' ELSE '[' END;\n END IF;\n ELSIF m[1] IN ('<', '<=')\n THEN\n IF ts_upper IS NULL OR ts < ts_upper OR (ts = ts_upper AND m[1] = '<') THEN\n ts_upper := ts;\n upper_inclusive := CASE WHEN m[1] = '<' THEN ')' ELSE ']' END;\n END IF;\n END IF;\n END LOOP;\n RAISE DEBUG 'Constraint % for %: % %', colname, reloid::regclass, ts_lower, ts_upper;\n RETURN tstzrange(ts_lower, ts_upper, lower_inclusive || upper_inclusive);\nEND;\n$$ LANGUAGE plpgsql STRICT STABLE;\n\nCREATE OR REPLACE FUNCTION get_partition_name(relid regclass) RETURNS text AS $$\n SELECT (parse_ident(relid::text))[cardinality(parse_ident(relid::text))];\n$$ LANGUAGE SQL STABLE STRICT;\n\nCREATE OR REPLACE VIEW partition_sys_meta AS\nSELECT\n partition,\n replace(\n replace(\n CASE WHEN level = 1 THEN partition_expr ELSE parent_partition_expr END,\n 'FOR VALUES IN (''',\n ''\n ),\n ''')',\n ''\n ) AS collection,\n level,\n c.reltuples,\n c.relhastriggers,\n partition_dtrange,\n COALESCE(\n get_tstz_constraint(c.oid, 'datetime'),\n partition_dtrange,\n inf_range\n ) as constraint_dtrange,\n COALESCE(\n get_tstz_constraint(c.oid, 'end_datetime'),\n inf_range\n ) as constraint_edtrange\nFROM\n pg_partition_tree('items')\n JOIN pg_class c ON (relid::regclass = c.oid)\n JOIN pg_class parent ON (parentrelid::regclass = parent.oid AND isleaf)\n LEFT JOIN pg_constraint edt ON (conrelid=c.oid AND contype='c')\n JOIN LATERAL get_partition_name(relid) AS partition ON TRUE\n JOIN LATERAL pg_get_expr(c.relpartbound, c.oid) as partition_expr ON TRUE\n JOIN LATERAL pg_get_expr(parent.relpartbound, parent.oid) as parent_partition_expr ON TRUE\n JOIN LATERAL tstzrange('-infinity', 'infinity','[]') as inf_range ON TRUE\n JOIN LATERAL COALESCE(constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), inf_range) as partition_dtrange ON TRUE\n JOIN LATERAL get_tstz_constraint(c.oid, 'datetime') as datetime_constraint ON TRUE\n JOIN LATERAL get_tstz_constraint(c.oid, 'end_datetime') as end_datetime_constraint ON TRUE\nWHERE isleaf\n;\n\nCREATE OR REPLACE VIEW partitions_view AS\nSELECT\n (parse_ident(relid::text))[cardinality(parse_ident(relid::text))] as partition,\n replace(\n replace(\n CASE WHEN level = 1 THEN partition_expr ELSE parent_partition_expr END,\n 'FOR VALUES IN (''',\n ''\n ),\n ''')',\n ''\n ) AS collection,\n level,\n c.reltuples,\n c.relhastriggers,\n partition_dtrange,\n COALESCE(\n get_tstz_constraint(c.oid, 'datetime'),\n partition_dtrange,\n inf_range\n ) as constraint_dtrange,\n COALESCE(\n get_tstz_constraint(c.oid, 'end_datetime'),\n inf_range\n ) as constraint_edtrange,\n dtrange,\n edtrange,\n spatial,\n last_updated\nFROM\n pg_partition_tree('items')\n JOIN pg_class c ON (relid::regclass = c.oid)\n JOIN pg_class parent ON (parentrelid::regclass = parent.oid AND isleaf)\n LEFT JOIN pg_constraint edt ON (conrelid=c.oid AND contype='c')\n JOIN LATERAL get_partition_name(relid) AS partition ON TRUE\n JOIN LATERAL pg_get_expr(c.relpartbound, c.oid) as partition_expr ON TRUE\n JOIN LATERAL pg_get_expr(parent.relpartbound, parent.oid) as parent_partition_expr ON TRUE\n JOIN LATERAL tstzrange('-infinity', 'infinity','[]') as inf_range ON TRUE\n JOIN LATERAL COALESCE(constraint_tstzrange(pg_get_expr(c.relpartbound, c.oid)), inf_range) as partition_dtrange ON TRUE\n JOIN LATERAL get_tstz_constraint(c.oid, 'datetime') as datetime_constraint ON TRUE\n JOIN LATERAL get_tstz_constraint(c.oid, 'end_datetime') as end_datetime_constraint ON TRUE\n LEFT JOIN pgstac.partition_stats USING (partition)\nWHERE isleaf\n;\n\nCREATE MATERIALIZED VIEW partitions AS\nSELECT * FROM partitions_view;\nCREATE UNIQUE INDEX ON partitions (partition);\n\nCREATE MATERIALIZED VIEW partition_steps AS\nSELECT\n partition as name,\n date_trunc('month',lower(partition_dtrange)) as sdate,\n date_trunc('month', upper(partition_dtrange)) + '1 month'::interval as edate\n FROM partitions_view WHERE partition_dtrange IS NOT NULL AND partition_dtrange != 'empty'::tstzrange\n ORDER BY dtrange ASC\n;\n\n\nCREATE OR REPLACE FUNCTION update_partition_stats_q(_partition text, istrigger boolean default false) RETURNS VOID AS $$\nDECLARE\nBEGIN\n PERFORM run_or_queue(\n format('SELECT update_partition_stats(%L, %L);', _partition, istrigger)\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION update_partition_stats(_partition text, istrigger boolean default false) RETURNS VOID AS $$\nDECLARE\n dtrange tstzrange;\n edtrange tstzrange;\n cdtrange tstzrange;\n cedtrange tstzrange;\n extent geometry;\n collection text;\nBEGIN\n RAISE NOTICE 'Updating stats for %.', _partition;\n EXECUTE format(\n $q$\n SELECT\n tstzrange(min(datetime), max(datetime),'[]'),\n tstzrange(min(end_datetime), max(end_datetime), '[]')\n FROM %I\n $q$,\n _partition\n ) INTO dtrange, edtrange;\n EXECUTE format('ANALYZE %I;', _partition);\n extent := st_estimatedextent('pgstac', _partition, 'geometry');\n RAISE DEBUG 'Estimated Extent: %', extent;\n INSERT INTO partition_stats (partition, dtrange, edtrange, spatial, last_updated)\n SELECT _partition, dtrange, edtrange, extent, now()\n ON CONFLICT (partition) DO\n UPDATE SET\n dtrange=EXCLUDED.dtrange,\n edtrange=EXCLUDED.edtrange,\n spatial=EXCLUDED.spatial,\n last_updated=EXCLUDED.last_updated\n ;\n\n SELECT\n constraint_dtrange, constraint_edtrange, pv.collection\n INTO cdtrange, cedtrange, collection\n FROM partitions_view pv WHERE partition = _partition;\n\n RAISE NOTICE 'Checking if we need to modify constraints...';\n RAISE NOTICE 'cdtrange: % dtrange: % cedtrange: % edtrange: %',cdtrange, dtrange, cedtrange, edtrange;\n IF\n (cdtrange IS DISTINCT FROM dtrange OR edtrange IS DISTINCT FROM cedtrange)\n AND NOT istrigger\n THEN\n RAISE NOTICE 'Modifying Constraints';\n RAISE NOTICE 'Existing % %', cdtrange, cedtrange;\n RAISE NOTICE 'New % %', dtrange, edtrange;\n PERFORM drop_table_constraints(_partition);\n PERFORM create_table_constraints(_partition, dtrange, edtrange);\n END IF;\n REFRESH MATERIALIZED VIEW partitions;\n REFRESH MATERIALIZED VIEW partition_steps;\n RAISE NOTICE 'Checking if we need to update collection extents.';\n IF get_setting_bool('update_collection_extent') THEN\n RAISE NOTICE 'updating collection extent for %', collection;\n PERFORM run_or_queue(format($q$\n UPDATE collections\n SET content = jsonb_set_lax(\n content,\n '{extent}'::text[],\n collection_extent(%L, FALSE),\n true,\n 'use_json_null'\n ) WHERE id=%L\n ;\n $q$, collection, collection));\n ELSE\n RAISE NOTICE 'Not updating collection extent for %', collection;\n END IF;\n\nEND;\n$$ LANGUAGE PLPGSQL STRICT SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION partition_name( IN collection text, IN dt timestamptz, OUT partition_name text, OUT partition_range tstzrange) AS $$\nDECLARE\n c RECORD;\n parent_name text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n parent_name := format('_items_%s', c.key);\n\n\n IF c.partition_trunc = 'year' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYY'));\n ELSIF c.partition_trunc = 'month' THEN\n partition_name := format('%s_%s', parent_name, to_char(dt,'YYYYMM'));\n ELSE\n partition_name := parent_name;\n partition_range := tstzrange('-infinity'::timestamptz, 'infinity'::timestamptz, '[]');\n END IF;\n IF partition_range IS NULL THEN\n partition_range := tstzrange(\n date_trunc(c.partition_trunc::text, dt),\n date_trunc(c.partition_trunc::text, dt) + concat('1 ', c.partition_trunc)::interval\n );\n END IF;\n RETURN;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION drop_table_constraints(t text) RETURNS text AS $$\nDECLARE\n q text;\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM partitions_view WHERE partition=t) THEN\n RETURN NULL;\n END IF;\n FOR q IN SELECT FORMAT(\n $q$\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n $q$,\n t,\n conname\n ) FROM pg_constraint\n WHERE conrelid=t::regclass::oid AND contype='c'\n LOOP\n EXECUTE q;\n END LOOP;\n RETURN t;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE OR REPLACE FUNCTION create_table_constraints(t text, _dtrange tstzrange, _edtrange tstzrange) RETURNS text AS $$\nDECLARE\n q text;\nBEGIN\n IF NOT EXISTS (SELECT 1 FROM partitions_view WHERE partition=t) THEN\n RETURN NULL;\n END IF;\n RAISE NOTICE 'Creating Table Constraints for % % %', t, _dtrange, _edtrange;\n IF _dtrange = 'empty' AND _edtrange = 'empty' THEN\n q :=format(\n $q$\n DO $block$\n BEGIN\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n ALTER TABLE %I\n ADD CONSTRAINT %I\n CHECK (((datetime IS NULL) AND (end_datetime IS NULL))) NOT VALID\n ;\n ALTER TABLE %I\n VALIDATE CONSTRAINT %I\n ;\n\n\n\n EXCEPTION WHEN others THEN\n RAISE WARNING '%%, Issue Altering Constraints. Please run update_partition_stats(%I)', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n $block$;\n $q$,\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t),\n t\n );\n ELSE\n q :=format(\n $q$\n DO $block$\n BEGIN\n\n ALTER TABLE %I DROP CONSTRAINT IF EXISTS %I;\n ALTER TABLE %I\n ADD CONSTRAINT %I\n CHECK (\n (datetime >= %L)\n AND (datetime <= %L)\n AND (end_datetime >= %L)\n AND (end_datetime <= %L)\n ) NOT VALID\n ;\n ALTER TABLE %I\n VALIDATE CONSTRAINT %I\n ;\n\n\n\n EXCEPTION WHEN others THEN\n RAISE WARNING '%%, Issue Altering Constraints. Please run update_partition_stats(%I)', SQLERRM USING ERRCODE = SQLSTATE;\n END;\n $block$;\n $q$,\n t,\n format('%s_dt', t),\n t,\n format('%s_dt', t),\n lower(_dtrange),\n upper(_dtrange),\n lower(_edtrange),\n upper(_edtrange),\n t,\n format('%s_dt', t),\n t\n );\n END IF;\n PERFORM run_or_queue(q);\n RETURN t;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION check_partition(\n _collection text,\n _dtrange tstzrange,\n _edtrange tstzrange\n) RETURNS text AS $$\nDECLARE\n c RECORD;\n pm RECORD;\n _partition_name text;\n _partition_dtrange tstzrange;\n _constraint_dtrange tstzrange;\n _constraint_edtrange tstzrange;\n q text;\n deferrable_q text;\n err_context text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=_collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', _collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n\n IF c.partition_trunc IS NOT NULL THEN\n _partition_dtrange := tstzrange(\n date_trunc(c.partition_trunc, lower(_dtrange)),\n date_trunc(c.partition_trunc, lower(_dtrange)) + (concat('1 ', c.partition_trunc))::interval,\n '[)'\n );\n ELSE\n _partition_dtrange := '[-infinity, infinity]'::tstzrange;\n END IF;\n\n IF NOT _partition_dtrange @> _dtrange THEN\n RAISE EXCEPTION 'dtrange % is greater than the partition size % for collection %', _dtrange, c.partition_trunc, _collection;\n END IF;\n\n\n IF c.partition_trunc = 'year' THEN\n _partition_name := format('_items_%s_%s', c.key, to_char(lower(_partition_dtrange),'YYYY'));\n ELSIF c.partition_trunc = 'month' THEN\n _partition_name := format('_items_%s_%s', c.key, to_char(lower(_partition_dtrange),'YYYYMM'));\n ELSE\n _partition_name := format('_items_%s', c.key);\n END IF;\n\n SELECT * INTO pm FROM partition_sys_meta WHERE collection=_collection AND partition_dtrange @> _dtrange;\n IF FOUND THEN\n RAISE NOTICE '% % %', _edtrange, _dtrange, pm;\n _constraint_edtrange :=\n tstzrange(\n least(\n lower(_edtrange),\n nullif(lower(pm.constraint_edtrange), '-infinity')\n ),\n greatest(\n upper(_edtrange),\n nullif(upper(pm.constraint_edtrange), 'infinity')\n ),\n '[]'\n );\n _constraint_dtrange :=\n tstzrange(\n least(\n lower(_dtrange),\n nullif(lower(pm.constraint_dtrange), '-infinity')\n ),\n greatest(\n upper(_dtrange),\n nullif(upper(pm.constraint_dtrange), 'infinity')\n ),\n '[]'\n );\n\n IF pm.constraint_edtrange @> _edtrange AND pm.constraint_dtrange @> _dtrange THEN\n RETURN pm.partition;\n ELSE\n PERFORM drop_table_constraints(_partition_name);\n END IF;\n ELSE\n _constraint_edtrange := _edtrange;\n _constraint_dtrange := _dtrange;\n END IF;\n RAISE NOTICE 'EXISTING CONSTRAINTS % %, NEW % %', pm.constraint_dtrange, pm.constraint_edtrange, _constraint_dtrange, _constraint_edtrange;\n RAISE NOTICE 'Creating partition % %', _partition_name, _partition_dtrange;\n IF c.partition_trunc IS NULL THEN\n q := format(\n $q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n GRANT ALL ON %I to pgstac_ingest;\n $q$,\n _partition_name,\n _collection,\n concat(_partition_name,'_pk'),\n _partition_name,\n _partition_name\n );\n ELSE\n q := format(\n $q$\n CREATE TABLE IF NOT EXISTS %I partition OF items FOR VALUES IN (%L) PARTITION BY RANGE (datetime);\n CREATE TABLE IF NOT EXISTS %I partition OF %I FOR VALUES FROM (%L) TO (%L);\n CREATE UNIQUE INDEX IF NOT EXISTS %I ON %I (id);\n GRANT ALL ON %I TO pgstac_ingest;\n $q$,\n format('_items_%s', c.key),\n _collection,\n _partition_name,\n format('_items_%s', c.key),\n lower(_partition_dtrange),\n upper(_partition_dtrange),\n format('%s_pk', _partition_name),\n _partition_name,\n _partition_name\n );\n END IF;\n\n BEGIN\n EXECUTE q;\n EXCEPTION\n WHEN duplicate_table THEN\n RAISE NOTICE 'Partition % already exists.', _partition_name;\n WHEN others THEN\n GET STACKED DIAGNOSTICS err_context = PG_EXCEPTION_CONTEXT;\n RAISE INFO 'Error Name:%',SQLERRM;\n RAISE INFO 'Error State:%', SQLSTATE;\n RAISE INFO 'Error Context:%', err_context;\n END;\n PERFORM maintain_partitions(_partition_name);\n PERFORM update_partition_stats_q(_partition_name, true);\n REFRESH MATERIALIZED VIEW partitions;\n REFRESH MATERIALIZED VIEW partition_steps;\n RETURN _partition_name;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION repartition(_collection text, _partition_trunc text, triggered boolean DEFAULT FALSE) RETURNS text AS $$\nDECLARE\n c RECORD;\n q text;\n from_trunc text;\nBEGIN\n SELECT * INTO c FROM pgstac.collections WHERE id=_collection;\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Collection % does not exist', _collection USING ERRCODE = 'foreign_key_violation', HINT = 'Make sure collection exists before adding items';\n END IF;\n IF triggered THEN\n RAISE NOTICE 'Converting % to % partitioning via Trigger', _collection, _partition_trunc;\n ELSE\n RAISE NOTICE 'Converting % from using % to % partitioning', _collection, c.partition_trunc, _partition_trunc;\n IF c.partition_trunc IS NOT DISTINCT FROM _partition_trunc THEN\n RAISE NOTICE 'Collection % already set to use partition by %', _collection, _partition_trunc;\n RETURN _collection;\n END IF;\n END IF;\n\n IF EXISTS (SELECT 1 FROM partitions_view WHERE collection=_collection LIMIT 1) THEN\n EXECUTE format(\n $q$\n CREATE TEMP TABLE changepartitionstaging ON COMMIT DROP AS SELECT * FROM %I;\n DROP TABLE IF EXISTS %I CASCADE;\n WITH p AS (\n SELECT\n collection,\n CASE\n WHEN %L IS NULL THEN '-infinity'::timestamptz\n ELSE date_trunc(%L, datetime)\n END as d,\n tstzrange(min(datetime),max(datetime),'[]') as dtrange,\n tstzrange(min(end_datetime),max(end_datetime),'[]') as edtrange\n FROM changepartitionstaging\n GROUP BY 1,2\n ) SELECT check_partition(collection, dtrange, edtrange) FROM p;\n INSERT INTO items SELECT * FROM changepartitionstaging;\n DROP TABLE changepartitionstaging;\n $q$,\n concat('_items_', c.key),\n concat('_items_', c.key),\n c.partition_trunc,\n c.partition_trunc\n );\n END IF;\n RETURN _collection;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE OR REPLACE FUNCTION collections_trigger_func() RETURNS TRIGGER AS $$\nDECLARE\n q text;\n partition_name text := format('_items_%s', NEW.key);\n partition_exists boolean := false;\n partition_empty boolean := true;\n err_context text;\n loadtemp boolean := FALSE;\nBEGIN\n RAISE NOTICE 'Collection Trigger. % %', NEW.id, NEW.key;\n IF TG_OP = 'UPDATE' AND NEW.partition_trunc IS DISTINCT FROM OLD.partition_trunc THEN\n PERFORM repartition(NEW.id, NEW.partition_trunc, TRUE);\n END IF;\n RETURN NEW;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE TRIGGER collections_trigger AFTER\nINSERT\nOR\nUPDATE ON collections\nFOR EACH ROW EXECUTE FUNCTION collections_trigger_func();\n" }, { "path": "src/pgstac/sql/004_search.sql", "content": "\nCREATE OR REPLACE FUNCTION chunker(\n IN _where text,\n OUT s timestamptz,\n OUT e timestamptz\n) RETURNS SETOF RECORD AS $$\nDECLARE\n explain jsonb;\nBEGIN\n IF _where IS NULL THEN\n _where := ' TRUE ';\n END IF;\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s;', _where)\n INTO explain;\n RAISE DEBUG 'EXPLAIN: %', explain;\n\n RETURN QUERY\n WITH t AS (\n SELECT j->>0 as p FROM\n jsonb_path_query(\n explain,\n 'strict $.**.\"Relation Name\" ? (@ != null)'\n ) j\n ),\n parts AS (\n SELECT sdate, edate FROM t JOIN partition_steps ON (t.p = name)\n ),\n times AS (\n SELECT sdate FROM parts\n UNION\n SELECT edate FROM parts\n ),\n uniq AS (\n SELECT DISTINCT sdate FROM times ORDER BY sdate\n ),\n last AS (\n SELECT sdate, lead(sdate, 1) over () as edate FROM uniq\n )\n SELECT sdate, edate FROM last WHERE edate IS NOT NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION partition_queries(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN partitions text[] DEFAULT NULL\n) RETURNS SETOF text AS $$\nDECLARE\n query text;\n sdate timestamptz;\n edate timestamptz;\nBEGIN\nIF _where IS NULL OR trim(_where) = '' THEN\n _where = ' TRUE ';\nEND IF;\nRAISE NOTICE 'Getting chunks for % %', _where, _orderby;\nIF _orderby ILIKE 'datetime d%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 DESC LOOP\n RETURN NEXT format($q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n $q$,\n sdate,\n edate,\n _where,\n _orderby\n );\n END LOOP;\nELSIF _orderby ILIKE 'datetime a%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 ASC LOOP\n RETURN NEXT format($q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n $q$,\n sdate,\n edate,\n _where,\n _orderby\n );\n END LOOP;\nELSE\n query := format($q$\n SELECT * FROM items\n WHERE %s\n ORDER BY %s\n $q$, _where, _orderby\n );\n\n RETURN NEXT query;\n RETURN;\nEND IF;\n\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\nCREATE OR REPLACE FUNCTION partition_query_view(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN _limit int DEFAULT 10\n) RETURNS text AS $$\n WITH p AS (\n SELECT * FROM partition_queries(_where, _orderby) p\n )\n SELECT\n CASE WHEN EXISTS (SELECT 1 FROM p) THEN\n (SELECT format($q$\n SELECT * FROM (\n %s\n ) total LIMIT %s\n $q$,\n string_agg(\n format($q$ SELECT * FROM ( %s ) AS sub $q$, p),\n '\n UNION ALL\n '\n ),\n _limit\n ))\n ELSE NULL\n END FROM p;\n$$ LANGUAGE SQL IMMUTABLE;\n\n\nCREATE OR REPLACE FUNCTION q_to_tsquery (jinput jsonb)\n RETURNS tsquery\n AS $$\nDECLARE\n input text;\n processed_text text;\n temp_text text;\n quote_array text[];\n placeholder text := '@QUOTE@';\nBEGIN\n IF jsonb_typeof(jinput) = 'string' THEN\n input := jinput->>0;\n ELSIF jsonb_typeof(jinput) = 'array' THEN\n input := array_to_string(\n array(select jsonb_array_elements_text(jinput)),\n ' OR '\n );\n ELSE\n RAISE EXCEPTION 'Input must be a string or an array of strings.';\n END IF;\n -- Extract all quoted phrases and store in array\n quote_array := regexp_matches(input, '\"[^\"]*\"', 'g');\n\n -- Replace each quoted part with a unique placeholder if there are any quoted phrases\n IF array_length(quote_array, 1) IS NOT NULL THEN\n processed_text := input;\n FOR i IN array_lower(quote_array, 1) .. array_upper(quote_array, 1) LOOP\n processed_text := replace(processed_text, quote_array[i], placeholder || i || placeholder);\n END LOOP;\n ELSE\n processed_text := input;\n END IF;\n\n -- Replace non-quoted text using regular expressions\n\n -- , -> |\n processed_text := regexp_replace(processed_text, ',(?=(?:[^\"]*\"[^\"]*\")*[^\"]*$)', ' | ', 'g');\n\n -- and -> &\n processed_text := regexp_replace(processed_text, '\\s+AND\\s+', ' & ', 'gi');\n\n -- or -> |\n processed_text := regexp_replace(processed_text, '\\s+OR\\s+', ' | ', 'gi');\n\n -- + ->\n processed_text := regexp_replace(processed_text, '^\\s*\\+([a-zA-Z0-9_]+)', '\\1', 'g'); -- +term at start\n processed_text := regexp_replace(processed_text, '\\s*\\+([a-zA-Z0-9_]+)', ' & \\1', 'g'); -- +term elsewhere\n\n -- - -> !\n processed_text := regexp_replace(processed_text, '^\\s*\\-([a-zA-Z0-9_]+)', '! \\1', 'g'); -- -term at start\n processed_text := regexp_replace(processed_text, '\\s*\\-([a-zA-Z0-9_]+)', ' & ! \\1', 'g'); -- -term elsewhere\n\n -- terms separated with spaces are assumed to represent adjacent terms. loop through these\n -- occurrences and replace them with the adjacency operator (<->)\n LOOP\n temp_text := regexp_replace(processed_text, '([a-zA-Z0-9_]+)\\s+([a-zA-Z0-9_]+)(?!\\s*[&|<>])', '\\1 <-> \\2', 'g');\n IF temp_text = processed_text THEN\n EXIT; -- No more replacements were made\n END IF;\n processed_text := temp_text;\n END LOOP;\n\n\n -- Replace placeholders back with quoted phrases if there were any\n IF array_length(quote_array, 1) IS NOT NULL THEN\n FOR i IN array_lower(quote_array, 1) .. array_upper(quote_array, 1) LOOP\n processed_text := replace(processed_text, placeholder || i || placeholder, '''' || substring(quote_array[i] from 2 for length(quote_array[i]) - 2) || '''');\n END LOOP;\n END IF;\n\n -- Print processed_text to the console for debugging purposes\n RAISE NOTICE 'processed_text: %', processed_text;\n\n RETURN to_tsquery('english', processed_text);\nEND;\n$$\nLANGUAGE plpgsql;\n\n\nCREATE OR REPLACE FUNCTION stac_search_to_where(j jsonb) RETURNS text AS $$\nDECLARE\n where_segments text[];\n _where text;\n dtrange tstzrange;\n collections text[];\n geom geometry;\n sdate timestamptz;\n edate timestamptz;\n filterlang text;\n filter jsonb := j->'filter';\n ft_query tsquery;\nBEGIN\n IF j ? 'ids' THEN\n where_segments := where_segments || format('id = ANY (%L) ', to_text_array(j->'ids'));\n END IF;\n\n IF j ? 'collections' THEN\n collections := to_text_array(j->'collections');\n where_segments := where_segments || format('collection = ANY (%L) ', collections);\n END IF;\n\n IF j ? 'datetime' THEN\n dtrange := parse_dtrange(j->'datetime');\n sdate := lower(dtrange);\n edate := upper(dtrange);\n\n where_segments := where_segments || format(' datetime <= %L::timestamptz AND end_datetime >= %L::timestamptz ',\n edate,\n sdate\n );\n END IF;\n\n IF j ? 'q' THEN\n ft_query := q_to_tsquery(j->'q');\n where_segments := where_segments || format(\n $quote$\n (\n to_tsvector('english', content->'properties'->>'description') ||\n to_tsvector('english', coalesce(content->'properties'->>'title', '')) ||\n to_tsvector('english', coalesce(content->'properties'->>'keywords', ''))\n ) @@ %L\n $quote$,\n ft_query\n );\n END IF;\n\n geom := stac_geom(j);\n IF geom IS NOT NULL THEN\n where_segments := where_segments || format('st_intersects(geometry, %L)',geom);\n END IF;\n\n filterlang := COALESCE(\n j->>'filter-lang',\n get_setting('default_filter_lang', j->'conf')\n );\n IF NOT filter @? '$.**.op' THEN\n filterlang := 'cql-json';\n END IF;\n\n IF filterlang NOT IN ('cql-json','cql2-json') AND j ? 'filter' THEN\n RAISE EXCEPTION '% is not a supported filter-lang. Please use cql-json or cql2-json.', filterlang;\n END IF;\n\n IF j ? 'query' AND j ? 'filter' THEN\n RAISE EXCEPTION 'Can only use either query or filter at one time.';\n END IF;\n\n IF j ? 'query' THEN\n filter := query_to_cql2(j->'query');\n ELSIF filterlang = 'cql-json' THEN\n filter := cql1_to_cql2(filter);\n END IF;\n RAISE NOTICE 'FILTER: %', filter;\n where_segments := where_segments || cql2_query(filter);\n IF cardinality(where_segments) < 1 THEN\n RETURN ' TRUE ';\n END IF;\n\n _where := array_to_string(array_remove(where_segments, NULL), ' AND ');\n\n IF _where IS NULL OR BTRIM(_where) = '' THEN\n RETURN ' TRUE ';\n END IF;\n RETURN _where;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE;\n\n\nCREATE OR REPLACE FUNCTION parse_sort_dir(_dir text, reverse boolean default false) RETURNS text AS $$\n WITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n ) SELECT\n CASE\n WHEN NOT reverse THEN d\n WHEN d = 'ASC' THEN 'DESC'\n WHEN d = 'DESC' THEN 'ASC'\n END\n FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION sort_dir_to_op(_dir text, prev boolean default false) RETURNS text AS $$\n WITH t AS (\n SELECT COALESCE(upper(_dir), 'ASC') as d\n ) SELECT\n CASE\n WHEN d = 'ASC' AND prev THEN '<='\n WHEN d = 'DESC' AND prev THEN '>='\n WHEN d = 'ASC' THEN '>='\n WHEN d = 'DESC' THEN '<='\n END\n FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION sort_sqlorderby(\n _search jsonb DEFAULT NULL,\n reverse boolean DEFAULT FALSE\n) RETURNS text AS $$\n WITH sortby AS (\n SELECT coalesce(_search->'sortby','[{\"field\":\"datetime\", \"direction\":\"desc\"}]') as sort\n ), withid AS (\n SELECT CASE\n WHEN sort @? '$[*] ? (@.field == \"id\")' THEN sort\n ELSE sort || '[{\"field\":\"id\", \"direction\":\"desc\"}]'::jsonb\n END as sort\n FROM sortby\n ), withid_rows AS (\n SELECT jsonb_array_elements(sort) as value FROM withid\n ),sorts AS (\n SELECT\n coalesce(\n (queryable(value->>'field')).expression\n ) as key,\n parse_sort_dir(value->>'direction', reverse) as dir\n FROM withid_rows\n )\n SELECT array_to_string(\n array_agg(concat(key, ' ', dir)),\n ', '\n ) FROM sorts;\n$$ LANGUAGE SQL;\n\nCREATE OR REPLACE FUNCTION get_sort_dir(sort_item jsonb) RETURNS text AS $$\n SELECT CASE WHEN sort_item->>'direction' ILIKE 'desc%' THEN 'DESC' ELSE 'ASC' END;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\n\nCREATE OR REPLACE FUNCTION get_token_val_str(\n _field text,\n _item items\n) RETURNS text AS $$\nDECLARE\n q text;\n literal text;\nBEGIN\n q := format($q$ SELECT quote_literal(%s) FROM (SELECT $1.*) as r;$q$, _field);\n EXECUTE q INTO literal USING _item;\n RETURN literal;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\n\nCREATE OR REPLACE FUNCTION get_token_record(IN _token text, OUT prev BOOLEAN, OUT item items) RETURNS RECORD AS $$\nDECLARE\n _itemid text := _token;\n _collectionid text;\nBEGIN\n IF _token IS NULL THEN\n RETURN;\n END IF;\n RAISE NOTICE 'Looking for token: %', _token;\n prev := FALSE;\n IF _token ILIKE 'prev:%' THEN\n _itemid := replace(_token, 'prev:','');\n prev := TRUE;\n ELSIF _token ILIKE 'next:%' THEN\n _itemid := replace(_token, 'next:', '');\n END IF;\n SELECT id INTO _collectionid FROM collections WHERE _itemid LIKE concat(id,':%');\n IF FOUND THEN\n _itemid := replace(_itemid, concat(_collectionid,':'), '');\n SELECT * INTO item FROM items WHERE id=_itemid AND collection=_collectionid;\n ELSE\n SELECT * INTO item FROM items WHERE id=_itemid;\n END IF;\n IF item IS NULL THEN\n RAISE EXCEPTION 'Could not find item using token: % item: % collection: %', _token, _itemid, _collectionid;\n END IF;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE STRICT;\n\n\nCREATE OR REPLACE FUNCTION get_token_filter(\n _sortby jsonb DEFAULT '[{\"field\":\"datetime\",\"direction\":\"desc\"}]'::jsonb,\n token_item items DEFAULT NULL,\n prev boolean DEFAULT FALSE,\n inclusive boolean DEFAULT FALSE\n) RETURNS text AS $$\nDECLARE\n ltop text := '<';\n gtop text := '>';\n dir text;\n sort record;\n orfilter text := '';\n orfilters text[] := '{}'::text[];\n andfilters text[] := '{}'::text[];\n output text;\n token_where text;\nBEGIN\n IF _sortby IS NULL OR _sortby = '[]'::jsonb THEN\n _sortby := '[{\"field\":\"datetime\",\"direction\":\"desc\"}]'::jsonb;\n END IF;\n _sortby := _sortby || jsonb_build_object('field','id','direction',_sortby->0->>'direction');\n RAISE NOTICE 'Getting Token Filter. % %', _sortby, token_item;\n IF inclusive THEN\n orfilters := orfilters || format('( id=%L AND collection=%L )' , token_item.id, token_item.collection);\n END IF;\n\n FOR sort IN\n WITH s1 AS (\n SELECT\n _row,\n (queryable(value->>'field')).expression as _field,\n (value->>'field' = 'id') as _isid,\n get_sort_dir(value) as _dir\n FROM jsonb_array_elements(_sortby)\n WITH ORDINALITY AS t(value, _row)\n )\n SELECT\n _row,\n _field,\n _dir,\n get_token_val_str(_field, token_item) as _val\n FROM s1\n WHERE _row <= (SELECT min(_row) FROM s1 WHERE _isid)\n LOOP\n orfilter := NULL;\n RAISE NOTICE 'SORT: %', sort;\n IF sort._val IS NOT NULL AND ((prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC')) THEN\n orfilter := format($f$(\n (%s %s %s) OR (%s IS NULL)\n )$f$,\n sort._field,\n ltop,\n sort._val,\n sort._val\n );\n ELSIF sort._val IS NULL AND ((prev AND sort._dir = 'ASC') OR (NOT prev AND sort._dir = 'DESC')) THEN\n RAISE NOTICE '< but null';\n orfilter := format('%s IS NOT NULL', sort._field);\n ELSIF sort._val IS NULL THEN\n RAISE NOTICE '> but null';\n ELSE\n orfilter := format($f$(\n (%s %s %s) OR (%s IS NULL)\n )$f$,\n sort._field,\n gtop,\n sort._val,\n sort._field\n );\n END IF;\n RAISE NOTICE 'ORFILTER: %', orfilter;\n\n IF orfilter IS NOT NULL THEN\n IF sort._row = 1 THEN\n orfilters := orfilters || orfilter;\n ELSE\n orfilters := orfilters || format('(%s AND %s)', array_to_string(andfilters, ' AND '), orfilter);\n END IF;\n END IF;\n IF sort._val IS NOT NULL THEN\n andfilters := andfilters || format('%s = %s', sort._field, sort._val);\n ELSE\n andfilters := andfilters || format('%s IS NULL', sort._field);\n END IF;\n END LOOP;\n\n output := array_to_string(orfilters, ' OR ');\n\n token_where := concat('(',coalesce(output,'true'),')');\n IF trim(token_where) = '' THEN\n token_where := NULL;\n END IF;\n RAISE NOTICE 'TOKEN_WHERE: %',token_where;\n RETURN token_where;\n END;\n$$ LANGUAGE PLPGSQL SET transform_null_equals TO TRUE\n;\n\nCREATE OR REPLACE FUNCTION search_hash(jsonb, jsonb) RETURNS text AS $$\n SELECT md5(concat(($1 - '{token,limit,context,includes,excludes}'::text[])::text,$2::text));\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\nDROP FUNCTION IF EXISTS search_tohash(jsonb);\n\nCREATE TABLE IF NOT EXISTS searches(\n hash text GENERATED ALWAYS AS (search_hash(search, metadata)) STORED PRIMARY KEY,\n search jsonb NOT NULL,\n _where text,\n orderby text,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n metadata jsonb DEFAULT '{}'::jsonb NOT NULL\n);\n\nCREATE TABLE IF NOT EXISTS search_wheres(\n id bigint generated always as identity primary key,\n _where text NOT NULL,\n lastused timestamptz DEFAULT now(),\n usecount bigint DEFAULT 0,\n statslastupdated timestamptz,\n estimated_count bigint,\n estimated_cost float,\n time_to_estimate float,\n total_count bigint,\n time_to_count float,\n partitions text[]\n);\n\nCREATE INDEX IF NOT EXISTS search_wheres_partitions ON search_wheres USING GIN (partitions);\nCREATE UNIQUE INDEX IF NOT EXISTS search_wheres_where ON search_wheres ((md5(_where)));\n\nCREATE OR REPLACE FUNCTION where_stats(\n inwhere text,\n updatestats boolean default false,\n conf jsonb default null\n) RETURNS search_wheres AS $$\nDECLARE\n t timestamptz;\n i interval;\n explain_json jsonb;\n partitions text[];\n sw search_wheres%ROWTYPE;\n inwhere_hash text := md5(inwhere);\n _context text := lower(context(conf));\n _stats_ttl interval := context_stats_ttl(conf);\n _estimated_cost_threshold float := context_estimated_cost(conf);\n _estimated_count_threshold int := context_estimated_count(conf);\n ro bool := pgstac.readonly(conf);\nBEGIN\n -- If updatestats is true then set ttl to 0\n IF updatestats THEN\n RAISE DEBUG 'Updatestats set to TRUE, setting TTL to 0';\n _stats_ttl := '0'::interval;\n END IF;\n\n -- If we don't need to calculate context, just return\n IF _context = 'off' THEN\n sw._where = inwhere;\n RETURN sw;\n END IF;\n\n -- Get any stats that we have.\n IF NOT ro THEN\n -- If there is a lock where another process is\n -- updating the stats, wait so that we don't end up calculating a bunch of times.\n SELECT * INTO sw FROM search_wheres WHERE md5(_where)=inwhere_hash FOR UPDATE;\n ELSE\n SELECT * INTO sw FROM search_wheres WHERE md5(_where)=inwhere_hash;\n END IF;\n\n -- If there is a cached row, figure out if we need to update\n IF\n sw IS NOT NULL\n AND sw.statslastupdated IS NOT NULL\n AND sw.total_count IS NOT NULL\n AND now() - sw.statslastupdated <= _stats_ttl\n THEN\n -- we have a cached row with data that is within our ttl\n RAISE DEBUG 'Stats present in table and lastupdated within ttl: %', sw;\n IF NOT ro THEN\n RAISE DEBUG 'Updating search_wheres only bumping lastused and usecount';\n UPDATE search_wheres SET\n lastused = now(),\n usecount = search_wheres.usecount + 1\n WHERE md5(_where) = inwhere_hash\n RETURNING * INTO sw;\n END IF;\n RAISE DEBUG 'Returning cached counts. %', sw;\n RETURN sw;\n END IF;\n\n -- Calculate estimated cost and rows\n -- Use explain to get estimated count/cost\n IF sw.estimated_count IS NULL OR sw.estimated_cost IS NULL THEN\n RAISE DEBUG 'Calculating estimated stats';\n t := clock_timestamp();\n EXECUTE format('EXPLAIN (format json) SELECT 1 FROM items WHERE %s', inwhere)\n INTO explain_json;\n RAISE DEBUG 'Time for just the explain: %', clock_timestamp() - t;\n i := clock_timestamp() - t;\n\n sw.estimated_count := explain_json->0->'Plan'->'Plan Rows';\n sw.estimated_cost := explain_json->0->'Plan'->'Total Cost';\n sw.time_to_estimate := extract(epoch from i);\n END IF;\n\n RAISE DEBUG 'ESTIMATED_COUNT: %, THRESHOLD %', sw.estimated_count, _estimated_count_threshold;\n RAISE DEBUG 'ESTIMATED_COST: %, THRESHOLD %', sw.estimated_cost, _estimated_cost_threshold;\n\n -- If context is set to auto and the costs are within the threshold return the estimated costs\n IF\n _context = 'auto'\n AND sw.estimated_count >= _estimated_count_threshold\n AND sw.estimated_cost >= _estimated_cost_threshold\n THEN\n IF NOT ro THEN\n INSERT INTO search_wheres (\n _where,\n lastused,\n usecount,\n statslastupdated,\n estimated_count,\n estimated_cost,\n time_to_estimate,\n total_count,\n time_to_count\n ) VALUES (\n inwhere,\n now(),\n 1,\n now(),\n sw.estimated_count,\n sw.estimated_cost,\n sw.time_to_estimate,\n null,\n null\n ) ON CONFLICT ((md5(_where)))\n DO UPDATE SET\n lastused = EXCLUDED.lastused,\n usecount = search_wheres.usecount + 1,\n statslastupdated = EXCLUDED.statslastupdated,\n estimated_count = EXCLUDED.estimated_count,\n estimated_cost = EXCLUDED.estimated_cost,\n time_to_estimate = EXCLUDED.time_to_estimate,\n total_count = EXCLUDED.total_count,\n time_to_count = EXCLUDED.time_to_count\n RETURNING * INTO sw;\n END IF;\n RAISE DEBUG 'Estimates are within thresholds, returning estimates. %', sw;\n RETURN sw;\n END IF;\n\n -- Calculate Actual Count\n t := clock_timestamp();\n RAISE NOTICE 'Calculating actual count...';\n EXECUTE format(\n 'SELECT count(*) FROM items WHERE %s',\n inwhere\n ) INTO sw.total_count;\n i := clock_timestamp() - t;\n RAISE NOTICE 'Actual Count: % -- %', sw.total_count, i;\n sw.time_to_count := extract(epoch FROM i);\n\n IF NOT ro THEN\n INSERT INTO search_wheres (\n _where,\n lastused,\n usecount,\n statslastupdated,\n estimated_count,\n estimated_cost,\n time_to_estimate,\n total_count,\n time_to_count\n ) VALUES (\n inwhere,\n now(),\n 1,\n now(),\n sw.estimated_count,\n sw.estimated_cost,\n sw.time_to_estimate,\n sw.total_count,\n sw.time_to_count\n ) ON CONFLICT ((md5(_where)))\n DO UPDATE SET\n lastused = EXCLUDED.lastused,\n usecount = search_wheres.usecount + 1,\n statslastupdated = EXCLUDED.statslastupdated,\n estimated_count = EXCLUDED.estimated_count,\n estimated_cost = EXCLUDED.estimated_cost,\n time_to_estimate = EXCLUDED.time_to_estimate,\n total_count = EXCLUDED.total_count,\n time_to_count = EXCLUDED.time_to_count\n RETURNING * INTO sw;\n END IF;\n RAISE DEBUG 'Returning with actual count. %', sw;\n RETURN sw;\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION search_query(\n _search jsonb = '{}'::jsonb,\n updatestats boolean = false,\n _metadata jsonb = '{}'::jsonb\n) RETURNS searches AS $$\nDECLARE\n search searches%ROWTYPE;\n cached_search searches%ROWTYPE;\n pexplain jsonb;\n t timestamptz;\n i interval;\n doupdate boolean := FALSE;\n insertfound boolean := FALSE;\n ro boolean := pgstac.readonly();\n found_search text;\nBEGIN\n RAISE NOTICE 'SEARCH: %', _search;\n -- Calculate hash, where clause, and order by statement\n search.search := _search;\n search.metadata := _metadata;\n search.hash := search_hash(_search, _metadata);\n search._where := stac_search_to_where(_search);\n search.orderby := sort_sqlorderby(_search);\n search.lastused := now();\n search.usecount := 1;\n\n -- If we are in read only mode, directly return search\n IF ro THEN\n RETURN search;\n END IF;\n\n RAISE NOTICE 'Updating Statistics for search: %s', search;\n -- Update statistics for times used and and when last used\n -- If the entry is locked, rather than waiting, skip updating the stats\n INSERT INTO searches (search, lastused, usecount, metadata)\n VALUES (search.search, now(), 1, search.metadata)\n ON CONFLICT DO NOTHING\n RETURNING * INTO cached_search\n ;\n\n IF NOT FOUND OR cached_search IS NULL THEN\n UPDATE searches SET\n lastused = now(),\n usecount = searches.usecount + 1\n WHERE hash = (\n SELECT hash FROM searches WHERE hash=search.hash FOR UPDATE SKIP LOCKED\n )\n RETURNING * INTO cached_search\n ;\n END IF;\n\n IF cached_search IS NOT NULL THEN\n cached_search._where = search._where;\n cached_search.orderby = search.orderby;\n RETURN cached_search;\n END IF;\n RETURN search;\n\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\nCREATE OR REPLACE FUNCTION search_fromhash(\n _hash text\n) RETURNS searches AS $$\n SELECT * FROM search_query((SELECT search FROM searches WHERE hash=_hash LIMIT 1));\n$$ LANGUAGE SQL STRICT;\n\nCREATE OR REPLACE FUNCTION search_rows(\n IN _where text DEFAULT 'TRUE',\n IN _orderby text DEFAULT 'datetime DESC, id DESC',\n IN partitions text[] DEFAULT NULL,\n IN _limit int DEFAULT 10\n) RETURNS SETOF items AS $$\nDECLARE\n base_query text;\n query text;\n sdate timestamptz;\n edate timestamptz;\n n int;\n records_left int := _limit;\n timer timestamptz := clock_timestamp();\n full_timer timestamptz := clock_timestamp();\nBEGIN\nIF _where IS NULL OR trim(_where) = '' THEN\n _where = ' TRUE ';\nEND IF;\nRAISE NOTICE 'Getting chunks for % %', _where, _orderby;\n\nbase_query := $q$\n SELECT * FROM items\n WHERE\n datetime >= %L AND datetime < %L\n AND (%s)\n ORDER BY %s\n LIMIT %L\n$q$;\n\nIF _orderby ILIKE 'datetime d%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 DESC LOOP\n RAISE NOTICE 'Running Query for % to %. %', sdate, edate, age_ms(full_timer);\n query := format(\n base_query,\n sdate,\n edate,\n _where,\n _orderby,\n records_left\n );\n RAISE DEBUG 'QUERY: %', query;\n timer := clock_timestamp();\n RETURN QUERY EXECUTE query;\n\n GET DIAGNOSTICS n = ROW_COUNT;\n records_left := records_left - n;\n RAISE NOTICE 'Returned %/% Rows From % to %. % to go. Time: %ms', n, _limit, sdate, edate, records_left, age_ms(timer);\n timer := clock_timestamp();\n IF records_left <= 0 THEN\n RAISE NOTICE 'SEARCH_ROWS TOOK %ms', age_ms(full_timer);\n RETURN;\n END IF;\n END LOOP;\nELSIF _orderby ILIKE 'datetime a%' THEN\n FOR sdate, edate IN SELECT * FROM chunker(_where) ORDER BY 1 ASC LOOP\n RAISE NOTICE 'Running Query for % to %. %', sdate, edate, age_ms(full_timer);\n query := format(\n base_query,\n sdate,\n edate,\n _where,\n _orderby,\n records_left\n );\n RAISE DEBUG 'QUERY: %', query;\n timer := clock_timestamp();\n RETURN QUERY EXECUTE query;\n\n GET DIAGNOSTICS n = ROW_COUNT;\n records_left := records_left - n;\n RAISE NOTICE 'Returned %/% Rows From % to %. % to go. Time: %ms', n, _limit, sdate, edate, records_left, age_ms(timer);\n timer := clock_timestamp();\n IF records_left <= 0 THEN\n RAISE NOTICE 'SEARCH_ROWS TOOK %ms', age_ms(full_timer);\n RETURN;\n END IF;\n END LOOP;\nELSE\n query := format($q$\n SELECT * FROM items\n WHERE %s\n ORDER BY %s\n LIMIT %L\n $q$, _where, _orderby, _limit\n );\n RAISE DEBUG 'QUERY: %', query;\n timer := clock_timestamp();\n RETURN QUERY EXECUTE query;\n RAISE NOTICE 'FULL QUERY TOOK %ms', age_ms(timer);\nEND IF;\nRAISE NOTICE 'SEARCH_ROWS TOOK %ms', age_ms(full_timer);\nRETURN;\nEND;\n$$ LANGUAGE PLPGSQL SET SEARCH_PATH TO pgstac,public;\n\n\nCREATE UNLOGGED TABLE format_item_cache(\n id text,\n collection text,\n fields text,\n hydrated bool,\n output jsonb,\n lastused timestamptz DEFAULT now(),\n usecount int DEFAULT 1,\n timetoformat float,\n PRIMARY KEY (collection, id, fields, hydrated)\n);\nCREATE INDEX ON format_item_cache (lastused);\n\nCREATE OR REPLACE FUNCTION format_item(_item items, _fields jsonb DEFAULT '{}', _hydrated bool DEFAULT TRUE) RETURNS jsonb AS $$\nDECLARE\n cache bool := get_setting_bool('format_cache');\n _output jsonb := null;\n t timestamptz := clock_timestamp();\nBEGIN\n IF cache THEN\n SELECT output INTO _output FROM format_item_cache\n WHERE id=_item.id AND collection=_item.collection AND fields=_fields::text AND hydrated=_hydrated;\n END IF;\n IF _output IS NULL THEN\n IF _hydrated THEN\n _output := content_hydrate(_item, _fields);\n ELSE\n _output := content_nonhydrated(_item, _fields);\n END IF;\n END IF;\n IF cache THEN\n INSERT INTO format_item_cache (id, collection, fields, hydrated, output, timetoformat)\n VALUES (_item.id, _item.collection, _fields::text, _hydrated, _output, age_ms(t))\n ON CONFLICT(collection, id, fields, hydrated) DO\n UPDATE\n SET lastused=now(), usecount = format_item_cache.usecount + 1\n ;\n END IF;\n RETURN _output;\n\nEND;\n$$ LANGUAGE PLPGSQL SECURITY DEFINER;\n\n\nCREATE OR REPLACE FUNCTION search(_search jsonb = '{}'::jsonb) RETURNS jsonb AS $$\nDECLARE\n searches searches%ROWTYPE;\n _where text;\n orderby text;\n search_where search_wheres%ROWTYPE;\n total_count bigint;\n token record;\n token_prev boolean;\n token_item items%ROWTYPE;\n token_where text;\n full_where text;\n init_ts timestamptz := clock_timestamp();\n timer timestamptz := clock_timestamp();\n hydrate bool := NOT (_search->'conf'->>'nohydrate' IS NOT NULL AND (_search->'conf'->>'nohydrate')::boolean = true);\n prev text;\n next text;\n context jsonb;\n collection jsonb;\n out_records jsonb;\n out_len int;\n _limit int := coalesce((_search->>'limit')::int, 10);\n _querylimit int;\n _fields jsonb := coalesce(_search->'fields', '{}'::jsonb);\n has_prev boolean := FALSE;\n has_next boolean := FALSE;\n links jsonb := '[]'::jsonb;\n base_url text:= concat(rtrim(base_url(_search->'conf'),'/'));\nBEGIN\n searches := search_query(_search);\n _where := searches._where;\n orderby := searches.orderby;\n search_where := where_stats(_where);\n total_count := coalesce(search_where.total_count, search_where.estimated_count);\n RAISE NOTICE 'SEARCH:TOKEN: %', _search->>'token';\n token := get_token_record(_search->>'token');\n RAISE NOTICE '***TOKEN: %', token;\n _querylimit := _limit + 1;\n IF token IS NOT NULL THEN\n token_prev := token.prev;\n token_item := token.item;\n token_where := get_token_filter(_search->'sortby', token_item, token_prev, FALSE);\n RAISE DEBUG 'TOKEN_WHERE: % (%ms from search start)', token_where, age_ms(timer);\n IF token_prev THEN -- if we are using a prev token, we know has_next is true\n RAISE DEBUG 'There is a previous token, so automatically setting has_next to true';\n has_next := TRUE;\n orderby := sort_sqlorderby(_search, TRUE);\n ELSE\n RAISE DEBUG 'There is a next token, so automatically setting has_prev to true';\n has_prev := TRUE;\n\n END IF;\n ELSE -- if there was no token, we know there is no prev\n RAISE DEBUG 'There is no token, so we know there is no prev. setting has_prev to false';\n has_prev := FALSE;\n END IF;\n\n full_where := concat_ws(' AND ', _where, token_where);\n RAISE NOTICE 'FULL WHERE CLAUSE: %', full_where;\n RAISE NOTICE 'Time to get counts and build query %', age_ms(timer);\n timer := clock_timestamp();\n\n IF hydrate THEN\n RAISE NOTICE 'Getting hydrated data.';\n ELSE\n RAISE NOTICE 'Getting non-hydrated data.';\n END IF;\n RAISE NOTICE 'CACHE SET TO %', get_setting_bool('format_cache');\n RAISE NOTICE 'Time to set hydration/formatting %', age_ms(timer);\n timer := clock_timestamp();\n SELECT jsonb_agg(format_item(i, _fields, hydrate)) INTO out_records\n FROM search_rows(\n full_where,\n orderby,\n search_where.partitions,\n _querylimit\n ) as i;\n\n RAISE NOTICE 'Time to fetch rows %', age_ms(timer);\n timer := clock_timestamp();\n\n\n IF token_prev THEN\n out_records := flip_jsonb_array(out_records);\n END IF;\n\n RAISE NOTICE 'Query returned % records.', jsonb_array_length(out_records);\n RAISE DEBUG 'TOKEN: % %', token_item.id, token_item.collection;\n RAISE DEBUG 'RECORD_1: % %', out_records->0->>'id', out_records->0->>'collection';\n RAISE DEBUG 'RECORD-1: % %', out_records->-1->>'id', out_records->-1->>'collection';\n\n -- REMOVE records that were from our token\n IF out_records->0->>'id' = token_item.id AND out_records->0->>'collection' = token_item.collection THEN\n out_records := out_records - 0;\n ELSIF out_records->-1->>'id' = token_item.id AND out_records->-1->>'collection' = token_item.collection THEN\n out_records := out_records - -1;\n END IF;\n\n out_len := jsonb_array_length(out_records);\n\n IF out_len = _limit + 1 THEN\n IF token_prev THEN\n has_prev := TRUE;\n out_records := out_records - 0;\n ELSE\n has_next := TRUE;\n out_records := out_records - -1;\n END IF;\n END IF;\n\n\n links := links || jsonb_build_object(\n 'rel', 'root',\n 'type', 'application/json',\n 'href', base_url\n ) || jsonb_build_object(\n 'rel', 'self',\n 'type', 'application/json',\n 'href', concat(base_url, '/search')\n );\n\n IF has_next THEN\n next := concat(out_records->-1->>'collection', ':', out_records->-1->>'id');\n RAISE NOTICE 'HAS NEXT | %', next;\n links := links || jsonb_build_object(\n 'rel', 'next',\n 'type', 'application/geo+json',\n 'method', 'GET',\n 'href', concat(base_url, '/search?token=next:', next)\n );\n END IF;\n\n IF has_prev THEN\n prev := concat(out_records->0->>'collection', ':', out_records->0->>'id');\n RAISE NOTICE 'HAS PREV | %', prev;\n links := links || jsonb_build_object(\n 'rel', 'prev',\n 'type', 'application/geo+json',\n 'method', 'GET',\n 'href', concat(base_url, '/search?token=prev:', prev)\n );\n END IF;\n\n RAISE NOTICE 'Time to get prev/next %', age_ms(timer);\n timer := clock_timestamp();\n\n\n collection := jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb),\n 'links', links\n );\n\n\n\n IF context(_search->'conf') != 'off' THEN\n collection := collection || jsonb_strip_nulls(jsonb_build_object(\n 'numberMatched', total_count,\n 'numberReturned', coalesce(jsonb_array_length(out_records), 0)\n ));\n ELSE\n collection := collection || jsonb_strip_nulls(jsonb_build_object(\n 'numberReturned', coalesce(jsonb_array_length(out_records), 0)\n ));\n END IF;\n\n IF get_setting_bool('timing', _search->'conf') THEN\n collection = collection || jsonb_build_object('timing', age_ms(init_ts));\n END IF;\n\n RAISE NOTICE 'Time to build final json %', age_ms(timer);\n timer := clock_timestamp();\n\n RAISE NOTICE 'Total Time: %', age_ms(current_timestamp);\n RAISE NOTICE 'RETURNING % records. NEXT: %. PREV: %', collection->>'numberReturned', collection->>'next', collection->>'prev';\n RETURN collection;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION search_cursor(_search jsonb = '{}'::jsonb) RETURNS refcursor AS $$\nDECLARE\n curs refcursor;\n searches searches%ROWTYPE;\n _where text;\n _orderby text;\n q text;\n\nBEGIN\n searches := search_query(_search);\n _where := searches._where;\n _orderby := searches.orderby;\n\n OPEN curs FOR\n WITH p AS (\n SELECT * FROM partition_queries(_where, _orderby) p\n )\n SELECT\n CASE WHEN EXISTS (SELECT 1 FROM p) THEN\n (SELECT format($q$\n SELECT * FROM (\n %s\n ) total\n $q$,\n string_agg(\n format($q$ SELECT * FROM ( %s ) AS sub $q$, p),\n '\n UNION ALL\n '\n )\n ))\n ELSE NULL\n END FROM p;\n RETURN curs;\nEND;\n$$ LANGUAGE PLPGSQL;\n" }, { "path": "src/pgstac/sql/004a_collectionsearch.sql", "content": "CREATE OR REPLACE VIEW collections_asitems AS\nSELECT\n id,\n geometry,\n 'collections' AS collection,\n datetime,\n end_datetime,\n jsonb_build_object(\n 'properties', content - '{links,assets,stac_version,stac_extensions}',\n 'links', content->'links',\n 'assets', content->'assets',\n 'stac_version', content->'stac_version',\n 'stac_extensions', content->'stac_extensions'\n ) AS content,\n content as collectionjson\nFROM collections;\n\n\nCREATE OR REPLACE FUNCTION collection_search_matched(\n IN _search jsonb DEFAULT '{}'::jsonb,\n OUT matched bigint\n) RETURNS bigint AS $$\nDECLARE\n _where text := stac_search_to_where(_search);\nBEGIN\n EXECUTE format(\n $query$\n SELECT\n count(*)\n FROM\n collections_asitems\n WHERE %s\n ;\n $query$,\n _where\n ) INTO matched;\n RETURN;\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION collection_search_rows(\n _search jsonb DEFAULT '{}'::jsonb\n) RETURNS SETOF jsonb AS $$\nDECLARE\n _where text := stac_search_to_where(_search);\n _limit int := coalesce((_search->>'limit')::int, 10);\n _fields jsonb := coalesce(_search->'fields', '{}'::jsonb);\n _orderby text;\n _offset int := COALESCE((_search->>'offset')::int, 0);\nBEGIN\n _orderby := sort_sqlorderby(\n jsonb_build_object(\n 'sortby',\n coalesce(\n _search->'sortby',\n '[{\"field\": \"id\", \"direction\": \"asc\"}]'::jsonb\n )\n )\n );\n RETURN QUERY EXECUTE format(\n $query$\n SELECT\n jsonb_fields(collectionjson, %L) as c\n FROM\n collections_asitems\n WHERE %s\n ORDER BY %s\n LIMIT %L\n OFFSET %L\n ;\n $query$,\n _fields,\n _where,\n _orderby,\n _limit,\n _offset\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nCREATE OR REPLACE FUNCTION collection_search(\n _search jsonb DEFAULT '{}'::jsonb\n) RETURNS jsonb AS $$\nDECLARE\n out_records jsonb;\n number_matched bigint := collection_search_matched(_search);\n number_returned bigint;\n _limit int := coalesce((_search->>'limit')::float::int, 10);\n _offset int := coalesce((_search->>'offset')::float::int, 0);\n links jsonb := '[]';\n ret jsonb;\n base_url text:= concat(rtrim(base_url(_search->'conf'),'/'), '/collections');\n prevoffset int;\n nextoffset int;\nBEGIN\n SELECT\n coalesce(jsonb_agg(c), '[]')\n INTO out_records\n FROM collection_search_rows(_search) c;\n\n number_returned := jsonb_array_length(out_records);\n RAISE DEBUG 'nm: %, nr: %, l:%, o:%', number_matched, number_returned, _limit, _offset;\n\n\n\n IF _limit <= number_matched AND number_matched > 0 THEN --need to have paging links\n nextoffset := least(_offset + _limit, number_matched - 1);\n prevoffset := greatest(_offset - _limit, 0);\n\n IF _offset > 0 THEN\n links := links || jsonb_build_object(\n 'rel', 'prev',\n 'type', 'application/json',\n 'method', 'GET' ,\n 'href', base_url,\n 'body', jsonb_build_object('offset', prevoffset),\n 'merge', TRUE\n );\n END IF;\n\n IF (_offset + _limit < number_matched) THEN\n links := links || jsonb_build_object(\n 'rel', 'next',\n 'type', 'application/json',\n 'method', 'GET' ,\n 'href', base_url,\n 'body', jsonb_build_object('offset', nextoffset),\n 'merge', TRUE\n );\n END IF;\n\n END IF;\n\n ret := jsonb_build_object(\n 'collections', out_records,\n 'numberMatched', number_matched,\n 'numberReturned', number_returned,\n 'links', links\n );\n RETURN ret;\n\nEND;\n$$ LANGUAGE PLPGSQL STABLE PARALLEL SAFE;\n" }, { "path": "src/pgstac/sql/005_tileutils.sql", "content": "SET SEARCH_PATH TO pgstac, public;\n\nCREATE OR REPLACE FUNCTION tileenvelope(zoom int, x int, y int) RETURNS geometry AS $$\nWITH t AS (\n SELECT\n 20037508.3427892 as merc_max,\n -20037508.3427892 as merc_min,\n (2 * 20037508.3427892) / (2 ^ zoom) as tile_size\n)\nSELECT st_makeenvelope(\n merc_min + (tile_size * x),\n merc_max - (tile_size * (y + 1)),\n merc_min + (tile_size * (x + 1)),\n merc_max - (tile_size * y),\n 3857\n) FROM t;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;DROP FUNCTION IF EXISTS mercgrid;\n\n\nCREATE OR REPLACE FUNCTION ftime() RETURNS interval as $$\nSELECT age(clock_timestamp(), transaction_timestamp());\n$$ LANGUAGE SQL;\n" }, { "path": "src/pgstac/sql/006_tilesearch.sql", "content": "SET SEARCH_PATH to pgstac, public;\n\nDROP FUNCTION IF EXISTS geometrysearch;\nCREATE OR REPLACE FUNCTION geometrysearch(\n IN geom geometry,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE, -- Return as soon as the passed in geometry is full covered\n IN skipcovered boolean DEFAULT TRUE -- Skip any items that would show up completely under the previous items\n) RETURNS jsonb AS $$\nDECLARE\n search searches%ROWTYPE;\n curs refcursor;\n _where text;\n query text;\n iter_record items%ROWTYPE;\n out_records jsonb := '{}'::jsonb[];\n exit_flag boolean := FALSE;\n counter int := 1;\n scancounter int := 1;\n remaining_limit int := _scanlimit;\n tilearea float;\n unionedgeom geometry;\n clippedgeom geometry;\n unionedgeom_area float := 0;\n prev_area float := 0;\n excludes text[];\n includes text[];\n\nBEGIN\n DROP TABLE IF EXISTS pgstac_results;\n CREATE TEMP TABLE pgstac_results (content jsonb) ON COMMIT DROP;\n\n -- If the passed in geometry is not an area set exitwhenfull and skipcovered to false\n IF ST_GeometryType(geom) !~* 'polygon' THEN\n RAISE NOTICE 'GEOMETRY IS NOT AN AREA';\n skipcovered = FALSE;\n exitwhenfull = FALSE;\n END IF;\n\n -- If skipcovered is true then you will always want to exit when the passed in geometry is full\n IF skipcovered THEN\n exitwhenfull := TRUE;\n END IF;\n\n search := search_fromhash(queryhash);\n\n IF search IS NULL THEN\n RAISE EXCEPTION 'Search with Query Hash % Not Found', queryhash;\n END IF;\n\n tilearea := st_area(geom);\n _where := format('%s AND st_intersects(geometry, %L::geometry)', search._where, geom);\n\n\n FOR query IN SELECT * FROM partition_queries(_where, search.orderby) LOOP\n query := format('%s LIMIT %L', query, remaining_limit);\n RAISE NOTICE '%', query;\n OPEN curs FOR EXECUTE query;\n LOOP\n FETCH curs INTO iter_record;\n EXIT WHEN NOT FOUND;\n IF exitwhenfull OR skipcovered THEN -- If we are not using exitwhenfull or skipcovered, we do not need to do expensive geometry operations\n clippedgeom := st_intersection(geom, iter_record.geometry);\n\n IF unionedgeom IS NULL THEN\n unionedgeom := clippedgeom;\n ELSE\n unionedgeom := st_union(unionedgeom, clippedgeom);\n END IF;\n\n unionedgeom_area := st_area(unionedgeom);\n\n IF skipcovered AND prev_area = unionedgeom_area THEN\n scancounter := scancounter + 1;\n CONTINUE;\n END IF;\n\n prev_area := unionedgeom_area;\n\n RAISE NOTICE '% % % %', unionedgeom_area/tilearea, counter, scancounter, ftime();\n END IF;\n RAISE NOTICE '% %', iter_record, content_hydrate(iter_record, fields);\n INSERT INTO pgstac_results (content) VALUES (content_hydrate(iter_record, fields));\n\n IF counter >= _limit\n OR scancounter > _scanlimit\n OR ftime() > _timelimit\n OR (exitwhenfull AND unionedgeom_area >= tilearea)\n THEN\n exit_flag := TRUE;\n EXIT;\n END IF;\n counter := counter + 1;\n scancounter := scancounter + 1;\n\n END LOOP;\n CLOSE curs;\n EXIT WHEN exit_flag;\n remaining_limit := _scanlimit - scancounter;\n END LOOP;\n\n SELECT jsonb_agg(content) INTO out_records FROM pgstac_results WHERE content IS NOT NULL;\n\n RETURN jsonb_build_object(\n 'type', 'FeatureCollection',\n 'features', coalesce(out_records, '[]'::jsonb)\n );\nEND;\n$$ LANGUAGE PLPGSQL;\n\nDROP FUNCTION IF EXISTS geojsonsearch;\nCREATE OR REPLACE FUNCTION geojsonsearch(\n IN geojson jsonb,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_geomfromgeojson(geojson),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\n\nDROP FUNCTION IF EXISTS xyzsearch;\nCREATE OR REPLACE FUNCTION xyzsearch(\n IN _x int,\n IN _y int,\n IN _z int,\n IN queryhash text,\n IN fields jsonb DEFAULT NULL,\n IN _scanlimit int DEFAULT 10000,\n IN _limit int DEFAULT 100,\n IN _timelimit interval DEFAULT '5 seconds'::interval,\n IN exitwhenfull boolean DEFAULT TRUE,\n IN skipcovered boolean DEFAULT TRUE\n) RETURNS jsonb AS $$\n SELECT * FROM geometrysearch(\n st_transform(tileenvelope(_z, _x, _y), 4326),\n queryhash,\n fields,\n _scanlimit,\n _limit,\n _timelimit,\n exitwhenfull,\n skipcovered\n );\n$$ LANGUAGE SQL;\n" }, { "path": "src/pgstac/sql/997_maintenance.sql", "content": "\nCREATE OR REPLACE PROCEDURE analyze_items() AS $$\nDECLARE\n q text;\n timeout_ts timestamptz;\nBEGIN\n timeout_ts := statement_timestamp() + queue_timeout();\n WHILE clock_timestamp() < timeout_ts LOOP\n SELECT format('ANALYZE (VERBOSE, SKIP_LOCKED) %I;', relname) INTO q\n FROM pg_stat_user_tables\n WHERE relname like '_item%' AND (n_mod_since_analyze>0 OR last_analyze IS NULL) LIMIT 1;\n IF NOT FOUND THEN\n EXIT;\n END IF;\n RAISE NOTICE '%', q;\n EXECUTE q;\n COMMIT;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE PROCEDURE validate_constraints() AS $$\nDECLARE\n q text;\nBEGIN\n FOR q IN\n SELECT\n FORMAT(\n 'ALTER TABLE %I.%I VALIDATE CONSTRAINT %I;',\n nsp.nspname,\n cls.relname,\n con.conname\n )\n\n FROM pg_constraint AS con\n JOIN pg_class AS cls\n ON con.conrelid = cls.oid\n JOIN pg_namespace AS nsp\n ON cls.relnamespace = nsp.oid\n WHERE convalidated = FALSE AND contype in ('c','f')\n AND nsp.nspname = 'pgstac'\n LOOP\n RAISE NOTICE '%', q;\n PERFORM run_or_queue(q);\n COMMIT;\n END LOOP;\nEND;\n$$ LANGUAGE PLPGSQL;\n\n\nCREATE OR REPLACE FUNCTION collection_extent(_collection text, runupdate boolean default false) RETURNS jsonb AS $$\nDECLARE\n geom_extent geometry;\n mind timestamptz;\n maxd timestamptz;\n extent jsonb;\nBEGIN\n IF runupdate THEN\n PERFORM update_partition_stats_q(partition)\n FROM partitions_view WHERE collection=_collection;\n END IF;\n SELECT\n min(lower(dtrange)),\n max(upper(edtrange)),\n st_extent(spatial)\n INTO\n mind,\n maxd,\n geom_extent\n FROM partitions_view\n WHERE collection=_collection;\n\n IF geom_extent IS NOT NULL AND mind IS NOT NULL AND maxd IS NOT NULL THEN\n extent := jsonb_build_object(\n 'spatial', jsonb_build_object(\n 'bbox', to_jsonb(array[array[st_xmin(geom_extent), st_ymin(geom_extent), st_xmax(geom_extent), st_ymax(geom_extent)]])\n ),\n 'temporal', jsonb_build_object(\n 'interval', to_jsonb(array[array[mind, maxd]])\n )\n );\n RETURN extent;\n END IF;\n RETURN NULL;\nEND;\n$$ LANGUAGE PLPGSQL;\n" }, { "path": "src/pgstac/sql/998_idempotent_post.sql", "content": "DO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('id', '{\"title\": \"Item ID\",\"description\": \"Item identifier\",\"$ref\": \"https://schemas.stacspec.org/v1.0.0/item-spec/json-schema/item.json#/definitions/core/allOf/2/properties/id\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('geometry', '{\"title\": \"Item Geometry\",\"description\": \"Item Geometry\",\"$ref\": \"https://geojson.org/schema/Feature.json\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDO $$\n BEGIN\n INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES\n ('datetime','{\"description\": \"Datetime\",\"type\": \"string\",\"title\": \"Acquired\",\"format\": \"date-time\",\"pattern\": \"(\\\\+00:00|Z)$\"}', null, null);\n EXCEPTION WHEN unique_violation THEN\n RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;\n END\n$$;\n\nDELETE FROM queryables a USING queryables b\n WHERE a.name = b.name AND a.collection_ids IS NOT DISTINCT FROM b.collection_ids AND a.id > b.id;\n\n\nINSERT INTO pgstac_settings (name, value) VALUES\n ('context', 'off'),\n ('context_estimated_count', '100000'),\n ('context_estimated_cost', '100000'),\n ('context_stats_ttl', '1 day'),\n ('default_filter_lang', 'cql2-json'),\n ('additional_properties', 'true'),\n ('use_queue', 'false'),\n ('queue_timeout', '10 minutes'),\n ('update_collection_extent', 'false'),\n ('format_cache', 'false'),\n ('readonly', 'false')\nON CONFLICT DO NOTHING\n;\n\n\nINSERT INTO cql2_ops (op, template, types) VALUES\n ('eq', '%s = %s', NULL),\n ('neq', '%s != %s', NULL),\n ('ne', '%s != %s', NULL),\n ('!=', '%s != %s', NULL),\n ('<>', '%s != %s', NULL),\n ('lt', '%s < %s', NULL),\n ('lte', '%s <= %s', NULL),\n ('gt', '%s > %s', NULL),\n ('gte', '%s >= %s', NULL),\n ('le', '%s <= %s', NULL),\n ('ge', '%s >= %s', NULL),\n ('=', '%s = %s', NULL),\n ('<', '%s < %s', NULL),\n ('<=', '%s <= %s', NULL),\n ('>', '%s > %s', NULL),\n ('>=', '%s >= %s', NULL),\n ('like', '%s LIKE %s', NULL),\n ('ilike', '%s ILIKE %s', NULL),\n ('+', '%s + %s', NULL),\n ('-', '%s - %s', NULL),\n ('*', '%s * %s', NULL),\n ('/', '%s / %s', NULL),\n ('not', 'NOT (%s)', NULL),\n ('between', '%s BETWEEN %s AND %s', NULL),\n ('isnull', '%s IS NULL', NULL),\n ('upper', 'upper(%s)', NULL),\n ('lower', 'lower(%s)', NULL),\n ('casei', 'upper(%s)', NULL),\n ('accenti', 'unaccent(%s)', NULL)\nON CONFLICT (op) DO UPDATE\n SET\n template = EXCLUDED.template\n;\n\n\nALTER FUNCTION to_text COST 5000;\nALTER FUNCTION to_float COST 5000;\nALTER FUNCTION to_int COST 5000;\nALTER FUNCTION to_tstz COST 5000;\nALTER FUNCTION to_text_array COST 5000;\n\nALTER FUNCTION update_partition_stats SECURITY DEFINER;\nALTER FUNCTION partition_after_triggerfunc SECURITY DEFINER;\nALTER FUNCTION drop_table_constraints SECURITY DEFINER;\nALTER FUNCTION create_table_constraints SECURITY DEFINER;\nALTER FUNCTION check_partition SECURITY DEFINER;\nALTER FUNCTION repartition SECURITY DEFINER;\nALTER FUNCTION where_stats SECURITY DEFINER;\nALTER FUNCTION search_query SECURITY DEFINER;\nALTER FUNCTION format_item SECURITY DEFINER;\nALTER FUNCTION maintain_index SECURITY DEFINER;\n\nGRANT USAGE ON SCHEMA pgstac to pgstac_read;\nGRANT ALL ON SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON SCHEMA pgstac to pgstac_admin;\n\n-- pgstac_read role limited to using function apis\nGRANT EXECUTE ON FUNCTION search TO pgstac_read;\nGRANT EXECUTE ON FUNCTION search_query TO pgstac_read;\nGRANT EXECUTE ON FUNCTION item_by_id TO pgstac_read;\nGRANT EXECUTE ON FUNCTION get_item TO pgstac_read;\nGRANT SELECT ON ALL TABLES IN SCHEMA pgstac TO pgstac_read;\n\n\nGRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA pgstac to pgstac_ingest;\nGRANT ALL ON ALL TABLES IN SCHEMA pgstac to pgstac_ingest;\nGRANT USAGE ON ALL SEQUENCES IN SCHEMA pgstac to pgstac_ingest;\n\nREVOKE ALL PRIVILEGES ON PROCEDURE run_queued_queries FROM public;\nGRANT ALL ON PROCEDURE run_queued_queries TO pgstac_admin;\n\nREVOKE ALL PRIVILEGES ON FUNCTION run_queued_queries_intransaction FROM public;\nGRANT ALL ON FUNCTION run_queued_queries_intransaction TO pgstac_admin;\n\nRESET ROLE;\n\nSET ROLE pgstac_ingest;\nSELECT update_partition_stats_q(partition) FROM partitions_view;\n" }, { "path": "src/pgstac/sql/999_version.sql", "content": "SELECT set_version('unreleased');\n" }, { "path": "src/pgstac/tests/basic/collection_searches.sql", "content": "\nSET pgstac.context TO 'on';\nSET pgstac.\"default_filter_lang\" TO 'cql2-json';\nWITH t AS (\n SELECT\n row_number() over () as id,\n x,\n y\n FROM\n generate_series(-180, 170, 10) as x,\n generate_series(-90, 80, 10) as y\n), t1 AS (\n SELECT\n concat('testcollection_', id) as id,\n x as minx,\n y as miny,\n x+10 as maxx,\n y+10 as maxy,\n '2000-01-01'::timestamptz + (concat(id, ' weeks'))::interval as sdt,\n '2000-01-01'::timestamptz + (concat(id, ' weeks'))::interval + ('2 months')::interval as edt\n FROM t\n)\nSELECT\n create_collection(format($q$\n {\n \"id\": \"%s\",\n \"type\": \"Collection\",\n \"title\": \"My Test Collection.\",\n \"description\": \"Description of my test collection.\",\n \"extent\": {\n \"spatial\": {\"bbox\": [[%s, %s, %s, %s]]},\n \"temporal\": {\"interval\": [[%I, %I]]}\n },\n \"stac_extensions\":[]\n }\n $q$,\n id, minx, miny, maxx, maxy, sdt, edt\n )::jsonb)\nFROM t1;\n\nselect collection_search('{\"ids\":[\"testcollection_1\",\"testcollection_2\"],\"limit\":10, \"sortby\":[{\"field\":\"id\",\"direction\":\"desc\"}]}');\n\nselect collection_search('{\"ids\":[\"testcollection_1\",\"testcollection_2\"],\"limit\":10, \"sortby\":[{\"field\":\"id\",\"direction\":\"asc\"}]}');\n\nselect collection_search('{\"ids\":[\"testcollection_1\",\"testcollection_2\",\"testcollection_3\"],\"limit\":1, \"sortby\":[{\"field\":\"id\",\"direction\":\"desc\"}]}');\n\nselect collection_search('{\"ids\":[\"testcollection_1\",\"testcollection_2\"],\"limit\":1, \"offset\":10, \"sortby\":[{\"field\":\"datetime\",\"direction\":\"desc\"}]}');\n\nselect collection_search('{\"filter\":{\"op\":\"eq\", \"args\":[{\"property\":\"title\"},\"My Test Collection.\"]},\"limit\":10, \"sortby\":[{\"field\":\"datetime\",\"direction\":\"desc\"}]}');\n\nselect collection_search('{\"datetime\":[\"2012-01-01\",\"2012-01-02\"], \"filter\":{\"op\":\"eq\", \"args\":[{\"property\":\"title\"},\"My Test Collection.\"]},\"limit\":10, \"sortby\":[{\"field\":\"datetime\",\"direction\":\"desc\"}]}');\n\nselect collection_search('{\"ids\":[\"testcollection_1\",\"testcollection_2\"], \"fields\": {\"include\": [\"title\"]}}');\n" }, { "path": "src/pgstac/tests/basic/collection_searches.sql.out", "content": "SET pgstac.context TO 'on';\nSET\nSET pgstac.\"default_filter_lang\" TO 'cql2-json';\nSET\nWITH t AS (\n SELECT\n row_number() over () as id,\n x,\n y\n FROM\n generate_series(-180, 170, 10) as x,\n generate_series(-90, 80, 10) as y\n), t1 AS (\n SELECT\n concat('testcollection_', id) as id,\n x as minx,\n y as miny,\n x+10 as maxx,\n y+10 as maxy,\n '2000-01-01'::timestamptz + (concat(id, ' weeks'))::interval as sdt,\n '2000-01-01'::timestamptz + (concat(id, ' weeks'))::interval + ('2 months')::interval as edt\n FROM t\n)\nSELECT\n create_collection(format($q$\n {\n \"id\": \"%s\",\n \"type\": \"Collection\",\n \"title\": \"My Test Collection.\",\n \"description\": \"Description of my test collection.\",\n \"extent\": {\n \"spatial\": {\"bbox\": [[%s, %s, %s, %s]]},\n \"temporal\": {\"interval\": [[%I, %I]]}\n },\n \"stac_extensions\":[]\n }\n $q$,\n id, minx, miny, maxx, maxy, sdt, edt\n )::jsonb)\nFROM t1;\n\nselect collection_search('{\"ids\":[\"testcollection_1\",\"testcollection_2\"],\"limit\":10, \"sortby\":[{\"field\":\"id\",\"direction\":\"desc\"}]}');\n {\"links\": [], \"collections\": [{\"id\": \"testcollection_2\", \"type\": \"Collection\", \"title\": \"My Test Collection.\", \"extent\": {\"spatial\": {\"bbox\": [[-170, -90, -160, -80]]}, \"temporal\": {\"interval\": [[\"2000-01-15 00:00:00+00\", \"2000-03-15 00:00:00+00\"]]}}, \"description\": \"Description of my test collection.\", \"stac_extensions\": []}, {\"id\": \"testcollection_1\", \"type\": \"Collection\", \"title\": \"My Test Collection.\", \"extent\": {\"spatial\": {\"bbox\": [[-180, -90, -170, -80]]}, \"temporal\": {\"interval\": [[\"2000-01-08 00:00:00+00\", \"2000-03-08 00:00:00+00\"]]}}, \"description\": \"Description of my test collection.\", \"stac_extensions\": []}], \"numberMatched\": 2, \"numberReturned\": 2}\n\nselect collection_search('{\"ids\":[\"testcollection_1\",\"testcollection_2\"],\"limit\":10, \"sortby\":[{\"field\":\"id\",\"direction\":\"asc\"}]}');\n {\"links\": [], \"collections\": [{\"id\": \"testcollection_1\", \"type\": \"Collection\", \"title\": \"My Test Collection.\", \"extent\": {\"spatial\": {\"bbox\": [[-180, -90, -170, -80]]}, \"temporal\": {\"interval\": [[\"2000-01-08 00:00:00+00\", \"2000-03-08 00:00:00+00\"]]}}, \"description\": \"Description of my test collection.\", \"stac_extensions\": []}, {\"id\": \"testcollection_2\", \"type\": \"Collection\", \"title\": \"My Test Collection.\", \"extent\": {\"spatial\": {\"bbox\": [[-170, -90, -160, -80]]}, \"temporal\": {\"interval\": [[\"2000-01-15 00:00:00+00\", \"2000-03-15 00:00:00+00\"]]}}, \"description\": \"Description of my test collection.\", \"stac_extensions\": []}], \"numberMatched\": 2, \"numberReturned\": 2}\n\nselect collection_search('{\"ids\":[\"testcollection_1\",\"testcollection_2\",\"testcollection_3\"],\"limit\":1, \"sortby\":[{\"field\":\"id\",\"direction\":\"desc\"}]}');\n {\"links\": [{\"rel\": \"next\", \"body\": {\"offset\": 1}, \"href\": \"./collections\", \"type\": \"application/json\", \"merge\": true, \"method\": \"GET\"}], \"collections\": [{\"id\": \"testcollection_3\", \"type\": \"Collection\", \"title\": \"My Test Collection.\", \"extent\": {\"spatial\": {\"bbox\": [[-160, -90, -150, -80]]}, \"temporal\": {\"interval\": [[\"2000-01-22 00:00:00+00\", \"2000-03-22 00:00:00+00\"]]}}, \"description\": \"Description of my test collection.\", \"stac_extensions\": []}], \"numberMatched\": 3, \"numberReturned\": 1}\n\nselect collection_search('{\"ids\":[\"testcollection_1\",\"testcollection_2\"],\"limit\":1, \"offset\":10, \"sortby\":[{\"field\":\"datetime\",\"direction\":\"desc\"}]}');\n {\"links\": [{\"rel\": \"prev\", \"body\": {\"offset\": 9}, \"href\": \"./collections\", \"type\": \"application/json\", \"merge\": true, \"method\": \"GET\"}], \"collections\": [], \"numberMatched\": 2, \"numberReturned\": 0}\n\nselect collection_search('{\"filter\":{\"op\":\"eq\", \"args\":[{\"property\":\"title\"},\"My Test Collection.\"]},\"limit\":10, \"sortby\":[{\"field\":\"datetime\",\"direction\":\"desc\"}]}');\n {\"links\": [{\"rel\": \"next\", \"body\": {\"offset\": 10}, \"href\": \"./collections\", \"type\": \"application/json\", \"merge\": true, \"method\": \"GET\"}], \"collections\": [{\"id\": \"testcollection_648\", \"type\": \"Collection\", \"title\": \"My Test Collection.\", \"extent\": {\"spatial\": {\"bbox\": [[170, 80, 180, 90]]}, \"temporal\": {\"interval\": [[\"2012-06-02 00:00:00+00\", \"2012-08-02 00:00:00+00\"]]}}, \"description\": \"Description of my test collection.\", \"stac_extensions\": []}, {\"id\": \"testcollection_647\", \"type\": \"Collection\", \"title\": \"My Test Collection.\", \"extent\": {\"spatial\": {\"bbox\": [[160, 80, 170, 90]]}, \"temporal\": {\"interval\": [[\"2012-05-26 00:00:00+00\", \"2012-07-26 00:00:00+00\"]]}}, \"description\": \"Description of my test collection.\", \"stac_extensions\": []}, {\"id\": \"testcollection_646\", \"type\": \"Collection\", \"title\": \"My Test Collection.\", \"extent\": {\"spatial\": {\"bbox\": [[150, 80, 160, 90]]}, \"temporal\": {\"interval\": [[\"2012-05-19 00:00:00+00\", \"2012-07-19 00:00:00+00\"]]}}, \"description\": \"Description of my test collection.\", \"stac_extensions\": []}, {\"id\": \"testcollection_645\", \"type\": \"Collection\", \"title\": \"My Test Collection.\", \"extent\": {\"spatial\": {\"bbox\": [[140, 80, 150, 90]]}, \"temporal\": {\"interval\": [[\"2012-05-12 00:00:00+00\", \"2012-07-12 00:00:00+00\"]]}}, \"description\": \"Description of my test collection.\", \"stac_extensions\": []}, {\"id\": \"testcollection_644\", \"type\": \"Collection\", \"title\": \"My Test Collection.\", \"extent\": {\"spatial\": {\"bbox\": [[130, 80, 140, 90]]}, \"temporal\": {\"interval\": [[\"2012-05-05 00:00:00+00\", \"2012-07-05 00:00:00+00\"]]}}, \"description\": \"Description of my test collection.\", \"stac_extensions\": []}, {\"id\": \"testcollection_643\", \"type\": \"Collection\", \"title\": \"My Test Collection.\", \"extent\": {\"spatial\": {\"bbox\": [[120, 80, 130, 90]]}, \"temporal\": {\"interval\": [[\"2012-04-28 00:00:00+00\", \"2012-06-28 00:00:00+00\"]]}}, \"description\": \"Description of my test collection.\", \"stac_extensions\": []}, {\"id\": \"testcollection_642\", \"type\": \"Collection\", \"title\": \"My Test Collection.\", \"extent\": {\"spatial\": {\"bbox\": [[110, 80, 120, 90]]}, \"temporal\": {\"interval\": [[\"2012-04-21 00:00:00+00\", \"2012-06-21 00:00:00+00\"]]}}, \"description\": \"Description of my test collection.\", \"stac_extensions\": []}, {\"id\": \"testcollection_641\", \"type\": \"Collection\", \"title\": \"My Test Collection.\", \"extent\": {\"spatial\": {\"bbox\": [[100, 80, 110, 90]]}, \"temporal\": {\"interval\": [[\"2012-04-14 00:00:00+00\", \"2012-06-14 00:00:00+00\"]]}}, \"description\": \"Description of my test collection.\", \"stac_extensions\": []}, {\"id\": \"testcollection_640\", \"type\": \"Collection\", \"title\": \"My Test Collection.\", \"extent\": {\"spatial\": {\"bbox\": [[90, 80, 100, 90]]}, \"temporal\": {\"interval\": [[\"2012-04-07 00:00:00+00\", \"2012-06-07 00:00:00+00\"]]}}, \"description\": \"Description of my test collection.\", \"stac_extensions\": []}, {\"id\": \"testcollection_639\", \"type\": \"Collection\", \"title\": \"My Test Collection.\", \"extent\": {\"spatial\": {\"bbox\": [[80, 80, 90, 90]]}, \"temporal\": {\"interval\": [[\"2012-03-31 00:00:00+00\", \"2012-05-31 00:00:00+00\"]]}}, \"description\": \"Description of my test collection.\", \"stac_extensions\": []}], \"numberMatched\": 648, \"numberReturned\": 10}\n\nselect collection_search('{\"datetime\":[\"2012-01-01\",\"2012-01-02\"], \"filter\":{\"op\":\"eq\", \"args\":[{\"property\":\"title\"},\"My Test Collection.\"]},\"limit\":10, \"sortby\":[{\"field\":\"datetime\",\"direction\":\"desc\"}]}');\n {\"links\": [], \"collections\": [{\"id\": \"testcollection_626\", \"type\": \"Collection\", \"title\": \"My Test Collection.\", \"extent\": {\"spatial\": {\"bbox\": [[-50, 80, -40, 90]]}, \"temporal\": {\"interval\": [[\"2011-12-31 00:00:00+00\", \"2012-02-29 00:00:00+00\"]]}}, \"description\": \"Description of my test collection.\", \"stac_extensions\": []}, {\"id\": \"testcollection_625\", \"type\": \"Collection\", \"title\": \"My Test Collection.\", \"extent\": {\"spatial\": {\"bbox\": [[-60, 80, -50, 90]]}, \"temporal\": {\"interval\": [[\"2011-12-24 00:00:00+00\", \"2012-02-24 00:00:00+00\"]]}}, \"description\": \"Description of my test collection.\", \"stac_extensions\": []}, {\"id\": \"testcollection_624\", \"type\": \"Collection\", \"title\": \"My Test Collection.\", \"extent\": {\"spatial\": {\"bbox\": [[-70, 80, -60, 90]]}, \"temporal\": {\"interval\": [[\"2011-12-17 00:00:00+00\", \"2012-02-17 00:00:00+00\"]]}}, \"description\": \"Description of my test collection.\", \"stac_extensions\": []}, {\"id\": \"testcollection_623\", \"type\": \"Collection\", \"title\": \"My Test Collection.\", \"extent\": {\"spatial\": {\"bbox\": [[-80, 80, -70, 90]]}, \"temporal\": {\"interval\": [[\"2011-12-10 00:00:00+00\", \"2012-02-10 00:00:00+00\"]]}}, \"description\": \"Description of my test collection.\", \"stac_extensions\": []}, {\"id\": \"testcollection_622\", \"type\": \"Collection\", \"title\": \"My Test Collection.\", \"extent\": {\"spatial\": {\"bbox\": [[-90, 80, -80, 90]]}, \"temporal\": {\"interval\": [[\"2011-12-03 00:00:00+00\", \"2012-02-03 00:00:00+00\"]]}}, \"description\": \"Description of my test collection.\", \"stac_extensions\": []}, {\"id\": \"testcollection_621\", \"type\": \"Collection\", \"title\": \"My Test Collection.\", \"extent\": {\"spatial\": {\"bbox\": [[-100, 80, -90, 90]]}, \"temporal\": {\"interval\": [[\"2011-11-26 00:00:00+00\", \"2012-01-26 00:00:00+00\"]]}}, \"description\": \"Description of my test collection.\", \"stac_extensions\": []}, {\"id\": \"testcollection_620\", \"type\": \"Collection\", \"title\": \"My Test Collection.\", \"extent\": {\"spatial\": {\"bbox\": [[-110, 80, -100, 90]]}, \"temporal\": {\"interval\": [[\"2011-11-19 00:00:00+00\", \"2012-01-19 00:00:00+00\"]]}}, \"description\": \"Description of my test collection.\", \"stac_extensions\": []}, {\"id\": \"testcollection_619\", \"type\": \"Collection\", \"title\": \"My Test Collection.\", \"extent\": {\"spatial\": {\"bbox\": [[-120, 80, -110, 90]]}, \"temporal\": {\"interval\": [[\"2011-11-12 00:00:00+00\", \"2012-01-12 00:00:00+00\"]]}}, \"description\": \"Description of my test collection.\", \"stac_extensions\": []}, {\"id\": \"testcollection_618\", \"type\": \"Collection\", \"title\": \"My Test Collection.\", \"extent\": {\"spatial\": {\"bbox\": [[-130, 80, -120, 90]]}, \"temporal\": {\"interval\": [[\"2011-11-05 00:00:00+00\", \"2012-01-05 00:00:00+00\"]]}}, \"description\": \"Description of my test collection.\", \"stac_extensions\": []}], \"numberMatched\": 9, \"numberReturned\": 9}\n\nselect collection_search('{\"ids\":[\"testcollection_1\",\"testcollection_2\"], \"fields\": {\"include\": [\"title\"]}}');\n {\"links\": [], \"collections\": [{\"id\": \"testcollection_1\", \"title\": \"My Test Collection.\"}, {\"id\": \"testcollection_2\", \"title\": \"My Test Collection.\"}], \"numberMatched\": 2, \"numberReturned\": 2}\n" }, { "path": "src/pgstac/tests/basic/cql2_searches.sql", "content": "SET ROLE pgstac_read;\nSET pgstac.\"default_filter_lang\" TO 'cql2-json';\n\nSELECT search('{\"ids\":[\"pgstac-test-item-0097\"],\"fields\":{\"include\":[\"id\"]}}');\n\nSELECT search('{\"ids\":[\"pgstac-test-item-0097\",\"pgstac-test-item-0003\"],\"fields\":{\"include\":[\"id\"]}}');\n\n\nSELECT search('{\"collections\":[\"pgstac-test-collection\"],\"fields\":{\"include\":[\"id\"]}, \"limit\": 1}');\n\nSELECT search('{\"collections\":[\"something\"]}');\n\nSELECT search('{\"collections\":[\"something\"],\"fields\":{\"include\":[\"id\"]}}');\n\nSELECT search('{\"filter\":{\"op\":\"t_intersects\", \"args\":[{\"property\":\"datetime\"},\"2011-08-16T00:00:00Z/2011-08-17T00:00:00Z\"]}, \"fields\":{\"include\":[\"id\",\"properties.datetime\",\"properties.eo:cloud_cover\"]},\"sortby\":[{\"field\":\"datetime\",\"direction\":\"desc\"},{\"field\":\"id\",\"direction\":\"asc\"}]}');\n\nSELECT search('{\"filter\":{\"op\":\"t_intersects\", \"args\":[{\"property\":\"datetime\"},\"2011-08-16T00:00:00Z/2011-08-17T00:00:00Z\"]}, \"fields\":{\"include\":[\"id\",\"properties.datetime\",\"properties.eo:cloud_cover\"]},\"sortby\":[{\"field\":\"datetime\",\"direction\":\"desc\"},{\"field\":\"id\",\"direction\":\"asc\"}]}');\n\nSELECT search('{\"filter\":{\"op\":\"t_intersects\", \"args\":[{\"property\":\"datetime\"},\"2011-08-16T00:00:00Z/2011-08-17T00:00:00Z\"]}, \"fields\":{\"include\":[\"id\",\"properties.datetime\",\"properties.eo:cloud_cover\"]},\"sortby\":[{\"field\":\"datetime\",\"direction\":\"desc\"},{\"field\":\"id\",\"direction\":\"asc\"}]}');\n\nSELECT search('{\"filter\":{\"op\":\"eq\", \"args\":[{\"property\":\"eo:cloud_cover\"},36]}, \"fields\":{\"include\":[\"id\",\"properties.datetime\",\"properties.eo:cloud_cover\"]},\"sortby\":[{\"field\":\"datetime\",\"direction\":\"desc\"},{\"field\":\"id\",\"direction\":\"asc\"}]}');\n\nSELECT search('{\"filter\":{\"op\":\"lt\", \"args\":[{\"property\":\"eo:cloud_cover\"},25]}, \"fields\":{\"include\":[\"id\",\"properties.datetime\",\"properties.eo:cloud_cover\"]},\"sortby\":[{\"field\":\"eo:cloud_cover\",\"direction\":\"asc\"}]}');\n\nSELECT search('{\"filter\":{\"op\":\"in\",\"args\":[{\"property\":\"id\"},[\"pgstac-test-item-0097\"]]},\"fields\":{\"include\":[\"id\"]}}');\n\n\nSELECT search('{\"filter\":{\"op\":\"in\",\"args\":[{\"property\":\"id\"},[\"pgstac-test-item-0097\",\"pgstac-test-item-0003\"]]},\"fields\":{\"include\":[\"id\"]}}');\n\nSELECT search('{\"filter\":{\"op\":\"in\",\"args\":[{\"property\":\"collection\"},[\"pgstac-test-collection\"]]},\"fields\":{\"include\":[\"id\"]}, \"limit\": 1}');\n\nSELECT search('{\"filter\":{\"op\":\"in\",\"args\":[{\"property\":\"collection\"},[\"nonexistent\"]]}}');\n\nSELECT search('{\"filter\":{\"op\":\"in\",\"args\":[{\"property\":\"collection\"},[\"nonexistent\"]]}, \"conf\":{\"context\":\"off\"}}');\n\nSELECT search('{\"conf\": {\"nohydrate\": true}, \"limit\": 2}');\n\nSELECT search('{\"filter\":{\"op\":\"in\",\"args\":[{\"property\":\"naip:state\"},[\"zz\",\"xx\"]]},\"fields\":{\"include\":[\"id\"]}}');\n\nSELECT search('{\"filter\":{\"op\":\"in\",\"args\":[{\"property\":\"naip:year\"},[2012,2013]]},\"fields\":{\"include\":[\"id\"]}}');\n\nSELECT search('{\"filter\":{\"op\":\"a_equals\",\"args\":[{\"property\":\"proj:bbox\"},[654842, 3423507, 661516, 3431125]]},\"fields\":{\"include\":[\"id\"]}}');\n\nSELECT search('{\"filter\":{\"op\":\"a_equals\",\"args\":[[654842, 3423507, 661516, 3431125],{\"property\":\"proj:bbox\"}]},\"fields\":{\"include\":[\"id\"]}}');\n\nSELECT search('{\"filter\":{\"op\":\"a_equals\",\"args\":[[654842, 3423507, 661516],{\"property\":\"proj:bbox\"}]},\"fields\":{\"include\":[\"id\"]}}');\n\nSELECT search('{\"filter\":{\"op\":\"a_overlaps\",\"args\":[{\"property\":\"proj:bbox\"},[654842, 3423507, 661516, 12345]]},\"fields\":{\"include\":[\"id\"]}}');\n\nSELECT search('{\"filter\":{\"op\":\"a_overlaps\",\"args\":[{\"property\":\"proj:bbox\"},[12345]]},\"fields\":{\"include\":[\"id\"]}}');\n\nSELECT search('{\"filter\":{\"op\":\"a_contains\",\"args\":[{\"property\":\"proj:bbox\"},[654842, 3423507, 661516]]},\"fields\":{\"include\":[\"id\"]}}');\n\nSELECT search('{\"filter\":{\"op\":\"a_contains\",\"args\":[{\"property\":\"proj:bbox\"},[654842, 3423507, 661516, 12345]]},\"fields\":{\"include\":[\"id\"]}}');\n\nSELECT search('{\"filter\":{\"op\":\"a_contained_by\",\"args\":[{\"property\":\"proj:bbox\"},[654842, 3423507, 661516]]},\"fields\":{\"include\":[\"id\"]}}');\n\nSELECT search('{\"filter\":{\"op\":\"a_contained_by\",\"args\":[{\"property\":\"proj:bbox\"},[654842, 3423507, 661516, 3431125, 234324]]},\"fields\":{\"include\":[\"id\"]}}');\n\n-- Test Paging\nSELECT search('{\"fields\":{\"include\":[\"id\",\"properties.datetime\",\"properties.eo:cloud_cover\"]},\"sortby\":[{\"field\":\"properties.eo:cloud_cover\",\"direction\":\"asc\"},{\"field\":\"datetime\",\"direction\":\"desc\"},{\"field\":\"id\",\"direction\":\"asc\"}]}');\n\nSELECT search('{\"token\":\"next:pgstac-test-item-0048\", \"fields\":{\"include\":[\"id\",\"properties.datetime\",\"properties.eo:cloud_cover\"]},\"sortby\":[{\"field\":\"properties.eo:cloud_cover\",\"direction\":\"asc\"},{\"field\":\"datetime\",\"direction\":\"desc\"},{\"field\":\"id\",\"direction\":\"asc\"}]}');\n\nSELECT search('{\"token\":\"next:pgstac-test-item-0016\", \"fields\":{\"include\":[\"id\",\"properties.datetime\",\"properties.eo:cloud_cover\"]},\"sortby\":[{\"field\":\"properties.eo:cloud_cover\",\"direction\":\"asc\"},{\"field\":\"datetime\",\"direction\":\"desc\"},{\"field\":\"id\",\"direction\":\"asc\"}]}');\n\nSELECT search('{\"token\":\"prev:pgstac-test-item-0030\", \"fields\":{\"include\":[\"id\",\"properties.datetime\",\"properties.eo:cloud_cover\"]},\"sortby\":[{\"field\":\"properties.eo:cloud_cover\",\"direction\":\"asc\"},{\"field\":\"datetime\",\"direction\":\"desc\"},{\"field\":\"id\",\"direction\":\"asc\"}]}');\n\nSELECT search('{\"token\":\"prev:pgstac-test-item-0054\", \"fields\":{\"include\":[\"id\",\"properties.datetime\",\"properties.eo:cloud_cover\"]},\"sortby\":[{\"field\":\"properties.eo:cloud_cover\",\"direction\":\"asc\"},{\"field\":\"datetime\",\"direction\":\"desc\"},{\"field\":\"id\",\"direction\":\"asc\"}]}');\n\n-- Test paging without fields extension\nSELECT jsonb_path_query(search('{\"fields\":{\"include\":[\"id\",\"properties.datetime\",\"properties.eo:cloud_cover\"]},\"sortby\":[{\"field\":\"properties.eo:cloud_cover\",\"direction\":\"asc\"},{\"field\":\"datetime\",\"direction\":\"desc\"},{\"field\":\"id\",\"direction\":\"asc\"}]}'), '$.features[*].id');\n\nSELECT jsonb_path_query(search('{\"token\":\"next:pgstac-test-item-0048\",\"sortby\":[{\"field\":\"properties.eo:cloud_cover\",\"direction\":\"asc\"},{\"field\":\"datetime\",\"direction\":\"desc\"},{\"field\":\"id\",\"direction\":\"asc\"}]}'), '$.features[*].id');\n\nSELECT jsonb_path_query(search('{\"token\":\"next:pgstac-test-item-0016\",\"sortby\":[{\"field\":\"properties.eo:cloud_cover\",\"direction\":\"asc\"},{\"field\":\"datetime\",\"direction\":\"desc\"},{\"field\":\"id\",\"direction\":\"asc\"}]}'), '$.features[*].id');\n\nSELECT jsonb_path_query(search('{\"token\":\"prev:pgstac-test-item-0030\",\"sortby\":[{\"field\":\"properties.eo:cloud_cover\",\"direction\":\"asc\"},{\"field\":\"datetime\",\"direction\":\"desc\"},{\"field\":\"id\",\"direction\":\"asc\"}]}'), '$.features[*].id');\n\nSELECT jsonb_path_query(search('{\"token\":\"prev:pgstac-test-item-0054\",\"sortby\":[{\"field\":\"properties.eo:cloud_cover\",\"direction\":\"asc\"},{\"field\":\"datetime\",\"direction\":\"desc\"},{\"field\":\"id\",\"direction\":\"asc\"}]}'), '$.features[*].id');\n\nSELECT search('{\"collections\": [\"pgstac-test-collection\"], \"limit\": 1}');\nSELECT search('{\"collections\": [\"pgstac-test-collection\"], \"limit\": 1, \"token\": \"next:pgstac-test-item-0001\"}');\n" }, { "path": "src/pgstac/tests/basic/cql2_searches.sql.out", "content": "SET ROLE pgstac_read;\nSET\nSET pgstac.\"default_filter_lang\" TO 'cql2-json';\nSET\nSELECT search('{\"ids\":[\"pgstac-test-item-0097\"],\"fields\":{\"include\":[\"id\"]}}');\n {\"type\": \"FeatureCollection\", \"links\": [{\"rel\": \"root\", \"href\": \".\", \"type\": \"application/json\"}, {\"rel\": \"self\", \"href\": \"./search\", \"type\": \"application/json\"}], \"features\": [{\"id\": \"pgstac-test-item-0097\", \"collection\": \"pgstac-test-collection\"}], \"numberMatched\": 1, \"numberReturned\": 1}\n\nSELECT search('{\"ids\":[\"pgstac-test-item-0097\",\"pgstac-test-item-0003\"],\"fields\":{\"include\":[\"id\"]}}');\n {\"type\": \"FeatureCollection\", \"links\": [{\"rel\": \"root\", \"href\": \".\", \"type\": \"application/json\"}, {\"rel\": \"self\", \"href\": \"./search\", \"type\": \"application/json\"}], \"features\": [{\"id\": \"pgstac-test-item-0003\", \"collection\": \"pgstac-test-collection\"}, {\"id\": \"pgstac-test-item-0097\", \"collection\": \"pgstac-test-collection\"}], \"numberMatched\": 2, \"numberReturned\": 2}\n\nSELECT search('{\"collections\":[\"pgstac-test-collection\"],\"fields\":{\"include\":[\"id\"]}, \"limit\": 1}');\n {\"type\": \"FeatureCollection\", \"links\": [{\"rel\": \"root\", \"href\": \".\", \"type\": \"application/json\"}, {\"rel\": \"self\", \"href\": \"./search\", \"type\": \"application/json\"}, {\"rel\": \"next\", \"href\": \"./search?token=next:pgstac-test-collection:pgstac-test-item-0003\", \"type\": \"application/geo+json\", \"method\": \"GET\"}], \"features\": [{\"id\": \"pgstac-test-item-0003\", \"collection\": \"pgstac-test-collection\"}], \"numberMatched\": 100, \"numberReturned\": 1}\n\nSELECT search('{\"collections\":[\"something\"]}');\n {\"type\": \"FeatureCollection\", \"links\": [{\"rel\": \"root\", \"href\": \".\", \"type\": \"application/json\"}, {\"rel\": \"self\", \"href\": \"./search\", \"type\": \"application/json\"}], \"features\": [], \"numberMatched\": 0, \"numberReturned\": 0}\n\nSELECT search('{\"collections\":[\"something\"],\"fields\":{\"include\":[\"id\"]}}');\n {\"type\": \"FeatureCollection\", \"links\": [{\"rel\": \"root\", \"href\": \".\", \"type\": \"application/json\"}, {\"rel\": \"self\", \"href\": \"./search\", \"type\": \"application/json\"}], \"features\": [], \"numberMatched\": 0, \"numberReturned\": 0}\n\nSELECT search('{\"filter\":{\"op\":\"t_intersects\", \"args\":[{\"property\":\"datetime\"},\"2011-08-16T00:00:00Z/2011-08-17T00:00:00Z\"]}, \"fields\":{\"include\":[\"id\",\"properties.datetime\",\"properties.eo:cloud_cover\"]},\"sortby\":[{\"field\":\"datetime\",\"direction\":\"desc\"},{\"field\":\"id\",\"direction\":\"asc\"}]}');\n {\"type\": \"FeatureCollection\", \"links\": [{\"rel\": \"root\", \"href\": \".\", \"type\": \"application/json\"}, {\"rel\": \"self\", \"href\": \"./search\", \"type\": \"application/json\"}, {\"rel\": \"next\", \"href\": \"./search?token=next:pgstac-test-collection:pgstac-test-item-0016\", \"type\": \"application/geo+json\", \"method\": \"GET\"}], \"features\": [{\"id\": \"pgstac-test-item-0007\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-17T00:00:00Z\", \"eo:cloud_cover\": 59}}, {\"id\": \"pgstac-test-item-0008\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-17T00:00:00Z\", \"eo:cloud_cover\": 64}}, {\"id\": \"pgstac-test-item-0009\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-17T00:00:00Z\", \"eo:cloud_cover\": 61}}, {\"id\": \"pgstac-test-item-0010\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-17T00:00:00Z\", \"eo:cloud_cover\": 31}}, {\"id\": \"pgstac-test-item-0011\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-17T00:00:00Z\", \"eo:cloud_cover\": 41}}, {\"id\": \"pgstac-test-item-0012\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-17T00:00:00Z\", \"eo:cloud_cover\": 4}}, {\"id\": \"pgstac-test-item-0013\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-17T00:00:00Z\", \"eo:cloud_cover\": 2}}, {\"id\": \"pgstac-test-item-0014\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-16T00:00:00Z\", \"eo:cloud_cover\": 17}}, {\"id\": \"pgstac-test-item-0015\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-16T00:00:00Z\", \"eo:cloud_cover\": 54}}, {\"id\": \"pgstac-test-item-0016\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-16T00:00:00Z\", \"eo:cloud_cover\": 13}}], \"numberMatched\": 57, \"numberReturned\": 10}\n\nSELECT search('{\"filter\":{\"op\":\"t_intersects\", \"args\":[{\"property\":\"datetime\"},\"2011-08-16T00:00:00Z/2011-08-17T00:00:00Z\"]}, \"fields\":{\"include\":[\"id\",\"properties.datetime\",\"properties.eo:cloud_cover\"]},\"sortby\":[{\"field\":\"datetime\",\"direction\":\"desc\"},{\"field\":\"id\",\"direction\":\"asc\"}]}');\n {\"type\": \"FeatureCollection\", \"links\": [{\"rel\": \"root\", \"href\": \".\", \"type\": \"application/json\"}, {\"rel\": \"self\", \"href\": \"./search\", \"type\": \"application/json\"}, {\"rel\": \"next\", \"href\": \"./search?token=next:pgstac-test-collection:pgstac-test-item-0016\", \"type\": \"application/geo+json\", \"method\": \"GET\"}], \"features\": [{\"id\": \"pgstac-test-item-0007\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-17T00:00:00Z\", \"eo:cloud_cover\": 59}}, {\"id\": \"pgstac-test-item-0008\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-17T00:00:00Z\", \"eo:cloud_cover\": 64}}, {\"id\": \"pgstac-test-item-0009\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-17T00:00:00Z\", \"eo:cloud_cover\": 61}}, {\"id\": \"pgstac-test-item-0010\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-17T00:00:00Z\", \"eo:cloud_cover\": 31}}, {\"id\": \"pgstac-test-item-0011\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-17T00:00:00Z\", \"eo:cloud_cover\": 41}}, {\"id\": \"pgstac-test-item-0012\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-17T00:00:00Z\", \"eo:cloud_cover\": 4}}, {\"id\": \"pgstac-test-item-0013\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-17T00:00:00Z\", \"eo:cloud_cover\": 2}}, {\"id\": \"pgstac-test-item-0014\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-16T00:00:00Z\", \"eo:cloud_cover\": 17}}, {\"id\": \"pgstac-test-item-0015\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-16T00:00:00Z\", \"eo:cloud_cover\": 54}}, {\"id\": \"pgstac-test-item-0016\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-16T00:00:00Z\", \"eo:cloud_cover\": 13}}], \"numberMatched\": 57, \"numberReturned\": 10}\n\nSELECT search('{\"filter\":{\"op\":\"t_intersects\", \"args\":[{\"property\":\"datetime\"},\"2011-08-16T00:00:00Z/2011-08-17T00:00:00Z\"]}, \"fields\":{\"include\":[\"id\",\"properties.datetime\",\"properties.eo:cloud_cover\"]},\"sortby\":[{\"field\":\"datetime\",\"direction\":\"desc\"},{\"field\":\"id\",\"direction\":\"asc\"}]}');\n {\"type\": \"FeatureCollection\", \"links\": [{\"rel\": \"root\", \"href\": \".\", \"type\": \"application/json\"}, {\"rel\": \"self\", \"href\": \"./search\", \"type\": \"application/json\"}, {\"rel\": \"next\", \"href\": \"./search?token=next:pgstac-test-collection:pgstac-test-item-0016\", \"type\": \"application/geo+json\", \"method\": \"GET\"}], \"features\": [{\"id\": \"pgstac-test-item-0007\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-17T00:00:00Z\", \"eo:cloud_cover\": 59}}, {\"id\": \"pgstac-test-item-0008\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-17T00:00:00Z\", \"eo:cloud_cover\": 64}}, {\"id\": \"pgstac-test-item-0009\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-17T00:00:00Z\", \"eo:cloud_cover\": 61}}, {\"id\": \"pgstac-test-item-0010\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-17T00:00:00Z\", \"eo:cloud_cover\": 31}}, {\"id\": \"pgstac-test-item-0011\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-17T00:00:00Z\", \"eo:cloud_cover\": 41}}, {\"id\": \"pgstac-test-item-0012\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-17T00:00:00Z\", \"eo:cloud_cover\": 4}}, {\"id\": \"pgstac-test-item-0013\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-17T00:00:00Z\", \"eo:cloud_cover\": 2}}, {\"id\": \"pgstac-test-item-0014\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-16T00:00:00Z\", \"eo:cloud_cover\": 17}}, {\"id\": \"pgstac-test-item-0015\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-16T00:00:00Z\", \"eo:cloud_cover\": 54}}, {\"id\": \"pgstac-test-item-0016\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-16T00:00:00Z\", \"eo:cloud_cover\": 13}}], \"numberMatched\": 57, \"numberReturned\": 10}\n\nSELECT search('{\"filter\":{\"op\":\"eq\", \"args\":[{\"property\":\"eo:cloud_cover\"},36]}, \"fields\":{\"include\":[\"id\",\"properties.datetime\",\"properties.eo:cloud_cover\"]},\"sortby\":[{\"field\":\"datetime\",\"direction\":\"desc\"},{\"field\":\"id\",\"direction\":\"asc\"}]}');\n {\"type\": \"FeatureCollection\", \"links\": [{\"rel\": \"root\", \"href\": \".\", \"type\": \"application/json\"}, {\"rel\": \"self\", \"href\": \"./search\", \"type\": \"application/json\"}], \"features\": [{\"id\": \"pgstac-test-item-0087\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-01T00:00:00Z\", \"eo:cloud_cover\": 36}}, {\"id\": \"pgstac-test-item-0089\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-07-31T00:00:00Z\", \"eo:cloud_cover\": 36}}], \"numberMatched\": 2, \"numberReturned\": 2}\n\nSELECT search('{\"filter\":{\"op\":\"lt\", \"args\":[{\"property\":\"eo:cloud_cover\"},25]}, \"fields\":{\"include\":[\"id\",\"properties.datetime\",\"properties.eo:cloud_cover\"]},\"sortby\":[{\"field\":\"eo:cloud_cover\",\"direction\":\"asc\"}]}');\n {\"type\": \"FeatureCollection\", \"links\": [{\"rel\": \"root\", \"href\": \".\", \"type\": \"application/json\"}, {\"rel\": \"self\", \"href\": \"./search\", \"type\": \"application/json\"}, {\"rel\": \"next\", \"href\": \"./search?token=next:pgstac-test-collection:pgstac-test-item-0012\", \"type\": \"application/geo+json\", \"method\": \"GET\"}], \"features\": [{\"id\": \"pgstac-test-item-0097\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-07-31T00:00:00Z\", \"eo:cloud_cover\": 1}}, {\"id\": \"pgstac-test-item-0063\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-16T00:00:00Z\", \"eo:cloud_cover\": 2}}, {\"id\": \"pgstac-test-item-0013\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-17T00:00:00Z\", \"eo:cloud_cover\": 2}}, {\"id\": \"pgstac-test-item-0085\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-01T00:00:00Z\", \"eo:cloud_cover\": 3}}, {\"id\": \"pgstac-test-item-0073\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-15T00:00:00Z\", \"eo:cloud_cover\": 3}}, {\"id\": \"pgstac-test-item-0041\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-16T00:00:00Z\", \"eo:cloud_cover\": 3}}, {\"id\": \"pgstac-test-item-0034\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-16T00:00:00Z\", \"eo:cloud_cover\": 3}}, {\"id\": \"pgstac-test-item-0005\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-24T00:00:00Z\", \"eo:cloud_cover\": 3}}, {\"id\": \"pgstac-test-item-0048\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-16T00:00:00Z\", \"eo:cloud_cover\": 4}}, {\"id\": \"pgstac-test-item-0012\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-17T00:00:00Z\", \"eo:cloud_cover\": 4}}], \"numberMatched\": 31, \"numberReturned\": 10}\n\nSELECT search('{\"filter\":{\"op\":\"in\",\"args\":[{\"property\":\"id\"},[\"pgstac-test-item-0097\"]]},\"fields\":{\"include\":[\"id\"]}}');\n {\"type\": \"FeatureCollection\", \"links\": [{\"rel\": \"root\", \"href\": \".\", \"type\": \"application/json\"}, {\"rel\": \"self\", \"href\": \"./search\", \"type\": \"application/json\"}], \"features\": [{\"id\": \"pgstac-test-item-0097\", \"collection\": \"pgstac-test-collection\"}], \"numberMatched\": 1, \"numberReturned\": 1}\n\nSELECT search('{\"filter\":{\"op\":\"in\",\"args\":[{\"property\":\"id\"},[\"pgstac-test-item-0097\",\"pgstac-test-item-0003\"]]},\"fields\":{\"include\":[\"id\"]}}');\n {\"type\": \"FeatureCollection\", \"links\": [{\"rel\": \"root\", \"href\": \".\", \"type\": \"application/json\"}, {\"rel\": \"self\", \"href\": \"./search\", \"type\": \"application/json\"}], \"features\": [{\"id\": \"pgstac-test-item-0003\", \"collection\": \"pgstac-test-collection\"}, {\"id\": \"pgstac-test-item-0097\", \"collection\": \"pgstac-test-collection\"}], \"numberMatched\": 2, \"numberReturned\": 2}\n\nSELECT search('{\"filter\":{\"op\":\"in\",\"args\":[{\"property\":\"collection\"},[\"pgstac-test-collection\"]]},\"fields\":{\"include\":[\"id\"]}, \"limit\": 1}');\n {\"type\": \"FeatureCollection\", \"links\": [{\"rel\": \"root\", \"href\": \".\", \"type\": \"application/json\"}, {\"rel\": \"self\", \"href\": \"./search\", \"type\": \"application/json\"}, {\"rel\": \"next\", \"href\": \"./search?token=next:pgstac-test-collection:pgstac-test-item-0003\", \"type\": \"application/geo+json\", \"method\": \"GET\"}], \"features\": [{\"id\": \"pgstac-test-item-0003\", \"collection\": \"pgstac-test-collection\"}], \"numberMatched\": 100, \"numberReturned\": 1}\n\nSELECT search('{\"filter\":{\"op\":\"in\",\"args\":[{\"property\":\"collection\"},[\"nonexistent\"]]}}');\n {\"type\": \"FeatureCollection\", \"links\": [{\"rel\": \"root\", \"href\": \".\", \"type\": \"application/json\"}, {\"rel\": \"self\", \"href\": \"./search\", \"type\": \"application/json\"}], \"features\": [], \"numberMatched\": 0, \"numberReturned\": 0}\n\nSELECT search('{\"filter\":{\"op\":\"in\",\"args\":[{\"property\":\"collection\"},[\"nonexistent\"]]}, \"conf\":{\"context\":\"off\"}}');\n {\"type\": \"FeatureCollection\", \"links\": [{\"rel\": \"root\", \"href\": \".\", \"type\": \"application/json\"}, {\"rel\": \"self\", \"href\": \"./search\", \"type\": \"application/json\"}], \"features\": [], \"numberReturned\": 0}\n\nSELECT search('{\"conf\": {\"nohydrate\": true}, \"limit\": 2}');\n {\"type\": \"FeatureCollection\", \"links\": [{\"rel\": \"root\", \"href\": \".\", \"type\": \"application/json\"}, {\"rel\": \"self\", \"href\": \"./search\", \"type\": \"application/json\"}, {\"rel\": \"next\", \"href\": \"./search?token=next:pgstac-test-collection:pgstac-test-item-0002\", \"type\": \"application/geo+json\", \"method\": \"GET\"}], \"features\": [{\"id\": \"pgstac-test-item-0003\", \"bbox\": [-85.379245, 30.933949, -85.308201, 31.003555], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30085/m_3008506_nw_16_1_20110825.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30085/m_3008506_nw_16_1_20110825.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30085/m_3008506_nw_16_1_20110825.200.jpg\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-85.309412, 30.933949], [-85.308201, 31.002658], [-85.378084, 31.003555], [-85.379245, 30.934843], [-85.309412, 30.933949]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-25T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [654842, 3423507, 661516, 3431125], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7618, 6674], \"eo:cloud_cover\": 28, \"proj:transform\": [1, 0, 654842, 0, -1, 3431125, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}, {\"id\": \"pgstac-test-item-0002\", \"bbox\": [-85.504167, 30.934008, -85.433293, 31.003486], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30085/m_3008505_nw_16_1_20110825.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30085/m_3008505_nw_16_1_20110825.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30085/m_3008505_nw_16_1_20110825.200.jpg\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-85.434414, 30.934008], [-85.433293, 31.002658], [-85.503096, 31.003486], [-85.504167, 30.934834], [-85.434414, 30.934008]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-25T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [642906, 3423339, 649572, 3430950], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7611, 6666], \"eo:cloud_cover\": 33, \"proj:transform\": [1, 0, 642906, 0, -1, 3430950, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}], \"numberMatched\": 100, \"numberReturned\": 2}\n\nSELECT search('{\"filter\":{\"op\":\"in\",\"args\":[{\"property\":\"naip:state\"},[\"zz\",\"xx\"]]},\"fields\":{\"include\":[\"id\"]}}');\n {\"type\": \"FeatureCollection\", \"links\": [{\"rel\": \"root\", \"href\": \".\", \"type\": \"application/json\"}, {\"rel\": \"self\", \"href\": \"./search\", \"type\": \"application/json\"}], \"features\": [{\"id\": \"pgstac-test-item-0085\", \"collection\": \"pgstac-test-collection\"}, {\"id\": \"pgstac-test-item-0100\", \"collection\": \"pgstac-test-collection\"}], \"numberMatched\": 2, \"numberReturned\": 2}\n\nSELECT search('{\"filter\":{\"op\":\"in\",\"args\":[{\"property\":\"naip:year\"},[2012,2013]]},\"fields\":{\"include\":[\"id\"]}}');\n {\"type\": \"FeatureCollection\", \"links\": [{\"rel\": \"root\", \"href\": \".\", \"type\": \"application/json\"}, {\"rel\": \"self\", \"href\": \"./search\", \"type\": \"application/json\"}], \"features\": [{\"id\": \"pgstac-test-item-0085\", \"collection\": \"pgstac-test-collection\"}, {\"id\": \"pgstac-test-item-0100\", \"collection\": \"pgstac-test-collection\"}], \"numberMatched\": 2, \"numberReturned\": 2}\n\nSELECT search('{\"filter\":{\"op\":\"a_equals\",\"args\":[{\"property\":\"proj:bbox\"},[654842, 3423507, 661516, 3431125]]},\"fields\":{\"include\":[\"id\"]}}');\n {\"type\": \"FeatureCollection\", \"links\": [{\"rel\": \"root\", \"href\": \".\", \"type\": \"application/json\"}, {\"rel\": \"self\", \"href\": \"./search\", \"type\": \"application/json\"}], \"features\": [{\"id\": \"pgstac-test-item-0003\", \"collection\": \"pgstac-test-collection\"}], \"numberMatched\": 1, \"numberReturned\": 1}\n\nSELECT search('{\"filter\":{\"op\":\"a_equals\",\"args\":[[654842, 3423507, 661516, 3431125],{\"property\":\"proj:bbox\"}]},\"fields\":{\"include\":[\"id\"]}}');\n {\"type\": \"FeatureCollection\", \"links\": [{\"rel\": \"root\", \"href\": \".\", \"type\": \"application/json\"}, {\"rel\": \"self\", \"href\": \"./search\", \"type\": \"application/json\"}], \"features\": [{\"id\": \"pgstac-test-item-0003\", \"collection\": \"pgstac-test-collection\"}], \"numberMatched\": 1, \"numberReturned\": 1}\n\nSELECT search('{\"filter\":{\"op\":\"a_equals\",\"args\":[[654842, 3423507, 661516],{\"property\":\"proj:bbox\"}]},\"fields\":{\"include\":[\"id\"]}}');\n {\"type\": \"FeatureCollection\", \"links\": [{\"rel\": \"root\", \"href\": \".\", \"type\": \"application/json\"}, {\"rel\": \"self\", \"href\": \"./search\", \"type\": \"application/json\"}], \"features\": [], \"numberMatched\": 0, \"numberReturned\": 0}\n\nSELECT search('{\"filter\":{\"op\":\"a_overlaps\",\"args\":[{\"property\":\"proj:bbox\"},[654842, 3423507, 661516, 12345]]},\"fields\":{\"include\":[\"id\"]}}');\n {\"type\": \"FeatureCollection\", \"links\": [{\"rel\": \"root\", \"href\": \".\", \"type\": \"application/json\"}, {\"rel\": \"self\", \"href\": \"./search\", \"type\": \"application/json\"}], \"features\": [{\"id\": \"pgstac-test-item-0003\", \"collection\": \"pgstac-test-collection\"}], \"numberMatched\": 1, \"numberReturned\": 1}\n\nSELECT search('{\"filter\":{\"op\":\"a_overlaps\",\"args\":[{\"property\":\"proj:bbox\"},[12345]]},\"fields\":{\"include\":[\"id\"]}}');\n {\"type\": \"FeatureCollection\", \"links\": [{\"rel\": \"root\", \"href\": \".\", \"type\": \"application/json\"}, {\"rel\": \"self\", \"href\": \"./search\", \"type\": \"application/json\"}], \"features\": [], \"numberMatched\": 0, \"numberReturned\": 0}\n\nSELECT search('{\"filter\":{\"op\":\"a_contains\",\"args\":[{\"property\":\"proj:bbox\"},[654842, 3423507, 661516]]},\"fields\":{\"include\":[\"id\"]}}');\n {\"type\": \"FeatureCollection\", \"links\": [{\"rel\": \"root\", \"href\": \".\", \"type\": \"application/json\"}, {\"rel\": \"self\", \"href\": \"./search\", \"type\": \"application/json\"}], \"features\": [{\"id\": \"pgstac-test-item-0003\", \"collection\": \"pgstac-test-collection\"}], \"numberMatched\": 1, \"numberReturned\": 1}\n\nSELECT search('{\"filter\":{\"op\":\"a_contains\",\"args\":[{\"property\":\"proj:bbox\"},[654842, 3423507, 661516, 12345]]},\"fields\":{\"include\":[\"id\"]}}');\n {\"type\": \"FeatureCollection\", \"links\": [{\"rel\": \"root\", \"href\": \".\", \"type\": \"application/json\"}, {\"rel\": \"self\", \"href\": \"./search\", \"type\": \"application/json\"}], \"features\": [], \"numberMatched\": 0, \"numberReturned\": 0}\n\nSELECT search('{\"filter\":{\"op\":\"a_contained_by\",\"args\":[{\"property\":\"proj:bbox\"},[654842, 3423507, 661516]]},\"fields\":{\"include\":[\"id\"]}}');\n {\"type\": \"FeatureCollection\", \"links\": [{\"rel\": \"root\", \"href\": \".\", \"type\": \"application/json\"}, {\"rel\": \"self\", \"href\": \"./search\", \"type\": \"application/json\"}], \"features\": [], \"numberMatched\": 0, \"numberReturned\": 0}\n\nSELECT search('{\"filter\":{\"op\":\"a_contained_by\",\"args\":[{\"property\":\"proj:bbox\"},[654842, 3423507, 661516, 3431125, 234324]]},\"fields\":{\"include\":[\"id\"]}}');\n {\"type\": \"FeatureCollection\", \"links\": [{\"rel\": \"root\", \"href\": \".\", \"type\": \"application/json\"}, {\"rel\": \"self\", \"href\": \"./search\", \"type\": \"application/json\"}], \"features\": [{\"id\": \"pgstac-test-item-0003\", \"collection\": \"pgstac-test-collection\"}], \"numberMatched\": 1, \"numberReturned\": 1}\n\n-- Test Paging\nSELECT search('{\"fields\":{\"include\":[\"id\",\"properties.datetime\",\"properties.eo:cloud_cover\"]},\"sortby\":[{\"field\":\"properties.eo:cloud_cover\",\"direction\":\"asc\"},{\"field\":\"datetime\",\"direction\":\"desc\"},{\"field\":\"id\",\"direction\":\"asc\"}]}');\n {\"type\": \"FeatureCollection\", \"links\": [{\"rel\": \"root\", \"href\": \".\", \"type\": \"application/json\"}, {\"rel\": \"self\", \"href\": \"./search\", \"type\": \"application/json\"}, {\"rel\": \"next\", \"href\": \"./search?token=next:pgstac-test-collection:pgstac-test-item-0048\", \"type\": \"application/geo+json\", \"method\": \"GET\"}], \"features\": [{\"id\": \"pgstac-test-item-0097\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-07-31T00:00:00Z\", \"eo:cloud_cover\": 1}}, {\"id\": \"pgstac-test-item-0013\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-17T00:00:00Z\", \"eo:cloud_cover\": 2}}, {\"id\": \"pgstac-test-item-0063\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-16T00:00:00Z\", \"eo:cloud_cover\": 2}}, {\"id\": \"pgstac-test-item-0005\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-24T00:00:00Z\", \"eo:cloud_cover\": 3}}, {\"id\": \"pgstac-test-item-0034\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-16T00:00:00Z\", \"eo:cloud_cover\": 3}}, {\"id\": \"pgstac-test-item-0041\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-16T00:00:00Z\", \"eo:cloud_cover\": 3}}, {\"id\": \"pgstac-test-item-0073\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-15T00:00:00Z\", \"eo:cloud_cover\": 3}}, {\"id\": \"pgstac-test-item-0085\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-01T00:00:00Z\", \"eo:cloud_cover\": 3}}, {\"id\": \"pgstac-test-item-0012\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-17T00:00:00Z\", \"eo:cloud_cover\": 4}}, {\"id\": \"pgstac-test-item-0048\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-16T00:00:00Z\", \"eo:cloud_cover\": 4}}], \"numberMatched\": 100, \"numberReturned\": 10}\n\nSELECT search('{\"token\":\"next:pgstac-test-item-0048\", \"fields\":{\"include\":[\"id\",\"properties.datetime\",\"properties.eo:cloud_cover\"]},\"sortby\":[{\"field\":\"properties.eo:cloud_cover\",\"direction\":\"asc\"},{\"field\":\"datetime\",\"direction\":\"desc\"},{\"field\":\"id\",\"direction\":\"asc\"}]}');\n {\"type\": \"FeatureCollection\", \"links\": [{\"rel\": \"root\", \"href\": \".\", \"type\": \"application/json\"}, {\"rel\": \"self\", \"href\": \"./search\", \"type\": \"application/json\"}, {\"rel\": \"next\", \"href\": \"./search?token=next:pgstac-test-collection:pgstac-test-item-0016\", \"type\": \"application/geo+json\", \"method\": \"GET\"}, {\"rel\": \"prev\", \"href\": \"./search?token=prev:pgstac-test-collection:pgstac-test-item-0054\", \"type\": \"application/geo+json\", \"method\": \"GET\"}], \"features\": [{\"id\": \"pgstac-test-item-0054\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-16T00:00:00Z\", \"eo:cloud_cover\": 5}}, {\"id\": \"pgstac-test-item-0032\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-16T00:00:00Z\", \"eo:cloud_cover\": 7}}, {\"id\": \"pgstac-test-item-0057\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-16T00:00:00Z\", \"eo:cloud_cover\": 7}}, {\"id\": \"pgstac-test-item-0075\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-15T00:00:00Z\", \"eo:cloud_cover\": 8}}, {\"id\": \"pgstac-test-item-0049\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-16T00:00:00Z\", \"eo:cloud_cover\": 9}}, {\"id\": \"pgstac-test-item-0040\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-16T00:00:00Z\", \"eo:cloud_cover\": 10}}, {\"id\": \"pgstac-test-item-0036\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-16T00:00:00Z\", \"eo:cloud_cover\": 12}}, {\"id\": \"pgstac-test-item-0039\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-16T00:00:00Z\", \"eo:cloud_cover\": 12}}, {\"id\": \"pgstac-test-item-0052\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-16T00:00:00Z\", \"eo:cloud_cover\": 12}}, {\"id\": \"pgstac-test-item-0016\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-16T00:00:00Z\", \"eo:cloud_cover\": 13}}], \"numberMatched\": 100, \"numberReturned\": 10}\n\nSELECT search('{\"token\":\"next:pgstac-test-item-0016\", \"fields\":{\"include\":[\"id\",\"properties.datetime\",\"properties.eo:cloud_cover\"]},\"sortby\":[{\"field\":\"properties.eo:cloud_cover\",\"direction\":\"asc\"},{\"field\":\"datetime\",\"direction\":\"desc\"},{\"field\":\"id\",\"direction\":\"asc\"}]}');\n {\"type\": \"FeatureCollection\", \"links\": [{\"rel\": \"root\", \"href\": \".\", \"type\": \"application/json\"}, {\"rel\": \"self\", \"href\": \"./search\", \"type\": \"application/json\"}, {\"rel\": \"next\", \"href\": \"./search?token=next:pgstac-test-collection:pgstac-test-item-0053\", \"type\": \"application/geo+json\", \"method\": \"GET\"}, {\"rel\": \"prev\", \"href\": \"./search?token=prev:pgstac-test-collection:pgstac-test-item-0030\", \"type\": \"application/geo+json\", \"method\": \"GET\"}], \"features\": [{\"id\": \"pgstac-test-item-0030\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-16T00:00:00Z\", \"eo:cloud_cover\": 16}}, {\"id\": \"pgstac-test-item-0031\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-16T00:00:00Z\", \"eo:cloud_cover\": 16}}, {\"id\": \"pgstac-test-item-0046\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-16T00:00:00Z\", \"eo:cloud_cover\": 16}}, {\"id\": \"pgstac-test-item-0014\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-16T00:00:00Z\", \"eo:cloud_cover\": 17}}, {\"id\": \"pgstac-test-item-0078\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-15T00:00:00Z\", \"eo:cloud_cover\": 21}}, {\"id\": \"pgstac-test-item-0051\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-16T00:00:00Z\", \"eo:cloud_cover\": 22}}, {\"id\": \"pgstac-test-item-0068\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-15T00:00:00Z\", \"eo:cloud_cover\": 22}}, {\"id\": \"pgstac-test-item-0004\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-24T00:00:00Z\", \"eo:cloud_cover\": 23}}, {\"id\": \"pgstac-test-item-0088\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-01T00:00:00Z\", \"eo:cloud_cover\": 23}}, {\"id\": \"pgstac-test-item-0053\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-16T00:00:00Z\", \"eo:cloud_cover\": 24}}], \"numberMatched\": 100, \"numberReturned\": 10}\n\nSELECT search('{\"token\":\"prev:pgstac-test-item-0030\", \"fields\":{\"include\":[\"id\",\"properties.datetime\",\"properties.eo:cloud_cover\"]},\"sortby\":[{\"field\":\"properties.eo:cloud_cover\",\"direction\":\"asc\"},{\"field\":\"datetime\",\"direction\":\"desc\"},{\"field\":\"id\",\"direction\":\"asc\"}]}');\n {\"type\": \"FeatureCollection\", \"links\": [{\"rel\": \"root\", \"href\": \".\", \"type\": \"application/json\"}, {\"rel\": \"self\", \"href\": \"./search\", \"type\": \"application/json\"}, {\"rel\": \"next\", \"href\": \"./search?token=next:pgstac-test-collection:pgstac-test-item-0016\", \"type\": \"application/geo+json\", \"method\": \"GET\"}, {\"rel\": \"prev\", \"href\": \"./search?token=prev:pgstac-test-collection:pgstac-test-item-0054\", \"type\": \"application/geo+json\", \"method\": \"GET\"}], \"features\": [{\"id\": \"pgstac-test-item-0054\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-16T00:00:00Z\", \"eo:cloud_cover\": 5}}, {\"id\": \"pgstac-test-item-0032\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-16T00:00:00Z\", \"eo:cloud_cover\": 7}}, {\"id\": \"pgstac-test-item-0057\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-16T00:00:00Z\", \"eo:cloud_cover\": 7}}, {\"id\": \"pgstac-test-item-0075\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-15T00:00:00Z\", \"eo:cloud_cover\": 8}}, {\"id\": \"pgstac-test-item-0049\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-16T00:00:00Z\", \"eo:cloud_cover\": 9}}, {\"id\": \"pgstac-test-item-0040\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-16T00:00:00Z\", \"eo:cloud_cover\": 10}}, {\"id\": \"pgstac-test-item-0036\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-16T00:00:00Z\", \"eo:cloud_cover\": 12}}, {\"id\": \"pgstac-test-item-0039\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-16T00:00:00Z\", \"eo:cloud_cover\": 12}}, {\"id\": \"pgstac-test-item-0052\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-16T00:00:00Z\", \"eo:cloud_cover\": 12}}, {\"id\": \"pgstac-test-item-0016\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-16T00:00:00Z\", \"eo:cloud_cover\": 13}}], \"numberMatched\": 100, \"numberReturned\": 10}\n\nSELECT search('{\"token\":\"prev:pgstac-test-item-0054\", \"fields\":{\"include\":[\"id\",\"properties.datetime\",\"properties.eo:cloud_cover\"]},\"sortby\":[{\"field\":\"properties.eo:cloud_cover\",\"direction\":\"asc\"},{\"field\":\"datetime\",\"direction\":\"desc\"},{\"field\":\"id\",\"direction\":\"asc\"}]}');\n {\"type\": \"FeatureCollection\", \"links\": [{\"rel\": \"root\", \"href\": \".\", \"type\": \"application/json\"}, {\"rel\": \"self\", \"href\": \"./search\", \"type\": \"application/json\"}, {\"rel\": \"next\", \"href\": \"./search?token=next:pgstac-test-collection:pgstac-test-item-0048\", \"type\": \"application/geo+json\", \"method\": \"GET\"}], \"features\": [{\"id\": \"pgstac-test-item-0097\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-07-31T00:00:00Z\", \"eo:cloud_cover\": 1}}, {\"id\": \"pgstac-test-item-0013\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-17T00:00:00Z\", \"eo:cloud_cover\": 2}}, {\"id\": \"pgstac-test-item-0063\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-16T00:00:00Z\", \"eo:cloud_cover\": 2}}, {\"id\": \"pgstac-test-item-0005\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-24T00:00:00Z\", \"eo:cloud_cover\": 3}}, {\"id\": \"pgstac-test-item-0034\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-16T00:00:00Z\", \"eo:cloud_cover\": 3}}, {\"id\": \"pgstac-test-item-0041\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-16T00:00:00Z\", \"eo:cloud_cover\": 3}}, {\"id\": \"pgstac-test-item-0073\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-15T00:00:00Z\", \"eo:cloud_cover\": 3}}, {\"id\": \"pgstac-test-item-0085\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-01T00:00:00Z\", \"eo:cloud_cover\": 3}}, {\"id\": \"pgstac-test-item-0012\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-17T00:00:00Z\", \"eo:cloud_cover\": 4}}, {\"id\": \"pgstac-test-item-0048\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-16T00:00:00Z\", \"eo:cloud_cover\": 4}}], \"numberMatched\": 100, \"numberReturned\": 10}\n\n-- Test paging without fields extension\nSELECT jsonb_path_query(search('{\"fields\":{\"include\":[\"id\",\"properties.datetime\",\"properties.eo:cloud_cover\"]},\"sortby\":[{\"field\":\"properties.eo:cloud_cover\",\"direction\":\"asc\"},{\"field\":\"datetime\",\"direction\":\"desc\"},{\"field\":\"id\",\"direction\":\"asc\"}]}'), '$.features[*].id');\n \"pgstac-test-item-0097\"\n \"pgstac-test-item-0013\"\n \"pgstac-test-item-0063\"\n \"pgstac-test-item-0005\"\n \"pgstac-test-item-0034\"\n \"pgstac-test-item-0041\"\n \"pgstac-test-item-0073\"\n \"pgstac-test-item-0085\"\n \"pgstac-test-item-0012\"\n \"pgstac-test-item-0048\"\n\nSELECT jsonb_path_query(search('{\"token\":\"next:pgstac-test-item-0048\",\"sortby\":[{\"field\":\"properties.eo:cloud_cover\",\"direction\":\"asc\"},{\"field\":\"datetime\",\"direction\":\"desc\"},{\"field\":\"id\",\"direction\":\"asc\"}]}'), '$.features[*].id');\n \"pgstac-test-item-0054\"\n \"pgstac-test-item-0032\"\n \"pgstac-test-item-0057\"\n \"pgstac-test-item-0075\"\n \"pgstac-test-item-0049\"\n \"pgstac-test-item-0040\"\n \"pgstac-test-item-0036\"\n \"pgstac-test-item-0039\"\n \"pgstac-test-item-0052\"\n \"pgstac-test-item-0016\"\n\nSELECT jsonb_path_query(search('{\"token\":\"next:pgstac-test-item-0016\",\"sortby\":[{\"field\":\"properties.eo:cloud_cover\",\"direction\":\"asc\"},{\"field\":\"datetime\",\"direction\":\"desc\"},{\"field\":\"id\",\"direction\":\"asc\"}]}'), '$.features[*].id');\n \"pgstac-test-item-0030\"\n \"pgstac-test-item-0031\"\n \"pgstac-test-item-0046\"\n \"pgstac-test-item-0014\"\n \"pgstac-test-item-0078\"\n \"pgstac-test-item-0051\"\n \"pgstac-test-item-0068\"\n \"pgstac-test-item-0004\"\n \"pgstac-test-item-0088\"\n \"pgstac-test-item-0053\"\n\nSELECT jsonb_path_query(search('{\"token\":\"prev:pgstac-test-item-0030\",\"sortby\":[{\"field\":\"properties.eo:cloud_cover\",\"direction\":\"asc\"},{\"field\":\"datetime\",\"direction\":\"desc\"},{\"field\":\"id\",\"direction\":\"asc\"}]}'), '$.features[*].id');\n \"pgstac-test-item-0054\"\n \"pgstac-test-item-0032\"\n \"pgstac-test-item-0057\"\n \"pgstac-test-item-0075\"\n \"pgstac-test-item-0049\"\n \"pgstac-test-item-0040\"\n \"pgstac-test-item-0036\"\n \"pgstac-test-item-0039\"\n \"pgstac-test-item-0052\"\n \"pgstac-test-item-0016\"\n\nSELECT jsonb_path_query(search('{\"token\":\"prev:pgstac-test-item-0054\",\"sortby\":[{\"field\":\"properties.eo:cloud_cover\",\"direction\":\"asc\"},{\"field\":\"datetime\",\"direction\":\"desc\"},{\"field\":\"id\",\"direction\":\"asc\"}]}'), '$.features[*].id');\n \"pgstac-test-item-0097\"\n \"pgstac-test-item-0013\"\n \"pgstac-test-item-0063\"\n \"pgstac-test-item-0005\"\n \"pgstac-test-item-0034\"\n \"pgstac-test-item-0041\"\n \"pgstac-test-item-0073\"\n \"pgstac-test-item-0085\"\n \"pgstac-test-item-0012\"\n \"pgstac-test-item-0048\"\n\nSELECT search('{\"collections\": [\"pgstac-test-collection\"], \"limit\": 1}');\n {\"type\": \"FeatureCollection\", \"links\": [{\"rel\": \"root\", \"href\": \".\", \"type\": \"application/json\"}, {\"rel\": \"self\", \"href\": \"./search\", \"type\": \"application/json\"}, {\"rel\": \"next\", \"href\": \"./search?token=next:pgstac-test-collection:pgstac-test-item-0003\", \"type\": \"application/geo+json\", \"method\": \"GET\"}], \"features\": [{\"id\": \"pgstac-test-item-0003\", \"bbox\": [-85.379245, 30.933949, -85.308201, 31.003555], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30085/m_3008506_nw_16_1_20110825.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30085/m_3008506_nw_16_1_20110825.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30085/m_3008506_nw_16_1_20110825.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-85.309412, 30.933949], [-85.308201, 31.002658], [-85.378084, 31.003555], [-85.379245, 30.934843], [-85.309412, 30.933949]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-25T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [654842, 3423507, 661516, 3431125], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7618, 6674], \"eo:cloud_cover\": 28, \"proj:transform\": [1, 0, 654842, 0, -1, 3431125, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}], \"numberMatched\": 100, \"numberReturned\": 1}\n\nSELECT search('{\"collections\": [\"pgstac-test-collection\"], \"limit\": 1, \"token\": \"next:pgstac-test-item-0001\"}');\n {\"type\": \"FeatureCollection\", \"links\": [{\"rel\": \"root\", \"href\": \".\", \"type\": \"application/json\"}, {\"rel\": \"self\", \"href\": \"./search\", \"type\": \"application/json\"}, {\"rel\": \"next\", \"href\": \"./search?token=next:pgstac-test-collection:pgstac-test-item-0006\", \"type\": \"application/geo+json\", \"method\": \"GET\"}, {\"rel\": \"prev\", \"href\": \"./search?token=prev:pgstac-test-collection:pgstac-test-item-0006\", \"type\": \"application/geo+json\", \"method\": \"GET\"}], \"features\": [{\"id\": \"pgstac-test-item-0006\", \"bbox\": [-87.253322, 30.934677, -87.184223, 31.00282], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008707_nw_16_1_20110824.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008707_nw_16_1_20110824.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008707_nw_16_1_20110824.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.184223, 30.934794], [-87.184354, 31.00282], [-87.253322, 31.002703], [-87.253142, 30.934677], [-87.184223, 30.934794]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-24T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [475817, 3422390, 482401, 3429929], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7539, 6584], \"eo:cloud_cover\": 100, \"proj:transform\": [1, 0, 475817, 0, -1, 3429929, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}], \"numberMatched\": 100, \"numberReturned\": 1}\n\n" }, { "path": "src/pgstac/tests/basic/cql_searches.sql", "content": "SET pgstac.\"default_filter_lang\" TO 'cql-json';\n\nSELECT search('{\"fields\":{\"include\":[\"id\",\"properties.datetime\",\"properties.eo:cloud_cover\"]},\"sortby\":[{\"field\":\"datetime\",\"direction\":\"desc\"},{\"field\":\"id\",\"direction\":\"asc\"}]}');\n\n-- Test Paging\nSELECT search('{\"fields\":{\"include\":[\"id\"]},\"sortby\":[{\"field\":\"datetime\",\"direction\":\"desc\"},{\"field\":\"id\",\"direction\":\"asc\"}]}');\n\nSELECT search('{\"token\":\"next:pgstac-test-item-0010\", \"fields\":{\"include\":[\"id\",\"properties.datetime\",\"properties.eo:cloud_cover\"]},\"sortby\":[{\"field\":\"datetime\",\"direction\":\"desc\"},{\"field\":\"id\",\"direction\":\"asc\"}]}');\n\nSELECT search('{\"token\":\"next:pgstac-test-item-0020\", \"fields\":{\"include\":[\"id\",\"properties.datetime\",\"properties.eo:cloud_cover\"]},\"sortby\":[{\"field\":\"datetime\",\"direction\":\"desc\"},{\"field\":\"id\",\"direction\":\"asc\"}]}');\n\nSELECT search('{\"token\":\"prev:pgstac-test-item-0021\", \"fields\":{\"include\":[\"id\",\"properties.datetime\",\"properties.eo:cloud_cover\"]},\"sortby\":[{\"field\":\"datetime\",\"direction\":\"desc\"},{\"field\":\"id\",\"direction\":\"asc\"}]}');\n\nSELECT search('{\"token\":\"next:pgstac-test-item-0011\", \"fields\":{\"include\":[\"id\",\"properties.datetime\",\"properties.eo:cloud_cover\"]},\"sortby\":[{\"field\":\"datetime\",\"direction\":\"desc\"},{\"field\":\"id\",\"direction\":\"asc\"}]}');\n--\n\nSELECT search('{\"datetime\":\"2011-08-16T00:00:00Z/2011-08-17T00:00:00Z\", \"fields\":{\"include\":[\"id\",\"properties.datetime\"]},\"sortby\":[{\"field\":\"datetime\",\"direction\":\"desc\"},{\"field\":\"id\",\"direction\":\"asc\"}]}');\n\nSELECT search('{\"datetime\":[\"2011-08-16T00:00:00Z\",\"2011-08-17T00:00:00Z\"], \"fields\":{\"include\":[\"id\",\"properties.datetime\"]},\"sortby\":[{\"field\":\"datetime\",\"direction\":\"desc\"},{\"field\":\"id\",\"direction\":\"asc\"}]}');\n\nSELECT search('{\"filter\":{\"anyinteracts\":[{\"property\":\"datetime\"},[\"2011-08-16T00:00:00Z\",\"2011-08-17T00:00:00Z\"]]}, \"fields\":{\"include\":[\"id\",\"properties.datetime\"]},\"sortby\":[{\"field\":\"datetime\",\"direction\":\"desc\"},{\"field\":\"id\",\"direction\":\"asc\"}]}');\n\nSELECT search('{\"filter\":{\"eq\":[{\"property\":\"eo:cloud_cover\"},36]}, \"fields\":{\"include\":[\"id\",\"properties.datetime\",\"properties.eo:cloud_cover\"]},\"sortby\":[{\"field\":\"datetime\",\"direction\":\"desc\"},{\"field\":\"id\",\"direction\":\"asc\"}]}');\n\nSELECT search('{\"filter\":{\"lt\":[{\"property\":\"eo:cloud_cover\"},25]}, \"fields\":{\"include\":[\"id\",\"properties.datetime\",\"properties.eo:cloud_cover\"]},\"sortby\":[{\"field\":\"eo:cloud_cover\",\"direction\":\"asc\"}]}');\n\nSELECT search('{\"ids\":[\"pgstac-test-item-0097\"],\"fields\":{\"include\":[\"id\"]}}');\n\nSELECT search('{\"ids\":[\"pgstac-test-item-0097\",\"pgstac-test-item-0003\"],\"fields\":{\"include\":[\"id\"]}}');\n\nSELECT search('{\"ids\":[\"bogusid\"],\"fields\":{\"include\":[\"id\"]}}');\n\nSELECT search('{\"collections\":[\"pgstac-test-collection\"],\"fields\":{\"include\":[\"id\"]}, \"limit\": 1}');\n\nSELECT search('{\"collections\":[\"something\"]}');\n\nSELECT search('{\"collections\":[\"something\"],\"fields\":{\"include\":[\"id\"]}}');\n\nSELECT usecount IS NOT NULL and usecount > 0 AND lastused IS NOT NULL AND lastused < clock_timestamp() FROM search_query(jsonb_build_object('collections',ARRAY[random()::text]));\n\nSELECT hash, search, _where, orderby, metadata from search_query('{\"collections\":[\"pgstac-test-collection\"]}'::jsonb, _metadata=>'{\"meta\":\"value\"}'::jsonb);\n\nSELECT hash, search, _where, orderby, metadata from search_query('{\"collections\":[\"pgstac-test-collection\"]}'::jsonb, _metadata=>'{\"meta\":\"value\"}'::jsonb);\n\nSELECT usecount IS NOT NULL and usecount > 0 AND lastused IS NOT NULL AND lastused < clock_timestamp() FROM search_query('{\"collections\":[\"pgstac-test-collection\"]}');\n" }, { "path": "src/pgstac/tests/basic/cql_searches.sql.out", "content": "SET pgstac.\"default_filter_lang\" TO 'cql-json';\nSET\nSELECT search('{\"fields\":{\"include\":[\"id\",\"properties.datetime\",\"properties.eo:cloud_cover\"]},\"sortby\":[{\"field\":\"datetime\",\"direction\":\"desc\"},{\"field\":\"id\",\"direction\":\"asc\"}]}');\n {\"type\": \"FeatureCollection\", \"links\": [{\"rel\": \"root\", \"href\": \".\", \"type\": \"application/json\"}, {\"rel\": \"self\", \"href\": \"./search\", \"type\": \"application/json\"}, {\"rel\": \"next\", \"href\": \"./search?token=next:pgstac-test-collection:pgstac-test-item-0010\", \"type\": \"application/geo+json\", \"method\": \"GET\"}], \"features\": [{\"id\": \"pgstac-test-item-0001\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-25T00:00:00Z\", \"eo:cloud_cover\": 89}}, {\"id\": \"pgstac-test-item-0002\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-25T00:00:00Z\", \"eo:cloud_cover\": 33}}, {\"id\": \"pgstac-test-item-0003\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-25T00:00:00Z\", \"eo:cloud_cover\": 28}}, {\"id\": \"pgstac-test-item-0004\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-24T00:00:00Z\", \"eo:cloud_cover\": 23}}, {\"id\": \"pgstac-test-item-0005\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-24T00:00:00Z\", \"eo:cloud_cover\": 3}}, {\"id\": \"pgstac-test-item-0006\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-24T00:00:00Z\", \"eo:cloud_cover\": 100}}, {\"id\": \"pgstac-test-item-0007\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-17T00:00:00Z\", \"eo:cloud_cover\": 59}}, {\"id\": \"pgstac-test-item-0008\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-17T00:00:00Z\", \"eo:cloud_cover\": 64}}, {\"id\": \"pgstac-test-item-0009\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-17T00:00:00Z\", \"eo:cloud_cover\": 61}}, {\"id\": \"pgstac-test-item-0010\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-17T00:00:00Z\", \"eo:cloud_cover\": 31}}], \"numberMatched\": 100, \"numberReturned\": 10}\n\n-- Test Paging\nSELECT search('{\"fields\":{\"include\":[\"id\"]},\"sortby\":[{\"field\":\"datetime\",\"direction\":\"desc\"},{\"field\":\"id\",\"direction\":\"asc\"}]}');\n {\"type\": \"FeatureCollection\", \"links\": [{\"rel\": \"root\", \"href\": \".\", \"type\": \"application/json\"}, {\"rel\": \"self\", \"href\": \"./search\", \"type\": \"application/json\"}, {\"rel\": \"next\", \"href\": \"./search?token=next:pgstac-test-collection:pgstac-test-item-0010\", \"type\": \"application/geo+json\", \"method\": \"GET\"}], \"features\": [{\"id\": \"pgstac-test-item-0001\", \"collection\": \"pgstac-test-collection\"}, {\"id\": \"pgstac-test-item-0002\", \"collection\": \"pgstac-test-collection\"}, {\"id\": \"pgstac-test-item-0003\", \"collection\": \"pgstac-test-collection\"}, {\"id\": \"pgstac-test-item-0004\", \"collection\": \"pgstac-test-collection\"}, {\"id\": \"pgstac-test-item-0005\", \"collection\": \"pgstac-test-collection\"}, {\"id\": \"pgstac-test-item-0006\", \"collection\": \"pgstac-test-collection\"}, {\"id\": \"pgstac-test-item-0007\", \"collection\": \"pgstac-test-collection\"}, {\"id\": \"pgstac-test-item-0008\", \"collection\": \"pgstac-test-collection\"}, {\"id\": \"pgstac-test-item-0009\", \"collection\": \"pgstac-test-collection\"}, {\"id\": \"pgstac-test-item-0010\", \"collection\": \"pgstac-test-collection\"}], \"numberMatched\": 100, \"numberReturned\": 10}\n\nSELECT search('{\"token\":\"next:pgstac-test-item-0010\", \"fields\":{\"include\":[\"id\",\"properties.datetime\",\"properties.eo:cloud_cover\"]},\"sortby\":[{\"field\":\"datetime\",\"direction\":\"desc\"},{\"field\":\"id\",\"direction\":\"asc\"}]}');\n {\"type\": \"FeatureCollection\", \"links\": [{\"rel\": \"root\", \"href\": \".\", \"type\": \"application/json\"}, {\"rel\": \"self\", \"href\": \"./search\", \"type\": \"application/json\"}, {\"rel\": \"next\", \"href\": \"./search?token=next:pgstac-test-collection:pgstac-test-item-0020\", \"type\": \"application/geo+json\", \"method\": \"GET\"}, {\"rel\": \"prev\", \"href\": \"./search?token=prev:pgstac-test-collection:pgstac-test-item-0011\", \"type\": \"application/geo+json\", \"method\": \"GET\"}], \"features\": [{\"id\": \"pgstac-test-item-0011\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-17T00:00:00Z\", \"eo:cloud_cover\": 41}}, {\"id\": \"pgstac-test-item-0012\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-17T00:00:00Z\", \"eo:cloud_cover\": 4}}, {\"id\": \"pgstac-test-item-0013\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-17T00:00:00Z\", \"eo:cloud_cover\": 2}}, {\"id\": \"pgstac-test-item-0014\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-16T00:00:00Z\", \"eo:cloud_cover\": 17}}, {\"id\": \"pgstac-test-item-0015\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-16T00:00:00Z\", \"eo:cloud_cover\": 54}}, {\"id\": \"pgstac-test-item-0016\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-16T00:00:00Z\", \"eo:cloud_cover\": 13}}, {\"id\": \"pgstac-test-item-0017\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-16T00:00:00Z\", \"eo:cloud_cover\": 59}}, {\"id\": \"pgstac-test-item-0018\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-16T00:00:00Z\", \"eo:cloud_cover\": 29}}, {\"id\": \"pgstac-test-item-0019\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-16T00:00:00Z\", \"eo:cloud_cover\": 52}}, {\"id\": \"pgstac-test-item-0020\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-16T00:00:00Z\", \"eo:cloud_cover\": 39}}], \"numberMatched\": 100, \"numberReturned\": 10}\n\nSELECT search('{\"token\":\"next:pgstac-test-item-0020\", \"fields\":{\"include\":[\"id\",\"properties.datetime\",\"properties.eo:cloud_cover\"]},\"sortby\":[{\"field\":\"datetime\",\"direction\":\"desc\"},{\"field\":\"id\",\"direction\":\"asc\"}]}');\n {\"type\": \"FeatureCollection\", \"links\": [{\"rel\": \"root\", \"href\": \".\", \"type\": \"application/json\"}, {\"rel\": \"self\", \"href\": \"./search\", \"type\": \"application/json\"}, {\"rel\": \"next\", \"href\": \"./search?token=next:pgstac-test-collection:pgstac-test-item-0030\", \"type\": \"application/geo+json\", \"method\": \"GET\"}, {\"rel\": \"prev\", \"href\": \"./search?token=prev:pgstac-test-collection:pgstac-test-item-0021\", \"type\": \"application/geo+json\", \"method\": \"GET\"}], \"features\": [{\"id\": \"pgstac-test-item-0021\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-16T00:00:00Z\", \"eo:cloud_cover\": 29}}, {\"id\": \"pgstac-test-item-0022\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-16T00:00:00Z\", \"eo:cloud_cover\": 84}}, {\"id\": \"pgstac-test-item-0023\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-16T00:00:00Z\", \"eo:cloud_cover\": 56}}, {\"id\": \"pgstac-test-item-0024\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-16T00:00:00Z\", \"eo:cloud_cover\": 58}}, {\"id\": \"pgstac-test-item-0025\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-16T00:00:00Z\", \"eo:cloud_cover\": 65}}, {\"id\": \"pgstac-test-item-0026\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-16T00:00:00Z\", \"eo:cloud_cover\": 52}}, {\"id\": \"pgstac-test-item-0027\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-16T00:00:00Z\", \"eo:cloud_cover\": 43}}, {\"id\": \"pgstac-test-item-0028\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-16T00:00:00Z\", \"eo:cloud_cover\": 46}}, {\"id\": \"pgstac-test-item-0029\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-16T00:00:00Z\", \"eo:cloud_cover\": 42}}, {\"id\": \"pgstac-test-item-0030\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-16T00:00:00Z\", \"eo:cloud_cover\": 16}}], \"numberMatched\": 100, \"numberReturned\": 10}\n\nSELECT search('{\"token\":\"prev:pgstac-test-item-0021\", \"fields\":{\"include\":[\"id\",\"properties.datetime\",\"properties.eo:cloud_cover\"]},\"sortby\":[{\"field\":\"datetime\",\"direction\":\"desc\"},{\"field\":\"id\",\"direction\":\"asc\"}]}');\n {\"type\": \"FeatureCollection\", \"links\": [{\"rel\": \"root\", \"href\": \".\", \"type\": \"application/json\"}, {\"rel\": \"self\", \"href\": \"./search\", \"type\": \"application/json\"}, {\"rel\": \"next\", \"href\": \"./search?token=next:pgstac-test-collection:pgstac-test-item-0020\", \"type\": \"application/geo+json\", \"method\": \"GET\"}, {\"rel\": \"prev\", \"href\": \"./search?token=prev:pgstac-test-collection:pgstac-test-item-0011\", \"type\": \"application/geo+json\", \"method\": \"GET\"}], \"features\": [{\"id\": \"pgstac-test-item-0011\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-17T00:00:00Z\", \"eo:cloud_cover\": 41}}, {\"id\": \"pgstac-test-item-0012\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-17T00:00:00Z\", \"eo:cloud_cover\": 4}}, {\"id\": \"pgstac-test-item-0013\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-17T00:00:00Z\", \"eo:cloud_cover\": 2}}, {\"id\": \"pgstac-test-item-0014\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-16T00:00:00Z\", \"eo:cloud_cover\": 17}}, {\"id\": \"pgstac-test-item-0015\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-16T00:00:00Z\", \"eo:cloud_cover\": 54}}, {\"id\": \"pgstac-test-item-0016\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-16T00:00:00Z\", \"eo:cloud_cover\": 13}}, {\"id\": \"pgstac-test-item-0017\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-16T00:00:00Z\", \"eo:cloud_cover\": 59}}, {\"id\": \"pgstac-test-item-0018\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-16T00:00:00Z\", \"eo:cloud_cover\": 29}}, {\"id\": \"pgstac-test-item-0019\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-16T00:00:00Z\", \"eo:cloud_cover\": 52}}, {\"id\": \"pgstac-test-item-0020\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-16T00:00:00Z\", \"eo:cloud_cover\": 39}}], \"numberMatched\": 100, \"numberReturned\": 10}\n\nSELECT search('{\"token\":\"next:pgstac-test-item-0011\", \"fields\":{\"include\":[\"id\",\"properties.datetime\",\"properties.eo:cloud_cover\"]},\"sortby\":[{\"field\":\"datetime\",\"direction\":\"desc\"},{\"field\":\"id\",\"direction\":\"asc\"}]}');\n {\"type\": \"FeatureCollection\", \"links\": [{\"rel\": \"root\", \"href\": \".\", \"type\": \"application/json\"}, {\"rel\": \"self\", \"href\": \"./search\", \"type\": \"application/json\"}, {\"rel\": \"next\", \"href\": \"./search?token=next:pgstac-test-collection:pgstac-test-item-0021\", \"type\": \"application/geo+json\", \"method\": \"GET\"}, {\"rel\": \"prev\", \"href\": \"./search?token=prev:pgstac-test-collection:pgstac-test-item-0012\", \"type\": \"application/geo+json\", \"method\": \"GET\"}], \"features\": [{\"id\": \"pgstac-test-item-0012\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-17T00:00:00Z\", \"eo:cloud_cover\": 4}}, {\"id\": \"pgstac-test-item-0013\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-17T00:00:00Z\", \"eo:cloud_cover\": 2}}, {\"id\": \"pgstac-test-item-0014\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-16T00:00:00Z\", \"eo:cloud_cover\": 17}}, {\"id\": \"pgstac-test-item-0015\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-16T00:00:00Z\", \"eo:cloud_cover\": 54}}, {\"id\": \"pgstac-test-item-0016\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-16T00:00:00Z\", \"eo:cloud_cover\": 13}}, {\"id\": \"pgstac-test-item-0017\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-16T00:00:00Z\", \"eo:cloud_cover\": 59}}, {\"id\": \"pgstac-test-item-0018\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-16T00:00:00Z\", \"eo:cloud_cover\": 29}}, {\"id\": \"pgstac-test-item-0019\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-16T00:00:00Z\", \"eo:cloud_cover\": 52}}, {\"id\": \"pgstac-test-item-0020\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-16T00:00:00Z\", \"eo:cloud_cover\": 39}}, {\"id\": \"pgstac-test-item-0021\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-16T00:00:00Z\", \"eo:cloud_cover\": 29}}], \"numberMatched\": 100, \"numberReturned\": 10}\n\n--\nSELECT search('{\"datetime\":\"2011-08-16T00:00:00Z/2011-08-17T00:00:00Z\", \"fields\":{\"include\":[\"id\",\"properties.datetime\"]},\"sortby\":[{\"field\":\"datetime\",\"direction\":\"desc\"},{\"field\":\"id\",\"direction\":\"asc\"}]}');\n {\"type\": \"FeatureCollection\", \"links\": [{\"rel\": \"root\", \"href\": \".\", \"type\": \"application/json\"}, {\"rel\": \"self\", \"href\": \"./search\", \"type\": \"application/json\"}, {\"rel\": \"next\", \"href\": \"./search?token=next:pgstac-test-collection:pgstac-test-item-0016\", \"type\": \"application/geo+json\", \"method\": \"GET\"}], \"features\": [{\"id\": \"pgstac-test-item-0007\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-17T00:00:00Z\"}}, {\"id\": \"pgstac-test-item-0008\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-17T00:00:00Z\"}}, {\"id\": \"pgstac-test-item-0009\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-17T00:00:00Z\"}}, {\"id\": \"pgstac-test-item-0010\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-17T00:00:00Z\"}}, {\"id\": \"pgstac-test-item-0011\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-17T00:00:00Z\"}}, {\"id\": \"pgstac-test-item-0012\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-17T00:00:00Z\"}}, {\"id\": \"pgstac-test-item-0013\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-17T00:00:00Z\"}}, {\"id\": \"pgstac-test-item-0014\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-16T00:00:00Z\"}}, {\"id\": \"pgstac-test-item-0015\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-16T00:00:00Z\"}}, {\"id\": \"pgstac-test-item-0016\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-16T00:00:00Z\"}}], \"numberMatched\": 57, \"numberReturned\": 10}\n\nSELECT search('{\"datetime\":[\"2011-08-16T00:00:00Z\",\"2011-08-17T00:00:00Z\"], \"fields\":{\"include\":[\"id\",\"properties.datetime\"]},\"sortby\":[{\"field\":\"datetime\",\"direction\":\"desc\"},{\"field\":\"id\",\"direction\":\"asc\"}]}');\n {\"type\": \"FeatureCollection\", \"links\": [{\"rel\": \"root\", \"href\": \".\", \"type\": \"application/json\"}, {\"rel\": \"self\", \"href\": \"./search\", \"type\": \"application/json\"}, {\"rel\": \"next\", \"href\": \"./search?token=next:pgstac-test-collection:pgstac-test-item-0016\", \"type\": \"application/geo+json\", \"method\": \"GET\"}], \"features\": [{\"id\": \"pgstac-test-item-0007\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-17T00:00:00Z\"}}, {\"id\": \"pgstac-test-item-0008\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-17T00:00:00Z\"}}, {\"id\": \"pgstac-test-item-0009\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-17T00:00:00Z\"}}, {\"id\": \"pgstac-test-item-0010\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-17T00:00:00Z\"}}, {\"id\": \"pgstac-test-item-0011\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-17T00:00:00Z\"}}, {\"id\": \"pgstac-test-item-0012\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-17T00:00:00Z\"}}, {\"id\": \"pgstac-test-item-0013\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-17T00:00:00Z\"}}, {\"id\": \"pgstac-test-item-0014\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-16T00:00:00Z\"}}, {\"id\": \"pgstac-test-item-0015\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-16T00:00:00Z\"}}, {\"id\": \"pgstac-test-item-0016\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-16T00:00:00Z\"}}], \"numberMatched\": 57, \"numberReturned\": 10}\n\nSELECT search('{\"filter\":{\"anyinteracts\":[{\"property\":\"datetime\"},[\"2011-08-16T00:00:00Z\",\"2011-08-17T00:00:00Z\"]]}, \"fields\":{\"include\":[\"id\",\"properties.datetime\"]},\"sortby\":[{\"field\":\"datetime\",\"direction\":\"desc\"},{\"field\":\"id\",\"direction\":\"asc\"}]}');\n {\"type\": \"FeatureCollection\", \"links\": [{\"rel\": \"root\", \"href\": \".\", \"type\": \"application/json\"}, {\"rel\": \"self\", \"href\": \"./search\", \"type\": \"application/json\"}, {\"rel\": \"next\", \"href\": \"./search?token=next:pgstac-test-collection:pgstac-test-item-0016\", \"type\": \"application/geo+json\", \"method\": \"GET\"}], \"features\": [{\"id\": \"pgstac-test-item-0007\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-17T00:00:00Z\"}}, {\"id\": \"pgstac-test-item-0008\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-17T00:00:00Z\"}}, {\"id\": \"pgstac-test-item-0009\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-17T00:00:00Z\"}}, {\"id\": \"pgstac-test-item-0010\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-17T00:00:00Z\"}}, {\"id\": \"pgstac-test-item-0011\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-17T00:00:00Z\"}}, {\"id\": \"pgstac-test-item-0012\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-17T00:00:00Z\"}}, {\"id\": \"pgstac-test-item-0013\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-17T00:00:00Z\"}}, {\"id\": \"pgstac-test-item-0014\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-16T00:00:00Z\"}}, {\"id\": \"pgstac-test-item-0015\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-16T00:00:00Z\"}}, {\"id\": \"pgstac-test-item-0016\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-16T00:00:00Z\"}}], \"numberMatched\": 57, \"numberReturned\": 10}\n\nSELECT search('{\"filter\":{\"eq\":[{\"property\":\"eo:cloud_cover\"},36]}, \"fields\":{\"include\":[\"id\",\"properties.datetime\",\"properties.eo:cloud_cover\"]},\"sortby\":[{\"field\":\"datetime\",\"direction\":\"desc\"},{\"field\":\"id\",\"direction\":\"asc\"}]}');\n {\"type\": \"FeatureCollection\", \"links\": [{\"rel\": \"root\", \"href\": \".\", \"type\": \"application/json\"}, {\"rel\": \"self\", \"href\": \"./search\", \"type\": \"application/json\"}], \"features\": [{\"id\": \"pgstac-test-item-0087\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-01T00:00:00Z\", \"eo:cloud_cover\": 36}}, {\"id\": \"pgstac-test-item-0089\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-07-31T00:00:00Z\", \"eo:cloud_cover\": 36}}], \"numberMatched\": 2, \"numberReturned\": 2}\n\nSELECT search('{\"filter\":{\"lt\":[{\"property\":\"eo:cloud_cover\"},25]}, \"fields\":{\"include\":[\"id\",\"properties.datetime\",\"properties.eo:cloud_cover\"]},\"sortby\":[{\"field\":\"eo:cloud_cover\",\"direction\":\"asc\"}]}');\n {\"type\": \"FeatureCollection\", \"links\": [{\"rel\": \"root\", \"href\": \".\", \"type\": \"application/json\"}, {\"rel\": \"self\", \"href\": \"./search\", \"type\": \"application/json\"}, {\"rel\": \"next\", \"href\": \"./search?token=next:pgstac-test-collection:pgstac-test-item-0012\", \"type\": \"application/geo+json\", \"method\": \"GET\"}], \"features\": [{\"id\": \"pgstac-test-item-0097\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-07-31T00:00:00Z\", \"eo:cloud_cover\": 1}}, {\"id\": \"pgstac-test-item-0063\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-16T00:00:00Z\", \"eo:cloud_cover\": 2}}, {\"id\": \"pgstac-test-item-0013\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-17T00:00:00Z\", \"eo:cloud_cover\": 2}}, {\"id\": \"pgstac-test-item-0085\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-01T00:00:00Z\", \"eo:cloud_cover\": 3}}, {\"id\": \"pgstac-test-item-0073\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-15T00:00:00Z\", \"eo:cloud_cover\": 3}}, {\"id\": \"pgstac-test-item-0041\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-16T00:00:00Z\", \"eo:cloud_cover\": 3}}, {\"id\": \"pgstac-test-item-0034\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-16T00:00:00Z\", \"eo:cloud_cover\": 3}}, {\"id\": \"pgstac-test-item-0005\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-24T00:00:00Z\", \"eo:cloud_cover\": 3}}, {\"id\": \"pgstac-test-item-0048\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-16T00:00:00Z\", \"eo:cloud_cover\": 4}}, {\"id\": \"pgstac-test-item-0012\", \"collection\": \"pgstac-test-collection\", \"properties\": {\"datetime\": \"2011-08-17T00:00:00Z\", \"eo:cloud_cover\": 4}}], \"numberMatched\": 31, \"numberReturned\": 10}\n\nSELECT search('{\"ids\":[\"pgstac-test-item-0097\"],\"fields\":{\"include\":[\"id\"]}}');\n {\"type\": \"FeatureCollection\", \"links\": [{\"rel\": \"root\", \"href\": \".\", \"type\": \"application/json\"}, {\"rel\": \"self\", \"href\": \"./search\", \"type\": \"application/json\"}], \"features\": [{\"id\": \"pgstac-test-item-0097\", \"collection\": \"pgstac-test-collection\"}], \"numberMatched\": 1, \"numberReturned\": 1}\n\nSELECT search('{\"ids\":[\"pgstac-test-item-0097\",\"pgstac-test-item-0003\"],\"fields\":{\"include\":[\"id\"]}}');\n {\"type\": \"FeatureCollection\", \"links\": [{\"rel\": \"root\", \"href\": \".\", \"type\": \"application/json\"}, {\"rel\": \"self\", \"href\": \"./search\", \"type\": \"application/json\"}], \"features\": [{\"id\": \"pgstac-test-item-0003\", \"collection\": \"pgstac-test-collection\"}, {\"id\": \"pgstac-test-item-0097\", \"collection\": \"pgstac-test-collection\"}], \"numberMatched\": 2, \"numberReturned\": 2}\n\nSELECT search('{\"ids\":[\"bogusid\"],\"fields\":{\"include\":[\"id\"]}}');\n {\"type\": \"FeatureCollection\", \"links\": [{\"rel\": \"root\", \"href\": \".\", \"type\": \"application/json\"}, {\"rel\": \"self\", \"href\": \"./search\", \"type\": \"application/json\"}], \"features\": [], \"numberMatched\": 0, \"numberReturned\": 0}\n\nSELECT search('{\"collections\":[\"pgstac-test-collection\"],\"fields\":{\"include\":[\"id\"]}, \"limit\": 1}');\n {\"type\": \"FeatureCollection\", \"links\": [{\"rel\": \"root\", \"href\": \".\", \"type\": \"application/json\"}, {\"rel\": \"self\", \"href\": \"./search\", \"type\": \"application/json\"}, {\"rel\": \"next\", \"href\": \"./search?token=next:pgstac-test-collection:pgstac-test-item-0003\", \"type\": \"application/geo+json\", \"method\": \"GET\"}], \"features\": [{\"id\": \"pgstac-test-item-0003\", \"collection\": \"pgstac-test-collection\"}], \"numberMatched\": 100, \"numberReturned\": 1}\n\nSELECT search('{\"collections\":[\"something\"]}');\n {\"type\": \"FeatureCollection\", \"links\": [{\"rel\": \"root\", \"href\": \".\", \"type\": \"application/json\"}, {\"rel\": \"self\", \"href\": \"./search\", \"type\": \"application/json\"}], \"features\": [], \"numberMatched\": 0, \"numberReturned\": 0}\n\nSELECT search('{\"collections\":[\"something\"],\"fields\":{\"include\":[\"id\"]}}');\n {\"type\": \"FeatureCollection\", \"links\": [{\"rel\": \"root\", \"href\": \".\", \"type\": \"application/json\"}, {\"rel\": \"self\", \"href\": \"./search\", \"type\": \"application/json\"}], \"features\": [], \"numberMatched\": 0, \"numberReturned\": 0}\n\nSELECT usecount IS NOT NULL and usecount > 0 AND lastused IS NOT NULL AND lastused < clock_timestamp() FROM search_query(jsonb_build_object('collections',ARRAY[random()::text]));\n t\n\nSELECT hash, search, _where, orderby, metadata from search_query('{\"collections\":[\"pgstac-test-collection\"]}'::jsonb, _metadata=>'{\"meta\":\"value\"}'::jsonb);\n 06efe6c09f0d61fd212e882325041a73 | {\"collections\": [\"pgstac-test-collection\"]} | collection = ANY ('{pgstac-test-collection}') | datetime DESC, id DESC | {\"meta\": \"value\"}\n\nSELECT hash, search, _where, orderby, metadata from search_query('{\"collections\":[\"pgstac-test-collection\"]}'::jsonb, _metadata=>'{\"meta\":\"value\"}'::jsonb);\n 06efe6c09f0d61fd212e882325041a73 | {\"collections\": [\"pgstac-test-collection\"]} | collection = ANY ('{pgstac-test-collection}') | datetime DESC, id DESC | {\"meta\": \"value\"}\n\nSELECT usecount IS NOT NULL and usecount > 0 AND lastused IS NOT NULL AND lastused < clock_timestamp() FROM search_query('{\"collections\":[\"pgstac-test-collection\"]}');\n t\n\n" }, { "path": "src/pgstac/tests/basic/crud_functions.sql", "content": "-- run tests as pgstac_ingest\nSET ROLE pgstac_ingest;\nSET pgstac.use_queue=FALSE;\nSELECT get_setting_bool('use_queue');\nSET pgstac.update_collection_extent=TRUE;\nSELECT get_setting_bool('update_collection_extent');\n--create base data to use with tests\nCREATE TEMP TABLE test_items AS\nSELECT jsonb_build_object(\n 'id', concat('pgstactest-crudtest-', (row_number() over ())::text),\n 'collection', 'pgstactest-crudtest',\n 'geometry', '{\"type\": \"Polygon\", \"coordinates\": [[[-85.309412, 30.933949], [-85.308201, 31.002658], [-85.378084, 31.003555], [-85.379245, 30.934843], [-85.309412, 30.933949]]]}'::json,\n 'properties', jsonb_build_object( 'datetime', g::text)\n) as content FROM generate_series('2020-01-01'::timestamptz, '2022-01-01'::timestamptz, '1 month'::interval) g;\n\n--test collection partioned by year\nINSERT INTO collections (content, partition_trunc) VALUES ('{\"id\":\"pgstactest-crudtest\"}', 'year');\n\n-- Create an item\nSELECT create_item((SELECT content FROM test_items LIMIT 1));\nSELECT * FROM items WHERE collection='pgstactest-crudtest';\n\n-- Check to see if extent got updated\nSELECT content->'extent' FROM collections WHERE id='pgstactest-crudtest';\n\n-- Update item with new datetime that is in a different partition\nSELECT update_item((SELECT content || '{\"properties\":{\"datetime\":\"2023-01-01 00:00:00Z\"}}'::jsonb FROM test_items LIMIT 1));\nSELECT * FROM items WHERE collection='pgstactest-crudtest';\n\n-- Check to see if extent got updated\nSELECT content->'extent' FROM collections WHERE id='pgstactest-crudtest';\n\n-- Update item with new datetime that is in a different partition\nSELECT upsert_item((SELECT content || '{\"properties\":{\"datetime\":\"2023-02-01 00:00:00Z\"}}'::jsonb FROM test_items LIMIT 1));\nSELECT * FROM items WHERE collection='pgstactest-crudtest';\n\n-- Delete an item\nSELECT delete_item('pgstactest-crudtest-1', 'pgstactest-crudtest');\nSELECT * FROM items WHERE collection='pgstactest-crudtest';\n\nWITH c AS (SELECT content FROM test_items LIMIT 2),\naggregated AS (SELECT jsonb_agg(content) as items FROM c)\nSELECT create_items(items) FROM aggregated;\nSELECT * FROM items WHERE collection='pgstactest-crudtest';\n\nDELETE FROM items WHERE collection='pgstactest-crudtest';\n\n-- upsert items that do not exist yet\nWITH c AS (SELECT content FROM test_items LIMIT 2),\naggregated AS (SELECT jsonb_agg(content) as items FROM c)\nSELECT upsert_items(items) FROM aggregated;\nSELECT * FROM items WHERE collection='pgstactest-crudtest';\n\n-- upsert items that already exist and are to be modified\nWITH c AS (SELECT content || '{\"properties\":{\"datetime\":\"2023-02-01 00:00:00Z\"}}'::jsonb as content FROM test_items LIMIT 2),\naggregated AS (SELECT jsonb_agg(content) as items FROM c)\nSELECT upsert_items(items) FROM aggregated;\nSELECT * FROM items WHERE collection='pgstactest-crudtest';\n\n-- turn off update_collection_extent then add an item and verify that the extent did not get updated automatically\nSET pgstac.update_collection_extent=FALSE;\nSELECT get_setting_bool('update_collection_extent');\nSELECT update_item((SELECT content || '{\"properties\":{\"datetime\":\"2024-01-01 00:00:00Z\"}}'::jsonb FROM test_items LIMIT 1));\n\nSELECT content->'extent' FROM collections WHERE id='pgstactest-crudtest';\n\n-- check formatting of temporal extent\nSELECT collection_temporal_extent('pgstactest-crudtest');\n\n" }, { "path": "src/pgstac/tests/basic/crud_functions.sql.out", "content": "-- run tests as pgstac_ingest\nSET ROLE pgstac_ingest;\nSET\nSET pgstac.use_queue=FALSE;\nSET\nSELECT get_setting_bool('use_queue');\n f\n\nSET pgstac.update_collection_extent=TRUE;\nSET\nSELECT get_setting_bool('update_collection_extent');\n t\n\n--create base data to use with tests\nCREATE TEMP TABLE test_items AS\nSELECT jsonb_build_object(\n 'id', concat('pgstactest-crudtest-', (row_number() over ())::text),\n 'collection', 'pgstactest-crudtest',\n 'geometry', '{\"type\": \"Polygon\", \"coordinates\": [[[-85.309412, 30.933949], [-85.308201, 31.002658], [-85.378084, 31.003555], [-85.379245, 30.934843], [-85.309412, 30.933949]]]}'::json,\n 'properties', jsonb_build_object( 'datetime', g::text)\n) as content FROM generate_series('2020-01-01'::timestamptz, '2022-01-01'::timestamptz, '1 month'::interval) g;\nSELECT 25\n--test collection partioned by year\nINSERT INTO collections (content, partition_trunc) VALUES ('{\"id\":\"pgstactest-crudtest\"}', 'year');\nINSERT 0 1\n-- Create an item\nSELECT create_item((SELECT content FROM test_items LIMIT 1));\n\n\nSELECT * FROM items WHERE collection='pgstactest-crudtest';\n pgstactest-crudtest-1 | 0103000020E610000001000000050000005B3FFD67CD5355C0C4211B4817EF3E400CE6AF90B95355C0A112D731AE003F4004C93B87325855C0BEBC00FBE8003F40FA0AD28C455855C000E5EFDE51EF3E405B3FFD67CD5355C0C4211B4817EF3E40 | pgstactest-crudtest | 2020-01-01 00:00:00+00 | 2020-01-01 00:00:00+00 | {\"properties\": {\"datetime\": \"2020-01-01 00:00:00+00\"}} |\n\n-- Check to see if extent got updated\nSELECT content->'extent' FROM collections WHERE id='pgstactest-crudtest';\n{\"spatial\": {\"bbox\": [[-85.3792495727539, 30.933948516845703, -85.30819702148438, 31.003555297851562]]}, \"temporal\": {\"interval\": [[\"2020-01-01T00:00:00+00:00\", \"2020-01-01T00:00:00+00:00\"]]}}\n\n\n-- Update item with new datetime that is in a different partition\nSELECT update_item((SELECT content || '{\"properties\":{\"datetime\":\"2023-01-01 00:00:00Z\"}}'::jsonb FROM test_items LIMIT 1));\n\nSELECT * FROM items WHERE collection='pgstactest-crudtest';\n pgstactest-crudtest-1 | 0103000020E610000001000000050000005B3FFD67CD5355C0C4211B4817EF3E400CE6AF90B95355C0A112D731AE003F4004C93B87325855C0BEBC00FBE8003F40FA0AD28C455855C000E5EFDE51EF3E405B3FFD67CD5355C0C4211B4817EF3E40 | pgstactest-crudtest | 2023-01-01 00:00:00+00 | 2023-01-01 00:00:00+00 | {\"properties\": {\"datetime\": \"2023-01-01 00:00:00Z\"}} |\n\n-- Check to see if extent got updated\nSELECT content->'extent' FROM collections WHERE id='pgstactest-crudtest';\n{\"spatial\": {\"bbox\": [[-85.3792495727539, 30.933948516845703, -85.30819702148438, 31.003555297851562]]}, \"temporal\": {\"interval\": [[\"2023-01-01T00:00:00+00:00\", \"2023-01-01T00:00:00+00:00\"]]}}\n\n-- Update item with new datetime that is in a different partition\nSELECT upsert_item((SELECT content || '{\"properties\":{\"datetime\":\"2023-02-01 00:00:00Z\"}}'::jsonb FROM test_items LIMIT 1));\n\n\nSELECT * FROM items WHERE collection='pgstactest-crudtest';\n pgstactest-crudtest-1 | 0103000020E610000001000000050000005B3FFD67CD5355C0C4211B4817EF3E400CE6AF90B95355C0A112D731AE003F4004C93B87325855C0BEBC00FBE8003F40FA0AD28C455855C000E5EFDE51EF3E405B3FFD67CD5355C0C4211B4817EF3E40 | pgstactest-crudtest | 2023-02-01 00:00:00+00 | 2023-02-01 00:00:00+00 | {\"properties\": {\"datetime\": \"2023-02-01 00:00:00Z\"}} |\n\n-- Delete an item\nSELECT delete_item('pgstactest-crudtest-1', 'pgstactest-crudtest');\n\n\nSELECT * FROM items WHERE collection='pgstactest-crudtest';\n\nWITH c AS (SELECT content FROM test_items LIMIT 2),\naggregated AS (SELECT jsonb_agg(content) as items FROM c)\nSELECT create_items(items) FROM aggregated;\n\n\nSELECT * FROM items WHERE collection='pgstactest-crudtest';\n pgstactest-crudtest-1 | 0103000020E610000001000000050000005B3FFD67CD5355C0C4211B4817EF3E400CE6AF90B95355C0A112D731AE003F4004C93B87325855C0BEBC00FBE8003F40FA0AD28C455855C000E5EFDE51EF3E405B3FFD67CD5355C0C4211B4817EF3E40 | pgstactest-crudtest | 2020-01-01 00:00:00+00 | 2020-01-01 00:00:00+00 | {\"properties\": {\"datetime\": \"2020-01-01 00:00:00+00\"}} |\n pgstactest-crudtest-2 | 0103000020E610000001000000050000005B3FFD67CD5355C0C4211B4817EF3E400CE6AF90B95355C0A112D731AE003F4004C93B87325855C0BEBC00FBE8003F40FA0AD28C455855C000E5EFDE51EF3E405B3FFD67CD5355C0C4211B4817EF3E40 | pgstactest-crudtest | 2020-02-01 00:00:00+00 | 2020-02-01 00:00:00+00 | {\"properties\": {\"datetime\": \"2020-02-01 00:00:00+00\"}} |\n\nDELETE FROM items WHERE collection='pgstactest-crudtest';\nDELETE 2\n-- upsert items that do not exist yet\nWITH c AS (SELECT content FROM test_items LIMIT 2),\naggregated AS (SELECT jsonb_agg(content) as items FROM c)\nSELECT upsert_items(items) FROM aggregated;\n\n\nSELECT * FROM items WHERE collection='pgstactest-crudtest';\n pgstactest-crudtest-1 | 0103000020E610000001000000050000005B3FFD67CD5355C0C4211B4817EF3E400CE6AF90B95355C0A112D731AE003F4004C93B87325855C0BEBC00FBE8003F40FA0AD28C455855C000E5EFDE51EF3E405B3FFD67CD5355C0C4211B4817EF3E40 | pgstactest-crudtest | 2020-01-01 00:00:00+00 | 2020-01-01 00:00:00+00 | {\"properties\": {\"datetime\": \"2020-01-01 00:00:00+00\"}} |\n pgstactest-crudtest-2 | 0103000020E610000001000000050000005B3FFD67CD5355C0C4211B4817EF3E400CE6AF90B95355C0A112D731AE003F4004C93B87325855C0BEBC00FBE8003F40FA0AD28C455855C000E5EFDE51EF3E405B3FFD67CD5355C0C4211B4817EF3E40 | pgstactest-crudtest | 2020-02-01 00:00:00+00 | 2020-02-01 00:00:00+00 | {\"properties\": {\"datetime\": \"2020-02-01 00:00:00+00\"}} |\n\n-- upsert items that already exist and are to be modified\nWITH c AS (SELECT content || '{\"properties\":{\"datetime\":\"2023-02-01 00:00:00Z\"}}'::jsonb as content FROM test_items LIMIT 2),\naggregated AS (SELECT jsonb_agg(content) as items FROM c)\nSELECT upsert_items(items) FROM aggregated;\n\n\nSELECT * FROM items WHERE collection='pgstactest-crudtest';\n pgstactest-crudtest-1 | 0103000020E610000001000000050000005B3FFD67CD5355C0C4211B4817EF3E400CE6AF90B95355C0A112D731AE003F4004C93B87325855C0BEBC00FBE8003F40FA0AD28C455855C000E5EFDE51EF3E405B3FFD67CD5355C0C4211B4817EF3E40 | pgstactest-crudtest | 2023-02-01 00:00:00+00 | 2023-02-01 00:00:00+00 | {\"properties\": {\"datetime\": \"2023-02-01 00:00:00Z\"}} |\n pgstactest-crudtest-2 | 0103000020E610000001000000050000005B3FFD67CD5355C0C4211B4817EF3E400CE6AF90B95355C0A112D731AE003F4004C93B87325855C0BEBC00FBE8003F40FA0AD28C455855C000E5EFDE51EF3E405B3FFD67CD5355C0C4211B4817EF3E40 | pgstactest-crudtest | 2023-02-01 00:00:00+00 | 2023-02-01 00:00:00+00 | {\"properties\": {\"datetime\": \"2023-02-01 00:00:00Z\"}} |\n\n-- turn off update_collection_extent then add an item and verify that the extent did not get updated automatically\nSET pgstac.update_collection_extent=FALSE;\nSET\nSELECT get_setting_bool('update_collection_extent');\n f\n\nSELECT update_item((SELECT content || '{\"properties\":{\"datetime\":\"2024-01-01 00:00:00Z\"}}'::jsonb FROM test_items LIMIT 1));\n\nSELECT content->'extent' FROM collections WHERE id='pgstactest-crudtest';\n {\"spatial\": {\"bbox\": [[-85.3792495727539, 30.933948516845703, -85.30819702148438, 31.003555297851562]]}, \"temporal\": {\"interval\": [[\"2023-02-01T00:00:00+00:00\", \"2023-02-01T00:00:00+00:00\"]]}}\n\n-- check formatting of temporal extent\nSELECT collection_temporal_extent('pgstactest-crudtest');\n [[\"2023-02-01T00:00:00+00:00\", \"2024-01-01T00:00:00+00:00\"]]\n\n" }, { "path": "src/pgstac/tests/basic/free_text.sql", "content": "\nSET pgstac.context TO 'on';\nSET pgstac.\"default_filter_lang\" TO 'cql2-json';\n\nCREATE TEMP TABLE temp_collections (\n id SERIAL PRIMARY KEY,\n title TEXT,\n description TEXT,\n keywords TEXT,\n minx NUMERIC,\n miny NUMERIC,\n maxx NUMERIC,\n maxy NUMERIC,\n sdt TIMESTAMPTZ,\n edt TIMESTAMPTZ\n);\n\nINSERT INTO temp_collections (\n title, description, keywords, minx, miny, maxx, maxy, sdt, edt\n) VALUES\n -- no keywords\n (\n 'Stranger Things',\n 'Some teenagers drop out of school to fight scary monsters',\n null,\n -180, -90, 180, 90,\n '2016-01-01T00:00:00Z',\n '2025-12-31T23:59:59Z'\n ),\n (\n 'The Bear',\n 'Another story about why you should not start a restaurant',\n 'restaurant, funny, sad, great',\n -180, -90, 180, 90,\n '2022-01-01T00:00:00Z',\n '2025-12-31T23:59:59Z'\n ),\n (\n 'Godzilla',\n 'A large lizard takes its revenge',\n 'scary, lizard, monster',\n -180, -90, 180, 90,\n '1954-01-01T00:00:00Z',\n null\n ),\n (\n 'Chefs Table',\n 'Another great story that make you wonder if you should go to a restaurant',\n 'restaurant, food, michelin',\n -180, -90, 180, 90,\n '2019-01-01T00:00:00Z',\n '2025-12-31T23:59:59Z'\n ),\n -- no title\n (\n null,\n 'A humoristic portrayal of office life',\n 'Scranton, paper',\n -180, -90, 180, 90,\n '2005-01-01T00:00:00Z',\n '2013-12-31T23:59:59Z'\n );\n\nSELECT\n create_collection(jsonb_build_object(\n 'id', format('testcollection_%s', id),\n 'type', 'Collection',\n 'title', title,\n 'description', description,\n 'extent', jsonb_build_object(\n 'spatial', jsonb_build_array(jsonb_build_array(minx, miny, maxx, maxy)),\n 'temporal', jsonb_build_array(jsonb_build_array(sdt, edt))\n ),\n 'stac_extensions', jsonb_build_array(),\n 'keywords', string_to_array(keywords, ', ')\n ))\nFROM temp_collections;\n\nselect collection_search('{\"q\": \"monsters\"}');\n\nselect collection_search('{\"q\": \"lizard\"}');\n\nselect collection_search('{\"q\": \"scary OR funny\"}');\n\nselect collection_search('{\"q\": \"(scary AND revenge) OR (funny AND sad)\"}');\n\nselect collection_search('{\"q\": \"\\\"great story\\\"\"}');\n\nselect collection_search('{\"q\": \"monster -school\"}');\n\nselect collection_search('{\"q\": \"+restaurant -sad\"}');\n\nselect collection_search('{\"q\": \"+restaurant\"}');\n\nselect collection_search('{\"q\": \"bear or stranger\"}');\n\nselect collection_search('{\"q\": \"bear OR stranger\"}');\n\nselect collection_search('{\"q\": \"bear, stranger\"}');\n\nselect collection_search('{\"q\": \"bear AND stranger\"}');\n\nselect collection_search('{\"q\": \"bear and stranger\"}');\n\nselect collection_search('{\"q\": \"\\\"bear or stranger\\\"\"}');\n\nselect collection_search('{\"q\": \"office\"}');\n\nselect collection_search('{\"q\": [\"bear\", \"stranger\"]}');\n\nselect collection_search('{\"q\": \"large lizard\"}');\n\nselect collection_search('{\"q\": \"teenagers fight monsters\"}');\n\nselect collection_search('{\"q\": \"scary monsters\"}');\n" }, { "path": "src/pgstac/tests/basic/free_text.sql.out", "content": "SET pgstac.context TO 'on';\nSET\nSET pgstac.\"default_filter_lang\" TO 'cql2-json';\nSET\nCREATE TEMP TABLE temp_collections (\n id SERIAL PRIMARY KEY,\n title TEXT,\n description TEXT,\n keywords TEXT,\n minx NUMERIC,\n miny NUMERIC,\n maxx NUMERIC,\n maxy NUMERIC,\n sdt TIMESTAMPTZ,\n edt TIMESTAMPTZ\n);\nCREATE TABLE\nINSERT INTO temp_collections (\n title, description, keywords, minx, miny, maxx, maxy, sdt, edt\n) VALUES\n -- no keywords\n (\n 'Stranger Things',\n 'Some teenagers drop out of school to fight scary monsters',\n null,\n -180, -90, 180, 90,\n '2016-01-01T00:00:00Z',\n '2025-12-31T23:59:59Z'\n ),\n (\n 'The Bear',\n 'Another story about why you should not start a restaurant',\n 'restaurant, funny, sad, great',\n -180, -90, 180, 90,\n '2022-01-01T00:00:00Z',\n '2025-12-31T23:59:59Z'\n ),\n (\n 'Godzilla',\n 'A large lizard takes its revenge',\n 'scary, lizard, monster',\n -180, -90, 180, 90,\n '1954-01-01T00:00:00Z',\n null\n ),\n (\n 'Chefs Table',\n 'Another great story that make you wonder if you should go to a restaurant',\n 'restaurant, food, michelin',\n -180, -90, 180, 90,\n '2019-01-01T00:00:00Z',\n '2025-12-31T23:59:59Z'\n ),\n -- no title\n (\n null,\n 'A humoristic portrayal of office life',\n 'Scranton, paper',\n -180, -90, 180, 90,\n '2005-01-01T00:00:00Z',\n '2013-12-31T23:59:59Z'\n );\nINSERT 0 5\nSELECT\n create_collection(jsonb_build_object(\n 'id', format('testcollection_%s', id),\n 'type', 'Collection',\n 'title', title,\n 'description', description,\n 'extent', jsonb_build_object(\n 'spatial', jsonb_build_array(jsonb_build_array(minx, miny, maxx, maxy)),\n 'temporal', jsonb_build_array(jsonb_build_array(sdt, edt))\n ),\n 'stac_extensions', jsonb_build_array(),\n 'keywords', string_to_array(keywords, ', ')\n ))\nFROM temp_collections;\n\n\n\n\nselect collection_search('{\"q\": \"monsters\"}');\n {\"links\": [], \"collections\": [{\"id\": \"testcollection_1\", \"type\": \"Collection\", \"title\": \"Stranger Things\", \"extent\": {\"spatial\": [[-180, -90, 180, 90]], \"temporal\": [[\"2016-01-01T00:00:00+00:00\", \"2025-12-31T23:59:59+00:00\"]]}, \"keywords\": null, \"description\": \"Some teenagers drop out of school to fight scary monsters\", \"stac_extensions\": []}, {\"id\": \"testcollection_3\", \"type\": \"Collection\", \"title\": \"Godzilla\", \"extent\": {\"spatial\": [[-180, -90, 180, 90]], \"temporal\": [[\"1954-01-01T00:00:00+00:00\", null]]}, \"keywords\": [\"scary\", \"lizard\", \"monster\"], \"description\": \"A large lizard takes its revenge\", \"stac_extensions\": []}], \"numberMatched\": 2, \"numberReturned\": 2}\n\n\nselect collection_search('{\"q\": \"lizard\"}');\n {\"links\": [], \"collections\": [{\"id\": \"testcollection_3\", \"type\": \"Collection\", \"title\": \"Godzilla\", \"extent\": {\"spatial\": [[-180, -90, 180, 90]], \"temporal\": [[\"1954-01-01T00:00:00+00:00\", null]]}, \"keywords\": [\"scary\", \"lizard\", \"monster\"], \"description\": \"A large lizard takes its revenge\", \"stac_extensions\": []}], \"numberMatched\": 1, \"numberReturned\": 1}\n\n\nselect collection_search('{\"q\": \"scary OR funny\"}');\n {\"links\": [], \"collections\": [{\"id\": \"testcollection_1\", \"type\": \"Collection\", \"title\": \"Stranger Things\", \"extent\": {\"spatial\": [[-180, -90, 180, 90]], \"temporal\": [[\"2016-01-01T00:00:00+00:00\", \"2025-12-31T23:59:59+00:00\"]]}, \"keywords\": null, \"description\": \"Some teenagers drop out of school to fight scary monsters\", \"stac_extensions\": []}, {\"id\": \"testcollection_2\", \"type\": \"Collection\", \"title\": \"The Bear\", \"extent\": {\"spatial\": [[-180, -90, 180, 90]], \"temporal\": [[\"2022-01-01T00:00:00+00:00\", \"2025-12-31T23:59:59+00:00\"]]}, \"keywords\": [\"restaurant\", \"funny\", \"sad\", \"great\"], \"description\": \"Another story about why you should not start a restaurant\", \"stac_extensions\": []}, {\"id\": \"testcollection_3\", \"type\": \"Collection\", \"title\": \"Godzilla\", \"extent\": {\"spatial\": [[-180, -90, 180, 90]], \"temporal\": [[\"1954-01-01T00:00:00+00:00\", null]]}, \"keywords\": [\"scary\", \"lizard\", \"monster\"], \"description\": \"A large lizard takes its revenge\", \"stac_extensions\": []}], \"numberMatched\": 3, \"numberReturned\": 3}\n\nselect collection_search('{\"q\": \"(scary AND revenge) OR (funny AND sad)\"}');\n {\"links\": [], \"collections\": [{\"id\": \"testcollection_2\", \"type\": \"Collection\", \"title\": \"The Bear\", \"extent\": {\"spatial\": [[-180, -90, 180, 90]], \"temporal\": [[\"2022-01-01T00:00:00+00:00\", \"2025-12-31T23:59:59+00:00\"]]}, \"keywords\": [\"restaurant\", \"funny\", \"sad\", \"great\"], \"description\": \"Another story about why you should not start a restaurant\", \"stac_extensions\": []}, {\"id\": \"testcollection_3\", \"type\": \"Collection\", \"title\": \"Godzilla\", \"extent\": {\"spatial\": [[-180, -90, 180, 90]], \"temporal\": [[\"1954-01-01T00:00:00+00:00\", null]]}, \"keywords\": [\"scary\", \"lizard\", \"monster\"], \"description\": \"A large lizard takes its revenge\", \"stac_extensions\": []}], \"numberMatched\": 2, \"numberReturned\": 2}\n\nselect collection_search('{\"q\": \"\\\"great story\\\"\"}');\n {\"links\": [], \"collections\": [{\"id\": \"testcollection_4\", \"type\": \"Collection\", \"title\": \"Chefs Table\", \"extent\": {\"spatial\": [[-180, -90, 180, 90]], \"temporal\": [[\"2019-01-01T00:00:00+00:00\", \"2025-12-31T23:59:59+00:00\"]]}, \"keywords\": [\"restaurant\", \"food\", \"michelin\"], \"description\": \"Another great story that make you wonder if you should go to a restaurant\", \"stac_extensions\": []}], \"numberMatched\": 1, \"numberReturned\": 1}\n\nselect collection_search('{\"q\": \"monster -school\"}');\n {\"links\": [], \"collections\": [{\"id\": \"testcollection_3\", \"type\": \"Collection\", \"title\": \"Godzilla\", \"extent\": {\"spatial\": [[-180, -90, 180, 90]], \"temporal\": [[\"1954-01-01T00:00:00+00:00\", null]]}, \"keywords\": [\"scary\", \"lizard\", \"monster\"], \"description\": \"A large lizard takes its revenge\", \"stac_extensions\": []}], \"numberMatched\": 1, \"numberReturned\": 1}\n\nselect collection_search('{\"q\": \"+restaurant -sad\"}');\n {\"links\": [], \"collections\": [{\"id\": \"testcollection_4\", \"type\": \"Collection\", \"title\": \"Chefs Table\", \"extent\": {\"spatial\": [[-180, -90, 180, 90]], \"temporal\": [[\"2019-01-01T00:00:00+00:00\", \"2025-12-31T23:59:59+00:00\"]]}, \"keywords\": [\"restaurant\", \"food\", \"michelin\"], \"description\": \"Another great story that make you wonder if you should go to a restaurant\", \"stac_extensions\": []}], \"numberMatched\": 1, \"numberReturned\": 1}\n\nselect collection_search('{\"q\": \"+restaurant\"}');\n {\"links\": [], \"collections\": [{\"id\": \"testcollection_2\", \"type\": \"Collection\", \"title\": \"The Bear\", \"extent\": {\"spatial\": [[-180, -90, 180, 90]], \"temporal\": [[\"2022-01-01T00:00:00+00:00\", \"2025-12-31T23:59:59+00:00\"]]}, \"keywords\": [\"restaurant\", \"funny\", \"sad\", \"great\"], \"description\": \"Another story about why you should not start a restaurant\", \"stac_extensions\": []}, {\"id\": \"testcollection_4\", \"type\": \"Collection\", \"title\": \"Chefs Table\", \"extent\": {\"spatial\": [[-180, -90, 180, 90]], \"temporal\": [[\"2019-01-01T00:00:00+00:00\", \"2025-12-31T23:59:59+00:00\"]]}, \"keywords\": [\"restaurant\", \"food\", \"michelin\"], \"description\": \"Another great story that make you wonder if you should go to a restaurant\", \"stac_extensions\": []}], \"numberMatched\": 2, \"numberReturned\": 2}\n\nselect collection_search('{\"q\": \"bear or stranger\"}');\n {\"links\": [], \"collections\": [{\"id\": \"testcollection_1\", \"type\": \"Collection\", \"title\": \"Stranger Things\", \"extent\": {\"spatial\": [[-180, -90, 180, 90]], \"temporal\": [[\"2016-01-01T00:00:00+00:00\", \"2025-12-31T23:59:59+00:00\"]]}, \"keywords\": null, \"description\": \"Some teenagers drop out of school to fight scary monsters\", \"stac_extensions\": []}, {\"id\": \"testcollection_2\", \"type\": \"Collection\", \"title\": \"The Bear\", \"extent\": {\"spatial\": [[-180, -90, 180, 90]], \"temporal\": [[\"2022-01-01T00:00:00+00:00\", \"2025-12-31T23:59:59+00:00\"]]}, \"keywords\": [\"restaurant\", \"funny\", \"sad\", \"great\"], \"description\": \"Another story about why you should not start a restaurant\", \"stac_extensions\": []}], \"numberMatched\": 2, \"numberReturned\": 2}\n\nselect collection_search('{\"q\": \"bear OR stranger\"}');\n {\"links\": [], \"collections\": [{\"id\": \"testcollection_1\", \"type\": \"Collection\", \"title\": \"Stranger Things\", \"extent\": {\"spatial\": [[-180, -90, 180, 90]], \"temporal\": [[\"2016-01-01T00:00:00+00:00\", \"2025-12-31T23:59:59+00:00\"]]}, \"keywords\": null, \"description\": \"Some teenagers drop out of school to fight scary monsters\", \"stac_extensions\": []}, {\"id\": \"testcollection_2\", \"type\": \"Collection\", \"title\": \"The Bear\", \"extent\": {\"spatial\": [[-180, -90, 180, 90]], \"temporal\": [[\"2022-01-01T00:00:00+00:00\", \"2025-12-31T23:59:59+00:00\"]]}, \"keywords\": [\"restaurant\", \"funny\", \"sad\", \"great\"], \"description\": \"Another story about why you should not start a restaurant\", \"stac_extensions\": []}], \"numberMatched\": 2, \"numberReturned\": 2}\n\nselect collection_search('{\"q\": \"bear, stranger\"}');\n {\"links\": [], \"collections\": [{\"id\": \"testcollection_1\", \"type\": \"Collection\", \"title\": \"Stranger Things\", \"extent\": {\"spatial\": [[-180, -90, 180, 90]], \"temporal\": [[\"2016-01-01T00:00:00+00:00\", \"2025-12-31T23:59:59+00:00\"]]}, \"keywords\": null, \"description\": \"Some teenagers drop out of school to fight scary monsters\", \"stac_extensions\": []}, {\"id\": \"testcollection_2\", \"type\": \"Collection\", \"title\": \"The Bear\", \"extent\": {\"spatial\": [[-180, -90, 180, 90]], \"temporal\": [[\"2022-01-01T00:00:00+00:00\", \"2025-12-31T23:59:59+00:00\"]]}, \"keywords\": [\"restaurant\", \"funny\", \"sad\", \"great\"], \"description\": \"Another story about why you should not start a restaurant\", \"stac_extensions\": []}], \"numberMatched\": 2, \"numberReturned\": 2}\n\nselect collection_search('{\"q\": \"bear AND stranger\"}');\n {\"links\": [], \"collections\": [], \"numberMatched\": 0, \"numberReturned\": 0}\n\nselect collection_search('{\"q\": \"bear and stranger\"}');\n {\"links\": [], \"collections\": [], \"numberMatched\": 0, \"numberReturned\": 0}\n\nselect collection_search('{\"q\": \"\\\"bear or stranger\\\"\"}');\n {\"links\": [], \"collections\": [], \"numberMatched\": 0, \"numberReturned\": 0}\n\n\nselect collection_search('{\"q\": \"office\"}');\n {\"links\": [], \"collections\": [{\"id\": \"testcollection_5\", \"type\": \"Collection\", \"title\": null, \"extent\": {\"spatial\": [[-180, -90, 180, 90]], \"temporal\": [[\"2005-01-01T00:00:00+00:00\", \"2013-12-31T23:59:59+00:00\"]]}, \"keywords\": [\"Scranton\", \"paper\"], \"description\": \"A humoristic portrayal of office life\", \"stac_extensions\": []}], \"numberMatched\": 1, \"numberReturned\": 1}\n\nselect collection_search('{\"q\": [\"bear\", \"stranger\"]}');\n {\"links\": [], \"collections\": [{\"id\": \"testcollection_1\", \"type\": \"Collection\", \"title\": \"Stranger Things\", \"extent\": {\"spatial\": [[-180, -90, 180, 90]], \"temporal\": [[\"2016-01-01T00:00:00+00:00\", \"2025-12-31T23:59:59+00:00\"]]}, \"keywords\": null, \"description\": \"Some teenagers drop out of school to fight scary monsters\", \"stac_extensions\": []}, {\"id\": \"testcollection_2\", \"type\": \"Collection\", \"title\": \"The Bear\", \"extent\": {\"spatial\": [[-180, -90, 180, 90]], \"temporal\": [[\"2022-01-01T00:00:00+00:00\", \"2025-12-31T23:59:59+00:00\"]]}, \"keywords\": [\"restaurant\", \"funny\", \"sad\", \"great\"], \"description\": \"Another story about why you should not start a restaurant\", \"stac_extensions\": []}], \"numberMatched\": 2, \"numberReturned\": 2}\n\nselect collection_search('{\"q\": \"large lizard\"}');\n {\"links\": [], \"collections\": [{\"id\": \"testcollection_3\", \"type\": \"Collection\", \"title\": \"Godzilla\", \"extent\": {\"spatial\": [[-180, -90, 180, 90]], \"temporal\": [[\"1954-01-01T00:00:00+00:00\", null]]}, \"keywords\": [\"scary\", \"lizard\", \"monster\"], \"description\": \"A large lizard takes its revenge\", \"stac_extensions\": []}], \"numberMatched\": 1, \"numberReturned\": 1}\n\nselect collection_search('{\"q\": \"teenagers fight monsters\"}');\n {\"links\": [], \"collections\": [], \"numberMatched\": 0, \"numberReturned\": 0}\n\nselect collection_search('{\"q\": \"scary monsters\"}');\n {\"links\": [], \"collections\": [{\"id\": \"testcollection_1\", \"type\": \"Collection\", \"title\": \"Stranger Things\", \"extent\": {\"spatial\": [[-180, -90, 180, 90]], \"temporal\": [[\"2016-01-01T00:00:00+00:00\", \"2025-12-31T23:59:59+00:00\"]]}, \"keywords\": null, \"description\": \"Some teenagers drop out of school to fight scary monsters\", \"stac_extensions\": []}], \"numberMatched\": 1, \"numberReturned\": 1}\n" }, { "path": "src/pgstac/tests/basic/partitions.sql", "content": "-- run tests as pgstac_ingest\nSET ROLE pgstac_ingest;\nSET pgstac.use_queue=FALSE;\nSELECT get_setting_bool('use_queue');\nSET pgstac.update_collection_extent=TRUE;\nSELECT get_setting_bool('update_collection_extent');\n--create base data to use with tests\nCREATE TEMP TABLE test_items AS\nSELECT jsonb_build_object(\n 'id', concat('pgstactest-partitioned-', (row_number() over ())::text),\n 'collection', 'pgstactest-partitioned',\n 'geometry', '{\"type\": \"Polygon\", \"coordinates\": [[[-85.309412, 30.933949], [-85.308201, 31.002658], [-85.378084, 31.003555], [-85.379245, 30.934843], [-85.309412, 30.933949]]]}'::json,\n 'properties', jsonb_build_object( 'datetime', g::text)\n) as content FROM generate_series('2020-01-01'::timestamptz, '2022-01-01'::timestamptz, '1 week'::interval) g;\n\n--test non-partitioned collection\nINSERT INTO collections (content) VALUES ('{\"id\":\"pgstactest-partitioned\"}');\nINSERT INTO items_staging(content)\nSELECT content FROM test_items;\nSELECT count(*) FROM partitions WHERE collection='pgstactest-partitioned';\n\n--test collection partioned by year\nINSERT INTO collections (content, partition_trunc) VALUES ('{\"id\":\"pgstactest-partitioned-year\"}', 'year');\nINSERT INTO items_staging(content)\nSELECT content || '{\"collection\":\"pgstactest-partitioned-year\"}'::jsonb FROM test_items;\nSELECT count(*) FROM partitions WHERE collection='pgstactest-partitioned-year';\n\n--test collection partioned by month\nINSERT INTO collections (content, partition_trunc) VALUES ('{\"id\":\"pgstactest-partitioned-month\"}', 'month');\nINSERT INTO items_staging(content)\nSELECT content || '{\"collection\":\"pgstactest-partitioned-month\"}'::jsonb FROM test_items;\nSELECT count(*) FROM partitions WHERE collection='pgstactest-partitioned-month';\n\n--test repartitioning from year to non partitioned\nUPDATE collections SET partition_trunc=NULL WHERE id='pgstactest-partitioned-year';\nSELECT count(*) FROM partitions WHERE collection='pgstactest-partitioned-year';\nSELECT count(*) FROM items WHERE collection='pgstactest-partitioned-year';\n\n--test repartitioning from non-partitioned to year\nUPDATE collections SET partition_trunc='year' WHERE id='pgstactest-partitioned';\nSELECT count(*) FROM partitions WHERE collection='pgstactest-partitioned';\nSELECT count(*) FROM items WHERE collection='pgstactest-partitioned';\n\n--check that partition stats have been updated\nSELECT count(*) FROM partitions WHERE collection='pgstactest-partitioned' and spatial IS NULL;\n\n--test noop for repartitioning\nUPDATE collections SET content=content || '{\"foo\":\"bar\"}'::jsonb WHERE id='pgstactest-partitioned-month';\nSELECT count(*) FROM partitions WHERE collection='pgstactest-partitioned-month';\nSELECT count(*) FROM items WHERE collection='pgstactest-partitioned-month';\n\n--test using query queue\nSET pgstac.use_queue=TRUE;\nSELECT get_setting_bool('use_queue');\n\nINSERT INTO collections (content, partition_trunc) VALUES ('{\"id\":\"pgstactest-partitioned-q\"}', 'month');\nINSERT INTO items_staging(content)\nSELECT content || '{\"collection\":\"pgstactest-partitioned-q\"}'::jsonb FROM test_items;\nSELECT count(*) FROM partitions WHERE collection='pgstactest-partitioned-q';\n\n--check that partition stats haven't been updated\nSELECT count(*) FROM partitions WHERE collection='pgstactest-partitioned-q' and spatial IS NULL;\n\n--check that queue has items\nSELECT count(*)>0 FROM query_queue;\n\n--run queue items to update partition stats\nSELECT run_queued_queries_intransaction()>0;\n\n--check that queue has been emptied\nSELECT count(*) FROM query_queue;\nSELECT run_queued_queries_intransaction();\n\n--check that partition stats have been updated\nSELECT count(*) FROM partitions WHERE collection='pgstactest-partitioned-q' and spatial IS NULL;\n\n--check that collection extents have been updated\nSELECT id, content->'extent' FROM collections WHERE id LIKE 'pgstactest-partitioned%' ORDER BY id;\n\n--check that values for datetimes that are non 4 digit or that have very high precision are ingesting correctly and that partitioning is working for them\nSET pgstac.use_queue=FALSE;\nSELECT get_setting_bool('use_queue');\n\nINSERT INTO test_items (content)\nSELECT jsonb_build_object(\n 'id', 'pgstactest-partitioned-whackyyear',\n 'collection', 'pgstactest-partitioned',\n 'geometry', '{\"type\": \"Polygon\", \"coordinates\": [[[-85.309412, 30.933949], [-85.308201, 31.002658], [-85.378084, 31.003555], [-85.379245, 30.934843], [-85.309412, 30.933949]]]}'::json,\n 'properties', jsonb_build_object( 'datetime', '10000-01-01T00:00:00Z')\n);\nINSERT INTO test_items (content)\nSELECT jsonb_build_object(\n 'id', 'pgstactest-partitioned-whackyprecision',\n 'collection', 'pgstactest-partitioned',\n 'geometry', '{\"type\": \"Polygon\", \"coordinates\": [[[-85.309412, 30.933949], [-85.308201, 31.002658], [-85.378084, 31.003555], [-85.379245, 30.934843], [-85.309412, 30.933949]]]}'::json,\n 'properties', jsonb_build_object( 'datetime', '2000-01-01T00:00:00.12389878917192387129837Z')\n);\nINSERT INTO test_items (content)\nSELECT jsonb_build_object(\n 'id', 'pgstactest-partitioned-startend',\n 'collection', 'pgstactest-partitioned',\n 'geometry', '{\"type\": \"Polygon\", \"coordinates\": [[[-85.309412, 30.933949], [-85.308201, 31.002658], [-85.378084, 31.003555], [-85.379245, 30.934843], [-85.309412, 30.933949]]]}'::json,\n 'properties', jsonb_build_object( 'start_datetime', '2000-01-01T00:00:00.12389878917192387129837Z', 'end_datetime', '99999-01-01T00:00:00Z')\n);\n\nINSERT INTO collections (content, partition_trunc) VALUES ('{\"id\":\"pgstactest-partitioned-oddballs\"}', 'month');\nINSERT INTO items_staging(content)\nSELECT content || '{\"collection\":\"pgstactest-partitioned-oddballs\"}'::jsonb FROM test_items;\n\nSELECT count(*) FROM partitions WHERE collection='pgstactest-partitioned-oddballs';\n\nSELECT collection, partition_dtrange, constraint_dtrange, constraint_edtrange, dtrange, edtrange\nFROM partitions\nWHERE collection='pgstactest-partitioned-oddballs'\nORDER BY partition_dtrange;\n" }, { "path": "src/pgstac/tests/basic/partitions.sql.out", "content": "-- run tests as pgstac_ingest\nSET ROLE pgstac_ingest;\nSET\nSET pgstac.use_queue=FALSE;\nSET\nSELECT get_setting_bool('use_queue');\n f\n\nSET pgstac.update_collection_extent=TRUE;\nSET\nSELECT get_setting_bool('update_collection_extent');\n t\n\n--create base data to use with tests\nCREATE TEMP TABLE test_items AS\nSELECT jsonb_build_object(\n 'id', concat('pgstactest-partitioned-', (row_number() over ())::text),\n 'collection', 'pgstactest-partitioned',\n 'geometry', '{\"type\": \"Polygon\", \"coordinates\": [[[-85.309412, 30.933949], [-85.308201, 31.002658], [-85.378084, 31.003555], [-85.379245, 30.934843], [-85.309412, 30.933949]]]}'::json,\n 'properties', jsonb_build_object( 'datetime', g::text)\n) as content FROM generate_series('2020-01-01'::timestamptz, '2022-01-01'::timestamptz, '1 week'::interval) g;\nSELECT 105\n--test non-partitioned collection\nINSERT INTO collections (content) VALUES ('{\"id\":\"pgstactest-partitioned\"}');\nINSERT 0 1\nINSERT INTO items_staging(content)\nSELECT content FROM test_items;\nINSERT 0 105\nSELECT count(*) FROM partitions WHERE collection='pgstactest-partitioned';\n 1\n\n--test collection partioned by year\nINSERT INTO collections (content, partition_trunc) VALUES ('{\"id\":\"pgstactest-partitioned-year\"}', 'year');\nINSERT 0 1\nINSERT INTO items_staging(content)\nSELECT content || '{\"collection\":\"pgstactest-partitioned-year\"}'::jsonb FROM test_items;\nINSERT 0 105\nSELECT count(*) FROM partitions WHERE collection='pgstactest-partitioned-year';\n 2\n\n--test collection partioned by month\nINSERT INTO collections (content, partition_trunc) VALUES ('{\"id\":\"pgstactest-partitioned-month\"}', 'month');\nINSERT 0 1\nINSERT INTO items_staging(content)\nSELECT content || '{\"collection\":\"pgstactest-partitioned-month\"}'::jsonb FROM test_items;\nINSERT 0 105\nSELECT count(*) FROM partitions WHERE collection='pgstactest-partitioned-month';\n 24\n\n--test repartitioning from year to non partitioned\nUPDATE collections SET partition_trunc=NULL WHERE id='pgstactest-partitioned-year';\nUPDATE 1\nSELECT count(*) FROM partitions WHERE collection='pgstactest-partitioned-year';\n 1\n\nSELECT count(*) FROM items WHERE collection='pgstactest-partitioned-year';\n 105\n\n--test repartitioning from non-partitioned to year\nUPDATE collections SET partition_trunc='year' WHERE id='pgstactest-partitioned';\nUPDATE 1\nSELECT count(*) FROM partitions WHERE collection='pgstactest-partitioned';\n 2\n\nSELECT count(*) FROM items WHERE collection='pgstactest-partitioned';\n 105\n\n--check that partition stats have been updated\nSELECT count(*) FROM partitions WHERE collection='pgstactest-partitioned' and spatial IS NULL;\n 0\n\n--test noop for repartitioning\nUPDATE collections SET content=content || '{\"foo\":\"bar\"}'::jsonb WHERE id='pgstactest-partitioned-month';\nUPDATE 1\nSELECT count(*) FROM partitions WHERE collection='pgstactest-partitioned-month';\n 24\n\nSELECT count(*) FROM items WHERE collection='pgstactest-partitioned-month';\n 105\n\n--test using query queue\nSET pgstac.use_queue=TRUE;\nSET\nSELECT get_setting_bool('use_queue');\n t\n\nINSERT INTO collections (content, partition_trunc) VALUES ('{\"id\":\"pgstactest-partitioned-q\"}', 'month');\nINSERT 0 1\nINSERT INTO items_staging(content)\nSELECT content || '{\"collection\":\"pgstactest-partitioned-q\"}'::jsonb FROM test_items;\nINSERT 0 105\nSELECT count(*) FROM partitions WHERE collection='pgstactest-partitioned-q';\n 24\n\n--check that partition stats haven't been updated\nSELECT count(*) FROM partitions WHERE collection='pgstactest-partitioned-q' and spatial IS NULL;\n 24\n\n--check that queue has items\nSELECT count(*)>0 FROM query_queue;\n t\n\n--run queue items to update partition stats\nSELECT run_queued_queries_intransaction()>0;\n t\n\n--check that queue has been emptied\nSELECT count(*) FROM query_queue;\n 0\n\nSELECT run_queued_queries_intransaction();\n 0\n\n--check that partition stats have been updated\nSELECT count(*) FROM partitions WHERE collection='pgstactest-partitioned-q' and spatial IS NULL;\n 0\n\n--check that collection extents have been updated\nSELECT id, content->'extent' FROM collections WHERE id LIKE 'pgstactest-partitioned%' ORDER BY id;\n pgstactest-partitioned | {\"spatial\": {\"bbox\": [[-85.3792495727539, 30.933948516845703, -85.30819702148438, 31.003555297851562]]}, \"temporal\": {\"interval\": [[\"2020-01-01T00:00:00+00:00\", \"2021-12-29T00:00:00+00:00\"]]}}\n pgstactest-partitioned-month | {\"spatial\": {\"bbox\": [[-85.3792495727539, 30.933948516845703, -85.30819702148438, 31.003555297851562]]}, \"temporal\": {\"interval\": [[\"2020-01-01T00:00:00+00:00\", \"2021-12-29T00:00:00+00:00\"]]}}\n pgstactest-partitioned-q | {\"spatial\": {\"bbox\": [[-85.3792495727539, 30.933948516845703, -85.30819702148438, 31.003555297851562]]}, \"temporal\": {\"interval\": [[\"2020-01-01T00:00:00+00:00\", \"2021-12-29T00:00:00+00:00\"]]}}\n pgstactest-partitioned-year | {\"spatial\": {\"bbox\": [[-85.3792495727539, 30.933948516845703, -85.30819702148438, 31.003555297851562]]}, \"temporal\": {\"interval\": [[\"2020-01-01T00:00:00+00:00\", \"2021-12-29T00:00:00+00:00\"]]}}\n\n--check that values for datetimes that are non 4 digit or that have very high precision are ingesting correctly and that partitioning is working for them\nSET pgstac.use_queue=FALSE;\nSET\nSELECT get_setting_bool('use_queue');\n f\n\nINSERT INTO test_items (content)\nSELECT jsonb_build_object(\n 'id', 'pgstactest-partitioned-whackyyear',\n 'collection', 'pgstactest-partitioned',\n 'geometry', '{\"type\": \"Polygon\", \"coordinates\": [[[-85.309412, 30.933949], [-85.308201, 31.002658], [-85.378084, 31.003555], [-85.379245, 30.934843], [-85.309412, 30.933949]]]}'::json,\n 'properties', jsonb_build_object( 'datetime', '10000-01-01T00:00:00Z')\n);\nINSERT 0 1\nINSERT INTO test_items (content)\nSELECT jsonb_build_object(\n 'id', 'pgstactest-partitioned-whackyprecision',\n 'collection', 'pgstactest-partitioned',\n 'geometry', '{\"type\": \"Polygon\", \"coordinates\": [[[-85.309412, 30.933949], [-85.308201, 31.002658], [-85.378084, 31.003555], [-85.379245, 30.934843], [-85.309412, 30.933949]]]}'::json,\n 'properties', jsonb_build_object( 'datetime', '2000-01-01T00:00:00.12389878917192387129837Z')\n);\nINSERT 0 1\nINSERT INTO test_items (content)\nSELECT jsonb_build_object(\n 'id', 'pgstactest-partitioned-startend',\n 'collection', 'pgstactest-partitioned',\n 'geometry', '{\"type\": \"Polygon\", \"coordinates\": [[[-85.309412, 30.933949], [-85.308201, 31.002658], [-85.378084, 31.003555], [-85.379245, 30.934843], [-85.309412, 30.933949]]]}'::json,\n 'properties', jsonb_build_object( 'start_datetime', '2000-01-01T00:00:00.12389878917192387129837Z', 'end_datetime', '99999-01-01T00:00:00Z')\n);\nINSERT 0 1\nINSERT INTO collections (content, partition_trunc) VALUES ('{\"id\":\"pgstactest-partitioned-oddballs\"}', 'month');\nINSERT 0 1\nINSERT INTO items_staging(content)\nSELECT content || '{\"collection\":\"pgstactest-partitioned-oddballs\"}'::jsonb FROM test_items;\nINSERT 0 108\nSELECT count(*) FROM partitions WHERE collection='pgstactest-partitioned-oddballs';\n 26\n\nSELECT collection, partition_dtrange, constraint_dtrange, constraint_edtrange, dtrange, edtrange\nFROM partitions\nWHERE collection='pgstactest-partitioned-oddballs'\nORDER BY partition_dtrange;\n pgstactest-partitioned-oddballs | [\"2000-01-01 00:00:00+00\",\"2000-02-01 00:00:00+00\") | [\"2000-01-01 00:00:00+00\",\"2000-02-01 00:00:00+00\") | [-infinity,infinity] | [\"2000-01-01 00:00:00.123899+00\",\"2000-01-01 00:00:00.123899+00\"] | [\"2000-01-01 00:00:00.123899+00\",\"99999-01-01 00:00:00+00\"]\n pgstactest-partitioned-oddballs | [\"2020-01-01 00:00:00+00\",\"2020-02-01 00:00:00+00\") | [\"2020-01-01 00:00:00+00\",\"2020-02-01 00:00:00+00\") | [-infinity,infinity] | [\"2020-01-01 00:00:00+00\",\"2020-01-29 00:00:00+00\"] | [\"2020-01-01 00:00:00+00\",\"2020-01-29 00:00:00+00\"]\n pgstactest-partitioned-oddballs | [\"2020-02-01 00:00:00+00\",\"2020-03-01 00:00:00+00\") | [\"2020-02-01 00:00:00+00\",\"2020-03-01 00:00:00+00\") | [-infinity,infinity] | [\"2020-02-05 00:00:00+00\",\"2020-02-26 00:00:00+00\"] | [\"2020-02-05 00:00:00+00\",\"2020-02-26 00:00:00+00\"]\n pgstactest-partitioned-oddballs | [\"2020-03-01 00:00:00+00\",\"2020-04-01 00:00:00+00\") | [\"2020-03-01 00:00:00+00\",\"2020-04-01 00:00:00+00\") | [-infinity,infinity] | [\"2020-03-04 00:00:00+00\",\"2020-03-25 00:00:00+00\"] | [\"2020-03-04 00:00:00+00\",\"2020-03-25 00:00:00+00\"]\n pgstactest-partitioned-oddballs | [\"2020-04-01 00:00:00+00\",\"2020-05-01 00:00:00+00\") | [\"2020-04-01 00:00:00+00\",\"2020-05-01 00:00:00+00\") | [-infinity,infinity] | [\"2020-04-01 00:00:00+00\",\"2020-04-29 00:00:00+00\"] | [\"2020-04-01 00:00:00+00\",\"2020-04-29 00:00:00+00\"]\n pgstactest-partitioned-oddballs | [\"2020-05-01 00:00:00+00\",\"2020-06-01 00:00:00+00\") | [\"2020-05-01 00:00:00+00\",\"2020-06-01 00:00:00+00\") | [-infinity,infinity] | [\"2020-05-06 00:00:00+00\",\"2020-05-27 00:00:00+00\"] | [\"2020-05-06 00:00:00+00\",\"2020-05-27 00:00:00+00\"]\n pgstactest-partitioned-oddballs | [\"2020-06-01 00:00:00+00\",\"2020-07-01 00:00:00+00\") | [\"2020-06-01 00:00:00+00\",\"2020-07-01 00:00:00+00\") | [-infinity,infinity] | [\"2020-06-03 00:00:00+00\",\"2020-06-24 00:00:00+00\"] | [\"2020-06-03 00:00:00+00\",\"2020-06-24 00:00:00+00\"]\n pgstactest-partitioned-oddballs | [\"2020-07-01 00:00:00+00\",\"2020-08-01 00:00:00+00\") | [\"2020-07-01 00:00:00+00\",\"2020-08-01 00:00:00+00\") | [-infinity,infinity] | [\"2020-07-01 00:00:00+00\",\"2020-07-29 00:00:00+00\"] | [\"2020-07-01 00:00:00+00\",\"2020-07-29 00:00:00+00\"]\n pgstactest-partitioned-oddballs | [\"2020-08-01 00:00:00+00\",\"2020-09-01 00:00:00+00\") | [\"2020-08-01 00:00:00+00\",\"2020-09-01 00:00:00+00\") | [-infinity,infinity] | [\"2020-08-05 00:00:00+00\",\"2020-08-26 00:00:00+00\"] | [\"2020-08-05 00:00:00+00\",\"2020-08-26 00:00:00+00\"]\n pgstactest-partitioned-oddballs | [\"2020-09-01 00:00:00+00\",\"2020-10-01 00:00:00+00\") | [\"2020-09-01 00:00:00+00\",\"2020-10-01 00:00:00+00\") | [-infinity,infinity] | [\"2020-09-02 00:00:00+00\",\"2020-09-30 00:00:00+00\"] | [\"2020-09-02 00:00:00+00\",\"2020-09-30 00:00:00+00\"]\n pgstactest-partitioned-oddballs | [\"2020-10-01 00:00:00+00\",\"2020-11-01 00:00:00+00\") | [\"2020-10-01 00:00:00+00\",\"2020-11-01 00:00:00+00\") | [-infinity,infinity] | [\"2020-10-07 00:00:00+00\",\"2020-10-28 00:00:00+00\"] | [\"2020-10-07 00:00:00+00\",\"2020-10-28 00:00:00+00\"]\n pgstactest-partitioned-oddballs | [\"2020-11-01 00:00:00+00\",\"2020-12-01 00:00:00+00\") | [\"2020-11-01 00:00:00+00\",\"2020-12-01 00:00:00+00\") | [-infinity,infinity] | [\"2020-11-04 00:00:00+00\",\"2020-11-25 00:00:00+00\"] | [\"2020-11-04 00:00:00+00\",\"2020-11-25 00:00:00+00\"]\n pgstactest-partitioned-oddballs | [\"2020-12-01 00:00:00+00\",\"2021-01-01 00:00:00+00\") | [\"2020-12-01 00:00:00+00\",\"2021-01-01 00:00:00+00\") | [-infinity,infinity] | [\"2020-12-02 00:00:00+00\",\"2020-12-30 00:00:00+00\"] | [\"2020-12-02 00:00:00+00\",\"2020-12-30 00:00:00+00\"]\n pgstactest-partitioned-oddballs | [\"2021-01-01 00:00:00+00\",\"2021-02-01 00:00:00+00\") | [\"2021-01-01 00:00:00+00\",\"2021-02-01 00:00:00+00\") | [-infinity,infinity] | [\"2021-01-06 00:00:00+00\",\"2021-01-27 00:00:00+00\"] | [\"2021-01-06 00:00:00+00\",\"2021-01-27 00:00:00+00\"]\n pgstactest-partitioned-oddballs | [\"2021-02-01 00:00:00+00\",\"2021-03-01 00:00:00+00\") | [\"2021-02-01 00:00:00+00\",\"2021-03-01 00:00:00+00\") | [-infinity,infinity] | [\"2021-02-03 00:00:00+00\",\"2021-02-24 00:00:00+00\"] | [\"2021-02-03 00:00:00+00\",\"2021-02-24 00:00:00+00\"]\n pgstactest-partitioned-oddballs | [\"2021-03-01 00:00:00+00\",\"2021-04-01 00:00:00+00\") | [\"2021-03-01 00:00:00+00\",\"2021-04-01 00:00:00+00\") | [-infinity,infinity] | [\"2021-03-03 00:00:00+00\",\"2021-03-31 00:00:00+00\"] | [\"2021-03-03 00:00:00+00\",\"2021-03-31 00:00:00+00\"]\n pgstactest-partitioned-oddballs | [\"2021-04-01 00:00:00+00\",\"2021-05-01 00:00:00+00\") | [\"2021-04-01 00:00:00+00\",\"2021-05-01 00:00:00+00\") | [-infinity,infinity] | [\"2021-04-07 00:00:00+00\",\"2021-04-28 00:00:00+00\"] | [\"2021-04-07 00:00:00+00\",\"2021-04-28 00:00:00+00\"]\n pgstactest-partitioned-oddballs | [\"2021-05-01 00:00:00+00\",\"2021-06-01 00:00:00+00\") | [\"2021-05-01 00:00:00+00\",\"2021-06-01 00:00:00+00\") | [-infinity,infinity] | [\"2021-05-05 00:00:00+00\",\"2021-05-26 00:00:00+00\"] | [\"2021-05-05 00:00:00+00\",\"2021-05-26 00:00:00+00\"]\n pgstactest-partitioned-oddballs | [\"2021-06-01 00:00:00+00\",\"2021-07-01 00:00:00+00\") | [\"2021-06-01 00:00:00+00\",\"2021-07-01 00:00:00+00\") | [-infinity,infinity] | [\"2021-06-02 00:00:00+00\",\"2021-06-30 00:00:00+00\"] | [\"2021-06-02 00:00:00+00\",\"2021-06-30 00:00:00+00\"]\n pgstactest-partitioned-oddballs | [\"2021-07-01 00:00:00+00\",\"2021-08-01 00:00:00+00\") | [\"2021-07-01 00:00:00+00\",\"2021-08-01 00:00:00+00\") | [-infinity,infinity] | [\"2021-07-07 00:00:00+00\",\"2021-07-28 00:00:00+00\"] | [\"2021-07-07 00:00:00+00\",\"2021-07-28 00:00:00+00\"]\n pgstactest-partitioned-oddballs | [\"2021-08-01 00:00:00+00\",\"2021-09-01 00:00:00+00\") | [\"2021-08-01 00:00:00+00\",\"2021-09-01 00:00:00+00\") | [-infinity,infinity] | [\"2021-08-04 00:00:00+00\",\"2021-08-25 00:00:00+00\"] | [\"2021-08-04 00:00:00+00\",\"2021-08-25 00:00:00+00\"]\n pgstactest-partitioned-oddballs | [\"2021-09-01 00:00:00+00\",\"2021-10-01 00:00:00+00\") | [\"2021-09-01 00:00:00+00\",\"2021-10-01 00:00:00+00\") | [-infinity,infinity] | [\"2021-09-01 00:00:00+00\",\"2021-09-29 00:00:00+00\"] | [\"2021-09-01 00:00:00+00\",\"2021-09-29 00:00:00+00\"]\n pgstactest-partitioned-oddballs | [\"2021-10-01 00:00:00+00\",\"2021-11-01 00:00:00+00\") | [\"2021-10-01 00:00:00+00\",\"2021-11-01 00:00:00+00\") | [-infinity,infinity] | [\"2021-10-06 00:00:00+00\",\"2021-10-27 00:00:00+00\"] | [\"2021-10-06 00:00:00+00\",\"2021-10-27 00:00:00+00\"]\n pgstactest-partitioned-oddballs | [\"2021-11-01 00:00:00+00\",\"2021-12-01 00:00:00+00\") | [\"2021-11-01 00:00:00+00\",\"2021-12-01 00:00:00+00\") | [-infinity,infinity] | [\"2021-11-03 00:00:00+00\",\"2021-11-24 00:00:00+00\"] | [\"2021-11-03 00:00:00+00\",\"2021-11-24 00:00:00+00\"]\n pgstactest-partitioned-oddballs | [\"2021-12-01 00:00:00+00\",\"2022-01-01 00:00:00+00\") | [\"2021-12-01 00:00:00+00\",\"2022-01-01 00:00:00+00\") | [-infinity,infinity] | [\"2021-12-01 00:00:00+00\",\"2021-12-29 00:00:00+00\"] | [\"2021-12-01 00:00:00+00\",\"2021-12-29 00:00:00+00\"]\n pgstactest-partitioned-oddballs | [\"10000-01-01 00:00:00+00\",\"10000-02-01 00:00:00+00\") | [\"10000-01-01 00:00:00+00\",\"10000-02-01 00:00:00+00\") | [-infinity,infinity] | [\"10000-01-01 00:00:00+00\",\"10000-01-01 00:00:00+00\"] | [\"10000-01-01 00:00:00+00\",\"10000-01-01 00:00:00+00\"]\n" }, { "path": "src/pgstac/tests/basic/search_path.sql", "content": "-- Test that partition views and tables work identically with and without pgstac in search_path\nSET ROLE pgstac_ingest;\nSET pgstac.use_queue=FALSE;\n\n-- Set up test data with pgstac in search_path\nINSERT INTO collections (content, partition_trunc) VALUES ('{\"id\":\"pgstactest-searchpath\"}', 'month');\nCREATE TEMP TABLE sp_test_items AS\nSELECT jsonb_build_object(\n 'id', concat('pgstactest-searchpath-', (row_number() over ())::text),\n 'collection', 'pgstactest-searchpath',\n 'geometry', '{\"type\": \"Polygon\", \"coordinates\": [[[-85.309412, 30.933949], [-85.308201, 31.002658], [-85.378084, 31.003555], [-85.379245, 30.934843], [-85.309412, 30.933949]]]}'::json,\n 'properties', jsonb_build_object('datetime', g::text)\n) as content FROM generate_series('2020-01-01'::timestamptz, '2020-06-01'::timestamptz, '1 week'::interval) g;\nINSERT INTO items_staging(content) SELECT content FROM sp_test_items;\n\n-- Capture results with pgstac in search_path\nCREATE TEMP TABLE sp_partition_sys_meta AS\nSELECT * FROM partition_sys_meta WHERE collection='pgstactest-searchpath' ORDER BY partition;\n\nCREATE TEMP TABLE sp_partition_stats AS\nSELECT * FROM partition_stats WHERE partition IN (SELECT partition FROM sp_partition_sys_meta) ORDER BY partition;\n\nCREATE TEMP TABLE sp_partitions AS\nSELECT * FROM partitions WHERE collection='pgstactest-searchpath' ORDER BY partition;\n\nCREATE TEMP TABLE sp_partitions_view AS\nSELECT * FROM partitions_view WHERE collection='pgstactest-searchpath' ORDER BY partition;\n\nCREATE TEMP TABLE sp_partition_steps AS\nSELECT * FROM partition_steps WHERE name IN (SELECT partition FROM sp_partition_sys_meta) ORDER BY name;\n\n-- Verify we have data\nSELECT count(*) > 0 AS has_partition_sys_meta FROM sp_partition_sys_meta;\nSELECT count(*) > 0 AS has_partition_stats FROM sp_partition_stats;\nSELECT count(*) > 0 AS has_partitions FROM sp_partitions;\nSELECT count(*) > 0 AS has_partitions_view FROM sp_partitions_view;\nSELECT count(*) > 0 AS has_partition_steps FROM sp_partition_steps;\n\n-- Now remove pgstac from search_path\nSET search_path TO public;\n\n-- partition_sys_meta: compare counts and key columns\nSELECT (\n SELECT count(*) FROM pgstac.partition_sys_meta WHERE collection='pgstactest-searchpath'\n) = (\n SELECT count(*) FROM sp_partition_sys_meta\n) AS partition_sys_meta_count_match;\n\nSELECT count(*) = 0 AS partition_sys_meta_data_match FROM (\n (SELECT partition, collection, constraint_dtrange, constraint_edtrange FROM pgstac.partition_sys_meta WHERE collection='pgstactest-searchpath'\n EXCEPT\n SELECT partition, collection, constraint_dtrange, constraint_edtrange FROM sp_partition_sys_meta)\n UNION ALL\n (SELECT partition, collection, constraint_dtrange, constraint_edtrange FROM sp_partition_sys_meta\n EXCEPT\n SELECT partition, collection, constraint_dtrange, constraint_edtrange FROM pgstac.partition_sys_meta WHERE collection='pgstactest-searchpath')\n) diff;\n\n-- partition_stats: compare counts and key columns\nSELECT (\n SELECT count(*) FROM pgstac.partition_stats WHERE partition IN (SELECT partition FROM sp_partition_sys_meta)\n) = (\n SELECT count(*) FROM sp_partition_stats\n) AS partition_stats_count_match;\n\nSELECT count(*) = 0 AS partition_stats_data_match FROM (\n (SELECT partition, dtrange, edtrange FROM pgstac.partition_stats WHERE partition IN (SELECT partition FROM sp_partition_sys_meta)\n EXCEPT\n SELECT partition, dtrange, edtrange FROM sp_partition_stats)\n UNION ALL\n (SELECT partition, dtrange, edtrange FROM sp_partition_stats\n EXCEPT\n SELECT partition, dtrange, edtrange FROM pgstac.partition_stats WHERE partition IN (SELECT partition FROM sp_partition_sys_meta))\n) diff;\n\n-- partitions (materialized view): compare counts and key columns\nSELECT (\n SELECT count(*) FROM pgstac.partitions WHERE collection='pgstactest-searchpath'\n) = (\n SELECT count(*) FROM sp_partitions\n) AS partitions_count_match;\n\nSELECT count(*) = 0 AS partitions_data_match FROM (\n (SELECT partition, collection, partition_dtrange, constraint_dtrange, constraint_edtrange, dtrange, edtrange FROM pgstac.partitions WHERE collection='pgstactest-searchpath'\n EXCEPT\n SELECT partition, collection, partition_dtrange, constraint_dtrange, constraint_edtrange, dtrange, edtrange FROM sp_partitions)\n UNION ALL\n (SELECT partition, collection, partition_dtrange, constraint_dtrange, constraint_edtrange, dtrange, edtrange FROM sp_partitions\n EXCEPT\n SELECT partition, collection, partition_dtrange, constraint_dtrange, constraint_edtrange, dtrange, edtrange FROM pgstac.partitions WHERE collection='pgstactest-searchpath')\n) diff;\n\n-- partitions_view: compare counts and key columns\nSELECT (\n SELECT count(*) FROM pgstac.partitions_view WHERE collection='pgstactest-searchpath'\n) = (\n SELECT count(*) FROM sp_partitions_view\n) AS partitions_view_count_match;\n\nSELECT count(*) = 0 AS partitions_view_data_match FROM (\n (SELECT partition, collection, partition_dtrange, constraint_dtrange, constraint_edtrange, dtrange, edtrange FROM pgstac.partitions_view WHERE collection='pgstactest-searchpath'\n EXCEPT\n SELECT partition, collection, partition_dtrange, constraint_dtrange, constraint_edtrange, dtrange, edtrange FROM sp_partitions_view)\n UNION ALL\n (SELECT partition, collection, partition_dtrange, constraint_dtrange, constraint_edtrange, dtrange, edtrange FROM sp_partitions_view\n EXCEPT\n SELECT partition, collection, partition_dtrange, constraint_dtrange, constraint_edtrange, dtrange, edtrange FROM pgstac.partitions_view WHERE collection='pgstactest-searchpath')\n) diff;\n\n-- partition_steps: compare counts and key columns\nSELECT (\n SELECT count(*) FROM pgstac.partition_steps WHERE name IN (SELECT partition FROM sp_partition_sys_meta)\n) = (\n SELECT count(*) FROM sp_partition_steps\n) AS partition_steps_count_match;\n\nSELECT count(*) = 0 AS partition_steps_data_match FROM (\n (SELECT name, sdate, edate FROM pgstac.partition_steps WHERE name IN (SELECT partition FROM sp_partition_sys_meta)\n EXCEPT\n SELECT name, sdate, edate FROM sp_partition_steps)\n UNION ALL\n (SELECT name, sdate, edate FROM sp_partition_steps\n EXCEPT\n SELECT name, sdate, edate FROM pgstac.partition_steps WHERE name IN (SELECT partition FROM sp_partition_sys_meta))\n) diff;\n\n-- Restore search_path\nSET search_path TO pgstac, public;\n" }, { "path": "src/pgstac/tests/basic/search_path.sql.out", "content": "-- Test that partition views and tables work identically with and without pgstac in search_path\nSET ROLE pgstac_ingest;\nSET\nSET pgstac.use_queue=FALSE;\nSET\n-- Set up test data with pgstac in search_path\nINSERT INTO collections (content, partition_trunc) VALUES ('{\"id\":\"pgstactest-searchpath\"}', 'month');\nINSERT 0 1\nCREATE TEMP TABLE sp_test_items AS\nSELECT jsonb_build_object(\n 'id', concat('pgstactest-searchpath-', (row_number() over ())::text),\n 'collection', 'pgstactest-searchpath',\n 'geometry', '{\"type\": \"Polygon\", \"coordinates\": [[[-85.309412, 30.933949], [-85.308201, 31.002658], [-85.378084, 31.003555], [-85.379245, 30.934843], [-85.309412, 30.933949]]]}'::json,\n 'properties', jsonb_build_object('datetime', g::text)\n) as content FROM generate_series('2020-01-01'::timestamptz, '2020-06-01'::timestamptz, '1 week'::interval) g;\nSELECT 22\nINSERT INTO items_staging(content) SELECT content FROM sp_test_items;\nINSERT 0 22\n-- Capture results with pgstac in search_path\nCREATE TEMP TABLE sp_partition_sys_meta AS\nSELECT * FROM partition_sys_meta WHERE collection='pgstactest-searchpath' ORDER BY partition;\nSELECT 5\nCREATE TEMP TABLE sp_partition_stats AS\nSELECT * FROM partition_stats WHERE partition IN (SELECT partition FROM sp_partition_sys_meta) ORDER BY partition;\nSELECT 5\nCREATE TEMP TABLE sp_partitions AS\nSELECT * FROM partitions WHERE collection='pgstactest-searchpath' ORDER BY partition;\nSELECT 5\nCREATE TEMP TABLE sp_partitions_view AS\nSELECT * FROM partitions_view WHERE collection='pgstactest-searchpath' ORDER BY partition;\nSELECT 5\nCREATE TEMP TABLE sp_partition_steps AS\nSELECT * FROM partition_steps WHERE name IN (SELECT partition FROM sp_partition_sys_meta) ORDER BY name;\nSELECT 5\n-- Verify we have data\nSELECT count(*) > 0 AS has_partition_sys_meta FROM sp_partition_sys_meta;\n t\n\nSELECT count(*) > 0 AS has_partition_stats FROM sp_partition_stats;\n t\n\nSELECT count(*) > 0 AS has_partitions FROM sp_partitions;\n t\n\nSELECT count(*) > 0 AS has_partitions_view FROM sp_partitions_view;\n t\n\nSELECT count(*) > 0 AS has_partition_steps FROM sp_partition_steps;\n t\n\n-- Now remove pgstac from search_path\nSET search_path TO public;\nSET\n-- partition_sys_meta: compare counts and key columns\nSELECT (\n SELECT count(*) FROM pgstac.partition_sys_meta WHERE collection='pgstactest-searchpath'\n) = (\n SELECT count(*) FROM sp_partition_sys_meta\n) AS partition_sys_meta_count_match;\n t\n\nSELECT count(*) = 0 AS partition_sys_meta_data_match FROM (\n (SELECT partition, collection, constraint_dtrange, constraint_edtrange FROM pgstac.partition_sys_meta WHERE collection='pgstactest-searchpath'\n EXCEPT\n SELECT partition, collection, constraint_dtrange, constraint_edtrange FROM sp_partition_sys_meta)\n UNION ALL\n (SELECT partition, collection, constraint_dtrange, constraint_edtrange FROM sp_partition_sys_meta\n EXCEPT\n SELECT partition, collection, constraint_dtrange, constraint_edtrange FROM pgstac.partition_sys_meta WHERE collection='pgstactest-searchpath')\n) diff;\n t\n\n-- partition_stats: compare counts and key columns\nSELECT (\n SELECT count(*) FROM pgstac.partition_stats WHERE partition IN (SELECT partition FROM sp_partition_sys_meta)\n) = (\n SELECT count(*) FROM sp_partition_stats\n) AS partition_stats_count_match;\n t\n\nSELECT count(*) = 0 AS partition_stats_data_match FROM (\n (SELECT partition, dtrange, edtrange FROM pgstac.partition_stats WHERE partition IN (SELECT partition FROM sp_partition_sys_meta)\n EXCEPT\n SELECT partition, dtrange, edtrange FROM sp_partition_stats)\n UNION ALL\n (SELECT partition, dtrange, edtrange FROM sp_partition_stats\n EXCEPT\n SELECT partition, dtrange, edtrange FROM pgstac.partition_stats WHERE partition IN (SELECT partition FROM sp_partition_sys_meta))\n) diff;\n t\n\n-- partitions (materialized view): compare counts and key columns\nSELECT (\n SELECT count(*) FROM pgstac.partitions WHERE collection='pgstactest-searchpath'\n) = (\n SELECT count(*) FROM sp_partitions\n) AS partitions_count_match;\n t\n\nSELECT count(*) = 0 AS partitions_data_match FROM (\n (SELECT partition, collection, partition_dtrange, constraint_dtrange, constraint_edtrange, dtrange, edtrange FROM pgstac.partitions WHERE collection='pgstactest-searchpath'\n EXCEPT\n SELECT partition, collection, partition_dtrange, constraint_dtrange, constraint_edtrange, dtrange, edtrange FROM sp_partitions)\n UNION ALL\n (SELECT partition, collection, partition_dtrange, constraint_dtrange, constraint_edtrange, dtrange, edtrange FROM sp_partitions\n EXCEPT\n SELECT partition, collection, partition_dtrange, constraint_dtrange, constraint_edtrange, dtrange, edtrange FROM pgstac.partitions WHERE collection='pgstactest-searchpath')\n) diff;\n t\n\n-- partitions_view: compare counts and key columns\nSELECT (\n SELECT count(*) FROM pgstac.partitions_view WHERE collection='pgstactest-searchpath'\n) = (\n SELECT count(*) FROM sp_partitions_view\n) AS partitions_view_count_match;\n t\n\nSELECT count(*) = 0 AS partitions_view_data_match FROM (\n (SELECT partition, collection, partition_dtrange, constraint_dtrange, constraint_edtrange, dtrange, edtrange FROM pgstac.partitions_view WHERE collection='pgstactest-searchpath'\n EXCEPT\n SELECT partition, collection, partition_dtrange, constraint_dtrange, constraint_edtrange, dtrange, edtrange FROM sp_partitions_view)\n UNION ALL\n (SELECT partition, collection, partition_dtrange, constraint_dtrange, constraint_edtrange, dtrange, edtrange FROM sp_partitions_view\n EXCEPT\n SELECT partition, collection, partition_dtrange, constraint_dtrange, constraint_edtrange, dtrange, edtrange FROM pgstac.partitions_view WHERE collection='pgstactest-searchpath')\n) diff;\n t\n\n-- partition_steps: compare counts and key columns\nSELECT (\n SELECT count(*) FROM pgstac.partition_steps WHERE name IN (SELECT partition FROM sp_partition_sys_meta)\n) = (\n SELECT count(*) FROM sp_partition_steps\n) AS partition_steps_count_match;\n t\n\nSELECT count(*) = 0 AS partition_steps_data_match FROM (\n (SELECT name, sdate, edate FROM pgstac.partition_steps WHERE name IN (SELECT partition FROM sp_partition_sys_meta)\n EXCEPT\n SELECT name, sdate, edate FROM sp_partition_steps)\n UNION ALL\n (SELECT name, sdate, edate FROM sp_partition_steps\n EXCEPT\n SELECT name, sdate, edate FROM pgstac.partition_steps WHERE name IN (SELECT partition FROM sp_partition_sys_meta))\n) diff;\n t\n\n-- Restore search_path\nSET search_path TO pgstac, public;\nSET\n" }, { "path": "src/pgstac/tests/basic/xyz_searches.sql", "content": "SET pgstac.\"default_filter_lang\" TO 'cql-json';\n\nSELECT hash from search_query('{\"collections\":[\"pgstac-test-collection\"]}');\n\nSELECT hash, search, metadata FROM search_fromhash('2bbae9a0ef0bbb5ffaca06603ce621d7');\n\nSELECT xyzsearch(8615, 13418, 15, '2bbae9a0ef0bbb5ffaca06603ce621d7', '{\"include\":[\"id\"]}'::jsonb);\n\nSELECT xyzsearch(1048, 1682, 12, '2bbae9a0ef0bbb5ffaca06603ce621d7', '{\"include\":[\"id\"]}'::jsonb);\n\nSELECT xyzsearch(1048, 1682, 12, '2bbae9a0ef0bbb5ffaca06603ce621d7', '{\"include\":[\"id\"]}'::jsonb, NULL, 1);\n\nSELECT xyzsearch(16792, 26892, 16, '2bbae9a0ef0bbb5ffaca06603ce621d7', '{\"include\":[\"id\"]}'::jsonb, exitwhenfull => true);\n\nSELECT xyzsearch(16792, 26892, 16, '2bbae9a0ef0bbb5ffaca06603ce621d7', '{\"include\":[\"id\"]}'::jsonb, exitwhenfull => false, skipcovered => false);\n\nSELECT geojsonsearch('{\"type\": \"Point\",\"coordinates\": [-87.75608539581299,30.692471153735646]}', '2bbae9a0ef0bbb5ffaca06603ce621d7', '{\"include\":[\"id\"]}'::jsonb, exitwhenfull => true, skipcovered => true);\n\nSELECT geojsonsearch('{\"type\": \"Point\",\"coordinates\": [-87.75608539581299,30.692471153735646]}', '2bbae9a0ef0bbb5ffaca06603ce621d7', '{\"include\":[\"id\"]}'::jsonb, exitwhenfull => false, skipcovered => false) s;\n" }, { "path": "src/pgstac/tests/basic/xyz_searches.sql.out", "content": "SET pgstac.\"default_filter_lang\" TO 'cql-json';\nSET\nSELECT hash from search_query('{\"collections\":[\"pgstac-test-collection\"]}');\n 2bbae9a0ef0bbb5ffaca06603ce621d7\n\nSELECT hash, search, metadata FROM search_fromhash('2bbae9a0ef0bbb5ffaca06603ce621d7');\n 2bbae9a0ef0bbb5ffaca06603ce621d7 | {\"collections\": [\"pgstac-test-collection\"]} | {}\n\nSELECT xyzsearch(8615, 13418, 15, '2bbae9a0ef0bbb5ffaca06603ce621d7', '{\"include\":[\"id\"]}'::jsonb);\n {\"type\": \"FeatureCollection\", \"features\": [{\"id\": \"pgstac-test-item-0003\", \"collection\": \"pgstac-test-collection\"}]}\n\nSELECT xyzsearch(1048, 1682, 12, '2bbae9a0ef0bbb5ffaca06603ce621d7', '{\"include\":[\"id\"]}'::jsonb);\n {\"type\": \"FeatureCollection\", \"features\": [{\"id\": \"pgstac-test-item-0050\", \"collection\": \"pgstac-test-collection\"}, {\"id\": \"pgstac-test-item-0049\", \"collection\": \"pgstac-test-collection\"}, {\"id\": \"pgstac-test-item-0048\", \"collection\": \"pgstac-test-collection\"}, {\"id\": \"pgstac-test-item-0047\", \"collection\": \"pgstac-test-collection\"}, {\"id\": \"pgstac-test-item-0100\", \"collection\": \"pgstac-test-collection\"}, {\"id\": \"pgstac-test-item-0089\", \"collection\": \"pgstac-test-collection\"}]}\n\nSELECT xyzsearch(1048, 1682, 12, '2bbae9a0ef0bbb5ffaca06603ce621d7', '{\"include\":[\"id\"]}'::jsonb, NULL, 1);\n {\"type\": \"FeatureCollection\", \"features\": [{\"id\": \"pgstac-test-item-0050\", \"collection\": \"pgstac-test-collection\"}]}\n\nSELECT xyzsearch(16792, 26892, 16, '2bbae9a0ef0bbb5ffaca06603ce621d7', '{\"include\":[\"id\"]}'::jsonb, exitwhenfull => true);\n {\"type\": \"FeatureCollection\", \"features\": [{\"id\": \"pgstac-test-item-0098\", \"collection\": \"pgstac-test-collection\"}, {\"id\": \"pgstac-test-item-0097\", \"collection\": \"pgstac-test-collection\"}]}\n\nSELECT xyzsearch(16792, 26892, 16, '2bbae9a0ef0bbb5ffaca06603ce621d7', '{\"include\":[\"id\"]}'::jsonb, exitwhenfull => false, skipcovered => false);\n {\"type\": \"FeatureCollection\", \"features\": [{\"id\": \"pgstac-test-item-0098\", \"collection\": \"pgstac-test-collection\"}, {\"id\": \"pgstac-test-item-0097\", \"collection\": \"pgstac-test-collection\"}, {\"id\": \"pgstac-test-item-0091\", \"collection\": \"pgstac-test-collection\"}]}\n\nSELECT geojsonsearch('{\"type\": \"Point\",\"coordinates\": [-87.75608539581299,30.692471153735646]}', '2bbae9a0ef0bbb5ffaca06603ce621d7', '{\"include\":[\"id\"]}'::jsonb, exitwhenfull => true, skipcovered => true);\n{\"type\": \"FeatureCollection\", \"features\": [{\"id\": \"pgstac-test-item-0097\", \"collection\": \"pgstac-test-collection\"}]}\n\nSELECT geojsonsearch('{\"type\": \"Point\",\"coordinates\": [-87.75608539581299,30.692471153735646]}', '2bbae9a0ef0bbb5ffaca06603ce621d7', '{\"include\":[\"id\"]}'::jsonb, exitwhenfull => false, skipcovered => false) s;\n{\"type\": \"FeatureCollection\", \"features\": [{\"id\": \"pgstac-test-item-0097\", \"collection\": \"pgstac-test-collection\"}]}\n" }, { "path": "src/pgstac/tests/pgtap/001_core.sql", "content": "-- Check that schema exists\nSELECT has_schema('pgstac'::name);\n\n-- Check that PostGIS extension are installed and available on the path\nSELECT has_extension('postgis');\n\nSELECT has_table('pgstac'::name, 'migrations'::name);\n\n\nSELECT has_function('pgstac'::name, 'to_text_array', ARRAY['jsonb']);\nSELECT results_eq(\n $$ SELECT to_text_array('[\"a\",\"b\",\"c\"]'::jsonb) $$,\n $$ SELECT '{a,b,c}'::text[] $$,\n 'to_text_array returns text[] from jsonb array'\n);\n\nSET pgstac.readonly to 'false';\n\nSELECT results_eq(\n $$ SELECT pgstac.readonly(); $$,\n $$ SELECT FALSE; $$,\n 'Readonly is set to false'\n);\n\nSELECT lives_ok(\n $$ SELECT search('{}'); $$,\n 'Search works with readonly mode set to off in readwrite mode.'\n);\n\nRESET pgstac.context;\nSELECT is_definer('update_partition_stats');\nSELECT is_definer('partition_after_triggerfunc');\nSELECT is_definer('drop_table_constraints');\nSELECT is_definer('create_table_constraints');\nSELECT is_definer('check_partition');\nSELECT is_definer('repartition');\nSELECT is_definer('where_stats');\nSELECT is_definer('search_query');\nSELECT is_definer('format_item');\nSELECT is_definer('maintain_index');\n" }, { "path": "src/pgstac/tests/pgtap/001a_jsonutils.sql", "content": "SELECT has_function('pgstac'::name, 'to_text_array', ARRAY['jsonb']);\n\nSELECT results_eq(\n $$ SELECT to_text_array('[\"a\",\"b\",\"c\"]'::jsonb) $$,\n $$ SELECT '{a,b,c}'::text[] $$,\n 'textarr returns text[] from jsonb array'\n);\n" }, { "path": "src/pgstac/tests/pgtap/001b_cursorutils.sql", "content": "" }, { "path": "src/pgstac/tests/pgtap/001s_stacutils.sql", "content": "SELECT has_function('pgstac'::name, 'stac_geom', ARRAY['jsonb']);\nSELECT has_function('pgstac'::name, 'stac_datetime', ARRAY['jsonb']);\nSELECT has_function('pgstac'::name, 'stac_end_datetime', ARRAY['jsonb']);\nSELECT has_function('pgstac'::name, 'stac_daterange', ARRAY['jsonb']);\n" }, { "path": "src/pgstac/tests/pgtap/002_collections.sql", "content": "SELECT has_table('pgstac'::name, 'collections'::name);\nSELECT col_is_pk('pgstac'::name, 'collections'::name, 'key', 'collections has primary key');\n\nSELECT has_function('pgstac'::name, 'create_collection', ARRAY['jsonb']);\nSELECT has_function('pgstac'::name, 'update_collection', ARRAY['jsonb']);\nSELECT has_function('pgstac'::name, 'upsert_collection', ARRAY['jsonb']);\nSELECT has_function('pgstac'::name, 'get_collection', ARRAY['text']);\nSELECT has_function('pgstac'::name, 'delete_collection', ARRAY['text']);\nSELECT has_function('pgstac'::name, 'all_collections', '{}'::text[]);\n\nDELETE FROM collections WHERE id in ('pgstac-test-collection', 'pgstac-test-collection2');\n\\copy collections (content) FROM 'tests/testdata/collections.ndjson';\n\\copy items_staging (content) FROM 'tests/testdata/items.ndjson';\n\nSELECT results_eq($$\n SELECT count(*) FROM partitions_view WHERE collection='pgstac-test-collection';\n $$,\n $$ SELECT 1::bigint $$,\n 'Test that partition metadata only has one record after adding items data for one collection'\n);\n\nSELECT results_eq($$\n SELECT count(*) FROM partitions_view WHERE collection='pgstac-test-collection2';\n $$,\n $$ SELECT 0::bigint $$,\n 'Test that partition metadata does not have collection 2 yet'\n);\n\nSELECT create_item('{\"id\": \"pgstac-test-item-0003\", \"bbox\": [-85.379245, 30.933949, -85.308201, 31.003555], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30085/m_3008506_nw_16_1_20110825.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30085/m_3008506_nw_16_1_20110825.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30085/m_3008506_nw_16_1_20110825.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-85.309412, 30.933949], [-85.308201, 31.002658], [-85.378084, 31.003555], [-85.379245, 30.934843], [-85.309412, 30.933949]]]}, \"collection\": \"pgstac-test-collection2\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-25T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [654842, 3423507, 661516, 3431125], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7618, 6674], \"eo:cloud_cover\": 28, \"proj:transform\": [1, 0, 654842, 0, -1, 3431125, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}');\n\n\nSELECT results_eq($$\n SELECT count(*) FROM partitions_view WHERE collection='pgstac-test-collection2';\n $$,\n $$ SELECT 1::bigint $$,\n 'Test that partition metadata only has record for collection 2'\n);\n\nDELETE FROM collections WHERE id='pgstac-test-collection2';\n\nSELECT results_eq($$\n SELECT count(*) FROM partition_sys_meta WHERE collection='pgstac-test-collection2';\n $$,\n $$ SELECT 0::bigint $$,\n 'Test that sys meta does not have for collection 2 record after removing collection 2'\n);\n\n\nDELETE FROM collections WHERE id='pgstac-test-collection';\n\nSELECT results_eq($$\n SELECT count(*) FROM partition_sys_meta WHERE collection='pgstac-test-collection';\n $$,\n $$ SELECT 0::bigint $$,\n 'Test that sys meta does not have for collection 1 record after removing collection 1'\n);\n\n\\copy collections (content) FROM 'tests/testdata/collections.ndjson';\n\\copy items_staging (content) FROM 'tests/testdata/items.ndjson';\n\nSELECT results_eq($$\n SELECT count(*) FROM partitions_view WHERE collection='pgstac-test-collection';\n $$,\n $$ SELECT 1::bigint $$,\n 'Test that partition metadata only has one record after adding items data for one collection'\n);\n\nSELECT results_eq($$\n SELECT count(*) FROM partitions_view WHERE collection='pgstac-test-collection2';\n $$,\n $$ SELECT 0::bigint $$,\n 'Test that partition metadata does not have collection 2 yet'\n);\n\nSELECT create_item('{\"id\": \"pgstac-test-item-0003\", \"bbox\": [-85.379245, 30.933949, -85.308201, 31.003555], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30085/m_3008506_nw_16_1_20110825.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30085/m_3008506_nw_16_1_20110825.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30085/m_3008506_nw_16_1_20110825.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-85.309412, 30.933949], [-85.308201, 31.002658], [-85.378084, 31.003555], [-85.379245, 30.934843], [-85.309412, 30.933949]]]}, \"collection\": \"pgstac-test-collection2\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-25T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [654842, 3423507, 661516, 3431125], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7618, 6674], \"eo:cloud_cover\": 28, \"proj:transform\": [1, 0, 654842, 0, -1, 3431125, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}');\n\n\nSELECT results_eq($$\n SELECT count(*) FROM partitions_view WHERE collection='pgstac-test-collection2';\n $$,\n $$ SELECT 1::bigint $$,\n 'Test that partition metadata only has record for collection 2'\n);\n" }, { "path": "src/pgstac/tests/pgtap/002a_queryables.sql", "content": "SELECT results_eq(\n $$ SELECT sort_sqlorderby('{\"sortby\":{\"field\":\"properties.eo:cloud_cover\"}}'); $$,\n $$ SELECT sort_sqlorderby('{\"sortby\":{\"field\":\"eo:cloud_cover\"}}'); $$,\n 'Make sure that sortby with/without properties prefix return the same sort statement.'\n);\n\nSET pgstac.\"default_filter_lang\" TO 'cql2-json';\n\nSELECT results_eq(\n $$ SELECT stac_search_to_where('{\"filter\":{\"op\":\"eq\",\"args\":[{\"property\":\"eo:cloud_cover\"},0]}}'); $$,\n $$ SELECT stac_search_to_where('{\"filter\":{\"op\":\"eq\",\"args\":[{\"property\":\"properties.eo:cloud_cover\"},0]}}'); $$,\n 'Make sure that CQL2 filter works the same with/without properties prefix.'\n);\n\nSET pgstac.\"default_filter_lang\" TO 'cql-json';\n\nSELECT results_eq(\n $$ SELECT stac_search_to_where('{\"filter\":{\"eq\":[{\"property\":\"eo:cloud_cover\"},0]}}'); $$,\n $$ SELECT stac_search_to_where('{\"filter\":{\"eq\":[{\"property\":\"properties.eo:cloud_cover\"},0]}}'); $$,\n 'Make sure that CQL filter works the same with/without properties prefix.'\n);\n\nDELETE FROM collections WHERE id in ('pgstac-test-collection', 'pgstac-test-collection2');\n\nSELECT results_eq(\n $$ SELECT all_collections(); $$,\n $$ SELECT '[]'::jsonb; $$,\n 'Make sure all_collections returns an empty array when the collection table is empty.'\n);\n\n\\copy collections (content) FROM 'tests/testdata/collections.ndjson';\n\nSELECT results_eq(\n $$ SELECT get_queryables('pgstac-test-collection') -> 'properties' ? 'datetime'; $$,\n $$ SELECT true; $$,\n 'Make sure valid schema object is returned for a existing collection.'\n);\n\nSELECT results_eq(\n $$ SELECT get_queryables('foo'); $$,\n $$ SELECT NULL::jsonb; $$,\n 'Make sure null is returned for a non-existant collection.'\n);\n\nSELECT lives_ok(\n $$ DELETE FROM queryables WHERE name IN ('testqueryable', 'testqueryable2', 'testqueryable3'); $$,\n 'Make sure test queryable does not exist.'\n);\n\nSELECT lives_ok(\n $$ INSERT INTO queryables (name, collection_ids) VALUES ('testqueryable', null); $$,\n 'Can add a new queryable that applies to all collections.'\n);\n\nselect is(\n (SELECT count(*) from collections where id = 'pgstac-test-collection'),\n '1',\n 'Make sure test collection exists.'\n);\n\nSELECT lives_ok(\n $$ INSERT INTO queryables (name, collection_ids) VALUES ('testqueryable3', '{pgstac-test-collection}'); $$,\n 'Can add a new queryable to a specific existing collection.'\n);\n\nSELECT throws_ok(\n $$ INSERT INTO queryables (name, collection_ids) VALUES ('testqueryable2', '{nonexistent}'); $$,\n '23503'\n);\n\nSELECT throws_ok(\n $$ INSERT INTO queryables (name, collection_ids) VALUES ('testqueryable', '{pgstac-test-collection}'); $$,\n '23505'\n);\n\nSELECT lives_ok(\n $$ UPDATE queryables SET collection_ids = '{pgstac-test-collection}' WHERE name='testqueryable'; $$,\n 'Can update a queryable from null to a single collection.'\n);\n\nSET pgstac.additional_properties to 'false';\n\nSELECT results_eq(\n $$ SELECT pgstac.additional_properties(); $$,\n $$ SELECT FALSE; $$,\n 'Make sure additional_properties is set to false'\n);\n\nSELECT throws_ok(\n $$ SELECT search('{\"filter\": {\"eq\": [{\"property\": \"xyzzy\"}, \"dummy\"]}}'); $$,\n 'Term xyzzy is not found in queryables.',\n 'Make sure a term not present in the list of queryables cannot be used in a filter'\n);\n\nSELECT lives_ok(\n $$ SELECT search('{\"filter\": {\"eq\": [{\"property\": \"datetime\"}, \"2020-11-11T00:00:00Z\"]}}'); $$,\n 'Make sure a term present in the list of queryables can be used in a filter'\n);\n\nSELECT lives_ok(\n $$ SELECT search('{\"filter\": {\"s_intersects\": [{\"property\": \"geometry\"}, {\"type\": \"Point\", \"coordinates\": [0, 0]}]}}'); $$,\n 'Make sure a term present in the list of queryables can be used in a filter'\n);\n\nSELECT lives_ok(\n $$ SELECT search('{\"filter\": {\"and\": [{\"t_after\": [{\"property\": \"datetime\"}, \"2020-11-11T00:00:00\"]}, {\"t_before\": [{\"property\": \"datetime\"}, \"2022-11-11T00:00:00\"]}]}}'); $$,\n 'Make sure that only arguments that are properties are checked'\n);\n\nSELECT throws_ok(\n $$ SELECT search('{\"filter\": {\"and\": [{\"t_after\": [{\"property\": \"datetime\"}, \"2020-11-11T00:00:00\"]}, {\"eq\": [{\"property\": \"xyzzy\"}, \"dummy\"]}]}}'); $$,\n 'Term xyzzy is not found in queryables.',\n 'Make sure a term not present in the list of queryables cannot be used in a filter with nested arguments'\n);\n\nSET pgstac.additional_properties to 'true';\n\nSELECT results_eq(\n $$ SELECT pgstac.additional_properties(); $$,\n $$ SELECT TRUE; $$,\n 'Make sure additional_properties is set to true'\n);\n\nSELECT lives_ok(\n $$ SELECT search('{\"filter\": {\"eq\": [{\"property\": \"xyzzy\"}, \"dummy\"]}}'); $$,\n 'Make sure a term not present in the list of queryables can be used in a filter'\n);\n\nSELECT lives_ok(\n $$ SELECT search('{\"filter\": {\"eq\": [{\"property\": \"datetime\"}, \"2020-11-11T00:00:00Z\"]}}'); $$,\n 'Make sure a term present in the list of queryables can be used in a filter'\n);\n\nRESET pgstac.additional_properties;\n" }, { "path": "src/pgstac/tests/pgtap/003_items.sql", "content": "SELECT has_table('pgstac'::name, 'items'::name);\n\n\nSELECT is_indexed('pgstac'::name, 'items'::name, 'geometry');\n\nSELECT is_partitioned('pgstac'::name,'items'::name);\n\n\nSELECT has_function('pgstac'::name, 'get_item', ARRAY['text','text']);\nSELECT has_function('pgstac'::name, 'delete_item', ARRAY['text','text']);\nSELECT has_function('pgstac'::name, 'create_item', ARRAY['jsonb']);\nSELECT has_function('pgstac'::name, 'update_item', ARRAY['jsonb']);\nSELECT has_function('pgstac'::name, 'upsert_item', ARRAY['jsonb']);\nSELECT has_function('pgstac'::name, 'create_items', ARRAY['jsonb']);\nSELECT has_function('pgstac'::name, 'upsert_items', ARRAY['jsonb']);\n\n\n-- tools to update collection extents based on extents in items\nSELECT has_function('pgstac'::name, 'collection_bbox', ARRAY['text']);\nSELECT has_function('pgstac'::name, 'collection_temporal_extent', ARRAY['text']);\nSELECT has_function('pgstac'::name, 'update_collection_extents', '{}'::text[]);\n\nDELETE FROM collections WHERE id in ('pgstac-test-collection', 'pgstac-test-collection2');\n\\copy collections (content) FROM 'tests/testdata/collections.ndjson';\n\nSELECT create_item('{\"id\": \"pgstac-test-item-0003\", \"bbox\": [-85.379245, 30.933949, -85.308201, 31.003555], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30085/m_3008506_nw_16_1_20110825.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30085/m_3008506_nw_16_1_20110825.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30085/m_3008506_nw_16_1_20110825.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-85.309412, 30.933949], [-85.308201, 31.002658], [-85.378084, 31.003555], [-85.379245, 30.934843], [-85.309412, 30.933949]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-25T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [654842, 3423507, 661516, 3431125], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7618, 6674], \"eo:cloud_cover\": 28, \"proj:transform\": [1, 0, 654842, 0, -1, 3431125, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}');\n\nSELECT results_eq($$\n SELECT content->'properties'->>'eo:cloud_cover' FROM items WHERE collection='pgstac-test-collection';\n $$,$$\n SELECT '28';\n $$,\n 'Test create_item function'\n);\n\nSELECT update_item('{\"id\": \"pgstac-test-item-0003\", \"bbox\": [-85.379245, 30.933949, -85.308201, 31.003555], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30085/m_3008506_nw_16_1_20110825.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30085/m_3008506_nw_16_1_20110825.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30085/m_3008506_nw_16_1_20110825.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-85.309412, 30.933949], [-85.308201, 31.002658], [-85.378084, 31.003555], [-85.379245, 30.934843], [-85.309412, 30.933949]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-25T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [654842, 3423507, 661516, 3431125], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7618, 6674], \"eo:cloud_cover\": 29, \"proj:transform\": [1, 0, 654842, 0, -1, 3431125, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}');\n\nSELECT results_eq($$\n SELECT content->'properties'->>'eo:cloud_cover' FROM items WHERE collection='pgstac-test-collection';\n $$,$$\n SELECT '29';\n $$,\n 'Test update_item function'\n);\n\nselect delete_item('pgstac-test-item-0003');\n\nSELECT results_eq($$\n SELECT count(*) FROM items WHERE collection='pgstac-test-collection';\n $$,$$\n SELECT 0::bigint;\n $$,\n 'Test delete_item function'\n);\n" }, { "path": "src/pgstac/tests/pgtap/004_search.sql", "content": "-- CREATE fixtures for testing search - as tests are run within a transaction, these will not persist\n\n\\copy items_staging (content) FROM 'tests/testdata/items.ndjson'\n\nSET pgstac.context TO 'on';\nSET pgstac.\"default_filter_lang\" TO 'cql-json';\n\nSELECT has_function('pgstac'::name, 'parse_dtrange', ARRAY['jsonb','timestamptz']);\n\n\nSELECT results_eq($$ SELECT parse_dtrange('[\"2020-01-01\",\"2021-01-01\"]'::jsonb) $$, $$ SELECT '[\"2020-01-01 00:00:00+00\",\"2021-01-01 00:00:00+00\")'::tstzrange $$, 'daterange passed as array range');\n\n\nSELECT results_eq($$ SELECT parse_dtrange('\"2020-01-01/2021-01-01\"'::jsonb) $$, $$ SELECT '[\"2020-01-01 00:00:00+00\",\"2021-01-01 00:00:00+00\")'::tstzrange $$, 'date range passed as string range');\n\n\nSELECT results_eq($$ SELECT parse_dtrange('\"2020-01-01/..\"'::jsonb) $$, $$ SELECT '[\"2020-01-01 00:00:00+00\",infinity)'::tstzrange $$, 'date range passed as string range');\n\n\nSELECT results_eq($$ SELECT parse_dtrange('\"2020-01-01/\"'::jsonb) $$, $$ SELECT '[\"2020-01-01 00:00:00+00\",infinity)'::tstzrange $$, 'date range passed as string range');\n\n\nSELECT results_eq($$ SELECT parse_dtrange('\"../2020-01-01\"'::jsonb) $$, $$ SELECT '[-infinity,\"2020-01-01 00:00:00+00\")'::tstzrange $$, 'date range passed as string range');\n\n\nSELECT results_eq($$ SELECT parse_dtrange('\"/2020-01-01\"'::jsonb) $$, $$ SELECT '[-infinity,\"2020-01-01 00:00:00+00\")'::tstzrange $$, 'date range passed as string range');\n\n\nSELECT has_function('pgstac'::name, 'bbox_geom', ARRAY['jsonb']);\n\n\nSELECT results_eq($$ SELECT bbox_geom('[0,1,2,3]') $$, $$ SELECT 'SRID=4326;POLYGON((0 1,0 3,2 3,2 1,0 1))'::geometry $$, '2d bbox');\n\n\nSELECT results_eq($$ SELECT bbox_geom('[0,1,2,3,4,5]'::jsonb) $$, $$ SELECT '010F0000A0E610000006000000010300008001000000050000000000000000000000000000000000F03F00000000000000400000000000000000000000000000104000000000000000400000000000000840000000000000104000000000000000400000000000000840000000000000F03F00000000000000400000000000000000000000000000F03F0000000000000040010300008001000000050000000000000000000000000000000000F03F00000000000014400000000000000840000000000000F03F00000000000014400000000000000840000000000000104000000000000014400000000000000000000000000000104000000000000014400000000000000000000000000000F03F0000000000001440010300008001000000050000000000000000000000000000000000F03F00000000000000400000000000000000000000000000F03F00000000000014400000000000000000000000000000104000000000000014400000000000000000000000000000104000000000000000400000000000000000000000000000F03F0000000000000040010300008001000000050000000000000000000840000000000000F03F00000000000000400000000000000840000000000000104000000000000000400000000000000840000000000000104000000000000014400000000000000840000000000000F03F00000000000014400000000000000840000000000000F03F0000000000000040010300008001000000050000000000000000000000000000000000F03F00000000000000400000000000000840000000000000F03F00000000000000400000000000000840000000000000F03F00000000000014400000000000000000000000000000F03F00000000000014400000000000000000000000000000F03F000000000000004001030000800100000005000000000000000000000000000000000010400000000000000040000000000000000000000000000010400000000000001440000000000000084000000000000010400000000000001440000000000000084000000000000010400000000000000040000000000000000000000000000010400000000000000040'::geometry $$, '3d bbox');\n\n\n\nSELECT has_function('pgstac'::name, 'sort_sqlorderby', ARRAY['jsonb','boolean']);\n\nSELECT results_eq($$\n SELECT sort_sqlorderby('{\"sortby\":[{\"field\":\"datetime\",\"direction\":\"desc\"},{\"field\":\"eo:cloud_cover\",\"direction\":\"asc\"}]}'::jsonb);\n $$,$$\n SELECT 'datetime DESC, to_int(content->''properties''->''eo:cloud_cover'') ASC, id DESC';\n $$,\n 'Test creation of sort sql'\n);\n\n\nSELECT results_eq($$\n SELECT sort_sqlorderby('{\"sortby\":[{\"field\":\"datetime\",\"direction\":\"desc\"},{\"field\":\"eo:cloud_cover\",\"direction\":\"asc\"}]}'::jsonb, true);\n $$,$$\n SELECT 'datetime ASC, to_int(content->''properties''->''eo:cloud_cover'') DESC, id ASC';\n $$,\n 'Test creation of reverse sort sql'\n);\n\n\nSELECT has_function('pgstac'::name, 'search', ARRAY['jsonb']);\n\n\nSELECT results_eq($$\n SELECT search('{\"collections\": [\"pgstac-test-collection\"], \"limit\": 10, \"sortby\":[{\"field\":\"id\",\"direction\":\"asc\"}], \"token\": \"prev:pgstac-test-item-0011\"}')\n $$,$$\n SELECT search('{\"collections\": [\"pgstac-test-collection\"], \"limit\": 10, \"sortby\":[{\"field\":\"id\",\"direction\":\"asc\"}]}')\n $$,\n 'Test prev token when reading first token_type=prev (https://github.com/stac-utils/pgstac/issues/140)'\n);\n\n\nSELECT has_function('pgstac'::name, 'search_query', ARRAY['jsonb','boolean','jsonb']);\n\n\nSELECT results_eq($$\n SELECT BTRIM(stac_search_to_where($q$\n {\n \"intersects\":\n {\n \"type\": \"Polygon\",\n \"coordinates\": [[\n [-77.0824, 38.7886], [-77.0189, 38.7886],\n [-77.0189, 38.8351], [-77.0824, 38.8351],\n [-77.0824, 38.7886]\n ]]\n }\n }\n $q$),E' \\n');\n $$, $$\n SELECT BTRIM($r$\n st_intersects(geometry, '0103000020E61000000100000005000000304CA60A464553C014D044D8F06443403E7958A8354153C014D044D8F06443403E7958A8354153C0DE718A8EE46A4340304CA60A464553C0DE718A8EE46A4340304CA60A464553C014D044D8F0644340')\n $r$,E' \\n');\n $$, 'Make sure that intersects returns valid query'\n);\n\n-- CQL 2 Tests from examples at https://github.com/radiantearth/stac-api-spec/blob/f5da775080ff3ff46d454c2888b6e796ee956faf/fragments/filter/README.md\n\nSET pgstac.\"default_filter_lang\" TO 'cql2-json';\n\nSELECT results_eq($$\n SELECT BTRIM(stac_search_to_where($q$\n {\n \"filter\": {\n \"op\" : \"and\",\n \"args\": [\n {\n \"op\": \"=\",\n \"args\": [ { \"property\": \"id\" }, \"LC08_L1TP_060247_20180905_20180912_01_T1_L1TP\" ]\n },\n {\n \"op\": \"=\",\n \"args\" : [ { \"property\": \"collection\" }, \"landsat8_l1tp\" ]\n }\n ]\n }\n }\n $q$),E' \\n');\n $$, $$\n SELECT BTRIM($r$\n (id = 'LC08_L1TP_060247_20180905_20180912_01_T1_L1TP' AND collection = 'landsat8_l1tp')\n $r$,E' \\n');\n $$, 'Test Example 1'\n);\n\n\nSELECT results_eq($$\n SELECT BTRIM(stac_search_to_where($q$\n {\n \"filter-lang\": \"cql2-json\",\n \"filter\": {\n \"op\": \"and\",\n \"args\": [\n {\n \"op\": \"=\",\n \"args\": [ { \"property\": \"collection\" }, \"landsat8_l1tp\" ]\n },\n {\n \"op\": \"<=\",\n \"args\": [ { \"property\": \"eo:cloud_cover\" }, \"10\" ]\n },\n {\n \"op\": \">=\",\n \"args\": [ { \"property\": \"datetime\" }, {\"timestamp\": \"2021-04-08T04:39:23Z\"} ]\n },\n {\n \"op\": \"s_intersects\",\n \"args\": [\n {\n \"property\": \"geometry\"\n },\n {\n \"type\": \"Polygon\",\n \"coordinates\": [\n [\n [43.5845, -79.5442],\n [43.6079, -79.4893],\n [43.5677, -79.4632],\n [43.6129, -79.3925],\n [43.6223, -79.3238],\n [43.6576, -79.3163],\n [43.7945, -79.1178],\n [43.8144, -79.1542],\n [43.8555, -79.1714],\n [43.7509, -79.6390],\n [43.5845, -79.5442]\n ]\n ]\n }\n ]\n }\n ]\n }\n }\n $q$),E' \\n');\n $$, $$\n SELECT BTRIM($r$\n (collection = 'landsat8_l1tp' AND to_int(content->'properties'->'eo:cloud_cover') <= to_int('\"10\"') AND datetime >= '2021-04-08 04:39:23+00'::timestamptz AND st_intersects(geometry, '0103000020E6100000010000000B000000894160E5D0CA4540ED9E3C2CD4E253C0849ECDAACFCD4540B37BF2B050DF53C038F8C264AAC8454076E09C11A5DD53C0F5DBD78173CE454085EB51B81ED953C08126C286A7CF4540789CA223B9D453C0C0EC9E3C2CD4454063EE5A423ED453C004560E2DB2E5454001DE02098AC753C063EE5A423EE84540C442AD69DEC953C02FDD240681ED454034A2B437F8CA53C08048BF7D1DE0454037894160E5E853C0894160E5D0CA4540ED9E3C2CD4E253C0'::geometry))\n $r$,E' \\n');\n $$, 'Test Example 2'\n);\n\n\nSELECT results_eq($$\n SELECT BTRIM(stac_search_to_where($q$\n {\n \"filter-lang\": \"cql2-json\",\n \"filter\": {\n \"op\": \"and\",\n \"args\": [\n {\n \"op\": \">\",\n \"args\": [ { \"property\": \"sentinel:data_coverage\" }, \"50\" ]\n },\n {\n \"op\": \"<\",\n \"args\": [ { \"property\": \"eo:cloud_cover\" }, 10 ]\n }\n ]\n }\n }\n $q$),E' \\n');\n $$, $$\n SELECT BTRIM($r$\n (to_text(content->'properties'->'sentinel:data_coverage') > to_text('\"50\"') AND to_int(content->'properties'->'eo:cloud_cover') < to_int('10'))\n $r$,E' \\n');\n $$, 'Test Example 3'\n);\n\n\n\nSELECT results_eq($$\n SELECT BTRIM(stac_search_to_where($q$\n {\n \"filter-lang\": \"cql2-json\",\n \"filter\": {\n \"op\": \"or\",\n \"args\": [\n {\n \"op\": \">\",\n \"args\": [ { \"property\": \"sentinel:data_coverage\" }, 50 ]\n },\n {\n \"op\": \"<\",\n \"args\": [ { \"property\": \"eo:cloud_cover\" }, 10 ]\n }\n ]\n }\n }\n $q$),E' \\n');\n $$, $$\n SELECT BTRIM($r$\n (to_float(content->'properties'->'sentinel:data_coverage') > to_float('50') OR to_int(content->'properties'->'eo:cloud_cover') < to_int('10'))\n $r$,E' \\n');\n $$, 'Test Example 4'\n);\n\n\n\nSELECT results_eq($$\n SELECT BTRIM(stac_search_to_where($q$\n {\n \"filter-lang\": \"cql2-json\",\n \"filter\": {\n \"op\": \"eq\",\n \"args\": [\n { \"property\": \"prop1\" },\n { \"property\": \"prop2\" }\n ]\n }\n }\n $q$),E' \\n');\n $$, $$\n SELECT BTRIM($r$\n to_text(content->'properties'->'prop1') = to_text(content->'properties'->'prop2')\n $r$,E' \\n');\n $$, 'Test Example 5'\n);\n\n\nSELECT results_eq($$\n SELECT BTRIM(stac_search_to_where($q$\n {\n \"filter-lang\": \"cql2-json\",\n \"filter\": {\n \"op\": \"t_intersects\",\n \"args\": [\n { \"property\": \"datetime\" },\n { \"interval\": [ \"2020-11-11T00:00:00Z\", \"2020-11-12T00:00:00Z\"] }\n ]\n }\n }\n $q$),E' \\n');\n $$, $$\n SELECT BTRIM($r$\n (datetime <= '2020-11-12 00:00:00+00'::timestamptz AND end_datetime >= '2020-11-11 00:00:00+00'::timestamptz)\n $r$,E' \\n');\n $$, 'Test Example 6'\n);\n\n\n\nSELECT results_eq($$\n SELECT BTRIM(stac_search_to_where($q$\n {\n \"filter-lang\": \"cql2-json\",\n \"filter\": {\n \"op\": \"s_intersects\",\n \"args\": [\n { \"property\": \"geometry\" } ,\n {\n \"type\": \"Polygon\",\n \"coordinates\": [[\n [-77.0824, 38.7886], [-77.0189, 38.7886],\n [-77.0189, 38.8351], [-77.0824, 38.8351],\n [-77.0824, 38.7886]\n ]]\n }\n ]\n }\n }\n $q$),E' \\n');\n $$, $$\n SELECT BTRIM($r$\n st_intersects(geometry, '0103000020E61000000100000005000000304CA60A464553C014D044D8F06443403E7958A8354153C014D044D8F06443403E7958A8354153C0DE718A8EE46A4340304CA60A464553C0DE718A8EE46A4340304CA60A464553C014D044D8F0644340'::geometry)\n $r$,E' \\n');\n $$, 'Test Example 7'\n);\n\n\n\nSELECT results_eq($$\n SELECT BTRIM(stac_search_to_where($q$\n {\n \"filter\": {\n \"op\": \"or\" ,\n \"args\": [\n {\n \"op\": \"s_intersects\",\n \"args\": [\n { \"property\": \"geometry\" } ,\n {\n \"type\": \"Polygon\",\n \"coordinates\": [[\n [-77.0824, 38.7886], [-77.0189, 38.7886],\n [-77.0189, 38.8351], [-77.0824, 38.8351],\n [-77.0824, 38.7886]\n ]]\n }\n ]\n },\n {\n \"op\": \"s_intersects\",\n \"args\": [\n { \"property\": \"geometry\" } ,\n {\n \"type\": \"Polygon\",\n \"coordinates\": [[\n [-79.0935, 38.7886], [-79.0290, 38.7886],\n [-79.0290, 38.8351], [-79.0935, 38.8351],\n [-79.0935, 38.7886]\n ]]\n }\n ]\n }\n ]\n }\n }\n $q$),E' \\n');\n $$, $$\n SELECT BTRIM($r$\n (st_intersects(geometry, '0103000020E61000000100000005000000304CA60A464553C014D044D8F06443403E7958A8354153C014D044D8F06443403E7958A8354153C0DE718A8EE46A4340304CA60A464553C0DE718A8EE46A4340304CA60A464553C014D044D8F0644340'::geometry) OR st_intersects(geometry, '0103000020E61000000100000005000000448B6CE7FBC553C014D044D8F064434060E5D022DBC153C014D044D8F064434060E5D022DBC153C0DE718A8EE46A4340448B6CE7FBC553C0DE718A8EE46A4340448B6CE7FBC553C014D044D8F0644340'::geometry))\n $r$,E' \\n');\n $$, 'Test Example 8'\n);\n\n\nSELECT results_eq($$\n SELECT BTRIM(stac_search_to_where($q$\n {\n \"filter-lang\": \"cql2-json\",\n \"filter\": {\n \"op\": \"or\",\n \"args\": [\n {\n \"op\": \">=\",\n \"args\": [ { \"property\": \"sentinel:data_coverage\" }, 50 ]\n },\n {\n \"op\": \">=\",\n \"args\": [ { \"property\": \"landsat:coverage_percent\" }, 50 ]\n },\n {\n \"op\": \"and\",\n \"args\": [\n {\n \"op\": \"isNull\",\n \"args\": { \"property\": \"sentinel:data_coverage\" }\n },\n {\n \"op\": \"isNull\",\n \"args\": { \"property\": \"landsat:coverage_percent\" }\n }\n ]\n }\n ]\n }\n }\n $q$),E' \\n');\n $$, $$\n SELECT BTRIM($r$\n (to_float(content->'properties'->'sentinel:data_coverage') >= to_float('50') OR to_float(content->'properties'->'landsat:coverage_percent') >= to_float('50') OR (to_text(content->'properties'->'sentinel:data_coverage') IS NULL AND to_text(content->'properties'->'landsat:coverage_percent') IS NULL))\n $r$,E' \\n');\n $$, 'Test Example 9'\n);\n\n\nSELECT results_eq($$\n SELECT BTRIM(stac_search_to_where($q$\n {\n \"filter-lang\": \"cql2-json\",\n \"filter\": {\n \"op\": \"between\",\n \"args\": [\n { \"property\": \"eo:cloud_cover\" },\n 0, 50\n ]\n }\n }\n $q$),E' \\n');\n $$, $$\n SELECT BTRIM($r$\n to_int(content->'properties'->'eo:cloud_cover') BETWEEN to_int('0') AND to_int('50')\n $r$,E' \\n');\n $$, 'Test Example 10'\n);\n\n\nSELECT results_eq($$\n SELECT BTRIM(stac_search_to_where($q$\n {\n \"filter-lang\": \"cql2-json\",\n \"filter\": {\n \"op\": \"like\",\n \"args\": [\n { \"property\": \"mission\" },\n \"sentinel%\"\n ]\n }\n }\n $q$),E' \\n');\n $$, $$\n SELECT BTRIM($r$\n to_text(content->'properties'->'mission') LIKE to_text('\"sentinel%\"')\n $r$,E' \\n');\n $$, 'Test Example 11'\n);\n\nSELECT results_eq($$\n SELECT BTRIM(stac_search_to_where($q$\n {\n \"filter-lang\": \"cql2-json\",\n \"filter\": {\n \"op\": \"eq\",\n \"args\": [\n {\"upper\": { \"property\": \"mission\" }},\n {\"upper\": \"sentinel\"}\n ]\n }\n }\n $q$),E' \\n');\n $$, $$\n SELECT BTRIM($r$\n upper(to_text(content->'properties'->'mission')) = upper(to_text('\"sentinel\"'))\n $r$,E' \\n');\n $$, 'Test upper'\n);\n\nSELECT results_eq($$\n SELECT BTRIM(stac_search_to_where($q$\n {\n \"filter-lang\": \"cql2-json\",\n \"filter\": {\n \"op\": \"eq\",\n \"args\": [\n {\"lower\": { \"property\": \"mission\" }},\n {\"lower\": \"sentinel\"}\n ]\n }\n }\n $q$),E' \\n');\n $$, $$\n SELECT BTRIM($r$\n lower(to_text(content->'properties'->'mission')) = lower(to_text('\"sentinel\"'))\n $r$,E' \\n');\n $$, 'Test lower'\n);\n\nSELECT results_eq($$\n SELECT BTRIM(stac_search_to_where($q$\n {\n \"filter-lang\": \"cql2-json\",\n \"filter\": {\n \"op\": \"eq\",\n \"args\": [\n {\"op\": \"casei\", \"args\":[{ \"property\": \"mission\" }]},\n {\"op\": \"casei\", \"args\":[\"sentinel\"]}\n ]\n }\n }\n $q$),E' \\n');\n $$, $$\n SELECT BTRIM($r$\n upper(to_text(content->'properties'->'mission')) = upper(to_text('\"sentinel\"'))\n $r$,E' \\n');\n $$, 'Test casei'\n);\n\nSELECT results_eq($$\n SELECT BTRIM(stac_search_to_where($q$\n {\n \"filter-lang\": \"cql2-json\",\n \"filter\": {\n \"op\": \"eq\",\n \"args\": [\n {\"op\": \"accenti\", \"args\":[{ \"property\": \"mission\" }]},\n {\"op\": \"accenti\", \"args\":[\"sentinel\"]}\n ]\n }\n }\n $q$),E' \\n');\n $$, $$\n SELECT BTRIM($r$\n unaccent(to_text(content->'properties'->'mission')) = unaccent(to_text('\"sentinel\"'))\n $r$,E' \\n');\n $$, 'Test accenti'\n);\n\n\n\n/* template\nSELECT results_eq($$\n\n $$,$$\n\n $$,\n 'Test that ...'\n);\n*/\n\nCREATE OR REPLACE FUNCTION pg_temp.isnull(j jsonb) RETURNS boolean AS $$\n SELECT nullif(j, 'null'::jsonb) IS NULL;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION pg_temp.isnull(t text) RETURNS boolean AS $$\n SELECT t IS NULL;\n$$ LANGUAGE SQL IMMUTABLE PARALLEL SAFE;\n\nCREATE OR REPLACE FUNCTION pg_temp.prev(j jsonb) RETURNS text AS $$\n SELECT split_part(jsonb_path_query_first(j, '$.links[*] ? (@.rel == \"prev\") .href')->>0, 'token=', 2);\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION pg_temp.next(j jsonb) RETURNS text AS $$\n SELECT split_part(jsonb_path_query_first(j, '$.links[*] ? (@.rel == \"next\") .href')->>0, 'token=', 2);\n$$ LANGUAGE SQL IMMUTABLE STRICT;\n\nCREATE OR REPLACE FUNCTION pg_temp.testpaging(testsortdir text, iddir text) RETURNS SETOF TEXT LANGUAGE plpgsql AS $$\nDECLARE\n searchfilter jsonb;\n searchresult jsonb;\n offsetids text;\n searchresultids text;\n page int := 0;\n token text;\nBEGIN\n RAISE NOTICE 'Testing % %', testsortdir, iddir;\n -- Create collection with items that have a field with nulls and duplicate values\n DELETE FROM items WHERE collection = 'pgstac-test-collection2';\n\n INSERT INTO collections (content) VALUES ('{\"id\":\"pgstac-test-collection2\"}'::jsonb) ON CONFLICT DO NOTHING;\n PERFORM check_partition('pgstac-test-collection2', '[2011-01-01,2012-01-01)', '[2011-01-01,2012-01-01)');\n\n INSERT INTO items (id, collection, datetime, end_datetime, geometry, content)\n SELECT concat(id, '_2'), 'pgstac-test-collection2', datetime, end_datetime, geometry, content FROM items WHERE collection='pgstac-test-collection';\n\n UPDATE items SET content = '{\"properties\":{\"testsort\":1}}'::jsonb\n WHERE collection = 'pgstac-test-collection2' AND\n id <= 'pgstac-test-item-0005_2';\n UPDATE items SET content = '{\"properties\":{\"testsort\":2}}'::jsonb\n WHERE collection = 'pgstac-test-collection2' AND\n id > 'pgstac-test-item-0005_2' and id <= 'pgstac-test-item-0010_2';\n UPDATE items SET content = '{\"properties\":{\"testsort\":3}}'::jsonb\n WHERE collection = 'pgstac-test-collection2' AND\n id > 'pgstac-test-item-0010' and id <= 'pgstac-test-item-0015_2';\n\n RETURN NEXT results_eq(\n $q$\n SELECT count(*) FROM items WHERE collection = 'pgstac-test-collection2';\n $q$, $q$\n SELECT 100::bigint;\n $q$,\n 'pgstac-test-collection2 has 100 items'\n );\n\n searchfilter := '{\"collections\":[\"pgstac-test-collection2\"],\"fields\":{\"include\":[\"id\",\"properties.datetime\",\"properties.testsort\"]},\"sortby\":[{\"field\":\"testsort\",\"direction\":null},{\"field\":\"id\",\"direction\":null}]}'::jsonb;\n\n searchfilter := jsonb_set(searchfilter, '{sortby,0,direction}'::text[], to_jsonb(testsortdir));\n searchfilter := jsonb_set(searchfilter, '{sortby,1,direction}'::text[], to_jsonb(iddir));\n\n RAISE NOTICE 'SORTBY: %', searchfilter->>'sortby';\n\n searchresult := search(searchfilter);\n\n RETURN NEXT ok(pg_temp.isnull(pg_temp.prev(searchresult)), 'first prev is null');\n\n -- page up\n WHILE page <= 100 LOOP\n EXECUTE format($q$\n WITH t AS (\n SELECT id\n FROM items\n WHERE collection='pgstac-test-collection2'\n ORDER BY content->'properties'->>'testsort' %s, id %s\n OFFSET %L LIMIT 10\n ) SELECT string_agg(id, ',') FROM t\n $q$,\n testsortdir,\n iddir,\n page\n ) INTO offsetids;\n EXECUTE format($q$\n SELECT string_agg(q->>0, ',') FROM jsonb_path_query(%L, '$.features[*].id') as q;\n $q$, searchresult) INTO searchresultids;\n RAISE NOTICE 'O: %', offsetids;\n RAISE NOTICE 'S: %', searchresultids;\n RETURN NEXT results_eq(\n format($q$\n SELECT id\n FROM items\n WHERE collection='pgstac-test-collection2'\n ORDER BY content->'properties'->>'testsort' %s, id %s\n OFFSET %L LIMIT 10\n $q$,\n testsortdir,\n iddir,\n page\n ),\n format($q$\n SELECT q->>0 FROM jsonb_path_query(%L, '$.features[*].id') as q;\n $q$, searchresult),\n format('Going up %s/%s page:%s results match using offset', testsortdir, iddir, page)\n );\n\n IF pg_temp.isnull(pg_temp.next(searchresult)) THEN\n EXIT;\n END IF;\n searchfilter := searchfilter || jsonb_build_object('token', pg_temp.next(searchresult));\n RAISE NOTICE 'SEARCHFILTER: %', searchfilter;\n searchresult := search(searchfilter);\n RAISE NOTICE 'SEARCHRESULT: %', searchresult;\n RAISE NOTICE 'PAGE:% TOKEN:% LINKS:%', page, searchfilter->>'token', searchresult->'links';\n page := page + 10;\n END LOOP;\n\n RETURN NEXT ok(pg_temp.isnull(pg_temp.next(searchresult)), 'last next is null');\n RETURN NEXT ok(page=90, 'last page going up is 90');\n -- page down\n WHILE page >= 0 LOOP\n IF page < 10 THEN\n EXIT;\n END IF;\n page := page - 10;\n searchfilter := searchfilter || jsonb_build_object('token', pg_temp.prev(searchresult));\n RAISE NOTICE 'SEARCHFILTER: %', searchfilter;\n searchresult := search(searchfilter);\n RAISE NOTICE 'SEARCHRESULT: %', searchresult;\n RAISE NOTICE 'PAGE:% TOKEN:% LINKS:%', page, searchfilter->>'token', searchresult->>'links';\n EXECUTE format($q$\n WITH t AS (\n SELECT id\n FROM items\n WHERE collection='pgstac-test-collection2'\n ORDER BY content->'properties'->>'testsort' %s, id %s\n OFFSET %L LIMIT 10\n ) SELECT string_agg(id, ',') FROM t\n $q$,\n testsortdir,\n iddir,\n page\n ) INTO offsetids;\n EXECUTE format($q$\n SELECT string_agg(q->>0, ',') FROM jsonb_path_query(%L, '$.features[*].id') as q;\n $q$, searchresult) INTO searchresultids;\n RAISE NOTICE 'O: %', offsetids;\n RAISE NOTICE 'S: %', searchresultids;\n RETURN NEXT results_eq(\n format($q$\n SELECT id\n FROM items\n WHERE collection='pgstac-test-collection2'\n ORDER BY content->'properties'->>'testsort' %s, id %s\n OFFSET %L LIMIT 10\n $q$,\n testsortdir,\n iddir,\n page\n ),\n format($q$\n SELECT q->>0 FROM jsonb_path_query(%L, '$.features[*].id') as q;\n $q$, searchresult),\n format('Going down %s/%s page:%s results match using offset', testsortdir, iddir, page)\n );\n\n IF pg_temp.isnull(pg_temp.prev(searchresult)) THEN\n EXIT;\n END IF;\n END LOOP;\n RETURN NEXT ok(pg_temp.isnull(pg_temp.prev(searchresult)), 'last prev is null');\n RETURN NEXT ok(page=0, 'last page going down is 0');\nEND;\n$$;\n\nSELECT * FROM pg_temp.testpaging('asc','asc');\nSELECT * FROM pg_temp.testpaging('asc','desc');\nSELECT * FROM pg_temp.testpaging('desc','desc');\nSELECT * FROM pg_temp.testpaging('desc','asc');\n\n\\copy items_staging (content) FROM 'tests/testdata/items_duplicate_ids.ndjson'\n\nSELECT is(\n (SELECT jsonb_array_length(search('{\"ids\": [\"pgstac-test-item-duplicated\"]}')->'features')),\n '2',\n 'Make sure all matching items are returned when items with the same ID are in multiple collections, no collections specified. #192'\n);\n\nSELECT is(\n (SELECT jsonb_array_length(search('{\"ids\": [\"pgstac-test-item-duplicated\"], \"collections\": [\"pgstac-test-collection\"]}')->'features')),\n '1',\n 'Make sure all matching items are returned when items with the same ID are in multiple collections, some collections specified. #192'\n);\n\nSELECT is(\n (SELECT jsonb_array_length(search('{\"ids\": [\"pgstac-test-item-duplicated\"], \"collections\": [\"pgstac-test-collection\", \"pgstac-test-collection2\"]}')->'features')),\n '2',\n 'Make sure all matching items are returned when items with the same ID are in multiple collections, all collections specified. #192'\n);\n" }, { "path": "src/pgstac/tests/pgtap/004a_collectionsearch.sql", "content": "-- CREATE fixtures for testing search - as tests are run within a transaction, these will not persist\n\nSET pgstac.context TO 'on';\nSET pgstac.\"default_filter_lang\" TO 'cql2-json';\n\nWITH t AS (\n SELECT\n row_number() over () as id,\n x,\n y\n FROM\n generate_series(-180, 170, 10) as x,\n generate_series(-90, 80, 10) as y\n), t1 AS (\n SELECT\n concat('testcollection_', id) as id,\n x as minx,\n y as miny,\n x+10 as maxx,\n y+10 as maxy,\n '2000-01-01'::timestamptz + (concat(id, ' weeks'))::interval as sdt,\n '2000-01-01'::timestamptz + (concat(id, ' weeks'))::interval + ('2 months')::interval as edt\n FROM t\n)\nSELECT\n create_collection(format($q$\n {\n \"id\": \"%s\",\n \"type\": \"Collection\",\n \"title\": \"My Test Collection.\",\n \"description\": \"Description of my test collection.\",\n \"extent\": {\n \"spatial\": {\"bbox\": [[%s, %s, %s, %s]]},\n \"temporal\": {\"interval\": [[%I, %I]]}\n },\n \"stac_extensions\":[]\n }\n $q$,\n id, minx, miny, maxx, maxy, sdt, edt\n )::jsonb)\nFROM t1;\n\nSELECT has_function('pgstac'::name, 'collection_search', ARRAY['jsonb']);\n\n\nSELECT results_eq($$\n select collection_search('{\"ids\":[\"testcollection_1\",\"testcollection_2\"],\"limit\":1, \"sortby\":[{\"field\":\"id\",\"direction\":\"asc\"}]}');\n $$,$$\n SELECT '{\"links\": [{\"rel\": \"next\", \"body\": {\"offset\": 1}, \"href\": \"./collections\", \"type\": \"application/json\", \"merge\": true, \"method\": \"GET\"}], \"numberMatched\": 2, \"numberReturned\": 1, \"collections\": [{\"id\": \"testcollection_1\", \"type\": \"Collection\", \"title\": \"My Test Collection.\", \"extent\": {\"spatial\": {\"bbox\": [[-180, -90, -170, -80]]}, \"temporal\": {\"interval\": [[\"2000-01-08 00:00:00+00\", \"2000-03-08 00:00:00+00\"]]}}, \"description\": \"Description of my test collection.\", \"stac_extensions\": []}]}'::jsonb\n $$,\n 'Test search passing in collection ids'\n);\n\nSELECT results_eq($$\n select collection_search('{\"ids\":[\"testcollection_1\",\"testcollection_2\"],\"limit\":1, \"sortby\":[{\"field\":\"id\",\"direction\":\"desc\"}]}');\n $$,$$\n SELECT '{\"links\": [{\"rel\": \"next\", \"body\": {\"offset\": 1}, \"href\": \"./collections\", \"type\": \"application/json\", \"merge\": true, \"method\": \"GET\"}], \"numberMatched\": 2, \"numberReturned\": 1, \"collections\": [{\"id\": \"testcollection_2\", \"type\": \"Collection\", \"title\": \"My Test Collection.\", \"extent\": {\"spatial\": {\"bbox\": [[-170, -90, -160, -80]]}, \"temporal\": {\"interval\": [[\"2000-01-15 00:00:00+00\", \"2000-03-15 00:00:00+00\"]]}}, \"description\": \"Description of my test collection.\", \"stac_extensions\": []}]}'::jsonb\n $$,\n 'Test search passing in collection ids with descending sort'\n);\n\nSELECT results_eq($$\n select collection_search('{\"limit\":1}') - '{collections}'::text[];\n $$,$$\n SELECT '{\"links\": [{\"rel\": \"next\", \"body\": {\"offset\": 1}, \"href\": \"./collections\", \"type\": \"application/json\", \"merge\": true, \"method\": \"GET\"}], \"numberMatched\": 650, \"numberReturned\": 1}'::jsonb\n $$,\n 'Test search limit 1 - next link'\n);\n\nSELECT results_eq($$\n select collection_search('{\"limit\":1, \"offset\":649}') - '{collections}'::text[];\n $$,$$\n SELECT '{\"links\": [{\"rel\": \"prev\", \"body\": {\"offset\": 648}, \"href\": \"./collections\", \"type\": \"application/json\", \"merge\": true, \"method\": \"GET\"}], \"numberMatched\": 650, \"numberReturned\": 1}'::jsonb\n $$,\n 'Test search limit 1, offset 649 - no next link'\n);\n\nSELECT results_eq($$\n select collection_search('{\"limit\": 700}') - '{collections}'::text[];\n $$,$$\n SELECT '{\"links\": [], \"numberMatched\": 650, \"numberReturned\": 650}'::jsonb\n $$,\n 'Test search limit 2 - no prev/next links'\n);\n\nSET pgstac.base_url='https://test.com/';\nSELECT results_eq($$\n select collection_search('{\"ids\":[\"testcollection_1\",\"testcollection_2\"],\"limit\":1, \"sortby\":[{\"field\":\"id\",\"direction\":\"asc\"}]}');\n $$,$$\n SELECT '{\"links\": [{\"rel\": \"next\", \"body\": {\"offset\": 1}, \"href\": \"https://test.com/collections\", \"type\": \"application/json\", \"merge\": true, \"method\": \"GET\"}], \"numberMatched\": 2, \"numberReturned\": 1, \"collections\": [{\"id\": \"testcollection_1\", \"type\": \"Collection\", \"title\": \"My Test Collection.\", \"extent\": {\"spatial\": {\"bbox\": [[-180, -90, -170, -80]]}, \"temporal\": {\"interval\": [[\"2000-01-08 00:00:00+00\", \"2000-03-08 00:00:00+00\"]]}}, \"description\": \"Description of my test collection.\", \"stac_extensions\": []}]}'::jsonb\n $$,\n 'Test search passing in collection ids with base_url set'\n);\n" }, { "path": "src/pgstac/tests/pgtap/005_tileutils.sql", "content": "SELECT has_function('pgstac'::name, 'tileenvelope', ARRAY['int', 'int', 'int']);\nSELECT has_function('pgstac'::name, 'ftime', ARRAY[]::text[]);\n" }, { "path": "src/pgstac/tests/pgtap/006_tilesearch.sql", "content": "SELECT has_function('pgstac'::name, 'geometrysearch', ARRAY['geometry','text','jsonb','int','int','interval','boolean','boolean']);\nSELECT has_function('pgstac'::name, 'geojsonsearch', ARRAY['jsonb','text','jsonb','int','int','interval','boolean','boolean']);\nSELECT has_function('pgstac'::name, 'xyzsearch', ARRAY['int','int','int','text','jsonb','int','int','interval','boolean','boolean']);\n" }, { "path": "src/pgstac/tests/pgtap/9999_readonly.sql", "content": "SET transaction_read_only TO 'on';\n\nSELECT results_eq(\n $$ SHOW transaction_read_only; $$,\n $$ SELECT 'on'; $$,\n 'Transaction set to read only'\n);\n\nSELECT throws_ok(\n $$ SELECT search('{}'); $$,\n '25006'\n);\n\nSET pgstac.readonly to 'true';\nSELECT results_eq(\n $$ SELECT pgstac.readonly(); $$,\n $$ SELECT TRUE; $$,\n 'Readonly is set to true'\n);\n\nSELECT lives_ok(\n $$ SELECT search('{}'); $$,\n 'Search works with readonly mode set to on in readonly mode.'\n);\n\nSET pgstac.context TO 'on';\nSELECT lives_ok(\n $$ SELECT search('{}'); $$,\n 'Search works with readonly mode set to on in readonly mode and the context extension enabled.'\n);\nRESET pgstac.readonly;\n" }, { "path": "src/pgstac/tests/pgtap/999_version.sql", "content": "" }, { "path": "src/pgstac/tests/pgtap.sql", "content": "\\unset ECHO\n\\set QUIET 1\n-- Turn off echo and keep things quiet.\n\n-- Format the output for nice TAP.\n\\pset format unaligned\n\\pset tuples_only true\n\\pset pager off\n\\timing off\n\n-- Revert all changes on failure.\n\\set ON_ERROR_STOP true\n\n-- Load the TAP functions.\nBEGIN;\nCREATE EXTENSION IF NOT EXISTS pgtap;\nSET SEARCH_PATH TO pgstac, pgtap, public;\n\n-- Plan the tests.\nSELECT plan(229);\n--SELECT * FROM no_plan();\n\n-- Run the tests.\n\n-- Core\n\\i tests/pgtap/001_core.sql\n\\i tests/pgtap/001a_jsonutils.sql\n\\i tests/pgtap/001b_cursorutils.sql\n\\i tests/pgtap/001s_stacutils.sql\n\\i tests/pgtap/002_collections.sql\n\\i tests/pgtap/002a_queryables.sql\n\\i tests/pgtap/003_items.sql\n\\i tests/pgtap/004_search.sql\n\\i tests/pgtap/004a_collectionsearch.sql\n\\i tests/pgtap/005_tileutils.sql\n\\i tests/pgtap/006_tilesearch.sql\n\\i tests/pgtap/999_version.sql\n\n-- Finish the tests and clean up.\nSELECT * FROM finish();\nROLLBACK;\n" }, { "path": "src/pgstac/tests/testdata/collections.json", "content": "{\n \"id\": \"pgstac-test-collection\",\n \"stac_version\": \"1.0.0-beta.2\",\n \"description\": \"The National Agriculture Imagery Program (NAIP) acquires aerial imagery\\nduring the agricultural growing seasons in the continental U.S.\\n\\nNAIP projects are contracted each year based upon available funding and the\\nFSA imagery acquisition cycle. Beginning in 2003, NAIP was acquired on\\na 5-year cycle. 2008 was a transition year, and a three-year cycle began\\nin 2009.\\n\\nNAIP imagery is acquired at a one-meter ground sample distance (GSD) with a\\nhorizontal accuracy that matches within six meters of photo-identifiable\\nground control points, which are used during image inspection.\\n\\nOlder images were collected using 3 bands (Red, Green, and Blue: RGB), but\\nnewer imagery is usually collected with an additional near-infrared band\\n(RGBN).\",\n \"links\": [\n {\n \"rel\": \"root\",\n \"href\": \"/collection.json\",\n \"type\": \"application/json\"\n },\n {\n \"rel\": \"self\",\n \"href\": \"/collection.json\",\n \"type\": \"application/json\"\n }\n ],\n \"stac_extensions\": [],\n \"title\": \"NAIP: National Agriculture Imagery Program\",\n \"extent\": {\n \"spatial\": {\n \"bbox\": [\n [\n -124.784,\n 24.744,\n -66.951,\n 49.346\n ]\n ]\n },\n \"temporal\": {\n \"interval\": [\n [\n \"2011-01-01T00:00:00Z\",\n \"2019-01-01T00:00:00Z\"\n ]\n ]\n }\n },\n \"license\": \"PDDL-1.0\",\n \"providers\": [\n {\n \"name\": \"USDA Farm Service Agency\",\n \"roles\": [\n \"producer\",\n \"licensor\"\n ],\n \"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\"\n }\n ],\n \"item_assets\": {\n \"image\": {\n \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\",\n \"roles\": [\n \"data\"\n ],\n \"title\": \"RGBIR COG tile\",\n \"eo:bands\": [\n {\n \"name\": \"Red\",\n \"common_name\": \"red\"\n },\n {\n \"name\": \"Green\",\n \"common_name\": \"green\"\n },\n {\n \"name\": \"Blue\",\n \"common_name\": \"blue\"\n },\n {\n \"name\": \"NIR\",\n \"common_name\": \"nir\",\n \"description\": \"near-infrared\"\n }\n ]\n },\n \"metadata\": {\n \"type\": \"text/plain\",\n \"roles\": [\n \"metadata\"\n ],\n \"title\": \"FGDC Metdata\"\n },\n \"thumbnail\": {\n \"type\": \"image/jpeg\",\n \"roles\": [\n \"thumbnail\"\n ],\n \"title\": \"Thumbnail\"\n }\n }\n}\n" }, { "path": "src/pgstac/tests/testdata/collections.ndjson", "content": "{\"id\":\"pgstac-test-collection\",\"stac_version\":\"1.0.0-beta.2\",\"description\":\"The National Agriculture Imagery Program (NAIP) acquires aerial imagery\\\\nduring the agricultural growing seasons in the continental U.S.\\\\n\\\\nNAIP projects are contracted each year based upon available funding and the\\\\nFSA imagery acquisition cycle. Beginning in 2003, NAIP was acquired on\\\\na 5-year cycle. 2008 was a transition year, and a three-year cycle began\\\\nin 2009.\\\\n\\\\nNAIP imagery is acquired at a one-meter ground sample distance (GSD) with a\\\\nhorizontal accuracy that matches within six meters of photo-identifiable\\\\nground control points, which are used during image inspection.\\\\n\\\\nOlder images were collected using 3 bands (Red, Green, and Blue: RGB), but\\\\nnewer imagery is usually collected with an additional near-infrared band\\\\n(RGBN).\",\"links\":[{\"rel\":\"root\",\"href\":\"/collection.json\",\"type\":\"application/json\"},{\"rel\":\"self\",\"href\":\"/collection.json\",\"type\":\"application/json\"}],\"stac_extensions\":[],\"title\":\"NAIP: National Agriculture Imagery Program\",\"extent\":{\"spatial\":{\"bbox\":[[-124.784,24.744,-66.951,49.346]]},\"temporal\":{\"interval\":[[\"2011-01-01T00:00:00Z\",\"2019-01-01T00:00:00Z\"]]}},\"license\":\"PDDL-1.0\",\"providers\":[{\"name\":\"USDA Farm Service Agency\",\"roles\":[\"producer\",\"licensor\"],\"url\":\"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\"}],\"item_assets\":{\"image\":{\"type\":\"image/tiff; application=geotiff; profile=cloud-optimized\",\"roles\":[\"data\"],\"title\":\"RGBIR COG tile\",\"eo:bands\":[{\"name\":\"Red\",\"common_name\":\"red\"},{\"name\":\"Green\",\"common_name\":\"green\"},{\"name\":\"Blue\",\"common_name\":\"blue\"},{\"name\":\"NIR\",\"common_name\":\"nir\",\"description\":\"near-infrared\"}]},\"metadata\":{\"type\":\"text/plain\",\"roles\":[\"metadata\"],\"title\":\"FGDC Metdata\"},\"thumbnail\":{\"type\":\"image/jpeg\",\"roles\":[\"thumbnail\"],\"title\":\"Thumbnail\"}}}\n{\"id\":\"pgstac-test-collection2\",\"stac_version\":\"1.0.0-beta.2\",\"description\":\"The National Agriculture Imagery Program (NAIP) acquires aerial imagery\\\\nduring the agricultural growing seasons in the continental U.S.\\\\n\\\\nNAIP projects are contracted each year based upon available funding and the\\\\nFSA imagery acquisition cycle. Beginning in 2003, NAIP was acquired on\\\\na 5-year cycle. 2008 was a transition year, and a three-year cycle began\\\\nin 2009.\\\\n\\\\nNAIP imagery is acquired at a one-meter ground sample distance (GSD) with a\\\\nhorizontal accuracy that matches within six meters of photo-identifiable\\\\nground control points, which are used during image inspection.\\\\n\\\\nOlder images were collected using 3 bands (Red, Green, and Blue: RGB), but\\\\nnewer imagery is usually collected with an additional near-infrared band\\\\n(RGBN).\",\"links\":[{\"rel\":\"root\",\"href\":\"/collection.json\",\"type\":\"application/json\"},{\"rel\":\"self\",\"href\":\"/collection.json\",\"type\":\"application/json\"}],\"stac_extensions\":[],\"title\":\"NAIP: National Agriculture Imagery Program\",\"extent\":{\"spatial\":{\"bbox\":[[-124.784,24.744,-66.951,49.346]]},\"temporal\":{\"interval\":[[\"2011-01-01T00:00:00Z\",\"2019-01-01T00:00:00Z\"]]}},\"license\":\"PDDL-1.0\",\"providers\":[{\"name\":\"USDA Farm Service Agency\",\"roles\":[\"producer\",\"licensor\"],\"url\":\"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\"}],\"item_assets\":{\"image\":{\"type\":\"image/tiff; application=geotiff; profile=cloud-optimized\",\"roles\":[\"data\"],\"title\":\"RGBIR COG tile\",\"eo:bands\":[{\"name\":\"Red\",\"common_name\":\"red\"},{\"name\":\"Green\",\"common_name\":\"green\"},{\"name\":\"Blue\",\"common_name\":\"blue\"},{\"name\":\"NIR\",\"common_name\":\"nir\",\"description\":\"near-infrared\"}]},\"metadata\":{\"type\":\"text/plain\",\"roles\":[\"metadata\"],\"title\":\"FGDC Metdata\"},\"thumbnail\":{\"type\":\"image/jpeg\",\"roles\":[\"thumbnail\"],\"title\":\"Thumbnail\"}}}\n" }, { "path": "src/pgstac/tests/testdata/items.ndjson", "content": "{\"id\": \"pgstac-test-item-0003\", \"bbox\": [-85.379245, 30.933949, -85.308201, 31.003555], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30085/m_3008506_nw_16_1_20110825.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30085/m_3008506_nw_16_1_20110825.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30085/m_3008506_nw_16_1_20110825.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-85.309412, 30.933949], [-85.308201, 31.002658], [-85.378084, 31.003555], [-85.379245, 30.934843], [-85.309412, 30.933949]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-25T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [654842, 3423507, 661516, 3431125], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7618, 6674], \"eo:cloud_cover\": 28, \"proj:transform\": [1, 0, 654842, 0, -1, 3431125, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0004\", \"bbox\": [-87.190775, 30.934712, -87.121772, 31.002794], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008707_ne_16_1_20110824.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008707_ne_16_1_20110824.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008707_ne_16_1_20110824.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.121772, 30.934795], [-87.121858, 31.002794], [-87.190775, 31.002711], [-87.190639, 30.934712], [-87.121772, 30.934795]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-24T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [481788, 3422382, 488367, 3429918], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7536, 6579], \"eo:cloud_cover\": 23, \"proj:transform\": [1, 0, 481788, 0, -1, 3429918, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0005\", \"bbox\": [-87.128238, 30.934744, -87.059311, 31.002757], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008708_nw_16_1_20110824.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008708_nw_16_1_20110824.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008708_nw_16_1_20110824.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.059311, 30.934794], [-87.059353, 31.002757], [-87.128238, 31.002707], [-87.128147, 30.934744], [-87.059311, 30.934794]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-24T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [487758, 3422377, 494334, 3429909], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7532, 6576], \"eo:cloud_cover\": 3, \"proj:transform\": [1, 0, 487758, 0, -1, 3429909, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0006\", \"bbox\": [-87.253322, 30.934677, -87.184223, 31.00282], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008707_nw_16_1_20110824.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008707_nw_16_1_20110824.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008707_nw_16_1_20110824.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.184223, 30.934794], [-87.184354, 31.00282], [-87.253322, 31.002703], [-87.253142, 30.934677], [-87.184223, 30.934794]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-24T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [475817, 3422390, 482401, 3429929], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7539, 6584], \"eo:cloud_cover\": 100, \"proj:transform\": [1, 0, 475817, 0, -1, 3429929, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0007\", \"bbox\": [-87.440935, 30.622081, -87.371617, 30.690415], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008721_se_16_1_20110817.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008721_se_16_1_20110817.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008721_se_16_1_20110817.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.371617, 30.622297], [-87.371878, 30.690415], [-87.440935, 30.690199], [-87.440626, 30.622081], [-87.371617, 30.622297]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-17T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [457770, 3387803, 464384, 3395352], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7549, 6614], \"eo:cloud_cover\": 59, \"proj:transform\": [1, 0, 457770, 0, -1, 3395352, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0008\", \"bbox\": [-87.503478, 30.622053, -87.434074, 30.690447], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008721_sw_16_1_20110817.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008721_sw_16_1_20110817.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008721_sw_16_1_20110817.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.434074, 30.622302], [-87.434379, 30.690447], [-87.503478, 30.690198], [-87.503124, 30.622053], [-87.434074, 30.622302]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-17T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [451780, 3387825, 458398, 3395377], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7552, 6618], \"eo:cloud_cover\": 64, \"proj:transform\": [1, 0, 451780, 0, -1, 3395377, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0009\", \"bbox\": [-87.503478, 30.68455, -87.434072, 30.752944], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008721_nw_16_1_20110817.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008721_nw_16_1_20110817.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008721_nw_16_1_20110817.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.434072, 30.684799], [-87.434377, 30.752944], [-87.503478, 30.752694], [-87.503125, 30.68455], [-87.434072, 30.684799]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-17T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [451811, 3394751, 458425, 3402303], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7552, 6614], \"eo:cloud_cover\": 61, \"proj:transform\": [1, 0, 451811, 0, -1, 3402303, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0010\", \"bbox\": [-87.440937, 30.684579, -87.371606, 30.752912], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008721_ne_16_1_20110817.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008721_ne_16_1_20110817.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008721_ne_16_1_20110817.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.371606, 30.684794], [-87.371867, 30.752912], [-87.440937, 30.752696], [-87.440628, 30.684579], [-87.371606, 30.684794]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-17T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [457797, 3394729, 464408, 3402278], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7549, 6611], \"eo:cloud_cover\": 31, \"proj:transform\": [1, 0, 457797, 0, -1, 3402278, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0011\", \"bbox\": [-87.315859, 30.934648, -87.246684, 31.002851], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008706_ne_16_1_20110817.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008706_ne_16_1_20110817.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008706_ne_16_1_20110817.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.246684, 30.934798], [-87.246859, 31.002851], [-87.315859, 31.0027], [-87.315635, 30.934648], [-87.246684, 30.934798]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-17T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [469847, 3422402, 476434, 3429944], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7542, 6587], \"eo:cloud_cover\": 41, \"proj:transform\": [1, 0, 469847, 0, -1, 3429944, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0012\", \"bbox\": [-87.440942, 30.93458, -87.371596, 31.002921], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008705_ne_16_1_20110817.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008705_ne_16_1_20110817.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008705_ne_16_1_20110817.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.371596, 30.934797], [-87.37186, 31.002921], [-87.440942, 31.002704], [-87.440629, 30.93458], [-87.371596, 30.934797]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-17T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [457906, 3422435, 464501, 3429985], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7550, 6595], \"eo:cloud_cover\": 4, \"proj:transform\": [1, 0, 457906, 0, -1, 3429985, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0013\", \"bbox\": [-87.378406, 30.934615, -87.309145, 31.002887], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008706_nw_16_1_20110817.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008706_nw_16_1_20110817.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008706_nw_16_1_20110817.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.309145, 30.934799], [-87.309365, 31.002887], [-87.378406, 31.002704], [-87.378137, 30.934615], [-87.309145, 30.934799]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-17T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [463876, 3422417, 470467, 3429963], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7546, 6591], \"eo:cloud_cover\": 2, \"proj:transform\": [1, 0, 463876, 0, -1, 3429963, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0014\", \"bbox\": [-87.628547, 30.496984, -87.558991, 30.565507], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008728_sw_16_1_20110816.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008728_sw_16_1_20110816.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008728_sw_16_1_20110816.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.558991, 30.497299], [-87.559382, 30.565507], [-87.628547, 30.565192], [-87.628108, 30.496984], [-87.558991, 30.497299]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [439724, 3374025, 446357, 3381584], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7559, 6633], \"eo:cloud_cover\": 17, \"proj:transform\": [1, 0, 439724, 0, -1, 3381584, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0015\", \"bbox\": [-86.940689, 30.934744, -86.871762, 31.002757], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30086/m_3008601_ne_16_1_20110816.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30086/m_3008601_ne_16_1_20110816.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30086/m_3008601_ne_16_1_20110816.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-86.871853, 30.934744], [-86.871762, 31.002707], [-86.940647, 31.002757], [-86.940689, 30.934794], [-86.871853, 30.934744]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [505666, 3422377, 512242, 3429909], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7532, 6576], \"eo:cloud_cover\": 54, \"proj:transform\": [1, 0, 505666, 0, -1, 3429909, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0016\", \"bbox\": [-87.003143, 30.934773, -86.93431, 31.002726], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30086/m_3008601_nw_16_1_20110816.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30086/m_3008601_nw_16_1_20110816.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30086/m_3008601_nw_16_1_20110816.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-86.934356, 30.934773], [-86.93431, 31.002709], [-87.003143, 31.002726], [-87.00314, 30.934789], [-86.934356, 30.934773]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [499700, 3422375, 506271, 3429904], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7529, 6571], \"eo:cloud_cover\": 13, \"proj:transform\": [1, 0, 499700, 0, -1, 3429904, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0017\", \"bbox\": [-86.815777, 30.934677, -86.746678, 31.00282], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30086/m_3008602_ne_16_1_20110816.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30086/m_3008602_ne_16_1_20110816.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30086/m_3008602_ne_16_1_20110816.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-86.746858, 30.934677], [-86.746678, 31.002703], [-86.815646, 31.00282], [-86.815777, 30.934794], [-86.746858, 30.934677]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [517599, 3422390, 524183, 3429929], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7539, 6584], \"eo:cloud_cover\": 59, \"proj:transform\": [1, 0, 517599, 0, -1, 3429929, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0018\", \"bbox\": [-86.878228, 30.934712, -86.809225, 31.002794], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30086/m_3008602_nw_16_1_20110816.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30086/m_3008602_nw_16_1_20110816.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30086/m_3008602_nw_16_1_20110816.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-86.809361, 30.934712], [-86.809225, 31.002711], [-86.878142, 31.002794], [-86.878228, 30.934795], [-86.809361, 30.934712]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [511633, 3422382, 518212, 3429918], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7536, 6579], \"eo:cloud_cover\": 29, \"proj:transform\": [1, 0, 511633, 0, -1, 3429918, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0019\", \"bbox\": [-87.566035, 30.934518, -87.496517, 31.002979], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008704_ne_16_1_20110816.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008704_ne_16_1_20110816.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008704_ne_16_1_20110816.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.496517, 30.934802], [-87.49687, 31.002979], [-87.566035, 31.002694], [-87.565633, 30.934518], [-87.496517, 30.934802]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [445964, 3422482, 452567, 3430038], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7556, 6603], \"eo:cloud_cover\": 52, \"proj:transform\": [1, 0, 445964, 0, -1, 3430038, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0020\", \"bbox\": [-87.62857, 30.934491, -87.558977, 31.003012], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008704_nw_16_1_20110816.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008704_nw_16_1_20110816.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008704_nw_16_1_20110816.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.558977, 30.934808], [-87.559374, 31.003012], [-87.62857, 31.002694], [-87.628123, 30.934491], [-87.558977, 30.934808]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [439994, 3422511, 446600, 3430070], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7559, 6606], \"eo:cloud_cover\": 39, \"proj:transform\": [1, 0, 439994, 0, -1, 3430070, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0021\", \"bbox\": [-87.628569, 30.871988, -87.55898, 30.940509], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008704_sw_16_1_20110816.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008704_sw_16_1_20110816.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008704_sw_16_1_20110816.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.55898, 30.872305], [-87.559377, 30.940509], [-87.628569, 30.940191], [-87.628123, 30.871988], [-87.55898, 30.872305]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [439955, 3415584, 446565, 3423143], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7559, 6610], \"eo:cloud_cover\": 29, \"proj:transform\": [1, 0, 439955, 0, -1, 3423143, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0022\", \"bbox\": [-87.503488, 30.93455, -87.434056, 31.002952], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008705_nw_16_1_20110816.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008705_nw_16_1_20110816.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008705_nw_16_1_20110816.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.434056, 30.934801], [-87.434365, 31.002952], [-87.503488, 31.002701], [-87.503131, 30.93455], [-87.434056, 30.934801]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [451935, 3422457, 458534, 3430010], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7553, 6599], \"eo:cloud_cover\": 84, \"proj:transform\": [1, 0, 451935, 0, -1, 3430010, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0023\", \"bbox\": [-87.06569, 30.934773, -86.996857, 31.002726], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008708_ne_16_1_20110816.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008708_ne_16_1_20110816.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008708_ne_16_1_20110816.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-86.99686, 30.934789], [-86.996857, 31.002726], [-87.06569, 31.002709], [-87.065644, 30.934773], [-86.99686, 30.934789]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [493729, 3422375, 500300, 3429904], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7529, 6571], \"eo:cloud_cover\": 56, \"proj:transform\": [1, 0, 493729, 0, -1, 3429904, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0024\", \"bbox\": [-87.941273, 30.809325, -87.871271, 30.878173], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008709_ne_16_1_20110816.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008709_ne_16_1_20110816.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008709_ne_16_1_20110816.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.871271, 30.80981], [-87.871889, 30.878173], [-87.941273, 30.877687], [-87.940605, 30.809325], [-87.871271, 30.80981]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [410024, 3408849, 416657, 3416426], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7577, 6633], \"eo:cloud_cover\": 58, \"proj:transform\": [1, 0, 410024, 0, -1, 3416426, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0025\", \"bbox\": [-88.003818, 30.809299, -87.933721, 30.878206], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008709_nw_16_1_20110816.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008709_nw_16_1_20110816.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008709_nw_16_1_20110816.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.933721, 30.809817], [-87.934384, 30.878206], [-88.003818, 30.877686], [-88.003106, 30.809299], [-87.933721, 30.809817]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [404045, 3408898, 410683, 3416478], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7580, 6638], \"eo:cloud_cover\": 65, \"proj:transform\": [1, 0, 404045, 0, -1, 3416478, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0026\", \"bbox\": [-87.941269, 30.746832, -87.871272, 30.815671], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008709_se_16_1_20110816.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008709_se_16_1_20110816.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008709_se_16_1_20110816.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.871272, 30.747317], [-87.871889, 30.815671], [-87.941269, 30.815185], [-87.940603, 30.746832], [-87.871272, 30.747317]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [409966, 3401923, 416603, 3409499], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7576, 6637], \"eo:cloud_cover\": 52, \"proj:transform\": [1, 0, 409966, 0, -1, 3409499, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0027\", \"bbox\": [-88.003816, 30.746797, -87.933724, 30.815704], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008709_sw_16_1_20110816.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008709_sw_16_1_20110816.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008709_sw_16_1_20110816.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.933724, 30.747315], [-87.934385, 30.815704], [-88.003816, 30.815185], [-88.003106, 30.746797], [-87.933724, 30.747315]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [403983, 3401971, 410625, 3409551], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7580, 6642], \"eo:cloud_cover\": 43, \"proj:transform\": [1, 0, 403983, 0, -1, 3409551, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0028\", \"bbox\": [-87.878737, 30.809358, -87.80881, 30.878137], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008710_nw_16_1_20110816.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008710_nw_16_1_20110816.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008710_nw_16_1_20110816.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.80881, 30.809809], [-87.809384, 30.878137], [-87.878737, 30.877684], [-87.878114, 30.809358], [-87.80881, 30.809809]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [416002, 3408804, 422632, 3416377], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7573, 6630], \"eo:cloud_cover\": 46, \"proj:transform\": [1, 0, 416002, 0, -1, 3416377, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0029\", \"bbox\": [-87.878732, 30.746864, -87.80881, 30.815634], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008710_sw_16_1_20110816.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008710_sw_16_1_20110816.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008710_sw_16_1_20110816.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.80881, 30.747315], [-87.809382, 30.815634], [-87.878732, 30.815182], [-87.878111, 30.746864], [-87.80881, 30.747315]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [415948, 3401878, 422582, 3409450], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7572, 6634], \"eo:cloud_cover\": 42, \"proj:transform\": [1, 0, 415948, 0, -1, 3409450, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0030\", \"bbox\": [-87.691106, 30.809455, -87.621436, 30.878045], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008711_ne_16_1_20110816.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008711_ne_16_1_20110816.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008711_ne_16_1_20110816.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.621436, 30.809805], [-87.621876, 30.878045], [-87.691106, 30.877694], [-87.690617, 30.809455], [-87.621436, 30.809805]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [433938, 3408689, 440556, 3416252], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7563, 6618], \"eo:cloud_cover\": 16, \"proj:transform\": [1, 0, 433938, 0, -1, 3416252, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0031\", \"bbox\": [-87.691108, 30.74696, -87.621442, 30.815542], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008711_se_16_1_20110816.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008711_se_16_1_20110816.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008711_se_16_1_20110816.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.621442, 30.74731], [-87.62188, 30.815542], [-87.691108, 30.815191], [-87.69062, 30.74696], [-87.621442, 30.74731]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [433895, 3401763, 440517, 3409325], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7562, 6622], \"eo:cloud_cover\": 16, \"proj:transform\": [1, 0, 433895, 0, -1, 3409325, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0032\", \"bbox\": [-87.628569, 30.809484, -87.558973, 30.878015], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008712_nw_16_1_20110816.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008712_nw_16_1_20110816.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008712_nw_16_1_20110816.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.558973, 30.809801], [-87.559369, 30.878015], [-87.628569, 30.877697], [-87.628124, 30.809484], [-87.558973, 30.809801]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [439916, 3408657, 446531, 3416217], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7560, 6615], \"eo:cloud_cover\": 7, \"proj:transform\": [1, 0, 439916, 0, -1, 3416217, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0033\", \"bbox\": [-87.566019, 30.747015, -87.496524, 30.815477], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008712_se_16_1_20110816.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008712_se_16_1_20110816.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008712_se_16_1_20110816.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.496524, 30.747298], [-87.496874, 30.815477], [-87.566019, 30.815193], [-87.56562, 30.747015], [-87.496524, 30.747298]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [445860, 3401702, 452474, 3409258], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7556, 6614], \"eo:cloud_cover\": 49, \"proj:transform\": [1, 0, 445860, 0, -1, 3409258, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0034\", \"bbox\": [-87.628559, 30.746989, -87.558978, 30.815511], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008712_sw_16_1_20110816.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008712_sw_16_1_20110816.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008712_sw_16_1_20110816.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.558978, 30.747305], [-87.559372, 30.815511], [-87.628559, 30.815194], [-87.628115, 30.746989], [-87.558978, 30.747305]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [439878, 3401731, 446496, 3409290], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7559, 6618], \"eo:cloud_cover\": 3, \"proj:transform\": [1, 0, 439878, 0, -1, 3409290, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0035\", \"bbox\": [-87.941266, 30.68433, -87.871274, 30.753168], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008717_ne_16_1_20110816.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008717_ne_16_1_20110816.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008717_ne_16_1_20110816.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.871274, 30.684813], [-87.871889, 30.753168], [-87.941266, 30.752684], [-87.940602, 30.68433], [-87.871274, 30.684813]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [409908, 3394996, 416549, 3402572], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7576, 6641], \"eo:cloud_cover\": 33, \"proj:transform\": [1, 0, 409908, 0, -1, 3402572, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0036\", \"bbox\": [-88.003814, 30.684295, -87.933727, 30.753202], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008717_nw_16_1_20110816.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008717_nw_16_1_20110816.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008717_nw_16_1_20110816.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.933727, 30.684813], [-87.934386, 30.753202], [-88.003814, 30.752684], [-88.003106, 30.684295], [-87.933727, 30.684813]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [403921, 3395044, 410567, 3402624], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7580, 6646], \"eo:cloud_cover\": 12, \"proj:transform\": [1, 0, 403921, 0, -1, 3402624, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0037\", \"bbox\": [-87.941265, 30.621827, -87.871277, 30.690665], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008717_se_16_1_20110816.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008717_se_16_1_20110816.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008717_se_16_1_20110816.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.871277, 30.62231], [-87.871891, 30.690665], [-87.941265, 30.690181], [-87.940603, 30.621827], [-87.871277, 30.62231]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [409850, 3388069, 416495, 3395645], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7576, 6645], \"eo:cloud_cover\": 54, \"proj:transform\": [1, 0, 409850, 0, -1, 3395645, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0038\", \"bbox\": [-88.003804, 30.621802, -87.933732, 30.6907], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008717_sw_16_1_20110816.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008717_sw_16_1_20110816.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008717_sw_16_1_20110816.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.933732, 30.622318], [-87.934389, 30.6907], [-88.003804, 30.690182], [-88.003098, 30.621802], [-87.933732, 30.622318]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [403860, 3388118, 410509, 3395697], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7579, 6649], \"eo:cloud_cover\": 77, \"proj:transform\": [1, 0, 403860, 0, -1, 3395697, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0039\", \"bbox\": [-87.878728, 30.684361, -87.808821, 30.75314], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008718_nw_16_1_20110816.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008718_nw_16_1_20110816.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008718_nw_16_1_20110816.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.808821, 30.684811], [-87.809392, 30.75314], [-87.878728, 30.752689], [-87.878109, 30.684361], [-87.808821, 30.684811]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [415894, 3394951, 422531, 3402524], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7573, 6637], \"eo:cloud_cover\": 12, \"proj:transform\": [1, 0, 415894, 0, -1, 3402524, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0040\", \"bbox\": [-87.878725, 30.621858, -87.808823, 30.690637], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008718_sw_16_1_20110816.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008718_sw_16_1_20110816.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008718_sw_16_1_20110816.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.808823, 30.622307], [-87.809392, 30.690637], [-87.878725, 30.690186], [-87.878107, 30.621858], [-87.808823, 30.622307]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [415840, 3388024, 422481, 3395597], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7573, 6641], \"eo:cloud_cover\": 10, \"proj:transform\": [1, 0, 415840, 0, -1, 3395597, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0041\", \"bbox\": [-87.6911, 30.684456, -87.621438, 30.753047], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008719_ne_16_1_20110816.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008719_ne_16_1_20110816.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008719_ne_16_1_20110816.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.621438, 30.684805], [-87.621875, 30.753047], [-87.6911, 30.752696], [-87.690613, 30.684456], [-87.621438, 30.684805]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [433853, 3394836, 440479, 3402399], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7563, 6626], \"eo:cloud_cover\": 3, \"proj:transform\": [1, 0, 433853, 0, -1, 3402399, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0042\", \"bbox\": [-87.691103, 30.62196, -87.621445, 30.690542], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008719_se_16_1_20110816.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008719_se_16_1_20110816.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008719_se_16_1_20110816.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.621445, 30.622309], [-87.621882, 30.690542], [-87.691103, 30.690192], [-87.690618, 30.62196], [-87.621445, 30.622309]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [433810, 3387910, 440440, 3395472], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7562, 6630], \"eo:cloud_cover\": 99, \"proj:transform\": [1, 0, 433810, 0, -1, 3395472, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0043\", \"bbox\": [-87.566019, 30.684519, -87.496527, 30.752981], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008720_ne_16_1_20110816.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008720_ne_16_1_20110816.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008720_ne_16_1_20110816.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.496527, 30.684801], [-87.496877, 30.752981], [-87.566019, 30.752698], [-87.565621, 30.684519], [-87.496527, 30.684801]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [445825, 3394776, 452443, 3402332], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7556, 6618], \"eo:cloud_cover\": 80, \"proj:transform\": [1, 0, 445825, 0, -1, 3402332, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0044\", \"bbox\": [-87.62856, 30.684484, -87.558983, 30.753015], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008720_nw_16_1_20110816.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008720_nw_16_1_20110816.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008720_nw_16_1_20110816.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.558983, 30.6848], [-87.559376, 30.753015], [-87.62856, 30.752699], [-87.628117, 30.684484], [-87.558983, 30.6848]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [439839, 3394804, 446461, 3402364], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7560, 6622], \"eo:cloud_cover\": 30, \"proj:transform\": [1, 0, 439839, 0, -1, 3402364, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0045\", \"bbox\": [-87.56602, 30.622022, -87.496532, 30.690476], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008720_se_16_1_20110816.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008720_se_16_1_20110816.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008720_se_16_1_20110816.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.496532, 30.622304], [-87.49688, 30.690476], [-87.56602, 30.690193], [-87.565623, 30.622022], [-87.496532, 30.622304]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [445790, 3387850, 452412, 3395405], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7555, 6622], \"eo:cloud_cover\": 82, \"proj:transform\": [1, 0, 445790, 0, -1, 3395405, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0046\", \"bbox\": [-87.628562, 30.621988, -87.558988, 30.69051], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008720_sw_16_1_20110816.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008720_sw_16_1_20110816.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008720_sw_16_1_20110816.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.558988, 30.622303], [-87.559381, 30.69051], [-87.628562, 30.690194], [-87.62812, 30.621988], [-87.558988, 30.622303]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [439800, 3387878, 446426, 3395437], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7559, 6626], \"eo:cloud_cover\": 16, \"proj:transform\": [1, 0, 439800, 0, -1, 3395437, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0047\", \"bbox\": [-87.941265, 30.559332, -87.871282, 30.628171], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008725_ne_16_1_20110816.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008725_ne_16_1_20110816.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008725_ne_16_1_20110816.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.871282, 30.559814], [-87.871893, 30.628171], [-87.941265, 30.627687], [-87.940604, 30.559332], [-87.871282, 30.559814]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [409792, 3381143, 416441, 3388719], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7576, 6649], \"eo:cloud_cover\": 43, \"proj:transform\": [1, 0, 409792, 0, -1, 3388719, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0048\", \"bbox\": [-87.941255, 30.496828, -87.871287, 30.565666], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008725_se_16_1_20110816.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008725_se_16_1_20110816.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008725_se_16_1_20110816.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.871287, 30.497309], [-87.871897, 30.565666], [-87.941255, 30.565183], [-87.940596, 30.496828], [-87.871287, 30.497309]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [409735, 3374216, 416387, 3381792], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7576, 6652], \"eo:cloud_cover\": 4, \"proj:transform\": [1, 0, 409735, 0, -1, 3381792, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0049\", \"bbox\": [-87.878723, 30.559362, -87.808825, 30.628132], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008726_nw_16_1_20110816.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008726_nw_16_1_20110816.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008726_nw_16_1_20110816.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.808825, 30.559811], [-87.809393, 30.628132], [-87.878723, 30.627682], [-87.878107, 30.559362], [-87.808825, 30.559811]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [415786, 3381098, 422431, 3388670], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7572, 6645], \"eo:cloud_cover\": 9, \"proj:transform\": [1, 0, 415786, 0, -1, 3388670, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0050\", \"bbox\": [-87.878723, 30.496867, -87.808828, 30.565637], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008726_sw_16_1_20110816.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008726_sw_16_1_20110816.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008726_sw_16_1_20110816.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.808828, 30.497315], [-87.809395, 30.565637], [-87.878723, 30.565187], [-87.878108, 30.496867], [-87.808828, 30.497315]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [415732, 3374172, 422381, 3381744], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7572, 6649], \"eo:cloud_cover\": 84, \"proj:transform\": [1, 0, 415732, 0, -1, 3381744, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0051\", \"bbox\": [-87.691097, 30.559454, -87.621453, 30.628045], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008727_ne_16_1_20110816.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008727_ne_16_1_20110816.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008727_ne_16_1_20110816.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.621453, 30.559803], [-87.621889, 30.628045], [-87.691097, 30.627696], [-87.690613, 30.559454], [-87.621453, 30.559803]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [433768, 3380983, 440401, 3388546], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7563, 6633], \"eo:cloud_cover\": 22, \"proj:transform\": [1, 0, 433768, 0, -1, 3388546, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0052\", \"bbox\": [-87.691091, 30.496957, -87.621451, 30.565539], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008727_se_16_1_20110816.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008727_se_16_1_20110816.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008727_se_16_1_20110816.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.621451, 30.497305], [-87.621886, 30.565539], [-87.691091, 30.565191], [-87.690608, 30.496957], [-87.621451, 30.497305]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [433726, 3374057, 440363, 3381619], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7562, 6637], \"eo:cloud_cover\": 12, \"proj:transform\": [1, 0, 433726, 0, -1, 3381619, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0053\", \"bbox\": [-87.56601, 30.559516, -87.496536, 30.627979], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008728_ne_16_1_20110816.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008728_ne_16_1_20110816.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008728_ne_16_1_20110816.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.496536, 30.559797], [-87.496884, 30.627979], [-87.56601, 30.627696], [-87.565614, 30.559516], [-87.496536, 30.559797]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [445756, 3380923, 452381, 3388479], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7556, 6625], \"eo:cloud_cover\": 24, \"proj:transform\": [1, 0, 445756, 0, -1, 3388479, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0054\", \"bbox\": [-87.628554, 30.559491, -87.558995, 30.628014], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008728_nw_16_1_20110816.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008728_nw_16_1_20110816.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008728_nw_16_1_20110816.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.558995, 30.559806], [-87.559386, 30.628014], [-87.628554, 30.627698], [-87.628114, 30.559491], [-87.558995, 30.559806]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [439762, 3380952, 446391, 3388511], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7559, 6629], \"eo:cloud_cover\": 5, \"proj:transform\": [1, 0, 439762, 0, -1, 3388511, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0055\", \"bbox\": [-87.566012, 30.497018, -87.496531, 30.565481], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008728_se_16_1_20110816.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008728_se_16_1_20110816.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008728_se_16_1_20110816.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.496531, 30.497299], [-87.496878, 30.565481], [-87.566012, 30.565199], [-87.565617, 30.497018], [-87.496531, 30.497299]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [445721, 3373997, 452351, 3381553], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7556, 6630], \"eo:cloud_cover\": 79, \"proj:transform\": [1, 0, 445721, 0, -1, 3381553, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0056\", \"bbox\": [-87.941283, 30.934327, -87.871262, 31.003175], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008701_ne_16_1_20110816.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008701_ne_16_1_20110816.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008701_ne_16_1_20110816.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.871262, 30.934813], [-87.871883, 31.003175], [-87.941283, 31.002688], [-87.940613, 30.934327], [-87.871262, 30.934813]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [410140, 3422703, 416766, 3430280], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7577, 6626], \"eo:cloud_cover\": 62, \"proj:transform\": [1, 0, 410140, 0, -1, 3430280, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0057\", \"bbox\": [-88.003827, 30.9343, -87.93372, 31.003207], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008701_nw_16_1_20110816.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008701_nw_16_1_20110816.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008701_nw_16_1_20110816.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.93372, 30.93482], [-87.934386, 31.003207], [-88.003827, 31.002686], [-88.003111, 30.9343], [-87.93372, 30.93482]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [404169, 3422752, 410799, 3430332], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7580, 6630], \"eo:cloud_cover\": 7, \"proj:transform\": [1, 0, 404169, 0, -1, 3430332, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0058\", \"bbox\": [-87.941277, 30.871827, -87.871261, 30.940674], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008701_se_16_1_20110816.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008701_se_16_1_20110816.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008701_se_16_1_20110816.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.871261, 30.872312], [-87.87188, 30.940674], [-87.941277, 30.940187], [-87.940609, 30.871827], [-87.871261, 30.872312]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [410082, 3415776, 416712, 3423353], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7577, 6630], \"eo:cloud_cover\": 53, \"proj:transform\": [1, 0, 410082, 0, -1, 3423353, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0059\", \"bbox\": [-88.003822, 30.8718, -87.93372, 30.940707], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008701_sw_16_1_20110816.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008701_sw_16_1_20110816.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008701_sw_16_1_20110816.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.93372, 30.872319], [-87.934384, 30.940707], [-88.003822, 30.940186], [-88.003108, 30.8718], [-87.93372, 30.872319]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [404107, 3415825, 410741, 3423405], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7580, 6634], \"eo:cloud_cover\": 57, \"proj:transform\": [1, 0, 404107, 0, -1, 3423405, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0066\", \"bbox\": [-86.441023, 30.934491, -86.37143, 31.003012], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30086/m_3008605_ne_16_1_20110815.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30086/m_3008605_ne_16_1_20110815.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30086/m_3008605_ne_16_1_20110815.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-86.371877, 30.934491], [-86.37143, 31.002694], [-86.440626, 31.003012], [-86.441023, 30.934808], [-86.371877, 30.934491]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-15T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [553400, 3422511, 560006, 3430070], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7559, 6606], \"eo:cloud_cover\": 86, \"proj:transform\": [1, 0, 553400, 0, -1, 3430070, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0060\", \"bbox\": [-87.878739, 30.934361, -87.808804, 31.00314], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008702_nw_16_1_20110816.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008702_nw_16_1_20110816.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008702_nw_16_1_20110816.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.808804, 30.934813], [-87.80938, 31.00314], [-87.878739, 31.002686], [-87.878114, 30.934361], [-87.808804, 30.934813]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [416111, 3422658, 422733, 3430231], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7573, 6622], \"eo:cloud_cover\": 70, \"proj:transform\": [1, 0, 416111, 0, -1, 3430231, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0061\", \"bbox\": [-87.878743, 30.87186, -87.808801, 30.940638], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008702_sw_16_1_20110816.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008702_sw_16_1_20110816.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008702_sw_16_1_20110816.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.808801, 30.872311], [-87.809376, 30.940638], [-87.878743, 30.940186], [-87.878119, 30.87186], [-87.808801, 30.872311]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [416056, 3415731, 422683, 3423304], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7573, 6627], \"eo:cloud_cover\": 94, \"proj:transform\": [1, 0, 416056, 0, -1, 3423304, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0062\", \"bbox\": [-87.691116, 30.934452, -87.621426, 31.003042], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008703_ne_16_1_20110816.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008703_ne_16_1_20110816.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008703_ne_16_1_20110816.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.621426, 30.934803], [-87.621868, 31.003042], [-87.691116, 31.00269], [-87.690624, 30.934452], [-87.621426, 30.934803]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [434023, 3422542, 440634, 3430105], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7563, 6611], \"eo:cloud_cover\": 85, \"proj:transform\": [1, 0, 434023, 0, -1, 3430105, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0063\", \"bbox\": [-87.691116, 30.871958, -87.621431, 30.940539], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008703_se_16_1_20110816.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008703_se_16_1_20110816.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008703_se_16_1_20110816.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.621431, 30.872309], [-87.621872, 30.940539], [-87.691116, 30.940188], [-87.690626, 30.871958], [-87.621431, 30.872309]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [433980, 3415616, 440595, 3423178], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7562, 6615], \"eo:cloud_cover\": 2, \"proj:transform\": [1, 0, 433980, 0, -1, 3423178, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0064\", \"bbox\": [-86.565944, 30.93455, -86.496512, 31.002952], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30086/m_3008604_ne_16_1_20110815.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30086/m_3008604_ne_16_1_20110815.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30086/m_3008604_ne_16_1_20110815.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-86.496869, 30.93455], [-86.496512, 31.002701], [-86.565635, 31.002952], [-86.565944, 30.934801], [-86.496869, 30.93455]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-15T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [541466, 3422457, 548065, 3430010], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7553, 6599], \"eo:cloud_cover\": 40, \"proj:transform\": [1, 0, 541466, 0, -1, 3430010, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0065\", \"bbox\": [-86.628404, 30.93458, -86.559058, 31.002921], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30086/m_3008604_nw_16_1_20110815.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30086/m_3008604_nw_16_1_20110815.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30086/m_3008604_nw_16_1_20110815.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-86.559371, 30.93458], [-86.559058, 31.002704], [-86.62814, 31.002921], [-86.628404, 30.934797], [-86.559371, 30.93458]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-15T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [535499, 3422435, 542094, 3429985], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7550, 6595], \"eo:cloud_cover\": 27, \"proj:transform\": [1, 0, 535499, 0, -1, 3429985, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0067\", \"bbox\": [-86.503483, 30.934518, -86.433965, 31.002979], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30086/m_3008605_nw_16_1_20110815.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30086/m_3008605_nw_16_1_20110815.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30086/m_3008605_nw_16_1_20110815.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-86.434367, 30.934518], [-86.433965, 31.002694], [-86.50313, 31.002979], [-86.503483, 30.934802], [-86.434367, 30.934518]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-15T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [547433, 3422482, 554036, 3430038], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7556, 6603], \"eo:cloud_cover\": 35, \"proj:transform\": [1, 0, 547433, 0, -1, 3430038, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0068\", \"bbox\": [-86.316114, 30.934419, -86.246339, 31.003078], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30086/m_3008606_ne_16_1_20110815.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30086/m_3008606_ne_16_1_20110815.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30086/m_3008606_ne_16_1_20110815.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-86.246875, 30.934419], [-86.246339, 31.002692], [-86.315627, 31.003078], [-86.316114, 30.934803], [-86.246875, 30.934419]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-15T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [565333, 3422577, 571948, 3430144], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7567, 6615], \"eo:cloud_cover\": 22, \"proj:transform\": [1, 0, 565333, 0, -1, 3430144, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0069\", \"bbox\": [-86.378574, 30.934452, -86.308884, 31.003042], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30086/m_3008606_nw_16_1_20110815.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30086/m_3008606_nw_16_1_20110815.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30086/m_3008606_nw_16_1_20110815.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-86.309376, 30.934452], [-86.308884, 31.00269], [-86.378132, 31.003042], [-86.378574, 30.934803], [-86.309376, 30.934452]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-15T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [559366, 3422542, 565977, 3430105], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7563, 6611], \"eo:cloud_cover\": 97, \"proj:transform\": [1, 0, 559366, 0, -1, 3430105, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0070\", \"bbox\": [-86.191196, 30.934361, -86.121261, 31.00314], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30086/m_3008607_ne_16_1_20110815.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30086/m_3008607_ne_16_1_20110815.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30086/m_3008607_ne_16_1_20110815.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-86.121886, 30.934361], [-86.121261, 31.002686], [-86.19062, 31.00314], [-86.191196, 30.934813], [-86.121886, 30.934361]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-15T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [577267, 3422658, 583889, 3430231], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7573, 6622], \"eo:cloud_cover\": 32, \"proj:transform\": [1, 0, 577267, 0, -1, 3430231, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0071\", \"bbox\": [-86.253655, 30.934391, -86.183805, 31.00311], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30086/m_3008607_nw_16_1_20110815.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30086/m_3008607_nw_16_1_20110815.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30086/m_3008607_nw_16_1_20110815.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-86.184386, 30.934391], [-86.183805, 31.002691], [-86.253123, 31.00311], [-86.253655, 30.93481], [-86.184386, 30.934391]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-15T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [571300, 3422616, 577918, 3430186], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7570, 6618], \"eo:cloud_cover\": 68, \"proj:transform\": [1, 0, 571300, 0, -1, 3430186, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0072\", \"bbox\": [-86.06628, 30.9343, -85.996173, 31.003207], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30086/m_3008608_ne_16_1_20110815.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30086/m_3008608_ne_16_1_20110815.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30086/m_3008608_ne_16_1_20110815.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-85.996889, 30.9343], [-85.996173, 31.002686], [-86.065614, 31.003207], [-86.06628, 30.93482], [-85.996889, 30.9343]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-15T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [589201, 3422752, 595831, 3430332], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7580, 6630], \"eo:cloud_cover\": 30, \"proj:transform\": [1, 0, 589201, 0, -1, 3430332, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0073\", \"bbox\": [-86.128738, 30.934327, -86.058717, 31.003175], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30086/m_3008608_nw_16_1_20110815.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30086/m_3008608_nw_16_1_20110815.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30086/m_3008608_nw_16_1_20110815.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-86.059387, 30.934327], [-86.058717, 31.002688], [-86.128117, 31.003175], [-86.128738, 30.934813], [-86.059387, 30.934327]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-15T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [583234, 3422703, 589860, 3430280], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7577, 6626], \"eo:cloud_cover\": 3, \"proj:transform\": [1, 0, 583234, 0, -1, 3430280, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0074\", \"bbox\": [-85.816455, 30.934167, -85.746007, 31.00333], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30085/m_3008502_ne_16_1_20110815.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30085/m_3008502_ne_16_1_20110815.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30085/m_3008502_ne_16_1_20110815.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-85.746902, 30.934167], [-85.746007, 31.002673], [-85.815609, 31.00333], [-85.816455, 30.934822], [-85.746902, 30.934167]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-15T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [613069, 3422979, 619715, 3430573], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7594, 6646], \"eo:cloud_cover\": 52, \"proj:transform\": [1, 0, 613069, 0, -1, 3430573, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0075\", \"bbox\": [-86.003823, 30.934269, -85.933631, 31.003236], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30085/m_3008501_nw_16_1_20110815.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30085/m_3008501_nw_16_1_20110815.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30085/m_3008501_nw_16_1_20110815.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-85.934391, 30.934269], [-85.933631, 31.002681], [-86.003112, 31.003236], [-86.003823, 30.934823], [-85.934391, 30.934269]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-15T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [595168, 3422804, 601802, 3430387], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7583, 6634], \"eo:cloud_cover\": 8, \"proj:transform\": [1, 0, 595168, 0, -1, 3430387, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0076\", \"bbox\": [-85.941366, 30.934235, -85.871089, 31.00327], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30085/m_3008501_ne_16_1_20110815.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30085/m_3008501_ne_16_1_20110815.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30085/m_3008501_ne_16_1_20110815.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-85.871894, 30.934235], [-85.871089, 31.002681], [-85.940611, 31.00327], [-85.941366, 30.934823], [-85.871894, 30.934235]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-15T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [601135, 3422859, 607773, 3430446], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7587, 6638], \"eo:cloud_cover\": 59, \"proj:transform\": [1, 0, 601135, 0, -1, 3430446, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0077\", \"bbox\": [-86.753316, 30.934648, -86.684141, 31.002851], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30086/m_3008603_nw_16_1_20110815.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30086/m_3008603_nw_16_1_20110815.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30086/m_3008603_nw_16_1_20110815.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-86.684365, 30.934648], [-86.684141, 31.0027], [-86.753141, 31.002851], [-86.753316, 30.934798], [-86.684365, 30.934648]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-15T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [523566, 3422402, 530153, 3429944], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7542, 6587], \"eo:cloud_cover\": 64, \"proj:transform\": [1, 0, 523566, 0, -1, 3429944, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0078\", \"bbox\": [-86.690855, 30.934615, -86.621594, 31.002887], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30086/m_3008603_ne_16_1_20110815.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30086/m_3008603_ne_16_1_20110815.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30086/m_3008603_ne_16_1_20110815.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-86.621863, 30.934615], [-86.621594, 31.002704], [-86.690635, 31.002887], [-86.690855, 30.934799], [-86.621863, 30.934615]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-15T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [529533, 3422417, 536124, 3429963], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7546, 6591], \"eo:cloud_cover\": 21, \"proj:transform\": [1, 0, 529533, 0, -1, 3429963, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0079\", \"bbox\": [-85.629082, 30.934072, -85.558378, 31.003423], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30085/m_3008504_nw_16_1_20110815.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30085/m_3008504_nw_16_1_20110815.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30085/m_3008504_nw_16_1_20110815.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-85.559409, 30.934072], [-85.558378, 31.002663], [-85.628101, 31.003423], [-85.629082, 30.934829], [-85.559409, 30.934072]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-15T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [630971, 3423185, 637629, 3430789], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7604, 6658], \"eo:cloud_cover\": 65, \"proj:transform\": [1, 0, 630971, 0, -1, 3430789, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0080\", \"bbox\": [-85.566619, 30.934046, -85.49583, 31.003456], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30085/m_3008504_ne_16_1_20110815.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30085/m_3008504_ne_16_1_20110815.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30085/m_3008504_ne_16_1_20110815.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-85.496906, 30.934046], [-85.49583, 31.002662], [-85.565593, 31.003456], [-85.566619, 30.934838], [-85.496906, 30.934046]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-15T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [636939, 3423261, 643601, 3430868], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7607, 6662], \"eo:cloud_cover\": 95, \"proj:transform\": [1, 0, 636939, 0, -1, 3430868, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0081\", \"bbox\": [-85.753989, 30.934141, -85.683456, 31.003364], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30085/m_3008503_nw_16_1_20110815.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30085/m_3008503_nw_16_1_20110815.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30085/m_3008503_nw_16_1_20110815.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-85.684396, 30.934141], [-85.683456, 31.002673], [-85.753099, 31.003364], [-85.753989, 30.934831], [-85.684396, 30.934141]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-15T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [619037, 3423045, 625687, 3430642], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7597, 6650], \"eo:cloud_cover\": 39, \"proj:transform\": [1, 0, 619037, 0, -1, 3430642, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0082\", \"bbox\": [-85.691535, 30.934103, -85.620917, 31.003395], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30085/m_3008503_ne_16_1_20110815.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30085/m_3008503_ne_16_1_20110815.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30085/m_3008503_ne_16_1_20110815.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-85.621902, 30.934103], [-85.620917, 31.002669], [-85.6906, 31.003395], [-85.691535, 30.934827], [-85.621902, 30.934103]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-15T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [625004, 3423113, 631658, 3430714], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7601, 6654], \"eo:cloud_cover\": 26, \"proj:transform\": [1, 0, 625004, 0, -1, 3430714, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0001\", \"bbox\": [-85.441706, 30.933975, -85.370747, 31.003522], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30085/m_3008505_ne_16_1_20110825.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30085/m_3008505_ne_16_1_20110825.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30085/m_3008505_ne_16_1_20110825.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-85.371913, 30.933975], [-85.370747, 31.00266], [-85.440589, 31.003522], [-85.441706, 30.934836], [-85.371913, 30.933975]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-25T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [648874, 3423421, 655544, 3431036], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7615, 6670], \"eo:cloud_cover\": 89, \"proj:transform\": [1, 0, 648874, 0, -1, 3431036, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0002\", \"bbox\": [-85.504167, 30.934008, -85.433293, 31.003486], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30085/m_3008505_nw_16_1_20110825.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30085/m_3008505_nw_16_1_20110825.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30085/m_3008505_nw_16_1_20110825.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-85.434414, 30.934008], [-85.433293, 31.002658], [-85.503096, 31.003486], [-85.504167, 30.934834], [-85.434414, 30.934008]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-25T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [642906, 3423339, 649572, 3430950], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7611, 6666], \"eo:cloud_cover\": 33, \"proj:transform\": [1, 0, 642906, 0, -1, 3430950, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0083\", \"bbox\": [-85.87891, 30.934198, -85.808547, 31.003302], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30085/m_3008502_nw_16_1_20110815.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30085/m_3008502_nw_16_1_20110815.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30085/m_3008502_nw_16_1_20110815.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-85.809397, 30.934198], [-85.808547, 31.002679], [-85.87811, 31.003302], [-85.87891, 30.934819], [-85.809397, 30.934198]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-15T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [607102, 3422917, 613744, 3430508], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7591, 6642], \"eo:cloud_cover\": 26, \"proj:transform\": [1, 0, 607102, 0, -1, 3430508, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0086\", \"bbox\": [-87.816197, 30.87189, -87.746352, 30.940609], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008702_se_16_1_20110801.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008702_se_16_1_20110801.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008702_se_16_1_20110801.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.746352, 30.872308], [-87.746882, 30.940609], [-87.816197, 30.94019], [-87.815618, 30.87189], [-87.746352, 30.872308]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-01T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [422031, 3415689, 428653, 3423259], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7570, 6622], \"eo:cloud_cover\": 40, \"proj:transform\": [1, 0, 422031, 0, -1, 3423259, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0087\", \"bbox\": [-87.753661, 30.934419, -87.683886, 31.003078], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008703_nw_16_1_20110801.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008703_nw_16_1_20110801.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008703_nw_16_1_20110801.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.683886, 30.934803], [-87.684373, 31.003078], [-87.753661, 31.002692], [-87.753125, 30.934419], [-87.683886, 30.934803]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-01T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [428052, 3422577, 434667, 3430144], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7567, 6615], \"eo:cloud_cover\": 36, \"proj:transform\": [1, 0, 428052, 0, -1, 3430144, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0088\", \"bbox\": [-87.753652, 30.871926, -87.683891, 30.940576], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008703_sw_16_1_20110801.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008703_sw_16_1_20110801.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008703_sw_16_1_20110801.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.683891, 30.87231], [-87.684377, 30.940576], [-87.753652, 30.94019], [-87.753117, 30.871926], [-87.683891, 30.87231]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-01T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [428006, 3415651, 434624, 3423217], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7566, 6618], \"eo:cloud_cover\": 23, \"proj:transform\": [1, 0, 428006, 0, -1, 3423217, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0089\", \"bbox\": [-87.816182, 30.55939, -87.746368, 30.6281], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008726_ne_16_1_20110731.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008726_ne_16_1_20110731.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008726_ne_16_1_20110731.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.746368, 30.559805], [-87.746892, 30.6281], [-87.816182, 30.627684], [-87.815609, 30.55939], [-87.746368, 30.559805]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-07-31T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [421780, 3381056, 428421, 3388625], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7569, 6641], \"eo:cloud_cover\": 36, \"proj:transform\": [1, 0, 421780, 0, -1, 3388625, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0090\", \"bbox\": [-87.81619, 30.809396, -87.746349, 30.878106], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008710_ne_16_1_20110731.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008710_ne_16_1_20110731.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008710_ne_16_1_20110731.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.746349, 30.809814], [-87.746878, 30.878106], [-87.81619, 30.877688], [-87.815612, 30.809396], [-87.746349, 30.809814]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-07-31T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [421981, 3408763, 428607, 3416332], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7569, 6626], \"eo:cloud_cover\": 84, \"proj:transform\": [1, 0, 421981, 0, -1, 3416332, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0091\", \"bbox\": [-87.816185, 30.621895, -87.746357, 30.690605], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008718_se_16_1_20110731.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008718_se_16_1_20110731.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008718_se_16_1_20110731.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.746357, 30.62231], [-87.746882, 30.690605], [-87.816185, 30.690188], [-87.815611, 30.621895], [-87.746357, 30.62231]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-07-31T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [421830, 3387983, 428468, 3395552], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7569, 6638], \"eo:cloud_cover\": 31, \"proj:transform\": [1, 0, 421830, 0, -1, 3395552, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0092\", \"bbox\": [-87.816194, 30.746893, -87.746358, 30.815603], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008710_se_16_1_20110731.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008710_se_16_1_20110731.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008710_se_16_1_20110731.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.746358, 30.74731], [-87.746885, 30.815603], [-87.816194, 30.815185], [-87.815618, 30.746893], [-87.746358, 30.74731]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-07-31T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [421930, 3401836, 428560, 3409405], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7569, 6630], \"eo:cloud_cover\": 24, \"proj:transform\": [1, 0, 421930, 0, -1, 3409405, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0093\", \"bbox\": [-87.753639, 30.559423, -87.683911, 30.628074], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008727_nw_16_1_20110731.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008727_nw_16_1_20110731.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008727_nw_16_1_20110731.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.683911, 30.559805], [-87.68439, 30.628074], [-87.753639, 30.627692], [-87.753111, 30.559423], [-87.683911, 30.559805]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-07-31T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [427774, 3381018, 434411, 3388584], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7566, 6637], \"eo:cloud_cover\": 61, \"proj:transform\": [1, 0, 427774, 0, -1, 3388584, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0094\", \"bbox\": [-87.753654, 30.809423, -87.683898, 30.878073], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008711_nw_16_1_20110731.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008711_nw_16_1_20110731.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008711_nw_16_1_20110731.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.683898, 30.809806], [-87.684382, 30.878073], [-87.753654, 30.877688], [-87.75312, 30.809423], [-87.683898, 30.809806]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-07-31T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [427959, 3408724, 434581, 3416290], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7566, 6622], \"eo:cloud_cover\": 29, \"proj:transform\": [1, 0, 427959, 0, -1, 3416290, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0095\", \"bbox\": [-87.753646, 30.746928, -87.683895, 30.815579], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008711_sw_16_1_20110731.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008711_sw_16_1_20110731.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008711_sw_16_1_20110731.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.683895, 30.747312], [-87.684378, 30.815579], [-87.753646, 30.815194], [-87.753114, 30.746928], [-87.683895, 30.747312]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-07-31T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [427913, 3401798, 434539, 3409364], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7566, 6626], \"eo:cloud_cover\": 63, \"proj:transform\": [1, 0, 427913, 0, -1, 3409364, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0096\", \"bbox\": [-87.753639, 30.684424, -87.683903, 30.753075], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008719_nw_16_1_20110731.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008719_nw_16_1_20110731.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008719_nw_16_1_20110731.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.683903, 30.684807], [-87.684385, 30.753075], [-87.753639, 30.752691], [-87.753109, 30.684424], [-87.683903, 30.684807]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-07-31T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [427867, 3394871, 434496, 3402437], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7566, 6629], \"eo:cloud_cover\": 26, \"proj:transform\": [1, 0, 427867, 0, -1, 3402437, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0097\", \"bbox\": [-87.816189, 30.68439, -87.746357, 30.753109], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008718_ne_16_1_20110731.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008718_ne_16_1_20110731.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008718_ne_16_1_20110731.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.746357, 30.684806], [-87.746883, 30.753109], [-87.816189, 30.752692], [-87.815614, 30.68439], [-87.746357, 30.684806]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-07-31T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [421880, 3394909, 428514, 3402479], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7570, 6634], \"eo:cloud_cover\": 1, \"proj:transform\": [1, 0, 421880, 0, -1, 3402479, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0098\", \"bbox\": [-87.753644, 30.621929, -87.683901, 30.69057], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008719_sw_16_1_20110731.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008719_sw_16_1_20110731.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008719_sw_16_1_20110731.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.683901, 30.622311], [-87.684382, 30.69057], [-87.753644, 30.690187], [-87.753115, 30.621929], [-87.683901, 30.622311]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-07-31T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [427820, 3387945, 434454, 3395510], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7565, 6634], \"eo:cloud_cover\": 73, \"proj:transform\": [1, 0, 427820, 0, -1, 3395510, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0099\", \"bbox\": [-87.753635, 30.496927, -87.683911, 30.565569], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008727_sw_16_1_20110731.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008727_sw_16_1_20110731.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008727_sw_16_1_20110731.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.683911, 30.497308], [-87.684389, 30.565569], [-87.753635, 30.565186], [-87.753109, 30.496927], [-87.683911, 30.497308]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-07-31T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [427728, 3374092, 434369, 3381657], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7565, 6641], \"eo:cloud_cover\": 90, \"proj:transform\": [1, 0, 427728, 0, -1, 3381657, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0084\", \"bbox\": [-85.316796, 30.93392, -85.245667, 31.003585], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30085/m_3008506_ne_16_1_20110802.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30085/m_3008506_ne_16_1_20110802.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30085/m_3008506_ne_16_1_20110802.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-85.246924, 30.93392], [-85.245667, 31.002654], [-85.315589, 31.003585], [-85.316796, 30.934848], [-85.246924, 30.93392]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-02T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [660809, 3423596, 667487, 3431217], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7621, 6678], \"eo:cloud_cover\": 52, \"proj:transform\": [1, 0, 660809, 0, -1, 3431217, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0085\", \"bbox\": [-87.816195, 30.934391, -87.746345, 31.00311], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008702_ne_16_1_20110801.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008702_ne_16_1_20110801.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008702_ne_16_1_20110801.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.746345, 30.93481], [-87.746877, 31.00311], [-87.816195, 31.002691], [-87.815614, 30.934391], [-87.746345, 30.93481]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-01T00:00:00Z\", \"naip:year\": \"2012\", \"proj:bbox\": [422082, 3422616, 428700, 3430186], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"xx\", \"proj:shape\": [7570, 6618], \"eo:cloud_cover\": 3, \"proj:transform\": [1, 0, 422082, 0, -1, 3430186, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0100\", \"bbox\": [-87.816179, 30.496894, -87.74637, 30.565604], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008726_se_16_1_20110731.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008726_se_16_1_20110731.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008726_se_16_1_20110731.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.74637, 30.497308], [-87.746892, 30.565604], [-87.816179, 30.565188], [-87.815608, 30.496894], [-87.74637, 30.497308]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-07-31T00:00:00Z\", \"naip:year\": 2013, \"proj:bbox\": [421730, 3374130, 428375, 3381699], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"zz\", \"proj:shape\": [7569, 6645], \"eo:cloud_cover\": 50, \"proj:transform\": [1, 0, 421730, 0, -1, 3381699, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n" }, { "path": "src/pgstac/tests/testdata/items.pgcopy", "content": "pgstac-test-item-0003\t0103000020E610000001000000050000005B3FFD67CD5355C0C4211B4817EF3E400CE6AF90B95355C0A112D731AE003F4004C93B87325855C0BEBC00FBE8003F40FA0AD28C455855C000E5EFDE51EF3E405B3FFD67CD5355C0C4211B4817EF3E40\tpgstac-test-collection\t2011-08-25 00:00:00+00\t2011-08-25 00:00:00+00\t{\"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30085/m_3008506_nw_16_1_20110825.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30085/m_3008506_nw_16_1_20110825.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30085/m_3008506_nw_16_1_20110825.200.jpg\"}}, \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-25T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [654842, 3423507, 661516, 3431125], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7618, 6674], \"eo:cloud_cover\": 28, \"proj:transform\": [1, 0, 654842, 0, -1, 3431125, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}\npgstac-test-item-0004\t0103000020E610000001000000050000006364C91CCBC755C0DE76A1B94EEF3E4062307F85CCC755C002A08A1BB7003F403E7958A835CC55C0E75608ABB1003F40E695EB6D33CC55C0C32D1F4949EF3E406364C91CCBC755C0DE76A1B94EEF3E40\tpgstac-test-collection\t2011-08-24 00:00:00+00\t2011-08-24 00:00:00+00\t{\"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008707_ne_16_1_20110824.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008707_ne_16_1_20110824.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008707_ne_16_1_20110824.200.jpg\"}}, \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-24T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [481788, 3422382, 488367, 3429918], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7536, 6579], \"eo:cloud_cover\": 23, \"proj:transform\": [1, 0, 481788, 0, -1, 3429918, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}\npgstac-test-item-0005\t0103000020E61000000100000005000000C1525DC0CBC355C03D7FDAA84EEF3E40D97A8670CCC355C0C7D5C8AEB4003F40AF06280D35C855C06478EC67B1003F402785798F33C855C0D921FE614BEF3E40C1525DC0CBC355C03D7FDAA84EEF3E40\tpgstac-test-collection\t2011-08-24 00:00:00+00\t2011-08-24 00:00:00+00\t{\"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008708_nw_16_1_20110824.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008708_nw_16_1_20110824.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008708_nw_16_1_20110824.200.jpg\"}}, \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-24T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [487758, 3422377, 494334, 3429909], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7532, 6576], \"eo:cloud_cover\": 3, \"proj:transform\": [1, 0, 487758, 0, -1, 3429909, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}\npgstac-test-item-0006\t0103000020E61000000100000005000000F20A444FCACB55C03D7FDAA84EEF3E40C138B874CCCB55C054C6BFCFB8003F40DE567A6D36D055C0E199D024B1003F409ECF807A33D055C0CA52EBFD46EF3E40F20A444FCACB55C03D7FDAA84EEF3E40\tpgstac-test-collection\t2011-08-24 00:00:00+00\t2011-08-24 00:00:00+00\t{\"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008707_nw_16_1_20110824.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008707_nw_16_1_20110824.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008707_nw_16_1_20110824.200.jpg\"}}, \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-24T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [475817, 3422390, 482401, 3429929], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7539, 6584], \"eo:cloud_cover\": 100, \"proj:transform\": [1, 0, 475817, 0, -1, 3429929, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}\npgstac-test-item-0007\t0103000020E61000000100000005000000CF68AB92C8D755C01F662FDB4E9F3E40850662D9CCD755C0F8AA9509BFB03E405A2A6F4738DC55C05EBBB4E1B0B03E401BF1643733DC55C086764EB3409F3E40CF68AB92C8D755C01F662FDB4E9F3E40\tpgstac-test-collection\t2011-08-17 00:00:00+00\t2011-08-17 00:00:00+00\t{\"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008721_se_16_1_20110817.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008721_se_16_1_20110817.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008721_se_16_1_20110817.200.jpg\"}}, \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-17T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [457770, 3387803, 464384, 3395352], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7549, 6614], \"eo:cloud_cover\": 59, \"proj:transform\": [1, 0, 457770, 0, -1, 3395352, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}\npgstac-test-item-0008\t0103000020E61000000100000005000000CF8250DEC7DB55C0433C122F4F9F3E406EC493DDCCDB55C00E9F7422C1B03E405A10CAFB38E055C0BDC3EDD0B0B03E404B75012F33E055C0F2608BDD3E9F3E40CF8250DEC7DB55C0433C122F4F9F3E40\tpgstac-test-collection\t2011-08-17 00:00:00+00\t2011-08-17 00:00:00+00\t{\"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008721_sw_16_1_20110817.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008721_sw_16_1_20110817.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008721_sw_16_1_20110817.200.jpg\"}}, \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-17T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [451780, 3387825, 458398, 3395377], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7552, 6618], \"eo:cloud_cover\": 64, \"proj:transform\": [1, 0, 451780, 0, -1, 3395377, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}\npgstac-test-item-0009\t0103000020E61000000100000005000000FF06EDD5C7DB55C06155BDFC4EAF3E409D4830D5CCDB55C02CB81FF0C0C03E405A10CAFB38E055C03BE5D18DB0C03E403333333333E055C0107A36AB3EAF3E40FF06EDD5C7DB55C06155BDFC4EAF3E40\tpgstac-test-collection\t2011-08-17 00:00:00+00\t2011-08-17 00:00:00+00\t{\"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008721_nw_16_1_20110817.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008721_nw_16_1_20110817.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008721_nw_16_1_20110817.200.jpg\"}}, \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-17T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [451811, 3394751, 458425, 3402303], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7552, 6614], \"eo:cloud_cover\": 61, \"proj:transform\": [1, 0, 451811, 0, -1, 3402303, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}\npgstac-test-item-0010\t0103000020E61000000100000005000000D53F8864C8D755C03D7FDAA84EAF3E408BDD3EABCCD755C015C440D7BEC03E402BA6D24F38DC55C07CD45FAFB0C03E40EC6CC83F33DC55C04487C09140AF3E40D53F8864C8D755C03D7FDAA84EAF3E40\tpgstac-test-collection\t2011-08-17 00:00:00+00\t2011-08-17 00:00:00+00\t{\"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008721_ne_16_1_20110817.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008721_ne_16_1_20110817.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008721_ne_16_1_20110817.200.jpg\"}}, \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-17T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [457797, 3394729, 464408, 3402278], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7549, 6611], \"eo:cloud_cover\": 31, \"proj:transform\": [1, 0, 457797, 0, -1, 3402278, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}\npgstac-test-item-0011\t0103000020E61000000100000005000000931CB0ABC9CF55C0C05DF6EB4EEF3E404AEEB089CCCF55C0CAC2D7D7BA003F406DC9AA0837D455C0FFB27BF2B0003F40459E245D33D455C09545611745EF3E40931CB0ABC9CF55C0C05DF6EB4EEF3E40\tpgstac-test-collection\t2011-08-17 00:00:00+00\t2011-08-17 00:00:00+00\t{\"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008706_ne_16_1_20110817.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008706_ne_16_1_20110817.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008706_ne_16_1_20110817.200.jpg\"}}, \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-17T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [469847, 3422402, 476434, 3429944], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7542, 6587], \"eo:cloud_cover\": 41, \"proj:transform\": [1, 0, 469847, 0, -1, 3429944, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}\npgstac-test-item-0012\t0103000020E61000000100000005000000C3D4963AC8D755C01F662FDB4EEF3E4032ACE28DCCD755C0BC783F6EBF003F40B45BCB6438DC55C082919735B1003F40D42AFA4333DC55C0E57E87A240EF3E40C3D4963AC8D755C01F662FDB4EEF3E40\tpgstac-test-collection\t2011-08-17 00:00:00+00\t2011-08-17 00:00:00+00\t{\"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008705_ne_16_1_20110817.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008705_ne_16_1_20110817.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008705_ne_16_1_20110817.200.jpg\"}}, \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-17T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [457906, 3422435, 464501, 3429985], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7550, 6595], \"eo:cloud_cover\": 4, \"proj:transform\": [1, 0, 457906, 0, -1, 3429985, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}\npgstac-test-item-0013\t0103000020E61000000100000005000000342E1C08C9D355C06155BDFC4EEF3E40BB61DBA2CCD355C06495D233BD003F400DA7CCCD37D855C082919735B1003F40151A886533D855C0DE59BBED42EF3E40342E1C08C9D355C06155BDFC4EEF3E40\tpgstac-test-collection\t2011-08-17 00:00:00+00\t2011-08-17 00:00:00+00\t{\"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008706_nw_16_1_20110817.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008706_nw_16_1_20110817.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008706_nw_16_1_20110817.200.jpg\"}}, \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-17T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [463876, 3422417, 470467, 3429963], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7546, 6591], \"eo:cloud_cover\": 2, \"proj:transform\": [1, 0, 463876, 0, -1, 3429963, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}\npgstac-test-item-0014\t0103000020E6100000010000000500000089F02F82C6E355C06155BDFC4E7F3E4026FE28EACCE355C0B9A81611C5903E40EE3F321D3AE855C0F9F5436CB0903E40C896E5EB32E855C0A1A2EA573A7F3E4089F02F82C6E355C06155BDFC4E7F3E40\tpgstac-test-collection\t2011-08-16 00:00:00+00\t2011-08-16 00:00:00+00\t{\"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008728_sw_16_1_20110816.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008728_sw_16_1_20110816.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008728_sw_16_1_20110816.200.jpg\"}}, \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [439724, 3374025, 446357, 3381584], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7559, 6633], \"eo:cloud_cover\": 17, \"proj:transform\": [1, 0, 439724, 0, -1, 3381584, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}\npgstac-test-item-0015\t0103000020E61000000100000005000000D97A8670CCB755C0D921FE614BEF3E4051F9D7F2CAB755C06478EC67B1003F402785798F33BC55C0C7D5C8AEB4003F403FADA23F34BC55C03D7FDAA84EEF3E40D97A8670CCB755C0D921FE614BEF3E40\tpgstac-test-collection\t2011-08-16 00:00:00+00\t2011-08-16 00:00:00+00\t{\"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30086/m_3008601_ne_16_1_20110816.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30086/m_3008601_ne_16_1_20110816.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30086/m_3008601_ne_16_1_20110816.200.jpg\"}}, \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [505666, 3422377, 512242, 3429909], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7532, 6576], \"eo:cloud_cover\": 54, \"proj:transform\": [1, 0, 505666, 0, -1, 3429909, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}\npgstac-test-item-0016\t0103000020E6100000010000000500000091B41B7DCCBB55C00E2F88484DEF3E40D9942BBCCBBB55C0A5677A89B1003F40868DB27E33C055C051D9B0A6B2003F40CE531D7233C055C019A9F7544EEF3E4091B41B7DCCBB55C00E2F88484DEF3E40\tpgstac-test-collection\t2011-08-16 00:00:00+00\t2011-08-16 00:00:00+00\t{\"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30086/m_3008601_nw_16_1_20110816.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30086/m_3008601_nw_16_1_20110816.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30086/m_3008601_nw_16_1_20110816.200.jpg\"}}, \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [499700, 3422375, 506271, 3429904], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7529, 6571], \"eo:cloud_cover\": 13, \"proj:transform\": [1, 0, 499700, 0, -1, 3429904, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}\npgstac-test-item-0017\t0103000020E6100000010000000500000062307F85CCAF55C0CA52EBFD46EF3E4022A98592C9AF55C0E199D024B1003F403FC7478B33B455C054C6BFCFB8003F400EF5BBB035B455C03D7FDAA84EEF3E4062307F85CCAF55C0CA52EBFD46EF3E40\tpgstac-test-collection\t2011-08-16 00:00:00+00\t2011-08-16 00:00:00+00\t{\"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30086/m_3008602_ne_16_1_20110816.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30086/m_3008602_ne_16_1_20110816.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30086/m_3008602_ne_16_1_20110816.200.jpg\"}}, \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [517599, 3422390, 524183, 3429929], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7539, 6584], \"eo:cloud_cover\": 59, \"proj:transform\": [1, 0, 517599, 0, -1, 3429929, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}\npgstac-test-item-0018\t0103000020E610000001000000050000001A6A1492CCB355C0C32D1F4949EF3E40C286A757CAB355C0E75608ABB1003F409ECF807A33B855C002A08A1BB7003F409D9B36E334B855C0DE76A1B94EEF3E401A6A1492CCB355C0C32D1F4949EF3E40\tpgstac-test-collection\t2011-08-16 00:00:00+00\t2011-08-16 00:00:00+00\t{\"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30086/m_3008602_nw_16_1_20110816.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30086/m_3008602_nw_16_1_20110816.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30086/m_3008602_nw_16_1_20110816.200.jpg\"}}, \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [511633, 3422382, 518212, 3429918], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7536, 6579], \"eo:cloud_cover\": 29, \"proj:transform\": [1, 0, 511633, 0, -1, 3429918, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}\npgstac-test-item-0019\t0103000020E610000001000000050000001D3A3DEFC6DF55C0433C122F4FEF3E404417D4B7CCDF55C02593533BC3003F400C59DDEA39E455C03BE5D18DB0003F407522C15433E455C0F98557923CEF3E401D3A3DEFC6DF55C0433C122F4FEF3E40\tpgstac-test-collection\t2011-08-16 00:00:00+00\t2011-08-16 00:00:00+00\t{\"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008704_ne_16_1_20110816.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008704_ne_16_1_20110816.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008704_ne_16_1_20110816.200.jpg\"}}, \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [445964, 3422482, 452567, 3430038], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7556, 6603], \"eo:cloud_cover\": 52, \"proj:transform\": [1, 0, 445964, 0, -1, 3430038, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}\npgstac-test-item-0020\t0103000020E61000000100000005000000D68D7747C6E355C0070ABC934FEF3E40E50E9BC8CCE355C0DC7EF964C5003F40CA4FAA7D3AE855C03BE5D18DB0003F4063B7CF2A33E855C006685BCD3AEF3E40D68D7747C6E355C0070ABC934FEF3E40\tpgstac-test-collection\t2011-08-16 00:00:00+00\t2011-08-16 00:00:00+00\t{\"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008704_nw_16_1_20110816.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008704_nw_16_1_20110816.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008704_nw_16_1_20110816.200.jpg\"}}, \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [439994, 3422511, 446600, 3430070], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7559, 6606], \"eo:cloud_cover\": 39, \"proj:transform\": [1, 0, 439994, 0, -1, 3430070, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}\npgstac-test-item-0021\t0103000020E610000001000000050000008FC70C54C6E355C0252367614FDF3E409D4830D5CCE355C0FA97A432C5F03E40E29178793AE855C058FE7C5BB0F03E4063B7CF2A33E855C02481069B3ADF3E408FC70C54C6E355C0252367614FDF3E40\tpgstac-test-collection\t2011-08-16 00:00:00+00\t2011-08-16 00:00:00+00\t{\"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008704_sw_16_1_20110816.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008704_sw_16_1_20110816.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008704_sw_16_1_20110816.200.jpg\"}}, \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [439955, 3415584, 446565, 3423143], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7559, 6610], \"eo:cloud_cover\": 29, \"proj:transform\": [1, 0, 439955, 0, -1, 3423143, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}\npgstac-test-item-0022\t0103000020E610000001000000050000007C28D192C7DB55C0A2444B1E4FEF3E40BB61DBA2CCDB55C032755776C1003F406C7BBB2539E055C09FAA4203B1003F40A4A65D4C33E055C0107A36AB3EEF3E407C28D192C7DB55C0A2444B1E4FEF3E40\tpgstac-test-collection\t2011-08-16 00:00:00+00\t2011-08-16 00:00:00+00\t{\"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008705_nw_16_1_20110816.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008705_nw_16_1_20110816.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008705_nw_16_1_20110816.200.jpg\"}}, \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [451935, 3422457, 458534, 3430010], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7553, 6599], \"eo:cloud_cover\": 84, \"proj:transform\": [1, 0, 451935, 0, -1, 3430010, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}\npgstac-test-item-0023\t0103000020E6100000010000000500000032ACE28DCCBF55C019A9F7544EEF3E407A724D81CCBF55C051D9B0A6B2003F40276BD44334C455C0A5677A89B1003F406F4BE48233C455C00E2F88484DEF3E4032ACE28DCCBF55C019A9F7544EEF3E40\tpgstac-test-collection\t2011-08-16 00:00:00+00\t2011-08-16 00:00:00+00\t{\"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008708_ne_16_1_20110816.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008708_ne_16_1_20110816.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008708_ne_16_1_20110816.200.jpg\"}}, \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [493729, 3422375, 500300, 3429904], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7529, 6571], \"eo:cloud_cover\": 56, \"proj:transform\": [1, 0, 493729, 0, -1, 3429904, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}\npgstac-test-item-0024\t0103000020E6100000010000000500000001BD70E7C2F755C048F949B54FCF3E407F2F8507CDF755C0EF3A1BF2CFE03E40E6E61BD13DFC55C0D61F6118B0E03E40105D50DF32FC55C0D0D556EC2FCF3E4001BD70E7C2F755C048F949B54FCF3E40\tpgstac-test-collection\t2011-08-16 00:00:00+00\t2011-08-16 00:00:00+00\t{\"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008709_ne_16_1_20110816.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008709_ne_16_1_20110816.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008709_ne_16_1_20110816.200.jpg\"}}, \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [410024, 3408849, 416657, 3416426], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7577, 6633], \"eo:cloud_cover\": 58, \"proj:transform\": [1, 0, 410024, 0, -1, 3416426, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}\npgstac-test-item-0025\t0103000020E61000000100000005000000A9A5B915C2FB55C0ADBEBA2A50CF3E40F6798CF2CCFB55C0A626C11BD2E03E40B648DA8D3E0056C035289A07B0E03E40F81A82E3320056C07DAF21382ECF3E40A9A5B915C2FB55C0ADBEBA2A50CF3E40\tpgstac-test-collection\t2011-08-16 00:00:00+00\t2011-08-16 00:00:00+00\t{\"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008709_nw_16_1_20110816.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008709_nw_16_1_20110816.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008709_nw_16_1_20110816.200.jpg\"}}, \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [404045, 3408898, 410683, 3416478], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7580, 6638], \"eo:cloud_cover\": 65, \"proj:transform\": [1, 0, 404045, 0, -1, 3416478, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}\npgstac-test-item-0026\t0103000020E61000000100000005000000EA7AA2EBC2F755C0ADBEBA2A50BF3E407F2F8507CDF755C0AD4B8DD0CFD03E4046EF54C03DFC55C09430D3F6AFD03E403FE1ECD632FC55C0359BC76130BF3E40EA7AA2EBC2F755C0ADBEBA2A50BF3E40\tpgstac-test-collection\t2011-08-16 00:00:00+00\t2011-08-16 00:00:00+00\t{\"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008709_se_16_1_20110816.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008709_se_16_1_20110816.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008709_se_16_1_20110816.200.jpg\"}}, \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [409966, 3401923, 416603, 3409499], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7576, 6637], \"eo:cloud_cover\": 52, \"proj:transform\": [1, 0, 409966, 0, -1, 3409499, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}\npgstac-test-item-0027\t0103000020E6100000010000000500000061DF4E22C2FB55C06CCF2C0950BF3E40DF37BEF6CCFB55C0653733FAD1D03E40E6CC76853E0056C09430D3F6AFD03E40F81A82E3320056C03CC093162EBF3E4061DF4E22C2FB55C06CCF2C0950BF3E40\tpgstac-test-collection\t2011-08-16 00:00:00+00\t2011-08-16 00:00:00+00\t{\"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008709_sw_16_1_20110816.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008709_sw_16_1_20110816.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008709_sw_16_1_20110816.200.jpg\"}}, \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [403983, 3401971, 410625, 3409551], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7580, 6642], \"eo:cloud_cover\": 43, \"proj:transform\": [1, 0, 403983, 0, -1, 3409551, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}\npgstac-test-item-0028\t0103000020E6100000010000000500000060AB048BC3F355C0A80183A44FCF3E40F6798CF2CCF355C055682096CDE03E4040321D3A3DF855C0F3380CE6AFE03E40390A100533F855C087C1FC1532CF3E4060AB048BC3F355C0A80183A44FCF3E40\tpgstac-test-collection\t2011-08-16 00:00:00+00\t2011-08-16 00:00:00+00\t{\"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008710_nw_16_1_20110816.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008710_nw_16_1_20110816.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008710_nw_16_1_20110816.200.jpg\"}}, \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [416002, 3408804, 422632, 3416377], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7573, 6630], \"eo:cloud_cover\": 46, \"proj:transform\": [1, 0, 416002, 0, -1, 3416377, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}\npgstac-test-item-0029\t0103000020E6100000010000000500000060AB048BC3F355C06CCF2C0950BF3E4026FE28EACCF355C07381CB63CDD03E40B77C24253DF855C0B2497EC4AFD03E4081D07AF832F855C04B8FA67A32BF3E4060AB048BC3F355C06CCF2C0950BF3E40\tpgstac-test-collection\t2011-08-16 00:00:00+00\t2011-08-16 00:00:00+00\t{\"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008710_sw_16_1_20110816.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008710_sw_16_1_20110816.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008710_sw_16_1_20110816.200.jpg\"}}, \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [415948, 3401878, 422582, 3409450], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7572, 6634], \"eo:cloud_cover\": 42, \"proj:transform\": [1, 0, 415948, 0, -1, 3409450, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}\npgstac-test-item-0030\t0103000020E61000000100000005000000A723809BC5E755C0252367614FCF3E40B58AFED0CCE755C0946A9F8EC7E03E407104A9143BEC55C03BE5D18DB0E03E40F243A51133EC55C06C95607138CF3E40A723809BC5E755C0252367614FCF3E40\tpgstac-test-collection\t2011-08-16 00:00:00+00\t2011-08-16 00:00:00+00\t{\"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008711_ne_16_1_20110816.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008711_ne_16_1_20110816.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008711_ne_16_1_20110816.200.jpg\"}}, \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [433938, 3408689, 440556, 3416252], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7563, 6618], \"eo:cloud_cover\": 16, \"proj:transform\": [1, 0, 433938, 0, -1, 3416252, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}\npgstac-test-item-0031\t0103000020E610000001000000050000001897AAB4C5E755C048F949B54FBF3E405682C5E1CCE755C0B2834A5CC7D03E4041800C1D3BEC55C058FE7C5BB0D03E40AA7D3A1E33EC55C0906B43C538BF3E401897AAB4C5E755C048F949B54FBF3E40\tpgstac-test-collection\t2011-08-16 00:00:00+00\t2011-08-16 00:00:00+00\t{\"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008711_se_16_1_20110816.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008711_se_16_1_20110816.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008711_se_16_1_20110816.200.jpg\"}}, \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [433895, 3401763, 440517, 3409325], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7562, 6622], \"eo:cloud_cover\": 16, \"proj:transform\": [1, 0, 433895, 0, -1, 3409325, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}\npgstac-test-item-0032\t0103000020E610000001000000050000003596B036C6E355C0A2444B1E4FCF3E405C59A2B3CCE355C0BF654E97C5E03E40E29178793AE855C01DCC26C0B0E03E404B75012F33E855C0A1A2EA573ACF3E403596B036C6E355C0A2444B1E4FCF3E40\tpgstac-test-collection\t2011-08-16 00:00:00+00\t2011-08-16 00:00:00+00\t{\"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008712_nw_16_1_20110816.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008712_nw_16_1_20110816.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008712_nw_16_1_20110816.200.jpg\"}}, \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [439916, 3408657, 446531, 3416217], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7560, 6615], \"eo:cloud_cover\": 7, \"proj:transform\": [1, 0, 439916, 0, -1, 3416217, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}\npgstac-test-item-0033\t0103000020E61000000100000005000000766B990CC7DF55C0C05DF6EB4EBF3E40E50E9BC8CCDF55C0E4A3C519C3D03E40897AC1A739E455C09AED0A7DB0D03E40AA7D3A1E33E455C0179F02603CBF3E40766B990CC7DF55C0C05DF6EB4EBF3E40\tpgstac-test-collection\t2011-08-16 00:00:00+00\t2011-08-16 00:00:00+00\t{\"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008712_se_16_1_20110816.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008712_se_16_1_20110816.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008712_se_16_1_20110816.200.jpg\"}}, \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [445860, 3401702, 452474, 3409258], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7556, 6614], \"eo:cloud_cover\": 49, \"proj:transform\": [1, 0, 445860, 0, -1, 3409258, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}\npgstac-test-item-0034\t0103000020E61000000100000005000000BE4BA94BC6E355C0252367614FBF3E40149337C0CCE355C03C873254C5D03E40D026874F3AE855C03BE5D18DB0D03E4021C8410933E855C0C478CDAB3ABF3E40BE4BA94BC6E355C0252367614FBF3E40\tpgstac-test-collection\t2011-08-16 00:00:00+00\t2011-08-16 00:00:00+00\t{\"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008712_sw_16_1_20110816.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008712_sw_16_1_20110816.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008712_sw_16_1_20110816.200.jpg\"}}, \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [439878, 3401731, 446496, 3409290], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7559, 6618], \"eo:cloud_cover\": 3, \"proj:transform\": [1, 0, 439878, 0, -1, 3409290, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}\npgstac-test-item-0035\t0103000020E61000000100000005000000BAF605F4C2F755C02AE09EE74FAF3E407F2F8507CDF755C0CB64389ECFC03E408DB5BFB33DFC55C0F3380CE6AFC03E405723BBD232FC55C0F3AB394030AF3E40BAF605F4C2F755C02AE09EE74FAF3E40\tpgstac-test-collection\t2011-08-16 00:00:00+00\t2011-08-16 00:00:00+00\t{\"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008717_ne_16_1_20110816.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008717_ne_16_1_20110816.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008717_ne_16_1_20110816.200.jpg\"}}, \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [409908, 3394996, 416549, 3402572], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7576, 6641], \"eo:cloud_cover\": 33, \"proj:transform\": [1, 0, 409908, 0, -1, 3402572, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}\npgstac-test-item-0036\t0103000020E610000001000000050000001A19E42EC2FB55C02AE09EE74FAF3E40C7F5EFFACCFB55C02448A5D8D1C03E401651137D3E0056C0F3380CE6AFC03E40F81A82E3320056C0FAD005F52DAF3E401A19E42EC2FB55C02AE09EE74FAF3E40\tpgstac-test-collection\t2011-08-16 00:00:00+00\t2011-08-16 00:00:00+00\t{\"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008717_nw_16_1_20110816.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008717_nw_16_1_20110816.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008717_nw_16_1_20110816.200.jpg\"}}, \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [403921, 3395044, 410567, 3402624], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7580, 6646], \"eo:cloud_cover\": 12, \"proj:transform\": [1, 0, 403921, 0, -1, 3402624, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}\npgstac-test-item-0037\t0103000020E6100000010000000500000073309B00C3F755C048F949B54F9F3E4050ABE80FCDF755C0E97DE36BCFB03E40A5F78DAF3DFC55C01152B7B3AFB03E403FE1ECD632FC55C011C5E40D309F3E4073309B00C3F755C048F949B54F9F3E40\tpgstac-test-collection\t2011-08-16 00:00:00+00\t2011-08-16 00:00:00+00\t{\"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008717_se_16_1_20110816.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008717_se_16_1_20110816.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008717_se_16_1_20110816.200.jpg\"}}, \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [409850, 3388069, 416495, 3395645], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7576, 6645], \"eo:cloud_cover\": 54, \"proj:transform\": [1, 0, 409850, 0, -1, 3395645, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}\npgstac-test-item-0038\t0103000020E61000000100000005000000A3CEDC43C2FB55C04EB6813B509F3E407F2F8507CDFB55C0E25817B7D1B03E4004E621533E0056C0B2497EC4AFB03E40B62BF4C1320056C05F96766A2E9F3E40A3CEDC43C2FB55C04EB6813B509F3E40\tpgstac-test-collection\t2011-08-16 00:00:00+00\t2011-08-16 00:00:00+00\t{\"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008717_sw_16_1_20110816.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008717_sw_16_1_20110816.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008717_sw_16_1_20110816.200.jpg\"}}, \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [403860, 3388118, 410509, 3395697], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7579, 6649], \"eo:cloud_cover\": 77, \"proj:transform\": [1, 0, 403860, 0, -1, 3395697, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}\npgstac-test-item-0039\t0103000020E610000001000000050000005AD427B9C3F355C0E9F010C64FAF3E4038691A14CDF355C0374F75C8CDC03E4016855D143DF855C0170FEF39B0C03E40B05417F032F855C069A8514832AF3E405AD427B9C3F355C0E9F010C64FAF3E40\tpgstac-test-collection\t2011-08-16 00:00:00+00\t2011-08-16 00:00:00+00\t{\"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008718_nw_16_1_20110816.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008718_nw_16_1_20110816.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008718_nw_16_1_20110816.200.jpg\"}}, \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [415894, 3394951, 422531, 3402524], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7573, 6637], \"eo:cloud_cover\": 12, \"proj:transform\": [1, 0, 415894, 0, -1, 3402524, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}\npgstac-test-item-0040\t0103000020E610000001000000050000002B508BC1C3F355C06612F5824F9F3E4038691A14CDF355C055682096CDB03E405E4BC8073DF855C035289A07B0B03E40E0D8B3E732F855C087C1FC15329F3E402B508BC1C3F355C06612F5824F9F3E40\tpgstac-test-collection\t2011-08-16 00:00:00+00\t2011-08-16 00:00:00+00\t{\"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008718_sw_16_1_20110816.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008718_sw_16_1_20110816.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008718_sw_16_1_20110816.200.jpg\"}}, \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [415840, 3388024, 422481, 3395597], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7573, 6641], \"eo:cloud_cover\": 10, \"proj:transform\": [1, 0, 415840, 0, -1, 3395597, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}\npgstac-test-item-0041\t0103000020E61000000100000005000000779FE3A3C5E755C0252367614FAF3E40CDCCCCCCCCE755C0D5592DB0C7C03E4000917EFB3AEC55C07CD45FAFB0C03E40514CDE0033EC55C00D8D278238AF3E40779FE3A3C5E755C0252367614FAF3E40\tpgstac-test-collection\t2011-08-16 00:00:00+00\t2011-08-16 00:00:00+00\t{\"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008719_ne_16_1_20110816.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008719_ne_16_1_20110816.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008719_ne_16_1_20110816.200.jpg\"}}, \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [433853, 3394836, 440479, 3402399], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7563, 6626], \"eo:cloud_cover\": 3, \"proj:transform\": [1, 0, 433853, 0, -1, 3402399, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}\npgstac-test-item-0042\t0103000020E61000000100000005000000D0D03FC1C5E755C0A80183A44F9F3E4026FE28EACCE755C0B2834A5CC7B03E40B8CA13083BEC55C0F9F5436CB0B03E40DA01D71533EC55C0906B43C5389F3E40D0D03FC1C5E755C0A80183A44F9F3E40\tpgstac-test-collection\t2011-08-16 00:00:00+00\t2011-08-16 00:00:00+00\t{\"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008719_se_16_1_20110816.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008719_se_16_1_20110816.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008719_se_16_1_20110816.200.jpg\"}}, \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [433810, 3387910, 440440, 3395472], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7562, 6630], \"eo:cloud_cover\": 99, \"proj:transform\": [1, 0, 433810, 0, -1, 3395472, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}\npgstac-test-item-0043\t0103000020E610000001000000050000002FA52E19C7DF55C0A2444B1E4FAF3E409D4830D5CCDF55C06682E15CC3C03E40897AC1A739E455C0BDC3EDD0B0C03E40923B6C2233E455C09A7D1EA33CAF3E402FA52E19C7DF55C0A2444B1E4FAF3E40\tpgstac-test-collection\t2011-08-16 00:00:00+00\t2011-08-16 00:00:00+00\t{\"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008720_ne_16_1_20110816.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008720_ne_16_1_20110816.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008720_ne_16_1_20110816.200.jpg\"}}, \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [445825, 3394776, 452443, 3402332], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7556, 6618], \"eo:cloud_cover\": 80, \"proj:transform\": [1, 0, 445825, 0, -1, 3402332, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}\npgstac-test-item-0044\t0103000020E610000001000000050000004701A260C6E355C0014D840D4FAF3E40B58AFED0CCE355C0BF654E97C5C03E40B8E4B8533AE855C05EBBB4E1B0C03E40F243A51133E855C0A1A2EA573AAF3E404701A260C6E355C0014D840D4FAF3E40\tpgstac-test-collection\t2011-08-16 00:00:00+00\t2011-08-16 00:00:00+00\t{\"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008720_nw_16_1_20110816.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008720_nw_16_1_20110816.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008720_nw_16_1_20110816.200.jpg\"}}, \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [439839, 3394804, 446461, 3402364], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7560, 6622], \"eo:cloud_cover\": 30, \"proj:transform\": [1, 0, 439839, 0, -1, 3402364, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}\npgstac-test-item-0045\t0103000020E61000000100000005000000B85A272EC7DF55C0842BA0504F9F3E405682C5E1CCDF55C043ACFE08C3B03E407138F3AB39E455C09AED0A7DB0B03E4063B7CF2A33E455C07C6473D53C9F3E40B85A272EC7DF55C0842BA0504F9F3E40\tpgstac-test-collection\t2011-08-16 00:00:00+00\t2011-08-16 00:00:00+00\t{\"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008720_se_16_1_20110816.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008720_se_16_1_20110816.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008720_se_16_1_20110816.200.jpg\"}}, \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [445790, 3387850, 452412, 3395405], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7555, 6622], \"eo:cloud_cover\": 82, \"proj:transform\": [1, 0, 445790, 0, -1, 3395405, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}\npgstac-test-item-0046\t0103000020E61000000100000005000000D0B69A75C6E355C0E333D93F4F9F3E403E40F7E5CCE355C09B8F6B43C5B03E4089601C5C3AE855C03BE5D18DB0B03E40AA7D3A1E33E855C02481069B3A9F3E40D0B69A75C6E355C0E333D93F4F9F3E40\tpgstac-test-collection\t2011-08-16 00:00:00+00\t2011-08-16 00:00:00+00\t{\"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008720_sw_16_1_20110816.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008720_sw_16_1_20110816.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008720_sw_16_1_20110816.200.jpg\"}}, \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [439800, 3387878, 446426, 3395437], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7559, 6626], \"eo:cloud_cover\": 16, \"proj:transform\": [1, 0, 439800, 0, -1, 3395437, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}\npgstac-test-item-0047\t0103000020E61000000100000005000000FBE59315C3F755C0CBD765F84F8F3E4020274C18CDF755C0AD4B8DD0CFA03E40A5F78DAF3DFC55C0D61F6118B0A03E40279F1EDB32FC55C0359BC761308F3E40FBE59315C3F755C0CBD765F84F8F3E40\tpgstac-test-collection\t2011-08-16 00:00:00+00\t2011-08-16 00:00:00+00\t{\"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008725_ne_16_1_20110816.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008725_ne_16_1_20110816.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008725_ne_16_1_20110816.200.jpg\"}}, \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [409792, 3381143, 416441, 3388719], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7576, 6649], \"eo:cloud_cover\": 43, \"proj:transform\": [1, 0, 409792, 0, -1, 3388719, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}\npgstac-test-item-0048\t0103000020E61000000100000005000000849B8C2AC3F755C0A80183A44F7F3E40C11E1329CDF755C08A75AA7CCF903E40938C9C853DFC55C0534145D5AF903E40E6AF90B932FC55C0B2BCAB1E307F3E40849B8C2AC3F755C0A80183A44F7F3E40\tpgstac-test-collection\t2011-08-16 00:00:00+00\t2011-08-16 00:00:00+00\t{\"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008725_se_16_1_20110816.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008725_se_16_1_20110816.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008725_se_16_1_20110816.200.jpg\"}}, \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [409735, 3374216, 416387, 3381792], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7576, 6652], \"eo:cloud_cover\": 4, \"proj:transform\": [1, 0, 409735, 0, -1, 3381792, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}\npgstac-test-item-0049\t0103000020E61000000100000005000000FBCBEEC9C3F355C0E9F010C64F8F3E4020274C18CDF355C032923D42CDA03E408DCF64FF3CF855C0B2497EC4AFA03E40E0D8B3E732F855C00AA01859328F3E40FBCBEEC9C3F355C0E9F010C64F8F3E40\tpgstac-test-collection\t2011-08-16 00:00:00+00\t2011-08-16 00:00:00+00\t{\"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008726_nw_16_1_20110816.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008726_nw_16_1_20110816.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008726_nw_16_1_20110816.200.jpg\"}}, \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [415786, 3381098, 422431, 3388670], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7572, 6645], \"eo:cloud_cover\": 9, \"proj:transform\": [1, 0, 415786, 0, -1, 3388670, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}\npgstac-test-item-0050\t0103000020E61000000100000005000000B40584D6C3F355C06CCF2C09507F3E40F0A2AF20CDF355C055682096CD903E408DCF64FF3CF855C0D61F6118B0903E40C896E5EB32F855C02E76FBAC327F3E40B40584D6C3F355C06CCF2C09507F3E40\tpgstac-test-collection\t2011-08-16 00:00:00+00\t2011-08-16 00:00:00+00\t{\"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008726_sw_16_1_20110816.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008726_sw_16_1_20110816.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008726_sw_16_1_20110816.200.jpg\"}}, \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [415732, 3374172, 422381, 3381744], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7572, 6649], \"eo:cloud_cover\": 84, \"proj:transform\": [1, 0, 415732, 0, -1, 3381744, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}\npgstac-test-item-0051\t0103000020E6100000010000000500000012C0CDE2C5E755C0E333D93F4F8F3E407F2F8507CDE755C0946A9F8EC7A03E404757E9EE3AEC55C07CD45FAFB0A03E40514CDE0033EC55C0CB9D9960388F3E4012C0CDE2C5E755C0E333D93F4F8F3E40\tpgstac-test-collection\t2011-08-16 00:00:00+00\t2011-08-16 00:00:00+00\t{\"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008727_ne_16_1_20110816.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008727_ne_16_1_20110816.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008727_ne_16_1_20110816.200.jpg\"}}, \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [433768, 3380983, 440401, 3388546], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7563, 6633], \"eo:cloud_cover\": 22, \"proj:transform\": [1, 0, 433768, 0, -1, 3388546, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}\npgstac-test-item-0052\t0103000020E6100000010000000500000041446ADAC5E755C0252367614F7F3E40C7F5EFFACCE755C0D09CF529C7903E40D6E3BED53AEC55C058FE7C5BB0903E40C896E5EB32EC55C0AD84EE92387F3E4041446ADAC5E755C0252367614F7F3E40\tpgstac-test-collection\t2011-08-16 00:00:00+00\t2011-08-16 00:00:00+00\t{\"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008727_se_16_1_20110816.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008727_se_16_1_20110816.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008727_se_16_1_20110816.200.jpg\"}}, \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [433726, 3374057, 440363, 3381619], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7562, 6637], \"eo:cloud_cover\": 12, \"proj:transform\": [1, 0, 433726, 0, -1, 3381619, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}\npgstac-test-item-0053\t0103000020E610000001000000050000005952EE3EC7DF55C01F662FDB4E8F3E40F6798CF2CCDF55C02593533BC3A03E4060CD018239E455C07CD45FAFB0A03E40390A100533E455C0B796C9703C8F3E405952EE3EC7DF55C01F662FDB4E8F3E40\tpgstac-test-collection\t2011-08-16 00:00:00+00\t2011-08-16 00:00:00+00\t{\"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008728_ne_16_1_20110816.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008728_ne_16_1_20110816.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008728_ne_16_1_20110816.200.jpg\"}}, \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [445756, 3380923, 452381, 3388479], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7556, 6625], \"eo:cloud_cover\": 24, \"proj:transform\": [1, 0, 445756, 0, -1, 3388479, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}\npgstac-test-item-0054\t0103000020E6100000010000000500000029E8F692C6E355C0C51A2E724F8F3E40C7F5EFFACCE355C01E6E8786C5A03E4047718E3A3AE855C0BDC3EDD0B0A03E40390A100533E855C006685BCD3A8F3E4029E8F692C6E355C0C51A2E724F8F3E40\tpgstac-test-collection\t2011-08-16 00:00:00+00\t2011-08-16 00:00:00+00\t{\"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008728_nw_16_1_20110816.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008728_nw_16_1_20110816.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008728_nw_16_1_20110816.200.jpg\"}}, \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [439762, 3380952, 446391, 3388511], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7559, 6629], \"eo:cloud_cover\": 5, \"proj:transform\": [1, 0, 439762, 0, -1, 3388511, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}\npgstac-test-item-0055\t0103000020E61000000100000005000000D09CF529C7DF55C06155BDFC4E7F3E40850662D9CCDF55C06682E15CC3903E403049658A39E455C05EBBB4E1B0903E40F243A51133E455C0F98557923C7F3E40D09CF529C7DF55C06155BDFC4E7F3E40\tpgstac-test-collection\t2011-08-16 00:00:00+00\t2011-08-16 00:00:00+00\t{\"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008728_se_16_1_20110816.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008728_se_16_1_20110816.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008728_se_16_1_20110816.200.jpg\"}}, \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [445721, 3373997, 452351, 3381553], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7556, 6630], \"eo:cloud_cover\": 79, \"proj:transform\": [1, 0, 445721, 0, -1, 3381553, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}\npgstac-test-item-0056\t0103000020E61000000100000005000000D80FB1C1C2F755C02AE09EE74FEF3E400EBC5AEECCF755C0302AA913D0003F40F8510DFB3DFC55C076172829B0003F40514CDE0033FC55C011C5E40D30EF3E40D80FB1C1C2F755C02AE09EE74FEF3E40\tpgstac-test-collection\t2011-08-16 00:00:00+00\t2011-08-16 00:00:00+00\t{\"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008701_ne_16_1_20110816.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008701_ne_16_1_20110816.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008701_ne_16_1_20110816.200.jpg\"}}, \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [410140, 3422703, 416766, 3430280], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7577, 6626], \"eo:cloud_cover\": 62, \"proj:transform\": [1, 0, 410140, 0, -1, 3430280, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}\npgstac-test-item-0057\t0103000020E61000000100000005000000C0E78711C2FB55C08FA50F5D50EF3E40C7F5EFFACCFB55C0471E882CD2003F40E0F599B33E0056C035289A07B0003F4081D07AF8320056C01EA7E8482EEF3E40C0E78711C2FB55C08FA50F5D50EF3E40\tpgstac-test-collection\t2011-08-16 00:00:00+00\t2011-08-16 00:00:00+00\t{\"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008701_nw_16_1_20110816.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008701_nw_16_1_20110816.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008701_nw_16_1_20110816.200.jpg\"}}, \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [404169, 3422752, 410799, 3430332], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7580, 6630], \"eo:cloud_cover\": 7, \"proj:transform\": [1, 0, 404169, 0, -1, 3430332, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}\npgstac-test-item-0058\t0103000020E61000000100000005000000F0517FBDC2F755C08AE8D7D64FDF3E405682C5E1CCF755C09032E202D0F03E4087DEE2E13DFC55C0D61F6118B0F03E40B05417F032FC55C011C5E40D30DF3E40F0517FBDC2F755C08AE8D7D64FDF3E40\tpgstac-test-collection\t2011-08-16 00:00:00+00\t2011-08-16 00:00:00+00\t{\"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008701_se_16_1_20110816.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008701_se_16_1_20110816.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008701_se_16_1_20110816.200.jpg\"}}, \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [410082, 3415776, 416712, 3423353], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7577, 6630], \"eo:cloud_cover\": 53, \"proj:transform\": [1, 0, 410082, 0, -1, 3423353, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}\npgstac-test-item-0059\t0103000020E61000000100000005000000C0E78711C2FB55C0EFAD484C50DF3E40F6798CF2CCFB55C0471E882CD2F03E405740A19E3E0056C035289A07B0F03E40C896E5EB320056C01EA7E8482EDF3E40C0E78711C2FB55C0EFAD484C50DF3E40\tpgstac-test-collection\t2011-08-16 00:00:00+00\t2011-08-16 00:00:00+00\t{\"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008701_sw_16_1_20110816.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008701_sw_16_1_20110816.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008701_sw_16_1_20110816.200.jpg\"}}, \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [404107, 3415825, 410741, 3423405], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7580, 6634], \"eo:cloud_cover\": 57, \"proj:transform\": [1, 0, 404107, 0, -1, 3423405, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}\npgstac-test-item-0066\t0103000020E610000001000000050000009D4830D5CC9755C006685BCD3AEF3E4036B05582C59755C03BE5D18DB0003F401BF16437339C55C0DC7EF964C5003F402A7288B8399C55C0070ABC934FEF3E409D4830D5CC9755C006685BCD3AEF3E40\tpgstac-test-collection\t2011-08-15 00:00:00+00\t2011-08-15 00:00:00+00\t{\"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30086/m_3008605_ne_16_1_20110815.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30086/m_3008605_ne_16_1_20110815.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30086/m_3008605_ne_16_1_20110815.200.jpg\"}}, \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-15T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [553400, 3422511, 560006, 3430070], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7559, 6606], \"eo:cloud_cover\": 86, \"proj:transform\": [1, 0, 553400, 0, -1, 3430070, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}\npgstac-test-item-0060\t0103000020E61000000100000005000000EF37DA71C3F355C02AE09EE74FEF3E405682C5E1CCF355C0374F75C8CD003F4010AE80423DF855C035289A07B0003F40390A100533F855C069A8514832EF3E40EF37DA71C3F355C02AE09EE74FEF3E40\tpgstac-test-collection\t2011-08-16 00:00:00+00\t2011-08-16 00:00:00+00\t{\"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008702_nw_16_1_20110816.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008702_nw_16_1_20110816.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008702_nw_16_1_20110816.200.jpg\"}}, \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [416111, 3422658, 422733, 3430231], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7573, 6622], \"eo:cloud_cover\": 70, \"proj:transform\": [1, 0, 416111, 0, -1, 3430231, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}\npgstac-test-item-0061\t0103000020E6100000010000000500000037FE4465C3F355C0E9F010C64FDF3E40B58AFED0CCF355C0F65FE7A6CDF03E40B1A547533DF855C035289A07B0F03E40C2BF081A33F855C0C9B08A3732DF3E4037FE4465C3F355C0E9F010C64FDF3E40\tpgstac-test-collection\t2011-08-16 00:00:00+00\t2011-08-16 00:00:00+00\t{\"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008702_sw_16_1_20110816.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008702_sw_16_1_20110816.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008702_sw_16_1_20110816.200.jpg\"}}, \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [416056, 3415731, 422683, 3423304], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7573, 6627], \"eo:cloud_cover\": 94, \"proj:transform\": [1, 0, 416056, 0, -1, 3423304, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}\npgstac-test-item-0062\t0103000020E6100000010000000500000095B88E71C5E755C0E333D93F4FEF3E40749B70AFCCE755C0B2834A5CC7003F40826F9A3E3BEC55C0B806B64AB0003F404B75012F33EC55C08AAE0B3F38EF3E4095B88E71C5E755C0E333D93F4FEF3E40\tpgstac-test-collection\t2011-08-16 00:00:00+00\t2011-08-16 00:00:00+00\t{\"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008703_ne_16_1_20110816.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008703_ne_16_1_20110816.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008703_ne_16_1_20110816.200.jpg\"}}, \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [434023, 3422542, 440634, 3430105], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7563, 6611], \"eo:cloud_cover\": 85, \"proj:transform\": [1, 0, 434023, 0, -1, 3430105, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}\npgstac-test-item-0063\t0103000020E610000001000000050000001E6E8786C5E755C0A80183A44FDF3E40149337C0CCE755C0D09CF529C7F03E40826F9A3E3BEC55C076172829B0F03E401BF1643733EC55C04E7CB5A338DF3E401E6E8786C5E755C0A80183A44FDF3E40\tpgstac-test-collection\t2011-08-16 00:00:00+00\t2011-08-16 00:00:00+00\t{\"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008703_se_16_1_20110816.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008703_se_16_1_20110816.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008703_se_16_1_20110816.200.jpg\"}}, \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [433980, 3415616, 440595, 3423178], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7562, 6615], \"eo:cloud_cover\": 2, \"proj:transform\": [1, 0, 433980, 0, -1, 3423178, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}\npgstac-test-item-0064\t0103000020E610000001000000050000005C59A2B3CC9F55C0107A36AB3EEF3E40948444DAC69F55C09FAA4203B1003F40459E245D33A455C032755776C1003F4084D72E6D38A455C0A2444B1E4FEF3E405C59A2B3CC9F55C0107A36AB3EEF3E40\tpgstac-test-collection\t2011-08-15 00:00:00+00\t2011-08-15 00:00:00+00\t{\"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30086/m_3008604_ne_16_1_20110815.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30086/m_3008604_ne_16_1_20110815.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30086/m_3008604_ne_16_1_20110815.200.jpg\"}}, \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-15T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [541466, 3422457, 548065, 3430010], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7553, 6599], \"eo:cloud_cover\": 40, \"proj:transform\": [1, 0, 541466, 0, -1, 3430010, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}\npgstac-test-item-0065\t0103000020E610000001000000050000002CD505BCCCA355C0E57E87A240EF3E404CA4349BC7A355C082919735B1003F40CE531D7233A855C0BC783F6EBF003F403D2B69C537A855C01F662FDB4EEF3E402CD505BCCCA355C0E57E87A240EF3E40\tpgstac-test-collection\t2011-08-15 00:00:00+00\t2011-08-15 00:00:00+00\t{\"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30086/m_3008604_nw_16_1_20110815.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30086/m_3008604_nw_16_1_20110815.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30086/m_3008604_nw_16_1_20110815.200.jpg\"}}, \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-15T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [535499, 3422435, 542094, 3429985], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7550, 6595], \"eo:cloud_cover\": 27, \"proj:transform\": [1, 0, 535499, 0, -1, 3429985, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}\npgstac-test-item-0067\t0103000020E610000001000000050000008BDD3EABCC9B55C0F98557923CEF3E40F4A62215C69B55C03BE5D18DB0003F40BCE82B4833A055C02593533BC3003F40E3C5C21039A055C0433C122F4FEF3E408BDD3EABCC9B55C0F98557923CEF3E40\tpgstac-test-collection\t2011-08-15 00:00:00+00\t2011-08-15 00:00:00+00\t{\"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30086/m_3008605_nw_16_1_20110815.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30086/m_3008605_nw_16_1_20110815.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30086/m_3008605_nw_16_1_20110815.200.jpg\"}}, \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-15T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [547433, 3422482, 554036, 3430038], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7556, 6603], \"eo:cloud_cover\": 35, \"proj:transform\": [1, 0, 547433, 0, -1, 3430038, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}\npgstac-test-item-0068\t0103000020E61000000100000005000000CDCCCCCCCC8F55C0D2C2651536EF3E40AE2EA704C48F55C0F9F5436CB0003F4004AF963B339455C04B5645B8C9003F40B2F336363B9455C0E333D93F4FEF3E40CDCCCCCCCC8F55C0D2C2651536EF3E40\tpgstac-test-collection\t2011-08-15 00:00:00+00\t2011-08-15 00:00:00+00\t{\"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30086/m_3008606_ne_16_1_20110815.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30086/m_3008606_ne_16_1_20110815.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30086/m_3008606_ne_16_1_20110815.200.jpg\"}}, \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-15T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [565333, 3422577, 571948, 3430144], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7567, 6615], \"eo:cloud_cover\": 22, \"proj:transform\": [1, 0, 565333, 0, -1, 3430144, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}\npgstac-test-item-0069\t0103000020E61000000100000005000000B58AFED0CC9355C08AAE0B3F38EF3E407E9065C1C49355C0B806B64AB0003F408C648F50339855C0B2834A5CC7003F406B47718E3A9855C0E333D93F4FEF3E40B58AFED0CC9355C08AAE0B3F38EF3E40\tpgstac-test-collection\t2011-08-15 00:00:00+00\t2011-08-15 00:00:00+00\t{\"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30086/m_3008606_nw_16_1_20110815.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30086/m_3008606_nw_16_1_20110815.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30086/m_3008606_nw_16_1_20110815.200.jpg\"}}, \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-15T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [559366, 3422542, 565977, 3430105], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7563, 6611], \"eo:cloud_cover\": 97, \"proj:transform\": [1, 0, 559366, 0, -1, 3430105, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}\npgstac-test-item-0070\t0103000020E61000000100000005000000C7F5EFFACC8755C069A8514832EF3E40F0517FBDC28755C035289A07B0003F40AA7D3A1E338C55C0374F75C8CD003F4011C8258E3C8C55C02AE09EE74FEF3E40C7F5EFFACC8755C069A8514832EF3E40\tpgstac-test-collection\t2011-08-15 00:00:00+00\t2011-08-15 00:00:00+00\t{\"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30086/m_3008607_ne_16_1_20110815.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30086/m_3008607_ne_16_1_20110815.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30086/m_3008607_ne_16_1_20110815.200.jpg\"}}, \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-15T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [577267, 3422658, 583889, 3430231], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7573, 6622], \"eo:cloud_cover\": 32, \"proj:transform\": [1, 0, 577267, 0, -1, 3430231, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}\npgstac-test-item-0071\t0103000020E61000000100000005000000C7F5EFFACC8B55C03FADA23F34EF3E40D8F50B76C38B55C058FE7C5BB0003F4063B7CF2A339055C0624A24D1CB003F40E15D2EE23B9055C048F949B54FEF3E40C7F5EFFACC8B55C03FADA23F34EF3E40\tpgstac-test-collection\t2011-08-15 00:00:00+00\t2011-08-15 00:00:00+00\t{\"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30086/m_3008607_nw_16_1_20110815.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30086/m_3008607_nw_16_1_20110815.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30086/m_3008607_nw_16_1_20110815.200.jpg\"}}, \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-15T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [571300, 3422616, 577918, 3430186], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7570, 6618], \"eo:cloud_cover\": 68, \"proj:transform\": [1, 0, 571300, 0, -1, 3430186, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}\npgstac-test-item-0072\t0103000020E610000001000000050000007F2F8507CD7F55C01EA7E8482EEF3E40200A664CC17F55C035289A07B0003F40390A1005338455C0471E882CD2003F40401878EE3D8455C08FA50F5D50EF3E407F2F8507CD7F55C01EA7E8482EEF3E40\tpgstac-test-collection\t2011-08-15 00:00:00+00\t2011-08-15 00:00:00+00\t{\"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30086/m_3008608_ne_16_1_20110815.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30086/m_3008608_ne_16_1_20110815.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30086/m_3008608_ne_16_1_20110815.200.jpg\"}}, \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-15T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [589201, 3422752, 595831, 3430332], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7580, 6630], \"eo:cloud_cover\": 30, \"proj:transform\": [1, 0, 589201, 0, -1, 3430332, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}\npgstac-test-item-0073\t0103000020E61000000100000005000000AFB321FFCC8355C011C5E40D30EF3E4008AEF204C28355C076172829B0003F40F243A511338855C0302AA913D0003F4028F04E3E3D8855C02AE09EE74FEF3E40AFB321FFCC8355C011C5E40D30EF3E40\tpgstac-test-collection\t2011-08-15 00:00:00+00\t2011-08-15 00:00:00+00\t{\"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30086/m_3008608_nw_16_1_20110815.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30086/m_3008608_nw_16_1_20110815.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30086/m_3008608_nw_16_1_20110815.200.jpg\"}}, \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-15T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [583234, 3422703, 589860, 3430280], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7577, 6626], \"eo:cloud_cover\": 3, \"proj:transform\": [1, 0, 583234, 0, -1, 3430280, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}\npgstac-test-item-0074\t0103000020E610000001000000050000004AD40B3ECD6F55C09F008A9125EF3E4093E52494BE6F55C00C957F2DAF003F40B05417F0327455C07E18213CDA003F40F7E978CC407455C0D1949D7E50EF3E404AD40B3ECD6F55C09F008A9125EF3E40\tpgstac-test-collection\t2011-08-15 00:00:00+00\t2011-08-15 00:00:00+00\t{\"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30085/m_3008502_ne_16_1_20110815.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30085/m_3008502_ne_16_1_20110815.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30085/m_3008502_ne_16_1_20110815.200.jpg\"}}, \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-15T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [613069, 3422979, 619715, 3430573], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7594, 6646], \"eo:cloud_cover\": 52, \"proj:transform\": [1, 0, 613069, 0, -1, 3430573, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}\npgstac-test-item-0075\t0103000020E6100000010000000500000050ABE80FCD7B55C0A8AAD0402CEF3E4009E23C9CC07B55C01152B7B3AF003F40698EACFC328055C07C2B1213D4003F403FFED2A23E8055C0728C648F50EF3E4050ABE80FCD7B55C0A8AAD0402CEF3E40\tpgstac-test-collection\t2011-08-15 00:00:00+00\t2011-08-15 00:00:00+00\t{\"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30085/m_3008501_nw_16_1_20110815.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30085/m_3008501_nw_16_1_20110815.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30085/m_3008501_nw_16_1_20110815.200.jpg\"}}, \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-15T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [595168, 3422804, 601802, 3430387], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7583, 6634], \"eo:cloud_cover\": 8, \"proj:transform\": [1, 0, 595168, 0, -1, 3430387, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}\npgstac-test-item-0076\t0103000020E6100000010000000500000008E57D1CCD7755C050C763062AEF3E40F1B913ECBF7755C01152B7B3AF003F4081D07AF8327C55C0D40E7F4DD6003F403FE42D573F7C55C0728C648F50EF3E4008E57D1CCD7755C050C763062AEF3E40\tpgstac-test-collection\t2011-08-15 00:00:00+00\t2011-08-15 00:00:00+00\t{\"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30085/m_3008501_ne_16_1_20110815.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30085/m_3008501_ne_16_1_20110815.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30085/m_3008501_ne_16_1_20110815.200.jpg\"}}, \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-15T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [601135, 3422859, 607773, 3430446], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7587, 6638], \"eo:cloud_cover\": 59, \"proj:transform\": [1, 0, 601135, 0, -1, 3430446, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}\npgstac-test-item-0077\t0103000020E61000000100000005000000BB61DBA2CCAB55C09545611745EF3E40933655F7C8AB55C0FFB27BF2B0003F40B6114F7633B055C0CAC2D7D7BA003F406DE34F5436B055C0C05DF6EB4EEF3E40BB61DBA2CCAB55C09545611745EF3E40\tpgstac-test-collection\t2011-08-15 00:00:00+00\t2011-08-15 00:00:00+00\t{\"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30086/m_3008603_nw_16_1_20110815.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30086/m_3008603_nw_16_1_20110815.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30086/m_3008603_nw_16_1_20110815.200.jpg\"}}, \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-15T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [523566, 3422402, 530153, 3429944], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7542, 6587], \"eo:cloud_cover\": 64, \"proj:transform\": [1, 0, 523566, 0, -1, 3429944, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}\npgstac-test-item-0078\t0103000020E61000000100000005000000EBE5779ACCA755C0DE59BBED42EF3E40F3583332C8A755C082919735B1003F40459E245D33AC55C06495D233BD003F40CCD1E3F736AC55C06155BDFC4EEF3E40EBE5779ACCA755C0DE59BBED42EF3E40\tpgstac-test-collection\t2011-08-15 00:00:00+00\t2011-08-15 00:00:00+00\t{\"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30086/m_3008603_ne_16_1_20110815.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30086/m_3008603_ne_16_1_20110815.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30086/m_3008603_ne_16_1_20110815.200.jpg\"}}, \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-15T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [529533, 3422417, 536124, 3429963], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7546, 6591], \"eo:cloud_cover\": 21, \"proj:transform\": [1, 0, 529533, 0, -1, 3429963, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}\npgstac-test-item-0079\t0103000020E61000000100000005000000A305685BCD6355C0FC1BB4571FEF3E4094331477BC6355C0C4E8B985AE003F406F6589CE326855C0E10D6954E0003F40252026E1426855C0365A0EF450EF3E40A305685BCD6355C0FC1BB4571FEF3E40\tpgstac-test-collection\t2011-08-15 00:00:00+00\t2011-08-15 00:00:00+00\t{\"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30085/m_3008504_nw_16_1_20110815.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30085/m_3008504_nw_16_1_20110815.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30085/m_3008504_nw_16_1_20110815.200.jpg\"}}, \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-15T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [630971, 3423185, 637629, 3430789], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7604, 6658], \"eo:cloud_cover\": 65, \"proj:transform\": [1, 0, 630971, 0, -1, 3430789, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}\npgstac-test-item-0080\t0103000020E61000000100000005000000EACBD24ECD5F55C0A9F57EA31DEF3E400B98C0ADBB5F55C024F1F274AE003F402E76FBAC326455C098F90E7EE2003F40B492567C436455C0DC0E0D8B51EF3E40EACBD24ECD5F55C0A9F57EA31DEF3E40\tpgstac-test-collection\t2011-08-15 00:00:00+00\t2011-08-15 00:00:00+00\t{\"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30085/m_3008504_ne_16_1_20110815.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30085/m_3008504_ne_16_1_20110815.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30085/m_3008504_ne_16_1_20110815.200.jpg\"}}, \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-15T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [636939, 3423261, 643601, 3430868], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7607, 6662], \"eo:cloud_cover\": 95, \"proj:transform\": [1, 0, 636939, 0, -1, 3430868, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}\npgstac-test-item-0081\t0103000020E61000000100000005000000D960E124CD6B55C04DDA54DD23EF3E4052103CBEBD6B55C00C957F2DAF003F409FE925C6327055C0D7FB8D76DC003F40CD22145B417055C077499C1551EF3E40D960E124CD6B55C04DDA54DD23EF3E40\tpgstac-test-collection\t2011-08-15 00:00:00+00\t2011-08-15 00:00:00+00\t{\"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30085/m_3008503_nw_16_1_20110815.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30085/m_3008503_nw_16_1_20110815.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30085/m_3008503_nw_16_1_20110815.200.jpg\"}}, \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-15T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [619037, 3423045, 625687, 3430642], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7597, 6650], \"eo:cloud_cover\": 39, \"proj:transform\": [1, 0, 619037, 0, -1, 3430642, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}\npgstac-test-item-0082\t0103000020E610000001000000050000004AD40B3ECD6755C07218CC5F21EF3E40F321A81ABD6755C089B663EAAE003F4087A757CA326C55C04DF8A57EDE003F408542041C426C55C0F46A80D250EF3E404AD40B3ECD6755C07218CC5F21EF3E40\tpgstac-test-collection\t2011-08-15 00:00:00+00\t2011-08-15 00:00:00+00\t{\"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30085/m_3008503_ne_16_1_20110815.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30085/m_3008503_ne_16_1_20110815.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30085/m_3008503_ne_16_1_20110815.200.jpg\"}}, \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-15T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [625004, 3423113, 631658, 3430714], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7601, 6654], \"eo:cloud_cover\": 26, \"proj:transform\": [1, 0, 625004, 0, -1, 3430714, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}\npgstac-test-item-0001\t0103000020E6100000010000000500000044FD2E6CCD5755C0174850FC18EF3E40C405A051BA5755C0E2016553AE003F408D7E349C325C55C007D15AD1E6003F409B1C3EE9445C55C09B1F7F6951EF3E4044FD2E6CCD5755C0174850FC18EF3E40\tpgstac-test-collection\t2011-08-25 00:00:00+00\t2011-08-25 00:00:00+00\t{\"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30085/m_3008505_ne_16_1_20110825.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30085/m_3008505_ne_16_1_20110825.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30085/m_3008505_ne_16_1_20110825.200.jpg\"}}, \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-25T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [648874, 3423421, 655544, 3431036], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7615, 6670], \"eo:cloud_cover\": 89, \"proj:transform\": [1, 0, 648874, 0, -1, 3431036, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}\npgstac-test-item-0002\t0103000020E610000001000000050000002CBB6070CD5B55C0CE33F6251BEF3E407D259012BB5B55C0A112D731AE003F40E6AF90B9326055C06DFE5F75E4003F403C2EAA45446055C05930F14751EF3E402CBB6070CD5B55C0CE33F6251BEF3E40\tpgstac-test-collection\t2011-08-25 00:00:00+00\t2011-08-25 00:00:00+00\t{\"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30085/m_3008505_nw_16_1_20110825.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30085/m_3008505_nw_16_1_20110825.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30085/m_3008505_nw_16_1_20110825.200.jpg\"}}, \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-25T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [642906, 3423339, 649572, 3430950], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7611, 6666], \"eo:cloud_cover\": 33, \"proj:transform\": [1, 0, 642906, 0, -1, 3430950, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}\npgstac-test-item-0083\t0103000020E61000000100000005000000C11E1329CD7355C015FDA19927EF3E40DA91EA3BBF7355C0D0622992AF003F40991249F4327855C0EB025E66D8003F402788BA0F407855C0EFAD484C50EF3E40C11E1329CD7355C015FDA19927EF3E40\tpgstac-test-collection\t2011-08-15 00:00:00+00\t2011-08-15 00:00:00+00\t{\"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30085/m_3008502_nw_16_1_20110815.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30085/m_3008502_nw_16_1_20110815.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30085/m_3008502_nw_16_1_20110815.200.jpg\"}}, \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-15T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [607102, 3422917, 613744, 3430508], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7591, 6642], \"eo:cloud_cover\": 26, \"proj:transform\": [1, 0, 607102, 0, -1, 3430508, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}\npgstac-test-item-0086\t0103000020E6100000010000000500000078D32D3BC4EF55C0070ABC934FDF3E4026FE28EACCEF55C0C1525DC0CBF03E40F98557923CF455C0B806B64AB0F03E40DA01D71533F455C09EB5DB2E34DF3E4078D32D3BC4EF55C0070ABC934FDF3E40\tpgstac-test-collection\t2011-08-01 00:00:00+00\t2011-08-01 00:00:00+00\t{\"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008702_se_16_1_20110801.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008702_se_16_1_20110801.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008702_se_16_1_20110801.200.jpg\"}}, \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-01T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [422031, 3415689, 428653, 3423259], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7570, 6622], \"eo:cloud_cover\": 40, \"proj:transform\": [1, 0, 422031, 0, -1, 3423259, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}\npgstac-test-item-0087\t0103000020E610000001000000050000004E0CC9C9C4EB55C0E333D93F4FEF3E40FC5069C4CCEB55C04B5645B8C9003F4052D158FB3BF055C0F9F5436CB0003F403333333333F055C0D2C2651536EF3E404E0CC9C9C4EB55C0E333D93F4FEF3E40\tpgstac-test-collection\t2011-08-01 00:00:00+00\t2011-08-01 00:00:00+00\t{\"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008703_nw_16_1_20110801.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008703_nw_16_1_20110801.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008703_nw_16_1_20110801.200.jpg\"}}, \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-01T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [428052, 3422577, 434667, 3430144], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7567, 6615], \"eo:cloud_cover\": 36, \"proj:transform\": [1, 0, 428052, 0, -1, 3430144, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}\npgstac-test-item-0088\t0103000020E61000000100000005000000D7C1C1DEC4EB55C048F949B54FDF3E409D4830D5CCEB55C00A67B796C9F03E40292499D53BF055C0B806B64AB0F03E40F243A51133F055C03788D68A36DF3E40D7C1C1DEC4EB55C048F949B54FDF3E40\tpgstac-test-collection\t2011-08-01 00:00:00+00\t2011-08-01 00:00:00+00\t{\"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008703_sw_16_1_20110801.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008703_sw_16_1_20110801.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008703_sw_16_1_20110801.200.jpg\"}}, \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-01T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [428006, 3415651, 434624, 3423217], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7566, 6618], \"eo:cloud_cover\": 23, \"proj:transform\": [1, 0, 428006, 0, -1, 3423217, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}\npgstac-test-item-0089\t0103000020E61000000100000005000000FBB1497EC4EF55C0252367614F8F3E4038691A14CDEF55C01B9E5E29CBA03E405E656D533CF455C0F3380CE6AFA03E40B05417F032F455C09EB5DB2E348F3E40FBB1497EC4EF55C0252367614F8F3E40\tpgstac-test-collection\t2011-07-31 00:00:00+00\t2011-07-31 00:00:00+00\t{\"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008726_ne_16_1_20110731.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008726_ne_16_1_20110731.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008726_ne_16_1_20110731.200.jpg\"}}, \"properties\": {\"gsd\": 1, \"datetime\": \"2011-07-31T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [421780, 3381056, 428421, 3388625], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7569, 6641], \"eo:cloud_cover\": 36, \"proj:transform\": [1, 0, 421780, 0, -1, 3388625, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}\npgstac-test-item-0090\t0103000020E61000000100000005000000BF99982EC4EF55C0CBD765F84FCF3E40850662D9CCEF55C0DF6B088ECBE03E40A054FB743CF455C076172829B0E03E40698EACFC32F455C06283859334CF3E40BF99982EC4EF55C0CBD765F84FCF3E40\tpgstac-test-collection\t2011-07-31 00:00:00+00\t2011-07-31 00:00:00+00\t{\"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008710_ne_16_1_20110731.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008710_ne_16_1_20110731.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008710_ne_16_1_20110731.200.jpg\"}}, \"properties\": {\"gsd\": 1, \"datetime\": \"2011-07-31T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [421981, 3408763, 428607, 3416332], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7569, 6626], \"eo:cloud_cover\": 84, \"proj:transform\": [1, 0, 421981, 0, -1, 3416332, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}\npgstac-test-item-0091\t0103000020E6100000010000000500000001892650C4EF55C048F949B54F9F3E4026FE28EACCEF55C03F74417DCBB03E40179F02603CF455C076172829B0B03E4081D07AF832F455C0C18BBE82349F3E4001892650C4EF55C048F949B54F9F3E40\tpgstac-test-collection\t2011-07-31 00:00:00+00\t2011-07-31 00:00:00+00\t{\"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008718_se_16_1_20110731.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008718_se_16_1_20110731.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008718_se_16_1_20110731.200.jpg\"}}, \"properties\": {\"gsd\": 1, \"datetime\": \"2011-07-31T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [421830, 3387983, 428468, 3395552], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7569, 6638], \"eo:cloud_cover\": 31, \"proj:transform\": [1, 0, 421830, 0, -1, 3395552, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}\npgstac-test-item-0092\t0103000020E61000000100000005000000E9465854C4EF55C048F949B54FBF3E40DF37BEF6CCEF55C0FD84B35BCBD03E40404CC2853CF455C09430D3F6AFD03E40DA01D71533F455C0809C306134BF3E40E9465854C4EF55C048F949B54FBF3E40\tpgstac-test-collection\t2011-07-31 00:00:00+00\t2011-07-31 00:00:00+00\t{\"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008710_se_16_1_20110731.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008710_se_16_1_20110731.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008710_se_16_1_20110731.200.jpg\"}}, \"properties\": {\"gsd\": 1, \"datetime\": \"2011-07-31T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [421930, 3401836, 428560, 3409405], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7569, 6630], \"eo:cloud_cover\": 24, \"proj:transform\": [1, 0, 421930, 0, -1, 3409405, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}\npgstac-test-item-0093\t0103000020E61000000100000005000000FA97A432C5EB55C0252367614F8F3E4067EDB60BCDEB55C0C9772975C9A03E405F7F129F3BF055C0F9F5436CB0A03E4081D07AF832F055C055A18158368F3E40FA97A432C5EB55C0252367614F8F3E40\tpgstac-test-collection\t2011-07-31 00:00:00+00\t2011-07-31 00:00:00+00\t{\"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008727_nw_16_1_20110731.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008727_nw_16_1_20110731.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008727_nw_16_1_20110731.200.jpg\"}}, \"properties\": {\"gsd\": 1, \"datetime\": \"2011-07-31T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [427774, 3381018, 434411, 3388584], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7566, 6637], \"eo:cloud_cover\": 61, \"proj:transform\": [1, 0, 427774, 0, -1, 3388584, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}\npgstac-test-item-0094\t0103000020E6100000010000000500000030F31DFCC4EB55C0C51A2E724FCF3E4026FE28EACCEB55C028806264C9E03E40F99FFCDD3BF055C076172829B0E03E40AA7D3A1E33F055C055A1815836CF3E4030F31DFCC4EB55C0C51A2E724FCF3E40\tpgstac-test-collection\t2011-07-31 00:00:00+00\t2011-07-31 00:00:00+00\t{\"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008711_nw_16_1_20110731.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008711_nw_16_1_20110731.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008711_nw_16_1_20110731.200.jpg\"}}, \"properties\": {\"gsd\": 1, \"datetime\": \"2011-07-31T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [427959, 3408724, 434581, 3416290], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7566, 6622], \"eo:cloud_cover\": 29, \"proj:transform\": [1, 0, 427959, 0, -1, 3416290, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}\npgstac-test-item-0095\t0103000020E6100000010000000500000078B988EFC4EB55C08AE8D7D64FBF3E40850662D9CCEB55C0EC4D0CC9C9D03E40B8B06EBC3BF055C03BE5D18DB0D03E40390A100533F055C0797764AC36BF3E4078B988EFC4EB55C08AE8D7D64FBF3E40\tpgstac-test-collection\t2011-07-31 00:00:00+00\t2011-07-31 00:00:00+00\t{\"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008711_sw_16_1_20110731.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008711_sw_16_1_20110731.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008711_sw_16_1_20110731.200.jpg\"}}, \"properties\": {\"gsd\": 1, \"datetime\": \"2011-07-31T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [427913, 3401798, 434539, 3409364], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7566, 6626], \"eo:cloud_cover\": 63, \"proj:transform\": [1, 0, 427913, 0, -1, 3409364, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}\npgstac-test-item-0096\t0103000020E61000000100000005000000B9A81611C5EB55C06612F5824FAF3E40DF37BEF6CCEB55C0696FF085C9C03E405F7F129F3BF055C058FE7C5BB0C03E40B05417F032F055C0F698486936AF3E40B9A81611C5EB55C06612F5824FAF3E40\tpgstac-test-collection\t2011-07-31 00:00:00+00\t2011-07-31 00:00:00+00\t{\"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008719_nw_16_1_20110731.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008719_nw_16_1_20110731.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008719_nw_16_1_20110731.200.jpg\"}}, \"properties\": {\"gsd\": 1, \"datetime\": \"2011-07-31T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [427867, 3394871, 434496, 3402437], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7566, 6629], \"eo:cloud_cover\": 26, \"proj:transform\": [1, 0, 427867, 0, -1, 3402437, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}\npgstac-test-item-0097\t0103000020E6100000010000000500000001892650C4EF55C0C51A2E724FAF3E400EBC5AEECCEF55C0C1525DC0CBC03E40B796C9703CF455C0F9F5436CB0C03E40390A100533F455C09EB5DB2E34AF3E4001892650C4EF55C0C51A2E724FAF3E40\tpgstac-test-collection\t2011-07-31 00:00:00+00\t2011-07-31 00:00:00+00\t{\"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008718_ne_16_1_20110731.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008718_ne_16_1_20110731.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008718_ne_16_1_20110731.200.jpg\"}}, \"properties\": {\"gsd\": 1, \"datetime\": \"2011-07-31T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [421880, 3394909, 428514, 3402479], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7570, 6634], \"eo:cloud_cover\": 1, \"proj:transform\": [1, 0, 421880, 0, -1, 3402479, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}\npgstac-test-item-0098\t0103000020E61000000100000005000000E92CB308C5EB55C0E9F010C64F9F3E4026FE28EACCEB55C046990D32C9B03E40E7340BB43BF055C0D61F6118B0B03E4021C8410933F055C01A6F2BBD369F3E40E92CB308C5EB55C0E9F010C64F9F3E40\tpgstac-test-collection\t2011-07-31 00:00:00+00\t2011-07-31 00:00:00+00\t{\"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008719_sw_16_1_20110731.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008719_sw_16_1_20110731.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008719_sw_16_1_20110731.200.jpg\"}}, \"properties\": {\"gsd\": 1, \"datetime\": \"2011-07-31T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [427820, 3387945, 434454, 3395510], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7565, 6634], \"eo:cloud_cover\": 73, \"proj:transform\": [1, 0, 427820, 0, -1, 3395510, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}\npgstac-test-item-0099\t0103000020E61000000100000005000000FA97A432C5EB55C0070ABC934F7F3E407F2F8507CDEB55C0A5A14621C9903E40BE874B8E3BF055C035289A07B0903E40B05417F032F055C0D87F9D9B367F3E40FA97A432C5EB55C0070ABC934F7F3E40\tpgstac-test-collection\t2011-07-31 00:00:00+00\t2011-07-31 00:00:00+00\t{\"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008727_sw_16_1_20110731.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008727_sw_16_1_20110731.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008727_sw_16_1_20110731.200.jpg\"}}, \"properties\": {\"gsd\": 1, \"datetime\": \"2011-07-31T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [427728, 3374092, 434369, 3381657], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7565, 6641], \"eo:cloud_cover\": 90, \"proj:transform\": [1, 0, 427728, 0, -1, 3381657, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}\npgstac-test-item-0084\t0103000020E610000001000000050000003E26529ACD4F55C09014916115EF3E4036AD1402B94F55C01E34BBEEAD003F408D7E349C325455C094C151F2EA003F403BE0BA62465455C023BBD23252EF3E403E26529ACD4F55C09014916115EF3E40\tpgstac-test-collection\t2011-08-02 00:00:00+00\t2011-08-02 00:00:00+00\t{\"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30085/m_3008506_ne_16_1_20110802.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30085/m_3008506_ne_16_1_20110802.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30085/m_3008506_ne_16_1_20110802.200.jpg\"}}, \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-02T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [660809, 3423596, 667487, 3431217], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7621, 6678], \"eo:cloud_cover\": 52, \"proj:transform\": [1, 0, 660809, 0, -1, 3431217, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}\npgstac-test-item-0085\t0103000020E610000001000000050000001FA2D11DC4EF55C048F949B54FEF3E409D4830D5CCEF55C0624A24D1CB003F40280AF4893CF455C058FE7C5BB0003F40390A100533F455C03FADA23F34EF3E401FA2D11DC4EF55C048F949B54FEF3E40\tpgstac-test-collection\t2011-08-01 00:00:00+00\t2011-08-01 00:00:00+00\t{\"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008702_ne_16_1_20110801.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008702_ne_16_1_20110801.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008702_ne_16_1_20110801.200.jpg\"}}, \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-01T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [422082, 3422616, 428700, 3430186], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7570, 6618], \"eo:cloud_cover\": 3, \"proj:transform\": [1, 0, 422082, 0, -1, 3430186, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}\npgstac-test-item-0100\t0103000020E61000000100000005000000CB2DAD86C4EF55C0070ABC934F7F3E4038691A14CDEF55C09E7C7A6CCB903E40A62BD8463CF455C076172829B0903E40C896E5EB32F455C02194F771347F3E40CB2DAD86C4EF55C0070ABC934F7F3E40\tpgstac-test-collection\t2011-07-31 00:00:00+00\t2011-07-31 00:00:00+00\t{\"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008726_se_16_1_20110731.tif\"}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008726_se_16_1_20110731.txt\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008726_se_16_1_20110731.200.jpg\"}}, \"properties\": {\"gsd\": 1, \"datetime\": \"2011-07-31T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [421730, 3374130, 428375, 3381699], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7569, 6645], \"eo:cloud_cover\": 50, \"proj:transform\": [1, 0, 421730, 0, -1, 3381699, 0, 0, 1]}, \"stac_extensions\": [\"eo\", \"projection\"]}\n" }, { "path": "src/pgstac/tests/testdata/items_duplicate_ids.ndjson", "content": "{\"id\": \"pgstac-test-item-duplicated\", \"bbox\": [-87.816179, 30.496894, -87.74637, 30.565604], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008726_se_16_1_20110731.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008726_se_16_1_20110731.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008726_se_16_1_20110731.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.74637, 30.497308], [-87.746892, 30.565604], [-87.816179, 30.565188], [-87.815608, 30.496894], [-87.74637, 30.497308]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-07-31T00:00:00Z\", \"naip:year\": 2013, \"proj:bbox\": [421730, 3374130, 428375, 3381699], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"zz\", \"proj:shape\": [7569, 6645], \"eo:cloud_cover\": 50, \"proj:transform\": [1, 0, 421730, 0, -1, 3381699, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-duplicated\", \"bbox\": [-87.816179, 30.496894, -87.74637, 30.565604], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008726_se_16_1_20110731.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008726_se_16_1_20110731.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008726_se_16_1_20110731.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.74637, 30.497308], [-87.746892, 30.565604], [-87.816179, 30.565188], [-87.815608, 30.496894], [-87.74637, 30.497308]]]}, \"collection\": \"pgstac-test-collection2\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-07-31T00:00:00Z\", \"naip:year\": 2013, \"proj:bbox\": [421730, 3374130, 428375, 3381699], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"zz\", \"proj:shape\": [7569, 6645], \"eo:cloud_cover\": 50, \"proj:transform\": [1, 0, 421730, 0, -1, 3381699, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n" }, { "path": "src/pgstac/tests/testdata/items_for_delsert.ndjson", "content": "{\"id\": \"pgstac-test-item-9993\", \"bbox\": [-85.379245, 30.933949, -85.308201, 31.003555], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30085/m_3008506_nw_16_1_20110825.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30085/m_3008506_nw_16_1_20110825.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30085/m_3008506_nw_16_1_20110825.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-85.309412, 30.933949], [-85.308201, 31.002658], [-85.378084, 31.003555], [-85.379245, 30.934843], [-85.309412, 30.933949]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-25T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [654842, 3423507, 661516, 3431125], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7618, 6674], \"eo:cloud_cover\": 28, \"proj:transform\": [1, 0, 654842, 0, -1, 3431125, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-9994\", \"bbox\": [-87.190775, 30.934712, -87.121772, 31.002794], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008707_ne_16_1_20110824.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008707_ne_16_1_20110824.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008707_ne_16_1_20110824.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.121772, 30.934795], [-87.121858, 31.002794], [-87.190775, 31.002711], [-87.190639, 30.934712], [-87.121772, 30.934795]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-24T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [481788, 3422382, 488367, 3429918], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7536, 6579], \"eo:cloud_cover\": 23, \"proj:transform\": [1, 0, 481788, 0, -1, 3429918, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-9995\", \"bbox\": [-87.128238, 30.934744, -87.059311, 31.002757], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008708_nw_16_1_20110824.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008708_nw_16_1_20110824.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008708_nw_16_1_20110824.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.059311, 30.934794], [-87.059353, 31.002757], [-87.128238, 31.002707], [-87.128147, 30.934744], [-87.059311, 30.934794]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-24T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [487758, 3422377, 494334, 3429909], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7532, 6576], \"eo:cloud_cover\": 3, \"proj:transform\": [1, 0, 487758, 0, -1, 3429909, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0006\", \"bbox\": [-87.253322, 30.934677, -87.184223, 31.00282], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008707_nw_16_1_20110824.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008707_nw_16_1_20110824.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008707_nw_16_1_20110824.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.184223, 30.934794], [-87.184354, 31.00282], [-87.253322, 31.002703], [-87.253142, 30.934677], [-87.184223, 30.934794]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-24T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [475817, 3422390, 482401, 3429929], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7539, 6584], \"eo:cloud_cover\": 100, \"proj:transform\": [1, 0, 475817, 0, -1, 3429929, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0007\", \"bbox\": [-87.440935, 30.622081, -87.371617, 30.690415], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008721_se_16_1_20110817.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008721_se_16_1_20110817.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008721_se_16_1_20110817.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.371617, 30.622297], [-87.371878, 30.690415], [-87.440935, 30.690199], [-87.440626, 30.622081], [-87.371617, 30.622297]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-17T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [457770, 3387803, 464384, 3395352], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7549, 6614], \"eo:cloud_cover\": 59, \"proj:transform\": [1, 0, 457770, 0, -1, 3395352, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0008\", \"bbox\": [-87.503478, 30.622053, -87.434074, 30.690447], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008721_sw_16_1_20110817.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008721_sw_16_1_20110817.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008721_sw_16_1_20110817.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.434074, 30.622302], [-87.434379, 30.690447], [-87.503478, 30.690198], [-87.503124, 30.622053], [-87.434074, 30.622302]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 999, \"datetime\": \"2011-08-17T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [451780, 3387825, 458398, 3395377], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7552, 6618], \"eo:cloud_cover\": 64, \"proj:transform\": [1, 0, 451780, 0, -1, 3395377, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n" }, { "path": "src/pgstac/tests/testdata/items_private.ndjson", "content": "{\"id\": \"pgstac-test-item-0003\", \"private\": {\"created_by\": \"stac-task\"}, \"bbox\": [-85.379245, 30.933949, -85.308201, 31.003555], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30085/m_3008506_nw_16_1_20110825.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30085/m_3008506_nw_16_1_20110825.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30085/m_3008506_nw_16_1_20110825.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-85.309412, 30.933949], [-85.308201, 31.002658], [-85.378084, 31.003555], [-85.379245, 30.934843], [-85.309412, 30.933949]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-25T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [654842, 3423507, 661516, 3431125], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7618, 6674], \"eo:cloud_cover\": 28, \"proj:transform\": [1, 0, 654842, 0, -1, 3431125, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0004\", \"private\": {\"created_by\": \"stac-task\"}, \"bbox\": [-87.190775, 30.934712, -87.121772, 31.002794], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008707_ne_16_1_20110824.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008707_ne_16_1_20110824.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008707_ne_16_1_20110824.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.121772, 30.934795], [-87.121858, 31.002794], [-87.190775, 31.002711], [-87.190639, 30.934712], [-87.121772, 30.934795]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-24T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [481788, 3422382, 488367, 3429918], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7536, 6579], \"eo:cloud_cover\": 23, \"proj:transform\": [1, 0, 481788, 0, -1, 3429918, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0005\", \"private\": {\"created_by\": \"stac-task\"}, \"bbox\": [-87.128238, 30.934744, -87.059311, 31.002757], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008708_nw_16_1_20110824.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008708_nw_16_1_20110824.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008708_nw_16_1_20110824.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.059311, 30.934794], [-87.059353, 31.002757], [-87.128238, 31.002707], [-87.128147, 30.934744], [-87.059311, 30.934794]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-24T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [487758, 3422377, 494334, 3429909], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7532, 6576], \"eo:cloud_cover\": 3, \"proj:transform\": [1, 0, 487758, 0, -1, 3429909, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0006\", \"private\": {\"created_by\": \"stac-task\"}, \"bbox\": [-87.253322, 30.934677, -87.184223, 31.00282], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008707_nw_16_1_20110824.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008707_nw_16_1_20110824.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008707_nw_16_1_20110824.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.184223, 30.934794], [-87.184354, 31.00282], [-87.253322, 31.002703], [-87.253142, 30.934677], [-87.184223, 30.934794]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-24T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [475817, 3422390, 482401, 3429929], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7539, 6584], \"eo:cloud_cover\": 100, \"proj:transform\": [1, 0, 475817, 0, -1, 3429929, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0007\", \"private\": {\"created_by\": \"stac-task\"}, \"bbox\": [-87.440935, 30.622081, -87.371617, 30.690415], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008721_se_16_1_20110817.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008721_se_16_1_20110817.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008721_se_16_1_20110817.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.371617, 30.622297], [-87.371878, 30.690415], [-87.440935, 30.690199], [-87.440626, 30.622081], [-87.371617, 30.622297]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-17T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [457770, 3387803, 464384, 3395352], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7549, 6614], \"eo:cloud_cover\": 59, \"proj:transform\": [1, 0, 457770, 0, -1, 3395352, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0008\", \"private\": {\"created_by\": \"stac-task\"}, \"bbox\": [-87.503478, 30.622053, -87.434074, 30.690447], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008721_sw_16_1_20110817.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008721_sw_16_1_20110817.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008721_sw_16_1_20110817.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.434074, 30.622302], [-87.434379, 30.690447], [-87.503478, 30.690198], [-87.503124, 30.622053], [-87.434074, 30.622302]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-17T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [451780, 3387825, 458398, 3395377], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7552, 6618], \"eo:cloud_cover\": 64, \"proj:transform\": [1, 0, 451780, 0, -1, 3395377, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0009\", \"private\": {\"created_by\": \"stac-task\"}, \"bbox\": [-87.503478, 30.68455, -87.434072, 30.752944], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008721_nw_16_1_20110817.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008721_nw_16_1_20110817.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008721_nw_16_1_20110817.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.434072, 30.684799], [-87.434377, 30.752944], [-87.503478, 30.752694], [-87.503125, 30.68455], [-87.434072, 30.684799]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-17T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [451811, 3394751, 458425, 3402303], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7552, 6614], \"eo:cloud_cover\": 61, \"proj:transform\": [1, 0, 451811, 0, -1, 3402303, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0010\", \"private\": {\"created_by\": \"stac-task\"}, \"bbox\": [-87.440937, 30.684579, -87.371606, 30.752912], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008721_ne_16_1_20110817.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008721_ne_16_1_20110817.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008721_ne_16_1_20110817.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.371606, 30.684794], [-87.371867, 30.752912], [-87.440937, 30.752696], [-87.440628, 30.684579], [-87.371606, 30.684794]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-17T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [457797, 3394729, 464408, 3402278], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7549, 6611], \"eo:cloud_cover\": 31, \"proj:transform\": [1, 0, 457797, 0, -1, 3402278, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0011\", \"private\": {\"created_by\": \"stac-task\"}, \"bbox\": [-87.315859, 30.934648, -87.246684, 31.002851], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008706_ne_16_1_20110817.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008706_ne_16_1_20110817.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008706_ne_16_1_20110817.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.246684, 30.934798], [-87.246859, 31.002851], [-87.315859, 31.0027], [-87.315635, 30.934648], [-87.246684, 30.934798]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-17T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [469847, 3422402, 476434, 3429944], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7542, 6587], \"eo:cloud_cover\": 41, \"proj:transform\": [1, 0, 469847, 0, -1, 3429944, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0012\", \"private\": {\"created_by\": \"stac-task\"}, \"bbox\": [-87.440942, 30.93458, -87.371596, 31.002921], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008705_ne_16_1_20110817.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008705_ne_16_1_20110817.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008705_ne_16_1_20110817.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.371596, 30.934797], [-87.37186, 31.002921], [-87.440942, 31.002704], [-87.440629, 30.93458], [-87.371596, 30.934797]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-17T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [457906, 3422435, 464501, 3429985], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7550, 6595], \"eo:cloud_cover\": 4, \"proj:transform\": [1, 0, 457906, 0, -1, 3429985, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0013\", \"private\": {\"created_by\": \"stac-task\"}, \"bbox\": [-87.378406, 30.934615, -87.309145, 31.002887], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008706_nw_16_1_20110817.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008706_nw_16_1_20110817.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008706_nw_16_1_20110817.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.309145, 30.934799], [-87.309365, 31.002887], [-87.378406, 31.002704], [-87.378137, 30.934615], [-87.309145, 30.934799]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-17T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [463876, 3422417, 470467, 3429963], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7546, 6591], \"eo:cloud_cover\": 2, \"proj:transform\": [1, 0, 463876, 0, -1, 3429963, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0014\", \"private\": {\"created_by\": \"stac-task\"}, \"bbox\": [-87.628547, 30.496984, -87.558991, 30.565507], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008728_sw_16_1_20110816.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008728_sw_16_1_20110816.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008728_sw_16_1_20110816.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.558991, 30.497299], [-87.559382, 30.565507], [-87.628547, 30.565192], [-87.628108, 30.496984], [-87.558991, 30.497299]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [439724, 3374025, 446357, 3381584], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7559, 6633], \"eo:cloud_cover\": 17, \"proj:transform\": [1, 0, 439724, 0, -1, 3381584, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0015\", \"private\": {\"created_by\": \"stac-task\"}, \"bbox\": [-86.940689, 30.934744, -86.871762, 31.002757], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30086/m_3008601_ne_16_1_20110816.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30086/m_3008601_ne_16_1_20110816.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30086/m_3008601_ne_16_1_20110816.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-86.871853, 30.934744], [-86.871762, 31.002707], [-86.940647, 31.002757], [-86.940689, 30.934794], [-86.871853, 30.934744]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [505666, 3422377, 512242, 3429909], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7532, 6576], \"eo:cloud_cover\": 54, \"proj:transform\": [1, 0, 505666, 0, -1, 3429909, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0016\", \"private\": {\"created_by\": \"stac-task\"}, \"bbox\": [-87.003143, 30.934773, -86.93431, 31.002726], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30086/m_3008601_nw_16_1_20110816.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30086/m_3008601_nw_16_1_20110816.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30086/m_3008601_nw_16_1_20110816.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-86.934356, 30.934773], [-86.93431, 31.002709], [-87.003143, 31.002726], [-87.00314, 30.934789], [-86.934356, 30.934773]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [499700, 3422375, 506271, 3429904], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7529, 6571], \"eo:cloud_cover\": 13, \"proj:transform\": [1, 0, 499700, 0, -1, 3429904, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0017\", \"private\": {\"created_by\": \"stac-task\"}, \"bbox\": [-86.815777, 30.934677, -86.746678, 31.00282], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30086/m_3008602_ne_16_1_20110816.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30086/m_3008602_ne_16_1_20110816.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30086/m_3008602_ne_16_1_20110816.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-86.746858, 30.934677], [-86.746678, 31.002703], [-86.815646, 31.00282], [-86.815777, 30.934794], [-86.746858, 30.934677]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [517599, 3422390, 524183, 3429929], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7539, 6584], \"eo:cloud_cover\": 59, \"proj:transform\": [1, 0, 517599, 0, -1, 3429929, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0018\", \"private\": {\"created_by\": \"stac-task\"}, \"bbox\": [-86.878228, 30.934712, -86.809225, 31.002794], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30086/m_3008602_nw_16_1_20110816.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30086/m_3008602_nw_16_1_20110816.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30086/m_3008602_nw_16_1_20110816.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-86.809361, 30.934712], [-86.809225, 31.002711], [-86.878142, 31.002794], [-86.878228, 30.934795], [-86.809361, 30.934712]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [511633, 3422382, 518212, 3429918], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7536, 6579], \"eo:cloud_cover\": 29, \"proj:transform\": [1, 0, 511633, 0, -1, 3429918, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0019\", \"private\": {\"created_by\": \"stac-task\"}, \"bbox\": [-87.566035, 30.934518, -87.496517, 31.002979], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008704_ne_16_1_20110816.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008704_ne_16_1_20110816.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008704_ne_16_1_20110816.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.496517, 30.934802], [-87.49687, 31.002979], [-87.566035, 31.002694], [-87.565633, 30.934518], [-87.496517, 30.934802]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [445964, 3422482, 452567, 3430038], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7556, 6603], \"eo:cloud_cover\": 52, \"proj:transform\": [1, 0, 445964, 0, -1, 3430038, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0020\", \"private\": {\"created_by\": \"stac-task\"}, \"bbox\": [-87.62857, 30.934491, -87.558977, 31.003012], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008704_nw_16_1_20110816.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008704_nw_16_1_20110816.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008704_nw_16_1_20110816.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.558977, 30.934808], [-87.559374, 31.003012], [-87.62857, 31.002694], [-87.628123, 30.934491], [-87.558977, 30.934808]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [439994, 3422511, 446600, 3430070], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7559, 6606], \"eo:cloud_cover\": 39, \"proj:transform\": [1, 0, 439994, 0, -1, 3430070, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0021\", \"private\": {\"created_by\": \"stac-task\"}, \"bbox\": [-87.628569, 30.871988, -87.55898, 30.940509], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008704_sw_16_1_20110816.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008704_sw_16_1_20110816.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008704_sw_16_1_20110816.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.55898, 30.872305], [-87.559377, 30.940509], [-87.628569, 30.940191], [-87.628123, 30.871988], [-87.55898, 30.872305]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [439955, 3415584, 446565, 3423143], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7559, 6610], \"eo:cloud_cover\": 29, \"proj:transform\": [1, 0, 439955, 0, -1, 3423143, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0022\", \"private\": {\"created_by\": \"stac-task\"}, \"bbox\": [-87.503488, 30.93455, -87.434056, 31.002952], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008705_nw_16_1_20110816.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008705_nw_16_1_20110816.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008705_nw_16_1_20110816.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.434056, 30.934801], [-87.434365, 31.002952], [-87.503488, 31.002701], [-87.503131, 30.93455], [-87.434056, 30.934801]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [451935, 3422457, 458534, 3430010], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7553, 6599], \"eo:cloud_cover\": 84, \"proj:transform\": [1, 0, 451935, 0, -1, 3430010, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0023\", \"private\": {\"created_by\": \"stac-task\"}, \"bbox\": [-87.06569, 30.934773, -86.996857, 31.002726], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008708_ne_16_1_20110816.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008708_ne_16_1_20110816.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008708_ne_16_1_20110816.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-86.99686, 30.934789], [-86.996857, 31.002726], [-87.06569, 31.002709], [-87.065644, 30.934773], [-86.99686, 30.934789]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [493729, 3422375, 500300, 3429904], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7529, 6571], \"eo:cloud_cover\": 56, \"proj:transform\": [1, 0, 493729, 0, -1, 3429904, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0024\", \"private\": {\"created_by\": \"stac-task\"}, \"bbox\": [-87.941273, 30.809325, -87.871271, 30.878173], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008709_ne_16_1_20110816.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008709_ne_16_1_20110816.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008709_ne_16_1_20110816.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.871271, 30.80981], [-87.871889, 30.878173], [-87.941273, 30.877687], [-87.940605, 30.809325], [-87.871271, 30.80981]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [410024, 3408849, 416657, 3416426], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7577, 6633], \"eo:cloud_cover\": 58, \"proj:transform\": [1, 0, 410024, 0, -1, 3416426, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0025\", \"private\": {\"created_by\": \"stac-task\"}, \"bbox\": [-88.003818, 30.809299, -87.933721, 30.878206], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008709_nw_16_1_20110816.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008709_nw_16_1_20110816.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008709_nw_16_1_20110816.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.933721, 30.809817], [-87.934384, 30.878206], [-88.003818, 30.877686], [-88.003106, 30.809299], [-87.933721, 30.809817]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [404045, 3408898, 410683, 3416478], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7580, 6638], \"eo:cloud_cover\": 65, \"proj:transform\": [1, 0, 404045, 0, -1, 3416478, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0026\", \"private\": {\"created_by\": \"stac-task\"}, \"bbox\": [-87.941269, 30.746832, -87.871272, 30.815671], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008709_se_16_1_20110816.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008709_se_16_1_20110816.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008709_se_16_1_20110816.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.871272, 30.747317], [-87.871889, 30.815671], [-87.941269, 30.815185], [-87.940603, 30.746832], [-87.871272, 30.747317]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [409966, 3401923, 416603, 3409499], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7576, 6637], \"eo:cloud_cover\": 52, \"proj:transform\": [1, 0, 409966, 0, -1, 3409499, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0027\", \"private\": {\"created_by\": \"stac-task\"}, \"bbox\": [-88.003816, 30.746797, -87.933724, 30.815704], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008709_sw_16_1_20110816.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008709_sw_16_1_20110816.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008709_sw_16_1_20110816.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.933724, 30.747315], [-87.934385, 30.815704], [-88.003816, 30.815185], [-88.003106, 30.746797], [-87.933724, 30.747315]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [403983, 3401971, 410625, 3409551], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7580, 6642], \"eo:cloud_cover\": 43, \"proj:transform\": [1, 0, 403983, 0, -1, 3409551, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0028\", \"private\": {\"created_by\": \"stac-task\"}, \"bbox\": [-87.878737, 30.809358, -87.80881, 30.878137], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008710_nw_16_1_20110816.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008710_nw_16_1_20110816.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008710_nw_16_1_20110816.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.80881, 30.809809], [-87.809384, 30.878137], [-87.878737, 30.877684], [-87.878114, 30.809358], [-87.80881, 30.809809]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [416002, 3408804, 422632, 3416377], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7573, 6630], \"eo:cloud_cover\": 46, \"proj:transform\": [1, 0, 416002, 0, -1, 3416377, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0029\", \"private\": {\"created_by\": \"stac-task\"}, \"bbox\": [-87.878732, 30.746864, -87.80881, 30.815634], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008710_sw_16_1_20110816.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008710_sw_16_1_20110816.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008710_sw_16_1_20110816.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.80881, 30.747315], [-87.809382, 30.815634], [-87.878732, 30.815182], [-87.878111, 30.746864], [-87.80881, 30.747315]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [415948, 3401878, 422582, 3409450], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7572, 6634], \"eo:cloud_cover\": 42, \"proj:transform\": [1, 0, 415948, 0, -1, 3409450, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0030\", \"private\": {\"created_by\": \"stac-task\"}, \"bbox\": [-87.691106, 30.809455, -87.621436, 30.878045], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008711_ne_16_1_20110816.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008711_ne_16_1_20110816.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008711_ne_16_1_20110816.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.621436, 30.809805], [-87.621876, 30.878045], [-87.691106, 30.877694], [-87.690617, 30.809455], [-87.621436, 30.809805]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [433938, 3408689, 440556, 3416252], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7563, 6618], \"eo:cloud_cover\": 16, \"proj:transform\": [1, 0, 433938, 0, -1, 3416252, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0031\", \"private\": {\"created_by\": \"stac-task\"}, \"bbox\": [-87.691108, 30.74696, -87.621442, 30.815542], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008711_se_16_1_20110816.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008711_se_16_1_20110816.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008711_se_16_1_20110816.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.621442, 30.74731], [-87.62188, 30.815542], [-87.691108, 30.815191], [-87.69062, 30.74696], [-87.621442, 30.74731]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [433895, 3401763, 440517, 3409325], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7562, 6622], \"eo:cloud_cover\": 16, \"proj:transform\": [1, 0, 433895, 0, -1, 3409325, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0032\", \"private\": {\"created_by\": \"stac-task\"}, \"bbox\": [-87.628569, 30.809484, -87.558973, 30.878015], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008712_nw_16_1_20110816.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008712_nw_16_1_20110816.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008712_nw_16_1_20110816.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.558973, 30.809801], [-87.559369, 30.878015], [-87.628569, 30.877697], [-87.628124, 30.809484], [-87.558973, 30.809801]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [439916, 3408657, 446531, 3416217], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7560, 6615], \"eo:cloud_cover\": 7, \"proj:transform\": [1, 0, 439916, 0, -1, 3416217, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0033\", \"private\": {\"created_by\": \"stac-task\"}, \"bbox\": [-87.566019, 30.747015, -87.496524, 30.815477], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008712_se_16_1_20110816.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008712_se_16_1_20110816.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008712_se_16_1_20110816.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.496524, 30.747298], [-87.496874, 30.815477], [-87.566019, 30.815193], [-87.56562, 30.747015], [-87.496524, 30.747298]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [445860, 3401702, 452474, 3409258], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7556, 6614], \"eo:cloud_cover\": 49, \"proj:transform\": [1, 0, 445860, 0, -1, 3409258, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0034\", \"private\": {\"created_by\": \"stac-task\"}, \"bbox\": [-87.628559, 30.746989, -87.558978, 30.815511], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008712_sw_16_1_20110816.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008712_sw_16_1_20110816.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008712_sw_16_1_20110816.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.558978, 30.747305], [-87.559372, 30.815511], [-87.628559, 30.815194], [-87.628115, 30.746989], [-87.558978, 30.747305]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [439878, 3401731, 446496, 3409290], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7559, 6618], \"eo:cloud_cover\": 3, \"proj:transform\": [1, 0, 439878, 0, -1, 3409290, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0035\", \"private\": {\"created_by\": \"stac-task\"}, \"bbox\": [-87.941266, 30.68433, -87.871274, 30.753168], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008717_ne_16_1_20110816.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008717_ne_16_1_20110816.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008717_ne_16_1_20110816.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.871274, 30.684813], [-87.871889, 30.753168], [-87.941266, 30.752684], [-87.940602, 30.68433], [-87.871274, 30.684813]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [409908, 3394996, 416549, 3402572], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7576, 6641], \"eo:cloud_cover\": 33, \"proj:transform\": [1, 0, 409908, 0, -1, 3402572, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0036\", \"private\": {\"created_by\": \"stac-task\"}, \"bbox\": [-88.003814, 30.684295, -87.933727, 30.753202], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008717_nw_16_1_20110816.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008717_nw_16_1_20110816.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008717_nw_16_1_20110816.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.933727, 30.684813], [-87.934386, 30.753202], [-88.003814, 30.752684], [-88.003106, 30.684295], [-87.933727, 30.684813]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [403921, 3395044, 410567, 3402624], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7580, 6646], \"eo:cloud_cover\": 12, \"proj:transform\": [1, 0, 403921, 0, -1, 3402624, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0037\", \"private\": {\"created_by\": \"stac-task\"}, \"bbox\": [-87.941265, 30.621827, -87.871277, 30.690665], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008717_se_16_1_20110816.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008717_se_16_1_20110816.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008717_se_16_1_20110816.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.871277, 30.62231], [-87.871891, 30.690665], [-87.941265, 30.690181], [-87.940603, 30.621827], [-87.871277, 30.62231]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [409850, 3388069, 416495, 3395645], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7576, 6645], \"eo:cloud_cover\": 54, \"proj:transform\": [1, 0, 409850, 0, -1, 3395645, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0038\", \"private\": {\"created_by\": \"stac-task\"}, \"bbox\": [-88.003804, 30.621802, -87.933732, 30.6907], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008717_sw_16_1_20110816.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008717_sw_16_1_20110816.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008717_sw_16_1_20110816.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.933732, 30.622318], [-87.934389, 30.6907], [-88.003804, 30.690182], [-88.003098, 30.621802], [-87.933732, 30.622318]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [403860, 3388118, 410509, 3395697], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7579, 6649], \"eo:cloud_cover\": 77, \"proj:transform\": [1, 0, 403860, 0, -1, 3395697, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0039\", \"private\": {\"created_by\": \"stac-task\"}, \"bbox\": [-87.878728, 30.684361, -87.808821, 30.75314], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008718_nw_16_1_20110816.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008718_nw_16_1_20110816.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008718_nw_16_1_20110816.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.808821, 30.684811], [-87.809392, 30.75314], [-87.878728, 30.752689], [-87.878109, 30.684361], [-87.808821, 30.684811]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [415894, 3394951, 422531, 3402524], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7573, 6637], \"eo:cloud_cover\": 12, \"proj:transform\": [1, 0, 415894, 0, -1, 3402524, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0040\", \"private\": {\"created_by\": \"stac-task\"}, \"bbox\": [-87.878725, 30.621858, -87.808823, 30.690637], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008718_sw_16_1_20110816.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008718_sw_16_1_20110816.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008718_sw_16_1_20110816.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.808823, 30.622307], [-87.809392, 30.690637], [-87.878725, 30.690186], [-87.878107, 30.621858], [-87.808823, 30.622307]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [415840, 3388024, 422481, 3395597], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7573, 6641], \"eo:cloud_cover\": 10, \"proj:transform\": [1, 0, 415840, 0, -1, 3395597, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0041\", \"private\": {\"created_by\": \"stac-task\"}, \"bbox\": [-87.6911, 30.684456, -87.621438, 30.753047], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008719_ne_16_1_20110816.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008719_ne_16_1_20110816.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008719_ne_16_1_20110816.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.621438, 30.684805], [-87.621875, 30.753047], [-87.6911, 30.752696], [-87.690613, 30.684456], [-87.621438, 30.684805]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [433853, 3394836, 440479, 3402399], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7563, 6626], \"eo:cloud_cover\": 3, \"proj:transform\": [1, 0, 433853, 0, -1, 3402399, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0042\", \"private\": {\"created_by\": \"stac-task\"}, \"bbox\": [-87.691103, 30.62196, -87.621445, 30.690542], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008719_se_16_1_20110816.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008719_se_16_1_20110816.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008719_se_16_1_20110816.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.621445, 30.622309], [-87.621882, 30.690542], [-87.691103, 30.690192], [-87.690618, 30.62196], [-87.621445, 30.622309]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [433810, 3387910, 440440, 3395472], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7562, 6630], \"eo:cloud_cover\": 99, \"proj:transform\": [1, 0, 433810, 0, -1, 3395472, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0043\", \"private\": {\"created_by\": \"stac-task\"}, \"bbox\": [-87.566019, 30.684519, -87.496527, 30.752981], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008720_ne_16_1_20110816.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008720_ne_16_1_20110816.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008720_ne_16_1_20110816.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.496527, 30.684801], [-87.496877, 30.752981], [-87.566019, 30.752698], [-87.565621, 30.684519], [-87.496527, 30.684801]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [445825, 3394776, 452443, 3402332], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7556, 6618], \"eo:cloud_cover\": 80, \"proj:transform\": [1, 0, 445825, 0, -1, 3402332, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0044\", \"private\": {\"created_by\": \"stac-task\"}, \"bbox\": [-87.62856, 30.684484, -87.558983, 30.753015], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008720_nw_16_1_20110816.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008720_nw_16_1_20110816.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008720_nw_16_1_20110816.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.558983, 30.6848], [-87.559376, 30.753015], [-87.62856, 30.752699], [-87.628117, 30.684484], [-87.558983, 30.6848]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [439839, 3394804, 446461, 3402364], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7560, 6622], \"eo:cloud_cover\": 30, \"proj:transform\": [1, 0, 439839, 0, -1, 3402364, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0045\", \"private\": {\"created_by\": \"stac-task\"}, \"bbox\": [-87.56602, 30.622022, -87.496532, 30.690476], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008720_se_16_1_20110816.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008720_se_16_1_20110816.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008720_se_16_1_20110816.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.496532, 30.622304], [-87.49688, 30.690476], [-87.56602, 30.690193], [-87.565623, 30.622022], [-87.496532, 30.622304]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [445790, 3387850, 452412, 3395405], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7555, 6622], \"eo:cloud_cover\": 82, \"proj:transform\": [1, 0, 445790, 0, -1, 3395405, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0046\", \"private\": {\"created_by\": \"stac-task\"}, \"bbox\": [-87.628562, 30.621988, -87.558988, 30.69051], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008720_sw_16_1_20110816.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008720_sw_16_1_20110816.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008720_sw_16_1_20110816.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.558988, 30.622303], [-87.559381, 30.69051], [-87.628562, 30.690194], [-87.62812, 30.621988], [-87.558988, 30.622303]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [439800, 3387878, 446426, 3395437], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7559, 6626], \"eo:cloud_cover\": 16, \"proj:transform\": [1, 0, 439800, 0, -1, 3395437, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0047\", \"private\": {\"created_by\": \"stac-task\"}, \"bbox\": [-87.941265, 30.559332, -87.871282, 30.628171], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008725_ne_16_1_20110816.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008725_ne_16_1_20110816.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008725_ne_16_1_20110816.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.871282, 30.559814], [-87.871893, 30.628171], [-87.941265, 30.627687], [-87.940604, 30.559332], [-87.871282, 30.559814]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [409792, 3381143, 416441, 3388719], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7576, 6649], \"eo:cloud_cover\": 43, \"proj:transform\": [1, 0, 409792, 0, -1, 3388719, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0048\", \"private\": {\"created_by\": \"stac-task\"}, \"bbox\": [-87.941255, 30.496828, -87.871287, 30.565666], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008725_se_16_1_20110816.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008725_se_16_1_20110816.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008725_se_16_1_20110816.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.871287, 30.497309], [-87.871897, 30.565666], [-87.941255, 30.565183], [-87.940596, 30.496828], [-87.871287, 30.497309]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [409735, 3374216, 416387, 3381792], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7576, 6652], \"eo:cloud_cover\": 4, \"proj:transform\": [1, 0, 409735, 0, -1, 3381792, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0049\", \"private\": {\"created_by\": \"stac-task\"}, \"bbox\": [-87.878723, 30.559362, -87.808825, 30.628132], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008726_nw_16_1_20110816.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008726_nw_16_1_20110816.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008726_nw_16_1_20110816.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.808825, 30.559811], [-87.809393, 30.628132], [-87.878723, 30.627682], [-87.878107, 30.559362], [-87.808825, 30.559811]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [415786, 3381098, 422431, 3388670], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7572, 6645], \"eo:cloud_cover\": 9, \"proj:transform\": [1, 0, 415786, 0, -1, 3388670, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0050\", \"private\": {\"created_by\": \"stac-task\"}, \"bbox\": [-87.878723, 30.496867, -87.808828, 30.565637], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008726_sw_16_1_20110816.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008726_sw_16_1_20110816.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008726_sw_16_1_20110816.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.808828, 30.497315], [-87.809395, 30.565637], [-87.878723, 30.565187], [-87.878108, 30.496867], [-87.808828, 30.497315]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [415732, 3374172, 422381, 3381744], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7572, 6649], \"eo:cloud_cover\": 84, \"proj:transform\": [1, 0, 415732, 0, -1, 3381744, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0051\", \"private\": {\"created_by\": \"stac-task\"}, \"bbox\": [-87.691097, 30.559454, -87.621453, 30.628045], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008727_ne_16_1_20110816.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008727_ne_16_1_20110816.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008727_ne_16_1_20110816.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.621453, 30.559803], [-87.621889, 30.628045], [-87.691097, 30.627696], [-87.690613, 30.559454], [-87.621453, 30.559803]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [433768, 3380983, 440401, 3388546], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7563, 6633], \"eo:cloud_cover\": 22, \"proj:transform\": [1, 0, 433768, 0, -1, 3388546, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0052\", \"private\": {\"created_by\": \"stac-task\"}, \"bbox\": [-87.691091, 30.496957, -87.621451, 30.565539], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008727_se_16_1_20110816.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008727_se_16_1_20110816.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008727_se_16_1_20110816.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.621451, 30.497305], [-87.621886, 30.565539], [-87.691091, 30.565191], [-87.690608, 30.496957], [-87.621451, 30.497305]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [433726, 3374057, 440363, 3381619], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7562, 6637], \"eo:cloud_cover\": 12, \"proj:transform\": [1, 0, 433726, 0, -1, 3381619, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0053\", \"private\": {\"created_by\": \"stac-task\"}, \"bbox\": [-87.56601, 30.559516, -87.496536, 30.627979], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008728_ne_16_1_20110816.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008728_ne_16_1_20110816.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008728_ne_16_1_20110816.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.496536, 30.559797], [-87.496884, 30.627979], [-87.56601, 30.627696], [-87.565614, 30.559516], [-87.496536, 30.559797]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [445756, 3380923, 452381, 3388479], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7556, 6625], \"eo:cloud_cover\": 24, \"proj:transform\": [1, 0, 445756, 0, -1, 3388479, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0054\", \"private\": {\"created_by\": \"stac-task\"}, \"bbox\": [-87.628554, 30.559491, -87.558995, 30.628014], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008728_nw_16_1_20110816.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008728_nw_16_1_20110816.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008728_nw_16_1_20110816.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.558995, 30.559806], [-87.559386, 30.628014], [-87.628554, 30.627698], [-87.628114, 30.559491], [-87.558995, 30.559806]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [439762, 3380952, 446391, 3388511], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7559, 6629], \"eo:cloud_cover\": 5, \"proj:transform\": [1, 0, 439762, 0, -1, 3388511, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0055\", \"private\": {\"created_by\": \"stac-task\"}, \"bbox\": [-87.566012, 30.497018, -87.496531, 30.565481], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008728_se_16_1_20110816.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008728_se_16_1_20110816.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008728_se_16_1_20110816.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.496531, 30.497299], [-87.496878, 30.565481], [-87.566012, 30.565199], [-87.565617, 30.497018], [-87.496531, 30.497299]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [445721, 3373997, 452351, 3381553], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7556, 6630], \"eo:cloud_cover\": 79, \"proj:transform\": [1, 0, 445721, 0, -1, 3381553, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0056\", \"private\": {\"created_by\": \"stac-task\"}, \"bbox\": [-87.941283, 30.934327, -87.871262, 31.003175], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008701_ne_16_1_20110816.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008701_ne_16_1_20110816.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008701_ne_16_1_20110816.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.871262, 30.934813], [-87.871883, 31.003175], [-87.941283, 31.002688], [-87.940613, 30.934327], [-87.871262, 30.934813]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [410140, 3422703, 416766, 3430280], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7577, 6626], \"eo:cloud_cover\": 62, \"proj:transform\": [1, 0, 410140, 0, -1, 3430280, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0057\", \"private\": {\"created_by\": \"stac-task\"}, \"bbox\": [-88.003827, 30.9343, -87.93372, 31.003207], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008701_nw_16_1_20110816.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008701_nw_16_1_20110816.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008701_nw_16_1_20110816.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.93372, 30.93482], [-87.934386, 31.003207], [-88.003827, 31.002686], [-88.003111, 30.9343], [-87.93372, 30.93482]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [404169, 3422752, 410799, 3430332], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7580, 6630], \"eo:cloud_cover\": 7, \"proj:transform\": [1, 0, 404169, 0, -1, 3430332, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0058\", \"private\": {\"created_by\": \"stac-task\"}, \"bbox\": [-87.941277, 30.871827, -87.871261, 30.940674], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008701_se_16_1_20110816.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008701_se_16_1_20110816.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008701_se_16_1_20110816.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.871261, 30.872312], [-87.87188, 30.940674], [-87.941277, 30.940187], [-87.940609, 30.871827], [-87.871261, 30.872312]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [410082, 3415776, 416712, 3423353], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7577, 6630], \"eo:cloud_cover\": 53, \"proj:transform\": [1, 0, 410082, 0, -1, 3423353, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0059\", \"private\": {\"created_by\": \"stac-task\"}, \"bbox\": [-88.003822, 30.8718, -87.93372, 30.940707], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008701_sw_16_1_20110816.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008701_sw_16_1_20110816.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008701_sw_16_1_20110816.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.93372, 30.872319], [-87.934384, 30.940707], [-88.003822, 30.940186], [-88.003108, 30.8718], [-87.93372, 30.872319]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [404107, 3415825, 410741, 3423405], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7580, 6634], \"eo:cloud_cover\": 57, \"proj:transform\": [1, 0, 404107, 0, -1, 3423405, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0066\", \"private\": {\"created_by\": \"stac-task\"}, \"bbox\": [-86.441023, 30.934491, -86.37143, 31.003012], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30086/m_3008605_ne_16_1_20110815.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30086/m_3008605_ne_16_1_20110815.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30086/m_3008605_ne_16_1_20110815.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-86.371877, 30.934491], [-86.37143, 31.002694], [-86.440626, 31.003012], [-86.441023, 30.934808], [-86.371877, 30.934491]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-15T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [553400, 3422511, 560006, 3430070], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7559, 6606], \"eo:cloud_cover\": 86, \"proj:transform\": [1, 0, 553400, 0, -1, 3430070, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0060\", \"private\": {\"created_by\": \"stac-task\"}, \"bbox\": [-87.878739, 30.934361, -87.808804, 31.00314], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008702_nw_16_1_20110816.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008702_nw_16_1_20110816.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008702_nw_16_1_20110816.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.808804, 30.934813], [-87.80938, 31.00314], [-87.878739, 31.002686], [-87.878114, 30.934361], [-87.808804, 30.934813]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [416111, 3422658, 422733, 3430231], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7573, 6622], \"eo:cloud_cover\": 70, \"proj:transform\": [1, 0, 416111, 0, -1, 3430231, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0061\", \"private\": {\"created_by\": \"stac-task\"}, \"bbox\": [-87.878743, 30.87186, -87.808801, 30.940638], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008702_sw_16_1_20110816.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008702_sw_16_1_20110816.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008702_sw_16_1_20110816.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.808801, 30.872311], [-87.809376, 30.940638], [-87.878743, 30.940186], [-87.878119, 30.87186], [-87.808801, 30.872311]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [416056, 3415731, 422683, 3423304], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7573, 6627], \"eo:cloud_cover\": 94, \"proj:transform\": [1, 0, 416056, 0, -1, 3423304, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0062\", \"private\": {\"created_by\": \"stac-task\"}, \"bbox\": [-87.691116, 30.934452, -87.621426, 31.003042], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008703_ne_16_1_20110816.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008703_ne_16_1_20110816.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008703_ne_16_1_20110816.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.621426, 30.934803], [-87.621868, 31.003042], [-87.691116, 31.00269], [-87.690624, 30.934452], [-87.621426, 30.934803]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [434023, 3422542, 440634, 3430105], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7563, 6611], \"eo:cloud_cover\": 85, \"proj:transform\": [1, 0, 434023, 0, -1, 3430105, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0063\", \"private\": {\"created_by\": \"stac-task\"}, \"bbox\": [-87.691116, 30.871958, -87.621431, 30.940539], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008703_se_16_1_20110816.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008703_se_16_1_20110816.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008703_se_16_1_20110816.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.621431, 30.872309], [-87.621872, 30.940539], [-87.691116, 30.940188], [-87.690626, 30.871958], [-87.621431, 30.872309]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-16T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [433980, 3415616, 440595, 3423178], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7562, 6615], \"eo:cloud_cover\": 2, \"proj:transform\": [1, 0, 433980, 0, -1, 3423178, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0064\", \"private\": {\"created_by\": \"stac-task\"}, \"bbox\": [-86.565944, 30.93455, -86.496512, 31.002952], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30086/m_3008604_ne_16_1_20110815.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30086/m_3008604_ne_16_1_20110815.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30086/m_3008604_ne_16_1_20110815.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-86.496869, 30.93455], [-86.496512, 31.002701], [-86.565635, 31.002952], [-86.565944, 30.934801], [-86.496869, 30.93455]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-15T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [541466, 3422457, 548065, 3430010], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7553, 6599], \"eo:cloud_cover\": 40, \"proj:transform\": [1, 0, 541466, 0, -1, 3430010, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0065\", \"private\": {\"created_by\": \"stac-task\"}, \"bbox\": [-86.628404, 30.93458, -86.559058, 31.002921], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30086/m_3008604_nw_16_1_20110815.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30086/m_3008604_nw_16_1_20110815.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30086/m_3008604_nw_16_1_20110815.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-86.559371, 30.93458], [-86.559058, 31.002704], [-86.62814, 31.002921], [-86.628404, 30.934797], [-86.559371, 30.93458]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-15T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [535499, 3422435, 542094, 3429985], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7550, 6595], \"eo:cloud_cover\": 27, \"proj:transform\": [1, 0, 535499, 0, -1, 3429985, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0067\", \"private\": {\"created_by\": \"stac-task\"}, \"bbox\": [-86.503483, 30.934518, -86.433965, 31.002979], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30086/m_3008605_nw_16_1_20110815.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30086/m_3008605_nw_16_1_20110815.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30086/m_3008605_nw_16_1_20110815.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-86.434367, 30.934518], [-86.433965, 31.002694], [-86.50313, 31.002979], [-86.503483, 30.934802], [-86.434367, 30.934518]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-15T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [547433, 3422482, 554036, 3430038], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7556, 6603], \"eo:cloud_cover\": 35, \"proj:transform\": [1, 0, 547433, 0, -1, 3430038, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0068\", \"private\": {\"created_by\": \"stac-task\"}, \"bbox\": [-86.316114, 30.934419, -86.246339, 31.003078], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30086/m_3008606_ne_16_1_20110815.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30086/m_3008606_ne_16_1_20110815.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30086/m_3008606_ne_16_1_20110815.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-86.246875, 30.934419], [-86.246339, 31.002692], [-86.315627, 31.003078], [-86.316114, 30.934803], [-86.246875, 30.934419]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-15T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [565333, 3422577, 571948, 3430144], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7567, 6615], \"eo:cloud_cover\": 22, \"proj:transform\": [1, 0, 565333, 0, -1, 3430144, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0069\", \"private\": {\"created_by\": \"stac-task\"}, \"bbox\": [-86.378574, 30.934452, -86.308884, 31.003042], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30086/m_3008606_nw_16_1_20110815.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30086/m_3008606_nw_16_1_20110815.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30086/m_3008606_nw_16_1_20110815.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-86.309376, 30.934452], [-86.308884, 31.00269], [-86.378132, 31.003042], [-86.378574, 30.934803], [-86.309376, 30.934452]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-15T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [559366, 3422542, 565977, 3430105], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7563, 6611], \"eo:cloud_cover\": 97, \"proj:transform\": [1, 0, 559366, 0, -1, 3430105, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0070\", \"private\": {\"created_by\": \"stac-task\"}, \"bbox\": [-86.191196, 30.934361, -86.121261, 31.00314], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30086/m_3008607_ne_16_1_20110815.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30086/m_3008607_ne_16_1_20110815.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30086/m_3008607_ne_16_1_20110815.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-86.121886, 30.934361], [-86.121261, 31.002686], [-86.19062, 31.00314], [-86.191196, 30.934813], [-86.121886, 30.934361]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-15T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [577267, 3422658, 583889, 3430231], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7573, 6622], \"eo:cloud_cover\": 32, \"proj:transform\": [1, 0, 577267, 0, -1, 3430231, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0071\", \"private\": {\"created_by\": \"stac-task\"}, \"bbox\": [-86.253655, 30.934391, -86.183805, 31.00311], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30086/m_3008607_nw_16_1_20110815.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30086/m_3008607_nw_16_1_20110815.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30086/m_3008607_nw_16_1_20110815.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-86.184386, 30.934391], [-86.183805, 31.002691], [-86.253123, 31.00311], [-86.253655, 30.93481], [-86.184386, 30.934391]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-15T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [571300, 3422616, 577918, 3430186], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7570, 6618], \"eo:cloud_cover\": 68, \"proj:transform\": [1, 0, 571300, 0, -1, 3430186, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0072\", \"private\": {\"created_by\": \"stac-task\"}, \"bbox\": [-86.06628, 30.9343, -85.996173, 31.003207], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30086/m_3008608_ne_16_1_20110815.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30086/m_3008608_ne_16_1_20110815.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30086/m_3008608_ne_16_1_20110815.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-85.996889, 30.9343], [-85.996173, 31.002686], [-86.065614, 31.003207], [-86.06628, 30.93482], [-85.996889, 30.9343]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-15T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [589201, 3422752, 595831, 3430332], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7580, 6630], \"eo:cloud_cover\": 30, \"proj:transform\": [1, 0, 589201, 0, -1, 3430332, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0073\", \"private\": {\"created_by\": \"stac-task\"}, \"bbox\": [-86.128738, 30.934327, -86.058717, 31.003175], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30086/m_3008608_nw_16_1_20110815.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30086/m_3008608_nw_16_1_20110815.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30086/m_3008608_nw_16_1_20110815.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-86.059387, 30.934327], [-86.058717, 31.002688], [-86.128117, 31.003175], [-86.128738, 30.934813], [-86.059387, 30.934327]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-15T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [583234, 3422703, 589860, 3430280], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7577, 6626], \"eo:cloud_cover\": 3, \"proj:transform\": [1, 0, 583234, 0, -1, 3430280, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0074\", \"private\": {\"created_by\": \"stac-task\"}, \"bbox\": [-85.816455, 30.934167, -85.746007, 31.00333], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30085/m_3008502_ne_16_1_20110815.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30085/m_3008502_ne_16_1_20110815.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30085/m_3008502_ne_16_1_20110815.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-85.746902, 30.934167], [-85.746007, 31.002673], [-85.815609, 31.00333], [-85.816455, 30.934822], [-85.746902, 30.934167]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-15T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [613069, 3422979, 619715, 3430573], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7594, 6646], \"eo:cloud_cover\": 52, \"proj:transform\": [1, 0, 613069, 0, -1, 3430573, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0075\", \"private\": {\"created_by\": \"stac-task\"}, \"bbox\": [-86.003823, 30.934269, -85.933631, 31.003236], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30085/m_3008501_nw_16_1_20110815.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30085/m_3008501_nw_16_1_20110815.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30085/m_3008501_nw_16_1_20110815.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-85.934391, 30.934269], [-85.933631, 31.002681], [-86.003112, 31.003236], [-86.003823, 30.934823], [-85.934391, 30.934269]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-15T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [595168, 3422804, 601802, 3430387], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7583, 6634], \"eo:cloud_cover\": 8, \"proj:transform\": [1, 0, 595168, 0, -1, 3430387, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0076\", \"private\": {\"created_by\": \"stac-task\"}, \"bbox\": [-85.941366, 30.934235, -85.871089, 31.00327], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30085/m_3008501_ne_16_1_20110815.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30085/m_3008501_ne_16_1_20110815.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30085/m_3008501_ne_16_1_20110815.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-85.871894, 30.934235], [-85.871089, 31.002681], [-85.940611, 31.00327], [-85.941366, 30.934823], [-85.871894, 30.934235]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-15T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [601135, 3422859, 607773, 3430446], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7587, 6638], \"eo:cloud_cover\": 59, \"proj:transform\": [1, 0, 601135, 0, -1, 3430446, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0077\", \"private\": {\"created_by\": \"stac-task\"}, \"bbox\": [-86.753316, 30.934648, -86.684141, 31.002851], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30086/m_3008603_nw_16_1_20110815.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30086/m_3008603_nw_16_1_20110815.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30086/m_3008603_nw_16_1_20110815.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-86.684365, 30.934648], [-86.684141, 31.0027], [-86.753141, 31.002851], [-86.753316, 30.934798], [-86.684365, 30.934648]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-15T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [523566, 3422402, 530153, 3429944], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7542, 6587], \"eo:cloud_cover\": 64, \"proj:transform\": [1, 0, 523566, 0, -1, 3429944, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0078\", \"private\": {\"created_by\": \"stac-task\"}, \"bbox\": [-86.690855, 30.934615, -86.621594, 31.002887], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30086/m_3008603_ne_16_1_20110815.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30086/m_3008603_ne_16_1_20110815.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30086/m_3008603_ne_16_1_20110815.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-86.621863, 30.934615], [-86.621594, 31.002704], [-86.690635, 31.002887], [-86.690855, 30.934799], [-86.621863, 30.934615]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-15T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [529533, 3422417, 536124, 3429963], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7546, 6591], \"eo:cloud_cover\": 21, \"proj:transform\": [1, 0, 529533, 0, -1, 3429963, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0079\", \"private\": {\"created_by\": \"stac-task\"}, \"bbox\": [-85.629082, 30.934072, -85.558378, 31.003423], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30085/m_3008504_nw_16_1_20110815.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30085/m_3008504_nw_16_1_20110815.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30085/m_3008504_nw_16_1_20110815.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-85.559409, 30.934072], [-85.558378, 31.002663], [-85.628101, 31.003423], [-85.629082, 30.934829], [-85.559409, 30.934072]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-15T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [630971, 3423185, 637629, 3430789], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7604, 6658], \"eo:cloud_cover\": 65, \"proj:transform\": [1, 0, 630971, 0, -1, 3430789, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0080\", \"private\": {\"created_by\": \"stac-task\"}, \"bbox\": [-85.566619, 30.934046, -85.49583, 31.003456], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30085/m_3008504_ne_16_1_20110815.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30085/m_3008504_ne_16_1_20110815.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30085/m_3008504_ne_16_1_20110815.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-85.496906, 30.934046], [-85.49583, 31.002662], [-85.565593, 31.003456], [-85.566619, 30.934838], [-85.496906, 30.934046]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-15T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [636939, 3423261, 643601, 3430868], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7607, 6662], \"eo:cloud_cover\": 95, \"proj:transform\": [1, 0, 636939, 0, -1, 3430868, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0081\", \"private\": {\"created_by\": \"stac-task\"}, \"bbox\": [-85.753989, 30.934141, -85.683456, 31.003364], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30085/m_3008503_nw_16_1_20110815.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30085/m_3008503_nw_16_1_20110815.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30085/m_3008503_nw_16_1_20110815.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-85.684396, 30.934141], [-85.683456, 31.002673], [-85.753099, 31.003364], [-85.753989, 30.934831], [-85.684396, 30.934141]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-15T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [619037, 3423045, 625687, 3430642], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7597, 6650], \"eo:cloud_cover\": 39, \"proj:transform\": [1, 0, 619037, 0, -1, 3430642, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0082\", \"private\": {\"created_by\": \"stac-task\"}, \"bbox\": [-85.691535, 30.934103, -85.620917, 31.003395], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30085/m_3008503_ne_16_1_20110815.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30085/m_3008503_ne_16_1_20110815.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30085/m_3008503_ne_16_1_20110815.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-85.621902, 30.934103], [-85.620917, 31.002669], [-85.6906, 31.003395], [-85.691535, 30.934827], [-85.621902, 30.934103]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-15T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [625004, 3423113, 631658, 3430714], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7601, 6654], \"eo:cloud_cover\": 26, \"proj:transform\": [1, 0, 625004, 0, -1, 3430714, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0001\", \"private\": {\"created_by\": \"stac-task\"}, \"bbox\": [-85.441706, 30.933975, -85.370747, 31.003522], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30085/m_3008505_ne_16_1_20110825.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30085/m_3008505_ne_16_1_20110825.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30085/m_3008505_ne_16_1_20110825.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-85.371913, 30.933975], [-85.370747, 31.00266], [-85.440589, 31.003522], [-85.441706, 30.934836], [-85.371913, 30.933975]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-25T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [648874, 3423421, 655544, 3431036], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7615, 6670], \"eo:cloud_cover\": 89, \"proj:transform\": [1, 0, 648874, 0, -1, 3431036, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0002\", \"private\": {\"created_by\": \"stac-task\"}, \"bbox\": [-85.504167, 30.934008, -85.433293, 31.003486], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30085/m_3008505_nw_16_1_20110825.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30085/m_3008505_nw_16_1_20110825.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30085/m_3008505_nw_16_1_20110825.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-85.434414, 30.934008], [-85.433293, 31.002658], [-85.503096, 31.003486], [-85.504167, 30.934834], [-85.434414, 30.934008]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-25T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [642906, 3423339, 649572, 3430950], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7611, 6666], \"eo:cloud_cover\": 33, \"proj:transform\": [1, 0, 642906, 0, -1, 3430950, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0083\", \"private\": {\"created_by\": \"stac-task\"}, \"bbox\": [-85.87891, 30.934198, -85.808547, 31.003302], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30085/m_3008502_nw_16_1_20110815.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30085/m_3008502_nw_16_1_20110815.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30085/m_3008502_nw_16_1_20110815.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-85.809397, 30.934198], [-85.808547, 31.002679], [-85.87811, 31.003302], [-85.87891, 30.934819], [-85.809397, 30.934198]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-15T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [607102, 3422917, 613744, 3430508], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7591, 6642], \"eo:cloud_cover\": 26, \"proj:transform\": [1, 0, 607102, 0, -1, 3430508, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0086\", \"private\": {\"created_by\": \"stac-task\"}, \"bbox\": [-87.816197, 30.87189, -87.746352, 30.940609], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008702_se_16_1_20110801.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008702_se_16_1_20110801.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008702_se_16_1_20110801.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.746352, 30.872308], [-87.746882, 30.940609], [-87.816197, 30.94019], [-87.815618, 30.87189], [-87.746352, 30.872308]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-01T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [422031, 3415689, 428653, 3423259], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7570, 6622], \"eo:cloud_cover\": 40, \"proj:transform\": [1, 0, 422031, 0, -1, 3423259, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0087\", \"private\": {\"created_by\": \"stac-task\"}, \"bbox\": [-87.753661, 30.934419, -87.683886, 31.003078], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008703_nw_16_1_20110801.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008703_nw_16_1_20110801.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008703_nw_16_1_20110801.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.683886, 30.934803], [-87.684373, 31.003078], [-87.753661, 31.002692], [-87.753125, 30.934419], [-87.683886, 30.934803]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-01T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [428052, 3422577, 434667, 3430144], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7567, 6615], \"eo:cloud_cover\": 36, \"proj:transform\": [1, 0, 428052, 0, -1, 3430144, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0088\", \"private\": {\"created_by\": \"stac-task\"}, \"bbox\": [-87.753652, 30.871926, -87.683891, 30.940576], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008703_sw_16_1_20110801.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008703_sw_16_1_20110801.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008703_sw_16_1_20110801.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.683891, 30.87231], [-87.684377, 30.940576], [-87.753652, 30.94019], [-87.753117, 30.871926], [-87.683891, 30.87231]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-01T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [428006, 3415651, 434624, 3423217], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7566, 6618], \"eo:cloud_cover\": 23, \"proj:transform\": [1, 0, 428006, 0, -1, 3423217, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0089\", \"private\": {\"created_by\": \"stac-task\"}, \"bbox\": [-87.816182, 30.55939, -87.746368, 30.6281], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008726_ne_16_1_20110731.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008726_ne_16_1_20110731.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008726_ne_16_1_20110731.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.746368, 30.559805], [-87.746892, 30.6281], [-87.816182, 30.627684], [-87.815609, 30.55939], [-87.746368, 30.559805]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-07-31T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [421780, 3381056, 428421, 3388625], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7569, 6641], \"eo:cloud_cover\": 36, \"proj:transform\": [1, 0, 421780, 0, -1, 3388625, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0090\", \"private\": {\"created_by\": \"stac-task\"}, \"bbox\": [-87.81619, 30.809396, -87.746349, 30.878106], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008710_ne_16_1_20110731.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008710_ne_16_1_20110731.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008710_ne_16_1_20110731.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.746349, 30.809814], [-87.746878, 30.878106], [-87.81619, 30.877688], [-87.815612, 30.809396], [-87.746349, 30.809814]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-07-31T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [421981, 3408763, 428607, 3416332], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7569, 6626], \"eo:cloud_cover\": 84, \"proj:transform\": [1, 0, 421981, 0, -1, 3416332, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0091\", \"private\": {\"created_by\": \"stac-task\"}, \"bbox\": [-87.816185, 30.621895, -87.746357, 30.690605], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008718_se_16_1_20110731.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008718_se_16_1_20110731.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008718_se_16_1_20110731.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.746357, 30.62231], [-87.746882, 30.690605], [-87.816185, 30.690188], [-87.815611, 30.621895], [-87.746357, 30.62231]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-07-31T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [421830, 3387983, 428468, 3395552], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7569, 6638], \"eo:cloud_cover\": 31, \"proj:transform\": [1, 0, 421830, 0, -1, 3395552, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0092\", \"private\": {\"created_by\": \"stac-task\"}, \"bbox\": [-87.816194, 30.746893, -87.746358, 30.815603], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008710_se_16_1_20110731.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008710_se_16_1_20110731.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008710_se_16_1_20110731.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.746358, 30.74731], [-87.746885, 30.815603], [-87.816194, 30.815185], [-87.815618, 30.746893], [-87.746358, 30.74731]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-07-31T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [421930, 3401836, 428560, 3409405], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7569, 6630], \"eo:cloud_cover\": 24, \"proj:transform\": [1, 0, 421930, 0, -1, 3409405, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0093\", \"private\": {\"created_by\": \"stac-task\"}, \"bbox\": [-87.753639, 30.559423, -87.683911, 30.628074], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008727_nw_16_1_20110731.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008727_nw_16_1_20110731.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008727_nw_16_1_20110731.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.683911, 30.559805], [-87.68439, 30.628074], [-87.753639, 30.627692], [-87.753111, 30.559423], [-87.683911, 30.559805]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-07-31T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [427774, 3381018, 434411, 3388584], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7566, 6637], \"eo:cloud_cover\": 61, \"proj:transform\": [1, 0, 427774, 0, -1, 3388584, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0094\", \"private\": {\"created_by\": \"stac-task\"}, \"bbox\": [-87.753654, 30.809423, -87.683898, 30.878073], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008711_nw_16_1_20110731.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008711_nw_16_1_20110731.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008711_nw_16_1_20110731.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.683898, 30.809806], [-87.684382, 30.878073], [-87.753654, 30.877688], [-87.75312, 30.809423], [-87.683898, 30.809806]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-07-31T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [427959, 3408724, 434581, 3416290], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7566, 6622], \"eo:cloud_cover\": 29, \"proj:transform\": [1, 0, 427959, 0, -1, 3416290, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0095\", \"private\": {\"created_by\": \"stac-task\"}, \"bbox\": [-87.753646, 30.746928, -87.683895, 30.815579], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008711_sw_16_1_20110731.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008711_sw_16_1_20110731.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008711_sw_16_1_20110731.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.683895, 30.747312], [-87.684378, 30.815579], [-87.753646, 30.815194], [-87.753114, 30.746928], [-87.683895, 30.747312]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-07-31T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [427913, 3401798, 434539, 3409364], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7566, 6626], \"eo:cloud_cover\": 63, \"proj:transform\": [1, 0, 427913, 0, -1, 3409364, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0096\", \"private\": {\"created_by\": \"stac-task\"}, \"bbox\": [-87.753639, 30.684424, -87.683903, 30.753075], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008719_nw_16_1_20110731.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008719_nw_16_1_20110731.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008719_nw_16_1_20110731.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.683903, 30.684807], [-87.684385, 30.753075], [-87.753639, 30.752691], [-87.753109, 30.684424], [-87.683903, 30.684807]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-07-31T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [427867, 3394871, 434496, 3402437], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7566, 6629], \"eo:cloud_cover\": 26, \"proj:transform\": [1, 0, 427867, 0, -1, 3402437, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0097\", \"private\": {\"created_by\": \"stac-task\"}, \"bbox\": [-87.816189, 30.68439, -87.746357, 30.753109], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008718_ne_16_1_20110731.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008718_ne_16_1_20110731.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008718_ne_16_1_20110731.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.746357, 30.684806], [-87.746883, 30.753109], [-87.816189, 30.752692], [-87.815614, 30.68439], [-87.746357, 30.684806]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-07-31T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [421880, 3394909, 428514, 3402479], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7570, 6634], \"eo:cloud_cover\": 1, \"proj:transform\": [1, 0, 421880, 0, -1, 3402479, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0098\", \"private\": {\"created_by\": \"stac-task\"}, \"bbox\": [-87.753644, 30.621929, -87.683901, 30.69057], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008719_sw_16_1_20110731.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008719_sw_16_1_20110731.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008719_sw_16_1_20110731.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.683901, 30.622311], [-87.684382, 30.69057], [-87.753644, 30.690187], [-87.753115, 30.621929], [-87.683901, 30.622311]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-07-31T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [427820, 3387945, 434454, 3395510], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7565, 6634], \"eo:cloud_cover\": 73, \"proj:transform\": [1, 0, 427820, 0, -1, 3395510, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0099\", \"private\": {\"created_by\": \"stac-task\"}, \"bbox\": [-87.753635, 30.496927, -87.683911, 30.565569], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008727_sw_16_1_20110731.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008727_sw_16_1_20110731.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008727_sw_16_1_20110731.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.683911, 30.497308], [-87.684389, 30.565569], [-87.753635, 30.565186], [-87.753109, 30.496927], [-87.683911, 30.497308]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-07-31T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [427728, 3374092, 434369, 3381657], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7565, 6641], \"eo:cloud_cover\": 90, \"proj:transform\": [1, 0, 427728, 0, -1, 3381657, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0084\", \"private\": {\"created_by\": \"stac-task\"}, \"bbox\": [-85.316796, 30.93392, -85.245667, 31.003585], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30085/m_3008506_ne_16_1_20110802.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30085/m_3008506_ne_16_1_20110802.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30085/m_3008506_ne_16_1_20110802.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-85.246924, 30.93392], [-85.245667, 31.002654], [-85.315589, 31.003585], [-85.316796, 30.934848], [-85.246924, 30.93392]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-02T00:00:00Z\", \"naip:year\": \"2011\", \"proj:bbox\": [660809, 3423596, 667487, 3431217], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"al\", \"proj:shape\": [7621, 6678], \"eo:cloud_cover\": 52, \"proj:transform\": [1, 0, 660809, 0, -1, 3431217, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0085\", \"private\": {\"created_by\": \"stac-task\"}, \"bbox\": [-87.816195, 30.934391, -87.746345, 31.00311], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008702_ne_16_1_20110801.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008702_ne_16_1_20110801.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008702_ne_16_1_20110801.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.746345, 30.93481], [-87.746877, 31.00311], [-87.816195, 31.002691], [-87.815614, 30.934391], [-87.746345, 30.93481]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-08-01T00:00:00Z\", \"naip:year\": \"2012\", \"proj:bbox\": [422082, 3422616, 428700, 3430186], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"xx\", \"proj:shape\": [7570, 6618], \"eo:cloud_cover\": 3, \"proj:transform\": [1, 0, 422082, 0, -1, 3430186, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n{\"id\": \"pgstac-test-item-0100\", \"private\": {\"created_by\": \"stac-task\"}, \"bbox\": [-87.816179, 30.496894, -87.74637, 30.565604], \"type\": \"Feature\", \"links\": [], \"assets\": {\"image\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008726_se_16_1_20110731.tif\", \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\", \"roles\": [\"data\"], \"title\": \"RGBIR COG tile\", \"eo:bands\": [{\"name\": \"Red\", \"common_name\": \"red\"}, {\"name\": \"Green\", \"common_name\": \"green\"}, {\"name\": \"Blue\", \"common_name\": \"blue\"}, {\"name\": \"NIR\", \"common_name\": \"nir\", \"description\": \"near-infrared\"}]}, \"metadata\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_fgdc_2011/30087/m_3008726_se_16_1_20110731.txt\", \"type\": \"text/plain\", \"roles\": [\"metadata\"], \"title\": \"FGDC Metdata\"}, \"thumbnail\": {\"href\": \"https://naipeuwest.blob.core.windows.net/naip/v002/al/2011/al_100cm_2011/30087/m_3008726_se_16_1_20110731.200.jpg\", \"type\": \"image/jpeg\", \"roles\": [\"thumbnail\"], \"title\": \"Thumbnail\"}}, \"geometry\": {\"type\": \"Polygon\", \"coordinates\": [[[-87.74637, 30.497308], [-87.746892, 30.565604], [-87.816179, 30.565188], [-87.815608, 30.496894], [-87.74637, 30.497308]]]}, \"collection\": \"pgstac-test-collection\", \"properties\": {\"gsd\": 1, \"datetime\": \"2011-07-31T00:00:00Z\", \"naip:year\": 2013, \"proj:bbox\": [421730, 3374130, 428375, 3381699], \"proj:epsg\": 26916, \"providers\": [{\"url\": \"https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/\", \"name\": \"USDA Farm Service Agency\", \"roles\": [\"producer\", \"licensor\"]}], \"naip:state\": \"zz\", \"proj:shape\": [7569, 6645], \"eo:cloud_cover\": 50, \"proj:transform\": [1, 0, 421730, 0, -1, 3381699, 0, 0, 1]}, \"stac_version\": \"1.0.0-beta.2\", \"stac_extensions\": [\"eo\", \"projection\"]}\n" }, { "path": "src/pypgstac/README.md", "content": "# pypgstac\n\nPython tools for working with PgSTAC\n" }, { "path": "src/pypgstac/examples/load_queryables_example.py", "content": "#!/usr/bin/env python\n\"\"\"\nExample script demonstrating how to load queryables into PgSTAC.\n\nThis script shows how to use the load_queryables function both from the command line\nand programmatically.\n\"\"\"\n\nimport sys\nfrom pathlib import Path\n\n# Add the parent directory to the path so we can import pypgstac\nsys.path.append(str(Path(__file__).parent.parent))\n\nfrom pypgstac.pypgstac import PgstacCLI\n\n\ndef load_for_specific_collections(\n cli, sample_file, collection_ids, delete_missing=False,\n):\n \"\"\"Load queryables for specific collections.\n\n Args:\n cli: PgstacCLI instance\n sample_file: Path to the queryables file\n collection_ids: List of collection IDs to apply queryables to\n delete_missing: If True, delete properties not present in the file\n \"\"\"\n cli.load_queryables(\n str(sample_file), collection_ids=collection_ids, delete_missing=delete_missing,\n )\n\n\ndef main():\n \"\"\"Demonstrate loading queryables into PgSTAC.\"\"\"\n # Get the path to the sample queryables file\n sample_file = Path(__file__).parent / \"sample_queryables.json\"\n\n # Check if the file exists\n if not sample_file.exists():\n return\n\n # Create a PgstacCLI instance\n # This will use the standard PostgreSQL environment variables for connection\n cli = PgstacCLI()\n\n # Load queryables for all collections\n cli.load_queryables(str(sample_file))\n\n # Example of loading for specific collections\n load_for_specific_collections(cli, sample_file, [\"landsat-8\", \"sentinel-2\"])\n\n # Example of loading queryables with delete_missing=True\n # This will delete properties not present in the file\n cli.load_queryables(str(sample_file), delete_missing=True)\n\n # Example of loading for specific collections with delete_missing=True\n # This will delete properties not present in the file, but only for the specified collections\n load_for_specific_collections(\n cli, sample_file, [\"landsat-8\", \"sentinel-2\"], delete_missing=True,\n )\n\n\nif __name__ == \"__main__\":\n main()\n" }, { "path": "src/pypgstac/examples/sample_queryables.json", "content": "{\n \"$schema\": \"https://json-schema.org/draft/2019-09/schema\",\n \"$id\": \"https://example.com/stac/queryables\",\n \"type\": \"object\",\n \"title\": \"Queryables for Example STAC API\",\n \"description\": \"Queryable names for the Example STAC API\",\n \"properties\": {\n \"id\": {\n \"description\": \"Item identifier\",\n \"type\": \"string\"\n },\n \"collection\": {\n \"description\": \"Collection identifier\",\n \"type\": \"string\"\n },\n \"datetime\": {\n \"description\": \"Datetime\",\n \"type\": \"string\",\n \"format\": \"date-time\"\n },\n \"geometry\": {\n \"description\": \"Geometry\",\n \"type\": \"object\"\n },\n \"eo:cloud_cover\": {\n \"description\": \"Cloud cover percentage\",\n \"type\": \"number\",\n \"minimum\": 0,\n \"maximum\": 100\n },\n \"platform\": {\n \"description\": \"Platform name\",\n \"type\": \"string\",\n \"enum\": [\"landsat-8\", \"sentinel-2\"]\n },\n \"instrument\": {\n \"description\": \"Instrument name\",\n \"type\": \"string\"\n },\n \"gsd\": {\n \"description\": \"Ground sample distance in meters\",\n \"type\": \"number\"\n },\n \"view:off_nadir\": {\n \"description\": \"Off-nadir angle in degrees\",\n \"type\": \"number\"\n },\n \"view:sun_azimuth\": {\n \"description\": \"Sun azimuth angle in degrees\",\n \"type\": \"number\"\n },\n \"view:sun_elevation\": {\n \"description\": \"Sun elevation angle in degrees\",\n \"type\": \"number\"\n },\n \"sci:doi\": {\n \"description\": \"Digital Object Identifier\",\n \"type\": \"string\"\n },\n \"created\": {\n \"description\": \"Date and time the item was created\",\n \"type\": \"string\",\n \"format\": \"date-time\"\n },\n \"updated\": {\n \"description\": \"Date and time the item was last updated\",\n \"type\": \"string\",\n \"format\": \"date-time\"\n },\n \"landcover:classes\": {\n \"description\": \"Land cover classes\",\n \"type\": \"array\",\n \"items\": {\n \"type\": \"string\"\n }\n }\n },\n \"additionalProperties\": true\n}\n" }, { "path": "src/pypgstac/pyproject.toml", "content": "[project]\nname = \"pypgstac\"\nversion = \"0.9.11-dev\"\ndescription = \"Schema, functions and a python library for storing and accessing STAC collections and items in PostgreSQL\"\nreadme = \"README.md\"\nrequires-python = \">=3.11\"\nlicense = \"MIT\"\nauthors = [{ name = \"David Bitner\", email = \"bitner@dbspatial.com\" }]\nkeywords = [\"STAC\", \"Postgresql\", \"PgSTAC\"]\nclassifiers = [\n \"Intended Audience :: Developers\",\n \"Intended Audience :: Information Technology\",\n \"Intended Audience :: Science/Research\",\n \"License :: OSI Approved :: MIT License\",\n \"Programming Language :: Python :: 3.11\",\n \"Programming Language :: Python :: 3.12\",\n \"Programming Language :: Python :: 3.13\",\n \"Programming Language :: Python :: 3.14\",\n]\ndependencies = [\n \"cachetools>=5.3.0\",\n \"fire>=0.7.0\",\n \"hydraters>=0.1.0\",\n \"orjson>=3.11.0\",\n \"plpygis>=0.5.0\",\n \"pydantic>=2.10,<3\",\n \"pydantic-settings>=2,<3\",\n \"python-dateutil>=2.8.0\",\n \"smart-open>=5.0\",\n \"tenacity>=8.1.0\",\n \"version-parser>=1.0.1\",\n]\n\n[project.optional-dependencies]\ntest = [\n \"morecantile>=6.2,<7.1\",\n \"pytest>=8.3,<9.1\",\n \"pytest-benchmark>=5.1,<5.3\",\n \"pytest-cov>=6.0,<7.2\",\n \"pystac[validation]==1.*\",\n \"types-cachetools>=5.5\",\n]\ndev = [\n \"types-setuptools\",\n \"ruff==0.15.12\",\n \"ty==0.0.35\",\n \"pre-commit==4.6.0\",\n]\npsycopg = [\"psycopg[binary]>=3.1.0\", \"psycopg-pool>=3.1.0\"]\nmigrations = [\"psycopg2-binary\", \"migra\"]\ndocs = [\n \"jupyter\",\n \"pandas\",\n \"seaborn\",\n \"mkdocs-jupyter\",\n \"folium\"\n]\n\n\n[project.urls]\nHomepage = \"https://stac-utils.github.io/pgstac/\"\nDocumentation = \"https://stac-utils.github.io/pgstac/\"\nIssues = \"https://github.com/stac-utils/pgstac/issues\"\nSource = \"https://github.com/stac-utils/pgstac\"\nChangelog = \"https://stac-utils.github.io/pgstac/release-notes/\"\n\n[project.scripts]\npypgstac = \"pypgstac.pypgstac:cli\"\n\n[build-system]\nrequires = [\"hatchling\"]\nbuild-backend = \"hatchling.build\"\n\n[tool.coverage.run]\nbranch = true\nparallel = true\n\n[tool.coverage.report]\nexclude_lines = [\"no cov\", \"if __name__ == .__main__.:\", \"if TYPE_CHECKING:\"]\n\n[tool.ruff]\ntarget-version = \"py311\"\nline-length = 88\n\n[tool.ruff.lint]\nselect = [\n \"E\", # pycodestyle errors\n \"W\", # pycodestyle warnings\n \"F\", # pyflakes\n \"I\", # isort\n \"B\", # flake8-bugbear\n \"C4\", # flake8-comprehensions\n \"T20\", # flake8-print\n \"Q\", # flake8-quotes\n \"DTZ\", # flake8-datetimez\n \"ERA\", # eradicate\n \"PLC\",\n \"PLE\",\n \"PLW\",\n \"COM\", # flake8-commas\n]\nignore = [\n \"E501\", # line too long, handled by formatter\n \"B008\", # do not perform function calls in argument defaults\n \"C901\", # too complex\n \"B905\",\n]\n\n[tool.ruff.lint.isort]\nknown-first-party = [\"pypgstac\"]\n\n[tool.ty.environment]\npython-version = \"3.11\"\n\n[tool.ty.src]\ninclude = [\"src/pypgstac\", \"tests\"]\n\n[tool.pydocstyle]\nselect = \"D1\"\nmatch = \"(?!test).*.py\"\n\n[tool.pytest.ini_options]\naddopts = \"-vv --benchmark-skip\"\n" }, { "path": "src/pypgstac/src/pypgstac/__init__.py", "content": "\"\"\"pyPgSTAC Version.\"\"\"\n\nfrom pypgstac.version import __version__\n\n__all__ = [\"__version__\"]\n" }, { "path": "src/pypgstac/src/pypgstac/db.py", "content": "\"\"\"Base library for database interaction with PgSTAC.\"\"\"\n\nimport atexit\nimport logging\nimport time\nfrom collections.abc import Generator\nfrom pathlib import Path\nfrom types import TracebackType\nfrom typing import Any\n\nimport orjson\nimport psycopg\nfrom psycopg import Connection, rows, sql\nfrom psycopg.abc import Params\nfrom psycopg.types import json as psycopg_json\nfrom psycopg.types.json import set_json_dumps, set_json_loads\nfrom psycopg_pool import ConnectionPool\nfrom pydantic_settings import BaseSettings, SettingsConfigDict\nfrom tenacity import retry, retry_if_exception_type, stop_after_attempt\n\nlogger = logging.getLogger(__name__)\n\n\ndef dumps(data: dict) -> str:\n \"\"\"Dump dictionary as string.\"\"\"\n return orjson.dumps(data).decode()\n\n\nset_json_dumps(dumps)\nset_json_loads(orjson.loads)\n\n\ndef pg_notice_handler(notice: psycopg.errors.Diagnostic) -> None:\n \"\"\"Add PG messages to logging.\"\"\"\n msg = f\"{notice.severity} - {notice.message_primary}\"\n logger.info(msg)\n\n\nclass Settings(BaseSettings):\n \"\"\"Base Settings for Database Connection.\"\"\"\n\n db_min_conn_size: int = 0\n db_max_conn_size: int = 1\n db_max_queries: int = 5\n db_max_idle: int = 5\n db_num_workers: int = 1\n db_retries: int = 3\n\n model_config = SettingsConfigDict(env_file=Path(\".env\"), extra=\"ignore\")\n\n\nsettings = Settings()\n\n\nclass PgstacDB:\n \"\"\"Base class for interacting with PgSTAC Database.\"\"\"\n\n def __init__(\n self,\n dsn: str | None = \"\",\n pool: ConnectionPool | None = None,\n connection: Connection | None = None,\n commit_on_exit: bool = True,\n debug: bool = False,\n use_queue: bool = False,\n ) -> None:\n \"\"\"Initialize Database.\"\"\"\n self.dsn: str\n if dsn is not None:\n self.dsn = dsn\n else:\n self.dsn = \"\"\n self.pool = pool\n self.connection = connection\n self.commit_on_exit = commit_on_exit\n self.initial_version = \"0.1.9\"\n self.debug = debug\n self.use_queue = use_queue\n if self.debug:\n logging.basicConfig(level=logging.DEBUG)\n\n def get_pool(self) -> ConnectionPool:\n \"\"\"Get Database Pool.\"\"\"\n if self.pool is None:\n self.pool = ConnectionPool(\n conninfo=self.dsn,\n min_size=settings.db_min_conn_size,\n max_size=settings.db_max_conn_size,\n max_waiting=settings.db_max_queries,\n max_idle=settings.db_max_idle,\n num_workers=settings.db_num_workers,\n open=True,\n )\n return self.pool\n\n def open(self) -> None:\n \"\"\"Open database pool connection.\"\"\"\n self.get_pool()\n\n def close(self) -> None:\n \"\"\"Close database pool connection.\"\"\"\n if self.pool is not None:\n self.pool.close()\n\n def connect(self) -> Connection:\n \"\"\"Return database connection.\"\"\"\n pool = self.get_pool()\n if self.connection is None or self.connection.closed or self.connection.broken:\n self.connection = pool.getconn()\n self.connection.autocommit = True\n if self.debug:\n self.connection.add_notice_handler(pg_notice_handler)\n self.connection.execute(\n \"SET CLIENT_MIN_MESSAGES TO NOTICE;\",\n prepare=False,\n )\n if self.use_queue:\n self.connection.execute(\n \"SET pgstac.use_queue TO TRUE;\",\n prepare=False,\n )\n atexit.register(self.disconnect)\n self.connection.execute(\n \"\"\"\n SELECT\n CASE\n WHEN\n current_setting('search_path', false) ~* '\\\\mpgstac\\\\M'\n THEN current_setting('search_path', false)\n ELSE set_config(\n 'search_path',\n 'pgstac,' || current_setting('search_path', false),\n false\n )\n END\n ;\n SET application_name TO 'pgstac';\n \"\"\",\n prepare=False,\n )\n return self.connection\n\n def wait(self) -> None:\n \"\"\"Block until database connection is ready.\"\"\"\n cnt: int = 0\n while cnt < 60:\n try:\n self.connect()\n self.query(\"SELECT 1;\")\n return None\n except psycopg.errors.OperationalError:\n time.sleep(1)\n cnt += 1\n raise psycopg.errors.CannotConnectNow\n\n def disconnect(self) -> None:\n \"\"\"Disconnect from database.\"\"\"\n try:\n if self.connection is not None:\n if self.commit_on_exit:\n self.connection.commit()\n else:\n self.connection.rollback()\n except Exception:\n pass\n try:\n if self.pool is not None and self.connection is not None:\n self.pool.putconn(self.connection)\n except Exception:\n pass\n\n self.connection = None\n self.pool = None\n\n def __enter__(self) -> Any:\n \"\"\"Enter used for context.\"\"\"\n self.connect()\n return self\n\n def __exit__(\n self,\n exc_type: type[BaseException] | None,\n exc: BaseException | None,\n traceback: TracebackType | None,\n ) -> None:\n \"\"\"Exit used for context.\"\"\"\n self.disconnect()\n\n @retry(\n stop=stop_after_attempt(settings.db_retries),\n retry=retry_if_exception_type(psycopg.errors.OperationalError),\n reraise=True,\n )\n def query(\n self,\n query: Any,\n args: Params | None = None,\n row_factory: rows.BaseRowFactory = rows.tuple_row,\n ) -> Generator:\n \"\"\"Query the database with parameters.\"\"\"\n conn = self.connect()\n try:\n with conn.cursor(row_factory=row_factory) as cursor:\n if args is None:\n rows = cursor.execute(query, prepare=False)\n else:\n rows = cursor.execute(query, args)\n if rows:\n for row in rows:\n yield row\n else:\n yield None\n except psycopg.errors.OperationalError as e:\n # If we get an operational error check the pool and retry\n logger.warning(f\"OPERATIONAL ERROR: {e}\")\n if self.pool is None:\n self.get_pool()\n else:\n self.pool.check()\n raise e\n except psycopg.errors.DatabaseError as e:\n if conn is not None:\n conn.rollback()\n raise e\n\n def query_one(self, *args: Any, **kwargs: Any) -> tuple[Any, ...] | str | None:\n \"\"\"Return results from a query that returns a single row.\"\"\"\n try:\n r = next(self.query(*args, **kwargs))\n except StopIteration:\n return None\n\n if r is None:\n return None\n if len(r) == 1:\n return r[0]\n return r\n\n def run_queued(self) -> str:\n try:\n self.connect().execute(\"\"\"\n CALL run_queued_queries();\n \"\"\")\n return \"Ran Queued Queries\"\n except Exception as e:\n return f\"Error Running Queued Queries: {e}\"\n\n @property\n def version(self) -> str | None:\n \"\"\"Get the current version number from a pgstac database.\"\"\"\n try:\n version = self.query_one(\n \"\"\"\n SELECT version from pgstac.migrations\n order by datetime desc, version desc limit 1;\n \"\"\",\n )\n logger.debug(f\"VERSION: {version}\")\n if isinstance(version, bytes):\n version = version.decode()\n if isinstance(version, str):\n return version\n except psycopg.errors.UndefinedTable:\n logger.debug(\"PgSTAC is not installed.\")\n if self.connection is not None:\n self.connection.rollback()\n return None\n\n @property\n def pg_version(self) -> str:\n \"\"\"Get the current pg version number from a pgstac database.\"\"\"\n version = self.query_one(\n \"\"\"\n SHOW server_version_num;\n \"\"\",\n )\n logger.debug(f\"PG VERSION: {version}.\")\n if isinstance(version, bytes):\n version = version.decode()\n if isinstance(version, str):\n if int(version) < 130000:\n major, minor, patch = tuple(\n map(int, [version[i : i + 2] for i in range(0, len(version), 2)]),\n )\n raise Exception(\n f\"PgSTAC requires PostgreSQL 13+, current version is: {major}.{minor}.{patch}\",\n ) # noqa: E501\n return version\n else:\n if self.connection is not None:\n self.connection.rollback()\n raise Exception(\"Could not find PG version.\")\n\n def func(self, function_name: str, *args: Any) -> Generator:\n \"\"\"Call a database function.\"\"\"\n placeholders = sql.SQL(\", \").join(sql.Placeholder() * len(args))\n func = sql.Identifier(function_name)\n cleaned_args = []\n for arg in args:\n if isinstance(arg, dict):\n cleaned_args.append(psycopg_json.Jsonb(arg))\n else:\n cleaned_args.append(arg)\n base_query = sql.SQL(\"SELECT * FROM {}({});\").format(func, placeholders)\n return self.query(base_query, cleaned_args)\n\n def search(self, query: dict | str | psycopg_json.Jsonb = \"{}\") -> str:\n \"\"\"Search PgSTAC.\"\"\"\n return dumps(next(self.func(\"search\", query))[0])\n" }, { "path": "src/pypgstac/src/pypgstac/hydration.py", "content": "\"\"\"Hydrate data in pypgstac rather than on the database.\"\"\"\n\nfrom copy import deepcopy\nfrom typing import Any, Mapping, cast\n\nfrom hydraters import hydrate\n\n# Marker value to indicate that a key should not be rehydrated\nDO_NOT_MERGE_MARKER = \"𒍟※\"\n\n\ndef hydrate_py(base_item: dict[str, Any], item: dict[str, Any]) -> dict[str, Any]:\n \"\"\"Hydrate item in-place with base_item properties.\n\n This will not perform a deep copy; values of the original item will be referenced\n in the return item.\n \"\"\"\n\n # Merge will mutate i, but create deep copies of values in the base item\n # This will prevent the base item values from being mutated, e.g. by\n # filtering out fields in `filter_fields`.\n def merge(b: dict[str, Any], i: dict[str, Any]) -> None:\n for key, _ in b.items():\n if key in i:\n if isinstance(b[key], dict) and isinstance(i.get(key), dict):\n # Recurse on dicts to merge values\n merge(b[key], i[key])\n elif isinstance(b[key], list) and isinstance(i.get(key), list):\n # Merge unequal lists, assume uniform types\n if len(b[key]) == len(i[key]):\n for bb, ii in zip(b[key], i[key]):\n # Make sure we're merging two dicts\n if isinstance(bb, dict) and isinstance(ii, dict):\n merge(bb, ii)\n else:\n # If item has a different length, then just use the item value\n continue\n else:\n # Key exists on item but isn't a dict or list, keep item value\n if i[key] == DO_NOT_MERGE_MARKER:\n # Key was marked as do-not-merge, drop it from the item\n del i[key]\n else:\n # Keep the item value\n continue\n\n else:\n # Keys in base item that are not in item are simply copied over\n i[key] = deepcopy(b[key])\n\n merge(base_item, item)\n return item\n\n\ndef dehydrate(base_item: dict[str, Any], full_item: dict[str, Any]) -> dict[str, Any]:\n \"\"\"\n Get a recursive difference between a base item and an incoming item to dehydrate.\n\n For keys of dicts within items, if the base item contains a key not present\n in the incoming item, then a do-no-merge value is added indicating that the\n key should not be rehydrated with the corresponding base item value. This will allow\n collection item-assets to contain keys that may not be present on individual items.\n \"\"\"\n\n def strip(\n base_value: Mapping[str, Any],\n item_value: Mapping[str, Any],\n ) -> dict[str, Any]:\n out: dict = {}\n for key, value in item_value.items():\n if base_value is None or key not in base_value:\n # Nothing on base; preserve item value in the dehydrated item\n out[key] = value\n continue\n\n if base_value[key] == value:\n # Equal values; do not include in the dehydrated item\n continue\n\n if isinstance(base_value[key], list) and isinstance(value, list):\n if len(base_value[key]) == len(value):\n # Equal length lists dehydrate dicts at each matching index\n # and use incoming item values for other types\n out[key] = []\n for bv, v in zip(base_value[key], value):\n if isinstance(bv, dict) and isinstance(v, dict):\n bv_mapping = cast(Mapping[str, Any], bv)\n v_mapping = cast(Mapping[str, Any], v)\n dehydrated = strip(bv_mapping, v_mapping)\n apply_marked_keys(bv_mapping, v_mapping, dehydrated)\n out[key].append(dehydrated)\n else:\n out[key].append(v)\n else:\n # Unequal length lists are not dehydrated and just use the\n # incoming item value\n out[key] = value\n continue\n\n if value is None or value == []:\n # Don't keep empty values\n continue\n\n if isinstance(value, dict) and isinstance(base_value[key], Mapping):\n # After dehdrating a dict, mark any keys that are present on the\n # base item but not in the incoming item as `do-not-merge` during\n # rehydration\n dehydrated = strip(base_value[key], value)\n apply_marked_keys(base_value[key], value, dehydrated)\n out[key] = dehydrated\n continue\n else:\n # Unequal non-dict values are copied over from the incoming item\n out[key] = value\n\n # Mark any top-level keys from the base_item that are not in the incoming item\n apply_marked_keys(base_value, item_value, out)\n\n return out\n\n return strip(base_item, full_item)\n\n\ndef apply_marked_keys(\n base_item: Mapping[str, Any],\n full_item: Mapping[str, Any],\n dehydrated: dict[str, Any],\n) -> None:\n \"\"\"Mark keys.\n\n Mark any keys that are present on the base item but not in the incoming item\n as `do-not-merge` on the dehydrated item. This will prevent they key from\n being rehydrated.\n\n This modifies the dehydrated item in-place.\n \"\"\"\n try:\n marked_keys = [key for key in base_item if key not in full_item]\n marked_dict = dict.fromkeys(marked_keys, DO_NOT_MERGE_MARKER)\n dehydrated.update(marked_dict)\n except TypeError:\n pass\n\n\n__all__ = [\n \"apply_marked_keys\",\n \"dehydrate\",\n \"hydrate\",\n \"hydrate_py\",\n]\n" }, { "path": "src/pypgstac/src/pypgstac/load.py", "content": "\"\"\"Utilities to bulk load data into pgstac from json/ndjson.\"\"\"\n\nimport contextlib\nimport itertools\nimport logging\nimport re\nimport sys\nimport time\nfrom dataclasses import dataclass\nfrom datetime import UTC, datetime\nfrom enum import Enum\nfrom pathlib import Path\nfrom typing import (\n Any,\n BinaryIO,\n Generator,\n Iterable,\n Iterator,\n TextIO,\n)\n\nimport orjson\nimport psycopg\nfrom cachetools.func import lru_cache\nfrom orjson import JSONDecodeError\nfrom plpygis.geometry import Geometry\nfrom psycopg import sql\nfrom smart_open import open\nfrom tenacity import (\n retry,\n retry_if_exception_type,\n stop_after_attempt,\n wait_random_exponential,\n)\nfrom version_parser import Version as V\n\nfrom .db import PgstacDB\nfrom .hydration import dehydrate\nfrom .version import __version__\n\nlogger = logging.getLogger(__name__)\n\nMIN_DATETIME_UTC = datetime.min.replace(tzinfo=UTC)\nMAX_DATETIME_UTC = datetime.max.replace(tzinfo=UTC)\n\n\ndef _normalize_version_for_parse(version: str) -> str:\n \"\"\"Extract the numeric semver prefix for version_parser compatibility.\"\"\"\n match = re.match(r\"^(\\d+\\.\\d+\\.\\d+)\", str(version))\n if match is not None:\n return match.group(1)\n return str(version)\n\n\n@dataclass\nclass Partition:\n name: str\n collection: str\n datetime_range_min: str\n datetime_range_max: str\n end_datetime_range_min: str\n end_datetime_range_max: str\n requires_update: bool\n\n\ndef chunked_iterable(iterable: Iterable, size: int | None = 10000) -> Iterable:\n \"\"\"Chunk an iterable.\"\"\"\n it = iter(iterable)\n while True:\n chunk = tuple(itertools.islice(it, size))\n if not chunk:\n break\n yield chunk\n\n\nclass Tables(str, Enum):\n \"\"\"Available tables for loading.\"\"\"\n\n items = \"items\"\n collections = \"collections\"\n\n\nclass Methods(str, Enum):\n \"\"\"Available methods for loading data.\"\"\"\n\n insert = \"insert\"\n ignore = \"ignore\"\n upsert = \"upsert\"\n delsert = \"delsert\"\n insert_ignore = \"insert_ignore\"\n\n\n@contextlib.contextmanager\ndef open_std(\n filename: str,\n mode: str = \"r\",\n *args: Any,\n **kwargs: Any,\n) -> Generator[Any, None, None]:\n \"\"\"Open files and i/o streams transparently.\"\"\"\n fh: TextIO | BinaryIO\n if (\n filename is None\n or filename == \"-\"\n or filename == \"stdin\"\n or filename == \"stdout\"\n ):\n stream = sys.stdin if \"r\" in mode else sys.stdout\n fh = stream.buffer if \"b\" in mode else stream\n close = False\n else:\n fh = open(filename, mode, *args, **kwargs)\n close = True\n\n try:\n yield fh\n finally:\n if close:\n try:\n fh.close()\n except AttributeError:\n pass\n\n\ndef read_json(file: Path | str | Iterator[Any] = \"stdin\") -> Iterable:\n \"\"\"Load data from an ndjson or json file.\"\"\"\n if file is None:\n file = \"stdin\"\n if isinstance(file, str):\n open_file: Any = open_std(file, \"r\")\n with open_file as f:\n # Try reading line by line as ndjson\n try:\n for line in f:\n lineout = line.strip().replace(\"\\\\\\\\\", \"\\\\\").replace(\"\\\\\\\\\", \"\\\\\")\n yield orjson.loads(lineout)\n except JSONDecodeError:\n # If reading first line as json fails, try reading entire file\n logger.info(\"First line could not be parsed as json, trying full file.\")\n try:\n f.seek(0)\n json = orjson.loads(f.read())\n if isinstance(json, list):\n for record in json:\n yield record\n else:\n yield json\n except JSONDecodeError:\n logger.info(\"File cannot be read as json\")\n raise\n elif isinstance(file, Iterable):\n for line in file:\n if isinstance(line, dict):\n yield line\n elif isinstance(line, (str, bytes, bytearray, memoryview)):\n yield orjson.loads(line)\n else:\n raise TypeError(\"Unsupported json input type in iterable.\")\n\n\nclass Loader:\n \"\"\"Utilities for loading data.\"\"\"\n\n db: PgstacDB\n _partition_cache: dict[str, Partition]\n\n def __init__(self, db: PgstacDB):\n self.db = db\n self._partition_cache: dict[str, Partition] = {}\n\n def check_version(self) -> None:\n db_version = self.db.version\n if db_version is None:\n raise Exception(\"Failed to detect the target database version.\")\n\n if db_version != \"unreleased\":\n v1 = V(_normalize_version_for_parse(db_version))\n v2 = V(_normalize_version_for_parse(__version__))\n if (v1.get_major_version(), v1.get_minor_version()) != (\n v2.get_major_version(),\n v2.get_minor_version(),\n ):\n raise Exception(\n f\"pypgstac version {__version__}\"\n \" is not compatible with the target\"\n f\" database version {self.db.version}.\"\n f\" database version {db_version}.\",\n )\n\n @lru_cache(maxsize=128)\n def collection_json(self, collection_id: str) -> tuple[dict[str, Any], int, str]:\n \"\"\"Get collection.\"\"\"\n res = self.db.query_one(\n \"SELECT base_item, key, partition_trunc FROM collections WHERE id=%s\",\n (collection_id,),\n )\n if isinstance(res, tuple):\n base_item, key, partition_trunc = res\n else:\n raise Exception(f\"Error getting info for {collection_id}.\")\n if key is None:\n raise Exception(\n f\"Collection {collection_id} is not present in the database\",\n )\n logger.debug(f\"Found {collection_id} with base_item {base_item}\")\n return base_item, key, partition_trunc\n\n def load_collections(\n self,\n file: Path | str | Iterator[Any] = \"stdin\",\n insert_mode: Methods | None = Methods.insert,\n ) -> None:\n \"\"\"Load a collections json or ndjson file.\"\"\"\n self.check_version()\n\n if file is None:\n file = \"stdin\"\n conn = self.db.connect()\n with conn.cursor() as cur:\n with conn.transaction():\n cur.execute(\n \"\"\"\n DROP TABLE IF EXISTS tmp_collections;\n CREATE TEMP TABLE tmp_collections\n (content jsonb) ON COMMIT DROP;\n \"\"\",\n )\n with cur.copy(\"COPY tmp_collections (content) FROM stdin;\") as copy:\n for collection in read_json(file):\n copy.write_row((orjson.dumps(collection).decode(),))\n if insert_mode in (\n None,\n Methods.insert,\n ):\n cur.execute(\n \"\"\"\n INSERT INTO collections (content)\n SELECT content FROM tmp_collections;\n \"\"\",\n )\n logger.debug(cur.statusmessage)\n logger.debug(f\"Rows affected: {cur.rowcount}\")\n elif insert_mode in (\n Methods.insert_ignore,\n Methods.ignore,\n ):\n cur.execute(\n \"\"\"\n INSERT INTO collections (content)\n SELECT content FROM tmp_collections\n ON CONFLICT DO NOTHING;\n \"\"\",\n )\n logger.debug(cur.statusmessage)\n logger.debug(f\"Rows affected: {cur.rowcount}\")\n elif insert_mode == Methods.upsert:\n cur.execute(\n \"\"\"\n INSERT INTO collections (content)\n SELECT content FROM tmp_collections\n ON CONFLICT (id) DO\n UPDATE SET content=EXCLUDED.content;\n \"\"\",\n )\n logger.debug(cur.statusmessage)\n logger.debug(f\"Rows affected: {cur.rowcount}\")\n else:\n raise Exception(\n \"Available modes are insert, ignore, and upsert.\"\n f\"You entered {insert_mode}.\",\n )\n\n @retry(\n stop=stop_after_attempt(10),\n wait=wait_random_exponential(multiplier=1, max=120),\n retry=(\n retry_if_exception_type(psycopg.errors.CheckViolation)\n | retry_if_exception_type(psycopg.errors.DeadlockDetected)\n | retry_if_exception_type(psycopg.errors.SerializationFailure)\n | retry_if_exception_type(psycopg.errors.LockNotAvailable)\n | retry_if_exception_type(psycopg.errors.ObjectInUse)\n ),\n reraise=True,\n before_sleep=lambda retry_state: (\n setattr(\n retry_state.args[1],\n \"requires_update\",\n True,\n )\n if retry_state.outcome is not None\n and isinstance(\n retry_state.outcome.exception(),\n psycopg.errors.CheckViolation,\n )\n else None\n ),\n )\n def load_partition(\n self,\n partition: Partition,\n items: Iterable[dict[str, Any]],\n insert_mode: Methods | None = Methods.insert,\n ) -> None:\n \"\"\"Load items data for a single partition.\"\"\"\n conn = self.db.connect()\n t = time.perf_counter()\n\n logger.debug(f\"Loading data for partition: {partition}.\")\n with conn.cursor() as cur:\n if partition.requires_update:\n with conn.transaction():\n cur.execute(\n \"\"\"\n SELECT check_partition(\n %s,\n tstzrange(%s, %s, '[]'),\n tstzrange(%s, %s, '[]')\n );\n \"\"\",\n (\n partition.collection,\n partition.datetime_range_min,\n partition.datetime_range_max,\n partition.end_datetime_range_min,\n partition.end_datetime_range_max,\n ),\n )\n\n logger.debug(\n f\"Adding or updating partition {partition.name} \"\n f\"took {time.perf_counter() - t}s\",\n )\n partition.requires_update = False\n else:\n logger.debug(\n f\"Partition {partition.name} does not require an update.\",\n )\n\n with conn.transaction():\n t = time.perf_counter()\n if insert_mode in (\n None,\n Methods.insert,\n ):\n with cur.copy(\n sql.SQL(\n \"\"\"\n COPY {}\n (id, collection, datetime,\n end_datetime, geometry,\n content, private)\n FROM stdin;\n \"\"\",\n ).format(sql.Identifier(partition.name)),\n ) as copy:\n for item in items:\n item.pop(\"partition\", None)\n copy.write_row(\n (\n item[\"id\"],\n item[\"collection\"],\n item[\"datetime\"],\n item[\"end_datetime\"],\n item[\"geometry\"],\n item[\"content\"],\n item.get(\"private\", None),\n ),\n )\n logger.debug(cur.statusmessage)\n logger.debug(f\"Rows affected: {cur.rowcount}\")\n elif insert_mode in (\n Methods.insert_ignore,\n Methods.upsert,\n Methods.delsert,\n Methods.ignore,\n ):\n cur.execute(\n \"\"\"\n DROP TABLE IF EXISTS items_ingest_temp;\n CREATE TEMP TABLE items_ingest_temp\n ON COMMIT DROP AS SELECT * FROM items LIMIT 0;\n \"\"\",\n )\n with cur.copy(\n \"\"\"\n COPY items_ingest_temp\n (id, collection, datetime,\n end_datetime, geometry,\n content, private)\n FROM stdin;\n \"\"\",\n ) as copy:\n for item in items:\n item.pop(\"partition\", None)\n copy.write_row(\n (\n item[\"id\"],\n item[\"collection\"],\n item[\"datetime\"],\n item[\"end_datetime\"],\n item[\"geometry\"],\n item[\"content\"],\n item.get(\"private\", None),\n ),\n )\n logger.debug(cur.statusmessage)\n logger.debug(f\"Copied rows: {cur.rowcount}\")\n\n cur.execute(\n sql.SQL(\n \"\"\"\n LOCK TABLE ONLY {} IN EXCLUSIVE MODE;\n \"\"\",\n ).format(sql.Identifier(partition.name)),\n )\n if insert_mode in (\n Methods.ignore,\n Methods.insert_ignore,\n ):\n cur.execute(\n sql.SQL(\n \"\"\"\n INSERT INTO {}\n SELECT *\n FROM items_ingest_temp ON CONFLICT DO NOTHING;\n \"\"\",\n ).format(sql.Identifier(partition.name)),\n )\n logger.debug(cur.statusmessage)\n logger.debug(f\"Rows affected: {cur.rowcount}\")\n elif insert_mode == Methods.upsert:\n cur.execute(\n sql.SQL(\n \"\"\"\n INSERT INTO {} AS t SELECT * FROM items_ingest_temp\n ON CONFLICT (id) DO UPDATE\n SET\n datetime = EXCLUDED.datetime,\n end_datetime = EXCLUDED.end_datetime,\n geometry = EXCLUDED.geometry,\n collection = EXCLUDED.collection,\n content = EXCLUDED.content\n WHERE t IS DISTINCT FROM EXCLUDED\n ;\n \"\"\",\n ).format(sql.Identifier(partition.name)),\n )\n logger.debug(cur.statusmessage)\n logger.debug(f\"Rows affected: {cur.rowcount}\")\n elif insert_mode == Methods.delsert:\n cur.execute(\n sql.SQL(\n \"\"\"\n WITH deletes AS (\n DELETE FROM items i USING items_ingest_temp s\n WHERE\n i.id = s.id\n AND i.collection = s.collection\n )\n INSERT INTO {} AS t SELECT * FROM items_ingest_temp\n ON CONFLICT (id) DO UPDATE\n SET\n datetime = EXCLUDED.datetime,\n end_datetime = EXCLUDED.end_datetime,\n geometry = EXCLUDED.geometry,\n collection = EXCLUDED.collection,\n content = EXCLUDED.content\n WHERE t IS DISTINCT FROM EXCLUDED\n ;\n \"\"\",\n ).format(sql.Identifier(partition.name)),\n )\n logger.debug(cur.statusmessage)\n logger.debug(f\"Rows affected: {cur.rowcount}\")\n else:\n raise Exception(\n \"Available modes are insert, ignore, upsert, and delsert.\"\n f\"You entered {insert_mode}.\",\n )\n logger.debug(\"Updating Partition Stats\")\n cur.execute(\"SELECT update_partition_stats_q(%s);\", (partition.name,))\n logger.debug(cur.statusmessage)\n logger.debug(f\"Rows affected: {cur.rowcount}\")\n logger.debug(\n f\"Copying data for {partition} took {time.perf_counter() - t} seconds\",\n )\n\n def _partition_update(self, item: dict[str, Any]) -> str:\n \"\"\"Update the cached partition with the item information and return the name.\n\n This method will mark the partition as dirty if the bounds of the partition\n need to be updated based on this item.\n \"\"\"\n p = item.get(\"partition\", None)\n if p is None:\n _, key, partition_trunc = self.collection_json(item[\"collection\"])\n if partition_trunc == \"year\":\n pd = item[\"datetime\"].replace(\"-\", \"\")[:4]\n p = f\"_items_{key}_{pd}\"\n elif partition_trunc == \"month\":\n pd = item[\"datetime\"].replace(\"-\", \"\")[:6]\n p = f\"_items_{key}_{pd}\"\n else:\n p = f\"_items_{key}\"\n item[\"partition\"] = p\n\n partition_name: str = p\n\n partition: Partition | None = None\n\n if partition_name not in self._partition_cache:\n # Read the partition information from the database if it exists\n db_rows = list(\n self.db.query(\n \"\"\"\n SELECT\n nullif(lower(constraint_dtrange),'-infinity')\n as datetime_range_min,\n nullif(upper(constraint_dtrange),'infinity')\n as datetime_range_max,\n nullif(lower(constraint_edtrange),'-infinity')\n as end_datetime_range_min,\n nullif(upper(constraint_edtrange),'infinity')\n as end_datetime_range_max\n FROM partition_sys_meta WHERE partition=%s;\n \"\"\",\n [partition_name],\n ),\n )\n if db_rows:\n datetime_range_min: datetime = db_rows[0][0] or MIN_DATETIME_UTC\n datetime_range_max: datetime = db_rows[0][1] or MAX_DATETIME_UTC\n end_datetime_range_min: datetime = db_rows[0][2] or MIN_DATETIME_UTC\n end_datetime_range_max: datetime = db_rows[0][3] or MAX_DATETIME_UTC\n\n partition = Partition(\n name=partition_name,\n collection=item[\"collection\"],\n datetime_range_min=datetime_range_min.isoformat(),\n datetime_range_max=datetime_range_max.isoformat(),\n end_datetime_range_min=end_datetime_range_min.isoformat(),\n end_datetime_range_max=end_datetime_range_max.isoformat(),\n requires_update=False,\n )\n\n else:\n partition = self._partition_cache[partition_name]\n\n if partition:\n # Only update the partition if the item is outside the current bounds\n if item[\"datetime\"] < partition.datetime_range_min:\n partition.datetime_range_min = item[\"datetime\"]\n partition.requires_update = True\n if item[\"datetime\"] > partition.datetime_range_max:\n partition.datetime_range_max = item[\"datetime\"]\n partition.requires_update = True\n if item[\"end_datetime\"] < partition.end_datetime_range_min:\n partition.end_datetime_range_min = item[\"end_datetime\"]\n partition.requires_update = True\n if item[\"end_datetime\"] > partition.end_datetime_range_max:\n partition.end_datetime_range_max = item[\"end_datetime\"]\n partition.requires_update = True\n else:\n # No partition exists yet; create a new one from item\n partition = Partition(\n name=partition_name,\n collection=item[\"collection\"],\n datetime_range_min=item[\"datetime\"],\n datetime_range_max=item[\"datetime\"],\n end_datetime_range_min=item[\"end_datetime\"],\n end_datetime_range_max=item[\"end_datetime\"],\n requires_update=True,\n )\n\n self._partition_cache[partition_name] = partition\n\n return partition_name\n\n def read_dehydrated(self, file: Path | str = \"stdin\") -> Generator:\n if file is None:\n file = \"stdin\"\n if isinstance(file, str):\n open_file: Any = open_std(file, \"r\")\n with open_file as f:\n # Note: if 'content' is changed to be anything\n # but the last field, the logic below will break.\n fields = [\n \"id\",\n \"geometry\",\n \"collection\",\n \"datetime\",\n \"end_datetime\",\n \"content\",\n ]\n\n for line in f:\n tab_split = line.split(\"\\t\")\n item = {}\n for i, field in enumerate(fields):\n if field == \"content\":\n # Join the remaining splits in case\n # there were any tabs in the JSON content.\n content_value = \"\\t\".join(tab_split[i:])\n # Replace quote characters that can be\n # written on export and causes failures.\n content_value = content_value.replace(r'\\\\\"', r\"\\\"\")\n item[field] = content_value\n else:\n item[field] = tab_split[i]\n item[\"partition\"] = self._partition_update(item)\n yield item\n\n def read_hydrated(\n self,\n file: Path | str | Iterator[Any] = \"stdin\",\n ) -> Generator:\n for line in read_json(file):\n item = self.format_item(line)\n item[\"partition\"] = self._partition_update(item)\n yield item\n\n def load_items(\n self,\n file: Path | str | Iterator[Any] = \"stdin\",\n insert_mode: Methods | None = Methods.insert,\n dehydrated: bool | None = False,\n chunksize: int | None = 10000,\n ) -> None:\n \"\"\"Load items json records.\"\"\"\n self.check_version()\n\n if file is None:\n file = \"stdin\"\n t = time.perf_counter()\n self._partition_cache = {}\n\n if dehydrated and isinstance(file, str):\n items = self.read_dehydrated(file)\n else:\n items = self.read_hydrated(file)\n\n for chunkin in chunked_iterable(items, chunksize):\n chunk = list(chunkin)\n chunk.sort(key=lambda x: x[\"partition\"])\n for k, g in itertools.groupby(chunk, lambda x: x[\"partition\"]):\n self.load_partition(self._partition_cache[k], list(g), insert_mode)\n\n logger.debug(f\"Adding data to database took {time.perf_counter() - t} seconds.\")\n\n def format_item(self, _item: Path | str | dict[str, Any]) -> dict[str, Any]:\n \"\"\"Format an item to insert into a record.\"\"\"\n out: dict[str, Any] = {}\n item: dict[str, Any]\n if not isinstance(_item, dict):\n try:\n item = orjson.loads(str(_item).replace(\"\\\\\\\\\", \"\\\\\"))\n except Exception:\n raise\n else:\n item = _item\n\n base_item, key, partition_trunc = self.collection_json(item[\"collection\"])\n\n out[\"id\"] = item.get(\"id\")\n out[\"collection\"] = item.get(\"collection\")\n properties: dict[str, Any] = item.get(\"properties\", {})\n\n dt: str | None = properties.get(\"datetime\")\n edt: str | None = properties.get(\"end_datetime\")\n sdt: str | None = properties.get(\"start_datetime\")\n\n if edt is not None and sdt is not None:\n out[\"datetime\"] = sdt\n out[\"end_datetime\"] = edt\n elif dt is not None:\n out[\"datetime\"] = dt\n out[\"end_datetime\"] = dt\n else:\n raise Exception(\"Invalid datetime encountered\")\n\n if out[\"datetime\"] is None or out[\"end_datetime\"] is None:\n raise Exception(\n f\"Datetime must be set. OUT: {out} Properties: {properties}\",\n )\n\n if partition_trunc == \"year\":\n pd = out[\"datetime\"].replace(\"-\", \"\")[:4]\n partition = f\"_items_{key}_{pd}\"\n elif partition_trunc == \"month\":\n pd = out[\"datetime\"].replace(\"-\", \"\")[:6]\n partition = f\"_items_{key}_{pd}\"\n else:\n partition = f\"_items_{key}\"\n\n out[\"partition\"] = partition\n\n geojson = item.get(\"geometry\")\n if geojson is None:\n geometry = None\n else:\n geom = Geometry.from_geojson(geojson)\n if geom is None:\n raise Exception(f\"Invalid geometry encountered: {geojson}\")\n geometry = str(geom.ewkb)\n out[\"geometry\"] = geometry\n\n content = dehydrate(base_item, item)\n\n # Remove keys from the dehydrated item content which are stored directly\n # on the table row.\n content.pop(\"id\", None)\n content.pop(\"collection\", None)\n content.pop(\"geometry\", None)\n\n if (private := content.pop(\"private\", None)) is not None:\n out[\"private\"] = orjson.dumps(private).decode()\n else:\n out[\"private\"] = None\n\n out[\"content\"] = orjson.dumps(content).decode()\n\n return out\n\n def __hash__(self) -> int:\n \"\"\"Return hash so that the LRU deocrator can cache without the class.\"\"\"\n return 0\n" }, { "path": "src/pypgstac/src/pypgstac/migrate.py", "content": "\"\"\"Utilities to help migrate pgstac schema.\"\"\"\n\nimport glob\nimport logging\nimport os\nimport re\nfrom collections import defaultdict\nfrom collections.abc import Iterator\nfrom typing import Any, cast\n\nfrom smart_open import open\n\nfrom . import __version__\nfrom .db import PgstacDB\n\ndirname = os.path.dirname(__file__)\nmigrations_dir = os.path.join(dirname, \"migrations\")\n\nlogger = logging.getLogger(__name__)\n\n\nclass MigrationPath:\n \"\"\"Calculate path from migration files to get from one version to the next.\"\"\"\n\n def __init__(self, path: str, f: str, t: str) -> None:\n \"\"\"Initialize MigrationPath.\"\"\"\n self.path = path\n if f is None:\n f = \"init\"\n if t is None:\n raise Exception('Must set \"to\" version')\n if f == t:\n raise Exception(\"No Migration Necessary\")\n\n self.f = f\n self.t = t\n\n def parse_filename(self, filename: str) -> list[str]:\n \"\"\"Get version numbers from filename.\"\"\"\n filename = os.path.splitext(os.path.basename(filename))[0].replace(\n \"pgstac.\",\n \"\",\n )\n return filename.split(\"-\")\n\n def get_files(self) -> Iterator[str]:\n \"\"\"Find all migration files available.\"\"\"\n path = self.path.rstrip(\"/\")\n return glob.iglob(f\"{path}/*.sql\")\n\n def build_graph(self) -> dict[str, list[str]]:\n \"\"\"Build a graph to get from one version to another.\"\"\"\n graph = defaultdict(list)\n for file in self.get_files():\n parts = self.parse_filename(file)\n if len(parts) == 2:\n graph[parts[0]].append(parts[1])\n else:\n graph[\"init\"].append(parts[0])\n return graph\n\n def build_path(self) -> list[str] | None:\n \"\"\"Create the path of ordered files needed to migrate.\"\"\"\n graph = self.build_graph()\n explored: list[str] = []\n q = [[self.f]]\n\n while q:\n path = q.pop(0)\n node = path[-1]\n if node not in explored:\n neighbours = graph[node]\n for neighbour in neighbours:\n new_path = list(path)\n new_path.append(neighbour)\n q.append(new_path)\n if neighbour == self.t:\n return new_path\n explored.append(node)\n return None\n\n def migrations(self) -> list[str]:\n \"\"\"Return the list of migrations needed in order.\"\"\"\n path = self.build_path()\n if path is None:\n raise Exception(\n f\"Could not determine path to get from {self.f} to {self.t}.\",\n )\n if len(path) == 1:\n return [f\"pgstac.{path[0]}.sql\"]\n files = []\n for idx in range(len(path) - 1):\n f = f\"pgstac.{path[idx]}-{path[idx + 1]}.sql\"\n f = f.replace(\"--init\", \"\")\n files.append(f\"pgstac.{path[idx]}-{path[idx + 1]}.sql\")\n return files\n\n\ndef get_sql(file: str) -> str:\n \"\"\"Get sql from a file as a string.\"\"\"\n sqlstrs = []\n file = re.sub(\"[0-9]+[.][0-9]+[.][0-9]+-dev\", \"unreleased\", file)\n fp = os.path.join(migrations_dir, file)\n file_handle: Any = open(fp)\n\n with file_handle as fd:\n sqlstrs.extend(fd.readlines())\n return \"\\n\".join(sqlstrs)\n\n\nclass Migrate:\n \"\"\"Utilities for migrating pgstac database.\"\"\"\n\n def __init__(self, db: PgstacDB, schema: str = \"pgstac\"):\n \"\"\"Prepare for migration.\"\"\"\n self.db = db\n self.schema = schema\n\n def run_migration(self, toversion: str | None = None) -> str:\n \"\"\"Migrate a pgstac database to current version.\"\"\"\n if toversion is None:\n toversion = __version__\n files = []\n if re.search(r\"-dev$\", toversion):\n logger.info(\"using unreleased version\")\n toversion = \"unreleased\"\n\n major, minor, patch = tuple(\n map(\n int,\n [\n self.db.pg_version[i : i + 2]\n for i in range(0, len(self.db.pg_version), 2)\n ],\n ),\n )\n logger.info(f\"Migrating PgSTAC on PostgreSQL Version {major}.{minor}.{patch}\")\n oldversion = self.db.version\n if oldversion == toversion:\n logger.info(f\"Target database already at version: {toversion}\")\n return toversion\n if oldversion is None:\n logger.info(f\"No pgstac version set, installing {toversion} from scratch.\")\n files.append(os.path.join(migrations_dir, f\"pgstac.{toversion}.sql\"))\n else:\n logger.info(f\"Migrating from {oldversion} to {toversion}.\")\n m = MigrationPath(migrations_dir, oldversion, toversion)\n files = m.migrations()\n\n if len(files) < 1:\n raise Exception(\"Could not find migration files\")\n\n conn = self.db.connect()\n\n with conn.cursor() as cur:\n conn.autocommit = False\n for file in files:\n logger.debug(f\"Running migration file {file}.\")\n migration_sql = get_sql(file)\n # Migration SQL is loaded from trusted local migration files.\n cur.execute(cast(Any, migration_sql))\n logger.debug(cur.statusmessage)\n logger.debug(cur.rowcount)\n\n logger.debug(f\"Database migrated to {toversion}\")\n\n newversion = self.db.version\n if conn is not None:\n if newversion == toversion:\n conn.commit()\n else:\n conn.rollback()\n raise Exception(\n \"Migration failed, database rolled back to previous state.\",\n )\n\n logger.debug(f\"New Version: {newversion}\")\n if newversion is None:\n raise Exception(\"Migration failed to report a new version.\")\n return newversion\n" }, { "path": "src/pypgstac/src/pypgstac/py.typed", "content": "" }, { "path": "src/pypgstac/src/pypgstac/pypgstac.py", "content": "\"\"\"Command utilities for managing pgstac.\"\"\"\n\nimport logging\nimport sys\n\nimport fire\nimport orjson\nfrom smart_open import open\n\nfrom pypgstac.db import PgstacDB\nfrom pypgstac.load import Loader, Methods, Tables, read_json\nfrom pypgstac.migrate import Migrate\n\n\nclass PgstacCLI:\n \"\"\"CLI for PgSTAC.\"\"\"\n\n def __init__(\n self,\n dsn: str | None = \"\",\n version: bool = False,\n debug: bool = False,\n usequeue: bool = False,\n ):\n \"\"\"Initialize PgSTAC CLI.\"\"\"\n if version:\n sys.exit(0)\n\n self.dsn = dsn\n self._db = PgstacDB(dsn=dsn, debug=debug, use_queue=usequeue)\n if debug:\n logging.basicConfig(level=logging.DEBUG)\n sys.tracebacklimit = 1000\n\n @property\n def initversion(self) -> str:\n \"\"\"Return earliest migration version.\"\"\"\n return \"0.1.9\"\n\n @property\n def version(self) -> str | None:\n \"\"\"Get PgSTAC version installed on database.\"\"\"\n return self._db.version\n\n @property\n def pg_version(self) -> str:\n \"\"\"Get PostgreSQL server version installed on database.\"\"\"\n return self._db.pg_version\n\n def pgready(self) -> None:\n \"\"\"Wait for a pgstac database to accept connections.\"\"\"\n self._db.wait()\n\n def search(self, query: str) -> str:\n \"\"\"Search PgSTAC.\"\"\"\n return self._db.search(query)\n\n def migrate(self, toversion: str | None = None) -> str:\n \"\"\"Migrate PgSTAC Database.\"\"\"\n migrator = Migrate(self._db)\n return migrator.run_migration(toversion=toversion)\n\n def load(\n self,\n table: Tables,\n file: str,\n method: Methods | None = Methods.insert,\n dehydrated: bool | None = False,\n chunksize: int | None = 10000,\n ) -> None:\n \"\"\"Load collections or items into PgSTAC.\"\"\"\n loader = Loader(db=self._db)\n if table == \"collections\":\n loader.load_collections(file, method)\n if table == \"items\":\n loader.load_items(file, method, dehydrated, chunksize)\n\n def runqueue(self) -> str:\n return self._db.run_queued()\n\n def loadextensions(self) -> None:\n conn = self._db.connect()\n\n with conn.cursor() as cur:\n cur.execute(\n \"\"\"\n INSERT INTO stac_extensions (url)\n SELECT DISTINCT\n substring(\n jsonb_array_elements_text(content->'stac_extensions') FROM E'^[^#]*'\n )\n FROM collections\n ON CONFLICT DO NOTHING;\n \"\"\",\n )\n conn.commit()\n\n urls = self._db.query(\n \"\"\"\n SELECT url FROM stac_extensions WHERE content IS NULL;\n \"\"\",\n )\n if urls:\n for u in urls:\n url = u[0]\n try:\n with open(url, \"r\") as f:\n content = f.read()\n self._db.query(\n \"\"\"\n UPDATE pgstac.stac_extensions\n SET content=%s\n WHERE url=%s\n ;\n \"\"\",\n [content, url],\n )\n conn.commit()\n except Exception:\n pass\n\n def load_queryables(\n self,\n file: str,\n collection_ids: list[str] | None = None,\n delete_missing: bool | None = False,\n index_fields: list[str] | None = None,\n ) -> None:\n \"\"\"Load queryables from a JSON file.\n\n Args:\n file: Path to the JSON file containing queryables definition\n collection_ids: Comma-separated list of collection IDs to apply the\n queryables to\n delete_missing: If True, delete properties not present in the file.\n If collection_ids is specified, only delete properties\n for those collections.\n index_fields: List of field names to create indexes for. If not provided,\n no indexes will be created. Creating too many indexes can\n negatively impact performance.\n \"\"\"\n\n # Read the queryables JSON file\n queryables_data = None\n for item in read_json(file):\n queryables_data = item\n break # We only need the first item\n\n if not queryables_data:\n raise ValueError(f\"No valid JSON data found in {file}\")\n\n # Extract properties from the queryables definition\n properties = queryables_data.get(\"properties\", {})\n if not properties:\n raise ValueError(\"No properties found in queryables definition\")\n\n conn = self._db.connect()\n with conn.cursor() as cur:\n with conn.transaction():\n # Insert each property as a queryable\n for name, definition in properties.items():\n # Skip core fields that are already indexed\n if name in (\n \"id\",\n \"geometry\",\n \"datetime\",\n \"end_datetime\",\n \"collection\",\n ):\n continue\n\n # Determine property wrapper based on type\n property_wrapper = \"to_text\" # default\n if definition.get(\"type\") == \"number\":\n property_wrapper = \"to_float\"\n elif definition.get(\"type\") == \"integer\":\n property_wrapper = \"to_int\"\n elif definition.get(\"format\") == \"date-time\":\n property_wrapper = \"to_tstz\"\n elif definition.get(\"type\") == \"array\":\n property_wrapper = \"to_text_array\"\n\n # Determine if this field should be indexed\n property_index_type = None\n if index_fields and name in index_fields:\n property_index_type = \"BTREE\"\n\n # First delete any existing queryable with the same name\n if not collection_ids:\n # If no collection_ids specified, delete queryables\n # with NULL collection_ids\n cur.execute(\n \"\"\"\n DELETE FROM queryables\n WHERE name = %s AND collection_ids IS NULL\n \"\"\",\n [name],\n )\n else:\n # Delete queryables with matching name and collection_ids\n cur.execute(\n \"\"\"\n DELETE FROM queryables\n WHERE name = %s AND collection_ids = %s::text[]\n \"\"\",\n [name, collection_ids],\n )\n\n # Also delete queryables with NULL collection_ids\n cur.execute(\n \"\"\"\n DELETE FROM queryables\n WHERE name = %s AND collection_ids IS NULL\n \"\"\",\n [name],\n )\n\n # Then insert the new queryable\n cur.execute(\n \"\"\"\n INSERT INTO queryables\n (name, collection_ids, definition, property_wrapper,\n property_index_type)\n VALUES (%s, %s, %s, %s, %s)\n \"\"\",\n [\n name,\n collection_ids,\n orjson.dumps(definition).decode(),\n property_wrapper,\n property_index_type,\n ],\n )\n\n # If delete_missing is True,\n # delete all queryables that were not in the file\n if delete_missing:\n # Get the list of property names from the file\n property_names = list(properties.keys())\n\n # Skip core fields that are already indexed\n core_fields = [\n \"id\",\n \"geometry\",\n \"datetime\",\n \"end_datetime\",\n \"collection\",\n ]\n property_names = [\n name for name in property_names if name not in core_fields\n ]\n\n if not property_names:\n # If no valid properties, don't delete anything\n pass\n elif not collection_ids:\n # If no collection_ids specified,\n # delete queryables with NULL collection_ids\n # that are not in the property_names list\n placeholders = \", \".join([\"%s\"] * len(property_names))\n core_placeholders = \", \".join([\"%s\"] * len(core_fields))\n\n # Build the query with proper placeholders\n query = f\"\"\"\n DELETE FROM queryables\n WHERE collection_ids IS NULL\n AND name NOT IN ({placeholders})\n AND name NOT IN ({core_placeholders})\n \"\"\"\n\n # Flatten the parameters\n params = property_names + core_fields\n\n cur.execute(query, params)\n else:\n # Delete queryables with matching collection_ids\n # that are not in the property_names list\n placeholders = \", \".join([\"%s\"] * len(property_names))\n core_placeholders = \", \".join([\"%s\"] * len(core_fields))\n\n # Build the query with proper placeholders\n query = f\"\"\"\n DELETE FROM queryables\n WHERE collection_ids = %s::text[]\n AND name NOT IN ({placeholders})\n AND name NOT IN ({core_placeholders})\n \"\"\"\n\n # Flatten the parameters\n params = [collection_ids] + property_names + core_fields\n\n cur.execute(query, params)\n\n # Trigger index creation only if index_fields were provided\n if index_fields and len(index_fields) > 0:\n cur.execute(\"SELECT maintain_partitions();\")\n\n\ndef cli() -> None:\n \"\"\"Wrap fire call for CLI.\"\"\"\n fire.Fire(PgstacCLI)\n\n\nif __name__ == \"__main__\":\n fire.Fire(PgstacCLI)\n" }, { "path": "src/pypgstac/src/pypgstac/version.py", "content": "\"\"\"Version.\"\"\"\n\n__version__ = \"0.9.11-dev\"\n" }, { "path": "src/pypgstac/tests/__init__.py", "content": "" }, { "path": "src/pypgstac/tests/conftest.py", "content": "\"\"\"Fixtures for pypgstac tests.\"\"\"\n\nimport os\nfrom typing import Generator\n\nimport psycopg\nimport pytest\n\nfrom pypgstac.db import PgstacDB\nfrom pypgstac.load import Loader\nfrom pypgstac.migrate import Migrate\n\n\n@pytest.fixture(scope=\"function\")\ndef db() -> Generator:\n \"\"\"Fixture to get a fresh database.\"\"\"\n origdb: str = os.getenv(\"PGDATABASE\", \"\")\n\n with psycopg.connect(autocommit=True) as conn:\n try:\n conn.execute(\n \"\"\"\n CREATE DATABASE pypgstactestdb\n TEMPLATE pgstac_test_db_template;\n \"\"\",\n )\n except psycopg.errors.DuplicateDatabase:\n try:\n conn.execute(\n \"\"\"\n DROP DATABASE pypgstactestdb WITH (FORCE);\n \"\"\",\n )\n conn.execute(\n \"\"\"\n CREATE DATABASE pypgstactestdb\n TEMPLATE pgstac_test_db_template;\n \"\"\",\n )\n except psycopg.errors.InsufficientPrivilege:\n try:\n conn.execute(\"DROP DATABASE pypgstactestdb;\")\n conn.execute(\n \"\"\"\n CREATE DATABASE pypgstactestdb\n TEMPLATE pgstac_test_db_template;\n \"\"\",\n )\n except Exception:\n pass\n\n os.environ[\"PGDATABASE\"] = \"pypgstactestdb\"\n\n pgdb = PgstacDB()\n\n yield pgdb\n\n pgdb.close()\n os.environ[\"PGDATABASE\"] = origdb\n\n with psycopg.connect(autocommit=True) as conn:\n try:\n conn.execute(\"DROP DATABASE pypgstactestdb WITH (FORCE);\")\n except psycopg.errors.InsufficientPrivilege:\n try:\n conn.execute(\"DROP DATABASE pypgstactestdb;\")\n except Exception:\n pass\n\n\n@pytest.fixture(scope=\"function\")\ndef loader(db: PgstacDB) -> Loader:\n \"\"\"Fixture to get a loader and an empty pgstac.\"\"\"\n if False:\n db.query(\"DROP SCHEMA IF EXISTS pgstac CASCADE;\")\n Migrate(db).run_migration()\n ldr = Loader(db)\n return ldr\n" }, { "path": "src/pypgstac/tests/data-files/hydration/collections/chloris-biomass.json", "content": "{\n \"id\": \"chloris-biomass\",\n \"type\": \"Collection\",\n \"links\": [\n {\n \"rel\": \"items\",\n \"type\": \"application/geo+json\",\n \"href\": \"https://planetarycomputer.microsoft.com/api/stac/v1/collections/chloris-biomass/items\"\n },\n {\n \"rel\": \"parent\",\n \"type\": \"application/json\",\n \"href\": \"https://planetarycomputer.microsoft.com/api/stac/v1/\"\n },\n {\n \"rel\": \"root\",\n \"type\": \"application/json\",\n \"href\": \"https://planetarycomputer.microsoft.com/api/stac/v1/\"\n },\n {\n \"rel\": \"self\",\n \"type\": \"application/json\",\n \"href\": \"https://planetarycomputer.microsoft.com/api/stac/v1/collections/chloris-biomass\"\n },\n {\n \"rel\": \"license\",\n \"href\": \"https://spdx.org/licenses/CC-BY-NC-SA-4.0.html\",\n \"title\": \"Creative Commons Attribution Non Commercial Share Alike 4.0 International\"\n },\n {\n \"rel\": \"describedby\",\n \"href\": \"https://planetarycomputer.microsoft.com/dataset/chloris-biomass\",\n \"title\": \"Human readable dataset overview and reference\",\n \"type\": \"text/html\"\n }\n ],\n \"title\": \"Chloris Biomass\",\n \"assets\": {\n \"thumbnail\": {\n \"href\": \"https://ai4edatasetspublicassets.blob.core.windows.net/assets/pc_thumbnails/chloris-biomass.jpg\",\n \"type\": \"image/jpg\",\n \"roles\": [\n \"thumbnail\"\n ],\n \"title\": \"Chloris Biomass\"\n }\n },\n \"extent\": {\n \"spatial\": {\n \"bbox\": [\n [\n -179.95,\n -60,\n 179.95,\n 90\n ]\n ]\n },\n \"temporal\": {\n \"interval\": [\n [\n \"2003-07-31T00:00:00Z\",\n \"2019-07-31T00:00:00Z\"\n ]\n ]\n }\n },\n \"license\": \"CC-BY-NC-SA-4.0\",\n \"keywords\": [\n \"Chloris\",\n \"Biomass\",\n \"MODIS\",\n \"Carbon\"\n ],\n \"providers\": [\n {\n \"url\": \"http://chloris.earth/\",\n \"name\": \"Chloris\",\n \"roles\": [\n \"producer\",\n \"licensor\"\n ]\n },\n {\n \"url\": \"https://planetarycomputer.microsoft.com\",\n \"name\": \"Microsoft\",\n \"roles\": [\n \"host\",\n \"processor\"\n ]\n }\n ],\n \"summaries\": {\n \"gsd\": [\n 4633\n ]\n },\n \"description\": \"The Chloris Global Biomass 2003 - 2019 dataset provides estimates of stock and change in aboveground biomass for Earth's terrestrial woody vegetation ecosystems. It covers the period 2003 - 2019, at annual time steps. The global dataset has a circa 4.6 km spatial resolution.\\n\\nThe maps and data sets were generated by combining multiple remote sensing measurements from space borne satellites, processed using state-of-the-art machine learning and statistical methods, validated with field data from multiple countries. The dataset provides direct estimates of aboveground stock and change, and are not based on land use or land cover area change, and as such they include gains and losses of carbon stock in all types of woody vegetation - whether natural or plantations.\\n\\nAnnual stocks are expressed in units of tons of biomass. Annual changes in stocks are expressed in units of CO2 equivalent, i.e., the amount of CO2 released from or taken up by terrestrial ecosystems for that specific pixel.\\n\\nThe spatial data sets are available on [Microsoft’s Planetary Computer](https://planetarycomputer.microsoft.com/dataset/chloris-biomass) under a Creative Common license of the type Attribution-Non Commercial-Share Alike [CC BY-NC-SA](https://spdx.org/licenses/CC-BY-NC-SA-4.0.html).\\n\\n[Chloris Geospatial](https://chloris.earth/) is a mission-driven technology company that develops software and data products on the state of natural capital for use by business, governments, and the social sector.\\n\",\n \"item_assets\": {\n \"biomass\": {\n \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\",\n \"roles\": [\n \"data\"\n ],\n \"title\": \"Annual estimates of aboveground woody biomass.\",\n \"raster:bands\": [\n {\n \"unit\": \"tonnes\",\n \"nodata\": 2147483647,\n \"data_type\": \"uint32\"\n }\n ]\n },\n \"biomass_wm\": {\n \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\",\n \"roles\": [\n \"data\"\n ],\n \"title\": \"Annual estimates of aboveground woody biomass (Web Mercator).\",\n \"raster:bands\": [\n {\n \"unit\": \"tonnes\",\n \"nodata\": 2147483647,\n \"data_type\": \"uint32\"\n }\n ]\n },\n \"biomass_change\": {\n \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\",\n \"roles\": [\n \"data\"\n ],\n \"title\": \"Annual estimates of changes (gains and losses) in aboveground woody biomass from the previous year.\",\n \"raster:bands\": [\n {\n \"unit\": \"tonnes\",\n \"nodata\": -32768,\n \"data_type\": \"int16\"\n }\n ]\n },\n \"biomass_change_wm\": {\n \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\",\n \"roles\": [\n \"data\"\n ],\n \"title\": \"Annual estimates of changes (gains and losses) in aboveground woody biomass from the previous year (Web Mercator).\",\n \"raster:bands\": [\n {\n \"unit\": \"tonnes\",\n \"nodata\": -32768,\n \"data_type\": \"int16\"\n }\n ]\n }\n },\n \"stac_version\": \"1.0.0\",\n \"msft:container\": \"chloris-biomass\",\n \"stac_extensions\": [\n \"https://stac-extensions.github.io/projection/v1.0.0/schema.json\",\n \"https://stac-extensions.github.io/raster/v1.1.0/schema.json\"\n ],\n \"msft:storage_account\": \"ai4edataeuwest\",\n \"msft:short_description\": \"The Chloris Global Biomass 2003 - 2019 dataset provides estimates of stock and change in aboveground biomass for Earth's terrestrial woody vegetation ecosystems during the period 2003 - 2019, at annual time steps. The global dataset has a circa 4.6 km spatial resolution.\"\n}" }, { "path": "src/pypgstac/tests/data-files/hydration/collections/landsat-c2-l1.json", "content": "{\n \"id\": \"landsat-c2-l1\",\n \"type\": \"Collection\",\n \"links\": [\n {\n \"rel\": \"items\",\n \"type\": \"application/geo+json\",\n \"href\": \"https://pct-apis-staging.westeurope.cloudapp.azure.com/stac/collections/landsat-c2-l1/items\"\n },\n {\n \"rel\": \"parent\",\n \"type\": \"application/json\",\n \"href\": \"https://pct-apis-staging.westeurope.cloudapp.azure.com/stac/\"\n },\n {\n \"rel\": \"root\",\n \"type\": \"application/json\",\n \"href\": \"https://pct-apis-staging.westeurope.cloudapp.azure.com/stac/\"\n },\n {\n \"rel\": \"self\",\n \"type\": \"application/json\",\n \"href\": \"https://pct-apis-staging.westeurope.cloudapp.azure.com/stac/collections/landsat-c2-l1\"\n },\n {\n \"rel\": \"cite-as\",\n \"href\": \"https://doi.org/10.5066/P9AF14YV\",\n \"title\": \"Landsat 1-5 MSS Collection 2 Level-1\"\n },\n {\n \"rel\": \"license\",\n \"href\": \"https://www.usgs.gov/core-science-systems/hdds/data-policy\",\n \"title\": \"Public Domain\"\n },\n {\n \"rel\": \"describedby\",\n \"href\": \"https://planetarycomputer.microsoft.com/dataset/landsat-c2-l1\",\n \"title\": \"Human readable dataset overview and reference\",\n \"type\": \"text/html\"\n }\n ],\n \"title\": \"Landsat Collection 2 Level-1\",\n \"assets\": {\n \"thumbnail\": {\n \"href\": \"https://ai4edatasetspublicassets.blob.core.windows.net/assets/pc_thumbnails/landsat-c2-l1-thumb.png\",\n \"type\": \"image/png\",\n \"roles\": [\"thumbnail\"],\n \"title\": \"Landsat Collection 2 Level-1 thumbnail\"\n }\n },\n \"extent\": {\n \"spatial\": {\n \"bbox\": [[-180, -90, 180, 90]]\n },\n \"temporal\": {\n \"interval\": [[\"1972-07-25T00:00:00Z\", \"2013-01-07T23:23:59Z\"]]\n }\n },\n \"license\": \"proprietary\",\n \"keywords\": [\"Landsat\", \"USGS\", \"NASA\", \"Satellite\", \"Global\", \"Imagery\"],\n \"providers\": [\n {\n \"url\": \"https://landsat.gsfc.nasa.gov/\",\n \"name\": \"NASA\",\n \"roles\": [\"producer\", \"licensor\"]\n },\n {\n \"url\": \"https://www.usgs.gov/landsat-missions/landsat-collection-2-level-1-data\",\n \"name\": \"USGS\",\n \"roles\": [\"producer\", \"processor\", \"licensor\"]\n },\n {\n \"url\": \"https://planetarycomputer.microsoft.com\",\n \"name\": \"Microsoft\",\n \"roles\": [\"host\"]\n }\n ],\n \"summaries\": {\n \"gsd\": [79],\n \"sci:doi\": [\"10.5066/P9AF14YV\"],\n \"eo:bands\": [\n {\n \"name\": \"B4\",\n \"common_name\": \"green\",\n \"description\": \"Visible green (Landsat 1-3 Band B4)\",\n \"center_wavelength\": 0.55,\n \"full_width_half_max\": 0.1\n },\n {\n \"name\": \"B5\",\n \"common_name\": \"red\",\n \"description\": \"Visible red (Landsat 1-3 Band B5)\",\n \"center_wavelength\": 0.65,\n \"full_width_half_max\": 0.1\n },\n {\n \"name\": \"B6\",\n \"common_name\": \"nir08\",\n \"description\": \"Near infrared (Landsat 1-3 Band B6)\",\n \"center_wavelength\": 0.75,\n \"full_width_half_max\": 0.1\n },\n {\n \"name\": \"B7\",\n \"common_name\": \"nir09\",\n \"description\": \"Near infrared (Landsat 1-3 Band B7)\",\n \"center_wavelength\": 0.95,\n \"full_width_half_max\": 0.3\n },\n {\n \"name\": \"B1\",\n \"common_name\": \"green\",\n \"description\": \"Visible green (Landsat 4-5 Band B1)\",\n \"center_wavelength\": 0.55,\n \"full_width_half_max\": 0.1\n },\n {\n \"name\": \"B2\",\n \"common_name\": \"red\",\n \"description\": \"Visible red (Landsat 4-5 Band B2)\",\n \"center_wavelength\": 0.65,\n \"full_width_half_max\": 0.1\n },\n {\n \"name\": \"B3\",\n \"common_name\": \"nir08\",\n \"description\": \"Near infrared (Landsat 4-5 Band B3)\",\n \"center_wavelength\": 0.75,\n \"full_width_half_max\": 0.1\n },\n {\n \"name\": \"B4\",\n \"common_name\": \"nir09\",\n \"description\": \"Near infrared (Landsat 4-5 Band B4)\",\n \"center_wavelength\": 0.95,\n \"full_width_half_max\": 0.3\n }\n ],\n \"platform\": [\n \"landsat-1\",\n \"landsat-2\",\n \"landsat-3\",\n \"landsat-4\",\n \"landsat-5\"\n ],\n \"instruments\": [\"mss\"],\n \"view:off_nadir\": [0]\n },\n \"description\": \"Landsat Collection 2 Level-1 data, consisting of quantized and calibrated scaled Digital Numbers (DN) representing the multispectral image data. These [Level-1](https://www.usgs.gov/landsat-missions/landsat-collection-2-level-1-data) data can be [rescaled](https://www.usgs.gov/landsat-missions/using-usgs-landsat-level-1-data-product) to top of atmosphere (TOA) reflectance and/or radiance. Thermal band data can be rescaled to TOA brightness temperature.\\\\n\\\\nThis dataset represents the global archive of Level-1 data from [Landsat Collection 2](https://www.usgs.gov/core-science-systems/nli/landsat/landsat-collection-2) acquired by the [Multispectral Scanner System](https://landsat.gsfc.nasa.gov/multispectral-scanner-system/) onboard Landsat 1 through Landsat 5 from July 7, 1972 to January 7, 2013. Images are stored in [cloud-optimized GeoTIFF](https://www.cogeo.org/) format.\\\\n\",\n \"item_assets\": {\n \"red\": {\n \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\",\n \"roles\": [\"data\"],\n \"title\": \"Red Band\",\n \"description\": \"Collection 2 Level-1 Red Band Top of Atmosphere Radiance\",\n \"raster:bands\": [\n {\n \"unit\": \"watt/steradian/square_meter/micrometer\",\n \"nodata\": 0,\n \"data_type\": \"uint8\",\n \"spatial_resolution\": 60\n }\n ]\n },\n \"green\": {\n \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\",\n \"roles\": [\"data\"],\n \"title\": \"Green Band\",\n \"description\": \"Collection 2 Level-1 Green Band Top of Atmosphere Radiance\",\n \"raster:bands\": [\n {\n \"unit\": \"watt/steradian/square_meter/micrometer\",\n \"nodata\": 0,\n \"data_type\": \"uint8\",\n \"spatial_resolution\": 60\n }\n ]\n },\n \"nir08\": {\n \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\",\n \"roles\": [\"data\"],\n \"title\": \"Near Infrared Band 0.8\",\n \"description\": \"Collection 2 Level-1 Near Infrared Band 0.8 Top of Atmosphere Radiance\",\n \"raster:bands\": [\n {\n \"unit\": \"watt/steradian/square_meter/micrometer\",\n \"nodata\": 0,\n \"data_type\": \"uint8\",\n \"spatial_resolution\": 60\n }\n ]\n },\n \"nir09\": {\n \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\",\n \"roles\": [\"data\"],\n \"title\": \"Near Infrared Band 0.9\",\n \"description\": \"Collection 2 Level-1 Near Infrared Band 0.9 Top of Atmosphere Radiance\",\n \"raster:bands\": [\n {\n \"unit\": \"watt/steradian/square_meter/micrometer\",\n \"nodata\": 0,\n \"data_type\": \"uint8\",\n \"spatial_resolution\": 60\n }\n ]\n },\n \"mtl.txt\": {\n \"type\": \"text/plain\",\n \"roles\": [\"metadata\"],\n \"title\": \"Product Metadata File (txt)\",\n \"description\": \"Collection 2 Level-1 Product Metadata File (txt)\"\n },\n \"mtl.xml\": {\n \"type\": \"application/xml\",\n \"roles\": [\"metadata\"],\n \"title\": \"Product Metadata File (xml)\",\n \"description\": \"Collection 2 Level-1 Product Metadata File (xml)\"\n },\n \"mtl.json\": {\n \"type\": \"application/json\",\n \"roles\": [\"metadata\"],\n \"title\": \"Product Metadata File (json)\",\n \"description\": \"Collection 2 Level-1 Product Metadata File (json)\"\n },\n \"qa_pixel\": {\n \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\",\n \"roles\": [\"cloud\"],\n \"title\": \"Pixel Quality Assessment Band\",\n \"description\": \"Collection 2 Level-1 Pixel Quality Assessment Band\",\n \"raster:bands\": [\n {\n \"unit\": \"bit index\",\n \"nodata\": 1,\n \"data_type\": \"uint16\",\n \"spatial_resolution\": 60\n }\n ]\n },\n \"qa_radsat\": {\n \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\",\n \"roles\": [\"saturation\"],\n \"title\": \"Radiometric Saturation and Dropped Pixel Quality Assessment Band\",\n \"description\": \"Collection 2 Level-1 Radiometric Saturation and Dropped Pixel Quality Assessment Band\",\n \"raster:bands\": [\n {\n \"unit\": \"bit index\",\n \"data_type\": \"uint16\",\n \"spatial_resolution\": 60\n }\n ]\n },\n \"thumbnail\": {\n \"type\": \"image/jpeg\",\n \"roles\": [\"thumbnail\"],\n \"title\": \"Thumbnail image\"\n },\n \"reduced_resolution_browse\": {\n \"type\": \"image/jpeg\",\n \"roles\": [\"overview\"],\n \"title\": \"Reduced resolution browse image\"\n }\n },\n \"stac_version\": \"1.0.0\",\n \"stac_extensions\": [\n \"https://stac-extensions.github.io/item-assets/v1.0.0/schema.json\",\n \"https://stac-extensions.github.io/view/v1.0.0/schema.json\",\n \"https://stac-extensions.github.io/scientific/v1.0.0/schema.json\",\n \"https://stac-extensions.github.io/raster/v1.0.0/schema.json\",\n \"https://stac-extensions.github.io/eo/v1.0.0/schema.json\"\n ]\n}\n" }, { "path": "src/pypgstac/tests/data-files/hydration/collections/sentinel-1-grd.json", "content": "{\n \"id\": \"sentinel-1-grd\",\n \"type\": \"Collection\",\n \"links\": [\n {\n \"rel\": \"items\",\n \"type\": \"application/geo+json\",\n \"href\": \"https://pct-apis-staging.westeurope.cloudapp.azure.com/stac/collections/sentinel-1-grd/items\"\n },\n {\n \"rel\": \"parent\",\n \"type\": \"application/json\",\n \"href\": \"https://pct-apis-staging.westeurope.cloudapp.azure.com/stac/\"\n },\n {\n \"rel\": \"root\",\n \"type\": \"application/json\",\n \"href\": \"https://pct-apis-staging.westeurope.cloudapp.azure.com/stac/\"\n },\n {\n \"rel\": \"self\",\n \"type\": \"application/json\",\n \"href\": \"https://pct-apis-staging.westeurope.cloudapp.azure.com/stac/collections/sentinel-1-grd\"\n },\n {\n \"rel\": \"license\",\n \"href\": \"https://sentinel.esa.int/documents/247904/690755/Sentinel_Data_Legal_Notice\",\n \"title\": \"Copernicus Sentinel data terms\"\n },\n {\n \"rel\": \"describedby\",\n \"href\": \"https://planetarycomputer.microsoft.com/dataset/sentinel-1-grd\",\n \"title\": \"Human readable dataset overview and reference\",\n \"type\": \"text/html\"\n }\n ],\n \"title\": \"Sentinel 1 Level-1 Ground Range Detected (GRD)\",\n \"assets\": {\n \"thumbnail\": {\n \"href\": \"https://ai4edatasetspublicassets.blob.core.windows.net/assets/pc_thumbnails/sentinel-1-grd.png\",\n \"type\": \"image/png\",\n \"roles\": [\n \"thumbnail\"\n ],\n \"title\": \"Sentinel 1 GRD\"\n }\n },\n \"extent\": {\n \"spatial\": {\n \"bbox\": [\n [\n -180,\n -90,\n 180,\n 90\n ]\n ]\n },\n \"temporal\": {\n \"interval\": [\n [\n \"2014-10-10T00:28:21Z\",\n null\n ]\n ]\n }\n },\n \"license\": \"proprietary\",\n \"keywords\": [\n \"ESA\",\n \"Copernicus\",\n \"Sentinel\",\n \"C-Band\",\n \"SAR\",\n \"GRD\"\n ],\n \"providers\": [\n {\n \"url\": \"https://earth.esa.int/web/guest/home\",\n \"name\": \"ESA\",\n \"roles\": [\n \"producer\",\n \"processor\",\n \"licensor\"\n ]\n },\n {\n \"url\": \"https://planetarycomputer.microsoft.com\",\n \"name\": \"Microsoft\",\n \"roles\": [\n \"host\"\n ]\n }\n ],\n \"summaries\": {\n \"platform\": [\n \"SENTINEL-1A\",\n \"SENTINEL-1B\"\n ],\n \"constellation\": [\n \"Sentinel-1\"\n ],\n \"s1:resolution\": [\n \"full\",\n \"high\",\n \"medium\"\n ],\n \"s1:orbit_source\": [\n \"DOWNLINK\",\n \"POEORB\",\n \"PREORB\",\n \"RESORB\"\n ],\n \"sar:looks_range\": [\n 5,\n 6,\n 3,\n 2\n ],\n \"sat:orbit_state\": [\n \"ascending\",\n \"descending\"\n ],\n \"sar:product_type\": [\n \"GRD\"\n ],\n \"sar:looks_azimuth\": [\n 1,\n 6,\n 2\n ],\n \"sar:polarizations\": [\n [\n \"VV\",\n \"VH\"\n ],\n [\n \"HH\",\n \"HV\"\n ],\n [\n \"VV\"\n ],\n [\n \"VH\"\n ],\n [\n \"HH\"\n ],\n [\n \"HV\"\n ]\n ],\n \"sar:frequency_band\": [\n \"C\"\n ],\n \"s1:processing_level\": [\n \"1\"\n ],\n \"sar:instrument_mode\": [\n \"IW\",\n \"EW\",\n \"SM\"\n ],\n \"sar:center_frequency\": [\n 5.405\n ],\n \"sar:resolution_range\": [\n 20,\n 23,\n 50,\n 93,\n 9\n ],\n \"s1:product_timeliness\": [\n \"NRT-10m\",\n \"NRT-1h\",\n \"NRT-3h\",\n \"Fast-24h\",\n \"Off-line\",\n \"Reprocessing\"\n ],\n \"sar:resolution_azimuth\": [\n 22,\n 23,\n 50,\n 87,\n 9\n ],\n \"sar:pixel_spacing_range\": [\n 10,\n 25,\n 40,\n 3.5\n ],\n \"sar:observation_direction\": [\n \"right\"\n ],\n \"sar:pixel_spacing_azimuth\": [\n 10,\n 25,\n 40,\n 3.5\n ],\n \"sar:looks_equivalent_number\": [\n 4.4,\n 29.7,\n 2.7,\n 10.7,\n 3.7\n ],\n \"sat:platform_international_designator\": [\n \"2014-016A\",\n \"2016-025A\",\n \"0000-000A\"\n ]\n },\n \"description\": \"...\",\n \"item_assets\": {\n \"hh\": {\n \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\",\n \"roles\": [\n \"data\"\n ],\n \"title\": \"HH: horizontal transmit, horizontal receive\",\n \"description\": \"Amplitude of signal transmitted with horizontal polarization and received with horizontal polarization with radiometric terrain correction applied.\"\n },\n \"hv\": {\n \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\",\n \"roles\": [\n \"data\"\n ],\n \"title\": \"HV: horizontal transmit, vertical receive\",\n \"description\": \"Amplitude of signal transmitted with horizontal polarization and received with vertical polarization with radiometric terrain correction applied.\"\n },\n \"vh\": {\n \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\",\n \"roles\": [\n \"data\"\n ],\n \"title\": \"VH: vertical transmit, horizontal receive\",\n \"description\": \"Amplitude of signal transmitted with vertical polarization and received with horizontal polarization with radiometric terrain correction applied.\"\n },\n \"vv\": {\n \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\",\n \"roles\": [\n \"data\"\n ],\n \"title\": \"VV: vertical transmit, vertical receive\",\n \"description\": \"Amplitude of signal transmitted with vertical polarization and received with vertical polarization with radiometric terrain correction applied.\"\n },\n \"thumbnail\": {\n \"type\": \"image/png\",\n \"roles\": [\n \"thumbnail\"\n ],\n \"title\": \"Preview Image\",\n \"description\": \"An averaged, decimated preview image in PNG format. Single polarisation products are represented with a grey scale image. Dual polarisation products are represented by a single composite colour image in RGB with the red channel (R) representing the co-polarisation VV or HH), the green channel (G) represents the cross-polarisation (VH or HV) and the blue channel (B) represents the ratio of the cross an co-polarisations.\"\n },\n \"safe-manifest\": {\n \"type\": \"application/xml\",\n \"roles\": [\n \"metadata\"\n ],\n \"title\": \"Manifest File\",\n \"description\": \"General product metadata in XML format. Contains a high-level textual description of the product and references to all of product's components, the product metadata, including the product identification and the resource references, and references to the physical location of each component file contained in the product.\"\n },\n \"schema-noise-hh\": {\n \"type\": \"application/xml\",\n \"roles\": [\n \"metadata\"\n ],\n \"title\": \"Noise Schema\",\n \"description\": \"Estimated thermal noise look-up tables\"\n },\n \"schema-noise-hv\": {\n \"type\": \"application/xml\",\n \"roles\": [\n \"metadata\"\n ],\n \"title\": \"Noise Schema\",\n \"description\": \"Estimated thermal noise look-up tables\"\n },\n \"schema-noise-vh\": {\n \"type\": \"application/xml\",\n \"roles\": [\n \"metadata\"\n ],\n \"title\": \"Noise Schema\",\n \"description\": \"Estimated thermal noise look-up tables\"\n },\n \"schema-noise-vv\": {\n \"type\": \"application/xml\",\n \"roles\": [\n \"metadata\"\n ],\n \"title\": \"Noise Schema\",\n \"description\": \"Estimated thermal noise look-up tables\"\n },\n \"schema-product-hh\": {\n \"type\": \"application/xml\",\n \"roles\": [\n \"metadata\"\n ],\n \"title\": \"Product Schema\",\n \"description\": \"Describes the main characteristics corresponding to the band: state of the platform during acquisition, image properties, Doppler information, geographic location, etc.\"\n },\n \"schema-product-hv\": {\n \"type\": \"application/xml\",\n \"roles\": [\n \"metadata\"\n ],\n \"title\": \"Product Schema\",\n \"description\": \"Describes the main characteristics corresponding to the band: state of the platform during acquisition, image properties, Doppler information, geographic location, etc.\"\n },\n \"schema-product-vh\": {\n \"type\": \"application/xml\",\n \"roles\": [\n \"metadata\"\n ],\n \"title\": \"Product Schema\",\n \"description\": \"Describes the main characteristics corresponding to the band: state of the platform during acquisition, image properties, Doppler information, geographic location, etc.\"\n },\n \"schema-product-vv\": {\n \"type\": \"application/xml\",\n \"roles\": [\n \"metadata\"\n ],\n \"title\": \"Product Schema\",\n \"description\": \"Describes the main characteristics corresponding to the band: state of the platform during acquisition, image properties, Doppler information, geographic location, etc.\"\n },\n \"schema-calibration-hh\": {\n \"type\": \"application/xml\",\n \"roles\": [\n \"metadata\"\n ],\n \"title\": \"Calibration Schema\",\n \"description\": \"Calibration metadata including calibration information and the beta nought, sigma nought, gamma and digital number look-up tables that can be used for absolute product calibration.\"\n },\n \"schema-calibration-hv\": {\n \"type\": \"application/xml\",\n \"roles\": [\n \"metadata\"\n ],\n \"title\": \"Calibration Schema\",\n \"description\": \"Calibration metadata including calibration information and the beta nought, sigma nought, gamma and digital number look-up tables that can be used for absolute product calibration.\"\n },\n \"schema-calibration-vh\": {\n \"type\": \"application/xml\",\n \"roles\": [\n \"metadata\"\n ],\n \"title\": \"Calibration Schema\",\n \"description\": \"Calibration metadata including calibration information and the beta nought, sigma nought, gamma and digital number look-up tables that can be used for absolute product calibration.\"\n },\n \"schema-calibration-vv\": {\n \"type\": \"application/xml\",\n \"roles\": [\n \"metadata\"\n ],\n \"title\": \"Calibration Schema\",\n \"description\": \"Calibration metadata including calibration information and the beta nought, sigma nought, gamma and digital number look-up tables that can be used for absolute product calibration.\"\n }\n },\n \"stac_version\": \"1.0.0\",\n \"msft:group_id\": \"sentinel-1\",\n \"msft:container\": \"s1-grd\",\n \"stac_extensions\": [\n \"https://stac-extensions.github.io/sar/v1.0.0/schema.json\",\n \"https://stac-extensions.github.io/sat/v1.0.0/schema.json\",\n \"https://stac-extensions.github.io/item-assets/v1.0.0/schema.json\"\n ],\n \"msft:storage_account\": \"sentinel1euwest\",\n \"msft:short_description\": \"Sentinel-1 Level-1 Ground Range Detected (GRD) products consist of focused SAR data that has been detected, multi-looked and projected to ground range using an Earth ellipsoid model.\"\n}" }, { "path": "src/pypgstac/tests/data-files/hydration/dehydrated-items/landsat-c2-l1/LM04_L1GS_001001_19830527_02_T2.json", "content": "{\n \"id\": \"LM04_L1GS_001001_19830527_02_T2\",\n \"properties\": {\n \"platform\": \"landsat-4\",\n \"instruments\": [\"mss\"],\n \"created\": \"2022-04-17T03:44:26.179982Z\",\n \"gsd\": 79,\n \"description\": \"Landsat Collection 2 Level-1\",\n \"eo:cloud_cover\": 32.0,\n \"view:off_nadir\": 0,\n \"view:sun_elevation\": 29.32047976,\n \"view:sun_azimuth\": 210.31823865,\n \"proj:epsg\": 32631,\n \"proj:shape\": [4308, 4371],\n \"proj:transform\": [60.0, 0.0, 378930.0, 0.0, -60.0, 9099030.0],\n \"landsat:cloud_cover_land\": 0.0,\n \"landsat:wrs_type\": \"2\",\n \"landsat:wrs_path\": \"001\",\n \"landsat:wrs_row\": \"001\",\n \"landsat:collection_category\": \"T2\",\n \"landsat:collection_number\": \"02\",\n \"landsat:correction\": \"L1GS\",\n \"landsat:scene_id\": \"LM40010011983147KIS00\",\n \"sci:doi\": \"10.5066/P9AF14YV\",\n \"datetime\": \"1983-05-27T13:36:40.094000Z\"\n },\n \"geometry\": {\n \"coordinates\": [\n [\n [6.3329893, 81.9297399],\n [-3.9422024, 81.0681921],\n [1.4107147, 79.6375797],\n [10.5596079, 80.371579],\n [6.3329893, 81.9297399]\n ]\n ],\n \"type\": \"Polygon\"\n },\n \"links\": [\n {\n \"rel\": \"cite-as\",\n \"href\": \"https://doi.org/10.5066/P9AF14YV\",\n \"title\": \"Landsat 1-5 MSS Collection 2 Level-1\"\n },\n {\n \"rel\": \"via\",\n \"href\": \"https://landsatlook.usgs.gov/stac-server/collections/landsat-c2l1/items/LM04_L1GS_001001_19830527_20210902_02_T2\",\n \"type\": \"application/json\",\n \"title\": \"USGS STAC Item\"\n }\n ],\n \"assets\": {\n \"thumbnail\": {\n \"href\": \"https://landsateuwest.blob.core.windows.net/landsat-c2/level-1/standard/mss/1983/001/001/LM04_L1GS_001001_19830527_20210902_02_T2/LM04_L1GS_001001_19830527_20210902_02_T2_thumb_small.jpeg\"\n },\n \"reduced_resolution_browse\": {\n \"href\": \"https://landsateuwest.blob.core.windows.net/landsat-c2/level-1/standard/mss/1983/001/001/LM04_L1GS_001001_19830527_20210902_02_T2/LM04_L1GS_001001_19830527_20210902_02_T2_thumb_large.jpeg\"\n },\n \"mtl.json\": {\n \"href\": \"https://landsateuwest.blob.core.windows.net/landsat-c2/level-1/standard/mss/1983/001/001/LM04_L1GS_001001_19830527_20210902_02_T2/LM04_L1GS_001001_19830527_20210902_02_T2_MTL.json\"\n },\n \"mtl.txt\": {\n \"href\": \"https://landsateuwest.blob.core.windows.net/landsat-c2/level-1/standard/mss/1983/001/001/LM04_L1GS_001001_19830527_20210902_02_T2/LM04_L1GS_001001_19830527_20210902_02_T2_MTL.txt\"\n },\n \"mtl.xml\": {\n \"href\": \"https://landsateuwest.blob.core.windows.net/landsat-c2/level-1/standard/mss/1983/001/001/LM04_L1GS_001001_19830527_20210902_02_T2/LM04_L1GS_001001_19830527_20210902_02_T2_MTL.xml\"\n },\n \"qa_pixel\": {\n \"href\": \"https://landsateuwest.blob.core.windows.net/landsat-c2/level-1/standard/mss/1983/001/001/LM04_L1GS_001001_19830527_20210902_02_T2/LM04_L1GS_001001_19830527_20210902_02_T2_QA_PIXEL.TIF\",\n \"description\": \"Collection 2 Level-1 Pixel Quality Assessment Band (QA_PIXEL)\",\n \"classification:bitfields\": [\n {\n \"name\": \"fill\",\n \"description\": \"Image or fill data\",\n \"offset\": 0,\n \"length\": 1,\n \"classes\": [\n { \"value\": 0, \"name\": \"not_fill\", \"description\": \"Image data\" },\n { \"value\": 1, \"name\": \"fill\", \"description\": \"Fill data\" }\n ]\n },\n {\n \"name\": \"cloud\",\n \"description\": \"Cloud mask\",\n \"offset\": 3,\n \"length\": 1,\n \"classes\": [\n {\n \"value\": 0,\n \"name\": \"not_cloud\",\n \"description\": \"Cloud confidence is not high\"\n },\n {\n \"value\": 1,\n \"name\": \"cloud\",\n \"description\": \"High confidence cloud\"\n }\n ]\n },\n {\n \"name\": \"cloud_confidence\",\n \"description\": \"Cloud confidence levels\",\n \"offset\": 8,\n \"length\": 2,\n \"classes\": [\n {\n \"value\": 0,\n \"name\": \"not_set\",\n \"description\": \"No confidence level set\"\n },\n {\n \"value\": 1,\n \"name\": \"low\",\n \"description\": \"Low confidence cloud\"\n },\n {\n \"value\": 2,\n \"name\": \"reserved\",\n \"description\": \"Reserved - value not used\"\n },\n {\n \"value\": 3,\n \"name\": \"high\",\n \"description\": \"High confidence cloud\"\n }\n ]\n }\n ]\n },\n \"qa_radsat\": {\n \"href\": \"https://landsateuwest.blob.core.windows.net/landsat-c2/level-1/standard/mss/1983/001/001/LM04_L1GS_001001_19830527_20210902_02_T2/LM04_L1GS_001001_19830527_20210902_02_T2_QA_RADSAT.TIF\",\n \"description\": \"Collection 2 Level-1 Radiometric Saturation and Dropped Pixel Quality Assessment Band (QA_RADSAT)\",\n \"classification:bitfields\": [\n {\n \"name\": \"band1\",\n \"description\": \"Band 1 radiometric saturation\",\n \"offset\": 0,\n \"length\": 1,\n \"classes\": [\n {\n \"value\": 0,\n \"name\": \"not_saturated\",\n \"description\": \"Band 1 not saturated\"\n },\n {\n \"value\": 1,\n \"name\": \"saturated\",\n \"description\": \"Band 1 saturated\"\n }\n ]\n },\n {\n \"name\": \"band2\",\n \"description\": \"Band 2 radiometric saturation\",\n \"offset\": 1,\n \"length\": 1,\n \"classes\": [\n {\n \"value\": 0,\n \"name\": \"not_saturated\",\n \"description\": \"Band 2 not saturated\"\n },\n {\n \"value\": 1,\n \"name\": \"saturated\",\n \"description\": \"Band 2 saturated\"\n }\n ]\n },\n {\n \"name\": \"band3\",\n \"description\": \"Band 3 radiometric saturation\",\n \"offset\": 2,\n \"length\": 1,\n \"classes\": [\n {\n \"value\": 0,\n \"name\": \"not_saturated\",\n \"description\": \"Band 3 not saturated\"\n },\n {\n \"value\": 1,\n \"name\": \"saturated\",\n \"description\": \"Band 3 saturated\"\n }\n ]\n },\n {\n \"name\": \"band4\",\n \"description\": \"Band 4 radiometric saturation\",\n \"offset\": 3,\n \"length\": 1,\n \"classes\": [\n {\n \"value\": 0,\n \"name\": \"not_saturated\",\n \"description\": \"Band 4 not saturated\"\n },\n {\n \"value\": 1,\n \"name\": \"saturated\",\n \"description\": \"Band 4 saturated\"\n }\n ]\n },\n {\n \"name\": \"band5\",\n \"description\": \"Band 5 radiometric saturation\",\n \"offset\": 4,\n \"length\": 1,\n \"classes\": [\n {\n \"value\": 0,\n \"name\": \"not_saturated\",\n \"description\": \"Band 5 not saturated\"\n },\n {\n \"value\": 1,\n \"name\": \"saturated\",\n \"description\": \"Band 5 saturated\"\n }\n ]\n },\n {\n \"name\": \"band6\",\n \"description\": \"Band 6 radiometric saturation\",\n \"offset\": 5,\n \"length\": 1,\n \"classes\": [\n {\n \"value\": 0,\n \"name\": \"not_saturated\",\n \"description\": \"Band 6 not saturated\"\n },\n {\n \"value\": 1,\n \"name\": \"saturated\",\n \"description\": \"Band 6 saturated\"\n }\n ]\n },\n {\n \"name\": \"band7\",\n \"description\": \"Band 7 radiometric saturation\",\n \"offset\": 6,\n \"length\": 1,\n \"classes\": [\n {\n \"value\": 0,\n \"name\": \"not_saturated\",\n \"description\": \"Band 7 not saturated\"\n },\n {\n \"value\": 1,\n \"name\": \"saturated\",\n \"description\": \"Band 7 saturated\"\n }\n ]\n },\n {\n \"name\": \"dropped\",\n \"description\": \"Dropped pixel\",\n \"offset\": 9,\n \"length\": 1,\n \"classes\": [\n {\n \"value\": 0,\n \"name\": \"not_dropped\",\n \"description\": \"Detector has a value - pixel present\"\n },\n {\n \"value\": 1,\n \"name\": \"dropped\",\n \"description\": \"Detector does not have a value - no data\"\n }\n ]\n }\n ]\n },\n \"green\": {\n \"href\": \"https://landsateuwest.blob.core.windows.net/landsat-c2/level-1/standard/mss/1983/001/001/LM04_L1GS_001001_19830527_20210902_02_T2/LM04_L1GS_001001_19830527_20210902_02_T2_B1.TIF\",\n \"description\": \"Collection 2 Level-1 Green Band (B1) Top of Atmosphere Radiance\",\n \"eo:bands\": [\n {\n \"name\": \"B1\",\n \"common_name\": \"green\",\n \"description\": \"Visible green\",\n \"center_wavelength\": 0.55,\n \"full_width_half_max\": 0.1\n }\n ],\n \"raster:bands\": [{ \"scale\": 0.8752, \"offset\": 2.9248 }]\n },\n \"red\": {\n \"href\": \"https://landsateuwest.blob.core.windows.net/landsat-c2/level-1/standard/mss/1983/001/001/LM04_L1GS_001001_19830527_20210902_02_T2/LM04_L1GS_001001_19830527_20210902_02_T2_B2.TIF\",\n \"eo:bands\": [\n {\n \"name\": \"B2\",\n \"common_name\": \"red\",\n \"description\": \"Visible red\",\n \"center_wavelength\": 0.65,\n \"full_width_half_max\": 0.1\n }\n ],\n \"raster:bands\": [{ \"scale\": 0.62008, \"offset\": 3.07992 }]\n },\n \"nir08\": {\n \"href\": \"https://landsateuwest.blob.core.windows.net/landsat-c2/level-1/standard/mss/1983/001/001/LM04_L1GS_001001_19830527_20210902_02_T2/LM04_L1GS_001001_19830527_20210902_02_T2_B3.TIF\",\n \"description\": \"Collection 2 Level-1 Near Infrared Band 0.8 (B3) Top of Atmosphere Radiance\",\n \"eo:bands\": [\n {\n \"name\": \"B3\",\n \"common_name\": \"nir08\",\n \"description\": \"Near infrared\",\n \"center_wavelength\": 0.75,\n \"full_width_half_max\": 0.1\n }\n ],\n \"raster:bands\": [{ \"scale\": 0.54921, \"offset\": 4.55079 }]\n },\n \"nir09\": {\n \"href\": \"https://landsateuwest.blob.core.windows.net/landsat-c2/level-1/standard/mss/1983/001/001/LM04_L1GS_001001_19830527_20210902_02_T2/LM04_L1GS_001001_19830527_20210902_02_T2_B4.TIF\",\n \"eo:bands\": [\n {\n \"name\": \"B4\",\n \"common_name\": \"nir09\",\n \"description\": \"Near infrared\",\n \"center_wavelength\": 0.95,\n \"full_width_half_max\": 0.3\n }\n ],\n \"raster:bands\": [{ \"unit\": \"𒍟※\" }]\n }\n },\n \"bbox\": [-4.69534637, 79.55635318, 11.95560413, 81.87586682],\n \"stac_extensions\": [\n \"https://stac-extensions.github.io/raster/v1.0.0/schema.json\",\n \"https://stac-extensions.github.io/eo/v1.0.0/schema.json\",\n \"https://stac-extensions.github.io/view/v1.0.0/schema.json\",\n \"https://stac-extensions.github.io/projection/v1.0.0/schema.json\",\n \"https://landsat.usgs.gov/stac/landsat-extension/v1.1.1/schema.json\",\n \"https://stac-extensions.github.io/classification/v1.0.0/schema.json\",\n \"https://stac-extensions.github.io/scientific/v1.0.0/schema.json\"\n ]\n}\n" }, { "path": "src/pypgstac/tests/data-files/hydration/raw-items/landsat-c2-l1/LM04_L1GS_001001_19830527_02_T2.json", "content": "{\n \"type\": \"Feature\",\n \"stac_version\": \"1.0.0\",\n \"id\": \"LM04_L1GS_001001_19830527_02_T2\",\n \"properties\": {\n \"platform\": \"landsat-4\",\n \"instruments\": [\"mss\"],\n \"created\": \"2022-04-17T03:44:26.179982Z\",\n \"gsd\": 79,\n \"description\": \"Landsat Collection 2 Level-1\",\n \"eo:cloud_cover\": 32.0,\n \"view:off_nadir\": 0,\n \"view:sun_elevation\": 29.32047976,\n \"view:sun_azimuth\": 210.31823865,\n \"proj:epsg\": 32631,\n \"proj:shape\": [4308, 4371],\n \"proj:transform\": [60.0, 0.0, 378930.0, 0.0, -60.0, 9099030.0],\n \"landsat:cloud_cover_land\": 0.0,\n \"landsat:wrs_type\": \"2\",\n \"landsat:wrs_path\": \"001\",\n \"landsat:wrs_row\": \"001\",\n \"landsat:collection_category\": \"T2\",\n \"landsat:collection_number\": \"02\",\n \"landsat:correction\": \"L1GS\",\n \"landsat:scene_id\": \"LM40010011983147KIS00\",\n \"sci:doi\": \"10.5066/P9AF14YV\",\n \"datetime\": \"1983-05-27T13:36:40.094000Z\"\n },\n \"geometry\": {\n \"coordinates\": [\n [\n [6.3329893, 81.9297399],\n [-3.9422024, 81.0681921],\n [1.4107147, 79.6375797],\n [10.5596079, 80.371579],\n [6.3329893, 81.9297399]\n ]\n ],\n \"type\": \"Polygon\"\n },\n \"links\": [\n {\n \"rel\": \"cite-as\",\n \"href\": \"https://doi.org/10.5066/P9AF14YV\",\n \"title\": \"Landsat 1-5 MSS Collection 2 Level-1\"\n },\n {\n \"rel\": \"via\",\n \"href\": \"https://landsatlook.usgs.gov/stac-server/collections/landsat-c2l1/items/LM04_L1GS_001001_19830527_20210902_02_T2\",\n \"type\": \"application/json\",\n \"title\": \"USGS STAC Item\"\n }\n ],\n \"assets\": {\n \"thumbnail\": {\n \"href\": \"https://landsateuwest.blob.core.windows.net/landsat-c2/level-1/standard/mss/1983/001/001/LM04_L1GS_001001_19830527_20210902_02_T2/LM04_L1GS_001001_19830527_20210902_02_T2_thumb_small.jpeg\",\n \"type\": \"image/jpeg\",\n \"title\": \"Thumbnail image\",\n \"roles\": [\"thumbnail\"]\n },\n \"reduced_resolution_browse\": {\n \"href\": \"https://landsateuwest.blob.core.windows.net/landsat-c2/level-1/standard/mss/1983/001/001/LM04_L1GS_001001_19830527_20210902_02_T2/LM04_L1GS_001001_19830527_20210902_02_T2_thumb_large.jpeg\",\n \"type\": \"image/jpeg\",\n \"title\": \"Reduced resolution browse image\",\n \"roles\": [\"overview\"]\n },\n \"mtl.json\": {\n \"href\": \"https://landsateuwest.blob.core.windows.net/landsat-c2/level-1/standard/mss/1983/001/001/LM04_L1GS_001001_19830527_20210902_02_T2/LM04_L1GS_001001_19830527_20210902_02_T2_MTL.json\",\n \"type\": \"application/json\",\n \"title\": \"Product Metadata File (json)\",\n \"description\": \"Collection 2 Level-1 Product Metadata File (json)\",\n \"roles\": [\"metadata\"]\n },\n \"mtl.txt\": {\n \"href\": \"https://landsateuwest.blob.core.windows.net/landsat-c2/level-1/standard/mss/1983/001/001/LM04_L1GS_001001_19830527_20210902_02_T2/LM04_L1GS_001001_19830527_20210902_02_T2_MTL.txt\",\n \"type\": \"text/plain\",\n \"title\": \"Product Metadata File (txt)\",\n \"description\": \"Collection 2 Level-1 Product Metadata File (txt)\",\n \"roles\": [\"metadata\"]\n },\n \"mtl.xml\": {\n \"href\": \"https://landsateuwest.blob.core.windows.net/landsat-c2/level-1/standard/mss/1983/001/001/LM04_L1GS_001001_19830527_20210902_02_T2/LM04_L1GS_001001_19830527_20210902_02_T2_MTL.xml\",\n \"type\": \"application/xml\",\n \"title\": \"Product Metadata File (xml)\",\n \"description\": \"Collection 2 Level-1 Product Metadata File (xml)\",\n \"roles\": [\"metadata\"]\n },\n \"qa_pixel\": {\n \"href\": \"https://landsateuwest.blob.core.windows.net/landsat-c2/level-1/standard/mss/1983/001/001/LM04_L1GS_001001_19830527_20210902_02_T2/LM04_L1GS_001001_19830527_20210902_02_T2_QA_PIXEL.TIF\",\n \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\",\n \"title\": \"Pixel Quality Assessment Band\",\n \"description\": \"Collection 2 Level-1 Pixel Quality Assessment Band (QA_PIXEL)\",\n \"classification:bitfields\": [\n {\n \"name\": \"fill\",\n \"description\": \"Image or fill data\",\n \"offset\": 0,\n \"length\": 1,\n \"classes\": [\n { \"value\": 0, \"name\": \"not_fill\", \"description\": \"Image data\" },\n { \"value\": 1, \"name\": \"fill\", \"description\": \"Fill data\" }\n ]\n },\n {\n \"name\": \"cloud\",\n \"description\": \"Cloud mask\",\n \"offset\": 3,\n \"length\": 1,\n \"classes\": [\n {\n \"value\": 0,\n \"name\": \"not_cloud\",\n \"description\": \"Cloud confidence is not high\"\n },\n {\n \"value\": 1,\n \"name\": \"cloud\",\n \"description\": \"High confidence cloud\"\n }\n ]\n },\n {\n \"name\": \"cloud_confidence\",\n \"description\": \"Cloud confidence levels\",\n \"offset\": 8,\n \"length\": 2,\n \"classes\": [\n {\n \"value\": 0,\n \"name\": \"not_set\",\n \"description\": \"No confidence level set\"\n },\n {\n \"value\": 1,\n \"name\": \"low\",\n \"description\": \"Low confidence cloud\"\n },\n {\n \"value\": 2,\n \"name\": \"reserved\",\n \"description\": \"Reserved - value not used\"\n },\n {\n \"value\": 3,\n \"name\": \"high\",\n \"description\": \"High confidence cloud\"\n }\n ]\n }\n ],\n \"raster:bands\": [\n {\n \"nodata\": 1,\n \"data_type\": \"uint16\",\n \"spatial_resolution\": 60,\n \"unit\": \"bit index\"\n }\n ],\n \"roles\": [\"cloud\"]\n },\n \"qa_radsat\": {\n \"href\": \"https://landsateuwest.blob.core.windows.net/landsat-c2/level-1/standard/mss/1983/001/001/LM04_L1GS_001001_19830527_20210902_02_T2/LM04_L1GS_001001_19830527_20210902_02_T2_QA_RADSAT.TIF\",\n \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\",\n \"title\": \"Radiometric Saturation and Dropped Pixel Quality Assessment Band\",\n \"description\": \"Collection 2 Level-1 Radiometric Saturation and Dropped Pixel Quality Assessment Band (QA_RADSAT)\",\n \"classification:bitfields\": [\n {\n \"name\": \"band1\",\n \"description\": \"Band 1 radiometric saturation\",\n \"offset\": 0,\n \"length\": 1,\n \"classes\": [\n {\n \"value\": 0,\n \"name\": \"not_saturated\",\n \"description\": \"Band 1 not saturated\"\n },\n {\n \"value\": 1,\n \"name\": \"saturated\",\n \"description\": \"Band 1 saturated\"\n }\n ]\n },\n {\n \"name\": \"band2\",\n \"description\": \"Band 2 radiometric saturation\",\n \"offset\": 1,\n \"length\": 1,\n \"classes\": [\n {\n \"value\": 0,\n \"name\": \"not_saturated\",\n \"description\": \"Band 2 not saturated\"\n },\n {\n \"value\": 1,\n \"name\": \"saturated\",\n \"description\": \"Band 2 saturated\"\n }\n ]\n },\n {\n \"name\": \"band3\",\n \"description\": \"Band 3 radiometric saturation\",\n \"offset\": 2,\n \"length\": 1,\n \"classes\": [\n {\n \"value\": 0,\n \"name\": \"not_saturated\",\n \"description\": \"Band 3 not saturated\"\n },\n {\n \"value\": 1,\n \"name\": \"saturated\",\n \"description\": \"Band 3 saturated\"\n }\n ]\n },\n {\n \"name\": \"band4\",\n \"description\": \"Band 4 radiometric saturation\",\n \"offset\": 3,\n \"length\": 1,\n \"classes\": [\n {\n \"value\": 0,\n \"name\": \"not_saturated\",\n \"description\": \"Band 4 not saturated\"\n },\n {\n \"value\": 1,\n \"name\": \"saturated\",\n \"description\": \"Band 4 saturated\"\n }\n ]\n },\n {\n \"name\": \"band5\",\n \"description\": \"Band 5 radiometric saturation\",\n \"offset\": 4,\n \"length\": 1,\n \"classes\": [\n {\n \"value\": 0,\n \"name\": \"not_saturated\",\n \"description\": \"Band 5 not saturated\"\n },\n {\n \"value\": 1,\n \"name\": \"saturated\",\n \"description\": \"Band 5 saturated\"\n }\n ]\n },\n {\n \"name\": \"band6\",\n \"description\": \"Band 6 radiometric saturation\",\n \"offset\": 5,\n \"length\": 1,\n \"classes\": [\n {\n \"value\": 0,\n \"name\": \"not_saturated\",\n \"description\": \"Band 6 not saturated\"\n },\n {\n \"value\": 1,\n \"name\": \"saturated\",\n \"description\": \"Band 6 saturated\"\n }\n ]\n },\n {\n \"name\": \"band7\",\n \"description\": \"Band 7 radiometric saturation\",\n \"offset\": 6,\n \"length\": 1,\n \"classes\": [\n {\n \"value\": 0,\n \"name\": \"not_saturated\",\n \"description\": \"Band 7 not saturated\"\n },\n {\n \"value\": 1,\n \"name\": \"saturated\",\n \"description\": \"Band 7 saturated\"\n }\n ]\n },\n {\n \"name\": \"dropped\",\n \"description\": \"Dropped pixel\",\n \"offset\": 9,\n \"length\": 1,\n \"classes\": [\n {\n \"value\": 0,\n \"name\": \"not_dropped\",\n \"description\": \"Detector has a value - pixel present\"\n },\n {\n \"value\": 1,\n \"name\": \"dropped\",\n \"description\": \"Detector does not have a value - no data\"\n }\n ]\n }\n ],\n \"raster:bands\": [\n { \"data_type\": \"uint16\", \"spatial_resolution\": 60, \"unit\": \"bit index\" }\n ],\n \"roles\": [\"saturation\"]\n },\n \"green\": {\n \"href\": \"https://landsateuwest.blob.core.windows.net/landsat-c2/level-1/standard/mss/1983/001/001/LM04_L1GS_001001_19830527_20210902_02_T2/LM04_L1GS_001001_19830527_20210902_02_T2_B1.TIF\",\n \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\",\n \"title\": \"Green Band\",\n \"description\": \"Collection 2 Level-1 Green Band (B1) Top of Atmosphere Radiance\",\n \"eo:bands\": [\n {\n \"name\": \"B1\",\n \"common_name\": \"green\",\n \"description\": \"Visible green\",\n \"center_wavelength\": 0.55,\n \"full_width_half_max\": 0.1\n }\n ],\n \"raster:bands\": [\n {\n \"nodata\": 0,\n \"data_type\": \"uint8\",\n \"spatial_resolution\": 60,\n \"unit\": \"watt/steradian/square_meter/micrometer\",\n \"scale\": 0.8752,\n \"offset\": 2.9248\n }\n ],\n \"roles\": [\"data\"]\n },\n \"red\": {\n \"href\": \"https://landsateuwest.blob.core.windows.net/landsat-c2/level-1/standard/mss/1983/001/001/LM04_L1GS_001001_19830527_20210902_02_T2/LM04_L1GS_001001_19830527_20210902_02_T2_B2.TIF\",\n \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\",\n \"title\": \"Red Band\",\n \"description\": \"Collection 2 Level-1 Red Band Top of Atmosphere Radiance\",\n \"eo:bands\": [\n {\n \"name\": \"B2\",\n \"common_name\": \"red\",\n \"description\": \"Visible red\",\n \"center_wavelength\": 0.65,\n \"full_width_half_max\": 0.1\n }\n ],\n \"raster:bands\": [\n {\n \"nodata\": 0,\n \"data_type\": \"uint8\",\n \"spatial_resolution\": 60,\n \"unit\": \"watt/steradian/square_meter/micrometer\",\n \"scale\": 0.62008,\n \"offset\": 3.07992\n }\n ],\n \"roles\": [\"data\"]\n },\n \"nir08\": {\n \"href\": \"https://landsateuwest.blob.core.windows.net/landsat-c2/level-1/standard/mss/1983/001/001/LM04_L1GS_001001_19830527_20210902_02_T2/LM04_L1GS_001001_19830527_20210902_02_T2_B3.TIF\",\n \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\",\n \"title\": \"Near Infrared Band 0.8\",\n \"description\": \"Collection 2 Level-1 Near Infrared Band 0.8 (B3) Top of Atmosphere Radiance\",\n \"eo:bands\": [\n {\n \"name\": \"B3\",\n \"common_name\": \"nir08\",\n \"description\": \"Near infrared\",\n \"center_wavelength\": 0.75,\n \"full_width_half_max\": 0.1\n }\n ],\n \"raster:bands\": [\n {\n \"nodata\": 0,\n \"data_type\": \"uint8\",\n \"spatial_resolution\": 60,\n \"unit\": \"watt/steradian/square_meter/micrometer\",\n \"scale\": 0.54921,\n \"offset\": 4.55079\n }\n ],\n \"roles\": [\"data\"]\n },\n \"nir09\": {\n \"href\": \"https://landsateuwest.blob.core.windows.net/landsat-c2/level-1/standard/mss/1983/001/001/LM04_L1GS_001001_19830527_20210902_02_T2/LM04_L1GS_001001_19830527_20210902_02_T2_B4.TIF\",\n \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\",\n \"title\": \"Near Infrared Band 0.9\",\n \"description\": \"Collection 2 Level-1 Near Infrared Band 0.9 Top of Atmosphere Radiance\",\n \"eo:bands\": [\n {\n \"name\": \"B4\",\n \"common_name\": \"nir09\",\n \"description\": \"Near infrared\",\n \"center_wavelength\": 0.95,\n \"full_width_half_max\": 0.3\n }\n ],\n \"raster:bands\": [\n {\n \"nodata\": 0,\n \"data_type\": \"uint8\",\n \"spatial_resolution\": 60\n }\n ],\n \"roles\": [\"data\"]\n }\n },\n \"bbox\": [-4.69534637, 79.55635318, 11.95560413, 81.87586682],\n \"stac_extensions\": [\n \"https://stac-extensions.github.io/raster/v1.0.0/schema.json\",\n \"https://stac-extensions.github.io/eo/v1.0.0/schema.json\",\n \"https://stac-extensions.github.io/view/v1.0.0/schema.json\",\n \"https://stac-extensions.github.io/projection/v1.0.0/schema.json\",\n \"https://landsat.usgs.gov/stac/landsat-extension/v1.1.1/schema.json\",\n \"https://stac-extensions.github.io/classification/v1.0.0/schema.json\",\n \"https://stac-extensions.github.io/scientific/v1.0.0/schema.json\"\n ],\n \"collection\": \"landsat-c2-l1\"\n}\n" }, { "path": "src/pypgstac/tests/data-files/hydration/raw-items/sentinel-1-grd/S1A_IW_GRDH_1SDV_20220428T034417_20220428T034442_042968_05213C.json", "content": "{\n \"id\": \"S1A_IW_GRDH_1SDV_20220428T034417_20220428T034442_042968_05213C\",\n \"bbox\": [\n 28.75231935,\n 1.1415594,\n 31.26726622,\n 3.11793585\n ],\n \"type\": \"Feature\",\n \"links\": [\n {\n \"rel\": \"collection\",\n \"type\": \"application/json\",\n \"href\": \"https://planetarycomputer-test.microsoft.com/stac/collections/sentinel-1-grd\"\n },\n {\n \"rel\": \"parent\",\n \"type\": \"application/json\",\n \"href\": \"https://planetarycomputer-test.microsoft.com/stac/collections/sentinel-1-grd\"\n },\n {\n \"rel\": \"root\",\n \"type\": \"application/json\",\n \"href\": \"https://planetarycomputer-test.microsoft.com/stac/\"\n },\n {\n \"rel\": \"self\",\n \"type\": \"application/geo+json\",\n \"href\": \"https://planetarycomputer-test.microsoft.com/stac/collections/sentinel-1-grd/items/S1A_IW_GRDH_1SDV_20220428T034417_20220428T034442_042968_05213C\"\n },\n {\n \"rel\": \"license\",\n \"href\": \"https://sentinel.esa.int/documents/247904/690755/Sentinel_Data_Legal_Notice\"\n },\n {\n \"rel\": \"preview\",\n \"href\": \"https://pct-apis-staging.westeurope.cloudapp.azure.com/data/item/map?collection=sentinel-1-grd&item=S1A_IW_GRDH_1SDV_20220428T034417_20220428T034442_042968_05213C\",\n \"title\": \"Map of item\",\n \"type\": \"text/html\"\n }\n ],\n \"assets\": {\n \"vh\": {\n \"href\": \"https://sentinel1euwest.blob.core.windows.net/s1-grd/GRD/2022/4/28/IW/DV/S1A_IW_GRDH_1SDV_20220428T034417_20220428T034442_042968_05213C_28ED/measurement/iw-vh.tiff\",\n \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\",\n \"roles\": [\n \"data\"\n ],\n \"title\": \"VH: vertical transmit, horizontal receive\",\n \"description\": \"Amplitude of signal transmitted with vertical polarization and received with horizontal polarization with radiometric terrain correction applied.\"\n },\n \"vv\": {\n \"href\": \"https://sentinel1euwest.blob.core.windows.net/s1-grd/GRD/2022/4/28/IW/DV/S1A_IW_GRDH_1SDV_20220428T034417_20220428T034442_042968_05213C_28ED/measurement/iw-vv.tiff\",\n \"type\": \"image/tiff; application=geotiff; profile=cloud-optimized\",\n \"roles\": [\n \"data\"\n ],\n \"title\": \"VV: vertical transmit, vertical receive\",\n \"description\": \"Amplitude of signal transmitted with vertical polarization and received with vertical polarization with radiometric terrain correction applied.\"\n },\n \"thumbnail\": {\n \"href\": \"https://sentinel1euwest.blob.core.windows.net/s1-grd/GRD/2022/4/28/IW/DV/S1A_IW_GRDH_1SDV_20220428T034417_20220428T034442_042968_05213C_28ED/preview/quick-look.png\",\n \"type\": \"image/png\",\n \"roles\": [\n \"thumbnail\"\n ],\n \"title\": \"Preview Image\",\n \"description\": \"An averaged, decimated preview image in PNG format. Single polarisation products are represented with a grey scale image. Dual polarisation products are represented by a single composite colour image in RGB with the red channel (R) representing the co-polarisation VV or HH), the green channel (G) represents the cross-polarisation (VH or HV) and the blue channel (B) represents the ratio of the cross an co-polarisations.\"\n },\n \"safe-manifest\": {\n \"href\": \"https://sentinel1euwest.blob.core.windows.net/s1-grd/GRD/2022/4/28/IW/DV/S1A_IW_GRDH_1SDV_20220428T034417_20220428T034442_042968_05213C_28ED/manifest.safe\",\n \"type\": \"application/xml\",\n \"roles\": [\n \"metadata\"\n ],\n \"title\": \"Manifest File\",\n \"description\": \"General product metadata in XML format. Contains a high-level textual description of the product and references to all of product's components, the product metadata, including the product identification and the resource references, and references to the physical location of each component file contained in the product.\"\n },\n \"schema-noise-vh\": {\n \"href\": \"https://sentinel1euwest.blob.core.windows.net/s1-grd/GRD/2022/4/28/IW/DV/S1A_IW_GRDH_1SDV_20220428T034417_20220428T034442_042968_05213C_28ED/annotation/calibration/noise-iw-vh.xml\",\n \"type\": \"application/xml\",\n \"roles\": [\n \"metadata\"\n ],\n \"title\": \"Noise Schema\",\n \"description\": \"Estimated thermal noise look-up tables\"\n },\n \"schema-noise-vv\": {\n \"href\": \"https://sentinel1euwest.blob.core.windows.net/s1-grd/GRD/2022/4/28/IW/DV/S1A_IW_GRDH_1SDV_20220428T034417_20220428T034442_042968_05213C_28ED/annotation/calibration/noise-iw-vv.xml\",\n \"type\": \"application/xml\",\n \"roles\": [\n \"metadata\"\n ],\n \"title\": \"Noise Schema\",\n \"description\": \"Estimated thermal noise look-up tables\"\n },\n \"schema-product-vh\": {\n \"href\": \"https://sentinel1euwest.blob.core.windows.net/s1-grd/GRD/2022/4/28/IW/DV/S1A_IW_GRDH_1SDV_20220428T034417_20220428T034442_042968_05213C_28ED/annotation/rfi/rfi-iw-vh.xml\",\n \"type\": \"application/xml\",\n \"roles\": [\n \"metadata\"\n ],\n \"title\": \"Product Schema\",\n \"description\": \"Describes the main characteristics corresponding to the band: state of the platform during acquisition, image properties, Doppler information, geographic location, etc.\"\n },\n \"schema-product-vv\": {\n \"href\": \"https://sentinel1euwest.blob.core.windows.net/s1-grd/GRD/2022/4/28/IW/DV/S1A_IW_GRDH_1SDV_20220428T034417_20220428T034442_042968_05213C_28ED/annotation/rfi/rfi-iw-vv.xml\",\n \"type\": \"application/xml\",\n \"roles\": [\n \"metadata\"\n ],\n \"title\": \"Product Schema\",\n \"description\": \"Describes the main characteristics corresponding to the band: state of the platform during acquisition, image properties, Doppler information, geographic location, etc.\"\n },\n \"schema-calibration-vh\": {\n \"href\": \"https://sentinel1euwest.blob.core.windows.net/s1-grd/GRD/2022/4/28/IW/DV/S1A_IW_GRDH_1SDV_20220428T034417_20220428T034442_042968_05213C_28ED/annotation/calibration/calibration-iw-vh.xml\",\n \"type\": \"application/xml\",\n \"roles\": [\n \"metadata\"\n ],\n \"title\": \"Calibration Schema\",\n \"description\": \"Calibration metadata including calibration information and the beta nought, sigma nought, gamma and digital number look-up tables that can be used for absolute product calibration.\"\n },\n \"schema-calibration-vv\": {\n \"href\": \"https://sentinel1euwest.blob.core.windows.net/s1-grd/GRD/2022/4/28/IW/DV/S1A_IW_GRDH_1SDV_20220428T034417_20220428T034442_042968_05213C_28ED/annotation/calibration/calibration-iw-vv.xml\",\n \"type\": \"application/xml\",\n \"roles\": [\n \"metadata\"\n ],\n \"title\": \"Calibration Schema\",\n \"description\": \"Calibration metadata including calibration information and the beta nought, sigma nought, gamma and digital number look-up tables that can be used for absolute product calibration.\"\n }\n },\n \"geometry\": {\n \"type\": \"Polygon\",\n \"coordinates\": [\n [\n [\n 31.2672662,\n 2.6527481\n ],\n [\n 30.144719,\n 2.8900456\n ],\n [\n 29.0610893,\n 3.1179358\n ],\n [\n 28.948047,\n 2.5753315\n ],\n [\n 28.9109774,\n 2.3942923\n ],\n [\n 28.873316,\n 2.2133714\n ],\n [\n 28.8368458,\n 2.0321919\n ],\n [\n 28.801199,\n 1.8508305\n ],\n [\n 28.7523193,\n 1.6096632\n ],\n [\n 30.9486881,\n 1.1415594\n ],\n [\n 30.959178,\n 1.2021369\n ],\n [\n 30.9997364,\n 1.3830464\n ],\n [\n 31.0794822,\n 1.7451525\n ],\n [\n 31.1183746,\n 1.9264115\n ],\n [\n 31.1528986,\n 2.1085995\n ],\n [\n 31.1867151,\n 2.2909337\n ],\n [\n 31.2672662,\n 2.6527481\n ]\n ]\n ]\n },\n \"collection\": \"sentinel-1-grd\",\n \"properties\": {\n \"datetime\": \"2022-04-28T03:44:30.207391Z\",\n \"platform\": \"SENTINEL-1A\",\n \"s1:shape\": [\n 25547,\n 16823\n ],\n \"end_datetime\": \"2022-04-28 03:44:42.707207+00:00\",\n \"constellation\": \"Sentinel-1\",\n \"s1:resolution\": \"high\",\n \"s1:datatake_id\": \"336188\",\n \"start_datetime\": \"2022-04-28 03:44:17.707575+00:00\",\n \"s1:orbit_source\": \"RESORB\",\n \"s1:slice_number\": \"5\",\n \"s1:total_slices\": \"14\",\n \"sar:looks_range\": 5,\n \"sat:orbit_state\": \"descending\",\n \"sar:product_type\": \"GRD\",\n \"sar:looks_azimuth\": 1,\n \"sar:polarizations\": [\n \"VV\",\n \"VH\"\n ],\n \"sar:frequency_band\": \"C\",\n \"sat:absolute_orbit\": 42968,\n \"sat:relative_orbit\": 21,\n \"s1:processing_level\": \"1\",\n \"sar:instrument_mode\": \"IW\",\n \"sar:center_frequency\": 5.405,\n \"sar:resolution_range\": 20,\n \"s1:product_timeliness\": \"Fast-24h\",\n \"sar:resolution_azimuth\": 22,\n \"sar:pixel_spacing_range\": 10,\n \"sar:observation_direction\": \"right\",\n \"sar:pixel_spacing_azimuth\": 10,\n \"sar:looks_equivalent_number\": 4.4,\n \"s1:instrument_configuration_ID\": \"7\",\n \"sat:platform_international_designator\": \"2014-016A\"\n },\n \"stac_extensions\": [\n \"https://stac-extensions.github.io/sar/v1.0.0/schema.json\",\n \"https://stac-extensions.github.io/sat/v1.0.0/schema.json\",\n \"https://stac-extensions.github.io/eo/v1.0.0/schema.json\"\n ],\n \"stac_version\": \"1.0.0\"\n}" }, { "path": "src/pypgstac/tests/data-files/load/dehydrated.txt", "content": "chloris_biomass_50km_2017\t0103000020E6100000010000000500000066666666667E66C0000000000080564066666666667E66C00000000000004EC066666666667E66400000000000004EC066666666667E6640000000000080564066666666667E66C00000000000805640\tchloris-biomass\t2016-07-31 00:00:00+00\t2017-07-31 00:00:00+00\t{\"bbox\": [-179.95, -60.0, 179.95, 90.0], \"links\": [], \"assets\": {\"biomass\": {\"href\": \"https://ai4edataeuwest.blob.core.windows.net/chloris-biomass/cog/bio_2017.tif\", \"roles\": \"𒍟※\", \"file:size\": 20568072}, \"biomass_wm\": {\"href\": \"https://ai4edataeuwest.blob.core.windows.net/chloris-biomass/cog/bio_2017_merc.tif\", \"roles\": \"𒍟※\", \"file:size\": 44219262, \"proj:bbox\": [-20037508.337301563, -8400523.19027713, 20034408.51494577, 149088735.41090125], \"proj:shape\": [33992, 8649], \"proj:transform\": [4633.12716525001, 0.0, -20037508.337301563, 0.0, -4633.127165250011, 149088735.41090125]}, \"biomass_change\": {\"href\": \"https://ai4edataeuwest.blob.core.windows.net/chloris-biomass/cog/bio_change_2016-2017.tif\", \"roles\": \"𒍟※\", \"file:size\": 11101605}, \"biomass_change_wm\": {\"href\": \"https://ai4edataeuwest.blob.core.windows.net/chloris-biomass/cog/bio_change_2016-2017_merc.tif\", \"roles\": \"𒍟※\", \"file:size\": 25997181, \"proj:bbox\": [-20037508.337301563, -8400523.19027713, 20034408.51494577, 149088735.41090125], \"proj:shape\": [33992, 8649], \"proj:transform\": [4633.12716525001, 0.0, -20037508.337301563, 0.0, -4633.127165250011, 149088735.41090125]}}, \"properties\": {\"gsd\": 4633, \"datetime\": \"2017-01-01T00:00:00Z\", \"proj:bbox\": [-20015109.354, -6671703.11790004, 20015109.35376009, 10007554.677], \"proj:epsg\": null, \"proj:wkt2\": \"PROJCS[\\\\\"unnamed\\\\\",GEOGCS[\\\\\"unnamed ellipse\\\\\",DATUM[\\\\\"unknown\\\\\",SPHEROID[\\\\\"unnamed\\\\\",6371007.181,0]],PRIMEM[\\\\\"Greenwich\\\\\",0],UNIT[\\\\\"degree\\\\\",0.0174532925199433,AUTHORITY[\\\\\"EPSG\\\\\",\\\\\"9122\\\\\"]]],PROJECTION[\\\\\"Sinusoidal\\\\\"],PARAMETER[\\\\\"longitude_of_center\\\\\",0],PARAMETER[\\\\\"false_easting\\\\\",0],PARAMETER[\\\\\"false_northing\\\\\",0],UNIT[\\\\\"metre\\\\\",1,AUTHORITY[\\\\\"EPSG\\\\\",\\\\\"9001\\\\\"]],AXIS[\\\\\"Easting\\\\\",EAST],AXIS[\\\\\"Northing\\\\\",NORTH]]\", \"proj:shape\": [3600, 8640], \"end_datetime\": \"2017-07-31T00:00:00Z\", \"proj:transform\": [4633.12716525001, 0.0, -20015109.354, 0.0, -4633.127165250011, 10007554.677], \"start_datetime\": \"2016-07-31T00:00:00Z\"}, \"stac_extensions\": [\"https://stac-extensions.github.io/file/v2.0.0/schema.json\", \"https://stac-extensions.github.io/raster/v1.0.0/schema.json\", \"https://stac-extensions.github.io/projection/v1.0.0/schema.json\"]}\nchloris_biomass_50km_2018\t0103000020E6100000010000000500000066666666667E66C0000000000080564066666666667E66C00000000000004EC066666666667E66400000000000004EC066666666667E6640000000000080564066666666667E66C00000000000805640\tchloris-biomass\t2017-07-31 00:00:00+00\t2018-07-31 00:00:00+00\t{\"bbox\": [-179.95, -60.0, 179.95, 90.0], \"links\": [], \"assets\": {\"biomass\": {\"href\": \"https://ai4edataeuwest.blob.core.windows.net/chloris-biomass/cog/bio_2018.tif\", \"roles\": \"𒍟※\", \"file:size\": 20581566}, \"biomass_wm\": {\"href\": \"https://ai4edataeuwest.blob.core.windows.net/chloris-biomass/cog/bio_2018_merc.tif\", \"roles\": \"𒍟※\", \"file:size\": 44264008, \"proj:bbox\": [-20037508.337301563, -8400523.19027713, 20034408.51494577, 149088735.41090125], \"proj:shape\": [33992, 8649], \"proj:transform\": [4633.12716525001, 0.0, -20037508.337301563, 0.0, -4633.127165250011, 149088735.41090125]}, \"biomass_change\": {\"href\": \"https://ai4edataeuwest.blob.core.windows.net/chloris-biomass/cog/bio_change_2017-2018.tif\", \"roles\": \"𒍟※\", \"file:size\": 11278226}, \"biomass_change_wm\": {\"href\": \"https://ai4edataeuwest.blob.core.windows.net/chloris-biomass/cog/bio_change_2017-2018_merc.tif\", \"roles\": \"𒍟※\", \"file:size\": 26334351, \"proj:bbox\": [-20037508.337301563, -8400523.19027713, 20034408.51494577, 149088735.41090125], \"proj:shape\": [33992, 8649], \"proj:transform\": [4633.12716525001, 0.0, -20037508.337301563, 0.0, -4633.127165250011, 149088735.41090125]}}, \"properties\": {\"gsd\": 4633, \"datetime\": \"2018-01-01T00:00:00Z\", \"proj:bbox\": [-20015109.354, -6671703.11790004, 20015109.35376009, 10007554.677], \"proj:epsg\": null, \"proj:wkt2\": \"PROJCS[\\\\\"unnamed\\\\\",GEOGCS[\\\\\"unnamed ellipse\\\\\",DATUM[\\\\\"unknown\\\\\",SPHEROID[\\\\\"unnamed\\\\\",6371007.181,0]],PRIMEM[\\\\\"Greenwich\\\\\",0],UNIT[\\\\\"degree\\\\\",0.0174532925199433,AUTHORITY[\\\\\"EPSG\\\\\",\\\\\"9122\\\\\"]]],PROJECTION[\\\\\"Sinusoidal\\\\\"],PARAMETER[\\\\\"longitude_of_center\\\\\",0],PARAMETER[\\\\\"false_easting\\\\\",0],PARAMETER[\\\\\"false_northing\\\\\",0],UNIT[\\\\\"metre\\\\\",1,AUTHORITY[\\\\\"EPSG\\\\\",\\\\\"9001\\\\\"]],AXIS[\\\\\"Easting\\\\\",EAST],AXIS[\\\\\"Northing\\\\\",NORTH]]\", \"proj:shape\": [3600, 8640], \"end_datetime\": \"2018-07-31T00:00:00Z\", \"proj:transform\": [4633.12716525001, 0.0, -20015109.354, 0.0, -4633.127165250011, 10007554.677], \"start_datetime\": \"2017-07-31T00:00:00Z\"}, \"stac_extensions\": [\"https://stac-extensions.github.io/file/v2.0.0/schema.json\", \"https://stac-extensions.github.io/raster/v1.0.0/schema.json\", \"https://stac-extensions.github.io/projection/v1.0.0/schema.json\"]}\n" }, { "path": "src/pypgstac/tests/data-files/queryables/test_queryables.json", "content": "{\n \"$schema\": \"https://json-schema.org/draft/2019-09/schema\",\n \"$id\": \"https://example.com/stac/queryables\",\n \"type\": \"object\",\n \"title\": \"Test Queryables for PgSTAC\",\n \"description\": \"Test queryable names for PgSTAC\",\n \"properties\": {\n \"id\": {\n \"description\": \"Item identifier\",\n \"type\": \"string\"\n },\n \"collection\": {\n \"description\": \"Collection identifier\",\n \"type\": \"string\"\n },\n \"datetime\": {\n \"description\": \"Datetime\",\n \"type\": \"string\",\n \"format\": \"date-time\"\n },\n \"geometry\": {\n \"description\": \"Geometry\",\n \"type\": \"object\"\n },\n \"test:string_prop\": {\n \"description\": \"Test string property\",\n \"type\": \"string\"\n },\n \"test:number_prop\": {\n \"description\": \"Test number property\",\n \"type\": \"number\",\n \"minimum\": 0,\n \"maximum\": 100\n },\n \"test:integer_prop\": {\n \"description\": \"Test integer property\",\n \"type\": \"integer\"\n },\n \"test:datetime_prop\": {\n \"description\": \"Test datetime property\",\n \"type\": \"string\",\n \"format\": \"date-time\"\n },\n \"test:array_prop\": {\n \"description\": \"Test array property\",\n \"type\": \"array\",\n \"items\": {\n \"type\": \"string\"\n }\n }\n },\n \"additionalProperties\": true\n}\n" }, { "path": "src/pypgstac/tests/hydration/__init__.py", "content": "" }, { "path": "src/pypgstac/tests/hydration/test_base_item.py", "content": "import json\nfrom pathlib import Path\nfrom typing import Any, Dict, cast\n\nfrom pypgstac.load import Loader\n\nHERE = Path(__file__).parent\nLANDSAT_COLLECTION = (\n HERE / \"..\" / \"data-files\" / \"hydration\" / \"collections\" / \"landsat-c2-l1.json\"\n)\n\n\ndef test_landsat_c2_l1(loader: Loader) -> None:\n \"\"\"Test that a base item is created when a collection is loaded and that it\n is equal to the item_assets of the collection\n .\n \"\"\"\n with open(LANDSAT_COLLECTION) as f:\n collection = json.load(f)\n loader.load_collections(str(LANDSAT_COLLECTION))\n\n base_item = cast(\n Dict[str, Any],\n loader.db.query_one(\n \"SELECT base_item FROM collections WHERE id=%s;\",\n (collection[\"id\"],),\n ),\n )\n\n assert type(base_item) is dict\n assert base_item[\"collection\"] == collection[\"id\"]\n assert base_item[\"assets\"] == collection[\"item_assets\"]\n" }, { "path": "src/pypgstac/tests/hydration/test_dehydrate.py", "content": "import json\nfrom pathlib import Path\nfrom typing import Any, Dict, cast\n\nfrom pypgstac import hydration\nfrom pypgstac.hydration import DO_NOT_MERGE_MARKER\nfrom pypgstac.load import Loader\n\nHERE = Path(__file__).parent\nLANDSAT_COLLECTION = (\n HERE / \"..\" / \"data-files\" / \"hydration\" / \"collections\" / \"landsat-c2-l1.json\"\n)\nLANDSAT_ITEM = (\n HERE\n / \"..\"\n / \"data-files\"\n / \"hydration\"\n / \"raw-items\"\n / \"landsat-c2-l1\"\n / \"LM04_L1GS_001001_19830527_02_T2.json\"\n)\n\n\nclass TestDehydrate:\n def dehydrate(\n self,\n base_item: Dict[str, Any],\n item: Dict[str, Any],\n ) -> Dict[str, Any]:\n return hydration.dehydrate(base_item, item)\n\n def test_landsat_c2_l1(self, loader: Loader) -> None:\n \"\"\"\n Test that a dehydrated item is created properly from a raw item against a\n base item from a collection.\n \"\"\"\n with open(LANDSAT_COLLECTION) as f:\n collection = json.load(f)\n loader.load_collections(str(LANDSAT_COLLECTION))\n\n with open(LANDSAT_ITEM) as f:\n item = json.load(f)\n\n base_item = cast(\n Dict[str, Any],\n loader.db.query_one(\n \"SELECT base_item FROM collections WHERE id=%s;\",\n (collection[\"id\"],),\n ),\n )\n\n assert type(base_item) is dict\n\n dehydrated = self.dehydrate(base_item, item)\n\n # Expect certain keys on base and not on dehydrated\n only_base_keys = [\"type\", \"collection\", \"stac_version\"]\n assert all(k in base_item for k in only_base_keys)\n assert not any(k in dehydrated for k in only_base_keys)\n\n # Expect certain keys on dehydrated and not on base\n only_dehydrated_keys = [\"id\", \"bbox\", \"geometry\", \"properties\"]\n assert not any(k in base_item for k in only_dehydrated_keys)\n assert all(k in dehydrated for k in only_dehydrated_keys)\n\n # Properties, links should be exactly the same pre- and post-dehydration\n assert item[\"properties\"] == dehydrated[\"properties\"]\n assert item[\"links\"] == dehydrated[\"links\"]\n\n # Check specific assets are dehydrated correctly\n thumbnail = dehydrated[\"assets\"][\"thumbnail\"]\n assert list(thumbnail.keys()) == [\"href\"]\n assert thumbnail[\"href\"] == item[\"assets\"][\"thumbnail\"][\"href\"]\n\n # Red asset raster bands have additional `scale` and `offset` keys\n red = dehydrated[\"assets\"][\"red\"]\n assert list(red.keys()) == [\"href\", \"eo:bands\", \"raster:bands\"]\n assert len(red[\"raster:bands\"]) == 1\n assert list(red[\"raster:bands\"][0].keys()) == [\"scale\", \"offset\"]\n item_red_rb = item[\"assets\"][\"red\"][\"raster:bands\"][0]\n assert red[\"raster:bands\"] == [\n {\"scale\": item_red_rb[\"scale\"], \"offset\": item_red_rb[\"offset\"]},\n ]\n\n # nir09 asset raster bands does not have a `unit` attribute, which is\n # present on base\n nir09 = dehydrated[\"assets\"][\"nir09\"]\n assert list(nir09.keys()) == [\"href\", \"eo:bands\", \"raster:bands\"]\n assert len(nir09[\"raster:bands\"]) == 1\n assert list(nir09[\"raster:bands\"][0].keys()) == [\"unit\"]\n assert nir09[\"raster:bands\"] == [{\"unit\": DO_NOT_MERGE_MARKER}]\n\n def test_single_depth_equals(self) -> None:\n base_item = {\"a\": \"first\", \"b\": \"second\", \"c\": \"third\"}\n item = {\"a\": \"first\", \"b\": \"second\", \"c\": \"third\"}\n dehydrated = self.dehydrate(base_item, item)\n assert dehydrated == {}\n\n def test_nested_equals(self) -> None:\n base_item = {\"a\": \"first\", \"b\": \"second\", \"c\": {\"d\": \"third\"}}\n item = {\"a\": \"first\", \"b\": \"second\", \"c\": {\"d\": \"third\"}}\n dehydrated = self.dehydrate(base_item, item)\n assert dehydrated == {}\n\n def test_nested_extra_keys(self) -> None:\n \"\"\"\n Test that items having nested dicts with keys not in base item preserve\n the additional keys in the dehydrated item.\n \"\"\"\n base_item = {\"a\": \"first\", \"b\": \"second\", \"c\": {\"d\": \"third\"}}\n item = {\n \"a\": \"first\",\n \"b\": \"second\",\n \"c\": {\"d\": \"third\", \"e\": \"fourth\", \"f\": \"fifth\"},\n }\n dehydrated = self.dehydrate(base_item, item)\n assert dehydrated == {\"c\": {\"e\": \"fourth\", \"f\": \"fifth\"}}\n\n def test_list_of_dicts_extra_keys(self) -> None:\n \"\"\"Test that an equal length list of dicts is dehydrated correctly.\"\"\"\n base_item = {\"a\": [{\"b1\": 1, \"b2\": 2}, {\"c1\": 1, \"c2\": 2}]}\n item = {\"a\": [{\"b1\": 1, \"b2\": 2, \"b3\": 3}, {\"c1\": 1, \"c2\": 2, \"c3\": 3}]}\n\n dehydrated = self.dehydrate(base_item, item)\n assert \"a\" in dehydrated\n assert dehydrated[\"a\"] == [{\"b3\": 3}, {\"c3\": 3}]\n\n def test_equal_len_list_of_mixed_types(self) -> None:\n \"\"\"\n Test that a list of equal length containing matched\n types at each index dehydrates\n dicts and preserves item-values of other types.\n \"\"\"\n base_item = {\"a\": [{\"b1\": 1, \"b2\": 2}, \"foo\", {\"c1\": 1, \"c2\": 2}, \"bar\"]}\n item = {\n \"a\": [\n {\"b1\": 1, \"b2\": 2, \"b3\": 3},\n \"far\",\n {\"c1\": 1, \"c2\": 2, \"c3\": 3},\n \"boo\",\n ],\n }\n\n dehydrated = self.dehydrate(base_item, item)\n assert \"a\" in dehydrated\n assert dehydrated[\"a\"] == [{\"b3\": 3}, \"far\", {\"c3\": 3}, \"boo\"]\n\n def test_unequal_len_list(self) -> None:\n \"\"\"Test that unequal length lists preserve the item value exactly.\"\"\"\n base_item = {\"a\": [{\"b1\": 1}, {\"c1\": 1}, {\"d1\": 1}]}\n item = {\"a\": [{\"b1\": 1, \"b2\": 2}, {\"c1\": 1, \"c2\": 2}]}\n\n dehydrated = self.dehydrate(base_item, item)\n assert \"a\" in dehydrated\n assert dehydrated[\"a\"] == item[\"a\"]\n\n def test_marked_non_merged_fields(self) -> None:\n base_item = {\"a\": \"first\", \"b\": \"second\", \"c\": {\"d\": \"third\", \"e\": \"fourth\"}}\n item = {\n \"a\": \"first\",\n \"b\": \"second\",\n \"c\": {\"d\": \"third\", \"f\": \"fifth\"},\n }\n dehydrated = self.dehydrate(base_item, item)\n assert dehydrated == {\"c\": {\"e\": DO_NOT_MERGE_MARKER, \"f\": \"fifth\"}}\n\n def test_marked_non_merged_fields_in_list(self) -> None:\n base_item = {\n \"a\": [{\"b\": \"first\", \"d\": \"third\"}, {\"c\": \"second\", \"e\": \"fourth\"}],\n }\n item = {\"a\": [{\"b\": \"first\"}, {\"c\": \"second\", \"f\": \"fifth\"}]}\n\n dehydrated = self.dehydrate(base_item, item)\n assert dehydrated == {\n \"a\": [\n {\"d\": DO_NOT_MERGE_MARKER},\n {\"e\": DO_NOT_MERGE_MARKER, \"f\": \"fifth\"},\n ],\n }\n\n def test_deeply_nested_dict(self) -> None:\n base_item = {\"a\": {\"b\": {\"c\": {\"d\": \"first\", \"d1\": \"second\"}}}}\n item = {\"a\": {\"b\": {\"c\": {\"d\": \"first\", \"d1\": \"second\", \"d2\": \"third\"}}}}\n\n dehydrated = self.dehydrate(base_item, item)\n assert dehydrated == {\"a\": {\"b\": {\"c\": {\"d2\": \"third\"}}}}\n\n def test_equal_list_of_non_dicts(self) -> None:\n \"\"\"Values of lists that match base_item should be dehydrated off.\"\"\"\n base_item = {\"assets\": {\"thumbnail\": {\"roles\": [\"thumbnail\"]}}}\n item = {\n \"assets\": {\"thumbnail\": {\"roles\": [\"thumbnail\"], \"href\": \"http://foo.com\"}},\n }\n\n dehydrated = self.dehydrate(base_item, item)\n assert dehydrated == {\"assets\": {\"thumbnail\": {\"href\": \"http://foo.com\"}}}\n\n def test_invalid_assets_marked(self) -> None:\n \"\"\"\n Assets can be included on item-assets that are not uniformly included on\n individual items. Ensure that base item asset keys without a matching item\n key are marked do-no-merge after dehydration.\n \"\"\"\n base_item = {\n \"type\": \"Feature\",\n \"assets\": {\n \"asset1\": {\"name\": \"Asset one\"},\n \"asset2\": {\"name\": \"Asset two\"},\n },\n }\n hydrated = {\n \"assets\": {\"asset1\": {\"name\": \"Asset one\", \"href\": \"http://foo.com\"}},\n }\n\n dehydrated = self.dehydrate(base_item, hydrated)\n\n assert dehydrated == {\n \"type\": DO_NOT_MERGE_MARKER,\n \"assets\": {\n \"asset1\": {\"href\": \"http://foo.com\"},\n \"asset2\": DO_NOT_MERGE_MARKER,\n },\n }\n\n def test_top_level_base_keys_marked(self) -> None:\n \"\"\"\n Top level keys on the base item not present on the incoming item should\n be marked as do not merge, no matter the nesting level.\n \"\"\"\n base_item = {\n \"single\": \"Feature\",\n \"double\": {\"nested\": \"value\"},\n \"triple\": {\"nested\": {\"deep\": \"value\"}},\n \"included\": \"value\",\n }\n hydrated = {\"included\": \"value\", \"unique\": \"value\"}\n\n dehydrated = self.dehydrate(base_item, hydrated)\n\n assert dehydrated == {\n \"single\": DO_NOT_MERGE_MARKER,\n \"double\": DO_NOT_MERGE_MARKER,\n \"triple\": DO_NOT_MERGE_MARKER,\n \"unique\": \"value\",\n }\n" }, { "path": "src/pypgstac/tests/hydration/test_dehydrate_pg.py", "content": "import os\nfrom contextlib import contextmanager\nfrom typing import Any, Dict, Generator\n\nimport psycopg\n\nfrom pypgstac.db import PgstacDB\nfrom pypgstac.migrate import Migrate\n\nfrom .test_dehydrate import TestDehydrate as TDehydrate\n\n\nclass TestDehydratePG(TDehydrate):\n \"\"\"Class to test Dehydration using pgstac.\"\"\"\n\n @contextmanager\n def db(self) -> Generator:\n \"\"\"Set up database connection.\"\"\"\n origdb: str = os.getenv(\"PGDATABASE\", \"\")\n with psycopg.connect(autocommit=True) as conn:\n try:\n conn.execute(\"CREATE DATABASE pgstactestdb;\")\n except psycopg.errors.DuplicateDatabase:\n pass\n\n os.environ[\"PGDATABASE\"] = \"pgstactestdb\"\n\n pgdb = PgstacDB()\n with psycopg.connect(autocommit=True) as conn:\n conn.execute(\"DROP SCHEMA IF EXISTS pgstac CASCADE;\")\n Migrate(pgdb).run_migration()\n\n yield pgdb\n\n pgdb.close()\n os.environ[\"PGDATABASE\"] = origdb\n\n def dehydrate(\n self,\n base_item: Dict[str, Any],\n item: Dict[str, Any],\n ) -> Dict[str, Any]:\n \"\"\"Dehydrate item using pgstac.\"\"\"\n with self.db() as db:\n return next(db.func(\"strip_jsonb\", item, base_item))[0]\n" }, { "path": "src/pypgstac/tests/hydration/test_hydrate.py", "content": "\"\"\"Test Hydration.\"\"\"\n\nimport json\nfrom copy import deepcopy\nfrom pathlib import Path\nfrom typing import Any, Dict, cast\n\nfrom pypgstac import hydration\nfrom pypgstac.hydration import DO_NOT_MERGE_MARKER\nfrom pypgstac.load import Loader\n\nHERE = Path(__file__).parent\nLANDSAT_COLLECTION = (\n HERE / \"..\" / \"data-files\" / \"hydration\" / \"collections\" / \"landsat-c2-l1.json\"\n)\nLANDSAT_DEHYDRATED_ITEM = (\n HERE\n / \"..\"\n / \"data-files\"\n / \"hydration\"\n / \"dehydrated-items\"\n / \"landsat-c2-l1\"\n / \"LM04_L1GS_001001_19830527_02_T2.json\"\n)\n\nLANDSAT_ITEM = (\n HERE\n / \"..\"\n / \"data-files\"\n / \"hydration\"\n / \"raw-items\"\n / \"landsat-c2-l1\"\n / \"LM04_L1GS_001001_19830527_02_T2.json\"\n)\n\n\nclass TestHydrate:\n def hydrate(\n self,\n base_item: Dict[str, Any],\n item: Dict[str, Any],\n ) -> Dict[str, Any]:\n hpy = hydration.hydrate_py(deepcopy(base_item), deepcopy(item))\n hrs = hydration.hydrate(deepcopy(base_item), deepcopy(item))\n assert hpy == hrs\n return hrs\n\n def test_landsat_c2_l1(self, loader: Loader) -> None:\n \"\"\"Test that a dehydrated item is is equal to the raw item it was dehydrated\n from, against the base item of the collection\n .\n \"\"\"\n with open(LANDSAT_COLLECTION) as f:\n collection = json.load(f)\n loader.load_collections(str(LANDSAT_COLLECTION))\n\n with open(LANDSAT_DEHYDRATED_ITEM) as f:\n dehydrated = json.load(f)\n\n with open(LANDSAT_ITEM) as f:\n raw_item = json.load(f)\n\n base_item = cast(\n Dict[str, Any],\n loader.db.query_one(\n \"SELECT base_item FROM collections WHERE id=%s;\",\n (collection[\"id\"],),\n ),\n )\n\n assert type(base_item) is dict\n\n hydrated = self.hydrate(base_item, dehydrated)\n assert hydrated == raw_item\n\n def test_full_hydrate(self) -> None:\n base_item = {\"a\": \"first\", \"b\": \"second\", \"c\": \"third\"}\n dehydrated: Dict[str, Any] = {}\n\n rehydrated = self.hydrate(base_item, dehydrated)\n assert rehydrated == base_item\n\n def test_full_nested(self) -> None:\n base_item = {\"a\": \"first\", \"b\": \"second\", \"c\": {\"d\": \"third\"}}\n dehydrated: Dict[str, Any] = {}\n\n rehydrated = self.hydrate(base_item, dehydrated)\n assert rehydrated == base_item\n\n def test_nested_extra_keys(self) -> None:\n \"\"\"\n Test that items having nested dicts with keys not in base item preserve\n the additional keys in the dehydrated item.\n \"\"\"\n base_item = {\"a\": \"first\", \"b\": \"second\", \"c\": {\"d\": \"third\"}}\n dehydrated = {\"c\": {\"e\": \"fourth\", \"f\": \"fifth\"}}\n hydrated = self.hydrate(base_item, dehydrated)\n\n assert hydrated == {\n \"a\": \"first\",\n \"b\": \"second\",\n \"c\": {\"d\": \"third\", \"e\": \"fourth\", \"f\": \"fifth\"},\n }\n\n def test_list_of_dicts_extra_keys(self) -> None:\n \"\"\"Test that an equal length list of dicts is hydrated correctly.\"\"\"\n base_item = {\"a\": [{\"b1\": 1, \"b2\": 2}, {\"c1\": 1, \"c2\": 2}]}\n dehydrated = {\"a\": [{\"b3\": 3}, {\"c3\": 3}]}\n\n hydrated = self.hydrate(base_item, dehydrated)\n assert hydrated == {\n \"a\": [{\"b1\": 1, \"b2\": 2, \"b3\": 3}, {\"c1\": 1, \"c2\": 2, \"c3\": 3}],\n }\n\n def test_equal_len_list_of_mixed_types(self) -> None:\n \"\"\"\n Test that a list of equal length containing matched types at\n each index dehydrates\n dicts and preserves item-values of other types.\n \"\"\"\n base_item = {\"a\": [{\"b1\": 1, \"b2\": 2}, \"foo\", {\"c1\": 1, \"c2\": 2}, \"bar\"]}\n dehydrated = {\"a\": [{\"b3\": 3}, \"far\", {\"c3\": 3}, \"boo\"]}\n\n hydrated = self.hydrate(base_item, dehydrated)\n assert hydrated == {\n \"a\": [\n {\"b1\": 1, \"b2\": 2, \"b3\": 3},\n \"far\",\n {\"c1\": 1, \"c2\": 2, \"c3\": 3},\n \"boo\",\n ],\n }\n\n def test_unequal_len_list(self) -> None:\n \"\"\"Test that unequal length lists preserve the item value exactly.\"\"\"\n base_item = {\"a\": [{\"b1\": 1}, {\"c1\": 1}, {\"d1\": 1}]}\n dehydrated = {\"a\": [{\"b1\": 1, \"b2\": 2}, {\"c1\": 1, \"c2\": 2}]}\n\n hydrated = self.hydrate(base_item, dehydrated)\n assert hydrated == dehydrated\n\n def test_marked_non_merged_fields(self) -> None:\n base_item = {\n \"a\": \"first\",\n \"b\": \"second\",\n \"c\": {\"d\": \"third\", \"e\": \"fourth\"},\n }\n dehydrated = {\"c\": {\"e\": DO_NOT_MERGE_MARKER, \"f\": \"fifth\"}}\n\n hydrated = self.hydrate(base_item, dehydrated)\n assert hydrated == {\n \"a\": \"first\",\n \"b\": \"second\",\n \"c\": {\"d\": \"third\", \"f\": \"fifth\"},\n }\n\n def test_marked_non_merged_fields_in_list(self) -> None:\n base_item = {\n \"a\": [{\"b\": \"first\", \"d\": \"third\"}, {\"c\": \"second\", \"e\": \"fourth\"}],\n }\n dehydrated = {\n \"a\": [\n {\"d\": DO_NOT_MERGE_MARKER},\n {\"e\": DO_NOT_MERGE_MARKER, \"f\": \"fifth\"},\n ],\n }\n\n hydrated = self.hydrate(base_item, dehydrated)\n assert hydrated == {\"a\": [{\"b\": \"first\"}, {\"c\": \"second\", \"f\": \"fifth\"}]}\n\n def test_deeply_nested_dict(self) -> None:\n base_item = {\"a\": {\"b\": {\"c\": {\"d\": \"first\", \"d1\": \"second\"}}}}\n dehydrated = {\"a\": {\"b\": {\"c\": {\"d2\": \"third\"}}}}\n\n hydrated = self.hydrate(base_item, dehydrated)\n assert hydrated == {\n \"a\": {\"b\": {\"c\": {\"d\": \"first\", \"d1\": \"second\", \"d2\": \"third\"}}},\n }\n\n def test_equal_list_of_non_dicts(self) -> None:\n \"\"\"Values of lists that match base_item should be hydrated back on.\"\"\"\n base_item = {\"assets\": {\"thumbnail\": {\"roles\": [\"thumbnail\"]}}}\n dehydrated = {\"assets\": {\"thumbnail\": {\"href\": \"http://foo.com\"}}}\n\n hydrated = self.hydrate(base_item, dehydrated)\n assert hydrated == {\n \"assets\": {\"thumbnail\": {\"roles\": [\"thumbnail\"], \"href\": \"http://foo.com\"}},\n }\n\n def test_invalid_assets_removed(self) -> None:\n \"\"\"\n Assets can be included on item-assets that are not uniformly included on\n individual items. Ensure that item asset keys from base_item aren't included\n after hydration.\n \"\"\"\n base_item = {\n \"type\": \"Feature\",\n \"assets\": {\n \"asset1\": {\"name\": \"Asset one\"},\n \"asset2\": {\"name\": \"Asset two\"},\n },\n }\n\n dehydrated = {\n \"assets\": {\n \"asset1\": {\"href\": \"http://foo.com\"},\n \"asset2\": DO_NOT_MERGE_MARKER,\n },\n }\n\n hydrated = self.hydrate(base_item, dehydrated)\n\n assert hydrated == {\n \"type\": \"Feature\",\n \"assets\": {\"asset1\": {\"name\": \"Asset one\", \"href\": \"http://foo.com\"}},\n }\n\n def test_top_level_base_keys_marked(self) -> None:\n \"\"\"\n Top level keys on the base item not present on the incoming item should\n be marked as do not merge, no matter the nesting level.\n \"\"\"\n base_item = {\n \"single\": \"Feature\",\n \"double\": {\"nested\": \"value\"},\n \"triple\": {\"nested\": {\"deep\": \"value\"}},\n \"included\": \"value\",\n }\n\n dehydrated = {\n \"single\": DO_NOT_MERGE_MARKER,\n \"double\": DO_NOT_MERGE_MARKER,\n \"triple\": DO_NOT_MERGE_MARKER,\n \"unique\": \"value\",\n }\n\n hydrated = self.hydrate(base_item, dehydrated)\n\n assert hydrated == {\"included\": \"value\", \"unique\": \"value\"}\n\n def test_base_none(self) -> None:\n base_item = {\"value\": None}\n dehydrated = {\"value\": {\"a\": \"b\"}}\n hydrated = self.hydrate(base_item, dehydrated)\n assert hydrated == {\"value\": {\"a\": \"b\"}}\n" }, { "path": "src/pypgstac/tests/hydration/test_hydrate_pg.py", "content": "\"\"\"Test Hydration in PgSTAC.\"\"\"\n\nimport os\nfrom contextlib import contextmanager\nfrom typing import Any, Dict, Generator\n\nimport psycopg\n\nfrom pypgstac.db import PgstacDB\nfrom pypgstac.migrate import Migrate\n\nfrom .test_hydrate import TestHydrate as THydrate\n\n\nclass TestHydratePG(THydrate):\n \"\"\"Test hydration using PgSTAC.\"\"\"\n\n @contextmanager\n def db(self) -> Generator[PgstacDB, None, None]:\n \"\"\"Set up database.\"\"\"\n origdb: str = os.getenv(\"PGDATABASE\", \"\")\n with psycopg.connect(autocommit=True) as conn:\n try:\n conn.execute(\"CREATE DATABASE pgstactestdb;\")\n except psycopg.errors.DuplicateDatabase:\n pass\n\n os.environ[\"PGDATABASE\"] = \"pgstactestdb\"\n\n pgdb = PgstacDB()\n with psycopg.connect(autocommit=True) as conn:\n conn.execute(\"DROP SCHEMA IF EXISTS pgstac CASCADE;\")\n Migrate(pgdb).run_migration()\n\n yield pgdb\n\n pgdb.close()\n os.environ[\"PGDATABASE\"] = origdb\n\n def hydrate(\n self,\n base_item: Dict[str, Any],\n item: Dict[str, Any],\n ) -> Dict[str, Any]:\n \"\"\"Hydrate using pgstac.\"\"\"\n with self.db() as db:\n return next(db.func(\"content_hydrate\", item, base_item))[0]\n" }, { "path": "src/pypgstac/tests/test_benchmark.py", "content": "import json\nimport uuid\nfrom datetime import datetime, timezone\nfrom math import ceil\nfrom typing import Any, Dict, Generator, Tuple\n\nimport morecantile\nimport psycopg\nimport pytest\n\nfrom pypgstac.load import Loader, Methods\n\nXMIN, YMIN = 0, 0\nAOI_WIDTH = 50\nAOI_HEIGHT = 50\n\n\nITEM_WIDTHS = [0.5, 0.75, 1, 1.5, 2, 3, 4, 5, 6, 8, 10]\nTMS = morecantile.tms.get(\"WebMercatorQuad\")\n\n\ndef generate_items(\n item_size: Tuple[float, float],\n collection_id: str,\n) -> Generator[Dict[str, Any], None, None]:\n item_width, item_height = item_size\n\n cols = ceil(AOI_WIDTH / item_width)\n rows = ceil(AOI_HEIGHT / item_height)\n\n # generate an item for each grid cell\n for row in range(rows):\n for col in range(cols):\n left = XMIN + (col * item_width)\n bottom = YMIN + (row * item_height)\n right = left + item_width\n top = bottom + item_height\n\n yield {\n \"type\": \"Feature\",\n \"stac_version\": \"1.0.0\",\n \"id\": str(uuid.uuid4()),\n \"collection\": collection_id,\n \"geometry\": {\n \"type\": \"Polygon\",\n \"coordinates\": [\n [\n [left, bottom],\n [right, bottom],\n [right, top],\n [left, top],\n [left, bottom],\n ],\n ],\n },\n \"bbox\": [left, bottom, right, top],\n \"properties\": {\n \"datetime\": datetime.now(timezone.utc).isoformat(),\n },\n }\n\n\n@pytest.fixture(scope=\"function\")\ndef search_hashes(loader: Loader) -> Dict[float, str]:\n search_hashes = {}\n for item_width in ITEM_WIDTHS:\n collection_id = f\"collection-{str(item_width)}\"\n collection = {\n \"type\": \"Collection\",\n \"id\": collection_id,\n \"stac_version\": \"1.0.0\",\n \"description\": f\"Minimal test collection {collection_id}\",\n \"license\": \"proprietary\",\n \"extent\": {\n \"spatial\": {\n \"bbox\": [XMIN, YMIN, XMIN + AOI_WIDTH, YMIN + AOI_HEIGHT],\n },\n \"temporal\": {\n \"interval\": [[datetime.now(timezone.utc).isoformat(), None]],\n },\n },\n }\n\n loader.load_collections(\n iter([collection]),\n insert_mode=Methods.insert,\n )\n loader.load_items(\n generate_items((item_width, item_width), collection_id),\n insert_mode=Methods.insert,\n )\n\n with psycopg.connect(autocommit=True) as conn:\n with conn.cursor() as cursor:\n cursor.execute(\n \"SELECT * FROM search_query(%s);\",\n (json.dumps({\"collections\": [collection_id]}),),\n )\n res = cursor.fetchone()\n assert res\n search_hashes[item_width] = res[0]\n\n return search_hashes\n\n\n@pytest.mark.benchmark(\n group=\"xyzsearch\",\n min_rounds=3,\n warmup=True,\n warmup_iterations=2,\n)\n@pytest.mark.parametrize(\"item_width\", ITEM_WIDTHS)\n@pytest.mark.parametrize(\"zoom\", range(3, 8 + 1))\ndef test1(\n benchmark,\n search_hashes: Dict[float, str],\n item_width: float,\n zoom: int,\n) -> None:\n # get a tile from the center of the full AOI\n xmid = XMIN + AOI_WIDTH / 2\n ymid = YMIN + AOI_HEIGHT / 2\n tiles = TMS.tiles(xmid, ymid, xmid + 1, ymid + 1, [zoom])\n tile = next(tiles)\n\n def xyzsearch_test():\n with psycopg.connect(autocommit=True) as conn:\n with conn.cursor() as cursor:\n cursor.execute(\n \"SELECT * FROM xyzsearch(%s, %s, %s, %s, %s, %s, %s, %s, %s, %s);\",\n (\n tile.x,\n tile.y,\n tile.z,\n search_hashes[item_width],\n json.dumps(\n {\n \"include\": [\"assets\", \"id\", \"bbox\", \"collection\"],\n },\n ), # fields\n 100000, # scan_limit,\n 100000, # items limit\n \"5 seconds\",\n True, # exitwhenfull\n True, # skipcovered\n ),\n )\n row = cursor.fetchone()\n assert row is not None\n _ = row[0]\n\n _ = benchmark(xyzsearch_test)\n" }, { "path": "src/pypgstac/tests/test_load.py", "content": "\"\"\"Tests for pypgstac.\"\"\"\n\nimport json\nimport re\nimport threading\nfrom datetime import datetime, timezone\nfrom pathlib import Path\nfrom unittest import mock\n\nimport pytest\nfrom psycopg.errors import UniqueViolation\nfrom version_parser import Version as V\n\nfrom pypgstac.db import PgstacDB\nfrom pypgstac.load import Loader, Methods, __version__, read_json\n\nHERE = Path(__file__).parent\nTEST_DATA_DIR = HERE.parent.parent / \"pgstac\" / \"tests\" / \"testdata\"\nTEST_COLLECTIONS_JSON = TEST_DATA_DIR / \"collections.json\"\nTEST_COLLECTIONS = TEST_DATA_DIR / \"collections.ndjson\"\nTEST_ITEMS = TEST_DATA_DIR / \"items_private.ndjson\"\nTEST_DEHYDRATED_ITEMS = TEST_DATA_DIR / \"items.pgcopy\"\n\nS1_GRD_COLLECTION = (\n HERE / \"data-files\" / \"hydration\" / \"collections\" / \"sentinel-1-grd.json\"\n)\n\nS1_GRD_ITEM = (\n HERE\n / \"data-files\"\n / \"hydration\"\n / \"raw-items\"\n / \"sentinel-1-grd\"\n / \"S1A_IW_GRDH_1SDV_20220428T034417_20220428T034442_042968_05213C.json\"\n)\n\n\ndef version_increment(source_version: str) -> str:\n source_version = re.sub(\"-dev$\", \"\", source_version)\n version = V(source_version)\n return \".\".join(\n map(\n str,\n [\n version.get_major_version(),\n version.get_minor_version(),\n version.get_patch_version() + 1,\n ],\n ),\n )\n\n\ndef test_load_collections_succeeds(loader: Loader) -> None:\n \"\"\"Test pypgstac collections loader.\"\"\"\n loader.load_collections(\n str(TEST_COLLECTIONS),\n insert_mode=Methods.insert,\n )\n\n\ndef test_load_collections_json_succeeds(loader: Loader) -> None:\n \"\"\"Test pypgstac collections loader.\"\"\"\n loader.load_collections(\n str(TEST_COLLECTIONS_JSON),\n insert_mode=Methods.insert,\n )\n\n\ndef test_load_collections_json_duplicates_fails(loader: Loader) -> None:\n \"\"\"Test pypgstac collections loader.\"\"\"\n loader.load_collections(\n str(TEST_COLLECTIONS_JSON),\n insert_mode=Methods.insert,\n )\n with pytest.raises(UniqueViolation):\n loader.load_collections(\n str(TEST_COLLECTIONS_JSON),\n insert_mode=Methods.insert,\n )\n\n\ndef test_load_collections_json_duplicates_with_upsert(loader: Loader) -> None:\n \"\"\"Test pypgstac collections loader.\"\"\"\n loader.load_collections(\n str(TEST_COLLECTIONS_JSON),\n insert_mode=Methods.insert,\n )\n loader.load_collections(\n str(TEST_COLLECTIONS_JSON),\n insert_mode=Methods.upsert,\n )\n\n\ndef test_load_collections_json_duplicates_with_ignore(loader: Loader) -> None:\n \"\"\"Test pypgstac collections loader.\"\"\"\n loader.load_collections(\n str(TEST_COLLECTIONS_JSON),\n insert_mode=Methods.insert,\n )\n loader.load_collections(\n str(TEST_COLLECTIONS_JSON),\n insert_mode=Methods.ignore,\n )\n\n\ndef test_load_items_duplicates_fails(loader: Loader) -> None:\n \"\"\"Test pypgstac collections loader.\"\"\"\n loader.load_collections(\n str(TEST_COLLECTIONS),\n insert_mode=Methods.insert,\n )\n loader.load_items(\n str(TEST_ITEMS),\n insert_mode=Methods.insert,\n )\n\n with pytest.raises(UniqueViolation):\n loader.load_items(\n str(TEST_ITEMS),\n insert_mode=Methods.insert,\n )\n\n\ndef test_load_items_succeeds(loader: Loader) -> None:\n \"\"\"Test pypgstac items loader.\"\"\"\n loader.load_collections(\n str(TEST_COLLECTIONS),\n insert_mode=Methods.upsert,\n )\n\n loader.load_items(\n str(TEST_ITEMS),\n insert_mode=Methods.insert,\n )\n\n\ndef test_load_items_ignore_succeeds(loader: Loader) -> None:\n \"\"\"Test pypgstac items ignore loader.\"\"\"\n loader.load_collections(\n str(TEST_COLLECTIONS),\n insert_mode=Methods.ignore,\n )\n\n loader.load_items(\n str(TEST_ITEMS),\n insert_mode=Methods.insert,\n )\n\n loader.load_items(\n str(TEST_ITEMS),\n insert_mode=Methods.ignore,\n )\n\n\ndef test_load_items_upsert_succeeds(loader: Loader) -> None:\n \"\"\"Test pypgstac items ignore loader.\"\"\"\n loader.load_collections(\n str(TEST_COLLECTIONS),\n insert_mode=Methods.ignore,\n )\n\n loader.load_items(\n str(TEST_ITEMS),\n insert_mode=Methods.insert,\n )\n\n loader.load_items(\n str(TEST_ITEMS),\n insert_mode=Methods.upsert,\n )\n\n\ndef test_load_items_delsert_succeeds(loader: Loader) -> None:\n \"\"\"Test pypgstac items ignore loader.\"\"\"\n loader.load_collections(\n str(TEST_COLLECTIONS),\n insert_mode=Methods.ignore,\n )\n\n loader.load_items(\n str(TEST_ITEMS),\n insert_mode=Methods.insert,\n )\n\n loader.load_items(\n str(TEST_ITEMS),\n insert_mode=Methods.delsert,\n )\n\n\ndef test_partition_loads_default(loader: Loader) -> None:\n \"\"\"Test pypgstac items ignore loader.\"\"\"\n loader.load_collections(\n str(TEST_COLLECTIONS_JSON),\n insert_mode=Methods.ignore,\n )\n\n loader.load_items(\n str(TEST_ITEMS),\n insert_mode=Methods.insert,\n )\n\n partitions = loader.db.query_one(\n \"\"\"\n SELECT count(*) from partitions;\n \"\"\",\n )\n\n assert partitions == 1\n\n\ndef test_partition_loads_month(loader: Loader) -> None:\n \"\"\"Test pypgstac items ignore loader.\"\"\"\n loader.load_collections(\n str(TEST_COLLECTIONS_JSON),\n insert_mode=Methods.ignore,\n )\n if loader.db.connection is not None:\n loader.db.connection.execute(\n \"\"\"\n UPDATE collections SET partition_trunc='month';\n \"\"\",\n )\n\n loader.load_items(\n str(TEST_ITEMS),\n insert_mode=Methods.insert,\n )\n\n partitions = loader.db.query_one(\n \"\"\"\n SELECT count(*) from partitions;\n \"\"\",\n )\n\n assert partitions == 2\n\n\ndef test_partition_loads_year(loader: Loader) -> None:\n \"\"\"Test pypgstac items ignore loader.\"\"\"\n loader.load_collections(\n str(TEST_COLLECTIONS_JSON),\n insert_mode=Methods.ignore,\n )\n if loader.db.connection is not None:\n loader.db.connection.execute(\n \"\"\"\n UPDATE collections SET partition_trunc='year';\n \"\"\",\n )\n\n loader.load_items(\n str(TEST_ITEMS),\n insert_mode=Methods.insert,\n )\n\n partitions = loader.db.query_one(\n \"\"\"\n SELECT count(*) from partitions;\n \"\"\",\n )\n\n assert partitions == 1\n\n\ndef test_load_items_dehydrated_ignore_succeeds(loader: Loader) -> None:\n \"\"\"Test pypgstac items ignore loader.\"\"\"\n loader.load_collections(\n str(TEST_COLLECTIONS),\n insert_mode=Methods.ignore,\n )\n\n loader.load_items(\n str(TEST_DEHYDRATED_ITEMS),\n insert_mode=Methods.insert,\n dehydrated=True,\n )\n\n loader.load_items(\n str(TEST_DEHYDRATED_ITEMS),\n insert_mode=Methods.ignore,\n dehydrated=True,\n )\n\n\ndef test_format_items_keys(loader: Loader) -> None:\n \"\"\"Test pypgstac items ignore loader.\"\"\"\n loader.load_collections(\n str(TEST_COLLECTIONS_JSON),\n insert_mode=Methods.ignore,\n )\n\n items_iter = read_json(str(TEST_ITEMS))\n item_json = next(iter(items_iter))\n out = loader.format_item(item_json)\n\n # Top level keys expected after format\n assert \"id\" in out\n assert \"collection\" in out\n assert \"geometry\" in out\n assert \"content\" in out\n assert \"private\" in out\n\n # Special keys expected not to be in the item content\n content_json = json.loads(out[\"content\"])\n assert \"id\" not in content_json\n assert \"collection\" not in content_json\n assert \"geometry\" not in content_json\n assert \"private\" not in content_json\n\n # Ensure bbox is included in content\n assert \"bbox\" in content_json\n\n\ndef test_s1_grd_load_and_query(loader: Loader) -> None:\n \"\"\"Test pypgstac items ignore loader.\"\"\"\n loader.load_collections(\n str(S1_GRD_COLLECTION),\n insert_mode=Methods.ignore,\n )\n\n loader.load_items(str(S1_GRD_ITEM), insert_mode=Methods.insert)\n\n search_body = {\n \"filter-lang\": \"cql2-json\",\n \"filter\": {\n \"op\": \"and\",\n \"args\": [\n {\n \"op\": \"=\",\n \"args\": [{\"property\": \"collection\"}, \"sentinel-1-grd\"],\n },\n {\n \"op\": \"=\",\n \"args\": [\n {\"property\": \"id\"},\n \"S1A_IW_GRDH_1SDV_20220428T034417_20220428T034442_042968_05213C\", # noqa: E501\n ],\n },\n ],\n },\n }\n\n res = next(\n loader.db.func(\n \"search\",\n search_body,\n ),\n )[0]\n res[\"features\"][0]\n\n\ndef test_load_dehydrated(loader: Loader) -> None:\n \"\"\"Test loader for items dumped directly out of item table.\"\"\"\n collections = [\n HERE / \"data-files\" / \"hydration\" / \"collections\" / \"chloris-biomass.json\",\n ]\n\n for collection in collections:\n loader.load_collections(\n str(collection),\n insert_mode=Methods.ignore,\n )\n\n dehydrated_items = HERE / \"data-files\" / \"load\" / \"dehydrated.txt\"\n\n loader.load_items(\n str(dehydrated_items),\n insert_mode=Methods.insert,\n dehydrated=True,\n )\n\n\ndef test_load_collections_incompatible_version(loader: Loader) -> None:\n \"\"\"Test pypgstac collections loader raises an exception for incompatible version.\"\"\"\n with mock.patch(\n \"pypgstac.db.PgstacDB.version\",\n new_callable=mock.PropertyMock,\n ) as mock_version:\n mock_version.return_value = \"dummy\"\n with pytest.raises(ValueError):\n loader.load_collections(\n str(TEST_COLLECTIONS_JSON),\n insert_mode=Methods.insert,\n )\n\n\ndef test_load_items_incompatible_version(loader: Loader) -> None:\n \"\"\"Test pypgstac items loader raises an exception for incompatible version.\"\"\"\n loader.load_collections(\n str(TEST_COLLECTIONS_JSON),\n insert_mode=Methods.insert,\n )\n with mock.patch(\n \"pypgstac.db.PgstacDB.version\",\n new_callable=mock.PropertyMock,\n ) as mock_version:\n mock_version.return_value = \"dummy\"\n with pytest.raises(ValueError):\n loader.load_items(\n str(TEST_ITEMS),\n insert_mode=Methods.insert,\n )\n\n\ndef test_load_compatible_major_minor_version(loader: Loader) -> None:\n \"\"\"Test pypgstac loader doesn't raise an exception.\"\"\"\n with mock.patch(\n \"pypgstac.load.__version__\",\n version_increment(__version__),\n ) as mock_version:\n loader.load_collections(\n str(TEST_COLLECTIONS_JSON),\n insert_mode=Methods.insert,\n )\n loader.load_items(\n str(TEST_ITEMS),\n insert_mode=Methods.insert,\n )\n assert mock_version != loader.db.version\n\n\ndef test_load_compatible_major_minor_version_with_dev_suffix(loader: Loader) -> None:\n \"\"\"Test pypgstac loader accepts dev-suffixed library versions.\"\"\"\n with mock.patch(\n \"pypgstac.load.__version__\",\n f\"{version_increment(__version__)}-dev\",\n ):\n loader.load_collections(\n str(TEST_COLLECTIONS_JSON),\n insert_mode=Methods.insert,\n )\n\n\ndef test_load_items_nopartitionconstraint_succeeds(loader: Loader) -> None:\n \"\"\"Test pypgstac items loader.\"\"\"\n loader.load_collections(\n str(TEST_COLLECTIONS),\n insert_mode=Methods.upsert,\n )\n loader.load_items(\n str(TEST_ITEMS),\n insert_mode=Methods.insert,\n )\n\n cdtmin = loader.db.query_one(\n \"\"\"\n SELECT lower(constraint_dtrange)::text\n FROM partition_sys_meta WHERE partition = '_items_1';\n \"\"\",\n )\n\n assert cdtmin == \"2011-07-31 00:00:00+00\"\n with loader.db.connect() as conn:\n conn.execute(\n \"\"\"\n ALTER TABLE _items_1 DROP CONSTRAINT _items_1_dt;\n \"\"\",\n )\n cdtmin = loader.db.query_one(\n \"\"\"\n SELECT lower(constraint_dtrange)::text\n FROM partition_sys_meta WHERE partition = '_items_1';\n \"\"\",\n )\n assert cdtmin == \"-infinity\"\n\n loader.load_items(\n str(TEST_ITEMS),\n insert_mode=Methods.upsert,\n )\n cdtmin = loader.db.query_one(\n \"\"\"\n SELECT lower(constraint_dtrange)::text\n FROM partition_sys_meta WHERE partition = '_items_1';\n \"\"\",\n )\n assert cdtmin == \"2011-07-31 00:00:00+00\"\n\n\ndef test_valid_srid(loader: Loader) -> None:\n \"\"\"Test pypgstac items have a valid srid.\n\n https://github.com/stac-utils/pgstac/issues/357\n \"\"\"\n loader.load_collections(\n str(TEST_COLLECTIONS_JSON),\n insert_mode=Methods.ignore,\n )\n loader.load_items(\n str(TEST_ITEMS),\n insert_mode=Methods.insert,\n )\n srid = loader.db.query_one(\n \"\"\"\n SELECT st_srid(geometry) from items LIMIT 1;\n \"\"\",\n )\n assert isinstance(srid, int)\n assert srid > 0\n\n\ndef _make_item(item_id: str, collection: str, dt: str) -> dict:\n \"\"\"Create a minimal STAC item with the given id, collection, and datetime.\"\"\"\n return {\n \"id\": item_id,\n \"type\": \"Feature\",\n \"collection\": collection,\n \"geometry\": {\n \"type\": \"Polygon\",\n \"coordinates\": [\n [\n [-85.31, 30.93],\n [-85.31, 31.00],\n [-85.38, 31.00],\n [-85.38, 30.93],\n [-85.31, 30.93],\n ],\n ],\n },\n \"bbox\": [-85.38, 30.93, -85.31, 31.00],\n \"links\": [],\n \"assets\": {},\n \"properties\": {\n \"datetime\": dt,\n },\n \"stac_version\": \"1.0.0\",\n \"stac_extensions\": [],\n }\n\n\ndef test_load_items_sequential_new_loader_per_item(db: PgstacDB) -> None:\n \"\"\"Test that creating a new Loader per iteration with now() datetimes works.\n\n Reproduces a pattern where a for loop creates a fresh Loader for each\n iteration and loads a single item with datetime=now(). Each Loader has\n an empty _partition_cache, so it queries partition bounds from the DB\n each time. With slightly different datetimes, each iteration may trigger\n check_partition to drop and recreate constraints unnecessarily.\n \"\"\"\n # Load the collection once\n loader = Loader(db)\n loader.load_collections(str(TEST_COLLECTIONS), insert_mode=Methods.upsert)\n\n num_items = 10\n collection_id = \"pgstac-test-collection\"\n\n for i in range(num_items):\n # Fresh loader each iteration — empty _partition_cache\n ldr = Loader(db)\n dt = datetime.now(timezone.utc).isoformat()\n item = _make_item(f\"race-seq-{i}\", collection_id, dt)\n ldr.load_items(iter([item]), insert_mode=Methods.upsert)\n\n count = db.query_one(\"SELECT count(*) FROM items;\")\n assert count == num_items, (\n f\"Expected {num_items} items but found {count}. \"\n \"Sequential new-Loader-per-item with now() datetimes failed.\"\n )\n\n\ndef test_load_items_concurrent_new_loader_per_item(db: PgstacDB) -> None:\n \"\"\"Test race condition with concurrent Loaders each loading one item.\n\n This replicates the scenario where multiple threads each instantiate a\n separate Loader and call load_items with a single item whose datetime\n is set to now(). Each Loader has its own _partition_cache, and the\n slightly different datetimes cause each to call check_partition, which\n drops and recreates partition constraints and refreshes materialized\n views. Concurrent execution triggers deadlocks, lock contention, and\n constraint violations.\n \"\"\"\n # Load the collection once\n loader = Loader(db)\n loader.load_collections(str(TEST_COLLECTIONS), insert_mode=Methods.upsert)\n\n num_items = 10\n collection_id = \"pgstac-test-collection\"\n errors: list = []\n\n def load_one_item(item_idx: int) -> None:\n try:\n ldr = Loader(PgstacDB())\n dt = datetime.now(timezone.utc).isoformat()\n item = _make_item(f\"race-concurrent-{item_idx}\", collection_id, dt)\n ldr.load_items(iter([item]), insert_mode=Methods.upsert)\n except Exception as e:\n errors.append((item_idx, e))\n\n threads = []\n for i in range(num_items):\n t = threading.Thread(target=load_one_item, args=(i,))\n threads.append(t)\n\n # Start all threads to maximize contention\n for t in threads:\n t.start()\n for t in threads:\n t.join(timeout=60)\n\n # Report any errors from threads\n if errors:\n error_msgs = [f\"Item {idx}: {type(e).__name__}: {e}\" for idx, e in errors]\n message = f\"{len(errors)}/{num_items} concurrent loads failed:\\n\" + \"\\n\".join(\n error_msgs,\n )\n assert not errors, message\n\n count = db.query_one(\"SELECT count(*) FROM items;\")\n assert count == num_items, (\n f\"Expected {num_items} items but found {count}. \"\n \"Concurrent new-Loader-per-item with now() datetimes lost items.\"\n )\n" }, { "path": "src/pypgstac/tests/test_queryables.py", "content": "\"\"\"Tests for pypgstac queryables functionality.\"\"\"\n\nfrom pathlib import Path\nfrom unittest.mock import MagicMock, patch\n\nimport pytest\n\nfrom pypgstac.db import PgstacDB\nfrom pypgstac.load import Loader, Methods\nfrom pypgstac.pypgstac import PgstacCLI\n\nHERE = Path(__file__).parent\nTEST_DATA_DIR = HERE.parent.parent / \"pgstac\" / \"tests\" / \"testdata\"\nTEST_COLLECTIONS_JSON = TEST_DATA_DIR / \"collections.json\"\nTEST_QUERYABLES_JSON = HERE / \"data-files\" / \"queryables\" / \"test_queryables.json\"\n\n\ndef test_load_queryables_succeeds(db: PgstacDB) -> None:\n \"\"\"Test pypgstac queryables loader.\"\"\"\n # Create a CLI instance\n cli = PgstacCLI(dsn=db.dsn)\n\n # Load the test queryables with index_fields specified for all fields\n cli.load_queryables(\n str(TEST_QUERYABLES_JSON),\n index_fields=[\n \"test:string_prop\",\n \"test:number_prop\",\n \"test:integer_prop\",\n \"test:datetime_prop\",\n \"test:array_prop\",\n ],\n )\n\n # Verify that the queryables were loaded\n result = db.query(\n \"\"\"\n SELECT name, property_wrapper, property_index_type\n FROM queryables\n WHERE name LIKE 'test:%'\n ORDER BY name;\n \"\"\",\n )\n\n # Convert result to a list of dictionaries for easier assertion\n queryables = [\n {\"name\": row[0], \"property_wrapper\": row[1], \"property_index_type\": row[2]}\n for row in result\n ]\n\n # Check that all test properties were loaded with correct wrappers\n assert len(queryables) == 5\n\n # Check string property\n string_prop = next(q for q in queryables if q[\"name\"] == \"test:string_prop\")\n assert string_prop[\"property_wrapper\"] == \"to_text\"\n assert string_prop[\"property_index_type\"] == \"BTREE\"\n\n # Check number property\n number_prop = next(q for q in queryables if q[\"name\"] == \"test:number_prop\")\n assert number_prop[\"property_wrapper\"] == \"to_float\"\n assert number_prop[\"property_index_type\"] == \"BTREE\"\n\n # Check integer property\n integer_prop = next(q for q in queryables if q[\"name\"] == \"test:integer_prop\")\n assert integer_prop[\"property_wrapper\"] == \"to_int\"\n assert integer_prop[\"property_index_type\"] == \"BTREE\"\n\n # Check datetime property\n datetime_prop = next(q for q in queryables if q[\"name\"] == \"test:datetime_prop\")\n assert datetime_prop[\"property_wrapper\"] == \"to_tstz\"\n assert datetime_prop[\"property_index_type\"] == \"BTREE\"\n\n # Check array property\n array_prop = next(q for q in queryables if q[\"name\"] == \"test:array_prop\")\n assert array_prop[\"property_wrapper\"] == \"to_text_array\"\n assert array_prop[\"property_index_type\"] == \"BTREE\"\n\n\ndef test_load_queryables_without_index_fields(db: PgstacDB) -> None:\n \"\"\"Test pypgstac queryables loader without index_fields parameter.\"\"\"\n # Create a CLI instance\n cli = PgstacCLI(dsn=db.dsn)\n\n # Load the test queryables without index_fields\n cli.load_queryables(str(TEST_QUERYABLES_JSON))\n\n # Verify that the queryables were loaded without indexes\n result = db.query(\n \"\"\"\n SELECT name, property_wrapper, property_index_type\n FROM queryables\n WHERE name LIKE 'test:%'\n ORDER BY name;\n \"\"\",\n )\n\n # Convert result to a list of dictionaries for easier assertion\n queryables = [\n {\"name\": row[0], \"property_wrapper\": row[1], \"property_index_type\": row[2]}\n for row in result\n ]\n\n # Check that all test properties were loaded with correct wrappers but no indexes\n assert len(queryables) == 5\n\n # Check that none of the properties have indexes\n for q in queryables:\n assert q[\"property_index_type\"] is None\n\n\ndef test_load_queryables_with_specific_index_fields(db: PgstacDB) -> None:\n \"\"\"Test pypgstac queryables loader with specific index_fields.\"\"\"\n # Create a CLI instance\n cli = PgstacCLI(dsn=db.dsn)\n\n # Load the test queryables with only specific index_fields\n cli.load_queryables(\n str(TEST_QUERYABLES_JSON),\n index_fields=[\"test:string_prop\", \"test:datetime_prop\"],\n )\n\n # Verify that only the specified fields have indexes\n result = db.query(\n \"\"\"\n SELECT name, property_wrapper, property_index_type\n FROM queryables\n WHERE name LIKE 'test:%'\n ORDER BY name;\n \"\"\",\n )\n\n # Convert result to a list of dictionaries for easier assertion\n queryables = [\n {\"name\": row[0], \"property_wrapper\": row[1], \"property_index_type\": row[2]}\n for row in result\n ]\n\n # Check that all properties are loaded\n assert len(queryables) == 5\n\n # Check that only the specified fields have indexes\n for q in queryables:\n if q[\"name\"] in [\"test:string_prop\", \"test:datetime_prop\"]:\n assert q[\"property_index_type\"] == \"BTREE\"\n else:\n assert q[\"property_index_type\"] is None\n\n\ndef test_load_queryables_empty_index_fields(db: PgstacDB) -> None:\n \"\"\"Test pypgstac queryables loader with empty index_fields.\"\"\"\n # Create a CLI instance\n cli = PgstacCLI(dsn=db.dsn)\n\n # Load the test queryables with empty index_fields\n cli.load_queryables(\n str(TEST_QUERYABLES_JSON),\n index_fields=[],\n )\n\n # Verify that no fields have indexes\n result = db.query(\n \"\"\"\n SELECT name, property_wrapper, property_index_type\n FROM queryables\n WHERE name LIKE 'test:%'\n ORDER BY name;\n \"\"\",\n )\n\n # Convert result to a list of dictionaries for easier assertion\n queryables = [\n {\"name\": row[0], \"property_wrapper\": row[1], \"property_index_type\": row[2]}\n for row in result\n ]\n\n # Check that no fields have indexes\n for q in queryables:\n assert q[\"property_index_type\"] is None\n\n\n@patch(\"pypgstac.pypgstac.PgstacDB.connect\")\ndef test_maintain_partitions_called_only_with_index_fields(mock_connect):\n \"\"\"Test that maintain_partitions is only called when index_fields is provided.\"\"\"\n # Mock the database connection\n mock_conn = MagicMock()\n mock_connect.return_value = mock_conn\n\n # Mock cursor\n mock_cursor = MagicMock()\n mock_conn.cursor.return_value.__enter__.return_value = mock_cursor\n\n # Create a CLI instance with the mocked connection\n cli = PgstacCLI(dsn=\"mock_dsn\")\n\n # Create a temporary file with test queryables\n test_file = HERE / \"data-files\" / \"queryables\" / \"temp_test.json\"\n with open(test_file, \"w\") as f:\n f.write(\n \"\"\"\n {\n \"type\": \"object\",\n \"title\": \"Test Properties\",\n \"properties\": {\n \"test:prop1\": {\n \"type\": \"string\",\n \"title\": \"Test Property 1\"\n },\n \"test:prop2\": {\n \"type\": \"integer\",\n \"title\": \"Test Property 2\"\n }\n }\n }\n \"\"\",\n )\n\n # Case 1: With index_fields\n cli.load_queryables(\n str(test_file),\n index_fields=[\"test:prop1\"],\n )\n\n # Check that maintain_partitions was called\n maintain_calls = [\n call_args\n for call_args in mock_cursor.execute.call_args_list\n if \"maintain_partitions\" in str(call_args)\n ]\n assert len(maintain_calls) == 1\n\n # Reset mock\n mock_cursor.reset_mock()\n\n # Case 2: Without index_fields\n cli.load_queryables(str(test_file))\n\n # Check that maintain_partitions was not called\n maintain_calls = [\n call_args\n for call_args in mock_cursor.execute.call_args_list\n if \"maintain_partitions\" in str(call_args)\n ]\n assert len(maintain_calls) == 0\n\n # Clean up\n test_file.unlink()\n\n\ndef test_load_queryables_with_collections(db: PgstacDB, loader: Loader) -> None:\n \"\"\"Test pypgstac queryables loader with specific collections.\"\"\"\n # Load test collections first\n loader.load_collections(\n str(TEST_COLLECTIONS_JSON),\n insert_mode=Methods.insert,\n )\n\n # Get collection IDs from the database\n result = db.query(\"SELECT id FROM collections LIMIT 2;\")\n collection_ids = [row[0] for row in result]\n\n # Create a CLI instance\n cli = PgstacCLI(dsn=db.dsn)\n\n # Load queryables for specific collections\n cli.load_queryables(\n str(TEST_QUERYABLES_JSON),\n collection_ids=collection_ids,\n index_fields=[\"test:string_prop\"],\n )\n\n # Verify that the queryables were loaded with the correct collection IDs\n result = db.query(\n \"\"\"\n SELECT name, collection_ids, property_index_type\n FROM queryables\n WHERE name LIKE 'test:%'\n ORDER BY name;\n \"\"\",\n )\n\n # Convert result to a list of dictionaries for easier assertion\n queryables = [\n {\"name\": row[0], \"collection_ids\": row[1], \"property_index_type\": row[2]}\n for row in result\n ]\n\n # Check that all queryables have the correct collection IDs\n assert len(queryables) == 5\n for q in queryables:\n assert set(q[\"collection_ids\"]) == set(collection_ids)\n # Check that only test:string_prop has an index\n if q[\"name\"] == \"test:string_prop\":\n assert q[\"property_index_type\"] == \"BTREE\"\n else:\n assert q[\"property_index_type\"] is None\n\n\ndef test_load_queryables_update(db: PgstacDB) -> None:\n \"\"\"Test updating existing queryables.\"\"\"\n # Create a CLI instance\n cli = PgstacCLI(dsn=db.dsn)\n\n # Load the test queryables with an index on number_prop\n cli.load_queryables(str(TEST_QUERYABLES_JSON), index_fields=[\"test:number_prop\"])\n\n # Modify the test queryables file to change property wrappers\n # This is simulated by directly updating the database\n db.query(\n \"\"\"\n UPDATE queryables\n SET property_wrapper = 'to_text'\n WHERE name = 'test:number_prop';\n \"\"\",\n )\n\n # Load the queryables again, but with a different index field\n cli.load_queryables(str(TEST_QUERYABLES_JSON), index_fields=[\"test:string_prop\"])\n\n # Verify that the property wrapper was updated and index changed\n result = db.query(\n \"\"\"\n SELECT name, property_wrapper, property_index_type\n FROM queryables\n WHERE name in ('test:number_prop', 'test:string_prop');\n \"\"\",\n )\n\n # Convert result to a list of dictionaries for easier assertion\n queryables = [\n {\"name\": row[0], \"property_wrapper\": row[1], \"property_index_type\": row[2]}\n for row in result\n ]\n\n # Find the properties\n number_prop = next(q for q in queryables if q[\"name\"] == \"test:number_prop\")\n string_prop = next(q for q in queryables if q[\"name\"] == \"test:string_prop\")\n\n # The property wrapper should be back to to_float\n assert number_prop[\"property_wrapper\"] == \"to_float\"\n # The index should be removed from number_prop\n assert number_prop[\"property_index_type\"] is None\n # The index should be added to string_prop\n assert string_prop[\"property_index_type\"] == \"BTREE\"\n\n\ndef test_load_queryables_invalid_json(db: PgstacDB) -> None:\n \"\"\"Test loading queryables with invalid JSON.\"\"\"\n # Create a CLI instance\n cli = PgstacCLI(dsn=db.dsn)\n\n # Create a temporary file with invalid JSON\n invalid_json_file = HERE / \"data-files\" / \"queryables\" / \"invalid.json\"\n with open(invalid_json_file, \"w\") as f:\n f.write(\"{\")\n\n # Loading should raise an exception\n with pytest.raises((ValueError, SyntaxError)):\n cli.load_queryables(str(invalid_json_file))\n\n # Clean up\n invalid_json_file.unlink()\n\n\ndef test_load_queryables_delete_missing(db: PgstacDB) -> None:\n \"\"\"Test loading queryables with delete_missing=True.\"\"\"\n # Create a CLI instance\n cli = PgstacCLI(dsn=db.dsn)\n\n # First, load the test queryables with indexes on all fields\n cli.load_queryables(\n str(TEST_QUERYABLES_JSON),\n index_fields=[\n \"test:string_prop\",\n \"test:number_prop\",\n \"test:integer_prop\",\n \"test:datetime_prop\",\n \"test:array_prop\",\n ],\n )\n\n # Create a temporary file with only one property\n partial_props_file = HERE / \"data-files\" / \"queryables\" / \"partial_props.json\"\n with open(partial_props_file, \"w\") as f:\n f.write(\n \"\"\"\n {\n \"type\": \"object\",\n \"title\": \"Partial Properties\",\n \"properties\": {\n \"test:string_prop\": {\n \"type\": \"string\",\n \"title\": \"String Property\"\n }\n }\n }\n \"\"\",\n )\n\n # Load the partial queryables with delete_missing=True and index the string property\n cli.load_queryables(\n str(partial_props_file),\n delete_missing=True,\n index_fields=[\"test:string_prop\"],\n )\n\n # Verify that only the string property remains and has an index\n result = db.query(\n \"\"\"\n SELECT name, property_index_type\n FROM queryables\n WHERE name LIKE 'test:%'\n ORDER BY name;\n \"\"\",\n )\n\n # Convert result to a list of dictionaries\n queryables = [{\"name\": row[0], \"property_index_type\": row[1]} for row in result]\n\n # Check that only the string property remains and has an index\n assert len(queryables) == 1\n assert queryables[0][\"name\"] == \"test:string_prop\"\n assert queryables[0][\"property_index_type\"] == \"BTREE\"\n\n # Clean up\n partial_props_file.unlink()\n\n\ndef test_load_queryables_delete_missing_with_collections(\n db: PgstacDB,\n loader: Loader,\n) -> None:\n \"\"\"Test loading queryables with delete_missing=True and specific collections.\"\"\"\n # Load test collections first\n loader.load_collections(\n str(TEST_COLLECTIONS_JSON),\n insert_mode=Methods.insert,\n )\n\n # Get collection IDs from the database\n result = db.query(\"SELECT id FROM collections LIMIT 2;\")\n collection_ids = [row[0] for row in result]\n\n # Create a CLI instance\n cli = PgstacCLI(dsn=db.dsn)\n\n # First, load all test queryables for the specific collections with indexes\n cli.load_queryables(\n str(TEST_QUERYABLES_JSON),\n collection_ids=collection_ids,\n index_fields=[\n \"test:string_prop\",\n \"test:number_prop\",\n \"test:integer_prop\",\n \"test:datetime_prop\",\n \"test:array_prop\",\n ],\n )\n\n # Create a temporary file with only one property\n partial_props_file = HERE / \"data-files\" / \"queryables\" / \"partial_props.json\"\n with open(partial_props_file, \"w\") as f:\n f.write(\n \"\"\"\n {\n \"type\": \"object\",\n \"title\": \"Partial Properties\",\n \"properties\": {\n \"test:string_prop\": {\n \"type\": \"string\",\n \"title\": \"String Property\"\n }\n }\n }\n \"\"\",\n )\n\n # Load the partial queryables with delete_missing=True for the specific collections\n # but without an index\n cli.load_queryables(\n str(partial_props_file),\n collection_ids=collection_ids,\n delete_missing=True,\n )\n\n # Verify that only the string property remains for the specific collections\n # and that it doesn't have an index\n result = db.query(\n \"\"\"\n SELECT name, collection_ids, property_index_type\n FROM queryables\n WHERE name LIKE 'test:%'\n ORDER BY name;\n \"\"\",\n )\n\n # Convert result to a list of dictionaries\n queryables = [\n {\"name\": row[0], \"collection_ids\": row[1], \"property_index_type\": row[2]}\n for row in result\n ]\n\n # Filter queryables for the specific collections\n specific_queryables = [\n q\n for q in queryables\n if q[\"collection_ids\"] and set(q[\"collection_ids\"]) == set(collection_ids)\n ]\n\n # Check that only the string property remains for the specific collections\n assert len(specific_queryables) == 1\n assert specific_queryables[0][\"name\"] == \"test:string_prop\"\n # Verify it doesn't have an index\n assert specific_queryables[0][\"property_index_type\"] is None\n\n # Clean up\n partial_props_file.unlink()\n\n\ndef test_load_queryables_no_properties(db: PgstacDB) -> None:\n \"\"\"Test loading queryables with no properties.\"\"\"\n # Create a CLI instance\n cli = PgstacCLI(dsn=db.dsn)\n\n # Create a temporary file with no properties\n no_props_file = HERE / \"data-files\" / \"queryables\" / \"no_props.json\"\n with open(no_props_file, \"w\") as f:\n f.write('{\"type\": \"object\", \"title\": \"No Properties\"}')\n\n # Loading should raise a ValueError\n with pytest.raises(\n ValueError,\n match=\"No properties found in queryables definition\",\n ):\n cli.load_queryables(str(no_props_file))\n\n # Clean up\n no_props_file.unlink()\n" } ]